diff --git a/.clang-format b/.clang-format index f3923a1f98583bef70d7beeac5954da858079a3c..2ffd69afc1a8280654a8763e044e27e08be7a69f 100644 --- a/.clang-format +++ b/.clang-format @@ -387,14 +387,14 @@ ForEachMacros: - 'rhl_for_each_entry_rcu' - 'rhl_for_each_rcu' - 'rht_for_each' - - 'rht_for_each_continue' + - 'rht_for_each_from' - 'rht_for_each_entry' - - 'rht_for_each_entry_continue' + - 'rht_for_each_entry_from' - 'rht_for_each_entry_rcu' - - 'rht_for_each_entry_rcu_continue' + - 'rht_for_each_entry_rcu_from' - 'rht_for_each_entry_safe' - 'rht_for_each_rcu' - - 'rht_for_each_rcu_continue' + - 'rht_for_each_rcu_from' - '__rq_for_each_bio' - 'rq_for_each_bvec' - 'rq_for_each_segment' diff --git a/.mailmap b/.mailmap index 907a5ffd3460ed305f4bd852a16d00c6b0ad3af8..0049c0b49056e3217852a1d11be98c7b3fc76962 100644 --- a/.mailmap +++ b/.mailmap @@ -16,6 +16,9 @@ Alan Cox Alan Cox Aleksey Gorelov Aleksandar Markovic +Alexei Starovoitov +Alexei Starovoitov +Alexei Starovoitov Al Viro Al Viro Andi Shyti @@ -46,6 +49,12 @@ Christoph Hellwig Christophe Ricard Corey Minyard Damian Hobson-Garcia +Daniel Borkmann +Daniel Borkmann +Daniel Borkmann +Daniel Borkmann +Daniel Borkmann +Daniel Borkmann David Brownell David Woodhouse Dengcheng Zhu diff --git a/Documentation/ABI/testing/sysfs-class-net-batman-adv b/Documentation/ABI/obsolete/sysfs-class-net-batman-adv similarity index 92% rename from Documentation/ABI/testing/sysfs-class-net-batman-adv rename to Documentation/ABI/obsolete/sysfs-class-net-batman-adv index 898106849e27f520490907af63260ccfca00de1f..5bdbc8d402561acb01e9722ccb11498ce9ff4b82 100644 --- a/Documentation/ABI/testing/sysfs-class-net-batman-adv +++ b/Documentation/ABI/obsolete/sysfs-class-net-batman-adv @@ -1,3 +1,5 @@ +This ABI is deprecated and will be removed after 2021. It is +replaced with the batadv generic netlink family. What: /sys/class/net//batman-adv/elp_interval Date: Feb 2014 diff --git a/Documentation/ABI/testing/sysfs-class-net-mesh b/Documentation/ABI/obsolete/sysfs-class-net-mesh similarity index 97% rename from Documentation/ABI/testing/sysfs-class-net-mesh rename to Documentation/ABI/obsolete/sysfs-class-net-mesh index c2b956d44a9520d276b7f13ee3d2617991127a5a..04c1a2932507eb65209b12f7465ed5ff55185bf9 100644 --- a/Documentation/ABI/testing/sysfs-class-net-mesh +++ b/Documentation/ABI/obsolete/sysfs-class-net-mesh @@ -1,3 +1,5 @@ +This ABI is deprecated and will be removed after 2021. It is +replaced with the batadv generic netlink family. What: /sys/class/net//mesh/aggregated_ogms Date: May 2010 diff --git a/Documentation/bpf/bpf_design_QA.rst b/Documentation/bpf/bpf_design_QA.rst index 10453c6271359c3f856f55ba1d72a56fd449fd96..cb402c59eca58eb01976f5db6ad1905de4ddf397 100644 --- a/Documentation/bpf/bpf_design_QA.rst +++ b/Documentation/bpf/bpf_design_QA.rst @@ -85,8 +85,33 @@ Q: Can loops be supported in a safe way? A: It's not clear yet. BPF developers are trying to find a way to -support bounded loops where the verifier can guarantee that -the program terminates in less than 4096 instructions. +support bounded loops. + +Q: What are the verifier limits? +-------------------------------- +A: The only limit known to the user space is BPF_MAXINSNS (4096). +It's the maximum number of instructions that the unprivileged bpf +program can have. The verifier has various internal limits. +Like the maximum number of instructions that can be explored during +program analysis. Currently, that limit is set to 1 million. +Which essentially means that the largest program can consist +of 1 million NOP instructions. There is a limit to the maximum number +of subsequent branches, a limit to the number of nested bpf-to-bpf +calls, a limit to the number of the verifier states per instruction, +a limit to the number of maps used by the program. +All these limits can be hit with a sufficiently complex program. +There are also non-numerical limits that can cause the program +to be rejected. The verifier used to recognize only pointer + constant +expressions. Now it can recognize pointer + bounded_register. +bpf_lookup_map_elem(key) had a requirement that 'key' must be +a pointer to the stack. Now, 'key' can be a pointer to map value. +The verifier is steadily getting 'smarter'. The limits are +being removed. The only way to know that the program is going to +be accepted by the verifier is to try to load it. +The bpf development process guarantees that the future kernel +versions will accept all bpf programs that were accepted by +the earlier versions. + Instruction level questions --------------------------- diff --git a/Documentation/bpf/btf.rst b/Documentation/bpf/btf.rst index 7313d354f20e6402e23161f24ddab227afb7c0a1..29396e6943b036ffa11b0ed5d54cfe35533c48b6 100644 --- a/Documentation/bpf/btf.rst +++ b/Documentation/bpf/btf.rst @@ -82,6 +82,8 @@ sequentially and type id is assigned to each recognized type starting from id #define BTF_KIND_RESTRICT 11 /* Restrict */ #define BTF_KIND_FUNC 12 /* Function */ #define BTF_KIND_FUNC_PROTO 13 /* Function Proto */ + #define BTF_KIND_VAR 14 /* Variable */ + #define BTF_KIND_DATASEC 15 /* Section */ Note that the type section encodes debug info, not just pure types. ``BTF_KIND_FUNC`` is not a type, and it represents a defined subprogram. @@ -393,6 +395,61 @@ refers to parameter type. If the function has variable arguments, the last parameter is encoded with ``name_off = 0`` and ``type = 0``. +2.2.14 BTF_KIND_VAR +~~~~~~~~~~~~~~~~~~~ + +``struct btf_type`` encoding requirement: + * ``name_off``: offset to a valid C identifier + * ``info.kind_flag``: 0 + * ``info.kind``: BTF_KIND_VAR + * ``info.vlen``: 0 + * ``type``: the type of the variable + +``btf_type`` is followed by a single ``struct btf_variable`` with the +following data:: + + struct btf_var { + __u32 linkage; + }; + +``struct btf_var`` encoding: + * ``linkage``: currently only static variable 0, or globally allocated + variable in ELF sections 1 + +Not all type of global variables are supported by LLVM at this point. +The following is currently available: + + * static variables with or without section attributes + * global variables with section attributes + +The latter is for future extraction of map key/value type id's from a +map definition. + +2.2.15 BTF_KIND_DATASEC +~~~~~~~~~~~~~~~~~~~~~~~ + +``struct btf_type`` encoding requirement: + * ``name_off``: offset to a valid name associated with a variable or + one of .data/.bss/.rodata + * ``info.kind_flag``: 0 + * ``info.kind``: BTF_KIND_DATASEC + * ``info.vlen``: # of variables + * ``size``: total section size in bytes (0 at compilation time, patched + to actual size by BPF loaders such as libbpf) + +``btf_type`` is followed by ``info.vlen`` number of ``struct btf_var_secinfo``.:: + + struct btf_var_secinfo { + __u32 type; + __u32 offset; + __u32 size; + }; + +``struct btf_var_secinfo`` encoding: + * ``type``: the type of the BTF_KIND_VAR variable + * ``offset``: the in-section offset of the variable + * ``size``: the size of the variable in bytes + 3. BTF Kernel API ***************** diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst index 4e77932959cc4d18ee9ce6652cf441ba1f8d68b7..d3fe4cac0c90a67c4072341ee8ca20ae9c62e573 100644 --- a/Documentation/bpf/index.rst +++ b/Documentation/bpf/index.rst @@ -36,6 +36,16 @@ Two sets of Questions and Answers (Q&A) are maintained. bpf_devel_QA +Program types +============= + +.. toctree:: + :maxdepth: 1 + + prog_cgroup_sysctl + prog_flow_dissector + + .. Links: .. _Documentation/networking/filter.txt: ../networking/filter.txt .. _man-pages: https://www.kernel.org/doc/man-pages/ diff --git a/Documentation/bpf/prog_cgroup_sysctl.rst b/Documentation/bpf/prog_cgroup_sysctl.rst new file mode 100644 index 0000000000000000000000000000000000000000..677d6c637cf3d977db1b73d0113efd46ec8f01e2 --- /dev/null +++ b/Documentation/bpf/prog_cgroup_sysctl.rst @@ -0,0 +1,125 @@ +.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) + +=========================== +BPF_PROG_TYPE_CGROUP_SYSCTL +=========================== + +This document describes ``BPF_PROG_TYPE_CGROUP_SYSCTL`` program type that +provides cgroup-bpf hook for sysctl. + +The hook has to be attached to a cgroup and will be called every time a +process inside that cgroup tries to read from or write to sysctl knob in proc. + +1. Attach type +************** + +``BPF_CGROUP_SYSCTL`` attach type has to be used to attach +``BPF_PROG_TYPE_CGROUP_SYSCTL`` program to a cgroup. + +2. Context +********** + +``BPF_PROG_TYPE_CGROUP_SYSCTL`` provides access to the following context from +BPF program:: + + struct bpf_sysctl { + __u32 write; + __u32 file_pos; + }; + +* ``write`` indicates whether sysctl value is being read (``0``) or written + (``1``). This field is read-only. + +* ``file_pos`` indicates file position sysctl is being accessed at, read + or written. This field is read-write. Writing to the field sets the starting + position in sysctl proc file ``read(2)`` will be reading from or ``write(2)`` + will be writing to. Writing zero to the field can be used e.g. to override + whole sysctl value by ``bpf_sysctl_set_new_value()`` on ``write(2)`` even + when it's called by user space on ``file_pos > 0``. Writing non-zero + value to the field can be used to access part of sysctl value starting from + specified ``file_pos``. Not all sysctl support access with ``file_pos != + 0``, e.g. writes to numeric sysctl entries must always be at file position + ``0``. See also ``kernel.sysctl_writes_strict`` sysctl. + +See `linux/bpf.h`_ for more details on how context field can be accessed. + +3. Return code +************** + +``BPF_PROG_TYPE_CGROUP_SYSCTL`` program must return one of the following +return codes: + +* ``0`` means "reject access to sysctl"; +* ``1`` means "proceed with access". + +If program returns ``0`` user space will get ``-1`` from ``read(2)`` or +``write(2)`` and ``errno`` will be set to ``EPERM``. + +4. Helpers +********** + +Since sysctl knob is represented by a name and a value, sysctl specific BPF +helpers focus on providing access to these properties: + +* ``bpf_sysctl_get_name()`` to get sysctl name as it is visible in + ``/proc/sys`` into provided by BPF program buffer; + +* ``bpf_sysctl_get_current_value()`` to get string value currently held by + sysctl into provided by BPF program buffer. This helper is available on both + ``read(2)`` from and ``write(2)`` to sysctl; + +* ``bpf_sysctl_get_new_value()`` to get new string value currently being + written to sysctl before actual write happens. This helper can be used only + on ``ctx->write == 1``; + +* ``bpf_sysctl_set_new_value()`` to override new string value currently being + written to sysctl before actual write happens. Sysctl value will be + overridden starting from the current ``ctx->file_pos``. If the whole value + has to be overridden BPF program can set ``file_pos`` to zero before calling + to the helper. This helper can be used only on ``ctx->write == 1``. New + string value set by the helper is treated and verified by kernel same way as + an equivalent string passed by user space. + +BPF program sees sysctl value same way as user space does in proc filesystem, +i.e. as a string. Since many sysctl values represent an integer or a vector +of integers, the following helpers can be used to get numeric value from the +string: + +* ``bpf_strtol()`` to convert initial part of the string to long integer + similar to user space `strtol(3)`_; +* ``bpf_strtoul()`` to convert initial part of the string to unsigned long + integer similar to user space `strtoul(3)`_; + +See `linux/bpf.h`_ for more details on helpers described here. + +5. Examples +*********** + +See `test_sysctl_prog.c`_ for an example of BPF program in C that access +sysctl name and value, parses string value to get vector of integers and uses +the result to make decision whether to allow or deny access to sysctl. + +6. Notes +******** + +``BPF_PROG_TYPE_CGROUP_SYSCTL`` is intended to be used in **trusted** root +environment, for example to monitor sysctl usage or catch unreasonable values +an application, running as root in a separate cgroup, is trying to set. + +Since `task_dfl_cgroup(current)` is called at `sys_read` / `sys_write` time it +may return results different from that at `sys_open` time, i.e. process that +opened sysctl file in proc filesystem may differ from process that is trying +to read from / write to it and two such processes may run in different +cgroups, what means ``BPF_PROG_TYPE_CGROUP_SYSCTL`` should not be used as a +security mechanism to limit sysctl usage. + +As with any cgroup-bpf program additional care should be taken if an +application running as root in a cgroup should not be allowed to +detach/replace BPF program attached by administrator. + +.. Links +.. _linux/bpf.h: ../../include/uapi/linux/bpf.h +.. _strtol(3): http://man7.org/linux/man-pages/man3/strtol.3p.html +.. _strtoul(3): http://man7.org/linux/man-pages/man3/strtoul.3p.html +.. _test_sysctl_prog.c: + ../../tools/testing/selftests/bpf/progs/test_sysctl_prog.c diff --git a/Documentation/networking/bpf_flow_dissector.rst b/Documentation/bpf/prog_flow_dissector.rst similarity index 98% rename from Documentation/networking/bpf_flow_dissector.rst rename to Documentation/bpf/prog_flow_dissector.rst index b375ae2ec2c4f43d4c59a5f12cb26b3a8f205df5..ed343abe541e120738acf8501eab3a02510af76b 100644 --- a/Documentation/networking/bpf_flow_dissector.rst +++ b/Documentation/bpf/prog_flow_dissector.rst @@ -1,8 +1,8 @@ .. SPDX-License-Identifier: GPL-2.0 -================== -BPF Flow Dissector -================== +============================ +BPF_PROG_TYPE_FLOW_DISSECTOR +============================ Overview ======== diff --git a/Documentation/devicetree/bindings/net/altera_tse.txt b/Documentation/devicetree/bindings/net/altera_tse.txt index 0e21df94a53ffa221e6262f4323dc2ff60068a00..0b7d4d3758ead7ae64ad73a224bb1c80c92b1b3c 100644 --- a/Documentation/devicetree/bindings/net/altera_tse.txt +++ b/Documentation/devicetree/bindings/net/altera_tse.txt @@ -46,9 +46,8 @@ Required properties: - reg: phy id used to communicate to phy. - device_type: Must be "ethernet-phy". -Optional properties: -- local-mac-address: See ethernet.txt in the same directory. -- max-frame-size: See ethernet.txt in the same directory. +The MAC address will be determined using the optional properties defined in +ethernet.txt. Example: diff --git a/Documentation/devicetree/bindings/net/amd-xgbe.txt b/Documentation/devicetree/bindings/net/amd-xgbe.txt index 93dcb79a5f16408ac92694a2831b66b74e232a4f..9c27dfcd11334b9f1dc870815b4ae011b08cd51f 100644 --- a/Documentation/devicetree/bindings/net/amd-xgbe.txt +++ b/Documentation/devicetree/bindings/net/amd-xgbe.txt @@ -24,8 +24,6 @@ Required properties: - phy-mode: See ethernet.txt file in the same directory Optional properties: -- mac-address: mac address to be assigned to the device. Can be overridden - by UEFI. - dma-coherent: Present if dma operations are coherent - amd,per-channel-interrupt: Indicates that Rx and Tx complete will generate a unique interrupt for each DMA channel - this requires an additional @@ -34,6 +32,9 @@ Optional properties: 0 - 1GbE and 10GbE (default) 1 - 2.5GbE and 10GbE +The MAC address will be determined using the optional properties defined in +ethernet.txt. + The following optional properties are represented by an array with each value corresponding to a particular speed. The first array value represents the setting for the 1GbE speed, the second value for the 2.5GbE speed and diff --git a/Documentation/devicetree/bindings/net/brcm,amac.txt b/Documentation/devicetree/bindings/net/brcm,amac.txt index 0bfad656a9ff26fc4f672e06f1123198bb9259e7..0120ebe932624379fc9f19b5ee809a26d16a8e30 100644 --- a/Documentation/devicetree/bindings/net/brcm,amac.txt +++ b/Documentation/devicetree/bindings/net/brcm,amac.txt @@ -16,8 +16,8 @@ Required properties: registers (required for Northstar2) - interrupts: Interrupt number -Optional properties: -- mac-address: See ethernet.txt file in the same directory +The MAC address will be determined using the optional properties +defined in ethernet.txt. Examples: diff --git a/Documentation/devicetree/bindings/net/cpsw.txt b/Documentation/devicetree/bindings/net/cpsw.txt index 3264e1978d25fde513e8857b27348f9edc153203..7c7ac5eb0313ba5a358a236066a582c4afa43ff3 100644 --- a/Documentation/devicetree/bindings/net/cpsw.txt +++ b/Documentation/devicetree/bindings/net/cpsw.txt @@ -49,10 +49,12 @@ Required properties: Optional properties: - dual_emac_res_vlan : Specifies VID to be used to segregate the ports -- mac-address : See ethernet.txt file in the same directory - phy_id : Specifies slave phy id (deprecated, use phy-handle) - phy-handle : See ethernet.txt file in the same directory +The MAC address will be determined using the optional properties +defined in ethernet.txt. + Slave sub-nodes: - fixed-link : See fixed-link.txt file in the same directory diff --git a/Documentation/devicetree/bindings/net/davinci_emac.txt b/Documentation/devicetree/bindings/net/davinci_emac.txt index ca83dcc84fb8ee5cfd876cf0bb3d8af5fd85ba6b..5e3579e72e2d041543628bf2a31dc28041f052d4 100644 --- a/Documentation/devicetree/bindings/net/davinci_emac.txt +++ b/Documentation/devicetree/bindings/net/davinci_emac.txt @@ -20,11 +20,12 @@ Required properties: Optional properties: - phy-handle: See ethernet.txt file in the same directory. If absent, davinci_emac driver defaults to 100/FULL. -- nvmem-cells: phandle, reference to an nvmem node for the MAC address -- nvmem-cell-names: string, should be "mac-address" if nvmem is to be used - ti,davinci-rmii-en: 1 byte, 1 means use RMII - ti,davinci-no-bd-ram: boolean, does EMAC have BD RAM? +The MAC address will be determined using the optional properties +defined in ethernet.txt. + Example (enbw_cmc board): eth0: emac@1e20000 { compatible = "ti,davinci-dm6467-emac"; diff --git a/Documentation/devicetree/bindings/net/dsa/dsa.txt b/Documentation/devicetree/bindings/net/dsa/dsa.txt index d66a5292b9d326cc7d4a68062fd4bfb9f3adb256..f66bb7ecdb8277a10ff03f484d608e000868db51 100644 --- a/Documentation/devicetree/bindings/net/dsa/dsa.txt +++ b/Documentation/devicetree/bindings/net/dsa/dsa.txt @@ -1,12 +1,6 @@ Distributed Switch Architecture Device Tree Bindings ---------------------------------------------------- -Two bindings exist, one of which has been deprecated due to -limitations. - -Current Binding ---------------- - Switches are true Linux devices and can be probed by any means. Once probed, they register to the DSA framework, passing a node pointer. This node is expected to fulfil the following binding, and @@ -71,9 +65,8 @@ properties, described in binding documents: Documentation/devicetree/bindings/net/fixed-link.txt for details. -- local-mac-address : See - Documentation/devicetree/bindings/net/ethernet.txt - for details. +The MAC address will be determined using the optional properties +defined in ethernet.txt. Example @@ -262,152 +255,3 @@ linked into one DSA cluster. }; }; }; - -Deprecated Binding ------------------- - -The deprecated binding makes use of a platform device to represent the -switches. The switches themselves are not Linux devices, and make use -of an MDIO bus for management. - -Required properties: -- compatible : Should be "marvell,dsa" -- #address-cells : Must be 2, first cell is the address on the MDIO bus - and second cell is the address in the switch tree. - Second cell is used only when cascading/chaining. -- #size-cells : Must be 0 -- dsa,ethernet : Should be a phandle to a valid Ethernet device node -- dsa,mii-bus : Should be a phandle to a valid MDIO bus device node - -Optional properties: -- interrupts : property with a value describing the switch - interrupt number (not supported by the driver) - -A DSA node can contain multiple switch chips which are therefore child nodes of -the parent DSA node. The maximum number of allowed child nodes is 4 -(DSA_MAX_SWITCHES). -Each of these switch child nodes should have the following required properties: - -- reg : Contains two fields. The first one describes the - address on the MII bus. The second is the switch - number that must be unique in cascaded configurations -- #address-cells : Must be 1 -- #size-cells : Must be 0 - -A switch child node has the following optional property: - -- eeprom-length : Set to the length of an EEPROM connected to the - switch. Must be set if the switch can not detect - the presence and/or size of a connected EEPROM, - otherwise optional. - -A switch may have multiple "port" children nodes - -Each port children node must have the following mandatory properties: -- reg : Describes the port address in the switch -- label : Describes the label associated with this port, special - labels are "cpu" to indicate a CPU port and "dsa" to - indicate an uplink/downlink port. - -Note that a port labelled "dsa" will imply checking for the uplink phandle -described below. - -Optional property: -- link : Should be a list of phandles to another switch's DSA port. - This property is only used when switches are being - chained/cascaded together. This port is used as outgoing port - towards the phandle port, which can be more than one hop away. - -- phy-handle : Phandle to a PHY on an external MDIO bus, not the - switch internal one. See - Documentation/devicetree/bindings/net/ethernet.txt - for details. - -- phy-mode : String representing the connection to the designated - PHY node specified by the 'phy-handle' property. See - Documentation/devicetree/bindings/net/ethernet.txt - for details. - -- mii-bus : Should be a phandle to a valid MDIO bus device node. - This mii-bus will be used in preference to the - global dsa,mii-bus defined above, for this switch. - -Optional subnodes: -- fixed-link : Fixed-link subnode describing a link to a non-MDIO - managed entity. See - Documentation/devicetree/bindings/net/fixed-link.txt - for details. - -Example: - - dsa@0 { - compatible = "marvell,dsa"; - #address-cells = <2>; - #size-cells = <0>; - - interrupts = <10>; - dsa,ethernet = <ðernet0>; - dsa,mii-bus = <&mii_bus0>; - - switch@0 { - #address-cells = <1>; - #size-cells = <0>; - reg = <16 0>; /* MDIO address 16, switch 0 in tree */ - - port@0 { - reg = <0>; - label = "lan1"; - phy-handle = <&phy0>; - }; - - port@1 { - reg = <1>; - label = "lan2"; - }; - - port@5 { - reg = <5>; - label = "cpu"; - }; - - switch0port6: port@6 { - reg = <6>; - label = "dsa"; - link = <&switch1port0 - &switch2port0>; - }; - }; - - switch@1 { - #address-cells = <1>; - #size-cells = <0>; - reg = <17 1>; /* MDIO address 17, switch 1 in tree */ - mii-bus = <&mii_bus1>; - reset-gpios = <&gpio5 1 GPIO_ACTIVE_LOW>; - - switch1port0: port@0 { - reg = <0>; - label = "dsa"; - link = <&switch0port6>; - }; - switch1port1: port@1 { - reg = <1>; - label = "dsa"; - link = <&switch2port1>; - }; - }; - - switch@2 { - #address-cells = <1>; - #size-cells = <0>; - reg = <18 2>; /* MDIO address 18, switch 2 in tree */ - mii-bus = <&mii_bus1>; - - switch2port0: port@0 { - reg = <0>; - label = "dsa"; - link = <&switch1port1 - &switch0port6>; - }; - }; - }; diff --git a/Documentation/devicetree/bindings/net/dsa/sja1105.txt b/Documentation/devicetree/bindings/net/dsa/sja1105.txt new file mode 100644 index 0000000000000000000000000000000000000000..13fd21074d484a43143be48837491d6893920728 --- /dev/null +++ b/Documentation/devicetree/bindings/net/dsa/sja1105.txt @@ -0,0 +1,156 @@ +NXP SJA1105 switch driver +========================= + +Required properties: + +- compatible: + Must be one of: + - "nxp,sja1105e" + - "nxp,sja1105t" + - "nxp,sja1105p" + - "nxp,sja1105q" + - "nxp,sja1105r" + - "nxp,sja1105s" + + Although the device ID could be detected at runtime, explicit bindings + are required in order to be able to statically check their validity. + For example, SGMII can only be specified on port 4 of R and S devices, + and the non-SGMII devices, while pin-compatible, are not equal in terms + of support for RGMII internal delays (supported on P/Q/R/S, but not on + E/T). + +Optional properties: + +- sja1105,role-mac: +- sja1105,role-phy: + Boolean properties that can be assigned under each port node. By + default (unless otherwise specified) a port is configured as MAC if it + is driving a PHY (phy-handle is present) or as PHY if it is PHY-less + (fixed-link specified, presumably because it is connected to a MAC). + The effect of this property (in either its implicit or explicit form) + is: + - In the case of MII or RMII it specifies whether the SJA1105 port is a + clock source or sink for this interface (not applicable for RGMII + where there is a Tx and an Rx clock). + - In the case of RGMII it affects the behavior regarding internal + delays: + 1. If sja1105,role-mac is specified, and the phy-mode property is one + of "rgmii-id", "rgmii-txid" or "rgmii-rxid", then the entity + designated to apply the delay/clock skew necessary for RGMII + is the PHY. The SJA1105 MAC does not apply any internal delays. + 2. If sja1105,role-phy is specified, and the phy-mode property is one + of the above, the designated entity to apply the internal delays + is the SJA1105 MAC (if hardware-supported). This is only supported + by the second-generation (P/Q/R/S) hardware. On a first-generation + E or T device, it is an error to specify an RGMII phy-mode other + than "rgmii" for a port that is in fixed-link mode. In that case, + the clock skew must either be added by the MAC at the other end of + the fixed-link, or by PCB serpentine traces on the board. + These properties are required, for example, in the case where SJA1105 + ports are at both ends of a MII/RMII PHY-less setup. One end would need + to have sja1105,role-mac, while the other sja1105,role-phy. + +See Documentation/devicetree/bindings/net/dsa/dsa.txt for the list of standard +DSA required and optional properties. + +Other observations +------------------ + +The SJA1105 SPI interface requires a CS-to-CLK time (t2 in UM10944) of at least +one half of t_CLK. At an SPI frequency of 1MHz, this means a minimum +cs_sck_delay of 500ns. Ensuring that this SPI timing requirement is observed +depends on the SPI bus master driver. + +Example +------- + +Ethernet switch connected via SPI to the host, CPU port wired to enet2: + +arch/arm/boot/dts/ls1021a-tsn.dts: + +/* SPI controller of the LS1021 */ +&dspi0 { + sja1105@1 { + reg = <0x1>; + #address-cells = <1>; + #size-cells = <0>; + compatible = "nxp,sja1105t"; + spi-max-frequency = <4000000>; + fsl,spi-cs-sck-delay = <1000>; + fsl,spi-sck-cs-delay = <1000>; + ports { + #address-cells = <1>; + #size-cells = <0>; + port@0 { + /* ETH5 written on chassis */ + label = "swp5"; + phy-handle = <&rgmii_phy6>; + phy-mode = "rgmii-id"; + reg = <0>; + /* Implicit "sja1105,role-mac;" */ + }; + port@1 { + /* ETH2 written on chassis */ + label = "swp2"; + phy-handle = <&rgmii_phy3>; + phy-mode = "rgmii-id"; + reg = <1>; + /* Implicit "sja1105,role-mac;" */ + }; + port@2 { + /* ETH3 written on chassis */ + label = "swp3"; + phy-handle = <&rgmii_phy4>; + phy-mode = "rgmii-id"; + reg = <2>; + /* Implicit "sja1105,role-mac;" */ + }; + port@3 { + /* ETH4 written on chassis */ + phy-handle = <&rgmii_phy5>; + label = "swp4"; + phy-mode = "rgmii-id"; + reg = <3>; + /* Implicit "sja1105,role-mac;" */ + }; + port@4 { + /* Internal port connected to eth2 */ + ethernet = <&enet2>; + phy-mode = "rgmii"; + reg = <4>; + /* Implicit "sja1105,role-phy;" */ + fixed-link { + speed = <1000>; + full-duplex; + }; + }; + }; + }; +}; + +/* MDIO controller of the LS1021 */ +&mdio0 { + /* BCM5464 */ + rgmii_phy3: ethernet-phy@3 { + reg = <0x3>; + }; + rgmii_phy4: ethernet-phy@4 { + reg = <0x4>; + }; + rgmii_phy5: ethernet-phy@5 { + reg = <0x5>; + }; + rgmii_phy6: ethernet-phy@6 { + reg = <0x6>; + }; +}; + +/* Ethernet master port of the LS1021 */ +&enet2 { + phy-connection-type = "rgmii"; + status = "ok"; + fixed-link { + speed = <1000>; + full-duplex; + }; +}; diff --git a/Documentation/devicetree/bindings/net/ethernet.txt b/Documentation/devicetree/bindings/net/ethernet.txt index a6862158058461f5af428498ea14c98aed1f7775..e88c3641d61363f47d973647a2120dabf18bc50c 100644 --- a/Documentation/devicetree/bindings/net/ethernet.txt +++ b/Documentation/devicetree/bindings/net/ethernet.txt @@ -4,12 +4,14 @@ NOTE: All 'phy*' properties documented below are Ethernet specific. For the generic PHY 'phys' property, see Documentation/devicetree/bindings/phy/phy-bindings.txt. -- local-mac-address: array of 6 bytes, specifies the MAC address that was - assigned to the network device; - mac-address: array of 6 bytes, specifies the MAC address that was last used by the boot program; should be used in cases where the MAC address assigned to the device by the boot program is different from the "local-mac-address" property; +- local-mac-address: array of 6 bytes, specifies the MAC address that was + assigned to the network device; +- nvmem-cells: phandle, reference to an nvmem node for the MAC address +- nvmem-cell-names: string, should be "mac-address" if nvmem is to be used - max-speed: number, specifies maximum speed in Mbit/s supported by the device; - max-frame-size: number, maximum transfer unit (IEEE defined MTU), rather than the maximum frame size (there's contradiction in the Devicetree @@ -36,7 +38,7 @@ Documentation/devicetree/bindings/phy/phy-bindings.txt. * "smii" * "xgmii" * "trgmii" - * "2000base-x", + * "1000base-x", * "2500base-x", * "rxaui" * "xaui" diff --git a/Documentation/devicetree/bindings/net/hisilicon-femac.txt b/Documentation/devicetree/bindings/net/hisilicon-femac.txt index d11af5ecace8594cfe92d72d5bcdc4d2b2e63932..5f96976f3cea7aeb777683b3ff5fb9290f6d6737 100644 --- a/Documentation/devicetree/bindings/net/hisilicon-femac.txt +++ b/Documentation/devicetree/bindings/net/hisilicon-femac.txt @@ -14,7 +14,6 @@ Required properties: the PHY reset signal(optional). - reset-names: should contain the reset signal name "mac"(required) and "phy"(optional). -- mac-address: see ethernet.txt [1]. - phy-mode: see ethernet.txt [1]. - phy-handle: see ethernet.txt [1]. - hisilicon,phy-reset-delays-us: triplet of delays if PHY reset signal given. @@ -22,6 +21,9 @@ Required properties: The 2nd cell is reset pulse in micro seconds. The 3rd cell is reset post-delay in micro seconds. +The MAC address will be determined using the optional properties +defined in ethernet.txt[1]. + [1] Documentation/devicetree/bindings/net/ethernet.txt Example: diff --git a/Documentation/devicetree/bindings/net/hisilicon-hix5hd2-gmac.txt b/Documentation/devicetree/bindings/net/hisilicon-hix5hd2-gmac.txt index eea73adc678f963888c3171006516272abd37f62..cddf46bf6b634f9a89a99e707a767c5d5de6c3df 100644 --- a/Documentation/devicetree/bindings/net/hisilicon-hix5hd2-gmac.txt +++ b/Documentation/devicetree/bindings/net/hisilicon-hix5hd2-gmac.txt @@ -18,7 +18,6 @@ Required properties: - #size-cells: must be <0>. - phy-mode: see ethernet.txt [1]. - phy-handle: see ethernet.txt [1]. -- mac-address: see ethernet.txt [1]. - clocks: clock phandle and specifier pair. - clock-names: contain the clock name "mac_core"(required) and "mac_ifc"(optional). - resets: should contain the phandle to the MAC core reset signal(optional), @@ -31,6 +30,9 @@ Required properties: The 2nd cell is reset pulse in micro seconds. The 3rd cell is reset post-delay in micro seconds. +The MAC address will be determined using the properties defined in +ethernet.txt[1]. + - PHY subnode: inherits from phy binding [2] [1] Documentation/devicetree/bindings/net/ethernet.txt diff --git a/Documentation/devicetree/bindings/net/keystone-netcp.txt b/Documentation/devicetree/bindings/net/keystone-netcp.txt index 04ba1dc34fd62150ae1357e8c0c7804907f56313..3a65aabc76a247eb420746b542b31515fdb96725 100644 --- a/Documentation/devicetree/bindings/net/keystone-netcp.txt +++ b/Documentation/devicetree/bindings/net/keystone-netcp.txt @@ -135,14 +135,14 @@ Optional properties: are swapped. The netcp driver will swap the two DWORDs back to the proper order when this property is set to 2 when it obtains the mac address from efuse. -- local-mac-address: the driver is designed to use the of_get_mac_address api - only if efuse-mac is 0. When efuse-mac is 0, the MAC - address is obtained from local-mac-address. If this - attribute is not present, then the driver will use a - random MAC address. - "netcp-device label": phandle to the device specification for each of NetCP sub-module attached to this interface. +The MAC address will be determined using the optional properties defined in +ethernet.txt, as provided by the of_get_mac_address API and only if efuse-mac +is set to 0. If any of the optional MAC address properties are not present, +then the driver will use random MAC address. + Example binding: netcp: netcp@2000000 { diff --git a/Documentation/devicetree/bindings/net/macb.txt b/Documentation/devicetree/bindings/net/macb.txt index 8b80515729d7145cc05c9293857212ba914e0607..9c5e94482b5f39dc3d34abb63ac0435454f21312 100644 --- a/Documentation/devicetree/bindings/net/macb.txt +++ b/Documentation/devicetree/bindings/net/macb.txt @@ -26,9 +26,8 @@ Required properties: Optional elements: 'tsu_clk' - clocks: Phandles to input clocks. -Optional properties: -- nvmem-cells: phandle, reference to an nvmem node for the MAC address -- nvmem-cell-names: string, should be "mac-address" if nvmem is to be used +The MAC address will be determined using the optional properties +defined in ethernet.txt. Optional properties for PHY child node: - reset-gpios : Should specify the gpio for phy reset diff --git a/Documentation/devicetree/bindings/net/marvell-pxa168.txt b/Documentation/devicetree/bindings/net/marvell-pxa168.txt index 845a148a346e9265e92f93f966a2e091b39ed058..5574af3554aaa57782249c97a036c1c732dbb8c3 100644 --- a/Documentation/devicetree/bindings/net/marvell-pxa168.txt +++ b/Documentation/devicetree/bindings/net/marvell-pxa168.txt @@ -11,7 +11,9 @@ Optional properties: - #address-cells: must be 1 when using sub-nodes. - #size-cells: must be 0 when using sub-nodes. - phy-handle: see ethernet.txt file in the same directory. -- local-mac-address: see ethernet.txt file in the same directory. + +The MAC address will be determined using the optional properties +defined in ethernet.txt. Sub-nodes: Each PHY can be represented as a sub-node. This is not mandatory. diff --git a/Documentation/devicetree/bindings/net/mdio-mux-meson-g12a.txt b/Documentation/devicetree/bindings/net/mdio-mux-meson-g12a.txt new file mode 100644 index 0000000000000000000000000000000000000000..3a96cbed92947988f5cfa2ef53a86a3a6a65c3ee --- /dev/null +++ b/Documentation/devicetree/bindings/net/mdio-mux-meson-g12a.txt @@ -0,0 +1,48 @@ +Properties for the MDIO bus multiplexer/glue of Amlogic G12a SoC family. + +This is a special case of a MDIO bus multiplexer. It allows to choose between +the internal mdio bus leading to the embedded 10/100 PHY or the external +MDIO bus. + +Required properties in addition to the generic multiplexer properties: +- compatible : amlogic,g12a-mdio-mux +- reg: physical address and length of the multiplexer/glue registers +- clocks: list of clock phandle, one for each entry clock-names. +- clock-names: should contain the following: + * "pclk" : peripheral clock. + * "clkin0" : platform crytal + * "clkin1" : SoC 50MHz MPLL + +Example : + +mdio_mux: mdio-multiplexer@4c000 { + compatible = "amlogic,g12a-mdio-mux"; + reg = <0x0 0x4c000 0x0 0xa4>; + clocks = <&clkc CLKID_ETH_PHY>, + <&xtal>, + <&clkc CLKID_MPLL_5OM>; + clock-names = "pclk", "clkin0", "clkin1"; + mdio-parent-bus = <&mdio0>; + #address-cells = <1>; + #size-cells = <0>; + + ext_mdio: mdio@0 { + reg = <0>; + #address-cells = <1>; + #size-cells = <0>; + }; + + int_mdio: mdio@1 { + reg = <1>; + #address-cells = <1>; + #size-cells = <0>; + + internal_ephy: ethernet-phy@8 { + compatible = "ethernet-phy-id0180.3301", + "ethernet-phy-ieee802.3-c22"; + interrupts = ; + reg = <8>; + max-speed = <100>; + }; + }; +}; diff --git a/Documentation/devicetree/bindings/net/microchip,enc28j60.txt b/Documentation/devicetree/bindings/net/microchip,enc28j60.txt index 24626e082b838f25a33e6fea0cebd8a0f75fcfef..a8275921a8964683fa7be15699b1ca4ce019abe0 100644 --- a/Documentation/devicetree/bindings/net/microchip,enc28j60.txt +++ b/Documentation/devicetree/bindings/net/microchip,enc28j60.txt @@ -21,8 +21,9 @@ Optional properties: - spi-max-frequency: Maximum frequency of the SPI bus when accessing the ENC28J60. According to the ENC28J80 datasheet, the chip allows a maximum of 20 MHz, however, board designs may need to limit this value. -- local-mac-address: See ethernet.txt in the same directory. +The MAC address will be determined using the optional properties +defined in ethernet.txt. Example (for NXP i.MX28 with pin control stuff for GPIO irq): diff --git a/Documentation/devicetree/bindings/net/microchip,lan78xx.txt b/Documentation/devicetree/bindings/net/microchip,lan78xx.txt index 76786a0f6d3d7c06d5ce09ba15c311621def50cf..11a679530ae65a2ae59bf740b210d15d26924a3d 100644 --- a/Documentation/devicetree/bindings/net/microchip,lan78xx.txt +++ b/Documentation/devicetree/bindings/net/microchip,lan78xx.txt @@ -7,9 +7,8 @@ The Device Tree properties, if present, override the OTP and EEPROM. Required properties: - compatible: Should be one of "usb424,7800", "usb424,7801" or "usb424,7850". -Optional properties: -- local-mac-address: see ethernet.txt -- mac-address: see ethernet.txt +The MAC address will be determined using the optional properties +defined in ethernet.txt. Optional properties of the embedded PHY: - microchip,led-modes: a 0..4 element vector, with each element configuring diff --git a/Documentation/devicetree/bindings/net/phy.txt b/Documentation/devicetree/bindings/net/phy.txt index 17c1d2bd00f6dd64ebb4891b3293d22a10c0b22b..9b9e5b1765ddf65c1f717ec756db1719af243079 100644 --- a/Documentation/devicetree/bindings/net/phy.txt +++ b/Documentation/devicetree/bindings/net/phy.txt @@ -51,6 +51,10 @@ Optional Properties: to ensure the integrated PHY is used. The absence of this property indicates the muxers should be configured so that the external PHY is used. +- resets: The reset-controller phandle and specifier for the PHY reset signal. + +- reset-names: Must be "phy" for the PHY reset signal. + - reset-gpios: The GPIO phandle and specifier for the PHY reset signal. - reset-assert-us: Delay after the reset was asserted in microseconds. @@ -67,6 +71,8 @@ ethernet-phy@0 { interrupts = <35 IRQ_TYPE_EDGE_RISING>; reg = <0>; + resets = <&rst 8>; + reset-names = "phy"; reset-gpios = <&gpio1 4 GPIO_ACTIVE_LOW>; reset-assert-us = <1000>; reset-deassert-us = <2000>; diff --git a/Documentation/devicetree/bindings/net/qca,qca7000.txt b/Documentation/devicetree/bindings/net/qca,qca7000.txt index e4a8a51086dfcbc2d766d7b3d21eb15b900f79fd..21c36e5249936c4c25677e396a457acbe3c9202e 100644 --- a/Documentation/devicetree/bindings/net/qca,qca7000.txt +++ b/Documentation/devicetree/bindings/net/qca,qca7000.txt @@ -23,7 +23,6 @@ Optional properties: Numbers smaller than 1000000 or greater than 16000000 are invalid. Missing the property will set the SPI frequency to 8000000 Hertz. -- local-mac-address : see ./ethernet.txt - qca,legacy-mode : Set the SPI data transfer of the QCA7000 to legacy mode. In this mode the SPI master must toggle the chip select between each data word. In burst mode these gaps aren't @@ -31,6 +30,9 @@ Optional properties: the QCA7000 is setup via GPIO pin strapping. If the property is missing the driver defaults to burst mode. +The MAC address will be determined using the optional properties +defined in ethernet.txt. + SPI Example: /* Freescale i.MX28 SPI master*/ diff --git a/Documentation/devicetree/bindings/net/qualcomm-bluetooth.txt b/Documentation/devicetree/bindings/net/qualcomm-bluetooth.txt index 824c0e23c5443092302e95790f62e0822d39197f..7ef6118abd3d2ad5968157d2099b8ea1cd1e3e67 100644 --- a/Documentation/devicetree/bindings/net/qualcomm-bluetooth.txt +++ b/Documentation/devicetree/bindings/net/qualcomm-bluetooth.txt @@ -11,20 +11,21 @@ Required properties: - compatible: should contain one of the following: * "qcom,qca6174-bt" * "qcom,wcn3990-bt" + * "qcom,wcn3998-bt" Optional properties for compatible string qcom,qca6174-bt: - enable-gpios: gpio specifier used to enable chip - clocks: clock provided to the controller (SUSCLK_32KHZ) -Required properties for compatible string qcom,wcn3990-bt: +Required properties for compatible string qcom,wcn399x-bt: - vddio-supply: VDD_IO supply regulator handle. - vddxo-supply: VDD_XO supply regulator handle. - vddrf-supply: VDD_RF supply regulator handle. - vddch0-supply: VDD_CH0 supply regulator handle. -Optional properties for compatible string qcom,wcn3990-bt: +Optional properties for compatible string qcom,wcn399x-bt: - max-speed: see Documentation/devicetree/bindings/serial/slave-device.txt diff --git a/Documentation/devicetree/bindings/net/samsung-sxgbe.txt b/Documentation/devicetree/bindings/net/samsung-sxgbe.txt index 46e591178911d28399c69b95f8b564b7f6d99dae..2cff6d8a585ac599dfb9fed7078b58821599cf24 100644 --- a/Documentation/devicetree/bindings/net/samsung-sxgbe.txt +++ b/Documentation/devicetree/bindings/net/samsung-sxgbe.txt @@ -21,10 +21,12 @@ Required properties: range. Optional properties: -- mac-address: 6 bytes, mac address - max-frame-size: Maximum Transfer Unit (IEEE defined MTU), rather than the maximum frame size. +The MAC address will be determined using the optional properties +defined in ethernet.txt. + Example: aliases { diff --git a/Documentation/devicetree/bindings/net/snps,dwc-qos-ethernet.txt b/Documentation/devicetree/bindings/net/snps,dwc-qos-ethernet.txt index 36f1aef585f03fbf6ee6e4b0b3a7a216a07fad09..ad3c6e109ce18f383d9ceb74fab85dd5723bf7af 100644 --- a/Documentation/devicetree/bindings/net/snps,dwc-qos-ethernet.txt +++ b/Documentation/devicetree/bindings/net/snps,dwc-qos-ethernet.txt @@ -103,8 +103,6 @@ Required properties: Optional properties: - dma-coherent: Present if dma operations are coherent -- mac-address: See ethernet.txt in the same directory -- local-mac-address: See ethernet.txt in the same directory - phy-reset-gpios: Phandle and specifier for any GPIO used to reset the PHY. See ../gpio/gpio.txt. - snps,en-lpi: If present it enables use of the AXI low-power interface @@ -133,6 +131,9 @@ Optional properties: - device_type: Must be "ethernet-phy". - fixed-mode device tree subnode: see fixed-link.txt in the same directory +The MAC address will be determined using the optional properties +defined in ethernet.txt. + Examples: ethernet2@40010000 { clock-names = "phy_ref_clk", "apb_pclk"; diff --git a/Documentation/devicetree/bindings/net/socionext,uniphier-ave4.txt b/Documentation/devicetree/bindings/net/socionext,uniphier-ave4.txt index fc8f01718690d2cbbbf271cbb0d3117ea8c6041c..4e85fc495e87ce33333681657f019b8b2f408d2e 100644 --- a/Documentation/devicetree/bindings/net/socionext,uniphier-ave4.txt +++ b/Documentation/devicetree/bindings/net/socionext,uniphier-ave4.txt @@ -31,8 +31,8 @@ Required properties: - socionext,syscon-phy-mode: A phandle to syscon with one argument that configures phy mode. The argument is the ID of MAC instance. -Optional properties: - - local-mac-address: See ethernet.txt in the same directory. +The MAC address will be determined using the optional properties +defined in ethernet.txt. Required subnode: - mdio: A container for child nodes representing phy nodes. diff --git a/Documentation/devicetree/bindings/net/socionext-netsec.txt b/Documentation/devicetree/bindings/net/socionext-netsec.txt index 0cff94fb04334926a1c3eb8c08969db29993bd71..9d6c9feb12ff191b707821b3314be092fbb7b393 100644 --- a/Documentation/devicetree/bindings/net/socionext-netsec.txt +++ b/Documentation/devicetree/bindings/net/socionext-netsec.txt @@ -26,11 +26,12 @@ Required properties: Optional properties: (See ethernet.txt file in the same directory) - dma-coherent: Boolean property, must only be present if memory accesses performed by the device are cache coherent. -- local-mac-address: See ethernet.txt in the same directory. -- mac-address: See ethernet.txt in the same directory. - max-speed: See ethernet.txt in the same directory. - max-frame-size: See ethernet.txt in the same directory. +The MAC address will be determined using the optional properties +defined in ethernet.txt. + Example: eth0: ethernet@522d0000 { compatible = "socionext,synquacer-netsec"; diff --git a/Documentation/devicetree/bindings/net/wireless/mediatek,mt76.txt b/Documentation/devicetree/bindings/net/wireless/mediatek,mt76.txt index 7b9a776230c05e6b349949344a8ecc1be32f9921..74665502f4cfd60a0a49692e31a2ad54e48c2682 100644 --- a/Documentation/devicetree/bindings/net/wireless/mediatek,mt76.txt +++ b/Documentation/devicetree/bindings/net/wireless/mediatek,mt76.txt @@ -13,11 +13,12 @@ properties: Optional properties: -- mac-address: See ethernet.txt in the parent directory -- local-mac-address: See ethernet.txt in the parent directory - ieee80211-freq-limit: See ieee80211.txt - mediatek,mtd-eeprom: Specify a MTD partition + offset containing EEPROM data +The driver is using of_get_mac_address API, so the MAC address can be as well +be set with corresponding optional properties defined in net/ethernet.txt. + Optional nodes: - led: Properties for a connected LED Optional properties: diff --git a/Documentation/devicetree/bindings/net/wireless/qca,ath9k.txt b/Documentation/devicetree/bindings/net/wireless/qca,ath9k.txt index b7396c8c271cd339784b777a234862aa3786cf1d..aaaeeb5f935bc8a3b0794dfd75aaf66e8701942b 100644 --- a/Documentation/devicetree/bindings/net/wireless/qca,ath9k.txt +++ b/Documentation/devicetree/bindings/net/wireless/qca,ath9k.txt @@ -34,9 +34,9 @@ Optional properties: ath9k wireless chip (in this case the calibration / EEPROM data will be loaded from userspace using the kernel firmware loader). -- mac-address: See ethernet.txt in the parent directory -- local-mac-address: See ethernet.txt in the parent directory +The MAC address will be determined using the optional properties defined in +net/ethernet.txt. In this example, the node is defined as child node of the PCI controller: &pci0 { diff --git a/Documentation/networking/batman-adv.rst b/Documentation/networking/batman-adv.rst index 245fb6c0ab6f55c828296c279c8867d47f629d1a..18020943ba2548d3f1dfcc353195ecb848a0ddb4 100644 --- a/Documentation/networking/batman-adv.rst +++ b/Documentation/networking/batman-adv.rst @@ -27,24 +27,8 @@ Load the batman-adv module into your kernel:: $ insmod batman-adv.ko The module is now waiting for activation. You must add some interfaces on which -batman can operate. After loading the module batman advanced will scan your -systems interfaces to search for compatible interfaces. Once found, it will -create subfolders in the ``/sys`` directories of each supported interface, -e.g.:: - - $ ls /sys/class/net/eth0/batman_adv/ - elp_interval iface_status mesh_iface throughput_override - -If an interface does not have the ``batman_adv`` subfolder, it probably is not -supported. Not supported interfaces are: loopback, non-ethernet and batman's -own interfaces. - -Note: After the module was loaded it will continuously watch for new -interfaces to verify the compatibility. There is no need to reload the module -if you plug your USB wifi adapter into your machine after batman advanced was -initially loaded. - -The batman-adv soft-interface can be created using the iproute2 tool ``ip``:: +batman-adv can operate. The batman-adv soft-interface can be created using the +iproute2 tool ``ip``:: $ ip link add name bat0 type batadv @@ -52,57 +36,46 @@ To activate a given interface simply attach it to the ``bat0`` interface:: $ ip link set dev eth0 master bat0 -Repeat this step for all interfaces you wish to add. Now batman starts +Repeat this step for all interfaces you wish to add. Now batman-adv starts using/broadcasting on this/these interface(s). -By reading the "iface_status" file you can check its status:: - - $ cat /sys/class/net/eth0/batman_adv/iface_status - active - To deactivate an interface you have to detach it from the "bat0" interface:: $ ip link set dev eth0 nomaster +The same can also be done using the batctl interface subcommand:: -All mesh wide settings can be found in batman's own interface folder:: + batctl -m bat0 interface create + batctl -m bat0 interface add -M eth0 - $ ls /sys/class/net/bat0/mesh/ - aggregated_ogms fragmentation isolation_mark routing_algo - ap_isolation gw_bandwidth log_level vlan0 - bonding gw_mode multicast_mode - bridge_loop_avoidance gw_sel_class network_coding - distributed_arp_table hop_penalty orig_interval +To detach eth0 and destroy bat0:: -There is a special folder for debugging information:: + batctl -m bat0 interface del -M eth0 + batctl -m bat0 interface destroy - $ ls /sys/kernel/debug/batman_adv/bat0/ - bla_backbone_table log neighbors transtable_local - bla_claim_table mcast_flags originators - dat_cache nc socket - gateways nc_nodes transtable_global +There are additional settings for each batadv mesh interface, vlan and hardif +which can be modified using batctl. Detailed information about this can be found +in its manual. -Some of the files contain all sort of status information regarding the mesh -network. For example, you can view the table of originators (mesh -participants) with:: +For instance, you can check the current originator interval (value +in milliseconds which determines how often batman-adv sends its broadcast +packets):: - $ cat /sys/kernel/debug/batman_adv/bat0/originators - -Other files allow to change batman's behaviour to better fit your requirements. -For instance, you can check the current originator interval (value in -milliseconds which determines how often batman sends its broadcast packets):: - - $ cat /sys/class/net/bat0/mesh/orig_interval + $ batctl -M bat0 orig_interval 1000 and also change its value:: - $ echo 3000 > /sys/class/net/bat0/mesh/orig_interval + $ batctl -M bat0 orig_interval 3000 In very mobile scenarios, you might want to adjust the originator interval to a lower value. This will make the mesh more responsive to topology changes, but will also increase the overhead. +Information about the current state can be accessed via the batadv generic +netlink family. batctl provides human readable version via its debug tables +subcommands. + Usage ===== @@ -147,43 +120,16 @@ batman-adv module. When building batman-adv as part of kernel, use "make menuconfig" and enable the option ``B.A.T.M.A.N. debugging`` (``CONFIG_BATMAN_ADV_DEBUG=y``). -Those additional debug messages can be accessed using a special file in -debugfs:: +Those additional debug messages can be accessed using the perf infrastructure:: - $ cat /sys/kernel/debug/batman_adv/bat0/log + $ trace-cmd stream -e batadv:batadv_dbg The additional debug output is by default disabled. It can be enabled during -run time. Following log_levels are defined: - -.. flat-table:: - - * - 0 - - All debug output disabled - * - 1 - - Enable messages related to routing / flooding / broadcasting - * - 2 - - Enable messages related to route added / changed / deleted - * - 4 - - Enable messages related to translation table operations - * - 8 - - Enable messages related to bridge loop avoidance - * - 16 - - Enable messages related to DAT, ARP snooping and parsing - * - 32 - - Enable messages related to network coding - * - 64 - - Enable messages related to multicast - * - 128 - - Enable messages related to throughput meter - * - 255 - - Enable all messages - -The debug output can be changed at runtime using the file -``/sys/class/net/bat0/mesh/log_level``. e.g.:: - - $ echo 6 > /sys/class/net/bat0/mesh/log_level - -will enable debug messages for when routes change. +run time:: + + $ batctl -m bat0 loglevel routes tt + +will enable debug messages for when routes and translation table entries change. Counters for different types of packets entering and leaving the batman-adv module are available through ethtool:: diff --git a/Documentation/networking/devlink-info-versions.rst b/Documentation/networking/devlink-info-versions.rst index c79ad85933831a257e3b15a8243488d51279f6ff..4316342b77468fa000a83b19c8b36a904185b612 100644 --- a/Documentation/networking/devlink-info-versions.rst +++ b/Documentation/networking/devlink-info-versions.rst @@ -41,3 +41,8 @@ fw.ncsi Version of the software responsible for supporting/handling the Network Controller Sideband Interface. + +fw.psid +======= + +Unique identifier of the firmware parameter set. diff --git a/Documentation/networking/dsa/bcm_sf2.txt b/Documentation/networking/dsa/bcm_sf2.rst similarity index 83% rename from Documentation/networking/dsa/bcm_sf2.txt rename to Documentation/networking/dsa/bcm_sf2.rst index eba3a2431e914ffb787eaf08d96c16b405f94d51..dee234039e1eccacf2309557dfab8dee910d2f76 100644 --- a/Documentation/networking/dsa/bcm_sf2.txt +++ b/Documentation/networking/dsa/bcm_sf2.rst @@ -1,3 +1,4 @@ +============================================= Broadcom Starfighter 2 Ethernet switch driver ============================================= @@ -25,27 +26,27 @@ are connected at a lower speed. The switch hardware block is typically interfaced using MMIO accesses and contains a bunch of sub-blocks/registers: -* SWITCH_CORE: common switch registers -* SWITCH_REG: external interfaces switch register -* SWITCH_MDIO: external MDIO bus controller (there is another one in SWITCH_CORE, +- ``SWITCH_CORE``: common switch registers +- ``SWITCH_REG``: external interfaces switch register +- ``SWITCH_MDIO``: external MDIO bus controller (there is another one in SWITCH_CORE, which is used for indirect PHY accesses) -* SWITCH_INDIR_RW: 64-bits wide register helper block -* SWITCH_INTRL2_0/1: Level-2 interrupt controllers -* SWITCH_ACB: Admission control block -* SWITCH_FCB: Fail-over control block +- ``SWITCH_INDIR_RW``: 64-bits wide register helper block +- ``SWITCH_INTRL2_0/1``: Level-2 interrupt controllers +- ``SWITCH_ACB``: Admission control block +- ``SWITCH_FCB``: Fail-over control block Implementation details ====================== -The driver is located in drivers/net/dsa/bcm_sf2.c and is implemented as a DSA -driver; see Documentation/networking/dsa/dsa.txt for details on the subsystem +The driver is located in ``drivers/net/dsa/bcm_sf2.c`` and is implemented as a DSA +driver; see ``Documentation/networking/dsa/dsa.rst`` for details on the subsystem and what it provides. The SF2 switch is configured to enable a Broadcom specific 4-bytes switch tag which gets inserted by the switch for every packet forwarded to the CPU interface, conversely, the CPU network interface should insert a similar tag for packets entering the CPU port. The tag format is described in -net/dsa/tag_brcm.c. +``net/dsa/tag_brcm.c``. Overall, the SF2 driver is a fairly regular DSA driver; there are a few specifics covered below. @@ -54,7 +55,7 @@ Device Tree probing ------------------- The DSA platform device driver is probed using a specific compatible string -provided in net/dsa/dsa.c. The reason for that is because the DSA subsystem gets +provided in ``net/dsa/dsa.c``. The reason for that is because the DSA subsystem gets registered as a platform device driver currently. DSA will provide the needed device_node pointers which are then accessible by the switch driver setup function to setup resources such as register ranges and interrupts. This @@ -70,7 +71,7 @@ Broadcom switches connected to a SF2 require the use of the DSA slave MDIO bus in order to properly configure them. By default, the SF2 pseudo-PHY address, and an external switch pseudo-PHY address will both be snooping for incoming MDIO transactions, since they are at the same address (30), resulting in some kind of -"double" programming. Using DSA, and setting ds->phys_mii_mask accordingly, we +"double" programming. Using DSA, and setting ``ds->phys_mii_mask`` accordingly, we selectively divert reads and writes towards external Broadcom switches pseudo-PHY addresses. Newer revisions of the SF2 hardware have introduced a configurable pseudo-PHY address which circumvents the initial design limitation. @@ -86,7 +87,7 @@ firmware gets reloaded. The SF2 driver relies on such events to properly set its MoCA interface carrier state and properly report this to the networking stack. The MoCA interfaces are supported using the PHY library's fixed PHY/emulated PHY -device and the switch driver registers a fixed_link_update callback for such +device and the switch driver registers a ``fixed_link_update`` callback for such PHYs which reflects the link state obtained from the interrupt handler. diff --git a/Documentation/networking/dsa/dsa.txt b/Documentation/networking/dsa/dsa.rst similarity index 67% rename from Documentation/networking/dsa/dsa.txt rename to Documentation/networking/dsa/dsa.rst index 43ef767bc4400053f9679d3ba41abbb8eea150a2..ca87068b9ab904a9f9342fdba01d638fef51df8e 100644 --- a/Documentation/networking/dsa/dsa.txt +++ b/Documentation/networking/dsa/dsa.rst @@ -1,10 +1,8 @@ -Distributed Switch Architecture -=============================== - -Introduction +============ +Architecture ============ -This document describes the Distributed Switch Architecture (DSA) subsystem +This document describes the **Distributed Switch Architecture (DSA)** subsystem design principles, limitations, interactions with other subsystems, and how to develop drivers for this subsystem as well as a TODO for developers interested in joining the effort. @@ -70,11 +68,11 @@ Switch tagging protocols DSA currently supports 5 different tagging protocols, and a tag-less mode as well. The different protocols are implemented in: -net/dsa/tag_trailer.c: Marvell's 4 trailer tag mode (legacy) -net/dsa/tag_dsa.c: Marvell's original DSA tag -net/dsa/tag_edsa.c: Marvell's enhanced DSA tag -net/dsa/tag_brcm.c: Broadcom's 4 bytes tag -net/dsa/tag_qca.c: Qualcomm's 2 bytes tag +- ``net/dsa/tag_trailer.c``: Marvell's 4 trailer tag mode (legacy) +- ``net/dsa/tag_dsa.c``: Marvell's original DSA tag +- ``net/dsa/tag_edsa.c``: Marvell's enhanced DSA tag +- ``net/dsa/tag_brcm.c``: Broadcom's 4 bytes tag +- ``net/dsa/tag_qca.c``: Qualcomm's 2 bytes tag The exact format of the tag protocol is vendor specific, but in general, they all contain something which: @@ -89,7 +87,7 @@ Master network devices are regular, unmodified Linux network device drivers for the CPU/management Ethernet interface. Such a driver might occasionally need to know whether DSA is enabled (e.g.: to enable/disable specific offload features), but the DSA subsystem has been proven to work with industry standard drivers: -e1000e, mv643xx_eth etc. without having to introduce modifications to these +``e1000e,`` ``mv643xx_eth`` etc. without having to introduce modifications to these drivers. Such network devices are also often referred to as conduit network devices since they act as a pipe between the host processor and the hardware Ethernet switch. @@ -100,40 +98,42 @@ Networking stack hooks When a master netdev is used with DSA, a small hook is placed in in the networking stack is in order to have the DSA subsystem process the Ethernet switch specific tagging protocol. DSA accomplishes this by registering a -specific (and fake) Ethernet type (later becoming skb->protocol) with the -networking stack, this is also known as a ptype or packet_type. A typical +specific (and fake) Ethernet type (later becoming ``skb->protocol``) with the +networking stack, this is also known as a ``ptype`` or ``packet_type``. A typical Ethernet Frame receive sequence looks like this: Master network device (e.g.: e1000e): -Receive interrupt fires: -- receive function is invoked -- basic packet processing is done: getting length, status etc. -- packet is prepared to be processed by the Ethernet layer by calling - eth_type_trans +1. Receive interrupt fires: + + - receive function is invoked + - basic packet processing is done: getting length, status etc. + - packet is prepared to be processed by the Ethernet layer by calling + ``eth_type_trans`` + +2. net/ethernet/eth.c:: + + eth_type_trans(skb, dev) + if (dev->dsa_ptr != NULL) + -> skb->protocol = ETH_P_XDSA -net/ethernet/eth.c: +3. drivers/net/ethernet/\*:: -eth_type_trans(skb, dev) - if (dev->dsa_ptr != NULL) - -> skb->protocol = ETH_P_XDSA + netif_receive_skb(skb) + -> iterate over registered packet_type + -> invoke handler for ETH_P_XDSA, calls dsa_switch_rcv() -drivers/net/ethernet/*: +4. net/dsa/dsa.c:: -netif_receive_skb(skb) - -> iterate over registered packet_type - -> invoke handler for ETH_P_XDSA, calls dsa_switch_rcv() + -> dsa_switch_rcv() + -> invoke switch tag specific protocol handler in 'net/dsa/tag_*.c' -net/dsa/dsa.c: - -> dsa_switch_rcv() - -> invoke switch tag specific protocol handler in - net/dsa/tag_*.c +5. net/dsa/tag_*.c: -net/dsa/tag_*.c: - -> inspect and strip switch tag protocol to determine originating port - -> locate per-port network device - -> invoke eth_type_trans() with the DSA slave network device - -> invoked netif_receive_skb() + - inspect and strip switch tag protocol to determine originating port + - locate per-port network device + - invoke ``eth_type_trans()`` with the DSA slave network device + - invoked ``netif_receive_skb()`` Past this point, the DSA slave network devices get delivered regular Ethernet frames that can be processed by the networking stack. @@ -162,7 +162,7 @@ invoke a specific transmit routine which takes care of adding the relevant switch tag in the Ethernet frames. These frames are then queued for transmission using the master network device -ndo_start_xmit() function, since they contain the appropriate switch tag, the +``ndo_start_xmit()`` function, since they contain the appropriate switch tag, the Ethernet switch will be able to process these incoming frames from the management interface and delivers these frames to the physical switch port. @@ -170,23 +170,25 @@ Graphical representation ------------------------ Summarized, this is basically how DSA looks like from a network device -perspective: - - - |--------------------------- - | CPU network device (eth0)| - ---------------------------- - | | - |--------------------------------------------| - | Switch driver | - |--------------------------------------------| - || || || - |-------| |-------| |-------| - | sw0p0 | | sw0p1 | | sw0p2 | - |-------| |-------| |-------| +perspective:: + + + |--------------------------- + | CPU network device (eth0)| + ---------------------------- + | | + |--------------------------------------------| + | Switch driver | + |--------------------------------------------| + || || || + |-------| |-------| |-------| + | sw0p0 | | sw0p1 | | sw0p2 | + |-------| |-------| |-------| + + Slave MDIO bus -------------- @@ -207,31 +209,32 @@ PHYs, external PHYs, or even external switches. Data structures --------------- -DSA data structures are defined in include/net/dsa.h as well as -net/dsa/dsa_priv.h. +DSA data structures are defined in ``include/net/dsa.h`` as well as +``net/dsa/dsa_priv.h``: -dsa_chip_data: platform data configuration for a given switch device, this -structure describes a switch device's parent device, its address, as well as -various properties of its ports: names/labels, and finally a routing table -indication (when cascading switches) +- ``dsa_chip_data``: platform data configuration for a given switch device, + this structure describes a switch device's parent device, its address, as + well as various properties of its ports: names/labels, and finally a routing + table indication (when cascading switches) -dsa_platform_data: platform device configuration data which can reference a -collection of dsa_chip_data structure if multiples switches are cascaded, the -master network device this switch tree is attached to needs to be referenced +- ``dsa_platform_data``: platform device configuration data which can reference + a collection of dsa_chip_data structure if multiples switches are cascaded, + the master network device this switch tree is attached to needs to be + referenced -dsa_switch_tree: structure assigned to the master network device under -"dsa_ptr", this structure references a dsa_platform_data structure as well as -the tagging protocol supported by the switch tree, and which receive/transmit -function hooks should be invoked, information about the directly attached switch -is also provided: CPU port. Finally, a collection of dsa_switch are referenced -to address individual switches in the tree. +- ``dsa_switch_tree``: structure assigned to the master network device under + ``dsa_ptr``, this structure references a dsa_platform_data structure as well as + the tagging protocol supported by the switch tree, and which receive/transmit + function hooks should be invoked, information about the directly attached + switch is also provided: CPU port. Finally, a collection of dsa_switch are + referenced to address individual switches in the tree. -dsa_switch: structure describing a switch device in the tree, referencing a -dsa_switch_tree as a backpointer, slave network devices, master network device, -and a reference to the backing dsa_switch_ops +- ``dsa_switch``: structure describing a switch device in the tree, referencing + a ``dsa_switch_tree`` as a backpointer, slave network devices, master network + device, and a reference to the backing``dsa_switch_ops`` -dsa_switch_ops: structure referencing function pointers, see below for a full -description. +- ``dsa_switch_ops``: structure referencing function pointers, see below for a + full description. Design limitations ================== @@ -240,7 +243,7 @@ Limits on the number of devices and ports ----------------------------------------- DSA currently limits the number of maximum switches within a tree to 4 -(DSA_MAX_SWITCHES), and the number of ports per switch to 12 (DSA_MAX_PORTS). +(``DSA_MAX_SWITCHES``), and the number of ports per switch to 12 (``DSA_MAX_PORTS``). These limits could be extended to support larger configurations would this need arise. @@ -279,15 +282,15 @@ Interactions with other subsystems DSA currently leverages the following subsystems: -- MDIO/PHY library: drivers/net/phy/phy.c, mdio_bus.c -- Switchdev: net/switchdev/* +- MDIO/PHY library: ``drivers/net/phy/phy.c``, ``mdio_bus.c`` +- Switchdev:``net/switchdev/*`` - Device Tree for various of_* functions MDIO/PHY library ---------------- Slave network devices exposed by DSA may or may not be interfacing with PHY -devices (struct phy_device as defined in include/linux/phy.h), but the DSA +devices (``struct phy_device`` as defined in ``include/linux/phy.h)``, but the DSA subsystem deals with all possible combinations: - internal PHY devices, built into the Ethernet switch hardware @@ -296,16 +299,16 @@ subsystem deals with all possible combinations: - special, non-autonegotiated or non MDIO-managed PHY devices: SFPs, MoCA; a.k.a fixed PHYs -The PHY configuration is done by the dsa_slave_phy_setup() function and the +The PHY configuration is done by the ``dsa_slave_phy_setup()`` function and the logic basically looks like this: - if Device Tree is used, the PHY device is looked up using the standard "phy-handle" property, if found, this PHY device is created and registered - using of_phy_connect() + using ``of_phy_connect()`` - if Device Tree is used, and the PHY device is "fixed", that is, conforms to the definition of a non-MDIO managed PHY as defined in - Documentation/devicetree/bindings/net/fixed-link.txt, the PHY is registered + ``Documentation/devicetree/bindings/net/fixed-link.txt``, the PHY is registered and connected transparently using the special fixed MDIO bus driver - finally, if the PHY is built into the switch, as is very common with @@ -331,8 +334,8 @@ Device Tree ----------- DSA features a standardized binding which is documented in -Documentation/devicetree/bindings/net/dsa/dsa.txt. PHY/MDIO library helper -functions such as of_get_phy_mode(), of_phy_connect() are also used to query +``Documentation/devicetree/bindings/net/dsa/dsa.txt``. PHY/MDIO library helper +functions such as ``of_get_phy_mode()``, ``of_phy_connect()`` are also used to query per-port PHY specific details: interface connection, MDIO bus location etc.. Driver development @@ -341,8 +344,8 @@ Driver development DSA switch drivers need to implement a dsa_switch_ops structure which will contain the various members described below. -register_switch_driver() registers this dsa_switch_ops in its internal list -of drivers to probe for. unregister_switch_driver() does the exact opposite. +``register_switch_driver()`` registers this dsa_switch_ops in its internal list +of drivers to probe for. ``unregister_switch_driver()`` does the exact opposite. Unless requested differently by setting the priv_size member accordingly, DSA does not allocate any driver private context space. @@ -350,17 +353,17 @@ does not allocate any driver private context space. Switch configuration -------------------- -- tag_protocol: this is to indicate what kind of tagging protocol is supported, - should be a valid value from the dsa_tag_protocol enum +- ``tag_protocol``: this is to indicate what kind of tagging protocol is supported, + should be a valid value from the ``dsa_tag_protocol`` enum -- probe: probe routine which will be invoked by the DSA platform device upon +- ``probe``: probe routine which will be invoked by the DSA platform device upon registration to test for the presence/absence of a switch device. For MDIO devices, it is recommended to issue a read towards internal registers using the switch pseudo-PHY and return whether this is a supported device. For other buses, return a non-NULL string -- setup: setup function for the switch, this function is responsible for setting - up the dsa_switch_ops private structure with all it needs: register maps, +- ``setup``: setup function for the switch, this function is responsible for setting + up the ``dsa_switch_ops`` private structure with all it needs: register maps, interrupts, mutexes, locks etc.. This function is also expected to properly configure the switch to separate all network interfaces from each other, that is, they should be isolated by the switch hardware itself, typically by creating @@ -375,27 +378,27 @@ Switch configuration PHY devices and link management ------------------------------- -- get_phy_flags: Some switches are interfaced to various kinds of Ethernet PHYs, +- ``get_phy_flags``: Some switches are interfaced to various kinds of Ethernet PHYs, if the PHY library PHY driver needs to know about information it cannot obtain on its own (e.g.: coming from switch memory mapped registers), this function should return a 32-bits bitmask of "flags", that is private between the switch - driver and the Ethernet PHY driver in drivers/net/phy/*. + driver and the Ethernet PHY driver in ``drivers/net/phy/\*``. -- phy_read: Function invoked by the DSA slave MDIO bus when attempting to read +- ``phy_read``: Function invoked by the DSA slave MDIO bus when attempting to read the switch port MDIO registers. If unavailable, return 0xffff for each read. For builtin switch Ethernet PHYs, this function should allow reading the link status, auto-negotiation results, link partner pages etc.. -- phy_write: Function invoked by the DSA slave MDIO bus when attempting to write +- ``phy_write``: Function invoked by the DSA slave MDIO bus when attempting to write to the switch port MDIO registers. If unavailable return a negative error code. -- adjust_link: Function invoked by the PHY library when a slave network device +- ``adjust_link``: Function invoked by the PHY library when a slave network device is attached to a PHY device. This function is responsible for appropriately configuring the switch port link parameters: speed, duplex, pause based on - what the phy_device is providing. + what the ``phy_device`` is providing. -- fixed_link_update: Function invoked by the PHY library, and specifically by +- ``fixed_link_update``: Function invoked by the PHY library, and specifically by the fixed PHY driver asking the switch driver for link parameters that could not be auto-negotiated, or obtained by reading the PHY registers through MDIO. This is particularly useful for specific kinds of hardware such as QSGMII, @@ -405,87 +408,87 @@ PHY devices and link management Ethtool operations ------------------ -- get_strings: ethtool function used to query the driver's strings, will +- ``get_strings``: ethtool function used to query the driver's strings, will typically return statistics strings, private flags strings etc. -- get_ethtool_stats: ethtool function used to query per-port statistics and +- ``get_ethtool_stats``: ethtool function used to query per-port statistics and return their values. DSA overlays slave network devices general statistics: RX/TX counters from the network device, with switch driver specific statistics per port -- get_sset_count: ethtool function used to query the number of statistics items +- ``get_sset_count``: ethtool function used to query the number of statistics items -- get_wol: ethtool function used to obtain Wake-on-LAN settings per-port, this +- ``get_wol``: ethtool function used to obtain Wake-on-LAN settings per-port, this function may, for certain implementations also query the master network device Wake-on-LAN settings if this interface needs to participate in Wake-on-LAN -- set_wol: ethtool function used to configure Wake-on-LAN settings per-port, +- ``set_wol``: ethtool function used to configure Wake-on-LAN settings per-port, direct counterpart to set_wol with similar restrictions -- set_eee: ethtool function which is used to configure a switch port EEE (Green +- ``set_eee``: ethtool function which is used to configure a switch port EEE (Green Ethernet) settings, can optionally invoke the PHY library to enable EEE at the PHY level if relevant. This function should enable EEE at the switch port MAC controller and data-processing logic -- get_eee: ethtool function which is used to query a switch port EEE settings, +- ``get_eee``: ethtool function which is used to query a switch port EEE settings, this function should return the EEE state of the switch port MAC controller and data-processing logic as well as query the PHY for its currently configured EEE settings -- get_eeprom_len: ethtool function returning for a given switch the EEPROM +- ``get_eeprom_len``: ethtool function returning for a given switch the EEPROM length/size in bytes -- get_eeprom: ethtool function returning for a given switch the EEPROM contents +- ``get_eeprom``: ethtool function returning for a given switch the EEPROM contents -- set_eeprom: ethtool function writing specified data to a given switch EEPROM +- ``set_eeprom``: ethtool function writing specified data to a given switch EEPROM -- get_regs_len: ethtool function returning the register length for a given +- ``get_regs_len``: ethtool function returning the register length for a given switch -- get_regs: ethtool function returning the Ethernet switch internal register +- ``get_regs``: ethtool function returning the Ethernet switch internal register contents. This function might require user-land code in ethtool to pretty-print register values and registers Power management ---------------- -- suspend: function invoked by the DSA platform device when the system goes to +- ``suspend``: function invoked by the DSA platform device when the system goes to suspend, should quiesce all Ethernet switch activities, but keep ports participating in Wake-on-LAN active as well as additional wake-up logic if supported -- resume: function invoked by the DSA platform device when the system resumes, +- ``resume``: function invoked by the DSA platform device when the system resumes, should resume all Ethernet switch activities and re-configure the switch to be in a fully active state -- port_enable: function invoked by the DSA slave network device ndo_open +- ``port_enable``: function invoked by the DSA slave network device ndo_open function when a port is administratively brought up, this function should be fully enabling a given switch port. DSA takes care of marking the port with - BR_STATE_BLOCKING if the port is a bridge member, or BR_STATE_FORWARDING if it + ``BR_STATE_BLOCKING`` if the port is a bridge member, or ``BR_STATE_FORWARDING`` if it was not, and propagating these changes down to the hardware -- port_disable: function invoked by the DSA slave network device ndo_close +- ``port_disable``: function invoked by the DSA slave network device ndo_close function when a port is administratively brought down, this function should be fully disabling a given switch port. DSA takes care of marking the port with - BR_STATE_DISABLED and propagating changes to the hardware if this port is + ``BR_STATE_DISABLED`` and propagating changes to the hardware if this port is disabled while being a bridge member Bridge layer ------------ -- port_bridge_join: bridge layer function invoked when a given switch port is +- ``port_bridge_join``: bridge layer function invoked when a given switch port is added to a bridge, this function should be doing the necessary at the switch level to permit the joining port from being added to the relevant logical domain for it to ingress/egress traffic with other members of the bridge. -- port_bridge_leave: bridge layer function invoked when a given switch port is +- ``port_bridge_leave``: bridge layer function invoked when a given switch port is removed from a bridge, this function should be doing the necessary at the switch level to deny the leaving port from ingress/egress traffic from the remaining bridge members. When the port leaves the bridge, it should be aged out at the switch hardware for the switch to (re) learn MAC addresses behind this port. -- port_stp_state_set: bridge layer function invoked when a given switch port STP +- ``port_stp_state_set``: bridge layer function invoked when a given switch port STP state is computed by the bridge layer and should be propagated to switch hardware to forward/block/learn traffic. The switch driver is responsible for computing a STP state change based on current and asked parameters and perform @@ -494,7 +497,7 @@ Bridge layer Bridge VLAN filtering --------------------- -- port_vlan_filtering: bridge layer function invoked when the bridge gets +- ``port_vlan_filtering``: bridge layer function invoked when the bridge gets configured for turning on or off VLAN filtering. If nothing specific needs to be done at the hardware level, this callback does not need to be implemented. When VLAN filtering is turned on, the hardware must be programmed with @@ -504,61 +507,61 @@ Bridge VLAN filtering accept any 802.1Q frames irrespective of their VLAN ID, and untagged frames are allowed. -- port_vlan_prepare: bridge layer function invoked when the bridge prepares the +- ``port_vlan_prepare``: bridge layer function invoked when the bridge prepares the configuration of a VLAN on the given port. If the operation is not supported - by the hardware, this function should return -EOPNOTSUPP to inform the bridge + by the hardware, this function should return ``-EOPNOTSUPP`` to inform the bridge code to fallback to a software implementation. No hardware setup must be done in this function. See port_vlan_add for this and details. -- port_vlan_add: bridge layer function invoked when a VLAN is configured +- ``port_vlan_add``: bridge layer function invoked when a VLAN is configured (tagged or untagged) for the given switch port -- port_vlan_del: bridge layer function invoked when a VLAN is removed from the +- ``port_vlan_del``: bridge layer function invoked when a VLAN is removed from the given switch port -- port_vlan_dump: bridge layer function invoked with a switchdev callback +- ``port_vlan_dump``: bridge layer function invoked with a switchdev callback function that the driver has to call for each VLAN the given port is a member of. A switchdev object is used to carry the VID and bridge flags. -- port_fdb_add: bridge layer function invoked when the bridge wants to install a +- ``port_fdb_add``: bridge layer function invoked when the bridge wants to install a Forwarding Database entry, the switch hardware should be programmed with the specified address in the specified VLAN Id in the forwarding database associated with this VLAN ID. If the operation is not supported, this - function should return -EOPNOTSUPP to inform the bridge code to fallback to + function should return ``-EOPNOTSUPP`` to inform the bridge code to fallback to a software implementation. -Note: VLAN ID 0 corresponds to the port private database, which, in the context -of DSA, would be the its port-based VLAN, used by the associated bridge device. +.. note:: VLAN ID 0 corresponds to the port private database, which, in the context + of DSA, would be the its port-based VLAN, used by the associated bridge device. -- port_fdb_del: bridge layer function invoked when the bridge wants to remove a +- ``port_fdb_del``: bridge layer function invoked when the bridge wants to remove a Forwarding Database entry, the switch hardware should be programmed to delete the specified MAC address from the specified VLAN ID if it was mapped into this port forwarding database -- port_fdb_dump: bridge layer function invoked with a switchdev callback +- ``port_fdb_dump``: bridge layer function invoked with a switchdev callback function that the driver has to call for each MAC address known to be behind the given port. A switchdev object is used to carry the VID and FDB info. -- port_mdb_prepare: bridge layer function invoked when the bridge prepares the +- ``port_mdb_prepare``: bridge layer function invoked when the bridge prepares the installation of a multicast database entry. If the operation is not supported, - this function should return -EOPNOTSUPP to inform the bridge code to fallback + this function should return ``-EOPNOTSUPP`` to inform the bridge code to fallback to a software implementation. No hardware setup must be done in this function. - See port_fdb_add for this and details. + See ``port_fdb_add`` for this and details. -- port_mdb_add: bridge layer function invoked when the bridge wants to install +- ``port_mdb_add``: bridge layer function invoked when the bridge wants to install a multicast database entry, the switch hardware should be programmed with the specified address in the specified VLAN ID in the forwarding database associated with this VLAN ID. -Note: VLAN ID 0 corresponds to the port private database, which, in the context -of DSA, would be the its port-based VLAN, used by the associated bridge device. +.. note:: VLAN ID 0 corresponds to the port private database, which, in the context + of DSA, would be the its port-based VLAN, used by the associated bridge device. -- port_mdb_del: bridge layer function invoked when the bridge wants to remove a +- ``port_mdb_del``: bridge layer function invoked when the bridge wants to remove a multicast database entry, the switch hardware should be programmed to delete the specified MAC address from the specified VLAN ID if it was mapped into this port forwarding database. -- port_mdb_dump: bridge layer function invoked with a switchdev callback +- ``port_mdb_dump``: bridge layer function invoked with a switchdev callback function that the driver has to call for each MAC address known to be behind the given port. A switchdev object is used to carry the VID and MDB info. @@ -577,7 +580,7 @@ two subsystems and get the best of both worlds. Other hanging fruits -------------------- -- making the number of ports fully dynamic and not dependent on DSA_MAX_PORTS +- making the number of ports fully dynamic and not dependent on ``DSA_MAX_PORTS`` - allowing more than one CPU/management interface: http://comments.gmane.org/gmane.linux.network/365657 - porting more drivers from other vendors: diff --git a/Documentation/networking/dsa/index.rst b/Documentation/networking/dsa/index.rst new file mode 100644 index 0000000000000000000000000000000000000000..0e5b7a9be40600aa96e794b9b2863f802c091c4f --- /dev/null +++ b/Documentation/networking/dsa/index.rst @@ -0,0 +1,11 @@ +=============================== +Distributed Switch Architecture +=============================== + +.. toctree:: + :maxdepth: 1 + + dsa + bcm_sf2 + lan9303 + sja1105 diff --git a/Documentation/networking/dsa/lan9303.txt b/Documentation/networking/dsa/lan9303.rst similarity index 85% rename from Documentation/networking/dsa/lan9303.txt rename to Documentation/networking/dsa/lan9303.rst index 144b02b95207eeac5590f949447e9f8b5812edf5..e3c820db28ad587f4587edacfd9fd37743a98fb5 100644 --- a/Documentation/networking/dsa/lan9303.txt +++ b/Documentation/networking/dsa/lan9303.rst @@ -1,3 +1,4 @@ +============================== LAN9303 Ethernet switch driver ============================== @@ -9,10 +10,9 @@ host master network interface (e.g. fixed link). Driver details ============== -The driver is implemented as a DSA driver, see -Documentation/networking/dsa/dsa.txt. +The driver is implemented as a DSA driver, see ``Documentation/networking/dsa/dsa.rst``. -See Documentation/devicetree/bindings/net/dsa/lan9303.txt for device tree +See ``Documentation/devicetree/bindings/net/dsa/lan9303.txt`` for device tree binding. The LAN9303 can be managed both via MDIO and I2C, both supported by this driver. diff --git a/Documentation/networking/dsa/sja1105.rst b/Documentation/networking/dsa/sja1105.rst new file mode 100644 index 0000000000000000000000000000000000000000..ea7bac438cfd277cc5fb8d588dc9c6b181133307 --- /dev/null +++ b/Documentation/networking/dsa/sja1105.rst @@ -0,0 +1,220 @@ +========================= +NXP SJA1105 switch driver +========================= + +Overview +======== + +The NXP SJA1105 is a family of 6 devices: + +- SJA1105E: First generation, no TTEthernet +- SJA1105T: First generation, TTEthernet +- SJA1105P: Second generation, no TTEthernet, no SGMII +- SJA1105Q: Second generation, TTEthernet, no SGMII +- SJA1105R: Second generation, no TTEthernet, SGMII +- SJA1105S: Second generation, TTEthernet, SGMII + +These are SPI-managed automotive switches, with all ports being gigabit +capable, and supporting MII/RMII/RGMII and optionally SGMII on one port. + +Being automotive parts, their configuration interface is geared towards +set-and-forget use, with minimal dynamic interaction at runtime. They +require a static configuration to be composed by software and packed +with CRC and table headers, and sent over SPI. + +The static configuration is composed of several configuration tables. Each +table takes a number of entries. Some configuration tables can be (partially) +reconfigured at runtime, some not. Some tables are mandatory, some not: + +============================= ================== ============================= +Table Mandatory Reconfigurable +============================= ================== ============================= +Schedule no no +Schedule entry points if Scheduling no +VL Lookup no no +VL Policing if VL Lookup no +VL Forwarding if VL Lookup no +L2 Lookup no no +L2 Policing yes no +VLAN Lookup yes yes +L2 Forwarding yes partially (fully on P/Q/R/S) +MAC Config yes partially (fully on P/Q/R/S) +Schedule Params if Scheduling no +Schedule Entry Points Params if Scheduling no +VL Forwarding Params if VL Forwarding no +L2 Lookup Params no partially (fully on P/Q/R/S) +L2 Forwarding Params yes no +Clock Sync Params no no +AVB Params no no +General Params yes partially +Retagging no yes +xMII Params yes no +SGMII no yes +============================= ================== ============================= + + +Also the configuration is write-only (software cannot read it back from the +switch except for very few exceptions). + +The driver creates a static configuration at probe time, and keeps it at +all times in memory, as a shadow for the hardware state. When required to +change a hardware setting, the static configuration is also updated. +If that changed setting can be transmitted to the switch through the dynamic +reconfiguration interface, it is; otherwise the switch is reset and +reprogrammed with the updated static configuration. + +Traffic support +=============== + +The switches do not support switch tagging in hardware. But they do support +customizing the TPID by which VLAN traffic is identified as such. The switch +driver is leveraging ``CONFIG_NET_DSA_TAG_8021Q`` by requesting that special +VLANs (with a custom TPID of ``ETH_P_EDSA`` instead of ``ETH_P_8021Q``) are +installed on its ports when not in ``vlan_filtering`` mode. This does not +interfere with the reception and transmission of real 802.1Q-tagged traffic, +because the switch does no longer parse those packets as VLAN after the TPID +change. +The TPID is restored when ``vlan_filtering`` is requested by the user through +the bridge layer, and general IP termination becomes no longer possible through +the switch netdevices in this mode. + +The switches have two programmable filters for link-local destination MACs. +These are used to trap BPDUs and PTP traffic to the master netdevice, and are +further used to support STP and 1588 ordinary clock/boundary clock +functionality. + +The following traffic modes are supported over the switch netdevices: + ++--------------------+------------+------------------+------------------+ +| | Standalone | Bridged with | Bridged with | +| | ports | vlan_filtering 0 | vlan_filtering 1 | ++====================+============+==================+==================+ +| Regular traffic | Yes | Yes | No (use master) | ++--------------------+------------+------------------+------------------+ +| Management traffic | Yes | Yes | Yes | +| (BPDU, PTP) | | | | ++--------------------+------------+------------------+------------------+ + +Switching features +================== + +The driver supports the configuration of L2 forwarding rules in hardware for +port bridging. The forwarding, broadcast and flooding domain between ports can +be restricted through two methods: either at the L2 forwarding level (isolate +one bridge's ports from another's) or at the VLAN port membership level +(isolate ports within the same bridge). The final forwarding decision taken by +the hardware is a logical AND of these two sets of rules. + +The hardware tags all traffic internally with a port-based VLAN (pvid), or it +decodes the VLAN information from the 802.1Q tag. Advanced VLAN classification +is not possible. Once attributed a VLAN tag, frames are checked against the +port's membership rules and dropped at ingress if they don't match any VLAN. +This behavior is available when switch ports are enslaved to a bridge with +``vlan_filtering 1``. + +Normally the hardware is not configurable with respect to VLAN awareness, but +by changing what TPID the switch searches 802.1Q tags for, the semantics of a +bridge with ``vlan_filtering 0`` can be kept (accept all traffic, tagged or +untagged), and therefore this mode is also supported. + +Segregating the switch ports in multiple bridges is supported (e.g. 2 + 2), but +all bridges should have the same level of VLAN awareness (either both have +``vlan_filtering`` 0, or both 1). Also an inevitable limitation of the fact +that VLAN awareness is global at the switch level is that once a bridge with +``vlan_filtering`` enslaves at least one switch port, the other un-bridged +ports are no longer available for standalone traffic termination. + +Topology and loop detection through STP is supported. + +L2 FDB manipulation (add/delete/dump) is currently possible for the first +generation devices. Aging time of FDB entries, as well as enabling fully static +management (no address learning and no flooding of unknown traffic) is not yet +configurable in the driver. + +A special comment about bridging with other netdevices (illustrated with an +example): + +A board has eth0, eth1, swp0@eth1, swp1@eth1, swp2@eth1, swp3@eth1. +The switch ports (swp0-3) are under br0. +It is desired that eth0 is turned into another switched port that communicates +with swp0-3. + +If br0 has vlan_filtering 0, then eth0 can simply be added to br0 with the +intended results. +If br0 has vlan_filtering 1, then a new br1 interface needs to be created that +enslaves eth0 and eth1 (the DSA master of the switch ports). This is because in +this mode, the switch ports beneath br0 are not capable of regular traffic, and +are only used as a conduit for switchdev operations. + +Device Tree bindings and board design +===================================== + +This section references ``Documentation/devicetree/bindings/net/dsa/sja1105.txt`` +and aims to showcase some potential switch caveats. + +RMII PHY role and out-of-band signaling +--------------------------------------- + +In the RMII spec, the 50 MHz clock signals are either driven by the MAC or by +an external oscillator (but not by the PHY). +But the spec is rather loose and devices go outside it in several ways. +Some PHYs go against the spec and may provide an output pin where they source +the 50 MHz clock themselves, in an attempt to be helpful. +On the other hand, the SJA1105 is only binary configurable - when in the RMII +MAC role it will also attempt to drive the clock signal. To prevent this from +happening it must be put in RMII PHY role. +But doing so has some unintended consequences. +In the RMII spec, the PHY can transmit extra out-of-band signals via RXD[1:0]. +These are practically some extra code words (/J/ and /K/) sent prior to the +preamble of each frame. The MAC does not have this out-of-band signaling +mechanism defined by the RMII spec. +So when the SJA1105 port is put in PHY role to avoid having 2 drivers on the +clock signal, inevitably an RMII PHY-to-PHY connection is created. The SJA1105 +emulates a PHY interface fully and generates the /J/ and /K/ symbols prior to +frame preambles, which the real PHY is not expected to understand. So the PHY +simply encodes the extra symbols received from the SJA1105-as-PHY onto the +100Base-Tx wire. +On the other side of the wire, some link partners might discard these extra +symbols, while others might choke on them and discard the entire Ethernet +frames that follow along. This looks like packet loss with some link partners +but not with others. +The take-away is that in RMII mode, the SJA1105 must be let to drive the +reference clock if connected to a PHY. + +RGMII fixed-link and internal delays +------------------------------------ + +As mentioned in the bindings document, the second generation of devices has +tunable delay lines as part of the MAC, which can be used to establish the +correct RGMII timing budget. +When powered up, these can shift the Rx and Tx clocks with a phase difference +between 73.8 and 101.7 degrees. +The catch is that the delay lines need to lock onto a clock signal with a +stable frequency. This means that there must be at least 2 microseconds of +silence between the clock at the old vs at the new frequency. Otherwise the +lock is lost and the delay lines must be reset (powered down and back up). +In RGMII the clock frequency changes with link speed (125 MHz at 1000 Mbps, 25 +MHz at 100 Mbps and 2.5 MHz at 10 Mbps), and link speed might change during the +AN process. +In the situation where the switch port is connected through an RGMII fixed-link +to a link partner whose link state life cycle is outside the control of Linux +(such as a different SoC), then the delay lines would remain unlocked (and +inactive) until there is manual intervention (ifdown/ifup on the switch port). +The take-away is that in RGMII mode, the switch's internal delays are only +reliable if the link partner never changes link speeds, or if it does, it does +so in a way that is coordinated with the switch port (practically, both ends of +the fixed-link are under control of the same Linux system). +As to why would a fixed-link interface ever change link speeds: there are +Ethernet controllers out there which come out of reset in 100 Mbps mode, and +their driver inevitably needs to change the speed and clock frequency if it's +required to work at gigabit. + +MDIO bus and PHY management +--------------------------- + +The SJA1105 does not have an MDIO bus and does not perform in-band AN either. +Therefore there is no link state notification coming from the switch device. +A board would need to hook up the PHYs connected to the switch to any other +MDIO bus available to Linux within the system (e.g. to the DSA master's MDIO +bus). Link state management then works by the driver manually keeping in sync +(over SPI commands) the MAC link speed with the settings negotiated by the PHY. diff --git a/Documentation/networking/index.rst b/Documentation/networking/index.rst index 984e68f9e0269507132846517a4c4c2b8d726216..f390fe3cfdfb85bcd5999c8e0f3b3b91920d5351 100644 --- a/Documentation/networking/index.rst +++ b/Documentation/networking/index.rst @@ -9,7 +9,6 @@ Contents: netdev-FAQ af_xdp batman-adv - bpf_flow_dissector can can_ucan_protocol device_drivers/freescale/dpaa2/index @@ -25,6 +24,7 @@ Contents: device_drivers/intel/i40e device_drivers/intel/iavf device_drivers/intel/ice + dsa/index devlink-info-versions ieee802154 kapi diff --git a/Documentation/networking/ip-sysctl.txt b/Documentation/networking/ip-sysctl.txt index c4ac35234f0551bdca1f774f3570fe100912474c..725b8bea58a78bba07c7cf4c0cbd5f615d3c065f 100644 --- a/Documentation/networking/ip-sysctl.txt +++ b/Documentation/networking/ip-sysctl.txt @@ -81,6 +81,11 @@ fib_multipath_hash_policy - INTEGER 0 - Layer 3 1 - Layer 4 +fib_sync_mem - UNSIGNED INTEGER + Amount of dirty memory from fib entries that can be backlogged before + synchronize_rcu is forced. + Default: 512kB Minimum: 64kB Maximum: 64MB + ip_forward_update_priority - INTEGER Whether to update SKB priority from "TOS" field in IPv4 header after it is forwarded. The new SKB priority is mapped from TOS field value @@ -1910,16 +1915,41 @@ enhanced_dad - BOOLEAN icmp/*: ratelimit - INTEGER - Limit the maximal rates for sending ICMPv6 packets. + Limit the maximal rates for sending ICMPv6 messages. 0 to disable any limiting, otherwise the minimal space between responses in milliseconds. Default: 1000 +ratemask - list of comma separated ranges + For ICMPv6 message types matching the ranges in the ratemask, limit + the sending of the message according to ratelimit parameter. + + The format used for both input and output is a comma separated + list of ranges (e.g. "0-127,129" for ICMPv6 message type 0 to 127 and + 129). Writing to the file will clear all previous ranges of ICMPv6 + message types and update the current list with the input. + + Refer to: https://www.iana.org/assignments/icmpv6-parameters/icmpv6-parameters.xhtml + for numerical values of ICMPv6 message types, e.g. echo request is 128 + and echo reply is 129. + + Default: 0-1,3-127 (rate limit ICMPv6 errors except Packet Too Big) + echo_ignore_all - BOOLEAN If set non-zero, then the kernel will ignore all ICMP ECHO requests sent to it over the IPv6 protocol. Default: 0 +echo_ignore_multicast - BOOLEAN + If set non-zero, then the kernel will ignore all ICMP ECHO + requests sent to it over the IPv6 protocol via multicast. + Default: 0 + +echo_ignore_anycast - BOOLEAN + If set non-zero, then the kernel will ignore all ICMP ECHO + requests sent to it over the IPv6 protocol destined to anycast address. + Default: 0 + xfrm6_gc_thresh - INTEGER (Obsolete since linux-4.14) The threshold at which we will start garbage collecting for IPv6 diff --git a/Documentation/packing.txt b/Documentation/packing.txt new file mode 100644 index 0000000000000000000000000000000000000000..f830c98645f103e43bc294e090abe07842f2f447 --- /dev/null +++ b/Documentation/packing.txt @@ -0,0 +1,149 @@ +================================================ +Generic bitfield packing and unpacking functions +================================================ + +Problem statement +----------------- + +When working with hardware, one has to choose between several approaches of +interfacing with it. +One can memory-map a pointer to a carefully crafted struct over the hardware +device's memory region, and access its fields as struct members (potentially +declared as bitfields). But writing code this way would make it less portable, +due to potential endianness mismatches between the CPU and the hardware device. +Additionally, one has to pay close attention when translating register +definitions from the hardware documentation into bit field indices for the +structs. Also, some hardware (typically networking equipment) tends to group +its register fields in ways that violate any reasonable word boundaries +(sometimes even 64 bit ones). This creates the inconvenience of having to +define "high" and "low" portions of register fields within the struct. +A more robust alternative to struct field definitions would be to extract the +required fields by shifting the appropriate number of bits. But this would +still not protect from endianness mismatches, except if all memory accesses +were performed byte-by-byte. Also the code can easily get cluttered, and the +high-level idea might get lost among the many bit shifts required. +Many drivers take the bit-shifting approach and then attempt to reduce the +clutter with tailored macros, but more often than not these macros take +shortcuts that still prevent the code from being truly portable. + +The solution +------------ + +This API deals with 2 basic operations: + - Packing a CPU-usable number into a memory buffer (with hardware + constraints/quirks) + - Unpacking a memory buffer (which has hardware constraints/quirks) + into a CPU-usable number. + +The API offers an abstraction over said hardware constraints and quirks, +over CPU endianness and therefore between possible mismatches between +the two. + +The basic unit of these API functions is the u64. From the CPU's +perspective, bit 63 always means bit offset 7 of byte 7, albeit only +logically. The question is: where do we lay this bit out in memory? + +The following examples cover the memory layout of a packed u64 field. +The byte offsets in the packed buffer are always implicitly 0, 1, ... 7. +What the examples show is where the logical bytes and bits sit. + +1. Normally (no quirks), we would do it like this: + +63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 +7 6 5 4 +31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 +3 2 1 0 + +That is, the MSByte (7) of the CPU-usable u64 sits at memory offset 0, and the +LSByte (0) of the u64 sits at memory offset 7. +This corresponds to what most folks would regard to as "big endian", where +bit i corresponds to the number 2^i. This is also referred to in the code +comments as "logical" notation. + + +2. If QUIRK_MSB_ON_THE_RIGHT is set, we do it like this: + +56 57 58 59 60 61 62 63 48 49 50 51 52 53 54 55 40 41 42 43 44 45 46 47 32 33 34 35 36 37 38 39 +7 6 5 4 +24 25 26 27 28 29 30 31 16 17 18 19 20 21 22 23 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 +3 2 1 0 + +That is, QUIRK_MSB_ON_THE_RIGHT does not affect byte positioning, but +inverts bit offsets inside a byte. + + +3. If QUIRK_LITTLE_ENDIAN is set, we do it like this: + +39 38 37 36 35 34 33 32 47 46 45 44 43 42 41 40 55 54 53 52 51 50 49 48 63 62 61 60 59 58 57 56 +4 5 6 7 +7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8 23 22 21 20 19 18 17 16 31 30 29 28 27 26 25 24 +0 1 2 3 + +Therefore, QUIRK_LITTLE_ENDIAN means that inside the memory region, every +byte from each 4-byte word is placed at its mirrored position compared to +the boundary of that word. + +4. If QUIRK_MSB_ON_THE_RIGHT and QUIRK_LITTLE_ENDIAN are both set, we do it + like this: + +32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 +4 5 6 7 +0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 +0 1 2 3 + + +5. If just QUIRK_LSW32_IS_FIRST is set, we do it like this: + +31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 +3 2 1 0 +63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 +7 6 5 4 + +In this case the 8 byte memory region is interpreted as follows: first +4 bytes correspond to the least significant 4-byte word, next 4 bytes to +the more significant 4-byte word. + + +6. If QUIRK_LSW32_IS_FIRST and QUIRK_MSB_ON_THE_RIGHT are set, we do it like + this: + +24 25 26 27 28 29 30 31 16 17 18 19 20 21 22 23 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 +3 2 1 0 +56 57 58 59 60 61 62 63 48 49 50 51 52 53 54 55 40 41 42 43 44 45 46 47 32 33 34 35 36 37 38 39 +7 6 5 4 + + +7. If QUIRK_LSW32_IS_FIRST and QUIRK_LITTLE_ENDIAN are set, it looks like + this: + +7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8 23 22 21 20 19 18 17 16 31 30 29 28 27 26 25 24 +0 1 2 3 +39 38 37 36 35 34 33 32 47 46 45 44 43 42 41 40 55 54 53 52 51 50 49 48 63 62 61 60 59 58 57 56 +4 5 6 7 + + +8. If QUIRK_LSW32_IS_FIRST, QUIRK_LITTLE_ENDIAN and QUIRK_MSB_ON_THE_RIGHT + are set, it looks like this: + +0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 +0 1 2 3 +32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 +4 5 6 7 + + +We always think of our offsets as if there were no quirk, and we translate +them afterwards, before accessing the memory region. + +Intended use +------------ + +Drivers that opt to use this API first need to identify which of the above 3 +quirk combinations (for a total of 8) match what the hardware documentation +describes. Then they should wrap the packing() function, creating a new +xxx_packing() that calls it using the proper QUIRK_* one-hot bits set. + +The packing() function returns an int-encoded error code, which protects the +programmer against incorrect API use. The errors are not expected to occur +durring runtime, therefore it is reasonable for xxx_packing() to return void +and simply swallow those errors. Optionally it can dump stack or print the +error description. diff --git a/MAINTAINERS b/MAINTAINERS index 235fa2042eb19e36c29febe6ae4320a1598dcf22..920a0a1545b7260a4bd08f0b80f6ad44853872d6 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2796,10 +2796,13 @@ M: Simon Wunderlich M: Antonio Quartulli L: b.a.t.m.a.n@lists.open-mesh.org (moderated for non-subscribers) W: https://www.open-mesh.org/ +B: https://www.open-mesh.org/projects/batman-adv/issues +C: irc://chat.freenode.net/batman Q: https://patchwork.open-mesh.org/project/batman/list/ +T: git https://git.open-mesh.org/linux-merge.git S: Maintained -F: Documentation/ABI/testing/sysfs-class-net-batman-adv -F: Documentation/ABI/testing/sysfs-class-net-mesh +F: Documentation/ABI/obsolete/sysfs-class-net-batman-adv +F: Documentation/ABI/obsolete/sysfs-class-net-mesh F: Documentation/networking/batman-adv.rst F: include/uapi/linux/batadv_packet.h F: include/uapi/linux/batman_adv.h @@ -9814,6 +9817,8 @@ F: Documentation/devicetree/bindings/mmc/mtk-sd.txt MEDIATEK MT76 WIRELESS LAN DRIVER M: Felix Fietkau M: Lorenzo Bianconi +R: Ryder Lee +R: Roy Luo L: linux-wireless@vger.kernel.org S: Maintained F: drivers/net/wireless/mediatek/mt76/ @@ -9912,15 +9917,6 @@ F: drivers/net/ethernet/mellanox/mlx5/core/accel/* F: drivers/net/ethernet/mellanox/mlx5/core/fpga/* F: include/linux/mlx5/mlx5_ifc_fpga.h -MELLANOX ETHERNET INNOVA IPSEC DRIVER -R: Boris Pismenny -L: netdev@vger.kernel.org -S: Supported -W: http://www.mellanox.com -Q: http://patchwork.ozlabs.org/project/netdev/list/ -F: drivers/net/ethernet/mellanox/mlx5/core/en_ipsec/* -F: drivers/net/ethernet/mellanox/mlx5/core/ipsec* - MELLANOX ETHERNET SWITCH DRIVERS M: Jiri Pirko M: Ido Schimmel @@ -10783,6 +10779,7 @@ L: linux-block@vger.kernel.org L: nbd@other.debian.org F: Documentation/blockdev/nbd.txt F: drivers/block/nbd.c +F: include/trace/events/nbd.h F: include/uapi/linux/nbd.h NETWORK DROP MONITOR @@ -11170,6 +11167,12 @@ S: Maintained F: Documentation/devicetree/bindings/sound/sgtl5000.txt F: sound/soc/codecs/sgtl5000* +NXP SJA1105 ETHERNET SWITCH DRIVER +M: Vladimir Oltean +L: linux-kernel@vger.kernel.org +S: Maintained +F: drivers/net/dsa/sja1105 + NXP TDA998X DRM DRIVER M: Russell King S: Maintained @@ -11723,6 +11726,14 @@ L: linux-i2c@vger.kernel.org S: Orphan F: drivers/i2c/busses/i2c-pasemi.c +PACKING +M: Vladimir Oltean +L: netdev@vger.kernel.org +S: Supported +F: lib/packing.c +F: include/linux/packing.h +F: Documentation/packing.txt + PADATA PARALLEL EXECUTION MECHANISM M: Steffen Klassert L: linux-crypto@vger.kernel.org @@ -13449,6 +13460,12 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-testing.g S: Maintained F: drivers/net/wireless/realtek/rtlwifi/ +REALTEK WIRELESS DRIVER (rtw88) +M: Yan-Hsuan Chuang +L: linux-wireless@vger.kernel.org +S: Maintained +F: drivers/net/wireless/realtek/rtw88/ + RTL8XXXU WIRELESS DRIVER (rtl8xxxu) M: Jes Sorensen L: linux-wireless@vger.kernel.org diff --git a/Makefile b/Makefile index cd8b71bfba76716ae24262a9cdfc7c09b5cb9482..bcf2e7628fe50b7297761dc59919cc8e22fc9560 100644 --- a/Makefile +++ b/Makefile @@ -401,6 +401,7 @@ NM = $(CROSS_COMPILE)nm STRIP = $(CROSS_COMPILE)strip OBJCOPY = $(CROSS_COMPILE)objcopy OBJDUMP = $(CROSS_COMPILE)objdump +PAHOLE = pahole LEX = flex YACC = bison AWK = awk @@ -455,7 +456,7 @@ KBUILD_LDFLAGS := GCC_PLUGINS_CFLAGS := export ARCH SRCARCH CONFIG_SHELL HOSTCC KBUILD_HOSTCFLAGS CROSS_COMPILE AS LD CC -export CPP AR NM STRIP OBJCOPY OBJDUMP KBUILD_HOSTLDFLAGS KBUILD_HOSTLDLIBS +export CPP AR NM STRIP OBJCOPY OBJDUMP PAHOLE KBUILD_HOSTLDFLAGS KBUILD_HOSTLDLIBS export MAKE LEX YACC AWK INSTALLKERNEL PERL PYTHON PYTHON2 PYTHON3 UTS_MACHINE export HOSTCXX KBUILD_HOSTCXXFLAGS LDFLAGS_MODULE CHECK CHECKFLAGS diff --git a/arch/alpha/include/uapi/asm/sockios.h b/arch/alpha/include/uapi/asm/sockios.h index ba287e4b01bfb5e1bd137cf28c7e5410aa1271ec..af92bc27c3be49c84b433a97ffd3b991e61d61a1 100644 --- a/arch/alpha/include/uapi/asm/sockios.h +++ b/arch/alpha/include/uapi/asm/sockios.h @@ -11,7 +11,7 @@ #define SIOCSPGRP _IOW('s', 8, pid_t) #define SIOCGPGRP _IOR('s', 9, pid_t) -#define SIOCGSTAMP 0x8906 /* Get stamp (timeval) */ -#define SIOCGSTAMPNS 0x8907 /* Get stamp (timespec) */ +#define SIOCGSTAMP_OLD 0x8906 /* Get stamp (timeval) */ +#define SIOCGSTAMPNS_OLD 0x8907 /* Get stamp (timespec) */ #endif /* _ASM_ALPHA_SOCKIOS_H */ diff --git a/arch/arm/mach-mvebu/kirkwood.c b/arch/arm/mach-mvebu/kirkwood.c index 0aa88105d46e5423c1d98d29ca0f65ab7968f143..9b5f4d665374cd201b0236e3b44bc851460c8a69 100644 --- a/arch/arm/mach-mvebu/kirkwood.c +++ b/arch/arm/mach-mvebu/kirkwood.c @@ -92,7 +92,8 @@ static void __init kirkwood_dt_eth_fixup(void) continue; /* skip disabled nodes or nodes with valid MAC address*/ - if (!of_device_is_available(pnp) || of_get_mac_address(np)) + if (!of_device_is_available(pnp) || + !IS_ERR(of_get_mac_address(np))) goto eth_fixup_skip; clk = of_clk_get(pnp, 0); diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h index 9c01f04db64dbeef606488fc177c2594cd6ec1c2..ec894de0ed4efc15e09ce564c8d19fd52d0e28ba 100644 --- a/arch/arm64/include/asm/insn.h +++ b/arch/arm64/include/asm/insn.h @@ -277,6 +277,7 @@ __AARCH64_INSN_FUNCS(adrp, 0x9F000000, 0x90000000) __AARCH64_INSN_FUNCS(prfm, 0x3FC00000, 0x39800000) __AARCH64_INSN_FUNCS(prfm_lit, 0xFF000000, 0xD8000000) __AARCH64_INSN_FUNCS(str_reg, 0x3FE0EC00, 0x38206800) +__AARCH64_INSN_FUNCS(ldadd, 0x3F20FC00, 0xB8200000) __AARCH64_INSN_FUNCS(ldr_reg, 0x3FE0EC00, 0x38606800) __AARCH64_INSN_FUNCS(ldr_lit, 0xBF000000, 0x18000000) __AARCH64_INSN_FUNCS(ldrsw_lit, 0xFF000000, 0x98000000) @@ -394,6 +395,13 @@ u32 aarch64_insn_gen_load_store_ex(enum aarch64_insn_register reg, enum aarch64_insn_register state, enum aarch64_insn_size_type size, enum aarch64_insn_ldst_type type); +u32 aarch64_insn_gen_ldadd(enum aarch64_insn_register result, + enum aarch64_insn_register address, + enum aarch64_insn_register value, + enum aarch64_insn_size_type size); +u32 aarch64_insn_gen_stadd(enum aarch64_insn_register address, + enum aarch64_insn_register value, + enum aarch64_insn_size_type size); u32 aarch64_insn_gen_add_sub_imm(enum aarch64_insn_register dst, enum aarch64_insn_register src, int imm, enum aarch64_insn_variant variant, diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c index 7820a4a688faaa2edbf0771902eb14b5f66aec0a..9e2b5882cdeb935c7e67f461c461d80422037ce0 100644 --- a/arch/arm64/kernel/insn.c +++ b/arch/arm64/kernel/insn.c @@ -734,6 +734,46 @@ u32 aarch64_insn_gen_load_store_ex(enum aarch64_insn_register reg, state); } +u32 aarch64_insn_gen_ldadd(enum aarch64_insn_register result, + enum aarch64_insn_register address, + enum aarch64_insn_register value, + enum aarch64_insn_size_type size) +{ + u32 insn = aarch64_insn_get_ldadd_value(); + + switch (size) { + case AARCH64_INSN_SIZE_32: + case AARCH64_INSN_SIZE_64: + break; + default: + pr_err("%s: unimplemented size encoding %d\n", __func__, size); + return AARCH64_BREAK_FAULT; + } + + insn = aarch64_insn_encode_ldst_size(size, insn); + + insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RT, insn, + result); + + insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, + address); + + return aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RS, insn, + value); +} + +u32 aarch64_insn_gen_stadd(enum aarch64_insn_register address, + enum aarch64_insn_register value, + enum aarch64_insn_size_type size) +{ + /* + * STADD is simply encoded as an alias for LDADD with XZR as + * the destination register. + */ + return aarch64_insn_gen_ldadd(AARCH64_INSN_REG_ZR, address, + value, size); +} + static u32 aarch64_insn_encode_prfm_imm(enum aarch64_insn_prfm_type type, enum aarch64_insn_prfm_target target, enum aarch64_insn_prfm_policy policy, diff --git a/arch/arm64/net/bpf_jit.h b/arch/arm64/net/bpf_jit.h index 783de51a6c4e5b33725d4904be6b09011dd48605..76606e87233f38c8b59e64d2c0cc1f5cffc3dd7a 100644 --- a/arch/arm64/net/bpf_jit.h +++ b/arch/arm64/net/bpf_jit.h @@ -100,11 +100,9 @@ #define A64_STXR(sf, Rt, Rn, Rs) \ A64_LSX(sf, Rt, Rn, Rs, STORE_EX) -/* Prefetch */ -#define A64_PRFM(Rn, type, target, policy) \ - aarch64_insn_gen_prefetch(Rn, AARCH64_INSN_PRFM_TYPE_##type, \ - AARCH64_INSN_PRFM_TARGET_##target, \ - AARCH64_INSN_PRFM_POLICY_##policy) +/* LSE atomics */ +#define A64_STADD(sf, Rn, Rs) \ + aarch64_insn_gen_stadd(Rn, Rs, A64_SIZE(sf)) /* Add/subtract (immediate) */ #define A64_ADDSUB_IMM(sf, Rd, Rn, imm12, type) \ diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index aaddc0217e73dfce56b44f2e7f19c26a64ac9c2b..df845cee438eb3478895d51e0cb29c75e41b2f2f 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -365,7 +365,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, const bool is64 = BPF_CLASS(code) == BPF_ALU64 || BPF_CLASS(code) == BPF_JMP; const bool isdw = BPF_SIZE(code) == BPF_DW; - u8 jmp_cond; + u8 jmp_cond, reg; s32 jmp_offset; #define check_imm(bits, imm) do { \ @@ -756,19 +756,28 @@ emit_cond_jmp: break; } break; + /* STX XADD: lock *(u32 *)(dst + off) += src */ case BPF_STX | BPF_XADD | BPF_W: /* STX XADD: lock *(u64 *)(dst + off) += src */ case BPF_STX | BPF_XADD | BPF_DW: - emit_a64_mov_i(1, tmp, off, ctx); - emit(A64_ADD(1, tmp, tmp, dst), ctx); - emit(A64_PRFM(tmp, PST, L1, STRM), ctx); - emit(A64_LDXR(isdw, tmp2, tmp), ctx); - emit(A64_ADD(isdw, tmp2, tmp2, src), ctx); - emit(A64_STXR(isdw, tmp2, tmp, tmp3), ctx); - jmp_offset = -3; - check_imm19(jmp_offset); - emit(A64_CBNZ(0, tmp3, jmp_offset), ctx); + if (!off) { + reg = dst; + } else { + emit_a64_mov_i(1, tmp, off, ctx); + emit(A64_ADD(1, tmp, tmp, dst), ctx); + reg = tmp; + } + if (cpus_have_cap(ARM64_HAS_LSE_ATOMICS)) { + emit(A64_STADD(isdw, reg, src), ctx); + } else { + emit(A64_LDXR(isdw, tmp2, reg), ctx); + emit(A64_ADD(isdw, tmp2, tmp2, src), ctx); + emit(A64_STXR(isdw, tmp2, reg, tmp3), ctx); + jmp_offset = -3; + check_imm19(jmp_offset); + emit(A64_CBNZ(0, tmp3, jmp_offset), ctx); + } break; default: diff --git a/arch/ia64/include/uapi/asm/sockios.h b/arch/ia64/include/uapi/asm/sockios.h deleted file mode 100644 index f27a12f95d2096bd3614956ee47f0384756c5250..0000000000000000000000000000000000000000 --- a/arch/ia64/include/uapi/asm/sockios.h +++ /dev/null @@ -1,21 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ -#ifndef _ASM_IA64_SOCKIOS_H -#define _ASM_IA64_SOCKIOS_H - -/* - * Socket-level I/O control calls. - * - * Based on . - * - * Modified 1998, 1999 - * David Mosberger-Tang , Hewlett-Packard Co - */ -#define FIOSETOWN 0x8901 -#define SIOCSPGRP 0x8902 -#define FIOGETOWN 0x8903 -#define SIOCGPGRP 0x8904 -#define SIOCATMARK 0x8905 -#define SIOCGSTAMP 0x8906 /* Get stamp (timeval) */ -#define SIOCGSTAMPNS 0x8907 /* Get stamp (timespec) */ - -#endif /* _ASM_IA64_SOCKIOS_H */ diff --git a/arch/mips/include/uapi/asm/sockios.h b/arch/mips/include/uapi/asm/sockios.h index 5b40a88593fa8180e9af018d98535a6717bef614..66f60234f290b5a1ae3e0788dccf1e0283d80ce5 100644 --- a/arch/mips/include/uapi/asm/sockios.h +++ b/arch/mips/include/uapi/asm/sockios.h @@ -21,7 +21,7 @@ #define SIOCSPGRP _IOW('s', 8, pid_t) #define SIOCGPGRP _IOR('s', 9, pid_t) -#define SIOCGSTAMP 0x8906 /* Get stamp (timeval) */ -#define SIOCGSTAMPNS 0x8907 /* Get stamp (timespec) */ +#define SIOCGSTAMP_OLD 0x8906 /* Get stamp (timeval) */ +#define SIOCGSTAMPNS_OLD 0x8907 /* Get stamp (timespec) */ #endif /* _ASM_SOCKIOS_H */ diff --git a/arch/parisc/include/uapi/asm/sockios.h b/arch/parisc/include/uapi/asm/sockios.h deleted file mode 100644 index 66a3ba64d53f6b0832dd5f80a9a6e158a92338b2..0000000000000000000000000000000000000000 --- a/arch/parisc/include/uapi/asm/sockios.h +++ /dev/null @@ -1,14 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ -#ifndef __ARCH_PARISC_SOCKIOS__ -#define __ARCH_PARISC_SOCKIOS__ - -/* Socket-level I/O control calls. */ -#define FIOSETOWN 0x8901 -#define SIOCSPGRP 0x8902 -#define FIOGETOWN 0x8903 -#define SIOCGPGRP 0x8904 -#define SIOCATMARK 0x8905 -#define SIOCGSTAMP 0x8906 /* Get stamp (timeval) */ -#define SIOCGSTAMPNS 0x8907 /* Get stamp (timespec) */ - -#endif diff --git a/arch/powerpc/sysdev/tsi108_dev.c b/arch/powerpc/sysdev/tsi108_dev.c index 1f1af12f23e22325855b4ec2fd8af99efd969c91..c92dcac85231c6381884199cfec717a08d3e36eb 100644 --- a/arch/powerpc/sysdev/tsi108_dev.c +++ b/arch/powerpc/sysdev/tsi108_dev.c @@ -105,7 +105,7 @@ static int __init tsi108_eth_of_init(void) } mac_addr = of_get_mac_address(np); - if (mac_addr) + if (!IS_ERR(mac_addr)) memcpy(tsi_eth_data.mac_addr, mac_addr, 6); ph = of_get_property(np, "mdio-handle", NULL); diff --git a/arch/sh/include/uapi/asm/sockios.h b/arch/sh/include/uapi/asm/sockios.h index 17313d2c35272ea740dd2b65b2dab0e91925213d..ef18a668456d74df9759e4e0cb54487e0685781c 100644 --- a/arch/sh/include/uapi/asm/sockios.h +++ b/arch/sh/include/uapi/asm/sockios.h @@ -10,6 +10,7 @@ #define SIOCSPGRP _IOW('s', 8, pid_t) #define SIOCGPGRP _IOR('s', 9, pid_t) -#define SIOCGSTAMP _IOR('s', 100, struct timeval) /* Get stamp (timeval) */ -#define SIOCGSTAMPNS _IOR('s', 101, struct timespec) /* Get stamp (timespec) */ +#define SIOCGSTAMP_OLD _IOR('s', 100, struct timeval) /* Get stamp (timeval) */ +#define SIOCGSTAMPNS_OLD _IOR('s', 101, struct timespec) /* Get stamp (timespec) */ + #endif /* __ASM_SH_SOCKIOS_H */ diff --git a/arch/sparc/include/uapi/asm/sockios.h b/arch/sparc/include/uapi/asm/sockios.h deleted file mode 100644 index 18a3ec14a847bb11eab202e3f100b8a267541ec7..0000000000000000000000000000000000000000 --- a/arch/sparc/include/uapi/asm/sockios.h +++ /dev/null @@ -1,15 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ -#ifndef _ASM_SPARC_SOCKIOS_H -#define _ASM_SPARC_SOCKIOS_H - -/* Socket-level I/O control calls. */ -#define FIOSETOWN 0x8901 -#define SIOCSPGRP 0x8902 -#define FIOGETOWN 0x8903 -#define SIOCGPGRP 0x8904 -#define SIOCATMARK 0x8905 -#define SIOCGSTAMP 0x8906 /* Get stamp (timeval) */ -#define SIOCGSTAMPNS 0x8907 /* Get stamp (timespec) */ - -#endif /* !(_ASM_SPARC_SOCKIOS_H) */ - diff --git a/arch/um/drivers/vector_kern.c b/arch/um/drivers/vector_kern.c index 596e7056f37607229afa47e280f01ede39b42412..e190e4ca52e1fd11a199de61ea2d1244d8676b87 100644 --- a/arch/um/drivers/vector_kern.c +++ b/arch/um/drivers/vector_kern.c @@ -1043,7 +1043,7 @@ static int vector_net_start_xmit(struct sk_buff *skb, struct net_device *dev) vector_send(vp->tx_queue); return NETDEV_TX_OK; } - if (skb->xmit_more) { + if (netdev_xmit_more()) { mod_timer(&vp->tl, vp->coalesce); return NETDEV_TX_OK; } diff --git a/arch/x86/include/uapi/asm/sockios.h b/arch/x86/include/uapi/asm/sockios.h deleted file mode 100644 index def6d4746ee7fef64057f79e8d76ac41688671d9..0000000000000000000000000000000000000000 --- a/arch/x86/include/uapi/asm/sockios.h +++ /dev/null @@ -1 +0,0 @@ -#include diff --git a/arch/x86/net/bpf_jit_comp32.c b/arch/x86/net/bpf_jit_comp32.c index 0d9cdffce6ac0a87f24f6c23913112b65492f053..b29e82f190c761cd322591f122dcb1d6f02a6398 100644 --- a/arch/x86/net/bpf_jit_comp32.c +++ b/arch/x86/net/bpf_jit_comp32.c @@ -117,6 +117,8 @@ static bool is_simm32(s64 value) #define IA32_JLE 0x7E #define IA32_JG 0x7F +#define COND_JMP_OPCODE_INVALID (0xFF) + /* * Map eBPF registers to IA32 32bit registers or stack scratch space. * @@ -698,19 +700,12 @@ static inline void emit_ia32_neg64(const u8 dst[], bool dstk, u8 **pprog) STACK_VAR(dst_hi)); } - /* xor ecx,ecx */ - EMIT2(0x31, add_2reg(0xC0, IA32_ECX, IA32_ECX)); - /* sub dreg_lo,ecx */ - EMIT2(0x2B, add_2reg(0xC0, dreg_lo, IA32_ECX)); - /* mov dreg_lo,ecx */ - EMIT2(0x89, add_2reg(0xC0, dreg_lo, IA32_ECX)); - - /* xor ecx,ecx */ - EMIT2(0x31, add_2reg(0xC0, IA32_ECX, IA32_ECX)); - /* sbb dreg_hi,ecx */ - EMIT2(0x19, add_2reg(0xC0, dreg_hi, IA32_ECX)); - /* mov dreg_hi,ecx */ - EMIT2(0x89, add_2reg(0xC0, dreg_hi, IA32_ECX)); + /* neg dreg_lo */ + EMIT2(0xF7, add_1reg(0xD8, dreg_lo)); + /* adc dreg_hi,0x0 */ + EMIT3(0x83, add_1reg(0xD0, dreg_hi), 0x00); + /* neg dreg_hi */ + EMIT2(0xF7, add_1reg(0xD8, dreg_hi)); if (dstk) { /* mov dword ptr [ebp+off],dreg_lo */ @@ -1613,6 +1608,75 @@ static inline void emit_push_r64(const u8 src[], u8 **pprog) *pprog = prog; } +static u8 get_cond_jmp_opcode(const u8 op, bool is_cmp_lo) +{ + u8 jmp_cond; + + /* Convert BPF opcode to x86 */ + switch (op) { + case BPF_JEQ: + jmp_cond = IA32_JE; + break; + case BPF_JSET: + case BPF_JNE: + jmp_cond = IA32_JNE; + break; + case BPF_JGT: + /* GT is unsigned '>', JA in x86 */ + jmp_cond = IA32_JA; + break; + case BPF_JLT: + /* LT is unsigned '<', JB in x86 */ + jmp_cond = IA32_JB; + break; + case BPF_JGE: + /* GE is unsigned '>=', JAE in x86 */ + jmp_cond = IA32_JAE; + break; + case BPF_JLE: + /* LE is unsigned '<=', JBE in x86 */ + jmp_cond = IA32_JBE; + break; + case BPF_JSGT: + if (!is_cmp_lo) + /* Signed '>', GT in x86 */ + jmp_cond = IA32_JG; + else + /* GT is unsigned '>', JA in x86 */ + jmp_cond = IA32_JA; + break; + case BPF_JSLT: + if (!is_cmp_lo) + /* Signed '<', LT in x86 */ + jmp_cond = IA32_JL; + else + /* LT is unsigned '<', JB in x86 */ + jmp_cond = IA32_JB; + break; + case BPF_JSGE: + if (!is_cmp_lo) + /* Signed '>=', GE in x86 */ + jmp_cond = IA32_JGE; + else + /* GE is unsigned '>=', JAE in x86 */ + jmp_cond = IA32_JAE; + break; + case BPF_JSLE: + if (!is_cmp_lo) + /* Signed '<=', LE in x86 */ + jmp_cond = IA32_JLE; + else + /* LE is unsigned '<=', JBE in x86 */ + jmp_cond = IA32_JBE; + break; + default: /* to silence GCC warning */ + jmp_cond = COND_JMP_OPCODE_INVALID; + break; + } + + return jmp_cond; +} + static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, int oldproglen, struct jit_context *ctx) { @@ -2069,10 +2133,6 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, case BPF_JMP | BPF_JLT | BPF_X: case BPF_JMP | BPF_JGE | BPF_X: case BPF_JMP | BPF_JLE | BPF_X: - case BPF_JMP | BPF_JSGT | BPF_X: - case BPF_JMP | BPF_JSLE | BPF_X: - case BPF_JMP | BPF_JSLT | BPF_X: - case BPF_JMP | BPF_JSGE | BPF_X: case BPF_JMP32 | BPF_JEQ | BPF_X: case BPF_JMP32 | BPF_JNE | BPF_X: case BPF_JMP32 | BPF_JGT | BPF_X: @@ -2118,6 +2178,40 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, EMIT2(0x39, add_2reg(0xC0, dreg_lo, sreg_lo)); goto emit_cond_jmp; } + case BPF_JMP | BPF_JSGT | BPF_X: + case BPF_JMP | BPF_JSLE | BPF_X: + case BPF_JMP | BPF_JSLT | BPF_X: + case BPF_JMP | BPF_JSGE | BPF_X: { + u8 dreg_lo = dstk ? IA32_EAX : dst_lo; + u8 dreg_hi = dstk ? IA32_EDX : dst_hi; + u8 sreg_lo = sstk ? IA32_ECX : src_lo; + u8 sreg_hi = sstk ? IA32_EBX : src_hi; + + if (dstk) { + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + EMIT3(0x8B, + add_2reg(0x40, IA32_EBP, + IA32_EDX), + STACK_VAR(dst_hi)); + } + + if (sstk) { + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_ECX), + STACK_VAR(src_lo)); + EMIT3(0x8B, + add_2reg(0x40, IA32_EBP, + IA32_EBX), + STACK_VAR(src_hi)); + } + + /* cmp dreg_hi,sreg_hi */ + EMIT2(0x39, add_2reg(0xC0, dreg_hi, sreg_hi)); + EMIT2(IA32_JNE, 10); + /* cmp dreg_lo,sreg_lo */ + EMIT2(0x39, add_2reg(0xC0, dreg_lo, sreg_lo)); + goto emit_cond_jmp_signed; + } case BPF_JMP | BPF_JSET | BPF_X: case BPF_JMP32 | BPF_JSET | BPF_X: { bool is_jmp64 = BPF_CLASS(insn->code) == BPF_JMP; @@ -2194,10 +2288,6 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, case BPF_JMP | BPF_JLT | BPF_K: case BPF_JMP | BPF_JGE | BPF_K: case BPF_JMP | BPF_JLE | BPF_K: - case BPF_JMP | BPF_JSGT | BPF_K: - case BPF_JMP | BPF_JSLE | BPF_K: - case BPF_JMP | BPF_JSLT | BPF_K: - case BPF_JMP | BPF_JSGE | BPF_K: case BPF_JMP32 | BPF_JEQ | BPF_K: case BPF_JMP32 | BPF_JNE | BPF_K: case BPF_JMP32 | BPF_JGT | BPF_K: @@ -2238,50 +2328,9 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, /* cmp dreg_lo,sreg_lo */ EMIT2(0x39, add_2reg(0xC0, dreg_lo, sreg_lo)); -emit_cond_jmp: /* Convert BPF opcode to x86 */ - switch (BPF_OP(code)) { - case BPF_JEQ: - jmp_cond = IA32_JE; - break; - case BPF_JSET: - case BPF_JNE: - jmp_cond = IA32_JNE; - break; - case BPF_JGT: - /* GT is unsigned '>', JA in x86 */ - jmp_cond = IA32_JA; - break; - case BPF_JLT: - /* LT is unsigned '<', JB in x86 */ - jmp_cond = IA32_JB; - break; - case BPF_JGE: - /* GE is unsigned '>=', JAE in x86 */ - jmp_cond = IA32_JAE; - break; - case BPF_JLE: - /* LE is unsigned '<=', JBE in x86 */ - jmp_cond = IA32_JBE; - break; - case BPF_JSGT: - /* Signed '>', GT in x86 */ - jmp_cond = IA32_JG; - break; - case BPF_JSLT: - /* Signed '<', LT in x86 */ - jmp_cond = IA32_JL; - break; - case BPF_JSGE: - /* Signed '>=', GE in x86 */ - jmp_cond = IA32_JGE; - break; - case BPF_JSLE: - /* Signed '<=', LE in x86 */ - jmp_cond = IA32_JLE; - break; - default: /* to silence GCC warning */ +emit_cond_jmp: jmp_cond = get_cond_jmp_opcode(BPF_OP(code), false); + if (jmp_cond == COND_JMP_OPCODE_INVALID) return -EFAULT; - } jmp_offset = addrs[i + insn->off] - addrs[i]; if (is_imm8(jmp_offset)) { EMIT2(jmp_cond, jmp_offset); @@ -2291,7 +2340,66 @@ emit_cond_jmp: /* Convert BPF opcode to x86 */ pr_err("cond_jmp gen bug %llx\n", jmp_offset); return -EFAULT; } + break; + } + case BPF_JMP | BPF_JSGT | BPF_K: + case BPF_JMP | BPF_JSLE | BPF_K: + case BPF_JMP | BPF_JSLT | BPF_K: + case BPF_JMP | BPF_JSGE | BPF_K: { + u8 dreg_lo = dstk ? IA32_EAX : dst_lo; + u8 dreg_hi = dstk ? IA32_EDX : dst_hi; + u8 sreg_lo = IA32_ECX; + u8 sreg_hi = IA32_EBX; + u32 hi; + if (dstk) { + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + EMIT3(0x8B, + add_2reg(0x40, IA32_EBP, + IA32_EDX), + STACK_VAR(dst_hi)); + } + + /* mov ecx,imm32 */ + EMIT2_off32(0xC7, add_1reg(0xC0, IA32_ECX), imm32); + hi = imm32 & (1 << 31) ? (u32)~0 : 0; + /* mov ebx,imm32 */ + EMIT2_off32(0xC7, add_1reg(0xC0, IA32_EBX), hi); + /* cmp dreg_hi,sreg_hi */ + EMIT2(0x39, add_2reg(0xC0, dreg_hi, sreg_hi)); + EMIT2(IA32_JNE, 10); + /* cmp dreg_lo,sreg_lo */ + EMIT2(0x39, add_2reg(0xC0, dreg_lo, sreg_lo)); + + /* + * For simplicity of branch offset computation, + * let's use fixed jump coding here. + */ +emit_cond_jmp_signed: /* Check the condition for low 32-bit comparison */ + jmp_cond = get_cond_jmp_opcode(BPF_OP(code), true); + if (jmp_cond == COND_JMP_OPCODE_INVALID) + return -EFAULT; + jmp_offset = addrs[i + insn->off] - addrs[i] + 8; + if (is_simm32(jmp_offset)) { + EMIT2_off32(0x0F, jmp_cond + 0x10, jmp_offset); + } else { + pr_err("cond_jmp gen bug %llx\n", jmp_offset); + return -EFAULT; + } + EMIT2(0xEB, 6); + + /* Check the condition for high 32-bit comparison */ + jmp_cond = get_cond_jmp_opcode(BPF_OP(code), false); + if (jmp_cond == COND_JMP_OPCODE_INVALID) + return -EFAULT; + jmp_offset = addrs[i + insn->off] - addrs[i]; + if (is_simm32(jmp_offset)) { + EMIT2_off32(0x0F, jmp_cond + 0x10, jmp_offset); + } else { + pr_err("cond_jmp gen bug %llx\n", jmp_offset); + return -EFAULT; + } break; } case BPF_JMP | BPF_JA: diff --git a/arch/xtensa/include/uapi/asm/sockios.h b/arch/xtensa/include/uapi/asm/sockios.h index fb8ac36071894fd47a09f0a2970f046ba480c458..1a1f58f4b75a293cf355a5638ad97a06b0de7626 100644 --- a/arch/xtensa/include/uapi/asm/sockios.h +++ b/arch/xtensa/include/uapi/asm/sockios.h @@ -26,7 +26,7 @@ #define SIOCSPGRP _IOW('s', 8, pid_t) #define SIOCGPGRP _IOR('s', 9, pid_t) -#define SIOCGSTAMP 0x8906 /* Get stamp (timeval) */ -#define SIOCGSTAMPNS 0x8907 /* Get stamp (timespec) */ +#define SIOCGSTAMP_OLD 0x8906 /* Get stamp (timeval) */ +#define SIOCGSTAMPNS_OLD 0x8907 /* Get stamp (timespec) */ #endif /* _XTENSA_SOCKIOS_H */ diff --git a/crypto/crypto_user_base.c b/crypto/crypto_user_base.c index f25d3f32c9c26901bb5e46ed21b9f35afbd2568a..e48da3b75c71d400d77de8512a9aa79409ae65d3 100644 --- a/crypto/crypto_user_base.c +++ b/crypto/crypto_user_base.c @@ -465,8 +465,8 @@ static int crypto_user_rcv_msg(struct sk_buff *skb, struct nlmsghdr *nlh, return err; } - err = nlmsg_parse(nlh, crypto_msg_min[type], attrs, CRYPTOCFGA_MAX, - crypto_policy, extack); + err = nlmsg_parse_deprecated(nlh, crypto_msg_min[type], attrs, + CRYPTOCFGA_MAX, crypto_policy, extack); if (err < 0) return err; diff --git a/drivers/atm/iphase.c b/drivers/atm/iphase.c index 82532c299bb5964a429e81353b9c5f94d9bb5ed2..5278c57dce739f9666ba684191d7c5717be7cd89 100644 --- a/drivers/atm/iphase.c +++ b/drivers/atm/iphase.c @@ -2826,8 +2826,8 @@ static int ia_ioctl(struct atm_dev *dev, unsigned int cmd, void __user *arg) case 0x6: { ia_cmds.status = 0; - printk("skb = 0x%lx\n", (long)skb_peek(&iadev->tx_backlog)); - printk("rtn_q: 0x%lx\n",(long)ia_deque_rtn_q(&iadev->tx_return_q)); + printk("skb = 0x%p\n", skb_peek(&iadev->tx_backlog)); + printk("rtn_q: 0x%p\n",ia_deque_rtn_q(&iadev->tx_return_q)); } break; case 0x8: diff --git a/drivers/block/drbd/drbd_nl.c b/drivers/block/drbd/drbd_nl.c index f2471172a96193fbcd5dc68adc4415286647290e..1cb5a0b85fd9603d0575a59958e8d23b4924622f 100644 --- a/drivers/block/drbd/drbd_nl.c +++ b/drivers/block/drbd/drbd_nl.c @@ -114,7 +114,7 @@ static int drbd_msg_put_info(struct sk_buff *skb, const char *info) if (!info || !info[0]) return 0; - nla = nla_nest_start(skb, DRBD_NLA_CFG_REPLY); + nla = nla_nest_start_noflag(skb, DRBD_NLA_CFG_REPLY); if (!nla) return err; @@ -135,7 +135,7 @@ static int drbd_msg_sprintf_info(struct sk_buff *skb, const char *fmt, ...) int err = -EMSGSIZE; int len; - nla = nla_nest_start(skb, DRBD_NLA_CFG_REPLY); + nla = nla_nest_start_noflag(skb, DRBD_NLA_CFG_REPLY); if (!nla) return err; @@ -3269,7 +3269,7 @@ static int nla_put_drbd_cfg_context(struct sk_buff *skb, struct drbd_device *device) { struct nlattr *nla; - nla = nla_nest_start(skb, DRBD_NLA_CFG_CONTEXT); + nla = nla_nest_start_noflag(skb, DRBD_NLA_CFG_CONTEXT); if (!nla) goto nla_put_failure; if (device && @@ -3837,7 +3837,7 @@ static int nla_put_status_info(struct sk_buff *skb, struct drbd_device *device, if (err) goto nla_put_failure; - nla = nla_nest_start(skb, DRBD_NLA_STATE_INFO); + nla = nla_nest_start_noflag(skb, DRBD_NLA_STATE_INFO); if (!nla) goto nla_put_failure; if (nla_put_u32(skb, T_sib_reason, sib ? sib->sib_reason : SIB_GET_STATUS_REPLY) || diff --git a/drivers/block/drbd/drbd_nla.c b/drivers/block/drbd/drbd_nla.c index 8e261cb5198bd1ab03f4926ccb0ebc35da766d67..6a09b0b98018573966cc39e9c73bd2a584448e5b 100644 --- a/drivers/block/drbd/drbd_nla.c +++ b/drivers/block/drbd/drbd_nla.c @@ -35,7 +35,8 @@ int drbd_nla_parse_nested(struct nlattr *tb[], int maxtype, struct nlattr *nla, err = drbd_nla_check_mandatory(maxtype, nla); if (!err) - err = nla_parse_nested(tb, maxtype, nla, policy, NULL); + err = nla_parse_nested_deprecated(tb, maxtype, nla, policy, + NULL); return err; } diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c index 90ba9f4c03f3b3899ae218c44bb0b832df45734f..053958a8a2bab3c9aeafc50776152d4032a51b76 100644 --- a/drivers/block/nbd.c +++ b/drivers/block/nbd.c @@ -44,6 +44,9 @@ #include #include +#define CREATE_TRACE_POINTS +#include + static DEFINE_IDR(nbd_index_idr); static DEFINE_MUTEX(nbd_index_mutex); static int nbd_total_devices = 0; @@ -510,6 +513,10 @@ static int nbd_send_cmd(struct nbd_device *nbd, struct nbd_cmd *cmd, int index) if (sent) { if (sent >= sizeof(request)) { skip = sent - sizeof(request); + + /* initialize handle for tracing purposes */ + handle = nbd_cmd_handle(cmd); + goto send_pages; } iov_iter_advance(&from, sent); @@ -526,11 +533,14 @@ static int nbd_send_cmd(struct nbd_device *nbd, struct nbd_cmd *cmd, int index) handle = nbd_cmd_handle(cmd); memcpy(request.handle, &handle, sizeof(handle)); + trace_nbd_send_request(&request, nbd->index, blk_mq_rq_from_pdu(cmd)); + dev_dbg(nbd_to_dev(nbd), "request %p: sending control (%s@%llu,%uB)\n", req, nbdcmd_to_ascii(type), (unsigned long long)blk_rq_pos(req) << 9, blk_rq_bytes(req)); result = sock_xmit(nbd, index, 1, &from, (type == NBD_CMD_WRITE) ? MSG_MORE : 0, &sent); + trace_nbd_header_sent(req, handle); if (result <= 0) { if (was_interrupted(result)) { /* If we havne't sent anything we can just return BUSY, @@ -603,6 +613,7 @@ send_pages: bio = next; } out: + trace_nbd_payload_sent(req, handle); nsock->pending = NULL; nsock->sent = 0; return 0; @@ -650,6 +661,7 @@ static struct nbd_cmd *nbd_read_stat(struct nbd_device *nbd, int index) tag, req); return ERR_PTR(-ENOENT); } + trace_nbd_header_received(req, handle); cmd = blk_mq_rq_to_pdu(req); mutex_lock(&cmd->lock); @@ -703,6 +715,7 @@ static struct nbd_cmd *nbd_read_stat(struct nbd_device *nbd, int index) } } out: + trace_nbd_payload_received(req, handle); mutex_unlock(&cmd->lock); return ret ? ERR_PTR(ret) : cmd; } @@ -1797,8 +1810,10 @@ again: ret = -EINVAL; goto out; } - ret = nla_parse_nested(socks, NBD_SOCK_MAX, attr, - nbd_sock_policy, info->extack); + ret = nla_parse_nested_deprecated(socks, NBD_SOCK_MAX, + attr, + nbd_sock_policy, + info->extack); if (ret != 0) { printk(KERN_ERR "nbd: error processing sock list\n"); ret = -EINVAL; @@ -1968,8 +1983,10 @@ static int nbd_genl_reconfigure(struct sk_buff *skb, struct genl_info *info) ret = -EINVAL; goto out; } - ret = nla_parse_nested(socks, NBD_SOCK_MAX, attr, - nbd_sock_policy, info->extack); + ret = nla_parse_nested_deprecated(socks, NBD_SOCK_MAX, + attr, + nbd_sock_policy, + info->extack); if (ret != 0) { printk(KERN_ERR "nbd: error processing sock list\n"); ret = -EINVAL; @@ -1999,22 +2016,22 @@ out: static const struct genl_ops nbd_connect_genl_ops[] = { { .cmd = NBD_CMD_CONNECT, - .policy = nbd_attr_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nbd_genl_connect, }, { .cmd = NBD_CMD_DISCONNECT, - .policy = nbd_attr_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nbd_genl_disconnect, }, { .cmd = NBD_CMD_RECONFIGURE, - .policy = nbd_attr_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nbd_genl_reconfigure, }, { .cmd = NBD_CMD_STATUS, - .policy = nbd_attr_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nbd_genl_status, }, }; @@ -2031,6 +2048,7 @@ static struct genl_family nbd_genl_family __ro_after_init = { .ops = nbd_connect_genl_ops, .n_ops = ARRAY_SIZE(nbd_connect_genl_ops), .maxattr = NBD_ATTR_MAX, + .policy = nbd_attr_policy, .mcgrps = nbd_mcast_grps, .n_mcgrps = ARRAY_SIZE(nbd_mcast_grps), }; @@ -2050,7 +2068,7 @@ static int populate_nbd_status(struct nbd_device *nbd, struct sk_buff *reply) */ if (refcount_read(&nbd->config_refs)) connected = 1; - dev_opt = nla_nest_start(reply, NBD_DEVICE_ITEM); + dev_opt = nla_nest_start_noflag(reply, NBD_DEVICE_ITEM); if (!dev_opt) return -EMSGSIZE; ret = nla_put_u32(reply, NBD_DEVICE_INDEX, nbd->index); @@ -2098,7 +2116,7 @@ static int nbd_genl_status(struct sk_buff *skb, struct genl_info *info) goto out; } - dev_list = nla_nest_start(reply, NBD_ATTR_DEVICE_LIST); + dev_list = nla_nest_start_noflag(reply, NBD_ATTR_DEVICE_LIST); if (index == -1) { ret = idr_for_each(&nbd_index_idr, &status_cb, reply); if (ret) { diff --git a/drivers/bluetooth/Kconfig b/drivers/bluetooth/Kconfig index 7b2e76e7f22fb50799aca97eb034e8e12dae4365..b9c34ff9a0d310ad450f68ff83465707ffdad1e1 100644 --- a/drivers/bluetooth/Kconfig +++ b/drivers/bluetooth/Kconfig @@ -336,7 +336,7 @@ config BT_MRVL The core driver to support Marvell Bluetooth devices. This driver is required if you want to support - Marvell Bluetooth devices, such as 8688/8787/8797/8887/8897/8977/8997. + Marvell Bluetooth devices, such as 8688/8787/8797/8887/8897/8977/8987/8997. Say Y here to compile Marvell Bluetooth driver into the kernel or say M to compile it as module. @@ -350,7 +350,7 @@ config BT_MRVL_SDIO The driver for Marvell Bluetooth chipsets with SDIO interface. This driver is required if you want to use Marvell Bluetooth - devices with SDIO interface. Currently SD8688/SD8787/SD8797/SD8887/SD8897/SD8977/SD8997 + devices with SDIO interface. Currently SD8688/SD8787/SD8797/SD8887/SD8897/SD8977/SD8987/SD8997 chipsets are supported. Say Y here to compile support for Marvell BT-over-SDIO driver @@ -379,6 +379,17 @@ config BT_WILINK Say Y here to compile support for Texas Instrument's WiLink7 driver into the kernel or say M to compile it as module (btwilink). +config BT_MTKSDIO + tristate "MediaTek HCI SDIO driver" + depends on MMC + help + MediaTek Bluetooth HCI SDIO driver. + This driver is required if you want to use MediaTek Bluetooth + with SDIO interface. + + Say Y here to compile support for MediaTek Bluetooth SDIO devices + into the kernel or say M to compile it as module (btmtksdio). + config BT_MTKUART tristate "MediaTek HCI UART driver" depends on SERIAL_DEV_BUS diff --git a/drivers/bluetooth/Makefile b/drivers/bluetooth/Makefile index b7e393cfc1e31470a05978361015788edae3807d..34887b9b3a8597e96b2bc981c821343861dbd60e 100644 --- a/drivers/bluetooth/Makefile +++ b/drivers/bluetooth/Makefile @@ -20,6 +20,7 @@ obj-$(CONFIG_BT_ATH3K) += ath3k.o obj-$(CONFIG_BT_MRVL) += btmrvl.o obj-$(CONFIG_BT_MRVL_SDIO) += btmrvl_sdio.o obj-$(CONFIG_BT_WILINK) += btwilink.o +obj-$(CONFIG_BT_MTKSDIO) += btmtksdio.o obj-$(CONFIG_BT_MTKUART) += btmtkuart.o obj-$(CONFIG_BT_QCOMSMD) += btqcomsmd.o obj-$(CONFIG_BT_BCM) += btbcm.o diff --git a/drivers/bluetooth/btbcm.c b/drivers/bluetooth/btbcm.c index d5d6e6e5da3bf99d223bc37f3255399e8c7b85fe..8e17963ab65ae5ecaafc1864b37ea9e0d22a0fa1 100644 --- a/drivers/bluetooth/btbcm.c +++ b/drivers/bluetooth/btbcm.c @@ -34,9 +34,11 @@ #define BDADDR_BCM20702A0 (&(bdaddr_t) {{0x00, 0xa0, 0x02, 0x70, 0x20, 0x00}}) #define BDADDR_BCM20702A1 (&(bdaddr_t) {{0x00, 0x00, 0xa0, 0x02, 0x70, 0x20}}) +#define BDADDR_BCM2076B1 (&(bdaddr_t) {{0x79, 0x56, 0x00, 0xa0, 0x76, 0x20}}) #define BDADDR_BCM43430A0 (&(bdaddr_t) {{0xac, 0x1f, 0x12, 0xa0, 0x43, 0x43}}) #define BDADDR_BCM4324B3 (&(bdaddr_t) {{0x00, 0x00, 0x00, 0xb3, 0x24, 0x43}}) #define BDADDR_BCM4330B1 (&(bdaddr_t) {{0x00, 0x00, 0x00, 0xb1, 0x30, 0x43}}) +#define BDADDR_BCM43341B (&(bdaddr_t) {{0xac, 0x1f, 0x00, 0x1b, 0x34, 0x43}}) int btbcm_check_bdaddr(struct hci_dev *hdev) { @@ -69,6 +71,9 @@ int btbcm_check_bdaddr(struct hci_dev *hdev) * The address 20:70:02:A0:00:00 indicates a BCM20702A1 controller * with no configured address. * + * The address 20:76:A0:00:56:79 indicates a BCM2076B1 controller + * with no configured address. + * * The address 43:24:B3:00:00:00 indicates a BCM4324B3 controller * with waiting for configuration state. * @@ -80,9 +85,11 @@ int btbcm_check_bdaddr(struct hci_dev *hdev) */ if (!bacmp(&bda->bdaddr, BDADDR_BCM20702A0) || !bacmp(&bda->bdaddr, BDADDR_BCM20702A1) || + !bacmp(&bda->bdaddr, BDADDR_BCM2076B1) || !bacmp(&bda->bdaddr, BDADDR_BCM4324B3) || !bacmp(&bda->bdaddr, BDADDR_BCM4330B1) || - !bacmp(&bda->bdaddr, BDADDR_BCM43430A0)) { + !bacmp(&bda->bdaddr, BDADDR_BCM43430A0) || + !bacmp(&bda->bdaddr, BDADDR_BCM43341B)) { bt_dev_info(hdev, "BCM: Using default device address (%pMR)", &bda->bdaddr); set_bit(HCI_QUIRK_INVALID_BDADDR, &hdev->quirks); @@ -333,6 +340,7 @@ struct bcm_subver_table { static const struct bcm_subver_table bcm_uart_subver_table[] = { { 0x4103, "BCM4330B1" }, /* 002.001.003 */ { 0x410e, "BCM43341B0" }, /* 002.001.014 */ + { 0x4204, "BCM2076B1" }, /* 002.002.004 */ { 0x4406, "BCM4324B3" }, /* 002.004.006 */ { 0x6109, "BCM4335C0" }, /* 003.001.009 */ { 0x610c, "BCM4354" }, /* 003.001.012 */ diff --git a/drivers/bluetooth/btmrvl_sdio.c b/drivers/bluetooth/btmrvl_sdio.c index 047b75ce1deb76b31d6d52a78028967f954a1a9d..0f3a020703ab233d871752f9c6f6d9bce7408f42 100644 --- a/drivers/bluetooth/btmrvl_sdio.c +++ b/drivers/bluetooth/btmrvl_sdio.c @@ -235,6 +235,29 @@ static const struct btmrvl_sdio_card_reg btmrvl_reg_8977 = { .fw_dump_end = 0xf8, }; +static const struct btmrvl_sdio_card_reg btmrvl_reg_8987 = { + .cfg = 0x00, + .host_int_mask = 0x08, + .host_intstatus = 0x0c, + .card_status = 0x5c, + .sq_read_base_addr_a0 = 0xf8, + .sq_read_base_addr_a1 = 0xf9, + .card_revision = 0xc8, + .card_fw_status0 = 0xe8, + .card_fw_status1 = 0xe9, + .card_rx_len = 0xea, + .card_rx_unit = 0xeb, + .io_port_0 = 0xe4, + .io_port_1 = 0xe5, + .io_port_2 = 0xe6, + .int_read_to_clear = true, + .host_int_rsr = 0x04, + .card_misc_cfg = 0xd8, + .fw_dump_ctrl = 0xf0, + .fw_dump_start = 0xf1, + .fw_dump_end = 0xf8, +}; + static const struct btmrvl_sdio_card_reg btmrvl_reg_8997 = { .cfg = 0x00, .host_int_mask = 0x08, @@ -312,6 +335,15 @@ static const struct btmrvl_sdio_device btmrvl_sdio_sd8977 = { .supports_fw_dump = true, }; +static const struct btmrvl_sdio_device btmrvl_sdio_sd8987 = { + .helper = NULL, + .firmware = "mrvl/sd8987_uapsta.bin", + .reg = &btmrvl_reg_8987, + .support_pscan_win_report = true, + .sd_blksz_fw_dl = 256, + .supports_fw_dump = true, +}; + static const struct btmrvl_sdio_device btmrvl_sdio_sd8997 = { .helper = NULL, .firmware = "mrvl/sd8997_uapsta.bin", @@ -343,6 +375,9 @@ static const struct sdio_device_id btmrvl_sdio_ids[] = { /* Marvell SD8977 Bluetooth device */ { SDIO_DEVICE(SDIO_VENDOR_ID_MARVELL, 0x9146), .driver_data = (unsigned long)&btmrvl_sdio_sd8977 }, + /* Marvell SD8987 Bluetooth device */ + { SDIO_DEVICE(SDIO_VENDOR_ID_MARVELL, 0x914A), + .driver_data = (unsigned long)&btmrvl_sdio_sd8987 }, /* Marvell SD8997 Bluetooth device */ { SDIO_DEVICE(SDIO_VENDOR_ID_MARVELL, 0x9142), .driver_data = (unsigned long)&btmrvl_sdio_sd8997 }, @@ -1797,4 +1832,5 @@ MODULE_FIRMWARE("mrvl/sd8797_uapsta.bin"); MODULE_FIRMWARE("mrvl/sd8887_uapsta.bin"); MODULE_FIRMWARE("mrvl/sd8897_uapsta.bin"); MODULE_FIRMWARE("mrvl/sd8977_uapsta.bin"); +MODULE_FIRMWARE("mrvl/sd8987_uapsta.bin"); MODULE_FIRMWARE("mrvl/sd8997_uapsta.bin"); diff --git a/drivers/bluetooth/btmtksdio.c b/drivers/bluetooth/btmtksdio.c new file mode 100644 index 0000000000000000000000000000000000000000..8133382884537048cd6352243fbf702ef9ea001d --- /dev/null +++ b/drivers/bluetooth/btmtksdio.c @@ -0,0 +1,1101 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2019 MediaTek Inc. + +/* + * Bluetooth support for MediaTek SDIO devices + * + * This file is written based on btsdio.c and btmtkuart.c. + * + * Author: Sean Wang + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#include +#include + +#include "h4_recv.h" + +#define VERSION "0.1" + +#define FIRMWARE_MT7663 "mediatek/mt7663pr2h.bin" +#define FIRMWARE_MT7668 "mediatek/mt7668pr2h.bin" + +#define MTKBTSDIO_AUTOSUSPEND_DELAY 8000 + +static bool enable_autosuspend; + +struct btmtksdio_data { + const char *fwname; +}; + +static const struct btmtksdio_data mt7663_data = { + .fwname = FIRMWARE_MT7663, +}; + +static const struct btmtksdio_data mt7668_data = { + .fwname = FIRMWARE_MT7668, +}; + +static const struct sdio_device_id btmtksdio_table[] = { + {SDIO_DEVICE(SDIO_VENDOR_ID_MEDIATEK, 0x7663), + .driver_data = (kernel_ulong_t)&mt7663_data }, + {SDIO_DEVICE(SDIO_VENDOR_ID_MEDIATEK, 0x7668), + .driver_data = (kernel_ulong_t)&mt7668_data }, + { } /* Terminating entry */ +}; + +#define MTK_REG_CHLPCR 0x4 /* W1S */ +#define C_INT_EN_SET BIT(0) +#define C_INT_EN_CLR BIT(1) +#define C_FW_OWN_REQ_SET BIT(8) /* For write */ +#define C_COM_DRV_OWN BIT(8) /* For read */ +#define C_FW_OWN_REQ_CLR BIT(9) + +#define MTK_REG_CSDIOCSR 0x8 +#define SDIO_RE_INIT_EN BIT(0) +#define SDIO_INT_CTL BIT(2) + +#define MTK_REG_CHCR 0xc +#define C_INT_CLR_CTRL BIT(1) + +/* CHISR have the same bits field definition with CHIER */ +#define MTK_REG_CHISR 0x10 +#define MTK_REG_CHIER 0x14 +#define FW_OWN_BACK_INT BIT(0) +#define RX_DONE_INT BIT(1) +#define TX_EMPTY BIT(2) +#define TX_FIFO_OVERFLOW BIT(8) +#define RX_PKT_LEN GENMASK(31, 16) + +#define MTK_REG_CTDR 0x18 + +#define MTK_REG_CRDR 0x1c + +#define MTK_SDIO_BLOCK_SIZE 256 + +#define BTMTKSDIO_TX_WAIT_VND_EVT 1 + +enum { + MTK_WMT_PATCH_DWNLD = 0x1, + MTK_WMT_TEST = 0x2, + MTK_WMT_WAKEUP = 0x3, + MTK_WMT_HIF = 0x4, + MTK_WMT_FUNC_CTRL = 0x6, + MTK_WMT_RST = 0x7, + MTK_WMT_SEMAPHORE = 0x17, +}; + +enum { + BTMTK_WMT_INVALID, + BTMTK_WMT_PATCH_UNDONE, + BTMTK_WMT_PATCH_DONE, + BTMTK_WMT_ON_UNDONE, + BTMTK_WMT_ON_DONE, + BTMTK_WMT_ON_PROGRESS, +}; + +struct mtkbtsdio_hdr { + __le16 len; + __le16 reserved; + u8 bt_type; +} __packed; + +struct mtk_wmt_hdr { + u8 dir; + u8 op; + __le16 dlen; + u8 flag; +} __packed; + +struct mtk_hci_wmt_cmd { + struct mtk_wmt_hdr hdr; + u8 data[256]; +} __packed; + +struct btmtk_hci_wmt_evt { + struct hci_event_hdr hhdr; + struct mtk_wmt_hdr whdr; +} __packed; + +struct btmtk_hci_wmt_evt_funcc { + struct btmtk_hci_wmt_evt hwhdr; + __be16 status; +} __packed; + +struct btmtk_tci_sleep { + u8 mode; + __le16 duration; + __le16 host_duration; + u8 host_wakeup_pin; + u8 time_compensation; +} __packed; + +struct btmtk_hci_wmt_params { + u8 op; + u8 flag; + u16 dlen; + const void *data; + u32 *status; +}; + +struct btmtksdio_dev { + struct hci_dev *hdev; + struct sdio_func *func; + struct device *dev; + + struct work_struct tx_work; + unsigned long tx_state; + struct sk_buff_head txq; + + struct sk_buff *evt_skb; + + const struct btmtksdio_data *data; +}; + +static int mtk_hci_wmt_sync(struct hci_dev *hdev, + struct btmtk_hci_wmt_params *wmt_params) +{ + struct btmtksdio_dev *bdev = hci_get_drvdata(hdev); + struct btmtk_hci_wmt_evt_funcc *wmt_evt_funcc; + u32 hlen, status = BTMTK_WMT_INVALID; + struct btmtk_hci_wmt_evt *wmt_evt; + struct mtk_hci_wmt_cmd wc; + struct mtk_wmt_hdr *hdr; + int err; + + hlen = sizeof(*hdr) + wmt_params->dlen; + if (hlen > 255) + return -EINVAL; + + hdr = (struct mtk_wmt_hdr *)&wc; + hdr->dir = 1; + hdr->op = wmt_params->op; + hdr->dlen = cpu_to_le16(wmt_params->dlen + 1); + hdr->flag = wmt_params->flag; + memcpy(wc.data, wmt_params->data, wmt_params->dlen); + + set_bit(BTMTKSDIO_TX_WAIT_VND_EVT, &bdev->tx_state); + + err = __hci_cmd_send(hdev, 0xfc6f, hlen, &wc); + if (err < 0) { + clear_bit(BTMTKSDIO_TX_WAIT_VND_EVT, &bdev->tx_state); + return err; + } + + /* The vendor specific WMT commands are all answered by a vendor + * specific event and will not have the Command Status or Command + * Complete as with usual HCI command flow control. + * + * After sending the command, wait for BTMTKSDIO_TX_WAIT_VND_EVT + * state to be cleared. The driver specific event receive routine + * will clear that state and with that indicate completion of the + * WMT command. + */ + err = wait_on_bit_timeout(&bdev->tx_state, BTMTKSDIO_TX_WAIT_VND_EVT, + TASK_INTERRUPTIBLE, HCI_INIT_TIMEOUT); + if (err == -EINTR) { + bt_dev_err(hdev, "Execution of wmt command interrupted"); + clear_bit(BTMTKSDIO_TX_WAIT_VND_EVT, &bdev->tx_state); + return err; + } + + if (err) { + bt_dev_err(hdev, "Execution of wmt command timed out"); + clear_bit(BTMTKSDIO_TX_WAIT_VND_EVT, &bdev->tx_state); + return -ETIMEDOUT; + } + + /* Parse and handle the return WMT event */ + wmt_evt = (struct btmtk_hci_wmt_evt *)bdev->evt_skb->data; + if (wmt_evt->whdr.op != hdr->op) { + bt_dev_err(hdev, "Wrong op received %d expected %d", + wmt_evt->whdr.op, hdr->op); + err = -EIO; + goto err_free_skb; + } + + switch (wmt_evt->whdr.op) { + case MTK_WMT_SEMAPHORE: + if (wmt_evt->whdr.flag == 2) + status = BTMTK_WMT_PATCH_UNDONE; + else + status = BTMTK_WMT_PATCH_DONE; + break; + case MTK_WMT_FUNC_CTRL: + wmt_evt_funcc = (struct btmtk_hci_wmt_evt_funcc *)wmt_evt; + if (be16_to_cpu(wmt_evt_funcc->status) == 0x404) + status = BTMTK_WMT_ON_DONE; + else if (be16_to_cpu(wmt_evt_funcc->status) == 0x420) + status = BTMTK_WMT_ON_PROGRESS; + else + status = BTMTK_WMT_ON_UNDONE; + break; + } + + if (wmt_params->status) + *wmt_params->status = status; + +err_free_skb: + kfree_skb(bdev->evt_skb); + bdev->evt_skb = NULL; + + return err; +} + +static int btmtksdio_tx_packet(struct btmtksdio_dev *bdev, + struct sk_buff *skb) +{ + struct mtkbtsdio_hdr *sdio_hdr; + int err; + + /* Make sure that there are enough rooms for SDIO header */ + if (unlikely(skb_headroom(skb) < sizeof(*sdio_hdr))) { + err = pskb_expand_head(skb, sizeof(*sdio_hdr), 0, + GFP_ATOMIC); + if (err < 0) + return err; + } + + /* Prepend MediaTek SDIO Specific Header */ + skb_push(skb, sizeof(*sdio_hdr)); + + sdio_hdr = (void *)skb->data; + sdio_hdr->len = cpu_to_le16(skb->len); + sdio_hdr->reserved = cpu_to_le16(0); + sdio_hdr->bt_type = hci_skb_pkt_type(skb); + + err = sdio_writesb(bdev->func, MTK_REG_CTDR, skb->data, + round_up(skb->len, MTK_SDIO_BLOCK_SIZE)); + if (err < 0) + goto err_skb_pull; + + bdev->hdev->stat.byte_tx += skb->len; + + kfree_skb(skb); + + return 0; + +err_skb_pull: + skb_pull(skb, sizeof(*sdio_hdr)); + + return err; +} + +static u32 btmtksdio_drv_own_query(struct btmtksdio_dev *bdev) +{ + return sdio_readl(bdev->func, MTK_REG_CHLPCR, NULL); +} + +static void btmtksdio_tx_work(struct work_struct *work) +{ + struct btmtksdio_dev *bdev = container_of(work, struct btmtksdio_dev, + tx_work); + struct sk_buff *skb; + int err; + + pm_runtime_get_sync(bdev->dev); + + sdio_claim_host(bdev->func); + + while ((skb = skb_dequeue(&bdev->txq))) { + err = btmtksdio_tx_packet(bdev, skb); + if (err < 0) { + bdev->hdev->stat.err_tx++; + skb_queue_head(&bdev->txq, skb); + break; + } + } + + sdio_release_host(bdev->func); + + pm_runtime_mark_last_busy(bdev->dev); + pm_runtime_put_autosuspend(bdev->dev); +} + +static int btmtksdio_recv_event(struct hci_dev *hdev, struct sk_buff *skb) +{ + struct btmtksdio_dev *bdev = hci_get_drvdata(hdev); + struct hci_event_hdr *hdr = (void *)skb->data; + int err; + + /* Fix up the vendor event id with 0xff for vendor specific instead + * of 0xe4 so that event send via monitoring socket can be parsed + * properly. + */ + if (hdr->evt == 0xe4) + hdr->evt = HCI_EV_VENDOR; + + /* When someone waits for the WMT event, the skb is being cloned + * and being processed the events from there then. + */ + if (test_bit(BTMTKSDIO_TX_WAIT_VND_EVT, &bdev->tx_state)) { + bdev->evt_skb = skb_clone(skb, GFP_KERNEL); + if (!bdev->evt_skb) { + err = -ENOMEM; + goto err_out; + } + } + + err = hci_recv_frame(hdev, skb); + if (err < 0) + goto err_free_skb; + + if (hdr->evt == HCI_EV_VENDOR) { + if (test_and_clear_bit(BTMTKSDIO_TX_WAIT_VND_EVT, + &bdev->tx_state)) { + /* Barrier to sync with other CPUs */ + smp_mb__after_atomic(); + wake_up_bit(&bdev->tx_state, BTMTKSDIO_TX_WAIT_VND_EVT); + } + } + + return 0; + +err_free_skb: + kfree_skb(bdev->evt_skb); + bdev->evt_skb = NULL; + +err_out: + return err; +} + +static const struct h4_recv_pkt mtk_recv_pkts[] = { + { H4_RECV_ACL, .recv = hci_recv_frame }, + { H4_RECV_SCO, .recv = hci_recv_frame }, + { H4_RECV_EVENT, .recv = btmtksdio_recv_event }, +}; + +static int btmtksdio_rx_packet(struct btmtksdio_dev *bdev, u16 rx_size) +{ + const struct h4_recv_pkt *pkts = mtk_recv_pkts; + int pkts_count = ARRAY_SIZE(mtk_recv_pkts); + struct mtkbtsdio_hdr *sdio_hdr; + int err, i, pad_size; + struct sk_buff *skb; + u16 dlen; + + if (rx_size < sizeof(*sdio_hdr)) + return -EILSEQ; + + /* A SDIO packet is exactly containing a Bluetooth packet */ + skb = bt_skb_alloc(rx_size, GFP_KERNEL); + if (!skb) + return -ENOMEM; + + skb_put(skb, rx_size); + + err = sdio_readsb(bdev->func, skb->data, MTK_REG_CRDR, rx_size); + if (err < 0) + goto err_kfree_skb; + + sdio_hdr = (void *)skb->data; + + /* We assume the default error as -EILSEQ simply to make the error path + * be cleaner. + */ + err = -EILSEQ; + + if (rx_size != le16_to_cpu(sdio_hdr->len)) { + bt_dev_err(bdev->hdev, "Rx size in sdio header is mismatched "); + goto err_kfree_skb; + } + + hci_skb_pkt_type(skb) = sdio_hdr->bt_type; + + /* Remove MediaTek SDIO header */ + skb_pull(skb, sizeof(*sdio_hdr)); + + /* We have to dig into the packet to get payload size and then know how + * many padding bytes at the tail, these padding bytes should be removed + * before the packet is indicated to the core layer. + */ + for (i = 0; i < pkts_count; i++) { + if (sdio_hdr->bt_type == (&pkts[i])->type) + break; + } + + if (i >= pkts_count) { + bt_dev_err(bdev->hdev, "Invalid bt type 0x%02x", + sdio_hdr->bt_type); + goto err_kfree_skb; + } + + /* Remaining bytes cannot hold a header*/ + if (skb->len < (&pkts[i])->hlen) { + bt_dev_err(bdev->hdev, "The size of bt header is mismatched"); + goto err_kfree_skb; + } + + switch ((&pkts[i])->lsize) { + case 1: + dlen = skb->data[(&pkts[i])->loff]; + break; + case 2: + dlen = get_unaligned_le16(skb->data + + (&pkts[i])->loff); + break; + default: + goto err_kfree_skb; + } + + pad_size = skb->len - (&pkts[i])->hlen - dlen; + + /* Remaining bytes cannot hold a payload */ + if (pad_size < 0) { + bt_dev_err(bdev->hdev, "The size of bt payload is mismatched"); + goto err_kfree_skb; + } + + /* Remove padding bytes */ + skb_trim(skb, skb->len - pad_size); + + /* Complete frame */ + (&pkts[i])->recv(bdev->hdev, skb); + + bdev->hdev->stat.byte_rx += rx_size; + + return 0; + +err_kfree_skb: + kfree_skb(skb); + + return err; +} + +static void btmtksdio_interrupt(struct sdio_func *func) +{ + struct btmtksdio_dev *bdev = sdio_get_drvdata(func); + u32 int_status; + u16 rx_size; + + /* It is required that the host gets ownership from the device before + * accessing any register, however, if SDIO host is not being released, + * a potential deadlock probably happens in a circular wait between SDIO + * IRQ work and PM runtime work. So, we have to explicitly release SDIO + * host here and claim again after the PM runtime work is all done. + */ + sdio_release_host(bdev->func); + + pm_runtime_get_sync(bdev->dev); + + sdio_claim_host(bdev->func); + + /* Disable interrupt */ + sdio_writel(func, C_INT_EN_CLR, MTK_REG_CHLPCR, 0); + + int_status = sdio_readl(func, MTK_REG_CHISR, NULL); + + /* Ack an interrupt as soon as possible before any operation on + * hardware. + * + * Note that we don't ack any status during operations to avoid race + * condition between the host and the device such as it's possible to + * mistakenly ack RX_DONE for the next packet and then cause interrupts + * not be raised again but there is still pending data in the hardware + * FIFO. + */ + sdio_writel(func, int_status, MTK_REG_CHISR, NULL); + + if (unlikely(!int_status)) + bt_dev_err(bdev->hdev, "CHISR is 0"); + + if (int_status & FW_OWN_BACK_INT) + bt_dev_dbg(bdev->hdev, "Get fw own back"); + + if (int_status & TX_EMPTY) + schedule_work(&bdev->tx_work); + else if (unlikely(int_status & TX_FIFO_OVERFLOW)) + bt_dev_warn(bdev->hdev, "Tx fifo overflow"); + + if (int_status & RX_DONE_INT) { + rx_size = (int_status & RX_PKT_LEN) >> 16; + + if (btmtksdio_rx_packet(bdev, rx_size) < 0) + bdev->hdev->stat.err_rx++; + } + + /* Enable interrupt */ + sdio_writel(func, C_INT_EN_SET, MTK_REG_CHLPCR, 0); + + pm_runtime_mark_last_busy(bdev->dev); + pm_runtime_put_autosuspend(bdev->dev); +} + +static int btmtksdio_open(struct hci_dev *hdev) +{ + struct btmtksdio_dev *bdev = hci_get_drvdata(hdev); + int err; + u32 status; + + sdio_claim_host(bdev->func); + + err = sdio_enable_func(bdev->func); + if (err < 0) + goto err_release_host; + + /* Get ownership from the device */ + sdio_writel(bdev->func, C_FW_OWN_REQ_CLR, MTK_REG_CHLPCR, &err); + if (err < 0) + goto err_disable_func; + + err = readx_poll_timeout(btmtksdio_drv_own_query, bdev, status, + status & C_COM_DRV_OWN, 2000, 1000000); + if (err < 0) { + bt_dev_err(bdev->hdev, "Cannot get ownership from device"); + goto err_disable_func; + } + + /* Disable interrupt & mask out all interrupt sources */ + sdio_writel(bdev->func, C_INT_EN_CLR, MTK_REG_CHLPCR, &err); + if (err < 0) + goto err_disable_func; + + sdio_writel(bdev->func, 0, MTK_REG_CHIER, &err); + if (err < 0) + goto err_disable_func; + + err = sdio_claim_irq(bdev->func, btmtksdio_interrupt); + if (err < 0) + goto err_disable_func; + + err = sdio_set_block_size(bdev->func, MTK_SDIO_BLOCK_SIZE); + if (err < 0) + goto err_release_irq; + + /* SDIO CMD 5 allows the SDIO device back to idle state an + * synchronous interrupt is supported in SDIO 4-bit mode + */ + sdio_writel(bdev->func, SDIO_INT_CTL | SDIO_RE_INIT_EN, + MTK_REG_CSDIOCSR, &err); + if (err < 0) + goto err_release_irq; + + /* Setup write-1-clear for CHISR register */ + sdio_writel(bdev->func, C_INT_CLR_CTRL, MTK_REG_CHCR, &err); + if (err < 0) + goto err_release_irq; + + /* Setup interrupt sources */ + sdio_writel(bdev->func, RX_DONE_INT | TX_EMPTY | TX_FIFO_OVERFLOW, + MTK_REG_CHIER, &err); + if (err < 0) + goto err_release_irq; + + /* Enable interrupt */ + sdio_writel(bdev->func, C_INT_EN_SET, MTK_REG_CHLPCR, &err); + if (err < 0) + goto err_release_irq; + + sdio_release_host(bdev->func); + + return 0; + +err_release_irq: + sdio_release_irq(bdev->func); + +err_disable_func: + sdio_disable_func(bdev->func); + +err_release_host: + sdio_release_host(bdev->func); + + return err; +} + +static int btmtksdio_close(struct hci_dev *hdev) +{ + struct btmtksdio_dev *bdev = hci_get_drvdata(hdev); + u32 status; + int err; + + sdio_claim_host(bdev->func); + + /* Disable interrupt */ + sdio_writel(bdev->func, C_INT_EN_CLR, MTK_REG_CHLPCR, NULL); + + sdio_release_irq(bdev->func); + + /* Return ownership to the device */ + sdio_writel(bdev->func, C_FW_OWN_REQ_SET, MTK_REG_CHLPCR, NULL); + + err = readx_poll_timeout(btmtksdio_drv_own_query, bdev, status, + !(status & C_COM_DRV_OWN), 2000, 1000000); + if (err < 0) + bt_dev_err(bdev->hdev, "Cannot return ownership to device"); + + sdio_disable_func(bdev->func); + + sdio_release_host(bdev->func); + + return 0; +} + +static int btmtksdio_flush(struct hci_dev *hdev) +{ + struct btmtksdio_dev *bdev = hci_get_drvdata(hdev); + + skb_queue_purge(&bdev->txq); + + cancel_work_sync(&bdev->tx_work); + + return 0; +} + +static int btmtksdio_func_query(struct hci_dev *hdev) +{ + struct btmtk_hci_wmt_params wmt_params; + int status, err; + u8 param = 0; + + /* Query whether the function is enabled */ + wmt_params.op = MTK_WMT_FUNC_CTRL; + wmt_params.flag = 4; + wmt_params.dlen = sizeof(param); + wmt_params.data = ¶m; + wmt_params.status = &status; + + err = mtk_hci_wmt_sync(hdev, &wmt_params); + if (err < 0) { + bt_dev_err(hdev, "Failed to query function status (%d)", err); + return err; + } + + return status; +} + +static int mtk_setup_firmware(struct hci_dev *hdev, const char *fwname) +{ + struct btmtk_hci_wmt_params wmt_params; + const struct firmware *fw; + const u8 *fw_ptr; + size_t fw_size; + int err, dlen; + u8 flag; + + err = request_firmware(&fw, fwname, &hdev->dev); + if (err < 0) { + bt_dev_err(hdev, "Failed to load firmware file (%d)", err); + return err; + } + + fw_ptr = fw->data; + fw_size = fw->size; + + /* The size of patch header is 30 bytes, should be skip */ + if (fw_size < 30) { + err = -EINVAL; + goto free_fw; + } + + fw_size -= 30; + fw_ptr += 30; + flag = 1; + + wmt_params.op = MTK_WMT_PATCH_DWNLD; + wmt_params.status = NULL; + + while (fw_size > 0) { + dlen = min_t(int, 250, fw_size); + + /* Tell device the position in sequence */ + if (fw_size - dlen <= 0) + flag = 3; + else if (fw_size < fw->size - 30) + flag = 2; + + wmt_params.flag = flag; + wmt_params.dlen = dlen; + wmt_params.data = fw_ptr; + + err = mtk_hci_wmt_sync(hdev, &wmt_params); + if (err < 0) { + bt_dev_err(hdev, "Failed to send wmt patch dwnld (%d)", + err); + goto free_fw; + } + + fw_size -= dlen; + fw_ptr += dlen; + } + + wmt_params.op = MTK_WMT_RST; + wmt_params.flag = 4; + wmt_params.dlen = 0; + wmt_params.data = NULL; + wmt_params.status = NULL; + + /* Activate funciton the firmware providing to */ + err = mtk_hci_wmt_sync(hdev, &wmt_params); + if (err < 0) { + bt_dev_err(hdev, "Failed to send wmt rst (%d)", err); + goto free_fw; + } + + /* Wait a few moments for firmware activation done */ + usleep_range(10000, 12000); + +free_fw: + release_firmware(fw); + return err; +} + +static int btmtksdio_setup(struct hci_dev *hdev) +{ + struct btmtksdio_dev *bdev = hci_get_drvdata(hdev); + struct btmtk_hci_wmt_params wmt_params; + ktime_t calltime, delta, rettime; + struct btmtk_tci_sleep tci_sleep; + unsigned long long duration; + struct sk_buff *skb; + int err, status; + u8 param = 0x1; + + calltime = ktime_get(); + + /* Query whether the firmware is already download */ + wmt_params.op = MTK_WMT_SEMAPHORE; + wmt_params.flag = 1; + wmt_params.dlen = 0; + wmt_params.data = NULL; + wmt_params.status = &status; + + err = mtk_hci_wmt_sync(hdev, &wmt_params); + if (err < 0) { + bt_dev_err(hdev, "Failed to query firmware status (%d)", err); + return err; + } + + if (status == BTMTK_WMT_PATCH_DONE) { + bt_dev_info(hdev, "Firmware already downloaded"); + goto ignore_setup_fw; + } + + /* Setup a firmware which the device definitely requires */ + err = mtk_setup_firmware(hdev, bdev->data->fwname); + if (err < 0) + return err; + +ignore_setup_fw: + /* Query whether the device is already enabled */ + err = readx_poll_timeout(btmtksdio_func_query, hdev, status, + status < 0 || status != BTMTK_WMT_ON_PROGRESS, + 2000, 5000000); + /* -ETIMEDOUT happens */ + if (err < 0) + return err; + + /* The other errors happen in btusb_mtk_func_query */ + if (status < 0) + return status; + + if (status == BTMTK_WMT_ON_DONE) { + bt_dev_info(hdev, "function already on"); + goto ignore_func_on; + } + + /* Enable Bluetooth protocol */ + wmt_params.op = MTK_WMT_FUNC_CTRL; + wmt_params.flag = 0; + wmt_params.dlen = sizeof(param); + wmt_params.data = ¶m; + wmt_params.status = NULL; + + err = mtk_hci_wmt_sync(hdev, &wmt_params); + if (err < 0) { + bt_dev_err(hdev, "Failed to send wmt func ctrl (%d)", err); + return err; + } + +ignore_func_on: + /* Apply the low power environment setup */ + tci_sleep.mode = 0x5; + tci_sleep.duration = cpu_to_le16(0x640); + tci_sleep.host_duration = cpu_to_le16(0x640); + tci_sleep.host_wakeup_pin = 0; + tci_sleep.time_compensation = 0; + + skb = __hci_cmd_sync(hdev, 0xfc7a, sizeof(tci_sleep), &tci_sleep, + HCI_INIT_TIMEOUT); + if (IS_ERR(skb)) { + err = PTR_ERR(skb); + bt_dev_err(hdev, "Failed to apply low power setting (%d)", err); + return err; + } + kfree_skb(skb); + + rettime = ktime_get(); + delta = ktime_sub(rettime, calltime); + duration = (unsigned long long)ktime_to_ns(delta) >> 10; + + pm_runtime_set_autosuspend_delay(bdev->dev, + MTKBTSDIO_AUTOSUSPEND_DELAY); + pm_runtime_use_autosuspend(bdev->dev); + + err = pm_runtime_set_active(bdev->dev); + if (err < 0) + return err; + + /* Default forbid runtime auto suspend, that can be allowed by + * enable_autosuspend flag or the PM runtime entry under sysfs. + */ + pm_runtime_forbid(bdev->dev); + pm_runtime_enable(bdev->dev); + + if (enable_autosuspend) + pm_runtime_allow(bdev->dev); + + bt_dev_info(hdev, "Device setup in %llu usecs", duration); + + return 0; +} + +static int btmtksdio_shutdown(struct hci_dev *hdev) +{ + struct btmtksdio_dev *bdev = hci_get_drvdata(hdev); + struct btmtk_hci_wmt_params wmt_params; + u8 param = 0x0; + int err; + + /* Get back the state to be consistent with the state + * in btmtksdio_setup. + */ + pm_runtime_get_sync(bdev->dev); + + /* Disable the device */ + wmt_params.op = MTK_WMT_FUNC_CTRL; + wmt_params.flag = 0; + wmt_params.dlen = sizeof(param); + wmt_params.data = ¶m; + wmt_params.status = NULL; + + err = mtk_hci_wmt_sync(hdev, &wmt_params); + if (err < 0) { + bt_dev_err(hdev, "Failed to send wmt func ctrl (%d)", err); + return err; + } + + pm_runtime_put_noidle(bdev->dev); + pm_runtime_disable(bdev->dev); + + return 0; +} + +static int btmtksdio_send_frame(struct hci_dev *hdev, struct sk_buff *skb) +{ + struct btmtksdio_dev *bdev = hci_get_drvdata(hdev); + + switch (hci_skb_pkt_type(skb)) { + case HCI_COMMAND_PKT: + hdev->stat.cmd_tx++; + break; + + case HCI_ACLDATA_PKT: + hdev->stat.acl_tx++; + break; + + case HCI_SCODATA_PKT: + hdev->stat.sco_tx++; + break; + + default: + return -EILSEQ; + } + + skb_queue_tail(&bdev->txq, skb); + + schedule_work(&bdev->tx_work); + + return 0; +} + +static int btmtksdio_probe(struct sdio_func *func, + const struct sdio_device_id *id) +{ + struct btmtksdio_dev *bdev; + struct hci_dev *hdev; + int err; + + bdev = devm_kzalloc(&func->dev, sizeof(*bdev), GFP_KERNEL); + if (!bdev) + return -ENOMEM; + + bdev->data = (void *)id->driver_data; + if (!bdev->data) + return -ENODEV; + + bdev->dev = &func->dev; + bdev->func = func; + + INIT_WORK(&bdev->tx_work, btmtksdio_tx_work); + skb_queue_head_init(&bdev->txq); + + /* Initialize and register HCI device */ + hdev = hci_alloc_dev(); + if (!hdev) { + dev_err(&func->dev, "Can't allocate HCI device\n"); + return -ENOMEM; + } + + bdev->hdev = hdev; + + hdev->bus = HCI_SDIO; + hci_set_drvdata(hdev, bdev); + + hdev->open = btmtksdio_open; + hdev->close = btmtksdio_close; + hdev->flush = btmtksdio_flush; + hdev->setup = btmtksdio_setup; + hdev->shutdown = btmtksdio_shutdown; + hdev->send = btmtksdio_send_frame; + SET_HCIDEV_DEV(hdev, &func->dev); + + hdev->manufacturer = 70; + set_bit(HCI_QUIRK_NON_PERSISTENT_SETUP, &hdev->quirks); + + err = hci_register_dev(hdev); + if (err < 0) { + dev_err(&func->dev, "Can't register HCI device\n"); + hci_free_dev(hdev); + return err; + } + + sdio_set_drvdata(func, bdev); + + /* pm_runtime_enable would be done after the firmware is being + * downloaded because the core layer probably already enables + * runtime PM for this func such as the case host->caps & + * MMC_CAP_POWER_OFF_CARD. + */ + if (pm_runtime_enabled(bdev->dev)) + pm_runtime_disable(bdev->dev); + + /* As explaination in drivers/mmc/core/sdio_bus.c tells us: + * Unbound SDIO functions are always suspended. + * During probe, the function is set active and the usage count + * is incremented. If the driver supports runtime PM, + * it should call pm_runtime_put_noidle() in its probe routine and + * pm_runtime_get_noresume() in its remove routine. + * + * So, put a pm_runtime_put_noidle here ! + */ + pm_runtime_put_noidle(bdev->dev); + + return 0; +} + +static void btmtksdio_remove(struct sdio_func *func) +{ + struct btmtksdio_dev *bdev = sdio_get_drvdata(func); + struct hci_dev *hdev; + + if (!bdev) + return; + + /* Be consistent the state in btmtksdio_probe */ + pm_runtime_get_noresume(bdev->dev); + + hdev = bdev->hdev; + + sdio_set_drvdata(func, NULL); + hci_unregister_dev(hdev); + hci_free_dev(hdev); +} + +#ifdef CONFIG_PM +static int btmtksdio_runtime_suspend(struct device *dev) +{ + struct sdio_func *func = dev_to_sdio_func(dev); + struct btmtksdio_dev *bdev; + u32 status; + int err; + + bdev = sdio_get_drvdata(func); + if (!bdev) + return 0; + + sdio_claim_host(bdev->func); + + sdio_writel(bdev->func, C_FW_OWN_REQ_SET, MTK_REG_CHLPCR, &err); + if (err < 0) + goto out; + + err = readx_poll_timeout(btmtksdio_drv_own_query, bdev, status, + !(status & C_COM_DRV_OWN), 2000, 1000000); +out: + bt_dev_info(bdev->hdev, "status (%d) return ownership to device", err); + + sdio_release_host(bdev->func); + + return err; +} + +static int btmtksdio_runtime_resume(struct device *dev) +{ + struct sdio_func *func = dev_to_sdio_func(dev); + struct btmtksdio_dev *bdev; + u32 status; + int err; + + bdev = sdio_get_drvdata(func); + if (!bdev) + return 0; + + sdio_claim_host(bdev->func); + + sdio_writel(bdev->func, C_FW_OWN_REQ_CLR, MTK_REG_CHLPCR, &err); + if (err < 0) + goto out; + + err = readx_poll_timeout(btmtksdio_drv_own_query, bdev, status, + status & C_COM_DRV_OWN, 2000, 1000000); +out: + bt_dev_info(bdev->hdev, "status (%d) get ownership from device", err); + + sdio_release_host(bdev->func); + + return err; +} + +static UNIVERSAL_DEV_PM_OPS(btmtksdio_pm_ops, btmtksdio_runtime_suspend, + btmtksdio_runtime_resume, NULL); +#define BTMTKSDIO_PM_OPS (&btmtksdio_pm_ops) +#else /* CONFIG_PM */ +#define BTMTKSDIO_PM_OPS NULL +#endif /* CONFIG_PM */ + +static struct sdio_driver btmtksdio_driver = { + .name = "btmtksdio", + .probe = btmtksdio_probe, + .remove = btmtksdio_remove, + .id_table = btmtksdio_table, + .drv = { + .owner = THIS_MODULE, + .pm = BTMTKSDIO_PM_OPS, + } +}; + +module_sdio_driver(btmtksdio_driver); + +module_param(enable_autosuspend, bool, 0644); +MODULE_PARM_DESC(enable_autosuspend, "Enable autosuspend by default"); + +MODULE_AUTHOR("Sean Wang "); +MODULE_DESCRIPTION("MediaTek Bluetooth SDIO driver ver " VERSION); +MODULE_VERSION(VERSION); +MODULE_LICENSE("GPL"); +MODULE_FIRMWARE(FIRMWARE_MT7663); +MODULE_FIRMWARE(FIRMWARE_MT7668); diff --git a/drivers/bluetooth/btmtkuart.c b/drivers/bluetooth/btmtkuart.c index b0b680dd69f49a9114ae7c91858703c912933489..f5dbeec8e2748b6a812b0ca41378de6d620b2e81 100644 --- a/drivers/bluetooth/btmtkuart.c +++ b/drivers/bluetooth/btmtkuart.c @@ -661,7 +661,7 @@ static int btmtkuart_change_baudrate(struct hci_dev *hdev) { struct btmtkuart_dev *bdev = hci_get_drvdata(hdev); struct btmtk_hci_wmt_params wmt_params; - u32 baudrate; + __le32 baudrate; u8 param; int err; diff --git a/drivers/bluetooth/btqca.c b/drivers/bluetooth/btqca.c index 612268574fc7841c68c7730303ebcff73302fca3..cc12eecd9e4dd20623115ccf1aacaaa56b763be7 100644 --- a/drivers/bluetooth/btqca.c +++ b/drivers/bluetooth/btqca.c @@ -336,7 +336,7 @@ int qca_uart_setup(struct hci_dev *hdev, uint8_t baudrate, { struct rome_config config; int err; - u8 rom_ver; + u8 rom_ver = 0; bt_dev_dbg(hdev, "QCA setup on UART"); @@ -344,7 +344,7 @@ int qca_uart_setup(struct hci_dev *hdev, uint8_t baudrate, /* Download rampatch file */ config.type = TLV_TYPE_PATCH; - if (soc_type == QCA_WCN3990) { + if (qca_is_wcn399x(soc_type)) { /* Firmware files to download are based on ROM version. * ROM version is derived from last two bytes of soc_ver. */ @@ -365,7 +365,7 @@ int qca_uart_setup(struct hci_dev *hdev, uint8_t baudrate, /* Download NVM configuration */ config.type = TLV_TYPE_NVM; - if (soc_type == QCA_WCN3990) + if (qca_is_wcn399x(soc_type)) snprintf(config.fwname, sizeof(config.fwname), "qca/crnv%02x.bin", rom_ver); else @@ -410,6 +410,7 @@ int qca_set_bdaddr(struct hci_dev *hdev, const bdaddr_t *bdaddr) } EXPORT_SYMBOL_GPL(qca_set_bdaddr); + MODULE_AUTHOR("Ben Young Tae Kim "); MODULE_DESCRIPTION("Bluetooth support for Qualcomm Atheros family ver " VERSION); MODULE_VERSION(VERSION); diff --git a/drivers/bluetooth/btqca.h b/drivers/bluetooth/btqca.h index c72c56ea74808b1e850e9e15759925bae72c0760..4c4fe2b5b7b74db7cd457fbbe6410d825875cc76 100644 --- a/drivers/bluetooth/btqca.h +++ b/drivers/bluetooth/btqca.h @@ -41,7 +41,7 @@ #define QCA_WCN3990_POWERON_PULSE 0xFC #define QCA_WCN3990_POWEROFF_PULSE 0xC0 -enum qca_bardrate { +enum qca_baudrate { QCA_BAUDRATE_115200 = 0, QCA_BAUDRATE_57600, QCA_BAUDRATE_38400, @@ -132,7 +132,8 @@ enum qca_btsoc_type { QCA_INVALID = -1, QCA_AR3002, QCA_ROME, - QCA_WCN3990 + QCA_WCN3990, + QCA_WCN3998, }; #if IS_ENABLED(CONFIG_BT_QCA) @@ -142,6 +143,10 @@ int qca_uart_setup(struct hci_dev *hdev, uint8_t baudrate, enum qca_btsoc_type soc_type, u32 soc_ver); int qca_read_soc_version(struct hci_dev *hdev, u32 *soc_version); int qca_set_bdaddr(struct hci_dev *hdev, const bdaddr_t *bdaddr); +static inline bool qca_is_wcn399x(enum qca_btsoc_type soc_type) +{ + return soc_type == QCA_WCN3990 || soc_type == QCA_WCN3998; +} #else static inline int qca_set_bdaddr_rome(struct hci_dev *hdev, const bdaddr_t *bdaddr) @@ -165,4 +170,8 @@ static inline int qca_set_bdaddr(struct hci_dev *hdev, const bdaddr_t *bdaddr) return -EOPNOTSUPP; } +static inline bool qca_is_wcn399x(enum qca_btsoc_type soc_type) +{ + return false; +} #endif diff --git a/drivers/bluetooth/btsdio.c b/drivers/bluetooth/btsdio.c index 282d1af1d3bae7aa4b8c770764626836a0d6a6a2..4cfa9abe03c81ad1515ac94bbb0704914d60188f 100644 --- a/drivers/bluetooth/btsdio.c +++ b/drivers/bluetooth/btsdio.c @@ -376,20 +376,7 @@ static struct sdio_driver btsdio_driver = { .id_table = btsdio_table, }; -static int __init btsdio_init(void) -{ - BT_INFO("Generic Bluetooth SDIO driver ver %s", VERSION); - - return sdio_register_driver(&btsdio_driver); -} - -static void __exit btsdio_exit(void) -{ - sdio_unregister_driver(&btsdio_driver); -} - -module_init(btsdio_init); -module_exit(btsdio_exit); +module_sdio_driver(btsdio_driver); MODULE_AUTHOR("Marcel Holtmann "); MODULE_DESCRIPTION("Generic Bluetooth SDIO driver ver " VERSION); diff --git a/drivers/bluetooth/hci_bcm.c b/drivers/bluetooth/hci_bcm.c index ddbe518c3e5b85095235ce5a73a08ce94cf28d46..b5d31d583d607ecc2c2b7fa3b7cce3904f83fc11 100644 --- a/drivers/bluetooth/hci_bcm.c +++ b/drivers/bluetooth/hci_bcm.c @@ -228,9 +228,15 @@ static int bcm_gpio_set_power(struct bcm_device *dev, bool powered) int err; if (powered && !dev->res_enabled) { - err = regulator_bulk_enable(BCM_NUM_SUPPLIES, dev->supplies); - if (err) - return err; + /* Intel Macs use bcm_apple_get_resources() and don't + * have regulator supplies configured. + */ + if (dev->supplies[0].supply) { + err = regulator_bulk_enable(BCM_NUM_SUPPLIES, + dev->supplies); + if (err) + return err; + } /* LPO clock needs to be 32.768 kHz */ err = clk_set_rate(dev->lpo_clk, 32768); @@ -259,7 +265,13 @@ static int bcm_gpio_set_power(struct bcm_device *dev, bool powered) if (!powered && dev->res_enabled) { clk_disable_unprepare(dev->txco_clk); clk_disable_unprepare(dev->lpo_clk); - regulator_bulk_disable(BCM_NUM_SUPPLIES, dev->supplies); + + /* Intel Macs use bcm_apple_get_resources() and don't + * have regulator supplies configured. + */ + if (dev->supplies[0].supply) + regulator_bulk_disable(BCM_NUM_SUPPLIES, + dev->supplies); } /* wait for device to power on and come out of reset */ diff --git a/drivers/bluetooth/hci_h5.c b/drivers/bluetooth/hci_h5.c index 069d1c8fde73e70c04d5e1690c5bc79ba509f2e9..3f02ae5601202c73a62df5a62eb8d145d427548c 100644 --- a/drivers/bluetooth/hci_h5.c +++ b/drivers/bluetooth/hci_h5.c @@ -536,7 +536,7 @@ static void h5_unslip_one_byte(struct h5 *h5, unsigned char c) skb_put_data(h5->rx_skb, byte, 1); h5->rx_pending--; - BT_DBG("unsliped 0x%02hhx, rx_pending %zu", *byte, h5->rx_pending); + BT_DBG("unslipped 0x%02hhx, rx_pending %zu", *byte, h5->rx_pending); } static void h5_reset_rx(struct h5 *h5) diff --git a/drivers/bluetooth/hci_qca.c b/drivers/bluetooth/hci_qca.c index 237aea34b69f1be277cb71b98305f7c6d3100c7b..57322c42bb2dd13f5b8482d2c6b4691f82b2c13e 100644 --- a/drivers/bluetooth/hci_qca.c +++ b/drivers/bluetooth/hci_qca.c @@ -54,9 +54,6 @@ #define HCI_IBS_WAKE_ACK 0xFC #define HCI_MAX_IBS_SIZE 10 -/* Controller states */ -#define STATE_IN_BAND_SLEEP_ENABLED 1 - #define IBS_WAKE_RETRANS_TIMEOUT_MS 100 #define IBS_TX_IDLE_TIMEOUT_MS 2000 #define CMD_TRANS_TIMEOUT_MS 100 @@ -67,6 +64,10 @@ /* Controller debug log header */ #define QCA_DEBUG_HANDLE 0x2EDC +enum qca_flags { + QCA_IBS_ENABLED, +}; + /* HCI_IBS transmit side sleep protocol states */ enum tx_ibs_states { HCI_IBS_TX_ASLEEP, @@ -174,6 +175,21 @@ static int qca_power_setup(struct hci_uart *hu, bool on); static void qca_power_shutdown(struct hci_uart *hu); static int qca_power_off(struct hci_dev *hdev); +static enum qca_btsoc_type qca_soc_type(struct hci_uart *hu) +{ + enum qca_btsoc_type soc_type; + + if (hu->serdev) { + struct qca_serdev *qsd = serdev_device_get_drvdata(hu->serdev); + + soc_type = qsd->btsoc_type; + } else { + soc_type = QCA_ROME; + } + + return soc_type; +} + static void __serial_clock_on(struct tty_struct *tty) { /* TODO: Some chipset requires to enable UART clock on client @@ -506,8 +522,10 @@ static int qca_open(struct hci_uart *hu) if (hu->serdev) { qcadev = serdev_device_get_drvdata(hu->serdev); - if (qcadev->btsoc_type != QCA_WCN3990) { + if (!qca_is_wcn399x(qcadev->btsoc_type)) { gpiod_set_value_cansleep(qcadev->bt_en, 1); + /* Controller needs time to bootup. */ + msleep(150); } else { hu->init_speed = qcadev->init_speed; hu->oper_speed = qcadev->oper_speed; @@ -612,7 +630,7 @@ static int qca_close(struct hci_uart *hu) if (hu->serdev) { qcadev = serdev_device_get_drvdata(hu->serdev); - if (qcadev->btsoc_type == QCA_WCN3990) + if (qca_is_wcn399x(qcadev->btsoc_type)) qca_power_shutdown(hu); else gpiod_set_value_cansleep(qcadev->bt_en, 0); @@ -775,7 +793,7 @@ static int qca_enqueue(struct hci_uart *hu, struct sk_buff *skb) /* Don't go to sleep in middle of patch download or * Out-Of-Band(GPIOs control) sleep is selected. */ - if (!test_bit(STATE_IN_BAND_SLEEP_ENABLED, &qca->flags)) { + if (!test_bit(QCA_IBS_ENABLED, &qca->flags)) { skb_queue_tail(&qca->txq, skb); spin_unlock_irqrestore(&qca->hci_ibs_lock, flags); return 0; @@ -963,7 +981,6 @@ static int qca_set_baudrate(struct hci_dev *hdev, uint8_t baudrate) { struct hci_uart *hu = hci_get_drvdata(hdev); struct qca_data *qca = hu->priv; - struct qca_serdev *qcadev; struct sk_buff *skb; u8 cmd[] = { 0x01, 0x48, 0xFC, 0x01, 0x00 }; @@ -985,18 +1002,17 @@ static int qca_set_baudrate(struct hci_dev *hdev, uint8_t baudrate) skb_queue_tail(&qca->txq, skb); hci_uart_tx_wakeup(hu); - qcadev = serdev_device_get_drvdata(hu->serdev); - /* Wait for the baudrate change request to be sent */ while (!skb_queue_empty(&qca->txq)) usleep_range(100, 200); - serdev_device_wait_until_sent(hu->serdev, + if (hu->serdev) + serdev_device_wait_until_sent(hu->serdev, msecs_to_jiffies(CMD_TRANS_TIMEOUT_MS)); /* Give the controller time to process the request */ - if (qcadev->btsoc_type == QCA_WCN3990) + if (qca_is_wcn399x(qca_soc_type(hu))) msleep(10); else msleep(300); @@ -1072,10 +1088,7 @@ static unsigned int qca_get_speed(struct hci_uart *hu, static int qca_check_speeds(struct hci_uart *hu) { - struct qca_serdev *qcadev; - - qcadev = serdev_device_get_drvdata(hu->serdev); - if (qcadev->btsoc_type == QCA_WCN3990) { + if (qca_is_wcn399x(qca_soc_type(hu))) { if (!qca_get_speed(hu, QCA_INIT_SPEED) && !qca_get_speed(hu, QCA_OPER_SPEED)) return -EINVAL; @@ -1091,7 +1104,6 @@ static int qca_check_speeds(struct hci_uart *hu) static int qca_set_speed(struct hci_uart *hu, enum qca_speed_type speed_type) { unsigned int speed, qca_baudrate; - struct qca_serdev *qcadev; int ret = 0; if (speed_type == QCA_INIT_SPEED) { @@ -1099,6 +1111,8 @@ static int qca_set_speed(struct hci_uart *hu, enum qca_speed_type speed_type) if (speed) host_set_baudrate(hu, speed); } else { + enum qca_btsoc_type soc_type = qca_soc_type(hu); + speed = qca_get_speed(hu, QCA_OPER_SPEED); if (!speed) return 0; @@ -1106,8 +1120,7 @@ static int qca_set_speed(struct hci_uart *hu, enum qca_speed_type speed_type) /* Disable flow control for wcn3990 to deassert RTS while * changing the baudrate of chip and host. */ - qcadev = serdev_device_get_drvdata(hu->serdev); - if (qcadev->btsoc_type == QCA_WCN3990) + if (qca_is_wcn399x(soc_type)) hci_uart_set_flow_control(hu, true); qca_baudrate = qca_get_baudrate_value(speed); @@ -1119,7 +1132,7 @@ static int qca_set_speed(struct hci_uart *hu, enum qca_speed_type speed_type) host_set_baudrate(hu, speed); error: - if (qcadev->btsoc_type == QCA_WCN3990) + if (qca_is_wcn399x(soc_type)) hci_uart_set_flow_control(hu, false); } @@ -1181,20 +1194,18 @@ static int qca_setup(struct hci_uart *hu) struct hci_dev *hdev = hu->hdev; struct qca_data *qca = hu->priv; unsigned int speed, qca_baudrate = QCA_BAUDRATE_115200; - struct qca_serdev *qcadev; + enum qca_btsoc_type soc_type = qca_soc_type(hu); int ret; int soc_ver = 0; - qcadev = serdev_device_get_drvdata(hu->serdev); - ret = qca_check_speeds(hu); if (ret) return ret; /* Patch downloading has to be done without IBS mode */ - clear_bit(STATE_IN_BAND_SLEEP_ENABLED, &qca->flags); + clear_bit(QCA_IBS_ENABLED, &qca->flags); - if (qcadev->btsoc_type == QCA_WCN3990) { + if (qca_is_wcn399x(soc_type)) { bt_dev_info(hdev, "setting up wcn3990"); /* Enable NON_PERSISTENT_SETUP QUIRK to ensure to execute @@ -1225,7 +1236,7 @@ static int qca_setup(struct hci_uart *hu) qca_baudrate = qca_get_baudrate_value(speed); } - if (qcadev->btsoc_type != QCA_WCN3990) { + if (!qca_is_wcn399x(soc_type)) { /* Get QCA version information */ ret = qca_read_soc_version(hdev, &soc_ver); if (ret) @@ -1234,9 +1245,9 @@ static int qca_setup(struct hci_uart *hu) bt_dev_info(hdev, "QCA controller version 0x%08x", soc_ver); /* Setup patch / NVM configurations */ - ret = qca_uart_setup(hdev, qca_baudrate, qcadev->btsoc_type, soc_ver); + ret = qca_uart_setup(hdev, qca_baudrate, soc_type, soc_ver); if (!ret) { - set_bit(STATE_IN_BAND_SLEEP_ENABLED, &qca->flags); + set_bit(QCA_IBS_ENABLED, &qca->flags); qca_debugfs_init(hdev); } else if (ret == -ENOENT) { /* No patch/nvm-config found, run with original fw/config */ @@ -1250,7 +1261,7 @@ static int qca_setup(struct hci_uart *hu) } /* Setup bdaddr */ - if (qcadev->btsoc_type == QCA_WCN3990) + if (qca_is_wcn399x(soc_type)) hu->hdev->set_bdaddr = qca_set_bdaddr; else hu->hdev->set_bdaddr = qca_set_bdaddr_rome; @@ -1273,7 +1284,7 @@ static struct hci_uart_proto qca_proto = { .dequeue = qca_dequeue, }; -static const struct qca_vreg_data qca_soc_data = { +static const struct qca_vreg_data qca_soc_data_wcn3990 = { .soc_type = QCA_WCN3990, .vregs = (struct qca_vreg []) { { "vddio", 1800000, 1900000, 15000 }, @@ -1284,6 +1295,17 @@ static const struct qca_vreg_data qca_soc_data = { .num_vregs = 4, }; +static const struct qca_vreg_data qca_soc_data_wcn3998 = { + .soc_type = QCA_WCN3998, + .vregs = (struct qca_vreg []) { + { "vddio", 1800000, 1900000, 10000 }, + { "vddxo", 1800000, 1900000, 80000 }, + { "vddrf", 1300000, 1352000, 300000 }, + { "vddch0", 3300000, 3300000, 450000 }, + }, + .num_vregs = 4, +}; + static void qca_power_shutdown(struct hci_uart *hu) { struct qca_data *qca = hu->priv; @@ -1294,7 +1316,7 @@ static void qca_power_shutdown(struct hci_uart *hu) * data in skb's. */ spin_lock_irqsave(&qca->hci_ibs_lock, flags); - clear_bit(STATE_IN_BAND_SLEEP_ENABLED, &qca->flags); + clear_bit(QCA_IBS_ENABLED, &qca->flags); qca_flush(hu); spin_unlock_irqrestore(&qca->hci_ibs_lock, flags); @@ -1417,8 +1439,8 @@ static int qca_serdev_probe(struct serdev_device *serdev) qcadev->serdev_hu.serdev = serdev; data = of_device_get_match_data(&serdev->dev); serdev_device_set_drvdata(serdev, qcadev); - if (data && data->soc_type == QCA_WCN3990) { - qcadev->btsoc_type = QCA_WCN3990; + if (data && qca_is_wcn399x(data->soc_type)) { + qcadev->btsoc_type = data->soc_type; qcadev->bt_power = devm_kzalloc(&serdev->dev, sizeof(struct qca_power), GFP_KERNEL); @@ -1482,7 +1504,7 @@ static void qca_serdev_remove(struct serdev_device *serdev) { struct qca_serdev *qcadev = serdev_device_get_drvdata(serdev); - if (qcadev->btsoc_type == QCA_WCN3990) + if (qca_is_wcn399x(qcadev->btsoc_type)) qca_power_shutdown(&qcadev->serdev_hu); else clk_disable_unprepare(qcadev->susclk); @@ -1492,7 +1514,8 @@ static void qca_serdev_remove(struct serdev_device *serdev) static const struct of_device_id qca_bluetooth_of_match[] = { { .compatible = "qcom,qca6174-bt" }, - { .compatible = "qcom,wcn3990-bt", .data = &qca_soc_data}, + { .compatible = "qcom,wcn3990-bt", .data = &qca_soc_data_wcn3990}, + { .compatible = "qcom,wcn3998-bt", .data = &qca_soc_data_wcn3998}, { /* sentinel */ } }; MODULE_DEVICE_TABLE(of, qca_bluetooth_of_match); diff --git a/drivers/infiniband/core/addr.c b/drivers/infiniband/core/addr.c index 0dce94e3c495619e1d825e9f0ee300da8ea2e779..744b6ec0acb0b2bf4323d636f21a3494e2f8fc14 100644 --- a/drivers/infiniband/core/addr.c +++ b/drivers/infiniband/core/addr.c @@ -42,7 +42,7 @@ #include #include #include -#include +#include #include #include #include @@ -86,8 +86,8 @@ static inline bool ib_nl_is_good_ip_resp(const struct nlmsghdr *nlh) if (nlh->nlmsg_flags & RDMA_NL_LS_F_ERR) return false; - ret = nla_parse(tb, LS_NLA_TYPE_MAX - 1, nlmsg_data(nlh), - nlmsg_len(nlh), ib_nl_addr_policy, NULL); + ret = nla_parse_deprecated(tb, LS_NLA_TYPE_MAX - 1, nlmsg_data(nlh), + nlmsg_len(nlh), ib_nl_addr_policy, NULL); if (ret) return false; @@ -351,7 +351,7 @@ static bool has_gateway(const struct dst_entry *dst, sa_family_t family) if (family == AF_INET) { rt = container_of(dst, struct rtable, dst); - return rt->rt_uses_gateway; + return rt->rt_gw_family == AF_INET; } rt6 = container_of(dst, struct rt6_info, dst); diff --git a/drivers/infiniband/core/iwpm_util.c b/drivers/infiniband/core/iwpm_util.c index a5d2a20ee6971a0990db47ccd1d1f70537c94513..41929bb83739f8d31c53d7ddbda38643dfd3c8b9 100644 --- a/drivers/infiniband/core/iwpm_util.c +++ b/drivers/infiniband/core/iwpm_util.c @@ -506,14 +506,14 @@ int iwpm_parse_nlmsg(struct netlink_callback *cb, int policy_max, int ret; const char *err_str = ""; - ret = nlmsg_validate(cb->nlh, nlh_len, policy_max - 1, nlmsg_policy, - NULL); + ret = nlmsg_validate_deprecated(cb->nlh, nlh_len, policy_max - 1, + nlmsg_policy, NULL); if (ret) { err_str = "Invalid attribute"; goto parse_nlmsg_error; } - ret = nlmsg_parse(cb->nlh, nlh_len, nltb, policy_max - 1, - nlmsg_policy, NULL); + ret = nlmsg_parse_deprecated(cb->nlh, nlh_len, nltb, policy_max - 1, + nlmsg_policy, NULL); if (ret) { err_str = "Unable to parse the nlmsg"; goto parse_nlmsg_error; diff --git a/drivers/infiniband/core/nldev.c b/drivers/infiniband/core/nldev.c index 11ed58d3fce59165ec414dfb478e67b658a6e512..85324012bf078de90f0f483e1026c11fa95a1a4b 100644 --- a/drivers/infiniband/core/nldev.c +++ b/drivers/infiniband/core/nldev.c @@ -292,7 +292,8 @@ static int fill_res_info_entry(struct sk_buff *msg, { struct nlattr *entry_attr; - entry_attr = nla_nest_start(msg, RDMA_NLDEV_ATTR_RES_SUMMARY_ENTRY); + entry_attr = nla_nest_start_noflag(msg, + RDMA_NLDEV_ATTR_RES_SUMMARY_ENTRY); if (!entry_attr) return -EMSGSIZE; @@ -327,7 +328,7 @@ static int fill_res_info(struct sk_buff *msg, struct ib_device *device) if (fill_nldev_handle(msg, device)) return -EMSGSIZE; - table_attr = nla_nest_start(msg, RDMA_NLDEV_ATTR_RES_SUMMARY); + table_attr = nla_nest_start_noflag(msg, RDMA_NLDEV_ATTR_RES_SUMMARY); if (!table_attr) return -EMSGSIZE; @@ -607,8 +608,8 @@ static int nldev_get_doit(struct sk_buff *skb, struct nlmsghdr *nlh, u32 index; int err; - err = nlmsg_parse(nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1, - nldev_policy, extack); + err = nlmsg_parse_deprecated(nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1, + nldev_policy, extack); if (err || !tb[RDMA_NLDEV_ATTR_DEV_INDEX]) return -EINVAL; @@ -652,8 +653,8 @@ static int nldev_set_doit(struct sk_buff *skb, struct nlmsghdr *nlh, u32 index; int err; - err = nlmsg_parse(nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1, nldev_policy, - extack); + err = nlmsg_parse_deprecated(nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1, + nldev_policy, extack); if (err || !tb[RDMA_NLDEV_ATTR_DEV_INDEX]) return -EINVAL; @@ -721,8 +722,8 @@ static int nldev_port_get_doit(struct sk_buff *skb, struct nlmsghdr *nlh, u32 port; int err; - err = nlmsg_parse(nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1, - nldev_policy, extack); + err = nlmsg_parse_deprecated(nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1, + nldev_policy, extack); if (err || !tb[RDMA_NLDEV_ATTR_DEV_INDEX] || !tb[RDMA_NLDEV_ATTR_PORT_INDEX]) @@ -777,8 +778,8 @@ static int nldev_port_get_dumpit(struct sk_buff *skb, int err; unsigned int p; - err = nlmsg_parse(cb->nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1, - nldev_policy, NULL); + err = nlmsg_parse_deprecated(cb->nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1, + nldev_policy, NULL); if (err || !tb[RDMA_NLDEV_ATTR_DEV_INDEX]) return -EINVAL; @@ -832,8 +833,8 @@ static int nldev_res_get_doit(struct sk_buff *skb, struct nlmsghdr *nlh, u32 index; int ret; - ret = nlmsg_parse(nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1, - nldev_policy, extack); + ret = nlmsg_parse_deprecated(nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1, + nldev_policy, extack); if (ret || !tb[RDMA_NLDEV_ATTR_DEV_INDEX]) return -EINVAL; @@ -981,8 +982,8 @@ static int res_get_common_doit(struct sk_buff *skb, struct nlmsghdr *nlh, struct sk_buff *msg; int ret; - ret = nlmsg_parse(nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1, - nldev_policy, extack); + ret = nlmsg_parse_deprecated(nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1, + nldev_policy, extack); if (ret || !tb[RDMA_NLDEV_ATTR_DEV_INDEX] || !fe->id || !tb[fe->id]) return -EINVAL; @@ -1070,8 +1071,8 @@ static int res_get_common_dumpit(struct sk_buff *skb, u32 index, port = 0; bool filled = false; - err = nlmsg_parse(cb->nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1, - nldev_policy, NULL); + err = nlmsg_parse_deprecated(cb->nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1, + nldev_policy, NULL); /* * Right now, we are expecting the device index to get res information, * but it is possible to extend this code to return all devices in @@ -1108,7 +1109,7 @@ static int res_get_common_dumpit(struct sk_buff *skb, goto err; } - table_attr = nla_nest_start(skb, fe->nldev_attr); + table_attr = nla_nest_start_noflag(skb, fe->nldev_attr); if (!table_attr) { ret = -EMSGSIZE; goto err; @@ -1134,7 +1135,7 @@ static int res_get_common_dumpit(struct sk_buff *skb, filled = true; - entry_attr = nla_nest_start(skb, fe->entry); + entry_attr = nla_nest_start_noflag(skb, fe->entry); if (!entry_attr) { ret = -EMSGSIZE; rdma_restrack_put(res); @@ -1249,8 +1250,8 @@ static int nldev_newlink(struct sk_buff *skb, struct nlmsghdr *nlh, char type[IFNAMSIZ]; int err; - err = nlmsg_parse(nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1, - nldev_policy, extack); + err = nlmsg_parse_deprecated(nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1, + nldev_policy, extack); if (err || !tb[RDMA_NLDEV_ATTR_DEV_NAME] || !tb[RDMA_NLDEV_ATTR_LINK_TYPE] || !tb[RDMA_NLDEV_ATTR_NDEV_NAME]) return -EINVAL; @@ -1293,8 +1294,8 @@ static int nldev_dellink(struct sk_buff *skb, struct nlmsghdr *nlh, u32 index; int err; - err = nlmsg_parse(nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1, - nldev_policy, extack); + err = nlmsg_parse_deprecated(nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1, + nldev_policy, extack); if (err || !tb[RDMA_NLDEV_ATTR_DEV_INDEX]) return -EINVAL; diff --git a/drivers/infiniband/core/sa_query.c b/drivers/infiniband/core/sa_query.c index 7925e45ea88ae81aadfbc239278633073817804f..bb534959abf0a575c3e69da9f6389a92d9d1c2b1 100644 --- a/drivers/infiniband/core/sa_query.c +++ b/drivers/infiniband/core/sa_query.c @@ -1028,8 +1028,8 @@ int ib_nl_handle_set_timeout(struct sk_buff *skb, !(NETLINK_CB(skb).sk)) return -EPERM; - ret = nla_parse(tb, LS_NLA_TYPE_MAX - 1, nlmsg_data(nlh), - nlmsg_len(nlh), ib_nl_policy, NULL); + ret = nla_parse_deprecated(tb, LS_NLA_TYPE_MAX - 1, nlmsg_data(nlh), + nlmsg_len(nlh), ib_nl_policy, NULL); attr = (const struct nlattr *)tb[LS_NLA_TYPE_TIMEOUT]; if (ret || !attr) goto settimeout_out; @@ -1080,8 +1080,8 @@ static inline int ib_nl_is_good_resolve_resp(const struct nlmsghdr *nlh) if (nlh->nlmsg_flags & RDMA_NL_LS_F_ERR) return 0; - ret = nla_parse(tb, LS_NLA_TYPE_MAX - 1, nlmsg_data(nlh), - nlmsg_len(nlh), ib_nl_policy, NULL); + ret = nla_parse_deprecated(tb, LS_NLA_TYPE_MAX - 1, nlmsg_data(nlh), + nlmsg_len(nlh), ib_nl_policy, NULL); if (ret) return 0; diff --git a/drivers/infiniband/hw/cxgb4/restrack.c b/drivers/infiniband/hw/cxgb4/restrack.c index 9a7520ee41e0f125a53e340e739df74a6cd1757b..f82d46ed969d9212ba5f3a65b42ea3e856af74cd 100644 --- a/drivers/infiniband/hw/cxgb4/restrack.c +++ b/drivers/infiniband/hw/cxgb4/restrack.c @@ -149,7 +149,7 @@ static int fill_res_qp_entry(struct sk_buff *msg, if (qhp->ucontext) return 0; - table_attr = nla_nest_start(msg, RDMA_NLDEV_ATTR_DRIVER); + table_attr = nla_nest_start_noflag(msg, RDMA_NLDEV_ATTR_DRIVER); if (!table_attr) goto err; @@ -216,7 +216,7 @@ static int fill_res_ep_entry(struct sk_buff *msg, if (!uep) return 0; - table_attr = nla_nest_start(msg, RDMA_NLDEV_ATTR_DRIVER); + table_attr = nla_nest_start_noflag(msg, RDMA_NLDEV_ATTR_DRIVER); if (!table_attr) goto err_free_uep; @@ -387,7 +387,7 @@ static int fill_res_cq_entry(struct sk_buff *msg, if (ibcq->uobject) return 0; - table_attr = nla_nest_start(msg, RDMA_NLDEV_ATTR_DRIVER); + table_attr = nla_nest_start_noflag(msg, RDMA_NLDEV_ATTR_DRIVER); if (!table_attr) goto err; @@ -447,7 +447,7 @@ static int fill_res_mr_entry(struct sk_buff *msg, if (!stag) return 0; - table_attr = nla_nest_start(msg, RDMA_NLDEV_ATTR_DRIVER); + table_attr = nla_nest_start_noflag(msg, RDMA_NLDEV_ATTR_DRIVER); if (!table_attr) goto err; diff --git a/drivers/infiniband/hw/hfi1/vnic_main.c b/drivers/infiniband/hw/hfi1/vnic_main.c index a922db58be1441020c665ee16ef081f5f3a2bfaf..2b07032dbddaacb213742cee71c503c858e77da4 100644 --- a/drivers/infiniband/hw/hfi1/vnic_main.c +++ b/drivers/infiniband/hw/hfi1/vnic_main.c @@ -423,8 +423,7 @@ tx_finish: static u16 hfi1_vnic_select_queue(struct net_device *netdev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { struct hfi1_vnic_vport_info *vinfo = opa_vnic_dev_priv(netdev); struct opa_vnic_skb_mdata *mdata; diff --git a/drivers/infiniband/hw/mlx5/cmd.c b/drivers/infiniband/hw/mlx5/cmd.c index 6bcc63aaa50bba036ee3612a929751f32f47f1fa..be95ac5aeb308625048f35a761900f40929761ff 100644 --- a/drivers/infiniband/hw/mlx5/cmd.c +++ b/drivers/infiniband/hw/mlx5/cmd.c @@ -148,7 +148,7 @@ int mlx5_cmd_alloc_memic(struct mlx5_memic *memic, phys_addr_t *addr, return ret; } - *addr = pci_resource_start(dev->pdev, 0) + + *addr = dev->bar_addr + MLX5_GET64(alloc_memic_out, out, memic_start_addr); return 0; @@ -167,7 +167,7 @@ int mlx5_cmd_dealloc_memic(struct mlx5_memic *memic, u64 addr, u64 length) u64 start_page_idx; int err; - addr -= pci_resource_start(dev->pdev, 0); + addr -= dev->bar_addr; start_page_idx = (addr - hw_start_addr) >> PAGE_SHIFT; MLX5_SET(dealloc_memic_in, in, opcode, MLX5_CMD_OP_DEALLOC_MEMIC); diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c index d3dd290ae1b176d609c14937c4ffedc2fe992a20..1aaa2056d188c221431821303f1f8c5013e5570f 100644 --- a/drivers/infiniband/hw/mlx5/main.c +++ b/drivers/infiniband/hw/mlx5/main.c @@ -181,7 +181,7 @@ static int mlx5_netdev_event(struct notifier_block *this, ibdev->rep->vport); if (rep_ndev == ndev) roce->netdev = ndev; - } else if (ndev->dev.parent == &mdev->pdev->dev) { + } else if (ndev->dev.parent == mdev->device) { roce->netdev = ndev; } write_unlock(&roce->netdev_lock); @@ -2011,7 +2011,7 @@ static phys_addr_t uar_index2pfn(struct mlx5_ib_dev *dev, fw_uars_per_page = MLX5_CAP_GEN(dev->mdev, uar_4k) ? MLX5_UARS_IN_PAGE : 1; - return (pci_resource_start(dev->mdev->pdev, 0) >> PAGE_SHIFT) + uar_idx / fw_uars_per_page; + return (dev->mdev->bar_addr >> PAGE_SHIFT) + uar_idx / fw_uars_per_page; } static int get_command(unsigned long offset) @@ -2202,7 +2202,7 @@ static int dm_mmap(struct ib_ucontext *context, struct vm_area_struct *vma) page_idx + npages) return -EINVAL; - pfn = ((pci_resource_start(dev->mdev->pdev, 0) + + pfn = ((dev->mdev->bar_addr + MLX5_CAP64_DEV_MEM(dev->mdev, memic_bar_start_addr)) >> PAGE_SHIFT) + page_idx; @@ -2285,7 +2285,7 @@ struct ib_dm *mlx5_ib_alloc_dm(struct ib_device *ibdev, goto err_free; start_offset = memic_addr & ~PAGE_MASK; - page_idx = (memic_addr - pci_resource_start(memic->dev->pdev, 0) - + page_idx = (memic_addr - memic->dev->bar_addr - MLX5_CAP64_DEV_MEM(memic->dev, memic_bar_start_addr)) >> PAGE_SHIFT; @@ -2328,7 +2328,7 @@ int mlx5_ib_dealloc_dm(struct ib_dm *ibdm) if (ret) return ret; - page_idx = (dm->dev_addr - pci_resource_start(memic->dev->pdev, 0) - + page_idx = (dm->dev_addr - memic->dev->bar_addr - MLX5_CAP64_DEV_MEM(memic->dev, memic_bar_start_addr)) >> PAGE_SHIFT; bitmap_clear(to_mucontext(ibdm->uobject->context)->dm_pages, @@ -4356,9 +4356,13 @@ static void delay_drop_handler(struct work_struct *work) static void handle_general_event(struct mlx5_ib_dev *ibdev, struct mlx5_eqe *eqe, struct ib_event *ibev) { + u8 port = (eqe->data.port.port >> 4) & 0xf; + switch (eqe->sub_type) { case MLX5_GENERAL_SUBTYPE_DELAY_DROP_TIMEOUT: - schedule_work(&ibdev->delay_drop.delay_drop_work); + if (mlx5_ib_port_link_layer(&ibdev->ib_dev, port) == + IB_LINK_LAYER_ETHERNET) + schedule_work(&ibdev->delay_drop.delay_drop_work); break; default: /* do nothing */ return; @@ -5675,7 +5679,8 @@ static int mlx5_ib_init_multiport_master(struct mlx5_ib_dev *dev) } if (bound) { - dev_dbg(&mpi->mdev->pdev->dev, "removing port from unaffiliated list.\n"); + dev_dbg(mpi->mdev->device, + "removing port from unaffiliated list.\n"); mlx5_ib_dbg(dev, "port %d bound\n", i + 1); list_del(&mpi->list); break; @@ -5874,7 +5879,7 @@ int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev) dev->ib_dev.local_dma_lkey = 0 /* not supported for now */; dev->ib_dev.phys_port_cnt = dev->num_ports; dev->ib_dev.num_comp_vectors = mlx5_comp_vectors_count(mdev); - dev->ib_dev.dev.parent = &mdev->pdev->dev; + dev->ib_dev.dev.parent = mdev->device; mutex_init(&dev->cap_mask_mutex); INIT_LIST_HEAD(&dev->qp_list); @@ -6563,7 +6568,8 @@ static void *mlx5_ib_add_slave_port(struct mlx5_core_dev *mdev) if (!bound) { list_add_tail(&mpi->list, &mlx5_ib_unaffiliated_port_list); - dev_dbg(&mdev->pdev->dev, "no suitable IB device found to bind to, added to unaffiliated list.\n"); + dev_dbg(mdev->device, + "no suitable IB device found to bind to, added to unaffiliated list.\n"); } mutex_unlock(&mlx5_ib_multiport_mutex); diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index c85f002558843ff50402bf14ba14ede892beb911..ca921fd4049963b84ad4f0541505ade7714eabbc 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -1194,8 +1194,7 @@ static struct ib_mr *mlx5_ib_get_memic_mr(struct ib_pd *pd, u64 memic_addr, MLX5_SET64(mkc, mkc, len, length); MLX5_SET(mkc, mkc, pd, to_mpd(pd)->pdn); MLX5_SET(mkc, mkc, qpn, 0xffffff); - MLX5_SET64(mkc, mkc, start_addr, - memic_addr - pci_resource_start(dev->mdev->pdev, 0)); + MLX5_SET64(mkc, mkc, start_addr, memic_addr - dev->mdev->bar_addr); err = mlx5_core_create_mkey(mdev, &mr->mmkey, in, inlen); if (err) diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c index 8aafb2208899302ab71aaae3d099d844ef375b9d..581144e224e24442c645bb979a1bcd91fe15bd09 100644 --- a/drivers/infiniband/hw/mlx5/qp.c +++ b/drivers/infiniband/hw/mlx5/qp.c @@ -5122,7 +5122,7 @@ out: wmb(); /* currently we support only regular doorbells */ - mlx5_write64((__be32 *)ctrl, bf->bfreg->map + bf->offset, NULL); + mlx5_write64((__be32 *)ctrl, bf->bfreg->map + bf->offset); /* Make sure doorbells don't leak out of SQ spinlock * and reach the HCA out of order. */ diff --git a/drivers/infiniband/hw/nes/nes_cm.c b/drivers/infiniband/hw/nes/nes_cm.c index 032883180f6524ce69872fd72f2e2519ad9e5056..0010a3ed64f154b4220db3ee757cf1f44affb82c 100644 --- a/drivers/infiniband/hw/nes/nes_cm.c +++ b/drivers/infiniband/hw/nes/nes_cm.c @@ -1407,7 +1407,7 @@ static int nes_addr_resolve_neigh(struct nes_vnic *nesvnic, u32 dst_ip, int arpi if (neigh->nud_state & NUD_VALID) { nes_debug(NES_DBG_CM, "Neighbor MAC address for 0x%08X" " is %pM, Gateway is 0x%08X \n", dst_ip, - neigh->ha, ntohl(rt->rt_gateway)); + neigh->ha, ntohl(rt->rt_gw4)); if (arpindex >= 0) { if (ether_addr_equal(nesadapter->arp_table[arpindex].mac_addr, neigh->ha)) { diff --git a/drivers/infiniband/ulp/opa_vnic/opa_vnic_netdev.c b/drivers/infiniband/ulp/opa_vnic/opa_vnic_netdev.c index ae70cd18903ef745c4bf1be6ce176c1eb4f4e013..aeff68f582d34749d797ff9faffa4303e275feb2 100644 --- a/drivers/infiniband/ulp/opa_vnic/opa_vnic_netdev.c +++ b/drivers/infiniband/ulp/opa_vnic/opa_vnic_netdev.c @@ -95,8 +95,7 @@ static netdev_tx_t opa_netdev_start_xmit(struct sk_buff *skb, } static u16 opa_vnic_select_queue(struct net_device *netdev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { struct opa_vnic_adapter *adapter = opa_vnic_priv(netdev); struct opa_vnic_skb_mdata *mdata; @@ -106,8 +105,7 @@ static u16 opa_vnic_select_queue(struct net_device *netdev, struct sk_buff *skb, mdata = skb_push(skb, sizeof(*mdata)); mdata->entropy = opa_vnic_calc_entropy(skb); mdata->vl = opa_vnic_get_vl(adapter, skb); - rc = adapter->rn_ops->ndo_select_queue(netdev, skb, - sb_dev, fallback); + rc = adapter->rn_ops->ndo_select_queue(netdev, skb, sb_dev); skb_pull(skb, sizeof(*mdata)); return rc; } diff --git a/drivers/isdn/gigaset/bas-gigaset.c b/drivers/isdn/gigaset/bas-gigaset.c index ecdeb89645d00454381ef4020986fdf5e38f2eec..149b1aca52a23e4cc28897064861abec8ce3326c 100644 --- a/drivers/isdn/gigaset/bas-gigaset.c +++ b/drivers/isdn/gigaset/bas-gigaset.c @@ -958,6 +958,7 @@ static void write_iso_callback(struct urb *urb) */ static int starturbs(struct bc_state *bcs) { + struct usb_device *udev = bcs->cs->hw.bas->udev; struct bas_bc_state *ubc = bcs->hw.bas; struct urb *urb; int j, k; @@ -975,8 +976,8 @@ static int starturbs(struct bc_state *bcs) rc = -EFAULT; goto error; } - usb_fill_int_urb(urb, bcs->cs->hw.bas->udev, - usb_rcvisocpipe(urb->dev, 3 + 2 * bcs->channel), + usb_fill_int_urb(urb, udev, + usb_rcvisocpipe(udev, 3 + 2 * bcs->channel), ubc->isoinbuf + k * BAS_INBUFSIZE, BAS_INBUFSIZE, read_iso_callback, bcs, BAS_FRAMETIME); @@ -1006,8 +1007,8 @@ static int starturbs(struct bc_state *bcs) rc = -EFAULT; goto error; } - usb_fill_int_urb(urb, bcs->cs->hw.bas->udev, - usb_sndisocpipe(urb->dev, 4 + 2 * bcs->channel), + usb_fill_int_urb(urb, udev, + usb_sndisocpipe(udev, 4 + 2 * bcs->channel), ubc->isooutbuf->data, sizeof(ubc->isooutbuf->data), write_iso_callback, &ubc->isoouturbs[k], diff --git a/drivers/isdn/hardware/mISDN/hfcpci.c b/drivers/isdn/hardware/mISDN/hfcpci.c index 362aa5450a5ec6764f71fcf9c0860fa54a066464..29c22d74afe052eea838d8d580977eee1fae578e 100644 --- a/drivers/isdn/hardware/mISDN/hfcpci.c +++ b/drivers/isdn/hardware/mISDN/hfcpci.c @@ -2041,9 +2041,9 @@ setup_hw(struct hfc_pci *hc) } printk(KERN_INFO - "HFC-PCI: defined at mem %#lx fifo %#lx(%#lx) IRQ %d HZ %d\n", - (u_long) hc->hw.pci_io, (u_long) hc->hw.fifos, - (u_long) hc->hw.dmahandle, hc->irq, HZ); + "HFC-PCI: defined at mem %#lx fifo %p(%pad) IRQ %d HZ %d\n", + (u_long) hc->hw.pci_io, hc->hw.fifos, + &hc->hw.dmahandle, hc->irq, HZ); /* enable memory mapped ports, disable busmaster */ pci_write_config_word(hc->pdev, PCI_COMMAND, PCI_ENA_MEMIO); diff --git a/drivers/isdn/hisax/config.c b/drivers/isdn/hisax/config.c index b12e6cae26c28d64c927151a25444e185ce24242..de965115a1834cd1e3619b6d6b7d0f9a0f394697 100644 --- a/drivers/isdn/hisax/config.c +++ b/drivers/isdn/hisax/config.c @@ -1294,9 +1294,9 @@ void HiSax_reportcard(int cardnr, int sel) printk(KERN_DEBUG "HiSax: reportcard No %d\n", cardnr + 1); printk(KERN_DEBUG "HiSax: Type %s\n", CardType[cs->typ]); printk(KERN_DEBUG "HiSax: debuglevel %x\n", cs->debug); - printk(KERN_DEBUG "HiSax: HiSax_reportcard address 0x%lX\n", - (ulong) & HiSax_reportcard); - printk(KERN_DEBUG "HiSax: cs 0x%lX\n", (ulong) cs); + printk(KERN_DEBUG "HiSax: HiSax_reportcard address 0x%px\n", + HiSax_reportcard); + printk(KERN_DEBUG "HiSax: cs 0x%px\n", cs); printk(KERN_DEBUG "HiSax: HW_Flags %lx bc0 flg %lx bc1 flg %lx\n", cs->HW_Flags, cs->bcs[0].Flag, cs->bcs[1].Flag); printk(KERN_DEBUG "HiSax: bcs 0 mode %d ch%d\n", diff --git a/drivers/isdn/i4l/isdn_ppp.c b/drivers/isdn/i4l/isdn_ppp.c index a7b275ea5de1da57471dc47d9cb66510adbd5d12..7e0f419c14f87294452bd9492bb20e57040f13d0 100644 --- a/drivers/isdn/i4l/isdn_ppp.c +++ b/drivers/isdn/i4l/isdn_ppp.c @@ -1888,8 +1888,9 @@ static u32 isdn_ppp_mp_get_seq(int short_seq, return seq; } -struct sk_buff *isdn_ppp_mp_discard(ippp_bundle *mp, - struct sk_buff *from, struct sk_buff *to) +static struct sk_buff *isdn_ppp_mp_discard(ippp_bundle *mp, + struct sk_buff *from, + struct sk_buff *to) { if (from) while (from != to) { @@ -1900,8 +1901,8 @@ struct sk_buff *isdn_ppp_mp_discard(ippp_bundle *mp, return from; } -void isdn_ppp_mp_reassembly(isdn_net_dev *net_dev, isdn_net_local *lp, - struct sk_buff *from, struct sk_buff *to) +static void isdn_ppp_mp_reassembly(isdn_net_dev *net_dev, isdn_net_local *lp, + struct sk_buff *from, struct sk_buff *to) { ippp_bundle *mp = net_dev->pb; int proto; diff --git a/drivers/media/rc/bpf-lirc.c b/drivers/media/rc/bpf-lirc.c index 390a722e6211320cd59764ac9de5cf8ff8cad9d6..ee657003c1a1efbe135187c49f3e202643f72dd5 100644 --- a/drivers/media/rc/bpf-lirc.c +++ b/drivers/media/rc/bpf-lirc.c @@ -97,6 +97,12 @@ lirc_mode2_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_map_update_elem_proto; case BPF_FUNC_map_delete_elem: return &bpf_map_delete_elem_proto; + case BPF_FUNC_map_push_elem: + return &bpf_map_push_elem_proto; + case BPF_FUNC_map_pop_elem: + return &bpf_map_pop_elem_proto; + case BPF_FUNC_map_peek_elem: + return &bpf_map_peek_elem_proto; case BPF_FUNC_ktime_get_ns: return &bpf_ktime_get_ns_proto; case BPF_FUNC_tail_call: diff --git a/drivers/net/Kconfig b/drivers/net/Kconfig index 7a96d168efc41dce1510fbac29522ce06851a8db..bc42f131f47c0874593c16e7284920e466259401 100644 --- a/drivers/net/Kconfig +++ b/drivers/net/Kconfig @@ -505,6 +505,7 @@ source "drivers/net/hyperv/Kconfig" config NETDEVSIM tristate "Simulated networking device" depends on DEBUG_FS + select NET_DEVLINK help This driver is a developer testing tool and software model that can be used to test various control path networking APIs, especially diff --git a/drivers/net/appletalk/ipddp.c b/drivers/net/appletalk/ipddp.c index 3d27616d9c85540304a8d78c4a2f050c0866b9a2..51cf5eca9c7f8de290d5498ef0b9a7f94190004c 100644 --- a/drivers/net/appletalk/ipddp.c +++ b/drivers/net/appletalk/ipddp.c @@ -116,11 +116,15 @@ static struct net_device * __init ipddp_init(void) */ static netdev_tx_t ipddp_xmit(struct sk_buff *skb, struct net_device *dev) { - __be32 paddr = skb_rtable(skb)->rt_gateway; + struct rtable *rtable = skb_rtable(skb); + __be32 paddr = 0; struct ddpehdr *ddp; struct ipddp_route *rt; struct atalk_addr *our_addr; + if (rtable->rt_gw_family == AF_INET) + paddr = rtable->rt_gw4; + spin_lock(&ipddp_route_lock); /* diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c index ee610721098e628d01ad988b1a9cad40aeef6f70..062fa7e3af4c267f68aa52cfeaa762994459cc41 100644 --- a/drivers/net/bonding/bond_main.c +++ b/drivers/net/bonding/bond_main.c @@ -4118,8 +4118,7 @@ static inline int bond_slave_override(struct bonding *bond, static u16 bond_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { /* This helper function exists to help dev_pick_tx get the correct * destination queue. Using a helper function skips a call to diff --git a/drivers/net/bonding/bond_netlink.c b/drivers/net/bonding/bond_netlink.c index b286f591242ebe5d9a3291408db08d412d2823f2..022044b59d6a830dc29b017e9b23dc2faa593776 100644 --- a/drivers/net/bonding/bond_netlink.c +++ b/drivers/net/bonding/bond_netlink.c @@ -546,7 +546,7 @@ static int bond_fill_info(struct sk_buff *skb, if (nla_put_u32(skb, IFLA_BOND_ARP_INTERVAL, bond->params.arp_interval)) goto nla_put_failure; - targets = nla_nest_start(skb, IFLA_BOND_ARP_IP_TARGET); + targets = nla_nest_start_noflag(skb, IFLA_BOND_ARP_IP_TARGET); if (!targets) goto nla_put_failure; @@ -644,7 +644,7 @@ static int bond_fill_info(struct sk_buff *skb, if (!bond_3ad_get_active_agg_info(bond, &info)) { struct nlattr *nest; - nest = nla_nest_start(skb, IFLA_BOND_AD_INFO); + nest = nla_nest_start_noflag(skb, IFLA_BOND_AD_INFO); if (!nest) goto nla_put_failure; @@ -711,7 +711,7 @@ static int bond_fill_linkxstats(struct sk_buff *skb, return -EINVAL; } - nest = nla_nest_start(skb, LINK_XSTATS_TYPE_BOND); + nest = nla_nest_start_noflag(skb, LINK_XSTATS_TYPE_BOND); if (!nest) return -EMSGSIZE; if (BOND_MODE(bond) == BOND_MODE_8023AD) { @@ -722,7 +722,7 @@ static int bond_fill_linkxstats(struct sk_buff *skb, else stats = &BOND_AD_INFO(bond).stats; - nest2 = nla_nest_start(skb, BOND_XSTATS_3AD); + nest2 = nla_nest_start_noflag(skb, BOND_XSTATS_3AD); if (!nest2) { nla_nest_end(skb, nest); return -EMSGSIZE; diff --git a/drivers/net/dsa/Kconfig b/drivers/net/dsa/Kconfig index 71bb3aebded4e46bb4d54e2aef2fc0951f1c8658..c6c5ecdbcaeff40189b44e8035a67e887de88dea 100644 --- a/drivers/net/dsa/Kconfig +++ b/drivers/net/dsa/Kconfig @@ -41,7 +41,7 @@ config NET_DSA_MT7530 config NET_DSA_MV88E6060 tristate "Marvell 88E6060 ethernet switch chip support" - depends on NET_DSA && NET_DSA_LEGACY + depends on NET_DSA select NET_DSA_TAG_TRAILER ---help--- This enables support for the Marvell 88E6060 ethernet switch @@ -51,6 +51,8 @@ source "drivers/net/dsa/microchip/Kconfig" source "drivers/net/dsa/mv88e6xxx/Kconfig" +source "drivers/net/dsa/sja1105/Kconfig" + config NET_DSA_QCA8K tristate "Qualcomm Atheros QCA8K Ethernet switch family support" depends on NET_DSA diff --git a/drivers/net/dsa/Makefile b/drivers/net/dsa/Makefile index 82e5d794c41f1523eeb406bec4f317acfce3374e..fefb6aaa82ba1d72bad2203c03f6bb1e659dca4b 100644 --- a/drivers/net/dsa/Makefile +++ b/drivers/net/dsa/Makefile @@ -18,3 +18,4 @@ obj-$(CONFIG_NET_DSA_VITESSE_VSC73XX) += vitesse-vsc73xx.o obj-y += b53/ obj-y += microchip/ obj-y += mv88e6xxx/ +obj-y += sja1105/ diff --git a/drivers/net/dsa/b53/b53_common.c b/drivers/net/dsa/b53/b53_common.c index 0852e5e0817721448c5cab9de89423a45181d6f6..c8040ecf4425bb85c8569d65e91e13dae0bff692 100644 --- a/drivers/net/dsa/b53/b53_common.c +++ b/drivers/net/dsa/b53/b53_common.c @@ -428,7 +428,6 @@ static void b53_enable_vlan(struct b53_device *dev, bool enable, b53_write8(dev, B53_CTRL_PAGE, B53_SWITCH_MODE, mgmt); dev->vlan_enabled = enable; - dev->vlan_filtering_enabled = enable_filtering; } static int b53_set_jumbo(struct b53_device *dev, bool enable, bool allow_10_100) @@ -665,7 +664,7 @@ int b53_configure_vlan(struct dsa_switch *ds) b53_do_vlan_op(dev, VTA_CMD_CLEAR); } - b53_enable_vlan(dev, false, dev->vlan_filtering_enabled); + b53_enable_vlan(dev, false, ds->vlan_filtering); b53_for_each_port(dev, i) b53_write16(dev, B53_VLAN_PAGE, @@ -966,6 +965,13 @@ static int b53_setup(struct dsa_switch *ds) b53_disable_port(ds, port); } + /* Let DSA handle the case were multiple bridges span the same switch + * device and different VLAN awareness settings are requested, which + * would be breaking filtering semantics for any of the other bridge + * devices. (not hardware supported) + */ + ds->vlan_filtering_is_global = true; + return ret; } @@ -1275,35 +1281,17 @@ EXPORT_SYMBOL(b53_phylink_mac_link_up); int b53_vlan_filtering(struct dsa_switch *ds, int port, bool vlan_filtering) { struct b53_device *dev = ds->priv; - struct net_device *bridge_dev; - unsigned int i; u16 pvid, new_pvid; - /* Handle the case were multiple bridges span the same switch device - * and one of them has a different setting than what is being requested - * which would be breaking filtering semantics for any of the other - * bridge devices. - */ - b53_for_each_port(dev, i) { - bridge_dev = dsa_to_port(ds, i)->bridge_dev; - if (bridge_dev && - bridge_dev != dsa_to_port(ds, port)->bridge_dev && - br_vlan_enabled(bridge_dev) != vlan_filtering) { - netdev_err(bridge_dev, - "VLAN filtering is global to the switch!\n"); - return -EINVAL; - } - } - b53_read16(dev, B53_VLAN_PAGE, B53_VLAN_PORT_DEF_TAG(port), &pvid); new_pvid = pvid; - if (dev->vlan_filtering_enabled && !vlan_filtering) { + if (!vlan_filtering) { /* Filtering is currently enabled, use the default PVID since * the bridge does not expect tagging anymore */ dev->ports[port].pvid = pvid; new_pvid = b53_default_pvid(dev); - } else if (!dev->vlan_filtering_enabled && vlan_filtering) { + } else { /* Filtering is currently disabled, restore the previous PVID */ new_pvid = dev->ports[port].pvid; } @@ -1329,7 +1317,7 @@ int b53_vlan_prepare(struct dsa_switch *ds, int port, if (vlan->vid_end > dev->num_vlans) return -ERANGE; - b53_enable_vlan(dev, true, dev->vlan_filtering_enabled); + b53_enable_vlan(dev, true, ds->vlan_filtering); return 0; } diff --git a/drivers/net/dsa/b53/b53_priv.h b/drivers/net/dsa/b53/b53_priv.h index e3441dcf2d2132a68b36b7e11f7e9da7f7dc8f10..f25bc80c4ffcfdc0fd591b21d3366191a192f950 100644 --- a/drivers/net/dsa/b53/b53_priv.h +++ b/drivers/net/dsa/b53/b53_priv.h @@ -139,7 +139,6 @@ struct b53_device { unsigned int num_vlans; struct b53_vlan *vlans; bool vlan_enabled; - bool vlan_filtering_enabled; unsigned int num_ports; struct b53_port *ports; }; diff --git a/drivers/net/dsa/bcm_sf2.c b/drivers/net/dsa/bcm_sf2.c index c8e3f05e1d729816eb9709611da1db6942c66bb4..4ccb3239f5f7994c398bd571fc06fb9435d927cb 100644 --- a/drivers/net/dsa/bcm_sf2.c +++ b/drivers/net/dsa/bcm_sf2.c @@ -1188,10 +1188,11 @@ static int bcm_sf2_sw_probe(struct platform_device *pdev) if (ret) goto out_mdio; - pr_info("Starfighter 2 top: %x.%02x, core: %x.%02x base: 0x%p, IRQs: %d, %d\n", - priv->hw_params.top_rev >> 8, priv->hw_params.top_rev & 0xff, - priv->hw_params.core_rev >> 8, priv->hw_params.core_rev & 0xff, - priv->core, priv->irq0, priv->irq1); + dev_info(&pdev->dev, + "Starfighter 2 top: %x.%02x, core: %x.%02x, IRQs: %d, %d\n", + priv->hw_params.top_rev >> 8, priv->hw_params.top_rev & 0xff, + priv->hw_params.core_rev >> 8, priv->hw_params.core_rev & 0xff, + priv->irq0, priv->irq1); return 0; diff --git a/drivers/net/dsa/lantiq_gswip.c b/drivers/net/dsa/lantiq_gswip.c index d8328866908c0cd8861bcfa9a4ad05e2e1f416b8..553831df58fed5638a3dfd8aebf5f5c01e558238 100644 --- a/drivers/net/dsa/lantiq_gswip.c +++ b/drivers/net/dsa/lantiq_gswip.c @@ -4,7 +4,25 @@ * * Copyright (C) 2010 Lantiq Deutschland * Copyright (C) 2012 John Crispin - * Copyright (C) 2017 - 2018 Hauke Mehrtens + * Copyright (C) 2017 - 2019 Hauke Mehrtens + * + * The VLAN and bridge model the GSWIP hardware uses does not directly + * matches the model DSA uses. + * + * The hardware has 64 possible table entries for bridges with one VLAN + * ID, one flow id and a list of ports for each bridge. All entries which + * match the same flow ID are combined in the mac learning table, they + * act as one global bridge. + * The hardware does not support VLAN filter on the port, but on the + * bridge, this driver converts the DSA model to the hardware. + * + * The CPU gets all the exception frames which do not match any forwarding + * rule and the CPU port is also added to all bridges. This makes it possible + * to handle all the special cases easily in software. + * At the initialization the driver allocates one bridge table entry for + * each switch port which is used when the port is used without an + * explicit bridge. This prevents the frames from being forwarded + * between all LAN ports by default. */ #include @@ -148,19 +166,29 @@ #define GSWIP_PCE_PMAP2 0x454 /* Default Multicast port map */ #define GSWIP_PCE_PMAP3 0x455 /* Default Unknown Unicast port map */ #define GSWIP_PCE_GCTRL_0 0x456 +#define GSWIP_PCE_GCTRL_0_MTFL BIT(0) /* MAC Table Flushing */ #define GSWIP_PCE_GCTRL_0_MC_VALID BIT(3) #define GSWIP_PCE_GCTRL_0_VLAN BIT(14) /* VLAN aware Switching */ #define GSWIP_PCE_GCTRL_1 0x457 #define GSWIP_PCE_GCTRL_1_MAC_GLOCK BIT(2) /* MAC Address table lock */ #define GSWIP_PCE_GCTRL_1_MAC_GLOCK_MOD BIT(3) /* Mac address table lock forwarding mode */ #define GSWIP_PCE_PCTRL_0p(p) (0x480 + ((p) * 0xA)) -#define GSWIP_PCE_PCTRL_0_INGRESS BIT(11) +#define GSWIP_PCE_PCTRL_0_TVM BIT(5) /* Transparent VLAN mode */ +#define GSWIP_PCE_PCTRL_0_VREP BIT(6) /* VLAN Replace Mode */ +#define GSWIP_PCE_PCTRL_0_INGRESS BIT(11) /* Accept special tag in ingress */ #define GSWIP_PCE_PCTRL_0_PSTATE_LISTEN 0x0 #define GSWIP_PCE_PCTRL_0_PSTATE_RX 0x1 #define GSWIP_PCE_PCTRL_0_PSTATE_TX 0x2 #define GSWIP_PCE_PCTRL_0_PSTATE_LEARNING 0x3 #define GSWIP_PCE_PCTRL_0_PSTATE_FORWARDING 0x7 #define GSWIP_PCE_PCTRL_0_PSTATE_MASK GENMASK(2, 0) +#define GSWIP_PCE_VCTRL(p) (0x485 + ((p) * 0xA)) +#define GSWIP_PCE_VCTRL_UVR BIT(0) /* Unknown VLAN Rule */ +#define GSWIP_PCE_VCTRL_VIMR BIT(3) /* VLAN Ingress Member violation rule */ +#define GSWIP_PCE_VCTRL_VEMR BIT(4) /* VLAN Egress Member violation rule */ +#define GSWIP_PCE_VCTRL_VSR BIT(5) /* VLAN Security */ +#define GSWIP_PCE_VCTRL_VID0 BIT(6) /* Priority Tagged Rule */ +#define GSWIP_PCE_DEFPVID(p) (0x486 + ((p) * 0xA)) #define GSWIP_MAC_FLEN 0x8C5 #define GSWIP_MAC_CTRL_2p(p) (0x905 + ((p) * 0xC)) @@ -183,6 +211,11 @@ #define GSWIP_SDMA_PCTRL_FCEN BIT(1) /* Flow Control Enable */ #define GSWIP_SDMA_PCTRL_PAUFWD BIT(1) /* Pause Frame Forwarding */ +#define GSWIP_TABLE_ACTIVE_VLAN 0x01 +#define GSWIP_TABLE_VLAN_MAPPING 0x02 +#define GSWIP_TABLE_MAC_BRIDGE 0x0b +#define GSWIP_TABLE_MAC_BRIDGE_STATIC 0x01 /* Static not, aging entry */ + #define XRX200_GPHY_FW_ALIGN (16 * 1024) struct gswip_hw_info { @@ -202,6 +235,12 @@ struct gswip_gphy_fw { char *fw_name; }; +struct gswip_vlan { + struct net_device *bridge; + u16 vid; + u8 fid; +}; + struct gswip_priv { __iomem void *gswip; __iomem void *mdio; @@ -211,8 +250,22 @@ struct gswip_priv { struct dsa_switch *ds; struct device *dev; struct regmap *rcu_regmap; + struct gswip_vlan vlans[64]; int num_gphy_fw; struct gswip_gphy_fw *gphy_fw; + u32 port_vlan_filter; +}; + +struct gswip_pce_table_entry { + u16 index; // PCE_TBL_ADDR.ADDR = pData->table_index + u16 table; // PCE_TBL_CTRL.ADDR = pData->table + u16 key[8]; + u16 val[5]; + u16 mask; + u8 gmap; + bool type; + bool valid; + bool key_mode; }; struct gswip_rmon_cnt_desc { @@ -447,10 +500,153 @@ static int gswip_mdio(struct gswip_priv *priv, struct device_node *mdio_np) return of_mdiobus_register(ds->slave_mii_bus, mdio_np); } +static int gswip_pce_table_entry_read(struct gswip_priv *priv, + struct gswip_pce_table_entry *tbl) +{ + int i; + int err; + u16 crtl; + u16 addr_mode = tbl->key_mode ? GSWIP_PCE_TBL_CTRL_OPMOD_KSRD : + GSWIP_PCE_TBL_CTRL_OPMOD_ADRD; + + err = gswip_switch_r_timeout(priv, GSWIP_PCE_TBL_CTRL, + GSWIP_PCE_TBL_CTRL_BAS); + if (err) + return err; + + gswip_switch_w(priv, tbl->index, GSWIP_PCE_TBL_ADDR); + gswip_switch_mask(priv, GSWIP_PCE_TBL_CTRL_ADDR_MASK | + GSWIP_PCE_TBL_CTRL_OPMOD_MASK, + tbl->table | addr_mode | GSWIP_PCE_TBL_CTRL_BAS, + GSWIP_PCE_TBL_CTRL); + + err = gswip_switch_r_timeout(priv, GSWIP_PCE_TBL_CTRL, + GSWIP_PCE_TBL_CTRL_BAS); + if (err) + return err; + + for (i = 0; i < ARRAY_SIZE(tbl->key); i++) + tbl->key[i] = gswip_switch_r(priv, GSWIP_PCE_TBL_KEY(i)); + + for (i = 0; i < ARRAY_SIZE(tbl->val); i++) + tbl->val[i] = gswip_switch_r(priv, GSWIP_PCE_TBL_VAL(i)); + + tbl->mask = gswip_switch_r(priv, GSWIP_PCE_TBL_MASK); + + crtl = gswip_switch_r(priv, GSWIP_PCE_TBL_CTRL); + + tbl->type = !!(crtl & GSWIP_PCE_TBL_CTRL_TYPE); + tbl->valid = !!(crtl & GSWIP_PCE_TBL_CTRL_VLD); + tbl->gmap = (crtl & GSWIP_PCE_TBL_CTRL_GMAP_MASK) >> 7; + + return 0; +} + +static int gswip_pce_table_entry_write(struct gswip_priv *priv, + struct gswip_pce_table_entry *tbl) +{ + int i; + int err; + u16 crtl; + u16 addr_mode = tbl->key_mode ? GSWIP_PCE_TBL_CTRL_OPMOD_KSWR : + GSWIP_PCE_TBL_CTRL_OPMOD_ADWR; + + err = gswip_switch_r_timeout(priv, GSWIP_PCE_TBL_CTRL, + GSWIP_PCE_TBL_CTRL_BAS); + if (err) + return err; + + gswip_switch_w(priv, tbl->index, GSWIP_PCE_TBL_ADDR); + gswip_switch_mask(priv, GSWIP_PCE_TBL_CTRL_ADDR_MASK | + GSWIP_PCE_TBL_CTRL_OPMOD_MASK, + tbl->table | addr_mode, + GSWIP_PCE_TBL_CTRL); + + for (i = 0; i < ARRAY_SIZE(tbl->key); i++) + gswip_switch_w(priv, tbl->key[i], GSWIP_PCE_TBL_KEY(i)); + + for (i = 0; i < ARRAY_SIZE(tbl->val); i++) + gswip_switch_w(priv, tbl->val[i], GSWIP_PCE_TBL_VAL(i)); + + gswip_switch_mask(priv, GSWIP_PCE_TBL_CTRL_ADDR_MASK | + GSWIP_PCE_TBL_CTRL_OPMOD_MASK, + tbl->table | addr_mode, + GSWIP_PCE_TBL_CTRL); + + gswip_switch_w(priv, tbl->mask, GSWIP_PCE_TBL_MASK); + + crtl = gswip_switch_r(priv, GSWIP_PCE_TBL_CTRL); + crtl &= ~(GSWIP_PCE_TBL_CTRL_TYPE | GSWIP_PCE_TBL_CTRL_VLD | + GSWIP_PCE_TBL_CTRL_GMAP_MASK); + if (tbl->type) + crtl |= GSWIP_PCE_TBL_CTRL_TYPE; + if (tbl->valid) + crtl |= GSWIP_PCE_TBL_CTRL_VLD; + crtl |= (tbl->gmap << 7) & GSWIP_PCE_TBL_CTRL_GMAP_MASK; + crtl |= GSWIP_PCE_TBL_CTRL_BAS; + gswip_switch_w(priv, crtl, GSWIP_PCE_TBL_CTRL); + + return gswip_switch_r_timeout(priv, GSWIP_PCE_TBL_CTRL, + GSWIP_PCE_TBL_CTRL_BAS); +} + +/* Add the LAN port into a bridge with the CPU port by + * default. This prevents automatic forwarding of + * packages between the LAN ports when no explicit + * bridge is configured. + */ +static int gswip_add_single_port_br(struct gswip_priv *priv, int port, bool add) +{ + struct gswip_pce_table_entry vlan_active = {0,}; + struct gswip_pce_table_entry vlan_mapping = {0,}; + unsigned int cpu_port = priv->hw_info->cpu_port; + unsigned int max_ports = priv->hw_info->max_ports; + int err; + + if (port >= max_ports) { + dev_err(priv->dev, "single port for %i supported\n", port); + return -EIO; + } + + vlan_active.index = port + 1; + vlan_active.table = GSWIP_TABLE_ACTIVE_VLAN; + vlan_active.key[0] = 0; /* vid */ + vlan_active.val[0] = port + 1 /* fid */; + vlan_active.valid = add; + err = gswip_pce_table_entry_write(priv, &vlan_active); + if (err) { + dev_err(priv->dev, "failed to write active VLAN: %d\n", err); + return err; + } + + if (!add) + return 0; + + vlan_mapping.index = port + 1; + vlan_mapping.table = GSWIP_TABLE_VLAN_MAPPING; + vlan_mapping.val[0] = 0 /* vid */; + vlan_mapping.val[1] = BIT(port) | BIT(cpu_port); + vlan_mapping.val[2] = 0; + err = gswip_pce_table_entry_write(priv, &vlan_mapping); + if (err) { + dev_err(priv->dev, "failed to write VLAN mapping: %d\n", err); + return err; + } + + return 0; +} + static int gswip_port_enable(struct dsa_switch *ds, int port, struct phy_device *phydev) { struct gswip_priv *priv = ds->priv; + int err; + + if (!dsa_is_cpu_port(ds, port)) { + err = gswip_add_single_port_br(priv, port, true); + if (err) + return err; + } /* RMON Counter Enable for port */ gswip_switch_w(priv, GSWIP_BM_PCFG_CNTEN, GSWIP_BM_PCFGp(port)); @@ -461,8 +657,6 @@ static int gswip_port_enable(struct dsa_switch *ds, int port, GSWIP_FDMA_PCTRLp(port)); gswip_switch_mask(priv, 0, GSWIP_SDMA_PCTRL_EN, GSWIP_SDMA_PCTRLp(port)); - gswip_switch_mask(priv, 0, GSWIP_PCE_PCTRL_0_INGRESS, - GSWIP_PCE_PCTRL_0p(port)); if (!dsa_is_cpu_port(ds, port)) { u32 macconf = GSWIP_MDIO_PHY_LINK_AUTO | @@ -535,6 +729,39 @@ static int gswip_pce_load_microcode(struct gswip_priv *priv) return 0; } +static int gswip_port_vlan_filtering(struct dsa_switch *ds, int port, + bool vlan_filtering) +{ + struct gswip_priv *priv = ds->priv; + struct net_device *bridge = dsa_to_port(ds, port)->bridge_dev; + + /* Do not allow changing the VLAN filtering options while in bridge */ + if (!!(priv->port_vlan_filter & BIT(port)) != vlan_filtering && bridge) + return -EIO; + + if (vlan_filtering) { + /* Use port based VLAN tag */ + gswip_switch_mask(priv, + GSWIP_PCE_VCTRL_VSR, + GSWIP_PCE_VCTRL_UVR | GSWIP_PCE_VCTRL_VIMR | + GSWIP_PCE_VCTRL_VEMR, + GSWIP_PCE_VCTRL(port)); + gswip_switch_mask(priv, GSWIP_PCE_PCTRL_0_TVM, 0, + GSWIP_PCE_PCTRL_0p(port)); + } else { + /* Use port based VLAN tag */ + gswip_switch_mask(priv, + GSWIP_PCE_VCTRL_UVR | GSWIP_PCE_VCTRL_VIMR | + GSWIP_PCE_VCTRL_VEMR, + GSWIP_PCE_VCTRL_VSR, + GSWIP_PCE_VCTRL(port)); + gswip_switch_mask(priv, 0, GSWIP_PCE_PCTRL_0_TVM, + GSWIP_PCE_PCTRL_0p(port)); + } + + return 0; +} + static int gswip_setup(struct dsa_switch *ds) { struct gswip_priv *priv = ds->priv; @@ -547,8 +774,10 @@ static int gswip_setup(struct dsa_switch *ds) gswip_switch_w(priv, 0, GSWIP_SWRES); /* disable port fetch/store dma on all ports */ - for (i = 0; i < priv->hw_info->max_ports; i++) + for (i = 0; i < priv->hw_info->max_ports; i++) { gswip_port_disable(ds, i); + gswip_port_vlan_filtering(ds, i, false); + } /* enable Switch */ gswip_mdio_mask(priv, 0, GSWIP_MDIO_GLOB_ENABLE, GSWIP_MDIO_GLOB); @@ -578,6 +807,10 @@ static int gswip_setup(struct dsa_switch *ds) gswip_switch_mask(priv, 0, GSWIP_FDMA_PCTRL_STEN, GSWIP_FDMA_PCTRLp(cpu_port)); + /* accept special tag in ingress direction */ + gswip_switch_mask(priv, 0, GSWIP_PCE_PCTRL_0_INGRESS, + GSWIP_PCE_PCTRL_0p(cpu_port)); + gswip_switch_mask(priv, 0, GSWIP_MAC_CTRL_2_MLEN, GSWIP_MAC_CTRL_2p(cpu_port)); gswip_switch_w(priv, VLAN_ETH_FRAME_LEN + 8, GSWIP_MAC_FLEN); @@ -587,10 +820,15 @@ static int gswip_setup(struct dsa_switch *ds) /* VLAN aware Switching */ gswip_switch_mask(priv, 0, GSWIP_PCE_GCTRL_0_VLAN, GSWIP_PCE_GCTRL_0); - /* Mac Address Table Lock */ - gswip_switch_mask(priv, 0, GSWIP_PCE_GCTRL_1_MAC_GLOCK | - GSWIP_PCE_GCTRL_1_MAC_GLOCK_MOD, - GSWIP_PCE_GCTRL_1); + /* Flush MAC Table */ + gswip_switch_mask(priv, 0, GSWIP_PCE_GCTRL_0_MTFL, GSWIP_PCE_GCTRL_0); + + err = gswip_switch_r_timeout(priv, GSWIP_PCE_GCTRL_0, + GSWIP_PCE_GCTRL_0_MTFL); + if (err) { + dev_err(priv->dev, "MAC flushing didn't finish\n"); + return err; + } gswip_port_enable(ds, cpu_port, NULL); return 0; @@ -602,6 +840,551 @@ static enum dsa_tag_protocol gswip_get_tag_protocol(struct dsa_switch *ds, return DSA_TAG_PROTO_GSWIP; } +static int gswip_vlan_active_create(struct gswip_priv *priv, + struct net_device *bridge, + int fid, u16 vid) +{ + struct gswip_pce_table_entry vlan_active = {0,}; + unsigned int max_ports = priv->hw_info->max_ports; + int idx = -1; + int err; + int i; + + /* Look for a free slot */ + for (i = max_ports; i < ARRAY_SIZE(priv->vlans); i++) { + if (!priv->vlans[i].bridge) { + idx = i; + break; + } + } + + if (idx == -1) + return -ENOSPC; + + if (fid == -1) + fid = idx; + + vlan_active.index = idx; + vlan_active.table = GSWIP_TABLE_ACTIVE_VLAN; + vlan_active.key[0] = vid; + vlan_active.val[0] = fid; + vlan_active.valid = true; + + err = gswip_pce_table_entry_write(priv, &vlan_active); + if (err) { + dev_err(priv->dev, "failed to write active VLAN: %d\n", err); + return err; + } + + priv->vlans[idx].bridge = bridge; + priv->vlans[idx].vid = vid; + priv->vlans[idx].fid = fid; + + return idx; +} + +static int gswip_vlan_active_remove(struct gswip_priv *priv, int idx) +{ + struct gswip_pce_table_entry vlan_active = {0,}; + int err; + + vlan_active.index = idx; + vlan_active.table = GSWIP_TABLE_ACTIVE_VLAN; + vlan_active.valid = false; + err = gswip_pce_table_entry_write(priv, &vlan_active); + if (err) + dev_err(priv->dev, "failed to delete active VLAN: %d\n", err); + priv->vlans[idx].bridge = NULL; + + return err; +} + +static int gswip_vlan_add_unaware(struct gswip_priv *priv, + struct net_device *bridge, int port) +{ + struct gswip_pce_table_entry vlan_mapping = {0,}; + unsigned int max_ports = priv->hw_info->max_ports; + unsigned int cpu_port = priv->hw_info->cpu_port; + bool active_vlan_created = false; + int idx = -1; + int i; + int err; + + /* Check if there is already a page for this bridge */ + for (i = max_ports; i < ARRAY_SIZE(priv->vlans); i++) { + if (priv->vlans[i].bridge == bridge) { + idx = i; + break; + } + } + + /* If this bridge is not programmed yet, add a Active VLAN table + * entry in a free slot and prepare the VLAN mapping table entry. + */ + if (idx == -1) { + idx = gswip_vlan_active_create(priv, bridge, -1, 0); + if (idx < 0) + return idx; + active_vlan_created = true; + + vlan_mapping.index = idx; + vlan_mapping.table = GSWIP_TABLE_VLAN_MAPPING; + /* VLAN ID byte, maps to the VLAN ID of vlan active table */ + vlan_mapping.val[0] = 0; + } else { + /* Read the existing VLAN mapping entry from the switch */ + vlan_mapping.index = idx; + vlan_mapping.table = GSWIP_TABLE_VLAN_MAPPING; + err = gswip_pce_table_entry_read(priv, &vlan_mapping); + if (err) { + dev_err(priv->dev, "failed to read VLAN mapping: %d\n", + err); + return err; + } + } + + /* Update the VLAN mapping entry and write it to the switch */ + vlan_mapping.val[1] |= BIT(cpu_port); + vlan_mapping.val[1] |= BIT(port); + err = gswip_pce_table_entry_write(priv, &vlan_mapping); + if (err) { + dev_err(priv->dev, "failed to write VLAN mapping: %d\n", err); + /* In case an Active VLAN was creaetd delete it again */ + if (active_vlan_created) + gswip_vlan_active_remove(priv, idx); + return err; + } + + gswip_switch_w(priv, 0, GSWIP_PCE_DEFPVID(port)); + return 0; +} + +static int gswip_vlan_add_aware(struct gswip_priv *priv, + struct net_device *bridge, int port, + u16 vid, bool untagged, + bool pvid) +{ + struct gswip_pce_table_entry vlan_mapping = {0,}; + unsigned int max_ports = priv->hw_info->max_ports; + unsigned int cpu_port = priv->hw_info->cpu_port; + bool active_vlan_created = false; + int idx = -1; + int fid = -1; + int i; + int err; + + /* Check if there is already a page for this bridge */ + for (i = max_ports; i < ARRAY_SIZE(priv->vlans); i++) { + if (priv->vlans[i].bridge == bridge) { + if (fid != -1 && fid != priv->vlans[i].fid) + dev_err(priv->dev, "one bridge with multiple flow ids\n"); + fid = priv->vlans[i].fid; + if (priv->vlans[i].vid == vid) { + idx = i; + break; + } + } + } + + /* If this bridge is not programmed yet, add a Active VLAN table + * entry in a free slot and prepare the VLAN mapping table entry. + */ + if (idx == -1) { + idx = gswip_vlan_active_create(priv, bridge, fid, vid); + if (idx < 0) + return idx; + active_vlan_created = true; + + vlan_mapping.index = idx; + vlan_mapping.table = GSWIP_TABLE_VLAN_MAPPING; + /* VLAN ID byte, maps to the VLAN ID of vlan active table */ + vlan_mapping.val[0] = vid; + } else { + /* Read the existing VLAN mapping entry from the switch */ + vlan_mapping.index = idx; + vlan_mapping.table = GSWIP_TABLE_VLAN_MAPPING; + err = gswip_pce_table_entry_read(priv, &vlan_mapping); + if (err) { + dev_err(priv->dev, "failed to read VLAN mapping: %d\n", + err); + return err; + } + } + + vlan_mapping.val[0] = vid; + /* Update the VLAN mapping entry and write it to the switch */ + vlan_mapping.val[1] |= BIT(cpu_port); + vlan_mapping.val[2] |= BIT(cpu_port); + vlan_mapping.val[1] |= BIT(port); + if (untagged) + vlan_mapping.val[2] &= ~BIT(port); + else + vlan_mapping.val[2] |= BIT(port); + err = gswip_pce_table_entry_write(priv, &vlan_mapping); + if (err) { + dev_err(priv->dev, "failed to write VLAN mapping: %d\n", err); + /* In case an Active VLAN was creaetd delete it again */ + if (active_vlan_created) + gswip_vlan_active_remove(priv, idx); + return err; + } + + if (pvid) + gswip_switch_w(priv, idx, GSWIP_PCE_DEFPVID(port)); + + return 0; +} + +static int gswip_vlan_remove(struct gswip_priv *priv, + struct net_device *bridge, int port, + u16 vid, bool pvid, bool vlan_aware) +{ + struct gswip_pce_table_entry vlan_mapping = {0,}; + unsigned int max_ports = priv->hw_info->max_ports; + unsigned int cpu_port = priv->hw_info->cpu_port; + int idx = -1; + int i; + int err; + + /* Check if there is already a page for this bridge */ + for (i = max_ports; i < ARRAY_SIZE(priv->vlans); i++) { + if (priv->vlans[i].bridge == bridge && + (!vlan_aware || priv->vlans[i].vid == vid)) { + idx = i; + break; + } + } + + if (idx == -1) { + dev_err(priv->dev, "bridge to leave does not exists\n"); + return -ENOENT; + } + + vlan_mapping.index = idx; + vlan_mapping.table = GSWIP_TABLE_VLAN_MAPPING; + err = gswip_pce_table_entry_read(priv, &vlan_mapping); + if (err) { + dev_err(priv->dev, "failed to read VLAN mapping: %d\n", err); + return err; + } + + vlan_mapping.val[1] &= ~BIT(port); + vlan_mapping.val[2] &= ~BIT(port); + err = gswip_pce_table_entry_write(priv, &vlan_mapping); + if (err) { + dev_err(priv->dev, "failed to write VLAN mapping: %d\n", err); + return err; + } + + /* In case all ports are removed from the bridge, remove the VLAN */ + if ((vlan_mapping.val[1] & ~BIT(cpu_port)) == 0) { + err = gswip_vlan_active_remove(priv, idx); + if (err) { + dev_err(priv->dev, "failed to write active VLAN: %d\n", + err); + return err; + } + } + + /* GSWIP 2.2 (GRX300) and later program here the VID directly. */ + if (pvid) + gswip_switch_w(priv, 0, GSWIP_PCE_DEFPVID(port)); + + return 0; +} + +static int gswip_port_bridge_join(struct dsa_switch *ds, int port, + struct net_device *bridge) +{ + struct gswip_priv *priv = ds->priv; + int err; + + /* When the bridge uses VLAN filtering we have to configure VLAN + * specific bridges. No bridge is configured here. + */ + if (!br_vlan_enabled(bridge)) { + err = gswip_vlan_add_unaware(priv, bridge, port); + if (err) + return err; + priv->port_vlan_filter &= ~BIT(port); + } else { + priv->port_vlan_filter |= BIT(port); + } + return gswip_add_single_port_br(priv, port, false); +} + +static void gswip_port_bridge_leave(struct dsa_switch *ds, int port, + struct net_device *bridge) +{ + struct gswip_priv *priv = ds->priv; + + gswip_add_single_port_br(priv, port, true); + + /* When the bridge uses VLAN filtering we have to configure VLAN + * specific bridges. No bridge is configured here. + */ + if (!br_vlan_enabled(bridge)) + gswip_vlan_remove(priv, bridge, port, 0, true, false); +} + +static int gswip_port_vlan_prepare(struct dsa_switch *ds, int port, + const struct switchdev_obj_port_vlan *vlan) +{ + struct gswip_priv *priv = ds->priv; + struct net_device *bridge = dsa_to_port(ds, port)->bridge_dev; + unsigned int max_ports = priv->hw_info->max_ports; + u16 vid; + int i; + int pos = max_ports; + + /* We only support VLAN filtering on bridges */ + if (!dsa_is_cpu_port(ds, port) && !bridge) + return -EOPNOTSUPP; + + for (vid = vlan->vid_begin; vid <= vlan->vid_end; ++vid) { + int idx = -1; + + /* Check if there is already a page for this VLAN */ + for (i = max_ports; i < ARRAY_SIZE(priv->vlans); i++) { + if (priv->vlans[i].bridge == bridge && + priv->vlans[i].vid == vid) { + idx = i; + break; + } + } + + /* If this VLAN is not programmed yet, we have to reserve + * one entry in the VLAN table. Make sure we start at the + * next position round. + */ + if (idx == -1) { + /* Look for a free slot */ + for (; pos < ARRAY_SIZE(priv->vlans); pos++) { + if (!priv->vlans[pos].bridge) { + idx = pos; + pos++; + break; + } + } + + if (idx == -1) + return -ENOSPC; + } + } + + return 0; +} + +static void gswip_port_vlan_add(struct dsa_switch *ds, int port, + const struct switchdev_obj_port_vlan *vlan) +{ + struct gswip_priv *priv = ds->priv; + struct net_device *bridge = dsa_to_port(ds, port)->bridge_dev; + bool untagged = vlan->flags & BRIDGE_VLAN_INFO_UNTAGGED; + bool pvid = vlan->flags & BRIDGE_VLAN_INFO_PVID; + u16 vid; + + /* We have to receive all packets on the CPU port and should not + * do any VLAN filtering here. This is also called with bridge + * NULL and then we do not know for which bridge to configure + * this. + */ + if (dsa_is_cpu_port(ds, port)) + return; + + for (vid = vlan->vid_begin; vid <= vlan->vid_end; ++vid) + gswip_vlan_add_aware(priv, bridge, port, vid, untagged, pvid); +} + +static int gswip_port_vlan_del(struct dsa_switch *ds, int port, + const struct switchdev_obj_port_vlan *vlan) +{ + struct gswip_priv *priv = ds->priv; + struct net_device *bridge = dsa_to_port(ds, port)->bridge_dev; + bool pvid = vlan->flags & BRIDGE_VLAN_INFO_PVID; + u16 vid; + int err; + + /* We have to receive all packets on the CPU port and should not + * do any VLAN filtering here. This is also called with bridge + * NULL and then we do not know for which bridge to configure + * this. + */ + if (dsa_is_cpu_port(ds, port)) + return 0; + + for (vid = vlan->vid_begin; vid <= vlan->vid_end; ++vid) { + err = gswip_vlan_remove(priv, bridge, port, vid, pvid, true); + if (err) + return err; + } + + return 0; +} + +static void gswip_port_fast_age(struct dsa_switch *ds, int port) +{ + struct gswip_priv *priv = ds->priv; + struct gswip_pce_table_entry mac_bridge = {0,}; + int i; + int err; + + for (i = 0; i < 2048; i++) { + mac_bridge.table = GSWIP_TABLE_MAC_BRIDGE; + mac_bridge.index = i; + + err = gswip_pce_table_entry_read(priv, &mac_bridge); + if (err) { + dev_err(priv->dev, "failed to read mac brigde: %d\n", + err); + return; + } + + if (!mac_bridge.valid) + continue; + + if (mac_bridge.val[1] & GSWIP_TABLE_MAC_BRIDGE_STATIC) + continue; + + if (((mac_bridge.val[0] & GENMASK(7, 4)) >> 4) != port) + continue; + + mac_bridge.valid = false; + err = gswip_pce_table_entry_write(priv, &mac_bridge); + if (err) { + dev_err(priv->dev, "failed to write mac brigde: %d\n", + err); + return; + } + } +} + +static void gswip_port_stp_state_set(struct dsa_switch *ds, int port, u8 state) +{ + struct gswip_priv *priv = ds->priv; + u32 stp_state; + + switch (state) { + case BR_STATE_DISABLED: + gswip_switch_mask(priv, GSWIP_SDMA_PCTRL_EN, 0, + GSWIP_SDMA_PCTRLp(port)); + return; + case BR_STATE_BLOCKING: + case BR_STATE_LISTENING: + stp_state = GSWIP_PCE_PCTRL_0_PSTATE_LISTEN; + break; + case BR_STATE_LEARNING: + stp_state = GSWIP_PCE_PCTRL_0_PSTATE_LEARNING; + break; + case BR_STATE_FORWARDING: + stp_state = GSWIP_PCE_PCTRL_0_PSTATE_FORWARDING; + break; + default: + dev_err(priv->dev, "invalid STP state: %d\n", state); + return; + } + + gswip_switch_mask(priv, 0, GSWIP_SDMA_PCTRL_EN, + GSWIP_SDMA_PCTRLp(port)); + gswip_switch_mask(priv, GSWIP_PCE_PCTRL_0_PSTATE_MASK, stp_state, + GSWIP_PCE_PCTRL_0p(port)); +} + +static int gswip_port_fdb(struct dsa_switch *ds, int port, + const unsigned char *addr, u16 vid, bool add) +{ + struct gswip_priv *priv = ds->priv; + struct net_device *bridge = dsa_to_port(ds, port)->bridge_dev; + struct gswip_pce_table_entry mac_bridge = {0,}; + unsigned int cpu_port = priv->hw_info->cpu_port; + int fid = -1; + int i; + int err; + + if (!bridge) + return -EINVAL; + + for (i = cpu_port; i < ARRAY_SIZE(priv->vlans); i++) { + if (priv->vlans[i].bridge == bridge) { + fid = priv->vlans[i].fid; + break; + } + } + + if (fid == -1) { + dev_err(priv->dev, "Port not part of a bridge\n"); + return -EINVAL; + } + + mac_bridge.table = GSWIP_TABLE_MAC_BRIDGE; + mac_bridge.key_mode = true; + mac_bridge.key[0] = addr[5] | (addr[4] << 8); + mac_bridge.key[1] = addr[3] | (addr[2] << 8); + mac_bridge.key[2] = addr[1] | (addr[0] << 8); + mac_bridge.key[3] = fid; + mac_bridge.val[0] = add ? BIT(port) : 0; /* port map */ + mac_bridge.val[1] = GSWIP_TABLE_MAC_BRIDGE_STATIC; + mac_bridge.valid = add; + + err = gswip_pce_table_entry_write(priv, &mac_bridge); + if (err) + dev_err(priv->dev, "failed to write mac brigde: %d\n", err); + + return err; +} + +static int gswip_port_fdb_add(struct dsa_switch *ds, int port, + const unsigned char *addr, u16 vid) +{ + return gswip_port_fdb(ds, port, addr, vid, true); +} + +static int gswip_port_fdb_del(struct dsa_switch *ds, int port, + const unsigned char *addr, u16 vid) +{ + return gswip_port_fdb(ds, port, addr, vid, false); +} + +static int gswip_port_fdb_dump(struct dsa_switch *ds, int port, + dsa_fdb_dump_cb_t *cb, void *data) +{ + struct gswip_priv *priv = ds->priv; + struct gswip_pce_table_entry mac_bridge = {0,}; + unsigned char addr[6]; + int i; + int err; + + for (i = 0; i < 2048; i++) { + mac_bridge.table = GSWIP_TABLE_MAC_BRIDGE; + mac_bridge.index = i; + + err = gswip_pce_table_entry_read(priv, &mac_bridge); + if (err) { + dev_err(priv->dev, "failed to write mac brigde: %d\n", + err); + return err; + } + + if (!mac_bridge.valid) + continue; + + addr[5] = mac_bridge.key[0] & 0xff; + addr[4] = (mac_bridge.key[0] >> 8) & 0xff; + addr[3] = mac_bridge.key[1] & 0xff; + addr[2] = (mac_bridge.key[1] >> 8) & 0xff; + addr[1] = mac_bridge.key[2] & 0xff; + addr[0] = (mac_bridge.key[2] >> 8) & 0xff; + if (mac_bridge.val[1] & GSWIP_TABLE_MAC_BRIDGE_STATIC) { + if (mac_bridge.val[0] & BIT(port)) + cb(addr, 0, true, data); + } else { + if (((mac_bridge.val[0] & GENMASK(7, 4)) >> 4) == port) + cb(addr, 0, false, data); + } + } + return 0; +} + static void gswip_phylink_validate(struct dsa_switch *ds, int port, unsigned long *supported, struct phylink_link_state *state) @@ -809,6 +1592,17 @@ static const struct dsa_switch_ops gswip_switch_ops = { .setup = gswip_setup, .port_enable = gswip_port_enable, .port_disable = gswip_port_disable, + .port_bridge_join = gswip_port_bridge_join, + .port_bridge_leave = gswip_port_bridge_leave, + .port_fast_age = gswip_port_fast_age, + .port_vlan_filtering = gswip_port_vlan_filtering, + .port_vlan_prepare = gswip_port_vlan_prepare, + .port_vlan_add = gswip_port_vlan_add, + .port_vlan_del = gswip_port_vlan_del, + .port_stp_state_set = gswip_port_stp_state_set, + .port_fdb_add = gswip_port_fdb_add, + .port_fdb_del = gswip_port_fdb_del, + .port_fdb_dump = gswip_port_fdb_dump, .phylink_validate = gswip_phylink_validate, .phylink_mac_config = gswip_phylink_mac_config, .phylink_mac_link_down = gswip_phylink_mac_link_down, diff --git a/drivers/net/dsa/microchip/ksz9477.c b/drivers/net/dsa/microchip/ksz9477.c index f16e1d7d8615d6191a42d158bf274a4e663e94d0..c026d15721f66bf3d03403b1f31d5362ec38bd9c 100644 --- a/drivers/net/dsa/microchip/ksz9477.c +++ b/drivers/net/dsa/microchip/ksz9477.c @@ -1144,6 +1144,7 @@ static phy_interface_t ksz9477_get_interface(struct ksz_device *dev, int port) interface = PHY_INTERFACE_MODE_GMII; if (gbit) break; + /* fall through */ case 0: interface = PHY_INTERFACE_MODE_MII; break; diff --git a/drivers/net/dsa/mt7530.c b/drivers/net/dsa/mt7530.c index 7357b4fc0185bb986c28c1fde8e0cda0b5d6d07a..8d531c5f21f31e53da08dd25b8bf88d76f282363 100644 --- a/drivers/net/dsa/mt7530.c +++ b/drivers/net/dsa/mt7530.c @@ -828,11 +828,9 @@ mt7530_port_set_vlan_unaware(struct dsa_switch *ds, int port) mt7530_rmw(priv, MT7530_PVC_P(port), VLAN_ATTR_MASK, VLAN_ATTR(MT7530_VLAN_TRANSPARENT)); - priv->ports[port].vlan_filtering = false; - for (i = 0; i < MT7530_NUM_PORTS; i++) { if (dsa_is_user_port(ds, i) && - priv->ports[i].vlan_filtering) { + dsa_port_is_vlan_filtering(&ds->ports[i])) { all_user_ports_removed = false; break; } @@ -891,8 +889,8 @@ mt7530_port_bridge_leave(struct dsa_switch *ds, int port, * And the other port's port matrix cannot be broken when the * other port is still a VLAN-aware port. */ - if (!priv->ports[i].vlan_filtering && - dsa_is_user_port(ds, i) && i != port) { + if (dsa_is_user_port(ds, i) && i != port && + !dsa_port_is_vlan_filtering(&ds->ports[i])) { if (dsa_to_port(ds, i)->bridge_dev != bridge) continue; if (priv->ports[i].enable) @@ -910,8 +908,6 @@ mt7530_port_bridge_leave(struct dsa_switch *ds, int port, PCR_MATRIX(BIT(MT7530_CPU_PORT))); priv->ports[port].pm = PCR_MATRIX(BIT(MT7530_CPU_PORT)); - mt7530_port_set_vlan_unaware(ds, port); - mutex_unlock(&priv->reg_mutex); } @@ -1013,10 +1009,6 @@ static int mt7530_port_vlan_filtering(struct dsa_switch *ds, int port, bool vlan_filtering) { - struct mt7530_priv *priv = ds->priv; - - priv->ports[port].vlan_filtering = vlan_filtering; - if (vlan_filtering) { /* The port is being kept as VLAN-unaware port when bridge is * set up with vlan_filtering not being set, Otherwise, the @@ -1025,6 +1017,8 @@ mt7530_port_vlan_filtering(struct dsa_switch *ds, int port, */ mt7530_port_set_vlan_aware(ds, port); mt7530_port_set_vlan_aware(ds, MT7530_CPU_PORT); + } else { + mt7530_port_set_vlan_unaware(ds, port); } return 0; @@ -1139,7 +1133,7 @@ mt7530_port_vlan_add(struct dsa_switch *ds, int port, /* The port is kept as VLAN-unaware if bridge with vlan_filtering not * being set. */ - if (!priv->ports[port].vlan_filtering) + if (!dsa_port_is_vlan_filtering(&ds->ports[port])) return; mutex_lock(&priv->reg_mutex); @@ -1170,7 +1164,7 @@ mt7530_port_vlan_del(struct dsa_switch *ds, int port, /* The port is kept as VLAN-unaware if bridge with vlan_filtering not * being set. */ - if (!priv->ports[port].vlan_filtering) + if (!dsa_port_is_vlan_filtering(&ds->ports[port])) return 0; mutex_lock(&priv->reg_mutex); diff --git a/drivers/net/dsa/mt7530.h b/drivers/net/dsa/mt7530.h index a95ed958df5b01978e70a5b4760d238dfe15d124..1eec7bdc283a7eeb98bcae0e5e449407ca577959 100644 --- a/drivers/net/dsa/mt7530.h +++ b/drivers/net/dsa/mt7530.h @@ -410,7 +410,6 @@ struct mt7530_port { bool enable; u32 pm; u16 pvid; - bool vlan_filtering; }; /* struct mt7530_priv - This is the main data structure for holding the state diff --git a/drivers/net/dsa/mv88e6060.c b/drivers/net/dsa/mv88e6060.c index 0b3e51f248c21a2477c9b1736b0f06b80e350e29..2a2489b5196d7c52dada2543ef1a62ea69abfbcf 100644 --- a/drivers/net/dsa/mv88e6060.c +++ b/drivers/net/dsa/mv88e6060.c @@ -1,11 +1,7 @@ +// SPDX-License-Identifier: GPL-2.0+ /* * net/dsa/mv88e6060.c - Driver for Marvell 88e6060 switch chips * Copyright (c) 2008-2009 Marvell Semiconductor - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. */ #include @@ -18,40 +14,16 @@ #include #include "mv88e6060.h" -static int reg_read(struct dsa_switch *ds, int addr, int reg) +static int reg_read(struct mv88e6060_priv *priv, int addr, int reg) { - struct mv88e6060_priv *priv = ds->priv; - return mdiobus_read_nested(priv->bus, priv->sw_addr + addr, reg); } -#define REG_READ(addr, reg) \ - ({ \ - int __ret; \ - \ - __ret = reg_read(ds, addr, reg); \ - if (__ret < 0) \ - return __ret; \ - __ret; \ - }) - - -static int reg_write(struct dsa_switch *ds, int addr, int reg, u16 val) +static int reg_write(struct mv88e6060_priv *priv, int addr, int reg, u16 val) { - struct mv88e6060_priv *priv = ds->priv; - return mdiobus_write_nested(priv->bus, priv->sw_addr + addr, reg, val); } -#define REG_WRITE(addr, reg, val) \ - ({ \ - int __ret; \ - \ - __ret = reg_write(ds, addr, reg, val); \ - if (__ret < 0) \ - return __ret; \ - }) - static const char *mv88e6060_get_name(struct mii_bus *bus, int sw_addr) { int ret; @@ -76,28 +48,7 @@ static enum dsa_tag_protocol mv88e6060_get_tag_protocol(struct dsa_switch *ds, return DSA_TAG_PROTO_TRAILER; } -static const char *mv88e6060_drv_probe(struct device *dsa_dev, - struct device *host_dev, int sw_addr, - void **_priv) -{ - struct mii_bus *bus = dsa_host_dev_to_mii_bus(host_dev); - struct mv88e6060_priv *priv; - const char *name; - - name = mv88e6060_get_name(bus, sw_addr); - if (name) { - priv = devm_kzalloc(dsa_dev, sizeof(*priv), GFP_KERNEL); - if (!priv) - return NULL; - *_priv = priv; - priv->bus = bus; - priv->sw_addr = sw_addr; - } - - return name; -} - -static int mv88e6060_switch_reset(struct dsa_switch *ds) +static int mv88e6060_switch_reset(struct mv88e6060_priv *priv) { int i; int ret; @@ -105,23 +56,32 @@ static int mv88e6060_switch_reset(struct dsa_switch *ds) /* Set all ports to the disabled state. */ for (i = 0; i < MV88E6060_PORTS; i++) { - ret = REG_READ(REG_PORT(i), PORT_CONTROL); - REG_WRITE(REG_PORT(i), PORT_CONTROL, - ret & ~PORT_CONTROL_STATE_MASK); + ret = reg_read(priv, REG_PORT(i), PORT_CONTROL); + if (ret < 0) + return ret; + ret = reg_write(priv, REG_PORT(i), PORT_CONTROL, + ret & ~PORT_CONTROL_STATE_MASK); + if (ret) + return ret; } /* Wait for transmit queues to drain. */ usleep_range(2000, 4000); /* Reset the switch. */ - REG_WRITE(REG_GLOBAL, GLOBAL_ATU_CONTROL, - GLOBAL_ATU_CONTROL_SWRESET | - GLOBAL_ATU_CONTROL_LEARNDIS); + ret = reg_write(priv, REG_GLOBAL, GLOBAL_ATU_CONTROL, + GLOBAL_ATU_CONTROL_SWRESET | + GLOBAL_ATU_CONTROL_LEARNDIS); + if (ret) + return ret; /* Wait up to one second for reset to complete. */ timeout = jiffies + 1 * HZ; while (time_before(jiffies, timeout)) { - ret = REG_READ(REG_GLOBAL, GLOBAL_STATUS); + ret = reg_read(priv, REG_GLOBAL, GLOBAL_STATUS); + if (ret < 0) + return ret; + if (ret & GLOBAL_STATUS_INIT_READY) break; @@ -133,61 +93,69 @@ static int mv88e6060_switch_reset(struct dsa_switch *ds) return 0; } -static int mv88e6060_setup_global(struct dsa_switch *ds) +static int mv88e6060_setup_global(struct mv88e6060_priv *priv) { + int ret; + /* Disable discarding of frames with excessive collisions, * set the maximum frame size to 1536 bytes, and mask all * interrupt sources. */ - REG_WRITE(REG_GLOBAL, GLOBAL_CONTROL, GLOBAL_CONTROL_MAX_FRAME_1536); + ret = reg_write(priv, REG_GLOBAL, GLOBAL_CONTROL, + GLOBAL_CONTROL_MAX_FRAME_1536); + if (ret) + return ret; /* Disable automatic address learning. */ - REG_WRITE(REG_GLOBAL, GLOBAL_ATU_CONTROL, - GLOBAL_ATU_CONTROL_LEARNDIS); - - return 0; + return reg_write(priv, REG_GLOBAL, GLOBAL_ATU_CONTROL, + GLOBAL_ATU_CONTROL_LEARNDIS); } -static int mv88e6060_setup_port(struct dsa_switch *ds, int p) +static int mv88e6060_setup_port(struct mv88e6060_priv *priv, int p) { int addr = REG_PORT(p); + int ret; /* Do not force flow control, disable Ingress and Egress * Header tagging, disable VLAN tunneling, and set the port * state to Forwarding. Additionally, if this is the CPU * port, enable Ingress and Egress Trailer tagging mode. */ - REG_WRITE(addr, PORT_CONTROL, - dsa_is_cpu_port(ds, p) ? + ret = reg_write(priv, addr, PORT_CONTROL, + dsa_is_cpu_port(priv->ds, p) ? PORT_CONTROL_TRAILER | PORT_CONTROL_INGRESS_MODE | PORT_CONTROL_STATE_FORWARDING : PORT_CONTROL_STATE_FORWARDING); + if (ret) + return ret; /* Port based VLAN map: give each port its own address * database, allow the CPU port to talk to each of the 'real' * ports, and allow each of the 'real' ports to only talk to * the CPU port. */ - REG_WRITE(addr, PORT_VLAN_MAP, - ((p & 0xf) << PORT_VLAN_MAP_DBNUM_SHIFT) | - (dsa_is_cpu_port(ds, p) ? dsa_user_ports(ds) : - BIT(dsa_to_port(ds, p)->cpu_dp->index))); + ret = reg_write(priv, addr, PORT_VLAN_MAP, + ((p & 0xf) << PORT_VLAN_MAP_DBNUM_SHIFT) | + (dsa_is_cpu_port(priv->ds, p) ? + dsa_user_ports(priv->ds) : + BIT(dsa_to_port(priv->ds, p)->cpu_dp->index))); + if (ret) + return ret; /* Port Association Vector: when learning source addresses * of packets, add the address to the address database using * a port bitmap that has only the bit for this port set and * the other bits clear. */ - REG_WRITE(addr, PORT_ASSOC_VECTOR, BIT(p)); - - return 0; + return reg_write(priv, addr, PORT_ASSOC_VECTOR, BIT(p)); } -static int mv88e6060_setup_addr(struct dsa_switch *ds) +static int mv88e6060_setup_addr(struct mv88e6060_priv *priv) { u8 addr[ETH_ALEN]; + int ret; u16 val; eth_random_addr(addr); @@ -199,34 +167,43 @@ static int mv88e6060_setup_addr(struct dsa_switch *ds) */ val &= 0xfeff; - REG_WRITE(REG_GLOBAL, GLOBAL_MAC_01, val); - REG_WRITE(REG_GLOBAL, GLOBAL_MAC_23, (addr[2] << 8) | addr[3]); - REG_WRITE(REG_GLOBAL, GLOBAL_MAC_45, (addr[4] << 8) | addr[5]); + ret = reg_write(priv, REG_GLOBAL, GLOBAL_MAC_01, val); + if (ret) + return ret; + + ret = reg_write(priv, REG_GLOBAL, GLOBAL_MAC_23, + (addr[2] << 8) | addr[3]); + if (ret) + return ret; - return 0; + return reg_write(priv, REG_GLOBAL, GLOBAL_MAC_45, + (addr[4] << 8) | addr[5]); } static int mv88e6060_setup(struct dsa_switch *ds) { + struct mv88e6060_priv *priv = ds->priv; int ret; int i; - ret = mv88e6060_switch_reset(ds); + priv->ds = ds; + + ret = mv88e6060_switch_reset(priv); if (ret < 0) return ret; /* @@@ initialise atu */ - ret = mv88e6060_setup_global(ds); + ret = mv88e6060_setup_global(priv); if (ret < 0) return ret; - ret = mv88e6060_setup_addr(ds); + ret = mv88e6060_setup_addr(priv); if (ret < 0) return ret; for (i = 0; i < MV88E6060_PORTS; i++) { - ret = mv88e6060_setup_port(ds, i); + ret = mv88e6060_setup_port(priv, i); if (ret < 0) return ret; } @@ -243,51 +220,93 @@ static int mv88e6060_port_to_phy_addr(int port) static int mv88e6060_phy_read(struct dsa_switch *ds, int port, int regnum) { + struct mv88e6060_priv *priv = ds->priv; int addr; addr = mv88e6060_port_to_phy_addr(port); if (addr == -1) return 0xffff; - return reg_read(ds, addr, regnum); + return reg_read(priv, addr, regnum); } static int mv88e6060_phy_write(struct dsa_switch *ds, int port, int regnum, u16 val) { + struct mv88e6060_priv *priv = ds->priv; int addr; addr = mv88e6060_port_to_phy_addr(port); if (addr == -1) return 0xffff; - return reg_write(ds, addr, regnum, val); + return reg_write(priv, addr, regnum, val); } static const struct dsa_switch_ops mv88e6060_switch_ops = { .get_tag_protocol = mv88e6060_get_tag_protocol, - .probe = mv88e6060_drv_probe, .setup = mv88e6060_setup, .phy_read = mv88e6060_phy_read, .phy_write = mv88e6060_phy_write, }; -static struct dsa_switch_driver mv88e6060_switch_drv = { - .ops = &mv88e6060_switch_ops, -}; - -static int __init mv88e6060_init(void) +static int mv88e6060_probe(struct mdio_device *mdiodev) { - register_switch_driver(&mv88e6060_switch_drv); - return 0; + struct device *dev = &mdiodev->dev; + struct mv88e6060_priv *priv; + struct dsa_switch *ds; + const char *name; + + priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL); + if (!priv) + return -ENOMEM; + + priv->bus = mdiodev->bus; + priv->sw_addr = mdiodev->addr; + + name = mv88e6060_get_name(priv->bus, priv->sw_addr); + if (!name) + return -ENODEV; + + dev_info(dev, "switch %s detected\n", name); + + ds = dsa_switch_alloc(dev, MV88E6060_PORTS); + if (!ds) + return -ENOMEM; + + ds->priv = priv; + ds->dev = dev; + ds->ops = &mv88e6060_switch_ops; + + dev_set_drvdata(dev, ds); + + return dsa_register_switch(ds); } -module_init(mv88e6060_init); -static void __exit mv88e6060_cleanup(void) +static void mv88e6060_remove(struct mdio_device *mdiodev) { - unregister_switch_driver(&mv88e6060_switch_drv); + struct dsa_switch *ds = dev_get_drvdata(&mdiodev->dev); + + dsa_unregister_switch(ds); } -module_exit(mv88e6060_cleanup); + +static const struct of_device_id mv88e6060_of_match[] = { + { + .compatible = "marvell,mv88e6060", + }, + { /* sentinel */ }, +}; + +static struct mdio_driver mv88e6060_driver = { + .probe = mv88e6060_probe, + .remove = mv88e6060_remove, + .mdiodrv.driver = { + .name = "mv88e6060", + .of_match_table = mv88e6060_of_match, + }, +}; + +mdio_module_driver(mv88e6060_driver); MODULE_AUTHOR("Lennert Buytenhek "); MODULE_DESCRIPTION("Driver for Marvell 88E6060 ethernet switch chip"); diff --git a/drivers/net/dsa/mv88e6060.h b/drivers/net/dsa/mv88e6060.h index 10249bd16292aadac4f0dc5a835514b0de0c0fc6..c0e7a0f2fb6a2ccd284449daa46d5085750b6375 100644 --- a/drivers/net/dsa/mv88e6060.h +++ b/drivers/net/dsa/mv88e6060.h @@ -117,6 +117,7 @@ struct mv88e6060_priv { */ struct mii_bus *bus; int sw_addr; + struct dsa_switch *ds; }; #endif diff --git a/drivers/net/dsa/mv88e6xxx/Makefile b/drivers/net/dsa/mv88e6xxx/Makefile index 50de304abe2fe80520d3475f4f70d53db25eba99..e85755dde90b98d503bb3ad225956d323b7ffb2a 100644 --- a/drivers/net/dsa/mv88e6xxx/Makefile +++ b/drivers/net/dsa/mv88e6xxx/Makefile @@ -12,3 +12,4 @@ mv88e6xxx-objs += phy.o mv88e6xxx-objs += port.o mv88e6xxx-$(CONFIG_NET_DSA_MV88E6XXX_PTP) += ptp.o mv88e6xxx-objs += serdes.o +mv88e6xxx-objs += smi.o diff --git a/drivers/net/dsa/mv88e6xxx/chip.c b/drivers/net/dsa/mv88e6xxx/chip.c index f4e2db44ad918ad3a01f0e6ffc5ac2707fd81b72..28414db979b0594aa266783e08e7db4036174e1a 100644 --- a/drivers/net/dsa/mv88e6xxx/chip.c +++ b/drivers/net/dsa/mv88e6xxx/chip.c @@ -43,6 +43,7 @@ #include "port.h" #include "ptp.h" #include "serdes.h" +#include "smi.h" static void assert_reg_lock(struct mv88e6xxx_chip *chip) { @@ -52,149 +53,6 @@ static void assert_reg_lock(struct mv88e6xxx_chip *chip) } } -/* The switch ADDR[4:1] configuration pins define the chip SMI device address - * (ADDR[0] is always zero, thus only even SMI addresses can be strapped). - * - * When ADDR is all zero, the chip uses Single-chip Addressing Mode, assuming it - * is the only device connected to the SMI master. In this mode it responds to - * all 32 possible SMI addresses, and thus maps directly the internal devices. - * - * When ADDR is non-zero, the chip uses Multi-chip Addressing Mode, allowing - * multiple devices to share the SMI interface. In this mode it responds to only - * 2 registers, used to indirectly access the internal SMI devices. - */ - -static int mv88e6xxx_smi_read(struct mv88e6xxx_chip *chip, - int addr, int reg, u16 *val) -{ - if (!chip->smi_ops) - return -EOPNOTSUPP; - - return chip->smi_ops->read(chip, addr, reg, val); -} - -static int mv88e6xxx_smi_write(struct mv88e6xxx_chip *chip, - int addr, int reg, u16 val) -{ - if (!chip->smi_ops) - return -EOPNOTSUPP; - - return chip->smi_ops->write(chip, addr, reg, val); -} - -static int mv88e6xxx_smi_single_chip_read(struct mv88e6xxx_chip *chip, - int addr, int reg, u16 *val) -{ - int ret; - - ret = mdiobus_read_nested(chip->bus, addr, reg); - if (ret < 0) - return ret; - - *val = ret & 0xffff; - - return 0; -} - -static int mv88e6xxx_smi_single_chip_write(struct mv88e6xxx_chip *chip, - int addr, int reg, u16 val) -{ - int ret; - - ret = mdiobus_write_nested(chip->bus, addr, reg, val); - if (ret < 0) - return ret; - - return 0; -} - -static const struct mv88e6xxx_bus_ops mv88e6xxx_smi_single_chip_ops = { - .read = mv88e6xxx_smi_single_chip_read, - .write = mv88e6xxx_smi_single_chip_write, -}; - -static int mv88e6xxx_smi_multi_chip_wait(struct mv88e6xxx_chip *chip) -{ - int ret; - int i; - - for (i = 0; i < 16; i++) { - ret = mdiobus_read_nested(chip->bus, chip->sw_addr, SMI_CMD); - if (ret < 0) - return ret; - - if ((ret & SMI_CMD_BUSY) == 0) - return 0; - } - - return -ETIMEDOUT; -} - -static int mv88e6xxx_smi_multi_chip_read(struct mv88e6xxx_chip *chip, - int addr, int reg, u16 *val) -{ - int ret; - - /* Wait for the bus to become free. */ - ret = mv88e6xxx_smi_multi_chip_wait(chip); - if (ret < 0) - return ret; - - /* Transmit the read command. */ - ret = mdiobus_write_nested(chip->bus, chip->sw_addr, SMI_CMD, - SMI_CMD_OP_22_READ | (addr << 5) | reg); - if (ret < 0) - return ret; - - /* Wait for the read command to complete. */ - ret = mv88e6xxx_smi_multi_chip_wait(chip); - if (ret < 0) - return ret; - - /* Read the data. */ - ret = mdiobus_read_nested(chip->bus, chip->sw_addr, SMI_DATA); - if (ret < 0) - return ret; - - *val = ret & 0xffff; - - return 0; -} - -static int mv88e6xxx_smi_multi_chip_write(struct mv88e6xxx_chip *chip, - int addr, int reg, u16 val) -{ - int ret; - - /* Wait for the bus to become free. */ - ret = mv88e6xxx_smi_multi_chip_wait(chip); - if (ret < 0) - return ret; - - /* Transmit the data to write. */ - ret = mdiobus_write_nested(chip->bus, chip->sw_addr, SMI_DATA, val); - if (ret < 0) - return ret; - - /* Transmit the write command. */ - ret = mdiobus_write_nested(chip->bus, chip->sw_addr, SMI_CMD, - SMI_CMD_OP_22_WRITE | (addr << 5) | reg); - if (ret < 0) - return ret; - - /* Wait for the write command to complete. */ - ret = mv88e6xxx_smi_multi_chip_wait(chip); - if (ret < 0) - return ret; - - return 0; -} - -static const struct mv88e6xxx_bus_ops mv88e6xxx_smi_multi_chip_ops = { - .read = mv88e6xxx_smi_multi_chip_read, - .write = mv88e6xxx_smi_multi_chip_write, -}; - int mv88e6xxx_read(struct mv88e6xxx_chip *chip, int addr, int reg, u16 *val) { int err; @@ -553,11 +411,28 @@ int mv88e6xxx_port_setup_mac(struct mv88e6xxx_chip *chip, int port, int link, int speed, int duplex, int pause, phy_interface_t mode) { + struct phylink_link_state state; int err; if (!chip->info->ops->port_set_link) return 0; + if (!chip->info->ops->port_link_state) + return 0; + + err = chip->info->ops->port_link_state(chip, port, &state); + if (err) + return err; + + /* Has anything actually changed? We don't expect the + * interface mode to change without one of the other + * parameters also changing + */ + if (state.link == link && + state.speed == speed && + state.duplex == duplex) + return 0; + /* Port's MAC control must not be changed unless the link is down */ err = chip->info->ops->port_set_link(chip, port, 0); if (err) @@ -2411,6 +2286,9 @@ static void mv88e6xxx_port_disable(struct dsa_switch *ds, int port) mutex_lock(&chip->reg_lock); + if (mv88e6xxx_port_set_state(chip, port, BR_STATE_DISABLED)) + dev_err(chip->dev, "failed to disable port\n"); + if (chip->info->ops->serdes_irq_free) chip->info->ops->serdes_irq_free(chip, port); @@ -2579,8 +2457,18 @@ static int mv88e6xxx_setup(struct dsa_switch *ds) /* Setup Switch Port Registers */ for (i = 0; i < mv88e6xxx_num_ports(chip); i++) { - if (dsa_is_unused_port(ds, i)) + if (dsa_is_unused_port(ds, i)) { + err = mv88e6xxx_port_set_state(chip, i, + BR_STATE_DISABLED); + if (err) + goto unlock; + + err = mv88e6xxx_serdes_power(chip, i, false); + if (err) + goto unlock; + continue; + } err = mv88e6xxx_setup_port(chip, i); if (err) @@ -4615,30 +4503,6 @@ static struct mv88e6xxx_chip *mv88e6xxx_alloc_chip(struct device *dev) return chip; } -static int mv88e6xxx_smi_init(struct mv88e6xxx_chip *chip, - struct mii_bus *bus, int sw_addr) -{ - if (sw_addr == 0) - chip->smi_ops = &mv88e6xxx_smi_single_chip_ops; - else if (chip->info->multi_chip) - chip->smi_ops = &mv88e6xxx_smi_multi_chip_ops; - else - return -EINVAL; - - chip->bus = bus; - chip->sw_addr = sw_addr; - - return 0; -} - -static void mv88e6xxx_ports_cmode_init(struct mv88e6xxx_chip *chip) -{ - int i; - - for (i = 0; i < mv88e6xxx_num_ports(chip); i++) - chip->ports[i].cmode = MV88E6XXX_PORT_STS_CMODE_INVALID; -} - static enum dsa_tag_protocol mv88e6xxx_get_tag_protocol(struct dsa_switch *ds, int port) { @@ -4647,58 +4511,6 @@ static enum dsa_tag_protocol mv88e6xxx_get_tag_protocol(struct dsa_switch *ds, return chip->info->tag_protocol; } -#if IS_ENABLED(CONFIG_NET_DSA_LEGACY) -static const char *mv88e6xxx_drv_probe(struct device *dsa_dev, - struct device *host_dev, int sw_addr, - void **priv) -{ - struct mv88e6xxx_chip *chip; - struct mii_bus *bus; - int err; - - bus = dsa_host_dev_to_mii_bus(host_dev); - if (!bus) - return NULL; - - chip = mv88e6xxx_alloc_chip(dsa_dev); - if (!chip) - return NULL; - - /* Legacy SMI probing will only support chips similar to 88E6085 */ - chip->info = &mv88e6xxx_table[MV88E6085]; - - err = mv88e6xxx_smi_init(chip, bus, sw_addr); - if (err) - goto free; - - err = mv88e6xxx_detect(chip); - if (err) - goto free; - - mv88e6xxx_ports_cmode_init(chip); - - mutex_lock(&chip->reg_lock); - err = mv88e6xxx_switch_reset(chip); - mutex_unlock(&chip->reg_lock); - if (err) - goto free; - - mv88e6xxx_phy_init(chip); - - err = mv88e6xxx_mdios_register(chip, NULL); - if (err) - goto free; - - *priv = chip; - - return chip->info->name; -free: - devm_kfree(dsa_dev, chip); - - return NULL; -} -#endif - static int mv88e6xxx_port_mdb_prepare(struct dsa_switch *ds, int port, const struct switchdev_obj_port_mdb *mdb) { @@ -4753,9 +4565,6 @@ static int mv88e6xxx_port_egress_floods(struct dsa_switch *ds, int port, } static const struct dsa_switch_ops mv88e6xxx_switch_ops = { -#if IS_ENABLED(CONFIG_NET_DSA_LEGACY) - .probe = mv88e6xxx_drv_probe, -#endif .get_tag_protocol = mv88e6xxx_get_tag_protocol, .setup = mv88e6xxx_setup, .adjust_link = mv88e6xxx_adjust_link, @@ -4801,10 +4610,6 @@ static const struct dsa_switch_ops mv88e6xxx_switch_ops = { .get_ts_info = mv88e6xxx_get_ts_info, }; -static struct dsa_switch_driver mv88e6xxx_switch_drv = { - .ops = &mv88e6xxx_switch_ops, -}; - static int mv88e6xxx_register_switch(struct mv88e6xxx_chip *chip) { struct device *dev = chip->dev; @@ -4915,7 +4720,6 @@ static int mv88e6xxx_probe(struct mdio_device *mdiodev) if (err) goto out; - mv88e6xxx_ports_cmode_init(chip); mv88e6xxx_phy_init(chip); if (chip->info->ops->get_eeprom) { @@ -4932,12 +4736,17 @@ static int mv88e6xxx_probe(struct mdio_device *mdiodev) if (err) goto out; - chip->irq = of_irq_get(np, 0); - if (chip->irq == -EPROBE_DEFER) { - err = chip->irq; - goto out; + if (np) { + chip->irq = of_irq_get(np, 0); + if (chip->irq == -EPROBE_DEFER) { + err = chip->irq; + goto out; + } } + if (pdata) + chip->irq = pdata->irq; + /* Has to be performed before the MDIO bus is created, because * the PHYs will link their interrupts to these interrupt * controllers @@ -5047,19 +4856,7 @@ static struct mdio_driver mv88e6xxx_driver = { }, }; -static int __init mv88e6xxx_init(void) -{ - register_switch_driver(&mv88e6xxx_switch_drv); - return mdio_driver_register(&mv88e6xxx_driver); -} -module_init(mv88e6xxx_init); - -static void __exit mv88e6xxx_cleanup(void) -{ - mdio_driver_unregister(&mv88e6xxx_driver); - unregister_switch_driver(&mv88e6xxx_switch_drv); -} -module_exit(mv88e6xxx_cleanup); +mdio_module_driver(mv88e6xxx_driver); MODULE_AUTHOR("Lennert Buytenhek "); MODULE_DESCRIPTION("Driver for Marvell 88E6XXX ethernet switch chips"); diff --git a/drivers/net/dsa/mv88e6xxx/chip.h b/drivers/net/dsa/mv88e6xxx/chip.h index 19c07dff044077cc057a3cb91fea5d4e5da15e76..faa3fa889f19a37de851f8e9b97da16e1238e83d 100644 --- a/drivers/net/dsa/mv88e6xxx/chip.h +++ b/drivers/net/dsa/mv88e6xxx/chip.h @@ -21,17 +21,6 @@ #include #include -#define SMI_CMD 0x00 -#define SMI_CMD_BUSY BIT(15) -#define SMI_CMD_CLAUSE_22 BIT(12) -#define SMI_CMD_OP_22_WRITE ((1 << 10) | SMI_CMD_BUSY | SMI_CMD_CLAUSE_22) -#define SMI_CMD_OP_22_READ ((2 << 10) | SMI_CMD_BUSY | SMI_CMD_CLAUSE_22) -#define SMI_CMD_OP_45_WRITE_ADDR ((0 << 10) | SMI_CMD_BUSY) -#define SMI_CMD_OP_45_WRITE_DATA ((1 << 10) | SMI_CMD_BUSY) -#define SMI_CMD_OP_45_READ_DATA ((2 << 10) | SMI_CMD_BUSY) -#define SMI_CMD_OP_45_READ_DATA_INC ((3 << 10) | SMI_CMD_BUSY) -#define SMI_DATA 0x01 - #define MV88E6XXX_N_FID 4096 /* PVT limits for 4-bit port and 5-bit switch */ diff --git a/drivers/net/dsa/mv88e6xxx/port.h b/drivers/net/dsa/mv88e6xxx/port.h index c7bed263a0f456c5c6908d0ce04c816c53a76b79..39c85e98fb92bc0a0834d8551a7700df548f6d62 100644 --- a/drivers/net/dsa/mv88e6xxx/port.h +++ b/drivers/net/dsa/mv88e6xxx/port.h @@ -52,7 +52,6 @@ #define MV88E6185_PORT_STS_CMODE_1000BASE_X 0x0005 #define MV88E6185_PORT_STS_CMODE_PHY 0x0006 #define MV88E6185_PORT_STS_CMODE_DISABLED 0x0007 -#define MV88E6XXX_PORT_STS_CMODE_INVALID 0xff /* Offset 0x01: MAC (or PCS or Physical) Control Register */ #define MV88E6XXX_PORT_MAC_CTL 0x01 diff --git a/drivers/net/dsa/mv88e6xxx/smi.c b/drivers/net/dsa/mv88e6xxx/smi.c new file mode 100644 index 0000000000000000000000000000000000000000..96f7d2685bdc541390c3fda7e0252c79649ffd99 --- /dev/null +++ b/drivers/net/dsa/mv88e6xxx/smi.c @@ -0,0 +1,158 @@ +/* + * Marvell 88E6xxx System Management Interface (SMI) support + * + * Copyright (c) 2008 Marvell Semiconductor + * + * Copyright (c) 2019 Vivien Didelot + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + */ + +#include "chip.h" +#include "smi.h" + +/* The switch ADDR[4:1] configuration pins define the chip SMI device address + * (ADDR[0] is always zero, thus only even SMI addresses can be strapped). + * + * When ADDR is all zero, the chip uses Single-chip Addressing Mode, assuming it + * is the only device connected to the SMI master. In this mode it responds to + * all 32 possible SMI addresses, and thus maps directly the internal devices. + * + * When ADDR is non-zero, the chip uses Multi-chip Addressing Mode, allowing + * multiple devices to share the SMI interface. In this mode it responds to only + * 2 registers, used to indirectly access the internal SMI devices. + */ + +static int mv88e6xxx_smi_direct_read(struct mv88e6xxx_chip *chip, + int dev, int reg, u16 *data) +{ + int ret; + + ret = mdiobus_read_nested(chip->bus, dev, reg); + if (ret < 0) + return ret; + + *data = ret & 0xffff; + + return 0; +} + +static int mv88e6xxx_smi_direct_write(struct mv88e6xxx_chip *chip, + int dev, int reg, u16 data) +{ + int ret; + + ret = mdiobus_write_nested(chip->bus, dev, reg, data); + if (ret < 0) + return ret; + + return 0; +} + +static int mv88e6xxx_smi_direct_wait(struct mv88e6xxx_chip *chip, + int dev, int reg, int bit, int val) +{ + u16 data; + int err; + int i; + + for (i = 0; i < 16; i++) { + err = mv88e6xxx_smi_direct_read(chip, dev, reg, &data); + if (err) + return err; + + if (!!(data >> bit) == !!val) + return 0; + } + + return -ETIMEDOUT; +} + +static const struct mv88e6xxx_bus_ops mv88e6xxx_smi_direct_ops = { + .read = mv88e6xxx_smi_direct_read, + .write = mv88e6xxx_smi_direct_write, +}; + +/* Offset 0x00: SMI Command Register + * Offset 0x01: SMI Data Register + */ + +static int mv88e6xxx_smi_indirect_read(struct mv88e6xxx_chip *chip, + int dev, int reg, u16 *data) +{ + int err; + + err = mv88e6xxx_smi_direct_wait(chip, chip->sw_addr, + MV88E6XXX_SMI_CMD, 15, 0); + if (err) + return err; + + err = mv88e6xxx_smi_direct_write(chip, chip->sw_addr, + MV88E6XXX_SMI_CMD, + MV88E6XXX_SMI_CMD_BUSY | + MV88E6XXX_SMI_CMD_MODE_22 | + MV88E6XXX_SMI_CMD_OP_22_READ | + (dev << 5) | reg); + if (err) + return err; + + err = mv88e6xxx_smi_direct_wait(chip, chip->sw_addr, + MV88E6XXX_SMI_CMD, 15, 0); + if (err) + return err; + + return mv88e6xxx_smi_direct_read(chip, chip->sw_addr, + MV88E6XXX_SMI_DATA, data); +} + +static int mv88e6xxx_smi_indirect_write(struct mv88e6xxx_chip *chip, + int dev, int reg, u16 data) +{ + int err; + + err = mv88e6xxx_smi_direct_wait(chip, chip->sw_addr, + MV88E6XXX_SMI_CMD, 15, 0); + if (err) + return err; + + err = mv88e6xxx_smi_direct_write(chip, chip->sw_addr, + MV88E6XXX_SMI_DATA, data); + if (err) + return err; + + err = mv88e6xxx_smi_direct_write(chip, chip->sw_addr, + MV88E6XXX_SMI_CMD, + MV88E6XXX_SMI_CMD_BUSY | + MV88E6XXX_SMI_CMD_MODE_22 | + MV88E6XXX_SMI_CMD_OP_22_WRITE | + (dev << 5) | reg); + if (err) + return err; + + return mv88e6xxx_smi_direct_wait(chip, chip->sw_addr, + MV88E6XXX_SMI_CMD, 15, 0); +} + +static const struct mv88e6xxx_bus_ops mv88e6xxx_smi_indirect_ops = { + .read = mv88e6xxx_smi_indirect_read, + .write = mv88e6xxx_smi_indirect_write, +}; + +int mv88e6xxx_smi_init(struct mv88e6xxx_chip *chip, + struct mii_bus *bus, int sw_addr) +{ + if (sw_addr == 0) + chip->smi_ops = &mv88e6xxx_smi_direct_ops; + else if (chip->info->multi_chip) + chip->smi_ops = &mv88e6xxx_smi_indirect_ops; + else + return -EINVAL; + + chip->bus = bus; + chip->sw_addr = sw_addr; + + return 0; +} diff --git a/drivers/net/dsa/mv88e6xxx/smi.h b/drivers/net/dsa/mv88e6xxx/smi.h new file mode 100644 index 0000000000000000000000000000000000000000..35e6403b65dcb49d37308d0473648249caf855a7 --- /dev/null +++ b/drivers/net/dsa/mv88e6xxx/smi.h @@ -0,0 +1,59 @@ +/* + * Marvell 88E6xxx System Management Interface (SMI) support + * + * Copyright (c) 2008 Marvell Semiconductor + * + * Copyright (c) 2019 Vivien Didelot + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + */ + +#ifndef _MV88E6XXX_SMI_H +#define _MV88E6XXX_SMI_H + +#include "chip.h" + +/* Offset 0x00: SMI Command Register */ +#define MV88E6XXX_SMI_CMD 0x00 +#define MV88E6XXX_SMI_CMD_BUSY 0x8000 +#define MV88E6XXX_SMI_CMD_MODE_MASK 0x1000 +#define MV88E6XXX_SMI_CMD_MODE_45 0x0000 +#define MV88E6XXX_SMI_CMD_MODE_22 0x1000 +#define MV88E6XXX_SMI_CMD_OP_MASK 0x0c00 +#define MV88E6XXX_SMI_CMD_OP_22_WRITE 0x0400 +#define MV88E6XXX_SMI_CMD_OP_22_READ 0x0800 +#define MV88E6XXX_SMI_CMD_OP_45_WRITE_ADDR 0x0000 +#define MV88E6XXX_SMI_CMD_OP_45_WRITE_DATA 0x0400 +#define MV88E6XXX_SMI_CMD_OP_45_READ_DATA 0x0800 +#define MV88E6XXX_SMI_CMD_OP_45_READ_DATA_INC 0x0c00 +#define MV88E6XXX_SMI_CMD_DEV_ADDR_MASK 0x003e +#define MV88E6XXX_SMI_CMD_REG_ADDR_MASK 0x001f + +/* Offset 0x01: SMI Data Register */ +#define MV88E6XXX_SMI_DATA 0x01 + +int mv88e6xxx_smi_init(struct mv88e6xxx_chip *chip, + struct mii_bus *bus, int sw_addr); + +static inline int mv88e6xxx_smi_read(struct mv88e6xxx_chip *chip, + int dev, int reg, u16 *data) +{ + if (chip->smi_ops && chip->smi_ops->read) + return chip->smi_ops->read(chip, dev, reg, data); + + return -EOPNOTSUPP; +} + +static inline int mv88e6xxx_smi_write(struct mv88e6xxx_chip *chip, + int dev, int reg, u16 data) +{ + if (chip->smi_ops && chip->smi_ops->write) + return chip->smi_ops->write(chip, dev, reg, data); + + return -EOPNOTSUPP; +} + +#endif /* _MV88E6XXX_SMI_H */ diff --git a/drivers/net/dsa/sja1105/Kconfig b/drivers/net/dsa/sja1105/Kconfig new file mode 100644 index 0000000000000000000000000000000000000000..757751a898199abba1d5f3becb740519fbfee98b --- /dev/null +++ b/drivers/net/dsa/sja1105/Kconfig @@ -0,0 +1,17 @@ +config NET_DSA_SJA1105 +tristate "NXP SJA1105 Ethernet switch family support" + depends on NET_DSA && SPI + select NET_DSA_TAG_SJA1105 + select PACKING + select CRC32 + help + This is the driver for the NXP SJA1105 automotive Ethernet switch + family. These are 5-port devices and are managed over an SPI + interface. Probing is handled based on OF bindings and so is the + linkage to phylib. The driver supports the following revisions: + - SJA1105E (Gen. 1, No TT-Ethernet) + - SJA1105T (Gen. 1, TT-Ethernet) + - SJA1105P (Gen. 2, No SGMII, No TT-Ethernet) + - SJA1105Q (Gen. 2, No SGMII, TT-Ethernet) + - SJA1105R (Gen. 2, SGMII, No TT-Ethernet) + - SJA1105S (Gen. 2, SGMII, TT-Ethernet) diff --git a/drivers/net/dsa/sja1105/Makefile b/drivers/net/dsa/sja1105/Makefile new file mode 100644 index 0000000000000000000000000000000000000000..1c2b55fec9596be6fdab9786ce4879efc38e2c40 --- /dev/null +++ b/drivers/net/dsa/sja1105/Makefile @@ -0,0 +1,9 @@ +obj-$(CONFIG_NET_DSA_SJA1105) += sja1105.o + +sja1105-objs := \ + sja1105_spi.o \ + sja1105_main.o \ + sja1105_ethtool.o \ + sja1105_clocking.o \ + sja1105_static_config.o \ + sja1105_dynamic_config.o \ diff --git a/drivers/net/dsa/sja1105/sja1105.h b/drivers/net/dsa/sja1105/sja1105.h new file mode 100644 index 0000000000000000000000000000000000000000..b043bfc408f254096d2c18c62fa84fcf71400f04 --- /dev/null +++ b/drivers/net/dsa/sja1105/sja1105.h @@ -0,0 +1,159 @@ +/* SPDX-License-Identifier: GPL-2.0 + * Copyright (c) 2018, Sensor-Technik Wiedemann GmbH + * Copyright (c) 2018-2019, Vladimir Oltean + */ +#ifndef _SJA1105_H +#define _SJA1105_H + +#include +#include +#include +#include "sja1105_static_config.h" + +#define SJA1105_NUM_PORTS 5 +#define SJA1105_NUM_TC 8 +#define SJA1105ET_FDB_BIN_SIZE 4 +/* The hardware value is in multiples of 10 ms. + * The passed parameter is in multiples of 1 ms. + */ +#define SJA1105_AGEING_TIME_MS(ms) ((ms) / 10) + +/* Keeps the different addresses between E/T and P/Q/R/S */ +struct sja1105_regs { + u64 device_id; + u64 prod_id; + u64 status; + u64 port_control; + u64 rgu; + u64 config; + u64 rmii_pll1; + u64 pad_mii_tx[SJA1105_NUM_PORTS]; + u64 cgu_idiv[SJA1105_NUM_PORTS]; + u64 rgmii_pad_mii_tx[SJA1105_NUM_PORTS]; + u64 mii_tx_clk[SJA1105_NUM_PORTS]; + u64 mii_rx_clk[SJA1105_NUM_PORTS]; + u64 mii_ext_tx_clk[SJA1105_NUM_PORTS]; + u64 mii_ext_rx_clk[SJA1105_NUM_PORTS]; + u64 rgmii_tx_clk[SJA1105_NUM_PORTS]; + u64 rmii_ref_clk[SJA1105_NUM_PORTS]; + u64 rmii_ext_tx_clk[SJA1105_NUM_PORTS]; + u64 mac[SJA1105_NUM_PORTS]; + u64 mac_hl1[SJA1105_NUM_PORTS]; + u64 mac_hl2[SJA1105_NUM_PORTS]; + u64 qlevel[SJA1105_NUM_PORTS]; +}; + +struct sja1105_info { + u64 device_id; + /* Needed for distinction between P and R, and between Q and S + * (since the parts with/without SGMII share the same + * switch core and device_id) + */ + u64 part_no; + const struct sja1105_dynamic_table_ops *dyn_ops; + const struct sja1105_table_ops *static_ops; + const struct sja1105_regs *regs; + int (*reset_cmd)(const void *ctx, const void *data); + int (*setup_rgmii_delay)(const void *ctx, int port); + const char *name; +}; + +struct sja1105_private { + struct sja1105_static_config static_config; + bool rgmii_rx_delay[SJA1105_NUM_PORTS]; + bool rgmii_tx_delay[SJA1105_NUM_PORTS]; + const struct sja1105_info *info; + struct gpio_desc *reset_gpio; + struct spi_device *spidev; + struct dsa_switch *ds; + struct sja1105_port ports[SJA1105_NUM_PORTS]; + /* Serializes transmission of management frames so that + * the switch doesn't confuse them with one another. + */ + struct mutex mgmt_lock; +}; + +#include "sja1105_dynamic_config.h" + +struct sja1105_spi_message { + u64 access; + u64 read_count; + u64 address; +}; + +typedef enum { + SPI_READ = 0, + SPI_WRITE = 1, +} sja1105_spi_rw_mode_t; + +/* From sja1105_spi.c */ +int sja1105_spi_send_packed_buf(const struct sja1105_private *priv, + sja1105_spi_rw_mode_t rw, u64 reg_addr, + void *packed_buf, size_t size_bytes); +int sja1105_spi_send_int(const struct sja1105_private *priv, + sja1105_spi_rw_mode_t rw, u64 reg_addr, + u64 *value, u64 size_bytes); +int sja1105_spi_send_long_packed_buf(const struct sja1105_private *priv, + sja1105_spi_rw_mode_t rw, u64 base_addr, + void *packed_buf, u64 buf_len); +int sja1105_static_config_upload(struct sja1105_private *priv); + +extern struct sja1105_info sja1105e_info; +extern struct sja1105_info sja1105t_info; +extern struct sja1105_info sja1105p_info; +extern struct sja1105_info sja1105q_info; +extern struct sja1105_info sja1105r_info; +extern struct sja1105_info sja1105s_info; + +/* From sja1105_clocking.c */ + +typedef enum { + XMII_MAC = 0, + XMII_PHY = 1, +} sja1105_mii_role_t; + +typedef enum { + XMII_MODE_MII = 0, + XMII_MODE_RMII = 1, + XMII_MODE_RGMII = 2, +} sja1105_phy_interface_t; + +typedef enum { + SJA1105_SPEED_10MBPS = 3, + SJA1105_SPEED_100MBPS = 2, + SJA1105_SPEED_1000MBPS = 1, + SJA1105_SPEED_AUTO = 0, +} sja1105_speed_t; + +int sja1105_clocking_setup_port(struct sja1105_private *priv, int port); +int sja1105_clocking_setup(struct sja1105_private *priv); + +/* From sja1105_ethtool.c */ +void sja1105_get_ethtool_stats(struct dsa_switch *ds, int port, u64 *data); +void sja1105_get_strings(struct dsa_switch *ds, int port, + u32 stringset, u8 *data); +int sja1105_get_sset_count(struct dsa_switch *ds, int port, int sset); + +/* From sja1105_dynamic_config.c */ +int sja1105_dynamic_config_read(struct sja1105_private *priv, + enum sja1105_blk_idx blk_idx, + int index, void *entry); +int sja1105_dynamic_config_write(struct sja1105_private *priv, + enum sja1105_blk_idx blk_idx, + int index, void *entry, bool keep); + +u8 sja1105_fdb_hash(struct sja1105_private *priv, const u8 *addr, u16 vid); + +/* Common implementations for the static and dynamic configs */ +size_t sja1105_l2_forwarding_entry_packing(void *buf, void *entry_ptr, + enum packing_op op); +size_t sja1105pqrs_l2_lookup_entry_packing(void *buf, void *entry_ptr, + enum packing_op op); +size_t sja1105et_l2_lookup_entry_packing(void *buf, void *entry_ptr, + enum packing_op op); +size_t sja1105_vlan_lookup_entry_packing(void *buf, void *entry_ptr, + enum packing_op op); +size_t sja1105pqrs_mac_config_entry_packing(void *buf, void *entry_ptr, + enum packing_op op); + +#endif diff --git a/drivers/net/dsa/sja1105/sja1105_clocking.c b/drivers/net/dsa/sja1105/sja1105_clocking.c new file mode 100644 index 0000000000000000000000000000000000000000..94bfe0ee50a8edc4265e8dae641c765dc436de64 --- /dev/null +++ b/drivers/net/dsa/sja1105/sja1105_clocking.c @@ -0,0 +1,601 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* Copyright (c) 2016-2018, NXP Semiconductors + * Copyright (c) 2018-2019, Vladimir Oltean + */ +#include +#include "sja1105.h" + +#define SJA1105_SIZE_CGU_CMD 4 + +struct sja1105_cfg_pad_mii_tx { + u64 d32_os; + u64 d32_ipud; + u64 d10_os; + u64 d10_ipud; + u64 ctrl_os; + u64 ctrl_ipud; + u64 clk_os; + u64 clk_ih; + u64 clk_ipud; +}; + +/* UM10944 Table 82. + * IDIV_0_C to IDIV_4_C control registers + * (addr. 10000Bh to 10000Fh) + */ +struct sja1105_cgu_idiv { + u64 clksrc; + u64 autoblock; + u64 idiv; + u64 pd; +}; + +/* PLL_1_C control register + * + * SJA1105 E/T: UM10944 Table 81 (address 10000Ah) + * SJA1105 P/Q/R/S: UM11040 Table 116 (address 10000Ah) + */ +struct sja1105_cgu_pll_ctrl { + u64 pllclksrc; + u64 msel; + u64 autoblock; + u64 psel; + u64 direct; + u64 fbsel; + u64 bypass; + u64 pd; +}; + +enum { + CLKSRC_MII0_TX_CLK = 0x00, + CLKSRC_MII0_RX_CLK = 0x01, + CLKSRC_MII1_TX_CLK = 0x02, + CLKSRC_MII1_RX_CLK = 0x03, + CLKSRC_MII2_TX_CLK = 0x04, + CLKSRC_MII2_RX_CLK = 0x05, + CLKSRC_MII3_TX_CLK = 0x06, + CLKSRC_MII3_RX_CLK = 0x07, + CLKSRC_MII4_TX_CLK = 0x08, + CLKSRC_MII4_RX_CLK = 0x09, + CLKSRC_PLL0 = 0x0B, + CLKSRC_PLL1 = 0x0E, + CLKSRC_IDIV0 = 0x11, + CLKSRC_IDIV1 = 0x12, + CLKSRC_IDIV2 = 0x13, + CLKSRC_IDIV3 = 0x14, + CLKSRC_IDIV4 = 0x15, +}; + +/* UM10944 Table 83. + * MIIx clock control registers 1 to 30 + * (addresses 100013h to 100035h) + */ +struct sja1105_cgu_mii_ctrl { + u64 clksrc; + u64 autoblock; + u64 pd; +}; + +static void sja1105_cgu_idiv_packing(void *buf, struct sja1105_cgu_idiv *idiv, + enum packing_op op) +{ + const int size = 4; + + sja1105_packing(buf, &idiv->clksrc, 28, 24, size, op); + sja1105_packing(buf, &idiv->autoblock, 11, 11, size, op); + sja1105_packing(buf, &idiv->idiv, 5, 2, size, op); + sja1105_packing(buf, &idiv->pd, 0, 0, size, op); +} + +static int sja1105_cgu_idiv_config(struct sja1105_private *priv, int port, + bool enabled, int factor) +{ + const struct sja1105_regs *regs = priv->info->regs; + struct device *dev = priv->ds->dev; + struct sja1105_cgu_idiv idiv; + u8 packed_buf[SJA1105_SIZE_CGU_CMD] = {0}; + + if (enabled && factor != 1 && factor != 10) { + dev_err(dev, "idiv factor must be 1 or 10\n"); + return -ERANGE; + } + + /* Payload for packed_buf */ + idiv.clksrc = 0x0A; /* 25MHz */ + idiv.autoblock = 1; /* Block clk automatically */ + idiv.idiv = factor - 1; /* Divide by 1 or 10 */ + idiv.pd = enabled ? 0 : 1; /* Power down? */ + sja1105_cgu_idiv_packing(packed_buf, &idiv, PACK); + + return sja1105_spi_send_packed_buf(priv, SPI_WRITE, + regs->cgu_idiv[port], packed_buf, + SJA1105_SIZE_CGU_CMD); +} + +static void +sja1105_cgu_mii_control_packing(void *buf, struct sja1105_cgu_mii_ctrl *cmd, + enum packing_op op) +{ + const int size = 4; + + sja1105_packing(buf, &cmd->clksrc, 28, 24, size, op); + sja1105_packing(buf, &cmd->autoblock, 11, 11, size, op); + sja1105_packing(buf, &cmd->pd, 0, 0, size, op); +} + +static int sja1105_cgu_mii_tx_clk_config(struct sja1105_private *priv, + int port, sja1105_mii_role_t role) +{ + const struct sja1105_regs *regs = priv->info->regs; + struct sja1105_cgu_mii_ctrl mii_tx_clk; + const int mac_clk_sources[] = { + CLKSRC_MII0_TX_CLK, + CLKSRC_MII1_TX_CLK, + CLKSRC_MII2_TX_CLK, + CLKSRC_MII3_TX_CLK, + CLKSRC_MII4_TX_CLK, + }; + const int phy_clk_sources[] = { + CLKSRC_IDIV0, + CLKSRC_IDIV1, + CLKSRC_IDIV2, + CLKSRC_IDIV3, + CLKSRC_IDIV4, + }; + u8 packed_buf[SJA1105_SIZE_CGU_CMD] = {0}; + int clksrc; + + if (role == XMII_MAC) + clksrc = mac_clk_sources[port]; + else + clksrc = phy_clk_sources[port]; + + /* Payload for packed_buf */ + mii_tx_clk.clksrc = clksrc; + mii_tx_clk.autoblock = 1; /* Autoblock clk while changing clksrc */ + mii_tx_clk.pd = 0; /* Power Down off => enabled */ + sja1105_cgu_mii_control_packing(packed_buf, &mii_tx_clk, PACK); + + return sja1105_spi_send_packed_buf(priv, SPI_WRITE, + regs->mii_tx_clk[port], packed_buf, + SJA1105_SIZE_CGU_CMD); +} + +static int +sja1105_cgu_mii_rx_clk_config(struct sja1105_private *priv, int port) +{ + const struct sja1105_regs *regs = priv->info->regs; + struct sja1105_cgu_mii_ctrl mii_rx_clk; + u8 packed_buf[SJA1105_SIZE_CGU_CMD] = {0}; + const int clk_sources[] = { + CLKSRC_MII0_RX_CLK, + CLKSRC_MII1_RX_CLK, + CLKSRC_MII2_RX_CLK, + CLKSRC_MII3_RX_CLK, + CLKSRC_MII4_RX_CLK, + }; + + /* Payload for packed_buf */ + mii_rx_clk.clksrc = clk_sources[port]; + mii_rx_clk.autoblock = 1; /* Autoblock clk while changing clksrc */ + mii_rx_clk.pd = 0; /* Power Down off => enabled */ + sja1105_cgu_mii_control_packing(packed_buf, &mii_rx_clk, PACK); + + return sja1105_spi_send_packed_buf(priv, SPI_WRITE, + regs->mii_rx_clk[port], packed_buf, + SJA1105_SIZE_CGU_CMD); +} + +static int +sja1105_cgu_mii_ext_tx_clk_config(struct sja1105_private *priv, int port) +{ + const struct sja1105_regs *regs = priv->info->regs; + struct sja1105_cgu_mii_ctrl mii_ext_tx_clk; + u8 packed_buf[SJA1105_SIZE_CGU_CMD] = {0}; + const int clk_sources[] = { + CLKSRC_IDIV0, + CLKSRC_IDIV1, + CLKSRC_IDIV2, + CLKSRC_IDIV3, + CLKSRC_IDIV4, + }; + + /* Payload for packed_buf */ + mii_ext_tx_clk.clksrc = clk_sources[port]; + mii_ext_tx_clk.autoblock = 1; /* Autoblock clk while changing clksrc */ + mii_ext_tx_clk.pd = 0; /* Power Down off => enabled */ + sja1105_cgu_mii_control_packing(packed_buf, &mii_ext_tx_clk, PACK); + + return sja1105_spi_send_packed_buf(priv, SPI_WRITE, + regs->mii_ext_tx_clk[port], + packed_buf, SJA1105_SIZE_CGU_CMD); +} + +static int +sja1105_cgu_mii_ext_rx_clk_config(struct sja1105_private *priv, int port) +{ + const struct sja1105_regs *regs = priv->info->regs; + struct sja1105_cgu_mii_ctrl mii_ext_rx_clk; + u8 packed_buf[SJA1105_SIZE_CGU_CMD] = {0}; + const int clk_sources[] = { + CLKSRC_IDIV0, + CLKSRC_IDIV1, + CLKSRC_IDIV2, + CLKSRC_IDIV3, + CLKSRC_IDIV4, + }; + + /* Payload for packed_buf */ + mii_ext_rx_clk.clksrc = clk_sources[port]; + mii_ext_rx_clk.autoblock = 1; /* Autoblock clk while changing clksrc */ + mii_ext_rx_clk.pd = 0; /* Power Down off => enabled */ + sja1105_cgu_mii_control_packing(packed_buf, &mii_ext_rx_clk, PACK); + + return sja1105_spi_send_packed_buf(priv, SPI_WRITE, + regs->mii_ext_rx_clk[port], + packed_buf, SJA1105_SIZE_CGU_CMD); +} + +static int sja1105_mii_clocking_setup(struct sja1105_private *priv, int port, + sja1105_mii_role_t role) +{ + struct device *dev = priv->ds->dev; + int rc; + + dev_dbg(dev, "Configuring MII-%s clocking\n", + (role == XMII_MAC) ? "MAC" : "PHY"); + /* If role is MAC, disable IDIV + * If role is PHY, enable IDIV and configure for 1/1 divider + */ + rc = sja1105_cgu_idiv_config(priv, port, (role == XMII_PHY), 1); + if (rc < 0) + return rc; + + /* Configure CLKSRC of MII_TX_CLK_n + * * If role is MAC, select TX_CLK_n + * * If role is PHY, select IDIV_n + */ + rc = sja1105_cgu_mii_tx_clk_config(priv, port, role); + if (rc < 0) + return rc; + + /* Configure CLKSRC of MII_RX_CLK_n + * Select RX_CLK_n + */ + rc = sja1105_cgu_mii_rx_clk_config(priv, port); + if (rc < 0) + return rc; + + if (role == XMII_PHY) { + /* Per MII spec, the PHY (which is us) drives the TX_CLK pin */ + + /* Configure CLKSRC of EXT_TX_CLK_n + * Select IDIV_n + */ + rc = sja1105_cgu_mii_ext_tx_clk_config(priv, port); + if (rc < 0) + return rc; + + /* Configure CLKSRC of EXT_RX_CLK_n + * Select IDIV_n + */ + rc = sja1105_cgu_mii_ext_rx_clk_config(priv, port); + if (rc < 0) + return rc; + } + return 0; +} + +static void +sja1105_cgu_pll_control_packing(void *buf, struct sja1105_cgu_pll_ctrl *cmd, + enum packing_op op) +{ + const int size = 4; + + sja1105_packing(buf, &cmd->pllclksrc, 28, 24, size, op); + sja1105_packing(buf, &cmd->msel, 23, 16, size, op); + sja1105_packing(buf, &cmd->autoblock, 11, 11, size, op); + sja1105_packing(buf, &cmd->psel, 9, 8, size, op); + sja1105_packing(buf, &cmd->direct, 7, 7, size, op); + sja1105_packing(buf, &cmd->fbsel, 6, 6, size, op); + sja1105_packing(buf, &cmd->bypass, 1, 1, size, op); + sja1105_packing(buf, &cmd->pd, 0, 0, size, op); +} + +static int sja1105_cgu_rgmii_tx_clk_config(struct sja1105_private *priv, + int port, sja1105_speed_t speed) +{ + const struct sja1105_regs *regs = priv->info->regs; + struct sja1105_cgu_mii_ctrl txc; + u8 packed_buf[SJA1105_SIZE_CGU_CMD] = {0}; + int clksrc; + + if (speed == SJA1105_SPEED_1000MBPS) { + clksrc = CLKSRC_PLL0; + } else { + int clk_sources[] = {CLKSRC_IDIV0, CLKSRC_IDIV1, CLKSRC_IDIV2, + CLKSRC_IDIV3, CLKSRC_IDIV4}; + clksrc = clk_sources[port]; + } + + /* RGMII: 125MHz for 1000, 25MHz for 100, 2.5MHz for 10 */ + txc.clksrc = clksrc; + /* Autoblock clk while changing clksrc */ + txc.autoblock = 1; + /* Power Down off => enabled */ + txc.pd = 0; + sja1105_cgu_mii_control_packing(packed_buf, &txc, PACK); + + return sja1105_spi_send_packed_buf(priv, SPI_WRITE, + regs->rgmii_tx_clk[port], + packed_buf, SJA1105_SIZE_CGU_CMD); +} + +/* AGU */ +static void +sja1105_cfg_pad_mii_tx_packing(void *buf, struct sja1105_cfg_pad_mii_tx *cmd, + enum packing_op op) +{ + const int size = 4; + + sja1105_packing(buf, &cmd->d32_os, 28, 27, size, op); + sja1105_packing(buf, &cmd->d32_ipud, 25, 24, size, op); + sja1105_packing(buf, &cmd->d10_os, 20, 19, size, op); + sja1105_packing(buf, &cmd->d10_ipud, 17, 16, size, op); + sja1105_packing(buf, &cmd->ctrl_os, 12, 11, size, op); + sja1105_packing(buf, &cmd->ctrl_ipud, 9, 8, size, op); + sja1105_packing(buf, &cmd->clk_os, 4, 3, size, op); + sja1105_packing(buf, &cmd->clk_ih, 2, 2, size, op); + sja1105_packing(buf, &cmd->clk_ipud, 1, 0, size, op); +} + +static int sja1105_rgmii_cfg_pad_tx_config(struct sja1105_private *priv, + int port) +{ + const struct sja1105_regs *regs = priv->info->regs; + struct sja1105_cfg_pad_mii_tx pad_mii_tx; + u8 packed_buf[SJA1105_SIZE_CGU_CMD] = {0}; + + /* Payload */ + pad_mii_tx.d32_os = 3; /* TXD[3:2] output stage: */ + /* high noise/high speed */ + pad_mii_tx.d10_os = 3; /* TXD[1:0] output stage: */ + /* high noise/high speed */ + pad_mii_tx.d32_ipud = 2; /* TXD[3:2] input stage: */ + /* plain input (default) */ + pad_mii_tx.d10_ipud = 2; /* TXD[1:0] input stage: */ + /* plain input (default) */ + pad_mii_tx.ctrl_os = 3; /* TX_CTL / TX_ER output stage */ + pad_mii_tx.ctrl_ipud = 2; /* TX_CTL / TX_ER input stage (default) */ + pad_mii_tx.clk_os = 3; /* TX_CLK output stage */ + pad_mii_tx.clk_ih = 0; /* TX_CLK input hysteresis (default) */ + pad_mii_tx.clk_ipud = 2; /* TX_CLK input stage (default) */ + sja1105_cfg_pad_mii_tx_packing(packed_buf, &pad_mii_tx, PACK); + + return sja1105_spi_send_packed_buf(priv, SPI_WRITE, + regs->rgmii_pad_mii_tx[port], + packed_buf, SJA1105_SIZE_CGU_CMD); +} + +static int sja1105_rgmii_clocking_setup(struct sja1105_private *priv, int port) +{ + struct device *dev = priv->ds->dev; + struct sja1105_mac_config_entry *mac; + sja1105_speed_t speed; + int rc; + + mac = priv->static_config.tables[BLK_IDX_MAC_CONFIG].entries; + speed = mac[port].speed; + + dev_dbg(dev, "Configuring port %d RGMII at speed %dMbps\n", + port, speed); + + switch (speed) { + case SJA1105_SPEED_1000MBPS: + /* 1000Mbps, IDIV disabled (125 MHz) */ + rc = sja1105_cgu_idiv_config(priv, port, false, 1); + break; + case SJA1105_SPEED_100MBPS: + /* 100Mbps, IDIV enabled, divide by 1 (25 MHz) */ + rc = sja1105_cgu_idiv_config(priv, port, true, 1); + break; + case SJA1105_SPEED_10MBPS: + /* 10Mbps, IDIV enabled, divide by 10 (2.5 MHz) */ + rc = sja1105_cgu_idiv_config(priv, port, true, 10); + break; + case SJA1105_SPEED_AUTO: + /* Skip CGU configuration if there is no speed available + * (e.g. link is not established yet) + */ + dev_dbg(dev, "Speed not available, skipping CGU config\n"); + return 0; + default: + rc = -EINVAL; + } + + if (rc < 0) { + dev_err(dev, "Failed to configure idiv\n"); + return rc; + } + rc = sja1105_cgu_rgmii_tx_clk_config(priv, port, speed); + if (rc < 0) { + dev_err(dev, "Failed to configure RGMII Tx clock\n"); + return rc; + } + rc = sja1105_rgmii_cfg_pad_tx_config(priv, port); + if (rc < 0) { + dev_err(dev, "Failed to configure Tx pad registers\n"); + return rc; + } + if (!priv->info->setup_rgmii_delay) + return 0; + + return priv->info->setup_rgmii_delay(priv, port); +} + +static int sja1105_cgu_rmii_ref_clk_config(struct sja1105_private *priv, + int port) +{ + const struct sja1105_regs *regs = priv->info->regs; + struct sja1105_cgu_mii_ctrl ref_clk; + u8 packed_buf[SJA1105_SIZE_CGU_CMD] = {0}; + const int clk_sources[] = { + CLKSRC_MII0_TX_CLK, + CLKSRC_MII1_TX_CLK, + CLKSRC_MII2_TX_CLK, + CLKSRC_MII3_TX_CLK, + CLKSRC_MII4_TX_CLK, + }; + + /* Payload for packed_buf */ + ref_clk.clksrc = clk_sources[port]; + ref_clk.autoblock = 1; /* Autoblock clk while changing clksrc */ + ref_clk.pd = 0; /* Power Down off => enabled */ + sja1105_cgu_mii_control_packing(packed_buf, &ref_clk, PACK); + + return sja1105_spi_send_packed_buf(priv, SPI_WRITE, + regs->rmii_ref_clk[port], + packed_buf, SJA1105_SIZE_CGU_CMD); +} + +static int +sja1105_cgu_rmii_ext_tx_clk_config(struct sja1105_private *priv, int port) +{ + const struct sja1105_regs *regs = priv->info->regs; + struct sja1105_cgu_mii_ctrl ext_tx_clk; + u8 packed_buf[SJA1105_SIZE_CGU_CMD] = {0}; + + /* Payload for packed_buf */ + ext_tx_clk.clksrc = CLKSRC_PLL1; + ext_tx_clk.autoblock = 1; /* Autoblock clk while changing clksrc */ + ext_tx_clk.pd = 0; /* Power Down off => enabled */ + sja1105_cgu_mii_control_packing(packed_buf, &ext_tx_clk, PACK); + + return sja1105_spi_send_packed_buf(priv, SPI_WRITE, + regs->rmii_ext_tx_clk[port], + packed_buf, SJA1105_SIZE_CGU_CMD); +} + +static int sja1105_cgu_rmii_pll_config(struct sja1105_private *priv) +{ + const struct sja1105_regs *regs = priv->info->regs; + u8 packed_buf[SJA1105_SIZE_CGU_CMD] = {0}; + struct sja1105_cgu_pll_ctrl pll = {0}; + struct device *dev = priv->ds->dev; + int rc; + + /* PLL1 must be enabled and output 50 Mhz. + * This is done by writing first 0x0A010941 to + * the PLL_1_C register and then deasserting + * power down (PD) 0x0A010940. + */ + + /* Step 1: PLL1 setup for 50Mhz */ + pll.pllclksrc = 0xA; + pll.msel = 0x1; + pll.autoblock = 0x1; + pll.psel = 0x1; + pll.direct = 0x0; + pll.fbsel = 0x1; + pll.bypass = 0x0; + pll.pd = 0x1; + + sja1105_cgu_pll_control_packing(packed_buf, &pll, PACK); + rc = sja1105_spi_send_packed_buf(priv, SPI_WRITE, regs->rmii_pll1, + packed_buf, SJA1105_SIZE_CGU_CMD); + if (rc < 0) { + dev_err(dev, "failed to configure PLL1 for 50MHz\n"); + return rc; + } + + /* Step 2: Enable PLL1 */ + pll.pd = 0x0; + + sja1105_cgu_pll_control_packing(packed_buf, &pll, PACK); + rc = sja1105_spi_send_packed_buf(priv, SPI_WRITE, regs->rmii_pll1, + packed_buf, SJA1105_SIZE_CGU_CMD); + if (rc < 0) { + dev_err(dev, "failed to enable PLL1\n"); + return rc; + } + return rc; +} + +static int sja1105_rmii_clocking_setup(struct sja1105_private *priv, int port, + sja1105_mii_role_t role) +{ + struct device *dev = priv->ds->dev; + int rc; + + dev_dbg(dev, "Configuring RMII-%s clocking\n", + (role == XMII_MAC) ? "MAC" : "PHY"); + /* AH1601.pdf chapter 2.5.1. Sources */ + if (role == XMII_MAC) { + /* Configure and enable PLL1 for 50Mhz output */ + rc = sja1105_cgu_rmii_pll_config(priv); + if (rc < 0) + return rc; + } + /* Disable IDIV for this port */ + rc = sja1105_cgu_idiv_config(priv, port, false, 1); + if (rc < 0) + return rc; + /* Source to sink mappings */ + rc = sja1105_cgu_rmii_ref_clk_config(priv, port); + if (rc < 0) + return rc; + if (role == XMII_MAC) { + rc = sja1105_cgu_rmii_ext_tx_clk_config(priv, port); + if (rc < 0) + return rc; + } + return 0; +} + +int sja1105_clocking_setup_port(struct sja1105_private *priv, int port) +{ + struct sja1105_xmii_params_entry *mii; + struct device *dev = priv->ds->dev; + sja1105_phy_interface_t phy_mode; + sja1105_mii_role_t role; + int rc; + + mii = priv->static_config.tables[BLK_IDX_XMII_PARAMS].entries; + + /* RGMII etc */ + phy_mode = mii->xmii_mode[port]; + /* MAC or PHY, for applicable types (not RGMII) */ + role = mii->phy_mac[port]; + + switch (phy_mode) { + case XMII_MODE_MII: + rc = sja1105_mii_clocking_setup(priv, port, role); + break; + case XMII_MODE_RMII: + rc = sja1105_rmii_clocking_setup(priv, port, role); + break; + case XMII_MODE_RGMII: + rc = sja1105_rgmii_clocking_setup(priv, port); + break; + default: + dev_err(dev, "Invalid interface mode specified: %d\n", + phy_mode); + return -EINVAL; + } + if (rc) + dev_err(dev, "Clocking setup for port %d failed: %d\n", + port, rc); + return rc; +} + +int sja1105_clocking_setup(struct sja1105_private *priv) +{ + int port, rc; + + for (port = 0; port < SJA1105_NUM_PORTS; port++) { + rc = sja1105_clocking_setup_port(priv, port); + if (rc < 0) + return rc; + } + return 0; +} diff --git a/drivers/net/dsa/sja1105/sja1105_dynamic_config.c b/drivers/net/dsa/sja1105/sja1105_dynamic_config.c new file mode 100644 index 0000000000000000000000000000000000000000..e73ab28bf63225770cef61056951f24ff73899cc --- /dev/null +++ b/drivers/net/dsa/sja1105/sja1105_dynamic_config.c @@ -0,0 +1,532 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2018-2019, Vladimir Oltean + */ +#include "sja1105.h" + +#define SJA1105_SIZE_DYN_CMD 4 + +#define SJA1105ET_SIZE_MAC_CONFIG_DYN_ENTRY \ + SJA1105_SIZE_DYN_CMD + +#define SJA1105ET_SIZE_L2_LOOKUP_DYN_CMD \ + (SJA1105_SIZE_DYN_CMD + SJA1105ET_SIZE_L2_LOOKUP_ENTRY) + +#define SJA1105PQRS_SIZE_L2_LOOKUP_DYN_CMD \ + (SJA1105_SIZE_DYN_CMD + SJA1105PQRS_SIZE_L2_LOOKUP_ENTRY) + +#define SJA1105_SIZE_VLAN_LOOKUP_DYN_CMD \ + (SJA1105_SIZE_DYN_CMD + 4 + SJA1105_SIZE_VLAN_LOOKUP_ENTRY) + +#define SJA1105_SIZE_L2_FORWARDING_DYN_CMD \ + (SJA1105_SIZE_DYN_CMD + SJA1105_SIZE_L2_FORWARDING_ENTRY) + +#define SJA1105ET_SIZE_MAC_CONFIG_DYN_CMD \ + (SJA1105_SIZE_DYN_CMD + SJA1105ET_SIZE_MAC_CONFIG_DYN_ENTRY) + +#define SJA1105PQRS_SIZE_MAC_CONFIG_DYN_CMD \ + (SJA1105_SIZE_DYN_CMD + SJA1105PQRS_SIZE_MAC_CONFIG_ENTRY) + +#define SJA1105ET_SIZE_L2_LOOKUP_PARAMS_DYN_CMD \ + SJA1105_SIZE_DYN_CMD + +#define SJA1105ET_SIZE_GENERAL_PARAMS_DYN_CMD \ + SJA1105_SIZE_DYN_CMD + +#define SJA1105_MAX_DYN_CMD_SIZE \ + SJA1105PQRS_SIZE_MAC_CONFIG_DYN_CMD + +static void +sja1105pqrs_l2_lookup_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd, + enum packing_op op) +{ + u8 *p = buf + SJA1105PQRS_SIZE_L2_LOOKUP_ENTRY; + const int size = SJA1105_SIZE_DYN_CMD; + + sja1105_packing(p, &cmd->valid, 31, 31, size, op); + sja1105_packing(p, &cmd->rdwrset, 30, 30, size, op); + sja1105_packing(p, &cmd->errors, 29, 29, size, op); + sja1105_packing(p, &cmd->valident, 27, 27, size, op); + /* Hack - The hardware takes the 'index' field within + * struct sja1105_l2_lookup_entry as the index on which this command + * will operate. However it will ignore everything else, so 'index' + * is logically part of command but physically part of entry. + * Populate the 'index' entry field from within the command callback, + * such that our API doesn't need to ask for a full-blown entry + * structure when e.g. a delete is requested. + */ + sja1105_packing(buf, &cmd->index, 29, 20, + SJA1105PQRS_SIZE_L2_LOOKUP_ENTRY, op); + /* TODO hostcmd */ +} + +static void +sja1105et_l2_lookup_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd, + enum packing_op op) +{ + u8 *p = buf + SJA1105ET_SIZE_L2_LOOKUP_ENTRY; + const int size = SJA1105_SIZE_DYN_CMD; + + sja1105_packing(p, &cmd->valid, 31, 31, size, op); + sja1105_packing(p, &cmd->rdwrset, 30, 30, size, op); + sja1105_packing(p, &cmd->errors, 29, 29, size, op); + sja1105_packing(p, &cmd->valident, 27, 27, size, op); + /* Hack - see comments above. */ + sja1105_packing(buf, &cmd->index, 29, 20, + SJA1105ET_SIZE_L2_LOOKUP_ENTRY, op); +} + +static void +sja1105et_mgmt_route_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd, + enum packing_op op) +{ + u8 *p = buf + SJA1105ET_SIZE_L2_LOOKUP_ENTRY; + u64 mgmtroute = 1; + + sja1105et_l2_lookup_cmd_packing(buf, cmd, op); + if (op == PACK) + sja1105_pack(p, &mgmtroute, 26, 26, SJA1105_SIZE_DYN_CMD); +} + +static size_t sja1105et_mgmt_route_entry_packing(void *buf, void *entry_ptr, + enum packing_op op) +{ + struct sja1105_mgmt_entry *entry = entry_ptr; + const size_t size = SJA1105ET_SIZE_L2_LOOKUP_ENTRY; + + /* UM10944: To specify if a PTP egress timestamp shall be captured on + * each port upon transmission of the frame, the LSB of VLANID in the + * ENTRY field provided by the host must be set. + * Bit 1 of VLANID then specifies the register where the timestamp for + * this port is stored in. + */ + sja1105_packing(buf, &entry->tsreg, 85, 85, size, op); + sja1105_packing(buf, &entry->takets, 84, 84, size, op); + sja1105_packing(buf, &entry->macaddr, 83, 36, size, op); + sja1105_packing(buf, &entry->destports, 35, 31, size, op); + sja1105_packing(buf, &entry->enfport, 30, 30, size, op); + return size; +} + +/* In E/T, entry is at addresses 0x27-0x28. There is a 4 byte gap at 0x29, + * and command is at 0x2a. Similarly in P/Q/R/S there is a 1 register gap + * between entry (0x2d, 0x2e) and command (0x30). + */ +static void +sja1105_vlan_lookup_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd, + enum packing_op op) +{ + u8 *p = buf + SJA1105_SIZE_VLAN_LOOKUP_ENTRY + 4; + const int size = SJA1105_SIZE_DYN_CMD; + + sja1105_packing(p, &cmd->valid, 31, 31, size, op); + sja1105_packing(p, &cmd->rdwrset, 30, 30, size, op); + sja1105_packing(p, &cmd->valident, 27, 27, size, op); + /* Hack - see comments above, applied for 'vlanid' field of + * struct sja1105_vlan_lookup_entry. + */ + sja1105_packing(buf, &cmd->index, 38, 27, + SJA1105_SIZE_VLAN_LOOKUP_ENTRY, op); +} + +static void +sja1105_l2_forwarding_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd, + enum packing_op op) +{ + u8 *p = buf + SJA1105_SIZE_L2_FORWARDING_ENTRY; + const int size = SJA1105_SIZE_DYN_CMD; + + sja1105_packing(p, &cmd->valid, 31, 31, size, op); + sja1105_packing(p, &cmd->errors, 30, 30, size, op); + sja1105_packing(p, &cmd->rdwrset, 29, 29, size, op); + sja1105_packing(p, &cmd->index, 4, 0, size, op); +} + +static void +sja1105et_mac_config_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd, + enum packing_op op) +{ + const int size = SJA1105_SIZE_DYN_CMD; + /* Yup, user manual definitions are reversed */ + u8 *reg1 = buf + 4; + + sja1105_packing(reg1, &cmd->valid, 31, 31, size, op); + sja1105_packing(reg1, &cmd->index, 26, 24, size, op); +} + +static size_t sja1105et_mac_config_entry_packing(void *buf, void *entry_ptr, + enum packing_op op) +{ + const int size = SJA1105ET_SIZE_MAC_CONFIG_DYN_ENTRY; + struct sja1105_mac_config_entry *entry = entry_ptr; + /* Yup, user manual definitions are reversed */ + u8 *reg1 = buf + 4; + u8 *reg2 = buf; + + sja1105_packing(reg1, &entry->speed, 30, 29, size, op); + sja1105_packing(reg1, &entry->drpdtag, 23, 23, size, op); + sja1105_packing(reg1, &entry->drpuntag, 22, 22, size, op); + sja1105_packing(reg1, &entry->retag, 21, 21, size, op); + sja1105_packing(reg1, &entry->dyn_learn, 20, 20, size, op); + sja1105_packing(reg1, &entry->egress, 19, 19, size, op); + sja1105_packing(reg1, &entry->ingress, 18, 18, size, op); + sja1105_packing(reg1, &entry->ing_mirr, 17, 17, size, op); + sja1105_packing(reg1, &entry->egr_mirr, 16, 16, size, op); + sja1105_packing(reg1, &entry->vlanprio, 14, 12, size, op); + sja1105_packing(reg1, &entry->vlanid, 11, 0, size, op); + sja1105_packing(reg2, &entry->tp_delin, 31, 16, size, op); + sja1105_packing(reg2, &entry->tp_delout, 15, 0, size, op); + /* MAC configuration table entries which can't be reconfigured: + * top, base, enabled, ifg, maxage, drpnona664 + */ + /* Bogus return value, not used anywhere */ + return 0; +} + +static void +sja1105pqrs_mac_config_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd, + enum packing_op op) +{ + const int size = SJA1105ET_SIZE_MAC_CONFIG_DYN_ENTRY; + u8 *p = buf + SJA1105PQRS_SIZE_MAC_CONFIG_ENTRY; + + sja1105_packing(p, &cmd->valid, 31, 31, size, op); + sja1105_packing(p, &cmd->errors, 30, 30, size, op); + sja1105_packing(p, &cmd->rdwrset, 29, 29, size, op); + sja1105_packing(p, &cmd->index, 2, 0, size, op); +} + +static void +sja1105et_l2_lookup_params_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd, + enum packing_op op) +{ + sja1105_packing(buf, &cmd->valid, 31, 31, + SJA1105ET_SIZE_L2_LOOKUP_PARAMS_DYN_CMD, op); +} + +static size_t +sja1105et_l2_lookup_params_entry_packing(void *buf, void *entry_ptr, + enum packing_op op) +{ + struct sja1105_l2_lookup_params_entry *entry = entry_ptr; + + sja1105_packing(buf, &entry->poly, 7, 0, + SJA1105ET_SIZE_L2_LOOKUP_PARAMS_DYN_CMD, op); + /* Bogus return value, not used anywhere */ + return 0; +} + +static void +sja1105et_general_params_cmd_packing(void *buf, struct sja1105_dyn_cmd *cmd, + enum packing_op op) +{ + const int size = SJA1105ET_SIZE_GENERAL_PARAMS_DYN_CMD; + + sja1105_packing(buf, &cmd->valid, 31, 31, size, op); + sja1105_packing(buf, &cmd->errors, 30, 30, size, op); +} + +static size_t +sja1105et_general_params_entry_packing(void *buf, void *entry_ptr, + enum packing_op op) +{ + struct sja1105_general_params_entry *entry = entry_ptr; + const int size = SJA1105ET_SIZE_GENERAL_PARAMS_DYN_CMD; + + sja1105_packing(buf, &entry->mirr_port, 2, 0, size, op); + /* Bogus return value, not used anywhere */ + return 0; +} + +#define OP_READ BIT(0) +#define OP_WRITE BIT(1) +#define OP_DEL BIT(2) + +/* SJA1105E/T: First generation */ +struct sja1105_dynamic_table_ops sja1105et_dyn_ops[BLK_IDX_MAX_DYN] = { + [BLK_IDX_L2_LOOKUP] = { + .entry_packing = sja1105et_l2_lookup_entry_packing, + .cmd_packing = sja1105et_l2_lookup_cmd_packing, + .access = (OP_READ | OP_WRITE | OP_DEL), + .max_entry_count = SJA1105_MAX_L2_LOOKUP_COUNT, + .packed_size = SJA1105ET_SIZE_L2_LOOKUP_DYN_CMD, + .addr = 0x20, + }, + [BLK_IDX_MGMT_ROUTE] = { + .entry_packing = sja1105et_mgmt_route_entry_packing, + .cmd_packing = sja1105et_mgmt_route_cmd_packing, + .access = (OP_READ | OP_WRITE), + .max_entry_count = SJA1105_NUM_PORTS, + .packed_size = SJA1105ET_SIZE_L2_LOOKUP_DYN_CMD, + .addr = 0x20, + }, + [BLK_IDX_L2_POLICING] = {0}, + [BLK_IDX_VLAN_LOOKUP] = { + .entry_packing = sja1105_vlan_lookup_entry_packing, + .cmd_packing = sja1105_vlan_lookup_cmd_packing, + .access = (OP_WRITE | OP_DEL), + .max_entry_count = SJA1105_MAX_VLAN_LOOKUP_COUNT, + .packed_size = SJA1105_SIZE_VLAN_LOOKUP_DYN_CMD, + .addr = 0x27, + }, + [BLK_IDX_L2_FORWARDING] = { + .entry_packing = sja1105_l2_forwarding_entry_packing, + .cmd_packing = sja1105_l2_forwarding_cmd_packing, + .max_entry_count = SJA1105_MAX_L2_FORWARDING_COUNT, + .access = OP_WRITE, + .packed_size = SJA1105_SIZE_L2_FORWARDING_DYN_CMD, + .addr = 0x24, + }, + [BLK_IDX_MAC_CONFIG] = { + .entry_packing = sja1105et_mac_config_entry_packing, + .cmd_packing = sja1105et_mac_config_cmd_packing, + .max_entry_count = SJA1105_MAX_MAC_CONFIG_COUNT, + .access = OP_WRITE, + .packed_size = SJA1105ET_SIZE_MAC_CONFIG_DYN_CMD, + .addr = 0x36, + }, + [BLK_IDX_L2_LOOKUP_PARAMS] = { + .entry_packing = sja1105et_l2_lookup_params_entry_packing, + .cmd_packing = sja1105et_l2_lookup_params_cmd_packing, + .max_entry_count = SJA1105_MAX_L2_LOOKUP_PARAMS_COUNT, + .access = OP_WRITE, + .packed_size = SJA1105ET_SIZE_L2_LOOKUP_PARAMS_DYN_CMD, + .addr = 0x38, + }, + [BLK_IDX_L2_FORWARDING_PARAMS] = {0}, + [BLK_IDX_GENERAL_PARAMS] = { + .entry_packing = sja1105et_general_params_entry_packing, + .cmd_packing = sja1105et_general_params_cmd_packing, + .max_entry_count = SJA1105_MAX_GENERAL_PARAMS_COUNT, + .access = OP_WRITE, + .packed_size = SJA1105ET_SIZE_GENERAL_PARAMS_DYN_CMD, + .addr = 0x34, + }, + [BLK_IDX_XMII_PARAMS] = {0}, +}; + +/* SJA1105P/Q/R/S: Second generation: TODO */ +struct sja1105_dynamic_table_ops sja1105pqrs_dyn_ops[BLK_IDX_MAX_DYN] = { + [BLK_IDX_L2_LOOKUP] = { + .entry_packing = sja1105pqrs_l2_lookup_entry_packing, + .cmd_packing = sja1105pqrs_l2_lookup_cmd_packing, + .access = (OP_READ | OP_WRITE | OP_DEL), + .max_entry_count = SJA1105_MAX_L2_LOOKUP_COUNT, + .packed_size = SJA1105ET_SIZE_L2_LOOKUP_DYN_CMD, + .addr = 0x24, + }, + [BLK_IDX_L2_POLICING] = {0}, + [BLK_IDX_VLAN_LOOKUP] = { + .entry_packing = sja1105_vlan_lookup_entry_packing, + .cmd_packing = sja1105_vlan_lookup_cmd_packing, + .access = (OP_READ | OP_WRITE | OP_DEL), + .max_entry_count = SJA1105_MAX_VLAN_LOOKUP_COUNT, + .packed_size = SJA1105_SIZE_VLAN_LOOKUP_DYN_CMD, + .addr = 0x2D, + }, + [BLK_IDX_L2_FORWARDING] = { + .entry_packing = sja1105_l2_forwarding_entry_packing, + .cmd_packing = sja1105_l2_forwarding_cmd_packing, + .max_entry_count = SJA1105_MAX_L2_FORWARDING_COUNT, + .access = OP_WRITE, + .packed_size = SJA1105_SIZE_L2_FORWARDING_DYN_CMD, + .addr = 0x2A, + }, + [BLK_IDX_MAC_CONFIG] = { + .entry_packing = sja1105pqrs_mac_config_entry_packing, + .cmd_packing = sja1105pqrs_mac_config_cmd_packing, + .max_entry_count = SJA1105_MAX_MAC_CONFIG_COUNT, + .access = (OP_READ | OP_WRITE), + .packed_size = SJA1105PQRS_SIZE_MAC_CONFIG_DYN_CMD, + .addr = 0x4B, + }, + [BLK_IDX_L2_LOOKUP_PARAMS] = { + .entry_packing = sja1105et_l2_lookup_params_entry_packing, + .cmd_packing = sja1105et_l2_lookup_params_cmd_packing, + .max_entry_count = SJA1105_MAX_L2_LOOKUP_PARAMS_COUNT, + .access = (OP_READ | OP_WRITE), + .packed_size = SJA1105ET_SIZE_L2_LOOKUP_PARAMS_DYN_CMD, + .addr = 0x38, + }, + [BLK_IDX_L2_FORWARDING_PARAMS] = {0}, + [BLK_IDX_GENERAL_PARAMS] = { + .entry_packing = sja1105et_general_params_entry_packing, + .cmd_packing = sja1105et_general_params_cmd_packing, + .max_entry_count = SJA1105_MAX_GENERAL_PARAMS_COUNT, + .access = OP_WRITE, + .packed_size = SJA1105ET_SIZE_GENERAL_PARAMS_DYN_CMD, + .addr = 0x34, + }, + [BLK_IDX_XMII_PARAMS] = {0}, +}; + +int sja1105_dynamic_config_read(struct sja1105_private *priv, + enum sja1105_blk_idx blk_idx, + int index, void *entry) +{ + const struct sja1105_dynamic_table_ops *ops; + struct sja1105_dyn_cmd cmd = {0}; + /* SPI payload buffer */ + u8 packed_buf[SJA1105_MAX_DYN_CMD_SIZE] = {0}; + int retries = 3; + int rc; + + if (blk_idx >= BLK_IDX_MAX_DYN) + return -ERANGE; + + ops = &priv->info->dyn_ops[blk_idx]; + + if (index >= ops->max_entry_count) + return -ERANGE; + if (!(ops->access & OP_READ)) + return -EOPNOTSUPP; + if (ops->packed_size > SJA1105_MAX_DYN_CMD_SIZE) + return -ERANGE; + if (!ops->cmd_packing) + return -EOPNOTSUPP; + if (!ops->entry_packing) + return -EOPNOTSUPP; + + cmd.valid = true; /* Trigger action on table entry */ + cmd.rdwrset = SPI_READ; /* Action is read */ + cmd.index = index; + ops->cmd_packing(packed_buf, &cmd, PACK); + + /* Send SPI write operation: read config table entry */ + rc = sja1105_spi_send_packed_buf(priv, SPI_WRITE, ops->addr, + packed_buf, ops->packed_size); + if (rc < 0) + return rc; + + /* Loop until we have confirmation that hardware has finished + * processing the command and has cleared the VALID field + */ + do { + memset(packed_buf, 0, ops->packed_size); + + /* Retrieve the read operation's result */ + rc = sja1105_spi_send_packed_buf(priv, SPI_READ, ops->addr, + packed_buf, ops->packed_size); + if (rc < 0) + return rc; + + cmd = (struct sja1105_dyn_cmd) {0}; + ops->cmd_packing(packed_buf, &cmd, UNPACK); + /* UM10944: [valident] will always be found cleared + * during a read access with MGMTROUTE set. + * So don't error out in that case. + */ + if (!cmd.valident && blk_idx != BLK_IDX_MGMT_ROUTE) + return -EINVAL; + cpu_relax(); + } while (cmd.valid && --retries); + + if (cmd.valid) + return -ETIMEDOUT; + + /* Don't dereference possibly NULL pointer - maybe caller + * only wanted to see whether the entry existed or not. + */ + if (entry) + ops->entry_packing(packed_buf, entry, UNPACK); + return 0; +} + +int sja1105_dynamic_config_write(struct sja1105_private *priv, + enum sja1105_blk_idx blk_idx, + int index, void *entry, bool keep) +{ + const struct sja1105_dynamic_table_ops *ops; + struct sja1105_dyn_cmd cmd = {0}; + /* SPI payload buffer */ + u8 packed_buf[SJA1105_MAX_DYN_CMD_SIZE] = {0}; + int rc; + + if (blk_idx >= BLK_IDX_MAX_DYN) + return -ERANGE; + + ops = &priv->info->dyn_ops[blk_idx]; + + if (index >= ops->max_entry_count) + return -ERANGE; + if (!(ops->access & OP_WRITE)) + return -EOPNOTSUPP; + if (!keep && !(ops->access & OP_DEL)) + return -EOPNOTSUPP; + if (ops->packed_size > SJA1105_MAX_DYN_CMD_SIZE) + return -ERANGE; + + cmd.valident = keep; /* If false, deletes entry */ + cmd.valid = true; /* Trigger action on table entry */ + cmd.rdwrset = SPI_WRITE; /* Action is write */ + cmd.index = index; + + if (!ops->cmd_packing) + return -EOPNOTSUPP; + ops->cmd_packing(packed_buf, &cmd, PACK); + + if (!ops->entry_packing) + return -EOPNOTSUPP; + /* Don't dereference potentially NULL pointer if just + * deleting a table entry is what was requested. For cases + * where 'index' field is physically part of entry structure, + * and needed here, we deal with that in the cmd_packing callback. + */ + if (keep) + ops->entry_packing(packed_buf, entry, PACK); + + /* Send SPI write operation: read config table entry */ + rc = sja1105_spi_send_packed_buf(priv, SPI_WRITE, ops->addr, + packed_buf, ops->packed_size); + if (rc < 0) + return rc; + + cmd = (struct sja1105_dyn_cmd) {0}; + ops->cmd_packing(packed_buf, &cmd, UNPACK); + if (cmd.errors) + return -EINVAL; + + return 0; +} + +static u8 sja1105_crc8_add(u8 crc, u8 byte, u8 poly) +{ + int i; + + for (i = 0; i < 8; i++) { + if ((crc ^ byte) & (1 << 7)) { + crc <<= 1; + crc ^= poly; + } else { + crc <<= 1; + } + byte <<= 1; + } + return crc; +} + +/* CRC8 algorithm with non-reversed input, non-reversed output, + * no input xor and no output xor. Code customized for receiving + * the SJA1105 E/T FDB keys (vlanid, macaddr) as input. CRC polynomial + * is also received as argument in the Koopman notation that the switch + * hardware stores it in. + */ +u8 sja1105_fdb_hash(struct sja1105_private *priv, const u8 *addr, u16 vid) +{ + struct sja1105_l2_lookup_params_entry *l2_lookup_params = + priv->static_config.tables[BLK_IDX_L2_LOOKUP_PARAMS].entries; + u64 poly_koopman = l2_lookup_params->poly; + /* Convert polynomial from Koopman to 'normal' notation */ + u8 poly = (u8)(1 + (poly_koopman << 1)); + u64 vlanid = l2_lookup_params->shared_learn ? 0 : vid; + u64 input = (vlanid << 48) | ether_addr_to_u64(addr); + u8 crc = 0; /* seed */ + int i; + + /* Mask the eight bytes starting from MSB one at a time */ + for (i = 56; i >= 0; i -= 8) { + u8 byte = (input & (0xffull << i)) >> i; + + crc = sja1105_crc8_add(crc, byte, poly); + } + return crc; +} diff --git a/drivers/net/dsa/sja1105/sja1105_dynamic_config.h b/drivers/net/dsa/sja1105/sja1105_dynamic_config.h new file mode 100644 index 0000000000000000000000000000000000000000..77be59546a55f086ec832575e12a1bc69a7ef986 --- /dev/null +++ b/drivers/net/dsa/sja1105/sja1105_dynamic_config.h @@ -0,0 +1,43 @@ +/* SPDX-License-Identifier: GPL-2.0 + * Copyright (c) 2019, Vladimir Oltean + */ +#ifndef _SJA1105_DYNAMIC_CONFIG_H +#define _SJA1105_DYNAMIC_CONFIG_H + +#include "sja1105.h" +#include + +struct sja1105_dyn_cmd { + u64 valid; + u64 rdwrset; + u64 errors; + u64 valident; + u64 index; +}; + +struct sja1105_dynamic_table_ops { + /* This returns size_t just to keep same prototype as the + * static config ops, of which we are reusing some functions. + */ + size_t (*entry_packing)(void *buf, void *entry_ptr, enum packing_op op); + void (*cmd_packing)(void *buf, struct sja1105_dyn_cmd *cmd, + enum packing_op op); + size_t max_entry_count; + size_t packed_size; + u64 addr; + u8 access; +}; + +struct sja1105_mgmt_entry { + u64 tsreg; + u64 takets; + u64 macaddr; + u64 destports; + u64 enfport; + u64 index; +}; + +extern struct sja1105_dynamic_table_ops sja1105et_dyn_ops[BLK_IDX_MAX_DYN]; +extern struct sja1105_dynamic_table_ops sja1105pqrs_dyn_ops[BLK_IDX_MAX_DYN]; + +#endif diff --git a/drivers/net/dsa/sja1105/sja1105_ethtool.c b/drivers/net/dsa/sja1105/sja1105_ethtool.c new file mode 100644 index 0000000000000000000000000000000000000000..ab581a28cd41710bf44721761bd02674ead4c1e0 --- /dev/null +++ b/drivers/net/dsa/sja1105/sja1105_ethtool.c @@ -0,0 +1,419 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2018-2019, Vladimir Oltean + */ +#include "sja1105.h" + +#define SJA1105_SIZE_MAC_AREA (0x02 * 4) +#define SJA1105_SIZE_HL1_AREA (0x10 * 4) +#define SJA1105_SIZE_HL2_AREA (0x4 * 4) +#define SJA1105_SIZE_QLEVEL_AREA (0x8 * 4) /* 0x4 to 0xB */ + +struct sja1105_port_status_mac { + u64 n_runt; + u64 n_soferr; + u64 n_alignerr; + u64 n_miierr; + u64 typeerr; + u64 sizeerr; + u64 tctimeout; + u64 priorerr; + u64 nomaster; + u64 memov; + u64 memerr; + u64 invtyp; + u64 intcyov; + u64 domerr; + u64 pcfbagdrop; + u64 spcprior; + u64 ageprior; + u64 portdrop; + u64 lendrop; + u64 bagdrop; + u64 policeerr; + u64 drpnona664err; + u64 spcerr; + u64 agedrp; +}; + +struct sja1105_port_status_hl1 { + u64 n_n664err; + u64 n_vlanerr; + u64 n_unreleased; + u64 n_sizeerr; + u64 n_crcerr; + u64 n_vlnotfound; + u64 n_ctpolerr; + u64 n_polerr; + u64 n_rxfrmsh; + u64 n_rxfrm; + u64 n_rxbytesh; + u64 n_rxbyte; + u64 n_txfrmsh; + u64 n_txfrm; + u64 n_txbytesh; + u64 n_txbyte; +}; + +struct sja1105_port_status_hl2 { + u64 n_qfull; + u64 n_part_drop; + u64 n_egr_disabled; + u64 n_not_reach; + u64 qlevel_hwm[8]; /* Only for P/Q/R/S */ + u64 qlevel[8]; /* Only for P/Q/R/S */ +}; + +struct sja1105_port_status { + struct sja1105_port_status_mac mac; + struct sja1105_port_status_hl1 hl1; + struct sja1105_port_status_hl2 hl2; +}; + +static void +sja1105_port_status_mac_unpack(void *buf, + struct sja1105_port_status_mac *status) +{ + /* Make pointer arithmetic work on 4 bytes */ + u32 *p = buf; + + sja1105_unpack(p + 0x0, &status->n_runt, 31, 24, 4); + sja1105_unpack(p + 0x0, &status->n_soferr, 23, 16, 4); + sja1105_unpack(p + 0x0, &status->n_alignerr, 15, 8, 4); + sja1105_unpack(p + 0x0, &status->n_miierr, 7, 0, 4); + sja1105_unpack(p + 0x1, &status->typeerr, 27, 27, 4); + sja1105_unpack(p + 0x1, &status->sizeerr, 26, 26, 4); + sja1105_unpack(p + 0x1, &status->tctimeout, 25, 25, 4); + sja1105_unpack(p + 0x1, &status->priorerr, 24, 24, 4); + sja1105_unpack(p + 0x1, &status->nomaster, 23, 23, 4); + sja1105_unpack(p + 0x1, &status->memov, 22, 22, 4); + sja1105_unpack(p + 0x1, &status->memerr, 21, 21, 4); + sja1105_unpack(p + 0x1, &status->invtyp, 19, 19, 4); + sja1105_unpack(p + 0x1, &status->intcyov, 18, 18, 4); + sja1105_unpack(p + 0x1, &status->domerr, 17, 17, 4); + sja1105_unpack(p + 0x1, &status->pcfbagdrop, 16, 16, 4); + sja1105_unpack(p + 0x1, &status->spcprior, 15, 12, 4); + sja1105_unpack(p + 0x1, &status->ageprior, 11, 8, 4); + sja1105_unpack(p + 0x1, &status->portdrop, 6, 6, 4); + sja1105_unpack(p + 0x1, &status->lendrop, 5, 5, 4); + sja1105_unpack(p + 0x1, &status->bagdrop, 4, 4, 4); + sja1105_unpack(p + 0x1, &status->policeerr, 3, 3, 4); + sja1105_unpack(p + 0x1, &status->drpnona664err, 2, 2, 4); + sja1105_unpack(p + 0x1, &status->spcerr, 1, 1, 4); + sja1105_unpack(p + 0x1, &status->agedrp, 0, 0, 4); +} + +static void +sja1105_port_status_hl1_unpack(void *buf, + struct sja1105_port_status_hl1 *status) +{ + /* Make pointer arithmetic work on 4 bytes */ + u32 *p = buf; + + sja1105_unpack(p + 0xF, &status->n_n664err, 31, 0, 4); + sja1105_unpack(p + 0xE, &status->n_vlanerr, 31, 0, 4); + sja1105_unpack(p + 0xD, &status->n_unreleased, 31, 0, 4); + sja1105_unpack(p + 0xC, &status->n_sizeerr, 31, 0, 4); + sja1105_unpack(p + 0xB, &status->n_crcerr, 31, 0, 4); + sja1105_unpack(p + 0xA, &status->n_vlnotfound, 31, 0, 4); + sja1105_unpack(p + 0x9, &status->n_ctpolerr, 31, 0, 4); + sja1105_unpack(p + 0x8, &status->n_polerr, 31, 0, 4); + sja1105_unpack(p + 0x7, &status->n_rxfrmsh, 31, 0, 4); + sja1105_unpack(p + 0x6, &status->n_rxfrm, 31, 0, 4); + sja1105_unpack(p + 0x5, &status->n_rxbytesh, 31, 0, 4); + sja1105_unpack(p + 0x4, &status->n_rxbyte, 31, 0, 4); + sja1105_unpack(p + 0x3, &status->n_txfrmsh, 31, 0, 4); + sja1105_unpack(p + 0x2, &status->n_txfrm, 31, 0, 4); + sja1105_unpack(p + 0x1, &status->n_txbytesh, 31, 0, 4); + sja1105_unpack(p + 0x0, &status->n_txbyte, 31, 0, 4); + status->n_rxfrm += status->n_rxfrmsh << 32; + status->n_rxbyte += status->n_rxbytesh << 32; + status->n_txfrm += status->n_txfrmsh << 32; + status->n_txbyte += status->n_txbytesh << 32; +} + +static void +sja1105_port_status_hl2_unpack(void *buf, + struct sja1105_port_status_hl2 *status) +{ + /* Make pointer arithmetic work on 4 bytes */ + u32 *p = buf; + + sja1105_unpack(p + 0x3, &status->n_qfull, 31, 0, 4); + sja1105_unpack(p + 0x2, &status->n_part_drop, 31, 0, 4); + sja1105_unpack(p + 0x1, &status->n_egr_disabled, 31, 0, 4); + sja1105_unpack(p + 0x0, &status->n_not_reach, 31, 0, 4); +} + +static void +sja1105pqrs_port_status_qlevel_unpack(void *buf, + struct sja1105_port_status_hl2 *status) +{ + /* Make pointer arithmetic work on 4 bytes */ + u32 *p = buf; + int i; + + for (i = 0; i < 8; i++) { + sja1105_unpack(p + i, &status->qlevel_hwm[i], 24, 16, 4); + sja1105_unpack(p + i, &status->qlevel[i], 8, 0, 4); + } +} + +static int sja1105_port_status_get_mac(struct sja1105_private *priv, + struct sja1105_port_status_mac *status, + int port) +{ + const struct sja1105_regs *regs = priv->info->regs; + u8 packed_buf[SJA1105_SIZE_MAC_AREA] = {0}; + int rc; + + /* MAC area */ + rc = sja1105_spi_send_packed_buf(priv, SPI_READ, regs->mac[port], + packed_buf, SJA1105_SIZE_MAC_AREA); + if (rc < 0) + return rc; + + sja1105_port_status_mac_unpack(packed_buf, status); + + return 0; +} + +static int sja1105_port_status_get_hl1(struct sja1105_private *priv, + struct sja1105_port_status_hl1 *status, + int port) +{ + const struct sja1105_regs *regs = priv->info->regs; + u8 packed_buf[SJA1105_SIZE_HL1_AREA] = {0}; + int rc; + + rc = sja1105_spi_send_packed_buf(priv, SPI_READ, regs->mac_hl1[port], + packed_buf, SJA1105_SIZE_HL1_AREA); + if (rc < 0) + return rc; + + sja1105_port_status_hl1_unpack(packed_buf, status); + + return 0; +} + +static int sja1105_port_status_get_hl2(struct sja1105_private *priv, + struct sja1105_port_status_hl2 *status, + int port) +{ + const struct sja1105_regs *regs = priv->info->regs; + u8 packed_buf[SJA1105_SIZE_QLEVEL_AREA] = {0}; + int rc; + + rc = sja1105_spi_send_packed_buf(priv, SPI_READ, regs->mac_hl2[port], + packed_buf, SJA1105_SIZE_HL2_AREA); + if (rc < 0) + return rc; + + sja1105_port_status_hl2_unpack(packed_buf, status); + + /* Code below is strictly P/Q/R/S specific. */ + if (priv->info->device_id == SJA1105E_DEVICE_ID || + priv->info->device_id == SJA1105T_DEVICE_ID) + return 0; + + rc = sja1105_spi_send_packed_buf(priv, SPI_READ, regs->qlevel[port], + packed_buf, SJA1105_SIZE_QLEVEL_AREA); + if (rc < 0) + return rc; + + sja1105pqrs_port_status_qlevel_unpack(packed_buf, status); + + return 0; +} + +static int sja1105_port_status_get(struct sja1105_private *priv, + struct sja1105_port_status *status, + int port) +{ + int rc; + + rc = sja1105_port_status_get_mac(priv, &status->mac, port); + if (rc < 0) + return rc; + rc = sja1105_port_status_get_hl1(priv, &status->hl1, port); + if (rc < 0) + return rc; + rc = sja1105_port_status_get_hl2(priv, &status->hl2, port); + if (rc < 0) + return rc; + + return 0; +} + +static char sja1105_port_stats[][ETH_GSTRING_LEN] = { + /* MAC-Level Diagnostic Counters */ + "n_runt", + "n_soferr", + "n_alignerr", + "n_miierr", + /* MAC-Level Diagnostic Flags */ + "typeerr", + "sizeerr", + "tctimeout", + "priorerr", + "nomaster", + "memov", + "memerr", + "invtyp", + "intcyov", + "domerr", + "pcfbagdrop", + "spcprior", + "ageprior", + "portdrop", + "lendrop", + "bagdrop", + "policeerr", + "drpnona664err", + "spcerr", + "agedrp", + /* High-Level Diagnostic Counters */ + "n_n664err", + "n_vlanerr", + "n_unreleased", + "n_sizeerr", + "n_crcerr", + "n_vlnotfound", + "n_ctpolerr", + "n_polerr", + "n_rxfrm", + "n_rxbyte", + "n_txfrm", + "n_txbyte", + "n_qfull", + "n_part_drop", + "n_egr_disabled", + "n_not_reach", +}; + +static char sja1105pqrs_extra_port_stats[][ETH_GSTRING_LEN] = { + /* Queue Levels */ + "qlevel_hwm_0", + "qlevel_hwm_1", + "qlevel_hwm_2", + "qlevel_hwm_3", + "qlevel_hwm_4", + "qlevel_hwm_5", + "qlevel_hwm_6", + "qlevel_hwm_7", + "qlevel_0", + "qlevel_1", + "qlevel_2", + "qlevel_3", + "qlevel_4", + "qlevel_5", + "qlevel_6", + "qlevel_7", +}; + +void sja1105_get_ethtool_stats(struct dsa_switch *ds, int port, u64 *data) +{ + struct sja1105_private *priv = ds->priv; + struct sja1105_port_status status; + int rc, i, k = 0; + + memset(&status, 0, sizeof(status)); + + rc = sja1105_port_status_get(priv, &status, port); + if (rc < 0) { + dev_err(ds->dev, "Failed to read port %d counters: %d\n", + port, rc); + return; + } + memset(data, 0, ARRAY_SIZE(sja1105_port_stats) * sizeof(u64)); + data[k++] = status.mac.n_runt; + data[k++] = status.mac.n_soferr; + data[k++] = status.mac.n_alignerr; + data[k++] = status.mac.n_miierr; + data[k++] = status.mac.typeerr; + data[k++] = status.mac.sizeerr; + data[k++] = status.mac.tctimeout; + data[k++] = status.mac.priorerr; + data[k++] = status.mac.nomaster; + data[k++] = status.mac.memov; + data[k++] = status.mac.memerr; + data[k++] = status.mac.invtyp; + data[k++] = status.mac.intcyov; + data[k++] = status.mac.domerr; + data[k++] = status.mac.pcfbagdrop; + data[k++] = status.mac.spcprior; + data[k++] = status.mac.ageprior; + data[k++] = status.mac.portdrop; + data[k++] = status.mac.lendrop; + data[k++] = status.mac.bagdrop; + data[k++] = status.mac.policeerr; + data[k++] = status.mac.drpnona664err; + data[k++] = status.mac.spcerr; + data[k++] = status.mac.agedrp; + data[k++] = status.hl1.n_n664err; + data[k++] = status.hl1.n_vlanerr; + data[k++] = status.hl1.n_unreleased; + data[k++] = status.hl1.n_sizeerr; + data[k++] = status.hl1.n_crcerr; + data[k++] = status.hl1.n_vlnotfound; + data[k++] = status.hl1.n_ctpolerr; + data[k++] = status.hl1.n_polerr; + data[k++] = status.hl1.n_rxfrm; + data[k++] = status.hl1.n_rxbyte; + data[k++] = status.hl1.n_txfrm; + data[k++] = status.hl1.n_txbyte; + data[k++] = status.hl2.n_qfull; + data[k++] = status.hl2.n_part_drop; + data[k++] = status.hl2.n_egr_disabled; + data[k++] = status.hl2.n_not_reach; + + if (priv->info->device_id == SJA1105E_DEVICE_ID || + priv->info->device_id == SJA1105T_DEVICE_ID) + return; + + memset(data + k, 0, ARRAY_SIZE(sja1105pqrs_extra_port_stats) * + sizeof(u64)); + for (i = 0; i < 8; i++) { + data[k++] = status.hl2.qlevel_hwm[i]; + data[k++] = status.hl2.qlevel[i]; + } +} + +void sja1105_get_strings(struct dsa_switch *ds, int port, + u32 stringset, u8 *data) +{ + struct sja1105_private *priv = ds->priv; + u8 *p = data; + int i; + + switch (stringset) { + case ETH_SS_STATS: + for (i = 0; i < ARRAY_SIZE(sja1105_port_stats); i++) { + strlcpy(p, sja1105_port_stats[i], ETH_GSTRING_LEN); + p += ETH_GSTRING_LEN; + } + if (priv->info->device_id == SJA1105E_DEVICE_ID || + priv->info->device_id == SJA1105T_DEVICE_ID) + return; + for (i = 0; i < ARRAY_SIZE(sja1105pqrs_extra_port_stats); i++) { + strlcpy(p, sja1105pqrs_extra_port_stats[i], + ETH_GSTRING_LEN); + p += ETH_GSTRING_LEN; + } + break; + } +} + +int sja1105_get_sset_count(struct dsa_switch *ds, int port, int sset) +{ + int count = ARRAY_SIZE(sja1105_port_stats); + struct sja1105_private *priv = ds->priv; + + if (sset != ETH_SS_STATS) + return -EOPNOTSUPP; + + if (priv->info->device_id == SJA1105PR_DEVICE_ID || + priv->info->device_id == SJA1105QS_DEVICE_ID) + count += ARRAY_SIZE(sja1105pqrs_extra_port_stats); + + return count; +} diff --git a/drivers/net/dsa/sja1105/sja1105_main.c b/drivers/net/dsa/sja1105/sja1105_main.c new file mode 100644 index 0000000000000000000000000000000000000000..50ff625c85d678588b3a0d32e00796b035c953ca --- /dev/null +++ b/drivers/net/dsa/sja1105/sja1105_main.c @@ -0,0 +1,1675 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2018, Sensor-Technik Wiedemann GmbH + * Copyright (c) 2018-2019, Vladimir Oltean + */ + +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "sja1105.h" + +static void sja1105_hw_reset(struct gpio_desc *gpio, unsigned int pulse_len, + unsigned int startup_delay) +{ + gpiod_set_value_cansleep(gpio, 1); + /* Wait for minimum reset pulse length */ + msleep(pulse_len); + gpiod_set_value_cansleep(gpio, 0); + /* Wait until chip is ready after reset */ + msleep(startup_delay); +} + +static void +sja1105_port_allow_traffic(struct sja1105_l2_forwarding_entry *l2_fwd, + int from, int to, bool allow) +{ + if (allow) { + l2_fwd[from].bc_domain |= BIT(to); + l2_fwd[from].reach_port |= BIT(to); + l2_fwd[from].fl_domain |= BIT(to); + } else { + l2_fwd[from].bc_domain &= ~BIT(to); + l2_fwd[from].reach_port &= ~BIT(to); + l2_fwd[from].fl_domain &= ~BIT(to); + } +} + +/* Structure used to temporarily transport device tree + * settings into sja1105_setup + */ +struct sja1105_dt_port { + phy_interface_t phy_mode; + sja1105_mii_role_t role; +}; + +static int sja1105_init_mac_settings(struct sja1105_private *priv) +{ + struct sja1105_mac_config_entry default_mac = { + /* Enable all 8 priority queues on egress. + * Every queue i holds top[i] - base[i] frames. + * Sum of top[i] - base[i] is 511 (max hardware limit). + */ + .top = {0x3F, 0x7F, 0xBF, 0xFF, 0x13F, 0x17F, 0x1BF, 0x1FF}, + .base = {0x0, 0x40, 0x80, 0xC0, 0x100, 0x140, 0x180, 0x1C0}, + .enabled = {true, true, true, true, true, true, true, true}, + /* Keep standard IFG of 12 bytes on egress. */ + .ifg = 0, + /* Always put the MAC speed in automatic mode, where it can be + * retrieved from the PHY object through phylib and + * sja1105_adjust_port_config. + */ + .speed = SJA1105_SPEED_AUTO, + /* No static correction for 1-step 1588 events */ + .tp_delin = 0, + .tp_delout = 0, + /* Disable aging for critical TTEthernet traffic */ + .maxage = 0xFF, + /* Internal VLAN (pvid) to apply to untagged ingress */ + .vlanprio = 0, + .vlanid = 0, + .ing_mirr = false, + .egr_mirr = false, + /* Don't drop traffic with other EtherType than ETH_P_IP */ + .drpnona664 = false, + /* Don't drop double-tagged traffic */ + .drpdtag = false, + /* Don't drop untagged traffic */ + .drpuntag = false, + /* Don't retag 802.1p (VID 0) traffic with the pvid */ + .retag = false, + /* Disable learning and I/O on user ports by default - + * STP will enable it. + */ + .dyn_learn = false, + .egress = false, + .ingress = false, + }; + struct sja1105_mac_config_entry *mac; + struct sja1105_table *table; + int i; + + table = &priv->static_config.tables[BLK_IDX_MAC_CONFIG]; + + /* Discard previous MAC Configuration Table */ + if (table->entry_count) { + kfree(table->entries); + table->entry_count = 0; + } + + table->entries = kcalloc(SJA1105_NUM_PORTS, + table->ops->unpacked_entry_size, GFP_KERNEL); + if (!table->entries) + return -ENOMEM; + + /* Override table based on phylib DT bindings */ + table->entry_count = SJA1105_NUM_PORTS; + + mac = table->entries; + + for (i = 0; i < SJA1105_NUM_PORTS; i++) { + mac[i] = default_mac; + if (i == dsa_upstream_port(priv->ds, i)) { + /* STP doesn't get called for CPU port, so we need to + * set the I/O parameters statically. + */ + mac[i].dyn_learn = true; + mac[i].ingress = true; + mac[i].egress = true; + } + } + + return 0; +} + +static int sja1105_init_mii_settings(struct sja1105_private *priv, + struct sja1105_dt_port *ports) +{ + struct device *dev = &priv->spidev->dev; + struct sja1105_xmii_params_entry *mii; + struct sja1105_table *table; + int i; + + table = &priv->static_config.tables[BLK_IDX_XMII_PARAMS]; + + /* Discard previous xMII Mode Parameters Table */ + if (table->entry_count) { + kfree(table->entries); + table->entry_count = 0; + } + + table->entries = kcalloc(SJA1105_MAX_XMII_PARAMS_COUNT, + table->ops->unpacked_entry_size, GFP_KERNEL); + if (!table->entries) + return -ENOMEM; + + /* Override table based on phylib DT bindings */ + table->entry_count = SJA1105_MAX_XMII_PARAMS_COUNT; + + mii = table->entries; + + for (i = 0; i < SJA1105_NUM_PORTS; i++) { + switch (ports[i].phy_mode) { + case PHY_INTERFACE_MODE_MII: + mii->xmii_mode[i] = XMII_MODE_MII; + break; + case PHY_INTERFACE_MODE_RMII: + mii->xmii_mode[i] = XMII_MODE_RMII; + break; + case PHY_INTERFACE_MODE_RGMII: + case PHY_INTERFACE_MODE_RGMII_ID: + case PHY_INTERFACE_MODE_RGMII_RXID: + case PHY_INTERFACE_MODE_RGMII_TXID: + mii->xmii_mode[i] = XMII_MODE_RGMII; + break; + default: + dev_err(dev, "Unsupported PHY mode %s!\n", + phy_modes(ports[i].phy_mode)); + } + + mii->phy_mac[i] = ports[i].role; + } + return 0; +} + +static int sja1105_init_static_fdb(struct sja1105_private *priv) +{ + struct sja1105_table *table; + + table = &priv->static_config.tables[BLK_IDX_L2_LOOKUP]; + + /* We only populate the FDB table through dynamic + * L2 Address Lookup entries + */ + if (table->entry_count) { + kfree(table->entries); + table->entry_count = 0; + } + return 0; +} + +static int sja1105_init_l2_lookup_params(struct sja1105_private *priv) +{ + struct sja1105_table *table; + struct sja1105_l2_lookup_params_entry default_l2_lookup_params = { + /* Learned FDB entries are forgotten after 300 seconds */ + .maxage = SJA1105_AGEING_TIME_MS(300000), + /* All entries within a FDB bin are available for learning */ + .dyn_tbsz = SJA1105ET_FDB_BIN_SIZE, + /* 2^8 + 2^5 + 2^3 + 2^2 + 2^1 + 1 in Koopman notation */ + .poly = 0x97, + /* This selects between Independent VLAN Learning (IVL) and + * Shared VLAN Learning (SVL) + */ + .shared_learn = false, + /* Don't discard management traffic based on ENFPORT - + * we don't perform SMAC port enforcement anyway, so + * what we are setting here doesn't matter. + */ + .no_enf_hostprt = false, + /* Don't learn SMAC for mac_fltres1 and mac_fltres0. + * Maybe correlate with no_linklocal_learn from bridge driver? + */ + .no_mgmt_learn = true, + }; + + table = &priv->static_config.tables[BLK_IDX_L2_LOOKUP_PARAMS]; + + if (table->entry_count) { + kfree(table->entries); + table->entry_count = 0; + } + + table->entries = kcalloc(SJA1105_MAX_L2_LOOKUP_PARAMS_COUNT, + table->ops->unpacked_entry_size, GFP_KERNEL); + if (!table->entries) + return -ENOMEM; + + table->entry_count = SJA1105_MAX_L2_LOOKUP_PARAMS_COUNT; + + /* This table only has a single entry */ + ((struct sja1105_l2_lookup_params_entry *)table->entries)[0] = + default_l2_lookup_params; + + return 0; +} + +static int sja1105_init_static_vlan(struct sja1105_private *priv) +{ + struct sja1105_table *table; + struct sja1105_vlan_lookup_entry pvid = { + .ving_mirr = 0, + .vegr_mirr = 0, + .vmemb_port = 0, + .vlan_bc = 0, + .tag_port = 0, + .vlanid = 0, + }; + int i; + + table = &priv->static_config.tables[BLK_IDX_VLAN_LOOKUP]; + + /* The static VLAN table will only contain the initial pvid of 0. + * All other VLANs are to be configured through dynamic entries, + * and kept in the static configuration table as backing memory. + * The pvid of 0 is sufficient to pass traffic while the ports are + * standalone and when vlan_filtering is disabled. When filtering + * gets enabled, the switchdev core sets up the VLAN ID 1 and sets + * it as the new pvid. Actually 'pvid 1' still comes up in 'bridge + * vlan' even when vlan_filtering is off, but it has no effect. + */ + if (table->entry_count) { + kfree(table->entries); + table->entry_count = 0; + } + + table->entries = kcalloc(1, table->ops->unpacked_entry_size, + GFP_KERNEL); + if (!table->entries) + return -ENOMEM; + + table->entry_count = 1; + + /* VLAN ID 0: all DT-defined ports are members; no restrictions on + * forwarding; always transmit priority-tagged frames as untagged. + */ + for (i = 0; i < SJA1105_NUM_PORTS; i++) { + pvid.vmemb_port |= BIT(i); + pvid.vlan_bc |= BIT(i); + pvid.tag_port &= ~BIT(i); + } + + ((struct sja1105_vlan_lookup_entry *)table->entries)[0] = pvid; + return 0; +} + +static int sja1105_init_l2_forwarding(struct sja1105_private *priv) +{ + struct sja1105_l2_forwarding_entry *l2fwd; + struct sja1105_table *table; + int i, j; + + table = &priv->static_config.tables[BLK_IDX_L2_FORWARDING]; + + if (table->entry_count) { + kfree(table->entries); + table->entry_count = 0; + } + + table->entries = kcalloc(SJA1105_MAX_L2_FORWARDING_COUNT, + table->ops->unpacked_entry_size, GFP_KERNEL); + if (!table->entries) + return -ENOMEM; + + table->entry_count = SJA1105_MAX_L2_FORWARDING_COUNT; + + l2fwd = table->entries; + + /* First 5 entries define the forwarding rules */ + for (i = 0; i < SJA1105_NUM_PORTS; i++) { + unsigned int upstream = dsa_upstream_port(priv->ds, i); + + for (j = 0; j < SJA1105_NUM_TC; j++) + l2fwd[i].vlan_pmap[j] = j; + + if (i == upstream) + continue; + + sja1105_port_allow_traffic(l2fwd, i, upstream, true); + sja1105_port_allow_traffic(l2fwd, upstream, i, true); + } + /* Next 8 entries define VLAN PCP mapping from ingress to egress. + * Create a one-to-one mapping. + */ + for (i = 0; i < SJA1105_NUM_TC; i++) + for (j = 0; j < SJA1105_NUM_PORTS; j++) + l2fwd[SJA1105_NUM_PORTS + i].vlan_pmap[j] = i; + + return 0; +} + +static int sja1105_init_l2_forwarding_params(struct sja1105_private *priv) +{ + struct sja1105_l2_forwarding_params_entry default_l2fwd_params = { + /* Disallow dynamic reconfiguration of vlan_pmap */ + .max_dynp = 0, + /* Use a single memory partition for all ingress queues */ + .part_spc = { SJA1105_MAX_FRAME_MEMORY, 0, 0, 0, 0, 0, 0, 0 }, + }; + struct sja1105_table *table; + + table = &priv->static_config.tables[BLK_IDX_L2_FORWARDING_PARAMS]; + + if (table->entry_count) { + kfree(table->entries); + table->entry_count = 0; + } + + table->entries = kcalloc(SJA1105_MAX_L2_FORWARDING_PARAMS_COUNT, + table->ops->unpacked_entry_size, GFP_KERNEL); + if (!table->entries) + return -ENOMEM; + + table->entry_count = SJA1105_MAX_L2_FORWARDING_PARAMS_COUNT; + + /* This table only has a single entry */ + ((struct sja1105_l2_forwarding_params_entry *)table->entries)[0] = + default_l2fwd_params; + + return 0; +} + +static int sja1105_init_general_params(struct sja1105_private *priv) +{ + struct sja1105_general_params_entry default_general_params = { + /* Disallow dynamic changing of the mirror port */ + .mirr_ptacu = 0, + .switchid = priv->ds->index, + /* Priority queue for link-local frames trapped to CPU */ + .hostprio = 0, + .mac_fltres1 = SJA1105_LINKLOCAL_FILTER_A, + .mac_flt1 = SJA1105_LINKLOCAL_FILTER_A_MASK, + .incl_srcpt1 = true, + .send_meta1 = false, + .mac_fltres0 = SJA1105_LINKLOCAL_FILTER_B, + .mac_flt0 = SJA1105_LINKLOCAL_FILTER_B_MASK, + .incl_srcpt0 = true, + .send_meta0 = false, + /* The destination for traffic matching mac_fltres1 and + * mac_fltres0 on all ports except host_port. Such traffic + * receieved on host_port itself would be dropped, except + * by installing a temporary 'management route' + */ + .host_port = dsa_upstream_port(priv->ds, 0), + /* Same as host port */ + .mirr_port = dsa_upstream_port(priv->ds, 0), + /* Link-local traffic received on casc_port will be forwarded + * to host_port without embedding the source port and device ID + * info in the destination MAC address (presumably because it + * is a cascaded port and a downstream SJA switch already did + * that). Default to an invalid port (to disable the feature) + * and overwrite this if we find any DSA (cascaded) ports. + */ + .casc_port = SJA1105_NUM_PORTS, + /* No TTEthernet */ + .vllupformat = 0, + .vlmarker = 0, + .vlmask = 0, + /* Only update correctionField for 1-step PTP (L2 transport) */ + .ignore2stf = 0, + /* Forcefully disable VLAN filtering by telling + * the switch that VLAN has a different EtherType. + */ + .tpid = ETH_P_SJA1105, + .tpid2 = ETH_P_SJA1105, + }; + struct sja1105_table *table; + int i, k = 0; + + for (i = 0; i < SJA1105_NUM_PORTS; i++) { + if (dsa_is_dsa_port(priv->ds, i)) + default_general_params.casc_port = i; + else if (dsa_is_user_port(priv->ds, i)) + priv->ports[i].mgmt_slot = k++; + } + + table = &priv->static_config.tables[BLK_IDX_GENERAL_PARAMS]; + + if (table->entry_count) { + kfree(table->entries); + table->entry_count = 0; + } + + table->entries = kcalloc(SJA1105_MAX_GENERAL_PARAMS_COUNT, + table->ops->unpacked_entry_size, GFP_KERNEL); + if (!table->entries) + return -ENOMEM; + + table->entry_count = SJA1105_MAX_GENERAL_PARAMS_COUNT; + + /* This table only has a single entry */ + ((struct sja1105_general_params_entry *)table->entries)[0] = + default_general_params; + + return 0; +} + +#define SJA1105_RATE_MBPS(speed) (((speed) * 64000) / 1000) + +static inline void +sja1105_setup_policer(struct sja1105_l2_policing_entry *policing, + int index) +{ + policing[index].sharindx = index; + policing[index].smax = 65535; /* Burst size in bytes */ + policing[index].rate = SJA1105_RATE_MBPS(1000); + policing[index].maxlen = ETH_FRAME_LEN + VLAN_HLEN + ETH_FCS_LEN; + policing[index].partition = 0; +} + +static int sja1105_init_l2_policing(struct sja1105_private *priv) +{ + struct sja1105_l2_policing_entry *policing; + struct sja1105_table *table; + int i, j, k; + + table = &priv->static_config.tables[BLK_IDX_L2_POLICING]; + + /* Discard previous L2 Policing Table */ + if (table->entry_count) { + kfree(table->entries); + table->entry_count = 0; + } + + table->entries = kcalloc(SJA1105_MAX_L2_POLICING_COUNT, + table->ops->unpacked_entry_size, GFP_KERNEL); + if (!table->entries) + return -ENOMEM; + + table->entry_count = SJA1105_MAX_L2_POLICING_COUNT; + + policing = table->entries; + + /* k sweeps through all unicast policers (0-39). + * bcast sweeps through policers 40-44. + */ + for (i = 0, k = 0; i < SJA1105_NUM_PORTS; i++) { + int bcast = (SJA1105_NUM_PORTS * SJA1105_NUM_TC) + i; + + for (j = 0; j < SJA1105_NUM_TC; j++, k++) + sja1105_setup_policer(policing, k); + + /* Set up this port's policer for broadcast traffic */ + sja1105_setup_policer(policing, bcast); + } + return 0; +} + +static int sja1105_static_config_load(struct sja1105_private *priv, + struct sja1105_dt_port *ports) +{ + int rc; + + sja1105_static_config_free(&priv->static_config); + rc = sja1105_static_config_init(&priv->static_config, + priv->info->static_ops, + priv->info->device_id); + if (rc) + return rc; + + /* Build static configuration */ + rc = sja1105_init_mac_settings(priv); + if (rc < 0) + return rc; + rc = sja1105_init_mii_settings(priv, ports); + if (rc < 0) + return rc; + rc = sja1105_init_static_fdb(priv); + if (rc < 0) + return rc; + rc = sja1105_init_static_vlan(priv); + if (rc < 0) + return rc; + rc = sja1105_init_l2_lookup_params(priv); + if (rc < 0) + return rc; + rc = sja1105_init_l2_forwarding(priv); + if (rc < 0) + return rc; + rc = sja1105_init_l2_forwarding_params(priv); + if (rc < 0) + return rc; + rc = sja1105_init_l2_policing(priv); + if (rc < 0) + return rc; + rc = sja1105_init_general_params(priv); + if (rc < 0) + return rc; + + /* Send initial configuration to hardware via SPI */ + return sja1105_static_config_upload(priv); +} + +static int sja1105_parse_rgmii_delays(struct sja1105_private *priv, + const struct sja1105_dt_port *ports) +{ + int i; + + for (i = 0; i < SJA1105_NUM_PORTS; i++) { + if (ports->role == XMII_MAC) + continue; + + if (ports->phy_mode == PHY_INTERFACE_MODE_RGMII_RXID || + ports->phy_mode == PHY_INTERFACE_MODE_RGMII_ID) + priv->rgmii_rx_delay[i] = true; + + if (ports->phy_mode == PHY_INTERFACE_MODE_RGMII_TXID || + ports->phy_mode == PHY_INTERFACE_MODE_RGMII_ID) + priv->rgmii_tx_delay[i] = true; + + if ((priv->rgmii_rx_delay[i] || priv->rgmii_tx_delay[i]) && + !priv->info->setup_rgmii_delay) + return -EINVAL; + } + return 0; +} + +static int sja1105_parse_ports_node(struct sja1105_private *priv, + struct sja1105_dt_port *ports, + struct device_node *ports_node) +{ + struct device *dev = &priv->spidev->dev; + struct device_node *child; + + for_each_child_of_node(ports_node, child) { + struct device_node *phy_node; + int phy_mode; + u32 index; + + /* Get switch port number from DT */ + if (of_property_read_u32(child, "reg", &index) < 0) { + dev_err(dev, "Port number not defined in device tree " + "(property \"reg\")\n"); + return -ENODEV; + } + + /* Get PHY mode from DT */ + phy_mode = of_get_phy_mode(child); + if (phy_mode < 0) { + dev_err(dev, "Failed to read phy-mode or " + "phy-interface-type property for port %d\n", + index); + return -ENODEV; + } + ports[index].phy_mode = phy_mode; + + phy_node = of_parse_phandle(child, "phy-handle", 0); + if (!phy_node) { + if (!of_phy_is_fixed_link(child)) { + dev_err(dev, "phy-handle or fixed-link " + "properties missing!\n"); + return -ENODEV; + } + /* phy-handle is missing, but fixed-link isn't. + * So it's a fixed link. Default to PHY role. + */ + ports[index].role = XMII_PHY; + } else { + /* phy-handle present => put port in MAC role */ + ports[index].role = XMII_MAC; + of_node_put(phy_node); + } + + /* The MAC/PHY role can be overridden with explicit bindings */ + if (of_property_read_bool(child, "sja1105,role-mac")) + ports[index].role = XMII_MAC; + else if (of_property_read_bool(child, "sja1105,role-phy")) + ports[index].role = XMII_PHY; + } + + return 0; +} + +static int sja1105_parse_dt(struct sja1105_private *priv, + struct sja1105_dt_port *ports) +{ + struct device *dev = &priv->spidev->dev; + struct device_node *switch_node = dev->of_node; + struct device_node *ports_node; + int rc; + + ports_node = of_get_child_by_name(switch_node, "ports"); + if (!ports_node) { + dev_err(dev, "Incorrect bindings: absent \"ports\" node\n"); + return -ENODEV; + } + + rc = sja1105_parse_ports_node(priv, ports, ports_node); + of_node_put(ports_node); + + return rc; +} + +/* Convert back and forth MAC speed from Mbps to SJA1105 encoding */ +static int sja1105_speed[] = { + [SJA1105_SPEED_AUTO] = 0, + [SJA1105_SPEED_10MBPS] = 10, + [SJA1105_SPEED_100MBPS] = 100, + [SJA1105_SPEED_1000MBPS] = 1000, +}; + +static sja1105_speed_t sja1105_get_speed_cfg(unsigned int speed_mbps) +{ + int i; + + for (i = SJA1105_SPEED_AUTO; i <= SJA1105_SPEED_1000MBPS; i++) + if (sja1105_speed[i] == speed_mbps) + return i; + return -EINVAL; +} + +/* Set link speed and enable/disable traffic I/O in the MAC configuration + * for a specific port. + * + * @speed_mbps: If 0, leave the speed unchanged, else adapt MAC to PHY speed. + * @enabled: Manage Rx and Tx settings for this port. If false, overrides the + * settings from the STP state, but not persistently (does not + * overwrite the static MAC info for this port). + */ +static int sja1105_adjust_port_config(struct sja1105_private *priv, int port, + int speed_mbps, bool enabled) +{ + struct sja1105_mac_config_entry dyn_mac; + struct sja1105_xmii_params_entry *mii; + struct sja1105_mac_config_entry *mac; + struct device *dev = priv->ds->dev; + sja1105_phy_interface_t phy_mode; + sja1105_speed_t speed; + int rc; + + mii = priv->static_config.tables[BLK_IDX_XMII_PARAMS].entries; + mac = priv->static_config.tables[BLK_IDX_MAC_CONFIG].entries; + + speed = sja1105_get_speed_cfg(speed_mbps); + if (speed_mbps && speed < 0) { + dev_err(dev, "Invalid speed %iMbps\n", speed_mbps); + return -EINVAL; + } + + /* If requested, overwrite SJA1105_SPEED_AUTO from the static MAC + * configuration table, since this will be used for the clocking setup, + * and we no longer need to store it in the static config (already told + * hardware we want auto during upload phase). + */ + if (speed_mbps) + mac[port].speed = speed; + else + mac[port].speed = SJA1105_SPEED_AUTO; + + /* On P/Q/R/S, one can read from the device via the MAC reconfiguration + * tables. On E/T, MAC reconfig tables are not readable, only writable. + * We have to *know* what the MAC looks like. For the sake of keeping + * the code common, we'll use the static configuration tables as a + * reasonable approximation for both E/T and P/Q/R/S. + */ + dyn_mac = mac[port]; + dyn_mac.ingress = enabled && mac[port].ingress; + dyn_mac.egress = enabled && mac[port].egress; + + /* Write to the dynamic reconfiguration tables */ + rc = sja1105_dynamic_config_write(priv, BLK_IDX_MAC_CONFIG, + port, &dyn_mac, true); + if (rc < 0) { + dev_err(dev, "Failed to write MAC config: %d\n", rc); + return rc; + } + + /* Reconfigure the PLLs for the RGMII interfaces (required 125 MHz at + * gigabit, 25 MHz at 100 Mbps and 2.5 MHz at 10 Mbps). For MII and + * RMII no change of the clock setup is required. Actually, changing + * the clock setup does interrupt the clock signal for a certain time + * which causes trouble for all PHYs relying on this signal. + */ + if (!enabled) + return 0; + + phy_mode = mii->xmii_mode[port]; + if (phy_mode != XMII_MODE_RGMII) + return 0; + + return sja1105_clocking_setup_port(priv, port); +} + +static void sja1105_adjust_link(struct dsa_switch *ds, int port, + struct phy_device *phydev) +{ + struct sja1105_private *priv = ds->priv; + + if (!phydev->link) + sja1105_adjust_port_config(priv, port, 0, false); + else + sja1105_adjust_port_config(priv, port, phydev->speed, true); +} + +static void sja1105_phylink_validate(struct dsa_switch *ds, int port, + unsigned long *supported, + struct phylink_link_state *state) +{ + /* Construct a new mask which exhaustively contains all link features + * supported by the MAC, and then apply that (logical AND) to what will + * be sent to the PHY for "marketing". + */ + __ETHTOOL_DECLARE_LINK_MODE_MASK(mask) = { 0, }; + struct sja1105_private *priv = ds->priv; + struct sja1105_xmii_params_entry *mii; + + mii = priv->static_config.tables[BLK_IDX_XMII_PARAMS].entries; + + /* The MAC does not support pause frames, and also doesn't + * support half-duplex traffic modes. + */ + phylink_set(mask, Autoneg); + phylink_set(mask, MII); + phylink_set(mask, 10baseT_Full); + phylink_set(mask, 100baseT_Full); + if (mii->xmii_mode[port] == XMII_MODE_RGMII) + phylink_set(mask, 1000baseT_Full); + + bitmap_and(supported, supported, mask, __ETHTOOL_LINK_MODE_MASK_NBITS); + bitmap_and(state->advertising, state->advertising, mask, + __ETHTOOL_LINK_MODE_MASK_NBITS); +} + +/* First-generation switches have a 4-way set associative TCAM that + * holds the FDB entries. An FDB index spans from 0 to 1023 and is comprised of + * a "bin" (grouping of 4 entries) and a "way" (an entry within a bin). + * For the placement of a newly learnt FDB entry, the switch selects the bin + * based on a hash function, and the way within that bin incrementally. + */ +static inline int sja1105et_fdb_index(int bin, int way) +{ + return bin * SJA1105ET_FDB_BIN_SIZE + way; +} + +static int sja1105_is_fdb_entry_in_bin(struct sja1105_private *priv, int bin, + const u8 *addr, u16 vid, + struct sja1105_l2_lookup_entry *match, + int *last_unused) +{ + int way; + + for (way = 0; way < SJA1105ET_FDB_BIN_SIZE; way++) { + struct sja1105_l2_lookup_entry l2_lookup = {0}; + int index = sja1105et_fdb_index(bin, way); + + /* Skip unused entries, optionally marking them + * into the return value + */ + if (sja1105_dynamic_config_read(priv, BLK_IDX_L2_LOOKUP, + index, &l2_lookup)) { + if (last_unused) + *last_unused = way; + continue; + } + + if (l2_lookup.macaddr == ether_addr_to_u64(addr) && + l2_lookup.vlanid == vid) { + if (match) + *match = l2_lookup; + return way; + } + } + /* Return an invalid entry index if not found */ + return -1; +} + +static int sja1105_fdb_add(struct dsa_switch *ds, int port, + const unsigned char *addr, u16 vid) +{ + struct sja1105_l2_lookup_entry l2_lookup = {0}; + struct sja1105_private *priv = ds->priv; + struct device *dev = ds->dev; + int last_unused = -1; + int bin, way; + + bin = sja1105_fdb_hash(priv, addr, vid); + + way = sja1105_is_fdb_entry_in_bin(priv, bin, addr, vid, + &l2_lookup, &last_unused); + if (way >= 0) { + /* We have an FDB entry. Is our port in the destination + * mask? If yes, we need to do nothing. If not, we need + * to rewrite the entry by adding this port to it. + */ + if (l2_lookup.destports & BIT(port)) + return 0; + l2_lookup.destports |= BIT(port); + } else { + int index = sja1105et_fdb_index(bin, way); + + /* We don't have an FDB entry. We construct a new one and + * try to find a place for it within the FDB table. + */ + l2_lookup.macaddr = ether_addr_to_u64(addr); + l2_lookup.destports = BIT(port); + l2_lookup.vlanid = vid; + + if (last_unused >= 0) { + way = last_unused; + } else { + /* Bin is full, need to evict somebody. + * Choose victim at random. If you get these messages + * often, you may need to consider changing the + * distribution function: + * static_config[BLK_IDX_L2_LOOKUP_PARAMS].entries->poly + */ + get_random_bytes(&way, sizeof(u8)); + way %= SJA1105ET_FDB_BIN_SIZE; + dev_warn(dev, "Warning, FDB bin %d full while adding entry for %pM. Evicting entry %u.\n", + bin, addr, way); + /* Evict entry */ + sja1105_dynamic_config_write(priv, BLK_IDX_L2_LOOKUP, + index, NULL, false); + } + } + l2_lookup.index = sja1105et_fdb_index(bin, way); + + return sja1105_dynamic_config_write(priv, BLK_IDX_L2_LOOKUP, + l2_lookup.index, &l2_lookup, + true); +} + +static int sja1105_fdb_del(struct dsa_switch *ds, int port, + const unsigned char *addr, u16 vid) +{ + struct sja1105_l2_lookup_entry l2_lookup = {0}; + struct sja1105_private *priv = ds->priv; + int index, bin, way; + bool keep; + + bin = sja1105_fdb_hash(priv, addr, vid); + way = sja1105_is_fdb_entry_in_bin(priv, bin, addr, vid, + &l2_lookup, NULL); + if (way < 0) + return 0; + index = sja1105et_fdb_index(bin, way); + + /* We have an FDB entry. Is our port in the destination mask? If yes, + * we need to remove it. If the resulting port mask becomes empty, we + * need to completely evict the FDB entry. + * Otherwise we just write it back. + */ + if (l2_lookup.destports & BIT(port)) + l2_lookup.destports &= ~BIT(port); + if (l2_lookup.destports) + keep = true; + else + keep = false; + + return sja1105_dynamic_config_write(priv, BLK_IDX_L2_LOOKUP, + index, &l2_lookup, keep); +} + +static int sja1105_fdb_dump(struct dsa_switch *ds, int port, + dsa_fdb_dump_cb_t *cb, void *data) +{ + struct sja1105_private *priv = ds->priv; + struct device *dev = ds->dev; + int i; + + for (i = 0; i < SJA1105_MAX_L2_LOOKUP_COUNT; i++) { + struct sja1105_l2_lookup_entry l2_lookup = {0}; + u8 macaddr[ETH_ALEN]; + int rc; + + rc = sja1105_dynamic_config_read(priv, BLK_IDX_L2_LOOKUP, + i, &l2_lookup); + /* No fdb entry at i, not an issue */ + if (rc == -EINVAL) + continue; + if (rc) { + dev_err(dev, "Failed to dump FDB: %d\n", rc); + return rc; + } + + /* FDB dump callback is per port. This means we have to + * disregard a valid entry if it's not for this port, even if + * only to revisit it later. This is inefficient because the + * 1024-sized FDB table needs to be traversed 4 times through + * SPI during a 'bridge fdb show' command. + */ + if (!(l2_lookup.destports & BIT(port))) + continue; + u64_to_ether_addr(l2_lookup.macaddr, macaddr); + cb(macaddr, l2_lookup.vlanid, false, data); + } + return 0; +} + +/* This callback needs to be present */ +static int sja1105_mdb_prepare(struct dsa_switch *ds, int port, + const struct switchdev_obj_port_mdb *mdb) +{ + return 0; +} + +static void sja1105_mdb_add(struct dsa_switch *ds, int port, + const struct switchdev_obj_port_mdb *mdb) +{ + sja1105_fdb_add(ds, port, mdb->addr, mdb->vid); +} + +static int sja1105_mdb_del(struct dsa_switch *ds, int port, + const struct switchdev_obj_port_mdb *mdb) +{ + return sja1105_fdb_del(ds, port, mdb->addr, mdb->vid); +} + +static int sja1105_bridge_member(struct dsa_switch *ds, int port, + struct net_device *br, bool member) +{ + struct sja1105_l2_forwarding_entry *l2_fwd; + struct sja1105_private *priv = ds->priv; + int i, rc; + + l2_fwd = priv->static_config.tables[BLK_IDX_L2_FORWARDING].entries; + + for (i = 0; i < SJA1105_NUM_PORTS; i++) { + /* Add this port to the forwarding matrix of the + * other ports in the same bridge, and viceversa. + */ + if (!dsa_is_user_port(ds, i)) + continue; + /* For the ports already under the bridge, only one thing needs + * to be done, and that is to add this port to their + * reachability domain. So we can perform the SPI write for + * them immediately. However, for this port itself (the one + * that is new to the bridge), we need to add all other ports + * to its reachability domain. So we do that incrementally in + * this loop, and perform the SPI write only at the end, once + * the domain contains all other bridge ports. + */ + if (i == port) + continue; + if (dsa_to_port(ds, i)->bridge_dev != br) + continue; + sja1105_port_allow_traffic(l2_fwd, i, port, member); + sja1105_port_allow_traffic(l2_fwd, port, i, member); + + rc = sja1105_dynamic_config_write(priv, BLK_IDX_L2_FORWARDING, + i, &l2_fwd[i], true); + if (rc < 0) + return rc; + } + + return sja1105_dynamic_config_write(priv, BLK_IDX_L2_FORWARDING, + port, &l2_fwd[port], true); +} + +static void sja1105_bridge_stp_state_set(struct dsa_switch *ds, int port, + u8 state) +{ + struct sja1105_private *priv = ds->priv; + struct sja1105_mac_config_entry *mac; + + mac = priv->static_config.tables[BLK_IDX_MAC_CONFIG].entries; + + switch (state) { + case BR_STATE_DISABLED: + case BR_STATE_BLOCKING: + /* From UM10944 description of DRPDTAG (why put this there?): + * "Management traffic flows to the port regardless of the state + * of the INGRESS flag". So BPDUs are still be allowed to pass. + * At the moment no difference between DISABLED and BLOCKING. + */ + mac[port].ingress = false; + mac[port].egress = false; + mac[port].dyn_learn = false; + break; + case BR_STATE_LISTENING: + mac[port].ingress = true; + mac[port].egress = false; + mac[port].dyn_learn = false; + break; + case BR_STATE_LEARNING: + mac[port].ingress = true; + mac[port].egress = false; + mac[port].dyn_learn = true; + break; + case BR_STATE_FORWARDING: + mac[port].ingress = true; + mac[port].egress = true; + mac[port].dyn_learn = true; + break; + default: + dev_err(ds->dev, "invalid STP state: %d\n", state); + return; + } + + sja1105_dynamic_config_write(priv, BLK_IDX_MAC_CONFIG, port, + &mac[port], true); +} + +static int sja1105_bridge_join(struct dsa_switch *ds, int port, + struct net_device *br) +{ + return sja1105_bridge_member(ds, port, br, true); +} + +static void sja1105_bridge_leave(struct dsa_switch *ds, int port, + struct net_device *br) +{ + sja1105_bridge_member(ds, port, br, false); +} + +static u8 sja1105_stp_state_get(struct sja1105_private *priv, int port) +{ + struct sja1105_mac_config_entry *mac; + + mac = priv->static_config.tables[BLK_IDX_MAC_CONFIG].entries; + + if (!mac[port].ingress && !mac[port].egress && !mac[port].dyn_learn) + return BR_STATE_BLOCKING; + if (mac[port].ingress && !mac[port].egress && !mac[port].dyn_learn) + return BR_STATE_LISTENING; + if (mac[port].ingress && !mac[port].egress && mac[port].dyn_learn) + return BR_STATE_LEARNING; + if (mac[port].ingress && mac[port].egress && mac[port].dyn_learn) + return BR_STATE_FORWARDING; + return -EINVAL; +} + +/* For situations where we need to change a setting at runtime that is only + * available through the static configuration, resetting the switch in order + * to upload the new static config is unavoidable. Back up the settings we + * modify at runtime (currently only MAC) and restore them after uploading, + * such that this operation is relatively seamless. + */ +static int sja1105_static_config_reload(struct sja1105_private *priv) +{ + struct sja1105_mac_config_entry *mac; + int speed_mbps[SJA1105_NUM_PORTS]; + u8 stp_state[SJA1105_NUM_PORTS]; + int rc, i; + + mac = priv->static_config.tables[BLK_IDX_MAC_CONFIG].entries; + + /* Back up settings changed by sja1105_adjust_port_config and + * sja1105_bridge_stp_state_set and restore their defaults. + */ + for (i = 0; i < SJA1105_NUM_PORTS; i++) { + speed_mbps[i] = sja1105_speed[mac[i].speed]; + mac[i].speed = SJA1105_SPEED_AUTO; + if (i == dsa_upstream_port(priv->ds, i)) { + mac[i].ingress = true; + mac[i].egress = true; + mac[i].dyn_learn = true; + } else { + stp_state[i] = sja1105_stp_state_get(priv, i); + mac[i].ingress = false; + mac[i].egress = false; + mac[i].dyn_learn = false; + } + } + + /* Reset switch and send updated static configuration */ + rc = sja1105_static_config_upload(priv); + if (rc < 0) + goto out; + + /* Configure the CGU (PLLs) for MII and RMII PHYs. + * For these interfaces there is no dynamic configuration + * needed, since PLLs have same settings at all speeds. + */ + rc = sja1105_clocking_setup(priv); + if (rc < 0) + goto out; + + for (i = 0; i < SJA1105_NUM_PORTS; i++) { + bool enabled = (speed_mbps[i] != 0); + + if (i != dsa_upstream_port(priv->ds, i)) + sja1105_bridge_stp_state_set(priv->ds, i, stp_state[i]); + + rc = sja1105_adjust_port_config(priv, i, speed_mbps[i], + enabled); + if (rc < 0) + goto out; + } +out: + return rc; +} + +/* The TPID setting belongs to the General Parameters table, + * which can only be partially reconfigured at runtime (and not the TPID). + * So a switch reset is required. + */ +static int sja1105_change_tpid(struct sja1105_private *priv, + u16 tpid, u16 tpid2) +{ + struct sja1105_general_params_entry *general_params; + struct sja1105_table *table; + + table = &priv->static_config.tables[BLK_IDX_GENERAL_PARAMS]; + general_params = table->entries; + general_params->tpid = tpid; + general_params->tpid2 = tpid2; + return sja1105_static_config_reload(priv); +} + +static int sja1105_pvid_apply(struct sja1105_private *priv, int port, u16 pvid) +{ + struct sja1105_mac_config_entry *mac; + + mac = priv->static_config.tables[BLK_IDX_MAC_CONFIG].entries; + + mac[port].vlanid = pvid; + + return sja1105_dynamic_config_write(priv, BLK_IDX_MAC_CONFIG, port, + &mac[port], true); +} + +static int sja1105_is_vlan_configured(struct sja1105_private *priv, u16 vid) +{ + struct sja1105_vlan_lookup_entry *vlan; + int count, i; + + vlan = priv->static_config.tables[BLK_IDX_VLAN_LOOKUP].entries; + count = priv->static_config.tables[BLK_IDX_VLAN_LOOKUP].entry_count; + + for (i = 0; i < count; i++) + if (vlan[i].vlanid == vid) + return i; + + /* Return an invalid entry index if not found */ + return -1; +} + +static int sja1105_vlan_apply(struct sja1105_private *priv, int port, u16 vid, + bool enabled, bool untagged) +{ + struct sja1105_vlan_lookup_entry *vlan; + struct sja1105_table *table; + bool keep = true; + int match, rc; + + table = &priv->static_config.tables[BLK_IDX_VLAN_LOOKUP]; + + match = sja1105_is_vlan_configured(priv, vid); + if (match < 0) { + /* Can't delete a missing entry. */ + if (!enabled) + return 0; + rc = sja1105_table_resize(table, table->entry_count + 1); + if (rc) + return rc; + match = table->entry_count - 1; + } + /* Assign pointer after the resize (it's new memory) */ + vlan = table->entries; + vlan[match].vlanid = vid; + if (enabled) { + vlan[match].vlan_bc |= BIT(port); + vlan[match].vmemb_port |= BIT(port); + } else { + vlan[match].vlan_bc &= ~BIT(port); + vlan[match].vmemb_port &= ~BIT(port); + } + /* Also unset tag_port if removing this VLAN was requested, + * just so we don't have a confusing bitmap (no practical purpose). + */ + if (untagged || !enabled) + vlan[match].tag_port &= ~BIT(port); + else + vlan[match].tag_port |= BIT(port); + /* If there's no port left as member of this VLAN, + * it's time for it to go. + */ + if (!vlan[match].vmemb_port) + keep = false; + + dev_dbg(priv->ds->dev, + "%s: port %d, vid %llu, broadcast domain 0x%llx, " + "port members 0x%llx, tagged ports 0x%llx, keep %d\n", + __func__, port, vlan[match].vlanid, vlan[match].vlan_bc, + vlan[match].vmemb_port, vlan[match].tag_port, keep); + + rc = sja1105_dynamic_config_write(priv, BLK_IDX_VLAN_LOOKUP, vid, + &vlan[match], keep); + if (rc < 0) + return rc; + + if (!keep) + return sja1105_table_delete_entry(table, match); + + return 0; +} + +static int sja1105_setup_8021q_tagging(struct dsa_switch *ds, bool enabled) +{ + int rc, i; + + for (i = 0; i < SJA1105_NUM_PORTS; i++) { + rc = dsa_port_setup_8021q_tagging(ds, i, enabled); + if (rc < 0) { + dev_err(ds->dev, "Failed to setup VLAN tagging for port %d: %d\n", + i, rc); + return rc; + } + } + dev_info(ds->dev, "%s switch tagging\n", + enabled ? "Enabled" : "Disabled"); + return 0; +} + +static enum dsa_tag_protocol +sja1105_get_tag_protocol(struct dsa_switch *ds, int port) +{ + return DSA_TAG_PROTO_SJA1105; +} + +/* This callback needs to be present */ +static int sja1105_vlan_prepare(struct dsa_switch *ds, int port, + const struct switchdev_obj_port_vlan *vlan) +{ + return 0; +} + +static int sja1105_vlan_filtering(struct dsa_switch *ds, int port, bool enabled) +{ + struct sja1105_private *priv = ds->priv; + int rc; + + if (enabled) + /* Enable VLAN filtering. */ + rc = sja1105_change_tpid(priv, ETH_P_8021Q, ETH_P_8021AD); + else + /* Disable VLAN filtering. */ + rc = sja1105_change_tpid(priv, ETH_P_SJA1105, ETH_P_SJA1105); + if (rc) + dev_err(ds->dev, "Failed to change VLAN Ethertype\n"); + + /* Switch port identification based on 802.1Q is only passable + * if we are not under a vlan_filtering bridge. So make sure + * the two configurations are mutually exclusive. + */ + return sja1105_setup_8021q_tagging(ds, !enabled); +} + +static void sja1105_vlan_add(struct dsa_switch *ds, int port, + const struct switchdev_obj_port_vlan *vlan) +{ + struct sja1105_private *priv = ds->priv; + u16 vid; + int rc; + + for (vid = vlan->vid_begin; vid <= vlan->vid_end; vid++) { + rc = sja1105_vlan_apply(priv, port, vid, true, vlan->flags & + BRIDGE_VLAN_INFO_UNTAGGED); + if (rc < 0) { + dev_err(ds->dev, "Failed to add VLAN %d to port %d: %d\n", + vid, port, rc); + return; + } + if (vlan->flags & BRIDGE_VLAN_INFO_PVID) { + rc = sja1105_pvid_apply(ds->priv, port, vid); + if (rc < 0) { + dev_err(ds->dev, "Failed to set pvid %d on port %d: %d\n", + vid, port, rc); + return; + } + } + } +} + +static int sja1105_vlan_del(struct dsa_switch *ds, int port, + const struct switchdev_obj_port_vlan *vlan) +{ + struct sja1105_private *priv = ds->priv; + u16 vid; + int rc; + + for (vid = vlan->vid_begin; vid <= vlan->vid_end; vid++) { + rc = sja1105_vlan_apply(priv, port, vid, false, vlan->flags & + BRIDGE_VLAN_INFO_UNTAGGED); + if (rc < 0) { + dev_err(ds->dev, "Failed to remove VLAN %d from port %d: %d\n", + vid, port, rc); + return rc; + } + } + return 0; +} + +/* The programming model for the SJA1105 switch is "all-at-once" via static + * configuration tables. Some of these can be dynamically modified at runtime, + * but not the xMII mode parameters table. + * Furthermode, some PHYs may not have crystals for generating their clocks + * (e.g. RMII). Instead, their 50MHz clock is supplied via the SJA1105 port's + * ref_clk pin. So port clocking needs to be initialized early, before + * connecting to PHYs is attempted, otherwise they won't respond through MDIO. + * Setting correct PHY link speed does not matter now. + * But dsa_slave_phy_setup is called later than sja1105_setup, so the PHY + * bindings are not yet parsed by DSA core. We need to parse early so that we + * can populate the xMII mode parameters table. + */ +static int sja1105_setup(struct dsa_switch *ds) +{ + struct sja1105_dt_port ports[SJA1105_NUM_PORTS]; + struct sja1105_private *priv = ds->priv; + int rc; + + rc = sja1105_parse_dt(priv, ports); + if (rc < 0) { + dev_err(ds->dev, "Failed to parse DT: %d\n", rc); + return rc; + } + + /* Error out early if internal delays are required through DT + * and we can't apply them. + */ + rc = sja1105_parse_rgmii_delays(priv, ports); + if (rc < 0) { + dev_err(ds->dev, "RGMII delay not supported\n"); + return rc; + } + + /* Create and send configuration down to device */ + rc = sja1105_static_config_load(priv, ports); + if (rc < 0) { + dev_err(ds->dev, "Failed to load static config: %d\n", rc); + return rc; + } + /* Configure the CGU (PHY link modes and speeds) */ + rc = sja1105_clocking_setup(priv); + if (rc < 0) { + dev_err(ds->dev, "Failed to configure MII clocking: %d\n", rc); + return rc; + } + /* On SJA1105, VLAN filtering per se is always enabled in hardware. + * The only thing we can do to disable it is lie about what the 802.1Q + * EtherType is. + * So it will still try to apply VLAN filtering, but all ingress + * traffic (except frames received with EtherType of ETH_P_SJA1105) + * will be internally tagged with a distorted VLAN header where the + * TPID is ETH_P_SJA1105, and the VLAN ID is the port pvid. + */ + ds->vlan_filtering_is_global = true; + + /* The DSA/switchdev model brings up switch ports in standalone mode by + * default, and that means vlan_filtering is 0 since they're not under + * a bridge, so it's safe to set up switch tagging at this time. + */ + return sja1105_setup_8021q_tagging(ds, true); +} + +static int sja1105_mgmt_xmit(struct dsa_switch *ds, int port, int slot, + struct sk_buff *skb) +{ + struct sja1105_mgmt_entry mgmt_route = {0}; + struct sja1105_private *priv = ds->priv; + struct ethhdr *hdr; + int timeout = 10; + int rc; + + hdr = eth_hdr(skb); + + mgmt_route.macaddr = ether_addr_to_u64(hdr->h_dest); + mgmt_route.destports = BIT(port); + mgmt_route.enfport = 1; + + rc = sja1105_dynamic_config_write(priv, BLK_IDX_MGMT_ROUTE, + slot, &mgmt_route, true); + if (rc < 0) { + kfree_skb(skb); + return rc; + } + + /* Transfer skb to the host port. */ + dsa_enqueue_skb(skb, ds->ports[port].slave); + + /* Wait until the switch has processed the frame */ + do { + rc = sja1105_dynamic_config_read(priv, BLK_IDX_MGMT_ROUTE, + slot, &mgmt_route); + if (rc < 0) { + dev_err_ratelimited(priv->ds->dev, + "failed to poll for mgmt route\n"); + continue; + } + + /* UM10944: The ENFPORT flag of the respective entry is + * cleared when a match is found. The host can use this + * flag as an acknowledgment. + */ + cpu_relax(); + } while (mgmt_route.enfport && --timeout); + + if (!timeout) { + /* Clean up the management route so that a follow-up + * frame may not match on it by mistake. + */ + sja1105_dynamic_config_write(priv, BLK_IDX_MGMT_ROUTE, + slot, &mgmt_route, false); + dev_err_ratelimited(priv->ds->dev, "xmit timed out\n"); + } + + return NETDEV_TX_OK; +} + +/* Deferred work is unfortunately necessary because setting up the management + * route cannot be done from atomit context (SPI transfer takes a sleepable + * lock on the bus) + */ +static netdev_tx_t sja1105_port_deferred_xmit(struct dsa_switch *ds, int port, + struct sk_buff *skb) +{ + struct sja1105_private *priv = ds->priv; + struct sja1105_port *sp = &priv->ports[port]; + int slot = sp->mgmt_slot; + + /* The tragic fact about the switch having 4x2 slots for installing + * management routes is that all of them except one are actually + * useless. + * If 2 slots are simultaneously configured for two BPDUs sent to the + * same (multicast) DMAC but on different egress ports, the switch + * would confuse them and redirect first frame it receives on the CPU + * port towards the port configured on the numerically first slot + * (therefore wrong port), then second received frame on second slot + * (also wrong port). + * So for all practical purposes, there needs to be a lock that + * prevents that from happening. The slot used here is utterly useless + * (could have simply been 0 just as fine), but we are doing it + * nonetheless, in case a smarter idea ever comes up in the future. + */ + mutex_lock(&priv->mgmt_lock); + + sja1105_mgmt_xmit(ds, port, slot, skb); + + mutex_unlock(&priv->mgmt_lock); + return NETDEV_TX_OK; +} + +/* The MAXAGE setting belongs to the L2 Forwarding Parameters table, + * which cannot be reconfigured at runtime. So a switch reset is required. + */ +static int sja1105_set_ageing_time(struct dsa_switch *ds, + unsigned int ageing_time) +{ + struct sja1105_l2_lookup_params_entry *l2_lookup_params; + struct sja1105_private *priv = ds->priv; + struct sja1105_table *table; + unsigned int maxage; + + table = &priv->static_config.tables[BLK_IDX_L2_LOOKUP_PARAMS]; + l2_lookup_params = table->entries; + + maxage = SJA1105_AGEING_TIME_MS(ageing_time); + + if (l2_lookup_params->maxage == maxage) + return 0; + + l2_lookup_params->maxage = maxage; + + return sja1105_static_config_reload(priv); +} + +static const struct dsa_switch_ops sja1105_switch_ops = { + .get_tag_protocol = sja1105_get_tag_protocol, + .setup = sja1105_setup, + .adjust_link = sja1105_adjust_link, + .set_ageing_time = sja1105_set_ageing_time, + .phylink_validate = sja1105_phylink_validate, + .get_strings = sja1105_get_strings, + .get_ethtool_stats = sja1105_get_ethtool_stats, + .get_sset_count = sja1105_get_sset_count, + .port_fdb_dump = sja1105_fdb_dump, + .port_fdb_add = sja1105_fdb_add, + .port_fdb_del = sja1105_fdb_del, + .port_bridge_join = sja1105_bridge_join, + .port_bridge_leave = sja1105_bridge_leave, + .port_stp_state_set = sja1105_bridge_stp_state_set, + .port_vlan_prepare = sja1105_vlan_prepare, + .port_vlan_filtering = sja1105_vlan_filtering, + .port_vlan_add = sja1105_vlan_add, + .port_vlan_del = sja1105_vlan_del, + .port_mdb_prepare = sja1105_mdb_prepare, + .port_mdb_add = sja1105_mdb_add, + .port_mdb_del = sja1105_mdb_del, + .port_deferred_xmit = sja1105_port_deferred_xmit, +}; + +static int sja1105_check_device_id(struct sja1105_private *priv) +{ + const struct sja1105_regs *regs = priv->info->regs; + u8 prod_id[SJA1105_SIZE_DEVICE_ID] = {0}; + struct device *dev = &priv->spidev->dev; + u64 device_id; + u64 part_no; + int rc; + + rc = sja1105_spi_send_int(priv, SPI_READ, regs->device_id, + &device_id, SJA1105_SIZE_DEVICE_ID); + if (rc < 0) + return rc; + + if (device_id != priv->info->device_id) { + dev_err(dev, "Expected device ID 0x%llx but read 0x%llx\n", + priv->info->device_id, device_id); + return -ENODEV; + } + + rc = sja1105_spi_send_packed_buf(priv, SPI_READ, regs->prod_id, + prod_id, SJA1105_SIZE_DEVICE_ID); + if (rc < 0) + return rc; + + sja1105_unpack(prod_id, &part_no, 19, 4, SJA1105_SIZE_DEVICE_ID); + + if (part_no != priv->info->part_no) { + dev_err(dev, "Expected part number 0x%llx but read 0x%llx\n", + priv->info->part_no, part_no); + return -ENODEV; + } + + return 0; +} + +static int sja1105_probe(struct spi_device *spi) +{ + struct device *dev = &spi->dev; + struct sja1105_private *priv; + struct dsa_switch *ds; + int rc, i; + + if (!dev->of_node) { + dev_err(dev, "No DTS bindings for SJA1105 driver\n"); + return -EINVAL; + } + + priv = devm_kzalloc(dev, sizeof(struct sja1105_private), GFP_KERNEL); + if (!priv) + return -ENOMEM; + + /* Configure the optional reset pin and bring up switch */ + priv->reset_gpio = devm_gpiod_get(dev, "reset", GPIOD_OUT_HIGH); + if (IS_ERR(priv->reset_gpio)) + dev_dbg(dev, "reset-gpios not defined, ignoring\n"); + else + sja1105_hw_reset(priv->reset_gpio, 1, 1); + + /* Populate our driver private structure (priv) based on + * the device tree node that was probed (spi) + */ + priv->spidev = spi; + spi_set_drvdata(spi, priv); + + /* Configure the SPI bus */ + spi->bits_per_word = 8; + rc = spi_setup(spi); + if (rc < 0) { + dev_err(dev, "Could not init SPI\n"); + return rc; + } + + priv->info = of_device_get_match_data(dev); + + /* Detect hardware device */ + rc = sja1105_check_device_id(priv); + if (rc < 0) { + dev_err(dev, "Device ID check failed: %d\n", rc); + return rc; + } + + dev_info(dev, "Probed switch chip: %s\n", priv->info->name); + + ds = dsa_switch_alloc(dev, SJA1105_NUM_PORTS); + if (!ds) + return -ENOMEM; + + ds->ops = &sja1105_switch_ops; + ds->priv = priv; + priv->ds = ds; + + /* Connections between dsa_port and sja1105_port */ + for (i = 0; i < SJA1105_NUM_PORTS; i++) { + struct sja1105_port *sp = &priv->ports[i]; + + ds->ports[i].priv = sp; + sp->dp = &ds->ports[i]; + } + mutex_init(&priv->mgmt_lock); + + return dsa_register_switch(priv->ds); +} + +static int sja1105_remove(struct spi_device *spi) +{ + struct sja1105_private *priv = spi_get_drvdata(spi); + + dsa_unregister_switch(priv->ds); + sja1105_static_config_free(&priv->static_config); + return 0; +} + +static const struct of_device_id sja1105_dt_ids[] = { + { .compatible = "nxp,sja1105e", .data = &sja1105e_info }, + { .compatible = "nxp,sja1105t", .data = &sja1105t_info }, + { .compatible = "nxp,sja1105p", .data = &sja1105p_info }, + { .compatible = "nxp,sja1105q", .data = &sja1105q_info }, + { .compatible = "nxp,sja1105r", .data = &sja1105r_info }, + { .compatible = "nxp,sja1105s", .data = &sja1105s_info }, + { /* sentinel */ }, +}; +MODULE_DEVICE_TABLE(of, sja1105_dt_ids); + +static struct spi_driver sja1105_driver = { + .driver = { + .name = "sja1105", + .owner = THIS_MODULE, + .of_match_table = of_match_ptr(sja1105_dt_ids), + }, + .probe = sja1105_probe, + .remove = sja1105_remove, +}; + +module_spi_driver(sja1105_driver); + +MODULE_AUTHOR("Vladimir Oltean "); +MODULE_AUTHOR("Georg Waibel "); +MODULE_DESCRIPTION("SJA1105 Driver"); +MODULE_LICENSE("GPL v2"); diff --git a/drivers/net/dsa/sja1105/sja1105_spi.c b/drivers/net/dsa/sja1105/sja1105_spi.c new file mode 100644 index 0000000000000000000000000000000000000000..244a94ccfc185a74fb154d2e0d1c57775f082c32 --- /dev/null +++ b/drivers/net/dsa/sja1105/sja1105_spi.c @@ -0,0 +1,590 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* Copyright (c) 2016-2018, NXP Semiconductors + * Copyright (c) 2018, Sensor-Technik Wiedemann GmbH + * Copyright (c) 2018-2019, Vladimir Oltean + */ +#include +#include +#include "sja1105.h" + +#define SJA1105_SIZE_PORT_CTRL 4 +#define SJA1105_SIZE_RESET_CMD 4 +#define SJA1105_SIZE_SPI_MSG_HEADER 4 +#define SJA1105_SIZE_SPI_MSG_MAXLEN (64 * 4) +#define SJA1105_SIZE_SPI_TRANSFER_MAX \ + (SJA1105_SIZE_SPI_MSG_HEADER + SJA1105_SIZE_SPI_MSG_MAXLEN) + +static int sja1105_spi_transfer(const struct sja1105_private *priv, + const void *tx, void *rx, int size) +{ + struct spi_device *spi = priv->spidev; + struct spi_transfer transfer = { + .tx_buf = tx, + .rx_buf = rx, + .len = size, + }; + struct spi_message msg; + int rc; + + if (size > SJA1105_SIZE_SPI_TRANSFER_MAX) { + dev_err(&spi->dev, "SPI message (%d) longer than max of %d\n", + size, SJA1105_SIZE_SPI_TRANSFER_MAX); + return -EMSGSIZE; + } + + spi_message_init(&msg); + spi_message_add_tail(&transfer, &msg); + + rc = spi_sync(spi, &msg); + if (rc < 0) { + dev_err(&spi->dev, "SPI transfer failed: %d\n", rc); + return rc; + } + + return rc; +} + +static void +sja1105_spi_message_pack(void *buf, const struct sja1105_spi_message *msg) +{ + const int size = SJA1105_SIZE_SPI_MSG_HEADER; + + memset(buf, 0, size); + + sja1105_pack(buf, &msg->access, 31, 31, size); + sja1105_pack(buf, &msg->read_count, 30, 25, size); + sja1105_pack(buf, &msg->address, 24, 4, size); +} + +/* If @rw is: + * - SPI_WRITE: creates and sends an SPI write message at absolute + * address reg_addr, taking size_bytes from *packed_buf + * - SPI_READ: creates and sends an SPI read message from absolute + * address reg_addr, writing size_bytes into *packed_buf + * + * This function should only be called if it is priorly known that + * @size_bytes is smaller than SIZE_SPI_MSG_MAXLEN. Larger packed buffers + * are chunked in smaller pieces by sja1105_spi_send_long_packed_buf below. + */ +int sja1105_spi_send_packed_buf(const struct sja1105_private *priv, + sja1105_spi_rw_mode_t rw, u64 reg_addr, + void *packed_buf, size_t size_bytes) +{ + u8 tx_buf[SJA1105_SIZE_SPI_TRANSFER_MAX] = {0}; + u8 rx_buf[SJA1105_SIZE_SPI_TRANSFER_MAX] = {0}; + const int msg_len = size_bytes + SJA1105_SIZE_SPI_MSG_HEADER; + struct sja1105_spi_message msg = {0}; + int rc; + + if (msg_len > SJA1105_SIZE_SPI_TRANSFER_MAX) + return -ERANGE; + + msg.access = rw; + msg.address = reg_addr; + if (rw == SPI_READ) + msg.read_count = size_bytes / 4; + + sja1105_spi_message_pack(tx_buf, &msg); + + if (rw == SPI_WRITE) + memcpy(tx_buf + SJA1105_SIZE_SPI_MSG_HEADER, + packed_buf, size_bytes); + + rc = sja1105_spi_transfer(priv, tx_buf, rx_buf, msg_len); + if (rc < 0) + return rc; + + if (rw == SPI_READ) + memcpy(packed_buf, rx_buf + SJA1105_SIZE_SPI_MSG_HEADER, + size_bytes); + + return 0; +} + +/* If @rw is: + * - SPI_WRITE: creates and sends an SPI write message at absolute + * address reg_addr, taking size_bytes from *packed_buf + * - SPI_READ: creates and sends an SPI read message from absolute + * address reg_addr, writing size_bytes into *packed_buf + * + * The u64 *value is unpacked, meaning that it's stored in the native + * CPU endianness and directly usable by software running on the core. + * + * This is a wrapper around sja1105_spi_send_packed_buf(). + */ +int sja1105_spi_send_int(const struct sja1105_private *priv, + sja1105_spi_rw_mode_t rw, u64 reg_addr, + u64 *value, u64 size_bytes) +{ + u8 packed_buf[SJA1105_SIZE_SPI_MSG_MAXLEN]; + int rc; + + if (size_bytes > SJA1105_SIZE_SPI_MSG_MAXLEN) + return -ERANGE; + + if (rw == SPI_WRITE) + sja1105_pack(packed_buf, value, 8 * size_bytes - 1, 0, + size_bytes); + + rc = sja1105_spi_send_packed_buf(priv, rw, reg_addr, packed_buf, + size_bytes); + + if (rw == SPI_READ) + sja1105_unpack(packed_buf, value, 8 * size_bytes - 1, 0, + size_bytes); + + return rc; +} + +/* Should be used if a @packed_buf larger than SJA1105_SIZE_SPI_MSG_MAXLEN + * must be sent/received. Splitting the buffer into chunks and assembling + * those into SPI messages is done automatically by this function. + */ +int sja1105_spi_send_long_packed_buf(const struct sja1105_private *priv, + sja1105_spi_rw_mode_t rw, u64 base_addr, + void *packed_buf, u64 buf_len) +{ + struct chunk { + void *buf_ptr; + int len; + u64 spi_address; + } chunk; + int distance_to_end; + int rc; + + /* Initialize chunk */ + chunk.buf_ptr = packed_buf; + chunk.spi_address = base_addr; + chunk.len = min_t(int, buf_len, SJA1105_SIZE_SPI_MSG_MAXLEN); + + while (chunk.len) { + rc = sja1105_spi_send_packed_buf(priv, rw, chunk.spi_address, + chunk.buf_ptr, chunk.len); + if (rc < 0) + return rc; + + chunk.buf_ptr += chunk.len; + chunk.spi_address += chunk.len / 4; + distance_to_end = (uintptr_t)(packed_buf + buf_len - + chunk.buf_ptr); + chunk.len = min(distance_to_end, SJA1105_SIZE_SPI_MSG_MAXLEN); + } + + return 0; +} + +/* Back-ported structure from UM11040 Table 112. + * Reset control register (addr. 100440h) + * In the SJA1105 E/T, only warm_rst and cold_rst are + * supported (exposed in UM10944 as rst_ctrl), but the bit + * offsets of warm_rst and cold_rst are actually reversed. + */ +struct sja1105_reset_cmd { + u64 switch_rst; + u64 cfg_rst; + u64 car_rst; + u64 otp_rst; + u64 warm_rst; + u64 cold_rst; + u64 por_rst; +}; + +static void +sja1105et_reset_cmd_pack(void *buf, const struct sja1105_reset_cmd *reset) +{ + const int size = SJA1105_SIZE_RESET_CMD; + + memset(buf, 0, size); + + sja1105_pack(buf, &reset->cold_rst, 3, 3, size); + sja1105_pack(buf, &reset->warm_rst, 2, 2, size); +} + +static void +sja1105pqrs_reset_cmd_pack(void *buf, const struct sja1105_reset_cmd *reset) +{ + const int size = SJA1105_SIZE_RESET_CMD; + + memset(buf, 0, size); + + sja1105_pack(buf, &reset->switch_rst, 8, 8, size); + sja1105_pack(buf, &reset->cfg_rst, 7, 7, size); + sja1105_pack(buf, &reset->car_rst, 5, 5, size); + sja1105_pack(buf, &reset->otp_rst, 4, 4, size); + sja1105_pack(buf, &reset->warm_rst, 3, 3, size); + sja1105_pack(buf, &reset->cold_rst, 2, 2, size); + sja1105_pack(buf, &reset->por_rst, 1, 1, size); +} + +static int sja1105et_reset_cmd(const void *ctx, const void *data) +{ + const struct sja1105_private *priv = ctx; + const struct sja1105_reset_cmd *reset = data; + const struct sja1105_regs *regs = priv->info->regs; + struct device *dev = priv->ds->dev; + u8 packed_buf[SJA1105_SIZE_RESET_CMD]; + + if (reset->switch_rst || + reset->cfg_rst || + reset->car_rst || + reset->otp_rst || + reset->por_rst) { + dev_err(dev, "Only warm and cold reset is supported " + "for SJA1105 E/T!\n"); + return -EINVAL; + } + + if (reset->warm_rst) + dev_dbg(dev, "Warm reset requested\n"); + if (reset->cold_rst) + dev_dbg(dev, "Cold reset requested\n"); + + sja1105et_reset_cmd_pack(packed_buf, reset); + + return sja1105_spi_send_packed_buf(priv, SPI_WRITE, regs->rgu, + packed_buf, SJA1105_SIZE_RESET_CMD); +} + +static int sja1105pqrs_reset_cmd(const void *ctx, const void *data) +{ + const struct sja1105_private *priv = ctx; + const struct sja1105_reset_cmd *reset = data; + const struct sja1105_regs *regs = priv->info->regs; + struct device *dev = priv->ds->dev; + u8 packed_buf[SJA1105_SIZE_RESET_CMD]; + + if (reset->switch_rst) + dev_dbg(dev, "Main reset for all functional modules requested\n"); + if (reset->cfg_rst) + dev_dbg(dev, "Chip configuration reset requested\n"); + if (reset->car_rst) + dev_dbg(dev, "Clock and reset control logic reset requested\n"); + if (reset->otp_rst) + dev_dbg(dev, "OTP read cycle for reading product " + "config settings requested\n"); + if (reset->warm_rst) + dev_dbg(dev, "Warm reset requested\n"); + if (reset->cold_rst) + dev_dbg(dev, "Cold reset requested\n"); + if (reset->por_rst) + dev_dbg(dev, "Power-on reset requested\n"); + + sja1105pqrs_reset_cmd_pack(packed_buf, reset); + + return sja1105_spi_send_packed_buf(priv, SPI_WRITE, regs->rgu, + packed_buf, SJA1105_SIZE_RESET_CMD); +} + +static int sja1105_cold_reset(const struct sja1105_private *priv) +{ + struct sja1105_reset_cmd reset = {0}; + + reset.cold_rst = 1; + return priv->info->reset_cmd(priv, &reset); +} + +static int sja1105_inhibit_tx(const struct sja1105_private *priv, + const unsigned long *port_bitmap) +{ + const struct sja1105_regs *regs = priv->info->regs; + u64 inhibit_cmd; + int port, rc; + + rc = sja1105_spi_send_int(priv, SPI_READ, regs->port_control, + &inhibit_cmd, SJA1105_SIZE_PORT_CTRL); + if (rc < 0) + return rc; + + for_each_set_bit(port, port_bitmap, SJA1105_NUM_PORTS) + inhibit_cmd |= BIT(port); + + return sja1105_spi_send_int(priv, SPI_WRITE, regs->port_control, + &inhibit_cmd, SJA1105_SIZE_PORT_CTRL); +} + +struct sja1105_status { + u64 configs; + u64 crcchkl; + u64 ids; + u64 crcchkg; +}; + +/* This is not reading the entire General Status area, which is also + * divergent between E/T and P/Q/R/S, but only the relevant bits for + * ensuring that the static config upload procedure was successful. + */ +static void sja1105_status_unpack(void *buf, struct sja1105_status *status) +{ + /* So that addition translates to 4 bytes */ + u32 *p = buf; + + /* device_id is missing from the buffer, but we don't + * want to diverge from the manual definition of the + * register addresses, so we'll back off one step with + * the register pointer, and never access p[0]. + */ + p--; + sja1105_unpack(p + 0x1, &status->configs, 31, 31, 4); + sja1105_unpack(p + 0x1, &status->crcchkl, 30, 30, 4); + sja1105_unpack(p + 0x1, &status->ids, 29, 29, 4); + sja1105_unpack(p + 0x1, &status->crcchkg, 28, 28, 4); +} + +static int sja1105_status_get(struct sja1105_private *priv, + struct sja1105_status *status) +{ + const struct sja1105_regs *regs = priv->info->regs; + u8 packed_buf[4]; + int rc; + + rc = sja1105_spi_send_packed_buf(priv, SPI_READ, + regs->status, + packed_buf, 4); + if (rc < 0) + return rc; + + sja1105_status_unpack(packed_buf, status); + + return 0; +} + +/* Not const because unpacking priv->static_config into buffers and preparing + * for upload requires the recalculation of table CRCs and updating the + * structures with these. + */ +static int +static_config_buf_prepare_for_upload(struct sja1105_private *priv, + void *config_buf, int buf_len) +{ + struct sja1105_static_config *config = &priv->static_config; + struct sja1105_table_header final_header; + sja1105_config_valid_t valid; + char *final_header_ptr; + int crc_len; + + valid = sja1105_static_config_check_valid(config); + if (valid != SJA1105_CONFIG_OK) { + dev_err(&priv->spidev->dev, + sja1105_static_config_error_msg[valid]); + return -EINVAL; + } + + /* Write Device ID and config tables to config_buf */ + sja1105_static_config_pack(config_buf, config); + /* Recalculate CRC of the last header (right now 0xDEADBEEF). + * Don't include the CRC field itself. + */ + crc_len = buf_len - 4; + /* Read the whole table header */ + final_header_ptr = config_buf + buf_len - SJA1105_SIZE_TABLE_HEADER; + sja1105_table_header_packing(final_header_ptr, &final_header, UNPACK); + /* Modify */ + final_header.crc = sja1105_crc32(config_buf, crc_len); + /* Rewrite */ + sja1105_table_header_packing(final_header_ptr, &final_header, PACK); + + return 0; +} + +#define RETRIES 10 + +int sja1105_static_config_upload(struct sja1105_private *priv) +{ + unsigned long port_bitmap = GENMASK_ULL(SJA1105_NUM_PORTS - 1, 0); + struct sja1105_static_config *config = &priv->static_config; + const struct sja1105_regs *regs = priv->info->regs; + struct device *dev = &priv->spidev->dev; + struct sja1105_status status; + int rc, retries = RETRIES; + u8 *config_buf; + int buf_len; + + buf_len = sja1105_static_config_get_length(config); + config_buf = kcalloc(buf_len, sizeof(char), GFP_KERNEL); + if (!config_buf) + return -ENOMEM; + + rc = static_config_buf_prepare_for_upload(priv, config_buf, buf_len); + if (rc < 0) { + dev_err(dev, "Invalid config, cannot upload\n"); + return -EINVAL; + } + /* Prevent PHY jabbering during switch reset by inhibiting + * Tx on all ports and waiting for current packet to drain. + * Otherwise, the PHY will see an unterminated Ethernet packet. + */ + rc = sja1105_inhibit_tx(priv, &port_bitmap); + if (rc < 0) { + dev_err(dev, "Failed to inhibit Tx on ports\n"); + return -ENXIO; + } + /* Wait for an eventual egress packet to finish transmission + * (reach IFG). It is guaranteed that a second one will not + * follow, and that switch cold reset is thus safe + */ + usleep_range(500, 1000); + do { + /* Put the SJA1105 in programming mode */ + rc = sja1105_cold_reset(priv); + if (rc < 0) { + dev_err(dev, "Failed to reset switch, retrying...\n"); + continue; + } + /* Wait for the switch to come out of reset */ + usleep_range(1000, 5000); + /* Upload the static config to the device */ + rc = sja1105_spi_send_long_packed_buf(priv, SPI_WRITE, + regs->config, + config_buf, buf_len); + if (rc < 0) { + dev_err(dev, "Failed to upload config, retrying...\n"); + continue; + } + /* Check that SJA1105 responded well to the config upload */ + rc = sja1105_status_get(priv, &status); + if (rc < 0) + continue; + + if (status.ids == 1) { + dev_err(dev, "Mismatch between hardware and static config " + "device id. Wrote 0x%llx, wants 0x%llx\n", + config->device_id, priv->info->device_id); + continue; + } + if (status.crcchkl == 1) { + dev_err(dev, "Switch reported invalid local CRC on " + "the uploaded config, retrying...\n"); + continue; + } + if (status.crcchkg == 1) { + dev_err(dev, "Switch reported invalid global CRC on " + "the uploaded config, retrying...\n"); + continue; + } + if (status.configs == 0) { + dev_err(dev, "Switch reported that configuration is " + "invalid, retrying...\n"); + continue; + } + } while (--retries && (status.crcchkl == 1 || status.crcchkg == 1 || + status.configs == 0 || status.ids == 1)); + + if (!retries) { + rc = -EIO; + dev_err(dev, "Failed to upload config to device, giving up\n"); + goto out; + } else if (retries != RETRIES - 1) { + dev_info(dev, "Succeeded after %d tried\n", RETRIES - retries); + } + + dev_info(dev, "Reset switch and programmed static config\n"); +out: + kfree(config_buf); + return rc; +} + +struct sja1105_regs sja1105et_regs = { + .device_id = 0x0, + .prod_id = 0x100BC3, + .status = 0x1, + .port_control = 0x11, + .config = 0x020000, + .rgu = 0x100440, + .pad_mii_tx = {0x100800, 0x100802, 0x100804, 0x100806, 0x100808}, + .rmii_pll1 = 0x10000A, + .cgu_idiv = {0x10000B, 0x10000C, 0x10000D, 0x10000E, 0x10000F}, + /* UM10944.pdf, Table 86, ACU Register overview */ + .rgmii_pad_mii_tx = {0x100800, 0x100802, 0x100804, 0x100806, 0x100808}, + .mac = {0x200, 0x202, 0x204, 0x206, 0x208}, + .mac_hl1 = {0x400, 0x410, 0x420, 0x430, 0x440}, + .mac_hl2 = {0x600, 0x610, 0x620, 0x630, 0x640}, + /* UM10944.pdf, Table 78, CGU Register overview */ + .mii_tx_clk = {0x100013, 0x10001A, 0x100021, 0x100028, 0x10002F}, + .mii_rx_clk = {0x100014, 0x10001B, 0x100022, 0x100029, 0x100030}, + .mii_ext_tx_clk = {0x100018, 0x10001F, 0x100026, 0x10002D, 0x100034}, + .mii_ext_rx_clk = {0x100019, 0x100020, 0x100027, 0x10002E, 0x100035}, + .rgmii_tx_clk = {0x100016, 0x10001D, 0x100024, 0x10002B, 0x100032}, + .rmii_ref_clk = {0x100015, 0x10001C, 0x100023, 0x10002A, 0x100031}, + .rmii_ext_tx_clk = {0x100018, 0x10001F, 0x100026, 0x10002D, 0x100034}, +}; + +struct sja1105_regs sja1105pqrs_regs = { + .device_id = 0x0, + .prod_id = 0x100BC3, + .status = 0x1, + .port_control = 0x12, + .config = 0x020000, + .rgu = 0x100440, + .pad_mii_tx = {0x100800, 0x100802, 0x100804, 0x100806, 0x100808}, + .rmii_pll1 = 0x10000A, + .cgu_idiv = {0x10000B, 0x10000C, 0x10000D, 0x10000E, 0x10000F}, + /* UM10944.pdf, Table 86, ACU Register overview */ + .rgmii_pad_mii_tx = {0x100800, 0x100802, 0x100804, 0x100806, 0x100808}, + .mac = {0x200, 0x202, 0x204, 0x206, 0x208}, + .mac_hl1 = {0x400, 0x410, 0x420, 0x430, 0x440}, + .mac_hl2 = {0x600, 0x610, 0x620, 0x630, 0x640}, + /* UM11040.pdf, Table 114 */ + .mii_tx_clk = {0x100013, 0x100019, 0x10001F, 0x100025, 0x10002B}, + .mii_rx_clk = {0x100014, 0x10001A, 0x100020, 0x100026, 0x10002C}, + .mii_ext_tx_clk = {0x100017, 0x10001D, 0x100023, 0x100029, 0x10002F}, + .mii_ext_rx_clk = {0x100018, 0x10001E, 0x100024, 0x10002A, 0x100030}, + .rgmii_tx_clk = {0x100016, 0x10001C, 0x100022, 0x100028, 0x10002E}, + .rmii_ref_clk = {0x100015, 0x10001B, 0x100021, 0x100027, 0x10002D}, + .rmii_ext_tx_clk = {0x100017, 0x10001D, 0x100023, 0x100029, 0x10002F}, + .qlevel = {0x604, 0x614, 0x624, 0x634, 0x644}, +}; + +struct sja1105_info sja1105e_info = { + .device_id = SJA1105E_DEVICE_ID, + .part_no = SJA1105ET_PART_NO, + .static_ops = sja1105e_table_ops, + .dyn_ops = sja1105et_dyn_ops, + .reset_cmd = sja1105et_reset_cmd, + .regs = &sja1105et_regs, + .name = "SJA1105E", +}; +struct sja1105_info sja1105t_info = { + .device_id = SJA1105T_DEVICE_ID, + .part_no = SJA1105ET_PART_NO, + .static_ops = sja1105t_table_ops, + .dyn_ops = sja1105et_dyn_ops, + .reset_cmd = sja1105et_reset_cmd, + .regs = &sja1105et_regs, + .name = "SJA1105T", +}; +struct sja1105_info sja1105p_info = { + .device_id = SJA1105PR_DEVICE_ID, + .part_no = SJA1105P_PART_NO, + .static_ops = sja1105p_table_ops, + .dyn_ops = sja1105pqrs_dyn_ops, + .reset_cmd = sja1105pqrs_reset_cmd, + .regs = &sja1105pqrs_regs, + .name = "SJA1105P", +}; +struct sja1105_info sja1105q_info = { + .device_id = SJA1105QS_DEVICE_ID, + .part_no = SJA1105Q_PART_NO, + .static_ops = sja1105q_table_ops, + .dyn_ops = sja1105pqrs_dyn_ops, + .reset_cmd = sja1105pqrs_reset_cmd, + .regs = &sja1105pqrs_regs, + .name = "SJA1105Q", +}; +struct sja1105_info sja1105r_info = { + .device_id = SJA1105PR_DEVICE_ID, + .part_no = SJA1105R_PART_NO, + .static_ops = sja1105r_table_ops, + .dyn_ops = sja1105pqrs_dyn_ops, + .reset_cmd = sja1105pqrs_reset_cmd, + .regs = &sja1105pqrs_regs, + .name = "SJA1105R", +}; +struct sja1105_info sja1105s_info = { + .device_id = SJA1105QS_DEVICE_ID, + .part_no = SJA1105S_PART_NO, + .static_ops = sja1105s_table_ops, + .dyn_ops = sja1105pqrs_dyn_ops, + .regs = &sja1105pqrs_regs, + .reset_cmd = sja1105pqrs_reset_cmd, + .name = "SJA1105S", +}; diff --git a/drivers/net/dsa/sja1105/sja1105_static_config.c b/drivers/net/dsa/sja1105/sja1105_static_config.c new file mode 100644 index 0000000000000000000000000000000000000000..b3c992b0abb04d5fb2e3d7a44a1e3c804123fdba --- /dev/null +++ b/drivers/net/dsa/sja1105/sja1105_static_config.c @@ -0,0 +1,987 @@ +// SPDX-License-Identifier: BSD-3-Clause +/* Copyright (c) 2016-2018, NXP Semiconductors + * Copyright (c) 2018-2019, Vladimir Oltean + */ +#include "sja1105_static_config.h" +#include +#include +#include +#include + +/* Convenience wrappers over the generic packing functions. These take into + * account the SJA1105 memory layout quirks and provide some level of + * programmer protection against incorrect API use. The errors are not expected + * to occur durring runtime, therefore printing and swallowing them here is + * appropriate instead of clutterring up higher-level code. + */ +void sja1105_pack(void *buf, const u64 *val, int start, int end, size_t len) +{ + int rc = packing(buf, (u64 *)val, start, end, len, + PACK, QUIRK_LSW32_IS_FIRST); + + if (likely(!rc)) + return; + + if (rc == -EINVAL) { + pr_err("Start bit (%d) expected to be larger than end (%d)\n", + start, end); + } else if (rc == -ERANGE) { + if ((start - end + 1) > 64) + pr_err("Field %d-%d too large for 64 bits!\n", + start, end); + else + pr_err("Cannot store %llx inside bits %d-%d (would truncate)\n", + *val, start, end); + } + dump_stack(); +} + +void sja1105_unpack(const void *buf, u64 *val, int start, int end, size_t len) +{ + int rc = packing((void *)buf, val, start, end, len, + UNPACK, QUIRK_LSW32_IS_FIRST); + + if (likely(!rc)) + return; + + if (rc == -EINVAL) + pr_err("Start bit (%d) expected to be larger than end (%d)\n", + start, end); + else if (rc == -ERANGE) + pr_err("Field %d-%d too large for 64 bits!\n", + start, end); + dump_stack(); +} + +void sja1105_packing(void *buf, u64 *val, int start, int end, + size_t len, enum packing_op op) +{ + int rc = packing(buf, val, start, end, len, op, QUIRK_LSW32_IS_FIRST); + + if (likely(!rc)) + return; + + if (rc == -EINVAL) { + pr_err("Start bit (%d) expected to be larger than end (%d)\n", + start, end); + } else if (rc == -ERANGE) { + if ((start - end + 1) > 64) + pr_err("Field %d-%d too large for 64 bits!\n", + start, end); + else + pr_err("Cannot store %llx inside bits %d-%d (would truncate)\n", + *val, start, end); + } + dump_stack(); +} + +/* Little-endian Ethernet CRC32 of data packed as big-endian u32 words */ +u32 sja1105_crc32(const void *buf, size_t len) +{ + unsigned int i; + u64 word; + u32 crc; + + /* seed */ + crc = ~0; + for (i = 0; i < len; i += 4) { + sja1105_unpack((void *)buf + i, &word, 31, 0, 4); + crc = crc32_le(crc, (u8 *)&word, 4); + } + return ~crc; +} + +static size_t sja1105et_general_params_entry_packing(void *buf, void *entry_ptr, + enum packing_op op) +{ + const size_t size = SJA1105ET_SIZE_GENERAL_PARAMS_ENTRY; + struct sja1105_general_params_entry *entry = entry_ptr; + + sja1105_packing(buf, &entry->vllupformat, 319, 319, size, op); + sja1105_packing(buf, &entry->mirr_ptacu, 318, 318, size, op); + sja1105_packing(buf, &entry->switchid, 317, 315, size, op); + sja1105_packing(buf, &entry->hostprio, 314, 312, size, op); + sja1105_packing(buf, &entry->mac_fltres1, 311, 264, size, op); + sja1105_packing(buf, &entry->mac_fltres0, 263, 216, size, op); + sja1105_packing(buf, &entry->mac_flt1, 215, 168, size, op); + sja1105_packing(buf, &entry->mac_flt0, 167, 120, size, op); + sja1105_packing(buf, &entry->incl_srcpt1, 119, 119, size, op); + sja1105_packing(buf, &entry->incl_srcpt0, 118, 118, size, op); + sja1105_packing(buf, &entry->send_meta1, 117, 117, size, op); + sja1105_packing(buf, &entry->send_meta0, 116, 116, size, op); + sja1105_packing(buf, &entry->casc_port, 115, 113, size, op); + sja1105_packing(buf, &entry->host_port, 112, 110, size, op); + sja1105_packing(buf, &entry->mirr_port, 109, 107, size, op); + sja1105_packing(buf, &entry->vlmarker, 106, 75, size, op); + sja1105_packing(buf, &entry->vlmask, 74, 43, size, op); + sja1105_packing(buf, &entry->tpid, 42, 27, size, op); + sja1105_packing(buf, &entry->ignore2stf, 26, 26, size, op); + sja1105_packing(buf, &entry->tpid2, 25, 10, size, op); + return size; +} + +static size_t +sja1105pqrs_general_params_entry_packing(void *buf, void *entry_ptr, + enum packing_op op) +{ + const size_t size = SJA1105PQRS_SIZE_GENERAL_PARAMS_ENTRY; + struct sja1105_general_params_entry *entry = entry_ptr; + + sja1105_packing(buf, &entry->vllupformat, 351, 351, size, op); + sja1105_packing(buf, &entry->mirr_ptacu, 350, 350, size, op); + sja1105_packing(buf, &entry->switchid, 349, 347, size, op); + sja1105_packing(buf, &entry->hostprio, 346, 344, size, op); + sja1105_packing(buf, &entry->mac_fltres1, 343, 296, size, op); + sja1105_packing(buf, &entry->mac_fltres0, 295, 248, size, op); + sja1105_packing(buf, &entry->mac_flt1, 247, 200, size, op); + sja1105_packing(buf, &entry->mac_flt0, 199, 152, size, op); + sja1105_packing(buf, &entry->incl_srcpt1, 151, 151, size, op); + sja1105_packing(buf, &entry->incl_srcpt0, 150, 150, size, op); + sja1105_packing(buf, &entry->send_meta1, 149, 149, size, op); + sja1105_packing(buf, &entry->send_meta0, 148, 148, size, op); + sja1105_packing(buf, &entry->casc_port, 147, 145, size, op); + sja1105_packing(buf, &entry->host_port, 144, 142, size, op); + sja1105_packing(buf, &entry->mirr_port, 141, 139, size, op); + sja1105_packing(buf, &entry->vlmarker, 138, 107, size, op); + sja1105_packing(buf, &entry->vlmask, 106, 75, size, op); + sja1105_packing(buf, &entry->tpid, 74, 59, size, op); + sja1105_packing(buf, &entry->ignore2stf, 58, 58, size, op); + sja1105_packing(buf, &entry->tpid2, 57, 42, size, op); + sja1105_packing(buf, &entry->queue_ts, 41, 41, size, op); + sja1105_packing(buf, &entry->egrmirrvid, 40, 29, size, op); + sja1105_packing(buf, &entry->egrmirrpcp, 28, 26, size, op); + sja1105_packing(buf, &entry->egrmirrdei, 25, 25, size, op); + sja1105_packing(buf, &entry->replay_port, 24, 22, size, op); + return size; +} + +static size_t +sja1105_l2_forwarding_params_entry_packing(void *buf, void *entry_ptr, + enum packing_op op) +{ + const size_t size = SJA1105_SIZE_L2_FORWARDING_PARAMS_ENTRY; + struct sja1105_l2_forwarding_params_entry *entry = entry_ptr; + int offset, i; + + sja1105_packing(buf, &entry->max_dynp, 95, 93, size, op); + for (i = 0, offset = 13; i < 8; i++, offset += 10) + sja1105_packing(buf, &entry->part_spc[i], + offset + 9, offset + 0, size, op); + return size; +} + +size_t sja1105_l2_forwarding_entry_packing(void *buf, void *entry_ptr, + enum packing_op op) +{ + const size_t size = SJA1105_SIZE_L2_FORWARDING_ENTRY; + struct sja1105_l2_forwarding_entry *entry = entry_ptr; + int offset, i; + + sja1105_packing(buf, &entry->bc_domain, 63, 59, size, op); + sja1105_packing(buf, &entry->reach_port, 58, 54, size, op); + sja1105_packing(buf, &entry->fl_domain, 53, 49, size, op); + for (i = 0, offset = 25; i < 8; i++, offset += 3) + sja1105_packing(buf, &entry->vlan_pmap[i], + offset + 2, offset + 0, size, op); + return size; +} + +static size_t +sja1105et_l2_lookup_params_entry_packing(void *buf, void *entry_ptr, + enum packing_op op) +{ + const size_t size = SJA1105ET_SIZE_L2_LOOKUP_PARAMS_ENTRY; + struct sja1105_l2_lookup_params_entry *entry = entry_ptr; + + sja1105_packing(buf, &entry->maxage, 31, 17, size, op); + sja1105_packing(buf, &entry->dyn_tbsz, 16, 14, size, op); + sja1105_packing(buf, &entry->poly, 13, 6, size, op); + sja1105_packing(buf, &entry->shared_learn, 5, 5, size, op); + sja1105_packing(buf, &entry->no_enf_hostprt, 4, 4, size, op); + sja1105_packing(buf, &entry->no_mgmt_learn, 3, 3, size, op); + return size; +} + +static size_t +sja1105pqrs_l2_lookup_params_entry_packing(void *buf, void *entry_ptr, + enum packing_op op) +{ + const size_t size = SJA1105PQRS_SIZE_L2_LOOKUP_PARAMS_ENTRY; + struct sja1105_l2_lookup_params_entry *entry = entry_ptr; + + sja1105_packing(buf, &entry->maxage, 57, 43, size, op); + sja1105_packing(buf, &entry->shared_learn, 27, 27, size, op); + sja1105_packing(buf, &entry->no_enf_hostprt, 26, 26, size, op); + sja1105_packing(buf, &entry->no_mgmt_learn, 25, 25, size, op); + return size; +} + +size_t sja1105et_l2_lookup_entry_packing(void *buf, void *entry_ptr, + enum packing_op op) +{ + const size_t size = SJA1105ET_SIZE_L2_LOOKUP_ENTRY; + struct sja1105_l2_lookup_entry *entry = entry_ptr; + + sja1105_packing(buf, &entry->vlanid, 95, 84, size, op); + sja1105_packing(buf, &entry->macaddr, 83, 36, size, op); + sja1105_packing(buf, &entry->destports, 35, 31, size, op); + sja1105_packing(buf, &entry->enfport, 30, 30, size, op); + sja1105_packing(buf, &entry->index, 29, 20, size, op); + return size; +} + +size_t sja1105pqrs_l2_lookup_entry_packing(void *buf, void *entry_ptr, + enum packing_op op) +{ + const size_t size = SJA1105PQRS_SIZE_L2_LOOKUP_ENTRY; + struct sja1105_l2_lookup_entry *entry = entry_ptr; + + /* These are static L2 lookup entries, so the structure + * should match UM11040 Table 16/17 definitions when + * LOCKEDS is 1. + */ + sja1105_packing(buf, &entry->vlanid, 81, 70, size, op); + sja1105_packing(buf, &entry->macaddr, 69, 22, size, op); + sja1105_packing(buf, &entry->destports, 21, 17, size, op); + sja1105_packing(buf, &entry->enfport, 16, 16, size, op); + sja1105_packing(buf, &entry->index, 15, 6, size, op); + return size; +} + +static size_t sja1105_l2_policing_entry_packing(void *buf, void *entry_ptr, + enum packing_op op) +{ + const size_t size = SJA1105_SIZE_L2_POLICING_ENTRY; + struct sja1105_l2_policing_entry *entry = entry_ptr; + + sja1105_packing(buf, &entry->sharindx, 63, 58, size, op); + sja1105_packing(buf, &entry->smax, 57, 42, size, op); + sja1105_packing(buf, &entry->rate, 41, 26, size, op); + sja1105_packing(buf, &entry->maxlen, 25, 15, size, op); + sja1105_packing(buf, &entry->partition, 14, 12, size, op); + return size; +} + +static size_t sja1105et_mac_config_entry_packing(void *buf, void *entry_ptr, + enum packing_op op) +{ + const size_t size = SJA1105ET_SIZE_MAC_CONFIG_ENTRY; + struct sja1105_mac_config_entry *entry = entry_ptr; + int offset, i; + + for (i = 0, offset = 72; i < 8; i++, offset += 19) { + sja1105_packing(buf, &entry->enabled[i], + offset + 0, offset + 0, size, op); + sja1105_packing(buf, &entry->base[i], + offset + 9, offset + 1, size, op); + sja1105_packing(buf, &entry->top[i], + offset + 18, offset + 10, size, op); + } + sja1105_packing(buf, &entry->ifg, 71, 67, size, op); + sja1105_packing(buf, &entry->speed, 66, 65, size, op); + sja1105_packing(buf, &entry->tp_delin, 64, 49, size, op); + sja1105_packing(buf, &entry->tp_delout, 48, 33, size, op); + sja1105_packing(buf, &entry->maxage, 32, 25, size, op); + sja1105_packing(buf, &entry->vlanprio, 24, 22, size, op); + sja1105_packing(buf, &entry->vlanid, 21, 10, size, op); + sja1105_packing(buf, &entry->ing_mirr, 9, 9, size, op); + sja1105_packing(buf, &entry->egr_mirr, 8, 8, size, op); + sja1105_packing(buf, &entry->drpnona664, 7, 7, size, op); + sja1105_packing(buf, &entry->drpdtag, 6, 6, size, op); + sja1105_packing(buf, &entry->drpuntag, 5, 5, size, op); + sja1105_packing(buf, &entry->retag, 4, 4, size, op); + sja1105_packing(buf, &entry->dyn_learn, 3, 3, size, op); + sja1105_packing(buf, &entry->egress, 2, 2, size, op); + sja1105_packing(buf, &entry->ingress, 1, 1, size, op); + return size; +} + +size_t sja1105pqrs_mac_config_entry_packing(void *buf, void *entry_ptr, + enum packing_op op) +{ + const size_t size = SJA1105PQRS_SIZE_MAC_CONFIG_ENTRY; + struct sja1105_mac_config_entry *entry = entry_ptr; + int offset, i; + + for (i = 0, offset = 104; i < 8; i++, offset += 19) { + sja1105_packing(buf, &entry->enabled[i], + offset + 0, offset + 0, size, op); + sja1105_packing(buf, &entry->base[i], + offset + 9, offset + 1, size, op); + sja1105_packing(buf, &entry->top[i], + offset + 18, offset + 10, size, op); + } + sja1105_packing(buf, &entry->ifg, 103, 99, size, op); + sja1105_packing(buf, &entry->speed, 98, 97, size, op); + sja1105_packing(buf, &entry->tp_delin, 96, 81, size, op); + sja1105_packing(buf, &entry->tp_delout, 80, 65, size, op); + sja1105_packing(buf, &entry->maxage, 64, 57, size, op); + sja1105_packing(buf, &entry->vlanprio, 56, 54, size, op); + sja1105_packing(buf, &entry->vlanid, 53, 42, size, op); + sja1105_packing(buf, &entry->ing_mirr, 41, 41, size, op); + sja1105_packing(buf, &entry->egr_mirr, 40, 40, size, op); + sja1105_packing(buf, &entry->drpnona664, 39, 39, size, op); + sja1105_packing(buf, &entry->drpdtag, 38, 38, size, op); + sja1105_packing(buf, &entry->drpuntag, 35, 35, size, op); + sja1105_packing(buf, &entry->retag, 34, 34, size, op); + sja1105_packing(buf, &entry->dyn_learn, 33, 33, size, op); + sja1105_packing(buf, &entry->egress, 32, 32, size, op); + sja1105_packing(buf, &entry->ingress, 31, 31, size, op); + return size; +} + +size_t sja1105_vlan_lookup_entry_packing(void *buf, void *entry_ptr, + enum packing_op op) +{ + const size_t size = SJA1105_SIZE_VLAN_LOOKUP_ENTRY; + struct sja1105_vlan_lookup_entry *entry = entry_ptr; + + sja1105_packing(buf, &entry->ving_mirr, 63, 59, size, op); + sja1105_packing(buf, &entry->vegr_mirr, 58, 54, size, op); + sja1105_packing(buf, &entry->vmemb_port, 53, 49, size, op); + sja1105_packing(buf, &entry->vlan_bc, 48, 44, size, op); + sja1105_packing(buf, &entry->tag_port, 43, 39, size, op); + sja1105_packing(buf, &entry->vlanid, 38, 27, size, op); + return size; +} + +static size_t sja1105_xmii_params_entry_packing(void *buf, void *entry_ptr, + enum packing_op op) +{ + const size_t size = SJA1105_SIZE_XMII_PARAMS_ENTRY; + struct sja1105_xmii_params_entry *entry = entry_ptr; + int offset, i; + + for (i = 0, offset = 17; i < 5; i++, offset += 3) { + sja1105_packing(buf, &entry->xmii_mode[i], + offset + 1, offset + 0, size, op); + sja1105_packing(buf, &entry->phy_mac[i], + offset + 2, offset + 2, size, op); + } + return size; +} + +size_t sja1105_table_header_packing(void *buf, void *entry_ptr, + enum packing_op op) +{ + const size_t size = SJA1105_SIZE_TABLE_HEADER; + struct sja1105_table_header *entry = entry_ptr; + + sja1105_packing(buf, &entry->block_id, 31, 24, size, op); + sja1105_packing(buf, &entry->len, 55, 32, size, op); + sja1105_packing(buf, &entry->crc, 95, 64, size, op); + return size; +} + +/* WARNING: the *hdr pointer is really non-const, because it is + * modifying the CRC of the header for a 2-stage packing operation + */ +void +sja1105_table_header_pack_with_crc(void *buf, struct sja1105_table_header *hdr) +{ + /* First pack the table as-is, then calculate the CRC, and + * finally put the proper CRC into the packed buffer + */ + memset(buf, 0, SJA1105_SIZE_TABLE_HEADER); + sja1105_table_header_packing(buf, hdr, PACK); + hdr->crc = sja1105_crc32(buf, SJA1105_SIZE_TABLE_HEADER - 4); + sja1105_pack(buf + SJA1105_SIZE_TABLE_HEADER - 4, &hdr->crc, 31, 0, 4); +} + +static void sja1105_table_write_crc(u8 *table_start, u8 *crc_ptr) +{ + u64 computed_crc; + int len_bytes; + + len_bytes = (uintptr_t)(crc_ptr - table_start); + computed_crc = sja1105_crc32(table_start, len_bytes); + sja1105_pack(crc_ptr, &computed_crc, 31, 0, 4); +} + +/* The block IDs that the switches support are unfortunately sparse, so keep a + * mapping table to "block indices" and translate back and forth so that we + * don't waste useless memory in struct sja1105_static_config. + * Also, since the block id comes from essentially untrusted input (unpacking + * the static config from userspace) it has to be sanitized (range-checked) + * before blindly indexing kernel memory with the blk_idx. + */ +static u64 blk_id_map[BLK_IDX_MAX] = { + [BLK_IDX_L2_LOOKUP] = BLKID_L2_LOOKUP, + [BLK_IDX_L2_POLICING] = BLKID_L2_POLICING, + [BLK_IDX_VLAN_LOOKUP] = BLKID_VLAN_LOOKUP, + [BLK_IDX_L2_FORWARDING] = BLKID_L2_FORWARDING, + [BLK_IDX_MAC_CONFIG] = BLKID_MAC_CONFIG, + [BLK_IDX_L2_LOOKUP_PARAMS] = BLKID_L2_LOOKUP_PARAMS, + [BLK_IDX_L2_FORWARDING_PARAMS] = BLKID_L2_FORWARDING_PARAMS, + [BLK_IDX_GENERAL_PARAMS] = BLKID_GENERAL_PARAMS, + [BLK_IDX_XMII_PARAMS] = BLKID_XMII_PARAMS, +}; + +const char *sja1105_static_config_error_msg[] = { + [SJA1105_CONFIG_OK] = "", + [SJA1105_MISSING_L2_POLICING_TABLE] = + "l2-policing-table needs to have at least one entry", + [SJA1105_MISSING_L2_FORWARDING_TABLE] = + "l2-forwarding-table is either missing or incomplete", + [SJA1105_MISSING_L2_FORWARDING_PARAMS_TABLE] = + "l2-forwarding-parameters-table is missing", + [SJA1105_MISSING_GENERAL_PARAMS_TABLE] = + "general-parameters-table is missing", + [SJA1105_MISSING_VLAN_TABLE] = + "vlan-lookup-table needs to have at least the default untagged VLAN", + [SJA1105_MISSING_XMII_TABLE] = + "xmii-table is missing", + [SJA1105_MISSING_MAC_TABLE] = + "mac-configuration-table needs to contain an entry for each port", + [SJA1105_OVERCOMMITTED_FRAME_MEMORY] = + "Not allowed to overcommit frame memory. L2 memory partitions " + "and VL memory partitions share the same space. The sum of all " + "16 memory partitions is not allowed to be larger than 929 " + "128-byte blocks (or 910 with retagging). Please adjust " + "l2-forwarding-parameters-table.part_spc and/or " + "vl-forwarding-parameters-table.partspc.", +}; + +sja1105_config_valid_t +static_config_check_memory_size(const struct sja1105_table *tables) +{ + const struct sja1105_l2_forwarding_params_entry *l2_fwd_params; + int i, mem = 0; + + l2_fwd_params = tables[BLK_IDX_L2_FORWARDING_PARAMS].entries; + + for (i = 0; i < 8; i++) + mem += l2_fwd_params->part_spc[i]; + + if (mem > SJA1105_MAX_FRAME_MEMORY) + return SJA1105_OVERCOMMITTED_FRAME_MEMORY; + + return SJA1105_CONFIG_OK; +} + +sja1105_config_valid_t +sja1105_static_config_check_valid(const struct sja1105_static_config *config) +{ + const struct sja1105_table *tables = config->tables; +#define IS_FULL(blk_idx) \ + (tables[blk_idx].entry_count == tables[blk_idx].ops->max_entry_count) + + if (tables[BLK_IDX_L2_POLICING].entry_count == 0) + return SJA1105_MISSING_L2_POLICING_TABLE; + + if (tables[BLK_IDX_VLAN_LOOKUP].entry_count == 0) + return SJA1105_MISSING_VLAN_TABLE; + + if (!IS_FULL(BLK_IDX_L2_FORWARDING)) + return SJA1105_MISSING_L2_FORWARDING_TABLE; + + if (!IS_FULL(BLK_IDX_MAC_CONFIG)) + return SJA1105_MISSING_MAC_TABLE; + + if (!IS_FULL(BLK_IDX_L2_FORWARDING_PARAMS)) + return SJA1105_MISSING_L2_FORWARDING_PARAMS_TABLE; + + if (!IS_FULL(BLK_IDX_GENERAL_PARAMS)) + return SJA1105_MISSING_GENERAL_PARAMS_TABLE; + + if (!IS_FULL(BLK_IDX_XMII_PARAMS)) + return SJA1105_MISSING_XMII_TABLE; + + return static_config_check_memory_size(tables); +#undef IS_FULL +} + +void +sja1105_static_config_pack(void *buf, struct sja1105_static_config *config) +{ + struct sja1105_table_header header = {0}; + enum sja1105_blk_idx i; + char *p = buf; + int j; + + sja1105_pack(p, &config->device_id, 31, 0, 4); + p += SJA1105_SIZE_DEVICE_ID; + + for (i = 0; i < BLK_IDX_MAX; i++) { + const struct sja1105_table *table; + char *table_start; + + table = &config->tables[i]; + if (!table->entry_count) + continue; + + header.block_id = blk_id_map[i]; + header.len = table->entry_count * + table->ops->packed_entry_size / 4; + sja1105_table_header_pack_with_crc(p, &header); + p += SJA1105_SIZE_TABLE_HEADER; + table_start = p; + for (j = 0; j < table->entry_count; j++) { + u8 *entry_ptr = table->entries; + + entry_ptr += j * table->ops->unpacked_entry_size; + memset(p, 0, table->ops->packed_entry_size); + table->ops->packing(p, entry_ptr, PACK); + p += table->ops->packed_entry_size; + } + sja1105_table_write_crc(table_start, p); + p += 4; + } + /* Final header: + * Block ID does not matter + * Length of 0 marks that header is final + * CRC will be replaced on-the-fly on "config upload" + */ + header.block_id = 0; + header.len = 0; + header.crc = 0xDEADBEEF; + memset(p, 0, SJA1105_SIZE_TABLE_HEADER); + sja1105_table_header_packing(p, &header, PACK); +} + +size_t +sja1105_static_config_get_length(const struct sja1105_static_config *config) +{ + unsigned int sum; + unsigned int header_count; + enum sja1105_blk_idx i; + + /* Ending header */ + header_count = 1; + sum = SJA1105_SIZE_DEVICE_ID; + + /* Tables (headers and entries) */ + for (i = 0; i < BLK_IDX_MAX; i++) { + const struct sja1105_table *table; + + table = &config->tables[i]; + if (table->entry_count) + header_count++; + + sum += table->ops->packed_entry_size * table->entry_count; + } + /* Headers have an additional CRC at the end */ + sum += header_count * (SJA1105_SIZE_TABLE_HEADER + 4); + /* Last header does not have an extra CRC because there is no data */ + sum -= 4; + + return sum; +} + +/* Compatibility matrices */ + +/* SJA1105E: First generation, no TTEthernet */ +struct sja1105_table_ops sja1105e_table_ops[BLK_IDX_MAX] = { + [BLK_IDX_L2_LOOKUP] = { + .packing = sja1105et_l2_lookup_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_lookup_entry), + .packed_entry_size = SJA1105ET_SIZE_L2_LOOKUP_ENTRY, + .max_entry_count = SJA1105_MAX_L2_LOOKUP_COUNT, + }, + [BLK_IDX_L2_POLICING] = { + .packing = sja1105_l2_policing_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_policing_entry), + .packed_entry_size = SJA1105_SIZE_L2_POLICING_ENTRY, + .max_entry_count = SJA1105_MAX_L2_POLICING_COUNT, + }, + [BLK_IDX_VLAN_LOOKUP] = { + .packing = sja1105_vlan_lookup_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_vlan_lookup_entry), + .packed_entry_size = SJA1105_SIZE_VLAN_LOOKUP_ENTRY, + .max_entry_count = SJA1105_MAX_VLAN_LOOKUP_COUNT, + }, + [BLK_IDX_L2_FORWARDING] = { + .packing = sja1105_l2_forwarding_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_forwarding_entry), + .packed_entry_size = SJA1105_SIZE_L2_FORWARDING_ENTRY, + .max_entry_count = SJA1105_MAX_L2_FORWARDING_COUNT, + }, + [BLK_IDX_MAC_CONFIG] = { + .packing = sja1105et_mac_config_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_mac_config_entry), + .packed_entry_size = SJA1105ET_SIZE_MAC_CONFIG_ENTRY, + .max_entry_count = SJA1105_MAX_MAC_CONFIG_COUNT, + }, + [BLK_IDX_L2_LOOKUP_PARAMS] = { + .packing = sja1105et_l2_lookup_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_lookup_params_entry), + .packed_entry_size = SJA1105ET_SIZE_L2_LOOKUP_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_L2_LOOKUP_PARAMS_COUNT, + }, + [BLK_IDX_L2_FORWARDING_PARAMS] = { + .packing = sja1105_l2_forwarding_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_forwarding_params_entry), + .packed_entry_size = SJA1105_SIZE_L2_FORWARDING_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_L2_FORWARDING_PARAMS_COUNT, + }, + [BLK_IDX_GENERAL_PARAMS] = { + .packing = sja1105et_general_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_general_params_entry), + .packed_entry_size = SJA1105ET_SIZE_GENERAL_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_GENERAL_PARAMS_COUNT, + }, + [BLK_IDX_XMII_PARAMS] = { + .packing = sja1105_xmii_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_xmii_params_entry), + .packed_entry_size = SJA1105_SIZE_XMII_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_XMII_PARAMS_COUNT, + }, +}; + +/* SJA1105T: First generation, TTEthernet */ +struct sja1105_table_ops sja1105t_table_ops[BLK_IDX_MAX] = { + [BLK_IDX_L2_LOOKUP] = { + .packing = sja1105et_l2_lookup_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_lookup_entry), + .packed_entry_size = SJA1105ET_SIZE_L2_LOOKUP_ENTRY, + .max_entry_count = SJA1105_MAX_L2_LOOKUP_COUNT, + }, + [BLK_IDX_L2_POLICING] = { + .packing = sja1105_l2_policing_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_policing_entry), + .packed_entry_size = SJA1105_SIZE_L2_POLICING_ENTRY, + .max_entry_count = SJA1105_MAX_L2_POLICING_COUNT, + }, + [BLK_IDX_VLAN_LOOKUP] = { + .packing = sja1105_vlan_lookup_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_vlan_lookup_entry), + .packed_entry_size = SJA1105_SIZE_VLAN_LOOKUP_ENTRY, + .max_entry_count = SJA1105_MAX_VLAN_LOOKUP_COUNT, + }, + [BLK_IDX_L2_FORWARDING] = { + .packing = sja1105_l2_forwarding_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_forwarding_entry), + .packed_entry_size = SJA1105_SIZE_L2_FORWARDING_ENTRY, + .max_entry_count = SJA1105_MAX_L2_FORWARDING_COUNT, + }, + [BLK_IDX_MAC_CONFIG] = { + .packing = sja1105et_mac_config_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_mac_config_entry), + .packed_entry_size = SJA1105ET_SIZE_MAC_CONFIG_ENTRY, + .max_entry_count = SJA1105_MAX_MAC_CONFIG_COUNT, + }, + [BLK_IDX_L2_LOOKUP_PARAMS] = { + .packing = sja1105et_l2_lookup_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_lookup_params_entry), + .packed_entry_size = SJA1105ET_SIZE_L2_LOOKUP_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_L2_LOOKUP_PARAMS_COUNT, + }, + [BLK_IDX_L2_FORWARDING_PARAMS] = { + .packing = sja1105_l2_forwarding_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_forwarding_params_entry), + .packed_entry_size = SJA1105_SIZE_L2_FORWARDING_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_L2_FORWARDING_PARAMS_COUNT, + }, + [BLK_IDX_GENERAL_PARAMS] = { + .packing = sja1105et_general_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_general_params_entry), + .packed_entry_size = SJA1105ET_SIZE_GENERAL_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_GENERAL_PARAMS_COUNT, + }, + [BLK_IDX_XMII_PARAMS] = { + .packing = sja1105_xmii_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_xmii_params_entry), + .packed_entry_size = SJA1105_SIZE_XMII_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_XMII_PARAMS_COUNT, + }, +}; + +/* SJA1105P: Second generation, no TTEthernet, no SGMII */ +struct sja1105_table_ops sja1105p_table_ops[BLK_IDX_MAX] = { + [BLK_IDX_L2_LOOKUP] = { + .packing = sja1105pqrs_l2_lookup_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_lookup_entry), + .packed_entry_size = SJA1105PQRS_SIZE_L2_LOOKUP_ENTRY, + .max_entry_count = SJA1105_MAX_L2_LOOKUP_COUNT, + }, + [BLK_IDX_L2_POLICING] = { + .packing = sja1105_l2_policing_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_policing_entry), + .packed_entry_size = SJA1105_SIZE_L2_POLICING_ENTRY, + .max_entry_count = SJA1105_MAX_L2_POLICING_COUNT, + }, + [BLK_IDX_VLAN_LOOKUP] = { + .packing = sja1105_vlan_lookup_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_vlan_lookup_entry), + .packed_entry_size = SJA1105_SIZE_VLAN_LOOKUP_ENTRY, + .max_entry_count = SJA1105_MAX_VLAN_LOOKUP_COUNT, + }, + [BLK_IDX_L2_FORWARDING] = { + .packing = sja1105_l2_forwarding_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_forwarding_entry), + .packed_entry_size = SJA1105_SIZE_L2_FORWARDING_ENTRY, + .max_entry_count = SJA1105_MAX_L2_FORWARDING_COUNT, + }, + [BLK_IDX_MAC_CONFIG] = { + .packing = sja1105pqrs_mac_config_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_mac_config_entry), + .packed_entry_size = SJA1105PQRS_SIZE_MAC_CONFIG_ENTRY, + .max_entry_count = SJA1105_MAX_MAC_CONFIG_COUNT, + }, + [BLK_IDX_L2_LOOKUP_PARAMS] = { + .packing = sja1105pqrs_l2_lookup_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_lookup_params_entry), + .packed_entry_size = SJA1105PQRS_SIZE_L2_LOOKUP_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_L2_LOOKUP_PARAMS_COUNT, + }, + [BLK_IDX_L2_FORWARDING_PARAMS] = { + .packing = sja1105_l2_forwarding_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_forwarding_params_entry), + .packed_entry_size = SJA1105_SIZE_L2_FORWARDING_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_L2_FORWARDING_PARAMS_COUNT, + }, + [BLK_IDX_GENERAL_PARAMS] = { + .packing = sja1105pqrs_general_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_general_params_entry), + .packed_entry_size = SJA1105PQRS_SIZE_GENERAL_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_GENERAL_PARAMS_COUNT, + }, + [BLK_IDX_XMII_PARAMS] = { + .packing = sja1105_xmii_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_xmii_params_entry), + .packed_entry_size = SJA1105_SIZE_XMII_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_XMII_PARAMS_COUNT, + }, +}; + +/* SJA1105Q: Second generation, TTEthernet, no SGMII */ +struct sja1105_table_ops sja1105q_table_ops[BLK_IDX_MAX] = { + [BLK_IDX_L2_LOOKUP] = { + .packing = sja1105pqrs_l2_lookup_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_lookup_entry), + .packed_entry_size = SJA1105PQRS_SIZE_L2_LOOKUP_ENTRY, + .max_entry_count = SJA1105_MAX_L2_LOOKUP_COUNT, + }, + [BLK_IDX_L2_POLICING] = { + .packing = sja1105_l2_policing_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_policing_entry), + .packed_entry_size = SJA1105_SIZE_L2_POLICING_ENTRY, + .max_entry_count = SJA1105_MAX_L2_POLICING_COUNT, + }, + [BLK_IDX_VLAN_LOOKUP] = { + .packing = sja1105_vlan_lookup_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_vlan_lookup_entry), + .packed_entry_size = SJA1105_SIZE_VLAN_LOOKUP_ENTRY, + .max_entry_count = SJA1105_MAX_VLAN_LOOKUP_COUNT, + }, + [BLK_IDX_L2_FORWARDING] = { + .packing = sja1105_l2_forwarding_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_forwarding_entry), + .packed_entry_size = SJA1105_SIZE_L2_FORWARDING_ENTRY, + .max_entry_count = SJA1105_MAX_L2_FORWARDING_COUNT, + }, + [BLK_IDX_MAC_CONFIG] = { + .packing = sja1105pqrs_mac_config_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_mac_config_entry), + .packed_entry_size = SJA1105PQRS_SIZE_MAC_CONFIG_ENTRY, + .max_entry_count = SJA1105_MAX_MAC_CONFIG_COUNT, + }, + [BLK_IDX_L2_LOOKUP_PARAMS] = { + .packing = sja1105pqrs_l2_lookup_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_lookup_params_entry), + .packed_entry_size = SJA1105PQRS_SIZE_L2_LOOKUP_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_L2_LOOKUP_PARAMS_COUNT, + }, + [BLK_IDX_L2_FORWARDING_PARAMS] = { + .packing = sja1105_l2_forwarding_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_forwarding_params_entry), + .packed_entry_size = SJA1105_SIZE_L2_FORWARDING_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_L2_FORWARDING_PARAMS_COUNT, + }, + [BLK_IDX_GENERAL_PARAMS] = { + .packing = sja1105pqrs_general_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_general_params_entry), + .packed_entry_size = SJA1105PQRS_SIZE_GENERAL_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_GENERAL_PARAMS_COUNT, + }, + [BLK_IDX_XMII_PARAMS] = { + .packing = sja1105_xmii_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_xmii_params_entry), + .packed_entry_size = SJA1105_SIZE_XMII_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_XMII_PARAMS_COUNT, + }, +}; + +/* SJA1105R: Second generation, no TTEthernet, SGMII */ +struct sja1105_table_ops sja1105r_table_ops[BLK_IDX_MAX] = { + [BLK_IDX_L2_LOOKUP] = { + .packing = sja1105pqrs_l2_lookup_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_lookup_entry), + .packed_entry_size = SJA1105PQRS_SIZE_L2_LOOKUP_ENTRY, + .max_entry_count = SJA1105_MAX_L2_LOOKUP_COUNT, + }, + [BLK_IDX_L2_POLICING] = { + .packing = sja1105_l2_policing_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_policing_entry), + .packed_entry_size = SJA1105_SIZE_L2_POLICING_ENTRY, + .max_entry_count = SJA1105_MAX_L2_POLICING_COUNT, + }, + [BLK_IDX_VLAN_LOOKUP] = { + .packing = sja1105_vlan_lookup_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_vlan_lookup_entry), + .packed_entry_size = SJA1105_SIZE_VLAN_LOOKUP_ENTRY, + .max_entry_count = SJA1105_MAX_VLAN_LOOKUP_COUNT, + }, + [BLK_IDX_L2_FORWARDING] = { + .packing = sja1105_l2_forwarding_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_forwarding_entry), + .packed_entry_size = SJA1105_SIZE_L2_FORWARDING_ENTRY, + .max_entry_count = SJA1105_MAX_L2_FORWARDING_COUNT, + }, + [BLK_IDX_MAC_CONFIG] = { + .packing = sja1105pqrs_mac_config_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_mac_config_entry), + .packed_entry_size = SJA1105PQRS_SIZE_MAC_CONFIG_ENTRY, + .max_entry_count = SJA1105_MAX_MAC_CONFIG_COUNT, + }, + [BLK_IDX_L2_LOOKUP_PARAMS] = { + .packing = sja1105pqrs_l2_lookup_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_lookup_params_entry), + .packed_entry_size = SJA1105PQRS_SIZE_L2_LOOKUP_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_L2_LOOKUP_PARAMS_COUNT, + }, + [BLK_IDX_L2_FORWARDING_PARAMS] = { + .packing = sja1105_l2_forwarding_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_forwarding_params_entry), + .packed_entry_size = SJA1105_SIZE_L2_FORWARDING_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_L2_FORWARDING_PARAMS_COUNT, + }, + [BLK_IDX_GENERAL_PARAMS] = { + .packing = sja1105pqrs_general_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_general_params_entry), + .packed_entry_size = SJA1105PQRS_SIZE_GENERAL_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_GENERAL_PARAMS_COUNT, + }, + [BLK_IDX_XMII_PARAMS] = { + .packing = sja1105_xmii_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_xmii_params_entry), + .packed_entry_size = SJA1105_SIZE_XMII_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_XMII_PARAMS_COUNT, + }, +}; + +/* SJA1105S: Second generation, TTEthernet, SGMII */ +struct sja1105_table_ops sja1105s_table_ops[BLK_IDX_MAX] = { + [BLK_IDX_L2_LOOKUP] = { + .packing = sja1105pqrs_l2_lookup_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_lookup_entry), + .packed_entry_size = SJA1105PQRS_SIZE_L2_LOOKUP_ENTRY, + .max_entry_count = SJA1105_MAX_L2_LOOKUP_COUNT, + }, + [BLK_IDX_L2_POLICING] = { + .packing = sja1105_l2_policing_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_policing_entry), + .packed_entry_size = SJA1105_SIZE_L2_POLICING_ENTRY, + .max_entry_count = SJA1105_MAX_L2_POLICING_COUNT, + }, + [BLK_IDX_VLAN_LOOKUP] = { + .packing = sja1105_vlan_lookup_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_vlan_lookup_entry), + .packed_entry_size = SJA1105_SIZE_VLAN_LOOKUP_ENTRY, + .max_entry_count = SJA1105_MAX_VLAN_LOOKUP_COUNT, + }, + [BLK_IDX_L2_FORWARDING] = { + .packing = sja1105_l2_forwarding_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_forwarding_entry), + .packed_entry_size = SJA1105_SIZE_L2_FORWARDING_ENTRY, + .max_entry_count = SJA1105_MAX_L2_FORWARDING_COUNT, + }, + [BLK_IDX_MAC_CONFIG] = { + .packing = sja1105pqrs_mac_config_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_mac_config_entry), + .packed_entry_size = SJA1105PQRS_SIZE_MAC_CONFIG_ENTRY, + .max_entry_count = SJA1105_MAX_MAC_CONFIG_COUNT, + }, + [BLK_IDX_L2_LOOKUP_PARAMS] = { + .packing = sja1105pqrs_l2_lookup_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_lookup_params_entry), + .packed_entry_size = SJA1105PQRS_SIZE_L2_LOOKUP_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_L2_LOOKUP_PARAMS_COUNT, + }, + [BLK_IDX_L2_FORWARDING_PARAMS] = { + .packing = sja1105_l2_forwarding_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_l2_forwarding_params_entry), + .packed_entry_size = SJA1105_SIZE_L2_FORWARDING_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_L2_FORWARDING_PARAMS_COUNT, + }, + [BLK_IDX_GENERAL_PARAMS] = { + .packing = sja1105pqrs_general_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_general_params_entry), + .packed_entry_size = SJA1105PQRS_SIZE_GENERAL_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_GENERAL_PARAMS_COUNT, + }, + [BLK_IDX_XMII_PARAMS] = { + .packing = sja1105_xmii_params_entry_packing, + .unpacked_entry_size = sizeof(struct sja1105_xmii_params_entry), + .packed_entry_size = SJA1105_SIZE_XMII_PARAMS_ENTRY, + .max_entry_count = SJA1105_MAX_XMII_PARAMS_COUNT, + }, +}; + +int sja1105_static_config_init(struct sja1105_static_config *config, + const struct sja1105_table_ops *static_ops, + u64 device_id) +{ + enum sja1105_blk_idx i; + + *config = (struct sja1105_static_config) {0}; + + /* Transfer static_ops array from priv into per-table ops + * for handier access + */ + for (i = 0; i < BLK_IDX_MAX; i++) + config->tables[i].ops = &static_ops[i]; + + config->device_id = device_id; + return 0; +} + +void sja1105_static_config_free(struct sja1105_static_config *config) +{ + enum sja1105_blk_idx i; + + for (i = 0; i < BLK_IDX_MAX; i++) { + if (config->tables[i].entry_count) { + kfree(config->tables[i].entries); + config->tables[i].entry_count = 0; + } + } +} + +int sja1105_table_delete_entry(struct sja1105_table *table, int i) +{ + size_t entry_size = table->ops->unpacked_entry_size; + u8 *entries = table->entries; + + if (i > table->entry_count) + return -ERANGE; + + memmove(entries + i * entry_size, entries + (i + 1) * entry_size, + (table->entry_count - i) * entry_size); + + table->entry_count--; + + return 0; +} + +/* No pointers to table->entries should be kept when this is called. */ +int sja1105_table_resize(struct sja1105_table *table, size_t new_count) +{ + size_t entry_size = table->ops->unpacked_entry_size; + void *new_entries, *old_entries = table->entries; + + if (new_count > table->ops->max_entry_count) + return -ERANGE; + + new_entries = kcalloc(new_count, entry_size, GFP_KERNEL); + if (!new_entries) + return -ENOMEM; + + memcpy(new_entries, old_entries, min(new_count, table->entry_count) * + entry_size); + + table->entries = new_entries; + table->entry_count = new_count; + kfree(old_entries); + return 0; +} diff --git a/drivers/net/dsa/sja1105/sja1105_static_config.h b/drivers/net/dsa/sja1105/sja1105_static_config.h new file mode 100644 index 0000000000000000000000000000000000000000..069ca8fd059ca851aa39e39178cbd48c9fb1598c --- /dev/null +++ b/drivers/net/dsa/sja1105/sja1105_static_config.h @@ -0,0 +1,253 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2016-2018, NXP Semiconductors + * Copyright (c) 2018-2019, Vladimir Oltean + */ +#ifndef _SJA1105_STATIC_CONFIG_H +#define _SJA1105_STATIC_CONFIG_H + +#include +#include +#include + +#define SJA1105_SIZE_DEVICE_ID 4 +#define SJA1105_SIZE_TABLE_HEADER 12 +#define SJA1105_SIZE_L2_POLICING_ENTRY 8 +#define SJA1105_SIZE_VLAN_LOOKUP_ENTRY 8 +#define SJA1105_SIZE_L2_FORWARDING_ENTRY 8 +#define SJA1105_SIZE_L2_FORWARDING_PARAMS_ENTRY 12 +#define SJA1105_SIZE_XMII_PARAMS_ENTRY 4 +#define SJA1105ET_SIZE_L2_LOOKUP_ENTRY 12 +#define SJA1105ET_SIZE_MAC_CONFIG_ENTRY 28 +#define SJA1105ET_SIZE_L2_LOOKUP_PARAMS_ENTRY 4 +#define SJA1105ET_SIZE_GENERAL_PARAMS_ENTRY 40 +#define SJA1105PQRS_SIZE_L2_LOOKUP_ENTRY 20 +#define SJA1105PQRS_SIZE_MAC_CONFIG_ENTRY 32 +#define SJA1105PQRS_SIZE_L2_LOOKUP_PARAMS_ENTRY 16 +#define SJA1105PQRS_SIZE_GENERAL_PARAMS_ENTRY 44 + +/* UM10944.pdf Page 11, Table 2. Configuration Blocks */ +enum { + BLKID_L2_LOOKUP = 0x05, + BLKID_L2_POLICING = 0x06, + BLKID_VLAN_LOOKUP = 0x07, + BLKID_L2_FORWARDING = 0x08, + BLKID_MAC_CONFIG = 0x09, + BLKID_L2_LOOKUP_PARAMS = 0x0D, + BLKID_L2_FORWARDING_PARAMS = 0x0E, + BLKID_GENERAL_PARAMS = 0x11, + BLKID_XMII_PARAMS = 0x4E, +}; + +enum sja1105_blk_idx { + BLK_IDX_L2_LOOKUP = 0, + BLK_IDX_L2_POLICING, + BLK_IDX_VLAN_LOOKUP, + BLK_IDX_L2_FORWARDING, + BLK_IDX_MAC_CONFIG, + BLK_IDX_L2_LOOKUP_PARAMS, + BLK_IDX_L2_FORWARDING_PARAMS, + BLK_IDX_GENERAL_PARAMS, + BLK_IDX_XMII_PARAMS, + BLK_IDX_MAX, + /* Fake block indices that are only valid for dynamic access */ + BLK_IDX_MGMT_ROUTE, + BLK_IDX_MAX_DYN, + BLK_IDX_INVAL = -1, +}; + +#define SJA1105_MAX_L2_LOOKUP_COUNT 1024 +#define SJA1105_MAX_L2_POLICING_COUNT 45 +#define SJA1105_MAX_VLAN_LOOKUP_COUNT 4096 +#define SJA1105_MAX_L2_FORWARDING_COUNT 13 +#define SJA1105_MAX_MAC_CONFIG_COUNT 5 +#define SJA1105_MAX_L2_LOOKUP_PARAMS_COUNT 1 +#define SJA1105_MAX_L2_FORWARDING_PARAMS_COUNT 1 +#define SJA1105_MAX_GENERAL_PARAMS_COUNT 1 +#define SJA1105_MAX_XMII_PARAMS_COUNT 1 + +#define SJA1105_MAX_FRAME_MEMORY 929 + +#define SJA1105E_DEVICE_ID 0x9C00000Cull +#define SJA1105T_DEVICE_ID 0x9E00030Eull +#define SJA1105PR_DEVICE_ID 0xAF00030Eull +#define SJA1105QS_DEVICE_ID 0xAE00030Eull + +#define SJA1105ET_PART_NO 0x9A83 +#define SJA1105P_PART_NO 0x9A84 +#define SJA1105Q_PART_NO 0x9A85 +#define SJA1105R_PART_NO 0x9A86 +#define SJA1105S_PART_NO 0x9A87 + +struct sja1105_general_params_entry { + u64 vllupformat; + u64 mirr_ptacu; + u64 switchid; + u64 hostprio; + u64 mac_fltres1; + u64 mac_fltres0; + u64 mac_flt1; + u64 mac_flt0; + u64 incl_srcpt1; + u64 incl_srcpt0; + u64 send_meta1; + u64 send_meta0; + u64 casc_port; + u64 host_port; + u64 mirr_port; + u64 vlmarker; + u64 vlmask; + u64 tpid; + u64 ignore2stf; + u64 tpid2; + /* P/Q/R/S only */ + u64 queue_ts; + u64 egrmirrvid; + u64 egrmirrpcp; + u64 egrmirrdei; + u64 replay_port; +}; + +struct sja1105_vlan_lookup_entry { + u64 ving_mirr; + u64 vegr_mirr; + u64 vmemb_port; + u64 vlan_bc; + u64 tag_port; + u64 vlanid; +}; + +struct sja1105_l2_lookup_entry { + u64 vlanid; + u64 macaddr; + u64 destports; + u64 enfport; + u64 index; +}; + +struct sja1105_l2_lookup_params_entry { + u64 maxage; /* Shared */ + u64 dyn_tbsz; /* E/T only */ + u64 poly; /* E/T only */ + u64 shared_learn; /* Shared */ + u64 no_enf_hostprt; /* Shared */ + u64 no_mgmt_learn; /* Shared */ +}; + +struct sja1105_l2_forwarding_entry { + u64 bc_domain; + u64 reach_port; + u64 fl_domain; + u64 vlan_pmap[8]; +}; + +struct sja1105_l2_forwarding_params_entry { + u64 max_dynp; + u64 part_spc[8]; +}; + +struct sja1105_l2_policing_entry { + u64 sharindx; + u64 smax; + u64 rate; + u64 maxlen; + u64 partition; +}; + +struct sja1105_mac_config_entry { + u64 top[8]; + u64 base[8]; + u64 enabled[8]; + u64 ifg; + u64 speed; + u64 tp_delin; + u64 tp_delout; + u64 maxage; + u64 vlanprio; + u64 vlanid; + u64 ing_mirr; + u64 egr_mirr; + u64 drpnona664; + u64 drpdtag; + u64 drpuntag; + u64 retag; + u64 dyn_learn; + u64 egress; + u64 ingress; +}; + +struct sja1105_xmii_params_entry { + u64 phy_mac[5]; + u64 xmii_mode[5]; +}; + +struct sja1105_table_header { + u64 block_id; + u64 len; + u64 crc; +}; + +struct sja1105_table_ops { + size_t (*packing)(void *buf, void *entry_ptr, enum packing_op op); + size_t unpacked_entry_size; + size_t packed_entry_size; + size_t max_entry_count; +}; + +struct sja1105_table { + const struct sja1105_table_ops *ops; + size_t entry_count; + void *entries; +}; + +struct sja1105_static_config { + u64 device_id; + struct sja1105_table tables[BLK_IDX_MAX]; +}; + +extern struct sja1105_table_ops sja1105e_table_ops[BLK_IDX_MAX]; +extern struct sja1105_table_ops sja1105t_table_ops[BLK_IDX_MAX]; +extern struct sja1105_table_ops sja1105p_table_ops[BLK_IDX_MAX]; +extern struct sja1105_table_ops sja1105q_table_ops[BLK_IDX_MAX]; +extern struct sja1105_table_ops sja1105r_table_ops[BLK_IDX_MAX]; +extern struct sja1105_table_ops sja1105s_table_ops[BLK_IDX_MAX]; + +size_t sja1105_table_header_packing(void *buf, void *hdr, enum packing_op op); +void +sja1105_table_header_pack_with_crc(void *buf, struct sja1105_table_header *hdr); +size_t +sja1105_static_config_get_length(const struct sja1105_static_config *config); + +typedef enum { + SJA1105_CONFIG_OK = 0, + SJA1105_MISSING_L2_POLICING_TABLE, + SJA1105_MISSING_L2_FORWARDING_TABLE, + SJA1105_MISSING_L2_FORWARDING_PARAMS_TABLE, + SJA1105_MISSING_GENERAL_PARAMS_TABLE, + SJA1105_MISSING_VLAN_TABLE, + SJA1105_MISSING_XMII_TABLE, + SJA1105_MISSING_MAC_TABLE, + SJA1105_OVERCOMMITTED_FRAME_MEMORY, +} sja1105_config_valid_t; + +extern const char *sja1105_static_config_error_msg[]; + +sja1105_config_valid_t +sja1105_static_config_check_valid(const struct sja1105_static_config *config); +void +sja1105_static_config_pack(void *buf, struct sja1105_static_config *config); +int sja1105_static_config_init(struct sja1105_static_config *config, + const struct sja1105_table_ops *static_ops, + u64 device_id); +void sja1105_static_config_free(struct sja1105_static_config *config); + +int sja1105_table_delete_entry(struct sja1105_table *table, int i); +int sja1105_table_resize(struct sja1105_table *table, size_t new_count); + +u32 sja1105_crc32(const void *buf, size_t len); + +void sja1105_pack(void *buf, const u64 *val, int start, int end, size_t len); +void sja1105_unpack(const void *buf, u64 *val, int start, int end, size_t len); +void sja1105_packing(void *buf, u64 *val, int start, int end, + size_t len, enum packing_op op); + +#endif diff --git a/drivers/net/dummy.c b/drivers/net/dummy.c index 0d15a12a4560b37a4ed21d8e4bdd3bb853adb894..3568129fb7dae08f89a63f7ecc0ce98e165eb83b 100644 --- a/drivers/net/dummy.c +++ b/drivers/net/dummy.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include #include @@ -131,21 +132,9 @@ static void dummy_get_drvinfo(struct net_device *dev, strlcpy(info->version, DRV_VERSION, sizeof(info->version)); } -static int dummy_get_ts_info(struct net_device *dev, - struct ethtool_ts_info *ts_info) -{ - ts_info->so_timestamping = SOF_TIMESTAMPING_TX_SOFTWARE | - SOF_TIMESTAMPING_RX_SOFTWARE | - SOF_TIMESTAMPING_SOFTWARE; - - ts_info->phc_index = -1; - - return 0; -}; - static const struct ethtool_ops dummy_ethtool_ops = { .get_drvinfo = dummy_get_drvinfo, - .get_ts_info = dummy_get_ts_info, + .get_ts_info = ethtool_op_get_ts_info, }; static void dummy_setup(struct net_device *dev) diff --git a/drivers/net/ethernet/aeroflex/greth.c b/drivers/net/ethernet/aeroflex/greth.c index 3155f7fa83eb64b3b4fda0c152fd3eec7bfe6384..90080a886cd9146e00c1138a38b36108b392fc7f 100644 --- a/drivers/net/ethernet/aeroflex/greth.c +++ b/drivers/net/ethernet/aeroflex/greth.c @@ -1458,7 +1458,7 @@ static int greth_of_probe(struct platform_device *ofdev) const u8 *addr; addr = of_get_mac_address(ofdev->dev.of_node); - if (addr) { + if (!IS_ERR(addr)) { for (i = 0; i < 6; i++) macaddr[i] = (unsigned int) addr[i]; } else { diff --git a/drivers/net/ethernet/allwinner/sun4i-emac.c b/drivers/net/ethernet/allwinner/sun4i-emac.c index e1acafa82214133044765421109a84ff2168dd91..37ebd890ef519a9fdee51bf9cc5f8d0946d9e2e5 100644 --- a/drivers/net/ethernet/allwinner/sun4i-emac.c +++ b/drivers/net/ethernet/allwinner/sun4i-emac.c @@ -870,7 +870,7 @@ static int emac_probe(struct platform_device *pdev) /* Read MAC-address from DT */ mac_addr = of_get_mac_address(np); - if (mac_addr) + if (!IS_ERR(mac_addr)) memcpy(ndev->dev_addr, mac_addr, ETH_ALEN); /* Check if the MAC address is valid, if not get a random one */ diff --git a/drivers/net/ethernet/altera/altera_tse_main.c b/drivers/net/ethernet/altera/altera_tse_main.c index aa1d1f5339d2a4b78ee2092d0c9683e1262e2581..877e67f4344b498e8d7907dea60feeeb4e5daf65 100644 --- a/drivers/net/ethernet/altera/altera_tse_main.c +++ b/drivers/net/ethernet/altera/altera_tse_main.c @@ -1537,7 +1537,7 @@ static int altera_tse_probe(struct platform_device *pdev) /* get default MAC address from device tree */ macaddr = of_get_mac_address(pdev->dev.of_node); - if (macaddr) + if (!IS_ERR(macaddr)) ether_addr_copy(ndev->dev_addr, macaddr); else eth_hw_addr_random(ndev); diff --git a/drivers/net/ethernet/amazon/ena/ena_com.c b/drivers/net/ethernet/amazon/ena/ena_com.c index 05798aa5bb73af91874beecef42b6b86fa35fcf3..7f8266b191ae7e5b712246b78689efe6584f7492 100644 --- a/drivers/net/ethernet/amazon/ena/ena_com.c +++ b/drivers/net/ethernet/amazon/ena/ena_com.c @@ -731,7 +731,7 @@ static int ena_com_config_llq_info(struct ena_com_dev *ena_dev, if (rc) pr_err("Cannot set LLQ configuration: %d\n", rc); - return 0; + return rc; } static int ena_com_wait_and_process_admin_cq_interrupts(struct ena_comp_ctx *comp_ctx, @@ -2194,7 +2194,7 @@ int ena_com_set_hash_function(struct ena_com_dev *ena_dev) if (unlikely(ret)) return ret; - if (get_resp.u.flow_hash_func.supported_func & (1 << rss->hash_func)) { + if (!(get_resp.u.flow_hash_func.supported_func & BIT(rss->hash_func))) { pr_err("Func hash %d isn't supported by device, abort\n", rss->hash_func); return -EOPNOTSUPP; @@ -2279,6 +2279,7 @@ int ena_com_fill_hash_function(struct ena_com_dev *ena_dev, return -EINVAL; } + rss->hash_func = func; rc = ena_com_set_hash_function(ena_dev); /* Restore the old function */ @@ -2801,7 +2802,11 @@ int ena_com_init_interrupt_moderation(struct ena_com_dev *ena_dev) /* if moderation is supported by device we set adaptive moderation */ delay_resolution = get_resp.u.intr_moderation.intr_delay_resolution; ena_com_update_intr_delay_resolution(ena_dev, delay_resolution); - ena_com_enable_adaptive_moderation(ena_dev); + + /* Disable adaptive moderation by default - can be enabled from + * ethtool + */ + ena_com_disable_adaptive_moderation(ena_dev); return 0; err: diff --git a/drivers/net/ethernet/amazon/ena/ena_ethtool.c b/drivers/net/ethernet/amazon/ena/ena_ethtool.c index f3a5a384e6e8f9d8620a81676c603b767e7e66ff..fe596bc30a96b0043d62698607a4c9b364fb5679 100644 --- a/drivers/net/ethernet/amazon/ena/ena_ethtool.c +++ b/drivers/net/ethernet/amazon/ena/ena_ethtool.c @@ -697,8 +697,8 @@ static int ena_set_rxfh(struct net_device *netdev, const u32 *indir, if (indir) { for (i = 0; i < ENA_RX_RSS_TABLE_SIZE; i++) { rc = ena_com_indirect_table_fill_entry(ena_dev, - ENA_IO_RXQ_IDX(indir[i]), - i); + i, + ENA_IO_RXQ_IDX(indir[i])); if (unlikely(rc)) { netif_err(adapter, drv, netdev, "Cannot fill indirect table (index is too large)\n"); diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c index a6eacf2099c3027c1e11c36fba99779808e64823..9c83642922c77f5ef63501172ae56c681b7304a6 100644 --- a/drivers/net/ethernet/amazon/ena/ena_netdev.c +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c @@ -224,28 +224,23 @@ static int ena_setup_tx_resources(struct ena_adapter *adapter, int qid) if (!tx_ring->tx_buffer_info) { tx_ring->tx_buffer_info = vzalloc(size); if (!tx_ring->tx_buffer_info) - return -ENOMEM; + goto err_tx_buffer_info; } size = sizeof(u16) * tx_ring->ring_size; tx_ring->free_tx_ids = vzalloc_node(size, node); if (!tx_ring->free_tx_ids) { tx_ring->free_tx_ids = vzalloc(size); - if (!tx_ring->free_tx_ids) { - vfree(tx_ring->tx_buffer_info); - return -ENOMEM; - } + if (!tx_ring->free_tx_ids) + goto err_free_tx_ids; } size = tx_ring->tx_max_header_size; tx_ring->push_buf_intermediate_buf = vzalloc_node(size, node); if (!tx_ring->push_buf_intermediate_buf) { tx_ring->push_buf_intermediate_buf = vzalloc(size); - if (!tx_ring->push_buf_intermediate_buf) { - vfree(tx_ring->tx_buffer_info); - vfree(tx_ring->free_tx_ids); - return -ENOMEM; - } + if (!tx_ring->push_buf_intermediate_buf) + goto err_push_buf_intermediate_buf; } /* Req id ring for TX out of order completions */ @@ -259,6 +254,15 @@ static int ena_setup_tx_resources(struct ena_adapter *adapter, int qid) tx_ring->next_to_clean = 0; tx_ring->cpu = ena_irq->cpu; return 0; + +err_push_buf_intermediate_buf: + vfree(tx_ring->free_tx_ids); + tx_ring->free_tx_ids = NULL; +err_free_tx_ids: + vfree(tx_ring->tx_buffer_info); + tx_ring->tx_buffer_info = NULL; +err_tx_buffer_info: + return -ENOMEM; } /* ena_free_tx_resources - Free I/O Tx Resources per Queue @@ -378,6 +382,7 @@ static int ena_setup_rx_resources(struct ena_adapter *adapter, rx_ring->free_rx_ids = vzalloc(size); if (!rx_ring->free_rx_ids) { vfree(rx_ring->rx_buffer_info); + rx_ring->rx_buffer_info = NULL; return -ENOMEM; } } @@ -1820,6 +1825,7 @@ err_setup_rx: err_setup_tx: ena_free_io_irq(adapter); err_req_irq: + ena_del_napi(adapter); return rc; } @@ -2236,7 +2242,7 @@ static netdev_tx_t ena_start_xmit(struct sk_buff *skb, struct net_device *dev) } } - if (netif_xmit_stopped(txq) || !skb->xmit_more) { + if (netif_xmit_stopped(txq) || !netdev_xmit_more()) { /* trigger the dma engine. ena_com_write_sq_doorbell() * has a mb */ @@ -2258,8 +2264,7 @@ error_drop_packet: } static u16 ena_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { u16 qid; /* we suspect that this is good for in--kernel network services that @@ -2269,7 +2274,7 @@ static u16 ena_select_queue(struct net_device *dev, struct sk_buff *skb, if (skb_rx_queue_recorded(skb)) qid = skb_get_rx_queue(skb); else - qid = fallback(dev, skb, NULL); + qid = netdev_pick_tx(dev, skb, NULL); return qid; } @@ -2292,7 +2297,7 @@ static void ena_config_host_info(struct ena_com_dev *ena_dev, host_info->bdf = (pdev->bus->number << 8) | pdev->devfn; host_info->os_type = ENA_ADMIN_OS_LINUX; host_info->kernel_ver = LINUX_VERSION_CODE; - strncpy(host_info->kernel_ver_str, utsname()->version, + strlcpy(host_info->kernel_ver_str, utsname()->version, sizeof(host_info->kernel_ver_str) - 1); host_info->os_dist = 0; strncpy(host_info->os_dist_str, utsname()->release, diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c index 4666084eda16a318a4042b81c2d3a03cb7664a19..d5fd49dd25f336aedbe166a8c6bdd74ef6296204 100644 --- a/drivers/net/ethernet/amd/xgbe/xgbe-dev.c +++ b/drivers/net/ethernet/amd/xgbe/xgbe-dev.c @@ -1887,7 +1887,7 @@ static void xgbe_dev_xmit(struct xgbe_channel *channel) smp_wmb(); ring->cur = cur_index + 1; - if (!packet->skb->xmit_more || + if (!netdev_xmit_more() || netif_xmit_stopped(netdev_get_tx_queue(pdata->netdev, channel->queue_index))) xgbe_tx_start_xmit(channel, ring); diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-drv.c b/drivers/net/ethernet/amd/xgbe/xgbe-drv.c index 0cc911f928b143c5e511b83f3e1a4494443a1f2e..3dd0cecddba80d6538dba78a0bbaf30332d9881f 100644 --- a/drivers/net/ethernet/amd/xgbe/xgbe-drv.c +++ b/drivers/net/ethernet/amd/xgbe/xgbe-drv.c @@ -1612,7 +1612,7 @@ static int xgbe_set_hwtstamp_settings(struct xgbe_prv_data *pdata, /* PTP v2, UDP, any kind of event packet */ case HWTSTAMP_FILTER_PTP_V2_L4_EVENT: XGMAC_SET_BITS(mac_tscr, MAC_TSCR, TSVER2ENA, 1); - /* PTP v1, UDP, any kind of event packet */ + /* Fall through - to PTP v1, UDP, any kind of event packet */ case HWTSTAMP_FILTER_PTP_V1_L4_EVENT: XGMAC_SET_BITS(mac_tscr, MAC_TSCR, TSIPV4ENA, 1); XGMAC_SET_BITS(mac_tscr, MAC_TSCR, TSIPV6ENA, 1); @@ -1623,7 +1623,7 @@ static int xgbe_set_hwtstamp_settings(struct xgbe_prv_data *pdata, /* PTP v2, UDP, Sync packet */ case HWTSTAMP_FILTER_PTP_V2_L4_SYNC: XGMAC_SET_BITS(mac_tscr, MAC_TSCR, TSVER2ENA, 1); - /* PTP v1, UDP, Sync packet */ + /* Fall through - to PTP v1, UDP, Sync packet */ case HWTSTAMP_FILTER_PTP_V1_L4_SYNC: XGMAC_SET_BITS(mac_tscr, MAC_TSCR, TSIPV4ENA, 1); XGMAC_SET_BITS(mac_tscr, MAC_TSCR, TSIPV6ENA, 1); @@ -1634,7 +1634,7 @@ static int xgbe_set_hwtstamp_settings(struct xgbe_prv_data *pdata, /* PTP v2, UDP, Delay_req packet */ case HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ: XGMAC_SET_BITS(mac_tscr, MAC_TSCR, TSVER2ENA, 1); - /* PTP v1, UDP, Delay_req packet */ + /* Fall through - to PTP v1, UDP, Delay_req packet */ case HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ: XGMAC_SET_BITS(mac_tscr, MAC_TSCR, TSIPV4ENA, 1); XGMAC_SET_BITS(mac_tscr, MAC_TSCR, TSIPV6ENA, 1); diff --git a/drivers/net/ethernet/aquantia/Kconfig b/drivers/net/ethernet/aquantia/Kconfig index 7d623e90dc19aae2a46840153a8484dae984e1cf..12472c5bb34d21a65f3ac8ff8dcebc1f590c4101 100644 --- a/drivers/net/ethernet/aquantia/Kconfig +++ b/drivers/net/ethernet/aquantia/Kconfig @@ -17,7 +17,8 @@ if NET_VENDOR_AQUANTIA config AQTION tristate "aQuantia AQtion(tm) Support" - depends on PCI && X86_64 + depends on PCI + depends on X86_64 || ARM64 || COMPILE_TEST ---help--- This enables the support for the aQuantia AQtion(tm) Ethernet card. diff --git a/drivers/net/ethernet/aquantia/atlantic/Makefile b/drivers/net/ethernet/aquantia/atlantic/Makefile index 4556630ee286cf42820ba2900727151012b4150c..1f99cf832476f0d7f5327672264e7c6990c74564 100644 --- a/drivers/net/ethernet/aquantia/atlantic/Makefile +++ b/drivers/net/ethernet/aquantia/atlantic/Makefile @@ -36,6 +36,7 @@ atlantic-objs := aq_main.o \ aq_ring.o \ aq_hw_utils.o \ aq_ethtool.o \ + aq_drvinfo.o \ aq_filters.o \ hw_atl/hw_atl_a0.o \ hw_atl/hw_atl_b0.o \ diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_cfg.h b/drivers/net/ethernet/aquantia/atlantic/aq_cfg.h index 3944ce7f087084b458dea5b6f9de3c213393d831..8f35c3f883f0062f89c3826eec4c16729a2d9851 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_cfg.h +++ b/drivers/net/ethernet/aquantia/atlantic/aq_cfg.h @@ -16,7 +16,7 @@ #define AQ_CFG_TCS_DEF 1U #define AQ_CFG_TXDS_DEF 4096U -#define AQ_CFG_RXDS_DEF 1024U +#define AQ_CFG_RXDS_DEF 2048U #define AQ_CFG_IS_POLLING_DEF 0U @@ -34,10 +34,16 @@ #define AQ_CFG_TCS_MAX 8U #define AQ_CFG_TX_FRAME_MAX (16U * 1024U) -#define AQ_CFG_RX_FRAME_MAX (4U * 1024U) +#define AQ_CFG_RX_FRAME_MAX (2U * 1024U) #define AQ_CFG_TX_CLEAN_BUDGET 256U +#define AQ_CFG_RX_REFILL_THRES 32U + +#define AQ_CFG_RX_HDR_SIZE 256U + +#define AQ_CFG_RX_PAGEORDER 0U + /* LRO */ #define AQ_CFG_IS_LRO_DEF 1U diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_common.h b/drivers/net/ethernet/aquantia/atlantic/aq_common.h index 6b6d1724676e622dcb1998e0af6bf9a2177d20e3..235bb3a72d669cae413160872169ce3df420f649 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_common.h +++ b/drivers/net/ethernet/aquantia/atlantic/aq_common.h @@ -41,9 +41,6 @@ #define AQ_DEVICE_ID_AQC111S 0x91B1 #define AQ_DEVICE_ID_AQC112S 0x92B1 -#define AQ_DEVICE_ID_AQC111E 0x51B1 -#define AQ_DEVICE_ID_AQC112E 0x52B1 - #define HW_ATL_NIC_NAME "aQuantia AQtion 10Gbit Network Adapter" #define AQ_HWREV_ANY 0 diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_drvinfo.c b/drivers/net/ethernet/aquantia/atlantic/aq_drvinfo.c new file mode 100644 index 0000000000000000000000000000000000000000..f5a92b2a5cd62f60ac6f519b4c157b458703452e --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic/aq_drvinfo.c @@ -0,0 +1,125 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* Copyright (C) 2014-2019 aQuantia Corporation. */ + +/* File aq_drvinfo.c: Definition of common code for firmware info in sys.*/ + +#include +#include +#include +#include +#include +#include +#include + +#include "aq_drvinfo.h" + +static int aq_hwmon_read(struct device *dev, enum hwmon_sensor_types type, + u32 attr, int channel, long *value) +{ + struct aq_nic_s *aq_nic = dev_get_drvdata(dev); + int temp; + int err; + + if (!aq_nic) + return -EIO; + + if (type != hwmon_temp) + return -EOPNOTSUPP; + + if (!aq_nic->aq_fw_ops->get_phy_temp) + return -EOPNOTSUPP; + + switch (attr) { + case hwmon_temp_input: + err = aq_nic->aq_fw_ops->get_phy_temp(aq_nic->aq_hw, &temp); + *value = temp; + return err; + default: + return -EOPNOTSUPP; + } +} + +static int aq_hwmon_read_string(struct device *dev, + enum hwmon_sensor_types type, + u32 attr, int channel, const char **str) +{ + struct aq_nic_s *aq_nic = dev_get_drvdata(dev); + + if (!aq_nic) + return -EIO; + + if (type != hwmon_temp) + return -EOPNOTSUPP; + + if (!aq_nic->aq_fw_ops->get_phy_temp) + return -EOPNOTSUPP; + + switch (attr) { + case hwmon_temp_label: + *str = "PHY Temperature"; + return 0; + default: + return -EOPNOTSUPP; + } +} + +static umode_t aq_hwmon_is_visible(const void *data, + enum hwmon_sensor_types type, + u32 attr, int channel) +{ + if (type != hwmon_temp) + return 0; + + switch (attr) { + case hwmon_temp_input: + case hwmon_temp_label: + return 0444; + default: + return 0; + } +} + +static const struct hwmon_ops aq_hwmon_ops = { + .is_visible = aq_hwmon_is_visible, + .read = aq_hwmon_read, + .read_string = aq_hwmon_read_string, +}; + +static u32 aq_hwmon_temp_config[] = { + HWMON_T_INPUT | HWMON_T_LABEL, + 0, +}; + +static const struct hwmon_channel_info aq_hwmon_temp = { + .type = hwmon_temp, + .config = aq_hwmon_temp_config, +}; + +static const struct hwmon_channel_info *aq_hwmon_info[] = { + &aq_hwmon_temp, + NULL, +}; + +static const struct hwmon_chip_info aq_hwmon_chip_info = { + .ops = &aq_hwmon_ops, + .info = aq_hwmon_info, +}; + +int aq_drvinfo_init(struct net_device *ndev) +{ + struct aq_nic_s *aq_nic = netdev_priv(ndev); + struct device *dev = &aq_nic->pdev->dev; + struct device *hwmon_dev; + int err = 0; + + hwmon_dev = devm_hwmon_device_register_with_info(dev, + ndev->name, + aq_nic, + &aq_hwmon_chip_info, + NULL); + + if (IS_ERR(hwmon_dev)) + err = PTR_ERR(hwmon_dev); + + return err; +} diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_drvinfo.h b/drivers/net/ethernet/aquantia/atlantic/aq_drvinfo.h new file mode 100644 index 0000000000000000000000000000000000000000..41fbb1358068aebaabadf8d44a3d23ea0f4a9088 --- /dev/null +++ b/drivers/net/ethernet/aquantia/atlantic/aq_drvinfo.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* Copyright (C) 2014-2017 aQuantia Corporation. */ + +/* File aq_drvinfo.h: Declaration of common code for firmware info in sys.*/ + +#ifndef AQ_DRVINFO_H +#define AQ_DRVINFO_H + +#include "aq_nic.h" +#include "aq_hw.h" +#include "hw_atl/hw_atl_utils.h" + +int aq_drvinfo_init(struct net_device *ndev); + +#endif /* AQ_DRVINFO_H */ diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_ethtool.c b/drivers/net/ethernet/aquantia/atlantic/aq_ethtool.c index a718d7a1f76c8b28782d8e829d235a36b05a0bc5..79da4809477071e65d5c5f934651f0a207dd6c50 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_ethtool.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_ethtool.c @@ -405,8 +405,10 @@ static int aq_ethtool_get_eee(struct net_device *ndev, struct ethtool_eee *eee) if (!aq_nic->aq_fw_ops->get_eee_rate) return -EOPNOTSUPP; + mutex_lock(&aq_nic->fwreq_mutex); err = aq_nic->aq_fw_ops->get_eee_rate(aq_nic->aq_hw, &rate, &supported_rates); + mutex_unlock(&aq_nic->fwreq_mutex); if (err < 0) return err; @@ -439,8 +441,10 @@ static int aq_ethtool_set_eee(struct net_device *ndev, struct ethtool_eee *eee) !aq_nic->aq_fw_ops->set_eee_rate)) return -EOPNOTSUPP; + mutex_lock(&aq_nic->fwreq_mutex); err = aq_nic->aq_fw_ops->get_eee_rate(aq_nic->aq_hw, &rate, &supported_rates); + mutex_unlock(&aq_nic->fwreq_mutex); if (err < 0) return err; @@ -452,20 +456,28 @@ static int aq_ethtool_set_eee(struct net_device *ndev, struct ethtool_eee *eee) cfg->eee_speeds = 0; } - return aq_nic->aq_fw_ops->set_eee_rate(aq_nic->aq_hw, rate); + mutex_lock(&aq_nic->fwreq_mutex); + err = aq_nic->aq_fw_ops->set_eee_rate(aq_nic->aq_hw, rate); + mutex_unlock(&aq_nic->fwreq_mutex); + + return err; } static int aq_ethtool_nway_reset(struct net_device *ndev) { struct aq_nic_s *aq_nic = netdev_priv(ndev); + int err = 0; if (unlikely(!aq_nic->aq_fw_ops->renegotiate)) return -EOPNOTSUPP; - if (netif_running(ndev)) - return aq_nic->aq_fw_ops->renegotiate(aq_nic->aq_hw); + if (netif_running(ndev)) { + mutex_lock(&aq_nic->fwreq_mutex); + err = aq_nic->aq_fw_ops->renegotiate(aq_nic->aq_hw); + mutex_unlock(&aq_nic->fwreq_mutex); + } - return 0; + return err; } static void aq_ethtool_get_pauseparam(struct net_device *ndev, @@ -503,7 +515,9 @@ static int aq_ethtool_set_pauseparam(struct net_device *ndev, else aq_nic->aq_hw->aq_nic_cfg->flow_control &= ~AQ_NIC_FC_TX; + mutex_lock(&aq_nic->fwreq_mutex); err = aq_nic->aq_fw_ops->set_flow_control(aq_nic->aq_hw); + mutex_unlock(&aq_nic->fwreq_mutex); return err; } diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_hw.h b/drivers/net/ethernet/aquantia/atlantic/aq_hw.h index 81aab73dc22f0fd33d56c7d984094f28133782af..95fd6c852a9dd7410eb73415a729a76ca0790294 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_hw.h +++ b/drivers/net/ethernet/aquantia/atlantic/aq_hw.h @@ -88,6 +88,8 @@ struct aq_stats_s { #define AQ_HW_IRQ_MSI 2U #define AQ_HW_IRQ_MSIX 3U +#define AQ_HW_SERVICE_IRQS 1U + #define AQ_HW_POWER_STATE_D0 0U #define AQ_HW_POWER_STATE_D3 3U @@ -259,6 +261,8 @@ struct aq_fw_ops { int (*update_stats)(struct aq_hw_s *self); + int (*get_phy_temp)(struct aq_hw_s *self, int *temp); + u32 (*get_flow_control)(struct aq_hw_s *self, u32 *fcmode); int (*set_flow_control)(struct aq_hw_s *self); diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_hw_utils.c b/drivers/net/ethernet/aquantia/atlantic/aq_hw_utils.c index d526c4f19d3434dee500184ee1ecdbd68c670188..22a1c784dc9c7305c571e053857e666540234c56 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_hw_utils.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_hw_utils.c @@ -53,6 +53,18 @@ void aq_hw_write_reg(struct aq_hw_s *hw, u32 reg, u32 value) writel(value, hw->mmio + reg); } +/* Most of 64-bit registers are in LSW, MSW form. + Counters are normally implemented by HW as latched pairs: + reading LSW first locks MSW, to overcome LSW overflow + */ +u64 aq_hw_read_reg64(struct aq_hw_s *hw, u32 reg) +{ + u64 value = aq_hw_read_reg(hw, reg); + + value |= (u64)aq_hw_read_reg(hw, reg + 4) << 32; + return value; +} + int aq_hw_err_from_flags(struct aq_hw_s *hw) { int err = 0; diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_hw_utils.h b/drivers/net/ethernet/aquantia/atlantic/aq_hw_utils.h index bc711238ca0ceab6e30e6fc32c24150c2d68acd0..bf73428ed689cf616dcd3bb1951cd1b160f53fb3 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_hw_utils.h +++ b/drivers/net/ethernet/aquantia/atlantic/aq_hw_utils.h @@ -35,6 +35,7 @@ void aq_hw_write_reg_bit(struct aq_hw_s *aq_hw, u32 addr, u32 msk, u32 aq_hw_read_reg_bit(struct aq_hw_s *aq_hw, u32 addr, u32 msk, u32 shift); u32 aq_hw_read_reg(struct aq_hw_s *hw, u32 reg); void aq_hw_write_reg(struct aq_hw_s *hw, u32 reg, u32 value); +u64 aq_hw_read_reg64(struct aq_hw_s *hw, u32 reg); int aq_hw_err_from_flags(struct aq_hw_s *hw); #endif /* AQ_HW_UTILS_H */ diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_main.c b/drivers/net/ethernet/aquantia/atlantic/aq_main.c index 2a11c1eefd8fb900cad8d4c958cbcc51998c2f28..1ea8b77fc1a719f0e4421aa5b4e8f748298ef8fa 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_main.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_main.c @@ -23,8 +23,17 @@ MODULE_VERSION(AQ_CFG_DRV_VERSION); MODULE_AUTHOR(AQ_CFG_DRV_AUTHOR); MODULE_DESCRIPTION(AQ_CFG_DRV_DESC); +static const char aq_ndev_driver_name[] = AQ_CFG_DRV_NAME; + static const struct net_device_ops aq_ndev_ops; +static struct workqueue_struct *aq_ndev_wq; + +void aq_ndev_schedule_work(struct work_struct *work) +{ + queue_work(aq_ndev_wq, work); +} + struct net_device *aq_ndev_alloc(void) { struct net_device *ndev = NULL; @@ -209,3 +218,35 @@ static const struct net_device_ops aq_ndev_ops = { .ndo_vlan_rx_add_vid = aq_ndo_vlan_rx_add_vid, .ndo_vlan_rx_kill_vid = aq_ndo_vlan_rx_kill_vid, }; + +static int __init aq_ndev_init_module(void) +{ + int ret; + + aq_ndev_wq = create_singlethread_workqueue(aq_ndev_driver_name); + if (!aq_ndev_wq) { + pr_err("Failed to create workqueue\n"); + return -ENOMEM; + } + + ret = aq_pci_func_register_driver(); + if (ret) { + destroy_workqueue(aq_ndev_wq); + return ret; + } + + return 0; +} + +static void __exit aq_ndev_exit_module(void) +{ + aq_pci_func_unregister_driver(); + + if (aq_ndev_wq) { + destroy_workqueue(aq_ndev_wq); + aq_ndev_wq = NULL; + } +} + +module_init(aq_ndev_init_module); +module_exit(aq_ndev_exit_module); diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_main.h b/drivers/net/ethernet/aquantia/atlantic/aq_main.h index ce92152eb43e596dc81f34713efcefab71e88e56..5448b82fb7ea15c37c6c9769752c7db12396714b 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_main.h +++ b/drivers/net/ethernet/aquantia/atlantic/aq_main.h @@ -13,7 +13,9 @@ #define AQ_MAIN_H #include "aq_common.h" +#include "aq_nic.h" +void aq_ndev_schedule_work(struct work_struct *work); struct net_device *aq_ndev_alloc(void); #endif /* AQ_MAIN_H */ diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_nic.c b/drivers/net/ethernet/aquantia/atlantic/aq_nic.c index ff83667410bd9fdb8183a483521996bbfd90fef3..e82d25a91bc18163e11902acc96179bf50364ece 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_nic.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_nic.c @@ -14,6 +14,7 @@ #include "aq_vec.h" #include "aq_hw.h" #include "aq_pci_func.h" +#include "aq_main.h" #include #include @@ -73,6 +74,7 @@ void aq_nic_cfg_start(struct aq_nic_s *self) cfg->tx_itr = aq_itr_tx; cfg->rx_itr = aq_itr_rx; + cfg->rxpageorder = AQ_CFG_RX_PAGEORDER; cfg->is_rss = AQ_CFG_IS_RSS_DEF; cfg->num_rss_queues = AQ_CFG_NUM_RSS_QUEUES_DEF; cfg->aq_rss.base_cpu_number = AQ_CFG_RSS_BASE_CPU_NUM_DEF; @@ -91,7 +93,8 @@ void aq_nic_cfg_start(struct aq_nic_s *self) /*rss rings */ cfg->vecs = min(cfg->aq_hw_caps->vecs, AQ_CFG_VECS_DEF); cfg->vecs = min(cfg->vecs, num_online_cpus()); - cfg->vecs = min(cfg->vecs, self->irqvecs); + if (self->irqvecs > AQ_HW_SERVICE_IRQS) + cfg->vecs = min(cfg->vecs, self->irqvecs - AQ_HW_SERVICE_IRQS); /* cfg->vecs should be power of 2 for RSS */ if (cfg->vecs >= 8U) cfg->vecs = 8U; @@ -115,6 +118,15 @@ void aq_nic_cfg_start(struct aq_nic_s *self) cfg->vecs = 1U; } + /* Check if we have enough vectors allocated for + * link status IRQ. If no - we'll know link state from + * slower service task. + */ + if (AQ_HW_SERVICE_IRQS > 0 && cfg->vecs + 1 <= self->irqvecs) + cfg->link_irq_vec = cfg->vecs; + else + cfg->link_irq_vec = 0; + cfg->link_speed_msk &= cfg->aq_hw_caps->link_speed_msk; cfg->features = cfg->aq_hw_caps->hw_features; } @@ -160,30 +172,48 @@ static int aq_nic_update_link_status(struct aq_nic_s *self) return 0; } -static void aq_nic_service_timer_cb(struct timer_list *t) +static irqreturn_t aq_linkstate_threaded_isr(int irq, void *private) { - struct aq_nic_s *self = from_timer(self, t, service_timer); - int ctimer = AQ_CFG_SERVICE_TIMER_INTERVAL; - int err = 0; + struct aq_nic_s *self = private; + + if (!self) + return IRQ_NONE; + + aq_nic_update_link_status(self); + + self->aq_hw_ops->hw_irq_enable(self->aq_hw, + BIT(self->aq_nic_cfg.link_irq_vec)); + return IRQ_HANDLED; +} + +static void aq_nic_service_task(struct work_struct *work) +{ + struct aq_nic_s *self = container_of(work, struct aq_nic_s, + service_task); + int err; if (aq_utils_obj_test(&self->flags, AQ_NIC_FLAGS_IS_NOT_READY)) - goto err_exit; + return; err = aq_nic_update_link_status(self); if (err) - goto err_exit; + return; + mutex_lock(&self->fwreq_mutex); if (self->aq_fw_ops->update_stats) self->aq_fw_ops->update_stats(self->aq_hw); + mutex_unlock(&self->fwreq_mutex); aq_nic_update_ndev_stats(self); +} + +static void aq_nic_service_timer_cb(struct timer_list *t) +{ + struct aq_nic_s *self = from_timer(self, t, service_timer); - /* If no link - use faster timer rate to detect link up asap */ - if (!netif_carrier_ok(self->ndev)) - ctimer = max(ctimer / 2, 1); + mod_timer(&self->service_timer, jiffies + AQ_CFG_SERVICE_TIMER_INTERVAL); -err_exit: - mod_timer(&self->service_timer, jiffies + ctimer); + aq_ndev_schedule_work(&self->service_task); } static void aq_nic_polling_timer_cb(struct timer_list *t) @@ -213,8 +243,10 @@ int aq_nic_ndev_register(struct aq_nic_s *self) if (err) goto err_exit; + mutex_lock(&self->fwreq_mutex); err = self->aq_fw_ops->get_mac_permanent(self->aq_hw, self->ndev->dev_addr); + mutex_unlock(&self->fwreq_mutex); if (err) goto err_exit; @@ -283,7 +315,9 @@ int aq_nic_init(struct aq_nic_s *self) unsigned int i = 0U; self->power_state = AQ_HW_POWER_STATE_D0; + mutex_lock(&self->fwreq_mutex); err = self->aq_hw_ops->hw_reset(self->aq_hw); + mutex_unlock(&self->fwreq_mutex); if (err < 0) goto err_exit; @@ -333,9 +367,11 @@ int aq_nic_start(struct aq_nic_s *self) err = aq_nic_update_interrupt_moderation_settings(self); if (err) goto err_exit; + + INIT_WORK(&self->service_task, aq_nic_service_task); + timer_setup(&self->service_timer, aq_nic_service_timer_cb, 0); - mod_timer(&self->service_timer, jiffies + - AQ_CFG_SERVICE_TIMER_INTERVAL); + aq_nic_service_timer_cb(&self->service_timer); if (self->aq_nic_cfg.is_polling) { timer_setup(&self->polling_timer, aq_nic_polling_timer_cb, 0); @@ -344,13 +380,25 @@ int aq_nic_start(struct aq_nic_s *self) } else { for (i = 0U, aq_vec = self->aq_vec[0]; self->aq_vecs > i; ++i, aq_vec = self->aq_vec[i]) { - err = aq_pci_func_alloc_irq(self, i, - self->ndev->name, aq_vec, + err = aq_pci_func_alloc_irq(self, i, self->ndev->name, + aq_vec_isr, aq_vec, aq_vec_get_affinity_mask(aq_vec)); if (err < 0) goto err_exit; } + if (self->aq_nic_cfg.link_irq_vec) { + int irqvec = pci_irq_vector(self->pdev, + self->aq_nic_cfg.link_irq_vec); + err = request_threaded_irq(irqvec, NULL, + aq_linkstate_threaded_isr, + IRQF_SHARED, + self->ndev->name, self); + if (err < 0) + goto err_exit; + self->msix_entry_mask |= (1 << self->aq_nic_cfg.link_irq_vec); + } + err = self->aq_hw_ops->hw_irq_enable(self->aq_hw, AQ_CFG_IRQ_MASK); if (err < 0) @@ -652,7 +700,14 @@ void aq_nic_get_stats(struct aq_nic_s *self, u64 *data) unsigned int i = 0U; unsigned int count = 0U; struct aq_vec_s *aq_vec = NULL; - struct aq_stats_s *stats = self->aq_hw_ops->hw_get_hw_stats(self->aq_hw); + struct aq_stats_s *stats; + + if (self->aq_fw_ops->update_stats) { + mutex_lock(&self->fwreq_mutex); + self->aq_fw_ops->update_stats(self->aq_hw); + mutex_unlock(&self->fwreq_mutex); + } + stats = self->aq_hw_ops->hw_get_hw_stats(self->aq_hw); if (!stats) goto err_exit; @@ -698,11 +753,12 @@ static void aq_nic_update_ndev_stats(struct aq_nic_s *self) struct net_device *ndev = self->ndev; struct aq_stats_s *stats = self->aq_hw_ops->hw_get_hw_stats(self->aq_hw); - ndev->stats.rx_packets = stats->uprc + stats->mprc + stats->bprc; - ndev->stats.rx_bytes = stats->ubrc + stats->mbrc + stats->bbrc; + ndev->stats.rx_packets = stats->dma_pkt_rc; + ndev->stats.rx_bytes = stats->dma_oct_rc; ndev->stats.rx_errors = stats->erpr; - ndev->stats.tx_packets = stats->uptc + stats->mptc + stats->bptc; - ndev->stats.tx_bytes = stats->ubtc + stats->mbtc + stats->bbtc; + ndev->stats.rx_dropped = stats->dpc; + ndev->stats.tx_packets = stats->dma_pkt_tc; + ndev->stats.tx_bytes = stats->dma_oct_tc; ndev->stats.tx_errors = stats->erpt; ndev->stats.multicast = stats->mprc; } @@ -839,7 +895,9 @@ int aq_nic_set_link_ksettings(struct aq_nic_s *self, self->aq_nic_cfg.is_autoneg = false; } + mutex_lock(&self->fwreq_mutex); err = self->aq_fw_ops->set_link_speed(self->aq_hw, rate); + mutex_unlock(&self->fwreq_mutex); if (err < 0) goto err_exit; @@ -872,6 +930,7 @@ int aq_nic_stop(struct aq_nic_s *self) netif_carrier_off(self->ndev); del_timer_sync(&self->service_timer); + cancel_work_sync(&self->service_task); self->aq_hw_ops->hw_irq_disable(self->aq_hw, AQ_CFG_IRQ_MASK); @@ -899,14 +958,22 @@ void aq_nic_deinit(struct aq_nic_s *self) self->aq_vecs > i; ++i, aq_vec = self->aq_vec[i]) aq_vec_deinit(aq_vec); - self->aq_fw_ops->deinit(self->aq_hw); + if (likely(self->aq_fw_ops->deinit)) { + mutex_lock(&self->fwreq_mutex); + self->aq_fw_ops->deinit(self->aq_hw); + mutex_unlock(&self->fwreq_mutex); + } if (self->power_state != AQ_HW_POWER_STATE_D0 || - self->aq_hw->aq_nic_cfg->wol) { - self->aq_fw_ops->set_power(self->aq_hw, - self->power_state, - self->ndev->dev_addr); - } + self->aq_hw->aq_nic_cfg->wol) + if (likely(self->aq_fw_ops->set_power)) { + mutex_lock(&self->fwreq_mutex); + self->aq_fw_ops->set_power(self->aq_hw, + self->power_state, + self->ndev->dev_addr); + mutex_unlock(&self->fwreq_mutex); + } + err_exit:; } diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_nic.h b/drivers/net/ethernet/aquantia/atlantic/aq_nic.h index 8e34c1e49bf2773c41e402664723b879da6f9ad7..c03d38ed105de820b4601620b2dc12b576bdeb24 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_nic.h +++ b/drivers/net/ethernet/aquantia/atlantic/aq_nic.h @@ -26,11 +26,13 @@ struct aq_nic_cfg_s { u64 features; u32 rxds; /* rx ring size, descriptors # */ u32 txds; /* tx ring size, descriptors # */ - u32 vecs; /* vecs==allocated irqs */ + u32 vecs; /* allocated rx/tx vectors */ + u32 link_irq_vec; u32 irq_type; u32 itr; u16 rx_itr; u16 tx_itr; + u32 rxpageorder; u32 num_rss_queues; u32 mtu; u32 flow_control; @@ -91,6 +93,7 @@ struct aq_nic_s { const struct aq_fw_ops *aq_fw_ops; struct aq_nic_cfg_s aq_nic_cfg; struct timer_list service_timer; + struct work_struct service_task; struct timer_list polling_timer; struct aq_hw_link_status_s link_status; struct { @@ -103,6 +106,8 @@ struct aq_nic_s { struct pci_dev *pdev; unsigned int msix_entry_mask; u32 irqvecs; + /* mutex to serialize FW interface access operations */ + struct mutex fwreq_mutex; struct aq_hw_rx_fltrs_s aq_hw_rx_fltrs; }; diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_pci_func.c b/drivers/net/ethernet/aquantia/atlantic/aq_pci_func.c index 0217ff4669a46f30b26c2f70d4f3f9e71813ece0..9cb0864d6d8da8a34419610425bcb9e726decb55 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_pci_func.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_pci_func.c @@ -20,6 +20,7 @@ #include "hw_atl/hw_atl_a0.h" #include "hw_atl/hw_atl_b0.h" #include "aq_filters.h" +#include "aq_drvinfo.h" static const struct pci_device_id aq_pci_tbl[] = { { PCI_VDEVICE(AQUANTIA, AQ_DEVICE_ID_0001), }, @@ -42,9 +43,6 @@ static const struct pci_device_id aq_pci_tbl[] = { { PCI_VDEVICE(AQUANTIA, AQ_DEVICE_ID_AQC111S), }, { PCI_VDEVICE(AQUANTIA, AQ_DEVICE_ID_AQC112S), }, - { PCI_VDEVICE(AQUANTIA, AQ_DEVICE_ID_AQC111E), }, - { PCI_VDEVICE(AQUANTIA, AQ_DEVICE_ID_AQC112E), }, - {} }; @@ -74,9 +72,6 @@ static const struct aq_board_revision_s hw_atl_boards[] = { { AQ_DEVICE_ID_AQC109S, AQ_HWREV_ANY, &hw_atl_ops_b1, &hw_atl_b0_caps_aqc109s, }, { AQ_DEVICE_ID_AQC111S, AQ_HWREV_ANY, &hw_atl_ops_b1, &hw_atl_b0_caps_aqc111s, }, { AQ_DEVICE_ID_AQC112S, AQ_HWREV_ANY, &hw_atl_ops_b1, &hw_atl_b0_caps_aqc112s, }, - - { AQ_DEVICE_ID_AQC111E, AQ_HWREV_ANY, &hw_atl_ops_b1, &hw_atl_b0_caps_aqc111e, }, - { AQ_DEVICE_ID_AQC112E, AQ_HWREV_ANY, &hw_atl_ops_b1, &hw_atl_b0_caps_aqc112e, }, }; MODULE_DEVICE_TABLE(pci, aq_pci_tbl); @@ -139,26 +134,27 @@ err_exit: } int aq_pci_func_alloc_irq(struct aq_nic_s *self, unsigned int i, - char *name, void *aq_vec, cpumask_t *affinity_mask) + char *name, irq_handler_t irq_handler, + void *irq_arg, cpumask_t *affinity_mask) { struct pci_dev *pdev = self->pdev; int err; if (pdev->msix_enabled || pdev->msi_enabled) - err = request_irq(pci_irq_vector(pdev, i), aq_vec_isr, 0, - name, aq_vec); + err = request_irq(pci_irq_vector(pdev, i), irq_handler, 0, + name, irq_arg); else err = request_irq(pci_irq_vector(pdev, i), aq_vec_isr_legacy, - IRQF_SHARED, name, aq_vec); + IRQF_SHARED, name, irq_arg); if (err >= 0) { self->msix_entry_mask |= (1 << i); - self->aq_vec[i] = aq_vec; - if (pdev->msix_enabled) + if (pdev->msix_enabled && affinity_mask) irq_set_affinity_hint(pci_irq_vector(pdev, i), affinity_mask); } + return err; } @@ -166,16 +162,22 @@ void aq_pci_func_free_irqs(struct aq_nic_s *self) { struct pci_dev *pdev = self->pdev; unsigned int i; + void *irq_data; for (i = 32U; i--;) { if (!((1U << i) & self->msix_entry_mask)) continue; - if (i >= AQ_CFG_VECS_MAX) + if (self->aq_nic_cfg.link_irq_vec && + i == self->aq_nic_cfg.link_irq_vec) + irq_data = self; + else if (i < AQ_CFG_VECS_MAX) + irq_data = self->aq_vec[i]; + else continue; if (pdev->msix_enabled) irq_set_affinity_hint(pci_irq_vector(pdev, i), NULL); - free_irq(pci_irq_vector(pdev, i), self->aq_vec[i]); + free_irq(pci_irq_vector(pdev, i), irq_data); self->msix_entry_mask &= ~(1U << i); } } @@ -185,7 +187,7 @@ unsigned int aq_pci_func_get_irq_type(struct aq_nic_s *self) if (self->pdev->msix_enabled) return AQ_HW_IRQ_MSIX; if (self->pdev->msi_enabled) - return AQ_HW_IRQ_MSIX; + return AQ_HW_IRQ_MSI; return AQ_HW_IRQ_LEGACY; } @@ -223,6 +225,8 @@ static int aq_pci_probe(struct pci_dev *pdev, SET_NETDEV_DEV(ndev, &pdev->dev); pci_set_drvdata(pdev, self); + mutex_init(&self->fwreq_mutex); + err = aq_pci_probe_get_hw_by_id(pdev, &self->aq_hw_ops, &aq_nic_get_cfg(self)->aq_hw_caps); if (err) @@ -268,6 +272,7 @@ static int aq_pci_probe(struct pci_dev *pdev, numvecs = min((u8)AQ_CFG_VECS_DEF, aq_nic_get_cfg(self)->aq_hw_caps->msix_irqs); numvecs = min(numvecs, num_online_cpus()); + numvecs += AQ_HW_SERVICE_IRQS; /*enable interrupts */ #if !AQ_CFG_FORCE_LEGACY_INT err = pci_alloc_irq_vectors(self->pdev, 1, numvecs, @@ -289,6 +294,8 @@ static int aq_pci_probe(struct pci_dev *pdev, if (err < 0) goto err_register; + aq_drvinfo_init(ndev); + return 0; err_register: @@ -365,4 +372,13 @@ static struct pci_driver aq_pci_ops = { .shutdown = aq_pci_shutdown, }; -module_pci_driver(aq_pci_ops); +int aq_pci_func_register_driver(void) +{ + return pci_register_driver(&aq_pci_ops); +} + +void aq_pci_func_unregister_driver(void) +{ + pci_unregister_driver(&aq_pci_ops); +} + diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_pci_func.h b/drivers/net/ethernet/aquantia/atlantic/aq_pci_func.h index aeee67bf69fab40aa685f5dcd40b259010ed7d2f..670f9a940d65f0524b0bbfec80da58fce742da11 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_pci_func.h +++ b/drivers/net/ethernet/aquantia/atlantic/aq_pci_func.h @@ -24,9 +24,12 @@ struct aq_board_revision_s { int aq_pci_func_init(struct pci_dev *pdev); int aq_pci_func_alloc_irq(struct aq_nic_s *self, unsigned int i, - char *name, void *aq_vec, - cpumask_t *affinity_mask); + char *name, irq_handler_t irq_handler, + void *irq_arg, cpumask_t *affinity_mask); void aq_pci_func_free_irqs(struct aq_nic_s *self); unsigned int aq_pci_func_get_irq_type(struct aq_nic_s *self); +int aq_pci_func_register_driver(void); +void aq_pci_func_unregister_driver(void); + #endif /* AQ_PCI_FUNC_H */ diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_ring.c b/drivers/net/ethernet/aquantia/atlantic/aq_ring.c index e2ffb159cbe2eeb5980a89aa688ebde8826fc7e6..350e385528fddd8697c3c1349336c57360e16418 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_ring.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_ring.c @@ -12,10 +12,89 @@ #include "aq_ring.h" #include "aq_nic.h" #include "aq_hw.h" +#include "aq_hw_utils.h" #include #include +static inline void aq_free_rxpage(struct aq_rxpage *rxpage, struct device *dev) +{ + unsigned int len = PAGE_SIZE << rxpage->order; + + dma_unmap_page(dev, rxpage->daddr, len, DMA_FROM_DEVICE); + + /* Drop the ref for being in the ring. */ + __free_pages(rxpage->page, rxpage->order); + rxpage->page = NULL; +} + +static int aq_get_rxpage(struct aq_rxpage *rxpage, unsigned int order, + struct device *dev) +{ + struct page *page; + dma_addr_t daddr; + int ret = -ENOMEM; + + page = dev_alloc_pages(order); + if (unlikely(!page)) + goto err_exit; + + daddr = dma_map_page(dev, page, 0, PAGE_SIZE << order, + DMA_FROM_DEVICE); + + if (unlikely(dma_mapping_error(dev, daddr))) + goto free_page; + + rxpage->page = page; + rxpage->daddr = daddr; + rxpage->order = order; + rxpage->pg_off = 0; + + return 0; + +free_page: + __free_pages(page, order); + +err_exit: + return ret; +} + +static int aq_get_rxpages(struct aq_ring_s *self, struct aq_ring_buff_s *rxbuf, + int order) +{ + int ret; + + if (rxbuf->rxdata.page) { + /* One means ring is the only user and can reuse */ + if (page_ref_count(rxbuf->rxdata.page) > 1) { + /* Try reuse buffer */ + rxbuf->rxdata.pg_off += AQ_CFG_RX_FRAME_MAX; + if (rxbuf->rxdata.pg_off + AQ_CFG_RX_FRAME_MAX <= + (PAGE_SIZE << order)) { + self->stats.rx.pg_flips++; + } else { + /* Buffer exhausted. We have other users and + * should release this page and realloc + */ + aq_free_rxpage(&rxbuf->rxdata, + aq_nic_get_dev(self->aq_nic)); + self->stats.rx.pg_losts++; + } + } else { + rxbuf->rxdata.pg_off = 0; + self->stats.rx.pg_reuses++; + } + } + + if (!rxbuf->rxdata.page) { + ret = aq_get_rxpage(&rxbuf->rxdata, order, + aq_nic_get_dev(self->aq_nic)); + return ret; + } + + return 0; +} + static struct aq_ring_s *aq_ring_alloc(struct aq_ring_s *self, struct aq_nic_s *aq_nic) { @@ -81,6 +160,11 @@ struct aq_ring_s *aq_ring_rx_alloc(struct aq_ring_s *self, self->idx = idx; self->size = aq_nic_cfg->rxds; self->dx_size = aq_nic_cfg->aq_hw_caps->rxd_size; + self->page_order = fls(AQ_CFG_RX_FRAME_MAX / PAGE_SIZE + + (AQ_CFG_RX_FRAME_MAX % PAGE_SIZE ? 1 : 0)) - 1; + + if (aq_nic_cfg->rxpageorder > self->page_order) + self->page_order = aq_nic_cfg->rxpageorder; self = aq_ring_alloc(self, aq_nic); if (!self) { @@ -201,22 +285,21 @@ int aq_ring_rx_clean(struct aq_ring_s *self, int budget) { struct net_device *ndev = aq_nic_get_ndev(self->aq_nic); - int err = 0; bool is_rsc_completed = true; + int err = 0; for (; (self->sw_head != self->hw_head) && budget; self->sw_head = aq_ring_next_dx(self, self->sw_head), --budget, ++(*work_done)) { struct aq_ring_buff_s *buff = &self->buff_ring[self->sw_head]; + struct aq_ring_buff_s *buff_ = NULL; struct sk_buff *skb = NULL; unsigned int next_ = 0U; unsigned int i = 0U; - struct aq_ring_buff_s *buff_ = NULL; + u16 hdr_len; - if (buff->is_error) { - __free_pages(buff->page, 0); + if (buff->is_error) continue; - } if (buff->is_cleaned) continue; @@ -246,45 +329,67 @@ int aq_ring_rx_clean(struct aq_ring_s *self, } } + dma_sync_single_range_for_cpu(aq_nic_get_dev(self->aq_nic), + buff->rxdata.daddr, + buff->rxdata.pg_off, + buff->len, DMA_FROM_DEVICE); + /* for single fragment packets use build_skb() */ if (buff->is_eop && buff->len <= AQ_CFG_RX_FRAME_MAX - AQ_SKB_ALIGN) { - skb = build_skb(page_address(buff->page), + skb = build_skb(aq_buf_vaddr(&buff->rxdata), AQ_CFG_RX_FRAME_MAX); if (unlikely(!skb)) { err = -ENOMEM; goto err_exit; } - skb_put(skb, buff->len); + page_ref_inc(buff->rxdata.page); } else { - skb = netdev_alloc_skb(ndev, ETH_HLEN); + skb = napi_alloc_skb(napi, AQ_CFG_RX_HDR_SIZE); if (unlikely(!skb)) { err = -ENOMEM; goto err_exit; } - skb_put(skb, ETH_HLEN); - memcpy(skb->data, page_address(buff->page), ETH_HLEN); - skb_add_rx_frag(skb, 0, buff->page, ETH_HLEN, - buff->len - ETH_HLEN, - SKB_TRUESIZE(buff->len - ETH_HLEN)); + hdr_len = buff->len; + if (hdr_len > AQ_CFG_RX_HDR_SIZE) + hdr_len = eth_get_headlen(skb->dev, + aq_buf_vaddr(&buff->rxdata), + AQ_CFG_RX_HDR_SIZE); + + memcpy(__skb_put(skb, hdr_len), aq_buf_vaddr(&buff->rxdata), + ALIGN(hdr_len, sizeof(long))); + + if (buff->len - hdr_len > 0) { + skb_add_rx_frag(skb, 0, buff->rxdata.page, + buff->rxdata.pg_off + hdr_len, + buff->len - hdr_len, + AQ_CFG_RX_FRAME_MAX); + page_ref_inc(buff->rxdata.page); + } if (!buff->is_eop) { - for (i = 1U, next_ = buff->next, - buff_ = &self->buff_ring[next_]; - true; next_ = buff_->next, - buff_ = &self->buff_ring[next_], ++i) { - skb_add_rx_frag(skb, i, - buff_->page, 0, + buff_ = buff; + i = 1U; + do { + next_ = buff_->next, + buff_ = &self->buff_ring[next_]; + + dma_sync_single_range_for_cpu( + aq_nic_get_dev(self->aq_nic), + buff_->rxdata.daddr, + buff_->rxdata.pg_off, buff_->len, - SKB_TRUESIZE(buff->len - - ETH_HLEN)); + DMA_FROM_DEVICE); + skb_add_rx_frag(skb, i++, + buff_->rxdata.page, + buff_->rxdata.pg_off, + buff_->len, + AQ_CFG_RX_FRAME_MAX); + page_ref_inc(buff_->rxdata.page); buff_->is_cleaned = 1; - - if (buff_->is_eop) - break; - } + } while (!buff_->is_eop); } } @@ -310,12 +415,15 @@ err_exit: int aq_ring_rx_fill(struct aq_ring_s *self) { - unsigned int pages_order = fls(AQ_CFG_RX_FRAME_MAX / PAGE_SIZE + - (AQ_CFG_RX_FRAME_MAX % PAGE_SIZE ? 1 : 0)) - 1; + unsigned int page_order = self->page_order; struct aq_ring_buff_s *buff = NULL; int err = 0; int i = 0; + if (aq_ring_avail_dx(self) < min_t(unsigned int, AQ_CFG_RX_REFILL_THRES, + self->size / 2)) + return err; + for (i = aq_ring_avail_dx(self); i--; self->sw_tail = aq_ring_next_dx(self, self->sw_tail)) { buff = &self->buff_ring[self->sw_tail]; @@ -323,30 +431,15 @@ int aq_ring_rx_fill(struct aq_ring_s *self) buff->flags = 0U; buff->len = AQ_CFG_RX_FRAME_MAX; - buff->page = alloc_pages(GFP_ATOMIC | __GFP_COMP, pages_order); - if (!buff->page) { - err = -ENOMEM; + err = aq_get_rxpages(self, buff, page_order); + if (err) goto err_exit; - } - - buff->pa = dma_map_page(aq_nic_get_dev(self->aq_nic), - buff->page, 0, - AQ_CFG_RX_FRAME_MAX, DMA_FROM_DEVICE); - - if (dma_mapping_error(aq_nic_get_dev(self->aq_nic), buff->pa)) { - err = -ENOMEM; - goto err_exit; - } + buff->pa = aq_buf_daddr(&buff->rxdata); buff = NULL; } err_exit: - if (err < 0) { - if (buff && buff->page) - __free_pages(buff->page, 0); - } - return err; } @@ -359,10 +452,7 @@ void aq_ring_rx_deinit(struct aq_ring_s *self) self->sw_head = aq_ring_next_dx(self, self->sw_head)) { struct aq_ring_buff_s *buff = &self->buff_ring[self->sw_head]; - dma_unmap_page(aq_nic_get_dev(self->aq_nic), buff->pa, - AQ_CFG_RX_FRAME_MAX, DMA_FROM_DEVICE); - - __free_pages(buff->page, 0); + aq_free_rxpage(&buff->rxdata, aq_nic_get_dev(self->aq_nic)); } err_exit:; diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_ring.h b/drivers/net/ethernet/aquantia/atlantic/aq_ring.h index ac1329f4051d7f3681e18f0e886fe9ec58accc88..cfffc301e746068c2f75822897d43e559de17bee 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_ring.h +++ b/drivers/net/ethernet/aquantia/atlantic/aq_ring.h @@ -17,6 +17,13 @@ struct page; struct aq_nic_cfg_s; +struct aq_rxpage { + struct page *page; + dma_addr_t daddr; + unsigned int order; + unsigned int pg_off; +}; + /* TxC SOP DX EOP * +----------+----------+----------+----------- * 8bytes|len l3,l4 | pa | pa | pa @@ -31,28 +38,21 @@ struct aq_nic_cfg_s; */ struct __packed aq_ring_buff_s { union { + /* RX/TX */ + dma_addr_t pa; /* RX */ struct { u32 rss_hash; u16 next; u8 is_hash_l4; u8 rsvd1; - struct page *page; + struct aq_rxpage rxdata; }; /* EOP */ struct { dma_addr_t pa_eop; struct sk_buff *skb; }; - /* DX */ - struct { - dma_addr_t pa; - }; - /* SOP */ - struct { - dma_addr_t pa_sop; - u32 len_pkt_sop; - }; /* TxC */ struct { u32 mss; @@ -91,6 +91,9 @@ struct aq_ring_stats_rx_s { u64 bytes; u64 lro_packets; u64 jumbo_packets; + u64 pg_losts; + u64 pg_flips; + u64 pg_reuses; }; struct aq_ring_stats_tx_s { @@ -116,6 +119,7 @@ struct aq_ring_s { unsigned int size; /* descriptors number */ unsigned int dx_size; /* TX or RX descriptor size, */ /* stored here for fater math */ + unsigned int page_order; union aq_ring_stats_s stats; dma_addr_t dx_ring_pa; }; @@ -126,6 +130,16 @@ struct aq_ring_param_s { cpumask_t affinity_mask; }; +static inline void *aq_buf_vaddr(struct aq_rxpage *rxpage) +{ + return page_to_virt(rxpage->page) + rxpage->pg_off; +} + +static inline dma_addr_t aq_buf_daddr(struct aq_rxpage *rxpage) +{ + return rxpage->daddr + rxpage->pg_off; +} + static inline unsigned int aq_ring_next_dx(struct aq_ring_s *self, unsigned int dx) { diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_vec.c b/drivers/net/ethernet/aquantia/atlantic/aq_vec.c index d335c334fa561ed2ae1a8dad45fcd9af822ee0a7..a2e4ca1782ae34f0052350b07d85af157addcfa7 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_vec.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_vec.c @@ -353,6 +353,9 @@ void aq_vec_add_stats(struct aq_vec_s *self, stats_rx->errors += rx->errors; stats_rx->jumbo_packets += rx->jumbo_packets; stats_rx->lro_packets += rx->lro_packets; + stats_rx->pg_losts += rx->pg_losts; + stats_rx->pg_flips += rx->pg_flips; + stats_rx->pg_reuses += rx->pg_reuses; stats_tx->packets += tx->packets; stats_tx->bytes += tx->bytes; diff --git a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_a0.c b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_a0.c index f6f8338153a28a52086f2045600c7363b8d8fdc4..9fe507fe2d7f5986545c0c144381781554753bf2 100644 --- a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_a0.c +++ b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_a0.c @@ -350,10 +350,10 @@ err_exit: static int hw_atl_a0_hw_init(struct aq_hw_s *self, u8 *mac_addr) { static u32 aq_hw_atl_igcr_table_[4][2] = { - { 0x20000000U, 0x20000000U }, /* AQ_IRQ_INVALID */ - { 0x20000080U, 0x20000080U }, /* AQ_IRQ_LEGACY */ - { 0x20000021U, 0x20000025U }, /* AQ_IRQ_MSI */ - { 0x20000022U, 0x20000026U } /* AQ_IRQ_MSIX */ + [AQ_HW_IRQ_INVALID] = { 0x20000000U, 0x20000000U }, + [AQ_HW_IRQ_LEGACY] = { 0x20000080U, 0x20000080U }, + [AQ_HW_IRQ_MSI] = { 0x20000021U, 0x20000025U }, + [AQ_HW_IRQ_MSIX] = { 0x20000022U, 0x20000026U }, }; int err = 0; @@ -619,8 +619,6 @@ err_exit: static int hw_atl_a0_hw_ring_rx_receive(struct aq_hw_s *self, struct aq_ring_s *ring) { - struct device *ndev = aq_nic_get_dev(ring->aq_nic); - for (; ring->hw_head != ring->sw_tail; ring->hw_head = aq_ring_next_dx(ring, ring->hw_head)) { struct aq_ring_buff_s *buff = NULL; @@ -687,8 +685,6 @@ static int hw_atl_a0_hw_ring_rx_receive(struct aq_hw_s *self, is_err &= ~0x18U; is_err &= ~0x04U; - dma_unmap_page(ndev, buff->pa, buff->len, DMA_FROM_DEVICE); - if (is_err || rxd_wb->type & 0x1000U) { /* status error or DMA error */ buff->is_error = 1U; diff --git a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_b0.c b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_b0.c index b31dba1b1a55713dc3c1a685bbb77ea6b9b4d9ae..bfcda12d73de5e57082df004ace2208de46f1d03 100644 --- a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_b0.c +++ b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_b0.c @@ -259,7 +259,13 @@ static int hw_atl_b0_hw_offload_set(struct aq_hw_s *self, hw_atl_rpo_lro_time_base_divider_set(self, 0x61AU); hw_atl_rpo_lro_inactive_interval_set(self, 0); - hw_atl_rpo_lro_max_coalescing_interval_set(self, 2); + /* the LRO timebase divider is 5 uS (0x61a), + * which is multiplied by 50(0x32) + * to get a maximum coalescing interval of 250 uS, + * which is the default value + */ + hw_atl_rpo_lro_max_coalescing_interval_set(self, 50); + hw_atl_rpo_lro_qsessions_lim_set(self, 1U); @@ -273,6 +279,10 @@ static int hw_atl_b0_hw_offload_set(struct aq_hw_s *self, hw_atl_rpo_lro_en_set(self, aq_nic_cfg->is_lro ? 0xFFFFFFFFU : 0U); + hw_atl_itr_rsc_en_set(self, + aq_nic_cfg->is_lro ? 0xFFFFFFFFU : 0U); + + hw_atl_itr_rsc_delay_set(self, 1U); } return aq_hw_err_from_flags(self); } @@ -378,10 +388,10 @@ err_exit: static int hw_atl_b0_hw_init(struct aq_hw_s *self, u8 *mac_addr) { static u32 aq_hw_atl_igcr_table_[4][2] = { - { 0x20000000U, 0x20000000U }, /* AQ_IRQ_INVALID */ - { 0x20000080U, 0x20000080U }, /* AQ_IRQ_LEGACY */ - { 0x20000021U, 0x20000025U }, /* AQ_IRQ_MSI */ - { 0x20000022U, 0x20000026U } /* AQ_IRQ_MSIX */ + [AQ_HW_IRQ_INVALID] = { 0x20000000U, 0x20000000U }, + [AQ_HW_IRQ_LEGACY] = { 0x20000080U, 0x20000080U }, + [AQ_HW_IRQ_MSI] = { 0x20000021U, 0x20000025U }, + [AQ_HW_IRQ_MSIX] = { 0x20000022U, 0x20000026U }, }; int err = 0; @@ -433,6 +443,11 @@ static int hw_atl_b0_hw_init(struct aq_hw_s *self, u8 *mac_addr) ((HW_ATL_B0_ERR_INT << 0x18) | (1U << 0x1F)) | ((HW_ATL_B0_ERR_INT << 0x10) | (1U << 0x17)), 0U); + /* Enable link interrupt */ + if (aq_nic_cfg->link_irq_vec) + hw_atl_reg_gen_irq_map_set(self, BIT(7) | + aq_nic_cfg->link_irq_vec, 3U); + hw_atl_b0_hw_offload_set(self, aq_nic_cfg); err_exit: @@ -654,8 +669,6 @@ err_exit: static int hw_atl_b0_hw_ring_rx_receive(struct aq_hw_s *self, struct aq_ring_s *ring) { - struct device *ndev = aq_nic_get_dev(ring->aq_nic); - for (; ring->hw_head != ring->sw_tail; ring->hw_head = aq_ring_next_dx(ring, ring->hw_head)) { struct aq_ring_buff_s *buff = NULL; @@ -697,8 +710,6 @@ static int hw_atl_b0_hw_ring_rx_receive(struct aq_hw_s *self, buff->is_cso_err = 0U; } - dma_unmap_page(ndev, buff->pa, buff->len, DMA_FROM_DEVICE); - if ((rx_stat & BIT(0)) || rxd_wb->type & 0x1000U) { /* MAC error or DMA error */ buff->is_error = 1U; diff --git a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_b0.h b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_b0.h index 2cc8dacfdc271b6aa6abf10b75031dc0e9524eee..b1c0b6850e60295230384bd12141a69964dd5786 100644 --- a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_b0.h +++ b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_b0.h @@ -32,9 +32,6 @@ extern const struct aq_hw_caps_s hw_atl_b0_caps_aqc109; #define hw_atl_b0_caps_aqc111s hw_atl_b0_caps_aqc108 #define hw_atl_b0_caps_aqc112s hw_atl_b0_caps_aqc109 -#define hw_atl_b0_caps_aqc111e hw_atl_b0_caps_aqc108 -#define hw_atl_b0_caps_aqc112e hw_atl_b0_caps_aqc109 - extern const struct aq_hw_ops hw_atl_ops_b0; #define hw_atl_ops_b1 hw_atl_ops_b0 diff --git a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_b0_internal.h b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_b0_internal.h index b318eefd36ae673cb27e668604472b6b736aa0a2..ea98a08d7820e9a7334245c0768c839c27032043 100644 --- a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_b0_internal.h +++ b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_b0_internal.h @@ -78,7 +78,7 @@ #define HW_ATL_B0_TC_MAX 1U #define HW_ATL_B0_RSS_MAX 8U -#define HW_ATL_B0_LRO_RXD_MAX 2U +#define HW_ATL_B0_LRO_RXD_MAX 16U #define HW_ATL_B0_RS_SLIP_ENABLED 0U /* (256k -1(max pay_len) - 54(header)) */ diff --git a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_llh.c b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_llh.c index 0722b8e0196492954ac8346b05181062e55e9af2..eaab25cd08b351afe4739dc30c1f9df6fe81a0ed 100644 --- a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_llh.c +++ b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_llh.c @@ -49,11 +49,6 @@ u32 hw_atl_glb_soft_res_get(struct aq_hw_s *aq_hw) HW_ATL_GLB_SOFT_RES_SHIFT); } -u32 hw_atl_reg_rx_dma_stat_counter7get(struct aq_hw_s *aq_hw) -{ - return aq_hw_read_reg(aq_hw, HW_ATL_RX_DMA_STAT_COUNTER7_ADR); -} - u32 hw_atl_reg_glb_mif_id_get(struct aq_hw_s *aq_hw) { return aq_hw_read_reg(aq_hw, HW_ATL_GLB_MIF_ID_ADR); @@ -65,44 +60,24 @@ u32 hw_atl_rpb_rx_dma_drop_pkt_cnt_get(struct aq_hw_s *aq_hw) return aq_hw_read_reg(aq_hw, HW_ATL_RPB_RX_DMA_DROP_PKT_CNT_ADR); } -u32 hw_atl_stats_rx_dma_good_octet_counterlsw_get(struct aq_hw_s *aq_hw) -{ - return aq_hw_read_reg(aq_hw, HW_ATL_STATS_RX_DMA_GOOD_OCTET_COUNTERLSW); -} - -u32 hw_atl_stats_rx_dma_good_pkt_counterlsw_get(struct aq_hw_s *aq_hw) +u64 hw_atl_stats_rx_dma_good_octet_counter_get(struct aq_hw_s *aq_hw) { - return aq_hw_read_reg(aq_hw, HW_ATL_STATS_RX_DMA_GOOD_PKT_COUNTERLSW); + return aq_hw_read_reg64(aq_hw, HW_ATL_STATS_RX_DMA_GOOD_OCTET_COUNTERLSW); } -u32 hw_atl_stats_tx_dma_good_octet_counterlsw_get(struct aq_hw_s *aq_hw) +u64 hw_atl_stats_rx_dma_good_pkt_counter_get(struct aq_hw_s *aq_hw) { - return aq_hw_read_reg(aq_hw, HW_ATL_STATS_TX_DMA_GOOD_OCTET_COUNTERLSW); + return aq_hw_read_reg64(aq_hw, HW_ATL_STATS_RX_DMA_GOOD_PKT_COUNTERLSW); } -u32 hw_atl_stats_tx_dma_good_pkt_counterlsw_get(struct aq_hw_s *aq_hw) +u64 hw_atl_stats_tx_dma_good_octet_counter_get(struct aq_hw_s *aq_hw) { - return aq_hw_read_reg(aq_hw, HW_ATL_STATS_TX_DMA_GOOD_PKT_COUNTERLSW); + return aq_hw_read_reg64(aq_hw, HW_ATL_STATS_TX_DMA_GOOD_OCTET_COUNTERLSW); } -u32 hw_atl_stats_rx_dma_good_octet_countermsw_get(struct aq_hw_s *aq_hw) +u64 hw_atl_stats_tx_dma_good_pkt_counter_get(struct aq_hw_s *aq_hw) { - return aq_hw_read_reg(aq_hw, HW_ATL_STATS_RX_DMA_GOOD_OCTET_COUNTERMSW); -} - -u32 hw_atl_stats_rx_dma_good_pkt_countermsw_get(struct aq_hw_s *aq_hw) -{ - return aq_hw_read_reg(aq_hw, HW_ATL_STATS_RX_DMA_GOOD_PKT_COUNTERMSW); -} - -u32 hw_atl_stats_tx_dma_good_octet_countermsw_get(struct aq_hw_s *aq_hw) -{ - return aq_hw_read_reg(aq_hw, HW_ATL_STATS_TX_DMA_GOOD_OCTET_COUNTERMSW); -} - -u32 hw_atl_stats_tx_dma_good_pkt_countermsw_get(struct aq_hw_s *aq_hw) -{ - return aq_hw_read_reg(aq_hw, HW_ATL_STATS_TX_DMA_GOOD_PKT_COUNTERMSW); + return aq_hw_read_reg64(aq_hw, HW_ATL_STATS_TX_DMA_GOOD_PKT_COUNTERLSW); } /* interrupt */ @@ -315,6 +290,21 @@ void hw_atl_itr_res_irq_set(struct aq_hw_s *aq_hw, u32 res_irq) HW_ATL_ITR_RES_SHIFT, res_irq); } +/* set RSC interrupt */ +void hw_atl_itr_rsc_en_set(struct aq_hw_s *aq_hw, u32 enable) +{ + aq_hw_write_reg(aq_hw, HW_ATL_ITR_RSC_EN_ADR, enable); +} + +/* set RSC delay */ +void hw_atl_itr_rsc_delay_set(struct aq_hw_s *aq_hw, u32 delay) +{ + aq_hw_write_reg_bit(aq_hw, HW_ATL_ITR_RSC_DELAY_ADR, + HW_ATL_ITR_RSC_DELAY_MSK, + HW_ATL_ITR_RSC_DELAY_SHIFT, + delay); +} + /* rdm */ void hw_atl_rdm_cpu_id_set(struct aq_hw_s *aq_hw, u32 cpuid, u32 dca) { diff --git a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_llh.h b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_llh.h index d46351890b16d992448c31522befa5915d4fefbb..2eb44e1cff7015f84b407d2914fd5150798a67e1 100644 --- a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_llh.h +++ b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_llh.h @@ -40,29 +40,17 @@ u32 hw_atl_glb_soft_res_get(struct aq_hw_s *aq_hw); u32 hw_atl_rpb_rx_dma_drop_pkt_cnt_get(struct aq_hw_s *aq_hw); -/* get rx dma good octet counter lsw */ -u32 hw_atl_stats_rx_dma_good_octet_counterlsw_get(struct aq_hw_s *aq_hw); +/* get rx dma good octet counter */ +u64 hw_atl_stats_rx_dma_good_octet_counter_get(struct aq_hw_s *aq_hw); -/* get rx dma good packet counter lsw */ -u32 hw_atl_stats_rx_dma_good_pkt_counterlsw_get(struct aq_hw_s *aq_hw); +/* get rx dma good packet counter */ +u64 hw_atl_stats_rx_dma_good_pkt_counter_get(struct aq_hw_s *aq_hw); -/* get tx dma good octet counter lsw */ -u32 hw_atl_stats_tx_dma_good_octet_counterlsw_get(struct aq_hw_s *aq_hw); +/* get tx dma good octet counter */ +u64 hw_atl_stats_tx_dma_good_octet_counter_get(struct aq_hw_s *aq_hw); -/* get tx dma good packet counter lsw */ -u32 hw_atl_stats_tx_dma_good_pkt_counterlsw_get(struct aq_hw_s *aq_hw); - -/* get rx dma good octet counter msw */ -u32 hw_atl_stats_rx_dma_good_octet_countermsw_get(struct aq_hw_s *aq_hw); - -/* get rx dma good packet counter msw */ -u32 hw_atl_stats_rx_dma_good_pkt_countermsw_get(struct aq_hw_s *aq_hw); - -/* get tx dma good octet counter msw */ -u32 hw_atl_stats_tx_dma_good_octet_countermsw_get(struct aq_hw_s *aq_hw); - -/* get tx dma good packet counter msw */ -u32 hw_atl_stats_tx_dma_good_pkt_countermsw_get(struct aq_hw_s *aq_hw); +/* get tx dma good packet counter */ +u64 hw_atl_stats_tx_dma_good_pkt_counter_get(struct aq_hw_s *aq_hw); /* get msm rx errors counter register */ u32 hw_atl_reg_mac_msm_rx_errs_cnt_get(struct aq_hw_s *aq_hw); @@ -82,9 +70,6 @@ u32 hw_atl_reg_mac_msm_rx_bcst_octets_counter1get(struct aq_hw_s *aq_hw); /* get msm rx unicast octets counter register 0 */ u32 hw_atl_reg_mac_msm_rx_ucst_octets_counter0get(struct aq_hw_s *aq_hw); -/* get rx dma statistics counter 7 */ -u32 hw_atl_reg_rx_dma_stat_counter7get(struct aq_hw_s *aq_hw); - /* get msm tx errors counter register */ u32 hw_atl_reg_mac_msm_tx_errs_cnt_get(struct aq_hw_s *aq_hw); @@ -152,6 +137,12 @@ u32 hw_atl_itr_res_irq_get(struct aq_hw_s *aq_hw); /* set reset interrupt */ void hw_atl_itr_res_irq_set(struct aq_hw_s *aq_hw, u32 res_irq); +/* set RSC interrupt */ +void hw_atl_itr_rsc_en_set(struct aq_hw_s *aq_hw, u32 enable); + +/* set RSC delay */ +void hw_atl_itr_rsc_delay_set(struct aq_hw_s *aq_hw, u32 delay); + /* rdm */ /* set cpu id */ diff --git a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_llh_internal.h b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_llh_internal.h index fb45bc2d99cfe09528b56f8bb32dad0d8c60ab6f..b64140924a0203ba81a73e9ab98e083a3659dacd 100644 --- a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_llh_internal.h +++ b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_llh_internal.h @@ -58,9 +58,6 @@ /* preprocessor definitions for msm rx unicast octets counter register 0 */ #define HW_ATL_MAC_MSM_RX_UCST_OCTETS_COUNTER0_ADR 0x000001b8u -/* preprocessor definitions for rx dma statistics counter 7 */ -#define HW_ATL_RX_DMA_STAT_COUNTER7_ADR 0x00006818u - /* preprocessor definitions for msm tx unicast frames counter register */ #define HW_ATL_MAC_MSM_TX_UCST_FRM_CNT_ADR 0x00000108u @@ -95,6 +92,19 @@ #define HW_ATL_ITR_RES_MSK 0x80000000 /* lower bit position of bitfield itr_reset */ #define HW_ATL_ITR_RES_SHIFT 31 + +/* register address for bitfield rsc_en */ +#define HW_ATL_ITR_RSC_EN_ADR 0x00002200 + +/* register address for bitfield rsc_delay */ +#define HW_ATL_ITR_RSC_DELAY_ADR 0x00002204 +/* bitmask for bitfield rsc_delay */ +#define HW_ATL_ITR_RSC_DELAY_MSK 0x0000000f +/* width of bitfield rsc_delay */ +#define HW_ATL_ITR_RSC_DELAY_WIDTH 4 +/* lower bit position of bitfield rsc_delay */ +#define HW_ATL_ITR_RSC_DELAY_SHIFT 0 + /* register address for bitfield dca{d}_cpuid[7:0] */ #define HW_ATL_RDM_DCADCPUID_ADR(dca) (0x00006100 + (dca) * 0x4) /* bitmask for bitfield dca{d}_cpuid[7:0] */ diff --git a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_utils.c b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_utils.c index eb4b99d56081c776d2369a32609c9f0afcab1631..1208f7ecdd76b6b974eb668920699969b5c27ebe 100644 --- a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_utils.c +++ b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_utils.c @@ -545,7 +545,7 @@ void hw_atl_utils_mpi_read_stats(struct aq_hw_s *self, pmbox->stats.ubtc = pmbox->stats.uptc * mtu; pmbox->stats.dpc = atomic_read(&self->dpc); } else { - pmbox->stats.dpc = hw_atl_reg_rx_dma_stat_counter7get(self); + pmbox->stats.dpc = hw_atl_rpb_rx_dma_drop_pkt_cnt_get(self); } err_exit:; @@ -763,6 +763,7 @@ static int hw_atl_fw1x_deinit(struct aq_hw_s *self) int hw_atl_utils_update_stats(struct aq_hw_s *self) { struct hw_atl_utils_mbox mbox; + struct aq_stats_s *cs = &self->curr_stats; hw_atl_utils_mpi_read_stats(self, &mbox); @@ -789,10 +790,11 @@ int hw_atl_utils_update_stats(struct aq_hw_s *self) AQ_SDELTA(dpc); } #undef AQ_SDELTA - self->curr_stats.dma_pkt_rc = hw_atl_stats_rx_dma_good_pkt_counterlsw_get(self); - self->curr_stats.dma_pkt_tc = hw_atl_stats_tx_dma_good_pkt_counterlsw_get(self); - self->curr_stats.dma_oct_rc = hw_atl_stats_rx_dma_good_octet_counterlsw_get(self); - self->curr_stats.dma_oct_tc = hw_atl_stats_tx_dma_good_octet_counterlsw_get(self); + + cs->dma_pkt_rc = hw_atl_stats_rx_dma_good_pkt_counter_get(self); + cs->dma_pkt_tc = hw_atl_stats_tx_dma_good_pkt_counter_get(self); + cs->dma_oct_rc = hw_atl_stats_rx_dma_good_octet_counter_get(self); + cs->dma_oct_tc = hw_atl_stats_tx_dma_good_octet_counter_get(self); memcpy(&self->last_stats, &mbox.stats, sizeof(mbox.stats)); @@ -960,6 +962,7 @@ const struct aq_fw_ops aq_fw_1x_ops = { .set_state = hw_atl_utils_mpi_set_state, .update_link_status = hw_atl_utils_mpi_get_link_status, .update_stats = hw_atl_utils_update_stats, + .get_phy_temp = NULL, .set_power = aq_fw1x_set_power, .set_eee_rate = NULL, .get_eee_rate = NULL, diff --git a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_utils_fw2x.c b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_utils_fw2x.c index fe6c5658e016dbcf0470250e04c8345460e988f5..fbc9d6ac841f38c4aa1a5b0a0cc41ed3c5d42a96 100644 --- a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_utils_fw2x.c +++ b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_utils_fw2x.c @@ -38,6 +38,7 @@ #define HW_ATL_FW2X_CTRL_WOL BIT(CTRL_WOL) #define HW_ATL_FW2X_CTRL_LINK_DROP BIT(CTRL_LINK_DROP) #define HW_ATL_FW2X_CTRL_PAUSE BIT(CTRL_PAUSE) +#define HW_ATL_FW2X_CTRL_TEMPERATURE BIT(CTRL_TEMPERATURE) #define HW_ATL_FW2X_CTRL_ASYMMETRIC_PAUSE BIT(CTRL_ASYMMETRIC_PAUSE) #define HW_ATL_FW2X_CTRL_FORCE_RECONNECT BIT(CTRL_FORCE_RECONNECT) @@ -310,6 +311,40 @@ static int aq_fw2x_update_stats(struct aq_hw_s *self) return hw_atl_utils_update_stats(self); } +static int aq_fw2x_get_phy_temp(struct aq_hw_s *self, int *temp) +{ + u32 mpi_opts = aq_hw_read_reg(self, HW_ATL_FW2X_MPI_CONTROL2_ADDR); + u32 temp_val = mpi_opts & HW_ATL_FW2X_CTRL_TEMPERATURE; + u32 phy_temp_offset; + u32 temp_res; + int err = 0; + u32 val; + + phy_temp_offset = self->mbox_addr + + offsetof(struct hw_atl_utils_mbox, info) + + offsetof(struct hw_aq_info, phy_temperature); + /* Toggle statistics bit for FW to 0x36C.18 (CTRL_TEMPERATURE) */ + mpi_opts = mpi_opts ^ HW_ATL_FW2X_CTRL_TEMPERATURE; + aq_hw_write_reg(self, HW_ATL_FW2X_MPI_CONTROL2_ADDR, mpi_opts); + /* Wait FW to report back */ + err = readx_poll_timeout_atomic(aq_fw2x_state2_get, self, val, + temp_val != + (val & HW_ATL_FW2X_CTRL_TEMPERATURE), + 1U, 10000U); + err = hw_atl_utils_fw_downld_dwords(self, phy_temp_offset, + &temp_res, 1); + + if (err) + return err; + + /* Convert PHY temperature from 1/256 degree Celsius + * to 1/1000 degree Celsius. + */ + *temp = temp_res * 1000 / 256; + + return 0; +} + static int aq_fw2x_set_sleep_proxy(struct aq_hw_s *self, u8 *mac) { struct hw_atl_utils_fw_rpc *rpc = NULL; @@ -509,6 +544,7 @@ const struct aq_fw_ops aq_fw_2x_ops = { .set_state = aq_fw2x_set_state, .update_link_status = aq_fw2x_update_link_status, .update_stats = aq_fw2x_update_stats, + .get_phy_temp = aq_fw2x_get_phy_temp, .set_power = aq_fw2x_set_power, .set_eee_rate = aq_fw2x_set_eee_rate, .get_eee_rate = aq_fw2x_get_eee_rate, diff --git a/drivers/net/ethernet/arc/emac_main.c b/drivers/net/ethernet/arc/emac_main.c index ff3d68532f5fbca48c9e19dca103cce3f055b6f6..7f89ad5c336d66d9676dac2f571ed02199ecd2df 100644 --- a/drivers/net/ethernet/arc/emac_main.c +++ b/drivers/net/ethernet/arc/emac_main.c @@ -960,7 +960,7 @@ int arc_emac_probe(struct net_device *ndev, int interface) /* Get MAC address from device tree */ mac_addr = of_get_mac_address(dev->of_node); - if (mac_addr) + if (!IS_ERR(mac_addr)) memcpy(ndev->dev_addr, mac_addr, ETH_ALEN); else eth_hw_addr_random(ndev); diff --git a/drivers/net/ethernet/aurora/nb8800.c b/drivers/net/ethernet/aurora/nb8800.c index 6f56276015a47d1069dd2dbcfd6335d7e74dcbb6..3c4967eecef12cf43b0890f2a66440fd1096c6ac 100644 --- a/drivers/net/ethernet/aurora/nb8800.c +++ b/drivers/net/ethernet/aurora/nb8800.c @@ -404,6 +404,7 @@ static int nb8800_xmit(struct sk_buff *skb, struct net_device *dev) unsigned int dma_len; unsigned int align; unsigned int next; + bool xmit_more; if (atomic_read(&priv->tx_free) <= NB8800_DESC_LOW) { netif_stop_queue(dev); @@ -423,9 +424,10 @@ static int nb8800_xmit(struct sk_buff *skb, struct net_device *dev) return NETDEV_TX_OK; } + xmit_more = netdev_xmit_more(); if (atomic_dec_return(&priv->tx_free) <= NB8800_DESC_LOW) { netif_stop_queue(dev); - skb->xmit_more = 0; + xmit_more = false; } next = priv->tx_next; @@ -450,7 +452,7 @@ static int nb8800_xmit(struct sk_buff *skb, struct net_device *dev) desc->n_addr = priv->tx_bufs[next].dma_desc; desc->config = DESC_BTS(2) | DESC_DS | DESC_EOF | dma_len; - if (!skb->xmit_more) + if (!xmit_more) desc->config |= DESC_EOC; txb->skb = skb; @@ -468,7 +470,7 @@ static int nb8800_xmit(struct sk_buff *skb, struct net_device *dev) priv->tx_next = next; - if (!skb->xmit_more) { + if (!xmit_more) { smp_wmb(); priv->tx_chain->ready = true; priv->tx_chain = NULL; @@ -1461,7 +1463,7 @@ static int nb8800_probe(struct platform_device *pdev) dev->irq = irq; mac = of_get_mac_address(pdev->dev.of_node); - if (mac) + if (!IS_ERR(mac)) ether_addr_copy(dev->dev_addr, mac); if (!is_valid_ether_addr(dev->dev_addr)) diff --git a/drivers/net/ethernet/broadcom/Kconfig b/drivers/net/ethernet/broadcom/Kconfig index 716bfbba59cf0052a3ff462b216f980b929c6c9e..461b2c0b2ed66b018eea343787c3f4fc0499c7c0 100644 --- a/drivers/net/ethernet/broadcom/Kconfig +++ b/drivers/net/ethernet/broadcom/Kconfig @@ -196,6 +196,7 @@ config BNXT depends on PCI select FW_LOADER select LIBCRC32C + select NET_DEVLINK ---help--- This driver supports Broadcom NetXtreme-C/E 10/25/40/50 gigabit Ethernet cards. To compile this driver as a module, choose M here: diff --git a/drivers/net/ethernet/broadcom/bcmsysport.c b/drivers/net/ethernet/broadcom/bcmsysport.c index bc3ac369cbe35571a9f43c7589fff01148fd68a0..c623896e3ccb857ea47649da4da305685ce4929f 100644 --- a/drivers/net/ethernet/broadcom/bcmsysport.c +++ b/drivers/net/ethernet/broadcom/bcmsysport.c @@ -116,15 +116,6 @@ static inline void dma_desc_set_addr(struct bcm_sysport_priv *priv, writel_relaxed(lower_32_bits(addr), d + DESC_ADDR_LO); } -static inline void tdma_port_write_desc_addr(struct bcm_sysport_priv *priv, - struct dma_desc *desc, - unsigned int port) -{ - /* Ports are latched, so write upper address first */ - tdma_writel(priv, desc->addr_status_len, TDMA_WRITE_PORT_HI(port)); - tdma_writel(priv, desc->addr_lo, TDMA_WRITE_PORT_LO(port)); -} - /* Ethtool operations */ static void bcm_sysport_set_rx_csum(struct net_device *dev, netdev_features_t wanted) @@ -1291,11 +1282,10 @@ static netdev_tx_t bcm_sysport_xmit(struct sk_buff *skb, struct bcm_sysport_tx_ring *ring; struct bcm_sysport_cb *cb; struct netdev_queue *txq; - struct dma_desc *desc; + u32 len_status, addr_lo; unsigned int skb_len; unsigned long flags; dma_addr_t mapping; - u32 len_status; u16 queue; int ret; @@ -1338,10 +1328,7 @@ static netdev_tx_t bcm_sysport_xmit(struct sk_buff *skb, dma_unmap_addr_set(cb, dma_addr, mapping); dma_unmap_len_set(cb, dma_len, skb_len); - /* Fetch a descriptor entry from our pool */ - desc = ring->desc_cpu; - - desc->addr_lo = lower_32_bits(mapping); + addr_lo = lower_32_bits(mapping); len_status = upper_32_bits(mapping) & DESC_ADDR_HI_MASK; len_status |= (skb_len << DESC_LEN_SHIFT); len_status |= (DESC_SOP | DESC_EOP | TX_STATUS_APP_CRC) << @@ -1354,16 +1341,9 @@ static netdev_tx_t bcm_sysport_xmit(struct sk_buff *skb, ring->curr_desc = 0; ring->desc_count--; - /* Ensure write completion of the descriptor status/length - * in DRAM before the System Port WRITE_PORT register latches - * the value - */ - wmb(); - desc->addr_status_len = len_status; - wmb(); - - /* Write this descriptor address to the RING write port */ - tdma_port_write_desc_addr(priv, desc, ring->index); + /* Ports are latched, so write upper address first */ + tdma_writel(priv, len_status, TDMA_WRITE_PORT_HI(ring->index)); + tdma_writel(priv, addr_lo, TDMA_WRITE_PORT_LO(ring->index)); /* Check ring space and update SW control flow */ if (ring->desc_count == 0) @@ -1489,28 +1469,14 @@ static int bcm_sysport_init_tx_ring(struct bcm_sysport_priv *priv, unsigned int index) { struct bcm_sysport_tx_ring *ring = &priv->tx_rings[index]; - struct device *kdev = &priv->pdev->dev; size_t size; - void *p; u32 reg; /* Simple descriptors partitioning for now */ size = 256; - /* We just need one DMA descriptor which is DMA-able, since writing to - * the port will allocate a new descriptor in its internal linked-list - */ - p = dma_alloc_coherent(kdev, sizeof(struct dma_desc), &ring->desc_dma, - GFP_KERNEL); - if (!p) { - netif_err(priv, hw, priv->netdev, "DMA alloc failed\n"); - return -ENOMEM; - } - ring->cbs = kcalloc(size, sizeof(struct bcm_sysport_cb), GFP_KERNEL); if (!ring->cbs) { - dma_free_coherent(kdev, sizeof(struct dma_desc), - ring->desc_cpu, ring->desc_dma); netif_err(priv, hw, priv->netdev, "CB allocation failed\n"); return -ENOMEM; } @@ -1523,7 +1489,6 @@ static int bcm_sysport_init_tx_ring(struct bcm_sysport_priv *priv, ring->size = size; ring->clean_index = 0; ring->alloc_size = ring->size; - ring->desc_cpu = p; ring->desc_count = ring->size; ring->curr_desc = 0; @@ -1578,8 +1543,8 @@ static int bcm_sysport_init_tx_ring(struct bcm_sysport_priv *priv, napi_enable(&ring->napi); netif_dbg(priv, hw, priv->netdev, - "TDMA cfg, size=%d, desc_cpu=%p switch q=%d,port=%d\n", - ring->size, ring->desc_cpu, ring->switch_queue, + "TDMA cfg, size=%d, switch q=%d,port=%d\n", + ring->size, ring->switch_queue, ring->switch_port); return 0; @@ -1589,7 +1554,6 @@ static void bcm_sysport_fini_tx_ring(struct bcm_sysport_priv *priv, unsigned int index) { struct bcm_sysport_tx_ring *ring = &priv->tx_rings[index]; - struct device *kdev = &priv->pdev->dev; u32 reg; /* Caller should stop the TDMA engine */ @@ -1611,12 +1575,6 @@ static void bcm_sysport_fini_tx_ring(struct bcm_sysport_priv *priv, kfree(ring->cbs); ring->cbs = NULL; - - if (ring->desc_dma) { - dma_free_coherent(kdev, sizeof(struct dma_desc), - ring->desc_cpu, ring->desc_dma); - ring->desc_dma = 0; - } ring->size = 0; ring->alloc_size = 0; @@ -2274,8 +2232,7 @@ static const struct ethtool_ops bcm_sysport_ethtool_ops = { }; static u16 bcm_sysport_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { struct bcm_sysport_priv *priv = netdev_priv(dev); u16 queue = skb_get_queue_mapping(skb); @@ -2283,7 +2240,7 @@ static u16 bcm_sysport_select_queue(struct net_device *dev, struct sk_buff *skb, unsigned int q, port; if (!netdev_uses_dsa(dev)) - return fallback(dev, skb, NULL); + return netdev_pick_tx(dev, skb, NULL); /* DSA tagging layer will have configured the correct queue */ q = BRCM_TAG_GET_QUEUE(queue); @@ -2291,7 +2248,7 @@ static u16 bcm_sysport_select_queue(struct net_device *dev, struct sk_buff *skb, tx_ring = priv->ring_map[q + port * priv->per_port_num_tx_queues]; if (unlikely(!tx_ring)) - return fallback(dev, skb, NULL); + return netdev_pick_tx(dev, skb, NULL); return tx_ring->index; } @@ -2548,7 +2505,7 @@ static int bcm_sysport_probe(struct platform_device *pdev) /* Initialize netdevice members */ macaddr = of_get_mac_address(dn); - if (!macaddr || !is_valid_ether_addr(macaddr)) { + if (IS_ERR(macaddr)) { dev_warn(&pdev->dev, "using random Ethernet MAC\n"); eth_hw_addr_random(dev); } else { @@ -2599,11 +2556,11 @@ static int bcm_sysport_probe(struct platform_device *pdev) priv->rev = topctrl_readl(priv, REV_CNTL) & REV_MASK; dev_info(&pdev->dev, - "Broadcom SYSTEMPORT%s" REV_FMT - " at 0x%p (irqs: %d, %d, TXQs: %d, RXQs: %d)\n", + "Broadcom SYSTEMPORT%s " REV_FMT + " (irqs: %d, %d, TXQs: %d, RXQs: %d)\n", priv->is_lite ? " Lite" : "", (priv->rev >> 8) & 0xff, priv->rev & 0xff, - priv->base, priv->irq0, priv->irq1, txq, rxq); + priv->irq0, priv->irq1, txq, rxq); return 0; diff --git a/drivers/net/ethernet/broadcom/bcmsysport.h b/drivers/net/ethernet/broadcom/bcmsysport.h index 0b192fea9c5d113542b9b2124d214b89844539aa..6f3141c86436e6731822dfbfe389e00166483a46 100644 --- a/drivers/net/ethernet/broadcom/bcmsysport.h +++ b/drivers/net/ethernet/broadcom/bcmsysport.h @@ -516,12 +516,6 @@ struct bcm_rsb { #define TDMA_DEBUG 0x64c -/* Transmit/Receive descriptor */ -struct dma_desc { - u32 addr_status_len; - u32 addr_lo; -}; - /* Number of Receive hardware descriptor words */ #define SP_NUM_HW_RX_DESC_WORDS 1024 #define SP_LT_NUM_HW_RX_DESC_WORDS 256 @@ -530,7 +524,7 @@ struct dma_desc { #define SP_NUM_TX_DESC 1536 #define SP_LT_NUM_TX_DESC 256 -#define WORDS_PER_DESC (sizeof(struct dma_desc) / sizeof(u32)) +#define WORDS_PER_DESC 2 /* Rx/Tx common counter group.*/ struct bcm_sysport_pkt_counters { @@ -718,7 +712,6 @@ struct bcm_sysport_net_dim { struct bcm_sysport_tx_ring { spinlock_t lock; /* Ring lock for tx reclaim/xmit */ struct napi_struct napi; /* NAPI per tx queue */ - dma_addr_t desc_dma; /* DMA cookie */ unsigned int index; /* Ring index */ unsigned int size; /* Ring current size */ unsigned int alloc_size; /* Ring one-time allocated size */ @@ -727,7 +720,6 @@ struct bcm_sysport_tx_ring { unsigned int c_index; /* Last consumer index */ unsigned int clean_index; /* Current clean index */ struct bcm_sysport_cb *cbs; /* Transmit control blocks */ - struct dma_desc *desc_cpu; /* CPU view of the descriptor */ struct bcm_sysport_priv *priv; /* private context backpointer */ unsigned long packets; /* packets statistics */ unsigned long bytes; /* bytes statistics */ diff --git a/drivers/net/ethernet/broadcom/bgmac-bcma.c b/drivers/net/ethernet/broadcom/bgmac-bcma.c index 6fe074c1588bb7c4568be9ace93cd5f754bab33a..34d18302b1a38e179be8615101f979768a1fd4fb 100644 --- a/drivers/net/ethernet/broadcom/bgmac-bcma.c +++ b/drivers/net/ethernet/broadcom/bgmac-bcma.c @@ -132,7 +132,7 @@ static int bgmac_probe(struct bcma_device *core) mac = of_get_mac_address(bgmac->dev->of_node); /* If no MAC address assigned via device tree, check SPROM */ - if (!mac) { + if (IS_ERR_OR_NULL(mac)) { switch (core->core_unit) { case 0: mac = sprom->et0mac; diff --git a/drivers/net/ethernet/broadcom/bgmac-platform.c b/drivers/net/ethernet/broadcom/bgmac-platform.c index 894eda5b13cf8fd0daf2fec8cf5131679708027b..6dc0dd91ad11e815e1f2fbcb7efbf64c1e36d078 100644 --- a/drivers/net/ethernet/broadcom/bgmac-platform.c +++ b/drivers/net/ethernet/broadcom/bgmac-platform.c @@ -193,7 +193,7 @@ static int bgmac_probe(struct platform_device *pdev) bgmac->dma_dev = &pdev->dev; mac_addr = of_get_mac_address(np); - if (mac_addr) + if (!IS_ERR(mac_addr)) ether_addr_copy(bgmac->net_dev->dev_addr, mac_addr); else dev_warn(&pdev->dev, "MAC address not present in device tree\n"); diff --git a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c index 0c8f5b546c6f42149fa3437b146c976c41f0eae6..008ad0ca89ba0ea7a20085ec4d69b1d010d77c88 100644 --- a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c +++ b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c @@ -1909,8 +1909,7 @@ void bnx2x_netif_stop(struct bnx2x *bp, int disable_hw) } u16 bnx2x_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { struct bnx2x *bp = netdev_priv(dev); @@ -1932,7 +1931,7 @@ u16 bnx2x_select_queue(struct net_device *dev, struct sk_buff *skb, } /* select a non-FCoE queue */ - return fallback(dev, skb, NULL) % + return netdev_pick_tx(dev, skb, NULL) % (BNX2X_NUM_ETH_QUEUES(bp) * bp->max_cos); } diff --git a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.h b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.h index 2d57af9c061cc80ac0d374dc7f2bdb33d9146ad3..c2f6e44e9a3f7e75b41f2542fedb9595fa2ef154 100644 --- a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.h +++ b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.h @@ -498,8 +498,7 @@ int bnx2x_set_vf_spoofchk(struct net_device *dev, int idx, bool val); /* select_queue callback */ u16 bnx2x_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback); + struct net_device *sb_dev); static inline void bnx2x_update_rx_prod(struct bnx2x *bp, struct bnx2x_fastpath *fp, diff --git a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_hsi.h b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_hsi.h index d9057c8bbeef7131cbfad85876d0b66b08cfeddd..78326a6c0aba7c33a13ed06effc5f20b1d2a26b0 100644 --- a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_hsi.h +++ b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_hsi.h @@ -3024,7 +3024,7 @@ struct afex_stats { #define BCM_5710_FW_MAJOR_VERSION 7 #define BCM_5710_FW_MINOR_VERSION 13 -#define BCM_5710_FW_REVISION_VERSION 1 +#define BCM_5710_FW_REVISION_VERSION 11 #define BCM_5710_FW_ENGINEERING_VERSION 0 #define BCM_5710_FW_COMPILE_FLAGS 1 @@ -3639,8 +3639,10 @@ struct client_init_rx_data { #define CLIENT_INIT_RX_DATA_TPA_EN_IPV6_SHIFT 1 #define CLIENT_INIT_RX_DATA_TPA_MODE (0x1<<2) #define CLIENT_INIT_RX_DATA_TPA_MODE_SHIFT 2 -#define CLIENT_INIT_RX_DATA_RESERVED5 (0x1F<<3) -#define CLIENT_INIT_RX_DATA_RESERVED5_SHIFT 3 +#define CLIENT_INIT_RX_DATA_TPA_OVER_VLAN_DISABLE (0x1<<3) +#define CLIENT_INIT_RX_DATA_TPA_OVER_VLAN_DISABLE_SHIFT 3 +#define CLIENT_INIT_RX_DATA_RESERVED5 (0xF<<4) +#define CLIENT_INIT_RX_DATA_RESERVED5_SHIFT 4 u8 vmqueue_mode_en_flg; u8 extra_data_over_sgl_en_flg; u8 cache_line_alignment_log_size; @@ -3831,7 +3833,7 @@ struct eth_classify_cmd_header { */ struct eth_classify_header { u8 rule_cnt; - u8 reserved0; + u8 warning_on_error; __le16 reserved1; __le32 echo; }; @@ -4752,6 +4754,8 @@ struct tpa_update_ramrod_data { __le32 sge_page_base_hi; __le16 sge_pause_thr_low; __le16 sge_pause_thr_high; + u8 tpa_over_vlan_disable; + u8 reserved[7]; }; @@ -4946,7 +4950,7 @@ struct fairness_vars_per_port { u32 upper_bound; u32 fair_threshold; u32 fairness_timeout; - u32 reserved0; + u32 size_thr; }; /* @@ -5415,7 +5419,9 @@ struct function_start_data { u8 sd_vlan_force_pri_val; u8 c2s_pri_tt_valid; u8 c2s_pri_default; - u8 reserved2[6]; + u8 tx_vlan_filtering_enable; + u8 tx_vlan_filtering_use_pvid; + u8 reserved2[4]; struct c2s_pri_trans_table_entry c2s_pri_trans_table; }; @@ -5448,7 +5454,8 @@ struct function_update_data { u8 reserved1; __le16 sd_vlan_tag; __le16 sd_vlan_eth_type; - __le16 reserved0; + u8 tx_vlan_filtering_pvid_change_flg; + u8 reserved0; __le32 reserved2; }; diff --git a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c index 3716c828ff5d9537ff4bb02678b0eeb819dd2853..03ac10b1cd1e1db4684c8af8eed33f950492ab5b 100644 --- a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c +++ b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c @@ -15347,27 +15347,47 @@ static int bnx2x_enable_ptp_packets(struct bnx2x *bp) return 0; } +#define BNX2X_P2P_DETECT_PARAM_MASK 0x5F5 +#define BNX2X_P2P_DETECT_RULE_MASK 0x3DBB +#define BNX2X_PTP_TX_ON_PARAM_MASK (BNX2X_P2P_DETECT_PARAM_MASK & 0x6AA) +#define BNX2X_PTP_TX_ON_RULE_MASK (BNX2X_P2P_DETECT_RULE_MASK & 0x3EEE) +#define BNX2X_PTP_V1_L4_PARAM_MASK (BNX2X_P2P_DETECT_PARAM_MASK & 0x7EE) +#define BNX2X_PTP_V1_L4_RULE_MASK (BNX2X_P2P_DETECT_RULE_MASK & 0x3FFE) +#define BNX2X_PTP_V2_L4_PARAM_MASK (BNX2X_P2P_DETECT_PARAM_MASK & 0x7EA) +#define BNX2X_PTP_V2_L4_RULE_MASK (BNX2X_P2P_DETECT_RULE_MASK & 0x3FEE) +#define BNX2X_PTP_V2_L2_PARAM_MASK (BNX2X_P2P_DETECT_PARAM_MASK & 0x6BF) +#define BNX2X_PTP_V2_L2_RULE_MASK (BNX2X_P2P_DETECT_RULE_MASK & 0x3EFF) +#define BNX2X_PTP_V2_PARAM_MASK (BNX2X_P2P_DETECT_PARAM_MASK & 0x6AA) +#define BNX2X_PTP_V2_RULE_MASK (BNX2X_P2P_DETECT_RULE_MASK & 0x3EEE) + int bnx2x_configure_ptp_filters(struct bnx2x *bp) { int port = BP_PORT(bp); + u32 param, rule; int rc; if (!bp->hwtstamp_ioctl_called) return 0; + param = port ? NIG_REG_P1_TLLH_PTP_PARAM_MASK : + NIG_REG_P0_TLLH_PTP_PARAM_MASK; + rule = port ? NIG_REG_P1_TLLH_PTP_RULE_MASK : + NIG_REG_P0_TLLH_PTP_RULE_MASK; switch (bp->tx_type) { case HWTSTAMP_TX_ON: bp->flags |= TX_TIMESTAMPING_EN; - REG_WR(bp, port ? NIG_REG_P1_TLLH_PTP_PARAM_MASK : - NIG_REG_P0_TLLH_PTP_PARAM_MASK, 0x6AA); - REG_WR(bp, port ? NIG_REG_P1_TLLH_PTP_RULE_MASK : - NIG_REG_P0_TLLH_PTP_RULE_MASK, 0x3EEE); + REG_WR(bp, param, BNX2X_PTP_TX_ON_PARAM_MASK); + REG_WR(bp, rule, BNX2X_PTP_TX_ON_RULE_MASK); break; case HWTSTAMP_TX_ONESTEP_SYNC: BNX2X_ERR("One-step timestamping is not supported\n"); return -ERANGE; } + param = port ? NIG_REG_P1_LLH_PTP_PARAM_MASK : + NIG_REG_P0_LLH_PTP_PARAM_MASK; + rule = port ? NIG_REG_P1_LLH_PTP_RULE_MASK : + NIG_REG_P0_LLH_PTP_RULE_MASK; switch (bp->rx_filter) { case HWTSTAMP_FILTER_NONE: break; @@ -15381,30 +15401,24 @@ int bnx2x_configure_ptp_filters(struct bnx2x *bp) case HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ: bp->rx_filter = HWTSTAMP_FILTER_PTP_V1_L4_EVENT; /* Initialize PTP detection for UDP/IPv4 events */ - REG_WR(bp, port ? NIG_REG_P1_LLH_PTP_PARAM_MASK : - NIG_REG_P0_LLH_PTP_PARAM_MASK, 0x7EE); - REG_WR(bp, port ? NIG_REG_P1_LLH_PTP_RULE_MASK : - NIG_REG_P0_LLH_PTP_RULE_MASK, 0x3FFE); + REG_WR(bp, param, BNX2X_PTP_V1_L4_PARAM_MASK); + REG_WR(bp, rule, BNX2X_PTP_V1_L4_RULE_MASK); break; case HWTSTAMP_FILTER_PTP_V2_L4_EVENT: case HWTSTAMP_FILTER_PTP_V2_L4_SYNC: case HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ: bp->rx_filter = HWTSTAMP_FILTER_PTP_V2_L4_EVENT; /* Initialize PTP detection for UDP/IPv4 or UDP/IPv6 events */ - REG_WR(bp, port ? NIG_REG_P1_LLH_PTP_PARAM_MASK : - NIG_REG_P0_LLH_PTP_PARAM_MASK, 0x7EA); - REG_WR(bp, port ? NIG_REG_P1_LLH_PTP_RULE_MASK : - NIG_REG_P0_LLH_PTP_RULE_MASK, 0x3FEE); + REG_WR(bp, param, BNX2X_PTP_V2_L4_PARAM_MASK); + REG_WR(bp, rule, BNX2X_PTP_V2_L4_RULE_MASK); break; case HWTSTAMP_FILTER_PTP_V2_L2_EVENT: case HWTSTAMP_FILTER_PTP_V2_L2_SYNC: case HWTSTAMP_FILTER_PTP_V2_L2_DELAY_REQ: bp->rx_filter = HWTSTAMP_FILTER_PTP_V2_L2_EVENT; /* Initialize PTP detection L2 events */ - REG_WR(bp, port ? NIG_REG_P1_LLH_PTP_PARAM_MASK : - NIG_REG_P0_LLH_PTP_PARAM_MASK, 0x6BF); - REG_WR(bp, port ? NIG_REG_P1_LLH_PTP_RULE_MASK : - NIG_REG_P0_LLH_PTP_RULE_MASK, 0x3EFF); + REG_WR(bp, param, BNX2X_PTP_V2_L2_PARAM_MASK); + REG_WR(bp, rule, BNX2X_PTP_V2_L2_RULE_MASK); break; case HWTSTAMP_FILTER_PTP_V2_EVENT: @@ -15412,10 +15426,8 @@ int bnx2x_configure_ptp_filters(struct bnx2x *bp) case HWTSTAMP_FILTER_PTP_V2_DELAY_REQ: bp->rx_filter = HWTSTAMP_FILTER_PTP_V2_EVENT; /* Initialize PTP detection L2, UDP/IPv4 or UDP/IPv6 events */ - REG_WR(bp, port ? NIG_REG_P1_LLH_PTP_PARAM_MASK : - NIG_REG_P0_LLH_PTP_PARAM_MASK, 0x6AA); - REG_WR(bp, port ? NIG_REG_P1_LLH_PTP_RULE_MASK : - NIG_REG_P0_LLH_PTP_RULE_MASK, 0x3EEE); + REG_WR(bp, param, BNX2X_PTP_V2_PARAM_MASK); + REG_WR(bp, rule, BNX2X_PTP_V2_RULE_MASK); break; } diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index 2a4341708c0fb04304e79338556df9fdd4a72bc9..8314c00d75376f15959413535d7369f9e551adc0 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -114,6 +114,7 @@ enum board_idx { BCM5745x_NPAR, BCM57508, BCM57504, + BCM57502, BCM58802, BCM58804, BCM58808, @@ -158,6 +159,7 @@ static const struct { [BCM5745x_NPAR] = { "Broadcom BCM5745x NetXtreme-E Ethernet Partition" }, [BCM57508] = { "Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet" }, [BCM57504] = { "Broadcom BCM57504 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet" }, + [BCM57502] = { "Broadcom BCM57502 NetXtreme-E 10Gb/25Gb/50Gb Ethernet" }, [BCM58802] = { "Broadcom BCM58802 NetXtreme-S 10Gb/25Gb/40Gb/50Gb Ethernet" }, [BCM58804] = { "Broadcom BCM58804 NetXtreme-S 10Gb/25Gb/40Gb/50Gb/100Gb Ethernet" }, [BCM58808] = { "Broadcom BCM58808 NetXtreme-S 10Gb/25Gb/40Gb/50Gb/100Gb Ethernet" }, @@ -205,6 +207,7 @@ static const struct pci_device_id bnxt_pci_tbl[] = { { PCI_VDEVICE(BROADCOM, 0x16f1), .driver_data = BCM57452 }, { PCI_VDEVICE(BROADCOM, 0x1750), .driver_data = BCM57508 }, { PCI_VDEVICE(BROADCOM, 0x1751), .driver_data = BCM57504 }, + { PCI_VDEVICE(BROADCOM, 0x1752), .driver_data = BCM57502 }, { PCI_VDEVICE(BROADCOM, 0xd802), .driver_data = BCM58802 }, { PCI_VDEVICE(BROADCOM, 0xd804), .driver_data = BCM58804 }, #ifdef CONFIG_BNXT_SRIOV @@ -216,6 +219,7 @@ static const struct pci_device_id bnxt_pci_tbl[] = { { PCI_VDEVICE(BROADCOM, 0x16dc), .driver_data = NETXTREME_E_VF }, { PCI_VDEVICE(BROADCOM, 0x16e1), .driver_data = NETXTREME_C_VF }, { PCI_VDEVICE(BROADCOM, 0x16e5), .driver_data = NETXTREME_C_VF }, + { PCI_VDEVICE(BROADCOM, 0x1806), .driver_data = NETXTREME_E_P5_VF }, { PCI_VDEVICE(BROADCOM, 0x1807), .driver_data = NETXTREME_E_P5_VF }, { PCI_VDEVICE(BROADCOM, 0xd800), .driver_data = NETXTREME_S_VF }, #endif @@ -551,13 +555,13 @@ normal_tx: prod = NEXT_TX(prod); txr->tx_prod = prod; - if (!skb->xmit_more || netif_xmit_stopped(txq)) + if (!netdev_xmit_more() || netif_xmit_stopped(txq)) bnxt_db_write(bp, &txr->tx_db, prod); tx_done: if (unlikely(bnxt_tx_avail(bp, txr) <= MAX_SKB_FRAGS + 1)) { - if (skb->xmit_more && !tx_buf->is_push) + if (netdev_xmit_more() && !tx_buf->is_push) bnxt_db_write(bp, &txr->tx_db, prod); netif_tx_stop_queue(txq); @@ -897,7 +901,7 @@ static struct sk_buff *bnxt_rx_page_skb(struct bnxt *bp, DMA_ATTR_WEAK_ORDERING); if (unlikely(!payload)) - payload = eth_get_headlen(data_ptr, len); + payload = eth_get_headlen(bp->dev, data_ptr, len); skb = napi_alloc_skb(&rxr->bnapi->napi, payload); if (!skb) { @@ -3393,6 +3397,12 @@ static void bnxt_free_port_stats(struct bnxt *bp) bp->hw_rx_port_stats_ext_map); bp->hw_rx_port_stats_ext = NULL; } + + if (bp->hw_pcie_stats) { + dma_free_coherent(&pdev->dev, sizeof(struct pcie_ctx_hw_stats), + bp->hw_pcie_stats, bp->hw_pcie_stats_map); + bp->hw_pcie_stats = NULL; + } } static void bnxt_free_ring_stats(struct bnxt *bp) @@ -3437,56 +3447,68 @@ static int bnxt_alloc_stats(struct bnxt *bp) cpr->hw_stats_ctx_id = INVALID_STATS_CTX_ID; } - if (BNXT_PF(bp) && bp->chip_num != CHIP_NUM_58700) { - if (bp->hw_rx_port_stats) - goto alloc_ext_stats; + if (BNXT_VF(bp) || bp->chip_num == CHIP_NUM_58700) + return 0; - bp->hw_port_stats_size = sizeof(struct rx_port_stats) + - sizeof(struct tx_port_stats) + 1024; + if (bp->hw_rx_port_stats) + goto alloc_ext_stats; - bp->hw_rx_port_stats = - dma_alloc_coherent(&pdev->dev, bp->hw_port_stats_size, - &bp->hw_rx_port_stats_map, - GFP_KERNEL); - if (!bp->hw_rx_port_stats) - return -ENOMEM; + bp->hw_port_stats_size = sizeof(struct rx_port_stats) + + sizeof(struct tx_port_stats) + 1024; - bp->hw_tx_port_stats = (void *)(bp->hw_rx_port_stats + 1) + - 512; - bp->hw_tx_port_stats_map = bp->hw_rx_port_stats_map + - sizeof(struct rx_port_stats) + 512; - bp->flags |= BNXT_FLAG_PORT_STATS; + bp->hw_rx_port_stats = + dma_alloc_coherent(&pdev->dev, bp->hw_port_stats_size, + &bp->hw_rx_port_stats_map, + GFP_KERNEL); + if (!bp->hw_rx_port_stats) + return -ENOMEM; + + bp->hw_tx_port_stats = (void *)(bp->hw_rx_port_stats + 1) + 512; + bp->hw_tx_port_stats_map = bp->hw_rx_port_stats_map + + sizeof(struct rx_port_stats) + 512; + bp->flags |= BNXT_FLAG_PORT_STATS; alloc_ext_stats: - /* Display extended statistics only if FW supports it */ - if (bp->hwrm_spec_code < 0x10804 || - bp->hwrm_spec_code == 0x10900) + /* Display extended statistics only if FW supports it */ + if (bp->hwrm_spec_code < 0x10804 || bp->hwrm_spec_code == 0x10900) + if (!(bp->fw_cap & BNXT_FW_CAP_EXT_STATS_SUPPORTED)) return 0; - if (bp->hw_rx_port_stats_ext) - goto alloc_tx_ext_stats; + if (bp->hw_rx_port_stats_ext) + goto alloc_tx_ext_stats; - bp->hw_rx_port_stats_ext = - dma_alloc_coherent(&pdev->dev, - sizeof(struct rx_port_stats_ext), - &bp->hw_rx_port_stats_ext_map, - GFP_KERNEL); - if (!bp->hw_rx_port_stats_ext) - return 0; + bp->hw_rx_port_stats_ext = + dma_alloc_coherent(&pdev->dev, sizeof(struct rx_port_stats_ext), + &bp->hw_rx_port_stats_ext_map, GFP_KERNEL); + if (!bp->hw_rx_port_stats_ext) + return 0; alloc_tx_ext_stats: - if (bp->hw_tx_port_stats_ext) - return 0; + if (bp->hw_tx_port_stats_ext) + goto alloc_pcie_stats; - if (bp->hwrm_spec_code >= 0x10902) { - bp->hw_tx_port_stats_ext = - dma_alloc_coherent(&pdev->dev, - sizeof(struct tx_port_stats_ext), - &bp->hw_tx_port_stats_ext_map, - GFP_KERNEL); - } - bp->flags |= BNXT_FLAG_PORT_STATS_EXT; + if (bp->hwrm_spec_code >= 0x10902 || + (bp->fw_cap & BNXT_FW_CAP_EXT_STATS_SUPPORTED)) { + bp->hw_tx_port_stats_ext = + dma_alloc_coherent(&pdev->dev, + sizeof(struct tx_port_stats_ext), + &bp->hw_tx_port_stats_ext_map, + GFP_KERNEL); } + bp->flags |= BNXT_FLAG_PORT_STATS_EXT; + +alloc_pcie_stats: + if (bp->hw_pcie_stats || + !(bp->fw_cap & BNXT_FW_CAP_PCIE_STATS_SUPPORTED)) + return 0; + + bp->hw_pcie_stats = + dma_alloc_coherent(&pdev->dev, sizeof(struct pcie_ctx_hw_stats), + &bp->hw_pcie_stats_map, GFP_KERNEL); + if (!bp->hw_pcie_stats) + return 0; + + bp->flags |= BNXT_FLAG_PCIE_STATS; return 0; } @@ -4205,16 +4227,25 @@ static int bnxt_hwrm_cfa_ntuple_filter_free(struct bnxt *bp, static int bnxt_hwrm_cfa_ntuple_filter_alloc(struct bnxt *bp, struct bnxt_ntuple_filter *fltr) { - struct bnxt_vnic_info *vnic = &bp->vnic_info[fltr->rxq + 1]; struct hwrm_cfa_ntuple_filter_alloc_input req = {0}; struct hwrm_cfa_ntuple_filter_alloc_output *resp; struct flow_keys *keys = &fltr->fkeys; + struct bnxt_vnic_info *vnic; + u32 dst_ena = 0; int rc = 0; bnxt_hwrm_cmd_hdr_init(bp, &req, HWRM_CFA_NTUPLE_FILTER_ALLOC, -1, -1); req.l2_filter_id = bp->vnic_info[0].fw_l2_filter_id[fltr->l2_fltr_idx]; - req.enables = cpu_to_le32(BNXT_NTP_FLTR_FLAGS); + if (bp->fw_cap & BNXT_FW_CAP_CFA_RFS_RING_TBL_IDX) { + dst_ena = CFA_NTUPLE_FILTER_ALLOC_REQ_ENABLES_RFS_RING_TBL_IDX; + req.rfs_ring_tbl_idx = cpu_to_le16(fltr->rxq); + vnic = &bp->vnic_info[0]; + } else { + vnic = &bp->vnic_info[fltr->rxq + 1]; + } + req.dst_id = cpu_to_le16(vnic->fw_vnic_id); + req.enables = cpu_to_le32(BNXT_NTP_FLTR_FLAGS | dst_ena); req.ethertype = htons(ETH_P_IP); memcpy(req.src_macaddr, fltr->src_mac_addr, ETH_ALEN); @@ -4252,7 +4283,6 @@ static int bnxt_hwrm_cfa_ntuple_filter_alloc(struct bnxt *bp, req.dst_port = keys->ports.dst; req.dst_port_mask = cpu_to_be16(0xffff); - req.dst_id = cpu_to_le16(vnic->fw_vnic_id); mutex_lock(&bp->hwrm_cmd_lock); rc = _hwrm_send_message(bp, &req, sizeof(req), HWRM_CMD_TIMEOUT); if (!rc) { @@ -5500,11 +5530,13 @@ static bool bnxt_need_reserve_rings(struct bnxt *bp) stat = bnxt_get_func_stat_ctxs(bp); if (BNXT_NEW_RM(bp) && (hw_resc->resv_rx_rings != rx || hw_resc->resv_cp_rings != cp || - hw_resc->resv_irqs < nq || hw_resc->resv_vnics != vnic || - hw_resc->resv_stat_ctxs != stat || + hw_resc->resv_vnics != vnic || hw_resc->resv_stat_ctxs != stat || (hw_resc->resv_hw_ring_grps != grp && !(bp->flags & BNXT_FLAG_CHIP_P5)))) return true; + if ((bp->flags & BNXT_FLAG_CHIP_P5) && BNXT_PF(bp) && + hw_resc->resv_irqs != nq) + return true; return false; } @@ -6053,6 +6085,8 @@ static int bnxt_hwrm_func_backing_store_qcaps(struct bnxt *bp) ctx->tqm_entries_multiple = 1; ctx->mrav_max_entries = le32_to_cpu(resp->mrav_max_entries); ctx->mrav_entry_size = le16_to_cpu(resp->mrav_entry_size); + ctx->mrav_num_entries_units = + le16_to_cpu(resp->mrav_num_entries_units); ctx->tim_entry_size = le16_to_cpu(resp->tim_entry_size); ctx->tim_max_entries = le32_to_cpu(resp->tim_max_entries); } else { @@ -6099,6 +6133,7 @@ static int bnxt_hwrm_func_backing_store_cfg(struct bnxt *bp, u32 enables) struct bnxt_ctx_pg_info *ctx_pg; __le32 *num_entries; __le64 *pg_dir; + u32 flags = 0; u8 *pg_attr; int i, rc; u32 ena; @@ -6158,6 +6193,9 @@ static int bnxt_hwrm_func_backing_store_cfg(struct bnxt *bp, u32 enables) if (enables & FUNC_BACKING_STORE_CFG_REQ_ENABLES_MRAV) { ctx_pg = &ctx->mrav_mem; req.mrav_num_entries = cpu_to_le32(ctx_pg->entries); + if (ctx->mrav_num_entries_units) + flags |= + FUNC_BACKING_STORE_CFG_REQ_FLAGS_MRAV_RESERVATION_SPLIT; req.mrav_entry_size = cpu_to_le16(ctx->mrav_entry_size); bnxt_hwrm_set_pg_attr(&ctx_pg->ring_mem, &req.mrav_pg_size_mrav_lvl, @@ -6184,6 +6222,7 @@ static int bnxt_hwrm_func_backing_store_cfg(struct bnxt *bp, u32 enables) *num_entries = cpu_to_le32(ctx_pg->entries); bnxt_hwrm_set_pg_attr(&ctx_pg->ring_mem, pg_attr, pg_dir); } + req.flags = cpu_to_le32(flags); rc = hwrm_send_message(bp, &req, sizeof(req), HWRM_CMD_TIMEOUT); if (rc) rc = -EIO; @@ -6322,6 +6361,7 @@ static int bnxt_alloc_ctx_mem(struct bnxt *bp) struct bnxt_ctx_pg_info *ctx_pg; struct bnxt_ctx_mem_info *ctx; u32 mem_size, ena, entries; + u32 num_mr, num_ah; u32 extra_srqs = 0; u32 extra_qps = 0; u8 pg_lvl = 1; @@ -6385,12 +6425,21 @@ static int bnxt_alloc_ctx_mem(struct bnxt *bp) goto skip_rdma; ctx_pg = &ctx->mrav_mem; - ctx_pg->entries = extra_qps * 4; + /* 128K extra is needed to accommodate static AH context + * allocation by f/w. + */ + num_mr = 1024 * 256; + num_ah = 1024 * 128; + ctx_pg->entries = num_mr + num_ah; mem_size = ctx->mrav_entry_size * ctx_pg->entries; rc = bnxt_alloc_ctx_pg_tbls(bp, ctx_pg, mem_size, 2); if (rc) return rc; ena = FUNC_BACKING_STORE_CFG_REQ_ENABLES_MRAV; + if (ctx->mrav_num_entries_units) + ctx_pg->entries = + ((num_mr / ctx->mrav_num_entries_units) << 16) | + (num_ah / ctx->mrav_num_entries_units); ctx_pg = &ctx->tim_mem; ctx_pg->entries = ctx->qp_mem.entries; @@ -6505,6 +6554,10 @@ static int __bnxt_hwrm_func_qcaps(struct bnxt *bp) bp->flags |= BNXT_FLAG_ROCEV1_CAP; if (flags & FUNC_QCAPS_RESP_FLAGS_ROCE_V2_SUPPORTED) bp->flags |= BNXT_FLAG_ROCEV2_CAP; + if (flags & FUNC_QCAPS_RESP_FLAGS_PCIE_STATS_SUPPORTED) + bp->fw_cap |= BNXT_FW_CAP_PCIE_STATS_SUPPORTED; + if (flags & FUNC_QCAPS_RESP_FLAGS_EXT_STATS_SUPPORTED) + bp->fw_cap |= BNXT_FW_CAP_EXT_STATS_SUPPORTED; bp->tx_push_thresh = 0; if (flags & FUNC_QCAPS_RESP_FLAGS_PUSH_MODE_SUPPORTED) @@ -6577,6 +6630,34 @@ static int bnxt_hwrm_func_qcaps(struct bnxt *bp) return 0; } +static int bnxt_hwrm_cfa_adv_flow_mgnt_qcaps(struct bnxt *bp) +{ + struct hwrm_cfa_adv_flow_mgnt_qcaps_input req = {0}; + struct hwrm_cfa_adv_flow_mgnt_qcaps_output *resp; + int rc = 0; + u32 flags; + + if (!(bp->fw_cap & BNXT_FW_CAP_CFA_ADV_FLOW)) + return 0; + + resp = bp->hwrm_cmd_resp_addr; + bnxt_hwrm_cmd_hdr_init(bp, &req, HWRM_CFA_ADV_FLOW_MGNT_QCAPS, -1, -1); + + mutex_lock(&bp->hwrm_cmd_lock); + rc = _hwrm_send_message(bp, &req, sizeof(req), HWRM_CMD_TIMEOUT); + if (rc) + goto hwrm_cfa_adv_qcaps_exit; + + flags = le32_to_cpu(resp->flags); + if (flags & + CFA_ADV_FLOW_MGNT_QCAPS_RESP_FLAGS_RFS_RING_TBL_IDX_SUPPORTED) + bp->fw_cap |= BNXT_FW_CAP_CFA_RFS_RING_TBL_IDX; + +hwrm_cfa_adv_qcaps_exit: + mutex_unlock(&bp->hwrm_cmd_lock); + return rc; +} + static int bnxt_hwrm_func_reset(struct bnxt *bp) { struct hwrm_func_reset_input req = {0}; @@ -6668,6 +6749,15 @@ static int bnxt_hwrm_ver_get(struct bnxt *bp) resp->hwrm_fw_maj_8b, resp->hwrm_fw_min_8b, resp->hwrm_fw_bld_8b, resp->hwrm_fw_rsvd_8b); + if (strlen(resp->active_pkg_name)) { + int fw_ver_len = strlen(bp->fw_ver_str); + + snprintf(bp->fw_ver_str + fw_ver_len, + FW_VER_STR_LEN - fw_ver_len - 1, "/pkg %s", + resp->active_pkg_name); + bp->fw_cap |= BNXT_FW_CAP_PKG_VER; + } + bp->hwrm_cmd_timeout = le16_to_cpu(resp->def_req_timeout); if (!bp->hwrm_cmd_timeout) bp->hwrm_cmd_timeout = DFLT_HWRM_CMD_TIMEOUT; @@ -6700,6 +6790,10 @@ static int bnxt_hwrm_ver_get(struct bnxt *bp) VER_GET_RESP_DEV_CAPS_CFG_TRUSTED_VF_SUPPORTED) bp->fw_cap |= BNXT_FW_CAP_TRUSTED_VF; + if (dev_caps_cfg & + VER_GET_RESP_DEV_CAPS_CFG_CFA_ADV_FLOW_MGNT_SUPPORTED) + bp->fw_cap |= BNXT_FW_CAP_CFA_ADV_FLOW; + hwrm_ver_get_exit: mutex_unlock(&bp->hwrm_cmd_lock); return rc; @@ -6805,6 +6899,19 @@ static int bnxt_hwrm_port_qstats_ext(struct bnxt *bp) return rc; } +static int bnxt_hwrm_pcie_qstats(struct bnxt *bp) +{ + struct hwrm_pcie_qstats_input req = {0}; + + if (!(bp->flags & BNXT_FLAG_PCIE_STATS)) + return 0; + + bnxt_hwrm_cmd_hdr_init(bp, &req, HWRM_PCIE_QSTATS, -1, -1); + req.pcie_stat_size = cpu_to_le16(sizeof(struct pcie_ctx_hw_stats)); + req.pcie_stat_host_addr = cpu_to_le64(bp->hw_pcie_stats_map); + return hwrm_send_message(bp, &req, sizeof(req), HWRM_CMD_TIMEOUT); +} + static void bnxt_hwrm_free_tunnel_ports(struct bnxt *bp) { if (bp->vxlan_port_cnt) { @@ -8652,7 +8759,7 @@ static int bnxt_hwrm_port_phy_read(struct bnxt *bp, u16 phy_addr, u16 reg, req.port_id = cpu_to_le16(bp->pf.port_id); req.phy_addr = phy_addr; req.reg_addr = cpu_to_le16(reg & 0x1f); - if (bp->link_info.support_speeds & BNXT_LINK_SPEED_MSK_10GB) { + if (mdio_phy_id_is_c45(phy_addr)) { req.cl45_mdio = 1; req.phy_addr = mdio_phy_id_prtad(phy_addr); req.dev_addr = mdio_phy_id_devad(phy_addr); @@ -8679,7 +8786,7 @@ static int bnxt_hwrm_port_phy_write(struct bnxt *bp, u16 phy_addr, u16 reg, req.port_id = cpu_to_le16(bp->pf.port_id); req.phy_addr = phy_addr; req.reg_addr = cpu_to_le16(reg & 0x1f); - if (bp->link_info.support_speeds & BNXT_LINK_SPEED_MSK_10GB) { + if (mdio_phy_id_is_c45(phy_addr)) { req.cl45_mdio = 1; req.phy_addr = mdio_phy_id_prtad(phy_addr); req.dev_addr = mdio_phy_id_devad(phy_addr); @@ -8997,8 +9104,11 @@ static bool bnxt_can_reserve_rings(struct bnxt *bp) /* If the chip and firmware supports RFS */ static bool bnxt_rfs_supported(struct bnxt *bp) { - if (bp->flags & BNXT_FLAG_CHIP_P5) + if (bp->flags & BNXT_FLAG_CHIP_P5) { + if (bp->fw_cap & BNXT_FW_CAP_CFA_RFS_RING_TBL_IDX) + return true; return false; + } if (BNXT_PF(bp) && !BNXT_CHIP_TYPE_NITRO_A0(bp)) return true; if (bp->flags & BNXT_FLAG_NEW_RSS_CAP) @@ -9013,7 +9123,7 @@ static bool bnxt_rfs_capable(struct bnxt *bp) int vnics, max_vnics, max_rss_ctxs; if (bp->flags & BNXT_FLAG_CHIP_P5) - return false; + return bnxt_rfs_supported(bp); if (!(bp->flags & BNXT_FLAG_MSIX_CAP) || !bnxt_can_reserve_rings(bp)) return false; @@ -9395,6 +9505,7 @@ static void bnxt_sp_task(struct work_struct *work) if (test_and_clear_bit(BNXT_PERIODIC_STATS_SP_EVENT, &bp->sp_event)) { bnxt_hwrm_port_qstats(bp); bnxt_hwrm_port_qstats_ext(bp); + bnxt_hwrm_pcie_qstats(bp); } if (test_and_clear_bit(BNXT_LINK_CHNG_SP_EVENT, &bp->sp_event)) { @@ -10065,23 +10176,6 @@ static int bnxt_bridge_setlink(struct net_device *dev, struct nlmsghdr *nlh, return rc; } -static int bnxt_get_phys_port_name(struct net_device *dev, char *buf, - size_t len) -{ - struct bnxt *bp = netdev_priv(dev); - int rc; - - /* The PF and it's VF-reps only support the switchdev framework */ - if (!BNXT_PF(bp)) - return -EOPNOTSUPP; - - rc = snprintf(buf, len, "p%d", bp->pf.port_id); - - if (rc >= len) - return -EOPNOTSUPP; - return 0; -} - int bnxt_get_port_parent_id(struct net_device *dev, struct netdev_phys_item_id *ppid) { @@ -10100,6 +10194,13 @@ int bnxt_get_port_parent_id(struct net_device *dev, return 0; } +static struct devlink_port *bnxt_get_devlink_port(struct net_device *dev) +{ + struct bnxt *bp = netdev_priv(dev); + + return &bp->dl_port; +} + static const struct net_device_ops bnxt_netdev_ops = { .ndo_open = bnxt_open, .ndo_start_xmit = bnxt_start_xmit, @@ -10131,8 +10232,7 @@ static const struct net_device_ops bnxt_netdev_ops = { .ndo_bpf = bnxt_xdp, .ndo_bridge_getlink = bnxt_bridge_getlink, .ndo_bridge_setlink = bnxt_bridge_setlink, - .ndo_get_port_parent_id = bnxt_get_port_parent_id, - .ndo_get_phys_port_name = bnxt_get_phys_port_name + .ndo_get_devlink_port = bnxt_get_devlink_port, }; static void bnxt_remove_one(struct pci_dev *pdev) @@ -10456,6 +10556,26 @@ static int bnxt_init_mac_addr(struct bnxt *bp) return rc; } +static int bnxt_pcie_dsn_get(struct bnxt *bp, u8 dsn[]) +{ + struct pci_dev *pdev = bp->pdev; + int pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_DSN); + u32 dw; + + if (!pos) { + netdev_info(bp->dev, "Unable do read adapter's DSN"); + return -EOPNOTSUPP; + } + + /* DSN (two dw) is at an offset of 4 from the cap pos */ + pos += 4; + pci_read_config_dword(pdev, pos, &dw); + put_unaligned_le32(dw, &dsn[0]); + pci_read_config_dword(pdev, pos + 4, &dw); + put_unaligned_le32(dw, &dsn[4]); + return 0; +} + static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) { static int version_printed; @@ -10589,6 +10709,12 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) rc = -1; goto init_err_pci_clean; } + + rc = bnxt_hwrm_cfa_adv_flow_mgnt_qcaps(bp); + if (rc) + netdev_warn(bp->dev, "hwrm query adv flow mgnt failure rc: %d\n", + rc); + rc = bnxt_init_mac_addr(bp); if (rc) { dev_err(&pdev->dev, "Unable to initialize mac address.\n"); @@ -10596,6 +10722,11 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) goto init_err_pci_clean; } + /* Read the adapter's DSN to use as the eswitch switch_id */ + rc = bnxt_pcie_dsn_get(bp, bp->switch_id); + if (rc) + goto init_err_pci_clean; + bnxt_hwrm_func_qcfg(bp); bnxt_hwrm_vnic_qcaps(bp); bnxt_hwrm_port_led_qcaps(bp); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.h b/drivers/net/ethernet/broadcom/bnxt/bnxt.h index cf81ace7a6e624db540a28e9f981f102074a8f29..eca36dd6b751ee5c37569d2f5d2229a435914ad9 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.h @@ -1227,6 +1227,7 @@ struct bnxt_ctx_mem_info { u16 mrav_entry_size; u16 tim_entry_size; u32 tim_max_entries; + u16 mrav_num_entries_units; u8 tqm_entries_multiple; u32 flags; @@ -1354,6 +1355,7 @@ struct bnxt { #define BNXT_FLAG_DIM 0x2000000 #define BNXT_FLAG_ROCE_MIRROR_CAP 0x4000000 #define BNXT_FLAG_PORT_STATS_EXT 0x10000000 + #define BNXT_FLAG_PCIE_STATS 0x40000000 #define BNXT_FLAG_ALL_CONFIG_FEATS (BNXT_FLAG_TPA | \ BNXT_FLAG_RFS | \ @@ -1480,6 +1482,11 @@ struct bnxt { #define BNXT_FW_CAP_KONG_MB_CHNL 0x00000080 #define BNXT_FW_CAP_OVS_64BIT_HANDLE 0x00000400 #define BNXT_FW_CAP_TRUSTED_VF 0x00000800 + #define BNXT_FW_CAP_PKG_VER 0x00004000 + #define BNXT_FW_CAP_CFA_ADV_FLOW 0x00008000 + #define BNXT_FW_CAP_CFA_RFS_RING_TBL_IDX 0x00010000 + #define BNXT_FW_CAP_PCIE_STATS_SUPPORTED 0x00020000 + #define BNXT_FW_CAP_EXT_STATS_SUPPORTED 0x00040000 #define BNXT_NEW_RM(bp) ((bp)->fw_cap & BNXT_FW_CAP_NEW_RM) u32 hwrm_spec_code; @@ -1498,10 +1505,12 @@ struct bnxt { struct tx_port_stats *hw_tx_port_stats; struct rx_port_stats_ext *hw_rx_port_stats_ext; struct tx_port_stats_ext *hw_tx_port_stats_ext; + struct pcie_ctx_hw_stats *hw_pcie_stats; dma_addr_t hw_rx_port_stats_map; dma_addr_t hw_tx_port_stats_map; dma_addr_t hw_rx_port_stats_ext_map; dma_addr_t hw_tx_port_stats_ext_map; + dma_addr_t hw_pcie_stats_map; int hw_port_stats_size; u16 fw_rx_stats_ext_size; u16 fw_tx_stats_ext_size; @@ -1634,6 +1643,9 @@ struct bnxt { #define BNXT_TX_STATS_EXT_OFFSET(counter) \ (offsetof(struct tx_port_stats_ext, counter) / 8) +#define BNXT_PCIE_STATS_OFFSET(counter) \ + (offsetof(struct pcie_ctx_hw_stats, counter) / 8) + #define I2C_DEV_ADDR_A0 0xa0 #define I2C_DEV_ADDR_A2 0xa2 #define SFF_DIAG_SUPPORT_OFFSET 0x5c diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c index e1feb97bcd81a0e8cb24e3e14048c5e0f04e207c..549c90d3e465f2fd4f7eb494cee47373b953aaf5 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c @@ -9,6 +9,7 @@ #include #include +#include #include "bnxt_hsi.h" #include "bnxt.h" #include "bnxt_vfr.h" @@ -228,6 +229,9 @@ int bnxt_dl_register(struct bnxt *bp) goto err_dl_unreg; } + devlink_port_attrs_set(&bp->dl_port, DEVLINK_PORT_FLAVOUR_PHYSICAL, + bp->pf.port_id, false, 0, + bp->switch_id, sizeof(bp->switch_id)); rc = devlink_port_register(dl, &bp->dl_port, bp->pf.port_id); if (rc) { netdev_err(bp->dev, "devlink_port_register failed"); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c index adabbe94a259f3e22c39140ffb165bfbcc9cba8b..b1263821a6e9d757a08a4475f0092b0d79f4d775 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c @@ -235,6 +235,9 @@ reset_coalesce: BNXT_TX_STATS_PRI_ENTRY(counter, 6), \ BNXT_TX_STATS_PRI_ENTRY(counter, 7) +#define BNXT_PCIE_STATS_ENTRY(counter) \ + { BNXT_PCIE_STATS_OFFSET(counter), __stringify(counter) } + enum { RX_TOTAL_DISCARDS, TX_TOTAL_DISCARDS, @@ -345,6 +348,10 @@ static const struct { BNXT_RX_STATS_EXT_ENTRY(resume_roce_pause_events), BNXT_RX_STATS_EXT_COS_ENTRIES, BNXT_RX_STATS_EXT_PFC_ENTRIES, + BNXT_RX_STATS_EXT_ENTRY(rx_bits), + BNXT_RX_STATS_EXT_ENTRY(rx_buffer_passed_threshold), + BNXT_RX_STATS_EXT_ENTRY(rx_pcs_symbol_err), + BNXT_RX_STATS_EXT_ENTRY(rx_corrected_bits), }; static const struct { @@ -383,6 +390,24 @@ static const struct { BNXT_TX_STATS_PRI_ENTRIES(tx_packets), }; +static const struct { + long offset; + char string[ETH_GSTRING_LEN]; +} bnxt_pcie_stats_arr[] = { + BNXT_PCIE_STATS_ENTRY(pcie_pl_signal_integrity), + BNXT_PCIE_STATS_ENTRY(pcie_dl_signal_integrity), + BNXT_PCIE_STATS_ENTRY(pcie_tl_signal_integrity), + BNXT_PCIE_STATS_ENTRY(pcie_link_integrity), + BNXT_PCIE_STATS_ENTRY(pcie_tx_traffic_rate), + BNXT_PCIE_STATS_ENTRY(pcie_rx_traffic_rate), + BNXT_PCIE_STATS_ENTRY(pcie_tx_dllp_statistics), + BNXT_PCIE_STATS_ENTRY(pcie_rx_dllp_statistics), + BNXT_PCIE_STATS_ENTRY(pcie_equalization_time), + BNXT_PCIE_STATS_ENTRY(pcie_ltssm_histogram[0]), + BNXT_PCIE_STATS_ENTRY(pcie_ltssm_histogram[2]), + BNXT_PCIE_STATS_ENTRY(pcie_recovery_histogram), +}; + #define BNXT_NUM_SW_FUNC_STATS ARRAY_SIZE(bnxt_sw_func_stats) #define BNXT_NUM_PORT_STATS ARRAY_SIZE(bnxt_port_stats_arr) #define BNXT_NUM_STATS_PRI \ @@ -390,6 +415,7 @@ static const struct { ARRAY_SIZE(bnxt_rx_pkts_pri_arr) + \ ARRAY_SIZE(bnxt_tx_bytes_pri_arr) + \ ARRAY_SIZE(bnxt_tx_pkts_pri_arr)) +#define BNXT_NUM_PCIE_STATS ARRAY_SIZE(bnxt_pcie_stats_arr) static int bnxt_get_num_stats(struct bnxt *bp) { @@ -407,6 +433,9 @@ static int bnxt_get_num_stats(struct bnxt *bp) num_stats += BNXT_NUM_STATS_PRI; } + if (bp->flags & BNXT_FLAG_PCIE_STATS) + num_stats += BNXT_NUM_PCIE_STATS; + return num_stats; } @@ -509,6 +538,14 @@ skip_ring_stats: } } } + if (bp->flags & BNXT_FLAG_PCIE_STATS) { + __le64 *pcie_stats = (__le64 *)bp->hw_pcie_stats; + + for (i = 0; i < BNXT_NUM_PCIE_STATS; i++, j++) { + buf[j] = le64_to_cpu(*(pcie_stats + + bnxt_pcie_stats_arr[i].offset)); + } + } } static void bnxt_get_strings(struct net_device *dev, u32 stringset, u8 *buf) @@ -609,6 +646,12 @@ static void bnxt_get_strings(struct net_device *dev, u32 stringset, u8 *buf) } } } + if (bp->flags & BNXT_FLAG_PCIE_STATS) { + for (i = 0; i < BNXT_NUM_PCIE_STATS; i++) { + strcpy(buf, bnxt_pcie_stats_arr[i].string); + buf += ETH_GSTRING_LEN; + } + } break; case ETH_SS_TEST: if (bp->num_tests) @@ -3262,7 +3305,8 @@ void bnxt_ethtool_init(struct bnxt *bp) struct net_device *dev = bp->dev; int i, rc; - bnxt_get_pkgver(dev); + if (!(bp->fw_cap & BNXT_FW_CAP_PKG_VER)) + bnxt_get_pkgver(dev); if (bp->hwrm_spec_code < 0x10704 || !BNXT_SINGLE_PF(bp)) return; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h index b6c610339501fc9a03483250441d07dd7ec6a87b..12bbb2a207d019471c9433cd5f92db3c82407b05 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h @@ -89,7 +89,10 @@ struct hwrm_short_input { __le16 signature; #define SHORT_REQ_SIGNATURE_SHORT_CMD 0x4321UL #define SHORT_REQ_SIGNATURE_LAST SHORT_REQ_SIGNATURE_SHORT_CMD - __le16 unused_0; + __le16 target_id; + #define SHORT_REQ_TARGET_ID_DEFAULT 0x0UL + #define SHORT_REQ_TARGET_ID_TOOLS 0xfffdUL + #define SHORT_REQ_TARGET_ID_LAST SHORT_REQ_TARGET_ID_TOOLS __le16 size; __le64 req_addr; }; @@ -211,6 +214,7 @@ struct cmd_nums { #define HWRM_FWD_RESP 0xd2UL #define HWRM_FWD_ASYNC_EVENT_CMPL 0xd3UL #define HWRM_OEM_CMD 0xd4UL + #define HWRM_PORT_PRBS_TEST 0xd5UL #define HWRM_TEMP_MONITOR_QUERY 0xe0UL #define HWRM_WOL_FILTER_ALLOC 0xf0UL #define HWRM_WOL_FILTER_FREE 0xf1UL @@ -262,6 +266,7 @@ struct cmd_nums { #define HWRM_CFA_EEM_QCFG 0x122UL #define HWRM_CFA_EEM_OP 0x123UL #define HWRM_CFA_ADV_FLOW_MGNT_QCAPS 0x124UL + #define HWRM_CFA_TFLIB 0x125UL #define HWRM_ENGINE_CKV_HELLO 0x12dUL #define HWRM_ENGINE_CKV_STATUS 0x12eUL #define HWRM_ENGINE_CKV_CKEK_ADD 0x12fUL @@ -272,6 +277,7 @@ struct cmd_nums { #define HWRM_ENGINE_CKV_RNG_GET 0x134UL #define HWRM_ENGINE_CKV_KEY_GEN 0x135UL #define HWRM_ENGINE_CKV_KEY_LABEL_CFG 0x136UL + #define HWRM_ENGINE_CKV_KEY_LABEL_QCFG 0x137UL #define HWRM_ENGINE_QG_CONFIG_QUERY 0x13cUL #define HWRM_ENGINE_QG_QUERY 0x13dUL #define HWRM_ENGINE_QG_METER_PROFILE_CONFIG_QUERY 0x13eUL @@ -312,6 +318,11 @@ struct cmd_nums { #define HWRM_SELFTEST_IRQ 0x202UL #define HWRM_SELFTEST_RETRIEVE_SERDES_DATA 0x203UL #define HWRM_PCIE_QSTATS 0x204UL + #define HWRM_MFG_FRU_WRITE_CONTROL 0x205UL + #define HWRM_MFG_TIMERS_QUERY 0x206UL + #define HWRM_MFG_OTP_CFG 0x207UL + #define HWRM_MFG_OTP_QCFG 0x208UL + #define HWRM_MFG_HDMA_TEST 0x209UL #define HWRM_DBG_READ_DIRECT 0xff10UL #define HWRM_DBG_READ_INDIRECT 0xff11UL #define HWRM_DBG_WRITE_DIRECT 0xff12UL @@ -325,6 +336,8 @@ struct cmd_nums { #define HWRM_DBG_FW_CLI 0xff1aUL #define HWRM_DBG_I2C_CMD 0xff1bUL #define HWRM_DBG_RING_INFO_GET 0xff1cUL + #define HWRM_DBG_CRASHDUMP_HEADER 0xff1dUL + #define HWRM_DBG_CRASHDUMP_ERASE 0xff1eUL #define HWRM_NVM_FACTORY_DEFAULTS 0xffeeUL #define HWRM_NVM_VALIDATE_OPTION 0xffefUL #define HWRM_NVM_FLUSH 0xfff0UL @@ -350,23 +363,26 @@ struct cmd_nums { /* ret_codes (size:64b/8B) */ struct ret_codes { __le16 error_code; - #define HWRM_ERR_CODE_SUCCESS 0x0UL - #define HWRM_ERR_CODE_FAIL 0x1UL - #define HWRM_ERR_CODE_INVALID_PARAMS 0x2UL - #define HWRM_ERR_CODE_RESOURCE_ACCESS_DENIED 0x3UL - #define HWRM_ERR_CODE_RESOURCE_ALLOC_ERROR 0x4UL - #define HWRM_ERR_CODE_INVALID_FLAGS 0x5UL - #define HWRM_ERR_CODE_INVALID_ENABLES 0x6UL - #define HWRM_ERR_CODE_UNSUPPORTED_TLV 0x7UL - #define HWRM_ERR_CODE_NO_BUFFER 0x8UL - #define HWRM_ERR_CODE_UNSUPPORTED_OPTION_ERR 0x9UL - #define HWRM_ERR_CODE_HOT_RESET_PROGRESS 0xaUL - #define HWRM_ERR_CODE_HOT_RESET_FAIL 0xbUL - #define HWRM_ERR_CODE_HWRM_ERROR 0xfUL - #define HWRM_ERR_CODE_TLV_ENCAPSULATED_RESPONSE 0x8000UL - #define HWRM_ERR_CODE_UNKNOWN_ERR 0xfffeUL - #define HWRM_ERR_CODE_CMD_NOT_SUPPORTED 0xffffUL - #define HWRM_ERR_CODE_LAST HWRM_ERR_CODE_CMD_NOT_SUPPORTED + #define HWRM_ERR_CODE_SUCCESS 0x0UL + #define HWRM_ERR_CODE_FAIL 0x1UL + #define HWRM_ERR_CODE_INVALID_PARAMS 0x2UL + #define HWRM_ERR_CODE_RESOURCE_ACCESS_DENIED 0x3UL + #define HWRM_ERR_CODE_RESOURCE_ALLOC_ERROR 0x4UL + #define HWRM_ERR_CODE_INVALID_FLAGS 0x5UL + #define HWRM_ERR_CODE_INVALID_ENABLES 0x6UL + #define HWRM_ERR_CODE_UNSUPPORTED_TLV 0x7UL + #define HWRM_ERR_CODE_NO_BUFFER 0x8UL + #define HWRM_ERR_CODE_UNSUPPORTED_OPTION_ERR 0x9UL + #define HWRM_ERR_CODE_HOT_RESET_PROGRESS 0xaUL + #define HWRM_ERR_CODE_HOT_RESET_FAIL 0xbUL + #define HWRM_ERR_CODE_NO_FLOW_COUNTER_DURING_ALLOC 0xcUL + #define HWRM_ERR_CODE_KEY_HASH_COLLISION 0xdUL + #define HWRM_ERR_CODE_KEY_ALREADY_EXISTS 0xeUL + #define HWRM_ERR_CODE_HWRM_ERROR 0xfUL + #define HWRM_ERR_CODE_TLV_ENCAPSULATED_RESPONSE 0x8000UL + #define HWRM_ERR_CODE_UNKNOWN_ERR 0xfffeUL + #define HWRM_ERR_CODE_CMD_NOT_SUPPORTED 0xffffUL + #define HWRM_ERR_CODE_LAST HWRM_ERR_CODE_CMD_NOT_SUPPORTED __le16 unused_0[3]; }; @@ -387,11 +403,15 @@ struct hwrm_err_output { #define HW_HASH_INDEX_SIZE 0x80 #define HW_HASH_KEY_SIZE 40 #define HWRM_RESP_VALID_KEY 1 +#define HWRM_TARGET_ID_BONO 0xFFF8 +#define HWRM_TARGET_ID_KONG 0xFFF9 +#define HWRM_TARGET_ID_APE 0xFFFA +#define HWRM_TARGET_ID_TOOLS 0xFFFD #define HWRM_VERSION_MAJOR 1 #define HWRM_VERSION_MINOR 10 #define HWRM_VERSION_UPDATE 0 -#define HWRM_VERSION_RSVD 47 -#define HWRM_VERSION_STR "1.10.0.47" +#define HWRM_VERSION_RSVD 69 +#define HWRM_VERSION_STR "1.10.0.69" /* hwrm_ver_get_input (size:192b/24B) */ struct hwrm_ver_get_input { @@ -442,6 +462,7 @@ struct hwrm_ver_get_output { #define VER_GET_RESP_DEV_CAPS_CFG_ADV_FLOW_COUNTERS_SUPPORTED 0x400UL #define VER_GET_RESP_DEV_CAPS_CFG_CFA_EEM_SUPPORTED 0x800UL #define VER_GET_RESP_DEV_CAPS_CFG_CFA_ADV_FLOW_MGNT_SUPPORTED 0x1000UL + #define VER_GET_RESP_DEV_CAPS_CFG_CFA_TFLIB_SUPPORTED 0x2000UL u8 roce_fw_maj_8b; u8 roce_fw_min_8b; u8 roce_fw_bld_8b; @@ -449,7 +470,7 @@ struct hwrm_ver_get_output { char hwrm_fw_name[16]; char mgmt_fw_name[16]; char netctrl_fw_name[16]; - u8 reserved2[16]; + char active_pkg_name[16]; char roce_fw_name[16]; __le16 chip_num; u8 chip_rev; @@ -1047,6 +1068,7 @@ struct hwrm_func_qcaps_output { #define FUNC_QCAPS_RESP_FLAGS_DYNAMIC_TX_RING_ALLOC 0x200000UL #define FUNC_QCAPS_RESP_FLAGS_HOT_RESET_CAPABLE 0x400000UL #define FUNC_QCAPS_RESP_FLAGS_ERROR_RECOVERY_CAPABLE 0x800000UL + #define FUNC_QCAPS_RESP_FLAGS_EXT_STATS_SUPPORTED 0x1000000UL u8 mac_address[6]; __le16 max_rsscos_ctx; __le16 max_cmpl_rings; @@ -1715,7 +1737,7 @@ struct hwrm_func_backing_store_qcaps_output { __le16 mrav_entry_size; __le16 tim_entry_size; __le32 tim_max_entries; - u8 unused_0[2]; + __le16 mrav_num_entries_units; u8 tqm_entries_multiple; u8 valid; }; @@ -1728,7 +1750,8 @@ struct hwrm_func_backing_store_cfg_input { __le16 target_id; __le64 resp_addr; __le32 flags; - #define FUNC_BACKING_STORE_CFG_REQ_FLAGS_PREBOOT_MODE 0x1UL + #define FUNC_BACKING_STORE_CFG_REQ_FLAGS_PREBOOT_MODE 0x1UL + #define FUNC_BACKING_STORE_CFG_REQ_FLAGS_MRAV_RESERVATION_SPLIT 0x2UL __le32 enables; #define FUNC_BACKING_STORE_CFG_REQ_ENABLES_QP 0x1UL #define FUNC_BACKING_STORE_CFG_REQ_ENABLES_SRQ 0x2UL @@ -2580,7 +2603,7 @@ struct hwrm_port_phy_qcfg_output { u8 valid; }; -/* hwrm_port_mac_cfg_input (size:320b/40B) */ +/* hwrm_port_mac_cfg_input (size:384b/48B) */ struct hwrm_port_mac_cfg_input { __le16 req_type; __le16 cmpl_ring; @@ -2601,6 +2624,7 @@ struct hwrm_port_mac_cfg_input { #define PORT_MAC_CFG_REQ_FLAGS_VLAN_PRI2COS_DISABLE 0x400UL #define PORT_MAC_CFG_REQ_FLAGS_TUNNEL_PRI2COS_DISABLE 0x800UL #define PORT_MAC_CFG_REQ_FLAGS_IP_DSCP2COS_DISABLE 0x1000UL + #define PORT_MAC_CFG_REQ_FLAGS_PTP_ONE_STEP_TX_TS 0x2000UL __le32 enables; #define PORT_MAC_CFG_REQ_ENABLES_IPG 0x1UL #define PORT_MAC_CFG_REQ_ENABLES_LPBK 0x2UL @@ -2610,6 +2634,7 @@ struct hwrm_port_mac_cfg_input { #define PORT_MAC_CFG_REQ_ENABLES_RX_TS_CAPTURE_PTP_MSG_TYPE 0x40UL #define PORT_MAC_CFG_REQ_ENABLES_TX_TS_CAPTURE_PTP_MSG_TYPE 0x80UL #define PORT_MAC_CFG_REQ_ENABLES_COS_FIELD_CFG 0x100UL + #define PORT_MAC_CFG_REQ_ENABLES_PTP_FREQ_ADJ_PPB 0x200UL __le16 port_id; u8 ipg; u8 lpbk; @@ -2642,6 +2667,8 @@ struct hwrm_port_mac_cfg_input { #define PORT_MAC_CFG_REQ_COS_FIELD_CFG_DEFAULT_COS_MASK 0xe0UL #define PORT_MAC_CFG_REQ_COS_FIELD_CFG_DEFAULT_COS_SFT 5 u8 unused_0[3]; + __s32 ptp_freq_adj_ppb; + u8 unused_1[4]; }; /* hwrm_port_mac_cfg_output (size:128b/16B) */ @@ -2680,8 +2707,9 @@ struct hwrm_port_mac_ptp_qcfg_output { __le16 seq_id; __le16 resp_len; u8 flags; - #define PORT_MAC_PTP_QCFG_RESP_FLAGS_DIRECT_ACCESS 0x1UL - #define PORT_MAC_PTP_QCFG_RESP_FLAGS_HWRM_ACCESS 0x2UL + #define PORT_MAC_PTP_QCFG_RESP_FLAGS_DIRECT_ACCESS 0x1UL + #define PORT_MAC_PTP_QCFG_RESP_FLAGS_HWRM_ACCESS 0x2UL + #define PORT_MAC_PTP_QCFG_RESP_FLAGS_ONE_STEP_TX_TS 0x4UL u8 unused_0[3]; __le32 rx_ts_reg_off_lower; __le32 rx_ts_reg_off_upper; @@ -2888,7 +2916,7 @@ struct tx_port_stats_ext { __le64 pfc_pri7_tx_transitions; }; -/* rx_port_stats_ext (size:2368b/296B) */ +/* rx_port_stats_ext (size:2624b/328B) */ struct rx_port_stats_ext { __le64 link_down_events; __le64 continuous_pause_events; @@ -2927,6 +2955,10 @@ struct rx_port_stats_ext { __le64 pfc_pri6_rx_transitions; __le64 pfc_pri7_rx_duration_us; __le64 pfc_pri7_rx_transitions; + __le64 rx_bits; + __le64 rx_buffer_passed_threshold; + __le64 rx_pcs_symbol_err; + __le64 rx_corrected_bits; }; /* hwrm_port_qstats_ext_input (size:320b/40B) */ @@ -3029,6 +3061,35 @@ struct hwrm_port_lpbk_clr_stats_output { u8 valid; }; +/* hwrm_port_ts_query_input (size:192b/24B) */ +struct hwrm_port_ts_query_input { + __le16 req_type; + __le16 cmpl_ring; + __le16 seq_id; + __le16 target_id; + __le64 resp_addr; + __le32 flags; + #define PORT_TS_QUERY_REQ_FLAGS_PATH 0x1UL + #define PORT_TS_QUERY_REQ_FLAGS_PATH_TX 0x0UL + #define PORT_TS_QUERY_REQ_FLAGS_PATH_RX 0x1UL + #define PORT_TS_QUERY_REQ_FLAGS_PATH_LAST PORT_TS_QUERY_REQ_FLAGS_PATH_RX + #define PORT_TS_QUERY_REQ_FLAGS_CURRENT_TIME 0x2UL + __le16 port_id; + u8 unused_0[2]; +}; + +/* hwrm_port_ts_query_output (size:192b/24B) */ +struct hwrm_port_ts_query_output { + __le16 error_code; + __le16 req_type; + __le16 seq_id; + __le16 resp_len; + __le64 ptp_msg_ts; + __le16 ptp_msg_seqid; + u8 unused_0[5]; + u8 valid; +}; + /* hwrm_port_phy_qcaps_input (size:192b/24B) */ struct hwrm_port_phy_qcaps_input { __le16 req_type; @@ -4703,7 +4764,8 @@ struct hwrm_vnic_qcaps_output { #define VNIC_QCAPS_RESP_FLAGS_RSS_DFLT_CR_CAP 0x20UL #define VNIC_QCAPS_RESP_FLAGS_ROCE_MIRRORING_CAPABLE_VNIC_CAP 0x40UL #define VNIC_QCAPS_RESP_FLAGS_OUTERMOST_RSS_CAP 0x80UL - u8 unused_1[7]; + __le16 max_aggs_supported; + u8 unused_1[5]; u8 valid; }; @@ -4723,6 +4785,7 @@ struct hwrm_vnic_tpa_cfg_input { #define VNIC_TPA_CFG_REQ_FLAGS_AGG_WITH_SAME_GRE_SEQ 0x20UL #define VNIC_TPA_CFG_REQ_FLAGS_GRO_IPID_CHECK 0x40UL #define VNIC_TPA_CFG_REQ_FLAGS_GRO_TTL_CHECK 0x80UL + #define VNIC_TPA_CFG_REQ_FLAGS_AGG_PACK_AS_GRO 0x100UL __le32 enables; #define VNIC_TPA_CFG_REQ_ENABLES_MAX_AGG_SEGS 0x1UL #define VNIC_TPA_CFG_REQ_ENABLES_MAX_AGGS 0x2UL @@ -5254,6 +5317,8 @@ struct hwrm_cfa_l2_filter_alloc_input { #define CFA_L2_FILTER_ALLOC_REQ_FLAGS_TRAFFIC_L2 (0x1UL << 4) #define CFA_L2_FILTER_ALLOC_REQ_FLAGS_TRAFFIC_ROCE (0x2UL << 4) #define CFA_L2_FILTER_ALLOC_REQ_FLAGS_TRAFFIC_LAST CFA_L2_FILTER_ALLOC_REQ_FLAGS_TRAFFIC_ROCE + #define CFA_L2_FILTER_ALLOC_REQ_FLAGS_XDP_DISABLE 0x40UL + #define CFA_L2_FILTER_ALLOC_REQ_FLAGS_SOURCE_VALID 0x80UL __le32 enables; #define CFA_L2_FILTER_ALLOC_REQ_ENABLES_L2_ADDR 0x1UL #define CFA_L2_FILTER_ALLOC_REQ_ENABLES_L2_ADDR_MASK 0x2UL @@ -5272,8 +5337,11 @@ struct hwrm_cfa_l2_filter_alloc_input { #define CFA_L2_FILTER_ALLOC_REQ_ENABLES_TUNNEL_TYPE 0x4000UL #define CFA_L2_FILTER_ALLOC_REQ_ENABLES_DST_ID 0x8000UL #define CFA_L2_FILTER_ALLOC_REQ_ENABLES_MIRROR_VNIC_ID 0x10000UL + #define CFA_L2_FILTER_ALLOC_REQ_ENABLES_NUM_VLANS 0x20000UL + #define CFA_L2_FILTER_ALLOC_REQ_ENABLES_T_NUM_VLANS 0x40000UL u8 l2_addr[6]; - u8 unused_0[2]; + u8 num_vlans; + u8 t_num_vlans; u8 l2_addr_mask[6]; __le16 l2_ovlan; __le16 l2_ovlan_mask; @@ -5338,6 +5406,16 @@ struct hwrm_cfa_l2_filter_alloc_output { __le16 resp_len; __le64 l2_filter_id; __le32 flow_id; + #define CFA_L2_FILTER_ALLOC_RESP_FLOW_ID_VALUE_MASK 0x3fffffffUL + #define CFA_L2_FILTER_ALLOC_RESP_FLOW_ID_VALUE_SFT 0 + #define CFA_L2_FILTER_ALLOC_RESP_FLOW_ID_TYPE 0x40000000UL + #define CFA_L2_FILTER_ALLOC_RESP_FLOW_ID_TYPE_INT (0x0UL << 30) + #define CFA_L2_FILTER_ALLOC_RESP_FLOW_ID_TYPE_EXT (0x1UL << 30) + #define CFA_L2_FILTER_ALLOC_RESP_FLOW_ID_TYPE_LAST CFA_L2_FILTER_ALLOC_RESP_FLOW_ID_TYPE_EXT + #define CFA_L2_FILTER_ALLOC_RESP_FLOW_ID_DIR 0x80000000UL + #define CFA_L2_FILTER_ALLOC_RESP_FLOW_ID_DIR_RX (0x0UL << 31) + #define CFA_L2_FILTER_ALLOC_RESP_FLOW_ID_DIR_TX (0x1UL << 31) + #define CFA_L2_FILTER_ALLOC_RESP_FLOW_ID_DIR_LAST CFA_L2_FILTER_ALLOC_RESP_FLOW_ID_DIR_TX u8 unused_0[3]; u8 valid; }; @@ -5504,6 +5582,16 @@ struct hwrm_cfa_tunnel_filter_alloc_output { __le16 resp_len; __le64 tunnel_filter_id; __le32 flow_id; + #define CFA_TUNNEL_FILTER_ALLOC_RESP_FLOW_ID_VALUE_MASK 0x3fffffffUL + #define CFA_TUNNEL_FILTER_ALLOC_RESP_FLOW_ID_VALUE_SFT 0 + #define CFA_TUNNEL_FILTER_ALLOC_RESP_FLOW_ID_TYPE 0x40000000UL + #define CFA_TUNNEL_FILTER_ALLOC_RESP_FLOW_ID_TYPE_INT (0x0UL << 30) + #define CFA_TUNNEL_FILTER_ALLOC_RESP_FLOW_ID_TYPE_EXT (0x1UL << 30) + #define CFA_TUNNEL_FILTER_ALLOC_RESP_FLOW_ID_TYPE_LAST CFA_TUNNEL_FILTER_ALLOC_RESP_FLOW_ID_TYPE_EXT + #define CFA_TUNNEL_FILTER_ALLOC_RESP_FLOW_ID_DIR 0x80000000UL + #define CFA_TUNNEL_FILTER_ALLOC_RESP_FLOW_ID_DIR_RX (0x0UL << 31) + #define CFA_TUNNEL_FILTER_ALLOC_RESP_FLOW_ID_DIR_TX (0x1UL << 31) + #define CFA_TUNNEL_FILTER_ALLOC_RESP_FLOW_ID_DIR_LAST CFA_TUNNEL_FILTER_ALLOC_RESP_FLOW_ID_DIR_TX u8 unused_0[3]; u8 valid; }; @@ -5646,7 +5734,7 @@ struct hwrm_cfa_encap_record_free_output { u8 valid; }; -/* hwrm_cfa_ntuple_filter_alloc_input (size:1024b/128B) */ +/* hwrm_cfa_ntuple_filter_alloc_input (size:1088b/136B) */ struct hwrm_cfa_ntuple_filter_alloc_input { __le16 req_type; __le16 cmpl_ring; @@ -5678,6 +5766,7 @@ struct hwrm_cfa_ntuple_filter_alloc_input { #define CFA_NTUPLE_FILTER_ALLOC_REQ_ENABLES_DST_ID 0x10000UL #define CFA_NTUPLE_FILTER_ALLOC_REQ_ENABLES_MIRROR_VNIC_ID 0x20000UL #define CFA_NTUPLE_FILTER_ALLOC_REQ_ENABLES_DST_MACADDR 0x40000UL + #define CFA_NTUPLE_FILTER_ALLOC_REQ_ENABLES_RFS_RING_TBL_IDX 0x80000UL __le64 l2_filter_id; u8 src_macaddr[6]; __be16 ethertype; @@ -5725,6 +5814,8 @@ struct hwrm_cfa_ntuple_filter_alloc_input { __be16 dst_port; __be16 dst_port_mask; __le64 ntuple_filter_id_hint; + __le16 rfs_ring_tbl_idx; + u8 unused_0[6]; }; /* hwrm_cfa_ntuple_filter_alloc_output (size:192b/24B) */ @@ -5735,6 +5826,16 @@ struct hwrm_cfa_ntuple_filter_alloc_output { __le16 resp_len; __le64 ntuple_filter_id; __le32 flow_id; + #define CFA_NTUPLE_FILTER_ALLOC_RESP_FLOW_ID_VALUE_MASK 0x3fffffffUL + #define CFA_NTUPLE_FILTER_ALLOC_RESP_FLOW_ID_VALUE_SFT 0 + #define CFA_NTUPLE_FILTER_ALLOC_RESP_FLOW_ID_TYPE 0x40000000UL + #define CFA_NTUPLE_FILTER_ALLOC_RESP_FLOW_ID_TYPE_INT (0x0UL << 30) + #define CFA_NTUPLE_FILTER_ALLOC_RESP_FLOW_ID_TYPE_EXT (0x1UL << 30) + #define CFA_NTUPLE_FILTER_ALLOC_RESP_FLOW_ID_TYPE_LAST CFA_NTUPLE_FILTER_ALLOC_RESP_FLOW_ID_TYPE_EXT + #define CFA_NTUPLE_FILTER_ALLOC_RESP_FLOW_ID_DIR 0x80000000UL + #define CFA_NTUPLE_FILTER_ALLOC_RESP_FLOW_ID_DIR_RX (0x0UL << 31) + #define CFA_NTUPLE_FILTER_ALLOC_RESP_FLOW_ID_DIR_TX (0x1UL << 31) + #define CFA_NTUPLE_FILTER_ALLOC_RESP_FLOW_ID_DIR_LAST CFA_NTUPLE_FILTER_ALLOC_RESP_FLOW_ID_DIR_TX u8 unused_0[3]; u8 valid; }; @@ -5934,19 +6035,20 @@ struct hwrm_cfa_flow_alloc_input { __le16 src_fid; __le32 tunnel_handle; __le16 action_flags; - #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_FWD 0x1UL - #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_RECYCLE 0x2UL - #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_DROP 0x4UL - #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_METER 0x8UL - #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_TUNNEL 0x10UL - #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_NAT_SRC 0x20UL - #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_NAT_DEST 0x40UL - #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_NAT_IPV4_ADDRESS 0x80UL - #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_L2_HEADER_REWRITE 0x100UL - #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_TTL_DECREMENT 0x200UL - #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_TUNNEL_IP 0x400UL - #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_FLOW_AGING_ENABLED 0x800UL - #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_PRI_HINT 0x1000UL + #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_FWD 0x1UL + #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_RECYCLE 0x2UL + #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_DROP 0x4UL + #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_METER 0x8UL + #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_TUNNEL 0x10UL + #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_NAT_SRC 0x20UL + #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_NAT_DEST 0x40UL + #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_NAT_IPV4_ADDRESS 0x80UL + #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_L2_HEADER_REWRITE 0x100UL + #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_TTL_DECREMENT 0x200UL + #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_TUNNEL_IP 0x400UL + #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_FLOW_AGING_ENABLED 0x800UL + #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_PRI_HINT 0x1000UL + #define CFA_FLOW_ALLOC_REQ_ACTION_FLAGS_NO_FLOW_COUNTER_ALLOC 0x2000UL __le16 dst_fid; __be16 l2_rewrite_vlan_tpid; __be16 l2_rewrite_vlan_tci; @@ -5997,6 +6099,16 @@ struct hwrm_cfa_flow_alloc_output { __le16 flow_handle; u8 unused_0[2]; __le32 flow_id; + #define CFA_FLOW_ALLOC_RESP_FLOW_ID_VALUE_MASK 0x3fffffffUL + #define CFA_FLOW_ALLOC_RESP_FLOW_ID_VALUE_SFT 0 + #define CFA_FLOW_ALLOC_RESP_FLOW_ID_TYPE 0x40000000UL + #define CFA_FLOW_ALLOC_RESP_FLOW_ID_TYPE_INT (0x0UL << 30) + #define CFA_FLOW_ALLOC_RESP_FLOW_ID_TYPE_EXT (0x1UL << 30) + #define CFA_FLOW_ALLOC_RESP_FLOW_ID_TYPE_LAST CFA_FLOW_ALLOC_RESP_FLOW_ID_TYPE_EXT + #define CFA_FLOW_ALLOC_RESP_FLOW_ID_DIR 0x80000000UL + #define CFA_FLOW_ALLOC_RESP_FLOW_ID_DIR_RX (0x0UL << 31) + #define CFA_FLOW_ALLOC_RESP_FLOW_ID_DIR_TX (0x1UL << 31) + #define CFA_FLOW_ALLOC_RESP_FLOW_ID_DIR_LAST CFA_FLOW_ALLOC_RESP_FLOW_ID_DIR_TX __le64 ext_flow_handle; __le32 flow_counter_id; u8 unused_1[3]; @@ -6011,7 +6123,8 @@ struct hwrm_cfa_flow_free_input { __le16 target_id; __le64 resp_addr; __le16 flow_handle; - u8 unused_0[6]; + __le16 unused_0; + __le32 flow_counter_id; __le64 ext_flow_handle; }; @@ -6199,8 +6312,10 @@ struct hwrm_cfa_eem_qcaps_output { __le16 seq_id; __le16 resp_len; __le32 flags; - #define CFA_EEM_QCAPS_RESP_FLAGS_PATH_TX 0x1UL - #define CFA_EEM_QCAPS_RESP_FLAGS_PATH_RX 0x2UL + #define CFA_EEM_QCAPS_RESP_FLAGS_PATH_TX 0x1UL + #define CFA_EEM_QCAPS_RESP_FLAGS_PATH_RX 0x2UL + #define CFA_EEM_QCAPS_RESP_FLAGS_CENTRALIZED_MEMORY_MODEL_SUPPORTED 0x4UL + #define CFA_EEM_QCAPS_RESP_FLAGS_DETACHED_CENTRALIZED_MEMORY_MODEL_SUPPORTED 0x8UL __le32 unused_0; __le32 supported; #define CFA_EEM_QCAPS_RESP_SUPPORTED_KEY0_TABLE 0x1UL @@ -6226,7 +6341,9 @@ struct hwrm_cfa_eem_cfg_input { #define CFA_EEM_CFG_REQ_FLAGS_PATH_TX 0x1UL #define CFA_EEM_CFG_REQ_FLAGS_PATH_RX 0x2UL #define CFA_EEM_CFG_REQ_FLAGS_PREFERRED_OFFLOAD 0x4UL - __le32 unused_0; + #define CFA_EEM_CFG_REQ_FLAGS_SECONDARY_PF 0x8UL + __le16 group_id; + __le16 unused_0; __le32 num_entries; __le32 unused_1; __le16 key0_ctx_id; @@ -6258,7 +6375,7 @@ struct hwrm_cfa_eem_qcfg_input { __le32 unused_0; }; -/* hwrm_cfa_eem_qcfg_output (size:128b/16B) */ +/* hwrm_cfa_eem_qcfg_output (size:192b/24B) */ struct hwrm_cfa_eem_qcfg_output { __le16 error_code; __le16 req_type; @@ -6269,6 +6386,8 @@ struct hwrm_cfa_eem_qcfg_output { #define CFA_EEM_QCFG_RESP_FLAGS_PATH_RX 0x2UL #define CFA_EEM_QCFG_RESP_FLAGS_PREFERRED_OFFLOAD 0x4UL __le32 num_entries; + u8 unused_0[7]; + u8 valid; }; /* hwrm_cfa_eem_op_input (size:192b/24B) */ @@ -6300,6 +6419,39 @@ struct hwrm_cfa_eem_op_output { u8 valid; }; +/* hwrm_cfa_adv_flow_mgnt_qcaps_input (size:256b/32B) */ +struct hwrm_cfa_adv_flow_mgnt_qcaps_input { + __le16 req_type; + __le16 cmpl_ring; + __le16 seq_id; + __le16 target_id; + __le64 resp_addr; + __le32 unused_0[4]; +}; + +/* hwrm_cfa_adv_flow_mgnt_qcaps_output (size:128b/16B) */ +struct hwrm_cfa_adv_flow_mgnt_qcaps_output { + __le16 error_code; + __le16 req_type; + __le16 seq_id; + __le16 resp_len; + __le32 flags; + #define CFA_ADV_FLOW_MGNT_QCAPS_RESP_FLAGS_FLOW_HND_16BIT_SUPPORTED 0x1UL + #define CFA_ADV_FLOW_MGNT_QCAPS_RESP_FLAGS_FLOW_HND_64BIT_SUPPORTED 0x2UL + #define CFA_ADV_FLOW_MGNT_QCAPS_RESP_FLAGS_FLOW_BATCH_DELETE_SUPPORTED 0x4UL + #define CFA_ADV_FLOW_MGNT_QCAPS_RESP_FLAGS_FLOW_RESET_ALL_SUPPORTED 0x8UL + #define CFA_ADV_FLOW_MGNT_QCAPS_RESP_FLAGS_NTUPLE_FLOW_DEST_FUNC_SUPPORTED 0x10UL + #define CFA_ADV_FLOW_MGNT_QCAPS_RESP_FLAGS_TX_EEM_FLOW_SUPPORTED 0x20UL + #define CFA_ADV_FLOW_MGNT_QCAPS_RESP_FLAGS_RX_EEM_FLOW_SUPPORTED 0x40UL + #define CFA_ADV_FLOW_MGNT_QCAPS_RESP_FLAGS_FLOW_COUNTER_ALLOC_SUPPORTED 0x80UL + #define CFA_ADV_FLOW_MGNT_QCAPS_RESP_FLAGS_RFS_RING_TBL_IDX_SUPPORTED 0x100UL + #define CFA_ADV_FLOW_MGNT_QCAPS_RESP_FLAGS_UNTAGGED_VLAN_SUPPORTED 0x200UL + #define CFA_ADV_FLOW_MGNT_QCAPS_RESP_FLAGS_XDP_SUPPORTED 0x400UL + #define CFA_ADV_FLOW_MGNT_QCAPS_RESP_FLAGS_L2_HEADER_SOURCE_FIELDS_SUPPORTED 0x800UL + u8 unused_0[3]; + u8 valid; +}; + /* hwrm_tunnel_dst_port_query_input (size:192b/24B) */ struct hwrm_tunnel_dst_port_query_input { __le16 req_type; @@ -6636,7 +6788,8 @@ struct hwrm_fw_qstatus_output { #define FW_QSTATUS_RESP_SELFRST_STATUS_SELFRSTNONE 0x0UL #define FW_QSTATUS_RESP_SELFRST_STATUS_SELFRSTASAP 0x1UL #define FW_QSTATUS_RESP_SELFRST_STATUS_SELFRSTPCIERST 0x2UL - #define FW_QSTATUS_RESP_SELFRST_STATUS_LAST FW_QSTATUS_RESP_SELFRST_STATUS_SELFRSTPCIERST + #define FW_QSTATUS_RESP_SELFRST_STATUS_SELFRSTPOWER 0x3UL + #define FW_QSTATUS_RESP_SELFRST_STATUS_LAST FW_QSTATUS_RESP_SELFRST_STATUS_SELFRSTPOWER u8 unused_0[6]; u8 valid; }; @@ -6659,8 +6812,8 @@ struct hwrm_fw_set_time_input { u8 unused_0; __le16 millisecond; __le16 zone; - #define FW_SET_TIME_REQ_ZONE_UTC 0x0UL - #define FW_SET_TIME_REQ_ZONE_UNKNOWN 0xffffUL + #define FW_SET_TIME_REQ_ZONE_UTC 0 + #define FW_SET_TIME_REQ_ZONE_UNKNOWN 65535 #define FW_SET_TIME_REQ_ZONE_LAST FW_SET_TIME_REQ_ZONE_UNKNOWN u8 unused_1[4]; }; @@ -7064,7 +7217,9 @@ struct hwrm_dbg_coredump_list_input { __le64 host_dest_addr; __le32 host_buf_len; __le16 seq_no; - u8 unused_0[2]; + u8 flags; + #define DBG_COREDUMP_LIST_REQ_FLAGS_CRASHDUMP 0x1UL + u8 unused_0[1]; }; /* hwrm_dbg_coredump_list_output (size:128b/16B) */ @@ -7392,7 +7547,9 @@ struct hwrm_nvm_get_dev_info_output { __le32 nvram_size; __le32 reserved_size; __le32 available_size; - u8 unused_0[3]; + u8 nvm_cfg_ver_maj; + u8 nvm_cfg_ver_min; + u8 nvm_cfg_ver_upd; u8 valid; }; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_vfr.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_vfr.c index 2bdd2da9aac7d5777a111b616cf2ecf178ecbe1c..f760921389a304214f854a4604f6233e6eafbfe5 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_vfr.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_vfr.c @@ -406,26 +406,6 @@ static void bnxt_vf_rep_netdev_init(struct bnxt *bp, struct bnxt_vf_rep *vf_rep, dev->min_mtu = ETH_ZLEN; } -static int bnxt_pcie_dsn_get(struct bnxt *bp, u8 dsn[]) -{ - struct pci_dev *pdev = bp->pdev; - int pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_DSN); - u32 dw; - - if (!pos) { - netdev_info(bp->dev, "Unable do read adapter's DSN"); - return -EOPNOTSUPP; - } - - /* DSN (two dw) is at an offset of 4 from the cap pos */ - pos += 4; - pci_read_config_dword(pdev, pos, &dw); - put_unaligned_le32(dw, &dsn[0]); - pci_read_config_dword(pdev, pos + 4, &dw); - put_unaligned_le32(dw, &dsn[4]); - return 0; -} - static int bnxt_vf_reps_create(struct bnxt *bp) { u16 *cfa_code_map = NULL, num_vfs = pci_num_vf(bp->pdev); @@ -490,11 +470,6 @@ static int bnxt_vf_reps_create(struct bnxt *bp) } } - /* Read the adapter's DSN to use as the eswitch switch_id */ - rc = bnxt_pcie_dsn_get(bp, bp->switch_id); - if (rc) - goto err; - /* publish cfa_code_map only after all VF-reps have been initialized */ bp->cfa_code_map = cfa_code_map; bp->eswitch_mode = DEVLINK_ESWITCH_MODE_SWITCHDEV; diff --git a/drivers/net/ethernet/broadcom/cnic.c b/drivers/net/ethernet/broadcom/cnic.c index 510dfc1c236bdb164bfaea4476a44f184137972e..57dc3cbff36ecd873a7e8c6ad123bc1b9bef9311 100644 --- a/drivers/net/ethernet/broadcom/cnic.c +++ b/drivers/net/ethernet/broadcom/cnic.c @@ -4038,15 +4038,14 @@ static void cnic_cm_process_kcqe(struct cnic_dev *dev, struct kcqe *kcqe) case L5CM_RAMROD_CMD_ID_CLOSE: { struct iscsi_kcqe *l5kcqe = (struct iscsi_kcqe *) kcqe; - if (l4kcqe->status != 0 || l5kcqe->completion_status != 0) { - netdev_warn(dev->netdev, "RAMROD CLOSE compl with status 0x%x completion status 0x%x\n", - l4kcqe->status, l5kcqe->completion_status); - opcode = L4_KCQE_OPCODE_VALUE_CLOSE_COMP; - /* Fall through */ - } else { + if (l4kcqe->status == 0 && l5kcqe->completion_status == 0) break; - } + + netdev_warn(dev->netdev, "RAMROD CLOSE compl with status 0x%x completion status 0x%x\n", + l4kcqe->status, l5kcqe->completion_status); + opcode = L4_KCQE_OPCODE_VALUE_CLOSE_COMP; } + /* Fall through */ case L4_KCQE_OPCODE_VALUE_RESET_RECEIVED: case L4_KCQE_OPCODE_VALUE_CLOSE_COMP: case L4_KCQE_OPCODE_VALUE_RESET_COMP: diff --git a/drivers/net/ethernet/broadcom/genet/bcmgenet.c b/drivers/net/ethernet/broadcom/genet/bcmgenet.c index 983245c0867c268be95b1ae52f57a2c0b4051507..374b9ff05c88727ed485dfcb0d27e79a23c97180 100644 --- a/drivers/net/ethernet/broadcom/genet/bcmgenet.c +++ b/drivers/net/ethernet/broadcom/genet/bcmgenet.c @@ -1665,7 +1665,7 @@ static netdev_tx_t bcmgenet_xmit(struct sk_buff *skb, struct net_device *dev) if (ring->free_bds <= (MAX_SKB_FRAGS + 1)) netif_tx_stop_queue(txq); - if (!skb->xmit_more || netif_xmit_stopped(txq)) + if (!netdev_xmit_more() || netif_xmit_stopped(txq)) /* Packets are ready, update producer index */ bcmgenet_tdma_ring_writel(priv, ring->index, ring->prod_index, TDMA_PROD_INDEX); @@ -3476,7 +3476,7 @@ static int bcmgenet_probe(struct platform_device *pdev) if (dn) { macaddr = of_get_mac_address(dn); - if (!macaddr) { + if (IS_ERR(macaddr)) { dev_err(&pdev->dev, "can't find MAC address\n"); err = -EINVAL; goto err; diff --git a/drivers/net/ethernet/broadcom/tg3.c b/drivers/net/ethernet/broadcom/tg3.c index 2aebd4bbb67dc420096a775d8c6c46ffa44e46c9..6d1f9c822548ef4191c798079be4cb467df20584 100644 --- a/drivers/net/ethernet/broadcom/tg3.c +++ b/drivers/net/ethernet/broadcom/tg3.c @@ -8151,7 +8151,7 @@ static netdev_tx_t tg3_start_xmit(struct sk_buff *skb, struct net_device *dev) netif_tx_wake_queue(txq); } - if (!skb->xmit_more || netif_xmit_stopped(txq)) { + if (!netdev_xmit_more() || netif_xmit_stopped(txq)) { /* Packets are ready, update Tx producer idx on card. */ tw32_tx_mbox(tnapi->prodmbox, entry); } @@ -12757,9 +12757,6 @@ static int tg3_set_phys_id(struct net_device *dev, { struct tg3 *tp = netdev_priv(dev); - if (!netif_running(tp->dev)) - return -EAGAIN; - switch (state) { case ETHTOOL_ID_ACTIVE: return 1; /* cycle on/off once per second */ diff --git a/drivers/net/ethernet/cadence/macb_main.c b/drivers/net/ethernet/cadence/macb_main.c index 3da2795e248638abbc8d50a194dd8061325aad41..c049410bc8889e710a086111d86ceea43343fd18 100644 --- a/drivers/net/ethernet/cadence/macb_main.c +++ b/drivers/net/ethernet/cadence/macb_main.c @@ -285,34 +285,22 @@ static void macb_set_hwaddr(struct macb *bp) static void macb_get_hwaddr(struct macb *bp) { - struct macb_platform_data *pdata; u32 bottom; u16 top; u8 addr[6]; int i; - pdata = dev_get_platdata(&bp->pdev->dev); - /* Check all 4 address register for valid address */ for (i = 0; i < 4; i++) { bottom = macb_or_gem_readl(bp, SA1B + i * 8); top = macb_or_gem_readl(bp, SA1T + i * 8); - if (pdata && pdata->rev_eth_addr) { - addr[5] = bottom & 0xff; - addr[4] = (bottom >> 8) & 0xff; - addr[3] = (bottom >> 16) & 0xff; - addr[2] = (bottom >> 24) & 0xff; - addr[1] = top & 0xff; - addr[0] = (top & 0xff00) >> 8; - } else { - addr[0] = bottom & 0xff; - addr[1] = (bottom >> 8) & 0xff; - addr[2] = (bottom >> 16) & 0xff; - addr[3] = (bottom >> 24) & 0xff; - addr[4] = top & 0xff; - addr[5] = (top >> 8) & 0xff; - } + addr[0] = bottom & 0xff; + addr[1] = (bottom >> 8) & 0xff; + addr[2] = (bottom >> 16) & 0xff; + addr[3] = (bottom >> 24) & 0xff; + addr[4] = top & 0xff; + addr[5] = (top >> 8) & 0xff; if (is_valid_ether_addr(addr)) { memcpy(bp->dev->dev_addr, addr, sizeof(addr)); @@ -510,12 +498,10 @@ static void macb_handle_link_change(struct net_device *dev) static int macb_mii_probe(struct net_device *dev) { struct macb *bp = netdev_priv(dev); - struct macb_platform_data *pdata; struct phy_device *phydev; struct device_node *np; - int phy_irq, ret, i; + int ret, i; - pdata = dev_get_platdata(&bp->pdev->dev); np = bp->pdev->dev.of_node; ret = 0; @@ -530,8 +516,6 @@ static int macb_mii_probe(struct net_device *dev) */ if (!bp->phy_node && !phy_find_first(bp->mii_bus)) { for (i = 0; i < PHY_MAX_ADDR; i++) { - struct phy_device *phydev; - phydev = mdiobus_scan(bp->mii_bus, i); if (IS_ERR(phydev) && PTR_ERR(phydev) != -ENODEV) { @@ -559,19 +543,6 @@ static int macb_mii_probe(struct net_device *dev) return -ENXIO; } - if (pdata) { - if (gpio_is_valid(pdata->phy_irq_pin)) { - ret = devm_gpio_request(&bp->pdev->dev, - pdata->phy_irq_pin, "phy int"); - if (!ret) { - phy_irq = gpio_to_irq(pdata->phy_irq_pin); - phydev->irq = (phy_irq < 0) ? PHY_POLL : phy_irq; - } - } else { - phydev->irq = PHY_POLL; - } - } - /* attach the mac to the phy */ ret = phy_connect_direct(dev, phydev, &macb_handle_link_change, bp->phy_interface); @@ -600,7 +571,6 @@ static int macb_mii_probe(struct net_device *dev) static int macb_mii_init(struct macb *bp) { - struct macb_platform_data *pdata; struct device_node *np; int err = -ENXIO; @@ -620,7 +590,6 @@ static int macb_mii_init(struct macb *bp) bp->pdev->name, bp->pdev->id); bp->mii_bus->priv = bp; bp->mii_bus->parent = &bp->pdev->dev; - pdata = dev_get_platdata(&bp->pdev->dev); dev_set_drvdata(&bp->dev->dev, bp->mii_bus); @@ -634,9 +603,6 @@ static int macb_mii_init(struct macb *bp) err = mdiobus_register(bp->mii_bus); } else { - if (pdata) - bp->mii_bus->phy_mask = pdata->phy_mask; - err = of_mdiobus_register(bp->mii_bus, np); } @@ -2461,12 +2427,12 @@ static int macb_open(struct net_device *dev) goto pm_exit; } - bp->macbgem_ops.mog_init_rings(bp); - macb_init_hw(bp); - for (q = 0, queue = bp->queues; q < bp->num_queues; ++q, ++queue) napi_enable(&queue->napi); + bp->macbgem_ops.mog_init_rings(bp); + macb_init_hw(bp); + /* schedule a link state check */ phy_start(dev->phydev); @@ -4052,7 +4018,6 @@ static int macb_probe(struct platform_device *pdev) struct clk *pclk, *hclk = NULL, *tx_clk = NULL, *rx_clk = NULL; struct clk *tsu_clk = NULL; unsigned int queue_mask, num_queues; - struct macb_platform_data *pdata; bool native_io; struct phy_device *phydev; struct net_device *dev; @@ -4172,27 +4137,21 @@ static int macb_probe(struct platform_device *pdev) bp->rx_intr_mask |= MACB_BIT(RXUBR); mac = of_get_mac_address(np); - if (mac) { + if (PTR_ERR(mac) == -EPROBE_DEFER) { + err = -EPROBE_DEFER; + goto err_out_free_netdev; + } else if (!IS_ERR(mac)) { ether_addr_copy(bp->dev->dev_addr, mac); } else { - err = nvmem_get_mac_address(&pdev->dev, bp->dev->dev_addr); - if (err) { - if (err == -EPROBE_DEFER) - goto err_out_free_netdev; - macb_get_hwaddr(bp); - } + macb_get_hwaddr(bp); } err = of_get_phy_mode(np); - if (err < 0) { - pdata = dev_get_platdata(&pdev->dev); - if (pdata && pdata->is_rmii) - bp->phy_interface = PHY_INTERFACE_MODE_RMII; - else - bp->phy_interface = PHY_INTERFACE_MODE_MII; - } else { + if (err < 0) + /* not found in DT, MII by default */ + bp->phy_interface = PHY_INTERFACE_MODE_MII; + else bp->phy_interface = err; - } /* IP specific init */ err = init(pdev); @@ -4362,8 +4321,7 @@ static int __maybe_unused macb_resume(struct device *dev) static int __maybe_unused macb_runtime_suspend(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct net_device *netdev = platform_get_drvdata(pdev); + struct net_device *netdev = dev_get_drvdata(dev); struct macb *bp = netdev_priv(netdev); if (!(device_may_wakeup(&bp->dev->dev))) { @@ -4379,8 +4337,7 @@ static int __maybe_unused macb_runtime_suspend(struct device *dev) static int __maybe_unused macb_runtime_resume(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct net_device *netdev = platform_get_drvdata(pdev); + struct net_device *netdev = dev_get_drvdata(dev); struct macb *bp = netdev_priv(netdev); if (!(device_may_wakeup(&bp->dev->dev))) { diff --git a/drivers/net/ethernet/cavium/Kconfig b/drivers/net/ethernet/cavium/Kconfig index 6650e2a5f171385036f761f008198724a2c99ba4..7612ab6b286dfd915604c7c50e9e2b970fe42b6f 100644 --- a/drivers/net/ethernet/cavium/Kconfig +++ b/drivers/net/ethernet/cavium/Kconfig @@ -68,6 +68,7 @@ config LIQUIDIO imply PTP_1588_CLOCK select FW_LOADER select LIBCRC32C + select NET_DEVLINK ---help--- This driver supports Cavium LiquidIO Intelligent Server Adapters based on CN66XX, CN68XX and CN23XX chips. diff --git a/drivers/net/ethernet/cavium/liquidio/lio_main.c b/drivers/net/ethernet/cavium/liquidio/lio_main.c index fb6f813cff65dcb7f608d4e1b859be9f38a6b13d..eab805579f9646b42a30e71757e3f6d6432ba7e2 100644 --- a/drivers/net/ethernet/cavium/liquidio/lio_main.c +++ b/drivers/net/ethernet/cavium/liquidio/lio_main.c @@ -2522,7 +2522,7 @@ static netdev_tx_t liquidio_xmit(struct sk_buff *skb, struct net_device *netdev) irh->vlan = skb_vlan_tag_get(skb) & 0xfff; } - xmit_more = skb->xmit_more; + xmit_more = netdev_xmit_more(); if (unlikely(cmdsetup.s.timestamp)) status = send_nic_timestamp_pkt(oct, &ndata, finfo, xmit_more); diff --git a/drivers/net/ethernet/cavium/liquidio/lio_vf_main.c b/drivers/net/ethernet/cavium/liquidio/lio_vf_main.c index 54b245797d2e61e1844a25c9674255c2dd80e69c..db0b90555acbcb368626757b803474710d135082 100644 --- a/drivers/net/ethernet/cavium/liquidio/lio_vf_main.c +++ b/drivers/net/ethernet/cavium/liquidio/lio_vf_main.c @@ -1585,7 +1585,7 @@ static netdev_tx_t liquidio_xmit(struct sk_buff *skb, struct net_device *netdev) irh->vlan = skb_vlan_tag_get(skb) & VLAN_VID_MASK; } - xmit_more = skb->xmit_more; + xmit_more = netdev_xmit_more(); if (unlikely(cmdsetup.s.timestamp)) status = send_nic_timestamp_pkt(oct, &ndata, finfo, xmit_more); diff --git a/drivers/net/ethernet/cavium/octeon/octeon_mgmt.c b/drivers/net/ethernet/cavium/octeon/octeon_mgmt.c index 5359c1021f428b04bc13ad2de9cc437fc47596cb..15b1130aa4aee45602c3185d6e0c9df1610b1284 100644 --- a/drivers/net/ethernet/cavium/octeon/octeon_mgmt.c +++ b/drivers/net/ethernet/cavium/octeon/octeon_mgmt.c @@ -1503,7 +1503,7 @@ static int octeon_mgmt_probe(struct platform_device *pdev) mac = of_get_mac_address(pdev->dev.of_node); - if (mac) + if (!IS_ERR(mac)) memcpy(netdev->dev_addr, mac, ETH_ALEN); else eth_hw_addr_random(netdev); diff --git a/drivers/net/ethernet/cavium/thunder/thunder_bgx.c b/drivers/net/ethernet/cavium/thunder/thunder_bgx.c index 673c57b8023fe3e2113cc1121f15d6700d12faa1..a65be851124f4b31841afaf62c859a4af75231f7 100644 --- a/drivers/net/ethernet/cavium/thunder/thunder_bgx.c +++ b/drivers/net/ethernet/cavium/thunder/thunder_bgx.c @@ -962,13 +962,13 @@ static void bgx_poll_for_sgmii_link(struct lmac *lmac) lmac->last_duplex = (an_result >> 1) & 0x1; switch (speed) { case 0: - lmac->last_speed = 10; + lmac->last_speed = SPEED_10; break; case 1: - lmac->last_speed = 100; + lmac->last_speed = SPEED_100; break; case 2: - lmac->last_speed = 1000; + lmac->last_speed = SPEED_1000; break; default: lmac->link_up = false; @@ -1012,10 +1012,10 @@ static void bgx_poll_for_link(struct work_struct *work) !(smu_link & SMU_RX_CTL_STATUS)) { lmac->link_up = 1; if (lmac->lmac_type == BGX_MODE_XLAUI) - lmac->last_speed = 40000; + lmac->last_speed = SPEED_40000; else - lmac->last_speed = 10000; - lmac->last_duplex = 1; + lmac->last_speed = SPEED_10000; + lmac->last_duplex = DUPLEX_FULL; } else { lmac->link_up = 0; lmac->last_speed = SPEED_UNKNOWN; @@ -1105,8 +1105,8 @@ static int bgx_lmac_enable(struct bgx *bgx, u8 lmacid) } else { /* Default to below link speed and duplex */ lmac->link_up = true; - lmac->last_speed = 1000; - lmac->last_duplex = 1; + lmac->last_speed = SPEED_1000; + lmac->last_duplex = DUPLEX_FULL; bgx_sgmii_change_link_state(lmac); return 0; } @@ -1484,7 +1484,7 @@ static int bgx_init_of_phy(struct bgx *bgx) break; mac = of_get_mac_address(node); - if (mac) + if (!IS_ERR(mac)) ether_addr_copy(bgx->lmac[lmac].mac, mac); SET_NETDEV_DEV(&bgx->lmac[lmac].netdev, &bgx->pdev->dev); diff --git a/drivers/net/ethernet/chelsio/cxgb3/l2t.c b/drivers/net/ethernet/chelsio/cxgb3/l2t.c index 0e9182d3f02c665306508d7b11278fb457292ede..b3e4118a15e700364292aba4c343f3dc67d9e849 100644 --- a/drivers/net/ethernet/chelsio/cxgb3/l2t.c +++ b/drivers/net/ethernet/chelsio/cxgb3/l2t.c @@ -443,9 +443,9 @@ found: struct l2t_data *t3_init_l2t(unsigned int l2t_capacity) { struct l2t_data *d; - int i, size = sizeof(*d) + l2t_capacity * sizeof(struct l2t_entry); + int i; - d = kvzalloc(size, GFP_KERNEL); + d = kvzalloc(struct_size(d, l2tab, l2t_capacity), GFP_KERNEL); if (!d) return NULL; diff --git a/drivers/net/ethernet/chelsio/cxgb3/l2t.h b/drivers/net/ethernet/chelsio/cxgb3/l2t.h index c2fd323c4078284ac84aa58825cc84958f108fe8..ea75f275023ffad95a4e2dfa46b1db51935417d8 100644 --- a/drivers/net/ethernet/chelsio/cxgb3/l2t.h +++ b/drivers/net/ethernet/chelsio/cxgb3/l2t.h @@ -75,8 +75,8 @@ struct l2t_data { struct l2t_entry *rover; /* starting point for next allocation */ atomic_t nfree; /* number of free entries */ rwlock_t lock; - struct l2t_entry l2tab[0]; struct rcu_head rcu_head; /* to handle rcu cleanup */ + struct l2t_entry l2tab[]; }; typedef void (*arp_failure_handler_func)(struct t3cdev * dev, diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4.h b/drivers/net/ethernet/chelsio/cxgb4/cxgb4.h index 956219c178e1276b4919e183cb44cc376530a3d8..a8fe0808823d860f8b5eed737d27a58343e9f365 100644 --- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4.h +++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4.h @@ -1575,9 +1575,11 @@ int t4_slow_intr_handler(struct adapter *adapter); int t4_wait_dev_ready(void __iomem *regs); +fw_port_cap32_t t4_link_acaps(struct adapter *adapter, unsigned int port, + struct link_config *lc); int t4_link_l1cfg_core(struct adapter *adap, unsigned int mbox, unsigned int port, struct link_config *lc, - bool sleep_ok, int timeout); + u8 sleep_ok, int timeout); static inline int t4_link_l1cfg(struct adapter *adapter, unsigned int mbox, unsigned int port, struct link_config *lc) diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_ethtool.c b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_ethtool.c index bec4711005ccd86fd69b9a01ed971757094b11ed..9e589302af9069f13915c2894c54aa8f14b37187 100644 --- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_ethtool.c +++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_ethtool.c @@ -442,7 +442,7 @@ static unsigned int speed_to_fw_caps(int speed) * Link Mode Mask. */ static void fw_caps_to_lmm(enum fw_port_type port_type, - unsigned int fw_caps, + fw_port_cap32_t fw_caps, unsigned long *link_mode_mask) { #define SET_LMM(__lmm_name) \ @@ -632,7 +632,10 @@ static int get_link_ksettings(struct net_device *dev, fw_caps_to_lmm(pi->port_type, pi->link_cfg.pcaps, link_ksettings->link_modes.supported); - fw_caps_to_lmm(pi->port_type, pi->link_cfg.acaps, + fw_caps_to_lmm(pi->port_type, + t4_link_acaps(pi->adapter, + pi->lport, + &pi->link_cfg), link_ksettings->link_modes.advertising); fw_caps_to_lmm(pi->port_type, pi->link_cfg.lpacaps, link_ksettings->link_modes.lp_advertising); @@ -642,22 +645,6 @@ static int get_link_ksettings(struct net_device *dev, : SPEED_UNKNOWN); base->duplex = DUPLEX_FULL; - if (pi->link_cfg.fc & PAUSE_RX) { - if (pi->link_cfg.fc & PAUSE_TX) { - ethtool_link_ksettings_add_link_mode(link_ksettings, - advertising, - Pause); - } else { - ethtool_link_ksettings_add_link_mode(link_ksettings, - advertising, - Asym_Pause); - } - } else if (pi->link_cfg.fc & PAUSE_TX) { - ethtool_link_ksettings_add_link_mode(link_ksettings, - advertising, - Asym_Pause); - } - base->autoneg = pi->link_cfg.autoneg; if (pi->link_cfg.pcaps & FW_PORT_CAP32_ANEG) ethtool_link_ksettings_add_link_mode(link_ksettings, diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_filter.c b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_filter.c index 5afb430000499ba55cad2da1f7a5619ff823d9be..4107007b6ec42727b195fce91c5079f35a0a3f58 100644 --- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_filter.c +++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_filter.c @@ -524,8 +524,7 @@ static int del_filter_wr(struct adapter *adapter, int fidx) return -ENOMEM; fwr = __skb_put(skb, len); - t4_mk_filtdelwr(f->tid, fwr, (adapter->flags & CXGB4_SHUTTING_DOWN) ? -1 - : adapter->sge.fw_evtq.abs_id); + t4_mk_filtdelwr(f->tid, fwr, adapter->sge.fw_evtq.abs_id); /* Mark the filter as "pending" and ship off the Filter Work Request. * When we get the Work Request Reply we'll clear the pending status. @@ -744,16 +743,40 @@ void clear_filter(struct adapter *adap, struct filter_entry *f) void clear_all_filters(struct adapter *adapter) { + struct net_device *dev = adapter->port[0]; unsigned int i; if (adapter->tids.ftid_tab) { struct filter_entry *f = &adapter->tids.ftid_tab[0]; unsigned int max_ftid = adapter->tids.nftids + adapter->tids.nsftids; - + /* Clear all TCAM filters */ for (i = 0; i < max_ftid; i++, f++) if (f->valid || f->pending) - clear_filter(adapter, f); + cxgb4_del_filter(dev, i, &f->fs); + } + + /* Clear all hash filters */ + if (is_hashfilter(adapter) && adapter->tids.tid_tab) { + struct filter_entry *f; + unsigned int sb; + + for (i = adapter->tids.hash_base; + i <= adapter->tids.ntids; i++) { + f = (struct filter_entry *) + adapter->tids.tid_tab[i]; + + if (f && (f->valid || f->pending)) + cxgb4_del_filter(dev, i, &f->fs); + } + + sb = t4_read_reg(adapter, LE_DB_SRVR_START_INDEX_A); + for (i = 0; i < sb; i++) { + f = (struct filter_entry *)adapter->tids.tid_tab[i]; + + if (f && (f->valid || f->pending)) + cxgb4_del_filter(dev, i, &f->fs); + } } } @@ -1568,9 +1591,8 @@ int cxgb4_del_filter(struct net_device *dev, int filter_id, struct filter_ctx ctx; int ret; - /* If we are shutting down the adapter do not wait for completion */ if (netdev2adap(dev)->flags & CXGB4_SHUTTING_DOWN) - return __cxgb4_del_filter(dev, filter_id, fs, NULL); + return 0; init_completion(&ctx.completion); @@ -1722,12 +1744,13 @@ void hash_filter_rpl(struct adapter *adap, const struct cpl_act_open_rpl *rpl) break; default: - dev_err(adap->pdev_dev, "%s: filter creation PROBLEM; status = %u\n", - __func__, status); + if (status != CPL_ERR_TCAM_FULL) + dev_err(adap->pdev_dev, "%s: filter creation PROBLEM; status = %u\n", + __func__, status); if (ctx) { if (status == CPL_ERR_TCAM_FULL) - ctx->result = -EAGAIN; + ctx->result = -ENOSPC; else ctx->result = -EINVAL; } diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c index 89179e31668786a25b8894faafd397d5db9f8564..715e4edcf4a2a59677c7397433c012a82f32c966 100644 --- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c +++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c @@ -979,8 +979,7 @@ freeout: } static u16 cxgb_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { int txq; @@ -1022,7 +1021,7 @@ static u16 cxgb_select_queue(struct net_device *dev, struct sk_buff *skb, return txq; } - return fallback(dev, skb, NULL) % dev->real_num_tx_queues; + return netdev_pick_tx(dev, skb, NULL) % dev->real_num_tx_queues; } static int closest_timer(const struct sge *s, int time) @@ -6025,6 +6024,11 @@ static void remove_one(struct pci_dev *pdev) return; } + /* If we allocated filters, free up state associated with any + * valid filters ... + */ + clear_all_filters(adapter); + adapter->flags |= CXGB4_SHUTTING_DOWN; if (adapter->pf == 4) { @@ -6055,11 +6059,6 @@ static void remove_one(struct pci_dev *pdev) if (IS_REACHABLE(CONFIG_THERMAL)) cxgb4_thermal_remove(adapter); - /* If we allocated filters, free up state associated with any - * valid filters ... - */ - clear_all_filters(adapter); - if (adapter->flags & CXGB4_FULL_INIT_DONE) cxgb_down(adapter); @@ -6161,15 +6160,24 @@ static int __init cxgb4_init_module(void) ret = pci_register_driver(&cxgb4_driver); if (ret < 0) - debugfs_remove(cxgb4_debugfs_root); + goto err_pci; #if IS_ENABLED(CONFIG_IPV6) if (!inet6addr_registered) { - register_inet6addr_notifier(&cxgb4_inet6addr_notifier); - inet6addr_registered = true; + ret = register_inet6addr_notifier(&cxgb4_inet6addr_notifier); + if (ret) + pci_unregister_driver(&cxgb4_driver); + else + inet6addr_registered = true; } #endif + if (ret == 0) + return ret; + +err_pci: + debugfs_remove(cxgb4_debugfs_root); + return ret; } diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_tc_flower.c b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_tc_flower.c index 82a8d19700602e6922765d647030a69d971de5ae..6e2d80008a79497bcc8b1bcb9206220d6308019c 100644 --- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_tc_flower.c +++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_tc_flower.c @@ -687,11 +687,8 @@ int cxgb4_tc_flower_replace(struct net_device *dev, ret = ctx.result; /* Check if hw returned error for filter creation */ - if (ret) { - netdev_err(dev, "%s: filter creation err %d\n", - __func__, ret); + if (ret) goto free_entry; - } ch_flower->tc_flower_cookie = cls->cookie; ch_flower->filter_id = ctx.tid; diff --git a/drivers/net/ethernet/chelsio/cxgb4/t4_hw.c b/drivers/net/ethernet/chelsio/cxgb4/t4_hw.c index a3544041ad3278f484f37d0d666d27fb1c0fbd35..f9b70be59792138266a428f8e42c5f73c01cb27c 100644 --- a/drivers/net/ethernet/chelsio/cxgb4/t4_hw.c +++ b/drivers/net/ethernet/chelsio/cxgb4/t4_hw.c @@ -3964,6 +3964,14 @@ void t4_ulprx_read_la(struct adapter *adap, u32 *la_buf) } } +/* The ADVERT_MASK is used to mask out all of the Advertised Firmware Port + * Capabilities which we control with separate controls -- see, for instance, + * Pause Frames and Forward Error Correction. In order to determine what the + * full set of Advertised Port Capabilities are, the base Advertised Port + * Capabilities (masked by ADVERT_MASK) must be combined with the Advertised + * Port Capabilities associated with those other controls. See + * t4_link_acaps() for how this is done. + */ #define ADVERT_MASK (FW_PORT_CAP32_SPEED_V(FW_PORT_CAP32_SPEED_M) | \ FW_PORT_CAP32_ANEG) @@ -4061,6 +4069,9 @@ static inline enum cc_pause fwcap_to_cc_pause(fw_port_cap32_t fw_pause) /* Translate Common Code Pause specification into Firmware Port Capabilities */ static inline fw_port_cap32_t cc_to_fwcap_pause(enum cc_pause cc_pause) { + /* Translate orthogonal RX/TX Pause Controls for L1 Configure + * commands, etc. + */ fw_port_cap32_t fw_pause = 0; if (cc_pause & PAUSE_RX) @@ -4070,6 +4081,19 @@ static inline fw_port_cap32_t cc_to_fwcap_pause(enum cc_pause cc_pause) if (!(cc_pause & PAUSE_AUTONEG)) fw_pause |= FW_PORT_CAP32_FORCE_PAUSE; + /* Translate orthogonal Pause controls into IEEE 802.3 Pause, + * Asymetrical Pause for use in reporting to upper layer OS code, etc. + * Note that these bits are ignored in L1 Configure commands. + */ + if (cc_pause & PAUSE_RX) { + if (cc_pause & PAUSE_TX) + fw_pause |= FW_PORT_CAP32_802_3_PAUSE; + else + fw_pause |= FW_PORT_CAP32_802_3_ASM_DIR; + } else if (cc_pause & PAUSE_TX) { + fw_pause |= FW_PORT_CAP32_802_3_ASM_DIR; + } + return fw_pause; } @@ -4100,31 +4124,22 @@ static inline fw_port_cap32_t cc_to_fwcap_fec(enum cc_fec cc_fec) } /** - * t4_link_l1cfg - apply link configuration to MAC/PHY + * t4_link_acaps - compute Link Advertised Port Capabilities * @adapter: the adapter - * @mbox: the Firmware Mailbox to use * @port: the Port ID * @lc: the Port's Link Configuration - * @sleep_ok: if true we may sleep while awaiting command completion - * @timeout: time to wait for command to finish before timing out - * (negative implies @sleep_ok=false) * - * Set up a port's MAC and PHY according to a desired link configuration. - * - If the PHY can auto-negotiate first decide what to advertise, then - * enable/disable auto-negotiation as desired, and reset. - * - If the PHY does not auto-negotiate just reset it. - * - If auto-negotiation is off set the MAC to the proper speed/duplex/FC, - * otherwise do it later based on the outcome of auto-negotiation. + * Synthesize the Advertised Port Capabilities we'll be using based on + * the base Advertised Port Capabilities (which have been filtered by + * ADVERT_MASK) plus the individual controls for things like Pause + * Frames, Forward Error Correction, MDI, etc. */ -int t4_link_l1cfg_core(struct adapter *adapter, unsigned int mbox, - unsigned int port, struct link_config *lc, - bool sleep_ok, int timeout) +fw_port_cap32_t t4_link_acaps(struct adapter *adapter, unsigned int port, + struct link_config *lc) { - unsigned int fw_caps = adapter->params.fw_caps_support; - fw_port_cap32_t fw_fc, cc_fec, fw_fec, rcap; - struct fw_port_cmd cmd; + fw_port_cap32_t fw_fc, fw_fec, acaps; unsigned int fw_mdi; - int ret; + char cc_fec; fw_mdi = (FW_PORT_CAP32_MDI_V(FW_PORT_CAP32_MDI_AUTO) & lc->pcaps); @@ -4151,18 +4166,15 @@ int t4_link_l1cfg_core(struct adapter *adapter, unsigned int mbox, * init_link_config(). */ if (!(lc->pcaps & FW_PORT_CAP32_ANEG)) { - if (lc->autoneg == AUTONEG_ENABLE) - return -EINVAL; - - rcap = lc->acaps | fw_fc | fw_fec; + acaps = lc->acaps | fw_fc | fw_fec; lc->fc = lc->requested_fc & ~PAUSE_AUTONEG; lc->fec = cc_fec; } else if (lc->autoneg == AUTONEG_DISABLE) { - rcap = lc->speed_caps | fw_fc | fw_fec | fw_mdi; + acaps = lc->speed_caps | fw_fc | fw_fec | fw_mdi; lc->fc = lc->requested_fc & ~PAUSE_AUTONEG; lc->fec = cc_fec; } else { - rcap = lc->acaps | fw_fc | fw_fec | fw_mdi; + acaps = lc->acaps | fw_fc | fw_fec | fw_mdi; } /* Some Requested Port Capabilities are trivially wrong if they exceed @@ -4173,15 +4185,50 @@ int t4_link_l1cfg_core(struct adapter *adapter, unsigned int mbox, * we need to exclude this from this check in order to maintain * compatibility ... */ - if ((rcap & ~lc->pcaps) & ~FW_PORT_CAP32_FORCE_PAUSE) { - dev_err(adapter->pdev_dev, - "Requested Port Capabilities %#x exceed Physical Port Capabilities %#x\n", - rcap, lc->pcaps); + if ((acaps & ~lc->pcaps) & ~FW_PORT_CAP32_FORCE_PAUSE) { + dev_err(adapter->pdev_dev, "Requested Port Capabilities %#x exceed Physical Port Capabilities %#x\n", + acaps, lc->pcaps); + return -EINVAL; + } + + return acaps; +} + +/** + * t4_link_l1cfg_core - apply link configuration to MAC/PHY + * @adapter: the adapter + * @mbox: the Firmware Mailbox to use + * @port: the Port ID + * @lc: the Port's Link Configuration + * @sleep_ok: if true we may sleep while awaiting command completion + * @timeout: time to wait for command to finish before timing out + * (negative implies @sleep_ok=false) + * + * Set up a port's MAC and PHY according to a desired link configuration. + * - If the PHY can auto-negotiate first decide what to advertise, then + * enable/disable auto-negotiation as desired, and reset. + * - If the PHY does not auto-negotiate just reset it. + * - If auto-negotiation is off set the MAC to the proper speed/duplex/FC, + * otherwise do it later based on the outcome of auto-negotiation. + */ +int t4_link_l1cfg_core(struct adapter *adapter, unsigned int mbox, + unsigned int port, struct link_config *lc, + u8 sleep_ok, int timeout) +{ + unsigned int fw_caps = adapter->params.fw_caps_support; + struct fw_port_cmd cmd; + fw_port_cap32_t rcap; + int ret; + + if (!(lc->pcaps & FW_PORT_CAP32_ANEG) && + lc->autoneg == AUTONEG_ENABLE) { return -EINVAL; } - /* And send that on to the Firmware ... + /* Compute our Requested Port Capabilities and send that on to the + * Firmware. */ + rcap = t4_link_acaps(adapter, port, lc); memset(&cmd, 0, sizeof(cmd)); cmd.op_to_portid = cpu_to_be32(FW_CMD_OP_V(FW_PORT_CMD) | FW_CMD_REQUEST_F | FW_CMD_EXEC_F | @@ -4211,7 +4258,7 @@ int t4_link_l1cfg_core(struct adapter *adapter, unsigned int mbox, rcap, -ret); return ret; } - return ret; + return 0; } /** diff --git a/drivers/net/ethernet/chelsio/cxgb4/t4fw_version.h b/drivers/net/ethernet/chelsio/cxgb4/t4fw_version.h index 9125ddd89dd123e6b7f08c0b20e68ddf53512690..a02b1dff403ecaef348c41c26e895597373d4483 100644 --- a/drivers/net/ethernet/chelsio/cxgb4/t4fw_version.h +++ b/drivers/net/ethernet/chelsio/cxgb4/t4fw_version.h @@ -36,8 +36,8 @@ #define __T4FW_VERSION_H__ #define T4FW_VERSION_MAJOR 0x01 -#define T4FW_VERSION_MINOR 0x16 -#define T4FW_VERSION_MICRO 0x09 +#define T4FW_VERSION_MINOR 0x17 +#define T4FW_VERSION_MICRO 0x03 #define T4FW_VERSION_BUILD 0x00 #define T4FW_MIN_VERSION_MAJOR 0x01 @@ -45,8 +45,8 @@ #define T4FW_MIN_VERSION_MICRO 0x00 #define T5FW_VERSION_MAJOR 0x01 -#define T5FW_VERSION_MINOR 0x16 -#define T5FW_VERSION_MICRO 0x09 +#define T5FW_VERSION_MINOR 0x17 +#define T5FW_VERSION_MICRO 0x03 #define T5FW_VERSION_BUILD 0x00 #define T5FW_MIN_VERSION_MAJOR 0x00 @@ -54,8 +54,8 @@ #define T5FW_MIN_VERSION_MICRO 0x00 #define T6FW_VERSION_MAJOR 0x01 -#define T6FW_VERSION_MINOR 0x16 -#define T6FW_VERSION_MICRO 0x09 +#define T6FW_VERSION_MINOR 0x17 +#define T6FW_VERSION_MICRO 0x03 #define T6FW_VERSION_BUILD 0x00 #define T6FW_MIN_VERSION_MAJOR 0x00 diff --git a/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c b/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c index adc4d481815ba90c228ef851bcd0d71408c767fb..6d4cf3d0b2f09099048f00c83f10a66e664f8982 100644 --- a/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c +++ b/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c @@ -518,8 +518,8 @@ static int fwevtq_handler(struct sge_rspq *rspq, const __be64 *rsp, break; } cpl = (void *)p; - /*FALLTHROUGH*/ } + /* Fall through */ case CPL_SGE_EGR_UPDATE: { /* @@ -1479,22 +1479,6 @@ static int cxgb4vf_get_link_ksettings(struct net_device *dev, base->duplex = DUPLEX_UNKNOWN; } - if (pi->link_cfg.fc & PAUSE_RX) { - if (pi->link_cfg.fc & PAUSE_TX) { - ethtool_link_ksettings_add_link_mode(link_ksettings, - advertising, - Pause); - } else { - ethtool_link_ksettings_add_link_mode(link_ksettings, - advertising, - Asym_Pause); - } - } else if (pi->link_cfg.fc & PAUSE_TX) { - ethtool_link_ksettings_add_link_mode(link_ksettings, - advertising, - Asym_Pause); - } - base->autoneg = pi->link_cfg.autoneg; if (pi->link_cfg.pcaps & FW_PORT_CAP32_ANEG) ethtool_link_ksettings_add_link_mode(link_ksettings, diff --git a/drivers/net/ethernet/chelsio/cxgb4vf/t4vf_hw.c b/drivers/net/ethernet/chelsio/cxgb4vf/t4vf_hw.c index 84dff74ca9cd8973c3fcd816a28ff9eb9a0614ad..8a389d617a238048f268be38771e37c808dd84e7 100644 --- a/drivers/net/ethernet/chelsio/cxgb4vf/t4vf_hw.c +++ b/drivers/net/ethernet/chelsio/cxgb4vf/t4vf_hw.c @@ -313,7 +313,17 @@ int t4vf_wr_mbox_core(struct adapter *adapter, const void *cmd, int size, return ret; } +/* In the Physical Function Driver Common Code, the ADVERT_MASK is used to + * mask out bits in the Advertised Port Capabilities which are managed via + * separate controls, like Pause Frames and Forward Error Correction. In the + * Virtual Function Common Code, since we never perform L1 Configuration on + * the Link, the only things we really need to filter out are things which + * we decode and report separately like Speed. + */ #define ADVERT_MASK (FW_PORT_CAP32_SPEED_V(FW_PORT_CAP32_SPEED_M) | \ + FW_PORT_CAP32_802_3_PAUSE | \ + FW_PORT_CAP32_802_3_ASM_DIR | \ + FW_PORT_CAP32_FEC_V(FW_PORT_CAP32_FEC_M) | \ FW_PORT_CAP32_ANEG) /** diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c index 733d9172425bf46398714c8a003c99fdfbd91f67..acb2856936d2027babd1c06b02896a91b4d3d2e5 100644 --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -897,7 +897,7 @@ static netdev_tx_t enic_hard_start_xmit(struct sk_buff *skb, if (vnic_wq_desc_avail(wq) < MAX_SKB_FRAGS + ENIC_DESC_MAX_SPLITS) netif_tx_stop_queue(txq); skb_tx_timestamp(skb); - if (!skb->xmit_more || netif_xmit_stopped(txq)) + if (!netdev_xmit_more() || netif_xmit_stopped(txq)) vnic_wq_doorbell(wq); spin_unlock(&enic->wq_lock[txq_map]); diff --git a/drivers/net/ethernet/cortina/gemini.c b/drivers/net/ethernet/cortina/gemini.c index 949103db8a8ad75afab234f879acddc7e8c4a515..9003eb6716cd8621840bbbc75e0ba0540dab7e0a 100644 --- a/drivers/net/ethernet/cortina/gemini.c +++ b/drivers/net/ethernet/cortina/gemini.c @@ -1235,8 +1235,6 @@ static int gmac_start_xmit(struct sk_buff *skb, struct net_device *netdev) int txq_num, nfrags; union dma_rwptr rw; - SKB_FRAG_ASSERT(skb); - if (skb->len >= 0x10000) goto out_drop_free; diff --git a/drivers/net/ethernet/davicom/dm9000.c b/drivers/net/ethernet/davicom/dm9000.c index c2586f44c29dc8a42eab98369ac4f52260c82e1a..953ee5616801113de773a3f6fc60a4773b92e133 100644 --- a/drivers/net/ethernet/davicom/dm9000.c +++ b/drivers/net/ethernet/davicom/dm9000.c @@ -1412,7 +1412,7 @@ static struct dm9000_plat_data *dm9000_parse_dt(struct device *dev) pdata->flags |= DM9000_PLATF_NO_EEPROM; mac_addr = of_get_mac_address(np); - if (mac_addr) + if (!IS_ERR(mac_addr)) memcpy(pdata->dev_addr, mac_addr, sizeof(pdata->dev_addr)); return pdata; diff --git a/drivers/net/ethernet/emulex/benet/be_main.c b/drivers/net/ethernet/emulex/benet/be_main.c index 3c7c04406a2bfa55d1f3c67e0019d488c7dff4a8..e2f9fbced17457b481a3e9ac2f1a9bc358a66b59 100644 --- a/drivers/net/ethernet/emulex/benet/be_main.c +++ b/drivers/net/ethernet/emulex/benet/be_main.c @@ -1376,7 +1376,7 @@ static netdev_tx_t be_xmit(struct sk_buff *skb, struct net_device *netdev) u16 q_idx = skb_get_queue_mapping(skb); struct be_tx_obj *txo = &adapter->tx_obj[q_idx]; struct be_wrb_params wrb_params = { 0 }; - bool flush = !skb->xmit_more; + bool flush = !netdev_xmit_more(); u16 wrb_cnt; skb = be_xmit_workarounds(adapter, skb, &wrb_params); diff --git a/drivers/net/ethernet/ethoc.c b/drivers/net/ethernet/ethoc.c index 0f3e7f21c6fa59e363fc2ee779f2fd1943022642..71da0490521b834eff62bcdf4216fd128c61ef9c 100644 --- a/drivers/net/ethernet/ethoc.c +++ b/drivers/net/ethernet/ethoc.c @@ -1153,7 +1153,7 @@ static int ethoc_probe(struct platform_device *pdev) const void *mac; mac = of_get_mac_address(pdev->dev.of_node); - if (mac) + if (!IS_ERR(mac)) ether_addr_copy(netdev->dev_addr, mac); priv->phy_id = -1; } diff --git a/drivers/net/ethernet/ezchip/nps_enet.c b/drivers/net/ethernet/ezchip/nps_enet.c index 659f1ad37e96a823a4ee3cba04be905d5c9c5dee..b4ce261550873a0b47c9ce0365862f126c20bd3f 100644 --- a/drivers/net/ethernet/ezchip/nps_enet.c +++ b/drivers/net/ethernet/ezchip/nps_enet.c @@ -616,7 +616,7 @@ static s32 nps_enet_probe(struct platform_device *pdev) /* set kernel MAC address to dev */ mac_addr = of_get_mac_address(dev->of_node); - if (mac_addr) + if (!IS_ERR(mac_addr)) ether_addr_copy(ndev->dev_addr, mac_addr); else eth_hw_addr_random(ndev); diff --git a/drivers/net/ethernet/freescale/dpaa/dpaa_eth.c b/drivers/net/ethernet/freescale/dpaa/dpaa_eth.c index dfebc30c4841211990b7888d61b01780e31f44aa..d3f2408dc9e8003d8e823b3402b771c87b02fa22 100644 --- a/drivers/net/ethernet/freescale/dpaa/dpaa_eth.c +++ b/drivers/net/ethernet/freescale/dpaa/dpaa_eth.c @@ -1648,7 +1648,7 @@ static struct sk_buff *dpaa_cleanup_tx_fd(const struct dpaa_priv *priv, qm_sg_entry_get_len(&sgt[0]), dma_dir); /* remaining pages were mapped with skb_frag_dma_map() */ - for (i = 1; i < nr_frags; i++) { + for (i = 1; i <= nr_frags; i++) { WARN_ON(qm_sg_entry_is_ext(&sgt[i])); dma_unmap_page(dev, qm_sg_addr(&sgt[i]), diff --git a/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c b/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c index dc339dc1adb21c30224fbce6eb0d60fd861c9388..63b1ecc18c26f7833988394835ee482a844ff6b8 100644 --- a/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c +++ b/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c @@ -435,7 +435,7 @@ static void dpaa2_eth_rx(struct dpaa2_eth_priv *priv, percpu_stats->rx_packets++; percpu_stats->rx_bytes += dpaa2_fd_get_len(fd); - napi_gro_receive(&ch->napi, skb); + list_add_tail(&skb->list, ch->rx_list); return; @@ -1113,12 +1113,16 @@ static int dpaa2_eth_poll(struct napi_struct *napi, int budget) struct dpaa2_eth_fq *fq, *txc_fq = NULL; struct netdev_queue *nq; int store_cleaned, work_done; + struct list_head rx_list; int err; ch = container_of(napi, struct dpaa2_eth_channel, napi); ch->xdp.res = 0; priv = ch->priv; + INIT_LIST_HEAD(&rx_list); + ch->rx_list = &rx_list; + do { err = pull_channel(ch); if (unlikely(err)) @@ -1162,6 +1166,8 @@ static int dpaa2_eth_poll(struct napi_struct *napi, int budget) work_done = max(rx_cleaned, 1); out: + netif_receive_skb_list(ch->rx_list); + if (txc_fq && txc_fq->dq_frames) { nq = netdev_get_tx_queue(priv->net_dev, txc_fq->flowid); netdev_tx_completed_queue(nq, txc_fq->dq_frames, @@ -2565,10 +2571,12 @@ static const struct dpaa2_eth_dist_fields dist_fields[] = { .rxnfc_field = RXH_L2DA, .cls_prot = NET_PROT_ETH, .cls_field = NH_FLD_ETH_DA, + .id = DPAA2_ETH_DIST_ETHDST, .size = 6, }, { .cls_prot = NET_PROT_ETH, .cls_field = NH_FLD_ETH_SA, + .id = DPAA2_ETH_DIST_ETHSRC, .size = 6, }, { /* This is the last ethertype field parsed: @@ -2577,28 +2585,33 @@ static const struct dpaa2_eth_dist_fields dist_fields[] = { */ .cls_prot = NET_PROT_ETH, .cls_field = NH_FLD_ETH_TYPE, + .id = DPAA2_ETH_DIST_ETHTYPE, .size = 2, }, { /* VLAN header */ .rxnfc_field = RXH_VLAN, .cls_prot = NET_PROT_VLAN, .cls_field = NH_FLD_VLAN_TCI, + .id = DPAA2_ETH_DIST_VLAN, .size = 2, }, { /* IP header */ .rxnfc_field = RXH_IP_SRC, .cls_prot = NET_PROT_IP, .cls_field = NH_FLD_IP_SRC, + .id = DPAA2_ETH_DIST_IPSRC, .size = 4, }, { .rxnfc_field = RXH_IP_DST, .cls_prot = NET_PROT_IP, .cls_field = NH_FLD_IP_DST, + .id = DPAA2_ETH_DIST_IPDST, .size = 4, }, { .rxnfc_field = RXH_L3_PROTO, .cls_prot = NET_PROT_IP, .cls_field = NH_FLD_IP_PROTO, + .id = DPAA2_ETH_DIST_IPPROTO, .size = 1, }, { /* Using UDP ports, this is functionally equivalent to raw @@ -2607,11 +2620,13 @@ static const struct dpaa2_eth_dist_fields dist_fields[] = { .rxnfc_field = RXH_L4_B_0_1, .cls_prot = NET_PROT_UDP, .cls_field = NH_FLD_UDP_PORT_SRC, + .id = DPAA2_ETH_DIST_L4SRC, .size = 2, }, { .rxnfc_field = RXH_L4_B_2_3, .cls_prot = NET_PROT_UDP, .cls_field = NH_FLD_UDP_PORT_DST, + .id = DPAA2_ETH_DIST_L4DST, .size = 2, }, }; @@ -2677,12 +2692,15 @@ static int config_cls_key(struct dpaa2_eth_priv *priv, dma_addr_t key) } /* Size of the Rx flow classification key */ -int dpaa2_eth_cls_key_size(void) +int dpaa2_eth_cls_key_size(u64 fields) { int i, size = 0; - for (i = 0; i < ARRAY_SIZE(dist_fields); i++) + for (i = 0; i < ARRAY_SIZE(dist_fields); i++) { + if (!(fields & dist_fields[i].id)) + continue; size += dist_fields[i].size; + } return size; } @@ -2703,6 +2721,24 @@ int dpaa2_eth_cls_fld_off(int prot, int field) return 0; } +/* Prune unused fields from the classification rule. + * Used when masking is not supported + */ +void dpaa2_eth_cls_trim_rule(void *key_mem, u64 fields) +{ + int off = 0, new_off = 0; + int i, size; + + for (i = 0; i < ARRAY_SIZE(dist_fields); i++) { + size = dist_fields[i].size; + if (dist_fields[i].id & fields) { + memcpy(key_mem + new_off, key_mem + off, size); + new_off += size; + } + off += size; + } +} + /* Set Rx distribution (hash or flow classification) key * flags is a combination of RXH_ bits */ @@ -2724,14 +2760,13 @@ static int dpaa2_eth_set_dist_key(struct net_device *net_dev, struct dpkg_extract *key = &cls_cfg.extracts[cls_cfg.num_extracts]; - /* For Rx hashing key we set only the selected fields. - * For Rx flow classification key we set all supported fields + /* For both Rx hashing and classification keys + * we set only the selected fields. */ - if (type == DPAA2_ETH_RX_DIST_HASH) { - if (!(flags & dist_fields[i].rxnfc_field)) - continue; + if (!(flags & dist_fields[i].id)) + continue; + if (type == DPAA2_ETH_RX_DIST_HASH) rx_hash_fields |= dist_fields[i].rxnfc_field; - } if (cls_cfg.num_extracts >= DPKG_MAX_NUM_OF_EXTRACTS) { dev_err(dev, "error adding key extraction rule, too many rules?\n"); @@ -2786,16 +2821,28 @@ free_key: int dpaa2_eth_set_hash(struct net_device *net_dev, u64 flags) { struct dpaa2_eth_priv *priv = netdev_priv(net_dev); + u64 key = 0; + int i; if (!dpaa2_eth_hash_enabled(priv)) return -EOPNOTSUPP; - return dpaa2_eth_set_dist_key(net_dev, DPAA2_ETH_RX_DIST_HASH, flags); + for (i = 0; i < ARRAY_SIZE(dist_fields); i++) + if (dist_fields[i].rxnfc_field & flags) + key |= dist_fields[i].id; + + return dpaa2_eth_set_dist_key(net_dev, DPAA2_ETH_RX_DIST_HASH, key); +} + +int dpaa2_eth_set_cls(struct net_device *net_dev, u64 flags) +{ + return dpaa2_eth_set_dist_key(net_dev, DPAA2_ETH_RX_DIST_CLS, flags); } -static int dpaa2_eth_set_cls(struct dpaa2_eth_priv *priv) +static int dpaa2_eth_set_default_cls(struct dpaa2_eth_priv *priv) { struct device *dev = priv->net_dev->dev.parent; + int err; /* Check if we actually support Rx flow classification */ if (dpaa2_eth_has_legacy_dist(priv)) { @@ -2803,8 +2850,7 @@ static int dpaa2_eth_set_cls(struct dpaa2_eth_priv *priv) return -EOPNOTSUPP; } - if (priv->dpni_attrs.options & DPNI_OPT_NO_FS || - !(priv->dpni_attrs.options & DPNI_OPT_HAS_KEY_MASKING)) { + if (!dpaa2_eth_fs_enabled(priv)) { dev_dbg(dev, "Rx cls disabled in DPNI options\n"); return -EOPNOTSUPP; } @@ -2814,9 +2860,21 @@ static int dpaa2_eth_set_cls(struct dpaa2_eth_priv *priv) return -EOPNOTSUPP; } + /* If there is no support for masking in the classification table, + * we don't set a default key, as it will depend on the rules + * added by the user at runtime. + */ + if (!dpaa2_eth_fs_mask_enabled(priv)) + goto out; + + err = dpaa2_eth_set_cls(priv->net_dev, DPAA2_ETH_DIST_ALL); + if (err) + return err; + +out: priv->rx_cls_enabled = 1; - return dpaa2_eth_set_dist_key(priv->net_dev, DPAA2_ETH_RX_DIST_CLS, 0); + return 0; } /* Bind the DPNI to its needed objects and resources: buffer pool, DPIOs, @@ -2851,7 +2909,7 @@ static int bind_dpni(struct dpaa2_eth_priv *priv) /* Configure the flow classification key; it includes all * supported header fields and cannot be modified at runtime */ - err = dpaa2_eth_set_cls(priv); + err = dpaa2_eth_set_default_cls(priv); if (err && err != -EOPNOTSUPP) dev_err(dev, "Failed to configure Rx classification key\n"); diff --git a/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.h b/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.h index 7879622aa3e6d8bdf348dc639235d5ca132afe74..5fb8f5c0dc9f8572bc7145706d29cdfba74e001f 100644 --- a/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.h +++ b/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.h @@ -334,6 +334,7 @@ struct dpaa2_eth_channel { struct dpaa2_eth_ch_stats stats; struct dpaa2_eth_ch_xdp xdp; struct xdp_rxq_info xdp_rxq; + struct list_head *rx_list; }; struct dpaa2_eth_dist_fields { @@ -341,6 +342,7 @@ struct dpaa2_eth_dist_fields { enum net_prot cls_prot; int cls_field; int size; + u64 id; }; struct dpaa2_eth_cls_rule { @@ -393,6 +395,7 @@ struct dpaa2_eth_priv { /* enabled ethtool hashing bits */ u64 rx_hash_fields; + u64 rx_cls_fields; struct dpaa2_eth_cls_rule *cls_rules; u8 rx_cls_enabled; struct bpf_prog *xdp_prog; @@ -436,6 +439,12 @@ static inline int dpaa2_eth_cmp_dpni_ver(struct dpaa2_eth_priv *priv, (dpaa2_eth_cmp_dpni_ver((priv), DPNI_RX_DIST_KEY_VER_MAJOR, \ DPNI_RX_DIST_KEY_VER_MINOR) < 0) +#define dpaa2_eth_fs_enabled(priv) \ + (!((priv)->dpni_attrs.options & DPNI_OPT_NO_FS)) + +#define dpaa2_eth_fs_mask_enabled(priv) \ + ((priv)->dpni_attrs.options & DPNI_OPT_HAS_KEY_MASKING) + #define dpaa2_eth_fs_count(priv) \ ((priv)->dpni_attrs.fs_entries) @@ -448,6 +457,18 @@ enum dpaa2_eth_rx_dist { DPAA2_ETH_RX_DIST_CLS }; +/* Unique IDs for the supported Rx classification header fields */ +#define DPAA2_ETH_DIST_ETHDST BIT(0) +#define DPAA2_ETH_DIST_ETHSRC BIT(1) +#define DPAA2_ETH_DIST_ETHTYPE BIT(2) +#define DPAA2_ETH_DIST_VLAN BIT(3) +#define DPAA2_ETH_DIST_IPSRC BIT(4) +#define DPAA2_ETH_DIST_IPDST BIT(5) +#define DPAA2_ETH_DIST_IPPROTO BIT(6) +#define DPAA2_ETH_DIST_L4SRC BIT(7) +#define DPAA2_ETH_DIST_L4DST BIT(8) +#define DPAA2_ETH_DIST_ALL (~0U) + static inline unsigned int dpaa2_eth_needed_headroom(struct dpaa2_eth_priv *priv, struct sk_buff *skb) @@ -482,7 +503,9 @@ static inline unsigned int dpaa2_eth_rx_head_room(struct dpaa2_eth_priv *priv) } int dpaa2_eth_set_hash(struct net_device *net_dev, u64 flags); -int dpaa2_eth_cls_key_size(void); +int dpaa2_eth_set_cls(struct net_device *net_dev, u64 key); +int dpaa2_eth_cls_key_size(u64 key); int dpaa2_eth_cls_fld_off(int prot, int field); +void dpaa2_eth_cls_trim_rule(void *key_mem, u64 fields); #endif /* __DPAA2_H */ diff --git a/drivers/net/ethernet/freescale/dpaa2/dpaa2-ethtool.c b/drivers/net/ethernet/freescale/dpaa2/dpaa2-ethtool.c index 591dfcf76adbced88e4fd11e4ddcfcd532c9a46e..76bd8d2872cc8e2899e73a71d4f99822d60ed23a 100644 --- a/drivers/net/ethernet/freescale/dpaa2/dpaa2-ethtool.c +++ b/drivers/net/ethernet/freescale/dpaa2/dpaa2-ethtool.c @@ -264,7 +264,7 @@ static void dpaa2_eth_get_ethtool_stats(struct net_device *net_dev, } static int prep_eth_rule(struct ethhdr *eth_value, struct ethhdr *eth_mask, - void *key, void *mask) + void *key, void *mask, u64 *fields) { int off; @@ -272,18 +272,21 @@ static int prep_eth_rule(struct ethhdr *eth_value, struct ethhdr *eth_mask, off = dpaa2_eth_cls_fld_off(NET_PROT_ETH, NH_FLD_ETH_TYPE); *(__be16 *)(key + off) = eth_value->h_proto; *(__be16 *)(mask + off) = eth_mask->h_proto; + *fields |= DPAA2_ETH_DIST_ETHTYPE; } if (!is_zero_ether_addr(eth_mask->h_source)) { off = dpaa2_eth_cls_fld_off(NET_PROT_ETH, NH_FLD_ETH_SA); ether_addr_copy(key + off, eth_value->h_source); ether_addr_copy(mask + off, eth_mask->h_source); + *fields |= DPAA2_ETH_DIST_ETHSRC; } if (!is_zero_ether_addr(eth_mask->h_dest)) { off = dpaa2_eth_cls_fld_off(NET_PROT_ETH, NH_FLD_ETH_DA); ether_addr_copy(key + off, eth_value->h_dest); ether_addr_copy(mask + off, eth_mask->h_dest); + *fields |= DPAA2_ETH_DIST_ETHDST; } return 0; @@ -291,7 +294,7 @@ static int prep_eth_rule(struct ethhdr *eth_value, struct ethhdr *eth_mask, static int prep_uip_rule(struct ethtool_usrip4_spec *uip_value, struct ethtool_usrip4_spec *uip_mask, - void *key, void *mask) + void *key, void *mask, u64 *fields) { int off; u32 tmp_value, tmp_mask; @@ -303,18 +306,21 @@ static int prep_uip_rule(struct ethtool_usrip4_spec *uip_value, off = dpaa2_eth_cls_fld_off(NET_PROT_IP, NH_FLD_IP_SRC); *(__be32 *)(key + off) = uip_value->ip4src; *(__be32 *)(mask + off) = uip_mask->ip4src; + *fields |= DPAA2_ETH_DIST_IPSRC; } if (uip_mask->ip4dst) { off = dpaa2_eth_cls_fld_off(NET_PROT_IP, NH_FLD_IP_DST); *(__be32 *)(key + off) = uip_value->ip4dst; *(__be32 *)(mask + off) = uip_mask->ip4dst; + *fields |= DPAA2_ETH_DIST_IPDST; } if (uip_mask->proto) { off = dpaa2_eth_cls_fld_off(NET_PROT_IP, NH_FLD_IP_PROTO); *(u8 *)(key + off) = uip_value->proto; *(u8 *)(mask + off) = uip_mask->proto; + *fields |= DPAA2_ETH_DIST_IPPROTO; } if (uip_mask->l4_4_bytes) { @@ -324,23 +330,26 @@ static int prep_uip_rule(struct ethtool_usrip4_spec *uip_value, off = dpaa2_eth_cls_fld_off(NET_PROT_UDP, NH_FLD_UDP_PORT_SRC); *(__be16 *)(key + off) = htons(tmp_value >> 16); *(__be16 *)(mask + off) = htons(tmp_mask >> 16); + *fields |= DPAA2_ETH_DIST_L4SRC; off = dpaa2_eth_cls_fld_off(NET_PROT_UDP, NH_FLD_UDP_PORT_DST); *(__be16 *)(key + off) = htons(tmp_value & 0xFFFF); *(__be16 *)(mask + off) = htons(tmp_mask & 0xFFFF); + *fields |= DPAA2_ETH_DIST_L4DST; } /* Only apply the rule for IPv4 frames */ off = dpaa2_eth_cls_fld_off(NET_PROT_ETH, NH_FLD_ETH_TYPE); *(__be16 *)(key + off) = htons(ETH_P_IP); *(__be16 *)(mask + off) = htons(0xFFFF); + *fields |= DPAA2_ETH_DIST_ETHTYPE; return 0; } static int prep_l4_rule(struct ethtool_tcpip4_spec *l4_value, struct ethtool_tcpip4_spec *l4_mask, - void *key, void *mask, u8 l4_proto) + void *key, void *mask, u8 l4_proto, u64 *fields) { int off; @@ -351,41 +360,47 @@ static int prep_l4_rule(struct ethtool_tcpip4_spec *l4_value, off = dpaa2_eth_cls_fld_off(NET_PROT_IP, NH_FLD_IP_SRC); *(__be32 *)(key + off) = l4_value->ip4src; *(__be32 *)(mask + off) = l4_mask->ip4src; + *fields |= DPAA2_ETH_DIST_IPSRC; } if (l4_mask->ip4dst) { off = dpaa2_eth_cls_fld_off(NET_PROT_IP, NH_FLD_IP_DST); *(__be32 *)(key + off) = l4_value->ip4dst; *(__be32 *)(mask + off) = l4_mask->ip4dst; + *fields |= DPAA2_ETH_DIST_IPDST; } if (l4_mask->psrc) { off = dpaa2_eth_cls_fld_off(NET_PROT_UDP, NH_FLD_UDP_PORT_SRC); *(__be16 *)(key + off) = l4_value->psrc; *(__be16 *)(mask + off) = l4_mask->psrc; + *fields |= DPAA2_ETH_DIST_L4SRC; } if (l4_mask->pdst) { off = dpaa2_eth_cls_fld_off(NET_PROT_UDP, NH_FLD_UDP_PORT_DST); *(__be16 *)(key + off) = l4_value->pdst; *(__be16 *)(mask + off) = l4_mask->pdst; + *fields |= DPAA2_ETH_DIST_L4DST; } /* Only apply the rule for IPv4 frames with the specified L4 proto */ off = dpaa2_eth_cls_fld_off(NET_PROT_ETH, NH_FLD_ETH_TYPE); *(__be16 *)(key + off) = htons(ETH_P_IP); *(__be16 *)(mask + off) = htons(0xFFFF); + *fields |= DPAA2_ETH_DIST_ETHTYPE; off = dpaa2_eth_cls_fld_off(NET_PROT_IP, NH_FLD_IP_PROTO); *(u8 *)(key + off) = l4_proto; *(u8 *)(mask + off) = 0xFF; + *fields |= DPAA2_ETH_DIST_IPPROTO; return 0; } static int prep_ext_rule(struct ethtool_flow_ext *ext_value, struct ethtool_flow_ext *ext_mask, - void *key, void *mask) + void *key, void *mask, u64 *fields) { int off; @@ -396,6 +411,7 @@ static int prep_ext_rule(struct ethtool_flow_ext *ext_value, off = dpaa2_eth_cls_fld_off(NET_PROT_VLAN, NH_FLD_VLAN_TCI); *(__be16 *)(key + off) = ext_value->vlan_tci; *(__be16 *)(mask + off) = ext_mask->vlan_tci; + *fields |= DPAA2_ETH_DIST_VLAN; } return 0; @@ -403,7 +419,7 @@ static int prep_ext_rule(struct ethtool_flow_ext *ext_value, static int prep_mac_ext_rule(struct ethtool_flow_ext *ext_value, struct ethtool_flow_ext *ext_mask, - void *key, void *mask) + void *key, void *mask, u64 *fields) { int off; @@ -411,36 +427,38 @@ static int prep_mac_ext_rule(struct ethtool_flow_ext *ext_value, off = dpaa2_eth_cls_fld_off(NET_PROT_ETH, NH_FLD_ETH_DA); ether_addr_copy(key + off, ext_value->h_dest); ether_addr_copy(mask + off, ext_mask->h_dest); + *fields |= DPAA2_ETH_DIST_ETHDST; } return 0; } -static int prep_cls_rule(struct ethtool_rx_flow_spec *fs, void *key, void *mask) +static int prep_cls_rule(struct ethtool_rx_flow_spec *fs, void *key, void *mask, + u64 *fields) { int err; switch (fs->flow_type & 0xFF) { case ETHER_FLOW: err = prep_eth_rule(&fs->h_u.ether_spec, &fs->m_u.ether_spec, - key, mask); + key, mask, fields); break; case IP_USER_FLOW: err = prep_uip_rule(&fs->h_u.usr_ip4_spec, - &fs->m_u.usr_ip4_spec, key, mask); + &fs->m_u.usr_ip4_spec, key, mask, fields); break; case TCP_V4_FLOW: err = prep_l4_rule(&fs->h_u.tcp_ip4_spec, &fs->m_u.tcp_ip4_spec, - key, mask, IPPROTO_TCP); + key, mask, IPPROTO_TCP, fields); break; case UDP_V4_FLOW: err = prep_l4_rule(&fs->h_u.udp_ip4_spec, &fs->m_u.udp_ip4_spec, - key, mask, IPPROTO_UDP); + key, mask, IPPROTO_UDP, fields); break; case SCTP_V4_FLOW: err = prep_l4_rule(&fs->h_u.sctp_ip4_spec, &fs->m_u.sctp_ip4_spec, key, mask, - IPPROTO_SCTP); + IPPROTO_SCTP, fields); break; default: return -EOPNOTSUPP; @@ -450,13 +468,14 @@ static int prep_cls_rule(struct ethtool_rx_flow_spec *fs, void *key, void *mask) return err; if (fs->flow_type & FLOW_EXT) { - err = prep_ext_rule(&fs->h_ext, &fs->m_ext, key, mask); + err = prep_ext_rule(&fs->h_ext, &fs->m_ext, key, mask, fields); if (err) return err; } if (fs->flow_type & FLOW_MAC_EXT) { - err = prep_mac_ext_rule(&fs->h_ext, &fs->m_ext, key, mask); + err = prep_mac_ext_rule(&fs->h_ext, &fs->m_ext, key, mask, + fields); if (err) return err; } @@ -473,6 +492,7 @@ static int do_cls_rule(struct net_device *net_dev, struct dpni_rule_cfg rule_cfg = { 0 }; struct dpni_fs_action_cfg fs_act = { 0 }; dma_addr_t key_iova; + u64 fields = 0; void *key_buf; int err; @@ -480,7 +500,7 @@ static int do_cls_rule(struct net_device *net_dev, fs->ring_cookie >= dpaa2_eth_queue_count(priv)) return -EINVAL; - rule_cfg.key_size = dpaa2_eth_cls_key_size(); + rule_cfg.key_size = dpaa2_eth_cls_key_size(DPAA2_ETH_DIST_ALL); /* allocate twice the key size, for the actual key and for mask */ key_buf = kzalloc(rule_cfg.key_size * 2, GFP_KERNEL); @@ -488,10 +508,36 @@ static int do_cls_rule(struct net_device *net_dev, return -ENOMEM; /* Fill the key and mask memory areas */ - err = prep_cls_rule(fs, key_buf, key_buf + rule_cfg.key_size); + err = prep_cls_rule(fs, key_buf, key_buf + rule_cfg.key_size, &fields); if (err) goto free_mem; + if (!dpaa2_eth_fs_mask_enabled(priv)) { + /* Masking allows us to configure a maximal key during init and + * use it for all flow steering rules. Without it, we include + * in the key only the fields actually used, so we need to + * extract the others from the final key buffer. + * + * Program the FS key if needed, or return error if previously + * set key can't be used for the current rule. User needs to + * delete existing rules in this case to allow for the new one. + */ + if (!priv->rx_cls_fields) { + err = dpaa2_eth_set_cls(net_dev, fields); + if (err) + goto free_mem; + + priv->rx_cls_fields = fields; + } else if (priv->rx_cls_fields != fields) { + netdev_err(net_dev, "No support for multiple FS keys, need to delete existing rules\n"); + err = -EOPNOTSUPP; + goto free_mem; + } + + dpaa2_eth_cls_trim_rule(key_buf, fields); + rule_cfg.key_size = dpaa2_eth_cls_key_size(fields); + } + key_iova = dma_map_single(dev, key_buf, rule_cfg.key_size * 2, DMA_TO_DEVICE); if (dma_mapping_error(dev, key_iova)) { @@ -500,7 +546,8 @@ static int do_cls_rule(struct net_device *net_dev, } rule_cfg.key_iova = key_iova; - rule_cfg.mask_iova = key_iova + rule_cfg.key_size; + if (dpaa2_eth_fs_mask_enabled(priv)) + rule_cfg.mask_iova = key_iova + rule_cfg.key_size; if (add) { if (fs->ring_cookie == RX_CLS_FLOW_DISC) @@ -522,6 +569,17 @@ free_mem: return err; } +static int num_rules(struct dpaa2_eth_priv *priv) +{ + int i, rules = 0; + + for (i = 0; i < dpaa2_eth_fs_count(priv); i++) + if (priv->cls_rules[i].in_use) + rules++; + + return rules; +} + static int update_cls_rule(struct net_device *net_dev, struct ethtool_rx_flow_spec *new_fs, int location) @@ -545,6 +603,9 @@ static int update_cls_rule(struct net_device *net_dev, return err; rule->in_use = 0; + + if (!dpaa2_eth_fs_mask_enabled(priv) && !num_rules(priv)) + priv->rx_cls_fields = 0; } /* If no new entry to add, return here */ @@ -581,9 +642,7 @@ static int dpaa2_eth_get_rxnfc(struct net_device *net_dev, break; case ETHTOOL_GRXCLSRLCNT: rxnfc->rule_cnt = 0; - for (i = 0; i < max_rules; i++) - if (priv->cls_rules[i].in_use) - rxnfc->rule_cnt++; + rxnfc->rule_cnt = num_rules(priv); rxnfc->data = max_rules; break; case ETHTOOL_GRXCLSRULE: diff --git a/drivers/net/ethernet/freescale/fec_main.c b/drivers/net/ethernet/freescale/fec_main.c index a96ad20ee4843e9cdd02c55a3e7286a51679efea..aa7d4e27c5d121497c8511dff2c300a81d351990 100644 --- a/drivers/net/ethernet/freescale/fec_main.c +++ b/drivers/net/ethernet/freescale/fec_main.c @@ -1655,7 +1655,7 @@ static void fec_get_mac(struct net_device *ndev) struct device_node *np = fep->pdev->dev.of_node; if (np) { const char *mac = of_get_mac_address(np); - if (mac) + if (!IS_ERR(mac)) iap = (unsigned char *) mac; } } diff --git a/drivers/net/ethernet/freescale/fec_mpc52xx.c b/drivers/net/ethernet/freescale/fec_mpc52xx.c index c1968b3ecec87cf8fcd807c4f34d0be4cc6e27e8..7b7e526869a7f8c04b262e3408581df99fc709fe 100644 --- a/drivers/net/ethernet/freescale/fec_mpc52xx.c +++ b/drivers/net/ethernet/freescale/fec_mpc52xx.c @@ -902,7 +902,7 @@ static int mpc52xx_fec_probe(struct platform_device *op) * First try to read MAC address from DT */ mac_addr = of_get_mac_address(np); - if (mac_addr) { + if (!IS_ERR(mac_addr)) { memcpy(ndev->dev_addr, mac_addr, ETH_ALEN); } else { struct mpc52xx_fec __iomem *fec = priv->fec; diff --git a/drivers/net/ethernet/freescale/fman/mac.c b/drivers/net/ethernet/freescale/fman/mac.c index 3c21486c6c8479186ba927f7f7f19a8c9f015314..9cd2c28d17df93390d6b57481d14eb56da6cc3db 100644 --- a/drivers/net/ethernet/freescale/fman/mac.c +++ b/drivers/net/ethernet/freescale/fman/mac.c @@ -724,7 +724,7 @@ static int mac_probe(struct platform_device *_of_dev) /* Get the MAC address */ mac_addr = of_get_mac_address(mac_node); - if (!mac_addr) { + if (IS_ERR(mac_addr)) { dev_err(dev, "of_get_mac_address(%pOF) failed\n", mac_node); err = -EINVAL; goto _return_of_get_parent; diff --git a/drivers/net/ethernet/freescale/fs_enet/fs_enet-main.c b/drivers/net/ethernet/freescale/fs_enet/fs_enet-main.c index 7c548ed535da5babe24c342243c82d737f257034..90ea7a115d0fb9a140d012551959bd943084873c 100644 --- a/drivers/net/ethernet/freescale/fs_enet/fs_enet-main.c +++ b/drivers/net/ethernet/freescale/fs_enet/fs_enet-main.c @@ -1014,7 +1014,7 @@ static int fs_enet_probe(struct platform_device *ofdev) spin_lock_init(&fep->tx_lock); mac_addr = of_get_mac_address(ofdev->dev.of_node); - if (mac_addr) + if (!IS_ERR(mac_addr)) memcpy(ndev->dev_addr, mac_addr, ETH_ALEN); ret = fep->ops->allocate_bd(ndev); diff --git a/drivers/net/ethernet/freescale/gianfar.c b/drivers/net/ethernet/freescale/gianfar.c index 45fcc96be90e06abb348843653313779918573f1..df13c693b03850b23c357b98997c3f29fff289cb 100644 --- a/drivers/net/ethernet/freescale/gianfar.c +++ b/drivers/net/ethernet/freescale/gianfar.c @@ -872,7 +872,7 @@ static int gfar_of_init(struct platform_device *ofdev, struct net_device **pdev) mac_addr = of_get_mac_address(np); - if (mac_addr) + if (!IS_ERR(mac_addr)) memcpy(dev->dev_addr, mac_addr, ETH_ALEN); if (model && !strcasecmp(model, "TSEC")) diff --git a/drivers/net/ethernet/freescale/ucc_geth.c b/drivers/net/ethernet/freescale/ucc_geth.c index eb3e65e8868f9f65047ab24e8094f56e307bb64b..216e99af2b5a1dc26d336240a76985a7d56c4c8c 100644 --- a/drivers/net/ethernet/freescale/ucc_geth.c +++ b/drivers/net/ethernet/freescale/ucc_geth.c @@ -3910,7 +3910,7 @@ static int ucc_geth_probe(struct platform_device* ofdev) } mac_addr = of_get_mac_address(np); - if (mac_addr) + if (!IS_ERR(mac_addr)) memcpy(dev->dev_addr, mac_addr, ETH_ALEN); ugeth->ug_info = ug_info; diff --git a/drivers/net/ethernet/freescale/ucc_geth_ethtool.c b/drivers/net/ethernet/freescale/ucc_geth_ethtool.c index 0beee2cc2ddd3164baf7211f49c3b5affef7a091..722b6de248164342af98f35104f0a2814e97e27e 100644 --- a/drivers/net/ethernet/freescale/ucc_geth_ethtool.c +++ b/drivers/net/ethernet/freescale/ucc_geth_ethtool.c @@ -252,14 +252,12 @@ uec_set_ringparam(struct net_device *netdev, return -EINVAL; } + if (netif_running(netdev)) + return -EBUSY; + ug_info->bdRingLenRx[queue] = ring->rx_pending; ug_info->bdRingLenTx[queue] = ring->tx_pending; - if (netif_running(netdev)) { - /* FIXME: restart automatically */ - netdev_info(netdev, "Please re-open the interface\n"); - } - return ret; } diff --git a/drivers/net/ethernet/hisilicon/hisi_femac.c b/drivers/net/ethernet/hisilicon/hisi_femac.c index 2c2808830e95723604b7356514ea64ca123fac3a..96c32ae320b0f128aef1084be052027a135bb5da 100644 --- a/drivers/net/ethernet/hisilicon/hisi_femac.c +++ b/drivers/net/ethernet/hisilicon/hisi_femac.c @@ -870,7 +870,7 @@ static int hisi_femac_drv_probe(struct platform_device *pdev) phy_modes(phy->interface)); mac_addr = of_get_mac_address(node); - if (mac_addr) + if (!IS_ERR(mac_addr)) ether_addr_copy(ndev->dev_addr, mac_addr); if (!is_valid_ether_addr(ndev->dev_addr)) { eth_hw_addr_random(ndev); diff --git a/drivers/net/ethernet/hisilicon/hix5hd2_gmac.c b/drivers/net/ethernet/hisilicon/hix5hd2_gmac.c index e5d853b7b454f4651e2dd2575c191559c4527cd1..b1cb58f0aaf64a45785fe04d2edc2733b289336f 100644 --- a/drivers/net/ethernet/hisilicon/hix5hd2_gmac.c +++ b/drivers/net/ethernet/hisilicon/hix5hd2_gmac.c @@ -1229,7 +1229,7 @@ static int hix5hd2_dev_probe(struct platform_device *pdev) } mac_addr = of_get_mac_address(node); - if (mac_addr) + if (!IS_ERR(mac_addr)) ether_addr_copy(ndev->dev_addr, mac_addr); if (!is_valid_ether_addr(ndev->dev_addr)) { eth_hw_addr_random(ndev); diff --git a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_main.c b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_main.c index 61eea6ac846fcfce9d254351221b998dd20a6e47..e05d2095d09bf869479e01bed78f789e62135237 100644 --- a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_main.c +++ b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_main.c @@ -2769,7 +2769,7 @@ static void set_promisc_tcam_enable(struct dsaf_device *dsaf_dev, u32 port) struct hns_mac_cb *mac_cb; u8 addr[ETH_ALEN] = {0}; u8 port_num; - u16 mskid; + int mskid; /* promisc use vague table match with vlanid = 0 & macaddr = 0 */ hns_dsaf_set_mac_key(dsaf_dev, &mac_key, 0x00, port, addr); diff --git a/drivers/net/ethernet/hisilicon/hns/hns_enet.c b/drivers/net/ethernet/hisilicon/hns/hns_enet.c index 4cd86ba1f050dcf786238c147cd2fc3b799797bc..65b985acae38595d66a90bcb9eaea6423b72f909 100644 --- a/drivers/net/ethernet/hisilicon/hns/hns_enet.c +++ b/drivers/net/ethernet/hisilicon/hns/hns_enet.c @@ -598,7 +598,7 @@ static int hns_nic_poll_rx_skb(struct hns_nic_ring_data *ring_data, } else { ring->stats.seg_pkt_cnt++; - pull_len = eth_get_headlen(va, HNS_RX_HEAD_SIZE); + pull_len = eth_get_headlen(ndev, va, HNS_RX_HEAD_SIZE); memcpy(__skb_put(skb, pull_len), va, ALIGN(pull_len, sizeof(long))); @@ -1962,8 +1962,7 @@ static void hns_nic_get_stats64(struct net_device *ndev, static u16 hns_nic_select_queue(struct net_device *ndev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { struct ethhdr *eth_hdr = (struct ethhdr *)skb->data; struct hns_nic_priv *priv = netdev_priv(ndev); @@ -1973,7 +1972,7 @@ hns_nic_select_queue(struct net_device *ndev, struct sk_buff *skb, is_multicast_ether_addr(eth_hdr->h_dest)) return 0; else - return fallback(ndev, skb, NULL); + return netdev_pick_tx(ndev, skb, NULL); } static const struct net_device_ops hns_nic_netdev_ops = { diff --git a/drivers/net/ethernet/hisilicon/hns3/hclge_mbx.h b/drivers/net/ethernet/hisilicon/hns3/hclge_mbx.h index 299b277bc7ae98294d0bf42aeee787e16f7beb8d..83e19c6b974edb08e7c3237d886ad58a196ce04e 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hclge_mbx.h +++ b/drivers/net/ethernet/hisilicon/hns3/hclge_mbx.h @@ -43,6 +43,8 @@ enum HCLGE_MBX_OPCODE { HCLGE_MBX_GET_QID_IN_PF, /* (VF -> PF) get queue id in pf */ HCLGE_MBX_LINK_STAT_MODE, /* (PF -> VF) link mode has changed */ HCLGE_MBX_GET_LINK_MODE, /* (VF -> PF) get the link mode of pf */ + HLCGE_MBX_PUSH_VLAN_INFO, /* (PF -> VF) push port base vlan */ + HCLGE_MBX_GET_MEDIA_TYPE, /* (VF -> PF) get media type */ HCLGE_MBX_GET_VF_FLR_STATUS = 200, /* (M7 -> PF) get vf reset status */ }; @@ -62,6 +64,8 @@ enum hclge_mbx_vlan_cfg_subcode { HCLGE_MBX_VLAN_FILTER = 0, /* set vlan filter */ HCLGE_MBX_VLAN_TX_OFF_CFG, /* set tx side vlan offload */ HCLGE_MBX_VLAN_RX_OFF_CFG, /* set rx side vlan offload */ + HCLGE_MBX_PORT_BASE_VLAN_CFG, /* set port based vlan configuration */ + HCLGE_MBX_GET_PORT_BASE_VLAN_STATE, /* get port based vlan state */ }; #define HCLGE_MBX_MAX_MSG_SIZE 16 @@ -80,12 +84,15 @@ struct hclgevf_mbx_resp_status { struct hclge_mbx_vf_to_pf_cmd { u8 rsv; u8 mbx_src_vfid; /* Auto filled by IMP */ - u8 rsv1[2]; + u8 mbx_need_resp; + u8 rsv1[1]; u8 msg_len; u8 rsv2[3]; u8 msg[HCLGE_MBX_MAX_MSG_SIZE]; }; +#define HCLGE_MBX_NEED_RESP_BIT BIT(0) + struct hclge_mbx_pf_to_vf_cmd { u8 dest_vfid; u8 rsv[3]; @@ -107,7 +114,7 @@ struct hclgevf_mbx_arq_ring { struct hclgevf_dev *hdev; u32 head; u32 tail; - u32 count; + atomic_t count; u16 msg_q[HCLGE_MBX_MAX_ARQ_MSG_NUM][HCLGE_MBX_MAX_ARQ_MSG_SIZE]; }; diff --git a/drivers/net/ethernet/hisilicon/hns3/hnae3.c b/drivers/net/ethernet/hisilicon/hns3/hnae3.c index 17ab4f4af6ad52259e3d0311ed2c751271579af7..fa8b8506b120a46173d88206b0b469d5b39eaef5 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hnae3.c +++ b/drivers/net/ethernet/hisilicon/hns3/hnae3.c @@ -76,8 +76,8 @@ static int hnae3_get_client_init_flag(struct hnae3_client *client, return inited; } -static int hnae3_match_n_instantiate(struct hnae3_client *client, - struct hnae3_ae_dev *ae_dev, bool is_reg) +static int hnae3_init_client_instance(struct hnae3_client *client, + struct hnae3_ae_dev *ae_dev) { int ret; @@ -87,23 +87,27 @@ static int hnae3_match_n_instantiate(struct hnae3_client *client, return 0; } - /* now, (un-)instantiate client by calling lower layer */ - if (is_reg) { - ret = ae_dev->ops->init_client_instance(client, ae_dev); - if (ret) - dev_err(&ae_dev->pdev->dev, - "fail to instantiate client, ret = %d\n", ret); + ret = ae_dev->ops->init_client_instance(client, ae_dev); + if (ret) + dev_err(&ae_dev->pdev->dev, + "fail to instantiate client, ret = %d\n", ret); - return ret; - } + return ret; +} + +static void hnae3_uninit_client_instance(struct hnae3_client *client, + struct hnae3_ae_dev *ae_dev) +{ + /* check if this client matches the type of ae_dev */ + if (!(hnae3_client_match(client->type, ae_dev->dev_type) && + hnae3_get_bit(ae_dev->flag, HNAE3_DEV_INITED_B))) + return; if (hnae3_get_client_init_flag(client, ae_dev)) { ae_dev->ops->uninit_client_instance(client, ae_dev); hnae3_set_client_init_flag(client, ae_dev, 0); } - - return 0; } int hnae3_register_client(struct hnae3_client *client) @@ -129,7 +133,7 @@ int hnae3_register_client(struct hnae3_client *client) /* if the client could not be initialized on current port, for * any error reasons, move on to next available port */ - ret = hnae3_match_n_instantiate(client, ae_dev, true); + ret = hnae3_init_client_instance(client, ae_dev); if (ret) dev_err(&ae_dev->pdev->dev, "match and instantiation failed for port, ret = %d\n", @@ -153,7 +157,7 @@ void hnae3_unregister_client(struct hnae3_client *client) mutex_lock(&hnae3_common_lock); /* un-initialize the client on every matched port */ list_for_each_entry(ae_dev, &hnae3_ae_dev_list, node) { - hnae3_match_n_instantiate(client, ae_dev, false); + hnae3_uninit_client_instance(client, ae_dev); } list_del(&client->node); @@ -205,7 +209,7 @@ void hnae3_register_ae_algo(struct hnae3_ae_algo *ae_algo) * initialize the figure out client instance */ list_for_each_entry(client, &hnae3_client_list, node) { - ret = hnae3_match_n_instantiate(client, ae_dev, true); + ret = hnae3_init_client_instance(client, ae_dev); if (ret) dev_err(&ae_dev->pdev->dev, "match and instantiation failed, ret = %d\n", @@ -243,7 +247,7 @@ void hnae3_unregister_ae_algo(struct hnae3_ae_algo *ae_algo) * un-initialize the figure out client instance */ list_for_each_entry(client, &hnae3_client_list, node) - hnae3_match_n_instantiate(client, ae_dev, false); + hnae3_uninit_client_instance(client, ae_dev); ae_algo->ops->uninit_ae_dev(ae_dev); hnae3_set_bit(ae_dev->flag, HNAE3_DEV_INITED_B, 0); @@ -301,7 +305,7 @@ int hnae3_register_ae_dev(struct hnae3_ae_dev *ae_dev) * initialize the figure out client instance */ list_for_each_entry(client, &hnae3_client_list, node) { - ret = hnae3_match_n_instantiate(client, ae_dev, true); + ret = hnae3_init_client_instance(client, ae_dev); if (ret) dev_err(&ae_dev->pdev->dev, "match and instantiation failed, ret = %d\n", @@ -343,7 +347,7 @@ void hnae3_unregister_ae_dev(struct hnae3_ae_dev *ae_dev) continue; list_for_each_entry(client, &hnae3_client_list, node) - hnae3_match_n_instantiate(client, ae_dev, false); + hnae3_uninit_client_instance(client, ae_dev); ae_algo->ops->uninit_ae_dev(ae_dev); hnae3_set_bit(ae_dev->flag, HNAE3_DEV_INITED_B, 0); diff --git a/drivers/net/ethernet/hisilicon/hns3/hnae3.h b/drivers/net/ethernet/hisilicon/hns3/hnae3.h index 38b430f11fc14f6d474faa4df7a28f2ef19fc287..ad21b0ef19469cd5cf907ffd094cb76d24fc49c0 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hnae3.h +++ b/drivers/net/ethernet/hisilicon/hns3/hnae3.h @@ -120,6 +120,25 @@ enum hnae3_media_type { HNAE3_MEDIA_TYPE_NONE, }; +/* must be consistent with definition in firmware */ +enum hnae3_module_type { + HNAE3_MODULE_TYPE_UNKNOWN = 0x00, + HNAE3_MODULE_TYPE_FIBRE_LR = 0x01, + HNAE3_MODULE_TYPE_FIBRE_SR = 0x02, + HNAE3_MODULE_TYPE_AOC = 0x03, + HNAE3_MODULE_TYPE_CR = 0x04, + HNAE3_MODULE_TYPE_KR = 0x05, + HNAE3_MODULE_TYPE_TP = 0x06, + +}; + +enum hnae3_fec_mode { + HNAE3_FEC_AUTO = 0, + HNAE3_FEC_BASER, + HNAE3_FEC_RS, + HNAE3_FEC_USER_DEF, +}; + enum hnae3_reset_notify_type { HNAE3_UP_CLIENT, HNAE3_DOWN_CLIENT, @@ -147,6 +166,13 @@ enum hnae3_flr_state { HNAE3_FLR_DONE, }; +enum hnae3_port_base_vlan_state { + HNAE3_PORT_BASE_VLAN_DISABLE, + HNAE3_PORT_BASE_VLAN_ENABLE, + HNAE3_PORT_BASE_VLAN_MODIFY, + HNAE3_PORT_BASE_VLAN_NOCHANGE, +}; + struct hnae3_vector_info { u8 __iomem *io_addr; int vector; @@ -223,10 +249,10 @@ struct hnae3_ae_dev { * non-ok * get_ksettings_an_result() * Get negotiation status,speed and duplex - * update_speed_duplex_h() - * Update hardware speed and duplex * get_media_type() * Get media type of MAC + * check_port_speed() + * Check target speed whether is supported * adjust_link() * Adjust link status * set_loopback() @@ -243,6 +269,8 @@ struct hnae3_ae_dev { * set auto autonegotiation of pause frame use * get_autoneg() * get auto autonegotiation of pause frame use + * restart_autoneg() + * restart autonegotiation * get_coalesce_usecs() * get usecs to delay a TX interrupt after a packet is sent * get_rx_max_coalesced_frames() @@ -333,11 +361,15 @@ struct hnae3_ae_ops { void (*get_ksettings_an_result)(struct hnae3_handle *handle, u8 *auto_neg, u32 *speed, u8 *duplex); - int (*update_speed_duplex_h)(struct hnae3_handle *handle); int (*cfg_mac_speed_dup_h)(struct hnae3_handle *handle, int speed, u8 duplex); - void (*get_media_type)(struct hnae3_handle *handle, u8 *media_type); + void (*get_media_type)(struct hnae3_handle *handle, u8 *media_type, + u8 *module_type); + int (*check_port_speed)(struct hnae3_handle *handle, u32 speed); + void (*get_fec)(struct hnae3_handle *handle, u8 *fec_ability, + u8 *fec_mode); + int (*set_fec)(struct hnae3_handle *handle, u32 fec_mode); void (*adjust_link)(struct hnae3_handle *handle, int speed, int duplex); int (*set_loopback)(struct hnae3_handle *handle, enum hnae3_loop loop_mode, bool en); @@ -353,6 +385,7 @@ struct hnae3_ae_ops { int (*set_autoneg)(struct hnae3_handle *handle, bool enable); int (*get_autoneg)(struct hnae3_handle *handle); + int (*restart_autoneg)(struct hnae3_handle *handle); void (*get_coalesce_usecs)(struct hnae3_handle *handle, u32 *tx_usecs, u32 *rx_usecs); @@ -385,7 +418,8 @@ struct hnae3_ae_ops { void (*update_stats)(struct hnae3_handle *handle, struct net_device_stats *net_stats); void (*get_stats)(struct hnae3_handle *handle, u64 *data); - + void (*get_mac_pause_stats)(struct hnae3_handle *handle, u64 *tx_cnt, + u64 *rx_cnt); void (*get_strings)(struct hnae3_handle *handle, u32 stringset, u8 *data); int (*get_sset_count)(struct hnae3_handle *handle, int stringset); @@ -578,8 +612,13 @@ struct hnae3_handle { u32 numa_node_mask; /* for multi-chip support */ + enum hnae3_port_base_vlan_state port_base_vlan_state; + u8 netdev_flags; struct dentry *hnae3_dbgfs; + + /* Network interface message level enabled bits */ + u32 msg_enable; }; #define hnae3_set_field(origin, mask, shift, val) \ diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3_debugfs.c b/drivers/net/ethernet/hisilicon/hns3/hns3_debugfs.c index 0de543faa5b151651287659c8c1f45879d73f7c4..fc4917ac44bece9c3fe42839f82164004ca4eb46 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3_debugfs.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3_debugfs.c @@ -239,6 +239,10 @@ static void hns3_dbg_help(struct hnae3_handle *h) dev_info(&h->pdev->dev, "queue info [number]\n"); dev_info(&h->pdev->dev, "queue map\n"); dev_info(&h->pdev->dev, "bd info [q_num] \n"); + + if (!hns3_is_phys_func(h->pdev)) + return; + dev_info(&h->pdev->dev, "dump fd tcam\n"); dev_info(&h->pdev->dev, "dump tc\n"); dev_info(&h->pdev->dev, "dump tm map [q_num]\n"); @@ -247,6 +251,9 @@ static void hns3_dbg_help(struct hnae3_handle *h) dev_info(&h->pdev->dev, "dump qos pri map\n"); dev_info(&h->pdev->dev, "dump qos buf cfg\n"); dev_info(&h->pdev->dev, "dump mng tbl\n"); + dev_info(&h->pdev->dev, "dump reset info\n"); + dev_info(&h->pdev->dev, "dump ncl_config (in hex)\n"); + dev_info(&h->pdev->dev, "dump mac tnl status\n"); memset(printf_buf, 0, HNS3_DBG_BUF_LEN); strncat(printf_buf, "dump reg [[bios common] [ssu ]", @@ -341,6 +348,8 @@ static ssize_t hns3_dbg_cmd_write(struct file *filp, const char __user *buffer, ret = hns3_dbg_bd_info(handle, cmd_buf); else if (handle->ae_algo->ops->dbg_run_cmd) ret = handle->ae_algo->ops->dbg_run_cmd(handle, cmd_buf); + else + ret = -EOPNOTSUPP; if (ret) hns3_dbg_help(handle); diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c b/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c index 162cb9afa0e705d1e7d668c1cf34eae64f4d65f2..18711e0f9bdf6cb79087b4fd1e6b480a2e8fde4f 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c @@ -35,6 +35,13 @@ static const char hns3_driver_string[] = static const char hns3_copyright[] = "Copyright (c) 2017 Huawei Corporation."; static struct hnae3_client client; +static int debug = -1; +module_param(debug, int, 0); +MODULE_PARM_DESC(debug, " Network interface message level setting"); + +#define DEFAULT_MSG_LEVEL (NETIF_MSG_PROBE | NETIF_MSG_LINK | \ + NETIF_MSG_IFDOWN | NETIF_MSG_IFUP) + /* hns3_pci_tbl - PCI Device ID Table * * Last entry must be all 0s @@ -67,7 +74,7 @@ static irqreturn_t hns3_irq_handle(int irq, void *vector) { struct hns3_enet_tqp_vector *tqp_vector = vector; - napi_schedule(&tqp_vector->napi); + napi_schedule_irqoff(&tqp_vector->napi); return IRQ_HANDLED; } @@ -730,95 +737,6 @@ static int hns3_get_l4_protocol(struct sk_buff *skb, u8 *ol4_proto, return 0; } -static void hns3_set_l2l3l4_len(struct sk_buff *skb, u8 ol4_proto, - u8 il4_proto, u32 *type_cs_vlan_tso, - u32 *ol_type_vlan_len_msec) -{ - union l3_hdr_info l3; - union l4_hdr_info l4; - unsigned char *l2_hdr; - u8 l4_proto = ol4_proto; - u32 ol2_len; - u32 ol3_len; - u32 ol4_len; - u32 l2_len; - u32 l3_len; - - l3.hdr = skb_network_header(skb); - l4.hdr = skb_transport_header(skb); - - /* compute L2 header size for normal packet, defined in 2 Bytes */ - l2_len = l3.hdr - skb->data; - hns3_set_field(*type_cs_vlan_tso, HNS3_TXD_L2LEN_S, l2_len >> 1); - - /* tunnel packet*/ - if (skb->encapsulation) { - /* compute OL2 header size, defined in 2 Bytes */ - ol2_len = l2_len; - hns3_set_field(*ol_type_vlan_len_msec, - HNS3_TXD_L2LEN_S, ol2_len >> 1); - - /* compute OL3 header size, defined in 4 Bytes */ - ol3_len = l4.hdr - l3.hdr; - hns3_set_field(*ol_type_vlan_len_msec, HNS3_TXD_L3LEN_S, - ol3_len >> 2); - - /* MAC in UDP, MAC in GRE (0x6558)*/ - if ((ol4_proto == IPPROTO_UDP) || (ol4_proto == IPPROTO_GRE)) { - /* switch MAC header ptr from outer to inner header.*/ - l2_hdr = skb_inner_mac_header(skb); - - /* compute OL4 header size, defined in 4 Bytes. */ - ol4_len = l2_hdr - l4.hdr; - hns3_set_field(*ol_type_vlan_len_msec, - HNS3_TXD_L4LEN_S, ol4_len >> 2); - - /* switch IP header ptr from outer to inner header */ - l3.hdr = skb_inner_network_header(skb); - - /* compute inner l2 header size, defined in 2 Bytes. */ - l2_len = l3.hdr - l2_hdr; - hns3_set_field(*type_cs_vlan_tso, HNS3_TXD_L2LEN_S, - l2_len >> 1); - } else { - /* skb packet types not supported by hardware, - * txbd len fild doesn't be filled. - */ - return; - } - - /* switch L4 header pointer from outer to inner */ - l4.hdr = skb_inner_transport_header(skb); - - l4_proto = il4_proto; - } - - /* compute inner(/normal) L3 header size, defined in 4 Bytes */ - l3_len = l4.hdr - l3.hdr; - hns3_set_field(*type_cs_vlan_tso, HNS3_TXD_L3LEN_S, l3_len >> 2); - - /* compute inner(/normal) L4 header size, defined in 4 Bytes */ - switch (l4_proto) { - case IPPROTO_TCP: - hns3_set_field(*type_cs_vlan_tso, HNS3_TXD_L4LEN_S, - l4.tcp->doff); - break; - case IPPROTO_SCTP: - hns3_set_field(*type_cs_vlan_tso, HNS3_TXD_L4LEN_S, - (sizeof(struct sctphdr) >> 2)); - break; - case IPPROTO_UDP: - hns3_set_field(*type_cs_vlan_tso, HNS3_TXD_L4LEN_S, - (sizeof(struct udphdr) >> 2)); - break; - default: - /* skb packet types not supported by hardware, - * txbd len fild doesn't be filled. - */ - return; - } -} - /* when skb->encapsulation is 0, skb->ip_summed is CHECKSUM_PARTIAL * and it is udp packet, which has a dest port as the IANA assigned. * the hardware is expected to do the checksum offload, but the @@ -827,12 +745,12 @@ static void hns3_set_l2l3l4_len(struct sk_buff *skb, u8 ol4_proto, */ static bool hns3_tunnel_csum_bug(struct sk_buff *skb) { -#define IANA_VXLAN_PORT 4789 union l4_hdr_info l4; l4.hdr = skb_transport_header(skb); - if (!(!skb->encapsulation && l4.udp->dest == htons(IANA_VXLAN_PORT))) + if (!(!skb->encapsulation && + l4.udp->dest == htons(IANA_VXLAN_UDP_PORT))) return false; skb_checksum_help(skb); @@ -840,46 +758,71 @@ static bool hns3_tunnel_csum_bug(struct sk_buff *skb) return true; } -static int hns3_set_l3l4_type_csum(struct sk_buff *skb, u8 ol4_proto, - u8 il4_proto, u32 *type_cs_vlan_tso, - u32 *ol_type_vlan_len_msec) +static void hns3_set_outer_l2l3l4(struct sk_buff *skb, u8 ol4_proto, + u32 *ol_type_vlan_len_msec) { + u32 l2_len, l3_len, l4_len; + unsigned char *il2_hdr; union l3_hdr_info l3; - u32 l4_proto = ol4_proto; + union l4_hdr_info l4; l3.hdr = skb_network_header(skb); + l4.hdr = skb_transport_header(skb); - /* define OL3 type and tunnel type(OL4).*/ - if (skb->encapsulation) { - /* define outer network header type.*/ - if (skb->protocol == htons(ETH_P_IP)) { - if (skb_is_gso(skb)) - hns3_set_field(*ol_type_vlan_len_msec, - HNS3_TXD_OL3T_S, - HNS3_OL3T_IPV4_CSUM); - else - hns3_set_field(*ol_type_vlan_len_msec, - HNS3_TXD_OL3T_S, - HNS3_OL3T_IPV4_NO_CSUM); - - } else if (skb->protocol == htons(ETH_P_IPV6)) { - hns3_set_field(*ol_type_vlan_len_msec, HNS3_TXD_OL3T_S, - HNS3_OL3T_IPV6); - } + /* compute OL2 header size, defined in 2 Bytes */ + l2_len = l3.hdr - skb->data; + hns3_set_field(*ol_type_vlan_len_msec, HNS3_TXD_L2LEN_S, l2_len >> 1); - /* define tunnel type(OL4).*/ - switch (l4_proto) { - case IPPROTO_UDP: + /* compute OL3 header size, defined in 4 Bytes */ + l3_len = l4.hdr - l3.hdr; + hns3_set_field(*ol_type_vlan_len_msec, HNS3_TXD_L3LEN_S, l3_len >> 2); + + il2_hdr = skb_inner_mac_header(skb); + /* compute OL4 header size, defined in 4 Bytes. */ + l4_len = il2_hdr - l4.hdr; + hns3_set_field(*ol_type_vlan_len_msec, HNS3_TXD_L4LEN_S, l4_len >> 2); + + /* define outer network header type */ + if (skb->protocol == htons(ETH_P_IP)) { + if (skb_is_gso(skb)) hns3_set_field(*ol_type_vlan_len_msec, - HNS3_TXD_TUNTYPE_S, - HNS3_TUN_MAC_IN_UDP); - break; - case IPPROTO_GRE: + HNS3_TXD_OL3T_S, + HNS3_OL3T_IPV4_CSUM); + else hns3_set_field(*ol_type_vlan_len_msec, - HNS3_TXD_TUNTYPE_S, - HNS3_TUN_NVGRE); - break; - default: + HNS3_TXD_OL3T_S, + HNS3_OL3T_IPV4_NO_CSUM); + + } else if (skb->protocol == htons(ETH_P_IPV6)) { + hns3_set_field(*ol_type_vlan_len_msec, HNS3_TXD_OL3T_S, + HNS3_OL3T_IPV6); + } + + if (ol4_proto == IPPROTO_UDP) + hns3_set_field(*ol_type_vlan_len_msec, HNS3_TXD_TUNTYPE_S, + HNS3_TUN_MAC_IN_UDP); + else if (ol4_proto == IPPROTO_GRE) + hns3_set_field(*ol_type_vlan_len_msec, HNS3_TXD_TUNTYPE_S, + HNS3_TUN_NVGRE); +} + +static int hns3_set_l2l3l4(struct sk_buff *skb, u8 ol4_proto, + u8 il4_proto, u32 *type_cs_vlan_tso, + u32 *ol_type_vlan_len_msec) +{ + unsigned char *l2_hdr = l2_hdr = skb->data; + u32 l4_proto = ol4_proto; + union l4_hdr_info l4; + union l3_hdr_info l3; + u32 l2_len, l3_len; + + l4.hdr = skb_transport_header(skb); + l3.hdr = skb_network_header(skb); + + /* handle encapsulation skb */ + if (skb->encapsulation) { + /* If this is a not UDP/GRE encapsulation skb */ + if (!(ol4_proto == IPPROTO_UDP || ol4_proto == IPPROTO_GRE)) { /* drop the skb tunnel packet if hardware don't support, * because hardware can't calculate csum when TSO. */ @@ -893,7 +836,12 @@ static int hns3_set_l3l4_type_csum(struct sk_buff *skb, u8 ol4_proto, return 0; } + hns3_set_outer_l2l3l4(skb, ol4_proto, ol_type_vlan_len_msec); + + /* switch to inner header */ + l2_hdr = skb_inner_mac_header(skb); l3.hdr = skb_inner_network_header(skb); + l4.hdr = skb_inner_transport_header(skb); l4_proto = il4_proto; } @@ -911,11 +859,22 @@ static int hns3_set_l3l4_type_csum(struct sk_buff *skb, u8 ol4_proto, HNS3_L3T_IPV6); } + /* compute inner(/normal) L2 header size, defined in 2 Bytes */ + l2_len = l3.hdr - l2_hdr; + hns3_set_field(*type_cs_vlan_tso, HNS3_TXD_L2LEN_S, l2_len >> 1); + + /* compute inner(/normal) L3 header size, defined in 4 Bytes */ + l3_len = l4.hdr - l3.hdr; + hns3_set_field(*type_cs_vlan_tso, HNS3_TXD_L3LEN_S, l3_len >> 2); + + /* compute inner(/normal) L4 header size, defined in 4 Bytes */ switch (l4_proto) { case IPPROTO_TCP: hns3_set_field(*type_cs_vlan_tso, HNS3_TXD_L4CS_B, 1); hns3_set_field(*type_cs_vlan_tso, HNS3_TXD_L4T_S, HNS3_L4T_TCP); + hns3_set_field(*type_cs_vlan_tso, HNS3_TXD_L4LEN_S, + l4.tcp->doff); break; case IPPROTO_UDP: if (hns3_tunnel_csum_bug(skb)) @@ -924,11 +883,15 @@ static int hns3_set_l3l4_type_csum(struct sk_buff *skb, u8 ol4_proto, hns3_set_field(*type_cs_vlan_tso, HNS3_TXD_L4CS_B, 1); hns3_set_field(*type_cs_vlan_tso, HNS3_TXD_L4T_S, HNS3_L4T_UDP); + hns3_set_field(*type_cs_vlan_tso, HNS3_TXD_L4LEN_S, + (sizeof(struct udphdr) >> 2)); break; case IPPROTO_SCTP: hns3_set_field(*type_cs_vlan_tso, HNS3_TXD_L4CS_B, 1); hns3_set_field(*type_cs_vlan_tso, HNS3_TXD_L4T_S, HNS3_L4T_SCTP); + hns3_set_field(*type_cs_vlan_tso, HNS3_TXD_L4LEN_S, + (sizeof(struct sctphdr) >> 2)); break; default: /* drop the skb tunnel packet if hardware don't support, @@ -963,6 +926,16 @@ static int hns3_fill_desc_vtags(struct sk_buff *skb, { #define HNS3_TX_VLAN_PRIO_SHIFT 13 + struct hnae3_handle *handle = tx_ring->tqp->handle; + + /* Since HW limitation, if port based insert VLAN enabled, only one VLAN + * header is allowed in skb, otherwise it will cause RAS error. + */ + if (unlikely(skb_vlan_tagged_multi(skb) && + handle->port_base_vlan_state == + HNAE3_PORT_BASE_VLAN_ENABLE)) + return -EINVAL; + if (skb->protocol == htons(ETH_P_8021Q) && !(tx_ring->tqp->handle->kinfo.netdev->features & NETIF_F_HW_VLAN_CTAG_TX)) { @@ -984,8 +957,16 @@ static int hns3_fill_desc_vtags(struct sk_buff *skb, * and use inner_vtag in one tag case. */ if (skb->protocol == htons(ETH_P_8021Q)) { - hns3_set_field(*out_vlan_flag, HNS3_TXD_OVLAN_B, 1); - *out_vtag = vlan_tag; + if (handle->port_base_vlan_state == + HNAE3_PORT_BASE_VLAN_DISABLE){ + hns3_set_field(*out_vlan_flag, + HNS3_TXD_OVLAN_B, 1); + *out_vtag = vlan_tag; + } else { + hns3_set_field(*inner_vlan_flag, + HNS3_TXD_VLAN_B, 1); + *inner_vtag = vlan_tag; + } } else { hns3_set_field(*inner_vlan_flag, HNS3_TXD_VLAN_B, 1); *inner_vtag = vlan_tag; @@ -1012,7 +993,6 @@ static int hns3_fill_desc(struct hns3_enet_ring *ring, void *priv, struct hns3_desc_cb *desc_cb = &ring->desc_cb[ring->next_to_use]; struct hns3_desc *desc = &ring->desc[ring->next_to_use]; struct device *dev = ring_to_dev(ring); - u16 bdtp_fe_sc_vld_ra_ri = 0; struct skb_frag_struct *frag; unsigned int frag_buf_num; int k, sizeoflast; @@ -1042,12 +1022,10 @@ static int hns3_fill_desc(struct hns3_enet_ring *ring, void *priv, ret = hns3_get_l4_protocol(skb, &ol4_proto, &il4_proto); if (unlikely(ret)) return ret; - hns3_set_l2l3l4_len(skb, ol4_proto, il4_proto, - &type_cs_vlan_tso, - &ol_type_vlan_len_msec); - ret = hns3_set_l3l4_type_csum(skb, ol4_proto, il4_proto, - &type_cs_vlan_tso, - &ol_type_vlan_len_msec); + + ret = hns3_set_l2l3l4(skb, ol4_proto, il4_proto, + &type_cs_vlan_tso, + &ol_type_vlan_len_msec); if (unlikely(ret)) return ret; @@ -1073,19 +1051,37 @@ static int hns3_fill_desc(struct hns3_enet_ring *ring, void *priv, dma = skb_frag_dma_map(dev, frag, 0, size, DMA_TO_DEVICE); } - if (unlikely(dma_mapping_error(ring->dev, dma))) { + if (unlikely(dma_mapping_error(dev, dma))) { ring->stats.sw_err_cnt++; return -ENOMEM; } desc_cb->length = size; + if (likely(size <= HNS3_MAX_BD_SIZE)) { + u16 bdtp_fe_sc_vld_ra_ri = 0; + + desc_cb->priv = priv; + desc_cb->dma = dma; + desc_cb->type = type; + desc->addr = cpu_to_le64(dma); + desc->tx.send_size = cpu_to_le16(size); + hns3_set_txbd_baseinfo(&bdtp_fe_sc_vld_ra_ri, frag_end); + desc->tx.bdtp_fe_sc_vld_ra_ri = + cpu_to_le16(bdtp_fe_sc_vld_ra_ri); + + ring_ptr_move_fw(ring, next_to_use); + return 0; + } + frag_buf_num = hns3_tx_bd_count(size); sizeoflast = size & HNS3_TX_LAST_SIZE_M; sizeoflast = sizeoflast ? sizeoflast : HNS3_MAX_BD_SIZE; /* When frag size is bigger than hardware limit, split this frag */ for (k = 0; k < frag_buf_num; k++) { + u16 bdtp_fe_sc_vld_ra_ri = 0; + /* The txbd's baseinfo of DESC_TYPE_PAGE & DESC_TYPE_SKB */ desc_cb->priv = priv; desc_cb->dma = dma + HNS3_MAX_BD_SIZE * k; @@ -1112,64 +1108,92 @@ static int hns3_fill_desc(struct hns3_enet_ring *ring, void *priv, return 0; } -static int hns3_nic_maybe_stop_tso(struct sk_buff **out_skb, int *bnum, - struct hns3_enet_ring *ring) +static int hns3_nic_bd_num(struct sk_buff *skb) { - struct sk_buff *skb = *out_skb; - struct sk_buff *new_skb = NULL; - struct skb_frag_struct *frag; - int bdnum_for_frag; - int frag_num; - int buf_num; - int size; - int i; + int size = skb_headlen(skb); + int i, bd_num; - size = skb_headlen(skb); - buf_num = hns3_tx_bd_count(size); + /* if the total len is within the max bd limit */ + if (likely(skb->len <= HNS3_MAX_BD_SIZE)) + return skb_shinfo(skb)->nr_frags + 1; + + bd_num = hns3_tx_bd_count(size); + + for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) { + struct skb_frag_struct *frag = &skb_shinfo(skb)->frags[i]; + int frag_bd_num; - frag_num = skb_shinfo(skb)->nr_frags; - for (i = 0; i < frag_num; i++) { - frag = &skb_shinfo(skb)->frags[i]; size = skb_frag_size(frag); - bdnum_for_frag = hns3_tx_bd_count(size); - if (unlikely(bdnum_for_frag > HNS3_MAX_BD_PER_FRAG)) + frag_bd_num = hns3_tx_bd_count(size); + + if (unlikely(frag_bd_num > HNS3_MAX_BD_PER_FRAG)) return -ENOMEM; - buf_num += bdnum_for_frag; + bd_num += frag_bd_num; } - if (unlikely(buf_num > HNS3_MAX_BD_PER_FRAG)) { - buf_num = hns3_tx_bd_count(skb->len); - if (ring_space(ring) < buf_num) - return -EBUSY; - /* manual split the send packet */ - new_skb = skb_copy(skb, GFP_ATOMIC); - if (!new_skb) - return -ENOMEM; - dev_kfree_skb_any(skb); - *out_skb = new_skb; - } + return bd_num; +} - if (unlikely(ring_space(ring) < buf_num)) - return -EBUSY; +static unsigned int hns3_gso_hdr_len(struct sk_buff *skb) +{ + if (!skb->encapsulation) + return skb_transport_offset(skb) + tcp_hdrlen(skb); - *bnum = buf_num; - return 0; + return skb_inner_transport_offset(skb) + inner_tcp_hdrlen(skb); } -static int hns3_nic_maybe_stop_tx(struct sk_buff **out_skb, int *bnum, - struct hns3_enet_ring *ring) +/* HW need every continuous 8 buffer data to be larger than MSS, + * we simplify it by ensuring skb_headlen + the first continuous + * 7 frags to to be larger than gso header len + mss, and the remaining + * continuous 7 frags to be larger than MSS except the last 7 frags. + */ +static bool hns3_skb_need_linearized(struct sk_buff *skb) +{ + int bd_limit = HNS3_MAX_BD_PER_FRAG - 1; + unsigned int tot_len = 0; + int i; + + for (i = 0; i < bd_limit; i++) + tot_len += skb_frag_size(&skb_shinfo(skb)->frags[i]); + + /* ensure headlen + the first 7 frags is greater than mss + header + * and the first 7 frags is greater than mss. + */ + if (((tot_len + skb_headlen(skb)) < (skb_shinfo(skb)->gso_size + + hns3_gso_hdr_len(skb))) || (tot_len < skb_shinfo(skb)->gso_size)) + return true; + + /* ensure the remaining continuous 7 buffer is greater than mss */ + for (i = 0; i < (skb_shinfo(skb)->nr_frags - bd_limit - 1); i++) { + tot_len -= skb_frag_size(&skb_shinfo(skb)->frags[i]); + tot_len += skb_frag_size(&skb_shinfo(skb)->frags[i + bd_limit]); + + if (tot_len < skb_shinfo(skb)->gso_size) + return true; + } + + return false; +} + +static int hns3_nic_maybe_stop_tx(struct hns3_enet_ring *ring, + struct sk_buff **out_skb) { struct sk_buff *skb = *out_skb; - struct sk_buff *new_skb = NULL; - int buf_num; + int bd_num; - /* No. of segments (plus a header) */ - buf_num = skb_shinfo(skb)->nr_frags + 1; + bd_num = hns3_nic_bd_num(skb); + if (bd_num < 0) + return bd_num; - if (unlikely(buf_num > HNS3_MAX_BD_PER_FRAG)) { - buf_num = hns3_tx_bd_count(skb->len); - if (ring_space(ring) < buf_num) + if (unlikely(bd_num > HNS3_MAX_BD_PER_FRAG)) { + struct sk_buff *new_skb; + + if (skb_is_gso(skb) && !hns3_skb_need_linearized(skb)) + goto out; + + bd_num = hns3_tx_bd_count(skb->len); + if (unlikely(ring_space(ring) < bd_num)) return -EBUSY; /* manual split the send packet */ new_skb = skb_copy(skb, GFP_ATOMIC); @@ -1177,14 +1201,17 @@ static int hns3_nic_maybe_stop_tx(struct sk_buff **out_skb, int *bnum, return -ENOMEM; dev_kfree_skb_any(skb); *out_skb = new_skb; + + u64_stats_update_begin(&ring->syncp); + ring->stats.tx_copy++; + u64_stats_update_end(&ring->syncp); } - if (unlikely(ring_space(ring) < buf_num)) +out: + if (unlikely(ring_space(ring) < bd_num)) return -EBUSY; - *bnum = buf_num; - - return 0; + return bd_num; } static void hns3_clear_desc(struct hns3_enet_ring *ring, int next_to_use_orig) @@ -1197,6 +1224,9 @@ static void hns3_clear_desc(struct hns3_enet_ring *ring, int next_to_use_orig) if (ring->next_to_use == next_to_use_orig) break; + /* rollback one */ + ring_ptr_move_bw(ring, next_to_use); + /* unmap the descriptor dma address */ if (ring->desc_cb[ring->next_to_use].type == DESC_TYPE_SKB) dma_unmap_single(dev, @@ -1210,9 +1240,7 @@ static void hns3_clear_desc(struct hns3_enet_ring *ring, int next_to_use_orig) DMA_TO_DEVICE); ring->desc_cb[ring->next_to_use].length = 0; - - /* rollback one */ - ring_ptr_move_bw(ring, next_to_use); + ring->desc_cb[ring->next_to_use].dma = 0; } } @@ -1225,7 +1253,6 @@ netdev_tx_t hns3_nic_net_xmit(struct sk_buff *skb, struct net_device *netdev) struct netdev_queue *dev_queue; struct skb_frag_struct *frag; int next_to_use_head; - int next_to_use_frag; int buf_num; int seg_num; int size; @@ -1235,22 +1262,23 @@ netdev_tx_t hns3_nic_net_xmit(struct sk_buff *skb, struct net_device *netdev) /* Prefetch the data used later */ prefetch(skb->data); - switch (priv->ops.maybe_stop_tx(&skb, &buf_num, ring)) { - case -EBUSY: - u64_stats_update_begin(&ring->syncp); - ring->stats.tx_busy++; - u64_stats_update_end(&ring->syncp); + buf_num = hns3_nic_maybe_stop_tx(ring, &skb); + if (unlikely(buf_num <= 0)) { + if (buf_num == -EBUSY) { + u64_stats_update_begin(&ring->syncp); + ring->stats.tx_busy++; + u64_stats_update_end(&ring->syncp); + goto out_net_tx_busy; + } else if (buf_num == -ENOMEM) { + u64_stats_update_begin(&ring->syncp); + ring->stats.sw_err_cnt++; + u64_stats_update_end(&ring->syncp); + } - goto out_net_tx_busy; - case -ENOMEM: - u64_stats_update_begin(&ring->syncp); - ring->stats.sw_err_cnt++; - u64_stats_update_end(&ring->syncp); - netdev_err(netdev, "no memory to xmit!\n"); + if (net_ratelimit()) + netdev_err(netdev, "xmit error: %d!\n", buf_num); goto out_err_tx_ok; - default: - break; } /* No. of segments (plus a header) */ @@ -1263,9 +1291,8 @@ netdev_tx_t hns3_nic_net_xmit(struct sk_buff *skb, struct net_device *netdev) ret = hns3_fill_desc(ring, skb, size, seg_num == 1 ? 1 : 0, DESC_TYPE_SKB); if (unlikely(ret)) - goto head_fill_err; + goto fill_err; - next_to_use_frag = ring->next_to_use; /* Fill the fragments */ for (i = 1; i < seg_num; i++) { frag = &skb_shinfo(skb)->frags[i - 1]; @@ -1276,7 +1303,7 @@ netdev_tx_t hns3_nic_net_xmit(struct sk_buff *skb, struct net_device *netdev) DESC_TYPE_PAGE); if (unlikely(ret)) - goto frag_fill_err; + goto fill_err; } /* Complete translate all packets */ @@ -1289,10 +1316,7 @@ netdev_tx_t hns3_nic_net_xmit(struct sk_buff *skb, struct net_device *netdev) return NETDEV_TX_OK; -frag_fill_err: - hns3_clear_desc(ring, next_to_use_frag); - -head_fill_err: +fill_err: hns3_clear_desc(ring, next_to_use_head); out_err_tx_ok: @@ -1355,13 +1379,6 @@ static int hns3_nic_set_features(struct net_device *netdev, bool enable; int ret; - if (changed & (NETIF_F_TSO | NETIF_F_TSO6)) { - if (features & (NETIF_F_TSO | NETIF_F_TSO6)) - priv->ops.maybe_stop_tx = hns3_nic_maybe_stop_tso; - else - priv->ops.maybe_stop_tx = hns3_nic_maybe_stop_tx; - } - if (changed & (NETIF_F_GRO_HW) && h->ae_algo->ops->set_gro_en) { enable = !!(features & NETIF_F_GRO_HW); ret = h->ae_algo->ops->set_gro_en(h, enable); @@ -1574,6 +1591,9 @@ static int hns3_nic_change_mtu(struct net_device *netdev, int new_mtu) struct hnae3_handle *h = hns3_get_handle(netdev); int ret; + if (hns3_nic_resetting(netdev)) + return -EBUSY; + if (!h->ae_algo->ops->set_mtu) return -EOPNOTSUPP; @@ -1590,13 +1610,19 @@ static int hns3_nic_change_mtu(struct net_device *netdev, int new_mtu) static bool hns3_get_tx_timeo_queue_info(struct net_device *ndev) { struct hns3_nic_priv *priv = netdev_priv(ndev); + struct hnae3_handle *h = hns3_get_handle(ndev); struct hns3_enet_ring *tx_ring = NULL; + struct napi_struct *napi; int timeout_queue = 0; int hw_head, hw_tail; + int fbd_num, fbd_oft; + int ebd_num, ebd_oft; + int bd_num, bd_err; + int ring_en, tc; int i; /* Find the stopped queue the same way the stack does */ - for (i = 0; i < ndev->real_num_tx_queues; i++) { + for (i = 0; i < ndev->num_tx_queues; i++) { struct netdev_queue *q; unsigned long trans_start; @@ -1617,21 +1643,66 @@ static bool hns3_get_tx_timeo_queue_info(struct net_device *ndev) return false; } + priv->tx_timeout_count++; + tx_ring = priv->ring_data[timeout_queue].ring; + napi = &tx_ring->tqp_vector->napi; + + netdev_info(ndev, + "tx_timeout count: %llu, queue id: %d, SW_NTU: 0x%x, SW_NTC: 0x%x, napi state: %lu\n", + priv->tx_timeout_count, timeout_queue, tx_ring->next_to_use, + tx_ring->next_to_clean, napi->state); + + netdev_info(ndev, + "tx_pkts: %llu, tx_bytes: %llu, io_err_cnt: %llu, sw_err_cnt: %llu\n", + tx_ring->stats.tx_pkts, tx_ring->stats.tx_bytes, + tx_ring->stats.io_err_cnt, tx_ring->stats.sw_err_cnt); + + netdev_info(ndev, + "seg_pkt_cnt: %llu, tx_err_cnt: %llu, restart_queue: %llu, tx_busy: %llu\n", + tx_ring->stats.seg_pkt_cnt, tx_ring->stats.tx_err_cnt, + tx_ring->stats.restart_queue, tx_ring->stats.tx_busy); + + /* When mac received many pause frames continuous, it's unable to send + * packets, which may cause tx timeout + */ + if (h->ae_algo->ops->update_stats && + h->ae_algo->ops->get_mac_pause_stats) { + u64 tx_pause_cnt, rx_pause_cnt; + + h->ae_algo->ops->update_stats(h, &ndev->stats); + h->ae_algo->ops->get_mac_pause_stats(h, &tx_pause_cnt, + &rx_pause_cnt); + netdev_info(ndev, "tx_pause_cnt: %llu, rx_pause_cnt: %llu\n", + tx_pause_cnt, rx_pause_cnt); + } hw_head = readl_relaxed(tx_ring->tqp->io_base + HNS3_RING_TX_RING_HEAD_REG); hw_tail = readl_relaxed(tx_ring->tqp->io_base + HNS3_RING_TX_RING_TAIL_REG); + fbd_num = readl_relaxed(tx_ring->tqp->io_base + + HNS3_RING_TX_RING_FBDNUM_REG); + fbd_oft = readl_relaxed(tx_ring->tqp->io_base + + HNS3_RING_TX_RING_OFFSET_REG); + ebd_num = readl_relaxed(tx_ring->tqp->io_base + + HNS3_RING_TX_RING_EBDNUM_REG); + ebd_oft = readl_relaxed(tx_ring->tqp->io_base + + HNS3_RING_TX_RING_EBD_OFFSET_REG); + bd_num = readl_relaxed(tx_ring->tqp->io_base + + HNS3_RING_TX_RING_BD_NUM_REG); + bd_err = readl_relaxed(tx_ring->tqp->io_base + + HNS3_RING_TX_RING_BD_ERR_REG); + ring_en = readl_relaxed(tx_ring->tqp->io_base + HNS3_RING_EN_REG); + tc = readl_relaxed(tx_ring->tqp->io_base + HNS3_RING_TX_RING_TC_REG); + netdev_info(ndev, - "tx_timeout count: %llu, queue id: %d, SW_NTU: 0x%x, SW_NTC: 0x%x, HW_HEAD: 0x%x, HW_TAIL: 0x%x, INT: 0x%x\n", - priv->tx_timeout_count, - timeout_queue, - tx_ring->next_to_use, - tx_ring->next_to_clean, - hw_head, - hw_tail, + "BD_NUM: 0x%x HW_HEAD: 0x%x, HW_TAIL: 0x%x, BD_ERR: 0x%x, INT: 0x%x\n", + bd_num, hw_head, hw_tail, bd_err, readl(tx_ring->tqp_vector->mask_addr)); + netdev_info(ndev, + "RING_EN: 0x%x, TC: 0x%x, FBD_NUM: 0x%x FBD_OFT: 0x%x, EBD_NUM: 0x%x, EBD_OFT: 0x%x\n", + ring_en, tc, fbd_num, fbd_oft, ebd_num, ebd_oft); return true; } @@ -1644,8 +1715,6 @@ static void hns3_nic_net_timeout(struct net_device *ndev) if (!hns3_get_tx_timeo_queue_info(ndev)) return; - priv->tx_timeout_count++; - /* request the reset, and let the hclge to determine * which reset level should be done */ @@ -1670,7 +1739,7 @@ static const struct net_device_ops hns3_nic_netdev_ops = { .ndo_set_vf_vlan = hns3_ndo_set_vf_vlan, }; -static bool hns3_is_phys_func(struct pci_dev *pdev) +bool hns3_is_phys_func(struct pci_dev *pdev) { u32 dev_id = pdev->device; @@ -2117,17 +2186,30 @@ static void hns3_reuse_buffer(struct hns3_enet_ring *ring, int i) ring->desc[i].rx.bd_base_info = 0; } -static void hns3_nic_reclaim_one_desc(struct hns3_enet_ring *ring, int *bytes, - int *pkts) +static void hns3_nic_reclaim_desc(struct hns3_enet_ring *ring, int head, + int *bytes, int *pkts) { - struct hns3_desc_cb *desc_cb = &ring->desc_cb[ring->next_to_clean]; + int ntc = ring->next_to_clean; + struct hns3_desc_cb *desc_cb; - (*pkts) += (desc_cb->type == DESC_TYPE_SKB); - (*bytes) += desc_cb->length; - /* desc_cb will be cleaned, after hnae3_free_buffer_detach*/ - hns3_free_buffer_detach(ring, ring->next_to_clean); + while (head != ntc) { + desc_cb = &ring->desc_cb[ntc]; + (*pkts) += (desc_cb->type == DESC_TYPE_SKB); + (*bytes) += desc_cb->length; + /* desc_cb will be cleaned, after hnae3_free_buffer_detach */ + hns3_free_buffer_detach(ring, ntc); - ring_ptr_move_fw(ring, next_to_clean); + if (++ntc == ring->desc_num) + ntc = 0; + + /* Issue prefetch for next Tx descriptor */ + prefetch(&ring->desc_cb[ntc]); + } + + /* This smp_store_release() pairs with smp_load_acquire() in + * ring_space called by hns3_nic_net_xmit. + */ + smp_store_release(&ring->next_to_clean, ntc); } static int is_valid_clean_head(struct hns3_enet_ring *ring, int h) @@ -2167,11 +2249,7 @@ void hns3_clean_tx_ring(struct hns3_enet_ring *ring) bytes = 0; pkts = 0; - while (head != ring->next_to_clean) { - hns3_nic_reclaim_one_desc(ring, &bytes, &pkts); - /* Issue prefetch for next Tx descriptor */ - prefetch(&ring->desc_cb[ring->next_to_clean]); - } + hns3_nic_reclaim_desc(ring, head, &bytes, &pkts); ring->tqp_vector->tx_group.total_bytes += bytes; ring->tqp_vector->tx_group.total_packets += pkts; @@ -2233,6 +2311,10 @@ hns3_nic_alloc_rx_buffers(struct hns3_enet_ring *ring, int cleand_count) break; } hns3_replace_buffer(ring, ring->next_to_use, &res_cbs); + + u64_stats_update_begin(&ring->syncp); + ring->stats.non_reuse_pg++; + u64_stats_update_end(&ring->syncp); } ring_ptr_move_fw(ring, next_to_use); @@ -2246,64 +2328,78 @@ static void hns3_nic_reuse_page(struct sk_buff *skb, int i, struct hns3_enet_ring *ring, int pull_len, struct hns3_desc_cb *desc_cb) { - struct hns3_desc *desc; - u32 truesize; - int size; - int last_offset; - bool twobufs; - - twobufs = ((PAGE_SIZE < 8192) && - hnae3_buf_size(ring) == HNS3_BUFFER_SIZE_2048); - - desc = &ring->desc[ring->next_to_clean]; - size = le16_to_cpu(desc->rx.size); - - truesize = hnae3_buf_size(ring); - - if (!twobufs) - last_offset = hnae3_page_size(ring) - hnae3_buf_size(ring); + struct hns3_desc *desc = &ring->desc[ring->next_to_clean]; + int size = le16_to_cpu(desc->rx.size); + u32 truesize = hnae3_buf_size(ring); skb_add_rx_frag(skb, i, desc_cb->priv, desc_cb->page_offset + pull_len, size - pull_len, truesize); - /* Avoid re-using remote pages,flag default unreuse */ - if (unlikely(page_to_nid(desc_cb->priv) != numa_node_id())) - return; - - if (twobufs) { - /* If we are only owner of page we can reuse it */ - if (likely(page_count(desc_cb->priv) == 1)) { - /* Flip page offset to other buffer */ - desc_cb->page_offset ^= truesize; - - desc_cb->reuse_flag = 1; - /* bump ref count on page before it is given*/ - get_page(desc_cb->priv); - } + /* Avoid re-using remote pages, or the stack is still using the page + * when page_offset rollback to zero, flag default unreuse + */ + if (unlikely(page_to_nid(desc_cb->priv) != numa_node_id()) || + (!desc_cb->page_offset && page_count(desc_cb->priv) > 1)) return; - } /* Move offset up to the next cache line */ desc_cb->page_offset += truesize; - if (desc_cb->page_offset <= last_offset) { + if (desc_cb->page_offset + truesize <= hnae3_page_size(ring)) { desc_cb->reuse_flag = 1; /* Bump ref count on page before it is given*/ get_page(desc_cb->priv); + } else if (page_count(desc_cb->priv) == 1) { + desc_cb->reuse_flag = 1; + desc_cb->page_offset = 0; + get_page(desc_cb->priv); } } +static int hns3_gro_complete(struct sk_buff *skb) +{ + __be16 type = skb->protocol; + struct tcphdr *th; + int depth = 0; + + while (type == htons(ETH_P_8021Q)) { + struct vlan_hdr *vh; + + if ((depth + VLAN_HLEN) > skb_headlen(skb)) + return -EFAULT; + + vh = (struct vlan_hdr *)(skb->data + depth); + type = vh->h_vlan_encapsulated_proto; + depth += VLAN_HLEN; + } + + if (type == htons(ETH_P_IP)) { + depth += sizeof(struct iphdr); + } else if (type == htons(ETH_P_IPV6)) { + depth += sizeof(struct ipv6hdr); + } else { + netdev_err(skb->dev, + "Error: FW GRO supports only IPv4/IPv6, not 0x%04x, depth: %d\n", + be16_to_cpu(type), depth); + return -EFAULT; + } + + th = (struct tcphdr *)(skb->data + depth); + skb_shinfo(skb)->gso_segs = NAPI_GRO_CB(skb)->count; + if (th->cwr) + skb_shinfo(skb)->gso_type |= SKB_GSO_TCP_ECN; + + skb->ip_summed = CHECKSUM_UNNECESSARY; + + return 0; +} + static void hns3_rx_checksum(struct hns3_enet_ring *ring, struct sk_buff *skb, - struct hns3_desc *desc) + u32 l234info, u32 bd_base_info, u32 ol_info) { struct net_device *netdev = ring->tqp->handle->kinfo.netdev; int l3_type, l4_type; - u32 bd_base_info; int ol4_type; - u32 l234info; - - bd_base_info = le32_to_cpu(desc->rx.bd_base_info); - l234info = le32_to_cpu(desc->rx.l234_info); skb->ip_summed = CHECKSUM_NONE; @@ -2312,12 +2408,6 @@ static void hns3_rx_checksum(struct hns3_enet_ring *ring, struct sk_buff *skb, if (!(netdev->features & NETIF_F_RXCSUM)) return; - /* We MUST enable hardware checksum before enabling hardware GRO */ - if (skb_shinfo(skb)->gso_size) { - skb->ip_summed = CHECKSUM_UNNECESSARY; - return; - } - /* check if hardware has done checksum */ if (!(bd_base_info & BIT(HNS3_RXD_L3L4P_B))) return; @@ -2332,7 +2422,7 @@ static void hns3_rx_checksum(struct hns3_enet_ring *ring, struct sk_buff *skb, return; } - ol4_type = hnae3_get_field(l234info, HNS3_RXD_OL4ID_M, + ol4_type = hnae3_get_field(ol_info, HNS3_RXD_OL4ID_M, HNS3_RXD_OL4ID_S); switch (ol4_type) { case HNS3_OL4_TYPE_MAC_IN_UDP: @@ -2370,6 +2460,7 @@ static bool hns3_parse_vlan_tag(struct hns3_enet_ring *ring, struct hns3_desc *desc, u32 l234info, u16 *vlan_tag) { + struct hnae3_handle *handle = ring->tqp->handle; struct pci_dev *pdev = ring->tqp->handle->pdev; if (pdev->revision == 0x20) { @@ -2382,14 +2473,35 @@ static bool hns3_parse_vlan_tag(struct hns3_enet_ring *ring, #define HNS3_STRP_OUTER_VLAN 0x1 #define HNS3_STRP_INNER_VLAN 0x2 +#define HNS3_STRP_BOTH 0x3 + /* Hardware always insert VLAN tag into RX descriptor when + * remove the tag from packet, driver needs to determine + * reporting which tag to stack. + */ switch (hnae3_get_field(l234info, HNS3_RXD_STRP_TAGP_M, HNS3_RXD_STRP_TAGP_S)) { case HNS3_STRP_OUTER_VLAN: + if (handle->port_base_vlan_state != + HNAE3_PORT_BASE_VLAN_DISABLE) + return false; + *vlan_tag = le16_to_cpu(desc->rx.ot_vlan_tag); return true; case HNS3_STRP_INNER_VLAN: + if (handle->port_base_vlan_state != + HNAE3_PORT_BASE_VLAN_DISABLE) + return false; + *vlan_tag = le16_to_cpu(desc->rx.vlan_tag); + return true; + case HNS3_STRP_BOTH: + if (handle->port_base_vlan_state == + HNAE3_PORT_BASE_VLAN_DISABLE) + *vlan_tag = le16_to_cpu(desc->rx.ot_vlan_tag); + else + *vlan_tag = le16_to_cpu(desc->rx.vlan_tag); + return true; default: return false; @@ -2437,7 +2549,7 @@ static int hns3_alloc_skb(struct hns3_enet_ring *ring, int length, ring->stats.seg_pkt_cnt++; u64_stats_update_end(&ring->syncp); - ring->pull_len = eth_get_headlen(va, HNS3_RX_HEAD_SIZE); + ring->pull_len = eth_get_headlen(netdev, va, HNS3_RX_HEAD_SIZE); __skb_put(skb, ring->pull_len); hns3_nic_reuse_page(skb, ring->frag_num++, ring, ring->pull_len, desc_cb); @@ -2512,8 +2624,9 @@ static int hns3_add_frag(struct hns3_enet_ring *ring, struct hns3_desc *desc, return 0; } -static void hns3_set_gro_param(struct sk_buff *skb, u32 l234info, - u32 bd_base_info) +static int hns3_set_gro_and_checksum(struct hns3_enet_ring *ring, + struct sk_buff *skb, u32 l234info, + u32 bd_base_info, u32 ol_info) { u16 gro_count; u32 l3_type; @@ -2521,12 +2634,11 @@ static void hns3_set_gro_param(struct sk_buff *skb, u32 l234info, gro_count = hnae3_get_field(l234info, HNS3_RXD_GRO_COUNT_M, HNS3_RXD_GRO_COUNT_S); /* if there is no HW GRO, do not set gro params */ - if (!gro_count) - return; + if (!gro_count) { + hns3_rx_checksum(ring, skb, l234info, bd_base_info, ol_info); + return 0; + } - /* tcp_gro_complete() will copy NAPI_GRO_CB(skb)->count - * to skb_shinfo(skb)->gso_segs - */ NAPI_GRO_CB(skb)->count = gro_count; l3_type = hnae3_get_field(l234info, HNS3_RXD_L3ID_M, @@ -2536,47 +2648,121 @@ static void hns3_set_gro_param(struct sk_buff *skb, u32 l234info, else if (l3_type == HNS3_L3_TYPE_IPV6) skb_shinfo(skb)->gso_type = SKB_GSO_TCPV6; else - return; + return -EFAULT; skb_shinfo(skb)->gso_size = hnae3_get_field(bd_base_info, HNS3_RXD_GRO_SIZE_M, HNS3_RXD_GRO_SIZE_S); - if (skb_shinfo(skb)->gso_size) - tcp_gro_complete(skb); + + return hns3_gro_complete(skb); } static void hns3_set_rx_skb_rss_type(struct hns3_enet_ring *ring, - struct sk_buff *skb) + struct sk_buff *skb, u32 rss_hash) { struct hnae3_handle *handle = ring->tqp->handle; enum pkt_hash_types rss_type; - struct hns3_desc *desc; - int last_bd; - - /* When driver handle the rss type, ring->next_to_clean indicates the - * first descriptor of next packet, need -1 here. - */ - last_bd = (ring->next_to_clean - 1 + ring->desc_num) % ring->desc_num; - desc = &ring->desc[last_bd]; - if (le32_to_cpu(desc->rx.rss_hash)) + if (rss_hash) rss_type = handle->kinfo.rss_type; else rss_type = PKT_HASH_TYPE_NONE; - skb_set_hash(skb, le32_to_cpu(desc->rx.rss_hash), rss_type); + skb_set_hash(skb, rss_hash, rss_type); } -static int hns3_handle_rx_bd(struct hns3_enet_ring *ring, - struct sk_buff **out_skb) +static int hns3_handle_bdinfo(struct hns3_enet_ring *ring, struct sk_buff *skb) { struct net_device *netdev = ring->tqp->handle->kinfo.netdev; enum hns3_pkt_l2t_type l2_frame_type; + u32 bd_base_info, l234info, ol_info; + struct hns3_desc *desc; + unsigned int len; + int pre_ntc, ret; + + /* bdinfo handled below is only valid on the last BD of the + * current packet, and ring->next_to_clean indicates the first + * descriptor of next packet, so need - 1 below. + */ + pre_ntc = ring->next_to_clean ? (ring->next_to_clean - 1) : + (ring->desc_num - 1); + desc = &ring->desc[pre_ntc]; + bd_base_info = le32_to_cpu(desc->rx.bd_base_info); + l234info = le32_to_cpu(desc->rx.l234_info); + ol_info = le32_to_cpu(desc->rx.ol_info); + + /* Based on hw strategy, the tag offloaded will be stored at + * ot_vlan_tag in two layer tag case, and stored at vlan_tag + * in one layer tag case. + */ + if (netdev->features & NETIF_F_HW_VLAN_CTAG_RX) { + u16 vlan_tag; + + if (hns3_parse_vlan_tag(ring, desc, l234info, &vlan_tag)) + __vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q), + vlan_tag); + } + + if (unlikely(!(bd_base_info & BIT(HNS3_RXD_VLD_B)))) { + u64_stats_update_begin(&ring->syncp); + ring->stats.non_vld_descs++; + u64_stats_update_end(&ring->syncp); + + return -EINVAL; + } + + if (unlikely(!desc->rx.pkt_len || (l234info & (BIT(HNS3_RXD_TRUNCAT_B) | + BIT(HNS3_RXD_L2E_B))))) { + u64_stats_update_begin(&ring->syncp); + if (l234info & BIT(HNS3_RXD_L2E_B)) + ring->stats.l2_err++; + else + ring->stats.err_pkt_len++; + u64_stats_update_end(&ring->syncp); + + return -EFAULT; + } + + len = skb->len; + + /* Do update ip stack process */ + skb->protocol = eth_type_trans(skb, netdev); + + /* This is needed in order to enable forwarding support */ + ret = hns3_set_gro_and_checksum(ring, skb, l234info, + bd_base_info, ol_info); + if (unlikely(ret)) { + u64_stats_update_begin(&ring->syncp); + ring->stats.rx_err_cnt++; + u64_stats_update_end(&ring->syncp); + return ret; + } + + l2_frame_type = hnae3_get_field(l234info, HNS3_RXD_DMAC_M, + HNS3_RXD_DMAC_S); + + u64_stats_update_begin(&ring->syncp); + ring->stats.rx_pkts++; + ring->stats.rx_bytes += len; + + if (l2_frame_type == HNS3_L2_TYPE_MULTICAST) + ring->stats.rx_multicast++; + + u64_stats_update_end(&ring->syncp); + + ring->tqp_vector->rx_group.total_bytes += len; + + hns3_set_rx_skb_rss_type(ring, skb, le32_to_cpu(desc->rx.rss_hash)); + return 0; +} + +static int hns3_handle_rx_bd(struct hns3_enet_ring *ring, + struct sk_buff **out_skb) +{ struct sk_buff *skb = ring->skb; struct hns3_desc_cb *desc_cb; struct hns3_desc *desc; u32 bd_base_info; - u32 l234info; int length; int ret; @@ -2636,64 +2822,13 @@ static int hns3_handle_rx_bd(struct hns3_enet_ring *ring, ALIGN(ring->pull_len, sizeof(long))); } - l234info = le32_to_cpu(desc->rx.l234_info); - bd_base_info = le32_to_cpu(desc->rx.bd_base_info); - - /* Based on hw strategy, the tag offloaded will be stored at - * ot_vlan_tag in two layer tag case, and stored at vlan_tag - * in one layer tag case. - */ - if (netdev->features & NETIF_F_HW_VLAN_CTAG_RX) { - u16 vlan_tag; - - if (hns3_parse_vlan_tag(ring, desc, l234info, &vlan_tag)) - __vlan_hwaccel_put_tag(skb, - htons(ETH_P_8021Q), - vlan_tag); - } - - if (unlikely(!(bd_base_info & BIT(HNS3_RXD_VLD_B)))) { - u64_stats_update_begin(&ring->syncp); - ring->stats.non_vld_descs++; - u64_stats_update_end(&ring->syncp); - - dev_kfree_skb_any(skb); - return -EINVAL; - } - - if (unlikely((!desc->rx.pkt_len) || - (l234info & (BIT(HNS3_RXD_TRUNCAT_B) | - BIT(HNS3_RXD_L2E_B))))) { - u64_stats_update_begin(&ring->syncp); - if (l234info & BIT(HNS3_RXD_L2E_B)) - ring->stats.l2_err++; - else - ring->stats.err_pkt_len++; - u64_stats_update_end(&ring->syncp); - + ret = hns3_handle_bdinfo(ring, skb); + if (unlikely(ret)) { dev_kfree_skb_any(skb); - return -EFAULT; + return ret; } - - l2_frame_type = hnae3_get_field(l234info, HNS3_RXD_DMAC_M, - HNS3_RXD_DMAC_S); - u64_stats_update_begin(&ring->syncp); - if (l2_frame_type == HNS3_L2_TYPE_MULTICAST) - ring->stats.rx_multicast++; - - ring->stats.rx_pkts++; - ring->stats.rx_bytes += skb->len; - u64_stats_update_end(&ring->syncp); - - ring->tqp_vector->rx_group.total_bytes += skb->len; - - /* This is needed in order to enable forwarding support */ - hns3_set_gro_param(skb, l234info, bd_base_info); - - hns3_rx_checksum(ring, skb, desc); *out_skb = skb; - hns3_set_rx_skb_rss_type(ring, skb); return 0; } @@ -2703,9 +2838,8 @@ int hns3_clean_rx_ring( void (*rx_fn)(struct hns3_enet_ring *, struct sk_buff *)) { #define RCB_NOF_ALLOC_RX_BUFF_ONCE 16 - struct net_device *netdev = ring->tqp->handle->kinfo.netdev; int recv_pkts, recv_bds, clean_count, err; - int unused_count = hns3_desc_unused(ring) - ring->pending_buf; + int unused_count = hns3_desc_unused(ring); struct sk_buff *skb = ring->skb; int num; @@ -2714,6 +2848,7 @@ int hns3_clean_rx_ring( recv_pkts = 0, recv_bds = 0, clean_count = 0; num -= unused_count; + unused_count -= ring->pending_buf; while (recv_pkts < budget && recv_bds < num) { /* Reuse or realloc buffers */ @@ -2740,8 +2875,6 @@ int hns3_clean_rx_ring( continue; } - /* Do update ip stack process */ - skb->protocol = eth_type_trans(skb, netdev); rx_fn(ring, skb); recv_bds += ring->pending_buf; clean_count += ring->pending_buf; @@ -2891,7 +3024,7 @@ static int hns3_nic_common_poll(struct napi_struct *napi, int budget) struct hns3_enet_tqp_vector *tqp_vector = container_of(napi, struct hns3_enet_tqp_vector, napi); bool clean_complete = true; - int rx_budget; + int rx_budget = budget; if (unlikely(test_bit(HNS3_NIC_STATE_DOWN, &priv->state))) { napi_complete(napi); @@ -2905,7 +3038,8 @@ static int hns3_nic_common_poll(struct napi_struct *napi, int budget) hns3_clean_tx_ring(ring); /* make sure rx ring budget not smaller than 1 */ - rx_budget = max(budget / tqp_vector->num_tqps, 1); + if (tqp_vector->num_tqps > 1) + rx_budget = max(budget / tqp_vector->num_tqps, 1); hns3_for_each_ring(ring, tqp_vector->rx_group) { int rx_cleaned = hns3_clean_rx_ring(ring, rx_budget, @@ -3316,6 +3450,7 @@ err: } devm_kfree(&pdev->dev, priv->ring_data); + priv->ring_data = NULL; return ret; } @@ -3324,12 +3459,16 @@ static void hns3_put_ring_config(struct hns3_nic_priv *priv) struct hnae3_handle *h = priv->ae_handle; int i; + if (!priv->ring_data) + return; + for (i = 0; i < h->kinfo.num_tqps; i++) { devm_kfree(priv->dev, priv->ring_data[i].ring); devm_kfree(priv->dev, priv->ring_data[i + h->kinfo.num_tqps].ring); } devm_kfree(priv->dev, priv->ring_data); + priv->ring_data = NULL; } static int hns3_alloc_ring_memory(struct hns3_enet_ring *ring) @@ -3339,8 +3478,8 @@ static int hns3_alloc_ring_memory(struct hns3_enet_ring *ring) if (ring->desc_num <= 0 || ring->buf_size <= 0) return -EINVAL; - ring->desc_cb = kcalloc(ring->desc_num, sizeof(ring->desc_cb[0]), - GFP_KERNEL); + ring->desc_cb = devm_kcalloc(ring_to_dev(ring), ring->desc_num, + sizeof(ring->desc_cb[0]), GFP_KERNEL); if (!ring->desc_cb) { ret = -ENOMEM; goto out; @@ -3361,7 +3500,7 @@ static int hns3_alloc_ring_memory(struct hns3_enet_ring *ring) out_with_desc: hns3_free_desc(ring); out_with_desc_cb: - kfree(ring->desc_cb); + devm_kfree(ring_to_dev(ring), ring->desc_cb); ring->desc_cb = NULL; out: return ret; @@ -3370,7 +3509,7 @@ out: static void hns3_fini_ring(struct hns3_enet_ring *ring) { hns3_free_desc(ring); - kfree(ring->desc_cb); + devm_kfree(ring_to_dev(ring), ring->desc_cb); ring->desc_cb = NULL; ring->next_to_clean = 0; ring->next_to_use = 0; @@ -3557,17 +3696,6 @@ static void hns3_del_all_fd_rules(struct net_device *netdev, bool clear_list) h->ae_algo->ops->del_all_fd_entries(h, clear_list); } -static void hns3_nic_set_priv_ops(struct net_device *netdev) -{ - struct hns3_nic_priv *priv = netdev_priv(netdev); - - if ((netdev->features & NETIF_F_TSO) || - (netdev->features & NETIF_F_TSO6)) - priv->ops.maybe_stop_tx = hns3_nic_maybe_stop_tso; - else - priv->ops.maybe_stop_tx = hns3_nic_maybe_stop_tx; -} - static int hns3_client_start(struct hnae3_handle *handle) { if (!handle->ae_algo->ops->client_start) @@ -3584,6 +3712,21 @@ static void hns3_client_stop(struct hnae3_handle *handle) handle->ae_algo->ops->client_stop(handle); } +static void hns3_info_show(struct hns3_nic_priv *priv) +{ + struct hnae3_knic_private_info *kinfo = &priv->ae_handle->kinfo; + + dev_info(priv->dev, "MAC address: %pM\n", priv->netdev->dev_addr); + dev_info(priv->dev, "Task queue pairs numbers: %d\n", kinfo->num_tqps); + dev_info(priv->dev, "RSS size: %d\n", kinfo->rss_size); + dev_info(priv->dev, "Allocated RSS size: %d\n", kinfo->req_rss_size); + dev_info(priv->dev, "RX buffer length: %d\n", kinfo->rx_buf_len); + dev_info(priv->dev, "Desc num per TX queue: %d\n", kinfo->num_tx_desc); + dev_info(priv->dev, "Desc num per RX queue: %d\n", kinfo->num_rx_desc); + dev_info(priv->dev, "Total number of enabled TCs: %d\n", kinfo->num_tc); + dev_info(priv->dev, "Max mtu size: %d\n", priv->netdev->max_mtu); +} + static int hns3_client_init(struct hnae3_handle *handle) { struct pci_dev *pdev = handle->pdev; @@ -3605,6 +3748,8 @@ static int hns3_client_init(struct hnae3_handle *handle) priv->tx_timeout_count = 0; set_bit(HNS3_NIC_STATE_DOWN, &priv->state); + handle->msg_enable = netif_msg_init(debug, DEFAULT_MSG_LEVEL); + handle->kinfo.netdev = netdev; handle->priv = (void *)priv; @@ -3617,7 +3762,6 @@ static int hns3_client_init(struct hnae3_handle *handle) netdev->netdev_ops = &hns3_nic_netdev_ops; SET_NETDEV_DEV(netdev, &pdev->dev); hns3_ethtool_set_ops(netdev); - hns3_nic_set_priv_ops(netdev); /* Carrier off reporting is important to ethtool even BEFORE open */ netif_carrier_off(netdev); @@ -3671,6 +3815,9 @@ static int hns3_client_init(struct hnae3_handle *handle) set_bit(HNS3_NIC_STATE_INITED, &priv->state); + if (netif_msg_drv(handle)) + hns3_info_show(priv); + return ret; out_client_start: @@ -3697,13 +3844,13 @@ static void hns3_client_uninit(struct hnae3_handle *handle, bool reset) struct hns3_nic_priv *priv = netdev_priv(netdev); int ret; - hns3_client_stop(handle); - hns3_remove_hw_addr(netdev); if (netdev->reg_state != NETREG_UNINITIALIZED) unregister_netdev(netdev); + hns3_client_stop(handle); + if (!test_and_clear_bit(HNS3_NIC_STATE_INITED, &priv->state)) { netdev_warn(netdev, "already uninitialized\n"); goto out_netdev_free; @@ -3729,8 +3876,6 @@ static void hns3_client_uninit(struct hnae3_handle *handle, bool reset) hns3_dbg_uninit(handle); - priv->ring_data = NULL; - out_netdev_free: free_netdev(netdev); } @@ -3745,11 +3890,13 @@ static void hns3_link_status_change(struct hnae3_handle *handle, bool linkup) if (linkup) { netif_carrier_on(netdev); netif_tx_wake_all_queues(netdev); - netdev_info(netdev, "link up\n"); + if (netif_msg_link(handle)) + netdev_info(netdev, "link up\n"); } else { netif_carrier_off(netdev); netif_tx_stop_all_queues(netdev); - netdev_info(netdev, "link down\n"); + if (netif_msg_link(handle)) + netdev_info(netdev, "link down\n"); } } @@ -3773,12 +3920,13 @@ static int hns3_recover_hw_addr(struct net_device *ndev) struct netdev_hw_addr *ha, *tmp; int ret = 0; + netif_addr_lock_bh(ndev); /* go through and sync uc_addr entries to the device */ list = &ndev->uc; list_for_each_entry_safe(ha, tmp, &list->list, list) { ret = hns3_nic_uc_sync(ndev, ha->addr); if (ret) - return ret; + goto out; } /* go through and sync mc_addr entries to the device */ @@ -3786,9 +3934,11 @@ static int hns3_recover_hw_addr(struct net_device *ndev) list_for_each_entry_safe(ha, tmp, &list->list, list) { ret = hns3_nic_mc_sync(ndev, ha->addr); if (ret) - return ret; + goto out; } +out: + netif_addr_unlock_bh(ndev); return ret; } @@ -3799,6 +3949,7 @@ static void hns3_remove_hw_addr(struct net_device *netdev) hns3_nic_uc_unsync(netdev, netdev->dev_addr); + netif_addr_lock_bh(netdev); /* go through and unsync uc_addr entries to the device */ list = &netdev->uc; list_for_each_entry_safe(ha, tmp, &list->list, list) @@ -3809,6 +3960,8 @@ static void hns3_remove_hw_addr(struct net_device *netdev) list_for_each_entry_safe(ha, tmp, &list->list, list) if (ha->refcount > 1) hns3_nic_mc_unsync(netdev, ha->addr); + + netif_addr_unlock_bh(netdev); } static void hns3_clear_tx_ring(struct hns3_enet_ring *ring) @@ -3850,6 +4003,13 @@ static int hns3_clear_rx_ring(struct hns3_enet_ring *ring) ring_ptr_move_fw(ring, next_to_use); } + /* Free the pending skb in rx ring */ + if (ring->skb) { + dev_kfree_skb_any(ring->skb); + ring->skb = NULL; + ring->pending_buf = 0; + } + return 0; } @@ -4048,18 +4208,24 @@ static int hns3_reset_notify_init_enet(struct hnae3_handle *handle) if (ret) goto err_uninit_vector; + ret = hns3_client_start(handle); + if (ret) { + dev_err(priv->dev, "hns3_client_start fail! ret=%d\n", ret); + goto err_uninit_ring; + } + set_bit(HNS3_NIC_STATE_INITED, &priv->state); return ret; +err_uninit_ring: + hns3_uninit_all_ring(priv); err_uninit_vector: hns3_nic_uninit_vector_data(priv); - priv->ring_data = NULL; err_dealloc_vector: hns3_nic_dealloc_vector_data(priv); err_put_ring: hns3_put_ring_config(priv); - priv->ring_data = NULL; return ret; } @@ -4101,7 +4267,7 @@ static int hns3_reset_notify_uninit_enet(struct hnae3_handle *handle) struct hns3_nic_priv *priv = netdev_priv(netdev); int ret; - if (!test_bit(HNS3_NIC_STATE_INITED, &priv->state)) { + if (!test_and_clear_bit(HNS3_NIC_STATE_INITED, &priv->state)) { netdev_warn(netdev, "already uninitialized\n"); return 0; } @@ -4121,9 +4287,6 @@ static int hns3_reset_notify_uninit_enet(struct hnae3_handle *handle) netdev_err(netdev, "uninit ring error\n"); hns3_put_ring_config(priv); - priv->ring_data = NULL; - - clear_bit(HNS3_NIC_STATE_INITED, &priv->state); return ret; } diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3_enet.h b/drivers/net/ethernet/hisilicon/hns3/hns3_enet.h index 75669cd0c31145fd763959f226175452dbb399bf..c14480f9b62577766d02e19db427e16aa1a6bee8 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3_enet.h +++ b/drivers/net/ethernet/hisilicon/hns3/hns3_enet.h @@ -42,8 +42,10 @@ enum hns3_nic_state { #define HNS3_RING_TX_RING_HEAD_REG 0x0005C #define HNS3_RING_TX_RING_FBDNUM_REG 0x00060 #define HNS3_RING_TX_RING_OFFSET_REG 0x00064 +#define HNS3_RING_TX_RING_EBDNUM_REG 0x00068 #define HNS3_RING_TX_RING_PKTNUM_RECORD_REG 0x0006C - +#define HNS3_RING_TX_RING_EBD_OFFSET_REG 0x00070 +#define HNS3_RING_TX_RING_BD_ERR_REG 0x00074 #define HNS3_RING_PREFETCH_EN_REG 0x0007C #define HNS3_RING_CFG_VF_NUM_REG 0x00080 #define HNS3_RING_ASID_REG 0x0008C @@ -374,6 +376,7 @@ struct ring_stats { u64 tx_err_cnt; u64 restart_queue; u64 tx_busy; + u64 tx_copy; }; struct { u64 rx_pkts; @@ -386,6 +389,7 @@ struct ring_stats { u64 l2_err; u64 l3l4_csum_err; u64 rx_multicast; + u64 non_reuse_pg; }; }; }; @@ -397,7 +401,6 @@ struct hns3_enet_ring { struct hns3_enet_ring *next; struct hns3_enet_tqp_vector *tqp_vector; struct hnae3_queue *tqp; - char ring_name[HNS3_RING_NAME_LEN]; struct device *dev; /* will be used for DMA mapping of descriptors */ /* statistic */ @@ -407,9 +410,6 @@ struct hns3_enet_ring { dma_addr_t desc_dma_addr; u32 buf_size; /* size for hnae_desc->addr, preset by AE */ u16 desc_num; /* total number of desc */ - u16 max_desc_num_per_pkt; - u16 max_raw_data_sz_per_desc; - u16 max_pkt_size; int next_to_use; /* idx of next spare desc */ /* idx of lastest sent desc, the ring is empty when equal to @@ -423,9 +423,6 @@ struct hns3_enet_ring { u32 flag; /* ring attribute */ - int numa_node; - cpumask_t affinity_mask; - int pending_buf; struct sk_buff *skb; struct sk_buff *tail_skb; @@ -442,11 +439,6 @@ struct hns3_nic_ring_data { void (*fini_process)(struct hns3_nic_ring_data *); }; -struct hns3_nic_ops { - int (*maybe_stop_tx)(struct sk_buff **out_skb, - int *bnum, struct hns3_enet_ring *ring); -}; - enum hns3_flow_level_range { HNS3_FLOW_LOW = 0, HNS3_FLOW_MID = 1, @@ -536,7 +528,6 @@ struct hns3_nic_priv { u32 port_id; struct net_device *netdev; struct device *dev; - struct hns3_nic_ops ops; /** * the cb for nic to manage the ring buffer, the first half of the @@ -577,18 +568,16 @@ union l4_hdr_info { unsigned char *hdr; }; -/* the distance between [begin, end) in a ring buffer - * note: there is a unuse slot between the begin and the end - */ -static inline int ring_dist(struct hns3_enet_ring *ring, int begin, int end) -{ - return (end - begin + ring->desc_num) % ring->desc_num; -} - static inline int ring_space(struct hns3_enet_ring *ring) { - return ring->desc_num - - ring_dist(ring, ring->next_to_clean, ring->next_to_use) - 1; + /* This smp_load_acquire() pairs with smp_store_release() in + * hns3_nic_reclaim_one_desc called by hns3_clean_tx_ring. + */ + int begin = smp_load_acquire(&ring->next_to_clean); + int end = READ_ONCE(ring->next_to_use); + + return ((end >= begin) ? (ring->desc_num - end + begin) : + (begin - end)) - 1; } static inline int is_ring_empty(struct hns3_enet_ring *ring) @@ -633,7 +622,7 @@ static inline bool hns3_nic_resetting(struct net_device *netdev) #define hnae3_queue_xmit(tqp, buf_num) writel_relaxed(buf_num, \ (tqp)->io_base + HNS3_RING_TX_RING_TAIL_REG) -#define ring_to_dev(ring) (&(ring)->tqp->handle->pdev->dev) +#define ring_to_dev(ring) ((ring)->dev) #define ring_to_dma_dir(ring) (HNAE3_IS_TX_RING(ring) ? \ DMA_TO_DEVICE : DMA_FROM_DEVICE) @@ -666,6 +655,7 @@ int hns3_init_all_ring(struct hns3_nic_priv *priv); int hns3_uninit_all_ring(struct hns3_nic_priv *priv); int hns3_nic_reset_all_ring(struct hnae3_handle *h); netdev_tx_t hns3_nic_net_xmit(struct sk_buff *skb, struct net_device *netdev); +bool hns3_is_phys_func(struct pci_dev *pdev); int hns3_clean_rx_ring( struct hns3_enet_ring *ring, int budget, void (*rx_fn)(struct hns3_enet_ring *, struct sk_buff *)); diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3_ethtool.c b/drivers/net/ethernet/hisilicon/hns3/hns3_ethtool.c index 359d4731fb2db5c6a3bb112d8136669736533137..d1588ea6132c1cd021ff2a3240c28448e015cf39 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3_ethtool.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3_ethtool.c @@ -29,6 +29,7 @@ static const struct hns3_stats hns3_txq_stats[] = { HNS3_TQP_STAT("errors", tx_err_cnt), HNS3_TQP_STAT("wake", restart_queue), HNS3_TQP_STAT("busy", tx_busy), + HNS3_TQP_STAT("copy", tx_copy), }; #define HNS3_TXQ_STATS_COUNT ARRAY_SIZE(hns3_txq_stats) @@ -48,6 +49,7 @@ static const struct hns3_stats hns3_rxq_stats[] = { HNS3_TQP_STAT("l2_err", l2_err), HNS3_TQP_STAT("l3l4_csum_err", l3l4_csum_err), HNS3_TQP_STAT("multicast", rx_multicast), + HNS3_TQP_STAT("non_reuse_pg", non_reuse_pg), }; #define HNS3_RXQ_STATS_COUNT ARRAY_SIZE(hns3_rxq_stats) @@ -483,6 +485,11 @@ static void hns3_get_stats(struct net_device *netdev, struct hnae3_handle *h = hns3_get_handle(netdev); u64 *p = data; + if (hns3_nic_resetting(netdev)) { + netdev_err(netdev, "dev resetting, could not get stats\n"); + return; + } + if (!h->ae_algo->ops->get_stats || !h->ae_algo->ops->update_stats) { netdev_err(netdev, "could not get any statistics\n"); return; @@ -599,6 +606,7 @@ static int hns3_get_link_ksettings(struct net_device *netdev, { struct hnae3_handle *h = hns3_get_handle(netdev); const struct hnae3_ae_ops *ops; + u8 module_type; u8 media_type; u8 link_stat; @@ -607,7 +615,7 @@ static int hns3_get_link_ksettings(struct net_device *netdev, ops = h->ae_algo->ops; if (ops->get_media_type) - ops->get_media_type(h, &media_type); + ops->get_media_type(h, &media_type, &module_type); else return -EOPNOTSUPP; @@ -617,7 +625,15 @@ static int hns3_get_link_ksettings(struct net_device *netdev, hns3_get_ksettings(h, cmd); break; case HNAE3_MEDIA_TYPE_FIBER: - cmd->base.port = PORT_FIBRE; + if (module_type == HNAE3_MODULE_TYPE_CR) + cmd->base.port = PORT_DA; + else + cmd->base.port = PORT_FIBRE; + + hns3_get_ksettings(h, cmd); + break; + case HNAE3_MEDIA_TYPE_BACKPLANE: + cmd->base.port = PORT_NONE; hns3_get_ksettings(h, cmd); break; case HNAE3_MEDIA_TYPE_COPPER: @@ -645,14 +661,79 @@ static int hns3_get_link_ksettings(struct net_device *netdev, return 0; } +static int hns3_check_ksettings_param(struct net_device *netdev, + const struct ethtool_link_ksettings *cmd) +{ + struct hnae3_handle *handle = hns3_get_handle(netdev); + const struct hnae3_ae_ops *ops = handle->ae_algo->ops; + u8 module_type = HNAE3_MODULE_TYPE_UNKNOWN; + u8 media_type = HNAE3_MEDIA_TYPE_UNKNOWN; + u8 autoneg; + u32 speed; + u8 duplex; + int ret; + + if (ops->get_ksettings_an_result) { + ops->get_ksettings_an_result(handle, &autoneg, &speed, &duplex); + if (cmd->base.autoneg == autoneg && cmd->base.speed == speed && + cmd->base.duplex == duplex) + return 0; + } + + if (ops->get_media_type) + ops->get_media_type(handle, &media_type, &module_type); + + if (cmd->base.duplex != DUPLEX_FULL && + media_type != HNAE3_MEDIA_TYPE_COPPER) { + netdev_err(netdev, + "only copper port supports half duplex!"); + return -EINVAL; + } + + if (ops->check_port_speed) { + ret = ops->check_port_speed(handle, cmd->base.speed); + if (ret) { + netdev_err(netdev, "unsupported speed\n"); + return ret; + } + } + + return 0; +} + static int hns3_set_link_ksettings(struct net_device *netdev, const struct ethtool_link_ksettings *cmd) { + struct hnae3_handle *handle = hns3_get_handle(netdev); + const struct hnae3_ae_ops *ops = handle->ae_algo->ops; + int ret = 0; + + /* Chip don't support this mode. */ + if (cmd->base.speed == SPEED_1000 && cmd->base.duplex == DUPLEX_HALF) + return -EINVAL; + /* Only support ksettings_set for netdev with phy attached for now */ if (netdev->phydev) return phy_ethtool_ksettings_set(netdev->phydev, cmd); - return -EOPNOTSUPP; + if (handle->pdev->revision == 0x20) + return -EOPNOTSUPP; + + ret = hns3_check_ksettings_param(netdev, cmd); + if (ret) + return ret; + + if (ops->set_autoneg) { + ret = ops->set_autoneg(handle, cmd->base.autoneg); + if (ret) + return ret; + } + + if (ops->cfg_mac_speed_dup_h) + ret = ops->cfg_mac_speed_dup_h(handle, cmd->base.speed, + cmd->base.duplex); + + return ret; } static u32 hns3_get_rss_key_size(struct net_device *netdev) @@ -857,19 +938,36 @@ static int hns3_set_rxnfc(struct net_device *netdev, struct ethtool_rxnfc *cmd) static int hns3_nway_reset(struct net_device *netdev) { + struct hnae3_handle *handle = hns3_get_handle(netdev); + const struct hnae3_ae_ops *ops = handle->ae_algo->ops; struct phy_device *phy = netdev->phydev; + int autoneg; if (!netif_running(netdev)) return 0; - /* Only support nway_reset for netdev with phy attached for now */ - if (!phy) + if (hns3_nic_resetting(netdev)) { + netdev_err(netdev, "dev resetting!"); + return -EBUSY; + } + + if (!ops->get_autoneg || !ops->restart_autoneg) return -EOPNOTSUPP; - if (phy->autoneg != AUTONEG_ENABLE) + autoneg = ops->get_autoneg(handle); + if (autoneg != AUTONEG_ENABLE) { + netdev_err(netdev, + "Autoneg is off, don't support to restart it\n"); return -EINVAL; + } + + if (phy) + return genphy_restart_aneg(phy); + + if (handle->pdev->revision == 0x20) + return -EOPNOTSUPP; - return genphy_restart_aneg(phy); + return ops->restart_autoneg(handle); } static void hns3_get_channels(struct net_device *netdev, @@ -1101,6 +1199,95 @@ static int hns3_set_phys_id(struct net_device *netdev, return h->ae_algo->ops->set_led_id(h, state); } +static u32 hns3_get_msglevel(struct net_device *netdev) +{ + struct hnae3_handle *h = hns3_get_handle(netdev); + + return h->msg_enable; +} + +static void hns3_set_msglevel(struct net_device *netdev, u32 msg_level) +{ + struct hnae3_handle *h = hns3_get_handle(netdev); + + h->msg_enable = msg_level; +} + +/* Translate local fec value into ethtool value. */ +static unsigned int loc_to_eth_fec(u8 loc_fec) +{ + u32 eth_fec = 0; + + if (loc_fec & BIT(HNAE3_FEC_AUTO)) + eth_fec |= ETHTOOL_FEC_AUTO; + if (loc_fec & BIT(HNAE3_FEC_RS)) + eth_fec |= ETHTOOL_FEC_RS; + if (loc_fec & BIT(HNAE3_FEC_BASER)) + eth_fec |= ETHTOOL_FEC_BASER; + + /* if nothing is set, then FEC is off */ + if (!eth_fec) + eth_fec = ETHTOOL_FEC_OFF; + + return eth_fec; +} + +/* Translate ethtool fec value into local value. */ +static unsigned int eth_to_loc_fec(unsigned int eth_fec) +{ + u32 loc_fec = 0; + + if (eth_fec & ETHTOOL_FEC_OFF) + return loc_fec; + + if (eth_fec & ETHTOOL_FEC_AUTO) + loc_fec |= BIT(HNAE3_FEC_AUTO); + if (eth_fec & ETHTOOL_FEC_RS) + loc_fec |= BIT(HNAE3_FEC_RS); + if (eth_fec & ETHTOOL_FEC_BASER) + loc_fec |= BIT(HNAE3_FEC_BASER); + + return loc_fec; +} + +static int hns3_get_fecparam(struct net_device *netdev, + struct ethtool_fecparam *fec) +{ + struct hnae3_handle *handle = hns3_get_handle(netdev); + const struct hnae3_ae_ops *ops = handle->ae_algo->ops; + u8 fec_ability; + u8 fec_mode; + + if (handle->pdev->revision == 0x20) + return -EOPNOTSUPP; + + if (!ops->get_fec) + return -EOPNOTSUPP; + + ops->get_fec(handle, &fec_ability, &fec_mode); + + fec->fec = loc_to_eth_fec(fec_ability); + fec->active_fec = loc_to_eth_fec(fec_mode); + + return 0; +} + +static int hns3_set_fecparam(struct net_device *netdev, + struct ethtool_fecparam *fec) +{ + struct hnae3_handle *handle = hns3_get_handle(netdev); + const struct hnae3_ae_ops *ops = handle->ae_algo->ops; + u32 fec_mode; + + if (handle->pdev->revision == 0x20) + return -EOPNOTSUPP; + + if (!ops->set_fec) + return -EOPNOTSUPP; + fec_mode = eth_to_loc_fec(fec->fec); + return ops->set_fec(handle, fec_mode); +} + static const struct ethtool_ops hns3vf_ethtool_ops = { .get_drvinfo = hns3_get_drvinfo, .get_ringparam = hns3_get_ringparam, @@ -1121,6 +1308,8 @@ static const struct ethtool_ops hns3vf_ethtool_ops = { .get_regs_len = hns3_get_regs_len, .get_regs = hns3_get_regs, .get_link = hns3_get_link, + .get_msglevel = hns3_get_msglevel, + .set_msglevel = hns3_set_msglevel, }; static const struct ethtool_ops hns3_ethtool_ops = { @@ -1150,6 +1339,10 @@ static const struct ethtool_ops hns3_ethtool_ops = { .get_regs_len = hns3_get_regs_len, .get_regs = hns3_get_regs, .set_phys_id = hns3_set_phys_id, + .get_msglevel = hns3_get_msglevel, + .set_msglevel = hns3_set_msglevel, + .get_fecparam = hns3_get_fecparam, + .set_fecparam = hns3_set_fecparam, }; void hns3_ethtool_set_ops(struct net_device *netdev) diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_cmd.c b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_cmd.c index 3a093a92eac516ac40d12309371c3780dc7b20f4..fbd904e3077cf6219b270df37a0058140a946784 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_cmd.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_cmd.c @@ -355,7 +355,7 @@ int hclge_cmd_init(struct hclge_dev *hdev) int ret; spin_lock_bh(&hdev->hw.cmq.csq.lock); - spin_lock_bh(&hdev->hw.cmq.crq.lock); + spin_lock(&hdev->hw.cmq.crq.lock); hdev->hw.cmq.csq.next_to_clean = 0; hdev->hw.cmq.csq.next_to_use = 0; @@ -364,7 +364,7 @@ int hclge_cmd_init(struct hclge_dev *hdev) hclge_cmd_init_regs(&hdev->hw); - spin_unlock_bh(&hdev->hw.cmq.crq.lock); + spin_unlock(&hdev->hw.cmq.crq.lock); spin_unlock_bh(&hdev->hw.cmq.csq.lock); clear_bit(HCLGE_STATE_CMD_DISABLE, &hdev->state); @@ -373,21 +373,26 @@ int hclge_cmd_init(struct hclge_dev *hdev) * reset may happen when lower level reset is being processed. */ if ((hclge_is_reset_pending(hdev))) { - set_bit(HCLGE_STATE_CMD_DISABLE, &hdev->state); - return -EBUSY; + ret = -EBUSY; + goto err_cmd_init; } ret = hclge_cmd_query_firmware_version(&hdev->hw, &version); if (ret) { dev_err(&hdev->pdev->dev, "firmware version query failed %d\n", ret); - return ret; + goto err_cmd_init; } hdev->fw_version = version; dev_info(&hdev->pdev->dev, "The firmware version is %08x\n", version); return 0; + +err_cmd_init: + set_bit(HCLGE_STATE_CMD_DISABLE, &hdev->state); + + return ret; } static void hclge_cmd_uninit_regs(struct hclge_hw *hw) @@ -411,7 +416,7 @@ static void hclge_destroy_queue(struct hclge_cmq_ring *ring) spin_unlock(&ring->lock); } -void hclge_destroy_cmd_queue(struct hclge_hw *hw) +static void hclge_destroy_cmd_queue(struct hclge_hw *hw) { hclge_destroy_queue(&hw->cmq.csq); hclge_destroy_queue(&hw->cmq.crq); diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_cmd.h b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_cmd.h index 3714733c96d99260bb228fc548801fdfa8ce7240..d79a209b80f62f9a2fb5faf22a83937ff9e7f659 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_cmd.h +++ b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_cmd.h @@ -109,7 +109,11 @@ enum hclge_opcode_type { HCLGE_OPC_QUERY_LINK_STATUS = 0x0307, HCLGE_OPC_CONFIG_MAX_FRM_SIZE = 0x0308, HCLGE_OPC_CONFIG_SPEED_DUP = 0x0309, + HCLGE_OPC_QUERY_MAC_TNL_INT = 0x0310, + HCLGE_OPC_MAC_TNL_INT_EN = 0x0311, + HCLGE_OPC_CLEAR_MAC_TNL_INT = 0x0312, HCLGE_OPC_SERDES_LOOPBACK = 0x0315, + HCLGE_OPC_CONFIG_FEC_MODE = 0x031A, /* PFC/Pause commands */ HCLGE_OPC_CFG_MAC_PAUSE_EN = 0x0701, @@ -237,8 +241,11 @@ enum hclge_opcode_type { /* Led command */ HCLGE_OPC_LED_STATUS_CFG = 0xB000, + /* NCL config command */ + HCLGE_OPC_QUERY_NCL_CONFIG = 0x7011, + /* SFP command */ - HCLGE_OPC_SFP_GET_SPEED = 0x7104, + HCLGE_OPC_GET_SFP_INFO = 0x7104, /* Error INT commands */ HCLGE_MAC_COMMON_INT_EN = 0x030E, @@ -593,9 +600,30 @@ struct hclge_config_auto_neg_cmd { u8 rsv[20]; }; -struct hclge_sfp_speed_cmd { - __le32 sfp_speed; - u32 rsv[5]; +struct hclge_sfp_info_cmd { + __le32 speed; + u8 query_type; /* 0: sfp speed, 1: active speed */ + u8 active_fec; + u8 autoneg; /* autoneg state */ + u8 autoneg_ability; /* whether support autoneg */ + __le32 speed_ability; /* speed ability for current media */ + __le32 module_type; + u8 rsv[8]; +}; + +#define HCLGE_MAC_CFG_FEC_AUTO_EN_B 0 +#define HCLGE_MAC_CFG_FEC_MODE_S 1 +#define HCLGE_MAC_CFG_FEC_MODE_M GENMASK(3, 1) +#define HCLGE_MAC_CFG_FEC_SET_DEF_B 0 +#define HCLGE_MAC_CFG_FEC_CLR_DEF_B 1 + +#define HCLGE_MAC_FEC_OFF 0 +#define HCLGE_MAC_FEC_BASER 1 +#define HCLGE_MAC_FEC_RS 2 +struct hclge_config_fec_cmd { + u8 fec_mode; + u8 default_config; + u8 rsv[22]; }; #define HCLGE_MAC_UPLINK_PORT 0x100 diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_debugfs.c b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_debugfs.c index 1192cf6f2321dadcb80da2e4a168cbc248d1d76c..a9ffb57c4607297f023c97e74cda46b331fb2c2c 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_debugfs.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_debugfs.c @@ -901,6 +901,109 @@ static void hclge_dbg_fd_tcam(struct hclge_dev *hdev) } } +static void hclge_dbg_dump_rst_info(struct hclge_dev *hdev) +{ + dev_info(&hdev->pdev->dev, "PF reset count: %d\n", + hdev->rst_stats.pf_rst_cnt); + dev_info(&hdev->pdev->dev, "FLR reset count: %d\n", + hdev->rst_stats.flr_rst_cnt); + dev_info(&hdev->pdev->dev, "CORE reset count: %d\n", + hdev->rst_stats.core_rst_cnt); + dev_info(&hdev->pdev->dev, "GLOBAL reset count: %d\n", + hdev->rst_stats.global_rst_cnt); + dev_info(&hdev->pdev->dev, "IMP reset count: %d\n", + hdev->rst_stats.imp_rst_cnt); + dev_info(&hdev->pdev->dev, "reset done count: %d\n", + hdev->rst_stats.reset_done_cnt); + dev_info(&hdev->pdev->dev, "HW reset done count: %d\n", + hdev->rst_stats.hw_reset_done_cnt); + dev_info(&hdev->pdev->dev, "reset count: %d\n", + hdev->rst_stats.reset_cnt); +} + +/* hclge_dbg_dump_ncl_config: print specified range of NCL_CONFIG file + * @hdev: pointer to struct hclge_dev + * @cmd_buf: string that contains offset and length + */ +static void hclge_dbg_dump_ncl_config(struct hclge_dev *hdev, char *cmd_buf) +{ +#define HCLGE_MAX_NCL_CONFIG_OFFSET 4096 +#define HCLGE_MAX_NCL_CONFIG_LENGTH (20 + 24 * 4) +#define HCLGE_CMD_DATA_NUM 6 + + struct hclge_desc desc[5]; + u32 byte_offset; + int bd_num = 5; + int offset; + int length; + int data0; + int ret; + int i; + int j; + + ret = sscanf(cmd_buf, "%x %x", &offset, &length); + if (ret != 2 || offset >= HCLGE_MAX_NCL_CONFIG_OFFSET || + length > HCLGE_MAX_NCL_CONFIG_OFFSET - offset) { + dev_err(&hdev->pdev->dev, "Invalid offset or length.\n"); + return; + } + if (offset < 0 || length <= 0) { + dev_err(&hdev->pdev->dev, "Non-positive offset or length.\n"); + return; + } + + dev_info(&hdev->pdev->dev, "offset | data\n"); + + while (length > 0) { + data0 = offset; + if (length >= HCLGE_MAX_NCL_CONFIG_LENGTH) + data0 |= HCLGE_MAX_NCL_CONFIG_LENGTH << 16; + else + data0 |= length << 16; + ret = hclge_dbg_cmd_send(hdev, desc, data0, bd_num, + HCLGE_OPC_QUERY_NCL_CONFIG); + if (ret) + return; + + byte_offset = offset; + for (i = 0; i < bd_num; i++) { + for (j = 0; j < HCLGE_CMD_DATA_NUM; j++) { + if (i == 0 && j == 0) + continue; + + dev_info(&hdev->pdev->dev, "0x%04x | 0x%08x\n", + byte_offset, + le32_to_cpu(desc[i].data[j])); + byte_offset += sizeof(u32); + length -= sizeof(u32); + if (length <= 0) + return; + } + } + offset += HCLGE_MAX_NCL_CONFIG_LENGTH; + } +} + +/* hclge_dbg_dump_mac_tnl_status: print message about mac tnl interrupt + * @hdev: pointer to struct hclge_dev + */ +static void hclge_dbg_dump_mac_tnl_status(struct hclge_dev *hdev) +{ +#define HCLGE_BILLION_NANO_SECONDS 1000000000 + + struct hclge_mac_tnl_stats stats; + unsigned long rem_nsec; + + dev_info(&hdev->pdev->dev, "Recently generated mac tnl interruption:\n"); + + while (kfifo_get(&hdev->mac_tnl_log, &stats)) { + rem_nsec = do_div(stats.time, HCLGE_BILLION_NANO_SECONDS); + dev_info(&hdev->pdev->dev, "[%07lu.%03lu]status = 0x%x\n", + (unsigned long)stats.time, rem_nsec / 1000, + stats.status); + } +} + int hclge_dbg_run_cmd(struct hnae3_handle *handle, char *cmd_buf) { struct hclge_vport *vport = hclge_get_vport(handle); @@ -924,6 +1027,13 @@ int hclge_dbg_run_cmd(struct hnae3_handle *handle, char *cmd_buf) hclge_dbg_dump_mng_table(hdev); } else if (strncmp(cmd_buf, "dump reg", 8) == 0) { hclge_dbg_dump_reg_cmd(hdev, cmd_buf); + } else if (strncmp(cmd_buf, "dump reset info", 15) == 0) { + hclge_dbg_dump_rst_info(hdev); + } else if (strncmp(cmd_buf, "dump ncl_config", 15) == 0) { + hclge_dbg_dump_ncl_config(hdev, + &cmd_buf[sizeof("dump ncl_config")]); + } else if (strncmp(cmd_buf, "dump mac tnl status", 19) == 0) { + hclge_dbg_dump_mac_tnl_status(hdev); } else { dev_info(&hdev->pdev->dev, "unknown command\n"); return -EINVAL; diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_err.c b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_err.c index 1f52d11f77b55c58d837918f384e172158caaca0..4ac80634c984994c7e53a6b8ffc2ca0836835b4b 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_err.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_err.c @@ -4,287 +4,468 @@ #include "hclge_err.h" static const struct hclge_hw_error hclge_imp_tcm_ecc_int[] = { - { .int_msk = BIT(1), .msg = "imp_itcm0_ecc_mbit_err" }, - { .int_msk = BIT(3), .msg = "imp_itcm1_ecc_mbit_err" }, - { .int_msk = BIT(5), .msg = "imp_itcm2_ecc_mbit_err" }, - { .int_msk = BIT(7), .msg = "imp_itcm3_ecc_mbit_err" }, - { .int_msk = BIT(9), .msg = "imp_dtcm0_mem0_ecc_mbit_err" }, - { .int_msk = BIT(11), .msg = "imp_dtcm0_mem1_ecc_mbit_err" }, - { .int_msk = BIT(13), .msg = "imp_dtcm1_mem0_ecc_mbit_err" }, - { .int_msk = BIT(15), .msg = "imp_dtcm1_mem1_ecc_mbit_err" }, - { .int_msk = BIT(17), .msg = "imp_itcm4_ecc_mbit_err" }, + { .int_msk = BIT(1), .msg = "imp_itcm0_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(3), .msg = "imp_itcm1_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(5), .msg = "imp_itcm2_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(7), .msg = "imp_itcm3_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(9), .msg = "imp_dtcm0_mem0_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(11), .msg = "imp_dtcm0_mem1_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(13), .msg = "imp_dtcm1_mem0_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(15), .msg = "imp_dtcm1_mem1_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(17), .msg = "imp_itcm4_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, { /* sentinel */ } }; static const struct hclge_hw_error hclge_cmdq_nic_mem_ecc_int[] = { - { .int_msk = BIT(1), .msg = "cmdq_nic_rx_depth_ecc_mbit_err" }, - { .int_msk = BIT(3), .msg = "cmdq_nic_tx_depth_ecc_mbit_err" }, - { .int_msk = BIT(5), .msg = "cmdq_nic_rx_tail_ecc_mbit_err" }, - { .int_msk = BIT(7), .msg = "cmdq_nic_tx_tail_ecc_mbit_err" }, - { .int_msk = BIT(9), .msg = "cmdq_nic_rx_head_ecc_mbit_err" }, - { .int_msk = BIT(11), .msg = "cmdq_nic_tx_head_ecc_mbit_err" }, - { .int_msk = BIT(13), .msg = "cmdq_nic_rx_addr_ecc_mbit_err" }, - { .int_msk = BIT(15), .msg = "cmdq_nic_tx_addr_ecc_mbit_err" }, - { .int_msk = BIT(17), .msg = "cmdq_rocee_rx_depth_ecc_mbit_err" }, - { .int_msk = BIT(19), .msg = "cmdq_rocee_tx_depth_ecc_mbit_err" }, - { .int_msk = BIT(21), .msg = "cmdq_rocee_rx_tail_ecc_mbit_err" }, - { .int_msk = BIT(23), .msg = "cmdq_rocee_tx_tail_ecc_mbit_err" }, - { .int_msk = BIT(25), .msg = "cmdq_rocee_rx_head_ecc_mbit_err" }, - { .int_msk = BIT(27), .msg = "cmdq_rocee_tx_head_ecc_mbit_err" }, - { .int_msk = BIT(29), .msg = "cmdq_rocee_rx_addr_ecc_mbit_err" }, - { .int_msk = BIT(31), .msg = "cmdq_rocee_tx_addr_ecc_mbit_err" }, + { .int_msk = BIT(1), .msg = "cmdq_nic_rx_depth_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(3), .msg = "cmdq_nic_tx_depth_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(5), .msg = "cmdq_nic_rx_tail_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(7), .msg = "cmdq_nic_tx_tail_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(9), .msg = "cmdq_nic_rx_head_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(11), .msg = "cmdq_nic_tx_head_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(13), .msg = "cmdq_nic_rx_addr_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(15), .msg = "cmdq_nic_tx_addr_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(17), .msg = "cmdq_rocee_rx_depth_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(19), .msg = "cmdq_rocee_tx_depth_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(21), .msg = "cmdq_rocee_rx_tail_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(23), .msg = "cmdq_rocee_tx_tail_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(25), .msg = "cmdq_rocee_rx_head_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(27), .msg = "cmdq_rocee_tx_head_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(29), .msg = "cmdq_rocee_rx_addr_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(31), .msg = "cmdq_rocee_tx_addr_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, { /* sentinel */ } }; static const struct hclge_hw_error hclge_tqp_int_ecc_int[] = { - { .int_msk = BIT(6), .msg = "tqp_int_cfg_even_ecc_mbit_err" }, - { .int_msk = BIT(7), .msg = "tqp_int_cfg_odd_ecc_mbit_err" }, - { .int_msk = BIT(8), .msg = "tqp_int_ctrl_even_ecc_mbit_err" }, - { .int_msk = BIT(9), .msg = "tqp_int_ctrl_odd_ecc_mbit_err" }, - { .int_msk = BIT(10), .msg = "tx_que_scan_int_ecc_mbit_err" }, - { .int_msk = BIT(11), .msg = "rx_que_scan_int_ecc_mbit_err" }, + { .int_msk = BIT(6), .msg = "tqp_int_cfg_even_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(7), .msg = "tqp_int_cfg_odd_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(8), .msg = "tqp_int_ctrl_even_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(9), .msg = "tqp_int_ctrl_odd_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(10), .msg = "tx_que_scan_int_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(11), .msg = "rx_que_scan_int_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, { /* sentinel */ } }; static const struct hclge_hw_error hclge_msix_sram_ecc_int[] = { - { .int_msk = BIT(1), .msg = "msix_nic_ecc_mbit_err" }, - { .int_msk = BIT(3), .msg = "msix_rocee_ecc_mbit_err" }, + { .int_msk = BIT(1), .msg = "msix_nic_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(3), .msg = "msix_rocee_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, { /* sentinel */ } }; static const struct hclge_hw_error hclge_igu_int[] = { - { .int_msk = BIT(0), .msg = "igu_rx_buf0_ecc_mbit_err" }, - { .int_msk = BIT(2), .msg = "igu_rx_buf1_ecc_mbit_err" }, + { .int_msk = BIT(0), .msg = "igu_rx_buf0_ecc_mbit_err", + .reset_level = HNAE3_CORE_RESET }, + { .int_msk = BIT(2), .msg = "igu_rx_buf1_ecc_mbit_err", + .reset_level = HNAE3_CORE_RESET }, { /* sentinel */ } }; static const struct hclge_hw_error hclge_igu_egu_tnl_int[] = { - { .int_msk = BIT(0), .msg = "rx_buf_overflow" }, - { .int_msk = BIT(1), .msg = "rx_stp_fifo_overflow" }, - { .int_msk = BIT(2), .msg = "rx_stp_fifo_undeflow" }, - { .int_msk = BIT(3), .msg = "tx_buf_overflow" }, - { .int_msk = BIT(4), .msg = "tx_buf_underrun" }, - { .int_msk = BIT(5), .msg = "rx_stp_buf_overflow" }, + { .int_msk = BIT(0), .msg = "rx_buf_overflow", + .reset_level = HNAE3_CORE_RESET }, + { .int_msk = BIT(1), .msg = "rx_stp_fifo_overflow", + .reset_level = HNAE3_CORE_RESET }, + { .int_msk = BIT(2), .msg = "rx_stp_fifo_undeflow", + .reset_level = HNAE3_CORE_RESET }, + { .int_msk = BIT(3), .msg = "tx_buf_overflow", + .reset_level = HNAE3_CORE_RESET }, + { .int_msk = BIT(4), .msg = "tx_buf_underrun", + .reset_level = HNAE3_CORE_RESET }, + { .int_msk = BIT(5), .msg = "rx_stp_buf_overflow", + .reset_level = HNAE3_CORE_RESET }, { /* sentinel */ } }; static const struct hclge_hw_error hclge_ncsi_err_int[] = { - { .int_msk = BIT(1), .msg = "ncsi_tx_ecc_mbit_err" }, + { .int_msk = BIT(1), .msg = "ncsi_tx_ecc_mbit_err", + .reset_level = HNAE3_NONE_RESET }, { /* sentinel */ } }; static const struct hclge_hw_error hclge_ppp_mpf_abnormal_int_st1[] = { - { .int_msk = BIT(0), .msg = "vf_vlan_ad_mem_ecc_mbit_err" }, - { .int_msk = BIT(1), .msg = "umv_mcast_group_mem_ecc_mbit_err" }, - { .int_msk = BIT(2), .msg = "umv_key_mem0_ecc_mbit_err" }, - { .int_msk = BIT(3), .msg = "umv_key_mem1_ecc_mbit_err" }, - { .int_msk = BIT(4), .msg = "umv_key_mem2_ecc_mbit_err" }, - { .int_msk = BIT(5), .msg = "umv_key_mem3_ecc_mbit_err" }, - { .int_msk = BIT(6), .msg = "umv_ad_mem_ecc_mbit_err" }, - { .int_msk = BIT(7), .msg = "rss_tc_mode_mem_ecc_mbit_err" }, - { .int_msk = BIT(8), .msg = "rss_idt_mem0_ecc_mbit_err" }, - { .int_msk = BIT(9), .msg = "rss_idt_mem1_ecc_mbit_err" }, - { .int_msk = BIT(10), .msg = "rss_idt_mem2_ecc_mbit_err" }, - { .int_msk = BIT(11), .msg = "rss_idt_mem3_ecc_mbit_err" }, - { .int_msk = BIT(12), .msg = "rss_idt_mem4_ecc_mbit_err" }, - { .int_msk = BIT(13), .msg = "rss_idt_mem5_ecc_mbit_err" }, - { .int_msk = BIT(14), .msg = "rss_idt_mem6_ecc_mbit_err" }, - { .int_msk = BIT(15), .msg = "rss_idt_mem7_ecc_mbit_err" }, - { .int_msk = BIT(16), .msg = "rss_idt_mem8_ecc_mbit_err" }, - { .int_msk = BIT(17), .msg = "rss_idt_mem9_ecc_mbit_err" }, - { .int_msk = BIT(18), .msg = "rss_idt_mem10_ecc_m1bit_err" }, - { .int_msk = BIT(19), .msg = "rss_idt_mem11_ecc_mbit_err" }, - { .int_msk = BIT(20), .msg = "rss_idt_mem12_ecc_mbit_err" }, - { .int_msk = BIT(21), .msg = "rss_idt_mem13_ecc_mbit_err" }, - { .int_msk = BIT(22), .msg = "rss_idt_mem14_ecc_mbit_err" }, - { .int_msk = BIT(23), .msg = "rss_idt_mem15_ecc_mbit_err" }, - { .int_msk = BIT(24), .msg = "port_vlan_mem_ecc_mbit_err" }, - { .int_msk = BIT(25), .msg = "mcast_linear_table_mem_ecc_mbit_err" }, - { .int_msk = BIT(26), .msg = "mcast_result_mem_ecc_mbit_err" }, - { .int_msk = BIT(27), - .msg = "flow_director_ad_mem0_ecc_mbit_err" }, - { .int_msk = BIT(28), - .msg = "flow_director_ad_mem1_ecc_mbit_err" }, - { .int_msk = BIT(29), - .msg = "rx_vlan_tag_memory_ecc_mbit_err" }, - { .int_msk = BIT(30), - .msg = "Tx_UP_mapping_config_mem_ecc_mbit_err" }, + { .int_msk = BIT(0), .msg = "vf_vlan_ad_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(1), .msg = "umv_mcast_group_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(2), .msg = "umv_key_mem0_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(3), .msg = "umv_key_mem1_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(4), .msg = "umv_key_mem2_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(5), .msg = "umv_key_mem3_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(6), .msg = "umv_ad_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(7), .msg = "rss_tc_mode_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(8), .msg = "rss_idt_mem0_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(9), .msg = "rss_idt_mem1_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(10), .msg = "rss_idt_mem2_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(11), .msg = "rss_idt_mem3_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(12), .msg = "rss_idt_mem4_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(13), .msg = "rss_idt_mem5_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(14), .msg = "rss_idt_mem6_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(15), .msg = "rss_idt_mem7_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(16), .msg = "rss_idt_mem8_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(17), .msg = "rss_idt_mem9_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(18), .msg = "rss_idt_mem10_ecc_m1bit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(19), .msg = "rss_idt_mem11_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(20), .msg = "rss_idt_mem12_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(21), .msg = "rss_idt_mem13_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(22), .msg = "rss_idt_mem14_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(23), .msg = "rss_idt_mem15_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(24), .msg = "port_vlan_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(25), .msg = "mcast_linear_table_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(26), .msg = "mcast_result_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(27), .msg = "flow_director_ad_mem0_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(28), .msg = "flow_director_ad_mem1_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(29), .msg = "rx_vlan_tag_memory_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(30), .msg = "Tx_UP_mapping_config_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, { /* sentinel */ } }; static const struct hclge_hw_error hclge_ppp_pf_abnormal_int[] = { - { .int_msk = BIT(0), .msg = "tx_vlan_tag_err" }, - { .int_msk = BIT(1), .msg = "rss_list_tc_unassigned_queue_err" }, + { .int_msk = BIT(0), .msg = "tx_vlan_tag_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(1), .msg = "rss_list_tc_unassigned_queue_err", + .reset_level = HNAE3_NONE_RESET }, { /* sentinel */ } }; static const struct hclge_hw_error hclge_ppp_mpf_abnormal_int_st3[] = { - { .int_msk = BIT(0), .msg = "hfs_fifo_mem_ecc_mbit_err" }, - { .int_msk = BIT(1), .msg = "rslt_descr_fifo_mem_ecc_mbit_err" }, - { .int_msk = BIT(2), .msg = "tx_vlan_tag_mem_ecc_mbit_err" }, - { .int_msk = BIT(3), .msg = "FD_CN0_memory_ecc_mbit_err" }, - { .int_msk = BIT(4), .msg = "FD_CN1_memory_ecc_mbit_err" }, - { .int_msk = BIT(5), .msg = "GRO_AD_memory_ecc_mbit_err" }, + { .int_msk = BIT(0), .msg = "hfs_fifo_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(1), .msg = "rslt_descr_fifo_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(2), .msg = "tx_vlan_tag_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(3), .msg = "FD_CN0_memory_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(4), .msg = "FD_CN1_memory_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(5), .msg = "GRO_AD_memory_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, { /* sentinel */ } }; static const struct hclge_hw_error hclge_tm_sch_rint[] = { - { .int_msk = BIT(1), .msg = "tm_sch_ecc_mbit_err" }, - { .int_msk = BIT(2), .msg = "tm_sch_port_shap_sub_fifo_wr_err" }, - { .int_msk = BIT(3), .msg = "tm_sch_port_shap_sub_fifo_rd_err" }, - { .int_msk = BIT(4), .msg = "tm_sch_pg_pshap_sub_fifo_wr_err" }, - { .int_msk = BIT(5), .msg = "tm_sch_pg_pshap_sub_fifo_rd_err" }, - { .int_msk = BIT(6), .msg = "tm_sch_pg_cshap_sub_fifo_wr_err" }, - { .int_msk = BIT(7), .msg = "tm_sch_pg_cshap_sub_fifo_rd_err" }, - { .int_msk = BIT(8), .msg = "tm_sch_pri_pshap_sub_fifo_wr_err" }, - { .int_msk = BIT(9), .msg = "tm_sch_pri_pshap_sub_fifo_rd_err" }, - { .int_msk = BIT(10), .msg = "tm_sch_pri_cshap_sub_fifo_wr_err" }, - { .int_msk = BIT(11), .msg = "tm_sch_pri_cshap_sub_fifo_rd_err" }, - { .int_msk = BIT(12), - .msg = "tm_sch_port_shap_offset_fifo_wr_err" }, - { .int_msk = BIT(13), - .msg = "tm_sch_port_shap_offset_fifo_rd_err" }, - { .int_msk = BIT(14), - .msg = "tm_sch_pg_pshap_offset_fifo_wr_err" }, - { .int_msk = BIT(15), - .msg = "tm_sch_pg_pshap_offset_fifo_rd_err" }, - { .int_msk = BIT(16), - .msg = "tm_sch_pg_cshap_offset_fifo_wr_err" }, - { .int_msk = BIT(17), - .msg = "tm_sch_pg_cshap_offset_fifo_rd_err" }, - { .int_msk = BIT(18), - .msg = "tm_sch_pri_pshap_offset_fifo_wr_err" }, - { .int_msk = BIT(19), - .msg = "tm_sch_pri_pshap_offset_fifo_rd_err" }, - { .int_msk = BIT(20), - .msg = "tm_sch_pri_cshap_offset_fifo_wr_err" }, - { .int_msk = BIT(21), - .msg = "tm_sch_pri_cshap_offset_fifo_rd_err" }, - { .int_msk = BIT(22), .msg = "tm_sch_rq_fifo_wr_err" }, - { .int_msk = BIT(23), .msg = "tm_sch_rq_fifo_rd_err" }, - { .int_msk = BIT(24), .msg = "tm_sch_nq_fifo_wr_err" }, - { .int_msk = BIT(25), .msg = "tm_sch_nq_fifo_rd_err" }, - { .int_msk = BIT(26), .msg = "tm_sch_roce_up_fifo_wr_err" }, - { .int_msk = BIT(27), .msg = "tm_sch_roce_up_fifo_rd_err" }, - { .int_msk = BIT(28), .msg = "tm_sch_rcb_byte_fifo_wr_err" }, - { .int_msk = BIT(29), .msg = "tm_sch_rcb_byte_fifo_rd_err" }, - { .int_msk = BIT(30), .msg = "tm_sch_ssu_byte_fifo_wr_err" }, - { .int_msk = BIT(31), .msg = "tm_sch_ssu_byte_fifo_rd_err" }, + { .int_msk = BIT(1), .msg = "tm_sch_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(2), .msg = "tm_sch_port_shap_sub_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(3), .msg = "tm_sch_port_shap_sub_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(4), .msg = "tm_sch_pg_pshap_sub_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(5), .msg = "tm_sch_pg_pshap_sub_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(6), .msg = "tm_sch_pg_cshap_sub_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(7), .msg = "tm_sch_pg_cshap_sub_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(8), .msg = "tm_sch_pri_pshap_sub_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(9), .msg = "tm_sch_pri_pshap_sub_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(10), .msg = "tm_sch_pri_cshap_sub_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(11), .msg = "tm_sch_pri_cshap_sub_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(12), .msg = "tm_sch_port_shap_offset_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(13), .msg = "tm_sch_port_shap_offset_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(14), .msg = "tm_sch_pg_pshap_offset_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(15), .msg = "tm_sch_pg_pshap_offset_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(16), .msg = "tm_sch_pg_cshap_offset_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(17), .msg = "tm_sch_pg_cshap_offset_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(18), .msg = "tm_sch_pri_pshap_offset_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(19), .msg = "tm_sch_pri_pshap_offset_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(20), .msg = "tm_sch_pri_cshap_offset_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(21), .msg = "tm_sch_pri_cshap_offset_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(22), .msg = "tm_sch_rq_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(23), .msg = "tm_sch_rq_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(24), .msg = "tm_sch_nq_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(25), .msg = "tm_sch_nq_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(26), .msg = "tm_sch_roce_up_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(27), .msg = "tm_sch_roce_up_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(28), .msg = "tm_sch_rcb_byte_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(29), .msg = "tm_sch_rcb_byte_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(30), .msg = "tm_sch_ssu_byte_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(31), .msg = "tm_sch_ssu_byte_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, { /* sentinel */ } }; static const struct hclge_hw_error hclge_qcn_fifo_rint[] = { - { .int_msk = BIT(0), .msg = "qcn_shap_gp0_sch_fifo_rd_err" }, - { .int_msk = BIT(1), .msg = "qcn_shap_gp0_sch_fifo_wr_err" }, - { .int_msk = BIT(2), .msg = "qcn_shap_gp1_sch_fifo_rd_err" }, - { .int_msk = BIT(3), .msg = "qcn_shap_gp1_sch_fifo_wr_err" }, - { .int_msk = BIT(4), .msg = "qcn_shap_gp2_sch_fifo_rd_err" }, - { .int_msk = BIT(5), .msg = "qcn_shap_gp2_sch_fifo_wr_err" }, - { .int_msk = BIT(6), .msg = "qcn_shap_gp3_sch_fifo_rd_err" }, - { .int_msk = BIT(7), .msg = "qcn_shap_gp3_sch_fifo_wr_err" }, - { .int_msk = BIT(8), .msg = "qcn_shap_gp0_offset_fifo_rd_err" }, - { .int_msk = BIT(9), .msg = "qcn_shap_gp0_offset_fifo_wr_err" }, - { .int_msk = BIT(10), .msg = "qcn_shap_gp1_offset_fifo_rd_err" }, - { .int_msk = BIT(11), .msg = "qcn_shap_gp1_offset_fifo_wr_err" }, - { .int_msk = BIT(12), .msg = "qcn_shap_gp2_offset_fifo_rd_err" }, - { .int_msk = BIT(13), .msg = "qcn_shap_gp2_offset_fifo_wr_err" }, - { .int_msk = BIT(14), .msg = "qcn_shap_gp3_offset_fifo_rd_err" }, - { .int_msk = BIT(15), .msg = "qcn_shap_gp3_offset_fifo_wr_err" }, - { .int_msk = BIT(16), .msg = "qcn_byte_info_fifo_rd_err" }, - { .int_msk = BIT(17), .msg = "qcn_byte_info_fifo_wr_err" }, + { .int_msk = BIT(0), .msg = "qcn_shap_gp0_sch_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(1), .msg = "qcn_shap_gp0_sch_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(2), .msg = "qcn_shap_gp1_sch_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(3), .msg = "qcn_shap_gp1_sch_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(4), .msg = "qcn_shap_gp2_sch_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(5), .msg = "qcn_shap_gp2_sch_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(6), .msg = "qcn_shap_gp3_sch_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(7), .msg = "qcn_shap_gp3_sch_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(8), .msg = "qcn_shap_gp0_offset_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(9), .msg = "qcn_shap_gp0_offset_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(10), .msg = "qcn_shap_gp1_offset_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(11), .msg = "qcn_shap_gp1_offset_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(12), .msg = "qcn_shap_gp2_offset_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(13), .msg = "qcn_shap_gp2_offset_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(14), .msg = "qcn_shap_gp3_offset_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(15), .msg = "qcn_shap_gp3_offset_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(16), .msg = "qcn_byte_info_fifo_rd_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(17), .msg = "qcn_byte_info_fifo_wr_err", + .reset_level = HNAE3_GLOBAL_RESET }, { /* sentinel */ } }; static const struct hclge_hw_error hclge_qcn_ecc_rint[] = { - { .int_msk = BIT(1), .msg = "qcn_byte_mem_ecc_mbit_err" }, - { .int_msk = BIT(3), .msg = "qcn_time_mem_ecc_mbit_err" }, - { .int_msk = BIT(5), .msg = "qcn_fb_mem_ecc_mbit_err" }, - { .int_msk = BIT(7), .msg = "qcn_link_mem_ecc_mbit_err" }, - { .int_msk = BIT(9), .msg = "qcn_rate_mem_ecc_mbit_err" }, - { .int_msk = BIT(11), .msg = "qcn_tmplt_mem_ecc_mbit_err" }, - { .int_msk = BIT(13), .msg = "qcn_shap_cfg_mem_ecc_mbit_err" }, - { .int_msk = BIT(15), .msg = "qcn_gp0_barrel_mem_ecc_mbit_err" }, - { .int_msk = BIT(17), .msg = "qcn_gp1_barrel_mem_ecc_mbit_err" }, - { .int_msk = BIT(19), .msg = "qcn_gp2_barrel_mem_ecc_mbit_err" }, - { .int_msk = BIT(21), .msg = "qcn_gp3_barral_mem_ecc_mbit_err" }, + { .int_msk = BIT(1), .msg = "qcn_byte_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(3), .msg = "qcn_time_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(5), .msg = "qcn_fb_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(7), .msg = "qcn_link_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(9), .msg = "qcn_rate_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(11), .msg = "qcn_tmplt_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(13), .msg = "qcn_shap_cfg_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(15), .msg = "qcn_gp0_barrel_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(17), .msg = "qcn_gp1_barrel_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(19), .msg = "qcn_gp2_barrel_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(21), .msg = "qcn_gp3_barral_mem_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, { /* sentinel */ } }; static const struct hclge_hw_error hclge_mac_afifo_tnl_int[] = { - { .int_msk = BIT(0), .msg = "egu_cge_afifo_ecc_1bit_err" }, - { .int_msk = BIT(1), .msg = "egu_cge_afifo_ecc_mbit_err" }, - { .int_msk = BIT(2), .msg = "egu_lge_afifo_ecc_1bit_err" }, - { .int_msk = BIT(3), .msg = "egu_lge_afifo_ecc_mbit_err" }, - { .int_msk = BIT(4), .msg = "cge_igu_afifo_ecc_1bit_err" }, - { .int_msk = BIT(5), .msg = "cge_igu_afifo_ecc_mbit_err" }, - { .int_msk = BIT(6), .msg = "lge_igu_afifo_ecc_1bit_err" }, - { .int_msk = BIT(7), .msg = "lge_igu_afifo_ecc_mbit_err" }, - { .int_msk = BIT(8), .msg = "cge_igu_afifo_overflow_err" }, - { .int_msk = BIT(9), .msg = "lge_igu_afifo_overflow_err" }, - { .int_msk = BIT(10), .msg = "egu_cge_afifo_underrun_err" }, - { .int_msk = BIT(11), .msg = "egu_lge_afifo_underrun_err" }, - { .int_msk = BIT(12), .msg = "egu_ge_afifo_underrun_err" }, - { .int_msk = BIT(13), .msg = "ge_igu_afifo_overflow_err" }, + { .int_msk = BIT(0), .msg = "egu_cge_afifo_ecc_1bit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(1), .msg = "egu_cge_afifo_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(2), .msg = "egu_lge_afifo_ecc_1bit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(3), .msg = "egu_lge_afifo_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(4), .msg = "cge_igu_afifo_ecc_1bit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(5), .msg = "cge_igu_afifo_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(6), .msg = "lge_igu_afifo_ecc_1bit_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(7), .msg = "lge_igu_afifo_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(8), .msg = "cge_igu_afifo_overflow_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(9), .msg = "lge_igu_afifo_overflow_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(10), .msg = "egu_cge_afifo_underrun_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(11), .msg = "egu_lge_afifo_underrun_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(12), .msg = "egu_ge_afifo_underrun_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(13), .msg = "ge_igu_afifo_overflow_err", + .reset_level = HNAE3_GLOBAL_RESET }, { /* sentinel */ } }; static const struct hclge_hw_error hclge_ppu_mpf_abnormal_int_st2[] = { - { .int_msk = BIT(13), .msg = "rpu_rx_pkt_bit32_ecc_mbit_err" }, - { .int_msk = BIT(14), .msg = "rpu_rx_pkt_bit33_ecc_mbit_err" }, - { .int_msk = BIT(15), .msg = "rpu_rx_pkt_bit34_ecc_mbit_err" }, - { .int_msk = BIT(16), .msg = "rpu_rx_pkt_bit35_ecc_mbit_err" }, - { .int_msk = BIT(17), .msg = "rcb_tx_ring_ecc_mbit_err" }, - { .int_msk = BIT(18), .msg = "rcb_rx_ring_ecc_mbit_err" }, - { .int_msk = BIT(19), .msg = "rcb_tx_fbd_ecc_mbit_err" }, - { .int_msk = BIT(20), .msg = "rcb_rx_ebd_ecc_mbit_err" }, - { .int_msk = BIT(21), .msg = "rcb_tso_info_ecc_mbit_err" }, - { .int_msk = BIT(22), .msg = "rcb_tx_int_info_ecc_mbit_err" }, - { .int_msk = BIT(23), .msg = "rcb_rx_int_info_ecc_mbit_err" }, - { .int_msk = BIT(24), .msg = "tpu_tx_pkt_0_ecc_mbit_err" }, - { .int_msk = BIT(25), .msg = "tpu_tx_pkt_1_ecc_mbit_err" }, - { .int_msk = BIT(26), .msg = "rd_bus_err" }, - { .int_msk = BIT(27), .msg = "wr_bus_err" }, - { .int_msk = BIT(28), .msg = "reg_search_miss" }, - { .int_msk = BIT(29), .msg = "rx_q_search_miss" }, - { .int_msk = BIT(30), .msg = "ooo_ecc_err_detect" }, - { .int_msk = BIT(31), .msg = "ooo_ecc_err_multpl" }, + { .int_msk = BIT(13), .msg = "rpu_rx_pkt_bit32_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(14), .msg = "rpu_rx_pkt_bit33_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(15), .msg = "rpu_rx_pkt_bit34_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(16), .msg = "rpu_rx_pkt_bit35_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(17), .msg = "rcb_tx_ring_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(18), .msg = "rcb_rx_ring_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(19), .msg = "rcb_tx_fbd_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(20), .msg = "rcb_rx_ebd_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(21), .msg = "rcb_tso_info_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(22), .msg = "rcb_tx_int_info_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(23), .msg = "rcb_rx_int_info_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(24), .msg = "tpu_tx_pkt_0_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(25), .msg = "tpu_tx_pkt_1_ecc_mbit_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(26), .msg = "rd_bus_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(27), .msg = "wr_bus_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(28), .msg = "reg_search_miss", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(29), .msg = "rx_q_search_miss", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(30), .msg = "ooo_ecc_err_detect", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(31), .msg = "ooo_ecc_err_multpl", + .reset_level = HNAE3_GLOBAL_RESET }, { /* sentinel */ } }; static const struct hclge_hw_error hclge_ppu_mpf_abnormal_int_st3[] = { - { .int_msk = BIT(4), .msg = "gro_bd_ecc_mbit_err" }, - { .int_msk = BIT(5), .msg = "gro_context_ecc_mbit_err" }, - { .int_msk = BIT(6), .msg = "rx_stash_cfg_ecc_mbit_err" }, - { .int_msk = BIT(7), .msg = "axi_rd_fbd_ecc_mbit_err" }, + { .int_msk = BIT(4), .msg = "gro_bd_ecc_mbit_err", + .reset_level = HNAE3_CORE_RESET }, + { .int_msk = BIT(5), .msg = "gro_context_ecc_mbit_err", + .reset_level = HNAE3_CORE_RESET }, + { .int_msk = BIT(6), .msg = "rx_stash_cfg_ecc_mbit_err", + .reset_level = HNAE3_CORE_RESET }, + { .int_msk = BIT(7), .msg = "axi_rd_fbd_ecc_mbit_err", + .reset_level = HNAE3_CORE_RESET }, { /* sentinel */ } }; static const struct hclge_hw_error hclge_ppu_pf_abnormal_int[] = { - { .int_msk = BIT(0), .msg = "over_8bd_no_fe" }, - { .int_msk = BIT(1), .msg = "tso_mss_cmp_min_err" }, - { .int_msk = BIT(2), .msg = "tso_mss_cmp_max_err" }, - { .int_msk = BIT(3), .msg = "tx_rd_fbd_poison" }, - { .int_msk = BIT(4), .msg = "rx_rd_ebd_poison" }, - { .int_msk = BIT(5), .msg = "buf_wait_timeout" }, + { .int_msk = BIT(0), .msg = "over_8bd_no_fe", + .reset_level = HNAE3_FUNC_RESET }, + { .int_msk = BIT(1), .msg = "tso_mss_cmp_min_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(2), .msg = "tso_mss_cmp_max_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(3), .msg = "tx_rd_fbd_poison", + .reset_level = HNAE3_FUNC_RESET }, + { .int_msk = BIT(4), .msg = "rx_rd_ebd_poison", + .reset_level = HNAE3_FUNC_RESET }, + { .int_msk = BIT(5), .msg = "buf_wait_timeout", + .reset_level = HNAE3_NONE_RESET }, { /* sentinel */ } }; static const struct hclge_hw_error hclge_ssu_com_err_int[] = { - { .int_msk = BIT(0), .msg = "buf_sum_err" }, - { .int_msk = BIT(1), .msg = "ppp_mb_num_err" }, - { .int_msk = BIT(2), .msg = "ppp_mbid_err" }, - { .int_msk = BIT(3), .msg = "ppp_rlt_mac_err" }, - { .int_msk = BIT(4), .msg = "ppp_rlt_host_err" }, - { .int_msk = BIT(5), .msg = "cks_edit_position_err" }, - { .int_msk = BIT(6), .msg = "cks_edit_condition_err" }, - { .int_msk = BIT(7), .msg = "vlan_edit_condition_err" }, - { .int_msk = BIT(8), .msg = "vlan_num_ot_err" }, - { .int_msk = BIT(9), .msg = "vlan_num_in_err" }, + { .int_msk = BIT(0), .msg = "buf_sum_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(1), .msg = "ppp_mb_num_err", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(2), .msg = "ppp_mbid_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(3), .msg = "ppp_rlt_mac_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(4), .msg = "ppp_rlt_host_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(5), .msg = "cks_edit_position_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(6), .msg = "cks_edit_condition_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(7), .msg = "vlan_edit_condition_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(8), .msg = "vlan_num_ot_err", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(9), .msg = "vlan_num_in_err", + .reset_level = HNAE3_GLOBAL_RESET }, { /* sentinel */ } }; #define HCLGE_SSU_MEM_ECC_ERR(x) \ - { .int_msk = BIT(x), .msg = "ssu_mem" #x "_ecc_mbit_err" } + { .int_msk = BIT(x), .msg = "ssu_mem" #x "_ecc_mbit_err", \ + .reset_level = HNAE3_GLOBAL_RESET } static const struct hclge_hw_error hclge_ssu_mem_ecc_err_int[] = { HCLGE_SSU_MEM_ECC_ERR(0), @@ -323,62 +504,106 @@ static const struct hclge_hw_error hclge_ssu_mem_ecc_err_int[] = { }; static const struct hclge_hw_error hclge_ssu_port_based_err_int[] = { - { .int_msk = BIT(0), .msg = "roc_pkt_without_key_port" }, - { .int_msk = BIT(1), .msg = "tpu_pkt_without_key_port" }, - { .int_msk = BIT(2), .msg = "igu_pkt_without_key_port" }, - { .int_msk = BIT(3), .msg = "roc_eof_mis_match_port" }, - { .int_msk = BIT(4), .msg = "tpu_eof_mis_match_port" }, - { .int_msk = BIT(5), .msg = "igu_eof_mis_match_port" }, - { .int_msk = BIT(6), .msg = "roc_sof_mis_match_port" }, - { .int_msk = BIT(7), .msg = "tpu_sof_mis_match_port" }, - { .int_msk = BIT(8), .msg = "igu_sof_mis_match_port" }, - { .int_msk = BIT(11), .msg = "ets_rd_int_rx_port" }, - { .int_msk = BIT(12), .msg = "ets_wr_int_rx_port" }, - { .int_msk = BIT(13), .msg = "ets_rd_int_tx_port" }, - { .int_msk = BIT(14), .msg = "ets_wr_int_tx_port" }, + { .int_msk = BIT(0), .msg = "roc_pkt_without_key_port", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(1), .msg = "tpu_pkt_without_key_port", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(2), .msg = "igu_pkt_without_key_port", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(3), .msg = "roc_eof_mis_match_port", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(4), .msg = "tpu_eof_mis_match_port", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(5), .msg = "igu_eof_mis_match_port", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(6), .msg = "roc_sof_mis_match_port", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(7), .msg = "tpu_sof_mis_match_port", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(8), .msg = "igu_sof_mis_match_port", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(11), .msg = "ets_rd_int_rx_port", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(12), .msg = "ets_wr_int_rx_port", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(13), .msg = "ets_rd_int_tx_port", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(14), .msg = "ets_wr_int_tx_port", + .reset_level = HNAE3_GLOBAL_RESET }, { /* sentinel */ } }; static const struct hclge_hw_error hclge_ssu_fifo_overflow_int[] = { - { .int_msk = BIT(0), .msg = "ig_mac_inf_int" }, - { .int_msk = BIT(1), .msg = "ig_host_inf_int" }, - { .int_msk = BIT(2), .msg = "ig_roc_buf_int" }, - { .int_msk = BIT(3), .msg = "ig_host_data_fifo_int" }, - { .int_msk = BIT(4), .msg = "ig_host_key_fifo_int" }, - { .int_msk = BIT(5), .msg = "tx_qcn_fifo_int" }, - { .int_msk = BIT(6), .msg = "rx_qcn_fifo_int" }, - { .int_msk = BIT(7), .msg = "tx_pf_rd_fifo_int" }, - { .int_msk = BIT(8), .msg = "rx_pf_rd_fifo_int" }, - { .int_msk = BIT(9), .msg = "qm_eof_fifo_int" }, - { .int_msk = BIT(10), .msg = "mb_rlt_fifo_int" }, - { .int_msk = BIT(11), .msg = "dup_uncopy_fifo_int" }, - { .int_msk = BIT(12), .msg = "dup_cnt_rd_fifo_int" }, - { .int_msk = BIT(13), .msg = "dup_cnt_drop_fifo_int" }, - { .int_msk = BIT(14), .msg = "dup_cnt_wrb_fifo_int" }, - { .int_msk = BIT(15), .msg = "host_cmd_fifo_int" }, - { .int_msk = BIT(16), .msg = "mac_cmd_fifo_int" }, - { .int_msk = BIT(17), .msg = "host_cmd_bitmap_empty_int" }, - { .int_msk = BIT(18), .msg = "mac_cmd_bitmap_empty_int" }, - { .int_msk = BIT(19), .msg = "dup_bitmap_empty_int" }, - { .int_msk = BIT(20), .msg = "out_queue_bitmap_empty_int" }, - { .int_msk = BIT(21), .msg = "bank2_bitmap_empty_int" }, - { .int_msk = BIT(22), .msg = "bank1_bitmap_empty_int" }, - { .int_msk = BIT(23), .msg = "bank0_bitmap_empty_int" }, + { .int_msk = BIT(0), .msg = "ig_mac_inf_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(1), .msg = "ig_host_inf_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(2), .msg = "ig_roc_buf_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(3), .msg = "ig_host_data_fifo_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(4), .msg = "ig_host_key_fifo_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(5), .msg = "tx_qcn_fifo_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(6), .msg = "rx_qcn_fifo_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(7), .msg = "tx_pf_rd_fifo_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(8), .msg = "rx_pf_rd_fifo_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(9), .msg = "qm_eof_fifo_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(10), .msg = "mb_rlt_fifo_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(11), .msg = "dup_uncopy_fifo_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(12), .msg = "dup_cnt_rd_fifo_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(13), .msg = "dup_cnt_drop_fifo_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(14), .msg = "dup_cnt_wrb_fifo_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(15), .msg = "host_cmd_fifo_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(16), .msg = "mac_cmd_fifo_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(17), .msg = "host_cmd_bitmap_empty_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(18), .msg = "mac_cmd_bitmap_empty_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(19), .msg = "dup_bitmap_empty_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(20), .msg = "out_queue_bitmap_empty_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(21), .msg = "bank2_bitmap_empty_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(22), .msg = "bank1_bitmap_empty_int", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(23), .msg = "bank0_bitmap_empty_int", + .reset_level = HNAE3_GLOBAL_RESET }, { /* sentinel */ } }; static const struct hclge_hw_error hclge_ssu_ets_tcg_int[] = { - { .int_msk = BIT(0), .msg = "ets_rd_int_rx_tcg" }, - { .int_msk = BIT(1), .msg = "ets_wr_int_rx_tcg" }, - { .int_msk = BIT(2), .msg = "ets_rd_int_tx_tcg" }, - { .int_msk = BIT(3), .msg = "ets_wr_int_tx_tcg" }, + { .int_msk = BIT(0), .msg = "ets_rd_int_rx_tcg", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(1), .msg = "ets_wr_int_rx_tcg", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(2), .msg = "ets_rd_int_tx_tcg", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(3), .msg = "ets_wr_int_tx_tcg", + .reset_level = HNAE3_GLOBAL_RESET }, { /* sentinel */ } }; static const struct hclge_hw_error hclge_ssu_port_based_pf_int[] = { - { .int_msk = BIT(0), .msg = "roc_pkt_without_key_port" }, - { .int_msk = BIT(9), .msg = "low_water_line_err_port" }, - { .int_msk = BIT(10), .msg = "hi_water_line_err_port" }, + { .int_msk = BIT(0), .msg = "roc_pkt_without_key_port", + .reset_level = HNAE3_GLOBAL_RESET }, + { .int_msk = BIT(9), .msg = "low_water_line_err_port", + .reset_level = HNAE3_NONE_RESET }, + { .int_msk = BIT(10), .msg = "hi_water_line_err_port", + .reset_level = HNAE3_GLOBAL_RESET }, { /* sentinel */ } }; @@ -406,16 +631,29 @@ static const struct hclge_hw_error hclge_rocee_qmm_ovf_err_int[] = { { /* sentinel */ } }; -static void hclge_log_error(struct device *dev, char *reg, - const struct hclge_hw_error *err, - u32 err_sts) +static enum hnae3_reset_type hclge_log_error(struct device *dev, char *reg, + const struct hclge_hw_error *err, + u32 err_sts) { + enum hnae3_reset_type reset_level = HNAE3_FUNC_RESET; + bool need_reset = false; + while (err->msg) { - if (err->int_msk & err_sts) + if (err->int_msk & err_sts) { dev_warn(dev, "%s %s found [error status=0x%x]\n", reg, err->msg, err_sts); + if (err->reset_level != HNAE3_NONE_RESET && + err->reset_level >= reset_level) { + reset_level = err->reset_level; + need_reset = true; + } + } err++; } + if (need_reset) + return reset_level; + else + return HNAE3_NONE_RESET; } /* hclge_cmd_query_error: read the error information @@ -454,6 +692,16 @@ static int hclge_cmd_query_error(struct hclge_dev *hdev, return ret; } +static int hclge_clear_mac_tnl_int(struct hclge_dev *hdev) +{ + struct hclge_desc desc; + + hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_CLEAR_MAC_TNL_INT, false); + desc.data[0] = cpu_to_le32(HCLGE_MAC_TNL_INT_CLR); + + return hclge_cmd_send(&hdev->hw, &desc, 1); +} + static int hclge_config_common_hw_err_int(struct hclge_dev *hdev, bool en) { struct device *dev = &hdev->pdev->dev; @@ -673,6 +921,21 @@ static int hclge_config_mac_err_int(struct hclge_dev *hdev, bool en) return ret; } +int hclge_config_mac_tnl_int(struct hclge_dev *hdev, bool en) +{ + struct hclge_desc desc; + + hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_MAC_TNL_INT_EN, false); + if (en) + desc.data[0] = cpu_to_le32(HCLGE_MAC_TNL_INT_EN); + else + desc.data[0] = 0; + + desc.data[1] = cpu_to_le32(HCLGE_MAC_TNL_INT_EN_MASK); + + return hclge_cmd_send(&hdev->hw, &desc, 1); +} + static int hclge_config_ppu_error_interrupts(struct hclge_dev *hdev, u32 cmd, bool en) { @@ -826,6 +1089,7 @@ static int hclge_handle_mpf_ras_error(struct hclge_dev *hdev, int num) { struct hnae3_ae_dev *ae_dev = hdev->ae_dev; + enum hnae3_reset_type reset_level; struct device *dev = &hdev->pdev->dev; __le32 *desc_data; u32 status; @@ -845,78 +1109,94 @@ static int hclge_handle_mpf_ras_error(struct hclge_dev *hdev, /* log HNS common errors */ status = le32_to_cpu(desc[0].data[0]); if (status) { - hclge_log_error(dev, "IMP_TCM_ECC_INT_STS", - &hclge_imp_tcm_ecc_int[0], status); - HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_GLOBAL_RESET); + reset_level = hclge_log_error(dev, "IMP_TCM_ECC_INT_STS", + &hclge_imp_tcm_ecc_int[0], + status); + HCLGE_SET_DEFAULT_RESET_REQUEST(reset_level); } status = le32_to_cpu(desc[0].data[1]); if (status) { - hclge_log_error(dev, "CMDQ_MEM_ECC_INT_STS", - &hclge_cmdq_nic_mem_ecc_int[0], status); - HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_GLOBAL_RESET); + reset_level = hclge_log_error(dev, "CMDQ_MEM_ECC_INT_STS", + &hclge_cmdq_nic_mem_ecc_int[0], + status); + HCLGE_SET_DEFAULT_RESET_REQUEST(reset_level); } if ((le32_to_cpu(desc[0].data[2])) & BIT(0)) { dev_warn(dev, "imp_rd_data_poison_err found\n"); - HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_GLOBAL_RESET); + HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_NONE_RESET); } status = le32_to_cpu(desc[0].data[3]); if (status) { - hclge_log_error(dev, "TQP_INT_ECC_INT_STS", - &hclge_tqp_int_ecc_int[0], status); - HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_CORE_RESET); + reset_level = hclge_log_error(dev, "TQP_INT_ECC_INT_STS", + &hclge_tqp_int_ecc_int[0], + status); + HCLGE_SET_DEFAULT_RESET_REQUEST(reset_level); } status = le32_to_cpu(desc[0].data[4]); if (status) { - hclge_log_error(dev, "MSIX_ECC_INT_STS", - &hclge_msix_sram_ecc_int[0], status); - HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_CORE_RESET); + reset_level = hclge_log_error(dev, "MSIX_ECC_INT_STS", + &hclge_msix_sram_ecc_int[0], + status); + HCLGE_SET_DEFAULT_RESET_REQUEST(reset_level); } /* log SSU(Storage Switch Unit) errors */ desc_data = (__le32 *)&desc[2]; status = le32_to_cpu(*(desc_data + 2)); if (status) { - hclge_log_error(dev, "SSU_ECC_MULTI_BIT_INT_0", - &hclge_ssu_mem_ecc_err_int[0], status); - HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_CORE_RESET); + reset_level = hclge_log_error(dev, "SSU_ECC_MULTI_BIT_INT_0", + &hclge_ssu_mem_ecc_err_int[0], + status); + HCLGE_SET_DEFAULT_RESET_REQUEST(reset_level); } status = le32_to_cpu(*(desc_data + 3)) & BIT(0); if (status) { dev_warn(dev, "SSU_ECC_MULTI_BIT_INT_1 ssu_mem32_ecc_mbit_err found [error status=0x%x]\n", status); - HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_CORE_RESET); + HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_GLOBAL_RESET); } status = le32_to_cpu(*(desc_data + 4)) & HCLGE_SSU_COMMON_ERR_INT_MASK; if (status) { - hclge_log_error(dev, "SSU_COMMON_ERR_INT", - &hclge_ssu_com_err_int[0], status); - HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_GLOBAL_RESET); + reset_level = hclge_log_error(dev, "SSU_COMMON_ERR_INT", + &hclge_ssu_com_err_int[0], + status); + HCLGE_SET_DEFAULT_RESET_REQUEST(reset_level); } /* log IGU(Ingress Unit) errors */ desc_data = (__le32 *)&desc[3]; status = le32_to_cpu(*desc_data) & HCLGE_IGU_INT_MASK; - if (status) - hclge_log_error(dev, "IGU_INT_STS", - &hclge_igu_int[0], status); + if (status) { + reset_level = hclge_log_error(dev, "IGU_INT_STS", + &hclge_igu_int[0], status); + HCLGE_SET_DEFAULT_RESET_REQUEST(reset_level); + } /* log PPP(Programmable Packet Process) errors */ desc_data = (__le32 *)&desc[4]; status = le32_to_cpu(*(desc_data + 1)); - if (status) - hclge_log_error(dev, "PPP_MPF_ABNORMAL_INT_ST1", - &hclge_ppp_mpf_abnormal_int_st1[0], status); + if (status) { + reset_level = + hclge_log_error(dev, "PPP_MPF_ABNORMAL_INT_ST1", + &hclge_ppp_mpf_abnormal_int_st1[0], + status); + HCLGE_SET_DEFAULT_RESET_REQUEST(reset_level); + } status = le32_to_cpu(*(desc_data + 3)) & HCLGE_PPP_MPF_INT_ST3_MASK; - if (status) - hclge_log_error(dev, "PPP_MPF_ABNORMAL_INT_ST3", - &hclge_ppp_mpf_abnormal_int_st3[0], status); + if (status) { + reset_level = + hclge_log_error(dev, "PPP_MPF_ABNORMAL_INT_ST3", + &hclge_ppp_mpf_abnormal_int_st3[0], + status); + HCLGE_SET_DEFAULT_RESET_REQUEST(reset_level); + } /* log PPU(RCB) errors */ desc_data = (__le32 *)&desc[5]; @@ -924,55 +1204,60 @@ static int hclge_handle_mpf_ras_error(struct hclge_dev *hdev, if (status) { dev_warn(dev, "PPU_MPF_ABNORMAL_INT_ST1 %s found\n", "rpu_rx_pkt_ecc_mbit_err"); - HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_CORE_RESET); + HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_GLOBAL_RESET); } status = le32_to_cpu(*(desc_data + 2)); if (status) { - hclge_log_error(dev, "PPU_MPF_ABNORMAL_INT_ST2", - &hclge_ppu_mpf_abnormal_int_st2[0], status); - HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_CORE_RESET); + reset_level = + hclge_log_error(dev, "PPU_MPF_ABNORMAL_INT_ST2", + &hclge_ppu_mpf_abnormal_int_st2[0], + status); + HCLGE_SET_DEFAULT_RESET_REQUEST(reset_level); } status = le32_to_cpu(*(desc_data + 3)) & HCLGE_PPU_MPF_INT_ST3_MASK; if (status) { - hclge_log_error(dev, "PPU_MPF_ABNORMAL_INT_ST3", - &hclge_ppu_mpf_abnormal_int_st3[0], status); - HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_CORE_RESET); + reset_level = + hclge_log_error(dev, "PPU_MPF_ABNORMAL_INT_ST3", + &hclge_ppu_mpf_abnormal_int_st3[0], + status); + HCLGE_SET_DEFAULT_RESET_REQUEST(reset_level); } /* log TM(Traffic Manager) errors */ desc_data = (__le32 *)&desc[6]; status = le32_to_cpu(*desc_data); if (status) { - hclge_log_error(dev, "TM_SCH_RINT", - &hclge_tm_sch_rint[0], status); - HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_CORE_RESET); + reset_level = hclge_log_error(dev, "TM_SCH_RINT", + &hclge_tm_sch_rint[0], status); + HCLGE_SET_DEFAULT_RESET_REQUEST(reset_level); } /* log QCN(Quantized Congestion Control) errors */ desc_data = (__le32 *)&desc[7]; status = le32_to_cpu(*desc_data) & HCLGE_QCN_FIFO_INT_MASK; if (status) { - hclge_log_error(dev, "QCN_FIFO_RINT", - &hclge_qcn_fifo_rint[0], status); - HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_CORE_RESET); + reset_level = hclge_log_error(dev, "QCN_FIFO_RINT", + &hclge_qcn_fifo_rint[0], status); + HCLGE_SET_DEFAULT_RESET_REQUEST(reset_level); } status = le32_to_cpu(*(desc_data + 1)) & HCLGE_QCN_ECC_INT_MASK; if (status) { - hclge_log_error(dev, "QCN_ECC_RINT", - &hclge_qcn_ecc_rint[0], status); - HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_CORE_RESET); + reset_level = hclge_log_error(dev, "QCN_ECC_RINT", + &hclge_qcn_ecc_rint[0], + status); + HCLGE_SET_DEFAULT_RESET_REQUEST(reset_level); } /* log NCSI errors */ desc_data = (__le32 *)&desc[9]; status = le32_to_cpu(*desc_data) & HCLGE_NCSI_ECC_INT_MASK; if (status) { - hclge_log_error(dev, "NCSI_ECC_INT_RPT", - &hclge_ncsi_err_int[0], status); - HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_CORE_RESET); + reset_level = hclge_log_error(dev, "NCSI_ECC_INT_RPT", + &hclge_ncsi_err_int[0], status); + HCLGE_SET_DEFAULT_RESET_REQUEST(reset_level); } /* clear all main PF RAS errors */ @@ -1000,6 +1285,7 @@ static int hclge_handle_pf_ras_error(struct hclge_dev *hdev, { struct hnae3_ae_dev *ae_dev = hdev->ae_dev; struct device *dev = &hdev->pdev->dev; + enum hnae3_reset_type reset_level; __le32 *desc_data; u32 status; int ret; @@ -1018,38 +1304,47 @@ static int hclge_handle_pf_ras_error(struct hclge_dev *hdev, /* log SSU(Storage Switch Unit) errors */ status = le32_to_cpu(desc[0].data[0]); if (status) { - hclge_log_error(dev, "SSU_PORT_BASED_ERR_INT", - &hclge_ssu_port_based_err_int[0], status); - HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_GLOBAL_RESET); + reset_level = hclge_log_error(dev, "SSU_PORT_BASED_ERR_INT", + &hclge_ssu_port_based_err_int[0], + status); + HCLGE_SET_DEFAULT_RESET_REQUEST(reset_level); } status = le32_to_cpu(desc[0].data[1]); if (status) { - hclge_log_error(dev, "SSU_FIFO_OVERFLOW_INT", - &hclge_ssu_fifo_overflow_int[0], status); - HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_GLOBAL_RESET); + reset_level = hclge_log_error(dev, "SSU_FIFO_OVERFLOW_INT", + &hclge_ssu_fifo_overflow_int[0], + status); + HCLGE_SET_DEFAULT_RESET_REQUEST(reset_level); } status = le32_to_cpu(desc[0].data[2]); if (status) { - hclge_log_error(dev, "SSU_ETS_TCG_INT", - &hclge_ssu_ets_tcg_int[0], status); - HCLGE_SET_DEFAULT_RESET_REQUEST(HNAE3_GLOBAL_RESET); + reset_level = hclge_log_error(dev, "SSU_ETS_TCG_INT", + &hclge_ssu_ets_tcg_int[0], + status); + HCLGE_SET_DEFAULT_RESET_REQUEST(reset_level); } /* log IGU(Ingress Unit) EGU(Egress Unit) TNL errors */ desc_data = (__le32 *)&desc[1]; status = le32_to_cpu(*desc_data) & HCLGE_IGU_EGU_TNL_INT_MASK; - if (status) - hclge_log_error(dev, "IGU_EGU_TNL_INT_STS", - &hclge_igu_egu_tnl_int[0], status); + if (status) { + reset_level = hclge_log_error(dev, "IGU_EGU_TNL_INT_STS", + &hclge_igu_egu_tnl_int[0], + status); + HCLGE_SET_DEFAULT_RESET_REQUEST(reset_level); + } /* log PPU(RCB) errors */ desc_data = (__le32 *)&desc[3]; status = le32_to_cpu(*desc_data) & HCLGE_PPU_PF_INT_RAS_MASK; - if (status) - hclge_log_error(dev, "PPU_PF_ABNORMAL_INT_ST0", - &hclge_ppu_pf_abnormal_int[0], status); + if (status) { + reset_level = hclge_log_error(dev, "PPU_PF_ABNORMAL_INT_ST0", + &hclge_ppu_pf_abnormal_int[0], + status); + HCLGE_SET_DEFAULT_RESET_REQUEST(reset_level); + } /* clear all PF RAS errors */ hclge_cmd_reuse_desc(&desc[0], false); @@ -1341,16 +1636,15 @@ pci_ers_result_t hclge_handle_hw_ras_error(struct hnae3_ae_dev *ae_dev) int hclge_handle_hw_msix_error(struct hclge_dev *hdev, unsigned long *reset_requests) { + struct hclge_mac_tnl_stats mac_tnl_stats; struct device *dev = &hdev->pdev->dev; u32 mpf_bd_num, pf_bd_num, bd_num; + enum hnae3_reset_type reset_level; struct hclge_desc desc_bd; struct hclge_desc *desc; __le32 *desc_data; - int ret = 0; u32 status; - - /* set default handling */ - set_bit(HNAE3_FUNC_RESET, reset_requests); + int ret; /* query the number of bds for the MSIx int status */ hclge_cmd_setup_basic_desc(&desc_bd, HCLGE_QUERY_MSIX_INT_STS_BD_NUM, @@ -1359,8 +1653,6 @@ int hclge_handle_hw_msix_error(struct hclge_dev *hdev, if (ret) { dev_err(dev, "fail(%d) to query msix int status bd num\n", ret); - /* reset everything for now */ - set_bit(HNAE3_GLOBAL_RESET, reset_requests); return ret; } @@ -1381,8 +1673,6 @@ int hclge_handle_hw_msix_error(struct hclge_dev *hdev, if (ret) { dev_err(dev, "query all mpf msix int cmd failed (%d)\n", ret); - /* reset everything for now */ - set_bit(HNAE3_GLOBAL_RESET, reset_requests); goto msi_error; } @@ -1390,9 +1680,10 @@ int hclge_handle_hw_msix_error(struct hclge_dev *hdev, desc_data = (__le32 *)&desc[1]; status = le32_to_cpu(*desc_data); if (status) { - hclge_log_error(dev, "MAC_AFIFO_TNL_INT_R", - &hclge_mac_afifo_tnl_int[0], status); - set_bit(HNAE3_GLOBAL_RESET, reset_requests); + reset_level = hclge_log_error(dev, "MAC_AFIFO_TNL_INT_R", + &hclge_mac_afifo_tnl_int[0], + status); + set_bit(reset_level, reset_requests); } /* log PPU(RCB) MPF errors */ @@ -1400,9 +1691,11 @@ int hclge_handle_hw_msix_error(struct hclge_dev *hdev, status = le32_to_cpu(*(desc_data + 2)) & HCLGE_PPU_MPF_INT_ST2_MSIX_MASK; if (status) { - hclge_log_error(dev, "PPU_MPF_ABNORMAL_INT_ST2", - &hclge_ppu_mpf_abnormal_int_st2[0], status); - set_bit(HNAE3_CORE_RESET, reset_requests); + reset_level = + hclge_log_error(dev, "PPU_MPF_ABNORMAL_INT_ST2", + &hclge_ppu_mpf_abnormal_int_st2[0], + status); + set_bit(reset_level, reset_requests); } /* clear all main PF MSIx errors */ @@ -1413,8 +1706,6 @@ int hclge_handle_hw_msix_error(struct hclge_dev *hdev, if (ret) { dev_err(dev, "clear all mpf msix int cmd failed (%d)\n", ret); - /* reset everything for now */ - set_bit(HNAE3_GLOBAL_RESET, reset_requests); goto msi_error; } @@ -1428,32 +1719,37 @@ int hclge_handle_hw_msix_error(struct hclge_dev *hdev, if (ret) { dev_err(dev, "query all pf msix int cmd failed (%d)\n", ret); - /* reset everything for now */ - set_bit(HNAE3_GLOBAL_RESET, reset_requests); goto msi_error; } /* log SSU PF errors */ status = le32_to_cpu(desc[0].data[0]) & HCLGE_SSU_PORT_INT_MSIX_MASK; if (status) { - hclge_log_error(dev, "SSU_PORT_BASED_ERR_INT", - &hclge_ssu_port_based_pf_int[0], status); - set_bit(HNAE3_GLOBAL_RESET, reset_requests); + reset_level = hclge_log_error(dev, "SSU_PORT_BASED_ERR_INT", + &hclge_ssu_port_based_pf_int[0], + status); + set_bit(reset_level, reset_requests); } /* read and log PPP PF errors */ desc_data = (__le32 *)&desc[2]; status = le32_to_cpu(*desc_data); - if (status) - hclge_log_error(dev, "PPP_PF_ABNORMAL_INT_ST0", - &hclge_ppp_pf_abnormal_int[0], status); + if (status) { + reset_level = hclge_log_error(dev, "PPP_PF_ABNORMAL_INT_ST0", + &hclge_ppp_pf_abnormal_int[0], + status); + set_bit(reset_level, reset_requests); + } /* log PPU(RCB) PF errors */ desc_data = (__le32 *)&desc[3]; status = le32_to_cpu(*desc_data) & HCLGE_PPU_PF_INT_MSIX_MASK; - if (status) - hclge_log_error(dev, "PPU_PF_ABNORMAL_INT_ST", - &hclge_ppu_pf_abnormal_int[0], status); + if (status) { + reset_level = hclge_log_error(dev, "PPU_PF_ABNORMAL_INT_ST", + &hclge_ppu_pf_abnormal_int[0], + status); + set_bit(reset_level, reset_requests); + } /* clear all PF MSIx errors */ hclge_cmd_reuse_desc(&desc[0], false); @@ -1463,8 +1759,31 @@ int hclge_handle_hw_msix_error(struct hclge_dev *hdev, if (ret) { dev_err(dev, "clear all pf msix int cmd failed (%d)\n", ret); - /* reset everything for now */ - set_bit(HNAE3_GLOBAL_RESET, reset_requests); + } + + /* query and clear mac tnl interruptions */ + hclge_cmd_setup_basic_desc(&desc[0], HCLGE_OPC_QUERY_MAC_TNL_INT, + true); + ret = hclge_cmd_send(&hdev->hw, &desc[0], 1); + if (ret) { + dev_err(dev, "query mac tnl int cmd failed (%d)\n", ret); + goto msi_error; + } + + status = le32_to_cpu(desc->data[0]); + if (status) { + /* When mac tnl interrupt occurs, we record current time and + * register status here in a fifo, then clear the status. So + * that if link status changes suddenly at some time, we can + * query them by debugfs. + */ + mac_tnl_stats.time = local_clock(); + mac_tnl_stats.status = status; + kfifo_put(&hdev->mac_tnl_log, mac_tnl_stats); + ret = hclge_clear_mac_tnl_int(hdev); + if (ret) + dev_err(dev, "clear mac tnl int failed (%d)\n", ret); + set_bit(HNAE3_NONE_RESET, reset_requests); } msi_error: diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_err.h b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_err.h index fc068280d3917f6ae893e78c1e95ac160844f3a8..9645590c92942c4a16ac52b7d9e0b729fa3b3fb3 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_err.h +++ b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_err.h @@ -47,6 +47,9 @@ #define HCLGE_NCSI_ERR_INT_TYPE 0x9 #define HCLGE_MAC_COMMON_ERR_INT_EN 0x107FF #define HCLGE_MAC_COMMON_ERR_INT_EN_MASK 0x107FF +#define HCLGE_MAC_TNL_INT_EN GENMASK(7, 0) +#define HCLGE_MAC_TNL_INT_EN_MASK GENMASK(7, 0) +#define HCLGE_MAC_TNL_INT_CLR GENMASK(7, 0) #define HCLGE_PPU_MPF_ABNORMAL_INT0_EN GENMASK(31, 0) #define HCLGE_PPU_MPF_ABNORMAL_INT0_EN_MASK GENMASK(31, 0) #define HCLGE_PPU_MPF_ABNORMAL_INT1_EN GENMASK(31, 0) @@ -112,8 +115,10 @@ struct hclge_hw_blk { struct hclge_hw_error { u32 int_msk; const char *msg; + enum hnae3_reset_type reset_level; }; +int hclge_config_mac_tnl_int(struct hclge_dev *hdev, bool en); int hclge_hw_error_set_state(struct hclge_dev *hdev, bool state); pci_ers_result_t hclge_handle_hw_ras_error(struct hnae3_ae_dev *ae_dev); int hclge_handle_hw_msix_error(struct hclge_dev *hdev, diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_main.c b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_main.c index deda606c51e7359851026b3b370791bc6ea161b6..d3b1f8cb11558148bda430ef0453047b499a9836 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_main.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_main.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include "hclge_cmd.h" #include "hclge_dcb.h" @@ -31,6 +32,7 @@ static int hclge_set_mac_mtu(struct hclge_dev *hdev, int new_mps); static int hclge_init_vlan_config(struct hclge_dev *hdev); static int hclge_reset_ae_dev(struct hnae3_ae_dev *ae_dev); +static bool hclge_get_hw_reset_stat(struct hnae3_handle *handle); static int hclge_set_umv_space(struct hclge_dev *hdev, u16 space_size, u16 *allocated_size, bool is_alloc); @@ -697,6 +699,16 @@ static void hclge_get_stats(struct hnae3_handle *handle, u64 *data) p = hclge_tqps_get_stats(handle, p); } +static void hclge_get_mac_pause_stat(struct hnae3_handle *handle, u64 *tx_cnt, + u64 *rx_cnt) +{ + struct hclge_vport *vport = hclge_get_vport(handle); + struct hclge_dev *hdev = vport->back; + + *tx_cnt = hdev->hw_stats.mac_stats.mac_tx_mac_pause_num; + *rx_cnt = hdev->hw_stats.mac_stats.mac_rx_mac_pause_num; +} + static int hclge_parse_func_status(struct hclge_dev *hdev, struct hclge_func_status_cmd *status) { @@ -833,33 +845,189 @@ static int hclge_parse_speed(int speed_cmd, int *speed) return 0; } -static void hclge_parse_fiber_link_mode(struct hclge_dev *hdev, - u8 speed_ability) +static int hclge_check_port_speed(struct hnae3_handle *handle, u32 speed) { - unsigned long *supported = hdev->hw.mac.supported; + struct hclge_vport *vport = hclge_get_vport(handle); + struct hclge_dev *hdev = vport->back; + u32 speed_ability = hdev->hw.mac.speed_ability; + u32 speed_bit = 0; - if (speed_ability & HCLGE_SUPPORT_1G_BIT) - linkmode_set_bit(ETHTOOL_LINK_MODE_1000baseX_Full_BIT, - supported); + switch (speed) { + case HCLGE_MAC_SPEED_10M: + speed_bit = HCLGE_SUPPORT_10M_BIT; + break; + case HCLGE_MAC_SPEED_100M: + speed_bit = HCLGE_SUPPORT_100M_BIT; + break; + case HCLGE_MAC_SPEED_1G: + speed_bit = HCLGE_SUPPORT_1G_BIT; + break; + case HCLGE_MAC_SPEED_10G: + speed_bit = HCLGE_SUPPORT_10G_BIT; + break; + case HCLGE_MAC_SPEED_25G: + speed_bit = HCLGE_SUPPORT_25G_BIT; + break; + case HCLGE_MAC_SPEED_40G: + speed_bit = HCLGE_SUPPORT_40G_BIT; + break; + case HCLGE_MAC_SPEED_50G: + speed_bit = HCLGE_SUPPORT_50G_BIT; + break; + case HCLGE_MAC_SPEED_100G: + speed_bit = HCLGE_SUPPORT_100G_BIT; + break; + default: + return -EINVAL; + } + + if (speed_bit & speed_ability) + return 0; + + return -EINVAL; +} +static void hclge_convert_setting_sr(struct hclge_mac *mac, u8 speed_ability) +{ if (speed_ability & HCLGE_SUPPORT_10G_BIT) linkmode_set_bit(ETHTOOL_LINK_MODE_10000baseSR_Full_BIT, - supported); + mac->supported); + if (speed_ability & HCLGE_SUPPORT_25G_BIT) + linkmode_set_bit(ETHTOOL_LINK_MODE_25000baseSR_Full_BIT, + mac->supported); + if (speed_ability & HCLGE_SUPPORT_40G_BIT) + linkmode_set_bit(ETHTOOL_LINK_MODE_40000baseSR4_Full_BIT, + mac->supported); + if (speed_ability & HCLGE_SUPPORT_50G_BIT) + linkmode_set_bit(ETHTOOL_LINK_MODE_50000baseSR2_Full_BIT, + mac->supported); + if (speed_ability & HCLGE_SUPPORT_100G_BIT) + linkmode_set_bit(ETHTOOL_LINK_MODE_100000baseSR4_Full_BIT, + mac->supported); +} +static void hclge_convert_setting_lr(struct hclge_mac *mac, u8 speed_ability) +{ + if (speed_ability & HCLGE_SUPPORT_10G_BIT) + linkmode_set_bit(ETHTOOL_LINK_MODE_10000baseLR_Full_BIT, + mac->supported); if (speed_ability & HCLGE_SUPPORT_25G_BIT) linkmode_set_bit(ETHTOOL_LINK_MODE_25000baseSR_Full_BIT, - supported); + mac->supported); + if (speed_ability & HCLGE_SUPPORT_50G_BIT) + linkmode_set_bit(ETHTOOL_LINK_MODE_50000baseLR_ER_FR_Full_BIT, + mac->supported); + if (speed_ability & HCLGE_SUPPORT_40G_BIT) + linkmode_set_bit(ETHTOOL_LINK_MODE_40000baseLR4_Full_BIT, + mac->supported); + if (speed_ability & HCLGE_SUPPORT_100G_BIT) + linkmode_set_bit(ETHTOOL_LINK_MODE_100000baseLR4_ER4_Full_BIT, + mac->supported); +} +static void hclge_convert_setting_cr(struct hclge_mac *mac, u8 speed_ability) +{ + if (speed_ability & HCLGE_SUPPORT_10G_BIT) + linkmode_set_bit(ETHTOOL_LINK_MODE_10000baseCR_Full_BIT, + mac->supported); + if (speed_ability & HCLGE_SUPPORT_25G_BIT) + linkmode_set_bit(ETHTOOL_LINK_MODE_25000baseCR_Full_BIT, + mac->supported); + if (speed_ability & HCLGE_SUPPORT_40G_BIT) + linkmode_set_bit(ETHTOOL_LINK_MODE_40000baseCR4_Full_BIT, + mac->supported); if (speed_ability & HCLGE_SUPPORT_50G_BIT) - linkmode_set_bit(ETHTOOL_LINK_MODE_50000baseSR2_Full_BIT, - supported); + linkmode_set_bit(ETHTOOL_LINK_MODE_50000baseCR2_Full_BIT, + mac->supported); + if (speed_ability & HCLGE_SUPPORT_100G_BIT) + linkmode_set_bit(ETHTOOL_LINK_MODE_100000baseCR4_Full_BIT, + mac->supported); +} +static void hclge_convert_setting_kr(struct hclge_mac *mac, u8 speed_ability) +{ + if (speed_ability & HCLGE_SUPPORT_1G_BIT) + linkmode_set_bit(ETHTOOL_LINK_MODE_1000baseKX_Full_BIT, + mac->supported); + if (speed_ability & HCLGE_SUPPORT_10G_BIT) + linkmode_set_bit(ETHTOOL_LINK_MODE_10000baseKR_Full_BIT, + mac->supported); + if (speed_ability & HCLGE_SUPPORT_25G_BIT) + linkmode_set_bit(ETHTOOL_LINK_MODE_25000baseKR_Full_BIT, + mac->supported); + if (speed_ability & HCLGE_SUPPORT_40G_BIT) + linkmode_set_bit(ETHTOOL_LINK_MODE_40000baseKR4_Full_BIT, + mac->supported); + if (speed_ability & HCLGE_SUPPORT_50G_BIT) + linkmode_set_bit(ETHTOOL_LINK_MODE_50000baseKR2_Full_BIT, + mac->supported); if (speed_ability & HCLGE_SUPPORT_100G_BIT) - linkmode_set_bit(ETHTOOL_LINK_MODE_100000baseSR4_Full_BIT, - supported); + linkmode_set_bit(ETHTOOL_LINK_MODE_100000baseKR4_Full_BIT, + mac->supported); +} - linkmode_set_bit(ETHTOOL_LINK_MODE_FIBRE_BIT, supported); - linkmode_set_bit(ETHTOOL_LINK_MODE_Pause_BIT, supported); +static void hclge_convert_setting_fec(struct hclge_mac *mac) +{ + linkmode_clear_bit(ETHTOOL_LINK_MODE_FEC_BASER_BIT, mac->supported); + linkmode_clear_bit(ETHTOOL_LINK_MODE_FEC_RS_BIT, mac->supported); + + switch (mac->speed) { + case HCLGE_MAC_SPEED_10G: + case HCLGE_MAC_SPEED_40G: + linkmode_set_bit(ETHTOOL_LINK_MODE_FEC_BASER_BIT, + mac->supported); + mac->fec_ability = + BIT(HNAE3_FEC_BASER) | BIT(HNAE3_FEC_AUTO); + break; + case HCLGE_MAC_SPEED_25G: + case HCLGE_MAC_SPEED_50G: + linkmode_set_bit(ETHTOOL_LINK_MODE_FEC_RS_BIT, + mac->supported); + mac->fec_ability = + BIT(HNAE3_FEC_BASER) | BIT(HNAE3_FEC_RS) | + BIT(HNAE3_FEC_AUTO); + break; + case HCLGE_MAC_SPEED_100G: + linkmode_set_bit(ETHTOOL_LINK_MODE_FEC_RS_BIT, mac->supported); + mac->fec_ability = BIT(HNAE3_FEC_RS) | BIT(HNAE3_FEC_AUTO); + break; + default: + mac->fec_ability = 0; + break; + } +} + +static void hclge_parse_fiber_link_mode(struct hclge_dev *hdev, + u8 speed_ability) +{ + struct hclge_mac *mac = &hdev->hw.mac; + + if (speed_ability & HCLGE_SUPPORT_1G_BIT) + linkmode_set_bit(ETHTOOL_LINK_MODE_1000baseX_Full_BIT, + mac->supported); + + hclge_convert_setting_sr(mac, speed_ability); + hclge_convert_setting_lr(mac, speed_ability); + hclge_convert_setting_cr(mac, speed_ability); + if (hdev->pdev->revision >= 0x21) + hclge_convert_setting_fec(mac); + + linkmode_set_bit(ETHTOOL_LINK_MODE_FIBRE_BIT, mac->supported); + linkmode_set_bit(ETHTOOL_LINK_MODE_Pause_BIT, mac->supported); + linkmode_set_bit(ETHTOOL_LINK_MODE_FEC_NONE_BIT, mac->supported); +} + +static void hclge_parse_backplane_link_mode(struct hclge_dev *hdev, + u8 speed_ability) +{ + struct hclge_mac *mac = &hdev->hw.mac; + + hclge_convert_setting_kr(mac, speed_ability); + if (hdev->pdev->revision >= 0x21) + hclge_convert_setting_fec(mac); + linkmode_set_bit(ETHTOOL_LINK_MODE_Backplane_BIT, mac->supported); + linkmode_set_bit(ETHTOOL_LINK_MODE_Pause_BIT, mac->supported); + linkmode_set_bit(ETHTOOL_LINK_MODE_FEC_NONE_BIT, mac->supported); } static void hclge_parse_copper_link_mode(struct hclge_dev *hdev, @@ -900,8 +1068,9 @@ static void hclge_parse_link_mode(struct hclge_dev *hdev, u8 speed_ability) hclge_parse_fiber_link_mode(hdev, speed_ability); else if (media_type == HNAE3_MEDIA_TYPE_COPPER) hclge_parse_copper_link_mode(hdev, speed_ability); + else if (media_type == HNAE3_MEDIA_TYPE_BACKPLANE) + hclge_parse_backplane_link_mode(hdev, speed_ability); } - static void hclge_parse_cfg(struct hclge_cfg *cfg, struct hclge_desc *desc) { struct hclge_cfg_param_cmd *req; @@ -1015,6 +1184,23 @@ static int hclge_get_cap(struct hclge_dev *hdev) return ret; } +static void hclge_init_kdump_kernel_config(struct hclge_dev *hdev) +{ +#define HCLGE_MIN_TX_DESC 64 +#define HCLGE_MIN_RX_DESC 64 + + if (!is_kdump_kernel()) + return; + + dev_info(&hdev->pdev->dev, + "Running kdump kernel. Using minimal resources\n"); + + /* minimal queue pairs equals to the number of vports */ + hdev->num_tqps = hdev->num_vmdq_vport + hdev->num_req_vfs + 1; + hdev->num_tx_desc = HCLGE_MIN_TX_DESC; + hdev->num_rx_desc = HCLGE_MIN_RX_DESC; +} + static int hclge_configure(struct hclge_dev *hdev) { struct hclge_cfg cfg; @@ -1074,6 +1260,8 @@ static int hclge_configure(struct hclge_dev *hdev) hdev->tx_sch_mode = HCLGE_FLAG_TC_BASE_SCH_MODE; + hclge_init_kdump_kernel_config(hdev); + return ret; } @@ -1337,6 +1525,8 @@ static int hclge_alloc_vport(struct hclge_dev *hdev) vport->back = hdev; vport->vport_id = i; vport->mps = HCLGE_MAC_DEFAULT_FRAME; + vport->port_base_vlan_cfg.state = HNAE3_PORT_BASE_VLAN_DISABLE; + vport->rxvlan_cfg.rx_vlan_offload_en = true; INIT_LIST_HEAD(&vport->vlan_list); INIT_LIST_HEAD(&vport->uc_mac_list); INIT_LIST_HEAD(&vport->mc_mac_list); @@ -1399,7 +1589,7 @@ static int hclge_tx_buffer_alloc(struct hclge_dev *hdev, return ret; } -static int hclge_get_tc_num(struct hclge_dev *hdev) +static u32 hclge_get_tc_num(struct hclge_dev *hdev) { int i, cnt = 0; @@ -1409,17 +1599,6 @@ static int hclge_get_tc_num(struct hclge_dev *hdev) return cnt; } -static int hclge_get_pfc_enalbe_num(struct hclge_dev *hdev) -{ - int i, cnt = 0; - - for (i = 0; i < HCLGE_MAX_TC_NUM; i++) - if (hdev->hw_tc_map & BIT(i) && - hdev->tm_info.hw_pfc_map & BIT(i)) - cnt++; - return cnt; -} - /* Get the number of pfc enabled TCs, which have private buffer */ static int hclge_get_pfc_priv_num(struct hclge_dev *hdev, struct hclge_pkt_buf_alloc *buf_alloc) @@ -1483,14 +1662,12 @@ static bool hclge_is_rx_buf_ok(struct hclge_dev *hdev, struct hclge_pkt_buf_alloc *buf_alloc, u32 rx_all) { - u32 shared_buf_min, shared_buf_tc, shared_std; - int tc_num, pfc_enable_num; + u32 shared_buf_min, shared_buf_tc, shared_std, hi_thrd, lo_thrd; + u32 tc_num = hclge_get_tc_num(hdev); u32 shared_buf, aligned_mps; u32 rx_priv; int i; - tc_num = hclge_get_tc_num(hdev); - pfc_enable_num = hclge_get_pfc_enalbe_num(hdev); aligned_mps = roundup(hdev->mps, HCLGE_BUF_SIZE_UNIT); if (hnae3_dev_dcb_supported(hdev)) @@ -1499,9 +1676,7 @@ static bool hclge_is_rx_buf_ok(struct hclge_dev *hdev, shared_buf_min = aligned_mps + HCLGE_NON_DCB_ADDITIONAL_BUF + hdev->dv_buf_size; - shared_buf_tc = pfc_enable_num * aligned_mps + - (tc_num - pfc_enable_num) * aligned_mps / 2 + - aligned_mps; + shared_buf_tc = tc_num * aligned_mps + aligned_mps; shared_std = roundup(max_t(u32, shared_buf_min, shared_buf_tc), HCLGE_BUF_SIZE_UNIT); @@ -1518,19 +1693,26 @@ static bool hclge_is_rx_buf_ok(struct hclge_dev *hdev, } else { buf_alloc->s_buf.self.high = aligned_mps + HCLGE_NON_DCB_ADDITIONAL_BUF; - buf_alloc->s_buf.self.low = - roundup(aligned_mps / 2, HCLGE_BUF_SIZE_UNIT); + buf_alloc->s_buf.self.low = aligned_mps; + } + + if (hnae3_dev_dcb_supported(hdev)) { + if (tc_num) + hi_thrd = (shared_buf - hdev->dv_buf_size) / tc_num; + else + hi_thrd = shared_buf - hdev->dv_buf_size; + + hi_thrd = max_t(u32, hi_thrd, 2 * aligned_mps); + hi_thrd = rounddown(hi_thrd, HCLGE_BUF_SIZE_UNIT); + lo_thrd = hi_thrd - aligned_mps / 2; + } else { + hi_thrd = aligned_mps + HCLGE_NON_DCB_ADDITIONAL_BUF; + lo_thrd = aligned_mps; } for (i = 0; i < HCLGE_MAX_TC_NUM; i++) { - if ((hdev->hw_tc_map & BIT(i)) && - (hdev->tm_info.hw_pfc_map & BIT(i))) { - buf_alloc->s_buf.tc_thrd[i].low = aligned_mps; - buf_alloc->s_buf.tc_thrd[i].high = 2 * aligned_mps; - } else { - buf_alloc->s_buf.tc_thrd[i].low = 0; - buf_alloc->s_buf.tc_thrd[i].high = aligned_mps; - } + buf_alloc->s_buf.tc_thrd[i].low = lo_thrd; + buf_alloc->s_buf.tc_thrd[i].high = hi_thrd; } return true; @@ -2095,6 +2277,16 @@ static int hclge_set_autoneg(struct hnae3_handle *handle, bool enable) struct hclge_vport *vport = hclge_get_vport(handle); struct hclge_dev *hdev = vport->back; + if (!hdev->hw.mac.support_autoneg) { + if (enable) { + dev_err(&hdev->pdev->dev, + "autoneg is not supported by current port\n"); + return -EOPNOTSUPP; + } else { + return 0; + } + } + return hclge_set_autoneg_en(hdev, enable); } @@ -2110,6 +2302,78 @@ static int hclge_get_autoneg(struct hnae3_handle *handle) return hdev->hw.mac.autoneg; } +static int hclge_restart_autoneg(struct hnae3_handle *handle) +{ + struct hclge_vport *vport = hclge_get_vport(handle); + struct hclge_dev *hdev = vport->back; + int ret; + + dev_dbg(&hdev->pdev->dev, "restart autoneg\n"); + + ret = hclge_notify_client(hdev, HNAE3_DOWN_CLIENT); + if (ret) + return ret; + return hclge_notify_client(hdev, HNAE3_UP_CLIENT); +} + +static int hclge_set_fec_hw(struct hclge_dev *hdev, u32 fec_mode) +{ + struct hclge_config_fec_cmd *req; + struct hclge_desc desc; + int ret; + + hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_CONFIG_FEC_MODE, false); + + req = (struct hclge_config_fec_cmd *)desc.data; + if (fec_mode & BIT(HNAE3_FEC_AUTO)) + hnae3_set_bit(req->fec_mode, HCLGE_MAC_CFG_FEC_AUTO_EN_B, 1); + if (fec_mode & BIT(HNAE3_FEC_RS)) + hnae3_set_field(req->fec_mode, HCLGE_MAC_CFG_FEC_MODE_M, + HCLGE_MAC_CFG_FEC_MODE_S, HCLGE_MAC_FEC_RS); + if (fec_mode & BIT(HNAE3_FEC_BASER)) + hnae3_set_field(req->fec_mode, HCLGE_MAC_CFG_FEC_MODE_M, + HCLGE_MAC_CFG_FEC_MODE_S, HCLGE_MAC_FEC_BASER); + + ret = hclge_cmd_send(&hdev->hw, &desc, 1); + if (ret) + dev_err(&hdev->pdev->dev, "set fec mode failed %d.\n", ret); + + return ret; +} + +static int hclge_set_fec(struct hnae3_handle *handle, u32 fec_mode) +{ + struct hclge_vport *vport = hclge_get_vport(handle); + struct hclge_dev *hdev = vport->back; + struct hclge_mac *mac = &hdev->hw.mac; + int ret; + + if (fec_mode && !(mac->fec_ability & fec_mode)) { + dev_err(&hdev->pdev->dev, "unsupported fec mode\n"); + return -EINVAL; + } + + ret = hclge_set_fec_hw(hdev, fec_mode); + if (ret) + return ret; + + mac->user_fec_mode = fec_mode | BIT(HNAE3_FEC_USER_DEF); + return 0; +} + +static void hclge_get_fec(struct hnae3_handle *handle, u8 *fec_ability, + u8 *fec_mode) +{ + struct hclge_vport *vport = hclge_get_vport(handle); + struct hclge_dev *hdev = vport->back; + struct hclge_mac *mac = &hdev->hw.mac; + + if (fec_ability) + *fec_ability = mac->fec_ability; + if (fec_mode) + *fec_mode = mac->fec_mode; +} + static int hclge_mac_init(struct hclge_dev *hdev) { struct hclge_mac *mac = &hdev->hw.mac; @@ -2127,6 +2391,15 @@ static int hclge_mac_init(struct hclge_dev *hdev) mac->link = 0; + if (mac->user_fec_mode & BIT(HNAE3_FEC_USER_DEF)) { + ret = hclge_set_fec_hw(hdev, mac->user_fec_mode); + if (ret) { + dev_err(&hdev->pdev->dev, + "Fec mode init fail, ret = %d\n", ret); + return ret; + } + } + ret = hclge_set_mac_mtu(hdev, hdev->mps); if (ret) { dev_err(&hdev->pdev->dev, "set mtu failed ret=%d\n", ret); @@ -2143,7 +2416,8 @@ static int hclge_mac_init(struct hclge_dev *hdev) static void hclge_mbx_task_schedule(struct hclge_dev *hdev) { - if (!test_and_set_bit(HCLGE_STATE_MBX_SERVICE_SCHED, &hdev->state)) + if (!test_bit(HCLGE_STATE_CMD_DISABLE, &hdev->state) && + !test_and_set_bit(HCLGE_STATE_MBX_SERVICE_SCHED, &hdev->state)) schedule_work(&hdev->mbx_service_task); } @@ -2222,6 +2496,7 @@ static void hclge_update_link_status(struct hclge_dev *hdev) for (i = 0; i < hdev->num_vmdq_vport + 1; i++) { handle = &hdev->vport[i].nic; client->ops->link_status_change(handle, state); + hclge_config_mac_tnl_int(hdev, state); rhandle = &hdev->vport[i].roce; if (rclient && rclient->ops->link_status_change) rclient->ops->link_status_change(rhandle, @@ -2231,14 +2506,35 @@ static void hclge_update_link_status(struct hclge_dev *hdev) } } +static void hclge_update_port_capability(struct hclge_mac *mac) +{ + /* firmware can not identify back plane type, the media type + * read from configuration can help deal it + */ + if (mac->media_type == HNAE3_MEDIA_TYPE_BACKPLANE && + mac->module_type == HNAE3_MODULE_TYPE_UNKNOWN) + mac->module_type = HNAE3_MODULE_TYPE_KR; + else if (mac->media_type == HNAE3_MEDIA_TYPE_COPPER) + mac->module_type = HNAE3_MODULE_TYPE_TP; + + if (mac->support_autoneg == true) { + linkmode_set_bit(ETHTOOL_LINK_MODE_Autoneg_BIT, mac->supported); + linkmode_copy(mac->advertising, mac->supported); + } else { + linkmode_clear_bit(ETHTOOL_LINK_MODE_Autoneg_BIT, + mac->supported); + linkmode_zero(mac->advertising); + } +} + static int hclge_get_sfp_speed(struct hclge_dev *hdev, u32 *speed) { - struct hclge_sfp_speed_cmd *resp = NULL; + struct hclge_sfp_info_cmd *resp = NULL; struct hclge_desc desc; int ret; - hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_SFP_GET_SPEED, true); - resp = (struct hclge_sfp_speed_cmd *)desc.data; + hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_GET_SFP_INFO, true); + resp = (struct hclge_sfp_info_cmd *)desc.data; ret = hclge_cmd_send(&hdev->hw, &desc, 1); if (ret == -EOPNOTSUPP) { dev_warn(&hdev->pdev->dev, @@ -2249,28 +2545,67 @@ static int hclge_get_sfp_speed(struct hclge_dev *hdev, u32 *speed) return ret; } - *speed = resp->sfp_speed; + *speed = le32_to_cpu(resp->speed); return 0; } -static int hclge_update_speed_duplex(struct hclge_dev *hdev) +static int hclge_get_sfp_info(struct hclge_dev *hdev, struct hclge_mac *mac) { - struct hclge_mac mac = hdev->hw.mac; - int speed; + struct hclge_sfp_info_cmd *resp; + struct hclge_desc desc; int ret; - /* get the speed from SFP cmd when phy - * doesn't exit. + hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_GET_SFP_INFO, true); + resp = (struct hclge_sfp_info_cmd *)desc.data; + + resp->query_type = QUERY_ACTIVE_SPEED; + + ret = hclge_cmd_send(&hdev->hw, &desc, 1); + if (ret == -EOPNOTSUPP) { + dev_warn(&hdev->pdev->dev, + "IMP does not support get SFP info %d\n", ret); + return ret; + } else if (ret) { + dev_err(&hdev->pdev->dev, "get sfp info failed %d\n", ret); + return ret; + } + + mac->speed = le32_to_cpu(resp->speed); + /* if resp->speed_ability is 0, it means it's an old version + * firmware, do not update these params */ - if (mac.phydev) + if (resp->speed_ability) { + mac->module_type = le32_to_cpu(resp->module_type); + mac->speed_ability = le32_to_cpu(resp->speed_ability); + mac->autoneg = resp->autoneg; + mac->support_autoneg = resp->autoneg_ability; + } else { + mac->speed_type = QUERY_SFP_SPEED; + } + + return 0; +} + +static int hclge_update_port_info(struct hclge_dev *hdev) +{ + struct hclge_mac *mac = &hdev->hw.mac; + int speed = HCLGE_MAC_SPEED_UNKNOWN; + int ret; + + /* get the port info from SFP cmd if not copper port */ + if (mac->media_type == HNAE3_MEDIA_TYPE_COPPER) return 0; - /* if IMP does not support get SFP/qSFP speed, return directly */ + /* if IMP does not support get SFP/qSFP info, return directly */ if (!hdev->support_sfp_query) return 0; - ret = hclge_get_sfp_speed(hdev, &speed); + if (hdev->pdev->revision >= 0x21) + ret = hclge_get_sfp_info(hdev, mac); + else + ret = hclge_get_sfp_speed(hdev, &speed); + if (ret == -EOPNOTSUPP) { hdev->support_sfp_query = false; return ret; @@ -2278,19 +2613,20 @@ static int hclge_update_speed_duplex(struct hclge_dev *hdev) return ret; } - if (speed == HCLGE_MAC_SPEED_UNKNOWN) - return 0; /* do nothing if no SFP */ - - /* must config full duplex for SFP */ - return hclge_cfg_mac_speed_dup(hdev, speed, HCLGE_MAC_FULL); -} - -static int hclge_update_speed_duplex_h(struct hnae3_handle *handle) -{ - struct hclge_vport *vport = hclge_get_vport(handle); - struct hclge_dev *hdev = vport->back; + if (hdev->pdev->revision >= 0x21) { + if (mac->speed_type == QUERY_ACTIVE_SPEED) { + hclge_update_port_capability(mac); + return 0; + } + return hclge_cfg_mac_speed_dup(hdev, mac->speed, + HCLGE_MAC_FULL); + } else { + if (speed == HCLGE_MAC_SPEED_UNKNOWN) + return 0; /* do nothing if no SFP */ - return hclge_update_speed_duplex(hdev); + /* must config full duplex for SFP */ + return hclge_cfg_mac_speed_dup(hdev, speed, HCLGE_MAC_FULL); + } } static int hclge_get_status(struct hnae3_handle *handle) @@ -2344,6 +2680,7 @@ static u32 hclge_check_event_cause(struct hclge_dev *hdev, u32 *clearval) set_bit(HNAE3_IMP_RESET, &hdev->reset_pending); set_bit(HCLGE_STATE_CMD_DISABLE, &hdev->state); *clearval = BIT(HCLGE_VECTOR0_IMPRESET_INT_B); + hdev->rst_stats.imp_rst_cnt++; return HCLGE_VECTOR0_EVENT_RST; } @@ -2352,6 +2689,7 @@ static u32 hclge_check_event_cause(struct hclge_dev *hdev, u32 *clearval) set_bit(HCLGE_STATE_CMD_DISABLE, &hdev->state); set_bit(HNAE3_GLOBAL_RESET, &hdev->reset_pending); *clearval = BIT(HCLGE_VECTOR0_GLOBALRESET_INT_B); + hdev->rst_stats.global_rst_cnt++; return HCLGE_VECTOR0_EVENT_RST; } @@ -2360,12 +2698,16 @@ static u32 hclge_check_event_cause(struct hclge_dev *hdev, u32 *clearval) set_bit(HCLGE_STATE_CMD_DISABLE, &hdev->state); set_bit(HNAE3_CORE_RESET, &hdev->reset_pending); *clearval = BIT(HCLGE_VECTOR0_CORERESET_INT_B); + hdev->rst_stats.core_rst_cnt++; return HCLGE_VECTOR0_EVENT_RST; } /* check for vector0 msix event source */ - if (msix_src_reg & HCLGE_VECTOR0_REG_MSIX_MASK) + if (msix_src_reg & HCLGE_VECTOR0_REG_MSIX_MASK) { + dev_dbg(&hdev->pdev->dev, "received event 0x%x\n", + msix_src_reg); return HCLGE_VECTOR0_EVENT_ERR; + } /* check for vector0 mailbox(=CMDQ RX) event source */ if (BIT(HCLGE_VECTOR0_RX_CMDQ_INT_B) & cmdq_src_reg) { @@ -2374,6 +2716,9 @@ static u32 hclge_check_event_cause(struct hclge_dev *hdev, u32 *clearval) return HCLGE_VECTOR0_EVENT_MBX; } + /* print other vector0 event source */ + dev_dbg(&hdev->pdev->dev, "cmdq_src_reg:0x%x, msix_src_reg:0x%x\n", + cmdq_src_reg, msix_src_reg); return HCLGE_VECTOR0_EVENT_OTHER; } @@ -2657,7 +3002,7 @@ static int hclge_set_all_vf_rst(struct hclge_dev *hdev, bool reset) return ret; } - if (!reset) + if (!reset || !test_bit(HCLGE_VPORT_STATE_ALIVE, &vport->state)) continue; /* Inform VF to process the reset. @@ -2694,9 +3039,18 @@ int hclge_func_reset_cmd(struct hclge_dev *hdev, int func_id) static void hclge_do_reset(struct hclge_dev *hdev) { + struct hnae3_handle *handle = &hdev->vport[0].nic; struct pci_dev *pdev = hdev->pdev; u32 val; + if (hclge_get_hw_reset_stat(handle)) { + dev_info(&pdev->dev, "Hardware reset not finish\n"); + dev_info(&pdev->dev, "func_rst_reg:0x%x, global_rst_reg:0x%x\n", + hclge_read_dev(&hdev->hw, HCLGE_FUN_RST_ING), + hclge_read_dev(&hdev->hw, HCLGE_GLOBAL_RESET_REG)); + return; + } + switch (hdev->reset_type) { case HNAE3_GLOBAL_RESET: val = hclge_read_dev(&hdev->hw, HCLGE_GLOBAL_RESET_REG); @@ -2775,6 +3129,10 @@ static enum hnae3_reset_type hclge_get_reset_level(struct hclge_dev *hdev, clear_bit(HNAE3_FLR_RESET, addr); } + if (hdev->reset_type != HNAE3_NONE_RESET && + rst_level < hdev->reset_type) + return HNAE3_NONE_RESET; + return rst_level; } @@ -2844,6 +3202,7 @@ static int hclge_reset_prepare_wait(struct hclge_dev *hdev) * after hclge_cmd_init is called. */ set_bit(HCLGE_STATE_CMD_DISABLE, &hdev->state); + hdev->rst_stats.pf_rst_cnt++; break; case HNAE3_FLR_RESET: /* There is no mechanism for PF to know if VF has stopped IO @@ -2852,6 +3211,7 @@ static int hclge_reset_prepare_wait(struct hclge_dev *hdev) msleep(100); set_bit(HCLGE_STATE_CMD_DISABLE, &hdev->state); set_bit(HNAE3_FLR_DOWN, &hdev->flr_state); + hdev->rst_stats.flr_rst_cnt++; break; case HNAE3_IMP_RESET: reg_val = hclge_read_dev(&hdev->hw, HCLGE_PF_OTHER_INT_REG); @@ -2932,7 +3292,7 @@ static void hclge_reset(struct hclge_dev *hdev) * know if device is undergoing reset */ ae_dev->reset_type = hdev->reset_type; - hdev->reset_count++; + hdev->rst_stats.reset_cnt++; /* perform reset of the stack & ae device for a client */ ret = hclge_notify_roce_client(hdev, HNAE3_DOWN_CLIENT); if (ret) @@ -2958,6 +3318,8 @@ static void hclge_reset(struct hclge_dev *hdev) goto err_reset; } + hdev->rst_stats.hw_reset_done_cnt++; + ret = hclge_notify_roce_client(hdev, HNAE3_UNINIT_CLIENT); if (ret) goto err_reset; @@ -3001,7 +3363,9 @@ static void hclge_reset(struct hclge_dev *hdev) hdev->last_reset_time = jiffies; hdev->reset_fail_cnt = 0; + hdev->rst_stats.reset_done_cnt++; ae_dev->reset_type = HNAE3_NONE_RESET; + del_timer(&hdev->reset_timer); return; @@ -3154,7 +3518,7 @@ static void hclge_service_task(struct work_struct *work) hdev->hw_stats.stats_timer = 0; } - hclge_update_speed_duplex(hdev); + hclge_update_port_info(hdev); hclge_update_link_status(hdev); hclge_update_vport_alive(hdev); hclge_service_complete(hdev); @@ -5194,7 +5558,7 @@ static unsigned long hclge_ae_dev_reset_cnt(struct hnae3_handle *handle) struct hclge_vport *vport = hclge_get_vport(handle); struct hclge_dev *hdev = vport->back; - return hdev->reset_count; + return hdev->rst_stats.hw_reset_done_cnt; } static void hclge_enable_fd(struct hnae3_handle *handle, bool enable) @@ -5282,8 +5646,8 @@ static int hclge_set_serdes_loopback(struct hclge_dev *hdev, bool en, #define HCLGE_SERDES_RETRY_MS 10 #define HCLGE_SERDES_RETRY_NUM 100 -#define HCLGE_MAC_LINK_STATUS_MS 20 -#define HCLGE_MAC_LINK_STATUS_NUM 10 +#define HCLGE_MAC_LINK_STATUS_MS 10 +#define HCLGE_MAC_LINK_STATUS_NUM 100 #define HCLGE_MAC_LINK_STATUS_DOWN 0 #define HCLGE_MAC_LINK_STATUS_UP 1 @@ -5942,8 +6306,11 @@ int hclge_add_uc_addr_common(struct hclge_vport *vport, } /* check if we just hit the duplicate */ - if (!ret) - ret = -EINVAL; + if (!ret) { + dev_warn(&hdev->pdev->dev, "VF %d mac(%pM) exists\n", + vport->vport_id, addr); + return 0; + } dev_err(&hdev->pdev->dev, "PF failed to add unicast entry(%pM) in the MAC table\n", @@ -6293,7 +6660,8 @@ static int hclge_set_mac_addr(struct hnae3_handle *handle, void *p, return -EINVAL; } - if (!is_first && hclge_rm_uc_addr(handle, hdev->hw.mac.mac_addr)) + if ((!is_first || is_kdump_kernel()) && + hclge_rm_uc_addr(handle, hdev->hw.mac.mac_addr)) dev_warn(&hdev->pdev->dev, "remove old uc mac address fail.\n"); @@ -6543,30 +6911,6 @@ static int hclge_set_vlan_filter_hw(struct hclge_dev *hdev, __be16 proto, return ret; } -int hclge_set_vlan_filter(struct hnae3_handle *handle, __be16 proto, - u16 vlan_id, bool is_kill) -{ - struct hclge_vport *vport = hclge_get_vport(handle); - struct hclge_dev *hdev = vport->back; - - return hclge_set_vlan_filter_hw(hdev, proto, vport->vport_id, vlan_id, - 0, is_kill); -} - -static int hclge_set_vf_vlan_filter(struct hnae3_handle *handle, int vfid, - u16 vlan, u8 qos, __be16 proto) -{ - struct hclge_vport *vport = hclge_get_vport(handle); - struct hclge_dev *hdev = vport->back; - - if ((vfid >= hdev->num_alloc_vfs) || (vlan > 4095) || (qos > 7)) - return -EINVAL; - if (proto != htons(ETH_P_8021Q)) - return -EPROTONOSUPPORT; - - return hclge_set_vlan_filter_hw(hdev, proto, vfid, vlan, qos, false); -} - static int hclge_set_vlan_tx_offload_cfg(struct hclge_vport *vport) { struct hclge_tx_vtag_cfg *vcfg = &vport->txvlan_cfg; @@ -6640,6 +6984,52 @@ static int hclge_set_vlan_rx_offload_cfg(struct hclge_vport *vport) return status; } +static int hclge_vlan_offload_cfg(struct hclge_vport *vport, + u16 port_base_vlan_state, + u16 vlan_tag) +{ + int ret; + + if (port_base_vlan_state == HNAE3_PORT_BASE_VLAN_DISABLE) { + vport->txvlan_cfg.accept_tag1 = true; + vport->txvlan_cfg.insert_tag1_en = false; + vport->txvlan_cfg.default_tag1 = 0; + } else { + vport->txvlan_cfg.accept_tag1 = false; + vport->txvlan_cfg.insert_tag1_en = true; + vport->txvlan_cfg.default_tag1 = vlan_tag; + } + + vport->txvlan_cfg.accept_untag1 = true; + + /* accept_tag2 and accept_untag2 are not supported on + * pdev revision(0x20), new revision support them, + * this two fields can not be configured by user. + */ + vport->txvlan_cfg.accept_tag2 = true; + vport->txvlan_cfg.accept_untag2 = true; + vport->txvlan_cfg.insert_tag2_en = false; + vport->txvlan_cfg.default_tag2 = 0; + + if (port_base_vlan_state == HNAE3_PORT_BASE_VLAN_DISABLE) { + vport->rxvlan_cfg.strip_tag1_en = false; + vport->rxvlan_cfg.strip_tag2_en = + vport->rxvlan_cfg.rx_vlan_offload_en; + } else { + vport->rxvlan_cfg.strip_tag1_en = + vport->rxvlan_cfg.rx_vlan_offload_en; + vport->rxvlan_cfg.strip_tag2_en = true; + } + vport->rxvlan_cfg.vlan1_vlan_prionly = false; + vport->rxvlan_cfg.vlan2_vlan_prionly = false; + + ret = hclge_set_vlan_tx_offload_cfg(vport); + if (ret) + return ret; + + return hclge_set_vlan_rx_offload_cfg(vport); +} + static int hclge_set_vlan_protocol_type(struct hclge_dev *hdev) { struct hclge_rx_vlan_type_cfg_cmd *rx_req; @@ -6730,34 +7120,14 @@ static int hclge_init_vlan_config(struct hclge_dev *hdev) return ret; for (i = 0; i < hdev->num_alloc_vport; i++) { - vport = &hdev->vport[i]; - vport->txvlan_cfg.accept_tag1 = true; - vport->txvlan_cfg.accept_untag1 = true; - - /* accept_tag2 and accept_untag2 are not supported on - * pdev revision(0x20), new revision support them. The - * value of this two fields will not return error when driver - * send command to fireware in revision(0x20). - * This two fields can not configured by user. - */ - vport->txvlan_cfg.accept_tag2 = true; - vport->txvlan_cfg.accept_untag2 = true; + u16 vlan_tag; - vport->txvlan_cfg.insert_tag1_en = false; - vport->txvlan_cfg.insert_tag2_en = false; - vport->txvlan_cfg.default_tag1 = 0; - vport->txvlan_cfg.default_tag2 = 0; - - ret = hclge_set_vlan_tx_offload_cfg(vport); - if (ret) - return ret; - - vport->rxvlan_cfg.strip_tag1_en = false; - vport->rxvlan_cfg.strip_tag2_en = true; - vport->rxvlan_cfg.vlan1_vlan_prionly = false; - vport->rxvlan_cfg.vlan2_vlan_prionly = false; + vport = &hdev->vport[i]; + vlan_tag = vport->port_base_vlan_cfg.vlan_info.vlan_tag; - ret = hclge_set_vlan_rx_offload_cfg(vport); + ret = hclge_vlan_offload_cfg(vport, + vport->port_base_vlan_cfg.state, + vlan_tag); if (ret) return ret; } @@ -6765,7 +7135,8 @@ static int hclge_init_vlan_config(struct hclge_dev *hdev) return hclge_set_vlan_filter(handle, htons(ETH_P_8021Q), 0, false); } -void hclge_add_vport_vlan_table(struct hclge_vport *vport, u16 vlan_id) +static void hclge_add_vport_vlan_table(struct hclge_vport *vport, u16 vlan_id, + bool writen_to_tbl) { struct hclge_vport_vlan_cfg *vlan; @@ -6777,14 +7148,38 @@ void hclge_add_vport_vlan_table(struct hclge_vport *vport, u16 vlan_id) if (!vlan) return; - vlan->hd_tbl_status = true; + vlan->hd_tbl_status = writen_to_tbl; vlan->vlan_id = vlan_id; list_add_tail(&vlan->node, &vport->vlan_list); } -void hclge_rm_vport_vlan_table(struct hclge_vport *vport, u16 vlan_id, - bool is_write_tbl) +static int hclge_add_vport_all_vlan_table(struct hclge_vport *vport) +{ + struct hclge_vport_vlan_cfg *vlan, *tmp; + struct hclge_dev *hdev = vport->back; + int ret; + + list_for_each_entry_safe(vlan, tmp, &vport->vlan_list, node) { + if (!vlan->hd_tbl_status) { + ret = hclge_set_vlan_filter_hw(hdev, htons(ETH_P_8021Q), + vport->vport_id, + vlan->vlan_id, 0, false); + if (ret) { + dev_err(&hdev->pdev->dev, + "restore vport vlan list failed, ret=%d\n", + ret); + return ret; + } + } + vlan->hd_tbl_status = true; + } + + return 0; +} + +static void hclge_rm_vport_vlan_table(struct hclge_vport *vport, u16 vlan_id, + bool is_write_tbl) { struct hclge_vport_vlan_cfg *vlan, *tmp; struct hclge_dev *hdev = vport->back; @@ -6847,14 +7242,203 @@ int hclge_en_hw_strip_rxvtag(struct hnae3_handle *handle, bool enable) { struct hclge_vport *vport = hclge_get_vport(handle); - vport->rxvlan_cfg.strip_tag1_en = false; - vport->rxvlan_cfg.strip_tag2_en = enable; + if (vport->port_base_vlan_cfg.state == HNAE3_PORT_BASE_VLAN_DISABLE) { + vport->rxvlan_cfg.strip_tag1_en = false; + vport->rxvlan_cfg.strip_tag2_en = enable; + } else { + vport->rxvlan_cfg.strip_tag1_en = enable; + vport->rxvlan_cfg.strip_tag2_en = true; + } vport->rxvlan_cfg.vlan1_vlan_prionly = false; vport->rxvlan_cfg.vlan2_vlan_prionly = false; + vport->rxvlan_cfg.rx_vlan_offload_en = enable; return hclge_set_vlan_rx_offload_cfg(vport); } +static int hclge_update_vlan_filter_entries(struct hclge_vport *vport, + u16 port_base_vlan_state, + struct hclge_vlan_info *new_info, + struct hclge_vlan_info *old_info) +{ + struct hclge_dev *hdev = vport->back; + int ret; + + if (port_base_vlan_state == HNAE3_PORT_BASE_VLAN_ENABLE) { + hclge_rm_vport_all_vlan_table(vport, false); + return hclge_set_vlan_filter_hw(hdev, + htons(new_info->vlan_proto), + vport->vport_id, + new_info->vlan_tag, + new_info->qos, false); + } + + ret = hclge_set_vlan_filter_hw(hdev, htons(old_info->vlan_proto), + vport->vport_id, old_info->vlan_tag, + old_info->qos, true); + if (ret) + return ret; + + return hclge_add_vport_all_vlan_table(vport); +} + +int hclge_update_port_base_vlan_cfg(struct hclge_vport *vport, u16 state, + struct hclge_vlan_info *vlan_info) +{ + struct hnae3_handle *nic = &vport->nic; + struct hclge_vlan_info *old_vlan_info; + struct hclge_dev *hdev = vport->back; + int ret; + + old_vlan_info = &vport->port_base_vlan_cfg.vlan_info; + + ret = hclge_vlan_offload_cfg(vport, state, vlan_info->vlan_tag); + if (ret) + return ret; + + if (state == HNAE3_PORT_BASE_VLAN_MODIFY) { + /* add new VLAN tag */ + ret = hclge_set_vlan_filter_hw(hdev, + htons(vlan_info->vlan_proto), + vport->vport_id, + vlan_info->vlan_tag, + vlan_info->qos, false); + if (ret) + return ret; + + /* remove old VLAN tag */ + ret = hclge_set_vlan_filter_hw(hdev, + htons(old_vlan_info->vlan_proto), + vport->vport_id, + old_vlan_info->vlan_tag, + old_vlan_info->qos, true); + if (ret) + return ret; + + goto update; + } + + ret = hclge_update_vlan_filter_entries(vport, state, vlan_info, + old_vlan_info); + if (ret) + return ret; + + /* update state only when disable/enable port based VLAN */ + vport->port_base_vlan_cfg.state = state; + if (state == HNAE3_PORT_BASE_VLAN_DISABLE) + nic->port_base_vlan_state = HNAE3_PORT_BASE_VLAN_DISABLE; + else + nic->port_base_vlan_state = HNAE3_PORT_BASE_VLAN_ENABLE; + +update: + vport->port_base_vlan_cfg.vlan_info.vlan_tag = vlan_info->vlan_tag; + vport->port_base_vlan_cfg.vlan_info.qos = vlan_info->qos; + vport->port_base_vlan_cfg.vlan_info.vlan_proto = vlan_info->vlan_proto; + + return 0; +} + +static u16 hclge_get_port_base_vlan_state(struct hclge_vport *vport, + enum hnae3_port_base_vlan_state state, + u16 vlan) +{ + if (state == HNAE3_PORT_BASE_VLAN_DISABLE) { + if (!vlan) + return HNAE3_PORT_BASE_VLAN_NOCHANGE; + else + return HNAE3_PORT_BASE_VLAN_ENABLE; + } else { + if (!vlan) + return HNAE3_PORT_BASE_VLAN_DISABLE; + else if (vport->port_base_vlan_cfg.vlan_info.vlan_tag == vlan) + return HNAE3_PORT_BASE_VLAN_NOCHANGE; + else + return HNAE3_PORT_BASE_VLAN_MODIFY; + } +} + +static int hclge_set_vf_vlan_filter(struct hnae3_handle *handle, int vfid, + u16 vlan, u8 qos, __be16 proto) +{ + struct hclge_vport *vport = hclge_get_vport(handle); + struct hclge_dev *hdev = vport->back; + struct hclge_vlan_info vlan_info; + u16 state; + int ret; + + if (hdev->pdev->revision == 0x20) + return -EOPNOTSUPP; + + /* qos is a 3 bits value, so can not be bigger than 7 */ + if (vfid >= hdev->num_alloc_vfs || vlan > VLAN_N_VID - 1 || qos > 7) + return -EINVAL; + if (proto != htons(ETH_P_8021Q)) + return -EPROTONOSUPPORT; + + vport = &hdev->vport[vfid]; + state = hclge_get_port_base_vlan_state(vport, + vport->port_base_vlan_cfg.state, + vlan); + if (state == HNAE3_PORT_BASE_VLAN_NOCHANGE) + return 0; + + vlan_info.vlan_tag = vlan; + vlan_info.qos = qos; + vlan_info.vlan_proto = ntohs(proto); + + /* update port based VLAN for PF */ + if (!vfid) { + hclge_notify_client(hdev, HNAE3_DOWN_CLIENT); + ret = hclge_update_port_base_vlan_cfg(vport, state, &vlan_info); + hclge_notify_client(hdev, HNAE3_UP_CLIENT); + + return ret; + } + + if (!test_bit(HCLGE_VPORT_STATE_ALIVE, &vport->state)) { + return hclge_update_port_base_vlan_cfg(vport, state, + &vlan_info); + } else { + ret = hclge_push_vf_port_base_vlan_info(&hdev->vport[0], + (u8)vfid, state, + vlan, qos, + ntohs(proto)); + return ret; + } +} + +int hclge_set_vlan_filter(struct hnae3_handle *handle, __be16 proto, + u16 vlan_id, bool is_kill) +{ + struct hclge_vport *vport = hclge_get_vport(handle); + struct hclge_dev *hdev = vport->back; + bool writen_to_tbl = false; + int ret = 0; + + /* when port based VLAN enabled, we use port based VLAN as the VLAN + * filter entry. In this case, we don't update VLAN filter table + * when user add new VLAN or remove exist VLAN, just update the vport + * VLAN list. The VLAN id in VLAN list won't be writen in VLAN filter + * table until port based VLAN disabled + */ + if (handle->port_base_vlan_state == HNAE3_PORT_BASE_VLAN_DISABLE) { + ret = hclge_set_vlan_filter_hw(hdev, proto, vport->vport_id, + vlan_id, 0, is_kill); + writen_to_tbl = true; + } + + if (ret) + return ret; + + if (is_kill) + hclge_rm_vport_vlan_table(vport, vlan_id, false); + else + hclge_add_vport_vlan_table(vport, vlan_id, + writen_to_tbl); + + return 0; +} + static int hclge_set_mac_mtu(struct hclge_dev *hdev, int new_mps) { struct hclge_config_max_frm_size_cmd *req; @@ -7199,13 +7783,13 @@ static int hclge_set_pauseparam(struct hnae3_handle *handle, u32 auto_neg, if (!fc_autoneg) return hclge_cfg_pauseparam(hdev, rx_en, tx_en); - /* Only support flow control negotiation for netdev with - * phy attached for now. - */ - if (!phydev) + if (phydev) + return phy_start_aneg(phydev); + + if (hdev->pdev->revision == 0x20) return -EOPNOTSUPP; - return phy_start_aneg(phydev); + return hclge_restart_autoneg(handle); } static void hclge_get_ksettings_an_result(struct hnae3_handle *handle, @@ -7222,13 +7806,17 @@ static void hclge_get_ksettings_an_result(struct hnae3_handle *handle, *auto_neg = hdev->hw.mac.autoneg; } -static void hclge_get_media_type(struct hnae3_handle *handle, u8 *media_type) +static void hclge_get_media_type(struct hnae3_handle *handle, u8 *media_type, + u8 *module_type) { struct hclge_vport *vport = hclge_get_vport(handle); struct hclge_dev *hdev = vport->back; if (media_type) *media_type = hdev->hw.mac.media_type; + + if (module_type) + *module_type = hdev->hw.mac.module_type; } static void hclge_get_mdix_mode(struct hnae3_handle *handle, @@ -7280,6 +7868,32 @@ static void hclge_get_mdix_mode(struct hnae3_handle *handle, *tp_mdix = ETH_TP_MDI; } +static void hclge_info_show(struct hclge_dev *hdev) +{ + struct device *dev = &hdev->pdev->dev; + + dev_info(dev, "PF info begin:\n"); + + dev_info(dev, "Task queue pairs numbers: %d\n", hdev->num_tqps); + dev_info(dev, "Desc num per TX queue: %d\n", hdev->num_tx_desc); + dev_info(dev, "Desc num per RX queue: %d\n", hdev->num_rx_desc); + dev_info(dev, "Numbers of vports: %d\n", hdev->num_alloc_vport); + dev_info(dev, "Numbers of vmdp vports: %d\n", hdev->num_vmdq_vport); + dev_info(dev, "Numbers of VF for this PF: %d\n", hdev->num_req_vfs); + dev_info(dev, "HW tc map: %d\n", hdev->hw_tc_map); + dev_info(dev, "Total buffer size for TX/RX: %d\n", hdev->pkt_buf_size); + dev_info(dev, "TX buffer size for each TC: %d\n", hdev->tx_buf_size); + dev_info(dev, "DV buffer size for each TC: %d\n", hdev->dv_buf_size); + dev_info(dev, "This is %s PF\n", + hdev->flag & HCLGE_FLAG_MAIN ? "main" : "not main"); + dev_info(dev, "DCB %s\n", + hdev->flag & HCLGE_FLAG_DCB_ENABLE ? "enable" : "disable"); + dev_info(dev, "MQPRIO %s\n", + hdev->flag & HCLGE_FLAG_MQPRIO_ENABLE ? "enable" : "disable"); + + dev_info(dev, "PF info end.\n"); +} + static int hclge_init_client_instance(struct hnae3_client *client, struct hnae3_ae_dev *ae_dev) { @@ -7301,6 +7915,9 @@ static int hclge_init_client_instance(struct hnae3_client *client, hnae3_set_client_init_flag(client, ae_dev, 1); + if (netif_msg_drv(&hdev->vport->nic)) + hclge_info_show(hdev); + if (hdev->roce_client && hnae3_dev_roce_supported(hdev)) { struct hnae3_client *rc = hdev->roce_client; @@ -7660,6 +8277,8 @@ static int hclge_init_ae_dev(struct hnae3_ae_dev *ae_dev) goto err_mdiobus_unreg; } + INIT_KFIFO(hdev->mac_tnl_log); + hclge_dcb_ops_set(hdev); timer_setup(&hdev->service_timer, hclge_service_timer, 0); @@ -7708,7 +8327,7 @@ static void hclge_reset_vport_state(struct hclge_dev *hdev) int i; for (i = 0; i < hdev->num_alloc_vport; i++) { - hclge_vport_start(vport); + hclge_vport_stop(vport); vport++; } } @@ -7813,6 +8432,7 @@ static void hclge_uninit_ae_dev(struct hnae3_ae_dev *ae_dev) hclge_enable_vector(&hdev->misc_vector, false); synchronize_irq(hdev->misc_vector.vector_irq); + hclge_config_mac_tnl_int(hdev, false); hclge_hw_error_set_state(hdev, false); hclge_cmd_uninit(hdev); hclge_misc_irq_uninit(hdev); @@ -8234,9 +8854,11 @@ static const struct hnae3_ae_ops hclge_ops = { .client_stop = hclge_client_stop, .get_status = hclge_get_status, .get_ksettings_an_result = hclge_get_ksettings_an_result, - .update_speed_duplex_h = hclge_update_speed_duplex_h, .cfg_mac_speed_dup_h = hclge_cfg_mac_speed_dup_h, .get_media_type = hclge_get_media_type, + .check_port_speed = hclge_check_port_speed, + .get_fec = hclge_get_fec, + .set_fec = hclge_set_fec, .get_rss_key_size = hclge_get_rss_key_size, .get_rss_indir_size = hclge_get_rss_indir_size, .get_rss = hclge_get_rss, @@ -8253,11 +8875,13 @@ static const struct hnae3_ae_ops hclge_ops = { .rm_mc_addr = hclge_rm_mc_addr, .set_autoneg = hclge_set_autoneg, .get_autoneg = hclge_get_autoneg, + .restart_autoneg = hclge_restart_autoneg, .get_pauseparam = hclge_get_pauseparam, .set_pauseparam = hclge_set_pauseparam, .set_mtu = hclge_set_mtu, .reset_queue = hclge_reset_tqp, .get_stats = hclge_get_stats, + .get_mac_pause_stats = hclge_get_mac_pause_stat, .update_stats = hclge_update_stats, .get_strings = hclge_get_strings, .get_sset_count = hclge_get_sset_count, diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_main.h b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_main.h index b57ac4beb31310fa05600ae9cc3b223e92fab400..dd06b11187b03058290475b15208777a67182262 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_main.h +++ b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_main.h @@ -7,6 +7,7 @@ #include #include #include +#include #include "hclge_cmd.h" #include "hnae3.h" @@ -188,6 +189,8 @@ enum HLCGE_PORT_TYPE { #define HCLGE_SUPPORT_25G_BIT BIT(2) #define HCLGE_SUPPORT_50G_BIT BIT(3) #define HCLGE_SUPPORT_100G_BIT BIT(4) +/* to be compatible with exsit board */ +#define HCLGE_SUPPORT_40G_BIT BIT(5) #define HCLGE_SUPPORT_100M_BIT BIT(6) #define HCLGE_SUPPORT_10M_BIT BIT(7) #define HCLGE_SUPPORT_GE \ @@ -235,15 +238,25 @@ enum HCLGE_MAC_DUPLEX { HCLGE_MAC_FULL }; +#define QUERY_SFP_SPEED 0 +#define QUERY_ACTIVE_SPEED 1 + struct hclge_mac { u8 phy_addr; u8 flag; - u8 media_type; + u8 media_type; /* port media type, e.g. fibre/copper/backplane */ u8 mac_addr[ETH_ALEN]; u8 autoneg; u8 duplex; + u8 support_autoneg; + u8 speed_type; /* 0: sfp speed, 1: active speed */ u32 speed; - int link; /* store the link status of mac & phy (if phy exit)*/ + u32 speed_ability; /* speed ability supported by current media */ + u32 module_type; /* sub media type, e.g. kr/cr/sr/lr */ + u32 fec_mode; /* active fec mode */ + u32 user_fec_mode; + u32 fec_ability; + int link; /* store the link status of mac & phy (if phy exit) */ struct phy_device *phydev; struct mii_bus *mdio_bus; phy_interface_t phy_if; @@ -649,6 +662,23 @@ struct hclge_vport_vlan_cfg { u16 vlan_id; }; +struct hclge_rst_stats { + u32 reset_done_cnt; /* the number of reset has completed */ + u32 hw_reset_done_cnt; /* the number of HW reset has completed */ + u32 pf_rst_cnt; /* the number of PF reset */ + u32 flr_rst_cnt; /* the number of FLR */ + u32 core_rst_cnt; /* the number of CORE reset */ + u32 global_rst_cnt; /* the number of GLOBAL */ + u32 imp_rst_cnt; /* the number of IMP reset */ + u32 reset_cnt; /* the number of reset */ +}; + +/* time and register status when mac tunnel interruption occur */ +struct hclge_mac_tnl_stats { + u64 time; + u32 status; +}; + /* For each bit of TCAM entry, it uses a pair of 'x' and * 'y' to indicate which value to match, like below: * ---------------------------------- @@ -675,6 +705,7 @@ struct hclge_vport_vlan_cfg { (y) = (_k_ ^ ~_v_) & (_k_); \ } while (0) +#define HCLGE_MAC_TNL_LOG_SIZE 8 #define HCLGE_VPORT_NUM 256 struct hclge_dev { struct pci_dev *pdev; @@ -691,7 +722,7 @@ struct hclge_dev { unsigned long default_reset_request; unsigned long reset_request; /* reset has been requested */ unsigned long reset_pending; /* client rst is pending to be served */ - unsigned long reset_count; /* the number of reset has been done */ + struct hclge_rst_stats rst_stats; u32 reset_fail_cnt; u32 fw_version; u16 num_vmdq_vport; /* Num vmdq vport this PF has set up */ @@ -791,6 +822,9 @@ struct hclge_dev { struct mutex umv_mutex; /* protect share_umv_size */ struct mutex vport_cfg_mutex; /* Protect stored vf table */ + + DECLARE_KFIFO(mac_tnl_log, struct hclge_mac_tnl_stats, + HCLGE_MAC_TNL_LOG_SIZE); }; /* VPort level vlan tag configuration for TX direction */ @@ -807,10 +841,11 @@ struct hclge_tx_vtag_cfg { /* VPort level vlan tag configuration for RX direction */ struct hclge_rx_vtag_cfg { - bool strip_tag1_en; /* Whether strip inner vlan tag */ - bool strip_tag2_en; /* Whether strip outer vlan tag */ - bool vlan1_vlan_prionly;/* Inner VLAN Tag up to descriptor Enable */ - bool vlan2_vlan_prionly;/* Outer VLAN Tag up to descriptor Enable */ + u8 rx_vlan_offload_en; /* Whether enable rx vlan offload */ + u8 strip_tag1_en; /* Whether strip inner vlan tag */ + u8 strip_tag2_en; /* Whether strip outer vlan tag */ + u8 vlan1_vlan_prionly; /* Inner VLAN Tag up to descriptor Enable */ + u8 vlan2_vlan_prionly; /* Outer VLAN Tag up to descriptor Enable */ }; struct hclge_rss_tuple_cfg { @@ -829,6 +864,17 @@ enum HCLGE_VPORT_STATE { HCLGE_VPORT_STATE_MAX }; +struct hclge_vlan_info { + u16 vlan_proto; /* so far support 802.1Q only */ + u16 qos; + u16 vlan_tag; +}; + +struct hclge_port_base_vlan_config { + u16 state; + struct hclge_vlan_info vlan_info; +}; + struct hclge_vport { u16 alloc_tqps; /* Allocated Tx/Rx queues */ @@ -842,9 +888,10 @@ struct hclge_vport { u16 alloc_rss_size; u16 qs_offset; - u16 bw_limit; /* VSI BW Limit (0 = disabled) */ + u32 bw_limit; /* VSI BW Limit (0 = disabled) */ u8 dwrr; + struct hclge_port_base_vlan_config port_base_vlan_cfg; struct hclge_tx_vtag_cfg txvlan_cfg; struct hclge_rx_vtag_cfg rxvlan_cfg; @@ -924,9 +971,11 @@ void hclge_rm_vport_mac_table(struct hclge_vport *vport, const u8 *mac_addr, void hclge_rm_vport_all_mac_table(struct hclge_vport *vport, bool is_del_list, enum HCLGE_MAC_ADDR_TYPE mac_type); void hclge_uninit_vport_mac_table(struct hclge_dev *hdev); -void hclge_add_vport_vlan_table(struct hclge_vport *vport, u16 vlan_id); -void hclge_rm_vport_vlan_table(struct hclge_vport *vport, u16 vlan_id, - bool is_write_tbl); void hclge_rm_vport_all_vlan_table(struct hclge_vport *vport, bool is_del_list); void hclge_uninit_vport_vlan_table(struct hclge_dev *hdev); +int hclge_update_port_base_vlan_cfg(struct hclge_vport *vport, u16 state, + struct hclge_vlan_info *vlan_info); +int hclge_push_vf_port_base_vlan_info(struct hclge_vport *vport, u8 vfid, + u16 state, u16 vlan_tag, u16 qos, + u16 vlan_proto); #endif diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_mbx.c b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_mbx.c index 306a23e486de522c6f910d05904dd0e35f3259d5..0e04e63f2a94ca2692b014207927f46821e7ab12 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_mbx.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_mbx.c @@ -212,8 +212,7 @@ static int hclge_set_vf_promisc_mode(struct hclge_vport *vport, } static int hclge_set_vf_uc_mac_addr(struct hclge_vport *vport, - struct hclge_mbx_vf_to_pf_cmd *mbx_req, - bool gen_resp) + struct hclge_mbx_vf_to_pf_cmd *mbx_req) { const u8 *mac_addr = (const u8 *)(&mbx_req->msg[2]); struct hclge_dev *hdev = vport->back; @@ -249,7 +248,7 @@ static int hclge_set_vf_uc_mac_addr(struct hclge_vport *vport, return -EIO; } - if (gen_resp) + if (mbx_req->mbx_need_resp & HCLGE_MBX_NEED_RESP_BIT) hclge_gen_resp_to_vf(vport, mbx_req, status, NULL, 0); return 0; @@ -289,9 +288,25 @@ static int hclge_set_vf_mc_mac_addr(struct hclge_vport *vport, return 0; } +int hclge_push_vf_port_base_vlan_info(struct hclge_vport *vport, u8 vfid, + u16 state, u16 vlan_tag, u16 qos, + u16 vlan_proto) +{ +#define MSG_DATA_SIZE 8 + + u8 msg_data[MSG_DATA_SIZE]; + + memcpy(&msg_data[0], &state, sizeof(u16)); + memcpy(&msg_data[2], &vlan_proto, sizeof(u16)); + memcpy(&msg_data[4], &qos, sizeof(u16)); + memcpy(&msg_data[6], &vlan_tag, sizeof(u16)); + + return hclge_send_mbx_msg(vport, msg_data, sizeof(msg_data), + HLCGE_MBX_PUSH_VLAN_INFO, vfid); +} + static int hclge_set_vf_vlan_cfg(struct hclge_vport *vport, - struct hclge_mbx_vf_to_pf_cmd *mbx_req, - bool gen_resp) + struct hclge_mbx_vf_to_pf_cmd *mbx_req) { int status = 0; @@ -305,19 +320,27 @@ static int hclge_set_vf_vlan_cfg(struct hclge_vport *vport, memcpy(&proto, &mbx_req->msg[5], sizeof(proto)); status = hclge_set_vlan_filter(handle, cpu_to_be16(proto), vlan, is_kill); - if (!status) - is_kill ? hclge_rm_vport_vlan_table(vport, vlan, false) - : hclge_add_vport_vlan_table(vport, vlan); } else if (mbx_req->msg[1] == HCLGE_MBX_VLAN_RX_OFF_CFG) { struct hnae3_handle *handle = &vport->nic; bool en = mbx_req->msg[2] ? true : false; status = hclge_en_hw_strip_rxvtag(handle, en); + } else if (mbx_req->msg[1] == HCLGE_MBX_PORT_BASE_VLAN_CFG) { + struct hclge_vlan_info *vlan_info; + u16 *state; + + state = (u16 *)&mbx_req->msg[2]; + vlan_info = (struct hclge_vlan_info *)&mbx_req->msg[4]; + status = hclge_update_port_base_vlan_cfg(vport, *state, + vlan_info); + } else if (mbx_req->msg[1] == HCLGE_MBX_GET_PORT_BASE_VLAN_STATE) { + u8 state; + + state = vport->port_base_vlan_cfg.state; + status = hclge_gen_resp_to_vf(vport, mbx_req, 0, &state, + sizeof(u8)); } - if (gen_resp) - status = hclge_gen_resp_to_vf(vport, mbx_req, status, NULL, 0); - return status; } @@ -385,24 +408,33 @@ static int hclge_get_vf_queue_depth(struct hclge_vport *vport, HCLGE_TQPS_DEPTH_INFO_LEN); } +static int hclge_get_vf_media_type(struct hclge_vport *vport, + struct hclge_mbx_vf_to_pf_cmd *mbx_req) +{ + struct hclge_dev *hdev = vport->back; + u8 resp_data[2]; + + resp_data[0] = hdev->hw.mac.media_type; + resp_data[1] = hdev->hw.mac.module_type; + return hclge_gen_resp_to_vf(vport, mbx_req, 0, resp_data, + sizeof(resp_data)); +} + static int hclge_get_link_info(struct hclge_vport *vport, struct hclge_mbx_vf_to_pf_cmd *mbx_req) { struct hclge_dev *hdev = vport->back; u16 link_status; - u8 msg_data[10]; - u16 media_type; + u8 msg_data[8]; u8 dest_vfid; u16 duplex; /* mac.link can only be 0 or 1 */ link_status = (u16)hdev->hw.mac.link; duplex = hdev->hw.mac.duplex; - media_type = hdev->hw.mac.media_type; memcpy(&msg_data[0], &link_status, sizeof(u16)); memcpy(&msg_data[2], &hdev->hw.mac.speed, sizeof(u32)); memcpy(&msg_data[6], &duplex, sizeof(u16)); - memcpy(&msg_data[8], &media_type, sizeof(u16)); dest_vfid = mbx_req->mbx_src_vfid; /* send this requested info to VF */ @@ -565,7 +597,7 @@ void hclge_mbx_handler(struct hclge_dev *hdev) ret); break; case HCLGE_MBX_SET_UNICAST: - ret = hclge_set_vf_uc_mac_addr(vport, req, true); + ret = hclge_set_vf_uc_mac_addr(vport, req); if (ret) dev_err(&hdev->pdev->dev, "PF fail(%d) to set VF UC MAC Addr\n", @@ -579,7 +611,7 @@ void hclge_mbx_handler(struct hclge_dev *hdev) ret); break; case HCLGE_MBX_SET_VLAN: - ret = hclge_set_vf_vlan_cfg(vport, req, false); + ret = hclge_set_vf_vlan_cfg(vport, req); if (ret) dev_err(&hdev->pdev->dev, "PF failed(%d) to config VF's VLAN\n", @@ -662,6 +694,13 @@ void hclge_mbx_handler(struct hclge_dev *hdev) hclge_rm_vport_all_vlan_table(vport, true); mutex_unlock(&hdev->vport_cfg_mutex); break; + case HCLGE_MBX_GET_MEDIA_TYPE: + ret = hclge_get_vf_media_type(vport, req); + if (ret) + dev_err(&hdev->pdev->dev, + "PF fail(%d) to media type for VF\n", + ret); + break; default: dev_err(&hdev->pdev->dev, "un-supported mailbox message, code = %d\n", diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_mdio.c b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_mdio.c index 48eda2c6fdae3e49eacb9094935c487743814418..1e8134892d775cca14e9ebe00c62d93940e4ff06 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_mdio.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_mdio.c @@ -3,6 +3,7 @@ #include #include +#include #include "hclge_cmd.h" #include "hclge_main.h" @@ -121,12 +122,18 @@ static int hclge_mdio_read(struct mii_bus *bus, int phyid, int regnum) int hclge_mac_mdio_config(struct hclge_dev *hdev) { +#define PHY_INEXISTENT 255 + struct hclge_mac *mac = &hdev->hw.mac; struct phy_device *phydev; struct mii_bus *mdio_bus; int ret; - if (hdev->hw.mac.phy_addr >= PHY_MAX_ADDR) { + if (hdev->hw.mac.phy_addr == PHY_INEXISTENT) { + dev_info(&hdev->pdev->dev, + "no phy device is connected to mdio bus\n"); + return 0; + } else if (hdev->hw.mac.phy_addr >= PHY_MAX_ADDR) { dev_err(&hdev->pdev->dev, "phy_addr(%d) is too large.\n", hdev->hw.mac.phy_addr); return -EINVAL; @@ -203,6 +210,8 @@ int hclge_mac_connect_phy(struct hnae3_handle *handle) linkmode_clear_bit(ETHTOOL_LINK_MODE_FIBRE_BIT, phydev->supported); + phydev->dev_flags |= MARVELL_PHY_LED0_LINK_LED1_ACTIVE; + ret = phy_connect_direct(netdev, phydev, hclge_mac_adjust_link, PHY_INTERFACE_MODE_SGMII); diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_tm.c b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_tm.c index aafc69f4bfdd6f5d2afce29a7b3affb80b2f036c..a7bbb6d3091a62b185ea28c8fa39a6da8f0a7072 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_tm.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_tm.c @@ -1331,8 +1331,11 @@ int hclge_pause_setup_hw(struct hclge_dev *hdev, bool init) ret = hclge_pfc_setup_hw(hdev); if (init && ret == -EOPNOTSUPP) dev_warn(&hdev->pdev->dev, "GE MAC does not support pfc\n"); - else + else if (ret) { + dev_err(&hdev->pdev->dev, "config pfc failed! ret = %d\n", + ret); return ret; + } return hclge_tm_bp_setup(hdev); } diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_cmd.c b/drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_cmd.c index 9441b453d38dfa9ac382e3c9f0e330aed09ab445..71f356fc24462415cb9770efa1388bf1f2fd4fca 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_cmd.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_cmd.c @@ -27,26 +27,39 @@ static int hclgevf_ring_space(struct hclgevf_cmq_ring *ring) return ring->desc_num - used - 1; } +static int hclgevf_is_valid_csq_clean_head(struct hclgevf_cmq_ring *ring, + int head) +{ + int ntu = ring->next_to_use; + int ntc = ring->next_to_clean; + + if (ntu > ntc) + return head >= ntc && head <= ntu; + + return head >= ntc || head <= ntu; +} + static int hclgevf_cmd_csq_clean(struct hclgevf_hw *hw) { + struct hclgevf_dev *hdev = container_of(hw, struct hclgevf_dev, hw); struct hclgevf_cmq_ring *csq = &hw->cmq.csq; - u16 ntc = csq->next_to_clean; - struct hclgevf_desc *desc; int clean = 0; u32 head; - desc = &csq->desc[ntc]; head = hclgevf_read_dev(hw, HCLGEVF_NIC_CSQ_HEAD_REG); - while (head != ntc) { - memset(desc, 0, sizeof(*desc)); - ntc++; - if (ntc == csq->desc_num) - ntc = 0; - desc = &csq->desc[ntc]; - clean++; + rmb(); /* Make sure head is ready before touch any data */ + + if (!hclgevf_is_valid_csq_clean_head(csq, head)) { + dev_warn(&hdev->pdev->dev, "wrong cmd head (%d, %d-%d)\n", head, + csq->next_to_use, csq->next_to_clean); + dev_warn(&hdev->pdev->dev, + "Disabling any further commands to IMP firmware\n"); + set_bit(HCLGEVF_STATE_CMD_DISABLE, &hdev->state); + return -EIO; } - csq->next_to_clean = ntc; + clean = (head - csq->next_to_clean + csq->desc_num) % csq->desc_num; + csq->next_to_clean = head; return clean; } @@ -321,13 +334,13 @@ int hclgevf_cmd_init(struct hclgevf_dev *hdev) int ret; spin_lock_bh(&hdev->hw.cmq.csq.lock); - spin_lock_bh(&hdev->hw.cmq.crq.lock); + spin_lock(&hdev->hw.cmq.crq.lock); /* initialize the pointers of async rx queue of mailbox */ hdev->arq.hdev = hdev; hdev->arq.head = 0; hdev->arq.tail = 0; - hdev->arq.count = 0; + atomic_set(&hdev->arq.count, 0); hdev->hw.cmq.csq.next_to_clean = 0; hdev->hw.cmq.csq.next_to_use = 0; hdev->hw.cmq.crq.next_to_clean = 0; @@ -335,7 +348,7 @@ int hclgevf_cmd_init(struct hclgevf_dev *hdev) hclgevf_cmd_init_regs(&hdev->hw); - spin_unlock_bh(&hdev->hw.cmq.crq.lock); + spin_unlock(&hdev->hw.cmq.crq.lock); spin_unlock_bh(&hdev->hw.cmq.csq.lock); clear_bit(HCLGEVF_STATE_CMD_DISABLE, &hdev->state); @@ -344,8 +357,8 @@ int hclgevf_cmd_init(struct hclgevf_dev *hdev) * reset may happen when lower level reset is being processed. */ if (hclgevf_is_reset_pending(hdev)) { - set_bit(HCLGEVF_STATE_CMD_DISABLE, &hdev->state); - return -EBUSY; + ret = -EBUSY; + goto err_cmd_init; } /* get firmware version */ @@ -353,13 +366,18 @@ int hclgevf_cmd_init(struct hclgevf_dev *hdev) if (ret) { dev_err(&hdev->pdev->dev, "failed(%d) to query firmware version\n", ret); - return ret; + goto err_cmd_init; } hdev->fw_version = version; dev_info(&hdev->pdev->dev, "The firmware version is %08x\n", version); return 0; + +err_cmd_init: + set_bit(HCLGEVF_STATE_CMD_DISABLE, &hdev->state); + + return ret; } static void hclgevf_cmd_uninit_regs(struct hclgevf_hw *hw) diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_main.c b/drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_main.c index 8bc28e6f465f1961fef7e7d070bee547dce6b9af..5d53467ee2d211f30af3cc0cafbe5b47cb07190a 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_main.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_main.c @@ -245,6 +245,27 @@ static int hclgevf_get_tc_info(struct hclgevf_dev *hdev) return 0; } +static int hclgevf_get_port_base_vlan_filter_state(struct hclgevf_dev *hdev) +{ + struct hnae3_handle *nic = &hdev->nic; + u8 resp_msg; + int ret; + + ret = hclgevf_send_mbx_msg(hdev, HCLGE_MBX_SET_VLAN, + HCLGE_MBX_GET_PORT_BASE_VLAN_STATE, + NULL, 0, true, &resp_msg, sizeof(u8)); + if (ret) { + dev_err(&hdev->pdev->dev, + "VF request to get port based vlan state failed %d", + ret); + return ret; + } + + nic->port_base_vlan_state = resp_msg; + + return 0; +} + static int hclgevf_get_queue_info(struct hclgevf_dev *hdev) { #define HCLGEVF_TQPS_RSS_INFO_LEN 6 @@ -307,6 +328,26 @@ static u16 hclgevf_get_qid_global(struct hnae3_handle *handle, u16 queue_id) return qid_in_pf; } +static int hclgevf_get_pf_media_type(struct hclgevf_dev *hdev) +{ + u8 resp_msg[2]; + int ret; + + ret = hclgevf_send_mbx_msg(hdev, HCLGE_MBX_GET_MEDIA_TYPE, 0, NULL, 0, + true, resp_msg, sizeof(resp_msg)); + if (ret) { + dev_err(&hdev->pdev->dev, + "VF request to get the pf port media type failed %d", + ret); + return ret; + } + + hdev->hw.mac.media_type = resp_msg[0]; + hdev->hw.mac.module_type = resp_msg[1]; + + return 0; +} + static int hclgevf_alloc_tqps(struct hclgevf_dev *hdev) { struct hclgevf_tqp *tqp; @@ -404,7 +445,7 @@ void hclgevf_update_link_status(struct hclgevf_dev *hdev, int link_state) } } -void hclgevf_update_link_mode(struct hclgevf_dev *hdev) +static void hclgevf_update_link_mode(struct hclgevf_dev *hdev) { #define HCLGEVF_ADVERTISING 0 #define HCLGEVF_SUPPORTED 1 @@ -1375,9 +1416,11 @@ static int hclgevf_reset_prepare_wait(struct hclgevf_dev *hdev) case HNAE3_VF_FUNC_RESET: ret = hclgevf_send_mbx_msg(hdev, HCLGE_MBX_RESET, 0, NULL, 0, true, NULL, sizeof(u8)); + hdev->rst_stats.vf_func_rst_cnt++; break; case HNAE3_FLR_RESET: set_bit(HNAE3_FLR_DOWN, &hdev->flr_state); + hdev->rst_stats.flr_rst_cnt++; break; default: break; @@ -1400,7 +1443,7 @@ static int hclgevf_reset(struct hclgevf_dev *hdev) * know if device is undergoing reset */ ae_dev->reset_type = hdev->reset_type; - hdev->reset_count++; + hdev->rst_stats.rst_cnt++; rtnl_lock(); /* bring down the nic to stop any ongoing TX/RX */ @@ -1426,6 +1469,8 @@ static int hclgevf_reset(struct hclgevf_dev *hdev) goto err_reset; } + hdev->rst_stats.hw_rst_done_cnt++; + rtnl_lock(); /* now, re-initialize the nic client and ae device*/ @@ -1444,6 +1489,7 @@ static int hclgevf_reset(struct hclgevf_dev *hdev) hdev->last_reset_time = jiffies; ae_dev->reset_type = HNAE3_NONE_RESET; + hdev->rst_stats.rst_done_cnt++; return ret; err_reset_lock: @@ -1455,6 +1501,8 @@ err_reset: */ hclgevf_cmd_init(hdev); dev_err(&hdev->pdev->dev, "failed to reset VF\n"); + if (hclgevf_is_reset_pending(hdev)) + hclgevf_reset_task_schedule(hdev); return ret; } @@ -1564,8 +1612,7 @@ static void hclgevf_get_misc_vector(struct hclgevf_dev *hdev) void hclgevf_reset_task_schedule(struct hclgevf_dev *hdev) { - if (!test_bit(HCLGEVF_STATE_RST_SERVICE_SCHED, &hdev->state) && - !test_bit(HCLGEVF_STATE_RST_HANDLING, &hdev->state)) { + if (!test_bit(HCLGEVF_STATE_RST_SERVICE_SCHED, &hdev->state)) { set_bit(HCLGEVF_STATE_RST_SERVICE_SCHED, &hdev->state); schedule_work(&hdev->rst_service_task); } @@ -1603,6 +1650,7 @@ static void hclgevf_service_timer(struct timer_list *t) mod_timer(&hdev->service_timer, jiffies + 5 * HZ); + hdev->stats_timer++; hclgevf_task_schedule(hdev); } @@ -1711,7 +1759,7 @@ static void hclgevf_keep_alive_task(struct work_struct *work) hdev = container_of(work, struct hclgevf_dev, keep_alive_task); - if (test_bit(HCLGEVF_STATE_RST_HANDLING, &hdev->state)) + if (test_bit(HCLGEVF_STATE_CMD_DISABLE, &hdev->state)) return; ret = hclgevf_send_mbx_msg(hdev, HCLGE_MBX_KEEP_ALIVE, 0, NULL, @@ -1723,9 +1771,16 @@ static void hclgevf_keep_alive_task(struct work_struct *work) static void hclgevf_service_task(struct work_struct *work) { + struct hnae3_handle *handle; struct hclgevf_dev *hdev; hdev = container_of(work, struct hclgevf_dev, service_task); + handle = &hdev->nic; + + if (hdev->stats_timer >= HCLGEVF_STATS_TIMER_INTERVAL) { + hclgevf_tqps_update_stats(handle); + hdev->stats_timer = 0; + } /* request the link status from the PF. PF would be able to tell VF * about such updates in future so we might remove this later @@ -1762,6 +1817,7 @@ static enum hclgevf_evt_cause hclgevf_check_evt_cause(struct hclgevf_dev *hdev, set_bit(HCLGEVF_STATE_CMD_DISABLE, &hdev->state); cmdq_src_reg &= ~BIT(HCLGEVF_VECTOR0_RST_INT_B); *clearval = cmdq_src_reg; + hdev->rst_stats.vf_rst_cnt++; return HCLGEVF_VECTOR0_EVENT_RST; } @@ -1814,6 +1870,11 @@ static int hclgevf_configure(struct hclgevf_dev *hdev) { int ret; + /* get current port based vlan state from PF */ + ret = hclgevf_get_port_base_vlan_filter_state(hdev); + if (ret) + return ret; + /* get queue configuration from PF */ ret = hclgevf_get_queue_info(hdev); if (ret) @@ -1824,6 +1885,10 @@ static int hclgevf_configure(struct hclgevf_dev *hdev) if (ret) return ret; + ret = hclgevf_get_pf_media_type(hdev); + if (ret) + return ret; + /* get tc configuration from PF */ return hclgevf_get_tc_info(hdev); } @@ -1986,8 +2051,10 @@ static void hclgevf_ae_stop(struct hnae3_handle *handle) set_bit(HCLGEVF_STATE_DOWN, &hdev->state); - for (i = 0; i < handle->kinfo.num_tqps; i++) - hclgevf_reset_tqp(handle, i); + if (hdev->reset_type != HNAE3_VF_RESET) + for (i = 0; i < handle->kinfo.num_tqps; i++) + if (hclgevf_reset_tqp(handle, i)) + break; /* reset tqp stats */ hclgevf_reset_tqp_stats(handle); @@ -2007,9 +2074,15 @@ static int hclgevf_set_alive(struct hnae3_handle *handle, bool alive) static int hclgevf_client_start(struct hnae3_handle *handle) { struct hclgevf_dev *hdev = hclgevf_ae_get_hdev(handle); + int ret; + + ret = hclgevf_set_alive(handle, true); + if (ret) + return ret; mod_timer(&hdev->keep_alive_timer, jiffies + 2 * HZ); - return hclgevf_set_alive(handle, true); + + return 0; } static void hclgevf_client_stop(struct hnae3_handle *handle) @@ -2051,6 +2124,10 @@ static void hclgevf_state_uninit(struct hclgevf_dev *hdev) { set_bit(HCLGEVF_STATE_DOWN, &hdev->state); + if (hdev->keep_alive_timer.function) + del_timer_sync(&hdev->keep_alive_timer); + if (hdev->keep_alive_task.func) + cancel_work_sync(&hdev->keep_alive_task); if (hdev->service_timer.function) del_timer_sync(&hdev->service_timer); if (hdev->service_task.func) @@ -2155,6 +2232,23 @@ static void hclgevf_misc_irq_uninit(struct hclgevf_dev *hdev) hclgevf_free_vector(hdev, 0); } +static void hclgevf_info_show(struct hclgevf_dev *hdev) +{ + struct device *dev = &hdev->pdev->dev; + + dev_info(dev, "VF info begin:\n"); + + dev_info(dev, "Task queue pairs numbers: %d\n", hdev->num_tqps); + dev_info(dev, "Desc num per TX queue: %d\n", hdev->num_tx_desc); + dev_info(dev, "Desc num per RX queue: %d\n", hdev->num_rx_desc); + dev_info(dev, "Numbers of vports: %d\n", hdev->num_alloc_vport); + dev_info(dev, "HW tc map: %d\n", hdev->hw_tc_map); + dev_info(dev, "PF media type of this VF: %d\n", + hdev->hw.mac.media_type); + + dev_info(dev, "VF info end.\n"); +} + static int hclgevf_init_client_instance(struct hnae3_client *client, struct hnae3_ae_dev *ae_dev) { @@ -2172,6 +2266,9 @@ static int hclgevf_init_client_instance(struct hnae3_client *client, hnae3_set_client_init_flag(client, ae_dev, 1); + if (netif_msg_drv(&hdev->nic)) + hclgevf_info_show(hdev); + if (hdev->roce_client && hnae3_dev_roce_supported(hdev)) { struct hnae3_client *rc = hdev->roce_client; @@ -2651,12 +2748,16 @@ static int hclgevf_gro_en(struct hnae3_handle *handle, bool enable) return hclgevf_config_gro(hdev, enable); } -static void hclgevf_get_media_type(struct hnae3_handle *handle, - u8 *media_type) +static void hclgevf_get_media_type(struct hnae3_handle *handle, u8 *media_type, + u8 *module_type) { struct hclgevf_dev *hdev = hclgevf_ae_get_hdev(handle); + if (media_type) *media_type = hdev->hw.mac.media_type; + + if (module_type) + *module_type = hdev->hw.mac.module_type; } static bool hclgevf_get_hw_reset_stat(struct hnae3_handle *handle) @@ -2677,7 +2778,7 @@ static unsigned long hclgevf_ae_dev_reset_cnt(struct hnae3_handle *handle) { struct hclgevf_dev *hdev = hclgevf_ae_get_hdev(handle); - return hdev->reset_count; + return hdev->rst_stats.hw_rst_done_cnt; } static void hclgevf_get_link_mode(struct hnae3_handle *handle, @@ -2756,6 +2857,31 @@ static void hclgevf_get_regs(struct hnae3_handle *handle, u32 *version, } } +void hclgevf_update_port_base_vlan_info(struct hclgevf_dev *hdev, u16 state, + u8 *port_base_vlan_info, u8 data_size) +{ + struct hnae3_handle *nic = &hdev->nic; + + rtnl_lock(); + hclgevf_notify_client(hdev, HNAE3_DOWN_CLIENT); + rtnl_unlock(); + + /* send msg to PF and wait update port based vlan info */ + hclgevf_send_mbx_msg(hdev, HCLGE_MBX_SET_VLAN, + HCLGE_MBX_PORT_BASE_VLAN_CFG, + port_base_vlan_info, data_size, + false, NULL, 0); + + if (state == HNAE3_PORT_BASE_VLAN_DISABLE) + nic->port_base_vlan_state = HNAE3_PORT_BASE_VLAN_DISABLE; + else + nic->port_base_vlan_state = HNAE3_PORT_BASE_VLAN_ENABLE; + + rtnl_lock(); + hclgevf_notify_client(hdev, HNAE3_UP_CLIENT); + rtnl_unlock(); +} + static const struct hnae3_ae_ops hclgevf_ops = { .init_ae_dev = hclgevf_init_ae_dev, .uninit_ae_dev = hclgevf_uninit_ae_dev, diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_main.h b/drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_main.h index c128863ee7d07f5bd7922a286e9d24a6c5fac6ed..cc52f54f8c088ecdae89c45335b99851e1b42f32 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_main.h +++ b/drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_main.h @@ -116,6 +116,8 @@ #define HCLGEVF_S_IP_BIT BIT(3) #define HCLGEVF_V_TAG_BIT BIT(4) +#define HCLGEVF_STATS_TIMER_INTERVAL (36) + enum hclgevf_evt_cause { HCLGEVF_VECTOR0_EVENT_RST, HCLGEVF_VECTOR0_EVENT_MBX, @@ -141,6 +143,7 @@ enum hclgevf_states { struct hclgevf_mac { u8 media_type; + u8 module_type; u8 mac_addr[ETH_ALEN]; int link; u8 duplex; @@ -210,6 +213,15 @@ struct hclgevf_misc_vector { int vector_irq; }; +struct hclgevf_rst_stats { + u32 rst_cnt; /* the number of reset */ + u32 vf_func_rst_cnt; /* the number of VF function reset */ + u32 flr_rst_cnt; /* the number of FLR */ + u32 vf_rst_cnt; /* the number of VF reset */ + u32 rst_done_cnt; /* the number of reset completed */ + u32 hw_rst_done_cnt; /* the number of HW reset completed */ +}; + struct hclgevf_dev { struct pci_dev *pdev; struct hnae3_ae_dev *ae_dev; @@ -227,7 +239,7 @@ struct hclgevf_dev { #define HCLGEVF_RESET_REQUESTED 0 #define HCLGEVF_RESET_PENDING 1 unsigned long reset_state; /* requested, pending */ - unsigned long reset_count; /* the number of reset has been done */ + struct hclgevf_rst_stats rst_stats; u32 reset_attempts; u32 fw_version; @@ -272,6 +284,7 @@ struct hclgevf_dev { struct hnae3_client *nic_client; struct hnae3_client *roce_client; u32 flag; + u32 stats_timer; }; static inline bool hclgevf_is_reset_pending(struct hclgevf_dev *hdev) @@ -290,4 +303,6 @@ void hclgevf_update_speed_duplex(struct hclgevf_dev *hdev, u32 speed, u8 duplex); void hclgevf_reset_task_schedule(struct hclgevf_dev *hdev); void hclgevf_mbx_task_schedule(struct hclgevf_dev *hdev); +void hclgevf_update_port_base_vlan_info(struct hclgevf_dev *hdev, u16 state, + u8 *port_base_vlan_info, u8 data_size); #endif diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_mbx.c b/drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_mbx.c index 7dc3c9f79169f119e53fb44c8ed929611a217878..30f2e9352cf3373cb8bd409aa2a7e1e730bf7472 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_mbx.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_mbx.c @@ -49,8 +49,8 @@ static int hclgevf_get_mbx_resp(struct hclgevf_dev *hdev, u16 code0, u16 code1, if (i >= HCLGEVF_MAX_TRY_TIMES) { dev_err(&hdev->pdev->dev, - "VF could not get mbx resp(=%d) from PF in %d tries\n", - hdev->mbx_resp.received_resp, i); + "VF could not get mbx(%d,%d) resp(=%d) from PF in %d tries\n", + code0, code1, hdev->mbx_resp.received_resp, i); return -EIO; } @@ -68,8 +68,11 @@ static int hclgevf_get_mbx_resp(struct hclgevf_dev *hdev, u16 code0, u16 code1, if (!(r_code0 == code0 && r_code1 == code1 && !mbx_resp->resp_status)) { dev_err(&hdev->pdev->dev, - "VF could not match resp code(code0=%d,code1=%d), %d", + "VF could not match resp code(code0=%d,code1=%d), %d\n", code0, code1, mbx_resp->resp_status); + dev_err(&hdev->pdev->dev, + "VF could not match resp r_code(r_code0=%d,r_code1=%d)\n", + r_code0, r_code1); return -EIO; } @@ -95,6 +98,8 @@ int hclgevf_send_mbx_msg(struct hclgevf_dev *hdev, u16 code, u16 subcode, } hclgevf_cmd_setup_basic_desc(&desc, HCLGEVF_OPC_MBX_VF_TO_PF, false); + req->mbx_need_resp |= need_resp ? HCLGE_MBX_NEED_RESP_BIT : + ~HCLGE_MBX_NEED_RESP_BIT; req->msg[0] = code; req->msg[1] = subcode; memcpy(&req->msg[2], msg_data, msg_len); @@ -198,6 +203,7 @@ void hclgevf_mbx_handler(struct hclgevf_dev *hdev) case HCLGE_MBX_LINK_STAT_CHANGE: case HCLGE_MBX_ASSERTING_RESET: case HCLGE_MBX_LINK_STAT_MODE: + case HLCGE_MBX_PUSH_VLAN_INFO: /* set this mbx event as pending. This is required as we * might loose interrupt event when mbx task is busy * handling. This shall be cleared when mbx task just @@ -208,7 +214,8 @@ void hclgevf_mbx_handler(struct hclgevf_dev *hdev) /* we will drop the async msg if we find ARQ as full * and continue with next message */ - if (hdev->arq.count >= HCLGE_MBX_MAX_ARQ_MSG_NUM) { + if (atomic_read(&hdev->arq.count) >= + HCLGE_MBX_MAX_ARQ_MSG_NUM) { dev_warn(&hdev->pdev->dev, "Async Q full, dropping msg(%d)\n", req->msg[1]); @@ -220,7 +227,7 @@ void hclgevf_mbx_handler(struct hclgevf_dev *hdev) memcpy(&msg_q[0], req->msg, HCLGE_MBX_MAX_ARQ_MSG_SIZE * sizeof(u16)); hclge_mbx_tail_ptr_move_arq(hdev->arq); - hdev->arq.count++; + atomic_inc(&hdev->arq.count); hclgevf_mbx_task_schedule(hdev); @@ -243,8 +250,8 @@ void hclgevf_mbx_handler(struct hclgevf_dev *hdev) void hclgevf_mbx_async_handler(struct hclgevf_dev *hdev) { enum hnae3_reset_type reset_type; - u16 link_status; - u16 *msg_q; + u16 link_status, state; + u16 *msg_q, *vlan_info; u8 duplex; u32 speed; u32 tail; @@ -272,7 +279,6 @@ void hclgevf_mbx_async_handler(struct hclgevf_dev *hdev) link_status = le16_to_cpu(msg_q[1]); memcpy(&speed, &msg_q[2], sizeof(speed)); duplex = (u8)le16_to_cpu(msg_q[4]); - hdev->hw.mac.media_type = (u8)le16_to_cpu(msg_q[5]); /* update upper layer with new link link status */ hclgevf_update_link_status(hdev, link_status); @@ -300,6 +306,12 @@ void hclgevf_mbx_async_handler(struct hclgevf_dev *hdev) hclgevf_reset_task_schedule(hdev); break; + case HLCGE_MBX_PUSH_VLAN_INFO: + state = le16_to_cpu(msg_q[1]); + vlan_info = &msg_q[1]; + hclgevf_update_port_base_vlan_info(hdev, state, + (u8 *)vlan_info, 8); + break; default: dev_err(&hdev->pdev->dev, "fetched unsupported(%d) message from arq\n", @@ -308,7 +320,7 @@ void hclgevf_mbx_async_handler(struct hclgevf_dev *hdev) } hclge_mbx_head_ptr_move_arq(hdev->arq); - hdev->arq.count--; + atomic_dec(&hdev->arq.count); msg_q = hdev->arq.msg_q[hdev->arq.head]; } } diff --git a/drivers/net/ethernet/huawei/hinic/hinic_tx.c b/drivers/net/ethernet/huawei/hinic/hinic_tx.c index e17bf33eba0c5e09f7708dc52566b342f4777dc6..0fbe8046824b2106eed2bbf62e54b477c97de7e7 100644 --- a/drivers/net/ethernet/huawei/hinic/hinic_tx.c +++ b/drivers/net/ethernet/huawei/hinic/hinic_tx.c @@ -518,7 +518,7 @@ process_sq_wqe: flush_skbs: netdev_txq = netdev_get_tx_queue(netdev, q_id); - if ((!skb->xmit_more) || (netif_xmit_stopped(netdev_txq))) + if ((!netdev_xmit_more()) || (netif_xmit_stopped(netdev_txq))) hinic_sq_write_db(txq->sq, prod_idx, wqe_size, 0); return err; diff --git a/drivers/net/ethernet/ibm/ehea/ehea_main.c b/drivers/net/ethernet/ibm/ehea/ehea_main.c index 90b62c1412c8f4715eaf1ab3ca14a9128f1f9046..707c8ba120c25d033af9768fbb76da12954abdbd 100644 --- a/drivers/net/ethernet/ibm/ehea/ehea_main.c +++ b/drivers/net/ethernet/ibm/ehea/ehea_main.c @@ -1463,7 +1463,7 @@ static int ehea_init_port_res(struct ehea_port *port, struct ehea_port_res *pr, memset(pr, 0, sizeof(struct ehea_port_res)); - pr->tx_bytes = rx_bytes; + pr->tx_bytes = tx_bytes; pr->tx_packets = tx_packets; pr->rx_bytes = rx_bytes; pr->rx_packets = rx_packets; diff --git a/drivers/net/ethernet/ibm/ehea/ehea_qmr.c b/drivers/net/ethernet/ibm/ehea/ehea_qmr.c index 5e4e37132bf257fda81a58715cf781748e767853..77ce17383aba8179ba9f8bffaccc72e7bd7e7468 100644 --- a/drivers/net/ethernet/ibm/ehea/ehea_qmr.c +++ b/drivers/net/ethernet/ibm/ehea/ehea_qmr.c @@ -123,8 +123,7 @@ struct ehea_cq *ehea_create_cq(struct ehea_adapter *adapter, int nr_of_cqe, u64 eq_handle, u32 cq_token) { struct ehea_cq *cq; - struct h_epa epa; - u64 *cq_handle_ref, hret, rpage; + u64 hret, rpage; u32 counter; int ret; void *vpage; @@ -139,8 +138,6 @@ struct ehea_cq *ehea_create_cq(struct ehea_adapter *adapter, cq->adapter = adapter; - cq_handle_ref = &cq->fw_handle; - hret = ehea_h_alloc_resource_cq(adapter->handle, &cq->attr, &cq->fw_handle, &cq->epas); if (hret != H_SUCCESS) { @@ -188,7 +185,6 @@ struct ehea_cq *ehea_create_cq(struct ehea_adapter *adapter, } hw_qeit_reset(&cq->hw_queue); - epa = cq->epas.kernel; ehea_reset_cq_ep(cq); ehea_reset_cq_n1(cq); diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c index dd71d5db727471d0047608a39d12dc78f3a8848a..d86b0e5895a66893aaa88390b6827a82b407fe38 100644 --- a/drivers/net/ethernet/ibm/ibmveth.c +++ b/drivers/net/ethernet/ibm/ibmveth.c @@ -93,7 +93,7 @@ struct ibmveth_stat { #define IBMVETH_STAT_OFF(stat) offsetof(struct ibmveth_adapter, stat) #define IBMVETH_GET_STAT(a, off) *((u64 *)(((unsigned long)(a)) + off)) -struct ibmveth_stat ibmveth_stats[] = { +static struct ibmveth_stat ibmveth_stats[] = { { "replenish_task_cycles", IBMVETH_STAT_OFF(replenish_task_cycles) }, { "replenish_no_mem", IBMVETH_STAT_OFF(replenish_no_mem) }, { "replenish_add_buff_failure", diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c index 3dfb2d131eb76f29c6129e8f29f477ab7be840db..b398d6c94dbdef34ce94e4de309ed34cd65f20cb 100644 --- a/drivers/net/ethernet/ibm/ibmvnic.c +++ b/drivers/net/ethernet/ibm/ibmvnic.c @@ -120,6 +120,7 @@ static int ibmvnic_reset_init(struct ibmvnic_adapter *); static void release_crq_queue(struct ibmvnic_adapter *); static int __ibmvnic_set_mac(struct net_device *netdev, struct sockaddr *p); static int init_crq_queue(struct ibmvnic_adapter *adapter); +static int send_query_phys_parms(struct ibmvnic_adapter *adapter); struct ibmvnic_stat { char name[ETH_GSTRING_LEN]; @@ -1968,13 +1969,11 @@ static void __ibmvnic_reset(struct work_struct *work) { struct ibmvnic_rwi *rwi; struct ibmvnic_adapter *adapter; - struct net_device *netdev; bool we_lock_rtnl = false; u32 reset_state; int rc = 0; adapter = container_of(work, struct ibmvnic_adapter, ibmvnic_reset); - netdev = adapter->netdev; /* netif_set_real_num_xx_queues needs to take rtnl lock here * unless wait_for_reset is set, in which case the rtnl lock @@ -2279,23 +2278,20 @@ static const struct net_device_ops ibmvnic_netdev_ops = { static int ibmvnic_get_link_ksettings(struct net_device *netdev, struct ethtool_link_ksettings *cmd) { - u32 supported, advertising; + struct ibmvnic_adapter *adapter = netdev_priv(netdev); + int rc; - supported = (SUPPORTED_1000baseT_Full | SUPPORTED_Autoneg | - SUPPORTED_FIBRE); - advertising = (ADVERTISED_1000baseT_Full | ADVERTISED_Autoneg | - ADVERTISED_FIBRE); - cmd->base.speed = SPEED_1000; - cmd->base.duplex = DUPLEX_FULL; + rc = send_query_phys_parms(adapter); + if (rc) { + adapter->speed = SPEED_UNKNOWN; + adapter->duplex = DUPLEX_UNKNOWN; + } + cmd->base.speed = adapter->speed; + cmd->base.duplex = adapter->duplex; cmd->base.port = PORT_FIBRE; cmd->base.phy_address = 0; cmd->base.autoneg = AUTONEG_ENABLE; - ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.supported, - supported); - ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.advertising, - advertising); - return 0; } @@ -2923,8 +2919,10 @@ static int init_sub_crq_irqs(struct ibmvnic_adapter *adapter) goto req_tx_irq_failed; } + snprintf(scrq->name, sizeof(scrq->name), "ibmvnic-%x-tx%d", + adapter->vdev->unit_address, i); rc = request_irq(scrq->irq, ibmvnic_interrupt_tx, - 0, "ibmvnic_tx", scrq); + 0, scrq->name, scrq); if (rc) { dev_err(dev, "Couldn't register tx irq 0x%x. rc=%d\n", @@ -2944,8 +2942,10 @@ static int init_sub_crq_irqs(struct ibmvnic_adapter *adapter) dev_err(dev, "Error mapping irq\n"); goto req_rx_irq_failed; } + snprintf(scrq->name, sizeof(scrq->name), "ibmvnic-%x-rx%d", + adapter->vdev->unit_address, i); rc = request_irq(scrq->irq, ibmvnic_interrupt_rx, - 0, "ibmvnic_rx", scrq); + 0, scrq->name, scrq); if (rc) { dev_err(dev, "Couldn't register rx irq 0x%x. rc=%d\n", scrq->irq, rc); @@ -4297,6 +4297,73 @@ out: } } +static int send_query_phys_parms(struct ibmvnic_adapter *adapter) +{ + union ibmvnic_crq crq; + int rc; + + memset(&crq, 0, sizeof(crq)); + crq.query_phys_parms.first = IBMVNIC_CRQ_CMD; + crq.query_phys_parms.cmd = QUERY_PHYS_PARMS; + init_completion(&adapter->fw_done); + rc = ibmvnic_send_crq(adapter, &crq); + if (rc) + return rc; + wait_for_completion(&adapter->fw_done); + return adapter->fw_done_rc ? -EIO : 0; +} + +static int handle_query_phys_parms_rsp(union ibmvnic_crq *crq, + struct ibmvnic_adapter *adapter) +{ + struct net_device *netdev = adapter->netdev; + int rc; + + rc = crq->query_phys_parms_rsp.rc.code; + if (rc) { + netdev_err(netdev, "Error %d in QUERY_PHYS_PARMS\n", rc); + return rc; + } + switch (cpu_to_be32(crq->query_phys_parms_rsp.speed)) { + case IBMVNIC_10MBPS: + adapter->speed = SPEED_10; + break; + case IBMVNIC_100MBPS: + adapter->speed = SPEED_100; + break; + case IBMVNIC_1GBPS: + adapter->speed = SPEED_1000; + break; + case IBMVNIC_10GBP: + adapter->speed = SPEED_10000; + break; + case IBMVNIC_25GBPS: + adapter->speed = SPEED_25000; + break; + case IBMVNIC_40GBPS: + adapter->speed = SPEED_40000; + break; + case IBMVNIC_50GBPS: + adapter->speed = SPEED_50000; + break; + case IBMVNIC_100GBPS: + adapter->speed = SPEED_100000; + break; + default: + netdev_warn(netdev, "Unknown speed 0x%08x\n", + cpu_to_be32(crq->query_phys_parms_rsp.speed)); + adapter->speed = SPEED_UNKNOWN; + } + if (crq->query_phys_parms_rsp.flags1 & IBMVNIC_FULL_DUPLEX) + adapter->duplex = DUPLEX_FULL; + else if (crq->query_phys_parms_rsp.flags1 & IBMVNIC_HALF_DUPLEX) + adapter->duplex = DUPLEX_HALF; + else + adapter->duplex = DUPLEX_UNKNOWN; + + return rc; +} + static void ibmvnic_handle_crq(union ibmvnic_crq *crq, struct ibmvnic_adapter *adapter) { @@ -4445,6 +4512,10 @@ static void ibmvnic_handle_crq(union ibmvnic_crq *crq, case GET_VPD_RSP: handle_vpd_rsp(crq, adapter); break; + case QUERY_PHYS_PARMS_RSP: + adapter->fw_done_rc = handle_query_phys_parms_rsp(crq, adapter); + complete(&adapter->fw_done); + break; default: netdev_err(netdev, "Got an invalid cmd type 0x%02x\n", gen_crq->cmd); @@ -4600,8 +4671,9 @@ static int init_crq_queue(struct ibmvnic_adapter *adapter) (unsigned long)adapter); netdev_dbg(adapter->netdev, "registering irq 0x%x\n", vdev->irq); - rc = request_irq(vdev->irq, ibmvnic_interrupt, 0, IBMVNIC_NAME, - adapter); + snprintf(crq->name, sizeof(crq->name), "ibmvnic-%x", + adapter->vdev->unit_address); + rc = request_irq(vdev->irq, ibmvnic_interrupt, 0, crq->name, adapter); if (rc) { dev_err(dev, "Couldn't register irq 0x%x. rc=%d\n", vdev->irq, rc); diff --git a/drivers/net/ethernet/ibm/ibmvnic.h b/drivers/net/ethernet/ibm/ibmvnic.h index f2018dbebfa527684a2dd8f91aeedfd1413d7fa1..cffdac372a33d7b3b2fc7282cec520626c6940de 100644 --- a/drivers/net/ethernet/ibm/ibmvnic.h +++ b/drivers/net/ethernet/ibm/ibmvnic.h @@ -377,11 +377,16 @@ struct ibmvnic_phys_parms { u8 flags2; #define IBMVNIC_LOGICAL_LNK_ACTIVE 0x80 __be32 speed; -#define IBMVNIC_AUTONEG 0x80 -#define IBMVNIC_10MBPS 0x40 -#define IBMVNIC_100MBPS 0x20 -#define IBMVNIC_1GBPS 0x10 -#define IBMVNIC_10GBPS 0x08 +#define IBMVNIC_AUTONEG 0x80000000 +#define IBMVNIC_10MBPS 0x40000000 +#define IBMVNIC_100MBPS 0x20000000 +#define IBMVNIC_1GBPS 0x10000000 +#define IBMVNIC_10GBP 0x08000000 +#define IBMVNIC_40GBPS 0x04000000 +#define IBMVNIC_100GBPS 0x02000000 +#define IBMVNIC_25GBPS 0x01000000 +#define IBMVNIC_50GBPS 0x00800000 +#define IBMVNIC_200GBPS 0x00400000 __be32 mtu; struct ibmvnic_rc rc; } __packed __aligned(8); @@ -850,6 +855,7 @@ struct ibmvnic_crq_queue { dma_addr_t msg_token; spinlock_t lock; bool active; + char name[32]; }; union sub_crq { @@ -876,6 +882,7 @@ struct ibmvnic_sub_crq_queue { struct sk_buff *rx_skb_top; struct ibmvnic_adapter *adapter; atomic_t used; + char name[32]; }; struct ibmvnic_long_term_buff { @@ -999,6 +1006,9 @@ struct ibmvnic_adapter { int phys_link_state; int logical_link_state; + u32 speed; + u8 duplex; + /* login data */ struct ibmvnic_login_buffer *login_buf; dma_addr_t login_buf_token; diff --git a/drivers/net/ethernet/intel/e100.c b/drivers/net/ethernet/intel/e100.c index 0fd268070fb4a3344735526f1bb299a80d62fb9b..a65d5a9ba7db37eb3485adca499e88e5f4684268 100644 --- a/drivers/net/ethernet/intel/e100.c +++ b/drivers/net/ethernet/intel/e100.c @@ -2797,7 +2797,7 @@ static int e100_set_features(struct net_device *netdev, netdev->features = features; e100_exec_cb(nic, NULL, e100_configure); - return 0; + return 1; } static const struct net_device_ops e100_netdev_ops = { diff --git a/drivers/net/ethernet/intel/e1000/e1000_main.c b/drivers/net/ethernet/intel/e1000/e1000_main.c index 466bf1ea186d299d1be7413f2554e7ea7df7b228..551de8c2fef2275b6f39a4820d5a109fb2849d04 100644 --- a/drivers/net/ethernet/intel/e1000/e1000_main.c +++ b/drivers/net/ethernet/intel/e1000/e1000_main.c @@ -820,7 +820,7 @@ static int e1000_set_features(struct net_device *netdev, else e1000_reset(adapter); - return 0; + return 1; } static const struct net_device_ops e1000_netdev_ops = { @@ -3267,7 +3267,7 @@ static netdev_tx_t e1000_xmit_frame(struct sk_buff *skb, /* Make sure there is space in the ring for the next send. */ e1000_maybe_stop_tx(netdev, tx_ring, desc_needed); - if (!skb->xmit_more || + if (!netdev_xmit_more() || netif_xmit_stopped(netdev_get_tx_queue(netdev, 0))) { writel(tx_ring->next_to_use, hw->hw_addr + tx_ring->tdt); } diff --git a/drivers/net/ethernet/intel/e1000e/netdev.c b/drivers/net/ethernet/intel/e1000e/netdev.c index 022c3ac0e40f2b0bcbe5c7625ce173b1e82272a5..0e09bede42a2bd2c912366a68863a52a22def8ee 100644 --- a/drivers/net/ethernet/intel/e1000e/netdev.c +++ b/drivers/net/ethernet/intel/e1000e/netdev.c @@ -5896,7 +5896,7 @@ static netdev_tx_t e1000_xmit_frame(struct sk_buff *skb, DIV_ROUND_UP(PAGE_SIZE, adapter->tx_fifo_limit) + 2)); - if (!skb->xmit_more || + if (!netdev_xmit_more() || netif_xmit_stopped(netdev_get_tx_queue(netdev, 0))) { if (adapter->flags2 & FLAG2_PCIM2PCI_ARBITER_WA) e1000e_update_tdt_wa(tx_ring, @@ -6996,7 +6996,7 @@ static int e1000_set_features(struct net_device *netdev, else e1000e_reset(adapter); - return 0; + return 1; } static const struct net_device_ops e1000e_netdev_ops = { @@ -7343,7 +7343,7 @@ static int e1000_probe(struct pci_dev *pdev, const struct pci_device_id *ent) dev_pm_set_driver_flags(&pdev->dev, DPM_FLAG_NEVER_SKIP); - if (pci_dev_run_wake(pdev)) + if (pci_dev_run_wake(pdev) && hw->mac.type < e1000_pch_cnp) pm_runtime_put_noidle(&pdev->dev); return 0; diff --git a/drivers/net/ethernet/intel/fm10k/fm10k_main.c b/drivers/net/ethernet/intel/fm10k/fm10k_main.c index cbf76a96e94e32f5944f1631e3b757589d37433d..90270b4a16829f12e9651a3d5542f74016643af1 100644 --- a/drivers/net/ethernet/intel/fm10k/fm10k_main.c +++ b/drivers/net/ethernet/intel/fm10k/fm10k_main.c @@ -280,7 +280,7 @@ static bool fm10k_add_rx_frag(struct fm10k_rx_buffer *rx_buffer, /* we need the header to contain the greater of either ETH_HLEN or * 60 bytes if the skb->len is less than 60 for skb_pad. */ - pull_len = eth_get_headlen(va, FM10K_RX_HDR_LEN); + pull_len = eth_get_headlen(skb->dev, va, FM10K_RX_HDR_LEN); /* align pull length to size of long to optimize memcpy performance */ memcpy(__skb_put(skb, pull_len), va, ALIGN(pull_len, sizeof(long))); @@ -1037,7 +1037,7 @@ static void fm10k_tx_map(struct fm10k_ring *tx_ring, fm10k_maybe_stop_tx(tx_ring, DESC_NEEDED); /* notify HW of packet */ - if (netif_xmit_stopped(txring_txq(tx_ring)) || !skb->xmit_more) { + if (netif_xmit_stopped(txring_txq(tx_ring)) || !netdev_xmit_more()) { writel(i, tx_ring->tail); } diff --git a/drivers/net/ethernet/intel/i40e/Makefile b/drivers/net/ethernet/intel/i40e/Makefile index 50590e8d1fd1389926a549a7ebd7c47a0ac9e28e..2f21b3e89fd03b605f76514297ea1790a9e89b53 100644 --- a/drivers/net/ethernet/intel/i40e/Makefile +++ b/drivers/net/ethernet/intel/i40e/Makefile @@ -21,6 +21,7 @@ i40e-objs := i40e_main.o \ i40e_diag.o \ i40e_txrx.o \ i40e_ptp.o \ + i40e_ddp.o \ i40e_client.o \ i40e_virtchnl_pf.o \ i40e_xsk.o diff --git a/drivers/net/ethernet/intel/i40e/i40e.h b/drivers/net/ethernet/intel/i40e/i40e.h index d3cc3427caad187ecf5bf967404f86182082790a..7ce42040b851d0a2841f90bffb6ea104d3d60f48 100644 --- a/drivers/net/ethernet/intel/i40e/i40e.h +++ b/drivers/net/ethernet/intel/i40e/i40e.h @@ -149,6 +149,7 @@ enum i40e_state_t { __I40E_CLIENT_L2_CHANGE, __I40E_CLIENT_RESET, __I40E_VIRTCHNL_OP_PENDING, + __I40E_RECOVERY_MODE, /* This must be last as it determines the size of the BITMAP */ __I40E_STATE_SIZE__, }; @@ -321,6 +322,29 @@ struct i40e_udp_port_config { u8 filter_index; }; +#define I40_DDP_FLASH_REGION 100 +#define I40E_PROFILE_INFO_SIZE 48 +#define I40E_MAX_PROFILE_NUM 16 +#define I40E_PROFILE_LIST_SIZE \ + (I40E_PROFILE_INFO_SIZE * I40E_MAX_PROFILE_NUM + 4) +#define I40E_DDP_PROFILE_PATH "intel/i40e/ddp/" +#define I40E_DDP_PROFILE_NAME_MAX 64 + +int i40e_ddp_load(struct net_device *netdev, const u8 *data, size_t size, + bool is_add); +int i40e_ddp_flash(struct net_device *netdev, struct ethtool_flash *flash); + +struct i40e_ddp_profile_list { + u32 p_count; + struct i40e_profile_info p_info[0]; +}; + +struct i40e_ddp_old_profile_list { + struct list_head list; + size_t old_ddp_size; + u8 old_ddp_buf[0]; +}; + /* macros related to FLX_PIT */ #define I40E_FLEX_SET_FSIZE(fsize) (((fsize) << \ I40E_PRTQF_FLX_PIT_FSIZE_SHIFT) & \ @@ -589,6 +613,8 @@ struct i40e_pf { struct sk_buff *ptp_tx_skb; unsigned long ptp_tx_start; struct hwtstamp_config tstamp_config; + struct timespec64 ptp_prev_hw_time; + ktime_t ptp_reset_start; struct mutex tmreg_lock; /* Used to protect the SYSTIME registers. */ u32 ptp_adj_mult; u32 tx_hwtstamp_timeouts; @@ -610,6 +636,8 @@ struct i40e_pf { u16 override_q_count; u16 last_sw_conf_flags; u16 last_sw_conf_valid_flags; + /* List to keep previous DDP profiles to be rolled back in the future */ + struct list_head ddp_old_prof; }; /** @@ -1083,6 +1111,8 @@ void i40e_ptp_rx_hwtstamp(struct i40e_pf *pf, struct sk_buff *skb, u8 index); void i40e_ptp_set_increment(struct i40e_pf *pf); int i40e_ptp_set_ts_config(struct i40e_pf *pf, struct ifreq *ifr); int i40e_ptp_get_ts_config(struct i40e_pf *pf, struct ifreq *ifr); +void i40e_ptp_save_hw_time(struct i40e_pf *pf); +void i40e_ptp_restore_hw_time(struct i40e_pf *pf); void i40e_ptp_init(struct i40e_pf *pf); void i40e_ptp_stop(struct i40e_pf *pf); int i40e_is_vsi_uplink_mode_veb(struct i40e_vsi *vsi); diff --git a/drivers/net/ethernet/intel/i40e/i40e_adminq.c b/drivers/net/ethernet/intel/i40e/i40e_adminq.c index 7ab61f6ebb5fa23582e8b5ce1d5e8ee55502b52b..243dcd4bec191f6fee7c83d11b9f3cbf6e2d2083 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_adminq.c +++ b/drivers/net/ethernet/intel/i40e/i40e_adminq.c @@ -608,6 +608,11 @@ i40e_status i40e_init_adminq(struct i40e_hw *hw) hw->aq.api_min_ver >= 7)) hw->flags |= I40E_HW_FLAG_802_1AD_CAPABLE; + if (hw->aq.api_maj_ver > 1 || + (hw->aq.api_maj_ver == 1 && + hw->aq.api_min_ver >= 8)) + hw->flags |= I40E_HW_FLAG_FW_LLDP_PERSISTENT; + if (hw->aq.api_maj_ver > I40E_FW_API_VERSION_MAJOR) { ret_code = I40E_ERR_FIRMWARE_API_VERSION; goto init_adminq_free_arq; @@ -749,7 +754,7 @@ i40e_status i40e_asq_send_command(struct i40e_hw *hw, if (val >= hw->aq.num_asq_entries) { i40e_debug(hw, I40E_DEBUG_AQ_MESSAGE, "AQTX: head overrun at %d\n", val); - status = I40E_ERR_QUEUE_EMPTY; + status = I40E_ERR_ADMIN_QUEUE_FULL; goto asq_send_command_error; } diff --git a/drivers/net/ethernet/intel/i40e/i40e_adminq_cmd.h b/drivers/net/ethernet/intel/i40e/i40e_adminq_cmd.h index 11506102471c14d8cf36f52488e10073d55c46dd..6536023fa074eb2e2d4a72c37eaae09ae6a51f62 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_adminq_cmd.h +++ b/drivers/net/ethernet/intel/i40e/i40e_adminq_cmd.h @@ -11,8 +11,8 @@ */ #define I40E_FW_API_VERSION_MAJOR 0x0001 -#define I40E_FW_API_VERSION_MINOR_X722 0x0006 -#define I40E_FW_API_VERSION_MINOR_X710 0x0007 +#define I40E_FW_API_VERSION_MINOR_X722 0x0008 +#define I40E_FW_API_VERSION_MINOR_X710 0x0008 #define I40E_FW_MINOR_VERSION(_h) ((_h)->mac.type == I40E_MAC_XL710 ? \ I40E_FW_API_VERSION_MINOR_X710 : \ @@ -261,6 +261,7 @@ enum i40e_admin_queue_opc { i40e_aqc_opc_get_cee_dcb_cfg = 0x0A07, i40e_aqc_opc_lldp_set_local_mib = 0x0A08, i40e_aqc_opc_lldp_stop_start_spec_agent = 0x0A09, + i40e_aqc_opc_lldp_restore = 0x0A0A, /* Tunnel commands */ i40e_aqc_opc_add_udp_tunnel = 0x0B00, @@ -1887,6 +1888,8 @@ enum i40e_aq_phy_type { I40E_PHY_TYPE_25GBASE_LR = 0x22, I40E_PHY_TYPE_25GBASE_AOC = 0x23, I40E_PHY_TYPE_25GBASE_ACC = 0x24, + I40E_PHY_TYPE_2_5GBASE_T = 0x30, + I40E_PHY_TYPE_5GBASE_T = 0x31, I40E_PHY_TYPE_MAX, I40E_PHY_TYPE_NOT_SUPPORTED_HIGH_TEMP = 0xFD, I40E_PHY_TYPE_EMPTY = 0xFE, @@ -1928,19 +1931,25 @@ enum i40e_aq_phy_type { BIT_ULL(I40E_PHY_TYPE_25GBASE_SR) | \ BIT_ULL(I40E_PHY_TYPE_25GBASE_LR) | \ BIT_ULL(I40E_PHY_TYPE_25GBASE_AOC) | \ - BIT_ULL(I40E_PHY_TYPE_25GBASE_ACC)) + BIT_ULL(I40E_PHY_TYPE_25GBASE_ACC) | \ + BIT_ULL(I40E_PHY_TYPE_2_5GBASE_T) | \ + BIT_ULL(I40E_PHY_TYPE_5GBASE_T)) +#define I40E_LINK_SPEED_2_5GB_SHIFT 0x0 #define I40E_LINK_SPEED_100MB_SHIFT 0x1 #define I40E_LINK_SPEED_1000MB_SHIFT 0x2 #define I40E_LINK_SPEED_10GB_SHIFT 0x3 #define I40E_LINK_SPEED_40GB_SHIFT 0x4 #define I40E_LINK_SPEED_20GB_SHIFT 0x5 #define I40E_LINK_SPEED_25GB_SHIFT 0x6 +#define I40E_LINK_SPEED_5GB_SHIFT 0x7 enum i40e_aq_link_speed { I40E_LINK_SPEED_UNKNOWN = 0, I40E_LINK_SPEED_100MB = BIT(I40E_LINK_SPEED_100MB_SHIFT), I40E_LINK_SPEED_1GB = BIT(I40E_LINK_SPEED_1000MB_SHIFT), + I40E_LINK_SPEED_2_5GB = (1 << I40E_LINK_SPEED_2_5GB_SHIFT), + I40E_LINK_SPEED_5GB = (1 << I40E_LINK_SPEED_5GB_SHIFT), I40E_LINK_SPEED_10GB = BIT(I40E_LINK_SPEED_10GB_SHIFT), I40E_LINK_SPEED_40GB = BIT(I40E_LINK_SPEED_40GB_SHIFT), I40E_LINK_SPEED_20GB = BIT(I40E_LINK_SPEED_20GB_SHIFT), @@ -1986,6 +1995,8 @@ struct i40e_aq_get_phy_abilities_resp { #define I40E_AQ_PHY_TYPE_EXT_25G_LR 0x08 #define I40E_AQ_PHY_TYPE_EXT_25G_AOC 0x10 #define I40E_AQ_PHY_TYPE_EXT_25G_ACC 0x20 +#define I40E_AQ_PHY_TYPE_EXT_2_5GBASE_T 0x40 +#define I40E_AQ_PHY_TYPE_EXT_5GBASE_T 0x80 u8 fec_cfg_curr_mod_ext_info; #define I40E_AQ_ENABLE_FEC_KR 0x01 #define I40E_AQ_ENABLE_FEC_RS 0x02 @@ -2498,18 +2509,19 @@ I40E_CHECK_CMD_LENGTH(i40e_aqc_lldp_update_tlv); /* Stop LLDP (direct 0x0A05) */ struct i40e_aqc_lldp_stop { u8 command; -#define I40E_AQ_LLDP_AGENT_STOP 0x0 -#define I40E_AQ_LLDP_AGENT_SHUTDOWN 0x1 +#define I40E_AQ_LLDP_AGENT_STOP 0x0 +#define I40E_AQ_LLDP_AGENT_SHUTDOWN 0x1 +#define I40E_AQ_LLDP_AGENT_STOP_PERSIST 0x2 u8 reserved[15]; }; I40E_CHECK_CMD_LENGTH(i40e_aqc_lldp_stop); /* Start LLDP (direct 0x0A06) */ - struct i40e_aqc_lldp_start { u8 command; -#define I40E_AQ_LLDP_AGENT_START 0x1 +#define I40E_AQ_LLDP_AGENT_START 0x1 +#define I40E_AQ_LLDP_AGENT_START_PERSIST 0x2 u8 reserved[15]; }; @@ -2633,6 +2645,16 @@ struct i40e_aqc_lldp_stop_start_specific_agent { I40E_CHECK_CMD_LENGTH(i40e_aqc_lldp_stop_start_specific_agent); +/* Restore LLDP Agent factory settings (direct 0x0A0A) */ +struct i40e_aqc_lldp_restore { + u8 command; +#define I40E_AQ_LLDP_AGENT_RESTORE_NOT 0x0 +#define I40E_AQ_LLDP_AGENT_RESTORE 0x1 + u8 reserved[15]; +}; + +I40E_CHECK_CMD_LENGTH(i40e_aqc_lldp_restore); + /* Add Udp Tunnel command and completion (direct 0x0B00) */ struct i40e_aqc_add_udp_tunnel { __le16 udp_port; diff --git a/drivers/net/ethernet/intel/i40e/i40e_client.c b/drivers/net/ethernet/intel/i40e/i40e_client.c index 5f3b8b9ff511d49a13e24f30f0d2a589915e40fe..e81530ca08d036ecf87f642fbf90c4c073d0850f 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_client.c +++ b/drivers/net/ethernet/intel/i40e/i40e_client.c @@ -578,11 +578,9 @@ static int i40e_client_setup_qvlist(struct i40e_info *ldev, struct i40e_hw *hw = &pf->hw; struct i40e_qv_info *qv_info; u32 v_idx, i, reg_idx, reg; - u32 size; - size = sizeof(struct i40e_qvlist_info) + - (sizeof(struct i40e_qv_info) * (qvlist_info->num_vectors - 1)); - ldev->qvlist_info = kzalloc(size, GFP_KERNEL); + ldev->qvlist_info = kzalloc(struct_size(ldev->qvlist_info, qv_info, + qvlist_info->num_vectors - 1), GFP_KERNEL); if (!ldev->qvlist_info) return -ENOMEM; ldev->qvlist_info->num_vectors = qvlist_info->num_vectors; diff --git a/drivers/net/ethernet/intel/i40e/i40e_common.c b/drivers/net/ethernet/intel/i40e/i40e_common.c index 97a9b1fb47638b47ee97ad7206279e71ba14526a..ecb1adaa54ecff44786a0d263659bc516524839f 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_common.c +++ b/drivers/net/ethernet/intel/i40e/i40e_common.c @@ -28,10 +28,14 @@ static i40e_status i40e_set_mac_type(struct i40e_hw *hw) case I40E_DEV_ID_QSFP_C: case I40E_DEV_ID_10G_BASE_T: case I40E_DEV_ID_10G_BASE_T4: + case I40E_DEV_ID_10G_B: + case I40E_DEV_ID_10G_SFP: case I40E_DEV_ID_20G_KR2: case I40E_DEV_ID_20G_KR2_A: case I40E_DEV_ID_25G_B: case I40E_DEV_ID_25G_SFP28: + case I40E_DEV_ID_X710_N3000: + case I40E_DEV_ID_XXV710_N3000: hw->mac.type = I40E_MAC_XL710; break; case I40E_DEV_ID_KX_X722: @@ -1149,6 +1153,8 @@ static enum i40e_media_type i40e_get_media_type(struct i40e_hw *hw) break; case I40E_PHY_TYPE_100BASE_TX: case I40E_PHY_TYPE_1000BASE_T: + case I40E_PHY_TYPE_2_5GBASE_T: + case I40E_PHY_TYPE_5GBASE_T: case I40E_PHY_TYPE_10GBASE_T: media = I40E_MEDIA_TYPE_BASET; break; @@ -1466,7 +1472,6 @@ static u32 i40e_led_is_mine(struct i40e_hw *hw, int idx) **/ u32 i40e_led_get(struct i40e_hw *hw) { - u32 current_mode = 0; u32 mode = 0; int i; @@ -1479,21 +1484,6 @@ u32 i40e_led_get(struct i40e_hw *hw) if (!gpio_val) continue; - /* ignore gpio LED src mode entries related to the activity - * LEDs - */ - current_mode = ((gpio_val & I40E_GLGEN_GPIO_CTL_LED_MODE_MASK) - >> I40E_GLGEN_GPIO_CTL_LED_MODE_SHIFT); - switch (current_mode) { - case I40E_COMBINED_ACTIVITY: - case I40E_FILTER_ACTIVITY: - case I40E_MAC_ACTIVITY: - case I40E_LINK_ACTIVITY: - continue; - default: - break; - } - mode = (gpio_val & I40E_GLGEN_GPIO_CTL_LED_MODE_MASK) >> I40E_GLGEN_GPIO_CTL_LED_MODE_SHIFT; break; @@ -1513,7 +1503,6 @@ u32 i40e_led_get(struct i40e_hw *hw) **/ void i40e_led_set(struct i40e_hw *hw, u32 mode, bool blink) { - u32 current_mode = 0; int i; if (mode & 0xfffffff0) @@ -1527,22 +1516,6 @@ void i40e_led_set(struct i40e_hw *hw, u32 mode, bool blink) if (!gpio_val) continue; - - /* ignore gpio LED src mode entries related to the activity - * LEDs - */ - current_mode = ((gpio_val & I40E_GLGEN_GPIO_CTL_LED_MODE_MASK) - >> I40E_GLGEN_GPIO_CTL_LED_MODE_SHIFT); - switch (current_mode) { - case I40E_COMBINED_ACTIVITY: - case I40E_FILTER_ACTIVITY: - case I40E_MAC_ACTIVITY: - case I40E_LINK_ACTIVITY: - continue; - default: - break; - } - gpio_val &= ~I40E_GLGEN_GPIO_CTL_LED_MODE_MASK; /* this & is a bit of paranoia, but serves as a range check */ gpio_val |= ((mode << I40E_GLGEN_GPIO_CTL_LED_MODE_SHIFT) & @@ -3656,15 +3629,55 @@ i40e_status i40e_aq_cfg_lldp_mib_change_event(struct i40e_hw *hw, return status; } +/** + * i40e_aq_restore_lldp + * @hw: pointer to the hw struct + * @setting: pointer to factory setting variable or NULL + * @restore: True if factory settings should be restored + * @cmd_details: pointer to command details structure or NULL + * + * Restore LLDP Agent factory settings if @restore set to True. In other case + * only returns factory setting in AQ response. + **/ +enum i40e_status_code +i40e_aq_restore_lldp(struct i40e_hw *hw, u8 *setting, bool restore, + struct i40e_asq_cmd_details *cmd_details) +{ + struct i40e_aq_desc desc; + struct i40e_aqc_lldp_restore *cmd = + (struct i40e_aqc_lldp_restore *)&desc.params.raw; + i40e_status status; + + if (!(hw->flags & I40E_HW_FLAG_FW_LLDP_PERSISTENT)) { + i40e_debug(hw, I40E_DEBUG_ALL, + "Restore LLDP not supported by current FW version.\n"); + return I40E_ERR_DEVICE_NOT_SUPPORTED; + } + + i40e_fill_default_direct_cmd_desc(&desc, i40e_aqc_opc_lldp_restore); + + if (restore) + cmd->command |= I40E_AQ_LLDP_AGENT_RESTORE; + + status = i40e_asq_send_command(hw, &desc, NULL, 0, cmd_details); + + if (setting) + *setting = cmd->command & 1; + + return status; +} + /** * i40e_aq_stop_lldp * @hw: pointer to the hw struct * @shutdown_agent: True if LLDP Agent needs to be Shutdown + * @persist: True if stop of LLDP should be persistent across power cycles * @cmd_details: pointer to command details structure or NULL * * Stop or Shutdown the embedded LLDP Agent **/ i40e_status i40e_aq_stop_lldp(struct i40e_hw *hw, bool shutdown_agent, + bool persist, struct i40e_asq_cmd_details *cmd_details) { struct i40e_aq_desc desc; @@ -3677,6 +3690,14 @@ i40e_status i40e_aq_stop_lldp(struct i40e_hw *hw, bool shutdown_agent, if (shutdown_agent) cmd->command |= I40E_AQ_LLDP_AGENT_SHUTDOWN; + if (persist) { + if (hw->flags & I40E_HW_FLAG_FW_LLDP_PERSISTENT) + cmd->command |= I40E_AQ_LLDP_AGENT_STOP_PERSIST; + else + i40e_debug(hw, I40E_DEBUG_ALL, + "Persistent Stop LLDP not supported by current FW version.\n"); + } + status = i40e_asq_send_command(hw, &desc, NULL, 0, cmd_details); return status; @@ -3686,13 +3707,14 @@ i40e_status i40e_aq_stop_lldp(struct i40e_hw *hw, bool shutdown_agent, * i40e_aq_start_lldp * @hw: pointer to the hw struct * @buff: buffer for result + * @persist: True if start of LLDP should be persistent across power cycles * @buff_size: buffer size * @cmd_details: pointer to command details structure or NULL * * Start the embedded LLDP Agent on all ports. **/ -i40e_status i40e_aq_start_lldp(struct i40e_hw *hw, - struct i40e_asq_cmd_details *cmd_details) +i40e_status i40e_aq_start_lldp(struct i40e_hw *hw, bool persist, + struct i40e_asq_cmd_details *cmd_details) { struct i40e_aq_desc desc; struct i40e_aqc_lldp_start *cmd = @@ -3702,6 +3724,15 @@ i40e_status i40e_aq_start_lldp(struct i40e_hw *hw, i40e_fill_default_direct_cmd_desc(&desc, i40e_aqc_opc_lldp_start); cmd->command = I40E_AQ_LLDP_AGENT_START; + + if (persist) { + if (hw->flags & I40E_HW_FLAG_FW_LLDP_PERSISTENT) + cmd->command |= I40E_AQ_LLDP_AGENT_START_PERSIST; + else + i40e_debug(hw, I40E_DEBUG_ALL, + "Persistent Start LLDP not supported by current FW version.\n"); + } + status = i40e_asq_send_command(hw, &desc, NULL, 0, cmd_details); return status; @@ -4873,6 +4904,7 @@ i40e_status i40e_read_phy_register(struct i40e_hw *hw, break; case I40E_DEV_ID_10G_BASE_T: case I40E_DEV_ID_10G_BASE_T4: + case I40E_DEV_ID_10G_BASE_T_BC: case I40E_DEV_ID_10G_BASE_T_X722: case I40E_DEV_ID_25G_B: case I40E_DEV_ID_25G_SFP28: @@ -5448,6 +5480,163 @@ i40e_find_segment_in_package(u32 segment_type, return NULL; } +/* Get section table in profile */ +#define I40E_SECTION_TABLE(profile, sec_tbl) \ + do { \ + struct i40e_profile_segment *p = (profile); \ + u32 count; \ + u32 *nvm; \ + count = p->device_table_count; \ + nvm = (u32 *)&p->device_table[count]; \ + sec_tbl = (struct i40e_section_table *)&nvm[nvm[0] + 1]; \ + } while (0) + +/* Get section header in profile */ +#define I40E_SECTION_HEADER(profile, offset) \ + (struct i40e_profile_section_header *)((u8 *)(profile) + (offset)) + +/** + * i40e_find_section_in_profile + * @section_type: the section type to search for (i.e., SECTION_TYPE_NOTE) + * @profile: pointer to the i40e segment header to be searched + * + * This function searches i40e segment for a particular section type. On + * success it returns a pointer to the section header, otherwise it will + * return NULL. + **/ +struct i40e_profile_section_header * +i40e_find_section_in_profile(u32 section_type, + struct i40e_profile_segment *profile) +{ + struct i40e_profile_section_header *sec; + struct i40e_section_table *sec_tbl; + u32 sec_off; + u32 i; + + if (profile->header.type != SEGMENT_TYPE_I40E) + return NULL; + + I40E_SECTION_TABLE(profile, sec_tbl); + + for (i = 0; i < sec_tbl->section_count; i++) { + sec_off = sec_tbl->section_offset[i]; + sec = I40E_SECTION_HEADER(profile, sec_off); + if (sec->section.type == section_type) + return sec; + } + + return NULL; +} + +/** + * i40e_ddp_exec_aq_section - Execute generic AQ for DDP + * @hw: pointer to the hw struct + * @aq: command buffer containing all data to execute AQ + **/ +static enum +i40e_status_code i40e_ddp_exec_aq_section(struct i40e_hw *hw, + struct i40e_profile_aq_section *aq) +{ + i40e_status status; + struct i40e_aq_desc desc; + u8 *msg = NULL; + u16 msglen; + + i40e_fill_default_direct_cmd_desc(&desc, aq->opcode); + desc.flags |= cpu_to_le16(aq->flags); + memcpy(desc.params.raw, aq->param, sizeof(desc.params.raw)); + + msglen = aq->datalen; + if (msglen) { + desc.flags |= cpu_to_le16((u16)(I40E_AQ_FLAG_BUF | + I40E_AQ_FLAG_RD)); + if (msglen > I40E_AQ_LARGE_BUF) + desc.flags |= cpu_to_le16((u16)I40E_AQ_FLAG_LB); + desc.datalen = cpu_to_le16(msglen); + msg = &aq->data[0]; + } + + status = i40e_asq_send_command(hw, &desc, msg, msglen, NULL); + + if (status) { + i40e_debug(hw, I40E_DEBUG_PACKAGE, + "unable to exec DDP AQ opcode %u, error %d\n", + aq->opcode, status); + return status; + } + + /* copy returned desc to aq_buf */ + memcpy(aq->param, desc.params.raw, sizeof(desc.params.raw)); + + return 0; +} + +/** + * i40e_validate_profile + * @hw: pointer to the hardware structure + * @profile: pointer to the profile segment of the package to be validated + * @track_id: package tracking id + * @rollback: flag if the profile is for rollback. + * + * Validates supported devices and profile's sections. + */ +static enum i40e_status_code +i40e_validate_profile(struct i40e_hw *hw, struct i40e_profile_segment *profile, + u32 track_id, bool rollback) +{ + struct i40e_profile_section_header *sec = NULL; + i40e_status status = 0; + struct i40e_section_table *sec_tbl; + u32 vendor_dev_id; + u32 dev_cnt; + u32 sec_off; + u32 i; + + if (track_id == I40E_DDP_TRACKID_INVALID) { + i40e_debug(hw, I40E_DEBUG_PACKAGE, "Invalid track_id\n"); + return I40E_NOT_SUPPORTED; + } + + dev_cnt = profile->device_table_count; + for (i = 0; i < dev_cnt; i++) { + vendor_dev_id = profile->device_table[i].vendor_dev_id; + if ((vendor_dev_id >> 16) == PCI_VENDOR_ID_INTEL && + hw->device_id == (vendor_dev_id & 0xFFFF)) + break; + } + if (dev_cnt && i == dev_cnt) { + i40e_debug(hw, I40E_DEBUG_PACKAGE, + "Device doesn't support DDP\n"); + return I40E_ERR_DEVICE_NOT_SUPPORTED; + } + + I40E_SECTION_TABLE(profile, sec_tbl); + + /* Validate sections types */ + for (i = 0; i < sec_tbl->section_count; i++) { + sec_off = sec_tbl->section_offset[i]; + sec = I40E_SECTION_HEADER(profile, sec_off); + if (rollback) { + if (sec->section.type == SECTION_TYPE_MMIO || + sec->section.type == SECTION_TYPE_AQ || + sec->section.type == SECTION_TYPE_RB_AQ) { + i40e_debug(hw, I40E_DEBUG_PACKAGE, + "Not a roll-back package\n"); + return I40E_NOT_SUPPORTED; + } + } else { + if (sec->section.type == SECTION_TYPE_RB_AQ || + sec->section.type == SECTION_TYPE_RB_MMIO) { + i40e_debug(hw, I40E_DEBUG_PACKAGE, + "Not an original package\n"); + return I40E_NOT_SUPPORTED; + } + } + } + + return status; +} + /** * i40e_write_profile * @hw: pointer to the hardware structure @@ -5463,47 +5652,99 @@ i40e_write_profile(struct i40e_hw *hw, struct i40e_profile_segment *profile, i40e_status status = 0; struct i40e_section_table *sec_tbl; struct i40e_profile_section_header *sec = NULL; - u32 dev_cnt; - u32 vendor_dev_id; - u32 *nvm; + struct i40e_profile_aq_section *ddp_aq; u32 section_size = 0; u32 offset = 0, info = 0; + u32 sec_off; u32 i; - dev_cnt = profile->device_table_count; + status = i40e_validate_profile(hw, profile, track_id, false); + if (status) + return status; - for (i = 0; i < dev_cnt; i++) { - vendor_dev_id = profile->device_table[i].vendor_dev_id; - if ((vendor_dev_id >> 16) == PCI_VENDOR_ID_INTEL) - if (hw->device_id == (vendor_dev_id & 0xFFFF)) + I40E_SECTION_TABLE(profile, sec_tbl); + + for (i = 0; i < sec_tbl->section_count; i++) { + sec_off = sec_tbl->section_offset[i]; + sec = I40E_SECTION_HEADER(profile, sec_off); + /* Process generic admin command */ + if (sec->section.type == SECTION_TYPE_AQ) { + ddp_aq = (struct i40e_profile_aq_section *)&sec[1]; + status = i40e_ddp_exec_aq_section(hw, ddp_aq); + if (status) { + i40e_debug(hw, I40E_DEBUG_PACKAGE, + "Failed to execute aq: section %d, opcode %u\n", + i, ddp_aq->opcode); break; + } + sec->section.type = SECTION_TYPE_RB_AQ; + } + + /* Skip any non-mmio sections */ + if (sec->section.type != SECTION_TYPE_MMIO) + continue; + + section_size = sec->section.size + + sizeof(struct i40e_profile_section_header); + + /* Write MMIO section */ + status = i40e_aq_write_ddp(hw, (void *)sec, (u16)section_size, + track_id, &offset, &info, NULL); + if (status) { + i40e_debug(hw, I40E_DEBUG_PACKAGE, + "Failed to write profile: section %d, offset %d, info %d\n", + i, offset, info); + break; + } } - if (i == dev_cnt) { - i40e_debug(hw, I40E_DEBUG_PACKAGE, "Device doesn't support DDP"); - return I40E_ERR_DEVICE_NOT_SUPPORTED; - } + return status; +} + +/** + * i40e_rollback_profile + * @hw: pointer to the hardware structure + * @profile: pointer to the profile segment of the package to be removed + * @track_id: package tracking id + * + * Rolls back previously loaded package. + */ +enum i40e_status_code +i40e_rollback_profile(struct i40e_hw *hw, struct i40e_profile_segment *profile, + u32 track_id) +{ + struct i40e_profile_section_header *sec = NULL; + i40e_status status = 0; + struct i40e_section_table *sec_tbl; + u32 offset = 0, info = 0; + u32 section_size = 0; + u32 sec_off; + int i; - nvm = (u32 *)&profile->device_table[dev_cnt]; - sec_tbl = (struct i40e_section_table *)&nvm[nvm[0] + 1]; + status = i40e_validate_profile(hw, profile, track_id, true); + if (status) + return status; - for (i = 0; i < sec_tbl->section_count; i++) { - sec = (struct i40e_profile_section_header *)((u8 *)profile + - sec_tbl->section_offset[i]); + I40E_SECTION_TABLE(profile, sec_tbl); - /* Skip 'AQ', 'note' and 'name' sections */ - if (sec->section.type != SECTION_TYPE_MMIO) + /* For rollback write sections in reverse */ + for (i = sec_tbl->section_count - 1; i >= 0; i--) { + sec_off = sec_tbl->section_offset[i]; + sec = I40E_SECTION_HEADER(profile, sec_off); + + /* Skip any non-rollback sections */ + if (sec->section.type != SECTION_TYPE_RB_MMIO) continue; section_size = sec->section.size + sizeof(struct i40e_profile_section_header); - /* Write profile */ + /* Write roll-back MMIO section */ status = i40e_aq_write_ddp(hw, (void *)sec, (u16)section_size, track_id, &offset, &info, NULL); if (status) { i40e_debug(hw, I40E_DEBUG_PACKAGE, - "Failed to write profile: offset %d, info %d", - offset, info); + "Failed to write profile: section %d, offset %d, info %d\n", + i, offset, info); break; } } diff --git a/drivers/net/ethernet/intel/i40e/i40e_dcb.c b/drivers/net/ethernet/intel/i40e/i40e_dcb.c index 56bff8faf37185fa9c6f45910aad6eb146dc07c5..292eeb3def10581b5ea8ef0bae730b04ba147454 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_dcb.c +++ b/drivers/net/ethernet/intel/i40e/i40e_dcb.c @@ -863,22 +863,23 @@ out: /** * i40e_init_dcb * @hw: pointer to the hw struct + * @enable_mib_change: enable mib change event * * Update DCB configuration from the Firmware **/ -i40e_status i40e_init_dcb(struct i40e_hw *hw) +i40e_status i40e_init_dcb(struct i40e_hw *hw, bool enable_mib_change) { i40e_status ret = 0; struct i40e_lldp_variables lldp_cfg; u8 adminstatus = 0; if (!hw->func_caps.dcb) - return ret; + return I40E_NOT_SUPPORTED; /* Read LLDP NVM area */ ret = i40e_read_lldp_cfg(hw, &lldp_cfg); if (ret) - return ret; + return I40E_ERR_NOT_READY; /* Get the LLDP AdminStatus for the current port */ adminstatus = lldp_cfg.adminstatus >> (hw->port * 4); @@ -887,7 +888,7 @@ i40e_status i40e_init_dcb(struct i40e_hw *hw) /* LLDP agent disabled */ if (!adminstatus) { hw->dcbx_status = I40E_DCBX_STATUS_DISABLED; - return ret; + return I40E_ERR_NOT_READY; } /* Get DCBX status */ @@ -896,26 +897,19 @@ i40e_status i40e_init_dcb(struct i40e_hw *hw) return ret; /* Check the DCBX Status */ - switch (hw->dcbx_status) { - case I40E_DCBX_STATUS_DONE: - case I40E_DCBX_STATUS_IN_PROGRESS: + if (hw->dcbx_status == I40E_DCBX_STATUS_DONE || + hw->dcbx_status == I40E_DCBX_STATUS_IN_PROGRESS) { /* Get current DCBX configuration */ ret = i40e_get_dcb_config(hw); if (ret) return ret; - break; - case I40E_DCBX_STATUS_DISABLED: - return ret; - case I40E_DCBX_STATUS_NOT_STARTED: - case I40E_DCBX_STATUS_MULTIPLE_PEERS: - default: - break; + } else if (hw->dcbx_status == I40E_DCBX_STATUS_DISABLED) { + return I40E_ERR_NOT_READY; } /* Configure the LLDP MIB change event */ - ret = i40e_aq_cfg_lldp_mib_change_event(hw, true, NULL); - if (ret) - return ret; + if (enable_mib_change) + ret = i40e_aq_cfg_lldp_mib_change_event(hw, true, NULL); return ret; } diff --git a/drivers/net/ethernet/intel/i40e/i40e_dcb.h b/drivers/net/ethernet/intel/i40e/i40e_dcb.h index 2b748a60a843ce704ddcd34849607c638c1f9b86..ddb48ae7cce468cfc8f63aa3e4aeec0d234db056 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_dcb.h +++ b/drivers/net/ethernet/intel/i40e/i40e_dcb.h @@ -124,5 +124,5 @@ i40e_status i40e_aq_get_dcb_config(struct i40e_hw *hw, u8 mib_type, u8 bridgetype, struct i40e_dcbx_config *dcbcfg); i40e_status i40e_get_dcb_config(struct i40e_hw *hw); -i40e_status i40e_init_dcb(struct i40e_hw *hw); +i40e_status i40e_init_dcb(struct i40e_hw *hw, bool enable_mib_change); #endif /* _I40E_DCB_H_ */ diff --git a/drivers/net/ethernet/intel/i40e/i40e_ddp.c b/drivers/net/ethernet/intel/i40e/i40e_ddp.c new file mode 100644 index 0000000000000000000000000000000000000000..5e08f100c4134b5d9c3873b5fc5c67d8e3dbcab9 --- /dev/null +++ b/drivers/net/ethernet/intel/i40e/i40e_ddp.c @@ -0,0 +1,481 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2013 - 2018 Intel Corporation. */ + +#include "i40e.h" + +#include + +/** + * i40e_ddp_profiles_eq - checks if DDP profiles are the equivalent + * @a: new profile info + * @b: old profile info + * + * checks if DDP profiles are the equivalent. + * Returns true if profiles are the same. + **/ +static bool i40e_ddp_profiles_eq(struct i40e_profile_info *a, + struct i40e_profile_info *b) +{ + return a->track_id == b->track_id && + !memcmp(&a->version, &b->version, sizeof(a->version)) && + !memcmp(&a->name, &b->name, I40E_DDP_NAME_SIZE); +} + +/** + * i40e_ddp_does_profile_exist - checks if DDP profile loaded already + * @hw: HW data structure + * @pinfo: DDP profile information structure + * + * checks if DDP profile loaded already. + * Returns >0 if the profile exists. + * Returns 0 if the profile is absent. + * Returns <0 if error. + **/ +static int i40e_ddp_does_profile_exist(struct i40e_hw *hw, + struct i40e_profile_info *pinfo) +{ + struct i40e_ddp_profile_list *profile_list; + u8 buff[I40E_PROFILE_LIST_SIZE]; + i40e_status status; + int i; + + status = i40e_aq_get_ddp_list(hw, buff, I40E_PROFILE_LIST_SIZE, 0, + NULL); + if (status) + return -1; + + profile_list = (struct i40e_ddp_profile_list *)buff; + for (i = 0; i < profile_list->p_count; i++) { + if (i40e_ddp_profiles_eq(pinfo, &profile_list->p_info[i])) + return 1; + } + return 0; +} + +/** + * i40e_ddp_profiles_overlap - checks if DDP profiles overlap. + * @new: new profile info + * @old: old profile info + * + * checks if DDP profiles overlap. + * Returns true if profiles are overlap. + **/ +static bool i40e_ddp_profiles_overlap(struct i40e_profile_info *new, + struct i40e_profile_info *old) +{ + unsigned int group_id_old = (u8)((old->track_id & 0x00FF0000) >> 16); + unsigned int group_id_new = (u8)((new->track_id & 0x00FF0000) >> 16); + + /* 0x00 group must be only the first */ + if (group_id_new == 0) + return true; + /* 0xFF group is compatible with anything else */ + if (group_id_new == 0xFF || group_id_old == 0xFF) + return false; + /* otherwise only profiles from the same group are compatible*/ + return group_id_old != group_id_new; +} + +/** + * i40e_ddp_does_profiles_ - checks if DDP overlaps with existing one. + * @hw: HW data structure + * @pinfo: DDP profile information structure + * + * checks if DDP profile overlaps with existing one. + * Returns >0 if the profile overlaps. + * Returns 0 if the profile is ok. + * Returns <0 if error. + **/ +static int i40e_ddp_does_profile_overlap(struct i40e_hw *hw, + struct i40e_profile_info *pinfo) +{ + struct i40e_ddp_profile_list *profile_list; + u8 buff[I40E_PROFILE_LIST_SIZE]; + i40e_status status; + int i; + + status = i40e_aq_get_ddp_list(hw, buff, I40E_PROFILE_LIST_SIZE, 0, + NULL); + if (status) + return -EIO; + + profile_list = (struct i40e_ddp_profile_list *)buff; + for (i = 0; i < profile_list->p_count; i++) { + if (i40e_ddp_profiles_overlap(pinfo, + &profile_list->p_info[i])) + return 1; + } + return 0; +} + +/** + * i40e_add_pinfo + * @hw: pointer to the hardware structure + * @profile: pointer to the profile segment of the package + * @profile_info_sec: buffer for information section + * @track_id: package tracking id + * + * Register a profile to the list of loaded profiles. + */ +static enum i40e_status_code +i40e_add_pinfo(struct i40e_hw *hw, struct i40e_profile_segment *profile, + u8 *profile_info_sec, u32 track_id) +{ + struct i40e_profile_section_header *sec; + struct i40e_profile_info *pinfo; + i40e_status status; + u32 offset = 0, info = 0; + + sec = (struct i40e_profile_section_header *)profile_info_sec; + sec->tbl_size = 1; + sec->data_end = sizeof(struct i40e_profile_section_header) + + sizeof(struct i40e_profile_info); + sec->section.type = SECTION_TYPE_INFO; + sec->section.offset = sizeof(struct i40e_profile_section_header); + sec->section.size = sizeof(struct i40e_profile_info); + pinfo = (struct i40e_profile_info *)(profile_info_sec + + sec->section.offset); + pinfo->track_id = track_id; + pinfo->version = profile->version; + pinfo->op = I40E_DDP_ADD_TRACKID; + + /* Clear reserved field */ + memset(pinfo->reserved, 0, sizeof(pinfo->reserved)); + memcpy(pinfo->name, profile->name, I40E_DDP_NAME_SIZE); + + status = i40e_aq_write_ddp(hw, (void *)sec, sec->data_end, + track_id, &offset, &info, NULL); + return status; +} + +/** + * i40e_del_pinfo - delete DDP profile info from NIC + * @hw: HW data structure + * @profile: DDP profile segment to be deleted + * @profile_info_sec: DDP profile section header + * @track_id: track ID of the profile for deletion + * + * Removes DDP profile from the NIC. + **/ +static enum i40e_status_code +i40e_del_pinfo(struct i40e_hw *hw, struct i40e_profile_segment *profile, + u8 *profile_info_sec, u32 track_id) +{ + struct i40e_profile_section_header *sec; + struct i40e_profile_info *pinfo; + i40e_status status; + u32 offset = 0, info = 0; + + sec = (struct i40e_profile_section_header *)profile_info_sec; + sec->tbl_size = 1; + sec->data_end = sizeof(struct i40e_profile_section_header) + + sizeof(struct i40e_profile_info); + sec->section.type = SECTION_TYPE_INFO; + sec->section.offset = sizeof(struct i40e_profile_section_header); + sec->section.size = sizeof(struct i40e_profile_info); + pinfo = (struct i40e_profile_info *)(profile_info_sec + + sec->section.offset); + pinfo->track_id = track_id; + pinfo->version = profile->version; + pinfo->op = I40E_DDP_REMOVE_TRACKID; + + /* Clear reserved field */ + memset(pinfo->reserved, 0, sizeof(pinfo->reserved)); + memcpy(pinfo->name, profile->name, I40E_DDP_NAME_SIZE); + + status = i40e_aq_write_ddp(hw, (void *)sec, sec->data_end, + track_id, &offset, &info, NULL); + return status; +} + +/** + * i40e_ddp_is_pkg_hdr_valid - performs basic pkg header integrity checks + * @netdev: net device structure (for logging purposes) + * @pkg_hdr: pointer to package header + * @size_huge: size of the whole DDP profile package in size_t + * + * Checks correctness of pkg header: Version, size too big/small, and + * all segment offsets alignment and boundaries. This function lets + * reject non DDP profile file to be loaded by administrator mistake. + **/ +static bool i40e_ddp_is_pkg_hdr_valid(struct net_device *netdev, + struct i40e_package_header *pkg_hdr, + size_t size_huge) +{ + u32 size = 0xFFFFFFFFU & size_huge; + u32 pkg_hdr_size; + u32 segment; + + if (!pkg_hdr) + return false; + + if (pkg_hdr->version.major > 0) { + struct i40e_ddp_version ver = pkg_hdr->version; + + netdev_err(netdev, "Unsupported DDP profile version %u.%u.%u.%u", + ver.major, ver.minor, ver.update, ver.draft); + return false; + } + if (size_huge > size) { + netdev_err(netdev, "Invalid DDP profile - size is bigger than 4G"); + return false; + } + if (size < (sizeof(struct i40e_package_header) + + sizeof(struct i40e_metadata_segment) + sizeof(u32) * 2)) { + netdev_err(netdev, "Invalid DDP profile - size is too small."); + return false; + } + + pkg_hdr_size = sizeof(u32) * (pkg_hdr->segment_count + 2U); + if (size < pkg_hdr_size) { + netdev_err(netdev, "Invalid DDP profile - too many segments"); + return false; + } + for (segment = 0; segment < pkg_hdr->segment_count; ++segment) { + u32 offset = pkg_hdr->segment_offset[segment]; + + if (0xFU & offset) { + netdev_err(netdev, + "Invalid DDP profile %u segment alignment", + segment); + return false; + } + if (pkg_hdr_size > offset || offset >= size) { + netdev_err(netdev, + "Invalid DDP profile %u segment offset", + segment); + return false; + } + } + + return true; +} + +/** + * i40e_ddp_load - performs DDP loading + * @netdev: net device structure + * @data: buffer containing recipe file + * @size: size of the buffer + * @is_add: true when loading profile, false when rolling back the previous one + * + * Checks correctness and loads DDP profile to the NIC. The function is + * also used for rolling back previously loaded profile. + **/ +int i40e_ddp_load(struct net_device *netdev, const u8 *data, size_t size, + bool is_add) +{ + u8 profile_info_sec[sizeof(struct i40e_profile_section_header) + + sizeof(struct i40e_profile_info)]; + struct i40e_metadata_segment *metadata_hdr; + struct i40e_profile_segment *profile_hdr; + struct i40e_profile_info pinfo; + struct i40e_package_header *pkg_hdr; + i40e_status status; + struct i40e_netdev_priv *np = netdev_priv(netdev); + struct i40e_vsi *vsi = np->vsi; + struct i40e_pf *pf = vsi->back; + u32 track_id; + int istatus; + + pkg_hdr = (struct i40e_package_header *)data; + if (!i40e_ddp_is_pkg_hdr_valid(netdev, pkg_hdr, size)) + return -EINVAL; + + if (size < (sizeof(struct i40e_package_header) + + sizeof(struct i40e_metadata_segment) + sizeof(u32) * 2)) { + netdev_err(netdev, "Invalid DDP recipe size."); + return -EINVAL; + } + + /* Find beginning of segment data in buffer */ + metadata_hdr = (struct i40e_metadata_segment *) + i40e_find_segment_in_package(SEGMENT_TYPE_METADATA, pkg_hdr); + if (!metadata_hdr) { + netdev_err(netdev, "Failed to find metadata segment in DDP recipe."); + return -EINVAL; + } + + track_id = metadata_hdr->track_id; + profile_hdr = (struct i40e_profile_segment *) + i40e_find_segment_in_package(SEGMENT_TYPE_I40E, pkg_hdr); + if (!profile_hdr) { + netdev_err(netdev, "Failed to find profile segment in DDP recipe."); + return -EINVAL; + } + + pinfo.track_id = track_id; + pinfo.version = profile_hdr->version; + if (is_add) + pinfo.op = I40E_DDP_ADD_TRACKID; + else + pinfo.op = I40E_DDP_REMOVE_TRACKID; + + memcpy(pinfo.name, profile_hdr->name, I40E_DDP_NAME_SIZE); + + /* Check if profile data already exists*/ + istatus = i40e_ddp_does_profile_exist(&pf->hw, &pinfo); + if (istatus < 0) { + netdev_err(netdev, "Failed to fetch loaded profiles."); + return istatus; + } + if (is_add) { + if (istatus > 0) { + netdev_err(netdev, "DDP profile already loaded."); + return -EINVAL; + } + istatus = i40e_ddp_does_profile_overlap(&pf->hw, &pinfo); + if (istatus < 0) { + netdev_err(netdev, "Failed to fetch loaded profiles."); + return istatus; + } + if (istatus > 0) { + netdev_err(netdev, "DDP profile overlaps with existing one."); + return -EINVAL; + } + } else { + if (istatus == 0) { + netdev_err(netdev, + "DDP profile for deletion does not exist."); + return -EINVAL; + } + } + + /* Load profile data */ + if (is_add) { + status = i40e_write_profile(&pf->hw, profile_hdr, track_id); + if (status) { + if (status == I40E_ERR_DEVICE_NOT_SUPPORTED) { + netdev_err(netdev, + "Profile is not supported by the device."); + return -EPERM; + } + netdev_err(netdev, "Failed to write DDP profile."); + return -EIO; + } + } else { + status = i40e_rollback_profile(&pf->hw, profile_hdr, track_id); + if (status) { + netdev_err(netdev, "Failed to remove DDP profile."); + return -EIO; + } + } + + /* Add/remove profile to/from profile list in FW */ + if (is_add) { + status = i40e_add_pinfo(&pf->hw, profile_hdr, profile_info_sec, + track_id); + if (status) { + netdev_err(netdev, "Failed to add DDP profile info."); + return -EIO; + } + } else { + status = i40e_del_pinfo(&pf->hw, profile_hdr, profile_info_sec, + track_id); + if (status) { + netdev_err(netdev, "Failed to restore DDP profile info."); + return -EIO; + } + } + + return 0; +} + +/** + * i40e_ddp_restore - restore previously loaded profile and remove from list + * @pf: PF data struct + * + * Restores previously loaded profile stored on the list in driver memory. + * After rolling back removes entry from the list. + **/ +static int i40e_ddp_restore(struct i40e_pf *pf) +{ + struct i40e_ddp_old_profile_list *entry; + struct net_device *netdev = pf->vsi[pf->lan_vsi]->netdev; + int status = 0; + + if (!list_empty(&pf->ddp_old_prof)) { + entry = list_first_entry(&pf->ddp_old_prof, + struct i40e_ddp_old_profile_list, + list); + status = i40e_ddp_load(netdev, entry->old_ddp_buf, + entry->old_ddp_size, false); + list_del(&entry->list); + kfree(entry); + } + return status; +} + +/** + * i40e_ddp_flash - callback function for ethtool flash feature + * @netdev: net device structure + * @flash: kernel flash structure + * + * Ethtool callback function used for loading and unloading DDP profiles. + **/ +int i40e_ddp_flash(struct net_device *netdev, struct ethtool_flash *flash) +{ + const struct firmware *ddp_config; + struct i40e_netdev_priv *np = netdev_priv(netdev); + struct i40e_vsi *vsi = np->vsi; + struct i40e_pf *pf = vsi->back; + int status = 0; + + /* Check for valid region first */ + if (flash->region != I40_DDP_FLASH_REGION) { + netdev_err(netdev, "Requested firmware region is not recognized by this driver."); + return -EINVAL; + } + if (pf->hw.bus.func != 0) { + netdev_err(netdev, "Any DDP operation is allowed only on Phy0 NIC interface"); + return -EINVAL; + } + + /* If the user supplied "-" instead of file name rollback previously + * stored profile. + */ + if (strncmp(flash->data, "-", 2) != 0) { + struct i40e_ddp_old_profile_list *list_entry; + char profile_name[sizeof(I40E_DDP_PROFILE_PATH) + + I40E_DDP_PROFILE_NAME_MAX]; + + profile_name[sizeof(profile_name) - 1] = 0; + strncpy(profile_name, I40E_DDP_PROFILE_PATH, + sizeof(profile_name) - 1); + strncat(profile_name, flash->data, I40E_DDP_PROFILE_NAME_MAX); + /* Load DDP recipe. */ + status = request_firmware(&ddp_config, profile_name, + &netdev->dev); + if (status) { + netdev_err(netdev, "DDP recipe file request failed."); + return status; + } + + status = i40e_ddp_load(netdev, ddp_config->data, + ddp_config->size, true); + + if (!status) { + list_entry = + kzalloc(sizeof(struct i40e_ddp_old_profile_list) + + ddp_config->size, GFP_KERNEL); + if (!list_entry) { + netdev_info(netdev, "Failed to allocate memory for previous DDP profile data."); + netdev_info(netdev, "New profile loaded but roll-back will be impossible."); + } else { + memcpy(list_entry->old_ddp_buf, + ddp_config->data, ddp_config->size); + list_entry->old_ddp_size = ddp_config->size; + list_add(&list_entry->list, &pf->ddp_old_prof); + } + } + + release_firmware(ddp_config); + } else { + if (!list_empty(&pf->ddp_old_prof)) { + status = i40e_ddp_restore(pf); + } else { + netdev_warn(netdev, "There is no DDP profile to restore."); + status = -ENOENT; + } + } + return status; +} diff --git a/drivers/net/ethernet/intel/i40e/i40e_debugfs.c b/drivers/net/ethernet/intel/i40e/i40e_debugfs.c index c67d485d6f99a9ea0028b8b5b7b84450f61c1891..7ea4f09229e43cf395789afd793079a8b5c42677 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_debugfs.c +++ b/drivers/net/ethernet/intel/i40e/i40e_debugfs.c @@ -1321,7 +1321,7 @@ static ssize_t i40e_dbg_command_write(struct file *filp, if (strncmp(&cmd_buf[5], "stop", 4) == 0) { int ret; - ret = i40e_aq_stop_lldp(&pf->hw, false, NULL); + ret = i40e_aq_stop_lldp(&pf->hw, false, false, NULL); if (ret) { dev_info(&pf->pdev->dev, "Stop LLDP AQ command failed =0x%x\n", @@ -1358,7 +1358,7 @@ static ssize_t i40e_dbg_command_write(struct file *filp, /* Continue and start FW LLDP anyways */ } - ret = i40e_aq_start_lldp(&pf->hw, NULL); + ret = i40e_aq_start_lldp(&pf->hw, false, NULL); if (ret) { dev_info(&pf->pdev->dev, "Start LLDP AQ command failed =0x%x\n", diff --git a/drivers/net/ethernet/intel/i40e/i40e_devids.h b/drivers/net/ethernet/intel/i40e/i40e_devids.h index 334b05ff685ac937ecfde38afb41176bb66240ac..bac4da031f9bf813f7ceac5435840ad38c1b545e 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_devids.h +++ b/drivers/net/ethernet/intel/i40e/i40e_devids.h @@ -5,6 +5,8 @@ #define _I40E_DEVIDS_H_ /* Device IDs */ +#define I40E_DEV_ID_X710_N3000 0x0CF8 +#define I40E_DEV_ID_XXV710_N3000 0x0D58 #define I40E_DEV_ID_SFP_XL710 0x1572 #define I40E_DEV_ID_QEMU 0x1574 #define I40E_DEV_ID_KX_B 0x1580 @@ -18,6 +20,9 @@ #define I40E_DEV_ID_10G_BASE_T4 0x1589 #define I40E_DEV_ID_25G_B 0x158A #define I40E_DEV_ID_25G_SFP28 0x158B +#define I40E_DEV_ID_10G_BASE_T_BC 0x15FF +#define I40E_DEV_ID_10G_B 0x104F +#define I40E_DEV_ID_10G_SFP 0x104E #define I40E_DEV_ID_KX_X722 0x37CE #define I40E_DEV_ID_QSFP_X722 0x37CF #define I40E_DEV_ID_SFP_X722 0x37D0 diff --git a/drivers/net/ethernet/intel/i40e/i40e_ethtool.c b/drivers/net/ethernet/intel/i40e/i40e_ethtool.c index 7874d0ec7fb0e1a5dc4c42aeed03d7342e8f4fef..7545b21bee3c1cd8816f7aed59ab016d8f6c9b64 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_ethtool.c +++ b/drivers/net/ethernet/intel/i40e/i40e_ethtool.c @@ -508,6 +508,20 @@ static void i40e_phy_type_to_ethtool(struct i40e_pf *pf, ethtool_link_ksettings_add_link_mode(ks, advertising, 10000baseT_Full); } + if (phy_types & I40E_CAP_PHY_TYPE_2_5GBASE_T) { + ethtool_link_ksettings_add_link_mode(ks, supported, + 2500baseT_Full); + if (hw_link_info->requested_speeds & I40E_LINK_SPEED_2_5GB) + ethtool_link_ksettings_add_link_mode(ks, advertising, + 2500baseT_Full); + } + if (phy_types & I40E_CAP_PHY_TYPE_5GBASE_T) { + ethtool_link_ksettings_add_link_mode(ks, supported, + 5000baseT_Full); + if (hw_link_info->requested_speeds & I40E_LINK_SPEED_5GB) + ethtool_link_ksettings_add_link_mode(ks, advertising, + 5000baseT_Full); + } if (phy_types & I40E_CAP_PHY_TYPE_XLAUI || phy_types & I40E_CAP_PHY_TYPE_XLPPI || phy_types & I40E_CAP_PHY_TYPE_40GBASE_AOC) @@ -535,17 +549,23 @@ static void i40e_phy_type_to_ethtool(struct i40e_pf *pf, ethtool_link_ksettings_add_link_mode(ks, advertising, 1000baseT_Full); } - if (phy_types & I40E_CAP_PHY_TYPE_40GBASE_SR4) + if (phy_types & I40E_CAP_PHY_TYPE_40GBASE_SR4) { ethtool_link_ksettings_add_link_mode(ks, supported, 40000baseSR4_Full); - if (phy_types & I40E_CAP_PHY_TYPE_40GBASE_LR4) + ethtool_link_ksettings_add_link_mode(ks, advertising, + 40000baseSR4_Full); + } + if (phy_types & I40E_CAP_PHY_TYPE_40GBASE_LR4) { ethtool_link_ksettings_add_link_mode(ks, supported, 40000baseLR4_Full); + ethtool_link_ksettings_add_link_mode(ks, advertising, + 40000baseLR4_Full); + } if (phy_types & I40E_CAP_PHY_TYPE_40GBASE_KR4) { ethtool_link_ksettings_add_link_mode(ks, supported, - 40000baseLR4_Full); + 40000baseKR4_Full); ethtool_link_ksettings_add_link_mode(ks, advertising, - 40000baseLR4_Full); + 40000baseKR4_Full); } if (phy_types & I40E_CAP_PHY_TYPE_20GBASE_KR2) { ethtool_link_ksettings_add_link_mode(ks, supported, @@ -668,13 +688,15 @@ static void i40e_phy_type_to_ethtool(struct i40e_pf *pf, phy_types & I40E_CAP_PHY_TYPE_25GBASE_KR || phy_types & I40E_CAP_PHY_TYPE_25GBASE_CR || phy_types & I40E_CAP_PHY_TYPE_20GBASE_KR2 || - phy_types & I40E_CAP_PHY_TYPE_10GBASE_T || phy_types & I40E_CAP_PHY_TYPE_10GBASE_SR || phy_types & I40E_CAP_PHY_TYPE_10GBASE_LR || phy_types & I40E_CAP_PHY_TYPE_10GBASE_KX4 || phy_types & I40E_CAP_PHY_TYPE_10GBASE_KR || phy_types & I40E_CAP_PHY_TYPE_10GBASE_CR1_CU || phy_types & I40E_CAP_PHY_TYPE_10GBASE_CR1 || + phy_types & I40E_CAP_PHY_TYPE_10GBASE_T || + phy_types & I40E_CAP_PHY_TYPE_5GBASE_T || + phy_types & I40E_CAP_PHY_TYPE_2_5GBASE_T || phy_types & I40E_CAP_PHY_TYPE_1000BASE_T_OPTICAL || phy_types & I40E_CAP_PHY_TYPE_1000BASE_T || phy_types & I40E_CAP_PHY_TYPE_1000BASE_SX || @@ -720,14 +742,20 @@ static void i40e_get_settings_link_up(struct i40e_hw *hw, case I40E_PHY_TYPE_40GBASE_AOC: ethtool_link_ksettings_add_link_mode(ks, supported, 40000baseCR4_Full); + ethtool_link_ksettings_add_link_mode(ks, advertising, + 40000baseCR4_Full); break; case I40E_PHY_TYPE_40GBASE_SR4: ethtool_link_ksettings_add_link_mode(ks, supported, 40000baseSR4_Full); + ethtool_link_ksettings_add_link_mode(ks, advertising, + 40000baseSR4_Full); break; case I40E_PHY_TYPE_40GBASE_LR4: ethtool_link_ksettings_add_link_mode(ks, supported, 40000baseLR4_Full); + ethtool_link_ksettings_add_link_mode(ks, advertising, + 40000baseLR4_Full); break; case I40E_PHY_TYPE_25GBASE_SR: case I40E_PHY_TYPE_25GBASE_LR: @@ -778,11 +806,17 @@ static void i40e_get_settings_link_up(struct i40e_hw *hw, 10000baseT_Full); break; case I40E_PHY_TYPE_10GBASE_T: + case I40E_PHY_TYPE_5GBASE_T: + case I40E_PHY_TYPE_2_5GBASE_T: case I40E_PHY_TYPE_1000BASE_T: case I40E_PHY_TYPE_100BASE_TX: ethtool_link_ksettings_add_link_mode(ks, supported, Autoneg); ethtool_link_ksettings_add_link_mode(ks, supported, 10000baseT_Full); + ethtool_link_ksettings_add_link_mode(ks, supported, + 5000baseT_Full); + ethtool_link_ksettings_add_link_mode(ks, supported, + 2500baseT_Full); ethtool_link_ksettings_add_link_mode(ks, supported, 1000baseT_Full); ethtool_link_ksettings_add_link_mode(ks, supported, @@ -791,6 +825,12 @@ static void i40e_get_settings_link_up(struct i40e_hw *hw, if (hw_link_info->requested_speeds & I40E_LINK_SPEED_10GB) ethtool_link_ksettings_add_link_mode(ks, advertising, 10000baseT_Full); + if (hw_link_info->requested_speeds & I40E_LINK_SPEED_5GB) + ethtool_link_ksettings_add_link_mode(ks, advertising, + 5000baseT_Full); + if (hw_link_info->requested_speeds & I40E_LINK_SPEED_2_5GB) + ethtool_link_ksettings_add_link_mode(ks, advertising, + 2500baseT_Full); if (hw_link_info->requested_speeds & I40E_LINK_SPEED_1GB) ethtool_link_ksettings_add_link_mode(ks, advertising, 1000baseT_Full); @@ -946,6 +986,12 @@ static void i40e_get_settings_link_up(struct i40e_hw *hw, case I40E_LINK_SPEED_10GB: ks->base.speed = SPEED_10000; break; + case I40E_LINK_SPEED_5GB: + ks->base.speed = SPEED_5000; + break; + case I40E_LINK_SPEED_2_5GB: + ks->base.speed = SPEED_2500; + break; case I40E_LINK_SPEED_1GB: ks->base.speed = SPEED_1000; break; @@ -1033,6 +1079,7 @@ static int i40e_get_link_ksettings(struct net_device *netdev, break; case I40E_MEDIA_TYPE_FIBER: ethtool_link_ksettings_add_link_mode(ks, supported, FIBRE); + ethtool_link_ksettings_add_link_mode(ks, advertising, FIBRE); ks->base.port = PORT_FIBRE; break; case I40E_MEDIA_TYPE_UNKNOWN: @@ -1230,6 +1277,12 @@ static int i40e_set_link_ksettings(struct net_device *netdev, ethtool_link_ksettings_test_link_mode(ks, advertising, 10000baseLR_Full)) config.link_speed |= I40E_LINK_SPEED_10GB; + if (ethtool_link_ksettings_test_link_mode(ks, advertising, + 2500baseT_Full)) + config.link_speed |= I40E_LINK_SPEED_2_5GB; + if (ethtool_link_ksettings_test_link_mode(ks, advertising, + 5000baseT_Full)) + config.link_speed |= I40E_LINK_SPEED_5GB; if (ethtool_link_ksettings_test_link_mode(ks, advertising, 20000baseKR2_Full)) config.link_speed |= I40E_LINK_SPEED_20GB; @@ -4945,7 +4998,7 @@ flags_complete: if (pf->flags & I40E_FLAG_DISABLE_FW_LLDP) { struct i40e_dcbx_config *dcbcfg; - i40e_aq_stop_lldp(&pf->hw, true, NULL); + i40e_aq_stop_lldp(&pf->hw, true, false, NULL); i40e_aq_set_dcb_parameters(&pf->hw, true, NULL); /* reset local_dcbx_config to default */ dcbcfg = &pf->hw.local_dcbx_config; @@ -4960,7 +5013,7 @@ flags_complete: dcbcfg->pfc.willing = 1; dcbcfg->pfc.pfccap = I40E_MAX_TRAFFIC_CLASS; } else { - i40e_aq_start_lldp(&pf->hw, NULL); + i40e_aq_start_lldp(&pf->hw, false, NULL); } } @@ -5128,6 +5181,12 @@ static int i40e_get_module_eeprom(struct net_device *netdev, return 0; } +static const struct ethtool_ops i40e_ethtool_recovery_mode_ops = { + .set_eeprom = i40e_set_eeprom, + .get_eeprom_len = i40e_get_eeprom_len, + .get_eeprom = i40e_get_eeprom, +}; + static const struct ethtool_ops i40e_ethtool_ops = { .get_drvinfo = i40e_get_drvinfo, .get_regs_len = i40e_get_regs_len, @@ -5171,9 +5230,16 @@ static const struct ethtool_ops i40e_ethtool_ops = { .set_link_ksettings = i40e_set_link_ksettings, .get_fecparam = i40e_get_fec_param, .set_fecparam = i40e_set_fec_param, + .flash_device = i40e_ddp_flash, }; void i40e_set_ethtool_ops(struct net_device *netdev) { - netdev->ethtool_ops = &i40e_ethtool_ops; + struct i40e_netdev_priv *np = netdev_priv(netdev); + struct i40e_pf *pf = np->vsi->back; + + if (!test_bit(__I40E_RECOVERY_MODE, pf->state)) + netdev->ethtool_ops = &i40e_ethtool_ops; + else + netdev->ethtool_ops = &i40e_ethtool_recovery_mode_ops; } diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c b/drivers/net/ethernet/intel/i40e/i40e_main.c index b1c265012c8ad03d65a6eb56e1a0d5e42371c6be..320562b39686e9296c83ea382b268ef14b4b401a 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_main.c +++ b/drivers/net/ethernet/intel/i40e/i40e_main.c @@ -27,7 +27,7 @@ static const char i40e_driver_string[] = #define DRV_VERSION_MAJOR 2 #define DRV_VERSION_MINOR 8 -#define DRV_VERSION_BUILD 10 +#define DRV_VERSION_BUILD 20 #define DRV_VERSION __stringify(DRV_VERSION_MAJOR) "." \ __stringify(DRV_VERSION_MINOR) "." \ __stringify(DRV_VERSION_BUILD) DRV_KERN @@ -46,6 +46,10 @@ static int i40e_setup_pf_filter_control(struct i40e_pf *pf); static void i40e_prep_for_reset(struct i40e_pf *pf, bool lock_acquired); static int i40e_reset(struct i40e_pf *pf); static void i40e_rebuild(struct i40e_pf *pf, bool reinit, bool lock_acquired); +static int i40e_setup_misc_vector_for_recovery_mode(struct i40e_pf *pf); +static int i40e_restore_interrupt_scheme(struct i40e_pf *pf); +static bool i40e_check_recovery_mode(struct i40e_pf *pf); +static int i40e_init_recovery_mode(struct i40e_pf *pf, struct i40e_hw *hw); static void i40e_fdir_sb_setup(struct i40e_pf *pf); static int i40e_veb_get_bw_info(struct i40e_veb *veb); static int i40e_get_capabilities(struct i40e_pf *pf, @@ -69,6 +73,8 @@ static const struct pci_device_id i40e_pci_tbl[] = { {PCI_VDEVICE(INTEL, I40E_DEV_ID_QSFP_C), 0}, {PCI_VDEVICE(INTEL, I40E_DEV_ID_10G_BASE_T), 0}, {PCI_VDEVICE(INTEL, I40E_DEV_ID_10G_BASE_T4), 0}, + {PCI_VDEVICE(INTEL, I40E_DEV_ID_10G_SFP), 0}, + {PCI_VDEVICE(INTEL, I40E_DEV_ID_10G_B), 0}, {PCI_VDEVICE(INTEL, I40E_DEV_ID_KX_X722), 0}, {PCI_VDEVICE(INTEL, I40E_DEV_ID_QSFP_X722), 0}, {PCI_VDEVICE(INTEL, I40E_DEV_ID_SFP_X722), 0}, @@ -77,6 +83,8 @@ static const struct pci_device_id i40e_pci_tbl[] = { {PCI_VDEVICE(INTEL, I40E_DEV_ID_SFP_I_X722), 0}, {PCI_VDEVICE(INTEL, I40E_DEV_ID_20G_KR2), 0}, {PCI_VDEVICE(INTEL, I40E_DEV_ID_20G_KR2_A), 0}, + {PCI_VDEVICE(INTEL, I40E_DEV_ID_X710_N3000), 0}, + {PCI_VDEVICE(INTEL, I40E_DEV_ID_XXV710_N3000), 0}, {PCI_VDEVICE(INTEL, I40E_DEV_ID_25G_B), 0}, {PCI_VDEVICE(INTEL, I40E_DEV_ID_25G_SFP28), 0}, /* required last entry */ @@ -278,8 +286,9 @@ struct i40e_vsi *i40e_find_vsi_from_id(struct i40e_pf *pf, u16 id) **/ void i40e_service_event_schedule(struct i40e_pf *pf) { - if (!test_bit(__I40E_DOWN, pf->state) && - !test_bit(__I40E_RESET_RECOVERY_PENDING, pf->state)) + if ((!test_bit(__I40E_DOWN, pf->state) && + !test_bit(__I40E_RESET_RECOVERY_PENDING, pf->state)) || + test_bit(__I40E_RECOVERY_MODE, pf->state)) queue_work(i40e_wq, &pf->service_task); } @@ -2107,11 +2116,22 @@ void i40e_aqc_add_filters(struct i40e_vsi *vsi, const char *vsi_name, fcnt = i40e_update_filter_state(num_add, list, add_head); if (fcnt != num_add) { - set_bit(__I40E_VSI_OVERFLOW_PROMISC, vsi->state); - dev_warn(&vsi->back->pdev->dev, - "Error %s adding RX filters on %s, promiscuous mode forced on\n", - i40e_aq_str(hw, aq_err), - vsi_name); + if (vsi->type == I40E_VSI_MAIN) { + set_bit(__I40E_VSI_OVERFLOW_PROMISC, vsi->state); + dev_warn(&vsi->back->pdev->dev, + "Error %s adding RX filters on %s, promiscuous mode forced on\n", + i40e_aq_str(hw, aq_err), vsi_name); + } else if (vsi->type == I40E_VSI_SRIOV || + vsi->type == I40E_VSI_VMDQ1 || + vsi->type == I40E_VSI_VMDQ2) { + dev_warn(&vsi->back->pdev->dev, + "Error %s adding RX filters on %s, please set promiscuous on manually for %s\n", + i40e_aq_str(hw, aq_err), vsi_name, vsi_name); + } else { + dev_warn(&vsi->back->pdev->dev, + "Error %s adding RX filters on %s, incorrect VSI type: %i.\n", + i40e_aq_str(hw, aq_err), vsi_name, vsi->type); + } } } @@ -2654,6 +2674,10 @@ void i40e_vlan_stripping_enable(struct i40e_vsi *vsi) struct i40e_vsi_context ctxt; i40e_status ret; + /* Don't modify stripping options if a port VLAN is active */ + if (vsi->info.pvid) + return; + if ((vsi->info.valid_sections & cpu_to_le16(I40E_AQ_VSI_PROP_VLAN_VALID)) && ((vsi->info.port_vlan_flags & I40E_AQ_VSI_PVLAN_MODE_MASK) == 0)) @@ -2684,6 +2708,10 @@ void i40e_vlan_stripping_disable(struct i40e_vsi *vsi) struct i40e_vsi_context ctxt; i40e_status ret; + /* Don't modify stripping options if a port VLAN is active */ + if (vsi->info.pvid) + return; + if ((vsi->info.valid_sections & cpu_to_le16(I40E_AQ_VSI_PROP_VLAN_VALID)) && ((vsi->info.port_vlan_flags & I40E_AQ_VSI_PVLAN_EMOD_MASK) == @@ -2949,9 +2977,9 @@ int i40e_vsi_add_pvid(struct i40e_vsi *vsi, u16 vid) **/ void i40e_vsi_remove_pvid(struct i40e_vsi *vsi) { - i40e_vlan_stripping_disable(vsi); - vsi->info.pvid = 0; + + i40e_vlan_stripping_disable(vsi); } /** @@ -4000,7 +4028,8 @@ static irqreturn_t i40e_intr(int irq, void *data) enable_intr: /* re-enable interrupt causes */ wr32(hw, I40E_PFINT_ICR0_ENA, ena_mask); - if (!test_bit(__I40E_DOWN, pf->state)) { + if (!test_bit(__I40E_DOWN, pf->state) || + test_bit(__I40E_RECOVERY_MODE, pf->state)) { i40e_service_event_schedule(pf); i40e_irq_dynamic_enable_icr0(pf); } @@ -6403,7 +6432,7 @@ static int i40e_init_pf_dcb(struct i40e_pf *pf) goto out; /* Get the initial DCB configuration */ - err = i40e_init_dcb(hw); + err = i40e_init_dcb(hw, true); if (!err) { /* Device/Function is not DCBX capable */ if ((!hw->func_caps.dcb) || @@ -6493,6 +6522,12 @@ void i40e_print_link_message(struct i40e_vsi *vsi, bool isup) case I40E_LINK_SPEED_10GB: speed = "10 G"; break; + case I40E_LINK_SPEED_5GB: + speed = "5 G"; + break; + case I40E_LINK_SPEED_2_5GB: + speed = "2.5 G"; + break; case I40E_LINK_SPEED_1GB: speed = "1000 M"; break; @@ -6846,10 +6881,12 @@ static int i40e_setup_tc(struct net_device *netdev, void *type_data) struct i40e_pf *pf = vsi->back; u8 enabled_tc = 0, num_tc, hw; bool need_reset = false; + int old_queue_pairs; int ret = -EINVAL; u16 mode; int i; + old_queue_pairs = vsi->num_queue_pairs; num_tc = mqprio_qopt->qopt.num_tc; hw = mqprio_qopt->qopt.hw; mode = mqprio_qopt->mode; @@ -6950,6 +6987,7 @@ config_tc: } ret = i40e_configure_queue_channels(vsi); if (ret) { + vsi->num_queue_pairs = old_queue_pairs; netdev_info(netdev, "Failed configuring queue channels\n"); need_reset = true; @@ -9290,6 +9328,11 @@ static void i40e_prep_for_reset(struct i40e_pf *pf, bool lock_acquired) dev_warn(&pf->pdev->dev, "shutdown_lan_hmc failed: %d\n", ret); } + + /* Save the current PTP time so that we can restore the time after the + * reset completes. + */ + i40e_ptp_save_hw_time(pf); } /** @@ -9382,6 +9425,7 @@ static int i40e_reset(struct i40e_pf *pf) **/ static void i40e_rebuild(struct i40e_pf *pf, bool reinit, bool lock_acquired) { + int old_recovery_mode_bit = test_bit(__I40E_RECOVERY_MODE, pf->state); struct i40e_vsi *vsi = pf->vsi[pf->lan_vsi]; struct i40e_hw *hw = &pf->hw; u8 set_fc_aq_fail = 0; @@ -9389,7 +9433,14 @@ static void i40e_rebuild(struct i40e_pf *pf, bool reinit, bool lock_acquired) u32 val; int v; - if (test_bit(__I40E_DOWN, pf->state)) + if (test_bit(__I40E_EMP_RESET_INTR_RECEIVED, pf->state) && + i40e_check_recovery_mode(pf)) { + i40e_set_ethtool_ops(pf->vsi[pf->lan_vsi]->netdev); + } + + if (test_bit(__I40E_DOWN, pf->state) && + !test_bit(__I40E_RECOVERY_MODE, pf->state) && + !old_recovery_mode_bit) goto clear_recovery; dev_dbg(&pf->pdev->dev, "Rebuilding internal switch\n"); @@ -9418,6 +9469,44 @@ static void i40e_rebuild(struct i40e_pf *pf, bool reinit, bool lock_acquired) if (test_and_clear_bit(__I40E_EMP_RESET_INTR_RECEIVED, pf->state)) i40e_verify_eeprom(pf); + /* if we are going out of or into recovery mode we have to act + * accordingly with regard to resources initialization + * and deinitialization + */ + if (test_bit(__I40E_RECOVERY_MODE, pf->state) || + old_recovery_mode_bit) { + if (i40e_get_capabilities(pf, + i40e_aqc_opc_list_func_capabilities)) + goto end_unlock; + + if (test_bit(__I40E_RECOVERY_MODE, pf->state)) { + /* we're staying in recovery mode so we'll reinitialize + * misc vector here + */ + if (i40e_setup_misc_vector_for_recovery_mode(pf)) + goto end_unlock; + } else { + if (!lock_acquired) + rtnl_lock(); + /* we're going out of recovery mode so we'll free + * the IRQ allocated specifically for recovery mode + * and restore the interrupt scheme + */ + free_irq(pf->pdev->irq, pf); + i40e_clear_interrupt_scheme(pf); + if (i40e_restore_interrupt_scheme(pf)) + goto end_unlock; + } + + /* tell the firmware that we're starting */ + i40e_send_version(pf); + + /* bail out in case recovery mode was detected, as there is + * no need for further configuration. + */ + goto end_unlock; + } + i40e_clear_pxe_mode(hw); ret = i40e_get_capabilities(pf, i40e_aqc_opc_list_func_capabilities); if (ret) @@ -9669,7 +9758,6 @@ static void i40e_handle_mdd_event(struct i40e_pf *pf) { struct i40e_hw *hw = &pf->hw; bool mdd_detected = false; - bool pf_mdd_detected = false; struct i40e_vf *vf; u32 reg; int i; @@ -9715,19 +9803,12 @@ static void i40e_handle_mdd_event(struct i40e_pf *pf) reg = rd32(hw, I40E_PF_MDET_TX); if (reg & I40E_PF_MDET_TX_VALID_MASK) { wr32(hw, I40E_PF_MDET_TX, 0xFFFF); - dev_info(&pf->pdev->dev, "TX driver issue detected, PF reset issued\n"); - pf_mdd_detected = true; + dev_dbg(&pf->pdev->dev, "TX driver issue detected on PF\n"); } reg = rd32(hw, I40E_PF_MDET_RX); if (reg & I40E_PF_MDET_RX_VALID_MASK) { wr32(hw, I40E_PF_MDET_RX, 0xFFFF); - dev_info(&pf->pdev->dev, "RX driver issue detected, PF reset issued\n"); - pf_mdd_detected = true; - } - /* Queue belongs to the PF, initiate a reset */ - if (pf_mdd_detected) { - set_bit(__I40E_PF_RESET_REQUESTED, pf->state); - i40e_service_event_schedule(pf); + dev_dbg(&pf->pdev->dev, "RX driver issue detected on PF\n"); } } @@ -9740,6 +9821,9 @@ static void i40e_handle_mdd_event(struct i40e_pf *pf) vf->num_mdd_events++; dev_info(&pf->pdev->dev, "TX driver issue detected on VF %d\n", i); + dev_info(&pf->pdev->dev, + "Use PF Control I/F to re-enable the VF\n"); + set_bit(I40E_VF_STATE_DISABLED, &vf->vf_states); } reg = rd32(hw, I40E_VP_MDET_RX(i)); @@ -9748,11 +9832,6 @@ static void i40e_handle_mdd_event(struct i40e_pf *pf) vf->num_mdd_events++; dev_info(&pf->pdev->dev, "RX driver issue detected on VF %d\n", i); - } - - if (vf->num_mdd_events > I40E_DEFAULT_NUM_MDD_EVENTS_ALLOWED) { - dev_info(&pf->pdev->dev, - "Too many MDD events on VF %d, disabled\n", i); dev_info(&pf->pdev->dev, "Use PF Control I/F to re-enable the VF\n"); set_bit(I40E_VF_STATE_DISABLED, &vf->vf_states); @@ -9879,31 +9958,38 @@ static void i40e_service_task(struct work_struct *work) unsigned long start_time = jiffies; /* don't bother with service tasks if a reset is in progress */ - if (test_bit(__I40E_RESET_RECOVERY_PENDING, pf->state)) + if (test_bit(__I40E_RESET_RECOVERY_PENDING, pf->state) || + test_bit(__I40E_SUSPENDED, pf->state)) return; if (test_and_set_bit(__I40E_SERVICE_SCHED, pf->state)) return; - i40e_detect_recover_hung(pf->vsi[pf->lan_vsi]); - i40e_sync_filters_subtask(pf); - i40e_reset_subtask(pf); - i40e_handle_mdd_event(pf); - i40e_vc_process_vflr_event(pf); - i40e_watchdog_subtask(pf); - i40e_fdir_reinit_subtask(pf); - if (test_and_clear_bit(__I40E_CLIENT_RESET, pf->state)) { - /* Client subtask will reopen next time through. */ - i40e_notify_client_of_netdev_close(pf->vsi[pf->lan_vsi], true); + if (!test_bit(__I40E_RECOVERY_MODE, pf->state)) { + i40e_detect_recover_hung(pf->vsi[pf->lan_vsi]); + i40e_sync_filters_subtask(pf); + i40e_reset_subtask(pf); + i40e_handle_mdd_event(pf); + i40e_vc_process_vflr_event(pf); + i40e_watchdog_subtask(pf); + i40e_fdir_reinit_subtask(pf); + if (test_and_clear_bit(__I40E_CLIENT_RESET, pf->state)) { + /* Client subtask will reopen next time through. */ + i40e_notify_client_of_netdev_close(pf->vsi[pf->lan_vsi], + true); + } else { + i40e_client_subtask(pf); + if (test_and_clear_bit(__I40E_CLIENT_L2_CHANGE, + pf->state)) + i40e_notify_client_of_l2_param_changes( + pf->vsi[pf->lan_vsi]); + } + i40e_sync_filters_subtask(pf); + i40e_sync_udp_filters_subtask(pf); } else { - i40e_client_subtask(pf); - if (test_and_clear_bit(__I40E_CLIENT_L2_CHANGE, - pf->state)) - i40e_notify_client_of_l2_param_changes( - pf->vsi[pf->lan_vsi]); - } - i40e_sync_filters_subtask(pf); - i40e_sync_udp_filters_subtask(pf); + i40e_reset_subtask(pf); + } + i40e_clean_adminq_subtask(pf); /* flush memory to make sure state is correct before next watchdog */ @@ -10725,6 +10811,48 @@ err_unwind: return err; } +/** + * i40e_setup_misc_vector_for_recovery_mode - Setup the misc vector to handle + * non queue events in recovery mode + * @pf: board private structure + * + * This sets up the handler for MSIX 0 or MSI/legacy, which is used to manage + * the non-queue interrupts, e.g. AdminQ and errors in recovery mode. + * This is handled differently than in recovery mode since no Tx/Rx resources + * are being allocated. + **/ +static int i40e_setup_misc_vector_for_recovery_mode(struct i40e_pf *pf) +{ + int err; + + if (pf->flags & I40E_FLAG_MSIX_ENABLED) { + err = i40e_setup_misc_vector(pf); + + if (err) { + dev_info(&pf->pdev->dev, + "MSI-X misc vector request failed, error %d\n", + err); + return err; + } + } else { + u32 flags = pf->flags & I40E_FLAG_MSI_ENABLED ? 0 : IRQF_SHARED; + + err = request_irq(pf->pdev->irq, i40e_intr, flags, + pf->int_name, pf); + + if (err) { + dev_info(&pf->pdev->dev, + "MSI/legacy misc vector request failed, error %d\n", + err); + return err; + } + i40e_enable_misc_int_causes(pf); + i40e_irq_dynamic_enable_icr0(pf); + } + + return 0; +} + /** * i40e_setup_misc_vector - Setup the misc vector to handle non queue events * @pf: board private structure @@ -13887,6 +14015,125 @@ void i40e_set_fec_in_flags(u8 fec_cfg, u32 *flags) *flags &= ~(I40E_FLAG_RS_FEC | I40E_FLAG_BASE_R_FEC); } +/** + * i40e_check_recovery_mode - check if we are running transition firmware + * @pf: board private structure + * + * Check registers indicating the firmware runs in recovery mode. Sets the + * appropriate driver state. + * + * Returns true if the recovery mode was detected, false otherwise + **/ +static bool i40e_check_recovery_mode(struct i40e_pf *pf) +{ + u32 val = rd32(&pf->hw, I40E_GL_FWSTS); + + if (val & I40E_GL_FWSTS_FWS1B_MASK) { + dev_notice(&pf->pdev->dev, "Firmware recovery mode detected. Limiting functionality.\n"); + dev_notice(&pf->pdev->dev, "Refer to the Intel(R) Ethernet Adapters and Devices User Guide for details on firmware recovery mode.\n"); + set_bit(__I40E_RECOVERY_MODE, pf->state); + + return true; + } + if (test_and_clear_bit(__I40E_RECOVERY_MODE, pf->state)) + dev_info(&pf->pdev->dev, "Reinitializing in normal mode with full functionality.\n"); + + return false; +} + +/** + * i40e_init_recovery_mode - initialize subsystems needed in recovery mode + * @pf: board private structure + * @hw: ptr to the hardware info + * + * This function does a minimal setup of all subsystems needed for running + * recovery mode. + * + * Returns 0 on success, negative on failure + **/ +static int i40e_init_recovery_mode(struct i40e_pf *pf, struct i40e_hw *hw) +{ + struct i40e_vsi *vsi; + int err; + int v_idx; + + pci_save_state(pf->pdev); + + /* set up periodic task facility */ + timer_setup(&pf->service_timer, i40e_service_timer, 0); + pf->service_timer_period = HZ; + + INIT_WORK(&pf->service_task, i40e_service_task); + clear_bit(__I40E_SERVICE_SCHED, pf->state); + + err = i40e_init_interrupt_scheme(pf); + if (err) + goto err_switch_setup; + + /* The number of VSIs reported by the FW is the minimum guaranteed + * to us; HW supports far more and we share the remaining pool with + * the other PFs. We allocate space for more than the guarantee with + * the understanding that we might not get them all later. + */ + if (pf->hw.func_caps.num_vsis < I40E_MIN_VSI_ALLOC) + pf->num_alloc_vsi = I40E_MIN_VSI_ALLOC; + else + pf->num_alloc_vsi = pf->hw.func_caps.num_vsis; + + /* Set up the vsi struct and our local tracking of the MAIN PF vsi. */ + pf->vsi = kcalloc(pf->num_alloc_vsi, sizeof(struct i40e_vsi *), + GFP_KERNEL); + if (!pf->vsi) { + err = -ENOMEM; + goto err_switch_setup; + } + + /* We allocate one VSI which is needed as absolute minimum + * in order to register the netdev + */ + v_idx = i40e_vsi_mem_alloc(pf, I40E_VSI_MAIN); + if (v_idx < 0) + goto err_switch_setup; + pf->lan_vsi = v_idx; + vsi = pf->vsi[v_idx]; + if (!vsi) + goto err_switch_setup; + vsi->alloc_queue_pairs = 1; + err = i40e_config_netdev(vsi); + if (err) + goto err_switch_setup; + err = register_netdev(vsi->netdev); + if (err) + goto err_switch_setup; + vsi->netdev_registered = true; + i40e_dbg_pf_init(pf); + + err = i40e_setup_misc_vector_for_recovery_mode(pf); + if (err) + goto err_switch_setup; + + /* tell the firmware that we're starting */ + i40e_send_version(pf); + + /* since everything's happy, start the service_task timer */ + mod_timer(&pf->service_timer, + round_jiffies(jiffies + pf->service_timer_period)); + + return 0; + +err_switch_setup: + i40e_reset_interrupt_capability(pf); + del_timer_sync(&pf->service_timer); + i40e_shutdown_adminq(hw); + iounmap(hw->hw_addr); + pci_disable_pcie_error_reporting(pf->pdev); + pci_release_mem_regions(pf->pdev); + pci_disable_device(pf->pdev); + kfree(pf); + + return err; +} + /** * i40e_probe - Device initialization routine * @pdev: PCI device information struct @@ -13984,6 +14231,7 @@ static int i40e_probe(struct pci_dev *pdev, const struct pci_device_id *ent) INIT_LIST_HEAD(&pf->l3_flex_pit_list); INIT_LIST_HEAD(&pf->l4_flex_pit_list); + INIT_LIST_HEAD(&pf->ddp_old_prof); /* set up the locks for the AQ, do this only once in probe * and destroy them only once in remove @@ -14011,13 +14259,14 @@ static int i40e_probe(struct pci_dev *pdev, const struct pci_device_id *ent) /* Reset here to make sure all is clean and to define PF 'n' */ i40e_clear_hw(hw); - err = i40e_pf_reset(hw); - if (err) { - dev_info(&pdev->dev, "Initial pf_reset failed: %d\n", err); - goto err_pf_reset; + if (!i40e_check_recovery_mode(pf)) { + err = i40e_pf_reset(hw); + if (err) { + dev_info(&pdev->dev, "Initial pf_reset failed: %d\n", err); + goto err_pf_reset; + } + pf->pfr_count++; } - pf->pfr_count++; - hw->aq.num_arq_entries = I40E_AQ_LEN; hw->aq.num_asq_entries = I40E_AQ_LEN; hw->aq.arq_buf_size = I40E_MAX_AQ_BUF_SIZE; @@ -14042,7 +14291,11 @@ static int i40e_probe(struct pci_dev *pdev, const struct pci_device_id *ent) if (err) { if (err == I40E_ERR_FIRMWARE_API_VERSION) dev_info(&pdev->dev, - "The driver for the device stopped because the NVM image is newer than expected. You must install the most recent version of the network driver.\n"); + "The driver for the device stopped because the NVM image v%u.%u is newer than expected v%u.%u. You must install the most recent version of the network driver.\n", + hw->aq.api_maj_ver, + hw->aq.api_min_ver, + I40E_FW_API_VERSION_MAJOR, + I40E_FW_MINOR_VERSION(hw)); else dev_info(&pdev->dev, "The driver for the device stopped because the device firmware failed to init. Try updating your NVM image.\n"); @@ -14051,19 +14304,28 @@ static int i40e_probe(struct pci_dev *pdev, const struct pci_device_id *ent) } i40e_get_oem_version(hw); - /* provide nvm, fw, api versions */ - dev_info(&pdev->dev, "fw %d.%d.%05d api %d.%d nvm %s\n", + /* provide nvm, fw, api versions, vendor:device id, subsys vendor:device id */ + dev_info(&pdev->dev, "fw %d.%d.%05d api %d.%d nvm %s [%04x:%04x] [%04x:%04x]\n", hw->aq.fw_maj_ver, hw->aq.fw_min_ver, hw->aq.fw_build, hw->aq.api_maj_ver, hw->aq.api_min_ver, - i40e_nvm_version_str(hw)); + i40e_nvm_version_str(hw), hw->vendor_id, hw->device_id, + hw->subsystem_vendor_id, hw->subsystem_device_id); if (hw->aq.api_maj_ver == I40E_FW_API_VERSION_MAJOR && hw->aq.api_min_ver > I40E_FW_MINOR_VERSION(hw)) dev_info(&pdev->dev, - "The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.\n"); + "The driver for the device detected a newer version of the NVM image v%u.%u than expected v%u.%u. Please install the most recent version of the network driver.\n", + hw->aq.api_maj_ver, + hw->aq.api_min_ver, + I40E_FW_API_VERSION_MAJOR, + I40E_FW_MINOR_VERSION(hw)); else if (hw->aq.api_maj_ver == 1 && hw->aq.api_min_ver < 4) dev_info(&pdev->dev, - "The driver for the device detected an older version of the NVM image than expected. Please update the NVM image.\n"); + "The driver for the device detected an older version of the NVM image v%u.%u than expected v%u.%u. Please update the NVM image.\n", + hw->aq.api_maj_ver, + hw->aq.api_min_ver, + I40E_FW_API_VERSION_MAJOR, + I40E_FW_MINOR_VERSION(hw)); i40e_verify_eeprom(pf); @@ -14072,6 +14334,7 @@ static int i40e_probe(struct pci_dev *pdev, const struct pci_device_id *ent) dev_warn(&pdev->dev, "This device is a pre-production adapter/LOM. Please be aware there may be issues with your hardware. If you are experiencing problems please contact your Intel or hardware representative who provided you with this hardware.\n"); i40e_clear_pxe_mode(hw); + err = i40e_get_capabilities(pf, i40e_aqc_opc_list_func_capabilities); if (err) goto err_adminq_setup; @@ -14082,6 +14345,9 @@ static int i40e_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto err_sw_init; } + if (test_bit(__I40E_RECOVERY_MODE, pf->state)) + return i40e_init_recovery_mode(pf, hw); + err = i40e_init_lan_hmc(hw, hw->func_caps.num_tx_qp, hw->func_caps.num_rx_qp, 0, 0); if (err) { @@ -14102,7 +14368,7 @@ static int i40e_probe(struct pci_dev *pdev, const struct pci_device_id *ent) */ if (pf->hw_features & I40E_HW_STOP_FW_LLDP) { dev_info(&pdev->dev, "Stopping firmware LLDP agent.\n"); - i40e_aq_stop_lldp(hw, true, NULL); + i40e_aq_stop_lldp(hw, true, false, NULL); } /* allow a platform config to override the HW addr */ @@ -14467,6 +14733,19 @@ static void i40e_remove(struct pci_dev *pdev) if (pf->service_task.func) cancel_work_sync(&pf->service_task); + if (test_bit(__I40E_RECOVERY_MODE, pf->state)) { + struct i40e_vsi *vsi = pf->vsi[0]; + + /* We know that we have allocated only one vsi for this PF, + * it was just for registering netdevice, so the interface + * could be visible in the 'ifconfig' output + */ + unregister_netdev(vsi->netdev); + free_netdev(vsi->netdev); + + goto unmap; + } + /* Client close must be called explicitly here because the timer * has been stopped. */ @@ -14516,6 +14795,12 @@ static void i40e_remove(struct pci_dev *pdev) ret_code); } +unmap: + /* Free MSI/legacy interrupt 0 when in recovery mode. */ + if (test_bit(__I40E_RECOVERY_MODE, pf->state) && + !(pf->flags & I40E_FLAG_MSIX_ENABLED)) + free_irq(pf->pdev->irq, pf); + /* shutdown the adminq */ i40e_shutdown_adminq(hw); @@ -14528,7 +14813,8 @@ static void i40e_remove(struct pci_dev *pdev) i40e_clear_interrupt_scheme(pf); for (i = 0; i < pf->num_alloc_vsi; i++) { if (pf->vsi[i]) { - i40e_vsi_clear_rings(pf->vsi[i]); + if (!test_bit(__I40E_RECOVERY_MODE, pf->state)) + i40e_vsi_clear_rings(pf->vsi[i]); i40e_vsi_clear(pf->vsi[i]); pf->vsi[i] = NULL; } @@ -14736,6 +15022,11 @@ static void i40e_shutdown(struct pci_dev *pdev) wr32(hw, I40E_PFPM_WUFC, (pf->wol_en ? I40E_PFPM_WUFC_MAG_MASK : 0)); + /* Free MSI/legacy interrupt 0 when in recovery mode. */ + if (test_bit(__I40E_RECOVERY_MODE, pf->state) && + !(pf->flags & I40E_FLAG_MSIX_ENABLED)) + free_irq(pf->pdev->irq, pf); + /* Since we're going to destroy queues during the * i40e_clear_interrupt_scheme() we should hold the RTNL lock for this * whole section diff --git a/drivers/net/ethernet/intel/i40e/i40e_nvm.c b/drivers/net/ethernet/intel/i40e/i40e_nvm.c index 0299e5bbb9022a457fa398ca7a401bc73bc6c1b8..c508b75c3c09e172431532d67621a07a27b4c9c3 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_nvm.c +++ b/drivers/net/ethernet/intel/i40e/i40e_nvm.c @@ -578,11 +578,10 @@ i40e_status i40e_update_nvm_checksum(struct i40e_hw *hw) __le16 le_sum; ret_code = i40e_calc_nvm_checksum(hw, &checksum); - if (!ret_code) { - le_sum = cpu_to_le16(checksum); + le_sum = cpu_to_le16(checksum); + if (!ret_code) ret_code = i40e_write_nvm_aq(hw, 0x00, I40E_SR_SW_CHECKSUM_WORD, 1, &le_sum, true); - } return ret_code; } diff --git a/drivers/net/ethernet/intel/i40e/i40e_prototype.h b/drivers/net/ethernet/intel/i40e/i40e_prototype.h index e08d754824b13e97f7e7db73d762668aeff87931..882627073dcea8ab12a66c4b5f4bd391b125900a 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_prototype.h +++ b/drivers/net/ethernet/intel/i40e/i40e_prototype.h @@ -203,14 +203,18 @@ i40e_status i40e_aq_get_lldp_mib(struct i40e_hw *hw, u8 bridge_type, i40e_status i40e_aq_cfg_lldp_mib_change_event(struct i40e_hw *hw, bool enable_update, struct i40e_asq_cmd_details *cmd_details); +enum i40e_status_code +i40e_aq_restore_lldp(struct i40e_hw *hw, u8 *setting, bool restore, + struct i40e_asq_cmd_details *cmd_details); i40e_status i40e_aq_stop_lldp(struct i40e_hw *hw, bool shutdown_agent, + bool persist, struct i40e_asq_cmd_details *cmd_details); i40e_status i40e_aq_set_dcb_parameters(struct i40e_hw *hw, bool dcb_enable, struct i40e_asq_cmd_details *cmd_details); -i40e_status i40e_aq_start_lldp(struct i40e_hw *hw, - struct i40e_asq_cmd_details *cmd_details); +i40e_status i40e_aq_start_lldp(struct i40e_hw *hw, bool persist, + struct i40e_asq_cmd_details *cmd_details); i40e_status i40e_aq_get_cee_dcb_config(struct i40e_hw *hw, void *buff, u16 buff_size, struct i40e_asq_cmd_details *cmd_details); @@ -429,10 +433,16 @@ i40e_status i40e_aq_get_ddp_list(struct i40e_hw *hw, void *buff, struct i40e_generic_seg_header * i40e_find_segment_in_package(u32 segment_type, struct i40e_package_header *pkg_header); +struct i40e_profile_section_header * +i40e_find_section_in_profile(u32 section_type, + struct i40e_profile_segment *profile); enum i40e_status_code i40e_write_profile(struct i40e_hw *hw, struct i40e_profile_segment *i40e_seg, u32 track_id); enum i40e_status_code +i40e_rollback_profile(struct i40e_hw *hw, struct i40e_profile_segment *i40e_seg, + u32 track_id); +enum i40e_status_code i40e_add_pinfo_to_list(struct i40e_hw *hw, struct i40e_profile_segment *profile, u8 *profile_info_sec, u32 track_id); diff --git a/drivers/net/ethernet/intel/i40e/i40e_ptp.c b/drivers/net/ethernet/intel/i40e/i40e_ptp.c index 31575c0bb884f2b0520b848d29420dc98438eba4..439c35f0c581cc66b8f69fb706ac0f4a82e8a8c3 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_ptp.c +++ b/drivers/net/ethernet/intel/i40e/i40e_ptp.c @@ -725,9 +725,56 @@ static long i40e_ptp_create_clock(struct i40e_pf *pf) pf->tstamp_config.rx_filter = HWTSTAMP_FILTER_NONE; pf->tstamp_config.tx_type = HWTSTAMP_TX_OFF; + /* Set the previous "reset" time to the current Kernel clock time */ + pf->ptp_prev_hw_time = ktime_to_timespec64(ktime_get_real()); + pf->ptp_reset_start = ktime_get(); + return 0; } +/** + * i40e_ptp_save_hw_time - Save the current PTP time as ptp_prev_hw_time + * @pf: Board private structure + * + * Read the current PTP time and save it into pf->ptp_prev_hw_time. This should + * be called at the end of preparing to reset, just before hardware reset + * occurs, in order to preserve the PTP time as close as possible across + * resets. + */ +void i40e_ptp_save_hw_time(struct i40e_pf *pf) +{ + /* don't try to access the PTP clock if it's not enabled */ + if (!(pf->flags & I40E_FLAG_PTP)) + return; + + i40e_ptp_gettimex(&pf->ptp_caps, &pf->ptp_prev_hw_time, NULL); + /* Get a monotonic starting time for this reset */ + pf->ptp_reset_start = ktime_get(); +} + +/** + * i40e_ptp_restore_hw_time - Restore the ptp_prev_hw_time + delta to PTP regs + * @pf: Board private structure + * + * Restore the PTP hardware clock registers. We previously cached the PTP + * hardware time as pf->ptp_prev_hw_time. To be as accurate as possible, + * update this value based on the time delta since the time was saved, using + * CLOCK_MONOTONIC (via ktime_get()) to calculate the time difference. + * + * This ensures that the hardware clock is restored to nearly what it should + * have been if a reset had not occurred. + */ +void i40e_ptp_restore_hw_time(struct i40e_pf *pf) +{ + ktime_t delta = ktime_sub(ktime_get(), pf->ptp_reset_start); + + /* Update the previous HW time with the ktime delta */ + timespec64_add_ns(&pf->ptp_prev_hw_time, ktime_to_ns(delta)); + + /* Restore the hardware clock registers */ + i40e_ptp_settime(&pf->ptp_caps, &pf->ptp_prev_hw_time); +} + /** * i40e_ptp_init - Initialize the 1588 support after device probe or reset * @pf: Board private structure @@ -735,6 +782,11 @@ static long i40e_ptp_create_clock(struct i40e_pf *pf) * This function sets device up for 1588 support. The first time it is run, it * will create a PHC clock device. It does not create a clock device if one * already exists. It also reconfigures the device after a reset. + * + * The first time a clock is created, i40e_ptp_create_clock will set + * pf->ptp_prev_hw_time to the current system time. During resets, it is + * expected that this timespec will be set to the last known PTP clock time, + * in order to preserve the clock time as close as possible across a reset. **/ void i40e_ptp_init(struct i40e_pf *pf) { @@ -766,7 +818,6 @@ void i40e_ptp_init(struct i40e_pf *pf) dev_err(&pf->pdev->dev, "%s: ptp_clock_register failed\n", __func__); } else if (pf->ptp_clock) { - struct timespec64 ts; u32 regval; if (pf->hw.debug_mask & I40E_DEBUG_LAN) @@ -787,9 +838,8 @@ void i40e_ptp_init(struct i40e_pf *pf) /* reset timestamping mode */ i40e_ptp_set_timestamp_mode(pf, &pf->tstamp_config); - /* Set the clock value. */ - ts = ktime_to_timespec64(ktime_get_real()); - i40e_ptp_settime(&pf->ptp_caps, &ts); + /* Restore the clock time based on last known value */ + i40e_ptp_restore_hw_time(pf); } } diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.c b/drivers/net/ethernet/intel/i40e/i40e_txrx.c index ffb611bbedfa42a5fc330a4e70e153eed125c49a..20a283702c9f7613fb010472807928b323ec9d2a 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_txrx.c +++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.c @@ -2035,7 +2035,8 @@ static struct sk_buff *i40e_construct_skb(struct i40e_ring *rx_ring, /* Determine available headroom for copy */ headlen = size; if (headlen > I40E_RX_HDR_SIZE) - headlen = eth_get_headlen(xdp->data, I40E_RX_HDR_SIZE); + headlen = eth_get_headlen(skb->dev, xdp->data, + I40E_RX_HDR_SIZE); /* align pull length to size of long to optimize memcpy performance */ memcpy(__skb_put(skb, headlen), xdp->data, @@ -3469,7 +3470,7 @@ static inline int i40e_tx_map(struct i40e_ring *tx_ring, struct sk_buff *skb, first->next_to_watch = tx_desc; /* notify HW of packet */ - if (netif_xmit_stopped(txring_txq(tx_ring)) || !skb->xmit_more) { + if (netif_xmit_stopped(txring_txq(tx_ring)) || !netdev_xmit_more()) { writel(i, tx_ring->tail); } diff --git a/drivers/net/ethernet/intel/i40e/i40e_type.h b/drivers/net/ethernet/intel/i40e/i40e_type.h index 2781ab91ca82f2b0a9249afbb293042e9705b3a5..8f43aa47c2631722e3cc4c13f02472976427c4a6 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_type.h +++ b/drivers/net/ethernet/intel/i40e/i40e_type.h @@ -252,6 +252,12 @@ struct i40e_phy_info { I40E_PHY_TYPE_OFFSET) #define I40E_CAP_PHY_TYPE_25GBASE_ACC BIT_ULL(I40E_PHY_TYPE_25GBASE_ACC + \ I40E_PHY_TYPE_OFFSET) +/* Offset for 2.5G/5G PHY Types value to bit number conversion */ +#define I40E_PHY_TYPE_OFFSET2 (-10) +#define I40E_CAP_PHY_TYPE_2_5GBASE_T BIT_ULL(I40E_PHY_TYPE_2_5GBASE_T + \ + I40E_PHY_TYPE_OFFSET2) +#define I40E_CAP_PHY_TYPE_5GBASE_T BIT_ULL(I40E_PHY_TYPE_5GBASE_T + \ + I40E_PHY_TYPE_OFFSET2) #define I40E_HW_CAP_MAX_GPIO 30 /* Capabilities of a PF or a VF or the whole device */ struct i40e_hw_capabilities { @@ -616,6 +622,7 @@ struct i40e_hw { #define I40E_HW_FLAG_AQ_PHY_ACCESS_CAPABLE BIT_ULL(2) #define I40E_HW_FLAG_NVM_READ_REQUIRES_LOCK BIT_ULL(3) #define I40E_HW_FLAG_FW_LLDP_STOPPABLE BIT_ULL(4) +#define I40E_HW_FLAG_FW_LLDP_PERSISTENT BIT_ULL(5) u64 flags; /* Used in set switch config AQ command */ @@ -1527,6 +1534,8 @@ struct i40e_generic_seg_header { struct i40e_metadata_segment { struct i40e_generic_seg_header header; struct i40e_ddp_version version; +#define I40E_DDP_TRACKID_RDONLY 0 +#define I40E_DDP_TRACKID_INVALID 0xFFFFFFFF u32 track_id; char name[I40E_DDP_NAME_SIZE]; }; @@ -1555,15 +1564,36 @@ struct i40e_profile_section_header { struct { #define SECTION_TYPE_INFO 0x00000010 #define SECTION_TYPE_MMIO 0x00000800 +#define SECTION_TYPE_RB_MMIO 0x00001800 #define SECTION_TYPE_AQ 0x00000801 +#define SECTION_TYPE_RB_AQ 0x00001801 #define SECTION_TYPE_NOTE 0x80000000 #define SECTION_TYPE_NAME 0x80000001 +#define SECTION_TYPE_PROTO 0x80000002 +#define SECTION_TYPE_PCTYPE 0x80000003 +#define SECTION_TYPE_PTYPE 0x80000004 u32 type; u32 offset; u32 size; } section; }; +struct i40e_profile_tlv_section_record { + u8 rtype; + u8 type; + u16 len; + u8 data[12]; +}; + +/* Generic AQ section in proflie */ +struct i40e_profile_aq_section { + u16 opcode; + u16 flags; + u8 param[16]; + u16 datalen; + u8 data[1]; +}; + struct i40e_profile_info { u32 track_id; struct i40e_ddp_version version; diff --git a/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c b/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c index 831d52bc3c9ae5b4073d4909f591a7802f63c1c6..479bc60c8f719e744a0d4e021f43118059101e1e 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c +++ b/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c @@ -181,7 +181,7 @@ static inline bool i40e_vc_isvalid_vsi_id(struct i40e_vf *vf, u16 vsi_id) * check for the valid queue id **/ static inline bool i40e_vc_isvalid_queue_id(struct i40e_vf *vf, u16 vsi_id, - u8 qid) + u16 qid) { struct i40e_pf *pf = vf->pf; struct i40e_vsi *vsi = i40e_find_vsi_from_id(pf, vsi_id); @@ -196,7 +196,7 @@ static inline bool i40e_vc_isvalid_queue_id(struct i40e_vf *vf, u16 vsi_id, * * check for the valid vector id **/ -static inline bool i40e_vc_isvalid_vector_id(struct i40e_vf *vf, u8 vector_id) +static inline bool i40e_vc_isvalid_vector_id(struct i40e_vf *vf, u32 vector_id) { struct i40e_pf *pf = vf->pf; @@ -441,14 +441,28 @@ static int i40e_config_iwarp_qvlist(struct i40e_vf *vf, u32 v_idx, i, reg_idx, reg; u32 next_q_idx, next_q_type; u32 msix_vf, size; + int ret = 0; + + msix_vf = pf->hw.func_caps.num_msix_vectors_vf; + + if (qvlist_info->num_vectors > msix_vf) { + dev_warn(&pf->pdev->dev, + "Incorrect number of iwarp vectors %u. Maximum %u allowed.\n", + qvlist_info->num_vectors, + msix_vf); + ret = -EINVAL; + goto err_out; + } size = sizeof(struct virtchnl_iwarp_qvlist_info) + (sizeof(struct virtchnl_iwarp_qv_info) * (qvlist_info->num_vectors - 1)); + kfree(vf->qvlist_info); vf->qvlist_info = kzalloc(size, GFP_KERNEL); - if (!vf->qvlist_info) - return -ENOMEM; - + if (!vf->qvlist_info) { + ret = -ENOMEM; + goto err_out; + } vf->qvlist_info->num_vectors = qvlist_info->num_vectors; msix_vf = pf->hw.func_caps.num_msix_vectors_vf; @@ -459,8 +473,10 @@ static int i40e_config_iwarp_qvlist(struct i40e_vf *vf, v_idx = qv_info->v_idx; /* Validate vector id belongs to this vf */ - if (!i40e_vc_isvalid_vector_id(vf, v_idx)) - goto err; + if (!i40e_vc_isvalid_vector_id(vf, v_idx)) { + ret = -EINVAL; + goto err_free; + } vf->qvlist_info->qv_info[i] = *qv_info; @@ -502,10 +518,11 @@ static int i40e_config_iwarp_qvlist(struct i40e_vf *vf, } return 0; -err: +err_free: kfree(vf->qvlist_info); vf->qvlist_info = NULL; - return -EINVAL; +err_out: + return ret; } /** @@ -1112,15 +1129,6 @@ static i40e_status i40e_config_vf_promiscuous_mode(struct i40e_vf *vf, if (!i40e_vc_isvalid_vsi_id(vf, vsi_id) || !vsi) return I40E_ERR_PARAM; - if (!test_bit(I40E_VIRTCHNL_VF_CAP_PRIVILEGE, &vf->vf_caps) && - (allmulti || alluni)) { - dev_err(&pf->pdev->dev, - "Unprivileged VF %d is attempting to configure promiscuous mode\n", - vf->vf_id); - /* Lie to the VF on purpose. */ - return 0; - } - if (vf->port_vlan_id) { aq_ret = i40e_aq_set_vsi_mc_promisc_on_vlan(hw, vsi->seid, allmulti, @@ -1997,8 +2005,31 @@ static int i40e_vc_config_promiscuous_mode_msg(struct i40e_vf *vf, u8 *msg) bool allmulti = false; bool alluni = false; - if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states)) - return I40E_ERR_PARAM; + if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states)) { + aq_ret = I40E_ERR_PARAM; + goto err_out; + } + if (!test_bit(I40E_VIRTCHNL_VF_CAP_PRIVILEGE, &vf->vf_caps)) { + dev_err(&pf->pdev->dev, + "Unprivileged VF %d is attempting to configure promiscuous mode\n", + vf->vf_id); + + /* Lie to the VF on purpose, because this is an error we can + * ignore. Unprivileged VF is not a virtual channel error. + */ + aq_ret = 0; + goto err_out; + } + + if (info->flags > I40E_MAX_VF_PROMISC_FLAGS) { + aq_ret = I40E_ERR_PARAM; + goto err_out; + } + + if (!i40e_vc_isvalid_vsi_id(vf, info->vsi_id)) { + aq_ret = I40E_ERR_PARAM; + goto err_out; + } /* Multicast promiscuous handling*/ if (info->flags & FLAG_VF_MULTICAST_PROMISC) @@ -2032,7 +2063,7 @@ static int i40e_vc_config_promiscuous_mode_msg(struct i40e_vf *vf, u8 *msg) clear_bit(I40E_VF_STATE_UC_PROMISC, &vf->vf_states); } } - +err_out: /* send the response to the VF */ return i40e_vc_send_resp_to_vf(vf, VIRTCHNL_OP_CONFIG_PROMISCUOUS_MODE, @@ -2054,17 +2085,16 @@ static int i40e_vc_config_queues_msg(struct i40e_vf *vf, u8 *msg) struct virtchnl_queue_pair_info *qpi; struct i40e_pf *pf = vf->pf; u16 vsi_id, vsi_queue_id = 0; + u16 num_qps_all = 0; i40e_status aq_ret = 0; int i, j = 0, idx = 0; - vsi_id = qci->vsi_id; - if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states)) { aq_ret = I40E_ERR_PARAM; goto error_param; } - if (!i40e_vc_isvalid_vsi_id(vf, vsi_id)) { + if (!i40e_vc_isvalid_vsi_id(vf, qci->vsi_id)) { aq_ret = I40E_ERR_PARAM; goto error_param; } @@ -2074,10 +2104,27 @@ static int i40e_vc_config_queues_msg(struct i40e_vf *vf, u8 *msg) goto error_param; } + if (vf->adq_enabled) { + for (i = 0; i < I40E_MAX_VF_VSI; i++) + num_qps_all += vf->ch[i].num_qps; + if (num_qps_all != qci->num_queue_pairs) { + aq_ret = I40E_ERR_PARAM; + goto error_param; + } + } + + vsi_id = qci->vsi_id; + for (i = 0; i < qci->num_queue_pairs; i++) { qpi = &qci->qpair[i]; if (!vf->adq_enabled) { + if (!i40e_vc_isvalid_queue_id(vf, vsi_id, + qpi->txq.queue_id)) { + aq_ret = I40E_ERR_PARAM; + goto error_param; + } + vsi_queue_id = qpi->txq.queue_id; if (qpi->txq.vsi_id != qci->vsi_id || @@ -2088,10 +2135,8 @@ static int i40e_vc_config_queues_msg(struct i40e_vf *vf, u8 *msg) } } - if (!i40e_vc_isvalid_queue_id(vf, vsi_id, vsi_queue_id)) { - aq_ret = I40E_ERR_PARAM; - goto error_param; - } + if (vf->adq_enabled) + vsi_id = vf->ch[idx].vsi_id; if (i40e_config_vsi_rx_queue(vf, vsi_id, vsi_queue_id, &qpi->rxq) || @@ -2115,7 +2160,6 @@ static int i40e_vc_config_queues_msg(struct i40e_vf *vf, u8 *msg) j++; vsi_queue_id++; } - vsi_id = vf->ch[idx].vsi_id; } } /* set vsi num_queue_pairs in use to num configured by VF */ @@ -2174,7 +2218,7 @@ static int i40e_vc_config_irq_map_msg(struct i40e_vf *vf, u8 *msg) struct virtchnl_irq_map_info *irqmap_info = (struct virtchnl_irq_map_info *)msg; struct virtchnl_vector_map *map; - u16 vsi_id, vector_id; + u16 vsi_id; i40e_status aq_ret = 0; int i; @@ -2183,16 +2227,21 @@ static int i40e_vc_config_irq_map_msg(struct i40e_vf *vf, u8 *msg) goto error_param; } + if (irqmap_info->num_vectors > + vf->pf->hw.func_caps.num_msix_vectors_vf) { + aq_ret = I40E_ERR_PARAM; + goto error_param; + } + for (i = 0; i < irqmap_info->num_vectors; i++) { map = &irqmap_info->vecmap[i]; - vector_id = map->vector_id; - vsi_id = map->vsi_id; /* validate msg params */ - if (!i40e_vc_isvalid_vector_id(vf, vector_id) || - !i40e_vc_isvalid_vsi_id(vf, vsi_id)) { + if (!i40e_vc_isvalid_vector_id(vf, map->vector_id) || + !i40e_vc_isvalid_vsi_id(vf, map->vsi_id)) { aq_ret = I40E_ERR_PARAM; goto error_param; } + vsi_id = map->vsi_id; if (i40e_validate_queue_map(vf, vsi_id, map->rxq_map)) { aq_ret = I40E_ERR_PARAM; @@ -2340,7 +2389,9 @@ static int i40e_vc_disable_queues_msg(struct i40e_vf *vf, u8 *msg) goto error_param; } - if ((0 == vqs->rx_queues) && (0 == vqs->tx_queues)) { + if ((vqs->rx_queues == 0 && vqs->tx_queues == 0) || + vqs->rx_queues > I40E_MAX_VF_QUEUES || + vqs->tx_queues > I40E_MAX_VF_QUEUES) { aq_ret = I40E_ERR_PARAM; goto error_param; } @@ -2454,8 +2505,10 @@ error_param: (u8 *)&stats, sizeof(stats)); } -/* If the VF is not trusted restrict the number of MAC/VLAN it can program */ -#define I40E_VC_MAX_MAC_ADDR_PER_VF 12 +/* If the VF is not trusted restrict the number of MAC/VLAN it can program + * MAC filters: 16 for multicast, 1 for MAC, 1 for broadcast + */ +#define I40E_VC_MAX_MAC_ADDR_PER_VF (16 + 1 + 1) #define I40E_VC_MAX_VLAN_PER_VF 8 /** @@ -2764,7 +2817,8 @@ static int i40e_vc_remove_vlan_msg(struct i40e_vf *vf, u8 *msg) vsi = pf->vsi[vf->lan_vsi_idx]; if (vsi->info.pvid) { - aq_ret = I40E_ERR_PARAM; + if (vfl->num_elements > 1 || vfl->vlan_id[0]) + aq_ret = I40E_ERR_PARAM; goto error_param; } @@ -3126,7 +3180,7 @@ static int i40e_validate_cloud_filter(struct i40e_vf *vf, } if (mask.dst_port & data.dst_port) { - if (!data.dst_port || be16_to_cpu(data.dst_port) > 0xFFFF) { + if (!data.dst_port) { dev_info(&pf->pdev->dev, "VF %d: Invalid Dest port\n", vf->vf_id); goto err; @@ -3134,7 +3188,7 @@ static int i40e_validate_cloud_filter(struct i40e_vf *vf, } if (mask.src_port & data.src_port) { - if (!data.src_port || be16_to_cpu(data.src_port) > 0xFFFF) { + if (!data.src_port) { dev_info(&pf->pdev->dev, "VF %d: Invalid Source port\n", vf->vf_id); goto err; @@ -3374,7 +3428,7 @@ static int i40e_vc_add_cloud_filter(struct i40e_vf *vf, u8 *msg) if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states)) { aq_ret = I40E_ERR_PARAM; - goto err; + goto err_out; } if (!vf->adq_enabled) { @@ -3382,7 +3436,7 @@ static int i40e_vc_add_cloud_filter(struct i40e_vf *vf, u8 *msg) "VF %d: ADq is not enabled, can't apply cloud filter\n", vf->vf_id); aq_ret = I40E_ERR_PARAM; - goto err; + goto err_out; } if (i40e_validate_cloud_filter(vf, vcf)) { @@ -3390,7 +3444,7 @@ static int i40e_vc_add_cloud_filter(struct i40e_vf *vf, u8 *msg) "VF %d: Invalid input/s, can't apply cloud filter\n", vf->vf_id); aq_ret = I40E_ERR_PARAM; - goto err; + goto err_out; } cfilter = kzalloc(sizeof(*cfilter), GFP_KERNEL); @@ -3451,13 +3505,17 @@ static int i40e_vc_add_cloud_filter(struct i40e_vf *vf, u8 *msg) "VF %d: Failed to add cloud filter, err %s aq_err %s\n", vf->vf_id, i40e_stat_str(&pf->hw, ret), i40e_aq_str(&pf->hw, pf->hw.aq.asq_last_status)); - goto err; + goto err_free; } INIT_HLIST_NODE(&cfilter->cloud_node); hlist_add_head(&cfilter->cloud_node, &vf->cloud_filter_list); + /* release the pointer passing it to the collection */ + cfilter = NULL; vf->num_cloud_filters++; -err: +err_free: + kfree(cfilter); +err_out: return i40e_vc_send_resp_to_vf(vf, VIRTCHNL_OP_ADD_CLOUD_FILTER, aq_ret); } @@ -4009,6 +4067,7 @@ int i40e_ndo_set_vf_port_vlan(struct net_device *netdev, int vf_id, { u16 vlanprio = vlan_id | (qos << I40E_VLAN_PRIORITY_SHIFT); struct i40e_netdev_priv *np = netdev_priv(netdev); + bool allmulti = false, alluni = false; struct i40e_pf *pf = np->vsi->back; struct i40e_vsi *vsi; struct i40e_vf *vf; @@ -4093,6 +4152,15 @@ int i40e_ndo_set_vf_port_vlan(struct net_device *netdev, int vf_id, } spin_unlock_bh(&vsi->mac_filter_hash_lock); + + /* disable promisc modes in case they were enabled */ + ret = i40e_config_vf_promiscuous_mode(vf, vf->lan_vsi_id, + allmulti, alluni); + if (ret) { + dev_err(&pf->pdev->dev, "Unable to config VF promiscuous mode\n"); + goto error_pvid; + } + if (vlan_id || qos) ret = i40e_vsi_add_pvid(vsi, vlanprio); else @@ -4119,6 +4187,12 @@ int i40e_ndo_set_vf_port_vlan(struct net_device *netdev, int vf_id, spin_unlock_bh(&vsi->mac_filter_hash_lock); + if (test_bit(I40E_VF_STATE_UC_PROMISC, &vf->vf_states)) + alluni = true; + + if (test_bit(I40E_VF_STATE_MC_PROMISC, &vf->vf_states)) + allmulti = true; + /* Schedule the worker thread to take care of applying changes */ i40e_service_event_schedule(vsi->back); @@ -4131,6 +4205,13 @@ int i40e_ndo_set_vf_port_vlan(struct net_device *netdev, int vf_id, * default LAN MAC address. */ vf->port_vlan_id = le16_to_cpu(vsi->info.pvid); + + ret = i40e_config_vf_promiscuous_mode(vf, vsi->id, allmulti, alluni); + if (ret) { + dev_err(&pf->pdev->dev, "Unable to config vf promiscuous mode\n"); + goto error_pvid; + } + ret = 0; error_pvid: diff --git a/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.h b/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.h index f9621026beefb0cbef63496c2b93101eed7f1092..f65cc0c1655029fa7119c364f18be581a0a6b398 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.h +++ b/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.h @@ -17,6 +17,8 @@ #define I40E_VLAN_MASK 0xFFF #define I40E_PRIORITY_MASK 0xE000 +#define I40E_MAX_VF_PROMISC_FLAGS 3 + /* Various queue ctrls */ enum i40e_queue_ctrl { I40E_QUEUE_CTRL_UNKNOWN = 0, diff --git a/drivers/net/ethernet/intel/iavf/i40e_adminq_cmd.h b/drivers/net/ethernet/intel/iavf/i40e_adminq_cmd.h index af4f94a6541e9d3d4498de8fdc94e48fd93349b3..e5ae4a1c0cff5efdc5d869f3dbd61276d3c65c7c 100644 --- a/drivers/net/ethernet/intel/iavf/i40e_adminq_cmd.h +++ b/drivers/net/ethernet/intel/iavf/i40e_adminq_cmd.h @@ -14,7 +14,7 @@ #define I40E_FW_API_VERSION_MAJOR 0x0001 #define I40E_FW_API_VERSION_MINOR_X722 0x0005 -#define I40E_FW_API_VERSION_MINOR_X710 0x0007 +#define I40E_FW_API_VERSION_MINOR_X710 0x0008 #define I40E_FW_MINOR_VERSION(_h) ((_h)->mac.type == I40E_MAC_XL710 ? \ I40E_FW_API_VERSION_MINOR_X710 : \ diff --git a/drivers/net/ethernet/intel/iavf/iavf_txrx.c b/drivers/net/ethernet/intel/iavf/iavf_txrx.c index 6bfef82e7607b4a49e30b6da997aa656b88ec9e9..06d1509d57f7040866c07b6a177b7bbcd65cd506 100644 --- a/drivers/net/ethernet/intel/iavf/iavf_txrx.c +++ b/drivers/net/ethernet/intel/iavf/iavf_txrx.c @@ -1315,7 +1315,7 @@ static struct sk_buff *iavf_construct_skb(struct iavf_ring *rx_ring, /* Determine available headroom for copy */ headlen = size; if (headlen > IAVF_RX_HDR_SIZE) - headlen = eth_get_headlen(va, IAVF_RX_HDR_SIZE); + headlen = eth_get_headlen(skb->dev, va, IAVF_RX_HDR_SIZE); /* align pull length to size of long to optimize memcpy performance */ memcpy(__skb_put(skb, headlen), va, ALIGN(headlen, sizeof(long))); @@ -2358,7 +2358,7 @@ static inline void iavf_tx_map(struct iavf_ring *tx_ring, struct sk_buff *skb, first->next_to_watch = tx_desc; /* notify HW of packet */ - if (netif_xmit_stopped(txring_txq(tx_ring)) || !skb->xmit_more) { + if (netif_xmit_stopped(txring_txq(tx_ring)) || !netdev_xmit_more()) { writel(i, tx_ring->tail); } diff --git a/drivers/net/ethernet/intel/ice/Makefile b/drivers/net/ethernet/intel/ice/Makefile index e5d6f684437e42ef65df30cd1cda6b8d64702c5d..2d140ba8378157c9c155f3866c1f651caa3fce8a 100644 --- a/drivers/net/ethernet/intel/ice/Makefile +++ b/drivers/net/ethernet/intel/ice/Makefile @@ -17,3 +17,4 @@ ice-y := ice_main.o \ ice_txrx.o \ ice_ethtool.o ice-$(CONFIG_PCI_IOV) += ice_virtchnl_pf.o ice_sriov.o +ice-$(CONFIG_DCB) += ice_dcb.o ice_dcb_lib.o diff --git a/drivers/net/ethernet/intel/ice/ice.h b/drivers/net/ethernet/intel/ice/ice.h index 89440775aea12df34d1fb43978be83e789aaec00..792e6e42030e164dd4800c105792d3e3db1b40d5 100644 --- a/drivers/net/ethernet/intel/ice/ice.h +++ b/drivers/net/ethernet/intel/ice/ice.h @@ -34,6 +34,7 @@ #include "ice_devids.h" #include "ice_type.h" #include "ice_txrx.h" +#include "ice_dcb.h" #include "ice_switch.h" #include "ice_common.h" #include "ice_sched.h" @@ -42,10 +43,21 @@ extern const char ice_drv_ver[]; #define ICE_BAR0 0 -#define ICE_DFLT_NUM_DESC 128 #define ICE_REQ_DESC_MULTIPLE 32 #define ICE_MIN_NUM_DESC ICE_REQ_DESC_MULTIPLE #define ICE_MAX_NUM_DESC 8160 +/* set default number of Rx/Tx descriptors to the minimum between + * ICE_MAX_NUM_DESC and the number of descriptors to fill up an entire page + */ +#define ICE_DFLT_NUM_RX_DESC min_t(u16, ICE_MAX_NUM_DESC, \ + ALIGN(PAGE_SIZE / \ + sizeof(union ice_32byte_rx_desc), \ + ICE_REQ_DESC_MULTIPLE)) +#define ICE_DFLT_NUM_TX_DESC min_t(u16, ICE_MAX_NUM_DESC, \ + ALIGN(PAGE_SIZE / \ + sizeof(struct ice_tx_desc), \ + ICE_REQ_DESC_MULTIPLE)) + #define ICE_DFLT_TRAFFIC_CLASS BIT(0) #define ICE_INT_NAME_STR_LEN (IFNAMSIZ + 16) #define ICE_ETHTOOL_FWVER_LEN 32 @@ -71,6 +83,8 @@ extern const char ice_drv_ver[]; #define ICE_MAX_QS_PER_VF 256 #define ICE_MIN_QS_PER_VF 1 #define ICE_DFLT_QS_PER_VF 4 +#define ICE_NONQ_VECS_VF 1 +#define ICE_MAX_SCATTER_QS_PER_VF 16 #define ICE_MAX_BASE_QS_PER_VF 16 #define ICE_MAX_INTR_PER_VF 65 #define ICE_MIN_INTR_PER_VF (ICE_MIN_QS_PER_VF + 1) @@ -114,6 +128,23 @@ extern const char ice_drv_ver[]; #define ice_for_each_q_vector(vsi, i) \ for ((i) = 0; (i) < (vsi)->num_q_vectors; (i)++) +#define ICE_UCAST_PROMISC_BITS (ICE_PROMISC_UCAST_TX | ICE_PROMISC_MCAST_TX | \ + ICE_PROMISC_UCAST_RX | ICE_PROMISC_MCAST_RX) + +#define ICE_UCAST_VLAN_PROMISC_BITS (ICE_PROMISC_UCAST_TX | \ + ICE_PROMISC_MCAST_TX | \ + ICE_PROMISC_UCAST_RX | \ + ICE_PROMISC_MCAST_RX | \ + ICE_PROMISC_VLAN_TX | \ + ICE_PROMISC_VLAN_RX) + +#define ICE_MCAST_PROMISC_BITS (ICE_PROMISC_MCAST_TX | ICE_PROMISC_MCAST_RX) + +#define ICE_MCAST_VLAN_PROMISC_BITS (ICE_PROMISC_MCAST_TX | \ + ICE_PROMISC_MCAST_RX | \ + ICE_PROMISC_VLAN_TX | \ + ICE_PROMISC_VLAN_RX) + struct ice_tc_info { u16 qoffset; u16 qcount_tx; @@ -123,7 +154,7 @@ struct ice_tc_info { struct ice_tc_cfg { u8 numtc; /* Total number of enabled TCs */ - u8 ena_tc; /* TX map */ + u8 ena_tc; /* Tx map */ struct ice_tc_info tc_info[ICE_MAX_TRAFFIC_CLASS]; }; @@ -134,7 +165,7 @@ struct ice_res_tracker { }; struct ice_qs_cfg { - struct mutex *qs_mutex; /* will be assgined to &pf->avail_q_mutex */ + struct mutex *qs_mutex; /* will be assigned to &pf->avail_q_mutex */ unsigned long *pf_map; unsigned long pf_map_size; unsigned int q_count; @@ -224,6 +255,8 @@ struct ice_vsi { s16 vf_id; /* VF ID for SR-IOV VSIs */ + u16 ethtype; /* Ethernet protocol for pause frame */ + /* RSS config */ u16 rss_table_size; /* HW RSS table size */ u16 rss_size; /* Allocated RSS queues */ @@ -247,6 +280,7 @@ struct ice_vsi { u8 irqs_ready; u8 current_isup; /* Sync 'link up' logging */ u8 stat_offsets_loaded; + u8 vlan_ena; /* queue information */ u8 tx_mapping_mode; /* ICE_MAP_MODE_[CONTIG|SCATTER] */ @@ -257,26 +291,34 @@ struct ice_vsi { u16 num_txq; /* Used Tx queues */ u16 alloc_rxq; /* Allocated Rx queues */ u16 num_rxq; /* Used Rx queues */ - u16 num_desc; + u16 num_rx_desc; + u16 num_tx_desc; struct ice_tc_cfg tc_cfg; } ____cacheline_internodealigned_in_smp; /* struct that defines an interrupt vector */ struct ice_q_vector { struct ice_vsi *vsi; - cpumask_t affinity_mask; - struct napi_struct napi; - struct ice_ring_container rx; - struct ice_ring_container tx; - struct irq_affinity_notify affinity_notify; + u16 v_idx; /* index in the vsi->q_vector array. */ - u8 num_ring_tx; /* total number of Tx rings in vector */ + u16 reg_idx; u8 num_ring_rx; /* total number of Rx rings in vector */ - char name[ICE_INT_NAME_STR_LEN]; + u8 num_ring_tx; /* total number of Tx rings in vector */ + u8 itr_countdown; /* when 0 should adjust adaptive ITR */ /* in usecs, need to use ice_intrl_to_usecs_reg() before writing this * value to the device */ u8 intrl; + + struct napi_struct napi; + + struct ice_ring_container rx; + struct ice_ring_container tx; + + cpumask_t affinity_mask; + struct irq_affinity_notify affinity_notify; + + char name[ICE_INT_NAME_STR_LEN]; } ____cacheline_internodealigned_in_smp; enum ice_pf_flags { @@ -285,7 +327,11 @@ enum ice_pf_flags { ICE_FLAG_RSS_ENA, ICE_FLAG_SRIOV_ENA, ICE_FLAG_SRIOV_CAPABLE, + ICE_FLAG_DCB_CAPABLE, + ICE_FLAG_DCB_ENA, ICE_FLAG_LINK_DOWN_ON_CLOSE_ENA, + ICE_FLAG_DISABLE_FW_LLDP, + ICE_FLAG_ETHTOOL_CTXT, /* set when ethtool holds RTNL lock */ ICE_PF_FLAGS_NBITS /* must be last */ }; @@ -324,8 +370,8 @@ struct ice_pf { u32 hw_oicr_idx; /* Other interrupt cause vector HW index */ u32 num_avail_hw_msix; /* remaining HW MSIX vectors left unclaimed */ u32 num_lan_msix; /* Total MSIX vectors for base driver */ - u16 num_lan_tx; /* num lan Tx queues setup */ - u16 num_lan_rx; /* num lan Rx queues setup */ + u16 num_lan_tx; /* num LAN Tx queues setup */ + u16 num_lan_rx; /* num LAN Rx queues setup */ u16 q_left_tx; /* remaining num Tx queues left unclaimed */ u16 q_left_rx; /* remaining num Rx queues left unclaimed */ u16 next_vsi; /* Next free slot in pf->vsi[] - 0-based! */ @@ -339,6 +385,9 @@ struct ice_pf { struct ice_hw_port_stats stats_prev; struct ice_hw hw; u8 stat_prev_loaded; /* has previous stats been loaded */ +#ifdef CONFIG_DCB + u16 dcbx_cap; +#endif /* CONFIG_DCB */ u32 tx_timeout_count; unsigned long tx_timeout_last_recovery; u32 tx_timeout_recovery_level; @@ -351,14 +400,15 @@ struct ice_netdev_priv { /** * ice_irq_dynamic_ena - Enable default interrupt generation settings - * @hw: pointer to hw struct - * @vsi: pointer to vsi struct, can be NULL + * @hw: pointer to HW struct + * @vsi: pointer to VSI struct, can be NULL * @q_vector: pointer to q_vector, can be NULL */ -static inline void ice_irq_dynamic_ena(struct ice_hw *hw, struct ice_vsi *vsi, - struct ice_q_vector *q_vector) +static inline void +ice_irq_dynamic_ena(struct ice_hw *hw, struct ice_vsi *vsi, + struct ice_q_vector *q_vector) { - u32 vector = (vsi && q_vector) ? vsi->hw_base_vector + q_vector->v_idx : + u32 vector = (vsi && q_vector) ? q_vector->reg_idx : ((struct ice_pf *)hw->back)->hw_oicr_idx; int itr = ICE_ITR_NONE; u32 val; @@ -374,10 +424,24 @@ static inline void ice_irq_dynamic_ena(struct ice_hw *hw, struct ice_vsi *vsi, wr32(hw, GLINT_DYN_CTL(vector), val); } -static inline void ice_vsi_set_tc_cfg(struct ice_vsi *vsi) +/** + * ice_find_vsi_by_type - Find and return VSI of a given type + * @pf: PF to search for VSI + * @type: Value indicating type of VSI we are looking for + */ +static inline struct ice_vsi * +ice_find_vsi_by_type(struct ice_pf *pf, enum ice_vsi_type type) { - vsi->tc_cfg.ena_tc = ICE_DFLT_TRAFFIC_CLASS; - vsi->tc_cfg.numtc = 1; + int i; + + for (i = 0; i < pf->num_alloc_vsi; i++) { + struct ice_vsi *vsi = pf->vsi[i]; + + if (vsi && vsi->type == type) + return vsi; + } + + return NULL; } void ice_set_ethtool_ops(struct net_device *netdev); @@ -388,5 +452,9 @@ int ice_get_rss(struct ice_vsi *vsi, u8 *seed, u8 *lut, u16 lut_size); void ice_fill_rss_lut(u8 *lut, u16 rss_table_size, u16 rss_size); void ice_print_link_msg(struct ice_vsi *vsi, bool isup); void ice_napi_del(struct ice_vsi *vsi); +#ifdef CONFIG_DCB +int ice_pf_ena_all_vsi(struct ice_pf *pf, bool locked); +void ice_pf_dis_all_vsi(struct ice_pf *pf, bool locked); +#endif /* CONFIG_DCB */ #endif /* _ICE_H_ */ diff --git a/drivers/net/ethernet/intel/ice/ice_adminq_cmd.h b/drivers/net/ethernet/intel/ice/ice_adminq_cmd.h index 242c784691819ed57094a625a1056db5592192ca..6ef083002f5b0fb25433635a4358fa64cccb58bc 100644 --- a/drivers/net/ethernet/intel/ice/ice_adminq_cmd.h +++ b/drivers/net/ethernet/intel/ice/ice_adminq_cmd.h @@ -62,7 +62,7 @@ struct ice_aqc_req_res { #define ICE_AQ_RES_NVM_WRITE_DFLT_TIMEOUT_MS 180000 #define ICE_AQ_RES_CHNG_LOCK_DFLT_TIMEOUT_MS 1000 #define ICE_AQ_RES_GLBL_LOCK_DFLT_TIMEOUT_MS 3000 - /* For SDP: pin id of the SDP */ + /* For SDP: pin ID of the SDP */ __le32 res_number; /* Status is only used for ICE_AQC_RES_ID_GLBL_LOCK */ __le16 status; @@ -747,6 +747,32 @@ struct ice_aqc_delete_elem { __le32 teid[1]; }; +/* Query Port ETS (indirect 0x040E) + * + * This indirect command is used to query port TC node configuration. + */ +struct ice_aqc_query_port_ets { + __le32 port_teid; + __le32 reserved; + __le32 addr_high; + __le32 addr_low; +}; + +struct ice_aqc_port_ets_elem { + u8 tc_valid_bits; + u8 reserved[3]; + /* 3 bits for UP per TC 0-7, 4th byte reserved */ + __le32 up2tc; + u8 tc_bw_share[8]; + __le32 port_eir_prof_id; + __le32 port_cir_prof_id; + /* 3 bits per Node priority to TC 0-7, 4th byte reserved */ + __le32 tc_node_prio; +#define ICE_TC_NODE_PRIO_S 0x4 + u8 reserved1[4]; + __le32 tc_node_teid[8]; /* Used for response, reserved in command */ +}; + /* Query Scheduler Resource Allocation (indirect 0x0412) * This indirect command retrieves the scheduler resources allocated by * EMP Firmware to the given PF. @@ -953,8 +979,9 @@ struct ice_aqc_set_phy_cfg_data { __le64 phy_type_low; /* Use values from ICE_PHY_TYPE_LOW_* */ __le64 phy_type_high; /* Use values from ICE_PHY_TYPE_HIGH_* */ u8 caps; -#define ICE_AQ_PHY_ENA_TX_PAUSE_ABILITY BIT(0) -#define ICE_AQ_PHY_ENA_RX_PAUSE_ABILITY BIT(1) +#define ICE_AQ_PHY_ENA_VALID_MASK ICE_M(0xef, 0) +#define ICE_AQ_PHY_ENA_TX_PAUSE_ABILITY BIT(0) +#define ICE_AQ_PHY_ENA_RX_PAUSE_ABILITY BIT(1) #define ICE_AQ_PHY_ENA_LOW_POWER BIT(2) #define ICE_AQ_PHY_ENA_LINK BIT(3) #define ICE_AQ_PHY_ENA_AUTO_LINK_UPDT BIT(5) @@ -1023,7 +1050,7 @@ struct ice_aqc_get_link_status_data { u8 ext_info; #define ICE_AQ_LINK_PHY_TEMP_ALARM BIT(0) #define ICE_AQ_LINK_EXCESSIVE_ERRORS BIT(1) /* Excessive Link Errors */ - /* Port TX Suspended */ + /* Port Tx Suspended */ #define ICE_AQ_LINK_TX_S 2 #define ICE_AQ_LINK_TX_M (0x03 << ICE_AQ_LINK_TX_S) #define ICE_AQ_LINK_TX_ACTIVE 0 @@ -1119,9 +1146,9 @@ struct ice_aqc_nvm { }; /** - * Send to PF command (indirect 0x0801) id is only used by PF + * Send to PF command (indirect 0x0801) ID is only used by PF * - * Send to VF command (indirect 0x0802) id is only used by PF + * Send to VF command (indirect 0x0802) ID is only used by PF * */ struct ice_aqc_pf_vf_msg { @@ -1131,6 +1158,126 @@ struct ice_aqc_pf_vf_msg { __le32 addr_low; }; +/* Get LLDP MIB (indirect 0x0A00) + * Note: This is also used by the LLDP MIB Change Event (0x0A01) + * as the format is the same. + */ +struct ice_aqc_lldp_get_mib { + u8 type; +#define ICE_AQ_LLDP_MIB_TYPE_S 0 +#define ICE_AQ_LLDP_MIB_TYPE_M (0x3 << ICE_AQ_LLDP_MIB_TYPE_S) +#define ICE_AQ_LLDP_MIB_LOCAL 0 +#define ICE_AQ_LLDP_MIB_REMOTE 1 +#define ICE_AQ_LLDP_MIB_LOCAL_AND_REMOTE 2 +#define ICE_AQ_LLDP_BRID_TYPE_S 2 +#define ICE_AQ_LLDP_BRID_TYPE_M (0x3 << ICE_AQ_LLDP_BRID_TYPE_S) +#define ICE_AQ_LLDP_BRID_TYPE_NEAREST_BRID 0 +#define ICE_AQ_LLDP_BRID_TYPE_NON_TPMR 1 +/* Tx pause flags in the 0xA01 event use ICE_AQ_LLDP_TX_* */ +#define ICE_AQ_LLDP_TX_S 0x4 +#define ICE_AQ_LLDP_TX_M (0x03 << ICE_AQ_LLDP_TX_S) +#define ICE_AQ_LLDP_TX_ACTIVE 0 +#define ICE_AQ_LLDP_TX_SUSPENDED 1 +#define ICE_AQ_LLDP_TX_FLUSHED 3 +/* The following bytes are reserved for the Get LLDP MIB command (0x0A00) + * and in the LLDP MIB Change Event (0x0A01). They are valid for the + * Get LLDP MIB (0x0A00) response only. + */ + u8 reserved1; + __le16 local_len; + __le16 remote_len; + u8 reserved2[2]; + __le32 addr_high; + __le32 addr_low; +}; + +/* Configure LLDP MIB Change Event (direct 0x0A01) */ +/* For MIB Change Event use ice_aqc_lldp_get_mib structure above */ +struct ice_aqc_lldp_set_mib_change { + u8 command; +#define ICE_AQ_LLDP_MIB_UPDATE_ENABLE 0x0 +#define ICE_AQ_LLDP_MIB_UPDATE_DIS 0x1 + u8 reserved[15]; +}; + +/* Stop LLDP (direct 0x0A05) */ +struct ice_aqc_lldp_stop { + u8 command; +#define ICE_AQ_LLDP_AGENT_STATE_MASK BIT(0) +#define ICE_AQ_LLDP_AGENT_STOP 0x0 +#define ICE_AQ_LLDP_AGENT_SHUTDOWN ICE_AQ_LLDP_AGENT_STATE_MASK +#define ICE_AQ_LLDP_AGENT_PERSIST_DIS BIT(1) + u8 reserved[15]; +}; + +/* Start LLDP (direct 0x0A06) */ +struct ice_aqc_lldp_start { + u8 command; +#define ICE_AQ_LLDP_AGENT_START BIT(0) +#define ICE_AQ_LLDP_AGENT_PERSIST_ENA BIT(1) + u8 reserved[15]; +}; + +/* Get CEE DCBX Oper Config (0x0A07) + * The command uses the generic descriptor struct and + * returns the struct below as an indirect response. + */ +struct ice_aqc_get_cee_dcb_cfg_resp { + u8 oper_num_tc; + u8 oper_prio_tc[4]; + u8 oper_tc_bw[8]; + u8 oper_pfc_en; + __le16 oper_app_prio; +#define ICE_AQC_CEE_APP_FCOE_S 0 +#define ICE_AQC_CEE_APP_FCOE_M (0x7 << ICE_AQC_CEE_APP_FCOE_S) +#define ICE_AQC_CEE_APP_ISCSI_S 3 +#define ICE_AQC_CEE_APP_ISCSI_M (0x7 << ICE_AQC_CEE_APP_ISCSI_S) +#define ICE_AQC_CEE_APP_FIP_S 8 +#define ICE_AQC_CEE_APP_FIP_M (0x7 << ICE_AQC_CEE_APP_FIP_S) + __le32 tlv_status; +#define ICE_AQC_CEE_PG_STATUS_S 0 +#define ICE_AQC_CEE_PG_STATUS_M (0x7 << ICE_AQC_CEE_PG_STATUS_S) +#define ICE_AQC_CEE_PFC_STATUS_S 3 +#define ICE_AQC_CEE_PFC_STATUS_M (0x7 << ICE_AQC_CEE_PFC_STATUS_S) +#define ICE_AQC_CEE_FCOE_STATUS_S 8 +#define ICE_AQC_CEE_FCOE_STATUS_M (0x7 << ICE_AQC_CEE_FCOE_STATUS_S) +#define ICE_AQC_CEE_ISCSI_STATUS_S 11 +#define ICE_AQC_CEE_ISCSI_STATUS_M (0x7 << ICE_AQC_CEE_ISCSI_STATUS_S) +#define ICE_AQC_CEE_FIP_STATUS_S 16 +#define ICE_AQC_CEE_FIP_STATUS_M (0x7 << ICE_AQC_CEE_FIP_STATUS_S) + u8 reserved[12]; +}; + +/* Set Local LLDP MIB (indirect 0x0A08) + * Used to replace the local MIB of a given LLDP agent. e.g. DCBx + */ +struct ice_aqc_lldp_set_local_mib { + u8 type; +#define SET_LOCAL_MIB_TYPE_DCBX_M BIT(0) +#define SET_LOCAL_MIB_TYPE_LOCAL_MIB 0 +#define SET_LOCAL_MIB_TYPE_CEE_M BIT(1) +#define SET_LOCAL_MIB_TYPE_CEE_WILLING 0 +#define SET_LOCAL_MIB_TYPE_CEE_NON_WILLING SET_LOCAL_MIB_TYPE_CEE_M + u8 reserved0; + __le16 length; + u8 reserved1[4]; + __le32 addr_high; + __le32 addr_low; +}; + +/* Stop/Start LLDP Agent (direct 0x0A09) + * Used for stopping/starting specific LLDP agent. e.g. DCBx. + * The same structure is used for the response, with the command field + * being used as the status field. + */ +struct ice_aqc_lldp_stop_start_specific_agent { + u8 command; +#define ICE_AQC_START_STOP_AGENT_M BIT(0) +#define ICE_AQC_START_STOP_AGENT_STOP_DCBX 0 +#define ICE_AQC_START_STOP_AGENT_START_DCBX ICE_AQC_START_STOP_AGENT_M + u8 reserved[15]; +}; + /* Get/Set RSS key (indirect 0x0B04/0x0B02) */ struct ice_aqc_get_set_rss_key { #define ICE_AQC_GSET_RSS_KEY_VSI_VALID BIT(15) @@ -1144,6 +1291,9 @@ struct ice_aqc_get_set_rss_key { #define ICE_AQC_GET_SET_RSS_KEY_DATA_RSS_KEY_SIZE 0x28 #define ICE_AQC_GET_SET_RSS_KEY_DATA_HASH_KEY_SIZE 0xC +#define ICE_GET_SET_RSS_KEY_EXTEND_KEY_SIZE \ + (ICE_AQC_GET_SET_RSS_KEY_DATA_RSS_KEY_SIZE + \ + ICE_AQC_GET_SET_RSS_KEY_DATA_HASH_KEY_SIZE) struct ice_aqc_get_set_rss_keys { u8 standard_rss_key[ICE_AQC_GET_SET_RSS_KEY_DATA_RSS_KEY_SIZE]; @@ -1185,7 +1335,7 @@ struct ice_aqc_get_set_rss_lut { __le32 addr_low; }; -/* Add TX LAN Queues (indirect 0x0C30) */ +/* Add Tx LAN Queues (indirect 0x0C30) */ struct ice_aqc_add_txqs { u8 num_qgrps; u8 reserved[3]; @@ -1194,7 +1344,7 @@ struct ice_aqc_add_txqs { __le32 addr_low; }; -/* This is the descriptor of each queue entry for the Add TX LAN Queues +/* This is the descriptor of each queue entry for the Add Tx LAN Queues * command (0x0C30). Only used within struct ice_aqc_add_tx_qgrp. */ struct ice_aqc_add_txqs_perq { @@ -1206,7 +1356,7 @@ struct ice_aqc_add_txqs_perq { struct ice_aqc_txsched_elem info; }; -/* The format of the command buffer for Add TX LAN Queues (0x0C30) +/* The format of the command buffer for Add Tx LAN Queues (0x0C30) * is an array of the following structs. Please note that the length of * each struct ice_aqc_add_tx_qgrp is variable due * to the variable number of queues in each group! @@ -1218,7 +1368,7 @@ struct ice_aqc_add_tx_qgrp { struct ice_aqc_add_txqs_perq txqs[1]; }; -/* Disable TX LAN Queues (indirect 0x0C31) */ +/* Disable Tx LAN Queues (indirect 0x0C31) */ struct ice_aqc_dis_txqs { u8 cmd_type; #define ICE_AQC_Q_DIS_CMD_S 0 @@ -1240,7 +1390,7 @@ struct ice_aqc_dis_txqs { __le32 addr_low; }; -/* The buffer for Disable TX LAN Queues (indirect 0x0C31) +/* The buffer for Disable Tx LAN Queues (indirect 0x0C31) * contains the following structures, arrayed one after the * other. * Note: Since the q_id is 16 bits wide, if the @@ -1387,8 +1537,15 @@ struct ice_aq_desc { struct ice_aqc_get_topo get_topo; struct ice_aqc_sched_elem_cmd sched_elem_cmd; struct ice_aqc_query_txsched_res query_sched_res; + struct ice_aqc_query_port_ets port_ets; struct ice_aqc_nvm nvm; struct ice_aqc_pf_vf_msg virt; + struct ice_aqc_lldp_get_mib lldp_get_mib; + struct ice_aqc_lldp_set_mib_change lldp_set_event; + struct ice_aqc_lldp_stop lldp_stop; + struct ice_aqc_lldp_start lldp_start; + struct ice_aqc_lldp_set_local_mib lldp_set_mib; + struct ice_aqc_lldp_stop_start_specific_agent lldp_agent_ctrl; struct ice_aqc_get_set_rss_lut get_set_rss_lut; struct ice_aqc_get_set_rss_key get_set_rss_key; struct ice_aqc_add_txqs add_txqs; @@ -1421,6 +1578,8 @@ struct ice_aq_desc { /* error codes */ enum ice_aq_err { ICE_AQ_RC_OK = 0, /* Success */ + ICE_AQ_RC_EPERM = 1, /* Operation not permitted */ + ICE_AQ_RC_ENOENT = 2, /* No such element */ ICE_AQ_RC_ENOMEM = 9, /* Out of memory */ ICE_AQ_RC_EBUSY = 12, /* Device or resource busy */ ICE_AQ_RC_EEXIST = 13, /* Object already exists */ @@ -1473,6 +1632,7 @@ enum ice_adminq_opc { ice_aqc_opc_get_sched_elems = 0x0404, ice_aqc_opc_suspend_sched_elems = 0x0409, ice_aqc_opc_resume_sched_elems = 0x040A, + ice_aqc_opc_query_port_ets = 0x040E, ice_aqc_opc_delete_sched_elems = 0x040F, ice_aqc_opc_query_sched_res = 0x0412, @@ -1490,6 +1650,14 @@ enum ice_adminq_opc { /* PF/VF mailbox commands */ ice_mbx_opc_send_msg_to_pf = 0x0801, ice_mbx_opc_send_msg_to_vf = 0x0802, + /* LLDP commands */ + ice_aqc_opc_lldp_get_mib = 0x0A00, + ice_aqc_opc_lldp_set_mib_change = 0x0A01, + ice_aqc_opc_lldp_stop = 0x0A05, + ice_aqc_opc_lldp_start = 0x0A06, + ice_aqc_opc_get_cee_dcb_cfg = 0x0A07, + ice_aqc_opc_lldp_set_local_mib = 0x0A08, + ice_aqc_opc_lldp_stop_start_specific_agent = 0x0A09, /* RSS commands */ ice_aqc_opc_set_rss_key = 0x0B02, @@ -1497,7 +1665,7 @@ enum ice_adminq_opc { ice_aqc_opc_get_rss_key = 0x0B04, ice_aqc_opc_get_rss_lut = 0x0B05, - /* TX queue handling commands/events */ + /* Tx queue handling commands/events */ ice_aqc_opc_add_txqs = 0x0C30, ice_aqc_opc_dis_txqs = 0x0C31, diff --git a/drivers/net/ethernet/intel/ice/ice_common.c b/drivers/net/ethernet/intel/ice/ice_common.c index 63f00344130047621cd59b089ee97b66fd5b8529..da7878529929fea347b07164527f2e43860eaf78 100644 --- a/drivers/net/ethernet/intel/ice/ice_common.c +++ b/drivers/net/ethernet/intel/ice/ice_common.c @@ -31,7 +31,7 @@ * @hw: pointer to the HW structure * * This function sets the MAC type of the adapter based on the - * vendor ID and device ID stored in the hw structure. + * vendor ID and device ID stored in the HW structure. */ static enum ice_status ice_set_mac_type(struct ice_hw *hw) { @@ -77,7 +77,7 @@ enum ice_status ice_clear_pf_cfg(struct ice_hw *hw) /** * ice_aq_manage_mac_read - manage MAC address read command - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @buf: a virtual buffer to hold the manage MAC read response * @buf_size: Size of the virtual buffer * @cd: pointer to command details structure or NULL @@ -262,7 +262,7 @@ static enum ice_media_type ice_get_media_type(struct ice_port_info *pi) * * Get Link Status (0x607). Returns the link status of the adapter. */ -static enum ice_status +enum ice_status ice_aq_get_link_info(struct ice_port_info *pi, bool ena_lse, struct ice_link_status *link, struct ice_sq_cd *cd) { @@ -331,7 +331,7 @@ ice_aq_get_link_info(struct ice_port_info *pi, bool ena_lse, /* flag cleared so calling functions don't call AQ again */ pi->phy.get_link_info = false; - return status; + return 0; } /** @@ -358,22 +358,22 @@ static void ice_init_flex_flags(struct ice_hw *hw, enum ice_rxdid prof_id) */ case ICE_RXDID_FLEX_NIC: case ICE_RXDID_FLEX_NIC_2: - ICE_PROG_FLG_ENTRY(hw, prof_id, ICE_RXFLG_PKT_FRG, - ICE_RXFLG_UDP_GRE, ICE_RXFLG_PKT_DSI, - ICE_RXFLG_FIN, idx++); + ICE_PROG_FLG_ENTRY(hw, prof_id, ICE_FLG_PKT_FRG, + ICE_FLG_UDP_GRE, ICE_FLG_PKT_DSI, + ICE_FLG_FIN, idx++); /* flex flag 1 is not used for flexi-flag programming, skipping * these four FLG64 bits. */ - ICE_PROG_FLG_ENTRY(hw, prof_id, ICE_RXFLG_SYN, ICE_RXFLG_RST, - ICE_RXFLG_PKT_DSI, ICE_RXFLG_PKT_DSI, idx++); - ICE_PROG_FLG_ENTRY(hw, prof_id, ICE_RXFLG_PKT_DSI, - ICE_RXFLG_PKT_DSI, ICE_RXFLG_EVLAN_x8100, - ICE_RXFLG_EVLAN_x9100, idx++); - ICE_PROG_FLG_ENTRY(hw, prof_id, ICE_RXFLG_VLAN_x8100, - ICE_RXFLG_TNL_VLAN, ICE_RXFLG_TNL_MAC, - ICE_RXFLG_TNL0, idx++); - ICE_PROG_FLG_ENTRY(hw, prof_id, ICE_RXFLG_TNL1, ICE_RXFLG_TNL2, - ICE_RXFLG_PKT_DSI, ICE_RXFLG_PKT_DSI, idx); + ICE_PROG_FLG_ENTRY(hw, prof_id, ICE_FLG_SYN, ICE_FLG_RST, + ICE_FLG_PKT_DSI, ICE_FLG_PKT_DSI, idx++); + ICE_PROG_FLG_ENTRY(hw, prof_id, ICE_FLG_PKT_DSI, + ICE_FLG_PKT_DSI, ICE_FLG_EVLAN_x8100, + ICE_FLG_EVLAN_x9100, idx++); + ICE_PROG_FLG_ENTRY(hw, prof_id, ICE_FLG_VLAN_x8100, + ICE_FLG_TNL_VLAN, ICE_FLG_TNL_MAC, + ICE_FLG_TNL0, idx++); + ICE_PROG_FLG_ENTRY(hw, prof_id, ICE_FLG_TNL1, ICE_FLG_TNL2, + ICE_FLG_PKT_DSI, ICE_FLG_PKT_DSI, idx); break; default: @@ -418,7 +418,7 @@ static void ice_init_flex_flds(struct ice_hw *hw, enum ice_rxdid prof_id) /** * ice_init_fltr_mgmt_struct - initializes filter management list and locks - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct */ static enum ice_status ice_init_fltr_mgmt_struct(struct ice_hw *hw) { @@ -438,7 +438,7 @@ static enum ice_status ice_init_fltr_mgmt_struct(struct ice_hw *hw) /** * ice_cleanup_fltr_mgmt_struct - cleanup filter management list and locks - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct */ static void ice_cleanup_fltr_mgmt_struct(struct ice_hw *hw) { @@ -477,7 +477,7 @@ static void ice_cleanup_fltr_mgmt_struct(struct ice_hw *hw) /** * ice_cfg_fw_log - configure FW logging - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @enable: enable certain FW logging events if true, disable all if false * * This function enables/disables the FW logging via Rx CQ events and a UART @@ -626,7 +626,7 @@ out: /** * ice_output_fw_log - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @desc: pointer to the AQ message descriptor * @buf: pointer to the buffer accompanying the AQ message * @@ -642,12 +642,12 @@ void ice_output_fw_log(struct ice_hw *hw, struct ice_aq_desc *desc, void *buf) /** * ice_get_itr_intrl_gran - determine int/intrl granularity - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * * Determines the itr/intrl granularities based on the maximum aggregate * bandwidth according to the device's configuration during power-on. */ -static enum ice_status ice_get_itr_intrl_gran(struct ice_hw *hw) +static void ice_get_itr_intrl_gran(struct ice_hw *hw) { u8 max_agg_bw = (rd32(hw, GL_PWR_MODE_CTL) & GL_PWR_MODE_CTL_CAR_MAX_BW_M) >> @@ -664,13 +664,7 @@ static enum ice_status ice_get_itr_intrl_gran(struct ice_hw *hw) hw->itr_gran = ICE_ITR_GRAN_MAX_25; hw->intrl_gran = ICE_INTRL_GRAN_MAX_25; break; - default: - ice_debug(hw, ICE_DBG_INIT, - "Failed to determine itr/intrl granularity\n"); - return ICE_ERR_CFG; } - - return 0; } /** @@ -697,9 +691,7 @@ enum ice_status ice_init_hw(struct ice_hw *hw) if (status) return status; - status = ice_get_itr_intrl_gran(hw); - if (status) - return status; + ice_get_itr_intrl_gran(hw); status = ice_init_all_ctrlq(hw); if (status) @@ -731,7 +723,7 @@ enum ice_status ice_init_hw(struct ice_hw *hw) goto err_unroll_cqinit; } - /* set the back pointer to hw */ + /* set the back pointer to HW */ hw->port_info->hw = hw; /* Initialize port_info struct with switch configuration data */ @@ -988,7 +980,7 @@ enum ice_status ice_reset(struct ice_hw *hw, enum ice_reset_req req) * @ice_rxq_ctx: pointer to the rxq context * @rxq_index: the index of the Rx queue * - * Copies rxq context from dense structure to hw register space + * Copies rxq context from dense structure to HW register space */ static enum ice_status ice_copy_rxq_ctx_to_hw(struct ice_hw *hw, u8 *ice_rxq_ctx, u32 rxq_index) @@ -1001,7 +993,7 @@ ice_copy_rxq_ctx_to_hw(struct ice_hw *hw, u8 *ice_rxq_ctx, u32 rxq_index) if (rxq_index > QRX_CTRL_MAX_INDEX) return ICE_ERR_PARAM; - /* Copy each dword separately to hw */ + /* Copy each dword separately to HW */ for (i = 0; i < ICE_RXQ_CTX_SIZE_DWORDS; i++) { wr32(hw, QRX_CONTEXT(i, rxq_index), *((u32 *)(ice_rxq_ctx + (i * sizeof(u32))))); @@ -1045,7 +1037,7 @@ static const struct ice_ctx_ele ice_rlan_ctx_info[] = { * @rxq_index: the index of the Rx queue * * Converts rxq context from sparse to dense structure and then writes - * it to hw register space + * it to HW register space */ enum ice_status ice_write_rxq_ctx(struct ice_hw *hw, struct ice_rlan_ctx *rlan_ctx, @@ -1100,8 +1092,9 @@ const struct ice_ctx_ele ice_tlan_ctx_info[] = { * * Dumps debug log about control command with descriptor contents. */ -void ice_debug_cq(struct ice_hw *hw, u32 __maybe_unused mask, void *desc, - void *buf, u16 buf_len) +void +ice_debug_cq(struct ice_hw *hw, u32 __maybe_unused mask, void *desc, void *buf, + u16 buf_len) { struct ice_aq_desc *cq_desc = (struct ice_aq_desc *)desc; u16 len; @@ -1143,7 +1136,7 @@ void ice_debug_cq(struct ice_hw *hw, u32 __maybe_unused mask, void *desc, /** * ice_aq_send_cmd - send FW Admin Queue command to FW Admin Queue - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @desc: descriptor describing the command * @buf: buffer to use for indirect commands (NULL for direct commands) * @buf_size: size of buffer for indirect commands (0 for direct commands) @@ -1160,7 +1153,7 @@ ice_aq_send_cmd(struct ice_hw *hw, struct ice_aq_desc *desc, void *buf, /** * ice_aq_get_fw_ver - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @cd: pointer to command details structure or NULL * * Get the firmware version (0x0001) from the admin queue commands @@ -1194,7 +1187,7 @@ enum ice_status ice_aq_get_fw_ver(struct ice_hw *hw, struct ice_sq_cd *cd) /** * ice_aq_q_shutdown - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @unloading: is the driver unloading itself * * Tell the Firmware that we're shutting down the AdminQ and whether @@ -1217,8 +1210,8 @@ enum ice_status ice_aq_q_shutdown(struct ice_hw *hw, bool unloading) /** * ice_aq_req_res - * @hw: pointer to the hw struct - * @res: resource id + * @hw: pointer to the HW struct + * @res: resource ID * @access: access type * @sdp_number: resource number * @timeout: the maximum time in ms that the driver may hold the resource @@ -1303,8 +1296,8 @@ ice_aq_req_res(struct ice_hw *hw, enum ice_aq_res_ids res, /** * ice_aq_release_res - * @hw: pointer to the hw struct - * @res: resource id + * @hw: pointer to the HW struct + * @res: resource ID * @sdp_number: resource number * @cd: pointer to command details structure or NULL * @@ -1330,7 +1323,7 @@ ice_aq_release_res(struct ice_hw *hw, enum ice_aq_res_ids res, u8 sdp_number, /** * ice_acquire_res * @hw: pointer to the HW structure - * @res: resource id + * @res: resource ID * @access: access type (read or write) * @timeout: timeout in milliseconds * @@ -1392,7 +1385,7 @@ ice_acquire_res_exit: /** * ice_release_res * @hw: pointer to the HW structure - * @res: resource id + * @res: resource ID * * This function will release a resource using the proper Admin Command. */ @@ -1404,7 +1397,7 @@ void ice_release_res(struct ice_hw *hw, enum ice_aq_res_ids res) status = ice_aq_release_res(hw, res, 0, NULL); /* there are some rare cases when trying to release the resource - * results in an admin Q timeout, so handle them correctly + * results in an admin queue timeout, so handle them correctly */ while ((status == ICE_ERR_AQ_TIMEOUT) && (total_delay < hw->adminq.sq_cmd_timeout)) { @@ -1415,13 +1408,15 @@ void ice_release_res(struct ice_hw *hw, enum ice_aq_res_ids res) } /** - * ice_get_guar_num_vsi - determine number of guar VSI for a PF - * @hw: pointer to the hw structure + * ice_get_num_per_func - determine number of resources per PF + * @hw: pointer to the HW structure + * @max: value to be evenly split between each PF * * Determine the number of valid functions by going through the bitmap returned - * from parsing capabilities and use this to calculate the number of VSI per PF. + * from parsing capabilities and use this to calculate the number of resources + * per PF based on the max value passed in. */ -static u32 ice_get_guar_num_vsi(struct ice_hw *hw) +static u32 ice_get_num_per_func(struct ice_hw *hw, u32 max) { u8 funcs; @@ -1432,12 +1427,12 @@ static u32 ice_get_guar_num_vsi(struct ice_hw *hw) if (!funcs) return 0; - return ICE_MAX_VSI / funcs; + return max / funcs; } /** * ice_parse_caps - parse function/device capabilities - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @buf: pointer to a buffer containing function/device capability records * @cap_count: number of capability records in the list * @opc: type of capabilities list to parse @@ -1512,7 +1507,8 @@ ice_parse_caps(struct ice_hw *hw, void *buf, u32 cap_count, "HW caps: Dev.VSI cnt = %d\n", dev_p->num_vsi_allocd_to_host); } else if (func_p) { - func_p->guar_num_vsi = ice_get_guar_num_vsi(hw); + func_p->guar_num_vsi = + ice_get_num_per_func(hw, ICE_MAX_VSI); ice_debug(hw, ICE_DBG_INIT, "HW caps: Func.VSI cnt = %d\n", number); @@ -1578,7 +1574,7 @@ ice_parse_caps(struct ice_hw *hw, void *buf, u32 cap_count, /** * ice_aq_discover_caps - query function/device capabilities - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @buf: a virtual buffer to hold the capabilities * @buf_size: Size of the virtual buffer * @cap_count: cap count needed if AQ err==ENOMEM @@ -1617,8 +1613,8 @@ ice_aq_discover_caps(struct ice_hw *hw, void *buf, u16 buf_size, u32 *cap_count, * @hw: pointer to the hardware structure * @opc: capabilities type to discover - pass in the command opcode */ -static enum ice_status ice_discover_caps(struct ice_hw *hw, - enum ice_adminq_opc opc) +static enum ice_status +ice_discover_caps(struct ice_hw *hw, enum ice_adminq_opc opc) { enum ice_status status; u32 cap_count; @@ -1677,7 +1673,7 @@ enum ice_status ice_get_caps(struct ice_hw *hw) /** * ice_aq_manage_mac_write - manage MAC address write command - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @mac_addr: MAC address to be written as LAA/LAA+WoL/Port address * @flags: flags to control write behavior * @cd: pointer to command details structure or NULL @@ -1705,7 +1701,7 @@ ice_aq_manage_mac_write(struct ice_hw *hw, const u8 *mac_addr, u8 flags, /** * ice_aq_clear_pxe_mode - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * * Tell the firmware that the driver is taking over from PXE (0x0110). */ @@ -1721,7 +1717,7 @@ static enum ice_status ice_aq_clear_pxe_mode(struct ice_hw *hw) /** * ice_clear_pxe_mode - clear pxe operations mode - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * * Make sure all PXE mode settings are cleared, including things * like descriptor fetch/write-back mode. @@ -1737,10 +1733,10 @@ void ice_clear_pxe_mode(struct ice_hw *hw) * @phy_type_low: lower part of phy_type * @phy_type_high: higher part of phy_type * - * This helper function will convert an entry in phy type structure + * This helper function will convert an entry in PHY type structure * [phy_type_low, phy_type_high] to its corresponding link speed. * Note: In the structure of [phy_type_low, phy_type_high], there should - * be one bit set, as this function will convert one phy type to its + * be one bit set, as this function will convert one PHY type to its * speed. * If no bit gets set, ICE_LINK_SPEED_UNKNOWN will be returned * If more than one bit gets set, ICE_LINK_SPEED_UNKNOWN will be returned @@ -1884,10 +1880,10 @@ void ice_update_phy_type(u64 *phy_type_low, u64 *phy_type_high, u16 link_speeds_bitmap) { - u16 speed = ICE_AQ_LINK_SPEED_UNKNOWN; u64 pt_high; u64 pt_low; int index; + u16 speed; /* We first check with low part of phy_type */ for (index = 0; index <= ICE_PHY_TYPE_LOW_MAX_INDEX; index++) { @@ -1910,7 +1906,7 @@ ice_update_phy_type(u64 *phy_type_low, u64 *phy_type_high, /** * ice_aq_set_phy_cfg - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @lport: logical port number * @cfg: structure with PHY configuration data to be set * @cd: pointer to command details structure or NULL @@ -1929,6 +1925,15 @@ ice_aq_set_phy_cfg(struct ice_hw *hw, u8 lport, if (!cfg) return ICE_ERR_PARAM; + /* Ensure that only valid bits of cfg->caps can be turned on. */ + if (cfg->caps & ~ICE_AQ_PHY_ENA_VALID_MASK) { + ice_debug(hw, ICE_DBG_PHY, + "Invalid bit is set in ice_aqc_set_phy_cfg_data->caps : 0x%x\n", + cfg->caps); + + cfg->caps &= ICE_AQ_PHY_ENA_VALID_MASK; + } + ice_fill_dflt_direct_cmd_desc(&desc, ice_aqc_opc_set_phy_cfg); desc.params.set_phy.lport_num = lport; desc.flags |= cpu_to_le16(ICE_AQ_FLAG_RD); @@ -2016,7 +2021,7 @@ ice_set_fc(struct ice_port_info *pi, u8 *aq_failures, bool ena_auto_link_update) if (!pcaps) return ICE_ERR_NO_MEMORY; - /* Get the current phy config */ + /* Get the current PHY config */ status = ice_aq_get_phy_caps(pi, false, ICE_AQC_REPORT_SW_CFG, pcaps, NULL); if (status) { @@ -2027,8 +2032,10 @@ ice_set_fc(struct ice_port_info *pi, u8 *aq_failures, bool ena_auto_link_update) /* clear the old pause settings */ cfg.caps = pcaps->caps & ~(ICE_AQC_PHY_EN_TX_LINK_PAUSE | ICE_AQC_PHY_EN_RX_LINK_PAUSE); + /* set the new capabilities */ cfg.caps |= pause_mask; + /* If the capabilities have changed, then set the new config */ if (cfg.caps != pcaps->caps) { int retry_count, retry_max = 10; @@ -2135,6 +2142,32 @@ ice_aq_set_link_restart_an(struct ice_port_info *pi, bool ena_link, return ice_aq_send_cmd(pi->hw, &desc, NULL, 0, cd); } +/** + * ice_aq_set_event_mask + * @hw: pointer to the HW struct + * @port_num: port number of the physical function + * @mask: event mask to be set + * @cd: pointer to command details structure or NULL + * + * Set event mask (0x0613) + */ +enum ice_status +ice_aq_set_event_mask(struct ice_hw *hw, u8 port_num, u16 mask, + struct ice_sq_cd *cd) +{ + struct ice_aqc_set_event_mask *cmd; + struct ice_aq_desc desc; + + cmd = &desc.params.set_event_mask; + + ice_fill_dflt_direct_cmd_desc(&desc, ice_aqc_opc_set_event_mask); + + cmd->lport_num = port_num; + + cmd->event_mask = cpu_to_le16(mask); + return ice_aq_send_cmd(hw, &desc, NULL, 0, cd); +} + /** * ice_aq_set_port_id_led * @pi: pointer to the port information @@ -2297,7 +2330,7 @@ ice_aq_set_rss_lut(struct ice_hw *hw, u16 vsi_handle, u8 lut_type, /** * __ice_aq_get_set_rss_key - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @vsi_id: VSI FW index * @key: pointer to key info struct * @set: set true to set the key, false to get the key @@ -2332,7 +2365,7 @@ ice_status __ice_aq_get_set_rss_key(struct ice_hw *hw, u16 vsi_id, /** * ice_aq_get_rss_key - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @vsi_handle: software VSI handle * @key: pointer to key info struct * @@ -2351,7 +2384,7 @@ ice_aq_get_rss_key(struct ice_hw *hw, u16 vsi_handle, /** * ice_aq_set_rss_key - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @vsi_handle: software VSI handle * @keys: pointer to key info struct * @@ -2436,7 +2469,7 @@ ice_aq_add_lan_txq(struct ice_hw *hw, u8 num_qgrps, * @num_qgrps: number of groups in the list * @qg_list: the list of groups to disable * @buf_size: the total size of the qg_list buffer in bytes - * @rst_src: if called due to reset, specifies the RST source + * @rst_src: if called due to reset, specifies the reset source * @vmvf_num: the relative VM or VF number that is undergoing the reset * @cd: pointer to command details structure or NULL * @@ -2476,7 +2509,7 @@ ice_aq_dis_lan_txq(struct ice_hw *hw, u8 num_qgrps, break; case ICE_VF_RESET: cmd->cmd_type = ICE_AQC_Q_DIS_CMD_VF_RESET; - /* In this case, FW expects vmvf_num to be absolute VF id */ + /* In this case, FW expects vmvf_num to be absolute VF ID */ cmd->vmvf_and_timeout |= cpu_to_le16((vmvf_num + hw->func_caps.vf_base_id) & ICE_AQC_Q_DIS_VMVF_NUM_M); @@ -2534,8 +2567,8 @@ do_aq: * @dest_ctx: the context to be written to * @ce_info: a description of the struct to be filled */ -static void ice_write_byte(u8 *src_ctx, u8 *dest_ctx, - const struct ice_ctx_ele *ce_info) +static void +ice_write_byte(u8 *src_ctx, u8 *dest_ctx, const struct ice_ctx_ele *ce_info) { u8 src_byte, dest_byte, mask; u8 *from, *dest; @@ -2573,8 +2606,8 @@ static void ice_write_byte(u8 *src_ctx, u8 *dest_ctx, * @dest_ctx: the context to be written to * @ce_info: a description of the struct to be filled */ -static void ice_write_word(u8 *src_ctx, u8 *dest_ctx, - const struct ice_ctx_ele *ce_info) +static void +ice_write_word(u8 *src_ctx, u8 *dest_ctx, const struct ice_ctx_ele *ce_info) { u16 src_word, mask; __le16 dest_word; @@ -2616,8 +2649,8 @@ static void ice_write_word(u8 *src_ctx, u8 *dest_ctx, * @dest_ctx: the context to be written to * @ce_info: a description of the struct to be filled */ -static void ice_write_dword(u8 *src_ctx, u8 *dest_ctx, - const struct ice_ctx_ele *ce_info) +static void +ice_write_dword(u8 *src_ctx, u8 *dest_ctx, const struct ice_ctx_ele *ce_info) { u32 src_dword, mask; __le32 dest_dword; @@ -2667,8 +2700,8 @@ static void ice_write_dword(u8 *src_ctx, u8 *dest_ctx, * @dest_ctx: the context to be written to * @ce_info: a description of the struct to be filled */ -static void ice_write_qword(u8 *src_ctx, u8 *dest_ctx, - const struct ice_ctx_ele *ce_info) +static void +ice_write_qword(u8 *src_ctx, u8 *dest_ctx, const struct ice_ctx_ele *ce_info) { u64 src_qword, mask; __le64 dest_qword; @@ -2749,25 +2782,51 @@ ice_set_ctx(u8 *src_ctx, u8 *dest_ctx, const struct ice_ctx_ele *ce_info) return 0; } +/** + * ice_get_lan_q_ctx - get the LAN queue context for the given VSI and TC + * @hw: pointer to the HW struct + * @vsi_handle: software VSI handle + * @tc: TC number + * @q_handle: software queue handle + */ +static struct ice_q_ctx * +ice_get_lan_q_ctx(struct ice_hw *hw, u16 vsi_handle, u8 tc, u16 q_handle) +{ + struct ice_vsi_ctx *vsi; + struct ice_q_ctx *q_ctx; + + vsi = ice_get_vsi_ctx(hw, vsi_handle); + if (!vsi) + return NULL; + if (q_handle >= vsi->num_lan_q_entries[tc]) + return NULL; + if (!vsi->lan_q_ctx[tc]) + return NULL; + q_ctx = vsi->lan_q_ctx[tc]; + return &q_ctx[q_handle]; +} + /** * ice_ena_vsi_txq * @pi: port information structure * @vsi_handle: software VSI handle - * @tc: tc number + * @tc: TC number + * @q_handle: software queue handle * @num_qgrps: Number of added queue groups * @buf: list of queue groups to be added * @buf_size: size of buffer for indirect command * @cd: pointer to command details structure or NULL * - * This function adds one lan q + * This function adds one LAN queue */ enum ice_status -ice_ena_vsi_txq(struct ice_port_info *pi, u16 vsi_handle, u8 tc, u8 num_qgrps, - struct ice_aqc_add_tx_qgrp *buf, u16 buf_size, +ice_ena_vsi_txq(struct ice_port_info *pi, u16 vsi_handle, u8 tc, u16 q_handle, + u8 num_qgrps, struct ice_aqc_add_tx_qgrp *buf, u16 buf_size, struct ice_sq_cd *cd) { struct ice_aqc_txsched_elem_data node = { 0 }; struct ice_sched_node *parent; + struct ice_q_ctx *q_ctx; enum ice_status status; struct ice_hw *hw; @@ -2784,6 +2843,14 @@ ice_ena_vsi_txq(struct ice_port_info *pi, u16 vsi_handle, u8 tc, u8 num_qgrps, mutex_lock(&pi->sched_lock); + q_ctx = ice_get_lan_q_ctx(hw, vsi_handle, tc, q_handle); + if (!q_ctx) { + ice_debug(hw, ICE_DBG_SCHED, "Enaq: invalid queue handle %d\n", + q_handle); + status = ICE_ERR_PARAM; + goto ena_txq_exit; + } + /* find a parent node */ parent = ice_sched_get_free_qparent(pi, vsi_handle, tc, ICE_SCHED_NODE_OWNER_LAN); @@ -2803,14 +2870,14 @@ ice_ena_vsi_txq(struct ice_port_info *pi, u16 vsi_handle, u8 tc, u8 num_qgrps, * Bit 5-6. * - Bit 7 is reserved. * Without setting the generic section as valid in valid_sections, the - * Admin Q command will fail with error code ICE_AQ_RC_EINVAL. + * Admin queue command will fail with error code ICE_AQ_RC_EINVAL. */ buf->txqs[0].info.valid_sections = ICE_AQC_ELEM_VALID_GENERIC; - /* add the lan q */ + /* add the LAN queue */ status = ice_aq_add_lan_txq(hw, num_qgrps, buf, buf_size, cd); if (status) { - ice_debug(hw, ICE_DBG_SCHED, "enable Q %d failed %d\n", + ice_debug(hw, ICE_DBG_SCHED, "enable queue %d failed %d\n", le16_to_cpu(buf->txqs[0].txq_id), hw->adminq.sq_last_status); goto ena_txq_exit; @@ -2818,8 +2885,9 @@ ice_ena_vsi_txq(struct ice_port_info *pi, u16 vsi_handle, u8 tc, u8 num_qgrps, node.node_teid = buf->txqs[0].q_teid; node.data.elem_type = ICE_AQC_ELEM_TYPE_LEAF; + q_ctx->q_handle = q_handle; - /* add a leaf node into schduler tree q layer */ + /* add a leaf node into schduler tree queue layer */ status = ice_sched_add_node(pi, hw->num_tx_sched_layers - 1, &node); ena_txq_exit: @@ -2830,35 +2898,43 @@ ena_txq_exit: /** * ice_dis_vsi_txq * @pi: port information structure + * @vsi_handle: software VSI handle + * @tc: TC number * @num_queues: number of queues + * @q_handles: pointer to software queue handle array * @q_ids: pointer to the q_id array * @q_teids: pointer to queue node teids - * @rst_src: if called due to reset, specifies the RST source + * @rst_src: if called due to reset, specifies the reset source * @vmvf_num: the relative VM or VF number that is undergoing the reset * @cd: pointer to command details structure or NULL * * This function removes queues and their corresponding nodes in SW DB */ enum ice_status -ice_dis_vsi_txq(struct ice_port_info *pi, u8 num_queues, u16 *q_ids, - u32 *q_teids, enum ice_disq_rst_src rst_src, u16 vmvf_num, +ice_dis_vsi_txq(struct ice_port_info *pi, u16 vsi_handle, u8 tc, u8 num_queues, + u16 *q_handles, u16 *q_ids, u32 *q_teids, + enum ice_disq_rst_src rst_src, u16 vmvf_num, struct ice_sq_cd *cd) { enum ice_status status = ICE_ERR_DOES_NOT_EXIST; struct ice_aqc_dis_txq_item qg_list; + struct ice_q_ctx *q_ctx; u16 i; if (!pi || pi->port_state != ICE_SCHED_PORT_STATE_READY) return ICE_ERR_CFG; - /* if queue is disabled already yet the disable queue command has to be - * sent to complete the VF reset, then call ice_aq_dis_lan_txq without - * any queue information - */ - if (!num_queues && rst_src) - return ice_aq_dis_lan_txq(pi->hw, 0, NULL, 0, rst_src, vmvf_num, - NULL); + if (!num_queues) { + /* if queue is disabled already yet the disable queue command + * has to be sent to complete the VF reset, then call + * ice_aq_dis_lan_txq without any queue information + */ + if (rst_src) + return ice_aq_dis_lan_txq(pi->hw, 0, NULL, 0, rst_src, + vmvf_num, NULL); + return ICE_ERR_CFG; + } mutex_lock(&pi->sched_lock); @@ -2868,6 +2944,17 @@ ice_dis_vsi_txq(struct ice_port_info *pi, u8 num_queues, u16 *q_ids, node = ice_sched_find_node_by_teid(pi->root, q_teids[i]); if (!node) continue; + q_ctx = ice_get_lan_q_ctx(pi->hw, vsi_handle, tc, q_handles[i]); + if (!q_ctx) { + ice_debug(pi->hw, ICE_DBG_SCHED, "invalid queue handle%d\n", + q_handles[i]); + continue; + } + if (q_ctx->q_handle != q_handles[i]) { + ice_debug(pi->hw, ICE_DBG_SCHED, "Err:handles %d %d\n", + q_ctx->q_handle, q_handles[i]); + continue; + } qg_list.parent_teid = node->info.parent_teid; qg_list.num_qs = 1; qg_list.q_id[0] = cpu_to_le16(q_ids[i]); @@ -2878,18 +2965,19 @@ ice_dis_vsi_txq(struct ice_port_info *pi, u8 num_queues, u16 *q_ids, if (status) break; ice_free_sched_node(pi, node); + q_ctx->q_handle = ICE_INVAL_Q_HANDLE; } mutex_unlock(&pi->sched_lock); return status; } /** - * ice_cfg_vsi_qs - configure the new/exisiting VSI queues + * ice_cfg_vsi_qs - configure the new/existing VSI queues * @pi: port information structure * @vsi_handle: software VSI handle * @tc_bitmap: TC bitmap * @maxqs: max queues array per TC - * @owner: lan or rdma + * @owner: LAN or RDMA * * This function adds/updates the VSI queues per TC. */ @@ -2908,7 +2996,7 @@ ice_cfg_vsi_qs(struct ice_port_info *pi, u16 vsi_handle, u8 tc_bitmap, mutex_lock(&pi->sched_lock); - for (i = 0; i < ICE_MAX_TRAFFIC_CLASS; i++) { + ice_for_each_traffic_class(i) { /* configuration is possible only if TC node is present */ if (!ice_sched_get_tc_node(pi, i)) continue; @@ -2924,13 +3012,13 @@ ice_cfg_vsi_qs(struct ice_port_info *pi, u16 vsi_handle, u8 tc_bitmap, } /** - * ice_cfg_vsi_lan - configure VSI lan queues + * ice_cfg_vsi_lan - configure VSI LAN queues * @pi: port information structure * @vsi_handle: software VSI handle * @tc_bitmap: TC bitmap - * @max_lanqs: max lan queues array per TC + * @max_lanqs: max LAN queues array per TC * - * This function adds/updates the VSI lan queues per TC. + * This function adds/updates the VSI LAN queues per TC. */ enum ice_status ice_cfg_vsi_lan(struct ice_port_info *pi, u16 vsi_handle, u8 tc_bitmap, @@ -2942,7 +3030,7 @@ ice_cfg_vsi_lan(struct ice_port_info *pi, u16 vsi_handle, u8 tc_bitmap, /** * ice_replay_pre_init - replay pre initialization - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * * Initializes required config data for VSI, FD, ACL, and RSS before replay. */ @@ -2966,7 +3054,7 @@ static enum ice_status ice_replay_pre_init(struct ice_hw *hw) /** * ice_replay_vsi - replay VSI configuration - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @vsi_handle: driver VSI handle * * Restore all VSI configuration after reset. It is required to call this @@ -2993,7 +3081,7 @@ enum ice_status ice_replay_vsi(struct ice_hw *hw, u16 vsi_handle) /** * ice_replay_post - post replay configuration cleanup - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * * Post replay cleanup. */ @@ -3012,8 +3100,9 @@ void ice_replay_post(struct ice_hw *hw) * @prev_stat: ptr to previous loaded stat value * @cur_stat: ptr to current stat value */ -void ice_stat_update40(struct ice_hw *hw, u32 hireg, u32 loreg, - bool prev_stat_loaded, u64 *prev_stat, u64 *cur_stat) +void +ice_stat_update40(struct ice_hw *hw, u32 hireg, u32 loreg, + bool prev_stat_loaded, u64 *prev_stat, u64 *cur_stat) { u64 new_data; @@ -3043,8 +3132,9 @@ void ice_stat_update40(struct ice_hw *hw, u32 hireg, u32 loreg, * @prev_stat: ptr to previous loaded stat value * @cur_stat: ptr to current stat value */ -void ice_stat_update32(struct ice_hw *hw, u32 reg, bool prev_stat_loaded, - u64 *prev_stat, u64 *cur_stat) +void +ice_stat_update32(struct ice_hw *hw, u32 reg, bool prev_stat_loaded, + u64 *prev_stat, u64 *cur_stat) { u32 new_data; @@ -3063,3 +3153,28 @@ void ice_stat_update32(struct ice_hw *hw, u32 reg, bool prev_stat_loaded, /* to manage the potential roll-over */ *cur_stat = (new_data + BIT_ULL(32)) - *prev_stat; } + +/** + * ice_sched_query_elem - query element information from HW + * @hw: pointer to the HW struct + * @node_teid: node TEID to be queried + * @buf: buffer to element information + * + * This function queries HW element information + */ +enum ice_status +ice_sched_query_elem(struct ice_hw *hw, u32 node_teid, + struct ice_aqc_get_elem *buf) +{ + u16 buf_size, num_elem_ret = 0; + enum ice_status status; + + buf_size = sizeof(*buf); + memset(buf, 0, buf_size); + buf->generic[0].node_teid = cpu_to_le32(node_teid); + status = ice_aq_query_sched_elems(hw, 1, buf, buf_size, &num_elem_ret, + NULL); + if (status || num_elem_ret != 1) + ice_debug(hw, ICE_DBG_SCHED, "query element failed\n"); + return status; +} diff --git a/drivers/net/ethernet/intel/ice/ice_common.h b/drivers/net/ethernet/intel/ice/ice_common.h index d7c7c2ed88234df8710593eed9e47d3aa7157ece..f1ddebf45231ebe09c0f4dd5773ee30c5cfed45b 100644 --- a/drivers/net/ethernet/intel/ice/ice_common.h +++ b/drivers/net/ethernet/intel/ice/ice_common.h @@ -9,8 +9,8 @@ #include "ice_switch.h" #include -void ice_debug_cq(struct ice_hw *hw, u32 mask, void *desc, void *buf, - u16 buf_len); +void +ice_debug_cq(struct ice_hw *hw, u32 mask, void *desc, void *buf, u16 buf_len); enum ice_status ice_init_hw(struct ice_hw *hw); void ice_deinit_hw(struct ice_hw *hw); enum ice_status ice_check_reset(struct ice_hw *hw); @@ -28,8 +28,8 @@ ice_acquire_res(struct ice_hw *hw, enum ice_aq_res_ids res, enum ice_aq_res_access_type access, u32 timeout); void ice_release_res(struct ice_hw *hw, enum ice_aq_res_ids res); enum ice_status ice_init_nvm(struct ice_hw *hw); -enum ice_status ice_read_sr_buf(struct ice_hw *hw, u16 offset, u16 *words, - u16 *data); +enum ice_status +ice_read_sr_buf(struct ice_hw *hw, u16 offset, u16 *words, u16 *data); enum ice_status ice_sq_send_cmd(struct ice_hw *hw, struct ice_ctl_q_info *cq, struct ice_aq_desc *desc, void *buf, u16 buf_size, @@ -89,25 +89,37 @@ enum ice_status ice_aq_set_link_restart_an(struct ice_port_info *pi, bool ena_link, struct ice_sq_cd *cd); enum ice_status +ice_aq_get_link_info(struct ice_port_info *pi, bool ena_lse, + struct ice_link_status *link, struct ice_sq_cd *cd); +enum ice_status +ice_aq_set_event_mask(struct ice_hw *hw, u8 port_num, u16 mask, + struct ice_sq_cd *cd); +enum ice_status ice_aq_set_port_id_led(struct ice_port_info *pi, bool is_orig_mode, struct ice_sq_cd *cd); enum ice_status -ice_dis_vsi_txq(struct ice_port_info *pi, u8 num_queues, u16 *q_ids, - u32 *q_teids, enum ice_disq_rst_src rst_src, u16 vmvf_num, - struct ice_sq_cd *cmd_details); +ice_dis_vsi_txq(struct ice_port_info *pi, u16 vsi_handle, u8 tc, u8 num_queues, + u16 *q_handle, u16 *q_ids, u32 *q_teids, + enum ice_disq_rst_src rst_src, u16 vmvf_num, + struct ice_sq_cd *cd); enum ice_status ice_cfg_vsi_lan(struct ice_port_info *pi, u16 vsi_handle, u8 tc_bitmap, u16 *max_lanqs); enum ice_status -ice_ena_vsi_txq(struct ice_port_info *pi, u16 vsi_handle, u8 tc, u8 num_qgrps, - struct ice_aqc_add_tx_qgrp *buf, u16 buf_size, +ice_ena_vsi_txq(struct ice_port_info *pi, u16 vsi_handle, u8 tc, u16 q_handle, + u8 num_qgrps, struct ice_aqc_add_tx_qgrp *buf, u16 buf_size, struct ice_sq_cd *cd); enum ice_status ice_replay_vsi(struct ice_hw *hw, u16 vsi_handle); void ice_replay_post(struct ice_hw *hw); void ice_output_fw_log(struct ice_hw *hw, struct ice_aq_desc *desc, void *buf); -void ice_stat_update40(struct ice_hw *hw, u32 hireg, u32 loreg, - bool prev_stat_loaded, u64 *prev_stat, u64 *cur_stat); -void ice_stat_update32(struct ice_hw *hw, u32 reg, bool prev_stat_loaded, - u64 *prev_stat, u64 *cur_stat); +void +ice_stat_update40(struct ice_hw *hw, u32 hireg, u32 loreg, + bool prev_stat_loaded, u64 *prev_stat, u64 *cur_stat); +void +ice_stat_update32(struct ice_hw *hw, u32 reg, bool prev_stat_loaded, + u64 *prev_stat, u64 *cur_stat); +enum ice_status +ice_sched_query_elem(struct ice_hw *hw, u32 node_teid, + struct ice_aqc_get_elem *buf); #endif /* _ICE_COMMON_H_ */ diff --git a/drivers/net/ethernet/intel/ice/ice_controlq.c b/drivers/net/ethernet/intel/ice/ice_controlq.c index 2bf5e11f559a85a10b38392b307cf0e16bb95f43..cc8cb5fdcdc13262e94d7972d1d3f09c07d837ee 100644 --- a/drivers/net/ethernet/intel/ice/ice_controlq.c +++ b/drivers/net/ethernet/intel/ice/ice_controlq.c @@ -51,7 +51,7 @@ static void ice_mailbox_init_regs(struct ice_hw *hw) /** * ice_check_sq_alive - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @cq: pointer to the specific Control queue * * Returns true if Queue is enabled else false. @@ -287,7 +287,7 @@ ice_cfg_sq_regs(struct ice_hw *hw, struct ice_ctl_q_info *cq) * @hw: pointer to the hardware structure * @cq: pointer to the specific Control queue * - * Configure base address and length registers for the receive (event q) + * Configure base address and length registers for the receive (event queue) */ static enum ice_status ice_cfg_rq_regs(struct ice_hw *hw, struct ice_ctl_q_info *cq) @@ -751,7 +751,7 @@ static u16 ice_clean_sq(struct ice_hw *hw, struct ice_ctl_q_info *cq) /** * ice_sq_done - check if FW has processed the Admin Send Queue (ATQ) - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @cq: pointer to the specific Control queue * * Returns true if the firmware has processed all descriptors on the @@ -767,7 +767,7 @@ static bool ice_sq_done(struct ice_hw *hw, struct ice_ctl_q_info *cq) /** * ice_sq_send_cmd - send command to Control Queue (ATQ) - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @cq: pointer to the specific Control queue * @desc: prefilled descriptor describing the command (non DMA mem) * @buf: buffer to use for indirect commands (or NULL for direct commands) @@ -962,7 +962,7 @@ void ice_fill_dflt_direct_cmd_desc(struct ice_aq_desc *desc, u16 opcode) /** * ice_clean_rq_elem - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @cq: pointer to the specific Control queue * @e: event info from the receive descriptor, includes any buffers * @pending: number of events that could be left to process diff --git a/drivers/net/ethernet/intel/ice/ice_controlq.h b/drivers/net/ethernet/intel/ice/ice_controlq.h index 0038a4109c994942b5d6e9b276cdb6b391cceef7..e0585394d984e818d48684d99c142fba56a876bc 100644 --- a/drivers/net/ethernet/intel/ice/ice_controlq.h +++ b/drivers/net/ethernet/intel/ice/ice_controlq.h @@ -79,6 +79,7 @@ struct ice_rq_event_info { /* Control Queue information */ struct ice_ctl_q_info { enum ice_ctl_q qtype; + enum ice_aq_err rq_last_status; /* last status on receive queue */ struct ice_ctl_q_ring rq; /* receive queue */ struct ice_ctl_q_ring sq; /* send queue */ u32 sq_cmd_timeout; /* send queue cmd write back timeout */ @@ -86,10 +87,9 @@ struct ice_ctl_q_info { u16 num_sq_entries; /* send queue depth */ u16 rq_buf_size; /* receive queue buffer size */ u16 sq_buf_size; /* send queue buffer size */ + enum ice_aq_err sq_last_status; /* last status on send queue */ struct mutex sq_lock; /* Send queue lock */ struct mutex rq_lock; /* Receive queue lock */ - enum ice_aq_err sq_last_status; /* last status on send queue */ - enum ice_aq_err rq_last_status; /* last status on receive queue */ }; #endif /* _ICE_CONTROLQ_H_ */ diff --git a/drivers/net/ethernet/intel/ice/ice_dcb.c b/drivers/net/ethernet/intel/ice/ice_dcb.c new file mode 100644 index 0000000000000000000000000000000000000000..8bbf48e04a1c9101c36c9418903425c1ab0dc01b --- /dev/null +++ b/drivers/net/ethernet/intel/ice/ice_dcb.c @@ -0,0 +1,1392 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2019, Intel Corporation. */ + +#include "ice_common.h" +#include "ice_sched.h" +#include "ice_dcb.h" + +/** + * ice_aq_get_lldp_mib + * @hw: pointer to the HW struct + * @bridge_type: type of bridge requested + * @mib_type: Local, Remote or both Local and Remote MIBs + * @buf: pointer to the caller-supplied buffer to store the MIB block + * @buf_size: size of the buffer (in bytes) + * @local_len: length of the returned Local LLDP MIB + * @remote_len: length of the returned Remote LLDP MIB + * @cd: pointer to command details structure or NULL + * + * Requests the complete LLDP MIB (entire packet). (0x0A00) + */ +static enum ice_status +ice_aq_get_lldp_mib(struct ice_hw *hw, u8 bridge_type, u8 mib_type, void *buf, + u16 buf_size, u16 *local_len, u16 *remote_len, + struct ice_sq_cd *cd) +{ + struct ice_aqc_lldp_get_mib *cmd; + struct ice_aq_desc desc; + enum ice_status status; + + cmd = &desc.params.lldp_get_mib; + + if (buf_size == 0 || !buf) + return ICE_ERR_PARAM; + + ice_fill_dflt_direct_cmd_desc(&desc, ice_aqc_opc_lldp_get_mib); + + cmd->type = mib_type & ICE_AQ_LLDP_MIB_TYPE_M; + cmd->type |= (bridge_type << ICE_AQ_LLDP_BRID_TYPE_S) & + ICE_AQ_LLDP_BRID_TYPE_M; + + desc.datalen = cpu_to_le16(buf_size); + + status = ice_aq_send_cmd(hw, &desc, buf, buf_size, cd); + if (!status) { + if (local_len) + *local_len = le16_to_cpu(cmd->local_len); + if (remote_len) + *remote_len = le16_to_cpu(cmd->remote_len); + } + + return status; +} + +/** + * ice_aq_cfg_lldp_mib_change + * @hw: pointer to the HW struct + * @ena_update: Enable or Disable event posting + * @cd: pointer to command details structure or NULL + * + * Enable or Disable posting of an event on ARQ when LLDP MIB + * associated with the interface changes (0x0A01) + */ +enum ice_status +ice_aq_cfg_lldp_mib_change(struct ice_hw *hw, bool ena_update, + struct ice_sq_cd *cd) +{ + struct ice_aqc_lldp_set_mib_change *cmd; + struct ice_aq_desc desc; + + cmd = &desc.params.lldp_set_event; + + ice_fill_dflt_direct_cmd_desc(&desc, ice_aqc_opc_lldp_set_mib_change); + + if (!ena_update) + cmd->command |= ICE_AQ_LLDP_MIB_UPDATE_DIS; + + return ice_aq_send_cmd(hw, &desc, NULL, 0, cd); +} + +/** + * ice_aq_stop_lldp + * @hw: pointer to the HW struct + * @shutdown_lldp_agent: True if LLDP Agent needs to be Shutdown + * False if LLDP Agent needs to be Stopped + * @cd: pointer to command details structure or NULL + * + * Stop or Shutdown the embedded LLDP Agent (0x0A05) + */ +enum ice_status +ice_aq_stop_lldp(struct ice_hw *hw, bool shutdown_lldp_agent, + struct ice_sq_cd *cd) +{ + struct ice_aqc_lldp_stop *cmd; + struct ice_aq_desc desc; + + cmd = &desc.params.lldp_stop; + + ice_fill_dflt_direct_cmd_desc(&desc, ice_aqc_opc_lldp_stop); + + if (shutdown_lldp_agent) + cmd->command |= ICE_AQ_LLDP_AGENT_SHUTDOWN; + + return ice_aq_send_cmd(hw, &desc, NULL, 0, cd); +} + +/** + * ice_aq_start_lldp + * @hw: pointer to the HW struct + * @cd: pointer to command details structure or NULL + * + * Start the embedded LLDP Agent on all ports. (0x0A06) + */ +enum ice_status ice_aq_start_lldp(struct ice_hw *hw, struct ice_sq_cd *cd) +{ + struct ice_aqc_lldp_start *cmd; + struct ice_aq_desc desc; + + cmd = &desc.params.lldp_start; + + ice_fill_dflt_direct_cmd_desc(&desc, ice_aqc_opc_lldp_start); + + cmd->command = ICE_AQ_LLDP_AGENT_START; + + return ice_aq_send_cmd(hw, &desc, NULL, 0, cd); +} + +/** + * ice_aq_set_lldp_mib - Set the LLDP MIB + * @hw: pointer to the HW struct + * @mib_type: Local, Remote or both Local and Remote MIBs + * @buf: pointer to the caller-supplied buffer to store the MIB block + * @buf_size: size of the buffer (in bytes) + * @cd: pointer to command details structure or NULL + * + * Set the LLDP MIB. (0x0A08) + */ +static enum ice_status +ice_aq_set_lldp_mib(struct ice_hw *hw, u8 mib_type, void *buf, u16 buf_size, + struct ice_sq_cd *cd) +{ + struct ice_aqc_lldp_set_local_mib *cmd; + struct ice_aq_desc desc; + + cmd = &desc.params.lldp_set_mib; + + if (buf_size == 0 || !buf) + return ICE_ERR_PARAM; + + ice_fill_dflt_direct_cmd_desc(&desc, ice_aqc_opc_lldp_set_local_mib); + + desc.flags |= cpu_to_le16((u16)ICE_AQ_FLAG_RD); + desc.datalen = cpu_to_le16(buf_size); + + cmd->type = mib_type; + cmd->length = cpu_to_le16(buf_size); + + return ice_aq_send_cmd(hw, &desc, buf, buf_size, cd); +} + +/** + * ice_get_dcbx_status + * @hw: pointer to the HW struct + * + * Get the DCBX status from the Firmware + */ +u8 ice_get_dcbx_status(struct ice_hw *hw) +{ + u32 reg; + + reg = rd32(hw, PRTDCB_GENS); + return (u8)((reg & PRTDCB_GENS_DCBX_STATUS_M) >> + PRTDCB_GENS_DCBX_STATUS_S); +} + +/** + * ice_parse_ieee_ets_common_tlv + * @buf: Data buffer to be parsed for ETS CFG/REC data + * @ets_cfg: Container to store parsed data + * + * Parses the common data of IEEE 802.1Qaz ETS CFG/REC TLV + */ +static void +ice_parse_ieee_ets_common_tlv(u8 *buf, struct ice_dcb_ets_cfg *ets_cfg) +{ + u8 offset = 0; + int i; + + /* Priority Assignment Table (4 octets) + * Octets:| 1 | 2 | 3 | 4 | + * ----------------------------------------- + * |pri0|pri1|pri2|pri3|pri4|pri5|pri6|pri7| + * ----------------------------------------- + * Bits:|7 4|3 0|7 4|3 0|7 4|3 0|7 4|3 0| + * ----------------------------------------- + */ + for (i = 0; i < 4; i++) { + ets_cfg->prio_table[i * 2] = + ((buf[offset] & ICE_IEEE_ETS_PRIO_1_M) >> + ICE_IEEE_ETS_PRIO_1_S); + ets_cfg->prio_table[i * 2 + 1] = + ((buf[offset] & ICE_IEEE_ETS_PRIO_0_M) >> + ICE_IEEE_ETS_PRIO_0_S); + offset++; + } + + /* TC Bandwidth Table (8 octets) + * Octets:| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | + * --------------------------------- + * |tc0|tc1|tc2|tc3|tc4|tc5|tc6|tc7| + * --------------------------------- + * + * TSA Assignment Table (8 octets) + * Octets:| 9 | 10| 11| 12| 13| 14| 15| 16| + * --------------------------------- + * |tc0|tc1|tc2|tc3|tc4|tc5|tc6|tc7| + * --------------------------------- + */ + ice_for_each_traffic_class(i) { + ets_cfg->tcbwtable[i] = buf[offset]; + ets_cfg->tsatable[i] = buf[ICE_MAX_TRAFFIC_CLASS + offset++]; + } +} + +/** + * ice_parse_ieee_etscfg_tlv + * @tlv: IEEE 802.1Qaz ETS CFG TLV + * @dcbcfg: Local store to update ETS CFG data + * + * Parses IEEE 802.1Qaz ETS CFG TLV + */ +static void +ice_parse_ieee_etscfg_tlv(struct ice_lldp_org_tlv *tlv, + struct ice_dcbx_cfg *dcbcfg) +{ + struct ice_dcb_ets_cfg *etscfg; + u8 *buf = tlv->tlvinfo; + + /* First Octet post subtype + * -------------------------- + * |will-|CBS | Re- | Max | + * |ing | |served| TCs | + * -------------------------- + * |1bit | 1bit|3 bits|3bits| + */ + etscfg = &dcbcfg->etscfg; + etscfg->willing = ((buf[0] & ICE_IEEE_ETS_WILLING_M) >> + ICE_IEEE_ETS_WILLING_S); + etscfg->cbs = ((buf[0] & ICE_IEEE_ETS_CBS_M) >> ICE_IEEE_ETS_CBS_S); + etscfg->maxtcs = ((buf[0] & ICE_IEEE_ETS_MAXTC_M) >> + ICE_IEEE_ETS_MAXTC_S); + + /* Begin parsing at Priority Assignment Table (offset 1 in buf) */ + ice_parse_ieee_ets_common_tlv(&buf[1], etscfg); +} + +/** + * ice_parse_ieee_etsrec_tlv + * @tlv: IEEE 802.1Qaz ETS REC TLV + * @dcbcfg: Local store to update ETS REC data + * + * Parses IEEE 802.1Qaz ETS REC TLV + */ +static void +ice_parse_ieee_etsrec_tlv(struct ice_lldp_org_tlv *tlv, + struct ice_dcbx_cfg *dcbcfg) +{ + u8 *buf = tlv->tlvinfo; + + /* Begin parsing at Priority Assignment Table (offset 1 in buf) */ + ice_parse_ieee_ets_common_tlv(&buf[1], &dcbcfg->etsrec); +} + +/** + * ice_parse_ieee_pfccfg_tlv + * @tlv: IEEE 802.1Qaz PFC CFG TLV + * @dcbcfg: Local store to update PFC CFG data + * + * Parses IEEE 802.1Qaz PFC CFG TLV + */ +static void +ice_parse_ieee_pfccfg_tlv(struct ice_lldp_org_tlv *tlv, + struct ice_dcbx_cfg *dcbcfg) +{ + u8 *buf = tlv->tlvinfo; + + /* ---------------------------------------- + * |will-|MBC | Re- | PFC | PFC Enable | + * |ing | |served| cap | | + * ----------------------------------------- + * |1bit | 1bit|2 bits|4bits| 1 octet | + */ + dcbcfg->pfc.willing = ((buf[0] & ICE_IEEE_PFC_WILLING_M) >> + ICE_IEEE_PFC_WILLING_S); + dcbcfg->pfc.mbc = ((buf[0] & ICE_IEEE_PFC_MBC_M) >> ICE_IEEE_PFC_MBC_S); + dcbcfg->pfc.pfccap = ((buf[0] & ICE_IEEE_PFC_CAP_M) >> + ICE_IEEE_PFC_CAP_S); + dcbcfg->pfc.pfcena = buf[1]; +} + +/** + * ice_parse_ieee_app_tlv + * @tlv: IEEE 802.1Qaz APP TLV + * @dcbcfg: Local store to update APP PRIO data + * + * Parses IEEE 802.1Qaz APP PRIO TLV + */ +static void +ice_parse_ieee_app_tlv(struct ice_lldp_org_tlv *tlv, + struct ice_dcbx_cfg *dcbcfg) +{ + u16 offset = 0; + u16 typelen; + int i = 0; + u16 len; + u8 *buf; + + typelen = ntohs(tlv->typelen); + len = ((typelen & ICE_LLDP_TLV_LEN_M) >> ICE_LLDP_TLV_LEN_S); + buf = tlv->tlvinfo; + + /* Removing sizeof(ouisubtype) and reserved byte from len. + * Remaining len div 3 is number of APP TLVs. + */ + len -= (sizeof(tlv->ouisubtype) + 1); + + /* Move offset to App Priority Table */ + offset++; + + /* Application Priority Table (3 octets) + * Octets:| 1 | 2 | 3 | + * ----------------------------------------- + * |Priority|Rsrvd| Sel | Protocol ID | + * ----------------------------------------- + * Bits:|23 21|20 19|18 16|15 0| + * ----------------------------------------- + */ + while (offset < len) { + dcbcfg->app[i].priority = ((buf[offset] & + ICE_IEEE_APP_PRIO_M) >> + ICE_IEEE_APP_PRIO_S); + dcbcfg->app[i].selector = ((buf[offset] & + ICE_IEEE_APP_SEL_M) >> + ICE_IEEE_APP_SEL_S); + dcbcfg->app[i].prot_id = (buf[offset + 1] << 0x8) | + buf[offset + 2]; + /* Move to next app */ + offset += 3; + i++; + if (i >= ICE_DCBX_MAX_APPS) + break; + } + + dcbcfg->numapps = i; +} + +/** + * ice_parse_ieee_tlv + * @tlv: IEEE 802.1Qaz TLV + * @dcbcfg: Local store to update ETS REC data + * + * Get the TLV subtype and send it to parsing function + * based on the subtype value + */ +static void +ice_parse_ieee_tlv(struct ice_lldp_org_tlv *tlv, struct ice_dcbx_cfg *dcbcfg) +{ + u32 ouisubtype; + u8 subtype; + + ouisubtype = ntohl(tlv->ouisubtype); + subtype = (u8)((ouisubtype & ICE_LLDP_TLV_SUBTYPE_M) >> + ICE_LLDP_TLV_SUBTYPE_S); + switch (subtype) { + case ICE_IEEE_SUBTYPE_ETS_CFG: + ice_parse_ieee_etscfg_tlv(tlv, dcbcfg); + break; + case ICE_IEEE_SUBTYPE_ETS_REC: + ice_parse_ieee_etsrec_tlv(tlv, dcbcfg); + break; + case ICE_IEEE_SUBTYPE_PFC_CFG: + ice_parse_ieee_pfccfg_tlv(tlv, dcbcfg); + break; + case ICE_IEEE_SUBTYPE_APP_PRI: + ice_parse_ieee_app_tlv(tlv, dcbcfg); + break; + default: + break; + } +} + +/** + * ice_parse_cee_pgcfg_tlv + * @tlv: CEE DCBX PG CFG TLV + * @dcbcfg: Local store to update ETS CFG data + * + * Parses CEE DCBX PG CFG TLV + */ +static void +ice_parse_cee_pgcfg_tlv(struct ice_cee_feat_tlv *tlv, + struct ice_dcbx_cfg *dcbcfg) +{ + struct ice_dcb_ets_cfg *etscfg; + u8 *buf = tlv->tlvinfo; + u16 offset = 0; + int i; + + etscfg = &dcbcfg->etscfg; + + if (tlv->en_will_err & ICE_CEE_FEAT_TLV_WILLING_M) + etscfg->willing = 1; + + etscfg->cbs = 0; + /* Priority Group Table (4 octets) + * Octets:| 1 | 2 | 3 | 4 | + * ----------------------------------------- + * |pri0|pri1|pri2|pri3|pri4|pri5|pri6|pri7| + * ----------------------------------------- + * Bits:|7 4|3 0|7 4|3 0|7 4|3 0|7 4|3 0| + * ----------------------------------------- + */ + for (i = 0; i < 4; i++) { + etscfg->prio_table[i * 2] = + ((buf[offset] & ICE_CEE_PGID_PRIO_1_M) >> + ICE_CEE_PGID_PRIO_1_S); + etscfg->prio_table[i * 2 + 1] = + ((buf[offset] & ICE_CEE_PGID_PRIO_0_M) >> + ICE_CEE_PGID_PRIO_0_S); + offset++; + } + + /* PG Percentage Table (8 octets) + * Octets:| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | + * --------------------------------- + * |pg0|pg1|pg2|pg3|pg4|pg5|pg6|pg7| + * --------------------------------- + */ + ice_for_each_traffic_class(i) + etscfg->tcbwtable[i] = buf[offset++]; + + /* Number of TCs supported (1 octet) */ + etscfg->maxtcs = buf[offset]; +} + +/** + * ice_parse_cee_pfccfg_tlv + * @tlv: CEE DCBX PFC CFG TLV + * @dcbcfg: Local store to update PFC CFG data + * + * Parses CEE DCBX PFC CFG TLV + */ +static void +ice_parse_cee_pfccfg_tlv(struct ice_cee_feat_tlv *tlv, + struct ice_dcbx_cfg *dcbcfg) +{ + u8 *buf = tlv->tlvinfo; + + if (tlv->en_will_err & ICE_CEE_FEAT_TLV_WILLING_M) + dcbcfg->pfc.willing = 1; + + /* ------------------------ + * | PFC Enable | PFC TCs | + * ------------------------ + * | 1 octet | 1 octet | + */ + dcbcfg->pfc.pfcena = buf[0]; + dcbcfg->pfc.pfccap = buf[1]; +} + +/** + * ice_parse_cee_app_tlv + * @tlv: CEE DCBX APP TLV + * @dcbcfg: Local store to update APP PRIO data + * + * Parses CEE DCBX APP PRIO TLV + */ +static void +ice_parse_cee_app_tlv(struct ice_cee_feat_tlv *tlv, struct ice_dcbx_cfg *dcbcfg) +{ + u16 len, typelen, offset = 0; + struct ice_cee_app_prio *app; + u8 i; + + typelen = ntohs(tlv->hdr.typelen); + len = ((typelen & ICE_LLDP_TLV_LEN_M) >> ICE_LLDP_TLV_LEN_S); + + dcbcfg->numapps = len / sizeof(*app); + if (!dcbcfg->numapps) + return; + if (dcbcfg->numapps > ICE_DCBX_MAX_APPS) + dcbcfg->numapps = ICE_DCBX_MAX_APPS; + + for (i = 0; i < dcbcfg->numapps; i++) { + u8 up, selector; + + app = (struct ice_cee_app_prio *)(tlv->tlvinfo + offset); + for (up = 0; up < ICE_MAX_USER_PRIORITY; up++) + if (app->prio_map & BIT(up)) + break; + + dcbcfg->app[i].priority = up; + + /* Get Selector from lower 2 bits, and convert to IEEE */ + selector = (app->upper_oui_sel & ICE_CEE_APP_SELECTOR_M); + switch (selector) { + case ICE_CEE_APP_SEL_ETHTYPE: + dcbcfg->app[i].selector = ICE_APP_SEL_ETHTYPE; + break; + case ICE_CEE_APP_SEL_TCPIP: + dcbcfg->app[i].selector = ICE_APP_SEL_TCPIP; + break; + default: + /* Keep selector as it is for unknown types */ + dcbcfg->app[i].selector = selector; + } + + dcbcfg->app[i].prot_id = ntohs(app->protocol); + /* Move to next app */ + offset += sizeof(*app); + } +} + +/** + * ice_parse_cee_tlv + * @tlv: CEE DCBX TLV + * @dcbcfg: Local store to update DCBX config data + * + * Get the TLV subtype and send it to parsing function + * based on the subtype value + */ +static void +ice_parse_cee_tlv(struct ice_lldp_org_tlv *tlv, struct ice_dcbx_cfg *dcbcfg) +{ + struct ice_cee_feat_tlv *sub_tlv; + u8 subtype, feat_tlv_count = 0; + u16 len, tlvlen, typelen; + u32 ouisubtype; + + ouisubtype = ntohl(tlv->ouisubtype); + subtype = (u8)((ouisubtype & ICE_LLDP_TLV_SUBTYPE_M) >> + ICE_LLDP_TLV_SUBTYPE_S); + /* Return if not CEE DCBX */ + if (subtype != ICE_CEE_DCBX_TYPE) + return; + + typelen = ntohs(tlv->typelen); + tlvlen = ((typelen & ICE_LLDP_TLV_LEN_M) >> ICE_LLDP_TLV_LEN_S); + len = sizeof(tlv->typelen) + sizeof(ouisubtype) + + sizeof(struct ice_cee_ctrl_tlv); + /* Return if no CEE DCBX Feature TLVs */ + if (tlvlen <= len) + return; + + sub_tlv = (struct ice_cee_feat_tlv *)((char *)tlv + len); + while (feat_tlv_count < ICE_CEE_MAX_FEAT_TYPE) { + u16 sublen; + + typelen = ntohs(sub_tlv->hdr.typelen); + sublen = ((typelen & ICE_LLDP_TLV_LEN_M) >> ICE_LLDP_TLV_LEN_S); + subtype = (u8)((typelen & ICE_LLDP_TLV_TYPE_M) >> + ICE_LLDP_TLV_TYPE_S); + switch (subtype) { + case ICE_CEE_SUBTYPE_PG_CFG: + ice_parse_cee_pgcfg_tlv(sub_tlv, dcbcfg); + break; + case ICE_CEE_SUBTYPE_PFC_CFG: + ice_parse_cee_pfccfg_tlv(sub_tlv, dcbcfg); + break; + case ICE_CEE_SUBTYPE_APP_PRI: + ice_parse_cee_app_tlv(sub_tlv, dcbcfg); + break; + default: + return; /* Invalid Sub-type return */ + } + feat_tlv_count++; + /* Move to next sub TLV */ + sub_tlv = (struct ice_cee_feat_tlv *) + ((char *)sub_tlv + sizeof(sub_tlv->hdr.typelen) + + sublen); + } +} + +/** + * ice_parse_org_tlv + * @tlv: Organization specific TLV + * @dcbcfg: Local store to update ETS REC data + * + * Currently only IEEE 802.1Qaz TLV is supported, all others + * will be returned + */ +static void +ice_parse_org_tlv(struct ice_lldp_org_tlv *tlv, struct ice_dcbx_cfg *dcbcfg) +{ + u32 ouisubtype; + u32 oui; + + ouisubtype = ntohl(tlv->ouisubtype); + oui = ((ouisubtype & ICE_LLDP_TLV_OUI_M) >> ICE_LLDP_TLV_OUI_S); + switch (oui) { + case ICE_IEEE_8021QAZ_OUI: + ice_parse_ieee_tlv(tlv, dcbcfg); + break; + case ICE_CEE_DCBX_OUI: + ice_parse_cee_tlv(tlv, dcbcfg); + break; + default: + break; + } +} + +/** + * ice_lldp_to_dcb_cfg + * @lldpmib: LLDPDU to be parsed + * @dcbcfg: store for LLDPDU data + * + * Parse DCB configuration from the LLDPDU + */ +enum ice_status ice_lldp_to_dcb_cfg(u8 *lldpmib, struct ice_dcbx_cfg *dcbcfg) +{ + struct ice_lldp_org_tlv *tlv; + enum ice_status ret = 0; + u16 offset = 0; + u16 typelen; + u16 type; + u16 len; + + if (!lldpmib || !dcbcfg) + return ICE_ERR_PARAM; + + /* set to the start of LLDPDU */ + lldpmib += ETH_HLEN; + tlv = (struct ice_lldp_org_tlv *)lldpmib; + while (1) { + typelen = ntohs(tlv->typelen); + type = ((typelen & ICE_LLDP_TLV_TYPE_M) >> ICE_LLDP_TLV_TYPE_S); + len = ((typelen & ICE_LLDP_TLV_LEN_M) >> ICE_LLDP_TLV_LEN_S); + offset += sizeof(typelen) + len; + + /* END TLV or beyond LLDPDU size */ + if (type == ICE_TLV_TYPE_END || offset > ICE_LLDPDU_SIZE) + break; + + switch (type) { + case ICE_TLV_TYPE_ORG: + ice_parse_org_tlv(tlv, dcbcfg); + break; + default: + break; + } + + /* Move to next TLV */ + tlv = (struct ice_lldp_org_tlv *) + ((char *)tlv + sizeof(tlv->typelen) + len); + } + + return ret; +} + +/** + * ice_aq_get_dcb_cfg + * @hw: pointer to the HW struct + * @mib_type: mib type for the query + * @bridgetype: bridge type for the query (remote) + * @dcbcfg: store for LLDPDU data + * + * Query DCB configuration from the firmware + */ +static enum ice_status +ice_aq_get_dcb_cfg(struct ice_hw *hw, u8 mib_type, u8 bridgetype, + struct ice_dcbx_cfg *dcbcfg) +{ + enum ice_status ret; + u8 *lldpmib; + + /* Allocate the LLDPDU */ + lldpmib = devm_kzalloc(ice_hw_to_dev(hw), ICE_LLDPDU_SIZE, GFP_KERNEL); + if (!lldpmib) + return ICE_ERR_NO_MEMORY; + + ret = ice_aq_get_lldp_mib(hw, bridgetype, mib_type, (void *)lldpmib, + ICE_LLDPDU_SIZE, NULL, NULL, NULL); + + if (!ret) + /* Parse LLDP MIB to get DCB configuration */ + ret = ice_lldp_to_dcb_cfg(lldpmib, dcbcfg); + + devm_kfree(ice_hw_to_dev(hw), lldpmib); + + return ret; +} + +/** + * ice_aq_start_stop_dcbx - Start/Stop DCBx service in FW + * @hw: pointer to the HW struct + * @start_dcbx_agent: True if DCBx Agent needs to be started + * False if DCBx Agent needs to be stopped + * @dcbx_agent_status: FW indicates back the DCBx agent status + * True if DCBx Agent is active + * False if DCBx Agent is stopped + * @cd: pointer to command details structure or NULL + * + * Start/Stop the embedded dcbx Agent. In case that this wrapper function + * returns ICE_SUCCESS, caller will need to check if FW returns back the same + * value as stated in dcbx_agent_status, and react accordingly. (0x0A09) + */ +enum ice_status +ice_aq_start_stop_dcbx(struct ice_hw *hw, bool start_dcbx_agent, + bool *dcbx_agent_status, struct ice_sq_cd *cd) +{ + struct ice_aqc_lldp_stop_start_specific_agent *cmd; + enum ice_status status; + struct ice_aq_desc desc; + u16 opcode; + + cmd = &desc.params.lldp_agent_ctrl; + + opcode = ice_aqc_opc_lldp_stop_start_specific_agent; + + ice_fill_dflt_direct_cmd_desc(&desc, opcode); + + if (start_dcbx_agent) + cmd->command = ICE_AQC_START_STOP_AGENT_START_DCBX; + + status = ice_aq_send_cmd(hw, &desc, NULL, 0, cd); + + *dcbx_agent_status = false; + + if (!status && + cmd->command == ICE_AQC_START_STOP_AGENT_START_DCBX) + *dcbx_agent_status = true; + + return status; +} + +/** + * ice_aq_get_cee_dcb_cfg + * @hw: pointer to the HW struct + * @buff: response buffer that stores CEE operational configuration + * @cd: pointer to command details structure or NULL + * + * Get CEE DCBX mode operational configuration from firmware (0x0A07) + */ +static enum ice_status +ice_aq_get_cee_dcb_cfg(struct ice_hw *hw, + struct ice_aqc_get_cee_dcb_cfg_resp *buff, + struct ice_sq_cd *cd) +{ + struct ice_aq_desc desc; + + ice_fill_dflt_direct_cmd_desc(&desc, ice_aqc_opc_get_cee_dcb_cfg); + + return ice_aq_send_cmd(hw, &desc, (void *)buff, sizeof(*buff), cd); +} + +/** + * ice_cee_to_dcb_cfg + * @cee_cfg: pointer to CEE configuration struct + * @dcbcfg: DCB configuration struct + * + * Convert CEE configuration from firmware to DCB configuration + */ +static void +ice_cee_to_dcb_cfg(struct ice_aqc_get_cee_dcb_cfg_resp *cee_cfg, + struct ice_dcbx_cfg *dcbcfg) +{ + u32 status, tlv_status = le32_to_cpu(cee_cfg->tlv_status); + u32 ice_aqc_cee_status_mask, ice_aqc_cee_status_shift; + u16 app_prio = le16_to_cpu(cee_cfg->oper_app_prio); + u8 i, err, sync, oper, app_index, ice_app_sel_type; + u16 ice_aqc_cee_app_mask, ice_aqc_cee_app_shift; + u16 ice_app_prot_id_type; + + /* CEE PG data to ETS config */ + dcbcfg->etscfg.maxtcs = cee_cfg->oper_num_tc; + + /* Note that the FW creates the oper_prio_tc nibbles reversed + * from those in the CEE Priority Group sub-TLV. + */ + for (i = 0; i < ICE_MAX_TRAFFIC_CLASS / 2; i++) { + dcbcfg->etscfg.prio_table[i * 2] = + ((cee_cfg->oper_prio_tc[i] & ICE_CEE_PGID_PRIO_0_M) >> + ICE_CEE_PGID_PRIO_0_S); + dcbcfg->etscfg.prio_table[i * 2 + 1] = + ((cee_cfg->oper_prio_tc[i] & ICE_CEE_PGID_PRIO_1_M) >> + ICE_CEE_PGID_PRIO_1_S); + } + + ice_for_each_traffic_class(i) { + dcbcfg->etscfg.tcbwtable[i] = cee_cfg->oper_tc_bw[i]; + + if (dcbcfg->etscfg.prio_table[i] == ICE_CEE_PGID_STRICT) { + /* Map it to next empty TC */ + dcbcfg->etscfg.prio_table[i] = cee_cfg->oper_num_tc - 1; + dcbcfg->etscfg.tsatable[i] = ICE_IEEE_TSA_STRICT; + } else { + dcbcfg->etscfg.tsatable[i] = ICE_IEEE_TSA_ETS; + } + } + + /* CEE PFC data to ETS config */ + dcbcfg->pfc.pfcena = cee_cfg->oper_pfc_en; + dcbcfg->pfc.pfccap = ICE_MAX_TRAFFIC_CLASS; + + app_index = 0; + for (i = 0; i < 3; i++) { + if (i == 0) { + /* FCoE APP */ + ice_aqc_cee_status_mask = ICE_AQC_CEE_FCOE_STATUS_M; + ice_aqc_cee_status_shift = ICE_AQC_CEE_FCOE_STATUS_S; + ice_aqc_cee_app_mask = ICE_AQC_CEE_APP_FCOE_M; + ice_aqc_cee_app_shift = ICE_AQC_CEE_APP_FCOE_S; + ice_app_sel_type = ICE_APP_SEL_ETHTYPE; + ice_app_prot_id_type = ICE_APP_PROT_ID_FCOE; + } else if (i == 1) { + /* iSCSI APP */ + ice_aqc_cee_status_mask = ICE_AQC_CEE_ISCSI_STATUS_M; + ice_aqc_cee_status_shift = ICE_AQC_CEE_ISCSI_STATUS_S; + ice_aqc_cee_app_mask = ICE_AQC_CEE_APP_ISCSI_M; + ice_aqc_cee_app_shift = ICE_AQC_CEE_APP_ISCSI_S; + ice_app_sel_type = ICE_APP_SEL_TCPIP; + ice_app_prot_id_type = ICE_APP_PROT_ID_ISCSI; + } else { + /* FIP APP */ + ice_aqc_cee_status_mask = ICE_AQC_CEE_FIP_STATUS_M; + ice_aqc_cee_status_shift = ICE_AQC_CEE_FIP_STATUS_S; + ice_aqc_cee_app_mask = ICE_AQC_CEE_APP_FIP_M; + ice_aqc_cee_app_shift = ICE_AQC_CEE_APP_FIP_S; + ice_app_sel_type = ICE_APP_SEL_ETHTYPE; + ice_app_prot_id_type = ICE_APP_PROT_ID_FIP; + } + + status = (tlv_status & ice_aqc_cee_status_mask) >> + ice_aqc_cee_status_shift; + err = (status & ICE_TLV_STATUS_ERR) ? 1 : 0; + sync = (status & ICE_TLV_STATUS_SYNC) ? 1 : 0; + oper = (status & ICE_TLV_STATUS_OPER) ? 1 : 0; + /* Add FCoE/iSCSI/FIP APP if Error is False and + * Oper/Sync is True + */ + if (!err && sync && oper) { + dcbcfg->app[app_index].priority = + (app_prio & ice_aqc_cee_app_mask) >> + ice_aqc_cee_app_shift; + dcbcfg->app[app_index].selector = ice_app_sel_type; + dcbcfg->app[app_index].prot_id = ice_app_prot_id_type; + app_index++; + } + } + + dcbcfg->numapps = app_index; +} + +/** + * ice_get_ieee_dcb_cfg + * @pi: port information structure + * @dcbx_mode: mode of DCBX (IEEE or CEE) + * + * Get IEEE or CEE mode DCB configuration from the Firmware + */ +static enum ice_status +ice_get_ieee_or_cee_dcb_cfg(struct ice_port_info *pi, u8 dcbx_mode) +{ + struct ice_dcbx_cfg *dcbx_cfg = NULL; + enum ice_status ret; + + if (!pi) + return ICE_ERR_PARAM; + + if (dcbx_mode == ICE_DCBX_MODE_IEEE) + dcbx_cfg = &pi->local_dcbx_cfg; + else if (dcbx_mode == ICE_DCBX_MODE_CEE) + dcbx_cfg = &pi->desired_dcbx_cfg; + + /* Get Local DCB Config in case of ICE_DCBX_MODE_IEEE + * or get CEE DCB Desired Config in case of ICE_DCBX_MODE_CEE + */ + ret = ice_aq_get_dcb_cfg(pi->hw, ICE_AQ_LLDP_MIB_LOCAL, + ICE_AQ_LLDP_BRID_TYPE_NEAREST_BRID, dcbx_cfg); + if (ret) + goto out; + + /* Get Remote DCB Config */ + dcbx_cfg = &pi->remote_dcbx_cfg; + ret = ice_aq_get_dcb_cfg(pi->hw, ICE_AQ_LLDP_MIB_REMOTE, + ICE_AQ_LLDP_BRID_TYPE_NEAREST_BRID, dcbx_cfg); + /* Don't treat ENOENT as an error for Remote MIBs */ + if (pi->hw->adminq.sq_last_status == ICE_AQ_RC_ENOENT) + ret = 0; + +out: + return ret; +} + +/** + * ice_get_dcb_cfg + * @pi: port information structure + * + * Get DCB configuration from the Firmware + */ +enum ice_status ice_get_dcb_cfg(struct ice_port_info *pi) +{ + struct ice_aqc_get_cee_dcb_cfg_resp cee_cfg; + struct ice_dcbx_cfg *dcbx_cfg; + enum ice_status ret; + + if (!pi) + return ICE_ERR_PARAM; + + ret = ice_aq_get_cee_dcb_cfg(pi->hw, &cee_cfg, NULL); + if (!ret) { + /* CEE mode */ + dcbx_cfg = &pi->local_dcbx_cfg; + dcbx_cfg->dcbx_mode = ICE_DCBX_MODE_CEE; + dcbx_cfg->tlv_status = le32_to_cpu(cee_cfg.tlv_status); + ice_cee_to_dcb_cfg(&cee_cfg, dcbx_cfg); + ret = ice_get_ieee_or_cee_dcb_cfg(pi, ICE_DCBX_MODE_CEE); + } else if (pi->hw->adminq.sq_last_status == ICE_AQ_RC_ENOENT) { + /* CEE mode not enabled try querying IEEE data */ + dcbx_cfg = &pi->local_dcbx_cfg; + dcbx_cfg->dcbx_mode = ICE_DCBX_MODE_IEEE; + ret = ice_get_ieee_or_cee_dcb_cfg(pi, ICE_DCBX_MODE_IEEE); + } + + return ret; +} + +/** + * ice_init_dcb + * @hw: pointer to the HW struct + * + * Update DCB configuration from the Firmware + */ +enum ice_status ice_init_dcb(struct ice_hw *hw) +{ + struct ice_port_info *pi = hw->port_info; + enum ice_status ret = 0; + + if (!hw->func_caps.common_cap.dcb) + return ICE_ERR_NOT_SUPPORTED; + + pi->is_sw_lldp = true; + + /* Get DCBX status */ + pi->dcbx_status = ice_get_dcbx_status(hw); + + if (pi->dcbx_status == ICE_DCBX_STATUS_DONE || + pi->dcbx_status == ICE_DCBX_STATUS_IN_PROGRESS) { + /* Get current DCBX configuration */ + ret = ice_get_dcb_cfg(pi); + pi->is_sw_lldp = (hw->adminq.sq_last_status == ICE_AQ_RC_EPERM); + if (ret) + return ret; + } else if (pi->dcbx_status == ICE_DCBX_STATUS_DIS) { + return ICE_ERR_NOT_READY; + } + + /* Configure the LLDP MIB change event */ + ret = ice_aq_cfg_lldp_mib_change(hw, true, NULL); + if (!ret) + pi->is_sw_lldp = false; + + return ret; +} + +/** + * ice_add_ieee_ets_common_tlv + * @buf: Data buffer to be populated with ice_dcb_ets_cfg data + * @ets_cfg: Container for ice_dcb_ets_cfg data + * + * Populate the TLV buffer with ice_dcb_ets_cfg data + */ +static void +ice_add_ieee_ets_common_tlv(u8 *buf, struct ice_dcb_ets_cfg *ets_cfg) +{ + u8 priority0, priority1; + u8 offset = 0; + int i; + + /* Priority Assignment Table (4 octets) + * Octets:| 1 | 2 | 3 | 4 | + * ----------------------------------------- + * |pri0|pri1|pri2|pri3|pri4|pri5|pri6|pri7| + * ----------------------------------------- + * Bits:|7 4|3 0|7 4|3 0|7 4|3 0|7 4|3 0| + * ----------------------------------------- + */ + for (i = 0; i < ICE_MAX_TRAFFIC_CLASS / 2; i++) { + priority0 = ets_cfg->prio_table[i * 2] & 0xF; + priority1 = ets_cfg->prio_table[i * 2 + 1] & 0xF; + buf[offset] = (priority0 << ICE_IEEE_ETS_PRIO_1_S) | priority1; + offset++; + } + + /* TC Bandwidth Table (8 octets) + * Octets:| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | + * --------------------------------- + * |tc0|tc1|tc2|tc3|tc4|tc5|tc6|tc7| + * --------------------------------- + * + * TSA Assignment Table (8 octets) + * Octets:| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | + * --------------------------------- + * |tc0|tc1|tc2|tc3|tc4|tc5|tc6|tc7| + * --------------------------------- + */ + ice_for_each_traffic_class(i) { + buf[offset] = ets_cfg->tcbwtable[i]; + buf[ICE_MAX_TRAFFIC_CLASS + offset] = ets_cfg->tsatable[i]; + offset++; + } +} + +/** + * ice_add_ieee_ets_tlv - Prepare ETS TLV in IEEE format + * @tlv: Fill the ETS config data in IEEE format + * @dcbcfg: Local store which holds the DCB Config + * + * Prepare IEEE 802.1Qaz ETS CFG TLV + */ +static void +ice_add_ieee_ets_tlv(struct ice_lldp_org_tlv *tlv, struct ice_dcbx_cfg *dcbcfg) +{ + struct ice_dcb_ets_cfg *etscfg; + u8 *buf = tlv->tlvinfo; + u8 maxtcwilling = 0; + u32 ouisubtype; + u16 typelen; + + typelen = ((ICE_TLV_TYPE_ORG << ICE_LLDP_TLV_TYPE_S) | + ICE_IEEE_ETS_TLV_LEN); + tlv->typelen = htons(typelen); + + ouisubtype = ((ICE_IEEE_8021QAZ_OUI << ICE_LLDP_TLV_OUI_S) | + ICE_IEEE_SUBTYPE_ETS_CFG); + tlv->ouisubtype = htonl(ouisubtype); + + /* First Octet post subtype + * -------------------------- + * |will-|CBS | Re- | Max | + * |ing | |served| TCs | + * -------------------------- + * |1bit | 1bit|3 bits|3bits| + */ + etscfg = &dcbcfg->etscfg; + if (etscfg->willing) + maxtcwilling = BIT(ICE_IEEE_ETS_WILLING_S); + maxtcwilling |= etscfg->maxtcs & ICE_IEEE_ETS_MAXTC_M; + buf[0] = maxtcwilling; + + /* Begin adding at Priority Assignment Table (offset 1 in buf) */ + ice_add_ieee_ets_common_tlv(&buf[1], etscfg); +} + +/** + * ice_add_ieee_etsrec_tlv - Prepare ETS Recommended TLV in IEEE format + * @tlv: Fill ETS Recommended TLV in IEEE format + * @dcbcfg: Local store which holds the DCB Config + * + * Prepare IEEE 802.1Qaz ETS REC TLV + */ +static void +ice_add_ieee_etsrec_tlv(struct ice_lldp_org_tlv *tlv, + struct ice_dcbx_cfg *dcbcfg) +{ + struct ice_dcb_ets_cfg *etsrec; + u8 *buf = tlv->tlvinfo; + u32 ouisubtype; + u16 typelen; + + typelen = ((ICE_TLV_TYPE_ORG << ICE_LLDP_TLV_TYPE_S) | + ICE_IEEE_ETS_TLV_LEN); + tlv->typelen = htons(typelen); + + ouisubtype = ((ICE_IEEE_8021QAZ_OUI << ICE_LLDP_TLV_OUI_S) | + ICE_IEEE_SUBTYPE_ETS_REC); + tlv->ouisubtype = htonl(ouisubtype); + + etsrec = &dcbcfg->etsrec; + + /* First Octet is reserved */ + /* Begin adding at Priority Assignment Table (offset 1 in buf) */ + ice_add_ieee_ets_common_tlv(&buf[1], etsrec); +} + +/** + * ice_add_ieee_pfc_tlv - Prepare PFC TLV in IEEE format + * @tlv: Fill PFC TLV in IEEE format + * @dcbcfg: Local store which holds the PFC CFG data + * + * Prepare IEEE 802.1Qaz PFC CFG TLV + */ +static void +ice_add_ieee_pfc_tlv(struct ice_lldp_org_tlv *tlv, struct ice_dcbx_cfg *dcbcfg) +{ + u8 *buf = tlv->tlvinfo; + u32 ouisubtype; + u16 typelen; + + typelen = ((ICE_TLV_TYPE_ORG << ICE_LLDP_TLV_TYPE_S) | + ICE_IEEE_PFC_TLV_LEN); + tlv->typelen = htons(typelen); + + ouisubtype = ((ICE_IEEE_8021QAZ_OUI << ICE_LLDP_TLV_OUI_S) | + ICE_IEEE_SUBTYPE_PFC_CFG); + tlv->ouisubtype = htonl(ouisubtype); + + /* ---------------------------------------- + * |will-|MBC | Re- | PFC | PFC Enable | + * |ing | |served| cap | | + * ----------------------------------------- + * |1bit | 1bit|2 bits|4bits| 1 octet | + */ + if (dcbcfg->pfc.willing) + buf[0] = BIT(ICE_IEEE_PFC_WILLING_S); + + if (dcbcfg->pfc.mbc) + buf[0] |= BIT(ICE_IEEE_PFC_MBC_S); + + buf[0] |= dcbcfg->pfc.pfccap & 0xF; + buf[1] = dcbcfg->pfc.pfcena; +} + +/** + * ice_add_ieee_app_pri_tlv - Prepare APP TLV in IEEE format + * @tlv: Fill APP TLV in IEEE format + * @dcbcfg: Local store which holds the APP CFG data + * + * Prepare IEEE 802.1Qaz APP CFG TLV + */ +static void +ice_add_ieee_app_pri_tlv(struct ice_lldp_org_tlv *tlv, + struct ice_dcbx_cfg *dcbcfg) +{ + u16 typelen, len, offset = 0; + u8 priority, selector, i = 0; + u8 *buf = tlv->tlvinfo; + u32 ouisubtype; + + /* No APP TLVs then just return */ + if (dcbcfg->numapps == 0) + return; + ouisubtype = ((ICE_IEEE_8021QAZ_OUI << ICE_LLDP_TLV_OUI_S) | + ICE_IEEE_SUBTYPE_APP_PRI); + tlv->ouisubtype = htonl(ouisubtype); + + /* Move offset to App Priority Table */ + offset++; + /* Application Priority Table (3 octets) + * Octets:| 1 | 2 | 3 | + * ----------------------------------------- + * |Priority|Rsrvd| Sel | Protocol ID | + * ----------------------------------------- + * Bits:|23 21|20 19|18 16|15 0| + * ----------------------------------------- + */ + while (i < dcbcfg->numapps) { + priority = dcbcfg->app[i].priority & 0x7; + selector = dcbcfg->app[i].selector & 0x7; + buf[offset] = (priority << ICE_IEEE_APP_PRIO_S) | selector; + buf[offset + 1] = (dcbcfg->app[i].prot_id >> 0x8) & 0xFF; + buf[offset + 2] = dcbcfg->app[i].prot_id & 0xFF; + /* Move to next app */ + offset += 3; + i++; + if (i >= ICE_DCBX_MAX_APPS) + break; + } + /* len includes size of ouisubtype + 1 reserved + 3*numapps */ + len = sizeof(tlv->ouisubtype) + 1 + (i * 3); + typelen = ((ICE_TLV_TYPE_ORG << ICE_LLDP_TLV_TYPE_S) | (len & 0x1FF)); + tlv->typelen = htons(typelen); +} + +/** + * ice_add_dcb_tlv - Add all IEEE TLVs + * @tlv: Fill TLV data in IEEE format + * @dcbcfg: Local store which holds the DCB Config + * @tlvid: Type of IEEE TLV + * + * Add tlv information + */ +static void +ice_add_dcb_tlv(struct ice_lldp_org_tlv *tlv, struct ice_dcbx_cfg *dcbcfg, + u16 tlvid) +{ + switch (tlvid) { + case ICE_IEEE_TLV_ID_ETS_CFG: + ice_add_ieee_ets_tlv(tlv, dcbcfg); + break; + case ICE_IEEE_TLV_ID_ETS_REC: + ice_add_ieee_etsrec_tlv(tlv, dcbcfg); + break; + case ICE_IEEE_TLV_ID_PFC_CFG: + ice_add_ieee_pfc_tlv(tlv, dcbcfg); + break; + case ICE_IEEE_TLV_ID_APP_PRI: + ice_add_ieee_app_pri_tlv(tlv, dcbcfg); + break; + default: + break; + } +} + +/** + * ice_dcb_cfg_to_lldp - Convert DCB configuration to MIB format + * @lldpmib: pointer to the HW struct + * @miblen: length of LLDP MIB + * @dcbcfg: Local store which holds the DCB Config + * + * Convert the DCB configuration to MIB format + */ +static void +ice_dcb_cfg_to_lldp(u8 *lldpmib, u16 *miblen, struct ice_dcbx_cfg *dcbcfg) +{ + u16 len, offset = 0, tlvid = ICE_TLV_ID_START; + struct ice_lldp_org_tlv *tlv; + u16 typelen; + + tlv = (struct ice_lldp_org_tlv *)lldpmib; + while (1) { + ice_add_dcb_tlv(tlv, dcbcfg, tlvid++); + typelen = ntohs(tlv->typelen); + len = (typelen & ICE_LLDP_TLV_LEN_M) >> ICE_LLDP_TLV_LEN_S; + if (len) + offset += len + 2; + /* END TLV or beyond LLDPDU size */ + if (tlvid >= ICE_TLV_ID_END_OF_LLDPPDU || + offset > ICE_LLDPDU_SIZE) + break; + /* Move to next TLV */ + if (len) + tlv = (struct ice_lldp_org_tlv *) + ((char *)tlv + sizeof(tlv->typelen) + len); + } + *miblen = offset; +} + +/** + * ice_set_dcb_cfg - Set the local LLDP MIB to FW + * @pi: port information structure + * + * Set DCB configuration to the Firmware + */ +enum ice_status ice_set_dcb_cfg(struct ice_port_info *pi) +{ + u8 mib_type, *lldpmib = NULL; + struct ice_dcbx_cfg *dcbcfg; + enum ice_status ret; + struct ice_hw *hw; + u16 miblen; + + if (!pi) + return ICE_ERR_PARAM; + + hw = pi->hw; + + /* update the HW local config */ + dcbcfg = &pi->local_dcbx_cfg; + /* Allocate the LLDPDU */ + lldpmib = devm_kzalloc(ice_hw_to_dev(hw), ICE_LLDPDU_SIZE, GFP_KERNEL); + if (!lldpmib) + return ICE_ERR_NO_MEMORY; + + mib_type = SET_LOCAL_MIB_TYPE_LOCAL_MIB; + if (dcbcfg->app_mode == ICE_DCBX_APPS_NON_WILLING) + mib_type |= SET_LOCAL_MIB_TYPE_CEE_NON_WILLING; + + ice_dcb_cfg_to_lldp(lldpmib, &miblen, dcbcfg); + ret = ice_aq_set_lldp_mib(hw, mib_type, (void *)lldpmib, miblen, + NULL); + + devm_kfree(ice_hw_to_dev(hw), lldpmib); + + return ret; +} + +/** + * ice_aq_query_port_ets - query port ets configuration + * @pi: port information structure + * @buf: pointer to buffer + * @buf_size: buffer size in bytes + * @cd: pointer to command details structure or NULL + * + * query current port ets configuration + */ +static enum ice_status +ice_aq_query_port_ets(struct ice_port_info *pi, + struct ice_aqc_port_ets_elem *buf, u16 buf_size, + struct ice_sq_cd *cd) +{ + struct ice_aqc_query_port_ets *cmd; + struct ice_aq_desc desc; + enum ice_status status; + + if (!pi) + return ICE_ERR_PARAM; + cmd = &desc.params.port_ets; + ice_fill_dflt_direct_cmd_desc(&desc, ice_aqc_opc_query_port_ets); + cmd->port_teid = pi->root->info.node_teid; + + status = ice_aq_send_cmd(pi->hw, &desc, buf, buf_size, cd); + return status; +} + +/** + * ice_update_port_tc_tree_cfg - update TC tree configuration + * @pi: port information structure + * @buf: pointer to buffer + * + * update the SW DB with the new TC changes + */ +static enum ice_status +ice_update_port_tc_tree_cfg(struct ice_port_info *pi, + struct ice_aqc_port_ets_elem *buf) +{ + struct ice_sched_node *node, *tc_node; + struct ice_aqc_get_elem elem; + enum ice_status status = 0; + u32 teid1, teid2; + u8 i, j; + + if (!pi) + return ICE_ERR_PARAM; + /* suspend the missing TC nodes */ + for (i = 0; i < pi->root->num_children; i++) { + teid1 = le32_to_cpu(pi->root->children[i]->info.node_teid); + ice_for_each_traffic_class(j) { + teid2 = le32_to_cpu(buf->tc_node_teid[j]); + if (teid1 == teid2) + break; + } + if (j < ICE_MAX_TRAFFIC_CLASS) + continue; + /* TC is missing */ + pi->root->children[i]->in_use = false; + } + /* add the new TC nodes */ + ice_for_each_traffic_class(j) { + teid2 = le32_to_cpu(buf->tc_node_teid[j]); + if (teid2 == ICE_INVAL_TEID) + continue; + /* Is it already present in the tree ? */ + for (i = 0; i < pi->root->num_children; i++) { + tc_node = pi->root->children[i]; + if (!tc_node) + continue; + teid1 = le32_to_cpu(tc_node->info.node_teid); + if (teid1 == teid2) { + tc_node->tc_num = j; + tc_node->in_use = true; + break; + } + } + if (i < pi->root->num_children) + continue; + /* new TC */ + status = ice_sched_query_elem(pi->hw, teid2, &elem); + if (!status) + status = ice_sched_add_node(pi, 1, &elem.generic[0]); + if (status) + break; + /* update the TC number */ + node = ice_sched_find_node_by_teid(pi->root, teid2); + if (node) + node->tc_num = j; + } + return status; +} + +/** + * ice_query_port_ets - query port ets configuration + * @pi: port information structure + * @buf: pointer to buffer + * @buf_size: buffer size in bytes + * @cd: pointer to command details structure or NULL + * + * query current port ets configuration and update the + * SW DB with the TC changes + */ +enum ice_status +ice_query_port_ets(struct ice_port_info *pi, + struct ice_aqc_port_ets_elem *buf, u16 buf_size, + struct ice_sq_cd *cd) +{ + enum ice_status status; + + mutex_lock(&pi->sched_lock); + status = ice_aq_query_port_ets(pi, buf, buf_size, cd); + if (!status) + status = ice_update_port_tc_tree_cfg(pi, buf); + mutex_unlock(&pi->sched_lock); + return status; +} diff --git a/drivers/net/ethernet/intel/ice/ice_dcb.h b/drivers/net/ethernet/intel/ice/ice_dcb.h new file mode 100644 index 0000000000000000000000000000000000000000..e7d4416e3a66cb334e915eb705adb9861e46ddac --- /dev/null +++ b/drivers/net/ethernet/intel/ice/ice_dcb.h @@ -0,0 +1,179 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2019, Intel Corporation. */ + +#ifndef _ICE_DCB_H_ +#define _ICE_DCB_H_ + +#include "ice_type.h" + +#define ICE_DCBX_STATUS_NOT_STARTED 0 +#define ICE_DCBX_STATUS_IN_PROGRESS 1 +#define ICE_DCBX_STATUS_DONE 2 +#define ICE_DCBX_STATUS_DIS 7 + +#define ICE_TLV_TYPE_END 0 +#define ICE_TLV_TYPE_ORG 127 + +#define ICE_IEEE_8021QAZ_OUI 0x0080C2 +#define ICE_IEEE_SUBTYPE_ETS_CFG 9 +#define ICE_IEEE_SUBTYPE_ETS_REC 10 +#define ICE_IEEE_SUBTYPE_PFC_CFG 11 +#define ICE_IEEE_SUBTYPE_APP_PRI 12 + +#define ICE_CEE_DCBX_OUI 0x001B21 +#define ICE_CEE_DCBX_TYPE 2 +#define ICE_CEE_SUBTYPE_PG_CFG 2 +#define ICE_CEE_SUBTYPE_PFC_CFG 3 +#define ICE_CEE_SUBTYPE_APP_PRI 4 +#define ICE_CEE_MAX_FEAT_TYPE 3 +/* Defines for LLDP TLV header */ +#define ICE_LLDP_TLV_LEN_S 0 +#define ICE_LLDP_TLV_LEN_M (0x01FF << ICE_LLDP_TLV_LEN_S) +#define ICE_LLDP_TLV_TYPE_S 9 +#define ICE_LLDP_TLV_TYPE_M (0x7F << ICE_LLDP_TLV_TYPE_S) +#define ICE_LLDP_TLV_SUBTYPE_S 0 +#define ICE_LLDP_TLV_SUBTYPE_M (0xFF << ICE_LLDP_TLV_SUBTYPE_S) +#define ICE_LLDP_TLV_OUI_S 8 +#define ICE_LLDP_TLV_OUI_M (0xFFFFFFUL << ICE_LLDP_TLV_OUI_S) + +/* Defines for IEEE ETS TLV */ +#define ICE_IEEE_ETS_MAXTC_S 0 +#define ICE_IEEE_ETS_MAXTC_M (0x7 << ICE_IEEE_ETS_MAXTC_S) +#define ICE_IEEE_ETS_CBS_S 6 +#define ICE_IEEE_ETS_CBS_M BIT(ICE_IEEE_ETS_CBS_S) +#define ICE_IEEE_ETS_WILLING_S 7 +#define ICE_IEEE_ETS_WILLING_M BIT(ICE_IEEE_ETS_WILLING_S) +#define ICE_IEEE_ETS_PRIO_0_S 0 +#define ICE_IEEE_ETS_PRIO_0_M (0x7 << ICE_IEEE_ETS_PRIO_0_S) +#define ICE_IEEE_ETS_PRIO_1_S 4 +#define ICE_IEEE_ETS_PRIO_1_M (0x7 << ICE_IEEE_ETS_PRIO_1_S) +#define ICE_CEE_PGID_PRIO_0_S 0 +#define ICE_CEE_PGID_PRIO_0_M (0xF << ICE_CEE_PGID_PRIO_0_S) +#define ICE_CEE_PGID_PRIO_1_S 4 +#define ICE_CEE_PGID_PRIO_1_M (0xF << ICE_CEE_PGID_PRIO_1_S) +#define ICE_CEE_PGID_STRICT 15 + +/* Defines for IEEE TSA types */ +#define ICE_IEEE_TSA_STRICT 0 +#define ICE_IEEE_TSA_ETS 2 + +/* Defines for IEEE PFC TLV */ +#define ICE_IEEE_PFC_CAP_S 0 +#define ICE_IEEE_PFC_CAP_M (0xF << ICE_IEEE_PFC_CAP_S) +#define ICE_IEEE_PFC_MBC_S 6 +#define ICE_IEEE_PFC_MBC_M BIT(ICE_IEEE_PFC_MBC_S) +#define ICE_IEEE_PFC_WILLING_S 7 +#define ICE_IEEE_PFC_WILLING_M BIT(ICE_IEEE_PFC_WILLING_S) + +/* Defines for IEEE APP TLV */ +#define ICE_IEEE_APP_SEL_S 0 +#define ICE_IEEE_APP_SEL_M (0x7 << ICE_IEEE_APP_SEL_S) +#define ICE_IEEE_APP_PRIO_S 5 +#define ICE_IEEE_APP_PRIO_M (0x7 << ICE_IEEE_APP_PRIO_S) + +/* TLV definitions for preparing MIB */ +#define ICE_IEEE_TLV_ID_ETS_CFG 3 +#define ICE_IEEE_TLV_ID_ETS_REC 4 +#define ICE_IEEE_TLV_ID_PFC_CFG 5 +#define ICE_IEEE_TLV_ID_APP_PRI 6 +#define ICE_TLV_ID_END_OF_LLDPPDU 7 +#define ICE_TLV_ID_START ICE_IEEE_TLV_ID_ETS_CFG + +#define ICE_IEEE_ETS_TLV_LEN 25 +#define ICE_IEEE_PFC_TLV_LEN 6 +#define ICE_IEEE_APP_TLV_LEN 11 + +/* IEEE 802.1AB LLDP Organization specific TLV */ +struct ice_lldp_org_tlv { + __be16 typelen; + __be32 ouisubtype; + u8 tlvinfo[1]; +} __packed; + +struct ice_cee_tlv_hdr { + __be16 typelen; + u8 operver; + u8 maxver; +}; + +struct ice_cee_ctrl_tlv { + struct ice_cee_tlv_hdr hdr; + __be32 seqno; + __be32 ackno; +}; + +struct ice_cee_feat_tlv { + struct ice_cee_tlv_hdr hdr; + u8 en_will_err; /* Bits: |En|Will|Err|Reserved(5)| */ +#define ICE_CEE_FEAT_TLV_ENA_M 0x80 +#define ICE_CEE_FEAT_TLV_WILLING_M 0x40 +#define ICE_CEE_FEAT_TLV_ERR_M 0x20 + u8 subtype; + u8 tlvinfo[1]; +}; + +struct ice_cee_app_prio { + __be16 protocol; + u8 upper_oui_sel; /* Bits: |Upper OUI(6)|Selector(2)| */ +#define ICE_CEE_APP_SELECTOR_M 0x03 + __be16 lower_oui; + u8 prio_map; +} __packed; + +u8 ice_get_dcbx_status(struct ice_hw *hw); +enum ice_status ice_lldp_to_dcb_cfg(u8 *lldpmib, struct ice_dcbx_cfg *dcbcfg); +enum ice_status ice_get_dcb_cfg(struct ice_port_info *pi); +enum ice_status ice_set_dcb_cfg(struct ice_port_info *pi); +enum ice_status ice_init_dcb(struct ice_hw *hw); +enum ice_status +ice_query_port_ets(struct ice_port_info *pi, + struct ice_aqc_port_ets_elem *buf, u16 buf_size, + struct ice_sq_cd *cmd_details); +#ifdef CONFIG_DCB +enum ice_status +ice_aq_stop_lldp(struct ice_hw *hw, bool shutdown_lldp_agent, + struct ice_sq_cd *cd); +enum ice_status ice_aq_start_lldp(struct ice_hw *hw, struct ice_sq_cd *cd); +enum ice_status +ice_aq_start_stop_dcbx(struct ice_hw *hw, bool start_dcbx_agent, + bool *dcbx_agent_status, struct ice_sq_cd *cd); +enum ice_status +ice_aq_cfg_lldp_mib_change(struct ice_hw *hw, bool ena_update, + struct ice_sq_cd *cd); +#else /* CONFIG_DCB */ +static inline enum ice_status +ice_aq_stop_lldp(struct ice_hw __always_unused *hw, + bool __always_unused shutdown_lldp_agent, + struct ice_sq_cd __always_unused *cd) +{ + return 0; +} + +static inline enum ice_status +ice_aq_start_lldp(struct ice_hw __always_unused *hw, + struct ice_sq_cd __always_unused *cd) +{ + return 0; +} + +static inline enum ice_status +ice_aq_start_stop_dcbx(struct ice_hw __always_unused *hw, + bool __always_unused start_dcbx_agent, + bool *dcbx_agent_status, + struct ice_sq_cd __always_unused *cd) +{ + *dcbx_agent_status = false; + + return 0; +} + +static inline enum ice_status +ice_aq_cfg_lldp_mib_change(struct ice_hw __always_unused *hw, + bool __always_unused ena_update, + struct ice_sq_cd __always_unused *cd) +{ + return 0; +} + +#endif /* CONFIG_DCB */ +#endif /* _ICE_DCB_H_ */ diff --git a/drivers/net/ethernet/intel/ice/ice_dcb_lib.c b/drivers/net/ethernet/intel/ice/ice_dcb_lib.c new file mode 100644 index 0000000000000000000000000000000000000000..3e81af1884fca750412babe37b056a9fb08d4eb1 --- /dev/null +++ b/drivers/net/ethernet/intel/ice/ice_dcb_lib.c @@ -0,0 +1,551 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2019, Intel Corporation. */ + +#include "ice_dcb_lib.h" + +/** + * ice_dcb_get_ena_tc - return bitmap of enabled TCs + * @dcbcfg: DCB config to evaluate for enabled TCs + */ +u8 ice_dcb_get_ena_tc(struct ice_dcbx_cfg *dcbcfg) +{ + u8 i, num_tc, ena_tc = 1; + + num_tc = ice_dcb_get_num_tc(dcbcfg); + + for (i = 0; i < num_tc; i++) + ena_tc |= BIT(i); + + return ena_tc; +} + +/** + * ice_dcb_get_num_tc - Get the number of TCs from DCBX config + * @dcbcfg: config to retrieve number of TCs from + */ +u8 ice_dcb_get_num_tc(struct ice_dcbx_cfg *dcbcfg) +{ + bool tc_unused = false; + u8 num_tc = 0; + u8 ret = 0; + int i; + + /* Scan the ETS Config Priority Table to find traffic classes + * enabled and create a bitmask of enabled TCs + */ + for (i = 0; i < CEE_DCBX_MAX_PRIO; i++) + num_tc |= BIT(dcbcfg->etscfg.prio_table[i]); + + /* Scan bitmask for contiguous TCs starting with TC0 */ + for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++) { + if (num_tc & BIT(i)) { + if (!tc_unused) { + ret++; + } else { + pr_err("Non-contiguous TCs - Disabling DCB\n"); + return 1; + } + } else { + tc_unused = true; + } + } + + /* There is always at least 1 TC */ + if (!ret) + ret = 1; + + return ret; +} + +/** + * ice_vsi_cfg_dcb_rings - Update rings to reflect DCB TC + * @vsi: VSI owner of rings being updated + */ +void ice_vsi_cfg_dcb_rings(struct ice_vsi *vsi) +{ + struct ice_ring *tx_ring, *rx_ring; + u16 qoffset, qcount; + int i, n; + + if (!test_bit(ICE_FLAG_DCB_ENA, vsi->back->flags)) { + /* Reset the TC information */ + for (i = 0; i < vsi->num_txq; i++) { + tx_ring = vsi->tx_rings[i]; + tx_ring->dcb_tc = 0; + } + for (i = 0; i < vsi->num_rxq; i++) { + rx_ring = vsi->rx_rings[i]; + rx_ring->dcb_tc = 0; + } + return; + } + + ice_for_each_traffic_class(n) { + if (!(vsi->tc_cfg.ena_tc & BIT(n))) + break; + + qoffset = vsi->tc_cfg.tc_info[n].qoffset; + qcount = vsi->tc_cfg.tc_info[n].qcount_tx; + for (i = qoffset; i < (qoffset + qcount); i++) { + tx_ring = vsi->tx_rings[i]; + rx_ring = vsi->rx_rings[i]; + tx_ring->dcb_tc = n; + rx_ring->dcb_tc = n; + } + } +} + +/** + * ice_pf_dcb_recfg - Reconfigure all VEBs and VSIs + * @pf: pointer to the PF struct + * + * Assumed caller has already disabled all VSIs before + * calling this function. Reconfiguring DCB based on + * local_dcbx_cfg. + */ +static void ice_pf_dcb_recfg(struct ice_pf *pf) +{ + struct ice_dcbx_cfg *dcbcfg = &pf->hw.port_info->local_dcbx_cfg; + u8 tc_map = 0; + int v, ret; + + /* Update each VSI */ + ice_for_each_vsi(pf, v) { + if (!pf->vsi[v]) + continue; + + if (pf->vsi[v]->type == ICE_VSI_PF) + tc_map = ice_dcb_get_ena_tc(dcbcfg); + else + tc_map = ICE_DFLT_TRAFFIC_CLASS; + + ret = ice_vsi_cfg_tc(pf->vsi[v], tc_map); + if (ret) + dev_err(&pf->pdev->dev, + "Failed to config TC for VSI index: %d\n", + pf->vsi[v]->idx); + else + ice_vsi_map_rings_to_vectors(pf->vsi[v]); + } +} + +/** + * ice_pf_dcb_cfg - Apply new DCB configuration + * @pf: pointer to the PF struct + * @new_cfg: DCBX config to apply + */ +static int ice_pf_dcb_cfg(struct ice_pf *pf, struct ice_dcbx_cfg *new_cfg) +{ + struct ice_dcbx_cfg *old_cfg, *curr_cfg; + struct ice_aqc_port_ets_elem buf = { 0 }; + int ret = 0; + + curr_cfg = &pf->hw.port_info->local_dcbx_cfg; + + /* Enable DCB tagging only when more than one TC */ + if (ice_dcb_get_num_tc(new_cfg) > 1) { + dev_dbg(&pf->pdev->dev, "DCB tagging enabled (num TC > 1)\n"); + set_bit(ICE_FLAG_DCB_ENA, pf->flags); + } else { + dev_dbg(&pf->pdev->dev, "DCB tagging disabled (num TC = 1)\n"); + clear_bit(ICE_FLAG_DCB_ENA, pf->flags); + } + + if (!memcmp(new_cfg, curr_cfg, sizeof(*new_cfg))) { + dev_dbg(&pf->pdev->dev, "No change in DCB config required\n"); + return ret; + } + + /* Store old config in case FW config fails */ + old_cfg = devm_kzalloc(&pf->pdev->dev, sizeof(*old_cfg), GFP_KERNEL); + memcpy(old_cfg, curr_cfg, sizeof(*old_cfg)); + + /* avoid race conditions by holding the lock while disabling and + * re-enabling the VSI + */ + rtnl_lock(); + ice_pf_dis_all_vsi(pf, true); + + memcpy(curr_cfg, new_cfg, sizeof(*curr_cfg)); + memcpy(&curr_cfg->etsrec, &curr_cfg->etscfg, sizeof(curr_cfg->etsrec)); + + /* Only send new config to HW if we are in SW LLDP mode. Otherwise, + * the new config came from the HW in the first place. + */ + if (pf->hw.port_info->is_sw_lldp) { + ret = ice_set_dcb_cfg(pf->hw.port_info); + if (ret) { + dev_err(&pf->pdev->dev, "Set DCB Config failed\n"); + /* Restore previous settings to local config */ + memcpy(curr_cfg, old_cfg, sizeof(*curr_cfg)); + goto out; + } + } + + ret = ice_query_port_ets(pf->hw.port_info, &buf, sizeof(buf), NULL); + if (ret) { + dev_err(&pf->pdev->dev, "Query Port ETS failed\n"); + goto out; + } + + ice_pf_dcb_recfg(pf); + +out: + ice_pf_ena_all_vsi(pf, true); + rtnl_unlock(); + devm_kfree(&pf->pdev->dev, old_cfg); + return ret; +} + +/** + * ice_dcb_rebuild - rebuild DCB post reset + * @pf: physical function instance + */ +void ice_dcb_rebuild(struct ice_pf *pf) +{ + struct ice_aqc_port_ets_elem buf = { 0 }; + struct ice_dcbx_cfg *prev_cfg; + enum ice_status ret; + u8 willing; + + ret = ice_query_port_ets(pf->hw.port_info, &buf, sizeof(buf), NULL); + if (ret) { + dev_err(&pf->pdev->dev, "Query Port ETS failed\n"); + goto dcb_error; + } + + /* If DCB was not enabled previously, we are done */ + if (!test_bit(ICE_FLAG_DCB_ENA, pf->flags)) + return; + + /* Save current willing state and force FW to unwilling */ + willing = pf->hw.port_info->local_dcbx_cfg.etscfg.willing; + pf->hw.port_info->local_dcbx_cfg.etscfg.willing = 0x0; + ret = ice_set_dcb_cfg(pf->hw.port_info); + if (ret) { + dev_err(&pf->pdev->dev, "Failed to set DCB to unwilling\n"); + goto dcb_error; + } + + /* Retrieve DCB config and ensure same as current in SW */ + prev_cfg = devm_kmemdup(&pf->pdev->dev, + &pf->hw.port_info->local_dcbx_cfg, + sizeof(*prev_cfg), GFP_KERNEL); + if (!prev_cfg) { + dev_err(&pf->pdev->dev, "Failed to alloc space for DCB cfg\n"); + goto dcb_error; + } + + ice_init_dcb(&pf->hw); + if (memcmp(prev_cfg, &pf->hw.port_info->local_dcbx_cfg, + sizeof(*prev_cfg))) { + /* difference in cfg detected - disable DCB till next MIB */ + dev_err(&pf->pdev->dev, "Set local MIB not accurate\n"); + devm_kfree(&pf->pdev->dev, prev_cfg); + goto dcb_error; + } + + /* fetched config congruent to previous configuration */ + devm_kfree(&pf->pdev->dev, prev_cfg); + + /* Configuration replayed - reset willing state to previous */ + pf->hw.port_info->local_dcbx_cfg.etscfg.willing = willing; + ret = ice_set_dcb_cfg(pf->hw.port_info); + if (ret) { + dev_err(&pf->pdev->dev, "Fail restoring prev willing state\n"); + goto dcb_error; + } + dev_info(&pf->pdev->dev, "DCB restored after reset\n"); + ret = ice_query_port_ets(pf->hw.port_info, &buf, sizeof(buf), NULL); + if (ret) { + dev_err(&pf->pdev->dev, "Query Port ETS failed\n"); + goto dcb_error; + } + + return; + +dcb_error: + dev_err(&pf->pdev->dev, "Disabling DCB until new settings occur\n"); + prev_cfg = devm_kzalloc(&pf->pdev->dev, sizeof(*prev_cfg), GFP_KERNEL); + prev_cfg->etscfg.willing = true; + prev_cfg->etscfg.tcbwtable[0] = ICE_TC_MAX_BW; + prev_cfg->etscfg.tsatable[0] = ICE_IEEE_TSA_ETS; + memcpy(&prev_cfg->etsrec, &prev_cfg->etscfg, sizeof(prev_cfg->etsrec)); + ice_pf_dcb_cfg(pf, prev_cfg); + devm_kfree(&pf->pdev->dev, prev_cfg); +} + +/** + * ice_dcb_init_cfg - set the initial DCB config in SW + * @pf: pf to apply config to + */ +static int ice_dcb_init_cfg(struct ice_pf *pf) +{ + struct ice_dcbx_cfg *newcfg; + struct ice_port_info *pi; + int ret = 0; + + pi = pf->hw.port_info; + newcfg = devm_kzalloc(&pf->pdev->dev, sizeof(*newcfg), GFP_KERNEL); + if (!newcfg) + return -ENOMEM; + + memcpy(newcfg, &pi->local_dcbx_cfg, sizeof(*newcfg)); + memset(&pi->local_dcbx_cfg, 0, sizeof(*newcfg)); + + dev_info(&pf->pdev->dev, "Configuring initial DCB values\n"); + if (ice_pf_dcb_cfg(pf, newcfg)) + ret = -EINVAL; + + devm_kfree(&pf->pdev->dev, newcfg); + + return ret; +} + +/** + * ice_dcb_sw_default_config - Apply a default DCB config + * @pf: pf to apply config to + */ +static int ice_dcb_sw_dflt_cfg(struct ice_pf *pf) +{ + struct ice_aqc_port_ets_elem buf = { 0 }; + struct ice_dcbx_cfg *dcbcfg; + struct ice_port_info *pi; + struct ice_hw *hw; + int ret; + + hw = &pf->hw; + pi = hw->port_info; + dcbcfg = devm_kzalloc(&pf->pdev->dev, sizeof(*dcbcfg), GFP_KERNEL); + + memset(dcbcfg, 0, sizeof(*dcbcfg)); + memset(&pi->local_dcbx_cfg, 0, sizeof(*dcbcfg)); + + dcbcfg->etscfg.willing = 1; + dcbcfg->etscfg.maxtcs = 8; + dcbcfg->etscfg.tcbwtable[0] = 100; + dcbcfg->etscfg.tsatable[0] = ICE_IEEE_TSA_ETS; + + memcpy(&dcbcfg->etsrec, &dcbcfg->etscfg, + sizeof(dcbcfg->etsrec)); + dcbcfg->etsrec.willing = 0; + + dcbcfg->pfc.willing = 1; + dcbcfg->pfc.pfccap = IEEE_8021QAZ_MAX_TCS; + + dcbcfg->numapps = 1; + dcbcfg->app[0].selector = ICE_APP_SEL_ETHTYPE; + dcbcfg->app[0].priority = 3; + dcbcfg->app[0].prot_id = ICE_APP_PROT_ID_FCOE; + + ret = ice_pf_dcb_cfg(pf, dcbcfg); + devm_kfree(&pf->pdev->dev, dcbcfg); + if (ret) + return ret; + + return ice_query_port_ets(pi, &buf, sizeof(buf), NULL); +} + +/** + * ice_init_pf_dcb - initialize DCB for a PF + * @pf: pf to initiialize DCB for + */ +int ice_init_pf_dcb(struct ice_pf *pf) +{ + struct device *dev = &pf->pdev->dev; + struct ice_port_info *port_info; + struct ice_hw *hw = &pf->hw; + int sw_default = 0; + int err; + + port_info = hw->port_info; + + /* check if device is DCB capable */ + if (!hw->func_caps.common_cap.dcb) { + dev_dbg(dev, "DCB not supported\n"); + return -EOPNOTSUPP; + } + + /* Best effort to put DCBx and LLDP into a good state */ + port_info->dcbx_status = ice_get_dcbx_status(hw); + if (port_info->dcbx_status != ICE_DCBX_STATUS_DONE && + port_info->dcbx_status != ICE_DCBX_STATUS_IN_PROGRESS) { + bool dcbx_status; + + /* Attempt to start LLDP engine. Ignore errors + * as this will error if it is already started + */ + ice_aq_start_lldp(hw, NULL); + + /* Attempt to start DCBX. Ignore errors as this + * will error if it is already started + */ + ice_aq_start_stop_dcbx(hw, true, &dcbx_status, NULL); + } + + err = ice_init_dcb(hw); + if (err) { + /* FW LLDP not in usable state, default to SW DCBx/LLDP */ + dev_info(&pf->pdev->dev, "FW LLDP not in usable state\n"); + hw->port_info->dcbx_status = ICE_DCBX_STATUS_NOT_STARTED; + hw->port_info->is_sw_lldp = true; + } + + if (port_info->dcbx_status == ICE_DCBX_STATUS_DIS) + dev_info(&pf->pdev->dev, "DCBX disabled\n"); + + /* LLDP disabled in FW */ + if (port_info->is_sw_lldp) { + sw_default = 1; + dev_info(&pf->pdev->dev, "DCBx/LLDP in SW mode.\n"); + } + + if (port_info->dcbx_status == ICE_DCBX_STATUS_NOT_STARTED) { + sw_default = 1; + dev_info(&pf->pdev->dev, "DCBX not started\n"); + } + + if (sw_default) { + err = ice_dcb_sw_dflt_cfg(pf); + if (err) { + dev_err(&pf->pdev->dev, + "Failed to set local DCB config %d\n", err); + err = -EIO; + goto dcb_init_err; + } + + pf->dcbx_cap = DCB_CAP_DCBX_HOST | DCB_CAP_DCBX_VER_IEEE; + set_bit(ICE_FLAG_DCB_CAPABLE, pf->flags); + set_bit(ICE_FLAG_DCB_ENA, pf->flags); + return 0; + } + + /* DCBX in FW and LLDP enabled in FW */ + pf->dcbx_cap = DCB_CAP_DCBX_LLD_MANAGED | DCB_CAP_DCBX_VER_IEEE; + + set_bit(ICE_FLAG_DCB_CAPABLE, pf->flags); + + err = ice_dcb_init_cfg(pf); + if (err) + goto dcb_init_err; + + dev_info(&pf->pdev->dev, "DCBX offload supported\n"); + return err; + +dcb_init_err: + dev_err(dev, "DCB init failed\n"); + return err; +} + +/** + * ice_update_dcb_stats - Update DCB stats counters + * @pf: PF whose stats needs to be updated + */ +void ice_update_dcb_stats(struct ice_pf *pf) +{ + struct ice_hw_port_stats *prev_ps, *cur_ps; + struct ice_hw *hw = &pf->hw; + u8 pf_id = hw->pf_id; + int i; + + prev_ps = &pf->stats_prev; + cur_ps = &pf->stats; + + for (i = 0; i < 8; i++) { + ice_stat_update32(hw, GLPRT_PXOFFRXC(pf_id, i), + pf->stat_prev_loaded, + &prev_ps->priority_xoff_rx[i], + &cur_ps->priority_xoff_rx[i]); + ice_stat_update32(hw, GLPRT_PXONRXC(pf_id, i), + pf->stat_prev_loaded, + &prev_ps->priority_xon_rx[i], + &cur_ps->priority_xon_rx[i]); + ice_stat_update32(hw, GLPRT_PXONTXC(pf_id, i), + pf->stat_prev_loaded, + &prev_ps->priority_xon_tx[i], + &cur_ps->priority_xon_tx[i]); + ice_stat_update32(hw, GLPRT_PXOFFTXC(pf_id, i), + pf->stat_prev_loaded, + &prev_ps->priority_xoff_tx[i], + &cur_ps->priority_xoff_tx[i]); + ice_stat_update32(hw, GLPRT_RXON2OFFCNT(pf_id, i), + pf->stat_prev_loaded, + &prev_ps->priority_xon_2_xoff[i], + &cur_ps->priority_xon_2_xoff[i]); + } +} + +/** + * ice_tx_prepare_vlan_flags_dcb - prepare VLAN tagging for DCB + * @tx_ring: ring to send buffer on + * @first: pointer to struct ice_tx_buf + */ +int +ice_tx_prepare_vlan_flags_dcb(struct ice_ring *tx_ring, + struct ice_tx_buf *first) +{ + struct sk_buff *skb = first->skb; + + if (!test_bit(ICE_FLAG_DCB_ENA, tx_ring->vsi->back->flags)) + return 0; + + /* Insert 802.1p priority into VLAN header */ + if ((first->tx_flags & (ICE_TX_FLAGS_HW_VLAN | ICE_TX_FLAGS_SW_VLAN)) || + skb->priority != TC_PRIO_CONTROL) { + first->tx_flags &= ~ICE_TX_FLAGS_VLAN_PR_M; + /* Mask the lower 3 bits to set the 802.1p priority */ + first->tx_flags |= (skb->priority & 0x7) << + ICE_TX_FLAGS_VLAN_PR_S; + if (first->tx_flags & ICE_TX_FLAGS_SW_VLAN) { + struct vlan_ethhdr *vhdr; + int rc; + + rc = skb_cow_head(skb, 0); + if (rc < 0) + return rc; + vhdr = (struct vlan_ethhdr *)skb->data; + vhdr->h_vlan_TCI = htons(first->tx_flags >> + ICE_TX_FLAGS_VLAN_S); + } else { + first->tx_flags |= ICE_TX_FLAGS_HW_VLAN; + } + } + + return 0; +} + +/** + * ice_dcb_process_lldp_set_mib_change - Process MIB change + * @pf: ptr to ice_pf + * @event: pointer to the admin queue receive event + */ +void +ice_dcb_process_lldp_set_mib_change(struct ice_pf *pf, + struct ice_rq_event_info *event) +{ + if (pf->dcbx_cap & DCB_CAP_DCBX_LLD_MANAGED) { + struct ice_dcbx_cfg *dcbcfg, *prev_cfg; + int err; + + prev_cfg = &pf->hw.port_info->local_dcbx_cfg; + dcbcfg = devm_kmemdup(&pf->pdev->dev, prev_cfg, + sizeof(*dcbcfg), GFP_KERNEL); + if (!dcbcfg) + return; + + err = ice_lldp_to_dcb_cfg(event->msg_buf, dcbcfg); + if (!err) + ice_pf_dcb_cfg(pf, dcbcfg); + + devm_kfree(&pf->pdev->dev, dcbcfg); + + /* Get updated DCBx data from firmware */ + err = ice_get_dcb_cfg(pf->hw.port_info); + if (err) + dev_err(&pf->pdev->dev, + "Failed to get DCB config\n"); + } else { + dev_dbg(&pf->pdev->dev, + "MIB Change Event in HOST mode\n"); + } +} diff --git a/drivers/net/ethernet/intel/ice/ice_dcb_lib.h b/drivers/net/ethernet/intel/ice/ice_dcb_lib.h new file mode 100644 index 0000000000000000000000000000000000000000..ca7b76faa03ca2ec9b48dd17550f84e06b73680e --- /dev/null +++ b/drivers/net/ethernet/intel/ice/ice_dcb_lib.h @@ -0,0 +1,61 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2019, Intel Corporation. */ + +#ifndef _ICE_DCB_LIB_H_ +#define _ICE_DCB_LIB_H_ + +#include "ice.h" +#include "ice_lib.h" + +#ifdef CONFIG_DCB +#define ICE_TC_MAX_BW 100 /* Default Max BW percentage */ + +void ice_dcb_rebuild(struct ice_pf *pf); +u8 ice_dcb_get_ena_tc(struct ice_dcbx_cfg *dcbcfg); +u8 ice_dcb_get_num_tc(struct ice_dcbx_cfg *dcbcfg); +void ice_vsi_cfg_dcb_rings(struct ice_vsi *vsi); +int ice_init_pf_dcb(struct ice_pf *pf); +void ice_update_dcb_stats(struct ice_pf *pf); +int +ice_tx_prepare_vlan_flags_dcb(struct ice_ring *tx_ring, + struct ice_tx_buf *first); +void +ice_dcb_process_lldp_set_mib_change(struct ice_pf *pf, + struct ice_rq_event_info *event); +static inline void +ice_set_cgd_num(struct ice_tlan_ctx *tlan_ctx, struct ice_ring *ring) +{ + tlan_ctx->cgd_num = ring->dcb_tc; +} +#else +#define ice_dcb_rebuild(pf) do {} while (0) + +static inline u8 ice_dcb_get_ena_tc(struct ice_dcbx_cfg __always_unused *dcbcfg) +{ + return ICE_DFLT_TRAFFIC_CLASS; +} + +static inline u8 ice_dcb_get_num_tc(struct ice_dcbx_cfg __always_unused *dcbcfg) +{ + return 1; +} + +static inline int ice_init_pf_dcb(struct ice_pf *pf) +{ + dev_dbg(&pf->pdev->dev, "DCB not supported\n"); + return -EOPNOTSUPP; +} + +static inline int +ice_tx_prepare_vlan_flags_dcb(struct ice_ring __always_unused *tx_ring, + struct ice_tx_buf __always_unused *first) +{ + return 0; +} + +#define ice_update_dcb_stats(pf) do {} while (0) +#define ice_vsi_cfg_dcb_rings(vsi) do {} while (0) +#define ice_dcb_process_lldp_set_mib_change(pf, event) do {} while (0) +#define ice_set_cgd_num(tlan_ctx, ring) do {} while (0) +#endif /* CONFIG_DCB */ +#endif /* _ICE_DCB_LIB_H_ */ diff --git a/drivers/net/ethernet/intel/ice/ice_ethtool.c b/drivers/net/ethernet/intel/ice/ice_ethtool.c index eb8d149e317ceab9be78fd96c6db0aed69cf23f6..1341fde8d53fa27f319bfd8fff75f439afcc959f 100644 --- a/drivers/net/ethernet/intel/ice/ice_ethtool.c +++ b/drivers/net/ethernet/intel/ice/ice_ethtool.c @@ -4,6 +4,8 @@ /* ethtool support for ice */ #include "ice.h" +#include "ice_lib.h" +#include "ice_dcb_lib.h" struct ice_stats { char stat_string[ETH_GSTRING_LEN]; @@ -33,8 +35,14 @@ static int ice_q_stats_len(struct net_device *netdev) #define ICE_PF_STATS_LEN ARRAY_SIZE(ice_gstrings_pf_stats) #define ICE_VSI_STATS_LEN ARRAY_SIZE(ice_gstrings_vsi_stats) -#define ICE_ALL_STATS_LEN(n) (ICE_PF_STATS_LEN + ICE_VSI_STATS_LEN + \ - ice_q_stats_len(n)) +#define ICE_PFC_STATS_LEN ( \ + (FIELD_SIZEOF(struct ice_pf, stats.priority_xoff_rx) + \ + FIELD_SIZEOF(struct ice_pf, stats.priority_xon_rx) + \ + FIELD_SIZEOF(struct ice_pf, stats.priority_xoff_tx) + \ + FIELD_SIZEOF(struct ice_pf, stats.priority_xon_tx)) \ + / sizeof(u64)) +#define ICE_ALL_STATS_LEN(n) (ICE_PF_STATS_LEN + ICE_PFC_STATS_LEN + \ + ICE_VSI_STATS_LEN + ice_q_stats_len(n)) static const struct ice_stats ice_gstrings_vsi_stats[] = { ICE_VSI_STAT("tx_unicast", eth_stats.tx_unicast), @@ -126,6 +134,7 @@ struct ice_priv_flag { static const struct ice_priv_flag ice_gstrings_priv_flags[] = { ICE_PRIV_FLAG("link-down-on-close", ICE_FLAG_LINK_DOWN_ON_CLOSE_ENA), + ICE_PRIV_FLAG("disable-fw-lldp", ICE_FLAG_DISABLE_FW_LLDP), }; #define ICE_PRIV_FLAG_ARRAY_SIZE ARRAY_SIZE(ice_gstrings_priv_flags) @@ -309,6 +318,22 @@ static void ice_get_strings(struct net_device *netdev, u32 stringset, u8 *data) p += ETH_GSTRING_LEN; } + for (i = 0; i < ICE_MAX_USER_PRIORITY; i++) { + snprintf(p, ETH_GSTRING_LEN, + "port.tx-priority-%u-xon", i); + p += ETH_GSTRING_LEN; + snprintf(p, ETH_GSTRING_LEN, + "port.tx-priority-%u-xoff", i); + p += ETH_GSTRING_LEN; + } + for (i = 0; i < ICE_MAX_USER_PRIORITY; i++) { + snprintf(p, ETH_GSTRING_LEN, + "port.rx-priority-%u-xon", i); + p += ETH_GSTRING_LEN; + snprintf(p, ETH_GSTRING_LEN, + "port.rx-priority-%u-xoff", i); + p += ETH_GSTRING_LEN; + } break; case ETH_SS_PRIV_FLAGS: for (i = 0; i < ICE_PRIV_FLAG_ARRAY_SIZE; i++) { @@ -382,13 +407,19 @@ static u32 ice_get_priv_flags(struct net_device *netdev) static int ice_set_priv_flags(struct net_device *netdev, u32 flags) { struct ice_netdev_priv *np = netdev_priv(netdev); + DECLARE_BITMAP(change_flags, ICE_PF_FLAGS_NBITS); + DECLARE_BITMAP(orig_flags, ICE_PF_FLAGS_NBITS); struct ice_vsi *vsi = np->vsi; struct ice_pf *pf = vsi->back; + int ret = 0; u32 i; if (flags > BIT(ICE_PRIV_FLAG_ARRAY_SIZE)) return -EINVAL; + set_bit(ICE_FLAG_ETHTOOL_CTXT, pf->flags); + + bitmap_copy(orig_flags, pf->flags, ICE_PF_FLAGS_NBITS); for (i = 0; i < ICE_PRIV_FLAG_ARRAY_SIZE; i++) { const struct ice_priv_flag *priv_flag; @@ -400,7 +431,79 @@ static int ice_set_priv_flags(struct net_device *netdev, u32 flags) clear_bit(priv_flag->bitno, pf->flags); } - return 0; + bitmap_xor(change_flags, pf->flags, orig_flags, ICE_PF_FLAGS_NBITS); + + if (test_bit(ICE_FLAG_DISABLE_FW_LLDP, change_flags)) { + if (test_bit(ICE_FLAG_DISABLE_FW_LLDP, pf->flags)) { + enum ice_status status; + + status = ice_aq_cfg_lldp_mib_change(&pf->hw, false, + NULL); + /* If unregistering for LLDP events fails, this is + * not an error state, as there shouldn't be any + * events to respond to. + */ + if (status) + dev_info(&pf->pdev->dev, + "Failed to unreg for LLDP events\n"); + + /* The AQ call to stop the FW LLDP agent will generate + * an error if the agent is already stopped. + */ + status = ice_aq_stop_lldp(&pf->hw, true, NULL); + if (status) + dev_warn(&pf->pdev->dev, + "Fail to stop LLDP agent\n"); + /* Use case for having the FW LLDP agent stopped + * will likely not need DCB, so failure to init is + * not a concern of ethtool + */ + status = ice_init_pf_dcb(pf); + if (status) + dev_warn(&pf->pdev->dev, "Fail to init DCB\n"); + } else { + enum ice_status status; + bool dcbx_agent_status; + + /* AQ command to start FW LLDP agent will return an + * error if the agent is already started + */ + status = ice_aq_start_lldp(&pf->hw, NULL); + if (status) + dev_warn(&pf->pdev->dev, + "Fail to start LLDP Agent\n"); + + /* AQ command to start FW DCBx agent will fail if + * the agent is already started + */ + status = ice_aq_start_stop_dcbx(&pf->hw, true, + &dcbx_agent_status, + NULL); + if (status) + dev_dbg(&pf->pdev->dev, + "Failed to start FW DCBX\n"); + + dev_info(&pf->pdev->dev, "FW DCBX agent is %s\n", + dcbx_agent_status ? "ACTIVE" : "DISABLED"); + + /* Failure to configure MIB change or init DCB is not + * relevant to ethtool. Print notification that + * registration/init failed but do not return error + * state to ethtool + */ + status = ice_aq_cfg_lldp_mib_change(&pf->hw, false, + NULL); + if (status) + dev_dbg(&pf->pdev->dev, + "Fail to reg for MIB change\n"); + + status = ice_init_pf_dcb(pf); + if (status) + dev_dbg(&pf->pdev->dev, "Fail to init DCB\n"); + } + } + clear_bit(ICE_FLAG_ETHTOOL_CTXT, pf->flags); + return ret; } static int ice_get_sset_count(struct net_device *netdev, int sset) @@ -486,6 +589,16 @@ ice_get_ethtool_stats(struct net_device *netdev, data[i++] = (ice_gstrings_pf_stats[j].sizeof_stat == sizeof(u64)) ? *(u64 *)p : *(u32 *)p; } + + for (j = 0; j < ICE_MAX_USER_PRIORITY; j++) { + data[i++] = pf->stats.priority_xon_tx[j]; + data[i++] = pf->stats.priority_xoff_tx[j]; + } + + for (j = 0; j < ICE_MAX_USER_PRIORITY; j++) { + data[i++] = pf->stats.priority_xon_rx[j]; + data[i++] = pf->stats.priority_xoff_rx[j]; + } } /** @@ -811,7 +924,7 @@ ice_get_settings_link_up(struct ethtool_link_ksettings *ks, link_info = &vsi->port_info->phy.link_info; - /* Initialize supported and advertised settings based on phy settings */ + /* Initialize supported and advertised settings based on PHY settings */ switch (link_info->phy_type_low) { case ICE_PHY_TYPE_LOW_100BASE_TX: ethtool_link_ksettings_add_link_mode(ks, supported, Autoneg); @@ -921,6 +1034,7 @@ ice_get_settings_link_up(struct ethtool_link_ksettings *ks, 25000baseCR_Full); break; case ICE_PHY_TYPE_LOW_25G_AUI_AOC_ACC: + case ICE_PHY_TYPE_LOW_25G_AUI_C2C: ethtool_link_ksettings_add_link_mode(ks, supported, 25000baseCR_Full); break; @@ -1137,10 +1251,10 @@ ice_get_settings_link_up(struct ethtool_link_ksettings *ks, */ static void ice_get_settings_link_down(struct ethtool_link_ksettings *ks, - struct net_device __always_unused *netdev) + struct net_device *netdev) { /* link is down and the driver needs to fall back on - * supported phy types to figure out what info to display + * supported PHY types to figure out what info to display */ ice_phy_type_to_ethtool(netdev, ks); @@ -1156,8 +1270,9 @@ ice_get_settings_link_down(struct ethtool_link_ksettings *ks, * * Reports speed/duplex settings based on media_type */ -static int ice_get_link_ksettings(struct net_device *netdev, - struct ethtool_link_ksettings *ks) +static int +ice_get_link_ksettings(struct net_device *netdev, + struct ethtool_link_ksettings *ks) { struct ice_netdev_priv *np = netdev_priv(netdev); struct ice_link_status *hw_link_info; @@ -1349,7 +1464,7 @@ ice_setup_autoneg(struct ice_port_info *p, struct ethtool_link_ksettings *ks, } else { /* If autoneg is currently enabled */ if (p->phy.link_info.an_info & ICE_AQ_AN_COMPLETED) { - /* If autoneg is supported 10GBASE_T is the only phy + /* If autoneg is supported 10GBASE_T is the only PHY * that can disable it, so otherwise return error */ if (ethtool_link_ksettings_test_link_mode(ks, @@ -1399,14 +1514,13 @@ ice_set_link_ksettings(struct net_device *netdev, if (!p) return -EOPNOTSUPP; - /* Check if this is lan vsi */ - for (idx = 0 ; idx < pf->num_alloc_vsi ; idx++) { + /* Check if this is LAN VSI */ + ice_for_each_vsi(pf, idx) if (pf->vsi[idx]->type == ICE_VSI_PF) { if (np->vsi != pf->vsi[idx]) return -EOPNOTSUPP; break; } - } if (p->phy.media_type != ICE_MEDIA_BASET && p->phy.media_type != ICE_MEDIA_FIBER && @@ -1464,7 +1578,7 @@ ice_set_link_ksettings(struct net_device *netdev, if (!abilities) return -ENOMEM; - /* Get the current phy config */ + /* Get the current PHY config */ status = ice_aq_get_phy_caps(p, false, ICE_AQC_REPORT_SW_CFG, abilities, NULL); if (status) { @@ -1559,15 +1673,16 @@ done: } /** - * ice_get_rxnfc - command to get RX flow classification rules + * ice_get_rxnfc - command to get Rx flow classification rules * @netdev: network interface device structure * @cmd: ethtool rxnfc command * @rule_locs: buffer to rturn Rx flow classification rules * * Returns Success if the command is supported. */ -static int ice_get_rxnfc(struct net_device *netdev, struct ethtool_rxnfc *cmd, - u32 __always_unused *rule_locs) +static int +ice_get_rxnfc(struct net_device *netdev, struct ethtool_rxnfc *cmd, + u32 __always_unused *rule_locs) { struct ice_netdev_priv *np = netdev_priv(netdev); struct ice_vsi *vsi = np->vsi; @@ -1821,18 +1936,21 @@ ice_get_pauseparam(struct net_device *netdev, struct ethtool_pauseparam *pause) struct ice_port_info *pi = np->vsi->port_info; struct ice_aqc_get_phy_caps_data *pcaps; struct ice_vsi *vsi = np->vsi; + struct ice_dcbx_cfg *dcbx_cfg; enum ice_status status; /* Initialize pause params */ pause->rx_pause = 0; pause->tx_pause = 0; + dcbx_cfg = &pi->local_dcbx_cfg; + pcaps = devm_kzalloc(&vsi->back->pdev->dev, sizeof(*pcaps), GFP_KERNEL); if (!pcaps) return; - /* Get current phy config */ + /* Get current PHY config */ status = ice_aq_get_phy_caps(pi, false, ICE_AQC_REPORT_SW_CFG, pcaps, NULL); if (status) @@ -1841,6 +1959,10 @@ ice_get_pauseparam(struct net_device *netdev, struct ethtool_pauseparam *pause) pause->autoneg = ((pcaps->caps & ICE_AQC_PHY_AN_MODE) ? AUTONEG_ENABLE : AUTONEG_DISABLE); + if (dcbx_cfg->pfc.pfcena) + /* PFC enabled so report LFC as off */ + goto out; + if (pcaps->caps & ICE_AQC_PHY_EN_TX_LINK_PAUSE) pause->tx_pause = 1; if (pcaps->caps & ICE_AQC_PHY_EN_RX_LINK_PAUSE) @@ -1861,6 +1983,7 @@ ice_set_pauseparam(struct net_device *netdev, struct ethtool_pauseparam *pause) struct ice_netdev_priv *np = netdev_priv(netdev); struct ice_link_status *hw_link_info; struct ice_pf *pf = np->vsi->back; + struct ice_dcbx_cfg *dcbx_cfg; struct ice_vsi *vsi = np->vsi; struct ice_hw *hw = &pf->hw; struct ice_port_info *pi; @@ -1871,6 +1994,7 @@ ice_set_pauseparam(struct net_device *netdev, struct ethtool_pauseparam *pause) pi = vsi->port_info; hw_link_info = &pi->phy.link_info; + dcbx_cfg = &pi->local_dcbx_cfg; link_up = hw_link_info->link_info & ICE_AQ_LINK_UP; /* Changing the port's flow control is not supported if this isn't the @@ -1893,6 +2017,10 @@ ice_set_pauseparam(struct net_device *netdev, struct ethtool_pauseparam *pause) netdev_info(netdev, "Autoneg did not complete so changing settings may not result in an actual change.\n"); } + if (dcbx_cfg->pfc.pfcena) { + netdev_info(netdev, "Priority flow control enabled. Cannot set link flow control.\n"); + return -EOPNOTSUPP; + } if (pause->rx_pause && pause->tx_pause) pi->fc.req_mode = ICE_FC_FULL; else if (pause->rx_pause && !pause->tx_pause) @@ -2021,11 +2149,12 @@ out: * @key: hash key * @hfunc: hash function * - * Returns -EINVAL if the table specifies an invalid queue id, otherwise + * Returns -EINVAL if the table specifies an invalid queue ID, otherwise * returns 0 after programming the table. */ -static int ice_set_rxfh(struct net_device *netdev, const u32 *indir, - const u8 *key, const u8 hfunc) +static int +ice_set_rxfh(struct net_device *netdev, const u32 *indir, const u8 *key, + const u8 hfunc) { struct ice_netdev_priv *np = netdev_priv(netdev); struct ice_vsi *vsi = np->vsi; @@ -2087,7 +2216,7 @@ enum ice_container_type { /** * ice_get_rc_coalesce - get ITR values for specific ring container * @ec: ethtool structure to fill with driver's coalesce settings - * @c_type: container type, RX or TX + * @c_type: container type, Rx or Tx * @rc: ring container that the ITR values will come from * * Query the device for ice_ring_container specific ITR values. This is @@ -2100,12 +2229,18 @@ static int ice_get_rc_coalesce(struct ethtool_coalesce *ec, enum ice_container_type c_type, struct ice_ring_container *rc) { - struct ice_pf *pf = rc->ring->vsi->back; + struct ice_pf *pf; + + if (!rc->ring) + return -EINVAL; + + pf = rc->ring->vsi->back; switch (c_type) { case ICE_RX_CONTAINER: ec->use_adaptive_rx_coalesce = ITR_IS_DYNAMIC(rc->itr_setting); ec->rx_coalesce_usecs = rc->itr_setting & ~ICE_ITR_DYNAMIC; + ec->rx_coalesce_usecs_high = rc->ring->q_vector->intrl; break; case ICE_TX_CONTAINER: ec->use_adaptive_tx_coalesce = ITR_IS_DYNAMIC(rc->itr_setting); @@ -2119,50 +2254,61 @@ ice_get_rc_coalesce(struct ethtool_coalesce *ec, enum ice_container_type c_type, return 0; } +/** + * ice_get_q_coalesce - get a queue's ITR/INTRL (coalesce) settings + * @vsi: VSI associated to the queue for getting ITR/INTRL (coalesce) settings + * @ec: coalesce settings to program the device with + * @q_num: update ITR/INTRL (coalesce) settings for this queue number/index + * + * Return 0 on success, and negative under the following conditions: + * 1. Getting Tx or Rx ITR/INTRL (coalesce) settings failed. + * 2. The q_num passed in is not a valid number/index for Tx and Rx rings. + */ +static int +ice_get_q_coalesce(struct ice_vsi *vsi, struct ethtool_coalesce *ec, int q_num) +{ + if (q_num < vsi->num_rxq && q_num < vsi->num_txq) { + if (ice_get_rc_coalesce(ec, ICE_RX_CONTAINER, + &vsi->rx_rings[q_num]->q_vector->rx)) + return -EINVAL; + if (ice_get_rc_coalesce(ec, ICE_TX_CONTAINER, + &vsi->tx_rings[q_num]->q_vector->tx)) + return -EINVAL; + } else if (q_num < vsi->num_rxq) { + if (ice_get_rc_coalesce(ec, ICE_RX_CONTAINER, + &vsi->rx_rings[q_num]->q_vector->rx)) + return -EINVAL; + } else if (q_num < vsi->num_txq) { + if (ice_get_rc_coalesce(ec, ICE_TX_CONTAINER, + &vsi->tx_rings[q_num]->q_vector->tx)) + return -EINVAL; + } else { + return -EINVAL; + } + + return 0; +} + /** * __ice_get_coalesce - get ITR/INTRL values for the device * @netdev: pointer to the netdev associated with this query * @ec: ethtool structure to fill with driver's coalesce settings * @q_num: queue number to get the coalesce settings for + * + * If the caller passes in a negative q_num then we return coalesce settings + * based on queue number 0, else use the actual q_num passed in. */ static int __ice_get_coalesce(struct net_device *netdev, struct ethtool_coalesce *ec, int q_num) { struct ice_netdev_priv *np = netdev_priv(netdev); - int tx = -EINVAL, rx = -EINVAL; struct ice_vsi *vsi = np->vsi; - if (q_num < 0) { - rx = ice_get_rc_coalesce(ec, ICE_RX_CONTAINER, - &vsi->rx_rings[0]->q_vector->rx); - tx = ice_get_rc_coalesce(ec, ICE_TX_CONTAINER, - &vsi->tx_rings[0]->q_vector->tx); - - goto update_coalesced_frames; - } - - if (q_num < vsi->num_rxq && q_num < vsi->num_txq) { - rx = ice_get_rc_coalesce(ec, ICE_RX_CONTAINER, - &vsi->rx_rings[q_num]->q_vector->rx); - tx = ice_get_rc_coalesce(ec, ICE_TX_CONTAINER, - &vsi->tx_rings[q_num]->q_vector->tx); - } else if (q_num < vsi->num_rxq) { - rx = ice_get_rc_coalesce(ec, ICE_RX_CONTAINER, - &vsi->rx_rings[q_num]->q_vector->rx); - } else if (q_num < vsi->num_txq) { - tx = ice_get_rc_coalesce(ec, ICE_TX_CONTAINER, - &vsi->tx_rings[q_num]->q_vector->tx); - } else { - /* q_num is invalid for both Rx and Tx queues */ - return -EINVAL; - } + if (q_num < 0) + q_num = 0; -update_coalesced_frames: - /* either q_num is invalid for both Rx and Tx queues or setting coalesce - * failed completely - */ - if (tx && rx) + if (ice_get_q_coalesce(vsi, ec, q_num)) return -EINVAL; if (q_num < vsi->num_txq) @@ -2180,15 +2326,16 @@ ice_get_coalesce(struct net_device *netdev, struct ethtool_coalesce *ec) return __ice_get_coalesce(netdev, ec, -1); } -static int ice_get_per_q_coalesce(struct net_device *netdev, u32 q_num, - struct ethtool_coalesce *ec) +static int +ice_get_per_q_coalesce(struct net_device *netdev, u32 q_num, + struct ethtool_coalesce *ec) { return __ice_get_coalesce(netdev, ec, q_num); } /** * ice_set_rc_coalesce - set ITR values for specific ring container - * @c_type: container type, RX or TX + * @c_type: container type, Rx or Tx * @ec: ethtool structure from user to update ITR settings * @rc: ring container that the ITR values will come from * @vsi: VSI associated to the ring container @@ -2213,6 +2360,23 @@ ice_set_rc_coalesce(enum ice_container_type c_type, struct ethtool_coalesce *ec, switch (c_type) { case ICE_RX_CONTAINER: + if (ec->rx_coalesce_usecs_high > ICE_MAX_INTRL || + (ec->rx_coalesce_usecs_high && + ec->rx_coalesce_usecs_high < pf->hw.intrl_gran)) { + netdev_info(vsi->netdev, + "Invalid value, rx-usecs-high valid values are 0 (disabled), %d-%d\n", + pf->hw.intrl_gran, ICE_MAX_INTRL); + return -EINVAL; + } + + if (ec->rx_coalesce_usecs_high != rc->ring->q_vector->intrl) { + rc->ring->q_vector->intrl = ec->rx_coalesce_usecs_high; + wr32(&pf->hw, GLINT_RATE(vsi->hw_base_vector + + rc->ring->q_vector->v_idx), + ice_intrl_usec_to_reg(ec->rx_coalesce_usecs_high, + pf->hw.intrl_gran)); + } + if (ec->rx_coalesce_usecs != itr_setting && ec->use_adaptive_rx_coalesce) { netdev_info(vsi->netdev, @@ -2235,6 +2399,12 @@ ice_set_rc_coalesce(enum ice_container_type c_type, struct ethtool_coalesce *ec, } break; case ICE_TX_CONTAINER: + if (ec->tx_coalesce_usecs_high) { + netdev_info(vsi->netdev, + "setting tx-usecs-high is not supported\n"); + return -EINVAL; + } + if (ec->tx_coalesce_usecs != itr_setting && ec->use_adaptive_tx_coalesce) { netdev_info(vsi->netdev, @@ -2264,54 +2434,77 @@ ice_set_rc_coalesce(enum ice_container_type c_type, struct ethtool_coalesce *ec, return 0; } +/** + * ice_set_q_coalesce - set a queue's ITR/INTRL (coalesce) settings + * @vsi: VSI associated to the queue that need updating + * @ec: coalesce settings to program the device with + * @q_num: update ITR/INTRL (coalesce) settings for this queue number/index + * + * Return 0 on success, and negative under the following conditions: + * 1. Setting Tx or Rx ITR/INTRL (coalesce) settings failed. + * 2. The q_num passed in is not a valid number/index for Tx and Rx rings. + */ +static int +ice_set_q_coalesce(struct ice_vsi *vsi, struct ethtool_coalesce *ec, int q_num) +{ + if (q_num < vsi->num_rxq && q_num < vsi->num_txq) { + if (ice_set_rc_coalesce(ICE_RX_CONTAINER, ec, + &vsi->rx_rings[q_num]->q_vector->rx, + vsi)) + return -EINVAL; + + if (ice_set_rc_coalesce(ICE_TX_CONTAINER, ec, + &vsi->tx_rings[q_num]->q_vector->tx, + vsi)) + return -EINVAL; + } else if (q_num < vsi->num_rxq) { + if (ice_set_rc_coalesce(ICE_RX_CONTAINER, ec, + &vsi->rx_rings[q_num]->q_vector->rx, + vsi)) + return -EINVAL; + } else if (q_num < vsi->num_txq) { + if (ice_set_rc_coalesce(ICE_TX_CONTAINER, ec, + &vsi->tx_rings[q_num]->q_vector->tx, + vsi)) + return -EINVAL; + } else { + return -EINVAL; + } + + return 0; +} + +/** + * __ice_set_coalesce - set ITR/INTRL values for the device + * @netdev: pointer to the netdev associated with this query + * @ec: ethtool structure to fill with driver's coalesce settings + * @q_num: queue number to get the coalesce settings for + * + * If the caller passes in a negative q_num then we set the coalesce settings + * for all Tx/Rx queues, else use the actual q_num passed in. + */ static int __ice_set_coalesce(struct net_device *netdev, struct ethtool_coalesce *ec, int q_num) { struct ice_netdev_priv *np = netdev_priv(netdev); - int rx = -EINVAL, tx = -EINVAL; struct ice_vsi *vsi = np->vsi; if (q_num < 0) { int i; ice_for_each_q_vector(vsi, i) { - struct ice_q_vector *q_vector = vsi->q_vectors[i]; - - if (ice_set_rc_coalesce(ICE_RX_CONTAINER, ec, - &q_vector->rx, vsi) || - ice_set_rc_coalesce(ICE_TX_CONTAINER, ec, - &q_vector->tx, vsi)) + if (ice_set_q_coalesce(vsi, ec, i)) return -EINVAL; } - goto set_work_lmt; } - if (q_num < vsi->num_rxq && q_num < vsi->num_txq) { - rx = ice_set_rc_coalesce(ICE_RX_CONTAINER, ec, - &vsi->rx_rings[q_num]->q_vector->rx, - vsi); - tx = ice_set_rc_coalesce(ICE_TX_CONTAINER, ec, - &vsi->tx_rings[q_num]->q_vector->tx, - vsi); - } else if (q_num < vsi->num_rxq) { - rx = ice_set_rc_coalesce(ICE_RX_CONTAINER, ec, - &vsi->rx_rings[q_num]->q_vector->rx, - vsi); - } else if (q_num < vsi->num_txq) { - tx = ice_set_rc_coalesce(ICE_TX_CONTAINER, ec, - &vsi->tx_rings[q_num]->q_vector->tx, - vsi); - } - - /* either q_num is invalid for both Rx and Tx queues or setting coalesce - * failed completely - */ - if (rx && tx) + if (ice_set_q_coalesce(vsi, ec, q_num)) return -EINVAL; set_work_lmt: + if (ec->tx_max_coalesced_frames_irq || ec->rx_max_coalesced_frames_irq) vsi->work_lmt = max(ec->tx_max_coalesced_frames_irq, ec->rx_max_coalesced_frames_irq); @@ -2325,8 +2518,9 @@ ice_set_coalesce(struct net_device *netdev, struct ethtool_coalesce *ec) return __ice_set_coalesce(netdev, ec, -1); } -static int ice_set_per_q_coalesce(struct net_device *netdev, u32 q_num, - struct ethtool_coalesce *ec) +static int +ice_set_per_q_coalesce(struct net_device *netdev, u32 q_num, + struct ethtool_coalesce *ec) { return __ice_set_coalesce(netdev, ec, q_num); } diff --git a/drivers/net/ethernet/intel/ice/ice_hw_autogen.h b/drivers/net/ethernet/intel/ice/ice_hw_autogen.h index 6bf5cc064270c1ebe6ac905f1d8be446ceee18ed..ec25f26069b07b15c3ad2ab582f2e956a8ce88d6 100644 --- a/drivers/net/ethernet/intel/ice/ice_hw_autogen.h +++ b/drivers/net/ethernet/intel/ice/ice_hw_autogen.h @@ -49,6 +49,9 @@ #define PF_MBX_ATQLEN_ATQLEN_M ICE_M(0x3FF, 0) #define PF_MBX_ATQLEN_ATQENABLE_M BIT(31) #define PF_MBX_ATQT 0x0022E300 +#define PRTDCB_GENS 0x00083020 +#define PRTDCB_GENS_DCBX_STATUS_S 0 +#define PRTDCB_GENS_DCBX_STATUS_M ICE_M(0x7, 0) #define GLFLXP_RXDID_FLAGS(_i, _j) (0x0045D000 + ((_i) * 4 + (_j) * 256)) #define GLFLXP_RXDID_FLAGS_FLEXIFLAG_4N_S 0 #define GLFLXP_RXDID_FLAGS_FLEXIFLAG_4N_M ICE_M(0x3F, 0) @@ -106,6 +109,16 @@ #define VPGEN_VFRTRIG_VFSWR_M BIT(0) #define PFHMC_ERRORDATA 0x00520500 #define PFHMC_ERRORINFO 0x00520400 +#define GLINT_CTL 0x0016CC54 +#define GLINT_CTL_DIS_AUTOMASK_M BIT(0) +#define GLINT_CTL_ITR_GRAN_200_S 16 +#define GLINT_CTL_ITR_GRAN_200_M ICE_M(0xF, 16) +#define GLINT_CTL_ITR_GRAN_100_S 20 +#define GLINT_CTL_ITR_GRAN_100_M ICE_M(0xF, 20) +#define GLINT_CTL_ITR_GRAN_50_S 24 +#define GLINT_CTL_ITR_GRAN_50_M ICE_M(0xF, 24) +#define GLINT_CTL_ITR_GRAN_25_S 28 +#define GLINT_CTL_ITR_GRAN_25_M ICE_M(0xF, 28) #define GLINT_DYN_CTL(_INT) (0x00160000 + ((_INT) * 4)) #define GLINT_DYN_CTL_INTENA_M BIT(0) #define GLINT_DYN_CTL_CLEARPBA_M BIT(1) @@ -150,11 +163,15 @@ #define PFINT_OICR_ENA 0x0016C900 #define QINT_RQCTL(_QRX) (0x00150000 + ((_QRX) * 4)) #define QINT_RQCTL_MSIX_INDX_S 0 +#define QINT_RQCTL_MSIX_INDX_M ICE_M(0x7FF, 0) #define QINT_RQCTL_ITR_INDX_S 11 +#define QINT_RQCTL_ITR_INDX_M ICE_M(0x3, 11) #define QINT_RQCTL_CAUSE_ENA_M BIT(30) #define QINT_TQCTL(_DBQM) (0x00140000 + ((_DBQM) * 4)) #define QINT_TQCTL_MSIX_INDX_S 0 +#define QINT_TQCTL_MSIX_INDX_M ICE_M(0x7FF, 0) #define QINT_TQCTL_ITR_INDX_S 11 +#define QINT_TQCTL_ITR_INDX_M ICE_M(0x3, 11) #define QINT_TQCTL_CAUSE_ENA_M BIT(30) #define VPINT_ALLOC(_VF) (0x001D1000 + ((_VF) * 4)) #define VPINT_ALLOC_FIRST_S 0 @@ -168,6 +185,8 @@ #define VPINT_ALLOC_PCI_LAST_S 12 #define VPINT_ALLOC_PCI_LAST_M ICE_M(0x7FF, 12) #define VPINT_ALLOC_PCI_VALID_M BIT(31) +#define VPINT_MBX_CTL(_VSI) (0x0016A000 + ((_VSI) * 4)) +#define VPINT_MBX_CTL_CAUSE_ENA_M BIT(30) #define GLLAN_RCTL_0 0x002941F8 #define QRX_CONTEXT(_i, _QRX) (0x00280000 + ((_i) * 8192 + (_QRX) * 4)) #define QRX_CTRL(_QRX) (0x00120000 + ((_QRX) * 4)) @@ -306,11 +325,16 @@ #define GLPRT_PTC64L(_i) (0x00380B80 + ((_i) * 8)) #define GLPRT_PTC9522H(_i) (0x00380D04 + ((_i) * 8)) #define GLPRT_PTC9522L(_i) (0x00380D00 + ((_i) * 8)) +#define GLPRT_PXOFFRXC(_i, _j) (0x00380500 + ((_i) * 8 + (_j) * 64)) +#define GLPRT_PXOFFTXC(_i, _j) (0x00380F40 + ((_i) * 8 + (_j) * 64)) +#define GLPRT_PXONRXC(_i, _j) (0x00380300 + ((_i) * 8 + (_j) * 64)) +#define GLPRT_PXONTXC(_i, _j) (0x00380D40 + ((_i) * 8 + (_j) * 64)) #define GLPRT_RFC(_i) (0x00380AC0 + ((_i) * 8)) #define GLPRT_RJC(_i) (0x00380B00 + ((_i) * 8)) #define GLPRT_RLEC(_i) (0x00380140 + ((_i) * 8)) #define GLPRT_ROC(_i) (0x00380240 + ((_i) * 8)) #define GLPRT_RUC(_i) (0x00380200 + ((_i) * 8)) +#define GLPRT_RXON2OFFCNT(_i, _j) (0x00380700 + ((_i) * 8 + (_j) * 64)) #define GLPRT_TDOLD(_i) (0x00381280 + ((_i) * 8)) #define GLPRT_UPRCH(_i) (0x00381304 + ((_i) * 8)) #define GLPRT_UPRCL(_i) (0x00381300 + ((_i) * 8)) diff --git a/drivers/net/ethernet/intel/ice/ice_lan_tx_rx.h b/drivers/net/ethernet/intel/ice/ice_lan_tx_rx.h index ef4c79b5aa32ad5a0dbc45864e78dee8f9eccd17..510a8c900e613806c93a25a66ef4ac8893dc7d8d 100644 --- a/drivers/net/ethernet/intel/ice/ice_lan_tx_rx.h +++ b/drivers/net/ethernet/intel/ice/ice_lan_tx_rx.h @@ -20,7 +20,7 @@ union ice_32byte_rx_desc { } lo_dword; union { __le32 rss; /* RSS Hash */ - __le32 fd_id; /* Flow Director filter id */ + __le32 fd_id; /* Flow Director filter ID */ } hi_dword; } qword0; struct { @@ -99,7 +99,7 @@ enum ice_rx_ptype_payload_layer { ICE_RX_PTYPE_PAYLOAD_LAYER_PAY4 = 3, }; -/* RX Flex Descriptor +/* Rx Flex Descriptor * This descriptor is used instead of the legacy version descriptor when * ice_rlan_ctx.adv_desc is set */ @@ -113,7 +113,7 @@ union ice_32b_rx_flex_desc { } read; struct { /* Qword 0 */ - u8 rxdid; /* descriptor builder profile id */ + u8 rxdid; /* descriptor builder profile ID */ u8 mir_id_umb_cast; /* mirror=[5:0], umb=[7:6] */ __le16 ptype_flex_flags0; /* ptype=[9:0], ff0=[15:10] */ __le16 pkt_len; /* [15:14] are reserved */ @@ -149,7 +149,7 @@ union ice_32b_rx_flex_desc { /* Rx Flex Descriptor NIC Profile * This descriptor corresponds to RxDID 2 which contains - * metadata fields for RSS, flow id and timestamp info + * metadata fields for RSS, flow ID and timestamp info */ struct ice_32b_rx_flex_desc_nic { /* Qword 0 */ @@ -208,23 +208,23 @@ enum ice_flex_rx_mdid { ICE_RX_MDID_HASH_HIGH, }; -/* Rx Flag64 packet flag bits */ -enum ice_rx_flg64_bits { - ICE_RXFLG_PKT_DSI = 0, - ICE_RXFLG_EVLAN_x8100 = 15, - ICE_RXFLG_EVLAN_x9100, - ICE_RXFLG_VLAN_x8100, - ICE_RXFLG_TNL_MAC = 22, - ICE_RXFLG_TNL_VLAN, - ICE_RXFLG_PKT_FRG, - ICE_RXFLG_FIN = 32, - ICE_RXFLG_SYN, - ICE_RXFLG_RST, - ICE_RXFLG_TNL0 = 38, - ICE_RXFLG_TNL1, - ICE_RXFLG_TNL2, - ICE_RXFLG_UDP_GRE, - ICE_RXFLG_RSVD = 63 +/* Rx/Tx Flag64 packet flag bits */ +enum ice_flg64_bits { + ICE_FLG_PKT_DSI = 0, + ICE_FLG_EVLAN_x8100 = 15, + ICE_FLG_EVLAN_x9100, + ICE_FLG_VLAN_x8100, + ICE_FLG_TNL_MAC = 22, + ICE_FLG_TNL_VLAN, + ICE_FLG_PKT_FRG, + ICE_FLG_FIN = 32, + ICE_FLG_SYN, + ICE_FLG_RST, + ICE_FLG_TNL0 = 38, + ICE_FLG_TNL1, + ICE_FLG_TNL2, + ICE_FLG_UDP_GRE, + ICE_FLG_RSVD = 63 }; /* for ice_32byte_rx_flex_desc.ptype_flexi_flags0 member */ @@ -322,7 +322,7 @@ enum ice_rlan_ctx_rx_hsplit_1 { ICE_RLAN_RX_HSPLIT_1_SPLIT_ALWAYS = 2, }; -/* TX Descriptor */ +/* Tx Descriptor */ struct ice_tx_desc { __le64 buf_addr; /* Address of descriptor's data buf */ __le64 cmd_type_offset_bsz; @@ -342,12 +342,12 @@ enum ice_tx_desc_cmd_bits { ICE_TX_DESC_CMD_EOP = 0x0001, ICE_TX_DESC_CMD_RS = 0x0002, ICE_TX_DESC_CMD_IL2TAG1 = 0x0008, - ICE_TX_DESC_CMD_IIPT_IPV6 = 0x0020, /* 2 BITS */ - ICE_TX_DESC_CMD_IIPT_IPV4 = 0x0040, /* 2 BITS */ - ICE_TX_DESC_CMD_IIPT_IPV4_CSUM = 0x0060, /* 2 BITS */ - ICE_TX_DESC_CMD_L4T_EOFT_TCP = 0x0100, /* 2 BITS */ - ICE_TX_DESC_CMD_L4T_EOFT_SCTP = 0x0200, /* 2 BITS */ - ICE_TX_DESC_CMD_L4T_EOFT_UDP = 0x0300, /* 2 BITS */ + ICE_TX_DESC_CMD_IIPT_IPV6 = 0x0020, + ICE_TX_DESC_CMD_IIPT_IPV4 = 0x0040, + ICE_TX_DESC_CMD_IIPT_IPV4_CSUM = 0x0060, + ICE_TX_DESC_CMD_L4T_EOFT_TCP = 0x0100, + ICE_TX_DESC_CMD_L4T_EOFT_SCTP = 0x0200, + ICE_TX_DESC_CMD_L4T_EOFT_UDP = 0x0300, }; #define ICE_TXD_QW1_OFFSET_S 16 diff --git a/drivers/net/ethernet/intel/ice/ice_lib.c b/drivers/net/ethernet/intel/ice/ice_lib.c index fa61203bee2694d028a0186c82f9ace77dd544ba..fbf1eba0cc2afe5c7038209e40ab01925d8d30b0 100644 --- a/drivers/net/ethernet/intel/ice/ice_lib.c +++ b/drivers/net/ethernet/intel/ice/ice_lib.c @@ -3,6 +3,7 @@ #include "ice.h" #include "ice_lib.h" +#include "ice_dcb_lib.h" /** * ice_setup_rx_ctx - Configure a receive ring context @@ -73,7 +74,7 @@ static int ice_setup_rx_ctx(struct ice_ring *ring) regval |= (rxdid << QRXFLXP_CNTXT_RXDID_IDX_S) & QRXFLXP_CNTXT_RXDID_IDX_M; - /* increasing context priority to pick up profile id; + /* increasing context priority to pick up profile ID; * default is 0x01; setting to 0x03 to ensure profile * is programming if prev context is of same priority */ @@ -124,6 +125,8 @@ ice_setup_tx_ctx(struct ice_ring *ring, struct ice_tlan_ctx *tlan_ctx, u16 pf_q) /* Transmit Queue Length */ tlan_ctx->qlen = ring->count; + ice_set_cgd_num(tlan_ctx, ring); + /* PF number */ tlan_ctx->pf_num = hw->pf_id; @@ -138,7 +141,7 @@ ice_setup_tx_ctx(struct ice_ring *ring, struct ice_tlan_ctx *tlan_ctx, u16 pf_q) tlan_ctx->vmvf_type = ICE_TLAN_CTX_VMVF_TYPE_PF; break; case ICE_VSI_VF: - /* Firmware expects vmvf_num to be absolute VF id */ + /* Firmware expects vmvf_num to be absolute VF ID */ tlan_ctx->vmvf_num = hw->func_caps.vf_base_id + vsi->vf_id; tlan_ctx->vmvf_type = ICE_TLAN_CTX_VMVF_TYPE_VF; break; @@ -175,17 +178,14 @@ static int ice_pf_rxq_wait(struct ice_pf *pf, int pf_q, bool ena) int i; for (i = 0; i < ICE_Q_WAIT_MAX_RETRY; i++) { - u32 rx_reg = rd32(&pf->hw, QRX_CTRL(pf_q)); - - if (ena == !!(rx_reg & QRX_CTRL_QENA_STAT_M)) - break; + if (ena == !!(rd32(&pf->hw, QRX_CTRL(pf_q)) & + QRX_CTRL_QENA_STAT_M)) + return 0; usleep_range(20, 40); } - if (i >= ICE_Q_WAIT_MAX_RETRY) - return -ETIMEDOUT; - return 0; + return -ETIMEDOUT; } /** @@ -197,19 +197,13 @@ static int ice_vsi_ctrl_rx_rings(struct ice_vsi *vsi, bool ena) { struct ice_pf *pf = vsi->back; struct ice_hw *hw = &pf->hw; - int i, j, ret = 0; + int i, ret = 0; for (i = 0; i < vsi->num_rxq; i++) { int pf_q = vsi->rxq_map[i]; u32 rx_reg; - for (j = 0; j < ICE_Q_WAIT_MAX_RETRY; j++) { - rx_reg = rd32(hw, QRX_CTRL(pf_q)); - if (((rx_reg >> QRX_CTRL_QENA_REQ_S) & 1) == - ((rx_reg >> QRX_CTRL_QENA_STAT_S) & 1)) - break; - usleep_range(1000, 2000); - } + rx_reg = rd32(hw, QRX_CTRL(pf_q)); /* Skip if the queue is already in the requested state */ if (ena == !!(rx_reg & QRX_CTRL_QENA_STAT_M)) @@ -238,12 +232,11 @@ static int ice_vsi_ctrl_rx_rings(struct ice_vsi *vsi, bool ena) /** * ice_vsi_alloc_arrays - Allocate queue and vector pointer arrays for the VSI * @vsi: VSI pointer - * @alloc_qvectors: a bool to specify if q_vectors need to be allocated. * * On error: returns error code (negative) * On success: returns 0 */ -static int ice_vsi_alloc_arrays(struct ice_vsi *vsi, bool alloc_qvectors) +static int ice_vsi_alloc_arrays(struct ice_vsi *vsi) { struct ice_pf *pf = vsi->back; @@ -258,15 +251,11 @@ static int ice_vsi_alloc_arrays(struct ice_vsi *vsi, bool alloc_qvectors) if (!vsi->rx_rings) goto err_rxrings; - if (alloc_qvectors) { - /* allocate memory for q_vector pointers */ - vsi->q_vectors = devm_kcalloc(&pf->pdev->dev, - vsi->num_q_vectors, - sizeof(*vsi->q_vectors), - GFP_KERNEL); - if (!vsi->q_vectors) - goto err_vectors; - } + /* allocate memory for q_vector pointers */ + vsi->q_vectors = devm_kcalloc(&pf->pdev->dev, vsi->num_q_vectors, + sizeof(*vsi->q_vectors), GFP_KERNEL); + if (!vsi->q_vectors) + goto err_vectors; return 0; @@ -279,25 +268,49 @@ err_txrings: } /** - * ice_vsi_set_num_qs - Set num queues, descriptors and vectors for a VSI + * ice_vsi_set_num_desc - Set number of descriptors for queues on this VSI * @vsi: the VSI being configured + */ +static void ice_vsi_set_num_desc(struct ice_vsi *vsi) +{ + switch (vsi->type) { + case ICE_VSI_PF: + vsi->num_rx_desc = ICE_DFLT_NUM_RX_DESC; + vsi->num_tx_desc = ICE_DFLT_NUM_TX_DESC; + break; + default: + dev_dbg(&vsi->back->pdev->dev, + "Not setting number of Tx/Rx descriptors for VSI type %d\n", + vsi->type); + break; + } +} + +/** + * ice_vsi_set_num_qs - Set number of queues, descriptors and vectors for a VSI + * @vsi: the VSI being configured + * @vf_id: ID of the VF being configured * * Return 0 on success and a negative value on error */ -static void ice_vsi_set_num_qs(struct ice_vsi *vsi) +static void ice_vsi_set_num_qs(struct ice_vsi *vsi, u16 vf_id) { struct ice_pf *pf = vsi->back; + struct ice_vf *vf = NULL; + + if (vsi->type == ICE_VSI_VF) + vsi->vf_id = vf_id; switch (vsi->type) { case ICE_VSI_PF: vsi->alloc_txq = pf->num_lan_tx; vsi->alloc_rxq = pf->num_lan_rx; - vsi->num_desc = ALIGN(ICE_DFLT_NUM_DESC, ICE_REQ_DESC_MULTIPLE); vsi->num_q_vectors = max_t(int, pf->num_lan_rx, pf->num_lan_tx); break; case ICE_VSI_VF: - vsi->alloc_txq = pf->num_vf_qps; - vsi->alloc_rxq = pf->num_vf_qps; + vf = &pf->vf[vsi->vf_id]; + vsi->alloc_txq = vf->num_vf_qs; + vsi->alloc_rxq = vf->num_vf_qs; /* pf->num_vf_msix includes (VF miscellaneous vector + * data queue interrupts). Since vsi->num_q_vectors is number * of queues vectors, subtract 1 from the original vector @@ -306,10 +319,11 @@ static void ice_vsi_set_num_qs(struct ice_vsi *vsi) vsi->num_q_vectors = pf->num_vf_msix - 1; break; default: - dev_warn(&vsi->back->pdev->dev, "Unknown VSI type %d\n", - vsi->type); + dev_warn(&pf->pdev->dev, "Unknown VSI type %d\n", vsi->type); break; } + + ice_vsi_set_num_desc(vsi); } /** @@ -370,16 +384,15 @@ void ice_vsi_delete(struct ice_vsi *vsi) } /** - * ice_vsi_free_arrays - clean up VSI resources + * ice_vsi_free_arrays - De-allocate queue and vector pointer arrays for the VSI * @vsi: pointer to VSI being cleared - * @free_qvectors: bool to specify if q_vectors should be deallocated */ -static void ice_vsi_free_arrays(struct ice_vsi *vsi, bool free_qvectors) +static void ice_vsi_free_arrays(struct ice_vsi *vsi) { struct ice_pf *pf = vsi->back; /* free the ring and vector containers */ - if (free_qvectors && vsi->q_vectors) { + if (vsi->q_vectors) { devm_kfree(&pf->pdev->dev, vsi->q_vectors); vsi->q_vectors = NULL; } @@ -427,7 +440,7 @@ int ice_vsi_clear(struct ice_vsi *vsi) if (vsi->idx < pf->next_vsi) pf->next_vsi = vsi->idx; - ice_vsi_free_arrays(vsi, true); + ice_vsi_free_arrays(vsi); mutex_unlock(&pf->sw_mutex); devm_kfree(&pf->pdev->dev, vsi); @@ -455,10 +468,12 @@ static irqreturn_t ice_msix_clean_rings(int __always_unused irq, void *data) * ice_vsi_alloc - Allocates the next available struct VSI in the PF * @pf: board private structure * @type: type of VSI + * @vf_id: ID of the VF being configured * * returns a pointer to a VSI on success, NULL on failure. */ -static struct ice_vsi *ice_vsi_alloc(struct ice_pf *pf, enum ice_vsi_type type) +static struct ice_vsi * +ice_vsi_alloc(struct ice_pf *pf, enum ice_vsi_type type, u16 vf_id) { struct ice_vsi *vsi = NULL; @@ -484,18 +499,21 @@ static struct ice_vsi *ice_vsi_alloc(struct ice_pf *pf, enum ice_vsi_type type) vsi->idx = pf->next_vsi; vsi->work_lmt = ICE_DFLT_IRQ_WORK; - ice_vsi_set_num_qs(vsi); + if (type == ICE_VSI_VF) + ice_vsi_set_num_qs(vsi, vf_id); + else + ice_vsi_set_num_qs(vsi, ICE_INVAL_VFID); switch (vsi->type) { case ICE_VSI_PF: - if (ice_vsi_alloc_arrays(vsi, true)) + if (ice_vsi_alloc_arrays(vsi)) goto err_rings; /* Setup default MSIX irq handler for VSI */ vsi->irq_handler = ice_msix_clean_rings; break; case ICE_VSI_VF: - if (ice_vsi_alloc_arrays(vsi, true)) + if (ice_vsi_alloc_arrays(vsi)) goto err_rings; break; default: @@ -547,7 +565,7 @@ static int __ice_vsi_get_qs_contig(struct ice_qs_cfg *qs_cfg) /** * __ice_vsi_get_qs_sc - Assign a scattered queues from PF to VSI - * @qs_cfg: gathered variables needed for PF->VSI queues assignment + * @qs_cfg: gathered variables needed for pf->vsi queues assignment * * Return 0 on success and -ENOMEM in case of no left space in PF queue bitmap */ @@ -579,11 +597,10 @@ err_scatter: /** * __ice_vsi_get_qs - helper function for assigning queues from PF to VSI - * @qs_cfg: gathered variables needed for PF->VSI queues assignment + * @qs_cfg: gathered variables needed for pf->vsi queues assignment * - * This is an internal function for assigning queues from the PF to VSI and - * initially tries to find contiguous space. If it is not successful to find - * contiguous space, then it tries with the scatter approach. + * This function first tries to find contiguous space. If it is not successful, + * it tries with the scatter approach. * * Return 0 on success and -ENOMEM in case of no left space in PF queue bitmap */ @@ -827,7 +844,7 @@ static void ice_vsi_setup_q_map(struct ice_vsi *vsi, struct ice_vsi_ctx *ctxt) /* find the (rounded up) power-of-2 of qcount */ pow = order_base_2(qcount_rx); - for (i = 0; i < ICE_MAX_TRAFFIC_CLASS; i++) { + ice_for_each_traffic_class(i) { if (!(vsi->tc_cfg.ena_tc & BIT(i))) { /* TC is not enabled */ vsi->tc_cfg.tc_info[i].qoffset = 0; @@ -852,7 +869,18 @@ static void ice_vsi_setup_q_map(struct ice_vsi *vsi, struct ice_vsi_ctx *ctxt) tx_count += tx_numq_tc; ctxt->info.tc_mapping[i] = cpu_to_le16(qmap); } - vsi->num_rxq = offset; + + /* if offset is non-zero, means it is calculated correctly based on + * enabled TCs for a given VSI otherwise qcount_rx will always + * be correct and non-zero because it is based off - VSI's + * allocated Rx queues which is at least 1 (hence qcount_tx will be + * at least 1) + */ + if (offset) + vsi->num_rxq = offset; + else + vsi->num_rxq = qcount_rx; + vsi->num_txq = tx_count; if (vsi->type == ICE_VSI_VF && vsi->num_txq != vsi->num_rxq) { @@ -881,6 +909,9 @@ static void ice_vsi_setup_q_map(struct ice_vsi *vsi, struct ice_vsi_ctx *ctxt) static void ice_set_rss_vsi_ctx(struct ice_vsi_ctx *ctxt, struct ice_vsi *vsi) { u8 lut_type, hash_type; + struct ice_pf *pf; + + pf = vsi->back; switch (vsi->type) { case ICE_VSI_PF: @@ -894,8 +925,7 @@ static void ice_set_rss_vsi_ctx(struct ice_vsi_ctx *ctxt, struct ice_vsi *vsi) hash_type = ICE_AQ_VSI_Q_OPT_RSS_TPLZ; break; default: - dev_warn(&vsi->back->pdev->dev, "Unknown VSI type %d\n", - vsi->type); + dev_warn(&pf->pdev->dev, "Unknown VSI type %d\n", vsi->type); return; } @@ -923,6 +953,7 @@ static int ice_vsi_init(struct ice_vsi *vsi) if (!ctxt) return -ENOMEM; + ctxt->info = vsi->info; switch (vsi->type) { case ICE_VSI_PF: ctxt->flags = ICE_AQ_VSI_TYPE_PF; @@ -948,6 +979,14 @@ static int ice_vsi_init(struct ice_vsi *vsi) ctxt->info.sw_id = vsi->port_info->sw_id; ice_vsi_setup_q_map(vsi, ctxt); + /* Enable MAC Antispoof with new VSI being initialized or updated */ + if (vsi->type == ICE_VSI_VF && pf->vf[vsi->vf_id].spoofchk) { + ctxt->info.valid_sections |= + cpu_to_le16(ICE_AQ_VSI_PROP_SECURITY_VALID); + ctxt->info.sec_flags |= + ICE_AQ_VSI_SEC_FLAG_ENA_MAC_ANTI_SPOOF; + } + ret = ice_add_vsi(hw, vsi->idx, ctxt, NULL); if (ret) { dev_err(&pf->pdev->dev, @@ -973,10 +1012,11 @@ static int ice_vsi_init(struct ice_vsi *vsi) static void ice_free_q_vector(struct ice_vsi *vsi, int v_idx) { struct ice_q_vector *q_vector; + struct ice_pf *pf = vsi->back; struct ice_ring *ring; if (!vsi->q_vectors[v_idx]) { - dev_dbg(&vsi->back->pdev->dev, "Queue vector at index %d not found\n", + dev_dbg(&pf->pdev->dev, "Queue vector at index %d not found\n", v_idx); return; } @@ -991,7 +1031,7 @@ static void ice_free_q_vector(struct ice_vsi *vsi, int v_idx) if (vsi->netdev) netif_napi_del(&q_vector->napi); - devm_kfree(&vsi->back->pdev->dev, q_vector); + devm_kfree(&pf->pdev->dev, q_vector); vsi->q_vectors[v_idx] = NULL; } @@ -1003,7 +1043,7 @@ void ice_vsi_free_q_vectors(struct ice_vsi *vsi) { int v_idx; - for (v_idx = 0; v_idx < vsi->num_q_vectors; v_idx++) + ice_for_each_q_vector(vsi, v_idx) ice_free_q_vector(vsi, v_idx); } @@ -1143,8 +1183,7 @@ static int ice_vsi_setup_vector_base(struct ice_vsi *vsi) num_q_vectors, vsi->idx); break; default: - dev_warn(&vsi->back->pdev->dev, "Unknown VSI type %d\n", - vsi->type); + dev_warn(&pf->pdev->dev, "Unknown VSI type %d\n", vsi->type); break; } @@ -1153,7 +1192,7 @@ static int ice_vsi_setup_vector_base(struct ice_vsi *vsi) "Failed to get tracking for %d HW vectors for VSI %d, err=%d\n", num_q_vectors, vsi->vsi_num, vsi->hw_base_vector); if (vsi->type != ICE_VSI_VF) { - ice_free_res(vsi->back->sw_irq_tracker, + ice_free_res(pf->sw_irq_tracker, vsi->sw_base_vector, vsi->idx); pf->num_avail_sw_msix += num_q_vectors; } @@ -1215,7 +1254,7 @@ static int ice_vsi_alloc_rings(struct ice_vsi *vsi) ring->ring_active = false; ring->vsi = vsi; ring->dev = &pf->pdev->dev; - ring->count = vsi->num_desc; + ring->count = vsi->num_tx_desc; vsi->tx_rings[i] = ring; } @@ -1234,7 +1273,7 @@ static int ice_vsi_alloc_rings(struct ice_vsi *vsi) ring->vsi = vsi; ring->netdev = vsi->netdev; ring->dev = &pf->pdev->dev; - ring->count = vsi->num_desc; + ring->count = vsi->num_rx_desc; vsi->rx_rings[i] = ring; } @@ -1253,7 +1292,11 @@ err_out: * through the MSI-X enabling code. On a constrained vector budget, we map Tx * and Rx rings to the vector as "efficiently" as possible. */ +#ifdef CONFIG_DCB +void ice_vsi_map_rings_to_vectors(struct ice_vsi *vsi) +#else static void ice_vsi_map_rings_to_vectors(struct ice_vsi *vsi) +#endif /* CONFIG_DCB */ { int q_vectors = vsi->num_q_vectors; int tx_rings_rem, rx_rings_rem; @@ -1339,7 +1382,6 @@ int ice_vsi_manage_rss_lut(struct ice_vsi *vsi, bool ena) */ static int ice_vsi_cfg_rss_lut_key(struct ice_vsi *vsi) { - u8 seed[ICE_AQC_GET_SET_RSS_KEY_DATA_RSS_KEY_SIZE]; struct ice_aqc_get_set_rss_keys *key; struct ice_pf *pf = vsi->back; enum ice_status status; @@ -1361,31 +1403,30 @@ static int ice_vsi_cfg_rss_lut_key(struct ice_vsi *vsi) vsi->rss_table_size); if (status) { - dev_err(&vsi->back->pdev->dev, + dev_err(&pf->pdev->dev, "set_rss_lut failed, error %d\n", status); err = -EIO; goto ice_vsi_cfg_rss_exit; } - key = devm_kzalloc(&vsi->back->pdev->dev, sizeof(*key), GFP_KERNEL); + key = devm_kzalloc(&pf->pdev->dev, sizeof(*key), GFP_KERNEL); if (!key) { err = -ENOMEM; goto ice_vsi_cfg_rss_exit; } if (vsi->rss_hkey_user) - memcpy(seed, vsi->rss_hkey_user, - ICE_AQC_GET_SET_RSS_KEY_DATA_RSS_KEY_SIZE); + memcpy(key, + (struct ice_aqc_get_set_rss_keys *)vsi->rss_hkey_user, + ICE_GET_SET_RSS_KEY_EXTEND_KEY_SIZE); else - netdev_rss_key_fill((void *)seed, - ICE_AQC_GET_SET_RSS_KEY_DATA_RSS_KEY_SIZE); - memcpy(&key->standard_rss_key, seed, - ICE_AQC_GET_SET_RSS_KEY_DATA_RSS_KEY_SIZE); + netdev_rss_key_fill((void *)key, + ICE_GET_SET_RSS_KEY_EXTEND_KEY_SIZE); status = ice_aq_set_rss_key(&pf->hw, vsi->idx, key); if (status) { - dev_err(&vsi->back->pdev->dev, "set_rss_key failed, error %d\n", + dev_err(&pf->pdev->dev, "set_rss_key failed, error %d\n", status); err = -EIO; } @@ -1397,12 +1438,12 @@ ice_vsi_cfg_rss_exit: } /** - * ice_add_mac_to_list - Add a mac address filter entry to the list + * ice_add_mac_to_list - Add a MAC address filter entry to the list * @vsi: the VSI to be forwarded to * @add_list: pointer to the list which contains MAC filter entries * @macaddr: the MAC address to be added. * - * Adds mac address filter entry to the temp list + * Adds MAC address filter entry to the temp list * * Returns 0 on success or ENOMEM on failure. */ @@ -1504,7 +1545,7 @@ void ice_free_fltr_list(struct device *dev, struct list_head *h) /** * ice_vsi_add_vlan - Add VSI membership for given VLAN * @vsi: the VSI being configured - * @vid: VLAN id to be added + * @vid: VLAN ID to be added */ int ice_vsi_add_vlan(struct ice_vsi *vsi, u16 vid) { @@ -1542,7 +1583,7 @@ int ice_vsi_add_vlan(struct ice_vsi *vsi, u16 vid) /** * ice_vsi_kill_vlan - Remove VSI membership for a given VLAN * @vsi: the VSI being configured - * @vid: VLAN id to be removed + * @vid: VLAN ID to be removed * * Returns 0 on success and negative on failure */ @@ -1551,7 +1592,8 @@ int ice_vsi_kill_vlan(struct ice_vsi *vsi, u16 vid) struct ice_fltr_list_entry *list; struct ice_pf *pf = vsi->back; LIST_HEAD(tmp_add_list); - int status = 0; + enum ice_status status; + int err = 0; list = devm_kzalloc(&pf->pdev->dev, sizeof(*list), GFP_KERNEL); if (!list) @@ -1567,14 +1609,20 @@ int ice_vsi_kill_vlan(struct ice_vsi *vsi, u16 vid) INIT_LIST_HEAD(&list->list_entry); list_add(&list->list_entry, &tmp_add_list); - if (ice_remove_vlan(&pf->hw, &tmp_add_list)) { - dev_err(&pf->pdev->dev, "Error removing VLAN %d on vsi %i\n", - vid, vsi->vsi_num); - status = -EIO; + status = ice_remove_vlan(&pf->hw, &tmp_add_list); + if (status == ICE_ERR_DOES_NOT_EXIST) { + dev_dbg(&pf->pdev->dev, + "Failed to remove VLAN %d on VSI %i, it does not exist, status: %d\n", + vid, vsi->vsi_num, status); + } else if (status) { + dev_err(&pf->pdev->dev, + "Error removing VLAN %d on vsi %i error: %d\n", + vid, vsi->vsi_num, status); + err = -EIO; } ice_free_fltr_list(&pf->pdev->dev, &tmp_add_list); - return status; + return err; } /** @@ -1586,7 +1634,6 @@ int ice_vsi_kill_vlan(struct ice_vsi *vsi, u16 vid) */ int ice_vsi_cfg_rxqs(struct ice_vsi *vsi) { - int err = 0; u16 i; if (vsi->type == ICE_VSI_VF) @@ -1601,14 +1648,19 @@ int ice_vsi_cfg_rxqs(struct ice_vsi *vsi) vsi->rx_buf_len = ICE_RXBUF_2048; setup_rings: /* set up individual rings */ - for (i = 0; i < vsi->num_rxq && !err; i++) - err = ice_setup_rx_ctx(vsi->rx_rings[i]); + for (i = 0; i < vsi->num_rxq; i++) { + int err; - if (err) { - dev_err(&vsi->back->pdev->dev, "ice_setup_rx_ctx failed\n"); - return -EIO; + err = ice_setup_rx_ctx(vsi->rx_rings[i]); + if (err) { + dev_err(&vsi->back->pdev->dev, + "ice_setup_rx_ctx failed for RxQ %d, err %d\n", + i, err); + return err; + } } - return err; + + return 0; } /** @@ -1640,7 +1692,7 @@ ice_vsi_cfg_txqs(struct ice_vsi *vsi, struct ice_ring **rings, int offset) num_q_grps = 1; /* set up and configure the Tx queues for each enabled TC */ - for (tc = 0; tc < ICE_MAX_TRAFFIC_CLASS; tc++) { + ice_for_each_traffic_class(tc) { if (!(vsi->tc_cfg.ena_tc & BIT(tc))) break; @@ -1660,10 +1712,10 @@ ice_vsi_cfg_txqs(struct ice_vsi *vsi, struct ice_ring **rings, int offset) rings[q_idx]->tail = pf->hw.hw_addr + QTX_COMM_DBELL(pf_q); status = ice_ena_vsi_txq(vsi->port_info, vsi->idx, tc, - num_q_grps, qg_buf, buf_len, - NULL); + i, num_q_grps, qg_buf, + buf_len, NULL); if (status) { - dev_err(&vsi->back->pdev->dev, + dev_err(&pf->pdev->dev, "Failed to set LAN Tx queue context, error: %d\n", status); err = -ENODEV; @@ -1707,7 +1759,7 @@ int ice_vsi_cfg_lan_txqs(struct ice_vsi *vsi) * This function converts a decimal interrupt rate limit in usecs to the format * expected by firmware. */ -static u32 ice_intrl_usec_to_reg(u8 intrl, u8 gran) +u32 ice_intrl_usec_to_reg(u8 intrl, u8 gran) { u32 val = intrl / gran; @@ -1716,18 +1768,50 @@ static u32 ice_intrl_usec_to_reg(u8 intrl, u8 gran) return 0; } +/** + * ice_cfg_itr_gran - set the ITR granularity to 2 usecs if not already set + * @hw: board specific structure + */ +static void ice_cfg_itr_gran(struct ice_hw *hw) +{ + u32 regval = rd32(hw, GLINT_CTL); + + /* no need to update global register if ITR gran is already set */ + if (!(regval & GLINT_CTL_DIS_AUTOMASK_M) && + (((regval & GLINT_CTL_ITR_GRAN_200_M) >> + GLINT_CTL_ITR_GRAN_200_S) == ICE_ITR_GRAN_US) && + (((regval & GLINT_CTL_ITR_GRAN_100_M) >> + GLINT_CTL_ITR_GRAN_100_S) == ICE_ITR_GRAN_US) && + (((regval & GLINT_CTL_ITR_GRAN_50_M) >> + GLINT_CTL_ITR_GRAN_50_S) == ICE_ITR_GRAN_US) && + (((regval & GLINT_CTL_ITR_GRAN_25_M) >> + GLINT_CTL_ITR_GRAN_25_S) == ICE_ITR_GRAN_US)) + return; + + regval = ((ICE_ITR_GRAN_US << GLINT_CTL_ITR_GRAN_200_S) & + GLINT_CTL_ITR_GRAN_200_M) | + ((ICE_ITR_GRAN_US << GLINT_CTL_ITR_GRAN_100_S) & + GLINT_CTL_ITR_GRAN_100_M) | + ((ICE_ITR_GRAN_US << GLINT_CTL_ITR_GRAN_50_S) & + GLINT_CTL_ITR_GRAN_50_M) | + ((ICE_ITR_GRAN_US << GLINT_CTL_ITR_GRAN_25_S) & + GLINT_CTL_ITR_GRAN_25_M); + wr32(hw, GLINT_CTL, regval); +} + /** * ice_cfg_itr - configure the initial interrupt throttle values * @hw: pointer to the HW structure * @q_vector: interrupt vector that's being configured - * @vector: HW vector index to apply the interrupt throttling to * * Configure interrupt throttling values for the ring containers that are * associated with the interrupt vector passed in. */ static void -ice_cfg_itr(struct ice_hw *hw, struct ice_q_vector *q_vector, u16 vector) +ice_cfg_itr(struct ice_hw *hw, struct ice_q_vector *q_vector) { + ice_cfg_itr_gran(hw); + if (q_vector->num_ring_rx) { struct ice_ring_container *rc = &q_vector->rx; @@ -1738,8 +1822,7 @@ ice_cfg_itr(struct ice_hw *hw, struct ice_q_vector *q_vector, u16 vector) rc->target_itr = ITR_TO_REG(rc->itr_setting); rc->next_update = jiffies + 1; rc->current_itr = rc->target_itr; - rc->latency_range = ICE_LOW_LATENCY; - wr32(hw, GLINT_ITR(rc->itr_idx, vector), + wr32(hw, GLINT_ITR(rc->itr_idx, q_vector->reg_idx), ITR_REG_ALIGN(rc->current_itr) >> ICE_ITR_GRAN_S); } @@ -1753,8 +1836,7 @@ ice_cfg_itr(struct ice_hw *hw, struct ice_q_vector *q_vector, u16 vector) rc->target_itr = ITR_TO_REG(rc->itr_setting); rc->next_update = jiffies + 1; rc->current_itr = rc->target_itr; - rc->latency_range = ICE_LOW_LATENCY; - wr32(hw, GLINT_ITR(rc->itr_idx, vector), + wr32(hw, GLINT_ITR(rc->itr_idx, q_vector->reg_idx), ITR_REG_ALIGN(rc->current_itr) >> ICE_ITR_GRAN_S); } } @@ -1766,17 +1848,17 @@ ice_cfg_itr(struct ice_hw *hw, struct ice_q_vector *q_vector, u16 vector) void ice_vsi_cfg_msix(struct ice_vsi *vsi) { struct ice_pf *pf = vsi->back; - u16 vector = vsi->hw_base_vector; struct ice_hw *hw = &pf->hw; u32 txq = 0, rxq = 0; int i, q; - for (i = 0; i < vsi->num_q_vectors; i++, vector++) { + for (i = 0; i < vsi->num_q_vectors; i++) { struct ice_q_vector *q_vector = vsi->q_vectors[i]; + u16 reg_idx = q_vector->reg_idx; - ice_cfg_itr(hw, q_vector, vector); + ice_cfg_itr(hw, q_vector); - wr32(hw, GLINT_RATE(vector), + wr32(hw, GLINT_RATE(reg_idx), ice_intrl_usec_to_reg(q_vector->intrl, hw->intrl_gran)); /* Both Transmit Queue Interrupt Cause Control register @@ -1791,33 +1873,37 @@ void ice_vsi_cfg_msix(struct ice_vsi *vsi) * tracked for this PF. */ for (q = 0; q < q_vector->num_ring_tx; q++) { - int itr_idx = q_vector->tx.itr_idx; + int itr_idx = (q_vector->tx.itr_idx << + QINT_TQCTL_ITR_INDX_S) & + QINT_TQCTL_ITR_INDX_M; u32 val; if (vsi->type == ICE_VSI_VF) - val = QINT_TQCTL_CAUSE_ENA_M | - (itr_idx << QINT_TQCTL_ITR_INDX_S) | - ((i + 1) << QINT_TQCTL_MSIX_INDX_S); + val = QINT_TQCTL_CAUSE_ENA_M | itr_idx | + (((i + 1) << QINT_TQCTL_MSIX_INDX_S) & + QINT_TQCTL_MSIX_INDX_M); else - val = QINT_TQCTL_CAUSE_ENA_M | - (itr_idx << QINT_TQCTL_ITR_INDX_S) | - (vector << QINT_TQCTL_MSIX_INDX_S); + val = QINT_TQCTL_CAUSE_ENA_M | itr_idx | + ((reg_idx << QINT_TQCTL_MSIX_INDX_S) & + QINT_TQCTL_MSIX_INDX_M); wr32(hw, QINT_TQCTL(vsi->txq_map[txq]), val); txq++; } for (q = 0; q < q_vector->num_ring_rx; q++) { - int itr_idx = q_vector->rx.itr_idx; + int itr_idx = (q_vector->rx.itr_idx << + QINT_RQCTL_ITR_INDX_S) & + QINT_RQCTL_ITR_INDX_M; u32 val; if (vsi->type == ICE_VSI_VF) - val = QINT_RQCTL_CAUSE_ENA_M | - (itr_idx << QINT_RQCTL_ITR_INDX_S) | - ((i + 1) << QINT_RQCTL_MSIX_INDX_S); + val = QINT_RQCTL_CAUSE_ENA_M | itr_idx | + (((i + 1) << QINT_RQCTL_MSIX_INDX_S) & + QINT_RQCTL_MSIX_INDX_M); else - val = QINT_RQCTL_CAUSE_ENA_M | - (itr_idx << QINT_RQCTL_ITR_INDX_S) | - (vector << QINT_RQCTL_MSIX_INDX_S); + val = QINT_RQCTL_CAUSE_ENA_M | itr_idx | + ((reg_idx << QINT_RQCTL_MSIX_INDX_S) & + QINT_RQCTL_MSIX_INDX_M); wr32(hw, QINT_RQCTL(vsi->rxq_map[rxq]), val); rxq++; } @@ -1848,6 +1934,10 @@ int ice_vsi_manage_vlan_insertion(struct ice_vsi *vsi) */ ctxt->info.vlan_flags = ICE_AQ_VSI_VLAN_MODE_ALL; + /* Preserve existing VLAN strip setting */ + ctxt->info.vlan_flags |= (vsi->info.vlan_flags & + ICE_AQ_VSI_VLAN_EMOD_M); + ctxt->info.valid_sections = cpu_to_le16(ICE_AQ_VSI_PROP_VLAN_VALID); status = ice_update_vsi(hw, vsi->idx, ctxt, NULL); @@ -1937,7 +2027,7 @@ int ice_vsi_stop_rx_rings(struct ice_vsi *vsi) * ice_vsi_stop_tx_rings - Disable Tx rings * @vsi: the VSI being configured * @rst_src: reset source - * @rel_vmvf_num: Relative id of VF/VM + * @rel_vmvf_num: Relative ID of VF/VM * @rings: Tx ring array to be stopped * @offset: offset within vsi->txq_map */ @@ -1947,10 +2037,10 @@ ice_vsi_stop_tx_rings(struct ice_vsi *vsi, enum ice_disq_rst_src rst_src, { struct ice_pf *pf = vsi->back; struct ice_hw *hw = &pf->hw; + int tc, q_idx = 0, err = 0; + u16 *q_ids, *q_handles, i; enum ice_status status; u32 *q_teids, val; - u16 *q_ids, i; - int err = 0; if (vsi->num_txq > ICE_LAN_TXQ_MAX_QDIS) return -EINVAL; @@ -1967,50 +2057,69 @@ ice_vsi_stop_tx_rings(struct ice_vsi *vsi, enum ice_disq_rst_src rst_src, goto err_alloc_q_ids; } - /* set up the Tx queue list to be disabled */ - ice_for_each_txq(vsi, i) { - u16 v_idx; + q_handles = devm_kcalloc(&pf->pdev->dev, vsi->num_txq, + sizeof(*q_handles), GFP_KERNEL); + if (!q_handles) { + err = -ENOMEM; + goto err_alloc_q_handles; + } - if (!rings || !rings[i] || !rings[i]->q_vector) { - err = -EINVAL; - goto err_out; - } + /* set up the Tx queue list to be disabled for each enabled TC */ + ice_for_each_traffic_class(tc) { + if (!(vsi->tc_cfg.ena_tc & BIT(tc))) + break; - q_ids[i] = vsi->txq_map[i + offset]; - q_teids[i] = rings[i]->txq_teid; + for (i = 0; i < vsi->tc_cfg.tc_info[tc].qcount_tx; i++) { + if (!rings || !rings[q_idx] || + !rings[q_idx]->q_vector) { + err = -EINVAL; + goto err_out; + } - /* clear cause_ena bit for disabled queues */ - val = rd32(hw, QINT_TQCTL(rings[i]->reg_idx)); - val &= ~QINT_TQCTL_CAUSE_ENA_M; - wr32(hw, QINT_TQCTL(rings[i]->reg_idx), val); + q_ids[i] = vsi->txq_map[q_idx + offset]; + q_teids[i] = rings[q_idx]->txq_teid; + q_handles[i] = i; - /* software is expected to wait for 100 ns */ - ndelay(100); + /* clear cause_ena bit for disabled queues */ + val = rd32(hw, QINT_TQCTL(rings[i]->reg_idx)); + val &= ~QINT_TQCTL_CAUSE_ENA_M; + wr32(hw, QINT_TQCTL(rings[i]->reg_idx), val); - /* trigger a software interrupt for the vector associated to - * the queue to schedule NAPI handler + /* software is expected to wait for 100 ns */ + ndelay(100); + + /* trigger a software interrupt for the vector + * associated to the queue to schedule NAPI handler + */ + wr32(hw, GLINT_DYN_CTL(rings[i]->q_vector->reg_idx), + GLINT_DYN_CTL_SWINT_TRIG_M | + GLINT_DYN_CTL_INTENA_MSK_M); + q_idx++; + } + status = ice_dis_vsi_txq(vsi->port_info, vsi->idx, tc, + vsi->num_txq, q_handles, q_ids, + q_teids, rst_src, rel_vmvf_num, NULL); + + /* if the disable queue command was exercised during an active + * reset flow, ICE_ERR_RESET_ONGOING is returned. This is not + * an error as the reset operation disables queues at the + * hardware level anyway. */ - v_idx = rings[i]->q_vector->v_idx; - wr32(hw, GLINT_DYN_CTL(vsi->hw_base_vector + v_idx), - GLINT_DYN_CTL_SWINT_TRIG_M | GLINT_DYN_CTL_INTENA_MSK_M); - } - status = ice_dis_vsi_txq(vsi->port_info, vsi->num_txq, q_ids, q_teids, - rst_src, rel_vmvf_num, NULL); - /* if the disable queue command was exercised during an active reset - * flow, ICE_ERR_RESET_ONGOING is returned. This is not an error as - * the reset operation disables queues at the hardware level anyway. - */ - if (status == ICE_ERR_RESET_ONGOING) { - dev_info(&pf->pdev->dev, - "Reset in progress. LAN Tx queues already disabled\n"); - } else if (status) { - dev_err(&pf->pdev->dev, - "Failed to disable LAN Tx queues, error: %d\n", - status); - err = -ENODEV; + if (status == ICE_ERR_RESET_ONGOING) { + dev_dbg(&pf->pdev->dev, + "Reset in progress. LAN Tx queues already disabled\n"); + } else if (status) { + dev_err(&pf->pdev->dev, + "Failed to disable LAN Tx queues, error: %d\n", + status); + err = -ENODEV; + } } err_out: + devm_kfree(&pf->pdev->dev, q_handles); + +err_alloc_q_handles: devm_kfree(&pf->pdev->dev, q_ids); err_alloc_q_ids: @@ -2023,10 +2132,11 @@ err_alloc_q_ids: * ice_vsi_stop_lan_tx_rings - Disable LAN Tx rings * @vsi: the VSI being configured * @rst_src: reset source - * @rel_vmvf_num: Relative id of VF/VM + * @rel_vmvf_num: Relative ID of VF/VM */ -int ice_vsi_stop_lan_tx_rings(struct ice_vsi *vsi, - enum ice_disq_rst_src rst_src, u16 rel_vmvf_num) +int +ice_vsi_stop_lan_tx_rings(struct ice_vsi *vsi, enum ice_disq_rst_src rst_src, + u16 rel_vmvf_num) { return ice_vsi_stop_tx_rings(vsi, rst_src, rel_vmvf_num, vsi->tx_rings, 0); @@ -2036,19 +2146,22 @@ int ice_vsi_stop_lan_tx_rings(struct ice_vsi *vsi, * ice_cfg_vlan_pruning - enable or disable VLAN pruning on the VSI * @vsi: VSI to enable or disable VLAN pruning on * @ena: set to true to enable VLAN pruning and false to disable it + * @vlan_promisc: enable valid security flags if not in VLAN promiscuous mode * * returns 0 if VSI is updated, negative otherwise */ -int ice_cfg_vlan_pruning(struct ice_vsi *vsi, bool ena) +int ice_cfg_vlan_pruning(struct ice_vsi *vsi, bool ena, bool vlan_promisc) { struct ice_vsi_ctx *ctxt; struct device *dev; + struct ice_pf *pf; int status; if (!vsi) return -EINVAL; - dev = &vsi->back->pdev->dev; + pf = vsi->back; + dev = &pf->pdev->dev; ctxt = devm_kzalloc(dev, sizeof(*ctxt), GFP_KERNEL); if (!ctxt) return -ENOMEM; @@ -2067,14 +2180,16 @@ int ice_cfg_vlan_pruning(struct ice_vsi *vsi, bool ena) ctxt->info.sw_flags2 &= ~ICE_AQ_VSI_SW_FLAG_RX_VLAN_PRUNE_ENA; } - ctxt->info.valid_sections = cpu_to_le16(ICE_AQ_VSI_PROP_SECURITY_VALID | - ICE_AQ_VSI_PROP_SW_VALID); + if (!vlan_promisc) + ctxt->info.valid_sections = + cpu_to_le16(ICE_AQ_VSI_PROP_SECURITY_VALID | + ICE_AQ_VSI_PROP_SW_VALID); - status = ice_update_vsi(&vsi->back->hw, vsi->idx, ctxt, NULL); + status = ice_update_vsi(&pf->hw, vsi->idx, ctxt, NULL); if (status) { netdev_err(vsi->netdev, "%sabling VLAN pruning on VSI handle: %d, VSI HW ID: %d failed, err = %d, aq_err = %d\n", ena ? "En" : "Dis", vsi->idx, vsi->vsi_num, status, - vsi->back->hw.adminq.sq_last_status); + pf->hw.adminq.sq_last_status); goto err_out; } @@ -2089,12 +2204,98 @@ err_out: return -EIO; } +static void ice_vsi_set_tc_cfg(struct ice_vsi *vsi) +{ + struct ice_dcbx_cfg *cfg = &vsi->port_info->local_dcbx_cfg; + + vsi->tc_cfg.ena_tc = ice_dcb_get_ena_tc(cfg); + vsi->tc_cfg.numtc = ice_dcb_get_num_tc(cfg); +} + +/** + * ice_vsi_set_q_vectors_reg_idx - set the HW register index for all q_vectors + * @vsi: VSI to set the q_vectors register index on + */ +static int +ice_vsi_set_q_vectors_reg_idx(struct ice_vsi *vsi) +{ + u16 i; + + if (!vsi || !vsi->q_vectors) + return -EINVAL; + + ice_for_each_q_vector(vsi, i) { + struct ice_q_vector *q_vector = vsi->q_vectors[i]; + + if (!q_vector) { + dev_err(&vsi->back->pdev->dev, + "Failed to set reg_idx on q_vector %d VSI %d\n", + i, vsi->vsi_num); + goto clear_reg_idx; + } + + q_vector->reg_idx = q_vector->v_idx + vsi->hw_base_vector; + } + + return 0; + +clear_reg_idx: + ice_for_each_q_vector(vsi, i) { + struct ice_q_vector *q_vector = vsi->q_vectors[i]; + + if (q_vector) + q_vector->reg_idx = 0; + } + + return -EINVAL; +} + +/** + * ice_vsi_add_rem_eth_mac - Program VSI ethertype based filter with rule + * @vsi: the VSI being configured + * @add_rule: boolean value to add or remove ethertype filter rule + */ +static void +ice_vsi_add_rem_eth_mac(struct ice_vsi *vsi, bool add_rule) +{ + struct ice_fltr_list_entry *list; + struct ice_pf *pf = vsi->back; + LIST_HEAD(tmp_add_list); + enum ice_status status; + + list = devm_kzalloc(&pf->pdev->dev, sizeof(*list), GFP_KERNEL); + if (!list) + return; + + list->fltr_info.lkup_type = ICE_SW_LKUP_ETHERTYPE; + list->fltr_info.fltr_act = ICE_DROP_PACKET; + list->fltr_info.flag = ICE_FLTR_TX; + list->fltr_info.src_id = ICE_SRC_ID_VSI; + list->fltr_info.vsi_handle = vsi->idx; + list->fltr_info.l_data.ethertype_mac.ethertype = vsi->ethtype; + + INIT_LIST_HEAD(&list->list_entry); + list_add(&list->list_entry, &tmp_add_list); + + if (add_rule) + status = ice_add_eth_mac(&pf->hw, &tmp_add_list); + else + status = ice_remove_eth_mac(&pf->hw, &tmp_add_list); + + if (status) + dev_err(&pf->pdev->dev, + "Failure Adding or Removing Ethertype on VSI %i error: %d\n", + vsi->vsi_num, status); + + ice_free_fltr_list(&pf->pdev->dev, &tmp_add_list); +} + /** * ice_vsi_setup - Set up a VSI by a given type * @pf: board private structure * @pi: pointer to the port_info instance * @type: VSI type - * @vf_id: defines VF id to which this VSI connects. This field is meant to be + * @vf_id: defines VF ID to which this VSI connects. This field is meant to be * used only for ICE_VSI_VF VSI type. For other VSI types, should * fill-in ICE_INVAL_VFID as input. * @@ -2112,7 +2313,11 @@ ice_vsi_setup(struct ice_pf *pf, struct ice_port_info *pi, struct ice_vsi *vsi; int ret, i; - vsi = ice_vsi_alloc(pf, type); + if (type == ICE_VSI_VF) + vsi = ice_vsi_alloc(pf, type, vf_id); + else + vsi = ice_vsi_alloc(pf, type, ICE_INVAL_VFID); + if (!vsi) { dev_err(dev, "could not allocate VSI\n"); return NULL; @@ -2120,6 +2325,9 @@ ice_vsi_setup(struct ice_pf *pf, struct ice_port_info *pi, vsi->port_info = pi; vsi->vsw = pf->first_sw; + if (vsi->type == ICE_VSI_PF) + vsi->ethtype = ETH_P_PAUSE; + if (vsi->type == ICE_VSI_VF) vsi->vf_id = vf_id; @@ -2132,7 +2340,7 @@ ice_vsi_setup(struct ice_pf *pf, struct ice_port_info *pi, /* set RSS capabilities */ ice_vsi_set_rss_params(vsi); - /* set tc configuration */ + /* set TC configuration */ ice_vsi_set_tc_cfg(vsi); /* create the VSI */ @@ -2150,6 +2358,10 @@ ice_vsi_setup(struct ice_pf *pf, struct ice_port_info *pi, if (ret) goto unroll_alloc_q_vector; + ret = ice_vsi_set_q_vectors_reg_idx(vsi); + if (ret) + goto unroll_vector_base; + ret = ice_vsi_alloc_rings(vsi); if (ret) goto unroll_vector_base; @@ -2188,6 +2400,10 @@ ice_vsi_setup(struct ice_pf *pf, struct ice_port_info *pi, } else { vsi->hw_base_vector = pf->vf[vf_id].first_vector_idx; } + ret = ice_vsi_set_q_vectors_reg_idx(vsi); + if (ret) + goto unroll_vector_base; + pf->q_left_tx -= vsi->alloc_txq; pf->q_left_rx -= vsi->alloc_rxq; break; @@ -2203,18 +2419,29 @@ ice_vsi_setup(struct ice_pf *pf, struct ice_port_info *pi, ret = ice_cfg_vsi_lan(vsi->port_info, vsi->idx, vsi->tc_cfg.ena_tc, max_txqs); if (ret) { - dev_info(&pf->pdev->dev, "Failed VSI lan queue config\n"); + dev_err(&pf->pdev->dev, + "VSI %d failed lan queue config, error %d\n", + vsi->vsi_num, ret); goto unroll_vector_base; } + /* Add switch rule to drop all Tx Flow Control Frames, of look up + * type ETHERTYPE from VSIs, and restrict malicious VF from sending + * out PAUSE or PFC frames. If enabled, FW can still send FC frames. + * The rule is added once for PF VSI in order to create appropriate + * recipe, since VSI/VSI list is ignored with drop action... + */ + if (vsi->type == ICE_VSI_PF) + ice_vsi_add_rem_eth_mac(vsi, true); + return vsi; unroll_vector_base: /* reclaim SW interrupts back to the common pool */ - ice_free_res(vsi->back->sw_irq_tracker, vsi->sw_base_vector, vsi->idx); + ice_free_res(pf->sw_irq_tracker, vsi->sw_base_vector, vsi->idx); pf->num_avail_sw_msix += vsi->num_q_vectors; /* reclaim HW interrupt back to the common pool */ - ice_free_res(vsi->back->hw_irq_tracker, vsi->hw_base_vector, vsi->idx); + ice_free_res(pf->hw_irq_tracker, vsi->hw_base_vector, vsi->idx); pf->num_avail_hw_msix += vsi->num_q_vectors; unroll_alloc_q_vector: ice_vsi_free_q_vectors(vsi); @@ -2281,7 +2508,7 @@ void ice_vsi_free_irq(struct ice_vsi *vsi) return; vsi->irqs_ready = false; - for (i = 0; i < vsi->num_q_vectors; i++) { + ice_for_each_q_vector(vsi, i) { u16 vector = i + base; int irq_num; @@ -2500,12 +2727,12 @@ void ice_vsi_dis_irq(struct ice_vsi *vsi) /* disable each interrupt */ if (test_bit(ICE_FLAG_MSIX_ENA, pf->flags)) { - for (i = vsi->hw_base_vector; - i < (vsi->num_q_vectors + vsi->hw_base_vector); i++) - wr32(hw, GLINT_DYN_CTL(i), 0); + ice_for_each_q_vector(vsi, i) + wr32(hw, GLINT_DYN_CTL(vsi->q_vectors[i]->reg_idx), 0); ice_flush(hw); - for (i = 0; i < vsi->num_q_vectors; i++) + + ice_for_each_q_vector(vsi, i) synchronize_irq(pf->msix_entries[i + base].vector); } } @@ -2551,22 +2778,23 @@ int ice_vsi_release(struct ice_vsi *vsi) /* reclaim interrupt vectors back to PF */ if (vsi->type != ICE_VSI_VF) { /* reclaim SW interrupts back to the common pool */ - ice_free_res(vsi->back->sw_irq_tracker, vsi->sw_base_vector, - vsi->idx); + ice_free_res(pf->sw_irq_tracker, vsi->sw_base_vector, vsi->idx); pf->num_avail_sw_msix += vsi->num_q_vectors; /* reclaim HW interrupts back to the common pool */ - ice_free_res(vsi->back->hw_irq_tracker, vsi->hw_base_vector, - vsi->idx); + ice_free_res(pf->hw_irq_tracker, vsi->hw_base_vector, vsi->idx); pf->num_avail_hw_msix += vsi->num_q_vectors; } else if (test_bit(ICE_VF_STATE_CFG_INTR, vf->vf_states)) { /* Reclaim VF resources back only while freeing all VFs or * vector reassignment is requested */ - ice_free_res(vsi->back->hw_irq_tracker, vf->first_vector_idx, + ice_free_res(pf->hw_irq_tracker, vf->first_vector_idx, vsi->idx); pf->num_avail_hw_msix += pf->num_vf_msix; } + if (vsi->type == ICE_VSI_PF) + ice_vsi_add_rem_eth_mac(vsi, false); + ice_remove_vsi_fltr(&pf->hw, vsi->idx); ice_rm_vsi_lan_cfg(vsi->port_info, vsi->idx); ice_vsi_delete(vsi); @@ -2596,6 +2824,7 @@ int ice_vsi_release(struct ice_vsi *vsi) int ice_vsi_rebuild(struct ice_vsi *vsi) { u16 max_txqs[ICE_MAX_TRAFFIC_CLASS] = { 0 }; + struct ice_vf *vf = NULL; struct ice_pf *pf; int ret, i; @@ -2603,16 +2832,38 @@ int ice_vsi_rebuild(struct ice_vsi *vsi) return -EINVAL; pf = vsi->back; + if (vsi->type == ICE_VSI_VF) + vf = &pf->vf[vsi->vf_id]; + ice_rm_vsi_lan_cfg(vsi->port_info, vsi->idx); ice_vsi_free_q_vectors(vsi); - ice_free_res(vsi->back->sw_irq_tracker, vsi->sw_base_vector, vsi->idx); - ice_free_res(vsi->back->hw_irq_tracker, vsi->hw_base_vector, vsi->idx); - vsi->sw_base_vector = 0; + + if (vsi->type != ICE_VSI_VF) { + /* reclaim SW interrupts back to the common pool */ + ice_free_res(pf->sw_irq_tracker, vsi->sw_base_vector, vsi->idx); + pf->num_avail_sw_msix += vsi->num_q_vectors; + vsi->sw_base_vector = 0; + /* reclaim HW interrupts back to the common pool */ + ice_free_res(pf->hw_irq_tracker, vsi->hw_base_vector, + vsi->idx); + pf->num_avail_hw_msix += vsi->num_q_vectors; + } else { + /* Reclaim VF resources back to the common pool for reset and + * and rebuild, with vector reassignment + */ + ice_free_res(pf->hw_irq_tracker, vf->first_vector_idx, + vsi->idx); + pf->num_avail_hw_msix += pf->num_vf_msix; + } vsi->hw_base_vector = 0; + ice_vsi_clear_rings(vsi); - ice_vsi_free_arrays(vsi, false); - ice_dev_onetime_setup(&vsi->back->hw); - ice_vsi_set_num_qs(vsi); + ice_vsi_free_arrays(vsi); + ice_dev_onetime_setup(&pf->hw); + if (vsi->type == ICE_VSI_VF) + ice_vsi_set_num_qs(vsi, vf->vf_id); + else + ice_vsi_set_num_qs(vsi, ICE_INVAL_VFID); ice_vsi_set_tc_cfg(vsi); /* Initialize VSI struct elements and create VSI in FW */ @@ -2620,7 +2871,7 @@ int ice_vsi_rebuild(struct ice_vsi *vsi) if (ret < 0) goto err_vsi; - ret = ice_vsi_alloc_arrays(vsi, false); + ret = ice_vsi_alloc_arrays(vsi); if (ret < 0) goto err_vsi; @@ -2634,6 +2885,10 @@ int ice_vsi_rebuild(struct ice_vsi *vsi) if (ret) goto err_vectors; + ret = ice_vsi_set_q_vectors_reg_idx(vsi); + if (ret) + goto err_vectors; + ret = ice_vsi_alloc_rings(vsi); if (ret) goto err_vectors; @@ -2643,7 +2898,7 @@ int ice_vsi_rebuild(struct ice_vsi *vsi) * receive traffic on first queue. Hence no need to capture * return value */ - if (test_bit(ICE_FLAG_RSS_ENA, vsi->back->flags)) + if (test_bit(ICE_FLAG_RSS_ENA, pf->flags)) ice_vsi_cfg_rss_lut_key(vsi); break; case ICE_VSI_VF: @@ -2655,12 +2910,16 @@ int ice_vsi_rebuild(struct ice_vsi *vsi) if (ret) goto err_vectors; + ret = ice_vsi_set_q_vectors_reg_idx(vsi); + if (ret) + goto err_vectors; + ret = ice_vsi_alloc_rings(vsi); if (ret) goto err_vectors; - vsi->back->q_left_tx -= vsi->alloc_txq; - vsi->back->q_left_rx -= vsi->alloc_rxq; + pf->q_left_tx -= vsi->alloc_txq; + pf->q_left_rx -= vsi->alloc_rxq; break; default: break; @@ -2673,8 +2932,9 @@ int ice_vsi_rebuild(struct ice_vsi *vsi) ret = ice_cfg_vsi_lan(vsi->port_info, vsi->idx, vsi->tc_cfg.ena_tc, max_txqs); if (ret) { - dev_info(&vsi->back->pdev->dev, - "Failed VSI lan queue config\n"); + dev_err(&pf->pdev->dev, + "VSI %d failed lan queue config, error %d\n", + vsi->vsi_num, ret); goto err_vectors; } return 0; @@ -2690,7 +2950,7 @@ err_rings: } err_vsi: ice_vsi_clear(vsi); - set_bit(__ICE_RESET_FAILED, vsi->back->state); + set_bit(__ICE_RESET_FAILED, pf->state); return ret; } @@ -2705,3 +2965,125 @@ bool ice_is_reset_in_progress(unsigned long *state) test_bit(__ICE_CORER_REQ, state) || test_bit(__ICE_GLOBR_REQ, state); } + +#ifdef CONFIG_DCB +/** + * ice_vsi_update_q_map - update our copy of the VSI info with new queue map + * @vsi: VSI being configured + * @ctx: the context buffer returned from AQ VSI update command + */ +static void ice_vsi_update_q_map(struct ice_vsi *vsi, struct ice_vsi_ctx *ctx) +{ + vsi->info.mapping_flags = ctx->info.mapping_flags; + memcpy(&vsi->info.q_mapping, &ctx->info.q_mapping, + sizeof(vsi->info.q_mapping)); + memcpy(&vsi->info.tc_mapping, ctx->info.tc_mapping, + sizeof(vsi->info.tc_mapping)); +} + +/** + * ice_vsi_cfg_netdev_tc - Setup the netdev TC configuration + * @vsi: the VSI being configured + * @ena_tc: TC map to be enabled + */ +static void ice_vsi_cfg_netdev_tc(struct ice_vsi *vsi, u8 ena_tc) +{ + struct net_device *netdev = vsi->netdev; + struct ice_pf *pf = vsi->back; + struct ice_dcbx_cfg *dcbcfg; + u8 netdev_tc; + int i; + + if (!netdev) + return; + + if (!ena_tc) { + netdev_reset_tc(netdev); + return; + } + + if (netdev_set_num_tc(netdev, vsi->tc_cfg.numtc)) + return; + + dcbcfg = &pf->hw.port_info->local_dcbx_cfg; + + ice_for_each_traffic_class(i) + if (vsi->tc_cfg.ena_tc & BIT(i)) + netdev_set_tc_queue(netdev, + vsi->tc_cfg.tc_info[i].netdev_tc, + vsi->tc_cfg.tc_info[i].qcount_tx, + vsi->tc_cfg.tc_info[i].qoffset); + + for (i = 0; i < ICE_MAX_USER_PRIORITY; i++) { + u8 ets_tc = dcbcfg->etscfg.prio_table[i]; + + /* Get the mapped netdev TC# for the UP */ + netdev_tc = vsi->tc_cfg.tc_info[ets_tc].netdev_tc; + netdev_set_prio_tc_map(netdev, i, netdev_tc); + } +} + +/** + * ice_vsi_cfg_tc - Configure VSI Tx Sched for given TC map + * @vsi: VSI to be configured + * @ena_tc: TC bitmap + * + * VSI queues expected to be quiesced before calling this function + */ +int ice_vsi_cfg_tc(struct ice_vsi *vsi, u8 ena_tc) +{ + u16 max_txqs[ICE_MAX_TRAFFIC_CLASS] = { 0 }; + struct ice_vsi_ctx *ctx; + struct ice_pf *pf = vsi->back; + enum ice_status status; + int i, ret = 0; + u8 num_tc = 0; + + ice_for_each_traffic_class(i) { + /* build bitmap of enabled TCs */ + if (ena_tc & BIT(i)) + num_tc++; + /* populate max_txqs per TC */ + max_txqs[i] = pf->num_lan_tx; + } + + vsi->tc_cfg.ena_tc = ena_tc; + vsi->tc_cfg.numtc = num_tc; + + ctx = devm_kzalloc(&pf->pdev->dev, sizeof(*ctx), GFP_KERNEL); + if (!ctx) + return -ENOMEM; + + ctx->vf_num = 0; + ctx->info = vsi->info; + + ice_vsi_setup_q_map(vsi, ctx); + + /* must to indicate which section of VSI context are being modified */ + ctx->info.valid_sections = cpu_to_le16(ICE_AQ_VSI_PROP_RXQ_MAP_VALID); + status = ice_update_vsi(&pf->hw, vsi->idx, ctx, NULL); + if (status) { + dev_info(&pf->pdev->dev, "Failed VSI Update\n"); + ret = -EIO; + goto out; + } + + status = ice_cfg_vsi_lan(vsi->port_info, vsi->idx, vsi->tc_cfg.ena_tc, + max_txqs); + + if (status) { + dev_err(&pf->pdev->dev, + "VSI %d failed TC config, error %d\n", + vsi->vsi_num, status); + ret = -EIO; + goto out; + } + ice_vsi_update_q_map(vsi, ctx); + vsi->info.valid_sections = 0; + + ice_vsi_cfg_netdev_tc(vsi, ena_tc); +out: + devm_kfree(&pf->pdev->dev, ctx); + return ret; +} +#endif /* CONFIG_DCB */ diff --git a/drivers/net/ethernet/intel/ice/ice_lib.h b/drivers/net/ethernet/intel/ice/ice_lib.h index 7988a53729a92b3fdfc1d4bc82359e744275d723..a91d3553cc89bd53ff2042cc5dc33ea2d6b21ebe 100644 --- a/drivers/net/ethernet/intel/ice/ice_lib.h +++ b/drivers/net/ethernet/intel/ice/ice_lib.h @@ -35,12 +35,16 @@ int ice_vsi_stop_lan_tx_rings(struct ice_vsi *vsi, enum ice_disq_rst_src rst_src, u16 rel_vmvf_num); -int ice_cfg_vlan_pruning(struct ice_vsi *vsi, bool ena); +int ice_cfg_vlan_pruning(struct ice_vsi *vsi, bool ena, bool vlan_promisc); void ice_vsi_delete(struct ice_vsi *vsi); int ice_vsi_clear(struct ice_vsi *vsi); +#ifdef CONFIG_DCB +int ice_vsi_cfg_tc(struct ice_vsi *vsi, u8 ena_tc); +#endif /* CONFIG_DCB */ + struct ice_vsi * ice_vsi_setup(struct ice_pf *pf, struct ice_port_info *pi, enum ice_vsi_type type, u16 vf_id); @@ -62,6 +66,10 @@ void ice_vsi_free_q_vectors(struct ice_vsi *vsi); void ice_vsi_put_qs(struct ice_vsi *vsi); +#ifdef CONFIG_DCB +void ice_vsi_map_rings_to_vectors(struct ice_vsi *vsi); +#endif /* CONFIG_DCB */ + void ice_vsi_dis_irq(struct ice_vsi *vsi); void ice_vsi_free_irq(struct ice_vsi *vsi); @@ -70,8 +78,7 @@ void ice_vsi_free_rx_rings(struct ice_vsi *vsi); void ice_vsi_free_tx_rings(struct ice_vsi *vsi); -int ice_vsi_cfg_tc(struct ice_vsi *vsi, u8 ena_tc); - int ice_vsi_manage_rss_lut(struct ice_vsi *vsi, bool ena); +u32 ice_intrl_usec_to_reg(u8 intrl, u8 gran); #endif /* !_ICE_LIB_H_ */ diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c index 47cc3f905b7ffc7e3bc83725bc6fb53652b1bb30..7843abf4d44df59309e9f33a887674a4782ad6bf 100644 --- a/drivers/net/ethernet/intel/ice/ice_main.c +++ b/drivers/net/ethernet/intel/ice/ice_main.c @@ -7,8 +7,9 @@ #include "ice.h" #include "ice_lib.h" +#include "ice_dcb_lib.h" -#define DRV_VERSION "0.7.2-k" +#define DRV_VERSION "0.7.4-k" #define DRV_SUMMARY "Intel(R) Ethernet Connection E800 Series Linux Driver" const char ice_drv_ver[] = DRV_VERSION; static const char ice_driver_string[] = DRV_SUMMARY; @@ -30,7 +31,6 @@ MODULE_PARM_DESC(debug, "netif level (0=none,...,16=all)"); static struct workqueue_struct *ice_wq; static const struct net_device_ops ice_netdev_ops; -static void ice_pf_dis_all_vsi(struct ice_pf *pf); static void ice_rebuild(struct ice_pf *pf); static void ice_vsi_release_all(struct ice_pf *pf); @@ -113,14 +113,14 @@ static void ice_check_for_hang_subtask(struct ice_pf *pf) } /** - * ice_add_mac_to_sync_list - creates list of mac addresses to be synced + * ice_add_mac_to_sync_list - creates list of MAC addresses to be synced * @netdev: the net device on which the sync is happening - * @addr: mac address to sync + * @addr: MAC address to sync * * This is a callback function which is called by the in kernel device sync * functions (like __dev_uc_sync, __dev_mc_sync, etc). This function only * populates the tmp_sync_list, which is later used by ice_add_mac to add the - * mac filters from the hardware. + * MAC filters from the hardware. */ static int ice_add_mac_to_sync_list(struct net_device *netdev, const u8 *addr) { @@ -134,14 +134,14 @@ static int ice_add_mac_to_sync_list(struct net_device *netdev, const u8 *addr) } /** - * ice_add_mac_to_unsync_list - creates list of mac addresses to be unsynced + * ice_add_mac_to_unsync_list - creates list of MAC addresses to be unsynced * @netdev: the net device on which the unsync is happening - * @addr: mac address to unsync + * @addr: MAC address to unsync * * This is a callback function which is called by the in kernel device unsync * functions (like __dev_uc_unsync, __dev_mc_unsync, etc). This function only * populates the tmp_unsync_list, which is later used by ice_remove_mac to - * delete the mac filters from the hardware. + * delete the MAC filters from the hardware. */ static int ice_add_mac_to_unsync_list(struct net_device *netdev, const u8 *addr) { @@ -167,6 +167,39 @@ static bool ice_vsi_fltr_changed(struct ice_vsi *vsi) test_bit(ICE_VSI_FLAG_VLAN_FLTR_CHANGED, vsi->flags); } +/** + * ice_cfg_promisc - Enable or disable promiscuous mode for a given PF + * @vsi: the VSI being configured + * @promisc_m: mask of promiscuous config bits + * @set_promisc: enable or disable promisc flag request + * + */ +static int ice_cfg_promisc(struct ice_vsi *vsi, u8 promisc_m, bool set_promisc) +{ + struct ice_hw *hw = &vsi->back->hw; + enum ice_status status = 0; + + if (vsi->type != ICE_VSI_PF) + return 0; + + if (vsi->vlan_ena) { + status = ice_set_vlan_vsi_promisc(hw, vsi->idx, promisc_m, + set_promisc); + } else { + if (set_promisc) + status = ice_set_vsi_promisc(hw, vsi->idx, promisc_m, + 0); + else + status = ice_clear_vsi_promisc(hw, vsi->idx, promisc_m, + 0); + } + + if (status) + return -EIO; + + return 0; +} + /** * ice_vsi_sync_fltr - Update the VSI filter list to the HW * @vsi: ptr to the VSI @@ -182,6 +215,7 @@ static int ice_vsi_sync_fltr(struct ice_vsi *vsi) struct ice_hw *hw = &pf->hw; enum ice_status status = 0; u32 changed_flags = 0; + u8 promisc_m; int err = 0; if (!vsi->netdev) @@ -211,7 +245,7 @@ static int ice_vsi_sync_fltr(struct ice_vsi *vsi) netif_addr_unlock_bh(netdev); } - /* Remove mac addresses in the unsync list */ + /* Remove MAC addresses in the unsync list */ status = ice_remove_mac(hw, &vsi->tmp_unsync_list); ice_free_fltr_list(dev, &vsi->tmp_unsync_list); if (status) { @@ -223,12 +257,16 @@ static int ice_vsi_sync_fltr(struct ice_vsi *vsi) } } - /* Add mac addresses in the sync list */ + /* Add MAC addresses in the sync list */ status = ice_add_mac(hw, &vsi->tmp_sync_list); ice_free_fltr_list(dev, &vsi->tmp_sync_list); - if (status) { + /* If filter is added successfully or already exists, do not go into + * 'if' condition and report it as error. Instead continue processing + * rest of the function. + */ + if (status && status != ICE_ERR_ALREADY_EXISTS) { netdev_err(netdev, "Failed to add MAC filters\n"); - /* If there is no more space for new umac filters, vsi + /* If there is no more space for new umac filters, VSI * should go into promiscuous mode. There should be some * space reserved for promiscuous filters. */ @@ -245,49 +283,56 @@ static int ice_vsi_sync_fltr(struct ice_vsi *vsi) } } /* check for changes in promiscuous modes */ - if (changed_flags & IFF_ALLMULTI) - netdev_warn(netdev, "Unsupported configuration\n"); + if (changed_flags & IFF_ALLMULTI) { + if (vsi->current_netdev_flags & IFF_ALLMULTI) { + if (vsi->vlan_ena) + promisc_m = ICE_MCAST_VLAN_PROMISC_BITS; + else + promisc_m = ICE_MCAST_PROMISC_BITS; + + err = ice_cfg_promisc(vsi, promisc_m, true); + if (err) { + netdev_err(netdev, "Error setting Multicast promiscuous mode on VSI %i\n", + vsi->vsi_num); + vsi->current_netdev_flags &= ~IFF_ALLMULTI; + goto out_promisc; + } + } else if (!(vsi->current_netdev_flags & IFF_ALLMULTI)) { + if (vsi->vlan_ena) + promisc_m = ICE_MCAST_VLAN_PROMISC_BITS; + else + promisc_m = ICE_MCAST_PROMISC_BITS; + + err = ice_cfg_promisc(vsi, promisc_m, false); + if (err) { + netdev_err(netdev, "Error clearing Multicast promiscuous mode on VSI %i\n", + vsi->vsi_num); + vsi->current_netdev_flags |= IFF_ALLMULTI; + goto out_promisc; + } + } + } if (((changed_flags & IFF_PROMISC) || promisc_forced_on) || test_bit(ICE_VSI_FLAG_PROMISC_CHANGED, vsi->flags)) { clear_bit(ICE_VSI_FLAG_PROMISC_CHANGED, vsi->flags); if (vsi->current_netdev_flags & IFF_PROMISC) { - /* Apply TX filter rule to get traffic from VMs */ - status = ice_cfg_dflt_vsi(hw, vsi->idx, true, - ICE_FLTR_TX); - if (status) { - netdev_err(netdev, "Error setting default VSI %i tx rule\n", - vsi->vsi_num); - vsi->current_netdev_flags &= ~IFF_PROMISC; - err = -EIO; - goto out_promisc; - } - /* Apply RX filter rule to get traffic from wire */ + /* Apply Rx filter rule to get traffic from wire */ status = ice_cfg_dflt_vsi(hw, vsi->idx, true, ICE_FLTR_RX); if (status) { - netdev_err(netdev, "Error setting default VSI %i rx rule\n", + netdev_err(netdev, "Error setting default VSI %i Rx rule\n", vsi->vsi_num); vsi->current_netdev_flags &= ~IFF_PROMISC; err = -EIO; goto out_promisc; } } else { - /* Clear TX filter rule to stop traffic from VMs */ - status = ice_cfg_dflt_vsi(hw, vsi->idx, false, - ICE_FLTR_TX); - if (status) { - netdev_err(netdev, "Error clearing default VSI %i tx rule\n", - vsi->vsi_num); - vsi->current_netdev_flags |= IFF_PROMISC; - err = -EIO; - goto out_promisc; - } - /* Clear RX filter to remove traffic from wire */ + /* Clear Rx filter to remove traffic from wire */ status = ice_cfg_dflt_vsi(hw, vsi->idx, false, ICE_FLTR_RX); if (status) { - netdev_err(netdev, "Error clearing default VSI %i rx rule\n", + netdev_err(netdev, "Error clearing default VSI %i Rx rule\n", vsi->vsi_num); vsi->current_netdev_flags |= IFF_PROMISC; err = -EIO; @@ -322,7 +367,7 @@ static void ice_sync_fltr_subtask(struct ice_pf *pf) clear_bit(ICE_FLAG_FLTR_SYNC, pf->flags); - for (v = 0; v < pf->num_alloc_vsi; v++) + ice_for_each_vsi(pf, v) if (pf->vsi[v] && ice_vsi_fltr_changed(pf->vsi[v]) && ice_vsi_sync_fltr(pf->vsi[v])) { /* come back and try again later */ @@ -331,6 +376,51 @@ static void ice_sync_fltr_subtask(struct ice_pf *pf) } } +/** + * ice_dis_vsi - pause a VSI + * @vsi: the VSI being paused + * @locked: is the rtnl_lock already held + */ +static void ice_dis_vsi(struct ice_vsi *vsi, bool locked) +{ + if (test_bit(__ICE_DOWN, vsi->state)) + return; + + set_bit(__ICE_NEEDS_RESTART, vsi->state); + + if (vsi->type == ICE_VSI_PF && vsi->netdev) { + if (netif_running(vsi->netdev)) { + if (!locked) { + rtnl_lock(); + vsi->netdev->netdev_ops->ndo_stop(vsi->netdev); + rtnl_unlock(); + } else { + vsi->netdev->netdev_ops->ndo_stop(vsi->netdev); + } + } else { + ice_vsi_close(vsi); + } + } +} + +/** + * ice_pf_dis_all_vsi - Pause all VSIs on a PF + * @pf: the PF + * @locked: is the rtnl_lock already held + */ +#ifdef CONFIG_DCB +void ice_pf_dis_all_vsi(struct ice_pf *pf, bool locked) +#else +static void ice_pf_dis_all_vsi(struct ice_pf *pf, bool locked) +#endif /* CONFIG_DCB */ +{ + int v; + + ice_for_each_vsi(pf, v) + if (pf->vsi[v]) + ice_dis_vsi(pf->vsi[v], locked); +} + /** * ice_prepare_for_reset - prep for the core to reset * @pf: board private structure @@ -342,12 +432,16 @@ ice_prepare_for_reset(struct ice_pf *pf) { struct ice_hw *hw = &pf->hw; + /* already prepared for reset */ + if (test_bit(__ICE_PREPARED_FOR_RESET, pf->state)) + return; + /* Notify VFs of impending reset */ if (ice_check_sq_alive(hw, &hw->mailboxq)) ice_vc_notify_reset(pf); /* disable the VSIs and their queues that are not already DOWN */ - ice_pf_dis_all_vsi(pf); + ice_pf_dis_all_vsi(pf, false); if (hw->port_info) ice_sched_clear_port(hw->port_info); @@ -394,6 +488,7 @@ static void ice_do_reset(struct ice_pf *pf, enum ice_reset_req reset_type) ice_rebuild(pf); clear_bit(__ICE_PREPARED_FOR_RESET, pf->state); clear_bit(__ICE_PFR_REQ, pf->state); + ice_reset_all_vfs(pf, true); } } @@ -416,10 +511,15 @@ static void ice_reset_subtask(struct ice_pf *pf) * for the reset now), poll for reset done, rebuild and return. */ if (test_bit(__ICE_RESET_OICR_RECV, pf->state)) { - clear_bit(__ICE_GLOBR_RECV, pf->state); - clear_bit(__ICE_CORER_RECV, pf->state); - if (!test_bit(__ICE_PREPARED_FOR_RESET, pf->state)) - ice_prepare_for_reset(pf); + /* Perform the largest reset requested */ + if (test_and_clear_bit(__ICE_CORER_RECV, pf->state)) + reset_type = ICE_RESET_CORER; + if (test_and_clear_bit(__ICE_GLOBR_RECV, pf->state)) + reset_type = ICE_RESET_GLOBR; + /* return if no valid reset type requested */ + if (reset_type == ICE_RESET_INVAL) + return; + ice_prepare_for_reset(pf); /* make sure we are ready to rebuild */ if (ice_check_reset(&pf->hw)) { @@ -429,13 +529,14 @@ static void ice_reset_subtask(struct ice_pf *pf) pf->hw.reset_ongoing = false; ice_rebuild(pf); /* clear bit to resume normal operations, but - * ICE_NEEDS_RESTART bit is set incase rebuild failed + * ICE_NEEDS_RESTART bit is set in case rebuild failed */ clear_bit(__ICE_RESET_OICR_RECV, pf->state); clear_bit(__ICE_PREPARED_FOR_RESET, pf->state); clear_bit(__ICE_PFR_REQ, pf->state); clear_bit(__ICE_CORER_REQ, pf->state); clear_bit(__ICE_GLOBR_REQ, pf->state); + ice_reset_all_vfs(pf, true); } return; @@ -469,6 +570,9 @@ void ice_print_link_msg(struct ice_vsi *vsi, bool isup) const char *speed; const char *fc; + if (!vsi) + return; + if (vsi->current_isup == isup) return; @@ -519,6 +623,9 @@ void ice_print_link_msg(struct ice_vsi *vsi, bool isup) case ICE_FC_RX_PAUSE: fc = "RX"; break; + case ICE_FC_NONE: + fc = "None"; + break; default: fc = "Unknown"; break; @@ -529,21 +636,22 @@ void ice_print_link_msg(struct ice_vsi *vsi, bool isup) } /** - * ice_vsi_link_event - update the vsi's netdev - * @vsi: the vsi on which the link event occurred - * @link_up: whether or not the vsi needs to be set up or down + * ice_vsi_link_event - update the VSI's netdev + * @vsi: the VSI on which the link event occurred + * @link_up: whether or not the VSI needs to be set up or down */ static void ice_vsi_link_event(struct ice_vsi *vsi, bool link_up) { - if (!vsi || test_bit(__ICE_DOWN, vsi->state)) + if (!vsi) + return; + + if (test_bit(__ICE_DOWN, vsi->state) || !vsi->netdev) return; if (vsi->type == ICE_VSI_PF) { - if (!vsi->netdev) { - dev_dbg(&vsi->back->pdev->dev, - "vsi->netdev is not initialized!\n"); + if (link_up == netif_carrier_ok(vsi->netdev)) return; - } + if (link_up) { netif_carrier_on(vsi->netdev); netif_tx_wake_all_queues(vsi->netdev); @@ -558,61 +666,51 @@ static void ice_vsi_link_event(struct ice_vsi *vsi, bool link_up) * ice_link_event - process the link event * @pf: pf that the link event is associated with * @pi: port_info for the port that the link event is associated with + * @link_up: true if the physical link is up and false if it is down + * @link_speed: current link speed received from the link event * - * Returns -EIO if ice_get_link_status() fails - * Returns 0 on success + * Returns 0 on success and negative on failure */ static int -ice_link_event(struct ice_pf *pf, struct ice_port_info *pi) +ice_link_event(struct ice_pf *pf, struct ice_port_info *pi, bool link_up, + u16 link_speed) { - u8 new_link_speed, old_link_speed; struct ice_phy_info *phy_info; - bool new_link_same_as_old; - bool new_link, old_link; - u8 lport; - u16 v; + struct ice_vsi *vsi; + u16 old_link_speed; + bool old_link; + int result; phy_info = &pi->phy; phy_info->link_info_old = phy_info->link_info; - /* Force ice_get_link_status() to update link info */ - phy_info->get_link_info = true; - old_link = (phy_info->link_info_old.link_info & ICE_AQ_LINK_UP); + old_link = !!(phy_info->link_info_old.link_info & ICE_AQ_LINK_UP); old_link_speed = phy_info->link_info_old.link_speed; - lport = pi->lport; - if (ice_get_link_status(pi, &new_link)) { + /* update the link info structures and re-enable link events, + * don't bail on failure due to other book keeping needed + */ + result = ice_update_link_info(pi); + if (result) dev_dbg(&pf->pdev->dev, - "Could not get link status for port %d\n", lport); - return -EIO; - } - - new_link_speed = phy_info->link_info.link_speed; + "Failed to update link status and re-enable link events for port %d\n", + pi->lport); - new_link_same_as_old = (new_link == old_link && - new_link_speed == old_link_speed); - - ice_for_each_vsi(pf, v) { - struct ice_vsi *vsi = pf->vsi[v]; - - if (!vsi || !vsi->port_info) - continue; + /* if the old link up/down and speed is the same as the new */ + if (link_up == old_link && link_speed == old_link_speed) + return result; - if (new_link_same_as_old && - (test_bit(__ICE_DOWN, vsi->state) || - new_link == netif_carrier_ok(vsi->netdev))) - continue; + vsi = ice_find_vsi_by_type(pf, ICE_VSI_PF); + if (!vsi || !vsi->port_info) + return -EINVAL; - if (vsi->port_info->lport == lport) { - ice_print_link_msg(vsi, new_link); - ice_vsi_link_event(vsi, new_link); - } - } + ice_vsi_link_event(vsi, link_up); + ice_print_link_msg(vsi, link_up); - if (!new_link_same_as_old && pf->num_alloc_vfs) + if (pf->num_alloc_vfs) ice_vc_notify_link_state(pf); - return 0; + return result; } /** @@ -635,18 +733,72 @@ static void ice_watchdog_subtask(struct ice_pf *pf) pf->serv_tmr_prev = jiffies; - if (ice_link_event(pf, pf->hw.port_info)) - dev_dbg(&pf->pdev->dev, "ice_link_event failed\n"); - /* Update the stats for active netdevs so the network stack * can look at updated numbers whenever it cares to */ ice_update_pf_stats(pf); - for (i = 0; i < pf->num_alloc_vsi; i++) + ice_for_each_vsi(pf, i) if (pf->vsi[i] && pf->vsi[i]->netdev) ice_update_vsi_stats(pf->vsi[i]); } +/** + * ice_init_link_events - enable/initialize link events + * @pi: pointer to the port_info instance + * + * Returns -EIO on failure, 0 on success + */ +static int ice_init_link_events(struct ice_port_info *pi) +{ + u16 mask; + + mask = ~((u16)(ICE_AQ_LINK_EVENT_UPDOWN | ICE_AQ_LINK_EVENT_MEDIA_NA | + ICE_AQ_LINK_EVENT_MODULE_QUAL_FAIL)); + + if (ice_aq_set_event_mask(pi->hw, pi->lport, mask, NULL)) { + dev_dbg(ice_hw_to_dev(pi->hw), + "Failed to set link event mask for port %d\n", + pi->lport); + return -EIO; + } + + if (ice_aq_get_link_info(pi, true, NULL, NULL)) { + dev_dbg(ice_hw_to_dev(pi->hw), + "Failed to enable link events for port %d\n", + pi->lport); + return -EIO; + } + + return 0; +} + +/** + * ice_handle_link_event - handle link event via ARQ + * @pf: pf that the link event is associated with + * @event: event structure containing link status info + */ +static int +ice_handle_link_event(struct ice_pf *pf, struct ice_rq_event_info *event) +{ + struct ice_aqc_get_link_status_data *link_data; + struct ice_port_info *port_info; + int status; + + link_data = (struct ice_aqc_get_link_status_data *)event->msg_buf; + port_info = pf->hw.port_info; + if (!port_info) + return -EINVAL; + + status = ice_link_event(pf, port_info, + !!(link_data->link_info & ICE_AQ_LINK_UP), + le16_to_cpu(link_data->link_speed)); + if (status) + dev_dbg(&pf->pdev->dev, + "Could not process link event, error %d\n", status); + + return status; +} + /** * __ice_clean_ctrlq - helper function to clean controlq rings * @pf: ptr to struct ice_pf @@ -750,12 +902,20 @@ static int __ice_clean_ctrlq(struct ice_pf *pf, enum ice_ctl_q q_type) opcode = le16_to_cpu(event.desc.opcode); switch (opcode) { + case ice_aqc_opc_get_link_status: + if (ice_handle_link_event(pf, &event)) + dev_err(&pf->pdev->dev, + "Could not handle link event\n"); + break; case ice_mbx_opc_send_msg_to_pf: ice_vc_process_vf_msg(pf, &event); break; case ice_aqc_opc_fw_logging: ice_output_fw_log(hw, &event.desc, event.msg_buf); break; + case ice_aqc_opc_lldp_set_mib_change: + ice_dcb_process_lldp_set_mib_change(pf, &event); + break; default: dev_dbg(&pf->pdev->dev, "%s Receive Queue unknown event 0x%04x ignored\n", @@ -876,6 +1036,18 @@ static void ice_service_task_stop(struct ice_pf *pf) clear_bit(__ICE_SERVICE_SCHED, pf->state); } +/** + * ice_service_task_restart - restart service task and schedule works + * @pf: board private structure + * + * This function is needed for suspend and resume works (e.g WoL scenario) + */ +static void ice_service_task_restart(struct ice_pf *pf) +{ + clear_bit(__ICE_SERVICE_DIS, pf->state); + ice_service_task_schedule(pf); +} + /** * ice_service_timer - timer callback to schedule service task * @t: pointer to timer_list @@ -901,7 +1073,7 @@ static void ice_handle_mdd_event(struct ice_pf *pf) u32 reg; int i; - if (!test_bit(__ICE_MDD_EVENT_PENDING, pf->state)) + if (!test_and_clear_bit(__ICE_MDD_EVENT_PENDING, pf->state)) return; /* find what triggered the MDD event */ @@ -993,10 +1165,12 @@ static void ice_handle_mdd_event(struct ice_pf *pf) for (i = 0; i < pf->num_alloc_vfs && mdd_detected; i++) { struct ice_vf *vf = &pf->vf[i]; + mdd_detected = false; + reg = rd32(hw, VP_MDET_TX_PQM(i)); if (reg & VP_MDET_TX_PQM_VALID_M) { wr32(hw, VP_MDET_TX_PQM(i), 0xFFFF); - vf->num_mdd_events++; + mdd_detected = true; dev_info(&pf->pdev->dev, "TX driver issue detected on VF %d\n", i); } @@ -1004,7 +1178,7 @@ static void ice_handle_mdd_event(struct ice_pf *pf) reg = rd32(hw, VP_MDET_TX_TCLAN(i)); if (reg & VP_MDET_TX_TCLAN_VALID_M) { wr32(hw, VP_MDET_TX_TCLAN(i), 0xFFFF); - vf->num_mdd_events++; + mdd_detected = true; dev_info(&pf->pdev->dev, "TX driver issue detected on VF %d\n", i); } @@ -1012,7 +1186,7 @@ static void ice_handle_mdd_event(struct ice_pf *pf) reg = rd32(hw, VP_MDET_TX_TDPU(i)); if (reg & VP_MDET_TX_TDPU_VALID_M) { wr32(hw, VP_MDET_TX_TDPU(i), 0xFFFF); - vf->num_mdd_events++; + mdd_detected = true; dev_info(&pf->pdev->dev, "TX driver issue detected on VF %d\n", i); } @@ -1020,26 +1194,19 @@ static void ice_handle_mdd_event(struct ice_pf *pf) reg = rd32(hw, VP_MDET_RX(i)); if (reg & VP_MDET_RX_VALID_M) { wr32(hw, VP_MDET_RX(i), 0xFFFF); - vf->num_mdd_events++; + mdd_detected = true; dev_info(&pf->pdev->dev, "RX driver issue detected on VF %d\n", i); } - if (vf->num_mdd_events > ICE_DFLT_NUM_MDD_EVENTS_ALLOWED) { - dev_info(&pf->pdev->dev, - "Too many MDD events on VF %d, disabled\n", i); + if (mdd_detected) { + vf->num_mdd_events++; dev_info(&pf->pdev->dev, "Use PF Control I/F to re-enable the VF\n"); set_bit(ICE_VF_STATE_DIS, vf->vf_states); } } - /* re-enable MDD interrupt cause */ - clear_bit(__ICE_MDD_EVENT_PENDING, pf->state); - reg = rd32(hw, PFINT_OICR_ENA); - reg |= PFINT_OICR_MAL_DETECT_M; - wr32(hw, PFINT_OICR_ENA, reg); - ice_flush(hw); } /** @@ -1089,7 +1256,7 @@ static void ice_service_task(struct work_struct *work) /** * ice_set_ctrlq_len - helper function to set controlq length - * @hw: pointer to the hw instance + * @hw: pointer to the HW instance */ static void ice_set_ctrlq_len(struct ice_hw *hw) { @@ -1111,8 +1278,9 @@ static void ice_set_ctrlq_len(struct ice_hw *hw) * This is a callback function used by the irq_set_affinity_notifier function * so that we may register to receive changes to the irq affinity masks. */ -static void ice_irq_affinity_notify(struct irq_affinity_notify *notify, - const cpumask_t *mask) +static void +ice_irq_affinity_notify(struct irq_affinity_notify *notify, + const cpumask_t *mask) { struct ice_q_vector *q_vector = container_of(notify, struct ice_q_vector, affinity_notify); @@ -1142,7 +1310,7 @@ static int ice_vsi_ena_irq(struct ice_vsi *vsi) if (test_bit(ICE_FLAG_MSIX_ENA, pf->flags)) { int i; - for (i = 0; i < vsi->num_q_vectors; i++) + ice_for_each_q_vector(vsi, i) ice_irq_dynamic_ena(hw, vsi, vsi->q_vectors[i]); } @@ -1184,10 +1352,9 @@ static int ice_vsi_req_irq_msix(struct ice_vsi *vsi, char *basename) /* skip this unused q_vector */ continue; } - err = devm_request_irq(&pf->pdev->dev, - pf->msix_entries[base + vector].vector, - vsi->irq_handler, 0, q_vector->name, - q_vector); + err = devm_request_irq(&pf->pdev->dev, irq_num, + vsi->irq_handler, 0, + q_vector->name, q_vector); if (err) { netdev_err(vsi->netdev, "MSIX request_irq failed, error: %d\n", err); @@ -1328,7 +1495,7 @@ static irqreturn_t ice_misc_intr(int __always_unused irq, void *data) rd32(hw, PFHMC_ERRORDATA)); } - /* Report and mask off any remaining unexpected interrupts */ + /* Report any remaining unexpected interrupts */ oicr &= ena_mask; if (oicr) { dev_dbg(&pf->pdev->dev, "unhandled interrupt oicr=0x%08x\n", @@ -1342,12 +1509,9 @@ static irqreturn_t ice_misc_intr(int __always_unused irq, void *data) set_bit(__ICE_PFR_REQ, pf->state); ice_service_task_schedule(pf); } - ena_mask &= ~oicr; } ret = IRQ_HANDLED; - /* re-enable interrupt causes that are not handled during this pass */ - wr32(hw, PFINT_OICR_ENA, ena_mask); if (!test_bit(__ICE_DOWN, pf->state)) { ice_service_task_schedule(pf); ice_irq_dynamic_ena(hw, NULL, NULL); @@ -1406,23 +1570,23 @@ static void ice_free_irq_msix_misc(struct ice_pf *pf) /** * ice_ena_ctrlq_interrupts - enable control queue interrupts * @hw: pointer to HW structure - * @v_idx: HW vector index to associate the control queue interrupts with + * @reg_idx: HW vector index to associate the control queue interrupts with */ -static void ice_ena_ctrlq_interrupts(struct ice_hw *hw, u16 v_idx) +static void ice_ena_ctrlq_interrupts(struct ice_hw *hw, u16 reg_idx) { u32 val; - val = ((v_idx & PFINT_OICR_CTL_MSIX_INDX_M) | + val = ((reg_idx & PFINT_OICR_CTL_MSIX_INDX_M) | PFINT_OICR_CTL_CAUSE_ENA_M); wr32(hw, PFINT_OICR_CTL, val); /* enable Admin queue Interrupt causes */ - val = ((v_idx & PFINT_FW_CTL_MSIX_INDX_M) | + val = ((reg_idx & PFINT_FW_CTL_MSIX_INDX_M) | PFINT_FW_CTL_CAUSE_ENA_M); wr32(hw, PFINT_FW_CTL, val); /* enable Mailbox queue Interrupt causes */ - val = ((v_idx & PFINT_MBX_CTL_MSIX_INDX_M) | + val = ((reg_idx & PFINT_MBX_CTL_MSIX_INDX_M) | PFINT_MBX_CTL_CAUSE_ENA_M); wr32(hw, PFINT_MBX_CTL, val); @@ -1510,7 +1674,7 @@ void ice_napi_del(struct ice_vsi *vsi) if (!vsi->netdev) return; - for (v_idx = 0; v_idx < vsi->num_q_vectors; v_idx++) + ice_for_each_q_vector(vsi, v_idx) netif_napi_del(&vsi->q_vectors[v_idx]->napi); } @@ -1529,7 +1693,7 @@ static void ice_napi_add(struct ice_vsi *vsi) if (!vsi->netdev) return; - for (v_idx = 0; v_idx < vsi->num_q_vectors; v_idx++) + ice_for_each_q_vector(vsi, v_idx) netif_napi_add(vsi->netdev, &vsi->q_vectors[v_idx]->napi, ice_napi_poll, NAPI_POLL_WEIGHT); } @@ -1649,18 +1813,20 @@ ice_pf_vsi_setup(struct ice_pf *pf, struct ice_port_info *pi) } /** - * ice_vlan_rx_add_vid - Add a vlan id filter to HW offload + * ice_vlan_rx_add_vid - Add a VLAN ID filter to HW offload * @netdev: network interface to be adjusted * @proto: unused protocol - * @vid: vlan id to be added + * @vid: VLAN ID to be added * - * net_device_ops implementation for adding vlan ids + * net_device_ops implementation for adding VLAN IDs */ -static int ice_vlan_rx_add_vid(struct net_device *netdev, - __always_unused __be16 proto, u16 vid) +static int +ice_vlan_rx_add_vid(struct net_device *netdev, __always_unused __be16 proto, + u16 vid) { struct ice_netdev_priv *np = netdev_priv(netdev); struct ice_vsi *vsi = np->vsi; + int ret; if (vid >= VLAN_N_VID) { netdev_err(netdev, "VLAN id requested %d is out of range %d\n", @@ -1673,33 +1839,39 @@ static int ice_vlan_rx_add_vid(struct net_device *netdev, /* Enable VLAN pruning when VLAN 0 is added */ if (unlikely(!vid)) { - int ret = ice_cfg_vlan_pruning(vsi, true); - + ret = ice_cfg_vlan_pruning(vsi, true, false); if (ret) return ret; } - /* Add all VLAN ids including 0 to the switch filter. VLAN id 0 is + /* Add all VLAN IDs including 0 to the switch filter. VLAN ID 0 is * needed to continue allowing all untagged packets since VLAN prune * list is applied to all packets by the switch */ - return ice_vsi_add_vlan(vsi, vid); + ret = ice_vsi_add_vlan(vsi, vid); + if (!ret) { + vsi->vlan_ena = true; + set_bit(ICE_VSI_FLAG_VLAN_FLTR_CHANGED, vsi->flags); + } + + return ret; } /** - * ice_vlan_rx_kill_vid - Remove a vlan id filter from HW offload + * ice_vlan_rx_kill_vid - Remove a VLAN ID filter from HW offload * @netdev: network interface to be adjusted * @proto: unused protocol - * @vid: vlan id to be removed + * @vid: VLAN ID to be removed * - * net_device_ops implementation for removing vlan ids + * net_device_ops implementation for removing VLAN IDs */ -static int ice_vlan_rx_kill_vid(struct net_device *netdev, - __always_unused __be16 proto, u16 vid) +static int +ice_vlan_rx_kill_vid(struct net_device *netdev, __always_unused __be16 proto, + u16 vid) { struct ice_netdev_priv *np = netdev_priv(netdev); struct ice_vsi *vsi = np->vsi; - int status; + int ret; if (vsi->info.pvid) return -EINVAL; @@ -1707,15 +1879,17 @@ static int ice_vlan_rx_kill_vid(struct net_device *netdev, /* Make sure ice_vsi_kill_vlan is successful before updating VLAN * information */ - status = ice_vsi_kill_vlan(vsi, vid); - if (status) - return status; + ret = ice_vsi_kill_vlan(vsi, vid); + if (ret) + return ret; /* Disable VLAN pruning when VLAN 0 is removed */ if (unlikely(!vid)) - status = ice_cfg_vlan_pruning(vsi, false); + ret = ice_cfg_vlan_pruning(vsi, false, false); - return status; + vsi->vlan_ena = false; + set_bit(ICE_VSI_FLAG_VLAN_FLTR_CHANGED, vsi->flags); + return ret; } /** @@ -2032,23 +2206,6 @@ static int ice_init_interrupt_scheme(struct ice_pf *pf) return 0; } -/** - * ice_verify_itr_gran - verify driver's assumption of ITR granularity - * @pf: pointer to the PF structure - * - * There is no error returned here because the driver will be able to handle a - * different ITR granularity, but interrupt moderation will not be accurate if - * the driver's assumptions are not verified. This assumption is made so we can - * use constants in the hot path instead of accessing structure members. - */ -static void ice_verify_itr_gran(struct ice_pf *pf) -{ - if (pf->hw.itr_gran != (ICE_ITR_GRAN_S << 1)) - dev_warn(&pf->pdev->dev, - "%d ITR granularity assumption is invalid, actual ITR granularity is %d. Interrupt moderation will be inaccurate!\n", - (ICE_ITR_GRAN_S << 1), pf->hw.itr_gran); -} - /** * ice_verify_cacheline_size - verify driver's assumption of 64 Byte cache lines * @pf: pointer to the PF structure @@ -2072,9 +2229,10 @@ static void ice_verify_cacheline_size(struct ice_pf *pf) * * Returns 0 on success, negative on failure */ -static int ice_probe(struct pci_dev *pdev, - const struct pci_device_id __always_unused *ent) +static int +ice_probe(struct pci_dev *pdev, const struct pci_device_id __always_unused *ent) { + struct device *dev = &pdev->dev; struct ice_pf *pf; struct ice_hw *hw; int err; @@ -2086,20 +2244,20 @@ static int ice_probe(struct pci_dev *pdev, err = pcim_iomap_regions(pdev, BIT(ICE_BAR0), pci_name(pdev)); if (err) { - dev_err(&pdev->dev, "BAR0 I/O map error %d\n", err); + dev_err(dev, "BAR0 I/O map error %d\n", err); return err; } - pf = devm_kzalloc(&pdev->dev, sizeof(*pf), GFP_KERNEL); + pf = devm_kzalloc(dev, sizeof(*pf), GFP_KERNEL); if (!pf) return -ENOMEM; /* set up for high or low dma */ - err = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64)); + err = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64)); if (err) - err = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32)); + err = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32)); if (err) { - dev_err(&pdev->dev, "DMA configuration failed: 0x%x\n", err); + dev_err(dev, "DMA configuration failed: 0x%x\n", err); return err; } @@ -2133,17 +2291,26 @@ static int ice_probe(struct pci_dev *pdev, err = ice_init_hw(hw); if (err) { - dev_err(&pdev->dev, "ice_init_hw failed: %d\n", err); + dev_err(dev, "ice_init_hw failed: %d\n", err); err = -EIO; goto err_exit_unroll; } - dev_info(&pdev->dev, "firmware %d.%d.%05d api %d.%d\n", + dev_info(dev, "firmware %d.%d.%05d api %d.%d\n", hw->fw_maj_ver, hw->fw_min_ver, hw->fw_build, hw->api_maj_ver, hw->api_min_ver); ice_init_pf(pf); + err = ice_init_pf_dcb(pf); + if (err) { + clear_bit(ICE_FLAG_DCB_CAPABLE, pf->flags); + clear_bit(ICE_FLAG_DCB_ENA, pf->flags); + + /* do not fail overall init if DCB init fails */ + err = 0; + } + ice_determine_q_usage(pf); pf->num_alloc_vsi = hw->func_caps.guar_num_vsi; @@ -2152,8 +2319,8 @@ static int ice_probe(struct pci_dev *pdev, goto err_init_pf_unroll; } - pf->vsi = devm_kcalloc(&pdev->dev, pf->num_alloc_vsi, - sizeof(*pf->vsi), GFP_KERNEL); + pf->vsi = devm_kcalloc(dev, pf->num_alloc_vsi, sizeof(*pf->vsi), + GFP_KERNEL); if (!pf->vsi) { err = -ENOMEM; goto err_init_pf_unroll; @@ -2161,8 +2328,7 @@ static int ice_probe(struct pci_dev *pdev, err = ice_init_interrupt_scheme(pf); if (err) { - dev_err(&pdev->dev, - "ice_init_interrupt_scheme failed: %d\n", err); + dev_err(dev, "ice_init_interrupt_scheme failed: %d\n", err); err = -EIO; goto err_init_interrupt_unroll; } @@ -2178,15 +2344,13 @@ static int ice_probe(struct pci_dev *pdev, if (test_bit(ICE_FLAG_MSIX_ENA, pf->flags)) { err = ice_req_irq_msix_misc(pf); if (err) { - dev_err(&pdev->dev, - "setup of misc vector failed: %d\n", err); + dev_err(dev, "setup of misc vector failed: %d\n", err); goto err_init_interrupt_unroll; } } /* create switch struct for the switch element created by FW on boot */ - pf->first_sw = devm_kzalloc(&pdev->dev, sizeof(*pf->first_sw), - GFP_KERNEL); + pf->first_sw = devm_kzalloc(dev, sizeof(*pf->first_sw), GFP_KERNEL); if (!pf->first_sw) { err = -ENOMEM; goto err_msix_misc_unroll; @@ -2204,8 +2368,7 @@ static int ice_probe(struct pci_dev *pdev, err = ice_setup_pf_sw(pf); if (err) { - dev_err(&pdev->dev, - "probe failed due to setup pf switch:%d\n", err); + dev_err(dev, "probe failed due to setup pf switch:%d\n", err); goto err_alloc_sw_unroll; } @@ -2214,8 +2377,13 @@ static int ice_probe(struct pci_dev *pdev, /* since everything is good, start the service timer */ mod_timer(&pf->serv_tmr, round_jiffies(jiffies + pf->serv_tmr_period)); + err = ice_init_link_events(pf->hw.port_info); + if (err) { + dev_err(dev, "ice_init_link_events failed: %d\n", err); + goto err_alloc_sw_unroll; + } + ice_verify_cacheline_size(pf); - ice_verify_itr_gran(pf); return 0; @@ -2227,7 +2395,7 @@ err_msix_misc_unroll: ice_free_irq_msix_misc(pf); err_init_interrupt_unroll: ice_clear_interrupt_scheme(pf); - devm_kfree(&pdev->dev, pf->vsi); + devm_kfree(dev, pf->vsi); err_init_pf_unroll: ice_deinit_pf(pf); ice_deinit_hw(hw); @@ -2272,6 +2440,136 @@ static void ice_remove(struct pci_dev *pdev) pci_disable_pcie_error_reporting(pdev); } +/** + * ice_pci_err_detected - warning that PCI error has been detected + * @pdev: PCI device information struct + * @err: the type of PCI error + * + * Called to warn that something happened on the PCI bus and the error handling + * is in progress. Allows the driver to gracefully prepare/handle PCI errors. + */ +static pci_ers_result_t +ice_pci_err_detected(struct pci_dev *pdev, enum pci_channel_state err) +{ + struct ice_pf *pf = pci_get_drvdata(pdev); + + if (!pf) { + dev_err(&pdev->dev, "%s: unrecoverable device error %d\n", + __func__, err); + return PCI_ERS_RESULT_DISCONNECT; + } + + if (!test_bit(__ICE_SUSPENDED, pf->state)) { + ice_service_task_stop(pf); + + if (!test_bit(__ICE_PREPARED_FOR_RESET, pf->state)) { + set_bit(__ICE_PFR_REQ, pf->state); + ice_prepare_for_reset(pf); + } + } + + return PCI_ERS_RESULT_NEED_RESET; +} + +/** + * ice_pci_err_slot_reset - a PCI slot reset has just happened + * @pdev: PCI device information struct + * + * Called to determine if the driver can recover from the PCI slot reset by + * using a register read to determine if the device is recoverable. + */ +static pci_ers_result_t ice_pci_err_slot_reset(struct pci_dev *pdev) +{ + struct ice_pf *pf = pci_get_drvdata(pdev); + pci_ers_result_t result; + int err; + u32 reg; + + err = pci_enable_device_mem(pdev); + if (err) { + dev_err(&pdev->dev, + "Cannot re-enable PCI device after reset, error %d\n", + err); + result = PCI_ERS_RESULT_DISCONNECT; + } else { + pci_set_master(pdev); + pci_restore_state(pdev); + pci_save_state(pdev); + pci_wake_from_d3(pdev, false); + + /* Check for life */ + reg = rd32(&pf->hw, GLGEN_RTRIG); + if (!reg) + result = PCI_ERS_RESULT_RECOVERED; + else + result = PCI_ERS_RESULT_DISCONNECT; + } + + err = pci_cleanup_aer_uncorrect_error_status(pdev); + if (err) + dev_dbg(&pdev->dev, + "pci_cleanup_aer_uncorrect_error_status failed, error %d\n", + err); + /* non-fatal, continue */ + + return result; +} + +/** + * ice_pci_err_resume - restart operations after PCI error recovery + * @pdev: PCI device information struct + * + * Called to allow the driver to bring things back up after PCI error and/or + * reset recovery have finished + */ +static void ice_pci_err_resume(struct pci_dev *pdev) +{ + struct ice_pf *pf = pci_get_drvdata(pdev); + + if (!pf) { + dev_err(&pdev->dev, + "%s failed, device is unrecoverable\n", __func__); + return; + } + + if (test_bit(__ICE_SUSPENDED, pf->state)) { + dev_dbg(&pdev->dev, "%s failed to resume normal operations!\n", + __func__); + return; + } + + ice_do_reset(pf, ICE_RESET_PFR); + ice_service_task_restart(pf); + mod_timer(&pf->serv_tmr, round_jiffies(jiffies + pf->serv_tmr_period)); +} + +/** + * ice_pci_err_reset_prepare - prepare device driver for PCI reset + * @pdev: PCI device information struct + */ +static void ice_pci_err_reset_prepare(struct pci_dev *pdev) +{ + struct ice_pf *pf = pci_get_drvdata(pdev); + + if (!test_bit(__ICE_SUSPENDED, pf->state)) { + ice_service_task_stop(pf); + + if (!test_bit(__ICE_PREPARED_FOR_RESET, pf->state)) { + set_bit(__ICE_PFR_REQ, pf->state); + ice_prepare_for_reset(pf); + } + } +} + +/** + * ice_pci_err_reset_done - PCI reset done, device driver reset can begin + * @pdev: PCI device information struct + */ +static void ice_pci_err_reset_done(struct pci_dev *pdev) +{ + ice_pci_err_resume(pdev); +} + /* ice_pci_tbl - PCI Device ID Table * * Wildcard entries (PCI_ANY_ID) should come last @@ -2289,12 +2587,21 @@ static const struct pci_device_id ice_pci_tbl[] = { }; MODULE_DEVICE_TABLE(pci, ice_pci_tbl); +static const struct pci_error_handlers ice_pci_err_handler = { + .error_detected = ice_pci_err_detected, + .slot_reset = ice_pci_err_slot_reset, + .reset_prepare = ice_pci_err_reset_prepare, + .reset_done = ice_pci_err_reset_done, + .resume = ice_pci_err_resume +}; + static struct pci_driver ice_driver = { .name = KBUILD_MODNAME, .id_table = ice_pci_tbl, .probe = ice_probe, .remove = ice_remove, .sriov_configure = ice_sriov_configure, + .err_handler = &ice_pci_err_handler }; /** @@ -2341,7 +2648,7 @@ static void __exit ice_module_exit(void) module_exit(ice_module_exit); /** - * ice_set_mac_address - NDO callback to set mac address + * ice_set_mac_address - NDO callback to set MAC address * @netdev: network interface device structure * @pi: pointer to an address structure * @@ -2378,14 +2685,14 @@ static int ice_set_mac_address(struct net_device *netdev, void *pi) return -EBUSY; } - /* When we change the mac address we also have to change the mac address - * based filter rules that were created previously for the old mac + /* When we change the MAC address we also have to change the MAC address + * based filter rules that were created previously for the old MAC * address. So first, we remove the old filter rule using ice_remove_mac * and then create a new filter rule using ice_add_mac. Note that for - * both these operations, we first need to form a "list" of mac - * addresses (even though in this case, we have only 1 mac address to be + * both these operations, we first need to form a "list" of MAC + * addresses (even though in this case, we have only 1 MAC address to be * added/removed) and this done using ice_add_mac_to_list. Depending on - * the ensuing operation this "list" of mac addresses is either to be + * the ensuing operation this "list" of MAC addresses is either to be * added or removed from the filter. */ err = ice_add_mac_to_list(vsi, &r_mac_list, netdev->dev_addr); @@ -2423,12 +2730,12 @@ free_lists: return err; } - /* change the netdev's mac address */ + /* change the netdev's MAC address */ memcpy(netdev->dev_addr, mac, netdev->addr_len); netdev_dbg(vsi->netdev, "updated mac address to %pM\n", netdev->dev_addr); - /* write new mac address to the firmware */ + /* write new MAC address to the firmware */ flags = ICE_AQC_MAN_MAC_UPDATE_LAA_WOL; status = ice_aq_manage_mac_write(hw, mac, flags, NULL); if (status) { @@ -2470,7 +2777,7 @@ static void ice_set_rx_mode(struct net_device *netdev) * @tb: pointer to array of nladdr (unused) * @dev: the net device pointer * @addr: the MAC address entry being added - * @vid: VLAN id + * @vid: VLAN ID * @flags: instructions from stack about fdb operation * @extack: netlink extended ack */ @@ -2510,11 +2817,12 @@ ice_fdb_add(struct ndmsg *ndm, struct nlattr __always_unused *tb[], * @tb: pointer to array of nladdr (unused) * @dev: the net device pointer * @addr: the MAC address entry being added - * @vid: VLAN id + * @vid: VLAN ID */ -static int ice_fdb_del(struct ndmsg *ndm, __always_unused struct nlattr *tb[], - struct net_device *dev, const unsigned char *addr, - __always_unused u16 vid) +static int +ice_fdb_del(struct ndmsg *ndm, __always_unused struct nlattr *tb[], + struct net_device *dev, const unsigned char *addr, + __always_unused u16 vid) { int err; @@ -2538,13 +2846,16 @@ static int ice_fdb_del(struct ndmsg *ndm, __always_unused struct nlattr *tb[], * @netdev: ptr to the netdev being adjusted * @features: the feature set that the stack is suggesting */ -static int ice_set_features(struct net_device *netdev, - netdev_features_t features) +static int +ice_set_features(struct net_device *netdev, netdev_features_t features) { struct ice_netdev_priv *np = netdev_priv(netdev); struct ice_vsi *vsi = np->vsi; int ret = 0; + /* Multiple features can be changed in one call so keep features in + * separate if/else statements to guarantee each feature is checked + */ if (features & NETIF_F_RXHASH && !(netdev->features & NETIF_F_RXHASH)) ret = ice_vsi_manage_rss_lut(vsi, true); else if (!(features & NETIF_F_RXHASH) && @@ -2557,8 +2868,9 @@ static int ice_set_features(struct net_device *netdev, else if (!(features & NETIF_F_HW_VLAN_CTAG_RX) && (netdev->features & NETIF_F_HW_VLAN_CTAG_RX)) ret = ice_vsi_manage_vlan_stripping(vsi, false); - else if ((features & NETIF_F_HW_VLAN_CTAG_TX) && - !(netdev->features & NETIF_F_HW_VLAN_CTAG_TX)) + + if ((features & NETIF_F_HW_VLAN_CTAG_TX) && + !(netdev->features & NETIF_F_HW_VLAN_CTAG_TX)) ret = ice_vsi_manage_vlan_insertion(vsi); else if (!(features & NETIF_F_HW_VLAN_CTAG_TX) && (netdev->features & NETIF_F_HW_VLAN_CTAG_TX)) @@ -2568,8 +2880,8 @@ static int ice_set_features(struct net_device *netdev, } /** - * ice_vsi_vlan_setup - Setup vlan offload properties on a VSI - * @vsi: VSI to setup vlan properties for + * ice_vsi_vlan_setup - Setup VLAN offload properties on a VSI + * @vsi: VSI to setup VLAN properties for */ static int ice_vsi_vlan_setup(struct ice_vsi *vsi) { @@ -2601,6 +2913,7 @@ static int ice_vsi_cfg(struct ice_vsi *vsi) if (err) return err; } + ice_vsi_cfg_dcb_rings(vsi); err = ice_vsi_cfg_lan_txqs(vsi); if (!err) @@ -2620,7 +2933,7 @@ static void ice_napi_enable_all(struct ice_vsi *vsi) if (!vsi->netdev) return; - for (q_idx = 0; q_idx < vsi->num_q_vectors; q_idx++) { + ice_for_each_q_vector(vsi, q_idx) { struct ice_q_vector *q_vector = vsi->q_vectors[q_idx]; if (q_vector->rx.ring || q_vector->tx.ring) @@ -2666,7 +2979,7 @@ static int ice_up_complete(struct ice_vsi *vsi) ice_service_task_schedule(pf); - return err; + return 0; } /** @@ -2693,8 +3006,8 @@ int ice_up(struct ice_vsi *vsi) * This function fetches stats from the ring considering the atomic operations * that needs to be performed to read u64 values in 32 bit machine. */ -static void ice_fetch_u64_stats_per_ring(struct ice_ring *ring, u64 *pkts, - u64 *bytes) +static void +ice_fetch_u64_stats_per_ring(struct ice_ring *ring, u64 *pkts, u64 *bytes) { unsigned int start; *pkts = 0; @@ -2911,6 +3224,8 @@ static void ice_update_pf_stats(struct ice_pf *pf) ice_stat_update32(hw, GLPRT_LXOFFTXC(pf_id), pf->stat_prev_loaded, &prev_ps->link_xoff_tx, &cur_ps->link_xoff_tx); + ice_update_dcb_stats(pf); + ice_stat_update32(hw, GLPRT_CRCERRS(pf_id), pf->stat_prev_loaded, &prev_ps->crc_errors, &cur_ps->crc_errors); @@ -2992,7 +3307,7 @@ static void ice_napi_disable_all(struct ice_vsi *vsi) if (!vsi->netdev) return; - for (q_idx = 0; q_idx < vsi->num_q_vectors; q_idx++) { + ice_for_each_q_vector(vsi, q_idx) { struct ice_q_vector *q_vector = vsi->q_vectors[q_idx]; if (q_vector->rx.ring || q_vector->tx.ring) @@ -3276,7 +3591,7 @@ static void ice_vsi_release_all(struct ice_pf *pf) if (!pf->vsi) return; - for (i = 0; i < pf->num_alloc_vsi; i++) { + ice_for_each_vsi(pf, i) { if (!pf->vsi[i]) continue; @@ -3289,46 +3604,30 @@ static void ice_vsi_release_all(struct ice_pf *pf) } /** - * ice_dis_vsi - pause a VSI - * @vsi: the VSI being paused + * ice_ena_vsi - resume a VSI + * @vsi: the VSI being resume * @locked: is the rtnl_lock already held */ -static void ice_dis_vsi(struct ice_vsi *vsi, bool locked) +static int ice_ena_vsi(struct ice_vsi *vsi, bool locked) { - if (test_bit(__ICE_DOWN, vsi->state)) - return; + int err = 0; - set_bit(__ICE_NEEDS_RESTART, vsi->state); + if (!test_bit(__ICE_NEEDS_RESTART, vsi->state)) + return err; + + clear_bit(__ICE_NEEDS_RESTART, vsi->state); + + if (vsi->netdev && vsi->type == ICE_VSI_PF) { + struct net_device *netd = vsi->netdev; - if (vsi->type == ICE_VSI_PF && vsi->netdev) { if (netif_running(vsi->netdev)) { - if (!locked) { + if (locked) { + err = netd->netdev_ops->ndo_open(netd); + } else { rtnl_lock(); - vsi->netdev->netdev_ops->ndo_stop(vsi->netdev); + err = netd->netdev_ops->ndo_open(netd); rtnl_unlock(); - } else { - vsi->netdev->netdev_ops->ndo_stop(vsi->netdev); } - } else { - ice_vsi_close(vsi); - } - } -} - -/** - * ice_ena_vsi - resume a VSI - * @vsi: the VSI being resume - */ -static int ice_ena_vsi(struct ice_vsi *vsi) -{ - int err = 0; - - if (test_and_clear_bit(__ICE_NEEDS_RESTART, vsi->state) && - vsi->netdev) { - if (netif_running(vsi->netdev)) { - rtnl_lock(); - err = vsi->netdev->netdev_ops->ndo_open(vsi->netdev); - rtnl_unlock(); } else { err = ice_vsi_open(vsi); } @@ -3337,30 +3636,22 @@ static int ice_ena_vsi(struct ice_vsi *vsi) return err; } -/** - * ice_pf_dis_all_vsi - Pause all VSIs on a PF - * @pf: the PF - */ -static void ice_pf_dis_all_vsi(struct ice_pf *pf) -{ - int v; - - ice_for_each_vsi(pf, v) - if (pf->vsi[v]) - ice_dis_vsi(pf->vsi[v], false); -} - /** * ice_pf_ena_all_vsi - Resume all VSIs on a PF * @pf: the PF + * @locked: is the rtnl_lock already held */ -static int ice_pf_ena_all_vsi(struct ice_pf *pf) +#ifdef CONFIG_DCB +int ice_pf_ena_all_vsi(struct ice_pf *pf, bool locked) +#else +static int ice_pf_ena_all_vsi(struct ice_pf *pf, bool locked) +#endif /* CONFIG_DCB */ { int v; ice_for_each_vsi(pf, v) if (pf->vsi[v]) - if (ice_ena_vsi(pf->vsi[v])) + if (ice_ena_vsi(pf->vsi[v], locked)) return -EIO; return 0; @@ -3375,16 +3666,12 @@ static int ice_vsi_rebuild_all(struct ice_pf *pf) int i; /* loop through pf->vsi array and reinit the VSI if found */ - for (i = 0; i < pf->num_alloc_vsi; i++) { + ice_for_each_vsi(pf, i) { int err; if (!pf->vsi[i]) continue; - /* VF VSI rebuild isn't supported yet */ - if (pf->vsi[i]->type == ICE_VSI_VF) - continue; - err = ice_vsi_rebuild(pf->vsi[i]); if (err) { dev_err(&pf->pdev->dev, @@ -3412,7 +3699,7 @@ static int ice_vsi_replay_all(struct ice_pf *pf) int i; /* loop through pf->vsi array and replay the VSI if found */ - for (i = 0; i < pf->num_alloc_vsi; i++) { + ice_for_each_vsi(pf, i) { if (!pf->vsi[i]) continue; @@ -3479,6 +3766,8 @@ static void ice_rebuild(struct ice_pf *pf) if (err) goto err_sched_init_port; + ice_dcb_rebuild(pf); + /* reset search_hint of irq_trackers to 0 since interrupts are * reclaimed and could be allocated from beginning during VSI rebuild */ @@ -3512,7 +3801,7 @@ static void ice_rebuild(struct ice_pf *pf) } /* restart the VSIs that were rebuilt and running before the reset */ - err = ice_pf_ena_all_vsi(pf); + err = ice_pf_ena_all_vsi(pf, false); if (err) { dev_err(&pf->pdev->dev, "error enabling VSIs\n"); /* no need to disable VSIs in tear down path in ice_rebuild() @@ -3521,9 +3810,7 @@ static void ice_rebuild(struct ice_pf *pf) goto err_vsi_rebuild; } - ice_reset_all_vfs(pf, true); - - for (i = 0; i < pf->num_alloc_vsi; i++) { + ice_for_each_vsi(pf, i) { bool link_up; if (!pf->vsi[i] || pf->vsi[i]->type != ICE_VSI_PF) @@ -3710,7 +3997,7 @@ int ice_get_rss(struct ice_vsi *vsi, u8 *seed, u8 *lut, u16 lut_size) /** * ice_bridge_getlink - Get the hardware bridge mode * @skb: skb buff - * @pid: process id + * @pid: process ID * @seq: RTNL message seq * @dev: the netdev being configured * @filter_mask: filter mask passed in @@ -3909,8 +4196,7 @@ static void ice_tx_timeout(struct net_device *netdev) /* Read interrupt register */ if (test_bit(ICE_FLAG_MSIX_ENA, pf->flags)) val = rd32(hw, - GLINT_DYN_CTL(tx_ring->q_vector->v_idx + - tx_ring->vsi->hw_base_vector)); + GLINT_DYN_CTL(tx_ring->q_vector->reg_idx)); netdev_info(netdev, "tx_timeout: VSI_num: %d, Q %d, NTC: 0x%x, HW_HEAD: 0x%x, NTU: 0x%x, INT: 0x%x\n", vsi->vsi_num, hung_queue, tx_ring->next_to_clean, diff --git a/drivers/net/ethernet/intel/ice/ice_nvm.c b/drivers/net/ethernet/intel/ice/ice_nvm.c index 413fdbbcc4d0ce79266bf1883110e9cb1b69956d..62571d33d0d6882203a704eb2ecd697f4ba11acd 100644 --- a/drivers/net/ethernet/intel/ice/ice_nvm.c +++ b/drivers/net/ethernet/intel/ice/ice_nvm.c @@ -5,7 +5,7 @@ /** * ice_aq_read_nvm - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @module_typeid: module pointer location in words from the NVM beginning * @offset: byte offset from the module beginning * @length: length of the section to be read (in bytes from the offset) @@ -235,7 +235,7 @@ ice_read_sr_word(struct ice_hw *hw, u16 offset, u16 *data) /** * ice_init_nvm - initializes NVM setting - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * * This function reads and populates NVM settings such as Shadow RAM size, * max_timeout, and blank_nvm_mode @@ -248,7 +248,7 @@ enum ice_status ice_init_nvm(struct ice_hw *hw) u32 fla, gens_stat; u8 sr_size; - /* The SR size is stored regardless of the nvm programming mode + /* The SR size is stored regardless of the NVM programming mode * as the blank mode may be used in the factory line. */ gens_stat = rd32(hw, GLNVM_GENS); diff --git a/drivers/net/ethernet/intel/ice/ice_sched.c b/drivers/net/ethernet/intel/ice/ice_sched.c index 56049739a250455d7ec1fbb9b5102593c5d88d13..8d49f83be7a5934cf94018bdecb4545238db5f8c 100644 --- a/drivers/net/ethernet/intel/ice/ice_sched.c +++ b/drivers/net/ethernet/intel/ice/ice_sched.c @@ -43,9 +43,9 @@ ice_sched_add_root_node(struct ice_port_info *pi, /** * ice_sched_find_node_by_teid - Find the Tx scheduler node in SW DB * @start_node: pointer to the starting ice_sched_node struct in a sub-tree - * @teid: node teid to search + * @teid: node TEID to search * - * This function searches for a node matching the teid in the scheduling tree + * This function searches for a node matching the TEID in the scheduling tree * from the SW DB. The search is recursive and is restricted by the number of * layers it has searched through; stopping at the max supported layer. * @@ -66,7 +66,7 @@ ice_sched_find_node_by_teid(struct ice_sched_node *start_node, u32 teid) start_node->info.data.elem_type == ICE_AQC_ELEM_TYPE_LEAF) return NULL; - /* Check if teid matches to any of the children nodes */ + /* Check if TEID matches to any of the children nodes */ for (i = 0; i < start_node->num_children; i++) if (ICE_TXSCHED_GET_NODE_TEID(start_node->children[i]) == teid) return start_node->children[i]; @@ -86,7 +86,7 @@ ice_sched_find_node_by_teid(struct ice_sched_node *start_node, u32 teid) /** * ice_aqc_send_sched_elem_cmd - send scheduling elements cmd - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @cmd_opc: cmd opcode * @elems_req: number of elements to request * @buf: pointer to buffer @@ -118,7 +118,7 @@ ice_aqc_send_sched_elem_cmd(struct ice_hw *hw, enum ice_adminq_opc cmd_opc, /** * ice_aq_query_sched_elems - query scheduler elements - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @elems_req: number of elements to query * @buf: pointer to buffer * @buf_size: buffer size in bytes @@ -127,7 +127,7 @@ ice_aqc_send_sched_elem_cmd(struct ice_hw *hw, enum ice_adminq_opc cmd_opc, * * Query scheduling elements (0x0404) */ -static enum ice_status +enum ice_status ice_aq_query_sched_elems(struct ice_hw *hw, u16 elems_req, struct ice_aqc_get_elem *buf, u16 buf_size, u16 *elems_ret, struct ice_sq_cd *cd) @@ -137,31 +137,6 @@ ice_aq_query_sched_elems(struct ice_hw *hw, u16 elems_req, elems_ret, cd); } -/** - * ice_sched_query_elem - query element information from hw - * @hw: pointer to the hw struct - * @node_teid: node teid to be queried - * @buf: buffer to element information - * - * This function queries HW element information - */ -static enum ice_status -ice_sched_query_elem(struct ice_hw *hw, u32 node_teid, - struct ice_aqc_get_elem *buf) -{ - u16 buf_size, num_elem_ret = 0; - enum ice_status status; - - buf_size = sizeof(*buf); - memset(buf, 0, buf_size); - buf->generic[0].node_teid = cpu_to_le32(node_teid); - status = ice_aq_query_sched_elems(hw, 1, buf, buf_size, &num_elem_ret, - NULL); - if (status || num_elem_ret != 1) - ice_debug(hw, ICE_DBG_SCHED, "query element failed\n"); - return status; -} - /** * ice_sched_add_node - Insert the Tx scheduler node in SW DB * @pi: port information structure @@ -226,7 +201,7 @@ ice_sched_add_node(struct ice_port_info *pi, u8 layer, /** * ice_aq_delete_sched_elems - delete scheduler elements - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @grps_req: number of groups to delete * @buf: pointer to buffer * @buf_size: buffer size in bytes @@ -246,13 +221,13 @@ ice_aq_delete_sched_elems(struct ice_hw *hw, u16 grps_req, } /** - * ice_sched_remove_elems - remove nodes from hw - * @hw: pointer to the hw struct + * ice_sched_remove_elems - remove nodes from HW + * @hw: pointer to the HW struct * @parent: pointer to the parent node * @num_nodes: number of nodes * @node_teids: array of node teids to be deleted * - * This function remove nodes from hw + * This function remove nodes from HW */ static enum ice_status ice_sched_remove_elems(struct ice_hw *hw, struct ice_sched_node *parent, @@ -276,7 +251,8 @@ ice_sched_remove_elems(struct ice_hw *hw, struct ice_sched_node *parent, status = ice_aq_delete_sched_elems(hw, 1, buf, buf_size, &num_groups_removed, NULL); if (status || num_groups_removed != 1) - ice_debug(hw, ICE_DBG_SCHED, "remove elements failed\n"); + ice_debug(hw, ICE_DBG_SCHED, "remove node failed FW error %d\n", + hw->adminq.sq_last_status); devm_kfree(ice_hw_to_dev(hw), buf); return status; @@ -284,7 +260,7 @@ ice_sched_remove_elems(struct ice_hw *hw, struct ice_sched_node *parent, /** * ice_sched_get_first_node - get the first node of the given layer - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @parent: pointer the base node of the subtree * @layer: layer number * @@ -360,12 +336,8 @@ void ice_free_sched_node(struct ice_port_info *pi, struct ice_sched_node *node) node->info.data.elem_type != ICE_AQC_ELEM_TYPE_ROOT_PORT && node->info.data.elem_type != ICE_AQC_ELEM_TYPE_LEAF) { u32 teid = le32_to_cpu(node->info.node_teid); - enum ice_status status; - status = ice_sched_remove_elems(hw, node->parent, 1, &teid); - if (status) - ice_debug(hw, ICE_DBG_SCHED, - "remove element failed %d\n", status); + ice_sched_remove_elems(hw, node->parent, 1, &teid); } parent = node->parent; /* root has no parent */ @@ -409,7 +381,7 @@ err_exit: /** * ice_aq_get_dflt_topo - gets default scheduler topology - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @lport: logical port number * @buf: pointer to buffer * @buf_size: buffer size in bytes @@ -439,7 +411,7 @@ ice_aq_get_dflt_topo(struct ice_hw *hw, u8 lport, /** * ice_aq_add_sched_elems - adds scheduling element - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @grps_req: the number of groups that are requested to be added * @buf: pointer to buffer * @buf_size: buffer size in bytes @@ -460,7 +432,7 @@ ice_aq_add_sched_elems(struct ice_hw *hw, u16 grps_req, /** * ice_aq_suspend_sched_elems - suspend scheduler elements - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @elems_req: number of elements to suspend * @buf: pointer to buffer * @buf_size: buffer size in bytes @@ -481,7 +453,7 @@ ice_aq_suspend_sched_elems(struct ice_hw *hw, u16 elems_req, /** * ice_aq_resume_sched_elems - resume scheduler elements - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @elems_req: number of elements to resume * @buf: pointer to buffer * @buf_size: buffer size in bytes @@ -502,7 +474,7 @@ ice_aq_resume_sched_elems(struct ice_hw *hw, u16 elems_req, /** * ice_aq_query_sched_res - query scheduler resource - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @buf_size: buffer size in bytes * @buf: pointer to buffer * @cd: pointer to command details structure or NULL @@ -521,13 +493,13 @@ ice_aq_query_sched_res(struct ice_hw *hw, u16 buf_size, } /** - * ice_sched_suspend_resume_elems - suspend or resume hw nodes - * @hw: pointer to the hw struct + * ice_sched_suspend_resume_elems - suspend or resume HW nodes + * @hw: pointer to the HW struct * @num_nodes: number of nodes * @node_teids: array of node teids to be suspended or resumed * @suspend: true means suspend / false means resume * - * This function suspends or resumes hw nodes + * This function suspends or resumes HW nodes */ static enum ice_status ice_sched_suspend_resume_elems(struct ice_hw *hw, u8 num_nodes, u32 *node_teids, @@ -561,10 +533,54 @@ ice_sched_suspend_resume_elems(struct ice_hw *hw, u8 num_nodes, u32 *node_teids, } /** - * ice_sched_clear_agg - clears the agg related information + * ice_alloc_lan_q_ctx - allocate LAN queue contexts for the given VSI and TC + * @hw: pointer to the HW struct + * @vsi_handle: VSI handle + * @tc: TC number + * @new_numqs: number of queues + */ +static enum ice_status +ice_alloc_lan_q_ctx(struct ice_hw *hw, u16 vsi_handle, u8 tc, u16 new_numqs) +{ + struct ice_vsi_ctx *vsi_ctx; + struct ice_q_ctx *q_ctx; + + vsi_ctx = ice_get_vsi_ctx(hw, vsi_handle); + if (!vsi_ctx) + return ICE_ERR_PARAM; + /* allocate LAN queue contexts */ + if (!vsi_ctx->lan_q_ctx[tc]) { + vsi_ctx->lan_q_ctx[tc] = devm_kcalloc(ice_hw_to_dev(hw), + new_numqs, + sizeof(*q_ctx), + GFP_KERNEL); + if (!vsi_ctx->lan_q_ctx[tc]) + return ICE_ERR_NO_MEMORY; + vsi_ctx->num_lan_q_entries[tc] = new_numqs; + return 0; + } + /* num queues are increased, update the queue contexts */ + if (new_numqs > vsi_ctx->num_lan_q_entries[tc]) { + u16 prev_num = vsi_ctx->num_lan_q_entries[tc]; + + q_ctx = devm_kcalloc(ice_hw_to_dev(hw), new_numqs, + sizeof(*q_ctx), GFP_KERNEL); + if (!q_ctx) + return ICE_ERR_NO_MEMORY; + memcpy(q_ctx, vsi_ctx->lan_q_ctx[tc], + prev_num * sizeof(*q_ctx)); + devm_kfree(ice_hw_to_dev(hw), vsi_ctx->lan_q_ctx[tc]); + vsi_ctx->lan_q_ctx[tc] = q_ctx; + vsi_ctx->num_lan_q_entries[tc] = new_numqs; + } + return 0; +} + +/** + * ice_sched_clear_agg - clears the aggregator related information * @hw: pointer to the hardware structure * - * This function removes agg list and free up agg related memory + * This function removes aggregator list and free up aggregator related memory * previously allocated. */ void ice_sched_clear_agg(struct ice_hw *hw) @@ -622,7 +638,7 @@ void ice_sched_clear_port(struct ice_port_info *pi) /** * ice_sched_cleanup_all - cleanup scheduler elements from SW DB for all ports - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * * Cleanup scheduling elements from SW DB for all the ports */ @@ -646,16 +662,16 @@ void ice_sched_cleanup_all(struct ice_hw *hw) } /** - * ice_sched_add_elems - add nodes to hw and SW DB + * ice_sched_add_elems - add nodes to HW and SW DB * @pi: port information structure * @tc_node: pointer to the branch node * @parent: pointer to the parent node * @layer: layer number to add nodes * @num_nodes: number of nodes * @num_nodes_added: pointer to num nodes added - * @first_node_teid: if new nodes are added then return the teid of first node + * @first_node_teid: if new nodes are added then return the TEID of first node * - * This function add nodes to hw as well as to SW DB for a given layer + * This function add nodes to HW as well as to SW DB for a given layer */ static enum ice_status ice_sched_add_elems(struct ice_port_info *pi, struct ice_sched_node *tc_node, @@ -697,7 +713,8 @@ ice_sched_add_elems(struct ice_port_info *pi, struct ice_sched_node *tc_node, status = ice_aq_add_sched_elems(hw, 1, buf, buf_size, &num_groups_added, NULL); if (status || num_groups_added != 1) { - ice_debug(hw, ICE_DBG_SCHED, "add elements failed\n"); + ice_debug(hw, ICE_DBG_SCHED, "add node failed FW Error %d\n", + hw->adminq.sq_last_status); devm_kfree(ice_hw_to_dev(hw), buf); return ICE_ERR_CFG; } @@ -748,7 +765,7 @@ ice_sched_add_elems(struct ice_port_info *pi, struct ice_sched_node *tc_node, * @parent: pointer to parent node * @layer: layer number to add nodes * @num_nodes: number of nodes to be added - * @first_node_teid: pointer to the first node teid + * @first_node_teid: pointer to the first node TEID * @num_nodes_added: pointer to number of nodes added * * This function add nodes to a given layer. @@ -800,7 +817,7 @@ ice_sched_add_nodes_to_layer(struct ice_port_info *pi, *num_nodes_added += num_added; } - /* Don't modify the first node teid memory if the first node was + /* Don't modify the first node TEID memory if the first node was * added already in the above call. Instead send some temp * memory for all other recursive calls. */ @@ -832,7 +849,7 @@ ice_sched_add_nodes_to_layer(struct ice_port_info *pi, /** * ice_sched_get_qgrp_layer - get the current queue group layer number - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * * This function returns the current queue group layer number */ @@ -844,7 +861,7 @@ static u8 ice_sched_get_qgrp_layer(struct ice_hw *hw) /** * ice_sched_get_vsi_layer - get the current VSI layer number - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * * This function returns the current VSI layer number */ @@ -855,7 +872,7 @@ static u8 ice_sched_get_vsi_layer(struct ice_hw *hw) * 7 4 * 5 or less sw_entry_point_layer */ - /* calculate the vsi layer based on number of layers. */ + /* calculate the VSI layer based on number of layers. */ if (hw->num_tx_sched_layers > ICE_VSI_LAYER_OFFSET + 1) { u8 layer = hw->num_tx_sched_layers - ICE_VSI_LAYER_OFFSET; @@ -973,7 +990,7 @@ enum ice_status ice_sched_init_port(struct ice_port_info *pi) goto err_init_port; } - /* If the last node is a leaf node then the index of the Q group + /* If the last node is a leaf node then the index of the queue group * layer is two less than the number of elements. */ if (num_elems > 2 && buf[0].generic[num_elems - 1].data.elem_type == @@ -1082,7 +1099,7 @@ sched_query_out: /** * ice_sched_find_node_in_subtree - Find node in part of base node subtree - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @base: pointer to the base node * @node: pointer to the node to search * @@ -1114,13 +1131,13 @@ ice_sched_find_node_in_subtree(struct ice_hw *hw, struct ice_sched_node *base, } /** - * ice_sched_get_free_qparent - Get a free lan or rdma q group node + * ice_sched_get_free_qparent - Get a free LAN or RDMA queue group node * @pi: port information structure * @vsi_handle: software VSI handle * @tc: branch number - * @owner: lan or rdma + * @owner: LAN or RDMA * - * This function retrieves a free lan or rdma q group node + * This function retrieves a free LAN or RDMA queue group node */ struct ice_sched_node * ice_sched_get_free_qparent(struct ice_port_info *pi, u16 vsi_handle, u8 tc, @@ -1138,11 +1155,11 @@ ice_sched_get_free_qparent(struct ice_port_info *pi, u16 vsi_handle, u8 tc, if (!vsi_ctx) return NULL; vsi_node = vsi_ctx->sched.vsi_node[tc]; - /* validate invalid VSI id */ + /* validate invalid VSI ID */ if (!vsi_node) goto lan_q_exit; - /* get the first q group node from VSI sub-tree */ + /* get the first queue group node from VSI sub-tree */ qgrp_node = ice_sched_get_first_node(pi->hw, vsi_node, qgrp_layer); while (qgrp_node) { /* make sure the qgroup node is part of the VSI subtree */ @@ -1158,12 +1175,12 @@ lan_q_exit: } /** - * ice_sched_get_vsi_node - Get a VSI node based on VSI id - * @hw: pointer to the hw struct + * ice_sched_get_vsi_node - Get a VSI node based on VSI ID + * @hw: pointer to the HW struct * @tc_node: pointer to the TC node * @vsi_handle: software VSI handle * - * This function retrieves a VSI node for a given VSI id from a given + * This function retrieves a VSI node for a given VSI ID from a given * TC branch */ static struct ice_sched_node * @@ -1188,7 +1205,7 @@ ice_sched_get_vsi_node(struct ice_hw *hw, struct ice_sched_node *tc_node, /** * ice_sched_calc_vsi_child_nodes - calculate number of VSI child nodes - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @num_qs: number of queues * @num_nodes: num nodes array * @@ -1204,7 +1221,7 @@ ice_sched_calc_vsi_child_nodes(struct ice_hw *hw, u16 num_qs, u16 *num_nodes) qgl = ice_sched_get_qgrp_layer(hw); vsil = ice_sched_get_vsi_layer(hw); - /* calculate num nodes from q group to VSI layer */ + /* calculate num nodes from queue group to VSI layer */ for (i = qgl; i > vsil; i--) { /* round to the next integer if there is a remainder */ num = DIV_ROUND_UP(num, hw->max_children[i]); @@ -1220,10 +1237,10 @@ ice_sched_calc_vsi_child_nodes(struct ice_hw *hw, u16 num_qs, u16 *num_nodes) * @vsi_handle: software VSI handle * @tc_node: pointer to the TC node * @num_nodes: pointer to the num nodes that needs to be added per layer - * @owner: node owner (lan or rdma) + * @owner: node owner (LAN or RDMA) * * This function adds the VSI child nodes to tree. It gets called for - * lan and rdma separately. + * LAN and RDMA separately. */ static enum ice_status ice_sched_add_vsi_child_nodes(struct ice_port_info *pi, u16 vsi_handle, @@ -1270,45 +1287,9 @@ ice_sched_add_vsi_child_nodes(struct ice_port_info *pi, u16 vsi_handle, return 0; } -/** - * ice_sched_rm_vsi_child_nodes - remove VSI child nodes from the tree - * @pi: port information structure - * @vsi_node: pointer to the VSI node - * @num_nodes: pointer to the num nodes that needs to be removed per layer - * @owner: node owner (lan or rdma) - * - * This function removes the VSI child nodes from the tree. It gets called for - * lan and rdma separately. - */ -static void -ice_sched_rm_vsi_child_nodes(struct ice_port_info *pi, - struct ice_sched_node *vsi_node, u16 *num_nodes, - u8 owner) -{ - struct ice_sched_node *node, *next; - u8 i, qgl, vsil; - u16 num; - - qgl = ice_sched_get_qgrp_layer(pi->hw); - vsil = ice_sched_get_vsi_layer(pi->hw); - - for (i = qgl; i > vsil; i--) { - num = num_nodes[i]; - node = ice_sched_get_first_node(pi->hw, vsi_node, i); - while (node && num) { - next = node->sibling; - if (node->owner == owner && !node->num_children) { - ice_free_sched_node(pi, node); - num--; - } - node = next; - } - } -} - /** * ice_sched_calc_vsi_support_nodes - calculate number of VSI support nodes - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @tc_node: pointer to TC node * @num_nodes: pointer to num nodes array * @@ -1427,7 +1408,7 @@ ice_sched_add_vsi_to_topo(struct ice_port_info *pi, u16 vsi_handle, u8 tc) /* calculate number of supported nodes needed for this VSI */ ice_sched_calc_vsi_support_nodes(hw, tc_node, num_nodes); - /* add vsi supported nodes to tc subtree */ + /* add VSI supported nodes to TC subtree */ return ice_sched_add_vsi_support_nodes(pi, vsi_handle, tc_node, num_nodes); } @@ -1446,7 +1427,6 @@ static enum ice_status ice_sched_update_vsi_child_nodes(struct ice_port_info *pi, u16 vsi_handle, u8 tc, u16 new_numqs, u8 owner) { - u16 prev_num_nodes[ICE_AQC_TOPO_MAX_LEVEL_NUM] = { 0 }; u16 new_num_nodes[ICE_AQC_TOPO_MAX_LEVEL_NUM] = { 0 }; struct ice_sched_node *vsi_node; struct ice_sched_node *tc_node; @@ -1454,7 +1434,6 @@ ice_sched_update_vsi_child_nodes(struct ice_port_info *pi, u16 vsi_handle, enum ice_status status = 0; struct ice_hw *hw = pi->hw; u16 prev_numqs; - u8 i; tc_node = ice_sched_get_tc_node(pi, tc); if (!tc_node) @@ -1468,41 +1447,30 @@ ice_sched_update_vsi_child_nodes(struct ice_port_info *pi, u16 vsi_handle, if (!vsi_ctx) return ICE_ERR_PARAM; - if (owner == ICE_SCHED_NODE_OWNER_LAN) - prev_numqs = vsi_ctx->sched.max_lanq[tc]; - else - return ICE_ERR_PARAM; - - /* num queues are not changed */ - if (prev_numqs == new_numqs) + prev_numqs = vsi_ctx->sched.max_lanq[tc]; + /* num queues are not changed or less than the previous number */ + if (new_numqs <= prev_numqs) + return status; + status = ice_alloc_lan_q_ctx(hw, vsi_handle, tc, new_numqs); + if (status) return status; - - /* calculate number of nodes based on prev/new number of qs */ - if (prev_numqs) - ice_sched_calc_vsi_child_nodes(hw, prev_numqs, prev_num_nodes); if (new_numqs) ice_sched_calc_vsi_child_nodes(hw, new_numqs, new_num_nodes); - - if (prev_numqs > new_numqs) { - for (i = 0; i < ICE_AQC_TOPO_MAX_LEVEL_NUM; i++) - new_num_nodes[i] = prev_num_nodes[i] - new_num_nodes[i]; - - ice_sched_rm_vsi_child_nodes(pi, vsi_node, new_num_nodes, - owner); - } else { - for (i = 0; i < ICE_AQC_TOPO_MAX_LEVEL_NUM; i++) - new_num_nodes[i] -= prev_num_nodes[i]; - - status = ice_sched_add_vsi_child_nodes(pi, vsi_handle, tc_node, - new_num_nodes, owner); - if (status) - return status; - } - + /* Keep the max number of queue configuration all the time. Update the + * tree only if number of queues > previous number of queues. This may + * leave some extra nodes in the tree if number of queues < previous + * number but that wouldn't harm anything. Removing those extra nodes + * may complicate the code if those nodes are part of SRL or + * individually rate limited. + */ + status = ice_sched_add_vsi_child_nodes(pi, vsi_handle, tc_node, + new_num_nodes, owner); + if (status) + return status; vsi_ctx->sched.max_lanq[tc] = new_numqs; - return status; + return 0; } /** @@ -1511,7 +1479,7 @@ ice_sched_update_vsi_child_nodes(struct ice_port_info *pi, u16 vsi_handle, * @vsi_handle: software VSI handle * @tc: TC number * @maxqs: max number of queues - * @owner: lan or rdma + * @owner: LAN or RDMA * @enable: TC enabled or disabled * * This function adds/updates VSI nodes based on the number of queues. If TC is @@ -1527,6 +1495,7 @@ ice_sched_cfg_vsi(struct ice_port_info *pi, u16 vsi_handle, u8 tc, u16 maxqs, enum ice_status status = 0; struct ice_hw *hw = pi->hw; + ice_debug(pi->hw, ICE_DBG_SCHED, "add/config VSI %d\n", vsi_handle); tc_node = ice_sched_get_tc_node(pi, tc); if (!tc_node) return ICE_ERR_PARAM; @@ -1535,7 +1504,7 @@ ice_sched_cfg_vsi(struct ice_port_info *pi, u16 vsi_handle, u8 tc, u16 maxqs, return ICE_ERR_PARAM; vsi_node = ice_sched_get_vsi_node(hw, tc_node, vsi_handle); - /* suspend the VSI if tc is not enabled */ + /* suspend the VSI if TC is not enabled */ if (!enable) { if (vsi_node && vsi_node->in_use) { u32 teid = le32_to_cpu(vsi_node->info.node_teid); @@ -1586,7 +1555,7 @@ ice_sched_cfg_vsi(struct ice_port_info *pi, u16 vsi_handle, u8 tc, u16 maxqs, } /** - * ice_sched_rm_agg_vsi_entry - remove agg related VSI info entry + * ice_sched_rm_agg_vsi_entry - remove aggregator related VSI info entry * @pi: port information structure * @vsi_handle: software VSI handle * @@ -1646,8 +1615,9 @@ ice_sched_rm_vsi_cfg(struct ice_port_info *pi, u16 vsi_handle, u8 owner) { enum ice_status status = ICE_ERR_PARAM; struct ice_vsi_ctx *vsi_ctx; - u8 i, j = 0; + u8 i; + ice_debug(pi->hw, ICE_DBG_SCHED, "removing VSI %d\n", vsi_handle); if (!ice_is_vsi_valid(pi->hw, vsi_handle)) return status; mutex_lock(&pi->sched_lock); @@ -1655,8 +1625,9 @@ ice_sched_rm_vsi_cfg(struct ice_port_info *pi, u16 vsi_handle, u8 owner) if (!vsi_ctx) goto exit_sched_rm_vsi_cfg; - for (i = 0; i < ICE_MAX_TRAFFIC_CLASS; i++) { + ice_for_each_traffic_class(i) { struct ice_sched_node *vsi_node, *tc_node; + u8 j = 0; tc_node = ice_sched_get_tc_node(pi, i); if (!tc_node) @@ -1689,7 +1660,7 @@ ice_sched_rm_vsi_cfg(struct ice_port_info *pi, u16 vsi_handle, u8 owner) ice_free_sched_node(pi, vsi_node); vsi_ctx->sched.vsi_node[i] = NULL; - /* clean up agg related vsi info if any */ + /* clean up aggregator related VSI info if any */ ice_sched_rm_agg_vsi_info(pi, vsi_handle); } if (owner == ICE_SCHED_NODE_OWNER_LAN) diff --git a/drivers/net/ethernet/intel/ice/ice_sched.h b/drivers/net/ethernet/intel/ice/ice_sched.h index bee8221ad146b43530c8add9320ab062366e3e8a..3902a8ad3025aa2e9f8cbbc4600cf42f2b86e462 100644 --- a/drivers/net/ethernet/intel/ice/ice_sched.h +++ b/drivers/net/ethernet/intel/ice/ice_sched.h @@ -24,6 +24,10 @@ struct ice_sched_agg_info { }; /* FW AQ command calls */ +enum ice_status +ice_aq_query_sched_elems(struct ice_hw *hw, u16 elems_req, + struct ice_aqc_get_elem *buf, u16 buf_size, + u16 *elems_ret, struct ice_sq_cd *cd); enum ice_status ice_sched_init_port(struct ice_port_info *pi); enum ice_status ice_sched_query_res_alloc(struct ice_hw *hw); void ice_sched_clear_port(struct ice_port_info *pi); diff --git a/drivers/net/ethernet/intel/ice/ice_status.h b/drivers/net/ethernet/intel/ice/ice_status.h index 683f48824a29ef21186ac5f87463684027163ee6..17afe6acb18a80f66cddc0909514b631002e26c2 100644 --- a/drivers/net/ethernet/intel/ice/ice_status.h +++ b/drivers/net/ethernet/intel/ice/ice_status.h @@ -12,6 +12,7 @@ enum ice_status { ICE_ERR_PARAM = -1, ICE_ERR_NOT_IMPL = -2, ICE_ERR_NOT_READY = -3, + ICE_ERR_NOT_SUPPORTED = -4, ICE_ERR_BAD_PTR = -5, ICE_ERR_INVAL_SIZE = -6, ICE_ERR_DEVICE_NOT_SUPPORTED = -8, diff --git a/drivers/net/ethernet/intel/ice/ice_switch.c b/drivers/net/ethernet/intel/ice/ice_switch.c index 09d1c314b68f39f982bc1eb49cfedae99b211c9b..9f1f595ae7e67ba9d107d560786be660b9c6fcf6 100644 --- a/drivers/net/ethernet/intel/ice/ice_switch.c +++ b/drivers/net/ethernet/intel/ice/ice_switch.c @@ -19,7 +19,7 @@ * byte 6 = 0x2: to identify it as locally administered SA MAC * byte 12 = 0x81 & byte 13 = 0x00: * In case of VLAN filter first two bytes defines ether type (0x8100) - * and remaining two bytes are placeholder for programming a given VLAN id + * and remaining two bytes are placeholder for programming a given VLAN ID * In case of Ether type filter it is treated as header without VLAN tag * and byte 12 and 13 is used to program a given Ether type instead */ @@ -51,7 +51,7 @@ static const u8 dummy_eth_header[DUMMY_ETH_HDR_LEN] = { 0x2, 0, 0, 0, 0, 0, /** * ice_aq_alloc_free_res - command to allocate/free resources - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @num_entries: number of resource entries in buffer * @buf: Indirect buffer to hold data parameters and response * @buf_size: size of buffer for indirect commands @@ -87,7 +87,7 @@ ice_aq_alloc_free_res(struct ice_hw *hw, u16 num_entries, /** * ice_init_def_sw_recp - initialize the recipe book keeping tables - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * * Allocate memory for the entire recipe table and initialize the structures/ * entries corresponding to basic recipes. @@ -163,7 +163,7 @@ ice_aq_get_sw_cfg(struct ice_hw *hw, struct ice_aqc_get_sw_cfg_resp *buf, /** * ice_aq_add_vsi - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @vsi_ctx: pointer to a VSI context struct * @cd: pointer to command details structure or NULL * @@ -206,7 +206,7 @@ ice_aq_add_vsi(struct ice_hw *hw, struct ice_vsi_ctx *vsi_ctx, /** * ice_aq_free_vsi - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @vsi_ctx: pointer to a VSI context struct * @keep_vsi_alloc: keep VSI allocation as part of this PF's resources * @cd: pointer to command details structure or NULL @@ -242,7 +242,7 @@ ice_aq_free_vsi(struct ice_hw *hw, struct ice_vsi_ctx *vsi_ctx, /** * ice_aq_update_vsi - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @vsi_ctx: pointer to a VSI context struct * @cd: pointer to command details structure or NULL * @@ -279,7 +279,7 @@ ice_aq_update_vsi(struct ice_hw *hw, struct ice_vsi_ctx *vsi_ctx, /** * ice_is_vsi_valid - check whether the VSI is valid or not - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @vsi_handle: VSI handle * * check whether the VSI is valid or not @@ -290,11 +290,11 @@ bool ice_is_vsi_valid(struct ice_hw *hw, u16 vsi_handle) } /** - * ice_get_hw_vsi_num - return the hw VSI number - * @hw: pointer to the hw struct + * ice_get_hw_vsi_num - return the HW VSI number + * @hw: pointer to the HW struct * @vsi_handle: VSI handle * - * return the hw VSI number + * return the HW VSI number * Caution: call this function only if VSI is valid (ice_is_vsi_valid) */ u16 ice_get_hw_vsi_num(struct ice_hw *hw, u16 vsi_handle) @@ -304,7 +304,7 @@ u16 ice_get_hw_vsi_num(struct ice_hw *hw, u16 vsi_handle) /** * ice_get_vsi_ctx - return the VSI context entry for a given VSI handle - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @vsi_handle: VSI handle * * return the VSI context entry for a given VSI handle @@ -316,21 +316,42 @@ struct ice_vsi_ctx *ice_get_vsi_ctx(struct ice_hw *hw, u16 vsi_handle) /** * ice_save_vsi_ctx - save the VSI context for a given VSI handle - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @vsi_handle: VSI handle * @vsi: VSI context pointer * * save the VSI context entry for a given VSI handle */ -static void ice_save_vsi_ctx(struct ice_hw *hw, u16 vsi_handle, - struct ice_vsi_ctx *vsi) +static void +ice_save_vsi_ctx(struct ice_hw *hw, u16 vsi_handle, struct ice_vsi_ctx *vsi) { hw->vsi_ctx[vsi_handle] = vsi; } +/** + * ice_clear_vsi_q_ctx - clear VSI queue contexts for all TCs + * @hw: pointer to the HW struct + * @vsi_handle: VSI handle + */ +static void ice_clear_vsi_q_ctx(struct ice_hw *hw, u16 vsi_handle) +{ + struct ice_vsi_ctx *vsi; + u8 i; + + vsi = ice_get_vsi_ctx(hw, vsi_handle); + if (!vsi) + return; + ice_for_each_traffic_class(i) { + if (vsi->lan_q_ctx[i]) { + devm_kfree(ice_hw_to_dev(hw), vsi->lan_q_ctx[i]); + vsi->lan_q_ctx[i] = NULL; + } + } +} + /** * ice_clear_vsi_ctx - clear the VSI context entry - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @vsi_handle: VSI handle * * clear the VSI context entry @@ -341,6 +362,7 @@ static void ice_clear_vsi_ctx(struct ice_hw *hw, u16 vsi_handle) vsi = ice_get_vsi_ctx(hw, vsi_handle); if (vsi) { + ice_clear_vsi_q_ctx(hw, vsi_handle); devm_kfree(ice_hw_to_dev(hw), vsi); hw->vsi_ctx[vsi_handle] = NULL; } @@ -348,7 +370,7 @@ static void ice_clear_vsi_ctx(struct ice_hw *hw, u16 vsi_handle) /** * ice_clear_all_vsi_ctx - clear all the VSI context entries - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct */ void ice_clear_all_vsi_ctx(struct ice_hw *hw) { @@ -360,7 +382,7 @@ void ice_clear_all_vsi_ctx(struct ice_hw *hw) /** * ice_add_vsi - add VSI context to the hardware and VSI handle list - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @vsi_handle: unique VSI handle provided by drivers * @vsi_ctx: pointer to a VSI context struct * @cd: pointer to command details structure or NULL @@ -383,7 +405,7 @@ ice_add_vsi(struct ice_hw *hw, u16 vsi_handle, struct ice_vsi_ctx *vsi_ctx, return status; tmp_vsi_ctx = ice_get_vsi_ctx(hw, vsi_handle); if (!tmp_vsi_ctx) { - /* Create a new vsi context */ + /* Create a new VSI context */ tmp_vsi_ctx = devm_kzalloc(ice_hw_to_dev(hw), sizeof(*tmp_vsi_ctx), GFP_KERNEL); if (!tmp_vsi_ctx) { @@ -398,12 +420,12 @@ ice_add_vsi(struct ice_hw *hw, u16 vsi_handle, struct ice_vsi_ctx *vsi_ctx, tmp_vsi_ctx->vsi_num = vsi_ctx->vsi_num; } - return status; + return 0; } /** * ice_free_vsi- free VSI context from hardware and VSI handle list - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @vsi_handle: unique VSI handle * @vsi_ctx: pointer to a VSI context struct * @keep_vsi_alloc: keep VSI allocation as part of this PF's resources @@ -428,7 +450,7 @@ ice_free_vsi(struct ice_hw *hw, u16 vsi_handle, struct ice_vsi_ctx *vsi_ctx, /** * ice_update_vsi - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @vsi_handle: unique VSI handle * @vsi_ctx: pointer to a VSI context struct * @cd: pointer to command details structure or NULL @@ -447,8 +469,8 @@ ice_update_vsi(struct ice_hw *hw, u16 vsi_handle, struct ice_vsi_ctx *vsi_ctx, /** * ice_aq_alloc_free_vsi_list - * @hw: pointer to the hw struct - * @vsi_list_id: VSI list id returned or used for lookup + * @hw: pointer to the HW struct + * @vsi_list_id: VSI list ID returned or used for lookup * @lkup_type: switch rule filter lookup type * @opc: switch rules population command type - pass in the command opcode * @@ -504,7 +526,7 @@ ice_aq_alloc_free_vsi_list_exit: /** * ice_aq_sw_rules - add/update/remove switch rules - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @rule_list: pointer to switch rule population list * @rule_list_sz: total size of the rule list in bytes * @num_rules: number of switch rules in the rule_list @@ -643,21 +665,43 @@ static void ice_fill_sw_info(struct ice_hw *hw, struct ice_fltr_info *fi) fi->fltr_act == ICE_FWD_TO_VSI_LIST || fi->fltr_act == ICE_FWD_TO_Q || fi->fltr_act == ICE_FWD_TO_QGRP)) { - fi->lb_en = true; - /* Do not set lan_en to TRUE if + /* Setting LB for prune actions will result in replicated + * packets to the internal switch that will be dropped. + */ + if (fi->lkup_type != ICE_SW_LKUP_VLAN) + fi->lb_en = true; + + /* Set lan_en to TRUE if * 1. The switch is a VEB AND * 2 - * 2.1 The lookup is MAC with unicast addr for MAC, OR - * 2.2 The lookup is MAC_VLAN with unicast addr for MAC + * 2.1 The lookup is a directional lookup like ethertype, + * promiscuous, ethertype-MAC, promiscuous-VLAN + * and default-port OR + * 2.2 The lookup is VLAN, OR + * 2.3 The lookup is MAC with mcast or bcast addr for MAC, OR + * 2.4 The lookup is MAC_VLAN with mcast or bcast addr for MAC. + * + * OR + * + * The switch is a VEPA. * - * In all other cases, the LAN enable has to be set to true. + * In all other cases, the LAN enable has to be set to false. */ - if (!(hw->evb_veb && - ((fi->lkup_type == ICE_SW_LKUP_MAC && - is_unicast_ether_addr(fi->l_data.mac.mac_addr)) || - (fi->lkup_type == ICE_SW_LKUP_MAC_VLAN && - is_unicast_ether_addr(fi->l_data.mac_vlan.mac_addr))))) + if (hw->evb_veb) { + if (fi->lkup_type == ICE_SW_LKUP_ETHERTYPE || + fi->lkup_type == ICE_SW_LKUP_PROMISC || + fi->lkup_type == ICE_SW_LKUP_ETHERTYPE_MAC || + fi->lkup_type == ICE_SW_LKUP_PROMISC_VLAN || + fi->lkup_type == ICE_SW_LKUP_DFLT || + fi->lkup_type == ICE_SW_LKUP_VLAN || + (fi->lkup_type == ICE_SW_LKUP_MAC && + !is_unicast_ether_addr(fi->l_data.mac.mac_addr)) || + (fi->lkup_type == ICE_SW_LKUP_MAC_VLAN && + !is_unicast_ether_addr(fi->l_data.mac.mac_addr))) + fi->lan_en = true; + } else { fi->lan_en = true; + } } } @@ -799,7 +843,7 @@ ice_fill_sw_rule(struct ice_hw *hw, struct ice_fltr_info *f_info, * @hw: pointer to the hardware structure * @m_ent: the management entry for which sw marker needs to be added * @sw_marker: sw marker to tag the Rx descriptor with - * @l_id: large action resource id + * @l_id: large action resource ID * * Create a large action to hold software marker and update the switch rule * entry pointed by m_ent with newly created large action @@ -811,8 +855,8 @@ ice_add_marker_act(struct ice_hw *hw, struct ice_fltr_mgmt_list_entry *m_ent, struct ice_aqc_sw_rules_elem *lg_act, *rx_tx; /* For software marker we need 3 large actions * 1. FWD action: FWD TO VSI or VSI LIST - * 2. GENERIC VALUE action to hold the profile id - * 3. GENERIC VALUE action to hold the software marker id + * 2. GENERIC VALUE action to hold the profile ID + * 3. GENERIC VALUE action to hold the software marker ID */ const u16 num_lg_acts = 3; enum ice_status status; @@ -875,13 +919,13 @@ ice_add_marker_act(struct ice_hw *hw, struct ice_fltr_mgmt_list_entry *m_ent, ice_fill_sw_rule(hw, &m_ent->fltr_info, rx_tx, ice_aqc_opc_update_sw_rules); - /* Update the action to point to the large action id */ + /* Update the action to point to the large action ID */ rx_tx->pdata.lkup_tx_rx.act = cpu_to_le32(ICE_SINGLE_ACT_PTR | ((l_id << ICE_SINGLE_ACT_PTR_VAL_S) & ICE_SINGLE_ACT_PTR_VAL_M)); - /* Use the filter rule id of the previously created rule with single + /* Use the filter rule ID of the previously created rule with single * act. Once the update happens, hardware will treat this as large * action */ @@ -904,10 +948,10 @@ ice_add_marker_act(struct ice_hw *hw, struct ice_fltr_mgmt_list_entry *m_ent, * @hw: pointer to the hardware structure * @vsi_handle_arr: array of VSI handles to set in the VSI mapping * @num_vsi: number of VSI handles in the array - * @vsi_list_id: VSI list id generated as part of allocate resource + * @vsi_list_id: VSI list ID generated as part of allocate resource * - * Helper function to create a new entry of VSI list id to VSI mapping - * using the given VSI list id + * Helper function to create a new entry of VSI list ID to VSI mapping + * using the given VSI list ID */ static struct ice_vsi_list_map_info * ice_create_vsi_list_map(struct ice_hw *hw, u16 *vsi_handle_arr, u16 num_vsi, @@ -935,13 +979,13 @@ ice_create_vsi_list_map(struct ice_hw *hw, u16 *vsi_handle_arr, u16 num_vsi, * @hw: pointer to the hardware structure * @vsi_handle_arr: array of VSI handles to form a VSI list * @num_vsi: number of VSI handles in the array - * @vsi_list_id: VSI list id generated as part of allocate resource + * @vsi_list_id: VSI list ID generated as part of allocate resource * @remove: Boolean value to indicate if this is a remove action * @opc: switch rules population command type - pass in the command opcode * @lkup_type: lookup type of the filter * * Call AQ command to add a new switch rule or update existing switch rule - * using the given VSI list id + * using the given VSI list ID */ static enum ice_status ice_update_vsi_list_rule(struct ice_hw *hw, u16 *vsi_handle_arr, u16 num_vsi, @@ -998,7 +1042,7 @@ exit: /** * ice_create_vsi_list_rule - Creates and populates a VSI list rule - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * @vsi_handle_arr: array of VSI handles to form a VSI list * @num_vsi: number of VSI handles in the array * @vsi_list_id: stores the ID of the VSI list to be created @@ -1092,7 +1136,7 @@ ice_create_pkt_fwd_rule_exit: * @f_info: filter information for switch rule * * Call AQ command to update a previously created switch rule with a - * VSI list id + * VSI list ID */ static enum ice_status ice_update_pkt_fwd_rule(struct ice_hw *hw, struct ice_fltr_info *f_info) @@ -1119,7 +1163,7 @@ ice_update_pkt_fwd_rule(struct ice_hw *hw, struct ice_fltr_info *f_info) /** * ice_update_sw_rule_bridge_mode - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * * Updates unicast switch filter rules based on VEB/VEPA mode */ @@ -1174,7 +1218,7 @@ enum ice_status ice_update_sw_rule_bridge_mode(struct ice_hw *hw) * Allocate a new VSI list and add two VSIs * to this list using switch rule command * Update the previously created switch rule with the - * newly created VSI list id + * newly created VSI list ID * if a VSI list was previously created * Add the new VSI to the previously created VSI list set * using the update switch rule command @@ -1255,7 +1299,7 @@ ice_add_update_vsi_list(struct ice_hw *hw, return 0; /* Update the previously created VSI list set with - * the new VSI id passed in + * the new VSI ID passed in */ vsi_list_id = cur_fltr->fwd_id.vsi_list_id; opcode = ice_aqc_opc_update_sw_rules; @@ -1263,7 +1307,7 @@ ice_add_update_vsi_list(struct ice_hw *hw, status = ice_update_vsi_list_rule(hw, &vsi_handle, 1, vsi_list_id, false, opcode, new_fltr->lkup_type); - /* update VSI list mapping info with new VSI id */ + /* update VSI list mapping info with new VSI ID */ if (!status) set_bit(vsi_handle, m_entry->vsi_list_info->vsi_map); } @@ -1305,7 +1349,7 @@ ice_find_rule_entry(struct ice_hw *hw, u8 recp_id, struct ice_fltr_info *f_info) * @hw: pointer to the hardware structure * @recp_id: lookup type for which VSI lists needs to be searched * @vsi_handle: VSI handle to be found in VSI list - * @vsi_list_id: VSI list id found containing vsi_handle + * @vsi_list_id: VSI list ID found containing vsi_handle * * Helper function to search a VSI list with single entry containing given VSI * handle element. This can be extended further to search VSI list with more @@ -1336,7 +1380,7 @@ ice_find_vsi_list_entry(struct ice_hw *hw, u8 recp_id, u16 vsi_handle, /** * ice_add_rule_internal - add rule for a given lookup type * @hw: pointer to the hardware structure - * @recp_id: lookup type (recipe id) for which rule has to be added + * @recp_id: lookup type (recipe ID) for which rule has to be added * @f_entry: structure containing MAC forwarding information * * Adds or updates the rule lists for a given recipe @@ -1381,7 +1425,7 @@ ice_add_rule_internal(struct ice_hw *hw, u8 recp_id, /** * ice_remove_vsi_list_rule * @hw: pointer to the hardware structure - * @vsi_list_id: VSI list id generated as part of allocate resource + * @vsi_list_id: VSI list ID generated as part of allocate resource * @lkup_type: switch rule filter lookup type * * The VSI list should be emptied before this function is called to remove the @@ -1506,7 +1550,7 @@ ice_rem_update_vsi_list(struct ice_hw *hw, u16 vsi_handle, /** * ice_remove_rule_internal - Remove a filter rule of a given type * @hw: pointer to the hardware structure - * @recp_id: recipe id for which the rule needs to removed + * @recp_id: recipe ID for which the rule needs to removed * @f_entry: rule entry containing filter information */ static enum ice_status @@ -1556,7 +1600,7 @@ ice_remove_rule_internal(struct ice_hw *hw, u8 recp_id, status = ice_rem_update_vsi_list(hw, vsi_handle, list_elem); if (status) goto exit; - /* if vsi count goes to zero after updating the vsi list */ + /* if VSI count goes to zero after updating the VSI list */ if (list_elem->vsi_count == 0) remove_rule = true; } @@ -1634,7 +1678,7 @@ ice_add_mac(struct ice_hw *hw, struct list_head *m_list) return ICE_ERR_PARAM; hw_vsi_id = ice_get_hw_vsi_num(hw, vsi_handle); m_list_itr->fltr_info.fwd_id.hw_vsi_id = hw_vsi_id; - /* update the src in case it is vsi num */ + /* update the src in case it is VSI num */ if (m_list_itr->fltr_info.src_id != ICE_SRC_ID_VSI) return ICE_ERR_PARAM; m_list_itr->fltr_info.src = hw_vsi_id; @@ -1710,7 +1754,7 @@ ice_add_mac(struct ice_hw *hw, struct list_head *m_list) ((u8 *)r_iter + (elem_sent * s_rule_size)); } - /* Fill up rule id based on the value returned from FW */ + /* Fill up rule ID based on the value returned from FW */ r_iter = s_rule; list_for_each_entry(m_list_itr, m_list, list_entry) { struct ice_fltr_info *f_info = &m_list_itr->fltr_info; @@ -1770,7 +1814,7 @@ ice_add_vlan_internal(struct ice_hw *hw, struct ice_fltr_list_entry *f_entry) ice_get_hw_vsi_num(hw, f_entry->fltr_info.vsi_handle); new_fltr = &f_entry->fltr_info; - /* VLAN id should only be 12 bits */ + /* VLAN ID should only be 12 bits */ if (new_fltr->l_data.vlan.vlan_id > ICE_MAX_VLAN_ID) return ICE_ERR_PARAM; @@ -1828,7 +1872,7 @@ ice_add_vlan_internal(struct ice_hw *hw, struct ice_fltr_list_entry *f_entry) } } } else if (v_list_itr->vsi_list_info->ref_cnt == 1) { - /* Update existing VSI list to add new VSI id only if it used + /* Update existing VSI list to add new VSI ID only if it used * by one VLAN rule. */ cur_fltr = &v_list_itr->fltr_info; @@ -1838,7 +1882,7 @@ ice_add_vlan_internal(struct ice_hw *hw, struct ice_fltr_list_entry *f_entry) /* If VLAN rule exists and VSI list being used by this rule is * referenced by more than 1 VLAN rule. Then create a new VSI * list appending previous VSI with new VSI and update existing - * VLAN rule to point to new VSI list id + * VLAN rule to point to new VSI list ID */ struct ice_fltr_info tmp_fltr; u16 vsi_handle_arr[2]; @@ -1925,6 +1969,65 @@ ice_add_vlan(struct ice_hw *hw, struct list_head *v_list) return 0; } +/** + * ice_add_eth_mac - Add ethertype and MAC based filter rule + * @hw: pointer to the hardware structure + * @em_list: list of ether type MAC filter, MAC is optional + */ +enum ice_status +ice_add_eth_mac(struct ice_hw *hw, struct list_head *em_list) +{ + struct ice_fltr_list_entry *em_list_itr; + + if (!em_list || !hw) + return ICE_ERR_PARAM; + + list_for_each_entry(em_list_itr, em_list, list_entry) { + enum ice_sw_lkup_type l_type = + em_list_itr->fltr_info.lkup_type; + + if (l_type != ICE_SW_LKUP_ETHERTYPE_MAC && + l_type != ICE_SW_LKUP_ETHERTYPE) + return ICE_ERR_PARAM; + + em_list_itr->fltr_info.flag = ICE_FLTR_TX; + em_list_itr->status = ice_add_rule_internal(hw, l_type, + em_list_itr); + if (em_list_itr->status) + return em_list_itr->status; + } + return 0; +} + +/** + * ice_remove_eth_mac - Remove an ethertype (or MAC) based filter rule + * @hw: pointer to the hardware structure + * @em_list: list of ethertype or ethertype MAC entries + */ +enum ice_status +ice_remove_eth_mac(struct ice_hw *hw, struct list_head *em_list) +{ + struct ice_fltr_list_entry *em_list_itr, *tmp; + + if (!em_list || !hw) + return ICE_ERR_PARAM; + + list_for_each_entry_safe(em_list_itr, tmp, em_list, list_entry) { + enum ice_sw_lkup_type l_type = + em_list_itr->fltr_info.lkup_type; + + if (l_type != ICE_SW_LKUP_ETHERTYPE_MAC && + l_type != ICE_SW_LKUP_ETHERTYPE) + return ICE_ERR_PARAM; + + em_list_itr->status = ice_remove_rule_internal(hw, l_type, + em_list_itr); + if (em_list_itr->status) + return em_list_itr->status; + } + return 0; +} + /** * ice_rem_sw_rule_info * @hw: pointer to the hardware structure @@ -2170,7 +2273,7 @@ ice_add_to_vsi_fltr_list(struct ice_hw *hw, u16 vsi_handle, struct ice_fltr_mgmt_list_entry *fm_entry; enum ice_status status = 0; - /* check to make sure VSI id is valid and within boundary */ + /* check to make sure VSI ID is valid and within boundary */ if (!ice_is_vsi_valid(hw, vsi_handle)) return ICE_ERR_PARAM; @@ -2189,6 +2292,291 @@ ice_add_to_vsi_fltr_list(struct ice_hw *hw, u16 vsi_handle, return status; } +/** + * ice_determine_promisc_mask + * @fi: filter info to parse + * + * Helper function to determine which ICE_PROMISC_ mask corresponds + * to given filter into. + */ +static u8 ice_determine_promisc_mask(struct ice_fltr_info *fi) +{ + u16 vid = fi->l_data.mac_vlan.vlan_id; + u8 *macaddr = fi->l_data.mac.mac_addr; + bool is_tx_fltr = false; + u8 promisc_mask = 0; + + if (fi->flag == ICE_FLTR_TX) + is_tx_fltr = true; + + if (is_broadcast_ether_addr(macaddr)) + promisc_mask |= is_tx_fltr ? + ICE_PROMISC_BCAST_TX : ICE_PROMISC_BCAST_RX; + else if (is_multicast_ether_addr(macaddr)) + promisc_mask |= is_tx_fltr ? + ICE_PROMISC_MCAST_TX : ICE_PROMISC_MCAST_RX; + else if (is_unicast_ether_addr(macaddr)) + promisc_mask |= is_tx_fltr ? + ICE_PROMISC_UCAST_TX : ICE_PROMISC_UCAST_RX; + if (vid) + promisc_mask |= is_tx_fltr ? + ICE_PROMISC_VLAN_TX : ICE_PROMISC_VLAN_RX; + + return promisc_mask; +} + +/** + * ice_remove_promisc - Remove promisc based filter rules + * @hw: pointer to the hardware structure + * @recp_id: recipe ID for which the rule needs to removed + * @v_list: list of promisc entries + */ +static enum ice_status +ice_remove_promisc(struct ice_hw *hw, u8 recp_id, + struct list_head *v_list) +{ + struct ice_fltr_list_entry *v_list_itr, *tmp; + + list_for_each_entry_safe(v_list_itr, tmp, v_list, list_entry) { + v_list_itr->status = + ice_remove_rule_internal(hw, recp_id, v_list_itr); + if (v_list_itr->status) + return v_list_itr->status; + } + return 0; +} + +/** + * ice_clear_vsi_promisc - clear specified promiscuous mode(s) for given VSI + * @hw: pointer to the hardware structure + * @vsi_handle: VSI handle to clear mode + * @promisc_mask: mask of promiscuous config bits to clear + * @vid: VLAN ID to clear VLAN promiscuous + */ +enum ice_status +ice_clear_vsi_promisc(struct ice_hw *hw, u16 vsi_handle, u8 promisc_mask, + u16 vid) +{ + struct ice_switch_info *sw = hw->switch_info; + struct ice_fltr_list_entry *fm_entry, *tmp; + struct list_head remove_list_head; + struct ice_fltr_mgmt_list_entry *itr; + struct list_head *rule_head; + struct mutex *rule_lock; /* Lock to protect filter rule list */ + enum ice_status status = 0; + u8 recipe_id; + + if (!ice_is_vsi_valid(hw, vsi_handle)) + return ICE_ERR_PARAM; + + if (vid) + recipe_id = ICE_SW_LKUP_PROMISC_VLAN; + else + recipe_id = ICE_SW_LKUP_PROMISC; + + rule_head = &sw->recp_list[recipe_id].filt_rules; + rule_lock = &sw->recp_list[recipe_id].filt_rule_lock; + + INIT_LIST_HEAD(&remove_list_head); + + mutex_lock(rule_lock); + list_for_each_entry(itr, rule_head, list_entry) { + u8 fltr_promisc_mask = 0; + + if (!ice_vsi_uses_fltr(itr, vsi_handle)) + continue; + + fltr_promisc_mask |= + ice_determine_promisc_mask(&itr->fltr_info); + + /* Skip if filter is not completely specified by given mask */ + if (fltr_promisc_mask & ~promisc_mask) + continue; + + status = ice_add_entry_to_vsi_fltr_list(hw, vsi_handle, + &remove_list_head, + &itr->fltr_info); + if (status) { + mutex_unlock(rule_lock); + goto free_fltr_list; + } + } + mutex_unlock(rule_lock); + + status = ice_remove_promisc(hw, recipe_id, &remove_list_head); + +free_fltr_list: + list_for_each_entry_safe(fm_entry, tmp, &remove_list_head, list_entry) { + list_del(&fm_entry->list_entry); + devm_kfree(ice_hw_to_dev(hw), fm_entry); + } + + return status; +} + +/** + * ice_set_vsi_promisc - set given VSI to given promiscuous mode(s) + * @hw: pointer to the hardware structure + * @vsi_handle: VSI handle to configure + * @promisc_mask: mask of promiscuous config bits + * @vid: VLAN ID to set VLAN promiscuous + */ +enum ice_status +ice_set_vsi_promisc(struct ice_hw *hw, u16 vsi_handle, u8 promisc_mask, u16 vid) +{ + enum { UCAST_FLTR = 1, MCAST_FLTR, BCAST_FLTR }; + struct ice_fltr_list_entry f_list_entry; + struct ice_fltr_info new_fltr; + enum ice_status status = 0; + bool is_tx_fltr; + u16 hw_vsi_id; + int pkt_type; + u8 recipe_id; + + if (!ice_is_vsi_valid(hw, vsi_handle)) + return ICE_ERR_PARAM; + hw_vsi_id = ice_get_hw_vsi_num(hw, vsi_handle); + + memset(&new_fltr, 0, sizeof(new_fltr)); + + if (promisc_mask & (ICE_PROMISC_VLAN_RX | ICE_PROMISC_VLAN_TX)) { + new_fltr.lkup_type = ICE_SW_LKUP_PROMISC_VLAN; + new_fltr.l_data.mac_vlan.vlan_id = vid; + recipe_id = ICE_SW_LKUP_PROMISC_VLAN; + } else { + new_fltr.lkup_type = ICE_SW_LKUP_PROMISC; + recipe_id = ICE_SW_LKUP_PROMISC; + } + + /* Separate filters must be set for each direction/packet type + * combination, so we will loop over the mask value, store the + * individual type, and clear it out in the input mask as it + * is found. + */ + while (promisc_mask) { + u8 *mac_addr; + + pkt_type = 0; + is_tx_fltr = false; + + if (promisc_mask & ICE_PROMISC_UCAST_RX) { + promisc_mask &= ~ICE_PROMISC_UCAST_RX; + pkt_type = UCAST_FLTR; + } else if (promisc_mask & ICE_PROMISC_UCAST_TX) { + promisc_mask &= ~ICE_PROMISC_UCAST_TX; + pkt_type = UCAST_FLTR; + is_tx_fltr = true; + } else if (promisc_mask & ICE_PROMISC_MCAST_RX) { + promisc_mask &= ~ICE_PROMISC_MCAST_RX; + pkt_type = MCAST_FLTR; + } else if (promisc_mask & ICE_PROMISC_MCAST_TX) { + promisc_mask &= ~ICE_PROMISC_MCAST_TX; + pkt_type = MCAST_FLTR; + is_tx_fltr = true; + } else if (promisc_mask & ICE_PROMISC_BCAST_RX) { + promisc_mask &= ~ICE_PROMISC_BCAST_RX; + pkt_type = BCAST_FLTR; + } else if (promisc_mask & ICE_PROMISC_BCAST_TX) { + promisc_mask &= ~ICE_PROMISC_BCAST_TX; + pkt_type = BCAST_FLTR; + is_tx_fltr = true; + } + + /* Check for VLAN promiscuous flag */ + if (promisc_mask & ICE_PROMISC_VLAN_RX) { + promisc_mask &= ~ICE_PROMISC_VLAN_RX; + } else if (promisc_mask & ICE_PROMISC_VLAN_TX) { + promisc_mask &= ~ICE_PROMISC_VLAN_TX; + is_tx_fltr = true; + } + + /* Set filter DA based on packet type */ + mac_addr = new_fltr.l_data.mac.mac_addr; + if (pkt_type == BCAST_FLTR) { + eth_broadcast_addr(mac_addr); + } else if (pkt_type == MCAST_FLTR || + pkt_type == UCAST_FLTR) { + /* Use the dummy ether header DA */ + ether_addr_copy(mac_addr, dummy_eth_header); + if (pkt_type == MCAST_FLTR) + mac_addr[0] |= 0x1; /* Set multicast bit */ + } + + /* Need to reset this to zero for all iterations */ + new_fltr.flag = 0; + if (is_tx_fltr) { + new_fltr.flag |= ICE_FLTR_TX; + new_fltr.src = hw_vsi_id; + } else { + new_fltr.flag |= ICE_FLTR_RX; + new_fltr.src = hw->port_info->lport; + } + + new_fltr.fltr_act = ICE_FWD_TO_VSI; + new_fltr.vsi_handle = vsi_handle; + new_fltr.fwd_id.hw_vsi_id = hw_vsi_id; + f_list_entry.fltr_info = new_fltr; + + status = ice_add_rule_internal(hw, recipe_id, &f_list_entry); + if (status) + goto set_promisc_exit; + } + +set_promisc_exit: + return status; +} + +/** + * ice_set_vlan_vsi_promisc + * @hw: pointer to the hardware structure + * @vsi_handle: VSI handle to configure + * @promisc_mask: mask of promiscuous config bits + * @rm_vlan_promisc: Clear VLANs VSI promisc mode + * + * Configure VSI with all associated VLANs to given promiscuous mode(s) + */ +enum ice_status +ice_set_vlan_vsi_promisc(struct ice_hw *hw, u16 vsi_handle, u8 promisc_mask, + bool rm_vlan_promisc) +{ + struct ice_switch_info *sw = hw->switch_info; + struct ice_fltr_list_entry *list_itr, *tmp; + struct list_head vsi_list_head; + struct list_head *vlan_head; + struct mutex *vlan_lock; /* Lock to protect filter rule list */ + enum ice_status status; + u16 vlan_id; + + INIT_LIST_HEAD(&vsi_list_head); + vlan_lock = &sw->recp_list[ICE_SW_LKUP_VLAN].filt_rule_lock; + vlan_head = &sw->recp_list[ICE_SW_LKUP_VLAN].filt_rules; + mutex_lock(vlan_lock); + status = ice_add_to_vsi_fltr_list(hw, vsi_handle, vlan_head, + &vsi_list_head); + mutex_unlock(vlan_lock); + if (status) + goto free_fltr_list; + + list_for_each_entry(list_itr, &vsi_list_head, list_entry) { + vlan_id = list_itr->fltr_info.l_data.vlan.vlan_id; + if (rm_vlan_promisc) + status = ice_clear_vsi_promisc(hw, vsi_handle, + promisc_mask, vlan_id); + else + status = ice_set_vsi_promisc(hw, vsi_handle, + promisc_mask, vlan_id); + if (status) + break; + } + +free_fltr_list: + list_for_each_entry_safe(list_itr, tmp, &vsi_list_head, list_entry) { + list_del(&list_itr->list_entry); + devm_kfree(ice_hw_to_dev(hw), list_itr); + } + return status; +} + /** * ice_remove_vsi_lkup_fltr - Remove lookup type filters for a VSI * @hw: pointer to the hardware structure @@ -2224,12 +2612,14 @@ ice_remove_vsi_lkup_fltr(struct ice_hw *hw, u16 vsi_handle, case ICE_SW_LKUP_VLAN: ice_remove_vlan(hw, &remove_list_head); break; + case ICE_SW_LKUP_PROMISC: + case ICE_SW_LKUP_PROMISC_VLAN: + ice_remove_promisc(hw, lkup, &remove_list_head); + break; case ICE_SW_LKUP_MAC_VLAN: case ICE_SW_LKUP_ETHERTYPE: case ICE_SW_LKUP_ETHERTYPE_MAC: - case ICE_SW_LKUP_PROMISC: case ICE_SW_LKUP_DFLT: - case ICE_SW_LKUP_PROMISC_VLAN: case ICE_SW_LKUP_LAST: default: ice_debug(hw, ICE_DBG_SW, "Unsupported lookup type %d\n", lkup); @@ -2263,7 +2653,7 @@ void ice_remove_vsi_fltr(struct ice_hw *hw, u16 vsi_handle) * ice_replay_vsi_fltr - Replay filters for requested VSI * @hw: pointer to the hardware structure * @vsi_handle: driver VSI handle - * @recp_id: Recipe id for which rules need to be replayed + * @recp_id: Recipe ID for which rules need to be replayed * @list_head: list for which filters need to be replayed * * Replays the filter of recipe recp_id for a VSI represented via vsi_handle. @@ -2287,7 +2677,7 @@ ice_replay_vsi_fltr(struct ice_hw *hw, u16 vsi_handle, u8 recp_id, f_entry.fltr_info = itr->fltr_info; if (itr->vsi_count < 2 && recp_id != ICE_SW_LKUP_VLAN && itr->fltr_info.vsi_handle == vsi_handle) { - /* update the src in case it is vsi num */ + /* update the src in case it is VSI num */ if (f_entry.fltr_info.src_id == ICE_SRC_ID_VSI) f_entry.fltr_info.src = hw_vsi_id; status = ice_add_rule_internal(hw, recp_id, &f_entry); @@ -2302,7 +2692,7 @@ ice_replay_vsi_fltr(struct ice_hw *hw, u16 vsi_handle, u8 recp_id, clear_bit(vsi_handle, itr->vsi_list_info->vsi_map); f_entry.fltr_info.vsi_handle = vsi_handle; f_entry.fltr_info.fltr_act = ICE_FWD_TO_VSI; - /* update the src in case it is vsi num */ + /* update the src in case it is VSI num */ if (f_entry.fltr_info.src_id == ICE_SRC_ID_VSI) f_entry.fltr_info.src = hw_vsi_id; if (recp_id == ICE_SW_LKUP_VLAN) @@ -2342,7 +2732,7 @@ enum ice_status ice_replay_vsi_all_fltr(struct ice_hw *hw, u16 vsi_handle) /** * ice_rm_all_sw_replay_rule_info - deletes filter replay rules - * @hw: pointer to the hw struct + * @hw: pointer to the HW struct * * Deletes the filter replay rules. */ diff --git a/drivers/net/ethernet/intel/ice/ice_switch.h b/drivers/net/ethernet/intel/ice/ice_switch.h index d5ef0bd58bf9789260bbf7575868f16a6123ec81..732b0b9b2e156616c984763f41490ef607dc446a 100644 --- a/drivers/net/ethernet/intel/ice/ice_switch.h +++ b/drivers/net/ethernet/intel/ice/ice_switch.h @@ -9,6 +9,13 @@ #define ICE_SW_CFG_MAX_BUF_LEN 2048 #define ICE_DFLT_VSI_INVAL 0xff #define ICE_VSI_INVAL_ID 0xffff +#define ICE_INVAL_Q_HANDLE 0xFFFF +#define ICE_INVAL_Q_HANDLE 0xFFFF + +/* VSI queue context structure */ +struct ice_q_ctx { + u16 q_handle; +}; /* VSI context structure for add/get/update/free operations */ struct ice_vsi_ctx { @@ -20,6 +27,8 @@ struct ice_vsi_ctx { struct ice_sched_vsi_info sched; u8 alloc_from_pool; u8 vf_num; + u16 num_lan_q_entries[ICE_MAX_TRAFFIC_CLASS]; + struct ice_q_ctx *lan_q_ctx[ICE_MAX_TRAFFIC_CLASS]; }; enum ice_sw_fwd_act_type { @@ -44,7 +53,7 @@ enum ice_sw_lkup_type { ICE_SW_LKUP_LAST }; -/* type of filter src id */ +/* type of filter src ID */ enum ice_src_id { ICE_SRC_ID_UNKNOWN = 0, ICE_SRC_ID_VSI, @@ -95,8 +104,8 @@ struct ice_fltr_info { /* Depending on filter action */ union { - /* queue id in case of ICE_FWD_TO_Q and starting - * queue id in case of ICE_FWD_TO_QGRP. + /* queue ID in case of ICE_FWD_TO_Q and starting + * queue ID in case of ICE_FWD_TO_QGRP. */ u16 q_id:11; u16 hw_vsi_id:10; @@ -143,7 +152,7 @@ struct ice_sw_recipe { DECLARE_BITMAP(r_bitmap, ICE_MAX_NUM_RECIPES); }; -/* Bookkeeping structure to hold bitmap of VSIs corresponding to VSI list id */ +/* Bookkeeping structure to hold bitmap of VSIs corresponding to VSI list ID */ struct ice_vsi_list_map_info { struct list_head list_entry; DECLARE_BITMAP(vsi_map, ICE_MAX_VSI); @@ -165,7 +174,7 @@ struct ice_fltr_list_entry { * used for VLAN membership. */ struct ice_fltr_mgmt_list_entry { - /* back pointer to VSI list id to VSI list mapping */ + /* back pointer to VSI list ID to VSI list mapping */ struct ice_vsi_list_map_info *vsi_list_info; u16 vsi_count; #define ICE_INVAL_LG_ACT_INDEX 0xffff @@ -178,6 +187,17 @@ struct ice_fltr_mgmt_list_entry { u8 counter_index; }; +enum ice_promisc_flags { + ICE_PROMISC_UCAST_RX = 0x1, + ICE_PROMISC_UCAST_TX = 0x2, + ICE_PROMISC_MCAST_RX = 0x4, + ICE_PROMISC_MCAST_TX = 0x8, + ICE_PROMISC_BCAST_RX = 0x10, + ICE_PROMISC_BCAST_TX = 0x20, + ICE_PROMISC_VLAN_RX = 0x40, + ICE_PROMISC_VLAN_TX = 0x80, +}; + /* VSI related commands */ enum ice_status ice_add_vsi(struct ice_hw *hw, u16 vsi_handle, struct ice_vsi_ctx *vsi_ctx, @@ -198,11 +218,27 @@ enum ice_status ice_get_initial_sw_cfg(struct ice_hw *hw); enum ice_status ice_update_sw_rule_bridge_mode(struct ice_hw *hw); enum ice_status ice_add_mac(struct ice_hw *hw, struct list_head *m_lst); enum ice_status ice_remove_mac(struct ice_hw *hw, struct list_head *m_lst); +enum ice_status +ice_add_eth_mac(struct ice_hw *hw, struct list_head *em_list); +enum ice_status +ice_remove_eth_mac(struct ice_hw *hw, struct list_head *em_list); void ice_remove_vsi_fltr(struct ice_hw *hw, u16 vsi_handle); -enum ice_status ice_add_vlan(struct ice_hw *hw, struct list_head *m_list); +enum ice_status +ice_add_vlan(struct ice_hw *hw, struct list_head *m_list); enum ice_status ice_remove_vlan(struct ice_hw *hw, struct list_head *v_list); + +/* Promisc/defport setup for VSIs */ enum ice_status ice_cfg_dflt_vsi(struct ice_hw *hw, u16 vsi_handle, bool set, u8 direction); +enum ice_status +ice_set_vsi_promisc(struct ice_hw *hw, u16 vsi_handle, u8 promisc_mask, + u16 vid); +enum ice_status +ice_clear_vsi_promisc(struct ice_hw *hw, u16 vsi_handle, u8 promisc_mask, + u16 vid); +enum ice_status +ice_set_vlan_vsi_promisc(struct ice_hw *hw, u16 vsi_handle, u8 promisc_mask, + bool rm_vlan_promisc); enum ice_status ice_init_def_sw_recp(struct ice_hw *hw); u16 ice_get_hw_vsi_num(struct ice_hw *hw, u16 vsi_handle); diff --git a/drivers/net/ethernet/intel/ice/ice_txrx.c b/drivers/net/ethernet/intel/ice/ice_txrx.c index 1af21bbe180eef0f6067d565b6c3100e5c8f552e..2364eaf33d236ff5f78b7a47a76cdb8af0445e2d 100644 --- a/drivers/net/ethernet/intel/ice/ice_txrx.c +++ b/drivers/net/ethernet/intel/ice/ice_txrx.c @@ -6,6 +6,7 @@ #include #include #include "ice.h" +#include "ice_dcb_lib.h" #define ICE_RX_HDR_SIZE 256 @@ -100,8 +101,8 @@ void ice_free_tx_ring(struct ice_ring *tx_ring) * * Returns true if there's any budget left (e.g. the clean is finished) */ -static bool ice_clean_tx_irq(struct ice_vsi *vsi, struct ice_ring *tx_ring, - int napi_budget) +static bool +ice_clean_tx_irq(struct ice_vsi *vsi, struct ice_ring *tx_ring, int napi_budget) { unsigned int total_bytes = 0, total_pkts = 0; unsigned int budget = vsi->work_lmt; @@ -236,9 +237,9 @@ int ice_setup_tx_ring(struct ice_ring *tx_ring) if (!tx_ring->tx_buf) return -ENOMEM; - /* round up to nearest 4K */ + /* round up to nearest page */ tx_ring->size = ALIGN(tx_ring->count * sizeof(struct ice_tx_desc), - 4096); + PAGE_SIZE); tx_ring->desc = dmam_alloc_coherent(dev, tx_ring->size, &tx_ring->dma, GFP_KERNEL); if (!tx_ring->desc) { @@ -282,8 +283,17 @@ void ice_clean_rx_ring(struct ice_ring *rx_ring) if (!rx_buf->page) continue; - dma_unmap_page(dev, rx_buf->dma, PAGE_SIZE, DMA_FROM_DEVICE); - __free_pages(rx_buf->page, 0); + /* Invalidate cache lines that may have been written to by + * device so that we avoid corrupting memory. + */ + dma_sync_single_range_for_cpu(dev, rx_buf->dma, + rx_buf->page_offset, + ICE_RXBUF_2048, DMA_FROM_DEVICE); + + /* free resources associated with mapping */ + dma_unmap_page_attrs(dev, rx_buf->dma, PAGE_SIZE, + DMA_FROM_DEVICE, ICE_RX_DMA_ATTR); + __page_frag_cache_drain(rx_buf->page, rx_buf->pagecnt_bias); rx_buf->page = NULL; rx_buf->page_offset = 0; @@ -339,9 +349,9 @@ int ice_setup_rx_ring(struct ice_ring *rx_ring) if (!rx_ring->rx_buf) return -ENOMEM; - /* round up to nearest 4K */ - rx_ring->size = rx_ring->count * sizeof(union ice_32byte_rx_desc); - rx_ring->size = ALIGN(rx_ring->size, 4096); + /* round up to nearest page */ + rx_ring->size = ALIGN(rx_ring->count * sizeof(union ice_32byte_rx_desc), + PAGE_SIZE); rx_ring->desc = dmam_alloc_coherent(dev, rx_ring->size, &rx_ring->dma, GFP_KERNEL); if (!rx_ring->desc) { @@ -389,8 +399,8 @@ static void ice_release_rx_desc(struct ice_ring *rx_ring, u32 val) * Returns true if the page was successfully allocated or * reused. */ -static bool ice_alloc_mapped_page(struct ice_ring *rx_ring, - struct ice_rx_buf *bi) +static bool +ice_alloc_mapped_page(struct ice_ring *rx_ring, struct ice_rx_buf *bi) { struct page *page = bi->page; dma_addr_t dma; @@ -409,7 +419,8 @@ static bool ice_alloc_mapped_page(struct ice_ring *rx_ring, } /* map page for use */ - dma = dma_map_page(rx_ring->dev, page, 0, PAGE_SIZE, DMA_FROM_DEVICE); + dma = dma_map_page_attrs(rx_ring->dev, page, 0, PAGE_SIZE, + DMA_FROM_DEVICE, ICE_RX_DMA_ATTR); /* if mapping failed free memory back to system since * there isn't much point in holding memory we can't use @@ -423,6 +434,8 @@ static bool ice_alloc_mapped_page(struct ice_ring *rx_ring, bi->dma = dma; bi->page = page; bi->page_offset = 0; + page_ref_add(page, USHRT_MAX - 1); + bi->pagecnt_bias = USHRT_MAX; return true; } @@ -444,7 +457,7 @@ bool ice_alloc_rx_bufs(struct ice_ring *rx_ring, u16 cleaned_count) if (!rx_ring->netdev || !cleaned_count) return false; - /* get the RX descriptor and buffer based on next_to_use */ + /* get the Rx descriptor and buffer based on next_to_use */ rx_desc = ICE_RX_DESC(rx_ring, ntu); bi = &rx_ring->rx_buf[ntu]; @@ -452,6 +465,12 @@ bool ice_alloc_rx_bufs(struct ice_ring *rx_ring, u16 cleaned_count) if (!ice_alloc_mapped_page(rx_ring, bi)) goto no_bufs; + /* sync the buffer for use by the device */ + dma_sync_single_range_for_device(rx_ring->dev, bi->dma, + bi->page_offset, + ICE_RXBUF_2048, + DMA_FROM_DEVICE); + /* Refresh the desc even if buffer_addrs didn't change * because each write-back erases this info. */ @@ -497,61 +516,43 @@ static bool ice_page_is_reserved(struct page *page) } /** - * ice_add_rx_frag - Add contents of Rx buffer to sk_buff - * @rx_buf: buffer containing page to add - * @rx_desc: descriptor containing length of buffer written by hardware - * @skb: sk_buf to place the data into - * - * This function will add the data contained in rx_buf->page to the skb. - * This is done either through a direct copy if the data in the buffer is - * less than the skb header size, otherwise it will just attach the page as - * a frag to the skb. + * ice_rx_buf_adjust_pg_offset - Prepare Rx buffer for reuse + * @rx_buf: Rx buffer to adjust + * @size: Size of adjustment * - * The function will then update the page offset if necessary and return - * true if the buffer can be reused by the adapter. + * Update the offset within page so that Rx buf will be ready to be reused. + * For systems with PAGE_SIZE < 8192 this function will flip the page offset + * so the second half of page assigned to Rx buffer will be used, otherwise + * the offset is moved by the @size bytes */ -static bool ice_add_rx_frag(struct ice_rx_buf *rx_buf, - union ice_32b_rx_flex_desc *rx_desc, - struct sk_buff *skb) +static void +ice_rx_buf_adjust_pg_offset(struct ice_rx_buf *rx_buf, unsigned int size) { #if (PAGE_SIZE < 8192) - unsigned int truesize = ICE_RXBUF_2048; + /* flip page offset to other buffer */ + rx_buf->page_offset ^= size; #else - unsigned int last_offset = PAGE_SIZE - ICE_RXBUF_2048; - unsigned int truesize; -#endif /* PAGE_SIZE < 8192) */ - - struct page *page; - unsigned int size; - - size = le16_to_cpu(rx_desc->wb.pkt_len) & - ICE_RX_FLX_DESC_PKT_LEN_M; - - page = rx_buf->page; + /* move offset up to the next cache line */ + rx_buf->page_offset += size; +#endif +} +/** + * ice_can_reuse_rx_page - Determine if page can be reused for another Rx + * @rx_buf: buffer containing the page + * + * If page is reusable, we have a green light for calling ice_reuse_rx_page, + * which will assign the current buffer to the buffer that next_to_alloc is + * pointing to; otherwise, the DMA mapping needs to be destroyed and + * page freed + */ +static bool ice_can_reuse_rx_page(struct ice_rx_buf *rx_buf) +{ #if (PAGE_SIZE >= 8192) - truesize = ALIGN(size, L1_CACHE_BYTES); -#endif /* PAGE_SIZE >= 8192) */ - - /* will the data fit in the skb we allocated? if so, just - * copy it as it is pretty small anyway - */ - if (size <= ICE_RX_HDR_SIZE && !skb_is_nonlinear(skb)) { - unsigned char *va = page_address(page) + rx_buf->page_offset; - - memcpy(__skb_put(skb, size), va, ALIGN(size, sizeof(long))); - - /* page is not reserved, we can reuse buffer as-is */ - if (likely(!ice_page_is_reserved(page))) - return true; - - /* this page cannot be reused so discard it */ - __free_pages(page, 0); - return false; - } - - skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, page, - rx_buf->page_offset, size, truesize); + unsigned int last_offset = PAGE_SIZE - ICE_RXBUF_2048; +#endif + unsigned int pagecnt_bias = rx_buf->pagecnt_bias; + struct page *page = rx_buf->page; /* avoid re-using remote pages */ if (unlikely(ice_page_is_reserved(page))) @@ -559,27 +560,52 @@ static bool ice_add_rx_frag(struct ice_rx_buf *rx_buf, #if (PAGE_SIZE < 8192) /* if we are only owner of page we can reuse it */ - if (unlikely(page_count(page) != 1)) + if (unlikely((page_count(page) - pagecnt_bias) > 1)) return false; - - /* flip page offset to other buffer */ - rx_buf->page_offset ^= truesize; #else - /* move offset up to the next cache line */ - rx_buf->page_offset += truesize; - if (rx_buf->page_offset > last_offset) return false; #endif /* PAGE_SIZE < 8192) */ - /* Even if we own the page, we are not allowed to use atomic_set() - * This would break get_page_unless_zero() users. + /* If we have drained the page fragment pool we need to update + * the pagecnt_bias and page count so that we fully restock the + * number of references the driver holds. */ - get_page(rx_buf->page); + if (unlikely(pagecnt_bias == 1)) { + page_ref_add(page, USHRT_MAX - 1); + rx_buf->pagecnt_bias = USHRT_MAX; + } return true; } +/** + * ice_add_rx_frag - Add contents of Rx buffer to sk_buff as a frag + * @rx_buf: buffer containing page to add + * @skb: sk_buff to place the data into + * @size: packet length from rx_desc + * + * This function will add the data contained in rx_buf->page to the skb. + * It will just attach the page as a frag to the skb. + * The function will then update the page offset. + */ +static void +ice_add_rx_frag(struct ice_rx_buf *rx_buf, struct sk_buff *skb, + unsigned int size) +{ +#if (PAGE_SIZE >= 8192) + unsigned int truesize = SKB_DATA_ALIGN(size); +#else + unsigned int truesize = ICE_RXBUF_2048; +#endif + + skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, rx_buf->page, + rx_buf->page_offset, size, truesize); + + /* page is being used so we must update the page offset */ + ice_rx_buf_adjust_pg_offset(rx_buf, truesize); +} + /** * ice_reuse_rx_page - page flip buffer and store it back on the ring * @rx_ring: Rx descriptor ring to store buffers on @@ -587,8 +613,8 @@ static bool ice_add_rx_frag(struct ice_rx_buf *rx_buf, * * Synchronizes page for reuse by the adapter */ -static void ice_reuse_rx_page(struct ice_ring *rx_ring, - struct ice_rx_buf *old_buf) +static void +ice_reuse_rx_page(struct ice_ring *rx_ring, struct ice_rx_buf *old_buf) { u16 nta = rx_ring->next_to_alloc; struct ice_rx_buf *new_buf; @@ -599,121 +625,132 @@ static void ice_reuse_rx_page(struct ice_ring *rx_ring, nta++; rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta : 0; - /* transfer page from old buffer to new buffer */ - *new_buf = *old_buf; + /* Transfer page from old buffer to new buffer. + * Move each member individually to avoid possible store + * forwarding stalls and unnecessary copy of skb. + */ + new_buf->dma = old_buf->dma; + new_buf->page = old_buf->page; + new_buf->page_offset = old_buf->page_offset; + new_buf->pagecnt_bias = old_buf->pagecnt_bias; } /** - * ice_fetch_rx_buf - Allocate skb and populate it + * ice_get_rx_buf - Fetch Rx buffer and synchronize data for use * @rx_ring: Rx descriptor ring to transact packets on - * @rx_desc: descriptor containing info written by hardware + * @skb: skb to be used + * @size: size of buffer to add to skb * - * This function allocates an skb on the fly, and populates it with the page - * data from the current receive descriptor, taking care to set up the skb - * correctly, as well as handling calling the page recycle function if - * necessary. + * This function will pull an Rx buffer from the ring and synchronize it + * for use by the CPU. */ -static struct sk_buff *ice_fetch_rx_buf(struct ice_ring *rx_ring, - union ice_32b_rx_flex_desc *rx_desc) +static struct ice_rx_buf * +ice_get_rx_buf(struct ice_ring *rx_ring, struct sk_buff **skb, + const unsigned int size) { struct ice_rx_buf *rx_buf; - struct sk_buff *skb; - struct page *page; rx_buf = &rx_ring->rx_buf[rx_ring->next_to_clean]; - page = rx_buf->page; - prefetchw(page); + prefetchw(rx_buf->page); + *skb = rx_buf->skb; - skb = rx_buf->skb; + /* we are reusing so sync this buffer for CPU use */ + dma_sync_single_range_for_cpu(rx_ring->dev, rx_buf->dma, + rx_buf->page_offset, size, + DMA_FROM_DEVICE); - if (likely(!skb)) { - u8 *page_addr = page_address(page) + rx_buf->page_offset; + /* We have pulled a buffer for use, so decrement pagecnt_bias */ + rx_buf->pagecnt_bias--; - /* prefetch first cache line of first page */ - prefetch(page_addr); + return rx_buf; +} + +/** + * ice_construct_skb - Allocate skb and populate it + * @rx_ring: Rx descriptor ring to transact packets on + * @rx_buf: Rx buffer to pull data from + * @size: the length of the packet + * + * This function allocates an skb. It then populates it with the page + * data from the current receive descriptor, taking care to set up the + * skb correctly. + */ +static struct sk_buff * +ice_construct_skb(struct ice_ring *rx_ring, struct ice_rx_buf *rx_buf, + unsigned int size) +{ + void *va = page_address(rx_buf->page) + rx_buf->page_offset; + unsigned int headlen; + struct sk_buff *skb; + + /* prefetch first cache line of first page */ + prefetch(va); #if L1_CACHE_BYTES < 128 - prefetch((void *)(page_addr + L1_CACHE_BYTES)); + prefetch((u8 *)va + L1_CACHE_BYTES); #endif /* L1_CACHE_BYTES */ - /* allocate a skb to store the frags */ - skb = __napi_alloc_skb(&rx_ring->q_vector->napi, - ICE_RX_HDR_SIZE, - GFP_ATOMIC | __GFP_NOWARN); - if (unlikely(!skb)) { - rx_ring->rx_stats.alloc_buf_failed++; - return NULL; - } - - /* we will be copying header into skb->data in - * pskb_may_pull so it is in our interest to prefetch - * it now to avoid a possible cache miss - */ - prefetchw(skb->data); + /* allocate a skb to store the frags */ + skb = __napi_alloc_skb(&rx_ring->q_vector->napi, ICE_RX_HDR_SIZE, + GFP_ATOMIC | __GFP_NOWARN); + if (unlikely(!skb)) + return NULL; - skb_record_rx_queue(skb, rx_ring->q_index); - } else { - /* we are reusing so sync this buffer for CPU use */ - dma_sync_single_range_for_cpu(rx_ring->dev, rx_buf->dma, - rx_buf->page_offset, - ICE_RXBUF_2048, - DMA_FROM_DEVICE); + skb_record_rx_queue(skb, rx_ring->q_index); + /* Determine available headroom for copy */ + headlen = size; + if (headlen > ICE_RX_HDR_SIZE) + headlen = eth_get_headlen(skb->dev, va, ICE_RX_HDR_SIZE); - rx_buf->skb = NULL; - } + /* align pull length to size of long to optimize memcpy performance */ + memcpy(__skb_put(skb, headlen), va, ALIGN(headlen, sizeof(long))); - /* pull page into skb */ - if (ice_add_rx_frag(rx_buf, rx_desc, skb)) { - /* hand second half of page back to the ring */ - ice_reuse_rx_page(rx_ring, rx_buf); - rx_ring->rx_stats.page_reuse_count++; + /* if we exhaust the linear part then add what is left as a frag */ + size -= headlen; + if (size) { +#if (PAGE_SIZE >= 8192) + unsigned int truesize = SKB_DATA_ALIGN(size); +#else + unsigned int truesize = ICE_RXBUF_2048; +#endif + skb_add_rx_frag(skb, 0, rx_buf->page, + rx_buf->page_offset + headlen, size, truesize); + /* buffer is used by skb, update page_offset */ + ice_rx_buf_adjust_pg_offset(rx_buf, truesize); } else { - /* we are not reusing the buffer so unmap it */ - dma_unmap_page(rx_ring->dev, rx_buf->dma, PAGE_SIZE, - DMA_FROM_DEVICE); + /* buffer is unused, reset bias back to rx_buf; data was copied + * onto skb's linear part so there's no need for adjusting + * page offset and we can reuse this buffer as-is + */ + rx_buf->pagecnt_bias++; } - /* clear contents of buffer_info */ - rx_buf->page = NULL; - return skb; } /** - * ice_pull_tail - ice specific version of skb_pull_tail - * @skb: pointer to current skb being adjusted + * ice_put_rx_buf - Clean up used buffer and either recycle or free + * @rx_ring: Rx descriptor ring to transact packets on + * @rx_buf: Rx buffer to pull data from * - * This function is an ice specific version of __pskb_pull_tail. The - * main difference between this version and the original function is that - * this function can make several assumptions about the state of things - * that allow for significant optimizations versus the standard function. - * As a result we can do things like drop a frag and maintain an accurate - * truesize for the skb. + * This function will clean up the contents of the rx_buf. It will + * either recycle the buffer or unmap it and free the associated resources. */ -static void ice_pull_tail(struct sk_buff *skb) +static void ice_put_rx_buf(struct ice_ring *rx_ring, struct ice_rx_buf *rx_buf) { - struct skb_frag_struct *frag = &skb_shinfo(skb)->frags[0]; - unsigned int pull_len; - unsigned char *va; - - /* it is valid to use page_address instead of kmap since we are - * working with pages allocated out of the lomem pool per - * alloc_page(GFP_ATOMIC) - */ - va = skb_frag_address(frag); - - /* we need the header to contain the greater of either ETH_HLEN or - * 60 bytes if the skb->len is less than 60 for skb_pad. - */ - pull_len = eth_get_headlen(va, ICE_RX_HDR_SIZE); - - /* align pull length to size of long to optimize memcpy performance */ - skb_copy_to_linear_data(skb, va, ALIGN(pull_len, sizeof(long))); + /* hand second half of page back to the ring */ + if (ice_can_reuse_rx_page(rx_buf)) { + ice_reuse_rx_page(rx_ring, rx_buf); + rx_ring->rx_stats.page_reuse_count++; + } else { + /* we are not reusing the buffer so unmap it */ + dma_unmap_page_attrs(rx_ring->dev, rx_buf->dma, PAGE_SIZE, + DMA_FROM_DEVICE, ICE_RX_DMA_ATTR); + __page_frag_cache_drain(rx_buf->page, rx_buf->pagecnt_bias); + } - /* update all of the pointers */ - skb_frag_size_sub(frag, pull_len); - frag->page_offset += pull_len; - skb->data_len -= pull_len; - skb->tail += pull_len; + /* clear contents of buffer_info */ + rx_buf->page = NULL; + rx_buf->skb = NULL; } /** @@ -730,10 +767,6 @@ static void ice_pull_tail(struct sk_buff *skb) */ static bool ice_cleanup_headers(struct sk_buff *skb) { - /* place header in linear portion of buffer */ - if (skb_is_nonlinear(skb)) - ice_pull_tail(skb); - /* if eth_skb_pad returns an error the skb was freed */ if (eth_skb_pad(skb)) return true; @@ -751,8 +784,8 @@ static bool ice_cleanup_headers(struct sk_buff *skb) * The status_error_len doesn't need to be shifted because it begins * at offset zero. */ -static bool ice_test_staterr(union ice_32b_rx_flex_desc *rx_desc, - const u16 stat_err_bits) +static bool +ice_test_staterr(union ice_32b_rx_flex_desc *rx_desc, const u16 stat_err_bits) { return !!(rx_desc->wb.status_error0 & cpu_to_le16(stat_err_bits)); @@ -769,9 +802,9 @@ static bool ice_test_staterr(union ice_32b_rx_flex_desc *rx_desc, * sk_buff in the next buffer to be chained and return true indicating * that this is in fact a non-EOP buffer. */ -static bool ice_is_non_eop(struct ice_ring *rx_ring, - union ice_32b_rx_flex_desc *rx_desc, - struct sk_buff *skb) +static bool +ice_is_non_eop(struct ice_ring *rx_ring, union ice_32b_rx_flex_desc *rx_desc, + struct sk_buff *skb) { u32 ntc = rx_ring->next_to_clean + 1; @@ -838,8 +871,9 @@ ice_rx_hash(struct ice_ring *rx_ring, union ice_32b_rx_flex_desc *rx_desc, * * skb->protocol must be set before this function is called */ -static void ice_rx_csum(struct ice_vsi *vsi, struct sk_buff *skb, - union ice_32b_rx_flex_desc *rx_desc, u8 ptype) +static void +ice_rx_csum(struct ice_vsi *vsi, struct sk_buff *skb, + union ice_32b_rx_flex_desc *rx_desc, u8 ptype) { struct ice_rx_ptype_decoded decoded; u32 rx_error, rx_status; @@ -909,9 +943,10 @@ checksum_fail: * order to populate the hash, checksum, VLAN, protocol, and * other fields within the skb. */ -static void ice_process_skb_fields(struct ice_ring *rx_ring, - union ice_32b_rx_flex_desc *rx_desc, - struct sk_buff *skb, u8 ptype) +static void +ice_process_skb_fields(struct ice_ring *rx_ring, + union ice_32b_rx_flex_desc *rx_desc, + struct sk_buff *skb, u8 ptype) { ice_rx_hash(rx_ring, rx_desc, skb, ptype); @@ -925,18 +960,17 @@ static void ice_process_skb_fields(struct ice_ring *rx_ring, * ice_receive_skb - Send a completed packet up the stack * @rx_ring: Rx ring in play * @skb: packet to send up - * @vlan_tag: vlan tag for packet + * @vlan_tag: VLAN tag for packet * * This function sends the completed packet (via. skb) up the stack using - * gro receive functions (with/without vlan tag) + * gro receive functions (with/without VLAN tag) */ -static void ice_receive_skb(struct ice_ring *rx_ring, struct sk_buff *skb, - u16 vlan_tag) +static void +ice_receive_skb(struct ice_ring *rx_ring, struct sk_buff *skb, u16 vlan_tag) { if ((rx_ring->netdev->features & NETIF_F_HW_VLAN_CTAG_RX) && - (vlan_tag & VLAN_VID_MASK)) { + (vlan_tag & VLAN_VID_MASK)) __vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q), vlan_tag); - } napi_gro_receive(&rx_ring->q_vector->napi, skb); } @@ -958,10 +992,12 @@ static int ice_clean_rx_irq(struct ice_ring *rx_ring, int budget) u16 cleaned_count = ICE_DESC_UNUSED(rx_ring); bool failure = false; - /* start the loop to process RX packets bounded by 'budget' */ + /* start the loop to process Rx packets bounded by 'budget' */ while (likely(total_rx_pkts < (unsigned int)budget)) { union ice_32b_rx_flex_desc *rx_desc; + struct ice_rx_buf *rx_buf; struct sk_buff *skb; + unsigned int size; u16 stat_err_bits; u16 vlan_tag = 0; u8 rx_ptype; @@ -973,7 +1009,7 @@ static int ice_clean_rx_irq(struct ice_ring *rx_ring, int budget) cleaned_count = 0; } - /* get the RX desc from RX ring based on 'next_to_clean' */ + /* get the Rx desc from Rx ring based on 'next_to_clean' */ rx_desc = ICE_RX_DESC(rx_ring, rx_ring->next_to_clean); /* status_error_len will always be zero for unused descriptors @@ -991,11 +1027,24 @@ static int ice_clean_rx_irq(struct ice_ring *rx_ring, int budget) */ dma_rmb(); + size = le16_to_cpu(rx_desc->wb.pkt_len) & + ICE_RX_FLX_DESC_PKT_LEN_M; + + rx_buf = ice_get_rx_buf(rx_ring, &skb, size); /* allocate (if needed) and populate skb */ - skb = ice_fetch_rx_buf(rx_ring, rx_desc); - if (!skb) + if (skb) + ice_add_rx_frag(rx_buf, skb, size); + else + skb = ice_construct_skb(rx_ring, rx_buf, size); + + /* exit if we failed to retrieve a buffer */ + if (!skb) { + rx_ring->rx_stats.alloc_buf_failed++; + rx_buf->pagecnt_bias++; break; + } + ice_put_rx_buf(rx_ring, rx_buf); cleaned_count++; /* skip if it is NOP desc */ @@ -1048,18 +1097,248 @@ static int ice_clean_rx_irq(struct ice_ring *rx_ring, int budget) return failure ? budget : (int)total_rx_pkts; } +/** + * ice_adjust_itr_by_size_and_speed - Adjust ITR based on current traffic + * @port_info: port_info structure containing the current link speed + * @avg_pkt_size: average size of Tx or Rx packets based on clean routine + * @itr: itr value to update + * + * Calculate how big of an increment should be applied to the ITR value passed + * in based on wmem_default, SKB overhead, Ethernet overhead, and the current + * link speed. + * + * The following is a calculation derived from: + * wmem_default / (size + overhead) = desired_pkts_per_int + * rate / bits_per_byte / (size + Ethernet overhead) = pkt_rate + * (desired_pkt_rate / pkt_rate) * usecs_per_sec = ITR value + * + * Assuming wmem_default is 212992 and overhead is 640 bytes per + * packet, (256 skb, 64 headroom, 320 shared info), we can reduce the + * formula down to: + * + * wmem_default * bits_per_byte * usecs_per_sec pkt_size + 24 + * ITR = -------------------------------------------- * -------------- + * rate pkt_size + 640 + */ +static unsigned int +ice_adjust_itr_by_size_and_speed(struct ice_port_info *port_info, + unsigned int avg_pkt_size, + unsigned int itr) +{ + switch (port_info->phy.link_info.link_speed) { + case ICE_AQ_LINK_SPEED_100GB: + itr += DIV_ROUND_UP(17 * (avg_pkt_size + 24), + avg_pkt_size + 640); + break; + case ICE_AQ_LINK_SPEED_50GB: + itr += DIV_ROUND_UP(34 * (avg_pkt_size + 24), + avg_pkt_size + 640); + break; + case ICE_AQ_LINK_SPEED_40GB: + itr += DIV_ROUND_UP(43 * (avg_pkt_size + 24), + avg_pkt_size + 640); + break; + case ICE_AQ_LINK_SPEED_25GB: + itr += DIV_ROUND_UP(68 * (avg_pkt_size + 24), + avg_pkt_size + 640); + break; + case ICE_AQ_LINK_SPEED_20GB: + itr += DIV_ROUND_UP(85 * (avg_pkt_size + 24), + avg_pkt_size + 640); + break; + case ICE_AQ_LINK_SPEED_10GB: + /* fall through */ + default: + itr += DIV_ROUND_UP(170 * (avg_pkt_size + 24), + avg_pkt_size + 640); + break; + } + + if ((itr & ICE_ITR_MASK) > ICE_ITR_ADAPTIVE_MAX_USECS) { + itr &= ICE_ITR_ADAPTIVE_LATENCY; + itr += ICE_ITR_ADAPTIVE_MAX_USECS; + } + + return itr; +} + +/** + * ice_update_itr - update the adaptive ITR value based on statistics + * @q_vector: structure containing interrupt and ring information + * @rc: structure containing ring performance data + * + * Stores a new ITR value based on packets and byte + * counts during the last interrupt. The advantage of per interrupt + * computation is faster updates and more accurate ITR for the current + * traffic pattern. Constants in this function were computed + * based on theoretical maximum wire speed and thresholds were set based + * on testing data as well as attempting to minimize response time + * while increasing bulk throughput. + */ +static void +ice_update_itr(struct ice_q_vector *q_vector, struct ice_ring_container *rc) +{ + unsigned long next_update = jiffies; + unsigned int packets, bytes, itr; + bool container_is_rx; + + if (!rc->ring || !ITR_IS_DYNAMIC(rc->itr_setting)) + return; + + /* If itr_countdown is set it means we programmed an ITR within + * the last 4 interrupt cycles. This has a side effect of us + * potentially firing an early interrupt. In order to work around + * this we need to throw out any data received for a few + * interrupts following the update. + */ + if (q_vector->itr_countdown) { + itr = rc->target_itr; + goto clear_counts; + } + + container_is_rx = (&q_vector->rx == rc); + /* For Rx we want to push the delay up and default to low latency. + * for Tx we want to pull the delay down and default to high latency. + */ + itr = container_is_rx ? + ICE_ITR_ADAPTIVE_MIN_USECS | ICE_ITR_ADAPTIVE_LATENCY : + ICE_ITR_ADAPTIVE_MAX_USECS | ICE_ITR_ADAPTIVE_LATENCY; + + /* If we didn't update within up to 1 - 2 jiffies we can assume + * that either packets are coming in so slow there hasn't been + * any work, or that there is so much work that NAPI is dealing + * with interrupt moderation and we don't need to do anything. + */ + if (time_after(next_update, rc->next_update)) + goto clear_counts; + + packets = rc->total_pkts; + bytes = rc->total_bytes; + + if (container_is_rx) { + /* If Rx there are 1 to 4 packets and bytes are less than + * 9000 assume insufficient data to use bulk rate limiting + * approach unless Tx is already in bulk rate limiting. We + * are likely latency driven. + */ + if (packets && packets < 4 && bytes < 9000 && + (q_vector->tx.target_itr & ICE_ITR_ADAPTIVE_LATENCY)) { + itr = ICE_ITR_ADAPTIVE_LATENCY; + goto adjust_by_size_and_speed; + } + } else if (packets < 4) { + /* If we have Tx and Rx ITR maxed and Tx ITR is running in + * bulk mode and we are receiving 4 or fewer packets just + * reset the ITR_ADAPTIVE_LATENCY bit for latency mode so + * that the Rx can relax. + */ + if (rc->target_itr == ICE_ITR_ADAPTIVE_MAX_USECS && + (q_vector->rx.target_itr & ICE_ITR_MASK) == + ICE_ITR_ADAPTIVE_MAX_USECS) + goto clear_counts; + } else if (packets > 32) { + /* If we have processed over 32 packets in a single interrupt + * for Tx assume we need to switch over to "bulk" mode. + */ + rc->target_itr &= ~ICE_ITR_ADAPTIVE_LATENCY; + } + + /* We have no packets to actually measure against. This means + * either one of the other queues on this vector is active or + * we are a Tx queue doing TSO with too high of an interrupt rate. + * + * Between 4 and 56 we can assume that our current interrupt delay + * is only slightly too low. As such we should increase it by a small + * fixed amount. + */ + if (packets < 56) { + itr = rc->target_itr + ICE_ITR_ADAPTIVE_MIN_INC; + if ((itr & ICE_ITR_MASK) > ICE_ITR_ADAPTIVE_MAX_USECS) { + itr &= ICE_ITR_ADAPTIVE_LATENCY; + itr += ICE_ITR_ADAPTIVE_MAX_USECS; + } + goto clear_counts; + } + + if (packets <= 256) { + itr = min(q_vector->tx.current_itr, q_vector->rx.current_itr); + itr &= ICE_ITR_MASK; + + /* Between 56 and 112 is our "goldilocks" zone where we are + * working out "just right". Just report that our current + * ITR is good for us. + */ + if (packets <= 112) + goto clear_counts; + + /* If packet count is 128 or greater we are likely looking + * at a slight overrun of the delay we want. Try halving + * our delay to see if that will cut the number of packets + * in half per interrupt. + */ + itr >>= 1; + itr &= ICE_ITR_MASK; + if (itr < ICE_ITR_ADAPTIVE_MIN_USECS) + itr = ICE_ITR_ADAPTIVE_MIN_USECS; + + goto clear_counts; + } + + /* The paths below assume we are dealing with a bulk ITR since + * number of packets is greater than 256. We are just going to have + * to compute a value and try to bring the count under control, + * though for smaller packet sizes there isn't much we can do as + * NAPI polling will likely be kicking in sooner rather than later. + */ + itr = ICE_ITR_ADAPTIVE_BULK; + +adjust_by_size_and_speed: + + /* based on checks above packets cannot be 0 so division is safe */ + itr = ice_adjust_itr_by_size_and_speed(q_vector->vsi->port_info, + bytes / packets, itr); + +clear_counts: + /* write back value */ + rc->target_itr = itr; + + /* next update should occur within next jiffy */ + rc->next_update = next_update + 1; + + rc->total_bytes = 0; + rc->total_pkts = 0; +} + /** * ice_buildreg_itr - build value for writing to the GLINT_DYN_CTL register * @itr_idx: interrupt throttling index - * @reg_itr: interrupt throttling value adjusted based on ITR granularity + * @itr: interrupt throttling value in usecs */ -static u32 ice_buildreg_itr(int itr_idx, u16 reg_itr) +static u32 ice_buildreg_itr(u16 itr_idx, u16 itr) { + /* The itr value is reported in microseconds, and the register value is + * recorded in 2 microsecond units. For this reason we only need to + * shift by the GLINT_DYN_CTL_INTERVAL_S - ICE_ITR_GRAN_S to apply this + * granularity as a shift instead of division. The mask makes sure the + * ITR value is never odd so we don't accidentally write into the field + * prior to the ITR field. + */ + itr &= ICE_ITR_MASK; + return GLINT_DYN_CTL_INTENA_M | GLINT_DYN_CTL_CLEARPBA_M | (itr_idx << GLINT_DYN_CTL_ITR_INDX_S) | - (reg_itr << GLINT_DYN_CTL_INTERVAL_S); + (itr << (GLINT_DYN_CTL_INTERVAL_S - ICE_ITR_GRAN_S)); } +/* The act of updating the ITR will cause it to immediately trigger. In order + * to prevent this from throwing off adaptive update statistics we defer the + * update so that it can only happen so often. So after either Tx or Rx are + * updated we make the adaptive scheme wait until either the ITR completely + * expires via the next_update expiration or we have been through at least + * 3 interrupts. + */ +#define ITR_COUNTDOWN_START 3 + /** * ice_update_ena_itr - Update ITR and re-enable MSIX interrupt * @vsi: the VSI associated with the q_vector @@ -1068,10 +1347,14 @@ static u32 ice_buildreg_itr(int itr_idx, u16 reg_itr) static void ice_update_ena_itr(struct ice_vsi *vsi, struct ice_q_vector *q_vector) { - struct ice_hw *hw = &vsi->back->hw; - struct ice_ring_container *rc; + struct ice_ring_container *tx = &q_vector->tx; + struct ice_ring_container *rx = &q_vector->rx; u32 itr_val; + /* This will do nothing if dynamic updates are not enabled */ + ice_update_itr(q_vector, tx); + ice_update_itr(q_vector, rx); + /* This block of logic allows us to get away with only updating * one ITR value with each interrupt. The idea is to perform a * pseudo-lazy update with the following criteria. @@ -1080,35 +1363,36 @@ ice_update_ena_itr(struct ice_vsi *vsi, struct ice_q_vector *q_vector) * 2. If we must reduce an ITR that is given highest priority. * 3. We then give priority to increasing ITR based on amount. */ - if (q_vector->rx.target_itr < q_vector->rx.current_itr) { - rc = &q_vector->rx; + if (rx->target_itr < rx->current_itr) { /* Rx ITR needs to be reduced, this is highest priority */ - itr_val = ice_buildreg_itr(rc->itr_idx, rc->target_itr); - rc->current_itr = rc->target_itr; - } else if ((q_vector->tx.target_itr < q_vector->tx.current_itr) || - ((q_vector->rx.target_itr - q_vector->rx.current_itr) < - (q_vector->tx.target_itr - q_vector->tx.current_itr))) { - rc = &q_vector->tx; + itr_val = ice_buildreg_itr(rx->itr_idx, rx->target_itr); + rx->current_itr = rx->target_itr; + q_vector->itr_countdown = ITR_COUNTDOWN_START; + } else if ((tx->target_itr < tx->current_itr) || + ((rx->target_itr - rx->current_itr) < + (tx->target_itr - tx->current_itr))) { /* Tx ITR needs to be reduced, this is second priority * Tx ITR needs to be increased more than Rx, fourth priority */ - itr_val = ice_buildreg_itr(rc->itr_idx, rc->target_itr); - rc->current_itr = rc->target_itr; - } else if (q_vector->rx.current_itr != q_vector->rx.target_itr) { - rc = &q_vector->rx; + itr_val = ice_buildreg_itr(tx->itr_idx, tx->target_itr); + tx->current_itr = tx->target_itr; + q_vector->itr_countdown = ITR_COUNTDOWN_START; + } else if (rx->current_itr != rx->target_itr) { /* Rx ITR needs to be increased, third priority */ - itr_val = ice_buildreg_itr(rc->itr_idx, rc->target_itr); - rc->current_itr = rc->target_itr; + itr_val = ice_buildreg_itr(rx->itr_idx, rx->target_itr); + rx->current_itr = rx->target_itr; + q_vector->itr_countdown = ITR_COUNTDOWN_START; } else { /* Still have to re-enable the interrupts */ itr_val = ice_buildreg_itr(ICE_ITR_NONE, 0); + if (q_vector->itr_countdown) + q_vector->itr_countdown--; } - if (!test_bit(__ICE_DOWN, vsi->state)) { - int vector = vsi->hw_base_vector + q_vector->v_idx; - - wr32(hw, GLINT_DYN_CTL(vector), itr_val); - } + if (!test_bit(__ICE_DOWN, vsi->state)) + wr32(&vsi->back->hw, + GLINT_DYN_CTL(q_vector->reg_idx), + itr_val); } /** @@ -1354,7 +1638,7 @@ ice_tx_map(struct ice_ring *tx_ring, struct ice_tx_buf *first, ice_maybe_stop_tx(tx_ring, DESC_NEEDED); /* notify HW of packet */ - if (netif_xmit_stopped(txring_txq(tx_ring)) || !skb->xmit_more) { + if (netif_xmit_stopped(txring_txq(tx_ring)) || !netdev_xmit_more()) { writel(i, tx_ring->tail); } @@ -1475,7 +1759,7 @@ int ice_tx_csum(struct ice_tx_buf *first, struct ice_tx_offload_params *off) } /** - * ice_tx_prepare_vlan_flags - prepare generic TX VLAN tagging flags for HW + * ice_tx_prepare_vlan_flags - prepare generic Tx VLAN tagging flags for HW * @tx_ring: ring to send buffer on * @first: pointer to struct ice_tx_buf * @@ -1501,7 +1785,7 @@ ice_tx_prepare_vlan_flags(struct ice_ring *tx_ring, struct ice_tx_buf *first) * to the encapsulated ethertype. */ skb->protocol = vlan_get_protocol(skb); - goto out; + return 0; } /* if we have a HW VLAN tag being added, default to the HW one */ @@ -1523,8 +1807,7 @@ ice_tx_prepare_vlan_flags(struct ice_ring *tx_ring, struct ice_tx_buf *first) first->tx_flags |= ICE_TX_FLAGS_SW_VLAN; } -out: - return 0; + return ice_tx_prepare_vlan_flags_dcb(tx_ring, first); } /** @@ -1561,6 +1844,7 @@ int ice_tso(struct ice_tx_buf *first, struct ice_tx_offload_params *off) if (err < 0) return err; + /* cppcheck-suppress unreadVariable */ ip.hdr = skb_network_header(skb); l4.hdr = skb_transport_header(skb); diff --git a/drivers/net/ethernet/intel/ice/ice_txrx.h b/drivers/net/ethernet/intel/ice/ice_txrx.h index fc358ea81816fb86fd8db1ce9b3ebce8531ce13d..66e05032ee56709a96f98785a195becd22670f11 100644 --- a/drivers/net/ethernet/intel/ice/ice_txrx.h +++ b/drivers/net/ethernet/intel/ice/ice_txrx.h @@ -45,8 +45,13 @@ #define ICE_TX_FLAGS_HW_VLAN BIT(1) #define ICE_TX_FLAGS_SW_VLAN BIT(2) #define ICE_TX_FLAGS_VLAN_M 0xffff0000 +#define ICE_TX_FLAGS_VLAN_PR_M 0xe0000000 +#define ICE_TX_FLAGS_VLAN_PR_S 29 #define ICE_TX_FLAGS_VLAN_S 16 +#define ICE_RX_DMA_ATTR \ + (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING) + struct ice_tx_buf { struct ice_tx_desc *next_to_watch; struct sk_buff *skb; @@ -73,6 +78,7 @@ struct ice_rx_buf { dma_addr_t dma; struct page *page; unsigned int page_offset; + u16 pagecnt_bias; }; struct ice_q_stats { @@ -124,11 +130,19 @@ enum ice_rx_dtype { #define ICE_ITR_DYNAMIC 0x8000 /* used as flag for itr_setting */ #define ITR_IS_DYNAMIC(setting) (!!((setting) & ICE_ITR_DYNAMIC)) #define ITR_TO_REG(setting) ((setting) & ~ICE_ITR_DYNAMIC) -#define ICE_ITR_GRAN_S 1 /* Assume ITR granularity is 2us */ +#define ICE_ITR_GRAN_S 1 /* ITR granularity is always 2us */ +#define ICE_ITR_GRAN_US BIT(ICE_ITR_GRAN_S) #define ICE_ITR_MASK 0x1FFE /* ITR register value alignment mask */ #define ITR_REG_ALIGN(setting) __ALIGN_MASK(setting, ~ICE_ITR_MASK) +#define ICE_ITR_ADAPTIVE_MIN_INC 0x0002 +#define ICE_ITR_ADAPTIVE_MIN_USECS 0x0002 +#define ICE_ITR_ADAPTIVE_MAX_USECS 0x00FA +#define ICE_ITR_ADAPTIVE_LATENCY 0x8000 +#define ICE_ITR_ADAPTIVE_BULK 0x0000 + #define ICE_DFLT_INTRL 0 +#define ICE_MAX_INTRL 236 /* Legacy or Advanced Mode Queue */ #define ICE_TX_ADVANCED 0 @@ -149,6 +163,9 @@ struct ice_ring { }; u16 q_index; /* Queue number of ring */ u32 txq_teid; /* Added Tx queue TEID */ +#ifdef CONFIG_DCB + u8 dcb_tc; /* Traffic class of ring */ +#endif /* CONFIG_DCB */ u16 count; /* Number of descriptors */ u16 reg_idx; /* HW register index of the ring */ @@ -173,21 +190,13 @@ struct ice_ring { u16 next_to_alloc; } ____cacheline_internodealigned_in_smp; -enum ice_latency_range { - ICE_LOWEST_LATENCY = 0, - ICE_LOW_LATENCY = 1, - ICE_BULK_LATENCY = 2, - ICE_ULTRA_LATENCY = 3, -}; - struct ice_ring_container { /* head of linked-list of rings */ struct ice_ring *ring; unsigned long next_update; /* jiffies value of next queue update */ unsigned int total_bytes; /* total bytes processed this int */ unsigned int total_pkts; /* total packets processed this int */ - enum ice_latency_range latency_range; - int itr_idx; /* index in the interrupt vector */ + u16 itr_idx; /* index in the interrupt vector */ u16 target_itr; /* value in usecs divided by the hw->itr_gran */ u16 current_itr; /* value in usecs divided by the hw->itr_gran */ /* high bit set means dynamic ITR, rest is used to store user diff --git a/drivers/net/ethernet/intel/ice/ice_type.h b/drivers/net/ethernet/intel/ice/ice_type.h index 17086d5b5c33b7f9b25ff14b0cbe035b8ce98fef..a862af4cbf78b9383c1ae7343567089150d5e4e7 100644 --- a/drivers/net/ethernet/intel/ice/ice_type.h +++ b/drivers/net/ethernet/intel/ice/ice_type.h @@ -24,6 +24,7 @@ static inline bool ice_is_tc_ena(u8 bitmap, u8 tc) /* debug masks - set these bits in hw->debug_mask to control output */ #define ICE_DBG_INIT BIT_ULL(1) #define ICE_DBG_LINK BIT_ULL(4) +#define ICE_DBG_PHY BIT_ULL(5) #define ICE_DBG_QCTX BIT_ULL(6) #define ICE_DBG_NVM BIT_ULL(7) #define ICE_DBG_LAN BIT_ULL(8) @@ -106,7 +107,7 @@ struct ice_link_status { }; /* Different reset sources for which a disable queue AQ call has to be made in - * order to clean the TX scheduler as a part of the reset + * order to clean the Tx scheduler as a part of the reset */ enum ice_disq_rst_src { ICE_NO_RESET = 0, @@ -128,11 +129,11 @@ struct ice_phy_info { struct ice_hw_common_caps { u32 valid_functions; - /* TX/RX queues */ - u16 num_rxq; /* Number/Total RX queues */ - u16 rxq_first_id; /* First queue ID for RX queues */ - u16 num_txq; /* Number/Total TX queues */ - u16 txq_first_id; /* First queue ID for TX queues */ + /* Tx/Rx queues */ + u16 num_rxq; /* Number/Total Rx queues */ + u16 rxq_first_id; /* First queue ID for Rx queues */ + u16 num_txq; /* Number/Total Tx queues */ + u16 txq_first_id; /* First queue ID for Tx queues */ /* MSI-X vectors */ u16 num_msix_vectors; @@ -147,6 +148,8 @@ struct ice_hw_common_caps { /* RSS related capabilities */ u16 rss_table_size; /* 512 for PFs and 64 for VFs */ u8 rss_table_entry_width; /* RSS Entry width in bits */ + + u8 dcb; }; /* Function specific capabilities */ @@ -209,12 +212,17 @@ struct ice_nvm_info { #define ICE_MAX_TRAFFIC_CLASS 8 #define ICE_TXSCHED_MAX_BRANCHES ICE_MAX_TRAFFIC_CLASS +#define ice_for_each_traffic_class(_i) \ + for ((_i) = 0; (_i) < ICE_MAX_TRAFFIC_CLASS; (_i)++) + +#define ICE_INVAL_TEID 0xFFFFFFFF + struct ice_sched_node { struct ice_sched_node *parent; struct ice_sched_node *sibling; /* next sibling in the same layer */ struct ice_sched_node **children; struct ice_aqc_txsched_elem_data info; - u32 agg_id; /* aggregator group id */ + u32 agg_id; /* aggregator group ID */ u16 vsi_handle; u8 in_use; /* suspended or in use */ u8 tx_sched_layer; /* Logical Layer (1-9) */ @@ -241,13 +249,12 @@ enum ice_agg_type { #define ICE_SCHED_DFLT_RL_PROF_ID 0 #define ICE_SCHED_DFLT_BW_WT 1 -/* vsi type list entry to locate corresponding vsi/ag nodes */ +/* VSI type list entry to locate corresponding VSI/ag nodes */ struct ice_sched_vsi_info { struct ice_sched_node *vsi_node[ICE_MAX_TRAFFIC_CLASS]; struct ice_sched_node *ag_node[ICE_MAX_TRAFFIC_CLASS]; struct list_head list_entry; u16 max_lanq[ICE_MAX_TRAFFIC_CLASS]; - u16 vsi_id; }; /* driver defines the policy */ @@ -257,15 +264,70 @@ struct ice_sched_tx_policy { u8 rdma_ena; }; +/* CEE or IEEE 802.1Qaz ETS Configuration data */ +struct ice_dcb_ets_cfg { + u8 willing; + u8 cbs; + u8 maxtcs; + u8 prio_table[ICE_MAX_TRAFFIC_CLASS]; + u8 tcbwtable[ICE_MAX_TRAFFIC_CLASS]; + u8 tsatable[ICE_MAX_TRAFFIC_CLASS]; +}; + +/* CEE or IEEE 802.1Qaz PFC Configuration data */ +struct ice_dcb_pfc_cfg { + u8 willing; + u8 mbc; + u8 pfccap; + u8 pfcena; +}; + +/* CEE or IEEE 802.1Qaz Application Priority data */ +struct ice_dcb_app_priority_table { + u16 prot_id; + u8 priority; + u8 selector; +}; + +#define ICE_MAX_USER_PRIORITY 8 +#define ICE_DCBX_MAX_APPS 32 +#define ICE_LLDPDU_SIZE 1500 +#define ICE_TLV_STATUS_OPER 0x1 +#define ICE_TLV_STATUS_SYNC 0x2 +#define ICE_TLV_STATUS_ERR 0x4 +#define ICE_APP_PROT_ID_FCOE 0x8906 +#define ICE_APP_PROT_ID_ISCSI 0x0cbc +#define ICE_APP_PROT_ID_FIP 0x8914 +#define ICE_APP_SEL_ETHTYPE 0x1 +#define ICE_APP_SEL_TCPIP 0x2 +#define ICE_CEE_APP_SEL_ETHTYPE 0x0 +#define ICE_CEE_APP_SEL_TCPIP 0x1 + +struct ice_dcbx_cfg { + u32 numapps; + u32 tlv_status; /* CEE mode TLV status */ + struct ice_dcb_ets_cfg etscfg; + struct ice_dcb_ets_cfg etsrec; + struct ice_dcb_pfc_cfg pfc; + struct ice_dcb_app_priority_table app[ICE_DCBX_MAX_APPS]; + u8 dcbx_mode; +#define ICE_DCBX_MODE_CEE 0x1 +#define ICE_DCBX_MODE_IEEE 0x2 + u8 app_mode; +#define ICE_DCBX_APPS_NON_WILLING 0x1 +}; + struct ice_port_info { struct ice_sched_node *root; /* Root Node per Port */ - struct ice_hw *hw; /* back pointer to hw instance */ + struct ice_hw *hw; /* back pointer to HW instance */ u32 last_node_teid; /* scheduler last node info */ u16 sw_id; /* Initial switch ID belongs to port */ u16 pf_vf_num; u8 port_state; #define ICE_SCHED_PORT_STATE_INIT 0x0 #define ICE_SCHED_PORT_STATE_READY 0x1 + u8 lport; +#define ICE_LPORT_MASK 0xff u16 dflt_tx_vsi_rule_id; u16 dflt_tx_vsi_num; u16 dflt_rx_vsi_rule_id; @@ -274,9 +336,14 @@ struct ice_port_info { struct ice_mac_info mac; struct ice_phy_info phy; struct mutex sched_lock; /* protect access to TXSched tree */ - u8 lport; -#define ICE_LPORT_MASK 0xff - u8 is_vf; + struct ice_dcbx_cfg local_dcbx_cfg; /* Oper/Local Cfg */ + /* DCBX info */ + struct ice_dcbx_cfg remote_dcbx_cfg; /* Peer Cfg */ + struct ice_dcbx_cfg desired_dcbx_cfg; /* CEE Desired Cfg */ + /* LLDP/DCBX Status */ + u8 dcbx_status:3; /* see ICE_DCBX_STATUS_DIS */ + u8 is_sw_lldp:1; + u8 is_vf:1; }; struct ice_switch_info { @@ -320,7 +387,7 @@ struct ice_hw { u8 pf_id; /* device profile info */ - /* TX Scheduler values */ + /* Tx Scheduler values */ u16 num_tx_sched_layers; u16 num_tx_sched_phys_layers; u8 flattened_layers; @@ -331,7 +398,7 @@ struct ice_hw { struct ice_vsi_ctx *vsi_ctx[ICE_MAX_VSI]; u8 evb_veb; /* true for VEB, false for VEPA */ - u8 reset_ongoing; /* true if hw is in reset, false otherwise */ + u8 reset_ongoing; /* true if HW is in reset, false otherwise */ struct ice_bus_info bus; struct ice_nvm_info nvm; struct ice_hw_dev_caps dev_caps; /* device capabilities */ @@ -410,6 +477,11 @@ struct ice_hw_port_stats { u64 link_xoff_rx; /* lxoffrxc */ u64 link_xon_tx; /* lxontxc */ u64 link_xoff_tx; /* lxofftxc */ + u64 priority_xon_rx[8]; /* pxonrxc[8] */ + u64 priority_xoff_rx[8]; /* pxoffrxc[8] */ + u64 priority_xon_tx[8]; /* pxontxc[8] */ + u64 priority_xoff_tx[8]; /* pxofftxc[8] */ + u64 priority_xon_2_xoff[8]; /* pxon2offc[8] */ u64 rx_size_64; /* prc64 */ u64 rx_size_127; /* prc127 */ u64 rx_size_255; /* prc255 */ diff --git a/drivers/net/ethernet/intel/ice/ice_virtchnl_pf.c b/drivers/net/ethernet/intel/ice/ice_virtchnl_pf.c index 57155b4a59dc1f54e2756104ddbf524750fe91d2..a805cbdd69be16aa1d96b8008de4f4a3ec9836f8 100644 --- a/drivers/net/ethernet/intel/ice/ice_virtchnl_pf.c +++ b/drivers/net/ethernet/intel/ice/ice_virtchnl_pf.c @@ -4,6 +4,37 @@ #include "ice.h" #include "ice_lib.h" +/** + * ice_err_to_virt err - translate errors for VF return code + * @ice_err: error return code + */ +static enum virtchnl_status_code ice_err_to_virt_err(enum ice_status ice_err) +{ + switch (ice_err) { + case ICE_SUCCESS: + return VIRTCHNL_STATUS_SUCCESS; + case ICE_ERR_BAD_PTR: + case ICE_ERR_INVAL_SIZE: + case ICE_ERR_DEVICE_NOT_SUPPORTED: + case ICE_ERR_PARAM: + case ICE_ERR_CFG: + return VIRTCHNL_STATUS_ERR_PARAM; + case ICE_ERR_NO_MEMORY: + return VIRTCHNL_STATUS_ERR_NO_MEMORY; + case ICE_ERR_NOT_READY: + case ICE_ERR_RESET_FAILED: + case ICE_ERR_FW_API_VER: + case ICE_ERR_AQ_ERROR: + case ICE_ERR_AQ_TIMEOUT: + case ICE_ERR_AQ_FULL: + case ICE_ERR_AQ_NO_WORK: + case ICE_ERR_AQ_EMPTY: + return VIRTCHNL_STATUS_ERR_ADMIN_QUEUE_ERROR; + default: + return VIRTCHNL_STATUS_ERR_NOT_SUPPORTED; + } +} + /** * ice_vc_vf_broadcast - Broadcast a message to all VFs on PF * @pf: pointer to the PF structure @@ -14,7 +45,7 @@ */ static void ice_vc_vf_broadcast(struct ice_pf *pf, enum virtchnl_ops v_opcode, - enum ice_status v_retval, u8 *msg, u16 msglen) + enum virtchnl_status_code v_retval, u8 *msg, u16 msglen) { struct ice_hw *hw = &pf->hw; struct ice_vf *vf = pf->vf; @@ -104,7 +135,8 @@ static void ice_vc_notify_vf_link_state(struct ice_vf *vf) ice_set_pfe_link(vf, &pfe, ls->link_speed, ls->link_info & ICE_AQ_LINK_UP); - ice_aq_send_msg_to_vf(hw, vf->vf_id, VIRTCHNL_OP_EVENT, 0, (u8 *)&pfe, + ice_aq_send_msg_to_vf(hw, vf->vf_id, VIRTCHNL_OP_EVENT, + VIRTCHNL_STATUS_SUCCESS, (u8 *)&pfe, sizeof(pfe), NULL); } @@ -343,11 +375,41 @@ static void ice_trigger_vf_reset(struct ice_vf *vf, bool is_vflr) } /** - * ice_vsi_set_pvid - Set port VLAN id for the VSI - * @vsi: the VSI being changed - * @vid: the VLAN id to set as a PVID + * ice_vsi_set_pvid_fill_ctxt - Set VSI ctxt for add PVID + * @ctxt: the VSI ctxt to fill + * @vid: the VLAN ID to set as a PVID + */ +static void ice_vsi_set_pvid_fill_ctxt(struct ice_vsi_ctx *ctxt, u16 vid) +{ + ctxt->info.vlan_flags = (ICE_AQ_VSI_VLAN_MODE_UNTAGGED | + ICE_AQ_VSI_PVLAN_INSERT_PVID | + ICE_AQ_VSI_VLAN_EMOD_STR); + ctxt->info.pvid = cpu_to_le16(vid); + ctxt->info.sw_flags2 |= ICE_AQ_VSI_SW_FLAG_RX_VLAN_PRUNE_ENA; + ctxt->info.valid_sections = cpu_to_le16(ICE_AQ_VSI_PROP_VLAN_VALID | + ICE_AQ_VSI_PROP_SW_VALID); +} + +/** + * ice_vsi_kill_pvid_fill_ctxt - Set VSI ctx for remove PVID + * @ctxt: the VSI ctxt to fill + */ +static void ice_vsi_kill_pvid_fill_ctxt(struct ice_vsi_ctx *ctxt) +{ + ctxt->info.vlan_flags = ICE_AQ_VSI_VLAN_EMOD_NOTHING; + ctxt->info.vlan_flags |= ICE_AQ_VSI_VLAN_MODE_ALL; + ctxt->info.sw_flags2 &= ~ICE_AQ_VSI_SW_FLAG_RX_VLAN_PRUNE_ENA; + ctxt->info.valid_sections = cpu_to_le16(ICE_AQ_VSI_PROP_VLAN_VALID | + ICE_AQ_VSI_PROP_SW_VALID); +} + +/** + * ice_vsi_manage_pvid - Enable or disable port VLAN for VSI + * @vsi: the VSI to update + * @vid: the VLAN ID to set as a PVID + * @enable: true for enable PVID false for disable */ -static int ice_vsi_set_pvid(struct ice_vsi *vsi, u16 vid) +static int ice_vsi_manage_pvid(struct ice_vsi *vsi, u16 vid, bool enable) { struct device *dev = &vsi->back->pdev->dev; struct ice_hw *hw = &vsi->back->hw; @@ -359,50 +421,31 @@ static int ice_vsi_set_pvid(struct ice_vsi *vsi, u16 vid) if (!ctxt) return -ENOMEM; - ctxt->info.vlan_flags = (ICE_AQ_VSI_VLAN_MODE_UNTAGGED | - ICE_AQ_VSI_PVLAN_INSERT_PVID | - ICE_AQ_VSI_VLAN_EMOD_STR); - ctxt->info.pvid = cpu_to_le16(vid); - ctxt->info.valid_sections = cpu_to_le16(ICE_AQ_VSI_PROP_VLAN_VALID); + ctxt->info = vsi->info; + if (enable) + ice_vsi_set_pvid_fill_ctxt(ctxt, vid); + else + ice_vsi_kill_pvid_fill_ctxt(ctxt); status = ice_update_vsi(hw, vsi->idx, ctxt, NULL); if (status) { - dev_info(dev, "update VSI for VLAN insert failed, err %d aq_err %d\n", + dev_info(dev, "update VSI for port VLAN failed, err %d aq_err %d\n", status, hw->adminq.sq_last_status); ret = -EIO; goto out; } - vsi->info.pvid = ctxt->info.pvid; - vsi->info.vlan_flags = ctxt->info.vlan_flags; + vsi->info = ctxt->info; out: devm_kfree(dev, ctxt); return ret; } -/** - * ice_vsi_kill_pvid - Remove port VLAN id from the VSI - * @vsi: the VSI being changed - */ -static int ice_vsi_kill_pvid(struct ice_vsi *vsi) -{ - struct ice_pf *pf = vsi->back; - - if (ice_vsi_manage_vlan_stripping(vsi, false)) { - dev_err(&pf->pdev->dev, "Error removing Port VLAN on VSI %i\n", - vsi->vsi_num); - return -ENODEV; - } - - vsi->info.pvid = 0; - return 0; -} - /** * ice_vf_vsi_setup - Set up a VF VSI * @pf: board private structure * @pi: pointer to the port_info instance - * @vf_id: defines VF id to which this VSI connects. + * @vf_id: defines VF ID to which this VSI connects. * * Returns pointer to the successfully allocated VSI struct on success, * otherwise returns NULL on failure. @@ -446,8 +489,10 @@ static int ice_alloc_vsi_res(struct ice_vf *vf) vsi->hw_base_vector += 1; /* Check if port VLAN exist before, and restore it accordingly */ - if (vf->port_vlan_id) - ice_vsi_set_pvid(vsi, vf->port_vlan_id); + if (vf->port_vlan_id) { + ice_vsi_manage_pvid(vsi, vf->port_vlan_id, true); + ice_vsi_add_vlan(vsi, vf->port_vlan_id & ICE_VLAN_M); + } eth_broadcast_addr(broadcast); @@ -468,7 +513,7 @@ static int ice_alloc_vsi_res(struct ice_vf *vf) /* Clear this bit after VF initialization since we shouldn't reclaim * and reassign interrupts for synchronous or asynchronous VFR events. - * We dont want to reconfigure interrupts since AVF driver doesn't + * We don't want to reconfigure interrupts since AVF driver doesn't * expect vector assignment to be changed unless there is a request for * more vectors. */ @@ -484,6 +529,8 @@ ice_alloc_vsi_res_exit: */ static int ice_alloc_vf_res(struct ice_vf *vf) { + struct ice_pf *pf = vf->pf; + int tx_rx_queue_left; int status; /* setup VF VSI and necessary resources */ @@ -491,6 +538,15 @@ static int ice_alloc_vf_res(struct ice_vf *vf) if (status) goto ice_alloc_vf_res_exit; + /* Update number of VF queues, in case VF had requested for queue + * changes + */ + tx_rx_queue_left = min_t(int, pf->q_left_tx, pf->q_left_rx); + tx_rx_queue_left += ICE_DFLT_QS_PER_VF; + if (vf->num_req_qs && vf->num_req_qs <= tx_rx_queue_left && + vf->num_req_qs != vf->num_vf_qs) + vf->num_vf_qs = vf->num_req_qs; + if (vf->trusted) set_bit(ICE_VIRTCHNL_VF_CAP_PRIVILEGE, &vf->vf_caps); else @@ -548,6 +604,10 @@ static void ice_ena_vf_mappings(struct ice_vf *vf) wr32(hw, GLINT_VECT2FUNC(v), reg); } + /* Map mailbox interrupt. We put an explicit 0 here to remind us that + * VF admin queue interrupts will go to VF MSI-X vector 0. + */ + wr32(hw, VPINT_MBX_CTL(abs_vf_id), VPINT_MBX_CTL_CAUSE_ENA_M | 0); /* set regardless of mapping mode */ wr32(hw, VPLAN_TXQ_MAPENA(vf->vf_id), VPLAN_TXQ_MAPENA_TX_ENA_M); @@ -749,6 +809,47 @@ static void ice_cleanup_and_realloc_vf(struct ice_vf *vf) wr32(hw, VFGEN_RSTAT(vf->vf_id), VIRTCHNL_VFR_VFACTIVE); } +/** + * ice_vf_set_vsi_promisc - set given VF VSI to given promiscuous mode(s) + * @vf: pointer to the VF info + * @vsi: the VSI being configured + * @promisc_m: mask of promiscuous config bits + * @rm_promisc: promisc flag request from the VF to remove or add filter + * + * This function configures VF VSI promiscuous mode, based on the VF requests, + * for Unicast, Multicast and VLAN + */ +static enum ice_status +ice_vf_set_vsi_promisc(struct ice_vf *vf, struct ice_vsi *vsi, u8 promisc_m, + bool rm_promisc) +{ + struct ice_pf *pf = vf->pf; + enum ice_status status = 0; + struct ice_hw *hw; + + hw = &pf->hw; + if (vf->num_vlan) { + status = ice_set_vlan_vsi_promisc(hw, vsi->idx, promisc_m, + rm_promisc); + } else if (vf->port_vlan_id) { + if (rm_promisc) + status = ice_clear_vsi_promisc(hw, vsi->idx, promisc_m, + vf->port_vlan_id); + else + status = ice_set_vsi_promisc(hw, vsi->idx, promisc_m, + vf->port_vlan_id); + } else { + if (rm_promisc) + status = ice_clear_vsi_promisc(hw, vsi->idx, promisc_m, + 0); + else + status = ice_set_vsi_promisc(hw, vsi->idx, promisc_m, + 0); + } + + return status; +} + /** * ice_reset_all_vfs - reset all allocated VFs in one go * @pf: pointer to the PF structure @@ -764,6 +865,7 @@ static void ice_cleanup_and_realloc_vf(struct ice_vf *vf) bool ice_reset_all_vfs(struct ice_pf *pf, bool is_vflr) { struct ice_hw *hw = &pf->hw; + struct ice_vf *vf; int v, i; /* If we don't have any VFs, then there is nothing to reset */ @@ -778,12 +880,17 @@ bool ice_reset_all_vfs(struct ice_pf *pf, bool is_vflr) for (v = 0; v < pf->num_alloc_vfs; v++) ice_trigger_vf_reset(&pf->vf[v], is_vflr); - /* Call Disable LAN Tx queue AQ call with VFR bit set and 0 - * queues to inform Firmware about VF reset. - */ - for (v = 0; v < pf->num_alloc_vfs; v++) - ice_dis_vsi_txq(pf->vsi[0]->port_info, 0, NULL, NULL, - ICE_VF_RESET, v, NULL); + for (v = 0; v < pf->num_alloc_vfs; v++) { + struct ice_vsi *vsi; + + vf = &pf->vf[v]; + vsi = pf->vsi[vf->lan_vsi_idx]; + if (test_bit(ICE_VF_STATE_ENA, vf->vf_states)) { + ice_vsi_stop_lan_tx_rings(vsi, ICE_VF_RESET, vf->vf_id); + ice_vsi_stop_rx_rings(vsi); + clear_bit(ICE_VF_STATE_ENA, vf->vf_states); + } + } /* HW requires some time to make sure it can flush the FIFO for a VF * when it resets it. Poll the VPGEN_VFRSTAT register for each VF in @@ -796,9 +903,9 @@ bool ice_reset_all_vfs(struct ice_pf *pf, bool is_vflr) /* Check each VF in sequence */ while (v < pf->num_alloc_vfs) { - struct ice_vf *vf = &pf->vf[v]; u32 reg; + vf = &pf->vf[v]; reg = rd32(hw, VPGEN_VFRSTAT(vf->vf_id)); if (!(reg & VPGEN_VFRSTAT_VFRD_M)) break; @@ -818,8 +925,18 @@ bool ice_reset_all_vfs(struct ice_pf *pf, bool is_vflr) usleep_range(10000, 20000); /* free VF resources to begin resetting the VSI state */ - for (v = 0; v < pf->num_alloc_vfs; v++) - ice_free_vf_res(&pf->vf[v]); + for (v = 0; v < pf->num_alloc_vfs; v++) { + vf = &pf->vf[v]; + + ice_free_vf_res(vf); + + /* Free VF queues as well, and reallocate later. + * If a given VF has different number of queues + * configured, the request for update will come + * via mailbox communication. + */ + vf->num_vf_qs = 0; + } if (ice_check_avail_res(pf)) { dev_err(&pf->pdev->dev, @@ -828,8 +945,15 @@ bool ice_reset_all_vfs(struct ice_pf *pf, bool is_vflr) } /* Finish the reset on each VF */ - for (v = 0; v < pf->num_alloc_vfs; v++) - ice_cleanup_and_realloc_vf(&pf->vf[v]); + for (v = 0; v < pf->num_alloc_vfs; v++) { + vf = &pf->vf[v]; + + vf->num_vf_qs = pf->num_vf_qps; + dev_dbg(&pf->pdev->dev, + "VF-id %d has %d queues configured\n", + vf->vf_id, vf->num_vf_qs); + ice_cleanup_and_realloc_vf(vf); + } ice_flush(hw); clear_bit(__ICE_VF_DIS, pf->state); @@ -847,9 +971,10 @@ bool ice_reset_all_vfs(struct ice_pf *pf, bool is_vflr) static bool ice_reset_vf(struct ice_vf *vf, bool is_vflr) { struct ice_pf *pf = vf->pf; - struct ice_hw *hw = &pf->hw; struct ice_vsi *vsi; + struct ice_hw *hw; bool rsd = false; + u8 promisc_m; u32 reg; int i; @@ -871,10 +996,11 @@ static bool ice_reset_vf(struct ice_vf *vf, bool is_vflr) /* Call Disable LAN Tx queue AQ call even when queues are not * enabled. This is needed for successful completiom of VFR */ - ice_dis_vsi_txq(vsi->port_info, 0, NULL, NULL, ICE_VF_RESET, - vf->vf_id, NULL); + ice_dis_vsi_txq(vsi->port_info, vsi->idx, 0, 0, NULL, NULL, + NULL, ICE_VF_RESET, vf->vf_id, NULL); } + hw = &pf->hw; /* poll VPGEN_VFRSTAT reg to make sure * that reset is complete */ @@ -900,6 +1026,21 @@ static bool ice_reset_vf(struct ice_vf *vf, bool is_vflr) usleep_range(10000, 20000); + /* disable promiscuous modes in case they were enabled + * ignore any error if disabling process failed + */ + if (test_bit(ICE_VF_STATE_UC_PROMISC, vf->vf_states) || + test_bit(ICE_VF_STATE_MC_PROMISC, vf->vf_states)) { + if (vf->port_vlan_id || vf->num_vlan) + promisc_m = ICE_UCAST_VLAN_PROMISC_BITS; + else + promisc_m = ICE_UCAST_PROMISC_BITS; + + vsi = pf->vsi[vf->lan_vsi_idx]; + if (ice_vf_set_vsi_promisc(vf, vsi, promisc_m, true)) + dev_err(&pf->pdev->dev, "disabling promiscuous mode failed\n"); + } + /* free VF resources to begin resetting the VSI state */ ice_free_vf_res(vf); @@ -938,7 +1079,7 @@ void ice_vc_notify_reset(struct ice_pf *pf) pfe.event = VIRTCHNL_EVENT_RESET_IMPENDING; pfe.severity = PF_EVENT_SEVERITY_CERTAIN_DOOM; - ice_vc_vf_broadcast(pf, VIRTCHNL_OP_EVENT, ICE_SUCCESS, + ice_vc_vf_broadcast(pf, VIRTCHNL_OP_EVENT, VIRTCHNL_STATUS_SUCCESS, (u8 *)&pfe, sizeof(struct virtchnl_pf_event)); } @@ -961,8 +1102,9 @@ static void ice_vc_notify_vf_reset(struct ice_vf *vf) pfe.event = VIRTCHNL_EVENT_RESET_IMPENDING; pfe.severity = PF_EVENT_SEVERITY_CERTAIN_DOOM; - ice_aq_send_msg_to_vf(&vf->pf->hw, vf->vf_id, VIRTCHNL_OP_EVENT, 0, - (u8 *)&pfe, sizeof(pfe), NULL); + ice_aq_send_msg_to_vf(&vf->pf->hw, vf->vf_id, VIRTCHNL_OP_EVENT, + VIRTCHNL_STATUS_SUCCESS, (u8 *)&pfe, sizeof(pfe), + NULL); } /** @@ -1012,7 +1154,7 @@ static int ice_alloc_vfs(struct ice_pf *pf, u16 num_alloc_vfs) pf->num_alloc_vfs = num_alloc_vfs; /* VF resources get allocated during reset */ - if (!ice_reset_all_vfs(pf, false)) + if (!ice_reset_all_vfs(pf, true)) goto err_unroll_sriov; goto err_unroll_intr; @@ -1131,21 +1273,10 @@ void ice_process_vflr_event(struct ice_pf *pf) int vf_id; u32 reg; - if (!test_bit(__ICE_VFLR_EVENT_PENDING, pf->state) || + if (!test_and_clear_bit(__ICE_VFLR_EVENT_PENDING, pf->state) || !pf->num_alloc_vfs) return; - /* Re-enable the VFLR interrupt cause here, before looking for which - * VF got reset. Otherwise, if another VF gets a reset while the - * first one is being processed, that interrupt will be lost, and - * that VF will be stuck in reset forever. - */ - reg = rd32(hw, PFINT_OICR_ENA); - reg |= PFINT_OICR_VFLR_M; - wr32(hw, PFINT_OICR_ENA, reg); - ice_flush(hw); - - clear_bit(__ICE_VFLR_EVENT_PENDING, pf->state); for (vf_id = 0; vf_id < pf->num_alloc_vfs; vf_id++) { struct ice_vf *vf = &pf->vf[vf_id]; u32 reg_idx, bit_idx; @@ -1182,8 +1313,9 @@ static void ice_vc_dis_vf(struct ice_vf *vf) * * send msg to VF */ -static int ice_vc_send_msg_to_vf(struct ice_vf *vf, u32 v_opcode, - enum ice_status v_retval, u8 *msg, u16 msglen) +static int +ice_vc_send_msg_to_vf(struct ice_vf *vf, u32 v_opcode, + enum virtchnl_status_code v_retval, u8 *msg, u16 msglen) { enum ice_status aq_ret; struct ice_pf *pf; @@ -1243,8 +1375,8 @@ static int ice_vc_get_ver_msg(struct ice_vf *vf, u8 *msg) if (VF_IS_V10(&vf->vf_ver)) info.minor = VIRTCHNL_VERSION_MINOR_NO_VF_CAPS; - return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_VERSION, ICE_SUCCESS, - (u8 *)&info, + return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_VERSION, + VIRTCHNL_STATUS_SUCCESS, (u8 *)&info, sizeof(struct virtchnl_version_info)); } @@ -1257,15 +1389,15 @@ static int ice_vc_get_ver_msg(struct ice_vf *vf, u8 *msg) */ static int ice_vc_get_vf_res_msg(struct ice_vf *vf, u8 *msg) { + enum virtchnl_status_code v_ret = VIRTCHNL_STATUS_SUCCESS; struct virtchnl_vf_resource *vfres = NULL; - enum ice_status aq_ret = 0; struct ice_pf *pf = vf->pf; struct ice_vsi *vsi; int len = 0; int ret; if (!test_bit(ICE_VF_STATE_INIT, vf->vf_states)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto err; } @@ -1273,7 +1405,7 @@ static int ice_vc_get_vf_res_msg(struct ice_vf *vf, u8 *msg) vfres = devm_kzalloc(&pf->pdev->dev, len, GFP_KERNEL); if (!vfres) { - aq_ret = ICE_ERR_NO_MEMORY; + v_ret = VIRTCHNL_STATUS_ERR_NO_MEMORY; len = 0; goto err; } @@ -1286,6 +1418,11 @@ static int ice_vc_get_vf_res_msg(struct ice_vf *vf, u8 *msg) vfres->vf_cap_flags = VIRTCHNL_VF_OFFLOAD_L2; vsi = pf->vsi[vf->lan_vsi_idx]; + if (!vsi) { + v_ret = VIRTCHNL_STATUS_ERR_PARAM; + goto err; + } + if (!vsi->info.pvid) vfres->vf_cap_flags |= VIRTCHNL_VF_OFFLOAD_VLAN; @@ -1336,7 +1473,7 @@ static int ice_vc_get_vf_res_msg(struct ice_vf *vf, u8 *msg) err: /* send the response back to the VF */ - ret = ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_GET_VF_RESOURCES, aq_ret, + ret = ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_GET_VF_RESOURCES, v_ret, (u8 *)vfres, len); devm_kfree(&pf->pdev->dev, vfres); @@ -1360,15 +1497,15 @@ static void ice_vc_reset_vf_msg(struct ice_vf *vf) /** * ice_find_vsi_from_id * @pf: the pf structure to search for the VSI - * @id: id of the VSI it is searching for + * @id: ID of the VSI it is searching for * - * searches for the VSI with the given id + * searches for the VSI with the given ID */ static struct ice_vsi *ice_find_vsi_from_id(struct ice_pf *pf, u16 id) { int i; - for (i = 0; i < pf->num_alloc_vsi; i++) + ice_for_each_vsi(pf, i) if (pf->vsi[i] && pf->vsi[i]->vsi_num == id) return pf->vsi[i]; @@ -1378,9 +1515,9 @@ static struct ice_vsi *ice_find_vsi_from_id(struct ice_pf *pf, u16 id) /** * ice_vc_isvalid_vsi_id * @vf: pointer to the VF info - * @vsi_id: VF relative VSI id + * @vsi_id: VF relative VSI ID * - * check for the valid VSI id + * check for the valid VSI ID */ static bool ice_vc_isvalid_vsi_id(struct ice_vf *vf, u16 vsi_id) { @@ -1395,10 +1532,10 @@ static bool ice_vc_isvalid_vsi_id(struct ice_vf *vf, u16 vsi_id) /** * ice_vc_isvalid_q_id * @vf: pointer to the VF info - * @vsi_id: VSI id - * @qid: VSI relative queue id + * @vsi_id: VSI ID + * @qid: VSI relative queue ID * - * check for the valid queue id + * check for the valid queue ID */ static bool ice_vc_isvalid_q_id(struct ice_vf *vf, u16 vsi_id, u8 qid) { @@ -1416,42 +1553,42 @@ static bool ice_vc_isvalid_q_id(struct ice_vf *vf, u16 vsi_id, u8 qid) */ static int ice_vc_config_rss_key(struct ice_vf *vf, u8 *msg) { + enum virtchnl_status_code v_ret = VIRTCHNL_STATUS_SUCCESS; struct virtchnl_rss_key *vrk = (struct virtchnl_rss_key *)msg; + struct ice_pf *pf = vf->pf; struct ice_vsi *vsi = NULL; - enum ice_status aq_ret; - int ret; if (!test_bit(ICE_VF_STATE_ACTIVE, vf->vf_states)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } if (!ice_vc_isvalid_vsi_id(vf, vrk->vsi_id)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } - vsi = ice_find_vsi_from_id(vf->pf, vrk->vsi_id); + vsi = pf->vsi[vf->lan_vsi_idx]; if (!vsi) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } if (vrk->key_len != ICE_VSIQF_HKEY_ARRAY_SIZE) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } if (!test_bit(ICE_FLAG_RSS_ENA, vf->pf->flags)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } - ret = ice_set_rss(vsi, vrk->key, NULL, 0); - aq_ret = ret ? ICE_ERR_PARAM : ICE_SUCCESS; + if (ice_set_rss(vsi, vrk->key, NULL, 0)) + v_ret = VIRTCHNL_STATUS_ERR_ADMIN_QUEUE_ERROR; error_param: - return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_CONFIG_RSS_KEY, aq_ret, + return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_CONFIG_RSS_KEY, v_ret, NULL, 0); } @@ -1465,40 +1602,40 @@ error_param: static int ice_vc_config_rss_lut(struct ice_vf *vf, u8 *msg) { struct virtchnl_rss_lut *vrl = (struct virtchnl_rss_lut *)msg; + enum virtchnl_status_code v_ret = VIRTCHNL_STATUS_SUCCESS; + struct ice_pf *pf = vf->pf; struct ice_vsi *vsi = NULL; - enum ice_status aq_ret; - int ret; if (!test_bit(ICE_VF_STATE_ACTIVE, vf->vf_states)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } if (!ice_vc_isvalid_vsi_id(vf, vrl->vsi_id)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } - vsi = ice_find_vsi_from_id(vf->pf, vrl->vsi_id); + vsi = pf->vsi[vf->lan_vsi_idx]; if (!vsi) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } if (vrl->lut_entries != ICE_VSIQF_HLUT_ARRAY_SIZE) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } if (!test_bit(ICE_FLAG_RSS_ENA, vf->pf->flags)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } - ret = ice_set_rss(vsi, NULL, vrl->lut, ICE_VSIQF_HLUT_ARRAY_SIZE); - aq_ret = ret ? ICE_ERR_PARAM : ICE_SUCCESS; + if (ice_set_rss(vsi, NULL, vrl->lut, ICE_VSIQF_HLUT_ARRAY_SIZE)) + v_ret = VIRTCHNL_STATUS_ERR_ADMIN_QUEUE_ERROR; error_param: - return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_CONFIG_RSS_LUT, aq_ret, + return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_CONFIG_RSS_LUT, v_ret, NULL, 0); } @@ -1511,25 +1648,26 @@ error_param: */ static int ice_vc_get_stats_msg(struct ice_vf *vf, u8 *msg) { + enum virtchnl_status_code v_ret = VIRTCHNL_STATUS_SUCCESS; struct virtchnl_queue_select *vqs = (struct virtchnl_queue_select *)msg; - enum ice_status aq_ret = 0; + struct ice_pf *pf = vf->pf; struct ice_eth_stats stats; struct ice_vsi *vsi; if (!test_bit(ICE_VF_STATE_ACTIVE, vf->vf_states)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } if (!ice_vc_isvalid_vsi_id(vf, vqs->vsi_id)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } - vsi = ice_find_vsi_from_id(vf->pf, vqs->vsi_id); + vsi = pf->vsi[vf->lan_vsi_idx]; if (!vsi) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } @@ -1540,7 +1678,7 @@ static int ice_vc_get_stats_msg(struct ice_vf *vf, u8 *msg) error_param: /* send the response to the VF */ - return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_GET_STATS, aq_ret, + return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_GET_STATS, v_ret, (u8 *)&stats, sizeof(stats)); } @@ -1553,29 +1691,30 @@ error_param: */ static int ice_vc_ena_qs_msg(struct ice_vf *vf, u8 *msg) { + enum virtchnl_status_code v_ret = VIRTCHNL_STATUS_SUCCESS; struct virtchnl_queue_select *vqs = (struct virtchnl_queue_select *)msg; - enum ice_status aq_ret = 0; + struct ice_pf *pf = vf->pf; struct ice_vsi *vsi; if (!test_bit(ICE_VF_STATE_ACTIVE, vf->vf_states)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } if (!ice_vc_isvalid_vsi_id(vf, vqs->vsi_id)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } if (!vqs->rx_queues && !vqs->tx_queues) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } - vsi = ice_find_vsi_from_id(vf->pf, vqs->vsi_id); + vsi = pf->vsi[vf->lan_vsi_idx]; if (!vsi) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } @@ -1584,15 +1723,15 @@ static int ice_vc_ena_qs_msg(struct ice_vf *vf, u8 *msg) * programmed using ice_vsi_cfg_txqs */ if (ice_vsi_start_rx_rings(vsi)) - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; /* Set flag to indicate that queues are enabled */ - if (!aq_ret) + if (v_ret == VIRTCHNL_STATUS_SUCCESS) set_bit(ICE_VF_STATE_ENA, vf->vf_states); error_param: /* send the response to the VF */ - return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_ENABLE_QUEUES, aq_ret, + return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_ENABLE_QUEUES, v_ret, NULL, 0); } @@ -1606,30 +1745,31 @@ error_param: */ static int ice_vc_dis_qs_msg(struct ice_vf *vf, u8 *msg) { + enum virtchnl_status_code v_ret = VIRTCHNL_STATUS_SUCCESS; struct virtchnl_queue_select *vqs = (struct virtchnl_queue_select *)msg; - enum ice_status aq_ret = 0; + struct ice_pf *pf = vf->pf; struct ice_vsi *vsi; if (!test_bit(ICE_VF_STATE_ACTIVE, vf->vf_states) && !test_bit(ICE_VF_STATE_ENA, vf->vf_states)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } if (!ice_vc_isvalid_vsi_id(vf, vqs->vsi_id)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } if (!vqs->rx_queues && !vqs->tx_queues) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } - vsi = ice_find_vsi_from_id(vf->pf, vqs->vsi_id); + vsi = pf->vsi[vf->lan_vsi_idx]; if (!vsi) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } @@ -1637,23 +1777,23 @@ static int ice_vc_dis_qs_msg(struct ice_vf *vf, u8 *msg) dev_err(&vsi->back->pdev->dev, "Failed to stop tx rings on VSI %d\n", vsi->vsi_num); - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; } if (ice_vsi_stop_rx_rings(vsi)) { dev_err(&vsi->back->pdev->dev, "Failed to stop rx rings on VSI %d\n", vsi->vsi_num); - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; } /* Clear enabled queues flag */ - if (!aq_ret) + if (v_ret == VIRTCHNL_STATUS_SUCCESS) clear_bit(ICE_VF_STATE_ENA, vf->vf_states); error_param: /* send the response to the VF */ - return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_DISABLE_QUEUES, aq_ret, + return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_DISABLE_QUEUES, v_ret, NULL, 0); } @@ -1666,22 +1806,30 @@ error_param: */ static int ice_vc_cfg_irq_map_msg(struct ice_vf *vf, u8 *msg) { + enum virtchnl_status_code v_ret = VIRTCHNL_STATUS_SUCCESS; struct virtchnl_irq_map_info *irqmap_info = (struct virtchnl_irq_map_info *)msg; u16 vsi_id, vsi_q_id, vector_id; struct virtchnl_vector_map *map; struct ice_vsi *vsi = NULL; struct ice_pf *pf = vf->pf; - enum ice_status aq_ret = 0; unsigned long qmap; + u16 num_q_vectors; int i; - if (!test_bit(ICE_VF_STATE_ACTIVE, vf->vf_states)) { - aq_ret = ICE_ERR_PARAM; + num_q_vectors = irqmap_info->num_vectors - ICE_NONQ_VECS_VF; + vsi = pf->vsi[vf->lan_vsi_idx]; + + if (!test_bit(ICE_VF_STATE_ACTIVE, vf->vf_states) || + !vsi || vsi->num_q_vectors < num_q_vectors || + irqmap_info->num_vectors == 0) { + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } - for (i = 0; i < irqmap_info->num_vectors; i++) { + for (i = 0; i < num_q_vectors; i++) { + struct ice_q_vector *q_vector = vsi->q_vectors[i]; + map = &irqmap_info->vecmap[i]; vector_id = map->vector_id; @@ -1689,40 +1837,30 @@ static int ice_vc_cfg_irq_map_msg(struct ice_vf *vf, u8 *msg) /* validate msg params */ if (!(vector_id < pf->hw.func_caps.common_cap .num_msix_vectors) || !ice_vc_isvalid_vsi_id(vf, vsi_id)) { - aq_ret = ICE_ERR_PARAM; - goto error_param; - } - - vsi = ice_find_vsi_from_id(vf->pf, vsi_id); - if (!vsi) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } /* lookout for the invalid queue index */ qmap = map->rxq_map; + q_vector->num_ring_rx = 0; for_each_set_bit(vsi_q_id, &qmap, ICE_MAX_BASE_QS_PER_VF) { - struct ice_q_vector *q_vector; - if (!ice_vc_isvalid_q_id(vf, vsi_id, vsi_q_id)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } - q_vector = vsi->q_vectors[i]; q_vector->num_ring_rx++; q_vector->rx.itr_idx = map->rxitr_idx; vsi->rx_rings[vsi_q_id]->q_vector = q_vector; } qmap = map->txq_map; + q_vector->num_ring_tx = 0; for_each_set_bit(vsi_q_id, &qmap, ICE_MAX_BASE_QS_PER_VF) { - struct ice_q_vector *q_vector; - if (!ice_vc_isvalid_q_id(vf, vsi_id, vsi_q_id)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } - q_vector = vsi->q_vectors[i]; q_vector->num_ring_tx++; q_vector->tx.itr_idx = map->txitr_idx; vsi->tx_rings[vsi_q_id]->q_vector = q_vector; @@ -1733,7 +1871,7 @@ static int ice_vc_cfg_irq_map_msg(struct ice_vf *vf, u8 *msg) ice_vsi_cfg_msix(vsi); error_param: /* send the response to the VF */ - return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_CONFIG_IRQ_MAP, aq_ret, + return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_CONFIG_IRQ_MAP, v_ret, NULL, 0); } @@ -1746,26 +1884,34 @@ error_param: */ static int ice_vc_cfg_qs_msg(struct ice_vf *vf, u8 *msg) { + enum virtchnl_status_code v_ret = VIRTCHNL_STATUS_SUCCESS; struct virtchnl_vsi_queue_config_info *qci = (struct virtchnl_vsi_queue_config_info *)msg; struct virtchnl_queue_pair_info *qpi; - enum ice_status aq_ret = 0; + struct ice_pf *pf = vf->pf; struct ice_vsi *vsi; int i; if (!test_bit(ICE_VF_STATE_ACTIVE, vf->vf_states)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } if (!ice_vc_isvalid_vsi_id(vf, qci->vsi_id)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } - vsi = ice_find_vsi_from_id(vf->pf, qci->vsi_id); + vsi = pf->vsi[vf->lan_vsi_idx]; if (!vsi) { - aq_ret = ICE_ERR_PARAM; + goto error_param; + } + + if (qci->num_queue_pairs > ICE_MAX_BASE_QS_PER_VF) { + dev_err(&pf->pdev->dev, + "VF-%d requesting more than supported number of queues: %d\n", + vf->vf_id, qci->num_queue_pairs); + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } @@ -1775,7 +1921,7 @@ static int ice_vc_cfg_qs_msg(struct ice_vf *vf, u8 *msg) qpi->rxq.vsi_id != qci->vsi_id || qpi->rxq.queue_id != qpi->txq.queue_id || !ice_vc_isvalid_q_id(vf, qci->vsi_id, qpi->txq.queue_id)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } /* copy Tx queue info from VF into VSI */ @@ -1785,13 +1931,13 @@ static int ice_vc_cfg_qs_msg(struct ice_vf *vf, u8 *msg) vsi->rx_rings[i]->dma = qpi->rxq.dma_ring_addr; vsi->rx_rings[i]->count = qpi->rxq.ring_len; if (qpi->rxq.databuffer_size > ((16 * 1024) - 128)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } vsi->rx_buf_len = qpi->rxq.databuffer_size; if (qpi->rxq.max_pkt_size >= (16 * 1024) || qpi->rxq.max_pkt_size < 64) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } vsi->max_frame = qpi->rxq.max_pkt_size; @@ -1802,15 +1948,16 @@ static int ice_vc_cfg_qs_msg(struct ice_vf *vf, u8 *msg) */ vsi->num_txq = qci->num_queue_pairs; vsi->num_rxq = qci->num_queue_pairs; + /* All queues of VF VSI are in TC 0 */ + vsi->tc_cfg.tc_info[0].qcount_tx = qci->num_queue_pairs; + vsi->tc_cfg.tc_info[0].qcount_rx = qci->num_queue_pairs; - if (!ice_vsi_cfg_lan_txqs(vsi) && !ice_vsi_cfg_rxqs(vsi)) - aq_ret = 0; - else - aq_ret = ICE_ERR_PARAM; + if (ice_vsi_cfg_lan_txqs(vsi) || ice_vsi_cfg_rxqs(vsi)) + v_ret = VIRTCHNL_STATUS_ERR_ADMIN_QUEUE_ERROR; error_param: /* send the response to the VF */ - return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_CONFIG_VSI_QUEUES, aq_ret, + return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_CONFIG_VSI_QUEUES, v_ret, NULL, 0); } @@ -1845,18 +1992,18 @@ static bool ice_can_vf_change_mac(struct ice_vf *vf) * ice_vc_handle_mac_addr_msg * @vf: pointer to the VF info * @msg: pointer to the msg buffer - * @set: true if mac filters are being set, false otherwise + * @set: true if MAC filters are being set, false otherwise * * add guest MAC address filter */ static int ice_vc_handle_mac_addr_msg(struct ice_vf *vf, u8 *msg, bool set) { + enum virtchnl_status_code v_ret = VIRTCHNL_STATUS_SUCCESS; struct virtchnl_ether_addr_list *al = (struct virtchnl_ether_addr_list *)msg; struct ice_pf *pf = vf->pf; enum virtchnl_ops vc_op; - enum ice_status ret; LIST_HEAD(mac_list); struct ice_vsi *vsi; int mac_count = 0; @@ -1869,19 +2016,27 @@ ice_vc_handle_mac_addr_msg(struct ice_vf *vf, u8 *msg, bool set) if (!test_bit(ICE_VF_STATE_ACTIVE, vf->vf_states) || !ice_vc_isvalid_vsi_id(vf, al->vsi_id)) { - ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto handle_mac_exit; } if (set && !ice_is_vf_trusted(vf) && (vf->num_mac + al->num_elements) > ICE_MAX_MACADDR_PER_VF) { dev_err(&pf->pdev->dev, - "Can't add more MAC addresses, because VF is not trusted, switch the VF to trusted mode in order to add more functionalities\n"); - ret = ICE_ERR_PARAM; + "Can't add more MAC addresses, because VF-%d is not trusted, switch the VF to trusted mode in order to add more functionalities\n", + vf->vf_id); + /* There is no need to let VF know about not being trusted + * to add more MAC addr, so we can just return success message. + */ + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto handle_mac_exit; } vsi = pf->vsi[vf->lan_vsi_idx]; + if (!vsi) { + v_ret = VIRTCHNL_STATUS_ERR_PARAM; + goto handle_mac_exit; + } for (i = 0; i < al->num_elements; i++) { u8 *maddr = al->list[i].addr; @@ -1893,40 +2048,39 @@ ice_vc_handle_mac_addr_msg(struct ice_vf *vf, u8 *msg, bool set) * already added. Just continue. */ dev_info(&pf->pdev->dev, - "mac %pM already set for VF %d\n", + "MAC %pM already set for VF %d\n", maddr, vf->vf_id); continue; } else { - /* VF can't remove dflt_lan_addr/bcast mac */ + /* VF can't remove dflt_lan_addr/bcast MAC */ dev_err(&pf->pdev->dev, - "can't remove mac %pM for VF %d\n", + "VF can't remove default MAC address or MAC %pM programmed by PF for VF %d\n", maddr, vf->vf_id); - ret = ICE_ERR_PARAM; - goto handle_mac_exit; + continue; } } /* check for the invalid cases and bail if necessary */ if (is_zero_ether_addr(maddr)) { dev_err(&pf->pdev->dev, - "invalid mac %pM provided for VF %d\n", + "invalid MAC %pM provided for VF %d\n", maddr, vf->vf_id); - ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto handle_mac_exit; } if (is_unicast_ether_addr(maddr) && !ice_can_vf_change_mac(vf)) { dev_err(&pf->pdev->dev, - "can't change unicast mac for untrusted VF %d\n", + "can't change unicast MAC for untrusted VF %d\n", vf->vf_id); - ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto handle_mac_exit; } - /* get here if maddr is multicast or if VF can change mac */ + /* get here if maddr is multicast or if VF can change MAC */ if (ice_add_mac_to_list(vsi, &mac_list, al->list[i].addr)) { - ret = ICE_ERR_NO_MEMORY; + v_ret = VIRTCHNL_STATUS_ERR_NO_MEMORY; goto handle_mac_exit; } mac_count++; @@ -1934,14 +2088,14 @@ ice_vc_handle_mac_addr_msg(struct ice_vf *vf, u8 *msg, bool set) /* program the updated filter list */ if (set) - ret = ice_add_mac(&pf->hw, &mac_list); + v_ret = ice_err_to_virt_err(ice_add_mac(&pf->hw, &mac_list)); else - ret = ice_remove_mac(&pf->hw, &mac_list); + v_ret = ice_err_to_virt_err(ice_remove_mac(&pf->hw, &mac_list)); - if (ret) { + if (v_ret) { dev_err(&pf->pdev->dev, - "can't update mac filters for VF %d, error %d\n", - vf->vf_id, ret); + "can't update MAC filters for VF %d, error %d\n", + vf->vf_id, v_ret); } else { if (set) vf->num_mac += mac_count; @@ -1952,7 +2106,7 @@ ice_vc_handle_mac_addr_msg(struct ice_vf *vf, u8 *msg, bool set) handle_mac_exit: ice_free_fltr_list(&pf->pdev->dev, &mac_list); /* send the response to the VF */ - return ice_vc_send_msg_to_vf(vf, vc_op, ret, NULL, 0); + return ice_vc_send_msg_to_vf(vf, vc_op, v_ret, NULL, 0); } /** @@ -1987,39 +2141,42 @@ static int ice_vc_del_mac_addr_msg(struct ice_vf *vf, u8 *msg) * VFs get a default number of queues but can use this message to request a * different number. If the request is successful, PF will reset the VF and * return 0. If unsuccessful, PF will send message informing VF of number of - * available queue pairs via virtchnl message response to vf. + * available queue pairs via virtchnl message response to VF. */ static int ice_vc_request_qs_msg(struct ice_vf *vf, u8 *msg) { + enum virtchnl_status_code v_ret = VIRTCHNL_STATUS_SUCCESS; struct virtchnl_vf_res_request *vfres = (struct virtchnl_vf_res_request *)msg; int req_queues = vfres->num_queue_pairs; - enum ice_status aq_ret = 0; struct ice_pf *pf = vf->pf; + int max_allowed_vf_queues; int tx_rx_queue_left; int cur_queues; if (!test_bit(ICE_VF_STATE_ACTIVE, vf->vf_states)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } - cur_queues = pf->num_vf_qps; + cur_queues = vf->num_vf_qs; tx_rx_queue_left = min_t(int, pf->q_left_tx, pf->q_left_rx); + max_allowed_vf_queues = tx_rx_queue_left + cur_queues; if (req_queues <= 0) { dev_err(&pf->pdev->dev, "VF %d tried to request %d queues. Ignoring.\n", vf->vf_id, req_queues); - } else if (req_queues > ICE_MAX_QS_PER_VF) { + } else if (req_queues > ICE_MAX_BASE_QS_PER_VF) { dev_err(&pf->pdev->dev, "VF %d tried to request more than %d queues.\n", - vf->vf_id, ICE_MAX_QS_PER_VF); - vfres->num_queue_pairs = ICE_MAX_QS_PER_VF; + vf->vf_id, ICE_MAX_BASE_QS_PER_VF); + vfres->num_queue_pairs = ICE_MAX_BASE_QS_PER_VF; } else if (req_queues - cur_queues > tx_rx_queue_left) { dev_warn(&pf->pdev->dev, "VF %d requested %d more queues, but only %d left.\n", vf->vf_id, req_queues - cur_queues, tx_rx_queue_left); - vfres->num_queue_pairs = tx_rx_queue_left + cur_queues; + vfres->num_queue_pairs = min_t(int, max_allowed_vf_queues, + ICE_MAX_BASE_QS_PER_VF); } else { /* request is successful, then reset VF */ vf->num_req_qs = req_queues; @@ -2033,18 +2190,18 @@ static int ice_vc_request_qs_msg(struct ice_vf *vf, u8 *msg) error_param: /* send the response to the VF */ return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_REQUEST_QUEUES, - aq_ret, (u8 *)vfres, sizeof(*vfres)); + v_ret, (u8 *)vfres, sizeof(*vfres)); } /** * ice_set_vf_port_vlan * @netdev: network interface device structure * @vf_id: VF identifier - * @vlan_id: VLAN id being set + * @vlan_id: VLAN ID being set * @qos: priority setting * @vlan_proto: VLAN protocol * - * program VF Port VLAN id and/or qos + * program VF Port VLAN ID and/or QoS */ int ice_set_vf_port_vlan(struct net_device *netdev, int vf_id, u16 vlan_id, u8 qos, @@ -2087,17 +2244,18 @@ ice_set_vf_port_vlan(struct net_device *netdev, int vf_id, u16 vlan_id, u8 qos, return ret; } - /* If pvid, then remove all filters on the old VLAN */ + /* If PVID, then remove all filters on the old VLAN */ if (vsi->info.pvid) ice_vsi_kill_vlan(vsi, (le16_to_cpu(vsi->info.pvid) & VLAN_VID_MASK)); if (vlan_id || qos) { - ret = ice_vsi_set_pvid(vsi, vlanprio); + ret = ice_vsi_manage_pvid(vsi, vlanprio, true); if (ret) goto error_set_pvid; } else { - ice_vsi_kill_pvid(vsi); + ice_vsi_manage_pvid(vsi, 0, false); + vsi->info.pvid = 0; } if (vlan_id) { @@ -2125,52 +2283,60 @@ error_set_pvid: * @msg: pointer to the msg buffer * @add_v: Add VLAN if true, otherwise delete VLAN * - * Process virtchnl op to add or remove programmed guest VLAN id + * Process virtchnl op to add or remove programmed guest VLAN ID */ static int ice_vc_process_vlan_msg(struct ice_vf *vf, u8 *msg, bool add_v) { + enum virtchnl_status_code v_ret = VIRTCHNL_STATUS_SUCCESS; struct virtchnl_vlan_filter_list *vfl = (struct virtchnl_vlan_filter_list *)msg; - enum ice_status aq_ret = 0; struct ice_pf *pf = vf->pf; + bool vlan_promisc = false; struct ice_vsi *vsi; + struct ice_hw *hw; + int status = 0; + u8 promisc_m; int i; if (!test_bit(ICE_VF_STATE_ACTIVE, vf->vf_states)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } if (!ice_vc_isvalid_vsi_id(vf, vfl->vsi_id)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } if (add_v && !ice_is_vf_trusted(vf) && vf->num_vlan >= ICE_MAX_VLAN_PER_VF) { dev_info(&pf->pdev->dev, - "VF is not trusted, switch the VF to trusted mode, in order to add more VLAN addresses\n"); - aq_ret = ICE_ERR_PARAM; + "VF-%d is not trusted, switch the VF to trusted mode, in order to add more VLAN addresses\n", + vf->vf_id); + /* There is no need to let VF know about being not trusted, + * so we can just return success message here + */ goto error_param; } for (i = 0; i < vfl->num_elements; i++) { if (vfl->vlan_id[i] > ICE_MAX_VLANID) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; dev_err(&pf->pdev->dev, "invalid VF VLAN id %d\n", vfl->vlan_id[i]); goto error_param; } } - vsi = ice_find_vsi_from_id(vf->pf, vfl->vsi_id); + hw = &pf->hw; + vsi = pf->vsi[vf->lan_vsi_idx]; if (!vsi) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } if (vsi->info.pvid) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } @@ -2178,38 +2344,94 @@ static int ice_vc_process_vlan_msg(struct ice_vf *vf, u8 *msg, bool add_v) dev_err(&pf->pdev->dev, "%sable VLAN stripping failed for VSI %i\n", add_v ? "en" : "dis", vsi->vsi_num); - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } + if (test_bit(ICE_VF_STATE_UC_PROMISC, vf->vf_states) || + test_bit(ICE_VF_STATE_MC_PROMISC, vf->vf_states)) + vlan_promisc = true; + if (add_v) { for (i = 0; i < vfl->num_elements; i++) { u16 vid = vfl->vlan_id[i]; - if (!ice_vsi_add_vlan(vsi, vid)) { - vf->num_vlan++; + if (!ice_is_vf_trusted(vf) && + vf->num_vlan >= ICE_MAX_VLAN_PER_VF) { + dev_info(&pf->pdev->dev, + "VF-%d is not trusted, switch the VF to trusted mode, in order to add more VLAN addresses\n", + vf->vf_id); + /* There is no need to let VF know about being + * not trusted, so we can just return success + * message here as well. + */ + goto error_param; + } + + if (ice_vsi_add_vlan(vsi, vid)) { + v_ret = VIRTCHNL_STATUS_ERR_PARAM; + goto error_param; + } - /* Enable VLAN pruning when VLAN 0 is added */ - if (unlikely(!vid)) - if (ice_cfg_vlan_pruning(vsi, true)) - aq_ret = ICE_ERR_PARAM; + vf->num_vlan++; + /* Enable VLAN pruning when VLAN is added */ + if (!vlan_promisc) { + status = ice_cfg_vlan_pruning(vsi, true, false); + if (status) { + v_ret = VIRTCHNL_STATUS_ERR_PARAM; + dev_err(&pf->pdev->dev, + "Enable VLAN pruning on VLAN ID: %d failed error-%d\n", + vid, status); + goto error_param; + } } else { - aq_ret = ICE_ERR_PARAM; + /* Enable Ucast/Mcast VLAN promiscuous mode */ + promisc_m = ICE_PROMISC_VLAN_TX | + ICE_PROMISC_VLAN_RX; + + status = ice_set_vsi_promisc(hw, vsi->idx, + promisc_m, vid); + if (status) { + v_ret = VIRTCHNL_STATUS_ERR_PARAM; + dev_err(&pf->pdev->dev, + "Enable Unicast/multicast promiscuous mode on VLAN ID:%d failed error-%d\n", + vid, status); + } } } } else { - for (i = 0; i < vfl->num_elements; i++) { + /* In case of non_trusted VF, number of VLAN elements passed + * to PF for removal might be greater than number of VLANs + * filter programmed for that VF - So, use actual number of + * VLANS added earlier with add VLAN opcode. In order to avoid + * removing VLAN that doesn't exist, which result to sending + * erroneous failed message back to the VF + */ + int num_vf_vlan; + + num_vf_vlan = vf->num_vlan; + for (i = 0; i < vfl->num_elements && i < num_vf_vlan; i++) { u16 vid = vfl->vlan_id[i]; /* Make sure ice_vsi_kill_vlan is successful before * updating VLAN information */ - if (!ice_vsi_kill_vlan(vsi, vid)) { - vf->num_vlan--; + if (ice_vsi_kill_vlan(vsi, vid)) { + v_ret = VIRTCHNL_STATUS_ERR_PARAM; + goto error_param; + } + + vf->num_vlan--; + /* Disable VLAN pruning when removing VLAN */ + ice_cfg_vlan_pruning(vsi, false, false); - /* Disable VLAN pruning when removing VLAN 0 */ - if (unlikely(!vid)) - ice_cfg_vlan_pruning(vsi, false); + /* Disable Unicast/Multicast VLAN promiscuous mode */ + if (vlan_promisc) { + promisc_m = ICE_PROMISC_VLAN_TX | + ICE_PROMISC_VLAN_RX; + + ice_clear_vsi_promisc(hw, vsi->idx, + promisc_m, vid); } } } @@ -2217,10 +2439,10 @@ static int ice_vc_process_vlan_msg(struct ice_vf *vf, u8 *msg, bool add_v) error_param: /* send the response to the VF */ if (add_v) - return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_ADD_VLAN, aq_ret, + return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_ADD_VLAN, v_ret, NULL, 0); else - return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_DEL_VLAN, aq_ret, + return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_DEL_VLAN, v_ret, NULL, 0); } @@ -2229,7 +2451,7 @@ error_param: * @vf: pointer to the VF info * @msg: pointer to the msg buffer * - * Add and program guest VLAN id + * Add and program guest VLAN ID */ static int ice_vc_add_vlan_msg(struct ice_vf *vf, u8 *msg) { @@ -2241,7 +2463,7 @@ static int ice_vc_add_vlan_msg(struct ice_vf *vf, u8 *msg) * @vf: pointer to the VF info * @msg: pointer to the msg buffer * - * remove programmed guest VLAN id + * remove programmed guest VLAN ID */ static int ice_vc_remove_vlan_msg(struct ice_vf *vf, u8 *msg) { @@ -2256,22 +2478,22 @@ static int ice_vc_remove_vlan_msg(struct ice_vf *vf, u8 *msg) */ static int ice_vc_ena_vlan_stripping(struct ice_vf *vf) { - enum ice_status aq_ret = 0; + enum virtchnl_status_code v_ret = VIRTCHNL_STATUS_SUCCESS; struct ice_pf *pf = vf->pf; struct ice_vsi *vsi; if (!test_bit(ICE_VF_STATE_ACTIVE, vf->vf_states)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } vsi = pf->vsi[vf->lan_vsi_idx]; if (ice_vsi_manage_vlan_stripping(vsi, true)) - aq_ret = ICE_ERR_AQ_ERROR; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; error_param: return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_ENABLE_VLAN_STRIPPING, - aq_ret, NULL, 0); + v_ret, NULL, 0); } /** @@ -2282,22 +2504,27 @@ error_param: */ static int ice_vc_dis_vlan_stripping(struct ice_vf *vf) { - enum ice_status aq_ret = 0; + enum virtchnl_status_code v_ret = VIRTCHNL_STATUS_SUCCESS; struct ice_pf *pf = vf->pf; struct ice_vsi *vsi; if (!test_bit(ICE_VF_STATE_ACTIVE, vf->vf_states)) { - aq_ret = ICE_ERR_PARAM; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; goto error_param; } vsi = pf->vsi[vf->lan_vsi_idx]; + if (!vsi) { + v_ret = VIRTCHNL_STATUS_ERR_PARAM; + goto error_param; + } + if (ice_vsi_manage_vlan_stripping(vsi, false)) - aq_ret = ICE_ERR_AQ_ERROR; + v_ret = VIRTCHNL_STATUS_ERR_PARAM; error_param: return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_DISABLE_VLAN_STRIPPING, - aq_ret, NULL, 0); + v_ret, NULL, 0); } /** @@ -2333,7 +2560,7 @@ void ice_vc_process_vf_msg(struct ice_pf *pf, struct ice_rq_event_info *event) /* Perform basic checks on the msg */ err = virtchnl_vc_validate_vf_msg(&vf->vf_ver, v_opcode, msg, msglen); if (err) { - if (err == VIRTCHNL_ERR_PARAM) + if (err == VIRTCHNL_STATUS_ERR_PARAM) err = -EPERM; else err = -EINVAL; @@ -2355,7 +2582,8 @@ void ice_vc_process_vf_msg(struct ice_pf *pf, struct ice_rq_event_info *event) error_handler: if (err) { - ice_vc_send_msg_to_vf(vf, v_opcode, ICE_ERR_PARAM, NULL, 0); + ice_vc_send_msg_to_vf(vf, v_opcode, VIRTCHNL_STATUS_ERR_PARAM, + NULL, 0); dev_err(&pf->pdev->dev, "Invalid message from VF %d, opcode %d, len %d, error %d\n", vf_id, v_opcode, msglen, err); return; @@ -2418,7 +2646,8 @@ error_handler: default: dev_err(&pf->pdev->dev, "Unsupported opcode %d from VF %d\n", v_opcode, vf_id); - err = ice_vc_send_msg_to_vf(vf, v_opcode, ICE_ERR_NOT_IMPL, + err = ice_vc_send_msg_to_vf(vf, v_opcode, + VIRTCHNL_STATUS_ERR_NOT_SUPPORTED, NULL, 0); break; } @@ -2427,7 +2656,7 @@ error_handler: * as it is busy with pending work. */ dev_info(&pf->pdev->dev, - "PF failed to honor VF %d, opcode %d\n, error %d\n", + "PF failed to honor VF %d, opcode %d, error %d\n", vf_id, v_opcode, err); } } @@ -2440,8 +2669,8 @@ error_handler: * * return VF configuration */ -int ice_get_vf_cfg(struct net_device *netdev, int vf_id, - struct ifla_vf_info *ivi) +int +ice_get_vf_cfg(struct net_device *netdev, int vf_id, struct ifla_vf_info *ivi) { struct ice_netdev_priv *np = netdev_priv(netdev); struct ice_vsi *vsi = np->vsi; @@ -2550,9 +2779,9 @@ out: * ice_set_vf_mac * @netdev: network interface device structure * @vf_id: VF identifier - * @mac: mac address + * @mac: MAC address * - * program VF mac address + * program VF MAC address */ int ice_set_vf_mac(struct net_device *netdev, int vf_id, u8 *mac) { @@ -2579,7 +2808,7 @@ int ice_set_vf_mac(struct net_device *netdev, int vf_id, u8 *mac) return -EINVAL; } - /* copy mac into dflt_lan_addr and trigger a VF reset. The reset + /* copy MAC into dflt_lan_addr and trigger a VF reset. The reset * flow will use the updated dflt_lan_addr and add a MAC filter * using ice_add_mac. Also set pf_set_mac to indicate that the PF has * set the MAC address for this VF. @@ -2587,7 +2816,7 @@ int ice_set_vf_mac(struct net_device *netdev, int vf_id, u8 *mac) ether_addr_copy(vf->dflt_lan_addr.addr, mac); vf->pf_set_mac = true; netdev_info(netdev, - "mac on VF %d set to %pM\n. VF driver will be reinitialized\n", + "MAC on VF %d set to %pM. VF driver will be reinitialized\n", vf_id, mac); ice_vc_dis_vf(vf); @@ -2690,7 +2919,8 @@ int ice_set_vf_link_state(struct net_device *netdev, int vf_id, int link_state) ice_set_pfe_link(vf, &pfe, ls->link_speed, vf->link_up); /* Notify the VF of its new link state */ - ice_aq_send_msg_to_vf(hw, vf->vf_id, VIRTCHNL_OP_EVENT, 0, (u8 *)&pfe, + ice_aq_send_msg_to_vf(hw, vf->vf_id, VIRTCHNL_OP_EVENT, + VIRTCHNL_STATUS_SUCCESS, (u8 *)&pfe, sizeof(pfe), NULL); return 0; diff --git a/drivers/net/ethernet/intel/ice/ice_virtchnl_pf.h b/drivers/net/ethernet/intel/ice/ice_virtchnl_pf.h index 01470a8ee03a000b647fb3d1b604c8d54f7b6248..3725aea16840b7a0c4ec29455ebcc3f49e51e3ee 100644 --- a/drivers/net/ethernet/intel/ice/ice_virtchnl_pf.h +++ b/drivers/net/ethernet/intel/ice/ice_virtchnl_pf.h @@ -48,10 +48,10 @@ enum ice_virtchnl_cap { struct ice_vf { struct ice_pf *pf; - s16 vf_id; /* VF id in the PF space */ + s16 vf_id; /* VF ID in the PF space */ u32 driver_caps; /* reported by VF driver */ int first_vector_idx; /* first vector index of this VF */ - struct ice_sw *vf_sw_id; /* switch id the VF VSIs connect to */ + struct ice_sw *vf_sw_id; /* switch ID the VF VSIs connect to */ struct virtchnl_version_info vf_ver; struct virtchnl_ether_addr dflt_lan_addr; u16 port_vlan_id; @@ -59,10 +59,10 @@ struct ice_vf { u8 trusted; u16 lan_vsi_idx; /* index into PF struct */ u16 lan_vsi_num; /* ID as used by firmware */ - u64 num_mdd_events; /* number of mdd events detected */ + u64 num_mdd_events; /* number of MDD events detected */ u64 num_inval_msgs; /* number of continuous invalid msgs */ u64 num_valid_msgs; /* number of valid msgs detected */ - unsigned long vf_caps; /* vf's adv. capabilities */ + unsigned long vf_caps; /* VF's adv. capabilities */ DECLARE_BITMAP(vf_states, ICE_VF_STATES_NBITS); /* VF runtime states */ unsigned int tx_rate; /* Tx bandwidth limit in Mbps */ u8 link_forced; @@ -70,6 +70,7 @@ struct ice_vf { u8 spoofchk; u16 num_mac; u16 num_vlan; + u16 num_vf_qs; /* num of queue configured per VF */ u8 num_req_qs; /* num of queue pairs requested by VF */ }; @@ -77,8 +78,8 @@ struct ice_vf { void ice_process_vflr_event(struct ice_pf *pf); int ice_sriov_configure(struct pci_dev *pdev, int num_vfs); int ice_set_vf_mac(struct net_device *netdev, int vf_id, u8 *mac); -int ice_get_vf_cfg(struct net_device *netdev, int vf_id, - struct ifla_vf_info *ivi); +int +ice_get_vf_cfg(struct net_device *netdev, int vf_id, struct ifla_vf_info *ivi); void ice_free_vfs(struct ice_pf *pf); void ice_vc_process_vf_msg(struct ice_pf *pf, struct ice_rq_event_info *event); @@ -86,11 +87,9 @@ void ice_vc_notify_link_state(struct ice_pf *pf); void ice_vc_notify_reset(struct ice_pf *pf); bool ice_reset_all_vfs(struct ice_pf *pf, bool is_vflr); -int ice_set_vf_port_vlan(struct net_device *netdev, int vf_id, - u16 vlan_id, u8 qos, __be16 vlan_proto); - -int ice_set_vf_bw(struct net_device *netdev, int vf_id, int min_tx_rate, - int max_tx_rate); +int +ice_set_vf_port_vlan(struct net_device *netdev, int vf_id, u16 vlan_id, u8 qos, + __be16 vlan_proto); int ice_set_vf_trust(struct net_device *netdev, int vf_id, bool trusted); @@ -162,12 +161,5 @@ ice_set_vf_link_state(struct net_device __always_unused *netdev, return -EOPNOTSUPP; } -static inline int -ice_set_vf_bw(struct net_device __always_unused *netdev, - int __always_unused vf_id, int __always_unused min_tx_rate, - int __always_unused max_tx_rate) -{ - return -EOPNOTSUPP; -} #endif /* CONFIG_PCI_IOV */ #endif /* _ICE_VIRTCHNL_PF_H_ */ diff --git a/drivers/net/ethernet/intel/igb/igb_ethtool.c b/drivers/net/ethernet/intel/igb/igb_ethtool.c index c5767106824521a2e78374310cbe5f15ecc4d125..c645d9e648e03b4ca17aafbc52e65f742ca21a37 100644 --- a/drivers/net/ethernet/intel/igb/igb_ethtool.c +++ b/drivers/net/ethernet/intel/igb/igb_ethtool.c @@ -3158,8 +3158,8 @@ static int igb_set_eee(struct net_device *netdev, } else if (!edata->eee_enabled) { dev_err(&adapter->pdev->dev, "Setting EEE options are not supported with EEE disabled\n"); - return -EINVAL; - } + return -EINVAL; + } adapter->eee_advert = ethtool_adv_to_mmd_eee_adv_t(edata->advertised); if (hw->dev_spec._82575.eee_disable != !edata->eee_enabled) { diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c index 1d71ec360b1c8203e9b4f815ee561d3e4c141475..39f33afc479c1abffc9d4f1546c077d364bd26c4 100644 --- a/drivers/net/ethernet/intel/igb/igb_main.c +++ b/drivers/net/ethernet/intel/igb/igb_main.c @@ -2480,7 +2480,7 @@ static int igb_set_features(struct net_device *netdev, else igb_reset(adapter); - return 0; + return 1; } static int igb_ndo_fdb_add(struct ndmsg *ndm, struct nlattr *tb[], @@ -3452,6 +3452,9 @@ static int igb_probe(struct pci_dev *pdev, const struct pci_device_id *ent) break; } } + + dev_pm_set_driver_flags(&pdev->dev, DPM_FLAG_NEVER_SKIP); + pm_runtime_put_noidle(&pdev->dev); return 0; @@ -6026,7 +6029,7 @@ static int igb_tx_map(struct igb_ring *tx_ring, /* Make sure there is space in the ring for the next send. */ igb_maybe_stop_tx(tx_ring, DESC_NEEDED); - if (netif_xmit_stopped(txring_txq(tx_ring)) || !skb->xmit_more) { + if (netif_xmit_stopped(txring_txq(tx_ring)) || !netdev_xmit_more()) { writel(i, tx_ring->tail); } return 0; @@ -8043,7 +8046,7 @@ static struct sk_buff *igb_construct_skb(struct igb_ring *rx_ring, /* Determine available headroom for copy */ headlen = size; if (headlen > IGB_RX_HDR_LEN) - headlen = eth_get_headlen(va, IGB_RX_HDR_LEN); + headlen = eth_get_headlen(skb->dev, va, IGB_RX_HDR_LEN); /* align pull length to size of long to optimize memcpy performance */ memcpy(__skb_put(skb, headlen), va, ALIGN(headlen, sizeof(long))); diff --git a/drivers/net/ethernet/intel/igc/igc.h b/drivers/net/ethernet/intel/igc/igc.h index 80faccc34cda2d44d08e491f3843866620f70e88..0f5534ce27b013c697727cf37d2fb38d0ce9d34c 100644 --- a/drivers/net/ethernet/intel/igc/igc.h +++ b/drivers/net/ethernet/intel/igc/igc.h @@ -29,9 +29,15 @@ unsigned int igc_get_max_rss_queues(struct igc_adapter *adapter); void igc_set_flag_queue_pairs(struct igc_adapter *adapter, const u32 max_rss_queues); int igc_reinit_queues(struct igc_adapter *adapter); +void igc_write_rss_indir_tbl(struct igc_adapter *adapter); bool igc_has_link(struct igc_adapter *adapter); void igc_reset(struct igc_adapter *adapter); int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx); +int igc_add_mac_steering_filter(struct igc_adapter *adapter, + const u8 *addr, u8 queue, u8 flags); +int igc_del_mac_steering_filter(struct igc_adapter *adapter, + const u8 *addr, u8 queue, u8 flags); +void igc_update_stats(struct igc_adapter *adapter); extern char igc_driver_name[]; extern char igc_driver_version[]; @@ -51,6 +57,13 @@ extern char igc_driver_version[]; #define IGC_FLAG_VLAN_PROMISC BIT(15) #define IGC_FLAG_RX_LEGACY BIT(16) +#define IGC_FLAG_RSS_FIELD_IPV4_UDP BIT(6) +#define IGC_FLAG_RSS_FIELD_IPV6_UDP BIT(7) + +#define IGC_MRQC_ENABLE_RSS_MQ 0x00000002 +#define IGC_MRQC_RSS_FIELD_IPV4_UDP 0x00400000 +#define IGC_MRQC_RSS_FIELD_IPV6_UDP 0x00800000 + #define IGC_START_ITR 648 /* ~6000 ints/sec */ #define IGC_4K_ITR 980 #define IGC_20K_ITR 196 @@ -284,15 +297,50 @@ struct igc_q_vector { struct igc_ring ring[0] ____cacheline_internodealigned_in_smp; }; +#define MAX_ETYPE_FILTER (4 - 1) + +enum igc_filter_match_flags { + IGC_FILTER_FLAG_ETHER_TYPE = 0x1, + IGC_FILTER_FLAG_VLAN_TCI = 0x2, + IGC_FILTER_FLAG_SRC_MAC_ADDR = 0x4, + IGC_FILTER_FLAG_DST_MAC_ADDR = 0x8, +}; + +/* RX network flow classification data structure */ +struct igc_nfc_input { + /* Byte layout in order, all values with MSB first: + * match_flags - 1 byte + * etype - 2 bytes + * vlan_tci - 2 bytes + */ + u8 match_flags; + __be16 etype; + __be16 vlan_tci; + u8 src_addr[ETH_ALEN]; + u8 dst_addr[ETH_ALEN]; +}; + +struct igc_nfc_filter { + struct hlist_node nfc_node; + struct igc_nfc_input filter; + unsigned long cookie; + u16 etype_reg_index; + u16 sw_idx; + u16 action; +}; + struct igc_mac_addr { u8 addr[ETH_ALEN]; u8 queue; u8 state; /* bitmask */ }; -#define IGC_MAC_STATE_DEFAULT 0x1 -#define IGC_MAC_STATE_MODIFIED 0x2 -#define IGC_MAC_STATE_IN_USE 0x4 +#define IGC_MAC_STATE_DEFAULT 0x1 +#define IGC_MAC_STATE_IN_USE 0x2 +#define IGC_MAC_STATE_SRC_ADDR 0x4 +#define IGC_MAC_STATE_QUEUE_STEERING 0x8 + +#define IGC_MAX_RXNFC_FILTERS 16 /* Board specific private data structure */ struct igc_adapter { @@ -356,12 +404,22 @@ struct igc_adapter { u16 tx_ring_count; u16 rx_ring_count; + u32 tx_hwtstamp_timeouts; + u32 tx_hwtstamp_skipped; + u32 rx_hwtstamp_cleared; u32 *shadow_vfta; u32 rss_queues; + u32 rss_indir_tbl_init; + + /* RX network flow classification support */ + struct hlist_head nfc_filter_list; + struct hlist_head cls_flower_list; + unsigned int nfc_filter_count; /* lock for RX network flow classification filter */ spinlock_t nfc_lock; + bool etype_bitmap[MAX_ETYPE_FILTER]; struct igc_mac_addr *mac_table; @@ -447,6 +505,10 @@ static inline s32 igc_read_phy_reg(struct igc_hw *hw, u32 offset, u16 *data) /* forward declaration */ void igc_reinit_locked(struct igc_adapter *); +int igc_add_filter(struct igc_adapter *adapter, + struct igc_nfc_filter *input); +int igc_erase_filter(struct igc_adapter *adapter, + struct igc_nfc_filter *input); #define igc_rx_pg_size(_ring) (PAGE_SIZE << igc_rx_pg_order(_ring)) diff --git a/drivers/net/ethernet/intel/igc/igc_base.h b/drivers/net/ethernet/intel/igc/igc_base.h index 76d4991d728443bdaaf15ddfd68708f6a133e346..58d1109d7f3fe8ecb509714585e810989ef6beab 100644 --- a/drivers/net/ethernet/intel/igc/igc_base.h +++ b/drivers/net/ethernet/intel/igc/igc_base.h @@ -1,8 +1,8 @@ /* SPDX-License-Identifier: GPL-2.0 */ /* Copyright (c) 2018 Intel Corporation */ -#ifndef _IGC_BASE_H -#define _IGC_BASE_H +#ifndef _IGC_BASE_H_ +#define _IGC_BASE_H_ /* forward declaration */ void igc_rx_fifo_flush_base(struct igc_hw *hw); diff --git a/drivers/net/ethernet/intel/igc/igc_defines.h b/drivers/net/ethernet/intel/igc/igc_defines.h index 7d1bdcd1225a376b09ffba76266abcdbf4595b29..a9a30268de59d2b986617f06fc5840ed82ec299c 100644 --- a/drivers/net/ethernet/intel/igc/igc_defines.h +++ b/drivers/net/ethernet/intel/igc/igc_defines.h @@ -310,6 +310,12 @@ IGC_RXDEXT_STATERR_CXE | \ IGC_RXDEXT_STATERR_RXE) +#define IGC_MRQC_RSS_FIELD_IPV4_TCP 0x00010000 +#define IGC_MRQC_RSS_FIELD_IPV4 0x00020000 +#define IGC_MRQC_RSS_FIELD_IPV6_TCP_EX 0x00040000 +#define IGC_MRQC_RSS_FIELD_IPV6 0x00100000 +#define IGC_MRQC_RSS_FIELD_IPV6_TCP 0x00200000 + /* Header split receive */ #define IGC_RFCTL_IPV6_EX_DIS 0x00010000 #define IGC_RFCTL_LEF 0x00040000 @@ -325,6 +331,10 @@ #define I225_RXPBSIZE_DEFAULT 0x000000A2 /* RXPBSIZE default */ #define I225_TXPBSIZE_DEFAULT 0x04000014 /* TXPBSIZE default */ +/* Receive Checksum Control */ +#define IGC_RXCSUM_CRCOFL 0x00000800 /* CRC32 offload enable */ +#define IGC_RXCSUM_PCSD 0x00002000 /* packet checksum disabled */ + /* GPY211 - I225 defines */ #define GPY_MMD_MASK 0xFFFF0000 #define GPY_MMD_SHIFT 16 @@ -390,4 +400,11 @@ #define IGC_N0_QUEUE -1 +#define IGC_MAX_MAC_HDR_LEN 127 +#define IGC_MAX_NETWORK_HDR_LEN 511 + +#define IGC_VLAPQF_QUEUE_SEL(_n, q_idx) ((q_idx) << ((_n) * 4)) +#define IGC_VLAPQF_P_VALID(_n) (0x1 << (3 + (_n) * 4)) +#define IGC_VLAPQF_QUEUE_MASK 0x03 + #endif /* _IGC_DEFINES_H_ */ diff --git a/drivers/net/ethernet/intel/igc/igc_ethtool.c b/drivers/net/ethernet/intel/igc/igc_ethtool.c index eff37a6c0afae34b3a224e56bc0977580a0ebdd4..ac98f1d9689218107ba72ef045ff62d2defea1b6 100644 --- a/drivers/net/ethernet/intel/igc/igc_ethtool.c +++ b/drivers/net/ethernet/intel/igc/igc_ethtool.c @@ -2,10 +2,120 @@ /* Copyright (c) 2018 Intel Corporation */ /* ethtool support for igc */ +#include #include #include "igc.h" +/* forward declaration */ +struct igc_stats { + char stat_string[ETH_GSTRING_LEN]; + int sizeof_stat; + int stat_offset; +}; + +#define IGC_STAT(_name, _stat) { \ + .stat_string = _name, \ + .sizeof_stat = FIELD_SIZEOF(struct igc_adapter, _stat), \ + .stat_offset = offsetof(struct igc_adapter, _stat) \ +} + +static const struct igc_stats igc_gstrings_stats[] = { + IGC_STAT("rx_packets", stats.gprc), + IGC_STAT("tx_packets", stats.gptc), + IGC_STAT("rx_bytes", stats.gorc), + IGC_STAT("tx_bytes", stats.gotc), + IGC_STAT("rx_broadcast", stats.bprc), + IGC_STAT("tx_broadcast", stats.bptc), + IGC_STAT("rx_multicast", stats.mprc), + IGC_STAT("tx_multicast", stats.mptc), + IGC_STAT("multicast", stats.mprc), + IGC_STAT("collisions", stats.colc), + IGC_STAT("rx_crc_errors", stats.crcerrs), + IGC_STAT("rx_no_buffer_count", stats.rnbc), + IGC_STAT("rx_missed_errors", stats.mpc), + IGC_STAT("tx_aborted_errors", stats.ecol), + IGC_STAT("tx_carrier_errors", stats.tncrs), + IGC_STAT("tx_window_errors", stats.latecol), + IGC_STAT("tx_abort_late_coll", stats.latecol), + IGC_STAT("tx_deferred_ok", stats.dc), + IGC_STAT("tx_single_coll_ok", stats.scc), + IGC_STAT("tx_multi_coll_ok", stats.mcc), + IGC_STAT("tx_timeout_count", tx_timeout_count), + IGC_STAT("rx_long_length_errors", stats.roc), + IGC_STAT("rx_short_length_errors", stats.ruc), + IGC_STAT("rx_align_errors", stats.algnerrc), + IGC_STAT("tx_tcp_seg_good", stats.tsctc), + IGC_STAT("tx_tcp_seg_failed", stats.tsctfc), + IGC_STAT("rx_flow_control_xon", stats.xonrxc), + IGC_STAT("rx_flow_control_xoff", stats.xoffrxc), + IGC_STAT("tx_flow_control_xon", stats.xontxc), + IGC_STAT("tx_flow_control_xoff", stats.xofftxc), + IGC_STAT("rx_long_byte_count", stats.gorc), + IGC_STAT("tx_dma_out_of_sync", stats.doosync), + IGC_STAT("tx_smbus", stats.mgptc), + IGC_STAT("rx_smbus", stats.mgprc), + IGC_STAT("dropped_smbus", stats.mgpdc), + IGC_STAT("os2bmc_rx_by_bmc", stats.o2bgptc), + IGC_STAT("os2bmc_tx_by_bmc", stats.b2ospc), + IGC_STAT("os2bmc_tx_by_host", stats.o2bspc), + IGC_STAT("os2bmc_rx_by_host", stats.b2ogprc), + IGC_STAT("tx_hwtstamp_timeouts", tx_hwtstamp_timeouts), + IGC_STAT("tx_hwtstamp_skipped", tx_hwtstamp_skipped), + IGC_STAT("rx_hwtstamp_cleared", rx_hwtstamp_cleared), +}; + +#define IGC_NETDEV_STAT(_net_stat) { \ + .stat_string = __stringify(_net_stat), \ + .sizeof_stat = FIELD_SIZEOF(struct rtnl_link_stats64, _net_stat), \ + .stat_offset = offsetof(struct rtnl_link_stats64, _net_stat) \ +} + +static const struct igc_stats igc_gstrings_net_stats[] = { + IGC_NETDEV_STAT(rx_errors), + IGC_NETDEV_STAT(tx_errors), + IGC_NETDEV_STAT(tx_dropped), + IGC_NETDEV_STAT(rx_length_errors), + IGC_NETDEV_STAT(rx_over_errors), + IGC_NETDEV_STAT(rx_frame_errors), + IGC_NETDEV_STAT(rx_fifo_errors), + IGC_NETDEV_STAT(tx_fifo_errors), + IGC_NETDEV_STAT(tx_heartbeat_errors) +}; + +enum igc_diagnostics_results { + TEST_REG = 0, + TEST_EEP, + TEST_IRQ, + TEST_LOOP, + TEST_LINK +}; + +static const char igc_gstrings_test[][ETH_GSTRING_LEN] = { + [TEST_REG] = "Register test (offline)", + [TEST_EEP] = "Eeprom test (offline)", + [TEST_IRQ] = "Interrupt test (offline)", + [TEST_LOOP] = "Loopback test (offline)", + [TEST_LINK] = "Link test (on/offline)" +}; + +#define IGC_TEST_LEN (sizeof(igc_gstrings_test) / ETH_GSTRING_LEN) + +#define IGC_GLOBAL_STATS_LEN \ + (sizeof(igc_gstrings_stats) / sizeof(struct igc_stats)) +#define IGC_NETDEV_STATS_LEN \ + (sizeof(igc_gstrings_net_stats) / sizeof(struct igc_stats)) +#define IGC_RX_QUEUE_STATS_LEN \ + (sizeof(struct igc_rx_queue_stats) / sizeof(u64)) +#define IGC_TX_QUEUE_STATS_LEN 3 /* packets, bytes, restart_queue */ +#define IGC_QUEUE_STATS_LEN \ + ((((struct igc_adapter *)netdev_priv(netdev))->num_rx_queues * \ + IGC_RX_QUEUE_STATS_LEN) + \ + (((struct igc_adapter *)netdev_priv(netdev))->num_tx_queues * \ + IGC_TX_QUEUE_STATS_LEN)) +#define IGC_STATS_LEN \ + (IGC_GLOBAL_STATS_LEN + IGC_NETDEV_STATS_LEN + IGC_QUEUE_STATS_LEN) + static const char igc_priv_flags_strings[][ETH_GSTRING_LEN] = { #define IGC_PRIV_FLAGS_LEGACY_RX BIT(0) "legacy-rx", @@ -545,6 +655,127 @@ static int igc_set_pauseparam(struct net_device *netdev, return retval; } +static void igc_get_strings(struct net_device *netdev, u32 stringset, u8 *data) +{ + struct igc_adapter *adapter = netdev_priv(netdev); + u8 *p = data; + int i; + + switch (stringset) { + case ETH_SS_TEST: + memcpy(data, *igc_gstrings_test, + IGC_TEST_LEN * ETH_GSTRING_LEN); + break; + case ETH_SS_STATS: + for (i = 0; i < IGC_GLOBAL_STATS_LEN; i++) { + memcpy(p, igc_gstrings_stats[i].stat_string, + ETH_GSTRING_LEN); + p += ETH_GSTRING_LEN; + } + for (i = 0; i < IGC_NETDEV_STATS_LEN; i++) { + memcpy(p, igc_gstrings_net_stats[i].stat_string, + ETH_GSTRING_LEN); + p += ETH_GSTRING_LEN; + } + for (i = 0; i < adapter->num_tx_queues; i++) { + sprintf(p, "tx_queue_%u_packets", i); + p += ETH_GSTRING_LEN; + sprintf(p, "tx_queue_%u_bytes", i); + p += ETH_GSTRING_LEN; + sprintf(p, "tx_queue_%u_restart", i); + p += ETH_GSTRING_LEN; + } + for (i = 0; i < adapter->num_rx_queues; i++) { + sprintf(p, "rx_queue_%u_packets", i); + p += ETH_GSTRING_LEN; + sprintf(p, "rx_queue_%u_bytes", i); + p += ETH_GSTRING_LEN; + sprintf(p, "rx_queue_%u_drops", i); + p += ETH_GSTRING_LEN; + sprintf(p, "rx_queue_%u_csum_err", i); + p += ETH_GSTRING_LEN; + sprintf(p, "rx_queue_%u_alloc_failed", i); + p += ETH_GSTRING_LEN; + } + /* BUG_ON(p - data != IGC_STATS_LEN * ETH_GSTRING_LEN); */ + break; + case ETH_SS_PRIV_FLAGS: + memcpy(data, igc_priv_flags_strings, + IGC_PRIV_FLAGS_STR_LEN * ETH_GSTRING_LEN); + break; + } +} + +static int igc_get_sset_count(struct net_device *netdev, int sset) +{ + switch (sset) { + case ETH_SS_STATS: + return IGC_STATS_LEN; + case ETH_SS_TEST: + return IGC_TEST_LEN; + case ETH_SS_PRIV_FLAGS: + return IGC_PRIV_FLAGS_STR_LEN; + default: + return -ENOTSUPP; + } +} + +static void igc_get_ethtool_stats(struct net_device *netdev, + struct ethtool_stats *stats, u64 *data) +{ + struct igc_adapter *adapter = netdev_priv(netdev); + struct rtnl_link_stats64 *net_stats = &adapter->stats64; + unsigned int start; + struct igc_ring *ring; + int i, j; + char *p; + + spin_lock(&adapter->stats64_lock); + igc_update_stats(adapter); + + for (i = 0; i < IGC_GLOBAL_STATS_LEN; i++) { + p = (char *)adapter + igc_gstrings_stats[i].stat_offset; + data[i] = (igc_gstrings_stats[i].sizeof_stat == + sizeof(u64)) ? *(u64 *)p : *(u32 *)p; + } + for (j = 0; j < IGC_NETDEV_STATS_LEN; j++, i++) { + p = (char *)net_stats + igc_gstrings_net_stats[j].stat_offset; + data[i] = (igc_gstrings_net_stats[j].sizeof_stat == + sizeof(u64)) ? *(u64 *)p : *(u32 *)p; + } + for (j = 0; j < adapter->num_tx_queues; j++) { + u64 restart2; + + ring = adapter->tx_ring[j]; + do { + start = u64_stats_fetch_begin_irq(&ring->tx_syncp); + data[i] = ring->tx_stats.packets; + data[i + 1] = ring->tx_stats.bytes; + data[i + 2] = ring->tx_stats.restart_queue; + } while (u64_stats_fetch_retry_irq(&ring->tx_syncp, start)); + do { + start = u64_stats_fetch_begin_irq(&ring->tx_syncp2); + restart2 = ring->tx_stats.restart_queue2; + } while (u64_stats_fetch_retry_irq(&ring->tx_syncp2, start)); + data[i + 2] += restart2; + + i += IGC_TX_QUEUE_STATS_LEN; + } + for (j = 0; j < adapter->num_rx_queues; j++) { + ring = adapter->rx_ring[j]; + do { + start = u64_stats_fetch_begin_irq(&ring->rx_syncp); + data[i] = ring->rx_stats.packets; + data[i + 1] = ring->rx_stats.bytes; + data[i + 2] = ring->rx_stats.drops; + data[i + 3] = ring->rx_stats.csum_err; + data[i + 4] = ring->rx_stats.alloc_failed; + } while (u64_stats_fetch_retry_irq(&ring->rx_syncp, start)); + i += IGC_RX_QUEUE_STATS_LEN; + } + spin_unlock(&adapter->stats64_lock); +} + static int igc_get_coalesce(struct net_device *netdev, struct ethtool_coalesce *ec) { @@ -643,6 +874,605 @@ static int igc_set_coalesce(struct net_device *netdev, return 0; } +#define ETHER_TYPE_FULL_MASK ((__force __be16)~0) +static int igc_get_ethtool_nfc_entry(struct igc_adapter *adapter, + struct ethtool_rxnfc *cmd) +{ + struct ethtool_rx_flow_spec *fsp = &cmd->fs; + struct igc_nfc_filter *rule = NULL; + + /* report total rule count */ + cmd->data = IGC_MAX_RXNFC_FILTERS; + + hlist_for_each_entry(rule, &adapter->nfc_filter_list, nfc_node) { + if (fsp->location <= rule->sw_idx) + break; + } + + if (!rule || fsp->location != rule->sw_idx) + return -EINVAL; + + if (rule->filter.match_flags) { + fsp->flow_type = ETHER_FLOW; + fsp->ring_cookie = rule->action; + if (rule->filter.match_flags & IGC_FILTER_FLAG_ETHER_TYPE) { + fsp->h_u.ether_spec.h_proto = rule->filter.etype; + fsp->m_u.ether_spec.h_proto = ETHER_TYPE_FULL_MASK; + } + if (rule->filter.match_flags & IGC_FILTER_FLAG_VLAN_TCI) { + fsp->flow_type |= FLOW_EXT; + fsp->h_ext.vlan_tci = rule->filter.vlan_tci; + fsp->m_ext.vlan_tci = htons(VLAN_PRIO_MASK); + } + if (rule->filter.match_flags & IGC_FILTER_FLAG_DST_MAC_ADDR) { + ether_addr_copy(fsp->h_u.ether_spec.h_dest, + rule->filter.dst_addr); + /* As we only support matching by the full + * mask, return the mask to userspace + */ + eth_broadcast_addr(fsp->m_u.ether_spec.h_dest); + } + if (rule->filter.match_flags & IGC_FILTER_FLAG_SRC_MAC_ADDR) { + ether_addr_copy(fsp->h_u.ether_spec.h_source, + rule->filter.src_addr); + /* As we only support matching by the full + * mask, return the mask to userspace + */ + eth_broadcast_addr(fsp->m_u.ether_spec.h_source); + } + + return 0; + } + return -EINVAL; +} + +static int igc_get_ethtool_nfc_all(struct igc_adapter *adapter, + struct ethtool_rxnfc *cmd, + u32 *rule_locs) +{ + struct igc_nfc_filter *rule; + int cnt = 0; + + /* report total rule count */ + cmd->data = IGC_MAX_RXNFC_FILTERS; + + hlist_for_each_entry(rule, &adapter->nfc_filter_list, nfc_node) { + if (cnt == cmd->rule_cnt) + return -EMSGSIZE; + rule_locs[cnt] = rule->sw_idx; + cnt++; + } + + cmd->rule_cnt = cnt; + + return 0; +} + +static int igc_get_rss_hash_opts(struct igc_adapter *adapter, + struct ethtool_rxnfc *cmd) +{ + cmd->data = 0; + + /* Report default options for RSS on igc */ + switch (cmd->flow_type) { + case TCP_V4_FLOW: + cmd->data |= RXH_L4_B_0_1 | RXH_L4_B_2_3; + /* Fall through */ + case UDP_V4_FLOW: + if (adapter->flags & IGC_FLAG_RSS_FIELD_IPV4_UDP) + cmd->data |= RXH_L4_B_0_1 | RXH_L4_B_2_3; + /* Fall through */ + case SCTP_V4_FLOW: + /* Fall through */ + case AH_ESP_V4_FLOW: + /* Fall through */ + case AH_V4_FLOW: + /* Fall through */ + case ESP_V4_FLOW: + /* Fall through */ + case IPV4_FLOW: + cmd->data |= RXH_IP_SRC | RXH_IP_DST; + break; + case TCP_V6_FLOW: + cmd->data |= RXH_L4_B_0_1 | RXH_L4_B_2_3; + /* Fall through */ + case UDP_V6_FLOW: + if (adapter->flags & IGC_FLAG_RSS_FIELD_IPV6_UDP) + cmd->data |= RXH_L4_B_0_1 | RXH_L4_B_2_3; + /* Fall through */ + case SCTP_V6_FLOW: + /* Fall through */ + case AH_ESP_V6_FLOW: + /* Fall through */ + case AH_V6_FLOW: + /* Fall through */ + case ESP_V6_FLOW: + /* Fall through */ + case IPV6_FLOW: + cmd->data |= RXH_IP_SRC | RXH_IP_DST; + break; + default: + return -EINVAL; + } + + return 0; +} + +static int igc_get_rxnfc(struct net_device *dev, struct ethtool_rxnfc *cmd, + u32 *rule_locs) +{ + struct igc_adapter *adapter = netdev_priv(dev); + int ret = -EOPNOTSUPP; + + switch (cmd->cmd) { + case ETHTOOL_GRXRINGS: + cmd->data = adapter->num_rx_queues; + ret = 0; + break; + case ETHTOOL_GRXCLSRLCNT: + cmd->rule_cnt = adapter->nfc_filter_count; + ret = 0; + break; + case ETHTOOL_GRXCLSRULE: + ret = igc_get_ethtool_nfc_entry(adapter, cmd); + break; + case ETHTOOL_GRXCLSRLALL: + ret = igc_get_ethtool_nfc_all(adapter, cmd, rule_locs); + break; + case ETHTOOL_GRXFH: + ret = igc_get_rss_hash_opts(adapter, cmd); + break; + default: + break; + } + + return ret; +} + +#define UDP_RSS_FLAGS (IGC_FLAG_RSS_FIELD_IPV4_UDP | \ + IGC_FLAG_RSS_FIELD_IPV6_UDP) +static int igc_set_rss_hash_opt(struct igc_adapter *adapter, + struct ethtool_rxnfc *nfc) +{ + u32 flags = adapter->flags; + + /* RSS does not support anything other than hashing + * to queues on src and dst IPs and ports + */ + if (nfc->data & ~(RXH_IP_SRC | RXH_IP_DST | + RXH_L4_B_0_1 | RXH_L4_B_2_3)) + return -EINVAL; + + switch (nfc->flow_type) { + case TCP_V4_FLOW: + case TCP_V6_FLOW: + if (!(nfc->data & RXH_IP_SRC) || + !(nfc->data & RXH_IP_DST) || + !(nfc->data & RXH_L4_B_0_1) || + !(nfc->data & RXH_L4_B_2_3)) + return -EINVAL; + break; + case UDP_V4_FLOW: + if (!(nfc->data & RXH_IP_SRC) || + !(nfc->data & RXH_IP_DST)) + return -EINVAL; + switch (nfc->data & (RXH_L4_B_0_1 | RXH_L4_B_2_3)) { + case 0: + flags &= ~IGC_FLAG_RSS_FIELD_IPV4_UDP; + break; + case (RXH_L4_B_0_1 | RXH_L4_B_2_3): + flags |= IGC_FLAG_RSS_FIELD_IPV4_UDP; + break; + default: + return -EINVAL; + } + break; + case UDP_V6_FLOW: + if (!(nfc->data & RXH_IP_SRC) || + !(nfc->data & RXH_IP_DST)) + return -EINVAL; + switch (nfc->data & (RXH_L4_B_0_1 | RXH_L4_B_2_3)) { + case 0: + flags &= ~IGC_FLAG_RSS_FIELD_IPV6_UDP; + break; + case (RXH_L4_B_0_1 | RXH_L4_B_2_3): + flags |= IGC_FLAG_RSS_FIELD_IPV6_UDP; + break; + default: + return -EINVAL; + } + break; + case AH_ESP_V4_FLOW: + case AH_V4_FLOW: + case ESP_V4_FLOW: + case SCTP_V4_FLOW: + case AH_ESP_V6_FLOW: + case AH_V6_FLOW: + case ESP_V6_FLOW: + case SCTP_V6_FLOW: + if (!(nfc->data & RXH_IP_SRC) || + !(nfc->data & RXH_IP_DST) || + (nfc->data & RXH_L4_B_0_1) || + (nfc->data & RXH_L4_B_2_3)) + return -EINVAL; + break; + default: + return -EINVAL; + } + + /* if we changed something we need to update flags */ + if (flags != adapter->flags) { + struct igc_hw *hw = &adapter->hw; + u32 mrqc = rd32(IGC_MRQC); + + if ((flags & UDP_RSS_FLAGS) && + !(adapter->flags & UDP_RSS_FLAGS)) + dev_err(&adapter->pdev->dev, + "enabling UDP RSS: fragmented packets may arrive out of order to the stack above\n"); + + adapter->flags = flags; + + /* Perform hash on these packet types */ + mrqc |= IGC_MRQC_RSS_FIELD_IPV4 | + IGC_MRQC_RSS_FIELD_IPV4_TCP | + IGC_MRQC_RSS_FIELD_IPV6 | + IGC_MRQC_RSS_FIELD_IPV6_TCP; + + mrqc &= ~(IGC_MRQC_RSS_FIELD_IPV4_UDP | + IGC_MRQC_RSS_FIELD_IPV6_UDP); + + if (flags & IGC_FLAG_RSS_FIELD_IPV4_UDP) + mrqc |= IGC_MRQC_RSS_FIELD_IPV4_UDP; + + if (flags & IGC_FLAG_RSS_FIELD_IPV6_UDP) + mrqc |= IGC_MRQC_RSS_FIELD_IPV6_UDP; + + wr32(IGC_MRQC, mrqc); + } + + return 0; +} + +static int igc_rxnfc_write_etype_filter(struct igc_adapter *adapter, + struct igc_nfc_filter *input) +{ + struct igc_hw *hw = &adapter->hw; + u8 i; + u32 etqf; + u16 etype; + + /* find an empty etype filter register */ + for (i = 0; i < MAX_ETYPE_FILTER; ++i) { + if (!adapter->etype_bitmap[i]) + break; + } + if (i == MAX_ETYPE_FILTER) { + dev_err(&adapter->pdev->dev, "ethtool -N: etype filters are all used.\n"); + return -EINVAL; + } + + adapter->etype_bitmap[i] = true; + + etqf = rd32(IGC_ETQF(i)); + etype = ntohs(input->filter.etype & ETHER_TYPE_FULL_MASK); + + etqf |= IGC_ETQF_FILTER_ENABLE; + etqf &= ~IGC_ETQF_ETYPE_MASK; + etqf |= (etype & IGC_ETQF_ETYPE_MASK); + + etqf &= ~IGC_ETQF_QUEUE_MASK; + etqf |= ((input->action << IGC_ETQF_QUEUE_SHIFT) + & IGC_ETQF_QUEUE_MASK); + etqf |= IGC_ETQF_QUEUE_ENABLE; + + wr32(IGC_ETQF(i), etqf); + + input->etype_reg_index = i; + + return 0; +} + +static int igc_rxnfc_write_vlan_prio_filter(struct igc_adapter *adapter, + struct igc_nfc_filter *input) +{ + struct igc_hw *hw = &adapter->hw; + u8 vlan_priority; + u16 queue_index; + u32 vlapqf; + + vlapqf = rd32(IGC_VLAPQF); + vlan_priority = (ntohs(input->filter.vlan_tci) & VLAN_PRIO_MASK) + >> VLAN_PRIO_SHIFT; + queue_index = (vlapqf >> (vlan_priority * 4)) & IGC_VLAPQF_QUEUE_MASK; + + /* check whether this vlan prio is already set */ + if (vlapqf & IGC_VLAPQF_P_VALID(vlan_priority) && + queue_index != input->action) { + dev_err(&adapter->pdev->dev, "ethtool rxnfc set vlan prio filter failed.\n"); + return -EEXIST; + } + + vlapqf |= IGC_VLAPQF_P_VALID(vlan_priority); + vlapqf |= IGC_VLAPQF_QUEUE_SEL(vlan_priority, input->action); + + wr32(IGC_VLAPQF, vlapqf); + + return 0; +} + +int igc_add_filter(struct igc_adapter *adapter, struct igc_nfc_filter *input) +{ + struct igc_hw *hw = &adapter->hw; + int err = -EINVAL; + + if (hw->mac.type == igc_i225 && + !(input->filter.match_flags & ~IGC_FILTER_FLAG_SRC_MAC_ADDR)) { + dev_err(&adapter->pdev->dev, + "i225 doesn't support flow classification rules specifying only source addresses.\n"); + return -EOPNOTSUPP; + } + + if (input->filter.match_flags & IGC_FILTER_FLAG_ETHER_TYPE) { + err = igc_rxnfc_write_etype_filter(adapter, input); + if (err) + return err; + } + + if (input->filter.match_flags & IGC_FILTER_FLAG_DST_MAC_ADDR) { + err = igc_add_mac_steering_filter(adapter, + input->filter.dst_addr, + input->action, 0); + err = min_t(int, err, 0); + if (err) + return err; + } + + if (input->filter.match_flags & IGC_FILTER_FLAG_SRC_MAC_ADDR) { + err = igc_add_mac_steering_filter(adapter, + input->filter.src_addr, + input->action, + IGC_MAC_STATE_SRC_ADDR); + err = min_t(int, err, 0); + if (err) + return err; + } + + if (input->filter.match_flags & IGC_FILTER_FLAG_VLAN_TCI) + err = igc_rxnfc_write_vlan_prio_filter(adapter, input); + + return err; +} + +static void igc_clear_etype_filter_regs(struct igc_adapter *adapter, + u16 reg_index) +{ + struct igc_hw *hw = &adapter->hw; + u32 etqf = rd32(IGC_ETQF(reg_index)); + + etqf &= ~IGC_ETQF_QUEUE_ENABLE; + etqf &= ~IGC_ETQF_QUEUE_MASK; + etqf &= ~IGC_ETQF_FILTER_ENABLE; + + wr32(IGC_ETQF(reg_index), etqf); + + adapter->etype_bitmap[reg_index] = false; +} + +static void igc_clear_vlan_prio_filter(struct igc_adapter *adapter, + u16 vlan_tci) +{ + struct igc_hw *hw = &adapter->hw; + u8 vlan_priority; + u32 vlapqf; + + vlan_priority = (vlan_tci & VLAN_PRIO_MASK) >> VLAN_PRIO_SHIFT; + + vlapqf = rd32(IGC_VLAPQF); + vlapqf &= ~IGC_VLAPQF_P_VALID(vlan_priority); + vlapqf &= ~IGC_VLAPQF_QUEUE_SEL(vlan_priority, + IGC_VLAPQF_QUEUE_MASK); + + wr32(IGC_VLAPQF, vlapqf); +} + +int igc_erase_filter(struct igc_adapter *adapter, struct igc_nfc_filter *input) +{ + if (input->filter.match_flags & IGC_FILTER_FLAG_ETHER_TYPE) + igc_clear_etype_filter_regs(adapter, + input->etype_reg_index); + + if (input->filter.match_flags & IGC_FILTER_FLAG_VLAN_TCI) + igc_clear_vlan_prio_filter(adapter, + ntohs(input->filter.vlan_tci)); + + if (input->filter.match_flags & IGC_FILTER_FLAG_SRC_MAC_ADDR) + igc_del_mac_steering_filter(adapter, input->filter.src_addr, + input->action, + IGC_MAC_STATE_SRC_ADDR); + + if (input->filter.match_flags & IGC_FILTER_FLAG_DST_MAC_ADDR) + igc_del_mac_steering_filter(adapter, input->filter.dst_addr, + input->action, 0); + + return 0; +} + +static int igc_update_ethtool_nfc_entry(struct igc_adapter *adapter, + struct igc_nfc_filter *input, + u16 sw_idx) +{ + struct igc_nfc_filter *rule, *parent; + int err = -EINVAL; + + parent = NULL; + rule = NULL; + + hlist_for_each_entry(rule, &adapter->nfc_filter_list, nfc_node) { + /* hash found, or no matching entry */ + if (rule->sw_idx >= sw_idx) + break; + parent = rule; + } + + /* if there is an old rule occupying our place remove it */ + if (rule && rule->sw_idx == sw_idx) { + if (!input) + err = igc_erase_filter(adapter, rule); + + hlist_del(&rule->nfc_node); + kfree(rule); + adapter->nfc_filter_count--; + } + + /* If no input this was a delete, err should be 0 if a rule was + * successfully found and removed from the list else -EINVAL + */ + if (!input) + return err; + + /* initialize node */ + INIT_HLIST_NODE(&input->nfc_node); + + /* add filter to the list */ + if (parent) + hlist_add_behind(&input->nfc_node, &parent->nfc_node); + else + hlist_add_head(&input->nfc_node, &adapter->nfc_filter_list); + + /* update counts */ + adapter->nfc_filter_count++; + + return 0; +} + +static int igc_add_ethtool_nfc_entry(struct igc_adapter *adapter, + struct ethtool_rxnfc *cmd) +{ + struct net_device *netdev = adapter->netdev; + struct ethtool_rx_flow_spec *fsp = + (struct ethtool_rx_flow_spec *)&cmd->fs; + struct igc_nfc_filter *input, *rule; + int err = 0; + + if (!(netdev->hw_features & NETIF_F_NTUPLE)) + return -EOPNOTSUPP; + + /* Don't allow programming if the action is a queue greater than + * the number of online Rx queues. + */ + if (fsp->ring_cookie == RX_CLS_FLOW_DISC || + fsp->ring_cookie >= adapter->num_rx_queues) { + dev_err(&adapter->pdev->dev, "ethtool -N: The specified action is invalid\n"); + return -EINVAL; + } + + /* Don't allow indexes to exist outside of available space */ + if (fsp->location >= IGC_MAX_RXNFC_FILTERS) { + dev_err(&adapter->pdev->dev, "Location out of range\n"); + return -EINVAL; + } + + if ((fsp->flow_type & ~FLOW_EXT) != ETHER_FLOW) + return -EINVAL; + + input = kzalloc(sizeof(*input), GFP_KERNEL); + if (!input) + return -ENOMEM; + + if (fsp->m_u.ether_spec.h_proto == ETHER_TYPE_FULL_MASK) { + input->filter.etype = fsp->h_u.ether_spec.h_proto; + input->filter.match_flags = IGC_FILTER_FLAG_ETHER_TYPE; + } + + /* Only support matching addresses by the full mask */ + if (is_broadcast_ether_addr(fsp->m_u.ether_spec.h_source)) { + input->filter.match_flags |= IGC_FILTER_FLAG_SRC_MAC_ADDR; + ether_addr_copy(input->filter.src_addr, + fsp->h_u.ether_spec.h_source); + } + + /* Only support matching addresses by the full mask */ + if (is_broadcast_ether_addr(fsp->m_u.ether_spec.h_dest)) { + input->filter.match_flags |= IGC_FILTER_FLAG_DST_MAC_ADDR; + ether_addr_copy(input->filter.dst_addr, + fsp->h_u.ether_spec.h_dest); + } + + if ((fsp->flow_type & FLOW_EXT) && fsp->m_ext.vlan_tci) { + if (fsp->m_ext.vlan_tci != htons(VLAN_PRIO_MASK)) { + err = -EINVAL; + goto err_out; + } + input->filter.vlan_tci = fsp->h_ext.vlan_tci; + input->filter.match_flags |= IGC_FILTER_FLAG_VLAN_TCI; + } + + input->action = fsp->ring_cookie; + input->sw_idx = fsp->location; + + spin_lock(&adapter->nfc_lock); + + hlist_for_each_entry(rule, &adapter->nfc_filter_list, nfc_node) { + if (!memcmp(&input->filter, &rule->filter, + sizeof(input->filter))) { + err = -EEXIST; + dev_err(&adapter->pdev->dev, + "ethtool: this filter is already set\n"); + goto err_out_w_lock; + } + } + + err = igc_add_filter(adapter, input); + if (err) + goto err_out_w_lock; + + igc_update_ethtool_nfc_entry(adapter, input, input->sw_idx); + + spin_unlock(&adapter->nfc_lock); + return 0; + +err_out_w_lock: + spin_unlock(&adapter->nfc_lock); +err_out: + kfree(input); + return err; +} + +static int igc_del_ethtool_nfc_entry(struct igc_adapter *adapter, + struct ethtool_rxnfc *cmd) +{ + struct ethtool_rx_flow_spec *fsp = + (struct ethtool_rx_flow_spec *)&cmd->fs; + int err; + + spin_lock(&adapter->nfc_lock); + err = igc_update_ethtool_nfc_entry(adapter, NULL, fsp->location); + spin_unlock(&adapter->nfc_lock); + + return err; +} + +static int igc_set_rxnfc(struct net_device *dev, struct ethtool_rxnfc *cmd) +{ + struct igc_adapter *adapter = netdev_priv(dev); + int ret = -EOPNOTSUPP; + + switch (cmd->cmd) { + case ETHTOOL_SRXFH: + ret = igc_set_rss_hash_opt(adapter, cmd); + break; + case ETHTOOL_SRXCLSRLINS: + ret = igc_add_ethtool_nfc_entry(adapter, cmd); + break; + case ETHTOOL_SRXCLSRLDEL: + ret = igc_del_ethtool_nfc_entry(adapter, cmd); + default: + break; + } + + return ret; +} + void igc_write_rss_indir_tbl(struct igc_adapter *adapter) { struct igc_hw *hw = &adapter->hw; @@ -885,17 +1715,13 @@ static int igc_get_link_ksettings(struct net_device *netdev, if (hw->mac.type == igc_i225 && (status & IGC_STATUS_SPEED_2500)) { speed = SPEED_2500; - hw_dbg("2500 Mbs, "); } else { speed = SPEED_1000; - hw_dbg("1000 Mbs, "); } } else if (status & IGC_STATUS_SPEED_100) { speed = SPEED_100; - hw_dbg("100 Mbs, "); } else { speed = SPEED_10; - hw_dbg("10 Mbs, "); } if ((status & IGC_STATUS_FD) || hw->phy.media_type != igc_media_type_copper) @@ -1011,8 +1837,13 @@ static const struct ethtool_ops igc_ethtool_ops = { .set_ringparam = igc_set_ringparam, .get_pauseparam = igc_get_pauseparam, .set_pauseparam = igc_set_pauseparam, + .get_strings = igc_get_strings, + .get_sset_count = igc_get_sset_count, + .get_ethtool_stats = igc_get_ethtool_stats, .get_coalesce = igc_get_coalesce, .set_coalesce = igc_set_coalesce, + .get_rxnfc = igc_get_rxnfc, + .set_rxnfc = igc_set_rxnfc, .get_rxfh_indir_size = igc_get_rxfh_indir_size, .get_rxfh = igc_get_rxfh, .set_rxfh = igc_set_rxfh, diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c index f8d692f6aa4fe06deb80e2680e9f1f616b07571b..34fa0e60a780af517c20b98dbc2b522de75b0192 100644 --- a/drivers/net/ethernet/intel/igc/igc_main.c +++ b/drivers/net/ethernet/intel/igc/igc_main.c @@ -620,6 +620,55 @@ static void igc_configure_tx(struct igc_adapter *adapter) */ static void igc_setup_mrqc(struct igc_adapter *adapter) { + struct igc_hw *hw = &adapter->hw; + u32 j, num_rx_queues; + u32 mrqc, rxcsum; + u32 rss_key[10]; + + netdev_rss_key_fill(rss_key, sizeof(rss_key)); + for (j = 0; j < 10; j++) + wr32(IGC_RSSRK(j), rss_key[j]); + + num_rx_queues = adapter->rss_queues; + + if (adapter->rss_indir_tbl_init != num_rx_queues) { + for (j = 0; j < IGC_RETA_SIZE; j++) + adapter->rss_indir_tbl[j] = + (j * num_rx_queues) / IGC_RETA_SIZE; + adapter->rss_indir_tbl_init = num_rx_queues; + } + igc_write_rss_indir_tbl(adapter); + + /* Disable raw packet checksumming so that RSS hash is placed in + * descriptor on writeback. No need to enable TCP/UDP/IP checksum + * offloads as they are enabled by default + */ + rxcsum = rd32(IGC_RXCSUM); + rxcsum |= IGC_RXCSUM_PCSD; + + /* Enable Receive Checksum Offload for SCTP */ + rxcsum |= IGC_RXCSUM_CRCOFL; + + /* Don't need to set TUOFL or IPOFL, they default to 1 */ + wr32(IGC_RXCSUM, rxcsum); + + /* Generate RSS hash based on packet types, TCP/UDP + * port numbers and/or IPv4/v6 src and dst addresses + */ + mrqc = IGC_MRQC_RSS_FIELD_IPV4 | + IGC_MRQC_RSS_FIELD_IPV4_TCP | + IGC_MRQC_RSS_FIELD_IPV6 | + IGC_MRQC_RSS_FIELD_IPV6_TCP | + IGC_MRQC_RSS_FIELD_IPV6_TCP_EX; + + if (adapter->flags & IGC_FLAG_RSS_FIELD_IPV4_UDP) + mrqc |= IGC_MRQC_RSS_FIELD_IPV4_UDP; + if (adapter->flags & IGC_FLAG_RSS_FIELD_IPV6_UDP) + mrqc |= IGC_MRQC_RSS_FIELD_IPV6_UDP; + + mrqc |= IGC_MRQC_ENABLE_RSS_MQ; + + wr32(IGC_MRQC, mrqc); } /** @@ -890,7 +939,7 @@ static int igc_tx_map(struct igc_ring *tx_ring, /* Make sure there is space in the ring for the next send. */ igc_maybe_stop_tx(tx_ring, DESC_NEEDED); - if (netif_xmit_stopped(txring_txq(tx_ring)) || !skb->xmit_more) { + if (netif_xmit_stopped(txring_txq(tx_ring)) || !netdev_xmit_more()) { writel(i, tx_ring->tail); } @@ -1145,7 +1194,7 @@ static struct sk_buff *igc_construct_skb(struct igc_ring *rx_ring, /* Determine available headroom for copy */ headlen = size; if (headlen > IGC_RX_HDR_LEN) - headlen = eth_get_headlen(va, IGC_RX_HDR_LEN); + headlen = eth_get_headlen(skb->dev, va, IGC_RX_HDR_LEN); /* align pull length to size of long to optimize memcpy performance */ memcpy(__skb_put(skb, headlen), va, ALIGN(headlen, sizeof(long))); @@ -1733,12 +1782,200 @@ void igc_up(struct igc_adapter *adapter) * igc_update_stats - Update the board statistics counters * @adapter: board private structure */ -static void igc_update_stats(struct igc_adapter *adapter) +void igc_update_stats(struct igc_adapter *adapter) { + struct rtnl_link_stats64 *net_stats = &adapter->stats64; + struct pci_dev *pdev = adapter->pdev; + struct igc_hw *hw = &adapter->hw; + u64 _bytes, _packets; + u64 bytes, packets; + unsigned int start; + u32 mpc; + int i; + + /* Prevent stats update while adapter is being reset, or if the pci + * connection is down. + */ + if (adapter->link_speed == 0) + return; + if (pci_channel_offline(pdev)) + return; + + packets = 0; + bytes = 0; + + rcu_read_lock(); + for (i = 0; i < adapter->num_rx_queues; i++) { + struct igc_ring *ring = adapter->rx_ring[i]; + u32 rqdpc = rd32(IGC_RQDPC(i)); + + if (hw->mac.type >= igc_i225) + wr32(IGC_RQDPC(i), 0); + + if (rqdpc) { + ring->rx_stats.drops += rqdpc; + net_stats->rx_fifo_errors += rqdpc; + } + + do { + start = u64_stats_fetch_begin_irq(&ring->rx_syncp); + _bytes = ring->rx_stats.bytes; + _packets = ring->rx_stats.packets; + } while (u64_stats_fetch_retry_irq(&ring->rx_syncp, start)); + bytes += _bytes; + packets += _packets; + } + + net_stats->rx_bytes = bytes; + net_stats->rx_packets = packets; + + packets = 0; + bytes = 0; + for (i = 0; i < adapter->num_tx_queues; i++) { + struct igc_ring *ring = adapter->tx_ring[i]; + + do { + start = u64_stats_fetch_begin_irq(&ring->tx_syncp); + _bytes = ring->tx_stats.bytes; + _packets = ring->tx_stats.packets; + } while (u64_stats_fetch_retry_irq(&ring->tx_syncp, start)); + bytes += _bytes; + packets += _packets; + } + net_stats->tx_bytes = bytes; + net_stats->tx_packets = packets; + rcu_read_unlock(); + + /* read stats registers */ + adapter->stats.crcerrs += rd32(IGC_CRCERRS); + adapter->stats.gprc += rd32(IGC_GPRC); + adapter->stats.gorc += rd32(IGC_GORCL); + rd32(IGC_GORCH); /* clear GORCL */ + adapter->stats.bprc += rd32(IGC_BPRC); + adapter->stats.mprc += rd32(IGC_MPRC); + adapter->stats.roc += rd32(IGC_ROC); + + adapter->stats.prc64 += rd32(IGC_PRC64); + adapter->stats.prc127 += rd32(IGC_PRC127); + adapter->stats.prc255 += rd32(IGC_PRC255); + adapter->stats.prc511 += rd32(IGC_PRC511); + adapter->stats.prc1023 += rd32(IGC_PRC1023); + adapter->stats.prc1522 += rd32(IGC_PRC1522); + adapter->stats.symerrs += rd32(IGC_SYMERRS); + adapter->stats.sec += rd32(IGC_SEC); + + mpc = rd32(IGC_MPC); + adapter->stats.mpc += mpc; + net_stats->rx_fifo_errors += mpc; + adapter->stats.scc += rd32(IGC_SCC); + adapter->stats.ecol += rd32(IGC_ECOL); + adapter->stats.mcc += rd32(IGC_MCC); + adapter->stats.latecol += rd32(IGC_LATECOL); + adapter->stats.dc += rd32(IGC_DC); + adapter->stats.rlec += rd32(IGC_RLEC); + adapter->stats.xonrxc += rd32(IGC_XONRXC); + adapter->stats.xontxc += rd32(IGC_XONTXC); + adapter->stats.xoffrxc += rd32(IGC_XOFFRXC); + adapter->stats.xofftxc += rd32(IGC_XOFFTXC); + adapter->stats.fcruc += rd32(IGC_FCRUC); + adapter->stats.gptc += rd32(IGC_GPTC); + adapter->stats.gotc += rd32(IGC_GOTCL); + rd32(IGC_GOTCH); /* clear GOTCL */ + adapter->stats.rnbc += rd32(IGC_RNBC); + adapter->stats.ruc += rd32(IGC_RUC); + adapter->stats.rfc += rd32(IGC_RFC); + adapter->stats.rjc += rd32(IGC_RJC); + adapter->stats.tor += rd32(IGC_TORH); + adapter->stats.tot += rd32(IGC_TOTH); + adapter->stats.tpr += rd32(IGC_TPR); + + adapter->stats.ptc64 += rd32(IGC_PTC64); + adapter->stats.ptc127 += rd32(IGC_PTC127); + adapter->stats.ptc255 += rd32(IGC_PTC255); + adapter->stats.ptc511 += rd32(IGC_PTC511); + adapter->stats.ptc1023 += rd32(IGC_PTC1023); + adapter->stats.ptc1522 += rd32(IGC_PTC1522); + + adapter->stats.mptc += rd32(IGC_MPTC); + adapter->stats.bptc += rd32(IGC_BPTC); + + adapter->stats.tpt += rd32(IGC_TPT); + adapter->stats.colc += rd32(IGC_COLC); + + adapter->stats.algnerrc += rd32(IGC_ALGNERRC); + + adapter->stats.tsctc += rd32(IGC_TSCTC); + adapter->stats.tsctfc += rd32(IGC_TSCTFC); + + adapter->stats.iac += rd32(IGC_IAC); + adapter->stats.icrxoc += rd32(IGC_ICRXOC); + adapter->stats.icrxptc += rd32(IGC_ICRXPTC); + adapter->stats.icrxatc += rd32(IGC_ICRXATC); + adapter->stats.ictxptc += rd32(IGC_ICTXPTC); + adapter->stats.ictxatc += rd32(IGC_ICTXATC); + adapter->stats.ictxqec += rd32(IGC_ICTXQEC); + adapter->stats.ictxqmtc += rd32(IGC_ICTXQMTC); + adapter->stats.icrxdmtc += rd32(IGC_ICRXDMTC); + + /* Fill out the OS statistics structure */ + net_stats->multicast = adapter->stats.mprc; + net_stats->collisions = adapter->stats.colc; + + /* Rx Errors */ + + /* RLEC on some newer hardware can be incorrect so build + * our own version based on RUC and ROC + */ + net_stats->rx_errors = adapter->stats.rxerrc + + adapter->stats.crcerrs + adapter->stats.algnerrc + + adapter->stats.ruc + adapter->stats.roc + + adapter->stats.cexterr; + net_stats->rx_length_errors = adapter->stats.ruc + + adapter->stats.roc; + net_stats->rx_crc_errors = adapter->stats.crcerrs; + net_stats->rx_frame_errors = adapter->stats.algnerrc; + net_stats->rx_missed_errors = adapter->stats.mpc; + + /* Tx Errors */ + net_stats->tx_errors = adapter->stats.ecol + + adapter->stats.latecol; + net_stats->tx_aborted_errors = adapter->stats.ecol; + net_stats->tx_window_errors = adapter->stats.latecol; + net_stats->tx_carrier_errors = adapter->stats.tncrs; + + /* Tx Dropped needs to be maintained elsewhere */ + + /* Management Stats */ + adapter->stats.mgptc += rd32(IGC_MGTPTC); + adapter->stats.mgprc += rd32(IGC_MGTPRC); + adapter->stats.mgpdc += rd32(IGC_MGTPDC); } static void igc_nfc_filter_exit(struct igc_adapter *adapter) { + struct igc_nfc_filter *rule; + + spin_lock(&adapter->nfc_lock); + + hlist_for_each_entry(rule, &adapter->nfc_filter_list, nfc_node) + igc_erase_filter(adapter, rule); + + hlist_for_each_entry(rule, &adapter->cls_flower_list, nfc_node) + igc_erase_filter(adapter, rule); + + spin_unlock(&adapter->nfc_lock); +} + +static void igc_nfc_filter_restore(struct igc_adapter *adapter) +{ + struct igc_nfc_filter *rule; + + spin_lock(&adapter->nfc_lock); + + hlist_for_each_entry(rule, &adapter->nfc_filter_list, nfc_node) + igc_add_filter(adapter, rule); + + spin_unlock(&adapter->nfc_lock); } /** @@ -1885,6 +2122,86 @@ static struct net_device_stats *igc_get_stats(struct net_device *netdev) return &netdev->stats; } +static netdev_features_t igc_fix_features(struct net_device *netdev, + netdev_features_t features) +{ + /* Since there is no support for separate Rx/Tx vlan accel + * enable/disable make sure Tx flag is always in same state as Rx. + */ + if (features & NETIF_F_HW_VLAN_CTAG_RX) + features |= NETIF_F_HW_VLAN_CTAG_TX; + else + features &= ~NETIF_F_HW_VLAN_CTAG_TX; + + return features; +} + +static int igc_set_features(struct net_device *netdev, + netdev_features_t features) +{ + netdev_features_t changed = netdev->features ^ features; + struct igc_adapter *adapter = netdev_priv(netdev); + + /* Add VLAN support */ + if (!(changed & (NETIF_F_RXALL | NETIF_F_NTUPLE))) + return 0; + + if (!(features & NETIF_F_NTUPLE)) { + struct hlist_node *node2; + struct igc_nfc_filter *rule; + + spin_lock(&adapter->nfc_lock); + hlist_for_each_entry_safe(rule, node2, + &adapter->nfc_filter_list, nfc_node) { + igc_erase_filter(adapter, rule); + hlist_del(&rule->nfc_node); + kfree(rule); + } + spin_unlock(&adapter->nfc_lock); + adapter->nfc_filter_count = 0; + } + + netdev->features = features; + + if (netif_running(netdev)) + igc_reinit_locked(adapter); + else + igc_reset(adapter); + + return 1; +} + +static netdev_features_t +igc_features_check(struct sk_buff *skb, struct net_device *dev, + netdev_features_t features) +{ + unsigned int network_hdr_len, mac_hdr_len; + + /* Make certain the headers can be described by a context descriptor */ + mac_hdr_len = skb_network_header(skb) - skb->data; + if (unlikely(mac_hdr_len > IGC_MAX_MAC_HDR_LEN)) + return features & ~(NETIF_F_HW_CSUM | + NETIF_F_SCTP_CRC | + NETIF_F_HW_VLAN_CTAG_TX | + NETIF_F_TSO | + NETIF_F_TSO6); + + network_hdr_len = skb_checksum_start(skb) - skb_network_header(skb); + if (unlikely(network_hdr_len > IGC_MAX_NETWORK_HDR_LEN)) + return features & ~(NETIF_F_HW_CSUM | + NETIF_F_SCTP_CRC | + NETIF_F_TSO | + NETIF_F_TSO6); + + /* We can only support IPv4 TSO in tunnels if we can mangle the + * inner IP ID field, so strip TSO if MANGLEID is not supported. + */ + if (skb->encapsulation && !(features & NETIF_F_TSO_MANGLEID)) + features &= ~NETIF_F_TSO; + + return features; +} + /** * igc_configure - configure the hardware for RX and TX * @adapter: private board structure @@ -1901,6 +2218,7 @@ static void igc_configure(struct igc_adapter *adapter) igc_setup_mrqc(adapter); igc_setup_rctl(adapter); + igc_nfc_filter_restore(adapter); igc_configure_tx(adapter); igc_configure_rx(adapter); @@ -1962,6 +2280,127 @@ static void igc_set_default_mac_filter(struct igc_adapter *adapter) igc_rar_set_index(adapter, 0); } +/* If the filter to be added and an already existing filter express + * the same address and address type, it should be possible to only + * override the other configurations, for example the queue to steer + * traffic. + */ +static bool igc_mac_entry_can_be_used(const struct igc_mac_addr *entry, + const u8 *addr, const u8 flags) +{ + if (!(entry->state & IGC_MAC_STATE_IN_USE)) + return true; + + if ((entry->state & IGC_MAC_STATE_SRC_ADDR) != + (flags & IGC_MAC_STATE_SRC_ADDR)) + return false; + + if (!ether_addr_equal(addr, entry->addr)) + return false; + + return true; +} + +/* Add a MAC filter for 'addr' directing matching traffic to 'queue', + * 'flags' is used to indicate what kind of match is made, match is by + * default for the destination address, if matching by source address + * is desired the flag IGC_MAC_STATE_SRC_ADDR can be used. + */ +static int igc_add_mac_filter_flags(struct igc_adapter *adapter, + const u8 *addr, const u8 queue, + const u8 flags) +{ + struct igc_hw *hw = &adapter->hw; + int rar_entries = hw->mac.rar_entry_count; + int i; + + if (is_zero_ether_addr(addr)) + return -EINVAL; + + /* Search for the first empty entry in the MAC table. + * Do not touch entries at the end of the table reserved for the VF MAC + * addresses. + */ + for (i = 0; i < rar_entries; i++) { + if (!igc_mac_entry_can_be_used(&adapter->mac_table[i], + addr, flags)) + continue; + + ether_addr_copy(adapter->mac_table[i].addr, addr); + adapter->mac_table[i].queue = queue; + adapter->mac_table[i].state |= IGC_MAC_STATE_IN_USE | flags; + + igc_rar_set_index(adapter, i); + return i; + } + + return -ENOSPC; +} + +int igc_add_mac_steering_filter(struct igc_adapter *adapter, + const u8 *addr, u8 queue, u8 flags) +{ + return igc_add_mac_filter_flags(adapter, addr, queue, + IGC_MAC_STATE_QUEUE_STEERING | flags); +} + +/* Remove a MAC filter for 'addr' directing matching traffic to + * 'queue', 'flags' is used to indicate what kind of match need to be + * removed, match is by default for the destination address, if + * matching by source address is to be removed the flag + * IGC_MAC_STATE_SRC_ADDR can be used. + */ +static int igc_del_mac_filter_flags(struct igc_adapter *adapter, + const u8 *addr, const u8 queue, + const u8 flags) +{ + struct igc_hw *hw = &adapter->hw; + int rar_entries = hw->mac.rar_entry_count; + int i; + + if (is_zero_ether_addr(addr)) + return -EINVAL; + + /* Search for matching entry in the MAC table based on given address + * and queue. Do not touch entries at the end of the table reserved + * for the VF MAC addresses. + */ + for (i = 0; i < rar_entries; i++) { + if (!(adapter->mac_table[i].state & IGC_MAC_STATE_IN_USE)) + continue; + if ((adapter->mac_table[i].state & flags) != flags) + continue; + if (adapter->mac_table[i].queue != queue) + continue; + if (!ether_addr_equal(adapter->mac_table[i].addr, addr)) + continue; + + /* When a filter for the default address is "deleted", + * we return it to its initial configuration + */ + if (adapter->mac_table[i].state & IGC_MAC_STATE_DEFAULT) { + adapter->mac_table[i].state = + IGC_MAC_STATE_DEFAULT | IGC_MAC_STATE_IN_USE; + } else { + adapter->mac_table[i].state = 0; + adapter->mac_table[i].queue = 0; + memset(adapter->mac_table[i].addr, 0, ETH_ALEN); + } + + igc_rar_set_index(adapter, i); + return 0; + } + + return -ENOENT; +} + +int igc_del_mac_steering_filter(struct igc_adapter *adapter, + const u8 *addr, u8 queue, u8 flags) +{ + return igc_del_mac_filter_flags(adapter, addr, queue, + IGC_MAC_STATE_QUEUE_STEERING | flags); +} + /** * igc_set_rx_mode - Secondary Unicast, Multicast and Promiscuous mode set * @netdev: network interface device structure @@ -3429,6 +3868,9 @@ static const struct net_device_ops igc_netdev_ops = { .ndo_set_mac_address = igc_set_mac, .ndo_change_mtu = igc_change_mtu, .ndo_get_stats = igc_get_stats, + .ndo_fix_features = igc_fix_features, + .ndo_set_features = igc_set_features, + .ndo_features_check = igc_features_check, }; /* PCIe configuration access */ @@ -3658,6 +4100,9 @@ static int igc_probe(struct pci_dev *pdev, if (err) goto err_sw_init; + /* copy netdev features into list of user selectable features */ + netdev->hw_features |= NETIF_F_NTUPLE; + /* MTU range: 68 - 9216 */ netdev->min_mtu = ETH_MIN_MTU; netdev->max_mtu = MAX_STD_JUMBO_FRAME_SIZE; diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h index 5afe7a8d3fafdd2cf16da8e83bdd46481eb43156..50d7c04dccf597f8b0553fd64abf1dd165965a4c 100644 --- a/drivers/net/ethernet/intel/igc/igc_regs.h +++ b/drivers/net/ethernet/intel/igc/igc_regs.h @@ -80,8 +80,23 @@ /* MSI-X Table Register Descriptions */ #define IGC_PBACL 0x05B68 /* MSIx PBA Clear - R/W 1 to clear */ +/* RSS registers */ +#define IGC_MRQC 0x05818 /* Multiple Receive Control - RW */ + +/* Filtering Registers */ +#define IGC_ETQF(_n) (0x05CB0 + (4 * (_n))) /* EType Queue Fltr */ + +/* ETQF register bit definitions */ +#define IGC_ETQF_FILTER_ENABLE BIT(26) +#define IGC_ETQF_QUEUE_ENABLE BIT(31) +#define IGC_ETQF_QUEUE_SHIFT 16 +#define IGC_ETQF_QUEUE_MASK 0x00070000 +#define IGC_ETQF_ETYPE_MASK 0x0000FFFF + /* Redirection Table - RW Array */ #define IGC_RETA(_i) (0x05C00 + ((_i) * 4)) +/* RSS Random Key - RW Array */ +#define IGC_RSSRK(_i) (0x05C80 + ((_i) * 4)) /* Receive Register Descriptions */ #define IGC_RCTL 0x00100 /* Rx Control - RW */ @@ -101,6 +116,7 @@ #define IGC_UTA 0x0A000 /* Unicast Table Array - RW */ #define IGC_RAL(_n) (0x05400 + ((_n) * 0x08)) #define IGC_RAH(_n) (0x05404 + ((_n) * 0x08)) +#define IGC_VLAPQF 0x055B0 /* VLAN Priority Queue Filter VLAPQF */ /* Transmit Register Descriptions */ #define IGC_TCTL 0x00400 /* Tx Control - RW */ diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c index 99e23cf6a73abc9d05c5c5b07d4ca779f13254a0..57fd9ee6de665255f22c7ff146ba398b7f4fd971 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c @@ -1800,7 +1800,7 @@ static void ixgbe_pull_tail(struct ixgbe_ring *rx_ring, * we need the header to contain the greater of either ETH_HLEN or * 60 bytes if the skb->len is less than 60 for skb_pad. */ - pull_len = eth_get_headlen(va, IXGBE_RX_HDR_SIZE); + pull_len = eth_get_headlen(skb->dev, va, IXGBE_RX_HDR_SIZE); /* align pull length to size of long to optimize memcpy performance */ skb_copy_to_linear_data(skb, va, ALIGN(pull_len, sizeof(long))); @@ -8297,7 +8297,7 @@ static int ixgbe_tx_map(struct ixgbe_ring *tx_ring, ixgbe_maybe_stop_tx(tx_ring, DESC_NEEDED); - if (netif_xmit_stopped(txring_txq(tx_ring)) || !skb->xmit_more) { + if (netif_xmit_stopped(txring_txq(tx_ring)) || !netdev_xmit_more()) { writel(i, tx_ring->tail); } @@ -8478,8 +8478,7 @@ static void ixgbe_atr(struct ixgbe_ring *ring, #ifdef IXGBE_FCOE static u16 ixgbe_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { struct ixgbe_adapter *adapter; struct ixgbe_ring_feature *f; @@ -8509,7 +8508,7 @@ static u16 ixgbe_select_queue(struct net_device *dev, struct sk_buff *skb, break; /* fall through */ default: - return fallback(dev, skb, sb_dev); + return netdev_pick_tx(dev, skb, sb_dev); } f = &adapter->ring_feature[RING_F_FCOE]; @@ -9791,7 +9790,7 @@ static int ixgbe_set_features(struct net_device *netdev, NETIF_F_HW_VLAN_CTAG_FILTER)) ixgbe_set_rx_mode(netdev); - return 0; + return 1; } /** diff --git a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c index 49e23afa05a2e58cfc1f6d9d296a23b2c1a71676..d189ed24766524676f2a1baa3651e53412bd1ad0 100644 --- a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c +++ b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c @@ -895,7 +895,8 @@ struct sk_buff *ixgbevf_construct_skb(struct ixgbevf_ring *rx_ring, /* Determine available headroom for copy */ headlen = size; if (headlen > IXGBEVF_RX_HDR_SIZE) - headlen = eth_get_headlen(xdp->data, IXGBEVF_RX_HDR_SIZE); + headlen = eth_get_headlen(skb->dev, xdp->data, + IXGBEVF_RX_HDR_SIZE); /* align pull length to size of long to optimize memcpy performance */ memcpy(__skb_put(skb, headlen), xdp->data, diff --git a/drivers/net/ethernet/lantiq_xrx200.c b/drivers/net/ethernet/lantiq_xrx200.c index d29104de0d53c06ae431b92bd6aa75cb579bd5ee..cda641ef89afaca7c236bed2f263ecb218b1be7c 100644 --- a/drivers/net/ethernet/lantiq_xrx200.c +++ b/drivers/net/ethernet/lantiq_xrx200.c @@ -478,7 +478,7 @@ static int xrx200_probe(struct platform_device *pdev) } mac = of_get_mac_address(np); - if (mac && is_valid_ether_addr(mac)) + if (!IS_ERR(mac)) ether_addr_copy(net_dev->dev_addr, mac); else eth_hw_addr_random(net_dev); diff --git a/drivers/net/ethernet/marvell/mv643xx_eth.c b/drivers/net/ethernet/marvell/mv643xx_eth.c index 292a668ce88e6f75580910eb1997e3b791a27d72..07e254fc96effec9428f5247ebcaf6b222476d3f 100644 --- a/drivers/net/ethernet/marvell/mv643xx_eth.c +++ b/drivers/net/ethernet/marvell/mv643xx_eth.c @@ -2749,7 +2749,7 @@ static int mv643xx_eth_shared_of_add_port(struct platform_device *pdev, } mac_addr = of_get_mac_address(pnp); - if (mac_addr) + if (!IS_ERR(mac_addr)) memcpy(ppd.mac_addr, mac_addr, ETH_ALEN); mv643xx_eth_property(pnp, "tx-queue-size", ppd.tx_queue_size); diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c index c0a3718b2e2a78616b3c02831aad80f31d0acf30..8186135883ed965f62ec76586649c3eafb63786f 100644 --- a/drivers/net/ethernet/marvell/mvneta.c +++ b/drivers/net/ethernet/marvell/mvneta.c @@ -2467,7 +2467,7 @@ out: if (txq->count >= txq->tx_stop_threshold) netif_tx_stop_queue(nq); - if (!skb->xmit_more || netif_xmit_stopped(nq) || + if (!netdev_xmit_more() || netif_xmit_stopped(nq) || txq->pending + frags > MVNETA_TXQ_DEC_SENT_MASK) mvneta_txq_pend_desc_add(pp, txq, frags); else @@ -3385,6 +3385,7 @@ static void mvneta_validate(struct net_device *ndev, unsigned long *supported, phylink_set(mask, 1000baseX_Full); } if (pp->comphy || state->interface == PHY_INTERFACE_MODE_2500BASEX) { + phylink_set(mask, 2500baseT_Full); phylink_set(mask, 2500baseX_Full); } @@ -4475,15 +4476,14 @@ static int mvneta_probe(struct platform_device *pdev) int err; int cpu; - dev = alloc_etherdev_mqs(sizeof(struct mvneta_port), txq_number, rxq_number); + dev = devm_alloc_etherdev_mqs(&pdev->dev, sizeof(struct mvneta_port), + txq_number, rxq_number); if (!dev) return -ENOMEM; dev->irq = irq_of_parse_and_map(dn, 0); - if (dev->irq == 0) { - err = -EINVAL; - goto err_free_netdev; - } + if (dev->irq == 0) + return -EINVAL; phy_mode = of_get_phy_mode(dn); if (phy_mode < 0) { @@ -4563,7 +4563,7 @@ static int mvneta_probe(struct platform_device *pdev) } dt_mac_addr = of_get_mac_address(dn); - if (dt_mac_addr) { + if (!IS_ERR(dt_mac_addr)) { mac_from = "device tree"; memcpy(dev->dev_addr, dt_mac_addr, ETH_ALEN); } else { @@ -4704,8 +4704,6 @@ err_free_phylink: phylink_destroy(pp->phylink); err_free_irq: irq_dispose_mapping(dev->irq); -err_free_netdev: - free_netdev(dev); return err; } @@ -4722,7 +4720,6 @@ static int mvneta_remove(struct platform_device *pdev) free_percpu(pp->stats); irq_dispose_mapping(dev->irq); phylink_destroy(pp->phylink); - free_netdev(dev); if (pp->bm_priv) { mvneta_bm_pool_destroy(pp->bm_priv, pp->pool_long, 1 << pp->id); diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h index ff0f4c503f534a792e22114af9395a81d896d4ef..6171270a016c6708db1d346cc71ae320b03eae99 100644 --- a/drivers/net/ethernet/marvell/mvpp2/mvpp2.h +++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2.h @@ -14,6 +14,7 @@ #include #include #include +#include /* Fifo Registers */ #define MVPP2_RX_DATA_FIFO_SIZE_REG(port) (0x00 + 4 * (port)) @@ -101,6 +102,7 @@ #define MVPP2_CLS_FLOW_TBL1_REG 0x1828 #define MVPP2_CLS_FLOW_TBL1_N_FIELDS_MASK 0x7 #define MVPP2_CLS_FLOW_TBL1_N_FIELDS(x) (x) +#define MVPP2_CLS_FLOW_TBL1_LU_TYPE(lu) (((lu) & 0x3f) << 3) #define MVPP2_CLS_FLOW_TBL1_PRIO_MASK 0x3f #define MVPP2_CLS_FLOW_TBL1_PRIO(x) ((x) << 9) #define MVPP2_CLS_FLOW_TBL1_SEQ_MASK 0x7 @@ -123,13 +125,18 @@ #define MVPP22_CLS_C2_TCAM_DATA2 0x1b18 #define MVPP22_CLS_C2_TCAM_DATA3 0x1b1c #define MVPP22_CLS_C2_TCAM_DATA4 0x1b20 +#define MVPP22_CLS_C2_LU_TYPE(lu) ((lu) & 0x3f) #define MVPP22_CLS_C2_PORT_ID(port) ((port) << 8) +#define MVPP22_CLS_C2_PORT_MASK (0xff << 8) +#define MVPP22_CLS_C2_TCAM_INV 0x1b24 +#define MVPP22_CLS_C2_TCAM_INV_BIT BIT(31) #define MVPP22_CLS_C2_HIT_CTR 0x1b50 #define MVPP22_CLS_C2_ACT 0x1b60 #define MVPP22_CLS_C2_ACT_RSS_EN(act) (((act) & 0x3) << 19) #define MVPP22_CLS_C2_ACT_FWD(act) (((act) & 0x7) << 13) #define MVPP22_CLS_C2_ACT_QHIGH(act) (((act) & 0x3) << 11) #define MVPP22_CLS_C2_ACT_QLOW(act) (((act) & 0x3) << 9) +#define MVPP22_CLS_C2_ACT_COLOR(act) ((act) & 0x7) #define MVPP22_CLS_C2_ATTR0 0x1b64 #define MVPP22_CLS_C2_ATTR0_QHIGH(qh) (((qh) & 0x1f) << 24) #define MVPP22_CLS_C2_ATTR0_QHIGH_MASK 0x1f @@ -610,6 +617,12 @@ #define MVPP2_BIT_TO_WORD(bit) ((bit) / 32) #define MVPP2_BIT_IN_WORD(bit) ((bit) % 32) +#define MVPP2_N_PRS_FLOWS 52 +#define MVPP2_N_RFS_ENTRIES_PER_FLOW 4 + +/* There are 7 supported high-level flows */ +#define MVPP2_N_RFS_RULES (MVPP2_N_RFS_ENTRIES_PER_FLOW * 7) + /* RSS constants */ #define MVPP22_RSS_TABLE_ENTRIES 32 @@ -710,6 +723,7 @@ enum mvpp2_prs_l3_cast { #define MVPP2_DESC_DMA_MASK DMA_BIT_MASK(40) /* Definitions */ +struct mvpp2_dbgfs_entries; /* Shared Packet Processor resources */ struct mvpp2 { @@ -771,6 +785,9 @@ struct mvpp2 { /* Debugfs root entry */ struct dentry *dbgfs_dir; + + /* Debugfs entries private data */ + struct mvpp2_dbgfs_entries *dbgfs_entries; }; struct mvpp2_pcpu_stats { @@ -802,6 +819,37 @@ struct mvpp2_queue_vector { struct cpumask *mask; }; +/* Internal represention of a Flow Steering rule */ +struct mvpp2_rfs_rule { + /* Rule location inside the flow*/ + int loc; + + /* Flow type, such as TCP_V4_FLOW, IP6_FLOW, etc. */ + int flow_type; + + /* Index of the C2 TCAM entry handling this rule */ + int c2_index; + + /* Header fields that needs to be extracted to match this flow */ + u16 hek_fields; + + /* CLS engine : only c2 is supported for now. */ + u8 engine; + + /* TCAM key and mask for C2-based steering. These fields should be + * encapsulated in a union should we add more engines. + */ + u64 c2_tcam; + u64 c2_tcam_mask; + + struct flow_rule *flow; +}; + +struct mvpp2_ethtool_fs { + struct mvpp2_rfs_rule rule; + struct ethtool_rxnfc rxnfc; +}; + struct mvpp2_port { u8 id; @@ -873,6 +921,10 @@ struct mvpp2_port { /* RSS indirection table */ u32 indir[MVPP22_RSS_TABLE_ENTRIES]; + + /* List of steering rules active on that port */ + struct mvpp2_ethtool_fs *rfs_rules[MVPP2_N_RFS_RULES]; + int n_rfs_rules; }; /* The mvpp2_tx_desc and mvpp2_rx_desc structures describe the diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c index efdb7a65683576a84806639630fce4d0928defcd..d046f7a1dcf5a556cc3ebc83af057a339f830409 100644 --- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c +++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.c @@ -22,302 +22,302 @@ } \ } -static struct mvpp2_cls_flow cls_flows[MVPP2_N_FLOWS] = { +static const struct mvpp2_cls_flow cls_flows[MVPP2_N_PRS_FLOWS] = { /* TCP over IPv4 flows, Not fragmented, no vlan tag */ - MVPP2_DEF_FLOW(TCP_V4_FLOW, MVPP2_FL_IP4_TCP_NF_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_TCP4, MVPP2_FL_IP4_TCP_NF_UNTAG, MVPP22_CLS_HEK_IP4_5T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP4 | MVPP2_PRS_RI_L4_TCP, MVPP2_PRS_IP_MASK | MVPP2_PRS_RI_VLAN_MASK), - MVPP2_DEF_FLOW(TCP_V4_FLOW, MVPP2_FL_IP4_TCP_NF_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_TCP4, MVPP2_FL_IP4_TCP_NF_UNTAG, MVPP22_CLS_HEK_IP4_5T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP4_OPT | MVPP2_PRS_RI_L4_TCP, MVPP2_PRS_IP_MASK | MVPP2_PRS_RI_VLAN_MASK), - MVPP2_DEF_FLOW(TCP_V4_FLOW, MVPP2_FL_IP4_TCP_NF_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_TCP4, MVPP2_FL_IP4_TCP_NF_UNTAG, MVPP22_CLS_HEK_IP4_5T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP4_OTHER | MVPP2_PRS_RI_L4_TCP, MVPP2_PRS_IP_MASK | MVPP2_PRS_RI_VLAN_MASK), /* TCP over IPv4 flows, Not fragmented, with vlan tag */ - MVPP2_DEF_FLOW(TCP_V4_FLOW, MVPP2_FL_IP4_TCP_NF_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_TCP4, MVPP2_FL_IP4_TCP_NF_TAG, MVPP22_CLS_HEK_IP4_5T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP4 | MVPP2_PRS_RI_L4_TCP, MVPP2_PRS_IP_MASK), - MVPP2_DEF_FLOW(TCP_V4_FLOW, MVPP2_FL_IP4_TCP_NF_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_TCP4, MVPP2_FL_IP4_TCP_NF_TAG, MVPP22_CLS_HEK_IP4_5T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP4_OPT | MVPP2_PRS_RI_L4_TCP, MVPP2_PRS_IP_MASK), - MVPP2_DEF_FLOW(TCP_V4_FLOW, MVPP2_FL_IP4_TCP_NF_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_TCP4, MVPP2_FL_IP4_TCP_NF_TAG, MVPP22_CLS_HEK_IP4_5T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP4_OTHER | MVPP2_PRS_RI_L4_TCP, MVPP2_PRS_IP_MASK), /* TCP over IPv4 flows, fragmented, no vlan tag */ - MVPP2_DEF_FLOW(TCP_V4_FLOW, MVPP2_FL_IP4_TCP_FRAG_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_TCP4, MVPP2_FL_IP4_TCP_FRAG_UNTAG, MVPP22_CLS_HEK_IP4_2T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP4 | MVPP2_PRS_RI_L4_TCP, MVPP2_PRS_IP_MASK | MVPP2_PRS_RI_VLAN_MASK), - MVPP2_DEF_FLOW(TCP_V4_FLOW, MVPP2_FL_IP4_TCP_FRAG_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_TCP4, MVPP2_FL_IP4_TCP_FRAG_UNTAG, MVPP22_CLS_HEK_IP4_2T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP4_OPT | MVPP2_PRS_RI_L4_TCP, MVPP2_PRS_IP_MASK | MVPP2_PRS_RI_VLAN_MASK), - MVPP2_DEF_FLOW(TCP_V4_FLOW, MVPP2_FL_IP4_TCP_FRAG_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_TCP4, MVPP2_FL_IP4_TCP_FRAG_UNTAG, MVPP22_CLS_HEK_IP4_2T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP4_OTHER | MVPP2_PRS_RI_L4_TCP, MVPP2_PRS_IP_MASK | MVPP2_PRS_RI_VLAN_MASK), /* TCP over IPv4 flows, fragmented, with vlan tag */ - MVPP2_DEF_FLOW(TCP_V4_FLOW, MVPP2_FL_IP4_TCP_FRAG_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_TCP4, MVPP2_FL_IP4_TCP_FRAG_TAG, MVPP22_CLS_HEK_IP4_2T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP4 | MVPP2_PRS_RI_L4_TCP, MVPP2_PRS_IP_MASK), - MVPP2_DEF_FLOW(TCP_V4_FLOW, MVPP2_FL_IP4_TCP_FRAG_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_TCP4, MVPP2_FL_IP4_TCP_FRAG_TAG, MVPP22_CLS_HEK_IP4_2T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP4_OPT | MVPP2_PRS_RI_L4_TCP, MVPP2_PRS_IP_MASK), - MVPP2_DEF_FLOW(TCP_V4_FLOW, MVPP2_FL_IP4_TCP_FRAG_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_TCP4, MVPP2_FL_IP4_TCP_FRAG_TAG, MVPP22_CLS_HEK_IP4_2T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP4_OTHER | MVPP2_PRS_RI_L4_TCP, MVPP2_PRS_IP_MASK), /* UDP over IPv4 flows, Not fragmented, no vlan tag */ - MVPP2_DEF_FLOW(UDP_V4_FLOW, MVPP2_FL_IP4_UDP_NF_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_UDP4, MVPP2_FL_IP4_UDP_NF_UNTAG, MVPP22_CLS_HEK_IP4_5T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP4 | MVPP2_PRS_RI_L4_UDP, MVPP2_PRS_IP_MASK | MVPP2_PRS_RI_VLAN_MASK), - MVPP2_DEF_FLOW(UDP_V4_FLOW, MVPP2_FL_IP4_UDP_NF_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_UDP4, MVPP2_FL_IP4_UDP_NF_UNTAG, MVPP22_CLS_HEK_IP4_5T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP4_OPT | MVPP2_PRS_RI_L4_UDP, MVPP2_PRS_IP_MASK | MVPP2_PRS_RI_VLAN_MASK), - MVPP2_DEF_FLOW(UDP_V4_FLOW, MVPP2_FL_IP4_UDP_NF_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_UDP4, MVPP2_FL_IP4_UDP_NF_UNTAG, MVPP22_CLS_HEK_IP4_5T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP4_OTHER | MVPP2_PRS_RI_L4_UDP, MVPP2_PRS_IP_MASK | MVPP2_PRS_RI_VLAN_MASK), /* UDP over IPv4 flows, Not fragmented, with vlan tag */ - MVPP2_DEF_FLOW(UDP_V4_FLOW, MVPP2_FL_IP4_UDP_NF_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_UDP4, MVPP2_FL_IP4_UDP_NF_TAG, MVPP22_CLS_HEK_IP4_5T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP4 | MVPP2_PRS_RI_L4_UDP, MVPP2_PRS_IP_MASK), - MVPP2_DEF_FLOW(UDP_V4_FLOW, MVPP2_FL_IP4_UDP_NF_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_UDP4, MVPP2_FL_IP4_UDP_NF_TAG, MVPP22_CLS_HEK_IP4_5T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP4_OPT | MVPP2_PRS_RI_L4_UDP, MVPP2_PRS_IP_MASK), - MVPP2_DEF_FLOW(UDP_V4_FLOW, MVPP2_FL_IP4_UDP_NF_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_UDP4, MVPP2_FL_IP4_UDP_NF_TAG, MVPP22_CLS_HEK_IP4_5T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP4_OTHER | MVPP2_PRS_RI_L4_UDP, MVPP2_PRS_IP_MASK), /* UDP over IPv4 flows, fragmented, no vlan tag */ - MVPP2_DEF_FLOW(UDP_V4_FLOW, MVPP2_FL_IP4_UDP_FRAG_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_UDP4, MVPP2_FL_IP4_UDP_FRAG_UNTAG, MVPP22_CLS_HEK_IP4_2T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP4 | MVPP2_PRS_RI_L4_UDP, MVPP2_PRS_IP_MASK | MVPP2_PRS_RI_VLAN_MASK), - MVPP2_DEF_FLOW(UDP_V4_FLOW, MVPP2_FL_IP4_UDP_FRAG_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_UDP4, MVPP2_FL_IP4_UDP_FRAG_UNTAG, MVPP22_CLS_HEK_IP4_2T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP4_OPT | MVPP2_PRS_RI_L4_UDP, MVPP2_PRS_IP_MASK | MVPP2_PRS_RI_VLAN_MASK), - MVPP2_DEF_FLOW(UDP_V4_FLOW, MVPP2_FL_IP4_UDP_FRAG_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_UDP4, MVPP2_FL_IP4_UDP_FRAG_UNTAG, MVPP22_CLS_HEK_IP4_2T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP4_OTHER | MVPP2_PRS_RI_L4_UDP, MVPP2_PRS_IP_MASK | MVPP2_PRS_RI_VLAN_MASK), /* UDP over IPv4 flows, fragmented, with vlan tag */ - MVPP2_DEF_FLOW(UDP_V4_FLOW, MVPP2_FL_IP4_UDP_FRAG_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_UDP4, MVPP2_FL_IP4_UDP_FRAG_TAG, MVPP22_CLS_HEK_IP4_2T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP4 | MVPP2_PRS_RI_L4_UDP, MVPP2_PRS_IP_MASK), - MVPP2_DEF_FLOW(UDP_V4_FLOW, MVPP2_FL_IP4_UDP_FRAG_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_UDP4, MVPP2_FL_IP4_UDP_FRAG_TAG, MVPP22_CLS_HEK_IP4_2T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP4_OPT | MVPP2_PRS_RI_L4_UDP, MVPP2_PRS_IP_MASK), - MVPP2_DEF_FLOW(UDP_V4_FLOW, MVPP2_FL_IP4_UDP_FRAG_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_UDP4, MVPP2_FL_IP4_UDP_FRAG_TAG, MVPP22_CLS_HEK_IP4_2T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP4_OTHER | MVPP2_PRS_RI_L4_UDP, MVPP2_PRS_IP_MASK), /* TCP over IPv6 flows, not fragmented, no vlan tag */ - MVPP2_DEF_FLOW(TCP_V6_FLOW, MVPP2_FL_IP6_TCP_NF_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_TCP6, MVPP2_FL_IP6_TCP_NF_UNTAG, MVPP22_CLS_HEK_IP6_5T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP6 | MVPP2_PRS_RI_L4_TCP, MVPP2_PRS_IP_MASK | MVPP2_PRS_RI_VLAN_MASK), - MVPP2_DEF_FLOW(TCP_V6_FLOW, MVPP2_FL_IP6_TCP_NF_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_TCP6, MVPP2_FL_IP6_TCP_NF_UNTAG, MVPP22_CLS_HEK_IP6_5T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP6_EXT | MVPP2_PRS_RI_L4_TCP, MVPP2_PRS_IP_MASK | MVPP2_PRS_RI_VLAN_MASK), /* TCP over IPv6 flows, not fragmented, with vlan tag */ - MVPP2_DEF_FLOW(TCP_V6_FLOW, MVPP2_FL_IP6_TCP_NF_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_TCP6, MVPP2_FL_IP6_TCP_NF_TAG, MVPP22_CLS_HEK_IP6_5T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP6 | MVPP2_PRS_RI_L4_TCP, MVPP2_PRS_IP_MASK), - MVPP2_DEF_FLOW(TCP_V6_FLOW, MVPP2_FL_IP6_TCP_NF_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_TCP6, MVPP2_FL_IP6_TCP_NF_TAG, MVPP22_CLS_HEK_IP6_5T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP6_EXT | MVPP2_PRS_RI_L4_TCP, MVPP2_PRS_IP_MASK), /* TCP over IPv6 flows, fragmented, no vlan tag */ - MVPP2_DEF_FLOW(TCP_V6_FLOW, MVPP2_FL_IP6_TCP_FRAG_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_TCP6, MVPP2_FL_IP6_TCP_FRAG_UNTAG, MVPP22_CLS_HEK_IP6_2T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP6 | MVPP2_PRS_RI_IP_FRAG_TRUE | MVPP2_PRS_RI_L4_TCP, MVPP2_PRS_IP_MASK | MVPP2_PRS_RI_VLAN_MASK), - MVPP2_DEF_FLOW(TCP_V6_FLOW, MVPP2_FL_IP6_TCP_FRAG_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_TCP6, MVPP2_FL_IP6_TCP_FRAG_UNTAG, MVPP22_CLS_HEK_IP6_2T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP6_EXT | MVPP2_PRS_RI_IP_FRAG_TRUE | MVPP2_PRS_RI_L4_TCP, MVPP2_PRS_IP_MASK | MVPP2_PRS_RI_VLAN_MASK), /* TCP over IPv6 flows, fragmented, with vlan tag */ - MVPP2_DEF_FLOW(TCP_V6_FLOW, MVPP2_FL_IP6_TCP_FRAG_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_TCP6, MVPP2_FL_IP6_TCP_FRAG_TAG, MVPP22_CLS_HEK_IP6_2T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP6 | MVPP2_PRS_RI_IP_FRAG_TRUE | MVPP2_PRS_RI_L4_TCP, MVPP2_PRS_IP_MASK), - MVPP2_DEF_FLOW(TCP_V6_FLOW, MVPP2_FL_IP6_TCP_FRAG_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_TCP6, MVPP2_FL_IP6_TCP_FRAG_TAG, MVPP22_CLS_HEK_IP6_2T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP6_EXT | MVPP2_PRS_RI_IP_FRAG_TRUE | MVPP2_PRS_RI_L4_TCP, MVPP2_PRS_IP_MASK), /* UDP over IPv6 flows, not fragmented, no vlan tag */ - MVPP2_DEF_FLOW(UDP_V6_FLOW, MVPP2_FL_IP6_UDP_NF_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_UDP6, MVPP2_FL_IP6_UDP_NF_UNTAG, MVPP22_CLS_HEK_IP6_5T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP6 | MVPP2_PRS_RI_L4_UDP, MVPP2_PRS_IP_MASK | MVPP2_PRS_RI_VLAN_MASK), - MVPP2_DEF_FLOW(UDP_V6_FLOW, MVPP2_FL_IP6_UDP_NF_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_UDP6, MVPP2_FL_IP6_UDP_NF_UNTAG, MVPP22_CLS_HEK_IP6_5T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP6_EXT | MVPP2_PRS_RI_L4_UDP, MVPP2_PRS_IP_MASK | MVPP2_PRS_RI_VLAN_MASK), /* UDP over IPv6 flows, not fragmented, with vlan tag */ - MVPP2_DEF_FLOW(UDP_V6_FLOW, MVPP2_FL_IP6_UDP_NF_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_UDP6, MVPP2_FL_IP6_UDP_NF_TAG, MVPP22_CLS_HEK_IP6_5T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP6 | MVPP2_PRS_RI_L4_UDP, MVPP2_PRS_IP_MASK), - MVPP2_DEF_FLOW(UDP_V6_FLOW, MVPP2_FL_IP6_UDP_NF_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_UDP6, MVPP2_FL_IP6_UDP_NF_TAG, MVPP22_CLS_HEK_IP6_5T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP6_EXT | MVPP2_PRS_RI_L4_UDP, MVPP2_PRS_IP_MASK), /* UDP over IPv6 flows, fragmented, no vlan tag */ - MVPP2_DEF_FLOW(UDP_V6_FLOW, MVPP2_FL_IP6_UDP_FRAG_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_UDP6, MVPP2_FL_IP6_UDP_FRAG_UNTAG, MVPP22_CLS_HEK_IP6_2T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP6 | MVPP2_PRS_RI_IP_FRAG_TRUE | MVPP2_PRS_RI_L4_UDP, MVPP2_PRS_IP_MASK | MVPP2_PRS_RI_VLAN_MASK), - MVPP2_DEF_FLOW(UDP_V6_FLOW, MVPP2_FL_IP6_UDP_FRAG_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_UDP6, MVPP2_FL_IP6_UDP_FRAG_UNTAG, MVPP22_CLS_HEK_IP6_2T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP6_EXT | MVPP2_PRS_RI_IP_FRAG_TRUE | MVPP2_PRS_RI_L4_UDP, MVPP2_PRS_IP_MASK | MVPP2_PRS_RI_VLAN_MASK), /* UDP over IPv6 flows, fragmented, with vlan tag */ - MVPP2_DEF_FLOW(UDP_V6_FLOW, MVPP2_FL_IP6_UDP_FRAG_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_UDP6, MVPP2_FL_IP6_UDP_FRAG_TAG, MVPP22_CLS_HEK_IP6_2T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP6 | MVPP2_PRS_RI_IP_FRAG_TRUE | MVPP2_PRS_RI_L4_UDP, MVPP2_PRS_IP_MASK), - MVPP2_DEF_FLOW(UDP_V6_FLOW, MVPP2_FL_IP6_UDP_FRAG_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_UDP6, MVPP2_FL_IP6_UDP_FRAG_TAG, MVPP22_CLS_HEK_IP6_2T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP6_EXT | MVPP2_PRS_RI_IP_FRAG_TRUE | MVPP2_PRS_RI_L4_UDP, MVPP2_PRS_IP_MASK), /* IPv4 flows, no vlan tag */ - MVPP2_DEF_FLOW(IPV4_FLOW, MVPP2_FL_IP4_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_IP4, MVPP2_FL_IP4_UNTAG, MVPP22_CLS_HEK_IP4_2T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP4, MVPP2_PRS_RI_VLAN_MASK | MVPP2_PRS_RI_L3_PROTO_MASK), - MVPP2_DEF_FLOW(IPV4_FLOW, MVPP2_FL_IP4_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_IP4, MVPP2_FL_IP4_UNTAG, MVPP22_CLS_HEK_IP4_2T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP4_OPT, MVPP2_PRS_RI_VLAN_MASK | MVPP2_PRS_RI_L3_PROTO_MASK), - MVPP2_DEF_FLOW(IPV4_FLOW, MVPP2_FL_IP4_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_IP4, MVPP2_FL_IP4_UNTAG, MVPP22_CLS_HEK_IP4_2T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP4_OTHER, MVPP2_PRS_RI_VLAN_MASK | MVPP2_PRS_RI_L3_PROTO_MASK), /* IPv4 flows, with vlan tag */ - MVPP2_DEF_FLOW(IPV4_FLOW, MVPP2_FL_IP4_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_IP4, MVPP2_FL_IP4_TAG, MVPP22_CLS_HEK_IP4_2T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP4, MVPP2_PRS_RI_L3_PROTO_MASK), - MVPP2_DEF_FLOW(IPV4_FLOW, MVPP2_FL_IP4_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_IP4, MVPP2_FL_IP4_TAG, MVPP22_CLS_HEK_IP4_2T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP4_OPT, MVPP2_PRS_RI_L3_PROTO_MASK), - MVPP2_DEF_FLOW(IPV4_FLOW, MVPP2_FL_IP4_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_IP4, MVPP2_FL_IP4_TAG, MVPP22_CLS_HEK_IP4_2T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP4_OTHER, MVPP2_PRS_RI_L3_PROTO_MASK), /* IPv6 flows, no vlan tag */ - MVPP2_DEF_FLOW(IPV6_FLOW, MVPP2_FL_IP6_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_IP6, MVPP2_FL_IP6_UNTAG, MVPP22_CLS_HEK_IP6_2T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP6, MVPP2_PRS_RI_VLAN_MASK | MVPP2_PRS_RI_L3_PROTO_MASK), - MVPP2_DEF_FLOW(IPV6_FLOW, MVPP2_FL_IP6_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_IP6, MVPP2_FL_IP6_UNTAG, MVPP22_CLS_HEK_IP6_2T, MVPP2_PRS_RI_VLAN_NONE | MVPP2_PRS_RI_L3_IP6, MVPP2_PRS_RI_VLAN_MASK | MVPP2_PRS_RI_L3_PROTO_MASK), /* IPv6 flows, with vlan tag */ - MVPP2_DEF_FLOW(IPV6_FLOW, MVPP2_FL_IP6_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_IP6, MVPP2_FL_IP6_TAG, MVPP22_CLS_HEK_IP6_2T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP6, MVPP2_PRS_RI_L3_PROTO_MASK), - MVPP2_DEF_FLOW(IPV6_FLOW, MVPP2_FL_IP6_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_IP6, MVPP2_FL_IP6_TAG, MVPP22_CLS_HEK_IP6_2T | MVPP22_CLS_HEK_OPT_VLAN, MVPP2_PRS_RI_L3_IP6, MVPP2_PRS_RI_L3_PROTO_MASK), /* Non IP flow, no vlan tag */ - MVPP2_DEF_FLOW(ETHER_FLOW, MVPP2_FL_NON_IP_UNTAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_ETHERNET, MVPP2_FL_NON_IP_UNTAG, 0, MVPP2_PRS_RI_VLAN_NONE, MVPP2_PRS_RI_VLAN_MASK), /* Non IP flow, with vlan tag */ - MVPP2_DEF_FLOW(ETHER_FLOW, MVPP2_FL_NON_IP_TAG, + MVPP2_DEF_FLOW(MVPP22_FLOW_ETHERNET, MVPP2_FL_NON_IP_TAG, MVPP22_CLS_HEK_OPT_VLAN, 0, 0), }; @@ -344,9 +344,9 @@ static void mvpp2_cls_flow_write(struct mvpp2 *priv, struct mvpp2_cls_flow_entry *fe) { mvpp2_write(priv, MVPP2_CLS_FLOW_INDEX_REG, fe->index); - mvpp2_write(priv, MVPP2_CLS_FLOW_TBL0_REG, fe->data[0]); - mvpp2_write(priv, MVPP2_CLS_FLOW_TBL1_REG, fe->data[1]); - mvpp2_write(priv, MVPP2_CLS_FLOW_TBL2_REG, fe->data[2]); + mvpp2_write(priv, MVPP2_CLS_FLOW_TBL0_REG, fe->data[0]); + mvpp2_write(priv, MVPP2_CLS_FLOW_TBL1_REG, fe->data[1]); + mvpp2_write(priv, MVPP2_CLS_FLOW_TBL2_REG, fe->data[2]); } u32 mvpp2_cls_lookup_hits(struct mvpp2 *priv, int index) @@ -429,12 +429,6 @@ static void mvpp2_cls_flow_port_id_sel(struct mvpp2_cls_flow_entry *fe, fe->data[0] &= ~MVPP2_CLS_FLOW_TBL0_PORT_ID_SEL; } -static void mvpp2_cls_flow_seq_set(struct mvpp2_cls_flow_entry *fe, u32 seq) -{ - fe->data[1] &= ~MVPP2_CLS_FLOW_TBL1_SEQ(MVPP2_CLS_FLOW_TBL1_SEQ_MASK); - fe->data[1] |= MVPP2_CLS_FLOW_TBL1_SEQ(seq); -} - static void mvpp2_cls_flow_last_set(struct mvpp2_cls_flow_entry *fe, bool is_last) { @@ -454,9 +448,22 @@ static void mvpp2_cls_flow_port_add(struct mvpp2_cls_flow_entry *fe, fe->data[0] |= MVPP2_CLS_FLOW_TBL0_PORT_ID(port); } +static void mvpp2_cls_flow_port_remove(struct mvpp2_cls_flow_entry *fe, + u32 port) +{ + fe->data[0] &= ~MVPP2_CLS_FLOW_TBL0_PORT_ID(port); +} + +static void mvpp2_cls_flow_lu_type_set(struct mvpp2_cls_flow_entry *fe, + u8 lu_type) +{ + fe->data[1] &= ~MVPP2_CLS_FLOW_TBL1_LU_TYPE(MVPP2_CLS_LU_TYPE_MASK); + fe->data[1] |= MVPP2_CLS_FLOW_TBL1_LU_TYPE(lu_type); +} + /* Initialize the parser entry for the given flow */ static void mvpp2_cls_flow_prs_init(struct mvpp2 *priv, - struct mvpp2_cls_flow *flow) + const struct mvpp2_cls_flow *flow) { mvpp2_prs_add_flow(priv, flow->flow_id, flow->prs_ri.ri, flow->prs_ri.ri_mask); @@ -464,7 +471,7 @@ static void mvpp2_cls_flow_prs_init(struct mvpp2 *priv, /* Initialize the Lookup Id table entry for the given flow */ static void mvpp2_cls_flow_lkp_init(struct mvpp2 *priv, - struct mvpp2_cls_flow *flow) + const struct mvpp2_cls_flow *flow) { struct mvpp2_cls_lookup_entry le; @@ -477,7 +484,7 @@ static void mvpp2_cls_flow_lkp_init(struct mvpp2 *priv, /* We point on the first lookup in the sequence for the flow, that is * the C2 lookup. */ - le.data |= MVPP2_CLS_LKP_FLOW_PTR(MVPP2_FLOW_C2_ENTRY(flow->flow_id)); + le.data |= MVPP2_CLS_LKP_FLOW_PTR(MVPP2_CLS_FLT_FIRST(flow->flow_id)); /* CLS is always enabled, RSS is enabled/disabled in C2 lookup */ le.data |= MVPP2_CLS_LKP_TBL_LOOKUP_EN_MASK; @@ -485,21 +492,111 @@ static void mvpp2_cls_flow_lkp_init(struct mvpp2 *priv, mvpp2_cls_lookup_write(priv, &le); } +static void mvpp2_cls_c2_write(struct mvpp2 *priv, + struct mvpp2_cls_c2_entry *c2) +{ + u32 val; + mvpp2_write(priv, MVPP22_CLS_C2_TCAM_IDX, c2->index); + + val = mvpp2_read(priv, MVPP22_CLS_C2_TCAM_INV); + if (c2->valid) + val &= ~MVPP22_CLS_C2_TCAM_INV_BIT; + else + val |= MVPP22_CLS_C2_TCAM_INV_BIT; + mvpp2_write(priv, MVPP22_CLS_C2_TCAM_INV, val); + + mvpp2_write(priv, MVPP22_CLS_C2_ACT, c2->act); + + mvpp2_write(priv, MVPP22_CLS_C2_ATTR0, c2->attr[0]); + mvpp2_write(priv, MVPP22_CLS_C2_ATTR1, c2->attr[1]); + mvpp2_write(priv, MVPP22_CLS_C2_ATTR2, c2->attr[2]); + mvpp2_write(priv, MVPP22_CLS_C2_ATTR3, c2->attr[3]); + + mvpp2_write(priv, MVPP22_CLS_C2_TCAM_DATA0, c2->tcam[0]); + mvpp2_write(priv, MVPP22_CLS_C2_TCAM_DATA1, c2->tcam[1]); + mvpp2_write(priv, MVPP22_CLS_C2_TCAM_DATA2, c2->tcam[2]); + mvpp2_write(priv, MVPP22_CLS_C2_TCAM_DATA3, c2->tcam[3]); + /* Writing TCAM_DATA4 flushes writes to TCAM_DATA0-4 and INV to HW */ + mvpp2_write(priv, MVPP22_CLS_C2_TCAM_DATA4, c2->tcam[4]); +} + +void mvpp2_cls_c2_read(struct mvpp2 *priv, int index, + struct mvpp2_cls_c2_entry *c2) +{ + u32 val; + mvpp2_write(priv, MVPP22_CLS_C2_TCAM_IDX, index); + + c2->index = index; + + c2->tcam[0] = mvpp2_read(priv, MVPP22_CLS_C2_TCAM_DATA0); + c2->tcam[1] = mvpp2_read(priv, MVPP22_CLS_C2_TCAM_DATA1); + c2->tcam[2] = mvpp2_read(priv, MVPP22_CLS_C2_TCAM_DATA2); + c2->tcam[3] = mvpp2_read(priv, MVPP22_CLS_C2_TCAM_DATA3); + c2->tcam[4] = mvpp2_read(priv, MVPP22_CLS_C2_TCAM_DATA4); + + c2->act = mvpp2_read(priv, MVPP22_CLS_C2_ACT); + + c2->attr[0] = mvpp2_read(priv, MVPP22_CLS_C2_ATTR0); + c2->attr[1] = mvpp2_read(priv, MVPP22_CLS_C2_ATTR1); + c2->attr[2] = mvpp2_read(priv, MVPP22_CLS_C2_ATTR2); + c2->attr[3] = mvpp2_read(priv, MVPP22_CLS_C2_ATTR3); + + val = mvpp2_read(priv, MVPP22_CLS_C2_TCAM_INV); + c2->valid = !(val & MVPP22_CLS_C2_TCAM_INV_BIT); +} + +static int mvpp2_cls_ethtool_flow_to_type(int flow_type) +{ + switch (flow_type & ~(FLOW_EXT | FLOW_MAC_EXT | FLOW_RSS)) { + case TCP_V4_FLOW: + return MVPP22_FLOW_TCP4; + case TCP_V6_FLOW: + return MVPP22_FLOW_TCP6; + case UDP_V4_FLOW: + return MVPP22_FLOW_UDP4; + case UDP_V6_FLOW: + return MVPP22_FLOW_UDP6; + case IPV4_FLOW: + return MVPP22_FLOW_IP4; + case IPV6_FLOW: + return MVPP22_FLOW_IP6; + default: + return -EOPNOTSUPP; + } +} + +static int mvpp2_cls_c2_port_flow_index(struct mvpp2_port *port, int loc) +{ + return MVPP22_CLS_C2_RFS_LOC(port->id, loc); +} + /* Initialize the flow table entries for the given flow */ -static void mvpp2_cls_flow_init(struct mvpp2 *priv, struct mvpp2_cls_flow *flow) +static void mvpp2_cls_flow_init(struct mvpp2 *priv, + const struct mvpp2_cls_flow *flow) { struct mvpp2_cls_flow_entry fe; - int i; + int i, pri = 0; + + /* Assign default values to all entries in the flow */ + for (i = MVPP2_CLS_FLT_FIRST(flow->flow_id); + i <= MVPP2_CLS_FLT_LAST(flow->flow_id); i++) { + memset(&fe, 0, sizeof(fe)); + fe.index = i; + mvpp2_cls_flow_pri_set(&fe, pri++); + + if (i == MVPP2_CLS_FLT_LAST(flow->flow_id)) + mvpp2_cls_flow_last_set(&fe, 1); + + mvpp2_cls_flow_write(priv, &fe); + } - /* C2 lookup */ - memset(&fe, 0, sizeof(fe)); - fe.index = MVPP2_FLOW_C2_ENTRY(flow->flow_id); + /* RSS config C2 lookup */ + mvpp2_cls_flow_read(priv, MVPP2_CLS_FLT_C2_RSS_ENTRY(flow->flow_id), + &fe); mvpp2_cls_flow_eng_set(&fe, MVPP22_CLS_ENGINE_C2); mvpp2_cls_flow_port_id_sel(&fe, true); - mvpp2_cls_flow_last_set(&fe, 0); - mvpp2_cls_flow_pri_set(&fe, 0); - mvpp2_cls_flow_seq_set(&fe, MVPP2_CLS_FLOW_SEQ_FIRST1); + mvpp2_cls_flow_lu_type_set(&fe, MVPP22_FLOW_ETHERNET); /* Add all ports */ for (i = 0; i < MVPP2_MAX_PORTS; i++) @@ -509,22 +606,19 @@ static void mvpp2_cls_flow_init(struct mvpp2 *priv, struct mvpp2_cls_flow *flow) /* C3Hx lookups */ for (i = 0; i < MVPP2_MAX_PORTS; i++) { - memset(&fe, 0, sizeof(fe)); - fe.index = MVPP2_PORT_FLOW_HASH_ENTRY(i, flow->flow_id); + mvpp2_cls_flow_read(priv, + MVPP2_CLS_FLT_HASH_ENTRY(i, flow->flow_id), + &fe); + /* Set a default engine. Will be overwritten when setting the + * real HEK parameters + */ + mvpp2_cls_flow_eng_set(&fe, MVPP22_CLS_ENGINE_C3HA); mvpp2_cls_flow_port_id_sel(&fe, true); - mvpp2_cls_flow_pri_set(&fe, i + 1); - mvpp2_cls_flow_seq_set(&fe, MVPP2_CLS_FLOW_SEQ_MIDDLE); mvpp2_cls_flow_port_add(&fe, BIT(i)); mvpp2_cls_flow_write(priv, &fe); } - - /* Update the last entry */ - mvpp2_cls_flow_last_set(&fe, 1); - mvpp2_cls_flow_seq_set(&fe, MVPP2_CLS_FLOW_SEQ_LAST); - - mvpp2_cls_flow_write(priv, &fe); } /* Adds a field to the Header Extracted Key generation parameters*/ @@ -555,6 +649,9 @@ static int mvpp2_flow_set_hek_fields(struct mvpp2_cls_flow_entry *fe, for_each_set_bit(i, &hash_opts, MVPP22_CLS_HEK_N_FIELDS) { switch (BIT(i)) { + case MVPP22_CLS_HEK_OPT_MAC_DA: + field_id = MVPP22_CLS_FIELD_MAC_DA; + break; case MVPP22_CLS_HEK_OPT_VLAN: field_id = MVPP22_CLS_FIELD_VLAN; break; @@ -586,9 +683,29 @@ static int mvpp2_flow_set_hek_fields(struct mvpp2_cls_flow_entry *fe, return 0; } -struct mvpp2_cls_flow *mvpp2_cls_flow_get(int flow) +/* Returns the size, in bits, of the corresponding HEK field */ +static int mvpp2_cls_hek_field_size(u32 field) { - if (flow >= MVPP2_N_FLOWS) + switch (field) { + case MVPP22_CLS_HEK_OPT_MAC_DA: + return 48; + case MVPP22_CLS_HEK_OPT_IP4SA: + case MVPP22_CLS_HEK_OPT_IP4DA: + return 32; + case MVPP22_CLS_HEK_OPT_IP6SA: + case MVPP22_CLS_HEK_OPT_IP6DA: + return 128; + case MVPP22_CLS_HEK_OPT_L4SIP: + case MVPP22_CLS_HEK_OPT_L4DIP: + return 16; + default: + return -1; + } +} + +const struct mvpp2_cls_flow *mvpp2_cls_flow_get(int flow) +{ + if (flow >= MVPP2_N_PRS_FLOWS) return NULL; return &cls_flows[flow]; @@ -608,21 +725,17 @@ struct mvpp2_cls_flow *mvpp2_cls_flow_get(int flow) static int mvpp2_port_rss_hash_opts_set(struct mvpp2_port *port, int flow_type, u16 requested_opts) { + const struct mvpp2_cls_flow *flow; struct mvpp2_cls_flow_entry fe; - struct mvpp2_cls_flow *flow; int i, engine, flow_index; u16 hash_opts; - for (i = 0; i < MVPP2_N_FLOWS; i++) { + for_each_cls_flow_id_with_type(i, flow_type) { flow = mvpp2_cls_flow_get(i); if (!flow) return -EINVAL; - if (flow->flow_type != flow_type) - continue; - - flow_index = MVPP2_PORT_FLOW_HASH_ENTRY(port->id, - flow->flow_id); + flow_index = MVPP2_CLS_FLT_HASH_ENTRY(port->id, flow->flow_id); mvpp2_cls_flow_read(port->priv, flow_index, &fe); @@ -697,21 +810,17 @@ u16 mvpp2_flow_get_hek_fields(struct mvpp2_cls_flow_entry *fe) */ static u16 mvpp2_port_rss_hash_opts_get(struct mvpp2_port *port, int flow_type) { + const struct mvpp2_cls_flow *flow; struct mvpp2_cls_flow_entry fe; - struct mvpp2_cls_flow *flow; int i, flow_index; u16 hash_opts = 0; - for (i = 0; i < MVPP2_N_FLOWS; i++) { + for_each_cls_flow_id_with_type(i, flow_type) { flow = mvpp2_cls_flow_get(i); if (!flow) return 0; - if (flow->flow_type != flow_type) - continue; - - flow_index = MVPP2_PORT_FLOW_HASH_ENTRY(port->id, - flow->flow_id); + flow_index = MVPP2_CLS_FLT_HASH_ENTRY(port->id, flow->flow_id); mvpp2_cls_flow_read(port->priv, flow_index, &fe); @@ -723,10 +832,10 @@ static u16 mvpp2_port_rss_hash_opts_get(struct mvpp2_port *port, int flow_type) static void mvpp2_cls_port_init_flows(struct mvpp2 *priv) { - struct mvpp2_cls_flow *flow; + const struct mvpp2_cls_flow *flow; int i; - for (i = 0; i < MVPP2_N_FLOWS; i++) { + for (i = 0; i < MVPP2_N_PRS_FLOWS; i++) { flow = mvpp2_cls_flow_get(i); if (!flow) break; @@ -737,47 +846,6 @@ static void mvpp2_cls_port_init_flows(struct mvpp2 *priv) } } -static void mvpp2_cls_c2_write(struct mvpp2 *priv, - struct mvpp2_cls_c2_entry *c2) -{ - mvpp2_write(priv, MVPP22_CLS_C2_TCAM_IDX, c2->index); - - /* Write TCAM */ - mvpp2_write(priv, MVPP22_CLS_C2_TCAM_DATA0, c2->tcam[0]); - mvpp2_write(priv, MVPP22_CLS_C2_TCAM_DATA1, c2->tcam[1]); - mvpp2_write(priv, MVPP22_CLS_C2_TCAM_DATA2, c2->tcam[2]); - mvpp2_write(priv, MVPP22_CLS_C2_TCAM_DATA3, c2->tcam[3]); - mvpp2_write(priv, MVPP22_CLS_C2_TCAM_DATA4, c2->tcam[4]); - - mvpp2_write(priv, MVPP22_CLS_C2_ACT, c2->act); - - mvpp2_write(priv, MVPP22_CLS_C2_ATTR0, c2->attr[0]); - mvpp2_write(priv, MVPP22_CLS_C2_ATTR1, c2->attr[1]); - mvpp2_write(priv, MVPP22_CLS_C2_ATTR2, c2->attr[2]); - mvpp2_write(priv, MVPP22_CLS_C2_ATTR3, c2->attr[3]); -} - -void mvpp2_cls_c2_read(struct mvpp2 *priv, int index, - struct mvpp2_cls_c2_entry *c2) -{ - mvpp2_write(priv, MVPP22_CLS_C2_TCAM_IDX, index); - - c2->index = index; - - c2->tcam[0] = mvpp2_read(priv, MVPP22_CLS_C2_TCAM_DATA0); - c2->tcam[1] = mvpp2_read(priv, MVPP22_CLS_C2_TCAM_DATA1); - c2->tcam[2] = mvpp2_read(priv, MVPP22_CLS_C2_TCAM_DATA2); - c2->tcam[3] = mvpp2_read(priv, MVPP22_CLS_C2_TCAM_DATA3); - c2->tcam[4] = mvpp2_read(priv, MVPP22_CLS_C2_TCAM_DATA4); - - c2->act = mvpp2_read(priv, MVPP22_CLS_C2_ACT); - - c2->attr[0] = mvpp2_read(priv, MVPP22_CLS_C2_ATTR0); - c2->attr[1] = mvpp2_read(priv, MVPP22_CLS_C2_ATTR1); - c2->attr[2] = mvpp2_read(priv, MVPP22_CLS_C2_ATTR2); - c2->attr[3] = mvpp2_read(priv, MVPP22_CLS_C2_ATTR3); -} - static void mvpp2_port_c2_cls_init(struct mvpp2_port *port) { struct mvpp2_cls_c2_entry c2; @@ -791,6 +859,10 @@ static void mvpp2_port_c2_cls_init(struct mvpp2_port *port) c2.tcam[4] = MVPP22_CLS_C2_PORT_ID(pmap); c2.tcam[4] |= MVPP22_CLS_C2_TCAM_EN(MVPP22_CLS_C2_PORT_ID(pmap)); + /* Match on Lookup Type */ + c2.tcam[4] |= MVPP22_CLS_C2_TCAM_EN(MVPP22_CLS_C2_LU_TYPE(MVPP2_CLS_LU_TYPE_MASK)); + c2.tcam[4] |= MVPP22_CLS_C2_LU_TYPE(MVPP22_FLOW_ETHERNET); + /* Update RSS status after matching this entry */ c2.act = MVPP22_CLS_C2_ACT_RSS_EN(MVPP22_C2_UPD_LOCK); @@ -809,6 +881,8 @@ static void mvpp2_port_c2_cls_init(struct mvpp2_port *port) c2.attr[0] = MVPP22_CLS_C2_ATTR0_QHIGH(qh) | MVPP22_CLS_C2_ATTR0_QLOW(ql); + c2.valid = true; + mvpp2_cls_c2_write(port->priv, &c2); } @@ -817,6 +891,7 @@ void mvpp2_cls_init(struct mvpp2 *priv) { struct mvpp2_cls_lookup_entry le; struct mvpp2_cls_flow_entry fe; + struct mvpp2_cls_c2_entry c2; int index; /* Enable classifier */ @@ -840,6 +915,14 @@ void mvpp2_cls_init(struct mvpp2 *priv) mvpp2_cls_lookup_write(priv, &le); } + /* Clear C2 TCAM engine table */ + memset(&c2, 0, sizeof(c2)); + c2.valid = false; + for (index = 0; index < MVPP22_CLS_C2_N_ENTRIES; index++) { + c2.index = index; + mvpp2_cls_c2_write(priv, &c2); + } + mvpp2_cls_port_init_flows(priv); } @@ -902,16 +985,28 @@ static void mvpp2_rss_port_c2_disable(struct mvpp2_port *port) mvpp2_cls_c2_write(port->priv, &c2); } -void mvpp22_rss_enable(struct mvpp2_port *port) +void mvpp22_port_rss_enable(struct mvpp2_port *port) { mvpp2_rss_port_c2_enable(port); } -void mvpp22_rss_disable(struct mvpp2_port *port) +void mvpp22_port_rss_disable(struct mvpp2_port *port) { mvpp2_rss_port_c2_disable(port); } +static void mvpp22_port_c2_lookup_disable(struct mvpp2_port *port, int entry) +{ + struct mvpp2_cls_c2_entry c2; + + mvpp2_cls_c2_read(port->priv, entry, &c2); + + /* Clear the port map so that the entry doesn't match anymore */ + c2.tcam[4] &= ~(MVPP22_CLS_C2_PORT_ID(BIT(port->id))); + + mvpp2_cls_c2_write(port->priv, &c2); +} + /* Set CPU queue number for oversize packets */ void mvpp2_cls_oversize_rxq_set(struct mvpp2_port *port) { @@ -928,6 +1023,289 @@ void mvpp2_cls_oversize_rxq_set(struct mvpp2_port *port) mvpp2_write(port->priv, MVPP2_CLS_SWFWD_PCTRL_REG, val); } +static int mvpp2_port_c2_tcam_rule_add(struct mvpp2_port *port, + struct mvpp2_rfs_rule *rule) +{ + struct flow_action_entry *act; + struct mvpp2_cls_c2_entry c2; + u8 qh, ql, pmap; + int index; + + memset(&c2, 0, sizeof(c2)); + + index = mvpp2_cls_c2_port_flow_index(port, rule->loc); + if (index < 0) + return -EINVAL; + c2.index = index; + + act = &rule->flow->action.entries[0]; + + rule->c2_index = c2.index; + + c2.tcam[0] = (rule->c2_tcam & 0xffff) | + ((rule->c2_tcam_mask & 0xffff) << 16); + c2.tcam[1] = ((rule->c2_tcam >> 16) & 0xffff) | + (((rule->c2_tcam_mask >> 16) & 0xffff) << 16); + c2.tcam[2] = ((rule->c2_tcam >> 32) & 0xffff) | + (((rule->c2_tcam_mask >> 32) & 0xffff) << 16); + c2.tcam[3] = ((rule->c2_tcam >> 48) & 0xffff) | + (((rule->c2_tcam_mask >> 48) & 0xffff) << 16); + + pmap = BIT(port->id); + c2.tcam[4] = MVPP22_CLS_C2_PORT_ID(pmap); + c2.tcam[4] |= MVPP22_CLS_C2_TCAM_EN(MVPP22_CLS_C2_PORT_ID(pmap)); + + /* Match on Lookup Type */ + c2.tcam[4] |= MVPP22_CLS_C2_TCAM_EN(MVPP22_CLS_C2_LU_TYPE(MVPP2_CLS_LU_TYPE_MASK)); + c2.tcam[4] |= MVPP22_CLS_C2_LU_TYPE(rule->loc); + + if (act->id == FLOW_ACTION_DROP) { + c2.act = MVPP22_CLS_C2_ACT_COLOR(MVPP22_C2_COL_RED_LOCK); + } else { + /* We want to keep the default color derived from the Header + * Parser drop entries, for VLAN and MAC filtering. This will + * assign a default color of Green or Red, and we want matches + * with a non-drop action to keep that color. + */ + c2.act = MVPP22_CLS_C2_ACT_COLOR(MVPP22_C2_COL_NO_UPD_LOCK); + + /* Mark packet as "forwarded to software", needed for RSS */ + c2.act |= MVPP22_CLS_C2_ACT_FWD(MVPP22_C2_FWD_SW_LOCK); + + c2.act |= MVPP22_CLS_C2_ACT_QHIGH(MVPP22_C2_UPD_LOCK) | + MVPP22_CLS_C2_ACT_QLOW(MVPP22_C2_UPD_LOCK); + + qh = ((act->queue.index + port->first_rxq) >> 3) & MVPP22_CLS_C2_ATTR0_QHIGH_MASK; + ql = (act->queue.index + port->first_rxq) & MVPP22_CLS_C2_ATTR0_QLOW_MASK; + + c2.attr[0] = MVPP22_CLS_C2_ATTR0_QHIGH(qh) | + MVPP22_CLS_C2_ATTR0_QLOW(ql); + } + + c2.valid = true; + + mvpp2_cls_c2_write(port->priv, &c2); + + return 0; +} + +static int mvpp2_port_c2_rfs_rule_insert(struct mvpp2_port *port, + struct mvpp2_rfs_rule *rule) +{ + return mvpp2_port_c2_tcam_rule_add(port, rule); +} + +static int mvpp2_port_cls_rfs_rule_remove(struct mvpp2_port *port, + struct mvpp2_rfs_rule *rule) +{ + const struct mvpp2_cls_flow *flow; + struct mvpp2_cls_flow_entry fe; + int index, i; + + for_each_cls_flow_id_containing_type(i, rule->flow_type) { + flow = mvpp2_cls_flow_get(i); + if (!flow) + return 0; + + index = MVPP2_CLS_FLT_C2_RFS(port->id, flow->flow_id, rule->loc); + + mvpp2_cls_flow_read(port->priv, index, &fe); + mvpp2_cls_flow_port_remove(&fe, BIT(port->id)); + mvpp2_cls_flow_write(port->priv, &fe); + } + + if (rule->c2_index >= 0) + mvpp22_port_c2_lookup_disable(port, rule->c2_index); + + return 0; +} + +static int mvpp2_port_flt_rfs_rule_insert(struct mvpp2_port *port, + struct mvpp2_rfs_rule *rule) +{ + const struct mvpp2_cls_flow *flow; + struct mvpp2 *priv = port->priv; + struct mvpp2_cls_flow_entry fe; + int index, ret, i; + + if (rule->engine != MVPP22_CLS_ENGINE_C2) + return -EOPNOTSUPP; + + ret = mvpp2_port_c2_rfs_rule_insert(port, rule); + if (ret) + return ret; + + for_each_cls_flow_id_containing_type(i, rule->flow_type) { + flow = mvpp2_cls_flow_get(i); + if (!flow) + return 0; + + index = MVPP2_CLS_FLT_C2_RFS(port->id, flow->flow_id, rule->loc); + + mvpp2_cls_flow_read(priv, index, &fe); + mvpp2_cls_flow_eng_set(&fe, rule->engine); + mvpp2_cls_flow_port_id_sel(&fe, true); + mvpp2_flow_set_hek_fields(&fe, rule->hek_fields); + mvpp2_cls_flow_lu_type_set(&fe, rule->loc); + mvpp2_cls_flow_port_add(&fe, 0xf); + + mvpp2_cls_flow_write(priv, &fe); + } + + return 0; +} + +static int mvpp2_cls_c2_build_match(struct mvpp2_rfs_rule *rule) +{ + struct flow_rule *flow = rule->flow; + int offs = 64; + + if (flow_rule_match_key(flow, FLOW_DISSECTOR_KEY_PORTS)) { + struct flow_match_ports match; + + flow_rule_match_ports(flow, &match); + if (match.mask->src) { + rule->hek_fields |= MVPP22_CLS_HEK_OPT_L4SIP; + offs -= mvpp2_cls_hek_field_size(MVPP22_CLS_HEK_OPT_L4SIP); + + rule->c2_tcam |= ((u64)ntohs(match.key->src)) << offs; + rule->c2_tcam_mask |= ((u64)ntohs(match.mask->src)) << offs; + } + + if (match.mask->dst) { + rule->hek_fields |= MVPP22_CLS_HEK_OPT_L4DIP; + offs -= mvpp2_cls_hek_field_size(MVPP22_CLS_HEK_OPT_L4DIP); + + rule->c2_tcam |= ((u64)ntohs(match.key->dst)) << offs; + rule->c2_tcam_mask |= ((u64)ntohs(match.mask->dst)) << offs; + } + } + + if (hweight16(rule->hek_fields) > MVPP2_FLOW_N_FIELDS) + return -EOPNOTSUPP; + + return 0; +} + +static int mvpp2_cls_rfs_parse_rule(struct mvpp2_rfs_rule *rule) +{ + struct flow_rule *flow = rule->flow; + struct flow_action_entry *act; + + act = &flow->action.entries[0]; + if (act->id != FLOW_ACTION_QUEUE && act->id != FLOW_ACTION_DROP) + return -EOPNOTSUPP; + + /* For now, only use the C2 engine which has a HEK size limited to 64 + * bits for TCAM matching. + */ + rule->engine = MVPP22_CLS_ENGINE_C2; + + if (mvpp2_cls_c2_build_match(rule)) + return -EINVAL; + + return 0; +} + +int mvpp2_ethtool_cls_rule_get(struct mvpp2_port *port, + struct ethtool_rxnfc *rxnfc) +{ + struct mvpp2_ethtool_fs *efs; + + if (rxnfc->fs.location >= MVPP2_N_RFS_RULES) + return -EINVAL; + + efs = port->rfs_rules[rxnfc->fs.location]; + if (!efs) + return -ENOENT; + + memcpy(rxnfc, &efs->rxnfc, sizeof(efs->rxnfc)); + + return 0; +} + +int mvpp2_ethtool_cls_rule_ins(struct mvpp2_port *port, + struct ethtool_rxnfc *info) +{ + struct ethtool_rx_flow_spec_input input = {}; + struct ethtool_rx_flow_rule *ethtool_rule; + struct mvpp2_ethtool_fs *efs, *old_efs; + int ret = 0; + + if (info->fs.location >= 4 || + info->fs.location < 0) + return -EINVAL; + + efs = kzalloc(sizeof(*efs), GFP_KERNEL); + if (!efs) + return -ENOMEM; + + input.fs = &info->fs; + + ethtool_rule = ethtool_rx_flow_rule_create(&input); + if (IS_ERR(ethtool_rule)) { + ret = PTR_ERR(ethtool_rule); + goto clean_rule; + } + + efs->rule.flow = ethtool_rule->rule; + efs->rule.flow_type = mvpp2_cls_ethtool_flow_to_type(info->fs.flow_type); + + ret = mvpp2_cls_rfs_parse_rule(&efs->rule); + if (ret) + goto clean_eth_rule; + + efs->rule.loc = info->fs.location; + + /* Replace an already existing rule */ + if (port->rfs_rules[efs->rule.loc]) { + old_efs = port->rfs_rules[efs->rule.loc]; + ret = mvpp2_port_cls_rfs_rule_remove(port, &old_efs->rule); + if (ret) + goto clean_eth_rule; + kfree(old_efs); + port->n_rfs_rules--; + } + + ret = mvpp2_port_flt_rfs_rule_insert(port, &efs->rule); + if (ret) + goto clean_eth_rule; + + memcpy(&efs->rxnfc, info, sizeof(*info)); + port->rfs_rules[efs->rule.loc] = efs; + port->n_rfs_rules++; + + return ret; + +clean_eth_rule: + ethtool_rx_flow_rule_destroy(ethtool_rule); +clean_rule: + kfree(efs); + return ret; +} + +int mvpp2_ethtool_cls_rule_del(struct mvpp2_port *port, + struct ethtool_rxnfc *info) +{ + struct mvpp2_ethtool_fs *efs; + int ret; + + efs = port->rfs_rules[info->fs.location]; + if (!efs) + return -EINVAL; + + /* Remove the rule from the engines. */ + ret = mvpp2_port_cls_rfs_rule_remove(port, &efs->rule); + if (ret) + return ret; + + port->n_rfs_rules--; + port->rfs_rules[info->fs.location] = NULL; + kfree(efs); + + return 0; +} + static inline u32 mvpp22_rxfh_indir(struct mvpp2_port *port, u32 rxq) { int nrxqs, cpu, cpus = num_possible_cpus(); @@ -965,19 +1343,22 @@ void mvpp22_rss_fill_table(struct mvpp2_port *port, u32 table) int mvpp2_ethtool_rxfh_set(struct mvpp2_port *port, struct ethtool_rxnfc *info) { u16 hash_opts = 0; + u32 flow_type; - switch (info->flow_type) { - case TCP_V4_FLOW: - case UDP_V4_FLOW: - case TCP_V6_FLOW: - case UDP_V6_FLOW: + flow_type = mvpp2_cls_ethtool_flow_to_type(info->flow_type); + + switch (flow_type) { + case MVPP22_FLOW_TCP4: + case MVPP22_FLOW_UDP4: + case MVPP22_FLOW_TCP6: + case MVPP22_FLOW_UDP6: if (info->data & RXH_L4_B_0_1) hash_opts |= MVPP22_CLS_HEK_OPT_L4SIP; if (info->data & RXH_L4_B_2_3) hash_opts |= MVPP22_CLS_HEK_OPT_L4DIP; /* Fallthrough */ - case IPV4_FLOW: - case IPV6_FLOW: + case MVPP22_FLOW_IP4: + case MVPP22_FLOW_IP6: if (info->data & RXH_L2DA) hash_opts |= MVPP22_CLS_HEK_OPT_MAC_DA; if (info->data & RXH_VLAN) @@ -994,15 +1375,18 @@ int mvpp2_ethtool_rxfh_set(struct mvpp2_port *port, struct ethtool_rxnfc *info) default: return -EOPNOTSUPP; } - return mvpp2_port_rss_hash_opts_set(port, info->flow_type, hash_opts); + return mvpp2_port_rss_hash_opts_set(port, flow_type, hash_opts); } int mvpp2_ethtool_rxfh_get(struct mvpp2_port *port, struct ethtool_rxnfc *info) { unsigned long hash_opts; + u32 flow_type; int i; - hash_opts = mvpp2_port_rss_hash_opts_get(port, info->flow_type); + flow_type = mvpp2_cls_ethtool_flow_to_type(info->flow_type); + + hash_opts = mvpp2_port_rss_hash_opts_get(port, flow_type); info->data = 0; for_each_set_bit(i, &hash_opts, MVPP22_CLS_HEK_N_FIELDS) { @@ -1037,7 +1421,7 @@ int mvpp2_ethtool_rxfh_get(struct mvpp2_port *port, struct ethtool_rxnfc *info) return 0; } -void mvpp22_rss_port_init(struct mvpp2_port *port) +void mvpp22_port_rss_init(struct mvpp2_port *port) { struct mvpp2 *priv = port->priv; int i; @@ -1065,10 +1449,10 @@ void mvpp22_rss_port_init(struct mvpp2_port *port) mvpp22_rss_fill_table(port, port->id); /* Configure default flows */ - mvpp2_port_rss_hash_opts_set(port, IPV4_FLOW, MVPP22_CLS_HEK_IP4_2T); - mvpp2_port_rss_hash_opts_set(port, IPV6_FLOW, MVPP22_CLS_HEK_IP6_2T); - mvpp2_port_rss_hash_opts_set(port, TCP_V4_FLOW, MVPP22_CLS_HEK_IP4_5T); - mvpp2_port_rss_hash_opts_set(port, TCP_V6_FLOW, MVPP22_CLS_HEK_IP6_5T); - mvpp2_port_rss_hash_opts_set(port, UDP_V4_FLOW, MVPP22_CLS_HEK_IP4_5T); - mvpp2_port_rss_hash_opts_set(port, UDP_V6_FLOW, MVPP22_CLS_HEK_IP6_5T); + mvpp2_port_rss_hash_opts_set(port, MVPP22_FLOW_IP4, MVPP22_CLS_HEK_IP4_2T); + mvpp2_port_rss_hash_opts_set(port, MVPP22_FLOW_IP6, MVPP22_CLS_HEK_IP6_2T); + mvpp2_port_rss_hash_opts_set(port, MVPP22_FLOW_TCP4, MVPP22_CLS_HEK_IP4_5T); + mvpp2_port_rss_hash_opts_set(port, MVPP22_FLOW_TCP6, MVPP22_CLS_HEK_IP6_5T); + mvpp2_port_rss_hash_opts_set(port, MVPP22_FLOW_UDP4, MVPP22_CLS_HEK_IP4_5T); + mvpp2_port_rss_hash_opts_set(port, MVPP22_FLOW_UDP6, MVPP22_CLS_HEK_IP6_5T); } diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.h b/drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.h index 089f05f298917ae394b9245cdb255259d269b3c7..56b617375a65afd695e6de3cb4a4485ee06d5829 100644 --- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.h +++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_cls.h @@ -71,14 +71,6 @@ enum mvpp2_cls_field_id { MVPP22_CLS_FIELD_L4DIP = 0x1e, }; -enum mvpp2_cls_flow_seq { - MVPP2_CLS_FLOW_SEQ_NORMAL = 0, - MVPP2_CLS_FLOW_SEQ_FIRST1, - MVPP2_CLS_FLOW_SEQ_FIRST2, - MVPP2_CLS_FLOW_SEQ_LAST, - MVPP2_CLS_FLOW_SEQ_MIDDLE -}; - /* Classifier C2 engine constants */ #define MVPP22_CLS_C2_TCAM_EN(data) ((data) << 16) @@ -100,39 +92,62 @@ enum mvpp22_cls_c2_fwd_action { MVPP22_C2_FWD_HW_LOW_LAT_LOCK, }; +enum mvpp22_cls_c2_color_action { + MVPP22_C2_COL_NO_UPD = 0, + MVPP22_C2_COL_NO_UPD_LOCK, + MVPP22_C2_COL_GREEN, + MVPP22_C2_COL_GREEN_LOCK, + MVPP22_C2_COL_YELLOW, + MVPP22_C2_COL_YELLOW_LOCK, + MVPP22_C2_COL_RED, /* Drop */ + MVPP22_C2_COL_RED_LOCK, /* Drop */ +}; + #define MVPP2_CLS_C2_TCAM_WORDS 5 #define MVPP2_CLS_C2_ATTR_WORDS 5 struct mvpp2_cls_c2_entry { u32 index; + /* TCAM lookup key */ u32 tcam[MVPP2_CLS_C2_TCAM_WORDS]; + /* Actions to perform upon TCAM match */ u32 act; + /* Attributes relative to the actions to perform */ u32 attr[MVPP2_CLS_C2_ATTR_WORDS]; + /* Entry validity */ + u8 valid; }; +#define MVPP22_FLOW_ETHER_BIT BIT(0) +#define MVPP22_FLOW_IP4_BIT BIT(1) +#define MVPP22_FLOW_IP6_BIT BIT(2) +#define MVPP22_FLOW_TCP_BIT BIT(3) +#define MVPP22_FLOW_UDP_BIT BIT(4) + +#define MVPP22_FLOW_TCP4 (MVPP22_FLOW_ETHER_BIT | MVPP22_FLOW_IP4_BIT | MVPP22_FLOW_TCP_BIT) +#define MVPP22_FLOW_TCP6 (MVPP22_FLOW_ETHER_BIT | MVPP22_FLOW_IP6_BIT | MVPP22_FLOW_TCP_BIT) +#define MVPP22_FLOW_UDP4 (MVPP22_FLOW_ETHER_BIT | MVPP22_FLOW_IP4_BIT | MVPP22_FLOW_UDP_BIT) +#define MVPP22_FLOW_UDP6 (MVPP22_FLOW_ETHER_BIT | MVPP22_FLOW_IP6_BIT | MVPP22_FLOW_UDP_BIT) +#define MVPP22_FLOW_IP4 (MVPP22_FLOW_ETHER_BIT | MVPP22_FLOW_IP4_BIT) +#define MVPP22_FLOW_IP6 (MVPP22_FLOW_ETHER_BIT | MVPP22_FLOW_IP6_BIT) +#define MVPP22_FLOW_ETHERNET (MVPP22_FLOW_ETHER_BIT) + /* Classifier C2 engine entries */ -#define MVPP22_CLS_C2_RSS_ENTRY(port) (port) -#define MVPP22_CLS_C2_N_ENTRIES MVPP2_MAX_PORTS +#define MVPP22_CLS_C2_N_ENTRIES 256 -/* RSS flow entries in the flow table. We have 2 entries per port for RSS. - * - * The first performs a lookup using the C2 TCAM engine, to tag the - * packet for software forwarding (needed for RSS), enable or disable RSS, and - * assign the default rx queue. - * - * The second configures the hash generation, by specifying which fields of the - * packet header are used to generate the hash, and specifies the relevant hash - * engine to use. +/* Number of per-port dedicated entries in the C2 TCAM */ +#define MVPP22_CLS_C2_PORT_N_FLOWS MVPP2_N_RFS_ENTRIES_PER_FLOW + +/* Each port has oen range per flow type + one entry controling the global RSS + * setting and the default rx queue */ -#define MVPP22_RSS_FLOW_C2_OFFS 0 -#define MVPP22_RSS_FLOW_HASH_OFFS 1 -#define MVPP22_RSS_FLOW_SIZE (MVPP22_RSS_FLOW_HASH_OFFS + 1) +#define MVPP22_CLS_C2_PORT_RANGE (MVPP22_CLS_C2_PORT_N_FLOWS + 1) +#define MVPP22_CLS_C2_PORT_FIRST(p) ((p) * MVPP22_CLS_C2_PORT_RANGE) +#define MVPP22_CLS_C2_RSS_ENTRY(p) (MVPP22_CLS_C2_PORT_FIRST((p) + 1) - 1) -#define MVPP22_RSS_FLOW_C2(port) ((port) * MVPP22_RSS_FLOW_SIZE + \ - MVPP22_RSS_FLOW_C2_OFFS) -#define MVPP22_RSS_FLOW_HASH(port) ((port) * MVPP22_RSS_FLOW_SIZE + \ - MVPP22_RSS_FLOW_HASH_OFFS) -#define MVPP22_RSS_FLOW_FIRST(port) MVPP22_RSS_FLOW_C2(port) +#define MVPP22_CLS_C2_PORT_FLOW_FIRST(p) (MVPP22_CLS_C2_PORT_FIRST(p)) + +#define MVPP22_CLS_C2_RFS_LOC(p, loc) (MVPP22_CLS_C2_PORT_FLOW_FIRST(p) + (loc)) /* Packet flow ID */ enum mvpp2_prs_flow { @@ -162,6 +177,11 @@ enum mvpp2_prs_flow { MVPP2_FL_LAST, }; +/* LU Type defined for all engines, and specified in the flow table */ +#define MVPP2_CLS_LU_TYPE_MASK 0x3f + +#define MVPP2_N_FLOWS (MVPP2_FL_LAST - MVPP2_FL_START) + struct mvpp2_cls_flow { /* The L2-L4 traffic flow type */ int flow_type; @@ -176,12 +196,48 @@ struct mvpp2_cls_flow { struct mvpp2_prs_result_info prs_ri; }; -#define MVPP2_N_FLOWS 52 +#define MVPP2_CLS_FLT_ENTRIES_PER_FLOW (MVPP2_MAX_PORTS + 1 + 16) +#define MVPP2_CLS_FLT_FIRST(id) (((id) - MVPP2_FL_START) * \ + MVPP2_CLS_FLT_ENTRIES_PER_FLOW) + +#define MVPP2_CLS_FLT_C2_RFS(port, id, rfs_n) (MVPP2_CLS_FLT_FIRST(id) + \ + ((port) * MVPP2_MAX_PORTS) + \ + (rfs_n)) + +#define MVPP2_CLS_FLT_C2_RSS_ENTRY(id) (MVPP2_CLS_FLT_C2_RFS(MVPP2_MAX_PORTS, id, 0)) +#define MVPP2_CLS_FLT_HASH_ENTRY(port, id) (MVPP2_CLS_FLT_C2_RSS_ENTRY(id) + 1 + (port)) +#define MVPP2_CLS_FLT_LAST(id) (MVPP2_CLS_FLT_FIRST(id) + \ + MVPP2_CLS_FLT_ENTRIES_PER_FLOW - 1) + +/* Iterate on each classifier flow id. Sets 'i' to be the index of the first + * entry in the cls_flows table for each different flow_id. + * This relies on entries having the same flow_id in the cls_flows table being + * contiguous. + */ +#define for_each_cls_flow_id(i) \ + for ((i) = 0; (i) < MVPP2_N_PRS_FLOWS; (i)++) \ + if ((i) > 0 && \ + cls_flows[(i)].flow_id == cls_flows[(i) - 1].flow_id) \ + continue; \ + else + +/* Iterate on each classifier flow that has a given flow_type. Sets 'i' to be + * the index of the first entry in the cls_flow table for each different flow_id + * that has the given flow_type. This allows to operate on all flows that + * matches a given ethtool flow type. + */ +#define for_each_cls_flow_id_with_type(i, type) \ + for_each_cls_flow_id((i)) \ + if (cls_flows[(i)].flow_type != (type)) \ + continue; \ + else + +#define for_each_cls_flow_id_containing_type(i, type) \ + for_each_cls_flow_id((i)) \ + if ((cls_flows[(i)].flow_type & (type)) != (type)) \ + continue; \ + else -#define MVPP2_ENTRIES_PER_FLOW (MVPP2_MAX_PORTS + 1) -#define MVPP2_FLOW_C2_ENTRY(id) ((id) * MVPP2_ENTRIES_PER_FLOW) -#define MVPP2_PORT_FLOW_HASH_ENTRY(port, id) ((id) * MVPP2_ENTRIES_PER_FLOW + \ - (port) + 1) struct mvpp2_cls_flow_entry { u32 index; u32 data[MVPP2_CLS_FLOWS_TBL_DATA_WORDS]; @@ -194,11 +250,10 @@ struct mvpp2_cls_lookup_entry { }; void mvpp22_rss_fill_table(struct mvpp2_port *port, u32 table); +void mvpp22_port_rss_init(struct mvpp2_port *port); -void mvpp22_rss_port_init(struct mvpp2_port *port); - -void mvpp22_rss_enable(struct mvpp2_port *port); -void mvpp22_rss_disable(struct mvpp2_port *port); +void mvpp22_port_rss_enable(struct mvpp2_port *port); +void mvpp22_port_rss_disable(struct mvpp2_port *port); int mvpp2_ethtool_rxfh_get(struct mvpp2_port *port, struct ethtool_rxnfc *info); int mvpp2_ethtool_rxfh_set(struct mvpp2_port *port, struct ethtool_rxnfc *info); @@ -213,7 +268,7 @@ int mvpp2_cls_flow_eng_get(struct mvpp2_cls_flow_entry *fe); u16 mvpp2_flow_get_hek_fields(struct mvpp2_cls_flow_entry *fe); -struct mvpp2_cls_flow *mvpp2_cls_flow_get(int flow); +const struct mvpp2_cls_flow *mvpp2_cls_flow_get(int flow); u32 mvpp2_cls_flow_hits(struct mvpp2 *priv, int index); @@ -230,4 +285,13 @@ u32 mvpp2_cls_c2_hit_count(struct mvpp2 *priv, int c2_index); void mvpp2_cls_c2_read(struct mvpp2 *priv, int index, struct mvpp2_cls_c2_entry *c2); +int mvpp2_ethtool_cls_rule_get(struct mvpp2_port *port, + struct ethtool_rxnfc *rxnfc); + +int mvpp2_ethtool_cls_rule_ins(struct mvpp2_port *port, + struct ethtool_rxnfc *info); + +int mvpp2_ethtool_cls_rule_del(struct mvpp2_port *port, + struct ethtool_rxnfc *info); + #endif diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_debugfs.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_debugfs.c index f9744a61e5dd6fc6282eddca38b2718dac899e12..0ee39ea47b6b63be44de58f5897eb44d054b1477 100644 --- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_debugfs.c +++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_debugfs.c @@ -18,22 +18,48 @@ struct mvpp2_dbgfs_prs_entry { struct mvpp2 *priv; }; +struct mvpp2_dbgfs_c2_entry { + int id; + struct mvpp2 *priv; +}; + struct mvpp2_dbgfs_flow_entry { int flow; struct mvpp2 *priv; }; +struct mvpp2_dbgfs_flow_tbl_entry { + int id; + struct mvpp2 *priv; +}; + struct mvpp2_dbgfs_port_flow_entry { struct mvpp2_port *port; struct mvpp2_dbgfs_flow_entry *dbg_fe; }; +struct mvpp2_dbgfs_entries { + /* Entries for Header Parser debug info */ + struct mvpp2_dbgfs_prs_entry prs_entries[MVPP2_PRS_TCAM_SRAM_SIZE]; + + /* Entries for Classifier C2 engine debug info */ + struct mvpp2_dbgfs_c2_entry c2_entries[MVPP22_CLS_C2_N_ENTRIES]; + + /* Entries for Classifier Flow Table debug info */ + struct mvpp2_dbgfs_flow_tbl_entry flt_entries[MVPP2_CLS_FLOWS_TBL_SIZE]; + + /* Entries for Classifier flows debug info */ + struct mvpp2_dbgfs_flow_entry flow_entries[MVPP2_N_PRS_FLOWS]; + + /* Entries for per-port flows debug info */ + struct mvpp2_dbgfs_port_flow_entry port_flow_entries[MVPP2_MAX_PORTS]; +}; + static int mvpp2_dbgfs_flow_flt_hits_show(struct seq_file *s, void *unused) { - struct mvpp2_dbgfs_flow_entry *entry = s->private; - int id = MVPP2_FLOW_C2_ENTRY(entry->flow); + struct mvpp2_dbgfs_flow_tbl_entry *entry = s->private; - u32 hits = mvpp2_cls_flow_hits(entry->priv, id); + u32 hits = mvpp2_cls_flow_hits(entry->priv, entry->id); seq_printf(s, "%u\n", hits); @@ -58,7 +84,7 @@ DEFINE_SHOW_ATTRIBUTE(mvpp2_dbgfs_flow_dec_hits); static int mvpp2_dbgfs_flow_type_show(struct seq_file *s, void *unused) { struct mvpp2_dbgfs_flow_entry *entry = s->private; - struct mvpp2_cls_flow *f; + const struct mvpp2_cls_flow *f; const char *flow_name; f = mvpp2_cls_flow_get(entry->flow); @@ -93,30 +119,12 @@ static int mvpp2_dbgfs_flow_type_show(struct seq_file *s, void *unused) return 0; } -static int mvpp2_dbgfs_flow_type_open(struct inode *inode, struct file *file) -{ - return single_open(file, mvpp2_dbgfs_flow_type_show, inode->i_private); -} - -static int mvpp2_dbgfs_flow_type_release(struct inode *inode, struct file *file) -{ - struct seq_file *seq = file->private_data; - struct mvpp2_dbgfs_flow_entry *flow_entry = seq->private; - - kfree(flow_entry); - return single_release(inode, file); -} - -static const struct file_operations mvpp2_dbgfs_flow_type_fops = { - .open = mvpp2_dbgfs_flow_type_open, - .read = seq_read, - .release = mvpp2_dbgfs_flow_type_release, -}; +DEFINE_SHOW_ATTRIBUTE(mvpp2_dbgfs_flow_type); static int mvpp2_dbgfs_flow_id_show(struct seq_file *s, void *unused) { - struct mvpp2_dbgfs_flow_entry *entry = s->private; - struct mvpp2_cls_flow *f; + const struct mvpp2_dbgfs_flow_entry *entry = s->private; + const struct mvpp2_cls_flow *f; f = mvpp2_cls_flow_get(entry->flow); if (!f) @@ -134,7 +142,7 @@ static int mvpp2_dbgfs_port_flow_hash_opt_show(struct seq_file *s, void *unused) struct mvpp2_dbgfs_port_flow_entry *entry = s->private; struct mvpp2_port *port = entry->port; struct mvpp2_cls_flow_entry fe; - struct mvpp2_cls_flow *f; + const struct mvpp2_cls_flow *f; int flow_index; u16 hash_opts; @@ -142,7 +150,7 @@ static int mvpp2_dbgfs_port_flow_hash_opt_show(struct seq_file *s, void *unused) if (!f) return -EINVAL; - flow_index = MVPP2_PORT_FLOW_HASH_ENTRY(entry->port->id, f->flow_id); + flow_index = MVPP2_CLS_FLT_HASH_ENTRY(entry->port->id, f->flow_id); mvpp2_cls_flow_read(port->priv, flow_index, &fe); @@ -153,42 +161,21 @@ static int mvpp2_dbgfs_port_flow_hash_opt_show(struct seq_file *s, void *unused) return 0; } -static int mvpp2_dbgfs_port_flow_hash_opt_open(struct inode *inode, - struct file *file) -{ - return single_open(file, mvpp2_dbgfs_port_flow_hash_opt_show, - inode->i_private); -} - -static int mvpp2_dbgfs_port_flow_hash_opt_release(struct inode *inode, - struct file *file) -{ - struct seq_file *seq = file->private_data; - struct mvpp2_dbgfs_port_flow_entry *flow_entry = seq->private; - - kfree(flow_entry); - return single_release(inode, file); -} - -static const struct file_operations mvpp2_dbgfs_port_flow_hash_opt_fops = { - .open = mvpp2_dbgfs_port_flow_hash_opt_open, - .read = seq_read, - .release = mvpp2_dbgfs_port_flow_hash_opt_release, -}; +DEFINE_SHOW_ATTRIBUTE(mvpp2_dbgfs_port_flow_hash_opt); static int mvpp2_dbgfs_port_flow_engine_show(struct seq_file *s, void *unused) { struct mvpp2_dbgfs_port_flow_entry *entry = s->private; struct mvpp2_port *port = entry->port; struct mvpp2_cls_flow_entry fe; - struct mvpp2_cls_flow *f; + const struct mvpp2_cls_flow *f; int flow_index, engine; f = mvpp2_cls_flow_get(entry->dbg_fe->flow); if (!f) return -EINVAL; - flow_index = MVPP2_PORT_FLOW_HASH_ENTRY(entry->port->id, f->flow_id); + flow_index = MVPP2_CLS_FLT_HASH_ENTRY(entry->port->id, f->flow_id); mvpp2_cls_flow_read(port->priv, flow_index, &fe); @@ -203,11 +190,10 @@ DEFINE_SHOW_ATTRIBUTE(mvpp2_dbgfs_port_flow_engine); static int mvpp2_dbgfs_flow_c2_hits_show(struct seq_file *s, void *unused) { - struct mvpp2_port *port = s->private; + struct mvpp2_dbgfs_c2_entry *entry = s->private; u32 hits; - hits = mvpp2_cls_c2_hit_count(port->priv, - MVPP22_CLS_C2_RSS_ENTRY(port->id)); + hits = mvpp2_cls_c2_hit_count(entry->priv, entry->id); seq_printf(s, "%u\n", hits); @@ -218,11 +204,11 @@ DEFINE_SHOW_ATTRIBUTE(mvpp2_dbgfs_flow_c2_hits); static int mvpp2_dbgfs_flow_c2_rxq_show(struct seq_file *s, void *unused) { - struct mvpp2_port *port = s->private; + struct mvpp2_dbgfs_c2_entry *entry = s->private; struct mvpp2_cls_c2_entry c2; u8 qh, ql; - mvpp2_cls_c2_read(port->priv, MVPP22_CLS_C2_RSS_ENTRY(port->id), &c2); + mvpp2_cls_c2_read(entry->priv, entry->id, &c2); qh = (c2.attr[0] >> MVPP22_CLS_C2_ATTR0_QHIGH_OFFS) & MVPP22_CLS_C2_ATTR0_QHIGH_MASK; @@ -239,11 +225,11 @@ DEFINE_SHOW_ATTRIBUTE(mvpp2_dbgfs_flow_c2_rxq); static int mvpp2_dbgfs_flow_c2_enable_show(struct seq_file *s, void *unused) { - struct mvpp2_port *port = s->private; + struct mvpp2_dbgfs_c2_entry *entry = s->private; struct mvpp2_cls_c2_entry c2; int enabled; - mvpp2_cls_c2_read(port->priv, MVPP22_CLS_C2_RSS_ENTRY(port->id), &c2); + mvpp2_cls_c2_read(entry->priv, entry->id, &c2); enabled = !!(c2.attr[2] & MVPP22_CLS_C2_ATTR2_RSS_EN); @@ -456,25 +442,7 @@ static int mvpp2_dbgfs_prs_valid_show(struct seq_file *s, void *unused) return 0; } -static int mvpp2_dbgfs_prs_valid_open(struct inode *inode, struct file *file) -{ - return single_open(file, mvpp2_dbgfs_prs_valid_show, inode->i_private); -} - -static int mvpp2_dbgfs_prs_valid_release(struct inode *inode, struct file *file) -{ - struct seq_file *seq = file->private_data; - struct mvpp2_dbgfs_prs_entry *entry = seq->private; - - kfree(entry); - return single_release(inode, file); -} - -static const struct file_operations mvpp2_dbgfs_prs_valid_fops = { - .open = mvpp2_dbgfs_prs_valid_open, - .read = seq_read, - .release = mvpp2_dbgfs_prs_valid_release, -}; +DEFINE_SHOW_ATTRIBUTE(mvpp2_dbgfs_prs_valid); static int mvpp2_dbgfs_flow_port_init(struct dentry *parent, struct mvpp2_port *port, @@ -487,10 +455,7 @@ static int mvpp2_dbgfs_flow_port_init(struct dentry *parent, if (IS_ERR(port_dir)) return PTR_ERR(port_dir); - /* This will be freed by 'hash_opts' release op */ - port_entry = kmalloc(sizeof(*port_entry), GFP_KERNEL); - if (!port_entry) - return -ENOMEM; + port_entry = &port->priv->dbgfs_entries->port_flow_entries[port->id]; port_entry->port = port; port_entry->dbg_fe = entry; @@ -518,17 +483,11 @@ static int mvpp2_dbgfs_flow_entry_init(struct dentry *parent, if (!flow_entry_dir) return -ENOMEM; - /* This will be freed by 'type' release op */ - entry = kmalloc(sizeof(*entry), GFP_KERNEL); - if (!entry) - return -ENOMEM; + entry = &priv->dbgfs_entries->flow_entries[flow]; entry->flow = flow; entry->priv = priv; - debugfs_create_file("flow_hits", 0444, flow_entry_dir, entry, - &mvpp2_dbgfs_flow_flt_hits_fops); - debugfs_create_file("dec_hits", 0444, flow_entry_dir, entry, &mvpp2_dbgfs_flow_dec_hits_fops); @@ -545,6 +504,7 @@ static int mvpp2_dbgfs_flow_entry_init(struct dentry *parent, if (ret) return ret; } + return 0; } @@ -557,7 +517,7 @@ static int mvpp2_dbgfs_flow_init(struct dentry *parent, struct mvpp2 *priv) if (!flow_dir) return -ENOMEM; - for (i = 0; i < MVPP2_N_FLOWS; i++) { + for (i = 0; i < MVPP2_N_PRS_FLOWS; i++) { ret = mvpp2_dbgfs_flow_entry_init(flow_dir, priv, i); if (ret) return ret; @@ -582,10 +542,7 @@ static int mvpp2_dbgfs_prs_entry_init(struct dentry *parent, if (!prs_entry_dir) return -ENOMEM; - /* The 'valid' entry's ops will free that */ - entry = kmalloc(sizeof(*entry), GFP_KERNEL); - if (!entry) - return -ENOMEM; + entry = &priv->dbgfs_entries->prs_entries[tid]; entry->tid = tid; entry->priv = priv; @@ -630,6 +587,98 @@ static int mvpp2_dbgfs_prs_init(struct dentry *parent, struct mvpp2 *priv) return 0; } +static int mvpp2_dbgfs_c2_entry_init(struct dentry *parent, + struct mvpp2 *priv, int id) +{ + struct mvpp2_dbgfs_c2_entry *entry; + struct dentry *c2_entry_dir; + char c2_entry_name[10]; + + if (id >= MVPP22_CLS_C2_N_ENTRIES) + return -EINVAL; + + sprintf(c2_entry_name, "%03d", id); + + c2_entry_dir = debugfs_create_dir(c2_entry_name, parent); + if (!c2_entry_dir) + return -ENOMEM; + + entry = &priv->dbgfs_entries->c2_entries[id]; + + entry->id = id; + entry->priv = priv; + + debugfs_create_file("hits", 0444, c2_entry_dir, entry, + &mvpp2_dbgfs_flow_c2_hits_fops); + + debugfs_create_file("default_rxq", 0444, c2_entry_dir, entry, + &mvpp2_dbgfs_flow_c2_rxq_fops); + + debugfs_create_file("rss_enable", 0444, c2_entry_dir, entry, + &mvpp2_dbgfs_flow_c2_enable_fops); + + return 0; +} + +static int mvpp2_dbgfs_flow_tbl_entry_init(struct dentry *parent, + struct mvpp2 *priv, int id) +{ + struct mvpp2_dbgfs_flow_tbl_entry *entry; + struct dentry *flow_tbl_entry_dir; + char flow_tbl_entry_name[10]; + + if (id >= MVPP2_CLS_FLOWS_TBL_SIZE) + return -EINVAL; + + sprintf(flow_tbl_entry_name, "%03d", id); + + flow_tbl_entry_dir = debugfs_create_dir(flow_tbl_entry_name, parent); + if (!flow_tbl_entry_dir) + return -ENOMEM; + + entry = &priv->dbgfs_entries->flt_entries[id]; + + entry->id = id; + entry->priv = priv; + + debugfs_create_file("hits", 0444, flow_tbl_entry_dir, entry, + &mvpp2_dbgfs_flow_flt_hits_fops); + + return 0; +} + +static int mvpp2_dbgfs_cls_init(struct dentry *parent, struct mvpp2 *priv) +{ + struct dentry *cls_dir, *c2_dir, *flow_tbl_dir; + int i, ret; + + cls_dir = debugfs_create_dir("classifier", parent); + if (!cls_dir) + return -ENOMEM; + + c2_dir = debugfs_create_dir("c2", cls_dir); + if (!c2_dir) + return -ENOMEM; + + for (i = 0; i < MVPP22_CLS_C2_N_ENTRIES; i++) { + ret = mvpp2_dbgfs_c2_entry_init(c2_dir, priv, i); + if (ret) + return ret; + } + + flow_tbl_dir = debugfs_create_dir("flow_table", cls_dir); + if (!flow_tbl_dir) + return -ENOMEM; + + for (i = 0; i < MVPP2_CLS_FLOWS_TBL_SIZE; i++) { + ret = mvpp2_dbgfs_flow_tbl_entry_init(flow_tbl_dir, priv, i); + if (ret) + return ret; + } + + return 0; +} + static int mvpp2_dbgfs_port_init(struct dentry *parent, struct mvpp2_port *port) { @@ -648,21 +697,14 @@ static int mvpp2_dbgfs_port_init(struct dentry *parent, debugfs_create_file("vid_filter", 0444, port_dir, port, &mvpp2_dbgfs_port_vid_fops); - debugfs_create_file("c2_hits", 0444, port_dir, port, - &mvpp2_dbgfs_flow_c2_hits_fops); - - debugfs_create_file("default_rxq", 0444, port_dir, port, - &mvpp2_dbgfs_flow_c2_rxq_fops); - - debugfs_create_file("rss_enable", 0444, port_dir, port, - &mvpp2_dbgfs_flow_c2_enable_fops); - return 0; } void mvpp2_dbgfs_cleanup(struct mvpp2 *priv) { debugfs_remove_recursive(priv->dbgfs_dir); + + kfree(priv->dbgfs_entries); } void mvpp2_dbgfs_init(struct mvpp2 *priv, const char *name) @@ -682,11 +724,18 @@ void mvpp2_dbgfs_init(struct mvpp2 *priv, const char *name) return; priv->dbgfs_dir = mvpp2_dir; + priv->dbgfs_entries = kzalloc(sizeof(*priv->dbgfs_entries), GFP_KERNEL); + if (!priv->dbgfs_entries) + goto err; ret = mvpp2_dbgfs_prs_init(mvpp2_dir, priv); if (ret) goto err; + ret = mvpp2_dbgfs_cls_init(mvpp2_dir, priv); + if (ret) + goto err; + for (i = 0; i < priv->port_count; i++) { ret = mvpp2_dbgfs_port_init(mvpp2_dir, priv->port_list[i]); if (ret) diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c index 25fbed2b8d94674d43133f327740048773827115..56d43d9b43eff9a0cdaa474be9421e9a77a75b28 100644 --- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c @@ -3741,9 +3741,9 @@ static int mvpp2_set_features(struct net_device *dev, if (changed & NETIF_F_RXHASH) { if (features & NETIF_F_RXHASH) - mvpp22_rss_enable(port); + mvpp22_port_rss_enable(port); else - mvpp22_rss_disable(port); + mvpp22_port_rss_disable(port); } return 0; @@ -3937,7 +3937,7 @@ static int mvpp2_ethtool_get_rxnfc(struct net_device *dev, struct ethtool_rxnfc *info, u32 *rules) { struct mvpp2_port *port = netdev_priv(dev); - int ret = 0; + int ret = 0, i, loc = 0; if (!mvpp22_rss_is_supported()) return -EOPNOTSUPP; @@ -3949,6 +3949,18 @@ static int mvpp2_ethtool_get_rxnfc(struct net_device *dev, case ETHTOOL_GRXRINGS: info->data = port->nrxqs; break; + case ETHTOOL_GRXCLSRLCNT: + info->rule_cnt = port->n_rfs_rules; + break; + case ETHTOOL_GRXCLSRULE: + ret = mvpp2_ethtool_cls_rule_get(port, info); + break; + case ETHTOOL_GRXCLSRLALL: + for (i = 0; i < MVPP2_N_RFS_RULES; i++) { + if (port->rfs_rules[i]) + rules[loc++] = i; + } + break; default: return -ENOTSUPP; } @@ -3969,6 +3981,12 @@ static int mvpp2_ethtool_set_rxnfc(struct net_device *dev, case ETHTOOL_SRXFH: ret = mvpp2_ethtool_rxfh_set(port, info); break; + case ETHTOOL_SRXCLSRLINS: + ret = mvpp2_ethtool_cls_rule_ins(port, info); + break; + case ETHTOOL_SRXCLSRLDEL: + ret = mvpp2_ethtool_cls_rule_del(port, info); + break; default: return -EOPNOTSUPP; } @@ -4301,7 +4319,7 @@ static int mvpp2_port_init(struct mvpp2_port *port) mvpp2_cls_port_config(port); if (mvpp22_rss_is_supported()) - mvpp22_rss_port_init(port); + mvpp22_port_rss_init(port); /* Provide an initial Rx packet size */ port->pkt_size = MVPP2_RX_PKT_SIZE(port->dev->mtu); @@ -4848,6 +4866,7 @@ static int mvpp2_port_probe(struct platform_device *pdev, struct mvpp2_port *port; struct mvpp2_port_pcpu *port_pcpu; struct device_node *port_node = to_of_node(port_fwnode); + netdev_features_t features; struct net_device *dev; struct resource *res; struct phylink *phylink; @@ -4856,7 +4875,6 @@ static int mvpp2_port_probe(struct platform_device *pdev, unsigned long flags = 0; bool has_tx_irqs; u32 id; - int features; int phy_mode; int err, i; diff --git a/drivers/net/ethernet/marvell/pxa168_eth.c b/drivers/net/ethernet/marvell/pxa168_eth.c index 35f2142aac5e328b5068850e22a7c77bed63b60b..ce037e8530fa651d719cd9c4c678ccacfd909370 100644 --- a/drivers/net/ethernet/marvell/pxa168_eth.c +++ b/drivers/net/ethernet/marvell/pxa168_eth.c @@ -1461,7 +1461,7 @@ static int pxa168_eth_probe(struct platform_device *pdev) if (pdev->dev.of_node) mac_addr = of_get_mac_address(pdev->dev.of_node); - if (mac_addr && is_valid_ether_addr(mac_addr)) { + if (!IS_ERR_OR_NULL(mac_addr)) { ether_addr_copy(dev->dev_addr, mac_addr); } else { /* try reading the mac address, if set by the bootloader */ diff --git a/drivers/net/ethernet/marvell/sky2.c b/drivers/net/ethernet/marvell/sky2.c index 49486c10ef81379b5ea5539bf8281db71d00473a..9d070cca3e9eb000925802b4ed6ff0cc95879d55 100644 --- a/drivers/net/ethernet/marvell/sky2.c +++ b/drivers/net/ethernet/marvell/sky2.c @@ -4804,7 +4804,7 @@ static struct net_device *sky2_init_netdev(struct sky2_hw *hw, unsigned port, * 2) from internal registers set by bootloader */ iap = of_get_mac_address(hw->pdev->dev.of_node); - if (iap) + if (!IS_ERR(iap)) memcpy(dev->dev_addr, iap, ETH_ALEN); else memcpy_fromio(dev->dev_addr, hw->regs + B2_MAC_1 + port * 8, diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.c b/drivers/net/ethernet/mediatek/mtk_eth_soc.c index 549d36497b8c10767e29333f90c80bababe783da..f9fbb3ffa3a61e696a8078a98f41be1496a9ac45 100644 --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c @@ -767,7 +767,8 @@ static int mtk_tx_map(struct sk_buff *skb, struct net_device *dev, */ wmb(); - if (netif_xmit_stopped(netdev_get_tx_queue(dev, 0)) || !skb->xmit_more) + if (netif_xmit_stopped(netdev_get_tx_queue(dev, 0)) || + !netdev_xmit_more()) mtk_w32(eth, txd->txd2, MTK_QTX_CTX_PTR); return 0; @@ -2027,7 +2028,7 @@ static int __init mtk_init(struct net_device *dev) const char *mac_addr; mac_addr = of_get_mac_address(mac->of_node); - if (mac_addr) + if (!IS_ERR(mac_addr)) ether_addr_copy(dev->dev_addr, mac_addr); /* If the mac address is invalid, use random mac address */ diff --git a/drivers/net/ethernet/mellanox/mlx4/Kconfig b/drivers/net/ethernet/mellanox/mlx4/Kconfig index ff8057ed97ee29ceb51b363e982878186e5d74d4..8491db57b0b0c28fb6db1ac58ce090652ddce9f5 100644 --- a/drivers/net/ethernet/mellanox/mlx4/Kconfig +++ b/drivers/net/ethernet/mellanox/mlx4/Kconfig @@ -26,6 +26,7 @@ config MLX4_EN_DCB config MLX4_CORE tristate depends on PCI + select NET_DEVLINK default n config MLX4_DEBUG diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c index 2cbd2bd7c67ceae405ff2fa8114925a3e984e17d..36a92b19e613d8dba0a5f57461e7de3d09635310 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c @@ -685,16 +685,15 @@ static void build_inline_wqe(struct mlx4_en_tx_desc *tx_desc, } u16 mlx4_en_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { struct mlx4_en_priv *priv = netdev_priv(dev); u16 rings_p_up = priv->num_tx_rings_p_up; if (netdev_get_num_tc(dev)) - return fallback(dev, skb, NULL); + return netdev_pick_tx(dev, skb, NULL); - return fallback(dev, skb, NULL) % rings_p_up; + return netdev_pick_tx(dev, skb, NULL) % rings_p_up; } static void mlx4_bf_copy(void __iomem *dst, const void *src, @@ -1043,7 +1042,7 @@ netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_device *dev) send_doorbell = __netdev_tx_sent_queue(ring->tx_queue, tx_info->nr_bytes, - skb->xmit_more); + netdev_xmit_more()); real_size = (real_size / 16) & 0x3f; diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h index 8137454e253497e37b638a88432daf01184c95cb..630f15977f091c1e28eceb7b6bc33414a69d5694 100644 --- a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h @@ -698,8 +698,7 @@ void mlx4_en_arm_cq(struct mlx4_en_priv *priv, struct mlx4_en_cq *cq); void mlx4_en_tx_irq(struct mlx4_cq *mcq); u16 mlx4_en_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback); + struct net_device *sb_dev); netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_device *dev); netdev_tx_t mlx4_en_xmit_frame(struct mlx4_en_rx_ring *rx_ring, struct mlx4_en_rx_alloc *frame, diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Kconfig b/drivers/net/ethernet/mellanox/mlx5/core/Kconfig index 6debffb8336b6805d54fb1b59a86580de730a148..9aca8086ee019eee25984afbcb4dae10512ec0ae 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/Kconfig +++ b/drivers/net/ethernet/mellanox/mlx5/core/Kconfig @@ -5,6 +5,7 @@ config MLX5_CORE tristate "Mellanox 5th generation network adapters (ConnectX series) core driver" depends on PCI + select NET_DEVLINK imply PTP_1588_CLOCK imply VXLAN default n diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile index 1a16f6d73cbc59a16294b124ceedb68f09796da9..243368dc23db087c2b39de0531c431cb6ec8395d 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile +++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile @@ -22,7 +22,8 @@ mlx5_core-y := main.o cmd.o debugfs.o fw.o eq.o uar.o pagealloc.o \ # mlx5_core-$(CONFIG_MLX5_CORE_EN) += en_main.o en_common.o en_fs.o en_ethtool.o \ en_tx.o en_rx.o en_dim.o en_txrx.o en/xdp.o en_stats.o \ - en_selftest.o en/port.o en/monitor_stats.o en/reporter_tx.o + en_selftest.o en/port.o en/monitor_stats.o en/reporter_tx.o \ + en/params.o # # Netdev extra @@ -35,7 +36,7 @@ mlx5_core-$(CONFIG_MLX5_ESWITCH) += en_rep.o en_tc.o en/tc_tun.o lib/port_tu # # Core extra # -mlx5_core-$(CONFIG_MLX5_ESWITCH) += eswitch.o eswitch_offloads.o ecpf.o +mlx5_core-$(CONFIG_MLX5_ESWITCH) += eswitch.o eswitch_offloads.o ecpf.o rdma.o mlx5_core-$(CONFIG_MLX5_MPFS) += lib/mpfs.o mlx5_core-$(CONFIG_VXLAN) += lib/vxlan.o mlx5_core-$(CONFIG_PTP_1588_CLOCK) += lib/clock.o @@ -57,5 +58,3 @@ mlx5_core-$(CONFIG_MLX5_EN_IPSEC) += en_accel/ipsec.o en_accel/ipsec_rxtx.o \ en_accel/ipsec_stats.o mlx5_core-$(CONFIG_MLX5_EN_TLS) += en_accel/tls.o en_accel/tls_rxtx.o en_accel/tls_stats.o - -CFLAGS_tracepoint.o := -I$(src) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/alloc.c b/drivers/net/ethernet/mellanox/mlx5/core/alloc.c index 9008e17126db552a9ba8dc8ba0a47e5e78a27392..549f962cd86e5eacd74682c8bf75ee31e2dfea01 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/alloc.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/alloc.c @@ -57,15 +57,16 @@ static void *mlx5_dma_zalloc_coherent_node(struct mlx5_core_dev *dev, int node) { struct mlx5_priv *priv = &dev->priv; + struct device *device = dev->device; int original_node; void *cpu_handle; mutex_lock(&priv->alloc_mutex); - original_node = dev_to_node(&dev->pdev->dev); - set_dev_node(&dev->pdev->dev, node); - cpu_handle = dma_alloc_coherent(&dev->pdev->dev, size, dma_handle, + original_node = dev_to_node(device); + set_dev_node(device, node); + cpu_handle = dma_alloc_coherent(device, size, dma_handle, GFP_KERNEL); - set_dev_node(&dev->pdev->dev, original_node); + set_dev_node(device, original_node); mutex_unlock(&priv->alloc_mutex); return cpu_handle; } @@ -110,7 +111,7 @@ EXPORT_SYMBOL(mlx5_buf_alloc); void mlx5_buf_free(struct mlx5_core_dev *dev, struct mlx5_frag_buf *buf) { - dma_free_coherent(&dev->pdev->dev, buf->size, buf->frags->buf, + dma_free_coherent(dev->device, buf->size, buf->frags->buf, buf->frags->map); kfree(buf->frags); @@ -139,7 +140,7 @@ int mlx5_frag_buf_alloc_node(struct mlx5_core_dev *dev, int size, if (!frag->buf) goto err_free_buf; if (frag->map & ((1 << buf->page_shift) - 1)) { - dma_free_coherent(&dev->pdev->dev, frag_sz, + dma_free_coherent(dev->device, frag_sz, buf->frags[i].buf, buf->frags[i].map); mlx5_core_warn(dev, "unexpected map alignment: %pad, page_shift=%d\n", &frag->map, buf->page_shift); @@ -152,7 +153,7 @@ int mlx5_frag_buf_alloc_node(struct mlx5_core_dev *dev, int size, err_free_buf: while (i--) - dma_free_coherent(&dev->pdev->dev, PAGE_SIZE, buf->frags[i].buf, + dma_free_coherent(dev->device, PAGE_SIZE, buf->frags[i].buf, buf->frags[i].map); kfree(buf->frags); err_out: @@ -168,7 +169,7 @@ void mlx5_frag_buf_free(struct mlx5_core_dev *dev, struct mlx5_frag_buf *buf) for (i = 0; i < buf->npages; i++) { int frag_sz = min_t(int, size, PAGE_SIZE); - dma_free_coherent(&dev->pdev->dev, frag_sz, buf->frags[i].buf, + dma_free_coherent(dev->device, frag_sz, buf->frags[i].buf, buf->frags[i].map); size -= frag_sz; } @@ -274,7 +275,7 @@ void mlx5_db_free(struct mlx5_core_dev *dev, struct mlx5_db *db) __set_bit(db->index, db->u.pgdir->bitmap); if (bitmap_full(db->u.pgdir->bitmap, db_per_page)) { - dma_free_coherent(&(dev->pdev->dev), PAGE_SIZE, + dma_free_coherent(dev->device, PAGE_SIZE, db->u.pgdir->db_page, db->u.pgdir->db_dma); list_del(&db->u.pgdir->list); bitmap_free(db->u.pgdir->bitmap); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/cmd.c b/drivers/net/ethernet/mellanox/mlx5/core/cmd.c index c087d1014b097a52ca144871669c48fadfd12b2b..937ba4bcb056e195a47a8923e50eb3989131f715 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/cmd.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/cmd.c @@ -1346,7 +1346,7 @@ static void set_wqname(struct mlx5_core_dev *dev) struct mlx5_cmd *cmd = &dev->cmd; snprintf(cmd->wq_name, sizeof(cmd->wq_name), "mlx5_cmd_%s", - dev_name(&dev->pdev->dev)); + dev_name(dev->device)); } static void clean_debug_files(struct mlx5_core_dev *dev) @@ -1851,7 +1851,7 @@ static void create_msg_cache(struct mlx5_core_dev *dev) static int alloc_cmd_page(struct mlx5_core_dev *dev, struct mlx5_cmd *cmd) { - struct device *ddev = &dev->pdev->dev; + struct device *ddev = dev->device; cmd->cmd_alloc_buf = dma_alloc_coherent(ddev, MLX5_ADAPTER_PAGE_SIZE, &cmd->alloc_dma, GFP_KERNEL); @@ -1882,7 +1882,7 @@ static int alloc_cmd_page(struct mlx5_core_dev *dev, struct mlx5_cmd *cmd) static void free_cmd_page(struct mlx5_core_dev *dev, struct mlx5_cmd *cmd) { - struct device *ddev = &dev->pdev->dev; + struct device *ddev = dev->device; dma_free_coherent(ddev, cmd->alloc_size, cmd->cmd_alloc_buf, cmd->alloc_dma); @@ -1901,14 +1901,13 @@ int mlx5_cmd_init(struct mlx5_core_dev *dev) memset(cmd, 0, sizeof(*cmd)); cmd_if_rev = cmdif_rev(dev); if (cmd_if_rev != CMD_IF_REV) { - dev_err(&dev->pdev->dev, - "Driver cmdif rev(%d) differs from firmware's(%d)\n", - CMD_IF_REV, cmd_if_rev); + mlx5_core_err(dev, + "Driver cmdif rev(%d) differs from firmware's(%d)\n", + CMD_IF_REV, cmd_if_rev); return -EINVAL; } - cmd->pool = dma_pool_create("mlx5_cmd", &dev->pdev->dev, size, align, - 0); + cmd->pool = dma_pool_create("mlx5_cmd", dev->device, size, align, 0); if (!cmd->pool) return -ENOMEM; @@ -1920,14 +1919,14 @@ int mlx5_cmd_init(struct mlx5_core_dev *dev) cmd->log_sz = cmd_l >> 4 & 0xf; cmd->log_stride = cmd_l & 0xf; if (1 << cmd->log_sz > MLX5_MAX_COMMANDS) { - dev_err(&dev->pdev->dev, "firmware reports too many outstanding commands %d\n", - 1 << cmd->log_sz); + mlx5_core_err(dev, "firmware reports too many outstanding commands %d\n", + 1 << cmd->log_sz); err = -EINVAL; goto err_free_page; } if (cmd->log_sz + cmd->log_stride > MLX5_ADAPTER_PAGE_SHIFT) { - dev_err(&dev->pdev->dev, "command queue size overflow\n"); + mlx5_core_err(dev, "command queue size overflow\n"); err = -EINVAL; goto err_free_page; } @@ -1938,8 +1937,8 @@ int mlx5_cmd_init(struct mlx5_core_dev *dev) cmd->cmdif_rev = ioread32be(&dev->iseg->cmdif_rev_fw_sub) >> 16; if (cmd->cmdif_rev > CMD_IF_REV) { - dev_err(&dev->pdev->dev, "driver does not support command interface version. driver %d, firmware %d\n", - CMD_IF_REV, cmd->cmdif_rev); + mlx5_core_err(dev, "driver does not support command interface version. driver %d, firmware %d\n", + CMD_IF_REV, cmd->cmdif_rev); err = -EOPNOTSUPP; goto err_free_page; } @@ -1955,7 +1954,7 @@ int mlx5_cmd_init(struct mlx5_core_dev *dev) cmd_h = (u32)((u64)(cmd->dma) >> 32); cmd_l = (u32)(cmd->dma); if (cmd_l & 0xfff) { - dev_err(&dev->pdev->dev, "invalid command queue address\n"); + mlx5_core_err(dev, "invalid command queue address\n"); err = -ENOMEM; goto err_free_page; } @@ -1975,7 +1974,7 @@ int mlx5_cmd_init(struct mlx5_core_dev *dev) set_wqname(dev); cmd->wq = create_singlethread_workqueue(cmd->wq_name); if (!cmd->wq) { - dev_err(&dev->pdev->dev, "failed to create command workqueue\n"); + mlx5_core_err(dev, "failed to create command workqueue\n"); err = -ENOMEM; goto err_cache; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer_tracepoint.h b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer_tracepoint.h index 83f90e9aff45fa16c9db2b40ec737f214b502ae7..3038be575923b36aeee6e441be376aca4177507d 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer_tracepoint.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer_tracepoint.h @@ -47,7 +47,7 @@ TRACE_EVENT(mlx5_fw, TP_ARGS(tracer, trace_timestamp, lost, event_id, msg), TP_STRUCT__entry( - __string(dev_name, dev_name(&tracer->dev->pdev->dev)) + __string(dev_name, dev_name(tracer->dev->device)) __field(u64, trace_timestamp) __field(bool, lost) __field(u8, event_id) @@ -55,7 +55,8 @@ TRACE_EVENT(mlx5_fw, ), TP_fast_assign( - __assign_str(dev_name, dev_name(&tracer->dev->pdev->dev)); + __assign_str(dev_name, + dev_name(tracer->dev->device)); __entry->trace_timestamp = trace_timestamp; __entry->lost = lost; __entry->event_id = event_id; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h index d3eaf2ceaa3979b79a65d83beaaf072567d48c00..3a183d690e235e6457c6148eaf030627b5f6d2ed 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h @@ -240,8 +240,8 @@ struct mlx5e_params { bool rx_cqe_compress_def; struct net_dim_cq_moder rx_cq_moderation; struct net_dim_cq_moder tx_cq_moderation; + bool tunneled_offload_en; bool lro_en; - u32 lro_wqe_sz; u8 tx_min_inline_mode; bool vlan_strip_disable; bool scatter_fcs_en; @@ -410,14 +410,17 @@ struct mlx5e_xdp_info_fifo { struct mlx5e_xdp_wqe_info { u8 num_wqebbs; - u8 num_ds; + u8 num_pkts; }; struct mlx5e_xdp_mpwqe { /* Current MPWQE session */ struct mlx5e_tx_wqe *wqe; u8 ds_count; + u8 pkt_count; u8 max_ds_count; + u8 complete; + u8 inline_on; }; struct mlx5e_xdpsq; @@ -429,7 +432,6 @@ struct mlx5e_xdpsq { /* dirtied @completion */ u32 xdpi_fifo_cc; u16 cc; - bool redirect_flush; /* dirtied @xmit */ u32 xdpi_fifo_pc ____cacheline_aligned_in_smp; @@ -462,10 +464,10 @@ struct mlx5e_xdpsq { struct mlx5e_icosq { /* data path */ + u16 cc; + u16 pc; - /* dirtied @xmit */ - u16 pc ____cacheline_aligned_in_smp; - + struct mlx5_wqe_ctrl_seg *doorbell_cseg; struct mlx5e_cq cq; /* write@xmit, read@completion */ @@ -532,7 +534,8 @@ typedef bool (*mlx5e_fp_post_rx_wqes)(struct mlx5e_rq *rq); typedef void (*mlx5e_fp_dealloc_wqe)(struct mlx5e_rq*, u16); enum mlx5e_rq_flag { - MLX5E_RQ_FLAG_XDP_XMIT = BIT(0), + MLX5E_RQ_FLAG_XDP_XMIT, + MLX5E_RQ_FLAG_XDP_REDIRECT, }; struct mlx5e_rq_frag_info { @@ -563,8 +566,10 @@ struct mlx5e_rq { struct mlx5e_mpw_info *info; mlx5e_fp_skb_from_cqe_mpwrq skb_from_cqe_mpwrq; u16 num_strides; + u16 actual_wq_head; u8 log_stride_sz; - bool umr_in_progress; + u8 umr_in_progress; + u8 umr_last_bulk; } mpwqe; }; struct { @@ -769,12 +774,12 @@ struct mlx5e_profile { void mlx5e_build_ptys2ethtool_map(void); u16 mlx5e_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback); + struct net_device *sb_dev); netdev_tx_t mlx5e_xmit(struct sk_buff *skb, struct net_device *dev); netdev_tx_t mlx5e_sq_xmit(struct mlx5e_txqsq *sq, struct sk_buff *skb, - struct mlx5e_tx_wqe *wqe, u16 pi); + struct mlx5e_tx_wqe *wqe, u16 pi, bool xmit_more); +void mlx5e_trigger_irq(struct mlx5e_icosq *sq); void mlx5e_completion_event(struct mlx5_core_cq *mcq); void mlx5e_cq_error_event(struct mlx5_core_cq *mcq, enum mlx5_event event); int mlx5e_napi_poll(struct napi_struct *napi, int budget); @@ -886,6 +891,53 @@ static inline bool mlx5e_tunnel_inner_ft_supported(struct mlx5_core_dev *mdev) MLX5_CAP_FLOWTABLE_NIC_RX(mdev, ft_field_support.inner_ip_version)); } +static inline bool mlx5_tx_swp_supported(struct mlx5_core_dev *mdev) +{ + return MLX5_CAP_ETH(mdev, swp) && + MLX5_CAP_ETH(mdev, swp_csum) && MLX5_CAP_ETH(mdev, swp_lso); +} + +struct mlx5e_swp_spec { + __be16 l3_proto; + u8 l4_proto; + u8 is_tun; + __be16 tun_l3_proto; + u8 tun_l4_proto; +}; + +static inline void +mlx5e_set_eseg_swp(struct sk_buff *skb, struct mlx5_wqe_eth_seg *eseg, + struct mlx5e_swp_spec *swp_spec) +{ + /* SWP offsets are in 2-bytes words */ + eseg->swp_outer_l3_offset = skb_network_offset(skb) / 2; + if (swp_spec->l3_proto == htons(ETH_P_IPV6)) + eseg->swp_flags |= MLX5_ETH_WQE_SWP_OUTER_L3_IPV6; + if (swp_spec->l4_proto) { + eseg->swp_outer_l4_offset = skb_transport_offset(skb) / 2; + if (swp_spec->l4_proto == IPPROTO_UDP) + eseg->swp_flags |= MLX5_ETH_WQE_SWP_OUTER_L4_UDP; + } + + if (swp_spec->is_tun) { + eseg->swp_inner_l3_offset = skb_inner_network_offset(skb) / 2; + if (swp_spec->tun_l3_proto == htons(ETH_P_IPV6)) + eseg->swp_flags |= MLX5_ETH_WQE_SWP_INNER_L3_IPV6; + } else { /* typically for ipsec when xfrm mode != XFRM_MODE_TUNNEL */ + eseg->swp_inner_l3_offset = skb_network_offset(skb) / 2; + if (swp_spec->l3_proto == htons(ETH_P_IPV6)) + eseg->swp_flags |= MLX5_ETH_WQE_SWP_INNER_L3_IPV6; + } + switch (swp_spec->tun_l4_proto) { + case IPPROTO_UDP: + eseg->swp_flags |= MLX5_ETH_WQE_SWP_INNER_L4_UDP; + /* fall through */ + case IPPROTO_TCP: + eseg->swp_inner_l4_offset = skb_inner_transport_offset(skb) / 2; + break; + } +} + static inline void mlx5e_sq_fetch_wqe(struct mlx5e_txqsq *sq, struct mlx5e_tx_wqe **wqe, u16 *pi) @@ -930,7 +982,7 @@ void mlx5e_notify_hw(struct mlx5_wq_cyc *wq, u16 pc, */ wmb(); - mlx5_write64((__be32 *)ctrl, uar_map, NULL); + mlx5_write64((__be32 *)ctrl, uar_map); } static inline void mlx5e_cq_arm(struct mlx5e_cq *cq) @@ -1042,6 +1094,7 @@ mlx5e_create_netdev(struct mlx5_core_dev *mdev, const struct mlx5e_profile *prof int mlx5e_attach_netdev(struct mlx5e_priv *priv); void mlx5e_detach_netdev(struct mlx5e_priv *priv); void mlx5e_destroy_netdev(struct mlx5e_priv *priv); +void mlx5e_set_netdev_mtu_boundaries(struct mlx5e_priv *priv); void mlx5e_build_nic_params(struct mlx5_core_dev *mdev, struct mlx5e_rss_params *rss_params, struct mlx5e_params *params, diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c new file mode 100644 index 0000000000000000000000000000000000000000..d3744bffbae3991b8e13fc258ae65ab9406c2a52 --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c @@ -0,0 +1,104 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* Copyright (c) 2019 Mellanox Technologies. */ + +#include "en/params.h" + +u32 mlx5e_rx_get_linear_frag_sz(struct mlx5e_params *params) +{ + u16 hw_mtu = MLX5E_SW2HW_MTU(params, params->sw_mtu); + u16 linear_rq_headroom = params->xdp_prog ? + XDP_PACKET_HEADROOM : MLX5_RX_HEADROOM; + u32 frag_sz; + + linear_rq_headroom += NET_IP_ALIGN; + + frag_sz = MLX5_SKB_FRAG_SZ(linear_rq_headroom + hw_mtu); + + if (params->xdp_prog && frag_sz < PAGE_SIZE) + frag_sz = PAGE_SIZE; + + return frag_sz; +} + +u8 mlx5e_mpwqe_log_pkts_per_wqe(struct mlx5e_params *params) +{ + u32 linear_frag_sz = mlx5e_rx_get_linear_frag_sz(params); + + return MLX5_MPWRQ_LOG_WQE_SZ - order_base_2(linear_frag_sz); +} + +bool mlx5e_rx_is_linear_skb(struct mlx5e_params *params) +{ + u32 frag_sz = mlx5e_rx_get_linear_frag_sz(params); + + return !params->lro_en && frag_sz <= PAGE_SIZE; +} + +#define MLX5_MAX_MPWQE_LOG_WQE_STRIDE_SZ ((BIT(__mlx5_bit_sz(wq, log_wqe_stride_size)) - 1) + \ + MLX5_MPWQE_LOG_STRIDE_SZ_BASE) +bool mlx5e_rx_mpwqe_is_linear_skb(struct mlx5_core_dev *mdev, + struct mlx5e_params *params) +{ + u32 frag_sz = mlx5e_rx_get_linear_frag_sz(params); + s8 signed_log_num_strides_param; + u8 log_num_strides; + + if (!mlx5e_rx_is_linear_skb(params)) + return false; + + if (order_base_2(frag_sz) > MLX5_MAX_MPWQE_LOG_WQE_STRIDE_SZ) + return false; + + if (MLX5_CAP_GEN(mdev, ext_stride_num_range)) + return true; + + log_num_strides = MLX5_MPWRQ_LOG_WQE_SZ - order_base_2(frag_sz); + signed_log_num_strides_param = + (s8)log_num_strides - MLX5_MPWQE_LOG_NUM_STRIDES_BASE; + + return signed_log_num_strides_param >= 0; +} + +u8 mlx5e_mpwqe_get_log_rq_size(struct mlx5e_params *params) +{ + u8 log_pkts_per_wqe = mlx5e_mpwqe_log_pkts_per_wqe(params); + + /* Numbers are unsigned, don't subtract to avoid underflow. */ + if (params->log_rq_mtu_frames < + log_pkts_per_wqe + MLX5E_PARAMS_MINIMUM_LOG_RQ_SIZE_MPW) + return MLX5E_PARAMS_MINIMUM_LOG_RQ_SIZE_MPW; + + return params->log_rq_mtu_frames - log_pkts_per_wqe; +} + +u8 mlx5e_mpwqe_get_log_stride_size(struct mlx5_core_dev *mdev, + struct mlx5e_params *params) +{ + if (mlx5e_rx_mpwqe_is_linear_skb(mdev, params)) + return order_base_2(mlx5e_rx_get_linear_frag_sz(params)); + + return MLX5_MPWRQ_DEF_LOG_STRIDE_SZ(mdev); +} + +u8 mlx5e_mpwqe_get_log_num_strides(struct mlx5_core_dev *mdev, + struct mlx5e_params *params) +{ + return MLX5_MPWRQ_LOG_WQE_SZ - + mlx5e_mpwqe_get_log_stride_size(mdev, params); +} + +u16 mlx5e_get_rq_headroom(struct mlx5_core_dev *mdev, + struct mlx5e_params *params) +{ + u16 linear_rq_headroom = params->xdp_prog ? + XDP_PACKET_HEADROOM : MLX5_RX_HEADROOM; + bool is_linear_skb; + + linear_rq_headroom += NET_IP_ALIGN; + + is_linear_skb = (params->rq_wq_type == MLX5_WQ_TYPE_CYCLIC) ? + mlx5e_rx_is_linear_skb(params) : + mlx5e_rx_mpwqe_is_linear_skb(mdev, params); + + return is_linear_skb ? linear_rq_headroom : 0; +} diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/params.h b/drivers/net/ethernet/mellanox/mlx5/core/en/params.h new file mode 100644 index 0000000000000000000000000000000000000000..b106a0236f368d18ec583f75a4974ed12d14dcf2 --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/params.h @@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2019 Mellanox Technologies. */ + +#ifndef __MLX5_EN_PARAMS_H__ +#define __MLX5_EN_PARAMS_H__ + +#include "en.h" + +u32 mlx5e_rx_get_linear_frag_sz(struct mlx5e_params *params); +u8 mlx5e_mpwqe_log_pkts_per_wqe(struct mlx5e_params *params); +bool mlx5e_rx_is_linear_skb(struct mlx5e_params *params); +bool mlx5e_rx_mpwqe_is_linear_skb(struct mlx5_core_dev *mdev, + struct mlx5e_params *params); +u8 mlx5e_mpwqe_get_log_rq_size(struct mlx5e_params *params); +u8 mlx5e_mpwqe_get_log_stride_size(struct mlx5_core_dev *mdev, + struct mlx5e_params *params); +u8 mlx5e_mpwqe_get_log_num_strides(struct mlx5_core_dev *mdev, + struct mlx5e_params *params); +u16 mlx5e_get_rq_headroom(struct mlx5_core_dev *mdev, + struct mlx5e_params *params); + +#endif /* __MLX5_EN_PARAMS_H__ */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/port_buffer.c b/drivers/net/ethernet/mellanox/mlx5/core/en/port_buffer.c index 4ab0d030b54486f67096a190471a747b1ac18c56..633b117eb13e81437c402d4ed05efd892bf68eb1 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/port_buffer.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/port_buffer.c @@ -167,23 +167,23 @@ static int update_xoff_threshold(struct mlx5e_port_buffer *port_buffer, } /** - * update_buffer_lossy() - * max_mtu: netdev's max_mtu - * pfc_en: current pfc configuration - * buffer: current prio to buffer mapping - * xoff: xoff value - * port_buffer: port receive buffer configuration - * change: + * update_buffer_lossy - Update buffer configuration based on pfc + * @max_mtu: netdev's max_mtu + * @pfc_en: current pfc configuration + * @buffer: current prio to buffer mapping + * @xoff: xoff value + * @port_buffer: port receive buffer configuration + * @change: * - * Update buffer configuration based on pfc configuraiton and priority - * to buffer mapping. - * Buffer's lossy bit is changed to: - * lossless if there is at least one PFC enabled priority mapped to this buffer - * lossy if all priorities mapped to this buffer are PFC disabled + * Update buffer configuration based on pfc configuraiton and + * priority to buffer mapping. + * Buffer's lossy bit is changed to: + * lossless if there is at least one PFC enabled priority + * mapped to this buffer lossy if all priorities mapped to + * this buffer are PFC disabled * - * Return: - * Return 0 if no error. - * Set change to true if buffer configuration is modified. + * @return: 0 if no error, + * sets change to true if buffer configuration was modified. */ static int update_buffer_lossy(unsigned int max_mtu, u8 pfc_en, u8 *buffer, u32 xoff, diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/tc_tun.c b/drivers/net/ethernet/mellanox/mlx5/core/en/tc_tun.c index eec07b34b4ad07c627eebf36f87557296cc1701a..fe5d4d7f15edc80426bed373e2bf646dfd39dde7 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/tc_tun.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/tc_tun.c @@ -74,7 +74,7 @@ static int mlx5e_route_lookup_ipv4(struct mlx5e_priv *priv, if (ret) return ret; - if (mlx5_lag_is_multipath(mdev) && !rt->rt_gateway) + if (mlx5_lag_is_multipath(mdev) && rt->rt_gw_family != AF_INET) return -ENETUNREACH; #else return -EOPNOTSUPP; @@ -100,7 +100,7 @@ static const char *mlx5e_netdev_kind(struct net_device *dev) if (dev->rtnl_link_ops) return dev->rtnl_link_ops->kind; else - return ""; + return "unknown"; } static int mlx5e_route_lookup_ipv6(struct mlx5e_priv *priv, @@ -640,8 +640,10 @@ int mlx5e_tc_tun_parse(struct net_device *filter_dev, headers_c, headers_v); } else { netdev_warn(priv->netdev, - "decapsulation offload is not supported for %s net device (%d)\n", - mlx5e_netdev_kind(filter_dev), tunnel_type); + "decapsulation offload is not supported for %s (kind: \"%s\")\n", + netdev_name(filter_dev), + mlx5e_netdev_kind(filter_dev)); + return -EOPNOTSUPP; } return err; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c index cad34d6f5f451b1bfdf4b363ef25cbe50cab9fcd..eb8ef78e562610007b7adb9a26aaeb72575ace43 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c @@ -105,7 +105,7 @@ bool mlx5e_xdp_handle(struct mlx5e_rq *rq, struct mlx5e_dma_info *di, if (unlikely(err)) goto xdp_abort; __set_bit(MLX5E_RQ_FLAG_XDP_XMIT, rq->flags); - rq->xdpsq.redirect_flush = true; + __set_bit(MLX5E_RQ_FLAG_XDP_REDIRECT, rq->flags); mlx5e_page_dma_unmap(rq, di); rq->stats->xdp_redirect++; return true; @@ -125,6 +125,7 @@ xdp_abort: static void mlx5e_xdp_mpwqe_session_start(struct mlx5e_xdpsq *sq) { struct mlx5e_xdp_mpwqe *session = &sq->mpwqe; + struct mlx5e_xdpsq_stats *stats = sq->stats; struct mlx5_wq_cyc *wq = &sq->wq; u8 wqebbs; u16 pi; @@ -132,7 +133,9 @@ static void mlx5e_xdp_mpwqe_session_start(struct mlx5e_xdpsq *sq) mlx5e_xdpsq_fetch_wqe(sq, &session->wqe); prefetchw(session->wqe->data); - session->ds_count = MLX5E_XDP_TX_EMPTY_DS_COUNT; + session->ds_count = MLX5E_XDP_TX_EMPTY_DS_COUNT; + session->pkt_count = 0; + session->complete = 0; pi = mlx5_wq_cyc_ctr2ix(wq, sq->pc); @@ -151,6 +154,10 @@ static void mlx5e_xdp_mpwqe_session_start(struct mlx5e_xdpsq *sq) MLX5E_XDP_MPW_MAX_WQEBBS); session->max_ds_count = MLX5_SEND_WQEBB_NUM_DS * wqebbs; + + mlx5e_xdp_update_inline_state(sq); + + stats->mpwqe++; } static void mlx5e_xdp_mpwqe_complete(struct mlx5e_xdpsq *sq) @@ -167,7 +174,7 @@ static void mlx5e_xdp_mpwqe_complete(struct mlx5e_xdpsq *sq) cseg->qpn_ds = cpu_to_be32((sq->sqn << 8) | ds_count); wi->num_wqebbs = DIV_ROUND_UP(ds_count, MLX5_SEND_WQEBB_NUM_DS); - wi->num_ds = ds_count - MLX5E_XDP_TX_EMPTY_DS_COUNT; + wi->num_pkts = session->pkt_count; sq->pc += wi->num_wqebbs; @@ -182,11 +189,9 @@ static bool mlx5e_xmit_xdp_frame_mpwqe(struct mlx5e_xdpsq *sq, struct mlx5e_xdp_mpwqe *session = &sq->mpwqe; struct mlx5e_xdpsq_stats *stats = sq->stats; - dma_addr_t dma_addr = xdpi->dma_addr; struct xdp_frame *xdpf = xdpi->xdpf; - unsigned int dma_len = xdpf->len; - if (unlikely(sq->hw_mtu < dma_len)) { + if (unlikely(sq->hw_mtu < xdpf->len)) { stats->err++; return false; } @@ -203,9 +208,10 @@ static bool mlx5e_xmit_xdp_frame_mpwqe(struct mlx5e_xdpsq *sq, mlx5e_xdp_mpwqe_session_start(sq); } - mlx5e_xdp_mpwqe_add_dseg(sq, dma_addr, dma_len); + mlx5e_xdp_mpwqe_add_dseg(sq, xdpi, stats); - if (unlikely(session->ds_count == session->max_ds_count)) + if (unlikely(session->complete || + session->ds_count == session->max_ds_count)) mlx5e_xdp_mpwqe_complete(sq); mlx5e_xdpi_fifo_push(&sq->db.xdpi_fifo, xdpi); @@ -269,12 +275,33 @@ static bool mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xdp_info * return true; } +static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *sq, + struct mlx5e_xdp_wqe_info *wi, + struct mlx5e_rq *rq, + bool recycle) +{ + struct mlx5e_xdp_info_fifo *xdpi_fifo = &sq->db.xdpi_fifo; + u16 i; + + for (i = 0; i < wi->num_pkts; i++) { + struct mlx5e_xdp_info xdpi = mlx5e_xdpi_fifo_pop(xdpi_fifo); + + if (rq) { + /* XDP_TX */ + mlx5e_page_release(rq, &xdpi.di, recycle); + } else { + /* XDP_REDIRECT */ + dma_unmap_single(sq->pdev, xdpi.dma_addr, + xdpi.xdpf->len, DMA_TO_DEVICE); + xdp_return_frame(xdpi.xdpf); + } + } +} + bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq, struct mlx5e_rq *rq) { - struct mlx5e_xdp_info_fifo *xdpi_fifo; struct mlx5e_xdpsq *sq; struct mlx5_cqe64 *cqe; - bool is_redirect; u16 sqcc; int i; @@ -287,9 +314,6 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq, struct mlx5e_rq *rq) if (!cqe) return false; - is_redirect = !rq; - xdpi_fifo = &sq->db.xdpi_fifo; - /* sq->cc must be updated only after mlx5_cqwq_update_db_record(), * otherwise a cq overrun may occur */ @@ -311,7 +335,7 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq, struct mlx5e_rq *rq) do { struct mlx5e_xdp_wqe_info *wi; - u16 ci, j; + u16 ci; last_wqe = (sqcc == wqe_counter); ci = mlx5_wq_cyc_ctr2ix(&sq->wq, sqcc); @@ -319,19 +343,7 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq, struct mlx5e_rq *rq) sqcc += wi->num_wqebbs; - for (j = 0; j < wi->num_ds; j++) { - struct mlx5e_xdp_info xdpi = - mlx5e_xdpi_fifo_pop(xdpi_fifo); - - if (is_redirect) { - dma_unmap_single(sq->pdev, xdpi.dma_addr, - xdpi.xdpf->len, DMA_TO_DEVICE); - xdp_return_frame(xdpi.xdpf); - } else { - /* Recycle RX page */ - mlx5e_page_release(rq, &xdpi.di, true); - } - } + mlx5e_free_xdpsq_desc(sq, wi, rq, true); } while (!last_wqe); } while ((++i < MLX5E_TX_CQ_POLL_BUDGET) && (cqe = mlx5_cqwq_get_cqe(&cq->wq))); @@ -348,31 +360,16 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq, struct mlx5e_rq *rq) void mlx5e_free_xdpsq_descs(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq) { - struct mlx5e_xdp_info_fifo *xdpi_fifo = &sq->db.xdpi_fifo; - bool is_redirect = !rq; - while (sq->cc != sq->pc) { struct mlx5e_xdp_wqe_info *wi; - u16 ci, i; + u16 ci; ci = mlx5_wq_cyc_ctr2ix(&sq->wq, sq->cc); wi = &sq->db.wqe_info[ci]; sq->cc += wi->num_wqebbs; - for (i = 0; i < wi->num_ds; i++) { - struct mlx5e_xdp_info xdpi = - mlx5e_xdpi_fifo_pop(xdpi_fifo); - - if (is_redirect) { - dma_unmap_single(sq->pdev, xdpi.dma_addr, - xdpi.xdpf->len, DMA_TO_DEVICE); - xdp_return_frame(xdpi.xdpf); - } else { - /* Recycle RX page */ - mlx5e_page_release(rq, &xdpi.di, false); - } - } + mlx5e_free_xdpsq_desc(sq, wi, rq, false); } } @@ -439,9 +436,9 @@ void mlx5e_xdp_rx_poll_complete(struct mlx5e_rq *rq) mlx5e_xmit_xdp_doorbell(xdpsq); - if (xdpsq->redirect_flush) { + if (test_bit(MLX5E_RQ_FLAG_XDP_REDIRECT, rq->flags)) { xdp_do_flush_map(); - xdpsq->redirect_flush = false; + __clear_bit(MLX5E_RQ_FLAG_XDP_REDIRECT, rq->flags); } } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h index 553956cadc8a00d6bed384aa6cdad86759d5b2dd..8b537a4b084030f84d203c795a9dc98c5a5d3524 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h @@ -74,16 +74,68 @@ static inline void mlx5e_xmit_xdp_doorbell(struct mlx5e_xdpsq *sq) } } +/* Enable inline WQEs to shift some load from a congested HCA (HW) to + * a less congested cpu (SW). + */ +static inline void mlx5e_xdp_update_inline_state(struct mlx5e_xdpsq *sq) +{ + u16 outstanding = sq->xdpi_fifo_pc - sq->xdpi_fifo_cc; + struct mlx5e_xdp_mpwqe *session = &sq->mpwqe; + +#define MLX5E_XDP_INLINE_WATERMARK_LOW 10 +#define MLX5E_XDP_INLINE_WATERMARK_HIGH 128 + + if (session->inline_on) { + if (outstanding <= MLX5E_XDP_INLINE_WATERMARK_LOW) + session->inline_on = 0; + return; + } + + /* inline is false */ + if (outstanding >= MLX5E_XDP_INLINE_WATERMARK_HIGH) + session->inline_on = 1; +} + static inline void -mlx5e_xdp_mpwqe_add_dseg(struct mlx5e_xdpsq *sq, dma_addr_t dma_addr, u16 dma_len) +mlx5e_xdp_mpwqe_add_dseg(struct mlx5e_xdpsq *sq, struct mlx5e_xdp_info *xdpi, + struct mlx5e_xdpsq_stats *stats) { struct mlx5e_xdp_mpwqe *session = &sq->mpwqe; + dma_addr_t dma_addr = xdpi->dma_addr; + struct xdp_frame *xdpf = xdpi->xdpf; struct mlx5_wqe_data_seg *dseg = - (struct mlx5_wqe_data_seg *)session->wqe + session->ds_count++; + (struct mlx5_wqe_data_seg *)session->wqe + session->ds_count; + u16 dma_len = xdpf->len; + session->pkt_count++; + +#define MLX5E_XDP_INLINE_WQE_SZ_THRSD (256 - sizeof(struct mlx5_wqe_inline_seg)) + + if (session->inline_on && dma_len <= MLX5E_XDP_INLINE_WQE_SZ_THRSD) { + struct mlx5_wqe_inline_seg *inline_dseg = + (struct mlx5_wqe_inline_seg *)dseg; + u16 ds_len = sizeof(*inline_dseg) + dma_len; + u16 ds_cnt = DIV_ROUND_UP(ds_len, MLX5_SEND_WQE_DS); + + if (unlikely(session->ds_count + ds_cnt > session->max_ds_count)) { + /* Not enough space for inline wqe, send with memory pointer */ + session->complete = true; + goto no_inline; + } + + inline_dseg->byte_count = cpu_to_be32(dma_len | MLX5_INLINE_SEG); + memcpy(inline_dseg->data, xdpf->data, dma_len); + + session->ds_count += ds_cnt; + stats->inlnw++; + return; + } + +no_inline: dseg->addr = cpu_to_be64(dma_addr); dseg->byte_count = cpu_to_be32(dma_len); dseg->lkey = sq->mkey_be; + session->ds_count++; } static inline void mlx5e_xdpsq_fetch_wqe(struct mlx5e_xdpsq *sq, @@ -110,5 +162,4 @@ mlx5e_xdpi_fifo_pop(struct mlx5e_xdp_info_fifo *fifo) { return fifo->xi[(*fifo->cc)++ & fifo->mask]; } - #endif diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h index 1dd225380a66075eeceebe290209bfaf20cddf8c..6da7c88742dca01484bad12223ec35fa9ed982c1 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h @@ -40,6 +40,57 @@ #include "en_accel/tls_rxtx.h" #include "en.h" +#if IS_ENABLED(CONFIG_GENEVE) +static inline bool mlx5_geneve_tx_allowed(struct mlx5_core_dev *mdev) +{ + return mlx5_tx_swp_supported(mdev); +} + +static inline void +mlx5e_tx_tunnel_accel(struct sk_buff *skb, struct mlx5_wqe_eth_seg *eseg) +{ + struct mlx5e_swp_spec swp_spec = {}; + unsigned int offset = 0; + __be16 l3_proto; + u8 l4_proto; + + l3_proto = vlan_get_protocol(skb); + switch (l3_proto) { + case htons(ETH_P_IP): + l4_proto = ip_hdr(skb)->protocol; + break; + case htons(ETH_P_IPV6): + l4_proto = ipv6_find_hdr(skb, &offset, -1, NULL, NULL); + break; + default: + return; + } + + if (l4_proto != IPPROTO_UDP || + udp_hdr(skb)->dest != cpu_to_be16(GENEVE_UDP_PORT)) + return; + swp_spec.l3_proto = l3_proto; + swp_spec.l4_proto = l4_proto; + swp_spec.is_tun = true; + if (inner_ip_hdr(skb)->version == 6) { + swp_spec.tun_l3_proto = htons(ETH_P_IPV6); + swp_spec.tun_l4_proto = inner_ipv6_hdr(skb)->nexthdr; + } else { + swp_spec.tun_l3_proto = htons(ETH_P_IP); + swp_spec.tun_l4_proto = inner_ip_hdr(skb)->protocol; + } + + mlx5e_set_eseg_swp(skb, eseg, &swp_spec); +} + +#else +static inline bool mlx5_geneve_tx_allowed(struct mlx5_core_dev *mdev) +{ + return false; +} + +#endif /* CONFIG_GENEVE */ + static inline void mlx5e_udp_gso_handle_tx_skb(struct sk_buff *skb) { diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_rxtx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_rxtx.c index 53608afd39b6c4f1c27a873d9b3531047d4f6c12..0dd17514caae82b7388608fe884be9e664364472 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_rxtx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_rxtx.c @@ -136,7 +136,7 @@ static void mlx5e_ipsec_set_swp(struct sk_buff *skb, struct mlx5_wqe_eth_seg *eseg, u8 mode, struct xfrm_offload *xo) { - u8 proto; + struct mlx5e_swp_spec swp_spec = {}; /* Tunnel Mode: * SWP: OutL3 InL3 InL4 @@ -146,35 +146,23 @@ static void mlx5e_ipsec_set_swp(struct sk_buff *skb, * SWP: OutL3 InL4 * InL3 * Pkt: MAC IP ESP L4 - * - * Offsets are in 2-byte words, counting from start of frame */ - eseg->swp_outer_l3_offset = skb_network_offset(skb) / 2; - if (skb->protocol == htons(ETH_P_IPV6)) - eseg->swp_flags |= MLX5_ETH_WQE_SWP_OUTER_L3_IPV6; - - if (mode == XFRM_MODE_TUNNEL) { - eseg->swp_inner_l3_offset = skb_inner_network_offset(skb) / 2; + swp_spec.l3_proto = skb->protocol; + swp_spec.is_tun = mode == XFRM_MODE_TUNNEL; + if (swp_spec.is_tun) { if (xo->proto == IPPROTO_IPV6) { - eseg->swp_flags |= MLX5_ETH_WQE_SWP_INNER_L3_IPV6; - proto = inner_ipv6_hdr(skb)->nexthdr; + swp_spec.tun_l3_proto = htons(ETH_P_IPV6); + swp_spec.tun_l4_proto = inner_ipv6_hdr(skb)->nexthdr; } else { - proto = inner_ip_hdr(skb)->protocol; + swp_spec.tun_l3_proto = htons(ETH_P_IP); + swp_spec.tun_l4_proto = inner_ip_hdr(skb)->protocol; } } else { - eseg->swp_inner_l3_offset = skb_network_offset(skb) / 2; - if (skb->protocol == htons(ETH_P_IPV6)) - eseg->swp_flags |= MLX5_ETH_WQE_SWP_INNER_L3_IPV6; - proto = xo->proto; - } - switch (proto) { - case IPPROTO_UDP: - eseg->swp_flags |= MLX5_ETH_WQE_SWP_INNER_L4_UDP; - /* Fall through */ - case IPPROTO_TCP: - eseg->swp_inner_l4_offset = skb_inner_transport_offset(skb) / 2; - break; + swp_spec.tun_l3_proto = skb->protocol; + swp_spec.tun_l4_proto = xo->proto; } + + mlx5e_set_eseg_swp(skb, eseg, &swp_spec); } void mlx5e_ipsec_set_iv_esn(struct sk_buff *skb, struct xfrm_state *x, diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c index be137d4a91692026acf2a2bce299667fc7228dcc..439bf5953885eb25f2aba6dd4aa1cef4cc0f4ecf 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c @@ -181,7 +181,6 @@ static void mlx5e_tls_complete_sync_skb(struct sk_buff *skb, */ nskb->ip_summed = CHECKSUM_PARTIAL; - nskb->xmit_more = 1; nskb->queue_mapping = skb->queue_mapping; } @@ -248,7 +247,7 @@ mlx5e_tls_handle_ooo(struct mlx5e_tls_offload_context_tx *context, sq->stats->tls_resync_bytes += nskb->len; mlx5e_tls_complete_sync_skb(skb, nskb, tcp_seq, headln, cpu_to_be64(info.rcd_sn)); - mlx5e_sq_xmit(sq, nskb, *wqe, *pi); + mlx5e_sq_xmit(sq, nskb, *wqe, *pi, true); mlx5e_sq_fetch_wqe(sq, wqe, pi); return skb; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c index 78dc8fe2a83c3499d290ceffcb273ebcd7ef1932..7efaa58ae034a0f9f4f016db37cddde72c0884a0 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c @@ -1561,7 +1561,7 @@ static int mlx5e_get_module_info(struct net_device *netdev, struct mlx5e_priv *priv = netdev_priv(netdev); struct mlx5_core_dev *dev = priv->mdev; int size_read = 0; - u8 data[4]; + u8 data[4] = {0}; size_read = mlx5_query_module_eeprom(dev, 0, 2, data); if (size_read < 2) @@ -1571,17 +1571,17 @@ static int mlx5e_get_module_info(struct net_device *netdev, switch (data[0]) { case MLX5_MODULE_ID_QSFP: modinfo->type = ETH_MODULE_SFF_8436; - modinfo->eeprom_len = ETH_MODULE_SFF_8436_LEN; + modinfo->eeprom_len = ETH_MODULE_SFF_8436_MAX_LEN; break; case MLX5_MODULE_ID_QSFP_PLUS: case MLX5_MODULE_ID_QSFP28: /* data[1] = revision id */ if (data[0] == MLX5_MODULE_ID_QSFP28 || data[1] >= 0x3) { modinfo->type = ETH_MODULE_SFF_8636; - modinfo->eeprom_len = ETH_MODULE_SFF_8636_LEN; + modinfo->eeprom_len = ETH_MODULE_SFF_8636_MAX_LEN; } else { modinfo->type = ETH_MODULE_SFF_8436; - modinfo->eeprom_len = ETH_MODULE_SFF_8436_LEN; + modinfo->eeprom_len = ETH_MODULE_SFF_8436_MAX_LEN; } break; case MLX5_MODULE_ID_SFP: diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c index 46157e2a1e5ac36121f8ec96f9f5a09417b5fa67..457cc39423f2ba26c6c40798904f5293e55d064f 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c @@ -34,6 +34,7 @@ #include #include #include +#include #include #include #include @@ -43,6 +44,7 @@ #include "en_rep.h" #include "en_accel/ipsec.h" #include "en_accel/ipsec_rxtx.h" +#include "en_accel/en_accel.h" #include "en_accel/tls.h" #include "accel/ipsec.h" #include "accel/tls.h" @@ -53,6 +55,7 @@ #include "lib/eq.h" #include "en/monitor_stats.h" #include "en/reporter.h" +#include "en/params.h" struct mlx5e_rq_param { u32 rqc[MLX5_ST_SZ_DW(rqc)]; @@ -101,108 +104,9 @@ bool mlx5e_check_fragmented_striding_rq_cap(struct mlx5_core_dev *mdev) return true; } -static u32 mlx5e_rx_get_linear_frag_sz(struct mlx5e_params *params) -{ - u16 hw_mtu = MLX5E_SW2HW_MTU(params, params->sw_mtu); - u16 linear_rq_headroom = params->xdp_prog ? - XDP_PACKET_HEADROOM : MLX5_RX_HEADROOM; - u32 frag_sz; - - linear_rq_headroom += NET_IP_ALIGN; - - frag_sz = MLX5_SKB_FRAG_SZ(linear_rq_headroom + hw_mtu); - - if (params->xdp_prog && frag_sz < PAGE_SIZE) - frag_sz = PAGE_SIZE; - - return frag_sz; -} - -static u8 mlx5e_mpwqe_log_pkts_per_wqe(struct mlx5e_params *params) -{ - u32 linear_frag_sz = mlx5e_rx_get_linear_frag_sz(params); - - return MLX5_MPWRQ_LOG_WQE_SZ - order_base_2(linear_frag_sz); -} - -static bool mlx5e_rx_is_linear_skb(struct mlx5_core_dev *mdev, - struct mlx5e_params *params) -{ - u32 frag_sz = mlx5e_rx_get_linear_frag_sz(params); - - return !params->lro_en && frag_sz <= PAGE_SIZE; -} - -#define MLX5_MAX_MPWQE_LOG_WQE_STRIDE_SZ ((BIT(__mlx5_bit_sz(wq, log_wqe_stride_size)) - 1) + \ - MLX5_MPWQE_LOG_STRIDE_SZ_BASE) -static bool mlx5e_rx_mpwqe_is_linear_skb(struct mlx5_core_dev *mdev, - struct mlx5e_params *params) -{ - u32 frag_sz = mlx5e_rx_get_linear_frag_sz(params); - s8 signed_log_num_strides_param; - u8 log_num_strides; - - if (!mlx5e_rx_is_linear_skb(mdev, params)) - return false; - - if (order_base_2(frag_sz) > MLX5_MAX_MPWQE_LOG_WQE_STRIDE_SZ) - return false; - - if (MLX5_CAP_GEN(mdev, ext_stride_num_range)) - return true; - - log_num_strides = MLX5_MPWRQ_LOG_WQE_SZ - order_base_2(frag_sz); - signed_log_num_strides_param = - (s8)log_num_strides - MLX5_MPWQE_LOG_NUM_STRIDES_BASE; - - return signed_log_num_strides_param >= 0; -} - -static u8 mlx5e_mpwqe_get_log_rq_size(struct mlx5e_params *params) -{ - if (params->log_rq_mtu_frames < - mlx5e_mpwqe_log_pkts_per_wqe(params) + MLX5E_PARAMS_MINIMUM_LOG_RQ_SIZE_MPW) - return MLX5E_PARAMS_MINIMUM_LOG_RQ_SIZE_MPW; - - return params->log_rq_mtu_frames - mlx5e_mpwqe_log_pkts_per_wqe(params); -} - -static u8 mlx5e_mpwqe_get_log_stride_size(struct mlx5_core_dev *mdev, - struct mlx5e_params *params) -{ - if (mlx5e_rx_mpwqe_is_linear_skb(mdev, params)) - return order_base_2(mlx5e_rx_get_linear_frag_sz(params)); - - return MLX5_MPWRQ_DEF_LOG_STRIDE_SZ(mdev); -} - -static u8 mlx5e_mpwqe_get_log_num_strides(struct mlx5_core_dev *mdev, - struct mlx5e_params *params) -{ - return MLX5_MPWRQ_LOG_WQE_SZ - - mlx5e_mpwqe_get_log_stride_size(mdev, params); -} - -static u16 mlx5e_get_rq_headroom(struct mlx5_core_dev *mdev, - struct mlx5e_params *params) -{ - u16 linear_rq_headroom = params->xdp_prog ? - XDP_PACKET_HEADROOM : MLX5_RX_HEADROOM; - bool is_linear_skb; - - linear_rq_headroom += NET_IP_ALIGN; - - is_linear_skb = (params->rq_wq_type == MLX5_WQ_TYPE_CYCLIC) ? - mlx5e_rx_is_linear_skb(mdev, params) : - mlx5e_rx_mpwqe_is_linear_skb(mdev, params); - - return is_linear_skb ? linear_rq_headroom : 0; -} - void mlx5e_init_rq_type_params(struct mlx5_core_dev *mdev, struct mlx5e_params *params) { - params->lro_wqe_sz = MLX5E_PARAMS_DEFAULT_LRO_WQE_SZ; params->log_rq_mtu_frames = is_kdump_kernel() ? MLX5E_PARAMS_MINIMUM_LOG_RQ_SIZE : MLX5E_PARAMS_DEFAULT_LOG_RQ_SIZE; @@ -469,7 +373,6 @@ static void mlx5e_init_frags_partition(struct mlx5e_rq *rq) } static int mlx5e_init_di_list(struct mlx5e_rq *rq, - struct mlx5e_params *params, int wq_sz, int cpu) { int len = wq_sz << rq->wqe.info.log_num_frags; @@ -597,7 +500,7 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c, goto err_free; } - err = mlx5e_init_di_list(rq, params, wq_sz, c->cpu); + err = mlx5e_init_di_list(rq, wq_sz, c->cpu); if (err) goto err_free; rq->post_wqes = mlx5e_post_rx_wqes; @@ -615,7 +518,7 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c, goto err_free; } - rq->wqe.skb_from_cqe = mlx5e_rx_is_linear_skb(mdev, params) ? + rq->wqe.skb_from_cqe = mlx5e_rx_is_linear_skb(params) ? mlx5e_skb_from_cqe_linear : mlx5e_skb_from_cqe_nonlinear; rq->mkey_be = c->mkey_be; @@ -902,10 +805,14 @@ static void mlx5e_free_rx_descs(struct mlx5e_rq *rq) if (rq->wq_type == MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ) { struct mlx5_wq_ll *wq = &rq->mpwqe.wq; + u16 head = wq->head; + int i; - /* UMR WQE (if in progress) is always at wq->head */ - if (rq->mpwqe.umr_in_progress) - rq->dealloc_wqe(rq, wq->head); + /* Outstanding UMR WQEs (in progress) start at wq->head */ + for (i = 0; i < rq->mpwqe.umr_in_progress; i++) { + rq->dealloc_wqe(rq, head); + head = mlx5_wq_ll_get_wqe_next_ix(wq, head); + } while (!mlx5_wq_ll_is_empty(wq)) { struct mlx5e_rx_wqe_ll *wqe; @@ -970,16 +877,8 @@ err_free_rq: static void mlx5e_activate_rq(struct mlx5e_rq *rq) { - struct mlx5e_icosq *sq = &rq->channel->icosq; - struct mlx5_wq_cyc *wq = &sq->wq; - struct mlx5e_tx_wqe *nopwqe; - - u16 pi = mlx5_wq_cyc_ctr2ix(wq, sq->pc); - set_bit(MLX5E_RQ_STATE_ENABLED, &rq->state); - sq->db.ico_wqe[pi].opcode = MLX5_OPCODE_NOP; - nopwqe = mlx5e_post_nop(wq, sq->sqn, &sq->pc); - mlx5e_notify_hw(wq, sq->pc, sq->uar_map, &nopwqe->ctrl); + mlx5e_trigger_irq(&rq->channel->icosq); } static void mlx5e_deactivate_rq(struct mlx5e_rq *rq) @@ -1091,7 +990,7 @@ static void mlx5e_free_icosq_db(struct mlx5e_icosq *sq) static int mlx5e_alloc_icosq_db(struct mlx5e_icosq *sq, int numa) { - u8 wq_sz = mlx5_wq_cyc_get_size(&sq->wq); + int wq_sz = mlx5_wq_cyc_get_size(&sq->wq); sq->db.ico_wqe = kvzalloc_node(array_size(wq_sz, sizeof(*sq->db.ico_wqe)), @@ -1527,7 +1426,7 @@ static int mlx5e_open_xdpsq(struct mlx5e_channel *c, dseg->lkey = sq->mkey_be; wi->num_wqebbs = 1; - wi->num_ds = 1; + wi->num_pkts = 1; } } @@ -1895,7 +1794,7 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix, c->tstamp = &priv->tstamp; c->ix = ix; c->cpu = cpu; - c->pdev = &priv->mdev->pdev->dev; + c->pdev = priv->mdev->device; c->netdev = priv->netdev; c->mkey_be = cpu_to_be32(priv->mdev->mlx5e_res.mkey.key); c->num_tc = params->num_tc; @@ -2053,7 +1952,7 @@ static void mlx5e_build_rq_frags_info(struct mlx5_core_dev *mdev, byte_count += MLX5E_METADATA_ETHER_LEN; #endif - if (mlx5e_rx_is_linear_skb(mdev, params)) { + if (mlx5e_rx_is_linear_skb(params)) { int frag_stride; frag_stride = mlx5e_rx_get_linear_frag_sz(params); @@ -2107,6 +2006,13 @@ static inline u8 mlx5e_get_rqwq_log_stride(u8 wq_type, int ndsegs) return order_base_2(sz); } +static u8 mlx5e_get_rq_log_wq_sz(void *rqc) +{ + void *wq = MLX5_ADDR_OF(rqc, rqc, wq); + + return MLX5_GET(wq, wq, log_wq_sz); +} + static void mlx5e_build_rq_param(struct mlx5e_priv *priv, struct mlx5e_params *params, struct mlx5e_rq_param *param) @@ -2141,7 +2047,7 @@ static void mlx5e_build_rq_param(struct mlx5e_priv *priv, MLX5_SET(rqc, rqc, vsd, params->vlan_strip_disable); MLX5_SET(rqc, rqc, scatter_fcs, params->scatter_fcs_en); - param->wq.buf_numa_node = dev_to_node(&mdev->pdev->dev); + param->wq.buf_numa_node = dev_to_node(mdev->device); } static void mlx5e_build_drop_rq_param(struct mlx5e_priv *priv, @@ -2156,7 +2062,7 @@ static void mlx5e_build_drop_rq_param(struct mlx5e_priv *priv, mlx5e_get_rqwq_log_stride(MLX5_WQ_TYPE_CYCLIC, 1)); MLX5_SET(rqc, rqc, counter_set_id, priv->drop_rq_q_counter); - param->wq.buf_numa_node = dev_to_node(&mdev->pdev->dev); + param->wq.buf_numa_node = dev_to_node(mdev->device); } static void mlx5e_build_sq_param_common(struct mlx5e_priv *priv, @@ -2168,7 +2074,7 @@ static void mlx5e_build_sq_param_common(struct mlx5e_priv *priv, MLX5_SET(wq, wq, log_wq_stride, ilog2(MLX5_SEND_WQE_BB)); MLX5_SET(wq, wq, pd, priv->mdev->mlx5e_res.pdn); - param->wq.buf_numa_node = dev_to_node(&priv->mdev->pdev->dev); + param->wq.buf_numa_node = dev_to_node(priv->mdev->device); } static void mlx5e_build_sq_param(struct mlx5e_priv *priv, @@ -2177,10 +2083,13 @@ static void mlx5e_build_sq_param(struct mlx5e_priv *priv, { void *sqc = param->sqc; void *wq = MLX5_ADDR_OF(sqc, sqc, wq); + bool allow_swp; + allow_swp = mlx5_geneve_tx_allowed(priv->mdev) || + !!MLX5_IPSEC_DEV(priv->mdev); mlx5e_build_sq_param_common(priv, param); MLX5_SET(wq, wq, log_wq_sz, params->log_sq_size); - MLX5_SET(sqc, sqc, allow_swp, !!MLX5_IPSEC_DEV(priv->mdev)); + MLX5_SET(sqc, sqc, allow_swp, allow_swp); } static void mlx5e_build_common_cq_param(struct mlx5e_priv *priv, @@ -2270,13 +2179,28 @@ static void mlx5e_build_xdpsq_param(struct mlx5e_priv *priv, param->is_mpw = MLX5E_GET_PFLAG(params, MLX5E_PFLAG_XDP_TX_MPWQE); } +static u8 mlx5e_build_icosq_log_wq_sz(struct mlx5e_params *params, + struct mlx5e_rq_param *rqp) +{ + switch (params->rq_wq_type) { + case MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ: + return order_base_2(MLX5E_UMR_WQEBBS) + + mlx5e_get_rq_log_wq_sz(rqp->rqc); + default: /* MLX5_WQ_TYPE_CYCLIC */ + return MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE; + } +} + static void mlx5e_build_channel_param(struct mlx5e_priv *priv, struct mlx5e_params *params, struct mlx5e_channel_param *cparam) { - u8 icosq_log_wq_sz = MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE; + u8 icosq_log_wq_sz; mlx5e_build_rq_param(priv, params, &cparam->rq); + + icosq_log_wq_sz = mlx5e_build_icosq_log_wq_sz(params, &cparam->rq); + mlx5e_build_sq_param(priv, params, &cparam->sq); mlx5e_build_xdpsq_param(priv, params, &cparam->xdp_sq); mlx5e_build_icosq_param(priv, icosq_log_wq_sz, &cparam->icosq); @@ -2332,14 +2256,18 @@ static void mlx5e_activate_channels(struct mlx5e_channels *chs) mlx5e_activate_channel(chs->c[i]); } +#define MLX5E_RQ_WQES_TIMEOUT 20000 /* msecs */ + static int mlx5e_wait_channels_min_rx_wqes(struct mlx5e_channels *chs) { int err = 0; int i; - for (i = 0; i < chs->num; i++) - err |= mlx5e_wait_for_min_rx_wqes(&chs->c[i]->rq, - err ? 0 : 20000); + for (i = 0; i < chs->num; i++) { + int timeout = err ? 0 : MLX5E_RQ_WQES_TIMEOUT; + + err |= mlx5e_wait_for_min_rx_wqes(&chs->c[i]->rq, timeout); + } return err ? -ETIMEDOUT : 0; } @@ -2636,7 +2564,7 @@ static void mlx5e_build_tir_ctx_lro(struct mlx5e_params *params, void *tirc) MLX5_TIRC_LRO_ENABLE_MASK_IPV4_LRO | MLX5_TIRC_LRO_ENABLE_MASK_IPV6_LRO); MLX5_SET(tirc, tirc, lro_max_ip_payload_size, - (params->lro_wqe_sz - ROUGH_MAX_L2_L3_HDR_SZ) >> 8); + (MLX5E_PARAMS_DEFAULT_LRO_WQE_SZ - ROUGH_MAX_L2_L3_HDR_SZ) >> 8); MLX5_SET(tirc, tirc, lro_timeout_period_usecs, params->lro_timeout); } @@ -2746,22 +2674,6 @@ free_in: return err; } -static void mlx5e_build_inner_indir_tir_ctx(struct mlx5e_priv *priv, - enum mlx5e_traffic_types tt, - u32 *tirc) -{ - MLX5_SET(tirc, tirc, transport_domain, priv->mdev->mlx5e_res.td.tdn); - - mlx5e_build_tir_ctx_lro(&priv->channels.params, tirc); - - MLX5_SET(tirc, tirc, disp_type, MLX5_TIRC_DISP_TYPE_INDIRECT); - MLX5_SET(tirc, tirc, indirect_table, priv->indir_rqt.rqtn); - MLX5_SET(tirc, tirc, tunneled_offload_en, 0x1); - - mlx5e_build_indir_tir_ctx_hash(&priv->rss_params, - &tirc_default_config[tt], tirc, true); -} - static int mlx5e_set_mtu(struct mlx5_core_dev *mdev, struct mlx5e_params *params, u16 mtu) { @@ -2811,6 +2723,21 @@ int mlx5e_set_dev_port_mtu(struct mlx5e_priv *priv) return 0; } +void mlx5e_set_netdev_mtu_boundaries(struct mlx5e_priv *priv) +{ + struct mlx5e_params *params = &priv->channels.params; + struct net_device *netdev = priv->netdev; + struct mlx5_core_dev *mdev = priv->mdev; + u16 max_mtu; + + /* MTU range: 68 - hw-specific max */ + netdev->min_mtu = ETH_MIN_MTU; + + mlx5_query_port_max_mtu(mdev, &max_mtu, 1); + netdev->max_mtu = min_t(unsigned int, MLX5E_HW2SW_MTU(params, max_mtu), + ETH_MAX_MTU); +} + static void mlx5e_netdev_set_tcs(struct net_device *netdev) { struct mlx5e_priv *priv = netdev_priv(netdev); @@ -3058,8 +2985,8 @@ static int mlx5e_alloc_drop_cq(struct mlx5_core_dev *mdev, struct mlx5e_cq *cq, struct mlx5e_cq_param *param) { - param->wq.buf_numa_node = dev_to_node(&mdev->pdev->dev); - param->wq.db_numa_node = dev_to_node(&mdev->pdev->dev); + param->wq.buf_numa_node = dev_to_node(mdev->device); + param->wq.db_numa_node = dev_to_node(mdev->device); return mlx5e_alloc_cq_common(mdev, param, cq); } @@ -3167,32 +3094,42 @@ static void mlx5e_cleanup_nic_tx(struct mlx5e_priv *priv) mlx5e_destroy_tis(priv->mdev, priv->tisn[tc]); } -static void mlx5e_build_indir_tir_ctx(struct mlx5e_priv *priv, - enum mlx5e_traffic_types tt, - u32 *tirc) +static void mlx5e_build_indir_tir_ctx_common(struct mlx5e_priv *priv, + u32 rqtn, u32 *tirc) { MLX5_SET(tirc, tirc, transport_domain, priv->mdev->mlx5e_res.td.tdn); + MLX5_SET(tirc, tirc, disp_type, MLX5_TIRC_DISP_TYPE_INDIRECT); + MLX5_SET(tirc, tirc, indirect_table, rqtn); + MLX5_SET(tirc, tirc, tunneled_offload_en, + priv->channels.params.tunneled_offload_en); mlx5e_build_tir_ctx_lro(&priv->channels.params, tirc); +} - MLX5_SET(tirc, tirc, disp_type, MLX5_TIRC_DISP_TYPE_INDIRECT); - MLX5_SET(tirc, tirc, indirect_table, priv->indir_rqt.rqtn); - +static void mlx5e_build_indir_tir_ctx(struct mlx5e_priv *priv, + enum mlx5e_traffic_types tt, + u32 *tirc) +{ + mlx5e_build_indir_tir_ctx_common(priv, priv->indir_rqt.rqtn, tirc); mlx5e_build_indir_tir_ctx_hash(&priv->rss_params, &tirc_default_config[tt], tirc, false); } static void mlx5e_build_direct_tir_ctx(struct mlx5e_priv *priv, u32 rqtn, u32 *tirc) { - MLX5_SET(tirc, tirc, transport_domain, priv->mdev->mlx5e_res.td.tdn); - - mlx5e_build_tir_ctx_lro(&priv->channels.params, tirc); - - MLX5_SET(tirc, tirc, disp_type, MLX5_TIRC_DISP_TYPE_INDIRECT); - MLX5_SET(tirc, tirc, indirect_table, rqtn); + mlx5e_build_indir_tir_ctx_common(priv, rqtn, tirc); MLX5_SET(tirc, tirc, rx_hash_fn, MLX5_RX_HASH_FN_INVERTED_XOR8); } +static void mlx5e_build_inner_indir_tir_ctx(struct mlx5e_priv *priv, + enum mlx5e_traffic_types tt, + u32 *tirc) +{ + mlx5e_build_indir_tir_ctx_common(priv, priv->indir_rqt.rqtn, tirc); + mlx5e_build_indir_tir_ctx_hash(&priv->rss_params, + &tirc_default_config[tt], tirc, true); +} + int mlx5e_create_indirect_tirs(struct mlx5e_priv *priv, bool inner_ttc) { struct mlx5e_tir *tir; @@ -3775,7 +3712,7 @@ int mlx5e_change_mtu(struct net_device *netdev, int new_mtu, new_channels.params.sw_mtu = new_mtu; if (params->xdp_prog && - !mlx5e_rx_is_linear_skb(priv->mdev, &new_channels.params)) { + !mlx5e_rx_is_linear_skb(&new_channels.params)) { netdev_err(netdev, "MTU(%d) > %d is not allowed while XDP enabled\n", new_mtu, mlx5e_xdp_max_mtu(params)); err = -EINVAL; @@ -4115,6 +4052,12 @@ static netdev_features_t mlx5e_tunnel_features_check(struct mlx5e_priv *priv, /* Verify if UDP port is being offloaded by HW */ if (mlx5_vxlan_lookup_port(priv->mdev->vxlan, port)) return features; + +#if IS_ENABLED(CONFIG_GENEVE) + /* Support Geneve offload for default UDP port */ + if (port == GENEVE_UDP_PORT && mlx5_geneve_tx_allowed(priv->mdev)) + return features; +#endif } out: @@ -4210,7 +4153,7 @@ static int mlx5e_xdp_allowed(struct mlx5e_priv *priv, struct bpf_prog *prog) new_channels.params = priv->channels.params; new_channels.params.xdp_prog = prog; - if (!mlx5e_rx_is_linear_skb(priv->mdev, &new_channels.params)) { + if (!mlx5e_rx_is_linear_skb(&new_channels.params)) { netdev_warn(netdev, "XDP is not allowed with MTU(%d) > %d\n", new_channels.params.sw_mtu, mlx5e_xdp_max_mtu(&new_channels.params)); @@ -4264,7 +4207,7 @@ static int mlx5e_xdp_set(struct net_device *netdev, struct bpf_prog *prog) mlx5e_set_rq_type(priv->mdev, &priv->channels.params); if (was_opened && reset) - mlx5e_open_locked(netdev); + err = mlx5e_open_locked(netdev); if (!test_bit(MLX5E_STATE_OPENED, &priv->state) || reset) goto unlock; @@ -4554,7 +4497,7 @@ void mlx5e_build_rq_params(struct mlx5_core_dev *mdev, if (!slow_pci_heuristic(mdev) && mlx5e_striding_rq_possible(mdev, params) && (mlx5e_rx_mpwqe_is_linear_skb(mdev, params) || - !mlx5e_rx_is_linear_skb(mdev, params))) + !mlx5e_rx_is_linear_skb(params))) MLX5E_SET_PFLAG(params, MLX5E_PFLAG_RX_STRIDING_RQ, true); mlx5e_set_rq_type(mdev, params); mlx5e_init_rq_type_params(mdev, params); @@ -4630,6 +4573,8 @@ void mlx5e_build_nic_params(struct mlx5_core_dev *mdev, /* RSS */ mlx5e_build_rss_params(rss_params, params->num_channels); + params->tunneled_offload_en = + mlx5e_tunnel_inner_ft_supported(mdev); } static void mlx5e_set_netdev_dev_addr(struct net_device *netdev) @@ -4651,7 +4596,7 @@ static void mlx5e_build_nic_netdev(struct net_device *netdev) bool fcs_supported; bool fcs_enabled; - SET_NETDEV_DEV(netdev, &mdev->pdev->dev); + SET_NETDEV_DEV(netdev, mdev->device); netdev->netdev_ops = &mlx5e_netdev_ops; @@ -4686,7 +4631,8 @@ static void mlx5e_build_nic_netdev(struct net_device *netdev) netdev->hw_features |= NETIF_F_HW_VLAN_CTAG_FILTER; netdev->hw_features |= NETIF_F_HW_VLAN_STAG_TX; - if (mlx5_vxlan_allowed(mdev->vxlan) || MLX5_CAP_ETH(mdev, tunnel_stateless_gre)) { + if (mlx5_vxlan_allowed(mdev->vxlan) || mlx5_geneve_tx_allowed(mdev) || + MLX5_CAP_ETH(mdev, tunnel_stateless_gre)) { netdev->hw_enc_features |= NETIF_F_IP_CSUM; netdev->hw_enc_features |= NETIF_F_IPV6_CSUM; netdev->hw_enc_features |= NETIF_F_TSO; @@ -4694,7 +4640,7 @@ static void mlx5e_build_nic_netdev(struct net_device *netdev) netdev->hw_enc_features |= NETIF_F_GSO_PARTIAL; } - if (mlx5_vxlan_allowed(mdev->vxlan)) { + if (mlx5_vxlan_allowed(mdev->vxlan) || mlx5_geneve_tx_allowed(mdev)) { netdev->hw_features |= NETIF_F_GSO_UDP_TUNNEL | NETIF_F_GSO_UDP_TUNNEL_CSUM; netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL | @@ -4913,7 +4859,6 @@ static void mlx5e_nic_enable(struct mlx5e_priv *priv) { struct net_device *netdev = priv->netdev; struct mlx5_core_dev *mdev = priv->mdev; - u16 max_mtu; mlx5e_init_l2_addr(priv); @@ -4921,10 +4866,7 @@ static void mlx5e_nic_enable(struct mlx5e_priv *priv) if (!netif_running(netdev)) mlx5_set_port_admin_status(mdev, MLX5_PORT_DOWN); - /* MTU range: 68 - hw-specific max */ - netdev->min_mtu = ETH_MIN_MTU; - mlx5_query_port_max_mtu(priv->mdev, &max_mtu, 1); - netdev->max_mtu = MLX5E_HW2SW_MTU(&priv->channels.params, max_mtu); + mlx5e_set_netdev_mtu_boundaries(priv); mlx5e_set_dev_port_mtu(priv); mlx5_lag_add(mdev, netdev); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c index a66b6ed80b302f2c236e303a233812ddd678401f..91e24f1cead885e73e5b1cfbd31fda7ebf809205 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c @@ -795,7 +795,8 @@ static int mlx5e_nic_rep_netdevice_event(struct notifier_block *nb, struct mlx5e_priv *priv = netdev_priv(rpriv->netdev); struct net_device *netdev = netdev_notifier_info_to_dev(ptr); - if (!mlx5e_tc_tun_device_to_offload(priv, netdev)) + if (!mlx5e_tc_tun_device_to_offload(priv, netdev) && + !is_vlan_dev(netdev)) return NOTIFY_OK; switch (event) { @@ -1374,6 +1375,7 @@ static void mlx5e_build_rep_params(struct net_device *netdev) mlx5e_set_rx_cq_mode_params(params, cq_period_mode); params->num_tc = 1; + params->tunneled_offload_en = false; mlx5_query_min_inline(mdev, ¶ms->tx_min_inline_mode); @@ -1389,7 +1391,7 @@ static void mlx5e_build_rep_netdev(struct net_device *netdev) struct mlx5_core_dev *mdev = priv->mdev; if (rep->vport == MLX5_VPORT_UPLINK) { - SET_NETDEV_DEV(netdev, &priv->mdev->pdev->dev); + SET_NETDEV_DEV(netdev, mdev->device); netdev->netdev_ops = &mlx5e_netdev_ops_uplink_rep; /* we want a persistent mac for the uplink rep */ mlx5_query_nic_vport_mac_address(mdev, 0, netdev->dev_addr); @@ -1623,13 +1625,7 @@ static void mlx5e_cleanup_rep_tx(struct mlx5e_priv *priv) static void mlx5e_vf_rep_enable(struct mlx5e_priv *priv) { - struct net_device *netdev = priv->netdev; - struct mlx5_core_dev *mdev = priv->mdev; - u16 max_mtu; - - netdev->min_mtu = ETH_MIN_MTU; - mlx5_query_port_max_mtu(mdev, &max_mtu, 1); - netdev->max_mtu = MLX5E_HW2SW_MTU(&priv->channels.params, max_mtu); + mlx5e_set_netdev_mtu_boundaries(priv); } static int uplink_rep_async_event(struct notifier_block *nb, unsigned long event, void *data) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c index c3b3002ff62f073f8c9fff88ea2fb74693474619..13133e7f088ed74288e9dcc54205d21b3c3c8c00 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c @@ -409,14 +409,15 @@ mlx5e_free_rx_mpwqe(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi, bool recycle mlx5e_page_release(rq, &dma_info[i], recycle); } -static void mlx5e_post_rx_mpwqe(struct mlx5e_rq *rq) +static void mlx5e_post_rx_mpwqe(struct mlx5e_rq *rq, u8 n) { struct mlx5_wq_ll *wq = &rq->mpwqe.wq; - struct mlx5e_rx_wqe_ll *wqe = mlx5_wq_ll_get_wqe(wq, wq->head); - rq->mpwqe.umr_in_progress = false; + do { + u16 next_wqe_index = mlx5_wq_ll_get_wqe_next_ix(wq, wq->head); - mlx5_wq_ll_push(wq, be16_to_cpu(wqe->next.next_wqe_index)); + mlx5_wq_ll_push(wq, next_wqe_index); + } while (--n); /* ensure wqes are visible to device before updating doorbell record */ dma_wmb(); @@ -426,7 +427,7 @@ static void mlx5e_post_rx_mpwqe(struct mlx5e_rq *rq) static inline u16 mlx5e_icosq_wrap_cnt(struct mlx5e_icosq *sq) { - return sq->pc >> MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE; + return mlx5_wq_cyc_get_ctr_wrap_cnt(&sq->wq, sq->pc); } static inline void mlx5e_fill_icosq_frag_edge(struct mlx5e_icosq *sq, @@ -478,8 +479,6 @@ static int mlx5e_alloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix) bitmap_zero(wi->xdp_xmit_bitmap, MLX5_MPWRQ_PAGES_PER_WQE); wi->consumed_strides = 0; - rq->mpwqe.umr_in_progress = true; - umr_wqe->ctrl.opmod_idx_opcode = cpu_to_be32((sq->pc << MLX5_WQE_CTRL_WQE_INDEX_SHIFT) | MLX5_OPCODE_UMR); @@ -487,7 +486,8 @@ static int mlx5e_alloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix) sq->db.ico_wqe[pi].opcode = MLX5_OPCODE_UMR; sq->pc += MLX5E_UMR_WQEBBS; - mlx5e_notify_hw(wq, sq->pc, sq->uar_map, &umr_wqe->ctrl); + + sq->doorbell_cseg = &umr_wqe->ctrl; return 0; @@ -542,37 +542,13 @@ bool mlx5e_post_rx_wqes(struct mlx5e_rq *rq) return !!err; } -static inline void mlx5e_poll_ico_single_cqe(struct mlx5e_cq *cq, - struct mlx5e_icosq *sq, - struct mlx5e_rq *rq, - struct mlx5_cqe64 *cqe) -{ - struct mlx5_wq_cyc *wq = &sq->wq; - u16 ci = mlx5_wq_cyc_ctr2ix(wq, be16_to_cpu(cqe->wqe_counter)); - struct mlx5e_sq_wqe_info *icowi = &sq->db.ico_wqe[ci]; - - mlx5_cqwq_pop(&cq->wq); - - if (unlikely(get_cqe_opcode(cqe) != MLX5_CQE_REQ)) { - netdev_WARN_ONCE(cq->channel->netdev, - "Bad OP in ICOSQ CQE: 0x%x\n", get_cqe_opcode(cqe)); - return; - } - - if (likely(icowi->opcode == MLX5_OPCODE_UMR)) { - mlx5e_post_rx_mpwqe(rq); - return; - } - - if (unlikely(icowi->opcode != MLX5_OPCODE_NOP)) - netdev_WARN_ONCE(cq->channel->netdev, - "Bad OPCODE in ICOSQ WQE info: 0x%x\n", icowi->opcode); -} - static void mlx5e_poll_ico_cq(struct mlx5e_cq *cq, struct mlx5e_rq *rq) { struct mlx5e_icosq *sq = container_of(cq, struct mlx5e_icosq, cq); struct mlx5_cqe64 *cqe; + u8 completed_umr = 0; + u16 sqcc; + int i; if (unlikely(!test_bit(MLX5E_SQ_STATE_ENABLED, &sq->state))) return; @@ -581,28 +557,96 @@ static void mlx5e_poll_ico_cq(struct mlx5e_cq *cq, struct mlx5e_rq *rq) if (likely(!cqe)) return; - /* by design, there's only a single cqe */ - mlx5e_poll_ico_single_cqe(cq, sq, rq, cqe); + /* sq->cc must be updated only after mlx5_cqwq_update_db_record(), + * otherwise a cq overrun may occur + */ + sqcc = sq->cc; + + i = 0; + do { + u16 wqe_counter; + bool last_wqe; + + mlx5_cqwq_pop(&cq->wq); + + wqe_counter = be16_to_cpu(cqe->wqe_counter); + + if (unlikely(get_cqe_opcode(cqe) != MLX5_CQE_REQ)) { + netdev_WARN_ONCE(cq->channel->netdev, + "Bad OP in ICOSQ CQE: 0x%x\n", get_cqe_opcode(cqe)); + break; + } + do { + struct mlx5e_sq_wqe_info *wi; + u16 ci; + + last_wqe = (sqcc == wqe_counter); + + ci = mlx5_wq_cyc_ctr2ix(&sq->wq, sqcc); + wi = &sq->db.ico_wqe[ci]; + + if (likely(wi->opcode == MLX5_OPCODE_UMR)) { + sqcc += MLX5E_UMR_WQEBBS; + completed_umr++; + } else if (likely(wi->opcode == MLX5_OPCODE_NOP)) { + sqcc++; + } else { + netdev_WARN_ONCE(cq->channel->netdev, + "Bad OPCODE in ICOSQ WQE info: 0x%x\n", + wi->opcode); + } + + } while (!last_wqe); + + } while ((++i < MLX5E_TX_CQ_POLL_BUDGET) && (cqe = mlx5_cqwq_get_cqe(&cq->wq))); + + sq->cc = sqcc; mlx5_cqwq_update_db_record(&cq->wq); + + if (likely(completed_umr)) { + mlx5e_post_rx_mpwqe(rq, completed_umr); + rq->mpwqe.umr_in_progress -= completed_umr; + } } bool mlx5e_post_rx_mpwqes(struct mlx5e_rq *rq) { + struct mlx5e_icosq *sq = &rq->channel->icosq; struct mlx5_wq_ll *wq = &rq->mpwqe.wq; + u8 missing, i; + u16 head; if (unlikely(!test_bit(MLX5E_RQ_STATE_ENABLED, &rq->state))) return false; - mlx5e_poll_ico_cq(&rq->channel->icosq.cq, rq); + mlx5e_poll_ico_cq(&sq->cq, rq); + + missing = mlx5_wq_ll_missing(wq) - rq->mpwqe.umr_in_progress; - if (mlx5_wq_ll_is_full(wq)) + if (unlikely(rq->mpwqe.umr_in_progress > rq->mpwqe.umr_last_bulk)) + rq->stats->congst_umr++; + +#define UMR_WQE_BULK (2) + if (likely(missing < UMR_WQE_BULK)) return false; - if (!rq->mpwqe.umr_in_progress) - mlx5e_alloc_rx_mpwqe(rq, wq->head); - else - rq->stats->congst_umr += mlx5_wq_ll_missing(wq) > 2; + head = rq->mpwqe.actual_wq_head; + i = missing; + do { + if (unlikely(mlx5e_alloc_rx_mpwqe(rq, head))) + break; + head = mlx5_wq_ll_get_wqe_next_ix(wq, head); + } while (--i); + + rq->mpwqe.umr_last_bulk = missing - i; + if (sq->doorbell_cseg) { + mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, sq->doorbell_cseg); + sq->doorbell_cseg = NULL; + } + + rq->mpwqe.umr_in_progress += rq->mpwqe.umr_last_bulk; + rq->mpwqe.actual_wq_head = head; return false; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c index b75aa8b8bf04eac8cac464c0c8550013154f6267..483d321d2151b61361b41ef9f1c8948ddd869beb 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c @@ -65,6 +65,8 @@ static const struct counter_desc sw_stats_desc[] = { { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xdp_drop) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xdp_redirect) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xdp_tx_xmit) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xdp_tx_mpwqe) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xdp_tx_inlnw) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xdp_tx_full) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xdp_tx_err) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xdp_tx_cqe) }, @@ -79,6 +81,8 @@ static const struct counter_desc sw_stats_desc[] = { { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_queue_wake) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_cqe_err) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_xdp_xmit) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_xdp_mpwqe) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_xdp_inlnw) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_xdp_full) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_xdp_err) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_xdp_cqes) }, @@ -89,7 +93,6 @@ static const struct counter_desc sw_stats_desc[] = { { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_buff_alloc_err) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cqe_compress_blks) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cqe_compress_pkts) }, - { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_page_reuse) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cache_reuse) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cache_full) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cache_empty) }, @@ -160,6 +163,8 @@ static void mlx5e_grp_sw_update_stats(struct mlx5e_priv *priv) s->rx_xdp_drop += rq_stats->xdp_drop; s->rx_xdp_redirect += rq_stats->xdp_redirect; s->rx_xdp_tx_xmit += xdpsq_stats->xmit; + s->rx_xdp_tx_mpwqe += xdpsq_stats->mpwqe; + s->rx_xdp_tx_inlnw += xdpsq_stats->inlnw; s->rx_xdp_tx_full += xdpsq_stats->full; s->rx_xdp_tx_err += xdpsq_stats->err; s->rx_xdp_tx_cqe += xdpsq_stats->cqes; @@ -170,7 +175,6 @@ static void mlx5e_grp_sw_update_stats(struct mlx5e_priv *priv) s->rx_buff_alloc_err += rq_stats->buff_alloc_err; s->rx_cqe_compress_blks += rq_stats->cqe_compress_blks; s->rx_cqe_compress_pkts += rq_stats->cqe_compress_pkts; - s->rx_page_reuse += rq_stats->page_reuse; s->rx_cache_reuse += rq_stats->cache_reuse; s->rx_cache_full += rq_stats->cache_full; s->rx_cache_empty += rq_stats->cache_empty; @@ -185,6 +189,8 @@ static void mlx5e_grp_sw_update_stats(struct mlx5e_priv *priv) s->ch_eq_rearm += ch_stats->eq_rearm; /* xdp redirect */ s->tx_xdp_xmit += xdpsq_red_stats->xmit; + s->tx_xdp_mpwqe += xdpsq_red_stats->mpwqe; + s->tx_xdp_inlnw += xdpsq_red_stats->inlnw; s->tx_xdp_full += xdpsq_red_stats->full; s->tx_xdp_err += xdpsq_red_stats->err; s->tx_xdp_cqes += xdpsq_red_stats->cqes; @@ -1212,7 +1218,6 @@ static const struct counter_desc rq_stats_desc[] = { { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, buff_alloc_err) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cqe_compress_blks) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cqe_compress_pkts) }, - { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, page_reuse) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cache_reuse) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cache_full) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cache_empty) }, @@ -1245,6 +1250,8 @@ static const struct counter_desc sq_stats_desc[] = { static const struct counter_desc rq_xdpsq_stats_desc[] = { { MLX5E_DECLARE_RQ_XDPSQ_STAT(struct mlx5e_xdpsq_stats, xmit) }, + { MLX5E_DECLARE_RQ_XDPSQ_STAT(struct mlx5e_xdpsq_stats, mpwqe) }, + { MLX5E_DECLARE_RQ_XDPSQ_STAT(struct mlx5e_xdpsq_stats, inlnw) }, { MLX5E_DECLARE_RQ_XDPSQ_STAT(struct mlx5e_xdpsq_stats, full) }, { MLX5E_DECLARE_RQ_XDPSQ_STAT(struct mlx5e_xdpsq_stats, err) }, { MLX5E_DECLARE_RQ_XDPSQ_STAT(struct mlx5e_xdpsq_stats, cqes) }, @@ -1252,6 +1259,8 @@ static const struct counter_desc rq_xdpsq_stats_desc[] = { static const struct counter_desc xdpsq_stats_desc[] = { { MLX5E_DECLARE_XDPSQ_STAT(struct mlx5e_xdpsq_stats, xmit) }, + { MLX5E_DECLARE_XDPSQ_STAT(struct mlx5e_xdpsq_stats, mpwqe) }, + { MLX5E_DECLARE_XDPSQ_STAT(struct mlx5e_xdpsq_stats, inlnw) }, { MLX5E_DECLARE_XDPSQ_STAT(struct mlx5e_xdpsq_stats, full) }, { MLX5E_DECLARE_XDPSQ_STAT(struct mlx5e_xdpsq_stats, err) }, { MLX5E_DECLARE_XDPSQ_STAT(struct mlx5e_xdpsq_stats, cqes) }, diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h index 16c3b785f282b109e9b2bc54bd4c136095be9b3f..cdddcc46971bcea2fd6f2f72e2d08ed4ec3fc6d2 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h @@ -77,6 +77,8 @@ struct mlx5e_sw_stats { u64 rx_xdp_drop; u64 rx_xdp_redirect; u64 rx_xdp_tx_xmit; + u64 rx_xdp_tx_mpwqe; + u64 rx_xdp_tx_inlnw; u64 rx_xdp_tx_full; u64 rx_xdp_tx_err; u64 rx_xdp_tx_cqe; @@ -91,6 +93,8 @@ struct mlx5e_sw_stats { u64 tx_queue_wake; u64 tx_cqe_err; u64 tx_xdp_xmit; + u64 tx_xdp_mpwqe; + u64 tx_xdp_inlnw; u64 tx_xdp_full; u64 tx_xdp_err; u64 tx_xdp_cqes; @@ -101,7 +105,6 @@ struct mlx5e_sw_stats { u64 rx_buff_alloc_err; u64 rx_cqe_compress_blks; u64 rx_cqe_compress_pkts; - u64 rx_page_reuse; u64 rx_cache_reuse; u64 rx_cache_full; u64 rx_cache_empty; @@ -201,7 +204,6 @@ struct mlx5e_rq_stats { u64 buff_alloc_err; u64 cqe_compress_blks; u64 cqe_compress_pkts; - u64 page_reuse; u64 cache_reuse; u64 cache_full; u64 cache_empty; @@ -241,6 +243,8 @@ struct mlx5e_sq_stats { struct mlx5e_xdpsq_stats { u64 xmit; + u64 mpwqe; + u64 inlnw; u64 full; u64 err; /* dirtied @completion */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c index d75dc44eb2ff63482820cd4d317af4577fcc1fbd..122f457091a2cb9e6207aeab953dd20c497f85d2 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c @@ -44,6 +44,7 @@ #include #include #include +#include #include "en.h" #include "en_rep.h" #include "en_tc.h" @@ -663,7 +664,8 @@ static int mlx5e_hairpin_flow_add(struct mlx5e_priv *priv, } netdev_dbg(priv->netdev, "add hairpin: tirn %x rqn %x peer %s sqn %x prio %d (log) data %d packets %d\n", - hp->tirn, hp->pair->rqn[0], hp->pair->peer_mdev->priv.name, + hp->tirn, hp->pair->rqn[0], + dev_name(hp->pair->peer_mdev->device), hp->pair->sqn[0], match_prio, params.log_data_size, params.log_num_packets); hpe->hp = hp; @@ -700,7 +702,7 @@ static void mlx5e_hairpin_flow_del(struct mlx5e_priv *priv, hpe = list_entry(next, struct mlx5e_hairpin_entry, flows); netdev_dbg(priv->netdev, "del hairpin: peer %s\n", - hpe->hp->pair->peer_mdev->priv.name); + dev_name(hpe->hp->pair->peer_mdev->device)); mlx5e_hairpin_destroy(hpe->hp); hash_del(&hpe->hairpin_hlist); @@ -1437,6 +1439,26 @@ static int parse_tunnel_attr(struct mlx5e_priv *priv, return 0; } +static void *get_match_headers_criteria(u32 flags, + struct mlx5_flow_spec *spec) +{ + return (flags & MLX5_FLOW_CONTEXT_ACTION_DECAP) ? + MLX5_ADDR_OF(fte_match_param, spec->match_criteria, + inner_headers) : + MLX5_ADDR_OF(fte_match_param, spec->match_criteria, + outer_headers); +} + +static void *get_match_headers_value(u32 flags, + struct mlx5_flow_spec *spec) +{ + return (flags & MLX5_FLOW_CONTEXT_ACTION_DECAP) ? + MLX5_ADDR_OF(fte_match_param, spec->match_value, + inner_headers) : + MLX5_ADDR_OF(fte_match_param, spec->match_value, + outer_headers); +} + static int __parse_cls_flower(struct mlx5e_priv *priv, struct mlx5_flow_spec *spec, struct tc_cls_flower_offload *f, @@ -1502,10 +1524,10 @@ static int __parse_cls_flower(struct mlx5e_priv *priv, /* In decap flow, header pointers should point to the inner * headers, outer header were already set by parse_tunnel_attr */ - headers_c = MLX5_ADDR_OF(fte_match_param, spec->match_criteria, - inner_headers); - headers_v = MLX5_ADDR_OF(fte_match_param, spec->match_value, - inner_headers); + headers_c = get_match_headers_criteria(MLX5_FLOW_CONTEXT_ACTION_DECAP, + spec); + headers_v = get_match_headers_value(MLX5_FLOW_CONTEXT_ACTION_DECAP, + spec); } if (flow_rule_match_key(rule, FLOW_DISSECTOR_KEY_BASIC)) { @@ -1520,11 +1542,23 @@ static int __parse_cls_flower(struct mlx5e_priv *priv, if (match.mask->n_proto) *match_level = MLX5_MATCH_L2; } - - if (flow_rule_match_key(rule, FLOW_DISSECTOR_KEY_VLAN)) { + if (flow_rule_match_key(rule, FLOW_DISSECTOR_KEY_VLAN) || + is_vlan_dev(filter_dev)) { + struct flow_dissector_key_vlan filter_dev_mask; + struct flow_dissector_key_vlan filter_dev_key; struct flow_match_vlan match; - flow_rule_match_vlan(rule, &match); + if (is_vlan_dev(filter_dev)) { + match.key = &filter_dev_key; + match.key->vlan_id = vlan_dev_vlan_id(filter_dev); + match.key->vlan_tpid = vlan_dev_vlan_proto(filter_dev); + match.key->vlan_priority = 0; + match.mask = &filter_dev_mask; + memset(match.mask, 0xff, sizeof(*match.mask)); + match.mask->vlan_priority = 0; + } else { + flow_rule_match_vlan(rule, &match); + } if (match.mask->vlan_id || match.mask->vlan_priority || match.mask->vlan_tpid) { @@ -1827,6 +1861,7 @@ static int parse_cls_flower(struct mlx5e_priv *priv, struct pedit_headers { struct ethhdr eth; + struct vlan_hdr vlan; struct iphdr ip4; struct ipv6hdr ip6; struct tcphdr tcp; @@ -1873,38 +1908,73 @@ struct mlx5_fields { u8 field; u8 size; u32 offset; + u32 match_offset; }; -#define OFFLOAD(fw_field, size, field, off) \ - {MLX5_ACTION_IN_FIELD_OUT_ ## fw_field, size, offsetof(struct pedit_headers, field) + (off)} +#define OFFLOAD(fw_field, size, field, off, match_field) \ + {MLX5_ACTION_IN_FIELD_OUT_ ## fw_field, size, \ + offsetof(struct pedit_headers, field) + (off), \ + MLX5_BYTE_OFF(fte_match_set_lyr_2_4, match_field)} + +static bool cmp_val_mask(void *valp, void *maskp, void *matchvalp, + void *matchmaskp, int size) +{ + bool same = false; + + switch (size) { + case sizeof(u8): + same = ((*(u8 *)valp) & (*(u8 *)maskp)) == + ((*(u8 *)matchvalp) & (*(u8 *)matchmaskp)); + break; + case sizeof(u16): + same = ((*(u16 *)valp) & (*(u16 *)maskp)) == + ((*(u16 *)matchvalp) & (*(u16 *)matchmaskp)); + break; + case sizeof(u32): + same = ((*(u32 *)valp) & (*(u32 *)maskp)) == + ((*(u32 *)matchvalp) & (*(u32 *)matchmaskp)); + break; + } + + return same; +} static struct mlx5_fields fields[] = { - OFFLOAD(DMAC_47_16, 4, eth.h_dest[0], 0), - OFFLOAD(DMAC_15_0, 2, eth.h_dest[4], 0), - OFFLOAD(SMAC_47_16, 4, eth.h_source[0], 0), - OFFLOAD(SMAC_15_0, 2, eth.h_source[4], 0), - OFFLOAD(ETHERTYPE, 2, eth.h_proto, 0), - - OFFLOAD(IP_TTL, 1, ip4.ttl, 0), - OFFLOAD(SIPV4, 4, ip4.saddr, 0), - OFFLOAD(DIPV4, 4, ip4.daddr, 0), - - OFFLOAD(SIPV6_127_96, 4, ip6.saddr.s6_addr32[0], 0), - OFFLOAD(SIPV6_95_64, 4, ip6.saddr.s6_addr32[1], 0), - OFFLOAD(SIPV6_63_32, 4, ip6.saddr.s6_addr32[2], 0), - OFFLOAD(SIPV6_31_0, 4, ip6.saddr.s6_addr32[3], 0), - OFFLOAD(DIPV6_127_96, 4, ip6.daddr.s6_addr32[0], 0), - OFFLOAD(DIPV6_95_64, 4, ip6.daddr.s6_addr32[1], 0), - OFFLOAD(DIPV6_63_32, 4, ip6.daddr.s6_addr32[2], 0), - OFFLOAD(DIPV6_31_0, 4, ip6.daddr.s6_addr32[3], 0), - OFFLOAD(IPV6_HOPLIMIT, 1, ip6.hop_limit, 0), - - OFFLOAD(TCP_SPORT, 2, tcp.source, 0), - OFFLOAD(TCP_DPORT, 2, tcp.dest, 0), - OFFLOAD(TCP_FLAGS, 1, tcp.ack_seq, 5), - - OFFLOAD(UDP_SPORT, 2, udp.source, 0), - OFFLOAD(UDP_DPORT, 2, udp.dest, 0), + OFFLOAD(DMAC_47_16, 4, eth.h_dest[0], 0, dmac_47_16), + OFFLOAD(DMAC_15_0, 2, eth.h_dest[4], 0, dmac_15_0), + OFFLOAD(SMAC_47_16, 4, eth.h_source[0], 0, smac_47_16), + OFFLOAD(SMAC_15_0, 2, eth.h_source[4], 0, smac_15_0), + OFFLOAD(ETHERTYPE, 2, eth.h_proto, 0, ethertype), + OFFLOAD(FIRST_VID, 2, vlan.h_vlan_TCI, 0, first_vid), + + OFFLOAD(IP_TTL, 1, ip4.ttl, 0, ttl_hoplimit), + OFFLOAD(SIPV4, 4, ip4.saddr, 0, src_ipv4_src_ipv6.ipv4_layout.ipv4), + OFFLOAD(DIPV4, 4, ip4.daddr, 0, dst_ipv4_dst_ipv6.ipv4_layout.ipv4), + + OFFLOAD(SIPV6_127_96, 4, ip6.saddr.s6_addr32[0], 0, + src_ipv4_src_ipv6.ipv6_layout.ipv6[0]), + OFFLOAD(SIPV6_95_64, 4, ip6.saddr.s6_addr32[1], 0, + src_ipv4_src_ipv6.ipv6_layout.ipv6[4]), + OFFLOAD(SIPV6_63_32, 4, ip6.saddr.s6_addr32[2], 0, + src_ipv4_src_ipv6.ipv6_layout.ipv6[8]), + OFFLOAD(SIPV6_31_0, 4, ip6.saddr.s6_addr32[3], 0, + src_ipv4_src_ipv6.ipv6_layout.ipv6[12]), + OFFLOAD(DIPV6_127_96, 4, ip6.daddr.s6_addr32[0], 0, + dst_ipv4_dst_ipv6.ipv6_layout.ipv6[0]), + OFFLOAD(DIPV6_95_64, 4, ip6.daddr.s6_addr32[1], 0, + dst_ipv4_dst_ipv6.ipv6_layout.ipv6[4]), + OFFLOAD(DIPV6_63_32, 4, ip6.daddr.s6_addr32[2], 0, + dst_ipv4_dst_ipv6.ipv6_layout.ipv6[8]), + OFFLOAD(DIPV6_31_0, 4, ip6.daddr.s6_addr32[3], 0, + dst_ipv4_dst_ipv6.ipv6_layout.ipv6[12]), + OFFLOAD(IPV6_HOPLIMIT, 1, ip6.hop_limit, 0, ttl_hoplimit), + + OFFLOAD(TCP_SPORT, 2, tcp.source, 0, tcp_sport), + OFFLOAD(TCP_DPORT, 2, tcp.dest, 0, tcp_dport), + OFFLOAD(TCP_FLAGS, 1, tcp.ack_seq, 5, tcp_flags), + + OFFLOAD(UDP_SPORT, 2, udp.source, 0, udp_sport), + OFFLOAD(UDP_DPORT, 2, udp.dest, 0, udp_dport), }; /* On input attr->max_mod_hdr_actions tells how many HW actions can be parsed at @@ -1913,9 +1983,14 @@ static struct mlx5_fields fields[] = { */ static int offload_pedit_fields(struct pedit_headers_action *hdrs, struct mlx5e_tc_flow_parse_attr *parse_attr, + u32 *action_flags, struct netlink_ext_ack *extack) { struct pedit_headers *set_masks, *add_masks, *set_vals, *add_vals; + void *headers_c = get_match_headers_criteria(*action_flags, + &parse_attr->spec); + void *headers_v = get_match_headers_value(*action_flags, + &parse_attr->spec); int i, action_size, nactions, max_actions, first, last, next_z; void *s_masks_p, *a_masks_p, *vals_p; struct mlx5_fields *f; @@ -1939,6 +2014,8 @@ static int offload_pedit_fields(struct pedit_headers_action *hdrs, nactions = parse_attr->num_mod_hdr_actions; for (i = 0; i < ARRAY_SIZE(fields); i++) { + bool skip; + f = &fields[i]; /* avoid seeing bits set from previous iterations */ s_mask = 0; @@ -1967,19 +2044,34 @@ static int offload_pedit_fields(struct pedit_headers_action *hdrs, return -EOPNOTSUPP; } + skip = false; if (s_mask) { + void *match_mask = headers_c + f->match_offset; + void *match_val = headers_v + f->match_offset; + cmd = MLX5_ACTION_TYPE_SET; mask = s_mask; vals_p = (void *)set_vals + f->offset; + /* don't rewrite if we have a match on the same value */ + if (cmp_val_mask(vals_p, s_masks_p, match_val, + match_mask, f->size)) + skip = true; /* clear to denote we consumed this field */ memset(s_masks_p, 0, f->size); } else { + u32 zero = 0; + cmd = MLX5_ACTION_TYPE_ADD; mask = a_mask; vals_p = (void *)add_vals + f->offset; + /* add 0 is no change */ + if (!memcmp(vals_p, &zero, f->size)) + skip = true; /* clear to denote we consumed this field */ memset(a_masks_p, 0, f->size); } + if (skip) + continue; field_bsize = f->size * BITS_PER_BYTE; @@ -2026,6 +2118,15 @@ static int offload_pedit_fields(struct pedit_headers_action *hdrs, return 0; } +static int mlx5e_flow_namespace_max_modify_action(struct mlx5_core_dev *mdev, + int namespace) +{ + if (namespace == MLX5_FLOW_NAMESPACE_FDB) /* FDB offloading */ + return MLX5_CAP_ESW_FLOWTABLE_FDB(mdev, max_modify_header_actions); + else /* namespace is MLX5_FLOW_NAMESPACE_KERNEL - NIC offloading */ + return MLX5_CAP_FLOWTABLE_NIC_RX(mdev, max_modify_header_actions); +} + static int alloc_mod_hdr_actions(struct mlx5e_priv *priv, struct pedit_headers_action *hdrs, int namespace, @@ -2037,11 +2138,7 @@ static int alloc_mod_hdr_actions(struct mlx5e_priv *priv, hdrs[TCA_PEDIT_KEY_EX_CMD_ADD].pedits; action_size = MLX5_UN_SZ_BYTES(set_action_in_add_action_in_auto); - if (namespace == MLX5_FLOW_NAMESPACE_FDB) /* FDB offloading */ - max_actions = MLX5_CAP_ESW_FLOWTABLE_FDB(priv->mdev, max_modify_header_actions); - else /* namespace is MLX5_FLOW_NAMESPACE_KERNEL - NIC offloading */ - max_actions = MLX5_CAP_FLOWTABLE_NIC_RX(priv->mdev, max_modify_header_actions); - + max_actions = mlx5e_flow_namespace_max_modify_action(priv->mdev, namespace); /* can get up to crazingly 16 HW actions in 32 bits pedit SW key */ max_actions = min(max_actions, nkeys * 16); @@ -2074,6 +2171,12 @@ static int parse_tc_pedit_action(struct mlx5e_priv *priv, goto out_err; } + if (!mlx5e_flow_namespace_max_modify_action(priv->mdev, namespace)) { + NL_SET_ERR_MSG_MOD(extack, + "The pedit offload action is not supported"); + goto out_err; + } + mask = act->mangle.mask; val = act->mangle.val; offset = act->mangle.offset; @@ -2092,6 +2195,7 @@ out_err: static int alloc_tc_pedit_action(struct mlx5e_priv *priv, int namespace, struct mlx5e_tc_flow_parse_attr *parse_attr, struct pedit_headers_action *hdrs, + u32 *action_flags, struct netlink_ext_ack *extack) { struct pedit_headers *cmd_masks; @@ -2104,7 +2208,7 @@ static int alloc_tc_pedit_action(struct mlx5e_priv *priv, int namespace, goto out_err; } - err = offload_pedit_fields(hdrs, parse_attr, extack); + err = offload_pedit_fields(hdrs, parse_attr, action_flags, extack); if (err < 0) goto out_dealloc_parsed_actions; @@ -2216,11 +2320,7 @@ static bool modify_header_match_supported(struct mlx5_flow_spec *spec, u8 ip_proto; int i; - if (actions & MLX5_FLOW_CONTEXT_ACTION_DECAP) - headers_v = MLX5_ADDR_OF(fte_match_param, spec->match_value, inner_headers); - else - headers_v = MLX5_ADDR_OF(fte_match_param, spec->match_value, outer_headers); - + headers_v = get_match_headers_value(actions, spec); ethertype = MLX5_GET(fte_match_set_lyr_2_4, headers_v, ethertype); /* for non-IP we only re-write MACs, so we're okay */ @@ -2266,7 +2366,8 @@ static bool actions_match_supported(struct mlx5e_priv *priv, actions = flow->nic_attr->action; if (flow->flags & MLX5E_TC_FLOW_EGRESS && - !(actions & MLX5_FLOW_CONTEXT_ACTION_DECAP)) + !((actions & MLX5_FLOW_CONTEXT_ACTION_DECAP) || + (actions & MLX5_FLOW_CONTEXT_ACTION_VLAN_POP))) return false; if (actions & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR) @@ -2291,6 +2392,74 @@ static bool same_hw_devs(struct mlx5e_priv *priv, struct mlx5e_priv *peer_priv) return (fsystem_guid == psystem_guid); } +static int add_vlan_rewrite_action(struct mlx5e_priv *priv, int namespace, + const struct flow_action_entry *act, + struct mlx5e_tc_flow_parse_attr *parse_attr, + struct pedit_headers_action *hdrs, + u32 *action, struct netlink_ext_ack *extack) +{ + u16 mask16 = VLAN_VID_MASK; + u16 val16 = act->vlan.vid & VLAN_VID_MASK; + const struct flow_action_entry pedit_act = { + .id = FLOW_ACTION_MANGLE, + .mangle.htype = FLOW_ACT_MANGLE_HDR_TYPE_ETH, + .mangle.offset = offsetof(struct vlan_ethhdr, h_vlan_TCI), + .mangle.mask = ~(u32)be16_to_cpu(*(__be16 *)&mask16), + .mangle.val = (u32)be16_to_cpu(*(__be16 *)&val16), + }; + u8 match_prio_mask, match_prio_val; + void *headers_c, *headers_v; + int err; + + headers_c = get_match_headers_criteria(*action, &parse_attr->spec); + headers_v = get_match_headers_value(*action, &parse_attr->spec); + + if (!(MLX5_GET(fte_match_set_lyr_2_4, headers_c, cvlan_tag) && + MLX5_GET(fte_match_set_lyr_2_4, headers_v, cvlan_tag))) { + NL_SET_ERR_MSG_MOD(extack, + "VLAN rewrite action must have VLAN protocol match"); + return -EOPNOTSUPP; + } + + match_prio_mask = MLX5_GET(fte_match_set_lyr_2_4, headers_c, first_prio); + match_prio_val = MLX5_GET(fte_match_set_lyr_2_4, headers_v, first_prio); + if (act->vlan.prio != (match_prio_val & match_prio_mask)) { + NL_SET_ERR_MSG_MOD(extack, + "Changing VLAN prio is not supported"); + return -EOPNOTSUPP; + } + + err = parse_tc_pedit_action(priv, &pedit_act, namespace, parse_attr, + hdrs, NULL); + *action |= MLX5_FLOW_CONTEXT_ACTION_MOD_HDR; + + return err; +} + +static int +add_vlan_prio_tag_rewrite_action(struct mlx5e_priv *priv, + struct mlx5e_tc_flow_parse_attr *parse_attr, + struct pedit_headers_action *hdrs, + u32 *action, struct netlink_ext_ack *extack) +{ + const struct flow_action_entry prio_tag_act = { + .vlan.vid = 0, + .vlan.prio = + MLX5_GET(fte_match_set_lyr_2_4, + get_match_headers_value(*action, + &parse_attr->spec), + first_prio) & + MLX5_GET(fte_match_set_lyr_2_4, + get_match_headers_criteria(*action, + &parse_attr->spec), + first_prio), + }; + + return add_vlan_rewrite_action(priv, MLX5_FLOW_NAMESPACE_FDB, + &prio_tag_act, parse_attr, hdrs, action, + extack); +} + static int parse_tc_nic_actions(struct mlx5e_priv *priv, struct flow_action *flow_action, struct mlx5e_tc_flow_parse_attr *parse_attr, @@ -2326,6 +2495,15 @@ static int parse_tc_nic_actions(struct mlx5e_priv *priv, action |= MLX5_FLOW_CONTEXT_ACTION_MOD_HDR | MLX5_FLOW_CONTEXT_ACTION_FWD_DEST; break; + case FLOW_ACTION_VLAN_MANGLE: + err = add_vlan_rewrite_action(priv, + MLX5_FLOW_NAMESPACE_KERNEL, + act, parse_attr, hdrs, + &action, extack); + if (err) + return err; + + break; case FLOW_ACTION_CSUM: if (csum_offload_supported(priv, action, act->csum_flags, @@ -2365,16 +2543,22 @@ static int parse_tc_nic_actions(struct mlx5e_priv *priv, } break; default: - return -EINVAL; + NL_SET_ERR_MSG_MOD(extack, "The offload action is not supported"); + return -EOPNOTSUPP; } } if (hdrs[TCA_PEDIT_KEY_EX_CMD_SET].pedits || hdrs[TCA_PEDIT_KEY_EX_CMD_ADD].pedits) { err = alloc_tc_pedit_action(priv, MLX5_FLOW_NAMESPACE_KERNEL, - parse_attr, hdrs, extack); + parse_attr, hdrs, &action, extack); if (err) return err; + /* in case all pedit actions are skipped, remove the MOD_HDR + * flag. + */ + if (parse_attr->num_mod_hdr_actions == 0) + action &= ~MLX5_FLOW_CONTEXT_ACTION_MOD_HDR; } attr->action = action; @@ -2544,8 +2728,7 @@ static int parse_tc_vlan_action(struct mlx5e_priv *priv, } break; default: - /* action is FLOW_ACT_VLAN_MANGLE */ - return -EOPNOTSUPP; + return -EINVAL; } attr->total_vlan = vlan_idx + 1; @@ -2553,15 +2736,60 @@ static int parse_tc_vlan_action(struct mlx5e_priv *priv, return 0; } +static int add_vlan_push_action(struct mlx5e_priv *priv, + struct mlx5_esw_flow_attr *attr, + struct net_device **out_dev, + u32 *action) +{ + struct net_device *vlan_dev = *out_dev; + struct flow_action_entry vlan_act = { + .id = FLOW_ACTION_VLAN_PUSH, + .vlan.vid = vlan_dev_vlan_id(vlan_dev), + .vlan.proto = vlan_dev_vlan_proto(vlan_dev), + .vlan.prio = 0, + }; + int err; + + err = parse_tc_vlan_action(priv, &vlan_act, attr, action); + if (err) + return err; + + *out_dev = dev_get_by_index_rcu(dev_net(vlan_dev), + dev_get_iflink(vlan_dev)); + if (is_vlan_dev(*out_dev)) + err = add_vlan_push_action(priv, attr, out_dev, action); + + return err; +} + +static int add_vlan_pop_action(struct mlx5e_priv *priv, + struct mlx5_esw_flow_attr *attr, + u32 *action) +{ + int nest_level = vlan_get_encap_level(attr->parse_attr->filter_dev); + struct flow_action_entry vlan_act = { + .id = FLOW_ACTION_VLAN_POP, + }; + int err = 0; + + while (nest_level--) { + err = parse_tc_vlan_action(priv, &vlan_act, attr, action); + if (err) + return err; + } + + return err; +} + static int parse_tc_fdb_actions(struct mlx5e_priv *priv, struct flow_action *flow_action, - struct mlx5e_tc_flow_parse_attr *parse_attr, struct mlx5e_tc_flow *flow, struct netlink_ext_ack *extack) { struct pedit_headers_action hdrs[2] = {}; struct mlx5_eswitch *esw = priv->mdev->priv.eswitch; struct mlx5_esw_flow_attr *attr = flow->esw_attr; + struct mlx5e_tc_flow_parse_attr *parse_attr = attr->parse_attr; struct mlx5e_rep_priv *rpriv = priv->ppriv; const struct ip_tunnel_info *info = NULL; const struct flow_action_entry *act; @@ -2633,6 +2861,20 @@ static int parse_tc_fdb_actions(struct mlx5e_priv *priv, uplink_upper == out_dev) out_dev = uplink_dev; + if (is_vlan_dev(out_dev)) { + err = add_vlan_push_action(priv, attr, + &out_dev, + &action); + if (err) + return err; + } + if (is_vlan_dev(parse_attr->filter_dev)) { + err = add_vlan_pop_action(priv, attr, + &action); + if (err) + return err; + } + if (!mlx5e_eswitch_rep(out_dev)) return -EOPNOTSUPP; @@ -2646,7 +2888,6 @@ static int parse_tc_fdb_actions(struct mlx5e_priv *priv, out_dev->ifindex; parse_attr->tun_info[attr->out_count] = *info; encap = false; - attr->parse_attr = parse_attr; attr->dests[attr->out_count].flags |= MLX5_ESW_DEST_ENCAP; attr->out_count++; @@ -2679,7 +2920,27 @@ static int parse_tc_fdb_actions(struct mlx5e_priv *priv, break; case FLOW_ACTION_VLAN_PUSH: case FLOW_ACTION_VLAN_POP: - err = parse_tc_vlan_action(priv, act, attr, &action); + if (act->id == FLOW_ACTION_VLAN_PUSH && + (action & MLX5_FLOW_CONTEXT_ACTION_VLAN_POP)) { + /* Replace vlan pop+push with vlan modify */ + action &= ~MLX5_FLOW_CONTEXT_ACTION_VLAN_POP; + err = add_vlan_rewrite_action(priv, + MLX5_FLOW_NAMESPACE_FDB, + act, parse_attr, hdrs, + &action, extack); + } else { + err = parse_tc_vlan_action(priv, act, attr, &action); + } + if (err) + return err; + + attr->split_count = attr->out_count; + break; + case FLOW_ACTION_VLAN_MANGLE: + err = add_vlan_rewrite_action(priv, + MLX5_FLOW_NAMESPACE_FDB, + act, parse_attr, hdrs, + &action, extack); if (err) return err; @@ -2705,16 +2966,39 @@ static int parse_tc_fdb_actions(struct mlx5e_priv *priv, break; } default: - return -EINVAL; + NL_SET_ERR_MSG_MOD(extack, "The offload action is not supported"); + return -EOPNOTSUPP; } } + if (MLX5_CAP_GEN(esw->dev, prio_tag_required) && + action & MLX5_FLOW_CONTEXT_ACTION_VLAN_POP) { + /* For prio tag mode, replace vlan pop with rewrite vlan prio + * tag rewrite. + */ + action &= ~MLX5_FLOW_CONTEXT_ACTION_VLAN_POP; + err = add_vlan_prio_tag_rewrite_action(priv, parse_attr, hdrs, + &action, extack); + if (err) + return err; + } + if (hdrs[TCA_PEDIT_KEY_EX_CMD_SET].pedits || hdrs[TCA_PEDIT_KEY_EX_CMD_ADD].pedits) { err = alloc_tc_pedit_action(priv, MLX5_FLOW_NAMESPACE_FDB, - parse_attr, hdrs, extack); + parse_attr, hdrs, &action, extack); if (err) return err; + /* in case all pedit actions are skipped, remove the MOD_HDR + * flag. we might have set split_count either by pedit or + * pop/push. if there is no pop/push either, reset it too. + */ + if (parse_attr->num_mod_hdr_actions == 0) { + action &= ~MLX5_FLOW_CONTEXT_ACTION_MOD_HDR; + if (!((action & MLX5_FLOW_CONTEXT_ACTION_VLAN_POP) || + (action & MLX5_FLOW_CONTEXT_ACTION_VLAN_PUSH))) + attr->split_count = 0; + } } attr->action = action; @@ -2883,7 +3167,7 @@ __mlx5e_add_fdb_flow(struct mlx5e_priv *priv, if (err) goto err_free; - err = parse_tc_fdb_actions(priv, &rule->action, parse_attr, flow, extack); + err = parse_tc_fdb_actions(priv, &rule->action, flow, extack); if (err) goto err_free; @@ -3080,6 +3364,7 @@ int mlx5e_configure_flower(struct net_device *dev, struct mlx5e_priv *priv, netdev_warn_once(priv->netdev, "flow cookie %lx already exists, ignoring\n", f->cookie); + err = -EEXIST; goto out; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c index 25a8f8260c14f619fefb9d3276210ca83c9e0d7e..7b61126fcec9741f999dde262077f59e0f65f7ee 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c @@ -32,6 +32,7 @@ #include #include +#include #include #include "en.h" #include "ipoib/ipoib.h" @@ -110,11 +111,10 @@ static inline int mlx5e_get_dscp_up(struct mlx5e_priv *priv, struct sk_buff *skb #endif u16 mlx5e_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { + int channel_ix = netdev_pick_tx(dev, skb, NULL); struct mlx5e_priv *priv = netdev_priv(dev); - int channel_ix = fallback(dev, skb, NULL); u16 num_channels; int up = 0; @@ -163,7 +163,7 @@ static inline u16 mlx5e_calc_min_inline(enum mlx5_inline_modes mode, case MLX5_INLINE_MODE_NONE: return 0; case MLX5_INLINE_MODE_TCP_UDP: - hlen = eth_get_headlen(skb->data, skb_headlen(skb)); + hlen = eth_get_headlen(skb->dev, skb->data, skb_headlen(skb)); if (hlen == ETH_HLEN && !skb_vlan_tag_present(skb)) hlen += VLAN_HLEN; break; @@ -297,7 +297,8 @@ static inline void mlx5e_fill_sq_frag_edge(struct mlx5e_txqsq *sq, static inline void mlx5e_txwqe_complete(struct mlx5e_txqsq *sq, struct sk_buff *skb, u8 opcode, u16 ds_cnt, u8 num_wqebbs, u32 num_bytes, u8 num_dma, - struct mlx5e_tx_wqe_info *wi, struct mlx5_wqe_ctrl_seg *cseg) + struct mlx5e_tx_wqe_info *wi, struct mlx5_wqe_ctrl_seg *cseg, + bool xmit_more) { struct mlx5_wq_cyc *wq = &sq->wq; @@ -320,14 +321,14 @@ mlx5e_txwqe_complete(struct mlx5e_txqsq *sq, struct sk_buff *skb, sq->stats->stopped++; } - if (!skb->xmit_more || netif_xmit_stopped(sq->txq)) + if (!xmit_more || netif_xmit_stopped(sq->txq)) mlx5e_notify_hw(wq, sq->pc, sq->uar_map, cseg); } #define INL_HDR_START_SZ (sizeof(((struct mlx5_wqe_eth_seg *)NULL)->inline_hdr.start)) netdev_tx_t mlx5e_sq_xmit(struct mlx5e_txqsq *sq, struct sk_buff *skb, - struct mlx5e_tx_wqe *wqe, u16 pi) + struct mlx5e_tx_wqe *wqe, u16 pi, bool xmit_more) { struct mlx5_wq_cyc *wq = &sq->wq; struct mlx5_wqe_ctrl_seg *cseg; @@ -360,7 +361,7 @@ netdev_tx_t mlx5e_sq_xmit(struct mlx5e_txqsq *sq, struct sk_buff *skb, } stats->bytes += num_bytes; - stats->xmit_more += skb->xmit_more; + stats->xmit_more += netdev_xmit_more(); headlen = skb->len - ihs - skb->data_len; ds_cnt += !!headlen; @@ -392,6 +393,10 @@ netdev_tx_t mlx5e_sq_xmit(struct mlx5e_txqsq *sq, struct sk_buff *skb, eseg = &wqe->eth; dseg = wqe->data; +#if IS_ENABLED(CONFIG_GENEVE) + if (skb->encapsulation) + mlx5e_tx_tunnel_accel(skb, eseg); +#endif mlx5e_txwqe_build_eseg_csum(sq, skb, eseg); eseg->mss = mss; @@ -419,7 +424,7 @@ netdev_tx_t mlx5e_sq_xmit(struct mlx5e_txqsq *sq, struct sk_buff *skb, goto err_drop; mlx5e_txwqe_complete(sq, skb, opcode, ds_cnt, num_wqebbs, num_bytes, - num_dma, wi, cseg); + num_dma, wi, cseg, xmit_more); return NETDEV_TX_OK; @@ -445,7 +450,7 @@ netdev_tx_t mlx5e_xmit(struct sk_buff *skb, struct net_device *dev) if (unlikely(!skb)) return NETDEV_TX_OK; - return mlx5e_sq_xmit(sq, skb, wqe, pi); + return mlx5e_sq_xmit(sq, skb, wqe, pi, netdev_xmit_more()); } static void mlx5e_dump_error_cqe(struct mlx5e_txqsq *sq, @@ -655,7 +660,7 @@ netdev_tx_t mlx5i_sq_xmit(struct mlx5e_txqsq *sq, struct sk_buff *skb, } stats->bytes += num_bytes; - stats->xmit_more += skb->xmit_more; + stats->xmit_more += netdev_xmit_more(); headlen = skb->len - ihs - skb->data_len; ds_cnt += !!headlen; @@ -700,7 +705,7 @@ netdev_tx_t mlx5i_sq_xmit(struct mlx5e_txqsq *sq, struct sk_buff *skb, goto err_drop; mlx5e_txwqe_complete(sq, skb, opcode, ds_cnt, num_wqebbs, num_bytes, - num_dma, wi, cseg); + num_dma, wi, cseg, false); return NETDEV_TX_OK; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c index b4af5e19f6acd63fb9b08375a533ec1fc9085820..f9862bf75491bb6da0be989bcd15414ab59ee8c2 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c @@ -71,6 +71,17 @@ static void mlx5e_handle_rx_dim(struct mlx5e_rq *rq) net_dim(&rq->dim, dim_sample); } +void mlx5e_trigger_irq(struct mlx5e_icosq *sq) +{ + struct mlx5_wq_cyc *wq = &sq->wq; + struct mlx5e_tx_wqe *nopwqe; + u16 pi = mlx5_wq_cyc_ctr2ix(wq, sq->pc); + + sq->db.ico_wqe[pi].opcode = MLX5_OPCODE_NOP; + nopwqe = mlx5e_post_nop(wq, sq->sqn, &sq->pc); + mlx5e_notify_hw(wq, sq->pc, sq->uar_map, &nopwqe->ctrl); +} + int mlx5e_napi_poll(struct napi_struct *napi, int budget) { struct mlx5e_channel *c = container_of(napi, struct mlx5e_channel, diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eq.c b/drivers/net/ethernet/mellanox/mlx5/core/eq.c index bb6e5b5d9681013f0e889d49a7567f2b7dae06a8..5aac97847721a9f337c21fd30880fdd53efd85f5 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eq.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/eq.c @@ -504,8 +504,7 @@ static u64 gather_async_events_mask(struct mlx5_core_dev *dev) if (MLX5_VPORT_MANAGER(dev)) async_event_mask |= (1ull << MLX5_EVENT_TYPE_NIC_VPORT_CHANGE); - if (MLX5_CAP_GEN(dev, port_type) == MLX5_CAP_PORT_TYPE_ETH && - MLX5_CAP_GEN(dev, general_notification_event)) + if (MLX5_CAP_GEN(dev, general_notification_event)) async_event_mask |= (1ull << MLX5_EVENT_TYPE_GENERAL_EVENT); if (MLX5_CAP_GEN(dev, port_module_event)) @@ -707,7 +706,7 @@ void mlx5_eq_update_ci(struct mlx5_eq *eq, u32 cc, bool arm) __raw_writel((__force u32)cpu_to_be32(val), addr); /* We still want ordering, just not swabbing, so add a barrier */ - mb(); + wmb(); } EXPORT_SYMBOL(mlx5_eq_update_ci); @@ -900,14 +899,12 @@ mlx5_comp_irq_get_affinity_mask(struct mlx5_core_dev *dev, int vector) } EXPORT_SYMBOL(mlx5_comp_irq_get_affinity_mask); +#ifdef CONFIG_RFS_ACCEL struct cpu_rmap *mlx5_eq_table_get_rmap(struct mlx5_core_dev *dev) { -#ifdef CONFIG_RFS_ACCEL return dev->priv.eq_table->rmap; -#else - return NULL; -#endif } +#endif struct mlx5_eq_comp *mlx5_eqn2comp_eq(struct mlx5_core_dev *dev, int eqn) { diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c index 8a67fd197b7923f67af1872eae280e0f5e3eb663..9ea0ccfe5ef598129ec5f9ad8b4564886f765d5a 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c @@ -72,25 +72,22 @@ static void esw_cleanup_vepa_rules(struct mlx5_eswitch *esw); MC_ADDR_CHANGE | \ PROMISC_CHANGE) -/* The vport getter/iterator are only valid after esw->total_vports - * and vport->vport are initialized in mlx5_eswitch_init. - */ -#define mlx5_esw_for_all_vports(esw, i, vport) \ - for ((i) = MLX5_VPORT_PF; \ - (vport) = &(esw)->vports[i], \ - (i) < (esw)->total_vports; (i)++) +struct mlx5_vport *__must_check +mlx5_eswitch_get_vport(struct mlx5_eswitch *esw, u16 vport_num) +{ + u16 idx; -#define mlx5_esw_for_each_vf_vport(esw, i, vport, nvfs) \ - for ((i) = MLX5_VPORT_FIRST_VF; \ - (vport) = &(esw)->vports[i], \ - (i) <= (nvfs); (i)++) + if (!esw || !MLX5_CAP_GEN(esw->dev, vport_group_manager)) + return ERR_PTR(-EPERM); -static struct mlx5_vport *mlx5_eswitch_get_vport(struct mlx5_eswitch *esw, - u16 vport_num) -{ - u16 idx = mlx5_eswitch_vport_num_to_index(esw, vport_num); + idx = mlx5_eswitch_vport_num_to_index(esw, vport_num); + + if (idx > esw->total_vports - 1) { + esw_debug(esw->dev, "vport out of range: num(0x%x), idx(0x%x)\n", + vport_num, idx); + return ERR_PTR(-EINVAL); + } - WARN_ON(idx > esw->total_vports - 1); return &esw->vports[idx]; } @@ -644,9 +641,8 @@ static int esw_del_mc_addr(struct mlx5_eswitch *esw, struct vport_addr *vaddr) /* Apply vport UC/MC list to HW l2 table and FDB table */ static void esw_apply_vport_addr_list(struct mlx5_eswitch *esw, - u16 vport_num, int list_type) + struct mlx5_vport *vport, int list_type) { - struct mlx5_vport *vport = mlx5_eswitch_get_vport(esw, vport_num); bool is_uc = list_type == MLX5_NVPRT_LIST_TYPE_UC; vport_addr_action vport_addr_add; vport_addr_action vport_addr_del; @@ -679,9 +675,8 @@ static void esw_apply_vport_addr_list(struct mlx5_eswitch *esw, /* Sync vport UC/MC list from vport context */ static void esw_update_vport_addr_list(struct mlx5_eswitch *esw, - u16 vport_num, int list_type) + struct mlx5_vport *vport, int list_type) { - struct mlx5_vport *vport = mlx5_eswitch_get_vport(esw, vport_num); bool is_uc = list_type == MLX5_NVPRT_LIST_TYPE_UC; u8 (*mac_list)[ETH_ALEN]; struct l2addr_node *node; @@ -710,12 +705,12 @@ static void esw_update_vport_addr_list(struct mlx5_eswitch *esw, if (!vport->enabled) goto out; - err = mlx5_query_nic_vport_mac_list(esw->dev, vport_num, list_type, + err = mlx5_query_nic_vport_mac_list(esw->dev, vport->vport, list_type, mac_list, &size); if (err) goto out; esw_debug(esw->dev, "vport[%d] context update %s list size (%d)\n", - vport_num, is_uc ? "UC" : "MC", size); + vport->vport, is_uc ? "UC" : "MC", size); for (i = 0; i < size; i++) { if (is_uc && !is_valid_ether_addr(mac_list[i])) @@ -753,10 +748,10 @@ static void esw_update_vport_addr_list(struct mlx5_eswitch *esw, if (!addr) { esw_warn(esw->dev, "Failed to add MAC(%pM) to vport[%d] DB\n", - mac_list[i], vport_num); + mac_list[i], vport->vport); continue; } - addr->vport = vport_num; + addr->vport = vport->vport; addr->action = MLX5_ACTION_ADD; } out: @@ -766,9 +761,9 @@ out: /* Sync vport UC/MC list from vport context * Must be called after esw_update_vport_addr_list */ -static void esw_update_vport_mc_promisc(struct mlx5_eswitch *esw, u16 vport_num) +static void esw_update_vport_mc_promisc(struct mlx5_eswitch *esw, + struct mlx5_vport *vport) { - struct mlx5_vport *vport = mlx5_eswitch_get_vport(esw, vport_num); struct l2addr_node *node; struct vport_addr *addr; struct hlist_head *hash; @@ -791,20 +786,20 @@ static void esw_update_vport_mc_promisc(struct mlx5_eswitch *esw, u16 vport_num) if (!addr) { esw_warn(esw->dev, "Failed to add allmulti MAC(%pM) to vport[%d] DB\n", - mac, vport_num); + mac, vport->vport); continue; } - addr->vport = vport_num; + addr->vport = vport->vport; addr->action = MLX5_ACTION_ADD; addr->mc_promisc = true; } } /* Apply vport rx mode to HW FDB table */ -static void esw_apply_vport_rx_mode(struct mlx5_eswitch *esw, u16 vport_num, +static void esw_apply_vport_rx_mode(struct mlx5_eswitch *esw, + struct mlx5_vport *vport, bool promisc, bool mc_promisc) { - struct mlx5_vport *vport = mlx5_eswitch_get_vport(esw, vport_num); struct esw_mc_addr *allmulti_addr = &esw->mc_promisc; if (IS_ERR_OR_NULL(vport->allmulti_rule) != mc_promisc) @@ -812,7 +807,7 @@ static void esw_apply_vport_rx_mode(struct mlx5_eswitch *esw, u16 vport_num, if (mc_promisc) { vport->allmulti_rule = - esw_fdb_set_vport_allmulti_rule(esw, vport_num); + esw_fdb_set_vport_allmulti_rule(esw, vport->vport); if (!allmulti_addr->uplink_rule) allmulti_addr->uplink_rule = esw_fdb_set_vport_allmulti_rule(esw, @@ -835,8 +830,8 @@ promisc: return; if (promisc) { - vport->promisc_rule = esw_fdb_set_vport_promisc_rule(esw, - vport_num); + vport->promisc_rule = + esw_fdb_set_vport_promisc_rule(esw, vport->vport); } else if (vport->promisc_rule) { mlx5_del_flow_rules(vport->promisc_rule); vport->promisc_rule = NULL; @@ -844,23 +839,23 @@ promisc: } /* Sync vport rx mode from vport context */ -static void esw_update_vport_rx_mode(struct mlx5_eswitch *esw, u16 vport_num) +static void esw_update_vport_rx_mode(struct mlx5_eswitch *esw, + struct mlx5_vport *vport) { - struct mlx5_vport *vport = mlx5_eswitch_get_vport(esw, vport_num); int promisc_all = 0; int promisc_uc = 0; int promisc_mc = 0; int err; err = mlx5_query_nic_vport_promisc(esw->dev, - vport_num, + vport->vport, &promisc_uc, &promisc_mc, &promisc_all); if (err) return; esw_debug(esw->dev, "vport[%d] context update rx mode promisc_all=%d, all_multi=%d\n", - vport_num, promisc_all, promisc_mc); + vport->vport, promisc_all, promisc_mc); if (!vport->info.trusted || !vport->enabled) { promisc_uc = 0; @@ -868,7 +863,7 @@ static void esw_update_vport_rx_mode(struct mlx5_eswitch *esw, u16 vport_num) promisc_all = 0; } - esw_apply_vport_rx_mode(esw, vport_num, promisc_all, + esw_apply_vport_rx_mode(esw, vport, promisc_all, (promisc_all || promisc_mc)); } @@ -883,27 +878,21 @@ static void esw_vport_change_handle_locked(struct mlx5_vport *vport) vport->vport, mac); if (vport->enabled_events & UC_ADDR_CHANGE) { - esw_update_vport_addr_list(esw, vport->vport, - MLX5_NVPRT_LIST_TYPE_UC); - esw_apply_vport_addr_list(esw, vport->vport, - MLX5_NVPRT_LIST_TYPE_UC); + esw_update_vport_addr_list(esw, vport, MLX5_NVPRT_LIST_TYPE_UC); + esw_apply_vport_addr_list(esw, vport, MLX5_NVPRT_LIST_TYPE_UC); } - if (vport->enabled_events & MC_ADDR_CHANGE) { - esw_update_vport_addr_list(esw, vport->vport, - MLX5_NVPRT_LIST_TYPE_MC); - } + if (vport->enabled_events & MC_ADDR_CHANGE) + esw_update_vport_addr_list(esw, vport, MLX5_NVPRT_LIST_TYPE_MC); if (vport->enabled_events & PROMISC_CHANGE) { - esw_update_vport_rx_mode(esw, vport->vport); + esw_update_vport_rx_mode(esw, vport); if (!IS_ERR_OR_NULL(vport->allmulti_rule)) - esw_update_vport_mc_promisc(esw, vport->vport); + esw_update_vport_mc_promisc(esw, vport); } - if (vport->enabled_events & (PROMISC_CHANGE | MC_ADDR_CHANGE)) { - esw_apply_vport_addr_list(esw, vport->vport, - MLX5_NVPRT_LIST_TYPE_MC); - } + if (vport->enabled_events & (PROMISC_CHANGE | MC_ADDR_CHANGE)) + esw_apply_vport_addr_list(esw, vport, MLX5_NVPRT_LIST_TYPE_MC); esw_debug(esw->dev, "vport[%d] Context Changed: Done\n", vport->vport); if (vport->enabled) @@ -922,8 +911,8 @@ static void esw_vport_change_handler(struct work_struct *work) mutex_unlock(&esw->state_lock); } -static int esw_vport_enable_egress_acl(struct mlx5_eswitch *esw, - struct mlx5_vport *vport) +int esw_vport_enable_egress_acl(struct mlx5_eswitch *esw, + struct mlx5_vport *vport) { int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in); struct mlx5_flow_group *vlan_grp = NULL; @@ -1006,8 +995,8 @@ out: return err; } -static void esw_vport_cleanup_egress_rules(struct mlx5_eswitch *esw, - struct mlx5_vport *vport) +void esw_vport_cleanup_egress_rules(struct mlx5_eswitch *esw, + struct mlx5_vport *vport) { if (!IS_ERR_OR_NULL(vport->egress.allowed_vlan)) mlx5_del_flow_rules(vport->egress.allowed_vlan); @@ -1019,8 +1008,8 @@ static void esw_vport_cleanup_egress_rules(struct mlx5_eswitch *esw, vport->egress.drop_rule = NULL; } -static void esw_vport_disable_egress_acl(struct mlx5_eswitch *esw, - struct mlx5_vport *vport) +void esw_vport_disable_egress_acl(struct mlx5_eswitch *esw, + struct mlx5_vport *vport) { if (IS_ERR_OR_NULL(vport->egress.acl)) return; @@ -1036,8 +1025,8 @@ static void esw_vport_disable_egress_acl(struct mlx5_eswitch *esw, vport->egress.acl = NULL; } -static int esw_vport_enable_ingress_acl(struct mlx5_eswitch *esw, - struct mlx5_vport *vport) +int esw_vport_enable_ingress_acl(struct mlx5_eswitch *esw, + struct mlx5_vport *vport) { int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in); struct mlx5_core_dev *dev = esw->dev; @@ -1168,8 +1157,8 @@ out: return err; } -static void esw_vport_cleanup_ingress_rules(struct mlx5_eswitch *esw, - struct mlx5_vport *vport) +void esw_vport_cleanup_ingress_rules(struct mlx5_eswitch *esw, + struct mlx5_vport *vport) { if (!IS_ERR_OR_NULL(vport->ingress.drop_rule)) mlx5_del_flow_rules(vport->ingress.drop_rule); @@ -1181,8 +1170,8 @@ static void esw_vport_cleanup_ingress_rules(struct mlx5_eswitch *esw, vport->ingress.allow_rule = NULL; } -static void esw_vport_disable_ingress_acl(struct mlx5_eswitch *esw, - struct mlx5_vport *vport) +void esw_vport_disable_ingress_acl(struct mlx5_eswitch *esw, + struct mlx5_vport *vport) { if (IS_ERR_OR_NULL(vport->ingress.acl)) return; @@ -1420,10 +1409,10 @@ static void esw_destroy_tsar(struct mlx5_eswitch *esw) esw->qos.enabled = false; } -static int esw_vport_enable_qos(struct mlx5_eswitch *esw, int vport_num, +static int esw_vport_enable_qos(struct mlx5_eswitch *esw, + struct mlx5_vport *vport, u32 initial_max_rate, u32 initial_bw_share) { - struct mlx5_vport *vport = mlx5_eswitch_get_vport(esw, vport_num); u32 sched_ctx[MLX5_ST_SZ_DW(scheduling_context)] = {0}; struct mlx5_core_dev *dev = esw->dev; void *vport_elem; @@ -1440,7 +1429,7 @@ static int esw_vport_enable_qos(struct mlx5_eswitch *esw, int vport_num, SCHEDULING_CONTEXT_ELEMENT_TYPE_VPORT); vport_elem = MLX5_ADDR_OF(scheduling_context, sched_ctx, element_attributes); - MLX5_SET(vport_element, vport_elem, vport_number, vport_num); + MLX5_SET(vport_element, vport_elem, vport_number, vport->vport); MLX5_SET(scheduling_context, sched_ctx, parent_element_id, esw->qos.root_tsar_id); MLX5_SET(scheduling_context, sched_ctx, max_average_bw, @@ -1453,7 +1442,7 @@ static int esw_vport_enable_qos(struct mlx5_eswitch *esw, int vport_num, &vport->qos.esw_tsar_ix); if (err) { esw_warn(esw->dev, "E-Switch create TSAR vport element failed (vport=%d,err=%d)\n", - vport_num, err); + vport->vport, err); return err; } @@ -1461,10 +1450,10 @@ static int esw_vport_enable_qos(struct mlx5_eswitch *esw, int vport_num, return 0; } -static void esw_vport_disable_qos(struct mlx5_eswitch *esw, int vport_num) +static void esw_vport_disable_qos(struct mlx5_eswitch *esw, + struct mlx5_vport *vport) { - struct mlx5_vport *vport = mlx5_eswitch_get_vport(esw, vport_num); - int err = 0; + int err; if (!vport->qos.enabled) return; @@ -1474,15 +1463,15 @@ static void esw_vport_disable_qos(struct mlx5_eswitch *esw, int vport_num) vport->qos.esw_tsar_ix); if (err) esw_warn(esw->dev, "E-Switch destroy TSAR vport element failed (vport=%d,err=%d)\n", - vport_num, err); + vport->vport, err); vport->qos.enabled = false; } -static int esw_vport_qos_config(struct mlx5_eswitch *esw, int vport_num, +static int esw_vport_qos_config(struct mlx5_eswitch *esw, + struct mlx5_vport *vport, u32 max_rate, u32 bw_share) { - struct mlx5_vport *vport = mlx5_eswitch_get_vport(esw, vport_num); u32 sched_ctx[MLX5_ST_SZ_DW(scheduling_context)] = {0}; struct mlx5_core_dev *dev = esw->dev; void *vport_elem; @@ -1499,7 +1488,7 @@ static int esw_vport_qos_config(struct mlx5_eswitch *esw, int vport_num, SCHEDULING_CONTEXT_ELEMENT_TYPE_VPORT); vport_elem = MLX5_ADDR_OF(scheduling_context, sched_ctx, element_attributes); - MLX5_SET(vport_element, vport_elem, vport_number, vport_num); + MLX5_SET(vport_element, vport_elem, vport_number, vport->vport); MLX5_SET(scheduling_context, sched_ctx, parent_element_id, esw->qos.root_tsar_id); MLX5_SET(scheduling_context, sched_ctx, max_average_bw, @@ -1515,7 +1504,7 @@ static int esw_vport_qos_config(struct mlx5_eswitch *esw, int vport_num, bitmask); if (err) { esw_warn(esw->dev, "E-Switch modify TSAR vport element failed (vport=%d,err=%d)\n", - vport_num, err); + vport->vport, err); return err; } @@ -1618,7 +1607,7 @@ static void esw_enable_vport(struct mlx5_eswitch *esw, struct mlx5_vport *vport, esw_apply_vport_conf(esw, vport); /* Attach vport to the eswitch rate limiter */ - if (esw_vport_enable_qos(esw, vport_num, vport->info.max_rate, + if (esw_vport_enable_qos(esw, vport, vport->info.max_rate, vport->qos.bw_share)) esw_warn(esw->dev, "Failed to attach vport %d to eswitch rate limiter", vport_num); @@ -1663,7 +1652,7 @@ static void esw_disable_vport(struct mlx5_eswitch *esw, */ esw_vport_change_handle_locked(vport); vport->enabled_events = 0; - esw_vport_disable_qos(esw, vport_num); + esw_vport_disable_qos(esw, vport); if (esw->manager_vport != vport_num && esw->mode == SRIOV_LEGACY) { mlx5_modify_vport_admin_state(esw->dev, @@ -1688,6 +1677,9 @@ static int eswitch_vport_event(struct notifier_block *nb, vport_num = be16_to_cpu(eqe->data.vport_change.vport_num); vport = mlx5_eswitch_get_vport(esw, vport_num); + if (IS_ERR(vport)) + return NOTIFY_OK; + if (vport->enabled) queue_work(esw->work_queue, &vport->vport_change_handler); @@ -1922,22 +1914,19 @@ void mlx5_eswitch_cleanup(struct mlx5_eswitch *esw) } /* Vport Administration */ -#define LEGAL_VPORT(esw, vport) (vport >= 0 && vport < esw->total_vports) - int mlx5_eswitch_set_vport_mac(struct mlx5_eswitch *esw, int vport, u8 mac[ETH_ALEN]) { - struct mlx5_vport *evport; + struct mlx5_vport *evport = mlx5_eswitch_get_vport(esw, vport); u64 node_guid; int err = 0; - if (!esw || !MLX5_CAP_GEN(esw->dev, vport_group_manager)) - return -EPERM; - if (!LEGAL_VPORT(esw, vport) || is_multicast_ether_addr(mac)) + if (IS_ERR(evport)) + return PTR_ERR(evport); + if (is_multicast_ether_addr(mac)) return -EINVAL; mutex_lock(&esw->state_lock); - evport = &esw->vports[vport]; if (evport->info.spoofchk && !is_valid_ether_addr(mac)) mlx5_core_warn(esw->dev, @@ -1972,16 +1961,15 @@ unlock: int mlx5_eswitch_set_vport_state(struct mlx5_eswitch *esw, int vport, int link_state) { - struct mlx5_vport *evport; + struct mlx5_vport *evport = mlx5_eswitch_get_vport(esw, vport); int err = 0; if (!ESW_ALLOWED(esw)) return -EPERM; - if (!LEGAL_VPORT(esw, vport)) - return -EINVAL; + if (IS_ERR(evport)) + return PTR_ERR(evport); mutex_lock(&esw->state_lock); - evport = &esw->vports[vport]; err = mlx5_modify_vport_admin_state(esw->dev, MLX5_VPORT_STATE_OP_MOD_ESW_VPORT, @@ -2003,14 +1991,10 @@ unlock: int mlx5_eswitch_get_vport_config(struct mlx5_eswitch *esw, int vport, struct ifla_vf_info *ivi) { - struct mlx5_vport *evport; - - if (!esw || !MLX5_CAP_GEN(esw->dev, vport_group_manager)) - return -EPERM; - if (!LEGAL_VPORT(esw, vport)) - return -EINVAL; + struct mlx5_vport *evport = mlx5_eswitch_get_vport(esw, vport); - evport = &esw->vports[vport]; + if (IS_ERR(evport)) + return PTR_ERR(evport); memset(ivi, 0, sizeof(*ivi)); ivi->vf = vport - 1; @@ -2032,16 +2016,17 @@ int mlx5_eswitch_get_vport_config(struct mlx5_eswitch *esw, int __mlx5_eswitch_set_vport_vlan(struct mlx5_eswitch *esw, int vport, u16 vlan, u8 qos, u8 set_flags) { - struct mlx5_vport *evport; + struct mlx5_vport *evport = mlx5_eswitch_get_vport(esw, vport); int err = 0; if (!ESW_ALLOWED(esw)) return -EPERM; - if (!LEGAL_VPORT(esw, vport) || (vlan > 4095) || (qos > 7)) + if (IS_ERR(evport)) + return PTR_ERR(evport); + if (vlan > 4095 || qos > 7) return -EINVAL; mutex_lock(&esw->state_lock); - evport = &esw->vports[vport]; err = modify_esw_vport_cvlan(esw->dev, vport, vlan, qos, set_flags); if (err) @@ -2075,17 +2060,16 @@ int mlx5_eswitch_set_vport_vlan(struct mlx5_eswitch *esw, int mlx5_eswitch_set_vport_spoofchk(struct mlx5_eswitch *esw, int vport, bool spoofchk) { - struct mlx5_vport *evport; + struct mlx5_vport *evport = mlx5_eswitch_get_vport(esw, vport); bool pschk; int err = 0; if (!ESW_ALLOWED(esw)) return -EPERM; - if (!LEGAL_VPORT(esw, vport)) - return -EINVAL; + if (IS_ERR(evport)) + return PTR_ERR(evport); mutex_lock(&esw->state_lock); - evport = &esw->vports[vport]; pschk = evport->info.spoofchk; evport->info.spoofchk = spoofchk; if (pschk && !is_valid_ether_addr(evport->info.mac)) @@ -2226,15 +2210,14 @@ out: int mlx5_eswitch_set_vport_trust(struct mlx5_eswitch *esw, int vport, bool setting) { - struct mlx5_vport *evport; + struct mlx5_vport *evport = mlx5_eswitch_get_vport(esw, vport); if (!ESW_ALLOWED(esw)) return -EPERM; - if (!LEGAL_VPORT(esw, vport)) - return -EINVAL; + if (IS_ERR(evport)) + return PTR_ERR(evport); mutex_lock(&esw->state_lock); - evport = &esw->vports[vport]; evport->info.trusted = setting; if (evport->enabled) esw_vport_change_handle_locked(evport); @@ -2284,7 +2267,7 @@ static int normalize_vports_min_rate(struct mlx5_eswitch *esw, u32 divider) if (bw_share == evport->qos.bw_share) continue; - err = esw_vport_qos_config(esw, evport->vport, vport_max_rate, + err = esw_vport_qos_config(esw, evport, vport_max_rate, bw_share); if (!err) evport->qos.bw_share = bw_share; @@ -2298,7 +2281,7 @@ static int normalize_vports_min_rate(struct mlx5_eswitch *esw, u32 divider) int mlx5_eswitch_set_vport_rate(struct mlx5_eswitch *esw, int vport, u32 max_rate, u32 min_rate) { - struct mlx5_vport *evport; + struct mlx5_vport *evport = mlx5_eswitch_get_vport(esw, vport); u32 fw_max_bw_share; u32 previous_min_rate; u32 divider; @@ -2308,8 +2291,8 @@ int mlx5_eswitch_set_vport_rate(struct mlx5_eswitch *esw, int vport, if (!ESW_ALLOWED(esw)) return -EPERM; - if (!LEGAL_VPORT(esw, vport)) - return -EINVAL; + if (IS_ERR(evport)) + return PTR_ERR(evport); fw_max_bw_share = MLX5_CAP_QOS(esw->dev, max_tsar_bw_share); min_rate_supported = MLX5_CAP_QOS(esw->dev, esw_bw_share) && @@ -2320,7 +2303,6 @@ int mlx5_eswitch_set_vport_rate(struct mlx5_eswitch *esw, int vport, return -EOPNOTSUPP; mutex_lock(&esw->state_lock); - evport = &esw->vports[vport]; if (min_rate == evport->info.min_rate) goto set_max_rate; @@ -2338,7 +2320,7 @@ set_max_rate: if (max_rate == evport->info.max_rate) goto unlock; - err = esw_vport_qos_config(esw, vport, max_rate, evport->qos.bw_share); + err = esw_vport_qos_config(esw, evport, max_rate, evport->qos.bw_share); if (!err) evport->info.max_rate = max_rate; @@ -2348,11 +2330,10 @@ unlock: } static int mlx5_eswitch_query_vport_drop_stats(struct mlx5_core_dev *dev, - int vport_idx, + struct mlx5_vport *vport, struct mlx5_vport_drop_stats *stats) { struct mlx5_eswitch *esw = dev->priv.eswitch; - struct mlx5_vport *vport = &esw->vports[vport_idx]; u64 rx_discard_vport_down, tx_discard_vport_down; u64 bytes = 0; int err = 0; @@ -2372,7 +2353,7 @@ static int mlx5_eswitch_query_vport_drop_stats(struct mlx5_core_dev *dev, !MLX5_CAP_GEN(dev, transmit_discard_vport_down)) return 0; - err = mlx5_query_vport_down_stats(dev, vport_idx, 1, + err = mlx5_query_vport_down_stats(dev, vport->vport, 1, &rx_discard_vport_down, &tx_discard_vport_down); if (err) @@ -2387,19 +2368,18 @@ static int mlx5_eswitch_query_vport_drop_stats(struct mlx5_core_dev *dev, } int mlx5_eswitch_get_vport_stats(struct mlx5_eswitch *esw, - int vport, + int vport_num, struct ifla_vf_stats *vf_stats) { + struct mlx5_vport *vport = mlx5_eswitch_get_vport(esw, vport_num); int outlen = MLX5_ST_SZ_BYTES(query_vport_counter_out); u32 in[MLX5_ST_SZ_DW(query_vport_counter_in)] = {0}; struct mlx5_vport_drop_stats stats = {0}; int err = 0; u32 *out; - if (!ESW_ALLOWED(esw)) - return -EPERM; - if (!LEGAL_VPORT(esw, vport)) - return -EINVAL; + if (IS_ERR(vport)) + return PTR_ERR(vport); out = kvzalloc(outlen, GFP_KERNEL); if (!out) @@ -2408,7 +2388,7 @@ int mlx5_eswitch_get_vport_stats(struct mlx5_eswitch *esw, MLX5_SET(query_vport_counter_in, in, opcode, MLX5_CMD_OP_QUERY_VPORT_COUNTER); MLX5_SET(query_vport_counter_in, in, op_mod, 0); - MLX5_SET(query_vport_counter_in, in, vport_number, vport); + MLX5_SET(query_vport_counter_in, in, vport_number, vport->vport); MLX5_SET(query_vport_counter_in, in, other_vport, 1); memset(out, 0, outlen); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h index 3f3cd32ae60a26c5544e15453ef9efcc6364d166..ed3fad689ec91d5e38d22b49f0c6e081250006c6 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h @@ -227,6 +227,18 @@ int esw_offloads_init(struct mlx5_eswitch *esw, int vf_nvports, int total_nvports); void esw_offloads_cleanup_reps(struct mlx5_eswitch *esw); int esw_offloads_init_reps(struct mlx5_eswitch *esw); +void esw_vport_cleanup_ingress_rules(struct mlx5_eswitch *esw, + struct mlx5_vport *vport); +int esw_vport_enable_ingress_acl(struct mlx5_eswitch *esw, + struct mlx5_vport *vport); +void esw_vport_cleanup_egress_rules(struct mlx5_eswitch *esw, + struct mlx5_vport *vport); +int esw_vport_enable_egress_acl(struct mlx5_eswitch *esw, + struct mlx5_vport *vport); +void esw_vport_disable_egress_acl(struct mlx5_eswitch *esw, + struct mlx5_vport *vport); +void esw_vport_disable_ingress_acl(struct mlx5_eswitch *esw, + struct mlx5_vport *vport); /* E-Switch API */ int mlx5_eswitch_init(struct mlx5_core_dev *dev); @@ -376,11 +388,11 @@ bool mlx5_esw_multipath_prereq(struct mlx5_core_dev *dev0, #define MLX5_DEBUG_ESWITCH_MASK BIT(3) -#define esw_info(dev, format, ...) \ - pr_info("(%s): E-Switch: " format, (dev)->priv.name, ##__VA_ARGS__) +#define esw_info(__dev, format, ...) \ + dev_info((__dev)->device, "E-Switch: " format, ##__VA_ARGS__) -#define esw_warn(dev, format, ...) \ - pr_warn("(%s): E-Switch: " format, (dev)->priv.name, ##__VA_ARGS__) +#define esw_warn(__dev, format, ...) \ + dev_warn((__dev)->device, "E-Switch: " format, ##__VA_ARGS__) #define esw_debug(dev, format, ...) \ mlx5_core_dbg_mask(dev, MLX5_DEBUG_ESWITCH_MASK, format, ##__VA_ARGS__) @@ -431,6 +443,54 @@ static inline int mlx5_eswitch_index_to_vport_num(struct mlx5_eswitch *esw, return index; } +/* TODO: This mlx5e_tc function shouldn't be called by eswitch */ +void mlx5e_tc_clean_fdb_peer_flows(struct mlx5_eswitch *esw); + +/* The vport getter/iterator are only valid after esw->total_vports + * and vport->vport are initialized in mlx5_eswitch_init. + */ +#define mlx5_esw_for_all_vports(esw, i, vport) \ + for ((i) = MLX5_VPORT_PF; \ + (vport) = &(esw)->vports[i], \ + (i) < (esw)->total_vports; (i)++) + +#define mlx5_esw_for_each_vf_vport(esw, i, vport, nvfs) \ + for ((i) = MLX5_VPORT_FIRST_VF; \ + (vport) = &(esw)->vports[(i)], \ + (i) <= (nvfs); (i)++) + +#define mlx5_esw_for_each_vf_vport_reverse(esw, i, vport, nvfs) \ + for ((i) = (nvfs); \ + (vport) = &(esw)->vports[(i)], \ + (i) >= MLX5_VPORT_FIRST_VF; (i)--) + +/* The rep getter/iterator are only valid after esw->total_vports + * and vport->vport are initialized in mlx5_eswitch_init. + */ +#define mlx5_esw_for_all_reps(esw, i, rep) \ + for ((i) = MLX5_VPORT_PF; \ + (rep) = &(esw)->offloads.vport_reps[i], \ + (i) < (esw)->total_vports; (i)++) + +#define mlx5_esw_for_each_vf_rep(esw, i, rep, nvfs) \ + for ((i) = MLX5_VPORT_FIRST_VF; \ + (rep) = &(esw)->offloads.vport_reps[i], \ + (i) <= (nvfs); (i)++) + +#define mlx5_esw_for_each_vf_rep_reverse(esw, i, rep, nvfs) \ + for ((i) = (nvfs); \ + (rep) = &(esw)->offloads.vport_reps[i], \ + (i) >= MLX5_VPORT_FIRST_VF; (i)--) + +#define mlx5_esw_for_each_vf_vport_num(esw, vport, nvfs) \ + for ((vport) = MLX5_VPORT_FIRST_VF; (vport) <= (nvfs); (vport)++) + +#define mlx5_esw_for_each_vf_vport_num_reverse(esw, vport, nvfs) \ + for ((vport) = (nvfs); (vport) >= MLX5_VPORT_FIRST_VF; (vport)--) + +struct mlx5_vport *__must_check +mlx5_eswitch_get_vport(struct mlx5_eswitch *esw, u16 vport_num); + #else /* CONFIG_MLX5_ESWITCH */ /* eswitch API stubs */ static inline int mlx5_eswitch_init(struct mlx5_core_dev *dev) { return 0; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c index 9b2d78ee22b88333fcc0b043d4234c340dcb3c32..e09ae27485ee33a99c88385b3c5b84c7ea19da11 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c @@ -37,17 +37,13 @@ #include #include "mlx5_core.h" #include "eswitch.h" +#include "rdma.h" #include "en.h" #include "fs_core.h" #include "lib/devcom.h" #include "ecpf.h" #include "lib/eq.h" -enum { - FDB_FAST_PATH = 0, - FDB_SLOW_PATH -}; - /* There are two match-all miss flows, one for unicast dst mac and * one for multicast. */ @@ -58,32 +54,6 @@ enum { #define UPLINK_REP_INDEX 0 -/* The rep getter/iterator are only valid after esw->total_vports - * and vport->vport are initialized in mlx5_eswitch_init. - */ -#define mlx5_esw_for_all_reps(esw, i, rep) \ - for ((i) = MLX5_VPORT_PF; \ - (rep) = &(esw)->offloads.vport_reps[i], \ - (i) < (esw)->total_vports; (i)++) - -#define mlx5_esw_for_each_vf_rep(esw, i, rep, nvfs) \ - for ((i) = MLX5_VPORT_FIRST_VF; \ - (rep) = &(esw)->offloads.vport_reps[i], \ - (i) <= (nvfs); (i)++) - -#define mlx5_esw_for_each_vf_rep_reverse(esw, i, rep, nvfs) \ - for ((i) = (nvfs); \ - (rep) = &(esw)->offloads.vport_reps[i], \ - (i) >= MLX5_VPORT_FIRST_VF; (i)--) - -#define mlx5_esw_for_each_vf_vport(esw, vport, nvfs) \ - for ((vport) = MLX5_VPORT_FIRST_VF; \ - (vport) <= (nvfs); (vport)++) - -#define mlx5_esw_for_each_vf_vport_reverse(esw, vport, nvfs) \ - for ((vport) = (nvfs); \ - (vport) >= MLX5_VPORT_FIRST_VF; (vport)--) - static struct mlx5_eswitch_rep *mlx5_eswitch_get_rep(struct mlx5_eswitch *esw, u16 vport_num) { @@ -363,7 +333,7 @@ static int esw_set_global_vlan_pop(struct mlx5_eswitch *esw, u8 val) esw_debug(esw->dev, "%s applying global %s policy\n", __func__, val ? "pop" : "none"); for (vf_vport = 1; vf_vport < esw->enabled_vports; vf_vport++) { rep = &esw->offloads.vport_reps[vf_vport]; - if (rep->rep_if[REP_ETH].state != REP_LOADED) + if (atomic_read(&rep->rep_if[REP_ETH].state) != REP_LOADED) continue; err = __mlx5_eswitch_set_vport_vlan(esw, rep->vport, 0, 0, val); @@ -663,7 +633,7 @@ static int esw_add_fdb_peer_miss_rules(struct mlx5_eswitch *esw, flows[mlx5_eswitch_ecpf_idx(esw)] = flow; } - mlx5_esw_for_each_vf_vport(esw, i, mlx5_core_max_vfs(esw->dev)) { + mlx5_esw_for_each_vf_vport_num(esw, i, mlx5_core_max_vfs(esw->dev)) { MLX5_SET(fte_match_set_misc, misc, source_port, i); flow = mlx5_add_flow_rules(esw->fdb_table.offloads.slow_fdb, spec, &flow_act, &dest, 1); @@ -681,7 +651,7 @@ static int esw_add_fdb_peer_miss_rules(struct mlx5_eswitch *esw, add_vf_flow_err: nvports = --i; - mlx5_esw_for_each_vf_vport_reverse(esw, i, nvports) + mlx5_esw_for_each_vf_vport_num_reverse(esw, i, nvports) mlx5_del_flow_rules(flows[i]); if (mlx5_ecpf_vport_exists(esw->dev)) @@ -704,7 +674,8 @@ static void esw_del_fdb_peer_miss_rules(struct mlx5_eswitch *esw) flows = esw->fdb_table.offloads.peer_miss_rules; - mlx5_esw_for_each_vf_vport_reverse(esw, i, mlx5_core_max_vfs(esw->dev)) + mlx5_esw_for_each_vf_vport_num_reverse(esw, i, + mlx5_core_max_vfs(esw->dev)) mlx5_del_flow_rules(flows[i]); if (mlx5_ecpf_vport_exists(esw->dev)) @@ -1287,13 +1258,13 @@ void esw_offloads_cleanup_reps(struct mlx5_eswitch *esw) int esw_offloads_init_reps(struct mlx5_eswitch *esw) { - int total_vfs = MLX5_TOTAL_VPORTS(esw->dev); + int total_vports = MLX5_TOTAL_VPORTS(esw->dev); struct mlx5_core_dev *dev = esw->dev; struct mlx5_eswitch_rep *rep; u8 hw_id[ETH_ALEN], rep_type; int vport; - esw->offloads.vport_reps = kcalloc(total_vfs, + esw->offloads.vport_reps = kcalloc(total_vports, sizeof(struct mlx5_eswitch_rep), GFP_KERNEL); if (!esw->offloads.vport_reps) @@ -1306,7 +1277,8 @@ int esw_offloads_init_reps(struct mlx5_eswitch *esw) ether_addr_copy(rep->hw_id, hw_id); for (rep_type = 0; rep_type < NUM_REP_TYPES; rep_type++) - rep->rep_if[rep_type].state = REP_UNREGISTERED; + atomic_set(&rep->rep_if[rep_type].state, + REP_UNREGISTERED); } return 0; @@ -1315,11 +1287,9 @@ int esw_offloads_init_reps(struct mlx5_eswitch *esw) static void __esw_offloads_unload_rep(struct mlx5_eswitch *esw, struct mlx5_eswitch_rep *rep, u8 rep_type) { - if (rep->rep_if[rep_type].state != REP_LOADED) - return; - - rep->rep_if[rep_type].unload(rep); - rep->rep_if[rep_type].state = REP_REGISTERED; + if (atomic_cmpxchg(&rep->rep_if[rep_type].state, + REP_LOADED, REP_REGISTERED) == REP_LOADED) + rep->rep_if[rep_type].unload(rep); } static void __unload_reps_special_vport(struct mlx5_eswitch *esw, u8 rep_type) @@ -1380,16 +1350,15 @@ static int __esw_offloads_load_rep(struct mlx5_eswitch *esw, { int err = 0; - if (rep->rep_if[rep_type].state != REP_REGISTERED) - return 0; - - err = rep->rep_if[rep_type].load(esw->dev, rep); - if (err) - return err; - - rep->rep_if[rep_type].state = REP_LOADED; + if (atomic_cmpxchg(&rep->rep_if[rep_type].state, + REP_REGISTERED, REP_LOADED) == REP_REGISTERED) { + err = rep->rep_if[rep_type].load(esw->dev, rep); + if (err) + atomic_set(&rep->rep_if[rep_type].state, + REP_REGISTERED); + } - return 0; + return err; } static int __load_reps_special_vport(struct mlx5_eswitch *esw, u8 rep_type) @@ -1523,8 +1492,6 @@ static int mlx5_esw_offloads_pair(struct mlx5_eswitch *esw, return 0; } -void mlx5e_tc_clean_fdb_peer_flows(struct mlx5_eswitch *esw); - static void mlx5_esw_offloads_unpair(struct mlx5_eswitch *esw) { mlx5e_tc_clean_fdb_peer_flows(esw); @@ -1607,6 +1574,169 @@ static void esw_offloads_devcom_cleanup(struct mlx5_eswitch *esw) mlx5_devcom_unregister_component(devcom, MLX5_DEVCOM_ESW_OFFLOADS); } +static int esw_vport_ingress_prio_tag_config(struct mlx5_eswitch *esw, + struct mlx5_vport *vport) +{ + struct mlx5_core_dev *dev = esw->dev; + struct mlx5_flow_act flow_act = {0}; + struct mlx5_flow_spec *spec; + int err = 0; + + /* For prio tag mode, there is only 1 FTEs: + * 1) Untagged packets - push prio tag VLAN, allow + * Unmatched traffic is allowed by default + */ + + if (!MLX5_CAP_ESW_INGRESS_ACL(dev, ft_support)) + return -EOPNOTSUPP; + + esw_vport_cleanup_ingress_rules(esw, vport); + + err = esw_vport_enable_ingress_acl(esw, vport); + if (err) { + mlx5_core_warn(esw->dev, + "failed to enable prio tag ingress acl (%d) on vport[%d]\n", + err, vport->vport); + return err; + } + + esw_debug(esw->dev, + "vport[%d] configure ingress rules\n", vport->vport); + + spec = kvzalloc(sizeof(*spec), GFP_KERNEL); + if (!spec) { + err = -ENOMEM; + goto out_no_mem; + } + + /* Untagged packets - push prio tag VLAN, allow */ + MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria, outer_headers.cvlan_tag); + MLX5_SET(fte_match_param, spec->match_value, outer_headers.cvlan_tag, 0); + spec->match_criteria_enable = MLX5_MATCH_OUTER_HEADERS; + flow_act.action = MLX5_FLOW_CONTEXT_ACTION_VLAN_PUSH | + MLX5_FLOW_CONTEXT_ACTION_ALLOW; + flow_act.vlan[0].ethtype = ETH_P_8021Q; + flow_act.vlan[0].vid = 0; + flow_act.vlan[0].prio = 0; + vport->ingress.allow_rule = + mlx5_add_flow_rules(vport->ingress.acl, spec, + &flow_act, NULL, 0); + if (IS_ERR(vport->ingress.allow_rule)) { + err = PTR_ERR(vport->ingress.allow_rule); + esw_warn(esw->dev, + "vport[%d] configure ingress untagged allow rule, err(%d)\n", + vport->vport, err); + vport->ingress.allow_rule = NULL; + goto out; + } + +out: + kvfree(spec); +out_no_mem: + if (err) + esw_vport_cleanup_ingress_rules(esw, vport); + return err; +} + +static int esw_vport_egress_prio_tag_config(struct mlx5_eswitch *esw, + struct mlx5_vport *vport) +{ + struct mlx5_flow_act flow_act = {0}; + struct mlx5_flow_spec *spec; + int err = 0; + + /* For prio tag mode, there is only 1 FTEs: + * 1) prio tag packets - pop the prio tag VLAN, allow + * Unmatched traffic is allowed by default + */ + + esw_vport_cleanup_egress_rules(esw, vport); + + err = esw_vport_enable_egress_acl(esw, vport); + if (err) { + mlx5_core_warn(esw->dev, + "failed to enable egress acl (%d) on vport[%d]\n", + err, vport->vport); + return err; + } + + esw_debug(esw->dev, + "vport[%d] configure prio tag egress rules\n", vport->vport); + + spec = kvzalloc(sizeof(*spec), GFP_KERNEL); + if (!spec) { + err = -ENOMEM; + goto out_no_mem; + } + + /* prio tag vlan rule - pop it so VF receives untagged packets */ + MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria, outer_headers.cvlan_tag); + MLX5_SET_TO_ONES(fte_match_param, spec->match_value, outer_headers.cvlan_tag); + MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria, outer_headers.first_vid); + MLX5_SET(fte_match_param, spec->match_value, outer_headers.first_vid, 0); + + spec->match_criteria_enable = MLX5_MATCH_OUTER_HEADERS; + flow_act.action = MLX5_FLOW_CONTEXT_ACTION_VLAN_POP | + MLX5_FLOW_CONTEXT_ACTION_ALLOW; + vport->egress.allowed_vlan = + mlx5_add_flow_rules(vport->egress.acl, spec, + &flow_act, NULL, 0); + if (IS_ERR(vport->egress.allowed_vlan)) { + err = PTR_ERR(vport->egress.allowed_vlan); + esw_warn(esw->dev, + "vport[%d] configure egress pop prio tag vlan rule failed, err(%d)\n", + vport->vport, err); + vport->egress.allowed_vlan = NULL; + goto out; + } + +out: + kvfree(spec); +out_no_mem: + if (err) + esw_vport_cleanup_egress_rules(esw, vport); + return err; +} + +static int esw_prio_tag_acls_config(struct mlx5_eswitch *esw, int nvports) +{ + struct mlx5_vport *vport = NULL; + int i, j; + int err; + + mlx5_esw_for_each_vf_vport(esw, i, vport, nvports) { + err = esw_vport_ingress_prio_tag_config(esw, vport); + if (err) + goto err_ingress; + err = esw_vport_egress_prio_tag_config(esw, vport); + if (err) + goto err_egress; + } + + return 0; + +err_egress: + esw_vport_disable_ingress_acl(esw, vport); +err_ingress: + mlx5_esw_for_each_vf_vport_reverse(esw, j, vport, i - 1) { + esw_vport_disable_egress_acl(esw, vport); + esw_vport_disable_ingress_acl(esw, vport); + } + + return err; +} + +static void esw_prio_tag_acls_cleanup(struct mlx5_eswitch *esw) +{ + struct mlx5_vport *vport; + int i; + + mlx5_esw_for_each_vf_vport(esw, i, vport, esw->nvports) { + esw_vport_disable_egress_acl(esw, vport); + esw_vport_disable_ingress_acl(esw, vport); + } +} + static int esw_offloads_steering_init(struct mlx5_eswitch *esw, int nvports) { int err; @@ -1614,6 +1744,12 @@ static int esw_offloads_steering_init(struct mlx5_eswitch *esw, int nvports) memset(&esw->fdb_table.offloads, 0, sizeof(struct offloads_fdb)); mutex_init(&esw->fdb_table.offloads.fdb_prio_lock); + if (MLX5_CAP_GEN(esw->dev, prio_tag_required)) { + err = esw_prio_tag_acls_config(esw, nvports); + if (err) + return err; + } + err = esw_create_offloads_fdb_tables(esw, nvports); if (err) return err; @@ -1642,6 +1778,8 @@ static void esw_offloads_steering_cleanup(struct mlx5_eswitch *esw) esw_destroy_vport_rx_group(esw); esw_destroy_offloads_table(esw); esw_destroy_offloads_fdb_tables(esw); + if (MLX5_CAP_GEN(esw->dev, prio_tag_required)) + esw_prio_tag_acls_cleanup(esw); } static void esw_host_params_event_handler(struct work_struct *work) @@ -1700,8 +1838,6 @@ int esw_offloads_init(struct mlx5_eswitch *esw, int vf_nvports, { int err; - mutex_init(&esw->fdb_table.offloads.fdb_prio_lock); - err = esw_offloads_steering_init(esw, total_nvports); if (err) return err; @@ -1719,6 +1855,8 @@ int esw_offloads_init(struct mlx5_eswitch *esw, int vf_nvports, esw->host_info.num_vfs = vf_nvports; } + mlx5_rdma_enable_roce(esw->dev); + return 0; err_reps: @@ -1757,6 +1895,7 @@ void esw_offloads_cleanup(struct mlx5_eswitch *esw) num_vfs = esw->dev->priv.sriov.num_vfs; } + mlx5_rdma_disable_roce(esw->dev); esw_offloads_devcom_cleanup(esw); esw_offloads_unload_all_reps(esw, num_vfs); esw_offloads_steering_cleanup(esw); @@ -2076,7 +2215,7 @@ void mlx5_eswitch_register_vport_reps(struct mlx5_eswitch *esw, rep_if->get_proto_dev = __rep_if->get_proto_dev; rep_if->priv = __rep_if->priv; - rep_if->state = REP_REGISTERED; + atomic_set(&rep_if->state, REP_REGISTERED); } } EXPORT_SYMBOL(mlx5_eswitch_register_vport_reps); @@ -2091,7 +2230,7 @@ void mlx5_eswitch_unregister_vport_reps(struct mlx5_eswitch *esw, u8 rep_type) __unload_reps_all_vport(esw, max_vf, rep_type); mlx5_esw_for_all_reps(esw, i, rep) - rep->rep_if[rep_type].state = REP_UNREGISTERED; + atomic_set(&rep->rep_if[rep_type].state, REP_UNREGISTERED); } EXPORT_SYMBOL(mlx5_eswitch_unregister_vport_reps); @@ -2111,7 +2250,7 @@ void *mlx5_eswitch_get_proto_dev(struct mlx5_eswitch *esw, rep = mlx5_eswitch_get_rep(esw, vport); - if (rep->rep_if[rep_type].state == REP_LOADED && + if (atomic_read(&rep->rep_if[rep_type].state) == REP_LOADED && rep->rep_if[rep_type].get_proto_dev) return rep->rep_if[rep_type].get_proto_dev(rep); return NULL; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/events.c b/drivers/net/ethernet/mellanox/mlx5/core/events.c index 5d5864e8df3c1ce4daefc99c2ffdcc4447044df4..a81e8d2168d8432d290767c702c136233a726594 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/events.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/events.c @@ -21,6 +21,7 @@ struct mlx5_event_nb { static int any_notifier(struct notifier_block *, unsigned long, void *); static int temp_warn(struct notifier_block *, unsigned long, void *); static int port_module(struct notifier_block *, unsigned long, void *); +static int pcie_core(struct notifier_block *, unsigned long, void *); /* handler which forwards the event to events->nh, driver notifiers */ static int forward_event(struct notifier_block *, unsigned long, void *); @@ -30,6 +31,7 @@ static struct mlx5_nb events_nbs_ref[] = { {.nb.notifier_call = any_notifier, .event_type = MLX5_EVENT_TYPE_NOTIFY_ANY }, {.nb.notifier_call = temp_warn, .event_type = MLX5_EVENT_TYPE_TEMP_WARN_EVENT }, {.nb.notifier_call = port_module, .event_type = MLX5_EVENT_TYPE_PORT_MODULE_EVENT }, + {.nb.notifier_call = pcie_core, .event_type = MLX5_EVENT_TYPE_GENERAL_EVENT }, /* Events to be forwarded (as is) to mlx5 core interfaces (mlx5e/mlx5_ib) */ {.nb.notifier_call = forward_event, .event_type = MLX5_EVENT_TYPE_PORT_CHANGE }, @@ -51,11 +53,14 @@ static struct mlx5_nb events_nbs_ref[] = { struct mlx5_events { struct mlx5_core_dev *dev; + struct workqueue_struct *wq; struct mlx5_event_nb notifiers[ARRAY_SIZE(events_nbs_ref)]; /* driver notifier chain */ struct atomic_notifier_head nh; /* port module events stats */ struct mlx5_pme_stats pme_stats; + /*pcie_core*/ + struct work_struct pcie_core_work; }; static const char *eqe_type_str(u8 type) @@ -249,6 +254,69 @@ static int port_module(struct notifier_block *nb, unsigned long type, void *data return NOTIFY_OK; } +enum { + MLX5_PCI_POWER_COULD_NOT_BE_READ = 0x0, + MLX5_PCI_POWER_SUFFICIENT_REPORTED = 0x1, + MLX5_PCI_POWER_INSUFFICIENT_REPORTED = 0x2, +}; + +static void mlx5_pcie_event(struct work_struct *work) +{ + u32 out[MLX5_ST_SZ_DW(mpein_reg)] = {0}; + u32 in[MLX5_ST_SZ_DW(mpein_reg)] = {0}; + struct mlx5_events *events; + struct mlx5_core_dev *dev; + u8 power_status; + u16 pci_power; + + events = container_of(work, struct mlx5_events, pcie_core_work); + dev = events->dev; + + if (!MLX5_CAP_MCAM_FEATURE(dev, pci_status_and_power)) + return; + + mlx5_core_access_reg(dev, in, sizeof(in), out, sizeof(out), + MLX5_REG_MPEIN, 0, 0); + power_status = MLX5_GET(mpein_reg, out, pwr_status); + pci_power = MLX5_GET(mpein_reg, out, pci_power); + + switch (power_status) { + case MLX5_PCI_POWER_COULD_NOT_BE_READ: + mlx5_core_info_rl(dev, + "PCIe slot power capability was not advertised.\n"); + break; + case MLX5_PCI_POWER_INSUFFICIENT_REPORTED: + mlx5_core_warn_rl(dev, + "Detected insufficient power on the PCIe slot (%uW).\n", + pci_power); + break; + case MLX5_PCI_POWER_SUFFICIENT_REPORTED: + mlx5_core_info_rl(dev, + "PCIe slot advertised sufficient power (%uW).\n", + pci_power); + break; + } +} + +static int pcie_core(struct notifier_block *nb, unsigned long type, void *data) +{ + struct mlx5_event_nb *event_nb = mlx5_nb_cof(nb, + struct mlx5_event_nb, + nb); + struct mlx5_events *events = event_nb->ctx; + struct mlx5_eqe *eqe = data; + + switch (eqe->sub_type) { + case MLX5_GENERAL_SUBTYPE_PCI_POWER_CHANGE_EVENT: + queue_work(events->wq, &events->pcie_core_work); + break; + default: + return NOTIFY_DONE; + } + + return NOTIFY_OK; +} + void mlx5_get_pme_stats(struct mlx5_core_dev *dev, struct mlx5_pme_stats *stats) { *stats = dev->priv.events->pme_stats; @@ -277,11 +345,17 @@ int mlx5_events_init(struct mlx5_core_dev *dev) ATOMIC_INIT_NOTIFIER_HEAD(&events->nh); events->dev = dev; dev->priv.events = events; + events->wq = create_singlethread_workqueue("mlx5_events"); + if (!events->wq) + return -ENOMEM; + INIT_WORK(&events->pcie_core_work, mlx5_pcie_event); + return 0; } void mlx5_events_cleanup(struct mlx5_core_dev *dev) { + destroy_workqueue(dev->priv.events->wq); kvfree(dev->priv.events); } @@ -304,6 +378,7 @@ void mlx5_events_stop(struct mlx5_core_dev *dev) for (i = ARRAY_SIZE(events_nbs_ref) - 1; i >= 0 ; i--) mlx5_eq_notifier_unregister(dev, &events->notifiers[i].nb); + flush_workqueue(events->wq); } int mlx5_notifier_register(struct mlx5_core_dev *dev, struct notifier_block *nb) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c b/drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c index 873541ef4c1b754b15209f7516bcee07378f6763..ca2296a2f9ee321cd388088369454f3f50e9301d 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c @@ -135,7 +135,7 @@ static void mlx5_fpga_conn_notify_hw(struct mlx5_fpga_conn *conn, void *wqe) *conn->qp.wq.sq.db = cpu_to_be32(conn->qp.sq.pc); /* Make sure that doorbell record is visible before ringing */ wmb(); - mlx5_write64(wqe, conn->fdev->conn_res.uar->map + MLX5_BF_OFFSET, NULL); + mlx5_write64(wqe, conn->fdev->conn_res.uar->map + MLX5_BF_OFFSET); } static void mlx5_fpga_conn_post_send(struct mlx5_fpga_conn *conn, diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fpga/core.h b/drivers/net/ethernet/mellanox/mlx5/core/fpga/core.h index 7e2e871dbf833b059d790bf161feb7264f2df093..52c9dee91ea465340155a14b9a4bcffc55b03cbb 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/fpga/core.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/fpga/core.h @@ -37,6 +37,7 @@ #include +#include "mlx5_core.h" #include "lib/eq.h" #include "fpga/cmd.h" @@ -62,26 +63,26 @@ struct mlx5_fpga_device { }; #define mlx5_fpga_dbg(__adev, format, ...) \ - dev_dbg(&(__adev)->mdev->pdev->dev, "FPGA: %s:%d:(pid %d): " format, \ - __func__, __LINE__, current->pid, ##__VA_ARGS__) + mlx5_core_dbg((__adev)->mdev, "FPGA: %s:%d:(pid %d): " format, \ + __func__, __LINE__, current->pid, ##__VA_ARGS__) #define mlx5_fpga_err(__adev, format, ...) \ - dev_err(&(__adev)->mdev->pdev->dev, "FPGA: %s:%d:(pid %d): " format, \ - __func__, __LINE__, current->pid, ##__VA_ARGS__) + mlx5_core_err((__adev)->mdev, "FPGA: %s:%d:(pid %d): " format, \ + __func__, __LINE__, current->pid, ##__VA_ARGS__) #define mlx5_fpga_warn(__adev, format, ...) \ - dev_warn(&(__adev)->mdev->pdev->dev, "FPGA: %s:%d:(pid %d): " format, \ - __func__, __LINE__, current->pid, ##__VA_ARGS__) + mlx5_core_warn((__adev)->mdev, "FPGA: %s:%d:(pid %d): " format, \ + __func__, __LINE__, current->pid, ##__VA_ARGS__) #define mlx5_fpga_warn_ratelimited(__adev, format, ...) \ - dev_warn_ratelimited(&(__adev)->mdev->pdev->dev, "FPGA: %s:%d: " \ - format, __func__, __LINE__, ##__VA_ARGS__) + mlx5_core_err_rl((__adev)->mdev, "FPGA: %s:%d: " \ + format, __func__, __LINE__, ##__VA_ARGS__) #define mlx5_fpga_notice(__adev, format, ...) \ - dev_notice(&(__adev)->mdev->pdev->dev, "FPGA: " format, ##__VA_ARGS__) + mlx5_core_info((__adev)->mdev, "FPGA: " format, ##__VA_ARGS__) #define mlx5_fpga_info(__adev, format, ...) \ - dev_info(&(__adev)->mdev->pdev->dev, "FPGA: " format, ##__VA_ARGS__) + mlx5_core_info((__adev)->mdev, "FPGA: " format, ##__VA_ARGS__) int mlx5_fpga_init(struct mlx5_core_dev *mdev); void mlx5_fpga_cleanup(struct mlx5_core_dev *mdev); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fpga/ipsec.c b/drivers/net/ethernet/mellanox/mlx5/core/fpga/ipsec.c index 5a22c5874f3bc30789ffe5fade6ebda0d50ee2b3..52c47d3dd5a5432efbe1b1816a86c71cc8cb09cc 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/fpga/ipsec.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/fpga/ipsec.c @@ -989,32 +989,33 @@ static enum fs_flow_table_type egress_to_fs_ft(bool egress) return egress ? FS_FT_NIC_TX : FS_FT_NIC_RX; } -static int fpga_ipsec_fs_create_flow_group(struct mlx5_core_dev *dev, +static int fpga_ipsec_fs_create_flow_group(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, u32 *in, - unsigned int *group_id, + struct mlx5_flow_group *fg, bool is_egress) { - int (*create_flow_group)(struct mlx5_core_dev *dev, + int (*create_flow_group)(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, u32 *in, - unsigned int *group_id) = + struct mlx5_flow_group *fg) = mlx5_fs_cmd_get_default(egress_to_fs_ft(is_egress))->create_flow_group; char *misc_params_c = MLX5_ADDR_OF(create_flow_group_in, in, match_criteria.misc_parameters); + struct mlx5_core_dev *dev = ns->dev; u32 saved_outer_esp_spi_mask; u8 match_criteria_enable; int ret; if (MLX5_CAP_FLOWTABLE(dev, flow_table_properties_nic_receive.ft_field_support.outer_esp_spi)) - return create_flow_group(dev, ft, in, group_id); + return create_flow_group(ns, ft, in, fg); match_criteria_enable = MLX5_GET(create_flow_group_in, in, match_criteria_enable); saved_outer_esp_spi_mask = MLX5_GET(fte_match_set_misc, misc_params_c, outer_esp_spi); if (!match_criteria_enable || !saved_outer_esp_spi_mask) - return create_flow_group(dev, ft, in, group_id); + return create_flow_group(ns, ft, in, fg); MLX5_SET(fte_match_set_misc, misc_params_c, outer_esp_spi, 0); @@ -1023,7 +1024,7 @@ static int fpga_ipsec_fs_create_flow_group(struct mlx5_core_dev *dev, MLX5_SET(create_flow_group_in, in, match_criteria_enable, match_criteria_enable & ~MLX5_MATCH_MISC_PARAMETERS); - ret = create_flow_group(dev, ft, in, group_id); + ret = create_flow_group(ns, ft, in, fg); MLX5_SET(fte_match_set_misc, misc_params_c, outer_esp_spi, saved_outer_esp_spi_mask); MLX5_SET(create_flow_group_in, in, match_criteria_enable, match_criteria_enable); @@ -1031,17 +1032,18 @@ static int fpga_ipsec_fs_create_flow_group(struct mlx5_core_dev *dev, return ret; } -static int fpga_ipsec_fs_create_fte(struct mlx5_core_dev *dev, +static int fpga_ipsec_fs_create_fte(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, struct mlx5_flow_group *fg, struct fs_fte *fte, bool is_egress) { - int (*create_fte)(struct mlx5_core_dev *dev, + int (*create_fte)(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, struct mlx5_flow_group *fg, struct fs_fte *fte) = mlx5_fs_cmd_get_default(egress_to_fs_ft(is_egress))->create_fte; + struct mlx5_core_dev *dev = ns->dev; struct mlx5_fpga_device *fdev = dev->fpga; struct mlx5_fpga_ipsec *fipsec = fdev->ipsec; struct mlx5_fpga_ipsec_rule *rule; @@ -1053,7 +1055,7 @@ static int fpga_ipsec_fs_create_fte(struct mlx5_core_dev *dev, !(fte->action.action & (MLX5_FLOW_CONTEXT_ACTION_ENCRYPT | MLX5_FLOW_CONTEXT_ACTION_DECRYPT))) - return create_fte(dev, ft, fg, fte); + return create_fte(ns, ft, fg, fte); rule = kzalloc(sizeof(*rule), GFP_KERNEL); if (!rule) @@ -1070,7 +1072,7 @@ static int fpga_ipsec_fs_create_fte(struct mlx5_core_dev *dev, WARN_ON(rule_insert(fipsec, rule)); modify_spec_mailbox(dev, fte, &mbox_mod); - ret = create_fte(dev, ft, fg, fte); + ret = create_fte(ns, ft, fg, fte); restore_spec_mailbox(fte, &mbox_mod); if (ret) { _rule_delete(fipsec, rule); @@ -1081,19 +1083,20 @@ static int fpga_ipsec_fs_create_fte(struct mlx5_core_dev *dev, return ret; } -static int fpga_ipsec_fs_update_fte(struct mlx5_core_dev *dev, +static int fpga_ipsec_fs_update_fte(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, - unsigned int group_id, + struct mlx5_flow_group *fg, int modify_mask, struct fs_fte *fte, bool is_egress) { - int (*update_fte)(struct mlx5_core_dev *dev, + int (*update_fte)(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, - unsigned int group_id, + struct mlx5_flow_group *fg, int modify_mask, struct fs_fte *fte) = mlx5_fs_cmd_get_default(egress_to_fs_ft(is_egress))->update_fte; + struct mlx5_core_dev *dev = ns->dev; bool is_esp = fte->action.esp_id; struct mailbox_mod mbox_mod; int ret; @@ -1102,24 +1105,25 @@ static int fpga_ipsec_fs_update_fte(struct mlx5_core_dev *dev, !(fte->action.action & (MLX5_FLOW_CONTEXT_ACTION_ENCRYPT | MLX5_FLOW_CONTEXT_ACTION_DECRYPT))) - return update_fte(dev, ft, group_id, modify_mask, fte); + return update_fte(ns, ft, fg, modify_mask, fte); modify_spec_mailbox(dev, fte, &mbox_mod); - ret = update_fte(dev, ft, group_id, modify_mask, fte); + ret = update_fte(ns, ft, fg, modify_mask, fte); restore_spec_mailbox(fte, &mbox_mod); return ret; } -static int fpga_ipsec_fs_delete_fte(struct mlx5_core_dev *dev, +static int fpga_ipsec_fs_delete_fte(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, struct fs_fte *fte, bool is_egress) { - int (*delete_fte)(struct mlx5_core_dev *dev, + int (*delete_fte)(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, struct fs_fte *fte) = mlx5_fs_cmd_get_default(egress_to_fs_ft(is_egress))->delete_fte; + struct mlx5_core_dev *dev = ns->dev; struct mlx5_fpga_device *fdev = dev->fpga; struct mlx5_fpga_ipsec *fipsec = fdev->ipsec; struct mlx5_fpga_ipsec_rule *rule; @@ -1131,7 +1135,7 @@ static int fpga_ipsec_fs_delete_fte(struct mlx5_core_dev *dev, !(fte->action.action & (MLX5_FLOW_CONTEXT_ACTION_ENCRYPT | MLX5_FLOW_CONTEXT_ACTION_DECRYPT))) - return delete_fte(dev, ft, fte); + return delete_fte(ns, ft, fte); rule = rule_search(fipsec, fte); if (!rule) @@ -1141,84 +1145,84 @@ static int fpga_ipsec_fs_delete_fte(struct mlx5_core_dev *dev, rule_delete(fipsec, rule); modify_spec_mailbox(dev, fte, &mbox_mod); - ret = delete_fte(dev, ft, fte); + ret = delete_fte(ns, ft, fte); restore_spec_mailbox(fte, &mbox_mod); return ret; } static int -mlx5_fpga_ipsec_fs_create_flow_group_egress(struct mlx5_core_dev *dev, +mlx5_fpga_ipsec_fs_create_flow_group_egress(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, u32 *in, - unsigned int *group_id) + struct mlx5_flow_group *fg) { - return fpga_ipsec_fs_create_flow_group(dev, ft, in, group_id, true); + return fpga_ipsec_fs_create_flow_group(ns, ft, in, fg, true); } static int -mlx5_fpga_ipsec_fs_create_fte_egress(struct mlx5_core_dev *dev, +mlx5_fpga_ipsec_fs_create_fte_egress(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, struct mlx5_flow_group *fg, struct fs_fte *fte) { - return fpga_ipsec_fs_create_fte(dev, ft, fg, fte, true); + return fpga_ipsec_fs_create_fte(ns, ft, fg, fte, true); } static int -mlx5_fpga_ipsec_fs_update_fte_egress(struct mlx5_core_dev *dev, +mlx5_fpga_ipsec_fs_update_fte_egress(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, - unsigned int group_id, + struct mlx5_flow_group *fg, int modify_mask, struct fs_fte *fte) { - return fpga_ipsec_fs_update_fte(dev, ft, group_id, modify_mask, fte, + return fpga_ipsec_fs_update_fte(ns, ft, fg, modify_mask, fte, true); } static int -mlx5_fpga_ipsec_fs_delete_fte_egress(struct mlx5_core_dev *dev, +mlx5_fpga_ipsec_fs_delete_fte_egress(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, struct fs_fte *fte) { - return fpga_ipsec_fs_delete_fte(dev, ft, fte, true); + return fpga_ipsec_fs_delete_fte(ns, ft, fte, true); } static int -mlx5_fpga_ipsec_fs_create_flow_group_ingress(struct mlx5_core_dev *dev, +mlx5_fpga_ipsec_fs_create_flow_group_ingress(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, u32 *in, - unsigned int *group_id) + struct mlx5_flow_group *fg) { - return fpga_ipsec_fs_create_flow_group(dev, ft, in, group_id, false); + return fpga_ipsec_fs_create_flow_group(ns, ft, in, fg, false); } static int -mlx5_fpga_ipsec_fs_create_fte_ingress(struct mlx5_core_dev *dev, +mlx5_fpga_ipsec_fs_create_fte_ingress(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, struct mlx5_flow_group *fg, struct fs_fte *fte) { - return fpga_ipsec_fs_create_fte(dev, ft, fg, fte, false); + return fpga_ipsec_fs_create_fte(ns, ft, fg, fte, false); } static int -mlx5_fpga_ipsec_fs_update_fte_ingress(struct mlx5_core_dev *dev, +mlx5_fpga_ipsec_fs_update_fte_ingress(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, - unsigned int group_id, + struct mlx5_flow_group *fg, int modify_mask, struct fs_fte *fte) { - return fpga_ipsec_fs_update_fte(dev, ft, group_id, modify_mask, fte, + return fpga_ipsec_fs_update_fte(ns, ft, fg, modify_mask, fte, false); } static int -mlx5_fpga_ipsec_fs_delete_fte_ingress(struct mlx5_core_dev *dev, +mlx5_fpga_ipsec_fs_delete_fte_ingress(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, struct fs_fte *fte) { - return fpga_ipsec_fs_delete_fte(dev, ft, fte, false); + return fpga_ipsec_fs_delete_fte(ns, ft, fte, false); } static struct mlx5_flow_cmds fpga_ipsec_ingress; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.c b/drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.c index c44ccb67c4a36777b3d87ca00ba1fc6027689c85..013b1ca4a79126b27390c04444831cbfa8bcd6c1 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.c @@ -39,7 +39,7 @@ #include "mlx5_core.h" #include "eswitch.h" -static int mlx5_cmd_stub_update_root_ft(struct mlx5_core_dev *dev, +static int mlx5_cmd_stub_update_root_ft(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, u32 underlay_qpn, bool disconnect) @@ -47,47 +47,43 @@ static int mlx5_cmd_stub_update_root_ft(struct mlx5_core_dev *dev, return 0; } -static int mlx5_cmd_stub_create_flow_table(struct mlx5_core_dev *dev, - u16 vport, - enum fs_flow_table_op_mod op_mod, - enum fs_flow_table_type type, - unsigned int level, +static int mlx5_cmd_stub_create_flow_table(struct mlx5_flow_root_namespace *ns, + struct mlx5_flow_table *ft, unsigned int log_size, - struct mlx5_flow_table *next_ft, - unsigned int *table_id, u32 flags) + struct mlx5_flow_table *next_ft) { return 0; } -static int mlx5_cmd_stub_destroy_flow_table(struct mlx5_core_dev *dev, +static int mlx5_cmd_stub_destroy_flow_table(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft) { return 0; } -static int mlx5_cmd_stub_modify_flow_table(struct mlx5_core_dev *dev, +static int mlx5_cmd_stub_modify_flow_table(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, struct mlx5_flow_table *next_ft) { return 0; } -static int mlx5_cmd_stub_create_flow_group(struct mlx5_core_dev *dev, +static int mlx5_cmd_stub_create_flow_group(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, u32 *in, - unsigned int *group_id) + struct mlx5_flow_group *fg) { return 0; } -static int mlx5_cmd_stub_destroy_flow_group(struct mlx5_core_dev *dev, +static int mlx5_cmd_stub_destroy_flow_group(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, - unsigned int group_id) + struct mlx5_flow_group *fg) { return 0; } -static int mlx5_cmd_stub_create_fte(struct mlx5_core_dev *dev, +static int mlx5_cmd_stub_create_fte(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, struct mlx5_flow_group *group, struct fs_fte *fte) @@ -95,28 +91,29 @@ static int mlx5_cmd_stub_create_fte(struct mlx5_core_dev *dev, return 0; } -static int mlx5_cmd_stub_update_fte(struct mlx5_core_dev *dev, +static int mlx5_cmd_stub_update_fte(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, - unsigned int group_id, + struct mlx5_flow_group *group, int modify_mask, struct fs_fte *fte) { return -EOPNOTSUPP; } -static int mlx5_cmd_stub_delete_fte(struct mlx5_core_dev *dev, +static int mlx5_cmd_stub_delete_fte(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, struct fs_fte *fte) { return 0; } -static int mlx5_cmd_update_root_ft(struct mlx5_core_dev *dev, +static int mlx5_cmd_update_root_ft(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, u32 underlay_qpn, bool disconnect) { u32 in[MLX5_ST_SZ_DW(set_flow_table_root_in)] = {0}; u32 out[MLX5_ST_SZ_DW(set_flow_table_root_out)] = {0}; + struct mlx5_core_dev *dev = ns->dev; if ((MLX5_CAP_GEN(dev, port_type) == MLX5_CAP_PORT_TYPE_IB) && underlay_qpn == 0) @@ -143,29 +140,26 @@ static int mlx5_cmd_update_root_ft(struct mlx5_core_dev *dev, return mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); } -static int mlx5_cmd_create_flow_table(struct mlx5_core_dev *dev, - u16 vport, - enum fs_flow_table_op_mod op_mod, - enum fs_flow_table_type type, - unsigned int level, +static int mlx5_cmd_create_flow_table(struct mlx5_flow_root_namespace *ns, + struct mlx5_flow_table *ft, unsigned int log_size, - struct mlx5_flow_table *next_ft, - unsigned int *table_id, u32 flags) + struct mlx5_flow_table *next_ft) { - int en_encap = !!(flags & MLX5_FLOW_TABLE_TUNNEL_EN_REFORMAT); - int en_decap = !!(flags & MLX5_FLOW_TABLE_TUNNEL_EN_DECAP); + int en_encap = !!(ft->flags & MLX5_FLOW_TABLE_TUNNEL_EN_REFORMAT); + int en_decap = !!(ft->flags & MLX5_FLOW_TABLE_TUNNEL_EN_DECAP); u32 out[MLX5_ST_SZ_DW(create_flow_table_out)] = {0}; u32 in[MLX5_ST_SZ_DW(create_flow_table_in)] = {0}; + struct mlx5_core_dev *dev = ns->dev; int err; MLX5_SET(create_flow_table_in, in, opcode, MLX5_CMD_OP_CREATE_FLOW_TABLE); - MLX5_SET(create_flow_table_in, in, table_type, type); - MLX5_SET(create_flow_table_in, in, flow_table_context.level, level); + MLX5_SET(create_flow_table_in, in, table_type, ft->type); + MLX5_SET(create_flow_table_in, in, flow_table_context.level, ft->level); MLX5_SET(create_flow_table_in, in, flow_table_context.log_size, log_size); - if (vport) { - MLX5_SET(create_flow_table_in, in, vport_number, vport); + if (ft->vport) { + MLX5_SET(create_flow_table_in, in, vport_number, ft->vport); MLX5_SET(create_flow_table_in, in, other_vport, 1); } @@ -174,13 +168,18 @@ static int mlx5_cmd_create_flow_table(struct mlx5_core_dev *dev, MLX5_SET(create_flow_table_in, in, flow_table_context.reformat_en, en_encap); - switch (op_mod) { + switch (ft->op_mod) { case FS_FT_OP_MOD_NORMAL: if (next_ft) { MLX5_SET(create_flow_table_in, in, - flow_table_context.table_miss_action, 1); + flow_table_context.table_miss_action, + MLX5_FLOW_TABLE_MISS_ACTION_FWD); MLX5_SET(create_flow_table_in, in, flow_table_context.table_miss_id, next_ft->id); + } else { + MLX5_SET(create_flow_table_in, in, + flow_table_context.table_miss_action, + ns->def_miss_action); } break; @@ -195,16 +194,17 @@ static int mlx5_cmd_create_flow_table(struct mlx5_core_dev *dev, err = mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); if (!err) - *table_id = MLX5_GET(create_flow_table_out, out, - table_id); + ft->id = MLX5_GET(create_flow_table_out, out, + table_id); return err; } -static int mlx5_cmd_destroy_flow_table(struct mlx5_core_dev *dev, +static int mlx5_cmd_destroy_flow_table(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft) { u32 in[MLX5_ST_SZ_DW(destroy_flow_table_in)] = {0}; u32 out[MLX5_ST_SZ_DW(destroy_flow_table_out)] = {0}; + struct mlx5_core_dev *dev = ns->dev; MLX5_SET(destroy_flow_table_in, in, opcode, MLX5_CMD_OP_DESTROY_FLOW_TABLE); @@ -218,12 +218,13 @@ static int mlx5_cmd_destroy_flow_table(struct mlx5_core_dev *dev, return mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); } -static int mlx5_cmd_modify_flow_table(struct mlx5_core_dev *dev, +static int mlx5_cmd_modify_flow_table(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, struct mlx5_flow_table *next_ft) { u32 in[MLX5_ST_SZ_DW(modify_flow_table_in)] = {0}; u32 out[MLX5_ST_SZ_DW(modify_flow_table_out)] = {0}; + struct mlx5_core_dev *dev = ns->dev; MLX5_SET(modify_flow_table_in, in, opcode, MLX5_CMD_OP_MODIFY_FLOW_TABLE); @@ -250,26 +251,29 @@ static int mlx5_cmd_modify_flow_table(struct mlx5_core_dev *dev, MLX5_MODIFY_FLOW_TABLE_MISS_TABLE_ID); if (next_ft) { MLX5_SET(modify_flow_table_in, in, - flow_table_context.table_miss_action, 1); + flow_table_context.table_miss_action, + MLX5_FLOW_TABLE_MISS_ACTION_FWD); MLX5_SET(modify_flow_table_in, in, flow_table_context.table_miss_id, next_ft->id); } else { MLX5_SET(modify_flow_table_in, in, - flow_table_context.table_miss_action, 0); + flow_table_context.table_miss_action, + ns->def_miss_action); } } return mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out)); } -static int mlx5_cmd_create_flow_group(struct mlx5_core_dev *dev, +static int mlx5_cmd_create_flow_group(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, u32 *in, - unsigned int *group_id) + struct mlx5_flow_group *fg) { u32 out[MLX5_ST_SZ_DW(create_flow_group_out)] = {0}; int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in); + struct mlx5_core_dev *dev = ns->dev; int err; MLX5_SET(create_flow_group_in, in, opcode, @@ -283,23 +287,24 @@ static int mlx5_cmd_create_flow_group(struct mlx5_core_dev *dev, err = mlx5_cmd_exec(dev, in, inlen, out, sizeof(out)); if (!err) - *group_id = MLX5_GET(create_flow_group_out, out, - group_id); + fg->id = MLX5_GET(create_flow_group_out, out, + group_id); return err; } -static int mlx5_cmd_destroy_flow_group(struct mlx5_core_dev *dev, +static int mlx5_cmd_destroy_flow_group(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, - unsigned int group_id) + struct mlx5_flow_group *fg) { u32 out[MLX5_ST_SZ_DW(destroy_flow_group_out)] = {0}; u32 in[MLX5_ST_SZ_DW(destroy_flow_group_in)] = {0}; + struct mlx5_core_dev *dev = ns->dev; MLX5_SET(destroy_flow_group_in, in, opcode, MLX5_CMD_OP_DESTROY_FLOW_GROUP); MLX5_SET(destroy_flow_group_in, in, table_type, ft->type); MLX5_SET(destroy_flow_group_in, in, table_id, ft->id); - MLX5_SET(destroy_flow_group_in, in, group_id, group_id); + MLX5_SET(destroy_flow_group_in, in, group_id, fg->id); if (ft->vport) { MLX5_SET(destroy_flow_group_in, in, vport_number, ft->vport); MLX5_SET(destroy_flow_group_in, in, other_vport, 1); @@ -505,23 +510,25 @@ err_out: return err; } -static int mlx5_cmd_create_fte(struct mlx5_core_dev *dev, +static int mlx5_cmd_create_fte(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, struct mlx5_flow_group *group, struct fs_fte *fte) { + struct mlx5_core_dev *dev = ns->dev; unsigned int group_id = group->id; return mlx5_cmd_set_fte(dev, 0, 0, ft, group_id, fte); } -static int mlx5_cmd_update_fte(struct mlx5_core_dev *dev, +static int mlx5_cmd_update_fte(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, - unsigned int group_id, + struct mlx5_flow_group *fg, int modify_mask, struct fs_fte *fte) { int opmod; + struct mlx5_core_dev *dev = ns->dev; int atomic_mod_cap = MLX5_CAP_FLOWTABLE(dev, flow_table_properties_nic_receive. flow_modify_en); @@ -529,15 +536,16 @@ static int mlx5_cmd_update_fte(struct mlx5_core_dev *dev, return -EOPNOTSUPP; opmod = 1; - return mlx5_cmd_set_fte(dev, opmod, modify_mask, ft, group_id, fte); + return mlx5_cmd_set_fte(dev, opmod, modify_mask, ft, fg->id, fte); } -static int mlx5_cmd_delete_fte(struct mlx5_core_dev *dev, +static int mlx5_cmd_delete_fte(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, struct fs_fte *fte) { u32 out[MLX5_ST_SZ_DW(delete_fte_out)] = {0}; u32 in[MLX5_ST_SZ_DW(delete_fte_in)] = {0}; + struct mlx5_core_dev *dev = ns->dev; MLX5_SET(delete_fte_in, in, opcode, MLX5_CMD_OP_DELETE_FLOW_TABLE_ENTRY); MLX5_SET(delete_fte_in, in, table_type, ft->type); @@ -853,6 +861,7 @@ const struct mlx5_flow_cmds *mlx5_fs_cmd_get_default(enum fs_flow_table_type typ case FS_FT_SNIFFER_RX: case FS_FT_SNIFFER_TX: case FS_FT_NIC_TX: + case FS_FT_RDMA_RX: return mlx5_fs_cmd_get_fw_cmds(); default: return mlx5_fs_cmd_get_stub_cmds(); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.h b/drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.h index 6228ba7bfa1a90fc2cafed3a13128e83391c903c..e340f9af2f5a7525393a56a82062d6990f180361 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.h @@ -36,45 +36,42 @@ #include "fs_core.h" struct mlx5_flow_cmds { - int (*create_flow_table)(struct mlx5_core_dev *dev, - u16 vport, - enum fs_flow_table_op_mod op_mod, - enum fs_flow_table_type type, - unsigned int level, unsigned int log_size, - struct mlx5_flow_table *next_ft, - unsigned int *table_id, u32 flags); - int (*destroy_flow_table)(struct mlx5_core_dev *dev, + int (*create_flow_table)(struct mlx5_flow_root_namespace *ns, + struct mlx5_flow_table *ft, + unsigned int log_size, + struct mlx5_flow_table *next_ft); + int (*destroy_flow_table)(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft); - int (*modify_flow_table)(struct mlx5_core_dev *dev, + int (*modify_flow_table)(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, struct mlx5_flow_table *next_ft); - int (*create_flow_group)(struct mlx5_core_dev *dev, + int (*create_flow_group)(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, u32 *in, - unsigned int *group_id); + struct mlx5_flow_group *fg); - int (*destroy_flow_group)(struct mlx5_core_dev *dev, + int (*destroy_flow_group)(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, - unsigned int group_id); + struct mlx5_flow_group *fg); - int (*create_fte)(struct mlx5_core_dev *dev, + int (*create_fte)(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, struct mlx5_flow_group *fg, struct fs_fte *fte); - int (*update_fte)(struct mlx5_core_dev *dev, + int (*update_fte)(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, - unsigned int group_id, + struct mlx5_flow_group *fg, int modify_mask, struct fs_fte *fte); - int (*delete_fte)(struct mlx5_core_dev *dev, + int (*delete_fte)(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, struct fs_fte *fte); - int (*update_root_ft)(struct mlx5_core_dev *dev, + int (*update_root_ft)(struct mlx5_flow_root_namespace *ns, struct mlx5_flow_table *ft, u32 underlay_qpn, bool disconnect); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c index 0be3eb86dd84e7abc0d7aac32d19b4d7643bc91d..fb5b61727ee70845cf9a980c5805eaf19f174640 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c @@ -403,7 +403,7 @@ static void del_hw_flow_table(struct fs_node *node) trace_mlx5_fs_del_ft(ft); if (node->active) { - err = root->cmds->destroy_flow_table(dev, ft); + err = root->cmds->destroy_flow_table(root, ft); if (err) mlx5_core_warn(dev, "flow steering can't destroy ft\n"); } @@ -435,7 +435,7 @@ static void modify_fte(struct fs_fte *fte) dev = get_dev(&fte->node); root = find_root(&ft->node); - err = root->cmds->update_fte(dev, ft, fg->id, fte->modify_mask, fte); + err = root->cmds->update_fte(root, ft, fg, fte->modify_mask, fte); if (err) mlx5_core_warn(dev, "%s can't del rule fg id=%d fte_index=%d\n", @@ -492,7 +492,7 @@ static void del_hw_fte(struct fs_node *node) dev = get_dev(&ft->node); root = find_root(&ft->node); if (node->active) { - err = root->cmds->delete_fte(dev, ft, fte); + err = root->cmds->delete_fte(root, ft, fte); if (err) mlx5_core_warn(dev, "flow steering can't delete fte in index %d of flow group id %d\n", @@ -532,7 +532,7 @@ static void del_hw_flow_group(struct fs_node *node) trace_mlx5_fs_del_fg(fg); root = find_root(&ft->node); - if (fg->node.active && root->cmds->destroy_flow_group(dev, ft, fg->id)) + if (fg->node.active && root->cmds->destroy_flow_group(root, ft, fg)) mlx5_core_warn(dev, "flow steering can't destroy fg %d of ft %d\n", fg->id, ft->id); } @@ -783,7 +783,7 @@ static int connect_fts_in_prio(struct mlx5_core_dev *dev, fs_for_each_ft(iter, prio) { i++; - err = root->cmds->modify_flow_table(dev, iter, ft); + err = root->cmds->modify_flow_table(root, iter, ft); if (err) { mlx5_core_warn(dev, "Failed to modify flow table %d\n", iter->id); @@ -819,7 +819,7 @@ static int update_root_ft_create(struct mlx5_flow_table *ft, struct fs_prio struct mlx5_flow_root_namespace *root = find_root(&prio->node); struct mlx5_ft_underlay_qp *uqp; int min_level = INT_MAX; - int err; + int err = 0; u32 qpn; if (root->root_ft) @@ -831,11 +831,11 @@ static int update_root_ft_create(struct mlx5_flow_table *ft, struct fs_prio if (list_empty(&root->underlay_qpns)) { /* Don't set any QPN (zero) in case QPN list is empty */ qpn = 0; - err = root->cmds->update_root_ft(root->dev, ft, qpn, false); + err = root->cmds->update_root_ft(root, ft, qpn, false); } else { list_for_each_entry(uqp, &root->underlay_qpns, list) { qpn = uqp->qpn; - err = root->cmds->update_root_ft(root->dev, ft, + err = root->cmds->update_root_ft(root, ft, qpn, false); if (err) break; @@ -871,7 +871,7 @@ static int _mlx5_modify_rule_destination(struct mlx5_flow_rule *rule, memcpy(&rule->dest_attr, dest, sizeof(*dest)); root = find_root(&ft->node); - err = root->cmds->update_fte(get_dev(&ft->node), ft, fg->id, + err = root->cmds->update_fte(root, ft, fg, modify_mask, fte); up_write_ref_node(&fte->node, false); @@ -1013,9 +1013,7 @@ static struct mlx5_flow_table *__mlx5_create_flow_table(struct mlx5_flow_namespa tree_init_node(&ft->node, del_hw_flow_table, del_sw_flow_table); log_table_sz = ft->max_fte ? ilog2(ft->max_fte) : 0; next_ft = find_next_chained_ft(fs_prio); - err = root->cmds->create_flow_table(root->dev, ft->vport, ft->op_mod, - ft->type, ft->level, log_table_sz, - next_ft, &ft->id, ft->flags); + err = root->cmds->create_flow_table(root, ft, log_table_sz, next_ft); if (err) goto free_ft; @@ -1032,7 +1030,7 @@ static struct mlx5_flow_table *__mlx5_create_flow_table(struct mlx5_flow_namespa trace_mlx5_fs_add_ft(ft); return ft; destroy_ft: - root->cmds->destroy_flow_table(root->dev, ft); + root->cmds->destroy_flow_table(root, ft); free_ft: kfree(ft); unlock_root: @@ -1114,7 +1112,6 @@ struct mlx5_flow_group *mlx5_create_flow_group(struct mlx5_flow_table *ft, start_flow_index); int end_index = MLX5_GET(create_flow_group_in, fg_in, end_flow_index); - struct mlx5_core_dev *dev = get_dev(&ft->node); struct mlx5_flow_group *fg; int err; @@ -1129,7 +1126,7 @@ struct mlx5_flow_group *mlx5_create_flow_group(struct mlx5_flow_table *ft, if (IS_ERR(fg)) return fg; - err = root->cmds->create_flow_group(dev, ft, fg_in, &fg->id); + err = root->cmds->create_flow_group(root, ft, fg_in, fg); if (err) { tree_put_node(&fg->node, false); return ERR_PTR(err); @@ -1269,11 +1266,9 @@ add_rule_fte(struct fs_fte *fte, fs_get_obj(ft, fg->node.parent); root = find_root(&fg->node); if (!(fte->status & FS_FTE_STATUS_EXISTING)) - err = root->cmds->create_fte(get_dev(&ft->node), - ft, fg, fte); + err = root->cmds->create_fte(root, ft, fg, fte); else - err = root->cmds->update_fte(get_dev(&ft->node), ft, fg->id, - modify_mask, fte); + err = root->cmds->update_fte(root, ft, fg, modify_mask, fte); if (err) goto free_handle; @@ -1339,7 +1334,6 @@ static int create_auto_flow_group(struct mlx5_flow_table *ft, struct mlx5_flow_group *fg) { struct mlx5_flow_root_namespace *root = find_root(&ft->node); - struct mlx5_core_dev *dev = get_dev(&ft->node); int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in); void *match_criteria_addr; u8 src_esw_owner_mask_on; @@ -1369,7 +1363,7 @@ static int create_auto_flow_group(struct mlx5_flow_table *ft, memcpy(match_criteria_addr, fg->mask.match_criteria, sizeof(fg->mask.match_criteria)); - err = root->cmds->create_flow_group(dev, ft, in, &fg->id); + err = root->cmds->create_flow_group(root, ft, in, fg); if (!err) { fg->node.active = true; trace_mlx5_fs_add_fg(fg); @@ -1941,12 +1935,12 @@ static int update_root_ft_destroy(struct mlx5_flow_table *ft) if (list_empty(&root->underlay_qpns)) { /* Don't set any QPN (zero) in case QPN list is empty */ qpn = 0; - err = root->cmds->update_root_ft(root->dev, new_root_ft, + err = root->cmds->update_root_ft(root, new_root_ft, qpn, false); } else { list_for_each_entry(uqp, &root->underlay_qpns, list) { qpn = uqp->qpn; - err = root->cmds->update_root_ft(root->dev, + err = root->cmds->update_root_ft(root, new_root_ft, qpn, false); if (err) @@ -2060,6 +2054,10 @@ struct mlx5_flow_namespace *mlx5_get_flow_namespace(struct mlx5_core_dev *dev, if (steering->sniffer_tx_root_ns) return &steering->sniffer_tx_root_ns->ns; return NULL; + case MLX5_FLOW_NAMESPACE_RDMA_RX: + if (steering->rdma_rx_root_ns) + return &steering->rdma_rx_root_ns->ns; + return NULL; default: break; } @@ -2456,6 +2454,7 @@ void mlx5_cleanup_fs(struct mlx5_core_dev *dev) steering->fdb_sub_ns = NULL; cleanup_root_ns(steering->sniffer_rx_root_ns); cleanup_root_ns(steering->sniffer_tx_root_ns); + cleanup_root_ns(steering->rdma_rx_root_ns); cleanup_root_ns(steering->egress_root_ns); mlx5_cleanup_fc_stats(dev); kmem_cache_destroy(steering->ftes_cache); @@ -2497,6 +2496,25 @@ static int init_sniffer_rx_root_ns(struct mlx5_flow_steering *steering) return 0; } +static int init_rdma_rx_root_ns(struct mlx5_flow_steering *steering) +{ + struct fs_prio *prio; + + steering->rdma_rx_root_ns = create_root_ns(steering, FS_FT_RDMA_RX); + if (!steering->rdma_rx_root_ns) + return -ENOMEM; + + steering->rdma_rx_root_ns->def_miss_action = + MLX5_FLOW_TABLE_MISS_ACTION_SWITCH_DOMAIN; + + /* Create single prio */ + prio = fs_create_prio(&steering->rdma_rx_root_ns->ns, 0, 1); + if (IS_ERR(prio)) { + cleanup_root_ns(steering->rdma_rx_root_ns); + return PTR_ERR(prio); + } + return 0; +} static int init_fdb_root_ns(struct mlx5_flow_steering *steering) { struct mlx5_flow_namespace *ns; @@ -2516,8 +2534,16 @@ static int init_fdb_root_ns(struct mlx5_flow_steering *steering) if (!steering->fdb_sub_ns) return -ENOMEM; + maj_prio = fs_create_prio(&steering->fdb_root_ns->ns, FDB_BYPASS_PATH, + 1); + if (IS_ERR(maj_prio)) { + err = PTR_ERR(maj_prio); + goto out_err; + } + levels = 2 * FDB_MAX_PRIO * (FDB_MAX_CHAIN + 1); - maj_prio = fs_create_prio_chained(&steering->fdb_root_ns->ns, 0, + maj_prio = fs_create_prio_chained(&steering->fdb_root_ns->ns, + FDB_FAST_PATH, levels); if (IS_ERR(maj_prio)) { err = PTR_ERR(maj_prio); @@ -2542,7 +2568,7 @@ static int init_fdb_root_ns(struct mlx5_flow_steering *steering) steering->fdb_sub_ns[chain] = ns; } - maj_prio = fs_create_prio(&steering->fdb_root_ns->ns, 1, 1); + maj_prio = fs_create_prio(&steering->fdb_root_ns->ns, FDB_SLOW_PATH, 1); if (IS_ERR(maj_prio)) { err = PTR_ERR(maj_prio); goto out_err; @@ -2725,6 +2751,13 @@ int mlx5_init_fs(struct mlx5_core_dev *dev) goto err; } + if (MLX5_CAP_FLOWTABLE_RDMA_RX(dev, ft_support) && + MLX5_CAP_FLOWTABLE_RDMA_RX(dev, table_miss_action_domain)) { + err = init_rdma_rx_root_ns(steering); + if (err) + goto err; + } + if (MLX5_IPSEC_DEV(dev) || MLX5_CAP_FLOWTABLE_NIC_TX(dev, ft_support)) { err = init_egress_root_ns(steering); if (err) @@ -2754,7 +2787,7 @@ int mlx5_fs_add_rx_underlay_qpn(struct mlx5_core_dev *dev, u32 underlay_qpn) goto update_ft_fail; } - err = root->cmds->update_root_ft(dev, root->root_ft, underlay_qpn, + err = root->cmds->update_root_ft(root, root->root_ft, underlay_qpn, false); if (err) { mlx5_core_warn(dev, "Failed adding underlay QPN (%u) to root FT err(%d)\n", @@ -2798,7 +2831,7 @@ int mlx5_fs_remove_rx_underlay_qpn(struct mlx5_core_dev *dev, u32 underlay_qpn) goto out; } - err = root->cmds->update_root_ft(dev, root->root_ft, underlay_qpn, + err = root->cmds->update_root_ft(root, root->root_ft, underlay_qpn, true); if (err) mlx5_core_warn(dev, "Failed removing underlay QPN (%u) from root FT err(%d)\n", diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.h b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.h index 87de0e4d9124e0bd21a53b41534c3add512683aa..a08c3d09a50f0cc80423ed594a601f52e55a0381 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.h @@ -67,6 +67,7 @@ enum fs_flow_table_type { FS_FT_FDB = 0X4, FS_FT_SNIFFER_RX = 0X5, FS_FT_SNIFFER_TX = 0X6, + FS_FT_RDMA_RX = 0X7, FS_FT_MAX_TYPE = FS_FT_SNIFFER_TX, }; @@ -90,6 +91,7 @@ struct mlx5_flow_steering { struct mlx5_flow_root_namespace **esw_ingress_root_ns; struct mlx5_flow_root_namespace *sniffer_tx_root_ns; struct mlx5_flow_root_namespace *sniffer_rx_root_ns; + struct mlx5_flow_root_namespace *rdma_rx_root_ns; struct mlx5_flow_root_namespace *egress_root_ns; }; @@ -150,7 +152,7 @@ struct mlx5_ft_underlay_qp { u32 qpn; }; -#define MLX5_FTE_MATCH_PARAM_RESERVED reserved_at_800 +#define MLX5_FTE_MATCH_PARAM_RESERVED reserved_at_a00 /* Calculate the fte_match_param length and without the reserved length. * Make sure the reserved field is the last. */ @@ -216,6 +218,7 @@ struct mlx5_flow_root_namespace { struct mutex chain_lock; struct list_head underlay_qpns; const struct mlx5_flow_cmds *cmds; + enum mlx5_flow_table_miss_action def_miss_action; }; int mlx5_init_fc_stats(struct mlx5_core_dev *dev); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c index cb9fa3430c5358678a6a7b719ed0d22e8d2eb15e..a2656f4008d9162b87ff2fc1f5639fcf8d3d4dda 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/health.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c @@ -152,11 +152,11 @@ static void health_recover(struct work_struct *work) nic_state = mlx5_get_nic_state(dev); if (nic_state == MLX5_NIC_IFC_INVALID) { - dev_err(&dev->pdev->dev, "health recovery flow aborted since the nic state is invalid\n"); + mlx5_core_err(dev, "health recovery flow aborted since the nic state is invalid\n"); return; } - dev_err(&dev->pdev->dev, "starting health recovery flow\n"); + mlx5_core_err(dev, "starting health recovery flow\n"); mlx5_recover_device(dev); } @@ -180,8 +180,8 @@ static void health_care(struct work_struct *work) if (!test_bit(MLX5_DROP_NEW_RECOVERY_WORK, &health->flags)) schedule_delayed_work(&health->recover_work, recover_delay); else - dev_err(&dev->pdev->dev, - "new health works are not permitted at this stage\n"); + mlx5_core_err(dev, + "new health works are not permitted at this stage\n"); spin_unlock_irqrestore(&health->wq_lock, flags); } @@ -228,18 +228,22 @@ static void print_health_info(struct mlx5_core_dev *dev) return; for (i = 0; i < ARRAY_SIZE(h->assert_var); i++) - dev_err(&dev->pdev->dev, "assert_var[%d] 0x%08x\n", i, ioread32be(h->assert_var + i)); + mlx5_core_err(dev, "assert_var[%d] 0x%08x\n", i, + ioread32be(h->assert_var + i)); - dev_err(&dev->pdev->dev, "assert_exit_ptr 0x%08x\n", ioread32be(&h->assert_exit_ptr)); - dev_err(&dev->pdev->dev, "assert_callra 0x%08x\n", ioread32be(&h->assert_callra)); + mlx5_core_err(dev, "assert_exit_ptr 0x%08x\n", + ioread32be(&h->assert_exit_ptr)); + mlx5_core_err(dev, "assert_callra 0x%08x\n", + ioread32be(&h->assert_callra)); sprintf(fw_str, "%d.%d.%d", fw_rev_maj(dev), fw_rev_min(dev), fw_rev_sub(dev)); - dev_err(&dev->pdev->dev, "fw_ver %s\n", fw_str); - dev_err(&dev->pdev->dev, "hw_id 0x%08x\n", ioread32be(&h->hw_id)); - dev_err(&dev->pdev->dev, "irisc_index %d\n", ioread8(&h->irisc_index)); - dev_err(&dev->pdev->dev, "synd 0x%x: %s\n", ioread8(&h->synd), hsynd_str(ioread8(&h->synd))); - dev_err(&dev->pdev->dev, "ext_synd 0x%04x\n", ioread16be(&h->ext_synd)); + mlx5_core_err(dev, "fw_ver %s\n", fw_str); + mlx5_core_err(dev, "hw_id 0x%08x\n", ioread32be(&h->hw_id)); + mlx5_core_err(dev, "irisc_index %d\n", ioread8(&h->irisc_index)); + mlx5_core_err(dev, "synd 0x%x: %s\n", ioread8(&h->synd), + hsynd_str(ioread8(&h->synd))); + mlx5_core_err(dev, "ext_synd 0x%04x\n", ioread16be(&h->ext_synd)); fw = ioread32be(&h->fw_ver); - dev_err(&dev->pdev->dev, "raw fw_ver 0x%08x\n", fw); + mlx5_core_err(dev, "raw fw_ver 0x%08x\n", fw); } static unsigned long get_next_poll_jiffies(void) @@ -262,8 +266,7 @@ void mlx5_trigger_health_work(struct mlx5_core_dev *dev) if (!test_bit(MLX5_DROP_NEW_HEALTH_WORK, &health->flags)) queue_work(health->wq, &health->work); else - dev_err(&dev->pdev->dev, - "new health works are not permitted at this stage\n"); + mlx5_core_err(dev, "new health works are not permitted at this stage\n"); spin_unlock_irqrestore(&health->wq_lock, flags); } @@ -284,7 +287,7 @@ static void poll_health(struct timer_list *t) health->prev = count; if (health->miss_counter == MAX_MISSES) { - dev_err(&dev->pdev->dev, "device's health compromised - reached miss count\n"); + mlx5_core_err(dev, "device's health compromised - reached miss count\n"); print_health_info(dev); } @@ -352,6 +355,13 @@ void mlx5_drain_health_recovery(struct mlx5_core_dev *dev) cancel_delayed_work_sync(&dev->priv.health.recover_work); } +void mlx5_health_flush(struct mlx5_core_dev *dev) +{ + struct mlx5_core_health *health = &dev->priv.health; + + flush_workqueue(health->wq); +} + void mlx5_health_cleanup(struct mlx5_core_dev *dev) { struct mlx5_core_health *health = &dev->priv.health; @@ -370,7 +380,7 @@ int mlx5_health_init(struct mlx5_core_dev *dev) return -ENOMEM; strcpy(name, "mlx5_health"); - strcat(name, dev_name(&dev->pdev->dev)); + strcat(name, dev_name(dev->device)); health->wq = create_singlethread_workqueue(name); kfree(name); if (!health->wq) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c b/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c index 4eac42555c7dccececfb9805fb984beb520fbb7b..ada1b7c0e0b8969a0a3397f89012c67aa8696bf2 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c @@ -68,6 +68,7 @@ static void mlx5i_build_nic_params(struct mlx5_core_dev *mdev, params->lro_en = false; params->hard_mtu = MLX5_IB_GRH_BYTES + MLX5_IPOIB_HARD_LEN; + params->tunneled_offload_en = false; } /* Called directly after IPoIB netdevice was created to initialize SW structs */ @@ -77,15 +78,14 @@ int mlx5i_init(struct mlx5_core_dev *mdev, void *ppriv) { struct mlx5e_priv *priv = mlx5i_epriv(netdev); - u16 max_mtu; int err; err = mlx5e_netdev_init(netdev, priv, mdev, profile, ppriv); if (err) return err; - mlx5_query_port_max_mtu(mdev, &max_mtu, 1); - netdev->mtu = max_mtu; + mlx5e_set_netdev_mtu_boundaries(priv); + netdev->mtu = netdev->max_mtu; mlx5e_build_nic_params(mdev, &priv->rss_params, &priv->channels.params, mlx5e_get_netdev_max_channels(netdev), diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag_mp.c b/drivers/net/ethernet/mellanox/mlx5/core/lag_mp.c index 5633f8572800e63efe51102365c40537c443a98c..8212bfd057338ac81f6e5f6add603412586dc434 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/lag_mp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/lag_mp.c @@ -122,7 +122,7 @@ static void mlx5_lag_fib_route_event(struct mlx5_lag *ldev, /* Handle add/replace event */ if (fi->fib_nhs == 1) { if (__mlx5_lag_is_active(ldev)) { - struct net_device *nh_dev = fi->fib_nh[0].nh_dev; + struct net_device *nh_dev = fi->fib_nh[0].fib_nh_dev; int i = mlx5_lag_dev_get_netdev_idx(ldev, nh_dev); mlx5_lag_set_port_affinity(ldev, ++i); @@ -134,10 +134,10 @@ static void mlx5_lag_fib_route_event(struct mlx5_lag *ldev, return; /* Verify next hops are ports of the same hca */ - if (!(fi->fib_nh[0].nh_dev == ldev->pf[0].netdev && - fi->fib_nh[1].nh_dev == ldev->pf[1].netdev) && - !(fi->fib_nh[0].nh_dev == ldev->pf[1].netdev && - fi->fib_nh[1].nh_dev == ldev->pf[0].netdev)) { + if (!(fi->fib_nh[0].fib_nh_dev == ldev->pf[0].netdev && + fi->fib_nh[1].fib_nh_dev == ldev->pf[1].netdev) && + !(fi->fib_nh[0].fib_nh_dev == ldev->pf[1].netdev && + fi->fib_nh[1].fib_nh_dev == ldev->pf[0].netdev)) { mlx5_core_warn(ldev->pf[0].dev, "Multipath offload require two ports of the same HCA\n"); return; } @@ -167,7 +167,7 @@ static void mlx5_lag_fib_nexthop_event(struct mlx5_lag *ldev, /* nh added/removed */ if (event == FIB_EVENT_NH_DEL) { - int i = mlx5_lag_dev_get_netdev_idx(ldev, fib_nh->nh_dev); + int i = mlx5_lag_dev_get_netdev_idx(ldev, fib_nh->fib_nh_dev); if (i >= 0) { i = (i + 1) % 2 + 1; /* peer port */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/port_tun.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/port_tun.c index 40f4a19b1ce16dbd15bf4b5887accd9712a94141..be69c1d7941aeb015646d0372d417a694b767980 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/lib/port_tun.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/port_tun.c @@ -80,10 +80,8 @@ void mlx5_init_port_tun_entropy(struct mlx5_tun_entropy *tun_entropy, mlx5_query_port_tun_entropy(mdev, &entropy_flags); tun_entropy->num_enabling_entries = 0; tun_entropy->num_disabling_entries = 0; - tun_entropy->enabled = entropy_flags.calc_enabled; - tun_entropy->enabled = - (entropy_flags.calc_supported) ? - entropy_flags.calc_enabled : true; + tun_entropy->enabled = entropy_flags.calc_supported ? + entropy_flags.calc_enabled : true; } static int mlx5_set_entropy(struct mlx5_tun_entropy *tun_entropy, diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/vxlan.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/vxlan.c index 9a8fd762167b3d5cdc66202395819e16b9bb1a99..b9d4f4e19ff93d5c7438fe75a990dce38a5912b9 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/lib/vxlan.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/vxlan.c @@ -33,6 +33,7 @@ #include #include #include +#include #include "mlx5_core.h" #include "vxlan.h" @@ -204,8 +205,8 @@ struct mlx5_vxlan *mlx5_vxlan_create(struct mlx5_core_dev *mdev) spin_lock_init(&vxlan->lock); hash_init(vxlan->htable); - /* Hardware adds 4789 by default */ - mlx5_vxlan_add_port(vxlan, 4789); + /* Hardware adds 4789 (IANA_VXLAN_UDP_PORT) by default */ + mlx5_vxlan_add_port(vxlan, IANA_VXLAN_UDP_PORT); return vxlan; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c index 76716419370df9738384675c3dc36e5d0ff6cf27..61fa1d162d288d12c7e9004838e175ff36b7176c 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c @@ -567,24 +567,23 @@ query_ex: static int set_hca_cap(struct mlx5_core_dev *dev) { - struct pci_dev *pdev = dev->pdev; int err; err = handle_hca_cap(dev); if (err) { - dev_err(&pdev->dev, "handle_hca_cap failed\n"); + mlx5_core_err(dev, "handle_hca_cap failed\n"); goto out; } err = handle_hca_cap_atomic(dev); if (err) { - dev_err(&pdev->dev, "handle_hca_cap_atomic failed\n"); + mlx5_core_err(dev, "handle_hca_cap_atomic failed\n"); goto out; } err = handle_hca_cap_odp(dev); if (err) { - dev_err(&pdev->dev, "handle_hca_cap_odp failed\n"); + mlx5_core_err(dev, "handle_hca_cap_odp failed\n"); goto out; } @@ -716,36 +715,28 @@ static int mlx5_core_set_issi(struct mlx5_core_dev *dev) return -EOPNOTSUPP; } -static int mlx5_pci_init(struct mlx5_core_dev *dev, struct mlx5_priv *priv) +static int mlx5_pci_init(struct mlx5_core_dev *dev, struct pci_dev *pdev, + const struct pci_device_id *id) { - struct pci_dev *pdev = dev->pdev; + struct mlx5_priv *priv = &dev->priv; int err = 0; - pci_set_drvdata(dev->pdev, dev); - strncpy(priv->name, dev_name(&pdev->dev), MLX5_MAX_NAME_LEN); - priv->name[MLX5_MAX_NAME_LEN - 1] = 0; - - mutex_init(&priv->pgdir_mutex); - INIT_LIST_HEAD(&priv->pgdir_list); - spin_lock_init(&priv->mkey_lock); + priv->pci_dev_data = id->driver_data; - mutex_init(&priv->alloc_mutex); + pci_set_drvdata(dev->pdev, dev); + dev->bar_addr = pci_resource_start(pdev, 0); priv->numa_node = dev_to_node(&dev->pdev->dev); - if (mlx5_debugfs_root) - priv->dbg_root = - debugfs_create_dir(pci_name(pdev), mlx5_debugfs_root); - err = mlx5_pci_enable_device(dev); if (err) { - dev_err(&pdev->dev, "Cannot enable PCI device, aborting\n"); - goto err_dbg; + mlx5_core_err(dev, "Cannot enable PCI device, aborting\n"); + return err; } err = request_bar(pdev); if (err) { - dev_err(&pdev->dev, "error requesting BARs, aborting\n"); + mlx5_core_err(dev, "error requesting BARs, aborting\n"); goto err_disable; } @@ -753,7 +744,7 @@ static int mlx5_pci_init(struct mlx5_core_dev *dev, struct mlx5_priv *priv) err = set_dma_caps(pdev); if (err) { - dev_err(&pdev->dev, "Failed setting DMA capabilities mask, aborting\n"); + mlx5_core_err(dev, "Failed setting DMA capabilities mask, aborting\n"); goto err_clr_master; } @@ -762,11 +753,11 @@ static int mlx5_pci_init(struct mlx5_core_dev *dev, struct mlx5_priv *priv) pci_enable_atomic_ops_to_root(pdev, PCI_EXP_DEVCAP2_ATOMIC_COMP128)) mlx5_core_dbg(dev, "Enabling pci atomics failed\n"); - dev->iseg_base = pci_resource_start(dev->pdev, 0); + dev->iseg_base = dev->bar_addr; dev->iseg = ioremap(dev->iseg_base, sizeof(*dev->iseg)); if (!dev->iseg) { err = -ENOMEM; - dev_err(&pdev->dev, "Failed mapping initialization segment, aborting\n"); + mlx5_core_err(dev, "Failed mapping initialization segment, aborting\n"); goto err_clr_master; } @@ -777,52 +768,47 @@ err_clr_master: release_bar(dev->pdev); err_disable: mlx5_pci_disable_device(dev); - -err_dbg: - debugfs_remove(priv->dbg_root); return err; } -static void mlx5_pci_close(struct mlx5_core_dev *dev, struct mlx5_priv *priv) +static void mlx5_pci_close(struct mlx5_core_dev *dev) { iounmap(dev->iseg); pci_clear_master(dev->pdev); release_bar(dev->pdev); mlx5_pci_disable_device(dev); - debugfs_remove_recursive(priv->dbg_root); } -static int mlx5_init_once(struct mlx5_core_dev *dev, struct mlx5_priv *priv) +static int mlx5_init_once(struct mlx5_core_dev *dev) { - struct pci_dev *pdev = dev->pdev; int err; - priv->devcom = mlx5_devcom_register_device(dev); - if (IS_ERR(priv->devcom)) - dev_err(&pdev->dev, "failed to register with devcom (0x%p)\n", - priv->devcom); + dev->priv.devcom = mlx5_devcom_register_device(dev); + if (IS_ERR(dev->priv.devcom)) + mlx5_core_err(dev, "failed to register with devcom (0x%p)\n", + dev->priv.devcom); err = mlx5_query_board_id(dev); if (err) { - dev_err(&pdev->dev, "query board id failed\n"); + mlx5_core_err(dev, "query board id failed\n"); goto err_devcom; } err = mlx5_eq_table_init(dev); if (err) { - dev_err(&pdev->dev, "failed to initialize eq\n"); + mlx5_core_err(dev, "failed to initialize eq\n"); goto err_devcom; } err = mlx5_events_init(dev); if (err) { - dev_err(&pdev->dev, "failed to initialize events\n"); + mlx5_core_err(dev, "failed to initialize events\n"); goto err_eq_cleanup; } err = mlx5_cq_debugfs_init(dev); if (err) { - dev_err(&pdev->dev, "failed to initialize cq debugfs\n"); + mlx5_core_err(dev, "failed to initialize cq debugfs\n"); goto err_events_cleanup; } @@ -838,31 +824,31 @@ static int mlx5_init_once(struct mlx5_core_dev *dev, struct mlx5_priv *priv) err = mlx5_init_rl_table(dev); if (err) { - dev_err(&pdev->dev, "Failed to init rate limiting\n"); + mlx5_core_err(dev, "Failed to init rate limiting\n"); goto err_tables_cleanup; } err = mlx5_mpfs_init(dev); if (err) { - dev_err(&pdev->dev, "Failed to init l2 table %d\n", err); + mlx5_core_err(dev, "Failed to init l2 table %d\n", err); goto err_rl_cleanup; } err = mlx5_eswitch_init(dev); if (err) { - dev_err(&pdev->dev, "Failed to init eswitch %d\n", err); + mlx5_core_err(dev, "Failed to init eswitch %d\n", err); goto err_mpfs_cleanup; } err = mlx5_sriov_init(dev); if (err) { - dev_err(&pdev->dev, "Failed to init sriov %d\n", err); + mlx5_core_err(dev, "Failed to init sriov %d\n", err); goto err_eswitch_cleanup; } err = mlx5_fpga_init(dev); if (err) { - dev_err(&pdev->dev, "Failed to init fpga device %d\n", err); + mlx5_core_err(dev, "Failed to init fpga device %d\n", err); goto err_sriov_cleanup; } @@ -912,93 +898,78 @@ static void mlx5_cleanup_once(struct mlx5_core_dev *dev) mlx5_devcom_unregister_device(dev->priv.devcom); } -static int mlx5_load_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv, - bool boot) +static int mlx5_function_setup(struct mlx5_core_dev *dev, bool boot) { - struct pci_dev *pdev = dev->pdev; int err; - dev->caps.embedded_cpu = mlx5_read_embedded_cpu(dev); - mutex_lock(&dev->intf_state_mutex); - if (test_bit(MLX5_INTERFACE_STATE_UP, &dev->intf_state)) { - dev_warn(&dev->pdev->dev, "%s: interface is up, NOP\n", - __func__); - goto out; - } - - dev_info(&pdev->dev, "firmware version: %d.%d.%d\n", fw_rev_maj(dev), - fw_rev_min(dev), fw_rev_sub(dev)); + mlx5_core_info(dev, "firmware version: %d.%d.%d\n", fw_rev_maj(dev), + fw_rev_min(dev), fw_rev_sub(dev)); /* Only PFs hold the relevant PCIe information for this query */ if (mlx5_core_is_pf(dev)) pcie_print_link_status(dev->pdev); - /* on load removing any previous indication of internal error, device is - * up - */ - dev->state = MLX5_DEVICE_STATE_UP; - /* wait for firmware to accept initialization segments configurations */ err = wait_fw_init(dev, FW_PRE_INIT_TIMEOUT_MILI); if (err) { - dev_err(&dev->pdev->dev, "Firmware over %d MS in pre-initializing state, aborting\n", - FW_PRE_INIT_TIMEOUT_MILI); - goto out_err; + mlx5_core_err(dev, "Firmware over %d MS in pre-initializing state, aborting\n", + FW_PRE_INIT_TIMEOUT_MILI); + return err; } err = mlx5_cmd_init(dev); if (err) { - dev_err(&pdev->dev, "Failed initializing command interface, aborting\n"); - goto out_err; + mlx5_core_err(dev, "Failed initializing command interface, aborting\n"); + return err; } err = wait_fw_init(dev, FW_INIT_TIMEOUT_MILI); if (err) { - dev_err(&dev->pdev->dev, "Firmware over %d MS in initializing state, aborting\n", - FW_INIT_TIMEOUT_MILI); + mlx5_core_err(dev, "Firmware over %d MS in initializing state, aborting\n", + FW_INIT_TIMEOUT_MILI); goto err_cmd_cleanup; } err = mlx5_core_enable_hca(dev, 0); if (err) { - dev_err(&pdev->dev, "enable hca failed\n"); + mlx5_core_err(dev, "enable hca failed\n"); goto err_cmd_cleanup; } err = mlx5_core_set_issi(dev); if (err) { - dev_err(&pdev->dev, "failed to set issi\n"); + mlx5_core_err(dev, "failed to set issi\n"); goto err_disable_hca; } err = mlx5_satisfy_startup_pages(dev, 1); if (err) { - dev_err(&pdev->dev, "failed to allocate boot pages\n"); + mlx5_core_err(dev, "failed to allocate boot pages\n"); goto err_disable_hca; } err = set_hca_ctrl(dev); if (err) { - dev_err(&pdev->dev, "set_hca_ctrl failed\n"); + mlx5_core_err(dev, "set_hca_ctrl failed\n"); goto reclaim_boot_pages; } err = set_hca_cap(dev); if (err) { - dev_err(&pdev->dev, "set_hca_cap failed\n"); + mlx5_core_err(dev, "set_hca_cap failed\n"); goto reclaim_boot_pages; } err = mlx5_satisfy_startup_pages(dev, 0); if (err) { - dev_err(&pdev->dev, "failed to allocate init pages\n"); + mlx5_core_err(dev, "failed to allocate init pages\n"); goto reclaim_boot_pages; } err = mlx5_cmd_init_hca(dev, sw_owner_id); if (err) { - dev_err(&pdev->dev, "init hca failed\n"); + mlx5_core_err(dev, "init hca failed\n"); goto reclaim_boot_pages; } @@ -1008,23 +979,50 @@ static int mlx5_load_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv, err = mlx5_query_hca_caps(dev); if (err) { - dev_err(&pdev->dev, "query hca failed\n"); - goto err_stop_poll; + mlx5_core_err(dev, "query hca failed\n"); + goto stop_health; } - if (boot) { - err = mlx5_init_once(dev, priv); - if (err) { - dev_err(&pdev->dev, "sw objs init failed\n"); - goto err_stop_poll; - } + return 0; + +stop_health: + mlx5_stop_health_poll(dev, boot); +reclaim_boot_pages: + mlx5_reclaim_startup_pages(dev); +err_disable_hca: + mlx5_core_disable_hca(dev, 0); +err_cmd_cleanup: + mlx5_cmd_cleanup(dev); + + return err; +} + +static int mlx5_function_teardown(struct mlx5_core_dev *dev, bool boot) +{ + int err; + + mlx5_stop_health_poll(dev, boot); + err = mlx5_cmd_teardown_hca(dev); + if (err) { + mlx5_core_err(dev, "tear_down_hca failed, skip cleanup\n"); + return err; } + mlx5_reclaim_startup_pages(dev); + mlx5_core_disable_hca(dev, 0); + mlx5_cmd_cleanup(dev); + + return 0; +} + +static int mlx5_load(struct mlx5_core_dev *dev) +{ + int err; dev->priv.uar = mlx5_get_uars_page(dev); if (IS_ERR(dev->priv.uar)) { - dev_err(&pdev->dev, "Failed allocating uar, aborting\n"); + mlx5_core_err(dev, "Failed allocating uar, aborting\n"); err = PTR_ERR(dev->priv.uar); - goto err_get_uars; + return err; } mlx5_events_start(dev); @@ -1032,132 +1030,155 @@ static int mlx5_load_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv, err = mlx5_eq_table_create(dev); if (err) { - dev_err(&pdev->dev, "Failed to create EQs\n"); + mlx5_core_err(dev, "Failed to create EQs\n"); goto err_eq_table; } err = mlx5_fw_tracer_init(dev->tracer); if (err) { - dev_err(&pdev->dev, "Failed to init FW tracer\n"); + mlx5_core_err(dev, "Failed to init FW tracer\n"); goto err_fw_tracer; } err = mlx5_fpga_device_start(dev); if (err) { - dev_err(&pdev->dev, "fpga device start failed %d\n", err); + mlx5_core_err(dev, "fpga device start failed %d\n", err); goto err_fpga_start; } err = mlx5_accel_ipsec_init(dev); if (err) { - dev_err(&pdev->dev, "IPSec device start failed %d\n", err); + mlx5_core_err(dev, "IPSec device start failed %d\n", err); goto err_ipsec_start; } err = mlx5_accel_tls_init(dev); if (err) { - dev_err(&pdev->dev, "TLS device start failed %d\n", err); + mlx5_core_err(dev, "TLS device start failed %d\n", err); goto err_tls_start; } err = mlx5_init_fs(dev); if (err) { - dev_err(&pdev->dev, "Failed to init flow steering\n"); + mlx5_core_err(dev, "Failed to init flow steering\n"); goto err_fs; } err = mlx5_core_set_hca_defaults(dev); if (err) { - dev_err(&pdev->dev, "Failed to set hca defaults\n"); + mlx5_core_err(dev, "Failed to set hca defaults\n"); goto err_fs; } err = mlx5_sriov_attach(dev); if (err) { - dev_err(&pdev->dev, "sriov init failed %d\n", err); + mlx5_core_err(dev, "sriov init failed %d\n", err); goto err_sriov; } err = mlx5_ec_init(dev); if (err) { - dev_err(&pdev->dev, "Failed to init embedded CPU\n"); + mlx5_core_err(dev, "Failed to init embedded CPU\n"); goto err_ec; } - if (mlx5_device_registered(dev)) { - mlx5_attach_device(dev); - } else { - err = mlx5_register_device(dev); - if (err) { - dev_err(&pdev->dev, "mlx5_register_device failed %d\n", err); - goto err_reg_dev; - } - } - - set_bit(MLX5_INTERFACE_STATE_UP, &dev->intf_state); -out: - mutex_unlock(&dev->intf_state_mutex); - return 0; -err_reg_dev: - mlx5_ec_cleanup(dev); - err_ec: mlx5_sriov_detach(dev); - err_sriov: mlx5_cleanup_fs(dev); - err_fs: mlx5_accel_tls_cleanup(dev); - err_tls_start: mlx5_accel_ipsec_cleanup(dev); - err_ipsec_start: mlx5_fpga_device_stop(dev); - err_fpga_start: mlx5_fw_tracer_cleanup(dev->tracer); - err_fw_tracer: mlx5_eq_table_destroy(dev); - err_eq_table: mlx5_pagealloc_stop(dev); mlx5_events_stop(dev); - mlx5_put_uars_page(dev, priv->uar); + mlx5_put_uars_page(dev, dev->priv.uar); + return err; +} + +static void mlx5_unload(struct mlx5_core_dev *dev) +{ + mlx5_ec_cleanup(dev); + mlx5_sriov_detach(dev); + mlx5_cleanup_fs(dev); + mlx5_accel_ipsec_cleanup(dev); + mlx5_accel_tls_cleanup(dev); + mlx5_fpga_device_stop(dev); + mlx5_fw_tracer_cleanup(dev->tracer); + mlx5_eq_table_destroy(dev); + mlx5_pagealloc_stop(dev); + mlx5_events_stop(dev); + mlx5_put_uars_page(dev, dev->priv.uar); +} -err_get_uars: - if (boot) - mlx5_cleanup_once(dev); +static int mlx5_load_one(struct mlx5_core_dev *dev, bool boot) +{ + int err = 0; -err_stop_poll: - mlx5_stop_health_poll(dev, boot); - if (mlx5_cmd_teardown_hca(dev)) { - dev_err(&dev->pdev->dev, "tear_down_hca failed, skip cleanup\n"); - goto out_err; + dev->caps.embedded_cpu = mlx5_read_embedded_cpu(dev); + mutex_lock(&dev->intf_state_mutex); + if (test_bit(MLX5_INTERFACE_STATE_UP, &dev->intf_state)) { + mlx5_core_warn(dev, "interface is up, NOP\n"); + goto out; } + /* remove any previous indication of internal error */ + dev->state = MLX5_DEVICE_STATE_UP; -reclaim_boot_pages: - mlx5_reclaim_startup_pages(dev); + err = mlx5_function_setup(dev, boot); + if (err) + goto out; -err_disable_hca: - mlx5_core_disable_hca(dev, 0); + if (boot) { + err = mlx5_init_once(dev); + if (err) { + mlx5_core_err(dev, "sw objs init failed\n"); + goto function_teardown; + } + } -err_cmd_cleanup: - mlx5_cmd_cleanup(dev); + err = mlx5_load(dev); + if (err) + goto err_load; -out_err: + if (mlx5_device_registered(dev)) { + mlx5_attach_device(dev); + } else { + err = mlx5_register_device(dev); + if (err) { + mlx5_core_err(dev, "register device failed %d\n", err); + goto err_reg_dev; + } + } + + set_bit(MLX5_INTERFACE_STATE_UP, &dev->intf_state); +out: + mutex_unlock(&dev->intf_state_mutex); + + return err; + +err_reg_dev: + mlx5_unload(dev); +err_load: + if (boot) + mlx5_cleanup_once(dev); +function_teardown: + mlx5_function_teardown(dev, boot); dev->state = MLX5_DEVICE_STATE_INTERNAL_ERROR; mutex_unlock(&dev->intf_state_mutex); return err; } -static int mlx5_unload_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv, - bool cleanup) +static int mlx5_unload_one(struct mlx5_core_dev *dev, bool cleanup) { int err = 0; @@ -1166,8 +1187,8 @@ static int mlx5_unload_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv, mutex_lock(&dev->intf_state_mutex); if (!test_bit(MLX5_INTERFACE_STATE_UP, &dev->intf_state)) { - dev_warn(&dev->pdev->dev, "%s: interface is down, NOP\n", - __func__); + mlx5_core_warn(dev, "%s: interface is down, NOP\n", + __func__); if (cleanup) mlx5_cleanup_once(dev); goto out; @@ -1178,30 +1199,12 @@ static int mlx5_unload_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv, if (mlx5_device_registered(dev)) mlx5_detach_device(dev); - mlx5_ec_cleanup(dev); - mlx5_sriov_detach(dev); - mlx5_cleanup_fs(dev); - mlx5_accel_ipsec_cleanup(dev); - mlx5_accel_tls_cleanup(dev); - mlx5_fpga_device_stop(dev); - mlx5_fw_tracer_cleanup(dev->tracer); - mlx5_eq_table_destroy(dev); - mlx5_pagealloc_stop(dev); - mlx5_events_stop(dev); - mlx5_put_uars_page(dev, priv->uar); + mlx5_unload(dev); + if (cleanup) mlx5_cleanup_once(dev); - mlx5_stop_health_poll(dev, cleanup); - - err = mlx5_cmd_teardown_hca(dev); - if (err) { - dev_err(&dev->pdev->dev, "tear_down_hca failed, skip cleanup\n"); - goto out; - } - mlx5_reclaim_startup_pages(dev); - mlx5_core_disable_hca(dev, 0); - mlx5_cmd_cleanup(dev); + mlx5_function_teardown(dev, cleanup); out: mutex_unlock(&dev->intf_state_mutex); return err; @@ -1218,29 +1221,12 @@ static const struct devlink_ops mlx5_devlink_ops = { #endif }; -#define MLX5_IB_MOD "mlx5_ib" -static int init_one(struct pci_dev *pdev, - const struct pci_device_id *id) +static int mlx5_mdev_init(struct mlx5_core_dev *dev, int profile_idx) { - struct mlx5_core_dev *dev; - struct devlink *devlink; - struct mlx5_priv *priv; + struct mlx5_priv *priv = &dev->priv; int err; - devlink = devlink_alloc(&mlx5_devlink_ops, sizeof(*dev)); - if (!devlink) { - dev_err(&pdev->dev, "kzalloc failed\n"); - return -ENOMEM; - } - - dev = devlink_priv(devlink); - priv = &dev->priv; - priv->pci_dev_data = id->driver_data; - - pci_set_drvdata(pdev, dev); - - dev->pdev = pdev; - dev->profile = &profile[prof_sel]; + dev->profile = &profile[profile_idx]; INIT_LIST_HEAD(&priv->ctx_list); spin_lock_init(&priv->ctx_lock); @@ -1252,25 +1238,75 @@ static int init_one(struct pci_dev *pdev, INIT_LIST_HEAD(&priv->bfregs.reg_head.list); INIT_LIST_HEAD(&priv->bfregs.wc_head.list); - err = mlx5_pci_init(dev, priv); - if (err) { - dev_err(&pdev->dev, "mlx5_pci_init failed with error code %d\n", err); - goto clean_dev; + mutex_init(&priv->alloc_mutex); + mutex_init(&priv->pgdir_mutex); + INIT_LIST_HEAD(&priv->pgdir_list); + spin_lock_init(&priv->mkey_lock); + + priv->dbg_root = debugfs_create_dir(dev_name(dev->device), + mlx5_debugfs_root); + if (!priv->dbg_root) { + dev_err(dev->device, "mlx5_core: error, Cannot create debugfs dir, aborting\n"); + return -ENOMEM; } err = mlx5_health_init(dev); - if (err) { - dev_err(&pdev->dev, "mlx5_health_init failed with error code %d\n", err); - goto close_pci; - } + if (err) + goto err_health_init; err = mlx5_pagealloc_init(dev); if (err) goto err_pagealloc_init; - err = mlx5_load_one(dev, priv, true); + return 0; + +err_pagealloc_init: + mlx5_health_cleanup(dev); +err_health_init: + debugfs_remove(dev->priv.dbg_root); + + return err; +} + +static void mlx5_mdev_uninit(struct mlx5_core_dev *dev) +{ + mlx5_pagealloc_cleanup(dev); + mlx5_health_cleanup(dev); + debugfs_remove_recursive(dev->priv.dbg_root); +} + +#define MLX5_IB_MOD "mlx5_ib" +static int init_one(struct pci_dev *pdev, const struct pci_device_id *id) +{ + struct mlx5_core_dev *dev; + struct devlink *devlink; + int err; + + devlink = devlink_alloc(&mlx5_devlink_ops, sizeof(*dev)); + if (!devlink) { + dev_err(&pdev->dev, "kzalloc failed\n"); + return -ENOMEM; + } + + dev = devlink_priv(devlink); + dev->device = &pdev->dev; + dev->pdev = pdev; + + err = mlx5_mdev_init(dev, prof_sel); + if (err) + goto mdev_init_err; + + err = mlx5_pci_init(dev, pdev, id); + if (err) { + mlx5_core_err(dev, "mlx5_pci_init failed with error code %d\n", + err); + goto pci_init_err; + } + + err = mlx5_load_one(dev, true); if (err) { - dev_err(&pdev->dev, "mlx5_load_one failed with error code %d\n", err); + mlx5_core_err(dev, "mlx5_load_one failed with error code %d\n", + err); goto err_load_one; } @@ -1284,14 +1320,13 @@ static int init_one(struct pci_dev *pdev, return 0; clean_load: - mlx5_unload_one(dev, priv, true); + mlx5_unload_one(dev, true); + err_load_one: - mlx5_pagealloc_cleanup(dev); -err_pagealloc_init: - mlx5_health_cleanup(dev); -close_pci: - mlx5_pci_close(dev, priv); -clean_dev: + mlx5_pci_close(dev); +pci_init_err: + mlx5_mdev_uninit(dev); +mdev_init_err: devlink_free(devlink); return err; @@ -1301,20 +1336,18 @@ static void remove_one(struct pci_dev *pdev) { struct mlx5_core_dev *dev = pci_get_drvdata(pdev); struct devlink *devlink = priv_to_devlink(dev); - struct mlx5_priv *priv = &dev->priv; devlink_unregister(devlink); mlx5_unregister_device(dev); - if (mlx5_unload_one(dev, priv, true)) { - dev_err(&dev->pdev->dev, "mlx5_unload_one failed\n"); - mlx5_health_cleanup(dev); + if (mlx5_unload_one(dev, true)) { + mlx5_core_err(dev, "mlx5_unload_one failed\n"); + mlx5_health_flush(dev); return; } - mlx5_pagealloc_cleanup(dev); - mlx5_health_cleanup(dev); - mlx5_pci_close(dev, priv); + mlx5_pci_close(dev); + mlx5_mdev_uninit(dev); devlink_free(devlink); } @@ -1322,12 +1355,11 @@ static pci_ers_result_t mlx5_pci_err_detected(struct pci_dev *pdev, pci_channel_state_t state) { struct mlx5_core_dev *dev = pci_get_drvdata(pdev); - struct mlx5_priv *priv = &dev->priv; - dev_info(&pdev->dev, "%s was called\n", __func__); + mlx5_core_info(dev, "%s was called\n", __func__); mlx5_enter_error_state(dev, false); - mlx5_unload_one(dev, priv, false); + mlx5_unload_one(dev, false); /* In case of kernel call drain the health wq */ if (state) { mlx5_drain_health_wq(dev); @@ -1354,7 +1386,9 @@ static int wait_vital(struct pci_dev *pdev) count = ioread32be(health->health_counter); if (count && count != 0xffffffff) { if (last_count && last_count != count) { - dev_info(&pdev->dev, "Counter value 0x%x after %d iterations\n", count, i); + mlx5_core_info(dev, + "wait vital counter value 0x%x after %d iterations\n", + count, i); return 0; } last_count = count; @@ -1370,12 +1404,12 @@ static pci_ers_result_t mlx5_pci_slot_reset(struct pci_dev *pdev) struct mlx5_core_dev *dev = pci_get_drvdata(pdev); int err; - dev_info(&pdev->dev, "%s was called\n", __func__); + mlx5_core_info(dev, "%s was called\n", __func__); err = mlx5_pci_enable_device(dev); if (err) { - dev_err(&pdev->dev, "%s: mlx5_pci_enable_device failed with error code: %d\n" - , __func__, err); + mlx5_core_err(dev, "%s: mlx5_pci_enable_device failed with error code: %d\n", + __func__, err); return PCI_ERS_RESULT_DISCONNECT; } @@ -1384,7 +1418,7 @@ static pci_ers_result_t mlx5_pci_slot_reset(struct pci_dev *pdev) pci_save_state(pdev); if (wait_vital(pdev)) { - dev_err(&pdev->dev, "%s: wait_vital timed out\n", __func__); + mlx5_core_err(dev, "%s: wait_vital timed out\n", __func__); return PCI_ERS_RESULT_DISCONNECT; } @@ -1394,17 +1428,16 @@ static pci_ers_result_t mlx5_pci_slot_reset(struct pci_dev *pdev) static void mlx5_pci_resume(struct pci_dev *pdev) { struct mlx5_core_dev *dev = pci_get_drvdata(pdev); - struct mlx5_priv *priv = &dev->priv; int err; - dev_info(&pdev->dev, "%s was called\n", __func__); + mlx5_core_info(dev, "%s was called\n", __func__); - err = mlx5_load_one(dev, priv, false); + err = mlx5_load_one(dev, false); if (err) - dev_err(&pdev->dev, "%s: mlx5_load_one failed with error code: %d\n" - , __func__, err); + mlx5_core_err(dev, "%s: mlx5_load_one failed with error code: %d\n", + __func__, err); else - dev_info(&pdev->dev, "%s: device recovered\n", __func__); + mlx5_core_info(dev, "%s: device recovered\n", __func__); } static const struct pci_error_handlers mlx5_err_handler = { @@ -1466,13 +1499,12 @@ succeed: static void shutdown(struct pci_dev *pdev) { struct mlx5_core_dev *dev = pci_get_drvdata(pdev); - struct mlx5_priv *priv = &dev->priv; int err; - dev_info(&pdev->dev, "Shutdown was called\n"); + mlx5_core_info(dev, "Shutdown was called\n"); err = mlx5_try_fast_unload(dev); if (err) - mlx5_unload_one(dev, priv, false); + mlx5_unload_one(dev, false); mlx5_pci_disable_device(dev); } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h index 7b331674622c118563c1bfc83e7f1b7bc78e62a5..22e69d4813e41586baea8cbf52ff2ba0462ae42a 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h @@ -41,6 +41,7 @@ #include #include #include +#include #define DRIVER_NAME "mlx5_core" #define DRIVER_VERSION "5.0-0" @@ -48,44 +49,57 @@ extern uint mlx5_core_debug_mask; #define mlx5_core_dbg(__dev, format, ...) \ - dev_dbg(&(__dev)->pdev->dev, "%s:%d:(pid %d): " format, \ + dev_dbg((__dev)->device, "%s:%d:(pid %d): " format, \ __func__, __LINE__, current->pid, \ ##__VA_ARGS__) -#define mlx5_core_dbg_once(__dev, format, ...) \ - dev_dbg_once(&(__dev)->pdev->dev, "%s:%d:(pid %d): " format, \ - __func__, __LINE__, current->pid, \ +#define mlx5_core_dbg_once(__dev, format, ...) \ + dev_dbg_once((__dev)->device, \ + "%s:%d:(pid %d): " format, \ + __func__, __LINE__, current->pid, \ ##__VA_ARGS__) -#define mlx5_core_dbg_mask(__dev, mask, format, ...) \ -do { \ - if ((mask) & mlx5_core_debug_mask) \ - mlx5_core_dbg(__dev, format, ##__VA_ARGS__); \ +#define mlx5_core_dbg_mask(__dev, mask, format, ...) \ +do { \ + if ((mask) & mlx5_core_debug_mask) \ + mlx5_core_dbg(__dev, format, ##__VA_ARGS__); \ } while (0) -#define mlx5_core_err(__dev, format, ...) \ - dev_err(&(__dev)->pdev->dev, "%s:%d:(pid %d): " format, \ - __func__, __LINE__, current->pid, \ +#define mlx5_core_err(__dev, format, ...) \ + dev_err((__dev)->device, "%s:%d:(pid %d): " format, \ + __func__, __LINE__, current->pid, \ ##__VA_ARGS__) -#define mlx5_core_err_rl(__dev, format, ...) \ - dev_err_ratelimited(&(__dev)->pdev->dev, \ - "%s:%d:(pid %d): " format, \ - __func__, __LINE__, current->pid, \ - ##__VA_ARGS__) +#define mlx5_core_err_rl(__dev, format, ...) \ + dev_err_ratelimited((__dev)->device, \ + "%s:%d:(pid %d): " format, \ + __func__, __LINE__, current->pid, \ + ##__VA_ARGS__) -#define mlx5_core_warn(__dev, format, ...) \ - dev_warn(&(__dev)->pdev->dev, "%s:%d:(pid %d): " format, \ - __func__, __LINE__, current->pid, \ - ##__VA_ARGS__) +#define mlx5_core_warn(__dev, format, ...) \ + dev_warn((__dev)->device, "%s:%d:(pid %d): " format, \ + __func__, __LINE__, current->pid, \ + ##__VA_ARGS__) #define mlx5_core_warn_once(__dev, format, ...) \ - dev_warn_once(&(__dev)->pdev->dev, "%s:%d:(pid %d): " format, \ + dev_warn_once((__dev)->device, "%s:%d:(pid %d): " format, \ __func__, __LINE__, current->pid, \ ##__VA_ARGS__) -#define mlx5_core_info(__dev, format, ...) \ - dev_info(&(__dev)->pdev->dev, format, ##__VA_ARGS__) +#define mlx5_core_warn_rl(__dev, format, ...) \ + dev_warn_ratelimited((__dev)->device, \ + "%s:%d:(pid %d): " format, \ + __func__, __LINE__, current->pid, \ + ##__VA_ARGS__) + +#define mlx5_core_info(__dev, format, ...) \ + dev_info((__dev)->device, format, ##__VA_ARGS__) + +#define mlx5_core_info_rl(__dev, format, ...) \ + dev_info_ratelimited((__dev)->device, \ + "%s:%d:(pid %d): " format, \ + __func__, __LINE__, current->pid, \ + ##__VA_ARGS__) enum { MLX5_CMD_DATA, /* print command payload only */ @@ -111,7 +125,6 @@ void mlx5_sriov_cleanup(struct mlx5_core_dev *dev); int mlx5_sriov_attach(struct mlx5_core_dev *dev); void mlx5_sriov_detach(struct mlx5_core_dev *dev); int mlx5_core_sriov_configure(struct pci_dev *dev, int num_vfs); -bool mlx5_sriov_is_enabled(struct mlx5_core_dev *dev); int mlx5_core_enable_hca(struct mlx5_core_dev *dev, u16 func_id); int mlx5_core_disable_hca(struct mlx5_core_dev *dev, u16 func_id); int mlx5_create_scheduling_element_cmd(struct mlx5_core_dev *dev, u8 hierarchy, @@ -176,6 +189,11 @@ int mlx5_firmware_flash(struct mlx5_core_dev *dev, const struct firmware *fw); void mlx5e_init(void); void mlx5e_cleanup(void); +static inline bool mlx5_sriov_is_enabled(struct mlx5_core_dev *dev) +{ + return pci_num_vf(dev->pdev) ? true : false; +} + static inline int mlx5_lag_is_lacp_owner(struct mlx5_core_dev *dev) { /* LACP owner conditions: diff --git a/drivers/net/ethernet/mellanox/mlx5/core/pagealloc.c b/drivers/net/ethernet/mellanox/mlx5/core/pagealloc.c index 41025387ff2c170c0790bb64f2a73e1fd0f8f267..91bd258ecf1b51c2e7c4c9b06fdf41cdc562731e 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/pagealloc.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/pagealloc.c @@ -200,7 +200,7 @@ static void free_4k(struct mlx5_core_dev *dev, u64 addr) rb_erase(&fwp->rb_node, &dev->priv.page_root); if (fwp->free_count != 1) list_del(&fwp->list); - dma_unmap_page(&dev->pdev->dev, addr & MLX5_U64_4K_PAGE_MASK, + dma_unmap_page(dev->device, addr & MLX5_U64_4K_PAGE_MASK, PAGE_SIZE, DMA_BIDIRECTIONAL); __free_page(fwp->page); kfree(fwp); @@ -211,11 +211,12 @@ static void free_4k(struct mlx5_core_dev *dev, u64 addr) static int alloc_system_page(struct mlx5_core_dev *dev, u16 func_id) { + struct device *device = dev->device; + int nid = dev_to_node(device); struct page *page; u64 zero_addr = 1; u64 addr; int err; - int nid = dev_to_node(&dev->pdev->dev); page = alloc_pages_node(nid, GFP_HIGHUSER, 0); if (!page) { @@ -223,9 +224,8 @@ static int alloc_system_page(struct mlx5_core_dev *dev, u16 func_id) return -ENOMEM; } map: - addr = dma_map_page(&dev->pdev->dev, page, 0, - PAGE_SIZE, DMA_BIDIRECTIONAL); - if (dma_mapping_error(&dev->pdev->dev, addr)) { + addr = dma_map_page(device, page, 0, PAGE_SIZE, DMA_BIDIRECTIONAL); + if (dma_mapping_error(device, addr)) { mlx5_core_warn(dev, "failed dma mapping page\n"); err = -ENOMEM; goto err_mapping; @@ -240,8 +240,7 @@ map: err = insert_page(dev, addr, page, func_id); if (err) { mlx5_core_err(dev, "failed to track allocated page\n"); - dma_unmap_page(&dev->pdev->dev, addr, PAGE_SIZE, - DMA_BIDIRECTIONAL); + dma_unmap_page(device, addr, PAGE_SIZE, DMA_BIDIRECTIONAL); } err_mapping: @@ -249,7 +248,7 @@ err_mapping: __free_page(page); if (zero_addr == 0) - dma_unmap_page(&dev->pdev->dev, zero_addr, PAGE_SIZE, + dma_unmap_page(device, zero_addr, PAGE_SIZE, DMA_BIDIRECTIONAL); return err; @@ -600,8 +599,7 @@ int mlx5_wait_for_pages(struct mlx5_core_dev *dev, int *pages) return 0; } - mlx5_core_dbg(dev, "Waiting for %d pages from %s\n", prev_pages, - dev->priv.name); + mlx5_core_dbg(dev, "Waiting for %d pages\n", prev_pages); while (*pages) { if (time_after(jiffies, end)) { mlx5_core_warn(dev, "aborting while there are %d pending pages\n", *pages); @@ -614,6 +612,6 @@ int mlx5_wait_for_pages(struct mlx5_core_dev *dev, int *pages) msleep(50); } - mlx5_core_dbg(dev, "All pages received from %s\n", dev->priv.name); + mlx5_core_dbg(dev, "All pages received\n"); return 0; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/port.c b/drivers/net/ethernet/mellanox/mlx5/core/port.c index 361468e0435dcc9fbb667716e483c4104ebf7fea..cc262b30aed532ca4240d7aaf985ba288384911f 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/port.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/port.c @@ -293,15 +293,36 @@ static int mlx5_query_module_num(struct mlx5_core_dev *dev, int *module_num) return 0; } +static int mlx5_eeprom_page(int offset) +{ + if (offset < MLX5_EEPROM_PAGE_LENGTH) + /* Addresses between 0-255 - page 00 */ + return 0; + + /* Addresses between 256 - 639 belongs to pages 01, 02 and 03 + * For example, offset = 400 belongs to page 02: + * 1 + ((400 - 256)/128) = 2 + */ + return 1 + ((offset - MLX5_EEPROM_PAGE_LENGTH) / + MLX5_EEPROM_HIGH_PAGE_LENGTH); +} + +static int mlx5_eeprom_high_page_offset(int page_num) +{ + if (!page_num) /* Page 0 always start from low page */ + return 0; + + /* High page */ + return page_num * MLX5_EEPROM_HIGH_PAGE_LENGTH; +} + int mlx5_query_module_eeprom(struct mlx5_core_dev *dev, u16 offset, u16 size, u8 *data) { + int module_num, page_num, status, err; u32 out[MLX5_ST_SZ_DW(mcia_reg)]; u32 in[MLX5_ST_SZ_DW(mcia_reg)]; - int module_num; u16 i2c_addr; - int status; - int err; void *ptr = MLX5_ADDR_OF(mcia_reg, out, dword_0); err = mlx5_query_module_num(dev, &module_num); @@ -311,8 +332,15 @@ int mlx5_query_module_eeprom(struct mlx5_core_dev *dev, memset(in, 0, sizeof(in)); size = min_t(int, size, MLX5_EEPROM_MAX_BYTES); - if (offset < MLX5_EEPROM_PAGE_LENGTH && - offset + size > MLX5_EEPROM_PAGE_LENGTH) + /* Get the page number related to the given offset */ + page_num = mlx5_eeprom_page(offset); + + /* Set the right offset according to the page number, + * For page_num > 0, relative offset is always >= 128 (high page). + */ + offset -= mlx5_eeprom_high_page_offset(page_num); + + if (offset + size > MLX5_EEPROM_PAGE_LENGTH) /* Cross pages read, read until offset 256 in low page */ size -= offset + size - MLX5_EEPROM_PAGE_LENGTH; @@ -321,7 +349,7 @@ int mlx5_query_module_eeprom(struct mlx5_core_dev *dev, MLX5_SET(mcia_reg, in, l, 0); MLX5_SET(mcia_reg, in, module, module_num); MLX5_SET(mcia_reg, in, i2c_device_address, i2c_addr); - MLX5_SET(mcia_reg, in, page_number, 0); + MLX5_SET(mcia_reg, in, page_number, page_num); MLX5_SET(mcia_reg, in, device_address, offset); MLX5_SET(mcia_reg, in, size, size); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/rdma.c b/drivers/net/ethernet/mellanox/mlx5/core/rdma.c new file mode 100644 index 0000000000000000000000000000000000000000..86f77456f873484742b2efef9919611f2fcf5f96 --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/rdma.c @@ -0,0 +1,182 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* Copyright (c) 2019 Mellanox Technologies */ + +#include +#include +#include + +#include "lib/mlx5.h" +#include "eswitch.h" +#include "fs_core.h" +#include "rdma.h" + +static void mlx5_rdma_disable_roce_steering(struct mlx5_core_dev *dev) +{ + struct mlx5_core_roce *roce = &dev->priv.roce; + + if (!roce->ft) + return; + + mlx5_del_flow_rules(roce->allow_rule); + mlx5_destroy_flow_group(roce->fg); + mlx5_destroy_flow_table(roce->ft); +} + +static int mlx5_rdma_enable_roce_steering(struct mlx5_core_dev *dev) +{ + int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in); + struct mlx5_core_roce *roce = &dev->priv.roce; + struct mlx5_flow_handle *flow_rule = NULL; + struct mlx5_flow_table_attr ft_attr = {}; + struct mlx5_flow_namespace *ns = NULL; + struct mlx5_flow_act flow_act = {}; + struct mlx5_flow_spec *spec; + struct mlx5_flow_table *ft; + struct mlx5_flow_group *fg; + void *match_criteria; + u32 *flow_group_in; + void *misc; + int err; + + if (!(MLX5_CAP_FLOWTABLE_RDMA_RX(dev, ft_support) && + MLX5_CAP_FLOWTABLE_RDMA_RX(dev, table_miss_action_domain))) + return -EOPNOTSUPP; + + flow_group_in = kvzalloc(inlen, GFP_KERNEL); + if (!flow_group_in) + return -ENOMEM; + spec = kvzalloc(sizeof(*spec), GFP_KERNEL); + if (!spec) { + kvfree(flow_group_in); + return -ENOMEM; + } + + ns = mlx5_get_flow_namespace(dev, MLX5_FLOW_NAMESPACE_RDMA_RX); + if (!ns) { + mlx5_core_err(dev, "Failed to get RDMA RX namespace"); + err = -EOPNOTSUPP; + goto free; + } + + ft_attr.max_fte = 1; + ft = mlx5_create_flow_table(ns, &ft_attr); + if (IS_ERR(ft)) { + mlx5_core_err(dev, "Failed to create RDMA RX flow table"); + err = PTR_ERR(ft); + goto free; + } + + MLX5_SET(create_flow_group_in, flow_group_in, match_criteria_enable, + MLX5_MATCH_MISC_PARAMETERS); + match_criteria = MLX5_ADDR_OF(create_flow_group_in, flow_group_in, + match_criteria); + MLX5_SET_TO_ONES(fte_match_param, match_criteria, + misc_parameters.source_port); + + fg = mlx5_create_flow_group(ft, flow_group_in); + if (IS_ERR(fg)) { + err = PTR_ERR(fg); + mlx5_core_err(dev, "Failed to create RDMA RX flow group err(%d)\n", err); + goto destroy_flow_table; + } + + spec->match_criteria_enable = MLX5_MATCH_MISC_PARAMETERS; + misc = MLX5_ADDR_OF(fte_match_param, spec->match_value, + misc_parameters); + MLX5_SET(fte_match_set_misc, misc, source_port, + dev->priv.eswitch->manager_vport); + misc = MLX5_ADDR_OF(fte_match_param, spec->match_criteria, + misc_parameters); + MLX5_SET_TO_ONES(fte_match_set_misc, misc, source_port); + + flow_act.action = MLX5_FLOW_CONTEXT_ACTION_ALLOW; + flow_rule = mlx5_add_flow_rules(ft, spec, &flow_act, NULL, 0); + if (IS_ERR(flow_rule)) { + err = PTR_ERR(flow_rule); + mlx5_core_err(dev, "Failed to add RoCE allow rule, err=%d\n", + err); + goto destroy_flow_group; + } + + kvfree(spec); + kvfree(flow_group_in); + roce->ft = ft; + roce->fg = fg; + roce->allow_rule = flow_rule; + + return 0; + +destroy_flow_table: + mlx5_destroy_flow_table(ft); +destroy_flow_group: + mlx5_destroy_flow_group(fg); +free: + kvfree(spec); + kvfree(flow_group_in); + return err; +} + +static void mlx5_rdma_del_roce_addr(struct mlx5_core_dev *dev) +{ + mlx5_core_roce_gid_set(dev, 0, 0, 0, + NULL, NULL, false, 0, 0); +} + +static void mlx5_rdma_make_default_gid(struct mlx5_core_dev *dev, union ib_gid *gid) +{ + u8 hw_id[ETH_ALEN]; + + mlx5_query_nic_vport_mac_address(dev, 0, hw_id); + gid->global.subnet_prefix = cpu_to_be64(0xfe80000000000000LL); + addrconf_addr_eui48(&gid->raw[8], hw_id); +} + +static int mlx5_rdma_add_roce_addr(struct mlx5_core_dev *dev) +{ + union ib_gid gid; + u8 mac[ETH_ALEN]; + + mlx5_rdma_make_default_gid(dev, &gid); + return mlx5_core_roce_gid_set(dev, 0, + MLX5_ROCE_VERSION_1, + 0, gid.raw, mac, + false, 0, 1); +} + +void mlx5_rdma_disable_roce(struct mlx5_core_dev *dev) +{ + mlx5_rdma_disable_roce_steering(dev); + mlx5_rdma_del_roce_addr(dev); + mlx5_nic_vport_disable_roce(dev); +} + +void mlx5_rdma_enable_roce(struct mlx5_core_dev *dev) +{ + int err; + + err = mlx5_nic_vport_enable_roce(dev); + if (err) { + mlx5_core_err(dev, "Failed to enable RoCE: %d\n", err); + return; + } + + err = mlx5_rdma_add_roce_addr(dev); + if (err) { + mlx5_core_err(dev, "Failed to add RoCE address: %d\n", err); + goto disable_roce; + } + + err = mlx5_rdma_enable_roce_steering(dev); + if (err) { + mlx5_core_err(dev, "Failed to enable RoCE steering: %d\n", err); + goto del_roce_addr; + } + + return; + +del_roce_addr: + mlx5_rdma_del_roce_addr(dev); +disable_roce: + mlx5_nic_vport_disable_roce(dev); + return; +} diff --git a/drivers/net/ethernet/mellanox/mlx5/core/rdma.h b/drivers/net/ethernet/mellanox/mlx5/core/rdma.h new file mode 100644 index 0000000000000000000000000000000000000000..750cff2a71a4bb2addd9c7617b2b5354c2d573d2 --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/rdma.h @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2019 Mellanox Technologies. */ + +#ifndef __MLX5_RDMA_H__ +#define __MLX5_RDMA_H__ + +#include "mlx5_core.h" + +#ifdef CONFIG_MLX5_ESWITCH + +void mlx5_rdma_enable_roce(struct mlx5_core_dev *dev); +void mlx5_rdma_disable_roce(struct mlx5_core_dev *dev); + +#else /* CONFIG_MLX5_ESWITCH */ + +static inline void mlx5_rdma_enable_roce(struct mlx5_core_dev *dev) {} +static inline void mlx5_rdma_disable_roce(struct mlx5_core_dev *dev) {} + +#endif /* CONFIG_MLX5_ESWITCH */ +#endif /* __MLX5_RDMA_H__ */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/sriov.c b/drivers/net/ethernet/mellanox/mlx5/core/sriov.c index 7b23fa8d2d60a74ae66427d3b9df3a990d6bb8ae..a249b3c3843db0e624948d86a824d6bea8b8c115 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/sriov.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/sriov.c @@ -36,13 +36,6 @@ #include "mlx5_core.h" #include "eswitch.h" -bool mlx5_sriov_is_enabled(struct mlx5_core_dev *dev) -{ - struct mlx5_core_sriov *sriov = &dev->priv.sriov; - - return !!sriov->num_vfs; -} - static int sriov_restore_guids(struct mlx5_core_dev *dev, int vf) { struct mlx5_core_sriov *sriov = &dev->priv.sriov; @@ -151,33 +144,10 @@ out: mlx5_core_warn(dev, "timeout reclaiming VFs pages\n"); } -static int mlx5_pci_enable_sriov(struct pci_dev *pdev, int num_vfs) -{ - struct mlx5_core_dev *dev = pci_get_drvdata(pdev); - int err = 0; - - if (pci_num_vf(pdev)) { - mlx5_core_warn(dev, "Unable to enable pci sriov, already enabled\n"); - return -EBUSY; - } - - err = pci_enable_sriov(pdev, num_vfs); - if (err) - mlx5_core_warn(dev, "pci_enable_sriov failed : %d\n", err); - - return err; -} - -static void mlx5_pci_disable_sriov(struct pci_dev *pdev) -{ - pci_disable_sriov(pdev); -} - static int mlx5_sriov_enable(struct pci_dev *pdev, int num_vfs) { struct mlx5_core_dev *dev = pci_get_drvdata(pdev); - struct mlx5_core_sriov *sriov = &dev->priv.sriov; - int err = 0; + int err; err = mlx5_device_enable_sriov(dev, num_vfs); if (err) { @@ -185,42 +155,37 @@ static int mlx5_sriov_enable(struct pci_dev *pdev, int num_vfs) return err; } - err = mlx5_pci_enable_sriov(pdev, num_vfs); + err = pci_enable_sriov(pdev, num_vfs); if (err) { - mlx5_core_warn(dev, "mlx5_pci_enable_sriov failed : %d\n", err); + mlx5_core_warn(dev, "pci_enable_sriov failed : %d\n", err); mlx5_device_disable_sriov(dev); - return err; } - - sriov->num_vfs = num_vfs; - - return 0; + return err; } static void mlx5_sriov_disable(struct pci_dev *pdev) { struct mlx5_core_dev *dev = pci_get_drvdata(pdev); - struct mlx5_core_sriov *sriov = &dev->priv.sriov; - mlx5_pci_disable_sriov(pdev); + pci_disable_sriov(pdev); mlx5_device_disable_sriov(dev); - sriov->num_vfs = 0; } int mlx5_core_sriov_configure(struct pci_dev *pdev, int num_vfs) { struct mlx5_core_dev *dev = pci_get_drvdata(pdev); + struct mlx5_core_sriov *sriov = &dev->priv.sriov; int err = 0; mlx5_core_dbg(dev, "requested num_vfs %d\n", num_vfs); - if (!mlx5_core_is_pf(dev)) - return -EPERM; if (num_vfs) err = mlx5_sriov_enable(pdev, num_vfs); else mlx5_sriov_disable(pdev); + if (!err) + sriov->num_vfs = num_vfs; return err ? err : num_vfs; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/transobj.c b/drivers/net/ethernet/mellanox/mlx5/core/transobj.c index c4d4b76096dc2e5884965966e9e1e63913b5abd4..b1068500f1df59658a03ce1fb107bc1ca49f7032 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/transobj.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/transobj.c @@ -182,16 +182,24 @@ out: } EXPORT_SYMBOL_GPL(mlx5_core_query_sq_state); +int mlx5_core_create_tir_out(struct mlx5_core_dev *dev, + u32 *in, int inlen, + u32 *out, int outlen) +{ + MLX5_SET(create_tir_in, in, opcode, MLX5_CMD_OP_CREATE_TIR); + + return mlx5_cmd_exec(dev, in, inlen, out, outlen); +} +EXPORT_SYMBOL(mlx5_core_create_tir_out); + int mlx5_core_create_tir(struct mlx5_core_dev *dev, u32 *in, int inlen, u32 *tirn) { - u32 out[MLX5_ST_SZ_DW(create_tir_out)] = {0}; + u32 out[MLX5_ST_SZ_DW(create_tir_out)] = {}; int err; - MLX5_SET(create_tir_in, in, opcode, MLX5_CMD_OP_CREATE_TIR); - - memset(out, 0, sizeof(out)); - err = mlx5_cmd_exec(dev, in, inlen, out, sizeof(out)); + err = mlx5_core_create_tir_out(dev, in, inlen, + out, sizeof(out)); if (!err) *tirn = MLX5_GET(create_tir_out, out, tirn); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/uar.c b/drivers/net/ethernet/mellanox/mlx5/core/uar.c index 94464723ff77de3c112fde55e61490675572c7e1..0d006224d7b057ecfdb9115f0835259ca1165494 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/uar.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/uar.c @@ -79,7 +79,7 @@ static u64 uar2pfn(struct mlx5_core_dev *mdev, u32 index) else system_page_index = index; - return (pci_resource_start(mdev->pdev, 0) >> PAGE_SHIFT) + system_page_index; + return (mdev->bar_addr >> PAGE_SHIFT) + system_page_index; } static void up_rel_func(struct kref *kref) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/vport.c b/drivers/net/ethernet/mellanox/mlx5/core/vport.c index ef95feca9961b2fe031ef56b3c857fc79365a309..95cdc8cbcba45a7411816dd15b8dc7554f5b07d6 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/vport.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/vport.c @@ -371,67 +371,6 @@ int mlx5_modify_nic_vport_mac_list(struct mlx5_core_dev *dev, } EXPORT_SYMBOL_GPL(mlx5_modify_nic_vport_mac_list); -int mlx5_query_nic_vport_vlans(struct mlx5_core_dev *dev, - u16 vport, - u16 vlans[], - int *size) -{ - u32 in[MLX5_ST_SZ_DW(query_nic_vport_context_in)]; - void *nic_vport_ctx; - int req_list_size; - int max_list_size; - int out_sz; - void *out; - int err; - int i; - - req_list_size = *size; - max_list_size = 1 << MLX5_CAP_GEN(dev, log_max_vlan_list); - if (req_list_size > max_list_size) { - mlx5_core_warn(dev, "Requested list size (%d) > (%d) max list size\n", - req_list_size, max_list_size); - req_list_size = max_list_size; - } - - out_sz = MLX5_ST_SZ_BYTES(modify_nic_vport_context_in) + - req_list_size * MLX5_ST_SZ_BYTES(vlan_layout); - - memset(in, 0, sizeof(in)); - out = kzalloc(out_sz, GFP_KERNEL); - if (!out) - return -ENOMEM; - - MLX5_SET(query_nic_vport_context_in, in, opcode, - MLX5_CMD_OP_QUERY_NIC_VPORT_CONTEXT); - MLX5_SET(query_nic_vport_context_in, in, allowed_list_type, - MLX5_NVPRT_LIST_TYPE_VLAN); - MLX5_SET(query_nic_vport_context_in, in, vport_number, vport); - - if (vport) - MLX5_SET(query_nic_vport_context_in, in, other_vport, 1); - - err = mlx5_cmd_exec(dev, in, sizeof(in), out, out_sz); - if (err) - goto out; - - nic_vport_ctx = MLX5_ADDR_OF(query_nic_vport_context_out, out, - nic_vport_context); - req_list_size = MLX5_GET(nic_vport_context, nic_vport_ctx, - allowed_list_size); - - *size = req_list_size; - for (i = 0; i < req_list_size; i++) { - void *vlan_addr = MLX5_ADDR_OF(nic_vport_context, - nic_vport_ctx, - current_uc_mac_address[i]); - vlans[i] = MLX5_GET(vlan_layout, vlan_addr, vlan); - } -out: - kfree(out); - return err; -} -EXPORT_SYMBOL_GPL(mlx5_query_nic_vport_vlans); - int mlx5_modify_nic_vport_vlans(struct mlx5_core_dev *dev, u16 vlans[], int list_size) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/wq.h b/drivers/net/ethernet/mellanox/mlx5/core/wq.h index ea934a48c90acec3618e189a51d246ccf5c9de89..1f87cce421e02469e4c30b136371e685071da37e 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/wq.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/wq.h @@ -134,6 +134,11 @@ static inline void mlx5_wq_cyc_update_db_record(struct mlx5_wq_cyc *wq) *wq->db = cpu_to_be32(wq->wqe_ctr); } +static inline u16 mlx5_wq_cyc_get_ctr_wrap_cnt(struct mlx5_wq_cyc *wq, u16 ctr) +{ + return ctr >> wq->fbc.log_sz; +} + static inline u16 mlx5_wq_cyc_ctr2ix(struct mlx5_wq_cyc *wq, u16 ctr) { return ctr & wq->fbc.sz_m1; @@ -243,6 +248,13 @@ static inline void *mlx5_wq_ll_get_wqe(struct mlx5_wq_ll *wq, u16 ix) return mlx5_frag_buf_get_wqe(&wq->fbc, ix); } +static inline u16 mlx5_wq_ll_get_wqe_next_ix(struct mlx5_wq_ll *wq, u16 ix) +{ + struct mlx5_wqe_srq_next_seg *wqe = mlx5_wq_ll_get_wqe(wq, ix); + + return be16_to_cpu(wqe->next_wqe_index); +} + static inline void mlx5_wq_ll_push(struct mlx5_wq_ll *wq, u16 head_next) { wq->head = head_next; diff --git a/drivers/net/ethernet/mellanox/mlxsw/Kconfig b/drivers/net/ethernet/mellanox/mlxsw/Kconfig index 9c195dfed0313f126659496d1593dc4b9d7704ae..b6b3ff0fe17f5c4e3a2c3f1460e6da57da4b79d8 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/Kconfig +++ b/drivers/net/ethernet/mellanox/mlxsw/Kconfig @@ -4,6 +4,7 @@ config MLXSW_CORE tristate "Mellanox Technologies Switch ASICs support" + select NET_DEVLINK ---help--- This driver supports Mellanox Technologies Switch ASICs family. diff --git a/drivers/net/ethernet/mellanox/mlxsw/Makefile b/drivers/net/ethernet/mellanox/mlxsw/Makefile index a01d15546e378510fe3decc6d5402540faf93fad..c4dc72e1ce638b60745245ee9a4dca0a82244f8e 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/Makefile +++ b/drivers/net/ethernet/mellanox/mlxsw/Makefile @@ -28,8 +28,8 @@ mlxsw_spectrum-objs := spectrum.o spectrum_buffers.o \ spectrum1_mr_tcam.o spectrum2_mr_tcam.o \ spectrum_mr_tcam.o spectrum_mr.o \ spectrum_qdisc.o spectrum_span.o \ - spectrum_nve.o spectrum_nve_vxlan.o + spectrum_nve.o spectrum_nve_vxlan.o \ + spectrum_dpipe.o mlxsw_spectrum-$(CONFIG_MLXSW_SPECTRUM_DCB) += spectrum_dcb.o -mlxsw_spectrum-$(CONFIG_NET_DEVLINK) += spectrum_dpipe.o obj-$(CONFIG_MLXSW_MINIMAL) += mlxsw_minimal.o mlxsw_minimal-objs := minimal.o diff --git a/drivers/net/ethernet/mellanox/mlxsw/core.c b/drivers/net/ethernet/mellanox/mlxsw/core.c index f26a4ca293637b48ccc9aa8b9e55f375f767eedb..bcbe07ec22be07e72c5d407f83483d7226fc338f 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/core.c +++ b/drivers/net/ethernet/mellanox/mlxsw/core.c @@ -781,7 +781,8 @@ mlxsw_devlink_sb_pool_get(struct devlink *devlink, static int mlxsw_devlink_sb_pool_set(struct devlink *devlink, unsigned int sb_index, u16 pool_index, u32 size, - enum devlink_sb_threshold_type threshold_type) + enum devlink_sb_threshold_type threshold_type, + struct netlink_ext_ack *extack) { struct mlxsw_core *mlxsw_core = devlink_priv(devlink); struct mlxsw_driver *mlxsw_driver = mlxsw_core->driver; @@ -789,7 +790,8 @@ mlxsw_devlink_sb_pool_set(struct devlink *devlink, if (!mlxsw_driver->sb_pool_set) return -EOPNOTSUPP; return mlxsw_driver->sb_pool_set(mlxsw_core, sb_index, - pool_index, size, threshold_type); + pool_index, size, threshold_type, + extack); } static void *__dl_port(struct devlink_port *devlink_port) @@ -829,7 +831,8 @@ static int mlxsw_devlink_sb_port_pool_get(struct devlink_port *devlink_port, static int mlxsw_devlink_sb_port_pool_set(struct devlink_port *devlink_port, unsigned int sb_index, u16 pool_index, - u32 threshold) + u32 threshold, + struct netlink_ext_ack *extack) { struct mlxsw_core *mlxsw_core = devlink_priv(devlink_port->devlink); struct mlxsw_driver *mlxsw_driver = mlxsw_core->driver; @@ -839,7 +842,7 @@ static int mlxsw_devlink_sb_port_pool_set(struct devlink_port *devlink_port, !mlxsw_core_port_check(mlxsw_core_port)) return -EOPNOTSUPP; return mlxsw_driver->sb_port_pool_set(mlxsw_core_port, sb_index, - pool_index, threshold); + pool_index, threshold, extack); } static int @@ -864,7 +867,8 @@ static int mlxsw_devlink_sb_tc_pool_bind_set(struct devlink_port *devlink_port, unsigned int sb_index, u16 tc_index, enum devlink_sb_pool_type pool_type, - u16 pool_index, u32 threshold) + u16 pool_index, u32 threshold, + struct netlink_ext_ack *extack) { struct mlxsw_core *mlxsw_core = devlink_priv(devlink_port->devlink); struct mlxsw_driver *mlxsw_driver = mlxsw_core->driver; @@ -875,7 +879,7 @@ mlxsw_devlink_sb_tc_pool_bind_set(struct devlink_port *devlink_port, return -EOPNOTSUPP; return mlxsw_driver->sb_tc_pool_bind_set(mlxsw_core_port, sb_index, tc_index, pool_type, - pool_index, threshold); + pool_index, threshold, extack); } static int mlxsw_devlink_sb_occ_snapshot(struct devlink *devlink, @@ -934,6 +938,46 @@ mlxsw_devlink_sb_occ_tc_port_bind_get(struct devlink_port *devlink_port, pool_type, p_cur, p_max); } +static int +mlxsw_devlink_info_get(struct devlink *devlink, struct devlink_info_req *req, + struct netlink_ext_ack *extack) +{ + struct mlxsw_core *mlxsw_core = devlink_priv(devlink); + char fw_info_psid[MLXSW_REG_MGIR_FW_INFO_PSID_SIZE]; + u32 hw_rev, fw_major, fw_minor, fw_sub_minor; + char mgir_pl[MLXSW_REG_MGIR_LEN]; + char buf[32]; + int err; + + err = devlink_info_driver_name_put(req, + mlxsw_core->bus_info->device_kind); + if (err) + return err; + + mlxsw_reg_mgir_pack(mgir_pl); + err = mlxsw_reg_query(mlxsw_core, MLXSW_REG(mgir), mgir_pl); + if (err) + return err; + mlxsw_reg_mgir_unpack(mgir_pl, &hw_rev, fw_info_psid, &fw_major, + &fw_minor, &fw_sub_minor); + + sprintf(buf, "%X", hw_rev); + err = devlink_info_version_fixed_put(req, "hw.revision", buf); + if (err) + return err; + + err = devlink_info_version_fixed_put(req, "fw.psid", fw_info_psid); + if (err) + return err; + + sprintf(buf, "%d.%d.%d", fw_major, fw_minor, fw_sub_minor); + err = devlink_info_version_running_put(req, "fw.version", buf); + if (err) + return err; + + return 0; +} + static int mlxsw_devlink_core_bus_device_reload(struct devlink *devlink, struct netlink_ext_ack *extack) { @@ -968,6 +1012,7 @@ static const struct devlink_ops mlxsw_devlink_ops = { .sb_occ_max_clear = mlxsw_devlink_sb_occ_max_clear, .sb_occ_port_pool_get = mlxsw_devlink_sb_occ_port_pool_get, .sb_occ_tc_port_bind_get = mlxsw_devlink_sb_occ_tc_port_bind_get, + .info_get = mlxsw_devlink_info_get, }; static int @@ -1718,7 +1763,11 @@ u64 mlxsw_core_res_get(struct mlxsw_core *mlxsw_core, } EXPORT_SYMBOL(mlxsw_core_res_get); -int mlxsw_core_port_init(struct mlxsw_core *mlxsw_core, u8 local_port) +int mlxsw_core_port_init(struct mlxsw_core *mlxsw_core, u8 local_port, + u32 port_number, bool split, + u32 split_port_subnumber, + const unsigned char *switch_id, + unsigned char switch_id_len) { struct devlink *devlink = priv_to_devlink(mlxsw_core); struct mlxsw_core_port *mlxsw_core_port = @@ -1727,6 +1776,9 @@ int mlxsw_core_port_init(struct mlxsw_core *mlxsw_core, u8 local_port) int err; mlxsw_core_port->local_port = local_port; + devlink_port_attrs_set(devlink_port, DEVLINK_PORT_FLAVOUR_PHYSICAL, + port_number, split, split_port_subnumber, + switch_id, switch_id_len); err = devlink_port_register(devlink, devlink_port, local_port); if (err) memset(mlxsw_core_port, 0, sizeof(*mlxsw_core_port)); @@ -1746,17 +1798,13 @@ void mlxsw_core_port_fini(struct mlxsw_core *mlxsw_core, u8 local_port) EXPORT_SYMBOL(mlxsw_core_port_fini); void mlxsw_core_port_eth_set(struct mlxsw_core *mlxsw_core, u8 local_port, - void *port_driver_priv, struct net_device *dev, - u32 port_number, bool split, - u32 split_port_subnumber) + void *port_driver_priv, struct net_device *dev) { struct mlxsw_core_port *mlxsw_core_port = &mlxsw_core->ports[local_port]; struct devlink_port *devlink_port = &mlxsw_core_port->devlink_port; mlxsw_core_port->port_driver_priv = port_driver_priv; - devlink_port_attrs_set(devlink_port, DEVLINK_PORT_FLAVOUR_PHYSICAL, - port_number, split, split_port_subnumber); devlink_port_type_eth_set(devlink_port, dev); } EXPORT_SYMBOL(mlxsw_core_port_eth_set); @@ -1796,16 +1844,18 @@ enum devlink_port_type mlxsw_core_port_type_get(struct mlxsw_core *mlxsw_core, } EXPORT_SYMBOL(mlxsw_core_port_type_get); -int mlxsw_core_port_get_phys_port_name(struct mlxsw_core *mlxsw_core, - u8 local_port, char *name, size_t len) + +struct devlink_port * +mlxsw_core_port_devlink_port_get(struct mlxsw_core *mlxsw_core, + u8 local_port) { struct mlxsw_core_port *mlxsw_core_port = &mlxsw_core->ports[local_port]; struct devlink_port *devlink_port = &mlxsw_core_port->devlink_port; - return devlink_port_get_phys_port_name(devlink_port, name, len); + return devlink_port; } -EXPORT_SYMBOL(mlxsw_core_port_get_phys_port_name); +EXPORT_SYMBOL(mlxsw_core_port_devlink_port_get); static void mlxsw_core_buf_dump_dbg(struct mlxsw_core *mlxsw_core, const char *buf, size_t size) diff --git a/drivers/net/ethernet/mellanox/mlxsw/core.h b/drivers/net/ethernet/mellanox/mlxsw/core.h index 8ec53f027575389b0cf38d5b8f4b123567e9acb2..917be621c9043d44f6e282b858b000b3996686b3 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/core.h +++ b/drivers/net/ethernet/mellanox/mlxsw/core.h @@ -164,20 +164,23 @@ void mlxsw_core_lag_mapping_clear(struct mlxsw_core *mlxsw_core, u16 lag_id, u8 local_port); void *mlxsw_core_port_driver_priv(struct mlxsw_core_port *mlxsw_core_port); -int mlxsw_core_port_init(struct mlxsw_core *mlxsw_core, u8 local_port); +int mlxsw_core_port_init(struct mlxsw_core *mlxsw_core, u8 local_port, + u32 port_number, bool split, + u32 split_port_subnumber, + const unsigned char *switch_id, + unsigned char switch_id_len); void mlxsw_core_port_fini(struct mlxsw_core *mlxsw_core, u8 local_port); void mlxsw_core_port_eth_set(struct mlxsw_core *mlxsw_core, u8 local_port, - void *port_driver_priv, struct net_device *dev, - u32 port_number, bool split, - u32 split_port_subnumber); + void *port_driver_priv, struct net_device *dev); void mlxsw_core_port_ib_set(struct mlxsw_core *mlxsw_core, u8 local_port, void *port_driver_priv); void mlxsw_core_port_clear(struct mlxsw_core *mlxsw_core, u8 local_port, void *port_driver_priv); enum devlink_port_type mlxsw_core_port_type_get(struct mlxsw_core *mlxsw_core, u8 local_port); -int mlxsw_core_port_get_phys_port_name(struct mlxsw_core *mlxsw_core, - u8 local_port, char *name, size_t len); +struct devlink_port * +mlxsw_core_port_devlink_port_get(struct mlxsw_core *mlxsw_core, + u8 local_port); int mlxsw_core_schedule_dw(struct delayed_work *dwork, unsigned long delay); bool mlxsw_core_schedule_work(struct work_struct *work); @@ -251,13 +254,14 @@ struct mlxsw_driver { struct devlink_sb_pool_info *pool_info); int (*sb_pool_set)(struct mlxsw_core *mlxsw_core, unsigned int sb_index, u16 pool_index, u32 size, - enum devlink_sb_threshold_type threshold_type); + enum devlink_sb_threshold_type threshold_type, + struct netlink_ext_ack *extack); int (*sb_port_pool_get)(struct mlxsw_core_port *mlxsw_core_port, unsigned int sb_index, u16 pool_index, u32 *p_threshold); int (*sb_port_pool_set)(struct mlxsw_core_port *mlxsw_core_port, unsigned int sb_index, u16 pool_index, - u32 threshold); + u32 threshold, struct netlink_ext_ack *extack); int (*sb_tc_pool_bind_get)(struct mlxsw_core_port *mlxsw_core_port, unsigned int sb_index, u16 tc_index, enum devlink_sb_pool_type pool_type, @@ -265,7 +269,8 @@ struct mlxsw_driver { int (*sb_tc_pool_bind_set)(struct mlxsw_core_port *mlxsw_core_port, unsigned int sb_index, u16 tc_index, enum devlink_sb_pool_type pool_type, - u16 pool_index, u32 threshold); + u16 pool_index, u32 threshold, + struct netlink_ext_ack *extack); int (*sb_occ_snapshot)(struct mlxsw_core *mlxsw_core, unsigned int sb_index); int (*sb_occ_max_clear)(struct mlxsw_core *mlxsw_core, diff --git a/drivers/net/ethernet/mellanox/mlxsw/minimal.c b/drivers/net/ethernet/mellanox/mlxsw/minimal.c index 00c390024350d9b47570ec718e51c78faa4b6cb4..cf2114273b72aae2446a233f2c945246857492f5 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/minimal.c +++ b/drivers/net/ethernet/mellanox/mlxsw/minimal.c @@ -51,33 +51,20 @@ static int mlxsw_m_port_dummy_open_stop(struct net_device *dev) return 0; } -static int -mlxsw_m_port_get_phys_port_name(struct net_device *dev, char *name, size_t len) -{ - struct mlxsw_m_port *mlxsw_m_port = netdev_priv(dev); - struct mlxsw_core *core = mlxsw_m_port->mlxsw_m->core; - u8 local_port = mlxsw_m_port->local_port; - - return mlxsw_core_port_get_phys_port_name(core, local_port, name, len); -} - -static int mlxsw_m_port_get_port_parent_id(struct net_device *dev, - struct netdev_phys_item_id *ppid) +static struct devlink_port * +mlxsw_m_port_get_devlink_port(struct net_device *dev) { struct mlxsw_m_port *mlxsw_m_port = netdev_priv(dev); struct mlxsw_m *mlxsw_m = mlxsw_m_port->mlxsw_m; - ppid->id_len = sizeof(mlxsw_m->base_mac); - memcpy(&ppid->id, &mlxsw_m->base_mac, ppid->id_len); - - return 0; + return mlxsw_core_port_devlink_port_get(mlxsw_m->core, + mlxsw_m_port->local_port); } static const struct net_device_ops mlxsw_m_port_netdev_ops = { .ndo_open = mlxsw_m_port_dummy_open_stop, .ndo_stop = mlxsw_m_port_dummy_open_stop, - .ndo_get_phys_port_name = mlxsw_m_port_get_phys_port_name, - .ndo_get_port_parent_id = mlxsw_m_port_get_port_parent_id, + .ndo_get_devlink_port = mlxsw_m_port_get_devlink_port, }; static int mlxsw_m_get_module_info(struct net_device *netdev, @@ -150,7 +137,10 @@ mlxsw_m_port_create(struct mlxsw_m *mlxsw_m, u8 local_port, u8 module) struct net_device *dev; int err; - err = mlxsw_core_port_init(mlxsw_m->core, local_port); + err = mlxsw_core_port_init(mlxsw_m->core, local_port, + module + 1, false, 0, + mlxsw_m->base_mac, + sizeof(mlxsw_m->base_mac)); if (err) { dev_err(mlxsw_m->bus_info->dev, "Port %d: Failed to init core port\n", local_port); @@ -190,7 +180,7 @@ mlxsw_m_port_create(struct mlxsw_m *mlxsw_m, u8 local_port, u8 module) } mlxsw_core_port_eth_set(mlxsw_m->core, mlxsw_m_port->local_port, - mlxsw_m_port, dev, module + 1, false, 0); + mlxsw_m_port, dev); return 0; diff --git a/drivers/net/ethernet/mellanox/mlxsw/reg.h b/drivers/net/ethernet/mellanox/mlxsw/reg.h index eb4c5e8964cd6210af966d5347cedf9fb64cc53d..e8002bfc1e8f13a1d6352042f70f8a7c248c3a57 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/reg.h +++ b/drivers/net/ethernet/mellanox/mlxsw/reg.h @@ -5210,6 +5210,42 @@ static inline void mlxsw_reg_pspa_pack(char *payload, u8 swid, u8 local_port) mlxsw_reg_pspa_sub_port_set(payload, 0); } +/* PPLR - Port Physical Loopback Register + * -------------------------------------- + * This register allows configuration of the port's loopback mode. + */ +#define MLXSW_REG_PPLR_ID 0x5018 +#define MLXSW_REG_PPLR_LEN 0x8 + +MLXSW_REG_DEFINE(pplr, MLXSW_REG_PPLR_ID, MLXSW_REG_PPLR_LEN); + +/* reg_pplr_local_port + * Local port number. + * Access: Index + */ +MLXSW_ITEM32(reg, pplr, local_port, 0x00, 16, 8); + +/* Phy local loopback. When set the port's egress traffic is looped back + * to the receiver and the port transmitter is disabled. + */ +#define MLXSW_REG_PPLR_LB_TYPE_BIT_PHY_LOCAL BIT(1) + +/* reg_pplr_lb_en + * Loopback enable. + * Access: RW + */ +MLXSW_ITEM32(reg, pplr, lb_en, 0x04, 0, 8); + +static inline void mlxsw_reg_pplr_pack(char *payload, u8 local_port, + bool phy_local) +{ + MLXSW_REG_ZERO(pplr, payload); + mlxsw_reg_pplr_local_port_set(payload, local_port); + mlxsw_reg_pplr_lb_en_set(payload, + phy_local ? + MLXSW_REG_PPLR_LB_TYPE_BIT_PHY_LOCAL : 0); +} + /* HTGT - Host Trap Group Table * ---------------------------- * Configures the properties for forwarding to CPU. @@ -8534,6 +8570,60 @@ static inline void mlxsw_reg_mpar_pack(char *payload, u8 local_port, mlxsw_reg_mpar_pa_id_set(payload, pa_id); } +/* MGIR - Management General Information Register + * ---------------------------------------------- + * MGIR register allows software to query the hardware and firmware general + * information. + */ +#define MLXSW_REG_MGIR_ID 0x9020 +#define MLXSW_REG_MGIR_LEN 0x9C + +MLXSW_REG_DEFINE(mgir, MLXSW_REG_MGIR_ID, MLXSW_REG_MGIR_LEN); + +/* reg_mgir_hw_info_device_hw_revision + * Access: RO + */ +MLXSW_ITEM32(reg, mgir, hw_info_device_hw_revision, 0x0, 16, 16); + +#define MLXSW_REG_MGIR_FW_INFO_PSID_SIZE 16 + +/* reg_mgir_fw_info_psid + * PSID (ASCII string). + * Access: RO + */ +MLXSW_ITEM_BUF(reg, mgir, fw_info_psid, 0x30, MLXSW_REG_MGIR_FW_INFO_PSID_SIZE); + +/* reg_mgir_fw_info_extended_major + * Access: RO + */ +MLXSW_ITEM32(reg, mgir, fw_info_extended_major, 0x44, 0, 32); + +/* reg_mgir_fw_info_extended_minor + * Access: RO + */ +MLXSW_ITEM32(reg, mgir, fw_info_extended_minor, 0x48, 0, 32); + +/* reg_mgir_fw_info_extended_sub_minor + * Access: RO + */ +MLXSW_ITEM32(reg, mgir, fw_info_extended_sub_minor, 0x4C, 0, 32); + +static inline void mlxsw_reg_mgir_pack(char *payload) +{ + MLXSW_REG_ZERO(mgir, payload); +} + +static inline void +mlxsw_reg_mgir_unpack(char *payload, u32 *hw_rev, char *fw_info_psid, + u32 *fw_major, u32 *fw_minor, u32 *fw_sub_minor) +{ + *hw_rev = mlxsw_reg_mgir_hw_info_device_hw_revision_get(payload); + mlxsw_reg_mgir_fw_info_psid_memcpy_from(payload, fw_info_psid); + *fw_major = mlxsw_reg_mgir_fw_info_extended_major_get(payload); + *fw_minor = mlxsw_reg_mgir_fw_info_extended_minor_get(payload); + *fw_sub_minor = mlxsw_reg_mgir_fw_info_extended_sub_minor_get(payload); +} + /* MRSR - Management Reset and Shutdown Register * --------------------------------------------- * MRSR register is used to reset or shutdown the switch or @@ -9927,6 +10017,7 @@ static const struct mlxsw_reg_info *mlxsw_reg_infos[] = { MLXSW_REG(pptb), MLXSW_REG(pbmc), MLXSW_REG(pspa), + MLXSW_REG(pplr), MLXSW_REG(htgt), MLXSW_REG(hpkt), MLXSW_REG(rgcr), @@ -9958,6 +10049,7 @@ static const struct mlxsw_reg_info *mlxsw_reg_infos[] = { MLXSW_REG(mcia), MLXSW_REG(mpat), MLXSW_REG(mpar), + MLXSW_REG(mgir), MLXSW_REG(mrsr), MLXSW_REG(mlcr), MLXSW_REG(mpsc), diff --git a/drivers/net/ethernet/mellanox/mlxsw/resources.h b/drivers/net/ethernet/mellanox/mlxsw/resources.h index 773ef7fdb2850cf8286e9ec0e6df74d3a6406210..33a9fc9ef6a43f0f5a24ae970fb953e23763aec0 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/resources.h +++ b/drivers/net/ethernet/mellanox/mlxsw/resources.h @@ -24,6 +24,8 @@ enum mlxsw_res_id { MLXSW_RES_ID_MAX_SYSTEM_PORT, MLXSW_RES_ID_MAX_LAG, MLXSW_RES_ID_MAX_LAG_MEMBERS, + MLXSW_RES_ID_LOCAL_PORTS_IN_1X, + MLXSW_RES_ID_LOCAL_PORTS_IN_2X, MLXSW_RES_ID_MAX_BUFFER_SIZE, MLXSW_RES_ID_CELL_SIZE, MLXSW_RES_ID_MAX_HEADROOM_SIZE, @@ -78,6 +80,8 @@ static u16 mlxsw_res_ids[] = { [MLXSW_RES_ID_MAX_SYSTEM_PORT] = 0x2502, [MLXSW_RES_ID_MAX_LAG] = 0x2520, [MLXSW_RES_ID_MAX_LAG_MEMBERS] = 0x2521, + [MLXSW_RES_ID_LOCAL_PORTS_IN_1X] = 0x2610, + [MLXSW_RES_ID_LOCAL_PORTS_IN_2X] = 0x2611, [MLXSW_RES_ID_MAX_BUFFER_SIZE] = 0x2802, /* Bytes */ [MLXSW_RES_ID_CELL_SIZE] = 0x2803, /* Bytes */ [MLXSW_RES_ID_MAX_HEADROOM_SIZE] = 0x2811, /* Bytes */ diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c index 6b8aa3761899b03e7c9211b7d3272a1027a57371..dbb425717f5ed4971ecdfc4b01f3bdfef555aac6 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c +++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c @@ -21,7 +21,7 @@ #include #include #include -#include +#include #include #include #include @@ -46,8 +46,8 @@ #define MLXSW_SP_FWREV_MINOR_TO_BRANCH(minor) ((minor) / 100) #define MLXSW_SP1_FWREV_MAJOR 13 -#define MLXSW_SP1_FWREV_MINOR 1910 -#define MLXSW_SP1_FWREV_SUBMINOR 622 +#define MLXSW_SP1_FWREV_MINOR 2000 +#define MLXSW_SP1_FWREV_SUBMINOR 1122 #define MLXSW_SP1_FWREV_CAN_RESET_MINOR 1702 static const struct mlxsw_fw_rev mlxsw_sp1_fw_rev = { @@ -1254,16 +1254,6 @@ static int mlxsw_sp_port_kill_vid(struct net_device *dev, return 0; } -static int mlxsw_sp_port_get_phys_port_name(struct net_device *dev, char *name, - size_t len) -{ - struct mlxsw_sp_port *mlxsw_sp_port = netdev_priv(dev); - - return mlxsw_core_port_get_phys_port_name(mlxsw_sp_port->mlxsw_sp->core, - mlxsw_sp_port->local_port, - name, len); -} - static struct mlxsw_sp_port_mall_tc_entry * mlxsw_sp_port_mall_tc_entry_find(struct mlxsw_sp_port *port, unsigned long cookie) { @@ -1279,21 +1269,19 @@ mlxsw_sp_port_mall_tc_entry_find(struct mlxsw_sp_port *port, static int mlxsw_sp_port_add_cls_matchall_mirror(struct mlxsw_sp_port *mlxsw_sp_port, struct mlxsw_sp_port_mall_mirror_tc_entry *mirror, - const struct tc_action *a, + const struct flow_action_entry *act, bool ingress) { enum mlxsw_sp_span_type span_type; - struct net_device *to_dev; - to_dev = tcf_mirred_dev(a); - if (!to_dev) { + if (!act->dev) { netdev_err(mlxsw_sp_port->dev, "Could not find requested device\n"); return -EINVAL; } mirror->ingress = ingress; span_type = ingress ? MLXSW_SP_SPAN_INGRESS : MLXSW_SP_SPAN_EGRESS; - return mlxsw_sp_span_mirror_add(mlxsw_sp_port, to_dev, span_type, + return mlxsw_sp_span_mirror_add(mlxsw_sp_port, act->dev, span_type, true, &mirror->span_id); } @@ -1312,7 +1300,7 @@ mlxsw_sp_port_del_cls_matchall_mirror(struct mlxsw_sp_port *mlxsw_sp_port, static int mlxsw_sp_port_add_cls_matchall_sample(struct mlxsw_sp_port *mlxsw_sp_port, struct tc_cls_matchall_offload *cls, - const struct tc_action *a, + const struct flow_action_entry *act, bool ingress) { int err; @@ -1323,18 +1311,18 @@ mlxsw_sp_port_add_cls_matchall_sample(struct mlxsw_sp_port *mlxsw_sp_port, netdev_err(mlxsw_sp_port->dev, "sample already active\n"); return -EEXIST; } - if (tcf_sample_rate(a) > MLXSW_REG_MPSC_RATE_MAX) { + if (act->sample.rate > MLXSW_REG_MPSC_RATE_MAX) { netdev_err(mlxsw_sp_port->dev, "sample rate not supported\n"); return -EOPNOTSUPP; } rcu_assign_pointer(mlxsw_sp_port->sample->psample_group, - tcf_sample_psample_group(a)); - mlxsw_sp_port->sample->truncate = tcf_sample_truncate(a); - mlxsw_sp_port->sample->trunc_size = tcf_sample_trunc_size(a); - mlxsw_sp_port->sample->rate = tcf_sample_rate(a); + act->sample.psample_group); + mlxsw_sp_port->sample->truncate = act->sample.truncate; + mlxsw_sp_port->sample->trunc_size = act->sample.trunc_size; + mlxsw_sp_port->sample->rate = act->sample.rate; - err = mlxsw_sp_port_sample_set(mlxsw_sp_port, true, tcf_sample_rate(a)); + err = mlxsw_sp_port_sample_set(mlxsw_sp_port, true, act->sample.rate); if (err) goto err_port_sample_set; return 0; @@ -1360,10 +1348,10 @@ static int mlxsw_sp_port_add_cls_matchall(struct mlxsw_sp_port *mlxsw_sp_port, { struct mlxsw_sp_port_mall_tc_entry *mall_tc_entry; __be16 protocol = f->common.protocol; - const struct tc_action *a; + struct flow_action_entry *act; int err; - if (!tcf_exts_has_one_action(f->exts)) { + if (!flow_offload_has_one_action(&f->rule->action)) { netdev_err(mlxsw_sp_port->dev, "only singular actions are supported\n"); return -EOPNOTSUPP; } @@ -1373,19 +1361,21 @@ static int mlxsw_sp_port_add_cls_matchall(struct mlxsw_sp_port *mlxsw_sp_port, return -ENOMEM; mall_tc_entry->cookie = f->cookie; - a = tcf_exts_first_action(f->exts); + act = &f->rule->action.entries[0]; - if (is_tcf_mirred_egress_mirror(a) && protocol == htons(ETH_P_ALL)) { + if (act->id == FLOW_ACTION_MIRRED && protocol == htons(ETH_P_ALL)) { struct mlxsw_sp_port_mall_mirror_tc_entry *mirror; mall_tc_entry->type = MLXSW_SP_PORT_MALL_MIRROR; mirror = &mall_tc_entry->mirror; err = mlxsw_sp_port_add_cls_matchall_mirror(mlxsw_sp_port, - mirror, a, ingress); - } else if (is_tcf_sample(a) && protocol == htons(ETH_P_ALL)) { + mirror, act, + ingress); + } else if (act->id == FLOW_ACTION_SAMPLE && + protocol == htons(ETH_P_ALL)) { mall_tc_entry->type = MLXSW_SP_PORT_MALL_SAMPLE; err = mlxsw_sp_port_add_cls_matchall_sample(mlxsw_sp_port, f, - a, ingress); + act, ingress); } else { err = -EOPNOTSUPP; } @@ -1679,6 +1669,25 @@ static int mlxsw_sp_feature_hw_tc(struct net_device *dev, bool enable) return 0; } +static int mlxsw_sp_feature_loopback(struct net_device *dev, bool enable) +{ + struct mlxsw_sp_port *mlxsw_sp_port = netdev_priv(dev); + char pplr_pl[MLXSW_REG_PPLR_LEN]; + int err; + + if (netif_running(dev)) + mlxsw_sp_port_admin_status_set(mlxsw_sp_port, false); + + mlxsw_reg_pplr_pack(pplr_pl, mlxsw_sp_port->local_port, enable); + err = mlxsw_reg_write(mlxsw_sp_port->mlxsw_sp->core, MLXSW_REG(pplr), + pplr_pl); + + if (netif_running(dev)) + mlxsw_sp_port_admin_status_set(mlxsw_sp_port, true); + + return err; +} + typedef int (*mlxsw_sp_feature_handler)(struct net_device *dev, bool enable); static int mlxsw_sp_handle_feature(struct net_device *dev, @@ -1710,20 +1719,30 @@ static int mlxsw_sp_handle_feature(struct net_device *dev, static int mlxsw_sp_set_features(struct net_device *dev, netdev_features_t features) { - return mlxsw_sp_handle_feature(dev, features, NETIF_F_HW_TC, + netdev_features_t oper_features = dev->features; + int err = 0; + + err |= mlxsw_sp_handle_feature(dev, features, NETIF_F_HW_TC, mlxsw_sp_feature_hw_tc); + err |= mlxsw_sp_handle_feature(dev, features, NETIF_F_LOOPBACK, + mlxsw_sp_feature_loopback); + + if (err) { + dev->features = oper_features; + return -EINVAL; + } + + return 0; } -static int mlxsw_sp_port_get_port_parent_id(struct net_device *dev, - struct netdev_phys_item_id *ppid) +static struct devlink_port * +mlxsw_sp_port_get_devlink_port(struct net_device *dev) { struct mlxsw_sp_port *mlxsw_sp_port = netdev_priv(dev); struct mlxsw_sp *mlxsw_sp = mlxsw_sp_port->mlxsw_sp; - ppid->id_len = sizeof(mlxsw_sp->base_mac); - memcpy(&ppid->id, &mlxsw_sp->base_mac, ppid->id_len); - - return 0; + return mlxsw_core_port_devlink_port_get(mlxsw_sp->core, + mlxsw_sp_port->local_port); } static const struct net_device_ops mlxsw_sp_port_netdev_ops = { @@ -1739,9 +1758,8 @@ static const struct net_device_ops mlxsw_sp_port_netdev_ops = { .ndo_get_offload_stats = mlxsw_sp_port_get_offload_stats, .ndo_vlan_rx_add_vid = mlxsw_sp_port_add_vid, .ndo_vlan_rx_kill_vid = mlxsw_sp_port_kill_vid, - .ndo_get_phys_port_name = mlxsw_sp_port_get_phys_port_name, .ndo_set_features = mlxsw_sp_set_features, - .ndo_get_port_parent_id = mlxsw_sp_port_get_port_parent_id, + .ndo_get_devlink_port = mlxsw_sp_port_get_devlink_port, }; static void mlxsw_sp_port_get_drvinfo(struct net_device *dev, @@ -3391,7 +3409,10 @@ static int mlxsw_sp_port_create(struct mlxsw_sp *mlxsw_sp, u8 local_port, struct net_device *dev; int err; - err = mlxsw_core_port_init(mlxsw_sp->core, local_port); + err = mlxsw_core_port_init(mlxsw_sp->core, local_port, + module + 1, split, lane / width, + mlxsw_sp->base_mac, + sizeof(mlxsw_sp->base_mac)); if (err) { dev_err(mlxsw_sp->bus_info->dev, "Port %d: Failed to init core port\n", local_port); @@ -3462,7 +3483,7 @@ static int mlxsw_sp_port_create(struct mlxsw_sp *mlxsw_sp, u8 local_port, dev->features |= NETIF_F_NETNS_LOCAL | NETIF_F_LLTX | NETIF_F_SG | NETIF_F_HW_VLAN_CTAG_FILTER | NETIF_F_HW_TC; - dev->hw_features |= NETIF_F_HW_TC; + dev->hw_features |= NETIF_F_HW_TC | NETIF_F_LOOPBACK; dev->min_mtu = 0; dev->max_mtu = ETH_MAX_MTU; @@ -3573,8 +3594,7 @@ static int mlxsw_sp_port_create(struct mlxsw_sp *mlxsw_sp, u8 local_port, } mlxsw_core_port_eth_set(mlxsw_sp->core, mlxsw_sp_port->local_port, - mlxsw_sp_port, dev, module + 1, - mlxsw_sp_port->split, lane / width); + mlxsw_sp_port, dev); mlxsw_core_schedule_dw(&mlxsw_sp_port->periodic_hw_stats.update_dw, 0); return 0; @@ -3710,14 +3730,14 @@ static u8 mlxsw_sp_cluster_base_port_get(u8 local_port) } static int mlxsw_sp_port_split_create(struct mlxsw_sp *mlxsw_sp, u8 base_port, - u8 module, unsigned int count) + u8 module, unsigned int count, u8 offset) { u8 width = MLXSW_PORT_MODULE_MAX_WIDTH / count; int err, i; for (i = 0; i < count; i++) { - err = mlxsw_sp_port_create(mlxsw_sp, base_port + i, true, - module, width, i * width); + err = mlxsw_sp_port_create(mlxsw_sp, base_port + i * offset, + true, module, width, i * width); if (err) goto err_port_create; } @@ -3726,8 +3746,8 @@ static int mlxsw_sp_port_split_create(struct mlxsw_sp *mlxsw_sp, u8 base_port, err_port_create: for (i--; i >= 0; i--) - if (mlxsw_sp_port_created(mlxsw_sp, base_port + i)) - mlxsw_sp_port_remove(mlxsw_sp, base_port + i); + if (mlxsw_sp_port_created(mlxsw_sp, base_port + i * offset)) + mlxsw_sp_port_remove(mlxsw_sp, base_port + i * offset); return err; } @@ -3758,11 +3778,19 @@ static int mlxsw_sp_port_split(struct mlxsw_core *mlxsw_core, u8 local_port, struct netlink_ext_ack *extack) { struct mlxsw_sp *mlxsw_sp = mlxsw_core_driver_priv(mlxsw_core); + u8 local_ports_in_1x, local_ports_in_2x, offset; struct mlxsw_sp_port *mlxsw_sp_port; u8 module, cur_width, base_port; int i; int err; + if (!MLXSW_CORE_RES_VALID(mlxsw_core, LOCAL_PORTS_IN_1X) || + !MLXSW_CORE_RES_VALID(mlxsw_core, LOCAL_PORTS_IN_2X)) + return -EIO; + + local_ports_in_1x = MLXSW_CORE_RES_GET(mlxsw_core, LOCAL_PORTS_IN_1X); + local_ports_in_2x = MLXSW_CORE_RES_GET(mlxsw_core, LOCAL_PORTS_IN_2X); + mlxsw_sp_port = mlxsw_sp->ports[local_port]; if (!mlxsw_sp_port) { dev_err(mlxsw_sp->bus_info->dev, "Port number \"%d\" does not exist\n", @@ -3788,13 +3816,15 @@ static int mlxsw_sp_port_split(struct mlxsw_core *mlxsw_core, u8 local_port, /* Make sure we have enough slave (even) ports for the split. */ if (count == 2) { + offset = local_ports_in_2x; base_port = local_port; - if (mlxsw_sp->ports[base_port + 1]) { + if (mlxsw_sp->ports[base_port + local_ports_in_2x]) { netdev_err(mlxsw_sp_port->dev, "Invalid split configuration\n"); NL_SET_ERR_MSG_MOD(extack, "Invalid split configuration"); return -EINVAL; } } else { + offset = local_ports_in_1x; base_port = mlxsw_sp_cluster_base_port_get(local_port); if (mlxsw_sp->ports[base_port + 1] || mlxsw_sp->ports[base_port + 3]) { @@ -3805,10 +3835,11 @@ static int mlxsw_sp_port_split(struct mlxsw_core *mlxsw_core, u8 local_port, } for (i = 0; i < count; i++) - if (mlxsw_sp_port_created(mlxsw_sp, base_port + i)) - mlxsw_sp_port_remove(mlxsw_sp, base_port + i); + if (mlxsw_sp_port_created(mlxsw_sp, base_port + i * offset)) + mlxsw_sp_port_remove(mlxsw_sp, base_port + i * offset); - err = mlxsw_sp_port_split_create(mlxsw_sp, base_port, module, count); + err = mlxsw_sp_port_split_create(mlxsw_sp, base_port, module, count, + offset); if (err) { dev_err(mlxsw_sp->bus_info->dev, "Failed to create split ports\n"); goto err_port_split_create; @@ -3825,11 +3856,19 @@ static int mlxsw_sp_port_unsplit(struct mlxsw_core *mlxsw_core, u8 local_port, struct netlink_ext_ack *extack) { struct mlxsw_sp *mlxsw_sp = mlxsw_core_driver_priv(mlxsw_core); + u8 local_ports_in_1x, local_ports_in_2x, offset; struct mlxsw_sp_port *mlxsw_sp_port; u8 cur_width, base_port; unsigned int count; int i; + if (!MLXSW_CORE_RES_VALID(mlxsw_core, LOCAL_PORTS_IN_1X) || + !MLXSW_CORE_RES_VALID(mlxsw_core, LOCAL_PORTS_IN_2X)) + return -EIO; + + local_ports_in_1x = MLXSW_CORE_RES_GET(mlxsw_core, LOCAL_PORTS_IN_1X); + local_ports_in_2x = MLXSW_CORE_RES_GET(mlxsw_core, LOCAL_PORTS_IN_2X); + mlxsw_sp_port = mlxsw_sp->ports[local_port]; if (!mlxsw_sp_port) { dev_err(mlxsw_sp->bus_info->dev, "Port number \"%d\" does not exist\n", @@ -3847,6 +3886,11 @@ static int mlxsw_sp_port_unsplit(struct mlxsw_core *mlxsw_core, u8 local_port, cur_width = mlxsw_sp_port->mapping.width; count = cur_width == 1 ? 4 : 2; + if (count == 2) + offset = local_ports_in_2x; + else + offset = local_ports_in_1x; + base_port = mlxsw_sp_cluster_base_port_get(local_port); /* Determine which ports to remove. */ @@ -3854,8 +3898,8 @@ static int mlxsw_sp_port_unsplit(struct mlxsw_core *mlxsw_core, u8 local_port, base_port = base_port + 2; for (i = 0; i < count; i++) - if (mlxsw_sp_port_created(mlxsw_sp, base_port + i)) - mlxsw_sp_port_remove(mlxsw_sp, base_port + i); + if (mlxsw_sp_port_created(mlxsw_sp, base_port + i * offset)) + mlxsw_sp_port_remove(mlxsw_sp, base_port + i * offset); mlxsw_sp_port_unsplit_create(mlxsw_sp, base_port, count); @@ -4238,7 +4282,7 @@ static int mlxsw_sp_lag_init(struct mlxsw_sp *mlxsw_sp) u32 seed; int err; - get_random_bytes(&seed, sizeof(seed)); + seed = jhash(mlxsw_sp->base_mac, sizeof(mlxsw_sp->base_mac), 0); mlxsw_reg_slcr_pack(slcr_pl, MLXSW_REG_SLCR_LAG_HASH_SMAC | MLXSW_REG_SLCR_LAG_HASH_DMAC | MLXSW_REG_SLCR_LAG_HASH_ETHERTYPE | diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum.h b/drivers/net/ethernet/mellanox/mlxsw/spectrum.h index da6278b0caa4992f5454d8e4a61ab6a8dafb854c..8601b3041acd9083365b6a23e017bbaa05a989bd 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/spectrum.h +++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum.h @@ -371,13 +371,14 @@ int mlxsw_sp_sb_pool_get(struct mlxsw_core *mlxsw_core, struct devlink_sb_pool_info *pool_info); int mlxsw_sp_sb_pool_set(struct mlxsw_core *mlxsw_core, unsigned int sb_index, u16 pool_index, u32 size, - enum devlink_sb_threshold_type threshold_type); + enum devlink_sb_threshold_type threshold_type, + struct netlink_ext_ack *extack); int mlxsw_sp_sb_port_pool_get(struct mlxsw_core_port *mlxsw_core_port, unsigned int sb_index, u16 pool_index, u32 *p_threshold); int mlxsw_sp_sb_port_pool_set(struct mlxsw_core_port *mlxsw_core_port, unsigned int sb_index, u16 pool_index, - u32 threshold); + u32 threshold, struct netlink_ext_ack *extack); int mlxsw_sp_sb_tc_pool_bind_get(struct mlxsw_core_port *mlxsw_core_port, unsigned int sb_index, u16 tc_index, enum devlink_sb_pool_type pool_type, @@ -385,7 +386,8 @@ int mlxsw_sp_sb_tc_pool_bind_get(struct mlxsw_core_port *mlxsw_core_port, int mlxsw_sp_sb_tc_pool_bind_set(struct mlxsw_core_port *mlxsw_core_port, unsigned int sb_index, u16 tc_index, enum devlink_sb_pool_type pool_type, - u16 pool_index, u32 threshold); + u16 pool_index, u32 threshold, + struct netlink_ext_ack *extack); int mlxsw_sp_sb_occ_snapshot(struct mlxsw_core *mlxsw_core, unsigned int sb_index); int mlxsw_sp_sb_occ_max_clear(struct mlxsw_core *mlxsw_core, diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl_tcam.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl_tcam.c index 8811f6513e36f7cddd13dc77580f6921065e105f..e993159e8e4cd7f8c16633ebcdc06a87a3a67be9 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl_tcam.c +++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl_tcam.c @@ -216,7 +216,6 @@ struct mlxsw_sp_acl_tcam_vregion { struct mlxsw_sp_acl_tcam_rehash_ctx ctx; } rehash; struct mlxsw_sp *mlxsw_sp; - bool failed_rollback; /* Indicates failed rollback during migration */ unsigned int ref_count; }; @@ -1256,11 +1255,8 @@ mlxsw_sp_acl_tcam_vchunk_migrate_start(struct mlxsw_sp *mlxsw_sp, struct mlxsw_sp_acl_tcam_chunk *new_chunk; new_chunk = mlxsw_sp_acl_tcam_chunk_create(mlxsw_sp, vchunk, region); - if (IS_ERR(new_chunk)) { - if (ctx->this_is_rollback) - vchunk->vregion->failed_rollback = true; + if (IS_ERR(new_chunk)) return PTR_ERR(new_chunk); - } vchunk->chunk2 = vchunk->chunk; vchunk->chunk = new_chunk; ctx->current_vchunk = vchunk; @@ -1318,8 +1314,13 @@ mlxsw_sp_acl_tcam_vchunk_migrate_one(struct mlxsw_sp *mlxsw_sp, err = mlxsw_sp_acl_tcam_ventry_migrate(mlxsw_sp, ventry, vchunk->chunk, credits); if (err) { - if (ctx->this_is_rollback) + if (ctx->this_is_rollback) { + /* Save the ventry which we ended with and try + * to continue later on. + */ + ctx->start_ventry = ventry; return err; + } /* Swap the chunk and chunk2 pointers so the follow-up * rollback call will see the original chunk pointer * in vchunk->chunk. @@ -1397,8 +1398,12 @@ mlxsw_sp_acl_tcam_vregion_migrate(struct mlxsw_sp *mlxsw_sp, ctx->this_is_rollback = true; err2 = mlxsw_sp_acl_tcam_vchunk_migrate_all(mlxsw_sp, vregion, ctx, credits); - if (err2) - vregion->failed_rollback = true; + if (err2) { + trace_mlxsw_sp_acl_tcam_vregion_rehash_rollback_failed(mlxsw_sp, + vregion); + dev_err(mlxsw_sp->bus_info->dev, "Failed to rollback during vregion migration fail\n"); + /* Let the rollback to be continued later on. */ + } } mutex_unlock(&vregion->lock); trace_mlxsw_sp_acl_tcam_vregion_migrate_end(mlxsw_sp, vregion); @@ -1423,8 +1428,6 @@ mlxsw_sp_acl_tcam_vregion_rehash_start(struct mlxsw_sp *mlxsw_sp, int err; trace_mlxsw_sp_acl_tcam_vregion_rehash(mlxsw_sp, vregion); - if (vregion->failed_rollback) - return -EBUSY; hints_priv = ops->region_rehash_hints_get(vregion->region->priv); if (IS_ERR(hints_priv)) @@ -1471,11 +1474,9 @@ mlxsw_sp_acl_tcam_vregion_rehash_end(struct mlxsw_sp *mlxsw_sp, struct mlxsw_sp_acl_tcam_region *unused_region = vregion->region2; const struct mlxsw_sp_acl_tcam_ops *ops = mlxsw_sp->acl_tcam_ops; - if (!vregion->failed_rollback) { - vregion->region2 = NULL; - mlxsw_sp_acl_tcam_group_region_detach(mlxsw_sp, unused_region); - mlxsw_sp_acl_tcam_region_destroy(mlxsw_sp, unused_region); - } + vregion->region2 = NULL; + mlxsw_sp_acl_tcam_group_region_detach(mlxsw_sp, unused_region); + mlxsw_sp_acl_tcam_region_destroy(mlxsw_sp, unused_region); ops->region_rehash_hints_put(ctx->hints_priv); ctx->hints_priv = NULL; } @@ -1506,11 +1507,6 @@ mlxsw_sp_acl_tcam_vregion_rehash(struct mlxsw_sp *mlxsw_sp, ctx, credits); if (err) { dev_err(mlxsw_sp->bus_info->dev, "Failed to migrate vregion\n"); - if (vregion->failed_rollback) { - trace_mlxsw_sp_acl_tcam_vregion_rehash_dis(mlxsw_sp, - vregion); - dev_err(mlxsw_sp->bus_info->dev, "Failed to rollback during vregion migration fail\n"); - } } if (*credits >= 0) diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_buffers.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_buffers.c index d633bef5f10512269547c00f718f552720dd29a3..8512dd49e4201abbbda983b6cb2889ef2853e829 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_buffers.c +++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_buffers.c @@ -6,6 +6,7 @@ #include #include #include +#include #include "spectrum.h" #include "core.h" @@ -15,6 +16,8 @@ struct mlxsw_sp_sb_pr { enum mlxsw_reg_sbpr_mode mode; u32 size; + u8 freeze_mode:1, + freeze_size:1; }; struct mlxsw_cp_sb_occ { @@ -27,6 +30,8 @@ struct mlxsw_sp_sb_cm { u32 max_buff; u16 pool_index; struct mlxsw_cp_sb_occ occ; + u8 freeze_pool:1, + freeze_thresh:1; }; #define MLXSW_SP_SB_INFI -1U @@ -48,7 +53,12 @@ struct mlxsw_sp_sb_pool_des { u8 pool; }; -/* Order ingress pools before egress pools. */ +#define MLXSW_SP_SB_POOL_ING 0 +#define MLXSW_SP_SB_POOL_EGR 4 +#define MLXSW_SP_SB_POOL_EGR_MC 8 +#define MLXSW_SP_SB_POOL_ING_CPU 9 +#define MLXSW_SP_SB_POOL_EGR_CPU 10 + static const struct mlxsw_sp_sb_pool_des mlxsw_sp1_sb_pool_dess[] = { {MLXSW_REG_SBXX_DIR_INGRESS, 0}, {MLXSW_REG_SBXX_DIR_INGRESS, 1}, @@ -59,6 +69,8 @@ static const struct mlxsw_sp_sb_pool_des mlxsw_sp1_sb_pool_dess[] = { {MLXSW_REG_SBXX_DIR_EGRESS, 2}, {MLXSW_REG_SBXX_DIR_EGRESS, 3}, {MLXSW_REG_SBXX_DIR_EGRESS, 15}, + {MLXSW_REG_SBXX_DIR_INGRESS, 4}, + {MLXSW_REG_SBXX_DIR_EGRESS, 4}, }; static const struct mlxsw_sp_sb_pool_des mlxsw_sp2_sb_pool_dess[] = { @@ -71,6 +83,8 @@ static const struct mlxsw_sp_sb_pool_des mlxsw_sp2_sb_pool_dess[] = { {MLXSW_REG_SBXX_DIR_EGRESS, 2}, {MLXSW_REG_SBXX_DIR_EGRESS, 3}, {MLXSW_REG_SBXX_DIR_EGRESS, 15}, + {MLXSW_REG_SBXX_DIR_INGRESS, 4}, + {MLXSW_REG_SBXX_DIR_EGRESS, 4}, }; #define MLXSW_SP_SB_ING_TC_COUNT 8 @@ -94,6 +108,7 @@ struct mlxsw_sp_sb_vals { unsigned int pool_count; const struct mlxsw_sp_sb_pool_des *pool_dess; const struct mlxsw_sp_sb_pm *pms; + const struct mlxsw_sp_sb_pm *pms_cpu; const struct mlxsw_sp_sb_pr *prs; const struct mlxsw_sp_sb_mm *mms; const struct mlxsw_sp_sb_cm *cms_ingress; @@ -275,7 +290,7 @@ static int mlxsw_sp_port_pb_init(struct mlxsw_sp_port *mlxsw_sp_port) { const u32 pbs[] = { [0] = MLXSW_SP_PB_HEADROOM * mlxsw_sp_port->mapping.width, - [9] = 2 * MLXSW_PORT_MAX_MTU, + [9] = MLXSW_PORT_MAX_MTU, }; struct mlxsw_sp *mlxsw_sp = mlxsw_sp_port->mlxsw_sp; char pbmc_pl[MLXSW_REG_PBMC_LEN]; @@ -390,46 +405,60 @@ static void mlxsw_sp_sb_ports_fini(struct mlxsw_sp *mlxsw_sp) .size = _size, \ } +#define MLXSW_SP_SB_PR_EXT(_mode, _size, _freeze_mode, _freeze_size) \ + { \ + .mode = _mode, \ + .size = _size, \ + .freeze_mode = _freeze_mode, \ + .freeze_size = _freeze_size, \ + } + #define MLXSW_SP1_SB_PR_INGRESS_SIZE 12440000 -#define MLXSW_SP1_SB_PR_INGRESS_MNG_SIZE (200 * 1000) #define MLXSW_SP1_SB_PR_EGRESS_SIZE 13232000 +#define MLXSW_SP1_SB_PR_CPU_SIZE (256 * 1000) +/* Order according to mlxsw_sp1_sb_pool_dess */ static const struct mlxsw_sp_sb_pr mlxsw_sp1_sb_prs[] = { - /* Ingress pools. */ MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC, MLXSW_SP1_SB_PR_INGRESS_SIZE), MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC, 0), MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC, 0), - MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC, - MLXSW_SP1_SB_PR_INGRESS_MNG_SIZE), - /* Egress pools. */ - MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC, - MLXSW_SP1_SB_PR_EGRESS_SIZE), + MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC, 0), + MLXSW_SP_SB_PR_EXT(MLXSW_REG_SBPR_MODE_DYNAMIC, + MLXSW_SP1_SB_PR_EGRESS_SIZE, true, false), MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC, 0), MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC, 0), MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC, 0), - MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_STATIC, MLXSW_SP_SB_INFI), + MLXSW_SP_SB_PR_EXT(MLXSW_REG_SBPR_MODE_STATIC, MLXSW_SP_SB_INFI, + true, true), + MLXSW_SP_SB_PR_EXT(MLXSW_REG_SBPR_MODE_DYNAMIC, + MLXSW_SP1_SB_PR_CPU_SIZE, true, false), + MLXSW_SP_SB_PR_EXT(MLXSW_REG_SBPR_MODE_DYNAMIC, + MLXSW_SP1_SB_PR_CPU_SIZE, true, false), }; #define MLXSW_SP2_SB_PR_INGRESS_SIZE 40960000 -#define MLXSW_SP2_SB_PR_INGRESS_MNG_SIZE (200 * 1000) #define MLXSW_SP2_SB_PR_EGRESS_SIZE 40960000 +#define MLXSW_SP2_SB_PR_CPU_SIZE (256 * 1000) +/* Order according to mlxsw_sp2_sb_pool_dess */ static const struct mlxsw_sp_sb_pr mlxsw_sp2_sb_prs[] = { - /* Ingress pools. */ MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC, MLXSW_SP2_SB_PR_INGRESS_SIZE), MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_STATIC, 0), MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_STATIC, 0), - MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC, - MLXSW_SP2_SB_PR_INGRESS_MNG_SIZE), - /* Egress pools. */ - MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC, - MLXSW_SP2_SB_PR_EGRESS_SIZE), + MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_STATIC, 0), + MLXSW_SP_SB_PR_EXT(MLXSW_REG_SBPR_MODE_DYNAMIC, + MLXSW_SP2_SB_PR_EGRESS_SIZE, true, false), MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_STATIC, 0), MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_STATIC, 0), MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_STATIC, 0), - MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_STATIC, MLXSW_SP_SB_INFI), + MLXSW_SP_SB_PR_EXT(MLXSW_REG_SBPR_MODE_STATIC, MLXSW_SP_SB_INFI, + true, true), + MLXSW_SP_SB_PR_EXT(MLXSW_REG_SBPR_MODE_DYNAMIC, + MLXSW_SP2_SB_PR_CPU_SIZE, true, false), + MLXSW_SP_SB_PR_EXT(MLXSW_REG_SBPR_MODE_DYNAMIC, + MLXSW_SP2_SB_PR_CPU_SIZE, true, false), }; static int mlxsw_sp_sb_prs_init(struct mlxsw_sp *mlxsw_sp, @@ -464,83 +493,106 @@ static int mlxsw_sp_sb_prs_init(struct mlxsw_sp *mlxsw_sp, .pool_index = _pool, \ } +#define MLXSW_SP_SB_CM_ING(_min_buff, _max_buff) \ + { \ + .min_buff = _min_buff, \ + .max_buff = _max_buff, \ + .pool_index = MLXSW_SP_SB_POOL_ING, \ + } + +#define MLXSW_SP_SB_CM_EGR(_min_buff, _max_buff) \ + { \ + .min_buff = _min_buff, \ + .max_buff = _max_buff, \ + .pool_index = MLXSW_SP_SB_POOL_EGR, \ + } + +#define MLXSW_SP_SB_CM_EGR_MC(_min_buff, _max_buff) \ + { \ + .min_buff = _min_buff, \ + .max_buff = _max_buff, \ + .pool_index = MLXSW_SP_SB_POOL_EGR_MC, \ + .freeze_pool = true, \ + .freeze_thresh = true, \ + } + static const struct mlxsw_sp_sb_cm mlxsw_sp1_sb_cms_ingress[] = { - MLXSW_SP_SB_CM(10000, 8, 0), - MLXSW_SP_SB_CM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN, 0), - MLXSW_SP_SB_CM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN, 0), - MLXSW_SP_SB_CM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN, 0), - MLXSW_SP_SB_CM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN, 0), - MLXSW_SP_SB_CM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN, 0), - MLXSW_SP_SB_CM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN, 0), - MLXSW_SP_SB_CM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN, 0), - MLXSW_SP_SB_CM(0, 0, 0), /* dummy, this PG does not exist */ - MLXSW_SP_SB_CM(20000, 1, 3), + MLXSW_SP_SB_CM_ING(10000, 8), + MLXSW_SP_SB_CM_ING(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), + MLXSW_SP_SB_CM_ING(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), + MLXSW_SP_SB_CM_ING(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), + MLXSW_SP_SB_CM_ING(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), + MLXSW_SP_SB_CM_ING(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), + MLXSW_SP_SB_CM_ING(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), + MLXSW_SP_SB_CM_ING(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), + MLXSW_SP_SB_CM_ING(0, 0), /* dummy, this PG does not exist */ + MLXSW_SP_SB_CM(10000, 8, MLXSW_SP_SB_POOL_ING_CPU), }; static const struct mlxsw_sp_sb_cm mlxsw_sp2_sb_cms_ingress[] = { - MLXSW_SP_SB_CM(0, 7, 0), - MLXSW_SP_SB_CM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN, 0), - MLXSW_SP_SB_CM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN, 0), - MLXSW_SP_SB_CM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN, 0), - MLXSW_SP_SB_CM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN, 0), - MLXSW_SP_SB_CM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN, 0), - MLXSW_SP_SB_CM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN, 0), - MLXSW_SP_SB_CM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN, 0), - MLXSW_SP_SB_CM(0, 0, 0), /* dummy, this PG does not exist */ - MLXSW_SP_SB_CM(20000, 1, 3), + MLXSW_SP_SB_CM_ING(0, 7), + MLXSW_SP_SB_CM_ING(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), + MLXSW_SP_SB_CM_ING(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), + MLXSW_SP_SB_CM_ING(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), + MLXSW_SP_SB_CM_ING(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), + MLXSW_SP_SB_CM_ING(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), + MLXSW_SP_SB_CM_ING(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), + MLXSW_SP_SB_CM_ING(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), + MLXSW_SP_SB_CM_ING(0, 0), /* dummy, this PG does not exist */ + MLXSW_SP_SB_CM(10000, 8, MLXSW_SP_SB_POOL_ING_CPU), }; static const struct mlxsw_sp_sb_cm mlxsw_sp1_sb_cms_egress[] = { - MLXSW_SP_SB_CM(1500, 9, 4), - MLXSW_SP_SB_CM(1500, 9, 4), - MLXSW_SP_SB_CM(1500, 9, 4), - MLXSW_SP_SB_CM(1500, 9, 4), - MLXSW_SP_SB_CM(1500, 9, 4), - MLXSW_SP_SB_CM(1500, 9, 4), - MLXSW_SP_SB_CM(1500, 9, 4), - MLXSW_SP_SB_CM(1500, 9, 4), - MLXSW_SP_SB_CM(0, MLXSW_SP_SB_INFI, 8), - MLXSW_SP_SB_CM(0, MLXSW_SP_SB_INFI, 8), - MLXSW_SP_SB_CM(0, MLXSW_SP_SB_INFI, 8), - MLXSW_SP_SB_CM(0, MLXSW_SP_SB_INFI, 8), - MLXSW_SP_SB_CM(0, MLXSW_SP_SB_INFI, 8), - MLXSW_SP_SB_CM(0, MLXSW_SP_SB_INFI, 8), - MLXSW_SP_SB_CM(0, MLXSW_SP_SB_INFI, 8), - MLXSW_SP_SB_CM(0, MLXSW_SP_SB_INFI, 8), - MLXSW_SP_SB_CM(1, 0xff, 4), + MLXSW_SP_SB_CM_EGR(1500, 9), + MLXSW_SP_SB_CM_EGR(1500, 9), + MLXSW_SP_SB_CM_EGR(1500, 9), + MLXSW_SP_SB_CM_EGR(1500, 9), + MLXSW_SP_SB_CM_EGR(1500, 9), + MLXSW_SP_SB_CM_EGR(1500, 9), + MLXSW_SP_SB_CM_EGR(1500, 9), + MLXSW_SP_SB_CM_EGR(1500, 9), + MLXSW_SP_SB_CM_EGR_MC(0, MLXSW_SP_SB_INFI), + MLXSW_SP_SB_CM_EGR_MC(0, MLXSW_SP_SB_INFI), + MLXSW_SP_SB_CM_EGR_MC(0, MLXSW_SP_SB_INFI), + MLXSW_SP_SB_CM_EGR_MC(0, MLXSW_SP_SB_INFI), + MLXSW_SP_SB_CM_EGR_MC(0, MLXSW_SP_SB_INFI), + MLXSW_SP_SB_CM_EGR_MC(0, MLXSW_SP_SB_INFI), + MLXSW_SP_SB_CM_EGR_MC(0, MLXSW_SP_SB_INFI), + MLXSW_SP_SB_CM_EGR_MC(0, MLXSW_SP_SB_INFI), + MLXSW_SP_SB_CM_EGR(1, 0xff), }; static const struct mlxsw_sp_sb_cm mlxsw_sp2_sb_cms_egress[] = { - MLXSW_SP_SB_CM(0, 7, 4), - MLXSW_SP_SB_CM(0, 7, 4), - MLXSW_SP_SB_CM(0, 7, 4), - MLXSW_SP_SB_CM(0, 7, 4), - MLXSW_SP_SB_CM(0, 7, 4), - MLXSW_SP_SB_CM(0, 7, 4), - MLXSW_SP_SB_CM(0, 7, 4), - MLXSW_SP_SB_CM(0, 7, 4), - MLXSW_SP_SB_CM(0, MLXSW_SP_SB_INFI, 8), - MLXSW_SP_SB_CM(0, MLXSW_SP_SB_INFI, 8), - MLXSW_SP_SB_CM(0, MLXSW_SP_SB_INFI, 8), - MLXSW_SP_SB_CM(0, MLXSW_SP_SB_INFI, 8), - MLXSW_SP_SB_CM(0, MLXSW_SP_SB_INFI, 8), - MLXSW_SP_SB_CM(0, MLXSW_SP_SB_INFI, 8), - MLXSW_SP_SB_CM(0, MLXSW_SP_SB_INFI, 8), - MLXSW_SP_SB_CM(0, MLXSW_SP_SB_INFI, 8), - MLXSW_SP_SB_CM(1, 0xff, 4), + MLXSW_SP_SB_CM_EGR(0, 7), + MLXSW_SP_SB_CM_EGR(0, 7), + MLXSW_SP_SB_CM_EGR(0, 7), + MLXSW_SP_SB_CM_EGR(0, 7), + MLXSW_SP_SB_CM_EGR(0, 7), + MLXSW_SP_SB_CM_EGR(0, 7), + MLXSW_SP_SB_CM_EGR(0, 7), + MLXSW_SP_SB_CM_EGR(0, 7), + MLXSW_SP_SB_CM_EGR_MC(0, MLXSW_SP_SB_INFI), + MLXSW_SP_SB_CM_EGR_MC(0, MLXSW_SP_SB_INFI), + MLXSW_SP_SB_CM_EGR_MC(0, MLXSW_SP_SB_INFI), + MLXSW_SP_SB_CM_EGR_MC(0, MLXSW_SP_SB_INFI), + MLXSW_SP_SB_CM_EGR_MC(0, MLXSW_SP_SB_INFI), + MLXSW_SP_SB_CM_EGR_MC(0, MLXSW_SP_SB_INFI), + MLXSW_SP_SB_CM_EGR_MC(0, MLXSW_SP_SB_INFI), + MLXSW_SP_SB_CM_EGR_MC(0, MLXSW_SP_SB_INFI), + MLXSW_SP_SB_CM_EGR(1, 0xff), }; -#define MLXSW_SP_CPU_PORT_SB_CM MLXSW_SP_SB_CM(0, 0, 4) +#define MLXSW_SP_CPU_PORT_SB_CM MLXSW_SP_SB_CM(0, 0, MLXSW_SP_SB_POOL_EGR_CPU) static const struct mlxsw_sp_sb_cm mlxsw_sp_cpu_port_sb_cms[] = { MLXSW_SP_CPU_PORT_SB_CM, - MLXSW_SP_SB_CM(MLXSW_PORT_MAX_MTU, 0, 4), - MLXSW_SP_SB_CM(MLXSW_PORT_MAX_MTU, 0, 4), - MLXSW_SP_SB_CM(MLXSW_PORT_MAX_MTU, 0, 4), - MLXSW_SP_SB_CM(MLXSW_PORT_MAX_MTU, 0, 4), - MLXSW_SP_SB_CM(MLXSW_PORT_MAX_MTU, 0, 4), + MLXSW_SP_SB_CM(1000, 8, MLXSW_SP_SB_POOL_EGR_CPU), + MLXSW_SP_SB_CM(1000, 8, MLXSW_SP_SB_POOL_EGR_CPU), + MLXSW_SP_SB_CM(1000, 8, MLXSW_SP_SB_POOL_EGR_CPU), + MLXSW_SP_SB_CM(1000, 8, MLXSW_SP_SB_POOL_EGR_CPU), + MLXSW_SP_SB_CM(1000, 8, MLXSW_SP_SB_POOL_EGR_CPU), MLXSW_SP_CPU_PORT_SB_CM, - MLXSW_SP_SB_CM(MLXSW_PORT_MAX_MTU, 0, 4), + MLXSW_SP_SB_CM(1000, 8, MLXSW_SP_SB_POOL_EGR_CPU), MLXSW_SP_CPU_PORT_SB_CM, MLXSW_SP_CPU_PORT_SB_CM, MLXSW_SP_CPU_PORT_SB_CM, @@ -648,80 +700,116 @@ static int mlxsw_sp_cpu_port_sb_cms_init(struct mlxsw_sp *mlxsw_sp) .max_buff = _max_buff, \ } +/* Order according to mlxsw_sp1_sb_pool_dess */ static const struct mlxsw_sp_sb_pm mlxsw_sp1_sb_pms[] = { - /* Ingress pools. */ MLXSW_SP_SB_PM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MAX), MLXSW_SP_SB_PM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), MLXSW_SP_SB_PM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), - MLXSW_SP_SB_PM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MAX), - /* Egress pools. */ + MLXSW_SP_SB_PM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), MLXSW_SP_SB_PM(0, 7), MLXSW_SP_SB_PM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), MLXSW_SP_SB_PM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), MLXSW_SP_SB_PM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), MLXSW_SP_SB_PM(10000, 90000), + MLXSW_SP_SB_PM(0, 8), /* 50% occupancy */ + MLXSW_SP_SB_PM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), }; +/* Order according to mlxsw_sp2_sb_pool_dess */ static const struct mlxsw_sp_sb_pm mlxsw_sp2_sb_pms[] = { - /* Ingress pools. */ MLXSW_SP_SB_PM(0, 7), MLXSW_SP_SB_PM(0, 0), MLXSW_SP_SB_PM(0, 0), - MLXSW_SP_SB_PM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MAX), - /* Egress pools. */ + MLXSW_SP_SB_PM(0, 0), MLXSW_SP_SB_PM(0, 7), MLXSW_SP_SB_PM(0, 0), MLXSW_SP_SB_PM(0, 0), MLXSW_SP_SB_PM(0, 0), MLXSW_SP_SB_PM(10000, 90000), + MLXSW_SP_SB_PM(0, 8), /* 50% occupancy */ + MLXSW_SP_SB_PM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN), }; -static int mlxsw_sp_port_sb_pms_init(struct mlxsw_sp_port *mlxsw_sp_port) +/* Order according to mlxsw_sp*_sb_pool_dess */ +static const struct mlxsw_sp_sb_pm mlxsw_sp_cpu_port_sb_pms[] = { + MLXSW_SP_SB_PM(0, 0), + MLXSW_SP_SB_PM(0, 0), + MLXSW_SP_SB_PM(0, 0), + MLXSW_SP_SB_PM(0, 0), + MLXSW_SP_SB_PM(0, 0), + MLXSW_SP_SB_PM(0, 0), + MLXSW_SP_SB_PM(0, 0), + MLXSW_SP_SB_PM(0, 0), + MLXSW_SP_SB_PM(0, 90000), + MLXSW_SP_SB_PM(0, 0), + MLXSW_SP_SB_PM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MAX), +}; + +static int mlxsw_sp_sb_pms_init(struct mlxsw_sp *mlxsw_sp, u8 local_port, + const struct mlxsw_sp_sb_pm *pms, + bool skip_ingress) { - struct mlxsw_sp *mlxsw_sp = mlxsw_sp_port->mlxsw_sp; - int i; - int err; + int i, err; for (i = 0; i < mlxsw_sp->sb_vals->pool_count; i++) { - const struct mlxsw_sp_sb_pm *pm = &mlxsw_sp->sb_vals->pms[i]; + const struct mlxsw_sp_sb_pm *pm = &pms[i]; + const struct mlxsw_sp_sb_pool_des *des; u32 max_buff; u32 min_buff; + des = &mlxsw_sp->sb_vals->pool_dess[i]; + if (skip_ingress && des->dir == MLXSW_REG_SBXX_DIR_INGRESS) + continue; + min_buff = mlxsw_sp_bytes_cells(mlxsw_sp, pm->min_buff); max_buff = pm->max_buff; if (mlxsw_sp_sb_pool_is_static(mlxsw_sp, i)) max_buff = mlxsw_sp_bytes_cells(mlxsw_sp, max_buff); - err = mlxsw_sp_sb_pm_write(mlxsw_sp, mlxsw_sp_port->local_port, - i, min_buff, max_buff); + err = mlxsw_sp_sb_pm_write(mlxsw_sp, local_port, i, min_buff, + max_buff); if (err) return err; } return 0; } -#define MLXSW_SP_SB_MM(_min_buff, _max_buff, _pool) \ +static int mlxsw_sp_port_sb_pms_init(struct mlxsw_sp_port *mlxsw_sp_port) +{ + struct mlxsw_sp *mlxsw_sp = mlxsw_sp_port->mlxsw_sp; + + return mlxsw_sp_sb_pms_init(mlxsw_sp, mlxsw_sp_port->local_port, + mlxsw_sp->sb_vals->pms, false); +} + +static int mlxsw_sp_cpu_port_sb_pms_init(struct mlxsw_sp *mlxsw_sp) +{ + return mlxsw_sp_sb_pms_init(mlxsw_sp, 0, mlxsw_sp->sb_vals->pms_cpu, + true); +} + +#define MLXSW_SP_SB_MM(_min_buff, _max_buff) \ { \ .min_buff = _min_buff, \ .max_buff = _max_buff, \ - .pool_index = _pool, \ + .pool_index = MLXSW_SP_SB_POOL_EGR, \ } static const struct mlxsw_sp_sb_mm mlxsw_sp_sb_mms[] = { - MLXSW_SP_SB_MM(0, 6, 4), - MLXSW_SP_SB_MM(0, 6, 4), - MLXSW_SP_SB_MM(0, 6, 4), - MLXSW_SP_SB_MM(0, 6, 4), - MLXSW_SP_SB_MM(0, 6, 4), - MLXSW_SP_SB_MM(0, 6, 4), - MLXSW_SP_SB_MM(0, 6, 4), - MLXSW_SP_SB_MM(0, 6, 4), - MLXSW_SP_SB_MM(0, 6, 4), - MLXSW_SP_SB_MM(0, 6, 4), - MLXSW_SP_SB_MM(0, 6, 4), - MLXSW_SP_SB_MM(0, 6, 4), - MLXSW_SP_SB_MM(0, 6, 4), - MLXSW_SP_SB_MM(0, 6, 4), - MLXSW_SP_SB_MM(0, 6, 4), + MLXSW_SP_SB_MM(0, 6), + MLXSW_SP_SB_MM(0, 6), + MLXSW_SP_SB_MM(0, 6), + MLXSW_SP_SB_MM(0, 6), + MLXSW_SP_SB_MM(0, 6), + MLXSW_SP_SB_MM(0, 6), + MLXSW_SP_SB_MM(0, 6), + MLXSW_SP_SB_MM(0, 6), + MLXSW_SP_SB_MM(0, 6), + MLXSW_SP_SB_MM(0, 6), + MLXSW_SP_SB_MM(0, 6), + MLXSW_SP_SB_MM(0, 6), + MLXSW_SP_SB_MM(0, 6), + MLXSW_SP_SB_MM(0, 6), + MLXSW_SP_SB_MM(0, 6), }; static int mlxsw_sp_sb_mms_init(struct mlxsw_sp *mlxsw_sp) @@ -755,21 +843,22 @@ static void mlxsw_sp_pool_count(struct mlxsw_sp *mlxsw_sp, { int i; - for (i = 0; i < mlxsw_sp->sb_vals->pool_count; ++i) + for (i = 0; i < mlxsw_sp->sb_vals->pool_count; ++i) { if (mlxsw_sp->sb_vals->pool_dess[i].dir == - MLXSW_REG_SBXX_DIR_EGRESS) - goto out; - WARN(1, "No egress pools\n"); + MLXSW_REG_SBXX_DIR_INGRESS) + (*p_ingress_len)++; + else + (*p_egress_len)++; + } -out: - *p_ingress_len = i; - *p_egress_len = mlxsw_sp->sb_vals->pool_count - i; + WARN(*p_egress_len == 0, "No egress pools\n"); } const struct mlxsw_sp_sb_vals mlxsw_sp1_sb_vals = { .pool_count = ARRAY_SIZE(mlxsw_sp1_sb_pool_dess), .pool_dess = mlxsw_sp1_sb_pool_dess, .pms = mlxsw_sp1_sb_pms, + .pms_cpu = mlxsw_sp_cpu_port_sb_pms, .prs = mlxsw_sp1_sb_prs, .mms = mlxsw_sp_sb_mms, .cms_ingress = mlxsw_sp1_sb_cms_ingress, @@ -785,6 +874,7 @@ const struct mlxsw_sp_sb_vals mlxsw_sp2_sb_vals = { .pool_count = ARRAY_SIZE(mlxsw_sp2_sb_pool_dess), .pool_dess = mlxsw_sp2_sb_pool_dess, .pms = mlxsw_sp2_sb_pms, + .pms_cpu = mlxsw_sp_cpu_port_sb_pms, .prs = mlxsw_sp2_sb_prs, .mms = mlxsw_sp_sb_mms, .cms_ingress = mlxsw_sp2_sb_cms_ingress, @@ -799,8 +889,8 @@ const struct mlxsw_sp_sb_vals mlxsw_sp2_sb_vals = { int mlxsw_sp_buffers_init(struct mlxsw_sp *mlxsw_sp) { u32 max_headroom_size; - u16 ing_pool_count; - u16 eg_pool_count; + u16 ing_pool_count = 0; + u16 eg_pool_count = 0; int err; if (!MLXSW_CORE_RES_VALID(mlxsw_sp->core, CELL_SIZE)) @@ -834,6 +924,9 @@ int mlxsw_sp_buffers_init(struct mlxsw_sp *mlxsw_sp) err = mlxsw_sp_cpu_port_sb_cms_init(mlxsw_sp); if (err) goto err_sb_cpu_port_sb_cms_init; + err = mlxsw_sp_cpu_port_sb_pms_init(mlxsw_sp); + if (err) + goto err_sb_cpu_port_pms_init; err = mlxsw_sp_sb_mms_init(mlxsw_sp); if (err) goto err_sb_mms_init; @@ -851,6 +944,7 @@ int mlxsw_sp_buffers_init(struct mlxsw_sp *mlxsw_sp) err_devlink_sb_register: err_sb_mms_init: +err_sb_cpu_port_pms_init: err_sb_cpu_port_sb_cms_init: err_sb_prs_init: mlxsw_sp_sb_ports_fini(mlxsw_sp); @@ -900,16 +994,32 @@ int mlxsw_sp_sb_pool_get(struct mlxsw_core *mlxsw_core, int mlxsw_sp_sb_pool_set(struct mlxsw_core *mlxsw_core, unsigned int sb_index, u16 pool_index, u32 size, - enum devlink_sb_threshold_type threshold_type) + enum devlink_sb_threshold_type threshold_type, + struct netlink_ext_ack *extack) { struct mlxsw_sp *mlxsw_sp = mlxsw_core_driver_priv(mlxsw_core); u32 pool_size = mlxsw_sp_bytes_cells(mlxsw_sp, size); + const struct mlxsw_sp_sb_pr *pr; enum mlxsw_reg_sbpr_mode mode; - if (size > MLXSW_CORE_RES_GET(mlxsw_sp->core, MAX_BUFFER_SIZE)) + mode = (enum mlxsw_reg_sbpr_mode) threshold_type; + pr = &mlxsw_sp->sb_vals->prs[pool_index]; + + if (size > MLXSW_CORE_RES_GET(mlxsw_sp->core, MAX_BUFFER_SIZE)) { + NL_SET_ERR_MSG_MOD(extack, "Exceeded shared buffer size"); return -EINVAL; + } + + if (pr->freeze_mode && pr->mode != mode) { + NL_SET_ERR_MSG_MOD(extack, "Changing this pool's threshold type is forbidden"); + return -EINVAL; + }; + + if (pr->freeze_size && pr->size != size) { + NL_SET_ERR_MSG_MOD(extack, "Changing this pool's size is forbidden"); + return -EINVAL; + }; - mode = (enum mlxsw_reg_sbpr_mode) threshold_type; return mlxsw_sp_sb_pr_write(mlxsw_sp, pool_index, mode, pool_size, false); } @@ -927,7 +1037,8 @@ static u32 mlxsw_sp_sb_threshold_out(struct mlxsw_sp *mlxsw_sp, u16 pool_index, } static int mlxsw_sp_sb_threshold_in(struct mlxsw_sp *mlxsw_sp, u16 pool_index, - u32 threshold, u32 *p_max_buff) + u32 threshold, u32 *p_max_buff, + struct netlink_ext_ack *extack) { struct mlxsw_sp_sb_pr *pr = mlxsw_sp_sb_pr_get(mlxsw_sp, pool_index); @@ -936,8 +1047,10 @@ static int mlxsw_sp_sb_threshold_in(struct mlxsw_sp *mlxsw_sp, u16 pool_index, val = threshold + MLXSW_SP_SB_THRESHOLD_TO_ALPHA_OFFSET; if (val < MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN || - val > MLXSW_REG_SBXX_DYN_MAX_BUFF_MAX) + val > MLXSW_REG_SBXX_DYN_MAX_BUFF_MAX) { + NL_SET_ERR_MSG_MOD(extack, "Invalid dynamic threshold value"); return -EINVAL; + } *p_max_buff = val; } else { *p_max_buff = mlxsw_sp_bytes_cells(mlxsw_sp, threshold); @@ -963,7 +1076,7 @@ int mlxsw_sp_sb_port_pool_get(struct mlxsw_core_port *mlxsw_core_port, int mlxsw_sp_sb_port_pool_set(struct mlxsw_core_port *mlxsw_core_port, unsigned int sb_index, u16 pool_index, - u32 threshold) + u32 threshold, struct netlink_ext_ack *extack) { struct mlxsw_sp_port *mlxsw_sp_port = mlxsw_core_port_driver_priv(mlxsw_core_port); @@ -973,7 +1086,7 @@ int mlxsw_sp_sb_port_pool_set(struct mlxsw_core_port *mlxsw_core_port, int err; err = mlxsw_sp_sb_threshold_in(mlxsw_sp, pool_index, - threshold, &max_buff); + threshold, &max_buff, extack); if (err) return err; @@ -1004,22 +1117,41 @@ int mlxsw_sp_sb_tc_pool_bind_get(struct mlxsw_core_port *mlxsw_core_port, int mlxsw_sp_sb_tc_pool_bind_set(struct mlxsw_core_port *mlxsw_core_port, unsigned int sb_index, u16 tc_index, enum devlink_sb_pool_type pool_type, - u16 pool_index, u32 threshold) + u16 pool_index, u32 threshold, + struct netlink_ext_ack *extack) { struct mlxsw_sp_port *mlxsw_sp_port = mlxsw_core_port_driver_priv(mlxsw_core_port); struct mlxsw_sp *mlxsw_sp = mlxsw_sp_port->mlxsw_sp; u8 local_port = mlxsw_sp_port->local_port; + const struct mlxsw_sp_sb_cm *cm; u8 pg_buff = tc_index; enum mlxsw_reg_sbxx_dir dir = (enum mlxsw_reg_sbxx_dir) pool_type; u32 max_buff; int err; - if (dir != mlxsw_sp->sb_vals->pool_dess[pool_index].dir) + if (dir != mlxsw_sp->sb_vals->pool_dess[pool_index].dir) { + NL_SET_ERR_MSG_MOD(extack, "Binding egress TC to ingress pool and vice versa is forbidden"); return -EINVAL; + } + + if (dir == MLXSW_REG_SBXX_DIR_INGRESS) + cm = &mlxsw_sp->sb_vals->cms_ingress[tc_index]; + else + cm = &mlxsw_sp->sb_vals->cms_egress[tc_index]; + + if (cm->freeze_pool && cm->pool_index != pool_index) { + NL_SET_ERR_MSG_MOD(extack, "Binding this TC to a different pool is forbidden"); + return -EINVAL; + } + + if (cm->freeze_thresh && cm->max_buff != threshold) { + NL_SET_ERR_MSG_MOD(extack, "Changing this TC's threshold is forbidden"); + return -EINVAL; + } err = mlxsw_sp_sb_threshold_in(mlxsw_sp, pool_index, - threshold, &max_buff); + threshold, &max_buff, extack); if (err) return err; diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_dpipe.h b/drivers/net/ethernet/mellanox/mlxsw/spectrum_dpipe.h index e689576231abe0d6607c0d63be409140c4cf929b..246dbb3c0e1bb5a12ea4109ff70273cb5174e928 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_dpipe.h +++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_dpipe.h @@ -4,24 +4,9 @@ #ifndef _MLXSW_PIPELINE_H_ #define _MLXSW_PIPELINE_H_ -#if IS_ENABLED(CONFIG_NET_DEVLINK) - int mlxsw_sp_dpipe_init(struct mlxsw_sp *mlxsw_sp); void mlxsw_sp_dpipe_fini(struct mlxsw_sp *mlxsw_sp); -#else - -static inline int mlxsw_sp_dpipe_init(struct mlxsw_sp *mlxsw_sp) -{ - return 0; -} - -static inline void mlxsw_sp_dpipe_fini(struct mlxsw_sp *mlxsw_sp) -{ -} - -#endif - #define MLXSW_SP_DPIPE_TABLE_NAME_ERIF "mlxsw_erif" #define MLXSW_SP_DPIPE_TABLE_NAME_HOST4 "mlxsw_host4" #define MLXSW_SP_DPIPE_TABLE_NAME_HOST6 "mlxsw_host6" diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c index 902e766a8ed33eabbe0b11075284878ba3fb3ad4..1cda8a248b12e1a554ee713371eeedbdaa377a40 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c +++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c @@ -13,9 +13,9 @@ #include #include #include -#include #include #include +#include #include #include #include @@ -2371,7 +2371,7 @@ static enum mlxsw_reg_rauht_op mlxsw_sp_rauht_op(bool adding) MLXSW_REG_RAUHT_OP_WRITE_DELETE; } -static void +static int mlxsw_sp_router_neigh_entry_op4(struct mlxsw_sp *mlxsw_sp, struct mlxsw_sp_neigh_entry *neigh_entry, enum mlxsw_reg_rauht_op op) @@ -2385,10 +2385,10 @@ mlxsw_sp_router_neigh_entry_op4(struct mlxsw_sp *mlxsw_sp, if (neigh_entry->counter_valid) mlxsw_reg_rauht_pack_counter(rauht_pl, neigh_entry->counter_index); - mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(rauht), rauht_pl); + return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(rauht), rauht_pl); } -static void +static int mlxsw_sp_router_neigh_entry_op6(struct mlxsw_sp *mlxsw_sp, struct mlxsw_sp_neigh_entry *neigh_entry, enum mlxsw_reg_rauht_op op) @@ -2402,7 +2402,7 @@ mlxsw_sp_router_neigh_entry_op6(struct mlxsw_sp *mlxsw_sp, if (neigh_entry->counter_valid) mlxsw_reg_rauht_pack_counter(rauht_pl, neigh_entry->counter_index); - mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(rauht), rauht_pl); + return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(rauht), rauht_pl); } bool mlxsw_sp_neigh_ipv6_ignore(struct mlxsw_sp_neigh_entry *neigh_entry) @@ -2424,20 +2424,33 @@ mlxsw_sp_neigh_entry_update(struct mlxsw_sp *mlxsw_sp, struct mlxsw_sp_neigh_entry *neigh_entry, bool adding) { + enum mlxsw_reg_rauht_op op = mlxsw_sp_rauht_op(adding); + int err; + if (!adding && !neigh_entry->connected) return; neigh_entry->connected = adding; if (neigh_entry->key.n->tbl->family == AF_INET) { - mlxsw_sp_router_neigh_entry_op4(mlxsw_sp, neigh_entry, - mlxsw_sp_rauht_op(adding)); + err = mlxsw_sp_router_neigh_entry_op4(mlxsw_sp, neigh_entry, + op); + if (err) + return; } else if (neigh_entry->key.n->tbl->family == AF_INET6) { if (mlxsw_sp_neigh_ipv6_ignore(neigh_entry)) return; - mlxsw_sp_router_neigh_entry_op6(mlxsw_sp, neigh_entry, - mlxsw_sp_rauht_op(adding)); + err = mlxsw_sp_router_neigh_entry_op6(mlxsw_sp, neigh_entry, + op); + if (err) + return; } else { WARN_ON_ONCE(1); + return; } + + if (adding) + neigh_entry->key.n->flags |= NTF_OFFLOADED; + else + neigh_entry->key.n->flags &= ~NTF_OFFLOADED; } void @@ -2873,12 +2886,13 @@ mlxsw_sp_nexthop6_group_cmp(const struct mlxsw_sp_nexthop_group *nh_grp, return false; list_for_each_entry(mlxsw_sp_rt6, &fib6_entry->rt6_list, list) { + struct fib6_nh *fib6_nh = &mlxsw_sp_rt6->rt->fib6_nh; struct in6_addr *gw; int ifindex, weight; - ifindex = mlxsw_sp_rt6->rt->fib6_nh.nh_dev->ifindex; - weight = mlxsw_sp_rt6->rt->fib6_nh.nh_weight; - gw = &mlxsw_sp_rt6->rt->fib6_nh.nh_gw; + ifindex = fib6_nh->fib_nh_dev->ifindex; + weight = fib6_nh->fib_nh_weight; + gw = &fib6_nh->fib_nh_gw6; if (!mlxsw_sp_nexthop6_group_has_nexthop(nh_grp, gw, ifindex, weight)) return false; @@ -2944,7 +2958,7 @@ mlxsw_sp_nexthop6_group_hash(struct mlxsw_sp_fib6_entry *fib6_entry, u32 seed) struct net_device *dev; list_for_each_entry(mlxsw_sp_rt6, &fib6_entry->rt6_list, list) { - dev = mlxsw_sp_rt6->rt->fib6_nh.nh_dev; + dev = mlxsw_sp_rt6->rt->fib6_nh.fib_nh_dev; val ^= dev->ifindex; } @@ -3610,7 +3624,7 @@ static bool mlxsw_sp_nexthop4_ipip_type(const struct mlxsw_sp *mlxsw_sp, const struct fib_nh *fib_nh, enum mlxsw_sp_ipip_type *p_ipipt) { - struct net_device *dev = fib_nh->nh_dev; + struct net_device *dev = fib_nh->fib_nh_dev; return dev && fib_nh->nh_parent->fib_type == RTN_UNICAST && @@ -3637,7 +3651,7 @@ static int mlxsw_sp_nexthop4_type_init(struct mlxsw_sp *mlxsw_sp, struct fib_nh *fib_nh) { const struct mlxsw_sp_ipip_ops *ipip_ops; - struct net_device *dev = fib_nh->nh_dev; + struct net_device *dev = fib_nh->fib_nh_dev; struct mlxsw_sp_ipip_entry *ipip_entry; struct mlxsw_sp_rif *rif; int err; @@ -3681,18 +3695,18 @@ static int mlxsw_sp_nexthop4_init(struct mlxsw_sp *mlxsw_sp, struct mlxsw_sp_nexthop *nh, struct fib_nh *fib_nh) { - struct net_device *dev = fib_nh->nh_dev; + struct net_device *dev = fib_nh->fib_nh_dev; struct in_device *in_dev; int err; nh->nh_grp = nh_grp; nh->key.fib_nh = fib_nh; #ifdef CONFIG_IP_ROUTE_MULTIPATH - nh->nh_weight = fib_nh->nh_weight; + nh->nh_weight = fib_nh->fib_nh_weight; #else nh->nh_weight = 1; #endif - memcpy(&nh->gw_addr, &fib_nh->nh_gw, sizeof(fib_nh->nh_gw)); + memcpy(&nh->gw_addr, &fib_nh->fib_nh_gw4, sizeof(fib_nh->fib_nh_gw4)); err = mlxsw_sp_nexthop_insert(mlxsw_sp, nh); if (err) return err; @@ -3705,7 +3719,7 @@ static int mlxsw_sp_nexthop4_init(struct mlxsw_sp *mlxsw_sp, in_dev = __in_dev_get_rtnl(dev); if (in_dev && IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN(in_dev) && - fib_nh->nh_flags & RTNH_F_LINKDOWN) + fib_nh->fib_nh_flags & RTNH_F_LINKDOWN) return 0; err = mlxsw_sp_nexthop4_type_init(mlxsw_sp, nh, fib_nh); @@ -3804,7 +3818,7 @@ static void mlxsw_sp_nexthop_rif_gone_sync(struct mlxsw_sp *mlxsw_sp, static bool mlxsw_sp_fi_is_gateway(const struct mlxsw_sp *mlxsw_sp, const struct fib_info *fi) { - return fi->fib_nh->nh_scope == RT_SCOPE_LINK || + return fi->fib_nh->fib_nh_scope == RT_SCOPE_LINK || mlxsw_sp_nexthop4_ipip_type(mlxsw_sp, fi->fib_nh, NULL); } @@ -3946,9 +3960,9 @@ mlxsw_sp_rt6_nexthop(struct mlxsw_sp_nexthop_group *nh_grp, struct mlxsw_sp_nexthop *nh = &nh_grp->nexthops[i]; struct fib6_info *rt = mlxsw_sp_rt6->rt; - if (nh->rif && nh->rif->dev == rt->fib6_nh.nh_dev && + if (nh->rif && nh->rif->dev == rt->fib6_nh.fib_nh_dev && ipv6_addr_equal((const struct in6_addr *) &nh->gw_addr, - &rt->fib6_nh.nh_gw)) + &rt->fib6_nh.fib_nh_gw6)) return nh; continue; } @@ -3966,7 +3980,7 @@ mlxsw_sp_fib4_entry_offload_set(struct mlxsw_sp_fib_entry *fib_entry) fib_entry->type == MLXSW_SP_FIB_ENTRY_TYPE_BLACKHOLE || fib_entry->type == MLXSW_SP_FIB_ENTRY_TYPE_IPIP_DECAP || fib_entry->type == MLXSW_SP_FIB_ENTRY_TYPE_NVE_DECAP) { - nh_grp->nexthops->key.fib_nh->nh_flags |= RTNH_F_OFFLOAD; + nh_grp->nexthops->key.fib_nh->fib_nh_flags |= RTNH_F_OFFLOAD; return; } @@ -3974,9 +3988,9 @@ mlxsw_sp_fib4_entry_offload_set(struct mlxsw_sp_fib_entry *fib_entry) struct mlxsw_sp_nexthop *nh = &nh_grp->nexthops[i]; if (nh->offloaded) - nh->key.fib_nh->nh_flags |= RTNH_F_OFFLOAD; + nh->key.fib_nh->fib_nh_flags |= RTNH_F_OFFLOAD; else - nh->key.fib_nh->nh_flags &= ~RTNH_F_OFFLOAD; + nh->key.fib_nh->fib_nh_flags &= ~RTNH_F_OFFLOAD; } } @@ -3992,7 +4006,7 @@ mlxsw_sp_fib4_entry_offload_unset(struct mlxsw_sp_fib_entry *fib_entry) for (i = 0; i < nh_grp->count; i++) { struct mlxsw_sp_nexthop *nh = &nh_grp->nexthops[i]; - nh->key.fib_nh->nh_flags &= ~RTNH_F_OFFLOAD; + nh->key.fib_nh->fib_nh_flags &= ~RTNH_F_OFFLOAD; } } @@ -4008,19 +4022,20 @@ mlxsw_sp_fib6_entry_offload_set(struct mlxsw_sp_fib_entry *fib_entry) if (fib_entry->type == MLXSW_SP_FIB_ENTRY_TYPE_LOCAL || fib_entry->type == MLXSW_SP_FIB_ENTRY_TYPE_BLACKHOLE) { list_first_entry(&fib6_entry->rt6_list, struct mlxsw_sp_rt6, - list)->rt->fib6_nh.nh_flags |= RTNH_F_OFFLOAD; + list)->rt->fib6_nh.fib_nh_flags |= RTNH_F_OFFLOAD; return; } list_for_each_entry(mlxsw_sp_rt6, &fib6_entry->rt6_list, list) { struct mlxsw_sp_nexthop_group *nh_grp = fib_entry->nh_group; + struct fib6_nh *fib6_nh = &mlxsw_sp_rt6->rt->fib6_nh; struct mlxsw_sp_nexthop *nh; nh = mlxsw_sp_rt6_nexthop(nh_grp, mlxsw_sp_rt6); if (nh && nh->offloaded) - mlxsw_sp_rt6->rt->fib6_nh.nh_flags |= RTNH_F_OFFLOAD; + fib6_nh->fib_nh_flags |= RTNH_F_OFFLOAD; else - mlxsw_sp_rt6->rt->fib6_nh.nh_flags &= ~RTNH_F_OFFLOAD; + fib6_nh->fib_nh_flags &= ~RTNH_F_OFFLOAD; } } @@ -4035,7 +4050,7 @@ mlxsw_sp_fib6_entry_offload_unset(struct mlxsw_sp_fib_entry *fib_entry) list_for_each_entry(mlxsw_sp_rt6, &fib6_entry->rt6_list, list) { struct fib6_info *rt = mlxsw_sp_rt6->rt; - rt->fib6_nh.nh_flags &= ~RTNH_F_OFFLOAD; + rt->fib6_nh.fib_nh_flags &= ~RTNH_F_OFFLOAD; } } @@ -4913,7 +4928,7 @@ static void mlxsw_sp_rt6_destroy(struct mlxsw_sp_rt6 *mlxsw_sp_rt6) static bool mlxsw_sp_fib6_rt_can_mp(const struct fib6_info *rt) { /* RTF_CACHE routes are ignored */ - return (rt->fib6_flags & (RTF_GATEWAY | RTF_ADDRCONF)) == RTF_GATEWAY; + return !(rt->fib6_flags & RTF_ADDRCONF) && rt->fib6_nh.fib_nh_gw_family; } static struct fib6_info * @@ -4972,8 +4987,8 @@ static bool mlxsw_sp_nexthop6_ipip_type(const struct mlxsw_sp *mlxsw_sp, const struct fib6_info *rt, enum mlxsw_sp_ipip_type *ret) { - return rt->fib6_nh.nh_dev && - mlxsw_sp_netdev_ipip_type(mlxsw_sp, rt->fib6_nh.nh_dev, ret); + return rt->fib6_nh.fib_nh_dev && + mlxsw_sp_netdev_ipip_type(mlxsw_sp, rt->fib6_nh.fib_nh_dev, ret); } static int mlxsw_sp_nexthop6_type_init(struct mlxsw_sp *mlxsw_sp, @@ -4983,7 +4998,7 @@ static int mlxsw_sp_nexthop6_type_init(struct mlxsw_sp *mlxsw_sp, { const struct mlxsw_sp_ipip_ops *ipip_ops; struct mlxsw_sp_ipip_entry *ipip_entry; - struct net_device *dev = rt->fib6_nh.nh_dev; + struct net_device *dev = rt->fib6_nh.fib_nh_dev; struct mlxsw_sp_rif *rif; int err; @@ -5026,11 +5041,11 @@ static int mlxsw_sp_nexthop6_init(struct mlxsw_sp *mlxsw_sp, struct mlxsw_sp_nexthop *nh, const struct fib6_info *rt) { - struct net_device *dev = rt->fib6_nh.nh_dev; + struct net_device *dev = rt->fib6_nh.fib_nh_dev; nh->nh_grp = nh_grp; - nh->nh_weight = rt->fib6_nh.nh_weight; - memcpy(&nh->gw_addr, &rt->fib6_nh.nh_gw, sizeof(nh->gw_addr)); + nh->nh_weight = rt->fib6_nh.fib_nh_weight; + memcpy(&nh->gw_addr, &rt->fib6_nh.fib_nh_gw6, sizeof(nh->gw_addr)); mlxsw_sp_nexthop_counter_alloc(mlxsw_sp, nh); list_add_tail(&nh->router_list_node, &mlxsw_sp->router->nexthop_list); @@ -5053,7 +5068,7 @@ static void mlxsw_sp_nexthop6_fini(struct mlxsw_sp *mlxsw_sp, static bool mlxsw_sp_rt6_is_gateway(const struct mlxsw_sp *mlxsw_sp, const struct fib6_info *rt) { - return rt->fib6_flags & RTF_GATEWAY || + return rt->fib6_nh.fib_nh_gw_family || mlxsw_sp_nexthop6_ipip_type(mlxsw_sp, rt, NULL); } @@ -6035,6 +6050,10 @@ static int mlxsw_sp_router_fib_rule_event(unsigned long event, fr_info = container_of(info, struct fib_rule_notifier_info, info); rule = fr_info->rule; + /* Rule only affects locally generated traffic */ + if (rule->iifindex == info->net->loopback_dev->ifindex) + return 0; + switch (info->family) { case AF_INET: if (!fib4_rule_default(rule) && !rule->l3mdev) @@ -6086,10 +6105,20 @@ static int mlxsw_sp_router_fib_event(struct notifier_block *nb, return notifier_from_errno(err); break; case FIB_EVENT_ENTRY_ADD: + case FIB_EVENT_ENTRY_REPLACE: /* fall through */ + case FIB_EVENT_ENTRY_APPEND: /* fall through */ if (router->aborted) { NL_SET_ERR_MSG_MOD(info->extack, "FIB offload was aborted. Not configuring route"); return notifier_from_errno(-EINVAL); } + if (info->family == AF_INET) { + struct fib_entry_notifier_info *fen_info = ptr; + + if (fen_info->fi->fib_nh_is_v6) { + NL_SET_ERR_MSG_MOD(info->extack, "IPv6 gateway with IPv4 route is not supported"); + return notifier_from_errno(-EINVAL); + } + } break; } @@ -7808,7 +7837,7 @@ static int mlxsw_sp_mp_hash_init(struct mlxsw_sp *mlxsw_sp) char recr2_pl[MLXSW_REG_RECR2_LEN]; u32 seed; - get_random_bytes(&seed, sizeof(seed)); + seed = jhash(mlxsw_sp->base_mac, sizeof(mlxsw_sp->base_mac), 0); mlxsw_reg_recr2_pack(recr2_pl, seed); mlxsw_sp_mp4_hash_init(recr2_pl); mlxsw_sp_mp6_hash_init(recr2_pl); diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_span.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_span.c index 536c23c578c348d103858c8fe48fec53d22d49a2..560a60e522f9e856682363bfd87ab93e55832199 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_span.c +++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_span.c @@ -316,7 +316,11 @@ mlxsw_sp_span_gretap4_route(const struct net_device *to_dev, dev = rt->dst.dev; *saddrp = fl4.saddr; - *daddrp = rt->rt_gateway; + if (rt->rt_gw_family == AF_INET) + *daddrp = rt->rt_gw4; + /* can not offload if route has an IPv6 gateway */ + else if (rt->rt_gw_family == AF_INET6) + dev = NULL; out: ip_rt_put(rt); diff --git a/drivers/net/ethernet/mellanox/mlxsw/switchib.c b/drivers/net/ethernet/mellanox/mlxsw/switchib.c index bcf2e79a21c853b631c38c07f4f12526459fdebe..0d9356b3f65d8b89f3f2eb15f82cbacf3faea14c 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/switchib.c +++ b/drivers/net/ethernet/mellanox/mlxsw/switchib.c @@ -30,6 +30,7 @@ struct mlxsw_sib { struct mlxsw_sib_port **ports; struct mlxsw_core *core; const struct mlxsw_bus_info *bus_info; + u8 hw_id[ETH_ALEN]; }; struct mlxsw_sib_port { @@ -102,6 +103,18 @@ mlxsw_sib_tx_v1_hdr_construct(struct sk_buff *skb, mlxsw_tx_v1_hdr_type_set(txhdr, MLXSW_TXHDR_TYPE_CONTROL); } +static int mlxsw_sib_hw_id_get(struct mlxsw_sib *mlxsw_sib) +{ + char spad_pl[MLXSW_REG_SPAD_LEN] = {0}; + int err; + + err = mlxsw_reg_query(mlxsw_sib->core, MLXSW_REG(spad), spad_pl); + if (err) + return err; + mlxsw_reg_spad_base_mac_memcpy_from(spad_pl, mlxsw_sib->hw_id); + return 0; +} + static int mlxsw_sib_port_admin_status_set(struct mlxsw_sib_port *mlxsw_sib_port, bool is_up) @@ -267,7 +280,9 @@ static int mlxsw_sib_port_create(struct mlxsw_sib *mlxsw_sib, u8 local_port, { int err; - err = mlxsw_core_port_init(mlxsw_sib->core, local_port); + err = mlxsw_core_port_init(mlxsw_sib->core, local_port, + module + 1, false, 0, + mlxsw_sib->hw_id, sizeof(mlxsw_sib->hw_id)); if (err) { dev_err(mlxsw_sib->bus_info->dev, "Port %d: Failed to init core port\n", local_port); @@ -439,6 +454,12 @@ static int mlxsw_sib_init(struct mlxsw_core *mlxsw_core, mlxsw_sib->core = mlxsw_core; mlxsw_sib->bus_info = mlxsw_bus_info; + err = mlxsw_sib_hw_id_get(mlxsw_sib); + if (err) { + dev_err(mlxsw_sib->bus_info->dev, "Failed to get switch HW ID\n"); + return err; + } + err = mlxsw_sib_ports_create(mlxsw_sib); if (err) { dev_err(mlxsw_sib->bus_info->dev, "Failed to create ports\n"); diff --git a/drivers/net/ethernet/mellanox/mlxsw/switchx2.c b/drivers/net/ethernet/mellanox/mlxsw/switchx2.c index 533fe6235b7c8d65a1fafc684b570479e4023b6a..fc4f19167262fe5ecce840bc6cb1a97c36da4295 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/switchx2.c +++ b/drivers/net/ethernet/mellanox/mlxsw/switchx2.c @@ -379,26 +379,14 @@ mlxsw_sx_port_get_stats64(struct net_device *dev, stats->tx_dropped = tx_dropped; } -static int mlxsw_sx_port_get_phys_port_name(struct net_device *dev, char *name, - size_t len) -{ - struct mlxsw_sx_port *mlxsw_sx_port = netdev_priv(dev); - - return mlxsw_core_port_get_phys_port_name(mlxsw_sx_port->mlxsw_sx->core, - mlxsw_sx_port->local_port, - name, len); -} - -static int mlxsw_sx_port_get_port_parent_id(struct net_device *dev, - struct netdev_phys_item_id *ppid) +static struct devlink_port * +mlxsw_sx_port_get_devlink_port(struct net_device *dev) { struct mlxsw_sx_port *mlxsw_sx_port = netdev_priv(dev); struct mlxsw_sx *mlxsw_sx = mlxsw_sx_port->mlxsw_sx; - ppid->id_len = sizeof(mlxsw_sx->hw_id); - memcpy(&ppid->id, &mlxsw_sx->hw_id, ppid->id_len); - - return 0; + return mlxsw_core_port_devlink_port_get(mlxsw_sx->core, + mlxsw_sx_port->local_port); } static const struct net_device_ops mlxsw_sx_port_netdev_ops = { @@ -407,8 +395,7 @@ static const struct net_device_ops mlxsw_sx_port_netdev_ops = { .ndo_start_xmit = mlxsw_sx_port_xmit, .ndo_change_mtu = mlxsw_sx_port_change_mtu, .ndo_get_stats64 = mlxsw_sx_port_get_stats64, - .ndo_get_phys_port_name = mlxsw_sx_port_get_phys_port_name, - .ndo_get_port_parent_id = mlxsw_sx_port_get_port_parent_id, + .ndo_get_devlink_port = mlxsw_sx_port_get_devlink_port, }; static void mlxsw_sx_port_get_drvinfo(struct net_device *dev, @@ -1102,7 +1089,7 @@ static int __mlxsw_sx_port_eth_create(struct mlxsw_sx *mlxsw_sx, u8 local_port, } mlxsw_core_port_eth_set(mlxsw_sx->core, mlxsw_sx_port->local_port, - mlxsw_sx_port, dev, module + 1, false, 0); + mlxsw_sx_port, dev); mlxsw_sx->ports[local_port] = mlxsw_sx_port; return 0; @@ -1127,7 +1114,9 @@ static int mlxsw_sx_port_eth_create(struct mlxsw_sx *mlxsw_sx, u8 local_port, { int err; - err = mlxsw_core_port_init(mlxsw_sx->core, local_port); + err = mlxsw_core_port_init(mlxsw_sx->core, local_port, + module + 1, false, 0, + mlxsw_sx->hw_id, sizeof(mlxsw_sx->hw_id)); if (err) { dev_err(mlxsw_sx->bus_info->dev, "Port %d: Failed to init core port\n", local_port); diff --git a/drivers/net/ethernet/micrel/ks8851.c b/drivers/net/ethernet/micrel/ks8851.c index 7849119d407aef1a7b92d6b0e047a8f74b4867f7..b44172a901edbeb5e899c8c9642cd1c84fbca1e3 100644 --- a/drivers/net/ethernet/micrel/ks8851.c +++ b/drivers/net/ethernet/micrel/ks8851.c @@ -425,7 +425,7 @@ static void ks8851_init_mac(struct ks8851_net *ks) const u8 *mac_addr; mac_addr = of_get_mac_address(ks->spidev->dev.of_node); - if (mac_addr) { + if (!IS_ERR(mac_addr)) { memcpy(dev->dev_addr, mac_addr, ETH_ALEN); ks8851_write_mac_addr(dev); return; diff --git a/drivers/net/ethernet/micrel/ks8851_mll.c b/drivers/net/ethernet/micrel/ks8851_mll.c index c946841c0a066d2e7eabd059092ed6cbbb156b01..dc76b0d152348bfb6b95abad8ef2f36e11d64c35 100644 --- a/drivers/net/ethernet/micrel/ks8851_mll.c +++ b/drivers/net/ethernet/micrel/ks8851_mll.c @@ -1327,7 +1327,7 @@ static int ks8851_probe(struct platform_device *pdev) /* overwriting the default MAC address */ if (pdev->dev.of_node) { mac = of_get_mac_address(pdev->dev.of_node); - if (mac) + if (!IS_ERR(mac)) memcpy(ks->mac_addr, mac, ETH_ALEN); } else { struct ks8851_mll_platform_data *pdata; diff --git a/drivers/net/ethernet/microchip/enc28j60.c b/drivers/net/ethernet/microchip/enc28j60.c index 8f72587b5a2cbb9c79b8d0377d779650eccb7446..0567e4f387a52f9591e360cf9a41d773e6e0b895 100644 --- a/drivers/net/ethernet/microchip/enc28j60.c +++ b/drivers/net/ethernet/microchip/enc28j60.c @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0+ /* * Microchip ENC28J60 ethernet driver (MAC + PHY) * @@ -5,11 +6,6 @@ * Author: Claudio Lanconelli * based on enc28j60.c written by David Anders for 2.4 kernel version * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * * $Id: enc28j60.c,v 1.22 2007/12/20 10:47:01 claudio Exp $ */ @@ -18,9 +14,9 @@ #include #include #include +#include #include #include -#include #include #include #include @@ -28,7 +24,6 @@ #include #include #include -#include #include "enc28j60_hw.h" @@ -41,10 +36,11 @@ (NETIF_MSG_PROBE | NETIF_MSG_IFUP | NETIF_MSG_IFDOWN | NETIF_MSG_LINK) /* Buffer size required for the largest SPI transfer (i.e., reading a - * frame). */ + * frame). + */ #define SPI_TRANSFER_BUF_LEN (4 + MAX_FRAMELEN) -#define TX_TIMEOUT (4 * HZ) +#define TX_TIMEOUT (4 * HZ) /* Max TX retries in case of collision as suggested by errata datasheet */ #define MAX_TX_RETRYCOUNT 16 @@ -83,11 +79,12 @@ static struct { /* * SPI read buffer - * wait for the SPI transfer and copy received data to destination + * Wait for the SPI transfer and copy received data to destination. */ static int spi_read_buf(struct enc28j60_net *priv, int len, u8 *data) { + struct device *dev = &priv->spi->dev; u8 *rx_buf = priv->spi_transfer_buf + 4; u8 *tx_buf = priv->spi_transfer_buf; struct spi_transfer tx = { @@ -113,8 +110,8 @@ spi_read_buf(struct enc28j60_net *priv, int len, u8 *data) ret = msg.status; } if (ret && netif_msg_drv(priv)) - printk(KERN_DEBUG DRV_NAME ": %s() failed: ret = %d\n", - __func__, ret); + dev_printk(KERN_DEBUG, dev, "%s() failed: ret = %d\n", + __func__, ret); return ret; } @@ -122,9 +119,9 @@ spi_read_buf(struct enc28j60_net *priv, int len, u8 *data) /* * SPI write buffer */ -static int spi_write_buf(struct enc28j60_net *priv, int len, - const u8 *data) +static int spi_write_buf(struct enc28j60_net *priv, int len, const u8 *data) { + struct device *dev = &priv->spi->dev; int ret; if (len > SPI_TRANSFER_BUF_LEN - 1 || len <= 0) @@ -134,8 +131,8 @@ static int spi_write_buf(struct enc28j60_net *priv, int len, memcpy(&priv->spi_transfer_buf[1], data, len); ret = spi_write(priv->spi, priv->spi_transfer_buf, len + 1); if (ret && netif_msg_drv(priv)) - printk(KERN_DEBUG DRV_NAME ": %s() failed: ret = %d\n", - __func__, ret); + dev_printk(KERN_DEBUG, dev, "%s() failed: ret = %d\n", + __func__, ret); } return ret; } @@ -143,9 +140,9 @@ static int spi_write_buf(struct enc28j60_net *priv, int len, /* * basic SPI read operation */ -static u8 spi_read_op(struct enc28j60_net *priv, u8 op, - u8 addr) +static u8 spi_read_op(struct enc28j60_net *priv, u8 op, u8 addr) { + struct device *dev = &priv->spi->dev; u8 tx_buf[2]; u8 rx_buf[4]; u8 val = 0; @@ -159,8 +156,8 @@ static u8 spi_read_op(struct enc28j60_net *priv, u8 op, tx_buf[0] = op | (addr & ADDR_MASK); ret = spi_write_then_read(priv->spi, tx_buf, 1, rx_buf, slen); if (ret) - printk(KERN_DEBUG DRV_NAME ": %s() failed: ret = %d\n", - __func__, ret); + dev_printk(KERN_DEBUG, dev, "%s() failed: ret = %d\n", + __func__, ret); else val = rx_buf[slen - 1]; @@ -170,28 +167,25 @@ static u8 spi_read_op(struct enc28j60_net *priv, u8 op, /* * basic SPI write operation */ -static int spi_write_op(struct enc28j60_net *priv, u8 op, - u8 addr, u8 val) +static int spi_write_op(struct enc28j60_net *priv, u8 op, u8 addr, u8 val) { + struct device *dev = &priv->spi->dev; int ret; priv->spi_transfer_buf[0] = op | (addr & ADDR_MASK); priv->spi_transfer_buf[1] = val; ret = spi_write(priv->spi, priv->spi_transfer_buf, 2); if (ret && netif_msg_drv(priv)) - printk(KERN_DEBUG DRV_NAME ": %s() failed: ret = %d\n", - __func__, ret); + dev_printk(KERN_DEBUG, dev, "%s() failed: ret = %d\n", + __func__, ret); return ret; } static void enc28j60_soft_reset(struct enc28j60_net *priv) { - if (netif_msg_hw(priv)) - printk(KERN_DEBUG DRV_NAME ": %s() enter\n", __func__); - spi_write_op(priv, ENC28J60_SOFT_RESET, 0, ENC28J60_SOFT_RESET); /* Errata workaround #1, CLKRDY check is unreliable, - * delay at least 1 mS instead */ + * delay at least 1 ms instead */ udelay(2000); } @@ -203,7 +197,7 @@ static void enc28j60_set_bank(struct enc28j60_net *priv, u8 addr) u8 b = (addr & BANK_MASK) >> 5; /* These registers (EIE, EIR, ESTAT, ECON2, ECON1) - * are present in all banks, no need to switch bank + * are present in all banks, no need to switch bank. */ if (addr >= EIE && addr <= ECON1) return; @@ -242,15 +236,13 @@ static void enc28j60_set_bank(struct enc28j60_net *priv, u8 addr) /* * Register bit field Set */ -static void nolock_reg_bfset(struct enc28j60_net *priv, - u8 addr, u8 mask) +static void nolock_reg_bfset(struct enc28j60_net *priv, u8 addr, u8 mask) { enc28j60_set_bank(priv, addr); spi_write_op(priv, ENC28J60_BIT_FIELD_SET, addr, mask); } -static void locked_reg_bfset(struct enc28j60_net *priv, - u8 addr, u8 mask) +static void locked_reg_bfset(struct enc28j60_net *priv, u8 addr, u8 mask) { mutex_lock(&priv->lock); nolock_reg_bfset(priv, addr, mask); @@ -260,15 +252,13 @@ static void locked_reg_bfset(struct enc28j60_net *priv, /* * Register bit field Clear */ -static void nolock_reg_bfclr(struct enc28j60_net *priv, - u8 addr, u8 mask) +static void nolock_reg_bfclr(struct enc28j60_net *priv, u8 addr, u8 mask) { enc28j60_set_bank(priv, addr); spi_write_op(priv, ENC28J60_BIT_FIELD_CLR, addr, mask); } -static void locked_reg_bfclr(struct enc28j60_net *priv, - u8 addr, u8 mask) +static void locked_reg_bfclr(struct enc28j60_net *priv, u8 addr, u8 mask) { mutex_lock(&priv->lock); nolock_reg_bfclr(priv, addr, mask); @@ -278,15 +268,13 @@ static void locked_reg_bfclr(struct enc28j60_net *priv, /* * Register byte read */ -static int nolock_regb_read(struct enc28j60_net *priv, - u8 address) +static int nolock_regb_read(struct enc28j60_net *priv, u8 address) { enc28j60_set_bank(priv, address); return spi_read_op(priv, ENC28J60_READ_CTRL_REG, address); } -static int locked_regb_read(struct enc28j60_net *priv, - u8 address) +static int locked_regb_read(struct enc28j60_net *priv, u8 address) { int ret; @@ -300,8 +288,7 @@ static int locked_regb_read(struct enc28j60_net *priv, /* * Register word read */ -static int nolock_regw_read(struct enc28j60_net *priv, - u8 address) +static int nolock_regw_read(struct enc28j60_net *priv, u8 address) { int rl, rh; @@ -312,8 +299,7 @@ static int nolock_regw_read(struct enc28j60_net *priv, return (rh << 8) | rl; } -static int locked_regw_read(struct enc28j60_net *priv, - u8 address) +static int locked_regw_read(struct enc28j60_net *priv, u8 address) { int ret; @@ -327,15 +313,13 @@ static int locked_regw_read(struct enc28j60_net *priv, /* * Register byte write */ -static void nolock_regb_write(struct enc28j60_net *priv, - u8 address, u8 data) +static void nolock_regb_write(struct enc28j60_net *priv, u8 address, u8 data) { enc28j60_set_bank(priv, address); spi_write_op(priv, ENC28J60_WRITE_CTRL_REG, address, data); } -static void locked_regb_write(struct enc28j60_net *priv, - u8 address, u8 data) +static void locked_regb_write(struct enc28j60_net *priv, u8 address, u8 data) { mutex_lock(&priv->lock); nolock_regb_write(priv, address, data); @@ -345,8 +329,7 @@ static void locked_regb_write(struct enc28j60_net *priv, /* * Register word write */ -static void nolock_regw_write(struct enc28j60_net *priv, - u8 address, u16 data) +static void nolock_regw_write(struct enc28j60_net *priv, u8 address, u16 data) { enc28j60_set_bank(priv, address); spi_write_op(priv, ENC28J60_WRITE_CTRL_REG, address, (u8) data); @@ -354,8 +337,7 @@ static void nolock_regw_write(struct enc28j60_net *priv, (u8) (data >> 8)); } -static void locked_regw_write(struct enc28j60_net *priv, - u8 address, u16 data) +static void locked_regw_write(struct enc28j60_net *priv, u8 address, u16 data) { mutex_lock(&priv->lock); nolock_regw_write(priv, address, data); @@ -364,20 +346,23 @@ static void locked_regw_write(struct enc28j60_net *priv, /* * Buffer memory read - * Select the starting address and execute a SPI buffer read + * Select the starting address and execute a SPI buffer read. */ -static void enc28j60_mem_read(struct enc28j60_net *priv, - u16 addr, int len, u8 *data) +static void enc28j60_mem_read(struct enc28j60_net *priv, u16 addr, int len, + u8 *data) { mutex_lock(&priv->lock); nolock_regw_write(priv, ERDPTL, addr); #ifdef CONFIG_ENC28J60_WRITEVERIFY if (netif_msg_drv(priv)) { + struct device *dev = &priv->spi->dev; u16 reg; + reg = nolock_regw_read(priv, ERDPTL); if (reg != addr) - printk(KERN_DEBUG DRV_NAME ": %s() error writing ERDPT " - "(0x%04x - 0x%04x)\n", __func__, reg, addr); + dev_printk(KERN_DEBUG, dev, + "%s() error writing ERDPT (0x%04x - 0x%04x)\n", + __func__, reg, addr); } #endif spi_read_buf(priv, len, data); @@ -390,6 +375,8 @@ static void enc28j60_mem_read(struct enc28j60_net *priv, static void enc28j60_packet_write(struct enc28j60_net *priv, int len, const u8 *data) { + struct device *dev = &priv->spi->dev; + mutex_lock(&priv->lock); /* Set the write pointer to start of transmit buffer area */ nolock_regw_write(priv, EWRPTL, TXSTART_INIT); @@ -398,9 +385,9 @@ enc28j60_packet_write(struct enc28j60_net *priv, int len, const u8 *data) u16 reg; reg = nolock_regw_read(priv, EWRPTL); if (reg != TXSTART_INIT) - printk(KERN_DEBUG DRV_NAME - ": %s() ERWPT:0x%04x != 0x%04x\n", - __func__, reg, TXSTART_INIT); + dev_printk(KERN_DEBUG, dev, + "%s() ERWPT:0x%04x != 0x%04x\n", + __func__, reg, TXSTART_INIT); } #endif /* Set the TXND pointer to correspond to the packet size given */ @@ -408,30 +395,28 @@ enc28j60_packet_write(struct enc28j60_net *priv, int len, const u8 *data) /* write per-packet control byte */ spi_write_op(priv, ENC28J60_WRITE_BUF_MEM, 0, 0x00); if (netif_msg_hw(priv)) - printk(KERN_DEBUG DRV_NAME - ": %s() after control byte ERWPT:0x%04x\n", - __func__, nolock_regw_read(priv, EWRPTL)); + dev_printk(KERN_DEBUG, dev, + "%s() after control byte ERWPT:0x%04x\n", + __func__, nolock_regw_read(priv, EWRPTL)); /* copy the packet into the transmit buffer */ spi_write_buf(priv, len, data); if (netif_msg_hw(priv)) - printk(KERN_DEBUG DRV_NAME - ": %s() after write packet ERWPT:0x%04x, len=%d\n", - __func__, nolock_regw_read(priv, EWRPTL), len); + dev_printk(KERN_DEBUG, dev, + "%s() after write packet ERWPT:0x%04x, len=%d\n", + __func__, nolock_regw_read(priv, EWRPTL), len); mutex_unlock(&priv->lock); } -static unsigned long msec20_to_jiffies; - static int poll_ready(struct enc28j60_net *priv, u8 reg, u8 mask, u8 val) { - unsigned long timeout = jiffies + msec20_to_jiffies; + struct device *dev = &priv->spi->dev; + unsigned long timeout = jiffies + msecs_to_jiffies(20); /* 20 msec timeout read */ while ((nolock_regb_read(priv, reg) & mask) != val) { if (time_after(jiffies, timeout)) { if (netif_msg_drv(priv)) - dev_dbg(&priv->spi->dev, - "reg %02x ready timeout!\n", reg); + dev_dbg(dev, "reg %02x ready timeout!\n", reg); return -ETIMEDOUT; } cpu_relax(); @@ -449,7 +434,7 @@ static int wait_phy_ready(struct enc28j60_net *priv) /* * PHY register read - * PHY registers are not accessed directly, but through the MII + * PHY registers are not accessed directly, but through the MII. */ static u16 enc28j60_phy_read(struct enc28j60_net *priv, u8 address) { @@ -465,7 +450,7 @@ static u16 enc28j60_phy_read(struct enc28j60_net *priv, u8 address) /* quit reading */ nolock_regb_write(priv, MICMD, 0x00); /* return the data */ - ret = nolock_regw_read(priv, MIRDL); + ret = nolock_regw_read(priv, MIRDL); mutex_unlock(&priv->lock); return ret; @@ -494,13 +479,13 @@ static int enc28j60_set_hw_macaddr(struct net_device *ndev) { int ret; struct enc28j60_net *priv = netdev_priv(ndev); + struct device *dev = &priv->spi->dev; mutex_lock(&priv->lock); if (!priv->hw_enable) { if (netif_msg_drv(priv)) - printk(KERN_INFO DRV_NAME - ": %s: Setting MAC address to %pM\n", - ndev->name, ndev->dev_addr); + dev_info(dev, "%s: Setting MAC address to %pM\n", + ndev->name, ndev->dev_addr); /* NOTE: MAC address in ENC28J60 is byte-backward */ nolock_regb_write(priv, MAADR5, ndev->dev_addr[0]); nolock_regb_write(priv, MAADR4, ndev->dev_addr[1]); @@ -511,9 +496,9 @@ static int enc28j60_set_hw_macaddr(struct net_device *ndev) ret = 0; } else { if (netif_msg_drv(priv)) - printk(KERN_DEBUG DRV_NAME - ": %s() Hardware must be disabled to set " - "Mac address\n", __func__); + dev_printk(KERN_DEBUG, dev, + "%s() Hardware must be disabled to set Mac address\n", + __func__); ret = -EBUSY; } mutex_unlock(&priv->lock); @@ -532,7 +517,7 @@ static int enc28j60_set_mac_address(struct net_device *dev, void *addr) if (!is_valid_ether_addr(address->sa_data)) return -EADDRNOTAVAIL; - memcpy(dev->dev_addr, address->sa_data, dev->addr_len); + ether_addr_copy(dev->dev_addr, address->sa_data); return enc28j60_set_hw_macaddr(dev); } @@ -541,33 +526,36 @@ static int enc28j60_set_mac_address(struct net_device *dev, void *addr) */ static void enc28j60_dump_regs(struct enc28j60_net *priv, const char *msg) { + struct device *dev = &priv->spi->dev; + mutex_lock(&priv->lock); - printk(KERN_DEBUG DRV_NAME " %s\n" - "HwRevID: 0x%02x\n" - "Cntrl: ECON1 ECON2 ESTAT EIR EIE\n" - " 0x%02x 0x%02x 0x%02x 0x%02x 0x%02x\n" - "MAC : MACON1 MACON3 MACON4\n" - " 0x%02x 0x%02x 0x%02x\n" - "Rx : ERXST ERXND ERXWRPT ERXRDPT ERXFCON EPKTCNT MAMXFL\n" - " 0x%04x 0x%04x 0x%04x 0x%04x " - "0x%02x 0x%02x 0x%04x\n" - "Tx : ETXST ETXND MACLCON1 MACLCON2 MAPHSUP\n" - " 0x%04x 0x%04x 0x%02x 0x%02x 0x%02x\n", - msg, nolock_regb_read(priv, EREVID), - nolock_regb_read(priv, ECON1), nolock_regb_read(priv, ECON2), - nolock_regb_read(priv, ESTAT), nolock_regb_read(priv, EIR), - nolock_regb_read(priv, EIE), nolock_regb_read(priv, MACON1), - nolock_regb_read(priv, MACON3), nolock_regb_read(priv, MACON4), - nolock_regw_read(priv, ERXSTL), nolock_regw_read(priv, ERXNDL), - nolock_regw_read(priv, ERXWRPTL), - nolock_regw_read(priv, ERXRDPTL), - nolock_regb_read(priv, ERXFCON), - nolock_regb_read(priv, EPKTCNT), - nolock_regw_read(priv, MAMXFLL), nolock_regw_read(priv, ETXSTL), - nolock_regw_read(priv, ETXNDL), - nolock_regb_read(priv, MACLCON1), - nolock_regb_read(priv, MACLCON2), - nolock_regb_read(priv, MAPHSUP)); + dev_printk(KERN_DEBUG, dev, + " %s\n" + "HwRevID: 0x%02x\n" + "Cntrl: ECON1 ECON2 ESTAT EIR EIE\n" + " 0x%02x 0x%02x 0x%02x 0x%02x 0x%02x\n" + "MAC : MACON1 MACON3 MACON4\n" + " 0x%02x 0x%02x 0x%02x\n" + "Rx : ERXST ERXND ERXWRPT ERXRDPT ERXFCON EPKTCNT MAMXFL\n" + " 0x%04x 0x%04x 0x%04x 0x%04x " + "0x%02x 0x%02x 0x%04x\n" + "Tx : ETXST ETXND MACLCON1 MACLCON2 MAPHSUP\n" + " 0x%04x 0x%04x 0x%02x 0x%02x 0x%02x\n", + msg, nolock_regb_read(priv, EREVID), + nolock_regb_read(priv, ECON1), nolock_regb_read(priv, ECON2), + nolock_regb_read(priv, ESTAT), nolock_regb_read(priv, EIR), + nolock_regb_read(priv, EIE), nolock_regb_read(priv, MACON1), + nolock_regb_read(priv, MACON3), nolock_regb_read(priv, MACON4), + nolock_regw_read(priv, ERXSTL), nolock_regw_read(priv, ERXNDL), + nolock_regw_read(priv, ERXWRPTL), + nolock_regw_read(priv, ERXRDPTL), + nolock_regb_read(priv, ERXFCON), + nolock_regb_read(priv, EPKTCNT), + nolock_regw_read(priv, MAMXFLL), nolock_regw_read(priv, ETXSTL), + nolock_regw_read(priv, ETXNDL), + nolock_regb_read(priv, MACLCON1), + nolock_regb_read(priv, MACLCON2), + nolock_regb_read(priv, MAPHSUP)); mutex_unlock(&priv->lock); } @@ -599,12 +587,13 @@ static u16 rx_packet_start(u16 ptr) static void nolock_rxfifo_init(struct enc28j60_net *priv, u16 start, u16 end) { + struct device *dev = &priv->spi->dev; u16 erxrdpt; if (start > 0x1FFF || end > 0x1FFF || start > end) { if (netif_msg_drv(priv)) - printk(KERN_ERR DRV_NAME ": %s(%d, %d) RXFIFO " - "bad parameters!\n", __func__, start, end); + dev_err(dev, "%s(%d, %d) RXFIFO bad parameters!\n", + __func__, start, end); return; } /* set receive buffer start + end */ @@ -617,10 +606,12 @@ static void nolock_rxfifo_init(struct enc28j60_net *priv, u16 start, u16 end) static void nolock_txfifo_init(struct enc28j60_net *priv, u16 start, u16 end) { + struct device *dev = &priv->spi->dev; + if (start > 0x1FFF || end > 0x1FFF || start > end) { if (netif_msg_drv(priv)) - printk(KERN_ERR DRV_NAME ": %s(%d, %d) TXFIFO " - "bad parameters!\n", __func__, start, end); + dev_err(dev, "%s(%d, %d) TXFIFO bad parameters!\n", + __func__, start, end); return; } /* set transmit buffer start + end */ @@ -630,14 +621,15 @@ static void nolock_txfifo_init(struct enc28j60_net *priv, u16 start, u16 end) /* * Low power mode shrinks power consumption about 100x, so we'd like - * the chip to be in that mode whenever it's inactive. (However, we - * can't stay in lowpower mode during suspend with WOL active.) + * the chip to be in that mode whenever it's inactive. (However, we + * can't stay in low power mode during suspend with WOL active.) */ static void enc28j60_lowpower(struct enc28j60_net *priv, bool is_low) { + struct device *dev = &priv->spi->dev; + if (netif_msg_drv(priv)) - dev_dbg(&priv->spi->dev, "%s power...\n", - is_low ? "low" : "high"); + dev_dbg(dev, "%s power...\n", is_low ? "low" : "high"); mutex_lock(&priv->lock); if (is_low) { @@ -656,11 +648,12 @@ static void enc28j60_lowpower(struct enc28j60_net *priv, bool is_low) static int enc28j60_hw_init(struct enc28j60_net *priv) { + struct device *dev = &priv->spi->dev; u8 reg; if (netif_msg_drv(priv)) - printk(KERN_DEBUG DRV_NAME ": %s() - %s\n", __func__, - priv->full_duplex ? "FullDuplex" : "HalfDuplex"); + dev_printk(KERN_DEBUG, dev, "%s() - %s\n", __func__, + priv->full_duplex ? "FullDuplex" : "HalfDuplex"); mutex_lock(&priv->lock); /* first reset the chip */ @@ -682,15 +675,15 @@ static int enc28j60_hw_init(struct enc28j60_net *priv) /* * Check the RevID. * If it's 0x00 or 0xFF probably the enc28j60 is not mounted or - * damaged + * damaged. */ reg = locked_regb_read(priv, EREVID); if (netif_msg_drv(priv)) - printk(KERN_INFO DRV_NAME ": chip RevID: 0x%02x\n", reg); + dev_info(dev, "chip RevID: 0x%02x\n", reg); if (reg == 0x00 || reg == 0xff) { if (netif_msg_drv(priv)) - printk(KERN_DEBUG DRV_NAME ": %s() Invalid RevId %d\n", - __func__, reg); + dev_printk(KERN_DEBUG, dev, "%s() Invalid RevId %d\n", + __func__, reg); return 0; } @@ -723,7 +716,7 @@ static int enc28j60_hw_init(struct enc28j60_net *priv) /* * MACLCON1 (default) * MACLCON2 (default) - * Set the maximum packet size which the controller will accept + * Set the maximum packet size which the controller will accept. */ locked_regw_write(priv, MAMXFLL, MAX_FRAMELEN); @@ -750,10 +743,12 @@ static int enc28j60_hw_init(struct enc28j60_net *priv) static void enc28j60_hw_enable(struct enc28j60_net *priv) { + struct device *dev = &priv->spi->dev; + /* enable interrupts */ if (netif_msg_hw(priv)) - printk(KERN_DEBUG DRV_NAME ": %s() enabling interrupts.\n", - __func__); + dev_printk(KERN_DEBUG, dev, "%s() enabling interrupts.\n", + __func__); enc28j60_phy_write(priv, PHIE, PHIE_PGEIE | PHIE_PLNKIE); @@ -772,7 +767,7 @@ static void enc28j60_hw_enable(struct enc28j60_net *priv) static void enc28j60_hw_disable(struct enc28j60_net *priv) { mutex_lock(&priv->lock); - /* disable interrutps and packet reception */ + /* disable interrupts and packet reception */ nolock_regb_write(priv, EIE, 0x00); nolock_reg_bfclr(priv, ECON1, ECON1_RXEN); priv->hw_enable = false; @@ -793,14 +788,12 @@ enc28j60_setlink(struct net_device *ndev, u8 autoneg, u16 speed, u8 duplex) priv->full_duplex = (duplex == DUPLEX_FULL); else { if (netif_msg_link(priv)) - dev_warn(&ndev->dev, - "unsupported link setting\n"); + netdev_warn(ndev, "unsupported link setting\n"); ret = -EOPNOTSUPP; } } else { if (netif_msg_link(priv)) - dev_warn(&ndev->dev, "Warning: hw must be disabled " - "to set link mode\n"); + netdev_warn(ndev, "Warning: hw must be disabled to set link mode\n"); ret = -EBUSY; } return ret; @@ -811,21 +804,23 @@ enc28j60_setlink(struct net_device *ndev, u8 autoneg, u16 speed, u8 duplex) */ static void enc28j60_read_tsv(struct enc28j60_net *priv, u8 tsv[TSV_SIZE]) { + struct device *dev = &priv->spi->dev; int endptr; endptr = locked_regw_read(priv, ETXNDL); if (netif_msg_hw(priv)) - printk(KERN_DEBUG DRV_NAME ": reading TSV at addr:0x%04x\n", - endptr + 1); + dev_printk(KERN_DEBUG, dev, "reading TSV at addr:0x%04x\n", + endptr + 1); enc28j60_mem_read(priv, endptr + 1, TSV_SIZE, tsv); } static void enc28j60_dump_tsv(struct enc28j60_net *priv, const char *msg, - u8 tsv[TSV_SIZE]) + u8 tsv[TSV_SIZE]) { + struct device *dev = &priv->spi->dev; u16 tmp1, tmp2; - printk(KERN_DEBUG DRV_NAME ": %s - TSV:\n", msg); + dev_printk(KERN_DEBUG, dev, "%s - TSV:\n", msg); tmp1 = tsv[1]; tmp1 <<= 8; tmp1 |= tsv[0]; @@ -834,30 +829,32 @@ static void enc28j60_dump_tsv(struct enc28j60_net *priv, const char *msg, tmp2 <<= 8; tmp2 |= tsv[4]; - printk(KERN_DEBUG DRV_NAME ": ByteCount: %d, CollisionCount: %d," - " TotByteOnWire: %d\n", tmp1, tsv[2] & 0x0f, tmp2); - printk(KERN_DEBUG DRV_NAME ": TxDone: %d, CRCErr:%d, LenChkErr: %d," - " LenOutOfRange: %d\n", TSV_GETBIT(tsv, TSV_TXDONE), - TSV_GETBIT(tsv, TSV_TXCRCERROR), - TSV_GETBIT(tsv, TSV_TXLENCHKERROR), - TSV_GETBIT(tsv, TSV_TXLENOUTOFRANGE)); - printk(KERN_DEBUG DRV_NAME ": Multicast: %d, Broadcast: %d, " - "PacketDefer: %d, ExDefer: %d\n", - TSV_GETBIT(tsv, TSV_TXMULTICAST), - TSV_GETBIT(tsv, TSV_TXBROADCAST), - TSV_GETBIT(tsv, TSV_TXPACKETDEFER), - TSV_GETBIT(tsv, TSV_TXEXDEFER)); - printk(KERN_DEBUG DRV_NAME ": ExCollision: %d, LateCollision: %d, " - "Giant: %d, Underrun: %d\n", - TSV_GETBIT(tsv, TSV_TXEXCOLLISION), - TSV_GETBIT(tsv, TSV_TXLATECOLLISION), - TSV_GETBIT(tsv, TSV_TXGIANT), TSV_GETBIT(tsv, TSV_TXUNDERRUN)); - printk(KERN_DEBUG DRV_NAME ": ControlFrame: %d, PauseFrame: %d, " - "BackPressApp: %d, VLanTagFrame: %d\n", - TSV_GETBIT(tsv, TSV_TXCONTROLFRAME), - TSV_GETBIT(tsv, TSV_TXPAUSEFRAME), - TSV_GETBIT(tsv, TSV_BACKPRESSUREAPP), - TSV_GETBIT(tsv, TSV_TXVLANTAGFRAME)); + dev_printk(KERN_DEBUG, dev, + "ByteCount: %d, CollisionCount: %d, TotByteOnWire: %d\n", + tmp1, tsv[2] & 0x0f, tmp2); + dev_printk(KERN_DEBUG, dev, + "TxDone: %d, CRCErr:%d, LenChkErr: %d, LenOutOfRange: %d\n", + TSV_GETBIT(tsv, TSV_TXDONE), + TSV_GETBIT(tsv, TSV_TXCRCERROR), + TSV_GETBIT(tsv, TSV_TXLENCHKERROR), + TSV_GETBIT(tsv, TSV_TXLENOUTOFRANGE)); + dev_printk(KERN_DEBUG, dev, + "Multicast: %d, Broadcast: %d, PacketDefer: %d, ExDefer: %d\n", + TSV_GETBIT(tsv, TSV_TXMULTICAST), + TSV_GETBIT(tsv, TSV_TXBROADCAST), + TSV_GETBIT(tsv, TSV_TXPACKETDEFER), + TSV_GETBIT(tsv, TSV_TXEXDEFER)); + dev_printk(KERN_DEBUG, dev, + "ExCollision: %d, LateCollision: %d, Giant: %d, Underrun: %d\n", + TSV_GETBIT(tsv, TSV_TXEXCOLLISION), + TSV_GETBIT(tsv, TSV_TXLATECOLLISION), + TSV_GETBIT(tsv, TSV_TXGIANT), TSV_GETBIT(tsv, TSV_TXUNDERRUN)); + dev_printk(KERN_DEBUG, dev, + "ControlFrame: %d, PauseFrame: %d, BackPressApp: %d, VLanTagFrame: %d\n", + TSV_GETBIT(tsv, TSV_TXCONTROLFRAME), + TSV_GETBIT(tsv, TSV_TXPAUSEFRAME), + TSV_GETBIT(tsv, TSV_BACKPRESSUREAPP), + TSV_GETBIT(tsv, TSV_TXVLANTAGFRAME)); } /* @@ -866,27 +863,29 @@ static void enc28j60_dump_tsv(struct enc28j60_net *priv, const char *msg, static void enc28j60_dump_rsv(struct enc28j60_net *priv, const char *msg, u16 pk_ptr, int len, u16 sts) { - printk(KERN_DEBUG DRV_NAME ": %s - NextPk: 0x%04x - RSV:\n", - msg, pk_ptr); - printk(KERN_DEBUG DRV_NAME ": ByteCount: %d, DribbleNibble: %d\n", len, - RSV_GETBIT(sts, RSV_DRIBBLENIBBLE)); - printk(KERN_DEBUG DRV_NAME ": RxOK: %d, CRCErr:%d, LenChkErr: %d," - " LenOutOfRange: %d\n", RSV_GETBIT(sts, RSV_RXOK), - RSV_GETBIT(sts, RSV_CRCERROR), - RSV_GETBIT(sts, RSV_LENCHECKERR), - RSV_GETBIT(sts, RSV_LENOUTOFRANGE)); - printk(KERN_DEBUG DRV_NAME ": Multicast: %d, Broadcast: %d, " - "LongDropEvent: %d, CarrierEvent: %d\n", - RSV_GETBIT(sts, RSV_RXMULTICAST), - RSV_GETBIT(sts, RSV_RXBROADCAST), - RSV_GETBIT(sts, RSV_RXLONGEVDROPEV), - RSV_GETBIT(sts, RSV_CARRIEREV)); - printk(KERN_DEBUG DRV_NAME ": ControlFrame: %d, PauseFrame: %d," - " UnknownOp: %d, VLanTagFrame: %d\n", - RSV_GETBIT(sts, RSV_RXCONTROLFRAME), - RSV_GETBIT(sts, RSV_RXPAUSEFRAME), - RSV_GETBIT(sts, RSV_RXUNKNOWNOPCODE), - RSV_GETBIT(sts, RSV_RXTYPEVLAN)); + struct device *dev = &priv->spi->dev; + + dev_printk(KERN_DEBUG, dev, "%s - NextPk: 0x%04x - RSV:\n", msg, pk_ptr); + dev_printk(KERN_DEBUG, dev, "ByteCount: %d, DribbleNibble: %d\n", + len, RSV_GETBIT(sts, RSV_DRIBBLENIBBLE)); + dev_printk(KERN_DEBUG, dev, + "RxOK: %d, CRCErr:%d, LenChkErr: %d, LenOutOfRange: %d\n", + RSV_GETBIT(sts, RSV_RXOK), + RSV_GETBIT(sts, RSV_CRCERROR), + RSV_GETBIT(sts, RSV_LENCHECKERR), + RSV_GETBIT(sts, RSV_LENOUTOFRANGE)); + dev_printk(KERN_DEBUG, dev, + "Multicast: %d, Broadcast: %d, LongDropEvent: %d, CarrierEvent: %d\n", + RSV_GETBIT(sts, RSV_RXMULTICAST), + RSV_GETBIT(sts, RSV_RXBROADCAST), + RSV_GETBIT(sts, RSV_RXLONGEVDROPEV), + RSV_GETBIT(sts, RSV_CARRIEREV)); + dev_printk(KERN_DEBUG, dev, + "ControlFrame: %d, PauseFrame: %d, UnknownOp: %d, VLanTagFrame: %d\n", + RSV_GETBIT(sts, RSV_RXCONTROLFRAME), + RSV_GETBIT(sts, RSV_RXPAUSEFRAME), + RSV_GETBIT(sts, RSV_RXUNKNOWNOPCODE), + RSV_GETBIT(sts, RSV_RXTYPEVLAN)); } static void dump_packet(const char *msg, int len, const char *data) @@ -904,20 +903,20 @@ static void dump_packet(const char *msg, int len, const char *data) static void enc28j60_hw_rx(struct net_device *ndev) { struct enc28j60_net *priv = netdev_priv(ndev); + struct device *dev = &priv->spi->dev; struct sk_buff *skb = NULL; u16 erxrdpt, next_packet, rxstat; u8 rsv[RSV_SIZE]; int len; if (netif_msg_rx_status(priv)) - printk(KERN_DEBUG DRV_NAME ": RX pk_addr:0x%04x\n", - priv->next_pk_ptr); + netdev_printk(KERN_DEBUG, ndev, "RX pk_addr:0x%04x\n", + priv->next_pk_ptr); if (unlikely(priv->next_pk_ptr > RXEND_INIT)) { if (netif_msg_rx_err(priv)) - dev_err(&ndev->dev, - "%s() Invalid packet address!! 0x%04x\n", - __func__, priv->next_pk_ptr); + netdev_err(ndev, "%s() Invalid packet address!! 0x%04x\n", + __func__, priv->next_pk_ptr); /* packet address corrupted: reset RX logic */ mutex_lock(&priv->lock); nolock_reg_bfclr(priv, ECON1, ECON1_RXEN); @@ -950,7 +949,7 @@ static void enc28j60_hw_rx(struct net_device *ndev) if (!RSV_GETBIT(rxstat, RSV_RXOK) || len > MAX_FRAMELEN) { if (netif_msg_rx_err(priv)) - dev_err(&ndev->dev, "Rx Error (%04x)\n", rxstat); + netdev_err(ndev, "Rx Error (%04x)\n", rxstat); ndev->stats.rx_errors++; if (RSV_GETBIT(rxstat, RSV_CRCERROR)) ndev->stats.rx_crc_errors++; @@ -962,8 +961,7 @@ static void enc28j60_hw_rx(struct net_device *ndev) skb = netdev_alloc_skb(ndev, len + NET_IP_ALIGN); if (!skb) { if (netif_msg_rx_err(priv)) - dev_err(&ndev->dev, - "out of memory for Rx'd frame\n"); + netdev_err(ndev, "out of memory for Rx'd frame\n"); ndev->stats.rx_dropped++; } else { skb_reserve(skb, NET_IP_ALIGN); @@ -983,12 +981,12 @@ static void enc28j60_hw_rx(struct net_device *ndev) /* * Move the RX read pointer to the start of the next * received packet. - * This frees the memory we just read out + * This frees the memory we just read out. */ erxrdpt = erxrdpt_workaround(next_packet, RXSTART_INIT, RXEND_INIT); if (netif_msg_hw(priv)) - printk(KERN_DEBUG DRV_NAME ": %s() ERXRDPT:0x%04x\n", - __func__, erxrdpt); + dev_printk(KERN_DEBUG, dev, "%s() ERXRDPT:0x%04x\n", + __func__, erxrdpt); mutex_lock(&priv->lock); nolock_regw_write(priv, ERXRDPTL, erxrdpt); @@ -997,9 +995,9 @@ static void enc28j60_hw_rx(struct net_device *ndev) u16 reg; reg = nolock_regw_read(priv, ERXRDPTL); if (reg != erxrdpt) - printk(KERN_DEBUG DRV_NAME ": %s() ERXRDPT verify " - "error (0x%04x - 0x%04x)\n", __func__, - reg, erxrdpt); + dev_printk(KERN_DEBUG, dev, + "%s() ERXRDPT verify error (0x%04x - 0x%04x)\n", + __func__, reg, erxrdpt); } #endif priv->next_pk_ptr = next_packet; @@ -1013,6 +1011,7 @@ static void enc28j60_hw_rx(struct net_device *ndev) */ static int enc28j60_get_free_rxfifo(struct enc28j60_net *priv) { + struct net_device *ndev = priv->netdev; int epkcnt, erxst, erxnd, erxwr, erxrd; int free_space; @@ -1035,8 +1034,8 @@ static int enc28j60_get_free_rxfifo(struct enc28j60_net *priv) } mutex_unlock(&priv->lock); if (netif_msg_rx_status(priv)) - printk(KERN_DEBUG DRV_NAME ": %s() free_space = %d\n", - __func__, free_space); + netdev_printk(KERN_DEBUG, ndev, "%s() free_space = %d\n", + __func__, free_space); return free_space; } @@ -1046,24 +1045,25 @@ static int enc28j60_get_free_rxfifo(struct enc28j60_net *priv) static void enc28j60_check_link_status(struct net_device *ndev) { struct enc28j60_net *priv = netdev_priv(ndev); + struct device *dev = &priv->spi->dev; u16 reg; int duplex; reg = enc28j60_phy_read(priv, PHSTAT2); if (netif_msg_hw(priv)) - printk(KERN_DEBUG DRV_NAME ": %s() PHSTAT1: %04x, " - "PHSTAT2: %04x\n", __func__, - enc28j60_phy_read(priv, PHSTAT1), reg); + dev_printk(KERN_DEBUG, dev, + "%s() PHSTAT1: %04x, PHSTAT2: %04x\n", __func__, + enc28j60_phy_read(priv, PHSTAT1), reg); duplex = reg & PHSTAT2_DPXSTAT; if (reg & PHSTAT2_LSTAT) { netif_carrier_on(ndev); if (netif_msg_ifup(priv)) - dev_info(&ndev->dev, "link up - %s\n", - duplex ? "Full duplex" : "Half duplex"); + netdev_info(ndev, "link up - %s\n", + duplex ? "Full duplex" : "Half duplex"); } else { if (netif_msg_ifdown(priv)) - dev_info(&ndev->dev, "link down\n"); + netdev_info(ndev, "link down\n"); netif_carrier_off(ndev); } } @@ -1089,8 +1089,8 @@ static void enc28j60_tx_clear(struct net_device *ndev, bool err) /* * RX handler - * ignore PKTIF because is unreliable! (look at the errata datasheet) - * check EPKTCNT is the suggested workaround. + * Ignore PKTIF because is unreliable! (Look at the errata datasheet) + * Check EPKTCNT is the suggested workaround. * We don't need to clear interrupt flag, automatically done when * enc28j60_hw_rx() decrements the packet counter. * Returns how many packet processed. @@ -1102,13 +1102,14 @@ static int enc28j60_rx_interrupt(struct net_device *ndev) pk_counter = locked_regb_read(priv, EPKTCNT); if (pk_counter && netif_msg_intr(priv)) - printk(KERN_DEBUG DRV_NAME ": intRX, pk_cnt: %d\n", pk_counter); + netdev_printk(KERN_DEBUG, ndev, "intRX, pk_cnt: %d\n", + pk_counter); if (pk_counter > priv->max_pk_counter) { /* update statistics */ priv->max_pk_counter = pk_counter; if (netif_msg_rx_status(priv) && priv->max_pk_counter > 1) - printk(KERN_DEBUG DRV_NAME ": RX max_pk_cnt: %d\n", - priv->max_pk_counter); + netdev_printk(KERN_DEBUG, ndev, "RX max_pk_cnt: %d\n", + priv->max_pk_counter); } ret = pk_counter; while (pk_counter-- > 0) @@ -1124,8 +1125,6 @@ static void enc28j60_irq_work_handler(struct work_struct *work) struct net_device *ndev = priv->netdev; int intflags, loop; - if (netif_msg_intr(priv)) - printk(KERN_DEBUG DRV_NAME ": %s() enter\n", __func__); /* disable further interrupts */ locked_reg_bfclr(priv, EIE, EIE_INTIE); @@ -1136,16 +1135,16 @@ static void enc28j60_irq_work_handler(struct work_struct *work) if ((intflags & EIR_DMAIF) != 0) { loop++; if (netif_msg_intr(priv)) - printk(KERN_DEBUG DRV_NAME - ": intDMA(%d)\n", loop); + netdev_printk(KERN_DEBUG, ndev, "intDMA(%d)\n", + loop); locked_reg_bfclr(priv, EIR, EIR_DMAIF); } /* LINK changed handler */ if ((intflags & EIR_LINKIF) != 0) { loop++; if (netif_msg_intr(priv)) - printk(KERN_DEBUG DRV_NAME - ": intLINK(%d)\n", loop); + netdev_printk(KERN_DEBUG, ndev, "intLINK(%d)\n", + loop); enc28j60_check_link_status(ndev); /* read PHIR to clear the flag */ enc28j60_phy_read(priv, PHIR); @@ -1156,13 +1155,12 @@ static void enc28j60_irq_work_handler(struct work_struct *work) bool err = false; loop++; if (netif_msg_intr(priv)) - printk(KERN_DEBUG DRV_NAME - ": intTX(%d)\n", loop); + netdev_printk(KERN_DEBUG, ndev, "intTX(%d)\n", + loop); priv->tx_retry_count = 0; if (locked_regb_read(priv, ESTAT) & ESTAT_TXABRT) { if (netif_msg_tx_err(priv)) - dev_err(&ndev->dev, - "Tx Error (aborted)\n"); + netdev_err(ndev, "Tx Error (aborted)\n"); err = true; } if (netif_msg_tx_done(priv)) { @@ -1179,8 +1177,8 @@ static void enc28j60_irq_work_handler(struct work_struct *work) loop++; if (netif_msg_intr(priv)) - printk(KERN_DEBUG DRV_NAME - ": intTXErr(%d)\n", loop); + netdev_printk(KERN_DEBUG, ndev, "intTXErr(%d)\n", + loop); locked_reg_bfclr(priv, ECON1, ECON1_TXRTS); enc28j60_read_tsv(priv, tsv); if (netif_msg_tx_err(priv)) @@ -1194,9 +1192,9 @@ static void enc28j60_irq_work_handler(struct work_struct *work) /* Transmit Late collision check for retransmit */ if (TSV_GETBIT(tsv, TSV_TXLATECOLLISION)) { if (netif_msg_tx_err(priv)) - printk(KERN_DEBUG DRV_NAME - ": LateCollision TXErr (%d)\n", - priv->tx_retry_count); + netdev_printk(KERN_DEBUG, ndev, + "LateCollision TXErr (%d)\n", + priv->tx_retry_count); if (priv->tx_retry_count++ < MAX_TX_RETRYCOUNT) locked_reg_bfset(priv, ECON1, ECON1_TXRTS); @@ -1210,13 +1208,12 @@ static void enc28j60_irq_work_handler(struct work_struct *work) if ((intflags & EIR_RXERIF) != 0) { loop++; if (netif_msg_intr(priv)) - printk(KERN_DEBUG DRV_NAME - ": intRXErr(%d)\n", loop); + netdev_printk(KERN_DEBUG, ndev, "intRXErr(%d)\n", + loop); /* Check free FIFO space to flag RX overrun */ if (enc28j60_get_free_rxfifo(priv) <= 0) { if (netif_msg_rx_err(priv)) - printk(KERN_DEBUG DRV_NAME - ": RX Overrun\n"); + netdev_printk(KERN_DEBUG, ndev, "RX Overrun\n"); ndev->stats.rx_dropped++; } locked_reg_bfclr(priv, EIR, EIR_RXERIF); @@ -1228,8 +1225,6 @@ static void enc28j60_irq_work_handler(struct work_struct *work) /* re-enable interrupts */ locked_reg_bfset(priv, EIE, EIE_INTIE); - if (netif_msg_intr(priv)) - printk(KERN_DEBUG DRV_NAME ": %s() exit\n", __func__); } /* @@ -1239,11 +1234,13 @@ static void enc28j60_irq_work_handler(struct work_struct *work) */ static void enc28j60_hw_tx(struct enc28j60_net *priv) { + struct net_device *ndev = priv->netdev; + BUG_ON(!priv->tx_skb); if (netif_msg_tx_queued(priv)) - printk(KERN_DEBUG DRV_NAME - ": Tx Packet Len:%d\n", priv->tx_skb->len); + netdev_printk(KERN_DEBUG, ndev, "Tx Packet Len:%d\n", + priv->tx_skb->len); if (netif_msg_pktdata(priv)) dump_packet(__func__, @@ -1253,6 +1250,7 @@ static void enc28j60_hw_tx(struct enc28j60_net *priv) #ifdef CONFIG_ENC28J60_WRITEVERIFY /* readback and verify written data */ if (netif_msg_drv(priv)) { + struct device *dev = &priv->spi->dev; int test_len, k; u8 test_buf[64]; /* limit the test to the first 64 bytes */ int okflag; @@ -1266,16 +1264,14 @@ static void enc28j60_hw_tx(struct enc28j60_net *priv) okflag = 1; for (k = 0; k < test_len; k++) { if (priv->tx_skb->data[k] != test_buf[k]) { - printk(KERN_DEBUG DRV_NAME - ": Error, %d location differ: " - "0x%02x-0x%02x\n", k, - priv->tx_skb->data[k], test_buf[k]); + dev_printk(KERN_DEBUG, dev, + "Error, %d location differ: 0x%02x-0x%02x\n", + k, priv->tx_skb->data[k], test_buf[k]); okflag = 0; } } if (!okflag) - printk(KERN_DEBUG DRV_NAME ": Tx write buffer, " - "verify ERROR!\n"); + dev_printk(KERN_DEBUG, dev, "Tx write buffer, verify ERROR!\n"); } #endif /* set TX request flag */ @@ -1287,14 +1283,11 @@ static netdev_tx_t enc28j60_send_packet(struct sk_buff *skb, { struct enc28j60_net *priv = netdev_priv(dev); - if (netif_msg_tx_queued(priv)) - printk(KERN_DEBUG DRV_NAME ": %s() enter\n", __func__); - /* If some error occurs while trying to transmit this * packet, you should return '1' from this function. * In such a case you _may not_ do anything to the * SKB, it is still owned by the network queueing - * layer when an error is returned. This means you + * layer when an error is returned. This means you * may not modify any SKB fields, you may not free * the SKB, etc. */ @@ -1337,7 +1330,7 @@ static void enc28j60_tx_timeout(struct net_device *ndev) struct enc28j60_net *priv = netdev_priv(ndev); if (netif_msg_timer(priv)) - dev_err(&ndev->dev, DRV_NAME " tx timeout\n"); + netdev_err(ndev, "tx timeout\n"); ndev->stats.tx_errors++; /* can't restart safely under softirq */ @@ -1356,13 +1349,9 @@ static int enc28j60_net_open(struct net_device *dev) { struct enc28j60_net *priv = netdev_priv(dev); - if (netif_msg_drv(priv)) - printk(KERN_DEBUG DRV_NAME ": %s() enter\n", __func__); - if (!is_valid_ether_addr(dev->dev_addr)) { if (netif_msg_ifup(priv)) - dev_err(&dev->dev, "invalid MAC address %pM\n", - dev->dev_addr); + netdev_err(dev, "invalid MAC address %pM\n", dev->dev_addr); return -EADDRNOTAVAIL; } /* Reset the hardware here (and take it out of low power mode) */ @@ -1370,7 +1359,7 @@ static int enc28j60_net_open(struct net_device *dev) enc28j60_hw_disable(priv); if (!enc28j60_hw_init(priv)) { if (netif_msg_ifup(priv)) - dev_err(&dev->dev, "hw_reset() failed\n"); + netdev_err(dev, "hw_reset() failed\n"); return -EINVAL; } /* Update the MAC address (in case user has changed it) */ @@ -1392,9 +1381,6 @@ static int enc28j60_net_close(struct net_device *dev) { struct enc28j60_net *priv = netdev_priv(dev); - if (netif_msg_drv(priv)) - printk(KERN_DEBUG DRV_NAME ": %s() enter\n", __func__); - enc28j60_hw_disable(priv); enc28j60_lowpower(priv, true); netif_stop_queue(dev); @@ -1415,16 +1401,16 @@ static void enc28j60_set_multicast_list(struct net_device *dev) if (dev->flags & IFF_PROMISC) { if (netif_msg_link(priv)) - dev_info(&dev->dev, "promiscuous mode\n"); + netdev_info(dev, "promiscuous mode\n"); priv->rxfilter = RXFILTER_PROMISC; } else if ((dev->flags & IFF_ALLMULTI) || !netdev_mc_empty(dev)) { if (netif_msg_link(priv)) - dev_info(&dev->dev, "%smulticast mode\n", - (dev->flags & IFF_ALLMULTI) ? "all-" : ""); + netdev_info(dev, "%smulticast mode\n", + (dev->flags & IFF_ALLMULTI) ? "all-" : ""); priv->rxfilter = RXFILTER_MULTI; } else { if (netif_msg_link(priv)) - dev_info(&dev->dev, "normal mode\n"); + netdev_info(dev, "normal mode\n"); priv->rxfilter = RXFILTER_NORMAL; } @@ -1436,20 +1422,21 @@ static void enc28j60_setrx_work_handler(struct work_struct *work) { struct enc28j60_net *priv = container_of(work, struct enc28j60_net, setrx_work); + struct device *dev = &priv->spi->dev; if (priv->rxfilter == RXFILTER_PROMISC) { if (netif_msg_drv(priv)) - printk(KERN_DEBUG DRV_NAME ": promiscuous mode\n"); + dev_printk(KERN_DEBUG, dev, "promiscuous mode\n"); locked_regb_write(priv, ERXFCON, 0x00); } else if (priv->rxfilter == RXFILTER_MULTI) { if (netif_msg_drv(priv)) - printk(KERN_DEBUG DRV_NAME ": multicast mode\n"); + dev_printk(KERN_DEBUG, dev, "multicast mode\n"); locked_regb_write(priv, ERXFCON, ERXFCON_UCEN | ERXFCON_CRCEN | ERXFCON_BCEN | ERXFCON_MCEN); } else { if (netif_msg_drv(priv)) - printk(KERN_DEBUG DRV_NAME ": normal mode\n"); + dev_printk(KERN_DEBUG, dev, "normal mode\n"); locked_regb_write(priv, ERXFCON, ERXFCON_UCEN | ERXFCON_CRCEN | ERXFCON_BCEN); @@ -1468,7 +1455,7 @@ static void enc28j60_restart_work_handler(struct work_struct *work) enc28j60_net_close(ndev); ret = enc28j60_net_open(ndev); if (unlikely(ret)) { - dev_info(&ndev->dev, " could not restart %d\n", ret); + netdev_info(ndev, "could not restart %d\n", ret); dev_close(ndev); } } @@ -1552,14 +1539,13 @@ static const struct net_device_ops enc28j60_netdev_ops = { static int enc28j60_probe(struct spi_device *spi) { + unsigned char macaddr[ETH_ALEN]; struct net_device *dev; struct enc28j60_net *priv; - const void *macaddr; int ret = 0; if (netif_msg_drv(&debug)) - dev_info(&spi->dev, DRV_NAME " Ethernet driver %s loaded\n", - DRV_VERSION); + dev_info(&spi->dev, "Ethernet driver %s loaded\n", DRV_VERSION); dev = alloc_etherdev(sizeof(struct enc28j60_net)); if (!dev) { @@ -1570,8 +1556,7 @@ static int enc28j60_probe(struct spi_device *spi) priv->netdev = dev; /* priv to netdev reference */ priv->spi = spi; /* priv to spi reference */ - priv->msg_enable = netif_msg_init(debug.msg_enable, - ENC28J60_MSG_DEFAULT); + priv->msg_enable = netif_msg_init(debug.msg_enable, ENC28J60_MSG_DEFAULT); mutex_init(&priv->lock); INIT_WORK(&priv->tx_work, enc28j60_tx_work_handler); INIT_WORK(&priv->setrx_work, enc28j60_setrx_work_handler); @@ -1582,13 +1567,12 @@ static int enc28j60_probe(struct spi_device *spi) if (!enc28j60_chipset_init(dev)) { if (netif_msg_probe(priv)) - dev_info(&spi->dev, DRV_NAME " chip not found\n"); + dev_info(&spi->dev, "chip not found\n"); ret = -EIO; goto error_irq; } - macaddr = of_get_mac_address(spi->dev.of_node); - if (macaddr) + if (device_get_mac_address(&spi->dev, macaddr, sizeof(macaddr))) ether_addr_copy(dev->dev_addr, macaddr); else eth_hw_addr_random(dev); @@ -1600,8 +1584,8 @@ static int enc28j60_probe(struct spi_device *spi) ret = request_irq(spi->irq, enc28j60_irq, 0, DRV_NAME, priv); if (ret < 0) { if (netif_msg_probe(priv)) - dev_err(&spi->dev, DRV_NAME ": request irq %d failed " - "(ret = %d)\n", spi->irq, ret); + dev_err(&spi->dev, "request irq %d failed (ret = %d)\n", + spi->irq, ret); goto error_irq; } @@ -1616,11 +1600,10 @@ static int enc28j60_probe(struct spi_device *spi) ret = register_netdev(dev); if (ret) { if (netif_msg_probe(priv)) - dev_err(&spi->dev, "register netdev " DRV_NAME - " failed (ret = %d)\n", ret); + dev_err(&spi->dev, "register netdev failed (ret = %d)\n", + ret); goto error_register; } - dev_info(&dev->dev, DRV_NAME " driver registered\n"); return 0; @@ -1636,9 +1619,6 @@ static int enc28j60_remove(struct spi_device *spi) { struct enc28j60_net *priv = spi_get_drvdata(spi); - if (netif_msg_drv(priv)) - printk(KERN_DEBUG DRV_NAME ": remove\n"); - unregister_netdev(priv->netdev); free_irq(spi->irq, priv); free_netdev(priv->netdev); @@ -1660,22 +1640,7 @@ static struct spi_driver enc28j60_driver = { .probe = enc28j60_probe, .remove = enc28j60_remove, }; - -static int __init enc28j60_init(void) -{ - msec20_to_jiffies = msecs_to_jiffies(20); - - return spi_register_driver(&enc28j60_driver); -} - -module_init(enc28j60_init); - -static void __exit enc28j60_exit(void) -{ - spi_unregister_driver(&enc28j60_driver); -} - -module_exit(enc28j60_exit); +module_spi_driver(enc28j60_driver); MODULE_DESCRIPTION(DRV_NAME " ethernet driver"); MODULE_AUTHOR("Claudio Lanconelli "); diff --git a/drivers/net/ethernet/netronome/Kconfig b/drivers/net/ethernet/netronome/Kconfig index 549898d5d450e328fe9308a96cae27daf5e9340b..f0d0e09f60e2162f52ed285ad3e2aa7f6330dea7 100644 --- a/drivers/net/ethernet/netronome/Kconfig +++ b/drivers/net/ethernet/netronome/Kconfig @@ -19,6 +19,7 @@ config NFP tristate "Netronome(R) NFP4000/NFP6000 NIC driver" depends on PCI && PCI_MSI depends on VXLAN || VXLAN=n + select NET_DEVLINK ---help--- This driver supports the Netronome(R) NFP4000/NFP6000 based cards working as a advanced Ethernet NIC. It works with both diff --git a/drivers/net/ethernet/netronome/nfp/Makefile b/drivers/net/ethernet/netronome/nfp/Makefile index 47c708f08ade56676b60190fd9e8e1b20ed62408..87bf784f8e8f8c596d7c7eea389ab7eb81cec663 100644 --- a/drivers/net/ethernet/netronome/nfp/Makefile +++ b/drivers/net/ethernet/netronome/nfp/Makefile @@ -15,6 +15,7 @@ nfp-objs := \ nfpcore/nfp_resource.o \ nfpcore/nfp_rtsym.o \ nfpcore/nfp_target.o \ + ccm.o \ nfp_asm.o \ nfp_app.o \ nfp_app_nic.o \ @@ -42,7 +43,8 @@ nfp-objs += \ flower/match.o \ flower/metadata.o \ flower/offload.o \ - flower/tunnel_conf.o + flower/tunnel_conf.o \ + flower/qos_conf.o endif ifeq ($(CONFIG_BPF_SYSCALL),y) diff --git a/drivers/net/ethernet/netronome/nfp/abm/ctrl.c b/drivers/net/ethernet/netronome/nfp/abm/ctrl.c index 9584f03f3efaf73fbd6894334ca47c98b2e18744..69e84ff7f2e5214e584c8b09722a069264acc0b0 100644 --- a/drivers/net/ethernet/netronome/nfp/abm/ctrl.c +++ b/drivers/net/ethernet/netronome/nfp/abm/ctrl.c @@ -261,10 +261,15 @@ int nfp_abm_ctrl_qm_disable(struct nfp_abm *abm) int nfp_abm_ctrl_prio_map_update(struct nfp_abm_link *alink, u32 *packed) { + const u32 cmd = NFP_NET_CFG_MBOX_CMD_PCI_DSCP_PRIOMAP_SET; struct nfp_net *nn = alink->vnic; unsigned int i; int err; + err = nfp_net_mbox_lock(nn, alink->abm->prio_map_len); + if (err) + return err; + /* Write data_len and wipe reserved */ nn_writeq(nn, nn->tlv_caps.mbox_off + NFP_NET_ABM_MBOX_DATALEN, alink->abm->prio_map_len); @@ -273,8 +278,7 @@ int nfp_abm_ctrl_prio_map_update(struct nfp_abm_link *alink, u32 *packed) nn_writel(nn, nn->tlv_caps.mbox_off + NFP_NET_ABM_MBOX_DATA + i, packed[i / sizeof(u32)]); - err = nfp_net_reconfig_mbox(nn, - NFP_NET_CFG_MBOX_CMD_PCI_DSCP_PRIOMAP_SET); + err = nfp_net_mbox_reconfig_and_unlock(nn, cmd); if (err) nfp_err(alink->abm->app->cpp, "setting DSCP -> VQ map failed with error %d\n", err); diff --git a/drivers/net/ethernet/netronome/nfp/abm/main.c b/drivers/net/ethernet/netronome/nfp/abm/main.c index 4d4ff5844c4735ef36f5a135e1df190e0089ee24..9183b3e85d217dec1328065f4594ce5e5895491d 100644 --- a/drivers/net/ethernet/netronome/nfp/abm/main.c +++ b/drivers/net/ethernet/netronome/nfp/abm/main.c @@ -53,7 +53,8 @@ nfp_abm_setup_tc(struct nfp_app *app, struct net_device *netdev, } } -static struct net_device *nfp_abm_repr_get(struct nfp_app *app, u32 port_id) +static struct net_device * +nfp_abm_repr_get(struct nfp_app *app, u32 port_id, bool *redir_egress) { enum nfp_repr_type rtype; struct nfp_reprs *reprs; @@ -549,5 +550,5 @@ const struct nfp_app_type app_abm = { .eswitch_mode_get = nfp_abm_eswitch_mode_get, .eswitch_mode_set = nfp_abm_eswitch_mode_set, - .repr_get = nfp_abm_repr_get, + .dev_get = nfp_abm_repr_get, }; diff --git a/drivers/net/ethernet/netronome/nfp/bpf/cmsg.c b/drivers/net/ethernet/netronome/nfp/bpf/cmsg.c index 9b6cfa697879391a611370e20daeb517d115aea7..bc9850e4ec5e646e7e21511a995c5315b1b26062 100644 --- a/drivers/net/ethernet/netronome/nfp/bpf/cmsg.c +++ b/drivers/net/ethernet/netronome/nfp/bpf/cmsg.c @@ -6,48 +6,13 @@ #include #include #include -#include +#include "../ccm.h" #include "../nfp_app.h" #include "../nfp_net.h" #include "fw.h" #include "main.h" -#define NFP_BPF_TAG_ALLOC_SPAN (U16_MAX / 4) - -static bool nfp_bpf_all_tags_busy(struct nfp_app_bpf *bpf) -{ - u16 used_tags; - - used_tags = bpf->tag_alloc_next - bpf->tag_alloc_last; - - return used_tags > NFP_BPF_TAG_ALLOC_SPAN; -} - -static int nfp_bpf_alloc_tag(struct nfp_app_bpf *bpf) -{ - /* All FW communication for BPF is request-reply. To make sure we - * don't reuse the message ID too early after timeout - limit the - * number of requests in flight. - */ - if (nfp_bpf_all_tags_busy(bpf)) { - cmsg_warn(bpf, "all FW request contexts busy!\n"); - return -EAGAIN; - } - - WARN_ON(__test_and_set_bit(bpf->tag_alloc_next, bpf->tag_allocator)); - return bpf->tag_alloc_next++; -} - -static void nfp_bpf_free_tag(struct nfp_app_bpf *bpf, u16 tag) -{ - WARN_ON(!__test_and_clear_bit(tag, bpf->tag_allocator)); - - while (!test_bit(bpf->tag_alloc_last, bpf->tag_allocator) && - bpf->tag_alloc_last != bpf->tag_alloc_next) - bpf->tag_alloc_last++; -} - static struct sk_buff * nfp_bpf_cmsg_alloc(struct nfp_app_bpf *bpf, unsigned int size) { @@ -87,149 +52,6 @@ nfp_bpf_cmsg_map_reply_size(struct nfp_app_bpf *bpf, unsigned int n) return size; } -static u8 nfp_bpf_cmsg_get_type(struct sk_buff *skb) -{ - struct cmsg_hdr *hdr; - - hdr = (struct cmsg_hdr *)skb->data; - - return hdr->type; -} - -static unsigned int nfp_bpf_cmsg_get_tag(struct sk_buff *skb) -{ - struct cmsg_hdr *hdr; - - hdr = (struct cmsg_hdr *)skb->data; - - return be16_to_cpu(hdr->tag); -} - -static struct sk_buff *__nfp_bpf_reply(struct nfp_app_bpf *bpf, u16 tag) -{ - unsigned int msg_tag; - struct sk_buff *skb; - - skb_queue_walk(&bpf->cmsg_replies, skb) { - msg_tag = nfp_bpf_cmsg_get_tag(skb); - if (msg_tag == tag) { - nfp_bpf_free_tag(bpf, tag); - __skb_unlink(skb, &bpf->cmsg_replies); - return skb; - } - } - - return NULL; -} - -static struct sk_buff *nfp_bpf_reply(struct nfp_app_bpf *bpf, u16 tag) -{ - struct sk_buff *skb; - - nfp_ctrl_lock(bpf->app->ctrl); - skb = __nfp_bpf_reply(bpf, tag); - nfp_ctrl_unlock(bpf->app->ctrl); - - return skb; -} - -static struct sk_buff *nfp_bpf_reply_drop_tag(struct nfp_app_bpf *bpf, u16 tag) -{ - struct sk_buff *skb; - - nfp_ctrl_lock(bpf->app->ctrl); - skb = __nfp_bpf_reply(bpf, tag); - if (!skb) - nfp_bpf_free_tag(bpf, tag); - nfp_ctrl_unlock(bpf->app->ctrl); - - return skb; -} - -static struct sk_buff * -nfp_bpf_cmsg_wait_reply(struct nfp_app_bpf *bpf, enum nfp_bpf_cmsg_type type, - int tag) -{ - struct sk_buff *skb; - int i, err; - - for (i = 0; i < 50; i++) { - udelay(4); - skb = nfp_bpf_reply(bpf, tag); - if (skb) - return skb; - } - - err = wait_event_interruptible_timeout(bpf->cmsg_wq, - skb = nfp_bpf_reply(bpf, tag), - msecs_to_jiffies(5000)); - /* We didn't get a response - try last time and atomically drop - * the tag even if no response is matched. - */ - if (!skb) - skb = nfp_bpf_reply_drop_tag(bpf, tag); - if (err < 0) { - cmsg_warn(bpf, "%s waiting for response to 0x%02x: %d\n", - err == ERESTARTSYS ? "interrupted" : "error", - type, err); - return ERR_PTR(err); - } - if (!skb) { - cmsg_warn(bpf, "timeout waiting for response to 0x%02x\n", - type); - return ERR_PTR(-ETIMEDOUT); - } - - return skb; -} - -static struct sk_buff * -nfp_bpf_cmsg_communicate(struct nfp_app_bpf *bpf, struct sk_buff *skb, - enum nfp_bpf_cmsg_type type, unsigned int reply_size) -{ - struct cmsg_hdr *hdr; - int tag; - - nfp_ctrl_lock(bpf->app->ctrl); - tag = nfp_bpf_alloc_tag(bpf); - if (tag < 0) { - nfp_ctrl_unlock(bpf->app->ctrl); - dev_kfree_skb_any(skb); - return ERR_PTR(tag); - } - - hdr = (void *)skb->data; - hdr->ver = CMSG_MAP_ABI_VERSION; - hdr->type = type; - hdr->tag = cpu_to_be16(tag); - - __nfp_app_ctrl_tx(bpf->app, skb); - - nfp_ctrl_unlock(bpf->app->ctrl); - - skb = nfp_bpf_cmsg_wait_reply(bpf, type, tag); - if (IS_ERR(skb)) - return skb; - - hdr = (struct cmsg_hdr *)skb->data; - if (hdr->type != __CMSG_REPLY(type)) { - cmsg_warn(bpf, "cmsg drop - wrong type 0x%02x != 0x%02lx!\n", - hdr->type, __CMSG_REPLY(type)); - goto err_free; - } - /* 0 reply_size means caller will do the validation */ - if (reply_size && skb->len != reply_size) { - cmsg_warn(bpf, "cmsg drop - type 0x%02x wrong size %d != %d!\n", - type, skb->len, reply_size); - goto err_free; - } - - return skb; -err_free: - dev_kfree_skb_any(skb); - return ERR_PTR(-EIO); -} - static int nfp_bpf_ctrl_rc_to_errno(struct nfp_app_bpf *bpf, struct cmsg_reply_map_simple *reply) @@ -275,8 +97,8 @@ nfp_bpf_ctrl_alloc_map(struct nfp_app_bpf *bpf, struct bpf_map *map) req->map_type = cpu_to_be32(map->map_type); req->map_flags = 0; - skb = nfp_bpf_cmsg_communicate(bpf, skb, CMSG_TYPE_MAP_ALLOC, - sizeof(*reply)); + skb = nfp_ccm_communicate(&bpf->ccm, skb, NFP_CCM_TYPE_BPF_MAP_ALLOC, + sizeof(*reply)); if (IS_ERR(skb)) return PTR_ERR(skb); @@ -310,8 +132,8 @@ void nfp_bpf_ctrl_free_map(struct nfp_app_bpf *bpf, struct nfp_bpf_map *nfp_map) req = (void *)skb->data; req->tid = cpu_to_be32(nfp_map->tid); - skb = nfp_bpf_cmsg_communicate(bpf, skb, CMSG_TYPE_MAP_FREE, - sizeof(*reply)); + skb = nfp_ccm_communicate(&bpf->ccm, skb, NFP_CCM_TYPE_BPF_MAP_FREE, + sizeof(*reply)); if (IS_ERR(skb)) { cmsg_warn(bpf, "leaking map - I/O error\n"); return; @@ -354,8 +176,7 @@ nfp_bpf_ctrl_reply_val(struct nfp_app_bpf *bpf, struct cmsg_reply_map_op *reply, } static int -nfp_bpf_ctrl_entry_op(struct bpf_offloaded_map *offmap, - enum nfp_bpf_cmsg_type op, +nfp_bpf_ctrl_entry_op(struct bpf_offloaded_map *offmap, enum nfp_ccm_type op, u8 *key, u8 *value, u64 flags, u8 *out_key, u8 *out_value) { struct nfp_bpf_map *nfp_map = offmap->dev_priv; @@ -386,8 +207,8 @@ nfp_bpf_ctrl_entry_op(struct bpf_offloaded_map *offmap, memcpy(nfp_bpf_ctrl_req_val(bpf, req, 0), value, map->value_size); - skb = nfp_bpf_cmsg_communicate(bpf, skb, op, - nfp_bpf_cmsg_map_reply_size(bpf, 1)); + skb = nfp_ccm_communicate(&bpf->ccm, skb, op, + nfp_bpf_cmsg_map_reply_size(bpf, 1)); if (IS_ERR(skb)) return PTR_ERR(skb); @@ -415,34 +236,34 @@ err_free: int nfp_bpf_ctrl_update_entry(struct bpf_offloaded_map *offmap, void *key, void *value, u64 flags) { - return nfp_bpf_ctrl_entry_op(offmap, CMSG_TYPE_MAP_UPDATE, + return nfp_bpf_ctrl_entry_op(offmap, NFP_CCM_TYPE_BPF_MAP_UPDATE, key, value, flags, NULL, NULL); } int nfp_bpf_ctrl_del_entry(struct bpf_offloaded_map *offmap, void *key) { - return nfp_bpf_ctrl_entry_op(offmap, CMSG_TYPE_MAP_DELETE, + return nfp_bpf_ctrl_entry_op(offmap, NFP_CCM_TYPE_BPF_MAP_DELETE, key, NULL, 0, NULL, NULL); } int nfp_bpf_ctrl_lookup_entry(struct bpf_offloaded_map *offmap, void *key, void *value) { - return nfp_bpf_ctrl_entry_op(offmap, CMSG_TYPE_MAP_LOOKUP, + return nfp_bpf_ctrl_entry_op(offmap, NFP_CCM_TYPE_BPF_MAP_LOOKUP, key, NULL, 0, NULL, value); } int nfp_bpf_ctrl_getfirst_entry(struct bpf_offloaded_map *offmap, void *next_key) { - return nfp_bpf_ctrl_entry_op(offmap, CMSG_TYPE_MAP_GETFIRST, + return nfp_bpf_ctrl_entry_op(offmap, NFP_CCM_TYPE_BPF_MAP_GETFIRST, NULL, NULL, 0, next_key, NULL); } int nfp_bpf_ctrl_getnext_entry(struct bpf_offloaded_map *offmap, void *key, void *next_key) { - return nfp_bpf_ctrl_entry_op(offmap, CMSG_TYPE_MAP_GETNEXT, + return nfp_bpf_ctrl_entry_op(offmap, NFP_CCM_TYPE_BPF_MAP_GETNEXT, key, NULL, 0, next_key, NULL); } @@ -456,54 +277,35 @@ unsigned int nfp_bpf_ctrl_cmsg_mtu(struct nfp_app_bpf *bpf) void nfp_bpf_ctrl_msg_rx(struct nfp_app *app, struct sk_buff *skb) { struct nfp_app_bpf *bpf = app->priv; - unsigned int tag; if (unlikely(skb->len < sizeof(struct cmsg_reply_map_simple))) { cmsg_warn(bpf, "cmsg drop - too short %d!\n", skb->len); - goto err_free; + dev_kfree_skb_any(skb); + return; } - if (nfp_bpf_cmsg_get_type(skb) == CMSG_TYPE_BPF_EVENT) { + if (nfp_ccm_get_type(skb) == NFP_CCM_TYPE_BPF_BPF_EVENT) { if (!nfp_bpf_event_output(bpf, skb->data, skb->len)) dev_consume_skb_any(skb); else dev_kfree_skb_any(skb); - return; } - nfp_ctrl_lock(bpf->app->ctrl); - - tag = nfp_bpf_cmsg_get_tag(skb); - if (unlikely(!test_bit(tag, bpf->tag_allocator))) { - cmsg_warn(bpf, "cmsg drop - no one is waiting for tag %u!\n", - tag); - goto err_unlock; - } - - __skb_queue_tail(&bpf->cmsg_replies, skb); - wake_up_interruptible_all(&bpf->cmsg_wq); - - nfp_ctrl_unlock(bpf->app->ctrl); - - return; -err_unlock: - nfp_ctrl_unlock(bpf->app->ctrl); -err_free: - dev_kfree_skb_any(skb); + nfp_ccm_rx(&bpf->ccm, skb); } void nfp_bpf_ctrl_msg_rx_raw(struct nfp_app *app, const void *data, unsigned int len) { + const struct nfp_ccm_hdr *hdr = data; struct nfp_app_bpf *bpf = app->priv; - const struct cmsg_hdr *hdr = data; if (unlikely(len < sizeof(struct cmsg_reply_map_simple))) { cmsg_warn(bpf, "cmsg drop - too short %d!\n", len); return; } - if (hdr->type == CMSG_TYPE_BPF_EVENT) + if (hdr->type == NFP_CCM_TYPE_BPF_BPF_EVENT) nfp_bpf_event_output(bpf, data, len); else cmsg_warn(bpf, "cmsg drop - msg type %d with raw buffer!\n", diff --git a/drivers/net/ethernet/netronome/nfp/bpf/fw.h b/drivers/net/ethernet/netronome/nfp/bpf/fw.h index 721921bcf120b8f126de424e4d5fc40e55487aa1..06c4286bd79e0a0b3578da221783a20d54d8a7f9 100644 --- a/drivers/net/ethernet/netronome/nfp/bpf/fw.h +++ b/drivers/net/ethernet/netronome/nfp/bpf/fw.h @@ -6,6 +6,7 @@ #include #include +#include "../ccm.h" /* Kernel's enum bpf_reg_type is not uABI so people may change it breaking * our FW ABI. In that case we will do translation in the driver. @@ -52,22 +53,6 @@ struct nfp_bpf_cap_tlv_maps { /* * Types defined for map related control messages */ -#define CMSG_MAP_ABI_VERSION 1 - -enum nfp_bpf_cmsg_type { - CMSG_TYPE_MAP_ALLOC = 1, - CMSG_TYPE_MAP_FREE = 2, - CMSG_TYPE_MAP_LOOKUP = 3, - CMSG_TYPE_MAP_UPDATE = 4, - CMSG_TYPE_MAP_DELETE = 5, - CMSG_TYPE_MAP_GETNEXT = 6, - CMSG_TYPE_MAP_GETFIRST = 7, - CMSG_TYPE_BPF_EVENT = 8, - __CMSG_TYPE_MAP_MAX, -}; - -#define CMSG_TYPE_MAP_REPLY_BIT 7 -#define __CMSG_REPLY(req) (BIT(CMSG_TYPE_MAP_REPLY_BIT) | (req)) /* BPF ABIv2 fixed-length control message fields */ #define CMSG_MAP_KEY_LW 16 @@ -84,19 +69,13 @@ enum nfp_bpf_cmsg_status { CMSG_RC_ERR_MAP_E2BIG = 7, }; -struct cmsg_hdr { - u8 type; - u8 ver; - __be16 tag; -}; - struct cmsg_reply_map_simple { - struct cmsg_hdr hdr; + struct nfp_ccm_hdr hdr; __be32 rc; }; struct cmsg_req_map_alloc_tbl { - struct cmsg_hdr hdr; + struct nfp_ccm_hdr hdr; __be32 key_size; /* in bytes */ __be32 value_size; /* in bytes */ __be32 max_entries; @@ -110,7 +89,7 @@ struct cmsg_reply_map_alloc_tbl { }; struct cmsg_req_map_free_tbl { - struct cmsg_hdr hdr; + struct nfp_ccm_hdr hdr; __be32 tid; }; @@ -120,7 +99,7 @@ struct cmsg_reply_map_free_tbl { }; struct cmsg_req_map_op { - struct cmsg_hdr hdr; + struct nfp_ccm_hdr hdr; __be32 tid; __be32 count; __be32 flags; @@ -135,7 +114,7 @@ struct cmsg_reply_map_op { }; struct cmsg_bpf_event { - struct cmsg_hdr hdr; + struct nfp_ccm_hdr hdr; __be32 cpu_id; __be64 map_ptr; __be32 data_size; diff --git a/drivers/net/ethernet/netronome/nfp/bpf/main.c b/drivers/net/ethernet/netronome/nfp/bpf/main.c index 275de9f4c61c635c69c5a6ac657e22511762998a..9c136da252214cd8c4fabd90e90e0b5e4e3c678e 100644 --- a/drivers/net/ethernet/netronome/nfp/bpf/main.c +++ b/drivers/net/ethernet/netronome/nfp/bpf/main.c @@ -442,14 +442,16 @@ static int nfp_bpf_init(struct nfp_app *app) bpf->app = app; app->priv = bpf; - skb_queue_head_init(&bpf->cmsg_replies); - init_waitqueue_head(&bpf->cmsg_wq); INIT_LIST_HEAD(&bpf->map_list); - err = rhashtable_init(&bpf->maps_neutral, &nfp_bpf_maps_neutral_params); + err = nfp_ccm_init(&bpf->ccm, app); if (err) goto err_free_bpf; + err = rhashtable_init(&bpf->maps_neutral, &nfp_bpf_maps_neutral_params); + if (err) + goto err_clean_ccm; + nfp_bpf_init_capabilities(bpf); err = nfp_bpf_parse_capabilities(app); @@ -474,6 +476,8 @@ static int nfp_bpf_init(struct nfp_app *app) err_free_neutral_maps: rhashtable_destroy(&bpf->maps_neutral); +err_clean_ccm: + nfp_ccm_clean(&bpf->ccm); err_free_bpf: kfree(bpf); return err; @@ -484,7 +488,7 @@ static void nfp_bpf_clean(struct nfp_app *app) struct nfp_app_bpf *bpf = app->priv; bpf_offload_dev_destroy(bpf->bpf_dev); - WARN_ON(!skb_queue_empty(&bpf->cmsg_replies)); + nfp_ccm_clean(&bpf->ccm); WARN_ON(!list_empty(&bpf->map_list)); WARN_ON(bpf->maps_in_use || bpf->map_elems_in_use); rhashtable_free_and_destroy(&bpf->maps_neutral, diff --git a/drivers/net/ethernet/netronome/nfp/bpf/main.h b/drivers/net/ethernet/netronome/nfp/bpf/main.h index b25a48218bcf7937320ee8e9fa174aaedbea744d..e54d1ac84df278737b39a5ee40bded7e1b43111e 100644 --- a/drivers/net/ethernet/netronome/nfp/bpf/main.h +++ b/drivers/net/ethernet/netronome/nfp/bpf/main.h @@ -14,6 +14,7 @@ #include #include +#include "../ccm.h" #include "../nfp_asm.h" #include "fw.h" @@ -84,16 +85,10 @@ enum pkt_vec { /** * struct nfp_app_bpf - bpf app priv structure * @app: backpointer to the app + * @ccm: common control message handler data * * @bpf_dev: BPF offload device handle * - * @tag_allocator: bitmap of control message tags in use - * @tag_alloc_next: next tag bit to allocate - * @tag_alloc_last: next tag bit to be freed - * - * @cmsg_replies: received cmsg replies waiting to be consumed - * @cmsg_wq: work queue for waiting for cmsg replies - * * @cmsg_key_sz: size of key in cmsg element array * @cmsg_val_sz: size of value in cmsg element array * @@ -132,16 +127,10 @@ enum pkt_vec { */ struct nfp_app_bpf { struct nfp_app *app; + struct nfp_ccm ccm; struct bpf_offload_dev *bpf_dev; - DECLARE_BITMAP(tag_allocator, U16_MAX + 1); - u16 tag_alloc_next; - u16 tag_alloc_last; - - struct sk_buff_head cmsg_replies; - struct wait_queue_head cmsg_wq; - unsigned int cmsg_key_sz; unsigned int cmsg_val_sz; diff --git a/drivers/net/ethernet/netronome/nfp/bpf/offload.c b/drivers/net/ethernet/netronome/nfp/bpf/offload.c index 15dce97650a5af251d0be613e66735221270a195..39c9fec222b45823eca3772a77b239938188fcd2 100644 --- a/drivers/net/ethernet/netronome/nfp/bpf/offload.c +++ b/drivers/net/ethernet/netronome/nfp/bpf/offload.c @@ -22,6 +22,7 @@ #include #include "main.h" +#include "../ccm.h" #include "../nfp_app.h" #include "../nfp_net_ctrl.h" #include "../nfp_net.h" @@ -452,7 +453,7 @@ int nfp_bpf_event_output(struct nfp_app_bpf *bpf, const void *data, if (len < sizeof(struct cmsg_bpf_event) + pkt_size + data_size) return -EINVAL; - if (cbe->hdr.ver != CMSG_MAP_ABI_VERSION) + if (cbe->hdr.ver != NFP_CCM_ABI_VERSION) return -EINVAL; rcu_read_lock(); diff --git a/drivers/net/ethernet/netronome/nfp/ccm.c b/drivers/net/ethernet/netronome/nfp/ccm.c new file mode 100644 index 0000000000000000000000000000000000000000..94476e41e261ab86889c9d7a54b3e3be8ee0b0e9 --- /dev/null +++ b/drivers/net/ethernet/netronome/nfp/ccm.c @@ -0,0 +1,220 @@ +// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +/* Copyright (C) 2016-2019 Netronome Systems, Inc. */ + +#include + +#include "ccm.h" +#include "nfp_app.h" +#include "nfp_net.h" + +#define NFP_CCM_TYPE_REPLY_BIT 7 +#define __NFP_CCM_REPLY(req) (BIT(NFP_CCM_TYPE_REPLY_BIT) | (req)) + +#define ccm_warn(app, msg...) nn_dp_warn(&(app)->ctrl->dp, msg) + +#define NFP_CCM_TAG_ALLOC_SPAN (U16_MAX / 4) + +static bool nfp_ccm_all_tags_busy(struct nfp_ccm *ccm) +{ + u16 used_tags; + + used_tags = ccm->tag_alloc_next - ccm->tag_alloc_last; + + return used_tags > NFP_CCM_TAG_ALLOC_SPAN; +} + +static int nfp_ccm_alloc_tag(struct nfp_ccm *ccm) +{ + /* CCM is for FW communication which is request-reply. To make sure + * we don't reuse the message ID too early after timeout - limit the + * number of requests in flight. + */ + if (unlikely(nfp_ccm_all_tags_busy(ccm))) { + ccm_warn(ccm->app, "all FW request contexts busy!\n"); + return -EAGAIN; + } + + WARN_ON(__test_and_set_bit(ccm->tag_alloc_next, ccm->tag_allocator)); + return ccm->tag_alloc_next++; +} + +static void nfp_ccm_free_tag(struct nfp_ccm *ccm, u16 tag) +{ + WARN_ON(!__test_and_clear_bit(tag, ccm->tag_allocator)); + + while (!test_bit(ccm->tag_alloc_last, ccm->tag_allocator) && + ccm->tag_alloc_last != ccm->tag_alloc_next) + ccm->tag_alloc_last++; +} + +static struct sk_buff *__nfp_ccm_reply(struct nfp_ccm *ccm, u16 tag) +{ + unsigned int msg_tag; + struct sk_buff *skb; + + skb_queue_walk(&ccm->replies, skb) { + msg_tag = nfp_ccm_get_tag(skb); + if (msg_tag == tag) { + nfp_ccm_free_tag(ccm, tag); + __skb_unlink(skb, &ccm->replies); + return skb; + } + } + + return NULL; +} + +static struct sk_buff * +nfp_ccm_reply(struct nfp_ccm *ccm, struct nfp_app *app, u16 tag) +{ + struct sk_buff *skb; + + nfp_ctrl_lock(app->ctrl); + skb = __nfp_ccm_reply(ccm, tag); + nfp_ctrl_unlock(app->ctrl); + + return skb; +} + +static struct sk_buff * +nfp_ccm_reply_drop_tag(struct nfp_ccm *ccm, struct nfp_app *app, u16 tag) +{ + struct sk_buff *skb; + + nfp_ctrl_lock(app->ctrl); + skb = __nfp_ccm_reply(ccm, tag); + if (!skb) + nfp_ccm_free_tag(ccm, tag); + nfp_ctrl_unlock(app->ctrl); + + return skb; +} + +static struct sk_buff * +nfp_ccm_wait_reply(struct nfp_ccm *ccm, struct nfp_app *app, + enum nfp_ccm_type type, int tag) +{ + struct sk_buff *skb; + int i, err; + + for (i = 0; i < 50; i++) { + udelay(4); + skb = nfp_ccm_reply(ccm, app, tag); + if (skb) + return skb; + } + + err = wait_event_interruptible_timeout(ccm->wq, + skb = nfp_ccm_reply(ccm, app, + tag), + msecs_to_jiffies(5000)); + /* We didn't get a response - try last time and atomically drop + * the tag even if no response is matched. + */ + if (!skb) + skb = nfp_ccm_reply_drop_tag(ccm, app, tag); + if (err < 0) { + ccm_warn(app, "%s waiting for response to 0x%02x: %d\n", + err == ERESTARTSYS ? "interrupted" : "error", + type, err); + return ERR_PTR(err); + } + if (!skb) { + ccm_warn(app, "timeout waiting for response to 0x%02x\n", type); + return ERR_PTR(-ETIMEDOUT); + } + + return skb; +} + +struct sk_buff * +nfp_ccm_communicate(struct nfp_ccm *ccm, struct sk_buff *skb, + enum nfp_ccm_type type, unsigned int reply_size) +{ + struct nfp_app *app = ccm->app; + struct nfp_ccm_hdr *hdr; + int reply_type, tag; + + nfp_ctrl_lock(app->ctrl); + tag = nfp_ccm_alloc_tag(ccm); + if (tag < 0) { + nfp_ctrl_unlock(app->ctrl); + dev_kfree_skb_any(skb); + return ERR_PTR(tag); + } + + hdr = (void *)skb->data; + hdr->ver = NFP_CCM_ABI_VERSION; + hdr->type = type; + hdr->tag = cpu_to_be16(tag); + + __nfp_app_ctrl_tx(app, skb); + + nfp_ctrl_unlock(app->ctrl); + + skb = nfp_ccm_wait_reply(ccm, app, type, tag); + if (IS_ERR(skb)) + return skb; + + reply_type = nfp_ccm_get_type(skb); + if (reply_type != __NFP_CCM_REPLY(type)) { + ccm_warn(app, "cmsg drop - wrong type 0x%02x != 0x%02lx!\n", + reply_type, __NFP_CCM_REPLY(type)); + goto err_free; + } + /* 0 reply_size means caller will do the validation */ + if (reply_size && skb->len != reply_size) { + ccm_warn(app, "cmsg drop - type 0x%02x wrong size %d != %d!\n", + type, skb->len, reply_size); + goto err_free; + } + + return skb; +err_free: + dev_kfree_skb_any(skb); + return ERR_PTR(-EIO); +} + +void nfp_ccm_rx(struct nfp_ccm *ccm, struct sk_buff *skb) +{ + struct nfp_app *app = ccm->app; + unsigned int tag; + + if (unlikely(skb->len < sizeof(struct nfp_ccm_hdr))) { + ccm_warn(app, "cmsg drop - too short %d!\n", skb->len); + goto err_free; + } + + nfp_ctrl_lock(app->ctrl); + + tag = nfp_ccm_get_tag(skb); + if (unlikely(!test_bit(tag, ccm->tag_allocator))) { + ccm_warn(app, "cmsg drop - no one is waiting for tag %u!\n", + tag); + goto err_unlock; + } + + __skb_queue_tail(&ccm->replies, skb); + wake_up_interruptible_all(&ccm->wq); + + nfp_ctrl_unlock(app->ctrl); + return; + +err_unlock: + nfp_ctrl_unlock(app->ctrl); +err_free: + dev_kfree_skb_any(skb); +} + +int nfp_ccm_init(struct nfp_ccm *ccm, struct nfp_app *app) +{ + ccm->app = app; + skb_queue_head_init(&ccm->replies); + init_waitqueue_head(&ccm->wq); + return 0; +} + +void nfp_ccm_clean(struct nfp_ccm *ccm) +{ + WARN_ON(!skb_queue_empty(&ccm->replies)); +} diff --git a/drivers/net/ethernet/netronome/nfp/ccm.h b/drivers/net/ethernet/netronome/nfp/ccm.h new file mode 100644 index 0000000000000000000000000000000000000000..e2fe4b867958802f05edfc71304cbeed9363ae17 --- /dev/null +++ b/drivers/net/ethernet/netronome/nfp/ccm.h @@ -0,0 +1,81 @@ +/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) */ +/* Copyright (C) 2016-2019 Netronome Systems, Inc. */ + +#ifndef NFP_CCM_H +#define NFP_CCM_H 1 + +#include +#include +#include + +struct nfp_app; + +/* Firmware ABI */ + +enum nfp_ccm_type { + NFP_CCM_TYPE_BPF_MAP_ALLOC = 1, + NFP_CCM_TYPE_BPF_MAP_FREE = 2, + NFP_CCM_TYPE_BPF_MAP_LOOKUP = 3, + NFP_CCM_TYPE_BPF_MAP_UPDATE = 4, + NFP_CCM_TYPE_BPF_MAP_DELETE = 5, + NFP_CCM_TYPE_BPF_MAP_GETNEXT = 6, + NFP_CCM_TYPE_BPF_MAP_GETFIRST = 7, + NFP_CCM_TYPE_BPF_BPF_EVENT = 8, + __NFP_CCM_TYPE_MAX, +}; + +#define NFP_CCM_ABI_VERSION 1 + +struct nfp_ccm_hdr { + u8 type; + u8 ver; + __be16 tag; +}; + +static inline u8 nfp_ccm_get_type(struct sk_buff *skb) +{ + struct nfp_ccm_hdr *hdr; + + hdr = (struct nfp_ccm_hdr *)skb->data; + + return hdr->type; +} + +static inline unsigned int nfp_ccm_get_tag(struct sk_buff *skb) +{ + struct nfp_ccm_hdr *hdr; + + hdr = (struct nfp_ccm_hdr *)skb->data; + + return be16_to_cpu(hdr->tag); +} + +/* Implementation */ + +/** + * struct nfp_ccm - common control message handling + * @tag_allocator: bitmap of control message tags in use + * @tag_alloc_next: next tag bit to allocate + * @tag_alloc_last: next tag bit to be freed + * + * @replies: received cmsg replies waiting to be consumed + * @wq: work queue for waiting for cmsg replies + */ +struct nfp_ccm { + struct nfp_app *app; + + DECLARE_BITMAP(tag_allocator, U16_MAX + 1); + u16 tag_alloc_next; + u16 tag_alloc_last; + + struct sk_buff_head replies; + struct wait_queue_head wq; +}; + +int nfp_ccm_init(struct nfp_ccm *ccm, struct nfp_app *app); +void nfp_ccm_clean(struct nfp_ccm *ccm); +void nfp_ccm_rx(struct nfp_ccm *ccm, struct sk_buff *skb); +struct sk_buff * +nfp_ccm_communicate(struct nfp_ccm *ccm, struct sk_buff *skb, + enum nfp_ccm_type type, unsigned int reply_size); +#endif diff --git a/drivers/net/ethernet/netronome/nfp/flower/action.c b/drivers/net/ethernet/netronome/nfp/flower/action.c index e336f6ee94f5c6d3a9108b85a6fb0529ecb7031d..c56e31d9f8a4f9836972996685e071a5f24962d4 100644 --- a/drivers/net/ethernet/netronome/nfp/flower/action.c +++ b/drivers/net/ethernet/netronome/nfp/flower/action.c @@ -160,9 +160,9 @@ nfp_fl_get_tun_from_act_l4_port(struct nfp_app *app, struct nfp_flower_priv *priv = app->priv; switch (tun->key.tp_dst) { - case htons(NFP_FL_VXLAN_PORT): + case htons(IANA_VXLAN_UDP_PORT): return NFP_FL_TUNNEL_VXLAN; - case htons(NFP_FL_GENEVE_PORT): + case htons(GENEVE_UDP_PORT): if (priv->flower_ext_feats & NFP_FL_FEATS_GENEVE) return NFP_FL_TUNNEL_GENEVE; /* FALLTHROUGH */ @@ -582,60 +582,23 @@ static u32 nfp_fl_csum_l4_to_flag(u8 ip_proto) } } -static int -nfp_fl_pedit(const struct flow_action_entry *act, - struct tc_cls_flower_offload *flow, - char *nfp_action, int *a_len, u32 *csum_updated) -{ - struct flow_rule *rule = tc_cls_flower_offload_flow_rule(flow); +struct nfp_flower_pedit_acts { struct nfp_fl_set_ipv6_addr set_ip6_dst, set_ip6_src; struct nfp_fl_set_ipv6_tc_hl_fl set_ip6_tc_hl_fl; struct nfp_fl_set_ip4_ttl_tos set_ip_ttl_tos; struct nfp_fl_set_ip4_addrs set_ip_addr; - enum flow_action_mangle_base htype; struct nfp_fl_set_tport set_tport; struct nfp_fl_set_eth set_eth; +}; + +static int +nfp_fl_commit_mangle(struct tc_cls_flower_offload *flow, char *nfp_action, + int *a_len, struct nfp_flower_pedit_acts *set_act, + u32 *csum_updated) +{ + struct flow_rule *rule = tc_cls_flower_offload_flow_rule(flow); size_t act_size = 0; u8 ip_proto = 0; - u32 offset; - int err; - - memset(&set_ip6_tc_hl_fl, 0, sizeof(set_ip6_tc_hl_fl)); - memset(&set_ip_ttl_tos, 0, sizeof(set_ip_ttl_tos)); - memset(&set_ip6_dst, 0, sizeof(set_ip6_dst)); - memset(&set_ip6_src, 0, sizeof(set_ip6_src)); - memset(&set_ip_addr, 0, sizeof(set_ip_addr)); - memset(&set_tport, 0, sizeof(set_tport)); - memset(&set_eth, 0, sizeof(set_eth)); - - htype = act->mangle.htype; - offset = act->mangle.offset; - - switch (htype) { - case TCA_PEDIT_KEY_EX_HDR_TYPE_ETH: - err = nfp_fl_set_eth(act, offset, &set_eth); - break; - case TCA_PEDIT_KEY_EX_HDR_TYPE_IP4: - err = nfp_fl_set_ip4(act, offset, &set_ip_addr, - &set_ip_ttl_tos); - break; - case TCA_PEDIT_KEY_EX_HDR_TYPE_IP6: - err = nfp_fl_set_ip6(act, offset, &set_ip6_dst, - &set_ip6_src, &set_ip6_tc_hl_fl); - break; - case TCA_PEDIT_KEY_EX_HDR_TYPE_TCP: - err = nfp_fl_set_tport(act, offset, &set_tport, - NFP_FL_ACTION_OPCODE_SET_TCP); - break; - case TCA_PEDIT_KEY_EX_HDR_TYPE_UDP: - err = nfp_fl_set_tport(act, offset, &set_tport, - NFP_FL_ACTION_OPCODE_SET_UDP); - break; - default: - return -EOPNOTSUPP; - } - if (err) - return err; if (flow_rule_match_key(rule, FLOW_DISSECTOR_KEY_BASIC)) { struct flow_match_basic match; @@ -644,77 +607,82 @@ nfp_fl_pedit(const struct flow_action_entry *act, ip_proto = match.key->ip_proto; } - if (set_eth.head.len_lw) { - act_size = sizeof(set_eth); - memcpy(nfp_action, &set_eth, act_size); + if (set_act->set_eth.head.len_lw) { + act_size = sizeof(set_act->set_eth); + memcpy(nfp_action, &set_act->set_eth, act_size); *a_len += act_size; } - if (set_ip_ttl_tos.head.len_lw) { + + if (set_act->set_ip_ttl_tos.head.len_lw) { nfp_action += act_size; - act_size = sizeof(set_ip_ttl_tos); - memcpy(nfp_action, &set_ip_ttl_tos, act_size); + act_size = sizeof(set_act->set_ip_ttl_tos); + memcpy(nfp_action, &set_act->set_ip_ttl_tos, act_size); *a_len += act_size; /* Hardware will automatically fix IPv4 and TCP/UDP checksum. */ *csum_updated |= TCA_CSUM_UPDATE_FLAG_IPV4HDR | nfp_fl_csum_l4_to_flag(ip_proto); } - if (set_ip_addr.head.len_lw) { + + if (set_act->set_ip_addr.head.len_lw) { nfp_action += act_size; - act_size = sizeof(set_ip_addr); - memcpy(nfp_action, &set_ip_addr, act_size); + act_size = sizeof(set_act->set_ip_addr); + memcpy(nfp_action, &set_act->set_ip_addr, act_size); *a_len += act_size; /* Hardware will automatically fix IPv4 and TCP/UDP checksum. */ *csum_updated |= TCA_CSUM_UPDATE_FLAG_IPV4HDR | nfp_fl_csum_l4_to_flag(ip_proto); } - if (set_ip6_tc_hl_fl.head.len_lw) { + + if (set_act->set_ip6_tc_hl_fl.head.len_lw) { nfp_action += act_size; - act_size = sizeof(set_ip6_tc_hl_fl); - memcpy(nfp_action, &set_ip6_tc_hl_fl, act_size); + act_size = sizeof(set_act->set_ip6_tc_hl_fl); + memcpy(nfp_action, &set_act->set_ip6_tc_hl_fl, act_size); *a_len += act_size; /* Hardware will automatically fix TCP/UDP checksum. */ *csum_updated |= nfp_fl_csum_l4_to_flag(ip_proto); } - if (set_ip6_dst.head.len_lw && set_ip6_src.head.len_lw) { + + if (set_act->set_ip6_dst.head.len_lw && + set_act->set_ip6_src.head.len_lw) { /* TC compiles set src and dst IPv6 address as a single action, * the hardware requires this to be 2 separate actions. */ nfp_action += act_size; - act_size = sizeof(set_ip6_src); - memcpy(nfp_action, &set_ip6_src, act_size); + act_size = sizeof(set_act->set_ip6_src); + memcpy(nfp_action, &set_act->set_ip6_src, act_size); *a_len += act_size; - act_size = sizeof(set_ip6_dst); - memcpy(&nfp_action[sizeof(set_ip6_src)], &set_ip6_dst, - act_size); + act_size = sizeof(set_act->set_ip6_dst); + memcpy(&nfp_action[sizeof(set_act->set_ip6_src)], + &set_act->set_ip6_dst, act_size); *a_len += act_size; /* Hardware will automatically fix TCP/UDP checksum. */ *csum_updated |= nfp_fl_csum_l4_to_flag(ip_proto); - } else if (set_ip6_dst.head.len_lw) { + } else if (set_act->set_ip6_dst.head.len_lw) { nfp_action += act_size; - act_size = sizeof(set_ip6_dst); - memcpy(nfp_action, &set_ip6_dst, act_size); + act_size = sizeof(set_act->set_ip6_dst); + memcpy(nfp_action, &set_act->set_ip6_dst, act_size); *a_len += act_size; /* Hardware will automatically fix TCP/UDP checksum. */ *csum_updated |= nfp_fl_csum_l4_to_flag(ip_proto); - } else if (set_ip6_src.head.len_lw) { + } else if (set_act->set_ip6_src.head.len_lw) { nfp_action += act_size; - act_size = sizeof(set_ip6_src); - memcpy(nfp_action, &set_ip6_src, act_size); + act_size = sizeof(set_act->set_ip6_src); + memcpy(nfp_action, &set_act->set_ip6_src, act_size); *a_len += act_size; /* Hardware will automatically fix TCP/UDP checksum. */ *csum_updated |= nfp_fl_csum_l4_to_flag(ip_proto); } - if (set_tport.head.len_lw) { + if (set_act->set_tport.head.len_lw) { nfp_action += act_size; - act_size = sizeof(set_tport); - memcpy(nfp_action, &set_tport, act_size); + act_size = sizeof(set_act->set_tport); + memcpy(nfp_action, &set_act->set_tport, act_size); *a_len += act_size; /* Hardware will automatically fix TCP/UDP checksum. */ @@ -725,7 +693,40 @@ nfp_fl_pedit(const struct flow_action_entry *act, } static int -nfp_flower_output_action(struct nfp_app *app, const struct flow_action_entry *act, +nfp_fl_pedit(const struct flow_action_entry *act, + struct tc_cls_flower_offload *flow, char *nfp_action, int *a_len, + u32 *csum_updated, struct nfp_flower_pedit_acts *set_act) +{ + enum flow_action_mangle_base htype; + u32 offset; + + htype = act->mangle.htype; + offset = act->mangle.offset; + + switch (htype) { + case TCA_PEDIT_KEY_EX_HDR_TYPE_ETH: + return nfp_fl_set_eth(act, offset, &set_act->set_eth); + case TCA_PEDIT_KEY_EX_HDR_TYPE_IP4: + return nfp_fl_set_ip4(act, offset, &set_act->set_ip_addr, + &set_act->set_ip_ttl_tos); + case TCA_PEDIT_KEY_EX_HDR_TYPE_IP6: + return nfp_fl_set_ip6(act, offset, &set_act->set_ip6_dst, + &set_act->set_ip6_src, + &set_act->set_ip6_tc_hl_fl); + case TCA_PEDIT_KEY_EX_HDR_TYPE_TCP: + return nfp_fl_set_tport(act, offset, &set_act->set_tport, + NFP_FL_ACTION_OPCODE_SET_TCP); + case TCA_PEDIT_KEY_EX_HDR_TYPE_UDP: + return nfp_fl_set_tport(act, offset, &set_act->set_tport, + NFP_FL_ACTION_OPCODE_SET_UDP); + default: + return -EOPNOTSUPP; + } +} + +static int +nfp_flower_output_action(struct nfp_app *app, + const struct flow_action_entry *act, struct nfp_fl_payload *nfp_fl, int *a_len, struct net_device *netdev, bool last, enum nfp_flower_tun_type *tun_type, int *tun_out_cnt, @@ -775,7 +776,8 @@ nfp_flower_loop_action(struct nfp_app *app, const struct flow_action_entry *act, struct nfp_fl_payload *nfp_fl, int *a_len, struct net_device *netdev, enum nfp_flower_tun_type *tun_type, int *tun_out_cnt, - int *out_cnt, u32 *csum_updated) + int *out_cnt, u32 *csum_updated, + struct nfp_flower_pedit_acts *set_act) { struct nfp_fl_set_ipv4_udp_tun *set_tun; struct nfp_fl_pre_tunnel *pre_tun; @@ -860,7 +862,7 @@ nfp_flower_loop_action(struct nfp_app *app, const struct flow_action_entry *act, return 0; case FLOW_ACTION_MANGLE: if (nfp_fl_pedit(act, flow, &nfp_fl->action_data[*a_len], - a_len, csum_updated)) + a_len, csum_updated, set_act)) return -EOPNOTSUPP; break; case FLOW_ACTION_CSUM: @@ -880,12 +882,49 @@ nfp_flower_loop_action(struct nfp_app *app, const struct flow_action_entry *act, return 0; } +static bool nfp_fl_check_mangle_start(struct flow_action *flow_act, + int current_act_idx) +{ + struct flow_action_entry current_act; + struct flow_action_entry prev_act; + + current_act = flow_act->entries[current_act_idx]; + if (current_act.id != FLOW_ACTION_MANGLE) + return false; + + if (current_act_idx == 0) + return true; + + prev_act = flow_act->entries[current_act_idx - 1]; + + return prev_act.id != FLOW_ACTION_MANGLE; +} + +static bool nfp_fl_check_mangle_end(struct flow_action *flow_act, + int current_act_idx) +{ + struct flow_action_entry current_act; + struct flow_action_entry next_act; + + current_act = flow_act->entries[current_act_idx]; + if (current_act.id != FLOW_ACTION_MANGLE) + return false; + + if (current_act_idx == flow_act->num_entries) + return true; + + next_act = flow_act->entries[current_act_idx + 1]; + + return next_act.id != FLOW_ACTION_MANGLE; +} + int nfp_flower_compile_action(struct nfp_app *app, struct tc_cls_flower_offload *flow, struct net_device *netdev, struct nfp_fl_payload *nfp_flow) { int act_len, act_cnt, err, tun_out_cnt, out_cnt, i; + struct nfp_flower_pedit_acts set_act; enum nfp_flower_tun_type tun_type; struct flow_action_entry *act; u32 csum_updated = 0; @@ -899,12 +938,18 @@ int nfp_flower_compile_action(struct nfp_app *app, out_cnt = 0; flow_action_for_each(i, act, &flow->rule->action) { + if (nfp_fl_check_mangle_start(&flow->rule->action, i)) + memset(&set_act, 0, sizeof(set_act)); err = nfp_flower_loop_action(app, act, flow, nfp_flow, &act_len, netdev, &tun_type, &tun_out_cnt, - &out_cnt, &csum_updated); + &out_cnt, &csum_updated, &set_act); if (err) return err; act_cnt++; + if (nfp_fl_check_mangle_end(&flow->rule->action, i)) + nfp_fl_commit_mangle(flow, + &nfp_flow->action_data[act_len], + &act_len, &set_act, &csum_updated); } /* We optimise when the action list is small, this can unfortunately diff --git a/drivers/net/ethernet/netronome/nfp/flower/cmsg.c b/drivers/net/ethernet/netronome/nfp/flower/cmsg.c index cf9e1118ee8fff37301de7ef4160ab3b99cc2f17..d5bbe3d6048bf3cec01fc5aadce5c75cc0d5f066 100644 --- a/drivers/net/ethernet/netronome/nfp/flower/cmsg.c +++ b/drivers/net/ethernet/netronome/nfp/flower/cmsg.c @@ -159,7 +159,7 @@ nfp_flower_cmsg_portmod_rx(struct nfp_app *app, struct sk_buff *skb) rtnl_lock(); rcu_read_lock(); - netdev = nfp_app_repr_get(app, be32_to_cpu(msg->portnum)); + netdev = nfp_app_dev_get(app, be32_to_cpu(msg->portnum), NULL); rcu_read_unlock(); if (!netdev) { nfp_flower_cmsg_warn(app, "ctrl msg for unknown port 0x%08x\n", @@ -192,7 +192,7 @@ nfp_flower_cmsg_portreify_rx(struct nfp_app *app, struct sk_buff *skb) msg = nfp_flower_cmsg_get_data(skb); rcu_read_lock(); - exists = !!nfp_app_repr_get(app, be32_to_cpu(msg->portnum)); + exists = !!nfp_app_dev_get(app, be32_to_cpu(msg->portnum), NULL); rcu_read_unlock(); if (!exists) { nfp_flower_cmsg_warn(app, "ctrl msg for unknown port 0x%08x\n", @@ -204,6 +204,50 @@ nfp_flower_cmsg_portreify_rx(struct nfp_app *app, struct sk_buff *skb) wake_up(&priv->reify_wait_queue); } +static void +nfp_flower_cmsg_merge_hint_rx(struct nfp_app *app, struct sk_buff *skb) +{ + unsigned int msg_len = nfp_flower_cmsg_get_data_len(skb); + struct nfp_flower_cmsg_merge_hint *msg; + struct nfp_fl_payload *sub_flows[2]; + int err, i, flow_cnt; + + msg = nfp_flower_cmsg_get_data(skb); + /* msg->count starts at 0 and always assumes at least 1 entry. */ + flow_cnt = msg->count + 1; + + if (msg_len < struct_size(msg, flow, flow_cnt)) { + nfp_flower_cmsg_warn(app, "Merge hint ctrl msg too short - %d bytes but expect %zd\n", + msg_len, struct_size(msg, flow, flow_cnt)); + return; + } + + if (flow_cnt != 2) { + nfp_flower_cmsg_warn(app, "Merge hint contains %d flows - two are expected\n", + flow_cnt); + return; + } + + rtnl_lock(); + for (i = 0; i < flow_cnt; i++) { + u32 ctx = be32_to_cpu(msg->flow[i].host_ctx); + + sub_flows[i] = nfp_flower_get_fl_payload_from_ctx(app, ctx); + if (!sub_flows[i]) { + nfp_flower_cmsg_warn(app, "Invalid flow in merge hint\n"); + goto err_rtnl_unlock; + } + } + + err = nfp_flower_merge_offloaded_flows(app, sub_flows[0], sub_flows[1]); + /* Only warn on memory fail. Hint veto will not break functionality. */ + if (err == -ENOMEM) + nfp_flower_cmsg_warn(app, "Flow merge memory fail.\n"); + +err_rtnl_unlock: + rtnl_unlock(); +} + static void nfp_flower_cmsg_process_one_rx(struct nfp_app *app, struct sk_buff *skb) { @@ -222,12 +266,21 @@ nfp_flower_cmsg_process_one_rx(struct nfp_app *app, struct sk_buff *skb) case NFP_FLOWER_CMSG_TYPE_PORT_MOD: nfp_flower_cmsg_portmod_rx(app, skb); break; + case NFP_FLOWER_CMSG_TYPE_MERGE_HINT: + if (app_priv->flower_ext_feats & NFP_FL_FEATS_FLOW_MERGE) { + nfp_flower_cmsg_merge_hint_rx(app, skb); + break; + } + goto err_default; case NFP_FLOWER_CMSG_TYPE_NO_NEIGH: nfp_tunnel_request_route(app, skb); break; case NFP_FLOWER_CMSG_TYPE_ACTIVE_TUNS: nfp_tunnel_keep_alive(app, skb); break; + case NFP_FLOWER_CMSG_TYPE_QOS_STATS: + nfp_flower_stats_rlim_reply(app, skb); + break; case NFP_FLOWER_CMSG_TYPE_LAG_CONFIG: if (app_priv->flower_ext_feats & NFP_FL_FEATS_LAG) { skb_stored = nfp_flower_lag_unprocessed_msg(app, skb); @@ -235,6 +288,7 @@ nfp_flower_cmsg_process_one_rx(struct nfp_app *app, struct sk_buff *skb) } /* fall through */ default: +err_default: nfp_flower_cmsg_warn(app, "Cannot handle invalid repr control type %u\n", type); goto out; diff --git a/drivers/net/ethernet/netronome/nfp/flower/cmsg.h b/drivers/net/ethernet/netronome/nfp/flower/cmsg.h index 0ed51e79db00ebb90f0f9e8b2d9e74631c7ed4dc..537f7fc19584472a91e1f3ba86b73f0cfce5c605 100644 --- a/drivers/net/ethernet/netronome/nfp/flower/cmsg.h +++ b/drivers/net/ethernet/netronome/nfp/flower/cmsg.h @@ -402,11 +402,13 @@ struct nfp_flower_cmsg_hdr { /* Types defined for port related control messages */ enum nfp_flower_cmsg_type_port { NFP_FLOWER_CMSG_TYPE_FLOW_ADD = 0, + NFP_FLOWER_CMSG_TYPE_FLOW_MOD = 1, NFP_FLOWER_CMSG_TYPE_FLOW_DEL = 2, NFP_FLOWER_CMSG_TYPE_LAG_CONFIG = 4, NFP_FLOWER_CMSG_TYPE_PORT_REIFY = 6, NFP_FLOWER_CMSG_TYPE_MAC_REPR = 7, NFP_FLOWER_CMSG_TYPE_PORT_MOD = 8, + NFP_FLOWER_CMSG_TYPE_MERGE_HINT = 9, NFP_FLOWER_CMSG_TYPE_NO_NEIGH = 10, NFP_FLOWER_CMSG_TYPE_TUN_MAC = 11, NFP_FLOWER_CMSG_TYPE_ACTIVE_TUNS = 12, @@ -414,6 +416,9 @@ enum nfp_flower_cmsg_type_port { NFP_FLOWER_CMSG_TYPE_TUN_IPS = 14, NFP_FLOWER_CMSG_TYPE_FLOW_STATS = 15, NFP_FLOWER_CMSG_TYPE_PORT_ECHO = 16, + NFP_FLOWER_CMSG_TYPE_QOS_MOD = 18, + NFP_FLOWER_CMSG_TYPE_QOS_DEL = 19, + NFP_FLOWER_CMSG_TYPE_QOS_STATS = 20, NFP_FLOWER_CMSG_TYPE_MAX = 32, }; @@ -451,6 +456,16 @@ struct nfp_flower_cmsg_portreify { #define NFP_FLOWER_CMSG_PORTREIFY_INFO_EXIST BIT(0) +/* NFP_FLOWER_CMSG_TYPE_FLOW_MERGE_HINT */ +struct nfp_flower_cmsg_merge_hint { + u8 reserved[3]; + u8 count; + struct { + __be32 host_ctx; + __be64 host_cookie; + } __packed flow[0]; +}; + enum nfp_flower_cmsg_port_type { NFP_FLOWER_CMSG_PORT_TYPE_UNSPEC = 0x0, NFP_FLOWER_CMSG_PORT_TYPE_PHYS_PORT = 0x1, @@ -473,6 +488,13 @@ enum nfp_flower_cmsg_port_vnic_type { #define NFP_FLOWER_CMSG_PORT_PCIE_Q GENMASK(5, 0) #define NFP_FLOWER_CMSG_PORT_PHYS_PORT_NUM GENMASK(7, 0) +static inline u32 nfp_flower_internal_port_get_port_id(u8 internal_port) +{ + return FIELD_PREP(NFP_FLOWER_CMSG_PORT_PHYS_PORT_NUM, internal_port) | + FIELD_PREP(NFP_FLOWER_CMSG_PORT_TYPE, + NFP_FLOWER_CMSG_PORT_TYPE_OTHER_PORT); +} + static inline u32 nfp_flower_cmsg_phys_port(u8 phys_port) { return FIELD_PREP(NFP_FLOWER_CMSG_PORT_PHYS_PORT_NUM, phys_port) | diff --git a/drivers/net/ethernet/netronome/nfp/flower/main.c b/drivers/net/ethernet/netronome/nfp/flower/main.c index 408089133599b55d686f8b876f43e2379e71ccd3..eb846133943b22736b6b62871fee10d599569493 100644 --- a/drivers/net/ethernet/netronome/nfp/flower/main.c +++ b/drivers/net/ethernet/netronome/nfp/flower/main.c @@ -22,6 +22,9 @@ #define NFP_FLOWER_ALLOWED_VER 0x0001000000010000UL +#define NFP_MIN_INT_PORT_ID 1 +#define NFP_MAX_INT_PORT_ID 256 + static const char *nfp_flower_extra_cap(struct nfp_app *app, struct nfp_net *nn) { return "FLOWER"; @@ -32,6 +35,113 @@ static enum devlink_eswitch_mode eswitch_mode_get(struct nfp_app *app) return DEVLINK_ESWITCH_MODE_SWITCHDEV; } +static int +nfp_flower_lookup_internal_port_id(struct nfp_flower_priv *priv, + struct net_device *netdev) +{ + struct net_device *entry; + int i, id = 0; + + rcu_read_lock(); + idr_for_each_entry(&priv->internal_ports.port_ids, entry, i) + if (entry == netdev) { + id = i; + break; + } + rcu_read_unlock(); + + return id; +} + +static int +nfp_flower_get_internal_port_id(struct nfp_app *app, struct net_device *netdev) +{ + struct nfp_flower_priv *priv = app->priv; + int id; + + id = nfp_flower_lookup_internal_port_id(priv, netdev); + if (id > 0) + return id; + + idr_preload(GFP_ATOMIC); + spin_lock_bh(&priv->internal_ports.lock); + id = idr_alloc(&priv->internal_ports.port_ids, netdev, + NFP_MIN_INT_PORT_ID, NFP_MAX_INT_PORT_ID, GFP_ATOMIC); + spin_unlock_bh(&priv->internal_ports.lock); + idr_preload_end(); + + return id; +} + +u32 nfp_flower_get_port_id_from_netdev(struct nfp_app *app, + struct net_device *netdev) +{ + int ext_port; + + if (nfp_netdev_is_nfp_repr(netdev)) { + return nfp_repr_get_port_id(netdev); + } else if (nfp_flower_internal_port_can_offload(app, netdev)) { + ext_port = nfp_flower_get_internal_port_id(app, netdev); + if (ext_port < 0) + return 0; + + return nfp_flower_internal_port_get_port_id(ext_port); + } + + return 0; +} + +static struct net_device * +nfp_flower_get_netdev_from_internal_port_id(struct nfp_app *app, int port_id) +{ + struct nfp_flower_priv *priv = app->priv; + struct net_device *netdev; + + rcu_read_lock(); + netdev = idr_find(&priv->internal_ports.port_ids, port_id); + rcu_read_unlock(); + + return netdev; +} + +static void +nfp_flower_free_internal_port_id(struct nfp_app *app, struct net_device *netdev) +{ + struct nfp_flower_priv *priv = app->priv; + int id; + + id = nfp_flower_lookup_internal_port_id(priv, netdev); + if (!id) + return; + + spin_lock_bh(&priv->internal_ports.lock); + idr_remove(&priv->internal_ports.port_ids, id); + spin_unlock_bh(&priv->internal_ports.lock); +} + +static int +nfp_flower_internal_port_event_handler(struct nfp_app *app, + struct net_device *netdev, + unsigned long event) +{ + if (event == NETDEV_UNREGISTER && + nfp_flower_internal_port_can_offload(app, netdev)) + nfp_flower_free_internal_port_id(app, netdev); + + return NOTIFY_OK; +} + +static void nfp_flower_internal_port_init(struct nfp_flower_priv *priv) +{ + spin_lock_init(&priv->internal_ports.lock); + idr_init(&priv->internal_ports.port_ids); +} + +static void nfp_flower_internal_port_cleanup(struct nfp_flower_priv *priv) +{ + idr_destroy(&priv->internal_ports.port_ids); +} + static struct nfp_flower_non_repr_priv * nfp_flower_non_repr_priv_lookup(struct nfp_app *app, struct net_device *netdev) { @@ -119,12 +229,21 @@ nfp_flower_repr_get_type_and_port(struct nfp_app *app, u32 port_id, u8 *port) } static struct net_device * -nfp_flower_repr_get(struct nfp_app *app, u32 port_id) +nfp_flower_dev_get(struct nfp_app *app, u32 port_id, bool *redir_egress) { enum nfp_repr_type repr_type; struct nfp_reprs *reprs; u8 port = 0; + /* Check if the port is internal. */ + if (FIELD_GET(NFP_FLOWER_CMSG_PORT_TYPE, port_id) == + NFP_FLOWER_CMSG_PORT_TYPE_OTHER_PORT) { + if (redir_egress) + *redir_egress = true; + port = FIELD_GET(NFP_FLOWER_CMSG_PORT_PHYS_PORT_NUM, port_id); + return nfp_flower_get_netdev_from_internal_port_id(app, port); + } + repr_type = nfp_flower_repr_get_type_and_port(app, port_id, &port); if (repr_type > NFP_REPR_TYPE_MAX) return NULL; @@ -641,11 +760,33 @@ static int nfp_flower_init(struct nfp_app *app) goto err_cleanup_metadata; } + if (app_priv->flower_ext_feats & NFP_FL_FEATS_FLOW_MOD) { + /* Tell the firmware that the driver supports flow merging. */ + err = nfp_rtsym_write_le(app->pf->rtbl, + "_abi_flower_merge_hint_enable", 1); + if (!err) { + app_priv->flower_ext_feats |= NFP_FL_FEATS_FLOW_MERGE; + nfp_flower_internal_port_init(app_priv); + } else if (err == -ENOENT) { + nfp_warn(app->cpp, "Flow merge not supported by FW.\n"); + } else { + goto err_lag_clean; + } + } else { + nfp_warn(app->cpp, "Flow mod/merge not supported by FW.\n"); + } + + if (app_priv->flower_ext_feats & NFP_FL_FEATS_VF_RLIM) + nfp_flower_qos_init(app); + INIT_LIST_HEAD(&app_priv->indr_block_cb_priv); INIT_LIST_HEAD(&app_priv->non_repr_priv); return 0; +err_lag_clean: + if (app_priv->flower_ext_feats & NFP_FL_FEATS_LAG) + nfp_flower_lag_cleanup(&app_priv->nfp_lag); err_cleanup_metadata: nfp_flower_metadata_cleanup(app); err_free_app_priv: @@ -661,9 +802,15 @@ static void nfp_flower_clean(struct nfp_app *app) skb_queue_purge(&app_priv->cmsg_skbs_low); flush_work(&app_priv->cmsg_work); + if (app_priv->flower_ext_feats & NFP_FL_FEATS_VF_RLIM) + nfp_flower_qos_cleanup(app); + if (app_priv->flower_ext_feats & NFP_FL_FEATS_LAG) nfp_flower_lag_cleanup(&app_priv->nfp_lag); + if (app_priv->flower_ext_feats & NFP_FL_FEATS_FLOW_MERGE) + nfp_flower_internal_port_cleanup(app_priv); + nfp_flower_metadata_cleanup(app); vfree(app->priv); app->priv = NULL; @@ -762,6 +909,10 @@ nfp_flower_netdev_event(struct nfp_app *app, struct net_device *netdev, if (ret & NOTIFY_STOP_MASK) return ret; + ret = nfp_flower_internal_port_event_handler(app, netdev, event); + if (ret & NOTIFY_STOP_MASK) + return ret; + return nfp_tunnel_mac_event_handler(app, netdev, event, ptr); } @@ -800,7 +951,7 @@ const struct nfp_app_type app_flower = { .sriov_disable = nfp_flower_sriov_disable, .eswitch_mode_get = eswitch_mode_get, - .repr_get = nfp_flower_repr_get, + .dev_get = nfp_flower_dev_get, .setup_tc = nfp_flower_setup_tc, }; diff --git a/drivers/net/ethernet/netronome/nfp/flower/main.h b/drivers/net/ethernet/netronome/nfp/flower/main.h index c0945a5fd1a47b9edc78f05d869bb8b6057e6709..40957a8dbfe6d5857e53a01d30401ea6bf909f6e 100644 --- a/drivers/net/ethernet/netronome/nfp/flower/main.h +++ b/drivers/net/ethernet/netronome/nfp/flower/main.h @@ -5,6 +5,7 @@ #define __NFP_FLOWER_H__ 1 #include "cmsg.h" +#include "../nfp_net.h" #include #include @@ -34,14 +35,14 @@ struct nfp_app; #define NFP_FL_MASK_REUSE_TIME_NS 40000 #define NFP_FL_MASK_ID_LOCATION 1 -#define NFP_FL_VXLAN_PORT 4789 -#define NFP_FL_GENEVE_PORT 6081 - /* Extra features bitmap. */ #define NFP_FL_FEATS_GENEVE BIT(0) #define NFP_FL_NBI_MTU_SETTING BIT(1) #define NFP_FL_FEATS_GENEVE_OPT BIT(2) #define NFP_FL_FEATS_VLAN_PCP BIT(3) +#define NFP_FL_FEATS_VF_RLIM BIT(4) +#define NFP_FL_FEATS_FLOW_MOD BIT(5) +#define NFP_FL_FEATS_FLOW_MERGE BIT(30) #define NFP_FL_FEATS_LAG BIT(31) struct nfp_fl_mask_id { @@ -117,6 +118,16 @@ struct nfp_fl_lag { struct sk_buff_head retrans_skbs; }; +/** + * struct nfp_fl_internal_ports - Flower APP priv data for additional ports + * @port_ids: Assignment of ids to any additional ports + * @lock: Lock for extra ports list + */ +struct nfp_fl_internal_ports { + struct idr port_ids; + spinlock_t lock; +}; + /** * struct nfp_flower_priv - Flower APP per-vNIC priv data * @app: Back pointer to app @@ -131,6 +142,7 @@ struct nfp_fl_lag { * @flow_table: Hash table used to store flower rules * @stats: Stored stats updates for flower rules * @stats_lock: Lock for flower rule stats updates + * @stats_ctx_table: Hash table to map stats contexts to its flow rule * @cmsg_work: Workqueue for control messages processing * @cmsg_skbs_high: List of higher priority skbs for control message * processing @@ -146,6 +158,10 @@ struct nfp_fl_lag { * @non_repr_priv: List of offloaded non-repr ports and their priv data * @active_mem_unit: Current active memory unit for flower rules * @total_mem_units: Total number of available memory units for flower rules + * @internal_ports: Internal port ids used in offloaded rules + * @qos_stats_work: Workqueue for qos stats processing + * @qos_rate_limiters: Current active qos rate limiters + * @qos_stats_lock: Lock on qos stats updates */ struct nfp_flower_priv { struct nfp_app *app; @@ -160,6 +176,7 @@ struct nfp_flower_priv { struct rhashtable flow_table; struct nfp_fl_stats *stats; spinlock_t stats_lock; /* lock stats */ + struct rhashtable stats_ctx_table; struct work_struct cmsg_work; struct sk_buff_head cmsg_skbs_high; struct sk_buff_head cmsg_skbs_low; @@ -172,6 +189,24 @@ struct nfp_flower_priv { struct list_head non_repr_priv; unsigned int active_mem_unit; unsigned int total_mem_units; + struct nfp_fl_internal_ports internal_ports; + struct delayed_work qos_stats_work; + unsigned int qos_rate_limiters; + spinlock_t qos_stats_lock; /* Protect the qos stats */ +}; + +/** + * struct nfp_fl_qos - Flower APP priv data for quality of service + * @netdev_port_id: NFP port number of repr with qos info + * @curr_stats: Currently stored stats updates for qos info + * @prev_stats: Previously stored updates for qos info + * @last_update: Stored time when last stats were updated + */ +struct nfp_fl_qos { + u32 netdev_port_id; + struct nfp_stat_pair curr_stats; + struct nfp_stat_pair prev_stats; + u64 last_update; }; /** @@ -180,14 +215,18 @@ struct nfp_flower_priv { * @lag_port_flags: Extended port flags to record lag state of repr * @mac_offloaded: Flag indicating a MAC address is offloaded for repr * @offloaded_mac_addr: MAC address that has been offloaded for repr + * @block_shared: Flag indicating if offload applies to shared blocks * @mac_list: List entry of reprs that share the same offloaded MAC + * @qos_table: Stored info on filters implementing qos */ struct nfp_flower_repr_priv { struct nfp_repr *nfp_repr; unsigned long lag_port_flags; bool mac_offloaded; u8 offloaded_mac_addr[ETH_ALEN]; + bool block_shared; struct list_head mac_list; + struct nfp_fl_qos qos_table; }; /** @@ -239,6 +278,25 @@ struct nfp_fl_payload { char *unmasked_data; char *mask_data; char *action_data; + struct list_head linked_flows; + bool in_hw; +}; + +struct nfp_fl_payload_link { + /* A link contains a pointer to a merge flow and an associated sub_flow. + * Each merge flow will feature in 2 links to its underlying sub_flows. + * A sub_flow will have at least 1 link to a merge flow or more if it + * has been used to create multiple merge flows. + * + * For a merge flow, 'linked_flows' in its nfp_fl_payload struct lists + * all links to sub_flows (sub_flow.flow) via merge.list. + * For a sub_flow, 'linked_flows' gives all links to merge flows it has + * formed (merge_flow.flow) via sub_flow.list. + */ + struct { + struct list_head list; + struct nfp_fl_payload *flow; + } merge_flow, sub_flow; }; extern const struct rhashtable_params nfp_flower_table_params; @@ -250,12 +308,40 @@ struct nfp_fl_stats_frame { __be64 stats_cookie; }; +static inline bool +nfp_flower_internal_port_can_offload(struct nfp_app *app, + struct net_device *netdev) +{ + struct nfp_flower_priv *app_priv = app->priv; + + if (!(app_priv->flower_ext_feats & NFP_FL_FEATS_FLOW_MERGE)) + return false; + if (!netdev->rtnl_link_ops) + return false; + if (!strcmp(netdev->rtnl_link_ops->kind, "openvswitch")) + return true; + + return false; +} + +/* The address of the merged flow acts as its cookie. + * Cookies supplied to us by TC flower are also addresses to allocated + * memory and thus this scheme should not generate any collisions. + */ +static inline bool nfp_flower_is_merge_flow(struct nfp_fl_payload *flow_pay) +{ + return flow_pay->tc_flower_cookie == (unsigned long)flow_pay; +} + int nfp_flower_metadata_init(struct nfp_app *app, u64 host_ctx_count, unsigned int host_ctx_split); void nfp_flower_metadata_cleanup(struct nfp_app *app); int nfp_flower_setup_tc(struct nfp_app *app, struct net_device *netdev, enum tc_setup_type type, void *type_data); +int nfp_flower_merge_offloaded_flows(struct nfp_app *app, + struct nfp_fl_payload *sub_flow1, + struct nfp_fl_payload *sub_flow2); int nfp_flower_compile_flow_match(struct nfp_app *app, struct tc_cls_flower_offload *flow, struct nfp_fl_key_ls *key_ls, @@ -270,6 +356,8 @@ int nfp_compile_flow_metadata(struct nfp_app *app, struct tc_cls_flower_offload *flow, struct nfp_fl_payload *nfp_flow, struct net_device *netdev); +void __nfp_modify_flow_metadata(struct nfp_flower_priv *priv, + struct nfp_fl_payload *nfp_flow); int nfp_modify_flow_metadata(struct nfp_app *app, struct nfp_fl_payload *nfp_flow); @@ -277,6 +365,8 @@ struct nfp_fl_payload * nfp_flower_search_fl_table(struct nfp_app *app, unsigned long tc_flower_cookie, struct net_device *netdev); struct nfp_fl_payload * +nfp_flower_get_fl_payload_from_ctx(struct nfp_app *app, u32 ctx_id); +struct nfp_fl_payload * nfp_flower_remove_fl_table(struct nfp_app *app, unsigned long tc_flower_cookie); void nfp_flower_rx_flow_stats(struct nfp_app *app, struct sk_buff *skb); @@ -302,6 +392,11 @@ int nfp_flower_lag_populate_pre_action(struct nfp_app *app, struct nfp_fl_pre_lag *pre_act); int nfp_flower_lag_get_output_id(struct nfp_app *app, struct net_device *master); +void nfp_flower_qos_init(struct nfp_app *app); +void nfp_flower_qos_cleanup(struct nfp_app *app); +int nfp_flower_setup_qos_offload(struct nfp_app *app, struct net_device *netdev, + struct tc_cls_matchall_offload *flow); +void nfp_flower_stats_rlim_reply(struct nfp_app *app, struct sk_buff *skb); int nfp_flower_reg_indir_block_handler(struct nfp_app *app, struct net_device *netdev, unsigned long event); @@ -314,4 +409,6 @@ void __nfp_flower_non_repr_priv_put(struct nfp_flower_non_repr_priv *non_repr_priv); void nfp_flower_non_repr_priv_put(struct nfp_app *app, struct net_device *netdev); +u32 nfp_flower_get_port_id_from_netdev(struct nfp_app *app, + struct net_device *netdev); #endif diff --git a/drivers/net/ethernet/netronome/nfp/flower/match.c b/drivers/net/ethernet/netronome/nfp/flower/match.c index 9b8b843d0340374a6372501e2c69eb298563be75..bfa4bf34911daf87ba27d2c47ed49e759fb55143 100644 --- a/drivers/net/ethernet/netronome/nfp/flower/match.c +++ b/drivers/net/ethernet/netronome/nfp/flower/match.c @@ -326,13 +326,12 @@ int nfp_flower_compile_flow_match(struct nfp_app *app, struct nfp_fl_payload *nfp_flow, enum nfp_flower_tun_type tun_type) { - u32 cmsg_port = 0; + u32 port_id; int err; u8 *ext; u8 *msk; - if (nfp_netdev_is_nfp_repr(netdev)) - cmsg_port = nfp_repr_get_port_id(netdev); + port_id = nfp_flower_get_port_id_from_netdev(app, netdev); memset(nfp_flow->unmasked_data, 0, key_ls->key_size); memset(nfp_flow->mask_data, 0, key_ls->key_size); @@ -358,13 +357,13 @@ int nfp_flower_compile_flow_match(struct nfp_app *app, /* Populate Exact Port data. */ err = nfp_flower_compile_port((struct nfp_flower_in_port *)ext, - cmsg_port, false, tun_type); + port_id, false, tun_type); if (err) return err; /* Populate Mask Port Data. */ err = nfp_flower_compile_port((struct nfp_flower_in_port *)msk, - cmsg_port, true, tun_type); + port_id, true, tun_type); if (err) return err; diff --git a/drivers/net/ethernet/netronome/nfp/flower/metadata.c b/drivers/net/ethernet/netronome/nfp/flower/metadata.c index 492837b852b6d3652123cc187d24dbf68bed6a19..3d326efdc814f1b50efc8913ab252e91a2669924 100644 --- a/drivers/net/ethernet/netronome/nfp/flower/metadata.c +++ b/drivers/net/ethernet/netronome/nfp/flower/metadata.c @@ -24,6 +24,18 @@ struct nfp_fl_flow_table_cmp_arg { unsigned long cookie; }; +struct nfp_fl_stats_ctx_to_flow { + struct rhash_head ht_node; + u32 stats_cxt; + struct nfp_fl_payload *flow; +}; + +static const struct rhashtable_params stats_ctx_table_params = { + .key_offset = offsetof(struct nfp_fl_stats_ctx_to_flow, stats_cxt), + .head_offset = offsetof(struct nfp_fl_stats_ctx_to_flow, ht_node), + .key_len = sizeof(u32), +}; + static int nfp_release_stats_entry(struct nfp_app *app, u32 stats_context_id) { struct nfp_flower_priv *priv = app->priv; @@ -264,9 +276,6 @@ nfp_check_mask_remove(struct nfp_app *app, char *mask_data, u32 mask_len, if (!mask_entry) return false; - if (meta_flags) - *meta_flags &= ~NFP_FL_META_FLAG_MANAGE_MASK; - *mask_id = mask_entry->mask_id; mask_entry->ref_cnt--; if (!mask_entry->ref_cnt) { @@ -285,25 +294,42 @@ int nfp_compile_flow_metadata(struct nfp_app *app, struct nfp_fl_payload *nfp_flow, struct net_device *netdev) { + struct nfp_fl_stats_ctx_to_flow *ctx_entry; struct nfp_flower_priv *priv = app->priv; struct nfp_fl_payload *check_entry; u8 new_mask_id; u32 stats_cxt; + int err; - if (nfp_get_stats_entry(app, &stats_cxt)) - return -ENOENT; + err = nfp_get_stats_entry(app, &stats_cxt); + if (err) + return err; nfp_flow->meta.host_ctx_id = cpu_to_be32(stats_cxt); nfp_flow->meta.host_cookie = cpu_to_be64(flow->cookie); nfp_flow->ingress_dev = netdev; + ctx_entry = kzalloc(sizeof(*ctx_entry), GFP_KERNEL); + if (!ctx_entry) { + err = -ENOMEM; + goto err_release_stats; + } + + ctx_entry->stats_cxt = stats_cxt; + ctx_entry->flow = nfp_flow; + + if (rhashtable_insert_fast(&priv->stats_ctx_table, &ctx_entry->ht_node, + stats_ctx_table_params)) { + err = -ENOMEM; + goto err_free_ctx_entry; + } + new_mask_id = 0; if (!nfp_check_mask_add(app, nfp_flow->mask_data, nfp_flow->meta.mask_len, &nfp_flow->meta.flags, &new_mask_id)) { - if (nfp_release_stats_entry(app, stats_cxt)) - return -EINVAL; - return -ENOENT; + err = -ENOENT; + goto err_remove_rhash; } nfp_flow->meta.flow_version = cpu_to_be64(priv->flower_version); @@ -317,43 +343,82 @@ int nfp_compile_flow_metadata(struct nfp_app *app, check_entry = nfp_flower_search_fl_table(app, flow->cookie, netdev); if (check_entry) { - if (nfp_release_stats_entry(app, stats_cxt)) - return -EINVAL; - - if (!nfp_check_mask_remove(app, nfp_flow->mask_data, - nfp_flow->meta.mask_len, - NULL, &new_mask_id)) - return -EINVAL; - - return -EEXIST; + err = -EEXIST; + goto err_remove_mask; } return 0; + +err_remove_mask: + nfp_check_mask_remove(app, nfp_flow->mask_data, nfp_flow->meta.mask_len, + NULL, &new_mask_id); +err_remove_rhash: + WARN_ON_ONCE(rhashtable_remove_fast(&priv->stats_ctx_table, + &ctx_entry->ht_node, + stats_ctx_table_params)); +err_free_ctx_entry: + kfree(ctx_entry); +err_release_stats: + nfp_release_stats_entry(app, stats_cxt); + + return err; +} + +void __nfp_modify_flow_metadata(struct nfp_flower_priv *priv, + struct nfp_fl_payload *nfp_flow) +{ + nfp_flow->meta.flags &= ~NFP_FL_META_FLAG_MANAGE_MASK; + nfp_flow->meta.flow_version = cpu_to_be64(priv->flower_version); + priv->flower_version++; } int nfp_modify_flow_metadata(struct nfp_app *app, struct nfp_fl_payload *nfp_flow) { + struct nfp_fl_stats_ctx_to_flow *ctx_entry; struct nfp_flower_priv *priv = app->priv; u8 new_mask_id = 0; u32 temp_ctx_id; + __nfp_modify_flow_metadata(priv, nfp_flow); + nfp_check_mask_remove(app, nfp_flow->mask_data, nfp_flow->meta.mask_len, &nfp_flow->meta.flags, &new_mask_id); - nfp_flow->meta.flow_version = cpu_to_be64(priv->flower_version); - priv->flower_version++; - /* Update flow payload with mask ids. */ nfp_flow->unmasked_data[NFP_FL_MASK_ID_LOCATION] = new_mask_id; - /* Release the stats ctx id. */ + /* Release the stats ctx id and ctx to flow table entry. */ temp_ctx_id = be32_to_cpu(nfp_flow->meta.host_ctx_id); + ctx_entry = rhashtable_lookup_fast(&priv->stats_ctx_table, &temp_ctx_id, + stats_ctx_table_params); + if (!ctx_entry) + return -ENOENT; + + WARN_ON_ONCE(rhashtable_remove_fast(&priv->stats_ctx_table, + &ctx_entry->ht_node, + stats_ctx_table_params)); + kfree(ctx_entry); + return nfp_release_stats_entry(app, temp_ctx_id); } +struct nfp_fl_payload * +nfp_flower_get_fl_payload_from_ctx(struct nfp_app *app, u32 ctx_id) +{ + struct nfp_fl_stats_ctx_to_flow *ctx_entry; + struct nfp_flower_priv *priv = app->priv; + + ctx_entry = rhashtable_lookup_fast(&priv->stats_ctx_table, &ctx_id, + stats_ctx_table_params); + if (!ctx_entry) + return NULL; + + return ctx_entry->flow; +} + static int nfp_fl_obj_cmpfn(struct rhashtable_compare_arg *arg, const void *obj) { @@ -403,6 +468,10 @@ int nfp_flower_metadata_init(struct nfp_app *app, u64 host_ctx_count, if (err) return err; + err = rhashtable_init(&priv->stats_ctx_table, &stats_ctx_table_params); + if (err) + goto err_free_flow_table; + get_random_bytes(&priv->mask_id_seed, sizeof(priv->mask_id_seed)); /* Init ring buffer and unallocated mask_ids. */ @@ -410,7 +479,7 @@ int nfp_flower_metadata_init(struct nfp_app *app, u64 host_ctx_count, kmalloc_array(NFP_FLOWER_MASK_ENTRY_RS, NFP_FLOWER_MASK_ELEMENT_RS, GFP_KERNEL); if (!priv->mask_ids.mask_id_free_list.buf) - goto err_free_flow_table; + goto err_free_stats_ctx_table; priv->mask_ids.init_unallocated = NFP_FLOWER_MASK_ENTRY_RS - 1; @@ -447,6 +516,8 @@ err_free_last_used: kfree(priv->mask_ids.last_used); err_free_mask_id: kfree(priv->mask_ids.mask_id_free_list.buf); +err_free_stats_ctx_table: + rhashtable_destroy(&priv->stats_ctx_table); err_free_flow_table: rhashtable_destroy(&priv->flow_table); return -ENOMEM; @@ -461,6 +532,8 @@ void nfp_flower_metadata_cleanup(struct nfp_app *app) rhashtable_free_and_destroy(&priv->flow_table, nfp_check_rhashtable_empty, NULL); + rhashtable_free_and_destroy(&priv->stats_ctx_table, + nfp_check_rhashtable_empty, NULL); kvfree(priv->stats); kfree(priv->mask_ids.mask_id_free_list.buf); kfree(priv->mask_ids.last_used); diff --git a/drivers/net/ethernet/netronome/nfp/flower/offload.c b/drivers/net/ethernet/netronome/nfp/flower/offload.c index 450d7296fd57424e8f2f89305e18afc0d34f1c91..1fbfeb43c53810c0987374a0851804749fa282b7 100644 --- a/drivers/net/ethernet/netronome/nfp/flower/offload.c +++ b/drivers/net/ethernet/netronome/nfp/flower/offload.c @@ -55,6 +55,28 @@ BIT(FLOW_DISSECTOR_KEY_ENC_IPV4_ADDRS) | \ BIT(FLOW_DISSECTOR_KEY_ENC_PORTS)) +#define NFP_FLOWER_MERGE_FIELDS \ + (NFP_FLOWER_LAYER_PORT | \ + NFP_FLOWER_LAYER_MAC | \ + NFP_FLOWER_LAYER_TP | \ + NFP_FLOWER_LAYER_IPV4 | \ + NFP_FLOWER_LAYER_IPV6) + +struct nfp_flower_merge_check { + union { + struct { + __be16 tci; + struct nfp_flower_mac_mpls l2; + struct nfp_flower_tp_ports l4; + union { + struct nfp_flower_ipv4 ipv4; + struct nfp_flower_ipv6 ipv6; + }; + }; + unsigned long vals[8]; + }; +}; + static int nfp_flower_xmit_flow(struct nfp_app *app, struct nfp_fl_payload *nfp_flow, u8 mtype) @@ -195,7 +217,7 @@ nfp_flower_calculate_key_layers(struct nfp_app *app, flow_rule_match_enc_opts(rule, &enc_op); switch (enc_ports.key->dst) { - case htons(NFP_FL_VXLAN_PORT): + case htons(IANA_VXLAN_UDP_PORT): *tun_type = NFP_FL_TUNNEL_VXLAN; key_layer |= NFP_FLOWER_LAYER_VXLAN; key_size += sizeof(struct nfp_flower_ipv4_udp_tun); @@ -203,7 +225,7 @@ nfp_flower_calculate_key_layers(struct nfp_app *app, if (enc_op.key) return -EOPNOTSUPP; break; - case htons(NFP_FL_GENEVE_PORT): + case htons(GENEVE_UDP_PORT): if (!(priv->flower_ext_feats & NFP_FL_FEATS_GENEVE)) return -EOPNOTSUPP; *tun_type = NFP_FL_TUNNEL_GENEVE; @@ -326,7 +348,7 @@ nfp_flower_calculate_key_layers(struct nfp_app *app, break; case cpu_to_be16(ETH_P_IPV6): - key_layer |= NFP_FLOWER_LAYER_IPV6; + key_layer |= NFP_FLOWER_LAYER_IPV6; key_size += sizeof(struct nfp_flower_ipv6); break; @@ -376,6 +398,8 @@ nfp_flower_allocate_new(struct nfp_fl_key_ls *key_layer) flow_pay->nfp_tun_ipv4_addr = 0; flow_pay->meta.flags = 0; + INIT_LIST_HEAD(&flow_pay->linked_flows); + flow_pay->in_hw = false; return flow_pay; @@ -388,6 +412,447 @@ err_free_flow: return NULL; } +static int +nfp_flower_update_merge_with_actions(struct nfp_fl_payload *flow, + struct nfp_flower_merge_check *merge, + u8 *last_act_id, int *act_out) +{ + struct nfp_fl_set_ipv6_tc_hl_fl *ipv6_tc_hl_fl; + struct nfp_fl_set_ip4_ttl_tos *ipv4_ttl_tos; + struct nfp_fl_set_ip4_addrs *ipv4_add; + struct nfp_fl_set_ipv6_addr *ipv6_add; + struct nfp_fl_push_vlan *push_vlan; + struct nfp_fl_set_tport *tport; + struct nfp_fl_set_eth *eth; + struct nfp_fl_act_head *a; + unsigned int act_off = 0; + u8 act_id = 0; + u8 *ports; + int i; + + while (act_off < flow->meta.act_len) { + a = (struct nfp_fl_act_head *)&flow->action_data[act_off]; + act_id = a->jump_id; + + switch (act_id) { + case NFP_FL_ACTION_OPCODE_OUTPUT: + if (act_out) + (*act_out)++; + break; + case NFP_FL_ACTION_OPCODE_PUSH_VLAN: + push_vlan = (struct nfp_fl_push_vlan *)a; + if (push_vlan->vlan_tci) + merge->tci = cpu_to_be16(0xffff); + break; + case NFP_FL_ACTION_OPCODE_POP_VLAN: + merge->tci = cpu_to_be16(0); + break; + case NFP_FL_ACTION_OPCODE_SET_IPV4_TUNNEL: + /* New tunnel header means l2 to l4 can be matched. */ + eth_broadcast_addr(&merge->l2.mac_dst[0]); + eth_broadcast_addr(&merge->l2.mac_src[0]); + memset(&merge->l4, 0xff, + sizeof(struct nfp_flower_tp_ports)); + memset(&merge->ipv4, 0xff, + sizeof(struct nfp_flower_ipv4)); + break; + case NFP_FL_ACTION_OPCODE_SET_ETHERNET: + eth = (struct nfp_fl_set_eth *)a; + for (i = 0; i < ETH_ALEN; i++) + merge->l2.mac_dst[i] |= eth->eth_addr_mask[i]; + for (i = 0; i < ETH_ALEN; i++) + merge->l2.mac_src[i] |= + eth->eth_addr_mask[ETH_ALEN + i]; + break; + case NFP_FL_ACTION_OPCODE_SET_IPV4_ADDRS: + ipv4_add = (struct nfp_fl_set_ip4_addrs *)a; + merge->ipv4.ipv4_src |= ipv4_add->ipv4_src_mask; + merge->ipv4.ipv4_dst |= ipv4_add->ipv4_dst_mask; + break; + case NFP_FL_ACTION_OPCODE_SET_IPV4_TTL_TOS: + ipv4_ttl_tos = (struct nfp_fl_set_ip4_ttl_tos *)a; + merge->ipv4.ip_ext.ttl |= ipv4_ttl_tos->ipv4_ttl_mask; + merge->ipv4.ip_ext.tos |= ipv4_ttl_tos->ipv4_tos_mask; + break; + case NFP_FL_ACTION_OPCODE_SET_IPV6_SRC: + ipv6_add = (struct nfp_fl_set_ipv6_addr *)a; + for (i = 0; i < 4; i++) + merge->ipv6.ipv6_src.in6_u.u6_addr32[i] |= + ipv6_add->ipv6[i].mask; + break; + case NFP_FL_ACTION_OPCODE_SET_IPV6_DST: + ipv6_add = (struct nfp_fl_set_ipv6_addr *)a; + for (i = 0; i < 4; i++) + merge->ipv6.ipv6_dst.in6_u.u6_addr32[i] |= + ipv6_add->ipv6[i].mask; + break; + case NFP_FL_ACTION_OPCODE_SET_IPV6_TC_HL_FL: + ipv6_tc_hl_fl = (struct nfp_fl_set_ipv6_tc_hl_fl *)a; + merge->ipv6.ip_ext.ttl |= + ipv6_tc_hl_fl->ipv6_hop_limit_mask; + merge->ipv6.ip_ext.tos |= ipv6_tc_hl_fl->ipv6_tc_mask; + merge->ipv6.ipv6_flow_label_exthdr |= + ipv6_tc_hl_fl->ipv6_label_mask; + break; + case NFP_FL_ACTION_OPCODE_SET_UDP: + case NFP_FL_ACTION_OPCODE_SET_TCP: + tport = (struct nfp_fl_set_tport *)a; + ports = (u8 *)&merge->l4.port_src; + for (i = 0; i < 4; i++) + ports[i] |= tport->tp_port_mask[i]; + break; + case NFP_FL_ACTION_OPCODE_PRE_TUNNEL: + case NFP_FL_ACTION_OPCODE_PRE_LAG: + case NFP_FL_ACTION_OPCODE_PUSH_GENEVE: + break; + default: + return -EOPNOTSUPP; + } + + act_off += a->len_lw << NFP_FL_LW_SIZ; + } + + if (last_act_id) + *last_act_id = act_id; + + return 0; +} + +static int +nfp_flower_populate_merge_match(struct nfp_fl_payload *flow, + struct nfp_flower_merge_check *merge, + bool extra_fields) +{ + struct nfp_flower_meta_tci *meta_tci; + u8 *mask = flow->mask_data; + u8 key_layer, match_size; + + memset(merge, 0, sizeof(struct nfp_flower_merge_check)); + + meta_tci = (struct nfp_flower_meta_tci *)mask; + key_layer = meta_tci->nfp_flow_key_layer; + + if (key_layer & ~NFP_FLOWER_MERGE_FIELDS && !extra_fields) + return -EOPNOTSUPP; + + merge->tci = meta_tci->tci; + mask += sizeof(struct nfp_flower_meta_tci); + + if (key_layer & NFP_FLOWER_LAYER_EXT_META) + mask += sizeof(struct nfp_flower_ext_meta); + + mask += sizeof(struct nfp_flower_in_port); + + if (key_layer & NFP_FLOWER_LAYER_MAC) { + match_size = sizeof(struct nfp_flower_mac_mpls); + memcpy(&merge->l2, mask, match_size); + mask += match_size; + } + + if (key_layer & NFP_FLOWER_LAYER_TP) { + match_size = sizeof(struct nfp_flower_tp_ports); + memcpy(&merge->l4, mask, match_size); + mask += match_size; + } + + if (key_layer & NFP_FLOWER_LAYER_IPV4) { + match_size = sizeof(struct nfp_flower_ipv4); + memcpy(&merge->ipv4, mask, match_size); + } + + if (key_layer & NFP_FLOWER_LAYER_IPV6) { + match_size = sizeof(struct nfp_flower_ipv6); + memcpy(&merge->ipv6, mask, match_size); + } + + return 0; +} + +static int +nfp_flower_can_merge(struct nfp_fl_payload *sub_flow1, + struct nfp_fl_payload *sub_flow2) +{ + /* Two flows can be merged if sub_flow2 only matches on bits that are + * either matched by sub_flow1 or set by a sub_flow1 action. This + * ensures that every packet that hits sub_flow1 and recirculates is + * guaranteed to hit sub_flow2. + */ + struct nfp_flower_merge_check sub_flow1_merge, sub_flow2_merge; + int err, act_out = 0; + u8 last_act_id = 0; + + err = nfp_flower_populate_merge_match(sub_flow1, &sub_flow1_merge, + true); + if (err) + return err; + + err = nfp_flower_populate_merge_match(sub_flow2, &sub_flow2_merge, + false); + if (err) + return err; + + err = nfp_flower_update_merge_with_actions(sub_flow1, &sub_flow1_merge, + &last_act_id, &act_out); + if (err) + return err; + + /* Must only be 1 output action and it must be the last in sequence. */ + if (act_out != 1 || last_act_id != NFP_FL_ACTION_OPCODE_OUTPUT) + return -EOPNOTSUPP; + + /* Reject merge if sub_flow2 matches on something that is not matched + * on or set in an action by sub_flow1. + */ + err = bitmap_andnot(sub_flow2_merge.vals, sub_flow2_merge.vals, + sub_flow1_merge.vals, + sizeof(struct nfp_flower_merge_check) * 8); + if (err) + return -EINVAL; + + return 0; +} + +static unsigned int +nfp_flower_copy_pre_actions(char *act_dst, char *act_src, int len, + bool *tunnel_act) +{ + unsigned int act_off = 0, act_len; + struct nfp_fl_act_head *a; + u8 act_id = 0; + + while (act_off < len) { + a = (struct nfp_fl_act_head *)&act_src[act_off]; + act_len = a->len_lw << NFP_FL_LW_SIZ; + act_id = a->jump_id; + + switch (act_id) { + case NFP_FL_ACTION_OPCODE_PRE_TUNNEL: + if (tunnel_act) + *tunnel_act = true; + /* fall through */ + case NFP_FL_ACTION_OPCODE_PRE_LAG: + memcpy(act_dst + act_off, act_src + act_off, act_len); + break; + default: + return act_off; + } + + act_off += act_len; + } + + return act_off; +} + +static int nfp_fl_verify_post_tun_acts(char *acts, int len) +{ + struct nfp_fl_act_head *a; + unsigned int act_off = 0; + + while (act_off < len) { + a = (struct nfp_fl_act_head *)&acts[act_off]; + if (a->jump_id != NFP_FL_ACTION_OPCODE_OUTPUT) + return -EOPNOTSUPP; + + act_off += a->len_lw << NFP_FL_LW_SIZ; + } + + return 0; +} + +static int +nfp_flower_merge_action(struct nfp_fl_payload *sub_flow1, + struct nfp_fl_payload *sub_flow2, + struct nfp_fl_payload *merge_flow) +{ + unsigned int sub1_act_len, sub2_act_len, pre_off1, pre_off2; + bool tunnel_act = false; + char *merge_act; + int err; + + /* The last action of sub_flow1 must be output - do not merge this. */ + sub1_act_len = sub_flow1->meta.act_len - sizeof(struct nfp_fl_output); + sub2_act_len = sub_flow2->meta.act_len; + + if (!sub2_act_len) + return -EINVAL; + + if (sub1_act_len + sub2_act_len > NFP_FL_MAX_A_SIZ) + return -EINVAL; + + /* A shortcut can only be applied if there is a single action. */ + if (sub1_act_len) + merge_flow->meta.shortcut = cpu_to_be32(NFP_FL_SC_ACT_NULL); + else + merge_flow->meta.shortcut = sub_flow2->meta.shortcut; + + merge_flow->meta.act_len = sub1_act_len + sub2_act_len; + merge_act = merge_flow->action_data; + + /* Copy any pre-actions to the start of merge flow action list. */ + pre_off1 = nfp_flower_copy_pre_actions(merge_act, + sub_flow1->action_data, + sub1_act_len, &tunnel_act); + merge_act += pre_off1; + sub1_act_len -= pre_off1; + pre_off2 = nfp_flower_copy_pre_actions(merge_act, + sub_flow2->action_data, + sub2_act_len, NULL); + merge_act += pre_off2; + sub2_act_len -= pre_off2; + + /* FW does a tunnel push when egressing, therefore, if sub_flow 1 pushes + * a tunnel, sub_flow 2 can only have output actions for a valid merge. + */ + if (tunnel_act) { + char *post_tun_acts = &sub_flow2->action_data[pre_off2]; + + err = nfp_fl_verify_post_tun_acts(post_tun_acts, sub2_act_len); + if (err) + return err; + } + + /* Copy remaining actions from sub_flows 1 and 2. */ + memcpy(merge_act, sub_flow1->action_data + pre_off1, sub1_act_len); + merge_act += sub1_act_len; + memcpy(merge_act, sub_flow2->action_data + pre_off2, sub2_act_len); + + return 0; +} + +/* Flow link code should only be accessed under RTNL. */ +static void nfp_flower_unlink_flow(struct nfp_fl_payload_link *link) +{ + list_del(&link->merge_flow.list); + list_del(&link->sub_flow.list); + kfree(link); +} + +static void nfp_flower_unlink_flows(struct nfp_fl_payload *merge_flow, + struct nfp_fl_payload *sub_flow) +{ + struct nfp_fl_payload_link *link; + + list_for_each_entry(link, &merge_flow->linked_flows, merge_flow.list) + if (link->sub_flow.flow == sub_flow) { + nfp_flower_unlink_flow(link); + return; + } +} + +static int nfp_flower_link_flows(struct nfp_fl_payload *merge_flow, + struct nfp_fl_payload *sub_flow) +{ + struct nfp_fl_payload_link *link; + + link = kmalloc(sizeof(*link), GFP_KERNEL); + if (!link) + return -ENOMEM; + + link->merge_flow.flow = merge_flow; + list_add_tail(&link->merge_flow.list, &merge_flow->linked_flows); + link->sub_flow.flow = sub_flow; + list_add_tail(&link->sub_flow.list, &sub_flow->linked_flows); + + return 0; +} + +/** + * nfp_flower_merge_offloaded_flows() - Merge 2 existing flows to single flow. + * @app: Pointer to the APP handle + * @sub_flow1: Initial flow matched to produce merge hint + * @sub_flow2: Post recirculation flow matched in merge hint + * + * Combines 2 flows (if valid) to a single flow, removing the initial from hw + * and offloading the new, merged flow. + * + * Return: negative value on error, 0 in success. + */ +int nfp_flower_merge_offloaded_flows(struct nfp_app *app, + struct nfp_fl_payload *sub_flow1, + struct nfp_fl_payload *sub_flow2) +{ + struct tc_cls_flower_offload merge_tc_off; + struct nfp_flower_priv *priv = app->priv; + struct nfp_fl_payload *merge_flow; + struct nfp_fl_key_ls merge_key_ls; + int err; + + ASSERT_RTNL(); + + if (sub_flow1 == sub_flow2 || + nfp_flower_is_merge_flow(sub_flow1) || + nfp_flower_is_merge_flow(sub_flow2)) + return -EINVAL; + + err = nfp_flower_can_merge(sub_flow1, sub_flow2); + if (err) + return err; + + merge_key_ls.key_size = sub_flow1->meta.key_len; + + merge_flow = nfp_flower_allocate_new(&merge_key_ls); + if (!merge_flow) + return -ENOMEM; + + merge_flow->tc_flower_cookie = (unsigned long)merge_flow; + merge_flow->ingress_dev = sub_flow1->ingress_dev; + + memcpy(merge_flow->unmasked_data, sub_flow1->unmasked_data, + sub_flow1->meta.key_len); + memcpy(merge_flow->mask_data, sub_flow1->mask_data, + sub_flow1->meta.mask_len); + + err = nfp_flower_merge_action(sub_flow1, sub_flow2, merge_flow); + if (err) + goto err_destroy_merge_flow; + + err = nfp_flower_link_flows(merge_flow, sub_flow1); + if (err) + goto err_destroy_merge_flow; + + err = nfp_flower_link_flows(merge_flow, sub_flow2); + if (err) + goto err_unlink_sub_flow1; + + merge_tc_off.cookie = merge_flow->tc_flower_cookie; + err = nfp_compile_flow_metadata(app, &merge_tc_off, merge_flow, + merge_flow->ingress_dev); + if (err) + goto err_unlink_sub_flow2; + + err = rhashtable_insert_fast(&priv->flow_table, &merge_flow->fl_node, + nfp_flower_table_params); + if (err) + goto err_release_metadata; + + err = nfp_flower_xmit_flow(app, merge_flow, + NFP_FLOWER_CMSG_TYPE_FLOW_MOD); + if (err) + goto err_remove_rhash; + + merge_flow->in_hw = true; + sub_flow1->in_hw = false; + + return 0; + +err_remove_rhash: + WARN_ON_ONCE(rhashtable_remove_fast(&priv->flow_table, + &merge_flow->fl_node, + nfp_flower_table_params)); +err_release_metadata: + nfp_modify_flow_metadata(app, merge_flow); +err_unlink_sub_flow2: + nfp_flower_unlink_flows(merge_flow, sub_flow2); +err_unlink_sub_flow1: + nfp_flower_unlink_flows(merge_flow, sub_flow1); +err_destroy_merge_flow: + kfree(merge_flow->action_data); + kfree(merge_flow->mask_data); + kfree(merge_flow->unmasked_data); + kfree(merge_flow); + return err; +} + /** * nfp_flower_add_offload() - Adds a new flow to hardware. * @app: Pointer to the APP handle @@ -454,6 +919,8 @@ nfp_flower_add_offload(struct nfp_app *app, struct net_device *netdev, if (port) port->tc_offload_cnt++; + flow_pay->in_hw = true; + /* Deallocate flow payload when flower rule has been destroyed. */ kfree(key_layer); @@ -475,6 +942,75 @@ err_free_key_ls: return err; } +static void +nfp_flower_remove_merge_flow(struct nfp_app *app, + struct nfp_fl_payload *del_sub_flow, + struct nfp_fl_payload *merge_flow) +{ + struct nfp_flower_priv *priv = app->priv; + struct nfp_fl_payload_link *link, *temp; + struct nfp_fl_payload *origin; + bool mod = false; + int err; + + link = list_first_entry(&merge_flow->linked_flows, + struct nfp_fl_payload_link, merge_flow.list); + origin = link->sub_flow.flow; + + /* Re-add rule the merge had overwritten if it has not been deleted. */ + if (origin != del_sub_flow) + mod = true; + + err = nfp_modify_flow_metadata(app, merge_flow); + if (err) { + nfp_flower_cmsg_warn(app, "Metadata fail for merge flow delete.\n"); + goto err_free_links; + } + + if (!mod) { + err = nfp_flower_xmit_flow(app, merge_flow, + NFP_FLOWER_CMSG_TYPE_FLOW_DEL); + if (err) { + nfp_flower_cmsg_warn(app, "Failed to delete merged flow.\n"); + goto err_free_links; + } + } else { + __nfp_modify_flow_metadata(priv, origin); + err = nfp_flower_xmit_flow(app, origin, + NFP_FLOWER_CMSG_TYPE_FLOW_MOD); + if (err) + nfp_flower_cmsg_warn(app, "Failed to revert merge flow.\n"); + origin->in_hw = true; + } + +err_free_links: + /* Clean any links connected with the merged flow. */ + list_for_each_entry_safe(link, temp, &merge_flow->linked_flows, + merge_flow.list) + nfp_flower_unlink_flow(link); + + kfree(merge_flow->action_data); + kfree(merge_flow->mask_data); + kfree(merge_flow->unmasked_data); + WARN_ON_ONCE(rhashtable_remove_fast(&priv->flow_table, + &merge_flow->fl_node, + nfp_flower_table_params)); + kfree_rcu(merge_flow, rcu); +} + +static void +nfp_flower_del_linked_merge_flows(struct nfp_app *app, + struct nfp_fl_payload *sub_flow) +{ + struct nfp_fl_payload_link *link, *temp; + + /* Remove any merge flow formed from the deleted sub_flow. */ + list_for_each_entry_safe(link, temp, &sub_flow->linked_flows, + sub_flow.list) + nfp_flower_remove_merge_flow(app, sub_flow, + link->merge_flow.flow); +} + /** * nfp_flower_del_offload() - Removes a flow from hardware. * @app: Pointer to the APP handle @@ -482,7 +1018,7 @@ err_free_key_ls: * @flow: TC flower classifier offload structure * * Removes a flow from the repeated hash structure and clears the - * action payload. + * action payload. Any flows merged from this are also deleted. * * Return: negative value on error, 0 if removed successfully. */ @@ -504,17 +1040,22 @@ nfp_flower_del_offload(struct nfp_app *app, struct net_device *netdev, err = nfp_modify_flow_metadata(app, nfp_flow); if (err) - goto err_free_flow; + goto err_free_merge_flow; if (nfp_flow->nfp_tun_ipv4_addr) nfp_tunnel_del_ipv4_off(app, nfp_flow->nfp_tun_ipv4_addr); + if (!nfp_flow->in_hw) { + err = 0; + goto err_free_merge_flow; + } + err = nfp_flower_xmit_flow(app, nfp_flow, NFP_FLOWER_CMSG_TYPE_FLOW_DEL); - if (err) - goto err_free_flow; + /* Fall through on error. */ -err_free_flow: +err_free_merge_flow: + nfp_flower_del_linked_merge_flows(app, nfp_flow); if (port) port->tc_offload_cnt--; kfree(nfp_flow->action_data); @@ -527,6 +1068,52 @@ err_free_flow: return err; } +static void +__nfp_flower_update_merge_stats(struct nfp_app *app, + struct nfp_fl_payload *merge_flow) +{ + struct nfp_flower_priv *priv = app->priv; + struct nfp_fl_payload_link *link; + struct nfp_fl_payload *sub_flow; + u64 pkts, bytes, used; + u32 ctx_id; + + ctx_id = be32_to_cpu(merge_flow->meta.host_ctx_id); + pkts = priv->stats[ctx_id].pkts; + /* Do not cycle subflows if no stats to distribute. */ + if (!pkts) + return; + bytes = priv->stats[ctx_id].bytes; + used = priv->stats[ctx_id].used; + + /* Reset stats for the merge flow. */ + priv->stats[ctx_id].pkts = 0; + priv->stats[ctx_id].bytes = 0; + + /* The merge flow has received stats updates from firmware. + * Distribute these stats to all subflows that form the merge. + * The stats will collected from TC via the subflows. + */ + list_for_each_entry(link, &merge_flow->linked_flows, merge_flow.list) { + sub_flow = link->sub_flow.flow; + ctx_id = be32_to_cpu(sub_flow->meta.host_ctx_id); + priv->stats[ctx_id].pkts += pkts; + priv->stats[ctx_id].bytes += bytes; + max_t(u64, priv->stats[ctx_id].used, used); + } +} + +static void +nfp_flower_update_merge_stats(struct nfp_app *app, + struct nfp_fl_payload *sub_flow) +{ + struct nfp_fl_payload_link *link; + + /* Get merge flows that the subflow forms to distribute their stats. */ + list_for_each_entry(link, &sub_flow->linked_flows, sub_flow.list) + __nfp_flower_update_merge_stats(app, link->merge_flow.flow); +} + /** * nfp_flower_get_stats() - Populates flow stats obtained from hardware. * @app: Pointer to the APP handle @@ -553,6 +1140,10 @@ nfp_flower_get_stats(struct nfp_app *app, struct net_device *netdev, ctx_id = be32_to_cpu(nfp_flow->meta.host_ctx_id); spin_lock_bh(&priv->stats_lock); + /* If request is for a sub_flow, update stats from merged flows. */ + if (!list_empty(&nfp_flow->linked_flows)) + nfp_flower_update_merge_stats(app, nfp_flow); + flow_stats_update(&flow->stats, priv->stats[ctx_id].bytes, priv->stats[ctx_id].pkts, priv->stats[ctx_id].used); @@ -594,6 +1185,9 @@ static int nfp_flower_setup_tc_block_cb(enum tc_setup_type type, case TC_SETUP_CLSFLOWER: return nfp_flower_repr_offload(repr->app, repr->netdev, type_data); + case TC_SETUP_CLSMATCHALL: + return nfp_flower_setup_qos_offload(repr->app, repr->netdev, + type_data); default: return -EOPNOTSUPP; } @@ -603,10 +1197,14 @@ static int nfp_flower_setup_tc_block(struct net_device *netdev, struct tc_block_offload *f) { struct nfp_repr *repr = netdev_priv(netdev); + struct nfp_flower_repr_priv *repr_priv; if (f->binder_type != TCF_BLOCK_BINDER_TYPE_CLSACT_INGRESS) return -EOPNOTSUPP; + repr_priv = repr->app_priv; + repr_priv->block_shared = tcf_block_shared(f->block); + switch (f->command) { case TC_BLOCK_BIND: return tcf_block_cb_register(f->block, @@ -682,7 +1280,9 @@ nfp_flower_setup_indr_tc_block(struct net_device *netdev, struct nfp_app *app, struct nfp_flower_priv *priv = app->priv; int err; - if (f->binder_type != TCF_BLOCK_BINDER_TYPE_CLSACT_INGRESS) + if (f->binder_type != TCF_BLOCK_BINDER_TYPE_CLSACT_INGRESS && + !(f->binder_type == TCF_BLOCK_BINDER_TYPE_CLSACT_EGRESS && + nfp_flower_internal_port_can_offload(app, netdev))) return -EOPNOTSUPP; switch (f->command) { diff --git a/drivers/net/ethernet/netronome/nfp/flower/qos_conf.c b/drivers/net/ethernet/netronome/nfp/flower/qos_conf.c new file mode 100644 index 0000000000000000000000000000000000000000..86e968cd5ffd3dc6ce9cc17ef29bffeb59ec5caa --- /dev/null +++ b/drivers/net/ethernet/netronome/nfp/flower/qos_conf.c @@ -0,0 +1,366 @@ +// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +/* Copyright (C) 2019 Netronome Systems, Inc. */ + +#include +#include +#include + +#include "cmsg.h" +#include "main.h" +#include "../nfp_port.h" + +#define NFP_FL_QOS_UPDATE msecs_to_jiffies(1000) + +struct nfp_police_cfg_head { + __be32 flags_opts; + __be32 port; +}; + +/* Police cmsg for configuring a trTCM traffic conditioner (8W/32B) + * See RFC 2698 for more details. + * ---------------------------------------------------------------- + * 3 2 1 + * 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 + * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + * | Flag options | + * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + * | Port Ingress | + * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + * | Token Bucket Peak | + * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + * | Token Bucket Committed | + * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + * | Peak Burst Size | + * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + * | Committed Burst Size | + * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + * | Peak Information Rate | + * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + * | Committed Information Rate | + * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + */ +struct nfp_police_config { + struct nfp_police_cfg_head head; + __be32 bkt_tkn_p; + __be32 bkt_tkn_c; + __be32 pbs; + __be32 cbs; + __be32 pir; + __be32 cir; +}; + +struct nfp_police_stats_reply { + struct nfp_police_cfg_head head; + __be64 pass_bytes; + __be64 pass_pkts; + __be64 drop_bytes; + __be64 drop_pkts; +}; + +static int +nfp_flower_install_rate_limiter(struct nfp_app *app, struct net_device *netdev, + struct tc_cls_matchall_offload *flow, + struct netlink_ext_ack *extack) +{ + struct flow_action_entry *action = &flow->rule->action.entries[0]; + struct nfp_flower_priv *fl_priv = app->priv; + struct nfp_flower_repr_priv *repr_priv; + struct nfp_police_config *config; + struct nfp_repr *repr; + struct sk_buff *skb; + u32 netdev_port_id; + u64 burst, rate; + + if (!nfp_netdev_is_nfp_repr(netdev)) { + NL_SET_ERR_MSG_MOD(extack, "unsupported offload: qos rate limit offload not supported on higher level port"); + return -EOPNOTSUPP; + } + repr = netdev_priv(netdev); + repr_priv = repr->app_priv; + + if (repr_priv->block_shared) { + NL_SET_ERR_MSG_MOD(extack, "unsupported offload: qos rate limit offload not supported on shared blocks"); + return -EOPNOTSUPP; + } + + if (repr->port->type != NFP_PORT_VF_PORT) { + NL_SET_ERR_MSG_MOD(extack, "unsupported offload: qos rate limit offload not supported on non-VF ports"); + return -EOPNOTSUPP; + } + + if (!flow_offload_has_one_action(&flow->rule->action)) { + NL_SET_ERR_MSG_MOD(extack, "unsupported offload: qos rate limit offload requires a single action"); + return -EOPNOTSUPP; + } + + if (flow->common.prio != (1 << 16)) { + NL_SET_ERR_MSG_MOD(extack, "unsupported offload: qos rate limit offload requires highest priority"); + return -EOPNOTSUPP; + } + + if (action->id != FLOW_ACTION_POLICE) { + NL_SET_ERR_MSG_MOD(extack, "unsupported offload: qos rate limit offload requires police action"); + return -EOPNOTSUPP; + } + + rate = action->police.rate_bytes_ps; + burst = div_u64(rate * PSCHED_NS2TICKS(action->police.burst), + PSCHED_TICKS_PER_SEC); + netdev_port_id = nfp_repr_get_port_id(netdev); + + skb = nfp_flower_cmsg_alloc(repr->app, sizeof(struct nfp_police_config), + NFP_FLOWER_CMSG_TYPE_QOS_MOD, GFP_KERNEL); + if (!skb) + return -ENOMEM; + + config = nfp_flower_cmsg_get_data(skb); + memset(config, 0, sizeof(struct nfp_police_config)); + config->head.port = cpu_to_be32(netdev_port_id); + config->bkt_tkn_p = cpu_to_be32(burst); + config->bkt_tkn_c = cpu_to_be32(burst); + config->pbs = cpu_to_be32(burst); + config->cbs = cpu_to_be32(burst); + config->pir = cpu_to_be32(rate); + config->cir = cpu_to_be32(rate); + nfp_ctrl_tx(repr->app->ctrl, skb); + + repr_priv->qos_table.netdev_port_id = netdev_port_id; + fl_priv->qos_rate_limiters++; + if (fl_priv->qos_rate_limiters == 1) + schedule_delayed_work(&fl_priv->qos_stats_work, + NFP_FL_QOS_UPDATE); + + return 0; +} + +static int +nfp_flower_remove_rate_limiter(struct nfp_app *app, struct net_device *netdev, + struct tc_cls_matchall_offload *flow, + struct netlink_ext_ack *extack) +{ + struct nfp_flower_priv *fl_priv = app->priv; + struct nfp_flower_repr_priv *repr_priv; + struct nfp_police_config *config; + struct nfp_repr *repr; + struct sk_buff *skb; + u32 netdev_port_id; + + if (!nfp_netdev_is_nfp_repr(netdev)) { + NL_SET_ERR_MSG_MOD(extack, "unsupported offload: qos rate limit offload not supported on higher level port"); + return -EOPNOTSUPP; + } + repr = netdev_priv(netdev); + + netdev_port_id = nfp_repr_get_port_id(netdev); + repr_priv = repr->app_priv; + + if (!repr_priv->qos_table.netdev_port_id) { + NL_SET_ERR_MSG_MOD(extack, "unsupported offload: cannot remove qos entry that does not exist"); + return -EOPNOTSUPP; + } + + skb = nfp_flower_cmsg_alloc(repr->app, sizeof(struct nfp_police_config), + NFP_FLOWER_CMSG_TYPE_QOS_DEL, GFP_KERNEL); + if (!skb) + return -ENOMEM; + + /* Clear all qos associate data for this interface */ + memset(&repr_priv->qos_table, 0, sizeof(struct nfp_fl_qos)); + fl_priv->qos_rate_limiters--; + if (!fl_priv->qos_rate_limiters) + cancel_delayed_work_sync(&fl_priv->qos_stats_work); + + config = nfp_flower_cmsg_get_data(skb); + memset(config, 0, sizeof(struct nfp_police_config)); + config->head.port = cpu_to_be32(netdev_port_id); + nfp_ctrl_tx(repr->app->ctrl, skb); + + return 0; +} + +void nfp_flower_stats_rlim_reply(struct nfp_app *app, struct sk_buff *skb) +{ + struct nfp_flower_priv *fl_priv = app->priv; + struct nfp_flower_repr_priv *repr_priv; + struct nfp_police_stats_reply *msg; + struct nfp_stat_pair *curr_stats; + struct nfp_stat_pair *prev_stats; + struct net_device *netdev; + struct nfp_repr *repr; + u32 netdev_port_id; + + msg = nfp_flower_cmsg_get_data(skb); + netdev_port_id = be32_to_cpu(msg->head.port); + rcu_read_lock(); + netdev = nfp_app_dev_get(app, netdev_port_id, NULL); + if (!netdev) + goto exit_unlock_rcu; + + repr = netdev_priv(netdev); + repr_priv = repr->app_priv; + curr_stats = &repr_priv->qos_table.curr_stats; + prev_stats = &repr_priv->qos_table.prev_stats; + + spin_lock_bh(&fl_priv->qos_stats_lock); + curr_stats->pkts = be64_to_cpu(msg->pass_pkts) + + be64_to_cpu(msg->drop_pkts); + curr_stats->bytes = be64_to_cpu(msg->pass_bytes) + + be64_to_cpu(msg->drop_bytes); + + if (!repr_priv->qos_table.last_update) { + prev_stats->pkts = curr_stats->pkts; + prev_stats->bytes = curr_stats->bytes; + } + + repr_priv->qos_table.last_update = jiffies; + spin_unlock_bh(&fl_priv->qos_stats_lock); + +exit_unlock_rcu: + rcu_read_unlock(); +} + +static void +nfp_flower_stats_rlim_request(struct nfp_flower_priv *fl_priv, + u32 netdev_port_id) +{ + struct nfp_police_cfg_head *head; + struct sk_buff *skb; + + skb = nfp_flower_cmsg_alloc(fl_priv->app, + sizeof(struct nfp_police_cfg_head), + NFP_FLOWER_CMSG_TYPE_QOS_STATS, + GFP_ATOMIC); + if (!skb) + return; + + head = nfp_flower_cmsg_get_data(skb); + memset(head, 0, sizeof(struct nfp_police_cfg_head)); + head->port = cpu_to_be32(netdev_port_id); + + nfp_ctrl_tx(fl_priv->app->ctrl, skb); +} + +static void +nfp_flower_stats_rlim_request_all(struct nfp_flower_priv *fl_priv) +{ + struct nfp_reprs *repr_set; + int i; + + rcu_read_lock(); + repr_set = rcu_dereference(fl_priv->app->reprs[NFP_REPR_TYPE_VF]); + if (!repr_set) + goto exit_unlock_rcu; + + for (i = 0; i < repr_set->num_reprs; i++) { + struct net_device *netdev; + + netdev = rcu_dereference(repr_set->reprs[i]); + if (netdev) { + struct nfp_repr *priv = netdev_priv(netdev); + struct nfp_flower_repr_priv *repr_priv; + u32 netdev_port_id; + + repr_priv = priv->app_priv; + netdev_port_id = repr_priv->qos_table.netdev_port_id; + if (!netdev_port_id) + continue; + + nfp_flower_stats_rlim_request(fl_priv, netdev_port_id); + } + } + +exit_unlock_rcu: + rcu_read_unlock(); +} + +static void update_stats_cache(struct work_struct *work) +{ + struct delayed_work *delayed_work; + struct nfp_flower_priv *fl_priv; + + delayed_work = to_delayed_work(work); + fl_priv = container_of(delayed_work, struct nfp_flower_priv, + qos_stats_work); + + nfp_flower_stats_rlim_request_all(fl_priv); + schedule_delayed_work(&fl_priv->qos_stats_work, NFP_FL_QOS_UPDATE); +} + +static int +nfp_flower_stats_rate_limiter(struct nfp_app *app, struct net_device *netdev, + struct tc_cls_matchall_offload *flow, + struct netlink_ext_ack *extack) +{ + struct nfp_flower_priv *fl_priv = app->priv; + struct nfp_flower_repr_priv *repr_priv; + struct nfp_stat_pair *curr_stats; + struct nfp_stat_pair *prev_stats; + u64 diff_bytes, diff_pkts; + struct nfp_repr *repr; + + if (!nfp_netdev_is_nfp_repr(netdev)) { + NL_SET_ERR_MSG_MOD(extack, "unsupported offload: qos rate limit offload not supported on higher level port"); + return -EOPNOTSUPP; + } + repr = netdev_priv(netdev); + + repr_priv = repr->app_priv; + if (!repr_priv->qos_table.netdev_port_id) { + NL_SET_ERR_MSG_MOD(extack, "unsupported offload: cannot find qos entry for stats update"); + return -EOPNOTSUPP; + } + + spin_lock_bh(&fl_priv->qos_stats_lock); + curr_stats = &repr_priv->qos_table.curr_stats; + prev_stats = &repr_priv->qos_table.prev_stats; + diff_pkts = curr_stats->pkts - prev_stats->pkts; + diff_bytes = curr_stats->bytes - prev_stats->bytes; + prev_stats->pkts = curr_stats->pkts; + prev_stats->bytes = curr_stats->bytes; + spin_unlock_bh(&fl_priv->qos_stats_lock); + + flow_stats_update(&flow->stats, diff_bytes, diff_pkts, + repr_priv->qos_table.last_update); + return 0; +} + +void nfp_flower_qos_init(struct nfp_app *app) +{ + struct nfp_flower_priv *fl_priv = app->priv; + + spin_lock_init(&fl_priv->qos_stats_lock); + INIT_DELAYED_WORK(&fl_priv->qos_stats_work, &update_stats_cache); +} + +void nfp_flower_qos_cleanup(struct nfp_app *app) +{ + struct nfp_flower_priv *fl_priv = app->priv; + + cancel_delayed_work_sync(&fl_priv->qos_stats_work); +} + +int nfp_flower_setup_qos_offload(struct nfp_app *app, struct net_device *netdev, + struct tc_cls_matchall_offload *flow) +{ + struct netlink_ext_ack *extack = flow->common.extack; + struct nfp_flower_priv *fl_priv = app->priv; + + if (!(fl_priv->flower_ext_feats & NFP_FL_FEATS_VF_RLIM)) { + NL_SET_ERR_MSG_MOD(extack, "unsupported offload: loaded firmware does not support qos rate limit offload"); + return -EOPNOTSUPP; + } + + switch (flow->command) { + case TC_CLSMATCHALL_REPLACE: + return nfp_flower_install_rate_limiter(app, netdev, flow, + extack); + case TC_CLSMATCHALL_DESTROY: + return nfp_flower_remove_rate_limiter(app, netdev, flow, + extack); + case TC_CLSMATCHALL_STATS: + return nfp_flower_stats_rate_limiter(app, netdev, flow, + extack); + default: + return -EOPNOTSUPP; + } +} diff --git a/drivers/net/ethernet/netronome/nfp/flower/tunnel_conf.c b/drivers/net/ethernet/netronome/nfp/flower/tunnel_conf.c index 4d78be4ec4e99b85a77f983da8b20041852ad11a..faa06edf95acc106a7c30c06de13d71e6267a584 100644 --- a/drivers/net/ethernet/netronome/nfp/flower/tunnel_conf.c +++ b/drivers/net/ethernet/netronome/nfp/flower/tunnel_conf.c @@ -171,7 +171,7 @@ void nfp_tunnel_keep_alive(struct nfp_app *app, struct sk_buff *skb) for (i = 0; i < count; i++) { ipv4_addr = payload->tun_info[i].ipv4; port = be32_to_cpu(payload->tun_info[i].egress_port); - netdev = nfp_app_repr_get(app, port); + netdev = nfp_app_dev_get(app, port, NULL); if (!netdev) continue; @@ -270,9 +270,10 @@ nfp_tun_write_neigh(struct net_device *netdev, struct nfp_app *app, struct flowi4 *flow, struct neighbour *neigh, gfp_t flag) { struct nfp_tun_neigh payload; + u32 port_id; - /* Only offload representor IPv4s for now. */ - if (!nfp_netdev_is_nfp_repr(netdev)) + port_id = nfp_flower_get_port_id_from_netdev(app, netdev); + if (!port_id) return; memset(&payload, 0, sizeof(struct nfp_tun_neigh)); @@ -290,7 +291,7 @@ nfp_tun_write_neigh(struct net_device *netdev, struct nfp_app *app, payload.src_ipv4 = flow->saddr; ether_addr_copy(payload.src_addr, netdev->dev_addr); neigh_ha_snapshot(payload.dst_addr, neigh, netdev); - payload.port_id = cpu_to_be32(nfp_repr_get_port_id(netdev)); + payload.port_id = cpu_to_be32(port_id); /* Add destination of new route to NFP cache. */ nfp_tun_add_route_to_cache(app, payload.dst_ipv4); @@ -366,7 +367,7 @@ void nfp_tunnel_request_route(struct nfp_app *app, struct sk_buff *skb) payload = nfp_flower_cmsg_get_data(skb); - netdev = nfp_app_repr_get(app, be32_to_cpu(payload->ingress_port)); + netdev = nfp_app_dev_get(app, be32_to_cpu(payload->ingress_port), NULL); if (!netdev) goto route_fail_warning; diff --git a/drivers/net/ethernet/netronome/nfp/nfp_app.h b/drivers/net/ethernet/netronome/nfp/nfp_app.h index f8d42271370509440439c483585c509e07616200..76d13af46a7a313d75d6e2b15ae1dbe57bf8b171 100644 --- a/drivers/net/ethernet/netronome/nfp/nfp_app.h +++ b/drivers/net/ethernet/netronome/nfp/nfp_app.h @@ -79,7 +79,7 @@ extern const struct nfp_app_type app_abm; * @eswitch_mode_set: set SR-IOV eswitch mode (under pf->lock) * @sriov_enable: app-specific sriov initialisation * @sriov_disable: app-specific sriov clean-up - * @repr_get: get representor netdev + * @dev_get: get representor or internal port representing netdev */ struct nfp_app_type { enum nfp_app_id id; @@ -143,7 +143,8 @@ struct nfp_app_type { enum devlink_eswitch_mode (*eswitch_mode_get)(struct nfp_app *app); int (*eswitch_mode_set)(struct nfp_app *app, u16 mode); - struct net_device *(*repr_get)(struct nfp_app *app, u32 id); + struct net_device *(*dev_get)(struct nfp_app *app, u32 id, + bool *redir_egress); }; /** @@ -397,12 +398,14 @@ static inline void nfp_app_sriov_disable(struct nfp_app *app) app->type->sriov_disable(app); } -static inline struct net_device *nfp_app_repr_get(struct nfp_app *app, u32 id) +static inline +struct net_device *nfp_app_dev_get(struct nfp_app *app, u32 id, + bool *redir_egress) { - if (unlikely(!app || !app->type->repr_get)) + if (unlikely(!app || !app->type->dev_get)) return NULL; - return app->type->repr_get(app, id); + return app->type->dev_get(app, id, redir_egress); } struct nfp_app *nfp_app_from_netdev(struct net_device *netdev); @@ -433,6 +436,6 @@ int nfp_app_nic_vnic_alloc(struct nfp_app *app, struct nfp_net *nn, int nfp_app_nic_vnic_init_phy_port(struct nfp_pf *pf, struct nfp_app *app, struct nfp_net *nn, unsigned int id); -struct devlink *nfp_devlink_get_devlink(struct net_device *netdev); +struct devlink_port *nfp_devlink_get_devlink_port(struct net_device *netdev); #endif diff --git a/drivers/net/ethernet/netronome/nfp/nfp_devlink.c b/drivers/net/ethernet/netronome/nfp/nfp_devlink.c index e9eca99cf4937c6e0abbe18791ef772a6f40f31f..c50fce42f473022ec3286dc832c4c2402761fac2 100644 --- a/drivers/net/ethernet/netronome/nfp/nfp_devlink.c +++ b/drivers/net/ethernet/netronome/nfp/nfp_devlink.c @@ -144,7 +144,8 @@ nfp_devlink_sb_pool_get(struct devlink *devlink, unsigned int sb_index, static int nfp_devlink_sb_pool_set(struct devlink *devlink, unsigned int sb_index, u16 pool_index, - u32 size, enum devlink_sb_threshold_type threshold_type) + u32 size, enum devlink_sb_threshold_type threshold_type, + struct netlink_ext_ack *extack) { struct nfp_pf *pf = devlink_priv(devlink); @@ -354,6 +355,8 @@ int nfp_devlink_port_register(struct nfp_app *app, struct nfp_port *port) { struct nfp_eth_table_port eth_port; struct devlink *devlink; + const u8 *serial; + int serial_len; int ret; rtnl_lock(); @@ -362,10 +365,10 @@ int nfp_devlink_port_register(struct nfp_app *app, struct nfp_port *port) if (ret) return ret; - devlink_port_type_eth_set(&port->dl_port, port->netdev); + serial_len = nfp_cpp_serial(port->app->cpp, &serial); devlink_port_attrs_set(&port->dl_port, DEVLINK_PORT_FLAVOUR_PHYSICAL, eth_port.label_port, eth_port.is_split, - eth_port.label_subport); + eth_port.label_subport, serial, serial_len); devlink = priv_to_devlink(app->pf); @@ -377,13 +380,23 @@ void nfp_devlink_port_unregister(struct nfp_port *port) devlink_port_unregister(&port->dl_port); } -struct devlink *nfp_devlink_get_devlink(struct net_device *netdev) +void nfp_devlink_port_type_eth_set(struct nfp_port *port) +{ + devlink_port_type_eth_set(&port->dl_port, port->netdev); +} + +void nfp_devlink_port_type_clear(struct nfp_port *port) { - struct nfp_app *app; + devlink_port_type_clear(&port->dl_port); +} + +struct devlink_port *nfp_devlink_get_devlink_port(struct net_device *netdev) +{ + struct nfp_port *port; - app = nfp_app_from_netdev(netdev); - if (!app) + port = nfp_port_from_netdev(netdev); + if (!port) return NULL; - return priv_to_devlink(app->pf); + return &port->dl_port; } diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c index f4c8776e42b680ab29058f4bf168781b60b32059..948d1a4b46439b61f7d4cd7ba197ce2b14b04d85 100644 --- a/drivers/net/ethernet/netronome/nfp/nfp_main.c +++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c @@ -294,6 +294,9 @@ static int nfp_pcie_sriov_disable(struct pci_dev *pdev) static int nfp_pcie_sriov_configure(struct pci_dev *pdev, int num_vfs) { + if (!pci_get_drvdata(pdev)) + return -ENOENT; + if (num_vfs == 0) return nfp_pcie_sriov_disable(pdev); else @@ -720,9 +723,13 @@ err_pci_disable: return err; } -static void nfp_pci_remove(struct pci_dev *pdev) +static void __nfp_pci_shutdown(struct pci_dev *pdev, bool unload_fw) { - struct nfp_pf *pf = pci_get_drvdata(pdev); + struct nfp_pf *pf; + + pf = pci_get_drvdata(pdev); + if (!pf) + return; nfp_hwmon_unregister(pf); @@ -733,7 +740,7 @@ static void nfp_pci_remove(struct pci_dev *pdev) vfree(pf->dumpspec); kfree(pf->rtbl); nfp_mip_close(pf->mip); - if (pf->fw_loaded) + if (unload_fw && pf->fw_loaded) nfp_fw_unload(pf); destroy_workqueue(pf->wq); @@ -749,11 +756,22 @@ static void nfp_pci_remove(struct pci_dev *pdev) pci_disable_device(pdev); } +static void nfp_pci_remove(struct pci_dev *pdev) +{ + __nfp_pci_shutdown(pdev, true); +} + +static void nfp_pci_shutdown(struct pci_dev *pdev) +{ + __nfp_pci_shutdown(pdev, false); +} + static struct pci_driver nfp_pci_driver = { .name = nfp_driver_name, .id_table = nfp_pci_device_ids, .probe = nfp_pci_probe, .remove = nfp_pci_remove, + .shutdown = nfp_pci_shutdown, .sriov_configure = nfp_pcie_sriov_configure, }; diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net.h b/drivers/net/ethernet/netronome/nfp/nfp_net.h index be37c2d6151c43f705515314e555f6fc643ccd91..df9aff2684ed0a57ed9c70451b384e270433ce67 100644 --- a/drivers/net/ethernet/netronome/nfp/nfp_net.h +++ b/drivers/net/ethernet/netronome/nfp/nfp_net.h @@ -539,12 +539,17 @@ struct nfp_net_dp { * @shared_handler: Handler for shared interrupts * @shared_name: Name for shared interrupt * @me_freq_mhz: ME clock_freq (MHz) - * @reconfig_lock: Protects HW reconfiguration request regs/machinery + * @reconfig_lock: Protects @reconfig_posted, @reconfig_timer_active, + * @reconfig_sync_present and HW reconfiguration request + * regs/machinery from async requests (sync must take + * @bar_lock) * @reconfig_posted: Pending reconfig bits coming from async sources * @reconfig_timer_active: Timer for reading reconfiguration results is pending * @reconfig_sync_present: Some thread is performing synchronous reconfig * @reconfig_timer: Timer for async reading of reconfig results * @reconfig_in_progress_update: Update FW is processing now (debug only) + * @bar_lock: vNIC config BAR access lock, protects: update, + * mailbox area * @link_up: Is the link up? * @link_status_lock: Protects @link_* and ensures atomicity with BAR reading * @rx_coalesce_usecs: RX interrupt moderation usecs delay parameter @@ -615,6 +620,8 @@ struct nfp_net { struct timer_list reconfig_timer; u32 reconfig_in_progress_update; + struct mutex bar_lock; + u32 rx_coalesce_usecs; u32 rx_coalesce_max_frames; u32 tx_coalesce_usecs; @@ -839,6 +846,16 @@ static inline void nfp_ctrl_unlock(struct nfp_net *nn) spin_unlock_bh(&nn->r_vecs[0].lock); } +static inline void nn_ctrl_bar_lock(struct nfp_net *nn) +{ + mutex_lock(&nn->bar_lock); +} + +static inline void nn_ctrl_bar_unlock(struct nfp_net *nn) +{ + mutex_unlock(&nn->bar_lock); +} + /* Globals */ extern const char nfp_driver_version[]; @@ -871,7 +888,9 @@ unsigned int nfp_net_rss_key_sz(struct nfp_net *nn); void nfp_net_rss_write_itbl(struct nfp_net *nn); void nfp_net_rss_write_key(struct nfp_net *nn); void nfp_net_coalesce_write_cfg(struct nfp_net *nn); -int nfp_net_reconfig_mbox(struct nfp_net *nn, u32 mbox_cmd); +int nfp_net_mbox_lock(struct nfp_net *nn, unsigned int data_size); +int nfp_net_mbox_reconfig(struct nfp_net *nn, u32 mbox_cmd); +int nfp_net_mbox_reconfig_and_unlock(struct nfp_net *nn, u32 mbox_cmd); unsigned int nfp_net_irqs_alloc(struct pci_dev *pdev, struct msix_entry *irq_entries, diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c index 6d1b8816552ea5cc56505b3d4d396396e7894300..b82b684f52ce6d3de7f2fc3462955af2267f3a4b 100644 --- a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c +++ b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -137,20 +138,37 @@ static bool nfp_net_reconfig_check_done(struct nfp_net *nn, bool last_check) return false; } -static int nfp_net_reconfig_wait(struct nfp_net *nn, unsigned long deadline) +static bool __nfp_net_reconfig_wait(struct nfp_net *nn, unsigned long deadline) { bool timed_out = false; + int i; + + /* Poll update field, waiting for NFP to ack the config. + * Do an opportunistic wait-busy loop, afterward sleep. + */ + for (i = 0; i < 50; i++) { + if (nfp_net_reconfig_check_done(nn, false)) + return false; + udelay(4); + } - /* Poll update field, waiting for NFP to ack the config */ while (!nfp_net_reconfig_check_done(nn, timed_out)) { - msleep(1); + usleep_range(250, 500); timed_out = time_is_before_eq_jiffies(deadline); } + return timed_out; +} + +static int nfp_net_reconfig_wait(struct nfp_net *nn, unsigned long deadline) +{ + if (__nfp_net_reconfig_wait(nn, deadline)) + return -EIO; + if (nn_readl(nn, NFP_NET_CFG_UPDATE) & NFP_NET_CFG_UPDATE_ERR) return -EIO; - return timed_out ? -EIO : 0; + return 0; } static void nfp_net_reconfig_timer(struct timer_list *t) @@ -243,7 +261,7 @@ static void nfp_net_reconfig_wait_posted(struct nfp_net *nn) } /** - * nfp_net_reconfig() - Reconfigure the firmware + * __nfp_net_reconfig() - Reconfigure the firmware * @nn: NFP Net device to reconfigure * @update: The value for the update field in the BAR config * @@ -253,10 +271,12 @@ static void nfp_net_reconfig_wait_posted(struct nfp_net *nn) * * Return: Negative errno on error, 0 on success */ -int nfp_net_reconfig(struct nfp_net *nn, u32 update) +static int __nfp_net_reconfig(struct nfp_net *nn, u32 update) { int ret; + lockdep_assert_held(&nn->bar_lock); + nfp_net_reconfig_sync_enter(nn); nfp_net_reconfig_start(nn, update); @@ -274,8 +294,31 @@ int nfp_net_reconfig(struct nfp_net *nn, u32 update) return ret; } +int nfp_net_reconfig(struct nfp_net *nn, u32 update) +{ + int ret; + + nn_ctrl_bar_lock(nn); + ret = __nfp_net_reconfig(nn, update); + nn_ctrl_bar_unlock(nn); + + return ret; +} + +int nfp_net_mbox_lock(struct nfp_net *nn, unsigned int data_size) +{ + if (nn->tlv_caps.mbox_len < NFP_NET_CFG_MBOX_SIMPLE_VAL + data_size) { + nn_err(nn, "mailbox too small for %u of data (%u)\n", + data_size, nn->tlv_caps.mbox_len); + return -EIO; + } + + nn_ctrl_bar_lock(nn); + return 0; +} + /** - * nfp_net_reconfig_mbox() - Reconfigure the firmware via the mailbox + * nfp_net_mbox_reconfig() - Reconfigure the firmware via the mailbox * @nn: NFP Net device to reconfigure * @mbox_cmd: The value for the mailbox command * @@ -283,19 +326,15 @@ int nfp_net_reconfig(struct nfp_net *nn, u32 update) * * Return: Negative errno on error, 0 on success */ -int nfp_net_reconfig_mbox(struct nfp_net *nn, u32 mbox_cmd) +int nfp_net_mbox_reconfig(struct nfp_net *nn, u32 mbox_cmd) { u32 mbox = nn->tlv_caps.mbox_off; int ret; - if (!nfp_net_has_mbox(&nn->tlv_caps)) { - nn_err(nn, "no mailbox present, command: %u\n", mbox_cmd); - return -EIO; - } - + lockdep_assert_held(&nn->bar_lock); nn_writeq(nn, mbox + NFP_NET_CFG_MBOX_SIMPLE_CMD, mbox_cmd); - ret = nfp_net_reconfig(nn, NFP_NET_CFG_UPDATE_MBOX); + ret = __nfp_net_reconfig(nn, NFP_NET_CFG_UPDATE_MBOX); if (ret) { nn_err(nn, "Mailbox update error\n"); return ret; @@ -304,6 +343,15 @@ int nfp_net_reconfig_mbox(struct nfp_net *nn, u32 mbox_cmd) return -nn_readl(nn, mbox + NFP_NET_CFG_MBOX_SIMPLE_RET); } +int nfp_net_mbox_reconfig_and_unlock(struct nfp_net *nn, u32 mbox_cmd) +{ + int ret; + + ret = nfp_net_mbox_reconfig(nn, mbox_cmd); + nn_ctrl_bar_unlock(nn); + return ret; +} + /* Interrupt configuration and handling */ @@ -909,7 +957,7 @@ static int nfp_net_tx(struct sk_buff *skb, struct net_device *netdev) nfp_net_tx_ring_stop(nd_q, tx_ring); tx_ring->wr_ptr_add += nr_frags + 1; - if (__netdev_tx_sent_queue(nd_q, txbuf->real_len, skb->xmit_more)) + if (__netdev_tx_sent_queue(nd_q, txbuf->real_len, netdev_xmit_more())) nfp_net_tx_xmit_more_flush(tx_ring); return NETDEV_TX_OK; @@ -1635,6 +1683,7 @@ static int nfp_net_rx(struct nfp_net_rx_ring *rx_ring, int budget) struct nfp_net_rx_buf *rxbuf; struct nfp_net_rx_desc *rxd; struct nfp_meta_parsed meta; + bool redir_egress = false; struct net_device *netdev; dma_addr_t new_dma_addr; u32 meta_len_xdp = 0; @@ -1770,13 +1819,16 @@ static int nfp_net_rx(struct nfp_net_rx_ring *rx_ring, int budget) struct nfp_net *nn; nn = netdev_priv(dp->netdev); - netdev = nfp_app_repr_get(nn->app, meta.portid); + netdev = nfp_app_dev_get(nn->app, meta.portid, + &redir_egress); if (unlikely(!netdev)) { nfp_net_rx_drop(dp, r_vec, rx_ring, rxbuf, NULL); continue; } - nfp_repr_inc_rx_stats(netdev, pkt_len); + + if (nfp_netdev_is_nfp_repr(netdev)) + nfp_repr_inc_rx_stats(netdev, pkt_len); } skb = build_skb(rxbuf->frag, true_bufsz); @@ -1811,7 +1863,13 @@ static int nfp_net_rx(struct nfp_net_rx_ring *rx_ring, int budget) if (meta_len_xdp) skb_metadata_set(skb, meta_len_xdp); - napi_gro_receive(&rx_ring->r_vec->napi, skb); + if (likely(!redir_egress)) { + napi_gro_receive(&rx_ring->r_vec->napi, skb); + } else { + skb->dev = netdev; + __skb_push(skb, ETH_HLEN); + dev_queue_xmit(skb); + } } if (xdp_prog) { @@ -3111,7 +3169,9 @@ static int nfp_net_change_mtu(struct net_device *netdev, int new_mtu) static int nfp_net_vlan_rx_add_vid(struct net_device *netdev, __be16 proto, u16 vid) { + const u32 cmd = NFP_NET_CFG_MBOX_CMD_CTAG_FILTER_ADD; struct nfp_net *nn = netdev_priv(netdev); + int err; /* Priority tagged packets with vlan id 0 are processed by the * NFP as untagged packets @@ -3119,17 +3179,23 @@ nfp_net_vlan_rx_add_vid(struct net_device *netdev, __be16 proto, u16 vid) if (!vid) return 0; + err = nfp_net_mbox_lock(nn, NFP_NET_CFG_VLAN_FILTER_SZ); + if (err) + return err; + nn_writew(nn, nn->tlv_caps.mbox_off + NFP_NET_CFG_VLAN_FILTER_VID, vid); nn_writew(nn, nn->tlv_caps.mbox_off + NFP_NET_CFG_VLAN_FILTER_PROTO, ETH_P_8021Q); - return nfp_net_reconfig_mbox(nn, NFP_NET_CFG_MBOX_CMD_CTAG_FILTER_ADD); + return nfp_net_mbox_reconfig_and_unlock(nn, cmd); } static int nfp_net_vlan_rx_kill_vid(struct net_device *netdev, __be16 proto, u16 vid) { + const u32 cmd = NFP_NET_CFG_MBOX_CMD_CTAG_FILTER_KILL; struct nfp_net *nn = netdev_priv(netdev); + int err; /* Priority tagged packets with vlan id 0 are processed by the * NFP as untagged packets @@ -3137,11 +3203,15 @@ nfp_net_vlan_rx_kill_vid(struct net_device *netdev, __be16 proto, u16 vid) if (!vid) return 0; + err = nfp_net_mbox_lock(nn, NFP_NET_CFG_VLAN_FILTER_SZ); + if (err) + return err; + nn_writew(nn, nn->tlv_caps.mbox_off + NFP_NET_CFG_VLAN_FILTER_VID, vid); nn_writew(nn, nn->tlv_caps.mbox_off + NFP_NET_CFG_VLAN_FILTER_PROTO, ETH_P_8021Q); - return nfp_net_reconfig_mbox(nn, NFP_NET_CFG_MBOX_CMD_CTAG_FILTER_KILL); + return nfp_net_mbox_reconfig_and_unlock(nn, cmd); } static void nfp_net_stat64(struct net_device *netdev, @@ -3324,8 +3394,11 @@ nfp_net_get_phys_port_name(struct net_device *netdev, char *name, size_t len) struct nfp_net *nn = netdev_priv(netdev); int n; + /* If port is defined, devlink_port is registered and devlink core + * is taking care of name formatting. + */ if (nn->port) - return nfp_port_get_phys_port_name(netdev, name, len); + return -EOPNOTSUPP; if (nn->dp.is_vf || nn->vnic_no_name) return -EOPNOTSUPP; @@ -3517,6 +3590,7 @@ const struct net_device_ops nfp_net_netdev_ops = { .ndo_set_vf_mac = nfp_app_set_vf_mac, .ndo_set_vf_vlan = nfp_app_set_vf_vlan, .ndo_set_vf_spoofchk = nfp_app_set_vf_spoofchk, + .ndo_set_vf_trust = nfp_app_set_vf_trust, .ndo_get_vf_config = nfp_app_get_vf_config, .ndo_set_vf_link_state = nfp_app_set_vf_link_state, .ndo_setup_tc = nfp_port_setup_tc, @@ -3530,8 +3604,7 @@ const struct net_device_ops nfp_net_netdev_ops = { .ndo_udp_tunnel_add = nfp_net_add_vxlan_port, .ndo_udp_tunnel_del = nfp_net_del_vxlan_port, .ndo_bpf = nfp_net_xdp, - .ndo_get_port_parent_id = nfp_port_get_port_parent_id, - .ndo_get_devlink = nfp_devlink_get_devlink, + .ndo_get_devlink_port = nfp_devlink_get_devlink_port, }; /** @@ -3548,7 +3621,7 @@ void nfp_net_info(struct nfp_net *nn) nn->fw_ver.resv, nn->fw_ver.class, nn->fw_ver.major, nn->fw_ver.minor, nn->max_mtu); - nn_info(nn, "CAP: %#x %s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s\n", + nn_info(nn, "CAP: %#x %s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s\n", nn->cap, nn->cap & NFP_NET_CFG_CTRL_PROMISC ? "PROMISC " : "", nn->cap & NFP_NET_CFG_CTRL_L2BC ? "L2BCFILT " : "", @@ -3564,7 +3637,6 @@ void nfp_net_info(struct nfp_net *nn) nn->cap & NFP_NET_CFG_CTRL_RSS ? "RSS1 " : "", nn->cap & NFP_NET_CFG_CTRL_RSS2 ? "RSS2 " : "", nn->cap & NFP_NET_CFG_CTRL_CTAG_FILTER ? "CTAG_FILTER " : "", - nn->cap & NFP_NET_CFG_CTRL_L2SWITCH ? "L2SWITCH " : "", nn->cap & NFP_NET_CFG_CTRL_MSIXAUTO ? "AUTOMASK " : "", nn->cap & NFP_NET_CFG_CTRL_IRQMOD ? "IRQMOD " : "", nn->cap & NFP_NET_CFG_CTRL_VXLAN ? "VXLAN " : "", @@ -3632,6 +3704,8 @@ nfp_net_alloc(struct pci_dev *pdev, void __iomem *ctrl_bar, bool needs_netdev, nn->dp.txd_cnt = NFP_NET_TX_DESCS_DEFAULT; nn->dp.rxd_cnt = NFP_NET_RX_DESCS_DEFAULT; + mutex_init(&nn->bar_lock); + spin_lock_init(&nn->reconfig_lock); spin_lock_init(&nn->link_status_lock); @@ -3659,6 +3733,9 @@ err_free_nn: void nfp_net_free(struct nfp_net *nn) { WARN_ON(timer_pending(&nn->reconfig_timer) || nn->reconfig_posted); + + mutex_destroy(&nn->bar_lock); + if (nn->dp.netdev) free_netdev(nn->dp.netdev); else @@ -3920,9 +3997,6 @@ int nfp_net_init(struct nfp_net *nn) nn->dp.ctrl |= NFP_NET_CFG_CTRL_IRQMOD; } - if (nn->dp.netdev) - nfp_net_netdev_init(nn); - /* Stash the re-configuration queue away. First odd queue in TX Bar */ nn->qcp_cfg = nn->tx_bar + NFP_QCP_QUEUE_ADDR_SZ; @@ -3935,6 +4009,9 @@ int nfp_net_init(struct nfp_net *nn) if (err) return err; + if (nn->dp.netdev) + nfp_net_netdev_init(nn); + nfp_net_vecs_init(nn); if (!nn->dp.netdev) diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h b/drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h index 372adea10e14e34179437ee23c5ef3ce51dac617..25919e3380718ccd10a961a49c731a64ef8d0211 100644 --- a/drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h +++ b/drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h @@ -104,8 +104,6 @@ #define NFP_NET_CFG_CTRL_RINGPRIO (0x1 << 19) /* Ring priorities */ #define NFP_NET_CFG_CTRL_MSIXAUTO (0x1 << 20) /* MSI-X auto-masking */ #define NFP_NET_CFG_CTRL_TXRWB (0x1 << 21) /* Write-back of TX ring*/ -#define NFP_NET_CFG_CTRL_L2SWITCH (0x1 << 22) /* L2 Switch */ -#define NFP_NET_CFG_CTRL_L2SWITCH_LOCAL (0x1 << 23) /* Switch to local */ #define NFP_NET_CFG_CTRL_VXLAN (0x1 << 24) /* VXLAN tunnel support */ #define NFP_NET_CFG_CTRL_NVGRE (0x1 << 25) /* NVGRE tunnel support */ #define NFP_NET_CFG_CTRL_BPF (0x1 << 27) /* BPF offload capable */ @@ -130,7 +128,6 @@ #define NFP_NET_CFG_UPDATE_TXRPRIO (0x1 << 3) /* TX Ring prio change */ #define NFP_NET_CFG_UPDATE_RXRPRIO (0x1 << 4) /* RX Ring prio change */ #define NFP_NET_CFG_UPDATE_MSIX (0x1 << 5) /* MSI-X change */ -#define NFP_NET_CFG_UPDATE_L2SWITCH (0x1 << 6) /* Switch changes */ #define NFP_NET_CFG_UPDATE_RESET (0x1 << 7) /* Update due to FLR */ #define NFP_NET_CFG_UPDATE_IRQMOD (0x1 << 8) /* IRQ mod change */ #define NFP_NET_CFG_UPDATE_VXLAN (0x1 << 9) /* VXLAN port change */ @@ -392,7 +389,6 @@ #define NFP_NET_CFG_MBOX_SIMPLE_CMD 0x0 #define NFP_NET_CFG_MBOX_SIMPLE_RET 0x4 #define NFP_NET_CFG_MBOX_SIMPLE_VAL 0x8 -#define NFP_NET_CFG_MBOX_SIMPLE_LEN 12 #define NFP_NET_CFG_MBOX_CMD_CTAG_FILTER_ADD 1 #define NFP_NET_CFG_MBOX_CMD_CTAG_FILTER_KILL 2 @@ -498,10 +494,4 @@ struct nfp_net_tlv_caps { int nfp_net_tlv_caps_parse(struct device *dev, u8 __iomem *ctrl_mem, struct nfp_net_tlv_caps *caps); - -static inline bool nfp_net_has_mbox(struct nfp_net_tlv_caps *caps) -{ - return caps->mbox_len >= NFP_NET_CFG_MBOX_SIMPLE_LEN; -} - #endif /* _NFP_NET_CTRL_H_ */ diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_ethtool.c b/drivers/net/ethernet/netronome/nfp/nfp_net_ethtool.c index 690b62718dbb3110a8cad67fa354336a729223cf..851e31e0ba8e01354707835b326d91aa31414924 100644 --- a/drivers/net/ethernet/netronome/nfp/nfp_net_ethtool.c +++ b/drivers/net/ethernet/netronome/nfp/nfp_net_ethtool.c @@ -18,6 +18,7 @@ #include #include #include +#include #include "nfpcore/nfp.h" #include "nfpcore/nfp_nsp.h" @@ -152,6 +153,8 @@ static const struct nfp_et_stat nfp_mac_et_stats[] = { #define NN_RVEC_GATHER_STATS 9 #define NN_RVEC_PER_Q_STATS 3 +#define SFP_SFF_REV_COMPLIANCE 1 + static void nfp_net_get_nspinfo(struct nfp_app *app, char *version) { struct nfp_nsp *nsp; @@ -1096,6 +1099,130 @@ nfp_app_get_dump_data(struct net_device *netdev, struct ethtool_dump *dump, buffer); } +static int +nfp_port_get_module_info(struct net_device *netdev, + struct ethtool_modinfo *modinfo) +{ + struct nfp_eth_table_port *eth_port; + struct nfp_port *port; + unsigned int read_len; + struct nfp_nsp *nsp; + int err = 0; + u8 data; + + port = nfp_port_from_netdev(netdev); + eth_port = nfp_port_get_eth_port(port); + if (!eth_port) + return -EOPNOTSUPP; + + nsp = nfp_nsp_open(port->app->cpp); + if (IS_ERR(nsp)) { + err = PTR_ERR(nsp); + netdev_err(netdev, "Failed to access the NSP: %d\n", err); + return err; + } + + if (!nfp_nsp_has_read_module_eeprom(nsp)) { + netdev_info(netdev, "reading module EEPROM not supported. Please update flash\n"); + err = -EOPNOTSUPP; + goto exit_close_nsp; + } + + switch (eth_port->interface) { + case NFP_INTERFACE_SFP: + case NFP_INTERFACE_SFP28: + err = nfp_nsp_read_module_eeprom(nsp, eth_port->eth_index, + SFP_SFF8472_COMPLIANCE, &data, + 1, &read_len); + if (err < 0) + goto exit_close_nsp; + + if (!data) { + modinfo->type = ETH_MODULE_SFF_8079; + modinfo->eeprom_len = ETH_MODULE_SFF_8079_LEN; + } else { + modinfo->type = ETH_MODULE_SFF_8472; + modinfo->eeprom_len = ETH_MODULE_SFF_8472_LEN; + } + break; + case NFP_INTERFACE_QSFP: + err = nfp_nsp_read_module_eeprom(nsp, eth_port->eth_index, + SFP_SFF_REV_COMPLIANCE, &data, + 1, &read_len); + if (err < 0) + goto exit_close_nsp; + + if (data < 0x3) { + modinfo->type = ETH_MODULE_SFF_8436; + modinfo->eeprom_len = ETH_MODULE_SFF_8436_LEN; + } else { + modinfo->type = ETH_MODULE_SFF_8636; + modinfo->eeprom_len = ETH_MODULE_SFF_8636_LEN; + } + break; + case NFP_INTERFACE_QSFP28: + modinfo->type = ETH_MODULE_SFF_8636; + modinfo->eeprom_len = ETH_MODULE_SFF_8636_LEN; + break; + default: + netdev_err(netdev, "Unsupported module 0x%x detected\n", + eth_port->interface); + err = -EINVAL; + } + +exit_close_nsp: + nfp_nsp_close(nsp); + return err; +} + +static int +nfp_port_get_module_eeprom(struct net_device *netdev, + struct ethtool_eeprom *eeprom, u8 *data) +{ + struct nfp_eth_table_port *eth_port; + struct nfp_port *port; + struct nfp_nsp *nsp; + int err; + + port = nfp_port_from_netdev(netdev); + eth_port = __nfp_port_get_eth_port(port); + if (!eth_port) + return -EOPNOTSUPP; + + nsp = nfp_nsp_open(port->app->cpp); + if (IS_ERR(nsp)) { + err = PTR_ERR(nsp); + netdev_err(netdev, "Failed to access the NSP: %d\n", err); + return err; + } + + if (!nfp_nsp_has_read_module_eeprom(nsp)) { + netdev_info(netdev, "reading module EEPROM not supported. Please update flash\n"); + err = -EOPNOTSUPP; + goto exit_close_nsp; + } + + err = nfp_nsp_read_module_eeprom(nsp, eth_port->eth_index, + eeprom->offset, data, eeprom->len, + &eeprom->len); + if (err < 0) { + if (eeprom->len) { + netdev_warn(netdev, + "Incomplete read from module EEPROM: %d\n", + err); + err = 0; + } else { + netdev_err(netdev, + "Reading from module EEPROM failed: %d\n", + err); + } + } + +exit_close_nsp: + nfp_nsp_close(nsp); + return err; +} + static int nfp_net_set_coalesce(struct net_device *netdev, struct ethtool_coalesce *ec) { @@ -1253,6 +1380,8 @@ static const struct ethtool_ops nfp_net_ethtool_ops = { .set_dump = nfp_app_set_dump, .get_dump_flag = nfp_app_get_dump_flag, .get_dump_data = nfp_app_get_dump_data, + .get_module_info = nfp_port_get_module_info, + .get_module_eeprom = nfp_port_get_module_eeprom, .get_coalesce = nfp_net_get_coalesce, .set_coalesce = nfp_net_set_coalesce, .get_channels = nfp_net_get_channels, @@ -1272,6 +1401,8 @@ const struct ethtool_ops nfp_port_ethtool_ops = { .set_dump = nfp_app_set_dump, .get_dump_flag = nfp_app_get_dump_flag, .get_dump_data = nfp_app_get_dump_data, + .get_module_info = nfp_port_get_module_info, + .get_module_eeprom = nfp_port_get_module_eeprom, .get_link_ksettings = nfp_net_get_link_ksettings, .set_link_ksettings = nfp_net_set_link_ksettings, .get_fecparam = nfp_port_get_fecparam, diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c index 08f5fdbd8e41a6480637a906ee0052e19c36d4cc..986464d4a20697e997cc18d9a7fe6dcb75ded347 100644 --- a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c +++ b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c @@ -150,34 +150,39 @@ nfp_net_pf_init_vnic(struct nfp_pf *pf, struct nfp_net *nn, unsigned int id) nn->id = id; + if (nn->port) { + err = nfp_devlink_port_register(pf->app, nn->port); + if (err) + return err; + } + err = nfp_net_init(nn); if (err) - return err; + goto err_devlink_port_clean; nfp_net_debugfs_vnic_add(nn, pf->ddir); - if (nn->port) { - err = nfp_devlink_port_register(pf->app, nn->port); - if (err) - goto err_dfs_clean; - } + if (nn->port) + nfp_devlink_port_type_eth_set(nn->port); nfp_net_info(nn); if (nfp_net_is_data_vnic(nn)) { err = nfp_app_vnic_init(pf->app, nn); if (err) - goto err_devlink_port_clean; + goto err_devlink_port_type_clean; } return 0; -err_devlink_port_clean: +err_devlink_port_type_clean: if (nn->port) - nfp_devlink_port_unregister(nn->port); -err_dfs_clean: + nfp_devlink_port_type_clear(nn->port); nfp_net_debugfs_dir_clean(&nn->debugfs_dir); nfp_net_clean(nn); +err_devlink_port_clean: + if (nn->port) + nfp_devlink_port_unregister(nn->port); return err; } @@ -221,9 +226,11 @@ static void nfp_net_pf_clean_vnic(struct nfp_pf *pf, struct nfp_net *nn) if (nfp_net_is_data_vnic(nn)) nfp_app_vnic_clean(pf->app, nn); if (nn->port) - nfp_devlink_port_unregister(nn->port); + nfp_devlink_port_type_clear(nn->port); nfp_net_debugfs_dir_clean(&nn->debugfs_dir); nfp_net_clean(nn); + if (nn->port) + nfp_devlink_port_unregister(nn->port); } static int nfp_net_pf_alloc_irqs(struct nfp_pf *pf) diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c b/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c index 94d228c044963b8494c737aa1c46700d3598fd11..036edcc1fa18142cc1b0ebc9a3c34ca98f5fd75b 100644 --- a/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c +++ b/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c @@ -267,13 +267,13 @@ const struct net_device_ops nfp_repr_netdev_ops = { .ndo_set_vf_mac = nfp_app_set_vf_mac, .ndo_set_vf_vlan = nfp_app_set_vf_vlan, .ndo_set_vf_spoofchk = nfp_app_set_vf_spoofchk, + .ndo_set_vf_trust = nfp_app_set_vf_trust, .ndo_get_vf_config = nfp_app_get_vf_config, .ndo_set_vf_link_state = nfp_app_set_vf_link_state, .ndo_fix_features = nfp_repr_fix_features, .ndo_set_features = nfp_port_set_features, .ndo_set_mac_address = eth_mac_addr, - .ndo_get_port_parent_id = nfp_port_get_port_parent_id, - .ndo_get_devlink = nfp_devlink_get_devlink, + .ndo_get_devlink_port = nfp_devlink_get_devlink_port, }; void diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c b/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c index b6ec46ed0540769215e80c9c4abacf5d24444079..3fdaaf8ed2ba494dc272ae2b452a70bfa39422e7 100644 --- a/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c +++ b/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c @@ -1,5 +1,5 @@ // SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) -/* Copyright (C) 2017 Netronome Systems, Inc. */ +/* Copyright (C) 2017-2019 Netronome Systems, Inc. */ #include #include @@ -146,6 +146,30 @@ int nfp_app_set_vf_spoofchk(struct net_device *netdev, int vf, bool enable) "spoofchk"); } +int nfp_app_set_vf_trust(struct net_device *netdev, int vf, bool enable) +{ + struct nfp_app *app = nfp_app_from_netdev(netdev); + unsigned int vf_offset; + u8 vf_ctrl; + int err; + + err = nfp_net_sriov_check(app, vf, NFP_NET_VF_CFG_MB_CAP_TRUST, + "trust"); + if (err) + return err; + + /* Write trust control bit to VF entry in VF config symbol */ + vf_offset = NFP_NET_VF_CFG_MB_SZ + vf * NFP_NET_VF_CFG_SZ + + NFP_NET_VF_CFG_CTRL; + vf_ctrl = readb(app->pf->vfcfg_tbl2 + vf_offset); + vf_ctrl &= ~NFP_NET_VF_CFG_CTRL_TRUST; + vf_ctrl |= FIELD_PREP(NFP_NET_VF_CFG_CTRL_TRUST, enable); + writeb(vf_ctrl, app->pf->vfcfg_tbl2 + vf_offset); + + return nfp_net_sriov_update(app, vf, NFP_NET_VF_CFG_MB_UPD_TRUST, + "trust"); +} + int nfp_app_set_vf_link_state(struct net_device *netdev, int vf, int link_state) { @@ -213,6 +237,7 @@ int nfp_app_get_vf_config(struct net_device *netdev, int vf, ivi->qos = FIELD_GET(NFP_NET_VF_CFG_VLAN_QOS, vlan_tci); ivi->spoofchk = FIELD_GET(NFP_NET_VF_CFG_CTRL_SPOOF, flags); + ivi->trusted = FIELD_GET(NFP_NET_VF_CFG_CTRL_TRUST, flags); ivi->linkstate = FIELD_GET(NFP_NET_VF_CFG_CTRL_LINK_STATE, flags); return 0; diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.h b/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.h index c9f09c5bb5ee9aaf2b52083102bfba0a39794cc9..a3db0cbf6425f0c8785031c817710622f44d3f6a 100644 --- a/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.h +++ b/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.h @@ -1,5 +1,5 @@ /* SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) */ -/* Copyright (C) 2017 Netronome Systems, Inc. */ +/* Copyright (C) 2017-2019 Netronome Systems, Inc. */ #ifndef _NFP_NET_SRIOV_H_ #define _NFP_NET_SRIOV_H_ @@ -19,12 +19,14 @@ #define NFP_NET_VF_CFG_MB_CAP_VLAN (0x1 << 1) #define NFP_NET_VF_CFG_MB_CAP_SPOOF (0x1 << 2) #define NFP_NET_VF_CFG_MB_CAP_LINK_STATE (0x1 << 3) +#define NFP_NET_VF_CFG_MB_CAP_TRUST (0x1 << 4) #define NFP_NET_VF_CFG_MB_RET 0x2 #define NFP_NET_VF_CFG_MB_UPD 0x4 #define NFP_NET_VF_CFG_MB_UPD_MAC (0x1 << 0) #define NFP_NET_VF_CFG_MB_UPD_VLAN (0x1 << 1) #define NFP_NET_VF_CFG_MB_UPD_SPOOF (0x1 << 2) #define NFP_NET_VF_CFG_MB_UPD_LINK_STATE (0x1 << 3) +#define NFP_NET_VF_CFG_MB_UPD_TRUST (0x1 << 4) #define NFP_NET_VF_CFG_MB_VF_NUM 0x7 /* VF config entry @@ -35,6 +37,7 @@ #define NFP_NET_VF_CFG_MAC_HI 0x0 #define NFP_NET_VF_CFG_MAC_LO 0x6 #define NFP_NET_VF_CFG_CTRL 0x4 +#define NFP_NET_VF_CFG_CTRL_TRUST 0x8 #define NFP_NET_VF_CFG_CTRL_SPOOF 0x4 #define NFP_NET_VF_CFG_CTRL_LINK_STATE 0x3 #define NFP_NET_VF_CFG_LS_MODE_AUTO 0 @@ -48,6 +51,7 @@ int nfp_app_set_vf_mac(struct net_device *netdev, int vf, u8 *mac); int nfp_app_set_vf_vlan(struct net_device *netdev, int vf, u16 vlan, u8 qos, __be16 vlan_proto); int nfp_app_set_vf_spoofchk(struct net_device *netdev, int vf, bool setting); +int nfp_app_set_vf_trust(struct net_device *netdev, int vf, bool setting); int nfp_app_set_vf_link_state(struct net_device *netdev, int vf, int link_state); int nfp_app_get_vf_config(struct net_device *netdev, int vf, diff --git a/drivers/net/ethernet/netronome/nfp/nfp_netvf_main.c b/drivers/net/ethernet/netronome/nfp/nfp_netvf_main.c index 1145849ca7bac044195f0778db8d882e37b9b8ba..e4977cdf767899a3a664919aee2f1687893b884a 100644 --- a/drivers/net/ethernet/netronome/nfp/nfp_netvf_main.c +++ b/drivers/net/ethernet/netronome/nfp/nfp_netvf_main.c @@ -282,8 +282,14 @@ err_free_vf: static void nfp_netvf_pci_remove(struct pci_dev *pdev) { - struct nfp_net_vf *vf = pci_get_drvdata(pdev); - struct nfp_net *nn = vf->nn; + struct nfp_net_vf *vf; + struct nfp_net *nn; + + vf = pci_get_drvdata(pdev); + if (!vf) + return; + + nn = vf->nn; /* Note, the order is slightly different from above as we need * to keep the nn pointer around till we have freed everything. @@ -317,4 +323,5 @@ struct pci_driver nfp_netvf_pci_driver = { .id_table = nfp_netvf_pci_device_ids, .probe = nfp_netvf_pci_probe, .remove = nfp_netvf_pci_remove, + .shutdown = nfp_netvf_pci_remove, }; diff --git a/drivers/net/ethernet/netronome/nfp/nfp_port.c b/drivers/net/ethernet/netronome/nfp/nfp_port.c index 93c5bfc0510b59975c72411e3e5d09b068a6984b..fcd16877e6e091943fe30c269c761b96a7952d66 100644 --- a/drivers/net/ethernet/netronome/nfp/nfp_port.c +++ b/drivers/net/ethernet/netronome/nfp/nfp_port.c @@ -30,22 +30,6 @@ struct nfp_port *nfp_port_from_netdev(struct net_device *netdev) return NULL; } -int nfp_port_get_port_parent_id(struct net_device *netdev, - struct netdev_phys_item_id *ppid) -{ - struct nfp_port *port; - const u8 *serial; - - port = nfp_port_from_netdev(netdev); - if (!port) - return -EOPNOTSUPP; - - ppid->id_len = nfp_cpp_serial(port->app->cpp, &serial); - memcpy(&ppid->id, serial, ppid->id_len); - - return 0; -} - int nfp_port_setup_tc(struct net_device *netdev, enum tc_setup_type type, void *type_data) { diff --git a/drivers/net/ethernet/netronome/nfp/nfp_port.h b/drivers/net/ethernet/netronome/nfp/nfp_port.h index 90ae053f5c07709140b2e34bc22d611c7fc21817..d7fd203bb18037fab3be23d7ccab50747c17289c 100644 --- a/drivers/net/ethernet/netronome/nfp/nfp_port.h +++ b/drivers/net/ethernet/netronome/nfp/nfp_port.h @@ -131,6 +131,8 @@ int nfp_net_refresh_port_table_sync(struct nfp_pf *pf); int nfp_devlink_port_register(struct nfp_app *app, struct nfp_port *port); void nfp_devlink_port_unregister(struct nfp_port *port); +void nfp_devlink_port_type_eth_set(struct nfp_port *port); +void nfp_devlink_port_type_clear(struct nfp_port *port); /** * Mac stats (0x0000 - 0x0200) diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c index 3a4e224a64b71745284bd1016951438ca916ecf1..42cf4fd875eacf34a4e287180b63a57192a1ea48 100644 --- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c +++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c @@ -79,6 +79,8 @@ #define NFP_VERSIONS_NCSI_OFF 22 #define NFP_VERSIONS_CFGR_OFF 26 +#define NSP_SFF_EEPROM_BLOCK_LEN 8 + enum nfp_nsp_cmd { SPCODE_NOOP = 0, /* No operation */ SPCODE_SOFT_RESET = 1, /* Soft reset the NFP */ @@ -95,6 +97,7 @@ enum nfp_nsp_cmd { SPCODE_FW_STORED = 16, /* If no FW loaded, load flash app FW */ SPCODE_HWINFO_LOOKUP = 17, /* Lookup HWinfo with overwrites etc. */ SPCODE_VERSIONS = 21, /* Report FW versions */ + SPCODE_READ_SFF_EEPROM = 22, /* Read module EEPROM */ }; struct nfp_nsp_dma_buf { @@ -965,3 +968,62 @@ const char *nfp_nsp_versions_get(enum nfp_nsp_versions id, bool flash, return (const char *)&buf[buf_off]; } + +static int +__nfp_nsp_module_eeprom(struct nfp_nsp *state, void *buf, unsigned int size) +{ + struct nfp_nsp_command_buf_arg module_eeprom = { + { + .code = SPCODE_READ_SFF_EEPROM, + .option = size, + }, + .in_buf = buf, + .in_size = size, + .out_buf = buf, + .out_size = size, + }; + + return nfp_nsp_command_buf(state, &module_eeprom); +} + +int nfp_nsp_read_module_eeprom(struct nfp_nsp *state, int eth_index, + unsigned int offset, void *data, + unsigned int len, unsigned int *read_len) +{ + struct eeprom_buf { + u8 metalen; + __le16 length; + __le16 offset; + __le16 readlen; + u8 eth_index; + u8 data[0]; + } __packed *buf; + int bufsz, ret; + + BUILD_BUG_ON(offsetof(struct eeprom_buf, data) % 8); + + /* Buffer must be large enough and rounded to the next block size. */ + bufsz = struct_size(buf, data, round_up(len, NSP_SFF_EEPROM_BLOCK_LEN)); + buf = kzalloc(bufsz, GFP_KERNEL); + if (!buf) + return -ENOMEM; + + buf->metalen = + offsetof(struct eeprom_buf, data) / NSP_SFF_EEPROM_BLOCK_LEN; + buf->length = cpu_to_le16(len); + buf->offset = cpu_to_le16(offset); + buf->eth_index = eth_index; + + ret = __nfp_nsp_module_eeprom(state, buf, bufsz); + + *read_len = min_t(unsigned int, len, le16_to_cpu(buf->readlen)); + if (*read_len) + memcpy(data, buf->data, *read_len); + + if (!ret && *read_len < len) + ret = -EIO; + + kfree(buf); + + return ret; +} diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.h b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.h index bd9c358c646fde11ae590d758a8e23b9b43eda52..22ee6985ee1c31408b55e2375e1e4197b70c912e 100644 --- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.h +++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.h @@ -22,6 +22,9 @@ int nfp_nsp_write_flash(struct nfp_nsp *state, const struct firmware *fw); int nfp_nsp_mac_reinit(struct nfp_nsp *state); int nfp_nsp_load_stored_fw(struct nfp_nsp *state); int nfp_nsp_hwinfo_lookup(struct nfp_nsp *state, void *buf, unsigned int size); +int nfp_nsp_read_module_eeprom(struct nfp_nsp *state, int eth_index, + unsigned int offset, void *data, + unsigned int len, unsigned int *read_len); static inline bool nfp_nsp_has_mac_reinit(struct nfp_nsp *state) { @@ -43,6 +46,11 @@ static inline bool nfp_nsp_has_versions(struct nfp_nsp *state) return nfp_nsp_get_abi_ver_minor(state) > 27; } +static inline bool nfp_nsp_has_read_module_eeprom(struct nfp_nsp *state) +{ + return nfp_nsp_get_abi_ver_minor(state) > 28; +} + enum nfp_eth_interface { NFP_INTERFACE_NONE = 0, NFP_INTERFACE_SFP = 1, diff --git a/drivers/net/ethernet/nxp/lpc_eth.c b/drivers/net/ethernet/nxp/lpc_eth.c index 89d17399fb5accd910564e13c788b309a522bcb3..da138edddd3256905741ff146db7967429ec3ab1 100644 --- a/drivers/net/ethernet/nxp/lpc_eth.c +++ b/drivers/net/ethernet/nxp/lpc_eth.c @@ -1368,7 +1368,7 @@ static int lpc_eth_drv_probe(struct platform_device *pdev) if (!is_valid_ether_addr(ndev->dev_addr)) { const char *macaddr = of_get_mac_address(np); - if (macaddr) + if (!IS_ERR(macaddr)) memcpy(ndev->dev_addr, macaddr, ETH_ALEN); } if (!is_valid_ether_addr(ndev->dev_addr)) diff --git a/drivers/net/ethernet/pasemi/pasemi_mac.c b/drivers/net/ethernet/pasemi/pasemi_mac.c index a5bf46310f60dc314b5ac5684f7ac9ac23ad15f1..5ffaee9f53b18d46d339f7b26ef5f1916333644a 100644 --- a/drivers/net/ethernet/pasemi/pasemi_mac.c +++ b/drivers/net/ethernet/pasemi/pasemi_mac.c @@ -1355,7 +1355,7 @@ static void pasemi_mac_queue_csdesc(const struct sk_buff *skb, const int nh_off = skb_network_offset(skb); const int nh_len = skb_network_header_len(skb); const int nfrags = skb_shinfo(skb)->nr_frags; - int cs_size, i, fill, hdr, cpyhdr, evt; + int cs_size, i, fill, hdr, evt; dma_addr_t csdma; fund = XCT_FUN_ST | XCT_FUN_RR_8BRES | @@ -1396,7 +1396,6 @@ static void pasemi_mac_queue_csdesc(const struct sk_buff *skb, fill++; /* Copy the result into the TCP packet */ - cpyhdr = fill; CS_DESC(csring, fill++) = XCT_FUN_O | XCT_FUN_FUN(csring->fun) | XCT_FUN_LLEN(2) | XCT_FUN_SE; CS_DESC(csring, fill++) = XCT_PTR_LEN(2) | XCT_PTR_ADDR(cs_dest) | XCT_PTR_T; @@ -1839,7 +1838,7 @@ static void __exit pasemi_mac_cleanup_module(void) pci_unregister_driver(&pasemi_mac_driver); } -int pasemi_mac_init_module(void) +static int pasemi_mac_init_module(void) { int err; diff --git a/drivers/net/ethernet/qlogic/qed/qed.h b/drivers/net/ethernet/qlogic/qed/qed.h index 127c89b22ef0da7d60680481467517f83ca7bc72..c5e96ce20f59b2c6f00e32fa876de32da1d6527e 100644 --- a/drivers/net/ethernet/qlogic/qed/qed.h +++ b/drivers/net/ethernet/qlogic/qed/qed.h @@ -496,6 +496,9 @@ enum qed_mf_mode_bit { /* Allow DSCP to TC mapping */ QED_MF_DSCP_TO_TC_MAP, + + /* Do not insert a vlan tag with id 0 */ + QED_MF_DONT_ADD_VLAN0_TAG, }; enum qed_ufp_mode { diff --git a/drivers/net/ethernet/qlogic/qed/qed_dcbx.c b/drivers/net/ethernet/qlogic/qed/qed_dcbx.c index 69966dfc6e3d12f6b754dd94f0ca7fcc4018713f..5c6a276f69ac467cfcdb5e5c59b911ca1564d925 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_dcbx.c +++ b/drivers/net/ethernet/qlogic/qed/qed_dcbx.c @@ -204,9 +204,7 @@ qed_dcbx_set_params(struct qed_dcbx_results *p_data, else p_data->arr[type].update = DONT_UPDATE_DCB_DSCP; - /* Do not add vlan tag 0 when DCB is enabled and port in UFP/OV mode */ - if ((test_bit(QED_MF_8021Q_TAGGING, &p_hwfn->cdev->mf_bits) || - test_bit(QED_MF_8021AD_TAGGING, &p_hwfn->cdev->mf_bits))) + if (test_bit(QED_MF_DONT_ADD_VLAN0_TAG, &p_hwfn->cdev->mf_bits)) p_data->arr[type].dont_add_vlan0 = true; /* QM reconf data */ diff --git a/drivers/net/ethernet/qlogic/qed/qed_dev.c b/drivers/net/ethernet/qlogic/qed/qed_dev.c index 866cdc86a3f27c879d4364089597ea499a0b1714..fccdb06fc5c568f0f2db424f0d7ecb0d7847652f 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_dev.c +++ b/drivers/net/ethernet/qlogic/qed/qed_dev.c @@ -3140,12 +3140,14 @@ static int qed_hw_get_nvm_info(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt) cdev->mf_bits = BIT(QED_MF_OVLAN_CLSS) | BIT(QED_MF_LLH_PROTO_CLSS) | BIT(QED_MF_UFP_SPECIFIC) | - BIT(QED_MF_8021Q_TAGGING); + BIT(QED_MF_8021Q_TAGGING) | + BIT(QED_MF_DONT_ADD_VLAN0_TAG); break; case NVM_CFG1_GLOB_MF_MODE_BD: cdev->mf_bits = BIT(QED_MF_OVLAN_CLSS) | BIT(QED_MF_LLH_PROTO_CLSS) | - BIT(QED_MF_8021AD_TAGGING); + BIT(QED_MF_8021AD_TAGGING) | + BIT(QED_MF_DONT_ADD_VLAN0_TAG); break; case NVM_CFG1_GLOB_MF_MODE_NPAR1_0: cdev->mf_bits = BIT(QED_MF_LLH_MAC_CLSS) | diff --git a/drivers/net/ethernet/qlogic/qede/qede.h b/drivers/net/ethernet/qlogic/qede/qede.h index 63a78162cfafd735fb3607c193ab79b2c75a54f6..92fe226980fda67f8c8b692b79cb2803d1121c1e 100644 --- a/drivers/net/ethernet/qlogic/qede/qede.h +++ b/drivers/net/ethernet/qlogic/qede/qede.h @@ -498,8 +498,7 @@ struct qede_reload_args { /* Datapath functions definition */ netdev_tx_t qede_start_xmit(struct sk_buff *skb, struct net_device *ndev); u16 qede_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback); + struct net_device *sb_dev); netdev_features_t qede_features_check(struct sk_buff *skb, struct net_device *dev, netdev_features_t features); diff --git a/drivers/net/ethernet/qlogic/qede/qede_ethtool.c b/drivers/net/ethernet/qlogic/qede/qede_ethtool.c index 4555c0b161efbda1356a0868117b65d4866d0746..8911a97ab0cacb0a78ae525f8d57405ae4ee3ae3 100644 --- a/drivers/net/ethernet/qlogic/qede/qede_ethtool.c +++ b/drivers/net/ethernet/qlogic/qede/qede_ethtool.c @@ -652,9 +652,9 @@ static void qede_get_drvinfo(struct net_device *ndev, { char mfw[ETHTOOL_FWVERS_LEN], storm[ETHTOOL_FWVERS_LEN]; struct qede_dev *edev = netdev_priv(ndev); + char mbi[ETHTOOL_FWVERS_LEN]; strlcpy(info->driver, "qede", sizeof(info->driver)); - strlcpy(info->version, DRV_MODULE_VERSION, sizeof(info->version)); snprintf(storm, ETHTOOL_FWVERS_LEN, "%d.%d.%d.%d", edev->dev_info.common.fw_major, @@ -668,13 +668,27 @@ static void qede_get_drvinfo(struct net_device *ndev, (edev->dev_info.common.mfw_rev >> 8) & 0xFF, edev->dev_info.common.mfw_rev & 0xFF); - if ((strlen(storm) + strlen(mfw) + strlen("mfw storm ")) < - sizeof(info->fw_version)) { + if ((strlen(storm) + strlen(DRV_MODULE_VERSION) + strlen("[storm] ")) < + sizeof(info->version)) + snprintf(info->version, sizeof(info->version), + "%s [storm %s]", DRV_MODULE_VERSION, storm); + else + snprintf(info->version, sizeof(info->version), + "%s %s", DRV_MODULE_VERSION, storm); + + if (edev->dev_info.common.mbi_version) { + snprintf(mbi, ETHTOOL_FWVERS_LEN, "%d.%d.%d", + (edev->dev_info.common.mbi_version & + QED_MBI_VERSION_2_MASK) >> QED_MBI_VERSION_2_OFFSET, + (edev->dev_info.common.mbi_version & + QED_MBI_VERSION_1_MASK) >> QED_MBI_VERSION_1_OFFSET, + (edev->dev_info.common.mbi_version & + QED_MBI_VERSION_0_MASK) >> QED_MBI_VERSION_0_OFFSET); snprintf(info->fw_version, sizeof(info->fw_version), - "mfw %s storm %s", mfw, storm); + "mbi %s [mfw %s]", mbi, mfw); } else { snprintf(info->fw_version, sizeof(info->fw_version), - "%s %s", mfw, storm); + "mfw %s", mfw); } strlcpy(info->bus_info, pci_name(edev->pdev), sizeof(info->bus_info)); diff --git a/drivers/net/ethernet/qlogic/qede/qede_fp.c b/drivers/net/ethernet/qlogic/qede/qede_fp.c index 6f7e3622c6b41816b0c166e33475c9582b5c7841..0ae28f0d2523069cb035f45454f928b2c398b418 100644 --- a/drivers/net/ethernet/qlogic/qede/qede_fp.c +++ b/drivers/net/ethernet/qlogic/qede/qede_fp.c @@ -1657,12 +1657,12 @@ netdev_tx_t qede_start_xmit(struct sk_buff *skb, struct net_device *ndev) txq->tx_db.data.bd_prod = cpu_to_le16(qed_chain_get_prod_idx(&txq->tx_pbl)); - if (!skb->xmit_more || netif_xmit_stopped(netdev_txq)) + if (!netdev_xmit_more() || netif_xmit_stopped(netdev_txq)) qede_update_tx_producer(txq); if (unlikely(qed_chain_get_elem_left(&txq->tx_pbl) < (MAX_SKB_FRAGS + 1))) { - if (skb->xmit_more) + if (netdev_xmit_more()) qede_update_tx_producer(txq); netif_tx_stop_queue(netdev_txq); @@ -1688,8 +1688,7 @@ netdev_tx_t qede_start_xmit(struct sk_buff *skb, struct net_device *ndev) } u16 qede_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { struct qede_dev *edev = netdev_priv(dev); int total_txq; @@ -1697,7 +1696,7 @@ u16 qede_select_queue(struct net_device *dev, struct sk_buff *skb, total_txq = QEDE_TSS_COUNT(edev) * edev->dev_info.num_tc; return QEDE_TSS_COUNT(edev) ? - fallback(dev, skb, NULL) % total_txq : 0; + netdev_pick_tx(dev, skb, NULL) % total_txq : 0; } /* 8B udp header + 8B base tunnel header + 32B option length */ diff --git a/drivers/net/ethernet/qualcomm/qca_spi.c b/drivers/net/ethernet/qualcomm/qca_spi.c index 97f92953bdb9c071deaf705a607589a12d064a53..b28360bc2255e85bc5bdfb45b1313268f1f5e833 100644 --- a/drivers/net/ethernet/qualcomm/qca_spi.c +++ b/drivers/net/ethernet/qualcomm/qca_spi.c @@ -966,7 +966,7 @@ qca_spi_probe(struct spi_device *spi) mac = of_get_mac_address(spi->dev.of_node); - if (mac) + if (!IS_ERR(mac)) ether_addr_copy(qca->net_dev->dev_addr, mac); if (!is_valid_ether_addr(qca->net_dev->dev_addr)) { diff --git a/drivers/net/ethernet/qualcomm/qca_uart.c b/drivers/net/ethernet/qualcomm/qca_uart.c index db6068cd7a1f21473e9c440850c3f80894439e29..590616846cd168735ee701e1724f333441a1d2ad 100644 --- a/drivers/net/ethernet/qualcomm/qca_uart.c +++ b/drivers/net/ethernet/qualcomm/qca_uart.c @@ -351,7 +351,7 @@ static int qca_uart_probe(struct serdev_device *serdev) mac = of_get_mac_address(serdev->dev.of_node); - if (mac) + if (!IS_ERR(mac)) ether_addr_copy(qca->net_dev->dev_addr, mac); if (!is_valid_ether_addr(qca->net_dev->dev_addr)) { diff --git a/drivers/net/ethernet/rdc/r6040.c b/drivers/net/ethernet/rdc/r6040.c index 04aa592f35c36f9782d891c65e2c0d810c9fd4cf..ad335bca3273dc4c1f29108116e8248952de5db8 100644 --- a/drivers/net/ethernet/rdc/r6040.c +++ b/drivers/net/ethernet/rdc/r6040.c @@ -840,7 +840,7 @@ static netdev_tx_t r6040_start_xmit(struct sk_buff *skb, skb_tx_timestamp(skb); /* Trigger the MAC to check the TX descriptor */ - if (!skb->xmit_more || netif_queue_stopped(dev)) + if (!netdev_xmit_more() || netif_queue_stopped(dev)) iowrite16(TM2TX, ioaddr + MTPR); lp->tx_insert_ptr = descptr->vndescp; diff --git a/drivers/net/ethernet/realtek/r8169.c b/drivers/net/ethernet/realtek/r8169.c index ed651dde6ef9ee8970bc68acb9c6fb9282411189..549be1c76a899e1e9d2d182beeef73ad8b77a83b 100644 --- a/drivers/net/ethernet/realtek/r8169.c +++ b/drivers/net/ethernet/realtek/r8169.c @@ -491,10 +491,6 @@ enum rtl_register_content { PCIDAC = (1 << 4), PCIMulRW = (1 << 3), #define INTT_MASK GENMASK(1, 0) - INTT_0 = 0x0000, // 8168 - INTT_1 = 0x0001, // 8168 - INTT_2 = 0x0002, // 8168 - INTT_3 = 0x0003, // 8168 /* rtl8169_PHYstatus */ TBI_Enable = 0x80, @@ -703,6 +699,8 @@ struct rtl8169_private { u32 ocp_base; }; +typedef void (*rtl_generic_fct)(struct rtl8169_private *tp); + MODULE_AUTHOR("Realtek and the Linux r8169 crew "); MODULE_DESCRIPTION("RealTek RTL-8169 Gigabit Ethernet driver"); module_param_named(debug, debug.msg_enable, int, 0); @@ -777,9 +775,9 @@ static bool rtl_loop_wait(struct rtl8169_private *tp, const struct rtl_cond *c, int i; for (i = 0; i < n; i++) { - delay(d); if (c->check(tp) == high) return true; + delay(d); } netif_err(tp, drv, tp->dev, "%s == %d (loop: %d, delay: %d).\n", c->msg, !high, n, d); @@ -1067,8 +1065,8 @@ DECLARE_RTL_COND(rtl_eriar_cond) return RTL_R32(tp, ERIAR) & ERIAR_FLAG; } -static void rtl_eri_write(struct rtl8169_private *tp, int addr, u32 mask, - u32 val, int type) +static void _rtl_eri_write(struct rtl8169_private *tp, int addr, u32 mask, + u32 val, int type) { BUG_ON((addr & 3) || (mask == 0)); RTL_W32(tp, ERIDR, val); @@ -1077,7 +1075,13 @@ static void rtl_eri_write(struct rtl8169_private *tp, int addr, u32 mask, rtl_udelay_loop_wait_low(tp, &rtl_eriar_cond, 100, 100); } -static u32 rtl_eri_read(struct rtl8169_private *tp, int addr, int type) +static void rtl_eri_write(struct rtl8169_private *tp, int addr, u32 mask, + u32 val) +{ + _rtl_eri_write(tp, addr, mask, val, ERIAR_EXGMAC); +} + +static u32 _rtl_eri_read(struct rtl8169_private *tp, int addr, int type) { RTL_W32(tp, ERIAR, ERIAR_READ_CMD | type | ERIAR_MASK_1111 | addr); @@ -1085,13 +1089,30 @@ static u32 rtl_eri_read(struct rtl8169_private *tp, int addr, int type) RTL_R32(tp, ERIDR) : ~0; } +static u32 rtl_eri_read(struct rtl8169_private *tp, int addr) +{ + return _rtl_eri_read(tp, addr, ERIAR_EXGMAC); +} + static void rtl_w0w1_eri(struct rtl8169_private *tp, int addr, u32 mask, u32 p, - u32 m, int type) + u32 m) { u32 val; - val = rtl_eri_read(tp, addr, type); - rtl_eri_write(tp, addr, mask, (val & ~m) | p, type); + val = rtl_eri_read(tp, addr); + rtl_eri_write(tp, addr, mask, (val & ~m) | p); +} + +static void rtl_eri_set_bits(struct rtl8169_private *tp, int addr, u32 mask, + u32 p) +{ + rtl_w0w1_eri(tp, addr, mask, p, 0); +} + +static void rtl_eri_clear_bits(struct rtl8169_private *tp, int addr, u32 mask, + u32 m) +{ + rtl_w0w1_eri(tp, addr, mask, 0, m); } static u32 r8168dp_ocp_read(struct rtl8169_private *tp, u8 mask, u16 reg) @@ -1103,7 +1124,7 @@ static u32 r8168dp_ocp_read(struct rtl8169_private *tp, u8 mask, u16 reg) static u32 r8168ep_ocp_read(struct rtl8169_private *tp, u8 mask, u16 reg) { - return rtl_eri_read(tp, reg, ERIAR_OOB); + return _rtl_eri_read(tp, reg, ERIAR_OOB); } static void r8168dp_ocp_write(struct rtl8169_private *tp, u8 mask, u16 reg, @@ -1117,13 +1138,13 @@ static void r8168dp_ocp_write(struct rtl8169_private *tp, u8 mask, u16 reg, static void r8168ep_ocp_write(struct rtl8169_private *tp, u8 mask, u16 reg, u32 data) { - rtl_eri_write(tp, reg, ((u32)mask & 0x0f) << ERIAR_MASK_SHIFT, - data, ERIAR_OOB); + _rtl_eri_write(tp, reg, ((u32)mask & 0x0f) << ERIAR_MASK_SHIFT, + data, ERIAR_OOB); } static void r8168dp_oob_notify(struct rtl8169_private *tp, u8 cmd) { - rtl_eri_write(tp, 0xe8, ERIAR_MASK_0001, cmd, ERIAR_EXGMAC); + rtl_eri_write(tp, 0xe8, ERIAR_MASK_0001, cmd); r8168dp_ocp_write(tp, 0x1, 0x30, 0x00000001); } @@ -1259,19 +1280,10 @@ static bool r8168_check_dash(struct rtl8169_private *tp) } } -struct exgmac_reg { - u16 addr; - u16 mask; - u32 val; -}; - -static void rtl_write_exgmac_batch(struct rtl8169_private *tp, - const struct exgmac_reg *r, int len) +static void rtl_reset_packet_filter(struct rtl8169_private *tp) { - while (len-- > 0) { - rtl_eri_write(tp, r->addr, r->mask, r->val, ERIAR_EXGMAC); - r++; - } + rtl_eri_clear_bits(tp, 0xdc, ERIAR_MASK_0001, BIT(0)); + rtl_eri_set_bits(tp, 0xdc, ERIAR_MASK_0001, BIT(0)); } DECLARE_RTL_COND(rtl_efusear_cond) @@ -1327,48 +1339,31 @@ static void rtl_link_chg_patch(struct rtl8169_private *tp) if (tp->mac_version == RTL_GIGA_MAC_VER_34 || tp->mac_version == RTL_GIGA_MAC_VER_38) { if (phydev->speed == SPEED_1000) { - rtl_eri_write(tp, 0x1bc, ERIAR_MASK_1111, 0x00000011, - ERIAR_EXGMAC); - rtl_eri_write(tp, 0x1dc, ERIAR_MASK_1111, 0x00000005, - ERIAR_EXGMAC); + rtl_eri_write(tp, 0x1bc, ERIAR_MASK_1111, 0x00000011); + rtl_eri_write(tp, 0x1dc, ERIAR_MASK_1111, 0x00000005); } else if (phydev->speed == SPEED_100) { - rtl_eri_write(tp, 0x1bc, ERIAR_MASK_1111, 0x0000001f, - ERIAR_EXGMAC); - rtl_eri_write(tp, 0x1dc, ERIAR_MASK_1111, 0x00000005, - ERIAR_EXGMAC); + rtl_eri_write(tp, 0x1bc, ERIAR_MASK_1111, 0x0000001f); + rtl_eri_write(tp, 0x1dc, ERIAR_MASK_1111, 0x00000005); } else { - rtl_eri_write(tp, 0x1bc, ERIAR_MASK_1111, 0x0000001f, - ERIAR_EXGMAC); - rtl_eri_write(tp, 0x1dc, ERIAR_MASK_1111, 0x0000003f, - ERIAR_EXGMAC); + rtl_eri_write(tp, 0x1bc, ERIAR_MASK_1111, 0x0000001f); + rtl_eri_write(tp, 0x1dc, ERIAR_MASK_1111, 0x0000003f); } - /* Reset packet filter */ - rtl_w0w1_eri(tp, 0xdc, ERIAR_MASK_0001, 0x00, 0x01, - ERIAR_EXGMAC); - rtl_w0w1_eri(tp, 0xdc, ERIAR_MASK_0001, 0x01, 0x00, - ERIAR_EXGMAC); + rtl_reset_packet_filter(tp); } else if (tp->mac_version == RTL_GIGA_MAC_VER_35 || tp->mac_version == RTL_GIGA_MAC_VER_36) { if (phydev->speed == SPEED_1000) { - rtl_eri_write(tp, 0x1bc, ERIAR_MASK_1111, 0x00000011, - ERIAR_EXGMAC); - rtl_eri_write(tp, 0x1dc, ERIAR_MASK_1111, 0x00000005, - ERIAR_EXGMAC); + rtl_eri_write(tp, 0x1bc, ERIAR_MASK_1111, 0x00000011); + rtl_eri_write(tp, 0x1dc, ERIAR_MASK_1111, 0x00000005); } else { - rtl_eri_write(tp, 0x1bc, ERIAR_MASK_1111, 0x0000001f, - ERIAR_EXGMAC); - rtl_eri_write(tp, 0x1dc, ERIAR_MASK_1111, 0x0000003f, - ERIAR_EXGMAC); + rtl_eri_write(tp, 0x1bc, ERIAR_MASK_1111, 0x0000001f); + rtl_eri_write(tp, 0x1dc, ERIAR_MASK_1111, 0x0000003f); } } else if (tp->mac_version == RTL_GIGA_MAC_VER_37) { if (phydev->speed == SPEED_10) { - rtl_eri_write(tp, 0x1d0, ERIAR_MASK_0011, 0x4d02, - ERIAR_EXGMAC); - rtl_eri_write(tp, 0x1dc, ERIAR_MASK_0011, 0x0060, - ERIAR_EXGMAC); + rtl_eri_write(tp, 0x1d0, ERIAR_MASK_0011, 0x4d02); + rtl_eri_write(tp, 0x1dc, ERIAR_MASK_0011, 0x0060a); } else { - rtl_eri_write(tp, 0x1d0, ERIAR_MASK_0011, 0x0000, - ERIAR_EXGMAC); + rtl_eri_write(tp, 0x1d0, ERIAR_MASK_0011, 0x0000); } } } @@ -1409,19 +1404,11 @@ static void __rtl8169_set_wol(struct rtl8169_private *tp, u32 wolopts) case RTL_GIGA_MAC_VER_40 ... RTL_GIGA_MAC_VER_51: tmp = ARRAY_SIZE(cfg) - 1; if (wolopts & WAKE_MAGIC) - rtl_w0w1_eri(tp, - 0x0dc, - ERIAR_MASK_0100, - MagicPacket_v2, - 0x0000, - ERIAR_EXGMAC); + rtl_eri_set_bits(tp, 0x0dc, ERIAR_MASK_0100, + MagicPacket_v2); else - rtl_w0w1_eri(tp, - 0x0dc, - ERIAR_MASK_0100, - 0x0000, - MagicPacket_v2, - ERIAR_EXGMAC); + rtl_eri_clear_bits(tp, 0x0dc, ERIAR_MASK_0100, + MagicPacket_v2); break; default: tmp = ARRAY_SIZE(cfg); @@ -2293,8 +2280,8 @@ struct phy_reg { u16 val; }; -static void rtl_writephy_batch(struct rtl8169_private *tp, - const struct phy_reg *regs, int len) +static void __rtl_writephy_batch(struct rtl8169_private *tp, + const struct phy_reg *regs, int len) { while (len-- > 0) { rtl_writephy(tp, regs->reg, regs->val); @@ -2302,6 +2289,8 @@ static void rtl_writephy_batch(struct rtl8169_private *tp, } } +#define rtl_writephy_batch(tp, a) __rtl_writephy_batch(tp, a, ARRAY_SIZE(a)) + #define PHY_READ 0x00000000 #define PHY_DATA_OR 0x10000000 #define PHY_DATA_AND 0x20000000 @@ -2564,7 +2553,11 @@ static void rtl_apply_firmware_cond(struct rtl8169_private *tp, u8 reg, u16 val) static void rtl8168_config_eee_mac(struct rtl8169_private *tp) { - rtl_w0w1_eri(tp, 0x1b0, ERIAR_MASK_1111, 0x0003, 0x0000, ERIAR_EXGMAC); + /* Adjust EEE LED frequency */ + if (tp->mac_version != RTL_GIGA_MAC_VER_38) + RTL_W8(tp, EEE_LED, RTL_R8(tp, EEE_LED) & ~0x07); + + rtl_eri_set_bits(tp, 0x1b0, ERIAR_MASK_1111, 0x0003); } static void rtl8168f_config_eee_phy(struct rtl8169_private *tp) @@ -2653,7 +2646,7 @@ static void rtl8169s_hw_phy_config(struct rtl8169_private *tp) { 0x00, 0x9200 } }; - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); } static void rtl8169sb_hw_phy_config(struct rtl8169_private *tp) @@ -2664,7 +2657,7 @@ static void rtl8169sb_hw_phy_config(struct rtl8169_private *tp) { 0x1f, 0x0000 } }; - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); } static void rtl8169scd_hw_phy_config_quirk(struct rtl8169_private *tp) @@ -2722,7 +2715,7 @@ static void rtl8169scd_hw_phy_config(struct rtl8169_private *tp) { 0x1f, 0x0000 } }; - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); rtl8169scd_hw_phy_config_quirk(tp); } @@ -2777,7 +2770,7 @@ static void rtl8169sce_hw_phy_config(struct rtl8169_private *tp) { 0x1f, 0x0000 } }; - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); } static void rtl8168bb_hw_phy_config(struct rtl8169_private *tp) @@ -2790,7 +2783,7 @@ static void rtl8168bb_hw_phy_config(struct rtl8169_private *tp) rtl_writephy(tp, 0x1f, 0x0001); rtl_patchphy(tp, 0x16, 1 << 0); - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); } static void rtl8168bef_hw_phy_config(struct rtl8169_private *tp) @@ -2801,7 +2794,7 @@ static void rtl8168bef_hw_phy_config(struct rtl8169_private *tp) { 0x1f, 0x0000 } }; - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); } static void rtl8168cp_1_hw_phy_config(struct rtl8169_private *tp) @@ -2814,7 +2807,7 @@ static void rtl8168cp_1_hw_phy_config(struct rtl8169_private *tp) { 0x1f, 0x0000 } }; - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); } static void rtl8168cp_2_hw_phy_config(struct rtl8169_private *tp) @@ -2829,7 +2822,7 @@ static void rtl8168cp_2_hw_phy_config(struct rtl8169_private *tp) rtl_patchphy(tp, 0x14, 1 << 5); rtl_patchphy(tp, 0x0d, 1 << 5); - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); } static void rtl8168c_1_hw_phy_config(struct rtl8169_private *tp) @@ -2854,7 +2847,7 @@ static void rtl8168c_1_hw_phy_config(struct rtl8169_private *tp) { 0x09, 0x0000 } }; - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); rtl_patchphy(tp, 0x14, 1 << 5); rtl_patchphy(tp, 0x0d, 1 << 5); @@ -2881,7 +2874,7 @@ static void rtl8168c_2_hw_phy_config(struct rtl8169_private *tp) { 0x1f, 0x0000 } }; - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); rtl_patchphy(tp, 0x16, 1 << 0); rtl_patchphy(tp, 0x14, 1 << 5); @@ -2903,7 +2896,7 @@ static void rtl8168c_3_hw_phy_config(struct rtl8169_private *tp) { 0x1f, 0x0000 } }; - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); rtl_patchphy(tp, 0x16, 1 << 0); rtl_patchphy(tp, 0x14, 1 << 5); @@ -2959,7 +2952,7 @@ static void rtl8168d_1_hw_phy_config(struct rtl8169_private *tp) { 0x0d, 0xf880 } }; - rtl_writephy_batch(tp, phy_reg_init_0, ARRAY_SIZE(phy_reg_init_0)); + rtl_writephy_batch(tp, phy_reg_init_0); /* * Rx Error Issue @@ -2980,7 +2973,7 @@ static void rtl8168d_1_hw_phy_config(struct rtl8169_private *tp) }; int val; - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); val = rtl_readphy(tp, 0x0d); @@ -3006,7 +2999,7 @@ static void rtl8168d_1_hw_phy_config(struct rtl8169_private *tp) { 0x06, 0x6662 } }; - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); } /* RSET couple improve */ @@ -3070,7 +3063,7 @@ static void rtl8168d_2_hw_phy_config(struct rtl8169_private *tp) { 0x0d, 0xf880 } }; - rtl_writephy_batch(tp, phy_reg_init_0, ARRAY_SIZE(phy_reg_init_0)); + rtl_writephy_batch(tp, phy_reg_init_0); if (rtl8168d_efuse_read(tp, 0x01) == 0xb1) { static const struct phy_reg phy_reg_init[] = { @@ -3084,7 +3077,7 @@ static void rtl8168d_2_hw_phy_config(struct rtl8169_private *tp) }; int val; - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); val = rtl_readphy(tp, 0x0d); if ((val & 0x00ff) != 0x006c) { @@ -3109,7 +3102,7 @@ static void rtl8168d_2_hw_phy_config(struct rtl8169_private *tp) { 0x06, 0x2642 } }; - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); } /* Fine tune PLL performance */ @@ -3187,7 +3180,7 @@ static void rtl8168d_3_hw_phy_config(struct rtl8169_private *tp) { 0x1f, 0x0000 } }; - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); } static void rtl8168d_4_hw_phy_config(struct rtl8169_private *tp) @@ -3202,7 +3195,7 @@ static void rtl8168d_4_hw_phy_config(struct rtl8169_private *tp) { 0x1f, 0x0000 } }; - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); rtl_patchphy(tp, 0x0d, 1 << 5); } @@ -3238,7 +3231,7 @@ static void rtl8168e_1_hw_phy_config(struct rtl8169_private *tp) rtl_apply_firmware(tp); - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); /* DCO enable for 10M IDLE Power */ rtl_writephy(tp, 0x1f, 0x0007); @@ -3286,14 +3279,11 @@ static void rtl_rar_exgmac_set(struct rtl8169_private *tp, u8 *addr) addr[2] | (addr[3] << 8), addr[4] | (addr[5] << 8) }; - const struct exgmac_reg e[] = { - { .addr = 0xe0, ERIAR_MASK_1111, .val = w[0] | (w[1] << 16) }, - { .addr = 0xe4, ERIAR_MASK_1111, .val = w[2] }, - { .addr = 0xf0, ERIAR_MASK_1111, .val = w[0] << 16 }, - { .addr = 0xf4, ERIAR_MASK_1111, .val = w[1] | (w[2] << 16) } - }; - rtl_write_exgmac_batch(tp, e, ARRAY_SIZE(e)); + rtl_eri_write(tp, 0xe0, ERIAR_MASK_1111, w[0] | (w[1] << 16)); + rtl_eri_write(tp, 0xe4, ERIAR_MASK_1111, w[2]); + rtl_eri_write(tp, 0xf0, ERIAR_MASK_1111, w[0] << 16); + rtl_eri_write(tp, 0xf4, ERIAR_MASK_1111, w[1] | (w[2] << 16)); } static void rtl8168e_2_hw_phy_config(struct rtl8169_private *tp) @@ -3327,7 +3317,7 @@ static void rtl8168e_2_hw_phy_config(struct rtl8169_private *tp) rtl_apply_firmware(tp); - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); /* For 4-corner performance improve */ rtl_writephy(tp, 0x1f, 0x0005); @@ -3436,7 +3426,7 @@ static void rtl8168f_1_hw_phy_config(struct rtl8169_private *tp) rtl_apply_firmware(tp); - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); rtl8168f_hw_phy_config(tp); @@ -3502,7 +3492,7 @@ static void rtl8411_hw_phy_config(struct rtl8169_private *tp) rtl_w0w1_phy(tp, 0x06, 0x4000, 0x0000); rtl_writephy(tp, 0x1f, 0x0000); - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); /* Modify green table for giga */ rtl_writephy(tp, 0x1f, 0x0005); @@ -3922,7 +3912,7 @@ static void rtl8102e_hw_phy_config(struct rtl8169_private *tp) rtl_patchphy(tp, 0x19, 1 << 13); rtl_patchphy(tp, 0x10, 1 << 15); - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); } static void rtl8105e_hw_phy_config(struct rtl8169_private *tp) @@ -3948,7 +3938,7 @@ static void rtl8105e_hw_phy_config(struct rtl8169_private *tp) rtl_apply_firmware(tp); - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_writephy_batch(tp, phy_reg_init); } static void rtl8402_hw_phy_config(struct rtl8169_private *tp) @@ -3961,7 +3951,7 @@ static void rtl8402_hw_phy_config(struct rtl8169_private *tp) rtl_apply_firmware(tp); /* EEE setting */ - rtl_eri_write(tp, 0x1b0, ERIAR_MASK_0011, 0x0000, ERIAR_EXGMAC); + rtl_eri_write(tp, 0x1b0, ERIAR_MASK_0011, 0x0000); rtl_writephy(tp, 0x1f, 0x0004); rtl_writephy(tp, 0x10, 0x401f); rtl_writephy(tp, 0x19, 0x7030); @@ -3984,139 +3974,73 @@ static void rtl8106e_hw_phy_config(struct rtl8169_private *tp) rtl_apply_firmware(tp); - rtl_eri_write(tp, 0x1b0, ERIAR_MASK_0011, 0x0000, ERIAR_EXGMAC); - rtl_writephy_batch(tp, phy_reg_init, ARRAY_SIZE(phy_reg_init)); + rtl_eri_write(tp, 0x1b0, ERIAR_MASK_0011, 0x0000); + rtl_writephy_batch(tp, phy_reg_init); - rtl_eri_write(tp, 0x1d0, ERIAR_MASK_0011, 0x0000, ERIAR_EXGMAC); + rtl_eri_write(tp, 0x1d0, ERIAR_MASK_0011, 0x0000); } static void rtl_hw_phy_config(struct net_device *dev) { + static const rtl_generic_fct phy_configs[] = { + /* PCI devices. */ + [RTL_GIGA_MAC_VER_01] = NULL, + [RTL_GIGA_MAC_VER_02] = rtl8169s_hw_phy_config, + [RTL_GIGA_MAC_VER_03] = rtl8169s_hw_phy_config, + [RTL_GIGA_MAC_VER_04] = rtl8169sb_hw_phy_config, + [RTL_GIGA_MAC_VER_05] = rtl8169scd_hw_phy_config, + [RTL_GIGA_MAC_VER_06] = rtl8169sce_hw_phy_config, + /* PCI-E devices. */ + [RTL_GIGA_MAC_VER_07] = rtl8102e_hw_phy_config, + [RTL_GIGA_MAC_VER_08] = rtl8102e_hw_phy_config, + [RTL_GIGA_MAC_VER_09] = rtl8102e_hw_phy_config, + [RTL_GIGA_MAC_VER_10] = NULL, + [RTL_GIGA_MAC_VER_11] = rtl8168bb_hw_phy_config, + [RTL_GIGA_MAC_VER_12] = rtl8168bef_hw_phy_config, + [RTL_GIGA_MAC_VER_13] = NULL, + [RTL_GIGA_MAC_VER_14] = NULL, + [RTL_GIGA_MAC_VER_15] = NULL, + [RTL_GIGA_MAC_VER_16] = NULL, + [RTL_GIGA_MAC_VER_17] = rtl8168bef_hw_phy_config, + [RTL_GIGA_MAC_VER_18] = rtl8168cp_1_hw_phy_config, + [RTL_GIGA_MAC_VER_19] = rtl8168c_1_hw_phy_config, + [RTL_GIGA_MAC_VER_20] = rtl8168c_2_hw_phy_config, + [RTL_GIGA_MAC_VER_21] = rtl8168c_3_hw_phy_config, + [RTL_GIGA_MAC_VER_22] = rtl8168c_4_hw_phy_config, + [RTL_GIGA_MAC_VER_23] = rtl8168cp_2_hw_phy_config, + [RTL_GIGA_MAC_VER_24] = rtl8168cp_2_hw_phy_config, + [RTL_GIGA_MAC_VER_25] = rtl8168d_1_hw_phy_config, + [RTL_GIGA_MAC_VER_26] = rtl8168d_2_hw_phy_config, + [RTL_GIGA_MAC_VER_27] = rtl8168d_3_hw_phy_config, + [RTL_GIGA_MAC_VER_28] = rtl8168d_4_hw_phy_config, + [RTL_GIGA_MAC_VER_29] = rtl8105e_hw_phy_config, + [RTL_GIGA_MAC_VER_30] = rtl8105e_hw_phy_config, + [RTL_GIGA_MAC_VER_31] = NULL, + [RTL_GIGA_MAC_VER_32] = rtl8168e_1_hw_phy_config, + [RTL_GIGA_MAC_VER_33] = rtl8168e_1_hw_phy_config, + [RTL_GIGA_MAC_VER_34] = rtl8168e_2_hw_phy_config, + [RTL_GIGA_MAC_VER_35] = rtl8168f_1_hw_phy_config, + [RTL_GIGA_MAC_VER_36] = rtl8168f_2_hw_phy_config, + [RTL_GIGA_MAC_VER_37] = rtl8402_hw_phy_config, + [RTL_GIGA_MAC_VER_38] = rtl8411_hw_phy_config, + [RTL_GIGA_MAC_VER_39] = rtl8106e_hw_phy_config, + [RTL_GIGA_MAC_VER_40] = rtl8168g_1_hw_phy_config, + [RTL_GIGA_MAC_VER_41] = NULL, + [RTL_GIGA_MAC_VER_42] = rtl8168g_2_hw_phy_config, + [RTL_GIGA_MAC_VER_43] = rtl8168g_2_hw_phy_config, + [RTL_GIGA_MAC_VER_44] = rtl8168g_2_hw_phy_config, + [RTL_GIGA_MAC_VER_45] = rtl8168h_1_hw_phy_config, + [RTL_GIGA_MAC_VER_46] = rtl8168h_2_hw_phy_config, + [RTL_GIGA_MAC_VER_47] = rtl8168h_1_hw_phy_config, + [RTL_GIGA_MAC_VER_48] = rtl8168h_2_hw_phy_config, + [RTL_GIGA_MAC_VER_49] = rtl8168ep_1_hw_phy_config, + [RTL_GIGA_MAC_VER_50] = rtl8168ep_2_hw_phy_config, + [RTL_GIGA_MAC_VER_51] = rtl8168ep_2_hw_phy_config, + }; struct rtl8169_private *tp = netdev_priv(dev); - switch (tp->mac_version) { - case RTL_GIGA_MAC_VER_01: - break; - case RTL_GIGA_MAC_VER_02: - case RTL_GIGA_MAC_VER_03: - rtl8169s_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_04: - rtl8169sb_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_05: - rtl8169scd_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_06: - rtl8169sce_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_07: - case RTL_GIGA_MAC_VER_08: - case RTL_GIGA_MAC_VER_09: - rtl8102e_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_11: - rtl8168bb_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_12: - rtl8168bef_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_17: - rtl8168bef_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_18: - rtl8168cp_1_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_19: - rtl8168c_1_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_20: - rtl8168c_2_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_21: - rtl8168c_3_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_22: - rtl8168c_4_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_23: - case RTL_GIGA_MAC_VER_24: - rtl8168cp_2_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_25: - rtl8168d_1_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_26: - rtl8168d_2_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_27: - rtl8168d_3_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_28: - rtl8168d_4_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_29: - case RTL_GIGA_MAC_VER_30: - rtl8105e_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_31: - /* None. */ - break; - case RTL_GIGA_MAC_VER_32: - case RTL_GIGA_MAC_VER_33: - rtl8168e_1_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_34: - rtl8168e_2_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_35: - rtl8168f_1_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_36: - rtl8168f_2_hw_phy_config(tp); - break; - - case RTL_GIGA_MAC_VER_37: - rtl8402_hw_phy_config(tp); - break; - - case RTL_GIGA_MAC_VER_38: - rtl8411_hw_phy_config(tp); - break; - - case RTL_GIGA_MAC_VER_39: - rtl8106e_hw_phy_config(tp); - break; - - case RTL_GIGA_MAC_VER_40: - rtl8168g_1_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_42: - case RTL_GIGA_MAC_VER_43: - case RTL_GIGA_MAC_VER_44: - rtl8168g_2_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_45: - case RTL_GIGA_MAC_VER_47: - rtl8168h_1_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_46: - case RTL_GIGA_MAC_VER_48: - rtl8168h_2_hw_phy_config(tp); - break; - - case RTL_GIGA_MAC_VER_49: - rtl8168ep_1_hw_phy_config(tp); - break; - case RTL_GIGA_MAC_VER_50: - case RTL_GIGA_MAC_VER_51: - rtl8168ep_2_hw_phy_config(tp); - break; - - case RTL_GIGA_MAC_VER_41: - default: - break; - } + if (phy_configs[tp->mac_version]) + phy_configs[tp->mac_version](tp); } static void rtl_schedule_task(struct rtl8169_private *tp, enum rtl_flag flag) @@ -4147,14 +4071,6 @@ static void rtl8169_init_phy(struct net_device *dev, struct rtl8169_private *tp) phy_speed_up(tp->phydev); genphy_soft_reset(tp->phydev); - - /* It was reported that several chips end up with 10MBit/Half on a - * 1GBit link after resuming from S3. For whatever reason the PHY on - * these chips doesn't properly start a renegotiation when soft-reset. - * Explicitly requesting a renegotiation fixes this. - */ - if (tp->phydev->autoneg == AUTONEG_ENABLE) - phy_restart_aneg(tp->phydev); } static void rtl_rar_set(struct rtl8169_private *tp, u8 *addr) @@ -4283,8 +4199,7 @@ static void r8168_pll_power_down(struct rtl8169_private *tp) case RTL_GIGA_MAC_VER_40: case RTL_GIGA_MAC_VER_41: case RTL_GIGA_MAC_VER_49: - rtl_w0w1_eri(tp, 0x1a8, ERIAR_MASK_1111, 0x00000000, - 0xfc000000, ERIAR_EXGMAC); + rtl_eri_clear_bits(tp, 0x1a8, ERIAR_MASK_1111, 0xfc000000); RTL_W8(tp, PMCH, RTL_R8(tp, PMCH) & ~0x80); break; } @@ -4312,8 +4227,7 @@ static void r8168_pll_power_up(struct rtl8169_private *tp) case RTL_GIGA_MAC_VER_41: case RTL_GIGA_MAC_VER_49: RTL_W8(tp, PMCH, RTL_R8(tp, PMCH) | 0xc0); - rtl_w0w1_eri(tp, 0x1a8, ERIAR_MASK_1111, 0xfc000000, - 0x00000000, ERIAR_EXGMAC); + rtl_eri_set_bits(tp, 0x1a8, ERIAR_MASK_1111, 0xfc000000); break; } @@ -4703,6 +4617,8 @@ static void rtl_hw_start(struct rtl8169_private *tp) rtl_set_rx_tx_desc_registers(tp); rtl_lock_config_regs(tp); + /* disable interrupt coalescing */ + RTL_W16(tp, IntrMitigate, 0x0000); /* Initially a 10 us delay. Turned it into a PCI commit. - FR */ RTL_R8(tp, IntrMask); RTL_W8(tp, ChipCmd, CmdTxEnb | CmdRxEnb); @@ -4735,12 +4651,6 @@ static void rtl_hw_start_8169(struct rtl8169_private *tp) rtl8169_set_magic_reg(tp, tp->mac_version); - /* - * Undocumented corner. Supposedly: - * (TxTimer << 12) | (TxPackets << 8) | (RxTimer << 4) | RxPackets - */ - RTL_W16(tp, IntrMitigate, 0x0000); - RTL_W32(tp, RxMissed, 0); } @@ -4801,8 +4711,8 @@ struct ephy_info { u16 bits; }; -static void rtl_ephy_init(struct rtl8169_private *tp, const struct ephy_info *e, - int len) +static void __rtl_ephy_init(struct rtl8169_private *tp, + const struct ephy_info *e, int len) { u16 w; @@ -4813,6 +4723,8 @@ static void rtl_ephy_init(struct rtl8169_private *tp, const struct ephy_info *e, } } +#define rtl_ephy_init(tp, a) __rtl_ephy_init(tp, a, ARRAY_SIZE(a)) + static void rtl_disable_clock_request(struct rtl8169_private *tp) { pcie_capability_clear_word(tp->pci_dev, PCI_EXP_LNKCTL, @@ -4844,6 +4756,24 @@ static void rtl_hw_aspm_clkreq_enable(struct rtl8169_private *tp, bool enable) udelay(10); } +static void rtl_set_fifo_size(struct rtl8169_private *tp, u16 rx_stat, + u16 tx_stat, u16 rx_dyn, u16 tx_dyn) +{ + /* Usage of dynamic vs. static FIFO is controlled by bit + * TXCFG_AUTO_FIFO. Exact meaning of FIFO values isn't known. + */ + rtl_eri_write(tp, 0xc8, ERIAR_MASK_1111, (rx_stat << 16) | rx_dyn); + rtl_eri_write(tp, 0xe8, ERIAR_MASK_1111, (tx_stat << 16) | tx_dyn); +} + +static void rtl8168g_set_pause_thresholds(struct rtl8169_private *tp, + u8 low, u8 high) +{ + /* FIFO thresholds for pause flow control */ + rtl_eri_write(tp, 0xcc, ERIAR_MASK_0001, low); + rtl_eri_write(tp, 0xd0, ERIAR_MASK_0001, high); +} + static void rtl_hw_start_8168bb(struct rtl8169_private *tp) { RTL_W8(tp, Config3, RTL_R8(tp, Config3) & ~Beacon_en); @@ -4893,7 +4823,7 @@ static void rtl_hw_start_8168cp_1(struct rtl8169_private *tp) rtl_set_def_aspm_entry_latency(tp); - rtl_ephy_init(tp, e_info_8168cp, ARRAY_SIZE(e_info_8168cp)); + rtl_ephy_init(tp, e_info_8168cp); __rtl_hw_start_8168cp(tp); } @@ -4941,7 +4871,7 @@ static void rtl_hw_start_8168c_1(struct rtl8169_private *tp) RTL_W8(tp, DBG_REG, 0x06 | FIX_NAK_1 | FIX_NAK_2); - rtl_ephy_init(tp, e_info_8168c_1, ARRAY_SIZE(e_info_8168c_1)); + rtl_ephy_init(tp, e_info_8168c_1); __rtl_hw_start_8168cp(tp); } @@ -4955,7 +4885,7 @@ static void rtl_hw_start_8168c_2(struct rtl8169_private *tp) rtl_set_def_aspm_entry_latency(tp); - rtl_ephy_init(tp, e_info_8168c_2, ARRAY_SIZE(e_info_8168c_2)); + rtl_ephy_init(tp, e_info_8168c_2); __rtl_hw_start_8168cp(tp); } @@ -5013,7 +4943,7 @@ static void rtl_hw_start_8168d_4(struct rtl8169_private *tp) RTL_W8(tp, MaxTxPacketSize, TxPacketMax); - rtl_ephy_init(tp, e_info_8168d_4, ARRAY_SIZE(e_info_8168d_4)); + rtl_ephy_init(tp, e_info_8168d_4); rtl_enable_clock_request(tp); } @@ -5038,7 +4968,7 @@ static void rtl_hw_start_8168e_1(struct rtl8169_private *tp) rtl_set_def_aspm_entry_latency(tp); - rtl_ephy_init(tp, e_info_8168e_1, ARRAY_SIZE(e_info_8168e_1)); + rtl_ephy_init(tp, e_info_8168e_1); if (tp->dev->mtu <= ETH_DATA_LEN) rtl_tx_performance_tweak(tp, PCI_EXP_DEVCTL_READRQ_4096B); @@ -5063,19 +4993,18 @@ static void rtl_hw_start_8168e_2(struct rtl8169_private *tp) rtl_set_def_aspm_entry_latency(tp); - rtl_ephy_init(tp, e_info_8168e_2, ARRAY_SIZE(e_info_8168e_2)); + rtl_ephy_init(tp, e_info_8168e_2); if (tp->dev->mtu <= ETH_DATA_LEN) rtl_tx_performance_tweak(tp, PCI_EXP_DEVCTL_READRQ_4096B); - rtl_eri_write(tp, 0xc0, ERIAR_MASK_0011, 0x0000, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xb8, ERIAR_MASK_0011, 0x0000, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xc8, ERIAR_MASK_1111, 0x00100002, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xe8, ERIAR_MASK_1111, 0x00100006, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xcc, ERIAR_MASK_1111, 0x00000050, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xd0, ERIAR_MASK_1111, 0x07ff0060, ERIAR_EXGMAC); - rtl_w0w1_eri(tp, 0x1b0, ERIAR_MASK_0001, 0x10, 0x00, ERIAR_EXGMAC); - rtl_w0w1_eri(tp, 0x0d4, ERIAR_MASK_0011, 0x0c00, 0xff00, ERIAR_EXGMAC); + rtl_eri_write(tp, 0xc0, ERIAR_MASK_0011, 0x0000); + rtl_eri_write(tp, 0xb8, ERIAR_MASK_0011, 0x0000); + rtl_set_fifo_size(tp, 0x10, 0x10, 0x02, 0x06); + rtl_eri_write(tp, 0xcc, ERIAR_MASK_1111, 0x00000050); + rtl_eri_write(tp, 0xd0, ERIAR_MASK_1111, 0x07ff0060); + rtl_eri_set_bits(tp, 0x1b0, ERIAR_MASK_0001, BIT(4)); + rtl_w0w1_eri(tp, 0x0d4, ERIAR_MASK_0011, 0x0c00, 0xff00); RTL_W8(tp, MaxTxPacketSize, EarlySize); @@ -5083,9 +5012,6 @@ static void rtl_hw_start_8168e_2(struct rtl8169_private *tp) RTL_W8(tp, MCU, RTL_R8(tp, MCU) & ~NOW_IS_OOB); - /* Adjust EEE LED frequency */ - RTL_W8(tp, EEE_LED, RTL_R8(tp, EEE_LED) & ~0x07); - rtl8168_config_eee_mac(tp); RTL_W8(tp, DLLPR, RTL_R8(tp, DLLPR) | PFM_EN); @@ -5101,16 +5027,14 @@ static void rtl_hw_start_8168f(struct rtl8169_private *tp) rtl_tx_performance_tweak(tp, PCI_EXP_DEVCTL_READRQ_4096B); - rtl_eri_write(tp, 0xc0, ERIAR_MASK_0011, 0x0000, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xb8, ERIAR_MASK_0011, 0x0000, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xc8, ERIAR_MASK_1111, 0x00100002, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xe8, ERIAR_MASK_1111, 0x00100006, ERIAR_EXGMAC); - rtl_w0w1_eri(tp, 0xdc, ERIAR_MASK_0001, 0x00, 0x01, ERIAR_EXGMAC); - rtl_w0w1_eri(tp, 0xdc, ERIAR_MASK_0001, 0x01, 0x00, ERIAR_EXGMAC); - rtl_w0w1_eri(tp, 0x1b0, ERIAR_MASK_0001, 0x10, 0x00, ERIAR_EXGMAC); - rtl_w0w1_eri(tp, 0x1d0, ERIAR_MASK_0001, 0x10, 0x00, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xcc, ERIAR_MASK_1111, 0x00000050, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xd0, ERIAR_MASK_1111, 0x00000060, ERIAR_EXGMAC); + rtl_eri_write(tp, 0xc0, ERIAR_MASK_0011, 0x0000); + rtl_eri_write(tp, 0xb8, ERIAR_MASK_0011, 0x0000); + rtl_set_fifo_size(tp, 0x10, 0x10, 0x02, 0x06); + rtl_reset_packet_filter(tp); + rtl_eri_set_bits(tp, 0x1b0, ERIAR_MASK_0001, BIT(4)); + rtl_eri_set_bits(tp, 0x1d0, ERIAR_MASK_0001, BIT(4)); + rtl_eri_write(tp, 0xcc, ERIAR_MASK_1111, 0x00000050); + rtl_eri_write(tp, 0xd0, ERIAR_MASK_1111, 0x00000060); RTL_W8(tp, MaxTxPacketSize, EarlySize); @@ -5135,12 +5059,9 @@ static void rtl_hw_start_8168f_1(struct rtl8169_private *tp) rtl_hw_start_8168f(tp); - rtl_ephy_init(tp, e_info_8168f_1, ARRAY_SIZE(e_info_8168f_1)); - - rtl_w0w1_eri(tp, 0x0d4, ERIAR_MASK_0011, 0x0c00, 0xff00, ERIAR_EXGMAC); + rtl_ephy_init(tp, e_info_8168f_1); - /* Adjust EEE LED frequency */ - RTL_W8(tp, EEE_LED, RTL_R8(tp, EEE_LED) & ~0x07); + rtl_w0w1_eri(tp, 0x0d4, ERIAR_MASK_0011, 0x0c00, 0xff00); } static void rtl_hw_start_8411(struct rtl8169_private *tp) @@ -5155,39 +5076,33 @@ static void rtl_hw_start_8411(struct rtl8169_private *tp) rtl_hw_start_8168f(tp); rtl_pcie_state_l2l3_disable(tp); - rtl_ephy_init(tp, e_info_8168f_1, ARRAY_SIZE(e_info_8168f_1)); + rtl_ephy_init(tp, e_info_8168f_1); - rtl_w0w1_eri(tp, 0x0d4, ERIAR_MASK_0011, 0x0c00, 0x0000, ERIAR_EXGMAC); + rtl_eri_set_bits(tp, 0x0d4, ERIAR_MASK_0011, 0x0c00); } static void rtl_hw_start_8168g(struct rtl8169_private *tp) { - rtl_eri_write(tp, 0xc8, ERIAR_MASK_0101, 0x080002, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xcc, ERIAR_MASK_0001, 0x38, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xd0, ERIAR_MASK_0001, 0x48, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xe8, ERIAR_MASK_1111, 0x00100006, ERIAR_EXGMAC); + rtl_set_fifo_size(tp, 0x08, 0x10, 0x02, 0x06); + rtl8168g_set_pause_thresholds(tp, 0x38, 0x48); rtl_set_def_aspm_entry_latency(tp); rtl_tx_performance_tweak(tp, PCI_EXP_DEVCTL_READRQ_4096B); - rtl_w0w1_eri(tp, 0xdc, ERIAR_MASK_0001, 0x00, 0x01, ERIAR_EXGMAC); - rtl_w0w1_eri(tp, 0xdc, ERIAR_MASK_0001, 0x01, 0x00, ERIAR_EXGMAC); - rtl_eri_write(tp, 0x2f8, ERIAR_MASK_0011, 0x1d8f, ERIAR_EXGMAC); + rtl_reset_packet_filter(tp); + rtl_eri_write(tp, 0x2f8, ERIAR_MASK_0011, 0x1d8f); RTL_W32(tp, MISC, RTL_R32(tp, MISC) & ~RXDV_GATED_EN); RTL_W8(tp, MaxTxPacketSize, EarlySize); - rtl_eri_write(tp, 0xc0, ERIAR_MASK_0011, 0x0000, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xb8, ERIAR_MASK_0011, 0x0000, ERIAR_EXGMAC); - - /* Adjust EEE LED frequency */ - RTL_W8(tp, EEE_LED, RTL_R8(tp, EEE_LED) & ~0x07); + rtl_eri_write(tp, 0xc0, ERIAR_MASK_0011, 0x0000); + rtl_eri_write(tp, 0xb8, ERIAR_MASK_0011, 0x0000); rtl8168_config_eee_mac(tp); - rtl_w0w1_eri(tp, 0x2fc, ERIAR_MASK_0001, 0x01, 0x06, ERIAR_EXGMAC); - rtl_w0w1_eri(tp, 0x1b0, ERIAR_MASK_0011, 0x0000, 0x1000, ERIAR_EXGMAC); + rtl_w0w1_eri(tp, 0x2fc, ERIAR_MASK_0001, 0x01, 0x06); + rtl_eri_clear_bits(tp, 0x1b0, ERIAR_MASK_0011, BIT(12)); rtl_pcie_state_l2l3_disable(tp); } @@ -5205,7 +5120,7 @@ static void rtl_hw_start_8168g_1(struct rtl8169_private *tp) /* disable aspm and clock request before access ephy */ rtl_hw_aspm_clkreq_enable(tp, false); - rtl_ephy_init(tp, e_info_8168g_1, ARRAY_SIZE(e_info_8168g_1)); + rtl_ephy_init(tp, e_info_8168g_1); rtl_hw_aspm_clkreq_enable(tp, true); } @@ -5223,7 +5138,7 @@ static void rtl_hw_start_8168g_2(struct rtl8169_private *tp) /* disable aspm and clock request before access ephy */ RTL_W8(tp, Config2, RTL_R8(tp, Config2) & ~ClkReqEn); RTL_W8(tp, Config5, RTL_R8(tp, Config5) & ~ASPM_en); - rtl_ephy_init(tp, e_info_8168g_2, ARRAY_SIZE(e_info_8168g_2)); + rtl_ephy_init(tp, e_info_8168g_2); } static void rtl_hw_start_8411_2(struct rtl8169_private *tp) @@ -5240,7 +5155,7 @@ static void rtl_hw_start_8411_2(struct rtl8169_private *tp) /* disable aspm and clock request before access ephy */ rtl_hw_aspm_clkreq_enable(tp, false); - rtl_ephy_init(tp, e_info_8411_2, ARRAY_SIZE(e_info_8411_2)); + rtl_ephy_init(tp, e_info_8411_2); rtl_hw_aspm_clkreq_enable(tp, true); } @@ -5259,34 +5174,28 @@ static void rtl_hw_start_8168h_1(struct rtl8169_private *tp) /* disable aspm and clock request before access ephy */ rtl_hw_aspm_clkreq_enable(tp, false); - rtl_ephy_init(tp, e_info_8168h_1, ARRAY_SIZE(e_info_8168h_1)); + rtl_ephy_init(tp, e_info_8168h_1); - rtl_eri_write(tp, 0xc8, ERIAR_MASK_0101, 0x00080002, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xcc, ERIAR_MASK_0001, 0x38, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xd0, ERIAR_MASK_0001, 0x48, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xe8, ERIAR_MASK_1111, 0x00100006, ERIAR_EXGMAC); + rtl_set_fifo_size(tp, 0x08, 0x10, 0x02, 0x06); + rtl8168g_set_pause_thresholds(tp, 0x38, 0x48); rtl_set_def_aspm_entry_latency(tp); rtl_tx_performance_tweak(tp, PCI_EXP_DEVCTL_READRQ_4096B); - rtl_w0w1_eri(tp, 0xdc, ERIAR_MASK_0001, 0x00, 0x01, ERIAR_EXGMAC); - rtl_w0w1_eri(tp, 0xdc, ERIAR_MASK_0001, 0x01, 0x00, ERIAR_EXGMAC); + rtl_reset_packet_filter(tp); - rtl_w0w1_eri(tp, 0xdc, ERIAR_MASK_1111, 0x0010, 0x00, ERIAR_EXGMAC); + rtl_eri_set_bits(tp, 0xdc, ERIAR_MASK_1111, BIT(4)); - rtl_w0w1_eri(tp, 0xd4, ERIAR_MASK_1111, 0x1f00, 0x00, ERIAR_EXGMAC); + rtl_eri_set_bits(tp, 0xd4, ERIAR_MASK_1111, 0x1f00); - rtl_eri_write(tp, 0x5f0, ERIAR_MASK_0011, 0x4f87, ERIAR_EXGMAC); + rtl_eri_write(tp, 0x5f0, ERIAR_MASK_0011, 0x4f87); RTL_W32(tp, MISC, RTL_R32(tp, MISC) & ~RXDV_GATED_EN); RTL_W8(tp, MaxTxPacketSize, EarlySize); - rtl_eri_write(tp, 0xc0, ERIAR_MASK_0011, 0x0000, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xb8, ERIAR_MASK_0011, 0x0000, ERIAR_EXGMAC); - - /* Adjust EEE LED frequency */ - RTL_W8(tp, EEE_LED, RTL_R8(tp, EEE_LED) & ~0x07); + rtl_eri_write(tp, 0xc0, ERIAR_MASK_0011, 0x0000); + rtl_eri_write(tp, 0xb8, ERIAR_MASK_0011, 0x0000); rtl8168_config_eee_mac(tp); @@ -5295,7 +5204,7 @@ static void rtl_hw_start_8168h_1(struct rtl8169_private *tp) RTL_W8(tp, DLLPR, RTL_R8(tp, DLLPR) & ~TX_10M_PS_EN); - rtl_w0w1_eri(tp, 0x1b0, ERIAR_MASK_0011, 0x0000, 0x1000, ERIAR_EXGMAC); + rtl_eri_clear_bits(tp, 0x1b0, ERIAR_MASK_0011, BIT(12)); rtl_pcie_state_l2l3_disable(tp); @@ -5345,34 +5254,28 @@ static void rtl_hw_start_8168ep(struct rtl8169_private *tp) { rtl8168ep_stop_cmac(tp); - rtl_eri_write(tp, 0xc8, ERIAR_MASK_0101, 0x00080002, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xcc, ERIAR_MASK_0001, 0x2f, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xd0, ERIAR_MASK_0001, 0x5f, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xe8, ERIAR_MASK_1111, 0x00100006, ERIAR_EXGMAC); + rtl_set_fifo_size(tp, 0x08, 0x10, 0x02, 0x06); + rtl8168g_set_pause_thresholds(tp, 0x2f, 0x5f); rtl_set_def_aspm_entry_latency(tp); rtl_tx_performance_tweak(tp, PCI_EXP_DEVCTL_READRQ_4096B); - rtl_w0w1_eri(tp, 0xdc, ERIAR_MASK_0001, 0x00, 0x01, ERIAR_EXGMAC); - rtl_w0w1_eri(tp, 0xdc, ERIAR_MASK_0001, 0x01, 0x00, ERIAR_EXGMAC); + rtl_reset_packet_filter(tp); - rtl_w0w1_eri(tp, 0xd4, ERIAR_MASK_1111, 0x1f80, 0x00, ERIAR_EXGMAC); + rtl_eri_set_bits(tp, 0xd4, ERIAR_MASK_1111, 0x1f80); - rtl_eri_write(tp, 0x5f0, ERIAR_MASK_0011, 0x4f87, ERIAR_EXGMAC); + rtl_eri_write(tp, 0x5f0, ERIAR_MASK_0011, 0x4f87); RTL_W32(tp, MISC, RTL_R32(tp, MISC) & ~RXDV_GATED_EN); RTL_W8(tp, MaxTxPacketSize, EarlySize); - rtl_eri_write(tp, 0xc0, ERIAR_MASK_0011, 0x0000, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xb8, ERIAR_MASK_0011, 0x0000, ERIAR_EXGMAC); - - /* Adjust EEE LED frequency */ - RTL_W8(tp, EEE_LED, RTL_R8(tp, EEE_LED) & ~0x07); + rtl_eri_write(tp, 0xc0, ERIAR_MASK_0011, 0x0000); + rtl_eri_write(tp, 0xb8, ERIAR_MASK_0011, 0x0000); rtl8168_config_eee_mac(tp); - rtl_w0w1_eri(tp, 0x2fc, ERIAR_MASK_0001, 0x01, 0x06, ERIAR_EXGMAC); + rtl_w0w1_eri(tp, 0x2fc, ERIAR_MASK_0001, 0x01, 0x06); RTL_W8(tp, DLLPR, RTL_R8(tp, DLLPR) & ~TX_10M_PS_EN); @@ -5391,7 +5294,7 @@ static void rtl_hw_start_8168ep_1(struct rtl8169_private *tp) /* disable aspm and clock request before access ephy */ rtl_hw_aspm_clkreq_enable(tp, false); - rtl_ephy_init(tp, e_info_8168ep_1, ARRAY_SIZE(e_info_8168ep_1)); + rtl_ephy_init(tp, e_info_8168ep_1); rtl_hw_start_8168ep(tp); @@ -5408,7 +5311,7 @@ static void rtl_hw_start_8168ep_2(struct rtl8169_private *tp) /* disable aspm and clock request before access ephy */ rtl_hw_aspm_clkreq_enable(tp, false); - rtl_ephy_init(tp, e_info_8168ep_2, ARRAY_SIZE(e_info_8168ep_2)); + rtl_ephy_init(tp, e_info_8168ep_2); rtl_hw_start_8168ep(tp); @@ -5430,7 +5333,7 @@ static void rtl_hw_start_8168ep_3(struct rtl8169_private *tp) /* disable aspm and clock request before access ephy */ rtl_hw_aspm_clkreq_enable(tp, false); - rtl_ephy_init(tp, e_info_8168ep_3, ARRAY_SIZE(e_info_8168ep_3)); + rtl_ephy_init(tp, e_info_8168ep_3); rtl_hw_start_8168ep(tp); @@ -5453,128 +5356,6 @@ static void rtl_hw_start_8168ep_3(struct rtl8169_private *tp) rtl_hw_aspm_clkreq_enable(tp, true); } -static void rtl_hw_start_8168(struct rtl8169_private *tp) -{ - RTL_W8(tp, MaxTxPacketSize, TxPacketMax); - - tp->cp_cmd &= ~INTT_MASK; - tp->cp_cmd |= PktCntrDisable | INTT_1; - RTL_W16(tp, CPlusCmd, tp->cp_cmd); - - RTL_W16(tp, IntrMitigate, 0x5100); - - /* Work around for RxFIFO overflow. */ - if (tp->mac_version == RTL_GIGA_MAC_VER_11) { - tp->irq_mask |= RxFIFOOver; - tp->irq_mask &= ~RxOverflow; - } - - switch (tp->mac_version) { - case RTL_GIGA_MAC_VER_11: - rtl_hw_start_8168bb(tp); - break; - - case RTL_GIGA_MAC_VER_12: - case RTL_GIGA_MAC_VER_17: - rtl_hw_start_8168bef(tp); - break; - - case RTL_GIGA_MAC_VER_18: - rtl_hw_start_8168cp_1(tp); - break; - - case RTL_GIGA_MAC_VER_19: - rtl_hw_start_8168c_1(tp); - break; - - case RTL_GIGA_MAC_VER_20: - rtl_hw_start_8168c_2(tp); - break; - - case RTL_GIGA_MAC_VER_21: - rtl_hw_start_8168c_3(tp); - break; - - case RTL_GIGA_MAC_VER_22: - rtl_hw_start_8168c_4(tp); - break; - - case RTL_GIGA_MAC_VER_23: - rtl_hw_start_8168cp_2(tp); - break; - - case RTL_GIGA_MAC_VER_24: - rtl_hw_start_8168cp_3(tp); - break; - - case RTL_GIGA_MAC_VER_25: - case RTL_GIGA_MAC_VER_26: - case RTL_GIGA_MAC_VER_27: - rtl_hw_start_8168d(tp); - break; - - case RTL_GIGA_MAC_VER_28: - rtl_hw_start_8168d_4(tp); - break; - - case RTL_GIGA_MAC_VER_31: - rtl_hw_start_8168dp(tp); - break; - - case RTL_GIGA_MAC_VER_32: - case RTL_GIGA_MAC_VER_33: - rtl_hw_start_8168e_1(tp); - break; - case RTL_GIGA_MAC_VER_34: - rtl_hw_start_8168e_2(tp); - break; - - case RTL_GIGA_MAC_VER_35: - case RTL_GIGA_MAC_VER_36: - rtl_hw_start_8168f_1(tp); - break; - - case RTL_GIGA_MAC_VER_38: - rtl_hw_start_8411(tp); - break; - - case RTL_GIGA_MAC_VER_40: - case RTL_GIGA_MAC_VER_41: - rtl_hw_start_8168g_1(tp); - break; - case RTL_GIGA_MAC_VER_42: - rtl_hw_start_8168g_2(tp); - break; - - case RTL_GIGA_MAC_VER_44: - rtl_hw_start_8411_2(tp); - break; - - case RTL_GIGA_MAC_VER_45: - case RTL_GIGA_MAC_VER_46: - rtl_hw_start_8168h_1(tp); - break; - - case RTL_GIGA_MAC_VER_49: - rtl_hw_start_8168ep_1(tp); - break; - - case RTL_GIGA_MAC_VER_50: - rtl_hw_start_8168ep_2(tp); - break; - - case RTL_GIGA_MAC_VER_51: - rtl_hw_start_8168ep_3(tp); - break; - - default: - netif_err(tp, drv, tp->dev, - "unknown chipset (mac_version = %d)\n", - tp->mac_version); - break; - } -} - static void rtl_hw_start_8102e_1(struct rtl8169_private *tp) { static const struct ephy_info e_info_8102e_1[] = { @@ -5603,7 +5384,7 @@ static void rtl_hw_start_8102e_1(struct rtl8169_private *tp) if ((cfg1 & LEDS0) && (cfg1 & LEDS1)) RTL_W8(tp, Config1, cfg1 & ~LEDS0); - rtl_ephy_init(tp, e_info_8102e_1, ARRAY_SIZE(e_info_8102e_1)); + rtl_ephy_init(tp, e_info_8102e_1); } static void rtl_hw_start_8102e_2(struct rtl8169_private *tp) @@ -5645,7 +5426,7 @@ static void rtl_hw_start_8105e_1(struct rtl8169_private *tp) RTL_W8(tp, MCU, RTL_R8(tp, MCU) | EN_NDP | EN_OOB_RESET); RTL_W8(tp, DLLPR, RTL_R8(tp, DLLPR) | PFM_EN); - rtl_ephy_init(tp, e_info_8105e_1, ARRAY_SIZE(e_info_8105e_1)); + rtl_ephy_init(tp, e_info_8105e_1); rtl_pcie_state_l2l3_disable(tp); } @@ -5670,17 +5451,15 @@ static void rtl_hw_start_8402(struct rtl8169_private *tp) RTL_W8(tp, MCU, RTL_R8(tp, MCU) & ~NOW_IS_OOB); - rtl_ephy_init(tp, e_info_8402, ARRAY_SIZE(e_info_8402)); + rtl_ephy_init(tp, e_info_8402); rtl_tx_performance_tweak(tp, PCI_EXP_DEVCTL_READRQ_4096B); - rtl_eri_write(tp, 0xc8, ERIAR_MASK_1111, 0x00000002, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xe8, ERIAR_MASK_1111, 0x00000006, ERIAR_EXGMAC); - rtl_w0w1_eri(tp, 0xdc, ERIAR_MASK_0001, 0x00, 0x01, ERIAR_EXGMAC); - rtl_w0w1_eri(tp, 0xdc, ERIAR_MASK_0001, 0x01, 0x00, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xc0, ERIAR_MASK_0011, 0x0000, ERIAR_EXGMAC); - rtl_eri_write(tp, 0xb8, ERIAR_MASK_0011, 0x0000, ERIAR_EXGMAC); - rtl_w0w1_eri(tp, 0x0d4, ERIAR_MASK_0011, 0x0e00, 0xff00, ERIAR_EXGMAC); + rtl_set_fifo_size(tp, 0x00, 0x00, 0x02, 0x06); + rtl_reset_packet_filter(tp); + rtl_eri_write(tp, 0xc0, ERIAR_MASK_0011, 0x0000); + rtl_eri_write(tp, 0xb8, ERIAR_MASK_0011, 0x0000); + rtl_w0w1_eri(tp, 0x0d4, ERIAR_MASK_0011, 0x0e00, 0xff00); rtl_pcie_state_l2l3_disable(tp); } @@ -5700,6 +5479,73 @@ static void rtl_hw_start_8106(struct rtl8169_private *tp) rtl_hw_aspm_clkreq_enable(tp, true); } +static void rtl_hw_config(struct rtl8169_private *tp) +{ + static const rtl_generic_fct hw_configs[] = { + [RTL_GIGA_MAC_VER_07] = rtl_hw_start_8102e_1, + [RTL_GIGA_MAC_VER_08] = rtl_hw_start_8102e_3, + [RTL_GIGA_MAC_VER_09] = rtl_hw_start_8102e_2, + [RTL_GIGA_MAC_VER_10] = NULL, + [RTL_GIGA_MAC_VER_11] = rtl_hw_start_8168bb, + [RTL_GIGA_MAC_VER_12] = rtl_hw_start_8168bef, + [RTL_GIGA_MAC_VER_13] = NULL, + [RTL_GIGA_MAC_VER_14] = NULL, + [RTL_GIGA_MAC_VER_15] = NULL, + [RTL_GIGA_MAC_VER_16] = NULL, + [RTL_GIGA_MAC_VER_17] = rtl_hw_start_8168bef, + [RTL_GIGA_MAC_VER_18] = rtl_hw_start_8168cp_1, + [RTL_GIGA_MAC_VER_19] = rtl_hw_start_8168c_1, + [RTL_GIGA_MAC_VER_20] = rtl_hw_start_8168c_2, + [RTL_GIGA_MAC_VER_21] = rtl_hw_start_8168c_3, + [RTL_GIGA_MAC_VER_22] = rtl_hw_start_8168c_4, + [RTL_GIGA_MAC_VER_23] = rtl_hw_start_8168cp_2, + [RTL_GIGA_MAC_VER_24] = rtl_hw_start_8168cp_3, + [RTL_GIGA_MAC_VER_25] = rtl_hw_start_8168d, + [RTL_GIGA_MAC_VER_26] = rtl_hw_start_8168d, + [RTL_GIGA_MAC_VER_27] = rtl_hw_start_8168d, + [RTL_GIGA_MAC_VER_28] = rtl_hw_start_8168d_4, + [RTL_GIGA_MAC_VER_29] = rtl_hw_start_8105e_1, + [RTL_GIGA_MAC_VER_30] = rtl_hw_start_8105e_2, + [RTL_GIGA_MAC_VER_31] = rtl_hw_start_8168dp, + [RTL_GIGA_MAC_VER_32] = rtl_hw_start_8168e_1, + [RTL_GIGA_MAC_VER_33] = rtl_hw_start_8168e_1, + [RTL_GIGA_MAC_VER_34] = rtl_hw_start_8168e_2, + [RTL_GIGA_MAC_VER_35] = rtl_hw_start_8168f_1, + [RTL_GIGA_MAC_VER_36] = rtl_hw_start_8168f_1, + [RTL_GIGA_MAC_VER_37] = rtl_hw_start_8402, + [RTL_GIGA_MAC_VER_38] = rtl_hw_start_8411, + [RTL_GIGA_MAC_VER_39] = rtl_hw_start_8106, + [RTL_GIGA_MAC_VER_40] = rtl_hw_start_8168g_1, + [RTL_GIGA_MAC_VER_41] = rtl_hw_start_8168g_1, + [RTL_GIGA_MAC_VER_42] = rtl_hw_start_8168g_2, + [RTL_GIGA_MAC_VER_43] = rtl_hw_start_8168g_2, + [RTL_GIGA_MAC_VER_44] = rtl_hw_start_8411_2, + [RTL_GIGA_MAC_VER_45] = rtl_hw_start_8168h_1, + [RTL_GIGA_MAC_VER_46] = rtl_hw_start_8168h_1, + [RTL_GIGA_MAC_VER_47] = rtl_hw_start_8168h_1, + [RTL_GIGA_MAC_VER_48] = rtl_hw_start_8168h_1, + [RTL_GIGA_MAC_VER_49] = rtl_hw_start_8168ep_1, + [RTL_GIGA_MAC_VER_50] = rtl_hw_start_8168ep_2, + [RTL_GIGA_MAC_VER_51] = rtl_hw_start_8168ep_3, + }; + + if (hw_configs[tp->mac_version]) + hw_configs[tp->mac_version](tp); +} + +static void rtl_hw_start_8168(struct rtl8169_private *tp) +{ + RTL_W8(tp, MaxTxPacketSize, TxPacketMax); + + /* Workaround for RxFIFO overflow. */ + if (tp->mac_version == RTL_GIGA_MAC_VER_11) { + tp->irq_mask |= RxFIFOOver; + tp->irq_mask &= ~RxOverflow; + } + + rtl_hw_config(tp); +} + static void rtl_hw_start_8101(struct rtl8169_private *tp) { if (tp->mac_version >= RTL_GIGA_MAC_VER_30) @@ -5715,43 +5561,7 @@ static void rtl_hw_start_8101(struct rtl8169_private *tp) tp->cp_cmd &= CPCMD_QUIRK_MASK; RTL_W16(tp, CPlusCmd, tp->cp_cmd); - switch (tp->mac_version) { - case RTL_GIGA_MAC_VER_07: - rtl_hw_start_8102e_1(tp); - break; - - case RTL_GIGA_MAC_VER_08: - rtl_hw_start_8102e_3(tp); - break; - - case RTL_GIGA_MAC_VER_09: - rtl_hw_start_8102e_2(tp); - break; - - case RTL_GIGA_MAC_VER_29: - rtl_hw_start_8105e_1(tp); - break; - case RTL_GIGA_MAC_VER_30: - rtl_hw_start_8105e_2(tp); - break; - - case RTL_GIGA_MAC_VER_37: - rtl_hw_start_8402(tp); - break; - - case RTL_GIGA_MAC_VER_39: - rtl_hw_start_8106(tp); - break; - case RTL_GIGA_MAC_VER_43: - rtl_hw_start_8168g_2(tp); - break; - case RTL_GIGA_MAC_VER_47: - case RTL_GIGA_MAC_VER_48: - rtl_hw_start_8168h_1(tp); - break; - } - - RTL_W16(tp, IntrMitigate, 0x0000); + rtl_hw_config(tp); } static int rtl8169_change_mtu(struct net_device *dev, int new_mtu) @@ -6268,7 +6078,7 @@ static netdev_tx_t rtl8169_start_xmit(struct sk_buff *skb, */ smp_mb(); if (rtl_tx_slots_avail(tp, MAX_SKB_FRAGS)) - netif_wake_queue(dev); + netif_start_queue(dev); } return NETDEV_TX_OK; @@ -6543,10 +6353,8 @@ static irqreturn_t rtl8169_interrupt(int irq, void *dev_instance) set_bit(RTL_FLAG_TASK_RESET_PENDING, tp->wk.flags); } - if (status & (RTL_EVENT_NAPI | LinkChg)) { - rtl_irq_disable(tp); - napi_schedule_irqoff(&tp->napi); - } + rtl_irq_disable(tp); + napi_schedule_irqoff(&tp->napi); out: rtl_ack_events(tp, status); @@ -6645,8 +6453,7 @@ static int r8169_phy_connect(struct rtl8169_private *tp) if (!tp->supports_gmii) phy_set_max_speed(phydev, SPEED_100); - /* Ensure to advertise everything, incl. pause */ - linkmode_copy(phydev->advertising, phydev->supported); + phy_support_asym_pause(phydev); phy_attached_info(phydev); @@ -7123,13 +6930,13 @@ static void rtl_read_mac_address(struct rtl8169_private *tp, switch (tp->mac_version) { case RTL_GIGA_MAC_VER_35 ... RTL_GIGA_MAC_VER_38: case RTL_GIGA_MAC_VER_40 ... RTL_GIGA_MAC_VER_51: - value = rtl_eri_read(tp, 0xe0, ERIAR_EXGMAC); + value = rtl_eri_read(tp, 0xe0); mac_addr[0] = (value >> 0) & 0xff; mac_addr[1] = (value >> 8) & 0xff; mac_addr[2] = (value >> 16) & 0xff; mac_addr[3] = (value >> 24) & 0xff; - value = rtl_eri_read(tp, 0xe4, ERIAR_EXGMAC); + value = rtl_eri_read(tp, 0xe4); mac_addr[4] = (value >> 0) & 0xff; mac_addr[5] = (value >> 8) & 0xff; break; diff --git a/drivers/net/ethernet/renesas/ravb_main.c b/drivers/net/ethernet/renesas/ravb_main.c index 316b47741d3f7a56976d46ff6d8f095502eb2b19..ef8f08931fe8b9d3cb75daee8a09c18aaa858f5f 100644 --- a/drivers/net/ethernet/renesas/ravb_main.c +++ b/drivers/net/ethernet/renesas/ravb_main.c @@ -111,7 +111,7 @@ static void ravb_set_buffer_align(struct sk_buff *skb) */ static void ravb_read_mac_address(struct net_device *ndev, const u8 *mac) { - if (mac) { + if (!IS_ERR(mac)) { ether_addr_copy(ndev->dev_addr, mac); } else { u32 mahr = ravb_read(ndev, MAHR); @@ -1607,8 +1607,7 @@ drop: } static u16 ravb_select_queue(struct net_device *ndev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { /* If skb needs TX timestamp, it is handled in network control queue */ return (skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP) ? RAVB_NC : @@ -1961,6 +1960,13 @@ static void ravb_set_config_mode(struct net_device *ndev) } } +static const struct soc_device_attribute ravb_delay_mode_quirk_match[] = { + { .soc_id = "r8a774c0" }, + { .soc_id = "r8a77990" }, + { .soc_id = "r8a77995" }, + { /* sentinel */ } +}; + /* Set tx and rx clock internal delay modes */ static void ravb_set_delay_mode(struct net_device *ndev) { @@ -1972,8 +1978,12 @@ static void ravb_set_delay_mode(struct net_device *ndev) set |= APSR_DM_RDM; if (priv->phy_interface == PHY_INTERFACE_MODE_RGMII_ID || - priv->phy_interface == PHY_INTERFACE_MODE_RGMII_TXID) - set |= APSR_DM_TDM; + priv->phy_interface == PHY_INTERFACE_MODE_RGMII_TXID) { + if (!WARN(soc_device_match(ravb_delay_mode_quirk_match), + "phy-mode %s requires TX clock internal delay mode which is not supported by this hardware revision. Please update device tree", + phy_modes(priv->phy_interface))) + set |= APSR_DM_TDM; + } ravb_modify(ndev, APSR, APSR_DM, set); } diff --git a/drivers/net/ethernet/renesas/sh_eth.c b/drivers/net/ethernet/renesas/sh_eth.c index ed30aebdb941a02e2136b89997ea5727613fd4de..7c4e282242d56cbd9f92c6b0b3289b1cffc8fae2 100644 --- a/drivers/net/ethernet/renesas/sh_eth.c +++ b/drivers/net/ethernet/renesas/sh_eth.c @@ -3192,7 +3192,7 @@ static struct sh_eth_plat_data *sh_eth_parse_dt(struct device *dev) pdata->phy_interface = ret; mac_addr = of_get_mac_address(np); - if (mac_addr) + if (!IS_ERR(mac_addr)) memcpy(pdata->mac_addr, mac_addr, ETH_ALEN); pdata->no_ether_link = diff --git a/drivers/net/ethernet/rocker/rocker_main.c b/drivers/net/ethernet/rocker/rocker_main.c index a71c900ca04f50396928d7a441cb57f429cf6737..7ae6c124bfe92ae360d754cf679ef4a166aa374d 100644 --- a/drivers/net/ethernet/rocker/rocker_main.c +++ b/drivers/net/ethernet/rocker/rocker_main.c @@ -2207,6 +2207,15 @@ static int rocker_router_fib_event(struct notifier_block *nb, switch (event) { case FIB_EVENT_ENTRY_ADD: /* fall through */ case FIB_EVENT_ENTRY_DEL: + if (info->family == AF_INET) { + struct fib_entry_notifier_info *fen_info = ptr; + + if (fen_info->fi->fib_nh_is_v6) { + NL_SET_ERR_MSG_MOD(info->extack, "IPv6 gateway with IPv4 route is not supported"); + return notifier_from_errno(-EINVAL); + } + } + memcpy(&fib_work->fen_info, ptr, sizeof(fib_work->fen_info)); /* Take referece on fib_info to prevent it from being * freed while work is queued. Release it afterwards. diff --git a/drivers/net/ethernet/rocker/rocker_ofdpa.c b/drivers/net/ethernet/rocker/rocker_ofdpa.c index fa296a7c255dfe581d59cc83a3f7db001a433c90..30a49802fb51e82ea20daeaca70feb47a27e53f3 100644 --- a/drivers/net/ethernet/rocker/rocker_ofdpa.c +++ b/drivers/net/ethernet/rocker/rocker_ofdpa.c @@ -2288,11 +2288,11 @@ static int ofdpa_port_fib_ipv4(struct ofdpa_port *ofdpa_port, __be32 dst, nh = fi->fib_nh; nh_on_port = (fi->fib_dev == ofdpa_port->dev); - has_gw = !!nh->nh_gw; + has_gw = !!nh->fib_nh_gw4; if (has_gw && nh_on_port) { err = ofdpa_port_ipv4_nh(ofdpa_port, flags, - nh->nh_gw, &index); + nh->fib_nh_gw4, &index); if (err) return err; @@ -2749,7 +2749,7 @@ static int ofdpa_fib4_add(struct rocker *rocker, fen_info->tb_id, 0); if (err) return err; - fen_info->fi->fib_nh->nh_flags |= RTNH_F_OFFLOAD; + fen_info->fi->fib_nh->fib_nh_flags |= RTNH_F_OFFLOAD; return 0; } @@ -2764,7 +2764,7 @@ static int ofdpa_fib4_del(struct rocker *rocker, ofdpa_port = ofdpa_port_dev_lower_find(fen_info->fi->fib_dev, rocker); if (!ofdpa_port) return 0; - fen_info->fi->fib_nh->nh_flags &= ~RTNH_F_OFFLOAD; + fen_info->fi->fib_nh->fib_nh_flags &= ~RTNH_F_OFFLOAD; return ofdpa_port_fib_ipv4(ofdpa_port, htonl(fen_info->dst), fen_info->dst_len, fen_info->fi, fen_info->tb_id, OFDPA_OP_FLAG_REMOVE); @@ -2791,7 +2791,7 @@ static void ofdpa_fib4_abort(struct rocker *rocker) rocker); if (!ofdpa_port) continue; - flow_entry->fi->fib_nh->nh_flags &= ~RTNH_F_OFFLOAD; + flow_entry->fi->fib_nh->fib_nh_flags &= ~RTNH_F_OFFLOAD; ofdpa_flow_tbl_del(ofdpa_port, OFDPA_OP_FLAG_REMOVE, flow_entry); } diff --git a/drivers/net/ethernet/samsung/sxgbe/sxgbe_platform.c b/drivers/net/ethernet/samsung/sxgbe/sxgbe_platform.c index fbd00cb0cb7d7204e97b9b11d8ae058c95148834..d2bc9412ba03b1d77c0320441bb0f7ac4440786a 100644 --- a/drivers/net/ethernet/samsung/sxgbe/sxgbe_platform.c +++ b/drivers/net/ethernet/samsung/sxgbe/sxgbe_platform.c @@ -124,7 +124,7 @@ static int sxgbe_platform_probe(struct platform_device *pdev) } /* Get MAC address if available (DT) */ - if (mac) + if (!IS_ERR_OR_NULL(mac)) ether_addr_copy(priv->dev->dev_addr, mac); /* Get the TX/RX IRQ numbers */ diff --git a/drivers/net/ethernet/sfc/falcon/tx.c b/drivers/net/ethernet/sfc/falcon/tx.c index 3409bbf5b19fffbc5ec3538e592173f854eb8791..c5059f456f374efe4baff7b64894de70df1bfdc1 100644 --- a/drivers/net/ethernet/sfc/falcon/tx.c +++ b/drivers/net/ethernet/sfc/falcon/tx.c @@ -321,7 +321,7 @@ netdev_tx_t ef4_enqueue_skb(struct ef4_tx_queue *tx_queue, struct sk_buff *skb) netdev_tx_sent_queue(tx_queue->core_txq, skb_len); /* Pass off to hardware */ - if (!skb->xmit_more || netif_xmit_stopped(tx_queue->core_txq)) { + if (!netdev_xmit_more() || netif_xmit_stopped(tx_queue->core_txq)) { struct ef4_tx_queue *txq2 = ef4_tx_queue_partner(tx_queue); /* There could be packets left on the partner queue if those @@ -333,7 +333,7 @@ netdev_tx_t ef4_enqueue_skb(struct ef4_tx_queue *tx_queue, struct sk_buff *skb) ef4_nic_push_buffers(tx_queue); } else { - tx_queue->xmit_more_available = skb->xmit_more; + tx_queue->xmit_more_available = netdev_xmit_more(); } tx_queue->tx_packets++; diff --git a/drivers/net/ethernet/sfc/mcdi_port.c b/drivers/net/ethernet/sfc/mcdi_port.c index 9382bb0b4d5ad67df6e5c79566e872604dd46996..a4bbfebe3d64d6bf991b55f578e324e8363aba92 100644 --- a/drivers/net/ethernet/sfc/mcdi_port.c +++ b/drivers/net/ethernet/sfc/mcdi_port.c @@ -342,6 +342,7 @@ static void efx_mcdi_phy_decode_link(struct efx_nic *efx, break; default: WARN_ON(1); + /* Fall through */ case MC_CMD_FCNTL_OFF: link_state->fc = 0; break; diff --git a/drivers/net/ethernet/sfc/tx.c b/drivers/net/ethernet/sfc/tx.c index 06c8f282263f7e23b63094dfb95870794013fa85..e182055ec2ebad88802b104540170f6e8936c4a6 100644 --- a/drivers/net/ethernet/sfc/tx.c +++ b/drivers/net/ethernet/sfc/tx.c @@ -478,8 +478,6 @@ static int efx_tx_tso_fallback(struct efx_tx_queue *tx_queue, next = skb->next; skb->next = NULL; - if (next) - skb->xmit_more = true; efx_enqueue_skb(tx_queue, skb); skb = next; } @@ -506,7 +504,7 @@ static int efx_tx_tso_fallback(struct efx_tx_queue *tx_queue, netdev_tx_t efx_enqueue_skb(struct efx_tx_queue *tx_queue, struct sk_buff *skb) { unsigned int old_insert_count = tx_queue->insert_count; - bool xmit_more = skb->xmit_more; + bool xmit_more = netdev_xmit_more(); bool data_mapped = false; unsigned int segments; unsigned int skb_len; @@ -533,7 +531,7 @@ netdev_tx_t efx_enqueue_skb(struct efx_tx_queue *tx_queue, struct sk_buff *skb) if (rc) goto err; #ifdef EFX_USE_PIO - } else if (skb_len <= efx_piobuf_size && !skb->xmit_more && + } else if (skb_len <= efx_piobuf_size && !xmit_more && efx_nic_may_tx_pio(tx_queue)) { /* Use PIO for short packets with an empty queue. */ if (efx_enqueue_skb_pio(tx_queue, skb)) @@ -559,8 +557,8 @@ netdev_tx_t efx_enqueue_skb(struct efx_tx_queue *tx_queue, struct sk_buff *skb) if (__netdev_tx_sent_queue(tx_queue->core_txq, skb_len, xmit_more)) { struct efx_tx_queue *txq2 = efx_tx_queue_partner(tx_queue); - /* There could be packets left on the partner queue if those - * SKBs had skb->xmit_more set. If we do not push those they + /* There could be packets left on the partner queue if + * xmit_more was set. If we do not push those they * could be left for a long time and cause a netdev watchdog. */ if (txq2->xmit_more_available) @@ -568,7 +566,7 @@ netdev_tx_t efx_enqueue_skb(struct efx_tx_queue *tx_queue, struct sk_buff *skb) efx_nic_push_buffers(tx_queue); } else { - tx_queue->xmit_more_available = skb->xmit_more; + tx_queue->xmit_more_available = xmit_more; } if (segments) { diff --git a/drivers/net/ethernet/socionext/sni_ave.c b/drivers/net/ethernet/socionext/sni_ave.c index bb6d5fb73035b340b989f368d90bad5c806a6d05..51a7b48db4bca4b16a517be4dbec68744939b502 100644 --- a/drivers/net/ethernet/socionext/sni_ave.c +++ b/drivers/net/ethernet/socionext/sni_ave.c @@ -1599,7 +1599,7 @@ static int ave_probe(struct platform_device *pdev) ndev->max_mtu = AVE_MAX_ETHFRAME - (ETH_HLEN + ETH_FCS_LEN); mac_addr = of_get_mac_address(np); - if (mac_addr) + if (!IS_ERR(mac_addr)) ether_addr_copy(ndev->dev_addr, mac_addr); /* if the mac address is invalid, use random mac address */ diff --git a/drivers/net/ethernet/stmicro/stmmac/dwxgmac2.h b/drivers/net/ethernet/stmicro/stmmac/dwxgmac2.h index 37d5e6fe74737f6c82acc223e2c93b043b7bb9f4..085b700a4994e599ec5ff2044c690a614f5ac4e6 100644 --- a/drivers/net/ethernet/stmicro/stmmac/dwxgmac2.h +++ b/drivers/net/ethernet/stmicro/stmmac/dwxgmac2.h @@ -143,6 +143,11 @@ #define XGMAC_RSF BIT(5) #define XGMAC_RTC GENMASK(1, 0) #define XGMAC_RTC_SHIFT 0 +#define XGMAC_MTL_RXQ_FLOW_CONTROL(x) (0x00001150 + (0x80 * (x))) +#define XGMAC_RFD GENMASK(31, 17) +#define XGMAC_RFD_SHIFT 17 +#define XGMAC_RFA GENMASK(15, 1) +#define XGMAC_RFA_SHIFT 1 #define XGMAC_MTL_QINTEN(x) (0x00001170 + (0x80 * (x))) #define XGMAC_RXOIE BIT(16) #define XGMAC_MTL_QINT_STATUS(x) (0x00001174 + (0x80 * (x))) diff --git a/drivers/net/ethernet/stmicro/stmmac/dwxgmac2_dma.c b/drivers/net/ethernet/stmicro/stmmac/dwxgmac2_dma.c index 2ba712b48a89ab3f71997666845b1804ef450e88..e79037f511e16c6d8bc7085c1612e3f5b0d81912 100644 --- a/drivers/net/ethernet/stmicro/stmmac/dwxgmac2_dma.c +++ b/drivers/net/ethernet/stmicro/stmmac/dwxgmac2_dma.c @@ -147,6 +147,52 @@ static void dwxgmac2_dma_rx_mode(void __iomem *ioaddr, int mode, value &= ~XGMAC_RQS; value |= (rqs << XGMAC_RQS_SHIFT) & XGMAC_RQS; + if ((fifosz >= 4096) && (qmode != MTL_QUEUE_AVB)) { + u32 flow = readl(ioaddr + XGMAC_MTL_RXQ_FLOW_CONTROL(channel)); + unsigned int rfd, rfa; + + value |= XGMAC_EHFC; + + /* Set Threshold for Activating Flow Control to min 2 frames, + * i.e. 1500 * 2 = 3000 bytes. + * + * Set Threshold for Deactivating Flow Control to min 1 frame, + * i.e. 1500 bytes. + */ + switch (fifosz) { + case 4096: + /* This violates the above formula because of FIFO size + * limit therefore overflow may occur in spite of this. + */ + rfd = 0x03; /* Full-2.5K */ + rfa = 0x01; /* Full-1.5K */ + break; + + case 8192: + rfd = 0x06; /* Full-4K */ + rfa = 0x0a; /* Full-6K */ + break; + + case 16384: + rfd = 0x06; /* Full-4K */ + rfa = 0x12; /* Full-10K */ + break; + + default: + rfd = 0x06; /* Full-4K */ + rfa = 0x1e; /* Full-16K */ + break; + } + + flow &= ~XGMAC_RFD; + flow |= rfd << XGMAC_RFD_SHIFT; + + flow &= ~XGMAC_RFA; + flow |= rfa << XGMAC_RFA_SHIFT; + + writel(flow, ioaddr + XGMAC_MTL_RXQ_FLOW_CONTROL(channel)); + } + writel(value, ioaddr + XGMAC_MTL_RXQ_OPMODE(channel)); /* Enable MTL RX overflow */ diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c index 48712437d0da8039e74c8cc34d8063ac666c03a8..5678b869cbffdfe5d10120d96f1e618d98f8d208 100644 --- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c +++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c @@ -74,7 +74,7 @@ MODULE_PARM_DESC(phyaddr, "Physical device address"); #define STMMAC_TX_THRESH (DMA_TX_SIZE / 4) #define STMMAC_RX_THRESH (DMA_RX_SIZE / 4) -static int flow_ctrl = FLOW_OFF; +static int flow_ctrl = FLOW_AUTO; module_param(flow_ctrl, int, 0644); MODULE_PARM_DESC(flow_ctrl, "Flow control ability [on/off]"); @@ -4262,7 +4262,7 @@ int stmmac_dvr_probe(struct device *device, priv->wol_irq = res->wol_irq; priv->lpi_irq = res->lpi_irq; - if (res->mac) + if (!IS_ERR_OR_NULL(res->mac)) memcpy(priv->dev->dev_addr, res->mac, ETH_ALEN); dev_set_drvdata(device, priv->dev); diff --git a/drivers/net/ethernet/sun/ldmvsw.c b/drivers/net/ethernet/sun/ldmvsw.c index 644e42c181ee6030dee0222479daf314a290a1f5..01ea0d6f88193821f0cfefbbba51709a4059a3c9 100644 --- a/drivers/net/ethernet/sun/ldmvsw.c +++ b/drivers/net/ethernet/sun/ldmvsw.c @@ -101,8 +101,7 @@ static struct vnet_port *vsw_tx_port_find(struct sk_buff *skb, } static u16 vsw_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { struct vnet_port *port = netdev_priv(dev); diff --git a/drivers/net/ethernet/sun/sunvnet.c b/drivers/net/ethernet/sun/sunvnet.c index 590172818b922f069a82765ff5042cd852093c27..96b883f965f638958be20b054029b579f6069cff 100644 --- a/drivers/net/ethernet/sun/sunvnet.c +++ b/drivers/net/ethernet/sun/sunvnet.c @@ -234,8 +234,7 @@ static struct vnet_port *vnet_tx_port_find(struct sk_buff *skb, } static u16 vnet_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { struct vnet *vp = netdev_priv(dev); struct vnet_port *port = __tx_port_find(vp, skb); diff --git a/drivers/net/ethernet/synopsys/dwc-xlgmac-hw.c b/drivers/net/ethernet/synopsys/dwc-xlgmac-hw.c index 99d86e39ff541c29a239b176ccfcb704c50e4ce8..bf6c1c6779ff37826fa27bf89be1a23bc6d33744 100644 --- a/drivers/net/ethernet/synopsys/dwc-xlgmac-hw.c +++ b/drivers/net/ethernet/synopsys/dwc-xlgmac-hw.c @@ -995,7 +995,7 @@ static void xlgmac_dev_xmit(struct xlgmac_channel *channel) smp_wmb(); ring->cur = cur_index + 1; - if (!pkt_info->skb->xmit_more || + if (!netdev_xmit_more() || netif_xmit_stopped(netdev_get_tx_queue(pdata->netdev, channel->queue_index))) xlgmac_tx_start_xmit(channel, ring); diff --git a/drivers/net/ethernet/ti/Kconfig b/drivers/net/ethernet/ti/Kconfig index 8b21b40a9fe5193298c3086d3867609a3eb58f12..afbdc9744230ac0fea85ea87c73075cda3c8fad8 100644 --- a/drivers/net/ethernet/ti/Kconfig +++ b/drivers/net/ethernet/ti/Kconfig @@ -20,7 +20,6 @@ config TI_DAVINCI_EMAC tristate "TI DaVinci EMAC Support" depends on ARM && ( ARCH_DAVINCI || ARCH_OMAP3 ) || COMPILE_TEST select TI_DAVINCI_MDIO - select TI_DAVINCI_CPDMA select PHYLIB ---help--- This driver supports TI's DaVinci Ethernet . @@ -38,16 +37,6 @@ config TI_DAVINCI_MDIO To compile this driver as a module, choose M here: the module will be called davinci_mdio. This is recommended. -config TI_DAVINCI_CPDMA - tristate "TI DaVinci CPDMA Support" - depends on ARCH_DAVINCI || ARCH_OMAP2PLUS || COMPILE_TEST - select GENERIC_ALLOCATOR - ---help--- - This driver supports TI's DaVinci CPDMA dma engine. - - To compile this driver as a module, choose M here: the module - will be called davinci_cpdma. This is recommended. - config TI_CPSW_PHY_SEL bool "TI CPSW Phy mode Selection (DEPRECATED)" default n @@ -55,17 +44,10 @@ config TI_CPSW_PHY_SEL This driver supports configuring of the phy mode connected to the CPSW. DEPRECATED: use PHY_TI_GMII_SEL. -config TI_CPSW_ALE - tristate "TI CPSW ALE Support" - ---help--- - This driver supports TI's CPSW ALE module. - config TI_CPSW tristate "TI CPSW Switch Support" depends on ARCH_DAVINCI || ARCH_OMAP2PLUS || COMPILE_TEST - select TI_DAVINCI_CPDMA select TI_DAVINCI_MDIO - select TI_CPSW_ALE select MFD_SYSCON select REGMAP ---help--- @@ -94,7 +76,6 @@ config TI_CPTS_MOD config TI_KEYSTONE_NETCP tristate "TI Keystone NETCP Core Support" - select TI_CPSW_ALE select TI_DAVINCI_MDIO depends on OF depends on KEYSTONE_NAVIGATOR_DMA && KEYSTONE_NAVIGATOR_QMSS diff --git a/drivers/net/ethernet/ti/Makefile b/drivers/net/ethernet/ti/Makefile index 0be551de821c884f54c4bdb254d75456664b0a4f..c3f53a40b48f20168fd9e98935ca10ab504ab8b7 100644 --- a/drivers/net/ethernet/ti/Makefile +++ b/drivers/net/ethernet/ti/Makefile @@ -8,16 +8,15 @@ obj-$(CONFIG_TI_DAVINCI_EMAC) += cpsw-common.o obj-$(CONFIG_TLAN) += tlan.o obj-$(CONFIG_CPMAC) += cpmac.o -obj-$(CONFIG_TI_DAVINCI_EMAC) += davinci_emac.o +obj-$(CONFIG_TI_DAVINCI_EMAC) += ti_davinci_emac.o +ti_davinci_emac-y := davinci_emac.o davinci_cpdma.o obj-$(CONFIG_TI_DAVINCI_MDIO) += davinci_mdio.o -obj-$(CONFIG_TI_DAVINCI_CPDMA) += davinci_cpdma.o obj-$(CONFIG_TI_CPSW_PHY_SEL) += cpsw-phy-sel.o -obj-$(CONFIG_TI_CPSW_ALE) += cpsw_ale.o obj-$(CONFIG_TI_CPTS_MOD) += cpts.o obj-$(CONFIG_TI_CPSW) += ti_cpsw.o -ti_cpsw-y := cpsw.o +ti_cpsw-y := cpsw.o davinci_cpdma.o cpsw_ale.o cpsw_priv.o cpsw_sl.o cpsw_ethtool.o obj-$(CONFIG_TI_KEYSTONE_NETCP) += keystone_netcp.o -keystone_netcp-y := netcp_core.o +keystone_netcp-y := netcp_core.o cpsw_ale.o obj-$(CONFIG_TI_KEYSTONE_NETCP_ETHSS) += keystone_netcp_ethss.o keystone_netcp_ethss-y := netcp_ethss.o netcp_sgmii.o netcp_xgbepcsr.o diff --git a/drivers/net/ethernet/ti/cpmac.c b/drivers/net/ethernet/ti/cpmac.c index e2d47b24a86922284ab508af8eb4bebfde2893e4..3a655a4dc10eed49b6dba2ecac9ff86381741e08 100644 --- a/drivers/net/ethernet/ti/cpmac.c +++ b/drivers/net/ethernet/ti/cpmac.c @@ -1,19 +1,7 @@ +// SPDX-License-Identifier: GPL-2.0+ /* * Copyright (C) 2006, 2007 Eugene Konev * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, write to the Free Software - * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ #include diff --git a/drivers/net/ethernet/ti/cpsw-common.c b/drivers/net/ethernet/ti/cpsw-common.c index 38d1cc557c116b1047e27daa7923a654a438d52e..bfa81bbfce3f23d930b09afe1997b3c435cc9071 100644 --- a/drivers/net/ethernet/ti/cpsw-common.c +++ b/drivers/net/ethernet/ti/cpsw-common.c @@ -1,14 +1,4 @@ -/* - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - */ +// SPDX-License-Identifier: GPL-2.0+ #include #include diff --git a/drivers/net/ethernet/ti/cpsw-phy-sel.c b/drivers/net/ethernet/ti/cpsw-phy-sel.c index fec275e2208d0daeb9bdae6d12ae9c046faeb92f..48e0924259f541425d190dabb4841623987f68e0 100644 --- a/drivers/net/ethernet/ti/cpsw-phy-sel.c +++ b/drivers/net/ethernet/ti/cpsw-phy-sel.c @@ -1,17 +1,10 @@ +// SPDX-License-Identifier: GPL-2.0 /* Texas Instruments Ethernet Switch Driver * * Copyright (C) 2013 Texas Instruments * * Module Author: Mugunthan V N * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * version 2 as published by the Free Software Foundation. - * - * This program is distributed "as is" WITHOUT ANY WARRANTY of any - * kind, whether express or implied; without even the implied warranty - * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include diff --git a/drivers/net/ethernet/ti/cpsw.c b/drivers/net/ethernet/ti/cpsw.c index a591583d120e1f598ab66f8d5e07daf15dc46321..b18eeb05b9930feb23d563767b2ef272891406ea 100644 --- a/drivers/net/ethernet/ti/cpsw.c +++ b/drivers/net/ethernet/ti/cpsw.c @@ -1,16 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Texas Instruments Ethernet Switch Driver * * Copyright (C) 2012 Texas Instruments * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License as - * published by the Free Software Foundation version 2. - * - * This program is distributed "as is" WITHOUT ANY WARRANTY of any - * kind, whether express or implied; without even the implied warranty - * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include @@ -44,138 +37,13 @@ #include "cpsw.h" #include "cpsw_ale.h" +#include "cpsw_priv.h" +#include "cpsw_sl.h" #include "cpts.h" #include "davinci_cpdma.h" #include -#define CPSW_DEBUG (NETIF_MSG_HW | NETIF_MSG_WOL | \ - NETIF_MSG_DRV | NETIF_MSG_LINK | \ - NETIF_MSG_IFUP | NETIF_MSG_INTR | \ - NETIF_MSG_PROBE | NETIF_MSG_TIMER | \ - NETIF_MSG_IFDOWN | NETIF_MSG_RX_ERR | \ - NETIF_MSG_TX_ERR | NETIF_MSG_TX_DONE | \ - NETIF_MSG_PKTDATA | NETIF_MSG_TX_QUEUED | \ - NETIF_MSG_RX_STATUS) - -#define cpsw_info(priv, type, format, ...) \ -do { \ - if (netif_msg_##type(priv) && net_ratelimit()) \ - dev_info(priv->dev, format, ## __VA_ARGS__); \ -} while (0) - -#define cpsw_err(priv, type, format, ...) \ -do { \ - if (netif_msg_##type(priv) && net_ratelimit()) \ - dev_err(priv->dev, format, ## __VA_ARGS__); \ -} while (0) - -#define cpsw_dbg(priv, type, format, ...) \ -do { \ - if (netif_msg_##type(priv) && net_ratelimit()) \ - dev_dbg(priv->dev, format, ## __VA_ARGS__); \ -} while (0) - -#define cpsw_notice(priv, type, format, ...) \ -do { \ - if (netif_msg_##type(priv) && net_ratelimit()) \ - dev_notice(priv->dev, format, ## __VA_ARGS__); \ -} while (0) - -#define ALE_ALL_PORTS 0x7 - -#define CPSW_MAJOR_VERSION(reg) (reg >> 8 & 0x7) -#define CPSW_MINOR_VERSION(reg) (reg & 0xff) -#define CPSW_RTL_VERSION(reg) ((reg >> 11) & 0x1f) - -#define CPSW_VERSION_1 0x19010a -#define CPSW_VERSION_2 0x19010c -#define CPSW_VERSION_3 0x19010f -#define CPSW_VERSION_4 0x190112 - -#define HOST_PORT_NUM 0 -#define CPSW_ALE_PORTS_NUM 3 -#define SLIVER_SIZE 0x40 - -#define CPSW1_HOST_PORT_OFFSET 0x028 -#define CPSW1_SLAVE_OFFSET 0x050 -#define CPSW1_SLAVE_SIZE 0x040 -#define CPSW1_CPDMA_OFFSET 0x100 -#define CPSW1_STATERAM_OFFSET 0x200 -#define CPSW1_HW_STATS 0x400 -#define CPSW1_CPTS_OFFSET 0x500 -#define CPSW1_ALE_OFFSET 0x600 -#define CPSW1_SLIVER_OFFSET 0x700 - -#define CPSW2_HOST_PORT_OFFSET 0x108 -#define CPSW2_SLAVE_OFFSET 0x200 -#define CPSW2_SLAVE_SIZE 0x100 -#define CPSW2_CPDMA_OFFSET 0x800 -#define CPSW2_HW_STATS 0x900 -#define CPSW2_STATERAM_OFFSET 0xa00 -#define CPSW2_CPTS_OFFSET 0xc00 -#define CPSW2_ALE_OFFSET 0xd00 -#define CPSW2_SLIVER_OFFSET 0xd80 -#define CPSW2_BD_OFFSET 0x2000 - -#define CPDMA_RXTHRESH 0x0c0 -#define CPDMA_RXFREE 0x0e0 -#define CPDMA_TXHDP 0x00 -#define CPDMA_RXHDP 0x20 -#define CPDMA_TXCP 0x40 -#define CPDMA_RXCP 0x60 - -#define CPSW_POLL_WEIGHT 64 -#define CPSW_RX_VLAN_ENCAP_HDR_SIZE 4 -#define CPSW_MIN_PACKET_SIZE (VLAN_ETH_ZLEN) -#define CPSW_MAX_PACKET_SIZE (VLAN_ETH_FRAME_LEN +\ - ETH_FCS_LEN +\ - CPSW_RX_VLAN_ENCAP_HDR_SIZE) - -#define RX_PRIORITY_MAPPING 0x76543210 -#define TX_PRIORITY_MAPPING 0x33221100 -#define CPDMA_TX_PRIORITY_MAP 0x76543210 - -#define CPSW_VLAN_AWARE BIT(1) -#define CPSW_RX_VLAN_ENCAP BIT(2) -#define CPSW_ALE_VLAN_AWARE 1 - -#define CPSW_FIFO_NORMAL_MODE (0 << 16) -#define CPSW_FIFO_DUAL_MAC_MODE (1 << 16) -#define CPSW_FIFO_RATE_LIMIT_MODE (2 << 16) - -#define CPSW_INTPACEEN (0x3f << 16) -#define CPSW_INTPRESCALE_MASK (0x7FF << 0) -#define CPSW_CMINTMAX_CNT 63 -#define CPSW_CMINTMIN_CNT 2 -#define CPSW_CMINTMAX_INTVL (1000 / CPSW_CMINTMIN_CNT) -#define CPSW_CMINTMIN_INTVL ((1000 / CPSW_CMINTMAX_CNT) + 1) - -#define cpsw_slave_index(cpsw, priv) \ - ((cpsw->data.dual_emac) ? priv->emac_port : \ - cpsw->data.active_slave) -#define IRQ_NUM 2 -#define CPSW_MAX_QUEUES 8 -#define CPSW_CPDMA_DESCS_POOL_SIZE_DEFAULT 256 -#define CPSW_FIFO_QUEUE_TYPE_SHIFT 16 -#define CPSW_FIFO_SHAPE_EN_SHIFT 16 -#define CPSW_FIFO_RATE_EN_SHIFT 20 -#define CPSW_TC_NUM 4 -#define CPSW_FIFO_SHAPERS_NUM (CPSW_TC_NUM - 1) -#define CPSW_PCT_MASK 0x7f - -#define CPSW_RX_VLAN_ENCAP_HDR_PRIO_SHIFT 29 -#define CPSW_RX_VLAN_ENCAP_HDR_PRIO_MSK GENMASK(2, 0) -#define CPSW_RX_VLAN_ENCAP_HDR_VID_SHIFT 16 -#define CPSW_RX_VLAN_ENCAP_HDR_PKT_TYPE_SHIFT 8 -#define CPSW_RX_VLAN_ENCAP_HDR_PKT_TYPE_MSK GENMASK(1, 0) -enum { - CPSW_RX_VLAN_ENCAP_HDR_PKT_VLAN_TAG = 0, - CPSW_RX_VLAN_ENCAP_HDR_PKT_RESERV, - CPSW_RX_VLAN_ENCAP_HDR_PKT_PRIO_TAG, - CPSW_RX_VLAN_ENCAP_HDR_PKT_UNTAG, -}; - static int debug_level; module_param(debug_level, int, 0); MODULE_PARM_DESC(debug_level, "cpsw debug level (NETIF_MSG bits)"); @@ -192,369 +60,6 @@ static int descs_pool_size = CPSW_CPDMA_DESCS_POOL_SIZE_DEFAULT; module_param(descs_pool_size, int, 0444); MODULE_PARM_DESC(descs_pool_size, "Number of CPDMA CPPI descriptors in pool"); -struct cpsw_wr_regs { - u32 id_ver; - u32 soft_reset; - u32 control; - u32 int_control; - u32 rx_thresh_en; - u32 rx_en; - u32 tx_en; - u32 misc_en; - u32 mem_allign1[8]; - u32 rx_thresh_stat; - u32 rx_stat; - u32 tx_stat; - u32 misc_stat; - u32 mem_allign2[8]; - u32 rx_imax; - u32 tx_imax; - -}; - -struct cpsw_ss_regs { - u32 id_ver; - u32 control; - u32 soft_reset; - u32 stat_port_en; - u32 ptype; - u32 soft_idle; - u32 thru_rate; - u32 gap_thresh; - u32 tx_start_wds; - u32 flow_control; - u32 vlan_ltype; - u32 ts_ltype; - u32 dlr_ltype; -}; - -/* CPSW_PORT_V1 */ -#define CPSW1_MAX_BLKS 0x00 /* Maximum FIFO Blocks */ -#define CPSW1_BLK_CNT 0x04 /* FIFO Block Usage Count (Read Only) */ -#define CPSW1_TX_IN_CTL 0x08 /* Transmit FIFO Control */ -#define CPSW1_PORT_VLAN 0x0c /* VLAN Register */ -#define CPSW1_TX_PRI_MAP 0x10 /* Tx Header Priority to Switch Pri Mapping */ -#define CPSW1_TS_CTL 0x14 /* Time Sync Control */ -#define CPSW1_TS_SEQ_LTYPE 0x18 /* Time Sync Sequence ID Offset and Msg Type */ -#define CPSW1_TS_VLAN 0x1c /* Time Sync VLAN1 and VLAN2 */ - -/* CPSW_PORT_V2 */ -#define CPSW2_CONTROL 0x00 /* Control Register */ -#define CPSW2_MAX_BLKS 0x08 /* Maximum FIFO Blocks */ -#define CPSW2_BLK_CNT 0x0c /* FIFO Block Usage Count (Read Only) */ -#define CPSW2_TX_IN_CTL 0x10 /* Transmit FIFO Control */ -#define CPSW2_PORT_VLAN 0x14 /* VLAN Register */ -#define CPSW2_TX_PRI_MAP 0x18 /* Tx Header Priority to Switch Pri Mapping */ -#define CPSW2_TS_SEQ_MTYPE 0x1c /* Time Sync Sequence ID Offset and Msg Type */ - -/* CPSW_PORT_V1 and V2 */ -#define SA_LO 0x20 /* CPGMAC_SL Source Address Low */ -#define SA_HI 0x24 /* CPGMAC_SL Source Address High */ -#define SEND_PERCENT 0x28 /* Transmit Queue Send Percentages */ - -/* CPSW_PORT_V2 only */ -#define RX_DSCP_PRI_MAP0 0x30 /* Rx DSCP Priority to Rx Packet Mapping */ -#define RX_DSCP_PRI_MAP1 0x34 /* Rx DSCP Priority to Rx Packet Mapping */ -#define RX_DSCP_PRI_MAP2 0x38 /* Rx DSCP Priority to Rx Packet Mapping */ -#define RX_DSCP_PRI_MAP3 0x3c /* Rx DSCP Priority to Rx Packet Mapping */ -#define RX_DSCP_PRI_MAP4 0x40 /* Rx DSCP Priority to Rx Packet Mapping */ -#define RX_DSCP_PRI_MAP5 0x44 /* Rx DSCP Priority to Rx Packet Mapping */ -#define RX_DSCP_PRI_MAP6 0x48 /* Rx DSCP Priority to Rx Packet Mapping */ -#define RX_DSCP_PRI_MAP7 0x4c /* Rx DSCP Priority to Rx Packet Mapping */ - -/* Bit definitions for the CPSW2_CONTROL register */ -#define PASS_PRI_TAGGED BIT(24) /* Pass Priority Tagged */ -#define VLAN_LTYPE2_EN BIT(21) /* VLAN LTYPE 2 enable */ -#define VLAN_LTYPE1_EN BIT(20) /* VLAN LTYPE 1 enable */ -#define DSCP_PRI_EN BIT(16) /* DSCP Priority Enable */ -#define TS_107 BIT(15) /* Tyme Sync Dest IP Address 107 */ -#define TS_320 BIT(14) /* Time Sync Dest Port 320 enable */ -#define TS_319 BIT(13) /* Time Sync Dest Port 319 enable */ -#define TS_132 BIT(12) /* Time Sync Dest IP Addr 132 enable */ -#define TS_131 BIT(11) /* Time Sync Dest IP Addr 131 enable */ -#define TS_130 BIT(10) /* Time Sync Dest IP Addr 130 enable */ -#define TS_129 BIT(9) /* Time Sync Dest IP Addr 129 enable */ -#define TS_TTL_NONZERO BIT(8) /* Time Sync Time To Live Non-zero enable */ -#define TS_ANNEX_F_EN BIT(6) /* Time Sync Annex F enable */ -#define TS_ANNEX_D_EN BIT(4) /* Time Sync Annex D enable */ -#define TS_LTYPE2_EN BIT(3) /* Time Sync LTYPE 2 enable */ -#define TS_LTYPE1_EN BIT(2) /* Time Sync LTYPE 1 enable */ -#define TS_TX_EN BIT(1) /* Time Sync Transmit Enable */ -#define TS_RX_EN BIT(0) /* Time Sync Receive Enable */ - -#define CTRL_V2_TS_BITS \ - (TS_320 | TS_319 | TS_132 | TS_131 | TS_130 | TS_129 |\ - TS_TTL_NONZERO | TS_ANNEX_D_EN | TS_LTYPE1_EN | VLAN_LTYPE1_EN) - -#define CTRL_V2_ALL_TS_MASK (CTRL_V2_TS_BITS | TS_TX_EN | TS_RX_EN) -#define CTRL_V2_TX_TS_BITS (CTRL_V2_TS_BITS | TS_TX_EN) -#define CTRL_V2_RX_TS_BITS (CTRL_V2_TS_BITS | TS_RX_EN) - - -#define CTRL_V3_TS_BITS \ - (TS_107 | TS_320 | TS_319 | TS_132 | TS_131 | TS_130 | TS_129 |\ - TS_TTL_NONZERO | TS_ANNEX_F_EN | TS_ANNEX_D_EN |\ - TS_LTYPE1_EN | VLAN_LTYPE1_EN) - -#define CTRL_V3_ALL_TS_MASK (CTRL_V3_TS_BITS | TS_TX_EN | TS_RX_EN) -#define CTRL_V3_TX_TS_BITS (CTRL_V3_TS_BITS | TS_TX_EN) -#define CTRL_V3_RX_TS_BITS (CTRL_V3_TS_BITS | TS_RX_EN) - -/* Bit definitions for the CPSW2_TS_SEQ_MTYPE register */ -#define TS_SEQ_ID_OFFSET_SHIFT (16) /* Time Sync Sequence ID Offset */ -#define TS_SEQ_ID_OFFSET_MASK (0x3f) -#define TS_MSG_TYPE_EN_SHIFT (0) /* Time Sync Message Type Enable */ -#define TS_MSG_TYPE_EN_MASK (0xffff) - -/* The PTP event messages - Sync, Delay_Req, Pdelay_Req, and Pdelay_Resp. */ -#define EVENT_MSG_BITS ((1<<0) | (1<<1) | (1<<2) | (1<<3)) - -/* Bit definitions for the CPSW1_TS_CTL register */ -#define CPSW_V1_TS_RX_EN BIT(0) -#define CPSW_V1_TS_TX_EN BIT(4) -#define CPSW_V1_MSG_TYPE_OFS 16 - -/* Bit definitions for the CPSW1_TS_SEQ_LTYPE register */ -#define CPSW_V1_SEQ_ID_OFS_SHIFT 16 - -#define CPSW_MAX_BLKS_TX 15 -#define CPSW_MAX_BLKS_TX_SHIFT 4 -#define CPSW_MAX_BLKS_RX 5 - -struct cpsw_host_regs { - u32 max_blks; - u32 blk_cnt; - u32 tx_in_ctl; - u32 port_vlan; - u32 tx_pri_map; - u32 cpdma_tx_pri_map; - u32 cpdma_rx_chan_map; -}; - -struct cpsw_sliver_regs { - u32 id_ver; - u32 mac_control; - u32 mac_status; - u32 soft_reset; - u32 rx_maxlen; - u32 __reserved_0; - u32 rx_pause; - u32 tx_pause; - u32 __reserved_1; - u32 rx_pri_map; -}; - -struct cpsw_hw_stats { - u32 rxgoodframes; - u32 rxbroadcastframes; - u32 rxmulticastframes; - u32 rxpauseframes; - u32 rxcrcerrors; - u32 rxaligncodeerrors; - u32 rxoversizedframes; - u32 rxjabberframes; - u32 rxundersizedframes; - u32 rxfragments; - u32 __pad_0[2]; - u32 rxoctets; - u32 txgoodframes; - u32 txbroadcastframes; - u32 txmulticastframes; - u32 txpauseframes; - u32 txdeferredframes; - u32 txcollisionframes; - u32 txsinglecollframes; - u32 txmultcollframes; - u32 txexcessivecollisions; - u32 txlatecollisions; - u32 txunderrun; - u32 txcarriersenseerrors; - u32 txoctets; - u32 octetframes64; - u32 octetframes65t127; - u32 octetframes128t255; - u32 octetframes256t511; - u32 octetframes512t1023; - u32 octetframes1024tup; - u32 netoctets; - u32 rxsofoverruns; - u32 rxmofoverruns; - u32 rxdmaoverruns; -}; - -struct cpsw_slave_data { - struct device_node *phy_node; - char phy_id[MII_BUS_ID_SIZE]; - int phy_if; - u8 mac_addr[ETH_ALEN]; - u16 dual_emac_res_vlan; /* Reserved VLAN for DualEMAC */ - struct phy *ifphy; -}; - -struct cpsw_platform_data { - struct cpsw_slave_data *slave_data; - u32 ss_reg_ofs; /* Subsystem control register offset */ - u32 channels; /* number of cpdma channels (symmetric) */ - u32 slaves; /* number of slave cpgmac ports */ - u32 active_slave; /* time stamping, ethtool and SIOCGMIIPHY slave */ - u32 ale_entries; /* ale table size */ - u32 bd_ram_size; /*buffer descriptor ram size */ - u32 mac_control; /* Mac control register */ - u16 default_vlan; /* Def VLAN for ALE lookup in VLAN aware mode*/ - bool dual_emac; /* Enable Dual EMAC mode */ -}; - -struct cpsw_slave { - void __iomem *regs; - struct cpsw_sliver_regs __iomem *sliver; - int slave_num; - u32 mac_control; - struct cpsw_slave_data *data; - struct phy_device *phy; - struct net_device *ndev; - u32 port_vlan; -}; - -static inline u32 slave_read(struct cpsw_slave *slave, u32 offset) -{ - return readl_relaxed(slave->regs + offset); -} - -static inline void slave_write(struct cpsw_slave *slave, u32 val, u32 offset) -{ - writel_relaxed(val, slave->regs + offset); -} - -struct cpsw_vector { - struct cpdma_chan *ch; - int budget; -}; - -struct cpsw_common { - struct device *dev; - struct cpsw_platform_data data; - struct napi_struct napi_rx; - struct napi_struct napi_tx; - struct cpsw_ss_regs __iomem *regs; - struct cpsw_wr_regs __iomem *wr_regs; - u8 __iomem *hw_stats; - struct cpsw_host_regs __iomem *host_port_regs; - u32 version; - u32 coal_intvl; - u32 bus_freq_mhz; - int rx_packet_max; - struct cpsw_slave *slaves; - struct cpdma_ctlr *dma; - struct cpsw_vector txv[CPSW_MAX_QUEUES]; - struct cpsw_vector rxv[CPSW_MAX_QUEUES]; - struct cpsw_ale *ale; - bool quirk_irq; - bool rx_irq_disabled; - bool tx_irq_disabled; - u32 irqs_table[IRQ_NUM]; - struct cpts *cpts; - int rx_ch_num, tx_ch_num; - int speed; - int usage_count; -}; - -struct cpsw_priv { - struct net_device *ndev; - struct device *dev; - u32 msg_enable; - u8 mac_addr[ETH_ALEN]; - bool rx_pause; - bool tx_pause; - bool mqprio_hw; - int fifo_bw[CPSW_TC_NUM]; - int shp_cfg_speed; - int tx_ts_enabled; - int rx_ts_enabled; - u32 emac_port; - struct cpsw_common *cpsw; -}; - -struct cpsw_stats { - char stat_string[ETH_GSTRING_LEN]; - int type; - int sizeof_stat; - int stat_offset; -}; - -enum { - CPSW_STATS, - CPDMA_RX_STATS, - CPDMA_TX_STATS, -}; - -#define CPSW_STAT(m) CPSW_STATS, \ - FIELD_SIZEOF(struct cpsw_hw_stats, m), \ - offsetof(struct cpsw_hw_stats, m) -#define CPDMA_RX_STAT(m) CPDMA_RX_STATS, \ - FIELD_SIZEOF(struct cpdma_chan_stats, m), \ - offsetof(struct cpdma_chan_stats, m) -#define CPDMA_TX_STAT(m) CPDMA_TX_STATS, \ - FIELD_SIZEOF(struct cpdma_chan_stats, m), \ - offsetof(struct cpdma_chan_stats, m) - -static const struct cpsw_stats cpsw_gstrings_stats[] = { - { "Good Rx Frames", CPSW_STAT(rxgoodframes) }, - { "Broadcast Rx Frames", CPSW_STAT(rxbroadcastframes) }, - { "Multicast Rx Frames", CPSW_STAT(rxmulticastframes) }, - { "Pause Rx Frames", CPSW_STAT(rxpauseframes) }, - { "Rx CRC Errors", CPSW_STAT(rxcrcerrors) }, - { "Rx Align/Code Errors", CPSW_STAT(rxaligncodeerrors) }, - { "Oversize Rx Frames", CPSW_STAT(rxoversizedframes) }, - { "Rx Jabbers", CPSW_STAT(rxjabberframes) }, - { "Undersize (Short) Rx Frames", CPSW_STAT(rxundersizedframes) }, - { "Rx Fragments", CPSW_STAT(rxfragments) }, - { "Rx Octets", CPSW_STAT(rxoctets) }, - { "Good Tx Frames", CPSW_STAT(txgoodframes) }, - { "Broadcast Tx Frames", CPSW_STAT(txbroadcastframes) }, - { "Multicast Tx Frames", CPSW_STAT(txmulticastframes) }, - { "Pause Tx Frames", CPSW_STAT(txpauseframes) }, - { "Deferred Tx Frames", CPSW_STAT(txdeferredframes) }, - { "Collisions", CPSW_STAT(txcollisionframes) }, - { "Single Collision Tx Frames", CPSW_STAT(txsinglecollframes) }, - { "Multiple Collision Tx Frames", CPSW_STAT(txmultcollframes) }, - { "Excessive Collisions", CPSW_STAT(txexcessivecollisions) }, - { "Late Collisions", CPSW_STAT(txlatecollisions) }, - { "Tx Underrun", CPSW_STAT(txunderrun) }, - { "Carrier Sense Errors", CPSW_STAT(txcarriersenseerrors) }, - { "Tx Octets", CPSW_STAT(txoctets) }, - { "Rx + Tx 64 Octet Frames", CPSW_STAT(octetframes64) }, - { "Rx + Tx 65-127 Octet Frames", CPSW_STAT(octetframes65t127) }, - { "Rx + Tx 128-255 Octet Frames", CPSW_STAT(octetframes128t255) }, - { "Rx + Tx 256-511 Octet Frames", CPSW_STAT(octetframes256t511) }, - { "Rx + Tx 512-1023 Octet Frames", CPSW_STAT(octetframes512t1023) }, - { "Rx + Tx 1024-Up Octet Frames", CPSW_STAT(octetframes1024tup) }, - { "Net Octets", CPSW_STAT(netoctets) }, - { "Rx Start of Frame Overruns", CPSW_STAT(rxsofoverruns) }, - { "Rx Middle of Frame Overruns", CPSW_STAT(rxmofoverruns) }, - { "Rx DMA Overruns", CPSW_STAT(rxdmaoverruns) }, -}; - -static const struct cpsw_stats cpsw_gstrings_ch_stats[] = { - { "head_enqueue", CPDMA_RX_STAT(head_enqueue) }, - { "tail_enqueue", CPDMA_RX_STAT(tail_enqueue) }, - { "pad_enqueue", CPDMA_RX_STAT(pad_enqueue) }, - { "misqueued", CPDMA_RX_STAT(misqueued) }, - { "desc_alloc_fail", CPDMA_RX_STAT(desc_alloc_fail) }, - { "pad_alloc_fail", CPDMA_RX_STAT(pad_alloc_fail) }, - { "runt_receive_buf", CPDMA_RX_STAT(runt_receive_buff) }, - { "runt_transmit_buf", CPDMA_RX_STAT(runt_transmit_buff) }, - { "empty_dequeue", CPDMA_RX_STAT(empty_dequeue) }, - { "busy_dequeue", CPDMA_RX_STAT(busy_dequeue) }, - { "good_dequeue", CPDMA_RX_STAT(good_dequeue) }, - { "requeue", CPDMA_RX_STAT(requeue) }, - { "teardown_dequeue", CPDMA_RX_STAT(teardown_dequeue) }, -}; - -#define CPSW_STATS_COMMON_LEN ARRAY_SIZE(cpsw_gstrings_stats) -#define CPSW_STATS_CH_LEN ARRAY_SIZE(cpsw_gstrings_ch_stats) - -#define ndev_to_cpsw(ndev) (((struct cpsw_priv *)netdev_priv(ndev))->cpsw) -#define napi_to_cpsw(napi) container_of(napi, struct cpsw_common, napi) #define for_each_slave(priv, func, arg...) \ do { \ struct cpsw_slave *slave; \ @@ -572,11 +77,6 @@ static const struct cpsw_stats cpsw_gstrings_ch_stats[] = { static int cpsw_ndo_vlan_rx_add_vid(struct net_device *ndev, __be16 proto, u16 vid); -static inline int cpsw_get_slave_port(u32 slave_num) -{ - return slave_num + 1; -} - static void cpsw_set_promiscious(struct net_device *ndev, bool enable) { struct cpsw_common *cpsw = ndev_to_cpsw(ndev); @@ -653,13 +153,6 @@ static void cpsw_set_promiscious(struct net_device *ndev, bool enable) } } -struct addr_sync_ctx { - struct net_device *ndev; - const u8 *addr; /* address to be synched */ - int consumed; /* number of address instances */ - int flush; /* flush flag */ -}; - /** * cpsw_set_mc - adds multicast entry to the table if it's not added or deletes * if it's not deleted @@ -800,12 +293,17 @@ static int cpsw_purge_all_mc(struct net_device *ndev, const u8 *addr, int num) static void cpsw_ndo_set_rx_mode(struct net_device *ndev) { - struct cpsw_common *cpsw = ndev_to_cpsw(ndev); + struct cpsw_priv *priv = netdev_priv(ndev); + struct cpsw_common *cpsw = priv->cpsw; + int slave_port = -1; + + if (cpsw->data.dual_emac) + slave_port = priv->emac_port + 1; if (ndev->flags & IFF_PROMISC) { /* Enable promiscuous mode */ cpsw_set_promiscious(ndev, true); - cpsw_ale_set_allmulti(cpsw->ale, IFF_ALLMULTI); + cpsw_ale_set_allmulti(cpsw->ale, IFF_ALLMULTI, slave_port); return; } else { /* Disable promiscuous mode */ @@ -813,14 +311,15 @@ static void cpsw_ndo_set_rx_mode(struct net_device *ndev) } /* Restore allmulti on vlans if necessary */ - cpsw_ale_set_allmulti(cpsw->ale, ndev->flags & IFF_ALLMULTI); + cpsw_ale_set_allmulti(cpsw->ale, + ndev->flags & IFF_ALLMULTI, slave_port); /* add/remove mcast address either for real netdev or for vlan */ __hw_addr_ref_sync_dev(&ndev->mc, ndev, cpsw_add_mc_addr, cpsw_del_mc_addr); } -static void cpsw_intr_enable(struct cpsw_common *cpsw) +void cpsw_intr_enable(struct cpsw_common *cpsw) { writel_relaxed(0xFF, &cpsw->wr_regs->tx_en); writel_relaxed(0xFF, &cpsw->wr_regs->rx_en); @@ -829,7 +328,7 @@ static void cpsw_intr_enable(struct cpsw_common *cpsw) return; } -static void cpsw_intr_disable(struct cpsw_common *cpsw) +void cpsw_intr_disable(struct cpsw_common *cpsw) { writel_relaxed(0, &cpsw->wr_regs->tx_en); writel_relaxed(0, &cpsw->wr_regs->rx_en); @@ -838,7 +337,7 @@ static void cpsw_intr_disable(struct cpsw_common *cpsw) return; } -static void cpsw_tx_handler(void *token, int len, int status) +void cpsw_tx_handler(void *token, int len, int status) { struct netdev_queue *txq; struct sk_buff *skb = token; @@ -970,11 +469,9 @@ requeue: dev_kfree_skb_any(new_skb); } -static void cpsw_split_res(struct net_device *ndev) +void cpsw_split_res(struct cpsw_common *cpsw) { - struct cpsw_priv *priv = netdev_priv(ndev); u32 consumed_rate = 0, bigest_rate = 0; - struct cpsw_common *cpsw = priv->cpsw; struct cpsw_vector *txv = cpsw->txv; int i, ch_weight, rlim_ch_num = 0; int budget, bigest_rate_ch = 0; @@ -1254,29 +751,32 @@ static void _cpsw_adjust_link(struct cpsw_slave *slave, slave_port = cpsw_get_slave_port(slave->slave_num); if (phy->link) { - mac_control = cpsw->data.mac_control; - - /* enable forwarding */ - cpsw_ale_control_set(cpsw->ale, slave_port, - ALE_PORT_STATE, ALE_PORT_STATE_FORWARD); + mac_control = CPSW_SL_CTL_GMII_EN; if (phy->speed == 1000) - mac_control |= BIT(7); /* GIGABITEN */ + mac_control |= CPSW_SL_CTL_GIG; if (phy->duplex) - mac_control |= BIT(0); /* FULLDUPLEXEN */ + mac_control |= CPSW_SL_CTL_FULLDUPLEX; /* set speed_in input in case RMII mode is used in 100Mbps */ if (phy->speed == 100) - mac_control |= BIT(15); + mac_control |= CPSW_SL_CTL_IFCTL_A; /* in band mode only works in 10Mbps RGMII mode */ else if ((phy->speed == 10) && phy_interface_is_rgmii(phy)) - mac_control |= BIT(18); /* In Band mode */ + mac_control |= CPSW_SL_CTL_EXT_EN; /* In Band mode */ if (priv->rx_pause) - mac_control |= BIT(3); + mac_control |= CPSW_SL_CTL_RX_FLOW_EN; if (priv->tx_pause) - mac_control |= BIT(4); + mac_control |= CPSW_SL_CTL_TX_FLOW_EN; + + if (mac_control != slave->mac_control) + cpsw_sl_ctl_set(slave->mac_sl, mac_control); + + /* enable forwarding */ + cpsw_ale_control_set(cpsw->ale, slave_port, + ALE_PORT_STATE, ALE_PORT_STATE_FORWARD); *link = true; @@ -1290,12 +790,14 @@ static void _cpsw_adjust_link(struct cpsw_slave *slave, /* disable forwarding */ cpsw_ale_control_set(cpsw->ale, slave_port, ALE_PORT_STATE, ALE_PORT_STATE_DISABLE); + + cpsw_sl_wait_for_idle(slave->mac_sl, 100); + + cpsw_sl_ctl_reset(slave->mac_sl); } - if (mac_control != slave->mac_control) { + if (mac_control != slave->mac_control) phy_print_status(phy); - writel_relaxed(mac_control, &slave->sliver->mac_control); - } slave->mac_control = mac_control; } @@ -1348,7 +850,7 @@ static void cpsw_adjust_link(struct net_device *ndev) if (link) { if (cpsw_need_resplit(cpsw)) - cpsw_split_res(ndev); + cpsw_split_res(cpsw); netif_carrier_on(ndev); if (netif_running(ndev)) @@ -1359,167 +861,6 @@ static void cpsw_adjust_link(struct net_device *ndev) } } -static int cpsw_get_coalesce(struct net_device *ndev, - struct ethtool_coalesce *coal) -{ - struct cpsw_common *cpsw = ndev_to_cpsw(ndev); - - coal->rx_coalesce_usecs = cpsw->coal_intvl; - return 0; -} - -static int cpsw_set_coalesce(struct net_device *ndev, - struct ethtool_coalesce *coal) -{ - struct cpsw_priv *priv = netdev_priv(ndev); - u32 int_ctrl; - u32 num_interrupts = 0; - u32 prescale = 0; - u32 addnl_dvdr = 1; - u32 coal_intvl = 0; - struct cpsw_common *cpsw = priv->cpsw; - - coal_intvl = coal->rx_coalesce_usecs; - - int_ctrl = readl(&cpsw->wr_regs->int_control); - prescale = cpsw->bus_freq_mhz * 4; - - if (!coal->rx_coalesce_usecs) { - int_ctrl &= ~(CPSW_INTPRESCALE_MASK | CPSW_INTPACEEN); - goto update_return; - } - - if (coal_intvl < CPSW_CMINTMIN_INTVL) - coal_intvl = CPSW_CMINTMIN_INTVL; - - if (coal_intvl > CPSW_CMINTMAX_INTVL) { - /* Interrupt pacer works with 4us Pulse, we can - * throttle further by dilating the 4us pulse. - */ - addnl_dvdr = CPSW_INTPRESCALE_MASK / prescale; - - if (addnl_dvdr > 1) { - prescale *= addnl_dvdr; - if (coal_intvl > (CPSW_CMINTMAX_INTVL * addnl_dvdr)) - coal_intvl = (CPSW_CMINTMAX_INTVL - * addnl_dvdr); - } else { - addnl_dvdr = 1; - coal_intvl = CPSW_CMINTMAX_INTVL; - } - } - - num_interrupts = (1000 * addnl_dvdr) / coal_intvl; - writel(num_interrupts, &cpsw->wr_regs->rx_imax); - writel(num_interrupts, &cpsw->wr_regs->tx_imax); - - int_ctrl |= CPSW_INTPACEEN; - int_ctrl &= (~CPSW_INTPRESCALE_MASK); - int_ctrl |= (prescale & CPSW_INTPRESCALE_MASK); - -update_return: - writel(int_ctrl, &cpsw->wr_regs->int_control); - - cpsw_notice(priv, timer, "Set coalesce to %d usecs.\n", coal_intvl); - cpsw->coal_intvl = coal_intvl; - - return 0; -} - -static int cpsw_get_sset_count(struct net_device *ndev, int sset) -{ - struct cpsw_common *cpsw = ndev_to_cpsw(ndev); - - switch (sset) { - case ETH_SS_STATS: - return (CPSW_STATS_COMMON_LEN + - (cpsw->rx_ch_num + cpsw->tx_ch_num) * - CPSW_STATS_CH_LEN); - default: - return -EOPNOTSUPP; - } -} - -static void cpsw_add_ch_strings(u8 **p, int ch_num, int rx_dir) -{ - int ch_stats_len; - int line; - int i; - - ch_stats_len = CPSW_STATS_CH_LEN * ch_num; - for (i = 0; i < ch_stats_len; i++) { - line = i % CPSW_STATS_CH_LEN; - snprintf(*p, ETH_GSTRING_LEN, - "%s DMA chan %ld: %s", rx_dir ? "Rx" : "Tx", - (long)(i / CPSW_STATS_CH_LEN), - cpsw_gstrings_ch_stats[line].stat_string); - *p += ETH_GSTRING_LEN; - } -} - -static void cpsw_get_strings(struct net_device *ndev, u32 stringset, u8 *data) -{ - struct cpsw_common *cpsw = ndev_to_cpsw(ndev); - u8 *p = data; - int i; - - switch (stringset) { - case ETH_SS_STATS: - for (i = 0; i < CPSW_STATS_COMMON_LEN; i++) { - memcpy(p, cpsw_gstrings_stats[i].stat_string, - ETH_GSTRING_LEN); - p += ETH_GSTRING_LEN; - } - - cpsw_add_ch_strings(&p, cpsw->rx_ch_num, 1); - cpsw_add_ch_strings(&p, cpsw->tx_ch_num, 0); - break; - } -} - -static void cpsw_get_ethtool_stats(struct net_device *ndev, - struct ethtool_stats *stats, u64 *data) -{ - u8 *p; - struct cpsw_common *cpsw = ndev_to_cpsw(ndev); - struct cpdma_chan_stats ch_stats; - int i, l, ch; - - /* Collect Davinci CPDMA stats for Rx and Tx Channel */ - for (l = 0; l < CPSW_STATS_COMMON_LEN; l++) - data[l] = readl(cpsw->hw_stats + - cpsw_gstrings_stats[l].stat_offset); - - for (ch = 0; ch < cpsw->rx_ch_num; ch++) { - cpdma_chan_get_stats(cpsw->rxv[ch].ch, &ch_stats); - for (i = 0; i < CPSW_STATS_CH_LEN; i++, l++) { - p = (u8 *)&ch_stats + - cpsw_gstrings_ch_stats[i].stat_offset; - data[l] = *(u32 *)p; - } - } - - for (ch = 0; ch < cpsw->tx_ch_num; ch++) { - cpdma_chan_get_stats(cpsw->txv[ch].ch, &ch_stats); - for (i = 0; i < CPSW_STATS_CH_LEN; i++, l++) { - p = (u8 *)&ch_stats + - cpsw_gstrings_ch_stats[i].stat_offset; - data[l] = *(u32 *)p; - } - } -} - -static inline int cpsw_tx_packet_submit(struct cpsw_priv *priv, - struct sk_buff *skb, - struct cpdma_chan *txch) -{ - struct cpsw_common *cpsw = priv->cpsw; - - skb_tx_timestamp(skb); - return cpdma_chan_submit(txch, skb, skb->data, skb->len, - priv->emac_port + cpsw->data.dual_emac); -} - static inline void cpsw_add_dual_emac_def_ale_entries( struct cpsw_priv *priv, struct cpsw_slave *slave, u32 slave_port) @@ -1542,24 +883,18 @@ static inline void cpsw_add_dual_emac_def_ale_entries( ALE_PORT_DROP_UNKNOWN_VLAN, 1); } -static void soft_reset_slave(struct cpsw_slave *slave) -{ - char name[32]; - - snprintf(name, sizeof(name), "slave-%d", slave->slave_num); - soft_reset(name, &slave->sliver->soft_reset); -} - static void cpsw_slave_open(struct cpsw_slave *slave, struct cpsw_priv *priv) { u32 slave_port; struct phy_device *phy; struct cpsw_common *cpsw = priv->cpsw; - soft_reset_slave(slave); + cpsw_sl_reset(slave->mac_sl, 100); + cpsw_sl_ctl_reset(slave->mac_sl); /* setup priority mapping */ - writel_relaxed(RX_PRIORITY_MAPPING, &slave->sliver->rx_pri_map); + cpsw_sl_reg_write(slave->mac_sl, CPSW_SL_RX_PRI_MAP, + RX_PRIORITY_MAPPING); switch (cpsw->version) { case CPSW_VERSION_1: @@ -1585,7 +920,8 @@ static void cpsw_slave_open(struct cpsw_slave *slave, struct cpsw_priv *priv) } /* setup max packet size, and mac address */ - writel_relaxed(cpsw->rx_packet_max, &slave->sliver->rx_maxlen); + cpsw_sl_reg_write(slave->mac_sl, CPSW_SL_RX_MAXLEN, + cpsw->rx_packet_max); cpsw_set_slave_mac(slave, priv); slave->mac_control = 0; /* no link yet */ @@ -1696,7 +1032,7 @@ static void cpsw_init_host_port(struct cpsw_priv *priv) } } -static int cpsw_fill_rx_channels(struct cpsw_priv *priv) +int cpsw_fill_rx_channels(struct cpsw_priv *priv) { struct cpsw_common *cpsw = priv->cpsw; struct sk_buff *skb; @@ -1748,7 +1084,8 @@ static void cpsw_slave_stop(struct cpsw_slave *slave, struct cpsw_common *cpsw) slave->phy = NULL; cpsw_ale_control_set(cpsw->ale, slave_port, ALE_PORT_STATE, ALE_PORT_STATE_DISABLE); - soft_reset_slave(slave); + cpsw_sl_reset(slave->mac_sl, 100); + cpsw_sl_ctl_reset(slave->mac_sl); } static int cpsw_tc_to_fifo(int tc, int num_tc) @@ -2114,7 +1451,7 @@ static int cpsw_ndo_stop(struct net_device *ndev) for_each_slave(priv, cpsw_slave_stop, cpsw); if (cpsw_need_resplit(cpsw)) - cpsw_split_res(ndev); + cpsw_split_res(cpsw); cpsw->usage_count--; pm_runtime_put_sync(cpsw->dev); @@ -2147,7 +1484,9 @@ static netdev_tx_t cpsw_ndo_start_xmit(struct sk_buff *skb, txch = cpsw->txv[q_idx].ch; txq = netdev_get_tx_queue(ndev, q_idx); - ret = cpsw_tx_packet_submit(priv, skb, txch); + skb_tx_timestamp(skb); + ret = cpdma_chan_submit(txch, skb, skb->data, skb->len, + priv->emac_port + cpsw->data.dual_emac); if (unlikely(ret != 0)) { cpsw_err(priv, tx_err, "desc submit failed\n"); goto fail; @@ -2418,18 +1757,6 @@ static int cpsw_ndo_set_mac_address(struct net_device *ndev, void *p) return 0; } -#ifdef CONFIG_NET_POLL_CONTROLLER -static void cpsw_ndo_poll_controller(struct net_device *ndev) -{ - struct cpsw_common *cpsw = ndev_to_cpsw(ndev); - - cpsw_intr_disable(cpsw); - cpsw_rx_interrupt(cpsw->irqs_table[0], cpsw); - cpsw_tx_interrupt(cpsw->irqs_table[1], cpsw); - cpsw_intr_enable(cpsw); -} -#endif - static inline int cpsw_add_vlan_ale_entry(struct cpsw_priv *priv, unsigned short vid) { @@ -2601,7 +1928,7 @@ static int cpsw_ndo_set_tx_maxrate(struct net_device *ndev, int queue, u32 rate) netdev_get_tx_queue(slave->ndev, queue)->tx_maxrate = rate; } - cpsw_split_res(ndev); + cpsw_split_res(cpsw); return ret; } @@ -2677,6 +2004,18 @@ static int cpsw_ndo_setup_tc(struct net_device *ndev, enum tc_setup_type type, } } +#ifdef CONFIG_NET_POLL_CONTROLLER +static void cpsw_ndo_poll_controller(struct net_device *ndev) +{ + struct cpsw_common *cpsw = ndev_to_cpsw(ndev); + + cpsw_intr_disable(cpsw); + cpsw_rx_interrupt(cpsw->irqs_table[0], cpsw); + cpsw_tx_interrupt(cpsw->irqs_table[1], cpsw); + cpsw_intr_enable(cpsw); +} +#endif + static const struct net_device_ops cpsw_netdev_ops = { .ndo_open = cpsw_ndo_open, .ndo_stop = cpsw_ndo_stop, @@ -2695,25 +2034,6 @@ static const struct net_device_ops cpsw_netdev_ops = { .ndo_setup_tc = cpsw_ndo_setup_tc, }; -static int cpsw_get_regs_len(struct net_device *ndev) -{ - struct cpsw_common *cpsw = ndev_to_cpsw(ndev); - - return cpsw->data.ale_entries * ALE_ENTRY_WORDS * sizeof(u32); -} - -static void cpsw_get_regs(struct net_device *ndev, - struct ethtool_regs *regs, void *p) -{ - u32 *reg = p; - struct cpsw_common *cpsw = ndev_to_cpsw(ndev); - - /* update CPSW IP version */ - regs->version = cpsw->version; - - cpsw_ale_dump(cpsw->ale, reg); -} - static void cpsw_get_drvinfo(struct net_device *ndev, struct ethtool_drvinfo *info) { @@ -2725,119 +2045,6 @@ static void cpsw_get_drvinfo(struct net_device *ndev, strlcpy(info->bus_info, pdev->name, sizeof(info->bus_info)); } -static u32 cpsw_get_msglevel(struct net_device *ndev) -{ - struct cpsw_priv *priv = netdev_priv(ndev); - return priv->msg_enable; -} - -static void cpsw_set_msglevel(struct net_device *ndev, u32 value) -{ - struct cpsw_priv *priv = netdev_priv(ndev); - priv->msg_enable = value; -} - -#if IS_ENABLED(CONFIG_TI_CPTS) -static int cpsw_get_ts_info(struct net_device *ndev, - struct ethtool_ts_info *info) -{ - struct cpsw_common *cpsw = ndev_to_cpsw(ndev); - - info->so_timestamping = - SOF_TIMESTAMPING_TX_HARDWARE | - SOF_TIMESTAMPING_TX_SOFTWARE | - SOF_TIMESTAMPING_RX_HARDWARE | - SOF_TIMESTAMPING_RX_SOFTWARE | - SOF_TIMESTAMPING_SOFTWARE | - SOF_TIMESTAMPING_RAW_HARDWARE; - info->phc_index = cpsw->cpts->phc_index; - info->tx_types = - (1 << HWTSTAMP_TX_OFF) | - (1 << HWTSTAMP_TX_ON); - info->rx_filters = - (1 << HWTSTAMP_FILTER_NONE) | - (1 << HWTSTAMP_FILTER_PTP_V1_L4_EVENT) | - (1 << HWTSTAMP_FILTER_PTP_V2_EVENT); - return 0; -} -#else -static int cpsw_get_ts_info(struct net_device *ndev, - struct ethtool_ts_info *info) -{ - info->so_timestamping = - SOF_TIMESTAMPING_TX_SOFTWARE | - SOF_TIMESTAMPING_RX_SOFTWARE | - SOF_TIMESTAMPING_SOFTWARE; - info->phc_index = -1; - info->tx_types = 0; - info->rx_filters = 0; - return 0; -} -#endif - -static int cpsw_get_link_ksettings(struct net_device *ndev, - struct ethtool_link_ksettings *ecmd) -{ - struct cpsw_priv *priv = netdev_priv(ndev); - struct cpsw_common *cpsw = priv->cpsw; - int slave_no = cpsw_slave_index(cpsw, priv); - - if (!cpsw->slaves[slave_no].phy) - return -EOPNOTSUPP; - - phy_ethtool_ksettings_get(cpsw->slaves[slave_no].phy, ecmd); - return 0; -} - -static int cpsw_set_link_ksettings(struct net_device *ndev, - const struct ethtool_link_ksettings *ecmd) -{ - struct cpsw_priv *priv = netdev_priv(ndev); - struct cpsw_common *cpsw = priv->cpsw; - int slave_no = cpsw_slave_index(cpsw, priv); - - if (cpsw->slaves[slave_no].phy) - return phy_ethtool_ksettings_set(cpsw->slaves[slave_no].phy, - ecmd); - else - return -EOPNOTSUPP; -} - -static void cpsw_get_wol(struct net_device *ndev, struct ethtool_wolinfo *wol) -{ - struct cpsw_priv *priv = netdev_priv(ndev); - struct cpsw_common *cpsw = priv->cpsw; - int slave_no = cpsw_slave_index(cpsw, priv); - - wol->supported = 0; - wol->wolopts = 0; - - if (cpsw->slaves[slave_no].phy) - phy_ethtool_get_wol(cpsw->slaves[slave_no].phy, wol); -} - -static int cpsw_set_wol(struct net_device *ndev, struct ethtool_wolinfo *wol) -{ - struct cpsw_priv *priv = netdev_priv(ndev); - struct cpsw_common *cpsw = priv->cpsw; - int slave_no = cpsw_slave_index(cpsw, priv); - - if (cpsw->slaves[slave_no].phy) - return phy_ethtool_set_wol(cpsw->slaves[slave_no].phy, wol); - else - return -EOPNOTSUPP; -} - -static void cpsw_get_pauseparam(struct net_device *ndev, - struct ethtool_pauseparam *pause) -{ - struct cpsw_priv *priv = netdev_priv(ndev); - - pause->autoneg = AUTONEG_DISABLE; - pause->rx_pause = priv->rx_pause ? true : false; - pause->tx_pause = priv->tx_pause ? true : false; -} - static int cpsw_set_pauseparam(struct net_device *ndev, struct ethtool_pauseparam *pause) { @@ -2851,316 +2058,10 @@ static int cpsw_set_pauseparam(struct net_device *ndev, return 0; } -static int cpsw_ethtool_op_begin(struct net_device *ndev) +static int cpsw_set_channels(struct net_device *ndev, + struct ethtool_channels *chs) { - struct cpsw_priv *priv = netdev_priv(ndev); - struct cpsw_common *cpsw = priv->cpsw; - int ret; - - ret = pm_runtime_get_sync(cpsw->dev); - if (ret < 0) { - cpsw_err(priv, drv, "ethtool begin failed %d\n", ret); - pm_runtime_put_noidle(cpsw->dev); - } - - return ret; -} - -static void cpsw_ethtool_op_complete(struct net_device *ndev) -{ - struct cpsw_priv *priv = netdev_priv(ndev); - int ret; - - ret = pm_runtime_put(priv->cpsw->dev); - if (ret < 0) - cpsw_err(priv, drv, "ethtool complete failed %d\n", ret); -} - -static void cpsw_get_channels(struct net_device *ndev, - struct ethtool_channels *ch) -{ - struct cpsw_common *cpsw = ndev_to_cpsw(ndev); - - ch->max_rx = cpsw->quirk_irq ? 1 : CPSW_MAX_QUEUES; - ch->max_tx = cpsw->quirk_irq ? 1 : CPSW_MAX_QUEUES; - ch->max_combined = 0; - ch->max_other = 0; - ch->other_count = 0; - ch->rx_count = cpsw->rx_ch_num; - ch->tx_count = cpsw->tx_ch_num; - ch->combined_count = 0; -} - -static int cpsw_check_ch_settings(struct cpsw_common *cpsw, - struct ethtool_channels *ch) -{ - if (cpsw->quirk_irq) { - dev_err(cpsw->dev, "Maximum one tx/rx queue is allowed"); - return -EOPNOTSUPP; - } - - if (ch->combined_count) - return -EINVAL; - - /* verify we have at least one channel in each direction */ - if (!ch->rx_count || !ch->tx_count) - return -EINVAL; - - if (ch->rx_count > cpsw->data.channels || - ch->tx_count > cpsw->data.channels) - return -EINVAL; - - return 0; -} - -static int cpsw_update_channels_res(struct cpsw_priv *priv, int ch_num, int rx) -{ - struct cpsw_common *cpsw = priv->cpsw; - void (*handler)(void *, int, int); - struct netdev_queue *queue; - struct cpsw_vector *vec; - int ret, *ch, vch; - - if (rx) { - ch = &cpsw->rx_ch_num; - vec = cpsw->rxv; - handler = cpsw_rx_handler; - } else { - ch = &cpsw->tx_ch_num; - vec = cpsw->txv; - handler = cpsw_tx_handler; - } - - while (*ch < ch_num) { - vch = rx ? *ch : 7 - *ch; - vec[*ch].ch = cpdma_chan_create(cpsw->dma, vch, handler, rx); - queue = netdev_get_tx_queue(priv->ndev, *ch); - queue->tx_maxrate = 0; - - if (IS_ERR(vec[*ch].ch)) - return PTR_ERR(vec[*ch].ch); - - if (!vec[*ch].ch) - return -EINVAL; - - cpsw_info(priv, ifup, "created new %d %s channel\n", *ch, - (rx ? "rx" : "tx")); - (*ch)++; - } - - while (*ch > ch_num) { - (*ch)--; - - ret = cpdma_chan_destroy(vec[*ch].ch); - if (ret) - return ret; - - cpsw_info(priv, ifup, "destroyed %d %s channel\n", *ch, - (rx ? "rx" : "tx")); - } - - return 0; -} - -static int cpsw_update_channels(struct cpsw_priv *priv, - struct ethtool_channels *ch) -{ - int ret; - - ret = cpsw_update_channels_res(priv, ch->rx_count, 1); - if (ret) - return ret; - - ret = cpsw_update_channels_res(priv, ch->tx_count, 0); - if (ret) - return ret; - - return 0; -} - -static void cpsw_suspend_data_pass(struct net_device *ndev) -{ - struct cpsw_common *cpsw = ndev_to_cpsw(ndev); - struct cpsw_slave *slave; - int i; - - /* Disable NAPI scheduling */ - cpsw_intr_disable(cpsw); - - /* Stop all transmit queues for every network device. - * Disable re-using rx descriptors with dormant_on. - */ - for (i = cpsw->data.slaves, slave = cpsw->slaves; i; i--, slave++) { - if (!(slave->ndev && netif_running(slave->ndev))) - continue; - - netif_tx_stop_all_queues(slave->ndev); - netif_dormant_on(slave->ndev); - } - - /* Handle rest of tx packets and stop cpdma channels */ - cpdma_ctlr_stop(cpsw->dma); -} - -static int cpsw_resume_data_pass(struct net_device *ndev) -{ - struct cpsw_priv *priv = netdev_priv(ndev); - struct cpsw_common *cpsw = priv->cpsw; - struct cpsw_slave *slave; - int i, ret; - - /* Allow rx packets handling */ - for (i = cpsw->data.slaves, slave = cpsw->slaves; i; i--, slave++) - if (slave->ndev && netif_running(slave->ndev)) - netif_dormant_off(slave->ndev); - - /* After this receive is started */ - if (cpsw->usage_count) { - ret = cpsw_fill_rx_channels(priv); - if (ret) - return ret; - - cpdma_ctlr_start(cpsw->dma); - cpsw_intr_enable(cpsw); - } - - /* Resume transmit for every affected interface */ - for (i = cpsw->data.slaves, slave = cpsw->slaves; i; i--, slave++) - if (slave->ndev && netif_running(slave->ndev)) - netif_tx_start_all_queues(slave->ndev); - - return 0; -} - -static int cpsw_set_channels(struct net_device *ndev, - struct ethtool_channels *chs) -{ - struct cpsw_priv *priv = netdev_priv(ndev); - struct cpsw_common *cpsw = priv->cpsw; - struct cpsw_slave *slave; - int i, ret; - - ret = cpsw_check_ch_settings(cpsw, chs); - if (ret < 0) - return ret; - - cpsw_suspend_data_pass(ndev); - ret = cpsw_update_channels(priv, chs); - if (ret) - goto err; - - for (i = cpsw->data.slaves, slave = cpsw->slaves; i; i--, slave++) { - if (!(slave->ndev && netif_running(slave->ndev))) - continue; - - /* Inform stack about new count of queues */ - ret = netif_set_real_num_tx_queues(slave->ndev, - cpsw->tx_ch_num); - if (ret) { - dev_err(priv->dev, "cannot set real number of tx queues\n"); - goto err; - } - - ret = netif_set_real_num_rx_queues(slave->ndev, - cpsw->rx_ch_num); - if (ret) { - dev_err(priv->dev, "cannot set real number of rx queues\n"); - goto err; - } - } - - if (cpsw->usage_count) - cpsw_split_res(ndev); - - ret = cpsw_resume_data_pass(ndev); - if (!ret) - return 0; -err: - dev_err(priv->dev, "cannot update channels number, closing device\n"); - dev_close(ndev); - return ret; -} - -static int cpsw_get_eee(struct net_device *ndev, struct ethtool_eee *edata) -{ - struct cpsw_priv *priv = netdev_priv(ndev); - struct cpsw_common *cpsw = priv->cpsw; - int slave_no = cpsw_slave_index(cpsw, priv); - - if (cpsw->slaves[slave_no].phy) - return phy_ethtool_get_eee(cpsw->slaves[slave_no].phy, edata); - else - return -EOPNOTSUPP; -} - -static int cpsw_set_eee(struct net_device *ndev, struct ethtool_eee *edata) -{ - struct cpsw_priv *priv = netdev_priv(ndev); - struct cpsw_common *cpsw = priv->cpsw; - int slave_no = cpsw_slave_index(cpsw, priv); - - if (cpsw->slaves[slave_no].phy) - return phy_ethtool_set_eee(cpsw->slaves[slave_no].phy, edata); - else - return -EOPNOTSUPP; -} - -static int cpsw_nway_reset(struct net_device *ndev) -{ - struct cpsw_priv *priv = netdev_priv(ndev); - struct cpsw_common *cpsw = priv->cpsw; - int slave_no = cpsw_slave_index(cpsw, priv); - - if (cpsw->slaves[slave_no].phy) - return genphy_restart_aneg(cpsw->slaves[slave_no].phy); - else - return -EOPNOTSUPP; -} - -static void cpsw_get_ringparam(struct net_device *ndev, - struct ethtool_ringparam *ering) -{ - struct cpsw_priv *priv = netdev_priv(ndev); - struct cpsw_common *cpsw = priv->cpsw; - - /* not supported */ - ering->tx_max_pending = 0; - ering->tx_pending = cpdma_get_num_tx_descs(cpsw->dma); - ering->rx_max_pending = descs_pool_size - CPSW_MAX_QUEUES; - ering->rx_pending = cpdma_get_num_rx_descs(cpsw->dma); -} - -static int cpsw_set_ringparam(struct net_device *ndev, - struct ethtool_ringparam *ering) -{ - struct cpsw_priv *priv = netdev_priv(ndev); - struct cpsw_common *cpsw = priv->cpsw; - int ret; - - /* ignore ering->tx_pending - only rx_pending adjustment is supported */ - - if (ering->rx_mini_pending || ering->rx_jumbo_pending || - ering->rx_pending < CPSW_MAX_QUEUES || - ering->rx_pending > (descs_pool_size - CPSW_MAX_QUEUES)) - return -EINVAL; - - if (ering->rx_pending == cpdma_get_num_rx_descs(cpsw->dma)) - return 0; - - cpsw_suspend_data_pass(ndev); - - cpdma_set_num_rx_descs(cpsw->dma, ering->rx_pending); - - if (cpsw->usage_count) - cpdma_chan_split_pool(cpsw->dma); - - ret = cpsw_resume_data_pass(ndev); - if (!ret) - return 0; - - dev_err(&ndev->dev, "cannot set ring params, closing device\n"); - dev_close(ndev); - return ret; + return cpsw_set_channels_common(ndev, chs, cpsw_rx_handler); } static const struct ethtool_ops cpsw_ethtool_ops = { @@ -3193,19 +2094,6 @@ static const struct ethtool_ops cpsw_ethtool_ops = { .set_ringparam = cpsw_set_ringparam, }; -static void cpsw_slave_init(struct cpsw_slave *slave, struct cpsw_common *cpsw, - u32 slave_reg_ofs, u32 sliver_reg_ofs) -{ - void __iomem *regs = cpsw->regs; - int slave_num = slave->slave_num; - struct cpsw_slave_data *data = cpsw->data.slave_data + slave_num; - - slave->data = data; - slave->regs = regs + slave_reg_ofs; - slave->sliver = regs + sliver_reg_ofs; - slave->port_vlan = data->dual_emac_res_vlan; -} - static int cpsw_probe_dt(struct cpsw_platform_data *data, struct platform_device *pdev) { @@ -3344,7 +2232,7 @@ static int cpsw_probe_dt(struct cpsw_platform_data *data, no_phy_slave: mac_addr = of_get_mac_address(slave_node); - if (mac_addr) { + if (!IS_ERR(mac_addr)) { memcpy(slave_data->mac_addr, mac_addr, ETH_ALEN); } else { ret = ti_cm_get_macid(&pdev->dev, i, @@ -3408,7 +2296,8 @@ static int cpsw_probe_dual_emac(struct cpsw_priv *priv) struct cpsw_priv *priv_sl2; int ret = 0; - ndev = alloc_etherdev_mq(sizeof(struct cpsw_priv), CPSW_MAX_QUEUES); + ndev = devm_alloc_etherdev_mqs(cpsw->dev, sizeof(struct cpsw_priv), + CPSW_MAX_QUEUES, CPSW_MAX_QUEUES); if (!ndev) { dev_err(cpsw->dev, "cpsw: error allocating net_device\n"); return -ENOMEM; @@ -3442,11 +2331,8 @@ static int cpsw_probe_dual_emac(struct cpsw_priv *priv) /* register the network device */ SET_NETDEV_DEV(ndev, cpsw->dev); ret = register_netdev(ndev); - if (ret) { + if (ret) dev_err(cpsw->dev, "cpsw: error registering net device\n"); - free_netdev(ndev); - ret = -ENODEV; - } return ret; } @@ -3467,63 +2353,74 @@ static const struct soc_device_attribute cpsw_soc_devices[] = { static int cpsw_probe(struct platform_device *pdev) { + struct device *dev = &pdev->dev; struct clk *clk; struct cpsw_platform_data *data; struct net_device *ndev; struct cpsw_priv *priv; - struct cpdma_params dma_params; - struct cpsw_ale_params ale_params; void __iomem *ss_regs; - void __iomem *cpts_regs; struct resource *res, *ss_res; struct gpio_descs *mode; - u32 slave_offset, sliver_offset, slave_size; const struct soc_device_attribute *soc; struct cpsw_common *cpsw; - int ret = 0, i, ch; + int ret = 0, ch; int irq; - cpsw = devm_kzalloc(&pdev->dev, sizeof(struct cpsw_common), GFP_KERNEL); + cpsw = devm_kzalloc(dev, sizeof(struct cpsw_common), GFP_KERNEL); if (!cpsw) return -ENOMEM; - cpsw->dev = &pdev->dev; + cpsw->dev = dev; - ndev = alloc_etherdev_mq(sizeof(struct cpsw_priv), CPSW_MAX_QUEUES); - if (!ndev) { - dev_err(&pdev->dev, "error allocating net_device\n"); - return -ENOMEM; - } - - platform_set_drvdata(pdev, ndev); - priv = netdev_priv(ndev); - priv->cpsw = cpsw; - priv->ndev = ndev; - priv->dev = &ndev->dev; - priv->msg_enable = netif_msg_init(debug_level, CPSW_DEBUG); - cpsw->rx_packet_max = max(rx_packet_max, 128); - - mode = devm_gpiod_get_array_optional(&pdev->dev, "mode", GPIOD_OUT_LOW); + mode = devm_gpiod_get_array_optional(dev, "mode", GPIOD_OUT_LOW); if (IS_ERR(mode)) { ret = PTR_ERR(mode); - dev_err(&pdev->dev, "gpio request failed, ret %d\n", ret); - goto clean_ndev_ret; + dev_err(dev, "gpio request failed, ret %d\n", ret); + return ret; } + clk = devm_clk_get(dev, "fck"); + if (IS_ERR(clk)) { + ret = PTR_ERR(clk); + dev_err(dev, "fck is not found %d\n", ret); + return ret; + } + cpsw->bus_freq_mhz = clk_get_rate(clk) / 1000000; + + ss_res = platform_get_resource(pdev, IORESOURCE_MEM, 0); + ss_regs = devm_ioremap_resource(dev, ss_res); + if (IS_ERR(ss_regs)) + return PTR_ERR(ss_regs); + cpsw->regs = ss_regs; + + res = platform_get_resource(pdev, IORESOURCE_MEM, 1); + cpsw->wr_regs = devm_ioremap_resource(dev, res); + if (IS_ERR(cpsw->wr_regs)) + return PTR_ERR(cpsw->wr_regs); + + /* RX IRQ */ + irq = platform_get_irq(pdev, 1); + if (irq < 0) + return irq; + cpsw->irqs_table[0] = irq; + + /* TX IRQ */ + irq = platform_get_irq(pdev, 2); + if (irq < 0) + return irq; + cpsw->irqs_table[1] = irq; + /* * This may be required here for child devices. */ - pm_runtime_enable(&pdev->dev); - - /* Select default pin state */ - pinctrl_pm_select_default_state(&pdev->dev); + pm_runtime_enable(dev); /* Need to enable clocks with runtime PM api to access module * registers */ - ret = pm_runtime_get_sync(&pdev->dev); + ret = pm_runtime_get_sync(dev); if (ret < 0) { - pm_runtime_put_noidle(&pdev->dev); + pm_runtime_put_noidle(dev); goto clean_runtime_disable_ret; } @@ -3531,170 +2428,72 @@ static int cpsw_probe(struct platform_device *pdev) if (ret) goto clean_dt_ret; - data = &cpsw->data; - cpsw->rx_ch_num = 1; - cpsw->tx_ch_num = 1; - - if (is_valid_ether_addr(data->slave_data[0].mac_addr)) { - memcpy(priv->mac_addr, data->slave_data[0].mac_addr, ETH_ALEN); - dev_info(&pdev->dev, "Detected MACID = %pM\n", priv->mac_addr); - } else { - eth_random_addr(priv->mac_addr); - dev_info(&pdev->dev, "Random MACID = %pM\n", priv->mac_addr); - } - - memcpy(ndev->dev_addr, priv->mac_addr, ETH_ALEN); + soc = soc_device_match(cpsw_soc_devices); + if (soc) + cpsw->quirk_irq = 1; - cpsw->slaves = devm_kcalloc(&pdev->dev, + data = &cpsw->data; + cpsw->slaves = devm_kcalloc(dev, data->slaves, sizeof(struct cpsw_slave), GFP_KERNEL); if (!cpsw->slaves) { ret = -ENOMEM; goto clean_dt_ret; } - for (i = 0; i < data->slaves; i++) - cpsw->slaves[i].slave_num = i; - - cpsw->slaves[0].ndev = ndev; - priv->emac_port = 0; - - clk = devm_clk_get(&pdev->dev, "fck"); - if (IS_ERR(clk)) { - dev_err(priv->dev, "fck is not found\n"); - ret = -ENODEV; - goto clean_dt_ret; - } - cpsw->bus_freq_mhz = clk_get_rate(clk) / 1000000; - - ss_res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - ss_regs = devm_ioremap_resource(&pdev->dev, ss_res); - if (IS_ERR(ss_regs)) { - ret = PTR_ERR(ss_regs); - goto clean_dt_ret; - } - cpsw->regs = ss_regs; - - cpsw->version = readl(&cpsw->regs->id_ver); - - res = platform_get_resource(pdev, IORESOURCE_MEM, 1); - cpsw->wr_regs = devm_ioremap_resource(&pdev->dev, res); - if (IS_ERR(cpsw->wr_regs)) { - ret = PTR_ERR(cpsw->wr_regs); - goto clean_dt_ret; - } - memset(&dma_params, 0, sizeof(dma_params)); - memset(&ale_params, 0, sizeof(ale_params)); + cpsw->rx_packet_max = max(rx_packet_max, CPSW_MAX_PACKET_SIZE); + cpsw->descs_pool_size = descs_pool_size; - switch (cpsw->version) { - case CPSW_VERSION_1: - cpsw->host_port_regs = ss_regs + CPSW1_HOST_PORT_OFFSET; - cpts_regs = ss_regs + CPSW1_CPTS_OFFSET; - cpsw->hw_stats = ss_regs + CPSW1_HW_STATS; - dma_params.dmaregs = ss_regs + CPSW1_CPDMA_OFFSET; - dma_params.txhdp = ss_regs + CPSW1_STATERAM_OFFSET; - ale_params.ale_regs = ss_regs + CPSW1_ALE_OFFSET; - slave_offset = CPSW1_SLAVE_OFFSET; - slave_size = CPSW1_SLAVE_SIZE; - sliver_offset = CPSW1_SLIVER_OFFSET; - dma_params.desc_mem_phys = 0; - break; - case CPSW_VERSION_2: - case CPSW_VERSION_3: - case CPSW_VERSION_4: - cpsw->host_port_regs = ss_regs + CPSW2_HOST_PORT_OFFSET; - cpts_regs = ss_regs + CPSW2_CPTS_OFFSET; - cpsw->hw_stats = ss_regs + CPSW2_HW_STATS; - dma_params.dmaregs = ss_regs + CPSW2_CPDMA_OFFSET; - dma_params.txhdp = ss_regs + CPSW2_STATERAM_OFFSET; - ale_params.ale_regs = ss_regs + CPSW2_ALE_OFFSET; - slave_offset = CPSW2_SLAVE_OFFSET; - slave_size = CPSW2_SLAVE_SIZE; - sliver_offset = CPSW2_SLIVER_OFFSET; - dma_params.desc_mem_phys = - (u32 __force) ss_res->start + CPSW2_BD_OFFSET; - break; - default: - dev_err(priv->dev, "unknown version 0x%08x\n", cpsw->version); - ret = -ENODEV; - goto clean_dt_ret; - } - for (i = 0; i < cpsw->data.slaves; i++) { - struct cpsw_slave *slave = &cpsw->slaves[i]; - - cpsw_slave_init(slave, cpsw, slave_offset, sliver_offset); - slave_offset += slave_size; - sliver_offset += SLIVER_SIZE; - } - - dma_params.dev = &pdev->dev; - dma_params.rxthresh = dma_params.dmaregs + CPDMA_RXTHRESH; - dma_params.rxfree = dma_params.dmaregs + CPDMA_RXFREE; - dma_params.rxhdp = dma_params.txhdp + CPDMA_RXHDP; - dma_params.txcp = dma_params.txhdp + CPDMA_TXCP; - dma_params.rxcp = dma_params.txhdp + CPDMA_RXCP; - - dma_params.num_chan = data->channels; - dma_params.has_soft_reset = true; - dma_params.min_packet_size = CPSW_MIN_PACKET_SIZE; - dma_params.desc_mem_size = data->bd_ram_size; - dma_params.desc_align = 16; - dma_params.has_ext_regs = true; - dma_params.desc_hw_addr = dma_params.desc_mem_phys; - dma_params.bus_freq_mhz = cpsw->bus_freq_mhz; - dma_params.descs_pool_size = descs_pool_size; - - cpsw->dma = cpdma_ctlr_create(&dma_params); - if (!cpsw->dma) { - dev_err(priv->dev, "error initializing dma\n"); - ret = -ENOMEM; + ret = cpsw_init_common(cpsw, ss_regs, ale_ageout, + ss_res->start + CPSW2_BD_OFFSET, + descs_pool_size); + if (ret) goto clean_dt_ret; - } - - soc = soc_device_match(cpsw_soc_devices); - if (soc) - cpsw->quirk_irq = 1; ch = cpsw->quirk_irq ? 0 : 7; cpsw->txv[0].ch = cpdma_chan_create(cpsw->dma, ch, cpsw_tx_handler, 0); if (IS_ERR(cpsw->txv[0].ch)) { - dev_err(priv->dev, "error initializing tx dma channel\n"); + dev_err(dev, "error initializing tx dma channel\n"); ret = PTR_ERR(cpsw->txv[0].ch); - goto clean_dma_ret; + goto clean_cpts; } cpsw->rxv[0].ch = cpdma_chan_create(cpsw->dma, 0, cpsw_rx_handler, 1); if (IS_ERR(cpsw->rxv[0].ch)) { - dev_err(priv->dev, "error initializing rx dma channel\n"); + dev_err(dev, "error initializing rx dma channel\n"); ret = PTR_ERR(cpsw->rxv[0].ch); - goto clean_dma_ret; + goto clean_cpts; } + cpsw_split_res(cpsw); - ale_params.dev = &pdev->dev; - ale_params.ale_ageout = ale_ageout; - ale_params.ale_entries = data->ale_entries; - ale_params.ale_ports = CPSW_ALE_PORTS_NUM; - - cpsw->ale = cpsw_ale_create(&ale_params); - if (!cpsw->ale) { - dev_err(priv->dev, "error initializing ale engine\n"); - ret = -ENODEV; - goto clean_dma_ret; + /* setup netdev */ + ndev = devm_alloc_etherdev_mqs(dev, sizeof(struct cpsw_priv), + CPSW_MAX_QUEUES, CPSW_MAX_QUEUES); + if (!ndev) { + dev_err(dev, "error allocating net_device\n"); + goto clean_cpts; } - cpsw->cpts = cpts_create(cpsw->dev, cpts_regs, cpsw->dev->of_node); - if (IS_ERR(cpsw->cpts)) { - ret = PTR_ERR(cpsw->cpts); - goto clean_dma_ret; - } + platform_set_drvdata(pdev, ndev); + priv = netdev_priv(ndev); + priv->cpsw = cpsw; + priv->ndev = ndev; + priv->dev = dev; + priv->msg_enable = netif_msg_init(debug_level, CPSW_DEBUG); + priv->emac_port = 0; - ndev->irq = platform_get_irq(pdev, 1); - if (ndev->irq < 0) { - dev_err(priv->dev, "error getting irq resource\n"); - ret = ndev->irq; - goto clean_dma_ret; + if (is_valid_ether_addr(data->slave_data[0].mac_addr)) { + memcpy(priv->mac_addr, data->slave_data[0].mac_addr, ETH_ALEN); + dev_info(dev, "Detected MACID = %pM\n", priv->mac_addr); + } else { + eth_random_addr(priv->mac_addr); + dev_info(dev, "Random MACID = %pM\n", priv->mac_addr); } + memcpy(ndev->dev_addr, priv->mac_addr, ETH_ALEN); + + cpsw->slaves[0].ndev = ndev; + ndev->features |= NETIF_F_HW_VLAN_CTAG_FILTER | NETIF_F_HW_VLAN_CTAG_RX; ndev->netdev_ops = &cpsw_netdev_ops; @@ -3705,15 +2504,14 @@ static int cpsw_probe(struct platform_device *pdev) netif_tx_napi_add(ndev, &cpsw->napi_tx, cpsw->quirk_irq ? cpsw_tx_poll : cpsw_tx_mq_poll, CPSW_POLL_WEIGHT); - cpsw_split_res(ndev); /* register the network device */ - SET_NETDEV_DEV(ndev, &pdev->dev); + SET_NETDEV_DEV(ndev, dev); ret = register_netdev(ndev); if (ret) { - dev_err(priv->dev, "error registering net device\n"); + dev_err(dev, "error registering net device\n"); ret = -ENODEV; - goto clean_dma_ret; + goto clean_cpts; } if (cpsw->data.dual_emac) { @@ -3731,40 +2529,24 @@ static int cpsw_probe(struct platform_device *pdev) * If anyone wants to implement support for those, make sure to * first request and append them to irqs_table array. */ - - /* RX IRQ */ - irq = platform_get_irq(pdev, 1); - if (irq < 0) { - ret = irq; - goto clean_dma_ret; - } - - cpsw->irqs_table[0] = irq; - ret = devm_request_irq(&pdev->dev, irq, cpsw_rx_interrupt, - 0, dev_name(&pdev->dev), cpsw); + ret = devm_request_irq(dev, cpsw->irqs_table[0], cpsw_rx_interrupt, + 0, dev_name(dev), cpsw); if (ret < 0) { - dev_err(priv->dev, "error attaching irq (%d)\n", ret); - goto clean_dma_ret; + dev_err(dev, "error attaching irq (%d)\n", ret); + goto clean_unregister_netdev_ret; } - /* TX IRQ */ - irq = platform_get_irq(pdev, 2); - if (irq < 0) { - ret = irq; - goto clean_dma_ret; - } - cpsw->irqs_table[1] = irq; - ret = devm_request_irq(&pdev->dev, irq, cpsw_tx_interrupt, + ret = devm_request_irq(dev, cpsw->irqs_table[1], cpsw_tx_interrupt, 0, dev_name(&pdev->dev), cpsw); if (ret < 0) { - dev_err(priv->dev, "error attaching irq (%d)\n", ret); - goto clean_dma_ret; + dev_err(dev, "error attaching irq (%d)\n", ret); + goto clean_unregister_netdev_ret; } cpsw_notice(priv, probe, "initialized device (regs %pa, irq %d, pool size %d)\n", - &ss_res->start, ndev->irq, dma_params.descs_pool_size); + &ss_res->start, cpsw->irqs_table[0], descs_pool_size); pm_runtime_put(&pdev->dev); @@ -3772,15 +2554,14 @@ static int cpsw_probe(struct platform_device *pdev) clean_unregister_netdev_ret: unregister_netdev(ndev); -clean_dma_ret: +clean_cpts: + cpts_release(cpsw->cpts); cpdma_ctlr_destroy(cpsw->dma); clean_dt_ret: cpsw_remove_dt(pdev); pm_runtime_put_sync(&pdev->dev); clean_runtime_disable_ret: pm_runtime_disable(&pdev->dev); -clean_ndev_ret: - free_netdev(priv->ndev); return ret; } @@ -3805,9 +2586,6 @@ static int cpsw_remove(struct platform_device *pdev) cpsw_remove_dt(pdev); pm_runtime_put_sync(&pdev->dev); pm_runtime_disable(&pdev->dev); - if (cpsw->data.dual_emac) - free_netdev(cpsw->slaves[1].ndev); - free_netdev(ndev); return 0; } diff --git a/drivers/net/ethernet/ti/cpsw.h b/drivers/net/ethernet/ti/cpsw.h index 907e05fc22e4cef43f5a87f9306ed5ae9195f263..35d602f03281c1147bfea84a14f559a9cbd30bd2 100644 --- a/drivers/net/ethernet/ti/cpsw.h +++ b/drivers/net/ethernet/ti/cpsw.h @@ -1,15 +1,8 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* Texas Instruments Ethernet Switch Driver * * Copyright (C) 2013 Texas Instruments * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * version 2 as published by the Free Software Foundation. - * - * This program is distributed "as is" WITHOUT ANY WARRANTY of any - * kind, whether express or implied; without even the implied warranty - * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #ifndef __CPSW_H__ #define __CPSW_H__ diff --git a/drivers/net/ethernet/ti/cpsw_ale.c b/drivers/net/ethernet/ti/cpsw_ale.c index 798c989d5d9342328544d44a389fd11b5b5d88c1..84025dcc78d59aea0dfad2708b49a319ad478d1e 100644 --- a/drivers/net/ethernet/ti/cpsw_ale.c +++ b/drivers/net/ethernet/ti/cpsw_ale.c @@ -1,16 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Texas Instruments N-Port Ethernet Switch Address Lookup Engine * * Copyright (C) 2012 Texas Instruments * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License as - * published by the Free Software Foundation version 2. - * - * This program is distributed "as is" WITHOUT ANY WARRANTY of any - * kind, whether express or implied; without even the implied warranty - * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include #include @@ -287,6 +280,9 @@ int cpsw_ale_flush_multicast(struct cpsw_ale *ale, int port_mask, int vid) if (cpsw_ale_get_mcast(ale_entry)) { u8 addr[6]; + if (cpsw_ale_get_super(ale_entry)) + continue; + cpsw_ale_get_addr(ale_entry, addr); if (!is_broadcast_ether_addr(addr)) cpsw_ale_flush_mcast(ale, ale_entry, port_mask); @@ -296,7 +292,6 @@ int cpsw_ale_flush_multicast(struct cpsw_ale *ale, int port_mask, int vid) } return 0; } -EXPORT_SYMBOL_GPL(cpsw_ale_flush_multicast); static inline void cpsw_ale_set_vlan_entry_type(u32 *ale_entry, int flags, u16 vid) @@ -334,7 +329,6 @@ int cpsw_ale_add_ucast(struct cpsw_ale *ale, const u8 *addr, int port, cpsw_ale_write(ale, idx, ale_entry); return 0; } -EXPORT_SYMBOL_GPL(cpsw_ale_add_ucast); int cpsw_ale_del_ucast(struct cpsw_ale *ale, const u8 *addr, int port, int flags, u16 vid) @@ -350,7 +344,6 @@ int cpsw_ale_del_ucast(struct cpsw_ale *ale, const u8 *addr, int port, cpsw_ale_write(ale, idx, ale_entry); return 0; } -EXPORT_SYMBOL_GPL(cpsw_ale_del_ucast); int cpsw_ale_add_mcast(struct cpsw_ale *ale, const u8 *addr, int port_mask, int flags, u16 vid, int mcast_state) @@ -365,7 +358,7 @@ int cpsw_ale_add_mcast(struct cpsw_ale *ale, const u8 *addr, int port_mask, cpsw_ale_set_vlan_entry_type(ale_entry, flags, vid); cpsw_ale_set_addr(ale_entry, addr); - cpsw_ale_set_super(ale_entry, (flags & ALE_BLOCKED) ? 1 : 0); + cpsw_ale_set_super(ale_entry, (flags & ALE_SUPER) ? 1 : 0); cpsw_ale_set_mcast_state(ale_entry, mcast_state); mask = cpsw_ale_get_port_mask(ale_entry, @@ -384,7 +377,6 @@ int cpsw_ale_add_mcast(struct cpsw_ale *ale, const u8 *addr, int port_mask, cpsw_ale_write(ale, idx, ale_entry); return 0; } -EXPORT_SYMBOL_GPL(cpsw_ale_add_mcast); int cpsw_ale_del_mcast(struct cpsw_ale *ale, const u8 *addr, int port_mask, int flags, u16 vid) @@ -407,7 +399,6 @@ int cpsw_ale_del_mcast(struct cpsw_ale *ale, const u8 *addr, int port_mask, cpsw_ale_write(ale, idx, ale_entry); return 0; } -EXPORT_SYMBOL_GPL(cpsw_ale_del_mcast); /* ALE NetCP NU switch specific vlan functions */ static void cpsw_ale_set_vlan_mcast(struct cpsw_ale *ale, u32 *ale_entry, @@ -458,7 +449,6 @@ int cpsw_ale_add_vlan(struct cpsw_ale *ale, u16 vid, int port, int untag, cpsw_ale_write(ale, idx, ale_entry); return 0; } -EXPORT_SYMBOL_GPL(cpsw_ale_add_vlan); int cpsw_ale_del_vlan(struct cpsw_ale *ale, u16 vid, int port_mask) { @@ -480,40 +470,39 @@ int cpsw_ale_del_vlan(struct cpsw_ale *ale, u16 vid, int port_mask) cpsw_ale_write(ale, idx, ale_entry); return 0; } -EXPORT_SYMBOL_GPL(cpsw_ale_del_vlan); -void cpsw_ale_set_allmulti(struct cpsw_ale *ale, int allmulti) +void cpsw_ale_set_allmulti(struct cpsw_ale *ale, int allmulti, int port) { u32 ale_entry[ALE_ENTRY_WORDS]; - int type, idx; int unreg_mcast = 0; - - /* Only bother doing the work if the setting is actually changing */ - if (ale->allmulti == allmulti) - return; - - /* Remember the new setting to check against next time */ - ale->allmulti = allmulti; + int type, idx; for (idx = 0; idx < ale->params.ale_entries; idx++) { + int vlan_members; + cpsw_ale_read(ale, idx, ale_entry); type = cpsw_ale_get_entry_type(ale_entry); if (type != ALE_TYPE_VLAN) continue; + vlan_members = + cpsw_ale_get_vlan_member_list(ale_entry, + ale->vlan_field_bits); + + if (port != -1 && !(vlan_members & BIT(port))) + continue; unreg_mcast = cpsw_ale_get_vlan_unreg_mcast(ale_entry, ale->vlan_field_bits); if (allmulti) - unreg_mcast |= 1; + unreg_mcast |= ALE_PORT_HOST; else - unreg_mcast &= ~1; + unreg_mcast &= ~ALE_PORT_HOST; cpsw_ale_set_vlan_unreg_mcast(ale_entry, unreg_mcast, ale->vlan_field_bits); cpsw_ale_write(ale, idx, ale_entry); } } -EXPORT_SYMBOL_GPL(cpsw_ale_set_allmulti); struct ale_control_info { const char *name; @@ -739,7 +728,6 @@ int cpsw_ale_control_set(struct cpsw_ale *ale, int port, int control, return 0; } -EXPORT_SYMBOL_GPL(cpsw_ale_control_set); int cpsw_ale_control_get(struct cpsw_ale *ale, int port, int control) { @@ -763,7 +751,6 @@ int cpsw_ale_control_get(struct cpsw_ale *ale, int port, int control) tmp = readl_relaxed(ale->params.ale_regs + offset) >> shift; return tmp & BITMASK(info->bits); } -EXPORT_SYMBOL_GPL(cpsw_ale_control_get); static void cpsw_ale_timer(struct timer_list *t) { @@ -788,14 +775,12 @@ void cpsw_ale_start(struct cpsw_ale *ale) add_timer(&ale->timer); } } -EXPORT_SYMBOL_GPL(cpsw_ale_start); void cpsw_ale_stop(struct cpsw_ale *ale) { del_timer_sync(&ale->timer); cpsw_ale_control_set(ale, 0, ALE_ENABLE, 0); } -EXPORT_SYMBOL_GPL(cpsw_ale_stop); struct cpsw_ale *cpsw_ale_create(struct cpsw_ale_params *params) { @@ -879,7 +864,6 @@ struct cpsw_ale *cpsw_ale_create(struct cpsw_ale_params *params) return ale; } -EXPORT_SYMBOL_GPL(cpsw_ale_create); void cpsw_ale_dump(struct cpsw_ale *ale, u32 *data) { @@ -890,8 +874,3 @@ void cpsw_ale_dump(struct cpsw_ale *ale, u32 *data) data += ALE_ENTRY_WORDS; } } -EXPORT_SYMBOL_GPL(cpsw_ale_dump); - -MODULE_LICENSE("GPL v2"); -MODULE_DESCRIPTION("TI CPSW ALE driver"); -MODULE_AUTHOR("Texas Instruments"); diff --git a/drivers/net/ethernet/ti/cpsw_ale.h b/drivers/net/ethernet/ti/cpsw_ale.h index cd07a3e96d5769797bf6a97c3c0d16e7d9de0b07..370df254eb1280dae11a43abde482a130374f058 100644 --- a/drivers/net/ethernet/ti/cpsw_ale.h +++ b/drivers/net/ethernet/ti/cpsw_ale.h @@ -1,16 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* * Texas Instruments N-Port Ethernet Switch Address Lookup Engine APIs * * Copyright (C) 2012 Texas Instruments * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License as - * published by the Free Software Foundation version 2. - * - * This program is distributed "as is" WITHOUT ANY WARRANTY of any - * kind, whether express or implied; without even the implied warranty - * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #ifndef __TI_CPSW_ALE_H__ #define __TI_CPSW_ALE_H__ @@ -37,7 +30,6 @@ struct cpsw_ale { struct cpsw_ale_params params; struct timer_list timer; unsigned long ageout; - int allmulti; u32 version; /* These bits are different on NetCP NU Switch ALE */ u32 port_mask_bits; @@ -116,7 +108,7 @@ int cpsw_ale_del_mcast(struct cpsw_ale *ale, const u8 *addr, int port_mask, int cpsw_ale_add_vlan(struct cpsw_ale *ale, u16 vid, int port, int untag, int reg_mcast, int unreg_mcast); int cpsw_ale_del_vlan(struct cpsw_ale *ale, u16 vid, int port); -void cpsw_ale_set_allmulti(struct cpsw_ale *ale, int allmulti); +void cpsw_ale_set_allmulti(struct cpsw_ale *ale, int allmulti, int port); int cpsw_ale_control_get(struct cpsw_ale *ale, int port, int control); int cpsw_ale_control_set(struct cpsw_ale *ale, int port, diff --git a/drivers/net/ethernet/ti/cpsw_ethtool.c b/drivers/net/ethernet/ti/cpsw_ethtool.c new file mode 100644 index 0000000000000000000000000000000000000000..a4a7ec0d2531281025497e06578c2554b028b7e4 --- /dev/null +++ b/drivers/net/ethernet/ti/cpsw_ethtool.c @@ -0,0 +1,719 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Texas Instruments Ethernet Switch Driver ethtool intf + * + * Copyright (C) 2019 Texas Instruments + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "cpsw.h" +#include "cpts.h" +#include "cpsw_ale.h" +#include "cpsw_priv.h" +#include "davinci_cpdma.h" + +struct cpsw_hw_stats { + u32 rxgoodframes; + u32 rxbroadcastframes; + u32 rxmulticastframes; + u32 rxpauseframes; + u32 rxcrcerrors; + u32 rxaligncodeerrors; + u32 rxoversizedframes; + u32 rxjabberframes; + u32 rxundersizedframes; + u32 rxfragments; + u32 __pad_0[2]; + u32 rxoctets; + u32 txgoodframes; + u32 txbroadcastframes; + u32 txmulticastframes; + u32 txpauseframes; + u32 txdeferredframes; + u32 txcollisionframes; + u32 txsinglecollframes; + u32 txmultcollframes; + u32 txexcessivecollisions; + u32 txlatecollisions; + u32 txunderrun; + u32 txcarriersenseerrors; + u32 txoctets; + u32 octetframes64; + u32 octetframes65t127; + u32 octetframes128t255; + u32 octetframes256t511; + u32 octetframes512t1023; + u32 octetframes1024tup; + u32 netoctets; + u32 rxsofoverruns; + u32 rxmofoverruns; + u32 rxdmaoverruns; +}; + +struct cpsw_stats { + char stat_string[ETH_GSTRING_LEN]; + int type; + int sizeof_stat; + int stat_offset; +}; + +enum { + CPSW_STATS, + CPDMA_RX_STATS, + CPDMA_TX_STATS, +}; + +#define CPSW_STAT(m) CPSW_STATS, \ + FIELD_SIZEOF(struct cpsw_hw_stats, m), \ + offsetof(struct cpsw_hw_stats, m) +#define CPDMA_RX_STAT(m) CPDMA_RX_STATS, \ + FIELD_SIZEOF(struct cpdma_chan_stats, m), \ + offsetof(struct cpdma_chan_stats, m) +#define CPDMA_TX_STAT(m) CPDMA_TX_STATS, \ + FIELD_SIZEOF(struct cpdma_chan_stats, m), \ + offsetof(struct cpdma_chan_stats, m) + +static const struct cpsw_stats cpsw_gstrings_stats[] = { + { "Good Rx Frames", CPSW_STAT(rxgoodframes) }, + { "Broadcast Rx Frames", CPSW_STAT(rxbroadcastframes) }, + { "Multicast Rx Frames", CPSW_STAT(rxmulticastframes) }, + { "Pause Rx Frames", CPSW_STAT(rxpauseframes) }, + { "Rx CRC Errors", CPSW_STAT(rxcrcerrors) }, + { "Rx Align/Code Errors", CPSW_STAT(rxaligncodeerrors) }, + { "Oversize Rx Frames", CPSW_STAT(rxoversizedframes) }, + { "Rx Jabbers", CPSW_STAT(rxjabberframes) }, + { "Undersize (Short) Rx Frames", CPSW_STAT(rxundersizedframes) }, + { "Rx Fragments", CPSW_STAT(rxfragments) }, + { "Rx Octets", CPSW_STAT(rxoctets) }, + { "Good Tx Frames", CPSW_STAT(txgoodframes) }, + { "Broadcast Tx Frames", CPSW_STAT(txbroadcastframes) }, + { "Multicast Tx Frames", CPSW_STAT(txmulticastframes) }, + { "Pause Tx Frames", CPSW_STAT(txpauseframes) }, + { "Deferred Tx Frames", CPSW_STAT(txdeferredframes) }, + { "Collisions", CPSW_STAT(txcollisionframes) }, + { "Single Collision Tx Frames", CPSW_STAT(txsinglecollframes) }, + { "Multiple Collision Tx Frames", CPSW_STAT(txmultcollframes) }, + { "Excessive Collisions", CPSW_STAT(txexcessivecollisions) }, + { "Late Collisions", CPSW_STAT(txlatecollisions) }, + { "Tx Underrun", CPSW_STAT(txunderrun) }, + { "Carrier Sense Errors", CPSW_STAT(txcarriersenseerrors) }, + { "Tx Octets", CPSW_STAT(txoctets) }, + { "Rx + Tx 64 Octet Frames", CPSW_STAT(octetframes64) }, + { "Rx + Tx 65-127 Octet Frames", CPSW_STAT(octetframes65t127) }, + { "Rx + Tx 128-255 Octet Frames", CPSW_STAT(octetframes128t255) }, + { "Rx + Tx 256-511 Octet Frames", CPSW_STAT(octetframes256t511) }, + { "Rx + Tx 512-1023 Octet Frames", CPSW_STAT(octetframes512t1023) }, + { "Rx + Tx 1024-Up Octet Frames", CPSW_STAT(octetframes1024tup) }, + { "Net Octets", CPSW_STAT(netoctets) }, + { "Rx Start of Frame Overruns", CPSW_STAT(rxsofoverruns) }, + { "Rx Middle of Frame Overruns", CPSW_STAT(rxmofoverruns) }, + { "Rx DMA Overruns", CPSW_STAT(rxdmaoverruns) }, +}; + +static const struct cpsw_stats cpsw_gstrings_ch_stats[] = { + { "head_enqueue", CPDMA_RX_STAT(head_enqueue) }, + { "tail_enqueue", CPDMA_RX_STAT(tail_enqueue) }, + { "pad_enqueue", CPDMA_RX_STAT(pad_enqueue) }, + { "misqueued", CPDMA_RX_STAT(misqueued) }, + { "desc_alloc_fail", CPDMA_RX_STAT(desc_alloc_fail) }, + { "pad_alloc_fail", CPDMA_RX_STAT(pad_alloc_fail) }, + { "runt_receive_buf", CPDMA_RX_STAT(runt_receive_buff) }, + { "runt_transmit_buf", CPDMA_RX_STAT(runt_transmit_buff) }, + { "empty_dequeue", CPDMA_RX_STAT(empty_dequeue) }, + { "busy_dequeue", CPDMA_RX_STAT(busy_dequeue) }, + { "good_dequeue", CPDMA_RX_STAT(good_dequeue) }, + { "requeue", CPDMA_RX_STAT(requeue) }, + { "teardown_dequeue", CPDMA_RX_STAT(teardown_dequeue) }, +}; + +#define CPSW_STATS_COMMON_LEN ARRAY_SIZE(cpsw_gstrings_stats) +#define CPSW_STATS_CH_LEN ARRAY_SIZE(cpsw_gstrings_ch_stats) + +u32 cpsw_get_msglevel(struct net_device *ndev) +{ + struct cpsw_priv *priv = netdev_priv(ndev); + + return priv->msg_enable; +} + +void cpsw_set_msglevel(struct net_device *ndev, u32 value) +{ + struct cpsw_priv *priv = netdev_priv(ndev); + + priv->msg_enable = value; +} + +int cpsw_get_coalesce(struct net_device *ndev, struct ethtool_coalesce *coal) +{ + struct cpsw_common *cpsw = ndev_to_cpsw(ndev); + + coal->rx_coalesce_usecs = cpsw->coal_intvl; + return 0; +} + +int cpsw_set_coalesce(struct net_device *ndev, struct ethtool_coalesce *coal) +{ + struct cpsw_priv *priv = netdev_priv(ndev); + u32 int_ctrl; + u32 num_interrupts = 0; + u32 prescale = 0; + u32 addnl_dvdr = 1; + u32 coal_intvl = 0; + struct cpsw_common *cpsw = priv->cpsw; + + coal_intvl = coal->rx_coalesce_usecs; + + int_ctrl = readl(&cpsw->wr_regs->int_control); + prescale = cpsw->bus_freq_mhz * 4; + + if (!coal->rx_coalesce_usecs) { + int_ctrl &= ~(CPSW_INTPRESCALE_MASK | CPSW_INTPACEEN); + goto update_return; + } + + if (coal_intvl < CPSW_CMINTMIN_INTVL) + coal_intvl = CPSW_CMINTMIN_INTVL; + + if (coal_intvl > CPSW_CMINTMAX_INTVL) { + /* Interrupt pacer works with 4us Pulse, we can + * throttle further by dilating the 4us pulse. + */ + addnl_dvdr = CPSW_INTPRESCALE_MASK / prescale; + + if (addnl_dvdr > 1) { + prescale *= addnl_dvdr; + if (coal_intvl > (CPSW_CMINTMAX_INTVL * addnl_dvdr)) + coal_intvl = (CPSW_CMINTMAX_INTVL + * addnl_dvdr); + } else { + addnl_dvdr = 1; + coal_intvl = CPSW_CMINTMAX_INTVL; + } + } + + num_interrupts = (1000 * addnl_dvdr) / coal_intvl; + writel(num_interrupts, &cpsw->wr_regs->rx_imax); + writel(num_interrupts, &cpsw->wr_regs->tx_imax); + + int_ctrl |= CPSW_INTPACEEN; + int_ctrl &= (~CPSW_INTPRESCALE_MASK); + int_ctrl |= (prescale & CPSW_INTPRESCALE_MASK); + +update_return: + writel(int_ctrl, &cpsw->wr_regs->int_control); + + cpsw_notice(priv, timer, "Set coalesce to %d usecs.\n", coal_intvl); + cpsw->coal_intvl = coal_intvl; + + return 0; +} + +int cpsw_get_sset_count(struct net_device *ndev, int sset) +{ + struct cpsw_common *cpsw = ndev_to_cpsw(ndev); + + switch (sset) { + case ETH_SS_STATS: + return (CPSW_STATS_COMMON_LEN + + (cpsw->rx_ch_num + cpsw->tx_ch_num) * + CPSW_STATS_CH_LEN); + default: + return -EOPNOTSUPP; + } +} + +static void cpsw_add_ch_strings(u8 **p, int ch_num, int rx_dir) +{ + int ch_stats_len; + int line; + int i; + + ch_stats_len = CPSW_STATS_CH_LEN * ch_num; + for (i = 0; i < ch_stats_len; i++) { + line = i % CPSW_STATS_CH_LEN; + snprintf(*p, ETH_GSTRING_LEN, + "%s DMA chan %ld: %s", rx_dir ? "Rx" : "Tx", + (long)(i / CPSW_STATS_CH_LEN), + cpsw_gstrings_ch_stats[line].stat_string); + *p += ETH_GSTRING_LEN; + } +} + +void cpsw_get_strings(struct net_device *ndev, u32 stringset, u8 *data) +{ + struct cpsw_common *cpsw = ndev_to_cpsw(ndev); + u8 *p = data; + int i; + + switch (stringset) { + case ETH_SS_STATS: + for (i = 0; i < CPSW_STATS_COMMON_LEN; i++) { + memcpy(p, cpsw_gstrings_stats[i].stat_string, + ETH_GSTRING_LEN); + p += ETH_GSTRING_LEN; + } + + cpsw_add_ch_strings(&p, cpsw->rx_ch_num, 1); + cpsw_add_ch_strings(&p, cpsw->tx_ch_num, 0); + break; + } +} + +void cpsw_get_ethtool_stats(struct net_device *ndev, + struct ethtool_stats *stats, u64 *data) +{ + u8 *p; + struct cpsw_common *cpsw = ndev_to_cpsw(ndev); + struct cpdma_chan_stats ch_stats; + int i, l, ch; + + /* Collect Davinci CPDMA stats for Rx and Tx Channel */ + for (l = 0; l < CPSW_STATS_COMMON_LEN; l++) + data[l] = readl(cpsw->hw_stats + + cpsw_gstrings_stats[l].stat_offset); + + for (ch = 0; ch < cpsw->rx_ch_num; ch++) { + cpdma_chan_get_stats(cpsw->rxv[ch].ch, &ch_stats); + for (i = 0; i < CPSW_STATS_CH_LEN; i++, l++) { + p = (u8 *)&ch_stats + + cpsw_gstrings_ch_stats[i].stat_offset; + data[l] = *(u32 *)p; + } + } + + for (ch = 0; ch < cpsw->tx_ch_num; ch++) { + cpdma_chan_get_stats(cpsw->txv[ch].ch, &ch_stats); + for (i = 0; i < CPSW_STATS_CH_LEN; i++, l++) { + p = (u8 *)&ch_stats + + cpsw_gstrings_ch_stats[i].stat_offset; + data[l] = *(u32 *)p; + } + } +} + +void cpsw_get_pauseparam(struct net_device *ndev, + struct ethtool_pauseparam *pause) +{ + struct cpsw_priv *priv = netdev_priv(ndev); + + pause->autoneg = AUTONEG_DISABLE; + pause->rx_pause = priv->rx_pause ? true : false; + pause->tx_pause = priv->tx_pause ? true : false; +} + +void cpsw_get_wol(struct net_device *ndev, struct ethtool_wolinfo *wol) +{ + struct cpsw_priv *priv = netdev_priv(ndev); + struct cpsw_common *cpsw = priv->cpsw; + int slave_no = cpsw_slave_index(cpsw, priv); + + wol->supported = 0; + wol->wolopts = 0; + + if (cpsw->slaves[slave_no].phy) + phy_ethtool_get_wol(cpsw->slaves[slave_no].phy, wol); +} + +int cpsw_set_wol(struct net_device *ndev, struct ethtool_wolinfo *wol) +{ + struct cpsw_priv *priv = netdev_priv(ndev); + struct cpsw_common *cpsw = priv->cpsw; + int slave_no = cpsw_slave_index(cpsw, priv); + + if (cpsw->slaves[slave_no].phy) + return phy_ethtool_set_wol(cpsw->slaves[slave_no].phy, wol); + else + return -EOPNOTSUPP; +} + +int cpsw_get_regs_len(struct net_device *ndev) +{ + struct cpsw_common *cpsw = ndev_to_cpsw(ndev); + + return cpsw->data.ale_entries * ALE_ENTRY_WORDS * sizeof(u32); +} + +void cpsw_get_regs(struct net_device *ndev, struct ethtool_regs *regs, void *p) +{ + u32 *reg = p; + struct cpsw_common *cpsw = ndev_to_cpsw(ndev); + + /* update CPSW IP version */ + regs->version = cpsw->version; + + cpsw_ale_dump(cpsw->ale, reg); +} + +int cpsw_ethtool_op_begin(struct net_device *ndev) +{ + struct cpsw_priv *priv = netdev_priv(ndev); + struct cpsw_common *cpsw = priv->cpsw; + int ret; + + ret = pm_runtime_get_sync(cpsw->dev); + if (ret < 0) { + cpsw_err(priv, drv, "ethtool begin failed %d\n", ret); + pm_runtime_put_noidle(cpsw->dev); + } + + return ret; +} + +void cpsw_ethtool_op_complete(struct net_device *ndev) +{ + struct cpsw_priv *priv = netdev_priv(ndev); + int ret; + + ret = pm_runtime_put(priv->cpsw->dev); + if (ret < 0) + cpsw_err(priv, drv, "ethtool complete failed %d\n", ret); +} + +void cpsw_get_channels(struct net_device *ndev, struct ethtool_channels *ch) +{ + struct cpsw_common *cpsw = ndev_to_cpsw(ndev); + + ch->max_rx = cpsw->quirk_irq ? 1 : CPSW_MAX_QUEUES; + ch->max_tx = cpsw->quirk_irq ? 1 : CPSW_MAX_QUEUES; + ch->max_combined = 0; + ch->max_other = 0; + ch->other_count = 0; + ch->rx_count = cpsw->rx_ch_num; + ch->tx_count = cpsw->tx_ch_num; + ch->combined_count = 0; +} + +int cpsw_get_link_ksettings(struct net_device *ndev, + struct ethtool_link_ksettings *ecmd) +{ + struct cpsw_priv *priv = netdev_priv(ndev); + struct cpsw_common *cpsw = priv->cpsw; + int slave_no = cpsw_slave_index(cpsw, priv); + + if (!cpsw->slaves[slave_no].phy) + return -EOPNOTSUPP; + + phy_ethtool_ksettings_get(cpsw->slaves[slave_no].phy, ecmd); + return 0; +} + +int cpsw_set_link_ksettings(struct net_device *ndev, + const struct ethtool_link_ksettings *ecmd) +{ + struct cpsw_priv *priv = netdev_priv(ndev); + struct cpsw_common *cpsw = priv->cpsw; + int slave_no = cpsw_slave_index(cpsw, priv); + + if (!cpsw->slaves[slave_no].phy) + return -EOPNOTSUPP; + + return phy_ethtool_ksettings_set(cpsw->slaves[slave_no].phy, ecmd); +} + +int cpsw_get_eee(struct net_device *ndev, struct ethtool_eee *edata) +{ + struct cpsw_priv *priv = netdev_priv(ndev); + struct cpsw_common *cpsw = priv->cpsw; + int slave_no = cpsw_slave_index(cpsw, priv); + + if (cpsw->slaves[slave_no].phy) + return phy_ethtool_get_eee(cpsw->slaves[slave_no].phy, edata); + else + return -EOPNOTSUPP; +} + +int cpsw_set_eee(struct net_device *ndev, struct ethtool_eee *edata) +{ + struct cpsw_priv *priv = netdev_priv(ndev); + struct cpsw_common *cpsw = priv->cpsw; + int slave_no = cpsw_slave_index(cpsw, priv); + + if (cpsw->slaves[slave_no].phy) + return phy_ethtool_set_eee(cpsw->slaves[slave_no].phy, edata); + else + return -EOPNOTSUPP; +} + +int cpsw_nway_reset(struct net_device *ndev) +{ + struct cpsw_priv *priv = netdev_priv(ndev); + struct cpsw_common *cpsw = priv->cpsw; + int slave_no = cpsw_slave_index(cpsw, priv); + + if (cpsw->slaves[slave_no].phy) + return genphy_restart_aneg(cpsw->slaves[slave_no].phy); + else + return -EOPNOTSUPP; +} + +static void cpsw_suspend_data_pass(struct net_device *ndev) +{ + struct cpsw_common *cpsw = ndev_to_cpsw(ndev); + struct cpsw_slave *slave; + int i; + + /* Disable NAPI scheduling */ + cpsw_intr_disable(cpsw); + + /* Stop all transmit queues for every network device. + * Disable re-using rx descriptors with dormant_on. + */ + for (i = cpsw->data.slaves, slave = cpsw->slaves; i; i--, slave++) { + if (!(slave->ndev && netif_running(slave->ndev))) + continue; + + netif_tx_stop_all_queues(slave->ndev); + netif_dormant_on(slave->ndev); + } + + /* Handle rest of tx packets and stop cpdma channels */ + cpdma_ctlr_stop(cpsw->dma); +} + +static int cpsw_resume_data_pass(struct net_device *ndev) +{ + struct cpsw_priv *priv = netdev_priv(ndev); + struct cpsw_common *cpsw = priv->cpsw; + struct cpsw_slave *slave; + int i, ret; + + /* Allow rx packets handling */ + for (i = cpsw->data.slaves, slave = cpsw->slaves; i; i--, slave++) + if (slave->ndev && netif_running(slave->ndev)) + netif_dormant_off(slave->ndev); + + /* After this receive is started */ + if (cpsw->usage_count) { + ret = cpsw_fill_rx_channels(priv); + if (ret) + return ret; + + cpdma_ctlr_start(cpsw->dma); + cpsw_intr_enable(cpsw); + } + + /* Resume transmit for every affected interface */ + for (i = cpsw->data.slaves, slave = cpsw->slaves; i; i--, slave++) + if (slave->ndev && netif_running(slave->ndev)) + netif_tx_start_all_queues(slave->ndev); + + return 0; +} + +static int cpsw_check_ch_settings(struct cpsw_common *cpsw, + struct ethtool_channels *ch) +{ + if (cpsw->quirk_irq) { + dev_err(cpsw->dev, "Maximum one tx/rx queue is allowed"); + return -EOPNOTSUPP; + } + + if (ch->combined_count) + return -EINVAL; + + /* verify we have at least one channel in each direction */ + if (!ch->rx_count || !ch->tx_count) + return -EINVAL; + + if (ch->rx_count > cpsw->data.channels || + ch->tx_count > cpsw->data.channels) + return -EINVAL; + + return 0; +} + +static int cpsw_update_channels_res(struct cpsw_priv *priv, int ch_num, int rx, + cpdma_handler_fn rx_handler) +{ + struct cpsw_common *cpsw = priv->cpsw; + void (*handler)(void *, int, int); + struct netdev_queue *queue; + struct cpsw_vector *vec; + int ret, *ch, vch; + + if (rx) { + ch = &cpsw->rx_ch_num; + vec = cpsw->rxv; + handler = rx_handler; + } else { + ch = &cpsw->tx_ch_num; + vec = cpsw->txv; + handler = cpsw_tx_handler; + } + + while (*ch < ch_num) { + vch = rx ? *ch : 7 - *ch; + vec[*ch].ch = cpdma_chan_create(cpsw->dma, vch, handler, rx); + queue = netdev_get_tx_queue(priv->ndev, *ch); + queue->tx_maxrate = 0; + + if (IS_ERR(vec[*ch].ch)) + return PTR_ERR(vec[*ch].ch); + + if (!vec[*ch].ch) + return -EINVAL; + + cpsw_info(priv, ifup, "created new %d %s channel\n", *ch, + (rx ? "rx" : "tx")); + (*ch)++; + } + + while (*ch > ch_num) { + (*ch)--; + + ret = cpdma_chan_destroy(vec[*ch].ch); + if (ret) + return ret; + + cpsw_info(priv, ifup, "destroyed %d %s channel\n", *ch, + (rx ? "rx" : "tx")); + } + + return 0; +} + +int cpsw_set_channels_common(struct net_device *ndev, + struct ethtool_channels *chs, + cpdma_handler_fn rx_handler) +{ + struct cpsw_priv *priv = netdev_priv(ndev); + struct cpsw_common *cpsw = priv->cpsw; + struct cpsw_slave *slave; + int i, ret; + + ret = cpsw_check_ch_settings(cpsw, chs); + if (ret < 0) + return ret; + + cpsw_suspend_data_pass(ndev); + + ret = cpsw_update_channels_res(priv, chs->rx_count, 1, rx_handler); + if (ret) + goto err; + + ret = cpsw_update_channels_res(priv, chs->tx_count, 0, rx_handler); + if (ret) + goto err; + + for (i = cpsw->data.slaves, slave = cpsw->slaves; i; i--, slave++) { + if (!(slave->ndev && netif_running(slave->ndev))) + continue; + + /* Inform stack about new count of queues */ + ret = netif_set_real_num_tx_queues(slave->ndev, + cpsw->tx_ch_num); + if (ret) { + dev_err(priv->dev, "cannot set real number of tx queues\n"); + goto err; + } + + ret = netif_set_real_num_rx_queues(slave->ndev, + cpsw->rx_ch_num); + if (ret) { + dev_err(priv->dev, "cannot set real number of rx queues\n"); + goto err; + } + } + + if (cpsw->usage_count) + cpsw_split_res(cpsw); + + ret = cpsw_resume_data_pass(ndev); + if (!ret) + return 0; +err: + dev_err(priv->dev, "cannot update channels number, closing device\n"); + dev_close(ndev); + return ret; +} + +void cpsw_get_ringparam(struct net_device *ndev, + struct ethtool_ringparam *ering) +{ + struct cpsw_priv *priv = netdev_priv(ndev); + struct cpsw_common *cpsw = priv->cpsw; + + /* not supported */ + ering->tx_max_pending = 0; + ering->tx_pending = cpdma_get_num_tx_descs(cpsw->dma); + ering->rx_max_pending = cpsw->descs_pool_size - CPSW_MAX_QUEUES; + ering->rx_pending = cpdma_get_num_rx_descs(cpsw->dma); +} + +int cpsw_set_ringparam(struct net_device *ndev, + struct ethtool_ringparam *ering) +{ + struct cpsw_priv *priv = netdev_priv(ndev); + struct cpsw_common *cpsw = priv->cpsw; + int ret; + + /* ignore ering->tx_pending - only rx_pending adjustment is supported */ + + if (ering->rx_mini_pending || ering->rx_jumbo_pending || + ering->rx_pending < CPSW_MAX_QUEUES || + ering->rx_pending > (cpsw->descs_pool_size - CPSW_MAX_QUEUES)) + return -EINVAL; + + if (ering->rx_pending == cpdma_get_num_rx_descs(cpsw->dma)) + return 0; + + cpsw_suspend_data_pass(ndev); + + cpdma_set_num_rx_descs(cpsw->dma, ering->rx_pending); + + if (cpsw->usage_count) + cpdma_chan_split_pool(cpsw->dma); + + ret = cpsw_resume_data_pass(ndev); + if (!ret) + return 0; + + dev_err(cpsw->dev, "cannot set ring params, closing device\n"); + dev_close(ndev); + return ret; +} + +#if IS_ENABLED(CONFIG_TI_CPTS) +int cpsw_get_ts_info(struct net_device *ndev, struct ethtool_ts_info *info) +{ + struct cpsw_common *cpsw = ndev_to_cpsw(ndev); + + info->so_timestamping = + SOF_TIMESTAMPING_TX_HARDWARE | + SOF_TIMESTAMPING_TX_SOFTWARE | + SOF_TIMESTAMPING_RX_HARDWARE | + SOF_TIMESTAMPING_RX_SOFTWARE | + SOF_TIMESTAMPING_SOFTWARE | + SOF_TIMESTAMPING_RAW_HARDWARE; + info->phc_index = cpsw->cpts->phc_index; + info->tx_types = + (1 << HWTSTAMP_TX_OFF) | + (1 << HWTSTAMP_TX_ON); + info->rx_filters = + (1 << HWTSTAMP_FILTER_NONE) | + (1 << HWTSTAMP_FILTER_PTP_V1_L4_EVENT) | + (1 << HWTSTAMP_FILTER_PTP_V2_EVENT); + return 0; +} +#else +int cpsw_get_ts_info(struct net_device *ndev, struct ethtool_ts_info *info) +{ + info->so_timestamping = + SOF_TIMESTAMPING_TX_SOFTWARE | + SOF_TIMESTAMPING_RX_SOFTWARE | + SOF_TIMESTAMPING_SOFTWARE; + info->phc_index = -1; + info->tx_types = 0; + info->rx_filters = 0; + return 0; +} +#endif diff --git a/drivers/net/ethernet/ti/cpsw_priv.c b/drivers/net/ethernet/ti/cpsw_priv.c new file mode 100644 index 0000000000000000000000000000000000000000..476d050a022c7d5b6006a248c25ee741eded7483 --- /dev/null +++ b/drivers/net/ethernet/ti/cpsw_priv.c @@ -0,0 +1,132 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Texas Instruments Ethernet Switch Driver + * + * Copyright (C) 2019 Texas Instruments + */ + +#include +#include +#include +#include +#include +#include +#include + +#include "cpts.h" +#include "cpsw_ale.h" +#include "cpsw_priv.h" +#include "cpsw_sl.h" +#include "davinci_cpdma.h" + +int cpsw_init_common(struct cpsw_common *cpsw, void __iomem *ss_regs, + int ale_ageout, phys_addr_t desc_mem_phys, + int descs_pool_size) +{ + u32 slave_offset, sliver_offset, slave_size; + struct cpsw_ale_params ale_params; + struct cpsw_platform_data *data; + struct cpdma_params dma_params; + struct device *dev = cpsw->dev; + void __iomem *cpts_regs; + int ret = 0, i; + + data = &cpsw->data; + cpsw->rx_ch_num = 1; + cpsw->tx_ch_num = 1; + + cpsw->version = readl(&cpsw->regs->id_ver); + + memset(&dma_params, 0, sizeof(dma_params)); + memset(&ale_params, 0, sizeof(ale_params)); + + switch (cpsw->version) { + case CPSW_VERSION_1: + cpsw->host_port_regs = ss_regs + CPSW1_HOST_PORT_OFFSET; + cpts_regs = ss_regs + CPSW1_CPTS_OFFSET; + cpsw->hw_stats = ss_regs + CPSW1_HW_STATS; + dma_params.dmaregs = ss_regs + CPSW1_CPDMA_OFFSET; + dma_params.txhdp = ss_regs + CPSW1_STATERAM_OFFSET; + ale_params.ale_regs = ss_regs + CPSW1_ALE_OFFSET; + slave_offset = CPSW1_SLAVE_OFFSET; + slave_size = CPSW1_SLAVE_SIZE; + sliver_offset = CPSW1_SLIVER_OFFSET; + dma_params.desc_mem_phys = 0; + break; + case CPSW_VERSION_2: + case CPSW_VERSION_3: + case CPSW_VERSION_4: + cpsw->host_port_regs = ss_regs + CPSW2_HOST_PORT_OFFSET; + cpts_regs = ss_regs + CPSW2_CPTS_OFFSET; + cpsw->hw_stats = ss_regs + CPSW2_HW_STATS; + dma_params.dmaregs = ss_regs + CPSW2_CPDMA_OFFSET; + dma_params.txhdp = ss_regs + CPSW2_STATERAM_OFFSET; + ale_params.ale_regs = ss_regs + CPSW2_ALE_OFFSET; + slave_offset = CPSW2_SLAVE_OFFSET; + slave_size = CPSW2_SLAVE_SIZE; + sliver_offset = CPSW2_SLIVER_OFFSET; + dma_params.desc_mem_phys = desc_mem_phys; + break; + default: + dev_err(dev, "unknown version 0x%08x\n", cpsw->version); + return -ENODEV; + } + + for (i = 0; i < cpsw->data.slaves; i++) { + struct cpsw_slave *slave = &cpsw->slaves[i]; + void __iomem *regs = cpsw->regs; + + slave->slave_num = i; + slave->data = &cpsw->data.slave_data[i]; + slave->regs = regs + slave_offset; + slave->port_vlan = slave->data->dual_emac_res_vlan; + slave->mac_sl = cpsw_sl_get("cpsw", dev, regs + sliver_offset); + if (IS_ERR(slave->mac_sl)) + return PTR_ERR(slave->mac_sl); + + slave_offset += slave_size; + sliver_offset += SLIVER_SIZE; + } + + ale_params.dev = dev; + ale_params.ale_ageout = ale_ageout; + ale_params.ale_entries = data->ale_entries; + ale_params.ale_ports = CPSW_ALE_PORTS_NUM; + + cpsw->ale = cpsw_ale_create(&ale_params); + if (!cpsw->ale) { + dev_err(dev, "error initializing ale engine\n"); + return -ENODEV; + } + + dma_params.dev = dev; + dma_params.rxthresh = dma_params.dmaregs + CPDMA_RXTHRESH; + dma_params.rxfree = dma_params.dmaregs + CPDMA_RXFREE; + dma_params.rxhdp = dma_params.txhdp + CPDMA_RXHDP; + dma_params.txcp = dma_params.txhdp + CPDMA_TXCP; + dma_params.rxcp = dma_params.txhdp + CPDMA_RXCP; + + dma_params.num_chan = data->channels; + dma_params.has_soft_reset = true; + dma_params.min_packet_size = CPSW_MIN_PACKET_SIZE; + dma_params.desc_mem_size = data->bd_ram_size; + dma_params.desc_align = 16; + dma_params.has_ext_regs = true; + dma_params.desc_hw_addr = dma_params.desc_mem_phys; + dma_params.bus_freq_mhz = cpsw->bus_freq_mhz; + dma_params.descs_pool_size = descs_pool_size; + + cpsw->dma = cpdma_ctlr_create(&dma_params); + if (!cpsw->dma) { + dev_err(dev, "error initializing dma\n"); + return -ENOMEM; + } + + cpsw->cpts = cpts_create(cpsw->dev, cpts_regs, cpsw->dev->of_node); + if (IS_ERR(cpsw->cpts)) { + ret = PTR_ERR(cpsw->cpts); + cpdma_ctlr_destroy(cpsw->dma); + } + + return ret; +} diff --git a/drivers/net/ethernet/ti/cpsw_priv.h b/drivers/net/ethernet/ti/cpsw_priv.h new file mode 100644 index 0000000000000000000000000000000000000000..04795b97ee715e9af292fec1dfeb899eca69b1a5 --- /dev/null +++ b/drivers/net/ethernet/ti/cpsw_priv.h @@ -0,0 +1,429 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Texas Instruments Ethernet Switch Driver + */ + +#ifndef DRIVERS_NET_ETHERNET_TI_CPSW_PRIV_H_ +#define DRIVERS_NET_ETHERNET_TI_CPSW_PRIV_H_ + +#include "davinci_cpdma.h" + +#define CPSW_DEBUG (NETIF_MSG_HW | NETIF_MSG_WOL | \ + NETIF_MSG_DRV | NETIF_MSG_LINK | \ + NETIF_MSG_IFUP | NETIF_MSG_INTR | \ + NETIF_MSG_PROBE | NETIF_MSG_TIMER | \ + NETIF_MSG_IFDOWN | NETIF_MSG_RX_ERR | \ + NETIF_MSG_TX_ERR | NETIF_MSG_TX_DONE | \ + NETIF_MSG_PKTDATA | NETIF_MSG_TX_QUEUED | \ + NETIF_MSG_RX_STATUS) + +#define cpsw_info(priv, type, format, ...) \ +do { \ + if (netif_msg_##type(priv) && net_ratelimit()) \ + dev_info(priv->dev, format, ## __VA_ARGS__); \ +} while (0) + +#define cpsw_err(priv, type, format, ...) \ +do { \ + if (netif_msg_##type(priv) && net_ratelimit()) \ + dev_err(priv->dev, format, ## __VA_ARGS__); \ +} while (0) + +#define cpsw_dbg(priv, type, format, ...) \ +do { \ + if (netif_msg_##type(priv) && net_ratelimit()) \ + dev_dbg(priv->dev, format, ## __VA_ARGS__); \ +} while (0) + +#define cpsw_notice(priv, type, format, ...) \ +do { \ + if (netif_msg_##type(priv) && net_ratelimit()) \ + dev_notice(priv->dev, format, ## __VA_ARGS__); \ +} while (0) + +#define ALE_ALL_PORTS 0x7 + +#define CPSW_MAJOR_VERSION(reg) (reg >> 8 & 0x7) +#define CPSW_MINOR_VERSION(reg) (reg & 0xff) +#define CPSW_RTL_VERSION(reg) ((reg >> 11) & 0x1f) + +#define CPSW_VERSION_1 0x19010a +#define CPSW_VERSION_2 0x19010c +#define CPSW_VERSION_3 0x19010f +#define CPSW_VERSION_4 0x190112 + +#define HOST_PORT_NUM 0 +#define CPSW_ALE_PORTS_NUM 3 +#define SLIVER_SIZE 0x40 + +#define CPSW1_HOST_PORT_OFFSET 0x028 +#define CPSW1_SLAVE_OFFSET 0x050 +#define CPSW1_SLAVE_SIZE 0x040 +#define CPSW1_CPDMA_OFFSET 0x100 +#define CPSW1_STATERAM_OFFSET 0x200 +#define CPSW1_HW_STATS 0x400 +#define CPSW1_CPTS_OFFSET 0x500 +#define CPSW1_ALE_OFFSET 0x600 +#define CPSW1_SLIVER_OFFSET 0x700 + +#define CPSW2_HOST_PORT_OFFSET 0x108 +#define CPSW2_SLAVE_OFFSET 0x200 +#define CPSW2_SLAVE_SIZE 0x100 +#define CPSW2_CPDMA_OFFSET 0x800 +#define CPSW2_HW_STATS 0x900 +#define CPSW2_STATERAM_OFFSET 0xa00 +#define CPSW2_CPTS_OFFSET 0xc00 +#define CPSW2_ALE_OFFSET 0xd00 +#define CPSW2_SLIVER_OFFSET 0xd80 +#define CPSW2_BD_OFFSET 0x2000 + +#define CPDMA_RXTHRESH 0x0c0 +#define CPDMA_RXFREE 0x0e0 +#define CPDMA_TXHDP 0x00 +#define CPDMA_RXHDP 0x20 +#define CPDMA_TXCP 0x40 +#define CPDMA_RXCP 0x60 + +#define CPSW_POLL_WEIGHT 64 +#define CPSW_RX_VLAN_ENCAP_HDR_SIZE 4 +#define CPSW_MIN_PACKET_SIZE (VLAN_ETH_ZLEN) +#define CPSW_MAX_PACKET_SIZE (VLAN_ETH_FRAME_LEN +\ + ETH_FCS_LEN +\ + CPSW_RX_VLAN_ENCAP_HDR_SIZE) + +#define RX_PRIORITY_MAPPING 0x76543210 +#define TX_PRIORITY_MAPPING 0x33221100 +#define CPDMA_TX_PRIORITY_MAP 0x76543210 + +#define CPSW_VLAN_AWARE BIT(1) +#define CPSW_RX_VLAN_ENCAP BIT(2) +#define CPSW_ALE_VLAN_AWARE 1 + +#define CPSW_FIFO_NORMAL_MODE (0 << 16) +#define CPSW_FIFO_DUAL_MAC_MODE (1 << 16) +#define CPSW_FIFO_RATE_LIMIT_MODE (2 << 16) + +#define CPSW_INTPACEEN (0x3f << 16) +#define CPSW_INTPRESCALE_MASK (0x7FF << 0) +#define CPSW_CMINTMAX_CNT 63 +#define CPSW_CMINTMIN_CNT 2 +#define CPSW_CMINTMAX_INTVL (1000 / CPSW_CMINTMIN_CNT) +#define CPSW_CMINTMIN_INTVL ((1000 / CPSW_CMINTMAX_CNT) + 1) + +#define IRQ_NUM 2 +#define CPSW_MAX_QUEUES 8 +#define CPSW_CPDMA_DESCS_POOL_SIZE_DEFAULT 256 +#define CPSW_FIFO_QUEUE_TYPE_SHIFT 16 +#define CPSW_FIFO_SHAPE_EN_SHIFT 16 +#define CPSW_FIFO_RATE_EN_SHIFT 20 +#define CPSW_TC_NUM 4 +#define CPSW_FIFO_SHAPERS_NUM (CPSW_TC_NUM - 1) +#define CPSW_PCT_MASK 0x7f + +#define CPSW_RX_VLAN_ENCAP_HDR_PRIO_SHIFT 29 +#define CPSW_RX_VLAN_ENCAP_HDR_PRIO_MSK GENMASK(2, 0) +#define CPSW_RX_VLAN_ENCAP_HDR_VID_SHIFT 16 +#define CPSW_RX_VLAN_ENCAP_HDR_PKT_TYPE_SHIFT 8 +#define CPSW_RX_VLAN_ENCAP_HDR_PKT_TYPE_MSK GENMASK(1, 0) +enum { + CPSW_RX_VLAN_ENCAP_HDR_PKT_VLAN_TAG = 0, + CPSW_RX_VLAN_ENCAP_HDR_PKT_RESERV, + CPSW_RX_VLAN_ENCAP_HDR_PKT_PRIO_TAG, + CPSW_RX_VLAN_ENCAP_HDR_PKT_UNTAG, +}; + +struct cpsw_wr_regs { + u32 id_ver; + u32 soft_reset; + u32 control; + u32 int_control; + u32 rx_thresh_en; + u32 rx_en; + u32 tx_en; + u32 misc_en; + u32 mem_allign1[8]; + u32 rx_thresh_stat; + u32 rx_stat; + u32 tx_stat; + u32 misc_stat; + u32 mem_allign2[8]; + u32 rx_imax; + u32 tx_imax; + +}; + +struct cpsw_ss_regs { + u32 id_ver; + u32 control; + u32 soft_reset; + u32 stat_port_en; + u32 ptype; + u32 soft_idle; + u32 thru_rate; + u32 gap_thresh; + u32 tx_start_wds; + u32 flow_control; + u32 vlan_ltype; + u32 ts_ltype; + u32 dlr_ltype; +}; + +/* CPSW_PORT_V1 */ +#define CPSW1_MAX_BLKS 0x00 /* Maximum FIFO Blocks */ +#define CPSW1_BLK_CNT 0x04 /* FIFO Block Usage Count (Read Only) */ +#define CPSW1_TX_IN_CTL 0x08 /* Transmit FIFO Control */ +#define CPSW1_PORT_VLAN 0x0c /* VLAN Register */ +#define CPSW1_TX_PRI_MAP 0x10 /* Tx Header Priority to Switch Pri Mapping */ +#define CPSW1_TS_CTL 0x14 /* Time Sync Control */ +#define CPSW1_TS_SEQ_LTYPE 0x18 /* Time Sync Sequence ID Offset and Msg Type */ +#define CPSW1_TS_VLAN 0x1c /* Time Sync VLAN1 and VLAN2 */ + +/* CPSW_PORT_V2 */ +#define CPSW2_CONTROL 0x00 /* Control Register */ +#define CPSW2_MAX_BLKS 0x08 /* Maximum FIFO Blocks */ +#define CPSW2_BLK_CNT 0x0c /* FIFO Block Usage Count (Read Only) */ +#define CPSW2_TX_IN_CTL 0x10 /* Transmit FIFO Control */ +#define CPSW2_PORT_VLAN 0x14 /* VLAN Register */ +#define CPSW2_TX_PRI_MAP 0x18 /* Tx Header Priority to Switch Pri Mapping */ +#define CPSW2_TS_SEQ_MTYPE 0x1c /* Time Sync Sequence ID Offset and Msg Type */ + +/* CPSW_PORT_V1 and V2 */ +#define SA_LO 0x20 /* CPGMAC_SL Source Address Low */ +#define SA_HI 0x24 /* CPGMAC_SL Source Address High */ +#define SEND_PERCENT 0x28 /* Transmit Queue Send Percentages */ + +/* CPSW_PORT_V2 only */ +#define RX_DSCP_PRI_MAP0 0x30 /* Rx DSCP Priority to Rx Packet Mapping */ +#define RX_DSCP_PRI_MAP1 0x34 /* Rx DSCP Priority to Rx Packet Mapping */ +#define RX_DSCP_PRI_MAP2 0x38 /* Rx DSCP Priority to Rx Packet Mapping */ +#define RX_DSCP_PRI_MAP3 0x3c /* Rx DSCP Priority to Rx Packet Mapping */ +#define RX_DSCP_PRI_MAP4 0x40 /* Rx DSCP Priority to Rx Packet Mapping */ +#define RX_DSCP_PRI_MAP5 0x44 /* Rx DSCP Priority to Rx Packet Mapping */ +#define RX_DSCP_PRI_MAP6 0x48 /* Rx DSCP Priority to Rx Packet Mapping */ +#define RX_DSCP_PRI_MAP7 0x4c /* Rx DSCP Priority to Rx Packet Mapping */ + +/* Bit definitions for the CPSW2_CONTROL register */ +#define PASS_PRI_TAGGED BIT(24) /* Pass Priority Tagged */ +#define VLAN_LTYPE2_EN BIT(21) /* VLAN LTYPE 2 enable */ +#define VLAN_LTYPE1_EN BIT(20) /* VLAN LTYPE 1 enable */ +#define DSCP_PRI_EN BIT(16) /* DSCP Priority Enable */ +#define TS_107 BIT(15) /* Tyme Sync Dest IP Address 107 */ +#define TS_320 BIT(14) /* Time Sync Dest Port 320 enable */ +#define TS_319 BIT(13) /* Time Sync Dest Port 319 enable */ +#define TS_132 BIT(12) /* Time Sync Dest IP Addr 132 enable */ +#define TS_131 BIT(11) /* Time Sync Dest IP Addr 131 enable */ +#define TS_130 BIT(10) /* Time Sync Dest IP Addr 130 enable */ +#define TS_129 BIT(9) /* Time Sync Dest IP Addr 129 enable */ +#define TS_TTL_NONZERO BIT(8) /* Time Sync Time To Live Non-zero enable */ +#define TS_ANNEX_F_EN BIT(6) /* Time Sync Annex F enable */ +#define TS_ANNEX_D_EN BIT(4) /* Time Sync Annex D enable */ +#define TS_LTYPE2_EN BIT(3) /* Time Sync LTYPE 2 enable */ +#define TS_LTYPE1_EN BIT(2) /* Time Sync LTYPE 1 enable */ +#define TS_TX_EN BIT(1) /* Time Sync Transmit Enable */ +#define TS_RX_EN BIT(0) /* Time Sync Receive Enable */ + +#define CTRL_V2_TS_BITS \ + (TS_320 | TS_319 | TS_132 | TS_131 | TS_130 | TS_129 |\ + TS_TTL_NONZERO | TS_ANNEX_D_EN | TS_LTYPE1_EN | VLAN_LTYPE1_EN) + +#define CTRL_V2_ALL_TS_MASK (CTRL_V2_TS_BITS | TS_TX_EN | TS_RX_EN) +#define CTRL_V2_TX_TS_BITS (CTRL_V2_TS_BITS | TS_TX_EN) +#define CTRL_V2_RX_TS_BITS (CTRL_V2_TS_BITS | TS_RX_EN) + + +#define CTRL_V3_TS_BITS \ + (TS_107 | TS_320 | TS_319 | TS_132 | TS_131 | TS_130 | TS_129 |\ + TS_TTL_NONZERO | TS_ANNEX_F_EN | TS_ANNEX_D_EN |\ + TS_LTYPE1_EN | VLAN_LTYPE1_EN) + +#define CTRL_V3_ALL_TS_MASK (CTRL_V3_TS_BITS | TS_TX_EN | TS_RX_EN) +#define CTRL_V3_TX_TS_BITS (CTRL_V3_TS_BITS | TS_TX_EN) +#define CTRL_V3_RX_TS_BITS (CTRL_V3_TS_BITS | TS_RX_EN) + +/* Bit definitions for the CPSW2_TS_SEQ_MTYPE register */ +#define TS_SEQ_ID_OFFSET_SHIFT (16) /* Time Sync Sequence ID Offset */ +#define TS_SEQ_ID_OFFSET_MASK (0x3f) +#define TS_MSG_TYPE_EN_SHIFT (0) /* Time Sync Message Type Enable */ +#define TS_MSG_TYPE_EN_MASK (0xffff) + +/* The PTP event messages - Sync, Delay_Req, Pdelay_Req, and Pdelay_Resp. */ +#define EVENT_MSG_BITS ((1<<0) | (1<<1) | (1<<2) | (1<<3)) + +/* Bit definitions for the CPSW1_TS_CTL register */ +#define CPSW_V1_TS_RX_EN BIT(0) +#define CPSW_V1_TS_TX_EN BIT(4) +#define CPSW_V1_MSG_TYPE_OFS 16 + +/* Bit definitions for the CPSW1_TS_SEQ_LTYPE register */ +#define CPSW_V1_SEQ_ID_OFS_SHIFT 16 + +#define CPSW_MAX_BLKS_TX 15 +#define CPSW_MAX_BLKS_TX_SHIFT 4 +#define CPSW_MAX_BLKS_RX 5 + +struct cpsw_host_regs { + u32 max_blks; + u32 blk_cnt; + u32 tx_in_ctl; + u32 port_vlan; + u32 tx_pri_map; + u32 cpdma_tx_pri_map; + u32 cpdma_rx_chan_map; +}; + +struct cpsw_slave_data { + struct device_node *phy_node; + char phy_id[MII_BUS_ID_SIZE]; + int phy_if; + u8 mac_addr[ETH_ALEN]; + u16 dual_emac_res_vlan; /* Reserved VLAN for DualEMAC */ + struct phy *ifphy; +}; + +struct cpsw_platform_data { + struct cpsw_slave_data *slave_data; + u32 ss_reg_ofs; /* Subsystem control register offset */ + u32 channels; /* number of cpdma channels (symmetric) */ + u32 slaves; /* number of slave cpgmac ports */ + u32 active_slave; /* time stamping, ethtool and SIOCGMIIPHY slave */ + u32 ale_entries; /* ale table size */ + u32 bd_ram_size; /*buffer descriptor ram size */ + u32 mac_control; /* Mac control register */ + u16 default_vlan; /* Def VLAN for ALE lookup in VLAN aware mode*/ + bool dual_emac; /* Enable Dual EMAC mode */ +}; + +struct cpsw_slave { + void __iomem *regs; + int slave_num; + u32 mac_control; + struct cpsw_slave_data *data; + struct phy_device *phy; + struct net_device *ndev; + u32 port_vlan; + struct cpsw_sl *mac_sl; +}; + +static inline u32 slave_read(struct cpsw_slave *slave, u32 offset) +{ + return readl_relaxed(slave->regs + offset); +} + +static inline void slave_write(struct cpsw_slave *slave, u32 val, u32 offset) +{ + writel_relaxed(val, slave->regs + offset); +} + +struct cpsw_vector { + struct cpdma_chan *ch; + int budget; +}; + +struct cpsw_common { + struct device *dev; + struct cpsw_platform_data data; + struct napi_struct napi_rx; + struct napi_struct napi_tx; + struct cpsw_ss_regs __iomem *regs; + struct cpsw_wr_regs __iomem *wr_regs; + u8 __iomem *hw_stats; + struct cpsw_host_regs __iomem *host_port_regs; + u32 version; + u32 coal_intvl; + u32 bus_freq_mhz; + int rx_packet_max; + int descs_pool_size; + struct cpsw_slave *slaves; + struct cpdma_ctlr *dma; + struct cpsw_vector txv[CPSW_MAX_QUEUES]; + struct cpsw_vector rxv[CPSW_MAX_QUEUES]; + struct cpsw_ale *ale; + bool quirk_irq; + bool rx_irq_disabled; + bool tx_irq_disabled; + u32 irqs_table[IRQ_NUM]; + struct cpts *cpts; + int rx_ch_num, tx_ch_num; + int speed; + int usage_count; +}; + +struct cpsw_priv { + struct net_device *ndev; + struct device *dev; + u32 msg_enable; + u8 mac_addr[ETH_ALEN]; + bool rx_pause; + bool tx_pause; + bool mqprio_hw; + int fifo_bw[CPSW_TC_NUM]; + int shp_cfg_speed; + int tx_ts_enabled; + int rx_ts_enabled; + u32 emac_port; + struct cpsw_common *cpsw; +}; + +#define ndev_to_cpsw(ndev) (((struct cpsw_priv *)netdev_priv(ndev))->cpsw) +#define napi_to_cpsw(napi) container_of(napi, struct cpsw_common, napi) + +#define cpsw_slave_index(cpsw, priv) \ + ((cpsw->data.dual_emac) ? priv->emac_port : \ + cpsw->data.active_slave) + +static inline int cpsw_get_slave_port(u32 slave_num) +{ + return slave_num + 1; +} + +struct addr_sync_ctx { + struct net_device *ndev; + const u8 *addr; /* address to be synched */ + int consumed; /* number of address instances */ + int flush; /* flush flag */ +}; + +int cpsw_init_common(struct cpsw_common *cpsw, void __iomem *ss_regs, + int ale_ageout, phys_addr_t desc_mem_phys, + int descs_pool_size); +void cpsw_split_res(struct cpsw_common *cpsw); +int cpsw_fill_rx_channels(struct cpsw_priv *priv); +void cpsw_intr_enable(struct cpsw_common *cpsw); +void cpsw_intr_disable(struct cpsw_common *cpsw); +void cpsw_tx_handler(void *token, int len, int status); + +/* ethtool */ +u32 cpsw_get_msglevel(struct net_device *ndev); +void cpsw_set_msglevel(struct net_device *ndev, u32 value); +int cpsw_get_coalesce(struct net_device *ndev, struct ethtool_coalesce *coal); +int cpsw_set_coalesce(struct net_device *ndev, struct ethtool_coalesce *coal); +int cpsw_get_sset_count(struct net_device *ndev, int sset); +void cpsw_get_strings(struct net_device *ndev, u32 stringset, u8 *data); +void cpsw_get_ethtool_stats(struct net_device *ndev, + struct ethtool_stats *stats, u64 *data); +void cpsw_get_pauseparam(struct net_device *ndev, + struct ethtool_pauseparam *pause); +void cpsw_get_wol(struct net_device *ndev, struct ethtool_wolinfo *wol); +int cpsw_set_wol(struct net_device *ndev, struct ethtool_wolinfo *wol); +int cpsw_get_regs_len(struct net_device *ndev); +void cpsw_get_regs(struct net_device *ndev, struct ethtool_regs *regs, void *p); +int cpsw_ethtool_op_begin(struct net_device *ndev); +void cpsw_ethtool_op_complete(struct net_device *ndev); +void cpsw_get_channels(struct net_device *ndev, struct ethtool_channels *ch); +int cpsw_get_link_ksettings(struct net_device *ndev, + struct ethtool_link_ksettings *ecmd); +int cpsw_set_link_ksettings(struct net_device *ndev, + const struct ethtool_link_ksettings *ecmd); +int cpsw_get_eee(struct net_device *ndev, struct ethtool_eee *edata); +int cpsw_set_eee(struct net_device *ndev, struct ethtool_eee *edata); +int cpsw_nway_reset(struct net_device *ndev); +void cpsw_get_ringparam(struct net_device *ndev, + struct ethtool_ringparam *ering); +int cpsw_set_ringparam(struct net_device *ndev, + struct ethtool_ringparam *ering); +int cpsw_set_channels_common(struct net_device *ndev, + struct ethtool_channels *chs, + cpdma_handler_fn rx_handler); +int cpsw_get_ts_info(struct net_device *ndev, struct ethtool_ts_info *info); + +#endif /* DRIVERS_NET_ETHERNET_TI_CPSW_PRIV_H_ */ diff --git a/drivers/net/ethernet/ti/cpsw_sl.c b/drivers/net/ethernet/ti/cpsw_sl.c new file mode 100644 index 0000000000000000000000000000000000000000..0c7531cb0f3982a851153fb6eacb8806310c3c07 --- /dev/null +++ b/drivers/net/ethernet/ti/cpsw_sl.c @@ -0,0 +1,328 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Texas Instruments Ethernet Switch media-access-controller (MAC) submodule/ + * Ethernet MAC Sliver (CPGMAC_SL) + * + * Copyright (C) 2019 Texas Instruments + * + */ + +#include +#include +#include + +#include "cpsw_sl.h" + +#define CPSW_SL_REG_NOTUSED U16_MAX + +static const u16 cpsw_sl_reg_map_cpsw[] = { + [CPSW_SL_IDVER] = 0x00, + [CPSW_SL_MACCONTROL] = 0x04, + [CPSW_SL_MACSTATUS] = 0x08, + [CPSW_SL_SOFT_RESET] = 0x0c, + [CPSW_SL_RX_MAXLEN] = 0x10, + [CPSW_SL_BOFFTEST] = 0x14, + [CPSW_SL_RX_PAUSE] = 0x18, + [CPSW_SL_TX_PAUSE] = 0x1c, + [CPSW_SL_EMCONTROL] = 0x20, + [CPSW_SL_RX_PRI_MAP] = 0x24, + [CPSW_SL_TX_GAP] = 0x28, +}; + +static const u16 cpsw_sl_reg_map_66ak2hk[] = { + [CPSW_SL_IDVER] = 0x00, + [CPSW_SL_MACCONTROL] = 0x04, + [CPSW_SL_MACSTATUS] = 0x08, + [CPSW_SL_SOFT_RESET] = 0x0c, + [CPSW_SL_RX_MAXLEN] = 0x10, + [CPSW_SL_BOFFTEST] = CPSW_SL_REG_NOTUSED, + [CPSW_SL_RX_PAUSE] = 0x18, + [CPSW_SL_TX_PAUSE] = 0x1c, + [CPSW_SL_EMCONTROL] = 0x20, + [CPSW_SL_RX_PRI_MAP] = 0x24, + [CPSW_SL_TX_GAP] = CPSW_SL_REG_NOTUSED, +}; + +static const u16 cpsw_sl_reg_map_66ak2x_xgbe[] = { + [CPSW_SL_IDVER] = 0x00, + [CPSW_SL_MACCONTROL] = 0x04, + [CPSW_SL_MACSTATUS] = 0x08, + [CPSW_SL_SOFT_RESET] = 0x0c, + [CPSW_SL_RX_MAXLEN] = 0x10, + [CPSW_SL_BOFFTEST] = CPSW_SL_REG_NOTUSED, + [CPSW_SL_RX_PAUSE] = 0x18, + [CPSW_SL_TX_PAUSE] = 0x1c, + [CPSW_SL_EMCONTROL] = 0x20, + [CPSW_SL_RX_PRI_MAP] = CPSW_SL_REG_NOTUSED, + [CPSW_SL_TX_GAP] = 0x28, +}; + +static const u16 cpsw_sl_reg_map_66ak2elg_am65[] = { + [CPSW_SL_IDVER] = CPSW_SL_REG_NOTUSED, + [CPSW_SL_MACCONTROL] = 0x00, + [CPSW_SL_MACSTATUS] = 0x04, + [CPSW_SL_SOFT_RESET] = 0x08, + [CPSW_SL_RX_MAXLEN] = CPSW_SL_REG_NOTUSED, + [CPSW_SL_BOFFTEST] = 0x0c, + [CPSW_SL_RX_PAUSE] = 0x10, + [CPSW_SL_TX_PAUSE] = 0x40, + [CPSW_SL_EMCONTROL] = 0x70, + [CPSW_SL_RX_PRI_MAP] = CPSW_SL_REG_NOTUSED, + [CPSW_SL_TX_GAP] = 0x74, +}; + +#define CPSW_SL_SOFT_RESET_BIT BIT(0) + +#define CPSW_SL_STATUS_PN_IDLE BIT(31) +#define CPSW_SL_AM65_STATUS_PN_E_IDLE BIT(30) +#define CPSW_SL_AM65_STATUS_PN_P_IDLE BIT(29) +#define CPSW_SL_AM65_STATUS_PN_TX_IDLE BIT(28) + +#define CPSW_SL_STATUS_IDLE_MASK_BASE (CPSW_SL_STATUS_PN_IDLE) + +#define CPSW_SL_STATUS_IDLE_MASK_K3 \ + (CPSW_SL_STATUS_IDLE_MASK_BASE | CPSW_SL_AM65_STATUS_PN_E_IDLE | \ + CPSW_SL_AM65_STATUS_PN_P_IDLE | CPSW_SL_AM65_STATUS_PN_TX_IDLE) + +#define CPSW_SL_CTL_FUNC_BASE \ + (CPSW_SL_CTL_FULLDUPLEX |\ + CPSW_SL_CTL_LOOPBACK |\ + CPSW_SL_CTL_RX_FLOW_EN |\ + CPSW_SL_CTL_TX_FLOW_EN |\ + CPSW_SL_CTL_GMII_EN |\ + CPSW_SL_CTL_TX_PACE |\ + CPSW_SL_CTL_GIG |\ + CPSW_SL_CTL_CMD_IDLE |\ + CPSW_SL_CTL_IFCTL_A |\ + CPSW_SL_CTL_IFCTL_B |\ + CPSW_SL_CTL_GIG_FORCE |\ + CPSW_SL_CTL_EXT_EN |\ + CPSW_SL_CTL_RX_CEF_EN |\ + CPSW_SL_CTL_RX_CSF_EN |\ + CPSW_SL_CTL_RX_CMF_EN) + +struct cpsw_sl { + struct device *dev; + void __iomem *sl_base; + const u16 *regs; + u32 control_features; + u32 idle_mask; +}; + +struct cpsw_sl_dev_id { + const char *device_id; + const u16 *regs; + const u32 control_features; + const u32 regs_offset; + const u32 idle_mask; +}; + +static const struct cpsw_sl_dev_id cpsw_sl_id_match[] = { + { + .device_id = "cpsw", + .regs = cpsw_sl_reg_map_cpsw, + .control_features = CPSW_SL_CTL_FUNC_BASE | + CPSW_SL_CTL_MTEST | + CPSW_SL_CTL_TX_SHORT_GAP_EN | + CPSW_SL_CTL_TX_SG_LIM_EN, + .idle_mask = CPSW_SL_STATUS_IDLE_MASK_BASE, + }, + { + .device_id = "66ak2hk", + .regs = cpsw_sl_reg_map_66ak2hk, + .control_features = CPSW_SL_CTL_FUNC_BASE | + CPSW_SL_CTL_TX_SHORT_GAP_EN, + .idle_mask = CPSW_SL_STATUS_IDLE_MASK_BASE, + }, + { + .device_id = "66ak2x_xgbe", + .regs = cpsw_sl_reg_map_66ak2x_xgbe, + .control_features = CPSW_SL_CTL_FUNC_BASE | + CPSW_SL_CTL_XGIG | + CPSW_SL_CTL_TX_SHORT_GAP_EN | + CPSW_SL_CTL_CRC_TYPE | + CPSW_SL_CTL_XGMII_EN, + .idle_mask = CPSW_SL_STATUS_IDLE_MASK_BASE, + }, + { + .device_id = "66ak2el", + .regs = cpsw_sl_reg_map_66ak2elg_am65, + .regs_offset = 0x330, + .control_features = CPSW_SL_CTL_FUNC_BASE | + CPSW_SL_CTL_MTEST | + CPSW_SL_CTL_TX_SHORT_GAP_EN | + CPSW_SL_CTL_CRC_TYPE | + CPSW_SL_CTL_EXT_EN_RX_FLO | + CPSW_SL_CTL_EXT_EN_TX_FLO | + CPSW_SL_CTL_TX_SG_LIM_EN, + .idle_mask = CPSW_SL_STATUS_IDLE_MASK_BASE, + }, + { + .device_id = "66ak2g", + .regs = cpsw_sl_reg_map_66ak2elg_am65, + .regs_offset = 0x330, + .control_features = CPSW_SL_CTL_FUNC_BASE | + CPSW_SL_CTL_MTEST | + CPSW_SL_CTL_CRC_TYPE | + CPSW_SL_CTL_EXT_EN_RX_FLO | + CPSW_SL_CTL_EXT_EN_TX_FLO, + }, + { + .device_id = "am65", + .regs = cpsw_sl_reg_map_66ak2elg_am65, + .regs_offset = 0x330, + .control_features = CPSW_SL_CTL_FUNC_BASE | + CPSW_SL_CTL_MTEST | + CPSW_SL_CTL_XGIG | + CPSW_SL_CTL_TX_SHORT_GAP_EN | + CPSW_SL_CTL_CRC_TYPE | + CPSW_SL_CTL_XGMII_EN | + CPSW_SL_CTL_EXT_EN_RX_FLO | + CPSW_SL_CTL_EXT_EN_TX_FLO | + CPSW_SL_CTL_TX_SG_LIM_EN | + CPSW_SL_CTL_EXT_EN_XGIG, + .idle_mask = CPSW_SL_STATUS_IDLE_MASK_K3, + }, + { }, +}; + +u32 cpsw_sl_reg_read(struct cpsw_sl *sl, enum cpsw_sl_regs reg) +{ + int val; + + if (sl->regs[reg] == CPSW_SL_REG_NOTUSED) { + dev_err(sl->dev, "cpsw_sl: not sup r reg: %04X\n", + sl->regs[reg]); + return 0; + } + + val = readl(sl->sl_base + sl->regs[reg]); + dev_dbg(sl->dev, "cpsw_sl: reg: %04X r 0x%08X\n", sl->regs[reg], val); + return val; +} + +void cpsw_sl_reg_write(struct cpsw_sl *sl, enum cpsw_sl_regs reg, u32 val) +{ + if (sl->regs[reg] == CPSW_SL_REG_NOTUSED) { + dev_err(sl->dev, "cpsw_sl: not sup w reg: %04X\n", + sl->regs[reg]); + return; + } + + dev_dbg(sl->dev, "cpsw_sl: reg: %04X w 0x%08X\n", sl->regs[reg], val); + writel(val, sl->sl_base + sl->regs[reg]); +} + +static const struct cpsw_sl_dev_id *cpsw_sl_match_id( + const struct cpsw_sl_dev_id *id, + const char *device_id) +{ + if (!id || !device_id) + return NULL; + + while (id->device_id) { + if (strcmp(device_id, id->device_id) == 0) + return id; + id++; + } + return NULL; +} + +struct cpsw_sl *cpsw_sl_get(const char *device_id, struct device *dev, + void __iomem *sl_base) +{ + const struct cpsw_sl_dev_id *sl_dev_id; + struct cpsw_sl *sl; + + sl = devm_kzalloc(dev, sizeof(struct cpsw_sl), GFP_KERNEL); + if (!sl) + return ERR_PTR(-ENOMEM); + sl->dev = dev; + sl->sl_base = sl_base; + + sl_dev_id = cpsw_sl_match_id(cpsw_sl_id_match, device_id); + if (!sl_dev_id) { + dev_err(sl->dev, "cpsw_sl: dev_id %s not found.\n", device_id); + return ERR_PTR(-EINVAL); + } + sl->regs = sl_dev_id->regs; + sl->control_features = sl_dev_id->control_features; + sl->idle_mask = sl_dev_id->idle_mask; + sl->sl_base += sl_dev_id->regs_offset; + + return sl; +} + +void cpsw_sl_reset(struct cpsw_sl *sl, unsigned long tmo) +{ + unsigned long timeout = jiffies + msecs_to_jiffies(tmo); + + /* Set the soft reset bit */ + cpsw_sl_reg_write(sl, CPSW_SL_SOFT_RESET, CPSW_SL_SOFT_RESET_BIT); + + /* Wait for the bit to clear */ + do { + usleep_range(100, 200); + } while ((cpsw_sl_reg_read(sl, CPSW_SL_SOFT_RESET) & + CPSW_SL_SOFT_RESET_BIT) && + time_after(timeout, jiffies)); + + if (cpsw_sl_reg_read(sl, CPSW_SL_SOFT_RESET) & CPSW_SL_SOFT_RESET_BIT) + dev_err(sl->dev, "cpsw_sl failed to soft-reset.\n"); +} + +u32 cpsw_sl_ctl_set(struct cpsw_sl *sl, u32 ctl_funcs) +{ + u32 val; + + if (ctl_funcs & ~sl->control_features) { + dev_err(sl->dev, "cpsw_sl: unsupported func 0x%08X\n", + ctl_funcs & (~sl->control_features)); + return -EINVAL; + } + + val = cpsw_sl_reg_read(sl, CPSW_SL_MACCONTROL); + val |= ctl_funcs; + cpsw_sl_reg_write(sl, CPSW_SL_MACCONTROL, val); + + return 0; +} + +u32 cpsw_sl_ctl_clr(struct cpsw_sl *sl, u32 ctl_funcs) +{ + u32 val; + + if (ctl_funcs & ~sl->control_features) { + dev_err(sl->dev, "cpsw_sl: unsupported func 0x%08X\n", + ctl_funcs & (~sl->control_features)); + return -EINVAL; + } + + val = cpsw_sl_reg_read(sl, CPSW_SL_MACCONTROL); + val &= ~ctl_funcs; + cpsw_sl_reg_write(sl, CPSW_SL_MACCONTROL, val); + + return 0; +} + +void cpsw_sl_ctl_reset(struct cpsw_sl *sl) +{ + cpsw_sl_reg_write(sl, CPSW_SL_MACCONTROL, 0); +} + +int cpsw_sl_wait_for_idle(struct cpsw_sl *sl, unsigned long tmo) +{ + unsigned long timeout = jiffies + msecs_to_jiffies(tmo); + + do { + usleep_range(100, 200); + } while (!(cpsw_sl_reg_read(sl, CPSW_SL_MACSTATUS) & + sl->idle_mask) && time_after(timeout, jiffies)); + + if (!(cpsw_sl_reg_read(sl, CPSW_SL_MACSTATUS) & sl->idle_mask)) { + dev_err(sl->dev, "cpsw_sl failed to soft-reset.\n"); + return -ETIMEDOUT; + } + + return 0; +} diff --git a/drivers/net/ethernet/ti/cpsw_sl.h b/drivers/net/ethernet/ti/cpsw_sl.h new file mode 100644 index 0000000000000000000000000000000000000000..a6d06a5a420fd3e05a87137aa86018d5efd990ee --- /dev/null +++ b/drivers/net/ethernet/ti/cpsw_sl.h @@ -0,0 +1,73 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Texas Instruments Ethernet Switch media-access-controller (MAC) submodule/ + * Ethernet MAC Sliver (CPGMAC_SL) APIs + * + * Copyright (C) 2019 Texas Instruments + * + */ + +#ifndef __TI_CPSW_SL_H__ +#define __TI_CPSW_SL_H__ + +#include + +enum cpsw_sl_regs { + CPSW_SL_IDVER, + CPSW_SL_MACCONTROL, + CPSW_SL_MACSTATUS, + CPSW_SL_SOFT_RESET, + CPSW_SL_RX_MAXLEN, + CPSW_SL_BOFFTEST, + CPSW_SL_RX_PAUSE, + CPSW_SL_TX_PAUSE, + CPSW_SL_EMCONTROL, + CPSW_SL_RX_PRI_MAP, + CPSW_SL_TX_GAP, +}; + +enum { + CPSW_SL_CTL_FULLDUPLEX = BIT(0), /* Full Duplex mode */ + CPSW_SL_CTL_LOOPBACK = BIT(1), /* Loop Back Mode */ + CPSW_SL_CTL_MTEST = BIT(2), /* Manufacturing Test mode */ + CPSW_SL_CTL_RX_FLOW_EN = BIT(3), /* Receive Flow Control Enable */ + CPSW_SL_CTL_TX_FLOW_EN = BIT(4), /* Transmit Flow Control Enable */ + CPSW_SL_CTL_GMII_EN = BIT(5), /* GMII Enable */ + CPSW_SL_CTL_TX_PACE = BIT(6), /* Transmit Pacing Enable */ + CPSW_SL_CTL_GIG = BIT(7), /* Gigabit Mode */ + CPSW_SL_CTL_XGIG = BIT(8), /* 10 Gigabit Mode */ + CPSW_SL_CTL_TX_SHORT_GAP_EN = BIT(10), /* Transmit Short Gap Enable */ + CPSW_SL_CTL_CMD_IDLE = BIT(11), /* Command Idle */ + CPSW_SL_CTL_CRC_TYPE = BIT(12), /* Port CRC Type */ + CPSW_SL_CTL_XGMII_EN = BIT(13), /* XGMII Enable */ + CPSW_SL_CTL_IFCTL_A = BIT(15), /* Interface Control A */ + CPSW_SL_CTL_IFCTL_B = BIT(16), /* Interface Control B */ + CPSW_SL_CTL_GIG_FORCE = BIT(17), /* Gigabit Mode Force */ + CPSW_SL_CTL_EXT_EN = BIT(18), /* External Control Enable */ + CPSW_SL_CTL_EXT_EN_RX_FLO = BIT(19), /* Ext RX Flow Control Enable */ + CPSW_SL_CTL_EXT_EN_TX_FLO = BIT(20), /* Ext TX Flow Control Enable */ + CPSW_SL_CTL_TX_SG_LIM_EN = BIT(21), /* TXt Short Gap Limit Enable */ + CPSW_SL_CTL_RX_CEF_EN = BIT(22), /* RX Copy Error Frames Enable */ + CPSW_SL_CTL_RX_CSF_EN = BIT(23), /* RX Copy Short Frames Enable */ + CPSW_SL_CTL_RX_CMF_EN = BIT(24), /* RX Copy MAC Control Frames Enable */ + CPSW_SL_CTL_EXT_EN_XGIG = BIT(25), /* Ext XGIG Control En, k3 only */ + + CPSW_SL_CTL_FUNCS_COUNT +}; + +struct cpsw_sl; + +struct cpsw_sl *cpsw_sl_get(const char *device_id, struct device *dev, + void __iomem *sl_base); + +void cpsw_sl_reset(struct cpsw_sl *sl, unsigned long tmo); + +u32 cpsw_sl_ctl_set(struct cpsw_sl *sl, u32 ctl_funcs); +u32 cpsw_sl_ctl_clr(struct cpsw_sl *sl, u32 ctl_funcs); +void cpsw_sl_ctl_reset(struct cpsw_sl *sl); +int cpsw_sl_wait_for_idle(struct cpsw_sl *sl, unsigned long tmo); + +u32 cpsw_sl_reg_read(struct cpsw_sl *sl, enum cpsw_sl_regs reg); +void cpsw_sl_reg_write(struct cpsw_sl *sl, enum cpsw_sl_regs reg, u32 val); + +#endif /* __TI_CPSW_SL_H__ */ diff --git a/drivers/net/ethernet/ti/cpts.c b/drivers/net/ethernet/ti/cpts.c index 2a9ba4acd7fabc06fbc001d7b40e02f4d5963caa..e257018ada71cd0920c80120d31b9eede0a199e6 100644 --- a/drivers/net/ethernet/ti/cpts.c +++ b/drivers/net/ethernet/ti/cpts.c @@ -1,21 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0+ /* * TI Common Platform Time Sync * * Copyright (C) 2012 Richard Cochran * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, write to the Free Software - * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ #include #include diff --git a/drivers/net/ethernet/ti/cpts.h b/drivers/net/ethernet/ti/cpts.h index d2c7decd59b6b3d3c8322f062f46d99a356c17da..024aab6af12fc187d8854377014de904f3d72ad4 100644 --- a/drivers/net/ethernet/ti/cpts.h +++ b/drivers/net/ethernet/ti/cpts.h @@ -1,21 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0+ */ /* * TI Common Platform Time Sync * * Copyright (C) 2012 Richard Cochran * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, write to the Free Software - * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ #ifndef _TI_CPTS_H_ #define _TI_CPTS_H_ diff --git a/drivers/net/ethernet/ti/davinci_cpdma.c b/drivers/net/ethernet/ti/davinci_cpdma.c index 4236dcdd56348512d6d2393ea27c19c0bd9078bf..35bf14d8e7af322ee2fc3bc02cff3065b5b1c067 100644 --- a/drivers/net/ethernet/ti/davinci_cpdma.c +++ b/drivers/net/ethernet/ti/davinci_cpdma.c @@ -1,16 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Texas Instruments CPDMA Driver * * Copyright (C) 2010 Texas Instruments * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License as - * published by the Free Software Foundation version 2. - * - * This program is distributed "as is" WITHOUT ANY WARRANTY of any - * kind, whether express or implied; without even the implied warranty - * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include #include @@ -527,7 +520,6 @@ struct cpdma_ctlr *cpdma_ctlr_create(struct cpdma_params *params) ctlr->num_chan = CPDMA_MAX_CHANNELS; return ctlr; } -EXPORT_SYMBOL_GPL(cpdma_ctlr_create); int cpdma_ctlr_start(struct cpdma_ctlr *ctlr) { @@ -588,7 +580,6 @@ int cpdma_ctlr_start(struct cpdma_ctlr *ctlr) spin_unlock_irqrestore(&ctlr->lock, flags); return 0; } -EXPORT_SYMBOL_GPL(cpdma_ctlr_start); int cpdma_ctlr_stop(struct cpdma_ctlr *ctlr) { @@ -621,7 +612,6 @@ int cpdma_ctlr_stop(struct cpdma_ctlr *ctlr) spin_unlock_irqrestore(&ctlr->lock, flags); return 0; } -EXPORT_SYMBOL_GPL(cpdma_ctlr_stop); int cpdma_ctlr_destroy(struct cpdma_ctlr *ctlr) { @@ -639,7 +629,6 @@ int cpdma_ctlr_destroy(struct cpdma_ctlr *ctlr) cpdma_desc_pool_destroy(ctlr); return ret; } -EXPORT_SYMBOL_GPL(cpdma_ctlr_destroy); int cpdma_ctlr_int_ctrl(struct cpdma_ctlr *ctlr, bool enable) { @@ -660,25 +649,21 @@ int cpdma_ctlr_int_ctrl(struct cpdma_ctlr *ctlr, bool enable) spin_unlock_irqrestore(&ctlr->lock, flags); return 0; } -EXPORT_SYMBOL_GPL(cpdma_ctlr_int_ctrl); void cpdma_ctlr_eoi(struct cpdma_ctlr *ctlr, u32 value) { dma_reg_write(ctlr, CPDMA_MACEOIVECTOR, value); } -EXPORT_SYMBOL_GPL(cpdma_ctlr_eoi); u32 cpdma_ctrl_rxchs_state(struct cpdma_ctlr *ctlr) { return dma_reg_read(ctlr, CPDMA_RXINTSTATMASKED); } -EXPORT_SYMBOL_GPL(cpdma_ctrl_rxchs_state); u32 cpdma_ctrl_txchs_state(struct cpdma_ctlr *ctlr) { return dma_reg_read(ctlr, CPDMA_TXINTSTATMASKED); } -EXPORT_SYMBOL_GPL(cpdma_ctrl_txchs_state); static void cpdma_chan_set_descs(struct cpdma_ctlr *ctlr, int rx, int desc_num, @@ -774,7 +759,6 @@ int cpdma_chan_split_pool(struct cpdma_ctlr *ctlr) return 0; } -EXPORT_SYMBOL_GPL(cpdma_chan_split_pool); /* cpdma_chan_set_weight - set weight of a channel in percentage. @@ -807,7 +791,6 @@ int cpdma_chan_set_weight(struct cpdma_chan *ch, int weight) spin_unlock_irqrestore(&ctlr->lock, flags); return ret; } -EXPORT_SYMBOL_GPL(cpdma_chan_set_weight); /* cpdma_chan_get_min_rate - get minimum allowed rate for channel * Should be called before cpdma_chan_set_rate. @@ -822,7 +805,6 @@ u32 cpdma_chan_get_min_rate(struct cpdma_ctlr *ctlr) return DIV_ROUND_UP(divident, divisor); } -EXPORT_SYMBOL_GPL(cpdma_chan_get_min_rate); /* cpdma_chan_set_rate - limits bandwidth for transmit channel. * The bandwidth * limited channels have to be in order beginning from lowest. @@ -867,7 +849,6 @@ err: spin_unlock_irqrestore(&ctlr->lock, flags); return ret; } -EXPORT_SYMBOL_GPL(cpdma_chan_set_rate); u32 cpdma_chan_get_rate(struct cpdma_chan *ch) { @@ -880,7 +861,6 @@ u32 cpdma_chan_get_rate(struct cpdma_chan *ch) return rate; } -EXPORT_SYMBOL_GPL(cpdma_chan_get_rate); struct cpdma_chan *cpdma_chan_create(struct cpdma_ctlr *ctlr, int chan_num, cpdma_handler_fn handler, int rx_type) @@ -940,7 +920,6 @@ struct cpdma_chan *cpdma_chan_create(struct cpdma_ctlr *ctlr, int chan_num, spin_unlock_irqrestore(&ctlr->lock, flags); return chan; } -EXPORT_SYMBOL_GPL(cpdma_chan_create); int cpdma_chan_get_rx_buf_num(struct cpdma_chan *chan) { @@ -953,7 +932,6 @@ int cpdma_chan_get_rx_buf_num(struct cpdma_chan *chan) return desc_num; } -EXPORT_SYMBOL_GPL(cpdma_chan_get_rx_buf_num); int cpdma_chan_destroy(struct cpdma_chan *chan) { @@ -975,7 +953,6 @@ int cpdma_chan_destroy(struct cpdma_chan *chan) spin_unlock_irqrestore(&ctlr->lock, flags); return 0; } -EXPORT_SYMBOL_GPL(cpdma_chan_destroy); int cpdma_chan_get_stats(struct cpdma_chan *chan, struct cpdma_chan_stats *stats) @@ -988,7 +965,6 @@ int cpdma_chan_get_stats(struct cpdma_chan *chan, spin_unlock_irqrestore(&chan->lock, flags); return 0; } -EXPORT_SYMBOL_GPL(cpdma_chan_get_stats); static void __cpdma_chan_submit(struct cpdma_chan *chan, struct cpdma_desc __iomem *desc) @@ -1095,7 +1071,6 @@ unlock_ret: spin_unlock_irqrestore(&chan->lock, flags); return ret; } -EXPORT_SYMBOL_GPL(cpdma_chan_submit); bool cpdma_check_free_tx_desc(struct cpdma_chan *chan) { @@ -1110,7 +1085,6 @@ bool cpdma_check_free_tx_desc(struct cpdma_chan *chan) spin_unlock_irqrestore(&chan->lock, flags); return free_tx_desc; } -EXPORT_SYMBOL_GPL(cpdma_check_free_tx_desc); static void __cpdma_chan_free(struct cpdma_chan *chan, struct cpdma_desc __iomem *desc, @@ -1204,7 +1178,6 @@ int cpdma_chan_process(struct cpdma_chan *chan, int quota) } return used; } -EXPORT_SYMBOL_GPL(cpdma_chan_process); int cpdma_chan_start(struct cpdma_chan *chan) { @@ -1224,7 +1197,6 @@ int cpdma_chan_start(struct cpdma_chan *chan) return 0; } -EXPORT_SYMBOL_GPL(cpdma_chan_start); int cpdma_chan_stop(struct cpdma_chan *chan) { @@ -1287,7 +1259,6 @@ int cpdma_chan_stop(struct cpdma_chan *chan) spin_unlock_irqrestore(&chan->lock, flags); return 0; } -EXPORT_SYMBOL_GPL(cpdma_chan_stop); int cpdma_chan_int_ctrl(struct cpdma_chan *chan, bool enable) { @@ -1329,25 +1300,19 @@ int cpdma_control_set(struct cpdma_ctlr *ctlr, int control, int value) return ret; } -EXPORT_SYMBOL_GPL(cpdma_control_set); int cpdma_get_num_rx_descs(struct cpdma_ctlr *ctlr) { return ctlr->num_rx_desc; } -EXPORT_SYMBOL_GPL(cpdma_get_num_rx_descs); int cpdma_get_num_tx_descs(struct cpdma_ctlr *ctlr) { return ctlr->num_tx_desc; } -EXPORT_SYMBOL_GPL(cpdma_get_num_tx_descs); void cpdma_set_num_rx_descs(struct cpdma_ctlr *ctlr, int num_rx_desc) { ctlr->num_rx_desc = num_rx_desc; ctlr->num_tx_desc = ctlr->pool->num_desc - ctlr->num_rx_desc; } -EXPORT_SYMBOL_GPL(cpdma_set_num_rx_descs); - -MODULE_LICENSE("GPL"); diff --git a/drivers/net/ethernet/ti/davinci_cpdma.h b/drivers/net/ethernet/ti/davinci_cpdma.h index d399af5389b8cee5946d2f0dbcf6685d236ab3ef..10376062dafafb74912b6ba2f720c5021b167a1f 100644 --- a/drivers/net/ethernet/ti/davinci_cpdma.h +++ b/drivers/net/ethernet/ti/davinci_cpdma.h @@ -1,16 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* * Texas Instruments CPDMA Driver * * Copyright (C) 2010 Texas Instruments * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License as - * published by the Free Software Foundation version 2. - * - * This program is distributed "as is" WITHOUT ANY WARRANTY of any - * kind, whether express or implied; without even the implied warranty - * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #ifndef __DAVINCI_CPDMA_H__ #define __DAVINCI_CPDMA_H__ @@ -34,8 +27,8 @@ struct cpdma_params { int num_chan; bool has_soft_reset; int min_packet_size; - u32 desc_mem_phys; - u32 desc_hw_addr; + dma_addr_t desc_mem_phys; + dma_addr_t desc_hw_addr; int desc_mem_size; int desc_align; u32 bus_freq_mhz; diff --git a/drivers/net/ethernet/ti/davinci_emac.c b/drivers/net/ethernet/ti/davinci_emac.c index 57450b174fc488e44b0e45a6b81e54b97e442c39..4bf65cab79e6cae5d7c5b20cdc87a130f8ad7918 100644 --- a/drivers/net/ethernet/ti/davinci_emac.c +++ b/drivers/net/ethernet/ti/davinci_emac.c @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0+ /* * DaVinci Ethernet Medium Access Controller * @@ -6,21 +7,6 @@ * Copyright (C) 2009 Texas Instruments. * * --------------------------------------------------------------------------- - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, write to the Free Software - * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. - * --------------------------------------------------------------------------- * History: * 0-5 A number of folks worked on this driver in bits and pieces but the major * contribution came from Suraj Iyer and Anant Gole @@ -1714,7 +1700,7 @@ davinci_emac_of_get_pdata(struct platform_device *pdev, struct emac_priv *priv) if (!is_valid_ether_addr(pdata->mac_addr)) { mac_addr = of_get_mac_address(np); - if (mac_addr) + if (!IS_ERR(mac_addr)) ether_addr_copy(pdata->mac_addr, mac_addr); } @@ -1912,15 +1898,11 @@ static int davinci_emac_probe(struct platform_device *pdev) ether_addr_copy(ndev->dev_addr, priv->mac_addr); if (!is_valid_ether_addr(priv->mac_addr)) { - /* Try nvmem if MAC wasn't passed over pdata or DT. */ - rc = nvmem_get_mac_address(&pdev->dev, priv->mac_addr); - if (rc) { - /* Use random MAC if still none obtained. */ - eth_hw_addr_random(ndev); - memcpy(priv->mac_addr, ndev->dev_addr, ndev->addr_len); - dev_warn(&pdev->dev, "using random MAC addr: %pM\n", - priv->mac_addr); - } + /* Use random MAC if still none obtained. */ + eth_hw_addr_random(ndev); + memcpy(priv->mac_addr, ndev->dev_addr, ndev->addr_len); + dev_warn(&pdev->dev, "using random MAC addr: %pM\n", + priv->mac_addr); } ndev->netdev_ops = &emac_netdev_ops; diff --git a/drivers/net/ethernet/ti/davinci_mdio.c b/drivers/net/ethernet/ti/davinci_mdio.c index a98aedae1b41d7baa424d6b5a51622ec3af5bee0..38b7f6d35759c5c73fb73e6eb9a8ed0ff6ee4c28 100644 --- a/drivers/net/ethernet/ti/davinci_mdio.c +++ b/drivers/net/ethernet/ti/davinci_mdio.c @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0+ /* * DaVinci MDIO Module driver * @@ -7,22 +8,6 @@ * * Copyright (C) 2009 Texas Instruments. * - * --------------------------------------------------------------------------- - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, write to the Free Software - * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. - * --------------------------------------------------------------------------- */ #include #include @@ -140,7 +125,7 @@ static void davinci_mdio_init_clk(struct davinci_mdio_data *data) static void davinci_mdio_enable(struct davinci_mdio_data *data) { /* set enable and clock divider */ - __raw_writel(data->clk_div | CONTROL_ENABLE, &data->regs->control); + writel(data->clk_div | CONTROL_ENABLE, &data->regs->control); } static int davinci_mdio_reset(struct mii_bus *bus) @@ -159,7 +144,7 @@ static int davinci_mdio_reset(struct mii_bus *bus) msleep(PHY_MAX_ADDR * data->access_time); /* dump hardware version info */ - ver = __raw_readl(&data->regs->version); + ver = readl(&data->regs->version); dev_info(data->dev, "davinci mdio revision %d.%d, bus freq %ld\n", (ver >> 8) & 0xff, ver & 0xff, @@ -169,7 +154,7 @@ static int davinci_mdio_reset(struct mii_bus *bus) goto done; /* get phy mask from the alive register */ - phy_mask = __raw_readl(&data->regs->alive); + phy_mask = readl(&data->regs->alive); if (phy_mask) { /* restrict mdio bus to live phys only */ dev_info(data->dev, "detected phy mask %x\n", ~phy_mask); @@ -196,11 +181,11 @@ static inline int wait_for_user_access(struct davinci_mdio_data *data) u32 reg; while (time_after(timeout, jiffies)) { - reg = __raw_readl(®s->user[0].access); + reg = readl(®s->user[0].access); if ((reg & USERACCESS_GO) == 0) return 0; - reg = __raw_readl(®s->control); + reg = readl(®s->control); if ((reg & CONTROL_IDLE) == 0) { usleep_range(100, 200); continue; @@ -216,7 +201,7 @@ static inline int wait_for_user_access(struct davinci_mdio_data *data) return -EAGAIN; } - reg = __raw_readl(®s->user[0].access); + reg = readl(®s->user[0].access); if ((reg & USERACCESS_GO) == 0) return 0; @@ -263,7 +248,7 @@ static int davinci_mdio_read(struct mii_bus *bus, int phy_id, int phy_reg) if (ret < 0) break; - __raw_writel(reg, &data->regs->user[0].access); + writel(reg, &data->regs->user[0].access); ret = wait_for_user_access(data); if (ret == -EAGAIN) @@ -271,7 +256,7 @@ static int davinci_mdio_read(struct mii_bus *bus, int phy_id, int phy_reg) if (ret < 0) break; - reg = __raw_readl(&data->regs->user[0].access); + reg = readl(&data->regs->user[0].access); ret = (reg & USERACCESS_ACK) ? (reg & USERACCESS_DATA) : -EIO; break; } @@ -307,7 +292,7 @@ static int davinci_mdio_write(struct mii_bus *bus, int phy_id, if (ret < 0) break; - __raw_writel(reg, &data->regs->user[0].access); + writel(reg, &data->regs->user[0].access); ret = wait_for_user_access(data); if (ret == -EAGAIN) @@ -412,9 +397,9 @@ static int davinci_mdio_probe(struct platform_device *pdev) data->dev = dev; res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - data->regs = devm_ioremap_resource(dev, res); - if (IS_ERR(data->regs)) - return PTR_ERR(data->regs); + data->regs = devm_ioremap(dev, res->start, resource_size(res)); + if (!data->regs) + return -ENOMEM; davinci_mdio_init_clk(data); @@ -472,9 +457,9 @@ static int davinci_mdio_runtime_suspend(struct device *dev) u32 ctrl; /* shutdown the scan state machine */ - ctrl = __raw_readl(&data->regs->control); + ctrl = readl(&data->regs->control); ctrl &= ~CONTROL_ENABLE; - __raw_writel(ctrl, &data->regs->control); + writel(ctrl, &data->regs->control); wait_for_idle(data); return 0; diff --git a/drivers/net/ethernet/ti/netcp.h b/drivers/net/ethernet/ti/netcp.h index c4ffdf47bad56c2df0a091850ab7d6f979c88ee4..43d5cd59b56bbaf7ede00c365efe04c43c032953 100644 --- a/drivers/net/ethernet/ti/netcp.h +++ b/drivers/net/ethernet/ti/netcp.h @@ -1,3 +1,4 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* * NetCP driver local header * @@ -8,15 +9,6 @@ * Santosh Shilimkar * Wingman Kwok * Murali Karicheri - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License as - * published by the Free Software Foundation version 2. - * - * This program is distributed "as is" WITHOUT ANY WARRANTY of any - * kind, whether express or implied; without even the implied warranty - * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #ifndef __NETCP_H__ #define __NETCP_H__ diff --git a/drivers/net/ethernet/ti/netcp_core.c b/drivers/net/ethernet/ti/netcp_core.c index d847f672a705f47c6b4bdf579917385b2640da79..642843945031c222a5f5225832c83df11aa87145 100644 --- a/drivers/net/ethernet/ti/netcp_core.c +++ b/drivers/net/ethernet/ti/netcp_core.c @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Keystone NetCP Core driver * @@ -8,15 +9,6 @@ * Santosh Shilimkar * Murali Karicheri * Wingman Kwok - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License as - * published by the Free Software Foundation version 2. - * - * This program is distributed "as is" WITHOUT ANY WARRANTY of any - * kind, whether express or implied; without even the implied warranty - * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include @@ -2045,7 +2037,7 @@ static int netcp_create_interface(struct netcp_device *netcp_device, devm_release_mem_region(dev, res.start, size); } else { mac_addr = of_get_mac_address(node_interface); - if (mac_addr) + if (!IS_ERR(mac_addr)) ether_addr_copy(ndev->dev_addr, mac_addr); else eth_random_addr(ndev->dev_addr); diff --git a/drivers/net/ethernet/ti/netcp_ethss.c b/drivers/net/ethernet/ti/netcp_ethss.c index 0a920c5936b24e1a14a8625a63f9bf2eed019ccb..ec179700c184081a306c1205226cd63acf13454e 100644 --- a/drivers/net/ethernet/ti/netcp_ethss.c +++ b/drivers/net/ethernet/ti/netcp_ethss.c @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Keystone GBE and XGBE subsystem code * @@ -7,15 +8,6 @@ * Cyril Chemparathy * Santosh Shilimkar * Wingman Kwok - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License as - * published by the Free Software Foundation version 2. - * - * This program is distributed "as is" WITHOUT ANY WARRANTY of any - * kind, whether express or implied; without even the implied warranty - * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include diff --git a/drivers/net/ethernet/ti/netcp_sgmii.c b/drivers/net/ethernet/ti/netcp_sgmii.c index 5d8419f658d04397a7be30c2a1b9a249f492e6e8..f7cf56d6351d75c54e413c3afad2da2a675c012b 100644 --- a/drivers/net/ethernet/ti/netcp_sgmii.c +++ b/drivers/net/ethernet/ti/netcp_sgmii.c @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* * SGMI module initialisation * @@ -6,14 +7,6 @@ * Sandeep Paulraj * Wingman Kwok * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License as - * published by the Free Software Foundation version 2. - * - * This program is distributed "as is" WITHOUT ANY WARRANTY of any - * kind, whether express or implied; without even the implied warranty - * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include "netcp.h" diff --git a/drivers/net/ethernet/ti/netcp_xgbepcsr.c b/drivers/net/ethernet/ti/netcp_xgbepcsr.c index 33571acc52b6ac0497fc6884280955f9cb416558..112778aedd8aff2670c94fca77996a2225b71a55 100644 --- a/drivers/net/ethernet/ti/netcp_xgbepcsr.c +++ b/drivers/net/ethernet/ti/netcp_xgbepcsr.c @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* * XGE PCSR module initialisation * @@ -5,14 +6,6 @@ * Authors: Sandeep Nair * WingMan Kwok * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License as - * published by the Free Software Foundation version 2. - * - * This program is distributed "as is" WITHOUT ANY WARRANTY of any - * kind, whether express or implied; without even the implied warranty - * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include "netcp.h" diff --git a/drivers/net/ethernet/wiznet/w5100.c b/drivers/net/ethernet/wiznet/w5100.c index 1713c2d2dccf22d6c374078ddc25cf16f9d110ac..8788953eaafdc3cba35dea23252d3ee99d793793 100644 --- a/drivers/net/ethernet/wiznet/w5100.c +++ b/drivers/net/ethernet/wiznet/w5100.c @@ -1158,7 +1158,7 @@ int w5100_probe(struct device *dev, const struct w5100_ops *ops, INIT_WORK(&priv->setrx_work, w5100_setrx_work); INIT_WORK(&priv->restart_work, w5100_restart_work); - if (mac_addr) + if (!IS_ERR_OR_NULL(mac_addr)) memcpy(ndev->dev_addr, mac_addr, ETH_ALEN); else eth_hw_addr_random(ndev); diff --git a/drivers/net/ethernet/xilinx/Kconfig b/drivers/net/ethernet/xilinx/Kconfig index da4ec575ccf9ba4aede92c0fd3a4842ed4257449..db448fad621b881cdb201cff51201ed978b5779a 100644 --- a/drivers/net/ethernet/xilinx/Kconfig +++ b/drivers/net/ethernet/xilinx/Kconfig @@ -5,7 +5,7 @@ config NET_VENDOR_XILINX bool "Xilinx devices" default y - depends on PPC || PPC32 || MICROBLAZE || ARCH_ZYNQ || MIPS + depends on PPC || PPC32 || MICROBLAZE || ARCH_ZYNQ || MIPS || X86 || COMPILE_TEST ---help--- If you have a network (Ethernet) card belonging to this class, say Y. @@ -33,8 +33,7 @@ config XILINX_AXI_EMAC config XILINX_LL_TEMAC tristate "Xilinx LL TEMAC (LocalLink Tri-mode Ethernet MAC) driver" - depends on (PPC || MICROBLAZE) - depends on !64BIT || BROKEN + depends on PPC || MICROBLAZE || X86 || COMPILE_TEST select PHYLIB ---help--- This driver supports the Xilinx 10/100/1000 LocalLink TEMAC diff --git a/drivers/net/ethernet/xilinx/ll_temac.h b/drivers/net/ethernet/xilinx/ll_temac.h index 107575225383297349666ce944667275a65af5e5..1aeda084b8f15e773e0abda0f7d27ba4a160ff74 100644 --- a/drivers/net/ethernet/xilinx/ll_temac.h +++ b/drivers/net/ethernet/xilinx/ll_temac.h @@ -334,6 +334,9 @@ struct temac_local { /* Connection to PHY device */ struct device_node *phy_node; + /* For non-device-tree devices */ + char phy_name[MII_BUS_ID_SIZE + 3]; + phy_interface_t phy_interface; /* MDIO bus data */ struct mii_bus *mii_bus; /* MII bus reference */ @@ -344,8 +347,10 @@ struct temac_local { #ifdef CONFIG_PPC_DCR dcr_host_t sdma_dcrs; #endif - u32 (*dma_in)(struct temac_local *, int); - void (*dma_out)(struct temac_local *, int, u32); + u32 (*temac_ior)(struct temac_local *lp, int offset); + void (*temac_iow)(struct temac_local *lp, int offset, u32 value); + u32 (*dma_in)(struct temac_local *lp, int reg); + void (*dma_out)(struct temac_local *lp, int reg, u32 value); int tx_irq; int rx_irq; @@ -353,7 +358,10 @@ struct temac_local { struct sk_buff **rx_skb; spinlock_t rx_lock; - struct mutex indirect_mutex; + /* For synchronization of indirect register access. Must be + * shared mutex between interfaces in same TEMAC block. + */ + struct mutex *indirect_mutex; u32 options; /* Current options word */ int last_link; unsigned int temac_features; @@ -367,18 +375,24 @@ struct temac_local { int tx_bd_next; int tx_bd_tail; int rx_bd_ci; + + /* DMA channel control setup */ + u32 tx_chnl_ctrl; + u32 rx_chnl_ctrl; }; +/* Wrappers for temac_ior()/temac_iow() function pointers above */ +#define temac_ior(lp, o) ((lp)->temac_ior(lp, o)) +#define temac_iow(lp, o, v) ((lp)->temac_iow(lp, o, v)) + /* xilinx_temac.c */ -u32 temac_ior(struct temac_local *lp, int offset); -void temac_iow(struct temac_local *lp, int offset, u32 value); int temac_indirect_busywait(struct temac_local *lp); u32 temac_indirect_in32(struct temac_local *lp, int reg); void temac_indirect_out32(struct temac_local *lp, int reg, u32 value); /* xilinx_temac_mdio.c */ -int temac_mdio_setup(struct temac_local *lp, struct device_node *np); +int temac_mdio_setup(struct temac_local *lp, struct platform_device *pdev); void temac_mdio_teardown(struct temac_local *lp); #endif /* XILINX_LL_TEMAC_H */ diff --git a/drivers/net/ethernet/xilinx/ll_temac_main.c b/drivers/net/ethernet/xilinx/ll_temac_main.c index 44efffbe79704ae555746834ddcb9982a2b12eec..997475c209c0da05d762b900a916b4ffdafad830 100644 --- a/drivers/net/ethernet/xilinx/ll_temac_main.c +++ b/drivers/net/ethernet/xilinx/ll_temac_main.c @@ -33,6 +33,7 @@ #include #include #include +#include #include #include #include @@ -51,6 +52,7 @@ #include #include #include +#include #include "ll_temac.h" @@ -61,14 +63,24 @@ * Low level register access functions */ -u32 temac_ior(struct temac_local *lp, int offset) +static u32 _temac_ior_be(struct temac_local *lp, int offset) { - return in_be32(lp->regs + offset); + return ioread32be(lp->regs + offset); } -void temac_iow(struct temac_local *lp, int offset, u32 value) +static void _temac_iow_be(struct temac_local *lp, int offset, u32 value) { - out_be32(lp->regs + offset, value); + return iowrite32be(value, lp->regs + offset); +} + +static u32 _temac_ior_le(struct temac_local *lp, int offset) +{ + return ioread32(lp->regs + offset); +} + +static void _temac_iow_le(struct temac_local *lp, int offset, u32 value) +{ + return iowrite32(value, lp->regs + offset); } int temac_indirect_busywait(struct temac_local *lp) @@ -80,7 +92,7 @@ int temac_indirect_busywait(struct temac_local *lp) WARN_ON(1); return -ETIMEDOUT; } - msleep(1); + usleep_range(500, 1000); } return 0; } @@ -119,23 +131,35 @@ void temac_indirect_out32(struct temac_local *lp, int reg, u32 value) } /** - * temac_dma_in32 - Memory mapped DMA read, this function expects a - * register input that is based on DCR word addresses which - * are then converted to memory mapped byte addresses + * temac_dma_in32_* - Memory mapped DMA read, these function expects a + * register input that is based on DCR word addresses which are then + * converted to memory mapped byte addresses. To be assigned to + * lp->dma_in32. */ -static u32 temac_dma_in32(struct temac_local *lp, int reg) +static u32 temac_dma_in32_be(struct temac_local *lp, int reg) { - return in_be32(lp->sdma_regs + (reg << 2)); + return ioread32be(lp->sdma_regs + (reg << 2)); +} + +static u32 temac_dma_in32_le(struct temac_local *lp, int reg) +{ + return ioread32(lp->sdma_regs + (reg << 2)); } /** - * temac_dma_out32 - Memory mapped DMA read, this function expects a - * register input that is based on DCR word addresses which - * are then converted to memory mapped byte addresses + * temac_dma_out32_* - Memory mapped DMA read, these function expects + * a register input that is based on DCR word addresses which are then + * converted to memory mapped byte addresses. To be assigned to + * lp->dma_out32. */ -static void temac_dma_out32(struct temac_local *lp, int reg, u32 value) +static void temac_dma_out32_be(struct temac_local *lp, int reg, u32 value) +{ + iowrite32be(value, lp->sdma_regs + (reg << 2)); +} + +static void temac_dma_out32_le(struct temac_local *lp, int reg, u32 value) { - out_be32(lp->sdma_regs + (reg << 2), value); + iowrite32(value, lp->sdma_regs + (reg << 2)); } /* DMA register access functions can be DCR based or memory mapped. @@ -187,7 +211,7 @@ static int temac_dcr_setup(struct temac_local *lp, struct platform_device *op, /* * temac_dcr_setup - This is a stub for when DCR is not supported, - * such as with MicroBlaze + * such as with MicroBlaze and x86 */ static int temac_dcr_setup(struct temac_local *lp, struct platform_device *op, struct device_node *np) @@ -225,7 +249,6 @@ static void temac_dma_bd_release(struct net_device *ndev) dma_free_coherent(ndev->dev.parent, sizeof(*lp->tx_bd_v) * TX_BD_NUM, lp->tx_bd_v, lp->tx_bd_p); - kfree(lp->rx_skb); } /** @@ -235,9 +258,11 @@ static int temac_dma_bd_init(struct net_device *ndev) { struct temac_local *lp = netdev_priv(ndev); struct sk_buff *skb; + dma_addr_t skb_dma_addr; int i; - lp->rx_skb = kcalloc(RX_BD_NUM, sizeof(*lp->rx_skb), GFP_KERNEL); + lp->rx_skb = devm_kcalloc(&ndev->dev, RX_BD_NUM, sizeof(*lp->rx_skb), + GFP_KERNEL); if (!lp->rx_skb) goto out; @@ -256,13 +281,13 @@ static int temac_dma_bd_init(struct net_device *ndev) goto out; for (i = 0; i < TX_BD_NUM; i++) { - lp->tx_bd_v[i].next = lp->tx_bd_p + - sizeof(*lp->tx_bd_v) * ((i + 1) % TX_BD_NUM); + lp->tx_bd_v[i].next = cpu_to_be32(lp->tx_bd_p + + sizeof(*lp->tx_bd_v) * ((i + 1) % TX_BD_NUM)); } for (i = 0; i < RX_BD_NUM; i++) { - lp->rx_bd_v[i].next = lp->rx_bd_p + - sizeof(*lp->rx_bd_v) * ((i + 1) % RX_BD_NUM); + lp->rx_bd_v[i].next = cpu_to_be32(lp->rx_bd_p + + sizeof(*lp->rx_bd_v) * ((i + 1) % RX_BD_NUM)); skb = netdev_alloc_skb_ip_align(ndev, XTE_MAX_JUMBO_FRAME_SIZE); @@ -271,31 +296,23 @@ static int temac_dma_bd_init(struct net_device *ndev) lp->rx_skb[i] = skb; /* returns physical address of skb->data */ - lp->rx_bd_v[i].phys = dma_map_single(ndev->dev.parent, - skb->data, - XTE_MAX_JUMBO_FRAME_SIZE, - DMA_FROM_DEVICE); - lp->rx_bd_v[i].len = XTE_MAX_JUMBO_FRAME_SIZE; - lp->rx_bd_v[i].app0 = STS_CTRL_APP0_IRQONEND; + skb_dma_addr = dma_map_single(ndev->dev.parent, skb->data, + XTE_MAX_JUMBO_FRAME_SIZE, + DMA_FROM_DEVICE); + lp->rx_bd_v[i].phys = cpu_to_be32(skb_dma_addr); + lp->rx_bd_v[i].len = cpu_to_be32(XTE_MAX_JUMBO_FRAME_SIZE); + lp->rx_bd_v[i].app0 = cpu_to_be32(STS_CTRL_APP0_IRQONEND); } - lp->dma_out(lp, TX_CHNL_CTRL, 0x10220400 | - CHNL_CTRL_IRQ_EN | - CHNL_CTRL_IRQ_DLY_EN | - CHNL_CTRL_IRQ_COAL_EN); - /* 0x10220483 */ - /* 0x00100483 */ - lp->dma_out(lp, RX_CHNL_CTRL, 0xff070000 | - CHNL_CTRL_IRQ_EN | - CHNL_CTRL_IRQ_DLY_EN | - CHNL_CTRL_IRQ_COAL_EN | - CHNL_CTRL_IRQ_IOE); - /* 0xff010283 */ - - lp->dma_out(lp, RX_CURDESC_PTR, lp->rx_bd_p); - lp->dma_out(lp, RX_TAILDESC_PTR, - lp->rx_bd_p + (sizeof(*lp->rx_bd_v) * (RX_BD_NUM - 1))); - lp->dma_out(lp, TX_CURDESC_PTR, lp->tx_bd_p); + /* Configure DMA channel (irq setup) */ + lp->dma_out(lp, TX_CHNL_CTRL, lp->tx_chnl_ctrl | + 0x00000400 | // Use 1 Bit Wide Counters. Currently Not Used! + CHNL_CTRL_IRQ_EN | CHNL_CTRL_IRQ_ERR_EN | + CHNL_CTRL_IRQ_DLY_EN | CHNL_CTRL_IRQ_COAL_EN); + lp->dma_out(lp, RX_CHNL_CTRL, lp->rx_chnl_ctrl | + CHNL_CTRL_IRQ_IOE | + CHNL_CTRL_IRQ_EN | CHNL_CTRL_IRQ_ERR_EN | + CHNL_CTRL_IRQ_DLY_EN | CHNL_CTRL_IRQ_COAL_EN); /* Init descriptor indexes */ lp->tx_bd_ci = 0; @@ -303,6 +320,15 @@ static int temac_dma_bd_init(struct net_device *ndev) lp->tx_bd_tail = 0; lp->rx_bd_ci = 0; + /* Enable RX DMA transfers */ + wmb(); + lp->dma_out(lp, RX_CURDESC_PTR, lp->rx_bd_p); + lp->dma_out(lp, RX_TAILDESC_PTR, + lp->rx_bd_p + (sizeof(*lp->rx_bd_v) * (RX_BD_NUM - 1))); + + /* Prepare for TX DMA transfer */ + lp->dma_out(lp, TX_CURDESC_PTR, lp->tx_bd_p); + return 0; out: @@ -319,7 +345,7 @@ static void temac_do_set_mac_address(struct net_device *ndev) struct temac_local *lp = netdev_priv(ndev); /* set up unicast MAC address filter set its mac address */ - mutex_lock(&lp->indirect_mutex); + mutex_lock(lp->indirect_mutex); temac_indirect_out32(lp, XTE_UAW0_OFFSET, (ndev->dev_addr[0]) | (ndev->dev_addr[1] << 8) | @@ -330,7 +356,7 @@ static void temac_do_set_mac_address(struct net_device *ndev) temac_indirect_out32(lp, XTE_UAW1_OFFSET, (ndev->dev_addr[4] & 0x000000ff) | (ndev->dev_addr[5] << 8)); - mutex_unlock(&lp->indirect_mutex); + mutex_unlock(lp->indirect_mutex); } static int temac_init_mac_address(struct net_device *ndev, const void *address) @@ -359,7 +385,7 @@ static void temac_set_multicast_list(struct net_device *ndev) u32 multi_addr_msw, multi_addr_lsw, val; int i; - mutex_lock(&lp->indirect_mutex); + mutex_lock(lp->indirect_mutex); if (ndev->flags & (IFF_ALLMULTI | IFF_PROMISC) || netdev_mc_count(ndev) > MULTICAST_CAM_TABLE_NUM) { /* @@ -398,7 +424,7 @@ static void temac_set_multicast_list(struct net_device *ndev) temac_indirect_out32(lp, XTE_MAW1_OFFSET, 0); dev_info(&ndev->dev, "Promiscuous mode disabled.\n"); } - mutex_unlock(&lp->indirect_mutex); + mutex_unlock(lp->indirect_mutex); } static struct temac_option { @@ -490,7 +516,7 @@ static u32 temac_setoptions(struct net_device *ndev, u32 options) struct temac_option *tp = &temac_options[0]; int reg; - mutex_lock(&lp->indirect_mutex); + mutex_lock(lp->indirect_mutex); while (tp->opt) { reg = temac_indirect_in32(lp, tp->reg) & ~tp->m_or; if (options & tp->opt) @@ -499,7 +525,7 @@ static u32 temac_setoptions(struct net_device *ndev, u32 options) tp++; } lp->options |= options; - mutex_unlock(&lp->indirect_mutex); + mutex_unlock(lp->indirect_mutex); return 0; } @@ -518,7 +544,7 @@ static void temac_device_reset(struct net_device *ndev) dev_dbg(&ndev->dev, "%s()\n", __func__); - mutex_lock(&lp->indirect_mutex); + mutex_lock(lp->indirect_mutex); /* Reset the receiver and wait for it to finish reset */ temac_indirect_out32(lp, XTE_RXC1_OFFSET, XTE_RXC1_RXRST_MASK); timeout = 1000; @@ -570,7 +596,7 @@ static void temac_device_reset(struct net_device *ndev) temac_indirect_out32(lp, XTE_TXC_OFFSET, 0); temac_indirect_out32(lp, XTE_FCC_OFFSET, XTE_FCC_RXFLO_MASK); - mutex_unlock(&lp->indirect_mutex); + mutex_unlock(lp->indirect_mutex); /* Sync default options with HW * but leave receiver and transmitter disabled. */ @@ -598,7 +624,7 @@ static void temac_adjust_link(struct net_device *ndev) /* hash together the state values to decide if something has changed */ link_state = phy->speed | (phy->duplex << 1) | phy->link; - mutex_lock(&lp->indirect_mutex); + mutex_lock(lp->indirect_mutex); if (lp->last_link != link_state) { mii_speed = temac_indirect_in32(lp, XTE_EMCFG_OFFSET); mii_speed &= ~XTE_EMCFG_LINKSPD_MASK; @@ -614,23 +640,52 @@ static void temac_adjust_link(struct net_device *ndev) lp->last_link = link_state; phy_print_status(phy); } - mutex_unlock(&lp->indirect_mutex); + mutex_unlock(lp->indirect_mutex); +} + +#ifdef CONFIG_64BIT + +static void ptr_to_txbd(void *p, struct cdmac_bd *bd) +{ + bd->app3 = (u32)(((u64)p) >> 32); + bd->app4 = (u32)((u64)p & 0xFFFFFFFF); +} + +static void *ptr_from_txbd(struct cdmac_bd *bd) +{ + return (void *)(((u64)(bd->app3) << 32) | bd->app4); } +#else + +static void ptr_to_txbd(void *p, struct cdmac_bd *bd) +{ + bd->app4 = (u32)p; +} + +static void *ptr_from_txbd(struct cdmac_bd *bd) +{ + return (void *)(bd->app4); +} + +#endif + static void temac_start_xmit_done(struct net_device *ndev) { struct temac_local *lp = netdev_priv(ndev); struct cdmac_bd *cur_p; unsigned int stat = 0; + struct sk_buff *skb; cur_p = &lp->tx_bd_v[lp->tx_bd_ci]; - stat = cur_p->app0; + stat = be32_to_cpu(cur_p->app0); while (stat & STS_CTRL_APP0_CMPLT) { - dma_unmap_single(ndev->dev.parent, cur_p->phys, cur_p->len, - DMA_TO_DEVICE); - if (cur_p->app4) - dev_consume_skb_irq((struct sk_buff *)cur_p->app4); + dma_unmap_single(ndev->dev.parent, be32_to_cpu(cur_p->phys), + be32_to_cpu(cur_p->len), DMA_TO_DEVICE); + skb = (struct sk_buff *)ptr_from_txbd(cur_p); + if (skb) + dev_consume_skb_irq(skb); cur_p->app0 = 0; cur_p->app1 = 0; cur_p->app2 = 0; @@ -638,14 +693,14 @@ static void temac_start_xmit_done(struct net_device *ndev) cur_p->app4 = 0; ndev->stats.tx_packets++; - ndev->stats.tx_bytes += cur_p->len; + ndev->stats.tx_bytes += be32_to_cpu(cur_p->len); lp->tx_bd_ci++; if (lp->tx_bd_ci >= TX_BD_NUM) lp->tx_bd_ci = 0; cur_p = &lp->tx_bd_v[lp->tx_bd_ci]; - stat = cur_p->app0; + stat = be32_to_cpu(cur_p->app0); } netif_wake_queue(ndev); @@ -679,7 +734,7 @@ temac_start_xmit(struct sk_buff *skb, struct net_device *ndev) { struct temac_local *lp = netdev_priv(ndev); struct cdmac_bd *cur_p; - dma_addr_t start_p, tail_p; + dma_addr_t start_p, tail_p, skb_dma_addr; int ii; unsigned long num_frag; skb_frag_t *frag; @@ -689,7 +744,7 @@ temac_start_xmit(struct sk_buff *skb, struct net_device *ndev) start_p = lp->tx_bd_p + sizeof(*lp->tx_bd_v) * lp->tx_bd_tail; cur_p = &lp->tx_bd_v[lp->tx_bd_tail]; - if (temac_check_tx_bd_space(lp, num_frag)) { + if (temac_check_tx_bd_space(lp, num_frag + 1)) { if (!netif_queue_stopped(ndev)) netif_stop_queue(ndev); return NETDEV_TX_BUSY; @@ -700,16 +755,18 @@ temac_start_xmit(struct sk_buff *skb, struct net_device *ndev) unsigned int csum_start_off = skb_checksum_start_offset(skb); unsigned int csum_index_off = csum_start_off + skb->csum_offset; - cur_p->app0 |= 1; /* TX Checksum Enabled */ - cur_p->app1 = (csum_start_off << 16) | csum_index_off; + cur_p->app0 |= cpu_to_be32(0x000001); /* TX Checksum Enabled */ + cur_p->app1 = cpu_to_be32((csum_start_off << 16) + | csum_index_off); cur_p->app2 = 0; /* initial checksum seed */ } - cur_p->app0 |= STS_CTRL_APP0_SOP; - cur_p->len = skb_headlen(skb); - cur_p->phys = dma_map_single(ndev->dev.parent, skb->data, - skb_headlen(skb), DMA_TO_DEVICE); - cur_p->app4 = (unsigned long)skb; + cur_p->app0 |= cpu_to_be32(STS_CTRL_APP0_SOP); + skb_dma_addr = dma_map_single(ndev->dev.parent, skb->data, + skb_headlen(skb), DMA_TO_DEVICE); + cur_p->len = cpu_to_be32(skb_headlen(skb)); + cur_p->phys = cpu_to_be32(skb_dma_addr); + ptr_to_txbd((void *)skb, cur_p); for (ii = 0; ii < num_frag; ii++) { lp->tx_bd_tail++; @@ -717,14 +774,16 @@ temac_start_xmit(struct sk_buff *skb, struct net_device *ndev) lp->tx_bd_tail = 0; cur_p = &lp->tx_bd_v[lp->tx_bd_tail]; - cur_p->phys = dma_map_single(ndev->dev.parent, - skb_frag_address(frag), - skb_frag_size(frag), DMA_TO_DEVICE); - cur_p->len = skb_frag_size(frag); + skb_dma_addr = dma_map_single(ndev->dev.parent, + skb_frag_address(frag), + skb_frag_size(frag), + DMA_TO_DEVICE); + cur_p->phys = cpu_to_be32(skb_dma_addr); + cur_p->len = cpu_to_be32(skb_frag_size(frag)); cur_p->app0 = 0; frag++; } - cur_p->app0 |= STS_CTRL_APP0_EOP; + cur_p->app0 |= cpu_to_be32(STS_CTRL_APP0_EOP); tail_p = lp->tx_bd_p + sizeof(*lp->tx_bd_v) * lp->tx_bd_tail; lp->tx_bd_tail++; @@ -734,6 +793,7 @@ temac_start_xmit(struct sk_buff *skb, struct net_device *ndev) skb_tx_timestamp(skb); /* Kick off the transfer */ + wmb(); lp->dma_out(lp, TX_TAILDESC_PTR, tail_p); /* DMA start */ return NETDEV_TX_OK; @@ -746,7 +806,7 @@ static void ll_temac_recv(struct net_device *ndev) struct sk_buff *skb, *new_skb; unsigned int bdstat; struct cdmac_bd *cur_p; - dma_addr_t tail_p; + dma_addr_t tail_p, skb_dma_addr; int length; unsigned long flags; @@ -755,14 +815,14 @@ static void ll_temac_recv(struct net_device *ndev) tail_p = lp->rx_bd_p + sizeof(*lp->rx_bd_v) * lp->rx_bd_ci; cur_p = &lp->rx_bd_v[lp->rx_bd_ci]; - bdstat = cur_p->app0; + bdstat = be32_to_cpu(cur_p->app0); while ((bdstat & STS_CTRL_APP0_CMPLT)) { skb = lp->rx_skb[lp->rx_bd_ci]; - length = cur_p->app4 & 0x3FFF; + length = be32_to_cpu(cur_p->app4) & 0x3FFF; - dma_unmap_single(ndev->dev.parent, cur_p->phys, length, - DMA_FROM_DEVICE); + dma_unmap_single(ndev->dev.parent, be32_to_cpu(cur_p->phys), + XTE_MAX_JUMBO_FRAME_SIZE, DMA_FROM_DEVICE); skb_put(skb, length); skb->protocol = eth_type_trans(skb, ndev); @@ -773,7 +833,12 @@ static void ll_temac_recv(struct net_device *ndev) (skb->protocol == htons(ETH_P_IP)) && (skb->len > 64)) { - skb->csum = cur_p->app3 & 0xFFFF; + /* Convert from device endianness (be32) to cpu + * endiannes, and if necessary swap the bytes + * (back) for proper IP checksum byte order + * (be16). + */ + skb->csum = htons(be32_to_cpu(cur_p->app3) & 0xFFFF); skb->ip_summed = CHECKSUM_COMPLETE; } @@ -790,11 +855,12 @@ static void ll_temac_recv(struct net_device *ndev) return; } - cur_p->app0 = STS_CTRL_APP0_IRQONEND; - cur_p->phys = dma_map_single(ndev->dev.parent, new_skb->data, - XTE_MAX_JUMBO_FRAME_SIZE, - DMA_FROM_DEVICE); - cur_p->len = XTE_MAX_JUMBO_FRAME_SIZE; + cur_p->app0 = cpu_to_be32(STS_CTRL_APP0_IRQONEND); + skb_dma_addr = dma_map_single(ndev->dev.parent, new_skb->data, + XTE_MAX_JUMBO_FRAME_SIZE, + DMA_FROM_DEVICE); + cur_p->phys = cpu_to_be32(skb_dma_addr); + cur_p->len = cpu_to_be32(XTE_MAX_JUMBO_FRAME_SIZE); lp->rx_skb[lp->rx_bd_ci] = new_skb; lp->rx_bd_ci++; @@ -802,7 +868,7 @@ static void ll_temac_recv(struct net_device *ndev) lp->rx_bd_ci = 0; cur_p = &lp->rx_bd_v[lp->rx_bd_ci]; - bdstat = cur_p->app0; + bdstat = be32_to_cpu(cur_p->app0); } lp->dma_out(lp, RX_TAILDESC_PTR, tail_p); @@ -820,8 +886,10 @@ static irqreturn_t ll_temac_tx_irq(int irq, void *_ndev) if (status & (IRQ_COAL | IRQ_DLY)) temac_start_xmit_done(lp->ndev); - if (status & 0x080) - dev_err(&ndev->dev, "DMA error 0x%x\n", status); + if (status & (IRQ_ERR | IRQ_DMAERR)) + dev_err_ratelimited(&ndev->dev, + "TX error 0x%x TX_CHNL_STS=0x%08x\n", + status, lp->dma_in(lp, TX_CHNL_STS)); return IRQ_HANDLED; } @@ -838,6 +906,10 @@ static irqreturn_t ll_temac_rx_irq(int irq, void *_ndev) if (status & (IRQ_COAL | IRQ_DLY)) ll_temac_recv(lp->ndev); + if (status & (IRQ_ERR | IRQ_DMAERR)) + dev_err_ratelimited(&ndev->dev, + "RX error 0x%x RX_CHNL_STS=0x%08x\n", + status, lp->dma_in(lp, RX_CHNL_STS)); return IRQ_HANDLED; } @@ -857,7 +929,14 @@ static int temac_open(struct net_device *ndev) dev_err(lp->dev, "of_phy_connect() failed\n"); return -ENODEV; } - + phy_start(phydev); + } else if (strlen(lp->phy_name) > 0) { + phydev = phy_connect(lp->ndev, lp->phy_name, temac_adjust_link, + lp->phy_interface); + if (IS_ERR(phydev)) { + dev_err(lp->dev, "phy_connect() failed\n"); + return PTR_ERR(phydev); + } phy_start(phydev); } @@ -977,22 +1056,25 @@ static const struct ethtool_ops temac_ethtool_ops = { .set_link_ksettings = phy_ethtool_set_link_ksettings, }; -static int temac_of_probe(struct platform_device *op) +static int temac_probe(struct platform_device *pdev) { - struct device_node *np; + struct ll_temac_platform_data *pdata = dev_get_platdata(&pdev->dev); + struct device_node *temac_np = dev_of_node(&pdev->dev), *dma_np; struct temac_local *lp; struct net_device *ndev; + struct resource *res; const void *addr; __be32 *p; + bool little_endian; int rc = 0; /* Init network device structure */ - ndev = alloc_etherdev(sizeof(*lp)); + ndev = devm_alloc_etherdev(&pdev->dev, sizeof(*lp)); if (!ndev) return -ENOMEM; - platform_set_drvdata(op, ndev); - SET_NETDEV_DEV(ndev, &op->dev); + platform_set_drvdata(pdev, ndev); + SET_NETDEV_DEV(ndev, &pdev->dev); ndev->flags &= ~IFF_MULTICAST; /* clear multicast */ ndev->features = NETIF_F_SG; ndev->netdev_ops = &temac_netdev_ops; @@ -1014,89 +1096,196 @@ static int temac_of_probe(struct platform_device *op) /* setup temac private info structure */ lp = netdev_priv(ndev); lp->ndev = ndev; - lp->dev = &op->dev; + lp->dev = &pdev->dev; lp->options = XTE_OPTION_DEFAULTS; spin_lock_init(&lp->rx_lock); - mutex_init(&lp->indirect_mutex); + + /* Setup mutex for synchronization of indirect register access */ + if (pdata) { + if (!pdata->indirect_mutex) { + dev_err(&pdev->dev, + "indirect_mutex missing in platform_data\n"); + return -EINVAL; + } + lp->indirect_mutex = pdata->indirect_mutex; + } else { + lp->indirect_mutex = devm_kmalloc(&pdev->dev, + sizeof(*lp->indirect_mutex), + GFP_KERNEL); + mutex_init(lp->indirect_mutex); + } /* map device registers */ - lp->regs = of_iomap(op->dev.of_node, 0); - if (!lp->regs) { - dev_err(&op->dev, "could not map temac regs.\n"); - rc = -ENOMEM; - goto nodev; + res = platform_get_resource(pdev, IORESOURCE_MEM, 0); + lp->regs = devm_ioremap_nocache(&pdev->dev, res->start, + resource_size(res)); + if (IS_ERR(lp->regs)) { + dev_err(&pdev->dev, "could not map TEMAC registers\n"); + return PTR_ERR(lp->regs); + } + + /* Select register access functions with the specified + * endianness mode. Default for OF devices is big-endian. + */ + little_endian = false; + if (temac_np) { + if (of_get_property(temac_np, "little-endian", NULL)) + little_endian = true; + } else if (pdata) { + little_endian = pdata->reg_little_endian; + } + if (little_endian) { + lp->temac_ior = _temac_ior_le; + lp->temac_iow = _temac_iow_le; + } else { + lp->temac_ior = _temac_ior_be; + lp->temac_iow = _temac_iow_be; } /* Setup checksum offload, but default to off if not specified */ lp->temac_features = 0; - p = (__be32 *)of_get_property(op->dev.of_node, "xlnx,txcsum", NULL); - if (p && be32_to_cpu(*p)) { - lp->temac_features |= TEMAC_FEATURE_TX_CSUM; + if (temac_np) { + p = (__be32 *)of_get_property(temac_np, "xlnx,txcsum", NULL); + if (p && be32_to_cpu(*p)) + lp->temac_features |= TEMAC_FEATURE_TX_CSUM; + p = (__be32 *)of_get_property(temac_np, "xlnx,rxcsum", NULL); + if (p && be32_to_cpu(*p)) + lp->temac_features |= TEMAC_FEATURE_RX_CSUM; + } else if (pdata) { + if (pdata->txcsum) + lp->temac_features |= TEMAC_FEATURE_TX_CSUM; + if (pdata->rxcsum) + lp->temac_features |= TEMAC_FEATURE_RX_CSUM; + } + if (lp->temac_features & TEMAC_FEATURE_TX_CSUM) /* Can checksum TCP/UDP over IPv4. */ ndev->features |= NETIF_F_IP_CSUM; - } - p = (__be32 *)of_get_property(op->dev.of_node, "xlnx,rxcsum", NULL); - if (p && be32_to_cpu(*p)) - lp->temac_features |= TEMAC_FEATURE_RX_CSUM; - - /* Find the DMA node, map the DMA registers, and decode the DMA IRQs */ - np = of_parse_phandle(op->dev.of_node, "llink-connected", 0); - if (!np) { - dev_err(&op->dev, "could not find DMA node\n"); - rc = -ENODEV; - goto err_iounmap; - } - /* Setup the DMA register accesses, could be DCR or memory mapped */ - if (temac_dcr_setup(lp, op, np)) { + /* Setup LocalLink DMA */ + if (temac_np) { + /* Find the DMA node, map the DMA registers, and + * decode the DMA IRQs. + */ + dma_np = of_parse_phandle(temac_np, "llink-connected", 0); + if (!dma_np) { + dev_err(&pdev->dev, "could not find DMA node\n"); + return -ENODEV; + } - /* no DCR in the device tree, try non-DCR */ - lp->sdma_regs = of_iomap(np, 0); - if (lp->sdma_regs) { - lp->dma_in = temac_dma_in32; - lp->dma_out = temac_dma_out32; - dev_dbg(&op->dev, "MEM base: %p\n", lp->sdma_regs); - } else { - dev_err(&op->dev, "unable to map DMA registers\n"); - of_node_put(np); - goto err_iounmap; + /* Setup the DMA register accesses, could be DCR or + * memory mapped. + */ + if (temac_dcr_setup(lp, pdev, dma_np)) { + /* no DCR in the device tree, try non-DCR */ + lp->sdma_regs = devm_of_iomap(&pdev->dev, dma_np, 0, + NULL); + if (IS_ERR(lp->sdma_regs)) { + dev_err(&pdev->dev, + "unable to map DMA registers\n"); + of_node_put(dma_np); + return PTR_ERR(lp->sdma_regs); + } + if (of_get_property(dma_np, "little-endian", NULL)) { + lp->dma_in = temac_dma_in32_le; + lp->dma_out = temac_dma_out32_le; + } else { + lp->dma_in = temac_dma_in32_be; + lp->dma_out = temac_dma_out32_be; + } + dev_dbg(&pdev->dev, "MEM base: %p\n", lp->sdma_regs); } - } - lp->rx_irq = irq_of_parse_and_map(np, 0); - lp->tx_irq = irq_of_parse_and_map(np, 1); + /* Get DMA RX and TX interrupts */ + lp->rx_irq = irq_of_parse_and_map(dma_np, 0); + lp->tx_irq = irq_of_parse_and_map(dma_np, 1); - of_node_put(np); /* Finished with the DMA node; drop the reference */ + /* Use defaults for IRQ delay/coalescing setup. These + * are configuration values, so does not belong in + * device-tree. + */ + lp->tx_chnl_ctrl = 0x10220000; + lp->rx_chnl_ctrl = 0xff070000; + + /* Finished with the DMA node; drop the reference */ + of_node_put(dma_np); + } else if (pdata) { + /* 2nd memory resource specifies DMA registers */ + res = platform_get_resource(pdev, IORESOURCE_MEM, 1); + lp->sdma_regs = devm_ioremap_nocache(&pdev->dev, res->start, + resource_size(res)); + if (IS_ERR(lp->sdma_regs)) { + dev_err(&pdev->dev, + "could not map DMA registers\n"); + return PTR_ERR(lp->sdma_regs); + } + if (pdata->dma_little_endian) { + lp->dma_in = temac_dma_in32_le; + lp->dma_out = temac_dma_out32_le; + } else { + lp->dma_in = temac_dma_in32_be; + lp->dma_out = temac_dma_out32_be; + } - if (!lp->rx_irq || !lp->tx_irq) { - dev_err(&op->dev, "could not determine irqs\n"); - rc = -ENOMEM; - goto err_iounmap_2; + /* Get DMA RX and TX interrupts */ + lp->rx_irq = platform_get_irq(pdev, 0); + lp->tx_irq = platform_get_irq(pdev, 1); + + /* IRQ delay/coalescing setup */ + if (pdata->tx_irq_timeout || pdata->tx_irq_count) + lp->tx_chnl_ctrl = (pdata->tx_irq_timeout << 24) | + (pdata->tx_irq_count << 16); + else + lp->tx_chnl_ctrl = 0x10220000; + if (pdata->rx_irq_timeout || pdata->rx_irq_count) + lp->rx_chnl_ctrl = (pdata->rx_irq_timeout << 24) | + (pdata->rx_irq_count << 16); + else + lp->rx_chnl_ctrl = 0xff070000; } + /* Error handle returned DMA RX and TX interrupts */ + if (lp->rx_irq < 0) { + if (lp->rx_irq != -EPROBE_DEFER) + dev_err(&pdev->dev, "could not get DMA RX irq\n"); + return lp->rx_irq; + } + if (lp->tx_irq < 0) { + if (lp->tx_irq != -EPROBE_DEFER) + dev_err(&pdev->dev, "could not get DMA TX irq\n"); + return lp->tx_irq; + } - /* Retrieve the MAC address */ - addr = of_get_mac_address(op->dev.of_node); - if (!addr) { - dev_err(&op->dev, "could not find MAC address\n"); - rc = -ENODEV; - goto err_iounmap_2; + if (temac_np) { + /* Retrieve the MAC address */ + addr = of_get_mac_address(temac_np); + if (IS_ERR(addr)) { + dev_err(&pdev->dev, "could not find MAC address\n"); + return -ENODEV; + } + temac_init_mac_address(ndev, addr); + } else if (pdata) { + temac_init_mac_address(ndev, pdata->mac_addr); } - temac_init_mac_address(ndev, addr); - rc = temac_mdio_setup(lp, op->dev.of_node); + rc = temac_mdio_setup(lp, pdev); if (rc) - dev_warn(&op->dev, "error registering MDIO bus\n"); - - lp->phy_node = of_parse_phandle(op->dev.of_node, "phy-handle", 0); - if (lp->phy_node) - dev_dbg(lp->dev, "using PHY node %pOF (%p)\n", np, np); + dev_warn(&pdev->dev, "error registering MDIO bus\n"); + + if (temac_np) { + lp->phy_node = of_parse_phandle(temac_np, "phy-handle", 0); + if (lp->phy_node) + dev_dbg(lp->dev, "using PHY node %pOF\n", temac_np); + } else if (pdata) { + snprintf(lp->phy_name, sizeof(lp->phy_name), + PHY_ID_FMT, lp->mii_bus->id, pdata->phy_addr); + lp->phy_interface = pdata->phy_interface; + } /* Add the device attributes */ rc = sysfs_create_group(&lp->dev->kobj, &temac_attr_group); if (rc) { dev_err(lp->dev, "Error creating sysfs files\n"); - goto err_iounmap_2; + goto err_sysfs_create; } rc = register_netdev(lp->ndev); @@ -1107,33 +1296,25 @@ static int temac_of_probe(struct platform_device *op) return 0; - err_register_ndev: +err_register_ndev: sysfs_remove_group(&lp->dev->kobj, &temac_attr_group); - err_iounmap_2: - if (lp->sdma_regs) - iounmap(lp->sdma_regs); - err_iounmap: - iounmap(lp->regs); - nodev: - free_netdev(ndev); - ndev = NULL; +err_sysfs_create: + if (lp->phy_node) + of_node_put(lp->phy_node); + temac_mdio_teardown(lp); return rc; } -static int temac_of_remove(struct platform_device *op) +static int temac_remove(struct platform_device *pdev) { - struct net_device *ndev = platform_get_drvdata(op); + struct net_device *ndev = platform_get_drvdata(pdev); struct temac_local *lp = netdev_priv(ndev); - temac_mdio_teardown(lp); unregister_netdev(ndev); sysfs_remove_group(&lp->dev->kobj, &temac_attr_group); - of_node_put(lp->phy_node); - lp->phy_node = NULL; - iounmap(lp->regs); - if (lp->sdma_regs) - iounmap(lp->sdma_regs); - free_netdev(ndev); + if (lp->phy_node) + of_node_put(lp->phy_node); + temac_mdio_teardown(lp); return 0; } @@ -1146,16 +1327,16 @@ static const struct of_device_id temac_of_match[] = { }; MODULE_DEVICE_TABLE(of, temac_of_match); -static struct platform_driver temac_of_driver = { - .probe = temac_of_probe, - .remove = temac_of_remove, +static struct platform_driver temac_driver = { + .probe = temac_probe, + .remove = temac_remove, .driver = { .name = "xilinx_temac", .of_match_table = temac_of_match, }, }; -module_platform_driver(temac_of_driver); +module_platform_driver(temac_driver); MODULE_DESCRIPTION("Xilinx LL_TEMAC Ethernet driver"); MODULE_AUTHOR("Yoshio Kashiwagi"); diff --git a/drivers/net/ethernet/xilinx/ll_temac_mdio.c b/drivers/net/ethernet/xilinx/ll_temac_mdio.c index f5e83ac6f7e2fa6bb1bcde1f24e5a4e549c92522..a4667326f745350fc90a121cf7672a9467c9ca8d 100644 --- a/drivers/net/ethernet/xilinx/ll_temac_mdio.c +++ b/drivers/net/ethernet/xilinx/ll_temac_mdio.c @@ -14,6 +14,7 @@ #include #include #include +#include #include "ll_temac.h" @@ -28,10 +29,10 @@ static int temac_mdio_read(struct mii_bus *bus, int phy_id, int reg) /* Write the PHY address to the MIIM Access Initiator register. * When the transfer completes, the PHY register value will appear * in the LSW0 register */ - mutex_lock(&lp->indirect_mutex); + mutex_lock(lp->indirect_mutex); temac_iow(lp, XTE_LSW0_OFFSET, (phy_id << 5) | reg); rc = temac_indirect_in32(lp, XTE_MIIMAI_OFFSET); - mutex_unlock(&lp->indirect_mutex); + mutex_unlock(lp->indirect_mutex); dev_dbg(lp->dev, "temac_mdio_read(phy_id=%i, reg=%x) == %x\n", phy_id, reg, rc); @@ -49,25 +50,34 @@ static int temac_mdio_write(struct mii_bus *bus, int phy_id, int reg, u16 val) /* First write the desired value into the write data register * and then write the address into the access initiator register */ - mutex_lock(&lp->indirect_mutex); + mutex_lock(lp->indirect_mutex); temac_indirect_out32(lp, XTE_MGTDR_OFFSET, val); temac_indirect_out32(lp, XTE_MIIMAI_OFFSET, (phy_id << 5) | reg); - mutex_unlock(&lp->indirect_mutex); + mutex_unlock(lp->indirect_mutex); return 0; } -int temac_mdio_setup(struct temac_local *lp, struct device_node *np) +int temac_mdio_setup(struct temac_local *lp, struct platform_device *pdev) { + struct ll_temac_platform_data *pdata = dev_get_platdata(&pdev->dev); + struct device_node *np = dev_of_node(&pdev->dev); struct mii_bus *bus; u32 bus_hz; int clk_div; int rc; struct resource res; + /* Get MDIO bus frequency (if specified) */ + bus_hz = 0; + if (np) + of_property_read_u32(np, "clock-frequency", &bus_hz); + else if (pdata) + bus_hz = pdata->mdio_clk_freq; + /* Calculate a reasonable divisor for the clock rate */ clk_div = 0x3f; /* worst-case default setting */ - if (of_property_read_u32(np, "clock-frequency", &bus_hz) == 0) { + if (bus_hz != 0) { clk_div = bus_hz / (2500 * 1000 * 2) - 1; if (clk_div < 1) clk_div = 1; @@ -77,17 +87,23 @@ int temac_mdio_setup(struct temac_local *lp, struct device_node *np) /* Enable the MDIO bus by asserting the enable bit and writing * in the clock config */ - mutex_lock(&lp->indirect_mutex); + mutex_lock(lp->indirect_mutex); temac_indirect_out32(lp, XTE_MC_OFFSET, 1 << 6 | clk_div); - mutex_unlock(&lp->indirect_mutex); + mutex_unlock(lp->indirect_mutex); - bus = mdiobus_alloc(); + bus = devm_mdiobus_alloc(&pdev->dev); if (!bus) return -ENOMEM; - of_address_to_resource(np, 0, &res); - snprintf(bus->id, MII_BUS_ID_SIZE, "%.8llx", - (unsigned long long)res.start); + if (np) { + of_address_to_resource(np, 0, &res); + snprintf(bus->id, MII_BUS_ID_SIZE, "%.8llx", + (unsigned long long)res.start); + } else if (pdata) { + snprintf(bus->id, MII_BUS_ID_SIZE, "%.8llx", + pdata->mdio_bus_id); + } + bus->priv = lp; bus->name = "Xilinx TEMAC MDIO"; bus->read = temac_mdio_read; @@ -98,23 +114,16 @@ int temac_mdio_setup(struct temac_local *lp, struct device_node *np) rc = of_mdiobus_register(bus, np); if (rc) - goto err_register; + return rc; - mutex_lock(&lp->indirect_mutex); + mutex_lock(lp->indirect_mutex); dev_dbg(lp->dev, "MDIO bus registered; MC:%x\n", temac_indirect_in32(lp, XTE_MC_OFFSET)); - mutex_unlock(&lp->indirect_mutex); + mutex_unlock(lp->indirect_mutex); return 0; - - err_register: - mdiobus_free(bus); - return rc; } void temac_mdio_teardown(struct temac_local *lp) { mdiobus_unregister(lp->mii_bus); - mdiobus_free(lp->mii_bus); - lp->mii_bus = NULL; } - diff --git a/drivers/net/ethernet/xilinx/xilinx_axienet_main.c b/drivers/net/ethernet/xilinx/xilinx_axienet_main.c index 4041c75997ba5ed52803b69b4c2e19bf41dee8e8..108fbc7f125a2fabfc2380d745ff1027b50475e2 100644 --- a/drivers/net/ethernet/xilinx/xilinx_axienet_main.c +++ b/drivers/net/ethernet/xilinx/xilinx_axienet_main.c @@ -1596,7 +1596,7 @@ static int axienet_probe(struct platform_device *pdev) /* Retrieve the MAC address */ mac_addr = of_get_mac_address(pdev->dev.of_node); - if (!mac_addr) { + if (IS_ERR(mac_addr)) { dev_err(&pdev->dev, "could not find MAC address\n"); goto free_netdev; } diff --git a/drivers/net/ethernet/xilinx/xilinx_emaclite.c b/drivers/net/ethernet/xilinx/xilinx_emaclite.c index b03a417d0073ebc033fc442acfa9b2523366ed94..69117075356327aa9904dc26f2ab8064dd9dd3fb 100644 --- a/drivers/net/ethernet/xilinx/xilinx_emaclite.c +++ b/drivers/net/ethernet/xilinx/xilinx_emaclite.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include #include @@ -1078,6 +1079,27 @@ static bool get_bool(struct platform_device *ofdev, const char *s) return (bool)*p; } +/** + * xemaclite_ethtools_get_drvinfo - Get various Axi Emac Lite driver info + * @ndev: Pointer to net_device structure + * @ed: Pointer to ethtool_drvinfo structure + * + * This implements ethtool command for getting the driver information. + * Issue "ethtool -i ethX" under linux prompt to execute this function. + */ +static void xemaclite_ethtools_get_drvinfo(struct net_device *ndev, + struct ethtool_drvinfo *ed) +{ + strlcpy(ed->driver, DRIVER_NAME, sizeof(ed->driver)); +} + +static const struct ethtool_ops xemaclite_ethtool_ops = { + .get_drvinfo = xemaclite_ethtools_get_drvinfo, + .get_link = ethtool_op_get_link, + .get_link_ksettings = phy_ethtool_get_link_ksettings, + .set_link_ksettings = phy_ethtool_set_link_ksettings, +}; + static const struct net_device_ops xemaclite_netdev_ops; /** @@ -1143,7 +1165,7 @@ static int xemaclite_of_probe(struct platform_device *ofdev) lp->rx_ping_pong = get_bool(ofdev, "xlnx,rx-ping-pong"); mac_address = of_get_mac_address(ofdev->dev.of_node); - if (mac_address) { + if (!IS_ERR(mac_address)) { /* Set the MAC address. */ memcpy(ndev->dev_addr, mac_address, ETH_ALEN); } else { @@ -1164,6 +1186,7 @@ static int xemaclite_of_probe(struct platform_device *ofdev) dev_info(dev, "MAC address is now %pM\n", ndev->dev_addr); ndev->netdev_ops = &xemaclite_netdev_ops; + ndev->ethtool_ops = &xemaclite_ethtool_ops; ndev->flags &= ~IFF_MULTICAST; ndev->watchdog_timeo = TX_TIMEOUT; @@ -1229,12 +1252,29 @@ xemaclite_poll_controller(struct net_device *ndev) } #endif +/* Ioctl MII Interface */ +static int xemaclite_ioctl(struct net_device *dev, struct ifreq *rq, int cmd) +{ + if (!dev->phydev || !netif_running(dev)) + return -EINVAL; + + switch (cmd) { + case SIOCGMIIPHY: + case SIOCGMIIREG: + case SIOCSMIIREG: + return phy_mii_ioctl(dev->phydev, rq, cmd); + default: + return -EOPNOTSUPP; + } +} + static const struct net_device_ops xemaclite_netdev_ops = { .ndo_open = xemaclite_open, .ndo_stop = xemaclite_close, .ndo_start_xmit = xemaclite_send, .ndo_set_mac_address = xemaclite_set_mac_address, .ndo_tx_timeout = xemaclite_tx_timeout, + .ndo_do_ioctl = xemaclite_ioctl, #ifdef CONFIG_NET_POLL_CONTROLLER .ndo_poll_controller = xemaclite_poll_controller, #endif diff --git a/drivers/net/geneve.c b/drivers/net/geneve.c index 5583d993480d2c11ed17afa189385627dcbad17b..98d1a45c06067d7aeaae5a31465a3cb036c7e080 100644 --- a/drivers/net/geneve.c +++ b/drivers/net/geneve.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -22,8 +23,6 @@ #define GENEVE_NETDEV_VER "0.6" -#define GENEVE_UDP_PORT 6081 - #define GENEVE_N_VID (1u << 24) #define GENEVE_VID_MASK (GENEVE_N_VID - 1) diff --git a/drivers/net/gtp.c b/drivers/net/gtp.c index 7a145172d50385f56f54e4b114ab6194fd672ca5..eaf4311b4004e6513d0b4a4a299724d78a9a8efd 100644 --- a/drivers/net/gtp.c +++ b/drivers/net/gtp.c @@ -1270,21 +1270,21 @@ static const struct nla_policy gtp_genl_policy[GTPA_MAX + 1] = { static const struct genl_ops gtp_genl_ops[] = { { .cmd = GTP_CMD_NEWPDP, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = gtp_genl_new_pdp, - .policy = gtp_genl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = GTP_CMD_DELPDP, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = gtp_genl_del_pdp, - .policy = gtp_genl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = GTP_CMD_GETPDP, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = gtp_genl_get_pdp, .dumpit = gtp_genl_dump_pdp, - .policy = gtp_genl_policy, .flags = GENL_ADMIN_PERM, }, }; @@ -1294,6 +1294,7 @@ static struct genl_family gtp_genl_family __ro_after_init = { .version = 0, .hdrsize = 0, .maxattr = GTPA_MAX, + .policy = gtp_genl_policy, .netnsok = true, .module = THIS_MODULE, .ops = gtp_genl_ops, diff --git a/drivers/net/hippi/rrunner.c b/drivers/net/hippi/rrunner.c index 029206e4da3b3719084ccdc4cbc7486313237cff..0f7025f3a384a8b92e9a6ae4c30f4b024853edc0 100644 --- a/drivers/net/hippi/rrunner.c +++ b/drivers/net/hippi/rrunner.c @@ -1298,11 +1298,11 @@ static void rr_dump(struct net_device *dev) if (rrpriv->tx_skbuff[cons]){ len = min_t(int, 0x80, rrpriv->tx_skbuff[cons]->len); printk("skbuff for cons %i is valid - dumping data (0x%x bytes - skbuff len 0x%x)\n", cons, len, rrpriv->tx_skbuff[cons]->len); - printk("mode 0x%x, size 0x%x,\n phys %08Lx, skbuff-addr %08lx, truesize 0x%x\n", + printk("mode 0x%x, size 0x%x,\n phys %08Lx, skbuff-addr %p, truesize 0x%x\n", rrpriv->tx_ring[cons].mode, rrpriv->tx_ring[cons].size, (unsigned long long) rrpriv->tx_ring[cons].addr.addrlo, - (unsigned long)rrpriv->tx_skbuff[cons]->data, + rrpriv->tx_skbuff[cons]->data, (unsigned int)rrpriv->tx_skbuff[cons]->truesize); for (i = 0; i < len; i++){ if (!(i & 7)) diff --git a/drivers/net/hyperv/netvsc.c b/drivers/net/hyperv/netvsc.c index e0dce373cdd9d875ded78bff545d76b32920f69c..ee198606854dbfa50d341c5d8a48dafe751483e2 100644 --- a/drivers/net/hyperv/netvsc.c +++ b/drivers/net/hyperv/netvsc.c @@ -875,12 +875,6 @@ static inline int netvsc_send_pkt( } else if (ret == -EAGAIN) { netif_tx_stop_queue(txq); ndev_ctx->eth_stats.stop_queue++; - if (atomic_read(&nvchan->queue_sends) < 1 && - !net_device->tx_disable) { - netif_tx_wake_queue(txq); - ndev_ctx->eth_stats.wake_queue++; - ret = -ENOSPC; - } } else { netdev_err(ndev, "Unable to send packet pages %u len %u, ret %d\n", @@ -888,6 +882,15 @@ static inline int netvsc_send_pkt( ret); } + if (netif_tx_queue_stopped(txq) && + atomic_read(&nvchan->queue_sends) < 1 && + !net_device->tx_disable) { + netif_tx_wake_queue(txq); + ndev_ctx->eth_stats.wake_queue++; + if (ret == -EAGAIN) + ret = -ENOSPC; + } + return ret; } @@ -966,7 +969,7 @@ int netvsc_send(struct net_device *ndev, /* Keep aggregating only if stack says more data is coming * and not doing mixed modes send and not flow blocked */ - xmit_more = skb->xmit_more && + xmit_more = netdev_xmit_more() && !packet->cp_partial && !netif_xmit_stopped(netdev_get_tx_queue(ndev, packet->q_idx)); diff --git a/drivers/net/hyperv/netvsc_drv.c b/drivers/net/hyperv/netvsc_drv.c index b20fb0fb595bde3d7455e2c6214812fef6732017..06393b2151021d4cafa4f7308f4070ad7f741932 100644 --- a/drivers/net/hyperv/netvsc_drv.c +++ b/drivers/net/hyperv/netvsc_drv.c @@ -328,7 +328,7 @@ static inline int netvsc_get_tx_queue(struct net_device *ndev, * If a valid queue has already been assigned, then use that. * Otherwise compute tx queue based on hash and the send table. * - * This is basically similar to default (__netdev_pick_tx) with the added step + * This is basically similar to default (netdev_pick_tx) with the added step * of using the host send_table when no other queue has been assigned. * * TODO support XPS - but get_xps_queue not exported @@ -351,8 +351,7 @@ static u16 netvsc_pick_tx(struct net_device *ndev, struct sk_buff *skb) } static u16 netvsc_select_queue(struct net_device *ndev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { struct net_device_context *ndc = netdev_priv(ndev); struct net_device *vf_netdev; @@ -364,10 +363,9 @@ static u16 netvsc_select_queue(struct net_device *ndev, struct sk_buff *skb, const struct net_device_ops *vf_ops = vf_netdev->netdev_ops; if (vf_ops->ndo_select_queue) - txq = vf_ops->ndo_select_queue(vf_netdev, skb, - sb_dev, fallback); + txq = vf_ops->ndo_select_queue(vf_netdev, skb, sb_dev); else - txq = fallback(vf_netdev, skb, NULL); + txq = netdev_pick_tx(vf_netdev, skb, NULL); /* Record the queue selected by VF so that it can be * used for common case where VF has more queues than diff --git a/drivers/net/ieee802154/mac802154_hwsim.c b/drivers/net/ieee802154/mac802154_hwsim.c index 3b88846de31b18236e423c3b941f69537ab3bcfa..b187ae1a6bd6172771196cb6a6b736cfbdde2ff7 100644 --- a/drivers/net/ieee802154/mac802154_hwsim.c +++ b/drivers/net/ieee802154/mac802154_hwsim.c @@ -227,14 +227,16 @@ static int append_radio_msg(struct sk_buff *skb, struct hwsim_phy *phy) return 0; } - nl_edges = nla_nest_start(skb, MAC802154_HWSIM_ATTR_RADIO_EDGES); + nl_edges = nla_nest_start_noflag(skb, + MAC802154_HWSIM_ATTR_RADIO_EDGES); if (!nl_edges) { rcu_read_unlock(); return -ENOBUFS; } list_for_each_entry_rcu(e, &phy->edges, list) { - nl_edge = nla_nest_start(skb, MAC802154_HWSIM_ATTR_RADIO_EDGE); + nl_edge = nla_nest_start_noflag(skb, + MAC802154_HWSIM_ATTR_RADIO_EDGE); if (!nl_edge) { rcu_read_unlock(); nla_nest_cancel(skb, nl_edges); @@ -428,9 +430,7 @@ static int hwsim_new_edge_nl(struct sk_buff *msg, struct genl_info *info) !info->attrs[MAC802154_HWSIM_ATTR_RADIO_EDGE]) return -EINVAL; - if (nla_parse_nested(edge_attrs, MAC802154_HWSIM_EDGE_ATTR_MAX, - info->attrs[MAC802154_HWSIM_ATTR_RADIO_EDGE], - hwsim_edge_policy, NULL)) + if (nla_parse_nested_deprecated(edge_attrs, MAC802154_HWSIM_EDGE_ATTR_MAX, info->attrs[MAC802154_HWSIM_ATTR_RADIO_EDGE], hwsim_edge_policy, NULL)) return -EINVAL; if (!edge_attrs[MAC802154_HWSIM_EDGE_ATTR_ENDPOINT_ID]) @@ -492,9 +492,7 @@ static int hwsim_del_edge_nl(struct sk_buff *msg, struct genl_info *info) !info->attrs[MAC802154_HWSIM_ATTR_RADIO_EDGE]) return -EINVAL; - if (nla_parse_nested(edge_attrs, MAC802154_HWSIM_EDGE_ATTR_MAX, - info->attrs[MAC802154_HWSIM_ATTR_RADIO_EDGE], - hwsim_edge_policy, NULL)) + if (nla_parse_nested_deprecated(edge_attrs, MAC802154_HWSIM_EDGE_ATTR_MAX, info->attrs[MAC802154_HWSIM_ATTR_RADIO_EDGE], hwsim_edge_policy, NULL)) return -EINVAL; if (!edge_attrs[MAC802154_HWSIM_EDGE_ATTR_ENDPOINT_ID]) @@ -542,9 +540,7 @@ static int hwsim_set_edge_lqi(struct sk_buff *msg, struct genl_info *info) !info->attrs[MAC802154_HWSIM_ATTR_RADIO_EDGE]) return -EINVAL; - if (nla_parse_nested(edge_attrs, MAC802154_HWSIM_EDGE_ATTR_MAX, - info->attrs[MAC802154_HWSIM_ATTR_RADIO_EDGE], - hwsim_edge_policy, NULL)) + if (nla_parse_nested_deprecated(edge_attrs, MAC802154_HWSIM_EDGE_ATTR_MAX, info->attrs[MAC802154_HWSIM_ATTR_RADIO_EDGE], hwsim_edge_policy, NULL)) return -EINVAL; if (!edge_attrs[MAC802154_HWSIM_EDGE_ATTR_ENDPOINT_ID] && @@ -598,37 +594,37 @@ static const struct nla_policy hwsim_genl_policy[MAC802154_HWSIM_ATTR_MAX + 1] = static const struct genl_ops hwsim_nl_ops[] = { { .cmd = MAC802154_HWSIM_CMD_NEW_RADIO, - .policy = hwsim_genl_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = hwsim_new_radio_nl, .flags = GENL_UNS_ADMIN_PERM, }, { .cmd = MAC802154_HWSIM_CMD_DEL_RADIO, - .policy = hwsim_genl_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = hwsim_del_radio_nl, .flags = GENL_UNS_ADMIN_PERM, }, { .cmd = MAC802154_HWSIM_CMD_GET_RADIO, - .policy = hwsim_genl_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = hwsim_get_radio_nl, .dumpit = hwsim_dump_radio_nl, }, { .cmd = MAC802154_HWSIM_CMD_NEW_EDGE, - .policy = hwsim_genl_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = hwsim_new_edge_nl, .flags = GENL_UNS_ADMIN_PERM, }, { .cmd = MAC802154_HWSIM_CMD_DEL_EDGE, - .policy = hwsim_genl_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = hwsim_del_edge_nl, .flags = GENL_UNS_ADMIN_PERM, }, { .cmd = MAC802154_HWSIM_CMD_SET_EDGE, - .policy = hwsim_genl_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = hwsim_set_edge_lqi, .flags = GENL_UNS_ADMIN_PERM, }, @@ -638,6 +634,7 @@ static struct genl_family hwsim_genl_family __ro_after_init = { .name = "MAC802154_HWSIM", .version = 1, .maxattr = MAC802154_HWSIM_ATTR_MAX, + .policy = hwsim_genl_policy, .module = THIS_MODULE, .ops = hwsim_nl_ops, .n_ops = ARRAY_SIZE(hwsim_nl_ops), diff --git a/drivers/net/loopback.c b/drivers/net/loopback.c index 2df7f60fe05220c19896a251b6b15239f4b95112..857e4bf998835eaeb6c4bf776dd77adabb0f36c9 100644 --- a/drivers/net/loopback.c +++ b/drivers/net/loopback.c @@ -128,21 +128,9 @@ static u32 always_on(struct net_device *dev) return 1; } -static int loopback_get_ts_info(struct net_device *netdev, - struct ethtool_ts_info *ts_info) -{ - ts_info->so_timestamping = SOF_TIMESTAMPING_TX_SOFTWARE | - SOF_TIMESTAMPING_RX_SOFTWARE | - SOF_TIMESTAMPING_SOFTWARE; - - ts_info->phc_index = -1; - - return 0; -}; - static const struct ethtool_ops loopback_ethtool_ops = { .get_link = always_on, - .get_ts_info = loopback_get_ts_info, + .get_ts_info = ethtool_op_get_ts_info, }; static int loopback_dev_init(struct net_device *dev) diff --git a/drivers/net/macsec.c b/drivers/net/macsec.c index 64a982563d59171e9220243a5d5eac9bed286529..009b2902c9d349b03995c7826734bd1b15e2704d 100644 --- a/drivers/net/macsec.c +++ b/drivers/net/macsec.c @@ -1611,9 +1611,7 @@ static int parse_sa_config(struct nlattr **attrs, struct nlattr **tb_sa) if (!attrs[MACSEC_ATTR_SA_CONFIG]) return -EINVAL; - if (nla_parse_nested(tb_sa, MACSEC_SA_ATTR_MAX, - attrs[MACSEC_ATTR_SA_CONFIG], - macsec_genl_sa_policy, NULL)) + if (nla_parse_nested_deprecated(tb_sa, MACSEC_SA_ATTR_MAX, attrs[MACSEC_ATTR_SA_CONFIG], macsec_genl_sa_policy, NULL)) return -EINVAL; return 0; @@ -1624,9 +1622,7 @@ static int parse_rxsc_config(struct nlattr **attrs, struct nlattr **tb_rxsc) if (!attrs[MACSEC_ATTR_RXSC_CONFIG]) return -EINVAL; - if (nla_parse_nested(tb_rxsc, MACSEC_RXSC_ATTR_MAX, - attrs[MACSEC_ATTR_RXSC_CONFIG], - macsec_genl_rxsc_policy, NULL)) + if (nla_parse_nested_deprecated(tb_rxsc, MACSEC_RXSC_ATTR_MAX, attrs[MACSEC_ATTR_RXSC_CONFIG], macsec_genl_rxsc_policy, NULL)) return -EINVAL; return 0; @@ -2175,8 +2171,9 @@ static int copy_tx_sa_stats(struct sk_buff *skb, return 0; } -static int copy_rx_sa_stats(struct sk_buff *skb, - struct macsec_rx_sa_stats __percpu *pstats) +static noinline_for_stack int +copy_rx_sa_stats(struct sk_buff *skb, + struct macsec_rx_sa_stats __percpu *pstats) { struct macsec_rx_sa_stats sum = {0, }; int cpu; @@ -2201,8 +2198,8 @@ static int copy_rx_sa_stats(struct sk_buff *skb, return 0; } -static int copy_rx_sc_stats(struct sk_buff *skb, - struct pcpu_rx_sc_stats __percpu *pstats) +static noinline_for_stack int +copy_rx_sc_stats(struct sk_buff *skb, struct pcpu_rx_sc_stats __percpu *pstats) { struct macsec_rx_sc_stats sum = {0, }; int cpu; @@ -2265,8 +2262,8 @@ static int copy_rx_sc_stats(struct sk_buff *skb, return 0; } -static int copy_tx_sc_stats(struct sk_buff *skb, - struct pcpu_tx_sc_stats __percpu *pstats) +static noinline_for_stack int +copy_tx_sc_stats(struct sk_buff *skb, struct pcpu_tx_sc_stats __percpu *pstats) { struct macsec_tx_sc_stats sum = {0, }; int cpu; @@ -2305,8 +2302,8 @@ static int copy_tx_sc_stats(struct sk_buff *skb, return 0; } -static int copy_secy_stats(struct sk_buff *skb, - struct pcpu_secy_stats __percpu *pstats) +static noinline_for_stack int +copy_secy_stats(struct sk_buff *skb, struct pcpu_secy_stats __percpu *pstats) { struct macsec_dev_stats sum = {0, }; int cpu; @@ -2364,7 +2361,8 @@ static int copy_secy_stats(struct sk_buff *skb, static int nla_put_secy(struct macsec_secy *secy, struct sk_buff *skb) { struct macsec_tx_sc *tx_sc = &secy->tx_sc; - struct nlattr *secy_nest = nla_nest_start(skb, MACSEC_ATTR_SECY); + struct nlattr *secy_nest = nla_nest_start_noflag(skb, + MACSEC_ATTR_SECY); u64 csid; if (!secy_nest) @@ -2410,8 +2408,9 @@ cancel: return 1; } -static int dump_secy(struct macsec_secy *secy, struct net_device *dev, - struct sk_buff *skb, struct netlink_callback *cb) +static noinline_for_stack int +dump_secy(struct macsec_secy *secy, struct net_device *dev, + struct sk_buff *skb, struct netlink_callback *cb) { struct macsec_rx_sc *rx_sc; struct macsec_tx_sc *tx_sc = &secy->tx_sc; @@ -2433,7 +2432,7 @@ static int dump_secy(struct macsec_secy *secy, struct net_device *dev, if (nla_put_secy(secy, skb)) goto nla_put_failure; - attr = nla_nest_start(skb, MACSEC_ATTR_TXSC_STATS); + attr = nla_nest_start_noflag(skb, MACSEC_ATTR_TXSC_STATS); if (!attr) goto nla_put_failure; if (copy_tx_sc_stats(skb, tx_sc->stats)) { @@ -2442,7 +2441,7 @@ static int dump_secy(struct macsec_secy *secy, struct net_device *dev, } nla_nest_end(skb, attr); - attr = nla_nest_start(skb, MACSEC_ATTR_SECY_STATS); + attr = nla_nest_start_noflag(skb, MACSEC_ATTR_SECY_STATS); if (!attr) goto nla_put_failure; if (copy_secy_stats(skb, macsec_priv(dev)->stats)) { @@ -2451,7 +2450,7 @@ static int dump_secy(struct macsec_secy *secy, struct net_device *dev, } nla_nest_end(skb, attr); - txsa_list = nla_nest_start(skb, MACSEC_ATTR_TXSA_LIST); + txsa_list = nla_nest_start_noflag(skb, MACSEC_ATTR_TXSA_LIST); if (!txsa_list) goto nla_put_failure; for (i = 0, j = 1; i < MACSEC_NUM_AN; i++) { @@ -2461,7 +2460,7 @@ static int dump_secy(struct macsec_secy *secy, struct net_device *dev, if (!tx_sa) continue; - txsa_nest = nla_nest_start(skb, j++); + txsa_nest = nla_nest_start_noflag(skb, j++); if (!txsa_nest) { nla_nest_cancel(skb, txsa_list); goto nla_put_failure; @@ -2476,7 +2475,7 @@ static int dump_secy(struct macsec_secy *secy, struct net_device *dev, goto nla_put_failure; } - attr = nla_nest_start(skb, MACSEC_SA_ATTR_STATS); + attr = nla_nest_start_noflag(skb, MACSEC_SA_ATTR_STATS); if (!attr) { nla_nest_cancel(skb, txsa_nest); nla_nest_cancel(skb, txsa_list); @@ -2494,7 +2493,7 @@ static int dump_secy(struct macsec_secy *secy, struct net_device *dev, } nla_nest_end(skb, txsa_list); - rxsc_list = nla_nest_start(skb, MACSEC_ATTR_RXSC_LIST); + rxsc_list = nla_nest_start_noflag(skb, MACSEC_ATTR_RXSC_LIST); if (!rxsc_list) goto nla_put_failure; @@ -2502,7 +2501,7 @@ static int dump_secy(struct macsec_secy *secy, struct net_device *dev, for_each_rxsc_rtnl(secy, rx_sc) { int k; struct nlattr *rxsa_list; - struct nlattr *rxsc_nest = nla_nest_start(skb, j++); + struct nlattr *rxsc_nest = nla_nest_start_noflag(skb, j++); if (!rxsc_nest) { nla_nest_cancel(skb, rxsc_list); @@ -2517,7 +2516,7 @@ static int dump_secy(struct macsec_secy *secy, struct net_device *dev, goto nla_put_failure; } - attr = nla_nest_start(skb, MACSEC_RXSC_ATTR_STATS); + attr = nla_nest_start_noflag(skb, MACSEC_RXSC_ATTR_STATS); if (!attr) { nla_nest_cancel(skb, rxsc_nest); nla_nest_cancel(skb, rxsc_list); @@ -2531,7 +2530,8 @@ static int dump_secy(struct macsec_secy *secy, struct net_device *dev, } nla_nest_end(skb, attr); - rxsa_list = nla_nest_start(skb, MACSEC_RXSC_ATTR_SA_LIST); + rxsa_list = nla_nest_start_noflag(skb, + MACSEC_RXSC_ATTR_SA_LIST); if (!rxsa_list) { nla_nest_cancel(skb, rxsc_nest); nla_nest_cancel(skb, rxsc_list); @@ -2545,7 +2545,7 @@ static int dump_secy(struct macsec_secy *secy, struct net_device *dev, if (!rx_sa) continue; - rxsa_nest = nla_nest_start(skb, k++); + rxsa_nest = nla_nest_start_noflag(skb, k++); if (!rxsa_nest) { nla_nest_cancel(skb, rxsa_list); nla_nest_cancel(skb, rxsc_nest); @@ -2553,7 +2553,8 @@ static int dump_secy(struct macsec_secy *secy, struct net_device *dev, goto nla_put_failure; } - attr = nla_nest_start(skb, MACSEC_SA_ATTR_STATS); + attr = nla_nest_start_noflag(skb, + MACSEC_SA_ATTR_STATS); if (!attr) { nla_nest_cancel(skb, rxsa_list); nla_nest_cancel(skb, rxsc_nest); @@ -2636,61 +2637,61 @@ done: static const struct genl_ops macsec_genl_ops[] = { { .cmd = MACSEC_CMD_GET_TXSC, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .dumpit = macsec_dump_txsc, - .policy = macsec_genl_policy, }, { .cmd = MACSEC_CMD_ADD_RXSC, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = macsec_add_rxsc, - .policy = macsec_genl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = MACSEC_CMD_DEL_RXSC, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = macsec_del_rxsc, - .policy = macsec_genl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = MACSEC_CMD_UPD_RXSC, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = macsec_upd_rxsc, - .policy = macsec_genl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = MACSEC_CMD_ADD_TXSA, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = macsec_add_txsa, - .policy = macsec_genl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = MACSEC_CMD_DEL_TXSA, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = macsec_del_txsa, - .policy = macsec_genl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = MACSEC_CMD_UPD_TXSA, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = macsec_upd_txsa, - .policy = macsec_genl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = MACSEC_CMD_ADD_RXSA, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = macsec_add_rxsa, - .policy = macsec_genl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = MACSEC_CMD_DEL_RXSA, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = macsec_del_rxsa, - .policy = macsec_genl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = MACSEC_CMD_UPD_RXSA, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = macsec_upd_rxsa, - .policy = macsec_genl_policy, .flags = GENL_ADMIN_PERM, }, }; @@ -2700,6 +2701,7 @@ static struct genl_family macsec_fam __ro_after_init = { .hdrsize = 0, .version = MACSEC_GENL_VERSION, .maxattr = MACSEC_ATTR_MAX, + .policy = macsec_genl_policy, .netnsok = true, .module = THIS_MODULE, .ops = macsec_genl_ops, diff --git a/drivers/net/macvlan.c b/drivers/net/macvlan.c index 0c0f105657d3b57486b2b9c94d33771769e0ccd8..b395423b19bcef0f434ed37ea9713e22909c6a1f 100644 --- a/drivers/net/macvlan.c +++ b/drivers/net/macvlan.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -34,6 +35,7 @@ #include #include #include +#include #define MACVLAN_HASH_BITS 8 #define MACVLAN_HASH_SIZE (1<netdev_ops; + struct ifreq ifrr; + int err = -EOPNOTSUPP; + + strncpy(ifrr.ifr_name, real_dev->name, IFNAMSIZ); + ifrr.ifr_ifru = ifr->ifr_ifru; + + switch (cmd) { + case SIOCSHWTSTAMP: + case SIOCGHWTSTAMP: + if (netif_device_present(real_dev) && ops->ndo_do_ioctl) + err = ops->ndo_do_ioctl(real_dev, &ifrr, cmd); + break; + } + + if (!err) + ifr->ifr_ifru = ifrr.ifr_ifru; + + return err; +} + /* * macvlan network devices have devices nesting below it and are a special * "super class" of normal network devices; split their locks off into a @@ -1020,6 +1046,26 @@ static int macvlan_ethtool_get_link_ksettings(struct net_device *dev, return __ethtool_get_link_ksettings(vlan->lowerdev, cmd); } +static int macvlan_ethtool_get_ts_info(struct net_device *dev, + struct ethtool_ts_info *info) +{ + struct net_device *real_dev = macvlan_dev_real_dev(dev); + const struct ethtool_ops *ops = real_dev->ethtool_ops; + struct phy_device *phydev = real_dev->phydev; + + if (phydev && phydev->drv && phydev->drv->ts_info) { + return phydev->drv->ts_info(phydev, info); + } else if (ops->get_ts_info) { + return ops->get_ts_info(real_dev, info); + } else { + info->so_timestamping = SOF_TIMESTAMPING_RX_SOFTWARE | + SOF_TIMESTAMPING_SOFTWARE; + info->phc_index = -1; + } + + return 0; +} + static netdev_features_t macvlan_fix_features(struct net_device *dev, netdev_features_t features) { @@ -1094,6 +1140,7 @@ static const struct ethtool_ops macvlan_ethtool_ops = { .get_link = ethtool_op_get_link, .get_link_ksettings = macvlan_ethtool_get_link_ksettings, .get_drvinfo = macvlan_ethtool_get_drvinfo, + .get_ts_info = macvlan_ethtool_get_ts_info, }; static const struct net_device_ops macvlan_netdev_ops = { @@ -1103,6 +1150,7 @@ static const struct net_device_ops macvlan_netdev_ops = { .ndo_stop = macvlan_stop, .ndo_start_xmit = macvlan_start_xmit, .ndo_change_mtu = macvlan_change_mtu, + .ndo_do_ioctl = macvlan_do_ioctl, .ndo_fix_features = macvlan_fix_features, .ndo_change_rx_flags = macvlan_change_rx_flags, .ndo_set_mac_address = macvlan_set_mac_address, @@ -1576,7 +1624,7 @@ static int macvlan_fill_info(struct sk_buff *skb, if (nla_put_u32(skb, IFLA_MACVLAN_MACADDR_COUNT, vlan->macaddr_count)) goto nla_put_failure; if (vlan->macaddr_count > 0) { - nest = nla_nest_start(skb, IFLA_MACVLAN_MACADDR_DATA); + nest = nla_nest_start_noflag(skb, IFLA_MACVLAN_MACADDR_DATA); if (nest == NULL) goto nla_put_failure; diff --git a/drivers/net/net_failover.c b/drivers/net/net_failover.c index ed1166adaa2fd8a270f55f575d18739e4be462ad..b16a1221d19b97029c84a9e91d06a70babe4a9db 100644 --- a/drivers/net/net_failover.c +++ b/drivers/net/net_failover.c @@ -115,8 +115,7 @@ static netdev_tx_t net_failover_start_xmit(struct sk_buff *skb, static u16 net_failover_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { struct net_failover_info *nfo_info = netdev_priv(dev); struct net_device *primary_dev; @@ -127,10 +126,9 @@ static u16 net_failover_select_queue(struct net_device *dev, const struct net_device_ops *ops = primary_dev->netdev_ops; if (ops->ndo_select_queue) - txq = ops->ndo_select_queue(primary_dev, skb, - sb_dev, fallback); + txq = ops->ndo_select_queue(primary_dev, skb, sb_dev); else - txq = fallback(primary_dev, skb, NULL); + txq = netdev_pick_tx(primary_dev, skb, NULL); qdisc_skb_cb(skb)->slave_dev_queue_mapping = skb->queue_mapping; diff --git a/drivers/net/netdevsim/Makefile b/drivers/net/netdevsim/Makefile index 0fee1d06c0848f274baedb51ef30594cf4358332..09f1315d2f2aa2e2640104691bdea858d10f1d95 100644 --- a/drivers/net/netdevsim/Makefile +++ b/drivers/net/netdevsim/Makefile @@ -3,17 +3,13 @@ obj-$(CONFIG_NETDEVSIM) += netdevsim.o netdevsim-objs := \ - netdev.o \ + netdev.o dev.o fib.o bus.o ifeq ($(CONFIG_BPF_SYSCALL),y) netdevsim-objs += \ bpf.o endif -ifneq ($(CONFIG_NET_DEVLINK),) -netdevsim-objs += devlink.o fib.o -endif - ifneq ($(CONFIG_XFRM_OFFLOAD),) netdevsim-objs += ipsec.o endif diff --git a/drivers/net/netdevsim/bpf.c b/drivers/net/netdevsim/bpf.c index f92c43453ec67e1408860561d6a9fe221158b7b3..2b74425822ab118b4fc723fcbaa3d91385c33e2b 100644 --- a/drivers/net/netdevsim/bpf.c +++ b/drivers/net/netdevsim/bpf.c @@ -27,7 +27,7 @@ bpf_verifier_log_write(env, "[netdevsim] " fmt, ##__VA_ARGS__) struct nsim_bpf_bound_prog { - struct netdevsim *ns; + struct nsim_dev *nsim_dev; struct bpf_prog *prog; struct dentry *ddir; const char *state; @@ -65,8 +65,8 @@ nsim_bpf_verify_insn(struct bpf_verifier_env *env, int insn_idx, int prev_insn) struct nsim_bpf_bound_prog *state; state = env->prog->aux->offload->dev_priv; - if (state->ns->bpf_bind_verifier_delay && !insn_idx) - msleep(state->ns->bpf_bind_verifier_delay); + if (state->nsim_dev->bpf_bind_verifier_delay && !insn_idx) + msleep(state->nsim_dev->bpf_bind_verifier_delay); if (insn_idx == env->prog->len - 1) pr_vlog(env, "Hello from netdevsim!\n"); @@ -213,7 +213,8 @@ nsim_xdp_set_prog(struct netdevsim *ns, struct netdev_bpf *bpf, return 0; } -static int nsim_bpf_create_prog(struct netdevsim *ns, struct bpf_prog *prog) +static int nsim_bpf_create_prog(struct nsim_dev *nsim_dev, + struct bpf_prog *prog) { struct nsim_bpf_bound_prog *state; char name[16]; @@ -222,13 +223,13 @@ static int nsim_bpf_create_prog(struct netdevsim *ns, struct bpf_prog *prog) if (!state) return -ENOMEM; - state->ns = ns; + state->nsim_dev = nsim_dev; state->prog = prog; state->state = "verify"; /* Program id is not populated yet when we create the state. */ - sprintf(name, "%u", ns->sdev->prog_id_gen++); - state->ddir = debugfs_create_dir(name, ns->sdev->ddir_bpf_bound_progs); + sprintf(name, "%u", nsim_dev->prog_id_gen++); + state->ddir = debugfs_create_dir(name, nsim_dev->ddir_bpf_bound_progs); if (IS_ERR_OR_NULL(state->ddir)) { kfree(state); return -ENOMEM; @@ -239,7 +240,7 @@ static int nsim_bpf_create_prog(struct netdevsim *ns, struct bpf_prog *prog) &state->state, &nsim_bpf_string_fops); debugfs_create_bool("loaded", 0400, state->ddir, &state->is_loaded); - list_add_tail(&state->l, &ns->sdev->bpf_bound_progs); + list_add_tail(&state->l, &nsim_dev->bpf_bound_progs); prog->aux->offload->dev_priv = state; @@ -248,12 +249,13 @@ static int nsim_bpf_create_prog(struct netdevsim *ns, struct bpf_prog *prog) static int nsim_bpf_verifier_prep(struct bpf_prog *prog) { - struct netdevsim *ns = bpf_offload_dev_priv(prog->aux->offload->offdev); + struct nsim_dev *nsim_dev = + bpf_offload_dev_priv(prog->aux->offload->offdev); - if (!ns->bpf_bind_accept) + if (!nsim_dev->bpf_bind_accept) return -EOPNOTSUPP; - return nsim_bpf_create_prog(ns, prog); + return nsim_bpf_create_prog(nsim_dev, prog); } static int nsim_bpf_translate(struct bpf_prog *prog) @@ -512,7 +514,7 @@ nsim_bpf_map_alloc(struct netdevsim *ns, struct bpf_offloaded_map *offmap) } offmap->dev_ops = &nsim_bpf_map_ops; - list_add_tail(&nmap->l, &ns->sdev->bpf_bound_maps); + list_add_tail(&nmap->l, &ns->nsim_dev->bpf_bound_maps); return 0; @@ -576,61 +578,68 @@ int nsim_bpf(struct net_device *dev, struct netdev_bpf *bpf) } } -int nsim_bpf_init(struct netdevsim *ns) +int nsim_bpf_dev_init(struct nsim_dev *nsim_dev) { int err; - if (ns->sdev->refcnt == 1) { - INIT_LIST_HEAD(&ns->sdev->bpf_bound_progs); - INIT_LIST_HEAD(&ns->sdev->bpf_bound_maps); + INIT_LIST_HEAD(&nsim_dev->bpf_bound_progs); + INIT_LIST_HEAD(&nsim_dev->bpf_bound_maps); - ns->sdev->ddir_bpf_bound_progs = - debugfs_create_dir("bpf_bound_progs", ns->sdev->ddir); - if (IS_ERR_OR_NULL(ns->sdev->ddir_bpf_bound_progs)) - return -ENOMEM; + nsim_dev->ddir_bpf_bound_progs = debugfs_create_dir("bpf_bound_progs", + nsim_dev->ddir); + if (IS_ERR_OR_NULL(nsim_dev->ddir_bpf_bound_progs)) + return -ENOMEM; - ns->sdev->bpf_dev = bpf_offload_dev_create(&nsim_bpf_dev_ops, - ns); - err = PTR_ERR_OR_ZERO(ns->sdev->bpf_dev); - if (err) - return err; - } + nsim_dev->bpf_dev = bpf_offload_dev_create(&nsim_bpf_dev_ops, nsim_dev); + err = PTR_ERR_OR_ZERO(nsim_dev->bpf_dev); + if (err) + return err; + + nsim_dev->bpf_bind_accept = true; + debugfs_create_bool("bpf_bind_accept", 0600, nsim_dev->ddir, + &nsim_dev->bpf_bind_accept); + debugfs_create_u32("bpf_bind_verifier_delay", 0600, nsim_dev->ddir, + &nsim_dev->bpf_bind_verifier_delay); + return 0; +} + +void nsim_bpf_dev_exit(struct nsim_dev *nsim_dev) +{ + WARN_ON(!list_empty(&nsim_dev->bpf_bound_progs)); + WARN_ON(!list_empty(&nsim_dev->bpf_bound_maps)); + bpf_offload_dev_destroy(nsim_dev->bpf_dev); +} + +int nsim_bpf_init(struct netdevsim *ns) +{ + struct dentry *ddir = ns->nsim_dev_port->ddir; + int err; - err = bpf_offload_dev_netdev_register(ns->sdev->bpf_dev, ns->netdev); + err = bpf_offload_dev_netdev_register(ns->nsim_dev->bpf_dev, + ns->netdev); if (err) - goto err_destroy_bdev; + return err; - debugfs_create_u32("bpf_offloaded_id", 0400, ns->ddir, + debugfs_create_u32("bpf_offloaded_id", 0400, ddir, &ns->bpf_offloaded_id); - ns->bpf_bind_accept = true; - debugfs_create_bool("bpf_bind_accept", 0600, ns->ddir, - &ns->bpf_bind_accept); - debugfs_create_u32("bpf_bind_verifier_delay", 0600, ns->ddir, - &ns->bpf_bind_verifier_delay); - ns->bpf_tc_accept = true; - debugfs_create_bool("bpf_tc_accept", 0600, ns->ddir, + debugfs_create_bool("bpf_tc_accept", 0600, ddir, &ns->bpf_tc_accept); - debugfs_create_bool("bpf_tc_non_bound_accept", 0600, ns->ddir, + debugfs_create_bool("bpf_tc_non_bound_accept", 0600, ddir, &ns->bpf_tc_non_bound_accept); ns->bpf_xdpdrv_accept = true; - debugfs_create_bool("bpf_xdpdrv_accept", 0600, ns->ddir, + debugfs_create_bool("bpf_xdpdrv_accept", 0600, ddir, &ns->bpf_xdpdrv_accept); ns->bpf_xdpoffload_accept = true; - debugfs_create_bool("bpf_xdpoffload_accept", 0600, ns->ddir, + debugfs_create_bool("bpf_xdpoffload_accept", 0600, ddir, &ns->bpf_xdpoffload_accept); ns->bpf_map_accept = true; - debugfs_create_bool("bpf_map_accept", 0600, ns->ddir, + debugfs_create_bool("bpf_map_accept", 0600, ddir, &ns->bpf_map_accept); return 0; - -err_destroy_bdev: - if (ns->sdev->refcnt == 1) - bpf_offload_dev_destroy(ns->sdev->bpf_dev); - return err; } void nsim_bpf_uninit(struct netdevsim *ns) @@ -638,11 +647,5 @@ void nsim_bpf_uninit(struct netdevsim *ns) WARN_ON(ns->xdp.prog); WARN_ON(ns->xdp_hw.prog); WARN_ON(ns->bpf_offloaded); - bpf_offload_dev_netdev_unregister(ns->sdev->bpf_dev, ns->netdev); - - if (ns->sdev->refcnt == 1) { - WARN_ON(!list_empty(&ns->sdev->bpf_bound_progs)); - WARN_ON(!list_empty(&ns->sdev->bpf_bound_maps)); - bpf_offload_dev_destroy(ns->sdev->bpf_dev); - } + bpf_offload_dev_netdev_unregister(ns->nsim_dev->bpf_dev, ns->netdev); } diff --git a/drivers/net/netdevsim/bus.c b/drivers/net/netdevsim/bus.c new file mode 100644 index 0000000000000000000000000000000000000000..1a0ff3d7747b258065dbc465dd9799bbbc0f6fd4 --- /dev/null +++ b/drivers/net/netdevsim/bus.c @@ -0,0 +1,341 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (C) 2017 Netronome Systems, Inc. + * Copyright (C) 2019 Mellanox Technologies. All rights reserved + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "netdevsim.h" + +static DEFINE_IDA(nsim_bus_dev_ids); +static LIST_HEAD(nsim_bus_dev_list); +static DEFINE_MUTEX(nsim_bus_dev_list_lock); + +static struct nsim_bus_dev *to_nsim_bus_dev(struct device *dev) +{ + return container_of(dev, struct nsim_bus_dev, dev); +} + +static int nsim_bus_dev_vfs_enable(struct nsim_bus_dev *nsim_bus_dev, + unsigned int num_vfs) +{ + nsim_bus_dev->vfconfigs = kcalloc(num_vfs, + sizeof(struct nsim_vf_config), + GFP_KERNEL); + if (!nsim_bus_dev->vfconfigs) + return -ENOMEM; + nsim_bus_dev->num_vfs = num_vfs; + + return 0; +} + +static void nsim_bus_dev_vfs_disable(struct nsim_bus_dev *nsim_bus_dev) +{ + kfree(nsim_bus_dev->vfconfigs); + nsim_bus_dev->vfconfigs = NULL; + nsim_bus_dev->num_vfs = 0; +} + +static ssize_t +nsim_bus_dev_numvfs_store(struct device *dev, struct device_attribute *attr, + const char *buf, size_t count) +{ + struct nsim_bus_dev *nsim_bus_dev = to_nsim_bus_dev(dev); + unsigned int num_vfs; + int ret; + + ret = kstrtouint(buf, 0, &num_vfs); + if (ret) + return ret; + + rtnl_lock(); + if (nsim_bus_dev->num_vfs == num_vfs) + goto exit_good; + if (nsim_bus_dev->num_vfs && num_vfs) { + ret = -EBUSY; + goto exit_unlock; + } + + if (num_vfs) { + ret = nsim_bus_dev_vfs_enable(nsim_bus_dev, num_vfs); + if (ret) + goto exit_unlock; + } else { + nsim_bus_dev_vfs_disable(nsim_bus_dev); + } +exit_good: + ret = count; +exit_unlock: + rtnl_unlock(); + + return ret; +} + +static ssize_t +nsim_bus_dev_numvfs_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nsim_bus_dev *nsim_bus_dev = to_nsim_bus_dev(dev); + + return sprintf(buf, "%u\n", nsim_bus_dev->num_vfs); +} + +static struct device_attribute nsim_bus_dev_numvfs_attr = + __ATTR(sriov_numvfs, 0664, nsim_bus_dev_numvfs_show, + nsim_bus_dev_numvfs_store); + +static ssize_t +new_port_store(struct device *dev, struct device_attribute *attr, + const char *buf, size_t count) +{ + struct nsim_bus_dev *nsim_bus_dev = to_nsim_bus_dev(dev); + unsigned int port_index; + int ret; + + ret = kstrtouint(buf, 0, &port_index); + if (ret) + return ret; + ret = nsim_dev_port_add(nsim_bus_dev, port_index); + return ret ? ret : count; +} + +static struct device_attribute nsim_bus_dev_new_port_attr = __ATTR_WO(new_port); + +static ssize_t +del_port_store(struct device *dev, struct device_attribute *attr, + const char *buf, size_t count) +{ + struct nsim_bus_dev *nsim_bus_dev = to_nsim_bus_dev(dev); + unsigned int port_index; + int ret; + + ret = kstrtouint(buf, 0, &port_index); + if (ret) + return ret; + ret = nsim_dev_port_del(nsim_bus_dev, port_index); + return ret ? ret : count; +} + +static struct device_attribute nsim_bus_dev_del_port_attr = __ATTR_WO(del_port); + +static struct attribute *nsim_bus_dev_attrs[] = { + &nsim_bus_dev_numvfs_attr.attr, + &nsim_bus_dev_new_port_attr.attr, + &nsim_bus_dev_del_port_attr.attr, + NULL, +}; + +static const struct attribute_group nsim_bus_dev_attr_group = { + .attrs = nsim_bus_dev_attrs, +}; + +static const struct attribute_group *nsim_bus_dev_attr_groups[] = { + &nsim_bus_dev_attr_group, + NULL, +}; + +static void nsim_bus_dev_release(struct device *dev) +{ + struct nsim_bus_dev *nsim_bus_dev = to_nsim_bus_dev(dev); + + nsim_bus_dev_vfs_disable(nsim_bus_dev); +} + +static struct device_type nsim_bus_dev_type = { + .groups = nsim_bus_dev_attr_groups, + .release = nsim_bus_dev_release, +}; + +static struct nsim_bus_dev * +nsim_bus_dev_new(unsigned int id, unsigned int port_count); + +static ssize_t +new_device_store(struct bus_type *bus, const char *buf, size_t count) +{ + struct nsim_bus_dev *nsim_bus_dev; + unsigned int port_count; + unsigned int id; + int err; + + err = sscanf(buf, "%u %u", &id, &port_count); + switch (err) { + case 1: + port_count = 1; + /* fall through */ + case 2: + if (id > INT_MAX) { + pr_err("Value of \"id\" is too big.\n"); + return -EINVAL; + } + break; + default: + pr_err("Format for adding new device is \"id port_count\" (uint uint).\n"); + return -EINVAL; + } + nsim_bus_dev = nsim_bus_dev_new(id, port_count); + if (IS_ERR(nsim_bus_dev)) + return PTR_ERR(nsim_bus_dev); + + mutex_lock(&nsim_bus_dev_list_lock); + list_add_tail(&nsim_bus_dev->list, &nsim_bus_dev_list); + mutex_unlock(&nsim_bus_dev_list_lock); + + return count; +} +static BUS_ATTR_WO(new_device); + +static void nsim_bus_dev_del(struct nsim_bus_dev *nsim_bus_dev); + +static ssize_t +del_device_store(struct bus_type *bus, const char *buf, size_t count) +{ + struct nsim_bus_dev *nsim_bus_dev, *tmp; + unsigned int id; + int err; + + err = sscanf(buf, "%u", &id); + switch (err) { + case 1: + if (id > INT_MAX) { + pr_err("Value of \"id\" is too big.\n"); + return -EINVAL; + } + break; + default: + pr_err("Format for deleting device is \"id\" (uint).\n"); + return -EINVAL; + } + + err = -ENOENT; + mutex_lock(&nsim_bus_dev_list_lock); + list_for_each_entry_safe(nsim_bus_dev, tmp, &nsim_bus_dev_list, list) { + if (nsim_bus_dev->dev.id != id) + continue; + list_del(&nsim_bus_dev->list); + nsim_bus_dev_del(nsim_bus_dev); + err = 0; + break; + } + mutex_unlock(&nsim_bus_dev_list_lock); + return !err ? count : err; +} +static BUS_ATTR_WO(del_device); + +static struct attribute *nsim_bus_attrs[] = { + &bus_attr_new_device.attr, + &bus_attr_del_device.attr, + NULL +}; +ATTRIBUTE_GROUPS(nsim_bus); + +static int nsim_bus_probe(struct device *dev) +{ + struct nsim_bus_dev *nsim_bus_dev = to_nsim_bus_dev(dev); + + return nsim_dev_probe(nsim_bus_dev); +} + +static int nsim_bus_remove(struct device *dev) +{ + struct nsim_bus_dev *nsim_bus_dev = to_nsim_bus_dev(dev); + + nsim_dev_remove(nsim_bus_dev); + return 0; +} + +static int nsim_num_vf(struct device *dev) +{ + struct nsim_bus_dev *nsim_bus_dev = to_nsim_bus_dev(dev); + + return nsim_bus_dev->num_vfs; +} + +static struct bus_type nsim_bus = { + .name = DRV_NAME, + .dev_name = DRV_NAME, + .bus_groups = nsim_bus_groups, + .probe = nsim_bus_probe, + .remove = nsim_bus_remove, + .num_vf = nsim_num_vf, +}; + +static struct nsim_bus_dev * +nsim_bus_dev_new(unsigned int id, unsigned int port_count) +{ + struct nsim_bus_dev *nsim_bus_dev; + int err; + + nsim_bus_dev = kzalloc(sizeof(*nsim_bus_dev), GFP_KERNEL); + if (!nsim_bus_dev) + return ERR_PTR(-ENOMEM); + + err = ida_alloc_range(&nsim_bus_dev_ids, id, id, GFP_KERNEL); + if (err < 0) + goto err_nsim_bus_dev_free; + nsim_bus_dev->dev.id = err; + nsim_bus_dev->dev.bus = &nsim_bus; + nsim_bus_dev->dev.type = &nsim_bus_dev_type; + nsim_bus_dev->port_count = port_count; + + err = device_register(&nsim_bus_dev->dev); + if (err) + goto err_nsim_bus_dev_id_free; + return nsim_bus_dev; + +err_nsim_bus_dev_id_free: + ida_free(&nsim_bus_dev_ids, nsim_bus_dev->dev.id); +err_nsim_bus_dev_free: + kfree(nsim_bus_dev); + return ERR_PTR(err); +} + +static void nsim_bus_dev_del(struct nsim_bus_dev *nsim_bus_dev) +{ + device_unregister(&nsim_bus_dev->dev); + ida_free(&nsim_bus_dev_ids, nsim_bus_dev->dev.id); + kfree(nsim_bus_dev); +} + +static struct device_driver nsim_driver = { + .name = DRV_NAME, + .bus = &nsim_bus, + .owner = THIS_MODULE, +}; + +int nsim_bus_init(void) +{ + int err; + + err = bus_register(&nsim_bus); + if (err) + return err; + err = driver_register(&nsim_driver); + if (err) + goto err_bus_unregister; + return 0; + +err_bus_unregister: + bus_unregister(&nsim_bus); + return err; +} + +void nsim_bus_exit(void) +{ + struct nsim_bus_dev *nsim_bus_dev, *tmp; + + mutex_lock(&nsim_bus_dev_list_lock); + list_for_each_entry_safe(nsim_bus_dev, tmp, &nsim_bus_dev_list, list) { + list_del(&nsim_bus_dev->list); + nsim_bus_dev_del(nsim_bus_dev); + } + mutex_unlock(&nsim_bus_dev_list_lock); + driver_unregister(&nsim_driver); + bus_unregister(&nsim_bus); +} diff --git a/drivers/net/netdevsim/dev.c b/drivers/net/netdevsim/dev.c new file mode 100644 index 0000000000000000000000000000000000000000..b509b941d5cab39fc55a29dca7ddf16208a5ddb1 --- /dev/null +++ b/drivers/net/netdevsim/dev.c @@ -0,0 +1,447 @@ +/* + * Copyright (c) 2018 Cumulus Networks. All rights reserved. + * Copyright (c) 2018 David Ahern + * Copyright (c) 2019 Mellanox Technologies. All rights reserved. + * + * This software is licensed under the GNU General License Version 2, + * June 1991 as shown in the file COPYING in the top-level directory of this + * source tree. + * + * THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" + * WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, + * BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS + * FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE + * OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME + * THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + */ + +#include +#include +#include +#include +#include +#include +#include + +#include "netdevsim.h" + +static struct dentry *nsim_dev_ddir; + +static int nsim_dev_debugfs_init(struct nsim_dev *nsim_dev) +{ + char dev_ddir_name[16]; + + sprintf(dev_ddir_name, DRV_NAME "%u", nsim_dev->nsim_bus_dev->dev.id); + nsim_dev->ddir = debugfs_create_dir(dev_ddir_name, nsim_dev_ddir); + if (IS_ERR_OR_NULL(nsim_dev->ddir)) + return PTR_ERR_OR_ZERO(nsim_dev->ddir) ?: -EINVAL; + nsim_dev->ports_ddir = debugfs_create_dir("ports", nsim_dev->ddir); + if (IS_ERR_OR_NULL(nsim_dev->ports_ddir)) + return PTR_ERR_OR_ZERO(nsim_dev->ports_ddir) ?: -EINVAL; + return 0; +} + +static void nsim_dev_debugfs_exit(struct nsim_dev *nsim_dev) +{ + debugfs_remove_recursive(nsim_dev->ports_ddir); + debugfs_remove_recursive(nsim_dev->ddir); +} + +static int nsim_dev_port_debugfs_init(struct nsim_dev *nsim_dev, + struct nsim_dev_port *nsim_dev_port) +{ + char port_ddir_name[16]; + char dev_link_name[32]; + + sprintf(port_ddir_name, "%u", nsim_dev_port->port_index); + nsim_dev_port->ddir = debugfs_create_dir(port_ddir_name, + nsim_dev->ports_ddir); + if (IS_ERR_OR_NULL(nsim_dev_port->ddir)) + return -ENOMEM; + + sprintf(dev_link_name, "../../../" DRV_NAME "%u", + nsim_dev->nsim_bus_dev->dev.id); + debugfs_create_symlink("dev", nsim_dev_port->ddir, dev_link_name); + + return 0; +} + +static void nsim_dev_port_debugfs_exit(struct nsim_dev_port *nsim_dev_port) +{ + debugfs_remove_recursive(nsim_dev_port->ddir); +} + +static u64 nsim_dev_ipv4_fib_resource_occ_get(void *priv) +{ + struct nsim_dev *nsim_dev = priv; + + return nsim_fib_get_val(nsim_dev->fib_data, + NSIM_RESOURCE_IPV4_FIB, false); +} + +static u64 nsim_dev_ipv4_fib_rules_res_occ_get(void *priv) +{ + struct nsim_dev *nsim_dev = priv; + + return nsim_fib_get_val(nsim_dev->fib_data, + NSIM_RESOURCE_IPV4_FIB_RULES, false); +} + +static u64 nsim_dev_ipv6_fib_resource_occ_get(void *priv) +{ + struct nsim_dev *nsim_dev = priv; + + return nsim_fib_get_val(nsim_dev->fib_data, + NSIM_RESOURCE_IPV6_FIB, false); +} + +static u64 nsim_dev_ipv6_fib_rules_res_occ_get(void *priv) +{ + struct nsim_dev *nsim_dev = priv; + + return nsim_fib_get_val(nsim_dev->fib_data, + NSIM_RESOURCE_IPV6_FIB_RULES, false); +} + +static int nsim_dev_resources_register(struct devlink *devlink) +{ + struct nsim_dev *nsim_dev = devlink_priv(devlink); + struct devlink_resource_size_params params = { + .size_max = (u64)-1, + .size_granularity = 1, + .unit = DEVLINK_RESOURCE_UNIT_ENTRY + }; + int err; + u64 n; + + /* Resources for IPv4 */ + err = devlink_resource_register(devlink, "IPv4", (u64)-1, + NSIM_RESOURCE_IPV4, + DEVLINK_RESOURCE_ID_PARENT_TOP, + ¶ms); + if (err) { + pr_err("Failed to register IPv4 top resource\n"); + goto out; + } + + n = nsim_fib_get_val(nsim_dev->fib_data, + NSIM_RESOURCE_IPV4_FIB, true); + err = devlink_resource_register(devlink, "fib", n, + NSIM_RESOURCE_IPV4_FIB, + NSIM_RESOURCE_IPV4, ¶ms); + if (err) { + pr_err("Failed to register IPv4 FIB resource\n"); + return err; + } + + n = nsim_fib_get_val(nsim_dev->fib_data, + NSIM_RESOURCE_IPV4_FIB_RULES, true); + err = devlink_resource_register(devlink, "fib-rules", n, + NSIM_RESOURCE_IPV4_FIB_RULES, + NSIM_RESOURCE_IPV4, ¶ms); + if (err) { + pr_err("Failed to register IPv4 FIB rules resource\n"); + return err; + } + + /* Resources for IPv6 */ + err = devlink_resource_register(devlink, "IPv6", (u64)-1, + NSIM_RESOURCE_IPV6, + DEVLINK_RESOURCE_ID_PARENT_TOP, + ¶ms); + if (err) { + pr_err("Failed to register IPv6 top resource\n"); + goto out; + } + + n = nsim_fib_get_val(nsim_dev->fib_data, + NSIM_RESOURCE_IPV6_FIB, true); + err = devlink_resource_register(devlink, "fib", n, + NSIM_RESOURCE_IPV6_FIB, + NSIM_RESOURCE_IPV6, ¶ms); + if (err) { + pr_err("Failed to register IPv6 FIB resource\n"); + return err; + } + + n = nsim_fib_get_val(nsim_dev->fib_data, + NSIM_RESOURCE_IPV6_FIB_RULES, true); + err = devlink_resource_register(devlink, "fib-rules", n, + NSIM_RESOURCE_IPV6_FIB_RULES, + NSIM_RESOURCE_IPV6, ¶ms); + if (err) { + pr_err("Failed to register IPv6 FIB rules resource\n"); + return err; + } + + devlink_resource_occ_get_register(devlink, + NSIM_RESOURCE_IPV4_FIB, + nsim_dev_ipv4_fib_resource_occ_get, + nsim_dev); + devlink_resource_occ_get_register(devlink, + NSIM_RESOURCE_IPV4_FIB_RULES, + nsim_dev_ipv4_fib_rules_res_occ_get, + nsim_dev); + devlink_resource_occ_get_register(devlink, + NSIM_RESOURCE_IPV6_FIB, + nsim_dev_ipv6_fib_resource_occ_get, + nsim_dev); + devlink_resource_occ_get_register(devlink, + NSIM_RESOURCE_IPV6_FIB_RULES, + nsim_dev_ipv6_fib_rules_res_occ_get, + nsim_dev); +out: + return err; +} + +static int nsim_dev_reload(struct devlink *devlink, + struct netlink_ext_ack *extack) +{ + struct nsim_dev *nsim_dev = devlink_priv(devlink); + enum nsim_resource_id res_ids[] = { + NSIM_RESOURCE_IPV4_FIB, NSIM_RESOURCE_IPV4_FIB_RULES, + NSIM_RESOURCE_IPV6_FIB, NSIM_RESOURCE_IPV6_FIB_RULES + }; + int i; + + for (i = 0; i < ARRAY_SIZE(res_ids); ++i) { + int err; + u64 val; + + err = devlink_resource_size_get(devlink, res_ids[i], &val); + if (!err) { + err = nsim_fib_set_max(nsim_dev->fib_data, + res_ids[i], val, extack); + if (err) + return err; + } + } + + return 0; +} + +static const struct devlink_ops nsim_dev_devlink_ops = { + .reload = nsim_dev_reload, +}; + +static struct nsim_dev * +nsim_dev_create(struct nsim_bus_dev *nsim_bus_dev, unsigned int port_count) +{ + struct nsim_dev *nsim_dev; + struct devlink *devlink; + int err; + + devlink = devlink_alloc(&nsim_dev_devlink_ops, sizeof(*nsim_dev)); + if (!devlink) + return ERR_PTR(-ENOMEM); + nsim_dev = devlink_priv(devlink); + nsim_dev->nsim_bus_dev = nsim_bus_dev; + nsim_dev->switch_id.id_len = sizeof(nsim_dev->switch_id.id); + get_random_bytes(nsim_dev->switch_id.id, nsim_dev->switch_id.id_len); + INIT_LIST_HEAD(&nsim_dev->port_list); + mutex_init(&nsim_dev->port_list_lock); + + nsim_dev->fib_data = nsim_fib_create(); + if (IS_ERR(nsim_dev->fib_data)) { + err = PTR_ERR(nsim_dev->fib_data); + goto err_devlink_free; + } + + err = nsim_dev_resources_register(devlink); + if (err) + goto err_fib_destroy; + + err = devlink_register(devlink, &nsim_bus_dev->dev); + if (err) + goto err_resources_unregister; + + err = nsim_dev_debugfs_init(nsim_dev); + if (err) + goto err_dl_unregister; + + err = nsim_bpf_dev_init(nsim_dev); + if (err) + goto err_debugfs_exit; + + return nsim_dev; + +err_debugfs_exit: + nsim_dev_debugfs_exit(nsim_dev); +err_dl_unregister: + devlink_unregister(devlink); +err_resources_unregister: + devlink_resources_unregister(devlink, NULL); +err_fib_destroy: + nsim_fib_destroy(nsim_dev->fib_data); +err_devlink_free: + devlink_free(devlink); + return ERR_PTR(err); +} + +static void nsim_dev_destroy(struct nsim_dev *nsim_dev) +{ + struct devlink *devlink = priv_to_devlink(nsim_dev); + + nsim_bpf_dev_exit(nsim_dev); + nsim_dev_debugfs_exit(nsim_dev); + devlink_unregister(devlink); + devlink_resources_unregister(devlink, NULL); + nsim_fib_destroy(nsim_dev->fib_data); + mutex_destroy(&nsim_dev->port_list_lock); + devlink_free(devlink); +} + +static int __nsim_dev_port_add(struct nsim_dev *nsim_dev, + unsigned int port_index) +{ + struct nsim_dev_port *nsim_dev_port; + struct devlink_port *devlink_port; + int err; + + nsim_dev_port = kzalloc(sizeof(*nsim_dev_port), GFP_KERNEL); + if (!nsim_dev_port) + return -ENOMEM; + nsim_dev_port->port_index = port_index; + + devlink_port = &nsim_dev_port->devlink_port; + devlink_port_attrs_set(devlink_port, DEVLINK_PORT_FLAVOUR_PHYSICAL, + port_index + 1, 0, 0, + nsim_dev->switch_id.id, + nsim_dev->switch_id.id_len); + err = devlink_port_register(priv_to_devlink(nsim_dev), devlink_port, + port_index); + if (err) + goto err_port_free; + + err = nsim_dev_port_debugfs_init(nsim_dev, nsim_dev_port); + if (err) + goto err_dl_port_unregister; + + nsim_dev_port->ns = nsim_create(nsim_dev, nsim_dev_port); + if (IS_ERR(nsim_dev_port->ns)) { + err = PTR_ERR(nsim_dev_port->ns); + goto err_port_debugfs_exit; + } + + devlink_port_type_eth_set(devlink_port, nsim_dev_port->ns->netdev); + list_add(&nsim_dev_port->list, &nsim_dev->port_list); + + return 0; + +err_port_debugfs_exit: + nsim_dev_port_debugfs_exit(nsim_dev_port); +err_dl_port_unregister: + devlink_port_unregister(devlink_port); +err_port_free: + kfree(nsim_dev_port); + return err; +} + +static void __nsim_dev_port_del(struct nsim_dev_port *nsim_dev_port) +{ + struct devlink_port *devlink_port = &nsim_dev_port->devlink_port; + + list_del(&nsim_dev_port->list); + devlink_port_type_clear(devlink_port); + nsim_destroy(nsim_dev_port->ns); + nsim_dev_port_debugfs_exit(nsim_dev_port); + devlink_port_unregister(devlink_port); + kfree(nsim_dev_port); +} + +static void nsim_dev_port_del_all(struct nsim_dev *nsim_dev) +{ + struct nsim_dev_port *nsim_dev_port, *tmp; + + list_for_each_entry_safe(nsim_dev_port, tmp, + &nsim_dev->port_list, list) + __nsim_dev_port_del(nsim_dev_port); +} + +int nsim_dev_probe(struct nsim_bus_dev *nsim_bus_dev) +{ + struct nsim_dev *nsim_dev; + int i; + int err; + + nsim_dev = nsim_dev_create(nsim_bus_dev, nsim_bus_dev->port_count); + if (IS_ERR(nsim_dev)) + return PTR_ERR(nsim_dev); + dev_set_drvdata(&nsim_bus_dev->dev, nsim_dev); + + for (i = 0; i < nsim_bus_dev->port_count; i++) { + err = __nsim_dev_port_add(nsim_dev, i); + if (err) + goto err_port_del_all; + } + return 0; + +err_port_del_all: + nsim_dev_port_del_all(nsim_dev); + nsim_dev_destroy(nsim_dev); + return err; +} + +void nsim_dev_remove(struct nsim_bus_dev *nsim_bus_dev) +{ + struct nsim_dev *nsim_dev = dev_get_drvdata(&nsim_bus_dev->dev); + + nsim_dev_port_del_all(nsim_dev); + nsim_dev_destroy(nsim_dev); +} + +static struct nsim_dev_port * +__nsim_dev_port_lookup(struct nsim_dev *nsim_dev, unsigned int port_index) +{ + struct nsim_dev_port *nsim_dev_port; + + list_for_each_entry(nsim_dev_port, &nsim_dev->port_list, list) + if (nsim_dev_port->port_index == port_index) + return nsim_dev_port; + return NULL; +} + +int nsim_dev_port_add(struct nsim_bus_dev *nsim_bus_dev, + unsigned int port_index) +{ + struct nsim_dev *nsim_dev = dev_get_drvdata(&nsim_bus_dev->dev); + int err; + + mutex_lock(&nsim_dev->port_list_lock); + if (__nsim_dev_port_lookup(nsim_dev, port_index)) + err = -EEXIST; + else + err = __nsim_dev_port_add(nsim_dev, port_index); + mutex_unlock(&nsim_dev->port_list_lock); + return err; +} + +int nsim_dev_port_del(struct nsim_bus_dev *nsim_bus_dev, + unsigned int port_index) +{ + struct nsim_dev *nsim_dev = dev_get_drvdata(&nsim_bus_dev->dev); + struct nsim_dev_port *nsim_dev_port; + int err = 0; + + mutex_lock(&nsim_dev->port_list_lock); + nsim_dev_port = __nsim_dev_port_lookup(nsim_dev, port_index); + if (!nsim_dev_port) + err = -ENOENT; + else + __nsim_dev_port_del(nsim_dev_port); + mutex_unlock(&nsim_dev->port_list_lock); + return err; +} + +int nsim_dev_init(void) +{ + nsim_dev_ddir = debugfs_create_dir(DRV_NAME, NULL); + if (IS_ERR_OR_NULL(nsim_dev_ddir)) + return -ENOMEM; + return 0; +} + +void nsim_dev_exit(void) +{ + debugfs_remove_recursive(nsim_dev_ddir); +} diff --git a/drivers/net/netdevsim/devlink.c b/drivers/net/netdevsim/devlink.c deleted file mode 100644 index 5135fc371f01ba27365b85c278b3c20b90e65c58..0000000000000000000000000000000000000000 --- a/drivers/net/netdevsim/devlink.c +++ /dev/null @@ -1,295 +0,0 @@ -/* - * Copyright (c) 2018 Cumulus Networks. All rights reserved. - * Copyright (c) 2018 David Ahern - * - * This software is licensed under the GNU General License Version 2, - * June 1991 as shown in the file COPYING in the top-level directory of this - * source tree. - * - * THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" - * WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, - * BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS - * FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE - * OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME - * THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. - */ - -#include -#include -#include - -#include "netdevsim.h" - -static unsigned int nsim_devlink_id; - -/* place holder until devlink and namespaces is sorted out */ -static struct net *nsim_devlink_net(struct devlink *devlink) -{ - return &init_net; -} - -/* IPv4 - */ -static u64 nsim_ipv4_fib_resource_occ_get(void *priv) -{ - struct net *net = priv; - - return nsim_fib_get_val(net, NSIM_RESOURCE_IPV4_FIB, false); -} - -static u64 nsim_ipv4_fib_rules_res_occ_get(void *priv) -{ - struct net *net = priv; - - return nsim_fib_get_val(net, NSIM_RESOURCE_IPV4_FIB_RULES, false); -} - -/* IPv6 - */ -static u64 nsim_ipv6_fib_resource_occ_get(void *priv) -{ - struct net *net = priv; - - return nsim_fib_get_val(net, NSIM_RESOURCE_IPV6_FIB, false); -} - -static u64 nsim_ipv6_fib_rules_res_occ_get(void *priv) -{ - struct net *net = priv; - - return nsim_fib_get_val(net, NSIM_RESOURCE_IPV6_FIB_RULES, false); -} - -static int devlink_resources_register(struct devlink *devlink) -{ - struct devlink_resource_size_params params = { - .size_max = (u64)-1, - .size_granularity = 1, - .unit = DEVLINK_RESOURCE_UNIT_ENTRY - }; - struct net *net = nsim_devlink_net(devlink); - int err; - u64 n; - - /* Resources for IPv4 */ - err = devlink_resource_register(devlink, "IPv4", (u64)-1, - NSIM_RESOURCE_IPV4, - DEVLINK_RESOURCE_ID_PARENT_TOP, - ¶ms); - if (err) { - pr_err("Failed to register IPv4 top resource\n"); - goto out; - } - - n = nsim_fib_get_val(net, NSIM_RESOURCE_IPV4_FIB, true); - err = devlink_resource_register(devlink, "fib", n, - NSIM_RESOURCE_IPV4_FIB, - NSIM_RESOURCE_IPV4, ¶ms); - if (err) { - pr_err("Failed to register IPv4 FIB resource\n"); - return err; - } - - n = nsim_fib_get_val(net, NSIM_RESOURCE_IPV4_FIB_RULES, true); - err = devlink_resource_register(devlink, "fib-rules", n, - NSIM_RESOURCE_IPV4_FIB_RULES, - NSIM_RESOURCE_IPV4, ¶ms); - if (err) { - pr_err("Failed to register IPv4 FIB rules resource\n"); - return err; - } - - /* Resources for IPv6 */ - err = devlink_resource_register(devlink, "IPv6", (u64)-1, - NSIM_RESOURCE_IPV6, - DEVLINK_RESOURCE_ID_PARENT_TOP, - ¶ms); - if (err) { - pr_err("Failed to register IPv6 top resource\n"); - goto out; - } - - n = nsim_fib_get_val(net, NSIM_RESOURCE_IPV6_FIB, true); - err = devlink_resource_register(devlink, "fib", n, - NSIM_RESOURCE_IPV6_FIB, - NSIM_RESOURCE_IPV6, ¶ms); - if (err) { - pr_err("Failed to register IPv6 FIB resource\n"); - return err; - } - - n = nsim_fib_get_val(net, NSIM_RESOURCE_IPV6_FIB_RULES, true); - err = devlink_resource_register(devlink, "fib-rules", n, - NSIM_RESOURCE_IPV6_FIB_RULES, - NSIM_RESOURCE_IPV6, ¶ms); - if (err) { - pr_err("Failed to register IPv6 FIB rules resource\n"); - return err; - } - - devlink_resource_occ_get_register(devlink, - NSIM_RESOURCE_IPV4_FIB, - nsim_ipv4_fib_resource_occ_get, - net); - devlink_resource_occ_get_register(devlink, - NSIM_RESOURCE_IPV4_FIB_RULES, - nsim_ipv4_fib_rules_res_occ_get, - net); - devlink_resource_occ_get_register(devlink, - NSIM_RESOURCE_IPV6_FIB, - nsim_ipv6_fib_resource_occ_get, - net); - devlink_resource_occ_get_register(devlink, - NSIM_RESOURCE_IPV6_FIB_RULES, - nsim_ipv6_fib_rules_res_occ_get, - net); -out: - return err; -} - -static int nsim_devlink_reload(struct devlink *devlink, - struct netlink_ext_ack *extack) -{ - enum nsim_resource_id res_ids[] = { - NSIM_RESOURCE_IPV4_FIB, NSIM_RESOURCE_IPV4_FIB_RULES, - NSIM_RESOURCE_IPV6_FIB, NSIM_RESOURCE_IPV6_FIB_RULES - }; - struct net *net = nsim_devlink_net(devlink); - int i; - - for (i = 0; i < ARRAY_SIZE(res_ids); ++i) { - int err; - u64 val; - - err = devlink_resource_size_get(devlink, res_ids[i], &val); - if (!err) { - err = nsim_fib_set_max(net, res_ids[i], val, extack); - if (err) - return err; - } - } - - return 0; -} - -static void nsim_devlink_net_reset(struct net *net) -{ - enum nsim_resource_id res_ids[] = { - NSIM_RESOURCE_IPV4_FIB, NSIM_RESOURCE_IPV4_FIB_RULES, - NSIM_RESOURCE_IPV6_FIB, NSIM_RESOURCE_IPV6_FIB_RULES - }; - int i; - - for (i = 0; i < ARRAY_SIZE(res_ids); ++i) { - if (nsim_fib_set_max(net, res_ids[i], (u64)-1, NULL)) { - pr_err("Failed to reset limit for resource %u\n", - res_ids[i]); - } - } -} - -static const struct devlink_ops nsim_devlink_ops = { - .reload = nsim_devlink_reload, -}; - -/* once devlink / namespace issues are sorted out - * this needs to be net in which a devlink instance - * is to be created. e.g., dev_net(ns->netdev) - */ -static struct net *nsim_to_net(struct netdevsim *ns) -{ - return &init_net; -} - -void nsim_devlink_teardown(struct netdevsim *ns) -{ - if (ns->devlink) { - struct net *net = nsim_to_net(ns); - bool *reg_devlink = net_generic(net, nsim_devlink_id); - - devlink_resources_unregister(ns->devlink, NULL); - devlink_unregister(ns->devlink); - devlink_free(ns->devlink); - ns->devlink = NULL; - - nsim_devlink_net_reset(net); - *reg_devlink = true; - } -} - -int nsim_devlink_setup(struct netdevsim *ns) -{ - struct net *net = nsim_to_net(ns); - bool *reg_devlink = net_generic(net, nsim_devlink_id); - struct devlink *devlink; - int err; - - /* only one device per namespace controls devlink */ - if (!*reg_devlink) { - ns->devlink = NULL; - return 0; - } - - devlink = devlink_alloc(&nsim_devlink_ops, 0); - if (!devlink) - return -ENOMEM; - - err = devlink_register(devlink, &ns->dev); - if (err) - goto err_devlink_free; - - err = devlink_resources_register(devlink); - if (err) - goto err_dl_unregister; - - ns->devlink = devlink; - - *reg_devlink = false; - - return 0; - -err_dl_unregister: - devlink_unregister(devlink); -err_devlink_free: - devlink_free(devlink); - - return err; -} - -/* Initialize per network namespace state */ -static int __net_init nsim_devlink_netns_init(struct net *net) -{ - bool *reg_devlink = net_generic(net, nsim_devlink_id); - - *reg_devlink = true; - - return 0; -} - -static struct pernet_operations nsim_devlink_net_ops = { - .init = nsim_devlink_netns_init, - .id = &nsim_devlink_id, - .size = sizeof(bool), -}; - -void nsim_devlink_exit(void) -{ - unregister_pernet_subsys(&nsim_devlink_net_ops); - nsim_fib_exit(); -} - -int nsim_devlink_init(void) -{ - int err; - - err = nsim_fib_init(); - if (err) - goto err_out; - - err = register_pernet_subsys(&nsim_devlink_net_ops); - if (err) - nsim_fib_exit(); - -err_out: - return err; -} diff --git a/drivers/net/netdevsim/fib.c b/drivers/net/netdevsim/fib.c index f61d094746c069bb7ac1c6db9eb77fe0820d2562..8c57ba7477726650bb6c0b188a2c8ebb989b14d7 100644 --- a/drivers/net/netdevsim/fib.c +++ b/drivers/net/netdevsim/fib.c @@ -18,7 +18,6 @@ #include #include #include -#include #include "netdevsim.h" @@ -33,15 +32,14 @@ struct nsim_per_fib_data { }; struct nsim_fib_data { + struct notifier_block fib_nb; struct nsim_per_fib_data ipv4; struct nsim_per_fib_data ipv6; }; -static unsigned int nsim_fib_net_id; - -u64 nsim_fib_get_val(struct net *net, enum nsim_resource_id res_id, bool max) +u64 nsim_fib_get_val(struct nsim_fib_data *fib_data, + enum nsim_resource_id res_id, bool max) { - struct nsim_fib_data *fib_data = net_generic(net, nsim_fib_net_id); struct nsim_fib_entry *entry; switch (res_id) { @@ -64,10 +62,10 @@ u64 nsim_fib_get_val(struct net *net, enum nsim_resource_id res_id, bool max) return max ? entry->max : entry->num; } -int nsim_fib_set_max(struct net *net, enum nsim_resource_id res_id, u64 val, +int nsim_fib_set_max(struct nsim_fib_data *fib_data, + enum nsim_resource_id res_id, u64 val, struct netlink_ext_ack *extack) { - struct nsim_fib_data *fib_data = net_generic(net, nsim_fib_net_id); struct nsim_fib_entry *entry; int err = 0; @@ -120,9 +118,9 @@ static int nsim_fib_rule_account(struct nsim_fib_entry *entry, bool add, return err; } -static int nsim_fib_rule_event(struct fib_notifier_info *info, bool add) +static int nsim_fib_rule_event(struct nsim_fib_data *data, + struct fib_notifier_info *info, bool add) { - struct nsim_fib_data *data = net_generic(info->net, nsim_fib_net_id); struct netlink_ext_ack *extack = info->extack; int err = 0; @@ -157,9 +155,9 @@ static int nsim_fib_account(struct nsim_fib_entry *entry, bool add, return err; } -static int nsim_fib_event(struct fib_notifier_info *info, bool add) +static int nsim_fib_event(struct nsim_fib_data *data, + struct fib_notifier_info *info, bool add) { - struct nsim_fib_data *data = net_generic(info->net, nsim_fib_net_id); struct netlink_ext_ack *extack = info->extack; int err = 0; @@ -178,18 +176,22 @@ static int nsim_fib_event(struct fib_notifier_info *info, bool add) static int nsim_fib_event_nb(struct notifier_block *nb, unsigned long event, void *ptr) { + struct nsim_fib_data *data = container_of(nb, struct nsim_fib_data, + fib_nb); struct fib_notifier_info *info = ptr; int err = 0; switch (event) { case FIB_EVENT_RULE_ADD: /* fall through */ case FIB_EVENT_RULE_DEL: - err = nsim_fib_rule_event(info, event == FIB_EVENT_RULE_ADD); + err = nsim_fib_rule_event(data, info, + event == FIB_EVENT_RULE_ADD); break; case FIB_EVENT_ENTRY_ADD: /* fall through */ case FIB_EVENT_ENTRY_DEL: - err = nsim_fib_event(info, event == FIB_EVENT_ENTRY_ADD); + err = nsim_fib_event(data, info, + event == FIB_EVENT_ENTRY_ADD); break; } @@ -199,30 +201,23 @@ static int nsim_fib_event_nb(struct notifier_block *nb, unsigned long event, /* inconsistent dump, trying again */ static void nsim_fib_dump_inconsistent(struct notifier_block *nb) { - struct nsim_fib_data *data; - struct net *net; - - rcu_read_lock(); - for_each_net_rcu(net) { - data = net_generic(net, nsim_fib_net_id); - - data->ipv4.fib.num = 0ULL; - data->ipv4.rules.num = 0ULL; + struct nsim_fib_data *data = container_of(nb, struct nsim_fib_data, + fib_nb); - data->ipv6.fib.num = 0ULL; - data->ipv6.rules.num = 0ULL; - } - rcu_read_unlock(); + data->ipv4.fib.num = 0ULL; + data->ipv4.rules.num = 0ULL; + data->ipv6.fib.num = 0ULL; + data->ipv6.rules.num = 0ULL; } -static struct notifier_block nsim_fib_nb = { - .notifier_call = nsim_fib_event_nb, -}; - -/* Initialize per network namespace state */ -static int __net_init nsim_fib_netns_init(struct net *net) +struct nsim_fib_data *nsim_fib_create(void) { - struct nsim_fib_data *data = net_generic(net, nsim_fib_net_id); + struct nsim_fib_data *data; + int err; + + data = kzalloc(sizeof(*data), GFP_KERNEL); + if (!data) + return ERR_PTR(-ENOMEM); data->ipv4.fib.max = (u64)-1; data->ipv4.rules.max = (u64)-1; @@ -230,37 +225,22 @@ static int __net_init nsim_fib_netns_init(struct net *net) data->ipv6.fib.max = (u64)-1; data->ipv6.rules.max = (u64)-1; - return 0; -} - -static struct pernet_operations nsim_fib_net_ops = { - .init = nsim_fib_netns_init, - .id = &nsim_fib_net_id, - .size = sizeof(struct nsim_fib_data), -}; - -void nsim_fib_exit(void) -{ - unregister_pernet_subsys(&nsim_fib_net_ops); - unregister_fib_notifier(&nsim_fib_nb); -} - -int nsim_fib_init(void) -{ - int err; - - err = register_pernet_subsys(&nsim_fib_net_ops); - if (err < 0) { - pr_err("Failed to register pernet subsystem\n"); - goto err_out; - } - - err = register_fib_notifier(&nsim_fib_nb, nsim_fib_dump_inconsistent); - if (err < 0) { + data->fib_nb.notifier_call = nsim_fib_event_nb; + err = register_fib_notifier(&data->fib_nb, nsim_fib_dump_inconsistent); + if (err) { pr_err("Failed to register fib notifier\n"); goto err_out; } + return data; + err_out: - return err; + kfree(data); + return ERR_PTR(err); +} + +void nsim_fib_destroy(struct nsim_fib_data *data) +{ + unregister_fib_notifier(&data->fib_nb); + kfree(data); } diff --git a/drivers/net/netdevsim/ipsec.c b/drivers/net/netdevsim/ipsec.c index 76e11d889bb6eab360cf629d7945235a07ee23a9..e27fc1a4516dfaa1ad87819b9b7b0cc7f5adde47 100644 --- a/drivers/net/netdevsim/ipsec.c +++ b/drivers/net/netdevsim/ipsec.c @@ -283,7 +283,8 @@ void nsim_ipsec_init(struct netdevsim *ns) ns->netdev->features |= NSIM_ESP_FEATURES; ns->netdev->hw_enc_features |= NSIM_ESP_FEATURES; - ns->ipsec.pfile = debugfs_create_file("ipsec", 0400, ns->ddir, ns, + ns->ipsec.pfile = debugfs_create_file("ipsec", 0400, + ns->nsim_dev_port->ddir, ns, &ipsec_dbg_fops); } diff --git a/drivers/net/netdevsim/netdev.c b/drivers/net/netdevsim/netdev.c index 75a50b59cb8f2030de5d2833cc747897ee92f1dd..e5c8aa08e1cd4f7c9646cbb5a54a6ab8bffe80d0 100644 --- a/drivers/net/netdevsim/netdev.c +++ b/drivers/net/netdevsim/netdev.c @@ -25,230 +25,6 @@ #include "netdevsim.h" -struct nsim_vf_config { - int link_state; - u16 min_tx_rate; - u16 max_tx_rate; - u16 vlan; - __be16 vlan_proto; - u16 qos; - u8 vf_mac[ETH_ALEN]; - bool spoofchk_enabled; - bool trusted; - bool rss_query_enabled; -}; - -static u32 nsim_dev_id; - -static struct dentry *nsim_ddir; -static struct dentry *nsim_sdev_ddir; - -static int nsim_num_vf(struct device *dev) -{ - struct netdevsim *ns = to_nsim(dev); - - return ns->num_vfs; -} - -static struct bus_type nsim_bus = { - .name = DRV_NAME, - .dev_name = DRV_NAME, - .num_vf = nsim_num_vf, -}; - -static int nsim_vfs_enable(struct netdevsim *ns, unsigned int num_vfs) -{ - ns->vfconfigs = kcalloc(num_vfs, sizeof(struct nsim_vf_config), - GFP_KERNEL); - if (!ns->vfconfigs) - return -ENOMEM; - ns->num_vfs = num_vfs; - - return 0; -} - -static void nsim_vfs_disable(struct netdevsim *ns) -{ - kfree(ns->vfconfigs); - ns->vfconfigs = NULL; - ns->num_vfs = 0; -} - -static ssize_t -nsim_numvfs_store(struct device *dev, struct device_attribute *attr, - const char *buf, size_t count) -{ - struct netdevsim *ns = to_nsim(dev); - unsigned int num_vfs; - int ret; - - ret = kstrtouint(buf, 0, &num_vfs); - if (ret) - return ret; - - rtnl_lock(); - if (ns->num_vfs == num_vfs) - goto exit_good; - if (ns->num_vfs && num_vfs) { - ret = -EBUSY; - goto exit_unlock; - } - - if (num_vfs) { - ret = nsim_vfs_enable(ns, num_vfs); - if (ret) - goto exit_unlock; - } else { - nsim_vfs_disable(ns); - } -exit_good: - ret = count; -exit_unlock: - rtnl_unlock(); - - return ret; -} - -static ssize_t -nsim_numvfs_show(struct device *dev, struct device_attribute *attr, char *buf) -{ - struct netdevsim *ns = to_nsim(dev); - - return sprintf(buf, "%u\n", ns->num_vfs); -} - -static struct device_attribute nsim_numvfs_attr = - __ATTR(sriov_numvfs, 0664, nsim_numvfs_show, nsim_numvfs_store); - -static struct attribute *nsim_dev_attrs[] = { - &nsim_numvfs_attr.attr, - NULL, -}; - -static const struct attribute_group nsim_dev_attr_group = { - .attrs = nsim_dev_attrs, -}; - -static const struct attribute_group *nsim_dev_attr_groups[] = { - &nsim_dev_attr_group, - NULL, -}; - -static void nsim_dev_release(struct device *dev) -{ - struct netdevsim *ns = to_nsim(dev); - - nsim_vfs_disable(ns); - free_netdev(ns->netdev); -} - -static struct device_type nsim_dev_type = { - .groups = nsim_dev_attr_groups, - .release = nsim_dev_release, -}; - -static int nsim_get_port_parent_id(struct net_device *dev, - struct netdev_phys_item_id *ppid) -{ - struct netdevsim *ns = netdev_priv(dev); - - ppid->id_len = sizeof(ns->sdev->switch_id); - memcpy(&ppid->id, &ns->sdev->switch_id, ppid->id_len); - return 0; -} - -static int nsim_init(struct net_device *dev) -{ - char sdev_ddir_name[10], sdev_link_name[32]; - struct netdevsim *ns = netdev_priv(dev); - int err; - - ns->netdev = dev; - ns->ddir = debugfs_create_dir(netdev_name(dev), nsim_ddir); - if (IS_ERR_OR_NULL(ns->ddir)) - return -ENOMEM; - - if (!ns->sdev) { - ns->sdev = kzalloc(sizeof(*ns->sdev), GFP_KERNEL); - if (!ns->sdev) { - err = -ENOMEM; - goto err_debugfs_destroy; - } - ns->sdev->refcnt = 1; - ns->sdev->switch_id = nsim_dev_id; - sprintf(sdev_ddir_name, "%u", ns->sdev->switch_id); - ns->sdev->ddir = debugfs_create_dir(sdev_ddir_name, - nsim_sdev_ddir); - if (IS_ERR_OR_NULL(ns->sdev->ddir)) { - err = PTR_ERR_OR_ZERO(ns->sdev->ddir) ?: -EINVAL; - goto err_sdev_free; - } - } else { - sprintf(sdev_ddir_name, "%u", ns->sdev->switch_id); - ns->sdev->refcnt++; - } - - sprintf(sdev_link_name, "../../" DRV_NAME "_sdev/%s", sdev_ddir_name); - debugfs_create_symlink("sdev", ns->ddir, sdev_link_name); - - err = nsim_bpf_init(ns); - if (err) - goto err_sdev_destroy; - - ns->dev.id = nsim_dev_id++; - ns->dev.bus = &nsim_bus; - ns->dev.type = &nsim_dev_type; - err = device_register(&ns->dev); - if (err) - goto err_bpf_uninit; - - SET_NETDEV_DEV(dev, &ns->dev); - - err = nsim_devlink_setup(ns); - if (err) - goto err_unreg_dev; - - nsim_ipsec_init(ns); - - return 0; - -err_unreg_dev: - device_unregister(&ns->dev); -err_bpf_uninit: - nsim_bpf_uninit(ns); -err_sdev_destroy: - if (!--ns->sdev->refcnt) { - debugfs_remove_recursive(ns->sdev->ddir); -err_sdev_free: - kfree(ns->sdev); - } -err_debugfs_destroy: - debugfs_remove_recursive(ns->ddir); - return err; -} - -static void nsim_uninit(struct net_device *dev) -{ - struct netdevsim *ns = netdev_priv(dev); - - nsim_ipsec_teardown(ns); - nsim_devlink_teardown(ns); - debugfs_remove_recursive(ns->ddir); - nsim_bpf_uninit(ns); - if (!--ns->sdev->refcnt) { - debugfs_remove_recursive(ns->sdev->ddir); - kfree(ns->sdev); - } -} - -static void nsim_free(struct net_device *dev) -{ - struct netdevsim *ns = netdev_priv(dev); - - device_unregister(&ns->dev); - /* netdev and vf state will be freed out of device_release() */ -} - static netdev_tx_t nsim_start_xmit(struct sk_buff *skb, struct net_device *dev) { struct netdevsim *ns = netdev_priv(dev); @@ -325,11 +101,12 @@ nsim_setup_tc_block(struct net_device *dev, struct tc_block_offload *f) static int nsim_set_vf_mac(struct net_device *dev, int vf, u8 *mac) { struct netdevsim *ns = netdev_priv(dev); + struct nsim_bus_dev *nsim_bus_dev = ns->nsim_bus_dev; /* Only refuse multicast addresses, zero address can mean unset/any. */ - if (vf >= ns->num_vfs || is_multicast_ether_addr(mac)) + if (vf >= nsim_bus_dev->num_vfs || is_multicast_ether_addr(mac)) return -EINVAL; - memcpy(ns->vfconfigs[vf].vf_mac, mac, ETH_ALEN); + memcpy(nsim_bus_dev->vfconfigs[vf].vf_mac, mac, ETH_ALEN); return 0; } @@ -338,13 +115,14 @@ static int nsim_set_vf_vlan(struct net_device *dev, int vf, u16 vlan, u8 qos, __be16 vlan_proto) { struct netdevsim *ns = netdev_priv(dev); + struct nsim_bus_dev *nsim_bus_dev = ns->nsim_bus_dev; - if (vf >= ns->num_vfs || vlan > 4095 || qos > 7) + if (vf >= nsim_bus_dev->num_vfs || vlan > 4095 || qos > 7) return -EINVAL; - ns->vfconfigs[vf].vlan = vlan; - ns->vfconfigs[vf].qos = qos; - ns->vfconfigs[vf].vlan_proto = vlan_proto; + nsim_bus_dev->vfconfigs[vf].vlan = vlan; + nsim_bus_dev->vfconfigs[vf].qos = qos; + nsim_bus_dev->vfconfigs[vf].vlan_proto = vlan_proto; return 0; } @@ -352,12 +130,13 @@ static int nsim_set_vf_vlan(struct net_device *dev, int vf, static int nsim_set_vf_rate(struct net_device *dev, int vf, int min, int max) { struct netdevsim *ns = netdev_priv(dev); + struct nsim_bus_dev *nsim_bus_dev = ns->nsim_bus_dev; - if (vf >= ns->num_vfs) + if (vf >= nsim_bus_dev->num_vfs) return -EINVAL; - ns->vfconfigs[vf].min_tx_rate = min; - ns->vfconfigs[vf].max_tx_rate = max; + nsim_bus_dev->vfconfigs[vf].min_tx_rate = min; + nsim_bus_dev->vfconfigs[vf].max_tx_rate = max; return 0; } @@ -365,10 +144,11 @@ static int nsim_set_vf_rate(struct net_device *dev, int vf, int min, int max) static int nsim_set_vf_spoofchk(struct net_device *dev, int vf, bool val) { struct netdevsim *ns = netdev_priv(dev); + struct nsim_bus_dev *nsim_bus_dev = ns->nsim_bus_dev; - if (vf >= ns->num_vfs) + if (vf >= nsim_bus_dev->num_vfs) return -EINVAL; - ns->vfconfigs[vf].spoofchk_enabled = val; + nsim_bus_dev->vfconfigs[vf].spoofchk_enabled = val; return 0; } @@ -376,10 +156,11 @@ static int nsim_set_vf_spoofchk(struct net_device *dev, int vf, bool val) static int nsim_set_vf_rss_query_en(struct net_device *dev, int vf, bool val) { struct netdevsim *ns = netdev_priv(dev); + struct nsim_bus_dev *nsim_bus_dev = ns->nsim_bus_dev; - if (vf >= ns->num_vfs) + if (vf >= nsim_bus_dev->num_vfs) return -EINVAL; - ns->vfconfigs[vf].rss_query_enabled = val; + nsim_bus_dev->vfconfigs[vf].rss_query_enabled = val; return 0; } @@ -387,10 +168,11 @@ static int nsim_set_vf_rss_query_en(struct net_device *dev, int vf, bool val) static int nsim_set_vf_trust(struct net_device *dev, int vf, bool val) { struct netdevsim *ns = netdev_priv(dev); + struct nsim_bus_dev *nsim_bus_dev = ns->nsim_bus_dev; - if (vf >= ns->num_vfs) + if (vf >= nsim_bus_dev->num_vfs) return -EINVAL; - ns->vfconfigs[vf].trusted = val; + nsim_bus_dev->vfconfigs[vf].trusted = val; return 0; } @@ -399,21 +181,22 @@ static int nsim_get_vf_config(struct net_device *dev, int vf, struct ifla_vf_info *ivi) { struct netdevsim *ns = netdev_priv(dev); + struct nsim_bus_dev *nsim_bus_dev = ns->nsim_bus_dev; - if (vf >= ns->num_vfs) + if (vf >= nsim_bus_dev->num_vfs) return -EINVAL; ivi->vf = vf; - ivi->linkstate = ns->vfconfigs[vf].link_state; - ivi->min_tx_rate = ns->vfconfigs[vf].min_tx_rate; - ivi->max_tx_rate = ns->vfconfigs[vf].max_tx_rate; - ivi->vlan = ns->vfconfigs[vf].vlan; - ivi->vlan_proto = ns->vfconfigs[vf].vlan_proto; - ivi->qos = ns->vfconfigs[vf].qos; - memcpy(&ivi->mac, ns->vfconfigs[vf].vf_mac, ETH_ALEN); - ivi->spoofchk = ns->vfconfigs[vf].spoofchk_enabled; - ivi->trusted = ns->vfconfigs[vf].trusted; - ivi->rss_query_en = ns->vfconfigs[vf].rss_query_enabled; + ivi->linkstate = nsim_bus_dev->vfconfigs[vf].link_state; + ivi->min_tx_rate = nsim_bus_dev->vfconfigs[vf].min_tx_rate; + ivi->max_tx_rate = nsim_bus_dev->vfconfigs[vf].max_tx_rate; + ivi->vlan = nsim_bus_dev->vfconfigs[vf].vlan; + ivi->vlan_proto = nsim_bus_dev->vfconfigs[vf].vlan_proto; + ivi->qos = nsim_bus_dev->vfconfigs[vf].qos; + memcpy(&ivi->mac, nsim_bus_dev->vfconfigs[vf].vf_mac, ETH_ALEN); + ivi->spoofchk = nsim_bus_dev->vfconfigs[vf].spoofchk_enabled; + ivi->trusted = nsim_bus_dev->vfconfigs[vf].trusted; + ivi->rss_query_en = nsim_bus_dev->vfconfigs[vf].rss_query_enabled; return 0; } @@ -421,8 +204,9 @@ nsim_get_vf_config(struct net_device *dev, int vf, struct ifla_vf_info *ivi) static int nsim_set_vf_link_state(struct net_device *dev, int vf, int state) { struct netdevsim *ns = netdev_priv(dev); + struct nsim_bus_dev *nsim_bus_dev = ns->nsim_bus_dev; - if (vf >= ns->num_vfs) + if (vf >= nsim_bus_dev->num_vfs) return -EINVAL; switch (state) { @@ -434,7 +218,7 @@ static int nsim_set_vf_link_state(struct net_device *dev, int vf, int state) return -EINVAL; } - ns->vfconfigs[vf].link_state = state; + nsim_bus_dev->vfconfigs[vf].link_state = state; return 0; } @@ -461,9 +245,14 @@ nsim_set_features(struct net_device *dev, netdev_features_t features) return 0; } +static struct devlink_port *nsim_get_devlink_port(struct net_device *dev) +{ + struct netdevsim *ns = netdev_priv(dev); + + return &ns->nsim_dev_port->devlink_port; +} + static const struct net_device_ops nsim_netdev_ops = { - .ndo_init = nsim_init, - .ndo_uninit = nsim_uninit, .ndo_start_xmit = nsim_start_xmit, .ndo_set_rx_mode = nsim_set_rx_mode, .ndo_set_mac_address = eth_mac_addr, @@ -481,7 +270,7 @@ static const struct net_device_ops nsim_netdev_ops = { .ndo_setup_tc = nsim_setup_tc, .ndo_set_features = nsim_set_features, .ndo_bpf = nsim_bpf, - .ndo_get_port_parent_id = nsim_get_port_parent_id, + .ndo_get_devlink_port = nsim_get_devlink_port, }; static void nsim_setup(struct net_device *dev) @@ -489,9 +278,6 @@ static void nsim_setup(struct net_device *dev) ether_setup(dev); eth_hw_addr_random(dev); - dev->netdev_ops = &nsim_netdev_ops; - dev->priv_destructor = nsim_free; - dev->tx_queue_len = 0; dev->flags |= IFF_NOARP; dev->flags &= ~IFF_MULTICAST; @@ -506,104 +292,102 @@ static void nsim_setup(struct net_device *dev) dev->max_mtu = ETH_MAX_MTU; } -static int nsim_validate(struct nlattr *tb[], struct nlattr *data[], - struct netlink_ext_ack *extack) +struct netdevsim * +nsim_create(struct nsim_dev *nsim_dev, struct nsim_dev_port *nsim_dev_port) { - if (tb[IFLA_ADDRESS]) { - if (nla_len(tb[IFLA_ADDRESS]) != ETH_ALEN) - return -EINVAL; - if (!is_valid_ether_addr(nla_data(tb[IFLA_ADDRESS]))) - return -EADDRNOTAVAIL; - } - return 0; + struct net_device *dev; + struct netdevsim *ns; + int err; + + dev = alloc_netdev(sizeof(*ns), "eth%d", NET_NAME_UNKNOWN, nsim_setup); + if (!dev) + return ERR_PTR(-ENOMEM); + + ns = netdev_priv(dev); + ns->netdev = dev; + ns->nsim_dev = nsim_dev; + ns->nsim_dev_port = nsim_dev_port; + ns->nsim_bus_dev = nsim_dev->nsim_bus_dev; + SET_NETDEV_DEV(dev, &ns->nsim_bus_dev->dev); + dev->netdev_ops = &nsim_netdev_ops; + + rtnl_lock(); + err = nsim_bpf_init(ns); + if (err) + goto err_free_netdev; + + nsim_ipsec_init(ns); + + err = register_netdevice(dev); + if (err) + goto err_ipsec_teardown; + rtnl_unlock(); + + return ns; + +err_ipsec_teardown: + nsim_ipsec_teardown(ns); + nsim_bpf_uninit(ns); + rtnl_unlock(); +err_free_netdev: + free_netdev(dev); + return ERR_PTR(err); } -static int nsim_newlink(struct net *src_net, struct net_device *dev, - struct nlattr *tb[], struct nlattr *data[], - struct netlink_ext_ack *extack) +void nsim_destroy(struct netdevsim *ns) { - struct netdevsim *ns = netdev_priv(dev); - - if (tb[IFLA_LINK]) { - struct net_device *joindev; - struct netdevsim *joinns; - - joindev = __dev_get_by_index(src_net, - nla_get_u32(tb[IFLA_LINK])); - if (!joindev) - return -ENODEV; - if (joindev->netdev_ops != &nsim_netdev_ops) - return -EINVAL; - - joinns = netdev_priv(joindev); - if (!joinns->sdev || !joinns->sdev->refcnt) - return -EINVAL; - ns->sdev = joinns->sdev; - } + struct net_device *dev = ns->netdev; - return register_netdevice(dev); + rtnl_lock(); + unregister_netdevice(dev); + nsim_ipsec_teardown(ns); + nsim_bpf_uninit(ns); + rtnl_unlock(); + free_netdev(dev); } -static void nsim_dellink(struct net_device *dev, struct list_head *head) +static int nsim_validate(struct nlattr *tb[], struct nlattr *data[], + struct netlink_ext_ack *extack) { - unregister_netdevice_queue(dev, head); + NL_SET_ERR_MSG_MOD(extack, "Please use: echo \"[ID] [PORT_COUNT]\" > /sys/bus/netdevsim/new_device"); + return -EOPNOTSUPP; } static struct rtnl_link_ops nsim_link_ops __read_mostly = { .kind = DRV_NAME, - .priv_size = sizeof(struct netdevsim), - .setup = nsim_setup, .validate = nsim_validate, - .newlink = nsim_newlink, - .dellink = nsim_dellink, }; static int __init nsim_module_init(void) { int err; - nsim_ddir = debugfs_create_dir(DRV_NAME, NULL); - if (IS_ERR_OR_NULL(nsim_ddir)) - return -ENOMEM; - - nsim_sdev_ddir = debugfs_create_dir(DRV_NAME "_sdev", NULL); - if (IS_ERR_OR_NULL(nsim_sdev_ddir)) { - err = -ENOMEM; - goto err_debugfs_destroy; - } - - err = bus_register(&nsim_bus); + err = nsim_dev_init(); if (err) - goto err_sdir_destroy; + return err; - err = nsim_devlink_init(); + err = nsim_bus_init(); if (err) - goto err_unreg_bus; + goto err_dev_exit; err = rtnl_link_register(&nsim_link_ops); if (err) - goto err_dl_fini; + goto err_bus_exit; return 0; -err_dl_fini: - nsim_devlink_exit(); -err_unreg_bus: - bus_unregister(&nsim_bus); -err_sdir_destroy: - debugfs_remove_recursive(nsim_sdev_ddir); -err_debugfs_destroy: - debugfs_remove_recursive(nsim_ddir); +err_bus_exit: + nsim_bus_exit(); +err_dev_exit: + nsim_dev_exit(); return err; } static void __exit nsim_module_exit(void) { rtnl_link_unregister(&nsim_link_ops); - nsim_devlink_exit(); - bus_unregister(&nsim_bus); - debugfs_remove_recursive(nsim_sdev_ddir); - debugfs_remove_recursive(nsim_ddir); + nsim_bus_exit(); + nsim_dev_exit(); } module_init(nsim_module_init); diff --git a/drivers/net/netdevsim/netdevsim.h b/drivers/net/netdevsim/netdevsim.h index 384c254fafc5c0513c54851f3e88c030d3e0847d..3f398797c2bc7855f6ae05baee28b25507463500 100644 --- a/drivers/net/netdevsim/netdevsim.h +++ b/drivers/net/netdevsim/netdevsim.h @@ -18,6 +18,7 @@ #include #include #include +#include #include #define DRV_NAME "netdevsim" @@ -26,26 +27,6 @@ #define NSIM_EA(extack, msg) NL_SET_ERR_MSG_MOD((extack), msg) -struct bpf_prog; -struct bpf_offload_dev; -struct dentry; -struct nsim_vf_config; - -struct netdevsim_shared_dev { - unsigned int refcnt; - u32 switch_id; - - struct dentry *ddir; - - struct bpf_offload_dev *bpf_dev; - - struct dentry *ddir_bpf_bound_progs; - u32 prog_id_gen; - - struct list_head bpf_bound_progs; - struct list_head bpf_bound_maps; -}; - #define NSIM_IPSEC_MAX_SA_COUNT 33 #define NSIM_IPSEC_VALID BIT(31) @@ -69,18 +50,14 @@ struct nsim_ipsec { struct netdevsim { struct net_device *netdev; + struct nsim_dev *nsim_dev; + struct nsim_dev_port *nsim_dev_port; u64 tx_packets; u64 tx_bytes; struct u64_stats_sync syncp; - struct device dev; - struct netdevsim_shared_dev *sdev; - - struct dentry *ddir; - - unsigned int num_vfs; - struct nsim_vf_config *vfconfigs; + struct nsim_bus_dev *nsim_bus_dev; struct bpf_prog *bpf_offloaded; u32 bpf_offloaded_id; @@ -88,22 +65,22 @@ struct netdevsim { struct xdp_attachment_info xdp; struct xdp_attachment_info xdp_hw; - bool bpf_bind_accept; - u32 bpf_bind_verifier_delay; - bool bpf_tc_accept; bool bpf_tc_non_bound_accept; bool bpf_xdpdrv_accept; bool bpf_xdpoffload_accept; bool bpf_map_accept; -#if IS_ENABLED(CONFIG_NET_DEVLINK) - struct devlink *devlink; -#endif struct nsim_ipsec ipsec; }; +struct netdevsim * +nsim_create(struct nsim_dev *nsim_dev, struct nsim_dev_port *nsim_dev_port); +void nsim_destroy(struct netdevsim *ns); + #ifdef CONFIG_BPF_SYSCALL +int nsim_bpf_dev_init(struct nsim_dev *nsim_dev); +void nsim_bpf_dev_exit(struct nsim_dev *nsim_dev); int nsim_bpf_init(struct netdevsim *ns); void nsim_bpf_uninit(struct netdevsim *ns); int nsim_bpf(struct net_device *dev, struct netdev_bpf *bpf); @@ -111,6 +88,15 @@ int nsim_bpf_disable_tc(struct netdevsim *ns); int nsim_bpf_setup_tc_block_cb(enum tc_setup_type type, void *type_data, void *cb_priv); #else + +static inline int nsim_bpf_dev_init(struct nsim_dev *nsim_dev) +{ + return 0; +} + +static inline void nsim_bpf_dev_exit(struct nsim_dev *nsim_dev) +{ +} static inline int nsim_bpf_init(struct netdevsim *ns) { return 0; @@ -138,7 +124,6 @@ nsim_bpf_setup_tc_block_cb(enum tc_setup_type type, void *type_data, } #endif -#if IS_ENABLED(CONFIG_NET_DEVLINK) enum nsim_resource_id { NSIM_RESOURCE_NONE, /* DEVLINK_RESOURCE_ID_PARENT_TOP */ NSIM_RESOURCE_IPV4, @@ -149,36 +134,47 @@ enum nsim_resource_id { NSIM_RESOURCE_IPV6_FIB_RULES, }; -int nsim_devlink_setup(struct netdevsim *ns); -void nsim_devlink_teardown(struct netdevsim *ns); +struct nsim_dev_port { + struct list_head list; + struct devlink_port devlink_port; + unsigned int port_index; + struct dentry *ddir; + struct netdevsim *ns; +}; -int nsim_devlink_init(void); -void nsim_devlink_exit(void); +struct nsim_dev { + struct nsim_bus_dev *nsim_bus_dev; + struct nsim_fib_data *fib_data; + struct dentry *ddir; + struct dentry *ports_ddir; + struct bpf_offload_dev *bpf_dev; + bool bpf_bind_accept; + u32 bpf_bind_verifier_delay; + struct dentry *ddir_bpf_bound_progs; + u32 prog_id_gen; + struct list_head bpf_bound_progs; + struct list_head bpf_bound_maps; + struct netdev_phys_item_id switch_id; + struct list_head port_list; + struct mutex port_list_lock; /* protects port list */ +}; -int nsim_fib_init(void); -void nsim_fib_exit(void); -u64 nsim_fib_get_val(struct net *net, enum nsim_resource_id res_id, bool max); -int nsim_fib_set_max(struct net *net, enum nsim_resource_id res_id, u64 val, +int nsim_dev_init(void); +void nsim_dev_exit(void); +int nsim_dev_probe(struct nsim_bus_dev *nsim_bus_dev); +void nsim_dev_remove(struct nsim_bus_dev *nsim_bus_dev); +int nsim_dev_port_add(struct nsim_bus_dev *nsim_bus_dev, + unsigned int port_index); +int nsim_dev_port_del(struct nsim_bus_dev *nsim_bus_dev, + unsigned int port_index); + +struct nsim_fib_data *nsim_fib_create(void); +void nsim_fib_destroy(struct nsim_fib_data *fib_data); +u64 nsim_fib_get_val(struct nsim_fib_data *fib_data, + enum nsim_resource_id res_id, bool max); +int nsim_fib_set_max(struct nsim_fib_data *fib_data, + enum nsim_resource_id res_id, u64 val, struct netlink_ext_ack *extack); -#else -static inline int nsim_devlink_setup(struct netdevsim *ns) -{ - return 0; -} - -static inline void nsim_devlink_teardown(struct netdevsim *ns) -{ -} - -static inline int nsim_devlink_init(void) -{ - return 0; -} - -static inline void nsim_devlink_exit(void) -{ -} -#endif #if IS_ENABLED(CONFIG_XFRM_OFFLOAD) void nsim_ipsec_init(struct netdevsim *ns); @@ -199,7 +195,26 @@ static inline bool nsim_ipsec_tx(struct netdevsim *ns, struct sk_buff *skb) } #endif -static inline struct netdevsim *to_nsim(struct device *ptr) -{ - return container_of(ptr, struct netdevsim, dev); -} +struct nsim_vf_config { + int link_state; + u16 min_tx_rate; + u16 max_tx_rate; + u16 vlan; + __be16 vlan_proto; + u16 qos; + u8 vf_mac[ETH_ALEN]; + bool spoofchk_enabled; + bool trusted; + bool rss_query_enabled; +}; + +struct nsim_bus_dev { + struct device dev; + struct list_head list; + unsigned int port_count; + unsigned int num_vfs; + struct nsim_vf_config *vfconfigs; +}; + +int nsim_bus_init(void); +void nsim_bus_exit(void); diff --git a/drivers/net/netdevsim/sdev.c b/drivers/net/netdevsim/sdev.c new file mode 100644 index 0000000000000000000000000000000000000000..6712da3340d662835a6864a0ce3fba35db5a0ec9 --- /dev/null +++ b/drivers/net/netdevsim/sdev.c @@ -0,0 +1,69 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2019 Mellanox Technologies. All rights reserved */ + +#include +#include +#include +#include + +#include "netdevsim.h" + +static struct dentry *nsim_sdev_ddir; + +static u32 nsim_sdev_id; + +struct netdevsim_shared_dev *nsim_sdev_get(struct netdevsim *joinns) +{ + struct netdevsim_shared_dev *sdev; + char sdev_ddir_name[10]; + int err; + + if (joinns) { + if (WARN_ON(!joinns->sdev)) + return ERR_PTR(-EINVAL); + sdev = joinns->sdev; + sdev->refcnt++; + return sdev; + } + + sdev = kzalloc(sizeof(*sdev), GFP_KERNEL); + if (!sdev) + return ERR_PTR(-ENOMEM); + sdev->refcnt = 1; + sdev->switch_id = nsim_sdev_id++; + + sprintf(sdev_ddir_name, "%u", sdev->switch_id); + sdev->ddir = debugfs_create_dir(sdev_ddir_name, nsim_sdev_ddir); + if (IS_ERR_OR_NULL(sdev->ddir)) { + err = PTR_ERR_OR_ZERO(sdev->ddir) ?: -EINVAL; + goto err_sdev_free; + } + + return sdev; + +err_sdev_free: + nsim_sdev_id--; + kfree(sdev); + return ERR_PTR(err); +} + +void nsim_sdev_put(struct netdevsim_shared_dev *sdev) +{ + if (--sdev->refcnt) + return; + debugfs_remove_recursive(sdev->ddir); + kfree(sdev); +} + +int nsim_sdev_init(void) +{ + nsim_sdev_ddir = debugfs_create_dir(DRV_NAME "_sdev", NULL); + if (IS_ERR_OR_NULL(nsim_sdev_ddir)) + return -ENOMEM; + return 0; +} + +void nsim_sdev_exit(void) +{ + debugfs_remove_recursive(nsim_sdev_ddir); +} diff --git a/drivers/net/phy/Kconfig b/drivers/net/phy/Kconfig index 520657945b8279debe7583d33c31c095899ddb7a..d6299710d634b0a3e54bcc315bd60cda62913aee 100644 --- a/drivers/net/phy/Kconfig +++ b/drivers/net/phy/Kconfig @@ -76,6 +76,17 @@ config MDIO_BUS_MUX_GPIO several child MDIO busses to a parent bus. Child bus selection is under the control of GPIO lines. +config MDIO_BUS_MUX_MESON_G12A + tristate "Amlogic G12a based MDIO bus multiplexer" + depends on ARCH_MESON || COMPILE_TEST + depends on OF_MDIO && HAS_IOMEM && COMMON_CLK + select MDIO_BUS_MUX + default m if ARCH_MESON + help + This module provides a driver for the MDIO multiplexer/glue of + the amlogic g12a SoC. The multiplexers connects either the external + or the internal MDIO bus to the parent bus. + config MDIO_BUS_MUX_MMIOREG tristate "MMIO device-controlled MDIO bus multiplexers" depends on OF_MDIO && HAS_IOMEM @@ -273,13 +284,13 @@ config BCM87XX_PHY Currently supports the BCM8706 and BCM8727 10G Ethernet PHYs. config BCM_CYGNUS_PHY - tristate "Broadcom Cygnus SoC internal PHY" - depends on ARCH_BCM_CYGNUS || COMPILE_TEST + tristate "Broadcom Cygnus/Omega SoC internal PHY" + depends on ARCH_BCM_IPROC || COMPILE_TEST depends on MDIO_BCM_IPROC select BCM_NET_PHYLIB ---help--- This PHY driver is for the 1G internal PHYs of the Broadcom - Cygnus Family SoC. + Cygnus and Omega Family SoC. Currently supports internal PHY's used in the BCM11300, BCM11320, BCM11350, BCM11360, BCM58300, BCM58302, @@ -397,7 +408,7 @@ config MICROCHIP_T1_PHY config MICROSEMI_PHY tristate "Microsemi PHYs" ---help--- - Currently supports VSC8530, VSC8531, VSC8540 and VSC8541 PHYs + Currently supports VSC8514, VSC8530, VSC8531, VSC8540 and VSC8541 PHYs config NATIONAL_PHY tristate "National Semiconductor PHYs" diff --git a/drivers/net/phy/Makefile b/drivers/net/phy/Makefile index ece5dae67174f87362e7950fcd4171b3c0c63d65..27d7f9f3b0de4c820ce1fdb6f7b1237df3601f5a 100644 --- a/drivers/net/phy/Makefile +++ b/drivers/net/phy/Makefile @@ -28,6 +28,7 @@ obj-$(CONFIG_MDIO_BITBANG) += mdio-bitbang.o obj-$(CONFIG_MDIO_BUS_MUX) += mdio-mux.o obj-$(CONFIG_MDIO_BUS_MUX_BCM_IPROC) += mdio-mux-bcm-iproc.o obj-$(CONFIG_MDIO_BUS_MUX_GPIO) += mdio-mux-gpio.o +obj-$(CONFIG_MDIO_BUS_MUX_MESON_G12A) += mdio-mux-meson-g12a.o obj-$(CONFIG_MDIO_BUS_MUX_MMIOREG) += mdio-mux-mmioreg.o obj-$(CONFIG_MDIO_BUS_MUX_MULTIPLEXER) += mdio-mux-multiplexer.o obj-$(CONFIG_MDIO_CAVIUM) += mdio-cavium.o diff --git a/drivers/net/phy/amd.c b/drivers/net/phy/amd.c index 65b4b0960b1e03c9a0c047db2e0f6c96811feec6..eef35f8c8d450cf0118202eda126866a80d8ee0e 100644 --- a/drivers/net/phy/amd.c +++ b/drivers/net/phy/amd.c @@ -60,7 +60,7 @@ static struct phy_driver am79c_driver[] = { { .phy_id = PHY_ID_AM79C874, .name = "AM79C874", .phy_id_mask = 0xfffffff0, - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = am79c_config_init, .ack_interrupt = am79c_ack_interrupt, .config_intr = am79c_config_intr, diff --git a/drivers/net/phy/aquantia_main.c b/drivers/net/phy/aquantia_main.c index 37218e5d7cc9525321984fa9dcae8cef8f2c6878..eed4fe3d871f2c477555f6f4eb8d8be6f5e73c48 100644 --- a/drivers/net/phy/aquantia_main.c +++ b/drivers/net/phy/aquantia_main.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include "aquantia.h" @@ -22,20 +23,33 @@ #define PHY_ID_AQCS109 0x03a1b5c2 #define PHY_ID_AQR405 0x03a1b4b0 +#define MDIO_PHYXS_VEND_IF_STATUS 0xe812 +#define MDIO_PHYXS_VEND_IF_STATUS_TYPE_MASK GENMASK(7, 3) +#define MDIO_PHYXS_VEND_IF_STATUS_TYPE_KR 0 +#define MDIO_PHYXS_VEND_IF_STATUS_TYPE_XFI 2 +#define MDIO_PHYXS_VEND_IF_STATUS_TYPE_SGMII 6 +#define MDIO_PHYXS_VEND_IF_STATUS_TYPE_OCSGMII 10 + #define MDIO_AN_VEND_PROV 0xc400 #define MDIO_AN_VEND_PROV_1000BASET_FULL BIT(15) #define MDIO_AN_VEND_PROV_1000BASET_HALF BIT(14) +#define MDIO_AN_VEND_PROV_DOWNSHIFT_EN BIT(4) +#define MDIO_AN_VEND_PROV_DOWNSHIFT_MASK GENMASK(3, 0) +#define MDIO_AN_VEND_PROV_DOWNSHIFT_DFLT 4 #define MDIO_AN_TX_VEND_STATUS1 0xc800 -#define MDIO_AN_TX_VEND_STATUS1_10BASET (0x0 << 1) -#define MDIO_AN_TX_VEND_STATUS1_100BASETX (0x1 << 1) -#define MDIO_AN_TX_VEND_STATUS1_1000BASET (0x2 << 1) -#define MDIO_AN_TX_VEND_STATUS1_10GBASET (0x3 << 1) -#define MDIO_AN_TX_VEND_STATUS1_2500BASET (0x4 << 1) -#define MDIO_AN_TX_VEND_STATUS1_5000BASET (0x5 << 1) -#define MDIO_AN_TX_VEND_STATUS1_RATE_MASK (0x7 << 1) +#define MDIO_AN_TX_VEND_STATUS1_RATE_MASK GENMASK(3, 1) +#define MDIO_AN_TX_VEND_STATUS1_10BASET 0 +#define MDIO_AN_TX_VEND_STATUS1_100BASETX 1 +#define MDIO_AN_TX_VEND_STATUS1_1000BASET 2 +#define MDIO_AN_TX_VEND_STATUS1_10GBASET 3 +#define MDIO_AN_TX_VEND_STATUS1_2500BASET 4 +#define MDIO_AN_TX_VEND_STATUS1_5000BASET 5 #define MDIO_AN_TX_VEND_STATUS1_FULL_DUPLEX BIT(0) +#define MDIO_AN_TX_VEND_INT_STATUS1 0xcc00 +#define MDIO_AN_TX_VEND_INT_STATUS1_DOWNSHIFT BIT(1) + #define MDIO_AN_TX_VEND_INT_STATUS2 0xcc01 #define MDIO_AN_TX_VEND_INT_MASK2 0xd401 @@ -44,8 +58,42 @@ #define MDIO_AN_RX_LP_STAT1 0xe820 #define MDIO_AN_RX_LP_STAT1_1000BASET_FULL BIT(15) #define MDIO_AN_RX_LP_STAT1_1000BASET_HALF BIT(14) +#define MDIO_AN_RX_LP_STAT1_SHORT_REACH BIT(13) +#define MDIO_AN_RX_LP_STAT1_AQRATE_DOWNSHIFT BIT(12) +#define MDIO_AN_RX_LP_STAT1_AQ_PHY BIT(2) + +#define MDIO_AN_RX_LP_STAT4 0xe823 +#define MDIO_AN_RX_LP_STAT4_FW_MAJOR GENMASK(15, 8) +#define MDIO_AN_RX_LP_STAT4_FW_MINOR GENMASK(7, 0) + +#define MDIO_AN_RX_VEND_STAT3 0xe832 +#define MDIO_AN_RX_VEND_STAT3_AFR BIT(0) + +/* MDIO_MMD_C22EXT */ +#define MDIO_C22EXT_STAT_SGMII_RX_GOOD_FRAMES 0xd292 +#define MDIO_C22EXT_STAT_SGMII_RX_BAD_FRAMES 0xd294 +#define MDIO_C22EXT_STAT_SGMII_RX_FALSE_CARRIER 0xd297 +#define MDIO_C22EXT_STAT_SGMII_TX_GOOD_FRAMES 0xd313 +#define MDIO_C22EXT_STAT_SGMII_TX_BAD_FRAMES 0xd315 +#define MDIO_C22EXT_STAT_SGMII_TX_FALSE_CARRIER 0xd317 +#define MDIO_C22EXT_STAT_SGMII_TX_COLLISIONS 0xd318 +#define MDIO_C22EXT_STAT_SGMII_TX_LINE_COLLISIONS 0xd319 +#define MDIO_C22EXT_STAT_SGMII_TX_FRAME_ALIGN_ERR 0xd31a +#define MDIO_C22EXT_STAT_SGMII_TX_RUNT_FRAMES 0xd31b /* Vendor specific 1, MDIO_MMD_VEND1 */ +#define VEND1_GLOBAL_FW_ID 0x0020 +#define VEND1_GLOBAL_FW_ID_MAJOR GENMASK(15, 8) +#define VEND1_GLOBAL_FW_ID_MINOR GENMASK(7, 0) + +#define VEND1_GLOBAL_RSVD_STAT1 0xc885 +#define VEND1_GLOBAL_RSVD_STAT1_FW_BUILD_ID GENMASK(7, 4) +#define VEND1_GLOBAL_RSVD_STAT1_PROV_ID GENMASK(3, 0) + +#define VEND1_GLOBAL_RSVD_STAT9 0xc88d +#define VEND1_GLOBAL_RSVD_STAT9_MODE GENMASK(7, 0) +#define VEND1_GLOBAL_RSVD_STAT9_1000BT2 0x23 + #define VEND1_GLOBAL_INT_STD_STATUS 0xfc00 #define VEND1_GLOBAL_INT_VEND_STATUS 0xfc01 @@ -72,6 +120,88 @@ #define VEND1_GLOBAL_INT_VEND_MASK_GLOBAL2 BIT(1) #define VEND1_GLOBAL_INT_VEND_MASK_GLOBAL3 BIT(0) +struct aqr107_hw_stat { + const char *name; + int reg; + int size; +}; + +#define SGMII_STAT(n, r, s) { n, MDIO_C22EXT_STAT_SGMII_ ## r, s } +static const struct aqr107_hw_stat aqr107_hw_stats[] = { + SGMII_STAT("sgmii_rx_good_frames", RX_GOOD_FRAMES, 26), + SGMII_STAT("sgmii_rx_bad_frames", RX_BAD_FRAMES, 26), + SGMII_STAT("sgmii_rx_false_carrier_events", RX_FALSE_CARRIER, 8), + SGMII_STAT("sgmii_tx_good_frames", TX_GOOD_FRAMES, 26), + SGMII_STAT("sgmii_tx_bad_frames", TX_BAD_FRAMES, 26), + SGMII_STAT("sgmii_tx_false_carrier_events", TX_FALSE_CARRIER, 8), + SGMII_STAT("sgmii_tx_collisions", TX_COLLISIONS, 8), + SGMII_STAT("sgmii_tx_line_collisions", TX_LINE_COLLISIONS, 8), + SGMII_STAT("sgmii_tx_frame_alignment_err", TX_FRAME_ALIGN_ERR, 16), + SGMII_STAT("sgmii_tx_runt_frames", TX_RUNT_FRAMES, 22), +}; +#define AQR107_SGMII_STAT_SZ ARRAY_SIZE(aqr107_hw_stats) + +struct aqr107_priv { + u64 sgmii_stats[AQR107_SGMII_STAT_SZ]; +}; + +static int aqr107_get_sset_count(struct phy_device *phydev) +{ + return AQR107_SGMII_STAT_SZ; +} + +static void aqr107_get_strings(struct phy_device *phydev, u8 *data) +{ + int i; + + for (i = 0; i < AQR107_SGMII_STAT_SZ; i++) + strscpy(data + i * ETH_GSTRING_LEN, aqr107_hw_stats[i].name, + ETH_GSTRING_LEN); +} + +static u64 aqr107_get_stat(struct phy_device *phydev, int index) +{ + const struct aqr107_hw_stat *stat = aqr107_hw_stats + index; + int len_l = min(stat->size, 16); + int len_h = stat->size - len_l; + u64 ret; + int val; + + val = phy_read_mmd(phydev, MDIO_MMD_C22EXT, stat->reg); + if (val < 0) + return U64_MAX; + + ret = val & GENMASK(len_l - 1, 0); + if (len_h) { + val = phy_read_mmd(phydev, MDIO_MMD_C22EXT, stat->reg + 1); + if (val < 0) + return U64_MAX; + + ret += (val & GENMASK(len_h - 1, 0)) << 16; + } + + return ret; +} + +static void aqr107_get_stats(struct phy_device *phydev, + struct ethtool_stats *stats, u64 *data) +{ + struct aqr107_priv *priv = phydev->priv; + u64 val; + int i; + + for (i = 0; i < AQR107_SGMII_STAT_SZ; i++) { + val = aqr107_get_stat(phydev, i); + if (val == U64_MAX) + phydev_err(phydev, "Reading HW Statistics failed for %s\n", + aqr107_hw_stats[i].name); + else + priv->sgmii_stats[i] += val; + + data[i] = priv->sgmii_stats[i]; + } +} + static int aqr_config_aneg(struct phy_device *phydev) { bool changed = false; @@ -112,41 +242,22 @@ static int aqr_config_aneg(struct phy_device *phydev) static int aqr_config_intr(struct phy_device *phydev) { + bool en = phydev->interrupts == PHY_INTERRUPT_ENABLED; int err; - if (phydev->interrupts == PHY_INTERRUPT_ENABLED) { - err = phy_write_mmd(phydev, MDIO_MMD_AN, - MDIO_AN_TX_VEND_INT_MASK2, - MDIO_AN_TX_VEND_INT_MASK2_LINK); - if (err < 0) - return err; - - err = phy_write_mmd(phydev, MDIO_MMD_VEND1, - VEND1_GLOBAL_INT_STD_MASK, - VEND1_GLOBAL_INT_STD_MASK_ALL); - if (err < 0) - return err; - - err = phy_write_mmd(phydev, MDIO_MMD_VEND1, - VEND1_GLOBAL_INT_VEND_MASK, - VEND1_GLOBAL_INT_VEND_MASK_GLOBAL3 | - VEND1_GLOBAL_INT_VEND_MASK_AN); - } else { - err = phy_write_mmd(phydev, MDIO_MMD_AN, - MDIO_AN_TX_VEND_INT_MASK2, 0); - if (err < 0) - return err; - - err = phy_write_mmd(phydev, MDIO_MMD_VEND1, - VEND1_GLOBAL_INT_STD_MASK, 0); - if (err < 0) - return err; - - err = phy_write_mmd(phydev, MDIO_MMD_VEND1, - VEND1_GLOBAL_INT_VEND_MASK, 0); - } + err = phy_write_mmd(phydev, MDIO_MMD_AN, MDIO_AN_TX_VEND_INT_MASK2, + en ? MDIO_AN_TX_VEND_INT_MASK2_LINK : 0); + if (err < 0) + return err; + + err = phy_write_mmd(phydev, MDIO_MMD_VEND1, VEND1_GLOBAL_INT_STD_MASK, + en ? VEND1_GLOBAL_INT_STD_MASK_ALL : 0); + if (err < 0) + return err; - return err; + return phy_write_mmd(phydev, MDIO_MMD_VEND1, VEND1_GLOBAL_INT_VEND_MASK, + en ? VEND1_GLOBAL_INT_VEND_MASK_GLOBAL3 | + VEND1_GLOBAL_INT_VEND_MASK_AN : 0); } static int aqr_ack_interrupt(struct phy_device *phydev) @@ -178,21 +289,315 @@ static int aqr_read_status(struct phy_device *phydev) return genphy_c45_read_status(phydev); } +static int aqr107_read_downshift_event(struct phy_device *phydev) +{ + int val; + + val = phy_read_mmd(phydev, MDIO_MMD_AN, MDIO_AN_TX_VEND_INT_STATUS1); + if (val < 0) + return val; + + return !!(val & MDIO_AN_TX_VEND_INT_STATUS1_DOWNSHIFT); +} + +static int aqr107_read_rate(struct phy_device *phydev) +{ + int val; + + val = phy_read_mmd(phydev, MDIO_MMD_AN, MDIO_AN_TX_VEND_STATUS1); + if (val < 0) + return val; + + switch (FIELD_GET(MDIO_AN_TX_VEND_STATUS1_RATE_MASK, val)) { + case MDIO_AN_TX_VEND_STATUS1_10BASET: + phydev->speed = SPEED_10; + break; + case MDIO_AN_TX_VEND_STATUS1_100BASETX: + phydev->speed = SPEED_100; + break; + case MDIO_AN_TX_VEND_STATUS1_1000BASET: + phydev->speed = SPEED_1000; + break; + case MDIO_AN_TX_VEND_STATUS1_2500BASET: + phydev->speed = SPEED_2500; + break; + case MDIO_AN_TX_VEND_STATUS1_5000BASET: + phydev->speed = SPEED_5000; + break; + case MDIO_AN_TX_VEND_STATUS1_10GBASET: + phydev->speed = SPEED_10000; + break; + default: + phydev->speed = SPEED_UNKNOWN; + break; + } + + if (val & MDIO_AN_TX_VEND_STATUS1_FULL_DUPLEX) + phydev->duplex = DUPLEX_FULL; + else + phydev->duplex = DUPLEX_HALF; + + return 0; +} + +static int aqr107_read_status(struct phy_device *phydev) +{ + int val, ret; + + ret = aqr_read_status(phydev); + if (ret) + return ret; + + if (!phydev->link || phydev->autoneg == AUTONEG_DISABLE) + return 0; + + val = phy_read_mmd(phydev, MDIO_MMD_PHYXS, MDIO_PHYXS_VEND_IF_STATUS); + if (val < 0) + return val; + + switch (FIELD_GET(MDIO_PHYXS_VEND_IF_STATUS_TYPE_MASK, val)) { + case MDIO_PHYXS_VEND_IF_STATUS_TYPE_KR: + case MDIO_PHYXS_VEND_IF_STATUS_TYPE_XFI: + phydev->interface = PHY_INTERFACE_MODE_10GKR; + break; + case MDIO_PHYXS_VEND_IF_STATUS_TYPE_SGMII: + phydev->interface = PHY_INTERFACE_MODE_SGMII; + break; + case MDIO_PHYXS_VEND_IF_STATUS_TYPE_OCSGMII: + phydev->interface = PHY_INTERFACE_MODE_2500BASEX; + break; + default: + phydev->interface = PHY_INTERFACE_MODE_NA; + break; + } + + val = aqr107_read_downshift_event(phydev); + if (val <= 0) + return val; + + phydev_warn(phydev, "Downshift occurred! Cabling may be defective.\n"); + + /* Read downshifted rate from vendor register */ + return aqr107_read_rate(phydev); +} + +static int aqr107_get_downshift(struct phy_device *phydev, u8 *data) +{ + int val, cnt, enable; + + val = phy_read_mmd(phydev, MDIO_MMD_AN, MDIO_AN_VEND_PROV); + if (val < 0) + return val; + + enable = FIELD_GET(MDIO_AN_VEND_PROV_DOWNSHIFT_EN, val); + cnt = FIELD_GET(MDIO_AN_VEND_PROV_DOWNSHIFT_MASK, val); + + *data = enable && cnt ? cnt : DOWNSHIFT_DEV_DISABLE; + + return 0; +} + +static int aqr107_set_downshift(struct phy_device *phydev, u8 cnt) +{ + int val = 0; + + if (!FIELD_FIT(MDIO_AN_VEND_PROV_DOWNSHIFT_MASK, cnt)) + return -E2BIG; + + if (cnt != DOWNSHIFT_DEV_DISABLE) { + val = MDIO_AN_VEND_PROV_DOWNSHIFT_EN; + val |= FIELD_PREP(MDIO_AN_VEND_PROV_DOWNSHIFT_MASK, cnt); + } + + return phy_modify_mmd(phydev, MDIO_MMD_AN, MDIO_AN_VEND_PROV, + MDIO_AN_VEND_PROV_DOWNSHIFT_EN | + MDIO_AN_VEND_PROV_DOWNSHIFT_MASK, val); +} + +static int aqr107_get_tunable(struct phy_device *phydev, + struct ethtool_tunable *tuna, void *data) +{ + switch (tuna->id) { + case ETHTOOL_PHY_DOWNSHIFT: + return aqr107_get_downshift(phydev, data); + default: + return -EOPNOTSUPP; + } +} + +static int aqr107_set_tunable(struct phy_device *phydev, + struct ethtool_tunable *tuna, const void *data) +{ + switch (tuna->id) { + case ETHTOOL_PHY_DOWNSHIFT: + return aqr107_set_downshift(phydev, *(const u8 *)data); + default: + return -EOPNOTSUPP; + } +} + +/* If we configure settings whilst firmware is still initializing the chip, + * then these settings may be overwritten. Therefore make sure chip + * initialization has completed. Use presence of the firmware ID as + * indicator for initialization having completed. + * The chip also provides a "reset completed" bit, but it's cleared after + * read. Therefore function would time out if called again. + */ +static int aqr107_wait_reset_complete(struct phy_device *phydev) +{ + int val, retries = 100; + + do { + val = phy_read_mmd(phydev, MDIO_MMD_VEND1, VEND1_GLOBAL_FW_ID); + if (val < 0) + return val; + msleep(20); + } while (!val && --retries); + + return val ? 0 : -ETIMEDOUT; +} + +static void aqr107_chip_info(struct phy_device *phydev) +{ + u8 fw_major, fw_minor, build_id, prov_id; + int val; + + val = phy_read_mmd(phydev, MDIO_MMD_VEND1, VEND1_GLOBAL_FW_ID); + if (val < 0) + return; + + fw_major = FIELD_GET(VEND1_GLOBAL_FW_ID_MAJOR, val); + fw_minor = FIELD_GET(VEND1_GLOBAL_FW_ID_MINOR, val); + + val = phy_read_mmd(phydev, MDIO_MMD_VEND1, VEND1_GLOBAL_RSVD_STAT1); + if (val < 0) + return; + + build_id = FIELD_GET(VEND1_GLOBAL_RSVD_STAT1_FW_BUILD_ID, val); + prov_id = FIELD_GET(VEND1_GLOBAL_RSVD_STAT1_PROV_ID, val); + + phydev_dbg(phydev, "FW %u.%u, Build %u, Provisioning %u\n", + fw_major, fw_minor, build_id, prov_id); +} + +static int aqr107_config_init(struct phy_device *phydev) +{ + int ret; + + /* Check that the PHY interface type is compatible */ + if (phydev->interface != PHY_INTERFACE_MODE_SGMII && + phydev->interface != PHY_INTERFACE_MODE_2500BASEX && + phydev->interface != PHY_INTERFACE_MODE_10GKR) + return -ENODEV; + + ret = aqr107_wait_reset_complete(phydev); + if (!ret) + aqr107_chip_info(phydev); + + /* ensure that a latched downshift event is cleared */ + aqr107_read_downshift_event(phydev); + + return aqr107_set_downshift(phydev, MDIO_AN_VEND_PROV_DOWNSHIFT_DFLT); +} + static int aqcs109_config_init(struct phy_device *phydev) { + int ret; + + /* Check that the PHY interface type is compatible */ + if (phydev->interface != PHY_INTERFACE_MODE_SGMII && + phydev->interface != PHY_INTERFACE_MODE_2500BASEX) + return -ENODEV; + + ret = aqr107_wait_reset_complete(phydev); + if (!ret) + aqr107_chip_info(phydev); + /* AQCS109 belongs to a chip family partially supporting 10G and 5G. * PMA speed ability bits are the same for all members of the family, * AQCS109 however supports speeds up to 2.5G only. */ - return phy_set_max_speed(phydev, SPEED_2500); + ret = phy_set_max_speed(phydev, SPEED_2500); + if (ret) + return ret; + + /* ensure that a latched downshift event is cleared */ + aqr107_read_downshift_event(phydev); + + return aqr107_set_downshift(phydev, MDIO_AN_VEND_PROV_DOWNSHIFT_DFLT); +} + +static void aqr107_link_change_notify(struct phy_device *phydev) +{ + u8 fw_major, fw_minor; + bool downshift, short_reach, afr; + int mode, val; + + if (phydev->state != PHY_RUNNING || phydev->autoneg == AUTONEG_DISABLE) + return; + + val = phy_read_mmd(phydev, MDIO_MMD_AN, MDIO_AN_RX_LP_STAT1); + /* call failed or link partner is no Aquantia PHY */ + if (val < 0 || !(val & MDIO_AN_RX_LP_STAT1_AQ_PHY)) + return; + + short_reach = val & MDIO_AN_RX_LP_STAT1_SHORT_REACH; + downshift = val & MDIO_AN_RX_LP_STAT1_AQRATE_DOWNSHIFT; + + val = phy_read_mmd(phydev, MDIO_MMD_AN, MDIO_AN_RX_LP_STAT4); + if (val < 0) + return; + + fw_major = FIELD_GET(MDIO_AN_RX_LP_STAT4_FW_MAJOR, val); + fw_minor = FIELD_GET(MDIO_AN_RX_LP_STAT4_FW_MINOR, val); + + val = phy_read_mmd(phydev, MDIO_MMD_AN, MDIO_AN_RX_VEND_STAT3); + if (val < 0) + return; + + afr = val & MDIO_AN_RX_VEND_STAT3_AFR; + + phydev_dbg(phydev, "Link partner is Aquantia PHY, FW %u.%u%s%s%s\n", + fw_major, fw_minor, + short_reach ? ", short reach mode" : "", + downshift ? ", fast-retrain downshift advertised" : "", + afr ? ", fast reframe advertised" : ""); + + val = phy_read_mmd(phydev, MDIO_MMD_VEND1, VEND1_GLOBAL_RSVD_STAT9); + if (val < 0) + return; + + mode = FIELD_GET(VEND1_GLOBAL_RSVD_STAT9_MODE, val); + if (mode == VEND1_GLOBAL_RSVD_STAT9_1000BT2) + phydev_info(phydev, "Aquantia 1000Base-T2 mode active\n"); +} + +static int aqr107_suspend(struct phy_device *phydev) +{ + return phy_set_bits_mmd(phydev, MDIO_MMD_VEND1, MDIO_CTRL1, + MDIO_CTRL1_LPOWER); +} + +static int aqr107_resume(struct phy_device *phydev) +{ + return phy_clear_bits_mmd(phydev, MDIO_MMD_VEND1, MDIO_CTRL1, + MDIO_CTRL1_LPOWER); +} + +static int aqr107_probe(struct phy_device *phydev) +{ + phydev->priv = devm_kzalloc(&phydev->mdio.dev, + sizeof(struct aqr107_priv), GFP_KERNEL); + if (!phydev->priv) + return -ENOMEM; + + return aqr_hwmon_probe(phydev); } static struct phy_driver aqr_driver[] = { { PHY_ID_MATCH_MODEL(PHY_ID_AQ1202), .name = "Aquantia AQ1202", - .aneg_done = genphy_c45_aneg_done, - .get_features = genphy_c45_pma_read_abilities, .config_aneg = aqr_config_aneg, .config_intr = aqr_config_intr, .ack_interrupt = aqr_ack_interrupt, @@ -201,8 +606,6 @@ static struct phy_driver aqr_driver[] = { { PHY_ID_MATCH_MODEL(PHY_ID_AQ2104), .name = "Aquantia AQ2104", - .aneg_done = genphy_c45_aneg_done, - .get_features = genphy_c45_pma_read_abilities, .config_aneg = aqr_config_aneg, .config_intr = aqr_config_intr, .ack_interrupt = aqr_ack_interrupt, @@ -211,8 +614,6 @@ static struct phy_driver aqr_driver[] = { { PHY_ID_MATCH_MODEL(PHY_ID_AQR105), .name = "Aquantia AQR105", - .aneg_done = genphy_c45_aneg_done, - .get_features = genphy_c45_pma_read_abilities, .config_aneg = aqr_config_aneg, .config_intr = aqr_config_intr, .ack_interrupt = aqr_ack_interrupt, @@ -221,8 +622,6 @@ static struct phy_driver aqr_driver[] = { { PHY_ID_MATCH_MODEL(PHY_ID_AQR106), .name = "Aquantia AQR106", - .aneg_done = genphy_c45_aneg_done, - .get_features = genphy_c45_pma_read_abilities, .config_aneg = aqr_config_aneg, .config_intr = aqr_config_intr, .ack_interrupt = aqr_ack_interrupt, @@ -231,31 +630,42 @@ static struct phy_driver aqr_driver[] = { { PHY_ID_MATCH_MODEL(PHY_ID_AQR107), .name = "Aquantia AQR107", - .aneg_done = genphy_c45_aneg_done, - .get_features = genphy_c45_pma_read_abilities, - .probe = aqr_hwmon_probe, + .probe = aqr107_probe, + .config_init = aqr107_config_init, .config_aneg = aqr_config_aneg, .config_intr = aqr_config_intr, .ack_interrupt = aqr_ack_interrupt, - .read_status = aqr_read_status, + .read_status = aqr107_read_status, + .get_tunable = aqr107_get_tunable, + .set_tunable = aqr107_set_tunable, + .suspend = aqr107_suspend, + .resume = aqr107_resume, + .get_sset_count = aqr107_get_sset_count, + .get_strings = aqr107_get_strings, + .get_stats = aqr107_get_stats, + .link_change_notify = aqr107_link_change_notify, }, { PHY_ID_MATCH_MODEL(PHY_ID_AQCS109), .name = "Aquantia AQCS109", - .aneg_done = genphy_c45_aneg_done, - .get_features = genphy_c45_pma_read_abilities, - .probe = aqr_hwmon_probe, + .probe = aqr107_probe, .config_init = aqcs109_config_init, .config_aneg = aqr_config_aneg, .config_intr = aqr_config_intr, .ack_interrupt = aqr_ack_interrupt, - .read_status = aqr_read_status, + .read_status = aqr107_read_status, + .get_tunable = aqr107_get_tunable, + .set_tunable = aqr107_set_tunable, + .suspend = aqr107_suspend, + .resume = aqr107_resume, + .get_sset_count = aqr107_get_sset_count, + .get_strings = aqr107_get_strings, + .get_stats = aqr107_get_stats, + .link_change_notify = aqr107_link_change_notify, }, { PHY_ID_MATCH_MODEL(PHY_ID_AQR405), .name = "Aquantia AQR405", - .aneg_done = genphy_c45_aneg_done, - .get_features = genphy_c45_pma_read_abilities, .config_aneg = aqr_config_aneg, .config_intr = aqr_config_intr, .ack_interrupt = aqr_ack_interrupt, diff --git a/drivers/net/phy/asix.c b/drivers/net/phy/asix.c index f14ba5366b911e7791472f9d1cbabaae8b6fb622..79bf7ef1fcfd6a210c409acb282251ef30cb1341 100644 --- a/drivers/net/phy/asix.c +++ b/drivers/net/phy/asix.c @@ -43,7 +43,7 @@ static struct phy_driver asix_driver[] = { { .phy_id = PHY_ID_ASIX_AX88796B, .name = "Asix Electronics AX88796B", .phy_id_mask = 0xfffffff0, - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .soft_reset = asix_soft_reset, } }; diff --git a/drivers/net/phy/at803x.c b/drivers/net/phy/at803x.c index f3e96191eb6f3533ab8b9fcdb1affe6d1a96e5ed..222ccd9ecfcebc0bf92beb82e177fba7c3bfaea5 100644 --- a/drivers/net/phy/at803x.c +++ b/drivers/net/phy/at803x.c @@ -324,8 +324,6 @@ static int at803x_config_intr(struct phy_device *phydev) static void at803x_link_change_notify(struct phy_device *phydev) { - struct at803x_priv *priv = phydev->priv; - /* * Conduct a hardware reset for AT8030 every time a link loss is * signalled. This is necessary to circumvent a hardware bug that @@ -333,25 +331,19 @@ static void at803x_link_change_notify(struct phy_device *phydev) * in the FIFO. In such cases, the FIFO enters an error mode it * cannot recover from by software. */ - if (phydev->state == PHY_NOLINK) { - if (phydev->mdio.reset && !priv->phy_reset) { - struct at803x_context context; + if (phydev->state == PHY_NOLINK && phydev->mdio.reset_gpio) { + struct at803x_context context; - at803x_context_save(phydev, &context); + at803x_context_save(phydev, &context); - phy_device_reset(phydev, 1); - msleep(1); - phy_device_reset(phydev, 0); - msleep(1); + phy_device_reset(phydev, 1); + msleep(1); + phy_device_reset(phydev, 0); + msleep(1); - at803x_context_restore(phydev, &context); + at803x_context_restore(phydev, &context); - phydev_dbg(phydev, "%s(): phy was reset\n", - __func__); - priv->phy_reset = true; - } - } else { - priv->phy_reset = false; + phydev_dbg(phydev, "%s(): phy was reset\n", __func__); } } @@ -397,7 +389,7 @@ static struct phy_driver at803x_driver[] = { .get_wol = at803x_get_wol, .suspend = at803x_suspend, .resume = at803x_resume, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .ack_interrupt = at803x_ack_interrupt, .config_intr = at803x_config_intr, }, { @@ -412,7 +404,7 @@ static struct phy_driver at803x_driver[] = { .get_wol = at803x_get_wol, .suspend = at803x_suspend, .resume = at803x_resume, - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .ack_interrupt = at803x_ack_interrupt, .config_intr = at803x_config_intr, }, { @@ -426,7 +418,7 @@ static struct phy_driver at803x_driver[] = { .get_wol = at803x_get_wol, .suspend = at803x_suspend, .resume = at803x_resume, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .aneg_done = at803x_aneg_done, .ack_interrupt = &at803x_ack_interrupt, .config_intr = &at803x_config_intr, diff --git a/drivers/net/phy/bcm-cygnus.c b/drivers/net/phy/bcm-cygnus.c index ab8e12922bf9789ed2be93b47ef6840a4e83aed3..9ccf28b0a04d1aa253d03826dc65f2e4cb77158e 100644 --- a/drivers/net/phy/bcm-cygnus.c +++ b/drivers/net/phy/bcm-cygnus.c @@ -10,6 +10,10 @@ #include #include +struct bcm_omega_phy_priv { + u64 *stats; +}; + /* Broadcom Cygnus Phy specific registers */ #define MII_BCM_CYGNUS_AFE_VDAC_ICTRL_0 0x91E5 /* VDAL Control register */ @@ -121,21 +125,162 @@ static int bcm_cygnus_resume(struct phy_device *phydev) return genphy_config_aneg(phydev); } +static int bcm_omega_config_init(struct phy_device *phydev) +{ + u8 count, rev; + int ret = 0; + + rev = phydev->phy_id & ~phydev->drv->phy_id_mask; + + pr_info_once("%s: %s PHY revision: 0x%02x\n", + phydev_name(phydev), phydev->drv->name, rev); + + /* Dummy read to a register to workaround an issue upon reset where the + * internal inverter may not allow the first MDIO transaction to pass + * the MDIO management controller and make us return 0xffff for such + * reads. + */ + phy_read(phydev, MII_BMSR); + + switch (rev) { + case 0x00: + ret = bcm_phy_28nm_a0b0_afe_config_init(phydev); + break; + default: + break; + } + + if (ret) + return ret; + + ret = bcm_phy_downshift_get(phydev, &count); + if (ret) + return ret; + + /* Only enable EEE if Wirespeed/downshift is disabled */ + ret = bcm_phy_set_eee(phydev, count == DOWNSHIFT_DEV_DISABLE); + if (ret) + return ret; + + return bcm_phy_enable_apd(phydev, true); +} + +static int bcm_omega_resume(struct phy_device *phydev) +{ + int ret; + + /* Re-apply workarounds coming out suspend/resume */ + ret = bcm_omega_config_init(phydev); + if (ret) + return ret; + + /* 28nm Gigabit PHYs come out of reset without any half-duplex + * or "hub" compliant advertised mode, fix that. This does not + * cause any problems with the PHY library since genphy_config_aneg() + * gracefully handles auto-negotiated and forced modes. + */ + return genphy_config_aneg(phydev); +} + +static int bcm_omega_get_tunable(struct phy_device *phydev, + struct ethtool_tunable *tuna, void *data) +{ + switch (tuna->id) { + case ETHTOOL_PHY_DOWNSHIFT: + return bcm_phy_downshift_get(phydev, (u8 *)data); + default: + return -EOPNOTSUPP; + } +} + +static int bcm_omega_set_tunable(struct phy_device *phydev, + struct ethtool_tunable *tuna, + const void *data) +{ + u8 count = *(u8 *)data; + int ret; + + switch (tuna->id) { + case ETHTOOL_PHY_DOWNSHIFT: + ret = bcm_phy_downshift_set(phydev, count); + break; + default: + return -EOPNOTSUPP; + } + + if (ret) + return ret; + + /* Disable EEE advertisement since this prevents the PHY + * from successfully linking up, trigger auto-negotiation restart + * to let the MAC decide what to do. + */ + ret = bcm_phy_set_eee(phydev, count == DOWNSHIFT_DEV_DISABLE); + if (ret) + return ret; + + return genphy_restart_aneg(phydev); +} + +static void bcm_omega_get_phy_stats(struct phy_device *phydev, + struct ethtool_stats *stats, u64 *data) +{ + struct bcm_omega_phy_priv *priv = phydev->priv; + + bcm_phy_get_stats(phydev, priv->stats, stats, data); +} + +static int bcm_omega_probe(struct phy_device *phydev) +{ + struct bcm_omega_phy_priv *priv; + + priv = devm_kzalloc(&phydev->mdio.dev, sizeof(*priv), GFP_KERNEL); + if (!priv) + return -ENOMEM; + + phydev->priv = priv; + + priv->stats = devm_kcalloc(&phydev->mdio.dev, + bcm_phy_get_sset_count(phydev), sizeof(u64), + GFP_KERNEL); + if (!priv->stats) + return -ENOMEM; + + return 0; +} + static struct phy_driver bcm_cygnus_phy_driver[] = { { .phy_id = PHY_ID_BCM_CYGNUS, .phy_id_mask = 0xfffffff0, .name = "Broadcom Cygnus PHY", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = bcm_cygnus_config_init, .ack_interrupt = bcm_phy_ack_intr, .config_intr = bcm_phy_config_intr, .suspend = genphy_suspend, .resume = bcm_cygnus_resume, -} }; +}, { + .phy_id = PHY_ID_BCM_OMEGA, + .phy_id_mask = 0xfffffff0, + .name = "Broadcom Omega Combo GPHY", + /* PHY_GBIT_FEATURES */ + .flags = PHY_IS_INTERNAL, + .config_init = bcm_omega_config_init, + .suspend = genphy_suspend, + .resume = bcm_omega_resume, + .get_tunable = bcm_omega_get_tunable, + .set_tunable = bcm_omega_set_tunable, + .get_sset_count = bcm_phy_get_sset_count, + .get_strings = bcm_phy_get_strings, + .get_stats = bcm_omega_get_phy_stats, + .probe = bcm_omega_probe, +} +}; static struct mdio_device_id __maybe_unused bcm_cygnus_phy_tbl[] = { { PHY_ID_BCM_CYGNUS, 0xfffffff0, }, + { PHY_ID_BCM_OMEGA, 0xfffffff0, }, { } }; MODULE_DEVICE_TABLE(mdio, bcm_cygnus_phy_tbl); diff --git a/drivers/net/phy/bcm-phy-lib.c b/drivers/net/phy/bcm-phy-lib.c index a75642051b8bc4db808bc156dfa2092e93a5c16f..e0d3310957ffaed2f6249fb5e25c2afdb4bba4c6 100644 --- a/drivers/net/phy/bcm-phy-lib.c +++ b/drivers/net/phy/bcm-phy-lib.c @@ -371,6 +371,58 @@ void bcm_phy_get_stats(struct phy_device *phydev, u64 *shadow, } EXPORT_SYMBOL_GPL(bcm_phy_get_stats); +void bcm_phy_r_rc_cal_reset(struct phy_device *phydev) +{ + /* Reset R_CAL/RC_CAL Engine */ + bcm_phy_write_exp_sel(phydev, 0x00b0, 0x0010); + + /* Disable Reset R_AL/RC_CAL Engine */ + bcm_phy_write_exp_sel(phydev, 0x00b0, 0x0000); +} +EXPORT_SYMBOL_GPL(bcm_phy_r_rc_cal_reset); + +int bcm_phy_28nm_a0b0_afe_config_init(struct phy_device *phydev) +{ + /* Increase VCO range to prevent unlocking problem of PLL at low + * temp + */ + bcm_phy_write_misc(phydev, PLL_PLLCTRL_1, 0x0048); + + /* Change Ki to 011 */ + bcm_phy_write_misc(phydev, PLL_PLLCTRL_2, 0x021b); + + /* Disable loading of TVCO buffer to bandgap, set bandgap trim + * to 111 + */ + bcm_phy_write_misc(phydev, PLL_PLLCTRL_4, 0x0e20); + + /* Adjust bias current trim by -3 */ + bcm_phy_write_misc(phydev, DSP_TAP10, 0x690b); + + /* Switch to CORE_BASE1E */ + phy_write(phydev, MII_BRCM_CORE_BASE1E, 0xd); + + bcm_phy_r_rc_cal_reset(phydev); + + /* write AFE_RXCONFIG_0 */ + bcm_phy_write_misc(phydev, AFE_RXCONFIG_0, 0xeb19); + + /* write AFE_RXCONFIG_1 */ + bcm_phy_write_misc(phydev, AFE_RXCONFIG_1, 0x9a3f); + + /* write AFE_RX_LP_COUNTER */ + bcm_phy_write_misc(phydev, AFE_RX_LP_COUNTER, 0x7fc0); + + /* write AFE_HPF_TRIM_OTHERS */ + bcm_phy_write_misc(phydev, AFE_HPF_TRIM_OTHERS, 0x000b); + + /* write AFTE_TX_CONFIG */ + bcm_phy_write_misc(phydev, AFE_TX_CONFIG, 0x0800); + + return 0; +} +EXPORT_SYMBOL_GPL(bcm_phy_28nm_a0b0_afe_config_init); + MODULE_DESCRIPTION("Broadcom PHY Library"); MODULE_LICENSE("GPL v2"); MODULE_AUTHOR("Broadcom Corporation"); diff --git a/drivers/net/phy/bcm-phy-lib.h b/drivers/net/phy/bcm-phy-lib.h index 17faaefcfd600b05d0d71d7605f60f0ce9e9bb53..5ecacb4e64f00109068fb996c3a400e270d24138 100644 --- a/drivers/net/phy/bcm-phy-lib.h +++ b/drivers/net/phy/bcm-phy-lib.h @@ -9,6 +9,24 @@ #include #include +/* 28nm only register definitions */ +#define MISC_ADDR(base, channel) base, channel + +#define DSP_TAP10 MISC_ADDR(0x0a, 0) +#define PLL_PLLCTRL_1 MISC_ADDR(0x32, 1) +#define PLL_PLLCTRL_2 MISC_ADDR(0x32, 2) +#define PLL_PLLCTRL_4 MISC_ADDR(0x33, 0) + +#define AFE_RXCONFIG_0 MISC_ADDR(0x38, 0) +#define AFE_RXCONFIG_1 MISC_ADDR(0x38, 1) +#define AFE_RXCONFIG_2 MISC_ADDR(0x38, 2) +#define AFE_RX_LP_COUNTER MISC_ADDR(0x38, 3) +#define AFE_TX_CONFIG MISC_ADDR(0x39, 0) +#define AFE_VDCA_ICTRL_0 MISC_ADDR(0x39, 1) +#define AFE_VDAC_OTHERS_0 MISC_ADDR(0x39, 3) +#define AFE_HPF_TRIM_OTHERS MISC_ADDR(0x3a, 0) + + int bcm_phy_write_exp(struct phy_device *phydev, u16 reg, u16 val); int bcm_phy_read_exp(struct phy_device *phydev, u16 reg); @@ -45,5 +63,7 @@ int bcm_phy_get_sset_count(struct phy_device *phydev); void bcm_phy_get_strings(struct phy_device *phydev, u8 *data); void bcm_phy_get_stats(struct phy_device *phydev, u64 *shadow, struct ethtool_stats *stats, u64 *data); +void bcm_phy_r_rc_cal_reset(struct phy_device *phydev); +int bcm_phy_28nm_a0b0_afe_config_init(struct phy_device *phydev); #endif /* _LINUX_BCM_PHY_LIB_H */ diff --git a/drivers/net/phy/bcm63xx.c b/drivers/net/phy/bcm63xx.c index 44e6cff419a094a7c1334ca15ef36084a61ecfb6..23f1958ba6ad4f6000aa0c5de4d22dca908ef248 100644 --- a/drivers/net/phy/bcm63xx.c +++ b/drivers/net/phy/bcm63xx.c @@ -64,7 +64,7 @@ static struct phy_driver bcm63xx_driver[] = { .phy_id = 0x00406000, .phy_id_mask = 0xfffffc00, .name = "Broadcom BCM63XX (1)", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .flags = PHY_IS_INTERNAL, .config_init = bcm63xx_config_init, .ack_interrupt = bcm_phy_ack_intr, @@ -73,7 +73,7 @@ static struct phy_driver bcm63xx_driver[] = { /* same phy as above, with just a different OUI */ .phy_id = 0x002bdc00, .phy_id_mask = 0xfffffc00, - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .flags = PHY_IS_INTERNAL, .config_init = bcm63xx_config_init, .ack_interrupt = bcm_phy_ack_intr, diff --git a/drivers/net/phy/bcm7xxx.c b/drivers/net/phy/bcm7xxx.c index b8415f8fae14fe24fafd6eb52023eacd7362bc16..8fc33867e524f865261b68f8c2f74ab0d8c6d660 100644 --- a/drivers/net/phy/bcm7xxx.c +++ b/drivers/net/phy/bcm7xxx.c @@ -37,77 +37,10 @@ #define MII_BCM7XXX_SHD_3_TL4 0x23 #define MII_BCM7XXX_TL4_RST_MSK (BIT(2) | BIT(1)) -/* 28nm only register definitions */ -#define MISC_ADDR(base, channel) base, channel - -#define DSP_TAP10 MISC_ADDR(0x0a, 0) -#define PLL_PLLCTRL_1 MISC_ADDR(0x32, 1) -#define PLL_PLLCTRL_2 MISC_ADDR(0x32, 2) -#define PLL_PLLCTRL_4 MISC_ADDR(0x33, 0) - -#define AFE_RXCONFIG_0 MISC_ADDR(0x38, 0) -#define AFE_RXCONFIG_1 MISC_ADDR(0x38, 1) -#define AFE_RXCONFIG_2 MISC_ADDR(0x38, 2) -#define AFE_RX_LP_COUNTER MISC_ADDR(0x38, 3) -#define AFE_TX_CONFIG MISC_ADDR(0x39, 0) -#define AFE_VDCA_ICTRL_0 MISC_ADDR(0x39, 1) -#define AFE_VDAC_OTHERS_0 MISC_ADDR(0x39, 3) -#define AFE_HPF_TRIM_OTHERS MISC_ADDR(0x3a, 0) - struct bcm7xxx_phy_priv { u64 *stats; }; -static void r_rc_cal_reset(struct phy_device *phydev) -{ - /* Reset R_CAL/RC_CAL Engine */ - bcm_phy_write_exp_sel(phydev, 0x00b0, 0x0010); - - /* Disable Reset R_AL/RC_CAL Engine */ - bcm_phy_write_exp_sel(phydev, 0x00b0, 0x0000); -} - -static int bcm7xxx_28nm_b0_afe_config_init(struct phy_device *phydev) -{ - /* Increase VCO range to prevent unlocking problem of PLL at low - * temp - */ - bcm_phy_write_misc(phydev, PLL_PLLCTRL_1, 0x0048); - - /* Change Ki to 011 */ - bcm_phy_write_misc(phydev, PLL_PLLCTRL_2, 0x021b); - - /* Disable loading of TVCO buffer to bandgap, set bandgap trim - * to 111 - */ - bcm_phy_write_misc(phydev, PLL_PLLCTRL_4, 0x0e20); - - /* Adjust bias current trim by -3 */ - bcm_phy_write_misc(phydev, DSP_TAP10, 0x690b); - - /* Switch to CORE_BASE1E */ - phy_write(phydev, MII_BRCM_CORE_BASE1E, 0xd); - - r_rc_cal_reset(phydev); - - /* write AFE_RXCONFIG_0 */ - bcm_phy_write_misc(phydev, AFE_RXCONFIG_0, 0xeb19); - - /* write AFE_RXCONFIG_1 */ - bcm_phy_write_misc(phydev, AFE_RXCONFIG_1, 0x9a3f); - - /* write AFE_RX_LP_COUNTER */ - bcm_phy_write_misc(phydev, AFE_RX_LP_COUNTER, 0x7fc0); - - /* write AFE_HPF_TRIM_OTHERS */ - bcm_phy_write_misc(phydev, AFE_HPF_TRIM_OTHERS, 0x000b); - - /* write AFTE_TX_CONFIG */ - bcm_phy_write_misc(phydev, AFE_TX_CONFIG, 0x0800); - - return 0; -} - static int bcm7xxx_28nm_d0_afe_config_init(struct phy_device *phydev) { /* AFE_RXCONFIG_0 */ @@ -143,7 +76,7 @@ static int bcm7xxx_28nm_d0_afe_config_init(struct phy_device *phydev) bcm_phy_write_misc(phydev, DSP_TAP10, 0x011b); /* Reset R_CAL/RC_CAL engine */ - r_rc_cal_reset(phydev); + bcm_phy_r_rc_cal_reset(phydev); return 0; } @@ -171,7 +104,7 @@ static int bcm7xxx_28nm_e0_plus_afe_config_init(struct phy_device *phydev) bcm_phy_write_misc(phydev, DSP_TAP10, 0x011b); /* Reset R_CAL/RC_CAL engine */ - r_rc_cal_reset(phydev); + bcm_phy_r_rc_cal_reset(phydev); return 0; } @@ -196,7 +129,7 @@ static int bcm7xxx_28nm_a0_patch_afe_config_init(struct phy_device *phydev) /* Enable ffe zero detection for Vitesse interoperability */ bcm_phy_write_misc(phydev, 0x26, 0x2, 0x0015); - r_rc_cal_reset(phydev); + bcm_phy_r_rc_cal_reset(phydev); return 0; } @@ -227,7 +160,7 @@ static int bcm7xxx_28nm_config_init(struct phy_device *phydev) switch (rev) { case 0xa0: case 0xb0: - ret = bcm7xxx_28nm_b0_afe_config_init(phydev); + ret = bcm_phy_28nm_a0b0_afe_config_init(phydev); break; case 0xd0: ret = bcm7xxx_28nm_d0_afe_config_init(phydev); @@ -605,7 +538,7 @@ static int bcm7xxx_28nm_probe(struct phy_device *phydev) .phy_id = (_oui), \ .phy_id_mask = 0xfffffff0, \ .name = _name, \ - .features = PHY_GBIT_FEATURES, \ + /* PHY_GBIT_FEATURES */ \ .flags = PHY_IS_INTERNAL, \ .config_init = bcm7xxx_28nm_config_init, \ .resume = bcm7xxx_28nm_resume, \ @@ -622,7 +555,7 @@ static int bcm7xxx_28nm_probe(struct phy_device *phydev) .phy_id = (_oui), \ .phy_id_mask = 0xfffffff0, \ .name = _name, \ - .features = PHY_BASIC_FEATURES, \ + /* PHY_BASIC_FEATURES */ \ .flags = PHY_IS_INTERNAL, \ .config_init = bcm7xxx_28nm_ephy_config_init, \ .resume = bcm7xxx_28nm_ephy_resume, \ @@ -637,7 +570,7 @@ static int bcm7xxx_28nm_probe(struct phy_device *phydev) .phy_id = (_oui), \ .phy_id_mask = 0xfffffff0, \ .name = _name, \ - .features = PHY_BASIC_FEATURES, \ + /* PHY_BASIC_FEATURES */ \ .flags = PHY_IS_INTERNAL, \ .config_init = bcm7xxx_config_init, \ .suspend = bcm7xxx_suspend, \ @@ -657,7 +590,6 @@ static struct phy_driver bcm7xxx_driver[] = { BCM7XXX_28NM_GPHY(PHY_ID_BCM7439, "Broadcom BCM7439"), BCM7XXX_28NM_GPHY(PHY_ID_BCM7439_2, "Broadcom BCM7439 (2)"), BCM7XXX_28NM_GPHY(PHY_ID_BCM7445, "Broadcom BCM7445"), - BCM7XXX_28NM_GPHY(PHY_ID_BCM_OMEGA, "Broadcom Omega Combo GPHY"), BCM7XXX_40NM_EPHY(PHY_ID_BCM7346, "Broadcom BCM7346"), BCM7XXX_40NM_EPHY(PHY_ID_BCM7362, "Broadcom BCM7362"), BCM7XXX_40NM_EPHY(PHY_ID_BCM7425, "Broadcom BCM7425"), diff --git a/drivers/net/phy/broadcom.c b/drivers/net/phy/broadcom.c index cb86a3e90c7de3ff41a7d821c135aec8dad9eef0..67fa05d6752369aa98d15fe059046647bbdebaed 100644 --- a/drivers/net/phy/broadcom.c +++ b/drivers/net/phy/broadcom.c @@ -610,7 +610,7 @@ static struct phy_driver broadcom_drivers[] = { .phy_id = PHY_ID_BCM5411, .phy_id_mask = 0xfffffff0, .name = "Broadcom BCM5411", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = bcm54xx_config_init, .ack_interrupt = bcm_phy_ack_intr, .config_intr = bcm_phy_config_intr, @@ -618,7 +618,7 @@ static struct phy_driver broadcom_drivers[] = { .phy_id = PHY_ID_BCM5421, .phy_id_mask = 0xfffffff0, .name = "Broadcom BCM5421", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = bcm54xx_config_init, .ack_interrupt = bcm_phy_ack_intr, .config_intr = bcm_phy_config_intr, @@ -626,7 +626,7 @@ static struct phy_driver broadcom_drivers[] = { .phy_id = PHY_ID_BCM54210E, .phy_id_mask = 0xfffffff0, .name = "Broadcom BCM54210E", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = bcm54xx_config_init, .ack_interrupt = bcm_phy_ack_intr, .config_intr = bcm_phy_config_intr, @@ -634,7 +634,7 @@ static struct phy_driver broadcom_drivers[] = { .phy_id = PHY_ID_BCM5461, .phy_id_mask = 0xfffffff0, .name = "Broadcom BCM5461", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = bcm54xx_config_init, .ack_interrupt = bcm_phy_ack_intr, .config_intr = bcm_phy_config_intr, @@ -642,7 +642,7 @@ static struct phy_driver broadcom_drivers[] = { .phy_id = PHY_ID_BCM54612E, .phy_id_mask = 0xfffffff0, .name = "Broadcom BCM54612E", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = bcm54xx_config_init, .ack_interrupt = bcm_phy_ack_intr, .config_intr = bcm_phy_config_intr, @@ -650,7 +650,7 @@ static struct phy_driver broadcom_drivers[] = { .phy_id = PHY_ID_BCM54616S, .phy_id_mask = 0xfffffff0, .name = "Broadcom BCM54616S", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = bcm54xx_config_init, .config_aneg = bcm54616s_config_aneg, .ack_interrupt = bcm_phy_ack_intr, @@ -659,7 +659,7 @@ static struct phy_driver broadcom_drivers[] = { .phy_id = PHY_ID_BCM5464, .phy_id_mask = 0xfffffff0, .name = "Broadcom BCM5464", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = bcm54xx_config_init, .ack_interrupt = bcm_phy_ack_intr, .config_intr = bcm_phy_config_intr, @@ -667,7 +667,7 @@ static struct phy_driver broadcom_drivers[] = { .phy_id = PHY_ID_BCM5481, .phy_id_mask = 0xfffffff0, .name = "Broadcom BCM5481", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = bcm54xx_config_init, .config_aneg = bcm5481_config_aneg, .ack_interrupt = bcm_phy_ack_intr, @@ -676,7 +676,7 @@ static struct phy_driver broadcom_drivers[] = { .phy_id = PHY_ID_BCM54810, .phy_id_mask = 0xfffffff0, .name = "Broadcom BCM54810", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = bcm54xx_config_init, .config_aneg = bcm5481_config_aneg, .ack_interrupt = bcm_phy_ack_intr, @@ -685,7 +685,7 @@ static struct phy_driver broadcom_drivers[] = { .phy_id = PHY_ID_BCM5482, .phy_id_mask = 0xfffffff0, .name = "Broadcom BCM5482", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = bcm5482_config_init, .read_status = bcm5482_read_status, .ack_interrupt = bcm_phy_ack_intr, @@ -694,7 +694,7 @@ static struct phy_driver broadcom_drivers[] = { .phy_id = PHY_ID_BCM50610, .phy_id_mask = 0xfffffff0, .name = "Broadcom BCM50610", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = bcm54xx_config_init, .ack_interrupt = bcm_phy_ack_intr, .config_intr = bcm_phy_config_intr, @@ -702,7 +702,7 @@ static struct phy_driver broadcom_drivers[] = { .phy_id = PHY_ID_BCM50610M, .phy_id_mask = 0xfffffff0, .name = "Broadcom BCM50610M", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = bcm54xx_config_init, .ack_interrupt = bcm_phy_ack_intr, .config_intr = bcm_phy_config_intr, @@ -710,7 +710,7 @@ static struct phy_driver broadcom_drivers[] = { .phy_id = PHY_ID_BCM57780, .phy_id_mask = 0xfffffff0, .name = "Broadcom BCM57780", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = bcm54xx_config_init, .ack_interrupt = bcm_phy_ack_intr, .config_intr = bcm_phy_config_intr, @@ -718,7 +718,7 @@ static struct phy_driver broadcom_drivers[] = { .phy_id = PHY_ID_BCMAC131, .phy_id_mask = 0xfffffff0, .name = "Broadcom BCMAC131", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = brcm_fet_config_init, .ack_interrupt = brcm_fet_ack_interrupt, .config_intr = brcm_fet_config_intr, @@ -726,7 +726,7 @@ static struct phy_driver broadcom_drivers[] = { .phy_id = PHY_ID_BCM5241, .phy_id_mask = 0xfffffff0, .name = "Broadcom BCM5241", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = brcm_fet_config_init, .ack_interrupt = brcm_fet_ack_interrupt, .config_intr = brcm_fet_config_intr, @@ -735,7 +735,7 @@ static struct phy_driver broadcom_drivers[] = { .phy_id_mask = 0xfffffff0, .name = "Broadcom BCM5395", .flags = PHY_IS_INTERNAL, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .get_sset_count = bcm_phy_get_sset_count, .get_strings = bcm_phy_get_strings, .get_stats = bcm53xx_phy_get_stats, @@ -744,7 +744,7 @@ static struct phy_driver broadcom_drivers[] = { .phy_id = PHY_ID_BCM89610, .phy_id_mask = 0xfffffff0, .name = "Broadcom BCM89610", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = bcm54xx_config_init, .ack_interrupt = bcm_phy_ack_intr, .config_intr = bcm_phy_config_intr, diff --git a/drivers/net/phy/cicada.c b/drivers/net/phy/cicada.c index 108ed24f84893654461aa94a48e5da20979ec8aa..9d1612a4d7e6c179159c826f1624ec616f6d7200 100644 --- a/drivers/net/phy/cicada.c +++ b/drivers/net/phy/cicada.c @@ -102,7 +102,7 @@ static struct phy_driver cis820x_driver[] = { .phy_id = 0x000fc410, .name = "Cicada Cis8201", .phy_id_mask = 0x000ffff0, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = &cis820x_config_init, .ack_interrupt = &cis820x_ack_interrupt, .config_intr = &cis820x_config_intr, @@ -110,7 +110,7 @@ static struct phy_driver cis820x_driver[] = { .phy_id = 0x000fc440, .name = "Cicada Cis8204", .phy_id_mask = 0x000fffc0, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = &cis820x_config_init, .ack_interrupt = &cis820x_ack_interrupt, .config_intr = &cis820x_config_intr, diff --git a/drivers/net/phy/davicom.c b/drivers/net/phy/davicom.c index bf39baa7f2c8a5937842a176ad82d3e03715b8cb..942f277463a4d4776f3a954eca782a00e71c52be 100644 --- a/drivers/net/phy/davicom.c +++ b/drivers/net/phy/davicom.c @@ -144,7 +144,7 @@ static struct phy_driver dm91xx_driver[] = { .phy_id = 0x0181b880, .name = "Davicom DM9161E", .phy_id_mask = 0x0ffffff0, - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = dm9161_config_init, .config_aneg = dm9161_config_aneg, .ack_interrupt = dm9161_ack_interrupt, @@ -153,7 +153,7 @@ static struct phy_driver dm91xx_driver[] = { .phy_id = 0x0181b8b0, .name = "Davicom DM9161B/C", .phy_id_mask = 0x0ffffff0, - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = dm9161_config_init, .config_aneg = dm9161_config_aneg, .ack_interrupt = dm9161_ack_interrupt, @@ -162,7 +162,7 @@ static struct phy_driver dm91xx_driver[] = { .phy_id = 0x0181b8a0, .name = "Davicom DM9161A", .phy_id_mask = 0x0ffffff0, - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = dm9161_config_init, .config_aneg = dm9161_config_aneg, .ack_interrupt = dm9161_ack_interrupt, @@ -171,7 +171,7 @@ static struct phy_driver dm91xx_driver[] = { .phy_id = 0x00181b80, .name = "Davicom DM9131", .phy_id_mask = 0x0ffffff0, - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .ack_interrupt = dm9161_ack_interrupt, .config_intr = dm9161_config_intr, } }; diff --git a/drivers/net/phy/dp83640.c b/drivers/net/phy/dp83640.c index 2fe2ebaf62d1b2c6a320efd375e5f4f5f3af142d..6580094161a9f6730e529124a08811f5ace9ff53 100644 --- a/drivers/net/phy/dp83640.c +++ b/drivers/net/phy/dp83640.c @@ -1514,7 +1514,7 @@ static struct phy_driver dp83640_driver = { .phy_id = DP83640_PHY_ID, .phy_id_mask = 0xfffffff0, .name = "NatSemi DP83640", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .probe = dp83640_probe, .remove = dp83640_remove, .soft_reset = dp83640_soft_reset, diff --git a/drivers/net/phy/dp83822.c b/drivers/net/phy/dp83822.c index 97d45bd5b38e382b678dc3ce814f813cf045d7d6..7ed4760fb1557ac5ac4c163e416c06532d8e9150 100644 --- a/drivers/net/phy/dp83822.c +++ b/drivers/net/phy/dp83822.c @@ -310,7 +310,7 @@ static int dp83822_resume(struct phy_device *phydev) { \ PHY_ID_MATCH_MODEL(_id), \ .name = (_name), \ - .features = PHY_BASIC_FEATURES, \ + /* PHY_BASIC_FEATURES */ \ .soft_reset = dp83822_phy_reset, \ .config_init = dp83822_config_init, \ .get_wol = dp83822_get_wol, \ diff --git a/drivers/net/phy/dp83848.c b/drivers/net/phy/dp83848.c index f55dc907c2f39b579acb9f8042f8692900e5fe47..6f9bc7d91f17fd9597918a6fd1c2434cceb91592 100644 --- a/drivers/net/phy/dp83848.c +++ b/drivers/net/phy/dp83848.c @@ -99,7 +99,7 @@ MODULE_DEVICE_TABLE(mdio, dp83848_tbl); .phy_id = _id, \ .phy_id_mask = 0xfffffff0, \ .name = _name, \ - .features = PHY_BASIC_FEATURES, \ + /* PHY_BASIC_FEATURES */ \ \ .soft_reset = genphy_soft_reset, \ .config_init = _config_init, \ diff --git a/drivers/net/phy/dp83867.c b/drivers/net/phy/dp83867.c index 8448d01819efbde7a973061c945177ad221ffd2c..fd35131a0c39fd714e401725c8942ba239f21e8a 100644 --- a/drivers/net/phy/dp83867.c +++ b/drivers/net/phy/dp83867.c @@ -315,7 +315,7 @@ static struct phy_driver dp83867_driver[] = { .phy_id = DP83867_PHY_ID, .phy_id_mask = 0xfffffff0, .name = "TI DP83867", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = dp83867_config_init, .soft_reset = dp83867_phy_reset, diff --git a/drivers/net/phy/dp83tc811.c b/drivers/net/phy/dp83tc811.c index e9704af1d239b8650edc56a40f6d12804371116e..ac27da16824dab5b62e6a9b57d883a5d53359252 100644 --- a/drivers/net/phy/dp83tc811.c +++ b/drivers/net/phy/dp83tc811.c @@ -338,7 +338,7 @@ static struct phy_driver dp83811_driver[] = { .phy_id = DP83TC811_PHY_ID, .phy_id_mask = 0xfffffff0, .name = "TI DP83TC811", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = dp83811_config_init, .config_aneg = dp83811_config_aneg, .soft_reset = dp83811_phy_reset, diff --git a/drivers/net/phy/et1011c.c b/drivers/net/phy/et1011c.c index 2aa367c04a8ea63765fde3bfb9af160402fbb9f7..09e07b902d3a337c3c2aa46f0c056f0eddf6b545 100644 --- a/drivers/net/phy/et1011c.c +++ b/drivers/net/phy/et1011c.c @@ -86,7 +86,7 @@ static struct phy_driver et1011c_driver[] = { { .phy_id = 0x0282f014, .name = "ET1011C", .phy_id_mask = 0xfffffff0, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_aneg = et1011c_config_aneg, .read_status = et1011c_read_status, } }; diff --git a/drivers/net/phy/fixed_phy.c b/drivers/net/phy/fixed_phy.c index 1acd8bfdb3bc30e7855e36eb18baee4cf9baf7c3..3ffe46df249ee2a482b748ac66a79ff31141690a 100644 --- a/drivers/net/phy/fixed_phy.c +++ b/drivers/net/phy/fixed_phy.c @@ -301,7 +301,7 @@ static struct phy_device *__fixed_phy_register(unsigned int irq, phy->supported); } - linkmode_copy(phy->advertising, phy->supported); + phy_advertise_supported(phy); ret = phy_device_register(phy); if (ret) { diff --git a/drivers/net/phy/icplus.c b/drivers/net/phy/icplus.c index ebef8354bc81347d2524f89f1a93f03dc7747fd2..d6e8516cd146ed9db359570cb6c308810775d8cc 100644 --- a/drivers/net/phy/icplus.c +++ b/drivers/net/phy/icplus.c @@ -311,7 +311,7 @@ static struct phy_driver icplus_driver[] = { .phy_id = 0x02430d80, .name = "ICPlus IP175C", .phy_id_mask = 0x0ffffff0, - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = &ip175c_config_init, .config_aneg = &ip175c_config_aneg, .read_status = &ip175c_read_status, @@ -321,7 +321,7 @@ static struct phy_driver icplus_driver[] = { .phy_id = 0x02430d90, .name = "ICPlus IP1001", .phy_id_mask = 0x0ffffff0, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = &ip1001_config_init, .suspend = genphy_suspend, .resume = genphy_resume, @@ -329,7 +329,7 @@ static struct phy_driver icplus_driver[] = { .phy_id = 0x02430c54, .name = "ICPlus IP101A/G", .phy_id_mask = 0x0ffffff0, - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .probe = ip101a_g_probe, .config_intr = ip101a_g_config_intr, .did_interrupt = ip101a_g_did_interrupt, diff --git a/drivers/net/phy/intel-xway.c b/drivers/net/phy/intel-xway.c index 02d9713318b6dfdc8f600281d756e33288636de3..b7875b36097fe246c8627717233ab04d03298b54 100644 --- a/drivers/net/phy/intel-xway.c +++ b/drivers/net/phy/intel-xway.c @@ -232,7 +232,7 @@ static struct phy_driver xway_gphy[] = { .phy_id = PHY_ID_PHY11G_1_3, .phy_id_mask = 0xffffffff, .name = "Intel XWAY PHY11G (PEF 7071/PEF 7072) v1.3", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = xway_gphy_config_init, .config_aneg = xway_gphy14_config_aneg, .ack_interrupt = xway_gphy_ack_interrupt, @@ -244,7 +244,7 @@ static struct phy_driver xway_gphy[] = { .phy_id = PHY_ID_PHY22F_1_3, .phy_id_mask = 0xffffffff, .name = "Intel XWAY PHY22F (PEF 7061) v1.3", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = xway_gphy_config_init, .config_aneg = xway_gphy14_config_aneg, .ack_interrupt = xway_gphy_ack_interrupt, @@ -256,7 +256,7 @@ static struct phy_driver xway_gphy[] = { .phy_id = PHY_ID_PHY11G_1_4, .phy_id_mask = 0xffffffff, .name = "Intel XWAY PHY11G (PEF 7071/PEF 7072) v1.4", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = xway_gphy_config_init, .config_aneg = xway_gphy14_config_aneg, .ack_interrupt = xway_gphy_ack_interrupt, @@ -268,7 +268,7 @@ static struct phy_driver xway_gphy[] = { .phy_id = PHY_ID_PHY22F_1_4, .phy_id_mask = 0xffffffff, .name = "Intel XWAY PHY22F (PEF 7061) v1.4", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = xway_gphy_config_init, .config_aneg = xway_gphy14_config_aneg, .ack_interrupt = xway_gphy_ack_interrupt, @@ -280,7 +280,7 @@ static struct phy_driver xway_gphy[] = { .phy_id = PHY_ID_PHY11G_1_5, .phy_id_mask = 0xffffffff, .name = "Intel XWAY PHY11G (PEF 7071/PEF 7072) v1.5 / v1.6", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = xway_gphy_config_init, .ack_interrupt = xway_gphy_ack_interrupt, .did_interrupt = xway_gphy_did_interrupt, @@ -291,7 +291,7 @@ static struct phy_driver xway_gphy[] = { .phy_id = PHY_ID_PHY22F_1_5, .phy_id_mask = 0xffffffff, .name = "Intel XWAY PHY22F (PEF 7061) v1.5 / v1.6", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = xway_gphy_config_init, .ack_interrupt = xway_gphy_ack_interrupt, .did_interrupt = xway_gphy_did_interrupt, @@ -302,7 +302,7 @@ static struct phy_driver xway_gphy[] = { .phy_id = PHY_ID_PHY11G_VR9_1_1, .phy_id_mask = 0xffffffff, .name = "Intel XWAY PHY11G (xRX v1.1 integrated)", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = xway_gphy_config_init, .ack_interrupt = xway_gphy_ack_interrupt, .did_interrupt = xway_gphy_did_interrupt, @@ -313,7 +313,7 @@ static struct phy_driver xway_gphy[] = { .phy_id = PHY_ID_PHY22F_VR9_1_1, .phy_id_mask = 0xffffffff, .name = "Intel XWAY PHY22F (xRX v1.1 integrated)", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = xway_gphy_config_init, .ack_interrupt = xway_gphy_ack_interrupt, .did_interrupt = xway_gphy_did_interrupt, @@ -324,7 +324,7 @@ static struct phy_driver xway_gphy[] = { .phy_id = PHY_ID_PHY11G_VR9_1_2, .phy_id_mask = 0xffffffff, .name = "Intel XWAY PHY11G (xRX v1.2 integrated)", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = xway_gphy_config_init, .ack_interrupt = xway_gphy_ack_interrupt, .did_interrupt = xway_gphy_did_interrupt, @@ -335,7 +335,7 @@ static struct phy_driver xway_gphy[] = { .phy_id = PHY_ID_PHY22F_VR9_1_2, .phy_id_mask = 0xffffffff, .name = "Intel XWAY PHY22F (xRX v1.2 integrated)", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = xway_gphy_config_init, .ack_interrupt = xway_gphy_ack_interrupt, .did_interrupt = xway_gphy_did_interrupt, diff --git a/drivers/net/phy/lxt.c b/drivers/net/phy/lxt.c index a93d673baf35dc2c4175c8badea698ab690fcdc3..31448628811929f063a379ac254771ea8dbd9f45 100644 --- a/drivers/net/phy/lxt.c +++ b/drivers/net/phy/lxt.c @@ -251,7 +251,7 @@ static struct phy_driver lxt97x_driver[] = { .phy_id = 0x78100000, .name = "LXT970", .phy_id_mask = 0xfffffff0, - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = lxt970_config_init, .ack_interrupt = lxt970_ack_interrupt, .config_intr = lxt970_config_intr, @@ -259,14 +259,14 @@ static struct phy_driver lxt97x_driver[] = { .phy_id = 0x001378e0, .name = "LXT971", .phy_id_mask = 0xfffffff0, - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .ack_interrupt = lxt971_ack_interrupt, .config_intr = lxt971_config_intr, }, { .phy_id = 0x00137a10, .name = "LXT973-A2", .phy_id_mask = 0xffffffff, - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .flags = 0, .probe = lxt973_probe, .config_aneg = lxt973_config_aneg, @@ -275,7 +275,7 @@ static struct phy_driver lxt97x_driver[] = { .phy_id = 0x00137a10, .name = "LXT973", .phy_id_mask = 0xfffffff0, - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .flags = 0, .probe = lxt973_probe, .config_aneg = lxt973_config_aneg, diff --git a/drivers/net/phy/marvell.c b/drivers/net/phy/marvell.c index f76c4048b9780e0f058abc1cb8dbc1cf7bbe93eb..a7796134e3be0e287dbaa65d96e163c073b703e3 100644 --- a/drivers/net/phy/marvell.c +++ b/drivers/net/phy/marvell.c @@ -29,6 +29,7 @@ #include #include #include +#include #include #include @@ -91,6 +92,14 @@ #define MII_88E1510_TEMP_SENSOR 0x1b #define MII_88E1510_TEMP_SENSOR_MASK 0xff +#define MII_88E1540_COPPER_CTRL3 0x1a +#define MII_88E1540_COPPER_CTRL3_LINK_DOWN_DELAY_MASK GENMASK(11, 10) +#define MII_88E1540_COPPER_CTRL3_LINK_DOWN_DELAY_00MS 0 +#define MII_88E1540_COPPER_CTRL3_LINK_DOWN_DELAY_10MS 1 +#define MII_88E1540_COPPER_CTRL3_LINK_DOWN_DELAY_20MS 2 +#define MII_88E1540_COPPER_CTRL3_LINK_DOWN_DELAY_40MS 3 +#define MII_88E1540_COPPER_CTRL3_FAST_LINK_DOWN BIT(9) + #define MII_88E6390_MISC_TEST 0x1b #define MII_88E6390_MISC_TEST_SAMPLE_1S 0 #define MII_88E6390_MISC_TEST_SAMPLE_10MS BIT(14) @@ -128,6 +137,7 @@ #define MII_PHY_LED_CTRL 16 #define MII_88E1121_PHY_LED_DEF 0x0030 #define MII_88E1510_PHY_LED_DEF 0x1177 +#define MII_88E1510_PHY_LED0_LINK_LED1_ACTIVE 0x1040 #define MII_M1011_PHY_STATUS 0x11 #define MII_M1011_PHY_STATUS_1000 0x8000 @@ -624,7 +634,10 @@ static void marvell_config_led(struct phy_device *phydev) * LED[2] .. Blink, Activity */ case MARVELL_PHY_FAMILY_ID(MARVELL_PHY_ID_88E1510): - def_config = MII_88E1510_PHY_LED_DEF; + if (phydev->dev_flags & MARVELL_PHY_LED0_LINK_LED1_ACTIVE) + def_config = MII_88E1510_PHY_LED0_LINK_LED1_ACTIVE; + else + def_config = MII_88E1510_PHY_LED_DEF; break; default: return; @@ -1025,6 +1038,101 @@ static int m88e1145_config_init(struct phy_device *phydev) return 0; } +static int m88e1540_get_fld(struct phy_device *phydev, u8 *msecs) +{ + int val; + + val = phy_read(phydev, MII_88E1540_COPPER_CTRL3); + if (val < 0) + return val; + + if (!(val & MII_88E1540_COPPER_CTRL3_FAST_LINK_DOWN)) { + *msecs = ETHTOOL_PHY_FAST_LINK_DOWN_OFF; + return 0; + } + + val = FIELD_GET(MII_88E1540_COPPER_CTRL3_LINK_DOWN_DELAY_MASK, val); + + switch (val) { + case MII_88E1540_COPPER_CTRL3_LINK_DOWN_DELAY_00MS: + *msecs = 0; + break; + case MII_88E1540_COPPER_CTRL3_LINK_DOWN_DELAY_10MS: + *msecs = 10; + break; + case MII_88E1540_COPPER_CTRL3_LINK_DOWN_DELAY_20MS: + *msecs = 20; + break; + case MII_88E1540_COPPER_CTRL3_LINK_DOWN_DELAY_40MS: + *msecs = 40; + break; + default: + return -EINVAL; + } + + return 0; +} + +static int m88e1540_set_fld(struct phy_device *phydev, const u8 *msecs) +{ + struct ethtool_eee eee; + int val, ret; + + if (*msecs == ETHTOOL_PHY_FAST_LINK_DOWN_OFF) + return phy_clear_bits(phydev, MII_88E1540_COPPER_CTRL3, + MII_88E1540_COPPER_CTRL3_FAST_LINK_DOWN); + + /* According to the Marvell data sheet EEE must be disabled for + * Fast Link Down detection to work properly + */ + ret = phy_ethtool_get_eee(phydev, &eee); + if (!ret && eee.eee_enabled) { + phydev_warn(phydev, "Fast Link Down detection requires EEE to be disabled!\n"); + return -EBUSY; + } + + if (*msecs <= 5) + val = MII_88E1540_COPPER_CTRL3_LINK_DOWN_DELAY_00MS; + else if (*msecs <= 15) + val = MII_88E1540_COPPER_CTRL3_LINK_DOWN_DELAY_10MS; + else if (*msecs <= 30) + val = MII_88E1540_COPPER_CTRL3_LINK_DOWN_DELAY_20MS; + else + val = MII_88E1540_COPPER_CTRL3_LINK_DOWN_DELAY_40MS; + + val = FIELD_PREP(MII_88E1540_COPPER_CTRL3_LINK_DOWN_DELAY_MASK, val); + + ret = phy_modify(phydev, MII_88E1540_COPPER_CTRL3, + MII_88E1540_COPPER_CTRL3_LINK_DOWN_DELAY_MASK, val); + if (ret) + return ret; + + return phy_set_bits(phydev, MII_88E1540_COPPER_CTRL3, + MII_88E1540_COPPER_CTRL3_FAST_LINK_DOWN); +} + +static int m88e1540_get_tunable(struct phy_device *phydev, + struct ethtool_tunable *tuna, void *data) +{ + switch (tuna->id) { + case ETHTOOL_PHY_FAST_LINK_DOWN: + return m88e1540_get_fld(phydev, data); + default: + return -EOPNOTSUPP; + } +} + +static int m88e1540_set_tunable(struct phy_device *phydev, + struct ethtool_tunable *tuna, const void *data) +{ + switch (tuna->id) { + case ETHTOOL_PHY_FAST_LINK_DOWN: + return m88e1540_set_fld(phydev, data); + default: + return -EOPNOTSUPP; + } +} + /* The VOD can be out of specification on link up. Poke an * undocumented register, in an undocumented page, with a magic value * to fix this. @@ -2024,7 +2132,7 @@ static struct phy_driver marvell_drivers[] = { .phy_id = MARVELL_PHY_ID_88E1101, .phy_id_mask = MARVELL_PHY_ID_MASK, .name = "Marvell 88E1101", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .probe = marvell_probe, .config_init = &marvell_config_init, .config_aneg = &m88e1101_config_aneg, @@ -2042,7 +2150,7 @@ static struct phy_driver marvell_drivers[] = { .phy_id = MARVELL_PHY_ID_88E1112, .phy_id_mask = MARVELL_PHY_ID_MASK, .name = "Marvell 88E1112", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .probe = marvell_probe, .config_init = &m88e1111_config_init, .config_aneg = &marvell_config_aneg, @@ -2060,7 +2168,7 @@ static struct phy_driver marvell_drivers[] = { .phy_id = MARVELL_PHY_ID_88E1111, .phy_id_mask = MARVELL_PHY_ID_MASK, .name = "Marvell 88E1111", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .probe = marvell_probe, .config_init = &m88e1111_config_init, .config_aneg = &marvell_config_aneg, @@ -2079,7 +2187,7 @@ static struct phy_driver marvell_drivers[] = { .phy_id = MARVELL_PHY_ID_88E1118, .phy_id_mask = MARVELL_PHY_ID_MASK, .name = "Marvell 88E1118", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .probe = marvell_probe, .config_init = &m88e1118_config_init, .config_aneg = &m88e1118_config_aneg, @@ -2097,7 +2205,7 @@ static struct phy_driver marvell_drivers[] = { .phy_id = MARVELL_PHY_ID_88E1121R, .phy_id_mask = MARVELL_PHY_ID_MASK, .name = "Marvell 88E1121R", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .probe = &m88e1121_probe, .config_init = &marvell_config_init, .config_aneg = &m88e1121_config_aneg, @@ -2117,7 +2225,7 @@ static struct phy_driver marvell_drivers[] = { .phy_id = MARVELL_PHY_ID_88E1318S, .phy_id_mask = MARVELL_PHY_ID_MASK, .name = "Marvell 88E1318S", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .probe = marvell_probe, .config_init = &m88e1318_config_init, .config_aneg = &m88e1318_config_aneg, @@ -2139,7 +2247,7 @@ static struct phy_driver marvell_drivers[] = { .phy_id = MARVELL_PHY_ID_88E1145, .phy_id_mask = MARVELL_PHY_ID_MASK, .name = "Marvell 88E1145", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .probe = marvell_probe, .config_init = &m88e1145_config_init, .config_aneg = &m88e1101_config_aneg, @@ -2158,7 +2266,7 @@ static struct phy_driver marvell_drivers[] = { .phy_id = MARVELL_PHY_ID_88E1149R, .phy_id_mask = MARVELL_PHY_ID_MASK, .name = "Marvell 88E1149R", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .probe = marvell_probe, .config_init = &m88e1149_config_init, .config_aneg = &m88e1118_config_aneg, @@ -2176,7 +2284,7 @@ static struct phy_driver marvell_drivers[] = { .phy_id = MARVELL_PHY_ID_88E1240, .phy_id_mask = MARVELL_PHY_ID_MASK, .name = "Marvell 88E1240", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .probe = marvell_probe, .config_init = &m88e1111_config_init, .config_aneg = &marvell_config_aneg, @@ -2194,7 +2302,7 @@ static struct phy_driver marvell_drivers[] = { .phy_id = MARVELL_PHY_ID_88E1116R, .phy_id_mask = MARVELL_PHY_ID_MASK, .name = "Marvell 88E1116R", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .probe = marvell_probe, .config_init = &m88e1116r_config_init, .ack_interrupt = &marvell_ack_interrupt, @@ -2234,7 +2342,7 @@ static struct phy_driver marvell_drivers[] = { .phy_id = MARVELL_PHY_ID_88E1540, .phy_id_mask = MARVELL_PHY_ID_MASK, .name = "Marvell 88E1540", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .probe = m88e1510_probe, .config_init = &marvell_config_init, .config_aneg = &m88e1510_config_aneg, @@ -2249,13 +2357,15 @@ static struct phy_driver marvell_drivers[] = { .get_sset_count = marvell_get_sset_count, .get_strings = marvell_get_strings, .get_stats = marvell_get_stats, + .get_tunable = m88e1540_get_tunable, + .set_tunable = m88e1540_set_tunable, }, { .phy_id = MARVELL_PHY_ID_88E1545, .phy_id_mask = MARVELL_PHY_ID_MASK, .name = "Marvell 88E1545", .probe = m88e1510_probe, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = &marvell_config_init, .config_aneg = &m88e1510_config_aneg, .read_status = &marvell_read_status, @@ -2274,7 +2384,7 @@ static struct phy_driver marvell_drivers[] = { .phy_id = MARVELL_PHY_ID_88E3016, .phy_id_mask = MARVELL_PHY_ID_MASK, .name = "Marvell 88E3016", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .probe = marvell_probe, .config_init = &m88e3016_config_init, .aneg_done = &marvell_aneg_done, @@ -2294,7 +2404,7 @@ static struct phy_driver marvell_drivers[] = { .phy_id = MARVELL_PHY_ID_88E6390, .phy_id_mask = MARVELL_PHY_ID_MASK, .name = "Marvell 88E6390", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .probe = m88e6390_probe, .config_init = &marvell_config_init, .config_aneg = &m88e6390_config_aneg, @@ -2309,6 +2419,8 @@ static struct phy_driver marvell_drivers[] = { .get_sset_count = marvell_get_sset_count, .get_strings = marvell_get_strings, .get_stats = marvell_get_stats, + .get_tunable = m88e1540_get_tunable, + .set_tunable = m88e1540_set_tunable, }, }; diff --git a/drivers/net/phy/marvell10g.c b/drivers/net/phy/marvell10g.c index 100b401b1f4a125343afbf880a1ced60cc033d42..238a20e13d6abfbf9e6cfa61033b6153eff28de5 100644 --- a/drivers/net/phy/marvell10g.c +++ b/drivers/net/phy/marvell10g.c @@ -48,6 +48,8 @@ enum { MV_AN_STAT1000 = 0x8001, /* 1000base-T status register */ /* Vendor2 MMD registers */ + MV_V2_PORT_CTRL = 0xf001, + MV_V2_PORT_CTRL_PWRDOWN = 0x0800, MV_V2_TEMP_CTRL = 0xf08a, MV_V2_TEMP_CTRL_MASK = 0xc000, MV_V2_TEMP_CTRL_SAMPLE = 0x0000, @@ -226,11 +228,19 @@ static int mv3310_probe(struct phy_device *phydev) static int mv3310_suspend(struct phy_device *phydev) { - return 0; + return phy_set_bits_mmd(phydev, MDIO_MMD_VEND2, MV_V2_PORT_CTRL, + MV_V2_PORT_CTRL_PWRDOWN); } static int mv3310_resume(struct phy_device *phydev) { + int ret; + + ret = phy_clear_bits_mmd(phydev, MDIO_MMD_VEND2, MV_V2_PORT_CTRL, + MV_V2_PORT_CTRL_PWRDOWN); + if (ret) + return ret; + return mv3310_hwmon_config(phydev, true); } @@ -472,8 +482,9 @@ static struct phy_driver mv3310_drivers[] = { .phy_id = MARVELL_PHY_ID_88E2110, .phy_id_mask = MARVELL_PHY_ID_MASK, .name = "mv88x2110", - .get_features = genphy_c45_pma_read_abilities, .probe = mv3310_probe, + .suspend = mv3310_suspend, + .resume = mv3310_resume, .soft_reset = genphy_no_soft_reset, .config_init = mv3310_config_init, .config_aneg = mv3310_config_aneg, diff --git a/drivers/net/phy/mdio-bcm-unimac.c b/drivers/net/phy/mdio-bcm-unimac.c index 8295bc7c8c204d540aed80e761c0640b740fbd9e..4a28fb29adaa8d5df841e61d3e86ab372fb7b267 100644 --- a/drivers/net/phy/mdio-bcm-unimac.c +++ b/drivers/net/phy/mdio-bcm-unimac.c @@ -92,10 +92,7 @@ static int unimac_mdio_poll(void *wait_func_data) usleep_range(1000, 2000); } while (--timeout); - if (!timeout) - return -ETIMEDOUT; - - return 0; + return -ETIMEDOUT; } static int unimac_mdio_read(struct mii_bus *bus, int phy_id, int reg) @@ -292,7 +289,7 @@ static int unimac_mdio_probe(struct platform_device *pdev) platform_set_drvdata(pdev, priv); - dev_info(&pdev->dev, "Broadcom UniMAC MDIO bus at 0x%p\n", priv->base); + dev_info(&pdev->dev, "Broadcom UniMAC MDIO bus\n"); return 0; diff --git a/drivers/net/phy/mdio-mux-meson-g12a.c b/drivers/net/phy/mdio-mux-meson-g12a.c new file mode 100644 index 0000000000000000000000000000000000000000..6fa29ea8e2a363f99d45eae647d78025eb831baf --- /dev/null +++ b/drivers/net/phy/mdio-mux-meson-g12a.c @@ -0,0 +1,380 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2019 Baylibre, SAS. + * Author: Jerome Brunet + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define ETH_PLL_STS 0x40 +#define ETH_PLL_CTL0 0x44 +#define PLL_CTL0_LOCK_DIG BIT(30) +#define PLL_CTL0_RST BIT(29) +#define PLL_CTL0_EN BIT(28) +#define PLL_CTL0_SEL BIT(23) +#define PLL_CTL0_N GENMASK(14, 10) +#define PLL_CTL0_M GENMASK(8, 0) +#define PLL_LOCK_TIMEOUT 1000000 +#define PLL_MUX_NUM_PARENT 2 +#define ETH_PLL_CTL1 0x48 +#define ETH_PLL_CTL2 0x4c +#define ETH_PLL_CTL3 0x50 +#define ETH_PLL_CTL4 0x54 +#define ETH_PLL_CTL5 0x58 +#define ETH_PLL_CTL6 0x5c +#define ETH_PLL_CTL7 0x60 + +#define ETH_PHY_CNTL0 0x80 +#define EPHY_G12A_ID 0x33000180 +#define ETH_PHY_CNTL1 0x84 +#define PHY_CNTL1_ST_MODE GENMASK(2, 0) +#define PHY_CNTL1_ST_PHYADD GENMASK(7, 3) +#define EPHY_DFLT_ADD 8 +#define PHY_CNTL1_MII_MODE GENMASK(15, 14) +#define EPHY_MODE_RMII 0x1 +#define PHY_CNTL1_CLK_EN BIT(16) +#define PHY_CNTL1_CLKFREQ BIT(17) +#define PHY_CNTL1_PHY_ENB BIT(18) +#define ETH_PHY_CNTL2 0x88 +#define PHY_CNTL2_USE_INTERNAL BIT(5) +#define PHY_CNTL2_SMI_SRC_MAC BIT(6) +#define PHY_CNTL2_RX_CLK_EPHY BIT(9) + +#define MESON_G12A_MDIO_EXTERNAL_ID 0 +#define MESON_G12A_MDIO_INTERNAL_ID 1 + +struct g12a_mdio_mux { + bool pll_is_enabled; + void __iomem *regs; + void *mux_handle; + struct clk *pclk; + struct clk *pll; +}; + +struct g12a_ephy_pll { + void __iomem *base; + struct clk_hw hw; +}; + +#define g12a_ephy_pll_to_dev(_hw) \ + container_of(_hw, struct g12a_ephy_pll, hw) + +static unsigned long g12a_ephy_pll_recalc_rate(struct clk_hw *hw, + unsigned long parent_rate) +{ + struct g12a_ephy_pll *pll = g12a_ephy_pll_to_dev(hw); + u32 val, m, n; + + val = readl(pll->base + ETH_PLL_CTL0); + m = FIELD_GET(PLL_CTL0_M, val); + n = FIELD_GET(PLL_CTL0_N, val); + + return parent_rate * m / n; +} + +static int g12a_ephy_pll_enable(struct clk_hw *hw) +{ + struct g12a_ephy_pll *pll = g12a_ephy_pll_to_dev(hw); + u32 val = readl(pll->base + ETH_PLL_CTL0); + + /* Apply both enable an reset */ + val |= PLL_CTL0_RST | PLL_CTL0_EN; + writel(val, pll->base + ETH_PLL_CTL0); + + /* Clear the reset to let PLL lock */ + val &= ~PLL_CTL0_RST; + writel(val, pll->base + ETH_PLL_CTL0); + + /* Poll on the digital lock instead of the usual analog lock + * This is done because bit 31 is unreliable on some SoC. Bit + * 31 may indicate that the PLL is not lock eventhough the clock + * is actually running + */ + return readl_poll_timeout(pll->base + ETH_PLL_CTL0, val, + val & PLL_CTL0_LOCK_DIG, 0, PLL_LOCK_TIMEOUT); +} + +static void g12a_ephy_pll_disable(struct clk_hw *hw) +{ + struct g12a_ephy_pll *pll = g12a_ephy_pll_to_dev(hw); + u32 val; + + val = readl(pll->base + ETH_PLL_CTL0); + val &= ~PLL_CTL0_EN; + val |= PLL_CTL0_RST; + writel(val, pll->base + ETH_PLL_CTL0); +} + +static int g12a_ephy_pll_is_enabled(struct clk_hw *hw) +{ + struct g12a_ephy_pll *pll = g12a_ephy_pll_to_dev(hw); + unsigned int val; + + val = readl(pll->base + ETH_PLL_CTL0); + + return (val & PLL_CTL0_LOCK_DIG) ? 1 : 0; +} + +static void g12a_ephy_pll_init(struct clk_hw *hw) +{ + struct g12a_ephy_pll *pll = g12a_ephy_pll_to_dev(hw); + + /* Apply PLL HW settings */ + writel(0x29c0040a, pll->base + ETH_PLL_CTL0); + writel(0x927e0000, pll->base + ETH_PLL_CTL1); + writel(0xac5f49e5, pll->base + ETH_PLL_CTL2); + writel(0x00000000, pll->base + ETH_PLL_CTL3); + writel(0x00000000, pll->base + ETH_PLL_CTL4); + writel(0x20200000, pll->base + ETH_PLL_CTL5); + writel(0x0000c002, pll->base + ETH_PLL_CTL6); + writel(0x00000023, pll->base + ETH_PLL_CTL7); +} + +static const struct clk_ops g12a_ephy_pll_ops = { + .recalc_rate = g12a_ephy_pll_recalc_rate, + .is_enabled = g12a_ephy_pll_is_enabled, + .enable = g12a_ephy_pll_enable, + .disable = g12a_ephy_pll_disable, + .init = g12a_ephy_pll_init, +}; + +static int g12a_enable_internal_mdio(struct g12a_mdio_mux *priv) +{ + int ret; + + /* Enable the phy clock */ + if (!priv->pll_is_enabled) { + ret = clk_prepare_enable(priv->pll); + if (ret) + return ret; + } + + priv->pll_is_enabled = true; + + /* Initialize ephy control */ + writel(EPHY_G12A_ID, priv->regs + ETH_PHY_CNTL0); + writel(FIELD_PREP(PHY_CNTL1_ST_MODE, 3) | + FIELD_PREP(PHY_CNTL1_ST_PHYADD, EPHY_DFLT_ADD) | + FIELD_PREP(PHY_CNTL1_MII_MODE, EPHY_MODE_RMII) | + PHY_CNTL1_CLK_EN | + PHY_CNTL1_CLKFREQ | + PHY_CNTL1_PHY_ENB, + priv->regs + ETH_PHY_CNTL1); + writel(PHY_CNTL2_USE_INTERNAL | + PHY_CNTL2_SMI_SRC_MAC | + PHY_CNTL2_RX_CLK_EPHY, + priv->regs + ETH_PHY_CNTL2); + + return 0; +} + +static int g12a_enable_external_mdio(struct g12a_mdio_mux *priv) +{ + /* Reset the mdio bus mux */ + writel_relaxed(0x0, priv->regs + ETH_PHY_CNTL2); + + /* Disable the phy clock if enabled */ + if (priv->pll_is_enabled) { + clk_disable_unprepare(priv->pll); + priv->pll_is_enabled = false; + } + + return 0; +} + +static int g12a_mdio_switch_fn(int current_child, int desired_child, + void *data) +{ + struct g12a_mdio_mux *priv = dev_get_drvdata(data); + + if (current_child == desired_child) + return 0; + + switch (desired_child) { + case MESON_G12A_MDIO_EXTERNAL_ID: + return g12a_enable_external_mdio(priv); + case MESON_G12A_MDIO_INTERNAL_ID: + return g12a_enable_internal_mdio(priv); + default: + return -EINVAL; + } +} + +static const struct of_device_id g12a_mdio_mux_match[] = { + { .compatible = "amlogic,g12a-mdio-mux", }, + {}, +}; +MODULE_DEVICE_TABLE(of, g12a_mdio_mux_match); + +static int g12a_ephy_glue_clk_register(struct device *dev) +{ + struct g12a_mdio_mux *priv = dev_get_drvdata(dev); + const char *parent_names[PLL_MUX_NUM_PARENT]; + struct clk_init_data init; + struct g12a_ephy_pll *pll; + struct clk_mux *mux; + struct clk *clk; + char *name; + int i; + + /* get the mux parents */ + for (i = 0; i < PLL_MUX_NUM_PARENT; i++) { + char in_name[8]; + + snprintf(in_name, sizeof(in_name), "clkin%d", i); + clk = devm_clk_get(dev, in_name); + if (IS_ERR(clk)) { + if (PTR_ERR(clk) != -EPROBE_DEFER) + dev_err(dev, "Missing clock %s\n", in_name); + return PTR_ERR(clk); + } + + parent_names[i] = __clk_get_name(clk); + } + + /* create the input mux */ + mux = devm_kzalloc(dev, sizeof(*mux), GFP_KERNEL); + if (!mux) + return -ENOMEM; + + name = kasprintf(GFP_KERNEL, "%s#mux", dev_name(dev)); + if (!name) + return -ENOMEM; + + init.name = name; + init.ops = &clk_mux_ro_ops; + init.flags = 0; + init.parent_names = parent_names; + init.num_parents = PLL_MUX_NUM_PARENT; + + mux->reg = priv->regs + ETH_PLL_CTL0; + mux->shift = __ffs(PLL_CTL0_SEL); + mux->mask = PLL_CTL0_SEL >> mux->shift; + mux->hw.init = &init; + + clk = devm_clk_register(dev, &mux->hw); + kfree(name); + if (IS_ERR(clk)) { + dev_err(dev, "failed to register input mux\n"); + return PTR_ERR(clk); + } + + /* create the pll */ + pll = devm_kzalloc(dev, sizeof(*pll), GFP_KERNEL); + if (!pll) + return -ENOMEM; + + name = kasprintf(GFP_KERNEL, "%s#pll", dev_name(dev)); + if (!name) + return -ENOMEM; + + init.name = name; + init.ops = &g12a_ephy_pll_ops; + init.flags = 0; + parent_names[0] = __clk_get_name(clk); + init.parent_names = parent_names; + init.num_parents = 1; + + pll->base = priv->regs; + pll->hw.init = &init; + + clk = devm_clk_register(dev, &pll->hw); + kfree(name); + if (IS_ERR(clk)) { + dev_err(dev, "failed to register input mux\n"); + return PTR_ERR(clk); + } + + priv->pll = clk; + + return 0; +} + +static int g12a_mdio_mux_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + struct g12a_mdio_mux *priv; + struct resource *res; + int ret; + + priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL); + if (!priv) + return -ENOMEM; + + platform_set_drvdata(pdev, priv); + + res = platform_get_resource(pdev, IORESOURCE_MEM, 0); + priv->regs = devm_ioremap_resource(dev, res); + if (IS_ERR(priv->regs)) + return PTR_ERR(priv->regs); + + priv->pclk = devm_clk_get(dev, "pclk"); + if (IS_ERR(priv->pclk)) { + ret = PTR_ERR(priv->pclk); + if (ret != -EPROBE_DEFER) + dev_err(dev, "failed to get peripheral clock\n"); + return ret; + } + + /* Make sure the device registers are clocked */ + ret = clk_prepare_enable(priv->pclk); + if (ret) { + dev_err(dev, "failed to enable peripheral clock"); + return ret; + } + + /* Register PLL in CCF */ + ret = g12a_ephy_glue_clk_register(dev); + if (ret) + goto err; + + ret = mdio_mux_init(dev, dev->of_node, g12a_mdio_switch_fn, + &priv->mux_handle, dev, NULL); + if (ret) { + if (ret != -EPROBE_DEFER) + dev_err(dev, "mdio multiplexer init failed: %d", ret); + goto err; + } + + return 0; + +err: + clk_disable_unprepare(priv->pclk); + return ret; +} + +static int g12a_mdio_mux_remove(struct platform_device *pdev) +{ + struct g12a_mdio_mux *priv = platform_get_drvdata(pdev); + + mdio_mux_uninit(priv->mux_handle); + + if (priv->pll_is_enabled) + clk_disable_unprepare(priv->pll); + + clk_disable_unprepare(priv->pclk); + + return 0; +} + +static struct platform_driver g12a_mdio_mux_driver = { + .probe = g12a_mdio_mux_probe, + .remove = g12a_mdio_mux_remove, + .driver = { + .name = "g12a-mdio_mux", + .of_match_table = g12a_mdio_mux_match, + }, +}; +module_platform_driver(g12a_mdio_mux_driver); + +MODULE_DESCRIPTION("Amlogic G12a MDIO multiplexer driver"); +MODULE_AUTHOR("Jerome Brunet "); +MODULE_LICENSE("GPL v2"); diff --git a/drivers/net/phy/mdio_bus.c b/drivers/net/phy/mdio_bus.c index 4be4cc09eb90b0c0f03e6fc5f1e3eb116671c0d9..bd04fe762056f49c5c28ac3256df820e0e7544ec 100644 --- a/drivers/net/phy/mdio_bus.c +++ b/drivers/net/phy/mdio_bus.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -55,10 +56,25 @@ static int mdiobus_register_gpiod(struct mdio_device *mdiodev) return PTR_ERR(gpiod); } - mdiodev->reset = gpiod; + mdiodev->reset_gpio = gpiod; - /* Assert the reset signal again */ - mdio_device_reset(mdiodev, 1); + return 0; +} + +static int mdiobus_register_reset(struct mdio_device *mdiodev) +{ + struct reset_control *reset = NULL; + + if (mdiodev->dev.of_node) + reset = devm_reset_control_get_exclusive(&mdiodev->dev, + "phy"); + if (PTR_ERR(reset) == -ENOENT || + PTR_ERR(reset) == -ENOTSUPP) + reset = NULL; + else if (IS_ERR(reset)) + return PTR_ERR(reset); + + mdiodev->reset_ctrl = reset; return 0; } @@ -74,6 +90,13 @@ int mdiobus_register_device(struct mdio_device *mdiodev) err = mdiobus_register_gpiod(mdiodev); if (err) return err; + + err = mdiobus_register_reset(mdiodev); + if (err) + return err; + + /* Assert the reset signal */ + mdio_device_reset(mdiodev, 1); } mdiodev->bus->mdio_map[mdiodev->addr] = mdiodev; @@ -446,8 +469,8 @@ void mdiobus_unregister(struct mii_bus *bus) if (!mdiodev) continue; - if (mdiodev->reset) - gpiod_put(mdiodev->reset); + if (mdiodev->reset_gpio) + gpiod_put(mdiodev->reset_gpio); mdiodev->device_remove(mdiodev); mdiodev->device_free(mdiodev); diff --git a/drivers/net/phy/mdio_device.c b/drivers/net/phy/mdio_device.c index 887076292e50fa9f5338b4658eb995760b739c5b..e282600bd83e269e7d91973599e7b7f0329da12e 100644 --- a/drivers/net/phy/mdio_device.c +++ b/drivers/net/phy/mdio_device.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include #include @@ -116,10 +117,18 @@ void mdio_device_reset(struct mdio_device *mdiodev, int value) { unsigned int d; - if (!mdiodev->reset) + if (!mdiodev->reset_gpio && !mdiodev->reset_ctrl) return; - gpiod_set_value(mdiodev->reset, value); + if (mdiodev->reset_gpio) + gpiod_set_value(mdiodev->reset_gpio, value); + + if (mdiodev->reset_ctrl) { + if (value) + reset_control_assert(mdiodev->reset_ctrl); + else + reset_control_deassert(mdiodev->reset_ctrl); + } d = value ? mdiodev->reset_assert_delay : mdiodev->reset_deassert_delay; if (d) diff --git a/drivers/net/phy/meson-gxl.c b/drivers/net/phy/meson-gxl.c index 0eec2913c289b83a77a238aca2da64e558378336..fa80d6dce8ee32a82a078c606bcfdefa9ea97d3e 100644 --- a/drivers/net/phy/meson-gxl.c +++ b/drivers/net/phy/meson-gxl.c @@ -224,24 +224,33 @@ static int meson_gxl_config_intr(struct phy_device *phydev) static struct phy_driver meson_gxl_phy[] = { { - .phy_id = 0x01814400, - .phy_id_mask = 0xfffffff0, + PHY_ID_MATCH_EXACT(0x01814400), .name = "Meson GXL Internal PHY", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .flags = PHY_IS_INTERNAL, .soft_reset = genphy_soft_reset, .config_init = meson_gxl_config_init, - .aneg_done = genphy_aneg_done, .read_status = meson_gxl_read_status, .ack_interrupt = meson_gxl_ack_interrupt, .config_intr = meson_gxl_config_intr, .suspend = genphy_suspend, .resume = genphy_resume, + }, { + PHY_ID_MATCH_EXACT(0x01803301), + .name = "Meson G12A Internal PHY", + /* PHY_BASIC_FEATURES */ + .flags = PHY_IS_INTERNAL, + .soft_reset = genphy_soft_reset, + .ack_interrupt = meson_gxl_ack_interrupt, + .config_intr = meson_gxl_config_intr, + .suspend = genphy_suspend, + .resume = genphy_resume, }, }; static struct mdio_device_id __maybe_unused meson_gxl_tbl[] = { - { 0x01814400, 0xfffffff0 }, + { PHY_ID_MATCH_VENDOR(0x01814400) }, + { PHY_ID_MATCH_VENDOR(0x01803301) }, { } }; diff --git a/drivers/net/phy/micrel.c b/drivers/net/phy/micrel.c index 352da24f1f337f02e572f1cf8b09e486e7e8f0f9..3c8186f269f9e088cee19bd509ba03e84ed58e1c 100644 --- a/drivers/net/phy/micrel.c +++ b/drivers/net/phy/micrel.c @@ -28,6 +28,7 @@ /* Operation Mode Strap Override */ #define MII_KSZPHY_OMSO 0x16 +#define KSZPHY_OMSO_FACTORY_TEST BIT(15) #define KSZPHY_OMSO_B_CAST_OFF BIT(9) #define KSZPHY_OMSO_NAND_TREE_ON BIT(5) #define KSZPHY_OMSO_RMII_OVERRIDE BIT(1) @@ -340,6 +341,18 @@ static int ksz8041_config_aneg(struct phy_device *phydev) return genphy_config_aneg(phydev); } +static int ksz8081_config_init(struct phy_device *phydev) +{ + /* KSZPHY_OMSO_FACTORY_TEST is set at de-assertion of the reset line + * based on the RXER (KSZ8081RNA/RND) or TXC (KSZ8081MNX/RNB) pin. If a + * pull-down is missing, the factory test mode should be cleared by + * manually writing a 0. + */ + phy_clear_bits(phydev, MII_KSZPHY_OMSO, KSZPHY_OMSO_FACTORY_TEST); + + return kszphy_config_init(phydev); +} + static int ksz8061_config_init(struct phy_device *phydev) { int ret; @@ -738,6 +751,31 @@ static int ksz8873mll_read_status(struct phy_device *phydev) return 0; } +static int ksz9031_get_features(struct phy_device *phydev) +{ + int ret; + + ret = genphy_read_abilities(phydev); + if (ret < 0) + return ret; + + /* Silicon Errata Sheet (DS80000691D or DS80000692D): + * Whenever the device's Asymmetric Pause capability is set to 1, + * link-up may fail after a link-up to link-down transition. + * + * Workaround: + * Do not enable the Asymmetric Pause capability bit. + */ + linkmode_clear_bit(ETHTOOL_LINK_MODE_Asym_Pause_BIT, phydev->supported); + + /* We force setting the Pause capability as the core will force the + * Asymmetric Pause capability to 1 otherwise. + */ + linkmode_set_bit(ETHTOOL_LINK_MODE_Pause_BIT, phydev->supported); + + return 0; +} + static int ksz9031_read_status(struct phy_device *phydev) { int err; @@ -908,7 +946,7 @@ static struct phy_driver ksphy_driver[] = { .phy_id = PHY_ID_KS8737, .phy_id_mask = MICREL_PHY_ID_MASK, .name = "Micrel KS8737", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .driver_data = &ks8737_type, .config_init = kszphy_config_init, .ack_interrupt = kszphy_ack_interrupt, @@ -919,7 +957,7 @@ static struct phy_driver ksphy_driver[] = { .phy_id = PHY_ID_KSZ8021, .phy_id_mask = 0x00ffffff, .name = "Micrel KSZ8021 or KSZ8031", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .driver_data = &ksz8021_type, .probe = kszphy_probe, .config_init = kszphy_config_init, @@ -934,7 +972,7 @@ static struct phy_driver ksphy_driver[] = { .phy_id = PHY_ID_KSZ8031, .phy_id_mask = 0x00ffffff, .name = "Micrel KSZ8031", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .driver_data = &ksz8021_type, .probe = kszphy_probe, .config_init = kszphy_config_init, @@ -949,7 +987,7 @@ static struct phy_driver ksphy_driver[] = { .phy_id = PHY_ID_KSZ8041, .phy_id_mask = MICREL_PHY_ID_MASK, .name = "Micrel KSZ8041", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .driver_data = &ksz8041_type, .probe = kszphy_probe, .config_init = ksz8041_config_init, @@ -965,7 +1003,7 @@ static struct phy_driver ksphy_driver[] = { .phy_id = PHY_ID_KSZ8041RNLI, .phy_id_mask = MICREL_PHY_ID_MASK, .name = "Micrel KSZ8041RNLI", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .driver_data = &ksz8041_type, .probe = kszphy_probe, .config_init = kszphy_config_init, @@ -980,7 +1018,7 @@ static struct phy_driver ksphy_driver[] = { .phy_id = PHY_ID_KSZ8051, .phy_id_mask = MICREL_PHY_ID_MASK, .name = "Micrel KSZ8051", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .driver_data = &ksz8051_type, .probe = kszphy_probe, .config_init = kszphy_config_init, @@ -995,7 +1033,7 @@ static struct phy_driver ksphy_driver[] = { .phy_id = PHY_ID_KSZ8001, .name = "Micrel KSZ8001 or KS8721", .phy_id_mask = 0x00fffffc, - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .driver_data = &ksz8041_type, .probe = kszphy_probe, .config_init = kszphy_config_init, @@ -1010,10 +1048,10 @@ static struct phy_driver ksphy_driver[] = { .phy_id = PHY_ID_KSZ8081, .name = "Micrel KSZ8081 or KSZ8091", .phy_id_mask = MICREL_PHY_ID_MASK, - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .driver_data = &ksz8081_type, .probe = kszphy_probe, - .config_init = kszphy_config_init, + .config_init = ksz8081_config_init, .ack_interrupt = kszphy_ack_interrupt, .config_intr = kszphy_config_intr, .get_sset_count = kszphy_get_sset_count, @@ -1025,7 +1063,7 @@ static struct phy_driver ksphy_driver[] = { .phy_id = PHY_ID_KSZ8061, .name = "Micrel KSZ8061", .phy_id_mask = MICREL_PHY_ID_MASK, - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = ksz8061_config_init, .ack_interrupt = kszphy_ack_interrupt, .config_intr = kszphy_config_intr, @@ -1035,7 +1073,7 @@ static struct phy_driver ksphy_driver[] = { .phy_id = PHY_ID_KSZ9021, .phy_id_mask = 0x000ffffe, .name = "Micrel KSZ9021 Gigabit PHY", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .driver_data = &ksz9021_type, .probe = kszphy_probe, .config_init = ksz9021_config_init, @@ -1052,9 +1090,9 @@ static struct phy_driver ksphy_driver[] = { .phy_id = PHY_ID_KSZ9031, .phy_id_mask = MICREL_PHY_ID_MASK, .name = "Micrel KSZ9031 Gigabit PHY", - .features = PHY_GBIT_FEATURES, .driver_data = &ksz9021_type, .probe = kszphy_probe, + .get_features = ksz9031_get_features, .config_init = ksz9031_config_init, .soft_reset = genphy_soft_reset, .read_status = ksz9031_read_status, @@ -1069,7 +1107,7 @@ static struct phy_driver ksphy_driver[] = { .phy_id = PHY_ID_KSZ9131, .phy_id_mask = MICREL_PHY_ID_MASK, .name = "Microchip KSZ9131 Gigabit PHY", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .driver_data = &ksz9021_type, .probe = kszphy_probe, .config_init = ksz9131_config_init, @@ -1085,7 +1123,7 @@ static struct phy_driver ksphy_driver[] = { .phy_id = PHY_ID_KSZ8873MLL, .phy_id_mask = MICREL_PHY_ID_MASK, .name = "Micrel KSZ8873MLL Switch", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = kszphy_config_init, .config_aneg = ksz8873mll_config_aneg, .read_status = ksz8873mll_read_status, @@ -1095,7 +1133,7 @@ static struct phy_driver ksphy_driver[] = { .phy_id = PHY_ID_KSZ886X, .phy_id_mask = MICREL_PHY_ID_MASK, .name = "Micrel KSZ886X Switch", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = kszphy_config_init, .suspend = genphy_suspend, .resume = genphy_resume, @@ -1103,7 +1141,7 @@ static struct phy_driver ksphy_driver[] = { .phy_id = PHY_ID_KSZ8795, .phy_id_mask = MICREL_PHY_ID_MASK, .name = "Micrel KSZ8795", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = kszphy_config_init, .config_aneg = ksz8873mll_config_aneg, .read_status = ksz8873mll_read_status, @@ -1113,7 +1151,7 @@ static struct phy_driver ksphy_driver[] = { .phy_id = PHY_ID_KSZ9477, .phy_id_mask = MICREL_PHY_ID_MASK, .name = "Microchip KSZ9477", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = kszphy_config_init, .suspend = genphy_suspend, .resume = genphy_resume, diff --git a/drivers/net/phy/microchip.c b/drivers/net/phy/microchip.c index c6cbb3aa8ae0939002a359988be8b08aff2d7ae9..eb1b3287fe0817a548424ab6e507f3664cf026b4 100644 --- a/drivers/net/phy/microchip.c +++ b/drivers/net/phy/microchip.c @@ -333,7 +333,7 @@ static struct phy_driver microchip_phy_driver[] = { .phy_id_mask = 0xfffffff0, .name = "Microchip LAN88xx", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .probe = lan88xx_probe, .remove = lan88xx_remove, diff --git a/drivers/net/phy/mscc.c b/drivers/net/phy/mscc.c index db50efb30df58bcacbed107bb1cf7eb2dd83ed21..28676af97b42c275f6d2df4a285d3d6d391ec7f6 100644 --- a/drivers/net/phy/mscc.c +++ b/drivers/net/phy/mscc.c @@ -85,12 +85,49 @@ enum rgmii_rx_clock_delay { #define LED_MODE_SEL_MASK(x) (GENMASK(3, 0) << LED_MODE_SEL_POS(x)) #define LED_MODE_SEL(x, mode) (((mode) << LED_MODE_SEL_POS(x)) & LED_MODE_SEL_MASK(x)) +#define MSCC_EXT_PAGE_CSR_CNTL_17 17 +#define MSCC_EXT_PAGE_CSR_CNTL_18 18 + +#define MSCC_EXT_PAGE_CSR_CNTL_19 19 +#define MSCC_PHY_CSR_CNTL_19_REG_ADDR(x) (x) +#define MSCC_PHY_CSR_CNTL_19_TARGET(x) ((x) << 12) +#define MSCC_PHY_CSR_CNTL_19_READ BIT(14) +#define MSCC_PHY_CSR_CNTL_19_CMD BIT(15) + +#define MSCC_EXT_PAGE_CSR_CNTL_20 20 +#define MSCC_PHY_CSR_CNTL_20_TARGET(x) (x) + +#define PHY_MCB_TARGET 0x07 +#define PHY_MCB_S6G_WRITE BIT(31) +#define PHY_MCB_S6G_READ BIT(30) + +#define PHY_S6G_PLL5G_CFG0 0x06 +#define PHY_S6G_LCPLL_CFG 0x11 +#define PHY_S6G_PLL_CFG 0x2b +#define PHY_S6G_COMMON_CFG 0x2c +#define PHY_S6G_GPC_CFG 0x2e +#define PHY_S6G_MISC_CFG 0x3b +#define PHY_MCB_S6G_CFG 0x3f +#define PHY_S6G_DFT_CFG2 0x3e +#define PHY_S6G_PLL_STATUS 0x31 +#define PHY_S6G_IB_STATUS0 0x2f + +#define PHY_S6G_SYS_RST_POS 31 +#define PHY_S6G_ENA_LANE_POS 18 +#define PHY_S6G_ENA_LOOP_POS 8 +#define PHY_S6G_QRATE_POS 6 +#define PHY_S6G_IF_MODE_POS 4 +#define PHY_S6G_PLL_ENA_OFFS_POS 21 +#define PHY_S6G_PLL_FSM_CTRL_DATA_POS 8 +#define PHY_S6G_PLL_FSM_ENA_POS 7 + #define MSCC_EXT_PAGE_ACCESS 31 #define MSCC_PHY_PAGE_STANDARD 0x0000 /* Standard registers */ #define MSCC_PHY_PAGE_EXTENDED 0x0001 /* Extended registers */ #define MSCC_PHY_PAGE_EXTENDED_2 0x0002 /* Extended reg - page 2 */ #define MSCC_PHY_PAGE_EXTENDED_3 0x0003 /* Extended reg - page 3 */ #define MSCC_PHY_PAGE_EXTENDED_4 0x0004 /* Extended reg - page 4 */ +#define MSCC_PHY_PAGE_CSR_CNTL MSCC_PHY_PAGE_EXTENDED_4 /* Extended reg - GPIO; this is a bank of registers that are shared for all PHYs * in the same package. */ @@ -216,6 +253,7 @@ enum rgmii_rx_clock_delay { #define MSCC_PHY_TR_MSB 18 /* Microsemi PHY ID's */ +#define PHY_ID_VSC8514 0x00070670 #define PHY_ID_VSC8530 0x00070560 #define PHY_ID_VSC8531 0x00070570 #define PHY_ID_VSC8540 0x00070760 @@ -1742,6 +1780,386 @@ static int vsc8584_did_interrupt(struct phy_device *phydev) return (rc < 0) ? 0 : rc & MII_VSC85XX_INT_MASK_MASK; } +static int vsc8514_config_pre_init(struct phy_device *phydev) +{ + /* These are the settings to override the silicon default + * values to handle hardware performance of PHY. They + * are set at Power-On state and remain until PHY Reset. + */ + const struct reg_val pre_init1[] = { + {0x0f90, 0x00688980}, + {0x0786, 0x00000003}, + {0x07fa, 0x0050100f}, + {0x0f82, 0x0012b002}, + {0x1686, 0x00000004}, + {0x168c, 0x00d2c46f}, + {0x17a2, 0x00000620}, + {0x16a0, 0x00eeffdd}, + {0x16a6, 0x00071448}, + {0x16a4, 0x0013132f}, + {0x16a8, 0x00000000}, + {0x0ffc, 0x00c0a028}, + {0x0fe8, 0x0091b06c}, + {0x0fea, 0x00041600}, + {0x0f80, 0x00fffaff}, + {0x0fec, 0x00901809}, + {0x0ffe, 0x00b01007}, + {0x16b0, 0x00eeff00}, + {0x16b2, 0x00007000}, + {0x16b4, 0x00000814}, + }; + unsigned int i; + u16 reg; + + phy_base_write(phydev, MSCC_EXT_PAGE_ACCESS, MSCC_PHY_PAGE_STANDARD); + + /* all writes below are broadcasted to all PHYs in the same package */ + reg = phy_base_read(phydev, MSCC_PHY_EXT_CNTL_STATUS); + reg |= SMI_BROADCAST_WR_EN; + phy_base_write(phydev, MSCC_PHY_EXT_CNTL_STATUS, reg); + + phy_base_write(phydev, MSCC_EXT_PAGE_ACCESS, MSCC_PHY_PAGE_TEST); + + reg = phy_base_read(phydev, MSCC_PHY_TEST_PAGE_8); + reg |= BIT(15); + phy_base_write(phydev, MSCC_PHY_TEST_PAGE_8, reg); + + phy_base_write(phydev, MSCC_EXT_PAGE_ACCESS, MSCC_PHY_PAGE_TR); + + for (i = 0; i < ARRAY_SIZE(pre_init1); i++) + vsc8584_csr_write(phydev, pre_init1[i].reg, pre_init1[i].val); + + phy_base_write(phydev, MSCC_EXT_PAGE_ACCESS, MSCC_PHY_PAGE_TEST); + + reg = phy_base_read(phydev, MSCC_PHY_TEST_PAGE_8); + reg &= ~BIT(15); + phy_base_write(phydev, MSCC_PHY_TEST_PAGE_8, reg); + + phy_base_write(phydev, MSCC_EXT_PAGE_ACCESS, MSCC_PHY_PAGE_STANDARD); + + reg = phy_base_read(phydev, MSCC_PHY_EXT_CNTL_STATUS); + reg &= ~SMI_BROADCAST_WR_EN; + phy_base_write(phydev, MSCC_PHY_EXT_CNTL_STATUS, reg); + + return 0; +} + +static u32 vsc85xx_csr_ctrl_phy_read(struct phy_device *phydev, + u32 target, u32 reg) +{ + unsigned long deadline; + u32 val, val_l, val_h; + + phy_base_write(phydev, MSCC_EXT_PAGE_ACCESS, MSCC_PHY_PAGE_CSR_CNTL); + + /* CSR registers are grouped under different Target IDs. + * 6-bit Target_ID is split between MSCC_EXT_PAGE_CSR_CNTL_20 and + * MSCC_EXT_PAGE_CSR_CNTL_19 registers. + * Target_ID[5:2] maps to bits[3:0] of MSCC_EXT_PAGE_CSR_CNTL_20 + * and Target_ID[1:0] maps to bits[13:12] of MSCC_EXT_PAGE_CSR_CNTL_19. + */ + + /* Setup the Target ID */ + phy_base_write(phydev, MSCC_EXT_PAGE_CSR_CNTL_20, + MSCC_PHY_CSR_CNTL_20_TARGET(target >> 2)); + + /* Trigger CSR Action - Read into the CSR's */ + phy_base_write(phydev, MSCC_EXT_PAGE_CSR_CNTL_19, + MSCC_PHY_CSR_CNTL_19_CMD | MSCC_PHY_CSR_CNTL_19_READ | + MSCC_PHY_CSR_CNTL_19_REG_ADDR(reg) | + MSCC_PHY_CSR_CNTL_19_TARGET(target & 0x3)); + + /* Wait for register access*/ + deadline = jiffies + msecs_to_jiffies(PROC_CMD_NCOMPLETED_TIMEOUT_MS); + do { + usleep_range(500, 1000); + val = phy_base_read(phydev, MSCC_EXT_PAGE_CSR_CNTL_19); + } while (time_before(jiffies, deadline) && + !(val & MSCC_PHY_CSR_CNTL_19_CMD)); + + if (!(val & MSCC_PHY_CSR_CNTL_19_CMD)) + return 0xffffffff; + + /* Read the Least Significant Word (LSW) (17) */ + val_l = phy_base_read(phydev, MSCC_EXT_PAGE_CSR_CNTL_17); + + /* Read the Most Significant Word (MSW) (18) */ + val_h = phy_base_read(phydev, MSCC_EXT_PAGE_CSR_CNTL_18); + + phy_base_write(phydev, MSCC_EXT_PAGE_ACCESS, + MSCC_PHY_PAGE_STANDARD); + + return (val_h << 16) | val_l; +} + +static int vsc85xx_csr_ctrl_phy_write(struct phy_device *phydev, + u32 target, u32 reg, u32 val) +{ + unsigned long deadline; + + phy_base_write(phydev, MSCC_EXT_PAGE_ACCESS, MSCC_PHY_PAGE_CSR_CNTL); + + /* CSR registers are grouped under different Target IDs. + * 6-bit Target_ID is split between MSCC_EXT_PAGE_CSR_CNTL_20 and + * MSCC_EXT_PAGE_CSR_CNTL_19 registers. + * Target_ID[5:2] maps to bits[3:0] of MSCC_EXT_PAGE_CSR_CNTL_20 + * and Target_ID[1:0] maps to bits[13:12] of MSCC_EXT_PAGE_CSR_CNTL_19. + */ + + /* Setup the Target ID */ + phy_base_write(phydev, MSCC_EXT_PAGE_CSR_CNTL_20, + MSCC_PHY_CSR_CNTL_20_TARGET(target >> 2)); + + /* Write the Least Significant Word (LSW) (17) */ + phy_base_write(phydev, MSCC_EXT_PAGE_CSR_CNTL_17, (u16)val); + + /* Write the Most Significant Word (MSW) (18) */ + phy_base_write(phydev, MSCC_EXT_PAGE_CSR_CNTL_18, (u16)(val >> 16)); + + /* Trigger CSR Action - Write into the CSR's */ + phy_base_write(phydev, MSCC_EXT_PAGE_CSR_CNTL_19, + MSCC_PHY_CSR_CNTL_19_CMD | + MSCC_PHY_CSR_CNTL_19_REG_ADDR(reg) | + MSCC_PHY_CSR_CNTL_19_TARGET(target & 0x3)); + + /* Wait for register access */ + deadline = jiffies + msecs_to_jiffies(PROC_CMD_NCOMPLETED_TIMEOUT_MS); + do { + usleep_range(500, 1000); + val = phy_base_read(phydev, MSCC_EXT_PAGE_CSR_CNTL_19); + } while (time_before(jiffies, deadline) && + !(val & MSCC_PHY_CSR_CNTL_19_CMD)); + + if (!(val & MSCC_PHY_CSR_CNTL_19_CMD)) + return -ETIMEDOUT; + + phy_base_write(phydev, MSCC_EXT_PAGE_ACCESS, + MSCC_PHY_PAGE_STANDARD); + + return 0; +} + +static int __phy_write_mcb_s6g(struct phy_device *phydev, u32 reg, u8 mcb, + u32 op) +{ + unsigned long deadline; + u32 val; + int ret; + + ret = vsc85xx_csr_ctrl_phy_write(phydev, PHY_MCB_TARGET, reg, + op | (1 << mcb)); + if (ret) + return -EINVAL; + + deadline = jiffies + msecs_to_jiffies(PROC_CMD_NCOMPLETED_TIMEOUT_MS); + do { + usleep_range(500, 1000); + val = vsc85xx_csr_ctrl_phy_read(phydev, PHY_MCB_TARGET, reg); + + if (val == 0xffffffff) + return -EIO; + + } while (time_before(jiffies, deadline) && (val & op)); + + if (val & op) + return -ETIMEDOUT; + + return 0; +} + +/* Trigger a read to the spcified MCB */ +static int phy_update_mcb_s6g(struct phy_device *phydev, u32 reg, u8 mcb) +{ + return __phy_write_mcb_s6g(phydev, reg, mcb, PHY_MCB_S6G_READ); +} + +/* Trigger a write to the spcified MCB */ +static int phy_commit_mcb_s6g(struct phy_device *phydev, u32 reg, u8 mcb) +{ + return __phy_write_mcb_s6g(phydev, reg, mcb, PHY_MCB_S6G_WRITE); +} + +static int vsc8514_config_init(struct phy_device *phydev) +{ + struct vsc8531_private *vsc8531 = phydev->priv; + unsigned long deadline; + u16 val, addr; + int ret, i; + u32 reg; + + phydev->mdix_ctrl = ETH_TP_MDI_AUTO; + + mutex_lock(&phydev->mdio.bus->mdio_lock); + + __phy_write(phydev, MSCC_EXT_PAGE_ACCESS, MSCC_PHY_PAGE_EXTENDED); + + addr = __phy_read(phydev, MSCC_PHY_EXT_PHY_CNTL_4); + addr >>= PHY_CNTL_4_ADDR_POS; + + val = __phy_read(phydev, MSCC_PHY_ACTIPHY_CNTL); + + if (val & PHY_ADDR_REVERSED) + vsc8531->base_addr = phydev->mdio.addr + addr; + else + vsc8531->base_addr = phydev->mdio.addr - addr; + + /* Some parts of the init sequence are identical for every PHY in the + * package. Some parts are modifying the GPIO register bank which is a + * set of registers that are affecting all PHYs, a few resetting the + * microprocessor common to all PHYs. + * All PHYs' interrupts mask register has to be zeroed before enabling + * any PHY's interrupt in this register. + * For all these reasons, we need to do the init sequence once and only + * once whatever is the first PHY in the package that is initialized and + * do the correct init sequence for all PHYs that are package-critical + * in this pre-init function. + */ + if (!vsc8584_is_pkg_init(phydev, val & PHY_ADDR_REVERSED ? 1 : 0)) + vsc8514_config_pre_init(phydev); + + vsc8531->pkg_init = true; + + phy_base_write(phydev, MSCC_EXT_PAGE_ACCESS, + MSCC_PHY_PAGE_EXTENDED_GPIO); + + val = phy_base_read(phydev, MSCC_PHY_MAC_CFG_FASTLINK); + + val &= ~MAC_CFG_MASK; + val |= MAC_CFG_QSGMII; + ret = phy_base_write(phydev, MSCC_PHY_MAC_CFG_FASTLINK, val); + + if (ret) + goto err; + + ret = vsc8584_cmd(phydev, + PROC_CMD_MCB_ACCESS_MAC_CONF | + PROC_CMD_RST_CONF_PORT | + PROC_CMD_READ_MOD_WRITE_PORT | PROC_CMD_QSGMII_MAC); + if (ret) + goto err; + + /* 6g mcb */ + phy_update_mcb_s6g(phydev, PHY_MCB_S6G_CFG, 0); + /* lcpll mcb */ + phy_update_mcb_s6g(phydev, PHY_S6G_LCPLL_CFG, 0); + /* pll5gcfg0 */ + ret = vsc85xx_csr_ctrl_phy_write(phydev, PHY_MCB_TARGET, + PHY_S6G_PLL5G_CFG0, 0x7036f145); + if (ret) + goto err; + + phy_commit_mcb_s6g(phydev, PHY_S6G_LCPLL_CFG, 0); + /* pllcfg */ + ret = vsc85xx_csr_ctrl_phy_write(phydev, PHY_MCB_TARGET, + PHY_S6G_PLL_CFG, + (3 << PHY_S6G_PLL_ENA_OFFS_POS) | + (120 << PHY_S6G_PLL_FSM_CTRL_DATA_POS) + | (0 << PHY_S6G_PLL_FSM_ENA_POS)); + if (ret) + goto err; + + /* commoncfg */ + ret = vsc85xx_csr_ctrl_phy_write(phydev, PHY_MCB_TARGET, + PHY_S6G_COMMON_CFG, + (0 << PHY_S6G_SYS_RST_POS) | + (0 << PHY_S6G_ENA_LANE_POS) | + (0 << PHY_S6G_ENA_LOOP_POS) | + (0 << PHY_S6G_QRATE_POS) | + (3 << PHY_S6G_IF_MODE_POS)); + if (ret) + goto err; + + /* misccfg */ + ret = vsc85xx_csr_ctrl_phy_write(phydev, PHY_MCB_TARGET, + PHY_S6G_MISC_CFG, 1); + if (ret) + goto err; + + /* gpcfg */ + ret = vsc85xx_csr_ctrl_phy_write(phydev, PHY_MCB_TARGET, + PHY_S6G_GPC_CFG, 768); + if (ret) + goto err; + + phy_commit_mcb_s6g(phydev, PHY_S6G_DFT_CFG2, 0); + + deadline = jiffies + msecs_to_jiffies(PROC_CMD_NCOMPLETED_TIMEOUT_MS); + do { + usleep_range(500, 1000); + phy_update_mcb_s6g(phydev, PHY_MCB_S6G_CFG, + 0); /* read 6G MCB into CSRs */ + reg = vsc85xx_csr_ctrl_phy_read(phydev, PHY_MCB_TARGET, + PHY_S6G_PLL_STATUS); + if (reg == 0xffffffff) { + mutex_unlock(&phydev->mdio.bus->mdio_lock); + return -EIO; + } + + } while (time_before(jiffies, deadline) && (reg & BIT(12))); + + if (reg & BIT(12)) { + mutex_unlock(&phydev->mdio.bus->mdio_lock); + return -ETIMEDOUT; + } + + /* misccfg */ + ret = vsc85xx_csr_ctrl_phy_write(phydev, PHY_MCB_TARGET, + PHY_S6G_MISC_CFG, 0); + if (ret) + goto err; + + phy_commit_mcb_s6g(phydev, PHY_MCB_S6G_CFG, 0); + + deadline = jiffies + msecs_to_jiffies(PROC_CMD_NCOMPLETED_TIMEOUT_MS); + do { + usleep_range(500, 1000); + phy_update_mcb_s6g(phydev, PHY_MCB_S6G_CFG, + 0); /* read 6G MCB into CSRs */ + reg = vsc85xx_csr_ctrl_phy_read(phydev, PHY_MCB_TARGET, + PHY_S6G_IB_STATUS0); + if (reg == 0xffffffff) { + mutex_unlock(&phydev->mdio.bus->mdio_lock); + return -EIO; + } + + } while (time_before(jiffies, deadline) && !(reg & BIT(8))); + + if (!(reg & BIT(8))) { + mutex_unlock(&phydev->mdio.bus->mdio_lock); + return -ETIMEDOUT; + } + + mutex_unlock(&phydev->mdio.bus->mdio_lock); + + ret = phy_write(phydev, MSCC_EXT_PAGE_ACCESS, MSCC_PHY_PAGE_STANDARD); + + if (ret) + return ret; + + ret = phy_modify(phydev, MSCC_PHY_EXT_PHY_CNTL_1, MEDIA_OP_MODE_MASK, + MEDIA_OP_MODE_COPPER); + + if (ret) + return ret; + + ret = genphy_soft_reset(phydev); + + if (ret) + return ret; + + for (i = 0; i < vsc8531->nleds; i++) { + ret = vsc85xx_led_cntl_set(phydev, i, vsc8531->leds_mode[i]); + if (ret) + return ret; + } + + return ret; + +err: + mutex_unlock(&phydev->mdio.bus->mdio_lock); + return ret; +} + static int vsc85xx_ack_interrupt(struct phy_device *phydev) { int rc = 0; @@ -1791,6 +2209,31 @@ static int vsc85xx_read_status(struct phy_device *phydev) return genphy_read_status(phydev); } +static int vsc8514_probe(struct phy_device *phydev) +{ + struct vsc8531_private *vsc8531; + u32 default_mode[4] = {VSC8531_LINK_1000_ACTIVITY, + VSC8531_LINK_100_ACTIVITY, VSC8531_LINK_ACTIVITY, + VSC8531_DUPLEX_COLLISION}; + + vsc8531 = devm_kzalloc(&phydev->mdio.dev, sizeof(*vsc8531), GFP_KERNEL); + if (!vsc8531) + return -ENOMEM; + + phydev->priv = vsc8531; + + vsc8531->nleds = 4; + vsc8531->supp_led_modes = VSC85XX_SUPP_LED_MODES; + vsc8531->hw_stats = vsc85xx_hw_stats; + vsc8531->nstats = ARRAY_SIZE(vsc85xx_hw_stats); + vsc8531->stats = devm_kmalloc_array(&phydev->mdio.dev, vsc8531->nstats, + sizeof(u64), GFP_KERNEL); + if (!vsc8531->stats) + return -ENOMEM; + + return vsc85xx_dt_led_modes_get(phydev, default_mode); +} + static int vsc8574_probe(struct phy_device *phydev) { struct vsc8531_private *vsc8531; @@ -1878,11 +2321,34 @@ static int vsc85xx_probe(struct phy_device *phydev) /* Microsemi VSC85xx PHYs */ static struct phy_driver vsc85xx_driver[] = { +{ + .phy_id = PHY_ID_VSC8514, + .name = "Microsemi GE VSC8514 SyncE", + .phy_id_mask = 0xfffffff0, + .soft_reset = &genphy_soft_reset, + .config_init = &vsc8514_config_init, + .config_aneg = &vsc85xx_config_aneg, + .read_status = &vsc85xx_read_status, + .ack_interrupt = &vsc85xx_ack_interrupt, + .config_intr = &vsc85xx_config_intr, + .suspend = &genphy_suspend, + .resume = &genphy_resume, + .probe = &vsc8514_probe, + .set_wol = &vsc85xx_wol_set, + .get_wol = &vsc85xx_wol_get, + .get_tunable = &vsc85xx_get_tunable, + .set_tunable = &vsc85xx_set_tunable, + .read_page = &vsc85xx_phy_read_page, + .write_page = &vsc85xx_phy_write_page, + .get_sset_count = &vsc85xx_get_sset_count, + .get_strings = &vsc85xx_get_strings, + .get_stats = &vsc85xx_get_stats, +}, { .phy_id = PHY_ID_VSC8530, .name = "Microsemi FE VSC8530", .phy_id_mask = 0xfffffff0, - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .soft_reset = &genphy_soft_reset, .config_init = &vsc85xx_config_init, .config_aneg = &vsc85xx_config_aneg, @@ -1907,7 +2373,7 @@ static struct phy_driver vsc85xx_driver[] = { .phy_id = PHY_ID_VSC8531, .name = "Microsemi VSC8531", .phy_id_mask = 0xfffffff0, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .soft_reset = &genphy_soft_reset, .config_init = &vsc85xx_config_init, .config_aneg = &vsc85xx_config_aneg, @@ -1932,7 +2398,7 @@ static struct phy_driver vsc85xx_driver[] = { .phy_id = PHY_ID_VSC8540, .name = "Microsemi FE VSC8540 SyncE", .phy_id_mask = 0xfffffff0, - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .soft_reset = &genphy_soft_reset, .config_init = &vsc85xx_config_init, .config_aneg = &vsc85xx_config_aneg, @@ -1957,7 +2423,7 @@ static struct phy_driver vsc85xx_driver[] = { .phy_id = PHY_ID_VSC8541, .name = "Microsemi VSC8541 SyncE", .phy_id_mask = 0xfffffff0, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .soft_reset = &genphy_soft_reset, .config_init = &vsc85xx_config_init, .config_aneg = &vsc85xx_config_aneg, @@ -1982,7 +2448,7 @@ static struct phy_driver vsc85xx_driver[] = { .phy_id = PHY_ID_VSC8574, .name = "Microsemi GE VSC8574 SyncE", .phy_id_mask = 0xfffffff0, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .soft_reset = &genphy_soft_reset, .config_init = &vsc8584_config_init, .config_aneg = &vsc85xx_config_aneg, @@ -2008,7 +2474,7 @@ static struct phy_driver vsc85xx_driver[] = { .phy_id = PHY_ID_VSC8584, .name = "Microsemi GE VSC8584 SyncE", .phy_id_mask = 0xfffffff0, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .soft_reset = &genphy_soft_reset, .config_init = &vsc8584_config_init, .config_aneg = &vsc85xx_config_aneg, @@ -2034,6 +2500,7 @@ static struct phy_driver vsc85xx_driver[] = { module_phy_driver(vsc85xx_driver); static struct mdio_device_id __maybe_unused vsc85xx_tbl[] = { + { PHY_ID_VSC8514, 0xfffffff0, }, { PHY_ID_VSC8530, 0xfffffff0, }, { PHY_ID_VSC8531, 0xfffffff0, }, { PHY_ID_VSC8540, 0xfffffff0, }, diff --git a/drivers/net/phy/national.c b/drivers/net/phy/national.c index 42282a86b68070ae1d19e8cd371d9cc7ffe57c76..a221dd552c3c649cc2932cebb7490e71629cb62b 100644 --- a/drivers/net/phy/national.c +++ b/drivers/net/phy/national.c @@ -128,7 +128,7 @@ static struct phy_driver dp83865_driver[] = { { .phy_id = DP83865_PHY_ID, .phy_id_mask = 0xfffffff0, .name = "NatSemi DP83865", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = ns_config_init, .ack_interrupt = ns_ack_interrupt, .config_intr = ns_config_intr, diff --git a/drivers/net/phy/phy-c45.c b/drivers/net/phy/phy-c45.c index 9e24d95694242aa88c5940b891da1c01a75a3198..abe13dfe50ad34e136f38a7394f3a1ef4b9edcbf 100644 --- a/drivers/net/phy/phy-c45.c +++ b/drivers/net/phy/phy-c45.c @@ -262,12 +262,30 @@ int genphy_c45_read_lpa(struct phy_device *phydev) { int val; + val = phy_read_mmd(phydev, MDIO_MMD_AN, MDIO_STAT1); + if (val < 0) + return val; + + if (!(val & MDIO_AN_STAT1_COMPLETE)) { + linkmode_clear_bit(ETHTOOL_LINK_MODE_Autoneg_BIT, + phydev->lp_advertising); + mii_10gbt_stat_mod_linkmode_lpa_t(phydev->lp_advertising, 0); + mii_adv_mod_linkmode_adv_t(phydev->lp_advertising, 0); + phydev->pause = 0; + phydev->asym_pause = 0; + + return 0; + } + + linkmode_mod_bit(ETHTOOL_LINK_MODE_Autoneg_BIT, phydev->lp_advertising, + val & MDIO_AN_STAT1_LPABLE); + /* Read the link partner's base page advertisement */ val = phy_read_mmd(phydev, MDIO_MMD_AN, MDIO_AN_LPA); if (val < 0) return val; - mii_lpa_mod_linkmode_lpa_t(phydev->lp_advertising, val); + mii_adv_mod_linkmode_adv_t(phydev->lp_advertising, val); phydev->pause = val & LPA_PAUSE_CAP ? 1 : 0; phydev->asym_pause = val & LPA_PAUSE_ASYM ? 1 : 0; @@ -498,21 +516,10 @@ int gen10g_config_aneg(struct phy_device *phydev) } EXPORT_SYMBOL_GPL(gen10g_config_aneg); -static int gen10g_read_status(struct phy_device *phydev) -{ - /* For now just lie and say it's 10G all the time */ - phydev->speed = SPEED_10000; - phydev->duplex = DUPLEX_FULL; - - return genphy_c45_read_link(phydev); -} - -struct phy_driver genphy_10g_driver = { +struct phy_driver genphy_c45_driver = { .phy_id = 0xffffffff, .phy_id_mask = 0xffffffff, - .name = "Generic 10G PHY", + .name = "Generic Clause 45 PHY", .soft_reset = genphy_no_soft_reset, - .features = PHY_10GBIT_FEATURES, - .config_aneg = gen10g_config_aneg, - .read_status = gen10g_read_status, + .read_status = genphy_c45_read_status, }; diff --git a/drivers/net/phy/phy-core.c b/drivers/net/phy/phy-core.c index 5016cd5fd7c7595c4e7d0818d13337cf59bbab53..3daf0214a242fef77909852a7b9676d1fe287eb3 100644 --- a/drivers/net/phy/phy-core.c +++ b/drivers/net/phy/phy-core.c @@ -8,6 +8,11 @@ const char *phy_speed_to_str(int speed) { + BUILD_BUG_ON_MSG(__ETHTOOL_LINK_MODE_MASK_NBITS != 67, + "Enum ethtool_link_mode_bit_indices and phylib are out of sync. " + "If a speed or mode has been added please update phy_speed_to_str " + "and the PHY settings array.\n"); + switch (speed) { case SPEED_10: return "10Mbps"; @@ -35,6 +40,8 @@ const char *phy_speed_to_str(int speed) return "56Gbps"; case SPEED_100000: return "100Gbps"; + case SPEED_200000: + return "200Gbps"; case SPEED_UNKNOWN: return "Unknown"; default: @@ -58,222 +65,81 @@ EXPORT_SYMBOL_GPL(phy_duplex_to_str); /* A mapping of all SUPPORTED settings to speed/duplex. This table * must be grouped by speed and sorted in descending match priority * - iow, descending speed. */ + +#define PHY_SETTING(s, d, b) { .speed = SPEED_ ## s, .duplex = DUPLEX_ ## d, \ + .bit = ETHTOOL_LINK_MODE_ ## b ## _BIT} + static const struct phy_setting settings[] = { + /* 200G */ + PHY_SETTING( 200000, FULL, 200000baseCR4_Full ), + PHY_SETTING( 200000, FULL, 200000baseKR4_Full ), + PHY_SETTING( 200000, FULL, 200000baseLR4_ER4_FR4_Full ), + PHY_SETTING( 200000, FULL, 200000baseDR4_Full ), + PHY_SETTING( 200000, FULL, 200000baseSR4_Full ), /* 100G */ - { - .speed = SPEED_100000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_100000baseCR4_Full_BIT, - }, - { - .speed = SPEED_100000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_100000baseKR4_Full_BIT, - }, - { - .speed = SPEED_100000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_100000baseLR4_ER4_Full_BIT, - }, - { - .speed = SPEED_100000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_100000baseSR4_Full_BIT, - }, + PHY_SETTING( 100000, FULL, 100000baseCR4_Full ), + PHY_SETTING( 100000, FULL, 100000baseKR4_Full ), + PHY_SETTING( 100000, FULL, 100000baseLR4_ER4_Full ), + PHY_SETTING( 100000, FULL, 100000baseSR4_Full ), + PHY_SETTING( 100000, FULL, 100000baseCR2_Full ), + PHY_SETTING( 100000, FULL, 100000baseKR2_Full ), + PHY_SETTING( 100000, FULL, 100000baseLR2_ER2_FR2_Full ), + PHY_SETTING( 100000, FULL, 100000baseDR2_Full ), + PHY_SETTING( 100000, FULL, 100000baseSR2_Full ), /* 56G */ - { - .speed = SPEED_56000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_56000baseCR4_Full_BIT, - }, - { - .speed = SPEED_56000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_56000baseKR4_Full_BIT, - }, - { - .speed = SPEED_56000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_56000baseLR4_Full_BIT, - }, - { - .speed = SPEED_56000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_56000baseSR4_Full_BIT, - }, + PHY_SETTING( 56000, FULL, 56000baseCR4_Full ), + PHY_SETTING( 56000, FULL, 56000baseKR4_Full ), + PHY_SETTING( 56000, FULL, 56000baseLR4_Full ), + PHY_SETTING( 56000, FULL, 56000baseSR4_Full ), /* 50G */ - { - .speed = SPEED_50000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_50000baseCR2_Full_BIT, - }, - { - .speed = SPEED_50000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_50000baseKR2_Full_BIT, - }, - { - .speed = SPEED_50000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_50000baseSR2_Full_BIT, - }, + PHY_SETTING( 50000, FULL, 50000baseCR2_Full ), + PHY_SETTING( 50000, FULL, 50000baseKR2_Full ), + PHY_SETTING( 50000, FULL, 50000baseSR2_Full ), + PHY_SETTING( 50000, FULL, 50000baseCR_Full ), + PHY_SETTING( 50000, FULL, 50000baseKR_Full ), + PHY_SETTING( 50000, FULL, 50000baseLR_ER_FR_Full ), + PHY_SETTING( 50000, FULL, 50000baseDR_Full ), + PHY_SETTING( 50000, FULL, 50000baseSR_Full ), /* 40G */ - { - .speed = SPEED_40000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_40000baseCR4_Full_BIT, - }, - { - .speed = SPEED_40000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_40000baseKR4_Full_BIT, - }, - { - .speed = SPEED_40000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_40000baseLR4_Full_BIT, - }, - { - .speed = SPEED_40000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_40000baseSR4_Full_BIT, - }, + PHY_SETTING( 40000, FULL, 40000baseCR4_Full ), + PHY_SETTING( 40000, FULL, 40000baseKR4_Full ), + PHY_SETTING( 40000, FULL, 40000baseLR4_Full ), + PHY_SETTING( 40000, FULL, 40000baseSR4_Full ), /* 25G */ - { - .speed = SPEED_25000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_25000baseCR_Full_BIT, - }, - { - .speed = SPEED_25000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_25000baseKR_Full_BIT, - }, - { - .speed = SPEED_25000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_25000baseSR_Full_BIT, - }, - + PHY_SETTING( 25000, FULL, 25000baseCR_Full ), + PHY_SETTING( 25000, FULL, 25000baseKR_Full ), + PHY_SETTING( 25000, FULL, 25000baseSR_Full ), /* 20G */ - { - .speed = SPEED_20000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_20000baseKR2_Full_BIT, - }, - { - .speed = SPEED_20000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_20000baseMLD2_Full_BIT, - }, + PHY_SETTING( 20000, FULL, 20000baseKR2_Full ), + PHY_SETTING( 20000, FULL, 20000baseMLD2_Full ), /* 10G */ - { - .speed = SPEED_10000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_10000baseCR_Full_BIT, - }, - { - .speed = SPEED_10000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_10000baseER_Full_BIT, - }, - { - .speed = SPEED_10000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_10000baseKR_Full_BIT, - }, - { - .speed = SPEED_10000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_10000baseKX4_Full_BIT, - }, - { - .speed = SPEED_10000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_10000baseLR_Full_BIT, - }, - { - .speed = SPEED_10000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_10000baseLRM_Full_BIT, - }, - { - .speed = SPEED_10000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_10000baseR_FEC_BIT, - }, - { - .speed = SPEED_10000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_10000baseSR_Full_BIT, - }, - { - .speed = SPEED_10000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_10000baseT_Full_BIT, - }, + PHY_SETTING( 10000, FULL, 10000baseCR_Full ), + PHY_SETTING( 10000, FULL, 10000baseER_Full ), + PHY_SETTING( 10000, FULL, 10000baseKR_Full ), + PHY_SETTING( 10000, FULL, 10000baseKX4_Full ), + PHY_SETTING( 10000, FULL, 10000baseLR_Full ), + PHY_SETTING( 10000, FULL, 10000baseLRM_Full ), + PHY_SETTING( 10000, FULL, 10000baseR_FEC ), + PHY_SETTING( 10000, FULL, 10000baseSR_Full ), + PHY_SETTING( 10000, FULL, 10000baseT_Full ), /* 5G */ - { - .speed = SPEED_5000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_5000baseT_Full_BIT, - }, - + PHY_SETTING( 5000, FULL, 5000baseT_Full ), /* 2.5G */ - { - .speed = SPEED_2500, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_2500baseT_Full_BIT, - }, - { - .speed = SPEED_2500, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_2500baseX_Full_BIT, - }, + PHY_SETTING( 2500, FULL, 2500baseT_Full ), + PHY_SETTING( 2500, FULL, 2500baseX_Full ), /* 1G */ - { - .speed = SPEED_1000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_1000baseKX_Full_BIT, - }, - { - .speed = SPEED_1000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_1000baseT_Full_BIT, - }, - { - .speed = SPEED_1000, - .duplex = DUPLEX_HALF, - .bit = ETHTOOL_LINK_MODE_1000baseT_Half_BIT, - }, - { - .speed = SPEED_1000, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_1000baseX_Full_BIT, - }, + PHY_SETTING( 1000, FULL, 1000baseKX_Full ), + PHY_SETTING( 1000, FULL, 1000baseT_Full ), + PHY_SETTING( 1000, HALF, 1000baseT_Half ), + PHY_SETTING( 1000, FULL, 1000baseX_Full ), /* 100M */ - { - .speed = SPEED_100, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_100baseT_Full_BIT, - }, - { - .speed = SPEED_100, - .duplex = DUPLEX_HALF, - .bit = ETHTOOL_LINK_MODE_100baseT_Half_BIT, - }, + PHY_SETTING( 100, FULL, 100baseT_Full ), + PHY_SETTING( 100, HALF, 100baseT_Half ), /* 10M */ - { - .speed = SPEED_10, - .duplex = DUPLEX_FULL, - .bit = ETHTOOL_LINK_MODE_10baseT_Full_BIT, - }, - { - .speed = SPEED_10, - .duplex = DUPLEX_HALF, - .bit = ETHTOOL_LINK_MODE_10baseT_Half_BIT, - }, + PHY_SETTING( 10, FULL, 10baseT_Full ), + PHY_SETTING( 10, HALF, 10baseT_Half ), }; +#undef PHY_SETTING /** * phy_lookup_setting - lookup a PHY setting @@ -362,7 +228,7 @@ int phy_set_max_speed(struct phy_device *phydev, u32 max_speed) if (err) return err; - linkmode_copy(phydev->advertising, phydev->supported); + phy_advertise_supported(phydev); return 0; } diff --git a/drivers/net/phy/phy.c b/drivers/net/phy/phy.c index 3745220c5c98a973ff9fded28549937702d01bb7..e8885429293ad3d3f383682018a03bedfd9ae222 100644 --- a/drivers/net/phy/phy.c +++ b/drivers/net/phy/phy.c @@ -43,7 +43,6 @@ static const char *phy_state_to_str(enum phy_state st) PHY_STATE_STR(NOLINK) PHY_STATE_STR(FORCING) PHY_STATE_STR(HALTED) - PHY_STATE_STR(RESUMING) } return NULL; @@ -61,6 +60,32 @@ static void phy_link_down(struct phy_device *phydev, bool do_carrier) phy_led_trigger_change_speed(phydev); } +static const char *phy_pause_str(struct phy_device *phydev) +{ + bool local_pause, local_asym_pause; + + if (phydev->autoneg == AUTONEG_DISABLE) + goto no_pause; + + local_pause = linkmode_test_bit(ETHTOOL_LINK_MODE_Pause_BIT, + phydev->advertising); + local_asym_pause = linkmode_test_bit(ETHTOOL_LINK_MODE_Asym_Pause_BIT, + phydev->advertising); + + if (local_pause && phydev->pause) + return "rx/tx"; + + if (local_asym_pause && phydev->asym_pause) { + if (local_pause) + return "rx"; + if (phydev->pause) + return "tx"; + } + +no_pause: + return "off"; +} + /** * phy_print_status - Convenience function to print out the current phy status * @phydev: the phy_device struct @@ -72,7 +97,7 @@ void phy_print_status(struct phy_device *phydev) "Link is Up - %s/%s - flow control %s\n", phy_speed_to_str(phydev->speed), phy_duplex_to_str(phydev->duplex), - phydev->pause ? "rx/tx" : "off"); + phy_pause_str(phydev)); } else { netdev_info(phydev->attached_dev, "Link is Down\n"); } @@ -214,10 +239,6 @@ static void phy_sanitize_settings(struct phy_device *phydev) { const struct phy_setting *setting; - /* Sanitize settings based on PHY capabilities */ - if (linkmode_test_bit(ETHTOOL_LINK_MODE_Autoneg_BIT, phydev->supported)) - phydev->autoneg = AUTONEG_DISABLE; - setting = phy_find_valid(phydev->speed, phydev->duplex, phydev->supported); if (setting) { @@ -863,10 +884,7 @@ void phy_start(struct phy_device *phydev) goto out; } - if (phydev->state == PHY_READY) - phydev->state = PHY_UP; - else - phydev->state = PHY_RESUMING; + phydev->state = PHY_UP; phy_start_machine(phydev); out: @@ -891,9 +909,6 @@ void phy_state_machine(struct work_struct *work) old_state = phydev->state; - if (phydev->drv && phydev->drv->link_change_notify) - phydev->drv->link_change_notify(phydev); - switch (phydev->state) { case PHY_DOWN: case PHY_READY: @@ -904,7 +919,6 @@ void phy_state_machine(struct work_struct *work) break; case PHY_NOLINK: case PHY_RUNNING: - case PHY_RESUMING: err = phy_check_link_status(phydev); break; case PHY_FORCING: @@ -940,10 +954,13 @@ void phy_state_machine(struct work_struct *work) if (err < 0) phy_error(phydev); - if (old_state != phydev->state) + if (old_state != phydev->state) { phydev_dbg(phydev, "PHY state change %s -> %s\n", phy_state_to_str(old_state), phy_state_to_str(phydev->state)); + if (phydev->drv && phydev->drv->link_change_notify) + phydev->drv->link_change_notify(phydev); + } /* Only re-schedule a PHY state machine change if we are polling the * PHY, if PHY_IGNORE_INTERRUPT is set, then we will be moving diff --git a/drivers/net/phy/phy_device.c b/drivers/net/phy/phy_device.c index 77068c545de0d33607981e7a94a32bf7ed1ff34c..dcc93a8731744e2e4b3e54e71291893e525e0c1f 100644 --- a/drivers/net/phy/phy_device.c +++ b/drivers/net/phy/phy_device.c @@ -225,7 +225,7 @@ static void phy_mdio_device_remove(struct mdio_device *mdiodev) } static struct phy_driver genphy_driver; -extern struct phy_driver genphy_10g_driver; +extern struct phy_driver genphy_c45_driver; static LIST_HEAD(phy_fixup_list); static DEFINE_MUTEX(phy_fixup_lock); @@ -1174,7 +1174,7 @@ int phy_attach_direct(struct net_device *dev, struct phy_device *phydev, */ if (!d->driver) { if (phydev->is_c45) - d->driver = &genphy_10g_driver.mdiodrv.driver; + d->driver = &genphy_c45_driver.mdiodrv.driver; else d->driver = &genphy_driver.mdiodrv.driver; @@ -1335,7 +1335,7 @@ EXPORT_SYMBOL_GPL(phy_driver_is_genphy); bool phy_driver_is_genphy_10g(struct phy_device *phydev) { return phy_driver_is_genphy_kind(phydev, - &genphy_10g_driver.mdiodrv.driver); + &genphy_c45_driver.mdiodrv.driver); } EXPORT_SYMBOL_GPL(phy_driver_is_genphy_10g); @@ -1710,23 +1710,19 @@ int genphy_update_link(struct phy_device *phydev) */ if (!phy_polling_mode(phydev)) { status = phy_read(phydev, MII_BMSR); - if (status < 0) { + if (status < 0) return status; - } else if (status & BMSR_LSTATUS) { - phydev->link = 1; - return 0; - } + else if (status & BMSR_LSTATUS) + goto done; } /* Read link and autonegotiation status */ status = phy_read(phydev, MII_BMSR); if (status < 0) return status; - - if ((status & BMSR_LSTATUS) == 0) - phydev->link = 0; - else - phydev->link = 1; +done: + phydev->link = status & BMSR_LSTATUS ? 1 : 0; + phydev->autoneg_complete = status & BMSR_ANEGCOMPLETE ? 1 : 0; return 0; } @@ -1743,23 +1739,26 @@ EXPORT_SYMBOL(genphy_update_link); */ int genphy_read_status(struct phy_device *phydev) { - int adv; - int err; - int lpa; - int lpagb = 0; + int adv, lpa, lpagb, err, old_link = phydev->link; /* Update the link, but return if there was an error */ err = genphy_update_link(phydev); if (err) return err; + /* why bother the PHY if nothing can have changed */ + if (phydev->autoneg == AUTONEG_ENABLE && old_link && phydev->link) + return 0; + + phydev->speed = SPEED_UNKNOWN; + phydev->duplex = DUPLEX_UNKNOWN; + phydev->pause = 0; + phydev->asym_pause = 0; + linkmode_zero(phydev->lp_advertising); - if (AUTONEG_ENABLE == phydev->autoneg) { - if (linkmode_test_bit(ETHTOOL_LINK_MODE_1000baseT_Half_BIT, - phydev->supported) || - linkmode_test_bit(ETHTOOL_LINK_MODE_1000baseT_Full_BIT, - phydev->supported)) { + if (phydev->autoneg == AUTONEG_ENABLE && phydev->autoneg_complete) { + if (phydev->is_gigabit_capable) { lpagb = phy_read(phydev, MII_STAT1000); if (lpagb < 0) return lpagb; @@ -1785,14 +1784,8 @@ int genphy_read_status(struct phy_device *phydev) return lpa; mii_lpa_mod_linkmode_lpa_t(phydev->lp_advertising, lpa); - - phydev->speed = SPEED_UNKNOWN; - phydev->duplex = DUPLEX_UNKNOWN; - phydev->pause = 0; - phydev->asym_pause = 0; - phy_resolve_aneg_linkmode(phydev); - } else { + } else if (phydev->autoneg == AUTONEG_DISABLE) { int bmcr = phy_read(phydev, MII_BMCR); if (bmcr < 0) @@ -1809,9 +1802,6 @@ int genphy_read_status(struct phy_device *phydev) phydev->speed = SPEED_100; else phydev->speed = SPEED_10; - - phydev->pause = 0; - phydev->asym_pause = 0; } return 0; @@ -1829,13 +1819,25 @@ EXPORT_SYMBOL(genphy_read_status); */ int genphy_soft_reset(struct phy_device *phydev) { + u16 res = BMCR_RESET; int ret; - ret = phy_set_bits(phydev, MII_BMCR, BMCR_RESET); + if (phydev->autoneg == AUTONEG_ENABLE) + res |= BMCR_ANRESTART; + + ret = phy_modify(phydev, MII_BMCR, BMCR_ISOLATE, res); if (ret < 0) return ret; - return phy_poll_reset(phydev); + ret = phy_poll_reset(phydev); + if (ret) + return ret; + + /* BMCR may be reset to defaults */ + if (phydev->autoneg == AUTONEG_DISABLE) + ret = genphy_setup_forced(phydev); + + return ret; } EXPORT_SYMBOL(genphy_soft_reset); @@ -1887,6 +1889,54 @@ int genphy_config_init(struct phy_device *phydev) } EXPORT_SYMBOL(genphy_config_init); +/** + * genphy_read_abilities - read PHY abilities from Clause 22 registers + * @phydev: target phy_device struct + * + * Description: Reads the PHY's abilities and populates + * phydev->supported accordingly. + * + * Returns: 0 on success, < 0 on failure + */ +int genphy_read_abilities(struct phy_device *phydev) +{ + int val; + + linkmode_set_bit_array(phy_basic_ports_array, + ARRAY_SIZE(phy_basic_ports_array), + phydev->supported); + + val = phy_read(phydev, MII_BMSR); + if (val < 0) + return val; + + linkmode_mod_bit(ETHTOOL_LINK_MODE_Autoneg_BIT, phydev->supported, + val & BMSR_ANEGCAPABLE); + + linkmode_mod_bit(ETHTOOL_LINK_MODE_100baseT_Full_BIT, phydev->supported, + val & BMSR_100FULL); + linkmode_mod_bit(ETHTOOL_LINK_MODE_100baseT_Half_BIT, phydev->supported, + val & BMSR_100HALF); + linkmode_mod_bit(ETHTOOL_LINK_MODE_10baseT_Full_BIT, phydev->supported, + val & BMSR_10FULL); + linkmode_mod_bit(ETHTOOL_LINK_MODE_10baseT_Half_BIT, phydev->supported, + val & BMSR_10HALF); + + if (val & BMSR_ESTATEN) { + val = phy_read(phydev, MII_ESTATUS); + if (val < 0) + return val; + + linkmode_mod_bit(ETHTOOL_LINK_MODE_1000baseT_Full_BIT, + phydev->supported, val & ESTATUS_1000_TFULL); + linkmode_mod_bit(ETHTOOL_LINK_MODE_1000baseT_Half_BIT, + phydev->supported, val & ESTATUS_1000_THALF); + } + + return 0; +} +EXPORT_SYMBOL(genphy_read_abilities); + /* This is used for the phy device which doesn't support the MMD extended * register access, but it does have side effect when we are trying to access * the MMD register via indirect method. @@ -1935,10 +1985,35 @@ EXPORT_SYMBOL(genphy_loopback); void phy_remove_link_mode(struct phy_device *phydev, u32 link_mode) { linkmode_clear_bit(link_mode, phydev->supported); - linkmode_copy(phydev->advertising, phydev->supported); + phy_advertise_supported(phydev); } EXPORT_SYMBOL(phy_remove_link_mode); +static void phy_copy_pause_bits(unsigned long *dst, unsigned long *src) +{ + linkmode_mod_bit(ETHTOOL_LINK_MODE_Asym_Pause_BIT, dst, + linkmode_test_bit(ETHTOOL_LINK_MODE_Asym_Pause_BIT, src)); + linkmode_mod_bit(ETHTOOL_LINK_MODE_Pause_BIT, dst, + linkmode_test_bit(ETHTOOL_LINK_MODE_Pause_BIT, src)); +} + +/** + * phy_advertise_supported - Advertise all supported modes + * @phydev: target phy_device struct + * + * Description: Called to advertise all supported modes, doesn't touch + * pause mode advertising. + */ +void phy_advertise_supported(struct phy_device *phydev) +{ + __ETHTOOL_DECLARE_LINK_MODE_MASK(new); + + linkmode_copy(new, phydev->supported); + phy_copy_pause_bits(new, phydev->advertising); + linkmode_copy(phydev->advertising, new); +} +EXPORT_SYMBOL(phy_advertise_supported); + /** * phy_support_sym_pause - Enable support of symmetrical pause * @phydev: target phy_device struct @@ -1949,8 +2024,7 @@ EXPORT_SYMBOL(phy_remove_link_mode); void phy_support_sym_pause(struct phy_device *phydev) { linkmode_clear_bit(ETHTOOL_LINK_MODE_Asym_Pause_BIT, phydev->supported); - linkmode_set_bit(ETHTOOL_LINK_MODE_Pause_BIT, phydev->supported); - linkmode_copy(phydev->advertising, phydev->supported); + phy_copy_pause_bits(phydev->advertising, phydev->supported); } EXPORT_SYMBOL(phy_support_sym_pause); @@ -1962,9 +2036,7 @@ EXPORT_SYMBOL(phy_support_sym_pause); */ void phy_support_asym_pause(struct phy_device *phydev) { - linkmode_set_bit(ETHTOOL_LINK_MODE_Pause_BIT, phydev->supported); - linkmode_set_bit(ETHTOOL_LINK_MODE_Asym_Pause_BIT, phydev->supported); - linkmode_copy(phydev->advertising, phydev->supported); + phy_copy_pause_bits(phydev->advertising, phydev->supported); } EXPORT_SYMBOL(phy_support_asym_pause); @@ -2044,11 +2116,14 @@ bool phy_validate_pause(struct phy_device *phydev, struct ethtool_pauseparam *pp) { if (!linkmode_test_bit(ETHTOOL_LINK_MODE_Pause_BIT, - phydev->supported) || - (!linkmode_test_bit(ETHTOOL_LINK_MODE_Asym_Pause_BIT, - phydev->supported) && - pp->rx_pause != pp->tx_pause)) + phydev->supported) && pp->rx_pause) + return false; + + if (!linkmode_test_bit(ETHTOOL_LINK_MODE_Asym_Pause_BIT, + phydev->supported) && + pp->rx_pause != pp->tx_pause) return false; + return true; } EXPORT_SYMBOL(phy_validate_pause); @@ -2104,14 +2179,30 @@ static int phy_probe(struct device *dev) */ if (phydrv->features) { linkmode_copy(phydev->supported, phydrv->features); - } else { + } else if (phydrv->get_features) { err = phydrv->get_features(phydev); - if (err) - goto out; + } else if (phydev->is_c45) { + err = genphy_c45_pma_read_abilities(phydev); + } else { + err = genphy_read_abilities(phydev); } + if (err) + goto out; + + if (!linkmode_test_bit(ETHTOOL_LINK_MODE_Autoneg_BIT, + phydev->supported)) + phydev->autoneg = 0; + + if (linkmode_test_bit(ETHTOOL_LINK_MODE_1000baseT_Half_BIT, + phydev->supported)) + phydev->is_gigabit_capable = 1; + if (linkmode_test_bit(ETHTOOL_LINK_MODE_1000baseT_Full_BIT, + phydev->supported)) + phydev->is_gigabit_capable = 1; + of_set_phy_supported(phydev); - linkmode_copy(phydev->advertising, phydev->supported); + phy_advertise_supported(phydev); /* Get the EEE modes we want to prohibit. We will ask * the PHY stop advertising these mode later on @@ -2177,11 +2268,11 @@ int phy_driver_register(struct phy_driver *new_driver, struct module *owner) int retval; /* Either the features are hard coded, or dynamically - * determine. It cannot be both or neither + * determined. It cannot be both. */ - if (WARN_ON((!new_driver->features && !new_driver->get_features) || - (new_driver->features && new_driver->get_features))) { - pr_err("%s: Driver features are missing\n", new_driver->name); + if (WARN_ON(new_driver->features && new_driver->get_features)) { + pr_err("%s: features and get_features must not both be set\n", + new_driver->name); return -EINVAL; } @@ -2243,8 +2334,7 @@ static struct phy_driver genphy_driver = { .phy_id_mask = 0xffffffff, .name = "Generic PHY", .soft_reset = genphy_no_soft_reset, - .config_init = genphy_config_init, - .features = PHY_GBIT_ALL_PORTS_FEATURES, + .get_features = genphy_read_abilities, .aneg_done = genphy_aneg_done, .suspend = genphy_suspend, .resume = genphy_resume, @@ -2261,14 +2351,14 @@ static int __init phy_init(void) features_init(); - rc = phy_driver_register(&genphy_10g_driver, THIS_MODULE); + rc = phy_driver_register(&genphy_c45_driver, THIS_MODULE); if (rc) - goto err_10g; + goto err_c45; rc = phy_driver_register(&genphy_driver, THIS_MODULE); if (rc) { - phy_driver_unregister(&genphy_10g_driver); -err_10g: + phy_driver_unregister(&genphy_c45_driver); +err_c45: mdio_bus_exit(); } @@ -2277,7 +2367,7 @@ err_10g: static void __exit phy_exit(void) { - phy_driver_unregister(&genphy_10g_driver); + phy_driver_unregister(&genphy_c45_driver); phy_driver_unregister(&genphy_driver); mdio_bus_exit(); } diff --git a/drivers/net/phy/qsemi.c b/drivers/net/phy/qsemi.c index 5486f6fb2ab2dc7a1428163a74615d650e93ab26..1b15a991ee061dd9cdbb97ed9c27643f0105cdef 100644 --- a/drivers/net/phy/qsemi.c +++ b/drivers/net/phy/qsemi.c @@ -110,7 +110,7 @@ static struct phy_driver qs6612_driver[] = { { .phy_id = 0x00181440, .name = "QS6612", .phy_id_mask = 0xfffffff0, - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = qs6612_config_init, .ack_interrupt = qs6612_ack_interrupt, .config_intr = qs6612_config_intr, diff --git a/drivers/net/phy/realtek.c b/drivers/net/phy/realtek.c index 10df52ccddfeed9970f8d688f3cebb2300f79722..d6a10f323117ba9e92e68727aaf2da77eb90e9da 100644 --- a/drivers/net/phy/realtek.c +++ b/drivers/net/phy/realtek.c @@ -151,21 +151,14 @@ static int rtl8211_config_aneg(struct phy_device *phydev) static int rtl8211c_config_init(struct phy_device *phydev) { /* RTL8211C has an issue when operating in Gigabit slave mode */ - phy_set_bits(phydev, MII_CTRL1000, - CTL1000_ENABLE_MASTER | CTL1000_AS_MASTER); - - return genphy_config_init(phydev); + return phy_set_bits(phydev, MII_CTRL1000, + CTL1000_ENABLE_MASTER | CTL1000_AS_MASTER); } static int rtl8211f_config_init(struct phy_device *phydev) { - int ret; u16 val = 0; - ret = genphy_config_init(phydev); - if (ret < 0) - return ret; - /* enable TX-delay for rgmii-id and rgmii-txid, otherwise disable it */ if (phydev->interface == PHY_INTERFACE_MODE_RGMII_ID || phydev->interface == PHY_INTERFACE_MODE_RGMII_TXID) @@ -192,10 +185,6 @@ static int rtl8366rb_config_init(struct phy_device *phydev) { int ret; - ret = genphy_config_init(phydev); - if (ret < 0) - return ret; - ret = phy_set_bits(phydev, RTL8366RB_POWER_SAVE, RTL8366RB_POWER_SAVE_ON); if (ret) { @@ -210,11 +199,9 @@ static struct phy_driver realtek_drvs[] = { { PHY_ID_MATCH_EXACT(0x00008201), .name = "RTL8201CP Ethernet", - .features = PHY_BASIC_FEATURES, }, { PHY_ID_MATCH_EXACT(0x001cc816), .name = "RTL8201F Fast Ethernet", - .features = PHY_BASIC_FEATURES, .ack_interrupt = &rtl8201_ack_interrupt, .config_intr = &rtl8201_config_intr, .suspend = genphy_suspend, @@ -224,14 +211,12 @@ static struct phy_driver realtek_drvs[] = { }, { PHY_ID_MATCH_EXACT(0x001cc910), .name = "RTL8211 Gigabit Ethernet", - .features = PHY_GBIT_FEATURES, .config_aneg = rtl8211_config_aneg, .read_mmd = &genphy_read_mmd_unsupported, .write_mmd = &genphy_write_mmd_unsupported, }, { PHY_ID_MATCH_EXACT(0x001cc912), .name = "RTL8211B Gigabit Ethernet", - .features = PHY_GBIT_FEATURES, .ack_interrupt = &rtl821x_ack_interrupt, .config_intr = &rtl8211b_config_intr, .read_mmd = &genphy_read_mmd_unsupported, @@ -241,14 +226,12 @@ static struct phy_driver realtek_drvs[] = { }, { PHY_ID_MATCH_EXACT(0x001cc913), .name = "RTL8211C Gigabit Ethernet", - .features = PHY_GBIT_FEATURES, .config_init = rtl8211c_config_init, .read_mmd = &genphy_read_mmd_unsupported, .write_mmd = &genphy_write_mmd_unsupported, }, { PHY_ID_MATCH_EXACT(0x001cc914), .name = "RTL8211DN Gigabit Ethernet", - .features = PHY_GBIT_FEATURES, .ack_interrupt = rtl821x_ack_interrupt, .config_intr = rtl8211e_config_intr, .suspend = genphy_suspend, @@ -256,7 +239,6 @@ static struct phy_driver realtek_drvs[] = { }, { PHY_ID_MATCH_EXACT(0x001cc915), .name = "RTL8211E Gigabit Ethernet", - .features = PHY_GBIT_FEATURES, .ack_interrupt = &rtl821x_ack_interrupt, .config_intr = &rtl8211e_config_intr, .suspend = genphy_suspend, @@ -264,7 +246,6 @@ static struct phy_driver realtek_drvs[] = { }, { PHY_ID_MATCH_EXACT(0x001cc916), .name = "RTL8211F Gigabit Ethernet", - .features = PHY_GBIT_FEATURES, .config_init = &rtl8211f_config_init, .ack_interrupt = &rtl8211f_ack_interrupt, .config_intr = &rtl8211f_config_intr, @@ -275,8 +256,6 @@ static struct phy_driver realtek_drvs[] = { }, { PHY_ID_MATCH_EXACT(0x001cc800), .name = "Generic Realtek PHY", - .features = PHY_GBIT_FEATURES, - .config_init = genphy_config_init, .suspend = genphy_suspend, .resume = genphy_resume, .read_page = rtl821x_read_page, @@ -284,7 +263,6 @@ static struct phy_driver realtek_drvs[] = { }, { PHY_ID_MATCH_EXACT(0x001cc961), .name = "RTL8366RB Gigabit Ethernet", - .features = PHY_GBIT_FEATURES, .config_init = &rtl8366rb_config_init, /* These interrupts are handled by the irq controller * embedded inside the RTL8366RB, they get unmasked when the diff --git a/drivers/net/phy/rockchip.c b/drivers/net/phy/rockchip.c index 95abf7072f3226ecb02897d372b71b2bd8d082e3..52f1f65320fe035c3722df83b3809b258b0fc0e1 100644 --- a/drivers/net/phy/rockchip.c +++ b/drivers/net/phy/rockchip.c @@ -104,41 +104,14 @@ static int rockchip_integrated_phy_config_init(struct phy_device *phydev) static void rockchip_link_change_notify(struct phy_device *phydev) { - int speed = SPEED_10; - - if (phydev->autoneg == AUTONEG_ENABLE) { - int reg = phy_read(phydev, MII_SPECIAL_CONTROL_STATUS); - - if (reg < 0) { - phydev_err(phydev, "phy_read err: %d.\n", reg); - return; - } - - if (reg & MII_SPEED_100) - speed = SPEED_100; - else if (reg & MII_SPEED_10) - speed = SPEED_10; - } else { - int bmcr = phy_read(phydev, MII_BMCR); - - if (bmcr < 0) { - phydev_err(phydev, "phy_read err: %d.\n", bmcr); - return; - } - - if (bmcr & BMCR_SPEED100) - speed = SPEED_100; - else - speed = SPEED_10; - } - /* * If mode switch happens from 10BT to 100BT, all DSP/AFE * registers are set to default values. So any AFE/DSP * registers have to be re-initialized in this case. */ - if ((phydev->speed == SPEED_10) && (speed == SPEED_100)) { + if (phydev->state == PHY_RUNNING && phydev->speed == SPEED_100) { int ret = rockchip_integrated_phy_analog_init(phydev); + if (ret) phydev_err(phydev, "rockchip_integrated_phy_analog_init err: %d.\n", ret); @@ -202,7 +175,7 @@ static struct phy_driver rockchip_phy_driver[] = { .phy_id = INTERNAL_EPHY_ID, .phy_id_mask = 0xfffffff0, .name = "Rockchip integrated EPHY", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .flags = 0, .link_change_notify = rockchip_link_change_notify, .soft_reset = genphy_soft_reset, diff --git a/drivers/net/phy/smsc.c b/drivers/net/phy/smsc.c index c94d3bfbc7727e825984558920eb58891bf17bfc..dc3d92d340c4d7a019c53706c96b0513cee13356 100644 --- a/drivers/net/phy/smsc.c +++ b/drivers/net/phy/smsc.c @@ -214,7 +214,7 @@ static struct phy_driver smsc_phy_driver[] = { .phy_id_mask = 0xfffffff0, .name = "SMSC LAN83C185", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .probe = smsc_phy_probe, @@ -233,7 +233,7 @@ static struct phy_driver smsc_phy_driver[] = { .phy_id_mask = 0xfffffff0, .name = "SMSC LAN8187", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .probe = smsc_phy_probe, @@ -257,7 +257,7 @@ static struct phy_driver smsc_phy_driver[] = { .phy_id_mask = 0xfffffff0, .name = "SMSC LAN8700", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .probe = smsc_phy_probe, @@ -282,7 +282,7 @@ static struct phy_driver smsc_phy_driver[] = { .phy_id_mask = 0xfffffff0, .name = "SMSC LAN911x Internal PHY", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .probe = smsc_phy_probe, @@ -300,7 +300,7 @@ static struct phy_driver smsc_phy_driver[] = { .phy_id_mask = 0xfffffff0, .name = "SMSC LAN8710/LAN8720", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .flags = PHY_RST_AFTER_CLK_EN, .probe = smsc_phy_probe, @@ -326,7 +326,7 @@ static struct phy_driver smsc_phy_driver[] = { .phy_id_mask = 0xfffffff0, .name = "SMSC LAN8740", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .probe = smsc_phy_probe, diff --git a/drivers/net/phy/ste10Xp.c b/drivers/net/phy/ste10Xp.c index 5b6acf431f9891a6c8bd93280f08fa753550b29b..d735a01380ed5926222565ace8f10685717a4ed8 100644 --- a/drivers/net/phy/ste10Xp.c +++ b/drivers/net/phy/ste10Xp.c @@ -81,7 +81,7 @@ static struct phy_driver ste10xp_pdriver[] = { .phy_id = STE101P_PHY_ID, .phy_id_mask = 0xfffffff0, .name = "STe101p", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = ste10Xp_config_init, .ack_interrupt = ste10Xp_ack_interrupt, .config_intr = ste10Xp_config_intr, @@ -91,7 +91,7 @@ static struct phy_driver ste10xp_pdriver[] = { .phy_id = STE100P_PHY_ID, .phy_id_mask = 0xffffffff, .name = "STe100p", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .config_init = ste10Xp_config_init, .ack_interrupt = ste10Xp_ack_interrupt, .config_intr = ste10Xp_config_intr, diff --git a/drivers/net/phy/uPD60620.c b/drivers/net/phy/uPD60620.c index 219fc7cdc2b3bafcf43fdc42e983b588b0b1e036..a32b3fd8a3705c1fb7e9c8d894631f560dc1196c 100644 --- a/drivers/net/phy/uPD60620.c +++ b/drivers/net/phy/uPD60620.c @@ -87,7 +87,7 @@ static struct phy_driver upd60620_driver[1] = { { .phy_id = UPD60620_PHY_ID, .phy_id_mask = 0xfffffffe, .name = "Renesas uPD60620", - .features = PHY_BASIC_FEATURES, + /* PHY_BASIC_FEATURES */ .flags = 0, .config_init = upd60620_config_init, .read_status = upd60620_read_status, diff --git a/drivers/net/phy/vitesse.c b/drivers/net/phy/vitesse.c index dc0dd87a66948bcbbae9f346aaaf34ef59038b52..43691b1acfd9da199116a7dbd8649b9dbdd666e3 100644 --- a/drivers/net/phy/vitesse.c +++ b/drivers/net/phy/vitesse.c @@ -61,7 +61,6 @@ #define PHY_ID_VSC8234 0x000fc620 #define PHY_ID_VSC8244 0x000fc6c0 -#define PHY_ID_VSC8514 0x00070670 #define PHY_ID_VSC8572 0x000704d0 #define PHY_ID_VSC8601 0x00070420 #define PHY_ID_VSC7385 0x00070450 @@ -293,7 +292,6 @@ static int vsc82xx_config_intr(struct phy_device *phydev) err = phy_write(phydev, MII_VSC8244_IMASK, (phydev->drv->phy_id == PHY_ID_VSC8234 || phydev->drv->phy_id == PHY_ID_VSC8244 || - phydev->drv->phy_id == PHY_ID_VSC8514 || phydev->drv->phy_id == PHY_ID_VSC8572 || phydev->drv->phy_id == PHY_ID_VSC8601) ? MII_VSC8244_IMASK_MASK : @@ -389,7 +387,7 @@ static struct phy_driver vsc82xx_driver[] = { .phy_id = PHY_ID_VSC8234, .name = "Vitesse VSC8234", .phy_id_mask = 0x000ffff0, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = &vsc824x_config_init, .config_aneg = &vsc82x4_config_aneg, .ack_interrupt = &vsc824x_ack_interrupt, @@ -398,16 +396,7 @@ static struct phy_driver vsc82xx_driver[] = { .phy_id = PHY_ID_VSC8244, .name = "Vitesse VSC8244", .phy_id_mask = 0x000fffc0, - .features = PHY_GBIT_FEATURES, - .config_init = &vsc824x_config_init, - .config_aneg = &vsc82x4_config_aneg, - .ack_interrupt = &vsc824x_ack_interrupt, - .config_intr = &vsc82xx_config_intr, -}, { - .phy_id = PHY_ID_VSC8514, - .name = "Vitesse VSC8514", - .phy_id_mask = 0x000ffff0, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = &vsc824x_config_init, .config_aneg = &vsc82x4_config_aneg, .ack_interrupt = &vsc824x_ack_interrupt, @@ -416,7 +405,7 @@ static struct phy_driver vsc82xx_driver[] = { .phy_id = PHY_ID_VSC8572, .name = "Vitesse VSC8572", .phy_id_mask = 0x000ffff0, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = &vsc824x_config_init, .config_aneg = &vsc82x4_config_aneg, .ack_interrupt = &vsc824x_ack_interrupt, @@ -425,7 +414,7 @@ static struct phy_driver vsc82xx_driver[] = { .phy_id = PHY_ID_VSC8601, .name = "Vitesse VSC8601", .phy_id_mask = 0x000ffff0, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = &vsc8601_config_init, .ack_interrupt = &vsc824x_ack_interrupt, .config_intr = &vsc82xx_config_intr, @@ -433,7 +422,7 @@ static struct phy_driver vsc82xx_driver[] = { .phy_id = PHY_ID_VSC7385, .name = "Vitesse VSC7385", .phy_id_mask = 0x000ffff0, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = vsc738x_config_init, .config_aneg = vsc73xx_config_aneg, .read_page = vsc73xx_read_page, @@ -442,7 +431,7 @@ static struct phy_driver vsc82xx_driver[] = { .phy_id = PHY_ID_VSC7388, .name = "Vitesse VSC7388", .phy_id_mask = 0x000ffff0, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = vsc738x_config_init, .config_aneg = vsc73xx_config_aneg, .read_page = vsc73xx_read_page, @@ -451,7 +440,7 @@ static struct phy_driver vsc82xx_driver[] = { .phy_id = PHY_ID_VSC7395, .name = "Vitesse VSC7395", .phy_id_mask = 0x000ffff0, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = vsc739x_config_init, .config_aneg = vsc73xx_config_aneg, .read_page = vsc73xx_read_page, @@ -460,7 +449,7 @@ static struct phy_driver vsc82xx_driver[] = { .phy_id = PHY_ID_VSC7398, .name = "Vitesse VSC7398", .phy_id_mask = 0x000ffff0, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = vsc739x_config_init, .config_aneg = vsc73xx_config_aneg, .read_page = vsc73xx_read_page, @@ -469,7 +458,7 @@ static struct phy_driver vsc82xx_driver[] = { .phy_id = PHY_ID_VSC8662, .name = "Vitesse VSC8662", .phy_id_mask = 0x000ffff0, - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = &vsc824x_config_init, .config_aneg = &vsc82x4_config_aneg, .ack_interrupt = &vsc824x_ack_interrupt, @@ -479,7 +468,7 @@ static struct phy_driver vsc82xx_driver[] = { .phy_id = PHY_ID_VSC8221, .phy_id_mask = 0x000ffff0, .name = "Vitesse VSC8221", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = &vsc8221_config_init, .ack_interrupt = &vsc824x_ack_interrupt, .config_intr = &vsc82xx_config_intr, @@ -488,7 +477,7 @@ static struct phy_driver vsc82xx_driver[] = { .phy_id = PHY_ID_VSC8211, .phy_id_mask = 0x000ffff0, .name = "Vitesse VSC8211", - .features = PHY_GBIT_FEATURES, + /* PHY_GBIT_FEATURES */ .config_init = &vsc8221_config_init, .ack_interrupt = &vsc824x_ack_interrupt, .config_intr = &vsc82xx_config_intr, @@ -499,7 +488,6 @@ module_phy_driver(vsc82xx_driver); static struct mdio_device_id __maybe_unused vitesse_tbl[] = { { PHY_ID_VSC8234, 0x000ffff0 }, { PHY_ID_VSC8244, 0x000fffc0 }, - { PHY_ID_VSC8514, 0x000ffff0 }, { PHY_ID_VSC8572, 0x000ffff0 }, { PHY_ID_VSC7385, 0x000ffff0 }, { PHY_ID_VSC7388, 0x000ffff0 }, diff --git a/drivers/net/sb1000.c b/drivers/net/sb1000.c index 941cfa8f1c2a5dbbb89d1ecba0b27f4579c58e5c..627b3a4405ada621c318ad5b5d1203adae010ae6 100644 --- a/drivers/net/sb1000.c +++ b/drivers/net/sb1000.c @@ -316,7 +316,7 @@ static int card_send_command(const int ioaddr[], const char* name, const unsigned char out[], unsigned char in[]) { - int status, x; + int status; if ((status = card_wait_for_busy_clear(ioaddr, name))) return status; @@ -345,9 +345,7 @@ card_send_command(const int ioaddr[], const char* name, out[0], out[1], out[2], out[3], out[4], out[5]); } - if (out[1] == 0x1b) { - x = (out[2] == 0x02); - } else { + if (out[1] != 0x1b) { if (out[0] >= 0x80 && in[0] != (out[1] | 0x80)) return -EIO; } @@ -490,14 +488,13 @@ sb1000_check_CRC(const int ioaddr[], const char* name) static const unsigned char Command0[6] = {0x80, 0x1f, 0x00, 0x00, 0x00, 0x00}; unsigned char st[7]; - int crc, status; + int status; /* check CRC */ if ((status = card_send_command(ioaddr, name, Command0, st))) return status; if (st[1] != st[3] || st[2] != st[4]) return -EIO; - crc = st[1] << 8 | st[2]; return 0; } diff --git a/drivers/net/team/team.c b/drivers/net/team/team.c index 16963f7a88f748fd0946fafef5c5a477e138cf63..2106045b3e16afb7093da36e953812d651aa3f9d 100644 --- a/drivers/net/team/team.c +++ b/drivers/net/team/team.c @@ -38,13 +38,11 @@ * Helpers **********/ -#define team_port_exists(dev) (dev->priv_flags & IFF_TEAM_PORT) - static struct team_port *team_port_get_rtnl(const struct net_device *dev) { struct team_port *port = rtnl_dereference(dev->rx_handler_data); - return team_port_exists(dev) ? port : NULL; + return netif_is_team_port(dev) ? port : NULL; } /* @@ -1143,7 +1141,7 @@ static int team_port_add(struct team *team, struct net_device *port_dev, return -EINVAL; } - if (team_port_exists(port_dev)) { + if (netif_is_team_port(port_dev)) { NL_SET_ERR_MSG(extack, "Device is already a port of a team device"); netdev_err(dev, "Device %s is already a port " "of a team device\n", portname); @@ -1724,8 +1722,7 @@ static netdev_tx_t team_xmit(struct sk_buff *skb, struct net_device *dev) } static u16 team_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { /* * This helper function exists to help dev_pick_tx get the correct @@ -2293,7 +2290,7 @@ static int team_nl_fill_one_option_get(struct sk_buff *skb, struct team *team, if (err) return err; - option_item = nla_nest_start(skb, TEAM_ATTR_ITEM_OPTION); + option_item = nla_nest_start_noflag(skb, TEAM_ATTR_ITEM_OPTION); if (!option_item) return -EMSGSIZE; @@ -2407,7 +2404,7 @@ start_again: if (nla_put_u32(skb, TEAM_ATTR_TEAM_IFINDEX, team->dev->ifindex)) goto nla_put_failure; - option_list = nla_nest_start(skb, TEAM_ATTR_LIST_OPTION); + option_list = nla_nest_start_noflag(skb, TEAM_ATTR_LIST_OPTION); if (!option_list) goto nla_put_failure; @@ -2513,9 +2510,11 @@ static int team_nl_cmd_options_set(struct sk_buff *skb, struct genl_info *info) err = -EINVAL; goto team_put; } - err = nla_parse_nested(opt_attrs, TEAM_ATTR_OPTION_MAX, - nl_option, team_nl_option_policy, - info->extack); + err = nla_parse_nested_deprecated(opt_attrs, + TEAM_ATTR_OPTION_MAX, + nl_option, + team_nl_option_policy, + info->extack); if (err) goto team_put; if (!opt_attrs[TEAM_ATTR_OPTION_NAME] || @@ -2629,7 +2628,7 @@ static int team_nl_fill_one_port_get(struct sk_buff *skb, { struct nlattr *port_item; - port_item = nla_nest_start(skb, TEAM_ATTR_ITEM_PORT); + port_item = nla_nest_start_noflag(skb, TEAM_ATTR_ITEM_PORT); if (!port_item) goto nest_cancel; if (nla_put_u32(skb, TEAM_ATTR_PORT_IFINDEX, port->dev->ifindex)) @@ -2684,7 +2683,7 @@ start_again: if (nla_put_u32(skb, TEAM_ATTR_TEAM_IFINDEX, team->dev->ifindex)) goto nla_put_failure; - port_list = nla_nest_start(skb, TEAM_ATTR_LIST_PORT); + port_list = nla_nest_start_noflag(skb, TEAM_ATTR_LIST_PORT); if (!port_list) goto nla_put_failure; @@ -2758,25 +2757,25 @@ static int team_nl_cmd_port_list_get(struct sk_buff *skb, static const struct genl_ops team_nl_ops[] = { { .cmd = TEAM_CMD_NOOP, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = team_nl_cmd_noop, - .policy = team_nl_policy, }, { .cmd = TEAM_CMD_OPTIONS_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = team_nl_cmd_options_set, - .policy = team_nl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = TEAM_CMD_OPTIONS_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = team_nl_cmd_options_get, - .policy = team_nl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = TEAM_CMD_PORT_LIST_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = team_nl_cmd_port_list_get, - .policy = team_nl_policy, .flags = GENL_ADMIN_PERM, }, }; @@ -2789,6 +2788,7 @@ static struct genl_family team_nl_family __ro_after_init = { .name = TEAM_GENL_NAME, .version = TEAM_GENL_VERSION, .maxattr = TEAM_ATTR_MAX, + .policy = team_nl_policy, .netnsok = true, .module = THIS_MODULE, .ops = team_nl_ops, diff --git a/drivers/net/tun.c b/drivers/net/tun.c index e9ca1c088d0b11611e4d80268ced3806db05cffb..9d72f8c76c158322cb54f67d8adefb6343a9283a 100644 --- a/drivers/net/tun.c +++ b/drivers/net/tun.c @@ -606,8 +606,7 @@ static u16 tun_ebpf_select_queue(struct tun_struct *tun, struct sk_buff *skb) } static u16 tun_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { struct tun_struct *tun = netdev_priv(dev); u16 ret; @@ -1043,7 +1042,7 @@ static int tun_net_close(struct net_device *dev) static void tun_automq_xmit(struct tun_struct *tun, struct sk_buff *skb) { #ifdef CONFIG_RPS - if (tun->numqueues == 1 && static_key_false(&rps_needed)) { + if (tun->numqueues == 1 && static_branch_unlikely(&rps_needed)) { /* Select queue was not called for the skbuff, so we extract the * RPS hash and save it into the flow_table here. */ @@ -1966,7 +1965,8 @@ drop: if (frags) { /* Exercise flow dissector code path. */ - u32 headlen = eth_get_headlen(skb->data, skb_headlen(skb)); + u32 headlen = eth_get_headlen(tun->dev, skb->data, + skb_headlen(skb)); if (unlikely(headlen > skb_headlen(skb))) { this_cpu_inc(tun->pcpu_stats->rx_dropped); @@ -2873,8 +2873,7 @@ err_free_dev: return err; } -static void tun_get_iff(struct net *net, struct tun_struct *tun, - struct ifreq *ifr) +static void tun_get_iff(struct tun_struct *tun, struct ifreq *ifr) { tun_debug(KERN_INFO, tun, "tun_get_iff\n"); @@ -3103,10 +3102,11 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd, tun_debug(KERN_INFO, tun, "tun_chr_ioctl cmd %u\n", cmd); + net = dev_net(tun->dev); ret = 0; switch (cmd) { case TUNGETIFF: - tun_get_iff(current->nsproxy->net_ns, tun, &ifr); + tun_get_iff(tun, &ifr); if (tfile->detached) ifr.ifr_flags |= IFF_DETACH_QUEUE; @@ -3328,6 +3328,13 @@ static long __tun_chr_ioctl(struct file *file, unsigned int cmd, ret = tun_net_change_carrier(tun->dev, (bool)carrier); break; + case TUNGETDEVNETNS: + ret = -EPERM; + if (!ns_capable(net->user_ns, CAP_NET_ADMIN)) + goto unlock; + ret = open_related_ns(&net->ns, get_net_ns); + break; + default: ret = -EINVAL; break; @@ -3457,7 +3464,7 @@ static void tun_chr_show_fdinfo(struct seq_file *m, struct file *file) rtnl_lock(); tun = tun_get(tfile); if (tun) - tun_get_iff(current->nsproxy->net_ns, tun, &ifr); + tun_get_iff(tun, &ifr); rtnl_unlock(); if (tun) diff --git a/drivers/net/usb/cdc_mbim.c b/drivers/net/usb/cdc_mbim.c index 0362acd5cdcaaf2debe7ab13941a06d1504ad21c..28321aca48fe230dd3ab05df2859ac94e8880f5d 100644 --- a/drivers/net/usb/cdc_mbim.c +++ b/drivers/net/usb/cdc_mbim.c @@ -23,6 +23,7 @@ #include #include #include +#include /* alternative VLAN for IP session 0 if not untagged */ #define MBIM_IPS0_VID 4094 diff --git a/drivers/net/usb/ipheth.c b/drivers/net/usb/ipheth.c index 3d8a70d3ea9bd67c91f85c6cab38d1afbf5f25f3..c247aed2dcebbe8025a8c9595c530c03a9240712 100644 --- a/drivers/net/usb/ipheth.c +++ b/drivers/net/usb/ipheth.c @@ -54,17 +54,6 @@ #include #define USB_VENDOR_APPLE 0x05ac -#define USB_PRODUCT_IPHONE 0x1290 -#define USB_PRODUCT_IPHONE_3G 0x1292 -#define USB_PRODUCT_IPHONE_3GS 0x1294 -#define USB_PRODUCT_IPHONE_4 0x1297 -#define USB_PRODUCT_IPAD 0x129a -#define USB_PRODUCT_IPAD_2 0x12a2 -#define USB_PRODUCT_IPAD_3 0x12a6 -#define USB_PRODUCT_IPAD_MINI 0x12ab -#define USB_PRODUCT_IPHONE_4_VZW 0x129c -#define USB_PRODUCT_IPHONE_4S 0x12a0 -#define USB_PRODUCT_IPHONE_5 0x12a8 #define IPHETH_USBINTF_CLASS 255 #define IPHETH_USBINTF_SUBCLASS 253 @@ -88,50 +77,9 @@ #define IPHETH_CARRIER_ON 0x04 static const struct usb_device_id ipheth_table[] = { - { USB_DEVICE_AND_INTERFACE_INFO( - USB_VENDOR_APPLE, USB_PRODUCT_IPHONE, - IPHETH_USBINTF_CLASS, IPHETH_USBINTF_SUBCLASS, - IPHETH_USBINTF_PROTO) }, - { USB_DEVICE_AND_INTERFACE_INFO( - USB_VENDOR_APPLE, USB_PRODUCT_IPHONE_3G, - IPHETH_USBINTF_CLASS, IPHETH_USBINTF_SUBCLASS, - IPHETH_USBINTF_PROTO) }, - { USB_DEVICE_AND_INTERFACE_INFO( - USB_VENDOR_APPLE, USB_PRODUCT_IPHONE_3GS, - IPHETH_USBINTF_CLASS, IPHETH_USBINTF_SUBCLASS, - IPHETH_USBINTF_PROTO) }, - { USB_DEVICE_AND_INTERFACE_INFO( - USB_VENDOR_APPLE, USB_PRODUCT_IPHONE_4, - IPHETH_USBINTF_CLASS, IPHETH_USBINTF_SUBCLASS, - IPHETH_USBINTF_PROTO) }, - { USB_DEVICE_AND_INTERFACE_INFO( - USB_VENDOR_APPLE, USB_PRODUCT_IPAD, - IPHETH_USBINTF_CLASS, IPHETH_USBINTF_SUBCLASS, - IPHETH_USBINTF_PROTO) }, - { USB_DEVICE_AND_INTERFACE_INFO( - USB_VENDOR_APPLE, USB_PRODUCT_IPAD_2, - IPHETH_USBINTF_CLASS, IPHETH_USBINTF_SUBCLASS, - IPHETH_USBINTF_PROTO) }, - { USB_DEVICE_AND_INTERFACE_INFO( - USB_VENDOR_APPLE, USB_PRODUCT_IPAD_3, - IPHETH_USBINTF_CLASS, IPHETH_USBINTF_SUBCLASS, - IPHETH_USBINTF_PROTO) }, - { USB_DEVICE_AND_INTERFACE_INFO( - USB_VENDOR_APPLE, USB_PRODUCT_IPAD_MINI, - IPHETH_USBINTF_CLASS, IPHETH_USBINTF_SUBCLASS, - IPHETH_USBINTF_PROTO) }, - { USB_DEVICE_AND_INTERFACE_INFO( - USB_VENDOR_APPLE, USB_PRODUCT_IPHONE_4_VZW, - IPHETH_USBINTF_CLASS, IPHETH_USBINTF_SUBCLASS, - IPHETH_USBINTF_PROTO) }, - { USB_DEVICE_AND_INTERFACE_INFO( - USB_VENDOR_APPLE, USB_PRODUCT_IPHONE_4S, - IPHETH_USBINTF_CLASS, IPHETH_USBINTF_SUBCLASS, - IPHETH_USBINTF_PROTO) }, - { USB_DEVICE_AND_INTERFACE_INFO( - USB_VENDOR_APPLE, USB_PRODUCT_IPHONE_5, - IPHETH_USBINTF_CLASS, IPHETH_USBINTF_SUBCLASS, - IPHETH_USBINTF_PROTO) }, + { USB_VENDOR_AND_INTERFACE_INFO(USB_VENDOR_APPLE, IPHETH_USBINTF_CLASS, + IPHETH_USBINTF_SUBCLASS, + IPHETH_USBINTF_PROTO) }, { } }; MODULE_DEVICE_TABLE(usb, ipheth_table); @@ -293,8 +241,6 @@ static int ipheth_carrier_set(struct ipheth_device *dev) struct usb_device *udev; int retval; - if (!dev) - return 0; if (!dev->confirmed_pairing) return 0; diff --git a/drivers/net/usb/qmi_wwan.c b/drivers/net/usb/qmi_wwan.c index 679e404a5224fc33b8591b16a59e0910adb4c4cd..5c3ac97519b7d245141ad91551b887a6cfc9b7fb 100644 --- a/drivers/net/usb/qmi_wwan.c +++ b/drivers/net/usb/qmi_wwan.c @@ -63,6 +63,7 @@ enum qmi_wwan_flags { enum qmi_wwan_quirks { QMI_WWAN_QUIRK_DTR = 1 << 0, /* needs "set DTR" request */ + QMI_WWAN_QUIRK_QUECTEL_DYNCFG = 1 << 1, /* check num. endpoints */ }; struct qmimux_hdr { @@ -845,6 +846,16 @@ static const struct driver_info qmi_wwan_info_quirk_dtr = { .data = QMI_WWAN_QUIRK_DTR, }; +static const struct driver_info qmi_wwan_info_quirk_quectel_dyncfg = { + .description = "WWAN/QMI device", + .flags = FLAG_WWAN | FLAG_SEND_ZLP, + .bind = qmi_wwan_bind, + .unbind = qmi_wwan_unbind, + .manage_power = qmi_wwan_manage_power, + .rx_fixup = qmi_wwan_rx_fixup, + .data = QMI_WWAN_QUIRK_DTR | QMI_WWAN_QUIRK_QUECTEL_DYNCFG, +}; + #define HUAWEI_VENDOR_ID 0x12D1 /* map QMI/wwan function by a fixed interface number */ @@ -865,6 +876,15 @@ static const struct driver_info qmi_wwan_info_quirk_dtr = { #define QMI_GOBI_DEVICE(vend, prod) \ QMI_FIXED_INTF(vend, prod, 0) +/* Quectel does not use fixed interface numbers on at least some of their + * devices. We need to check the number of endpoints to ensure that we bind to + * the correct interface. + */ +#define QMI_QUIRK_QUECTEL_DYNCFG(vend, prod) \ + USB_DEVICE_AND_INTERFACE_INFO(vend, prod, USB_CLASS_VENDOR_SPEC, \ + USB_SUBCLASS_VENDOR_SPEC, 0xff), \ + .driver_info = (unsigned long)&qmi_wwan_info_quirk_quectel_dyncfg + static const struct usb_device_id products[] = { /* 1. CDC ECM like devices match on the control interface */ { /* Huawei E392, E398 and possibly others sharing both device id and more... */ @@ -969,20 +989,9 @@ static const struct usb_device_id products[] = { USB_DEVICE_AND_INTERFACE_INFO(0x03f0, 0x581d, USB_CLASS_VENDOR_SPEC, 1, 7), .driver_info = (unsigned long)&qmi_wwan_info, }, - { /* Quectel EP06/EG06/EM06 */ - USB_DEVICE_AND_INTERFACE_INFO(0x2c7c, 0x0306, - USB_CLASS_VENDOR_SPEC, - USB_SUBCLASS_VENDOR_SPEC, - 0xff), - .driver_info = (unsigned long)&qmi_wwan_info_quirk_dtr, - }, - { /* Quectel EG12/EM12 */ - USB_DEVICE_AND_INTERFACE_INFO(0x2c7c, 0x0512, - USB_CLASS_VENDOR_SPEC, - USB_SUBCLASS_VENDOR_SPEC, - 0xff), - .driver_info = (unsigned long)&qmi_wwan_info_quirk_dtr, - }, + {QMI_QUIRK_QUECTEL_DYNCFG(0x2c7c, 0x0125)}, /* Quectel EC25, EC20 R2.0 Mini PCIe */ + {QMI_QUIRK_QUECTEL_DYNCFG(0x2c7c, 0x0306)}, /* Quectel EP06/EG06/EM06 */ + {QMI_QUIRK_QUECTEL_DYNCFG(0x2c7c, 0x0512)}, /* Quectel EG12/EM12 */ /* 3. Combined interface devices matching on interface number */ {QMI_FIXED_INTF(0x0408, 0xea42, 4)}, /* Yota / Megafon M100-1 */ @@ -1281,7 +1290,6 @@ static const struct usb_device_id products[] = { {QMI_FIXED_INTF(0x03f0, 0x9d1d, 1)}, /* HP lt4120 Snapdragon X5 LTE */ {QMI_FIXED_INTF(0x22de, 0x9061, 3)}, /* WeTelecom WPD-600N */ {QMI_QUIRK_SET_DTR(0x1e0e, 0x9001, 5)}, /* SIMCom 7100E, 7230E, 7600E ++ */ - {QMI_QUIRK_SET_DTR(0x2c7c, 0x0125, 4)}, /* Quectel EC25, EC20 R2.0 Mini PCIe */ {QMI_QUIRK_SET_DTR(0x2c7c, 0x0121, 4)}, /* Quectel EC21 Mini PCIe */ {QMI_QUIRK_SET_DTR(0x2c7c, 0x0191, 4)}, /* Quectel EG91 */ {QMI_FIXED_INTF(0x2c7c, 0x0296, 4)}, /* Quectel BG96 */ @@ -1361,27 +1369,12 @@ static bool quectel_ec20_detected(struct usb_interface *intf) return false; } -static bool quectel_diag_detected(struct usb_interface *intf) -{ - struct usb_device *dev = interface_to_usbdev(intf); - struct usb_interface_descriptor intf_desc = intf->cur_altsetting->desc; - u16 id_vendor = le16_to_cpu(dev->descriptor.idVendor); - u16 id_product = le16_to_cpu(dev->descriptor.idProduct); - - if (id_vendor != 0x2c7c || intf_desc.bNumEndpoints != 2) - return false; - - if (id_product == 0x0306 || id_product == 0x0512) - return true; - else - return false; -} - static int qmi_wwan_probe(struct usb_interface *intf, const struct usb_device_id *prod) { struct usb_device_id *id = (struct usb_device_id *)prod; struct usb_interface_descriptor *desc = &intf->cur_altsetting->desc; + const struct driver_info *info; /* Workaround to enable dynamic IDs. This disables usbnet * blacklisting functionality. Which, if required, can be @@ -1415,10 +1408,14 @@ static int qmi_wwan_probe(struct usb_interface *intf, * we need to match on class/subclass/protocol. These values are * identical for the diagnostic- and QMI-interface, but bNumEndpoints is * different. Ignore the current interface if the number of endpoints - * the number for the diag interface (two). + * equals the number for the diag interface (two). */ - if (quectel_diag_detected(intf)) - return -ENODEV; + info = (void *)&id->driver_info; + + if (info->data & QMI_WWAN_QUIRK_QUECTEL_DYNCFG) { + if (desc->bNumEndpoints == 2) + return -ENODEV; + } return usbnet_probe(intf, id); } diff --git a/drivers/net/usb/r8152.c b/drivers/net/usb/r8152.c index 86c8c64fbb0f33e920a5fbe77e2d73fe69c14991..b01bfa63860dbc9c305cdbe51520951eb97af9d9 100644 --- a/drivers/net/usb/r8152.c +++ b/drivers/net/usb/r8152.c @@ -1212,7 +1212,6 @@ static int vendor_mac_passthru_addr_read(struct r8152 *tp, struct sockaddr *sa) goto amacout; } memcpy(sa->sa_data, buf, 6); - ether_addr_copy(tp->netdev->dev_addr, sa->sa_data); netif_info(tp, probe, tp->netdev, "Using pass-thru MAC addr %pM\n", sa->sa_data); @@ -1221,43 +1220,57 @@ amacout: return ret; } -static int set_ethernet_addr(struct r8152 *tp) +static int determine_ethernet_addr(struct r8152 *tp, struct sockaddr *sa) { struct net_device *dev = tp->netdev; - struct sockaddr sa; int ret; + sa->sa_family = dev->type; + if (tp->version == RTL_VER_01) { - ret = pla_ocp_read(tp, PLA_IDR, 8, sa.sa_data); + ret = pla_ocp_read(tp, PLA_IDR, 8, sa->sa_data); } else { /* if device doesn't support MAC pass through this will * be expected to be non-zero */ - ret = vendor_mac_passthru_addr_read(tp, &sa); + ret = vendor_mac_passthru_addr_read(tp, sa); if (ret < 0) - ret = pla_ocp_read(tp, PLA_BACKUP, 8, sa.sa_data); + ret = pla_ocp_read(tp, PLA_BACKUP, 8, sa->sa_data); } if (ret < 0) { netif_err(tp, probe, dev, "Get ether addr fail\n"); - } else if (!is_valid_ether_addr(sa.sa_data)) { + } else if (!is_valid_ether_addr(sa->sa_data)) { netif_err(tp, probe, dev, "Invalid ether addr %pM\n", - sa.sa_data); + sa->sa_data); eth_hw_addr_random(dev); - ether_addr_copy(sa.sa_data, dev->dev_addr); - ret = rtl8152_set_mac_address(dev, &sa); + ether_addr_copy(sa->sa_data, dev->dev_addr); netif_info(tp, probe, dev, "Random ether addr %pM\n", - sa.sa_data); - } else { - if (tp->version == RTL_VER_01) - ether_addr_copy(dev->dev_addr, sa.sa_data); - else - ret = rtl8152_set_mac_address(dev, &sa); + sa->sa_data); + return 0; } return ret; } +static int set_ethernet_addr(struct r8152 *tp) +{ + struct net_device *dev = tp->netdev; + struct sockaddr sa; + int ret; + + ret = determine_ethernet_addr(tp, &sa); + if (ret < 0) + return ret; + + if (tp->version == RTL_VER_01) + ether_addr_copy(dev->dev_addr, sa.sa_data); + else + ret = rtl8152_set_mac_address(dev, &sa); + + return ret; +} + static void read_bulk_callback(struct urb *urb) { struct net_device *netdev; @@ -4264,10 +4277,18 @@ static int rtl8152_post_reset(struct usb_interface *intf) { struct r8152 *tp = usb_get_intfdata(intf); struct net_device *netdev; + struct sockaddr sa; if (!tp) return 0; + /* reset the MAC adddress in case of policy change */ + if (determine_ethernet_addr(tp, &sa) >= 0) { + rtnl_lock(); + dev_set_mac_address (tp->netdev, &sa, NULL); + rtnl_unlock(); + } + netdev = tp->netdev; if (!netif_running(netdev)) return 0; diff --git a/drivers/net/usb/smsc75xx.c b/drivers/net/usb/smsc75xx.c index ec287c9741e833eb2af7b2878ee08ff1941227b0..e4c2f3afce604544045a5562c29f76bb80654327 100644 --- a/drivers/net/usb/smsc75xx.c +++ b/drivers/net/usb/smsc75xx.c @@ -774,8 +774,8 @@ static void smsc75xx_init_mac_address(struct usbnet *dev) /* maybe the boot loader passed the MAC address in devicetree */ mac_addr = of_get_mac_address(dev->udev->dev.of_node); - if (mac_addr) { - memcpy(dev->net->dev_addr, mac_addr, ETH_ALEN); + if (!IS_ERR(mac_addr)) { + ether_addr_copy(dev->net->dev_addr, mac_addr); return; } diff --git a/drivers/net/usb/smsc95xx.c b/drivers/net/usb/smsc95xx.c index e3d08626828e4890a413d1081fa8beba8398636d..a0e119907c842a808ccfeb5017e393abdad2197d 100644 --- a/drivers/net/usb/smsc95xx.c +++ b/drivers/net/usb/smsc95xx.c @@ -917,8 +917,8 @@ static void smsc95xx_init_mac_address(struct usbnet *dev) /* maybe the boot loader passed the MAC address in devicetree */ mac_addr = of_get_mac_address(dev->udev->dev.of_node); - if (mac_addr) { - memcpy(dev->net->dev_addr, mac_addr, ETH_ALEN); + if (!IS_ERR(mac_addr)) { + ether_addr_copy(dev->net->dev_addr, mac_addr); return; } diff --git a/drivers/net/veth.c b/drivers/net/veth.c index 569e87a51a3336cb630fe2d752546e867e62f318..09a1433b08332996163a5344d91f5de430890d2d 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -162,18 +162,6 @@ static void veth_get_ethtool_stats(struct net_device *dev, } } -static int veth_get_ts_info(struct net_device *dev, - struct ethtool_ts_info *info) -{ - info->so_timestamping = - SOF_TIMESTAMPING_TX_SOFTWARE | - SOF_TIMESTAMPING_RX_SOFTWARE | - SOF_TIMESTAMPING_SOFTWARE; - info->phc_index = -1; - - return 0; -} - static const struct ethtool_ops veth_ethtool_ops = { .get_drvinfo = veth_get_drvinfo, .get_link = ethtool_op_get_link, @@ -181,7 +169,7 @@ static const struct ethtool_ops veth_ethtool_ops = { .get_sset_count = veth_get_sset_count, .get_ethtool_stats = veth_get_ethtool_stats, .get_link_ksettings = veth_get_link_ksettings, - .get_ts_info = veth_get_ts_info, + .get_ts_info = ethtool_op_get_ts_info, }; /* general routines */ diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 7eb38ea9ba56318ba9e6c4686558ff12d3221ce3..559c48e66afc4904048d1486b636742eb14e4608 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -31,7 +31,6 @@ #include #include #include -#include #include #include #include @@ -1568,7 +1567,7 @@ static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev) struct send_queue *sq = &vi->sq[qnum]; int err; struct netdev_queue *txq = netdev_get_tx_queue(dev, qnum); - bool kick = !skb->xmit_more; + bool kick = !netdev_xmit_more(); bool use_napi = sq->napi.weight; /* Free up any pending old buffers before queueing new ones. */ @@ -1588,7 +1587,8 @@ static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev) dev->stats.tx_fifo_errors++; if (net_ratelimit()) dev_warn(&dev->dev, - "Unexpected TXQ (%d) queue failure: %d\n", qnum, err); + "Unexpected TXQ (%d) queue failure: %d\n", + qnum, err); dev->stats.tx_dropped++; dev_kfree_skb_any(skb); return NETDEV_TX_OK; @@ -1925,7 +1925,7 @@ static int virtnet_vlan_rx_kill_vid(struct net_device *dev, return 0; } -static void virtnet_clean_affinity(struct virtnet_info *vi, long hcpu) +static void virtnet_clean_affinity(struct virtnet_info *vi) { int i; @@ -1949,7 +1949,7 @@ static void virtnet_set_affinity(struct virtnet_info *vi) int stride; if (!zalloc_cpumask_var(&mask, GFP_KERNEL)) { - virtnet_clean_affinity(vi, -1); + virtnet_clean_affinity(vi); return; } @@ -1999,7 +1999,7 @@ static int virtnet_cpu_down_prep(unsigned int cpu, struct hlist_node *node) struct virtnet_info *vi = hlist_entry_safe(node, struct virtnet_info, node); - virtnet_clean_affinity(vi, cpu); + virtnet_clean_affinity(vi); return 0; } @@ -2384,7 +2384,7 @@ static int virtnet_set_guest_offloads(struct virtnet_info *vi, u64 offloads) if (!virtnet_send_command(vi, VIRTIO_NET_CTRL_GUEST_OFFLOADS, VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET, &sg)) { - dev_warn(&vi->dev->dev, "Fail to set guest offload. \n"); + dev_warn(&vi->dev->dev, "Fail to set guest offload.\n"); return -EINVAL; } @@ -2735,7 +2735,7 @@ static void virtnet_del_vqs(struct virtnet_info *vi) { struct virtio_device *vdev = vi->vdev; - virtnet_clean_affinity(vi, -1); + virtnet_clean_affinity(vi); vdev->config->del_vqs(vdev); @@ -3115,8 +3115,9 @@ static int virtnet_probe(struct virtio_device *vdev) /* Should never trigger: MTU was previously validated * in virtnet_validate. */ - dev_err(&vdev->dev, "device MTU appears to have changed " - "it is now %d < %d", mtu, dev->min_mtu); + dev_err(&vdev->dev, + "device MTU appears to have changed it is now %d < %d", + mtu, dev->min_mtu); goto free; } diff --git a/drivers/net/vrf.c b/drivers/net/vrf.c index 9ee4d7402ca23296091939a59a5a6fec459a8472..cf7e6a92e73ca298b837296dc2b6e10b3b67e9ec 100644 --- a/drivers/net/vrf.c +++ b/drivers/net/vrf.c @@ -370,7 +370,7 @@ static int vrf_finish_output6(struct net *net, struct sock *sk, neigh = __neigh_create(&nd_tbl, nexthop, dst->dev, false); if (!IS_ERR(neigh)) { sock_confirm_neigh(skb, neigh); - ret = neigh_output(neigh, skb); + ret = neigh_output(neigh, skb, false); rcu_read_unlock_bh(); return ret; } @@ -549,7 +549,7 @@ static int vrf_finish_output(struct net *net, struct sock *sk, struct sk_buff *s struct net_device *dev = dst->dev; unsigned int hh_len = LL_RESERVED_SPACE(dev); struct neighbour *neigh; - u32 nexthop; + bool is_v6gw = false; int ret = -EINVAL; nf_reset(skb); @@ -572,13 +572,11 @@ static int vrf_finish_output(struct net *net, struct sock *sk, struct sk_buff *s rcu_read_lock_bh(); - nexthop = (__force u32)rt_nexthop(rt, ip_hdr(skb)->daddr); - neigh = __ipv4_neigh_lookup_noref(dev, nexthop); - if (unlikely(!neigh)) - neigh = __neigh_create(&arp_tbl, &nexthop, dev, false); + neigh = ip_neigh_for_gw(rt, skb, &is_v6gw); if (!IS_ERR(neigh)) { sock_confirm_neigh(skb, neigh); - ret = neigh_output(neigh, skb); + /* if crossing protocols, can not use the cached header */ + ret = neigh_output(neigh, skb, is_v6gw); rcu_read_unlock_bh(); return ret; } diff --git a/drivers/net/vxlan.c b/drivers/net/vxlan.c index d76dfed8d9bbef1d1ae8470686e417af2e531ac8..5994d5415a03324944472a9a875e3c1471bf5641 100644 --- a/drivers/net/vxlan.c +++ b/drivers/net/vxlan.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include diff --git a/drivers/net/wimax/i2400m/control.c b/drivers/net/wimax/i2400m/control.c index ef298d8525c5481c8df55f933cf0df6452ba41c0..4fe7c7e132c496611ddd1410c169cf2886adbc8b 100644 --- a/drivers/net/wimax/i2400m/control.c +++ b/drivers/net/wimax/i2400m/control.c @@ -352,6 +352,7 @@ void i2400m_report_tlv_system_state(struct i2400m *i2400m, case I2400M_SS_IDLE: d_printf(1, dev, "entering BS-negotiated idle mode\n"); + /* Fall through */ case I2400M_SS_DISCONNECTING: case I2400M_SS_DATA_PATH_CONNECTED: wimax_state_change(wimax_dev, WIMAX_ST_CONNECTED); diff --git a/drivers/net/wimax/i2400m/tx.c b/drivers/net/wimax/i2400m/tx.c index f20886ade1ccdf3147253e90f397263066eab431..ebd64e0837264bb9636bed91f58e6809edbaad47 100644 --- a/drivers/net/wimax/i2400m/tx.c +++ b/drivers/net/wimax/i2400m/tx.c @@ -640,8 +640,7 @@ void i2400m_tx_close(struct i2400m *i2400m) * figure out where the next TX message starts (and where the * offset to the moved header is). */ - hdr_size = sizeof(*tx_msg) - + le16_to_cpu(tx_msg->num_pls) * sizeof(tx_msg->pld[0]); + hdr_size = struct_size(tx_msg, pld, le16_to_cpu(tx_msg->num_pls)); hdr_size = ALIGN(hdr_size, I2400M_PL_ALIGN); tx_msg->offset = I2400M_TX_PLD_SIZE - hdr_size; tx_msg_moved = (void *) tx_msg + tx_msg->offset; diff --git a/drivers/net/wireless/ath/ath10k/testmode.c b/drivers/net/wireless/ath/ath10k/testmode.c index 6433ff10d80e10488d066c4312609b1d768a1a47..a29cfb9c72c24c07686f1dcdaa6b10381c5b3904 100644 --- a/drivers/net/wireless/ath/ath10k/testmode.c +++ b/drivers/net/wireless/ath/ath10k/testmode.c @@ -416,8 +416,8 @@ int ath10k_tm_cmd(struct ieee80211_hw *hw, struct ieee80211_vif *vif, struct nlattr *tb[ATH10K_TM_ATTR_MAX + 1]; int ret; - ret = nla_parse(tb, ATH10K_TM_ATTR_MAX, data, len, ath10k_tm_policy, - NULL); + ret = nla_parse_deprecated(tb, ATH10K_TM_ATTR_MAX, data, len, + ath10k_tm_policy, NULL); if (ret) return ret; diff --git a/drivers/net/wireless/ath/ath6kl/testmode.c b/drivers/net/wireless/ath/ath6kl/testmode.c index d8dcacda9add2b23e5f38bc66263190fe4ba4be1..f3906dbe5495ee56ae949c66f531e48532add242 100644 --- a/drivers/net/wireless/ath/ath6kl/testmode.c +++ b/drivers/net/wireless/ath/ath6kl/testmode.c @@ -74,8 +74,8 @@ int ath6kl_tm_cmd(struct wiphy *wiphy, struct wireless_dev *wdev, int err, buf_len; void *buf; - err = nla_parse(tb, ATH6KL_TM_ATTR_MAX, data, len, ath6kl_tm_policy, - NULL); + err = nla_parse_deprecated(tb, ATH6KL_TM_ATTR_MAX, data, len, + ath6kl_tm_policy, NULL); if (err) return err; diff --git a/drivers/net/wireless/ath/ath9k/init.c b/drivers/net/wireless/ath/ath9k/init.c index 98141b699c883cf327ca477cc087ae71297dbb27..a04d8616fe09f8ea26bea20513e9e80eaf374c1a 100644 --- a/drivers/net/wireless/ath/ath9k/init.c +++ b/drivers/net/wireless/ath/ath9k/init.c @@ -642,7 +642,7 @@ static int ath9k_of_init(struct ath_softc *sc) } mac = of_get_mac_address(np); - if (mac) + if (!IS_ERR(mac)) ether_addr_copy(common->macaddr, mac); return 0; diff --git a/drivers/net/wireless/ath/wcn36xx/testmode.c b/drivers/net/wireless/ath/wcn36xx/testmode.c index 51a038022c8b80404b9bd841c6fefd3b866ffe66..7ae14b4d2d0e95f8187465df28d0aaf29dc6cb1d 100644 --- a/drivers/net/wireless/ath/wcn36xx/testmode.c +++ b/drivers/net/wireless/ath/wcn36xx/testmode.c @@ -132,8 +132,8 @@ int wcn36xx_tm_cmd(struct ieee80211_hw *hw, struct ieee80211_vif *vif, unsigned short attr; wcn36xx_dbg_dump(WCN36XX_DBG_TESTMODE_DUMP, "Data:", data, len); - ret = nla_parse(tb, WCN36XX_TM_ATTR_MAX, data, len, - wcn36xx_tm_policy, NULL); + ret = nla_parse_deprecated(tb, WCN36XX_TM_ATTR_MAX, data, len, + wcn36xx_tm_policy, NULL); if (ret) return ret; diff --git a/drivers/net/wireless/ath/wil6210/cfg80211.c b/drivers/net/wireless/ath/wil6210/cfg80211.c index a1e226652b4abfff4f01fffd870e7c9616ffff98..804955d24b3040d946c5fbb24a48bcbadb0b0095 100644 --- a/drivers/net/wireless/ath/wil6210/cfg80211.c +++ b/drivers/net/wireless/ath/wil6210/cfg80211.c @@ -465,7 +465,7 @@ static int wil_cfg80211_validate_add_iface(struct wil6210_priv *wil, .num_different_channels = 1, }; - for (i = 0; i < wil->max_vifs; i++) { + for (i = 0; i < GET_MAX_VIFS(wil); i++) { if (wil->vifs[i]) { wdev = vif_to_wdev(wil->vifs[i]); params.iftype_num[wdev->iftype]++; @@ -486,7 +486,7 @@ static int wil_cfg80211_validate_change_iface(struct wil6210_priv *wil, }; bool check_combos = false; - for (i = 0; i < wil->max_vifs; i++) { + for (i = 0; i < GET_MAX_VIFS(wil); i++) { struct wil6210_vif *vif_pos = wil->vifs[i]; if (vif_pos && vif != vif_pos) { @@ -1274,7 +1274,12 @@ int wil_cfg80211_mgmt_tx(struct wiphy *wiphy, struct wireless_dev *wdev, params->wait); out: + /* when the sent packet was not acked by receiver(ACK=0), rc will + * be -EAGAIN. In this case this function needs to return success, + * the ACK=0 will be reflected in tx_status. + */ tx_status = (rc == 0); + rc = (rc == -EAGAIN) ? 0 : rc; cfg80211_mgmt_tx_status(wdev, cookie ? *cookie : 0, buf, len, tx_status, GFP_KERNEL); @@ -1806,7 +1811,7 @@ void wil_cfg80211_ap_recovery(struct wil6210_priv *wil) int rc, i; struct wiphy *wiphy = wil_to_wiphy(wil); - for (i = 0; i < wil->max_vifs; i++) { + for (i = 0; i < GET_MAX_VIFS(wil); i++) { struct wil6210_vif *vif = wil->vifs[i]; struct net_device *ndev; struct cfg80211_beacon_data bcon = {}; @@ -2620,8 +2625,8 @@ static int wil_rf_sector_get_cfg(struct wiphy *wiphy, if (!test_bit(WMI_FW_CAPABILITY_RF_SECTORS, wil->fw_capabilities)) return -EOPNOTSUPP; - rc = nla_parse(tb, QCA_ATTR_DMG_RF_SECTOR_MAX, data, data_len, - wil_rf_sector_policy, NULL); + rc = nla_parse_deprecated(tb, QCA_ATTR_DMG_RF_SECTOR_MAX, data, + data_len, wil_rf_sector_policy, NULL); if (rc) { wil_err(wil, "Invalid rf sector ATTR\n"); return rc; @@ -2679,13 +2684,13 @@ static int wil_rf_sector_get_cfg(struct wiphy *wiphy, QCA_ATTR_PAD)) goto nla_put_failure; - nl_cfgs = nla_nest_start(msg, QCA_ATTR_DMG_RF_SECTOR_CFG); + nl_cfgs = nla_nest_start_noflag(msg, QCA_ATTR_DMG_RF_SECTOR_CFG); if (!nl_cfgs) goto nla_put_failure; for (i = 0; i < WMI_MAX_RF_MODULES_NUM; i++) { if (!(rf_modules_vec & BIT(i))) continue; - nl_cfg = nla_nest_start(msg, i); + nl_cfg = nla_nest_start_noflag(msg, i); if (!nl_cfg) goto nla_put_failure; si = &reply.evt.sectors_info[i]; @@ -2740,8 +2745,8 @@ static int wil_rf_sector_set_cfg(struct wiphy *wiphy, if (!test_bit(WMI_FW_CAPABILITY_RF_SECTORS, wil->fw_capabilities)) return -EOPNOTSUPP; - rc = nla_parse(tb, QCA_ATTR_DMG_RF_SECTOR_MAX, data, data_len, - wil_rf_sector_policy, NULL); + rc = nla_parse_deprecated(tb, QCA_ATTR_DMG_RF_SECTOR_MAX, data, + data_len, wil_rf_sector_policy, NULL); if (rc) { wil_err(wil, "Invalid rf sector ATTR\n"); return rc; @@ -2773,9 +2778,11 @@ static int wil_rf_sector_set_cfg(struct wiphy *wiphy, cmd.sector_type = sector_type; nla_for_each_nested(nl_cfg, tb[QCA_ATTR_DMG_RF_SECTOR_CFG], tmp) { - rc = nla_parse_nested(tb2, QCA_ATTR_DMG_RF_SECTOR_CFG_MAX, - nl_cfg, wil_rf_sector_cfg_policy, - NULL); + rc = nla_parse_nested_deprecated(tb2, + QCA_ATTR_DMG_RF_SECTOR_CFG_MAX, + nl_cfg, + wil_rf_sector_cfg_policy, + NULL); if (rc) { wil_err(wil, "invalid sector cfg\n"); return -EINVAL; @@ -2847,8 +2854,8 @@ static int wil_rf_sector_get_selected(struct wiphy *wiphy, if (!test_bit(WMI_FW_CAPABILITY_RF_SECTORS, wil->fw_capabilities)) return -EOPNOTSUPP; - rc = nla_parse(tb, QCA_ATTR_DMG_RF_SECTOR_MAX, data, data_len, - wil_rf_sector_policy, NULL); + rc = nla_parse_deprecated(tb, QCA_ATTR_DMG_RF_SECTOR_MAX, data, + data_len, wil_rf_sector_policy, NULL); if (rc) { wil_err(wil, "Invalid rf sector ATTR\n"); return rc; @@ -2955,8 +2962,8 @@ static int wil_rf_sector_set_selected(struct wiphy *wiphy, if (!test_bit(WMI_FW_CAPABILITY_RF_SECTORS, wil->fw_capabilities)) return -EOPNOTSUPP; - rc = nla_parse(tb, QCA_ATTR_DMG_RF_SECTOR_MAX, data, data_len, - wil_rf_sector_policy, NULL); + rc = nla_parse_deprecated(tb, QCA_ATTR_DMG_RF_SECTOR_MAX, data, + data_len, wil_rf_sector_policy, NULL); if (rc) { wil_err(wil, "Invalid rf sector ATTR\n"); return rc; diff --git a/drivers/net/wireless/ath/wil6210/debugfs.c b/drivers/net/wireless/ath/wil6210/debugfs.c index 7ad4e53284390c0abbf1b070c58acb14ebae250c..df2adff6c33a1e5603c136b540d7bf6a2ac03fcc 100644 --- a/drivers/net/wireless/ath/wil6210/debugfs.c +++ b/drivers/net/wireless/ath/wil6210/debugfs.c @@ -207,6 +207,8 @@ static void wil_print_sring(struct seq_file *s, struct wil6210_priv *wil, seq_puts(s, "???\n"); } seq_printf(s, " desc_rdy_pol = %d\n", sring->desc_rdy_pol); + seq_printf(s, " invalid_buff_id_cnt = %d\n", + sring->invalid_buff_id_cnt); if (sring->va && (sring->size <= (1 << WIL_RING_SIZE_ORDER_MAX))) { uint i; @@ -258,6 +260,11 @@ static void wil_print_mbox_ring(struct seq_file *s, const char *prefix, wil_halp_vote(wil); + if (wil_mem_access_lock(wil)) { + wil_halp_unvote(wil); + return; + } + wil_memcpy_fromio_32(&r, off, sizeof(r)); wil_mbox_ring_le2cpus(&r); /* @@ -323,6 +330,7 @@ static void wil_print_mbox_ring(struct seq_file *s, const char *prefix, } out: seq_puts(s, "}\n"); + wil_mem_access_unlock(wil); wil_halp_unvote(wil); } @@ -601,6 +609,12 @@ static int memread_show(struct seq_file *s, void *data) if (ret < 0) return ret; + ret = wil_mem_access_lock(wil); + if (ret) { + wil_pm_runtime_put(wil); + return ret; + } + a = wmi_buffer(wil, cpu_to_le32(mem_addr)); if (a) @@ -608,6 +622,7 @@ static int memread_show(struct seq_file *s, void *data) else seq_printf(s, "[0x%08x] = INVALID\n", mem_addr); + wil_mem_access_unlock(wil); wil_pm_runtime_put(wil); return 0; @@ -626,10 +641,6 @@ static ssize_t wil_read_file_ioblob(struct file *file, char __user *user_buf, size_t unaligned_bytes, aligned_count, ret; int rc; - if (test_bit(wil_status_suspending, wil_blob->wil->status) || - test_bit(wil_status_suspended, wil_blob->wil->status)) - return 0; - if (pos < 0) return -EINVAL; @@ -656,11 +667,19 @@ static ssize_t wil_read_file_ioblob(struct file *file, char __user *user_buf, return rc; } + rc = wil_mem_access_lock(wil); + if (rc) { + kfree(buf); + wil_pm_runtime_put(wil); + return rc; + } + wil_memcpy_fromio_32(buf, (const void __iomem *) wil_blob->blob.data + aligned_pos, aligned_count); ret = copy_to_user(user_buf, buf + unaligned_bytes, count); + wil_mem_access_unlock(wil); wil_pm_runtime_put(wil); kfree(buf); @@ -1364,7 +1383,7 @@ static int link_show(struct seq_file *s, void *data) if (p->status != wil_sta_connected) continue; - vif = (mid < wil->max_vifs) ? wil->vifs[mid] : NULL; + vif = (mid < GET_MAX_VIFS(wil)) ? wil->vifs[mid] : NULL; if (vif) { rc = wil_cid_fill_sinfo(vif, i, sinfo); if (rc) @@ -1562,7 +1581,7 @@ __acquires(&p->tid_rx_lock) __releases(&p->tid_rx_lock) break; } mid = (p->status != wil_sta_unused) ? p->mid : U8_MAX; - if (mid < wil->max_vifs) { + if (mid < GET_MAX_VIFS(wil)) { struct wil6210_vif *vif = wil->vifs[mid]; if (vif->wdev.iftype == NL80211_IFTYPE_STATION && @@ -1628,7 +1647,7 @@ static int mids_show(struct seq_file *s, void *data) int i; mutex_lock(&wil->vif_mutex); - for (i = 0; i < wil->max_vifs; i++) { + for (i = 0; i < GET_MAX_VIFS(wil); i++) { vif = wil->vifs[i]; if (vif) { @@ -1849,7 +1868,7 @@ static int wil_link_stats_debugfs_show(struct seq_file *s, void *data) /* iterate over all MIDs and show per-cid statistics. Then show the * global statistics */ - for (i = 0; i < wil->max_vifs; i++) { + for (i = 0; i < GET_MAX_VIFS(wil); i++) { vif = wil->vifs[i]; seq_printf(s, "MID %d ", i); @@ -1905,7 +1924,7 @@ static ssize_t wil_link_stats_write(struct file *file, const char __user *buf, if (rc) return rc; - for (i = 0; i < wil->max_vifs; i++) { + for (i = 0; i < GET_MAX_VIFS(wil); i++) { vif = wil->vifs[i]; if (!vif) continue; @@ -2375,6 +2394,7 @@ static const struct dbg_off dbg_wil_regs[] = { {"RGF_MAC_MTRL_COUNTER_0", 0444, HOSTADDR(RGF_MAC_MTRL_COUNTER_0), doff_io32}, {"RGF_USER_USAGE_1", 0444, HOSTADDR(RGF_USER_USAGE_1), doff_io32}, + {"RGF_USER_USAGE_2", 0444, HOSTADDR(RGF_USER_USAGE_2), doff_io32}, {}, }; diff --git a/drivers/net/wireless/ath/wil6210/fw_inc.c b/drivers/net/wireless/ath/wil6210/fw_inc.c index 388b3d4717cae06f0bcd60f4004fe7224674c24f..3ec0f2fab9b7cb0e585ad70596f9dbe7efa2c10f 100644 --- a/drivers/net/wireless/ath/wil6210/fw_inc.c +++ b/drivers/net/wireless/ath/wil6210/fw_inc.c @@ -1,6 +1,6 @@ /* * Copyright (c) 2014-2017 Qualcomm Atheros, Inc. - * Copyright (c) 2018, The Linux Foundation. All rights reserved. + * Copyright (c) 2018-2019, The Linux Foundation. All rights reserved. * * Permission to use, copy, modify, and/or distribute this software for any * purpose with or without fee is hereby granted, provided that the above @@ -647,6 +647,8 @@ int wil_request_firmware(struct wil6210_priv *wil, const char *name, out: release_firmware(fw); + if (rc) + wil_err_fw(wil, "Loading <%s> failed, rc %d\n", name, rc); return rc; } @@ -741,6 +743,8 @@ int wil_request_board(struct wil6210_priv *wil, const char *name) out: release_firmware(brd); + if (rc) + wil_err_fw(wil, "Loading <%s> failed, rc %d\n", name, rc); return rc; } diff --git a/drivers/net/wireless/ath/wil6210/main.c b/drivers/net/wireless/ath/wil6210/main.c index 277abfdf3322c5b8a59b08093a7a9219a9105514..9b9c9ec015362a5587700a3cf382fc783e6c832b 100644 --- a/drivers/net/wireless/ath/wil6210/main.c +++ b/drivers/net/wireless/ath/wil6210/main.c @@ -184,6 +184,28 @@ void wil_memcpy_toio_32(volatile void __iomem *dst, const void *src, } } +/* Device memory access is prohibited while reset or suspend. + * wil_mem_access_lock protects accessing device memory in these cases + */ +int wil_mem_access_lock(struct wil6210_priv *wil) +{ + if (!down_read_trylock(&wil->mem_lock)) + return -EBUSY; + + if (test_bit(wil_status_suspending, wil->status) || + test_bit(wil_status_suspended, wil->status)) { + up_read(&wil->mem_lock); + return -EBUSY; + } + + return 0; +} + +void wil_mem_access_unlock(struct wil6210_priv *wil) +{ + up_read(&wil->mem_lock); +} + static void wil_ring_fini_tx(struct wil6210_priv *wil, int id) { struct wil_ring *ring = &wil->ring_tx[id]; @@ -663,7 +685,7 @@ void wil_bcast_fini_all(struct wil6210_priv *wil) int i; struct wil6210_vif *vif; - for (i = 0; i < wil->max_vifs; i++) { + for (i = 0; i < GET_MAX_VIFS(wil); i++) { vif = wil->vifs[i]; if (vif) wil_bcast_fini(vif); @@ -703,6 +725,7 @@ int wil_priv_init(struct wil6210_priv *wil) spin_lock_init(&wil->wmi_ev_lock); spin_lock_init(&wil->net_queue_lock); init_waitqueue_head(&wil->wq); + init_rwsem(&wil->mem_lock); wil->wmi_wq = create_singlethread_workqueue(WIL_NAME "_wmi"); if (!wil->wmi_wq) @@ -1390,13 +1413,22 @@ static int wil_get_otp_info(struct wil6210_priv *wil) u8 mac[8]; int mac_addr; - if (wil->hw_version >= HW_VER_TALYN_MB) - mac_addr = RGF_OTP_MAC_TALYN_MB; - else - mac_addr = RGF_OTP_MAC; + /* OEM MAC has precedence */ + mac_addr = RGF_OTP_OEM_MAC; + wil_memcpy_fromio_32(mac, wil->csr + HOSTADDR(mac_addr), sizeof(mac)); + + if (is_valid_ether_addr(mac)) { + wil_info(wil, "using OEM MAC %pM\n", mac); + } else { + if (wil->hw_version >= HW_VER_TALYN_MB) + mac_addr = RGF_OTP_MAC_TALYN_MB; + else + mac_addr = RGF_OTP_MAC; + + wil_memcpy_fromio_32(mac, wil->csr + HOSTADDR(mac_addr), + sizeof(mac)); + } - wil_memcpy_fromio_32(mac, wil->csr + HOSTADDR(mac_addr), - sizeof(mac)); if (!is_valid_ether_addr(mac)) { wil_err(wil, "Invalid MAC %pM\n", mac); return -EINVAL; @@ -1460,7 +1492,7 @@ void wil_abort_scan_all_vifs(struct wil6210_priv *wil, bool sync) lockdep_assert_held(&wil->vif_mutex); - for (i = 0; i < wil->max_vifs; i++) { + for (i = 0; i < GET_MAX_VIFS(wil); i++) { struct wil6210_vif *vif = wil->vifs[i]; if (vif) @@ -1500,11 +1532,6 @@ static void wil_pre_fw_config(struct wil6210_priv *wil) if (wil->hw_version < HW_VER_TALYN_MB) { wil_s(wil, RGF_CAF_ICR + offsetof(struct RGF_ICR, ICR), 0); wil_w(wil, RGF_CAF_ICR + offsetof(struct RGF_ICR, IMV), ~0); - } else { - wil_s(wil, - RGF_CAF_ICR_TALYN_MB + offsetof(struct RGF_ICR, ICR), 0); - wil_w(wil, RGF_CAF_ICR_TALYN_MB + - offsetof(struct RGF_ICR, IMV), ~0); } /* clear PAL_UNIT_ICR (potential D0->D3 leftover) * In Talyn-MB host cannot access this register due to @@ -1528,7 +1555,7 @@ static int wil_restore_vifs(struct wil6210_priv *wil) struct wireless_dev *wdev; int i, rc; - for (i = 0; i < wil->max_vifs; i++) { + for (i = 0; i < GET_MAX_VIFS(wil); i++) { vif = wil->vifs[i]; if (!vif) continue; @@ -1580,7 +1607,8 @@ int wil_reset(struct wil6210_priv *wil, bool load_fw) if (wil->hw_version == HW_VER_UNKNOWN) return -ENODEV; - if (test_bit(WIL_PLATFORM_CAPA_T_PWR_ON_0, wil->platform_capa)) { + if (test_bit(WIL_PLATFORM_CAPA_T_PWR_ON_0, wil->platform_capa) && + wil->hw_version < HW_VER_TALYN_MB) { wil_dbg_misc(wil, "Notify FW to set T_POWER_ON=0\n"); wil_s(wil, RGF_USER_USAGE_8, BIT_USER_SUPPORT_T_POWER_ON_0); } @@ -1599,20 +1627,11 @@ int wil_reset(struct wil6210_priv *wil, bool load_fw) } set_bit(wil_status_resetting, wil->status); - if (test_bit(wil_status_collecting_dumps, wil->status)) { - /* Device collects crash dump, cancel the reset. - * following crash dump collection, reset would take place. - */ - wil_dbg_misc(wil, "reject reset while collecting crash dump\n"); - rc = -EBUSY; - goto out; - } - mutex_lock(&wil->vif_mutex); wil_abort_scan_all_vifs(wil, false); mutex_unlock(&wil->vif_mutex); - for (i = 0; i < wil->max_vifs; i++) { + for (i = 0; i < GET_MAX_VIFS(wil); i++) { vif = wil->vifs[i]; if (vif) { cancel_work_sync(&vif->disconnect_worker); @@ -1782,7 +1801,9 @@ int __wil_up(struct wil6210_priv *wil) WARN_ON(!mutex_is_locked(&wil->mutex)); + down_write(&wil->mem_lock); rc = wil_reset(wil, true); + up_write(&wil->mem_lock); if (rc) return rc; @@ -1854,6 +1875,7 @@ int wil_up(struct wil6210_priv *wil) int __wil_down(struct wil6210_priv *wil) { + int rc; WARN_ON(!mutex_is_locked(&wil->mutex)); set_bit(wil_status_resetting, wil->status); @@ -1873,7 +1895,11 @@ int __wil_down(struct wil6210_priv *wil) wil_abort_scan_all_vifs(wil, false); mutex_unlock(&wil->vif_mutex); - return wil_reset(wil, false); + down_write(&wil->mem_lock); + rc = wil_reset(wil, false); + up_write(&wil->mem_lock); + + return rc; } int wil_down(struct wil6210_priv *wil) diff --git a/drivers/net/wireless/ath/wil6210/netdev.c b/drivers/net/wireless/ath/wil6210/netdev.c index b4e0eb1585b9871364d06855f389767d064e541a..59f041d708fe5c6c4c70ea0de4f022880dc5cbf9 100644 --- a/drivers/net/wireless/ath/wil6210/netdev.c +++ b/drivers/net/wireless/ath/wil6210/netdev.c @@ -1,6 +1,6 @@ /* * Copyright (c) 2012-2017 Qualcomm Atheros, Inc. - * Copyright (c) 2018, The Linux Foundation. All rights reserved. + * Copyright (c) 2018-2019, The Linux Foundation. All rights reserved. * * Permission to use, copy, modify, and/or distribute this software for any * purpose with or without fee is hereby granted, provided that the above @@ -27,7 +27,7 @@ bool wil_has_other_active_ifaces(struct wil6210_priv *wil, struct wil6210_vif *vif; struct net_device *ndev_i; - for (i = 0; i < wil->max_vifs; i++) { + for (i = 0; i < GET_MAX_VIFS(wil); i++) { vif = wil->vifs[i]; if (vif) { ndev_i = vif_to_ndev(vif); @@ -155,7 +155,7 @@ static int wil6210_netdev_poll_tx(struct napi_struct *napi, int budget) struct wil6210_vif *vif; if (!ring->va || !txdata->enabled || - txdata->mid >= wil->max_vifs) + txdata->mid >= GET_MAX_VIFS(wil)) continue; vif = wil->vifs[txdata->mid]; @@ -294,7 +294,7 @@ static u8 wil_vif_find_free_mid(struct wil6210_priv *wil) { u8 i; - for (i = 0; i < wil->max_vifs; i++) { + for (i = 0; i < GET_MAX_VIFS(wil); i++) { if (!wil->vifs[i]) return i; } @@ -500,7 +500,7 @@ void wil_vif_remove(struct wil6210_priv *wil, u8 mid) bool any_active = wil_has_active_ifaces(wil, true, false); ASSERT_RTNL(); - if (mid >= wil->max_vifs) { + if (mid >= GET_MAX_VIFS(wil)) { wil_err(wil, "invalid MID: %d\n", mid); return; } diff --git a/drivers/net/wireless/ath/wil6210/pcie_bus.c b/drivers/net/wireless/ath/wil6210/pcie_bus.c index c8c6613371d1bcbd2d0a8e079053bc08b2c41d89..3b82d6cfc218b7428299a59ccb34e377de172d71 100644 --- a/drivers/net/wireless/ath/wil6210/pcie_bus.c +++ b/drivers/net/wireless/ath/wil6210/pcie_bus.c @@ -1,6 +1,6 @@ /* * Copyright (c) 2012-2017 Qualcomm Atheros, Inc. - * Copyright (c) 2018, The Linux Foundation. All rights reserved. + * Copyright (c) 2018-2019, The Linux Foundation. All rights reserved. * * Permission to use, copy, modify, and/or distribute this software for any * purpose with or without fee is hereby granted, provided that the above @@ -176,7 +176,7 @@ static void wil_remove_all_additional_vifs(struct wil6210_priv *wil) struct wil6210_vif *vif; int i; - for (i = 1; i < wil->max_vifs; i++) { + for (i = 1; i < GET_MAX_VIFS(wil); i++) { vif = wil->vifs[i]; if (vif) { wil_vif_prepare_stop(vif); diff --git a/drivers/net/wireless/ath/wil6210/pm.c b/drivers/net/wireless/ath/wil6210/pm.c index 75fe9323547ca58d53475bd27bebf262be6e8e93..56143e7670ed93e6ead364b87d6555054ec76588 100644 --- a/drivers/net/wireless/ath/wil6210/pm.c +++ b/drivers/net/wireless/ath/wil6210/pm.c @@ -1,6 +1,6 @@ /* * Copyright (c) 2014,2017 Qualcomm Atheros, Inc. - * Copyright (c) 2018, The Linux Foundation. All rights reserved. + * Copyright (c) 2018-2019, The Linux Foundation. All rights reserved. * * Permission to use, copy, modify, and/or distribute this software for any * purpose with or without fee is hereby granted, provided that the above @@ -26,7 +26,7 @@ static void wil_pm_wake_connected_net_queues(struct wil6210_priv *wil) int i; mutex_lock(&wil->vif_mutex); - for (i = 0; i < wil->max_vifs; i++) { + for (i = 0; i < GET_MAX_VIFS(wil); i++) { struct wil6210_vif *vif = wil->vifs[i]; if (vif && test_bit(wil_vif_fwconnected, vif->status)) @@ -40,7 +40,7 @@ static void wil_pm_stop_all_net_queues(struct wil6210_priv *wil) int i; mutex_lock(&wil->vif_mutex); - for (i = 0; i < wil->max_vifs; i++) { + for (i = 0; i < GET_MAX_VIFS(wil); i++) { struct wil6210_vif *vif = wil->vifs[i]; if (vif) @@ -123,7 +123,7 @@ int wil_can_suspend(struct wil6210_priv *wil, bool is_runtime) /* interface is running */ mutex_lock(&wil->vif_mutex); - for (i = 0; i < wil->max_vifs; i++) { + for (i = 0; i < GET_MAX_VIFS(wil); i++) { struct wil6210_vif *vif = wil->vifs[i]; if (!vif) @@ -195,14 +195,18 @@ static int wil_suspend_keep_radio_on(struct wil6210_priv *wil) wil_dbg_pm(wil, "suspend keep radio on\n"); /* Prevent handling of new tx and wmi commands */ - set_bit(wil_status_suspending, wil->status); - if (test_bit(wil_status_collecting_dumps, wil->status)) { - /* Device collects crash dump, cancel the suspend */ - wil_dbg_pm(wil, "reject suspend while collecting crash dump\n"); - clear_bit(wil_status_suspending, wil->status); + rc = down_write_trylock(&wil->mem_lock); + if (!rc) { + wil_err(wil, + "device is busy. down_write_trylock failed, returned (0x%x)\n", + rc); wil->suspend_stats.rejected_by_host++; return -EBUSY; } + + set_bit(wil_status_suspending, wil->status); + up_write(&wil->mem_lock); + wil_pm_stop_all_net_queues(wil); if (!wil_is_tx_idle(wil)) { @@ -310,15 +314,18 @@ static int wil_suspend_radio_off(struct wil6210_priv *wil) wil_dbg_pm(wil, "suspend radio off\n"); - set_bit(wil_status_suspending, wil->status); - if (test_bit(wil_status_collecting_dumps, wil->status)) { - /* Device collects crash dump, cancel the suspend */ - wil_dbg_pm(wil, "reject suspend while collecting crash dump\n"); - clear_bit(wil_status_suspending, wil->status); + rc = down_write_trylock(&wil->mem_lock); + if (!rc) { + wil_err(wil, + "device is busy. down_write_trylock failed, returned (0x%x)\n", + rc); wil->suspend_stats.rejected_by_host++; return -EBUSY; } + set_bit(wil_status_suspending, wil->status); + up_write(&wil->mem_lock); + /* if netif up, hardware is alive, shut it down */ mutex_lock(&wil->vif_mutex); active_ifaces = wil_has_active_ifaces(wil, true, false); diff --git a/drivers/net/wireless/ath/wil6210/txrx_edma.c b/drivers/net/wireless/ath/wil6210/txrx_edma.c index c38773878ae3aaf98c5dad56667c0eb6cc65f57b..f6fce6ff73d9038fa3bbd30391e4cfddf9ead29f 100644 --- a/drivers/net/wireless/ath/wil6210/txrx_edma.c +++ b/drivers/net/wireless/ath/wil6210/txrx_edma.c @@ -29,6 +29,7 @@ #define WIL_EDMA_MAX_DATA_OFFSET (2) /* RX buffer size must be aligned to 4 bytes */ #define WIL_EDMA_RX_BUF_LEN_DEFAULT (2048) +#define MAX_INVALID_BUFF_ID_RETRY (3) static void wil_tx_desc_unmap_edma(struct device *dev, union wil_tx_desc *desc, @@ -312,7 +313,8 @@ static int wil_init_rx_buff_arr(struct wil6210_priv *wil, struct list_head *free = &wil->rx_buff_mgmt.free; int i; - wil->rx_buff_mgmt.buff_arr = kcalloc(size, sizeof(struct wil_rx_buff), + wil->rx_buff_mgmt.buff_arr = kcalloc(size + 1, + sizeof(struct wil_rx_buff), GFP_KERNEL); if (!wil->rx_buff_mgmt.buff_arr) return -ENOMEM; @@ -321,14 +323,16 @@ static int wil_init_rx_buff_arr(struct wil6210_priv *wil, INIT_LIST_HEAD(active); INIT_LIST_HEAD(free); - /* Linkify the list */ + /* Linkify the list. + * buffer id 0 should not be used (marks invalid id). + */ buff_arr = wil->rx_buff_mgmt.buff_arr; - for (i = 0; i < size; i++) { + for (i = 1; i <= size; i++) { list_add(&buff_arr[i].list, free); buff_arr[i].id = i; } - wil->rx_buff_mgmt.size = size; + wil->rx_buff_mgmt.size = size + 1; return 0; } @@ -428,6 +432,9 @@ static void wil_ring_free_edma(struct wil6210_priv *wil, struct wil_ring *ring) &ring->pa, ring->ctx); wil_move_all_rx_buff_to_free_list(wil, ring); + dma_free_coherent(dev, sizeof(*ring->edma_rx_swtail.va), + ring->edma_rx_swtail.va, + ring->edma_rx_swtail.pa); goto out; } @@ -804,18 +811,9 @@ static int wil_rx_error_check_edma(struct wil6210_priv *wil, struct sk_buff *skb, struct wil_net_stats *stats) { - int error; int l2_rx_status; - int l3_rx_status; - int l4_rx_status; void *msg = wil_skb_rxstatus(skb); - error = wil_rx_status_get_error(msg); - if (!error) { - skb->ip_summed = CHECKSUM_UNNECESSARY; - return 0; - } - l2_rx_status = wil_rx_status_get_l2_rx_status(msg); if (l2_rx_status != 0) { wil_dbg_txrx(wil, "L2 RX error, l2_rx_status=0x%x\n", @@ -844,17 +842,7 @@ static int wil_rx_error_check_edma(struct wil6210_priv *wil, return -EFAULT; } - l3_rx_status = wil_rx_status_get_l3_rx_status(msg); - l4_rx_status = wil_rx_status_get_l4_rx_status(msg); - if (!l3_rx_status && !l4_rx_status) - skb->ip_summed = CHECKSUM_UNNECESSARY; - /* If HW reports bad checksum, let IP stack re-check it - * For example, HW don't understand Microsoft IP stack that - * mis-calculates TCP checksum - if it should be 0x0, - * it writes 0xffff in violation of RFC 1624 - */ - else - stats->rx_csum_err++; + skb->ip_summed = wil_rx_status_get_checksum(msg, stats); return 0; } @@ -892,26 +880,50 @@ again: /* Extract the buffer ID from the status message */ buff_id = le16_to_cpu(wil_rx_status_get_buff_id(msg)); - if (unlikely(!wil_val_in_range(buff_id, 0, wil->rx_buff_mgmt.size))) { + + while (!buff_id) { + struct wil_rx_status_extended *s; + int invalid_buff_id_retry = 0; + + wil_dbg_txrx(wil, + "buff_id is not updated yet by HW, (swhead 0x%x)\n", + sring->swhead); + if (++invalid_buff_id_retry > MAX_INVALID_BUFF_ID_RETRY) + break; + + /* Read the status message again */ + s = (struct wil_rx_status_extended *) + (sring->va + (sring->elem_size * sring->swhead)); + *(struct wil_rx_status_extended *)msg = *s; + buff_id = le16_to_cpu(wil_rx_status_get_buff_id(msg)); + } + + if (unlikely(!wil_val_in_range(buff_id, 1, wil->rx_buff_mgmt.size))) { wil_err(wil, "Corrupt buff_id=%d, sring->swhead=%d\n", buff_id, sring->swhead); + wil_rx_status_reset_buff_id(sring); wil_sring_advance_swhead(sring); + sring->invalid_buff_id_cnt++; goto again; } - wil_sring_advance_swhead(sring); - /* Extract the SKB from the rx_buff management array */ skb = wil->rx_buff_mgmt.buff_arr[buff_id].skb; wil->rx_buff_mgmt.buff_arr[buff_id].skb = NULL; if (!skb) { wil_err(wil, "No Rx skb at buff_id %d\n", buff_id); + wil_rx_status_reset_buff_id(sring); /* Move the buffer from the active list to the free list */ - list_move(&wil->rx_buff_mgmt.buff_arr[buff_id].list, - &wil->rx_buff_mgmt.free); + list_move_tail(&wil->rx_buff_mgmt.buff_arr[buff_id].list, + &wil->rx_buff_mgmt.free); + wil_sring_advance_swhead(sring); + sring->invalid_buff_id_cnt++; goto again; } + wil_rx_status_reset_buff_id(sring); + wil_sring_advance_swhead(sring); + memcpy(&pa, skb->cb, sizeof(pa)); dma_unmap_single(dev, pa, sz, DMA_FROM_DEVICE); dmalen = le16_to_cpu(wil_rx_status_get_length(msg)); @@ -926,8 +938,8 @@ again: sizeof(struct wil_rx_status_extended), false); /* Move the buffer from the active list to the free list */ - list_move(&wil->rx_buff_mgmt.buff_arr[buff_id].list, - &wil->rx_buff_mgmt.free); + list_move_tail(&wil->rx_buff_mgmt.buff_arr[buff_id].list, + &wil->rx_buff_mgmt.free); eop = wil_rx_status_get_eop(msg); diff --git a/drivers/net/wireless/ath/wil6210/txrx_edma.h b/drivers/net/wireless/ath/wil6210/txrx_edma.h index 343516a03a1e4098dc9c561845c358b25e3c460f..bb4ff28b73e5fd5aa6f8e10e948af65ad454481b 100644 --- a/drivers/net/wireless/ath/wil6210/txrx_edma.h +++ b/drivers/net/wireless/ath/wil6210/txrx_edma.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2012-2016,2018, The Linux Foundation. All rights reserved. + * Copyright (c) 2012-2016,2018-2019, The Linux Foundation. All rights reserved. * * Permission to use, copy, modify, and/or distribute this software for any * purpose with or without fee is hereby granted, provided that the above @@ -427,6 +427,12 @@ static inline int wil_rx_status_get_eop(void *msg) /* EoP = End of Packet */ 30, 30); } +static inline void wil_rx_status_reset_buff_id(struct wil_status_ring *s) +{ + ((struct wil_rx_status_compressed *) + (s->va + (s->elem_size * s->swhead)))->buff_id = 0; +} + static inline __le16 wil_rx_status_get_buff_id(void *msg) { return ((struct wil_rx_status_compressed *)msg)->buff_id; @@ -511,6 +517,45 @@ static inline int wil_rx_status_get_l4_rx_status(void *msg) 5, 6); } +/* L4 L3 Expected result + * 0 0 Ok. No L3 and no L4 known protocols found. + * Treated as L2 packet. (no offloads on this packet) + * 0 1 Ok. It means that L3 was found, and checksum check passed. + * No known L4 protocol was found. + * 0 2 It means that L3 protocol was found, and checksum check failed. + * No L4 known protocol was found. + * 1 any Ok. It means that L4 was found, and checksum check passed. + * 3 0 Not a possible scenario. + * 3 1 Recalculate. It means that L3 protocol was found, and checksum + * passed. But L4 checksum failed. Need to see if really failed, + * or due to fragmentation. + * 3 2 Both L3 and L4 checksum check failed. + */ +static inline int wil_rx_status_get_checksum(void *msg, + struct wil_net_stats *stats) +{ + int l3_rx_status = wil_rx_status_get_l3_rx_status(msg); + int l4_rx_status = wil_rx_status_get_l4_rx_status(msg); + + if (l4_rx_status == 1) + return CHECKSUM_UNNECESSARY; + + if (l4_rx_status == 0 && l3_rx_status == 1) + return CHECKSUM_UNNECESSARY; + + if (l3_rx_status == 0 && l4_rx_status == 0) + /* L2 packet */ + return CHECKSUM_NONE; + + /* If HW reports bad checksum, let IP stack re-check it + * For example, HW doesn't understand Microsoft IP stack that + * mis-calculates TCP checksum - if it should be 0x0, + * it writes 0xffff in violation of RFC 1624 + */ + stats->rx_csum_err++; + return CHECKSUM_NONE; +} + static inline int wil_rx_status_get_security(void *msg) { return WIL_GET_BITS(((struct wil_rx_status_compressed *)msg)->d0, diff --git a/drivers/net/wireless/ath/wil6210/wil6210.h b/drivers/net/wireless/ath/wil6210/wil6210.h index e1b1039b13abd009297abf5e841d1c6f825e759b..8724d9975606b3575610b52375e4202d4e7eb60a 100644 --- a/drivers/net/wireless/ath/wil6210/wil6210.h +++ b/drivers/net/wireless/ath/wil6210/wil6210.h @@ -185,6 +185,7 @@ struct RGF_ICR { /* registers - FW addresses */ #define RGF_USER_USAGE_1 (0x880004) +#define RGF_USER_USAGE_2 (0x880008) #define RGF_USER_USAGE_6 (0x880018) #define BIT_USER_OOB_MODE BIT(31) #define BIT_USER_OOB_R2_MODE BIT(30) @@ -367,6 +368,7 @@ struct RGF_ICR { #define REVISION_ID_SPARROW_D0 (0x3) #define RGF_OTP_MAC_TALYN_MB (0x8a0304) +#define RGF_OTP_OEM_MAC (0x8a0334) #define RGF_OTP_MAC (0x8a0620) /* Talyn-MB */ @@ -566,10 +568,11 @@ struct wil_status_ring { bool is_rx; u8 desc_rdy_pol; /* Expected descriptor ready bit polarity */ struct wil_ring_rx_data rx_data; + u32 invalid_buff_id_cnt; /* relevant only for RX */ }; #define WIL_STA_TID_NUM (16) -#define WIL_MCS_MAX (12) /* Maximum MCS supported */ +#define WIL_MCS_MAX (15) /* Maximum MCS supported */ struct wil_net_stats { unsigned long rx_packets; @@ -660,7 +663,6 @@ enum { /* for wil6210_priv.status */ wil_status_suspending, /* suspend in progress */ wil_status_suspended, /* suspend completed, device is suspended */ wil_status_resuming, /* resume in progress */ - wil_status_collecting_dumps, /* crashdump collection in progress */ wil_status_last /* keep last */ }; @@ -992,6 +994,8 @@ struct wil6210_priv { struct wil_txrx_ops txrx_ops; struct mutex mutex; /* for wil6210_priv access in wil_{up|down} */ + /* for synchronizing device memory access while reset or suspend */ + struct rw_semaphore mem_lock; /* statistics */ atomic_t isr_count_rx, isr_count_tx; /* debugfs */ @@ -1060,6 +1064,7 @@ struct wil6210_priv { #define vif_to_wil(v) (v->wil) #define vif_to_ndev(v) (v->ndev) #define vif_to_wdev(v) (&v->wdev) +#define GET_MAX_VIFS(wil) min_t(int, (wil)->max_vifs, WIL_MAX_VIFS) static inline struct wil6210_vif *wdev_to_vif(struct wil6210_priv *wil, struct wireless_dev *wdev) @@ -1176,6 +1181,8 @@ void wil_memcpy_fromio_32(void *dst, const volatile void __iomem *src, size_t count); void wil_memcpy_toio_32(volatile void __iomem *dst, const void *src, size_t count); +int wil_mem_access_lock(struct wil6210_priv *wil); +void wil_mem_access_unlock(struct wil6210_priv *wil); struct wil6210_vif * wil_vif_alloc(struct wil6210_priv *wil, const char *name, diff --git a/drivers/net/wireless/ath/wil6210/wil_crash_dump.c b/drivers/net/wireless/ath/wil6210/wil_crash_dump.c index dc33a0b4c3fac55c67371aa7ef762a2c90320636..772cb00c20025fa50b5663cc3079edc84ee9a673 100644 --- a/drivers/net/wireless/ath/wil6210/wil_crash_dump.c +++ b/drivers/net/wireless/ath/wil6210/wil_crash_dump.c @@ -1,6 +1,6 @@ /* * Copyright (c) 2015,2017 Qualcomm Atheros, Inc. - * Copyright (c) 2018, The Linux Foundation. All rights reserved. + * Copyright (c) 2018-2019, The Linux Foundation. All rights reserved. * * Permission to use, copy, modify, and/or distribute this software for any * purpose with or without fee is hereby granted, provided that the above @@ -57,7 +57,7 @@ static int wil_fw_get_crash_dump_bounds(struct wil6210_priv *wil, int wil_fw_copy_crash_dump(struct wil6210_priv *wil, void *dest, u32 size) { - int i; + int i, rc; const struct fw_map *map; void *data; u32 host_min, dump_size, offset, len; @@ -73,14 +73,9 @@ int wil_fw_copy_crash_dump(struct wil6210_priv *wil, void *dest, u32 size) return -EINVAL; } - set_bit(wil_status_collecting_dumps, wil->status); - if (test_bit(wil_status_suspending, wil->status) || - test_bit(wil_status_suspended, wil->status) || - test_bit(wil_status_resetting, wil->status)) { - wil_err(wil, "cannot collect fw dump during suspend/reset\n"); - clear_bit(wil_status_collecting_dumps, wil->status); - return -EINVAL; - } + rc = wil_mem_access_lock(wil); + if (rc) + return rc; /* copy to crash dump area */ for (i = 0; i < ARRAY_SIZE(fw_mapping); i++) { @@ -100,8 +95,7 @@ int wil_fw_copy_crash_dump(struct wil6210_priv *wil, void *dest, u32 size) wil_memcpy_fromio_32((void * __force)(dest + offset), (const void __iomem * __force)data, len); } - - clear_bit(wil_status_collecting_dumps, wil->status); + wil_mem_access_unlock(wil); return 0; } diff --git a/drivers/net/wireless/ath/wil6210/wmi.c b/drivers/net/wireless/ath/wil6210/wmi.c index bda4a9712f91fb10e29db1b5e98cbe7ae215033a..d89cd41e78aca635b57f6bf769cf3e48088aa0e5 100644 --- a/drivers/net/wireless/ath/wil6210/wmi.c +++ b/drivers/net/wireless/ath/wil6210/wmi.c @@ -41,6 +41,7 @@ MODULE_PARM_DESC(led_id, #define WIL_WAIT_FOR_SUSPEND_RESUME_COMP 200 #define WIL_WMI_CALL_GENERAL_TO_MS 100 +#define WIL_WMI_PCP_STOP_TO_MS 5000 /** * WMI event receiving - theory of operations @@ -2195,7 +2196,8 @@ int wmi_pcp_stop(struct wil6210_vif *vif) return rc; return wmi_call(wil, WMI_PCP_STOP_CMDID, vif->mid, NULL, 0, - WMI_PCP_STOPPED_EVENTID, NULL, 0, 20); + WMI_PCP_STOPPED_EVENTID, NULL, 0, + WIL_WMI_PCP_STOP_TO_MS); } int wmi_set_ssid(struct wil6210_vif *vif, u8 ssid_len, const void *ssid) @@ -2957,6 +2959,10 @@ static const char *suspend_status2name(u8 status) switch (status) { case WMI_TRAFFIC_SUSPEND_REJECTED_LINK_NOT_IDLE: return "LINK_NOT_IDLE"; + case WMI_TRAFFIC_SUSPEND_REJECTED_DISCONNECT: + return "DISCONNECT"; + case WMI_TRAFFIC_SUSPEND_REJECTED_OTHER: + return "OTHER"; default: return "Untracked status"; } @@ -3046,6 +3052,9 @@ static void resume_triggers2string(u32 triggers, char *string, int str_size) if (triggers & WMI_RESUME_TRIGGER_WMI_EVT) strlcat(string, " WMI_EVT", str_size); + + if (triggers & WMI_RESUME_TRIGGER_DISCONNECT) + strlcat(string, " DISCONNECT", str_size); } int wmi_resume(struct wil6210_priv *wil) @@ -3196,7 +3205,7 @@ static void wmi_event_handle(struct wil6210_priv *wil, if (mid == MID_BROADCAST) mid = 0; - if (mid >= ARRAY_SIZE(wil->vifs) || mid >= wil->max_vifs) { + if (mid >= GET_MAX_VIFS(wil)) { wil_dbg_wmi(wil, "invalid mid %d, event skipped\n", mid); return; @@ -3502,8 +3511,9 @@ int wmi_mgmt_tx(struct wil6210_vif *vif, const u8 *buf, size_t len) rc = wmi_call(wil, WMI_SW_TX_REQ_CMDID, vif->mid, cmd, total, WMI_SW_TX_COMPLETE_EVENTID, &evt, sizeof(evt), 2000); if (!rc && evt.evt.status != WMI_FW_STATUS_SUCCESS) { - wil_err(wil, "mgmt_tx failed with status %d\n", evt.evt.status); - rc = -EINVAL; + wil_dbg_wmi(wil, "mgmt_tx failed with status %d\n", + evt.evt.status); + rc = -EAGAIN; } kfree(cmd); @@ -3555,9 +3565,9 @@ int wmi_mgmt_tx_ext(struct wil6210_vif *vif, const u8 *buf, size_t len, rc = wmi_call(wil, WMI_SW_TX_REQ_EXT_CMDID, vif->mid, cmd, total, WMI_SW_TX_COMPLETE_EVENTID, &evt, sizeof(evt), 2000); if (!rc && evt.evt.status != WMI_FW_STATUS_SUCCESS) { - wil_err(wil, "mgmt_tx_ext failed with status %d\n", - evt.evt.status); - rc = -EINVAL; + wil_dbg_wmi(wil, "mgmt_tx_ext failed with status %d\n", + evt.evt.status); + rc = -EAGAIN; } kfree(cmd); diff --git a/drivers/net/wireless/ath/wil6210/wmi.h b/drivers/net/wireless/ath/wil6210/wmi.h index b668758da994f8d66493615039efe0d4d38a12d0..da46fc8d39cfd0c0575f81de9a9ad5cb64645c82 100644 --- a/drivers/net/wireless/ath/wil6210/wmi.h +++ b/drivers/net/wireless/ath/wil6210/wmi.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2018, The Linux Foundation. All rights reserved. + * Copyright (c) 2018-2019, The Linux Foundation. All rights reserved. * Copyright (c) 2012-2017 Qualcomm Atheros, Inc. * Copyright (c) 2006-2012 Wilocity * @@ -104,6 +104,7 @@ enum wmi_fw_capability { WMI_FW_CAPABILITY_RAW_MODE = 24, WMI_FW_CAPABILITY_TX_REQ_EXT = 25, WMI_FW_CAPABILITY_CHANNEL_4 = 26, + WMI_FW_CAPABILITY_IPA = 27, WMI_FW_CAPABILITY_MAX, }; @@ -294,6 +295,7 @@ enum wmi_command_id { WMI_SET_AP_SLOT_SIZE_CMDID = 0xA0F, WMI_SET_VRING_PRIORITY_WEIGHT_CMDID = 0xA10, WMI_SET_VRING_PRIORITY_CMDID = 0xA11, + WMI_RBUFCAP_CFG_CMDID = 0xA12, WMI_SET_MAC_ADDRESS_CMDID = 0xF003, WMI_ABORT_SCAN_CMDID = 0xF007, WMI_SET_PROMISCUOUS_MODE_CMDID = 0xF041, @@ -979,10 +981,22 @@ enum wmi_rx_msg_type { WMI_RX_MSG_TYPE_EXTENDED = 0x01, }; +enum wmi_ring_add_irq_mode { + /* Backwards compatibility + * for DESC ring - interrupt disabled + * for STATUS ring - interrupt enabled + */ + WMI_RING_ADD_IRQ_MODE_BWC = 0x00, + WMI_RING_ADD_IRQ_MODE_DISABLE = 0x01, + WMI_RING_ADD_IRQ_MODE_ENABLE = 0x02, +}; + struct wmi_tx_status_ring_add_cmd { struct wmi_edma_ring_cfg ring_cfg; u8 irq_index; - u8 reserved[3]; + /* wmi_ring_add_irq_mode */ + u8 irq_mode; + u8 reserved[2]; } __packed; struct wmi_rx_status_ring_add_cmd { @@ -1016,7 +1030,10 @@ struct wmi_tx_desc_ring_add_cmd { u8 mac_ctrl; u8 to_resolution; u8 agg_max_wsize; - u8 reserved[3]; + u8 irq_index; + /* wmi_ring_add_irq_mode */ + u8 irq_mode; + u8 reserved; struct wmi_vring_cfg_schd schd_params; } __packed; @@ -1982,6 +1999,7 @@ enum wmi_event_id { WMI_BEAMFORMING_MGMT_DONE_EVENTID = 0x1836, WMI_BF_TXSS_MGMT_DONE_EVENTID = 0x1837, WMI_BF_RXSS_MGMT_DONE_EVENTID = 0x1839, + WMI_BF_TRIG_EVENTID = 0x183A, WMI_RS_MGMT_DONE_EVENTID = 0x1852, WMI_RF_MGMT_STATUS_EVENTID = 0x1853, WMI_BF_SM_MGMT_DONE_EVENTID = 0x1838, @@ -2082,6 +2100,7 @@ enum wmi_event_id { WMI_SET_AP_SLOT_SIZE_EVENTID = 0x1A0F, WMI_SET_VRING_PRIORITY_WEIGHT_EVENTID = 0x1A10, WMI_SET_VRING_PRIORITY_EVENTID = 0x1A11, + WMI_RBUFCAP_CFG_EVENTID = 0x1A12, WMI_SET_CHANNEL_EVENTID = 0x9000, WMI_ASSOC_REQ_EVENTID = 0x9001, WMI_EAPOL_RX_EVENTID = 0x9002, @@ -2267,7 +2286,9 @@ struct wmi_notify_req_done_event { __le32 status; __le64 tsf; s8 rssi; - u8 reserved0[3]; + /* enum wmi_edmg_tx_mode */ + u8 tx_mode; + u8 reserved0[2]; __le32 tx_tpt; __le32 tx_goodput; __le32 rx_goodput; @@ -2316,6 +2337,7 @@ enum wmi_disconnect_reason { WMI_DIS_REASON_PROFILE_MISMATCH = 0x0C, WMI_DIS_REASON_CONNECTION_EVICTED = 0x0D, WMI_DIS_REASON_IBSS_MERGE = 0x0E, + WMI_DIS_REASON_HIGH_TEMPERATURE = 0x0F, }; /* WMI_DISCONNECT_EVENTID */ @@ -3168,6 +3190,30 @@ struct wmi_brp_set_ant_limit_event { u8 reserved[3]; } __packed; +enum wmi_bf_type { + WMI_BF_TYPE_SLS = 0x00, + WMI_BF_TYPE_BRP_RX = 0x01, +}; + +/* WMI_BF_TRIG_CMDID */ +struct wmi_bf_trig_cmd { + /* enum wmi_bf_type - type of requested beamforming */ + u8 bf_type; + /* used only for WMI_BF_TYPE_BRP_RX */ + u8 cid; + /* used only for WMI_BF_TYPE_SLS */ + u8 dst_mac[WMI_MAC_LEN]; + u8 reserved[4]; +} __packed; + +/* WMI_BF_TRIG_EVENTID */ +struct wmi_bf_trig_event { + /* enum wmi_fw_status */ + u8 status; + u8 cid; + u8 reserved[2]; +} __packed; + /* broadcast connection ID */ #define WMI_LINK_MAINTAIN_CFG_CID_BROADCAST (0xFFFFFFFF) @@ -3263,6 +3309,8 @@ struct wmi_link_maintain_cfg_read_done_event { enum wmi_traffic_suspend_status { WMI_TRAFFIC_SUSPEND_APPROVED = 0x0, WMI_TRAFFIC_SUSPEND_REJECTED_LINK_NOT_IDLE = 0x1, + WMI_TRAFFIC_SUSPEND_REJECTED_DISCONNECT = 0x2, + WMI_TRAFFIC_SUSPEND_REJECTED_OTHER = 0x3, }; /* WMI_TRAFFIC_SUSPEND_EVENTID */ @@ -3282,6 +3330,7 @@ enum wmi_resume_trigger { WMI_RESUME_TRIGGER_UCAST_RX = 0x2, WMI_RESUME_TRIGGER_BCAST_RX = 0x4, WMI_RESUME_TRIGGER_WMI_EVT = 0x8, + WMI_RESUME_TRIGGER_DISCONNECT = 0x10, }; /* WMI_TRAFFIC_RESUME_EVENTID */ @@ -4057,4 +4106,38 @@ struct wmi_set_vring_priority_event { u8 reserved[3]; } __packed; +/* WMI_RADAR_PCI_CTRL_BLOCK struct */ +struct wmi_radar_pci_ctrl_block { + /* last fw tail address index */ + __le32 fw_tail_index; + /* last SW head address index known to FW */ + __le32 sw_head_index; + __le32 last_wr_pulse_tsf_low; + __le32 last_wr_pulse_count; + __le32 last_wr_in_bytes; + __le32 last_wr_pulse_id; + __le32 last_wr_burst_id; + /* When pre overflow detected, advance sw head in unit of pulses */ + __le32 sw_head_inc; + __le32 reserved[8]; +} __packed; + +/* WMI_RBUFCAP_CFG_CMD */ +struct wmi_rbufcap_cfg_cmd { + u8 enable; + u8 reserved; + /* RBUFCAP indicates rx space unavailable when number of rx + * descriptors drops below this threshold. Set 0 to use system + * default + */ + __le16 rx_desc_threshold; +} __packed; + +/* WMI_RBUFCAP_CFG_EVENTID */ +struct wmi_rbufcap_cfg_event { + /* enum wmi_fw_status */ + u8 status; + u8 reserved[3]; +} __packed; + #endif /* __WILOCITY_WMI_H__ */ diff --git a/drivers/net/wireless/atmel/at76c50x-usb.c b/drivers/net/wireless/atmel/at76c50x-usb.c index e99e766a302851e36b34f338cd819b6eaf616637..1cabae424839ebba77ee239c6ecd85a2b4b2c5bd 100644 --- a/drivers/net/wireless/atmel/at76c50x-usb.c +++ b/drivers/net/wireless/atmel/at76c50x-usb.c @@ -2585,8 +2585,8 @@ static int __init at76_mod_init(void) if (result < 0) printk(KERN_ERR DRIVER_NAME ": usb_register failed (status %d)\n", result); - - led_trigger_register_simple("at76_usb-tx", &ledtrig_tx); + else + led_trigger_register_simple("at76_usb-tx", &ledtrig_tx); return result; } diff --git a/drivers/net/wireless/broadcom/b43/phy_lp.c b/drivers/net/wireless/broadcom/b43/phy_lp.c index 46408a560814c5c17ed3a92a7daa5a555234c87e..6b7f0238723fd0bb44240471ed761320b5ed74df 100644 --- a/drivers/net/wireless/broadcom/b43/phy_lp.c +++ b/drivers/net/wireless/broadcom/b43/phy_lp.c @@ -1826,16 +1826,10 @@ static void lpphy_stop_tx_tone(struct b43_wldev *dev) } -static void lpphy_papd_cal(struct b43_wldev *dev, struct lpphy_tx_gains gains, - int mode, bool useindex, u8 index) -{ - //TODO -} - static void lpphy_papd_cal_txpwr(struct b43_wldev *dev) { struct b43_phy_lp *lpphy = dev->phy.lp; - struct lpphy_tx_gains gains, oldgains; + struct lpphy_tx_gains oldgains; int old_txpctl, old_afe_ovr, old_rf, old_bbmult; lpphy_read_tx_pctl_mode_from_hardware(dev); @@ -1848,11 +1842,6 @@ static void lpphy_papd_cal_txpwr(struct b43_wldev *dev) lpphy_set_tx_power_control(dev, B43_LPPHY_TXPCTL_OFF); - if (dev->dev->chip_id == 0x4325 && dev->dev->chip_rev == 0) - lpphy_papd_cal(dev, gains, 0, 1, 30); - else - lpphy_papd_cal(dev, gains, 0, 1, 65); - if (old_afe_ovr) lpphy_set_tx_gains(dev, oldgains); lpphy_set_bb_mult(dev, old_bbmult); diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcdc.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcdc.c index 73d3c1a0a7c946405258310e686b9c9bfee8efd4..98b168736df0800b998b53ab2d2743446e096555 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcdc.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcdc.c @@ -490,11 +490,18 @@ fail: return -ENOMEM; } -void brcmf_proto_bcdc_detach(struct brcmf_pub *drvr) +void brcmf_proto_bcdc_detach_pre_delif(struct brcmf_pub *drvr) +{ + struct brcmf_bcdc *bcdc = drvr->proto->pd; + + brcmf_fws_detach_pre_delif(bcdc->fws); +} + +void brcmf_proto_bcdc_detach_post_delif(struct brcmf_pub *drvr) { struct brcmf_bcdc *bcdc = drvr->proto->pd; drvr->proto->pd = NULL; - brcmf_fws_detach(bcdc->fws); + brcmf_fws_detach_post_delif(bcdc->fws); kfree(bcdc); } diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcdc.h b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcdc.h index 3b0e9eff21b5826883bd2a0377c3baeb023490c5..4bc52240ccea2e4b0c3a944170cfc4d0613e0f6f 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcdc.h +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcdc.h @@ -18,14 +18,16 @@ #ifdef CONFIG_BRCMFMAC_PROTO_BCDC int brcmf_proto_bcdc_attach(struct brcmf_pub *drvr); -void brcmf_proto_bcdc_detach(struct brcmf_pub *drvr); +void brcmf_proto_bcdc_detach_pre_delif(struct brcmf_pub *drvr); +void brcmf_proto_bcdc_detach_post_delif(struct brcmf_pub *drvr); void brcmf_proto_bcdc_txflowblock(struct device *dev, bool state); void brcmf_proto_bcdc_txcomplete(struct device *dev, struct sk_buff *txp, bool success); struct brcmf_fws_info *drvr_to_fws(struct brcmf_pub *drvr); #else static inline int brcmf_proto_bcdc_attach(struct brcmf_pub *drvr) { return 0; } -static inline void brcmf_proto_bcdc_detach(struct brcmf_pub *drvr) {} +static void brcmf_proto_bcdc_detach_pre_delif(struct brcmf_pub *drvr) {}; +static inline void brcmf_proto_bcdc_detach_post_delif(struct brcmf_pub *drvr) {} #endif #endif /* BRCMFMAC_BCDC_H */ diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcmsdh.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcmsdh.c index ec129864cc9cff71ee736ae73ff25c1c24a59c32..60aede5abb4de9d1512e132d34347d1c5cf84b57 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcmsdh.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcmsdh.c @@ -628,15 +628,13 @@ int brcmf_sdiod_send_buf(struct brcmf_sdio_dev *sdiodev, u8 *buf, uint nbytes) err = brcmf_sdiod_set_backplane_window(sdiodev, addr); if (err) - return err; + goto out; addr &= SBSDIO_SB_OFT_ADDR_MASK; addr |= SBSDIO_SB_ACCESS_2_4B_FLAG; - if (!err) - err = brcmf_sdiod_skbuff_write(sdiodev, sdiodev->func2, addr, - mypkt); - + err = brcmf_sdiod_skbuff_write(sdiodev, sdiodev->func2, addr, mypkt); +out: brcmu_pkt_buf_free_skb(mypkt); return err; diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bus.h b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bus.h index 3d441c5c745ccf30dddf7a38e36fe8a62617d958..2fe167eae22cc21ac2be8278197a581c88274f0c 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bus.h +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bus.h @@ -91,6 +91,7 @@ struct brcmf_bus_ops { int (*get_fwname)(struct device *dev, const char *ext, unsigned char *fw_name); void (*debugfs_create)(struct device *dev); + int (*reset)(struct device *dev); }; @@ -245,6 +246,15 @@ void brcmf_bus_debugfs_create(struct brcmf_bus *bus) return bus->ops->debugfs_create(bus->dev); } +static inline +int brcmf_bus_reset(struct brcmf_bus *bus) +{ + if (!bus->ops->reset) + return -EOPNOTSUPP; + + return bus->ops->reset(bus->dev); +} + /* * interface functions from common layer */ @@ -262,6 +272,8 @@ void brcmf_detach(struct device *dev); void brcmf_dev_reset(struct device *dev); /* Request from bus module to initiate a coredump */ void brcmf_dev_coredump(struct device *dev); +/* Indication that firmware has halted or crashed */ +void brcmf_fw_crashed(struct device *dev); /* Configure the "global" bus state used by upper layers */ void brcmf_bus_change_state(struct brcmf_bus *bus, enum brcmf_bus_state state); diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.c index e92f6351bd224d1ded06e645d7c9b23abf1c5382..8ee8af4e7ec4f3270faf0d47a47a86b1dc849716 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.c @@ -5464,6 +5464,8 @@ static s32 brcmf_get_assoc_ies(struct brcmf_cfg80211_info *cfg, conn_info->req_ie = kmemdup(cfg->extra_buf, conn_info->req_ie_len, GFP_KERNEL); + if (!conn_info->req_ie) + conn_info->req_ie_len = 0; } else { conn_info->req_ie_len = 0; conn_info->req_ie = NULL; @@ -5480,6 +5482,8 @@ static s32 brcmf_get_assoc_ies(struct brcmf_cfg80211_info *cfg, conn_info->resp_ie = kmemdup(cfg->extra_buf, conn_info->resp_ie_len, GFP_KERNEL); + if (!conn_info->resp_ie) + conn_info->resp_ie_len = 0; } else { conn_info->resp_ie_len = 0; conn_info->resp_ie = NULL; diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c index 4fbe8791f674959029bf27530c41a49299ba3886..7d6a08779693433e90eb9904756001307c9c70ac 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c @@ -841,17 +841,17 @@ static void brcmf_del_if(struct brcmf_pub *drvr, s32 bsscfgidx, bool rtnl_locked) { struct brcmf_if *ifp; + int ifidx; ifp = drvr->iflist[bsscfgidx]; - drvr->iflist[bsscfgidx] = NULL; if (!ifp) { bphy_err(drvr, "Null interface, bsscfgidx=%d\n", bsscfgidx); return; } brcmf_dbg(TRACE, "Enter, bsscfgidx=%d, ifidx=%d\n", bsscfgidx, ifp->ifidx); - if (drvr->if2bss[ifp->ifidx] == bsscfgidx) - drvr->if2bss[ifp->ifidx] = BRCMF_BSSIDX_INVALID; + ifidx = ifp->ifidx; + if (ifp->ndev) { if (bsscfgidx == 0) { if (ifp->ndev->netdev_ops == &brcmf_netdev_ops_pri) { @@ -879,6 +879,10 @@ static void brcmf_del_if(struct brcmf_pub *drvr, s32 bsscfgidx, brcmf_p2p_ifp_removed(ifp, rtnl_locked); kfree(ifp); } + + drvr->iflist[bsscfgidx] = NULL; + if (drvr->if2bss[ifidx] == bsscfgidx) + drvr->if2bss[ifidx] = BRCMF_BSSIDX_INVALID; } void brcmf_remove_interface(struct brcmf_if *ifp, bool rtnl_locked) @@ -1084,6 +1088,14 @@ static int brcmf_revinfo_read(struct seq_file *s, void *data) return 0; } +static void brcmf_core_bus_reset(struct work_struct *work) +{ + struct brcmf_pub *drvr = container_of(work, struct brcmf_pub, + bus_reset); + + brcmf_bus_reset(drvr->bus_if); +} + static int brcmf_bus_started(struct brcmf_pub *drvr, struct cfg80211_ops *ops) { int ret = -1; @@ -1155,6 +1167,8 @@ static int brcmf_bus_started(struct brcmf_pub *drvr, struct cfg80211_ops *ops) #endif #endif /* CONFIG_INET */ + INIT_WORK(&drvr->bus_reset, brcmf_core_bus_reset); + /* populate debugfs */ brcmf_debugfs_add_entry(drvr, "revinfo", brcmf_revinfo_read); brcmf_feat_debugfs_create(drvr); @@ -1273,6 +1287,18 @@ void brcmf_dev_coredump(struct device *dev) brcmf_dbg(TRACE, "failed to create coredump\n"); } +void brcmf_fw_crashed(struct device *dev) +{ + struct brcmf_bus *bus_if = dev_get_drvdata(dev); + struct brcmf_pub *drvr = bus_if->drvr; + + bphy_err(drvr, "Firmware has halted or crashed\n"); + + brcmf_dev_coredump(dev); + + schedule_work(&drvr->bus_reset); +} + void brcmf_detach(struct device *dev) { s32 i; @@ -1299,6 +1325,8 @@ void brcmf_detach(struct device *dev) brcmf_bus_change_state(bus_if, BRCMF_BUS_DOWN); + brcmf_proto_detach_pre_delif(drvr); + /* make sure primary interface removed last */ for (i = BRCMF_MAX_IFS-1; i > -1; i--) brcmf_remove_interface(drvr->iflist[i], false); @@ -1308,7 +1336,7 @@ void brcmf_detach(struct device *dev) brcmf_bus_stop(drvr->bus_if); - brcmf_proto_detach(drvr); + brcmf_proto_detach_post_delif(drvr); bus_if->drvr = NULL; wiphy_free(drvr->wiphy); diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.h b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.h index d8085ce579f4e3bf94c6f14135728fe59875edce..9f09aa31eeda92e852c1f5d24fc7702df44193c0 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.h +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.h @@ -143,6 +143,8 @@ struct brcmf_pub { struct notifier_block inet6addr_notifier; struct brcmf_mp_device *settings; + struct work_struct bus_reset; + u8 clmver[BRCMF_DCMD_SMLEN]; }; diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/dmi.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/dmi.c index 7535cb0d4ac045942e88d1dceb52ad006e989988..9f1417e000733b6a4378fe9a456546d3783acdf3 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/dmi.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/dmi.c @@ -31,6 +31,10 @@ struct brcmf_dmi_data { /* NOTE: Please keep all entries sorted alphabetically */ +static const struct brcmf_dmi_data acepc_t8_data = { + BRCM_CC_4345_CHIP_ID, 6, "acepc-t8" +}; + static const struct brcmf_dmi_data gpd_win_pocket_data = { BRCM_CC_4356_CHIP_ID, 2, "gpd-win-pocket" }; @@ -48,6 +52,28 @@ static const struct brcmf_dmi_data pov_tab_p1006w_data = { }; static const struct dmi_system_id dmi_platform_data[] = { + { + /* ACEPC T8 Cherry Trail Z8350 mini PC */ + .matches = { + DMI_EXACT_MATCH(DMI_BOARD_VENDOR, "To be filled by O.E.M."), + DMI_EXACT_MATCH(DMI_BOARD_NAME, "Cherry Trail CR"), + DMI_EXACT_MATCH(DMI_PRODUCT_SKU, "T8"), + /* also match on somewhat unique bios-version */ + DMI_EXACT_MATCH(DMI_BIOS_VERSION, "1.000"), + }, + .driver_data = (void *)&acepc_t8_data, + }, + { + /* ACEPC T11 Cherry Trail Z8350 mini PC, same wifi as the T8 */ + .matches = { + DMI_EXACT_MATCH(DMI_BOARD_VENDOR, "To be filled by O.E.M."), + DMI_EXACT_MATCH(DMI_BOARD_NAME, "Cherry Trail CR"), + DMI_EXACT_MATCH(DMI_PRODUCT_SKU, "T11"), + /* also match on somewhat unique bios-version */ + DMI_EXACT_MATCH(DMI_BIOS_VERSION, "1.000"), + }, + .driver_data = (void *)&acepc_t8_data, + }, { /* Match for the GPDwin which unfortunately uses somewhat * generic dmi strings, which is why we test for 4 strings. diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/firmware.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/firmware.c index 8209a42dea72c021f18338cad2f92574bc176b81..6a333dd80b2dc4f04f3d914c9809b1b7f1ce4606 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/firmware.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/firmware.c @@ -711,7 +711,6 @@ brcmf_fw_alloc_request(u32 chip, u32 chiprev, size_t mp_path_len; u32 i, j; char end = '\0'; - size_t reqsz; for (i = 0; i < table_size; i++) { if (mapping_table[i].chipid == chip && @@ -726,8 +725,7 @@ brcmf_fw_alloc_request(u32 chip, u32 chiprev, return NULL; } - reqsz = sizeof(*fwreq) + n_fwnames * sizeof(struct brcmf_fw_item); - fwreq = kzalloc(reqsz, GFP_KERNEL); + fwreq = kzalloc(struct_size(fwreq, items, n_fwnames), GFP_KERNEL); if (!fwreq) return NULL; @@ -743,6 +741,7 @@ brcmf_fw_alloc_request(u32 chip, u32 chiprev, for (j = 0; j < n_fwnames; j++) { fwreq->items[j].path = fwnames[j].path; + fwnames[j].path[0] = '\0'; /* check if firmware path is provided by module parameter */ if (brcmf_mp_global.firmware_path[0] != '\0') { strlcpy(fwnames[j].path, mp_path, diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fwsignal.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fwsignal.c index abeb305492e010c352f47de660b599c13883daf0..c22c49ae552e02a60e29c5db3512ab3ce1c65d66 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fwsignal.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fwsignal.c @@ -580,24 +580,6 @@ static bool brcmf_fws_ifidx_match(struct sk_buff *skb, void *arg) return ifidx == *(int *)arg; } -static void brcmf_fws_psq_flush(struct brcmf_fws_info *fws, struct pktq *q, - int ifidx) -{ - bool (*matchfn)(struct sk_buff *, void *) = NULL; - struct sk_buff *skb; - int prec; - - if (ifidx != -1) - matchfn = brcmf_fws_ifidx_match; - for (prec = 0; prec < q->num_prec; prec++) { - skb = brcmu_pktq_pdeq_match(q, prec, matchfn, &ifidx); - while (skb) { - brcmu_pkt_buf_free_skb(skb); - skb = brcmu_pktq_pdeq_match(q, prec, matchfn, &ifidx); - } - } -} - static void brcmf_fws_hanger_init(struct brcmf_fws_hanger *hanger) { int i; @@ -669,6 +651,28 @@ static inline int brcmf_fws_hanger_poppkt(struct brcmf_fws_hanger *h, return 0; } +static void brcmf_fws_psq_flush(struct brcmf_fws_info *fws, struct pktq *q, + int ifidx) +{ + bool (*matchfn)(struct sk_buff *, void *) = NULL; + struct sk_buff *skb; + int prec; + u32 hslot; + + if (ifidx != -1) + matchfn = brcmf_fws_ifidx_match; + for (prec = 0; prec < q->num_prec; prec++) { + skb = brcmu_pktq_pdeq_match(q, prec, matchfn, &ifidx); + while (skb) { + hslot = brcmf_skb_htod_tag_get_field(skb, HSLOT); + brcmf_fws_hanger_poppkt(&fws->hanger, hslot, &skb, + true); + brcmu_pkt_buf_free_skb(skb); + skb = brcmu_pktq_pdeq_match(q, prec, matchfn, &ifidx); + } + } +} + static int brcmf_fws_hanger_mark_suppressed(struct brcmf_fws_hanger *h, u32 slot_id) { @@ -2200,6 +2204,8 @@ void brcmf_fws_del_interface(struct brcmf_if *ifp) brcmf_fws_lock(fws); ifp->fws_desc = NULL; brcmf_dbg(TRACE, "deleting %s\n", entry->name); + brcmf_fws_macdesc_cleanup(fws, &fws->desc.iface[ifp->ifidx], + ifp->ifidx); brcmf_fws_macdesc_deinit(entry); brcmf_fws_cleanup(fws, ifp->ifidx); brcmf_fws_unlock(fws); @@ -2437,17 +2443,25 @@ struct brcmf_fws_info *brcmf_fws_attach(struct brcmf_pub *drvr) return fws; fail: - brcmf_fws_detach(fws); + brcmf_fws_detach_pre_delif(fws); + brcmf_fws_detach_post_delif(fws); return ERR_PTR(rc); } -void brcmf_fws_detach(struct brcmf_fws_info *fws) +void brcmf_fws_detach_pre_delif(struct brcmf_fws_info *fws) { if (!fws) return; - - if (fws->fws_wq) + if (fws->fws_wq) { destroy_workqueue(fws->fws_wq); + fws->fws_wq = NULL; + } +} + +void brcmf_fws_detach_post_delif(struct brcmf_fws_info *fws) +{ + if (!fws) + return; /* cleanup */ brcmf_fws_lock(fws); diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fwsignal.h b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fwsignal.h index 4e6835766d5d8fb287560ea224d50d1fb3df4442..749c06dcdc17ddf958daccd6fda42bbaeed55e1f 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fwsignal.h +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fwsignal.h @@ -19,7 +19,8 @@ #define FWSIGNAL_H_ struct brcmf_fws_info *brcmf_fws_attach(struct brcmf_pub *drvr); -void brcmf_fws_detach(struct brcmf_fws_info *fws); +void brcmf_fws_detach_pre_delif(struct brcmf_fws_info *fws); +void brcmf_fws_detach_post_delif(struct brcmf_fws_info *fws); void brcmf_fws_debugfs_create(struct brcmf_pub *drvr); bool brcmf_fws_queue_skbs(struct brcmf_fws_info *fws); bool brcmf_fws_fc_active(struct brcmf_fws_info *fws); diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/msgbuf.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/msgbuf.c index d3780eae7f19cdb1f669e12bec82817fd09933fa..9d1f9ff25bfade875212db27d744a3d8905fff40 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/msgbuf.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/msgbuf.c @@ -375,7 +375,7 @@ brcmf_msgbuf_get_pktid(struct device *dev, struct brcmf_msgbuf_pktids *pktids, struct brcmf_msgbuf_pktid *pktid; struct sk_buff *skb; - if (idx >= pktids->array_size) { + if (idx < 0 || idx >= pktids->array_size) { brcmf_err("Invalid packet id %d (max %d)\n", idx, pktids->array_size); return NULL; @@ -747,7 +747,7 @@ static void brcmf_msgbuf_txflow(struct brcmf_msgbuf *msgbuf, u16 flowid) tx_msghdr = (struct msgbuf_tx_msghdr *)ret_ptr; tx_msghdr->msg.msgtype = MSGBUF_TYPE_TX_POST; - tx_msghdr->msg.request_id = cpu_to_le32(pktid); + tx_msghdr->msg.request_id = cpu_to_le32(pktid + 1); tx_msghdr->msg.ifidx = brcmf_flowring_ifidx_get(flow, flowid); tx_msghdr->flags = BRCMF_MSGBUF_PKT_FLAGS_FRAME_802_3; tx_msghdr->flags |= (skb->priority & 0x07) << @@ -884,7 +884,7 @@ brcmf_msgbuf_process_txstatus(struct brcmf_msgbuf *msgbuf, void *buf) u16 flowid; tx_status = (struct msgbuf_tx_status *)buf; - idx = le32_to_cpu(tx_status->msg.request_id); + idx = le32_to_cpu(tx_status->msg.request_id) - 1; flowid = le16_to_cpu(tx_status->compl_hdr.flow_ring_id); flowid -= BRCMF_H2D_MSGRING_FLOWRING_IDSTART; skb = brcmf_msgbuf_get_pktid(msgbuf->drvr->bus_if->dev, diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/pcie.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/pcie.c index 58a6bc37935878db7669ffed776719fc0ecf1e87..83e4938527f4b611e6813e180a983c77b0f1e4d4 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/pcie.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/pcie.c @@ -345,6 +345,10 @@ static const u32 brcmf_ring_itemsize[BRCMF_NROF_COMMON_MSGRINGS] = { BRCMF_D2H_MSGRING_RX_COMPLETE_ITEMSIZE }; +static void brcmf_pcie_setup(struct device *dev, int ret, + struct brcmf_fw_request *fwreq); +static struct brcmf_fw_request * +brcmf_pcie_prepare_fw_request(struct brcmf_pciedev_info *devinfo); static u32 brcmf_pcie_read_reg32(struct brcmf_pciedev_info *devinfo, u32 reg_offset) @@ -671,6 +675,7 @@ static int brcmf_pcie_send_mb_data(struct brcmf_pciedev_info *devinfo, u32 htod_mb_data) { struct brcmf_pcie_shared_info *shared; + struct brcmf_core *core; u32 addr; u32 cur_htod_mb_data; u32 i; @@ -694,7 +699,11 @@ brcmf_pcie_send_mb_data(struct brcmf_pciedev_info *devinfo, u32 htod_mb_data) brcmf_pcie_write_tcm32(devinfo, addr, htod_mb_data); pci_write_config_dword(devinfo->pdev, BRCMF_PCIE_REG_SBMBX, 1); - pci_write_config_dword(devinfo->pdev, BRCMF_PCIE_REG_SBMBX, 1); + + /* Send mailbox interrupt twice as a hardware workaround */ + core = brcmf_chip_get_core(devinfo->ci, BCMA_CORE_PCIE2); + if (core->rev <= 13) + pci_write_config_dword(devinfo->pdev, BRCMF_PCIE_REG_SBMBX, 1); return 0; } @@ -730,7 +739,7 @@ static void brcmf_pcie_handle_mb_data(struct brcmf_pciedev_info *devinfo) } if (dtoh_mb_data & BRCMF_D2H_DEV_FWHALT) { brcmf_dbg(PCIE, "D2H_MB_DATA: FW HALT\n"); - brcmf_dev_coredump(&devinfo->pdev->dev); + brcmf_fw_crashed(&devinfo->pdev->dev); } } @@ -755,15 +764,22 @@ static void brcmf_pcie_bus_console_init(struct brcmf_pciedev_info *devinfo) console->base_addr, console->buf_addr, console->bufsize); } - -static void brcmf_pcie_bus_console_read(struct brcmf_pciedev_info *devinfo) +/** + * brcmf_pcie_bus_console_read - reads firmware messages + * + * @error: specifies if error has occurred (prints messages unconditionally) + */ +static void brcmf_pcie_bus_console_read(struct brcmf_pciedev_info *devinfo, + bool error) { + struct pci_dev *pdev = devinfo->pdev; + struct brcmf_bus *bus = dev_get_drvdata(&pdev->dev); struct brcmf_pcie_console *console; u32 addr; u8 ch; u32 newidx; - if (!BRCMF_FWCON_ON()) + if (!error && !BRCMF_FWCON_ON()) return; console = &devinfo->shared.console; @@ -787,7 +803,10 @@ static void brcmf_pcie_bus_console_read(struct brcmf_pciedev_info *devinfo) } if (ch == '\n') { console->log_str[console->log_idx] = 0; - pr_debug("CONSOLE: %s", console->log_str); + if (error) + brcmf_err(bus, "CONSOLE: %s", console->log_str); + else + pr_debug("CONSOLE: %s", console->log_str); console->log_idx = 0; } } @@ -848,7 +867,7 @@ static irqreturn_t brcmf_pcie_isr_thread(int irq, void *arg) &devinfo->pdev->dev); } } - brcmf_pcie_bus_console_read(devinfo); + brcmf_pcie_bus_console_read(devinfo, false); if (devinfo->state == BRCMFMAC_PCIE_STATE_UP) brcmf_pcie_intr_enable(devinfo); devinfo->in_irq = false; @@ -1409,6 +1428,38 @@ int brcmf_pcie_get_fwname(struct device *dev, const char *ext, u8 *fw_name) return 0; } +static int brcmf_pcie_reset(struct device *dev) +{ + struct brcmf_bus *bus_if = dev_get_drvdata(dev); + struct brcmf_pciedev *buspub = bus_if->bus_priv.pcie; + struct brcmf_pciedev_info *devinfo = buspub->devinfo; + struct brcmf_fw_request *fwreq; + int err; + + brcmf_pcie_bus_console_read(devinfo, true); + + brcmf_detach(dev); + + brcmf_pcie_release_irq(devinfo); + brcmf_pcie_release_scratchbuffers(devinfo); + brcmf_pcie_release_ringbuffers(devinfo); + brcmf_pcie_reset_device(devinfo); + + fwreq = brcmf_pcie_prepare_fw_request(devinfo); + if (!fwreq) { + dev_err(dev, "Failed to prepare FW request\n"); + return -ENOMEM; + } + + err = brcmf_fw_get_firmwares(dev, fwreq, brcmf_pcie_setup); + if (err) { + dev_err(dev, "Failed to prepare FW request\n"); + kfree(fwreq); + } + + return err; +} + static const struct brcmf_bus_ops brcmf_pcie_bus_ops = { .txdata = brcmf_pcie_tx, .stop = brcmf_pcie_down, @@ -1418,6 +1469,7 @@ static const struct brcmf_bus_ops brcmf_pcie_bus_ops = { .get_ramsize = brcmf_pcie_get_ramsize, .get_memdump = brcmf_pcie_get_memdump, .get_fwname = brcmf_pcie_get_fwname, + .reset = brcmf_pcie_reset, }; @@ -1778,7 +1830,7 @@ static void brcmf_pcie_setup(struct device *dev, int ret, if (brcmf_attach(&devinfo->pdev->dev, devinfo->settings) == 0) return; - brcmf_pcie_bus_console_read(devinfo); + brcmf_pcie_bus_console_read(devinfo, false); fail: device_release_driver(dev); diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/proto.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/proto.c index 024c643052bcc437c3849a91aca2f457dc4c2f83..c7964ccdda69a667ca3e043a6e3d598e8ad1a194 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/proto.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/proto.c @@ -67,16 +67,22 @@ fail: return -ENOMEM; } -void brcmf_proto_detach(struct brcmf_pub *drvr) +void brcmf_proto_detach_post_delif(struct brcmf_pub *drvr) { brcmf_dbg(TRACE, "Enter\n"); if (drvr->proto) { if (drvr->bus_if->proto_type == BRCMF_PROTO_BCDC) - brcmf_proto_bcdc_detach(drvr); + brcmf_proto_bcdc_detach_post_delif(drvr); else if (drvr->bus_if->proto_type == BRCMF_PROTO_MSGBUF) brcmf_proto_msgbuf_detach(drvr); kfree(drvr->proto); drvr->proto = NULL; } } + +void brcmf_proto_detach_pre_delif(struct brcmf_pub *drvr) +{ + if (drvr->proto && drvr->bus_if->proto_type == BRCMF_PROTO_BCDC) + brcmf_proto_bcdc_detach_pre_delif(drvr); +} diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/proto.h b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/proto.h index d3c3b9a815ad7f5377b7caa4a278538b28b5095b..72355aea902879533eb05f23bf272a717902aad5 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/proto.h +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/proto.h @@ -54,7 +54,8 @@ struct brcmf_proto { int brcmf_proto_attach(struct brcmf_pub *drvr); -void brcmf_proto_detach(struct brcmf_pub *drvr); +void brcmf_proto_detach_pre_delif(struct brcmf_pub *drvr); +void brcmf_proto_detach_post_delif(struct brcmf_pub *drvr); static inline int brcmf_proto_hdrpull(struct brcmf_pub *drvr, bool do_fws, struct sk_buff *skb, diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c index 4d104ab80fd8ec9bf7c68583c46cca7f2962fe39..22b73da428225f67675ca9f6bb7d8d07b9cecb95 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c @@ -622,6 +622,7 @@ BRCMF_FW_DEF(43430A0, "brcmfmac43430a0-sdio"); /* Note the names are not postfixed with a1 for backward compatibility */ BRCMF_FW_DEF(43430A1, "brcmfmac43430-sdio"); BRCMF_FW_DEF(43455, "brcmfmac43455-sdio"); +BRCMF_FW_DEF(43456, "brcmfmac43456-sdio"); BRCMF_FW_DEF(4354, "brcmfmac4354-sdio"); BRCMF_FW_DEF(4356, "brcmfmac4356-sdio"); BRCMF_FW_DEF(4373, "brcmfmac4373-sdio"); @@ -642,7 +643,8 @@ static const struct brcmf_firmware_mapping brcmf_sdio_fwnames[] = { BRCMF_FW_ENTRY(BRCM_CC_4339_CHIP_ID, 0xFFFFFFFF, 4339), BRCMF_FW_ENTRY(BRCM_CC_43430_CHIP_ID, 0x00000001, 43430A0), BRCMF_FW_ENTRY(BRCM_CC_43430_CHIP_ID, 0xFFFFFFFE, 43430A1), - BRCMF_FW_ENTRY(BRCM_CC_4345_CHIP_ID, 0xFFFFFFC0, 43455), + BRCMF_FW_ENTRY(BRCM_CC_4345_CHIP_ID, 0x00000200, 43456), + BRCMF_FW_ENTRY(BRCM_CC_4345_CHIP_ID, 0xFFFFFDC0, 43455), BRCMF_FW_ENTRY(BRCM_CC_4354_CHIP_ID, 0xFFFFFFFF, 4354), BRCMF_FW_ENTRY(BRCM_CC_4356_CHIP_ID, 0xFFFFFFFF, 4356), BRCMF_FW_ENTRY(CY_CC_4373_CHIP_ID, 0xFFFFFFFF, 4373), @@ -1090,8 +1092,8 @@ static u32 brcmf_sdio_hostmail(struct brcmf_sdio *bus) /* dongle indicates the firmware has halted/crashed */ if (hmb_data & HMB_DATA_FWHALT) { - brcmf_err("mailbox indicates firmware halted\n"); - brcmf_dev_coredump(&sdiod->func1->dev); + brcmf_dbg(SDIO, "mailbox indicates firmware halted\n"); + brcmf_fw_crashed(&sdiod->func1->dev); } /* Dongle recomposed rx frames, accept them again */ diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/usb.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/usb.c index e9cbfd077710af959a8f0edb51b2cdd5c569c95c..75fcd6752edcf190ebfd24f5cc74910b36f336e7 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/usb.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/usb.c @@ -160,7 +160,7 @@ struct brcmf_usbdev_info { struct usb_device *usbdev; struct device *dev; - struct mutex dev_init_lock; + struct completion dev_init_done; int ctl_in_pipe, ctl_out_pipe; struct urb *ctl_urb; /* URB for control endpoint */ @@ -445,22 +445,17 @@ fail: } -static void brcmf_usb_free_q(struct list_head *q, bool pending) +static void brcmf_usb_free_q(struct list_head *q) { struct brcmf_usbreq *req, *next; - int i = 0; + list_for_each_entry_safe(req, next, q, list) { if (!req->urb) { brcmf_err("bad req\n"); break; } - i++; - if (pending) { - usb_kill_urb(req->urb); - } else { - usb_free_urb(req->urb); - list_del_init(&req->list); - } + usb_free_urb(req->urb); + list_del_init(&req->list); } } @@ -682,12 +677,18 @@ static int brcmf_usb_up(struct device *dev) static void brcmf_cancel_all_urbs(struct brcmf_usbdev_info *devinfo) { + int i; + if (devinfo->ctl_urb) usb_kill_urb(devinfo->ctl_urb); if (devinfo->bulk_urb) usb_kill_urb(devinfo->bulk_urb); - brcmf_usb_free_q(&devinfo->tx_postq, true); - brcmf_usb_free_q(&devinfo->rx_postq, true); + if (devinfo->tx_reqs) + for (i = 0; i < devinfo->bus_pub.ntxq; i++) + usb_kill_urb(devinfo->tx_reqs[i].urb); + if (devinfo->rx_reqs) + for (i = 0; i < devinfo->bus_pub.nrxq; i++) + usb_kill_urb(devinfo->rx_reqs[i].urb); } static void brcmf_usb_down(struct device *dev) @@ -1023,8 +1024,8 @@ static void brcmf_usb_detach(struct brcmf_usbdev_info *devinfo) brcmf_dbg(USB, "Enter, devinfo %p\n", devinfo); /* free the URBS */ - brcmf_usb_free_q(&devinfo->rx_freeq, false); - brcmf_usb_free_q(&devinfo->tx_freeq, false); + brcmf_usb_free_q(&devinfo->rx_freeq); + brcmf_usb_free_q(&devinfo->tx_freeq); usb_free_urb(devinfo->ctl_urb); usb_free_urb(devinfo->bulk_urb); @@ -1193,11 +1194,11 @@ static void brcmf_usb_probe_phase2(struct device *dev, int ret, if (ret) goto error; - mutex_unlock(&devinfo->dev_init_lock); + complete(&devinfo->dev_init_done); return; error: brcmf_dbg(TRACE, "failed: dev=%s, err=%d\n", dev_name(dev), ret); - mutex_unlock(&devinfo->dev_init_lock); + complete(&devinfo->dev_init_done); device_release_driver(dev); } @@ -1265,7 +1266,7 @@ static int brcmf_usb_probe_cb(struct brcmf_usbdev_info *devinfo) if (ret) goto fail; /* we are done */ - mutex_unlock(&devinfo->dev_init_lock); + complete(&devinfo->dev_init_done); return 0; } bus->chip = bus_pub->devid; @@ -1325,11 +1326,10 @@ brcmf_usb_probe(struct usb_interface *intf, const struct usb_device_id *id) devinfo->usbdev = usb; devinfo->dev = &usb->dev; - /* Take an init lock, to protect for disconnect while still loading. + /* Init completion, to protect for disconnect while still loading. * Necessary because of the asynchronous firmware load construction */ - mutex_init(&devinfo->dev_init_lock); - mutex_lock(&devinfo->dev_init_lock); + init_completion(&devinfo->dev_init_done); usb_set_intfdata(intf, devinfo); @@ -1407,7 +1407,7 @@ brcmf_usb_probe(struct usb_interface *intf, const struct usb_device_id *id) return 0; fail: - mutex_unlock(&devinfo->dev_init_lock); + complete(&devinfo->dev_init_done); kfree(devinfo); usb_set_intfdata(intf, NULL); return ret; @@ -1422,7 +1422,7 @@ brcmf_usb_disconnect(struct usb_interface *intf) devinfo = (struct brcmf_usbdev_info *)usb_get_intfdata(intf); if (devinfo) { - mutex_lock(&devinfo->dev_init_lock); + wait_for_completion(&devinfo->dev_init_done); /* Make sure that devinfo still exists. Firmware probe routines * may have released the device and cleared the intfdata. */ diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/vendor.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/vendor.c index 8eff2753abadeb2704f87f88f7c5eabd75bf091b..d493021f6031852b478706022418293ab8939090 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/vendor.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/vendor.c @@ -35,9 +35,10 @@ static int brcmf_cfg80211_vndr_cmds_dcmd_handler(struct wiphy *wiphy, struct brcmf_if *ifp; const struct brcmf_vndr_dcmd_hdr *cmdhdr = data; struct sk_buff *reply; - int ret, payload, ret_len; + unsigned int payload, ret_len; void *dcmd_buf = NULL, *wr_pointer; u16 msglen, maxmsglen = PAGE_SIZE - 0x100; + int ret; if (len < sizeof(*cmdhdr)) { brcmf_err("vendor command too short: %d\n", len); @@ -65,7 +66,7 @@ static int brcmf_cfg80211_vndr_cmds_dcmd_handler(struct wiphy *wiphy, brcmf_err("oversize return buffer %d\n", ret_len); ret_len = BRCMF_DCMD_MAXLEN; } - payload = max(ret_len, len) + 1; + payload = max_t(unsigned int, ret_len, len) + 1; dcmd_buf = vzalloc(payload); if (NULL == dcmd_buf) return -ENOMEM; diff --git a/drivers/net/wireless/intel/iwlegacy/3945-debug.c b/drivers/net/wireless/intel/iwlegacy/3945-debug.c index a2960032be81252ad97ba82b6209dabcaa75a67c..4b912e707f38935de946c08d7a54f82940a3a144 100644 --- a/drivers/net/wireless/intel/iwlegacy/3945-debug.c +++ b/drivers/net/wireless/intel/iwlegacy/3945-debug.c @@ -185,7 +185,7 @@ il3945_ucode_rx_stats_read(struct file *file, char __user *user_buf, pos += scnprintf(buf + pos, bufsz - pos, "%-32s current" - "acumulative delta max\n", + "accumulative delta max\n", "Statistics_Rx - CCK:"); pos += scnprintf(buf + pos, bufsz - pos, @@ -273,7 +273,7 @@ il3945_ucode_rx_stats_read(struct file *file, char __user *user_buf, pos += scnprintf(buf + pos, bufsz - pos, "%-32s current" - "acumulative delta max\n", + "accumulative delta max\n", "Statistics_Rx - GENERAL:"); pos += scnprintf(buf + pos, bufsz - pos, @@ -346,7 +346,7 @@ il3945_ucode_tx_stats_read(struct file *file, char __user *user_buf, pos += scnprintf(buf + pos, bufsz - pos, "%-32s current" - "acumulative delta max\n", + "accumulative delta max\n", "Statistics_Tx:"); pos += scnprintf(buf + pos, bufsz - pos, @@ -447,7 +447,7 @@ il3945_ucode_general_stats_read(struct file *file, char __user *user_buf, pos += scnprintf(buf + pos, bufsz - pos, "%-32s current" - "acumulative delta max\n", + "accumulative delta max\n", "Statistics_General:"); pos += scnprintf(buf + pos, bufsz - pos, diff --git a/drivers/net/wireless/intel/iwlegacy/4965.c b/drivers/net/wireless/intel/iwlegacy/4965.c index ce4144a892173a14e24bd8cd9e660732e96a8e60..a20b6c885047bac9812d44a58f9f346b132b29dc 100644 --- a/drivers/net/wireless/intel/iwlegacy/4965.c +++ b/drivers/net/wireless/intel/iwlegacy/4965.c @@ -577,7 +577,6 @@ il4965_math_div_round(s32 num, s32 denom, s32 * res) sign = -sign; denom = -denom; } - *res = 1; *res = ((num * 2 + denom) / (denom * 2)) * sign; return 1; diff --git a/drivers/net/wireless/intel/iwlwifi/cfg/22000.c b/drivers/net/wireless/intel/iwlwifi/cfg/22000.c index 0a87d87fbb4f5dcf7f9cd5f99fcfb33600ca2297..a9c846c59289e8f34a23b89bf7f248785589788b 100644 --- a/drivers/net/wireless/intel/iwlwifi/cfg/22000.c +++ b/drivers/net/wireless/intel/iwlwifi/cfg/22000.c @@ -56,7 +56,7 @@ #include "iwl-config.h" /* Highest firmware API version supported */ -#define IWL_22000_UCODE_API_MAX 46 +#define IWL_22000_UCODE_API_MAX 48 /* Lowest firmware API version supported */ #define IWL_22000_UCODE_API_MIN 39 @@ -80,7 +80,6 @@ #define IWL_22000_QU_B_HR_B_FW_PRE "iwlwifi-Qu-b0-hr-b0-" #define IWL_22000_HR_B_FW_PRE "iwlwifi-QuQnj-b0-hr-b0-" #define IWL_22000_HR_A0_FW_PRE "iwlwifi-QuQnj-a0-hr-a0-" -#define IWL_22000_SU_Z0_FW_PRE "iwlwifi-su-z0-" #define IWL_QU_B_JF_B_FW_PRE "iwlwifi-Qu-b0-jf-b0-" #define IWL_QUZ_A_HR_B_FW_PRE "iwlwifi-QuZ-a0-hr-b0-" #define IWL_QNJ_B_JF_B_FW_PRE "iwlwifi-QuQnj-b0-jf-b0-" @@ -89,6 +88,7 @@ #define IWL_22000_SO_A_HR_B_FW_PRE "iwlwifi-so-a0-hr-b0-" #define IWL_22000_SO_A_GF_A_FW_PRE "iwlwifi-so-a0-gf-a0-" #define IWL_22000_TY_A_GF_A_FW_PRE "iwlwifi-ty-a0-gf-a0-" +#define IWL_22000_SO_A_GF4_A_FW_PRE "iwlwifi-so-a0-gf4-a0-" #define IWL_22000_HR_MODULE_FIRMWARE(api) \ IWL_22000_HR_FW_PRE __stringify(api) ".ucode" @@ -104,8 +104,6 @@ IWL_22000_HR_B_FW_PRE __stringify(api) ".ucode" #define IWL_22000_HR_A0_QNJ_MODULE_FIRMWARE(api) \ IWL_22000_HR_A0_FW_PRE __stringify(api) ".ucode" -#define IWL_22000_SU_Z0_MODULE_FIRMWARE(api) \ - IWL_22000_SU_Z0_FW_PRE __stringify(api) ".ucode" #define IWL_QUZ_A_HR_B_MODULE_FIRMWARE(api) \ IWL_QUZ_A_HR_B_FW_PRE __stringify(api) ".ucode" #define IWL_QU_B_JF_B_MODULE_FIRMWARE(api) \ @@ -180,7 +178,11 @@ static const struct iwl_ht_params iwl_22000_ht_params = { .dbgc_supported = true, \ .min_umac_error_event_table = 0x400000, \ .d3_debug_data_base_addr = 0x401000, \ - .d3_debug_data_length = 60 * 1024 + .d3_debug_data_length = 60 * 1024, \ + .fw_mon_smem_write_ptr_addr = 0xa0c16c, \ + .fw_mon_smem_write_ptr_msk = 0xfffff, \ + .fw_mon_smem_cycle_cnt_ptr_addr = 0xa0c174, \ + .fw_mon_smem_cycle_cnt_ptr_msk = 0xfffff #define IWL_DEVICE_AX200_COMMON \ IWL_DEVICE_22000_COMMON, \ @@ -190,7 +192,8 @@ static const struct iwl_ht_params iwl_22000_ht_params = { IWL_DEVICE_22000_COMMON, \ .device_family = IWL_DEVICE_FAMILY_22000, \ .base_params = &iwl_22000_base_params, \ - .csr = &iwl_csr_v1 + .csr = &iwl_csr_v1, \ + .gp2_reg_addr = 0xa02c68 #define IWL_DEVICE_22560 \ IWL_DEVICE_22000_COMMON, \ @@ -203,7 +206,9 @@ static const struct iwl_ht_params iwl_22000_ht_params = { .device_family = IWL_DEVICE_FAMILY_AX210, \ .base_params = &iwl_22560_base_params, \ .csr = &iwl_csr_v1, \ - .min_txq_size = 128 + .min_txq_size = 128, \ + .gp2_reg_addr = 0xd02c68, \ + .min_256_ba_txq_size = 512 const struct iwl_cfg iwl22000_2ac_cfg_hr = { .name = "Intel(R) Dual Band Wireless AC 22000", @@ -412,19 +417,6 @@ const struct iwl_cfg iwl22000_2ax_cfg_qnj_hr_a0 = { .max_tx_agg_size = IEEE80211_MAX_AMPDU_BUF_HT, }; -const struct iwl_cfg iwl22560_2ax_cfg_su_cdb = { - .name = "Intel(R) Dual Band Wireless AX 22560", - .fw_name_pre = IWL_22000_SU_Z0_FW_PRE, - IWL_DEVICE_22560, - .cdb = true, - /* - * This device doesn't support receiving BlockAck with a large bitmap - * so we need to restrict the size of transmitted aggregation to the - * HT size; mac80211 would otherwise pick the HE max (256) by default. - */ - .max_tx_agg_size = IEEE80211_MAX_AMPDU_BUF_HT, -}; - const struct iwl_cfg iwlax210_2ax_cfg_so_jf_a0 = { .name = "Intel(R) Wireless-AC 9560 160MHz", .fw_name_pre = IWL_22000_SO_A_JF_B_FW_PRE, @@ -440,12 +432,20 @@ const struct iwl_cfg iwlax210_2ax_cfg_so_hr_a0 = { const struct iwl_cfg iwlax210_2ax_cfg_so_gf_a0 = { .name = "Intel(R) Wi-Fi 7 AX211 160MHz", .fw_name_pre = IWL_22000_SO_A_GF_A_FW_PRE, + .uhb_supported = true, IWL_DEVICE_AX210, }; const struct iwl_cfg iwlax210_2ax_cfg_ty_gf_a0 = { .name = "Intel(R) Wi-Fi 7 AX210 160MHz", .fw_name_pre = IWL_22000_TY_A_GF_A_FW_PRE, + .uhb_supported = true, + IWL_DEVICE_AX210, +}; + +const struct iwl_cfg iwlax210_2ax_cfg_so_gf4_a0 = { + .name = "Intel(R) Wi-Fi 7 AX210 160MHz", + .fw_name_pre = IWL_22000_SO_A_GF4_A_FW_PRE, IWL_DEVICE_AX210, }; @@ -455,7 +455,6 @@ MODULE_FIRMWARE(IWL_22000_HR_A_F0_QNJ_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); MODULE_FIRMWARE(IWL_22000_HR_B_F0_QNJ_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); MODULE_FIRMWARE(IWL_22000_HR_B_QNJ_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); MODULE_FIRMWARE(IWL_22000_HR_A0_QNJ_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); -MODULE_FIRMWARE(IWL_22000_SU_Z0_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); MODULE_FIRMWARE(IWL_QU_B_JF_B_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); MODULE_FIRMWARE(IWL_QUZ_A_HR_B_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); MODULE_FIRMWARE(IWL_QNJ_B_JF_B_MODULE_FIRMWARE(IWL_22000_UCODE_API_MAX)); diff --git a/drivers/net/wireless/intel/iwlwifi/cfg/9000.c b/drivers/net/wireless/intel/iwlwifi/cfg/9000.c index 3225b64eb8454914fda7518904d43948b0aec9da..41bdd0eaf62c3fa3d58bfd11381a5b8b76167b24 100644 --- a/drivers/net/wireless/intel/iwlwifi/cfg/9000.c +++ b/drivers/net/wireless/intel/iwlwifi/cfg/9000.c @@ -6,7 +6,7 @@ * GPL LICENSE SUMMARY * * Copyright(c) 2015-2017 Intel Deutschland GmbH - * Copyright (C) 2018 Intel Corporation + * Copyright (C) 2018 - 2019 Intel Corporation * * This program is free software; you can redistribute it and/or modify * it under the terms of version 2 of the GNU General Public License as @@ -20,7 +20,7 @@ * BSD LICENSE * * Copyright(c) 2015-2017 Intel Deutschland GmbH - * Copyright (C) 2018 Intel Corporation + * Copyright (C) 2018 - 2019 Intel Corporation * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -148,7 +148,11 @@ static const struct iwl_tt_params iwl9000_tt_params = { .d3_debug_data_length = 92 * 1024, \ .ht_params = &iwl9000_ht_params, \ .nvm_ver = IWL9000_NVM_VERSION, \ - .max_ht_ampdu_exponent = IEEE80211_HT_MAX_AMPDU_64K + .max_ht_ampdu_exponent = IEEE80211_HT_MAX_AMPDU_64K, \ + .fw_mon_smem_write_ptr_addr = 0xa0476c, \ + .fw_mon_smem_write_ptr_msk = 0xfffff, \ + .fw_mon_smem_cycle_cnt_ptr_addr = 0xa04774, \ + .fw_mon_smem_cycle_cnt_ptr_msk = 0xfffff const struct iwl_cfg iwl9160_2ac_cfg = { diff --git a/drivers/net/wireless/intel/iwlwifi/fw/api/d3.h b/drivers/net/wireless/intel/iwlwifi/fw/api/d3.h index 86ea0784e1a37fee6c7e6612a6a63ec07e95abdb..31231b223aaeb44421ef391d3b6296651997df2c 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/api/d3.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/api/d3.h @@ -8,7 +8,7 @@ * Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH * Copyright(c) 2015 - 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * * This program is free software; you can redistribute it and/or modify * it under the terms of version 2 of the GNU General Public License as @@ -31,7 +31,7 @@ * Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH * Copyright(c) 2015 - 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -214,7 +214,7 @@ struct iwl_proto_offload_cmd_v3_large { #define IWL_WOWLAN_MIN_PATTERN_LEN 16 #define IWL_WOWLAN_MAX_PATTERN_LEN 128 -struct iwl_wowlan_pattern { +struct iwl_wowlan_pattern_v1 { u8 mask[IWL_WOWLAN_MAX_PATTERN_LEN / 8]; u8 pattern[IWL_WOWLAN_MAX_PATTERN_LEN]; u8 mask_size; @@ -227,7 +227,7 @@ struct iwl_wowlan_pattern { /** * struct iwl_wowlan_patterns_cmd - WoWLAN wakeup patterns */ -struct iwl_wowlan_patterns_cmd { +struct iwl_wowlan_patterns_cmd_v1 { /** * @n_patterns: number of patterns */ @@ -236,9 +236,129 @@ struct iwl_wowlan_patterns_cmd { /** * @patterns: the patterns, array length in @n_patterns */ - struct iwl_wowlan_pattern patterns[]; + struct iwl_wowlan_pattern_v1 patterns[]; } __packed; /* WOWLAN_PATTERN_ARRAY_API_S_VER_1 */ +#define IPV4_ADDR_SIZE 4 +#define IPV6_ADDR_SIZE 16 + +enum iwl_wowlan_pattern_type { + WOWLAN_PATTERN_TYPE_BITMASK, + WOWLAN_PATTERN_TYPE_IPV4_TCP_SYN, + WOWLAN_PATTERN_TYPE_IPV6_TCP_SYN, + WOWLAN_PATTERN_TYPE_IPV4_TCP_SYN_WILDCARD, + WOWLAN_PATTERN_TYPE_IPV6_TCP_SYN_WILDCARD, +}; /* WOWLAN_PATTERN_TYPE_API_E_VER_1 */ + +/** + * struct iwl_wowlan_ipv4_tcp_syn - WoWLAN IPv4 TCP SYN pattern data + */ +struct iwl_wowlan_ipv4_tcp_syn { + /** + * @src_addr: source IP address to match + */ + u8 src_addr[IPV4_ADDR_SIZE]; + + /** + * @dst_addr: destination IP address to match + */ + u8 dst_addr[IPV4_ADDR_SIZE]; + + /** + * @src_port: source TCP port to match + */ + __le16 src_port; + + /** + * @dst_port: destination TCP port to match + */ + __le16 dst_port; +} __packed; /* WOWLAN_IPV4_TCP_SYN_API_S_VER_1 */ + +/** + * struct iwl_wowlan_ipv6_tcp_syn - WoWLAN Ipv6 TCP SYN pattern data + */ +struct iwl_wowlan_ipv6_tcp_syn { + /** + * @src_addr: source IP address to match + */ + u8 src_addr[IPV6_ADDR_SIZE]; + + /** + * @dst_addr: destination IP address to match + */ + u8 dst_addr[IPV6_ADDR_SIZE]; + + /** + * @src_port: source TCP port to match + */ + __le16 src_port; + + /** + * @dst_port: destination TCP port to match + */ + __le16 dst_port; +} __packed; /* WOWLAN_IPV6_TCP_SYN_API_S_VER_1 */ + +/** + * union iwl_wowlan_pattern_data - Data for the different pattern types + * + * If wildcard addresses/ports are to be used, the union can be left + * undefined. + */ +union iwl_wowlan_pattern_data { + /** + * @bitmask: bitmask pattern data + */ + struct iwl_wowlan_pattern_v1 bitmask; + + /** + * @ipv4_tcp_syn: IPv4 TCP SYN pattern data + */ + struct iwl_wowlan_ipv4_tcp_syn ipv4_tcp_syn; + + /** + * @ipv6_tcp_syn: IPv6 TCP SYN pattern data + */ + struct iwl_wowlan_ipv6_tcp_syn ipv6_tcp_syn; +}; /* WOWLAN_PATTERN_API_U_VER_1 */ + +/** + * struct iwl_wowlan_pattern_v2 - Pattern entry for the WoWLAN wakeup patterns + */ +struct iwl_wowlan_pattern_v2 { + /** + * @pattern_type: defines the struct type to be used in the union + */ + u8 pattern_type; + + /** + * @reserved: reserved for alignment + */ + u8 reserved[3]; + + /** + * @u: the union containing the match data, or undefined for + * wildcard matches + */ + union iwl_wowlan_pattern_data u; +} __packed; /* WOWLAN_PATTERN_API_S_VER_2 */ + +/** + * struct iwl_wowlan_patterns_cmd - WoWLAN wakeup patterns command + */ +struct iwl_wowlan_patterns_cmd { + /** + * @n_patterns: number of patterns + */ + __le32 n_patterns; + + /** + * @patterns: the patterns, array length in @n_patterns + */ + struct iwl_wowlan_pattern_v2 patterns[]; +} __packed; /* WOWLAN_PATTERN_ARRAY_API_S_VER_2 */ + enum iwl_wowlan_wakeup_filters { IWL_WOWLAN_WAKEUP_MAGIC_PACKET = BIT(0), IWL_WOWLAN_WAKEUP_PATTERN_MATCH = BIT(1), @@ -383,7 +503,11 @@ enum iwl_wowlan_wakeup_reason { IWL_WOWLAN_WAKEUP_BY_D3_WAKEUP_HOST_TIMER = BIT(14), IWL_WOWLAN_WAKEUP_BY_RXFRAME_FILTERED_IN = BIT(15), IWL_WOWLAN_WAKEUP_BY_BEACON_FILTERED_IN = BIT(16), - + IWL_WAKEUP_BY_11W_UNPROTECTED_DEAUTH_OR_DISASSOC = BIT(17), + IWL_WAKEUP_BY_PATTERN_IPV4_TCP_SYN = BIT(18), + IWL_WAKEUP_BY_PATTERN_IPV4_TCP_SYN_WILDCARD = BIT(19), + IWL_WAKEUP_BY_PATTERN_IPV6_TCP_SYN = BIT(20), + IWL_WAKEUP_BY_PATTERN_IPV6_TCP_SYN_WILDCARD = BIT(21), }; /* WOWLAN_WAKE_UP_REASON_API_E_VER_2 */ struct iwl_wowlan_gtk_status_v1 { diff --git a/drivers/net/wireless/intel/iwlwifi/fw/api/dbg-tlv.h b/drivers/net/wireless/intel/iwlwifi/fw/api/dbg-tlv.h index 33858787817b0227df170ba774385dfcaa3f5fa8..f4202bc231a65d21e18450aba6716590f11c004a 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/api/dbg-tlv.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/api/dbg-tlv.h @@ -60,12 +60,13 @@ #include -/* +/** * struct iwl_fw_ini_header: Common Header for all debug group TLV's structures + * * @tlv_version: version info * @apply_point: &enum iwl_fw_ini_apply_point * @data: TLV data followed - **/ + */ struct iwl_fw_ini_header { __le32 tlv_version; __le32 apply_point; @@ -73,7 +74,7 @@ struct iwl_fw_ini_header { } __packed; /* FW_DEBUG_TLV_HEADER_S */ /** - * struct iwl_fw_ini_allocation_tlv - (IWL_FW_INI_TLV_TYPE_BUFFER_ALLOCATION) + * struct iwl_fw_ini_allocation_tlv - (IWL_UCODE_TLV_TYPE_BUFFER_ALLOCATION) * buffer allocation TLV - for debug * * @iwl_fw_ini_header: header @@ -84,7 +85,7 @@ struct iwl_fw_ini_header { * @max_fragments: the maximum allowed fragmentation in the desired memory * allocation above * @min_frag_size: the minimum allowed fragmentation size in bytes -*/ + */ struct iwl_fw_ini_allocation_tlv { struct iwl_fw_ini_header header; __le32 allocation_id; @@ -95,33 +96,52 @@ struct iwl_fw_ini_allocation_tlv { } __packed; /* FW_DEBUG_TLV_BUFFER_ALLOCATION_TLV_S_VER_1 */ /** - * struct iwl_fw_ini_hcmd (IWL_FW_INI_TLV_TYPE_HCMD) - * Generic Host command pass through TLV + * enum iwl_fw_ini_dbg_domain - debug domains + * allows to send host cmd or collect memory region if a given domain is enabled + * + * @IWL_FW_INI_DBG_DOMAIN_ALWAYS_ON: the default domain, always on + * @IWL_FW_INI_DBG_DOMAIN_REPORT_PS: power save domain + */ +enum iwl_fw_ini_dbg_domain { + IWL_FW_INI_DBG_DOMAIN_ALWAYS_ON = 0, + IWL_FW_INI_DBG_DOMAIN_REPORT_PS, +}; /* FW_DEBUG_TLV_DOMAIN_API_E_VER_1 */ + +/** + * struct iwl_fw_ini_hcmd * * @id: the debug configuration command type for instance: 0xf6 / 0xf5 / DHC * @group: the desired cmd group - * @padding: all zeros for dword alignment - * @data: all of the relevant command (0xf6/0xf5) to be sent -*/ + * @reserved: to align to FW struct + * @data: all of the relevant command data to be sent + */ struct iwl_fw_ini_hcmd { u8 id; u8 group; - __le16 padding; + __le16 reserved; u8 data[0]; -} __packed; /* FW_DEBUG_TLV_HCMD_DATA_S */ +} __packed; /* FW_DEBUG_TLV_HCMD_DATA_API_S_VER_1 */ /** - * struct iwl_fw_ini_hcmd_tlv + * struct iwl_fw_ini_hcmd_tlv - (IWL_UCODE_TLV_TYPE_HCMD) + * Generic Host command pass through TLV + * * @header: header + * @domain: send command only if the specific domain is enabled + * &enum iwl_fw_ini_dbg_domain + * @period_msec: period in which the hcmd will be sent to FW. Measured in msec + * (0 = one time command). * @hcmd: a variable length host-command to be sent to apply the configuration. */ struct iwl_fw_ini_hcmd_tlv { struct iwl_fw_ini_header header; + __le32 domain; + __le32 period_msec; struct iwl_fw_ini_hcmd hcmd; -} __packed; /* FW_DEBUG_TLV_HCMD_S_VER_1 */ +} __packed; /* FW_DEBUG_TLV_HCMD_API_S_VER_1 */ -/* - * struct iwl_fw_ini_debug_flow_tlv (IWL_FW_INI_TLV_TYPE_DEBUG_FLOW) +/** + * struct iwl_fw_ini_debug_flow_tlv - (IWL_UCODE_TLV_TYPE_DEBUG_FLOW) * * @header: header * @debug_flow_cfg: &enum iwl_fw_ini_debug_flow @@ -134,8 +154,20 @@ struct iwl_fw_ini_debug_flow_tlv { #define IWL_FW_INI_MAX_REGION_ID 64 #define IWL_FW_INI_MAX_NAME 32 +/** + * struct iwl_fw_ini_region_cfg_dhc - defines dhc response to dump. + * + * @id_and_grp: id and group of dhc response. + * @desc: dhc response descriptor. + */ +struct iwl_fw_ini_region_cfg_dhc { + __le32 id_and_grp; + __le32 desc; +} __packed; /* FW_DEBUG_TLV_REGION_DHC_API_S_VER_1 */ + /** * struct iwl_fw_ini_region_cfg_internal - meta data of internal memory region + * * @num_of_range: the amount of ranges in the region * @range_data_size: size of the data to read per range, in bytes. */ @@ -146,6 +178,7 @@ struct iwl_fw_ini_region_cfg_internal { /** * struct iwl_fw_ini_region_cfg_fifos - meta data of fifos region + * * @fid1: fifo id 1 - bitmap of lmac tx/rx fifos to include in the region * @fid2: fifo id 2 - bitmap of umac rx fifos to include in the region. * It is unused for tx. @@ -163,34 +196,43 @@ struct iwl_fw_ini_region_cfg_fifos { /** * struct iwl_fw_ini_region_cfg + * * @region_id: ID of this dump configuration * @region_type: &enum iwl_fw_ini_region_type - * @num_regions: amount of regions in the address array. + * @domain: dump this region only if the specific domain is enabled + * &enum iwl_fw_ini_dbg_domain * @name_len: name length * @name: file name to use for this region * @internal: used in case the region uses internal memory. * @allocation_id: For DRAM type field substitutes for allocation_id * @fifos: used in case of fifos region. + * @dhc_desc: dhc response descriptor. + * @notif_id_and_grp: dump this region only if the specific notification + * occurred. * @offset: offset to use for each memory base address * @start_addr: array of addresses. */ struct iwl_fw_ini_region_cfg { __le32 region_id; __le32 region_type; + __le32 domain; __le32 name_len; u8 name[IWL_FW_INI_MAX_NAME]; union { struct iwl_fw_ini_region_cfg_internal internal; __le32 allocation_id; struct iwl_fw_ini_region_cfg_fifos fifos; - }; + struct iwl_fw_ini_region_cfg_dhc dhc_desc; + __le32 notif_id_and_grp; + }; /* FW_DEBUG_TLV_REGION_EXT_INT_PARAMS_API_U_VER_1 */ __le32 offset; __le32 start_addr[]; -} __packed; /* FW_DEBUG_TLV_REGION_CONFIG_S */ +} __packed; /* FW_DEBUG_TLV_REGION_CONFIG_API_S_VER_1 */ /** - * struct iwl_fw_ini_region_tlv - (IWL_FW_INI_TLV_TYPE_REGION_CFG) - * DUMP sections define IDs and triggers that use those IDs TLV + * struct iwl_fw_ini_region_tlv - (IWL_UCODE_TLV_TYPE_REGIONS) + * defines memory regions to dump + * * @header: header * @num_regions: how many different region section and IDs are coming next * @region_config: list of dump configurations @@ -199,13 +241,12 @@ struct iwl_fw_ini_region_tlv { struct iwl_fw_ini_header header; __le32 num_regions; struct iwl_fw_ini_region_cfg region_config[]; -} __packed; /* FW_DEBUG_TLV_REGIONS_S_VER_1 */ +} __packed; /* FW_DEBUG_TLV_REGIONS_API_S_VER_1 */ /** - * struct iwl_fw_ini_trigger - (IWL_FW_INI_TLV_TYPE_DUMP_CFG) - * Region sections define IDs and triggers that use those IDs TLV + * struct iwl_fw_ini_trigger * - * @trigger_id: enum &iwl_fw_ini_tigger_id + * @trigger_id: &enum iwl_fw_ini_trigger_id * @override_trig: determines how apply trigger in case a trigger with the * same id is already in use. Using the first 2 bytes: * Byte 0: if 0, override trigger configuration, otherwise use the @@ -214,6 +255,7 @@ struct iwl_fw_ini_region_tlv { * existing trigger. * @dump_delay: delay from trigger fire to dump, in usec * @occurrences: max amount of times to be fired + * @reserved: to align to FW struct * @ignore_consec: ignore consecutive triggers, in usec * @force_restart: force FW restart * @multi_dut: initiate debug dump data on several DUTs @@ -226,17 +268,18 @@ struct iwl_fw_ini_trigger { __le32 override_trig; __le32 dump_delay; __le32 occurrences; + __le32 reserved; __le32 ignore_consec; __le32 force_restart; __le32 multi_dut; __le32 trigger_data; __le32 num_regions; __le32 data[]; -} __packed; /* FW_TLV_DEBUG_TRIGGER_CONFIG_S */ +} __packed; /* FW_TLV_DEBUG_TRIGGER_CONFIG_API_S_VER_1 */ /** - * struct iwl_fw_ini_trigger_tlv - (IWL_FW_INI_TLV_TYPE_TRIGGERS_CFG) - * DUMP sections define IDs and triggers that use those IDs TLV + * struct iwl_fw_ini_trigger_tlv - (IWL_UCODE_TLV_TYPE_TRIGGERS) + * Triggers that hold memory regions to dump in case a trigger fires * * @header: header * @num_triggers: how many different triggers section and IDs are coming next @@ -246,16 +289,18 @@ struct iwl_fw_ini_trigger_tlv { struct iwl_fw_ini_header header; __le32 num_triggers; struct iwl_fw_ini_trigger trigger_config[]; -} __packed; /* FW_TLV_DEBUG_TRIGGERS_S_VER_1 */ +} __packed; /* FW_TLV_DEBUG_TRIGGERS_API_S_VER_1 */ /** * enum iwl_fw_ini_trigger_id + * * @IWL_FW_TRIGGER_ID_FW_ASSERT: FW assert * @IWL_FW_TRIGGER_ID_FW_HW_ERROR: HW assert * @IWL_FW_TRIGGER_ID_FW_TFD_Q_HANG: TFD queue hang * @IWL_FW_TRIGGER_ID_FW_DEBUG_HOST_TRIGGER: FW debug notification - * @IWL_FW_TRIGGER_ID_FW_GENERIC_NOTIFOCATION: FW generic notification + * @IWL_FW_TRIGGER_ID_FW_GENERIC_NOTIFICATION: FW generic notification * @IWL_FW_TRIGGER_ID_USER_TRIGGER: User trigger + * @IWL_FW_TRIGGER_ID_PERIODIC_TRIGGER: triggers periodically * @IWL_FW_TRIGGER_ID_HOST_PEER_CLIENT_INACTIVITY: peer inactivity * @IWL_FW_TRIGGER_ID_HOST_TX_LATENCY_THRESHOLD_CROSSED: TX latency * threshold was crossed @@ -299,47 +344,51 @@ enum iwl_fw_ini_trigger_id { /* FW triggers */ IWL_FW_TRIGGER_ID_FW_DEBUG_HOST_TRIGGER = 4, - IWL_FW_TRIGGER_ID_FW_GENERIC_NOTIFOCATION = 5, + IWL_FW_TRIGGER_ID_FW_GENERIC_NOTIFICATION = 5, /* User trigger */ IWL_FW_TRIGGER_ID_USER_TRIGGER = 6, + /* periodic uses the data field for the interval time */ + IWL_FW_TRIGGER_ID_PERIODIC_TRIGGER = 7, + /* Host triggers */ - IWL_FW_TRIGGER_ID_HOST_PEER_CLIENT_INACTIVITY = 7, - IWL_FW_TRIGGER_ID_HOST_TX_LATENCY_THRESHOLD_CROSSED = 8, - IWL_FW_TRIGGER_ID_HOST_TX_RESPONSE_STATUS_FAILED = 9, - IWL_FW_TRIGGER_ID_HOST_OS_REQ_DEAUTH_PEER = 10, - IWL_FW_TRIGGER_ID_HOST_STOP_GO_REQUEST = 11, - IWL_FW_TRIGGER_ID_HOST_START_GO_REQUEST = 12, - IWL_FW_TRIGGER_ID_HOST_JOIN_GROUP_REQUEST = 13, - IWL_FW_TRIGGER_ID_HOST_SCAN_START = 14, - IWL_FW_TRIGGER_ID_HOST_SCAN_SUBMITTED = 15, - IWL_FW_TRIGGER_ID_HOST_SCAN_PARAMS = 16, - IWL_FW_TRIGGER_ID_HOST_CHECK_FOR_HANG = 17, - IWL_FW_TRIGGER_ID_HOST_BAR_RECEIVED = 18, - IWL_FW_TRIGGER_ID_HOST_AGG_TX_RESPONSE_STATUS_FAILED = 19, - IWL_FW_TRIGGER_ID_HOST_EAPOL_TX_RESPONSE_FAILED = 20, - IWL_FW_TRIGGER_ID_HOST_FAKE_TX_RESPONSE_SUSPECTED = 21, - IWL_FW_TRIGGER_ID_HOST_AUTH_REQ_FROM_ASSOC_CLIENT = 22, - IWL_FW_TRIGGER_ID_HOST_ROAM_COMPLETE = 23, - IWL_FW_TRIGGER_ID_HOST_AUTH_ASSOC_FAST_FAILED = 24, - IWL_FW_TRIGGER_ID_HOST_D3_START = 25, - IWL_FW_TRIGGER_ID_HOST_D3_END = 26, - IWL_FW_TRIGGER_ID_HOST_BSS_MISSED_BEACONS = 27, - IWL_FW_TRIGGER_ID_HOST_P2P_CLIENT_MISSED_BEACONS = 28, - IWL_FW_TRIGGER_ID_HOST_PEER_CLIENT_TX_FAILURES = 29, - IWL_FW_TRIGGER_ID_HOST_TX_WFD_ACTION_FRAME_FAILED = 30, - IWL_FW_TRIGGER_ID_HOST_AUTH_ASSOC_FAILED = 31, - IWL_FW_TRIGGER_ID_HOST_SCAN_COMPLETE = 32, - IWL_FW_TRIGGER_ID_HOST_SCAN_ABORT = 33, - IWL_FW_TRIGGER_ID_HOST_NIC_ALIVE = 34, - IWL_FW_TRIGGER_ID_HOST_CHANNEL_SWITCH_COMPLETE = 35, + IWL_FW_TRIGGER_ID_HOST_PEER_CLIENT_INACTIVITY = 8, + IWL_FW_TRIGGER_ID_HOST_TX_LATENCY_THRESHOLD_CROSSED = 9, + IWL_FW_TRIGGER_ID_HOST_TX_RESPONSE_STATUS_FAILED = 10, + IWL_FW_TRIGGER_ID_HOST_OS_REQ_DEAUTH_PEER = 11, + IWL_FW_TRIGGER_ID_HOST_STOP_GO_REQUEST = 12, + IWL_FW_TRIGGER_ID_HOST_START_GO_REQUEST = 13, + IWL_FW_TRIGGER_ID_HOST_JOIN_GROUP_REQUEST = 14, + IWL_FW_TRIGGER_ID_HOST_SCAN_START = 15, + IWL_FW_TRIGGER_ID_HOST_SCAN_SUBMITTED = 16, + IWL_FW_TRIGGER_ID_HOST_SCAN_PARAMS = 17, + IWL_FW_TRIGGER_ID_HOST_CHECK_FOR_HANG = 18, + IWL_FW_TRIGGER_ID_HOST_BAR_RECEIVED = 19, + IWL_FW_TRIGGER_ID_HOST_AGG_TX_RESPONSE_STATUS_FAILED = 20, + IWL_FW_TRIGGER_ID_HOST_EAPOL_TX_RESPONSE_FAILED = 21, + IWL_FW_TRIGGER_ID_HOST_FAKE_TX_RESPONSE_SUSPECTED = 22, + IWL_FW_TRIGGER_ID_HOST_AUTH_REQ_FROM_ASSOC_CLIENT = 23, + IWL_FW_TRIGGER_ID_HOST_ROAM_COMPLETE = 24, + IWL_FW_TRIGGER_ID_HOST_AUTH_ASSOC_FAST_FAILED = 25, + IWL_FW_TRIGGER_ID_HOST_D3_START = 26, + IWL_FW_TRIGGER_ID_HOST_D3_END = 27, + IWL_FW_TRIGGER_ID_HOST_BSS_MISSED_BEACONS = 28, + IWL_FW_TRIGGER_ID_HOST_P2P_CLIENT_MISSED_BEACONS = 29, + IWL_FW_TRIGGER_ID_HOST_PEER_CLIENT_TX_FAILURES = 30, + IWL_FW_TRIGGER_ID_HOST_TX_WFD_ACTION_FRAME_FAILED = 31, + IWL_FW_TRIGGER_ID_HOST_AUTH_ASSOC_FAILED = 32, + IWL_FW_TRIGGER_ID_HOST_SCAN_COMPLETE = 33, + IWL_FW_TRIGGER_ID_HOST_SCAN_ABORT = 34, + IWL_FW_TRIGGER_ID_HOST_NIC_ALIVE = 35, + IWL_FW_TRIGGER_ID_HOST_CHANNEL_SWITCH_COMPLETE = 36, IWL_FW_TRIGGER_ID_NUM, }; /* FW_DEBUG_TLV_TRIGGER_ID_E_VER_1 */ /** * enum iwl_fw_ini_apply_point + * * @IWL_FW_INI_APPLY_INVALID: invalid * @IWL_FW_INI_APPLY_EARLY: pre loading FW * @IWL_FW_INI_APPLY_AFTER_ALIVE: first cmd from host after alive @@ -360,6 +409,7 @@ enum iwl_fw_ini_apply_point { /** * enum iwl_fw_ini_allocation_id + * * @IWL_FW_INI_ALLOCATION_INVALID: invalid * @IWL_FW_INI_ALLOCATION_ID_DBGC1: allocation meant for DBGC1 configuration * @IWL_FW_INI_ALLOCATION_ID_DBGC2: allocation meant for DBGC2 configuration @@ -380,18 +430,22 @@ enum iwl_fw_ini_allocation_id { /** * enum iwl_fw_ini_buffer_location + * * @IWL_FW_INI_LOCATION_INVALID: invalid * @IWL_FW_INI_LOCATION_SRAM_PATH: SRAM location * @IWL_FW_INI_LOCATION_DRAM_PATH: DRAM location + * @IWL_FW_INI_LOCATION_NPK_PATH: NPK location */ enum iwl_fw_ini_buffer_location { IWL_FW_INI_LOCATION_INVALID, IWL_FW_INI_LOCATION_SRAM_PATH, IWL_FW_INI_LOCATION_DRAM_PATH, + IWL_FW_INI_LOCATION_NPK_PATH, }; /* FW_DEBUG_TLV_BUFFER_LOCATION_E_VER_1 */ /** * enum iwl_fw_ini_debug_flow + * * @IWL_FW_INI_DEBUG_INVALID: invalid * @IWL_FW_INI_DEBUG_DBTR_FLOW: undefined * @IWL_FW_INI_DEBUG_TB2DTF_FLOW: undefined @@ -404,6 +458,7 @@ enum iwl_fw_ini_debug_flow { /** * enum iwl_fw_ini_region_type + * * @IWL_FW_INI_REGION_INVALID: invalid * @IWL_FW_INI_REGION_DEVICE_MEMORY: device internal memory * @IWL_FW_INI_REGION_PERIPHERY_MAC: periphery registers of MAC @@ -416,6 +471,10 @@ enum iwl_fw_ini_debug_flow { * @IWL_FW_INI_REGION_RXF: RX fifo * @IWL_FW_INI_REGION_PAGING: paging memory * @IWL_FW_INI_REGION_CSR: CSR registers + * @IWL_FW_INI_REGION_NOTIFICATION: FW notification data + * @IWL_FW_INI_REGION_DHC: dhc response to dump + * @IWL_FW_INI_REGION_LMAC_ERROR_TABLE: lmac error table + * @IWL_FW_INI_REGION_UMAC_ERROR_TABLE: umac error table * @IWL_FW_INI_REGION_NUM: number of region types */ enum iwl_fw_ini_region_type { @@ -431,6 +490,10 @@ enum iwl_fw_ini_region_type { IWL_FW_INI_REGION_RXF, IWL_FW_INI_REGION_PAGING, IWL_FW_INI_REGION_CSR, + IWL_FW_INI_REGION_NOTIFICATION, + IWL_FW_INI_REGION_DHC, + IWL_FW_INI_REGION_LMAC_ERROR_TABLE, + IWL_FW_INI_REGION_UMAC_ERROR_TABLE, IWL_FW_INI_REGION_NUM }; /* FW_DEBUG_TLV_REGION_TYPE_E_VER_1 */ diff --git a/drivers/net/wireless/intel/iwlwifi/fw/api/location.h b/drivers/net/wireless/intel/iwlwifi/fw/api/location.h index 5dddb21c1c4dbc2f076fd24d06adcb6e0f164f51..8d78b0e671c0e4721dfddd06b7edcce1c242ec40 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/api/location.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/api/location.h @@ -674,6 +674,59 @@ struct iwl_tof_range_rsp_ap_entry_ntfy_v3 { __le32 papd_calib_output; } __packed; /* LOCATION_RANGE_RSP_AP_ETRY_NTFY_API_S_VER_3 */ +/** + * struct iwl_tof_range_rsp_ap_entry_ntfy_v4 - AP parameters (response) + * @bssid: BSSID of the AP + * @measure_status: current APs measurement status, one of + * &enum iwl_tof_entry_status. + * @measure_bw: Current AP Bandwidth: 0 20MHz, 1 40MHz, 2 80MHz + * @rtt: The Round Trip Time that took for the last measurement for + * current AP [pSec] + * @rtt_variance: The Variance of the RTT values measured for current AP + * @rtt_spread: The Difference between the maximum and the minimum RTT + * values measured for current AP in the current session [pSec] + * @rssi: RSSI as uploaded in the Channel Estimation notification + * @rssi_spread: The Difference between the maximum and the minimum RSSI values + * measured for current AP in the current session + * @last_burst: 1 if no more FTM sessions are scheduled for this responder + * @refusal_period: refusal period in case of + * @IWL_TOF_ENTRY_RESPONDER_CANNOT_COLABORATE [sec] + * @timestamp: The GP2 Clock [usec] where Channel Estimation notification was + * uploaded by the LMAC + * @start_tsf: measurement start time in TSF of the mac specified in the range + * request + * @rx_rate_n_flags: rate and flags of the last FTM frame received from this + * responder + * @tx_rate_n_flags: rate and flags of the last ack sent to this responder + * @t2t3_initiator: as calculated from the algo in the initiator + * @t1t4_responder: as calculated from the algo in the responder + * @common_calib: Calib val that was used in for this AP measurement + * @specific_calib: val that was used in for this AP measurement + * @papd_calib_output: The result of the tof papd calibration that was injected + * into the algorithm. + */ +struct iwl_tof_range_rsp_ap_entry_ntfy_v4 { + u8 bssid[ETH_ALEN]; + u8 measure_status; + u8 measure_bw; + __le32 rtt; + __le32 rtt_variance; + __le32 rtt_spread; + s8 rssi; + u8 rssi_spread; + u8 last_burst; + u8 refusal_period; + __le32 timestamp; + __le32 start_tsf; + __le32 rx_rate_n_flags; + __le32 tx_rate_n_flags; + __le32 t2t3_initiator; + __le32 t1t4_responder; + __le16 common_calib; + __le16 specific_calib; + __le32 papd_calib_output; +} __packed; /* LOCATION_RANGE_RSP_AP_ETRY_NTFY_API_S_VER_4 */ + /** * struct iwl_tof_range_rsp_ap_entry_ntfy - AP parameters (response) * @bssid: BSSID of the AP @@ -704,6 +757,8 @@ struct iwl_tof_range_rsp_ap_entry_ntfy_v3 { * @specific_calib: val that was used in for this AP measurement * @papd_calib_output: The result of the tof papd calibration that was injected * into the algorithm. + * @rttConfidence: a value between 0 - 31 that represents the rtt accuracy. + * @reserved: for alignment */ struct iwl_tof_range_rsp_ap_entry_ntfy { u8 bssid[ETH_ALEN]; @@ -725,7 +780,9 @@ struct iwl_tof_range_rsp_ap_entry_ntfy { __le16 common_calib; __le16 specific_calib; __le32 papd_calib_output; -} __packed; /* LOCATION_RANGE_RSP_AP_ETRY_NTFY_API_S_VER_4 */ + u8 rttConfidence; + u8 reserved[3]; +} __packed; /* LOCATION_RANGE_RSP_AP_ETRY_NTFY_API_S_VER_5 */ /** * enum iwl_tof_response_status - tof response status @@ -760,6 +817,22 @@ struct iwl_tof_range_rsp_ntfy_v5 { struct iwl_tof_range_rsp_ap_entry_ntfy_v3 ap[IWL_MVM_TOF_MAX_APS]; } __packed; /* LOCATION_RANGE_RSP_NTFY_API_S_VER_5 */ +/** + * struct iwl_tof_range_rsp_ntfy_v6 - ranging response notification + * @request_id: A Token ID of the corresponding Range request + * @num_of_aps: Number of APs results + * @last_report: 1 if no more FTM sessions are scheduled, 0 otherwise. + * @reserved: reserved + * @ap: per-AP data + */ +struct iwl_tof_range_rsp_ntfy_v6 { + u8 request_id; + u8 num_of_aps; + u8 last_report; + u8 reserved; + struct iwl_tof_range_rsp_ap_entry_ntfy_v4 ap[IWL_MVM_TOF_MAX_APS]; +} __packed; /* LOCATION_RANGE_RSP_NTFY_API_S_VER_6 */ + /** * struct iwl_tof_range_rsp_ntfy - ranging response notification * @request_id: A Token ID of the corresponding Range request @@ -774,7 +847,7 @@ struct iwl_tof_range_rsp_ntfy { u8 last_report; u8 reserved; struct iwl_tof_range_rsp_ap_entry_ntfy ap[IWL_MVM_TOF_MAX_APS]; -} __packed; /* LOCATION_RANGE_RSP_NTFY_API_S_VER_6 */ +} __packed; /* LOCATION_RANGE_RSP_NTFY_API_S_VER_7 */ #define IWL_MVM_TOF_MCSI_BUF_SIZE (245) /** diff --git a/drivers/net/wireless/intel/iwlwifi/fw/api/mac.h b/drivers/net/wireless/intel/iwlwifi/fw/api/mac.h index 941c50477003830773e285778e690193222395c0..85c5e367cbf1ccf1d7630bed1cfd416d503cd015 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/api/mac.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/api/mac.h @@ -541,6 +541,66 @@ enum iwl_he_htc_flags { #define IWL_HE_HTC_LINK_ADAP_UNSOLICITED (2 << IWL_HE_HTC_LINK_ADAP_POS) #define IWL_HE_HTC_LINK_ADAP_BOTH (3 << IWL_HE_HTC_LINK_ADAP_POS) +/** + * struct iwl_he_sta_context_cmd_v1 - configure FW to work with HE AP + * @sta_id: STA id + * @tid_limit: max num of TIDs in TX HE-SU multi-TID agg + * 0 - bad value, 1 - multi-tid not supported, 2..8 - tid limit + * @reserved1: reserved byte for future use + * @reserved2: reserved byte for future use + * @flags: see %iwl_11ax_sta_ctxt_flags + * @ref_bssid_addr: reference BSSID used by the AP + * @reserved0: reserved 2 bytes for aligning the ref_bssid_addr field to 8 bytes + * @htc_flags: which features are supported in HTC + * @frag_flags: frag support in A-MSDU + * @frag_level: frag support level + * @frag_max_num: max num of "open" MSDUs in the receiver (in power of 2) + * @frag_min_size: min frag size (except last frag) + * @pkt_ext: optional, exists according to PPE-present bit in the HE-PHY capa + * @bss_color: 11ax AP ID that is used in the HE SIG-A to mark inter BSS frame + * @htc_trig_based_pkt_ext: default PE in 4us units + * @frame_time_rts_th: HE duration RTS threshold, in units of 32us + * @rand_alloc_ecwmin: random CWmin = 2**ECWmin-1 + * @rand_alloc_ecwmax: random CWmax = 2**ECWmax-1 + * @reserved3: reserved byte for future use + * @trig_based_txf: MU EDCA Parameter set for the trigger based traffic queues + */ +struct iwl_he_sta_context_cmd_v1 { + u8 sta_id; + u8 tid_limit; + u8 reserved1; + u8 reserved2; + __le32 flags; + + /* The below fields are set via Multiple BSSID IE */ + u8 ref_bssid_addr[6]; + __le16 reserved0; + + /* The below fields are set via HE-capabilities IE */ + __le32 htc_flags; + + u8 frag_flags; + u8 frag_level; + u8 frag_max_num; + u8 frag_min_size; + + /* The below fields are set via PPE thresholds element */ + struct iwl_he_pkt_ext pkt_ext; + + /* The below fields are set via HE-Operation IE */ + u8 bss_color; + u8 htc_trig_based_pkt_ext; + __le16 frame_time_rts_th; + + /* Random access parameter set (i.e. RAPS) */ + u8 rand_alloc_ecwmin; + u8 rand_alloc_ecwmax; + __le16 reserved3; + + /* The below fields are set via MU EDCA parameter set element */ + struct iwl_he_backoff_conf trig_based_txf[AC_NUM]; +} __packed; /* STA_CONTEXT_DOT11AX_API_S_VER_1 */ + /** * struct iwl_he_sta_context_cmd - configure FW to work with HE AP * @sta_id: STA id @@ -564,6 +624,14 @@ enum iwl_he_htc_flags { * @rand_alloc_ecwmax: random CWmax = 2**ECWmax-1 * @reserved3: reserved byte for future use * @trig_based_txf: MU EDCA Parameter set for the trigger based traffic queues + * @max_bssid_indicator: indicator of the max bssid supported on the associated + * bss + * @bssid_index: index of the associated VAP + * @ema_ap: AP supports enhanced Multi BSSID advertisement + * @profile_periodicity: number of Beacon periods that are needed to receive the + * complete VAPs info + * @bssid_count: actual number of VAPs in the MultiBSS Set + * @reserved4: alignment */ struct iwl_he_sta_context_cmd { u8 sta_id; @@ -599,7 +667,14 @@ struct iwl_he_sta_context_cmd { /* The below fields are set via MU EDCA parameter set element */ struct iwl_he_backoff_conf trig_based_txf[AC_NUM]; -} __packed; /* STA_CONTEXT_DOT11AX_API_S */ + + u8 max_bssid_indicator; + u8 bssid_index; + u8 ema_ap; + u8 profile_periodicity; + u8 bssid_count; + u8 reserved4[3]; +} __packed; /* STA_CONTEXT_DOT11AX_API_S_VER_2 */ /** * struct iwl_he_monitor_cmd - configure air sniffer for HE diff --git a/drivers/net/wireless/intel/iwlwifi/fw/api/nvm-reg.h b/drivers/net/wireless/intel/iwlwifi/fw/api/nvm-reg.h index 93b392f0c6a4d1d7cd311852768f0c09ca30f969..97b49843e318526b66d84323acc375d2fb925e2d 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/api/nvm-reg.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/api/nvm-reg.h @@ -8,7 +8,7 @@ * Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH * Copyright(c) 2016 - 2017 Intel Deutschland GmbH - * Copyright (C) 2018 Intel Corporation + * Copyright(C) 2018 - 2019 Intel Corporation * * This program is free software; you can redistribute it and/or modify * it under the terms of version 2 of the GNU General Public License as @@ -31,7 +31,7 @@ * Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH * Copyright(c) 2016 - 2017 Intel Deutschland GmbH - * Copyright (C) 2018 Intel Corporation + * Copyright(C) 2018 - 2019 Intel Corporation * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -233,7 +233,8 @@ struct iwl_nvm_get_info_phy { __le32 rx_chains; } __packed; /* REGULATORY_NVM_GET_INFO_PHY_SKU_SECTION_S_VER_1 */ -#define IWL_NUM_CHANNELS (51) +#define IWL_NUM_CHANNELS_V1 51 +#define IWL_NUM_CHANNELS 110 /** * struct iwl_nvm_get_info_regulatory - regulatory information @@ -241,12 +242,38 @@ struct iwl_nvm_get_info_phy { * @channel_profile: regulatory data of this channel * @reserved: reserved */ -struct iwl_nvm_get_info_regulatory { +struct iwl_nvm_get_info_regulatory_v1 { __le32 lar_enabled; - __le16 channel_profile[IWL_NUM_CHANNELS]; + __le16 channel_profile[IWL_NUM_CHANNELS_V1]; __le16 reserved; } __packed; /* REGULATORY_NVM_GET_INFO_REGULATORY_S_VER_1 */ +/** + * struct iwl_nvm_get_info_regulatory - regulatory information + * @lar_enabled: is LAR enabled + * @n_channels: number of valid channels in the array + * @channel_profile: regulatory data of this channel + */ +struct iwl_nvm_get_info_regulatory { + __le32 lar_enabled; + __le32 n_channels; + __le32 channel_profile[IWL_NUM_CHANNELS]; +} __packed; /* REGULATORY_NVM_GET_INFO_REGULATORY_S_VER_2 */ + +/** + * struct iwl_nvm_get_info_rsp_v3 - response to get NVM data + * @general: general NVM data + * @mac_sku: data relating to MAC sku + * @phy_sku: data relating to PHY sku + * @regulatory: regulatory data + */ +struct iwl_nvm_get_info_rsp_v3 { + struct iwl_nvm_get_info_general general; + struct iwl_nvm_get_info_sku mac_sku; + struct iwl_nvm_get_info_phy phy_sku; + struct iwl_nvm_get_info_regulatory_v1 regulatory; +} __packed; /* REGULATORY_NVM_GET_INFO_RSP_API_S_VER_3 */ + /** * struct iwl_nvm_get_info_rsp - response to get NVM data * @general: general NVM data @@ -259,7 +286,7 @@ struct iwl_nvm_get_info_rsp { struct iwl_nvm_get_info_sku mac_sku; struct iwl_nvm_get_info_phy phy_sku; struct iwl_nvm_get_info_regulatory regulatory; -} __packed; /* REGULATORY_NVM_GET_INFO_RSP_API_S_VER_3 */ +} __packed; /* REGULATORY_NVM_GET_INFO_RSP_API_S_VER_4 */ /** * struct iwl_nvm_access_complete_cmd - NVM_ACCESS commands are completed diff --git a/drivers/net/wireless/intel/iwlwifi/fw/api/rx.h b/drivers/net/wireless/intel/iwlwifi/fw/api/rx.h index 6e8224ce890628268b1c60f6594eb87403ca25a6..d55312ef58c9c61e257a257b16af8ff74a7c80dc 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/api/rx.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/api/rx.h @@ -8,7 +8,7 @@ * Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH * Copyright(c) 2015 - 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * * This program is free software; you can redistribute it and/or modify * it under the terms of version 2 of the GNU General Public License as @@ -31,7 +31,7 @@ * Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH * Copyright(c) 2015 - 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -688,13 +688,6 @@ struct iwl_rx_mpdu_desc { #define IWL_RX_DESC_SIZE_V1 offsetofend(struct iwl_rx_mpdu_desc, v1) -#define IWL_CD_STTS_OPTIMIZED_POS 0 -#define IWL_CD_STTS_OPTIMIZED_MSK 0x01 -#define IWL_CD_STTS_TRANSFER_STATUS_POS 1 -#define IWL_CD_STTS_TRANSFER_STATUS_MSK 0x0E -#define IWL_CD_STTS_WIFI_STATUS_POS 4 -#define IWL_CD_STTS_WIFI_STATUS_MSK 0xF0 - #define RX_NO_DATA_CHAIN_A_POS 0 #define RX_NO_DATA_CHAIN_A_MSK (0xff << RX_NO_DATA_CHAIN_A_POS) #define RX_NO_DATA_CHAIN_B_POS 8 @@ -747,62 +740,6 @@ struct iwl_rx_no_data { __le32 rx_vec[2]; } __packed; /* RX_NO_DATA_NTFY_API_S_VER_1 */ -/** - * enum iwl_completion_desc_transfer_status - transfer status (bits 1-3) - * @IWL_CD_STTS_UNUSED: unused - * @IWL_CD_STTS_UNUSED_2: unused - * @IWL_CD_STTS_END_TRANSFER: successful transfer complete. - * In sniffer mode, when split is used, set in last CD completion. (RX) - * @IWL_CD_STTS_OVERFLOW: In sniffer mode, when using split - used for - * all CD completion. (RX) - * @IWL_CD_STTS_ABORTED: CR abort / close flow. (RX) - * @IWL_CD_STTS_ERROR: general error (RX) - */ -enum iwl_completion_desc_transfer_status { - IWL_CD_STTS_UNUSED, - IWL_CD_STTS_UNUSED_2, - IWL_CD_STTS_END_TRANSFER, - IWL_CD_STTS_OVERFLOW, - IWL_CD_STTS_ABORTED, - IWL_CD_STTS_ERROR, -}; - -/** - * enum iwl_completion_desc_wifi_status - wifi status (bits 4-7) - * @IWL_CD_STTS_VALID: the packet is valid (RX) - * @IWL_CD_STTS_FCS_ERR: frame check sequence error (RX) - * @IWL_CD_STTS_SEC_KEY_ERR: error handling the security key of rx (RX) - * @IWL_CD_STTS_DECRYPTION_ERR: error decrypting the frame (RX) - * @IWL_CD_STTS_DUP: duplicate packet (RX) - * @IWL_CD_STTS_ICV_MIC_ERR: MIC error (RX) - * @IWL_CD_STTS_INTERNAL_SNAP_ERR: problems removing the snap (RX) - * @IWL_CD_STTS_SEC_PORT_FAIL: security port fail (RX) - * @IWL_CD_STTS_BA_OLD_SN: block ack received old SN (RX) - * @IWL_CD_STTS_QOS_NULL: QoS null packet (RX) - * @IWL_CD_STTS_MAC_HDR_ERR: MAC header conversion error (RX) - * @IWL_CD_STTS_MAX_RETRANS: reached max number of retransmissions (TX) - * @IWL_CD_STTS_EX_LIFETIME: exceeded lifetime (TX) - * @IWL_CD_STTS_NOT_USED: completed but not used (RX) - * @IWL_CD_STTS_REPLAY_ERR: pn check failed, replay error (RX) - */ -enum iwl_completion_desc_wifi_status { - IWL_CD_STTS_VALID, - IWL_CD_STTS_FCS_ERR, - IWL_CD_STTS_SEC_KEY_ERR, - IWL_CD_STTS_DECRYPTION_ERR, - IWL_CD_STTS_DUP, - IWL_CD_STTS_ICV_MIC_ERR, - IWL_CD_STTS_INTERNAL_SNAP_ERR, - IWL_CD_STTS_SEC_PORT_FAIL, - IWL_CD_STTS_BA_OLD_SN, - IWL_CD_STTS_QOS_NULL, - IWL_CD_STTS_MAC_HDR_ERR, - IWL_CD_STTS_MAX_RETRANS, - IWL_CD_STTS_EX_LIFETIME, - IWL_CD_STTS_NOT_USED, - IWL_CD_STTS_REPLAY_ERR, -}; - struct iwl_frame_release { u8 baid; u8 reserved; diff --git a/drivers/net/wireless/intel/iwlwifi/fw/api/scan.h b/drivers/net/wireless/intel/iwlwifi/fw/api/scan.h index 890a939c463d60716bd6ea464481e6413c646da4..1a67a2a439ab4d9c4bb161f96a56a803eddacba6 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/api/scan.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/api/scan.h @@ -788,7 +788,53 @@ struct iwl_umac_scan_complete { __le32 reserved; } __packed; /* SCAN_COMPLETE_NTF_UMAC_API_S_VER_1 */ -#define SCAN_OFFLOAD_MATCHING_CHANNELS_LEN 5 +#define SCAN_OFFLOAD_MATCHING_CHANNELS_LEN_V1 5 +#define SCAN_OFFLOAD_MATCHING_CHANNELS_LEN 7 + +/** + * struct iwl_scan_offload_profile_match_v1 - match information + * @bssid: matched bssid + * @reserved: reserved + * @channel: channel where the match occurred + * @energy: energy + * @matching_feature: feature matches + * @matching_channels: bitmap of channels that matched, referencing + * the channels passed in the scan offload request. + */ +struct iwl_scan_offload_profile_match_v1 { + u8 bssid[ETH_ALEN]; + __le16 reserved; + u8 channel; + u8 energy; + u8 matching_feature; + u8 matching_channels[SCAN_OFFLOAD_MATCHING_CHANNELS_LEN_V1]; +} __packed; /* SCAN_OFFLOAD_PROFILE_MATCH_RESULTS_S_VER_1 */ + +/** + * struct iwl_scan_offload_profiles_query_v1 - match results query response + * @matched_profiles: bitmap of matched profiles, referencing the + * matches passed in the scan offload request + * @last_scan_age: age of the last offloaded scan + * @n_scans_done: number of offloaded scans done + * @gp2_d0u: GP2 when D0U occurred + * @gp2_invoked: GP2 when scan offload was invoked + * @resume_while_scanning: not used + * @self_recovery: obsolete + * @reserved: reserved + * @matches: array of match information, one for each match + */ +struct iwl_scan_offload_profiles_query_v1 { + __le32 matched_profiles; + __le32 last_scan_age; + __le32 n_scans_done; + __le32 gp2_d0u; + __le32 gp2_invoked; + u8 resume_while_scanning; + u8 self_recovery; + __le16 reserved; + struct iwl_scan_offload_profile_match_v1 matches[IWL_SCAN_MAX_PROFILES]; +} __packed; /* SCAN_OFFLOAD_PROFILES_QUERY_RSP_S_VER_2 */ + /** * struct iwl_scan_offload_profile_match - match information * @bssid: matched bssid @@ -797,7 +843,7 @@ struct iwl_umac_scan_complete { * @energy: energy * @matching_feature: feature matches * @matching_channels: bitmap of channels that matched, referencing - * the channels passed in tue scan offload request + * the channels passed in the scan offload request. */ struct iwl_scan_offload_profile_match { u8 bssid[ETH_ALEN]; @@ -806,7 +852,7 @@ struct iwl_scan_offload_profile_match { u8 energy; u8 matching_feature; u8 matching_channels[SCAN_OFFLOAD_MATCHING_CHANNELS_LEN]; -} __packed; /* SCAN_OFFLOAD_PROFILE_MATCH_RESULTS_S_VER_1 */ +} __packed; /* SCAN_OFFLOAD_PROFILE_MATCH_RESULTS_S_VER_2 */ /** * struct iwl_scan_offload_profiles_query - match results query response @@ -831,7 +877,7 @@ struct iwl_scan_offload_profiles_query { u8 self_recovery; __le16 reserved; struct iwl_scan_offload_profile_match matches[IWL_SCAN_MAX_PROFILES]; -} __packed; /* SCAN_OFFLOAD_PROFILES_QUERY_RSP_S_VER_2 */ +} __packed; /* SCAN_OFFLOAD_PROFILES_QUERY_RSP_S_VER_3 */ /** * struct iwl_umac_scan_iter_complete_notif - notifies end of scanning iteration diff --git a/drivers/net/wireless/intel/iwlwifi/fw/api/txq.h b/drivers/net/wireless/intel/iwlwifi/fw/api/txq.h index 6ac240b6eacec1a1874b679c98b9686faed3940e..73196cbc7fbefd8bb1511adda2ddbd7474664e2c 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/api/txq.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/api/txq.h @@ -8,6 +8,7 @@ * Copyright(c) 2007 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH * Copyright(c) 2016 - 2017 Intel Deutschland GmbH + * Copyright(c) 2019 Intel Corporation * * This program is free software; you can redistribute it and/or modify * it under the terms of version 2 of the GNU General Public License as @@ -30,6 +31,7 @@ * Copyright(c) 2005 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH * Copyright(c) 2016 - 2017 Intel Deutschland GmbH + * Copyright(c) 2019 Intel Corporation * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -133,6 +135,7 @@ enum iwl_tx_queue_cfg_actions { #define IWL_DEFAULT_QUEUE_SIZE 256 #define IWL_MGMT_QUEUE_SIZE 16 +#define IWL_CMD_QUEUE_SIZE 32 /** * struct iwl_tx_queue_cfg_cmd - txq hw scheduler config command * @sta_id: station id diff --git a/drivers/net/wireless/intel/iwlwifi/fw/dbg.c b/drivers/net/wireless/intel/iwlwifi/fw/dbg.c index d7380016f1c0d4f4d85fd9b063f7344f5bd954c0..5f52e40a29032dd290991324a8f31f512c99fd16 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/dbg.c +++ b/drivers/net/wireless/intel/iwlwifi/fw/dbg.c @@ -545,6 +545,7 @@ static const struct iwl_prph_range iwl_prph_dump_addr_22000[] = { { .start = 0x00a04590, .end = 0x00a04590 }, { .start = 0x00a04598, .end = 0x00a04598 }, { .start = 0x00a045c0, .end = 0x00a045f4 }, + { .start = 0x00a05c18, .end = 0x00a05c1c }, { .start = 0x00a0c000, .end = 0x00a0c018 }, { .start = 0x00a0c020, .end = 0x00a0c028 }, { .start = 0x00a0c038, .end = 0x00a0c094 }, @@ -557,6 +558,12 @@ static const struct iwl_prph_range iwl_prph_dump_addr_22000[] = { { .start = 0x00a0c1b0, .end = 0x00a0c1b8 }, }; +static const struct iwl_prph_range iwl_prph_dump_addr_ax210[] = { + { .start = 0x00d03c00, .end = 0x00d03c64 }, + { .start = 0x00d05c18, .end = 0x00d05c1c }, + { .start = 0x00d0c000, .end = 0x00d0c174 }, +}; + static void iwl_read_prph_block(struct iwl_trans *trans, u32 start, u32 len_bytes, __le32 *data) { @@ -675,7 +682,8 @@ static void iwl_fw_prph_handler(struct iwl_fw_runtime *fwrt, void *ptr, u32 range_len; if (fwrt->trans->cfg->device_family >= IWL_DEVICE_FAMILY_AX210) { - /* TODO */ + range_len = ARRAY_SIZE(iwl_prph_dump_addr_ax210); + handler(fwrt, iwl_prph_dump_addr_ax210, range_len, ptr); } else if (fwrt->trans->cfg->device_family >= IWL_DEVICE_FAMILY_22000) { range_len = ARRAY_SIZE(iwl_prph_dump_addr_22000); handler(fwrt, iwl_prph_dump_addr_22000, range_len, ptr); @@ -804,8 +812,8 @@ static void iwl_dump_paging(struct iwl_fw_runtime *fwrt, } static struct iwl_fw_error_dump_file * -_iwl_fw_error_dump(struct iwl_fw_runtime *fwrt, - struct iwl_fw_dump_ptrs *fw_error_dump) +iwl_fw_error_dump_file(struct iwl_fw_runtime *fwrt, + struct iwl_fw_dump_ptrs *fw_error_dump) { struct iwl_fw_error_dump_file *dump_file; struct iwl_fw_error_dump_data *dump_data; @@ -909,11 +917,8 @@ _iwl_fw_error_dump(struct iwl_fw_runtime *fwrt, dump_data->type = cpu_to_le32(IWL_FW_ERROR_DUMP_DEV_FW_INFO); dump_data->len = cpu_to_le32(sizeof(*dump_info)); dump_info = (void *)dump_data->data; - dump_info->device_family = - fwrt->trans->cfg->device_family == - IWL_DEVICE_FAMILY_7000 ? - cpu_to_le32(IWL_FW_ERROR_DUMP_FAMILY_7) : - cpu_to_le32(IWL_FW_ERROR_DUMP_FAMILY_8); + dump_info->hw_type = + cpu_to_le32(CSR_HW_REV_TYPE(fwrt->trans->hw_rev)); dump_info->hw_step = cpu_to_le32(CSR_HW_REV_STEP(fwrt->trans->hw_rev)); memcpy(dump_info->fw_human_readable, fwrt->fw->human_readable, @@ -967,10 +972,11 @@ _iwl_fw_error_dump(struct iwl_fw_runtime *fwrt, if (fifo_len) { iwl_fw_dump_rxf(fwrt, &dump_data); iwl_fw_dump_txf(fwrt, &dump_data); - if (radio_len) - iwl_read_radio_regs(fwrt, &dump_data); } + if (radio_len) + iwl_read_radio_regs(fwrt, &dump_data); + if (iwl_fw_dbg_type_on(fwrt, IWL_FW_ERROR_DUMP_ERROR_INFO) && fwrt->dump.desc) { dump_data->type = cpu_to_le32(IWL_FW_ERROR_DUMP_ERROR_INFO); @@ -1049,14 +1055,14 @@ static int iwl_dump_ini_prph_iter(struct iwl_fw_runtime *fwrt, { struct iwl_fw_ini_error_dump_range *range = range_ptr; __le32 *val = range->data; - u32 addr, prph_val, offset = le32_to_cpu(reg->offset); + u32 prph_val; + u32 addr = le32_to_cpu(reg->start_addr[idx]) + le32_to_cpu(reg->offset); int i; - range->start_addr = reg->start_addr[idx]; + range->start_addr = cpu_to_le64(addr); range->range_data_size = reg->internal.range_data_size; for (i = 0; i < le32_to_cpu(reg->internal.range_data_size); i += 4) { - addr = le32_to_cpu(range->start_addr) + i; - prph_val = iwl_read_prph(fwrt->trans, addr + offset); + prph_val = iwl_read_prph(fwrt->trans, addr + i); if (prph_val == 0x5a5a5a5a) return -EBUSY; *val++ = cpu_to_le32(prph_val); @@ -1071,16 +1077,13 @@ static int iwl_dump_ini_csr_iter(struct iwl_fw_runtime *fwrt, { struct iwl_fw_ini_error_dump_range *range = range_ptr; __le32 *val = range->data; - u32 addr, offset = le32_to_cpu(reg->offset); + u32 addr = le32_to_cpu(reg->start_addr[idx]) + le32_to_cpu(reg->offset); int i; - range->start_addr = reg->start_addr[idx]; + range->start_addr = cpu_to_le64(addr); range->range_data_size = reg->internal.range_data_size; - for (i = 0; i < le32_to_cpu(reg->internal.range_data_size); i += 4) { - addr = le32_to_cpu(range->start_addr) + i; - *val++ = cpu_to_le32(iwl_trans_read32(fwrt->trans, - addr + offset)); - } + for (i = 0; i < le32_to_cpu(reg->internal.range_data_size); i += 4) + *val++ = cpu_to_le32(iwl_trans_read32(fwrt->trans, addr + i)); return sizeof(*range) + le32_to_cpu(range->range_data_size); } @@ -1090,12 +1093,11 @@ static int iwl_dump_ini_dev_mem_iter(struct iwl_fw_runtime *fwrt, void *range_ptr, int idx) { struct iwl_fw_ini_error_dump_range *range = range_ptr; - u32 addr = le32_to_cpu(range->start_addr); - u32 offset = le32_to_cpu(reg->offset); + u32 addr = le32_to_cpu(reg->start_addr[idx]) + le32_to_cpu(reg->offset); - range->start_addr = reg->start_addr[idx]; + range->start_addr = cpu_to_le64(addr); range->range_data_size = reg->internal.range_data_size; - iwl_trans_read_mem_bytes(fwrt->trans, addr + offset, range->data, + iwl_trans_read_mem_bytes(fwrt->trans, addr, range->data, le32_to_cpu(reg->internal.range_data_size)); return sizeof(*range) + le32_to_cpu(range->range_data_size); @@ -1109,7 +1111,7 @@ iwl_dump_ini_paging_gen2_iter(struct iwl_fw_runtime *fwrt, struct iwl_fw_ini_error_dump_range *range = range_ptr; u32 page_size = fwrt->trans->init_dram.paging[idx].size; - range->start_addr = cpu_to_le32(idx); + range->start_addr = cpu_to_le64(idx); range->range_data_size = cpu_to_le32(page_size); memcpy(range->data, fwrt->trans->init_dram.paging[idx].block, page_size); @@ -1129,7 +1131,7 @@ static int iwl_dump_ini_paging_iter(struct iwl_fw_runtime *fwrt, dma_addr_t addr = fwrt->fw_paging_db[idx].fw_paging_phys; u32 page_size = fwrt->fw_paging_db[idx].fw_paging_size; - range->start_addr = cpu_to_le32(idx); + range->start_addr = cpu_to_le64(idx); range->range_data_size = cpu_to_le32(page_size); dma_sync_single_for_cpu(fwrt->trans->dev, addr, page_size, DMA_BIDIRECTIONAL); @@ -1152,7 +1154,7 @@ iwl_dump_ini_mon_dram_iter(struct iwl_fw_runtime *fwrt, if (start_addr == 0x5a5a5a5a) return -EBUSY; - range->start_addr = cpu_to_le32(start_addr); + range->start_addr = cpu_to_le64(start_addr); range->range_data_size = cpu_to_le32(fwrt->trans->fw_mon[idx].size); memcpy(range->data, fwrt->trans->fw_mon[idx].block, @@ -1228,10 +1230,11 @@ static int iwl_dump_ini_txf_iter(struct iwl_fw_runtime *fwrt, { struct iwl_fw_ini_fifo_error_dump_range *range = range_ptr; struct iwl_ini_txf_iter_data *iter; + struct iwl_fw_ini_error_dump_register *reg_dump = (void *)range->data; u32 offs = le32_to_cpu(reg->offset), addr; u32 registers_size = - le32_to_cpu(reg->fifos.num_of_registers) * sizeof(__le32); - __le32 *val = range->data; + le32_to_cpu(reg->fifos.num_of_registers) * sizeof(*reg_dump); + __le32 *data; unsigned long flags; int i; @@ -1249,11 +1252,18 @@ static int iwl_dump_ini_txf_iter(struct iwl_fw_runtime *fwrt, iwl_write_prph_no_grab(fwrt->trans, TXF_LARC_NUM + offs, iter->fifo); - /* read txf registers */ + /* + * read txf registers. for each register, write to the dump the + * register address and its value + */ for (i = 0; i < le32_to_cpu(reg->fifos.num_of_registers); i++) { addr = le32_to_cpu(reg->start_addr[i]) + offs; - *val++ = cpu_to_le32(iwl_read_prph_no_grab(fwrt->trans, addr)); + reg_dump->addr = cpu_to_le32(addr); + reg_dump->data = cpu_to_le32(iwl_read_prph_no_grab(fwrt->trans, + addr)); + + reg_dump++; } if (reg->fifos.header_only) { @@ -1270,8 +1280,9 @@ static int iwl_dump_ini_txf_iter(struct iwl_fw_runtime *fwrt, /* Read FIFO */ addr = TXF_READ_MODIFY_DATA + offs; - for (i = 0; i < iter->fifo_size; i += sizeof(__le32)) - *val++ = cpu_to_le32(iwl_read_prph_no_grab(fwrt->trans, addr)); + data = (void *)reg_dump; + for (i = 0; i < iter->fifo_size; i += sizeof(*data)) + *data++ = cpu_to_le32(iwl_read_prph_no_grab(fwrt->trans, addr)); out: iwl_trans_release_nic_access(fwrt->trans, &flags); @@ -1327,10 +1338,11 @@ static int iwl_dump_ini_rxf_iter(struct iwl_fw_runtime *fwrt, { struct iwl_fw_ini_fifo_error_dump_range *range = range_ptr; struct iwl_ini_rxf_data rxf_data; + struct iwl_fw_ini_error_dump_register *reg_dump = (void *)range->data; u32 offs = le32_to_cpu(reg->offset), addr; u32 registers_size = - le32_to_cpu(reg->fifos.num_of_registers) * sizeof(__le32); - __le32 *val = range->data; + le32_to_cpu(reg->fifos.num_of_registers) * sizeof(*reg_dump); + __le32 *data; unsigned long flags; int i; @@ -1341,17 +1353,22 @@ static int iwl_dump_ini_rxf_iter(struct iwl_fw_runtime *fwrt, if (!iwl_trans_grab_nic_access(fwrt->trans, &flags)) return -EBUSY; - offs += rxf_data.offset; - range->fifo_num = cpu_to_le32(rxf_data.fifo_num); range->num_of_registers = reg->fifos.num_of_registers; range->range_data_size = cpu_to_le32(rxf_data.size + registers_size); - /* read rxf registers */ + /* + * read rxf registers. for each register, write to the dump the + * register address and its value + */ for (i = 0; i < le32_to_cpu(reg->fifos.num_of_registers); i++) { addr = le32_to_cpu(reg->start_addr[i]) + offs; - *val++ = cpu_to_le32(iwl_read_prph_no_grab(fwrt->trans, addr)); + reg_dump->addr = cpu_to_le32(addr); + reg_dump->data = cpu_to_le32(iwl_read_prph_no_grab(fwrt->trans, + addr)); + + reg_dump++; } if (reg->fifos.header_only) { @@ -1359,6 +1376,12 @@ static int iwl_dump_ini_rxf_iter(struct iwl_fw_runtime *fwrt, goto out; } + /* + * region register have absolute value so apply rxf offset after + * reading the registers + */ + offs += rxf_data.offset; + /* Lock fence */ iwl_write_prph_no_grab(fwrt->trans, RXF_SET_FENCE_MODE + offs, 0x1); /* Set fence pointer to the same place like WR pointer */ @@ -1369,8 +1392,9 @@ static int iwl_dump_ini_rxf_iter(struct iwl_fw_runtime *fwrt, /* Read FIFO */ addr = RXF_FIFO_RD_FENCE_INC + offs; - for (i = 0; i < rxf_data.size; i += sizeof(__le32)) - *val++ = cpu_to_le32(iwl_read_prph_no_grab(fwrt->trans, addr)); + data = (void *)reg_dump; + for (i = 0; i < rxf_data.size; i += sizeof(*data)) + *data++ = cpu_to_le32(iwl_read_prph_no_grab(fwrt->trans, addr)); out: iwl_trans_release_nic_access(fwrt->trans, &flags); @@ -1384,32 +1408,86 @@ static void *iwl_dump_ini_mem_fill_header(struct iwl_fw_runtime *fwrt, { struct iwl_fw_ini_error_dump *dump = data; + dump->header.version = cpu_to_le32(IWL_INI_DUMP_MEM_VER); + return dump->ranges; } static void -*iwl_dump_ini_mon_dram_fill_header(struct iwl_fw_runtime *fwrt, - struct iwl_fw_ini_region_cfg *reg, - void *data) +*iwl_dump_ini_mon_fill_header(struct iwl_fw_runtime *fwrt, + struct iwl_fw_ini_region_cfg *reg, + struct iwl_fw_ini_monitor_dump *data, + u32 write_ptr_addr, u32 write_ptr_msk, + u32 cycle_cnt_addr, u32 cycle_cnt_msk) { - struct iwl_fw_ini_monitor_dram_dump *mon_dump = (void *)data; u32 write_ptr, cycle_cnt; unsigned long flags; if (!iwl_trans_grab_nic_access(fwrt->trans, &flags)) { - IWL_ERR(fwrt, "Failed to get DRAM monitor header\n"); + IWL_ERR(fwrt, "Failed to get monitor header\n"); return NULL; } - write_ptr = iwl_read_umac_prph_no_grab(fwrt->trans, - MON_BUFF_WRPTR_VER2); - cycle_cnt = iwl_read_umac_prph_no_grab(fwrt->trans, - MON_BUFF_CYCLE_CNT_VER2); + + write_ptr = iwl_read_prph_no_grab(fwrt->trans, write_ptr_addr); + cycle_cnt = iwl_read_prph_no_grab(fwrt->trans, cycle_cnt_addr); + iwl_trans_release_nic_access(fwrt->trans, &flags); - mon_dump->write_ptr = cpu_to_le32(write_ptr); - mon_dump->cycle_cnt = cpu_to_le32(cycle_cnt); + data->header.version = cpu_to_le32(IWL_INI_DUMP_MONITOR_VER); + data->write_ptr = cpu_to_le32(write_ptr & write_ptr_msk); + data->cycle_cnt = cpu_to_le32(cycle_cnt & cycle_cnt_msk); + + return data->ranges; +} + +static void +*iwl_dump_ini_mon_dram_fill_header(struct iwl_fw_runtime *fwrt, + struct iwl_fw_ini_region_cfg *reg, + void *data) +{ + struct iwl_fw_ini_monitor_dump *mon_dump = (void *)data; + u32 write_ptr_addr, write_ptr_msk, cycle_cnt_addr, cycle_cnt_msk; + + switch (fwrt->trans->cfg->device_family) { + case IWL_DEVICE_FAMILY_9000: + case IWL_DEVICE_FAMILY_22000: + write_ptr_addr = MON_BUFF_WRPTR_VER2; + write_ptr_msk = -1; + cycle_cnt_addr = MON_BUFF_CYCLE_CNT_VER2; + cycle_cnt_msk = -1; + break; + default: + IWL_ERR(fwrt, "Unsupported device family %d\n", + fwrt->trans->cfg->device_family); + return NULL; + } + + return iwl_dump_ini_mon_fill_header(fwrt, reg, mon_dump, write_ptr_addr, + write_ptr_msk, cycle_cnt_addr, + cycle_cnt_msk); +} + +static void +*iwl_dump_ini_mon_smem_fill_header(struct iwl_fw_runtime *fwrt, + struct iwl_fw_ini_region_cfg *reg, + void *data) +{ + struct iwl_fw_ini_monitor_dump *mon_dump = (void *)data; + const struct iwl_cfg *cfg = fwrt->trans->cfg; + + if (fwrt->trans->cfg->device_family != IWL_DEVICE_FAMILY_9000 && + fwrt->trans->cfg->device_family != IWL_DEVICE_FAMILY_22000) { + IWL_ERR(fwrt, "Unsupported device family %d\n", + fwrt->trans->cfg->device_family); + return NULL; + } + + return iwl_dump_ini_mon_fill_header(fwrt, reg, mon_dump, + cfg->fw_mon_smem_write_ptr_addr, + cfg->fw_mon_smem_write_ptr_msk, + cfg->fw_mon_smem_cycle_cnt_ptr_addr, + cfg->fw_mon_smem_cycle_cnt_ptr_msk); - return mon_dump->ranges; } static void *iwl_dump_ini_fifo_fill_header(struct iwl_fw_runtime *fwrt, @@ -1418,6 +1496,8 @@ static void *iwl_dump_ini_fifo_fill_header(struct iwl_fw_runtime *fwrt, { struct iwl_fw_ini_fifo_error_dump *dump = data; + dump->header.version = cpu_to_le32(IWL_INI_DUMP_FIFO_VER); + return dump->ranges; } @@ -1509,7 +1589,8 @@ static u32 iwl_dump_ini_paging_get_size(struct iwl_fw_runtime *fwrt, static u32 iwl_dump_ini_mon_dram_get_size(struct iwl_fw_runtime *fwrt, struct iwl_fw_ini_region_cfg *reg) { - u32 size = sizeof(struct iwl_fw_ini_monitor_dram_dump); + u32 size = sizeof(struct iwl_fw_ini_monitor_dump) + + sizeof(struct iwl_fw_ini_error_dump_range); if (fwrt->trans->num_blocks) size += fwrt->trans->fw_mon[0].size; @@ -1517,6 +1598,15 @@ static u32 iwl_dump_ini_mon_dram_get_size(struct iwl_fw_runtime *fwrt, return size; } +static u32 iwl_dump_ini_mon_smem_get_size(struct iwl_fw_runtime *fwrt, + struct iwl_fw_ini_region_cfg *reg) +{ + return sizeof(struct iwl_fw_ini_monitor_dump) + + iwl_dump_ini_mem_ranges(fwrt, reg) * + (sizeof(struct iwl_fw_ini_error_dump_range) + + le32_to_cpu(reg->internal.range_data_size)); +} + static u32 iwl_dump_ini_txf_get_size(struct iwl_fw_runtime *fwrt, struct iwl_fw_ini_region_cfg *reg) { @@ -1524,7 +1614,7 @@ static u32 iwl_dump_ini_txf_get_size(struct iwl_fw_runtime *fwrt, void *fifo_iter = fwrt->dump.fifo_iter; u32 size = 0; u32 fifo_hdr = sizeof(struct iwl_fw_ini_fifo_error_dump_range) + - le32_to_cpu(reg->fifos.num_of_registers) * sizeof(__le32); + le32_to_cpu(reg->fifos.num_of_registers) * sizeof(__le32) * 2; fwrt->dump.fifo_iter = &iter; while (iwl_ini_txf_iter(fwrt, reg)) { @@ -1547,7 +1637,7 @@ static u32 iwl_dump_ini_rxf_get_size(struct iwl_fw_runtime *fwrt, struct iwl_ini_rxf_data rx_data; u32 size = sizeof(struct iwl_fw_ini_fifo_error_dump) + sizeof(struct iwl_fw_ini_fifo_error_dump_range) + - le32_to_cpu(reg->fifos.num_of_registers) * sizeof(__le32); + le32_to_cpu(reg->fifos.num_of_registers) * sizeof(__le32) * 2; if (reg->fifos.header_only) return size; @@ -1584,27 +1674,31 @@ struct iwl_dump_ini_mem_ops { * @fwrt: fw runtime struct. * @data: dump memory data. * @reg: region to copy to the dump. + * @ops: memory dump operations. */ static void iwl_dump_ini_mem(struct iwl_fw_runtime *fwrt, - enum iwl_fw_ini_region_type type, struct iwl_fw_error_dump_data **data, struct iwl_fw_ini_region_cfg *reg, struct iwl_dump_ini_mem_ops *ops) { struct iwl_fw_ini_error_dump_header *header = (void *)(*data)->data; + u32 num_of_ranges, i, type = le32_to_cpu(reg->region_type); void *range; - u32 num_of_ranges, i; if (WARN_ON(!ops || !ops->get_num_of_ranges || !ops->get_size || !ops->fill_mem_hdr || !ops->fill_range)) return; + IWL_DEBUG_FW(fwrt, "WRT: collecting region: id=%d, type=%d\n", + le32_to_cpu(reg->region_id), type); + num_of_ranges = ops->get_num_of_ranges(fwrt, reg); (*data)->type = cpu_to_le32(type | INI_DUMP_BIT); (*data)->len = cpu_to_le32(ops->get_size(fwrt, reg)); + header->region_id = reg->region_id; header->num_of_ranges = cpu_to_le32(num_of_ranges); header->name_len = cpu_to_le32(min_t(int, IWL_FW_INI_MAX_NAME, le32_to_cpu(reg->name_len))); @@ -1612,7 +1706,8 @@ iwl_dump_ini_mem(struct iwl_fw_runtime *fwrt, range = ops->fill_mem_hdr(fwrt, reg, header); if (!range) { - IWL_ERR(fwrt, "Failed to fill region header: id=%d, type=%d\n", + IWL_ERR(fwrt, + "WRT: failed to fill region header: id=%d, type=%d\n", le32_to_cpu(reg->region_id), type); memset(*data, 0, le32_to_cpu((*data)->len)); return; @@ -1622,7 +1717,8 @@ iwl_dump_ini_mem(struct iwl_fw_runtime *fwrt, int range_size = ops->fill_range(fwrt, reg, range, i); if (range_size < 0) { - IWL_ERR(fwrt, "Failed to dump region: id=%d, type=%d\n", + IWL_ERR(fwrt, + "WRT: failed to dump region: id=%d, type=%d\n", le32_to_cpu(reg->region_id), type); memset(*data, 0, le32_to_cpu((*data)->len)); return; @@ -1643,23 +1739,30 @@ static int iwl_fw_ini_get_trigger_len(struct iwl_fw_runtime *fwrt, for (i = 0; i < le32_to_cpu(trigger->num_regions); i++) { u32 reg_id = le32_to_cpu(trigger->data[i]); struct iwl_fw_ini_region_cfg *reg; - enum iwl_fw_ini_region_type type; if (WARN_ON(reg_id >= ARRAY_SIZE(fwrt->dump.active_regs))) continue; reg = fwrt->dump.active_regs[reg_id]; - if (WARN(!reg, "Unassigned region %d\n", reg_id)) + if (!reg) { + IWL_WARN(fwrt, + "WRT: unassigned region id %d, skipping\n", + reg_id); continue; + } - type = le32_to_cpu(reg->region_type); - switch (type) { + /* currently the driver supports always on domain only */ + if (le32_to_cpu(reg->domain) != IWL_FW_INI_DBG_DOMAIN_ALWAYS_ON) + continue; + + switch (le32_to_cpu(reg->region_type)) { case IWL_FW_INI_REGION_DEVICE_MEMORY: case IWL_FW_INI_REGION_PERIPHERY_MAC: case IWL_FW_INI_REGION_PERIPHERY_PHY: case IWL_FW_INI_REGION_PERIPHERY_AUX: - case IWL_FW_INI_REGION_INTERNAL_BUFFER: case IWL_FW_INI_REGION_CSR: + case IWL_FW_INI_REGION_LMAC_ERROR_TABLE: + case IWL_FW_INI_REGION_UMAC_ERROR_TABLE: size += hdr_len + iwl_dump_ini_mem_get_size(fwrt, reg); break; case IWL_FW_INI_REGION_TXF: @@ -1668,7 +1771,7 @@ static int iwl_fw_ini_get_trigger_len(struct iwl_fw_runtime *fwrt, case IWL_FW_INI_REGION_RXF: size += hdr_len + iwl_dump_ini_rxf_get_size(fwrt, reg); break; - case IWL_FW_INI_REGION_PAGING: { + case IWL_FW_INI_REGION_PAGING: size += hdr_len; if (iwl_fw_dbg_is_paging_enabled(fwrt)) { size += iwl_dump_ini_paging_get_size(fwrt, reg); @@ -1677,13 +1780,16 @@ static int iwl_fw_ini_get_trigger_len(struct iwl_fw_runtime *fwrt, reg); } break; - } case IWL_FW_INI_REGION_DRAM_BUFFER: if (!fwrt->trans->num_blocks) break; size += hdr_len + iwl_dump_ini_mon_dram_get_size(fwrt, reg); break; + case IWL_FW_INI_REGION_INTERNAL_BUFFER: + size += hdr_len + + iwl_dump_ini_mon_smem_get_size(fwrt, reg); + break; case IWL_FW_INI_REGION_DRAM_IMR: /* Undefined yet */ default: @@ -1701,7 +1807,6 @@ static void iwl_fw_ini_dump_trigger(struct iwl_fw_runtime *fwrt, for (i = 0; i < num; i++) { u32 reg_id = le32_to_cpu(trigger->data[i]); - enum iwl_fw_ini_region_type type; struct iwl_fw_ini_region_cfg *reg; struct iwl_dump_ini_mem_ops ops; @@ -1713,15 +1818,19 @@ static void iwl_fw_ini_dump_trigger(struct iwl_fw_runtime *fwrt, if (!reg) continue; - type = le32_to_cpu(reg->region_type); - switch (type) { + /* currently the driver supports always on domain only */ + if (le32_to_cpu(reg->domain) != IWL_FW_INI_DBG_DOMAIN_ALWAYS_ON) + continue; + + switch (le32_to_cpu(reg->region_type)) { case IWL_FW_INI_REGION_DEVICE_MEMORY: - case IWL_FW_INI_REGION_INTERNAL_BUFFER: + case IWL_FW_INI_REGION_LMAC_ERROR_TABLE: + case IWL_FW_INI_REGION_UMAC_ERROR_TABLE: ops.get_num_of_ranges = iwl_dump_ini_mem_ranges; ops.get_size = iwl_dump_ini_mem_get_size; ops.fill_mem_hdr = iwl_dump_ini_mem_fill_header; ops.fill_range = iwl_dump_ini_dev_mem_iter; - iwl_dump_ini_mem(fwrt, type, data, reg, &ops); + iwl_dump_ini_mem(fwrt, data, reg, &ops); break; case IWL_FW_INI_REGION_PERIPHERY_MAC: case IWL_FW_INI_REGION_PERIPHERY_PHY: @@ -1730,16 +1839,23 @@ static void iwl_fw_ini_dump_trigger(struct iwl_fw_runtime *fwrt, ops.get_size = iwl_dump_ini_mem_get_size; ops.fill_mem_hdr = iwl_dump_ini_mem_fill_header; ops.fill_range = iwl_dump_ini_prph_iter; - iwl_dump_ini_mem(fwrt, type, data, reg, &ops); + iwl_dump_ini_mem(fwrt, data, reg, &ops); break; case IWL_FW_INI_REGION_DRAM_BUFFER: ops.get_num_of_ranges = iwl_dump_ini_mon_dram_ranges; ops.get_size = iwl_dump_ini_mon_dram_get_size; ops.fill_mem_hdr = iwl_dump_ini_mon_dram_fill_header; ops.fill_range = iwl_dump_ini_mon_dram_iter; - iwl_dump_ini_mem(fwrt, type, data, reg, &ops); + iwl_dump_ini_mem(fwrt, data, reg, &ops); + break; + case IWL_FW_INI_REGION_INTERNAL_BUFFER: + ops.get_num_of_ranges = iwl_dump_ini_mem_ranges; + ops.get_size = iwl_dump_ini_mon_smem_get_size; + ops.fill_mem_hdr = iwl_dump_ini_mon_smem_fill_header; + ops.fill_range = iwl_dump_ini_dev_mem_iter; + iwl_dump_ini_mem(fwrt, data, reg, &ops); break; - case IWL_FW_INI_REGION_PAGING: { + case IWL_FW_INI_REGION_PAGING: ops.fill_mem_hdr = iwl_dump_ini_mem_fill_header; if (iwl_fw_dbg_is_paging_enabled(fwrt)) { ops.get_num_of_ranges = @@ -1754,9 +1870,8 @@ static void iwl_fw_ini_dump_trigger(struct iwl_fw_runtime *fwrt, ops.fill_range = iwl_dump_ini_paging_gen2_iter; } - iwl_dump_ini_mem(fwrt, type, data, reg, &ops); + iwl_dump_ini_mem(fwrt, data, reg, &ops); break; - } case IWL_FW_INI_REGION_TXF: { struct iwl_ini_txf_iter_data iter = { .init = true }; void *fifo_iter = fwrt->dump.fifo_iter; @@ -1766,7 +1881,7 @@ static void iwl_fw_ini_dump_trigger(struct iwl_fw_runtime *fwrt, ops.get_size = iwl_dump_ini_txf_get_size; ops.fill_mem_hdr = iwl_dump_ini_fifo_fill_header; ops.fill_range = iwl_dump_ini_txf_iter; - iwl_dump_ini_mem(fwrt, type, data, reg, &ops); + iwl_dump_ini_mem(fwrt, data, reg, &ops); fwrt->dump.fifo_iter = fifo_iter; break; } @@ -1775,14 +1890,14 @@ static void iwl_fw_ini_dump_trigger(struct iwl_fw_runtime *fwrt, ops.get_size = iwl_dump_ini_rxf_get_size; ops.fill_mem_hdr = iwl_dump_ini_fifo_fill_header; ops.fill_range = iwl_dump_ini_rxf_iter; - iwl_dump_ini_mem(fwrt, type, data, reg, &ops); + iwl_dump_ini_mem(fwrt, data, reg, &ops); break; case IWL_FW_INI_REGION_CSR: ops.get_num_of_ranges = iwl_dump_ini_mem_ranges; ops.get_size = iwl_dump_ini_mem_get_size; ops.fill_mem_hdr = iwl_dump_ini_mem_fill_header; ops.fill_range = iwl_dump_ini_csr_iter; - iwl_dump_ini_mem(fwrt, type, data, reg, &ops); + iwl_dump_ini_mem(fwrt, data, reg, &ops); break; case IWL_FW_INI_REGION_DRAM_IMR: /* This is undefined yet */ @@ -1793,16 +1908,13 @@ static void iwl_fw_ini_dump_trigger(struct iwl_fw_runtime *fwrt, } static struct iwl_fw_error_dump_file * -_iwl_fw_error_ini_dump(struct iwl_fw_runtime *fwrt, - struct iwl_fw_dump_ptrs *fw_error_dump) +iwl_fw_error_ini_dump_file(struct iwl_fw_runtime *fwrt) { - int size, id = le32_to_cpu(fwrt->dump.desc->trig_desc.type); + int size; struct iwl_fw_error_dump_data *dump_data; struct iwl_fw_error_dump_file *dump_file; struct iwl_fw_ini_trigger *trigger; - - if (id == FW_DBG_TRIGGER_FW_ASSERT) - id = IWL_FW_TRIGGER_ID_FW_ASSERT; + enum iwl_fw_ini_trigger_id id = fwrt->dump.ini_trig_id; if (!iwl_fw_ini_trigger_on(fwrt, id)) return NULL; @@ -1819,8 +1931,6 @@ _iwl_fw_error_ini_dump(struct iwl_fw_runtime *fwrt, if (!dump_file) return NULL; - fw_error_dump->fwrt_ptr = dump_file; - dump_file->barker = cpu_to_le32(IWL_FW_ERROR_DUMP_BARKER); dump_data = (void *)dump_file->data; dump_file->file_len = cpu_to_le32(size); @@ -1830,47 +1940,27 @@ _iwl_fw_error_ini_dump(struct iwl_fw_runtime *fwrt, return dump_file; } -void iwl_fw_error_dump(struct iwl_fw_runtime *fwrt) +static void iwl_fw_error_dump(struct iwl_fw_runtime *fwrt) { - struct iwl_fw_dump_ptrs *fw_error_dump; + struct iwl_fw_dump_ptrs fw_error_dump = {}; struct iwl_fw_error_dump_file *dump_file; struct scatterlist *sg_dump_data; u32 file_len; u32 dump_mask = fwrt->fw->dbg.dump_mask; - IWL_DEBUG_INFO(fwrt, "WRT dump start\n"); - - /* there's no point in fw dump if the bus is dead */ - if (test_bit(STATUS_TRANS_DEAD, &fwrt->trans->status)) { - IWL_ERR(fwrt, "Skip fw error dump since bus is dead\n"); - goto out; - } - - fw_error_dump = kzalloc(sizeof(*fw_error_dump), GFP_KERNEL); - if (!fw_error_dump) - goto out; - - if (fwrt->trans->ini_valid) - dump_file = _iwl_fw_error_ini_dump(fwrt, fw_error_dump); - else - dump_file = _iwl_fw_error_dump(fwrt, fw_error_dump); - - if (!dump_file) { - kfree(fw_error_dump); + dump_file = iwl_fw_error_dump_file(fwrt, &fw_error_dump); + if (!dump_file) goto out; - } if (!fwrt->trans->ini_valid && fwrt->dump.monitor_only) dump_mask &= IWL_FW_ERROR_DUMP_FW_MONITOR; - if (!fwrt->trans->ini_valid) - fw_error_dump->trans_ptr = - iwl_trans_dump_data(fwrt->trans, dump_mask); - + fw_error_dump.trans_ptr = iwl_trans_dump_data(fwrt->trans, dump_mask); file_len = le32_to_cpu(dump_file->file_len); - fw_error_dump->fwrt_len = file_len; - if (fw_error_dump->trans_ptr) { - file_len += fw_error_dump->trans_ptr->len; + fw_error_dump.fwrt_len = file_len; + + if (fw_error_dump.trans_ptr) { + file_len += fw_error_dump.trans_ptr->len; dump_file->file_len = cpu_to_le32(file_len); } @@ -1878,27 +1968,49 @@ void iwl_fw_error_dump(struct iwl_fw_runtime *fwrt) if (sg_dump_data) { sg_pcopy_from_buffer(sg_dump_data, sg_nents(sg_dump_data), - fw_error_dump->fwrt_ptr, - fw_error_dump->fwrt_len, 0); - if (fw_error_dump->trans_ptr) + fw_error_dump.fwrt_ptr, + fw_error_dump.fwrt_len, 0); + if (fw_error_dump.trans_ptr) sg_pcopy_from_buffer(sg_dump_data, sg_nents(sg_dump_data), - fw_error_dump->trans_ptr->data, - fw_error_dump->trans_ptr->len, - fw_error_dump->fwrt_len); + fw_error_dump.trans_ptr->data, + fw_error_dump.trans_ptr->len, + fw_error_dump.fwrt_len); dev_coredumpsg(fwrt->trans->dev, sg_dump_data, file_len, GFP_KERNEL); } - vfree(fw_error_dump->fwrt_ptr); - vfree(fw_error_dump->trans_ptr); - kfree(fw_error_dump); + vfree(fw_error_dump.fwrt_ptr); + vfree(fw_error_dump.trans_ptr); out: iwl_fw_free_dump_desc(fwrt); clear_bit(IWL_FWRT_STATUS_DUMPING, &fwrt->status); - IWL_DEBUG_INFO(fwrt, "WRT dump done\n"); } -IWL_EXPORT_SYMBOL(iwl_fw_error_dump); + +static void iwl_fw_error_ini_dump(struct iwl_fw_runtime *fwrt) +{ + struct iwl_fw_error_dump_file *dump_file; + struct scatterlist *sg_dump_data; + u32 file_len; + + dump_file = iwl_fw_error_ini_dump_file(fwrt); + if (!dump_file) + goto out; + + file_len = le32_to_cpu(dump_file->file_len); + + sg_dump_data = alloc_sgtable(file_len); + if (sg_dump_data) { + sg_pcopy_from_buffer(sg_dump_data, sg_nents(sg_dump_data), + dump_file, file_len, 0); + dev_coredumpsg(fwrt->trans->dev, sg_dump_data, file_len, + GFP_KERNEL); + } + vfree(dump_file); +out: + fwrt->dump.ini_trig_id = IWL_FW_TRIGGER_ID_INVALID; + clear_bit(IWL_FWRT_STATUS_DUMPING, &fwrt->status); +} const struct iwl_fw_dump_desc iwl_dump_desc_assert = { .trig_desc = { @@ -1912,6 +2024,17 @@ int iwl_fw_dbg_collect_desc(struct iwl_fw_runtime *fwrt, bool monitor_only, unsigned int delay) { + u32 trig_type = le32_to_cpu(desc->trig_desc.type); + int ret; + + if (fwrt->trans->ini_valid) { + ret = iwl_fw_dbg_ini_collect(fwrt, trig_type); + if (!ret) + iwl_fw_free_dump_desc(fwrt); + + return ret; + } + if (test_and_set_bit(IWL_FWRT_STATUS_DUMPING, &fwrt->status)) return -EBUSY; @@ -1953,10 +2076,10 @@ int iwl_fw_dbg_error_collect(struct iwl_fw_runtime *fwrt, } IWL_EXPORT_SYMBOL(iwl_fw_dbg_error_collect); -int _iwl_fw_dbg_collect(struct iwl_fw_runtime *fwrt, - enum iwl_fw_dbg_trigger trig, - const char *str, size_t len, - struct iwl_fw_dbg_trigger_tlv *trigger) +int iwl_fw_dbg_collect(struct iwl_fw_runtime *fwrt, + enum iwl_fw_dbg_trigger trig, + const char *str, size_t len, + struct iwl_fw_dbg_trigger_tlv *trigger) { struct iwl_fw_dump_desc *desc; unsigned int delay = 0; @@ -1993,50 +2116,73 @@ int _iwl_fw_dbg_collect(struct iwl_fw_runtime *fwrt, return iwl_fw_dbg_collect_desc(fwrt, desc, monitor_only, delay); } -IWL_EXPORT_SYMBOL(_iwl_fw_dbg_collect); +IWL_EXPORT_SYMBOL(iwl_fw_dbg_collect); -int iwl_fw_dbg_collect(struct iwl_fw_runtime *fwrt, - u32 id, const char *str, size_t len) +int _iwl_fw_dbg_ini_collect(struct iwl_fw_runtime *fwrt, + enum iwl_fw_ini_trigger_id id) { - struct iwl_fw_dump_desc *desc; struct iwl_fw_ini_active_triggers *active; u32 occur, delay; - if (!fwrt->trans->ini_valid) - return _iwl_fw_dbg_collect(fwrt, id, str, len, NULL); - - if (id == FW_DBG_TRIGGER_USER) - id = IWL_FW_TRIGGER_ID_USER_TRIGGER; + if (WARN_ON(!iwl_fw_ini_trigger_on(fwrt, id))) + return -EINVAL; - active = &fwrt->dump.active_trigs[id]; + if (test_and_set_bit(IWL_FWRT_STATUS_DUMPING, &fwrt->status)) + return -EBUSY; - if (WARN_ON(!active->active)) + if (!iwl_fw_ini_trigger_on(fwrt, id)) { + IWL_WARN(fwrt, "WRT: Trigger %d is not active, aborting dump\n", + id); return -EINVAL; + } + active = &fwrt->dump.active_trigs[id]; delay = le32_to_cpu(active->trig->dump_delay); occur = le32_to_cpu(active->trig->occurrences); if (!occur) return 0; + active->trig->occurrences = cpu_to_le32(--occur); + if (le32_to_cpu(active->trig->force_restart)) { - IWL_WARN(fwrt, "Force restart: trigger %d fired.\n", id); + IWL_WARN(fwrt, "WRT: force restart: trigger %d fired.\n", id); iwl_force_nmi(fwrt->trans); return 0; } - desc = kzalloc(sizeof(*desc) + len, GFP_ATOMIC); - if (!desc) - return -ENOMEM; + if (test_and_set_bit(IWL_FWRT_STATUS_DUMPING, &fwrt->status)) + return -EBUSY; - active->trig->occurrences = cpu_to_le32(--occur); + fwrt->dump.ini_trig_id = id; - desc->len = len; - desc->trig_desc.type = cpu_to_le32(id); - memcpy(desc->trig_desc.data, str, len); + IWL_WARN(fwrt, "WRT: collecting data: ini trigger %d fired.\n", id); + + schedule_delayed_work(&fwrt->dump.wk, usecs_to_jiffies(delay)); - return iwl_fw_dbg_collect_desc(fwrt, desc, true, delay); + return 0; } -IWL_EXPORT_SYMBOL(iwl_fw_dbg_collect); +IWL_EXPORT_SYMBOL(_iwl_fw_dbg_ini_collect); + +int iwl_fw_dbg_ini_collect(struct iwl_fw_runtime *fwrt, u32 legacy_trigger_id) +{ + int id; + + switch (legacy_trigger_id) { + case FW_DBG_TRIGGER_FW_ASSERT: + case FW_DBG_TRIGGER_ALIVE_TIMEOUT: + case FW_DBG_TRIGGER_DRIVER: + id = IWL_FW_TRIGGER_ID_FW_ASSERT; + break; + case FW_DBG_TRIGGER_USER: + id = IWL_FW_TRIGGER_ID_USER_TRIGGER; + break; + default: + return -EIO; + } + + return _iwl_fw_dbg_ini_collect(fwrt, id); +} +IWL_EXPORT_SYMBOL(iwl_fw_dbg_ini_collect); int iwl_fw_dbg_collect_trig(struct iwl_fw_runtime *fwrt, struct iwl_fw_dbg_trigger_tlv *trigger, @@ -2064,8 +2210,8 @@ int iwl_fw_dbg_collect_trig(struct iwl_fw_runtime *fwrt, len = strlen(buf) + 1; } - ret = _iwl_fw_dbg_collect(fwrt, le32_to_cpu(trigger->id), buf, len, - trigger); + ret = iwl_fw_dbg_collect(fwrt, le32_to_cpu(trigger->id), buf, len, + trigger); if (ret) return ret; @@ -2139,9 +2285,20 @@ void iwl_fw_dbg_collect_sync(struct iwl_fw_runtime *fwrt) return; } + /* there's no point in fw dump if the bus is dead */ + if (test_bit(STATUS_TRANS_DEAD, &fwrt->trans->status)) { + IWL_ERR(fwrt, "Skip fw error dump since bus is dead\n"); + return; + } + iwl_fw_dbg_stop_recording(fwrt, ¶ms); - iwl_fw_error_dump(fwrt); + IWL_DEBUG_FW_INFO(fwrt, "WRT: data collection start\n"); + if (fwrt->trans->ini_valid) + iwl_fw_error_ini_dump(fwrt); + else + iwl_fw_error_dump(fwrt); + IWL_DEBUG_FW_INFO(fwrt, "WRT: data collection done\n"); /* start recording again if the firmware is not crashed */ if (!test_bit(STATUS_FW_ERROR, &fwrt->trans->status) && @@ -2211,12 +2368,14 @@ iwl_fw_dbg_buffer_allocation(struct iwl_fw_runtime *fwrt, u32 size) if (!virtual_addr) IWL_ERR(fwrt, "Failed to allocate debug memory\n"); + IWL_DEBUG_FW(trans, + "Allocated DRAM buffer[%d], size=0x%x\n", + trans->num_blocks, size); + trans->fw_mon[trans->num_blocks].block = virtual_addr; trans->fw_mon[trans->num_blocks].physical = phys_addr; trans->fw_mon[trans->num_blocks].size = size; trans->num_blocks++; - - IWL_DEBUG_FW(trans, "Allocated debug block of size %d\n", size); } static void iwl_fw_dbg_buffer_apply(struct iwl_fw_runtime *fwrt, @@ -2239,11 +2398,15 @@ static void iwl_fw_dbg_buffer_apply(struct iwl_fw_runtime *fwrt, if (buf_location == IWL_FW_INI_LOCATION_SRAM_PATH) { if (!WARN(pnt != IWL_FW_INI_APPLY_EARLY, - "Invalid apply point %d for SMEM buffer allocation", - pnt)) + "WRT: Invalid apply point %d for SMEM buffer allocation, aborting\n", + pnt)) { + IWL_DEBUG_FW(trans, + "WRT: applying SMEM buffer destination\n"); + /* set sram monitor by enabling bit 7 */ iwl_set_bit(fwrt->trans, CSR_HW_IF_CONFIG_REG, CSR_HW_IF_CONFIG_REG_BIT_MONITOR_SRAM); + } return; } @@ -2262,6 +2425,9 @@ static void iwl_fw_dbg_buffer_apply(struct iwl_fw_runtime *fwrt, if (trans->num_blocks == 1) return; + IWL_DEBUG_FW(trans, + "WRT: applying DRAM buffer[%d] destination\n", block_idx); + cmd->num_frags = cpu_to_le32(1); cmd->fragments[0].address = cpu_to_le64(trans->fw_mon[block_idx].physical); @@ -2273,7 +2439,8 @@ static void iwl_fw_dbg_buffer_apply(struct iwl_fw_runtime *fwrt, } static void iwl_fw_dbg_send_hcmd(struct iwl_fw_runtime *fwrt, - struct iwl_ucode_tlv *tlv) + struct iwl_ucode_tlv *tlv, + bool ext) { struct iwl_fw_ini_hcmd_tlv *hcmd_tlv = (void *)&tlv->data[0]; struct iwl_fw_ini_hcmd *data = &hcmd_tlv->hcmd; @@ -2285,6 +2452,14 @@ static void iwl_fw_dbg_send_hcmd(struct iwl_fw_runtime *fwrt, .data = { data->data, }, }; + /* currently the driver supports always on domain only */ + if (le32_to_cpu(hcmd_tlv->domain) != IWL_FW_INI_DBG_DOMAIN_ALWAYS_ON) + return; + + IWL_DEBUG_FW(fwrt, + "WRT: ext=%d. Sending host command id=0x%x, group=0x%x\n", + ext, data->id, data->group); + iwl_trans_send_cmd(fwrt->trans, &hcmd); } @@ -2294,24 +2469,32 @@ static void iwl_fw_dbg_update_regions(struct iwl_fw_runtime *fwrt, { void *iter = (void *)tlv->region_config; int i, size = le32_to_cpu(tlv->num_regions); + const char *err_st = + "WRT: ext=%d. Invalid region %s %d for apply point %d\n"; for (i = 0; i < size; i++) { struct iwl_fw_ini_region_cfg *reg = iter, **active; int id = le32_to_cpu(reg->region_id); u32 type = le32_to_cpu(reg->region_type); - if (WARN(id >= ARRAY_SIZE(fwrt->dump.active_regs), - "Invalid region id %d for apply point %d\n", id, pnt)) + if (WARN(id >= ARRAY_SIZE(fwrt->dump.active_regs), err_st, ext, + "id", id, pnt)) + break; + + if (WARN(type == 0 || type >= IWL_FW_INI_REGION_NUM, err_st, + ext, "type", type, pnt)) break; active = &fwrt->dump.active_regs[id]; if (*active) - IWL_WARN(fwrt->trans, "region TLV %d override\n", id); + IWL_WARN(fwrt->trans, + "WRT: ext=%d. Region id %d override\n", + ext, id); IWL_DEBUG_FW(fwrt, - "%s: apply point %d, activating region ID %d\n", - __func__, pnt, id); + "WRT: ext=%d. Activating region id %d\n", + ext, id); *active = reg; @@ -2319,7 +2502,15 @@ static void iwl_fw_dbg_update_regions(struct iwl_fw_runtime *fwrt, type == IWL_FW_INI_REGION_RXF) iter += le32_to_cpu(reg->fifos.num_of_registers) * sizeof(__le32); - else if (type != IWL_FW_INI_REGION_DRAM_BUFFER) + else if (type == IWL_FW_INI_REGION_DEVICE_MEMORY || + type == IWL_FW_INI_REGION_PERIPHERY_MAC || + type == IWL_FW_INI_REGION_PERIPHERY_PHY || + type == IWL_FW_INI_REGION_PERIPHERY_AUX || + type == IWL_FW_INI_REGION_INTERNAL_BUFFER || + type == IWL_FW_INI_REGION_PAGING || + type == IWL_FW_INI_REGION_CSR || + type == IWL_FW_INI_REGION_LMAC_ERROR_TABLE || + type == IWL_FW_INI_REGION_UMAC_ERROR_TABLE) iter += le32_to_cpu(reg->internal.num_of_ranges) * sizeof(__le32); @@ -2338,7 +2529,8 @@ static int iwl_fw_dbg_trig_realloc(struct iwl_fw_runtime *fwrt, ptr = krealloc(active->trig, size, GFP_KERNEL); if (!ptr) { - IWL_ERR(fwrt, "Failed to allocate memory for trigger %d\n", id); + IWL_ERR(fwrt, "WRT: Failed to allocate memory for trigger %d\n", + id); return -ENOMEM; } active->trig = ptr; @@ -2362,7 +2554,9 @@ static void iwl_fw_dbg_update_triggers(struct iwl_fw_runtime *fwrt, u32 trig_regs_size = le32_to_cpu(trig->num_regions) * sizeof(__le32); - if (WARN_ON(id >= ARRAY_SIZE(fwrt->dump.active_trigs))) + if (WARN(id >= ARRAY_SIZE(fwrt->dump.active_trigs), + "WRT: ext=%d. Invalid trigger id %d for apply point %d\n", + ext, id, apply_point)) break; active = &fwrt->dump.active_trigs[id]; @@ -2370,6 +2564,10 @@ static void iwl_fw_dbg_update_triggers(struct iwl_fw_runtime *fwrt, if (!active->active) { size_t trig_size = sizeof(*trig) + trig_regs_size; + IWL_DEBUG_FW(fwrt, + "WRT: ext=%d. Activating trigger %d\n", + ext, id); + if (iwl_fw_dbg_trig_realloc(fwrt, active, id, trig_size)) goto next; @@ -2388,8 +2586,16 @@ static void iwl_fw_dbg_update_triggers(struct iwl_fw_runtime *fwrt, int mem_to_add = trig_regs_size; if (region_override) { + IWL_DEBUG_FW(fwrt, + "WRT: ext=%d. Trigger %d regions override\n", + ext, id); + mem_to_add -= active_regs * sizeof(__le32); } else { + IWL_DEBUG_FW(fwrt, + "WRT: ext=%d. Trigger %d regions appending\n", + ext, id); + offset += active_regs; new_regs += active_regs; } @@ -2398,8 +2604,13 @@ static void iwl_fw_dbg_update_triggers(struct iwl_fw_runtime *fwrt, active->size + mem_to_add)) goto next; - if (conf_override) + if (conf_override) { + IWL_DEBUG_FW(fwrt, + "WRT: ext=%d. Trigger %d configuration override\n", + ext, id); + memcpy(active->trig, trig, sizeof(*trig)); + } memcpy(active->trig->data + offset, trig->data, trig_regs_size); @@ -2411,6 +2622,20 @@ static void iwl_fw_dbg_update_triggers(struct iwl_fw_runtime *fwrt, active->trig->occurrences = cpu_to_le32(-1); active->active = true; + + if (id == IWL_FW_TRIGGER_ID_PERIODIC_TRIGGER) { + u32 collect_interval = le32_to_cpu(trig->trigger_data); + + /* the minimum allowed interval is 50ms */ + if (collect_interval < 50) { + collect_interval = 50; + trig->trigger_data = + cpu_to_le32(collect_interval); + } + + mod_timer(&fwrt->dump.periodic_trig, + jiffies + msecs_to_jiffies(collect_interval)); + } next: iter += sizeof(*trig) + trig_regs_size; @@ -2440,11 +2665,11 @@ static void _iwl_fw_dbg_apply_point(struct iwl_fw_runtime *fwrt, case IWL_UCODE_TLV_TYPE_HCMD: if (pnt < IWL_FW_INI_APPLY_AFTER_ALIVE) { IWL_ERR(fwrt, - "Invalid apply point %x for host command\n", - pnt); + "WRT: ext=%d. Invalid apply point %d for host command\n", + ext, pnt); goto next; } - iwl_fw_dbg_send_hcmd(fwrt, tlv); + iwl_fw_dbg_send_hcmd(fwrt, tlv, ext); break; case IWL_UCODE_TLV_TYPE_REGIONS: iwl_fw_dbg_update_regions(fwrt, ini_tlv, ext, pnt); @@ -2455,7 +2680,9 @@ static void _iwl_fw_dbg_apply_point(struct iwl_fw_runtime *fwrt, case IWL_UCODE_TLV_TYPE_DEBUG_FLOW: break; default: - WARN_ONCE(1, "Invalid TLV %x for apply point\n", type); + WARN_ONCE(1, + "WRT: ext=%d. Invalid TLV 0x%x for apply point\n", + ext, type); break; } next: @@ -2469,6 +2696,8 @@ void iwl_fw_dbg_apply_point(struct iwl_fw_runtime *fwrt, void *data = &fwrt->trans->apply_points[apply_point]; int i; + IWL_DEBUG_FW(fwrt, "WRT: enabling apply point %d\n", apply_point); + if (apply_point == IWL_FW_INI_APPLY_EARLY) { for (i = 0; i < IWL_FW_INI_MAX_REGION_ID; i++) fwrt->dump.active_regs[i] = NULL; @@ -2487,8 +2716,34 @@ IWL_EXPORT_SYMBOL(iwl_fw_dbg_apply_point); void iwl_fwrt_stop_device(struct iwl_fw_runtime *fwrt) { + del_timer(&fwrt->dump.periodic_trig); iwl_fw_dbg_collect_sync(fwrt); iwl_trans_stop_device(fwrt->trans); } IWL_EXPORT_SYMBOL(iwl_fwrt_stop_device); + +void iwl_fw_dbg_periodic_trig_handler(struct timer_list *t) +{ + struct iwl_fw_runtime *fwrt; + enum iwl_fw_ini_trigger_id id = IWL_FW_TRIGGER_ID_PERIODIC_TRIGGER; + int ret; + typeof(fwrt->dump) *dump_ptr = container_of(t, typeof(fwrt->dump), + periodic_trig); + + fwrt = container_of(dump_ptr, typeof(*fwrt), dump); + + ret = _iwl_fw_dbg_ini_collect(fwrt, id); + if (!ret || ret == -EBUSY) { + struct iwl_fw_ini_trigger *trig = + fwrt->dump.active_trigs[id].trig; + u32 occur = le32_to_cpu(trig->occurrences); + u32 collect_interval = le32_to_cpu(trig->trigger_data); + + if (!occur) + return; + + mod_timer(&fwrt->dump.periodic_trig, + jiffies + msecs_to_jiffies(collect_interval)); + } +} diff --git a/drivers/net/wireless/intel/iwlwifi/fw/dbg.h b/drivers/net/wireless/intel/iwlwifi/fw/dbg.h index a199056234d3b5cf0aea64fcd4ff6d95afe85ba4..2a9e560a906bb6c287190f68cb91ee0981262e0d 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/dbg.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/dbg.h @@ -108,18 +108,17 @@ static inline void iwl_fw_free_dump_desc(struct iwl_fw_runtime *fwrt) fwrt->dump.umac_err_id = 0; } -void iwl_fw_error_dump(struct iwl_fw_runtime *fwrt); int iwl_fw_dbg_collect_desc(struct iwl_fw_runtime *fwrt, const struct iwl_fw_dump_desc *desc, bool monitor_only, unsigned int delay); int iwl_fw_dbg_error_collect(struct iwl_fw_runtime *fwrt, enum iwl_fw_dbg_trigger trig_type); -int _iwl_fw_dbg_collect(struct iwl_fw_runtime *fwrt, - enum iwl_fw_dbg_trigger trig, - const char *str, size_t len, - struct iwl_fw_dbg_trigger_tlv *trigger); +int _iwl_fw_dbg_ini_collect(struct iwl_fw_runtime *fwrt, + enum iwl_fw_ini_trigger_id id); +int iwl_fw_dbg_ini_collect(struct iwl_fw_runtime *fwrt, u32 legacy_trigger_id); int iwl_fw_dbg_collect(struct iwl_fw_runtime *fwrt, - u32 id, const char *str, size_t len); + enum iwl_fw_dbg_trigger trig, const char *str, + size_t len, struct iwl_fw_dbg_trigger_tlv *trigger); int iwl_fw_dbg_collect_trig(struct iwl_fw_runtime *fwrt, struct iwl_fw_dbg_trigger_tlv *trigger, const char *fmt, ...) __printf(3, 4); @@ -229,10 +228,8 @@ iwl_fw_ini_trigger_on(struct iwl_fw_runtime *fwrt, struct iwl_fw_ini_trigger *trig; u32 usec; - - - if (!fwrt->trans->ini_valid || id >= IWL_FW_TRIGGER_ID_NUM || - !fwrt->dump.active_trigs[id].active) + if (!fwrt->trans->ini_valid || id == IWL_FW_TRIGGER_ID_INVALID || + id >= IWL_FW_TRIGGER_ID_NUM || !fwrt->dump.active_trigs[id].active) return false; trig = fwrt->dump.active_trigs[id].trig; @@ -388,11 +385,13 @@ void iwl_fw_dbg_read_d3_debug_data(struct iwl_fw_runtime *fwrt); static inline void iwl_fw_flush_dump(struct iwl_fw_runtime *fwrt) { + del_timer(&fwrt->dump.periodic_trig); flush_delayed_work(&fwrt->dump.wk); } static inline void iwl_fw_cancel_dump(struct iwl_fw_runtime *fwrt) { + del_timer(&fwrt->dump.periodic_trig); cancel_delayed_work_sync(&fwrt->dump.wk); } @@ -461,4 +460,15 @@ static inline void iwl_fw_umac_set_alive_err_table(struct iwl_trans *trans, /* This bit is used to differentiate the legacy dump from the ini dump */ #define INI_DUMP_BIT BIT(31) +static inline void iwl_fw_error_collect(struct iwl_fw_runtime *fwrt) +{ + if (fwrt->trans->ini_valid && fwrt->trans->hw_error) { + _iwl_fw_dbg_ini_collect(fwrt, IWL_FW_TRIGGER_ID_FW_HW_ERROR); + fwrt->trans->hw_error = false; + } else { + iwl_fw_dbg_collect_desc(fwrt, &iwl_dump_desc_assert, false, 0); + } +} + +void iwl_fw_dbg_periodic_trig_handler(struct timer_list *t); #endif /* __iwl_fw_dbg_h__ */ diff --git a/drivers/net/wireless/intel/iwlwifi/fw/error-dump.h b/drivers/net/wireless/intel/iwlwifi/fw/error-dump.h index 9b5077bd46c399d3727b589edffdaa4e8b97d7e9..0feff4c33e39a256e8755a81f266c7de56bdf41d 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/error-dump.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/error-dump.h @@ -184,7 +184,7 @@ enum iwl_fw_error_dump_family { /** * struct iwl_fw_error_dump_info - info on the device / firmware - * @device_family: the family of the device (7 / 8) + * @hw_type: the type of the device * @hw_step: the step of the device * @fw_human_readable: human readable FW version * @dev_human_readable: name of the device @@ -196,7 +196,7 @@ enum iwl_fw_error_dump_family { * if the dump collection was not initiated by an assert, the value is 0 */ struct iwl_fw_error_dump_info { - __le32 device_family; + __le32 hw_type; __le32 hw_step; u8 fw_human_readable[FW_VER_HUMAN_READABLE_SZ]; u8 dev_human_readable[64]; @@ -211,6 +211,9 @@ struct iwl_fw_error_dump_info { * @fw_mon_wr_ptr: the position of the write pointer in the cyclic buffer * @fw_mon_base_ptr: base pointer of the data * @fw_mon_cycle_cnt: number of wraparounds + * @fw_mon_base_high_ptr: used in AX210 devices, the base adderss is 64 bit + * so fw_mon_base_ptr holds LSB 32 bits and fw_mon_base_high_ptr hold + * MSB 32 bits * @reserved: for future use * @data: captured data */ @@ -218,7 +221,8 @@ struct iwl_fw_error_dump_fw_mon { __le32 fw_mon_wr_ptr; __le32 fw_mon_base_ptr; __le32 fw_mon_cycle_cnt; - __le32 reserved[3]; + __le32 fw_mon_base_high_ptr; + __le32 reserved[2]; u8 data[]; } __packed; @@ -274,25 +278,33 @@ struct iwl_fw_error_dump_mem { u8 data[]; }; +#define IWL_INI_DUMP_MEM_VER 1 +#define IWL_INI_DUMP_MONITOR_VER 1 +#define IWL_INI_DUMP_FIFO_VER 1 + /** * struct iwl_fw_ini_error_dump_range - range of memory - * @start_addr: the start address of this range * @range_data_size: the size of this range, in bytes + * @start_addr: the start address of this range * @data: the actual memory */ struct iwl_fw_ini_error_dump_range { - __le32 start_addr; __le32 range_data_size; + __le64 start_addr; __le32 data[]; } __packed; /** * struct iwl_fw_ini_error_dump_header - ini region dump header + * @version: dump version + * @region_id: id of the region * @num_of_ranges: number of ranges in this region * @name_len: number of bytes allocated to the name string of this region * @name: name of the region */ struct iwl_fw_ini_error_dump_header { + __le32 version; + __le32 region_id; __le32 num_of_ranges; __le32 name_len; u8 name[IWL_FW_INI_MAX_NAME]; @@ -311,13 +323,24 @@ struct iwl_fw_ini_error_dump { /* This bit is used to differentiate between lmac and umac rxf */ #define IWL_RXF_UMAC_BIT BIT(31) +/** + * struct iwl_fw_ini_error_dump_register - ini register dump + * @addr: address of the register + * @data: data of the register + */ +struct iwl_fw_ini_error_dump_register { + __le32 addr; + __le32 data; +} __packed; + /** * struct iwl_fw_ini_fifo_error_dump_range - ini fifo range dump * @fifo_num: the fifo num. In case of rxf and umac rxf, set BIT(31) to * distinguish between lmac and umac * @num_of_registers: num of registers to dump, dword size each - * @range_data_size: the size of the registers and fifo data - * @data: fifo data + * @range_data_size: the size of the data + * @data: consist of + * num_of_registers * (register address + register value) + fifo data */ struct iwl_fw_ini_fifo_error_dump_range { __le32 fifo_num; @@ -351,13 +374,13 @@ struct iwl_fw_error_dump_rb { }; /** - * struct iwl_fw_ini_monitor_dram_dump - ini dram monitor dump + * struct iwl_fw_ini_monitor_dump - ini monitor dump * @header - header of the region - * @write_ptr - write pointer position in the dram + * @write_ptr - write pointer position in the buffer * @cycle_cnt - cycles count * @ranges - the memory ranges of this this region */ -struct iwl_fw_ini_monitor_dram_dump { +struct iwl_fw_ini_monitor_dump { struct iwl_fw_ini_error_dump_header header; __le32 write_ptr; __le32 cycle_cnt; diff --git a/drivers/net/wireless/intel/iwlwifi/fw/file.h b/drivers/net/wireless/intel/iwlwifi/fw/file.h index e06407dc088b14ab6b4a6593a678963654b82152..de9243d301352b45f8b91a4980f560ae3c90641d 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/file.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/file.h @@ -142,12 +142,14 @@ enum iwl_ucode_tlv_type { IWL_UCODE_TLV_FW_DBG_DEST = 38, IWL_UCODE_TLV_FW_DBG_CONF = 39, IWL_UCODE_TLV_FW_DBG_TRIGGER = 40, + IWL_UCODE_TLV_CMD_VERSIONS = 48, IWL_UCODE_TLV_FW_GSCAN_CAPA = 50, IWL_UCODE_TLV_FW_MEM_SEG = 51, IWL_UCODE_TLV_IML = 52, IWL_UCODE_TLV_UMAC_DEBUG_ADDRS = 54, IWL_UCODE_TLV_LMAC_DEBUG_ADDRS = 55, IWL_UCODE_TLV_FW_RECOVERY_INFO = 57, + IWL_UCODE_TLV_FW_FSEQ_VERSION = 60, IWL_UCODE_TLV_TYPE_BUFFER_ALLOCATION = IWL_UCODE_INI_TLV_GROUP + 0x1, IWL_UCODE_TLV_DEBUG_BASE = IWL_UCODE_TLV_TYPE_BUFFER_ALLOCATION, @@ -275,8 +277,15 @@ typedef unsigned int __bitwise iwl_ucode_tlv_api_t; * version of the beacon notification. * @IWL_UCODE_TLV_API_BEACON_FILTER_V4: This ucode supports v4 of * BEACON_FILTER_CONFIG_API_S_VER_4. + * @IWL_UCODE_TLV_API_REGULATORY_NVM_INFO: This ucode supports v4 of + * REGULATORY_NVM_GET_INFO_RSP_API_S. * @IWL_UCODE_TLV_API_FTM_NEW_RANGE_REQ: This ucode supports v7 of * LOCATION_RANGE_REQ_CMD_API_S and v6 of LOCATION_RANGE_RESP_NTFY_API_S. + * @IWL_UCODE_TLV_API_SCAN_OFFLOAD_CHANS: This ucode supports v2 of + * SCAN_OFFLOAD_PROFILE_MATCH_RESULTS_S and v3 of + * SCAN_OFFLOAD_PROFILES_QUERY_RSP_S. + * @IWL_UCODE_TLV_API_MBSSID_HE: This ucode supports v2 of + * STA_CONTEXT_DOT11AX_API_S * * @NUM_IWL_UCODE_TLV_API: number of bits used */ @@ -303,7 +312,12 @@ enum iwl_ucode_tlv_api { IWL_UCODE_TLV_API_REDUCE_TX_POWER = (__force iwl_ucode_tlv_api_t)45, IWL_UCODE_TLV_API_SHORT_BEACON_NOTIF = (__force iwl_ucode_tlv_api_t)46, IWL_UCODE_TLV_API_BEACON_FILTER_V4 = (__force iwl_ucode_tlv_api_t)47, + IWL_UCODE_TLV_API_REGULATORY_NVM_INFO = (__force iwl_ucode_tlv_api_t)48, IWL_UCODE_TLV_API_FTM_NEW_RANGE_REQ = (__force iwl_ucode_tlv_api_t)49, + IWL_UCODE_TLV_API_SCAN_OFFLOAD_CHANS = (__force iwl_ucode_tlv_api_t)50, + IWL_UCODE_TLV_API_MBSSID_HE = (__force iwl_ucode_tlv_api_t)52, + IWL_UCODE_TLV_API_WOWLAN_TCP_SYN_WAKE = (__force iwl_ucode_tlv_api_t)53, + IWL_UCODE_TLV_API_FTM_RTT_ACCURACY = (__force iwl_ucode_tlv_api_t)54, NUM_IWL_UCODE_TLV_API #ifdef __CHECKER__ @@ -353,6 +367,7 @@ typedef unsigned int __bitwise iwl_ucode_tlv_capa_t; * IWL_UCODE_TLV_CAPA_CHANNEL_SWITCH_CMD: firmware supports CSA command * @IWL_UCODE_TLV_CAPA_ULTRA_HB_CHANNELS: firmware supports ultra high band * (6 GHz). + * @IWL_UCODE_TLV_CAPA_CS_MODIFY: firmware supports modify action CSA command * @IWL_UCODE_TLV_CAPA_EXTENDED_DTS_MEASURE: extended DTS measurement * @IWL_UCODE_TLV_CAPA_SHORT_PM_TIMEOUTS: supports short PM timeouts * @IWL_UCODE_TLV_CAPA_BT_MPLUT_SUPPORT: supports bt-coex Multi-priority LUT @@ -423,6 +438,7 @@ enum iwl_ucode_tlv_capa { IWL_UCODE_TLV_CAPA_CHANNEL_SWITCH_CMD = (__force iwl_ucode_tlv_capa_t)46, IWL_UCODE_TLV_CAPA_ULTRA_HB_CHANNELS = (__force iwl_ucode_tlv_capa_t)48, IWL_UCODE_TLV_CAPA_FTM_CALIBRATED = (__force iwl_ucode_tlv_capa_t)47, + IWL_UCODE_TLV_CAPA_CS_MODIFY = (__force iwl_ucode_tlv_capa_t)49, /* set 2 */ IWL_UCODE_TLV_CAPA_EXTENDED_DTS_MEASURE = (__force iwl_ucode_tlv_capa_t)64, @@ -928,4 +944,20 @@ struct iwl_fw_dbg_conf_tlv { struct iwl_fw_dbg_conf_hcmd hcmd; } __packed; +#define IWL_FW_CMD_VER_UNKNOWN 99 + +/** + * struct iwl_fw_cmd_version - firmware command version entry + * @cmd: command ID + * @group: group ID + * @cmd_ver: command version + * @notif_ver: notification version + */ +struct iwl_fw_cmd_version { + u8 cmd; + u8 group; + u8 cmd_ver; + u8 notif_ver; +} __packed; + #endif /* __iwl_fw_file_h__ */ diff --git a/drivers/net/wireless/intel/iwlwifi/fw/img.h b/drivers/net/wireless/intel/iwlwifi/fw/img.h index f4c5a4d732068e7a7be08dae233b8cd25186e7d8..18ca5f152be6e9d9ffe07454237983b2a3d10be7 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/img.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/img.h @@ -8,7 +8,7 @@ * Copyright(c) 2008 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH * Copyright(c) 2016 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * * This program is free software; you can redistribute it and/or modify * it under the terms of version 2 of the GNU General Public License as @@ -31,7 +31,7 @@ * Copyright(c) 2005 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH * Copyright(c) 2016 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -109,6 +109,9 @@ struct iwl_ucode_capabilities { u32 error_log_size; unsigned long _api[BITS_TO_LONGS(NUM_IWL_UCODE_TLV_API)]; unsigned long _capa[BITS_TO_LONGS(NUM_IWL_UCODE_TLV_CAPA)]; + + const struct iwl_fw_cmd_version *cmd_versions; + u32 n_cmd_versions; }; static inline bool diff --git a/drivers/net/wireless/intel/iwlwifi/fw/init.c b/drivers/net/wireless/intel/iwlwifi/fw/init.c index 12310e3d2fc5aa7b544b08c95ad3086c3de799c1..4435c0ce301308ddd1549cd34053ccc23ad0d774 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/init.c +++ b/drivers/net/wireless/intel/iwlwifi/fw/init.c @@ -76,6 +76,8 @@ void iwl_fw_runtime_init(struct iwl_fw_runtime *fwrt, struct iwl_trans *trans, fwrt->ops_ctx = ops_ctx; INIT_DELAYED_WORK(&fwrt->dump.wk, iwl_fw_error_dump_wk); iwl_fwrt_dbgfs_register(fwrt, dbgfs_dir); + timer_setup(&fwrt->dump.periodic_trig, + iwl_fw_dbg_periodic_trig_handler, 0); } IWL_EXPORT_SYMBOL(iwl_fw_runtime_init); diff --git a/drivers/net/wireless/intel/iwlwifi/fw/runtime.h b/drivers/net/wireless/intel/iwlwifi/fw/runtime.h index a5fe1a8ca426e6eb0ffc2e26dcb00b2256f3ae47..a6402a0b38540050510ee9293b8bb3964b244986 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/runtime.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/runtime.h @@ -145,6 +145,8 @@ struct iwl_fw_runtime { u32 lmac_err_id[MAX_NUM_LMAC]; u32 umac_err_id; void *fifo_iter; + enum iwl_fw_ini_trigger_id ini_trig_id; + struct timer_list periodic_trig; } dump; #ifdef CONFIG_IWLWIFI_DEBUGFS struct { diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-config.h b/drivers/net/wireless/intel/iwlwifi/iwl-config.h index 93070848280a4e425f95de9842b5bbc42000fcfb..f3e69edf89071b5823caa6ed76bce52ed5160247 100644 --- a/drivers/net/wireless/intel/iwlwifi/iwl-config.h +++ b/drivers/net/wireless/intel/iwlwifi/iwl-config.h @@ -383,6 +383,9 @@ struct iwl_csr_params { * @bisr_workaround: BISR hardware workaround (for 22260 series devices) * @min_txq_size: minimum number of slots required in a TX queue * @umac_prph_offset: offset to add to UMAC periphery address + * @uhb_supported: ultra high band channels supported + * @min_256_ba_txq_size: minimum number of slots required in a TX queue which + * supports 256 BA aggregation * * We enable the driver to be backward compatible wrt. hardware features. * API differences in uCode shouldn't be handled here but through TLVs @@ -433,7 +436,8 @@ struct iwl_cfg { gen2:1, cdb:1, dbgc_supported:1, - bisr_workaround:1; + bisr_workaround:1, + uhb_supported:1; u8 valid_tx_ant; u8 valid_rx_ant; u8 non_shared_ant; @@ -450,6 +454,12 @@ struct iwl_cfg { u32 d3_debug_data_length; u32 min_txq_size; u32 umac_prph_offset; + u32 fw_mon_smem_write_ptr_addr; + u32 fw_mon_smem_write_ptr_msk; + u32 fw_mon_smem_cycle_cnt_ptr_addr; + u32 fw_mon_smem_cycle_cnt_ptr_msk; + u32 gp2_reg_addr; + u32 min_256_ba_txq_size; }; extern const struct iwl_csr_params iwl_csr_v1; @@ -568,11 +578,11 @@ extern const struct iwl_cfg iwl22000_2ax_cfg_qnj_hr_b0_f0; extern const struct iwl_cfg iwl22000_2ax_cfg_qnj_hr_b0; extern const struct iwl_cfg iwl9560_2ac_cfg_qnj_jf_b0; extern const struct iwl_cfg iwl22000_2ax_cfg_qnj_hr_a0; -extern const struct iwl_cfg iwl22560_2ax_cfg_su_cdb; extern const struct iwl_cfg iwlax210_2ax_cfg_so_jf_a0; extern const struct iwl_cfg iwlax210_2ax_cfg_so_hr_a0; extern const struct iwl_cfg iwlax210_2ax_cfg_so_gf_a0; extern const struct iwl_cfg iwlax210_2ax_cfg_ty_gf_a0; +extern const struct iwl_cfg iwlax210_2ax_cfg_so_gf4_a0; #endif /* CPTCFG_IWLMVM || CPTCFG_IWLFMAC */ #endif /* __IWL_CONFIG_H__ */ diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-csr.h b/drivers/net/wireless/intel/iwlwifi/iwl-csr.h index e539bc94eff7fdcee8e4c979fe0581710768b275..55355484600936c73fe1db54848e432e440c4116 100644 --- a/drivers/net/wireless/intel/iwlwifi/iwl-csr.h +++ b/drivers/net/wireless/intel/iwlwifi/iwl-csr.h @@ -8,7 +8,7 @@ * Copyright(c) 2005 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH * Copyright(c) 2016 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * * This program is free software; you can redistribute it and/or modify * it under the terms of version 2 of the GNU General Public License as @@ -30,7 +30,7 @@ * * Copyright(c) 2005 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -290,6 +290,7 @@ /* HW REV */ #define CSR_HW_REV_DASH(_val) (((_val) & 0x0000003) >> 0) #define CSR_HW_REV_STEP(_val) (((_val) & 0x000000C) >> 2) +#define CSR_HW_REV_TYPE(_val) (((_val) & 0x000FFF0) >> 4) /* HW RFID */ #define CSR_HW_RFID_FLAVOR(_val) (((_val) & 0x000000F) >> 0) @@ -337,6 +338,7 @@ enum { #define CSR_HW_RF_ID_TYPE_HR (0x0010A000) #define CSR_HW_RF_ID_TYPE_HRCDB (0x00109F00) #define CSR_HW_RF_ID_TYPE_GF (0x0010D000) +#define CSR_HW_RF_ID_TYPE_GF4 (0x0010E000) /* HW_RF CHIP ID */ #define CSR_HW_RF_ID_TYPE_CHIP_ID(_val) (((_val) >> 12) & 0xFFF) diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-dbg-tlv.c b/drivers/net/wireless/intel/iwlwifi/iwl-dbg-tlv.c index c7070760a10aa2d9b1e8bc465abbaa31c459c879..ba66f7fba0645962e27c3c951477920d82332f5f 100644 --- a/drivers/net/wireless/intel/iwlwifi/iwl-dbg-tlv.c +++ b/drivers/net/wireless/intel/iwlwifi/iwl-dbg-tlv.c @@ -5,7 +5,7 @@ * * GPL LICENSE SUMMARY * - * Copyright (C) 2018 Intel Corporation + * Copyright (C) 2018 - 2019 Intel Corporation * * This program is free software; you can redistribute it and/or modify * it under the terms of version 2 of the GNU General Public License as @@ -28,7 +28,7 @@ * * BSD LICENSE * - * Copyright (C) 2018 Intel Corporation + * Copyright (C) 2018 - 2019 Intel Corporation * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -73,6 +73,9 @@ void iwl_fw_dbg_copy_tlv(struct iwl_trans *trans, struct iwl_ucode_tlv *tlv, int copy_size = le32_to_cpu(tlv->length) + sizeof(*tlv); int offset_size = copy_size; + if (le32_to_cpu(header->tlv_version) != 1) + return; + if (WARN_ONCE(apply_point >= IWL_FW_INI_APPLY_NUM, "Invalid apply point id %d\n", apply_point)) return; @@ -133,7 +136,10 @@ void iwl_alloc_dbg_tlv(struct iwl_trans *trans, size_t len, const u8 *data, hdr = (void *)&tlv->data[0]; apply = le32_to_cpu(hdr->apply_point); - IWL_DEBUG_FW(trans, "Read TLV %x, apply point %d\n", + if (le32_to_cpu(hdr->tlv_version) != 1) + continue; + + IWL_DEBUG_FW(trans, "WRT: read TLV 0x%x, apply point %d\n", le32_to_cpu(tlv->type), apply); if (WARN_ON(apply >= IWL_FW_INI_APPLY_NUM)) diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-debug.h b/drivers/net/wireless/intel/iwlwifi/iwl-debug.h index 655ff5694560a672ab0c58d2dd5c5a7bc5613164..d3ba6a1422eeedbac976b238b14099ba56547834 100644 --- a/drivers/net/wireless/intel/iwlwifi/iwl-debug.h +++ b/drivers/net/wireless/intel/iwlwifi/iwl-debug.h @@ -218,5 +218,7 @@ do { \ #define IWL_DEBUG_TPT(p, f, a...) IWL_DEBUG(p, IWL_DL_TPT, f, ## a) #define IWL_DEBUG_RPM(p, f, a...) IWL_DEBUG(p, IWL_DL_RPM, f, ## a) #define IWL_DEBUG_LAR(p, f, a...) IWL_DEBUG(p, IWL_DL_LAR, f, ## a) +#define IWL_DEBUG_FW_INFO(p, f, a...) \ + IWL_DEBUG(p, IWL_DL_INFO | IWL_DL_FW, f, ## a) #endif diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-drv.c b/drivers/net/wireless/intel/iwlwifi/iwl-drv.c index 689a65b11cc33c9237a36be525df9a7e2474b11e..852d3cbfc719925340c23457145c7b2a80b180e9 100644 --- a/drivers/net/wireless/intel/iwlwifi/iwl-drv.c +++ b/drivers/net/wireless/intel/iwlwifi/iwl-drv.c @@ -179,6 +179,7 @@ static void iwl_dealloc_ucode(struct iwl_drv *drv) kfree(drv->fw.dbg.trigger_tlv[i]); kfree(drv->fw.dbg.mem_tlv); kfree(drv->fw.iml); + kfree(drv->fw.ucode_capa.cmd_versions); for (i = 0; i < IWL_UCODE_TYPE_MAX; i++) iwl_free_fw_img(drv, drv->fw.img + i); @@ -252,8 +253,8 @@ static int iwl_request_firmware(struct iwl_drv *drv, bool first) snprintf(drv->firmware_name, sizeof(drv->firmware_name), "%s%s.ucode", cfg->fw_name_pre, tag); - IWL_DEBUG_INFO(drv, "attempting to load firmware '%s'\n", - drv->firmware_name); + IWL_DEBUG_FW_INFO(drv, "attempting to load firmware '%s'\n", + drv->firmware_name); return request_firmware_nowait(THIS_MODULE, 1, drv->firmware_name, drv->trans->dev, @@ -1144,6 +1145,23 @@ static int iwl_parse_tlv_firmware(struct iwl_drv *drv, if (iwlwifi_mod_params.enable_ini) iwl_fw_dbg_copy_tlv(drv->trans, tlv, false); break; + case IWL_UCODE_TLV_CMD_VERSIONS: + if (tlv_len % sizeof(struct iwl_fw_cmd_version)) { + IWL_ERR(drv, + "Invalid length for command versions: %u\n", + tlv_len); + tlv_len /= sizeof(struct iwl_fw_cmd_version); + tlv_len *= sizeof(struct iwl_fw_cmd_version); + } + if (WARN_ON(capa->cmd_versions)) + return -EINVAL; + capa->cmd_versions = kmemdup(tlv_data, tlv_len, + GFP_KERNEL); + if (!capa->cmd_versions) + return -ENOMEM; + capa->n_cmd_versions = + tlv_len / sizeof(struct iwl_fw_cmd_version); + break; default: IWL_DEBUG_INFO(drv, "unknown TLV: %d\n", tlv_type); break; @@ -1318,8 +1336,8 @@ static void iwl_req_fw_callback(const struct firmware *ucode_raw, void *context) if (!ucode_raw) goto try_again; - IWL_DEBUG_INFO(drv, "Loaded firmware file '%s' (%zd bytes).\n", - drv->firmware_name, ucode_raw->size); + IWL_DEBUG_FW_INFO(drv, "Loaded firmware file '%s' (%zd bytes).\n", + drv->firmware_name, ucode_raw->size); /* Make sure that we got at least the API version number */ if (ucode_raw->size < 4) { diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-nvm-parse.c b/drivers/net/wireless/intel/iwlwifi/iwl-nvm-parse.c index 87d6de7efdd2356a48e08ce0acec08112bb50a28..d87a6bb3e45684052b3eadea0e3b05df89f7a158 100644 --- a/drivers/net/wireless/intel/iwlwifi/iwl-nvm-parse.c +++ b/drivers/net/wireless/intel/iwlwifi/iwl-nvm-parse.c @@ -130,7 +130,7 @@ enum nvm_sku_bits { /* * These are the channel numbers in the order that they are stored in the NVM */ -static const u8 iwl_nvm_channels[] = { +static const u16 iwl_nvm_channels[] = { /* 2.4 GHz */ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, /* 5 GHz */ @@ -139,7 +139,7 @@ static const u8 iwl_nvm_channels[] = { 149, 153, 157, 161, 165 }; -static const u8 iwl_ext_nvm_channels[] = { +static const u16 iwl_ext_nvm_channels[] = { /* 2.4 GHz */ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, /* 5 GHz */ @@ -148,14 +148,27 @@ static const u8 iwl_ext_nvm_channels[] = { 149, 153, 157, 161, 165, 169, 173, 177, 181 }; +static const u16 iwl_uhb_nvm_channels[] = { + /* 2.4 GHz */ + 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, + /* 5 GHz */ + 36, 40, 44, 48, 52, 56, 60, 64, 68, 72, 76, 80, 84, 88, 92, + 96, 100, 104, 108, 112, 116, 120, 124, 128, 132, 136, 140, 144, + 149, 153, 157, 161, 165, 169, 173, 177, 181, + /* 6-7 GHz */ + 189, 193, 197, 201, 205, 209, 213, 217, 221, 225, 229, 233, 237, 241, + 245, 249, 253, 257, 261, 265, 269, 273, 277, 281, 285, 289, 293, 297, + 301, 305, 309, 313, 317, 321, 325, 329, 333, 337, 341, 345, 349, 353, + 357, 361, 365, 369, 373, 377, 381, 385, 389, 393, 397, 401, 405, 409, + 413, 417, 421 +}; + #define IWL_NVM_NUM_CHANNELS ARRAY_SIZE(iwl_nvm_channels) #define IWL_NVM_NUM_CHANNELS_EXT ARRAY_SIZE(iwl_ext_nvm_channels) +#define IWL_NVM_NUM_CHANNELS_UHB ARRAY_SIZE(iwl_uhb_nvm_channels) #define NUM_2GHZ_CHANNELS 14 -#define NUM_2GHZ_CHANNELS_EXT 14 #define FIRST_2GHZ_HT_MINUS 5 #define LAST_2GHZ_HT_PLUS 9 -#define LAST_5GHZ_HT 165 -#define LAST_5GHZ_HT_FAMILY_8000 181 #define N_HW_ADDR_MASK 0xF /* rate data (static) */ @@ -213,7 +226,7 @@ enum iwl_nvm_channel_flags { }; static inline void iwl_nvm_print_channel_flags(struct device *dev, u32 level, - int chan, u16 flags) + int chan, u32 flags) { #define CHECK_AND_PRINT_I(x) \ ((flags & NVM_CHANNEL_##x) ? " " #x : "") @@ -244,20 +257,16 @@ static inline void iwl_nvm_print_channel_flags(struct device *dev, u32 level, } static u32 iwl_get_channel_flags(u8 ch_num, int ch_idx, bool is_5ghz, - u16 nvm_flags, const struct iwl_cfg *cfg) + u32 nvm_flags, const struct iwl_cfg *cfg) { u32 flags = IEEE80211_CHAN_NO_HT40; - u32 last_5ghz_ht = LAST_5GHZ_HT; - - if (cfg->nvm_type == IWL_NVM_EXT) - last_5ghz_ht = LAST_5GHZ_HT_FAMILY_8000; if (!is_5ghz && (nvm_flags & NVM_CHANNEL_40MHZ)) { if (ch_num <= LAST_2GHZ_HT_PLUS) flags &= ~IEEE80211_CHAN_NO_HT40PLUS; if (ch_num >= FIRST_2GHZ_HT_MINUS) flags &= ~IEEE80211_CHAN_NO_HT40MINUS; - } else if (ch_num <= last_5ghz_ht && (nvm_flags & NVM_CHANNEL_40MHZ)) { + } else if (nvm_flags & NVM_CHANNEL_40MHZ) { if ((ch_idx - NUM_2GHZ_CHANNELS) % 2 == 0) flags &= ~IEEE80211_CHAN_NO_HT40PLUS; else @@ -292,30 +301,36 @@ static u32 iwl_get_channel_flags(u8 ch_num, int ch_idx, bool is_5ghz, static int iwl_init_channel_map(struct device *dev, const struct iwl_cfg *cfg, struct iwl_nvm_data *data, - const __le16 * const nvm_ch_flags, - u32 sbands_flags) + const void * const nvm_ch_flags, + u32 sbands_flags, bool v4) { int ch_idx; int n_channels = 0; struct ieee80211_channel *channel; - u16 ch_flags; - int num_of_ch, num_2ghz_channels; - const u8 *nvm_chan; - - if (cfg->nvm_type != IWL_NVM_EXT) { - num_of_ch = IWL_NVM_NUM_CHANNELS; - nvm_chan = &iwl_nvm_channels[0]; - num_2ghz_channels = NUM_2GHZ_CHANNELS; - } else { + u32 ch_flags; + int num_of_ch, num_2ghz_channels = NUM_2GHZ_CHANNELS; + const u16 *nvm_chan; + + if (cfg->uhb_supported) { + num_of_ch = IWL_NVM_NUM_CHANNELS_UHB; + nvm_chan = iwl_uhb_nvm_channels; + } else if (cfg->nvm_type == IWL_NVM_EXT) { num_of_ch = IWL_NVM_NUM_CHANNELS_EXT; - nvm_chan = &iwl_ext_nvm_channels[0]; - num_2ghz_channels = NUM_2GHZ_CHANNELS_EXT; + nvm_chan = iwl_ext_nvm_channels; + } else { + num_of_ch = IWL_NVM_NUM_CHANNELS; + nvm_chan = iwl_nvm_channels; } for (ch_idx = 0; ch_idx < num_of_ch; ch_idx++) { bool is_5ghz = (ch_idx >= num_2ghz_channels); - ch_flags = __le16_to_cpup(nvm_ch_flags + ch_idx); + if (v4) + ch_flags = + __le32_to_cpup((__le32 *)nvm_ch_flags + ch_idx); + else + ch_flags = + __le16_to_cpup((__le16 *)nvm_ch_flags + ch_idx); if (is_5ghz && !data->sku_cap_band_52ghz_enable) continue; @@ -636,12 +651,7 @@ static struct ieee80211_sband_iftype_data iwl_he_capa[] = { static void iwl_init_he_hw_capab(struct ieee80211_supported_band *sband, u8 tx_chains, u8 rx_chains) { - if (sband->band == NL80211_BAND_2GHZ || - sband->band == NL80211_BAND_5GHZ) - sband->iftype_data = iwl_he_capa; - else - return; - + sband->iftype_data = iwl_he_capa; sband->n_iftype_data = ARRAY_SIZE(iwl_he_capa); /* If not 2x2, we need to indicate 1x1 in the Midamble RX Max NSTS */ @@ -661,15 +671,15 @@ static void iwl_init_he_hw_capab(struct ieee80211_supported_band *sband, static void iwl_init_sbands(struct device *dev, const struct iwl_cfg *cfg, struct iwl_nvm_data *data, - const __le16 *nvm_ch_flags, u8 tx_chains, - u8 rx_chains, u32 sbands_flags) + const void *nvm_ch_flags, u8 tx_chains, + u8 rx_chains, u32 sbands_flags, bool v4) { int n_channels; int n_used = 0; struct ieee80211_supported_band *sband; n_channels = iwl_init_channel_map(dev, cfg, data, nvm_ch_flags, - sbands_flags); + sbands_flags, v4); sband = &data->bands[NL80211_BAND_2GHZ]; sband->band = NL80211_BAND_2GHZ; sband->bitrates = &iwl_cfg80211_rates[RATES_24_OFFS]; @@ -1006,22 +1016,18 @@ iwl_parse_nvm_data(struct iwl_trans *trans, const struct iwl_cfg *cfg, sbands_flags |= IWL_NVM_SBANDS_FLAGS_NO_WIDE_IN_5GHZ; iwl_init_sbands(dev, cfg, data, ch_section, tx_chains, rx_chains, - sbands_flags); + sbands_flags, false); data->calib_version = 255; return data; } IWL_EXPORT_SYMBOL(iwl_parse_nvm_data); -static u32 iwl_nvm_get_regdom_bw_flags(const u8 *nvm_chan, +static u32 iwl_nvm_get_regdom_bw_flags(const u16 *nvm_chan, int ch_idx, u16 nvm_flags, const struct iwl_cfg *cfg) { u32 flags = NL80211_RRF_NO_HT40; - u32 last_5ghz_ht = LAST_5GHZ_HT; - - if (cfg->nvm_type == IWL_NVM_EXT) - last_5ghz_ht = LAST_5GHZ_HT_FAMILY_8000; if (ch_idx < NUM_2GHZ_CHANNELS && (nvm_flags & NVM_CHANNEL_40MHZ)) { @@ -1029,8 +1035,7 @@ static u32 iwl_nvm_get_regdom_bw_flags(const u8 *nvm_chan, flags &= ~NL80211_RRF_NO_HT40PLUS; if (nvm_chan[ch_idx] >= FIRST_2GHZ_HT_MINUS) flags &= ~NL80211_RRF_NO_HT40MINUS; - } else if (nvm_chan[ch_idx] <= last_5ghz_ht && - (nvm_flags & NVM_CHANNEL_40MHZ)) { + } else if (nvm_flags & NVM_CHANNEL_40MHZ) { if ((ch_idx - NUM_2GHZ_CHANNELS) % 2 == 0) flags &= ~NL80211_RRF_NO_HT40PLUS; else @@ -1074,18 +1079,26 @@ iwl_parse_nvm_mcc_info(struct device *dev, const struct iwl_cfg *cfg, int ch_idx; u16 ch_flags; u32 reg_rule_flags, prev_reg_rule_flags = 0; - const u8 *nvm_chan = cfg->nvm_type == IWL_NVM_EXT ? - iwl_ext_nvm_channels : iwl_nvm_channels; + const u16 *nvm_chan; struct ieee80211_regdomain *regd, *copy_rd; - int size_of_regd, regd_to_copy; struct ieee80211_reg_rule *rule; struct regdb_ptrs *regdb_ptrs; enum nl80211_band band; int center_freq, prev_center_freq = 0; int valid_rules = 0; bool new_rule; - int max_num_ch = cfg->nvm_type == IWL_NVM_EXT ? - IWL_NVM_NUM_CHANNELS_EXT : IWL_NVM_NUM_CHANNELS; + int max_num_ch; + + if (cfg->uhb_supported) { + max_num_ch = IWL_NVM_NUM_CHANNELS_UHB; + nvm_chan = iwl_uhb_nvm_channels; + } else if (cfg->nvm_type == IWL_NVM_EXT) { + max_num_ch = IWL_NVM_NUM_CHANNELS_EXT; + nvm_chan = iwl_ext_nvm_channels; + } else { + max_num_ch = IWL_NVM_NUM_CHANNELS; + nvm_chan = iwl_nvm_channels; + } if (WARN_ON(num_of_ch > max_num_ch)) num_of_ch = max_num_ch; @@ -1097,11 +1110,7 @@ iwl_parse_nvm_mcc_info(struct device *dev, const struct iwl_cfg *cfg, num_of_ch); /* build a regdomain rule for every valid channel */ - size_of_regd = - sizeof(struct ieee80211_regdomain) + - num_of_ch * sizeof(struct ieee80211_reg_rule); - - regd = kzalloc(size_of_regd, GFP_KERNEL); + regd = kzalloc(struct_size(regd, reg_rules, num_of_ch), GFP_KERNEL); if (!regd) return ERR_PTR(-ENOMEM); @@ -1177,14 +1186,10 @@ iwl_parse_nvm_mcc_info(struct device *dev, const struct iwl_cfg *cfg, * Narrow down regdom for unused regulatory rules to prevent hole * between reg rules to wmm rules. */ - regd_to_copy = sizeof(struct ieee80211_regdomain) + - valid_rules * sizeof(struct ieee80211_reg_rule); - - copy_rd = kmemdup(regd, regd_to_copy, GFP_KERNEL); - if (!copy_rd) { + copy_rd = kmemdup(regd, struct_size(regd, reg_rules, valid_rules), + GFP_KERNEL); + if (!copy_rd) copy_rd = ERR_PTR(-ENOMEM); - goto out; - } out: kfree(regdb_ptrs); @@ -1393,7 +1398,6 @@ struct iwl_nvm_data *iwl_get_nvm(struct iwl_trans *trans, const struct iwl_fw *fw) { struct iwl_nvm_get_info cmd = {}; - struct iwl_nvm_get_info_rsp *rsp; struct iwl_nvm_data *nvm; struct iwl_host_cmd hcmd = { .flags = CMD_WANT_SKB | CMD_SEND_IN_RFKILL, @@ -1408,12 +1412,24 @@ struct iwl_nvm_data *iwl_get_nvm(struct iwl_trans *trans, bool empty_otp; u32 mac_flags; u32 sbands_flags = 0; + /* + * All the values in iwl_nvm_get_info_rsp v4 are the same as + * in v3, except for the channel profile part of the + * regulatory. So we can just access the new struct, with the + * exception of the latter. + */ + struct iwl_nvm_get_info_rsp *rsp; + struct iwl_nvm_get_info_rsp_v3 *rsp_v3; + bool v4 = fw_has_api(&fw->ucode_capa, + IWL_UCODE_TLV_API_REGULATORY_NVM_INFO); + size_t rsp_size = v4 ? sizeof(*rsp) : sizeof(*rsp_v3); + void *channel_profile; ret = iwl_trans_send_cmd(trans, &hcmd); if (ret) return ERR_PTR(ret); - if (WARN(iwl_rx_packet_payload_len(hcmd.resp_pkt) != sizeof(*rsp), + if (WARN(iwl_rx_packet_payload_len(hcmd.resp_pkt) != rsp_size, "Invalid payload len in NVM response from FW %d", iwl_rx_packet_payload_len(hcmd.resp_pkt))) { ret = -EINVAL; @@ -1475,11 +1491,15 @@ struct iwl_nvm_data *iwl_get_nvm(struct iwl_trans *trans, sbands_flags |= IWL_NVM_SBANDS_FLAGS_LAR; } + rsp_v3 = (void *)rsp; + channel_profile = v4 ? (void *)rsp->regulatory.channel_profile : + (void *)rsp_v3->regulatory.channel_profile; + iwl_init_sbands(trans->dev, trans->cfg, nvm, - rsp->regulatory.channel_profile, + channel_profile, nvm->valid_tx_ant & fw->valid_tx_ant, nvm->valid_rx_ant & fw->valid_rx_ant, - sbands_flags); + sbands_flags, v4); iwl_free_resp(&hcmd); return nvm; diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-prph.h b/drivers/net/wireless/intel/iwlwifi/iwl-prph.h index 1af9f9e1ecd4edc55e23a098e6ddfba2d6187cc5..8e6a0c363c0dd514cca2c064a4568b5eee2bb537 100644 --- a/drivers/net/wireless/intel/iwlwifi/iwl-prph.h +++ b/drivers/net/wireless/intel/iwlwifi/iwl-prph.h @@ -368,6 +368,12 @@ #define MON_BUFF_WRPTR_VER2 (0xa03c24) #define MON_BUFF_CYCLE_CNT_VER2 (0xa03c28) #define MON_BUFF_SHIFT_VER2 (0x8) +/* FW monitor familiy AX210 and on */ +#define DBGC_CUR_DBGBUF_BASE_ADDR_LSB (0xd03c20) +#define DBGC_CUR_DBGBUF_BASE_ADDR_MSB (0xd03c24) +#define DBGC_CUR_DBGBUF_STATUS (0xd03c1c) +#define DBGC_DBGBUF_WRAP_AROUND (0xd03c2c) +#define DBGC_CUR_DBGBUF_STATUS_OFFSET_MSK (0x00ffffff) #define MON_DMARB_RD_CTL_ADDR (0xa03c60) #define MON_DMARB_RD_DATA_ADDR (0xa03c5c) diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-trans.h b/drivers/net/wireless/intel/iwlwifi/iwl-trans.h index d8690acee40c0c45668f6480b10daff295633500..1e4c9ef548ccf6f06a3ac56084437bcfbea4465c 100644 --- a/drivers/net/wireless/intel/iwlwifi/iwl-trans.h +++ b/drivers/net/wireless/intel/iwlwifi/iwl-trans.h @@ -274,7 +274,6 @@ struct iwl_rx_cmd_buffer { bool _page_stolen; u32 _rx_page_order; unsigned int truesize; - u8 status; }; static inline void *rxb_addr(struct iwl_rx_cmd_buffer *r) @@ -768,6 +767,7 @@ struct iwl_self_init_dram { * @umac_error_event_table: addr of umac error table * @error_event_table_tlv_status: bitmap that indicates what error table * pointers was recevied via TLV. use enum &iwl_error_event_table_status + * @hw_error: equals true if hw error interrupt was received from the FW */ struct iwl_trans { const struct iwl_trans_ops *ops; @@ -830,6 +830,7 @@ struct iwl_trans { u32 lmac_error_event_table[2]; u32 umac_error_event_table; unsigned int error_event_table_tlv_status; + bool hw_error; /* pointer to trans specific struct */ /*Ensure that this pointer will always be aligned to sizeof pointer */ diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/d3.c b/drivers/net/wireless/intel/iwlwifi/mvm/d3.c index 808bc6f363d0e4642d38daf28a4ea7165037dc32..60f5d337f16d4f8c2dd65ec13332a3684d4f1a69 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/d3.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/d3.c @@ -8,7 +8,7 @@ * Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH * Copyright(c) 2016 - 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * * This program is free software; you can redistribute it and/or modify * it under the terms of version 2 of the GNU General Public License as @@ -31,7 +31,7 @@ * Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH * Copyright(c) 2016 - 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -385,10 +385,10 @@ static void iwl_mvm_wowlan_program_keys(struct ieee80211_hw *hw, } } -static int iwl_mvm_send_patterns(struct iwl_mvm *mvm, - struct cfg80211_wowlan *wowlan) +static int iwl_mvm_send_patterns_v1(struct iwl_mvm *mvm, + struct cfg80211_wowlan *wowlan) { - struct iwl_wowlan_patterns_cmd *pattern_cmd; + struct iwl_wowlan_patterns_cmd_v1 *pattern_cmd; struct iwl_host_cmd cmd = { .id = WOWLAN_PATTERNS, .dataflags[0] = IWL_HCMD_DFL_NOCOPY, @@ -399,7 +399,7 @@ static int iwl_mvm_send_patterns(struct iwl_mvm *mvm, return 0; cmd.len[0] = sizeof(*pattern_cmd) + - wowlan->n_patterns * sizeof(struct iwl_wowlan_pattern); + wowlan->n_patterns * sizeof(struct iwl_wowlan_pattern_v1); pattern_cmd = kmalloc(cmd.len[0], GFP_KERNEL); if (!pattern_cmd) @@ -426,6 +426,50 @@ static int iwl_mvm_send_patterns(struct iwl_mvm *mvm, return err; } +static int iwl_mvm_send_patterns(struct iwl_mvm *mvm, + struct cfg80211_wowlan *wowlan) +{ + struct iwl_wowlan_patterns_cmd *pattern_cmd; + struct iwl_host_cmd cmd = { + .id = WOWLAN_PATTERNS, + .dataflags[0] = IWL_HCMD_DFL_NOCOPY, + }; + int i, err; + + if (!wowlan->n_patterns) + return 0; + + cmd.len[0] = sizeof(*pattern_cmd) + + wowlan->n_patterns * sizeof(struct iwl_wowlan_pattern_v2); + + pattern_cmd = kmalloc(cmd.len[0], GFP_KERNEL); + if (!pattern_cmd) + return -ENOMEM; + + pattern_cmd->n_patterns = cpu_to_le32(wowlan->n_patterns); + + for (i = 0; i < wowlan->n_patterns; i++) { + int mask_len = DIV_ROUND_UP(wowlan->patterns[i].pattern_len, 8); + + pattern_cmd->patterns[i].pattern_type = + WOWLAN_PATTERN_TYPE_BITMASK; + + memcpy(&pattern_cmd->patterns[i].u.bitmask.mask, + wowlan->patterns[i].mask, mask_len); + memcpy(&pattern_cmd->patterns[i].u.bitmask.pattern, + wowlan->patterns[i].pattern, + wowlan->patterns[i].pattern_len); + pattern_cmd->patterns[i].u.bitmask.mask_size = mask_len; + pattern_cmd->patterns[i].u.bitmask.pattern_size = + wowlan->patterns[i].pattern_len; + } + + cmd.data[0] = pattern_cmd; + err = iwl_mvm_send_cmd(mvm, &cmd); + kfree(pattern_cmd); + return err; +} + static int iwl_mvm_d3_reprogram(struct iwl_mvm *mvm, struct ieee80211_vif *vif, struct ieee80211_sta *ap_sta) { @@ -851,7 +895,11 @@ iwl_mvm_wowlan_config(struct iwl_mvm *mvm, if (ret) return ret; - ret = iwl_mvm_send_patterns(mvm, wowlan); + if (fw_has_api(&mvm->fw->ucode_capa, + IWL_UCODE_TLV_API_WOWLAN_TCP_SYN_WAKE)) + ret = iwl_mvm_send_patterns(mvm, wowlan); + else + ret = iwl_mvm_send_patterns_v1(mvm, wowlan); if (ret) return ret; @@ -1728,9 +1776,12 @@ void iwl_mvm_d0i3_update_keys(struct iwl_mvm *mvm, iwl_mvm_iter_d0i3_ap_keys(mvm, vif, iwl_mvm_d3_update_keys, >kdata); } +#define ND_QUERY_BUF_LEN (sizeof(struct iwl_scan_offload_profile_match) * \ + IWL_SCAN_MAX_PROFILES) + struct iwl_mvm_nd_query_results { u32 matched_profiles; - struct iwl_scan_offload_profile_match matches[IWL_SCAN_MAX_PROFILES]; + u8 matches[ND_QUERY_BUF_LEN]; }; static int @@ -1743,6 +1794,7 @@ iwl_mvm_netdetect_query_results(struct iwl_mvm *mvm, .flags = CMD_WANT_SKB, }; int ret, len; + size_t query_len, matches_len; ret = iwl_mvm_send_cmd(mvm, &cmd); if (ret) { @@ -1750,8 +1802,19 @@ iwl_mvm_netdetect_query_results(struct iwl_mvm *mvm, return ret; } + if (fw_has_api(&mvm->fw->ucode_capa, + IWL_UCODE_TLV_API_SCAN_OFFLOAD_CHANS)) { + query_len = sizeof(struct iwl_scan_offload_profiles_query); + matches_len = sizeof(struct iwl_scan_offload_profile_match) * + IWL_SCAN_MAX_PROFILES; + } else { + query_len = sizeof(struct iwl_scan_offload_profiles_query_v1); + matches_len = sizeof(struct iwl_scan_offload_profile_match_v1) * + IWL_SCAN_MAX_PROFILES; + } + len = iwl_rx_packet_payload_len(cmd.resp_pkt); - if (len < sizeof(*query)) { + if (len < query_len) { IWL_ERR(mvm, "Invalid scan offload profiles query response!\n"); ret = -EIO; goto out_free_resp; @@ -1760,7 +1823,7 @@ iwl_mvm_netdetect_query_results(struct iwl_mvm *mvm, query = (void *)cmd.resp_pkt->data; results->matched_profiles = le32_to_cpu(query->matched_profiles); - memcpy(results->matches, query->matches, sizeof(results->matches)); + memcpy(results->matches, query->matches, matches_len); #ifdef CONFIG_IWLWIFI_DEBUGFS mvm->last_netdetect_scans = le32_to_cpu(query->n_scans_done); @@ -1771,6 +1834,57 @@ out_free_resp: return ret; } +static int iwl_mvm_query_num_match_chans(struct iwl_mvm *mvm, + struct iwl_mvm_nd_query_results *query, + int idx) +{ + int n_chans = 0, i; + + if (fw_has_api(&mvm->fw->ucode_capa, + IWL_UCODE_TLV_API_SCAN_OFFLOAD_CHANS)) { + struct iwl_scan_offload_profile_match *matches = + (struct iwl_scan_offload_profile_match *)query->matches; + + for (i = 0; i < SCAN_OFFLOAD_MATCHING_CHANNELS_LEN; i++) + n_chans += hweight8(matches[idx].matching_channels[i]); + } else { + struct iwl_scan_offload_profile_match_v1 *matches = + (struct iwl_scan_offload_profile_match_v1 *)query->matches; + + for (i = 0; i < SCAN_OFFLOAD_MATCHING_CHANNELS_LEN_V1; i++) + n_chans += hweight8(matches[idx].matching_channels[i]); + } + + return n_chans; +} + +static void iwl_mvm_query_set_freqs(struct iwl_mvm *mvm, + struct iwl_mvm_nd_query_results *query, + struct cfg80211_wowlan_nd_match *match, + int idx) +{ + int i; + + if (fw_has_api(&mvm->fw->ucode_capa, + IWL_UCODE_TLV_API_SCAN_OFFLOAD_CHANS)) { + struct iwl_scan_offload_profile_match *matches = + (struct iwl_scan_offload_profile_match *)query->matches; + + for (i = 0; i < SCAN_OFFLOAD_MATCHING_CHANNELS_LEN * 8; i++) + if (matches[idx].matching_channels[i / 8] & (BIT(i % 8))) + match->channels[match->n_channels++] = + mvm->nd_channels[i]->center_freq; + } else { + struct iwl_scan_offload_profile_match_v1 *matches = + (struct iwl_scan_offload_profile_match_v1 *)query->matches; + + for (i = 0; i < SCAN_OFFLOAD_MATCHING_CHANNELS_LEN_V1 * 8; i++) + if (matches[idx].matching_channels[i / 8] & (BIT(i % 8))) + match->channels[match->n_channels++] = + mvm->nd_channels[i]->center_freq; + } +} + static void iwl_mvm_query_netdetect_reasons(struct iwl_mvm *mvm, struct ieee80211_vif *vif) { @@ -1783,7 +1897,7 @@ static void iwl_mvm_query_netdetect_reasons(struct iwl_mvm *mvm, struct iwl_wowlan_status *fw_status; unsigned long matched_profiles; u32 reasons = 0; - int i, j, n_matches, ret; + int i, n_matches, ret; fw_status = iwl_mvm_get_wakeup_status(mvm); if (!IS_ERR_OR_NULL(fw_status)) { @@ -1817,14 +1931,10 @@ static void iwl_mvm_query_netdetect_reasons(struct iwl_mvm *mvm, goto out_report_nd; for_each_set_bit(i, &matched_profiles, mvm->n_nd_match_sets) { - struct iwl_scan_offload_profile_match *fw_match; struct cfg80211_wowlan_nd_match *match; int idx, n_channels = 0; - fw_match = &query.matches[i]; - - for (j = 0; j < SCAN_OFFLOAD_MATCHING_CHANNELS_LEN; j++) - n_channels += hweight8(fw_match->matching_channels[j]); + n_channels = iwl_mvm_query_num_match_chans(mvm, &query, i); match = kzalloc(struct_size(match, channels, n_channels), GFP_KERNEL); @@ -1844,10 +1954,7 @@ static void iwl_mvm_query_netdetect_reasons(struct iwl_mvm *mvm, if (mvm->n_nd_channels < n_channels) continue; - for (j = 0; j < SCAN_OFFLOAD_MATCHING_CHANNELS_LEN * 8; j++) - if (fw_match->matching_channels[j / 8] & (BIT(j % 8))) - match->channels[match->n_channels++] = - mvm->nd_channels[j]->center_freq; + iwl_mvm_query_set_freqs(mvm, &query, match, i); } out_report_nd: @@ -2030,7 +2137,6 @@ out: * 2. We are using a unified image but had an error while exiting D3 */ set_bit(IWL_MVM_STATUS_HW_RESTART_REQUESTED, &mvm->status); - set_bit(IWL_MVM_STATUS_D3_RECONFIG, &mvm->status); /* * When switching images we return 1, which causes mac80211 * to do a reconfig with IEEE80211_RECONFIG_TYPE_RESTART. diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/debugfs-vif.c b/drivers/net/wireless/intel/iwlwifi/mvm/debugfs-vif.c index 6925527d8457a294ff423547d86511e5a2718ae8..f043eefabb4ec40120b4afe0dca2187237030043 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/debugfs-vif.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/debugfs-vif.c @@ -743,9 +743,8 @@ static ssize_t iwl_dbgfs_quota_min_read(struct file *file, #define MVM_DEBUGFS_READ_WRITE_FILE_OPS(name, bufsz) \ _MVM_DEBUGFS_READ_WRITE_FILE_OPS(name, bufsz, struct ieee80211_vif) #define MVM_DEBUGFS_ADD_FILE_VIF(name, parent, mode) do { \ - if (!debugfs_create_file(#name, mode, parent, vif, \ - &iwl_dbgfs_##name##_ops)) \ - goto err; \ + debugfs_create_file(#name, mode, parent, vif, \ + &iwl_dbgfs_##name##_ops); \ } while (0) MVM_DEBUGFS_READ_FILE_OPS(mac_params); @@ -811,12 +810,6 @@ void iwl_mvm_vif_dbgfs_register(struct iwl_mvm *mvm, struct ieee80211_vif *vif) mvmvif->dbgfs_slink = debugfs_create_symlink(dbgfs_dir->d_name.name, mvm->debugfs_dir, buf); - if (!mvmvif->dbgfs_slink) - IWL_ERR(mvm, "Can't create debugfs symbolic link under %pd\n", - dbgfs_dir); - return; -err: - IWL_ERR(mvm, "Can't create debugfs entity\n"); } void iwl_mvm_vif_dbgfs_clean(struct iwl_mvm *mvm, struct ieee80211_vif *vif) diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/debugfs.c b/drivers/net/wireless/intel/iwlwifi/mvm/debugfs.c index 776b24f5420006635c96520a4624bb2cd007a649..d4ff6b44de2c45e0d718ee5a3887472d6f752760 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/debugfs.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/debugfs.c @@ -1349,7 +1349,7 @@ static ssize_t iwl_dbgfs_fw_dbg_collect_write(struct iwl_mvm *mvm, return 0; iwl_fw_dbg_collect(&mvm->fwrt, FW_DBG_TRIGGER_USER, buf, - (count - 1)); + (count - 1), NULL); iwl_mvm_unref(mvm, IWL_MVM_REF_PRPH_WRITE); @@ -1696,9 +1696,8 @@ static ssize_t iwl_dbgfs_d0i3_refs_write(struct iwl_mvm *mvm, char *buf, #define MVM_DEBUGFS_READ_WRITE_FILE_OPS(name, bufsz) \ _MVM_DEBUGFS_READ_WRITE_FILE_OPS(name, bufsz, struct iwl_mvm) #define MVM_DEBUGFS_ADD_FILE_ALIAS(alias, name, parent, mode) do { \ - if (!debugfs_create_file(alias, mode, parent, mvm, \ - &iwl_dbgfs_##name##_ops)) \ - goto err; \ + debugfs_create_file(alias, mode, parent, mvm, \ + &iwl_dbgfs_##name##_ops); \ } while (0) #define MVM_DEBUGFS_ADD_FILE(name, parent, mode) \ MVM_DEBUGFS_ADD_FILE_ALIAS(#name, name, parent, mode) @@ -1709,9 +1708,8 @@ static ssize_t iwl_dbgfs_d0i3_refs_write(struct iwl_mvm *mvm, char *buf, _MVM_DEBUGFS_READ_WRITE_FILE_OPS(name, bufsz, struct ieee80211_sta) #define MVM_DEBUGFS_ADD_STA_FILE_ALIAS(alias, name, parent, mode) do { \ - if (!debugfs_create_file(alias, mode, parent, sta, \ - &iwl_dbgfs_##name##_ops)) \ - goto err; \ + debugfs_create_file(alias, mode, parent, sta, \ + &iwl_dbgfs_##name##_ops); \ } while (0) #define MVM_DEBUGFS_ADD_STA_FILE(name, parent, mode) \ MVM_DEBUGFS_ADD_STA_FILE_ALIAS(#name, name, parent, mode) @@ -2092,13 +2090,9 @@ void iwl_mvm_sta_add_debugfs(struct ieee80211_hw *hw, if (iwl_mvm_has_tlc_offload(mvm)) MVM_DEBUGFS_ADD_STA_FILE(rs_data, dir, 0400); - - return; -err: - IWL_ERR(mvm, "Can't create the mvm station debugfs entry\n"); } -int iwl_mvm_dbgfs_register(struct iwl_mvm *mvm, struct dentry *dbgfs_dir) +void iwl_mvm_dbgfs_register(struct iwl_mvm *mvm, struct dentry *dbgfs_dir) { struct dentry *bcast_dir __maybe_unused; char buf[100]; @@ -2142,14 +2136,10 @@ int iwl_mvm_dbgfs_register(struct iwl_mvm *mvm, struct dentry *dbgfs_dir) #endif MVM_DEBUGFS_ADD_FILE(he_sniffer_params, mvm->debugfs_dir, 0600); - if (!debugfs_create_bool("enable_scan_iteration_notif", - 0600, - mvm->debugfs_dir, - &mvm->scan_iter_notif_enabled)) - goto err; - if (!debugfs_create_bool("drop_bcn_ap_mode", 0600, - mvm->debugfs_dir, &mvm->drop_bcn_ap_mode)) - goto err; + debugfs_create_bool("enable_scan_iteration_notif", 0600, + mvm->debugfs_dir, &mvm->scan_iter_notif_enabled); + debugfs_create_bool("drop_bcn_ap_mode", 0600, mvm->debugfs_dir, + &mvm->drop_bcn_ap_mode); MVM_DEBUGFS_ADD_FILE(uapsd_noagg_bssids, mvm->debugfs_dir, S_IRUSR); @@ -2157,13 +2147,9 @@ int iwl_mvm_dbgfs_register(struct iwl_mvm *mvm, struct dentry *dbgfs_dir) if (mvm->fw->ucode_capa.flags & IWL_UCODE_TLV_FLAGS_BCAST_FILTERING) { bcast_dir = debugfs_create_dir("bcast_filtering", mvm->debugfs_dir); - if (!bcast_dir) - goto err; - if (!debugfs_create_bool("override", 0600, - bcast_dir, - &mvm->dbgfs_bcast_filtering.override)) - goto err; + debugfs_create_bool("override", 0600, bcast_dir, + &mvm->dbgfs_bcast_filtering.override); MVM_DEBUGFS_ADD_FILE_ALIAS("filters", bcast_filters, bcast_dir, 0600); @@ -2175,35 +2161,26 @@ int iwl_mvm_dbgfs_register(struct iwl_mvm *mvm, struct dentry *dbgfs_dir) #ifdef CONFIG_PM_SLEEP MVM_DEBUGFS_ADD_FILE(d3_sram, mvm->debugfs_dir, 0600); MVM_DEBUGFS_ADD_FILE(d3_test, mvm->debugfs_dir, 0400); - if (!debugfs_create_bool("d3_wake_sysassert", 0600, - mvm->debugfs_dir, &mvm->d3_wake_sysassert)) - goto err; - if (!debugfs_create_u32("last_netdetect_scans", 0400, - mvm->debugfs_dir, &mvm->last_netdetect_scans)) - goto err; + debugfs_create_bool("d3_wake_sysassert", 0600, mvm->debugfs_dir, + &mvm->d3_wake_sysassert); + debugfs_create_u32("last_netdetect_scans", 0400, mvm->debugfs_dir, + &mvm->last_netdetect_scans); #endif - if (!debugfs_create_u8("ps_disabled", 0400, - mvm->debugfs_dir, &mvm->ps_disabled)) - goto err; - if (!debugfs_create_blob("nvm_hw", 0400, - mvm->debugfs_dir, &mvm->nvm_hw_blob)) - goto err; - if (!debugfs_create_blob("nvm_sw", 0400, - mvm->debugfs_dir, &mvm->nvm_sw_blob)) - goto err; - if (!debugfs_create_blob("nvm_calib", 0400, - mvm->debugfs_dir, &mvm->nvm_calib_blob)) - goto err; - if (!debugfs_create_blob("nvm_prod", 0400, - mvm->debugfs_dir, &mvm->nvm_prod_blob)) - goto err; - if (!debugfs_create_blob("nvm_phy_sku", 0400, - mvm->debugfs_dir, &mvm->nvm_phy_sku_blob)) - goto err; - if (!debugfs_create_blob("nvm_reg", S_IRUSR, - mvm->debugfs_dir, &mvm->nvm_reg_blob)) - goto err; + debugfs_create_u8("ps_disabled", 0400, mvm->debugfs_dir, + &mvm->ps_disabled); + debugfs_create_blob("nvm_hw", 0400, mvm->debugfs_dir, + &mvm->nvm_hw_blob); + debugfs_create_blob("nvm_sw", 0400, mvm->debugfs_dir, + &mvm->nvm_sw_blob); + debugfs_create_blob("nvm_calib", 0400, mvm->debugfs_dir, + &mvm->nvm_calib_blob); + debugfs_create_blob("nvm_prod", 0400, mvm->debugfs_dir, + &mvm->nvm_prod_blob); + debugfs_create_blob("nvm_phy_sku", 0400, mvm->debugfs_dir, + &mvm->nvm_phy_sku_blob); + debugfs_create_blob("nvm_reg", S_IRUSR, + mvm->debugfs_dir, &mvm->nvm_reg_blob); debugfs_create_file("mem", 0600, dbgfs_dir, mvm, &iwl_dbgfs_mem_ops); @@ -2212,11 +2189,5 @@ int iwl_mvm_dbgfs_register(struct iwl_mvm *mvm, struct dentry *dbgfs_dir) * exists (before the opmode exists which removes the target.) */ snprintf(buf, 100, "../../%pd2", dbgfs_dir->d_parent); - if (!debugfs_create_symlink("iwlwifi", mvm->hw->wiphy->debugfsdir, buf)) - goto err; - - return 0; -err: - IWL_ERR(mvm, "Can't create the mvm debugfs directory\n"); - return -ENOMEM; + debugfs_create_symlink("iwlwifi", mvm->hw->wiphy->debugfsdir, buf); } diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/ftm-initiator.c b/drivers/net/wireless/intel/iwlwifi/mvm/ftm-initiator.c index 94132cfd1f56241b7e9548466c7050541c87e4c5..fec38a47696e05d739b373748f92e813371b6f88 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/ftm-initiator.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/ftm-initiator.c @@ -187,12 +187,24 @@ static void iwl_mvm_ftm_cmd(struct iwl_mvm *mvm, struct ieee80211_vif *vif, for (i = 0; i < ETH_ALEN; i++) cmd->macaddr_mask[i] = ~req->mac_addr_mask[i]; - if (vif->bss_conf.assoc) + if (vif->bss_conf.assoc) { memcpy(cmd->range_req_bssid, vif->bss_conf.bssid, ETH_ALEN); - else + + /* AP's TSF is only relevant if associated */ + for (i = 0; i < req->n_peers; i++) { + if (req->peers[i].report_ap_tsf) { + struct iwl_mvm_vif *mvmvif = + iwl_mvm_vif_from_mac80211(vif); + + cmd->tsf_mac_id = cpu_to_le32(mvmvif->id); + return; + } + } + } else { eth_broadcast_addr(cmd->range_req_bssid); + } - /* TODO: fill in tsf_mac_id if needed */ + /* Don't report AP's TSF */ cmd->tsf_mac_id = cpu_to_le32(0xff); } @@ -480,6 +492,7 @@ void iwl_mvm_ftm_range_resp(struct iwl_mvm *mvm, struct iwl_rx_cmd_buffer *rxb) { struct iwl_rx_packet *pkt = rxb_addr(rxb); struct iwl_tof_range_rsp_ntfy_v5 *fw_resp_v5 = (void *)pkt->data; + struct iwl_tof_range_rsp_ntfy_v6 *fw_resp_v6 = (void *)pkt->data; struct iwl_tof_range_rsp_ntfy *fw_resp = (void *)pkt->data; int i; bool new_api = fw_has_api(&mvm->fw->ucode_capa, @@ -519,8 +532,15 @@ void iwl_mvm_ftm_range_resp(struct iwl_mvm *mvm, struct iwl_rx_cmd_buffer *rxb) int peer_idx; if (new_api) { - fw_ap = &fw_resp->ap[i]; + if (fw_has_api(&mvm->fw->ucode_capa, + IWL_UCODE_TLV_API_FTM_RTT_ACCURACY)) + fw_ap = &fw_resp->ap[i]; + else + fw_ap = (void *)&fw_resp_v6->ap[i]; + result.final = fw_resp->ap[i].last_burst; + result.ap_tsf = le32_to_cpu(fw_ap->start_tsf); + result.ap_tsf_valid = 1; } else { /* the first part is the same for old and new APIs */ fw_ap = (void *)&fw_resp_v5->ap[i]; @@ -588,6 +608,11 @@ void iwl_mvm_ftm_range_resp(struct iwl_mvm *mvm, struct iwl_rx_cmd_buffer *rxb) mvm->ftm_initiator.req, &result, GFP_KERNEL); + if (fw_has_api(&mvm->fw->ucode_capa, + IWL_UCODE_TLV_API_FTM_RTT_ACCURACY)) + IWL_DEBUG_INFO(mvm, "RTT confidence: %hhu\n", + fw_ap->rttConfidence); + iwl_mvm_debug_range_resp(mvm, i, &result); } diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/mac-ctxt.c b/drivers/net/wireless/intel/iwlwifi/mvm/mac-ctxt.c index 6a70dece447dd1090d3f2d6d8b71c10a8d62b264..53c217af13c8b488becf591afbcb8fe7d7b10cf4 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/mac-ctxt.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/mac-ctxt.c @@ -8,7 +8,7 @@ * Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH * Copyright(c) 2015 - 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * * This program is free software; you can redistribute it and/or modify * it under the terms of version 2 of the GNU General Public License as @@ -31,7 +31,7 @@ * Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH * Copyright(c) 2015 - 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -262,9 +262,7 @@ int iwl_mvm_mac_ctxt_init(struct iwl_mvm *mvm, struct ieee80211_vif *vif) .preferred_tsf = NUM_TSF_IDS, .found_vif = false, }; - u32 ac; - int ret, i, queue_limit; - unsigned long used_hw_queues; + int ret, i; lockdep_assert_held(&mvm->mutex); @@ -341,37 +339,9 @@ int iwl_mvm_mac_ctxt_init(struct iwl_mvm *mvm, struct ieee80211_vif *vif) INIT_LIST_HEAD(&mvmvif->time_event_data.list); mvmvif->time_event_data.id = TE_MAX; - /* No need to allocate data queues to P2P Device MAC.*/ - if (vif->type == NL80211_IFTYPE_P2P_DEVICE) { - for (ac = 0; ac < IEEE80211_NUM_ACS; ac++) - vif->hw_queue[ac] = IEEE80211_INVAL_HW_QUEUE; - + /* No need to allocate data queues to P2P Device MAC and NAN.*/ + if (vif->type == NL80211_IFTYPE_P2P_DEVICE) return 0; - } - - /* - * queues in mac80211 almost entirely independent of - * the ones here - no real limit - */ - queue_limit = IEEE80211_MAX_QUEUES; - - /* - * Find available queues, and allocate them to the ACs. When in - * DQA-mode they aren't really used, and this is done only so the - * mac80211 ieee80211_check_queues() function won't fail - */ - for (ac = 0; ac < IEEE80211_NUM_ACS; ac++) { - u8 queue = find_first_zero_bit(&used_hw_queues, queue_limit); - - if (queue >= queue_limit) { - IWL_ERR(mvm, "Failed to allocate queue\n"); - ret = -EIO; - goto exit_fail; - } - - __set_bit(queue, &used_hw_queues); - vif->hw_queue[ac] = queue; - } /* Allocate the CAB queue for softAP and GO interfaces */ if (vif->type == NL80211_IFTYPE_AP || @@ -1143,9 +1113,7 @@ static void iwl_mvm_mac_ctxt_cmd_fill_ap(struct iwl_mvm *mvm, ieee80211_tu_to_usec(data.beacon_int * rand / 100); } else { - mvmvif->ap_beacon_time = - iwl_read_prph(mvm->trans, - DEVICE_SYSTEM_TIME_REG); + mvmvif->ap_beacon_time = iwl_mvm_get_systime(mvm); } } @@ -1573,6 +1541,7 @@ void iwl_mvm_channel_switch_noa_notif(struct iwl_mvm *mvm, rcu_read_lock(); vif = rcu_dereference(mvm->vif_id_to_mac[mac_id]); + mvmvif = iwl_mvm_vif_from_mac80211(vif); switch (vif->type) { case NL80211_IFTYPE_AP: @@ -1581,7 +1550,6 @@ void iwl_mvm_channel_switch_noa_notif(struct iwl_mvm *mvm, csa_vif != vif)) goto out_unlock; - mvmvif = iwl_mvm_vif_from_mac80211(csa_vif); csa_id = FW_CMD_ID_AND_COLOR(mvmvif->id, mvmvif->color); if (WARN(csa_id != id_n_color, "channel switch noa notification on unexpected vif (csa_vif=%d, notif=%d)", @@ -1602,6 +1570,7 @@ void iwl_mvm_channel_switch_noa_notif(struct iwl_mvm *mvm, return; case NL80211_IFTYPE_STATION: iwl_mvm_csa_client_absent(mvm, vif); + cancel_delayed_work(&mvmvif->csa_work); ieee80211_chswitch_done(vif, true); break; default: diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c b/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c index 6a3b11dd2edf53cf4352178a56189c1ebe99de66..5c52469288beb55ed5cd21f97a6c184b38c0c136 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c @@ -8,7 +8,7 @@ * Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH * Copyright(c) 2016 - 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * * This program is free software; you can redistribute it and/or modify * it under the terms of version 2 of the GNU General Public License as @@ -31,7 +31,7 @@ * Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH * Copyright(c) 2016 - 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -420,6 +420,7 @@ int iwl_mvm_init_fw_regd(struct iwl_mvm *mvm) const static u8 he_if_types_ext_capa_sta[] = { [0] = WLAN_EXT_CAPA1_EXT_CHANNEL_SWITCHING, + [2] = WLAN_EXT_CAPA3_MULTI_BSSID_SUPPORT, [7] = WLAN_EXT_CAPA8_OPMODE_NOTIF, [9] = WLAN_EXT_CAPA10_TWT_REQUESTER_SUPPORT, }; @@ -597,6 +598,9 @@ int iwl_mvm_mac_setup_register(struct iwl_mvm *mvm) BIT(NL80211_IFTYPE_ADHOC); hw->wiphy->flags |= WIPHY_FLAG_IBSS_RSN; + wiphy_ext_feature_set(hw->wiphy, NL80211_EXT_FEATURE_VHT_IBSS); + hw->wiphy->features |= NL80211_FEATURE_HT_IBSS; + hw->wiphy->regulatory_flags |= REGULATORY_ENABLE_RELAX_NO_IR; if (iwl_mvm_is_lar_supported(mvm)) hw->wiphy->regulatory_flags |= REGULATORY_WIPHY_SELF_MANAGED; @@ -732,6 +736,9 @@ int iwl_mvm_mac_setup_register(struct iwl_mvm *mvm) hw->wiphy->iftype_ext_capab = he_iftypes_ext_capa; hw->wiphy->num_iftype_ext_capab = ARRAY_SIZE(he_iftypes_ext_capa); + + ieee80211_hw_set(hw, SUPPORTS_MULTI_BSSID); + ieee80211_hw_set(hw, SUPPORTS_ONLY_HE_MULTI_BSSID); } mvm->rts_threshold = IEEE80211_MAX_RTS_THRESHOLD; @@ -1191,15 +1198,6 @@ static void iwl_mvm_cleanup_iterator(void *data, u8 *mac, static void iwl_mvm_restart_cleanup(struct iwl_mvm *mvm) { - /* clear the D3 reconfig, we only need it to avoid dumping a - * firmware coredump on reconfiguration, we shouldn't do that - * on D3->D0 transition - */ - if (!test_and_clear_bit(IWL_MVM_STATUS_D3_RECONFIG, &mvm->status)) { - mvm->fwrt.dump.desc = &iwl_dump_desc_assert; - iwl_fw_error_dump(&mvm->fwrt); - } - /* cleanup all stale references (scan, roc), but keep the * ucode_down ref until reconfig is complete */ @@ -1500,6 +1498,91 @@ static int iwl_mvm_set_tx_power(struct iwl_mvm *mvm, struct ieee80211_vif *vif, return iwl_mvm_send_cmd_pdu(mvm, REDUCE_TX_POWER_CMD, 0, len, &cmd); } +static int iwl_mvm_post_channel_switch(struct ieee80211_hw *hw, + struct ieee80211_vif *vif) +{ + struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif); + struct iwl_mvm *mvm = IWL_MAC80211_GET_MVM(hw); + int ret; + + mutex_lock(&mvm->mutex); + + if (mvmvif->csa_failed) { + mvmvif->csa_failed = false; + ret = -EIO; + goto out_unlock; + } + + if (vif->type == NL80211_IFTYPE_STATION) { + struct iwl_mvm_sta *mvmsta; + + mvmvif->csa_bcn_pending = false; + mvmsta = iwl_mvm_sta_from_staid_protected(mvm, + mvmvif->ap_sta_id); + + if (WARN_ON(!mvmsta)) { + ret = -EIO; + goto out_unlock; + } + + iwl_mvm_sta_modify_disable_tx(mvm, mvmsta, false); + + iwl_mvm_mac_ctxt_changed(mvm, vif, false, NULL); + + ret = iwl_mvm_enable_beacon_filter(mvm, vif, 0); + if (ret) + goto out_unlock; + + iwl_mvm_stop_session_protection(mvm, vif); + } + + mvmvif->ps_disabled = false; + + ret = iwl_mvm_power_update_ps(mvm); + +out_unlock: + mutex_unlock(&mvm->mutex); + + return ret; +} + +static void iwl_mvm_abort_channel_switch(struct ieee80211_hw *hw, + struct ieee80211_vif *vif) +{ + struct iwl_mvm *mvm = IWL_MAC80211_GET_MVM(hw); + struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif); + struct iwl_chan_switch_te_cmd cmd = { + .mac_id = cpu_to_le32(FW_CMD_ID_AND_COLOR(mvmvif->id, + mvmvif->color)), + .action = cpu_to_le32(FW_CTXT_ACTION_REMOVE), + }; + + IWL_DEBUG_MAC80211(mvm, "Abort CSA on mac %d\n", mvmvif->id); + + mutex_lock(&mvm->mutex); + WARN_ON(iwl_mvm_send_cmd_pdu(mvm, + WIDE_ID(MAC_CONF_GROUP, + CHANNEL_SWITCH_TIME_EVENT_CMD), + 0, sizeof(cmd), &cmd)); + mutex_unlock(&mvm->mutex); + + WARN_ON(iwl_mvm_post_channel_switch(hw, vif)); +} + +static void iwl_mvm_channel_switch_disconnect_wk(struct work_struct *wk) +{ + struct iwl_mvm *mvm; + struct iwl_mvm_vif *mvmvif; + struct ieee80211_vif *vif; + + mvmvif = container_of(wk, struct iwl_mvm_vif, csa_work.work); + vif = container_of((void *)mvmvif, struct ieee80211_vif, drv_priv); + mvm = mvmvif->mvm; + + iwl_mvm_abort_channel_switch(mvm->hw, vif); + ieee80211_chswitch_done(vif, false); +} + static int iwl_mvm_mac_add_interface(struct ieee80211_hw *hw, struct ieee80211_vif *vif) { @@ -1626,6 +1709,8 @@ static int iwl_mvm_mac_add_interface(struct ieee80211_hw *hw, } iwl_mvm_tcm_add_vif(mvm, vif); + INIT_DELAYED_WORK(&mvmvif->csa_work, + iwl_mvm_channel_switch_disconnect_wk); if (vif->type == NL80211_IFTYPE_MONITOR) mvm->monitor_on = true; @@ -2127,6 +2212,10 @@ static void iwl_mvm_cfg_he_sta(struct iwl_mvm *mvm, .frame_time_rts_th = cpu_to_le16(vif->bss_conf.frame_time_rts_th), }; + int size = fw_has_api(&mvm->fw->ucode_capa, + IWL_UCODE_TLV_API_MBSSID_HE) ? + sizeof(sta_ctxt_cmd) : + sizeof(struct iwl_he_sta_context_cmd_v1); struct ieee80211_sta *sta; u32 flags; int i; @@ -2254,16 +2343,18 @@ static void iwl_mvm_cfg_he_sta(struct iwl_mvm *mvm, /* Set the PPE thresholds accordingly */ if (low_th >= 0 && high_th >= 0) { - u8 ***pkt_ext_qam = - (void *)sta_ctxt_cmd.pkt_ext.pkt_ext_qam_th; + struct iwl_he_pkt_ext *pkt_ext = + (struct iwl_he_pkt_ext *)&sta_ctxt_cmd.pkt_ext; for (i = 0; i < MAX_HE_SUPP_NSS; i++) { u8 bw; for (bw = 0; bw < MAX_HE_CHANNEL_BW_INDX; bw++) { - pkt_ext_qam[i][bw][0] = low_th; - pkt_ext_qam[i][bw][1] = high_th; + pkt_ext->pkt_ext_qam_th[i][bw][0] = + low_th; + pkt_ext->pkt_ext_qam_th[i][bw][1] = + high_th; } } @@ -2308,13 +2399,23 @@ static void iwl_mvm_cfg_he_sta(struct iwl_mvm *mvm, (vif->bss_conf.uora_ocw_range >> 3) & 0x7; } - /* TODO: support Multi BSSID IE */ + if (vif->bss_conf.nontransmitted) { + flags |= STA_CTXT_HE_REF_BSSID_VALID; + ether_addr_copy(sta_ctxt_cmd.ref_bssid_addr, + vif->bss_conf.transmitter_bssid); + sta_ctxt_cmd.max_bssid_indicator = + vif->bss_conf.bssid_indicator; + sta_ctxt_cmd.bssid_index = vif->bss_conf.bssid_index; + sta_ctxt_cmd.ema_ap = vif->bss_conf.ema_ap; + sta_ctxt_cmd.profile_periodicity = + vif->bss_conf.profile_periodicity; + } sta_ctxt_cmd.flags = cpu_to_le32(flags); if (iwl_mvm_send_cmd_pdu(mvm, iwl_cmd_id(STA_HE_CTXT_CMD, DATA_PATH_GROUP, 0), - 0, sizeof(sta_ctxt_cmd), &sta_ctxt_cmd)) + 0, size, &sta_ctxt_cmd)) IWL_ERR(mvm, "Failed to config FW to work HE!\n"); } @@ -3612,7 +3713,7 @@ static int iwl_mvm_send_aux_roc_cmd(struct iwl_mvm *mvm, struct ieee80211_vif *vif, int duration) { - int res, time_reg = DEVICE_SYSTEM_TIME_REG; + int res; struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif); struct iwl_mvm_time_event_data *te_data = &mvmvif->hs_time_event_data; static const u16 time_event_response[] = { HOT_SPOT_CMD }; @@ -3638,7 +3739,7 @@ static int iwl_mvm_send_aux_roc_cmd(struct iwl_mvm *mvm, 0); /* Set the time and duration */ - tail->apply_time = cpu_to_le32(iwl_read_prph(mvm->trans, time_reg)); + tail->apply_time = cpu_to_le32(iwl_mvm_get_systime(mvm)); delay = AUX_ROC_MIN_DELAY; req_dur = MSEC_TO_TU(duration); @@ -4364,8 +4465,8 @@ static int __iwl_mvm_mac_testmode_cmd(struct iwl_mvm *mvm, int err; u32 noa_duration; - err = nla_parse(tb, IWL_MVM_TM_ATTR_MAX, data, len, iwl_mvm_tm_policy, - NULL); + err = nla_parse_deprecated(tb, IWL_MVM_TM_ATTR_MAX, data, len, + iwl_mvm_tm_policy, NULL); if (err) return err; @@ -4442,16 +4543,22 @@ static int iwl_mvm_schedule_client_csa(struct iwl_mvm *mvm, .action = cpu_to_le32(FW_CTXT_ACTION_ADD), .tsf = cpu_to_le32(chsw->timestamp), .cs_count = chsw->count, + .cs_mode = chsw->block_tx, }; lockdep_assert_held(&mvm->mutex); + if (chsw->delay) + cmd.cs_delayed_bcn_count = + DIV_ROUND_UP(chsw->delay, vif->bss_conf.beacon_int); + return iwl_mvm_send_cmd_pdu(mvm, WIDE_ID(MAC_CONF_GROUP, CHANNEL_SWITCH_TIME_EVENT_CMD), 0, sizeof(cmd), &cmd); } +#define IWL_MAX_CSA_BLOCK_TX 1500 static int iwl_mvm_pre_channel_switch(struct ieee80211_hw *hw, struct ieee80211_vif *vif, struct ieee80211_channel_switch *chsw) @@ -4516,8 +4623,18 @@ static int iwl_mvm_pre_channel_switch(struct ieee80211_hw *hw, ((vif->bss_conf.beacon_int * (chsw->count - 1) - IWL_MVM_CHANNEL_SWITCH_TIME_CLIENT) * 1024); - if (chsw->block_tx) + if (chsw->block_tx) { iwl_mvm_csa_client_absent(mvm, vif); + /* + * In case of undetermined / long time with immediate + * quiet monitor status to gracefully disconnect + */ + if (!chsw->count || + chsw->count * vif->bss_conf.beacon_int > + IWL_MAX_CSA_BLOCK_TX) + schedule_delayed_work(&mvmvif->csa_work, + msecs_to_jiffies(IWL_MAX_CSA_BLOCK_TX)); + } if (mvmvif->bf_data.bf_enabled) { ret = iwl_mvm_disable_beacon_filter(mvm, vif, 0); @@ -4532,6 +4649,9 @@ static int iwl_mvm_pre_channel_switch(struct ieee80211_hw *hw, iwl_mvm_schedule_csa_period(mvm, vif, vif->bss_conf.beacon_int, apply_time); + + mvmvif->csa_count = chsw->count; + mvmvif->csa_misbehave = false; break; default: break; @@ -4552,52 +4672,42 @@ out_unlock: return ret; } -static int iwl_mvm_post_channel_switch(struct ieee80211_hw *hw, - struct ieee80211_vif *vif) +static void iwl_mvm_channel_switch_rx_beacon(struct ieee80211_hw *hw, + struct ieee80211_vif *vif, + struct ieee80211_channel_switch *chsw) { - struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif); struct iwl_mvm *mvm = IWL_MAC80211_GET_MVM(hw); - int ret; - - mutex_lock(&mvm->mutex); - - if (mvmvif->csa_failed) { - mvmvif->csa_failed = false; - ret = -EIO; - goto out_unlock; - } - - if (vif->type == NL80211_IFTYPE_STATION) { - struct iwl_mvm_sta *mvmsta; + struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif); + struct iwl_chan_switch_te_cmd cmd = { + .mac_id = cpu_to_le32(FW_CMD_ID_AND_COLOR(mvmvif->id, + mvmvif->color)), + .action = cpu_to_le32(FW_CTXT_ACTION_MODIFY), + .tsf = cpu_to_le32(chsw->timestamp), + .cs_count = chsw->count, + .cs_mode = chsw->block_tx, + }; - mvmvif->csa_bcn_pending = false; - mvmsta = iwl_mvm_sta_from_staid_protected(mvm, - mvmvif->ap_sta_id); + if (!fw_has_capa(&mvm->fw->ucode_capa, IWL_UCODE_TLV_CAPA_CS_MODIFY)) + return; - if (WARN_ON(!mvmsta)) { - ret = -EIO; - goto out_unlock; + if (chsw->count >= mvmvif->csa_count && chsw->block_tx) { + if (mvmvif->csa_misbehave) { + /* Second time, give up on this AP*/ + iwl_mvm_abort_channel_switch(hw, vif); + ieee80211_chswitch_done(vif, false); + mvmvif->csa_misbehave = false; + return; } - - iwl_mvm_sta_modify_disable_tx(mvm, mvmsta, false); - - iwl_mvm_mac_ctxt_changed(mvm, vif, false, NULL); - - ret = iwl_mvm_enable_beacon_filter(mvm, vif, 0); - if (ret) - goto out_unlock; - - iwl_mvm_stop_session_protection(mvm, vif); + mvmvif->csa_misbehave = true; } + mvmvif->csa_count = chsw->count; - mvmvif->ps_disabled = false; - - ret = iwl_mvm_power_update_ps(mvm); - -out_unlock: - mutex_unlock(&mvm->mutex); + IWL_DEBUG_MAC80211(mvm, "Modify CSA on mac %d\n", mvmvif->id); - return ret; + WARN_ON(iwl_mvm_send_cmd_pdu(mvm, + WIDE_ID(MAC_CONF_GROUP, + CHANNEL_SWITCH_TIME_EVENT_CMD), + CMD_ASYNC, sizeof(cmd), &cmd)); } static void iwl_mvm_flush_no_vif(struct iwl_mvm *mvm, u32 queues, bool drop) @@ -5056,6 +5166,8 @@ const struct ieee80211_ops iwl_mvm_hw_ops = { .channel_switch = iwl_mvm_channel_switch, .pre_channel_switch = iwl_mvm_pre_channel_switch, .post_channel_switch = iwl_mvm_post_channel_switch, + .abort_channel_switch = iwl_mvm_abort_channel_switch, + .channel_switch_rx_beacon = iwl_mvm_channel_switch_rx_beacon, .tdls_channel_switch = iwl_mvm_tdls_channel_switch, .tdls_cancel_channel_switch = iwl_mvm_tdls_cancel_channel_switch, diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/mvm.h b/drivers/net/wireless/intel/iwlwifi/mvm/mvm.h index a50dc53df08698ff0afafbef8f74692007afc17b..8dc2a9850bc584c4a009caed8fd2c9725a7853ae 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/mvm.h +++ b/drivers/net/wireless/intel/iwlwifi/mvm/mvm.h @@ -8,7 +8,7 @@ * Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH * Copyright(c) 2016 - 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * * This program is free software; you can redistribute it and/or modify * it under the terms of version 2 of the GNU General Public License as @@ -31,7 +31,7 @@ * Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH * Copyright(c) 2016 - 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -490,6 +490,9 @@ struct iwl_mvm_vif { bool csa_countdown; bool csa_failed; u16 csa_target_freq; + u16 csa_count; + u16 csa_misbehave; + struct delayed_work csa_work; /* Indicates that we are waiting for a beacon on a new channel */ bool csa_bcn_pending; @@ -1199,7 +1202,6 @@ struct iwl_mvm { * @IWL_MVM_STATUS_IN_HW_RESTART: HW restart is active * @IWL_MVM_STATUS_IN_D0I3: NIC is in D0i3 * @IWL_MVM_STATUS_ROC_AUX_RUNNING: AUX remain-on-channel is running - * @IWL_MVM_STATUS_D3_RECONFIG: D3 reconfiguration is being done * @IWL_MVM_STATUS_FIRMWARE_RUNNING: firmware is running * @IWL_MVM_STATUS_NEED_FLUSH_P2P: need to flush P2P bcast STA */ @@ -1211,7 +1213,6 @@ enum iwl_mvm_status { IWL_MVM_STATUS_IN_HW_RESTART, IWL_MVM_STATUS_IN_D0I3, IWL_MVM_STATUS_ROC_AUX_RUNNING, - IWL_MVM_STATUS_D3_RECONFIG, IWL_MVM_STATUS_FIRMWARE_RUNNING, IWL_MVM_STATUS_NEED_FLUSH_P2P, }; @@ -1537,6 +1538,7 @@ void iwl_mvm_dump_nic_error_log(struct iwl_mvm *mvm); u8 first_antenna(u8 mask); u8 iwl_mvm_next_antenna(struct iwl_mvm *mvm, u8 valid, u8 last_idx); void iwl_mvm_get_sync_time(struct iwl_mvm *mvm, u32 *gp2, u64 *boottime); +u32 iwl_mvm_get_systime(struct iwl_mvm *mvm); /* Tx / Host Commands */ int __must_check iwl_mvm_send_cmd(struct iwl_mvm *mvm, @@ -1649,8 +1651,8 @@ void iwl_mvm_rx_rx_mpdu(struct iwl_mvm *mvm, struct napi_struct *napi, struct iwl_rx_cmd_buffer *rxb); void iwl_mvm_rx_mpdu_mq(struct iwl_mvm *mvm, struct napi_struct *napi, struct iwl_rx_cmd_buffer *rxb, int queue); -void iwl_mvm_rx_monitor_ndp(struct iwl_mvm *mvm, struct napi_struct *napi, - struct iwl_rx_cmd_buffer *rxb, int queue); +void iwl_mvm_rx_monitor_no_data(struct iwl_mvm *mvm, struct napi_struct *napi, + struct iwl_rx_cmd_buffer *rxb, int queue); void iwl_mvm_rx_frame_release(struct iwl_mvm *mvm, struct napi_struct *napi, struct iwl_rx_cmd_buffer *rxb, int queue); int iwl_mvm_notify_rx_queue(struct iwl_mvm *mvm, u32 rxq_mask, @@ -1784,14 +1786,13 @@ void iwl_mvm_rx_umac_scan_iter_complete_notif(struct iwl_mvm *mvm, /* MVM debugfs */ #ifdef CONFIG_IWLWIFI_DEBUGFS -int iwl_mvm_dbgfs_register(struct iwl_mvm *mvm, struct dentry *dbgfs_dir); +void iwl_mvm_dbgfs_register(struct iwl_mvm *mvm, struct dentry *dbgfs_dir); void iwl_mvm_vif_dbgfs_register(struct iwl_mvm *mvm, struct ieee80211_vif *vif); void iwl_mvm_vif_dbgfs_clean(struct iwl_mvm *mvm, struct ieee80211_vif *vif); #else -static inline int iwl_mvm_dbgfs_register(struct iwl_mvm *mvm, - struct dentry *dbgfs_dir) +static inline void iwl_mvm_dbgfs_register(struct iwl_mvm *mvm, + struct dentry *dbgfs_dir) { - return 0; } static inline void iwl_mvm_vif_dbgfs_register(struct iwl_mvm *mvm, struct ieee80211_vif *vif) @@ -2023,17 +2024,6 @@ static inline u32 iwl_mvm_flushable_queues(struct iwl_mvm *mvm) static inline void iwl_mvm_stop_device(struct iwl_mvm *mvm) { lockdep_assert_held(&mvm->mutex); - /* If IWL_MVM_STATUS_HW_RESTART_REQUESTED bit is set then we received - * an assert. Since we failed to bring the interface up, mac80211 - * will not attempt to reconfig the device, - * which handles the dump collection in assert flow, - * so trigger dump collection here. - */ - if (test_and_clear_bit(IWL_MVM_STATUS_HW_RESTART_REQUESTED, - &mvm->status)) - iwl_fw_dbg_collect_desc(&mvm->fwrt, &iwl_dump_desc_assert, - false, 0); - iwl_fw_cancel_timestamp(&mvm->fwrt); clear_bit(IWL_MVM_STATUS_FIRMWARE_RUNNING, &mvm->status); iwl_fwrt_stop_device(&mvm->fwrt); diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/ops.c b/drivers/net/wireless/intel/iwlwifi/mvm/ops.c index 13681b03c10e15a4a42d9a7c598b91d80726abd7..acd2fda124665f651cb939281df40ef1c3163bc6 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/ops.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/ops.c @@ -8,7 +8,7 @@ * Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH * Copyright(c) 2016 - 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * * This program is free software; you can redistribute it and/or modify * it under the terms of version 2 of the GNU General Public License as @@ -31,7 +31,7 @@ * Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH * Copyright(c) 2016 - 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -862,9 +862,7 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg, min_backoff = iwl_mvm_min_backoff(mvm); iwl_mvm_thermal_initialize(mvm, min_backoff); - err = iwl_mvm_dbgfs_register(mvm, dbgfs_dir); - if (err) - goto out_unregister; + iwl_mvm_dbgfs_register(mvm, dbgfs_dir); if (!iwl_mvm_has_new_rx_stats_api(mvm)) memset(&mvm->rx_stats_v3, 0, @@ -881,14 +879,6 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg, return op_mode; - out_unregister: - if (iwlmvm_mod_params.init_dbg) - return op_mode; - - ieee80211_unregister_hw(mvm->hw); - mvm->hw_registered = false; - iwl_mvm_leds_exit(mvm); - iwl_mvm_thermal_exit(mvm); out_free: iwl_fw_flush_dump(&mvm->fwrt); iwl_fw_runtime_free(&mvm->fwrt); @@ -1105,7 +1095,7 @@ static void iwl_mvm_rx_mq(struct iwl_op_mode *op_mode, else if (cmd == WIDE_ID(LEGACY_GROUP, FRAME_RELEASE)) iwl_mvm_rx_frame_release(mvm, napi, rxb, 0); else if (cmd == WIDE_ID(DATA_PATH_GROUP, RX_NO_DATA_NOTIF)) - iwl_mvm_rx_monitor_ndp(mvm, napi, rxb, 0); + iwl_mvm_rx_monitor_no_data(mvm, napi, rxb, 0); else iwl_mvm_rx_common(mvm, rxb, pkt); } @@ -1271,6 +1261,7 @@ static void iwl_mvm_reprobe_wk(struct work_struct *wk) void iwl_mvm_nic_restart(struct iwl_mvm *mvm, bool fw_error) { iwl_abort_notification_waits(&mvm->notif_wait); + del_timer(&mvm->fwrt.dump.periodic_trig); /* * This is a bit racy, but worst case we tell mac80211 about @@ -1291,8 +1282,7 @@ void iwl_mvm_nic_restart(struct iwl_mvm *mvm, bool fw_error) * can't recover this since we're already half suspended. */ if (!mvm->fw_restart && fw_error) { - iwl_fw_dbg_collect_desc(&mvm->fwrt, &iwl_dump_desc_assert, - false, 0); + iwl_fw_error_collect(&mvm->fwrt); } else if (test_bit(IWL_MVM_STATUS_IN_HW_RESTART, &mvm->status)) { struct iwl_mvm_reprobe *reprobe; @@ -1340,6 +1330,8 @@ void iwl_mvm_nic_restart(struct iwl_mvm *mvm, bool fw_error) } } + iwl_fw_error_collect(&mvm->fwrt); + if (fw_error && mvm->fw_restart > 0) mvm->fw_restart--; set_bit(IWL_MVM_STATUS_HW_RESTART_REQUESTED, &mvm->status); diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/rs-fw.c b/drivers/net/wireless/intel/iwlwifi/mvm/rs-fw.c index a28283ff7295afc24b7a37ded7c8d269cdfab221..659e21b2d4e7293ed913d60f9285c25d42862187 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/rs-fw.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/rs-fw.c @@ -6,7 +6,7 @@ * GPL LICENSE SUMMARY * * Copyright(c) 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * * This program is free software; you can redistribute it and/or modify * it under the terms of version 2 of the GNU General Public License as @@ -27,7 +27,7 @@ * BSD LICENSE * * Copyright(c) 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -116,8 +116,9 @@ static u8 rs_fw_sgi_cw_support(struct ieee80211_sta *sta) return supp; } -static u16 rs_fw_set_config_flags(struct iwl_mvm *mvm, - struct ieee80211_sta *sta) +static u16 rs_fw_get_config_flags(struct iwl_mvm *mvm, + struct ieee80211_sta *sta, + struct ieee80211_supported_band *sband) { struct ieee80211_sta_ht_cap *ht_cap = &sta->ht_cap; struct ieee80211_sta_vht_cap *vht_cap = &sta->vht_cap; @@ -147,6 +148,12 @@ static u16 rs_fw_set_config_flags(struct iwl_mvm *mvm, (vht_ena && (vht_cap->cap & IEEE80211_VHT_CAP_RXLDPC)))) flags |= IWL_TLC_MNG_CFG_FLAGS_LDPC_MSK; + /* consider our LDPC support in case of HE */ + if (sband->iftype_data && sband->iftype_data->he_cap.has_he && + !(sband->iftype_data->he_cap.he_cap_elem.phy_cap_info[1] & + IEEE80211_HE_PHY_CAP1_LDPC_CODING_IN_PAYLOAD)) + flags &= ~IWL_TLC_MNG_CFG_FLAGS_LDPC_MSK; + if (he_cap && he_cap->has_he && (he_cap->he_cap_elem.phy_cap_info[3] & IEEE80211_HE_PHY_CAP3_DCM_MAX_CONST_RX_MASK)) @@ -223,19 +230,43 @@ static u16 rs_fw_he_ieee80211_mcs_to_rs_mcs(u16 mcs) static void rs_fw_he_set_enabled_rates(const struct ieee80211_sta *sta, - const struct ieee80211_sta_he_cap *he_cap, + struct ieee80211_supported_band *sband, struct iwl_tlc_config_cmd *cmd) { - u16 mcs_160 = le16_to_cpu(sta->he_cap.he_mcs_nss_supp.rx_mcs_160); - u16 mcs_80 = le16_to_cpu(sta->he_cap.he_mcs_nss_supp.rx_mcs_80); + const struct ieee80211_sta_he_cap *he_cap = &sta->he_cap; + u16 mcs_160 = le16_to_cpu(he_cap->he_mcs_nss_supp.rx_mcs_160); + u16 mcs_80 = le16_to_cpu(he_cap->he_mcs_nss_supp.rx_mcs_80); + u16 tx_mcs_80 = + le16_to_cpu(sband->iftype_data->he_cap.he_mcs_nss_supp.tx_mcs_80); + u16 tx_mcs_160 = + le16_to_cpu(sband->iftype_data->he_cap.he_mcs_nss_supp.tx_mcs_160); int i; for (i = 0; i < sta->rx_nss && i < MAX_NSS; i++) { u16 _mcs_160 = (mcs_160 >> (2 * i)) & 0x3; u16 _mcs_80 = (mcs_80 >> (2 * i)) & 0x3; - + u16 _tx_mcs_160 = (tx_mcs_160 >> (2 * i)) & 0x3; + u16 _tx_mcs_80 = (tx_mcs_80 >> (2 * i)) & 0x3; + + /* If one side doesn't support - mark both as not supporting */ + if (_mcs_80 == IEEE80211_HE_MCS_NOT_SUPPORTED || + _tx_mcs_80 == IEEE80211_HE_MCS_NOT_SUPPORTED) { + _mcs_80 = IEEE80211_HE_MCS_NOT_SUPPORTED; + _tx_mcs_80 = IEEE80211_HE_MCS_NOT_SUPPORTED; + } + if (_mcs_80 > _tx_mcs_80) + _mcs_80 = _tx_mcs_80; cmd->ht_rates[i][0] = cpu_to_le16(rs_fw_he_ieee80211_mcs_to_rs_mcs(_mcs_80)); + + /* If one side doesn't support - mark both as not supporting */ + if (_mcs_160 == IEEE80211_HE_MCS_NOT_SUPPORTED || + _tx_mcs_160 == IEEE80211_HE_MCS_NOT_SUPPORTED) { + _mcs_160 = IEEE80211_HE_MCS_NOT_SUPPORTED; + _tx_mcs_160 = IEEE80211_HE_MCS_NOT_SUPPORTED; + } + if (_mcs_160 > _tx_mcs_160) + _mcs_160 = _tx_mcs_160; cmd->ht_rates[i][1] = cpu_to_le16(rs_fw_he_ieee80211_mcs_to_rs_mcs(_mcs_160)); } @@ -264,7 +295,7 @@ static void rs_fw_set_supp_rates(struct ieee80211_sta *sta, /* HT/VHT rates */ if (he_cap && he_cap->has_he) { cmd->mode = IWL_TLC_MNG_MODE_HE; - rs_fw_he_set_enabled_rates(sta, he_cap, cmd); + rs_fw_he_set_enabled_rates(sta, sband, cmd); } else if (vht_cap && vht_cap->vht_supported) { cmd->mode = IWL_TLC_MNG_MODE_VHT; rs_fw_vht_set_enabled_rates(sta, vht_cap, cmd); @@ -345,6 +376,37 @@ out: rcu_read_unlock(); } +static u16 rs_fw_get_max_amsdu_len(struct ieee80211_sta *sta) +{ + const struct ieee80211_sta_vht_cap *vht_cap = &sta->vht_cap; + const struct ieee80211_sta_ht_cap *ht_cap = &sta->ht_cap; + + if (vht_cap && vht_cap->vht_supported) { + switch (vht_cap->cap & IEEE80211_VHT_CAP_MAX_MPDU_MASK) { + case IEEE80211_VHT_CAP_MAX_MPDU_LENGTH_11454: + return IEEE80211_MAX_MPDU_LEN_VHT_11454; + case IEEE80211_VHT_CAP_MAX_MPDU_LENGTH_7991: + return IEEE80211_MAX_MPDU_LEN_VHT_7991; + default: + return IEEE80211_MAX_MPDU_LEN_VHT_3895; + } + + } else if (ht_cap && ht_cap->ht_supported) { + if (ht_cap->cap & IEEE80211_HT_CAP_MAX_AMSDU) + /* + * agg is offloaded so we need to assume that agg + * are enabled and max mpdu in ampdu is 4095 + * (spec 802.11-2016 9.3.2.1) + */ + return IEEE80211_MAX_MPDU_LEN_HT_BA; + else + return IEEE80211_MAX_MPDU_LEN_HT_3839; + } + + /* in legacy mode no amsdu is enabled so return zero */ + return 0; +} + void rs_fw_rate_init(struct iwl_mvm *mvm, struct ieee80211_sta *sta, enum nl80211_band band, bool update) { @@ -352,15 +414,16 @@ void rs_fw_rate_init(struct iwl_mvm *mvm, struct ieee80211_sta *sta, struct iwl_mvm_sta *mvmsta = iwl_mvm_sta_from_mac80211(sta); struct iwl_lq_sta_rs_fw *lq_sta = &mvmsta->lq_sta.rs_fw; u32 cmd_id = iwl_cmd_id(TLC_MNG_CONFIG_CMD, DATA_PATH_GROUP, 0); - struct ieee80211_supported_band *sband; + struct ieee80211_supported_band *sband = hw->wiphy->bands[band]; + u16 max_amsdu_len = rs_fw_get_max_amsdu_len(sta); struct iwl_tlc_config_cmd cfg_cmd = { .sta_id = mvmsta->sta_id, .max_ch_width = update ? rs_fw_bw_from_sta_bw(sta) : RATE_MCS_CHAN_WIDTH_20, - .flags = cpu_to_le16(rs_fw_set_config_flags(mvm, sta)), + .flags = cpu_to_le16(rs_fw_get_config_flags(mvm, sta, sband)), .chains = rs_fw_set_active_chains(iwl_mvm_get_valid_tx_ant(mvm)), - .max_mpdu_len = cpu_to_le16(sta->max_amsdu_len), .sgi_ch_width_supp = rs_fw_sgi_cw_support(sta), + .max_mpdu_len = cpu_to_le16(max_amsdu_len), .amsdu = iwl_mvm_is_csum_supported(mvm), }; int ret; @@ -370,9 +433,14 @@ void rs_fw_rate_init(struct iwl_mvm *mvm, struct ieee80211_sta *sta, #ifdef CONFIG_IWLWIFI_DEBUGFS iwl_mvm_reset_frame_stats(mvm); #endif - sband = hw->wiphy->bands[band]; rs_fw_set_supp_rates(sta, sband, &cfg_cmd); + /* + * since TLC offload works with one mode we can assume + * that only vht/ht is used and also set it as station max amsdu + */ + sta->max_amsdu_len = max_amsdu_len; + ret = iwl_mvm_send_cmd_pdu(mvm, cmd_id, 0, sizeof(cfg_cmd), &cfg_cmd); if (ret) IWL_ERR(mvm, "Failed to send rate scale config (%d)\n", ret); diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/rs.c b/drivers/net/wireless/intel/iwlwifi/mvm/rs.c index e231a44d242384958b0c6d0dd8221bc2623cb1aa..c182821ab22b166c51ddbe66fca24d69958f4ec6 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/rs.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/rs.c @@ -4078,9 +4078,8 @@ static ssize_t iwl_dbgfs_ss_force_write(struct iwl_lq_sta *lq_sta, char *buf, #define MVM_DEBUGFS_READ_WRITE_FILE_OPS(name, bufsz) \ _MVM_DEBUGFS_READ_WRITE_FILE_OPS(name, bufsz, struct iwl_lq_sta) #define MVM_DEBUGFS_ADD_FILE_RS(name, parent, mode) do { \ - if (!debugfs_create_file(#name, mode, parent, lq_sta, \ - &iwl_dbgfs_##name##_ops)) \ - goto err; \ + debugfs_create_file(#name, mode, parent, lq_sta, \ + &iwl_dbgfs_##name##_ops); \ } while (0) MVM_DEBUGFS_READ_WRITE_FILE_OPS(ss_force, 32); @@ -4108,9 +4107,6 @@ static void rs_drv_add_sta_debugfs(void *mvm, void *priv_sta, &lq_sta->pers.dbg_fixed_txp_reduction); MVM_DEBUGFS_ADD_FILE_RS(ss_force, dir, 0600); - return; -err: - IWL_ERR((struct iwl_mvm *)mvm, "Can't create debugfs entity\n"); } void rs_remove_sta_debugfs(void *mvm, void *mvm_sta) diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/rxmq.c b/drivers/net/wireless/intel/iwlwifi/mvm/rxmq.c index b516fd1867ecf9f1cac26b540bd90c3c14c1e7e1..1824566d08fc79eb61a11f35b65380a78294b140 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/rxmq.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/rxmq.c @@ -8,7 +8,7 @@ * Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH * Copyright(c) 2015 - 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * * This program is free software; you can redistribute it and/or modify * it under the terms of version 2 of the GNU General Public License as @@ -31,7 +31,7 @@ * Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH * Copyright(c) 2015 - 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -1699,8 +1699,8 @@ out: rcu_read_unlock(); } -void iwl_mvm_rx_monitor_ndp(struct iwl_mvm *mvm, struct napi_struct *napi, - struct iwl_rx_cmd_buffer *rxb, int queue) +void iwl_mvm_rx_monitor_no_data(struct iwl_mvm *mvm, struct napi_struct *napi, + struct iwl_rx_cmd_buffer *rxb, int queue) { struct ieee80211_rx_status *rx_status; struct iwl_rx_packet *pkt = rxb_addr(rxb); @@ -1721,10 +1721,6 @@ void iwl_mvm_rx_monitor_ndp(struct iwl_mvm *mvm, struct napi_struct *napi, if (unlikely(test_bit(IWL_MVM_STATUS_IN_HW_RESTART, &mvm->status))) return; - /* Currently only NDP type is supported */ - if (info_type != RX_NO_DATA_INFO_TYPE_NDP) - return; - energy_a = (rssi & RX_NO_DATA_CHAIN_A_MSK) >> RX_NO_DATA_CHAIN_A_POS; energy_b = (rssi & RX_NO_DATA_CHAIN_B_MSK) >> RX_NO_DATA_CHAIN_B_POS; channel = (rssi & RX_NO_DATA_CHANNEL_MSK) >> RX_NO_DATA_CHANNEL_POS; @@ -1746,9 +1742,22 @@ void iwl_mvm_rx_monitor_ndp(struct iwl_mvm *mvm, struct napi_struct *napi, /* 0-length PSDU */ rx_status->flag |= RX_FLAG_NO_PSDU; - /* currently this is the only type for which we get this notif */ - rx_status->zero_length_psdu_type = - IEEE80211_RADIOTAP_ZERO_LEN_PSDU_SOUNDING; + + switch (info_type) { + case RX_NO_DATA_INFO_TYPE_NDP: + rx_status->zero_length_psdu_type = + IEEE80211_RADIOTAP_ZERO_LEN_PSDU_SOUNDING; + break; + case RX_NO_DATA_INFO_TYPE_MU_UNMATCHED: + case RX_NO_DATA_INFO_TYPE_HE_TB_UNMATCHED: + rx_status->zero_length_psdu_type = + IEEE80211_RADIOTAP_ZERO_LEN_PSDU_NOT_CAPTURED; + break; + default: + rx_status->zero_length_psdu_type = + IEEE80211_RADIOTAP_ZERO_LEN_PSDU_VENDOR; + break; + } /* This may be overridden by iwl_mvm_rx_he() to HE_RU */ switch (rate_n_flags & RATE_MCS_CHAN_WIDTH_MSK) { diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/scan.c b/drivers/net/wireless/intel/iwlwifi/mvm/scan.c index 78694bc38e76a0017601f9af4ffc397edd0b9c2e..d9ddf9ff6428e306f9899f7a8b69f6c197ce9387 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/scan.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/scan.c @@ -1082,21 +1082,23 @@ static void iwl_mvm_fill_scan_dwell(struct iwl_mvm *mvm, dwell->extended = IWL_SCAN_DWELL_EXTENDED; } -static void iwl_mvm_fill_channels(struct iwl_mvm *mvm, u8 *channels) +static void iwl_mvm_fill_channels(struct iwl_mvm *mvm, u8 *channels, + u32 max_channels) { struct ieee80211_supported_band *band; int i, j = 0; band = &mvm->nvm_data->bands[NL80211_BAND_2GHZ]; - for (i = 0; i < band->n_channels; i++, j++) + for (i = 0; i < band->n_channels && j < max_channels; i++, j++) channels[j] = band->channels[i].hw_value; band = &mvm->nvm_data->bands[NL80211_BAND_5GHZ]; - for (i = 0; i < band->n_channels; i++, j++) + for (i = 0; i < band->n_channels && j < max_channels; i++, j++) channels[j] = band->channels[i].hw_value; } static void iwl_mvm_fill_scan_config_v1(struct iwl_mvm *mvm, void *config, - u32 flags, u8 channel_flags) + u32 flags, u8 channel_flags, + u32 max_channels) { enum iwl_mvm_scan_type type = iwl_mvm_get_scan_type(mvm, NULL); struct iwl_scan_config_v1 *cfg = config; @@ -1115,11 +1117,12 @@ static void iwl_mvm_fill_scan_config_v1(struct iwl_mvm *mvm, void *config, cfg->bcast_sta_id = mvm->aux_sta.sta_id; cfg->channel_flags = channel_flags; - iwl_mvm_fill_channels(mvm, cfg->channel_array); + iwl_mvm_fill_channels(mvm, cfg->channel_array, max_channels); } static void iwl_mvm_fill_scan_config(struct iwl_mvm *mvm, void *config, - u32 flags, u8 channel_flags) + u32 flags, u8 channel_flags, + u32 max_channels) { struct iwl_scan_config *cfg = config; @@ -1162,7 +1165,7 @@ static void iwl_mvm_fill_scan_config(struct iwl_mvm *mvm, void *config, cfg->bcast_sta_id = mvm->aux_sta.sta_id; cfg->channel_flags = channel_flags; - iwl_mvm_fill_channels(mvm, cfg->channel_array); + iwl_mvm_fill_channels(mvm, cfg->channel_array, max_channels); } int iwl_mvm_config_scan(struct iwl_mvm *mvm) @@ -1181,7 +1184,7 @@ int iwl_mvm_config_scan(struct iwl_mvm *mvm) u8 channel_flags; if (WARN_ON(num_channels > mvm->fw->ucode_capa.n_scan_channels)) - return -ENOBUFS; + num_channels = mvm->fw->ucode_capa.n_scan_channels; if (iwl_mvm_is_cdb_supported(mvm)) { type = iwl_mvm_get_scan_type_band(mvm, NULL, @@ -1234,9 +1237,11 @@ int iwl_mvm_config_scan(struct iwl_mvm *mvm) flags |= (iwl_mvm_is_scan_fragmented(hb_type)) ? SCAN_CONFIG_FLAG_SET_LMAC2_FRAGMENTED : SCAN_CONFIG_FLAG_CLEAR_LMAC2_FRAGMENTED; - iwl_mvm_fill_scan_config(mvm, cfg, flags, channel_flags); + iwl_mvm_fill_scan_config(mvm, cfg, flags, channel_flags, + num_channels); } else { - iwl_mvm_fill_scan_config_v1(mvm, cfg, flags, channel_flags); + iwl_mvm_fill_scan_config_v1(mvm, cfg, flags, channel_flags, + num_channels); } cmd.data[0] = cfg; diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/sta.c b/drivers/net/wireless/intel/iwlwifi/mvm/sta.c index 98d123dd7177845ff1676df2dbbdb2c492d89f6f..f545a737a92dfa8321b83ce00748c1833ec7c854 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/sta.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/sta.c @@ -746,7 +746,8 @@ static int iwl_mvm_find_free_queue(struct iwl_mvm *mvm, u8 sta_id, static int iwl_mvm_tvqm_enable_txq(struct iwl_mvm *mvm, u8 sta_id, u8 tid, unsigned int timeout) { - int queue, size = IWL_DEFAULT_QUEUE_SIZE; + int queue, size = max_t(u32, IWL_DEFAULT_QUEUE_SIZE, + mvm->trans->cfg->min_256_ba_txq_size); if (tid == IWL_MAX_TID_COUNT) { tid = IWL_MGMT_TID; @@ -2109,12 +2110,14 @@ int iwl_mvm_send_add_bcast_sta(struct iwl_mvm *mvm, struct ieee80211_vif *vif) if (!iwl_mvm_has_new_tx_api(mvm)) { if (vif->type == NL80211_IFTYPE_AP || - vif->type == NL80211_IFTYPE_ADHOC) + vif->type == NL80211_IFTYPE_ADHOC) { queue = mvm->probe_queue; - else if (vif->type == NL80211_IFTYPE_P2P_DEVICE) + } else if (vif->type == NL80211_IFTYPE_P2P_DEVICE) { queue = mvm->p2p_dev_queue; - else if (WARN(1, "Missing required TXQ for adding bcast STA\n")) + } else { + WARN(1, "Missing required TXQ for adding bcast STA\n"); return -EINVAL; + } bsta->tfd_queue_msk |= BIT(queue); @@ -2277,7 +2280,8 @@ int iwl_mvm_add_mcast_sta(struct iwl_mvm *mvm, struct ieee80211_vif *vif) static const u8 _maddr[] = {0x03, 0x00, 0x00, 0x00, 0x00, 0x00}; const u8 *maddr = _maddr; struct iwl_trans_txq_scd_cfg cfg = { - .fifo = IWL_MVM_TX_FIFO_MCAST, + .fifo = vif->type == NL80211_IFTYPE_AP ? + IWL_MVM_TX_FIFO_MCAST : IWL_MVM_TX_FIFO_BE, .sta_id = msta->sta_id, .tid = 0, .aggregate = false, diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/tdls.c b/drivers/net/wireless/intel/iwlwifi/mvm/tdls.c index 859aa5a4e6b5068cf381f4d3f3c497babdff877d..9df21a8d1fc1f48cd513236bcbe5789c285dca33 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/tdls.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/tdls.c @@ -7,7 +7,7 @@ * * Copyright(c) 2014 Intel Mobile Communications GmbH * Copyright(c) 2017 Intel Deutschland GmbH - * Copyright(C) 2018 Intel Corporation + * Copyright(C) 2018 - 2019 Intel Corporation * * This program is free software; you can redistribute it and/or modify * it under the terms of version 2 of the GNU General Public License as @@ -29,7 +29,7 @@ * * Copyright(c) 2014 Intel Mobile Communications GmbH * Copyright(c) 2017 Intel Deutschland GmbH - * Copyright(C) 2018 Intel Corporation + * Copyright(C) 2018 - 2019 Intel Corporation * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -252,8 +252,7 @@ static void iwl_mvm_tdls_update_cs_state(struct iwl_mvm *mvm, /* we only send requests to our switching peer - update sent time */ if (state == IWL_MVM_TDLS_SW_REQ_SENT) - mvm->tdls_cs.peer.sent_timestamp = - iwl_read_prph(mvm->trans, DEVICE_SYSTEM_TIME_REG); + mvm->tdls_cs.peer.sent_timestamp = iwl_mvm_get_systime(mvm); if (state == IWL_MVM_TDLS_SW_IDLE) mvm->tdls_cs.cur_sta_id = IWL_MVM_INVALID_STA; diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/time-event.c b/drivers/net/wireless/intel/iwlwifi/mvm/time-event.c index 9693fa4cdc39b1b3038aeb699ada80df0ec69fe2..4d34e5ab1bffb9191f709f0f26f498a0301584d7 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/time-event.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/time-event.c @@ -8,7 +8,7 @@ * Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH * Copyright(c) 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * * This program is free software; you can redistribute it and/or modify * it under the terms of version 2 of the GNU General Public License as @@ -31,7 +31,7 @@ * Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved. * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH * Copyright(c) 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -234,6 +234,7 @@ iwl_mvm_te_handle_notify_csa(struct iwl_mvm *mvm, break; } iwl_mvm_csa_client_absent(mvm, te_data->vif); + cancel_delayed_work(&mvmvif->csa_work); ieee80211_chswitch_done(te_data->vif, true); break; default: diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/utils.c b/drivers/net/wireless/intel/iwlwifi/mvm/utils.c index 4649327abb456bff84b3e455a56b13eefe17090d..b9914efc55c4b2f027db3954877645c4b7c27fb0 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/utils.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/utils.c @@ -1418,6 +1418,16 @@ void iwl_mvm_tcm_rm_vif(struct iwl_mvm *mvm, struct ieee80211_vif *vif) cancel_delayed_work_sync(&mvmvif->uapsd_nonagg_detected_wk); } +u32 iwl_mvm_get_systime(struct iwl_mvm *mvm) +{ + u32 reg_addr = DEVICE_SYSTEM_TIME_REG; + + if (mvm->trans->cfg->device_family >= IWL_DEVICE_FAMILY_22000 && + mvm->trans->cfg->gp2_reg_addr) + reg_addr = mvm->trans->cfg->gp2_reg_addr; + + return iwl_read_prph(mvm->trans, reg_addr); +} void iwl_mvm_get_sync_time(struct iwl_mvm *mvm, u32 *gp2, u64 *boottime) { @@ -1432,7 +1442,7 @@ void iwl_mvm_get_sync_time(struct iwl_mvm *mvm, u32 *gp2, u64 *boottime) iwl_mvm_power_update_device(mvm); } - *gp2 = iwl_read_prph(mvm->trans, DEVICE_SYSTEM_TIME_REG); + *gp2 = iwl_mvm_get_systime(mvm); *boottime = ktime_get_boot_ns(); if (!ps_disabled) { diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/ctxt-info-gen3.c b/drivers/net/wireless/intel/iwlwifi/pcie/ctxt-info-gen3.c index 1e36459948db02916ddc4abb796dd779a191b9f9..f496d1bcb64305ebde739ce91381a827ba579242 100644 --- a/drivers/net/wireless/intel/iwlwifi/pcie/ctxt-info-gen3.c +++ b/drivers/net/wireless/intel/iwlwifi/pcie/ctxt-info-gen3.c @@ -66,7 +66,8 @@ int iwl_pcie_ctxt_info_gen3_init(struct iwl_trans *trans, void *iml_img; u32 control_flags = 0; int ret; - int cmdq_size = max_t(u32, TFD_CMD_SLOTS, trans->cfg->min_txq_size); + int cmdq_size = max_t(u32, IWL_CMD_QUEUE_SIZE, + trans->cfg->min_txq_size); /* Allocate prph scratch */ prph_scratch = dma_alloc_coherent(trans->dev, sizeof(*prph_scratch), diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/ctxt-info.c b/drivers/net/wireless/intel/iwlwifi/pcie/ctxt-info.c index 9274e317cc777b041ef8553445e15fcc6165c6b9..8969b47bacf2cf08e628dc0077a15170b8fe99bc 100644 --- a/drivers/net/wireless/intel/iwlwifi/pcie/ctxt-info.c +++ b/drivers/net/wireless/intel/iwlwifi/pcie/ctxt-info.c @@ -6,7 +6,7 @@ * GPL LICENSE SUMMARY * * Copyright(c) 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * * This program is free software; you can redistribute it and/or modify * it under the terms of version 2 of the GNU General Public License as @@ -20,7 +20,7 @@ * BSD LICENSE * * Copyright(c) 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -210,7 +210,7 @@ int iwl_pcie_ctxt_info_init(struct iwl_trans *trans, ctxt_info->hcmd_cfg.cmd_queue_addr = cpu_to_le64(trans_pcie->txq[trans_pcie->cmd_queue]->dma_addr); ctxt_info->hcmd_cfg.cmd_queue_size = - TFD_QUEUE_CB_SIZE(TFD_CMD_SLOTS); + TFD_QUEUE_CB_SIZE(IWL_CMD_QUEUE_SIZE); /* allocate ucode sections in dram and set addresses */ ret = iwl_pcie_init_fw_sec(trans, fw, &ctxt_info->dram); diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/drv.c b/drivers/net/wireless/intel/iwlwifi/pcie/drv.c index 9f1af8da9dc181eb1dcc48b2f0fb1d1b7ffa9836..cd035061cdd5559a2b1faf1f227e936623433f3f 100644 --- a/drivers/net/wireless/intel/iwlwifi/pcie/drv.c +++ b/drivers/net/wireless/intel/iwlwifi/pcie/drv.c @@ -928,11 +928,6 @@ static const struct pci_device_id iwl_hw_card_ids[] = { {IWL_PCI_DEVICE(0x34F0, 0x1651, killer1650s_2ax_cfg_qu_b0_hr_b0)}, {IWL_PCI_DEVICE(0x34F0, 0x1652, killer1650i_2ax_cfg_qu_b0_hr_b0)}, {IWL_PCI_DEVICE(0x34F0, 0x4070, iwl_ax101_cfg_qu_hr)}, - {IWL_PCI_DEVICE(0x40C0, 0x0000, iwl22560_2ax_cfg_su_cdb)}, - {IWL_PCI_DEVICE(0x40C0, 0x0010, iwl22560_2ax_cfg_su_cdb)}, - {IWL_PCI_DEVICE(0x40c0, 0x0090, iwl22560_2ax_cfg_su_cdb)}, - {IWL_PCI_DEVICE(0x40C0, 0x0310, iwl22560_2ax_cfg_su_cdb)}, - {IWL_PCI_DEVICE(0x40C0, 0x0A10, iwl22560_2ax_cfg_su_cdb)}, {IWL_PCI_DEVICE(0x43F0, 0x0040, iwl_ax101_cfg_qu_hr)}, {IWL_PCI_DEVICE(0x43F0, 0x0070, iwl_ax101_cfg_qu_hr)}, {IWL_PCI_DEVICE(0x43F0, 0x0074, iwl_ax101_cfg_qu_hr)}, @@ -963,9 +958,6 @@ static const struct pci_device_id iwl_hw_card_ids[] = { {IWL_PCI_DEVICE(0x2723, 0x4080, iwl_ax200_cfg_cc)}, {IWL_PCI_DEVICE(0x2723, 0x4088, iwl_ax200_cfg_cc)}, - {IWL_PCI_DEVICE(0x1a56, 0x1653, killer1650w_2ax_cfg)}, - {IWL_PCI_DEVICE(0x1a56, 0x1654, killer1650x_2ax_cfg)}, - {IWL_PCI_DEVICE(0x2725, 0x0090, iwlax210_2ax_cfg_so_hr_a0)}, {IWL_PCI_DEVICE(0x7A70, 0x0090, iwlax210_2ax_cfg_so_hr_a0)}, {IWL_PCI_DEVICE(0x7A70, 0x0310, iwlax210_2ax_cfg_so_hr_a0)}, @@ -1047,9 +1039,7 @@ static int iwl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent) } /* register transport layer debugfs here */ - ret = iwl_trans_pcie_dbgfs_register(iwl_trans); - if (ret) - goto out_free_drv; + iwl_trans_pcie_dbgfs_register(iwl_trans); /* if RTPM is in use, enable it in our device */ if (iwl_trans->runtime_pm_mode != IWL_PLAT_PM_MODE_DISABLED) { @@ -1078,8 +1068,6 @@ static int iwl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent) return 0; -out_free_drv: - iwl_drv_stop(iwl_trans->drv); out_free_trans: iwl_trans_pcie_free(iwl_trans); return ret; diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/internal.h b/drivers/net/wireless/intel/iwlwifi/pcie/internal.h index 59213164f35e3814cd0d7618cf8f6f54fd873f5b..b513037dc0661e9eaae98f162841fbbe2e86e779 100644 --- a/drivers/net/wireless/intel/iwlwifi/pcie/internal.h +++ b/drivers/net/wireless/intel/iwlwifi/pcie/internal.h @@ -106,7 +106,6 @@ struct iwl_host_cmd; * @page: driver's pointer to the rxb page * @invalid: rxb is in driver ownership - not owned by HW * @vid: index of this rxb in the global table - * @size: size used from the buffer */ struct iwl_rx_mem_buffer { dma_addr_t page_dma; @@ -114,7 +113,6 @@ struct iwl_rx_mem_buffer { u16 vid; bool invalid; struct list_head list; - u32 size; }; /** @@ -135,46 +133,32 @@ struct isr_statistics { u32 unhandled; }; -#define IWL_RX_TD_TYPE_MSK 0xff000000 -#define IWL_RX_TD_SIZE_MSK 0x00ffffff -#define IWL_RX_TD_SIZE_2K BIT(11) -#define IWL_RX_TD_TYPE 0 - /** * struct iwl_rx_transfer_desc - transfer descriptor - * @type_n_size: buffer type (bit 0: external buff valid, - * bit 1: optional footer valid, bit 2-7: reserved) - * and buffer size * @addr: ptr to free buffer start address * @rbid: unique tag of the buffer * @reserved: reserved */ struct iwl_rx_transfer_desc { - __le32 type_n_size; - __le64 addr; __le16 rbid; - __le16 reserved; + __le16 reserved[3]; + __le64 addr; } __packed; -#define IWL_RX_CD_SIZE 0xffffff00 +#define IWL_RX_CD_FLAGS_FRAGMENTED BIT(0) /** * struct iwl_rx_completion_desc - completion descriptor - * @type: buffer type (bit 0: external buff valid, - * bit 1: optional footer valid, bit 2-7: reserved) - * @status: status of the completion * @reserved1: reserved * @rbid: unique tag of the received buffer - * @size: buffer size, masked by IWL_RX_CD_SIZE + * @flags: flags (0: fragmented, all others: reserved) * @reserved2: reserved */ struct iwl_rx_completion_desc { - u8 type; - u8 status; - __le16 reserved1; + __le32 reserved1; __le16 rbid; - __le32 size; - u8 reserved2[22]; + u8 flags; + u8 reserved2[25]; } __packed; /** @@ -306,10 +290,6 @@ struct iwl_cmd_meta { u32 tbs; }; - -#define TFD_TX_CMD_SLOTS 256 -#define TFD_CMD_SLOTS 32 - /* * The FH will write back to the first TB only, so we need to copy some data * into the buffer regardless of whether it should be mapped or not. @@ -556,7 +536,7 @@ struct iwl_trans_pcie { int ict_index; bool use_ict; bool is_down, opmode_down; - bool debug_rfkill; + s8 debug_rfkill; struct isr_statistics isr_stats; spinlock_t irq_lock; @@ -1002,7 +982,7 @@ static inline bool iwl_is_rfkill_set(struct iwl_trans *trans) lockdep_assert_held(&trans_pcie->mutex); - if (trans_pcie->debug_rfkill) + if (trans_pcie->debug_rfkill == 1) return true; return !(iwl_read32(trans, CSR_GP_CNTRL) & @@ -1046,12 +1026,9 @@ void iwl_trans_pcie_dump_regs(struct iwl_trans *trans); void iwl_trans_pcie_sync_nmi(struct iwl_trans *trans); #ifdef CONFIG_IWLWIFI_DEBUGFS -int iwl_trans_pcie_dbgfs_register(struct iwl_trans *trans); +void iwl_trans_pcie_dbgfs_register(struct iwl_trans *trans); #else -static inline int iwl_trans_pcie_dbgfs_register(struct iwl_trans *trans) -{ - return 0; -} +static inline void iwl_trans_pcie_dbgfs_register(struct iwl_trans *trans) { } #endif int iwl_pci_fw_exit_d0i3(struct iwl_trans *trans); diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/rx.c b/drivers/net/wireless/intel/iwlwifi/pcie/rx.c index 8d4f0628622bb7f948e522210f9aa02309f0c56e..31b3591f71d1151b6d2e38e46c52b66a84abb043 100644 --- a/drivers/net/wireless/intel/iwlwifi/pcie/rx.c +++ b/drivers/net/wireless/intel/iwlwifi/pcie/rx.c @@ -282,9 +282,8 @@ static void iwl_pcie_restock_bd(struct iwl_trans *trans, if (trans->cfg->device_family >= IWL_DEVICE_FAMILY_22560) { struct iwl_rx_transfer_desc *bd = rxq->bd; - bd[rxq->write].type_n_size = - cpu_to_le32((IWL_RX_TD_TYPE & IWL_RX_TD_TYPE_MSK) | - ((IWL_RX_TD_SIZE_2K >> 8) & IWL_RX_TD_SIZE_MSK)); + BUILD_BUG_ON(sizeof(*bd) != 2 * sizeof(u64)); + bd[rxq->write].addr = cpu_to_le64(rxb->page_dma); bd[rxq->write].rbid = cpu_to_le16(rxb->vid); } else { @@ -435,7 +434,7 @@ static struct page *iwl_pcie_rx_alloc_page(struct iwl_trans *trans, /* * Issue an error if we don't have enough pre-allocated * buffers. -` */ + */ if (!(gfp_mask & __GFP_NOWARN) && net_ratelimit()) IWL_CRIT(trans, "Failed to alloc_pages\n"); @@ -1265,9 +1264,6 @@ static void iwl_pcie_rx_handle_rb(struct iwl_trans *trans, .truesize = max_len, }; - if (trans->cfg->device_family >= IWL_DEVICE_FAMILY_22560) - rxcb.status = rxq->cd[i].status; - pkt = rxb_addr(&rxcb); if (pkt->len_n_flags == cpu_to_le32(FH_RSCSR_FRAME_INVALID)) { @@ -1394,6 +1390,8 @@ static struct iwl_rx_mem_buffer *iwl_pcie_get_rxb(struct iwl_trans *trans, struct iwl_rx_mem_buffer *rxb; u16 vid; + BUILD_BUG_ON(sizeof(struct iwl_rx_completion_desc) != 32); + if (!trans->cfg->mq_rx_supported) { rxb = rxq->queue[i]; rxq->queue[i] = NULL; @@ -1415,9 +1413,6 @@ static struct iwl_rx_mem_buffer *iwl_pcie_get_rxb(struct iwl_trans *trans, IWL_DEBUG_RX(trans, "Got virtual RB ID %u\n", (u32)rxb->vid); - if (trans->cfg->device_family >= IWL_DEVICE_FAMILY_22560) - rxb->size = le32_to_cpu(rxq->cd[i].size) & IWL_RX_CD_SIZE; - rxb->invalid = true; return rxb; @@ -1434,10 +1429,15 @@ out_err: static void iwl_pcie_rx_handle(struct iwl_trans *trans, int queue) { struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans); - struct iwl_rxq *rxq = &trans_pcie->rxq[queue]; + struct iwl_rxq *rxq; u32 r, i, count = 0; bool emergency = false; + if (WARN_ON_ONCE(!trans_pcie->rxq || !trans_pcie->rxq[queue].bd)) + return; + + rxq = &trans_pcie->rxq[queue]; + restart: spin_lock(&rxq->lock); /* uCode's read index (stored in shared DRAM) indicates the last Rx @@ -2212,6 +2212,7 @@ irqreturn_t iwl_pcie_irq_msix_handler(int irq, void *dev_id) "Hardware error detected. Restarting.\n"); isr_stats->hw++; + trans->hw_error = true; iwl_pcie_irq_handle_error(trans); } diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/trans-gen2.c b/drivers/net/wireless/intel/iwlwifi/pcie/trans-gen2.c index 9c203ca75de9069e9ce4a769d18b3af402ab7076..8507a7bdcfdd73d635d72cdcdf820a13a4f8297e 100644 --- a/drivers/net/wireless/intel/iwlwifi/pcie/trans-gen2.c +++ b/drivers/net/wireless/intel/iwlwifi/pcie/trans-gen2.c @@ -6,7 +6,7 @@ * GPL LICENSE SUMMARY * * Copyright(c) 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * * This program is free software; you can redistribute it and/or modify * it under the terms of version 2 of the GNU General Public License as @@ -20,7 +20,7 @@ * BSD LICENSE * * Copyright(c) 2017 Intel Deutschland GmbH - * Copyright(c) 2018 Intel Corporation + * Copyright(c) 2018 - 2019 Intel Corporation * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -234,7 +234,8 @@ void iwl_trans_pcie_gen2_stop_device(struct iwl_trans *trans, bool low_power) static int iwl_pcie_gen2_nic_init(struct iwl_trans *trans) { struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans); - int queue_size = max_t(u32, TFD_CMD_SLOTS, trans->cfg->min_txq_size); + int queue_size = max_t(u32, IWL_CMD_QUEUE_SIZE, + trans->cfg->min_txq_size); /* TODO: most of the logic can be removed in A0 - but not in Z0 */ spin_lock(&trans_pcie->irq_lock); diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/trans.c b/drivers/net/wireless/intel/iwlwifi/pcie/trans.c index 4f5eec7e44bd4fb997474520b687aba868307668..803fcbac415255d39e995a299e3ed031988e7436 100644 --- a/drivers/net/wireless/intel/iwlwifi/pcie/trans.c +++ b/drivers/net/wireless/intel/iwlwifi/pcie/trans.c @@ -896,6 +896,8 @@ void iwl_pcie_apply_destination(struct iwl_trans *trans) if (!trans->num_blocks) return; + IWL_DEBUG_FW(trans, + "WRT: applying DRAM buffer[0] destination\n"); iwl_write_umac_prph(trans, MON_BUFF_BASE_ADDR_VER2, trans->fw_mon[0].physical >> MON_BUFF_SHIFT_VER2); @@ -2441,9 +2443,8 @@ void iwl_pcie_dump_csr(struct iwl_trans *trans) #ifdef CONFIG_IWLWIFI_DEBUGFS /* create and remove of files */ #define DEBUGFS_ADD_FILE(name, parent, mode) do { \ - if (!debugfs_create_file(#name, mode, parent, trans, \ - &iwl_dbgfs_##name##_ops)) \ - goto err; \ + debugfs_create_file(#name, mode, parent, trans, \ + &iwl_dbgfs_##name##_ops); \ } while (0) /* file operation */ @@ -2686,16 +2687,17 @@ static ssize_t iwl_dbgfs_rfkill_write(struct file *file, { struct iwl_trans *trans = file->private_data; struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans); - bool old = trans_pcie->debug_rfkill; + bool new_value; int ret; - ret = kstrtobool_from_user(user_buf, count, &trans_pcie->debug_rfkill); + ret = kstrtobool_from_user(user_buf, count, &new_value); if (ret) return ret; - if (old == trans_pcie->debug_rfkill) + if (new_value == trans_pcie->debug_rfkill) return count; IWL_WARN(trans, "changing debug rfkill %d->%d\n", - old, trans_pcie->debug_rfkill); + trans_pcie->debug_rfkill, new_value); + trans_pcie->debug_rfkill = new_value; iwl_pcie_handle_rfkill_irq(trans); return count; @@ -2846,7 +2848,7 @@ static const struct file_operations iwl_dbgfs_monitor_data_ops = { }; /* Create the debugfs files and directories */ -int iwl_trans_pcie_dbgfs_register(struct iwl_trans *trans) +void iwl_trans_pcie_dbgfs_register(struct iwl_trans *trans) { struct dentry *dir = trans->dbgfs_dir; @@ -2857,11 +2859,6 @@ int iwl_trans_pcie_dbgfs_register(struct iwl_trans *trans) DEBUGFS_ADD_FILE(fh_reg, dir, 0400); DEBUGFS_ADD_FILE(rfkill, dir, 0600); DEBUGFS_ADD_FILE(monitor_data, dir, 0400); - return 0; - -err: - IWL_ERR(trans, "failed to create the trans debugfs entry\n"); - return -ENOMEM; } static void iwl_trans_pcie_debugfs_cleanup(struct iwl_trans *trans) @@ -3011,10 +3008,14 @@ static void iwl_trans_pcie_dump_pointers(struct iwl_trans *trans, struct iwl_fw_error_dump_fw_mon *fw_mon_data) { - u32 base, write_ptr, wrap_cnt; + u32 base, base_high, write_ptr, write_ptr_val, wrap_cnt; - /* If there was a dest TLV - use the values from there */ - if (trans->ini_valid) { + if (trans->cfg->device_family >= IWL_DEVICE_FAMILY_AX210) { + base = DBGC_CUR_DBGBUF_BASE_ADDR_LSB; + base_high = DBGC_CUR_DBGBUF_BASE_ADDR_MSB; + write_ptr = DBGC_CUR_DBGBUF_STATUS; + wrap_cnt = DBGC_DBGBUF_WRAP_AROUND; + } else if (trans->ini_valid) { base = iwl_umac_prph(trans, MON_BUFF_BASE_ADDR_VER2); write_ptr = iwl_umac_prph(trans, MON_BUFF_WRPTR_VER2); wrap_cnt = iwl_umac_prph(trans, MON_BUFF_CYCLE_CNT_VER2); @@ -3027,12 +3028,18 @@ iwl_trans_pcie_dump_pointers(struct iwl_trans *trans, write_ptr = MON_BUFF_WRPTR; wrap_cnt = MON_BUFF_CYCLE_CNT; } - fw_mon_data->fw_mon_wr_ptr = - cpu_to_le32(iwl_read_prph(trans, write_ptr)); + + write_ptr_val = iwl_read_prph(trans, write_ptr); fw_mon_data->fw_mon_cycle_cnt = cpu_to_le32(iwl_read_prph(trans, wrap_cnt)); fw_mon_data->fw_mon_base_ptr = cpu_to_le32(iwl_read_prph(trans, base)); + if (trans->cfg->device_family >= IWL_DEVICE_FAMILY_AX210) { + fw_mon_data->fw_mon_base_high_ptr = + cpu_to_le32(iwl_read_prph(trans, base_high)); + write_ptr_val &= DBGC_CUR_DBGBUF_STATUS_OFFSET_MSK; + } + fw_mon_data->fw_mon_wr_ptr = cpu_to_le32(write_ptr_val); } static u32 @@ -3043,9 +3050,10 @@ iwl_trans_pcie_dump_monitor(struct iwl_trans *trans, u32 len = 0; if ((trans->num_blocks && - trans->cfg->device_family == IWL_DEVICE_FAMILY_7000) || - (trans->dbg_dest_tlv && !trans->ini_valid) || - (trans->ini_valid && trans->num_blocks)) { + (trans->cfg->device_family == IWL_DEVICE_FAMILY_7000 || + trans->cfg->device_family >= IWL_DEVICE_FAMILY_AX210 || + trans->ini_valid)) || + (trans->dbg_dest_tlv && !trans->ini_valid)) { struct iwl_fw_error_dump_fw_mon *fw_mon_data; (*data)->type = cpu_to_le32(IWL_FW_ERROR_DUMP_FW_MONITOR); @@ -3164,8 +3172,10 @@ static struct iwl_trans_dump_data len = sizeof(*dump_data); /* host commands */ - len += sizeof(*data) + - cmdq->n_window * (sizeof(*txcmd) + TFD_MAX_PAYLOAD_SIZE); + if (dump_mask & BIT(IWL_FW_ERROR_DUMP_TXCMD)) + len += sizeof(*data) + + cmdq->n_window * (sizeof(*txcmd) + + TFD_MAX_PAYLOAD_SIZE); /* FW monitor */ if (dump_mask & BIT(IWL_FW_ERROR_DUMP_FW_MONITOR)) @@ -3411,7 +3421,7 @@ struct iwl_trans *iwl_trans_pcie_alloc(struct pci_dev *pdev, ret = -ENOMEM; goto out_no_pci; } - + trans_pcie->debug_rfkill = -1; if (!cfg->base_params->pcie_l1_allowed) { /* @@ -3539,6 +3549,9 @@ struct iwl_trans *iwl_trans_pcie_alloc(struct pci_dev *pdev, } else if (CSR_HW_RF_ID_TYPE_CHIP_ID(trans->hw_rf_id) == CSR_HW_RF_ID_TYPE_CHIP_ID(CSR_HW_RF_ID_TYPE_GF)) { trans->cfg = &iwlax210_2ax_cfg_so_gf_a0; + } else if (CSR_HW_RF_ID_TYPE_CHIP_ID(trans->hw_rf_id) == + CSR_HW_RF_ID_TYPE_CHIP_ID(CSR_HW_RF_ID_TYPE_GF4)) { + trans->cfg = &iwlax210_2ax_cfg_so_gf4_a0; } } else if (cfg == &iwl_ax101_cfg_qu_hr) { if (CSR_HW_RF_ID_TYPE_CHIP_ID(trans->hw_rf_id) == diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/tx.c b/drivers/net/wireless/intel/iwlwifi/pcie/tx.c index 7be73e2c4681cadc48ed5a838068d419196b413f..fa4245d0d4a8f848163c930126cd088f2f5c46b1 100644 --- a/drivers/net/wireless/intel/iwlwifi/pcie/tx.c +++ b/drivers/net/wireless/intel/iwlwifi/pcie/tx.c @@ -996,10 +996,11 @@ static int iwl_pcie_tx_alloc(struct iwl_trans *trans) bool cmd_queue = (txq_id == trans_pcie->cmd_queue); if (cmd_queue) - slots_num = max_t(u32, TFD_CMD_SLOTS, + slots_num = max_t(u32, IWL_CMD_QUEUE_SIZE, trans->cfg->min_txq_size); else - slots_num = TFD_TX_CMD_SLOTS; + slots_num = max_t(u32, IWL_DEFAULT_QUEUE_SIZE, + trans->cfg->min_256_ba_txq_size); trans_pcie->txq[txq_id] = &trans_pcie->txq_memory[txq_id]; ret = iwl_pcie_txq_alloc(trans, trans_pcie->txq[txq_id], slots_num, cmd_queue); @@ -1049,10 +1050,11 @@ int iwl_pcie_tx_init(struct iwl_trans *trans) bool cmd_queue = (txq_id == trans_pcie->cmd_queue); if (cmd_queue) - slots_num = max_t(u32, TFD_CMD_SLOTS, + slots_num = max_t(u32, IWL_CMD_QUEUE_SIZE, trans->cfg->min_txq_size); else - slots_num = TFD_TX_CMD_SLOTS; + slots_num = max_t(u32, IWL_DEFAULT_QUEUE_SIZE, + trans->cfg->min_256_ba_txq_size); ret = iwl_pcie_txq_init(trans, trans_pcie->txq[txq_id], slots_num, cmd_queue); if (ret) { diff --git a/drivers/net/wireless/intersil/p54/p54pci.c b/drivers/net/wireless/intersil/p54/p54pci.c index 27a49068d32d0a71e90b69c3a3b0289be956ff94..57ad56435dda53ed852f9a4935430fa9cdea7e52 100644 --- a/drivers/net/wireless/intersil/p54/p54pci.c +++ b/drivers/net/wireless/intersil/p54/p54pci.c @@ -554,7 +554,7 @@ static int p54p_probe(struct pci_dev *pdev, err = pci_enable_device(pdev); if (err) { dev_err(&pdev->dev, "Cannot enable new PCI device\n"); - return err; + goto err_put; } mem_addr = pci_resource_start(pdev, 0); @@ -639,6 +639,7 @@ static int p54p_probe(struct pci_dev *pdev, pci_release_regions(pdev); err_disable_dev: pci_disable_device(pdev); +err_put: pci_dev_put(pdev); return err; } diff --git a/drivers/net/wireless/mac80211_hwsim.c b/drivers/net/wireless/mac80211_hwsim.c index c71adb1f1f4170fa4638abd80011360061b72192..60ca13e0f15b7b02aca1e425f8a971555ac172e3 100644 --- a/drivers/net/wireless/mac80211_hwsim.c +++ b/drivers/net/wireless/mac80211_hwsim.c @@ -409,8 +409,8 @@ static int mac80211_hwsim_vendor_cmd_test(struct wiphy *wiphy, int err; u32 val; - err = nla_parse(tb, QCA_WLAN_VENDOR_ATTR_MAX, data, data_len, - hwsim_vendor_test_policy, NULL); + err = nla_parse_deprecated(tb, QCA_WLAN_VENDOR_ATTR_MAX, data, + data_len, hwsim_vendor_test_policy, NULL); if (err) return err; if (!tb[QCA_WLAN_VENDOR_ATTR_TEST]) @@ -1932,8 +1932,8 @@ static int mac80211_hwsim_testmode_cmd(struct ieee80211_hw *hw, struct sk_buff *skb; int err, ps; - err = nla_parse(tb, HWSIM_TM_ATTR_MAX, data, len, - hwsim_testmode_policy, NULL); + err = nla_parse_deprecated(tb, HWSIM_TM_ATTR_MAX, data, len, + hwsim_testmode_policy, NULL); if (err) return err; @@ -2806,6 +2806,12 @@ static int mac80211_hwsim_new_radio(struct genl_info *info, ieee80211_hw_set(hw, SIGNAL_DBM); ieee80211_hw_set(hw, SUPPORTS_PS); ieee80211_hw_set(hw, TDLS_WIDER_BW); + + /* We only have SW crypto and only implement the A-MPDU API + * (but don't really build A-MPDUs) so can have extended key + * support + */ + ieee80211_hw_set(hw, EXT_KEY_ID_NATIVE); if (rctbl) ieee80211_hw_set(hw, SUPPORTS_RC_TABLE); ieee80211_hw_set(hw, SUPPORTS_MULTI_BSSID); @@ -3627,35 +3633,35 @@ done: static const struct genl_ops hwsim_ops[] = { { .cmd = HWSIM_CMD_REGISTER, - .policy = hwsim_genl_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = hwsim_register_received_nl, .flags = GENL_UNS_ADMIN_PERM, }, { .cmd = HWSIM_CMD_FRAME, - .policy = hwsim_genl_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = hwsim_cloned_frame_received_nl, }, { .cmd = HWSIM_CMD_TX_INFO_FRAME, - .policy = hwsim_genl_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = hwsim_tx_info_frame_received_nl, }, { .cmd = HWSIM_CMD_NEW_RADIO, - .policy = hwsim_genl_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = hwsim_new_radio_nl, .flags = GENL_UNS_ADMIN_PERM, }, { .cmd = HWSIM_CMD_DEL_RADIO, - .policy = hwsim_genl_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = hwsim_del_radio_nl, .flags = GENL_UNS_ADMIN_PERM, }, { .cmd = HWSIM_CMD_GET_RADIO, - .policy = hwsim_genl_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = hwsim_get_radio_nl, .dumpit = hwsim_dump_radio_nl, }, @@ -3665,6 +3671,7 @@ static struct genl_family hwsim_genl_family __ro_after_init = { .name = "MAC80211_HWSIM", .version = 1, .maxattr = HWSIM_ATTR_MAX, + .policy = hwsim_genl_policy, .netnsok = true, .module = THIS_MODULE, .ops = hwsim_ops, @@ -3901,6 +3908,8 @@ static int __init init_mac80211_hwsim(void) param.p2p_device = support_p2p_device; param.use_chanctx = channels > 1; param.iftypes = HWSIM_IFTYPE_SUPPORT_MASK; + if (param.p2p_device) + param.iftypes |= BIT(NL80211_IFTYPE_P2P_DEVICE); err = mac80211_hwsim_new_radio(NULL, ¶m); if (err < 0) diff --git a/drivers/net/wireless/marvell/mwifiex/Kconfig b/drivers/net/wireless/marvell/mwifiex/Kconfig index 524fd565cb2a200feefd78700d61e385962bfddf..572d187a99f4e4b012c0539d5619997531349c9e 100644 --- a/drivers/net/wireless/marvell/mwifiex/Kconfig +++ b/drivers/net/wireless/marvell/mwifiex/Kconfig @@ -9,13 +9,13 @@ config MWIFIEX mwifiex. config MWIFIEX_SDIO - tristate "Marvell WiFi-Ex Driver for SD8786/SD8787/SD8797/SD8887/SD8897/SD8977/SD8997" + tristate "Marvell WiFi-Ex Driver for SD8786/SD8787/SD8797/SD8887/SD8897/SD8977/SD8987/SD8997" depends on MWIFIEX && MMC select FW_LOADER select WANT_DEV_COREDUMP ---help--- This adds support for wireless adapters based on Marvell - 8786/8787/8797/8887/8897/8997 chipsets with SDIO interface. + 8786/8787/8797/8887/8897/8977/8987/8997 chipsets with SDIO interface. If you choose to build it as a module, it will be called mwifiex_sdio. diff --git a/drivers/net/wireless/marvell/mwifiex/cfg80211.c b/drivers/net/wireless/marvell/mwifiex/cfg80211.c index c46f0a54a0c764ee563001aea96c30f8481fecb4..e11a4bb67172242d9017bfd0f7959e93de8961f7 100644 --- a/drivers/net/wireless/marvell/mwifiex/cfg80211.c +++ b/drivers/net/wireless/marvell/mwifiex/cfg80211.c @@ -4059,8 +4059,8 @@ static int mwifiex_tm_cmd(struct wiphy *wiphy, struct wireless_dev *wdev, if (!priv) return -EINVAL; - err = nla_parse(tb, MWIFIEX_TM_ATTR_MAX, data, len, mwifiex_tm_policy, - NULL); + err = nla_parse_deprecated(tb, MWIFIEX_TM_ATTR_MAX, data, len, + mwifiex_tm_policy, NULL); if (err) return err; @@ -4082,16 +4082,20 @@ static int mwifiex_tm_cmd(struct wiphy *wiphy, struct wireless_dev *wdev, if (mwifiex_send_cmd(priv, 0, 0, 0, hostcmd, true)) { dev_err(priv->adapter->dev, "Failed to process hostcmd\n"); + kfree(hostcmd); return -EFAULT; } /* process hostcmd response*/ skb = cfg80211_testmode_alloc_reply_skb(wiphy, hostcmd->len); - if (!skb) + if (!skb) { + kfree(hostcmd); return -ENOMEM; + } err = nla_put(skb, MWIFIEX_TM_ATTR_DATA, hostcmd->len, hostcmd->cmd); if (err) { + kfree(hostcmd); kfree_skb(skb); return -EMSGSIZE; } diff --git a/drivers/net/wireless/marvell/mwifiex/cfp.c b/drivers/net/wireless/marvell/mwifiex/cfp.c index bfe84e55df7762349d65657cb5148457500d43cc..f1522fb1c1e878ae385a04648bae9737f698e1f6 100644 --- a/drivers/net/wireless/marvell/mwifiex/cfp.c +++ b/drivers/net/wireless/marvell/mwifiex/cfp.c @@ -531,5 +531,8 @@ u8 mwifiex_adjust_data_rate(struct mwifiex_private *priv, rate_index = (rx_rate > MWIFIEX_RATE_INDEX_OFDM0) ? rx_rate - 1 : rx_rate; + if (rate_index >= MWIFIEX_MAX_AC_RX_RATES) + rate_index = MWIFIEX_MAX_AC_RX_RATES - 1; + return rate_index; } diff --git a/drivers/net/wireless/marvell/mwifiex/cmdevt.c b/drivers/net/wireless/marvell/mwifiex/cmdevt.c index 60db2b969e20d7d8f5b7119ed6636ccda132cdfd..8c35441fd9b7dbc2f3d4a1f46753323c8a29bce4 100644 --- a/drivers/net/wireless/marvell/mwifiex/cmdevt.c +++ b/drivers/net/wireless/marvell/mwifiex/cmdevt.c @@ -341,6 +341,12 @@ static int mwifiex_dnld_sleep_confirm_cmd(struct mwifiex_adapter *adapter) sleep_cfm_tmp = dev_alloc_skb(sizeof(struct mwifiex_opt_sleep_confirm) + MWIFIEX_TYPE_LEN); + if (!sleep_cfm_tmp) { + mwifiex_dbg(adapter, ERROR, + "SLEEP_CFM: dev_alloc_skb failed\n"); + return -ENOMEM; + } + skb_put(sleep_cfm_tmp, sizeof(struct mwifiex_opt_sleep_confirm) + MWIFIEX_TYPE_LEN); put_unaligned_le32(MWIFIEX_USB_TYPE_CMD, sleep_cfm_tmp->data); diff --git a/drivers/net/wireless/marvell/mwifiex/main.c b/drivers/net/wireless/marvell/mwifiex/main.c index 20cee5c397fb6f4d72dc6378401ee9bcf2af6708..f6da8edab7f1c109755ab1955a3c867fb40cac14 100644 --- a/drivers/net/wireless/marvell/mwifiex/main.c +++ b/drivers/net/wireless/marvell/mwifiex/main.c @@ -1282,8 +1282,7 @@ static struct net_device_stats *mwifiex_get_stats(struct net_device *dev) static u16 mwifiex_netdev_select_wmm_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { skb->priority = cfg80211_classify8021d(skb, NULL); return mwifiex_1d_to_wmm_queue[skb->priority]; diff --git a/drivers/net/wireless/marvell/mwifiex/sdio.c b/drivers/net/wireless/marvell/mwifiex/sdio.c index d5a70340a9457fda1f80efb38cec2fefdf0874f9..24c041dad9f633a5986d9c0557b2b24b04617579 100644 --- a/drivers/net/wireless/marvell/mwifiex/sdio.c +++ b/drivers/net/wireless/marvell/mwifiex/sdio.c @@ -491,6 +491,8 @@ static void mwifiex_sdio_coredump(struct device *dev) #define SDIO_DEVICE_ID_MARVELL_8801 (0x9139) /* Device ID for SD8977 */ #define SDIO_DEVICE_ID_MARVELL_8977 (0x9145) +/* Device ID for SD8987 */ +#define SDIO_DEVICE_ID_MARVELL_8987 (0x9149) /* Device ID for SD8997 */ #define SDIO_DEVICE_ID_MARVELL_8997 (0x9141) @@ -511,6 +513,8 @@ static const struct sdio_device_id mwifiex_ids[] = { .driver_data = (unsigned long)&mwifiex_sdio_sd8801}, {SDIO_DEVICE(SDIO_VENDOR_ID_MARVELL, SDIO_DEVICE_ID_MARVELL_8977), .driver_data = (unsigned long)&mwifiex_sdio_sd8977}, + {SDIO_DEVICE(SDIO_VENDOR_ID_MARVELL, SDIO_DEVICE_ID_MARVELL_8987), + .driver_data = (unsigned long)&mwifiex_sdio_sd8987}, {SDIO_DEVICE(SDIO_VENDOR_ID_MARVELL, SDIO_DEVICE_ID_MARVELL_8997), .driver_data = (unsigned long)&mwifiex_sdio_sd8997}, {}, @@ -2731,4 +2735,5 @@ MODULE_FIRMWARE(SD8797_DEFAULT_FW_NAME); MODULE_FIRMWARE(SD8897_DEFAULT_FW_NAME); MODULE_FIRMWARE(SD8887_DEFAULT_FW_NAME); MODULE_FIRMWARE(SD8977_DEFAULT_FW_NAME); +MODULE_FIRMWARE(SD8987_DEFAULT_FW_NAME); MODULE_FIRMWARE(SD8997_DEFAULT_FW_NAME); diff --git a/drivers/net/wireless/marvell/mwifiex/sdio.h b/drivers/net/wireless/marvell/mwifiex/sdio.h index 912de2cde8d9219d672c403e309c1619e5ab43d5..f672bdf52cc17bdebb759472e806583720bc5a1d 100644 --- a/drivers/net/wireless/marvell/mwifiex/sdio.h +++ b/drivers/net/wireless/marvell/mwifiex/sdio.h @@ -37,6 +37,7 @@ #define SD8887_DEFAULT_FW_NAME "mrvl/sd8887_uapsta.bin" #define SD8801_DEFAULT_FW_NAME "mrvl/sd8801_uapsta.bin" #define SD8977_DEFAULT_FW_NAME "mrvl/sd8977_uapsta.bin" +#define SD8987_DEFAULT_FW_NAME "mrvl/sd8987_uapsta.bin" #define SD8997_DEFAULT_FW_NAME "mrvl/sd8997_uapsta.bin" #define BLOCK_MODE 1 @@ -526,6 +527,58 @@ static const struct mwifiex_sdio_card_reg mwifiex_reg_sd8887 = { 0x68, 0x69, 0x6a}, }; +static const struct mwifiex_sdio_card_reg mwifiex_reg_sd8987 = { + .start_rd_port = 0, + .start_wr_port = 0, + .base_0_reg = 0xF8, + .base_1_reg = 0xF9, + .poll_reg = 0x5C, + .host_int_enable = UP_LD_HOST_INT_MASK | DN_LD_HOST_INT_MASK | + CMD_PORT_UPLD_INT_MASK | CMD_PORT_DNLD_INT_MASK, + .host_int_rsr_reg = 0x4, + .host_int_status_reg = 0x0C, + .host_int_mask_reg = 0x08, + .status_reg_0 = 0xE8, + .status_reg_1 = 0xE9, + .sdio_int_mask = 0xff, + .data_port_mask = 0xffffffff, + .io_port_0_reg = 0xE4, + .io_port_1_reg = 0xE5, + .io_port_2_reg = 0xE6, + .max_mp_regs = 196, + .rd_bitmap_l = 0x10, + .rd_bitmap_u = 0x11, + .rd_bitmap_1l = 0x12, + .rd_bitmap_1u = 0x13, + .wr_bitmap_l = 0x14, + .wr_bitmap_u = 0x15, + .wr_bitmap_1l = 0x16, + .wr_bitmap_1u = 0x17, + .rd_len_p0_l = 0x18, + .rd_len_p0_u = 0x19, + .card_misc_cfg_reg = 0xd8, + .card_cfg_2_1_reg = 0xd9, + .cmd_rd_len_0 = 0xc0, + .cmd_rd_len_1 = 0xc1, + .cmd_rd_len_2 = 0xc2, + .cmd_rd_len_3 = 0xc3, + .cmd_cfg_0 = 0xc4, + .cmd_cfg_1 = 0xc5, + .cmd_cfg_2 = 0xc6, + .cmd_cfg_3 = 0xc7, + .fw_dump_host_ready = 0xcc, + .fw_dump_ctrl = 0xf9, + .fw_dump_start = 0xf1, + .fw_dump_end = 0xf8, + .func1_dump_reg_start = 0x10, + .func1_dump_reg_end = 0x17, + .func1_scratch_reg = 0xE8, + .func1_spec_reg_num = 13, + .func1_spec_reg_table = {0x08, 0x58, 0x5C, 0x5D, 0x60, + 0x61, 0x62, 0x64, 0x65, 0x66, + 0x68, 0x69, 0x6a}, +}; + static const struct mwifiex_sdio_device mwifiex_sdio_sd8786 = { .firmware = SD8786_DEFAULT_FW_NAME, .reg = &mwifiex_reg_sd87xx, @@ -633,6 +686,22 @@ static const struct mwifiex_sdio_device mwifiex_sdio_sd8887 = { .can_ext_scan = true, }; +static const struct mwifiex_sdio_device mwifiex_sdio_sd8987 = { + .firmware = SD8987_DEFAULT_FW_NAME, + .reg = &mwifiex_reg_sd8987, + .max_ports = 32, + .mp_agg_pkt_limit = 16, + .tx_buf_size = MWIFIEX_TX_DATA_BUF_SIZE_2K, + .mp_tx_agg_buf_size = MWIFIEX_MP_AGGR_BUF_SIZE_MAX, + .mp_rx_agg_buf_size = MWIFIEX_MP_AGGR_BUF_SIZE_MAX, + .supports_sdio_new_mode = true, + .has_control_mask = false, + .can_dump_fw = true, + .fw_dump_enh = true, + .can_auto_tdls = true, + .can_ext_scan = true, +}; + static const struct mwifiex_sdio_device mwifiex_sdio_sd8801 = { .firmware = SD8801_DEFAULT_FW_NAME, .reg = &mwifiex_reg_sd87xx, diff --git a/drivers/net/wireless/marvell/mwifiex/sta_cmdresp.c b/drivers/net/wireless/marvell/mwifiex/sta_cmdresp.c index 69e3b624adbb9432fe49de863de2d0e95418c739..24b33e20e7a9abb15ff93d11123e1f5846faf915 100644 --- a/drivers/net/wireless/marvell/mwifiex/sta_cmdresp.c +++ b/drivers/net/wireless/marvell/mwifiex/sta_cmdresp.c @@ -1025,17 +1025,14 @@ mwifiex_create_custom_regdomain(struct mwifiex_private *priv, struct ieee80211_regdomain *regd; struct ieee80211_reg_rule *rule; bool new_rule; - int regd_size, idx, freq, prev_freq = 0; + int idx, freq, prev_freq = 0; u32 bw, prev_bw = 0; u8 chflags, prev_chflags = 0, valid_rules = 0; if (WARN_ON_ONCE(num_chan > NL80211_MAX_SUPP_REG_RULES)) return ERR_PTR(-EINVAL); - regd_size = sizeof(struct ieee80211_regdomain) + - num_chan * sizeof(struct ieee80211_reg_rule); - - regd = kzalloc(regd_size, GFP_KERNEL); + regd = kzalloc(struct_size(regd, reg_rules, num_chan), GFP_KERNEL); if (!regd) return ERR_PTR(-ENOMEM); diff --git a/drivers/net/wireless/marvell/mwifiex/sta_event.c b/drivers/net/wireless/marvell/mwifiex/sta_event.c index a327fc5b36e3e2d7971e3409f99b9b2c205a073c..8b3123cb84c8188b67602ced820cf99db1e064ac 100644 --- a/drivers/net/wireless/marvell/mwifiex/sta_event.c +++ b/drivers/net/wireless/marvell/mwifiex/sta_event.c @@ -27,9 +27,9 @@ #define MWIFIEX_IBSS_CONNECT_EVT_FIX_SIZE 12 -static int mwifiex_check_ibss_peer_capabilties(struct mwifiex_private *priv, - struct mwifiex_sta_node *sta_ptr, - struct sk_buff *event) +static int mwifiex_check_ibss_peer_capabilities(struct mwifiex_private *priv, + struct mwifiex_sta_node *sta_ptr, + struct sk_buff *event) { int evt_len, ele_len; u8 *curr; @@ -42,7 +42,7 @@ static int mwifiex_check_ibss_peer_capabilties(struct mwifiex_private *priv, evt_len = event->len; curr = event->data; - mwifiex_dbg_dump(priv->adapter, EVT_D, "ibss peer capabilties:", + mwifiex_dbg_dump(priv->adapter, EVT_D, "ibss peer capabilities:", event->data, event->len); skb_push(event, MWIFIEX_IBSS_CONNECT_EVT_FIX_SIZE); @@ -937,8 +937,8 @@ int mwifiex_process_sta_event(struct mwifiex_private *priv) ibss_sta_addr); sta_ptr = mwifiex_add_sta_entry(priv, ibss_sta_addr); if (sta_ptr && adapter->adhoc_11n_enabled) { - mwifiex_check_ibss_peer_capabilties(priv, sta_ptr, - adapter->event_skb); + mwifiex_check_ibss_peer_capabilities(priv, sta_ptr, + adapter->event_skb); if (sta_ptr->is_11n_enabled) for (i = 0; i < MAX_NUM_TID; i++) sta_ptr->ampdu_sta[i] = diff --git a/drivers/net/wireless/marvell/mwifiex/sta_rx.c b/drivers/net/wireless/marvell/mwifiex/sta_rx.c index fb28a5c7f4416bc84ed0b6a279c39af1db72d4f8..52a2ce2e78b081625caf933323e008b2bfe6dbc8 100644 --- a/drivers/net/wireless/marvell/mwifiex/sta_rx.c +++ b/drivers/net/wireless/marvell/mwifiex/sta_rx.c @@ -250,7 +250,8 @@ int mwifiex_process_sta_rx_packet(struct mwifiex_private *priv, local_rx_pd->nf); } } else { - if (rx_pkt_type != PKT_TYPE_BAR) + if (rx_pkt_type != PKT_TYPE_BAR && + local_rx_pd->priority < MAX_NUM_TID) priv->rx_seq[local_rx_pd->priority] = seq_num; memcpy(ta, priv->curr_bss_params.bss_descriptor.mac_address, ETH_ALEN); diff --git a/drivers/net/wireless/marvell/mwifiex/uap_event.c b/drivers/net/wireless/marvell/mwifiex/uap_event.c index ca759d9c025396ca14d0a73fb34d4feed56ad41b..86bfa1b9ef9d125b04a0f6fec225b5f3b7329ef6 100644 --- a/drivers/net/wireless/marvell/mwifiex/uap_event.c +++ b/drivers/net/wireless/marvell/mwifiex/uap_event.c @@ -23,8 +23,8 @@ #define MWIFIEX_BSS_START_EVT_FIX_SIZE 12 -static int mwifiex_check_uap_capabilties(struct mwifiex_private *priv, - struct sk_buff *event) +static int mwifiex_check_uap_capabilities(struct mwifiex_private *priv, + struct sk_buff *event) { int evt_len; u8 *curr; @@ -38,7 +38,7 @@ static int mwifiex_check_uap_capabilties(struct mwifiex_private *priv, evt_len = event->len; curr = event->data; - mwifiex_dbg_dump(priv->adapter, EVT_D, "uap capabilties:", + mwifiex_dbg_dump(priv->adapter, EVT_D, "uap capabilities:", event->data, event->len); skb_push(event, MWIFIEX_BSS_START_EVT_FIX_SIZE); @@ -201,7 +201,7 @@ int mwifiex_process_uap_event(struct mwifiex_private *priv) ETH_ALEN); if (priv->hist_data) mwifiex_hist_data_reset(priv); - mwifiex_check_uap_capabilties(priv, adapter->event_skb); + mwifiex_check_uap_capabilities(priv, adapter->event_skb); break; case EVENT_UAP_MIC_COUNTERMEASURES: /* For future development */ diff --git a/drivers/net/wireless/marvell/mwl8k.c b/drivers/net/wireless/marvell/mwl8k.c index 8e4e9b6919e0274cbdbc3a2cfde72b2e40c8fb6f..c4db6417748f346518619c65720c443968a3544e 100644 --- a/drivers/net/wireless/marvell/mwl8k.c +++ b/drivers/net/wireless/marvell/mwl8k.c @@ -441,6 +441,9 @@ static const struct ieee80211_rate mwl8k_rates_50[] = { #define MWL8K_CMD_UPDATE_STADB 0x1123 #define MWL8K_CMD_BASTREAM 0x1125 +#define MWL8K_LEGACY_5G_RATE_OFFSET \ + (ARRAY_SIZE(mwl8k_rates_24) - ARRAY_SIZE(mwl8k_rates_50)) + static const char *mwl8k_cmd_name(__le16 cmd, char *buf, int bufsize) { u16 command = le16_to_cpu(cmd); @@ -1016,8 +1019,9 @@ mwl8k_rxd_ap_process(void *_rxd, struct ieee80211_rx_status *status, if (rxd->channel > 14) { status->band = NL80211_BAND_5GHZ; - if (!(status->encoding == RX_ENC_HT)) - status->rate_idx -= 5; + if (!(status->encoding == RX_ENC_HT) && + status->rate_idx >= MWL8K_LEGACY_5G_RATE_OFFSET) + status->rate_idx -= MWL8K_LEGACY_5G_RATE_OFFSET; } else { status->band = NL80211_BAND_2GHZ; } @@ -1124,8 +1128,9 @@ mwl8k_rxd_sta_process(void *_rxd, struct ieee80211_rx_status *status, if (rxd->channel > 14) { status->band = NL80211_BAND_5GHZ; - if (!(status->encoding == RX_ENC_HT)) - status->rate_idx -= 5; + if (!(status->encoding == RX_ENC_HT) && + status->rate_idx >= MWL8K_LEGACY_5G_RATE_OFFSET) + status->rate_idx -= MWL8K_LEGACY_5G_RATE_OFFSET; } else { status->band = NL80211_BAND_2GHZ; } @@ -2234,8 +2239,10 @@ static int mwl8k_post_cmd(struct ieee80211_hw *hw, struct mwl8k_cmd_pkt *cmd) dma_size = le16_to_cpu(cmd->length); dma_addr = pci_map_single(priv->pdev, cmd, dma_size, PCI_DMA_BIDIRECTIONAL); - if (pci_dma_mapping_error(priv->pdev, dma_addr)) - return -ENOMEM; + if (pci_dma_mapping_error(priv->pdev, dma_addr)) { + rc = -ENOMEM; + goto exit; + } priv->hostcmd_wait = &cmd_wait; iowrite32(dma_addr, regs + MWL8K_HIU_GEN_PTR); @@ -2275,6 +2282,7 @@ static int mwl8k_post_cmd(struct ieee80211_hw *hw, struct mwl8k_cmd_pkt *cmd) ms); } +exit: if (bitmap) mwl8k_enable_bsses(hw, true, bitmap); @@ -4631,7 +4639,7 @@ static void mwl8k_tx_poll(unsigned long data) limit = 32; - spin_lock_bh(&priv->tx_lock); + spin_lock(&priv->tx_lock); for (i = 0; i < mwl8k_tx_queues(priv); i++) limit -= mwl8k_txq_reclaim(hw, i, limit, 0); @@ -4641,7 +4649,7 @@ static void mwl8k_tx_poll(unsigned long data) priv->tx_wait = NULL; } - spin_unlock_bh(&priv->tx_lock); + spin_unlock(&priv->tx_lock); if (limit) { writel(~MWL8K_A2H_INT_TX_DONE, diff --git a/drivers/net/wireless/mediatek/mt76/Kconfig b/drivers/net/wireless/mediatek/mt76/Kconfig index dbe8c70a8f73b32d22e4970f1197ae1d45e0e1c3..30e44e4c3c7d224c7a3e72d70f9b93c38fa6e579 100644 --- a/drivers/net/wireless/mediatek/mt76/Kconfig +++ b/drivers/net/wireless/mediatek/mt76/Kconfig @@ -22,3 +22,4 @@ config MT76x02_USB source "drivers/net/wireless/mediatek/mt76/mt76x0/Kconfig" source "drivers/net/wireless/mediatek/mt76/mt76x2/Kconfig" source "drivers/net/wireless/mediatek/mt76/mt7603/Kconfig" +source "drivers/net/wireless/mediatek/mt76/mt7615/Kconfig" diff --git a/drivers/net/wireless/mediatek/mt76/Makefile b/drivers/net/wireless/mediatek/mt76/Makefile index 3fd1b64b4aa75dc5707c1c120a7b019822fafbea..7beae2354a248cd6ce767f80aadac85bfa85c470 100644 --- a/drivers/net/wireless/mediatek/mt76/Makefile +++ b/drivers/net/wireless/mediatek/mt76/Makefile @@ -16,10 +16,11 @@ CFLAGS_mt76x02_trace.o := -I$(src) mt76x02-lib-y := mt76x02_util.o mt76x02_mac.o mt76x02_mcu.o \ mt76x02_eeprom.o mt76x02_phy.o mt76x02_mmio.o \ mt76x02_txrx.o mt76x02_trace.o mt76x02_debugfs.o \ - mt76x02_dfs.o + mt76x02_dfs.o mt76x02_beacon.o mt76x02-usb-y := mt76x02_usb_mcu.o mt76x02_usb_core.o obj-$(CONFIG_MT76x0_COMMON) += mt76x0/ obj-$(CONFIG_MT76x2_COMMON) += mt76x2/ obj-$(CONFIG_MT7603E) += mt7603/ +obj-$(CONFIG_MT7615E) += mt7615/ diff --git a/drivers/net/wireless/mediatek/mt76/agg-rx.c b/drivers/net/wireless/mediatek/mt76/agg-rx.c index 73c8b2805c978cb688e73bbd72156078eb98a492..27e3ff039c4813451fc7f0fb775da6a7dff734a2 100644 --- a/drivers/net/wireless/mediatek/mt76/agg-rx.c +++ b/drivers/net/wireless/mediatek/mt76/agg-rx.c @@ -135,7 +135,7 @@ mt76_rx_aggr_check_ctl(struct sk_buff *skb, struct sk_buff_head *frames) return; status->tid = le16_to_cpu(bar->control) >> 12; - seqno = le16_to_cpu(bar->start_seq_num) >> 4; + seqno = IEEE80211_SEQ_TO_SN(le16_to_cpu(bar->start_seq_num)); tid = rcu_dereference(wcid->aggr[status->tid]); if (!tid) return; diff --git a/drivers/net/wireless/mediatek/mt76/debugfs.c b/drivers/net/wireless/mediatek/mt76/debugfs.c index a5adf22c3ffad133d2a87acc8d0af9c46b73b248..c6a9fe2aef9d495b5108efeac9116193474a6941 100644 --- a/drivers/net/wireless/mediatek/mt76/debugfs.c +++ b/drivers/net/wireless/mediatek/mt76/debugfs.c @@ -43,14 +43,15 @@ mt76_queues_read(struct seq_file *s, void *data) int i; for (i = 0; i < ARRAY_SIZE(dev->q_tx); i++) { - struct mt76_queue *q = &dev->q_tx[i]; + struct mt76_sw_queue *q = &dev->q_tx[i]; - if (!q->ndesc) + if (!q->q) continue; seq_printf(s, "%d: queued=%d head=%d tail=%d swq_queued=%d\n", - i, q->queued, q->head, q->tail, q->swq_queued); + i, q->q->queued, q->q->head, q->q->tail, + q->swq_queued); } return 0; diff --git a/drivers/net/wireless/mediatek/mt76/dma.c b/drivers/net/wireless/mediatek/mt76/dma.c index 76629b98c78d78d81d7c61e0cd4771ba8d5f3c39..4381155375e1e0b6a7f108382ef3e09006c19eed 100644 --- a/drivers/net/wireless/mediatek/mt76/dma.c +++ b/drivers/net/wireless/mediatek/mt76/dma.c @@ -18,16 +18,20 @@ #include "mt76.h" #include "dma.h" -#define DMA_DUMMY_TXWI ((void *) ~0) - static int -mt76_dma_alloc_queue(struct mt76_dev *dev, struct mt76_queue *q) +mt76_dma_alloc_queue(struct mt76_dev *dev, struct mt76_queue *q, + int idx, int n_desc, int bufsize, + u32 ring_base) { int size; int i; spin_lock_init(&q->lock); - INIT_LIST_HEAD(&q->swq); + + q->regs = dev->mmio.regs + ring_base + idx * MT_RING_SIZE; + q->ndesc = n_desc; + q->buf_size = bufsize; + q->hw_idx = idx; size = q->ndesc * sizeof(struct mt76_desc); q->desc = dmam_alloc_coherent(dev->dev, size, &q->desc_dma, GFP_KERNEL); @@ -43,10 +47,10 @@ mt76_dma_alloc_queue(struct mt76_dev *dev, struct mt76_queue *q) for (i = 0; i < q->ndesc; i++) q->desc[i].ctrl = cpu_to_le32(MT_DMA_CTL_DMA_DONE); - iowrite32(q->desc_dma, &q->regs->desc_base); - iowrite32(0, &q->regs->cpu_idx); - iowrite32(0, &q->regs->dma_idx); - iowrite32(q->ndesc, &q->regs->ring_size); + writel(q->desc_dma, &q->regs->desc_base); + writel(0, &q->regs->cpu_idx); + writel(0, &q->regs->dma_idx); + writel(q->ndesc, &q->regs->ring_size); return 0; } @@ -61,7 +65,7 @@ mt76_dma_add_buf(struct mt76_dev *dev, struct mt76_queue *q, int i, idx = -1; if (txwi) - q->entry[q->head].txwi = DMA_DUMMY_TXWI; + q->entry[q->head].txwi = DMA_DUMMY_DATA; for (i = 0; i < nbufs; i += 2, buf += 2) { u32 buf0 = buf[0].addr, buf1 = 0; @@ -120,9 +124,12 @@ mt76_dma_tx_cleanup_idx(struct mt76_dev *dev, struct mt76_queue *q, int idx, DMA_TO_DEVICE); } - if (e->txwi == DMA_DUMMY_TXWI) + if (e->txwi == DMA_DUMMY_DATA) e->txwi = NULL; + if (e->skb == DMA_DUMMY_DATA) + e->skb = NULL; + *prev_e = *e; memset(e, 0, sizeof(*e)); } @@ -130,56 +137,64 @@ mt76_dma_tx_cleanup_idx(struct mt76_dev *dev, struct mt76_queue *q, int idx, static void mt76_dma_sync_idx(struct mt76_dev *dev, struct mt76_queue *q) { - iowrite32(q->desc_dma, &q->regs->desc_base); - iowrite32(q->ndesc, &q->regs->ring_size); - q->head = ioread32(&q->regs->dma_idx); + writel(q->desc_dma, &q->regs->desc_base); + writel(q->ndesc, &q->regs->ring_size); + q->head = readl(&q->regs->dma_idx); q->tail = q->head; - iowrite32(q->head, &q->regs->cpu_idx); + writel(q->head, &q->regs->cpu_idx); } static void mt76_dma_tx_cleanup(struct mt76_dev *dev, enum mt76_txq_id qid, bool flush) { - struct mt76_queue *q = &dev->q_tx[qid]; + struct mt76_sw_queue *sq = &dev->q_tx[qid]; + struct mt76_queue *q = sq->q; struct mt76_queue_entry entry; + unsigned int n_swq_queued[4] = {}; + unsigned int n_queued = 0; bool wake = false; - int last; + int i, last; - if (!q->ndesc) + if (!q) return; - spin_lock_bh(&q->lock); if (flush) last = -1; else - last = ioread32(&q->regs->dma_idx); + last = readl(&q->regs->dma_idx); - while (q->queued && q->tail != last) { + while ((q->queued > n_queued) && q->tail != last) { mt76_dma_tx_cleanup_idx(dev, q, q->tail, &entry); if (entry.schedule) - q->swq_queued--; + n_swq_queued[entry.qid]++; q->tail = (q->tail + 1) % q->ndesc; - q->queued--; + n_queued++; - if (entry.skb) { - spin_unlock_bh(&q->lock); - dev->drv->tx_complete_skb(dev, q, &entry, flush); - spin_lock_bh(&q->lock); - } + if (entry.skb) + dev->drv->tx_complete_skb(dev, qid, &entry); if (entry.txwi) { - mt76_put_txwi(dev, entry.txwi); + if (!(dev->drv->txwi_flags & MT_TXWI_NO_FREE)) + mt76_put_txwi(dev, entry.txwi); wake = !flush; } if (!flush && q->tail == last) - last = ioread32(&q->regs->dma_idx); + last = readl(&q->regs->dma_idx); } - if (!flush) - mt76_txq_schedule(dev, q); - else + spin_lock_bh(&q->lock); + + q->queued -= n_queued; + for (i = 0; i < ARRAY_SIZE(n_swq_queued); i++) { + if (!n_swq_queued[i]) + continue; + + dev->q_tx[i].swq_queued -= n_swq_queued[i]; + } + + if (flush) mt76_dma_sync_idx(dev, q); wake = wake && q->stopped && @@ -244,20 +259,20 @@ mt76_dma_dequeue(struct mt76_dev *dev, struct mt76_queue *q, bool flush, static void mt76_dma_kick_queue(struct mt76_dev *dev, struct mt76_queue *q) { - iowrite32(q->head, &q->regs->cpu_idx); + writel(q->head, &q->regs->cpu_idx); } static int mt76_dma_tx_queue_skb_raw(struct mt76_dev *dev, enum mt76_txq_id qid, struct sk_buff *skb, u32 tx_info) { - struct mt76_queue *q = &dev->q_tx[qid]; + struct mt76_queue *q = dev->q_tx[qid].q; struct mt76_queue_buf buf; dma_addr_t addr; addr = dma_map_single(dev->dev, skb->data, skb->len, DMA_TO_DEVICE); - if (dma_mapping_error(dev->dev, addr)) + if (unlikely(dma_mapping_error(dev->dev, addr))) return -ENOMEM; buf.addr = addr; @@ -271,80 +286,85 @@ mt76_dma_tx_queue_skb_raw(struct mt76_dev *dev, enum mt76_txq_id qid, return 0; } -int mt76_dma_tx_queue_skb(struct mt76_dev *dev, struct mt76_queue *q, - struct sk_buff *skb, struct mt76_wcid *wcid, - struct ieee80211_sta *sta) +static int +mt76_dma_tx_queue_skb(struct mt76_dev *dev, enum mt76_txq_id qid, + struct sk_buff *skb, struct mt76_wcid *wcid, + struct ieee80211_sta *sta) { + struct mt76_queue *q = dev->q_tx[qid].q; + struct mt76_tx_info tx_info = { + .skb = skb, + }; + int len, n = 0, ret = -ENOMEM; struct mt76_queue_entry e; struct mt76_txwi_cache *t; - struct mt76_queue_buf buf[32]; struct sk_buff *iter; dma_addr_t addr; - int len; - u32 tx_info = 0; - int n, ret; + u8 *txwi; t = mt76_get_txwi(dev); if (!t) { ieee80211_free_txskb(dev->hw, skb); return -ENOMEM; } + txwi = mt76_get_txwi_ptr(dev, t); skb->prev = skb->next = NULL; - dma_sync_single_for_cpu(dev->dev, t->dma_addr, sizeof(t->txwi), - DMA_TO_DEVICE); - ret = dev->drv->tx_prepare_skb(dev, &t->txwi, skb, q, wcid, sta, - &tx_info); - dma_sync_single_for_device(dev->dev, t->dma_addr, sizeof(t->txwi), - DMA_TO_DEVICE); - if (ret < 0) - goto free; + if (dev->drv->tx_aligned4_skbs) + mt76_insert_hdr_pad(skb); - len = skb->len - skb->data_len; + len = skb_headlen(skb); addr = dma_map_single(dev->dev, skb->data, len, DMA_TO_DEVICE); - if (dma_mapping_error(dev->dev, addr)) { - ret = -ENOMEM; + if (unlikely(dma_mapping_error(dev->dev, addr))) goto free; - } - n = 0; - buf[n].addr = t->dma_addr; - buf[n++].len = dev->drv->txwi_size; - buf[n].addr = addr; - buf[n++].len = len; + tx_info.buf[n].addr = t->dma_addr; + tx_info.buf[n++].len = dev->drv->txwi_size; + tx_info.buf[n].addr = addr; + tx_info.buf[n++].len = len; skb_walk_frags(skb, iter) { - if (n == ARRAY_SIZE(buf)) + if (n == ARRAY_SIZE(tx_info.buf)) goto unmap; addr = dma_map_single(dev->dev, iter->data, iter->len, DMA_TO_DEVICE); - if (dma_mapping_error(dev->dev, addr)) + if (unlikely(dma_mapping_error(dev->dev, addr))) goto unmap; - buf[n].addr = addr; - buf[n++].len = iter->len; + tx_info.buf[n].addr = addr; + tx_info.buf[n++].len = iter->len; } + tx_info.nbuf = n; + + dma_sync_single_for_cpu(dev->dev, t->dma_addr, dev->drv->txwi_size, + DMA_TO_DEVICE); + ret = dev->drv->tx_prepare_skb(dev, txwi, qid, wcid, sta, &tx_info); + dma_sync_single_for_device(dev->dev, t->dma_addr, dev->drv->txwi_size, + DMA_TO_DEVICE); + if (ret < 0) + goto unmap; - if (q->queued + (n + 1) / 2 >= q->ndesc - 1) + if (q->queued + (tx_info.nbuf + 1) / 2 >= q->ndesc - 1) { + ret = -ENOMEM; goto unmap; + } - return mt76_dma_add_buf(dev, q, buf, n, tx_info, skb, t); + return mt76_dma_add_buf(dev, q, tx_info.buf, tx_info.nbuf, + tx_info.info, tx_info.skb, t); unmap: - ret = -ENOMEM; for (n--; n > 0; n--) - dma_unmap_single(dev->dev, buf[n].addr, buf[n].len, - DMA_TO_DEVICE); + dma_unmap_single(dev->dev, tx_info.buf[n].addr, + tx_info.buf[n].len, DMA_TO_DEVICE); free: - e.skb = skb; + e.skb = tx_info.skb; e.txwi = t; - dev->drv->tx_complete_skb(dev, q, &e, true); + dev->drv->tx_complete_skb(dev, qid, &e); mt76_put_txwi(dev, t); return ret; } -EXPORT_SYMBOL_GPL(mt76_dma_tx_queue_skb); static int mt76_dma_rx_fill(struct mt76_dev *dev, struct mt76_queue *q) @@ -366,7 +386,7 @@ mt76_dma_rx_fill(struct mt76_dev *dev, struct mt76_queue *q) break; addr = dma_map_single(dev->dev, buf, len, DMA_FROM_DEVICE); - if (dma_mapping_error(dev->dev, addr)) { + if (unlikely(dma_mapping_error(dev->dev, addr))) { skb_free_frag(buf); break; } diff --git a/drivers/net/wireless/mediatek/mt76/dma.h b/drivers/net/wireless/mediatek/mt76/dma.h index e3292df5e9b25723e4acf08499fcc974ae31e1b7..03dd2bafa4e8bd28b8b55ec46bde2c7831a4cadf 100644 --- a/drivers/net/wireless/mediatek/mt76/dma.h +++ b/drivers/net/wireless/mediatek/mt76/dma.h @@ -16,6 +16,8 @@ #ifndef __MT76_DMA_H #define __MT76_DMA_H +#define DMA_DUMMY_DATA ((void *)~0) + #define MT_RING_SIZE 0x10 #define MT_DMA_CTL_SD_LEN1 GENMASK(13, 0) diff --git a/drivers/net/wireless/mediatek/mt76/eeprom.c b/drivers/net/wireless/mediatek/mt76/eeprom.c index a1529920d8776350a26c61a8256bb570a1784cc5..04964937a3af24c544b969c3791dd64f44e3d62e 100644 --- a/drivers/net/wireless/mediatek/mt76/eeprom.c +++ b/drivers/net/wireless/mediatek/mt76/eeprom.c @@ -94,7 +94,7 @@ mt76_eeprom_override(struct mt76_dev *dev) return; mac = of_get_mac_address(np); - if (mac) + if (!IS_ERR(mac)) memcpy(dev->macaddr, mac, ETH_ALEN); #endif diff --git a/drivers/net/wireless/mediatek/mt76/mac80211.c b/drivers/net/wireless/mediatek/mt76/mac80211.c index 316167404729fdcd8322c71bd5626785822ce90d..5b6a81ee457e906e79196856ccdd6e6a2c3bdeb6 100644 --- a/drivers/net/wireless/mediatek/mt76/mac80211.c +++ b/drivers/net/wireless/mediatek/mt76/mac80211.c @@ -214,6 +214,8 @@ mt76_init_sband(struct mt76_dev *dev, struct mt76_sband *msband, vht_cap->cap |= IEEE80211_VHT_CAP_RXLDPC | IEEE80211_VHT_CAP_RXSTBC_1 | IEEE80211_VHT_CAP_SHORT_GI_80 | + IEEE80211_VHT_CAP_RX_ANTENNA_PATTERN | + IEEE80211_VHT_CAP_TX_ANTENNA_PATTERN | (3 << IEEE80211_VHT_CAP_MAX_A_MPDU_LENGTH_EXPONENT_SHIFT); return 0; @@ -369,10 +371,16 @@ void mt76_unregister_device(struct mt76_dev *dev) mt76_tx_status_check(dev, NULL, true); ieee80211_unregister_hw(hw); - mt76_tx_free(dev); } EXPORT_SYMBOL_GPL(mt76_unregister_device); +void mt76_free_device(struct mt76_dev *dev) +{ + mt76_tx_free(dev); + ieee80211_free_hw(dev->hw); +} +EXPORT_SYMBOL_GPL(mt76_free_device); + void mt76_rx(struct mt76_dev *dev, enum mt76_rxq_id q, struct sk_buff *skb) { if (!test_bit(MT76_STATE_RUNNING, &dev->state)) { @@ -384,17 +392,20 @@ void mt76_rx(struct mt76_dev *dev, enum mt76_rxq_id q, struct sk_buff *skb) } EXPORT_SYMBOL_GPL(mt76_rx); -static bool mt76_has_tx_pending(struct mt76_dev *dev) +bool mt76_has_tx_pending(struct mt76_dev *dev) { + struct mt76_queue *q; int i; for (i = 0; i < ARRAY_SIZE(dev->q_tx); i++) { - if (dev->q_tx[i].queued) + q = dev->q_tx[i].q; + if (q && q->queued) return true; } return false; } +EXPORT_SYMBOL_GPL(mt76_has_tx_pending); void mt76_set_channel(struct mt76_dev *dev) { @@ -560,6 +571,7 @@ mt76_check_sta(struct mt76_dev *dev, struct sk_buff *skb) struct ieee80211_sta *sta; struct mt76_wcid *wcid = status->wcid; bool ps; + int i; if (ieee80211_is_pspoll(hdr->frame_control) && !wcid) { sta = ieee80211_find_sta_by_ifaddr(dev->hw, hdr->addr2, NULL); @@ -606,6 +618,20 @@ mt76_check_sta(struct mt76_dev *dev, struct sk_buff *skb) dev->drv->sta_ps(dev, sta, ps); ieee80211_sta_ps_transition(sta, ps); + + if (ps) + return; + + for (i = 0; i < ARRAY_SIZE(sta->txq); i++) { + struct mt76_txq *mtxq; + + if (!sta->txq[i]) + continue; + + mtxq = (struct mt76_txq *) sta->txq[i]->drv_priv; + if (!skb_queue_empty(&mtxq->retry_q)) + ieee80211_schedule_txq(dev->hw, sta->txq[i]); + } } void mt76_rx_complete(struct mt76_dev *dev, struct sk_buff_head *frames, @@ -737,7 +763,7 @@ int mt76_get_txpower(struct ieee80211_hw *hw, struct ieee80211_vif *vif, struct mt76_dev *dev = hw->priv; int n_chains = hweight8(dev->antenna_mask); - *dbm = dev->txpower_cur / 2; + *dbm = DIV_ROUND_UP(dev->txpower_cur, 2); /* convert from per-chain power to combined * output on 2x2 devices @@ -787,3 +813,10 @@ void mt76_csa_check(struct mt76_dev *dev) __mt76_csa_check, dev); } EXPORT_SYMBOL_GPL(mt76_csa_check); + +int +mt76_set_tim(struct ieee80211_hw *hw, struct ieee80211_sta *sta, bool set) +{ + return 0; +} +EXPORT_SYMBOL_GPL(mt76_set_tim); diff --git a/drivers/net/wireless/mediatek/mt76/mmio.c b/drivers/net/wireless/mediatek/mt76/mmio.c index 1d6bbce76041c2e4996f81bea00ac2fe69288bab..38368d19aa6f01a45155bd7fe7d035514cd15486 100644 --- a/drivers/net/wireless/mediatek/mt76/mmio.c +++ b/drivers/net/wireless/mediatek/mt76/mmio.c @@ -21,7 +21,7 @@ static u32 mt76_mmio_rr(struct mt76_dev *dev, u32 offset) { u32 val; - val = ioread32(dev->mmio.regs + offset); + val = readl(dev->mmio.regs + offset); trace_reg_rr(dev, offset, val); return val; @@ -30,7 +30,7 @@ static u32 mt76_mmio_rr(struct mt76_dev *dev, u32 offset) static void mt76_mmio_wr(struct mt76_dev *dev, u32 offset, u32 val) { trace_reg_wr(dev, offset, val); - iowrite32(val, dev->mmio.regs + offset); + writel(val, dev->mmio.regs + offset); } static u32 mt76_mmio_rmw(struct mt76_dev *dev, u32 offset, u32 mask, u32 val) @@ -70,6 +70,19 @@ static int mt76_mmio_rd_rp(struct mt76_dev *dev, u32 base, return 0; } +void mt76_set_irq_mask(struct mt76_dev *dev, u32 addr, + u32 clear, u32 set) +{ + unsigned long flags; + + spin_lock_irqsave(&dev->mmio.irq_lock, flags); + dev->mmio.irqmask &= ~clear; + dev->mmio.irqmask |= set; + mt76_mmio_wr(dev, addr, dev->mmio.irqmask); + spin_unlock_irqrestore(&dev->mmio.irq_lock, flags); +} +EXPORT_SYMBOL_GPL(mt76_set_irq_mask); + void mt76_mmio_init(struct mt76_dev *dev, void __iomem *regs) { static const struct mt76_bus_ops mt76_mmio_ops = { diff --git a/drivers/net/wireless/mediatek/mt76/mt76.h b/drivers/net/wireless/mediatek/mt76/mt76.h index bcbfd3c4a44b68199dbc53e0e5ea498cc9e78e89..8ecbf81a906f5b1374ee03e89d85163302c0919d 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76.h +++ b/drivers/net/wireless/mediatek/mt76/mt76.h @@ -69,6 +69,7 @@ enum mt76_txq_id { MT_TXQ_MCU, MT_TXQ_BEACON, MT_TXQ_CAB, + MT_TXQ_FWDL, __MT_TXQ_MAX }; @@ -83,12 +84,11 @@ struct mt76_queue_buf { int len; }; -struct mt76u_buf { - struct mt76_dev *dev; - struct urb *urb; - size_t len; - void *buf; - bool done; +struct mt76_tx_info { + struct mt76_queue_buf buf[32]; + struct sk_buff *skb; + int nbuf; + u32 info; }; struct mt76_queue_entry { @@ -98,9 +98,11 @@ struct mt76_queue_entry { }; union { struct mt76_txwi_cache *txwi; - struct mt76u_buf ubuf; + struct urb *urb; }; + enum mt76_txq_id qid; bool schedule; + bool done; }; struct mt76_queue_regs { @@ -117,9 +119,6 @@ struct mt76_queue { struct mt76_queue_entry *entry; struct mt76_desc *desc; - struct list_head swq; - int swq_queued; - u16 first; u16 head; u16 tail; @@ -134,7 +133,13 @@ struct mt76_queue { dma_addr_t desc_dma; struct sk_buff *rx_head; struct page_frag_cache rx_page; - spinlock_t rx_page_lock; +}; + +struct mt76_sw_queue { + struct mt76_queue *q; + + struct list_head swq; + int swq_queued; }; struct mt76_mcu_ops { @@ -150,13 +155,15 @@ struct mt76_mcu_ops { struct mt76_queue_ops { int (*init)(struct mt76_dev *dev); - int (*alloc)(struct mt76_dev *dev, struct mt76_queue *q); + int (*alloc)(struct mt76_dev *dev, struct mt76_queue *q, + int idx, int n_desc, int bufsize, + u32 ring_base); int (*add_buf)(struct mt76_dev *dev, struct mt76_queue *q, struct mt76_queue_buf *buf, int nbufs, u32 info, struct sk_buff *skb, void *txwi); - int (*tx_queue_skb)(struct mt76_dev *dev, struct mt76_queue *q, + int (*tx_queue_skb)(struct mt76_dev *dev, enum mt76_txq_id qid, struct sk_buff *skb, struct mt76_wcid *wcid, struct ieee80211_sta *sta); @@ -183,6 +190,11 @@ enum mt76_wcid_flags { DECLARE_EWMA(signal, 10, 8); +#define MT_WCID_TX_INFO_RATE GENMASK(15, 0) +#define MT_WCID_TX_INFO_NSS GENMASK(17, 16) +#define MT_WCID_TX_INFO_TXPWR_ADJ GENMASK(25, 18) +#define MT_WCID_TX_INFO_SET BIT(31) + struct mt76_wcid { struct mt76_rx_tid __rcu *aggr[IEEE80211_NUM_TIDS]; @@ -201,18 +213,14 @@ struct mt76_wcid { u8 rx_check_pn; u8 rx_key_pn[IEEE80211_NUM_TIDS][6]; - __le16 tx_rate; - bool tx_rate_set; - u8 tx_rate_nss; - s8 max_txpwr_adj; + u32 tx_info; bool sw_iv; u8 packet_id; }; struct mt76_txq { - struct list_head list; - struct mt76_queue *hwq; + struct mt76_sw_queue *swq; struct mt76_wcid *wcid; struct sk_buff_head retry_q; @@ -223,11 +231,11 @@ struct mt76_txq { }; struct mt76_txwi_cache { - u32 txwi[8]; - dma_addr_t dma_addr; struct list_head list; -}; + dma_addr_t dma_addr; + struct sk_buff *skb; +}; struct mt76_rx_tid { struct rcu_head rcu_head; @@ -280,18 +288,22 @@ struct mt76_hw_cap { bool has_5ghz; }; +#define MT_TXWI_NO_FREE BIT(0) + struct mt76_driver_ops { + bool tx_aligned4_skbs; + u32 txwi_flags; u16 txwi_size; void (*update_survey)(struct mt76_dev *dev); int (*tx_prepare_skb)(struct mt76_dev *dev, void *txwi_ptr, - struct sk_buff *skb, struct mt76_queue *q, - struct mt76_wcid *wcid, - struct ieee80211_sta *sta, u32 *tx_info); + enum mt76_txq_id qid, struct mt76_wcid *wcid, + struct ieee80211_sta *sta, + struct mt76_tx_info *tx_info); - void (*tx_complete_skb)(struct mt76_dev *dev, struct mt76_queue *q, - struct mt76_queue_entry *e, bool flush); + void (*tx_complete_skb)(struct mt76_dev *dev, enum mt76_txq_id qid, + struct mt76_queue_entry *e); bool (*tx_status_data)(struct mt76_dev *dev, u8 *update); @@ -378,7 +390,6 @@ struct mt76_usb { u8 data[32]; struct tasklet_struct rx_tasklet; - struct tasklet_struct tx_tasklet; struct delayed_work stat_work; u8 out_ep[__MT_EP_OUT_MAX]; @@ -435,11 +446,14 @@ struct mt76_dev { struct sk_buff_head rx_skb[__MT_RXQ_MAX]; struct list_head txwi_cache; - struct mt76_queue q_tx[__MT_TXQ_MAX]; + struct mt76_sw_queue q_tx[__MT_TXQ_MAX]; struct mt76_queue q_rx[__MT_RXQ_MAX]; const struct mt76_queue_ops *queue_ops; int tx_dma_idx[4]; + struct tasklet_struct tx_tasklet; + struct delayed_work mac_work; + wait_queue_head_t tx_wait; struct sk_buff_head status_list; @@ -455,6 +469,10 @@ struct mt76_dev { u8 antenna_mask; u16 chainmask; + struct tasklet_struct pre_tbtt_tasklet; + int beacon_int; + u8 beacon_mask; + struct mt76_sband sband_2g; struct mt76_sband sband_5g; struct debugfs_blob_wrapper eeprom; @@ -529,6 +547,9 @@ struct mt76_rx_status { #define mt76_rd_rp(dev, ...) (dev)->mt76.bus->rd_rp(&((dev)->mt76), __VA_ARGS__) #define mt76_mcu_send_msg(dev, ...) (dev)->mt76.mcu_ops->mcu_send_msg(&((dev)->mt76), __VA_ARGS__) +#define __mt76_mcu_send_msg(dev, ...) (dev)->mcu_ops->mcu_send_msg((dev), __VA_ARGS__) +#define mt76_mcu_restart(dev, ...) (dev)->mt76.mcu_ops->mcu_restart(&((dev)->mt76)) +#define __mt76_mcu_restart(dev, ...) (dev)->mcu_ops->mcu_restart((dev)) #define mt76_set(dev, offset, val) mt76_rmw(dev, offset, 0, val) #define mt76_clear(dev, offset, val) mt76_rmw(dev, offset, val, 0) @@ -572,6 +593,7 @@ static inline u16 mt76_rev(struct mt76_dev *dev) #define mt76_init_queues(dev) (dev)->mt76.queue_ops->init(&((dev)->mt76)) #define mt76_queue_alloc(dev, ...) (dev)->mt76.queue_ops->alloc(&((dev)->mt76), __VA_ARGS__) #define mt76_tx_queue_skb_raw(dev, ...) (dev)->mt76.queue_ops->tx_queue_skb_raw(&((dev)->mt76), __VA_ARGS__) +#define mt76_tx_queue_skb(dev, ...) (dev)->mt76.queue_ops->tx_queue_skb(&((dev)->mt76), __VA_ARGS__) #define mt76_queue_rx_reset(dev, ...) (dev)->mt76.queue_ops->rx_reset(&((dev)->mt76), __VA_ARGS__) #define mt76_queue_tx_cleanup(dev, ...) (dev)->mt76.queue_ops->tx_cleanup(&((dev)->mt76), __VA_ARGS__) #define mt76_queue_kick(dev, ...) (dev)->mt76.queue_ops->kick(&((dev)->mt76), __VA_ARGS__) @@ -597,6 +619,7 @@ struct mt76_dev *mt76_alloc_device(struct device *pdev, unsigned int size, int mt76_register_device(struct mt76_dev *dev, bool vht, struct ieee80211_rate *rates, int n_rates); void mt76_unregister_device(struct mt76_dev *dev); +void mt76_free_device(struct mt76_dev *dev); struct dentry *mt76_register_debugfs(struct mt76_dev *dev); void mt76_seq_puts_array(struct seq_file *file, const char *str, @@ -605,6 +628,12 @@ void mt76_seq_puts_array(struct seq_file *file, const char *str, int mt76_eeprom_init(struct mt76_dev *dev, int len); void mt76_eeprom_override(struct mt76_dev *dev); +static inline u8 * +mt76_get_txwi_ptr(struct mt76_dev *dev, struct mt76_txwi_cache *t) +{ + return (u8 *)t - dev->drv->txwi_size; +} + /* increment with wrap-around */ static inline int mt76_incr(int val, int size) { @@ -645,9 +674,19 @@ static inline struct mt76_tx_cb *mt76_tx_skb_cb(struct sk_buff *skb) return ((void *) IEEE80211_SKB_CB(skb)->status.status_driver_data); } -int mt76_dma_tx_queue_skb(struct mt76_dev *dev, struct mt76_queue *q, - struct sk_buff *skb, struct mt76_wcid *wcid, - struct ieee80211_sta *sta); +static inline void mt76_insert_hdr_pad(struct sk_buff *skb) +{ + int len = ieee80211_get_hdrlen_from_skb(skb); + + if (len % 4 == 0) + return; + + skb_push(skb, 2); + memmove(skb->data, skb->data + 2, len); + + skb->data[len] = 0; + skb->data[len + 1] = 0; +} void mt76_rx(struct mt76_dev *dev, enum mt76_rxq_id q, struct sk_buff *skb); void mt76_tx(struct mt76_dev *dev, struct ieee80211_sta *sta, @@ -657,13 +696,14 @@ void mt76_txq_remove(struct mt76_dev *dev, struct ieee80211_txq *txq); void mt76_wake_tx_queue(struct ieee80211_hw *hw, struct ieee80211_txq *txq); void mt76_stop_tx_queues(struct mt76_dev *dev, struct ieee80211_sta *sta, bool send_bar); -void mt76_txq_schedule(struct mt76_dev *dev, struct mt76_queue *hwq); +void mt76_txq_schedule(struct mt76_dev *dev, enum mt76_txq_id qid); void mt76_txq_schedule_all(struct mt76_dev *dev); void mt76_release_buffered_frames(struct ieee80211_hw *hw, struct ieee80211_sta *sta, u16 tids, int nframes, enum ieee80211_frame_release_type reason, bool more_data); +bool mt76_has_tx_pending(struct mt76_dev *dev); void mt76_set_channel(struct mt76_dev *dev); int mt76_get_survey(struct ieee80211_hw *hw, int idx, struct survey_info *survey); @@ -708,6 +748,8 @@ int mt76_get_txpower(struct ieee80211_hw *hw, struct ieee80211_vif *vif, void mt76_csa_check(struct mt76_dev *dev); void mt76_csa_finish(struct mt76_dev *dev); +int mt76_set_tim(struct ieee80211_hw *hw, struct ieee80211_sta *sta, bool set); + /* internal */ void mt76_tx_free(struct mt76_dev *dev); struct mt76_txwi_cache *mt76_get_txwi(struct mt76_dev *dev); @@ -738,8 +780,7 @@ static inline int mt76u_bulk_msg(struct mt76_dev *dev, void *data, int len, int *actual_len, int timeout) { - struct usb_interface *intf = to_usb_interface(dev->dev); - struct usb_device *udev = interface_to_usbdev(intf); + struct usb_device *udev = to_usb_device(dev->dev); struct mt76_usb *usb = &dev->usb; unsigned int pipe; @@ -757,10 +798,10 @@ int mt76u_vendor_request(struct mt76_dev *dev, u8 req, void mt76u_single_wr(struct mt76_dev *dev, const u8 req, const u16 offset, const u32 val); int mt76u_init(struct mt76_dev *dev, struct usb_interface *intf); -int mt76u_submit_rx_buffers(struct mt76_dev *dev); int mt76u_alloc_queues(struct mt76_dev *dev); -void mt76u_stop_queues(struct mt76_dev *dev); -void mt76u_stop_stat_wk(struct mt76_dev *dev); +void mt76u_stop_tx(struct mt76_dev *dev); +void mt76u_stop_rx(struct mt76_dev *dev); +int mt76u_resume_rx(struct mt76_dev *dev); void mt76u_queues_deinit(struct mt76_dev *dev); struct sk_buff * @@ -770,4 +811,6 @@ void mt76_mcu_rx_event(struct mt76_dev *dev, struct sk_buff *skb); struct sk_buff *mt76_mcu_get_response(struct mt76_dev *dev, unsigned long expires); +void mt76_set_irq_mask(struct mt76_dev *dev, u32 addr, u32 clear, u32 set); + #endif diff --git a/drivers/net/wireless/mediatek/mt76/mt7603/beacon.c b/drivers/net/wireless/mediatek/mt76/mt7603/beacon.c index 4dcb465095d19e9a0fe88c15a757a59fec802d87..58e68fbdbf75f58ecb8fede0902f11c27219cd44 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7603/beacon.c +++ b/drivers/net/wireless/mediatek/mt76/mt7603/beacon.c @@ -16,21 +16,20 @@ mt7603_update_beacon_iter(void *priv, u8 *mac, struct ieee80211_vif *vif) struct mt7603_vif *mvif = (struct mt7603_vif *)vif->drv_priv; struct sk_buff *skb = NULL; - if (!(dev->beacon_mask & BIT(mvif->idx))) + if (!(dev->mt76.beacon_mask & BIT(mvif->idx))) return; skb = ieee80211_beacon_get(mt76_hw(dev), vif); if (!skb) return; - mt76_dma_tx_queue_skb(&dev->mt76, &dev->mt76.q_tx[MT_TXQ_BEACON], skb, - &mvif->sta.wcid, NULL); + mt76_tx_queue_skb(dev, MT_TXQ_BEACON, skb, &mvif->sta.wcid, NULL); spin_lock_bh(&dev->ps_lock); mt76_wr(dev, MT_DMA_FQCR0, MT_DMA_FQCR0_BUSY | FIELD_PREP(MT_DMA_FQCR0_TARGET_WCID, mvif->sta.wcid.idx) | FIELD_PREP(MT_DMA_FQCR0_TARGET_QID, - dev->mt76.q_tx[MT_TXQ_CAB].hw_idx) | + dev->mt76.q_tx[MT_TXQ_CAB].q->hw_idx) | FIELD_PREP(MT_DMA_FQCR0_DEST_PORT_ID, 3) | FIELD_PREP(MT_DMA_FQCR0_DEST_QUEUE_ID, 8)); @@ -49,7 +48,7 @@ mt7603_add_buffered_bc(void *priv, u8 *mac, struct ieee80211_vif *vif) struct ieee80211_tx_info *info; struct sk_buff *skb; - if (!(dev->beacon_mask & BIT(mvif->idx))) + if (!(dev->mt76.beacon_mask & BIT(mvif->idx))) return; skb = ieee80211_get_buffered_bc(mt76_hw(dev), vif); @@ -73,10 +72,13 @@ void mt7603_pre_tbtt_tasklet(unsigned long arg) struct sk_buff *skb; int i, nframes; + if (mt76_hw(dev)->conf.flags & IEEE80211_CONF_OFFCHANNEL) + return; + data.dev = dev; __skb_queue_head_init(&data.q); - q = &dev->mt76.q_tx[MT_TXQ_BEACON]; + q = dev->mt76.q_tx[MT_TXQ_BEACON].q; spin_lock_bh(&q->lock); ieee80211_iterate_active_interfaces_atomic(mt76_hw(dev), IEEE80211_IFACE_ITER_RESUME_ALL, @@ -93,7 +95,7 @@ void mt7603_pre_tbtt_tasklet(unsigned long arg) if (dev->mt76.csa_complete) goto out; - q = &dev->mt76.q_tx[MT_TXQ_CAB]; + q = dev->mt76.q_tx[MT_TXQ_CAB].q; do { nframes = skb_queue_len(&data.q); ieee80211_iterate_active_interfaces_atomic(mt76_hw(dev), @@ -118,8 +120,7 @@ void mt7603_pre_tbtt_tasklet(unsigned long arg) struct ieee80211_vif *vif = info->control.vif; struct mt7603_vif *mvif = (struct mt7603_vif *)vif->drv_priv; - mt76_dma_tx_queue_skb(&dev->mt76, q, skb, &mvif->sta.wcid, - NULL); + mt76_tx_queue_skb(dev, MT_TXQ_CAB, skb, &mvif->sta.wcid, NULL); } mt76_queue_kick(dev, q); spin_unlock_bh(&q->lock); @@ -135,7 +136,8 @@ void mt7603_pre_tbtt_tasklet(unsigned long arg) out: mt76_queue_tx_cleanup(dev, MT_TXQ_BEACON, false); - if (dev->mt76.q_tx[MT_TXQ_BEACON].queued > hweight8(dev->beacon_mask)) + if (dev->mt76.q_tx[MT_TXQ_BEACON].q->queued > + hweight8(dev->mt76.beacon_mask)) dev->beacon_check++; } @@ -145,19 +147,19 @@ void mt7603_beacon_set_timer(struct mt7603_dev *dev, int idx, int intval) if (idx >= 0) { if (intval) - dev->beacon_mask |= BIT(idx); + dev->mt76.beacon_mask |= BIT(idx); else - dev->beacon_mask &= ~BIT(idx); + dev->mt76.beacon_mask &= ~BIT(idx); } - if (!dev->beacon_mask || (!intval && idx < 0)) { + if (!dev->mt76.beacon_mask || (!intval && idx < 0)) { mt7603_irq_disable(dev, MT_INT_MAC_IRQ3); mt76_clear(dev, MT_ARB_SCR, MT_ARB_SCR_BCNQ_OPMODE_MASK); mt76_wr(dev, MT_HW_INT_MASK(3), 0); return; } - dev->beacon_int = intval; + dev->mt76.beacon_int = intval; mt76_wr(dev, MT_TBTT, FIELD_PREP(MT_TBTT_PERIOD, intval) | MT_TBTT_CAL_ENABLE); @@ -175,10 +177,11 @@ void mt7603_beacon_set_timer(struct mt7603_dev *dev, int idx, int intval) mt76_set(dev, MT_WF_ARB_BCN_START, MT_WF_ARB_BCN_START_BSSn(0) | - ((dev->beacon_mask >> 1) * MT_WF_ARB_BCN_START_BSS0n(1))); + ((dev->mt76.beacon_mask >> 1) * + MT_WF_ARB_BCN_START_BSS0n(1))); mt7603_irq_enable(dev, MT_INT_MAC_IRQ3); - if (dev->beacon_mask & ~BIT(0)) + if (dev->mt76.beacon_mask & ~BIT(0)) mt76_set(dev, MT_LPON_SBTOR(0), MT_LPON_SBTOR_SUB_BSS_EN); else mt76_clear(dev, MT_LPON_SBTOR(0), MT_LPON_SBTOR_SUB_BSS_EN); diff --git a/drivers/net/wireless/mediatek/mt76/mt7603/core.c b/drivers/net/wireless/mediatek/mt76/mt7603/core.c index 1086dcd376a0cab0c1dd362ddd9c88771744d348..37e5644b45efa2f743a3d9597cd5cbad899c5fe0 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7603/core.c +++ b/drivers/net/wireless/mediatek/mt76/mt7603/core.c @@ -2,17 +2,6 @@ #include "mt7603.h" -void mt7603_set_irq_mask(struct mt7603_dev *dev, u32 clear, u32 set) -{ - unsigned long flags; - - spin_lock_irqsave(&dev->mt76.mmio.irq_lock, flags); - dev->mt76.mmio.irqmask &= ~clear; - dev->mt76.mmio.irqmask |= set; - mt76_wr(dev, MT_INT_MASK_CSR, dev->mt76.mmio.irqmask); - spin_unlock_irqrestore(&dev->mt76.mmio.irq_lock, flags); -} - void mt7603_rx_poll_complete(struct mt76_dev *mdev, enum mt76_rxq_id q) { struct mt7603_dev *dev = container_of(mdev, struct mt7603_dev, mt76); @@ -38,7 +27,7 @@ irqreturn_t mt7603_irq_handler(int irq, void *dev_instance) mt76_wr(dev, MT_HW_INT_STATUS(3), hwintr); if (hwintr & MT_HW_INT3_PRE_TBTT0) - tasklet_schedule(&dev->pre_tbtt_tasklet); + tasklet_schedule(&dev->mt76.pre_tbtt_tasklet); if ((hwintr & MT_HW_INT3_TBTT0) && dev->mt76.csa_complete) mt76_csa_finish(&dev->mt76); @@ -46,7 +35,7 @@ irqreturn_t mt7603_irq_handler(int irq, void *dev_instance) if (intr & MT_INT_TX_DONE_ALL) { mt7603_irq_disable(dev, MT_INT_TX_DONE_ALL); - tasklet_schedule(&dev->tx_tasklet); + tasklet_schedule(&dev->mt76.tx_tasklet); } if (intr & MT_INT_RX_DONE(0)) { @@ -64,8 +53,8 @@ irqreturn_t mt7603_irq_handler(int irq, void *dev_instance) u32 mt7603_reg_map(struct mt7603_dev *dev, u32 addr) { - u32 base = addr & GENMASK(31, 19); - u32 offset = addr & GENMASK(18, 0); + u32 base = addr & MT_MCU_PCIE_REMAP_2_BASE; + u32 offset = addr & MT_MCU_PCIE_REMAP_2_OFFSET; dev->bus_ops->wr(&dev->mt76, MT_MCU_PCIE_REMAP_2, base); diff --git a/drivers/net/wireless/mediatek/mt76/mt7603/dma.c b/drivers/net/wireless/mediatek/mt76/mt7603/dma.c index b3ae0aaea62a15b51b1ed2c23bc8857f88dec8c9..27e2d9f905533566af87ca04b7506789aefead8d 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7603/dma.c +++ b/drivers/net/wireless/mediatek/mt76/mt7603/dma.c @@ -5,18 +5,22 @@ #include "../dma.h" static int -mt7603_init_tx_queue(struct mt7603_dev *dev, struct mt76_queue *q, +mt7603_init_tx_queue(struct mt7603_dev *dev, struct mt76_sw_queue *q, int idx, int n_desc) { - int ret; + struct mt76_queue *hwq; + int err; - q->hw_idx = idx; - q->regs = dev->mt76.mmio.regs + MT_TX_RING_BASE + idx * MT_RING_SIZE; - q->ndesc = n_desc; + hwq = devm_kzalloc(dev->mt76.dev, sizeof(*hwq), GFP_KERNEL); + if (!hwq) + return -ENOMEM; - ret = mt76_queue_alloc(dev, q); - if (ret) - return ret; + err = mt76_queue_alloc(dev, hwq, idx, n_desc, 0, MT_TX_RING_BASE); + if (err < 0) + return err; + + INIT_LIST_HEAD(&q->swq); + q->q = hwq; mt7603_irq_enable(dev, MT_INT_TX_DONE(idx)); @@ -119,15 +123,12 @@ static int mt7603_init_rx_queue(struct mt7603_dev *dev, struct mt76_queue *q, int idx, int n_desc, int bufsize) { - int ret; + int err; - q->regs = dev->mt76.mmio.regs + MT_RX_RING_BASE + idx * MT_RING_SIZE; - q->ndesc = n_desc; - q->buf_size = bufsize; - - ret = mt76_queue_alloc(dev, q); - if (ret) - return ret; + err = mt76_queue_alloc(dev, q, idx, n_desc, bufsize, + MT_RX_RING_BASE); + if (err < 0) + return err; mt7603_irq_enable(dev, MT_INT_RX_DONE(idx)); @@ -144,6 +145,8 @@ mt7603_tx_tasklet(unsigned long data) for (i = MT_TXQ_MCU; i >= 0; i--) mt76_queue_tx_cleanup(dev, i, false); + mt76_txq_schedule_all(&dev->mt76); + mt7603_irq_enable(dev, MT_INT_TX_DONE_ALL); } @@ -163,7 +166,7 @@ int mt7603_dma_init(struct mt7603_dev *dev) init_waitqueue_head(&dev->mt76.mmio.mcu.wait); skb_queue_head_init(&dev->mt76.mmio.mcu.res_q); - tasklet_init(&dev->tx_tasklet, mt7603_tx_tasklet, (unsigned long)dev); + tasklet_init(&dev->mt76.tx_tasklet, mt7603_tx_tasklet, (unsigned long)dev); mt76_clear(dev, MT_WPDMA_GLO_CFG, MT_WPDMA_GLO_CFG_TX_DMA_EN | @@ -223,6 +226,6 @@ void mt7603_dma_cleanup(struct mt7603_dev *dev) MT_WPDMA_GLO_CFG_RX_DMA_EN | MT_WPDMA_GLO_CFG_TX_WRITEBACK_DONE); - tasklet_kill(&dev->tx_tasklet); + tasklet_kill(&dev->mt76.tx_tasklet); mt76_dma_cleanup(&dev->mt76); } diff --git a/drivers/net/wireless/mediatek/mt76/mt7603/init.c b/drivers/net/wireless/mediatek/mt76/mt7603/init.c index 3af45949e868909e3073335cc302411c5e6c9761..78cdbb70e1789a726c25a9a293f99422d792ffad 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7603/init.c +++ b/drivers/net/wireless/mediatek/mt76/mt7603/init.c @@ -167,7 +167,8 @@ mt7603_mac_init(struct mt7603_dev *dev) FIELD_PREP(MT_AGG_RETRY_CONTROL_BAR_LIMIT, 1) | FIELD_PREP(MT_AGG_RETRY_CONTROL_RTS_LIMIT, 15)); - mt76_rmw(dev, MT_DMA_DCR0, ~0xfffc, 4096); + mt76_wr(dev, MT_DMA_DCR0, MT_DMA_DCR0_RX_VEC_DROP | + FIELD_PREP(MT_DMA_DCR0_MAX_RX_LEN, 4096)); mt76_rmw(dev, MT_DMA_VCFR0, BIT(0), BIT(13)); mt76_rmw(dev, MT_DMA_TMCFR0, BIT(0) | BIT(1), BIT(13)); @@ -488,6 +489,7 @@ mt7603_init_txpower(struct mt7603_dev *dev, for (i = 0; i < sband->n_channels; i++) { chan = &sband->channels[i]; chan->max_power = target_power; + chan->orig_mpwr = target_power; } } @@ -512,8 +514,8 @@ int mt7603_register_device(struct mt7603_dev *dev) spin_lock_init(&dev->ps_lock); - INIT_DELAYED_WORK(&dev->mac_work, mt7603_mac_work); - tasklet_init(&dev->pre_tbtt_tasklet, mt7603_pre_tbtt_tasklet, + INIT_DELAYED_WORK(&dev->mt76.mac_work, mt7603_mac_work); + tasklet_init(&dev->mt76.pre_tbtt_tasklet, mt7603_pre_tbtt_tasklet, (unsigned long)dev); /* Check for 7688, which only has 1SS */ @@ -572,9 +574,9 @@ int mt7603_register_device(struct mt7603_dev *dev) void mt7603_unregister_device(struct mt7603_dev *dev) { - tasklet_disable(&dev->pre_tbtt_tasklet); + tasklet_disable(&dev->mt76.pre_tbtt_tasklet); mt76_unregister_device(&dev->mt76); mt7603_mcu_exit(dev); mt7603_dma_cleanup(dev); - ieee80211_free_hw(mt76_hw(dev)); + mt76_free_device(&dev->mt76); } diff --git a/drivers/net/wireless/mediatek/mt76/mt7603/mac.c b/drivers/net/wireless/mediatek/mt76/mt7603/mac.c index 5abc02b578185a6467571f549987dd147e2b3d3b..6d506e34c3eeb1d8370b26092023035260b34d13 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7603/mac.c +++ b/drivers/net/wireless/mediatek/mt76/mt7603/mac.c @@ -590,7 +590,7 @@ mt7603_mac_fill_rx(struct mt7603_dev *dev, struct sk_buff *skb) status->aggr = unicast && !ieee80211_is_qos_nullfunc(hdr->frame_control); status->tid = *ieee80211_get_qos_ctl(hdr) & IEEE80211_QOS_CTL_TID_MASK; - status->seqno = hdr->seq_ctrl >> 4; + status->seqno = IEEE80211_SEQ_TO_SN(le16_to_cpu(hdr->seq_ctrl)); return 0; } @@ -717,11 +717,11 @@ void mt7603_wtbl_set_rates(struct mt7603_dev *dev, struct mt7603_sta *sta, MT_WTBL_UPDATE_RATE_UPDATE | MT_WTBL_UPDATE_TX_COUNT_CLEAR); - if (!sta->wcid.tx_rate_set) + if (!(sta->wcid.tx_info & MT_WCID_TX_INFO_SET)) mt76_poll(dev, MT_WTBL_UPDATE, MT_WTBL_UPDATE_BUSY, 0, 5000); sta->rate_count = 2 * MT7603_RATE_RETRY * n_rates; - sta->wcid.tx_rate_set = true; + sta->wcid.tx_info |= MT_WCID_TX_INFO_SET; } static enum mt7603_cipher_type @@ -783,7 +783,7 @@ int mt7603_wtbl_set_key(struct mt7603_dev *dev, int wcid, static int mt7603_mac_write_txwi(struct mt7603_dev *dev, __le32 *txwi, - struct sk_buff *skb, struct mt76_queue *q, + struct sk_buff *skb, enum mt76_txq_id qid, struct mt76_wcid *wcid, struct ieee80211_sta *sta, int pid, struct ieee80211_key_conf *key) { @@ -792,6 +792,7 @@ mt7603_mac_write_txwi(struct mt7603_dev *dev, __le32 *txwi, struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)skb->data; struct ieee80211_bar *bar = (struct ieee80211_bar *)skb->data; struct ieee80211_vif *vif = info->control.vif; + struct mt76_queue *q = dev->mt76.q_tx[qid].q; struct mt7603_vif *mvif; int wlan_idx; int hdr_len = ieee80211_get_hdrlen_from_skb(skb); @@ -806,7 +807,7 @@ mt7603_mac_write_txwi(struct mt7603_dev *dev, __le32 *txwi, if (vif) { mvif = (struct mt7603_vif *)vif->drv_priv; vif_idx = mvif->idx; - if (vif_idx && q >= &dev->mt76.q_tx[MT_TXQ_BEACON]) + if (vif_idx && qid >= MT_TXQ_BEACON) vif_idx += 0x10; } @@ -880,7 +881,7 @@ mt7603_mac_write_txwi(struct mt7603_dev *dev, __le32 *txwi, } /* use maximum tx count for beacons and buffered multicast */ - if (q >= &dev->mt76.q_tx[MT_TXQ_BEACON]) + if (qid >= MT_TXQ_BEACON) tx_count = 0x1f; val = FIELD_PREP(MT_TXD3_REM_TX_COUNT, tx_count) | @@ -911,13 +912,13 @@ mt7603_mac_write_txwi(struct mt7603_dev *dev, __le32 *txwi, } int mt7603_tx_prepare_skb(struct mt76_dev *mdev, void *txwi_ptr, - struct sk_buff *skb, struct mt76_queue *q, - struct mt76_wcid *wcid, struct ieee80211_sta *sta, - u32 *tx_info) + enum mt76_txq_id qid, struct mt76_wcid *wcid, + struct ieee80211_sta *sta, + struct mt76_tx_info *tx_info) { struct mt7603_dev *dev = container_of(mdev, struct mt7603_dev, mt76); struct mt7603_sta *msta = container_of(wcid, struct mt7603_sta, wcid); - struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb); + struct ieee80211_tx_info *info = IEEE80211_SKB_CB(tx_info->skb); struct ieee80211_key_conf *key = info->control.hw_key; int pid; @@ -933,7 +934,7 @@ int mt7603_tx_prepare_skb(struct mt76_dev *mdev, void *txwi_ptr, mt7603_wtbl_set_ps(dev, msta, false); } - pid = mt76_tx_status_skb_add(mdev, wcid, skb); + pid = mt76_tx_status_skb_add(mdev, wcid, tx_info->skb); if (info->flags & IEEE80211_TX_CTL_RATE_CTRL_PROBE) { spin_lock_bh(&dev->mt76.lock); @@ -943,7 +944,8 @@ int mt7603_tx_prepare_skb(struct mt76_dev *mdev, void *txwi_ptr, spin_unlock_bh(&dev->mt76.lock); } - mt7603_mac_write_txwi(dev, txwi_ptr, skb, q, wcid, sta, pid, key); + mt7603_mac_write_txwi(dev, txwi_ptr, tx_info->skb, qid, wcid, + sta, pid, key); return 0; } @@ -1142,8 +1144,8 @@ out: rcu_read_unlock(); } -void mt7603_tx_complete_skb(struct mt76_dev *mdev, struct mt76_queue *q, - struct mt76_queue_entry *e, bool flush) +void mt7603_tx_complete_skb(struct mt76_dev *mdev, enum mt76_txq_id qid, + struct mt76_queue_entry *e) { struct mt7603_dev *dev = container_of(mdev, struct mt7603_dev, mt76); struct sk_buff *skb = e->skb; @@ -1153,7 +1155,7 @@ void mt7603_tx_complete_skb(struct mt76_dev *mdev, struct mt76_queue *q, return; } - if (q - dev->mt76.q_tx < 4) + if (qid < 4) dev->tx_hang_check = 0; mt76_tx_complete_skb(mdev, skb); @@ -1266,7 +1268,7 @@ static void mt7603_dma_sched_reset(struct mt7603_dev *dev) static void mt7603_mac_watchdog_reset(struct mt7603_dev *dev) { - int beacon_int = dev->beacon_int; + int beacon_int = dev->mt76.beacon_int; u32 mask = dev->mt76.mmio.irqmask; int i; @@ -1276,8 +1278,8 @@ static void mt7603_mac_watchdog_reset(struct mt7603_dev *dev) /* lock/unlock all queues to ensure that no tx is pending */ mt76_txq_schedule_all(&dev->mt76); - tasklet_disable(&dev->tx_tasklet); - tasklet_disable(&dev->pre_tbtt_tasklet); + tasklet_disable(&dev->mt76.tx_tasklet); + tasklet_disable(&dev->mt76.pre_tbtt_tasklet); napi_disable(&dev->mt76.napi[0]); napi_disable(&dev->mt76.napi[1]); @@ -1323,10 +1325,10 @@ skip_dma_reset: clear_bit(MT76_RESET, &dev->mt76.state); mutex_unlock(&dev->mt76.mutex); - tasklet_enable(&dev->tx_tasklet); - tasklet_schedule(&dev->tx_tasklet); + tasklet_enable(&dev->mt76.tx_tasklet); + tasklet_schedule(&dev->mt76.tx_tasklet); - tasklet_enable(&dev->pre_tbtt_tasklet); + tasklet_enable(&dev->mt76.pre_tbtt_tasklet); mt7603_beacon_set_timer(dev, -1, beacon_int); napi_enable(&dev->mt76.napi[0]); @@ -1385,17 +1387,17 @@ static bool mt7603_tx_hang(struct mt7603_dev *dev) int i; for (i = 0; i < 4; i++) { - q = &dev->mt76.q_tx[i]; + q = dev->mt76.q_tx[i].q; if (!q->queued) continue; prev_dma_idx = dev->tx_dma_idx[i]; - dma_idx = ioread32(&q->regs->dma_idx); + dma_idx = readl(&q->regs->dma_idx); dev->tx_dma_idx[i] = dma_idx; if (dma_idx == prev_dma_idx && - dma_idx != ioread32(&q->regs->cpu_idx)) + dma_idx != readl(&q->regs->cpu_idx)) break; } @@ -1666,7 +1668,7 @@ out: void mt7603_mac_work(struct work_struct *work) { struct mt7603_dev *dev = container_of(work, struct mt7603_dev, - mac_work.work); + mt76.mac_work.work); bool reset = false; mt76_tx_status_check(&dev->mt76, NULL, false); @@ -1719,6 +1721,6 @@ void mt7603_mac_work(struct work_struct *work) if (reset) mt7603_mac_watchdog_reset(dev); - ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mac_work, + ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mt76.mac_work, msecs_to_jiffies(MT7603_WATCHDOG_TIME)); } diff --git a/drivers/net/wireless/mediatek/mt76/mt7603/main.c b/drivers/net/wireless/mediatek/mt76/mt7603/main.c index a3c4ef198bfeea965fb3f8d71e9d622cc546bb1a..0a0334dc40d543b7a11f7beac596ce036da5a35d 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7603/main.c +++ b/drivers/net/wireless/mediatek/mt76/mt7603/main.c @@ -16,7 +16,7 @@ mt7603_start(struct ieee80211_hw *hw) mt7603_mac_start(dev); dev->survey_time = ktime_get_boottime(); set_bit(MT76_STATE_RUNNING, &dev->mt76.state); - mt7603_mac_work(&dev->mac_work.work); + mt7603_mac_work(&dev->mt76.mac_work.work); return 0; } @@ -27,7 +27,7 @@ mt7603_stop(struct ieee80211_hw *hw) struct mt7603_dev *dev = hw->priv; clear_bit(MT76_STATE_RUNNING, &dev->mt76.state); - cancel_delayed_work_sync(&dev->mac_work); + cancel_delayed_work_sync(&dev->mt76.mac_work); mt7603_mac_stop(dev); } @@ -132,11 +132,13 @@ mt7603_set_channel(struct mt7603_dev *dev, struct cfg80211_chan_def *def) u8 bw = MT_BW_20; bool failed = false; - cancel_delayed_work_sync(&dev->mac_work); + cancel_delayed_work_sync(&dev->mt76.mac_work); + tasklet_disable(&dev->mt76.pre_tbtt_tasklet); mutex_lock(&dev->mt76.mutex); set_bit(MT76_RESET, &dev->mt76.state); + mt7603_beacon_set_timer(dev, -1, 0); mt76_set_channel(&dev->mt76); mt7603_mac_stop(dev); @@ -171,7 +173,7 @@ mt7603_set_channel(struct mt7603_dev *dev, struct cfg80211_chan_def *def) mt76_txq_schedule_all(&dev->mt76); - ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mac_work, + ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mt76.mac_work, MT7603_WATCHDOG_TIME); /* reset channel stats */ @@ -186,10 +188,14 @@ mt7603_set_channel(struct mt7603_dev *dev, struct cfg80211_chan_def *def) mt7603_init_edcca(dev); out: + if (!(mt76_hw(dev)->conf.flags & IEEE80211_CONF_OFFCHANNEL)) + mt7603_beacon_set_timer(dev, -1, dev->mt76.beacon_int); mutex_unlock(&dev->mt76.mutex); + tasklet_enable(&dev->mt76.pre_tbtt_tasklet); + if (failed) - mt7603_mac_work(&dev->mac_work.work); + mt7603_mac_work(&dev->mt76.mac_work.work); return ret; } @@ -294,9 +300,9 @@ mt7603_bss_info_changed(struct ieee80211_hw *hw, struct ieee80211_vif *vif, if (changed & (BSS_CHANGED_BEACON_ENABLED | BSS_CHANGED_BEACON_INT)) { int beacon_int = !!info->enable_beacon * info->beacon_int; - tasklet_disable(&dev->pre_tbtt_tasklet); + tasklet_disable(&dev->mt76.pre_tbtt_tasklet); mt7603_beacon_set_timer(dev, mvif->idx, beacon_int); - tasklet_enable(&dev->pre_tbtt_tasklet); + tasklet_enable(&dev->mt76.pre_tbtt_tasklet); } mutex_unlock(&dev->mt76.mutex); @@ -492,7 +498,7 @@ mt7603_conf_tx(struct ieee80211_hw *hw, struct ieee80211_vif *vif, u16 queue, u16 cw_max = (1 << 10) - 1; u32 val; - queue = dev->mt76.q_tx[queue].hw_idx; + queue = dev->mt76.q_tx[queue].q->hw_idx; if (params->cw_min) cw_min = params->cw_min; @@ -535,7 +541,6 @@ mt7603_sw_scan(struct ieee80211_hw *hw, struct ieee80211_vif *vif, struct mt7603_dev *dev = hw->priv; set_bit(MT76_SCANNING, &dev->mt76.state); - mt7603_beacon_set_timer(dev, -1, 0); } static void @@ -544,7 +549,6 @@ mt7603_sw_scan_complete(struct ieee80211_hw *hw, struct ieee80211_vif *vif) struct mt7603_dev *dev = hw->priv; clear_bit(MT76_SCANNING, &dev->mt76.state); - mt7603_beacon_set_timer(dev, -1, dev->beacon_int); } static void @@ -593,7 +597,7 @@ mt7603_ampdu_action(struct ieee80211_hw *hw, struct ieee80211_vif *vif, mt7603_mac_tx_ba_reset(dev, msta->wcid.idx, tid, -1); break; case IEEE80211_AMPDU_TX_START: - mtxq->agg_ssn = *ssn << 4; + mtxq->agg_ssn = IEEE80211_SN_TO_SEQ(*ssn); ieee80211_start_tx_ba_cb_irqsafe(vif, sta->addr, tid); break; case IEEE80211_AMPDU_TX_STOP_CONT: @@ -664,12 +668,6 @@ static void mt7603_tx(struct ieee80211_hw *hw, struct ieee80211_tx_control *cont mt76_tx(&dev->mt76, control->sta, wcid, skb); } -static int -mt7603_set_tim(struct ieee80211_hw *hw, struct ieee80211_sta *sta, bool set) -{ - return 0; -} - const struct ieee80211_ops mt7603_ops = { .tx = mt7603_tx, .start = mt7603_start, @@ -691,7 +689,7 @@ const struct ieee80211_ops mt7603_ops = { .sta_rate_tbl_update = mt7603_sta_rate_tbl_update, .release_buffered_frames = mt7603_release_buffered_frames, .set_coverage_class = mt7603_set_coverage_class, - .set_tim = mt7603_set_tim, + .set_tim = mt76_set_tim, .get_survey = mt76_get_survey, }; diff --git a/drivers/net/wireless/mediatek/mt76/mt7603/mcu.c b/drivers/net/wireless/mediatek/mt76/mt7603/mcu.c index d06905ea8cc63f2b9851eb679655007c1e8f7486..6357b5658a32e332ef7a377ef859574b1de81436 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7603/mcu.c +++ b/drivers/net/wireless/mediatek/mt76/mt7603/mcu.c @@ -14,17 +14,14 @@ struct mt7603_fw_trailer { } __packed; static int -__mt7603_mcu_msg_send(struct mt7603_dev *dev, struct sk_buff *skb, int cmd, - int query, int *wait_seq) +__mt7603_mcu_msg_send(struct mt7603_dev *dev, struct sk_buff *skb, + int cmd, int *wait_seq) { int hdrlen = dev->mcu_running ? sizeof(struct mt7603_mcu_txd) : 12; struct mt76_dev *mdev = &dev->mt76; struct mt7603_mcu_txd *txd; u8 seq; - if (!skb) - return -EINVAL; - seq = ++mdev->mmio.mcu.msg_seq & 0xf; if (!seq) seq = ++mdev->mmio.mcu.msg_seq & 0xf; @@ -42,15 +39,14 @@ __mt7603_mcu_msg_send(struct mt7603_dev *dev, struct sk_buff *skb, int cmd, if (cmd < 0) { txd->cid = -cmd; + txd->set_query = MCU_Q_NA; } else { txd->cid = MCU_CMD_EXT_CID; txd->ext_cid = cmd; - if (query != MCU_Q_NA) - txd->ext_cid_ack = 1; + txd->set_query = MCU_Q_SET; + txd->ext_cid_ack = 1; } - txd->set_query = query; - if (wait_seq) *wait_seq = seq; @@ -58,21 +54,26 @@ __mt7603_mcu_msg_send(struct mt7603_dev *dev, struct sk_buff *skb, int cmd, } static int -mt7603_mcu_msg_send(struct mt7603_dev *dev, struct sk_buff *skb, int cmd, - int query) +mt7603_mcu_msg_send(struct mt76_dev *mdev, int cmd, const void *data, + int len, bool wait_resp) { - struct mt76_dev *mdev = &dev->mt76; + struct mt7603_dev *dev = container_of(mdev, struct mt7603_dev, mt76); unsigned long expires = jiffies + 3 * HZ; struct mt7603_mcu_rxd *rxd; + struct sk_buff *skb; int ret, seq; + skb = mt7603_mcu_msg_alloc(data, len); + if (!skb) + return -ENOMEM; + mutex_lock(&mdev->mmio.mcu.mutex); - ret = __mt7603_mcu_msg_send(dev, skb, cmd, query, &seq); + ret = __mt7603_mcu_msg_send(dev, skb, cmd, &seq); if (ret) goto out; - while (1) { + while (wait_resp) { bool check_seq = false; skb = mt76_mcu_get_response(&dev->mt76, expires); @@ -113,28 +114,22 @@ mt7603_mcu_init_download(struct mt7603_dev *dev, u32 addr, u32 len) .len = cpu_to_le32(len), .mode = cpu_to_le32(BIT(31)), }; - struct sk_buff *skb = mt7603_mcu_msg_alloc(&req, sizeof(req)); - return mt7603_mcu_msg_send(dev, skb, -MCU_CMD_TARGET_ADDRESS_LEN_REQ, - MCU_Q_NA); + return __mt76_mcu_send_msg(&dev->mt76, -MCU_CMD_TARGET_ADDRESS_LEN_REQ, + &req, sizeof(req), true); } static int mt7603_mcu_send_firmware(struct mt7603_dev *dev, const void *data, int len) { - struct sk_buff *skb; - int ret = 0; + int cur_len, ret = 0; while (len > 0) { - int cur_len = min_t(int, 4096 - sizeof(struct mt7603_mcu_txd), - len); - - skb = mt7603_mcu_msg_alloc(data, cur_len); - if (!skb) - return -ENOMEM; + cur_len = min_t(int, 4096 - sizeof(struct mt7603_mcu_txd), + len); - ret = __mt7603_mcu_msg_send(dev, skb, -MCU_CMD_FW_SCATTER, - MCU_Q_NA, NULL); + ret = __mt76_mcu_send_msg(&dev->mt76, -MCU_CMD_FW_SCATTER, + data, cur_len, false); if (ret) break; @@ -155,23 +150,19 @@ mt7603_mcu_start_firmware(struct mt7603_dev *dev, u32 addr) .override = cpu_to_le32(addr ? 1 : 0), .addr = cpu_to_le32(addr), }; - struct sk_buff *skb = mt7603_mcu_msg_alloc(&req, sizeof(req)); - return mt7603_mcu_msg_send(dev, skb, -MCU_CMD_FW_START_REQ, - MCU_Q_NA); + return __mt76_mcu_send_msg(&dev->mt76, -MCU_CMD_FW_START_REQ, + &req, sizeof(req), true); } static int -mt7603_mcu_restart(struct mt7603_dev *dev) +mt7603_mcu_restart(struct mt76_dev *dev) { - struct sk_buff *skb = mt7603_mcu_msg_alloc(NULL, 0); - - return mt7603_mcu_msg_send(dev, skb, -MCU_CMD_RESTART_DL_REQ, - MCU_Q_NA); + return __mt76_mcu_send_msg(dev, -MCU_CMD_RESTART_DL_REQ, + NULL, 0, true); } -static int -mt7603_load_firmware(struct mt7603_dev *dev) +static int mt7603_load_firmware(struct mt7603_dev *dev) { const struct firmware *fw; const struct mt7603_fw_trailer *hdr; @@ -261,6 +252,9 @@ running: mt76_clear(dev, MT_SCH_4, BIT(8)); dev->mcu_running = true; + snprintf(dev->mt76.hw->wiphy->fw_version, + sizeof(dev->mt76.hw->wiphy->fw_version), + "%.10s-%.15s", hdr->fw_ver, hdr->build_date); dev_info(dev->mt76.dev, "firmware init done\n"); out: @@ -271,14 +265,18 @@ out: int mt7603_mcu_init(struct mt7603_dev *dev) { - mutex_init(&dev->mt76.mmio.mcu.mutex); + static const struct mt76_mcu_ops mt7603_mcu_ops = { + .mcu_send_msg = mt7603_mcu_msg_send, + .mcu_restart = mt7603_mcu_restart, + }; + dev->mt76.mcu_ops = &mt7603_mcu_ops; return mt7603_load_firmware(dev); } void mt7603_mcu_exit(struct mt7603_dev *dev) { - mt7603_mcu_restart(dev); + __mt76_mcu_restart(&dev->mt76); skb_queue_purge(&dev->mt76.mmio.mcu.res_q); } @@ -360,27 +358,30 @@ int mt7603_mcu_set_eeprom(struct mt7603_dev *dev) .buffer_mode = 1, .len = ARRAY_SIZE(req_fields) - 1, }; - struct sk_buff *skb; - struct req_data *data; const int size = 0xff * sizeof(struct req_data); - u8 *eep = (u8 *)dev->mt76.eeprom.data; - int i; + u8 *req, *eep = (u8 *)dev->mt76.eeprom.data; + int i, ret, len = sizeof(req_hdr) + size; + struct req_data *data; BUILD_BUG_ON(ARRAY_SIZE(req_fields) * sizeof(*data) > size); - skb = mt7603_mcu_msg_alloc(NULL, size + sizeof(req_hdr)); - memcpy(skb_put(skb, sizeof(req_hdr)), &req_hdr, sizeof(req_hdr)); - data = (struct req_data *)skb_put(skb, size); - memset(data, 0, size); + req = kmalloc(len, GFP_KERNEL); + if (!req) + return -ENOMEM; + memcpy(req, &req_hdr, sizeof(req_hdr)); + data = (struct req_data *)(req + sizeof(req_hdr)); + memset(data, 0, size); for (i = 0; i < ARRAY_SIZE(req_fields); i++) { data[i].addr = cpu_to_le16(req_fields[i]); data[i].val = eep[req_fields[i]]; - data[i].pad = 0; } - return mt7603_mcu_msg_send(dev, skb, MCU_EXT_CMD_EFUSE_BUFFER_MODE, - MCU_Q_SET); + ret = __mt76_mcu_send_msg(&dev->mt76, MCU_EXT_CMD_EFUSE_BUFFER_MODE, + req, len, true); + kfree(req); + + return ret; } static int mt7603_mcu_set_tx_power(struct mt7603_dev *dev) @@ -415,7 +416,6 @@ static int mt7603_mcu_set_tx_power(struct mt7603_dev *dev) }, #undef EEP_VAL }; - struct sk_buff *skb; u8 *eep = (u8 *)dev->mt76.eeprom.data; memcpy(req.rate_power_delta, eep + MT_EE_TX_POWER_CCK, @@ -424,9 +424,8 @@ static int mt7603_mcu_set_tx_power(struct mt7603_dev *dev) memcpy(req.temp_comp_power, eep + MT_EE_STEP_NUM_NEG_6_7, sizeof(req.temp_comp_power)); - skb = mt7603_mcu_msg_alloc(&req, sizeof(req)); - return mt7603_mcu_msg_send(dev, skb, MCU_EXT_CMD_SET_TX_POWER_CTRL, - MCU_Q_SET); + return __mt76_mcu_send_msg(&dev->mt76, MCU_EXT_CMD_SET_TX_POWER_CTRL, + &req, sizeof(req), true); } int mt7603_mcu_set_channel(struct mt7603_dev *dev) @@ -450,10 +449,8 @@ int mt7603_mcu_set_channel(struct mt7603_dev *dev) .tx_streams = n_chains, .rx_streams = n_chains, }; - struct sk_buff *skb; s8 tx_power; - int ret; - int i; + int i, ret; if (dev->mt76.chandef.width == NL80211_CHAN_WIDTH_40) { req.bw = MT_BW_40; @@ -473,9 +470,8 @@ int mt7603_mcu_set_channel(struct mt7603_dev *dev) for (i = 0; i < ARRAY_SIZE(req.txpower); i++) req.txpower[i] = tx_power; - skb = mt7603_mcu_msg_alloc(&req, sizeof(req)); - ret = mt7603_mcu_msg_send(dev, skb, MCU_EXT_CMD_CHANNEL_SWITCH, - MCU_Q_SET); + ret = __mt76_mcu_send_msg(&dev->mt76, MCU_EXT_CMD_CHANNEL_SWITCH, + &req, sizeof(req), true); if (ret) return ret; diff --git a/drivers/net/wireless/mediatek/mt76/mt7603/mt7603.h b/drivers/net/wireless/mediatek/mt76/mt7603/mt7603.h index 6049f3b7c8fec429de86329d35662c4659f711ee..fa64bbaab0d2c7e74bbaaa7907bc3e1c9adc3708 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7603/mt7603.h +++ b/drivers/net/wireless/mediatek/mt76/mt7603/mt7603.h @@ -109,7 +109,6 @@ struct mt7603_dev { ktime_t survey_time; ktime_t ed_time; - int beacon_int; struct mt76_queue q_rx; @@ -126,8 +125,6 @@ struct mt7603_dev { s8 sensitivity; - u8 beacon_mask; - u8 beacon_check; u8 tx_hang_check; u8 tx_dma_check; @@ -143,10 +140,6 @@ struct mt7603_dev { u32 reset_test; unsigned int reset_cause[__RESET_CAUSE_MAX]; - - struct delayed_work mac_work; - struct tasklet_struct tx_tasklet; - struct tasklet_struct pre_tbtt_tasklet; }; extern const struct mt76_driver_ops mt7603_drv_ops; @@ -179,16 +172,14 @@ void mt7603_dma_cleanup(struct mt7603_dev *dev); int mt7603_mcu_init(struct mt7603_dev *dev); void mt7603_init_debugfs(struct mt7603_dev *dev); -void mt7603_set_irq_mask(struct mt7603_dev *dev, u32 clear, u32 set); - static inline void mt7603_irq_enable(struct mt7603_dev *dev, u32 mask) { - mt7603_set_irq_mask(dev, 0, mask); + mt76_set_irq_mask(&dev->mt76, MT_INT_MASK_CSR, 0, mask); } static inline void mt7603_irq_disable(struct mt7603_dev *dev, u32 mask) { - mt7603_set_irq_mask(dev, mask, 0); + mt76_set_irq_mask(&dev->mt76, MT_INT_MASK_CSR, mask, 0); } void mt7603_mac_dma_start(struct mt7603_dev *dev); @@ -225,12 +216,12 @@ void mt7603_wtbl_set_smps(struct mt7603_dev *dev, struct mt7603_sta *sta, void mt7603_filter_tx(struct mt7603_dev *dev, int idx, bool abort); int mt7603_tx_prepare_skb(struct mt76_dev *mdev, void *txwi_ptr, - struct sk_buff *skb, struct mt76_queue *q, - struct mt76_wcid *wcid, struct ieee80211_sta *sta, - u32 *tx_info); + enum mt76_txq_id qid, struct mt76_wcid *wcid, + struct ieee80211_sta *sta, + struct mt76_tx_info *tx_info); -void mt7603_tx_complete_skb(struct mt76_dev *mdev, struct mt76_queue *q, - struct mt76_queue_entry *e, bool flush); +void mt7603_tx_complete_skb(struct mt76_dev *mdev, enum mt76_txq_id qid, + struct mt76_queue_entry *e); void mt7603_queue_rx_skb(struct mt76_dev *mdev, enum mt76_rxq_id q, struct sk_buff *skb); diff --git a/drivers/net/wireless/mediatek/mt76/mt7603/regs.h b/drivers/net/wireless/mediatek/mt76/mt7603/regs.h index da6827ae6cee24ebef6bd6492fa26c4122efbe7a..9d257d5c309d10c1e6a7c2f99dd8b163460e5dd2 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7603/regs.h +++ b/drivers/net/wireless/mediatek/mt76/mt7603/regs.h @@ -233,6 +233,10 @@ #define MT_WF_DMA(ofs) (MT_WF_DMA_BASE + (ofs)) #define MT_DMA_DCR0 MT_WF_DMA(0x000) +#define MT_DMA_DCR0_MAX_RX_LEN GENMASK(15, 0) +#define MT_DMA_DCR0_DAMSDU BIT(16) +#define MT_DMA_DCR0_RX_VEC_DROP BIT(17) + #define MT_DMA_DCR1 MT_WF_DMA(0x004) #define MT_DMA_FQCR0 MT_WF_DMA(0x008) diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/Kconfig b/drivers/net/wireless/mediatek/mt76/mt7615/Kconfig new file mode 100644 index 0000000000000000000000000000000000000000..3b8aba09bd5e4d8f60ca062f8d46d2a0ac4169e9 --- /dev/null +++ b/drivers/net/wireless/mediatek/mt76/mt7615/Kconfig @@ -0,0 +1,7 @@ +config MT7615E + tristate "MediaTek MT7615E (PCIe) support" + select MT76_CORE + depends on MAC80211 + depends on PCI + help + This adds support for MT7615-based wireless PCIe devices. diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/Makefile b/drivers/net/wireless/mediatek/mt76/mt7615/Makefile new file mode 100644 index 0000000000000000000000000000000000000000..6397552f6ee311aa174d4a6628370339efab422c --- /dev/null +++ b/drivers/net/wireless/mediatek/mt76/mt7615/Makefile @@ -0,0 +1,5 @@ +#SPDX-License-Identifier: ISC + +obj-$(CONFIG_MT7615E) += mt7615e.o + +mt7615e-y := pci.o init.o dma.o eeprom.o main.o mcu.o mac.o diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/dma.c b/drivers/net/wireless/mediatek/mt76/mt7615/dma.c new file mode 100644 index 0000000000000000000000000000000000000000..3ec6582afd8f5f40cad36455bd1c3116edaf47e2 --- /dev/null +++ b/drivers/net/wireless/mediatek/mt76/mt7615/dma.c @@ -0,0 +1,205 @@ +// SPDX-License-Identifier: ISC +/* Copyright (C) 2019 MediaTek Inc. + * + * Author: Ryder Lee + * Roy Luo + * Lorenzo Bianconi + * Felix Fietkau + */ + +#include "mt7615.h" +#include "../dma.h" +#include "mac.h" + +static int +mt7615_init_tx_queues(struct mt7615_dev *dev, int n_desc) +{ + struct mt76_sw_queue *q; + struct mt76_queue *hwq; + int err, i; + + hwq = devm_kzalloc(dev->mt76.dev, sizeof(*hwq), GFP_KERNEL); + if (!hwq) + return -ENOMEM; + + err = mt76_queue_alloc(dev, hwq, 0, n_desc, 0, MT_TX_RING_BASE); + if (err < 0) + return err; + + for (i = 0; i < MT_TXQ_MCU; i++) { + q = &dev->mt76.q_tx[i]; + INIT_LIST_HEAD(&q->swq); + q->q = hwq; + } + + return 0; +} + +static int +mt7615_init_mcu_queue(struct mt7615_dev *dev, struct mt76_sw_queue *q, + int idx, int n_desc) +{ + struct mt76_queue *hwq; + int err; + + hwq = devm_kzalloc(dev->mt76.dev, sizeof(*hwq), GFP_KERNEL); + if (!hwq) + return -ENOMEM; + + err = mt76_queue_alloc(dev, hwq, idx, n_desc, 0, MT_TX_RING_BASE); + if (err < 0) + return err; + + INIT_LIST_HEAD(&q->swq); + q->q = hwq; + + return 0; +} + +void mt7615_queue_rx_skb(struct mt76_dev *mdev, enum mt76_rxq_id q, + struct sk_buff *skb) +{ + struct mt7615_dev *dev = container_of(mdev, struct mt7615_dev, mt76); + __le32 *rxd = (__le32 *)skb->data; + __le32 *end = (__le32 *)&skb->data[skb->len]; + enum rx_pkt_type type; + + type = FIELD_GET(MT_RXD0_PKT_TYPE, le32_to_cpu(rxd[0])); + + switch (type) { + case PKT_TYPE_TXS: + for (rxd++; rxd + 7 <= end; rxd += 7) + mt7615_mac_add_txs(dev, rxd); + dev_kfree_skb(skb); + break; + case PKT_TYPE_TXRX_NOTIFY: + mt7615_mac_tx_free(dev, skb); + break; + case PKT_TYPE_RX_EVENT: + mt76_mcu_rx_event(&dev->mt76, skb); + break; + case PKT_TYPE_NORMAL: + if (!mt7615_mac_fill_rx(dev, skb)) { + mt76_rx(&dev->mt76, q, skb); + return; + } + /* fall through */ + default: + dev_kfree_skb(skb); + break; + } +} + +static void mt7615_tx_tasklet(unsigned long data) +{ + struct mt7615_dev *dev = (struct mt7615_dev *)data; + static const u8 queue_map[] = { + MT_TXQ_MCU, + MT_TXQ_BE + }; + int i; + + for (i = 0; i < ARRAY_SIZE(queue_map); i++) + mt76_queue_tx_cleanup(dev, queue_map[i], false); + + mt76_txq_schedule_all(&dev->mt76); + + mt7615_irq_enable(dev, MT_INT_TX_DONE_ALL); +} + +int mt7615_dma_init(struct mt7615_dev *dev) +{ + int ret; + + mt76_dma_attach(&dev->mt76); + + tasklet_init(&dev->mt76.tx_tasklet, mt7615_tx_tasklet, + (unsigned long)dev); + + mt76_wr(dev, MT_WPDMA_GLO_CFG, + MT_WPDMA_GLO_CFG_TX_WRITEBACK_DONE | + MT_WPDMA_GLO_CFG_FIFO_LITTLE_ENDIAN | + MT_WPDMA_GLO_CFG_FIRST_TOKEN_ONLY | + MT_WPDMA_GLO_CFG_OMIT_TX_INFO); + + mt76_rmw_field(dev, MT_WPDMA_GLO_CFG, + MT_WPDMA_GLO_CFG_TX_BT_SIZE_BIT0, 0x1); + + mt76_rmw_field(dev, MT_WPDMA_GLO_CFG, + MT_WPDMA_GLO_CFG_TX_BT_SIZE_BIT21, 0x1); + + mt76_rmw_field(dev, MT_WPDMA_GLO_CFG, + MT_WPDMA_GLO_CFG_DMA_BURST_SIZE, 0x3); + + mt76_rmw_field(dev, MT_WPDMA_GLO_CFG, + MT_WPDMA_GLO_CFG_MULTI_DMA_EN, 0x3); + + mt76_wr(dev, MT_WPDMA_GLO_CFG1, 0x1); + mt76_wr(dev, MT_WPDMA_TX_PRE_CFG, 0xf0000); + mt76_wr(dev, MT_WPDMA_RX_PRE_CFG, 0xf7f0000); + mt76_wr(dev, MT_WPDMA_ABT_CFG, 0x4000026); + mt76_wr(dev, MT_WPDMA_ABT_CFG1, 0x18811881); + mt76_set(dev, 0x7158, BIT(16)); + mt76_clear(dev, 0x7000, BIT(23)); + mt76_wr(dev, MT_WPDMA_RST_IDX, ~0); + + ret = mt7615_init_tx_queues(dev, MT7615_TX_RING_SIZE); + if (ret) + return ret; + + ret = mt7615_init_mcu_queue(dev, &dev->mt76.q_tx[MT_TXQ_MCU], + MT7615_TXQ_MCU, + MT7615_TX_MCU_RING_SIZE); + if (ret) + return ret; + + ret = mt7615_init_mcu_queue(dev, &dev->mt76.q_tx[MT_TXQ_FWDL], + MT7615_TXQ_FWDL, + MT7615_TX_FWDL_RING_SIZE); + if (ret) + return ret; + + /* init rx queues */ + ret = mt76_queue_alloc(dev, &dev->mt76.q_rx[MT_RXQ_MCU], 1, + MT7615_RX_MCU_RING_SIZE, MT_RX_BUF_SIZE, + MT_RX_RING_BASE); + if (ret) + return ret; + + ret = mt76_queue_alloc(dev, &dev->mt76.q_rx[MT_RXQ_MAIN], 0, + MT7615_RX_RING_SIZE, MT_RX_BUF_SIZE, + MT_RX_RING_BASE); + if (ret) + return ret; + + mt76_wr(dev, MT_DELAY_INT_CFG, 0); + + ret = mt76_init_queues(dev); + if (ret < 0) + return ret; + + mt76_poll(dev, MT_WPDMA_GLO_CFG, + MT_WPDMA_GLO_CFG_TX_DMA_BUSY | + MT_WPDMA_GLO_CFG_RX_DMA_BUSY, 0, 1000); + + /* start dma engine */ + mt76_set(dev, MT_WPDMA_GLO_CFG, + MT_WPDMA_GLO_CFG_TX_DMA_EN | + MT_WPDMA_GLO_CFG_RX_DMA_EN); + + /* enable interrupts for TX/RX rings */ + mt7615_irq_enable(dev, MT_INT_RX_DONE_ALL | MT_INT_TX_DONE_ALL); + + return 0; +} + +void mt7615_dma_cleanup(struct mt7615_dev *dev) +{ + mt76_clear(dev, MT_WPDMA_GLO_CFG, + MT_WPDMA_GLO_CFG_TX_DMA_EN | + MT_WPDMA_GLO_CFG_RX_DMA_EN); + mt76_set(dev, MT_WPDMA_GLO_CFG, MT_WPDMA_GLO_CFG_SW_RESET); + + tasklet_kill(&dev->mt76.tx_tasklet); + mt76_dma_cleanup(&dev->mt76); +} diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/eeprom.c b/drivers/net/wireless/mediatek/mt76/mt7615/eeprom.c new file mode 100644 index 0000000000000000000000000000000000000000..dd5ab46a4f662b91e2bfe52523c2d7371b6a8507 --- /dev/null +++ b/drivers/net/wireless/mediatek/mt76/mt7615/eeprom.c @@ -0,0 +1,98 @@ +// SPDX-License-Identifier: ISC +/* Copyright (C) 2019 MediaTek Inc. + * + * Author: Ryder Lee + * Felix Fietkau + */ + +#include "mt7615.h" +#include "eeprom.h" + +static int mt7615_efuse_read(struct mt7615_dev *dev, u32 base, + u16 addr, u8 *data) +{ + u32 val; + int i; + + val = mt76_rr(dev, base + MT_EFUSE_CTRL); + val &= ~(MT_EFUSE_CTRL_AIN | MT_EFUSE_CTRL_MODE); + val |= FIELD_PREP(MT_EFUSE_CTRL_AIN, addr & ~0xf); + val |= MT_EFUSE_CTRL_KICK; + mt76_wr(dev, base + MT_EFUSE_CTRL, val); + + if (!mt76_poll(dev, base + MT_EFUSE_CTRL, MT_EFUSE_CTRL_KICK, 0, 1000)) + return -ETIMEDOUT; + + udelay(2); + + val = mt76_rr(dev, base + MT_EFUSE_CTRL); + if ((val & MT_EFUSE_CTRL_AOUT) == MT_EFUSE_CTRL_AOUT || + WARN_ON_ONCE(!(val & MT_EFUSE_CTRL_VALID))) { + memset(data, 0x0, 16); + return 0; + } + + for (i = 0; i < 4; i++) { + val = mt76_rr(dev, base + MT_EFUSE_RDATA(i)); + put_unaligned_le32(val, data + 4 * i); + } + + return 0; +} + +static int mt7615_efuse_init(struct mt7615_dev *dev) +{ + u32 base = mt7615_reg_map(dev, MT_EFUSE_BASE); + int len = MT7615_EEPROM_SIZE; + int ret, i; + void *buf; + + if (mt76_rr(dev, base + MT_EFUSE_BASE_CTRL) & MT_EFUSE_BASE_CTRL_EMPTY) + return -EINVAL; + + dev->mt76.otp.data = devm_kzalloc(dev->mt76.dev, len, GFP_KERNEL); + dev->mt76.otp.size = len; + if (!dev->mt76.otp.data) + return -ENOMEM; + + buf = dev->mt76.otp.data; + for (i = 0; i + 16 <= len; i += 16) { + ret = mt7615_efuse_read(dev, base, i, buf + i); + if (ret) + return ret; + } + + return 0; +} + +static int mt7615_eeprom_load(struct mt7615_dev *dev) +{ + int ret; + + ret = mt76_eeprom_init(&dev->mt76, MT7615_EEPROM_SIZE); + if (ret < 0) + return ret; + + return mt7615_efuse_init(dev); +} + +int mt7615_eeprom_init(struct mt7615_dev *dev) +{ + int ret; + + ret = mt7615_eeprom_load(dev); + if (ret < 0) + return ret; + + memcpy(dev->mt76.eeprom.data, dev->mt76.otp.data, MT7615_EEPROM_SIZE); + + dev->mt76.cap.has_2ghz = true; + dev->mt76.cap.has_5ghz = true; + + memcpy(dev->mt76.macaddr, dev->mt76.eeprom.data + MT_EE_MAC_ADDR, + ETH_ALEN); + + mt76_eeprom_override(&dev->mt76); + + return 0; +} diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/eeprom.h b/drivers/net/wireless/mediatek/mt76/mt7615/eeprom.h new file mode 100644 index 0000000000000000000000000000000000000000..a4cf166881717e6da5ffac969d4079acf849a7d7 --- /dev/null +++ b/drivers/net/wireless/mediatek/mt76/mt7615/eeprom.h @@ -0,0 +1,18 @@ +/* SPDX-License-Identifier: ISC */ +/* Copyright (C) 2019 MediaTek Inc. */ + +#ifndef __MT7615_EEPROM_H +#define __MT7615_EEPROM_H + +#include "mt7615.h" + +enum mt7615_eeprom_field { + MT_EE_CHIP_ID = 0x000, + MT_EE_VERSION = 0x002, + MT_EE_MAC_ADDR = 0x004, + MT_EE_NIC_CONF_0 = 0x034, + + __MT_EE_MAX = 0x3bf +}; + +#endif diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/init.c b/drivers/net/wireless/mediatek/mt76/mt7615/init.c new file mode 100644 index 0000000000000000000000000000000000000000..3ab3ff553ef2d547b2dca1f661270c62a6bb163a --- /dev/null +++ b/drivers/net/wireless/mediatek/mt76/mt7615/init.c @@ -0,0 +1,229 @@ +// SPDX-License-Identifier: ISC +/* Copyright (C) 2019 MediaTek Inc. + * + * Author: Roy Luo + * Ryder Lee + * Felix Fietkau + */ + +#include +#include "mt7615.h" +#include "mac.h" + +static void mt7615_phy_init(struct mt7615_dev *dev) +{ + /* disable band 0 rf low power beacon mode */ + mt76_rmw(dev, MT_WF_PHY_WF2_RFCTRL0, MT_WF_PHY_WF2_RFCTRL0_LPBCN_EN, + MT_WF_PHY_WF2_RFCTRL0_LPBCN_EN); +} + +static void mt7615_mac_init(struct mt7615_dev *dev) +{ + /* enable band 0 clk */ + mt76_rmw(dev, MT_CFG_CCR, + MT_CFG_CCR_MAC_D0_1X_GC_EN | MT_CFG_CCR_MAC_D0_2X_GC_EN, + MT_CFG_CCR_MAC_D0_1X_GC_EN | MT_CFG_CCR_MAC_D0_2X_GC_EN); + + mt76_rmw_field(dev, MT_TMAC_CTCR0, + MT_TMAC_CTCR0_INS_DDLMT_REFTIME, 0x3f); + mt76_rmw_field(dev, MT_TMAC_CTCR0, + MT_TMAC_CTCR0_INS_DDLMT_DENSITY, 0x3); + mt76_rmw(dev, MT_TMAC_CTCR0, + MT_TMAC_CTCR0_INS_DDLMT_VHT_SMPDU_EN | + MT_TMAC_CTCR0_INS_DDLMT_EN, + MT_TMAC_CTCR0_INS_DDLMT_VHT_SMPDU_EN | + MT_TMAC_CTCR0_INS_DDLMT_EN); + + mt7615_mcu_set_rts_thresh(dev, 0x92b); + + mt76_rmw(dev, MT_AGG_SCR, MT_AGG_SCR_NLNAV_MID_PTEC_DIS, + MT_AGG_SCR_NLNAV_MID_PTEC_DIS); + + mt7615_mcu_init_mac(dev); + + mt76_wr(dev, MT_DMA_DCR0, MT_DMA_DCR0_RX_VEC_DROP | + FIELD_PREP(MT_DMA_DCR0_MAX_RX_LEN, 3072)); + + mt76_wr(dev, MT_AGG_ARUCR, FIELD_PREP(MT_AGG_ARxCR_LIMIT(0), 7)); + mt76_wr(dev, MT_AGG_ARDCR, + FIELD_PREP(MT_AGG_ARxCR_LIMIT(0), 0) | + FIELD_PREP(MT_AGG_ARxCR_LIMIT(1), + max_t(int, 0, MT7615_RATE_RETRY - 2)) | + FIELD_PREP(MT_AGG_ARxCR_LIMIT(2), MT7615_RATE_RETRY - 1) | + FIELD_PREP(MT_AGG_ARxCR_LIMIT(3), MT7615_RATE_RETRY - 1) | + FIELD_PREP(MT_AGG_ARxCR_LIMIT(4), MT7615_RATE_RETRY - 1) | + FIELD_PREP(MT_AGG_ARxCR_LIMIT(5), MT7615_RATE_RETRY - 1) | + FIELD_PREP(MT_AGG_ARxCR_LIMIT(6), MT7615_RATE_RETRY - 1) | + FIELD_PREP(MT_AGG_ARxCR_LIMIT(7), MT7615_RATE_RETRY - 1)); + + mt76_wr(dev, MT_AGG_ARCR, + (MT_AGG_ARCR_INIT_RATE1 | + FIELD_PREP(MT_AGG_ARCR_RTS_RATE_THR, 2) | + MT_AGG_ARCR_RATE_DOWN_RATIO_EN | + FIELD_PREP(MT_AGG_ARCR_RATE_DOWN_RATIO, 1) | + FIELD_PREP(MT_AGG_ARCR_RATE_UP_EXTRA_TH, 4))); + + dev->mt76.global_wcid.idx = MT7615_WTBL_RESERVED; + dev->mt76.global_wcid.hw_key_idx = -1; + rcu_assign_pointer(dev->mt76.wcid[MT7615_WTBL_RESERVED], + &dev->mt76.global_wcid); +} + +static int mt7615_init_hardware(struct mt7615_dev *dev) +{ + int ret; + + mt76_wr(dev, MT_INT_SOURCE_CSR, ~0); + + spin_lock_init(&dev->token_lock); + idr_init(&dev->token); + + ret = mt7615_eeprom_init(dev); + if (ret < 0) + return ret; + + ret = mt7615_dma_init(dev); + if (ret) + return ret; + + set_bit(MT76_STATE_INITIALIZED, &dev->mt76.state); + + ret = mt7615_mcu_init(dev); + if (ret) + return ret; + + mt7615_mcu_set_eeprom(dev); + mt7615_mac_init(dev); + mt7615_phy_init(dev); + mt7615_mcu_ctrl_pm_state(dev, 0); + mt7615_mcu_del_wtbl_all(dev); + + return 0; +} + +#define CCK_RATE(_idx, _rate) { \ + .bitrate = _rate, \ + .flags = IEEE80211_RATE_SHORT_PREAMBLE, \ + .hw_value = (MT_PHY_TYPE_CCK << 8) | (_idx), \ + .hw_value_short = (MT_PHY_TYPE_CCK << 8) | (4 + (_idx)), \ +} + +#define OFDM_RATE(_idx, _rate) { \ + .bitrate = _rate, \ + .hw_value = (MT_PHY_TYPE_OFDM << 8) | (_idx), \ + .hw_value_short = (MT_PHY_TYPE_OFDM << 8) | (_idx), \ +} + +static struct ieee80211_rate mt7615_rates[] = { + CCK_RATE(0, 10), + CCK_RATE(1, 20), + CCK_RATE(2, 55), + CCK_RATE(3, 110), + OFDM_RATE(11, 60), + OFDM_RATE(15, 90), + OFDM_RATE(10, 120), + OFDM_RATE(14, 180), + OFDM_RATE(9, 240), + OFDM_RATE(13, 360), + OFDM_RATE(8, 480), + OFDM_RATE(12, 540), +}; + +static const struct ieee80211_iface_limit if_limits[] = { + { + .max = MT7615_MAX_INTERFACES, + .types = BIT(NL80211_IFTYPE_AP) | + BIT(NL80211_IFTYPE_STATION) + } +}; + +static const struct ieee80211_iface_combination if_comb[] = { + { + .limits = if_limits, + .n_limits = ARRAY_SIZE(if_limits), + .max_interfaces = 4, + .num_different_channels = 1, + .beacon_int_infra_match = true, + } +}; + +static int mt7615_init_debugfs(struct mt7615_dev *dev) +{ + struct dentry *dir; + + dir = mt76_register_debugfs(&dev->mt76); + if (!dir) + return -ENOMEM; + + return 0; +} + +int mt7615_register_device(struct mt7615_dev *dev) +{ + struct ieee80211_hw *hw = mt76_hw(dev); + struct wiphy *wiphy = hw->wiphy; + int ret; + + ret = mt7615_init_hardware(dev); + if (ret) + return ret; + + INIT_DELAYED_WORK(&dev->mt76.mac_work, mt7615_mac_work); + + hw->queues = 4; + hw->max_rates = 3; + hw->max_report_rates = 7; + hw->max_rate_tries = 11; + + hw->sta_data_size = sizeof(struct mt7615_sta); + hw->vif_data_size = sizeof(struct mt7615_vif); + + wiphy->iface_combinations = if_comb; + wiphy->n_iface_combinations = ARRAY_SIZE(if_comb); + + ieee80211_hw_set(hw, SUPPORTS_REORDERING_BUFFER); + ieee80211_hw_set(hw, TX_STATUS_NO_AMPDU_LEN); + + dev->mt76.sband_2g.sband.ht_cap.cap |= IEEE80211_HT_CAP_LDPC_CODING; + dev->mt76.sband_5g.sband.ht_cap.cap |= IEEE80211_HT_CAP_LDPC_CODING; + dev->mt76.sband_5g.sband.vht_cap.cap |= + IEEE80211_VHT_CAP_SHORT_GI_160 | + IEEE80211_VHT_CAP_MAX_MPDU_LENGTH_11454 | + IEEE80211_VHT_CAP_MAX_A_MPDU_LENGTH_EXPONENT_MASK | + IEEE80211_VHT_CAP_SUPP_CHAN_WIDTH_160_80PLUS80MHZ; + dev->mt76.chainmask = 0x404; + dev->mt76.antenna_mask = 0xf; + + wiphy->interface_modes = BIT(NL80211_IFTYPE_STATION) | + BIT(NL80211_IFTYPE_AP); + + ret = mt76_register_device(&dev->mt76, true, mt7615_rates, + ARRAY_SIZE(mt7615_rates)); + if (ret) + return ret; + + hw->max_tx_fragments = MT_TXP_MAX_BUF_NUM; + + return mt7615_init_debugfs(dev); +} + +void mt7615_unregister_device(struct mt7615_dev *dev) +{ + struct mt76_txwi_cache *txwi; + int id; + + spin_lock_bh(&dev->token_lock); + idr_for_each_entry(&dev->token, txwi, id) { + mt7615_txp_skb_unmap(&dev->mt76, txwi); + if (txwi->skb) + dev_kfree_skb_any(txwi->skb); + mt76_put_txwi(&dev->mt76, txwi); + } + spin_unlock_bh(&dev->token_lock); + idr_destroy(&dev->token); + mt76_unregister_device(&dev->mt76); + mt7615_mcu_exit(dev); + mt7615_dma_cleanup(dev); + + ieee80211_free_hw(mt76_hw(dev)); +} diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/mac.c b/drivers/net/wireless/mediatek/mt76/mt7615/mac.c new file mode 100644 index 0000000000000000000000000000000000000000..b8f48d10f27a5b1c0436fad698715bbf4128ba6d --- /dev/null +++ b/drivers/net/wireless/mediatek/mt76/mt7615/mac.c @@ -0,0 +1,775 @@ +// SPDX-License-Identifier: ISC +/* Copyright (C) 2019 MediaTek Inc. + * + * Author: Ryder Lee + * Roy Luo + * Felix Fietkau + * Lorenzo Bianconi + */ + +#include +#include +#include "mt7615.h" +#include "../dma.h" +#include "mac.h" + +static struct mt76_wcid *mt7615_rx_get_wcid(struct mt7615_dev *dev, + u8 idx, bool unicast) +{ + struct mt7615_sta *sta; + struct mt76_wcid *wcid; + + if (idx >= ARRAY_SIZE(dev->mt76.wcid)) + return NULL; + + wcid = rcu_dereference(dev->mt76.wcid[idx]); + if (unicast || !wcid) + return wcid; + + if (!wcid->sta) + return NULL; + + sta = container_of(wcid, struct mt7615_sta, wcid); + if (!sta->vif) + return NULL; + + return &sta->vif->sta.wcid; +} + +static int mt7615_get_rate(struct mt7615_dev *dev, + struct ieee80211_supported_band *sband, + int idx, bool cck) +{ + int offset = 0; + int len = sband->n_bitrates; + int i; + + if (cck) { + if (sband == &dev->mt76.sband_5g.sband) + return 0; + + idx &= ~BIT(2); /* short preamble */ + } else if (sband == &dev->mt76.sband_2g.sband) { + offset = 4; + } + + for (i = offset; i < len; i++) { + if ((sband->bitrates[i].hw_value & GENMASK(7, 0)) == idx) + return i; + } + + return 0; +} + +static void mt7615_insert_ccmp_hdr(struct sk_buff *skb, u8 key_id) +{ + struct mt76_rx_status *status = (struct mt76_rx_status *)skb->cb; + int hdr_len = ieee80211_get_hdrlen_from_skb(skb); + u8 *pn = status->iv; + u8 *hdr; + + __skb_push(skb, 8); + memmove(skb->data, skb->data + 8, hdr_len); + hdr = skb->data + hdr_len; + + hdr[0] = pn[5]; + hdr[1] = pn[4]; + hdr[2] = 0; + hdr[3] = 0x20 | (key_id << 6); + hdr[4] = pn[3]; + hdr[5] = pn[2]; + hdr[6] = pn[1]; + hdr[7] = pn[0]; + + status->flag &= ~RX_FLAG_IV_STRIPPED; +} + +int mt7615_mac_fill_rx(struct mt7615_dev *dev, struct sk_buff *skb) +{ + struct mt76_rx_status *status = (struct mt76_rx_status *)skb->cb; + struct ieee80211_supported_band *sband; + struct ieee80211_hdr *hdr; + __le32 *rxd = (__le32 *)skb->data; + u32 rxd0 = le32_to_cpu(rxd[0]); + u32 rxd1 = le32_to_cpu(rxd[1]); + u32 rxd2 = le32_to_cpu(rxd[2]); + bool unicast, remove_pad, insert_ccmp_hdr = false; + int i, idx; + + memset(status, 0, sizeof(*status)); + + unicast = (rxd1 & MT_RXD1_NORMAL_ADDR_TYPE) == MT_RXD1_NORMAL_U2M; + idx = FIELD_GET(MT_RXD2_NORMAL_WLAN_IDX, rxd2); + status->wcid = mt7615_rx_get_wcid(dev, idx, unicast); + + /* TODO: properly support DBDC */ + status->freq = dev->mt76.chandef.chan->center_freq; + status->band = dev->mt76.chandef.chan->band; + if (status->band == NL80211_BAND_5GHZ) + sband = &dev->mt76.sband_5g.sband; + else + sband = &dev->mt76.sband_2g.sband; + + if (rxd2 & MT_RXD2_NORMAL_FCS_ERR) + status->flag |= RX_FLAG_FAILED_FCS_CRC; + + if (rxd2 & MT_RXD2_NORMAL_TKIP_MIC_ERR) + status->flag |= RX_FLAG_MMIC_ERROR; + + if (FIELD_GET(MT_RXD2_NORMAL_SEC_MODE, rxd2) != 0 && + !(rxd2 & (MT_RXD2_NORMAL_CLM | MT_RXD2_NORMAL_CM))) { + status->flag |= RX_FLAG_DECRYPTED; + status->flag |= RX_FLAG_IV_STRIPPED; + status->flag |= RX_FLAG_MMIC_STRIPPED | RX_FLAG_MIC_STRIPPED; + } + + remove_pad = rxd1 & MT_RXD1_NORMAL_HDR_OFFSET; + + if (rxd2 & MT_RXD2_NORMAL_MAX_LEN_ERROR) + return -EINVAL; + + if (!sband->channels) + return -EINVAL; + + rxd += 4; + if (rxd0 & MT_RXD0_NORMAL_GROUP_4) { + rxd += 4; + if ((u8 *)rxd - skb->data >= skb->len) + return -EINVAL; + } + + if (rxd0 & MT_RXD0_NORMAL_GROUP_1) { + u8 *data = (u8 *)rxd; + + if (status->flag & RX_FLAG_DECRYPTED) { + status->iv[0] = data[5]; + status->iv[1] = data[4]; + status->iv[2] = data[3]; + status->iv[3] = data[2]; + status->iv[4] = data[1]; + status->iv[5] = data[0]; + + insert_ccmp_hdr = FIELD_GET(MT_RXD2_NORMAL_FRAG, rxd2); + } + rxd += 4; + if ((u8 *)rxd - skb->data >= skb->len) + return -EINVAL; + } + + if (rxd0 & MT_RXD0_NORMAL_GROUP_2) { + rxd += 2; + if ((u8 *)rxd - skb->data >= skb->len) + return -EINVAL; + } + + if (rxd0 & MT_RXD0_NORMAL_GROUP_3) { + u32 rxdg0 = le32_to_cpu(rxd[0]); + u32 rxdg1 = le32_to_cpu(rxd[1]); + u8 stbc = FIELD_GET(MT_RXV1_HT_STBC, rxdg0); + bool cck = false; + + i = FIELD_GET(MT_RXV1_TX_RATE, rxdg0); + switch (FIELD_GET(MT_RXV1_TX_MODE, rxdg0)) { + case MT_PHY_TYPE_CCK: + cck = true; + /* fall through */ + case MT_PHY_TYPE_OFDM: + i = mt7615_get_rate(dev, sband, i, cck); + break; + case MT_PHY_TYPE_HT_GF: + case MT_PHY_TYPE_HT: + status->encoding = RX_ENC_HT; + if (i > 31) + return -EINVAL; + break; + case MT_PHY_TYPE_VHT: + status->nss = FIELD_GET(MT_RXV2_NSTS, rxdg1) + 1; + status->encoding = RX_ENC_VHT; + break; + default: + return -EINVAL; + } + status->rate_idx = i; + + switch (FIELD_GET(MT_RXV1_FRAME_MODE, rxdg0)) { + case MT_PHY_BW_20: + break; + case MT_PHY_BW_40: + status->bw = RATE_INFO_BW_40; + break; + case MT_PHY_BW_80: + status->bw = RATE_INFO_BW_80; + break; + case MT_PHY_BW_160: + status->bw = RATE_INFO_BW_160; + break; + default: + return -EINVAL; + } + + if (rxdg0 & MT_RXV1_HT_SHORT_GI) + status->enc_flags |= RX_ENC_FLAG_SHORT_GI; + if (rxdg0 & MT_RXV1_HT_AD_CODE) + status->enc_flags |= RX_ENC_FLAG_LDPC; + + status->enc_flags |= RX_ENC_FLAG_STBC_MASK * stbc; + + /* TODO: RSSI */ + rxd += 6; + if ((u8 *)rxd - skb->data >= skb->len) + return -EINVAL; + } + + skb_pull(skb, (u8 *)rxd - skb->data + 2 * remove_pad); + + if (insert_ccmp_hdr) { + u8 key_id = FIELD_GET(MT_RXD1_NORMAL_KEY_ID, rxd1); + + mt7615_insert_ccmp_hdr(skb, key_id); + } + + hdr = (struct ieee80211_hdr *)skb->data; + if (!status->wcid || !ieee80211_is_data_qos(hdr->frame_control)) + return 0; + + status->aggr = unicast && + !ieee80211_is_qos_nullfunc(hdr->frame_control); + status->tid = *ieee80211_get_qos_ctl(hdr) & IEEE80211_QOS_CTL_TID_MASK; + status->seqno = IEEE80211_SEQ_TO_SN(le16_to_cpu(hdr->seq_ctrl)); + + return 0; +} + +void mt7615_sta_ps(struct mt76_dev *mdev, struct ieee80211_sta *sta, bool ps) +{ +} + +void mt7615_tx_complete_skb(struct mt76_dev *mdev, enum mt76_txq_id qid, + struct mt76_queue_entry *e) +{ + if (!e->txwi) { + dev_kfree_skb_any(e->skb); + return; + } + + /* error path */ + if (e->skb == DMA_DUMMY_DATA) { + struct mt76_txwi_cache *t; + struct mt7615_dev *dev; + struct mt7615_txp *txp; + u8 *txwi_ptr; + + txwi_ptr = mt76_get_txwi_ptr(mdev, e->txwi); + txp = (struct mt7615_txp *)(txwi_ptr + MT_TXD_SIZE); + dev = container_of(mdev, struct mt7615_dev, mt76); + + spin_lock_bh(&dev->token_lock); + t = idr_remove(&dev->token, le16_to_cpu(txp->token)); + spin_unlock_bh(&dev->token_lock); + e->skb = t ? t->skb : NULL; + } + + if (e->skb) + mt76_tx_complete_skb(mdev, e->skb); +} + +u16 mt7615_mac_tx_rate_val(struct mt7615_dev *dev, + const struct ieee80211_tx_rate *rate, + bool stbc, u8 *bw) +{ + u8 phy, nss, rate_idx; + u16 rateval; + + *bw = 0; + + if (rate->flags & IEEE80211_TX_RC_VHT_MCS) { + rate_idx = ieee80211_rate_get_vht_mcs(rate); + nss = ieee80211_rate_get_vht_nss(rate); + phy = MT_PHY_TYPE_VHT; + if (rate->flags & IEEE80211_TX_RC_40_MHZ_WIDTH) + *bw = 1; + else if (rate->flags & IEEE80211_TX_RC_80_MHZ_WIDTH) + *bw = 2; + else if (rate->flags & IEEE80211_TX_RC_160_MHZ_WIDTH) + *bw = 3; + } else if (rate->flags & IEEE80211_TX_RC_MCS) { + rate_idx = rate->idx; + nss = 1 + (rate->idx >> 3); + phy = MT_PHY_TYPE_HT; + if (rate->flags & IEEE80211_TX_RC_GREEN_FIELD) + phy = MT_PHY_TYPE_HT_GF; + if (rate->flags & IEEE80211_TX_RC_40_MHZ_WIDTH) + *bw = 1; + } else { + const struct ieee80211_rate *r; + int band = dev->mt76.chandef.chan->band; + u16 val; + + nss = 1; + r = &mt76_hw(dev)->wiphy->bands[band]->bitrates[rate->idx]; + if (rate->flags & IEEE80211_TX_RC_USE_SHORT_PREAMBLE) + val = r->hw_value_short; + else + val = r->hw_value; + + phy = val >> 8; + rate_idx = val & 0xff; + } + + rateval = (FIELD_PREP(MT_TX_RATE_IDX, rate_idx) | + FIELD_PREP(MT_TX_RATE_MODE, phy) | + FIELD_PREP(MT_TX_RATE_NSS, nss - 1)); + + if (stbc && nss == 1) + rateval |= MT_TX_RATE_STBC; + + return rateval; +} + +int mt7615_mac_write_txwi(struct mt7615_dev *dev, __le32 *txwi, + struct sk_buff *skb, struct mt76_wcid *wcid, + struct ieee80211_sta *sta, int pid, + struct ieee80211_key_conf *key) +{ + struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb); + struct ieee80211_tx_rate *rate = &info->control.rates[0]; + struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)skb->data; + struct ieee80211_vif *vif = info->control.vif; + int tx_count = 8; + u8 fc_type, fc_stype, p_fmt, q_idx, omac_idx = 0; + __le16 fc = hdr->frame_control; + u16 seqno = 0; + u32 val; + + if (vif) { + struct mt7615_vif *mvif = (struct mt7615_vif *)vif->drv_priv; + + omac_idx = mvif->omac_idx; + } + + if (sta) { + struct mt7615_sta *msta = (struct mt7615_sta *)sta->drv_priv; + + tx_count = msta->rate_count; + } + + fc_type = (le16_to_cpu(fc) & IEEE80211_FCTL_FTYPE) >> 2; + fc_stype = (le16_to_cpu(fc) & IEEE80211_FCTL_STYPE) >> 4; + + if (ieee80211_is_data(fc)) { + q_idx = skb_get_queue_mapping(skb); + p_fmt = MT_TX_TYPE_CT; + } else if (ieee80211_is_beacon(fc)) { + q_idx = MT_LMAC_BCN0; + p_fmt = MT_TX_TYPE_FW; + } else { + q_idx = MT_LMAC_ALTX0; + p_fmt = MT_TX_TYPE_CT; + } + + val = FIELD_PREP(MT_TXD0_TX_BYTES, skb->len + MT_TXD_SIZE) | + FIELD_PREP(MT_TXD0_P_IDX, MT_TX_PORT_IDX_LMAC) | + FIELD_PREP(MT_TXD0_Q_IDX, q_idx); + txwi[0] = cpu_to_le32(val); + + val = MT_TXD1_LONG_FORMAT | + FIELD_PREP(MT_TXD1_WLAN_IDX, wcid->idx) | + FIELD_PREP(MT_TXD1_HDR_FORMAT, MT_HDR_FORMAT_802_11) | + FIELD_PREP(MT_TXD1_HDR_INFO, + ieee80211_get_hdrlen_from_skb(skb) / 2) | + FIELD_PREP(MT_TXD1_TID, + skb->priority & IEEE80211_QOS_CTL_TID_MASK) | + FIELD_PREP(MT_TXD1_PKT_FMT, p_fmt) | + FIELD_PREP(MT_TXD1_OWN_MAC, omac_idx); + txwi[1] = cpu_to_le32(val); + + val = FIELD_PREP(MT_TXD2_FRAME_TYPE, fc_type) | + FIELD_PREP(MT_TXD2_SUB_TYPE, fc_stype) | + FIELD_PREP(MT_TXD2_MULTICAST, + is_multicast_ether_addr(hdr->addr1)); + txwi[2] = cpu_to_le32(val); + + if (!(info->flags & IEEE80211_TX_CTL_AMPDU)) + txwi[2] |= cpu_to_le32(MT_TXD2_BA_DISABLE); + + txwi[4] = 0; + txwi[6] = 0; + + if (rate->idx >= 0 && rate->count && + !(info->flags & IEEE80211_TX_CTL_RATE_CTRL_PROBE)) { + bool stbc = info->flags & IEEE80211_TX_CTL_STBC; + u8 bw; + u16 rateval = mt7615_mac_tx_rate_val(dev, rate, stbc, &bw); + + txwi[2] |= cpu_to_le32(MT_TXD2_FIX_RATE); + + val = MT_TXD6_FIXED_BW | + FIELD_PREP(MT_TXD6_BW, bw) | + FIELD_PREP(MT_TXD6_TX_RATE, rateval); + txwi[6] |= cpu_to_le32(val); + + if (rate->flags & IEEE80211_TX_RC_SHORT_GI) + txwi[6] |= cpu_to_le32(MT_TXD6_SGI); + + if (info->flags & IEEE80211_TX_CTL_LDPC) + txwi[6] |= cpu_to_le32(MT_TXD6_LDPC); + + if (!(rate->flags & (IEEE80211_TX_RC_MCS | + IEEE80211_TX_RC_VHT_MCS))) + txwi[2] |= cpu_to_le32(MT_TXD2_BA_DISABLE); + + tx_count = rate->count; + } + + if (!ieee80211_is_beacon(fc)) { + val = MT_TXD5_TX_STATUS_HOST | MT_TXD5_SW_POWER_MGMT | + FIELD_PREP(MT_TXD5_PID, pid); + txwi[5] = cpu_to_le32(val); + } else { + txwi[5] = 0; + /* use maximum tx count for beacons */ + tx_count = 0x1f; + } + + val = FIELD_PREP(MT_TXD3_REM_TX_COUNT, tx_count); + if (ieee80211_is_data_qos(hdr->frame_control)) { + seqno = IEEE80211_SEQ_TO_SN(le16_to_cpu(hdr->seq_ctrl)); + val |= MT_TXD3_SN_VALID; + } else if (ieee80211_is_back_req(hdr->frame_control)) { + struct ieee80211_bar *bar = (struct ieee80211_bar *)skb->data; + + seqno = IEEE80211_SEQ_TO_SN(le16_to_cpu(bar->start_seq_num)); + val |= MT_TXD3_SN_VALID; + } + val |= FIELD_PREP(MT_TXD3_SEQ, seqno); + + txwi[3] = cpu_to_le32(val); + + if (info->flags & IEEE80211_TX_CTL_NO_ACK) + txwi[3] |= cpu_to_le32(MT_TXD3_NO_ACK); + + if (key) + txwi[3] |= cpu_to_le32(MT_TXD3_PROTECT_FRAME); + + txwi[7] = FIELD_PREP(MT_TXD7_TYPE, fc_type) | + FIELD_PREP(MT_TXD7_SUB_TYPE, fc_stype); + + return 0; +} + +void mt7615_txp_skb_unmap(struct mt76_dev *dev, + struct mt76_txwi_cache *t) +{ + struct mt7615_txp *txp; + u8 *txwi; + int i; + + txwi = mt76_get_txwi_ptr(dev, t); + txp = (struct mt7615_txp *)(txwi + MT_TXD_SIZE); + for (i = 1; i < txp->nbuf; i++) + dma_unmap_single(dev->dev, le32_to_cpu(txp->buf[i]), + le16_to_cpu(txp->len[i]), DMA_TO_DEVICE); +} + +int mt7615_tx_prepare_skb(struct mt76_dev *mdev, void *txwi_ptr, + enum mt76_txq_id qid, struct mt76_wcid *wcid, + struct ieee80211_sta *sta, + struct mt76_tx_info *tx_info) +{ + struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)tx_info->skb->data; + struct mt7615_dev *dev = container_of(mdev, struct mt7615_dev, mt76); + struct mt7615_sta *msta = container_of(wcid, struct mt7615_sta, wcid); + struct ieee80211_tx_info *info = IEEE80211_SKB_CB(tx_info->skb); + struct ieee80211_key_conf *key = info->control.hw_key; + struct ieee80211_vif *vif = info->control.vif; + int i, pid, id, nbuf = tx_info->nbuf - 1; + u8 *txwi = (u8 *)txwi_ptr; + struct mt76_txwi_cache *t; + struct mt7615_txp *txp; + + if (!wcid) + wcid = &dev->mt76.global_wcid; + + pid = mt76_tx_status_skb_add(mdev, wcid, tx_info->skb); + + if (info->flags & IEEE80211_TX_CTL_RATE_CTRL_PROBE) { + spin_lock_bh(&dev->mt76.lock); + msta->rate_probe = true; + mt7615_mcu_set_rates(dev, msta, &info->control.rates[0], + msta->rates); + spin_unlock_bh(&dev->mt76.lock); + } + + mt7615_mac_write_txwi(dev, txwi_ptr, tx_info->skb, wcid, sta, + pid, key); + + txp = (struct mt7615_txp *)(txwi + MT_TXD_SIZE); + for (i = 0; i < nbuf; i++) { + txp->buf[i] = cpu_to_le32(tx_info->buf[i + 1].addr); + txp->len[i] = cpu_to_le16(tx_info->buf[i + 1].len); + } + txp->nbuf = nbuf; + + /* pass partial skb header to fw */ + tx_info->buf[1].len = MT_CT_PARSE_LEN; + tx_info->nbuf = MT_CT_DMA_BUF_NUM; + + txp->flags = cpu_to_le16(MT_CT_INFO_APPLY_TXD); + + if (!key) + txp->flags |= cpu_to_le16(MT_CT_INFO_NONE_CIPHER_FRAME); + + if (ieee80211_is_mgmt(hdr->frame_control)) + txp->flags |= cpu_to_le16(MT_CT_INFO_MGMT_FRAME); + + if (vif) { + struct mt7615_vif *mvif = (struct mt7615_vif *)vif->drv_priv; + + txp->bss_idx = mvif->idx; + } + + t = (struct mt76_txwi_cache *)(txwi + mdev->drv->txwi_size); + t->skb = tx_info->skb; + + spin_lock_bh(&dev->token_lock); + id = idr_alloc(&dev->token, t, 0, MT7615_TOKEN_SIZE, GFP_ATOMIC); + spin_unlock_bh(&dev->token_lock); + if (id < 0) + return id; + + txp->token = cpu_to_le16(id); + txp->rept_wds_wcid = 0xff; + tx_info->skb = DMA_DUMMY_DATA; + + return 0; +} + +static bool mt7615_fill_txs(struct mt7615_dev *dev, struct mt7615_sta *sta, + struct ieee80211_tx_info *info, __le32 *txs_data) +{ + struct ieee80211_supported_band *sband; + int i, idx, count, final_idx = 0; + bool fixed_rate, final_mpdu, ack_timeout; + bool probe, ampdu, cck = false; + u32 final_rate, final_rate_flags, final_nss, txs; + u8 pid; + + fixed_rate = info->status.rates[0].count; + probe = !!(info->flags & IEEE80211_TX_CTL_RATE_CTRL_PROBE); + + txs = le32_to_cpu(txs_data[1]); + final_mpdu = txs & MT_TXS1_ACKED_MPDU; + ampdu = !fixed_rate && (txs & MT_TXS1_AMPDU); + + txs = le32_to_cpu(txs_data[3]); + count = FIELD_GET(MT_TXS3_TX_COUNT, txs); + + txs = le32_to_cpu(txs_data[0]); + pid = FIELD_GET(MT_TXS0_PID, txs); + final_rate = FIELD_GET(MT_TXS0_TX_RATE, txs); + ack_timeout = txs & MT_TXS0_ACK_TIMEOUT; + + if (!ampdu && (txs & MT_TXS0_RTS_TIMEOUT)) + return false; + + if (txs & MT_TXS0_QUEUE_TIMEOUT) + return false; + + if (!ack_timeout) + info->flags |= IEEE80211_TX_STAT_ACK; + + info->status.ampdu_len = 1; + info->status.ampdu_ack_len = !!(info->flags & + IEEE80211_TX_STAT_ACK); + + if (ampdu || (info->flags & IEEE80211_TX_CTL_AMPDU)) + info->flags |= IEEE80211_TX_STAT_AMPDU | IEEE80211_TX_CTL_AMPDU; + + if (fixed_rate && !probe) { + info->status.rates[0].count = count; + goto out; + } + + for (i = 0, idx = 0; i < ARRAY_SIZE(info->status.rates); i++) { + int cur_count = min_t(int, count, 2 * MT7615_RATE_RETRY); + + if (!i && probe) { + cur_count = 1; + } else { + info->status.rates[i] = sta->rates[idx]; + idx++; + } + + if (i && info->status.rates[i].idx < 0) { + info->status.rates[i - 1].count += count; + break; + } + + if (!count) { + info->status.rates[i].idx = -1; + break; + } + + info->status.rates[i].count = cur_count; + final_idx = i; + count -= cur_count; + } + +out: + final_rate_flags = info->status.rates[final_idx].flags; + + switch (FIELD_GET(MT_TX_RATE_MODE, final_rate)) { + case MT_PHY_TYPE_CCK: + cck = true; + /* fall through */ + case MT_PHY_TYPE_OFDM: + if (dev->mt76.chandef.chan->band == NL80211_BAND_5GHZ) + sband = &dev->mt76.sband_5g.sband; + else + sband = &dev->mt76.sband_2g.sband; + final_rate &= MT_TX_RATE_IDX; + final_rate = mt7615_get_rate(dev, sband, final_rate, cck); + final_rate_flags = 0; + break; + case MT_PHY_TYPE_HT_GF: + case MT_PHY_TYPE_HT: + final_rate_flags |= IEEE80211_TX_RC_MCS; + final_rate &= MT_TX_RATE_IDX; + if (final_rate > 31) + return false; + break; + case MT_PHY_TYPE_VHT: + final_nss = FIELD_GET(MT_TX_RATE_NSS, final_rate); + final_rate_flags |= IEEE80211_TX_RC_VHT_MCS; + final_rate = (final_rate & MT_TX_RATE_IDX) | (final_nss << 4); + break; + default: + return false; + } + + info->status.rates[final_idx].idx = final_rate; + info->status.rates[final_idx].flags = final_rate_flags; + + return true; +} + +static bool mt7615_mac_add_txs_skb(struct mt7615_dev *dev, + struct mt7615_sta *sta, int pid, + __le32 *txs_data) +{ + struct mt76_dev *mdev = &dev->mt76; + struct sk_buff_head list; + struct sk_buff *skb; + + if (pid < MT_PACKET_ID_FIRST) + return false; + + mt76_tx_status_lock(mdev, &list); + skb = mt76_tx_status_skb_get(mdev, &sta->wcid, pid, &list); + if (skb) { + struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb); + + if (info->flags & IEEE80211_TX_CTL_RATE_CTRL_PROBE) { + spin_lock_bh(&dev->mt76.lock); + if (sta->rate_probe) { + mt7615_mcu_set_rates(dev, sta, NULL, + sta->rates); + sta->rate_probe = false; + } + spin_unlock_bh(&dev->mt76.lock); + } + + if (!mt7615_fill_txs(dev, sta, info, txs_data)) { + ieee80211_tx_info_clear_status(info); + info->status.rates[0].idx = -1; + } + + mt76_tx_status_skb_done(mdev, skb, &list); + } + mt76_tx_status_unlock(mdev, &list); + + return !!skb; +} + +void mt7615_mac_add_txs(struct mt7615_dev *dev, void *data) +{ + struct ieee80211_tx_info info = {}; + struct ieee80211_sta *sta = NULL; + struct mt7615_sta *msta = NULL; + struct mt76_wcid *wcid; + __le32 *txs_data = data; + u32 txs; + u8 wcidx; + u8 pid; + + txs = le32_to_cpu(txs_data[0]); + pid = FIELD_GET(MT_TXS0_PID, txs); + txs = le32_to_cpu(txs_data[2]); + wcidx = FIELD_GET(MT_TXS2_WCID, txs); + + if (pid == MT_PACKET_ID_NO_ACK) + return; + + if (wcidx >= ARRAY_SIZE(dev->mt76.wcid)) + return; + + rcu_read_lock(); + + wcid = rcu_dereference(dev->mt76.wcid[wcidx]); + if (!wcid) + goto out; + + msta = container_of(wcid, struct mt7615_sta, wcid); + sta = wcid_to_sta(wcid); + + if (mt7615_mac_add_txs_skb(dev, msta, pid, txs_data)) + goto out; + + if (wcidx >= MT7615_WTBL_STA || !sta) + goto out; + + if (mt7615_fill_txs(dev, msta, &info, txs_data)) + ieee80211_tx_status_noskb(mt76_hw(dev), sta, &info); + +out: + rcu_read_unlock(); +} + +void mt7615_mac_tx_free(struct mt7615_dev *dev, struct sk_buff *skb) +{ + struct mt7615_tx_free *free = (struct mt7615_tx_free *)skb->data; + struct mt76_dev *mdev = &dev->mt76; + struct mt76_txwi_cache *txwi; + u8 i, count; + + count = FIELD_GET(MT_TX_FREE_MSDU_ID_CNT, le16_to_cpu(free->ctrl)); + for (i = 0; i < count; i++) { + spin_lock_bh(&dev->token_lock); + txwi = idr_remove(&dev->token, le16_to_cpu(free->token[i])); + spin_unlock_bh(&dev->token_lock); + + if (!txwi) + continue; + + mt7615_txp_skb_unmap(mdev, txwi); + if (txwi->skb) { + mt76_tx_complete_skb(mdev, txwi->skb); + txwi->skb = NULL; + } + + mt76_put_txwi(mdev, txwi); + } + dev_kfree_skb(skb); +} + +void mt7615_mac_work(struct work_struct *work) +{ + struct mt7615_dev *dev; + + dev = (struct mt7615_dev *)container_of(work, struct mt76_dev, + mac_work.work); + + mt76_tx_status_check(&dev->mt76, NULL, false); + ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mt76.mac_work, + MT7615_WATCHDOG_TIME); +} diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/mac.h b/drivers/net/wireless/mediatek/mt76/mt7615/mac.h new file mode 100644 index 0000000000000000000000000000000000000000..18ad4b8a3807c49b28cf79994f1fabeb96d3ba8f --- /dev/null +++ b/drivers/net/wireless/mediatek/mt76/mt7615/mac.h @@ -0,0 +1,300 @@ +/* SPDX-License-Identifier: ISC */ +/* Copyright (C) 2019 MediaTek Inc. */ + +#ifndef __MT7615_MAC_H +#define __MT7615_MAC_H + +#define MT_CT_PARSE_LEN 72 +#define MT_CT_DMA_BUF_NUM 2 + +#define MT_RXD0_LENGTH GENMASK(15, 0) +#define MT_RXD0_PKT_TYPE GENMASK(31, 29) + +#define MT_RXD0_NORMAL_ETH_TYPE_OFS GENMASK(22, 16) +#define MT_RXD0_NORMAL_IP_SUM BIT(23) +#define MT_RXD0_NORMAL_UDP_TCP_SUM BIT(24) +#define MT_RXD0_NORMAL_GROUP_1 BIT(25) +#define MT_RXD0_NORMAL_GROUP_2 BIT(26) +#define MT_RXD0_NORMAL_GROUP_3 BIT(27) +#define MT_RXD0_NORMAL_GROUP_4 BIT(28) + +enum rx_pkt_type { + PKT_TYPE_TXS, + PKT_TYPE_TXRXV, + PKT_TYPE_NORMAL, + PKT_TYPE_RX_DUP_RFB, + PKT_TYPE_RX_TMR, + PKT_TYPE_RETRIEVE, + PKT_TYPE_TXRX_NOTIFY, + PKT_TYPE_RX_EVENT +}; + +#define MT_RXD1_NORMAL_BSSID GENMASK(31, 26) +#define MT_RXD1_NORMAL_PAYLOAD_FORMAT GENMASK(25, 24) +#define MT_RXD1_NORMAL_HDR_TRANS BIT(23) +#define MT_RXD1_NORMAL_HDR_OFFSET BIT(22) +#define MT_RXD1_NORMAL_MAC_HDR_LEN GENMASK(21, 16) +#define MT_RXD1_NORMAL_CH_FREQ GENMASK(15, 8) +#define MT_RXD1_NORMAL_KEY_ID GENMASK(7, 6) +#define MT_RXD1_NORMAL_BEACON_UC BIT(5) +#define MT_RXD1_NORMAL_BEACON_MC BIT(4) +#define MT_RXD1_NORMAL_BF_REPORT BIT(3) +#define MT_RXD1_NORMAL_ADDR_TYPE GENMASK(2, 1) +#define MT_RXD1_NORMAL_BCAST GENMASK(2, 1) +#define MT_RXD1_NORMAL_MCAST BIT(2) +#define MT_RXD1_NORMAL_U2M BIT(1) +#define MT_RXD1_NORMAL_HTC_VLD BIT(0) + +#define MT_RXD2_NORMAL_NON_AMPDU BIT(31) +#define MT_RXD2_NORMAL_NON_AMPDU_SUB BIT(30) +#define MT_RXD2_NORMAL_NDATA BIT(29) +#define MT_RXD2_NORMAL_NULL_FRAME BIT(28) +#define MT_RXD2_NORMAL_FRAG BIT(27) +#define MT_RXD2_NORMAL_INT_FRAME BIT(26) +#define MT_RXD2_NORMAL_HDR_TRANS_ERROR BIT(25) +#define MT_RXD2_NORMAL_MAX_LEN_ERROR BIT(24) +#define MT_RXD2_NORMAL_AMSDU_ERR BIT(23) +#define MT_RXD2_NORMAL_LEN_MISMATCH BIT(22) +#define MT_RXD2_NORMAL_TKIP_MIC_ERR BIT(21) +#define MT_RXD2_NORMAL_ICV_ERR BIT(20) +#define MT_RXD2_NORMAL_CLM BIT(19) +#define MT_RXD2_NORMAL_CM BIT(18) +#define MT_RXD2_NORMAL_FCS_ERR BIT(17) +#define MT_RXD2_NORMAL_SW_BIT BIT(16) +#define MT_RXD2_NORMAL_SEC_MODE GENMASK(15, 12) +#define MT_RXD2_NORMAL_TID GENMASK(11, 8) +#define MT_RXD2_NORMAL_WLAN_IDX GENMASK(7, 0) + +#define MT_RXD3_NORMAL_PF_STS GENMASK(31, 30) +#define MT_RXD3_NORMAL_PF_MODE BIT(29) +#define MT_RXD3_NORMAL_CLS_BITMAP GENMASK(28, 19) +#define MT_RXD3_NORMAL_WOL GENMASK(18, 14) +#define MT_RXD3_NORMAL_MAGIC_PKT BIT(13) +#define MT_RXD3_NORMAL_OFLD GENMASK(12, 11) +#define MT_RXD3_NORMAL_CLS BIT(10) +#define MT_RXD3_NORMAL_PATTERN_DROP BIT(9) +#define MT_RXD3_NORMAL_TSF_COMPARE_LOSS BIT(8) +#define MT_RXD3_NORMAL_RXV_SEQ GENMASK(7, 0) + +#define MT_RXV1_ACID_DET_H BIT(31) +#define MT_RXV1_ACID_DET_L BIT(30) +#define MT_RXV1_VHTA2_B8_B3 GENMASK(29, 24) +#define MT_RXV1_NUM_RX GENMASK(23, 22) +#define MT_RXV1_HT_NO_SOUND BIT(21) +#define MT_RXV1_HT_SMOOTH BIT(20) +#define MT_RXV1_HT_SHORT_GI BIT(19) +#define MT_RXV1_HT_AGGR BIT(18) +#define MT_RXV1_VHTA1_B22 BIT(17) +#define MT_RXV1_FRAME_MODE GENMASK(16, 15) +#define MT_RXV1_TX_MODE GENMASK(14, 12) +#define MT_RXV1_HT_EXT_LTF GENMASK(11, 10) +#define MT_RXV1_HT_AD_CODE BIT(9) +#define MT_RXV1_HT_STBC GENMASK(8, 7) +#define MT_RXV1_TX_RATE GENMASK(6, 0) + +#define MT_RXV2_SEL_ANT BIT(31) +#define MT_RXV2_VALID_BIT BIT(30) +#define MT_RXV2_NSTS GENMASK(29, 27) +#define MT_RXV2_GROUP_ID GENMASK(26, 21) +#define MT_RXV2_LENGTH GENMASK(20, 0) + +enum tx_header_format { + MT_HDR_FORMAT_802_3, + MT_HDR_FORMAT_CMD, + MT_HDR_FORMAT_802_11, + MT_HDR_FORMAT_802_11_EXT, +}; + +enum tx_pkt_type { + MT_TX_TYPE_CT, + MT_TX_TYPE_SF, + MT_TX_TYPE_CMD, + MT_TX_TYPE_FW, +}; + +enum tx_pkt_queue_idx { + MT_LMAC_AC00, + MT_LMAC_AC01, + MT_LMAC_AC02, + MT_LMAC_AC03, + MT_LMAC_ALTX0 = 0x10, + MT_LMAC_BMC0, + MT_LMAC_BCN0, + MT_LMAC_PSMP0, +}; + +enum tx_port_idx { + MT_TX_PORT_IDX_LMAC, + MT_TX_PORT_IDX_MCU +}; + +enum tx_mcu_port_q_idx { + MT_TX_MCU_PORT_RX_Q0 = 0, + MT_TX_MCU_PORT_RX_Q1, + MT_TX_MCU_PORT_RX_Q2, + MT_TX_MCU_PORT_RX_Q3, + MT_TX_MCU_PORT_RX_FWDL = 0x1e +}; + +enum tx_phy_bandwidth { + MT_PHY_BW_20, + MT_PHY_BW_40, + MT_PHY_BW_80, + MT_PHY_BW_160, +}; + +#define MT_CT_INFO_APPLY_TXD BIT(0) +#define MT_CT_INFO_COPY_HOST_TXD_ALL BIT(1) +#define MT_CT_INFO_MGMT_FRAME BIT(2) +#define MT_CT_INFO_NONE_CIPHER_FRAME BIT(3) +#define MT_CT_INFO_HSR2_TX BIT(4) + +#define MT_TXD_SIZE (8 * 4) + +#define MT_TXD0_P_IDX BIT(31) +#define MT_TXD0_Q_IDX GENMASK(30, 26) +#define MT_TXD0_UDP_TCP_SUM BIT(24) +#define MT_TXD0_IP_SUM BIT(23) +#define MT_TXD0_ETH_TYPE_OFFSET GENMASK(22, 16) +#define MT_TXD0_TX_BYTES GENMASK(15, 0) + +#define MT_TXD1_OWN_MAC GENMASK(31, 26) +#define MT_TXD1_PKT_FMT GENMASK(25, 24) +#define MT_TXD1_TID GENMASK(23, 21) +#define MT_TXD1_AMSDU BIT(20) +#define MT_TXD1_UNXV BIT(19) +#define MT_TXD1_HDR_PAD GENMASK(18, 17) +#define MT_TXD1_TXD_LEN BIT(16) +#define MT_TXD1_LONG_FORMAT BIT(15) +#define MT_TXD1_HDR_FORMAT GENMASK(14, 13) +#define MT_TXD1_HDR_INFO GENMASK(12, 8) +#define MT_TXD1_WLAN_IDX GENMASK(7, 0) + +#define MT_TXD2_FIX_RATE BIT(31) +#define MT_TXD2_TIMING_MEASURE BIT(30) +#define MT_TXD2_BA_DISABLE BIT(29) +#define MT_TXD2_POWER_OFFSET GENMASK(28, 24) +#define MT_TXD2_MAX_TX_TIME GENMASK(23, 16) +#define MT_TXD2_FRAG GENMASK(15, 14) +#define MT_TXD2_HTC_VLD BIT(13) +#define MT_TXD2_DURATION BIT(12) +#define MT_TXD2_BIP BIT(11) +#define MT_TXD2_MULTICAST BIT(10) +#define MT_TXD2_RTS BIT(9) +#define MT_TXD2_SOUNDING BIT(8) +#define MT_TXD2_NDPA BIT(7) +#define MT_TXD2_NDP BIT(6) +#define MT_TXD2_FRAME_TYPE GENMASK(5, 4) +#define MT_TXD2_SUB_TYPE GENMASK(3, 0) + +#define MT_TXD3_SN_VALID BIT(31) +#define MT_TXD3_PN_VALID BIT(30) +#define MT_TXD3_SEQ GENMASK(27, 16) +#define MT_TXD3_REM_TX_COUNT GENMASK(15, 11) +#define MT_TXD3_TX_COUNT GENMASK(10, 6) +#define MT_TXD3_PROTECT_FRAME BIT(1) +#define MT_TXD3_NO_ACK BIT(0) + +#define MT_TXD4_PN_LOW GENMASK(31, 0) + +#define MT_TXD5_PN_HIGH GENMASK(31, 16) +#define MT_TXD5_SW_POWER_MGMT BIT(13) +#define MT_TXD5_DA_SELECT BIT(11) +#define MT_TXD5_TX_STATUS_HOST BIT(10) +#define MT_TXD5_TX_STATUS_MCU BIT(9) +#define MT_TXD5_TX_STATUS_FMT BIT(8) +#define MT_TXD5_PID GENMASK(7, 0) + +#define MT_TXD6_FIXED_RATE BIT(31) +#define MT_TXD6_SGI BIT(30) +#define MT_TXD6_LDPC BIT(29) +#define MT_TXD6_TX_BF BIT(28) +#define MT_TXD6_TX_RATE GENMASK(27, 16) +#define MT_TXD6_ANT_ID GENMASK(15, 4) +#define MT_TXD6_DYN_BW BIT(3) +#define MT_TXD6_FIXED_BW BIT(2) +#define MT_TXD6_BW GENMASK(1, 0) + +#define MT_TXD7_TYPE GENMASK(21, 20) +#define MT_TXD7_SUB_TYPE GENMASK(19, 16) + +#define MT_TX_RATE_STBC BIT(11) +#define MT_TX_RATE_NSS GENMASK(10, 9) +#define MT_TX_RATE_MODE GENMASK(8, 6) +#define MT_TX_RATE_IDX GENMASK(5, 0) + +#define MT_TXP_MAX_BUF_NUM 6 + +struct mt7615_txp { + __le16 flags; + __le16 token; + u8 bss_idx; + u8 rept_wds_wcid; + u8 rsv; + u8 nbuf; + __le32 buf[MT_TXP_MAX_BUF_NUM]; + __le16 len[MT_TXP_MAX_BUF_NUM]; +} __packed; + +struct mt7615_tx_free { + __le16 rx_byte_cnt; + __le16 ctrl; + u8 txd_cnt; + u8 rsv[3]; + __le16 token[]; +} __packed; + +#define MT_TX_FREE_MSDU_ID_CNT GENMASK(6, 0) + +#define MT_TXS0_PID GENMASK(31, 24) +#define MT_TXS0_BA_ERROR BIT(22) +#define MT_TXS0_PS_FLAG BIT(21) +#define MT_TXS0_TXOP_TIMEOUT BIT(20) +#define MT_TXS0_BIP_ERROR BIT(19) + +#define MT_TXS0_QUEUE_TIMEOUT BIT(18) +#define MT_TXS0_RTS_TIMEOUT BIT(17) +#define MT_TXS0_ACK_TIMEOUT BIT(16) +#define MT_TXS0_ACK_ERROR_MASK GENMASK(18, 16) + +#define MT_TXS0_TX_STATUS_HOST BIT(15) +#define MT_TXS0_TX_STATUS_MCU BIT(14) +#define MT_TXS0_TXS_FORMAT BIT(13) +#define MT_TXS0_FIXED_RATE BIT(12) +#define MT_TXS0_TX_RATE GENMASK(11, 0) + +#define MT_TXS1_ANT_ID GENMASK(31, 20) +#define MT_TXS1_RESP_RATE GENMASK(19, 16) +#define MT_TXS1_BW GENMASK(15, 14) +#define MT_TXS1_I_TXBF BIT(13) +#define MT_TXS1_E_TXBF BIT(12) +#define MT_TXS1_TID GENMASK(11, 9) +#define MT_TXS1_AMPDU BIT(8) +#define MT_TXS1_ACKED_MPDU BIT(7) +#define MT_TXS1_TX_POWER_DBM GENMASK(6, 0) + +#define MT_TXS2_WCID GENMASK(31, 24) +#define MT_TXS2_RXV_SEQNO GENMASK(23, 16) +#define MT_TXS2_TX_DELAY GENMASK(15, 0) + +#define MT_TXS3_LAST_TX_RATE GENMASK(31, 29) +#define MT_TXS3_TX_COUNT GENMASK(28, 24) +#define MT_TXS3_F1_TSSI1 GENMASK(23, 12) +#define MT_TXS3_F1_TSSI0 GENMASK(11, 0) +#define MT_TXS3_F0_SEQNO GENMASK(11, 0) + +#define MT_TXS4_F0_TIMESTAMP GENMASK(31, 0) +#define MT_TXS4_F1_TSSI3 GENMASK(23, 12) +#define MT_TXS4_F1_TSSI2 GENMASK(11, 0) + +#define MT_TXS5_F0_FRONT_TIME GENMASK(24, 0) +#define MT_TXS5_F1_NOISE_2 GENMASK(23, 16) +#define MT_TXS5_F1_NOISE_1 GENMASK(15, 8) +#define MT_TXS5_F1_NOISE_0 GENMASK(7, 0) + +#define MT_TXS6_F1_RCPI_3 GENMASK(31, 24) +#define MT_TXS6_F1_RCPI_2 GENMASK(23, 16) +#define MT_TXS6_F1_RCPI_1 GENMASK(15, 8) +#define MT_TXS6_F1_RCPI_0 GENMASK(7, 0) + +#endif diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/main.c b/drivers/net/wireless/mediatek/mt76/mt7615/main.c new file mode 100644 index 0000000000000000000000000000000000000000..80e6b211f60b8f5ca15c6db16e94f73b40b226f5 --- /dev/null +++ b/drivers/net/wireless/mediatek/mt76/mt7615/main.c @@ -0,0 +1,499 @@ +// SPDX-License-Identifier: ISC +/* Copyright (C) 2019 MediaTek Inc. + * + * Author: Roy Luo + * Ryder Lee + * Felix Fietkau + */ + +#include +#include +#include +#include +#include "mt7615.h" + +static int mt7615_start(struct ieee80211_hw *hw) +{ + struct mt7615_dev *dev = hw->priv; + + set_bit(MT76_STATE_RUNNING, &dev->mt76.state); + ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mt76.mac_work, + MT7615_WATCHDOG_TIME); + + return 0; +} + +static void mt7615_stop(struct ieee80211_hw *hw) +{ + struct mt7615_dev *dev = hw->priv; + + clear_bit(MT76_STATE_RUNNING, &dev->mt76.state); + cancel_delayed_work_sync(&dev->mt76.mac_work); +} + +static int get_omac_idx(enum nl80211_iftype type, u32 mask) +{ + int i; + + switch (type) { + case NL80211_IFTYPE_AP: + /* ap use hw bssid 0 and ext bssid */ + if (~mask & BIT(HW_BSSID_0)) + return HW_BSSID_0; + + for (i = EXT_BSSID_1; i < EXT_BSSID_END; i++) + if (~mask & BIT(i)) + return i; + + break; + case NL80211_IFTYPE_STATION: + /* sta use hw bssid other than 0 */ + for (i = HW_BSSID_1; i < HW_BSSID_MAX; i++) + if (~mask & BIT(i)) + return i; + + break; + default: + WARN_ON(1); + break; + }; + + return -1; +} + +static int mt7615_add_interface(struct ieee80211_hw *hw, + struct ieee80211_vif *vif) +{ + struct mt7615_vif *mvif = (struct mt7615_vif *)vif->drv_priv; + struct mt7615_dev *dev = hw->priv; + struct mt76_txq *mtxq; + int idx, ret = 0; + + mutex_lock(&dev->mt76.mutex); + + mvif->idx = ffs(~dev->vif_mask) - 1; + if (mvif->idx >= MT7615_MAX_INTERFACES) { + ret = -ENOSPC; + goto out; + } + + mvif->omac_idx = get_omac_idx(vif->type, dev->omac_mask); + if (mvif->omac_idx < 0) { + ret = -ENOSPC; + goto out; + } + + /* TODO: DBDC support. Use band 0 and wmm 0 for now */ + mvif->band_idx = 0; + mvif->wmm_idx = 0; + + ret = mt7615_mcu_set_dev_info(dev, vif, 1); + if (ret) + goto out; + + dev->vif_mask |= BIT(mvif->idx); + dev->omac_mask |= BIT(mvif->omac_idx); + idx = MT7615_WTBL_RESERVED - 1 - mvif->idx; + mvif->sta.wcid.idx = idx; + mvif->sta.wcid.hw_key_idx = -1; + + rcu_assign_pointer(dev->mt76.wcid[idx], &mvif->sta.wcid); + mtxq = (struct mt76_txq *)vif->txq->drv_priv; + mtxq->wcid = &mvif->sta.wcid; + mt76_txq_init(&dev->mt76, vif->txq); + +out: + mutex_unlock(&dev->mt76.mutex); + + return ret; +} + +static void mt7615_remove_interface(struct ieee80211_hw *hw, + struct ieee80211_vif *vif) +{ + struct mt7615_vif *mvif = (struct mt7615_vif *)vif->drv_priv; + struct mt7615_dev *dev = hw->priv; + int idx = mvif->sta.wcid.idx; + + /* TODO: disable beacon for the bss */ + + mt7615_mcu_set_dev_info(dev, vif, 0); + + rcu_assign_pointer(dev->mt76.wcid[idx], NULL); + mt76_txq_remove(&dev->mt76, vif->txq); + + mutex_lock(&dev->mt76.mutex); + dev->vif_mask &= ~BIT(mvif->idx); + dev->omac_mask &= ~BIT(mvif->omac_idx); + mutex_unlock(&dev->mt76.mutex); +} + +static int mt7615_set_channel(struct mt7615_dev *dev, + struct cfg80211_chan_def *def) +{ + int ret; + + cancel_delayed_work_sync(&dev->mt76.mac_work); + set_bit(MT76_RESET, &dev->mt76.state); + + mt76_set_channel(&dev->mt76); + + ret = mt7615_mcu_set_channel(dev); + if (ret) + return ret; + + clear_bit(MT76_RESET, &dev->mt76.state); + + mt76_txq_schedule_all(&dev->mt76); + ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mt76.mac_work, + MT7615_WATCHDOG_TIME); + return 0; +} + +static int mt7615_set_key(struct ieee80211_hw *hw, enum set_key_cmd cmd, + struct ieee80211_vif *vif, struct ieee80211_sta *sta, + struct ieee80211_key_conf *key) +{ + struct mt7615_dev *dev = hw->priv; + struct mt7615_vif *mvif = (struct mt7615_vif *)vif->drv_priv; + struct mt7615_sta *msta = sta ? (struct mt7615_sta *)sta->drv_priv : + &mvif->sta; + struct mt76_wcid *wcid = &msta->wcid; + int idx = key->keyidx; + + /* The hardware does not support per-STA RX GTK, fallback + * to software mode for these. + */ + if ((vif->type == NL80211_IFTYPE_ADHOC || + vif->type == NL80211_IFTYPE_MESH_POINT) && + (key->cipher == WLAN_CIPHER_SUITE_TKIP || + key->cipher == WLAN_CIPHER_SUITE_CCMP) && + !(key->flags & IEEE80211_KEY_FLAG_PAIRWISE)) + return -EOPNOTSUPP; + + if (cmd == SET_KEY) { + key->hw_key_idx = wcid->idx; + wcid->hw_key_idx = idx; + } else { + if (idx == wcid->hw_key_idx) + wcid->hw_key_idx = -1; + + key = NULL; + } + mt76_wcid_key_setup(&dev->mt76, wcid, key); + + return mt7615_mcu_set_wtbl_key(dev, wcid->idx, key, cmd); +} + +static int mt7615_config(struct ieee80211_hw *hw, u32 changed) +{ + struct mt7615_dev *dev = hw->priv; + int ret = 0; + + if (changed & IEEE80211_CONF_CHANGE_CHANNEL) { + mutex_lock(&dev->mt76.mutex); + + ieee80211_stop_queues(hw); + ret = mt7615_set_channel(dev, &hw->conf.chandef); + ieee80211_wake_queues(hw); + + mutex_unlock(&dev->mt76.mutex); + } + + if (changed & IEEE80211_CONF_CHANGE_MONITOR) { + mutex_lock(&dev->mt76.mutex); + + if (!(hw->conf.flags & IEEE80211_CONF_MONITOR)) + dev->mt76.rxfilter |= MT_WF_RFCR_DROP_OTHER_UC; + else + dev->mt76.rxfilter &= ~MT_WF_RFCR_DROP_OTHER_UC; + + mt76_wr(dev, MT_WF_RFCR, dev->mt76.rxfilter); + + mutex_unlock(&dev->mt76.mutex); + } + return ret; +} + +static int +mt7615_conf_tx(struct ieee80211_hw *hw, struct ieee80211_vif *vif, u16 queue, + const struct ieee80211_tx_queue_params *params) +{ + struct mt7615_dev *dev = hw->priv; + static const u8 wmm_queue_map[] = { + [IEEE80211_AC_BK] = 0, + [IEEE80211_AC_BE] = 1, + [IEEE80211_AC_VI] = 2, + [IEEE80211_AC_VO] = 3, + }; + + /* TODO: hw wmm_set 1~3 */ + return mt7615_mcu_set_wmm(dev, wmm_queue_map[queue], params); +} + +static void mt7615_configure_filter(struct ieee80211_hw *hw, + unsigned int changed_flags, + unsigned int *total_flags, + u64 multicast) +{ + struct mt7615_dev *dev = hw->priv; + u32 flags = 0; + +#define MT76_FILTER(_flag, _hw) do { \ + flags |= *total_flags & FIF_##_flag; \ + dev->mt76.rxfilter &= ~(_hw); \ + dev->mt76.rxfilter |= !(flags & FIF_##_flag) * (_hw); \ + } while (0) + + dev->mt76.rxfilter &= ~(MT_WF_RFCR_DROP_OTHER_BSS | + MT_WF_RFCR_DROP_OTHER_BEACON | + MT_WF_RFCR_DROP_FRAME_REPORT | + MT_WF_RFCR_DROP_PROBEREQ | + MT_WF_RFCR_DROP_MCAST_FILTERED | + MT_WF_RFCR_DROP_MCAST | + MT_WF_RFCR_DROP_BCAST | + MT_WF_RFCR_DROP_DUPLICATE | + MT_WF_RFCR_DROP_A2_BSSID | + MT_WF_RFCR_DROP_UNWANTED_CTL | + MT_WF_RFCR_DROP_STBC_MULTI); + + MT76_FILTER(OTHER_BSS, MT_WF_RFCR_DROP_OTHER_TIM | + MT_WF_RFCR_DROP_A3_MAC | + MT_WF_RFCR_DROP_A3_BSSID); + + MT76_FILTER(FCSFAIL, MT_WF_RFCR_DROP_FCSFAIL); + + MT76_FILTER(CONTROL, MT_WF_RFCR_DROP_CTS | + MT_WF_RFCR_DROP_RTS | + MT_WF_RFCR_DROP_CTL_RSV | + MT_WF_RFCR_DROP_NDPA); + + *total_flags = flags; + mt76_wr(dev, MT_WF_RFCR, dev->mt76.rxfilter); +} + +static void mt7615_bss_info_changed(struct ieee80211_hw *hw, + struct ieee80211_vif *vif, + struct ieee80211_bss_conf *info, + u32 changed) +{ + struct mt7615_dev *dev = hw->priv; + + mutex_lock(&dev->mt76.mutex); + + /* TODO: sta mode connect/disconnect + * BSS_CHANGED_ASSOC | BSS_CHANGED_BSSID + */ + + /* TODO: update beacon content + * BSS_CHANGED_BEACON + */ + + if (changed & BSS_CHANGED_BEACON_ENABLED) { + if (info->enable_beacon) { + mt7615_mcu_set_bss_info(dev, vif, 1); + mt7615_mcu_add_wtbl_bmc(dev, vif); + mt7615_mcu_set_sta_rec_bmc(dev, vif, 1); + mt7615_mcu_set_bcn(dev, vif, 1); + } else { + mt7615_mcu_set_sta_rec_bmc(dev, vif, 0); + mt7615_mcu_del_wtbl_bmc(dev, vif); + mt7615_mcu_set_bss_info(dev, vif, 0); + mt7615_mcu_set_bcn(dev, vif, 0); + } + } + + mutex_unlock(&dev->mt76.mutex); +} + +int mt7615_sta_add(struct mt76_dev *mdev, struct ieee80211_vif *vif, + struct ieee80211_sta *sta) +{ + struct mt7615_dev *dev = container_of(mdev, struct mt7615_dev, mt76); + struct mt7615_sta *msta = (struct mt7615_sta *)sta->drv_priv; + struct mt7615_vif *mvif = (struct mt7615_vif *)vif->drv_priv; + int idx; + + idx = mt76_wcid_alloc(dev->mt76.wcid_mask, MT7615_WTBL_STA - 1); + if (idx < 0) + return -ENOSPC; + + msta->vif = mvif; + msta->wcid.sta = 1; + msta->wcid.idx = idx; + + mt7615_mcu_add_wtbl(dev, vif, sta); + mt7615_mcu_set_sta_rec(dev, vif, sta, 1); + + return 0; +} + +void mt7615_sta_assoc(struct mt76_dev *mdev, struct ieee80211_vif *vif, + struct ieee80211_sta *sta) +{ + struct mt7615_dev *dev = container_of(mdev, struct mt7615_dev, mt76); + + if (sta->ht_cap.ht_supported) + mt7615_mcu_set_ht_cap(dev, vif, sta); +} + +void mt7615_sta_remove(struct mt76_dev *mdev, struct ieee80211_vif *vif, + struct ieee80211_sta *sta) +{ + struct mt7615_dev *dev = container_of(mdev, struct mt7615_dev, mt76); + + mt7615_mcu_set_sta_rec(dev, vif, sta, 0); + mt7615_mcu_del_wtbl(dev, vif, sta); +} + +static void mt7615_sta_rate_tbl_update(struct ieee80211_hw *hw, + struct ieee80211_vif *vif, + struct ieee80211_sta *sta) +{ + struct mt7615_dev *dev = hw->priv; + struct mt7615_sta *msta = (struct mt7615_sta *)sta->drv_priv; + struct ieee80211_sta_rates *sta_rates = rcu_dereference(sta->rates); + int i; + + spin_lock_bh(&dev->mt76.lock); + for (i = 0; i < ARRAY_SIZE(msta->rates); i++) { + msta->rates[i].idx = sta_rates->rate[i].idx; + msta->rates[i].count = sta_rates->rate[i].count; + msta->rates[i].flags = sta_rates->rate[i].flags; + + if (msta->rates[i].idx < 0 || !msta->rates[i].count) + break; + } + msta->n_rates = i; + mt7615_mcu_set_rates(dev, msta, NULL, msta->rates); + msta->rate_probe = false; + spin_unlock_bh(&dev->mt76.lock); +} + +static void mt7615_tx(struct ieee80211_hw *hw, + struct ieee80211_tx_control *control, + struct sk_buff *skb) +{ + struct mt7615_dev *dev = hw->priv; + struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb); + struct ieee80211_vif *vif = info->control.vif; + struct mt76_wcid *wcid = &dev->mt76.global_wcid; + + if (control->sta) { + struct mt7615_sta *sta; + + sta = (struct mt7615_sta *)control->sta->drv_priv; + wcid = &sta->wcid; + } + + if (vif && !control->sta) { + struct mt7615_vif *mvif; + + mvif = (struct mt7615_vif *)vif->drv_priv; + wcid = &mvif->sta.wcid; + } + + mt76_tx(&dev->mt76, control->sta, wcid, skb); +} + +static int mt7615_set_rts_threshold(struct ieee80211_hw *hw, u32 val) +{ + struct mt7615_dev *dev = hw->priv; + + mutex_lock(&dev->mt76.mutex); + mt7615_mcu_set_rts_thresh(dev, val); + mutex_unlock(&dev->mt76.mutex); + + return 0; +} + +static int +mt7615_ampdu_action(struct ieee80211_hw *hw, struct ieee80211_vif *vif, + struct ieee80211_ampdu_params *params) +{ + enum ieee80211_ampdu_mlme_action action = params->action; + struct mt7615_dev *dev = hw->priv; + struct ieee80211_sta *sta = params->sta; + struct ieee80211_txq *txq = sta->txq[params->tid]; + struct mt7615_sta *msta = (struct mt7615_sta *)sta->drv_priv; + u16 tid = params->tid; + u16 *ssn = ¶ms->ssn; + struct mt76_txq *mtxq; + + if (!txq) + return -EINVAL; + + mtxq = (struct mt76_txq *)txq->drv_priv; + + switch (action) { + case IEEE80211_AMPDU_RX_START: + mt76_rx_aggr_start(&dev->mt76, &msta->wcid, tid, *ssn, + params->buf_size); + mt7615_mcu_set_rx_ba(dev, params, 1); + break; + case IEEE80211_AMPDU_RX_STOP: + mt76_rx_aggr_stop(&dev->mt76, &msta->wcid, tid); + mt7615_mcu_set_rx_ba(dev, params, 0); + break; + case IEEE80211_AMPDU_TX_OPERATIONAL: + mtxq->aggr = true; + mtxq->send_bar = false; + mt7615_mcu_set_tx_ba(dev, params, 1); + break; + case IEEE80211_AMPDU_TX_STOP_FLUSH: + case IEEE80211_AMPDU_TX_STOP_FLUSH_CONT: + mtxq->aggr = false; + ieee80211_send_bar(vif, sta->addr, tid, mtxq->agg_ssn); + mt7615_mcu_set_tx_ba(dev, params, 0); + break; + case IEEE80211_AMPDU_TX_START: + mtxq->agg_ssn = IEEE80211_SN_TO_SEQ(*ssn); + ieee80211_start_tx_ba_cb_irqsafe(vif, sta->addr, tid); + break; + case IEEE80211_AMPDU_TX_STOP_CONT: + mtxq->aggr = false; + mt7615_mcu_set_tx_ba(dev, params, 0); + ieee80211_stop_tx_ba_cb_irqsafe(vif, sta->addr, tid); + break; + } + + return 0; +} + +static void +mt7615_sw_scan(struct ieee80211_hw *hw, struct ieee80211_vif *vif, + const u8 *mac) +{ + struct mt7615_dev *dev = hw->priv; + + set_bit(MT76_SCANNING, &dev->mt76.state); +} + +static void +mt7615_sw_scan_complete(struct ieee80211_hw *hw, struct ieee80211_vif *vif) +{ + struct mt7615_dev *dev = hw->priv; + + clear_bit(MT76_SCANNING, &dev->mt76.state); +} + +const struct ieee80211_ops mt7615_ops = { + .tx = mt7615_tx, + .start = mt7615_start, + .stop = mt7615_stop, + .add_interface = mt7615_add_interface, + .remove_interface = mt7615_remove_interface, + .config = mt7615_config, + .conf_tx = mt7615_conf_tx, + .configure_filter = mt7615_configure_filter, + .bss_info_changed = mt7615_bss_info_changed, + .sta_state = mt76_sta_state, + .set_key = mt7615_set_key, + .ampdu_action = mt7615_ampdu_action, + .set_rts_threshold = mt7615_set_rts_threshold, + .wake_tx_queue = mt76_wake_tx_queue, + .sta_rate_tbl_update = mt7615_sta_rate_tbl_update, + .sw_scan_start = mt7615_sw_scan, + .sw_scan_complete = mt7615_sw_scan_complete, + .release_buffered_frames = mt76_release_buffered_frames, +}; diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/mcu.c b/drivers/net/wireless/mediatek/mt76/mt7615/mcu.c new file mode 100644 index 0000000000000000000000000000000000000000..ea67c6022fe6ff5894fa3cf066dfeddb1ab0d16a --- /dev/null +++ b/drivers/net/wireless/mediatek/mt76/mt7615/mcu.c @@ -0,0 +1,1655 @@ +// SPDX-License-Identifier: ISC +/* Copyright (C) 2019 MediaTek Inc. + * + * Author: Roy Luo + * Ryder Lee + */ + +#include +#include "mt7615.h" +#include "mcu.h" +#include "mac.h" +#include "eeprom.h" + +struct mt7615_patch_hdr { + char build_date[16]; + char platform[4]; + __be32 hw_sw_ver; + __be32 patch_ver; + __be16 checksum; +} __packed; + +struct mt7615_fw_trailer { + __le32 addr; + u8 chip_id; + u8 feature_set; + u8 eco_code; + char fw_ver[10]; + char build_date[15]; + __le32 len; +} __packed; + +#define MCU_PATCH_ADDRESS 0x80000 + +#define N9_REGION_NUM 2 +#define CR4_REGION_NUM 1 + +#define IMG_CRC_LEN 4 + +#define FW_FEATURE_SET_ENCRYPT BIT(0) +#define FW_FEATURE_SET_KEY_IDX GENMASK(2, 1) + +#define DL_MODE_ENCRYPT BIT(0) +#define DL_MODE_KEY_IDX GENMASK(2, 1) +#define DL_MODE_RESET_SEC_IV BIT(3) +#define DL_MODE_WORKING_PDA_CR4 BIT(4) +#define DL_MODE_NEED_RSP BIT(31) + +#define FW_START_OVERRIDE BIT(0) +#define FW_START_WORKING_PDA_CR4 BIT(2) + +static int __mt7615_mcu_msg_send(struct mt7615_dev *dev, struct sk_buff *skb, + int cmd, int query, int dest, int *wait_seq) +{ + struct mt7615_mcu_txd *mcu_txd; + u8 seq, q_idx, pkt_fmt; + enum mt76_txq_id qid; + u32 val; + __le32 *txd; + + if (!skb) + return -EINVAL; + + seq = ++dev->mt76.mmio.mcu.msg_seq & 0xf; + if (!seq) + seq = ++dev->mt76.mmio.mcu.msg_seq & 0xf; + + mcu_txd = (struct mt7615_mcu_txd *)skb_push(skb, + sizeof(struct mt7615_mcu_txd)); + memset(mcu_txd, 0, sizeof(struct mt7615_mcu_txd)); + + if (cmd != -MCU_CMD_FW_SCATTER) { + q_idx = MT_TX_MCU_PORT_RX_Q0; + pkt_fmt = MT_TX_TYPE_CMD; + } else { + q_idx = MT_TX_MCU_PORT_RX_FWDL; + pkt_fmt = MT_TX_TYPE_FW; + } + + txd = mcu_txd->txd; + + val = FIELD_PREP(MT_TXD0_TX_BYTES, cpu_to_le16(skb->len)) | + FIELD_PREP(MT_TXD0_P_IDX, MT_TX_PORT_IDX_MCU) | + FIELD_PREP(MT_TXD0_Q_IDX, q_idx); + txd[0] = cpu_to_le32(val); + + val = MT_TXD1_LONG_FORMAT | + FIELD_PREP(MT_TXD1_HDR_FORMAT, MT_HDR_FORMAT_CMD) | + FIELD_PREP(MT_TXD1_PKT_FMT, pkt_fmt); + txd[1] = cpu_to_le32(val); + + mcu_txd->len = cpu_to_le16(skb->len - sizeof(mcu_txd->txd)); + mcu_txd->pq_id = cpu_to_le16(MCU_PQ_ID(MT_TX_PORT_IDX_MCU, q_idx)); + mcu_txd->pkt_type = MCU_PKT_ID; + mcu_txd->seq = seq; + + if (cmd < 0) { + mcu_txd->cid = -cmd; + } else { + mcu_txd->cid = MCU_CMD_EXT_CID; + mcu_txd->ext_cid = cmd; + if (query != MCU_Q_NA) + mcu_txd->ext_cid_ack = 1; + } + + mcu_txd->set_query = query; + mcu_txd->s2d_index = dest; + + if (wait_seq) + *wait_seq = seq; + + if (test_bit(MT76_STATE_MCU_RUNNING, &dev->mt76.state)) + qid = MT_TXQ_MCU; + else + qid = MT_TXQ_FWDL; + + return mt76_tx_queue_skb_raw(dev, qid, skb, 0); +} + +static int mt7615_mcu_msg_send(struct mt7615_dev *dev, struct sk_buff *skb, + int cmd, int query, int dest, + struct sk_buff **skb_ret) +{ + unsigned long expires = jiffies + 10 * HZ; + struct mt7615_mcu_rxd *rxd; + int ret, seq; + + mutex_lock(&dev->mt76.mmio.mcu.mutex); + + ret = __mt7615_mcu_msg_send(dev, skb, cmd, query, dest, &seq); + if (ret) + goto out; + + while (1) { + skb = mt76_mcu_get_response(&dev->mt76, expires); + if (!skb) { + dev_err(dev->mt76.dev, "Message %d (seq %d) timeout\n", + cmd, seq); + ret = -ETIMEDOUT; + break; + } + + rxd = (struct mt7615_mcu_rxd *)skb->data; + if (seq != rxd->seq) + continue; + + if (skb_ret) { + int hdr_len = sizeof(*rxd); + + if (!test_bit(MT76_STATE_MCU_RUNNING, + &dev->mt76.state)) + hdr_len -= 4; + skb_pull(skb, hdr_len); + *skb_ret = skb; + } else { + dev_kfree_skb(skb); + } + + break; + } + +out: + mutex_unlock(&dev->mt76.mmio.mcu.mutex); + + return ret; +} + +static int mt7615_mcu_init_download(struct mt7615_dev *dev, u32 addr, + u32 len, u32 mode) +{ + struct { + __le32 addr; + __le32 len; + __le32 mode; + } req = { + .addr = cpu_to_le32(addr), + .len = cpu_to_le32(len), + .mode = cpu_to_le32(mode), + }; + struct sk_buff *skb = mt7615_mcu_msg_alloc(&req, sizeof(req)); + + return mt7615_mcu_msg_send(dev, skb, -MCU_CMD_TARGET_ADDRESS_LEN_REQ, + MCU_Q_NA, MCU_S2D_H2N, NULL); +} + +static int mt7615_mcu_send_firmware(struct mt7615_dev *dev, const void *data, + int len) +{ + struct sk_buff *skb; + int ret = 0; + + while (len > 0) { + int cur_len = min_t(int, 4096 - sizeof(struct mt7615_mcu_txd), + len); + + skb = mt7615_mcu_msg_alloc(data, cur_len); + if (!skb) + return -ENOMEM; + + ret = __mt7615_mcu_msg_send(dev, skb, -MCU_CMD_FW_SCATTER, + MCU_Q_NA, MCU_S2D_H2N, NULL); + if (ret) + break; + + data += cur_len; + len -= cur_len; + } + + return ret; +} + +static int mt7615_mcu_start_firmware(struct mt7615_dev *dev, u32 addr, + u32 option) +{ + struct { + __le32 option; + __le32 addr; + } req = { + .option = cpu_to_le32(option), + .addr = cpu_to_le32(addr), + }; + struct sk_buff *skb = mt7615_mcu_msg_alloc(&req, sizeof(req)); + + return mt7615_mcu_msg_send(dev, skb, -MCU_CMD_FW_START_REQ, + MCU_Q_NA, MCU_S2D_H2N, NULL); +} + +static int mt7615_mcu_restart(struct mt7615_dev *dev) +{ + struct sk_buff *skb = mt7615_mcu_msg_alloc(NULL, 0); + + return mt7615_mcu_msg_send(dev, skb, -MCU_CMD_RESTART_DL_REQ, + MCU_Q_NA, MCU_S2D_H2N, NULL); +} + +static int mt7615_mcu_patch_sem_ctrl(struct mt7615_dev *dev, bool get) +{ + struct { + __le32 operation; + } req = { + .operation = cpu_to_le32(get ? PATCH_SEM_GET : + PATCH_SEM_RELEASE), + }; + struct event { + u8 status; + u8 reserved[3]; + } *resp; + struct sk_buff *skb = mt7615_mcu_msg_alloc(&req, sizeof(req)); + struct sk_buff *skb_ret; + int ret; + + ret = mt7615_mcu_msg_send(dev, skb, -MCU_CMD_PATCH_SEM_CONTROL, + MCU_Q_NA, MCU_S2D_H2N, &skb_ret); + if (ret) + goto out; + + resp = (struct event *)(skb_ret->data); + ret = resp->status; + dev_kfree_skb(skb_ret); + +out: + return ret; +} + +static int mt7615_mcu_start_patch(struct mt7615_dev *dev) +{ + struct { + u8 check_crc; + u8 reserved[3]; + } req = { + .check_crc = 0, + }; + struct sk_buff *skb = mt7615_mcu_msg_alloc(&req, sizeof(req)); + + return mt7615_mcu_msg_send(dev, skb, -MCU_CMD_PATCH_FINISH_REQ, + MCU_Q_NA, MCU_S2D_H2N, NULL); +} + +static int mt7615_driver_own(struct mt7615_dev *dev) +{ + mt76_wr(dev, MT_CFG_LPCR_HOST, MT_CFG_LPCR_HOST_DRV_OWN); + if (!mt76_poll_msec(dev, MT_CFG_LPCR_HOST, + MT_CFG_LPCR_HOST_FW_OWN, 0, 500)) { + dev_err(dev->mt76.dev, "Timeout for driver own\n"); + return -EIO; + } + + return 0; +} + +static int mt7615_load_patch(struct mt7615_dev *dev) +{ + const struct firmware *fw; + const struct mt7615_patch_hdr *hdr; + const char *firmware = MT7615_ROM_PATCH; + int len, ret, sem; + + sem = mt7615_mcu_patch_sem_ctrl(dev, 1); + switch (sem) { + case PATCH_IS_DL: + return 0; + case PATCH_NOT_DL_SEM_SUCCESS: + break; + default: + dev_err(dev->mt76.dev, "Failed to get patch semaphore\n"); + return -EAGAIN; + } + + ret = request_firmware(&fw, firmware, dev->mt76.dev); + if (ret) + return ret; + + if (!fw || !fw->data || fw->size < sizeof(*hdr)) { + dev_err(dev->mt76.dev, "Invalid firmware\n"); + ret = -EINVAL; + goto out; + } + + hdr = (const struct mt7615_patch_hdr *)(fw->data); + + dev_info(dev->mt76.dev, "HW/SW Version: 0x%x, Build Time: %.16s\n", + be32_to_cpu(hdr->hw_sw_ver), hdr->build_date); + + len = fw->size - sizeof(*hdr); + + ret = mt7615_mcu_init_download(dev, MCU_PATCH_ADDRESS, len, + DL_MODE_NEED_RSP); + if (ret) { + dev_err(dev->mt76.dev, "Download request failed\n"); + goto out; + } + + ret = mt7615_mcu_send_firmware(dev, fw->data + sizeof(*hdr), len); + if (ret) { + dev_err(dev->mt76.dev, "Failed to send firmware to device\n"); + goto out; + } + + ret = mt7615_mcu_start_patch(dev); + if (ret) + dev_err(dev->mt76.dev, "Failed to start patch\n"); + +out: + release_firmware(fw); + + sem = mt7615_mcu_patch_sem_ctrl(dev, 0); + switch (sem) { + case PATCH_REL_SEM_SUCCESS: + break; + default: + ret = -EAGAIN; + dev_err(dev->mt76.dev, "Failed to release patch semaphore\n"); + break; + } + + return ret; +} + +static u32 gen_dl_mode(u8 feature_set, bool is_cr4) +{ + u32 ret = 0; + + ret |= (feature_set & FW_FEATURE_SET_ENCRYPT) ? + (DL_MODE_ENCRYPT | DL_MODE_RESET_SEC_IV) : 0; + ret |= FIELD_PREP(DL_MODE_KEY_IDX, + FIELD_GET(FW_FEATURE_SET_KEY_IDX, feature_set)); + ret |= DL_MODE_NEED_RSP; + ret |= is_cr4 ? DL_MODE_WORKING_PDA_CR4 : 0; + + return ret; +} + +static int mt7615_load_ram(struct mt7615_dev *dev) +{ + const struct firmware *fw; + const struct mt7615_fw_trailer *hdr; + const char *n9_firmware = MT7615_FIRMWARE_N9; + const char *cr4_firmware = MT7615_FIRMWARE_CR4; + u32 n9_ilm_addr, offset; + int i, ret; + + ret = request_firmware(&fw, n9_firmware, dev->mt76.dev); + if (ret) + return ret; + + if (!fw || !fw->data || fw->size < N9_REGION_NUM * sizeof(*hdr)) { + dev_err(dev->mt76.dev, "Invalid firmware\n"); + ret = -EINVAL; + goto out; + } + + hdr = (const struct mt7615_fw_trailer *)(fw->data + fw->size - + N9_REGION_NUM * sizeof(*hdr)); + + dev_info(dev->mt76.dev, "N9 Firmware Version: %.10s, Build Time: %.15s\n", + hdr->fw_ver, hdr->build_date); + + n9_ilm_addr = le32_to_cpu(hdr->addr); + + for (offset = 0, i = 0; i < N9_REGION_NUM; i++) { + u32 len, addr, mode; + + len = le32_to_cpu(hdr[i].len) + IMG_CRC_LEN; + addr = le32_to_cpu(hdr[i].addr); + mode = gen_dl_mode(hdr[i].feature_set, false); + + ret = mt7615_mcu_init_download(dev, addr, len, mode); + if (ret) { + dev_err(dev->mt76.dev, "Download request failed\n"); + goto out; + } + + ret = mt7615_mcu_send_firmware(dev, fw->data + offset, len); + if (ret) { + dev_err(dev->mt76.dev, "Failed to send firmware to device\n"); + goto out; + } + + offset += len; + } + + ret = mt7615_mcu_start_firmware(dev, n9_ilm_addr, FW_START_OVERRIDE); + if (ret) { + dev_err(dev->mt76.dev, "Failed to start N9 firmware\n"); + goto out; + } + + release_firmware(fw); + + ret = request_firmware(&fw, cr4_firmware, dev->mt76.dev); + if (ret) + return ret; + + if (!fw || !fw->data || fw->size < CR4_REGION_NUM * sizeof(*hdr)) { + dev_err(dev->mt76.dev, "Invalid firmware\n"); + ret = -EINVAL; + goto out; + } + + hdr = (const struct mt7615_fw_trailer *)(fw->data + fw->size - + CR4_REGION_NUM * sizeof(*hdr)); + + dev_info(dev->mt76.dev, "CR4 Firmware Version: %.10s, Build Time: %.15s\n", + hdr->fw_ver, hdr->build_date); + + for (offset = 0, i = 0; i < CR4_REGION_NUM; i++) { + u32 len, addr, mode; + + len = le32_to_cpu(hdr[i].len) + IMG_CRC_LEN; + addr = le32_to_cpu(hdr[i].addr); + mode = gen_dl_mode(hdr[i].feature_set, true); + + ret = mt7615_mcu_init_download(dev, addr, len, mode); + if (ret) { + dev_err(dev->mt76.dev, "Download request failed\n"); + goto out; + } + + ret = mt7615_mcu_send_firmware(dev, fw->data + offset, len); + if (ret) { + dev_err(dev->mt76.dev, "Failed to send firmware to device\n"); + goto out; + } + + offset += len; + } + + ret = mt7615_mcu_start_firmware(dev, 0, FW_START_WORKING_PDA_CR4); + if (ret) + dev_err(dev->mt76.dev, "Failed to start CR4 firmware\n"); + +out: + release_firmware(fw); + + return ret; +} + +static int mt7615_load_firmware(struct mt7615_dev *dev) +{ + int ret; + u32 val; + + val = mt76_get_field(dev, MT_TOP_MISC2, MT_TOP_MISC2_FW_STATE); + + if (val != FW_STATE_FW_DOWNLOAD) { + dev_err(dev->mt76.dev, "Firmware is not ready for download\n"); + return -EIO; + } + + ret = mt7615_load_patch(dev); + if (ret) + return ret; + + ret = mt7615_load_ram(dev); + if (ret) + return ret; + + if (!mt76_poll_msec(dev, MT_TOP_MISC2, MT_TOP_MISC2_FW_STATE, + FIELD_PREP(MT_TOP_MISC2_FW_STATE, + FW_STATE_CR4_RDY), 500)) { + dev_err(dev->mt76.dev, "Timeout for initializing firmware\n"); + return -EIO; + } + + dev_dbg(dev->mt76.dev, "Firmware init done\n"); + + return 0; +} + +int mt7615_mcu_init(struct mt7615_dev *dev) +{ + int ret; + + ret = mt7615_driver_own(dev); + if (ret) + return ret; + + ret = mt7615_load_firmware(dev); + if (ret) + return ret; + + set_bit(MT76_STATE_MCU_RUNNING, &dev->mt76.state); + + return 0; +} + +void mt7615_mcu_exit(struct mt7615_dev *dev) +{ + mt7615_mcu_restart(dev); + mt76_wr(dev, MT_CFG_LPCR_HOST, MT_CFG_LPCR_HOST_FW_OWN); + skb_queue_purge(&dev->mt76.mmio.mcu.res_q); +} + +int mt7615_mcu_set_eeprom(struct mt7615_dev *dev) +{ + struct req_data { + u8 val; + } __packed; + struct { + u8 buffer_mode; + u8 pad; + u16 len; + } __packed req_hdr = { + .buffer_mode = 1, + .len = __MT_EE_MAX - MT_EE_NIC_CONF_0, + }; + struct sk_buff *skb; + struct req_data *data; + const int size = (__MT_EE_MAX - MT_EE_NIC_CONF_0) * + sizeof(struct req_data); + u8 *eep = (u8 *)dev->mt76.eeprom.data; + u16 off; + + skb = mt7615_mcu_msg_alloc(NULL, size + sizeof(req_hdr)); + memcpy(skb_put(skb, sizeof(req_hdr)), &req_hdr, sizeof(req_hdr)); + data = (struct req_data *)skb_put(skb, size); + memset(data, 0, size); + + for (off = MT_EE_NIC_CONF_0; off < __MT_EE_MAX; off++) + data[off - MT_EE_NIC_CONF_0].val = eep[off]; + + return mt7615_mcu_msg_send(dev, skb, MCU_EXT_CMD_EFUSE_BUFFER_MODE, + MCU_Q_SET, MCU_S2D_H2N, NULL); +} + +int mt7615_mcu_init_mac(struct mt7615_dev *dev) +{ + struct { + u8 enable; + u8 band; + u8 rsv[2]; + } __packed req = { + .enable = 1, + .band = 0, + }; + struct sk_buff *skb = mt7615_mcu_msg_alloc(&req, sizeof(req)); + + return mt7615_mcu_msg_send(dev, skb, MCU_EXT_CMD_MAC_INIT_CTRL, + MCU_Q_SET, MCU_S2D_H2N, NULL); +} + +int mt7615_mcu_set_rts_thresh(struct mt7615_dev *dev, u32 val) +{ + struct { + u8 prot_idx; + u8 band; + u8 rsv[2]; + __le32 len_thresh; + __le32 pkt_thresh; + } __packed req = { + .prot_idx = 1, + .band = 0, + .len_thresh = cpu_to_le32(val), + .pkt_thresh = cpu_to_le32(0x2), + }; + struct sk_buff *skb = mt7615_mcu_msg_alloc(&req, sizeof(req)); + + return mt7615_mcu_msg_send(dev, skb, MCU_EXT_CMD_PROTECT_CTRL, + MCU_Q_SET, MCU_S2D_H2N, NULL); +} + +int mt7615_mcu_set_wmm(struct mt7615_dev *dev, u8 queue, + const struct ieee80211_tx_queue_params *params) +{ +#define WMM_AIFS_SET BIT(0) +#define WMM_CW_MIN_SET BIT(1) +#define WMM_CW_MAX_SET BIT(2) +#define WMM_TXOP_SET BIT(3) + struct req_data { + u8 number; + u8 rsv[3]; + u8 queue; + u8 valid; + u8 aifs; + u8 cw_min; + __le16 cw_max; + __le16 txop; + } __packed req = { + .number = 1, + .queue = queue, + .valid = WMM_AIFS_SET | WMM_TXOP_SET, + .aifs = params->aifs, + .txop = cpu_to_le16(params->txop), + }; + struct sk_buff *skb; + + if (params->cw_min) { + req.valid |= WMM_CW_MIN_SET; + req.cw_min = params->cw_min; + } + if (params->cw_max) { + req.valid |= WMM_CW_MAX_SET; + req.cw_max = cpu_to_le16(params->cw_max); + } + + skb = mt7615_mcu_msg_alloc(&req, sizeof(req)); + return mt7615_mcu_msg_send(dev, skb, MCU_EXT_CMD_EDCA_UPDATE, + MCU_Q_SET, MCU_S2D_H2N, NULL); +} + +int mt7615_mcu_ctrl_pm_state(struct mt7615_dev *dev, int enter) +{ +#define ENTER_PM_STATE 1 +#define EXIT_PM_STATE 2 + struct { + u8 pm_number; + u8 pm_state; + u8 bssid[ETH_ALEN]; + u8 dtim_period; + u8 wlan_idx; + __le16 bcn_interval; + __le32 aid; + __le32 rx_filter; + u8 band_idx; + u8 rsv[3]; + __le32 feature; + u8 omac_idx; + u8 wmm_idx; + u8 bcn_loss_cnt; + u8 bcn_sp_duration; + } __packed req = { + .pm_number = 5, + .pm_state = (enter) ? ENTER_PM_STATE : EXIT_PM_STATE, + .band_idx = 0, + }; + struct sk_buff *skb = mt7615_mcu_msg_alloc(&req, sizeof(req)); + + return mt7615_mcu_msg_send(dev, skb, MCU_EXT_CMD_PM_STATE_CTRL, + MCU_Q_SET, MCU_S2D_H2N, NULL); +} + +static int __mt7615_mcu_set_dev_info(struct mt7615_dev *dev, + struct dev_info *dev_info) +{ + struct req_hdr { + u8 omac_idx; + u8 band_idx; + __le16 tlv_num; + u8 is_tlv_append; + u8 rsv[3]; + } __packed req_hdr = {0}; + struct req_tlv { + __le16 tag; + __le16 len; + u8 active; + u8 band_idx; + u8 omac_addr[ETH_ALEN]; + } __packed; + struct sk_buff *skb; + u16 tlv_num = 0; + + skb = mt7615_mcu_msg_alloc(NULL, sizeof(req_hdr) + + sizeof(struct req_tlv)); + skb_reserve(skb, sizeof(req_hdr)); + + if (dev_info->feature & BIT(DEV_INFO_ACTIVE)) { + struct req_tlv req_tlv = { + .tag = cpu_to_le16(DEV_INFO_ACTIVE), + .len = cpu_to_le16(sizeof(req_tlv)), + .active = dev_info->enable, + .band_idx = dev_info->band_idx, + }; + memcpy(req_tlv.omac_addr, dev_info->omac_addr, ETH_ALEN); + memcpy(skb_put(skb, sizeof(req_tlv)), &req_tlv, + sizeof(req_tlv)); + tlv_num++; + } + + req_hdr.omac_idx = dev_info->omac_idx; + req_hdr.band_idx = dev_info->band_idx; + req_hdr.tlv_num = cpu_to_le16(tlv_num); + req_hdr.is_tlv_append = tlv_num ? 1 : 0; + + memcpy(skb_push(skb, sizeof(req_hdr)), &req_hdr, sizeof(req_hdr)); + + return mt7615_mcu_msg_send(dev, skb, MCU_EXT_CMD_DEV_INFO_UPDATE, + MCU_Q_SET, MCU_S2D_H2N, NULL); +} + +int mt7615_mcu_set_dev_info(struct mt7615_dev *dev, struct ieee80211_vif *vif, + int en) +{ + struct mt7615_vif *mvif = (struct mt7615_vif *)vif->drv_priv; + struct dev_info dev_info = {0}; + + dev_info.omac_idx = mvif->omac_idx; + memcpy(dev_info.omac_addr, vif->addr, ETH_ALEN); + dev_info.band_idx = mvif->band_idx; + dev_info.enable = en; + dev_info.feature = BIT(DEV_INFO_ACTIVE); + + return __mt7615_mcu_set_dev_info(dev, &dev_info); +} + +static void bss_info_omac_handler (struct mt7615_dev *dev, + struct bss_info *bss_info, + struct sk_buff *skb) +{ + struct bss_info_omac tlv = {0}; + + tlv.tag = cpu_to_le16(BSS_INFO_OMAC); + tlv.len = cpu_to_le16(sizeof(tlv)); + tlv.hw_bss_idx = (bss_info->omac_idx > EXT_BSSID_START) ? + HW_BSSID_0 : bss_info->omac_idx; + tlv.omac_idx = bss_info->omac_idx; + tlv.band_idx = bss_info->band_idx; + tlv.conn_type = cpu_to_le32(bss_info->conn_type); + + memcpy(skb_put(skb, sizeof(tlv)), &tlv, sizeof(tlv)); +} + +static void bss_info_basic_handler (struct mt7615_dev *dev, + struct bss_info *bss_info, + struct sk_buff *skb) +{ + struct bss_info_basic tlv = {0}; + + tlv.tag = cpu_to_le16(BSS_INFO_BASIC); + tlv.len = cpu_to_le16(sizeof(tlv)); + tlv.network_type = cpu_to_le32(bss_info->network_type); + tlv.active = bss_info->enable; + tlv.bcn_interval = cpu_to_le16(bss_info->bcn_interval); + memcpy(tlv.bssid, bss_info->bssid, ETH_ALEN); + tlv.wmm_idx = bss_info->wmm_idx; + tlv.dtim_period = bss_info->dtim_period; + tlv.bmc_tx_wlan_idx = bss_info->bmc_tx_wlan_idx; + + memcpy(skb_put(skb, sizeof(tlv)), &tlv, sizeof(tlv)); +} + +static void bss_info_ext_bss_handler (struct mt7615_dev *dev, + struct bss_info *bss_info, + struct sk_buff *skb) +{ +/* SIFS 20us + 512 byte beacon tranmitted by 1Mbps (3906us) */ +#define BCN_TX_ESTIMATE_TIME (4096 + 20) + struct bss_info_ext_bss tlv = {0}; + int ext_bss_idx; + + ext_bss_idx = bss_info->omac_idx - EXT_BSSID_START; + + if (ext_bss_idx < 0) + return; + + tlv.tag = cpu_to_le16(BSS_INFO_EXT_BSS); + tlv.len = cpu_to_le16(sizeof(tlv)); + tlv.mbss_tsf_offset = ext_bss_idx * BCN_TX_ESTIMATE_TIME; + + memcpy(skb_put(skb, sizeof(tlv)), &tlv, sizeof(tlv)); +} + +static struct bss_info_tag_handler bss_info_tag_handler[] = { + {BSS_INFO_OMAC, sizeof(struct bss_info_omac), bss_info_omac_handler}, + {BSS_INFO_BASIC, sizeof(struct bss_info_basic), bss_info_basic_handler}, + {BSS_INFO_RF_CH, sizeof(struct bss_info_rf_ch), NULL}, + {BSS_INFO_PM, 0, NULL}, + {BSS_INFO_UAPSD, 0, NULL}, + {BSS_INFO_ROAM_DETECTION, 0, NULL}, + {BSS_INFO_LQ_RM, 0, NULL}, + {BSS_INFO_EXT_BSS, sizeof(struct bss_info_ext_bss), bss_info_ext_bss_handler}, + {BSS_INFO_BMC_INFO, 0, NULL}, + {BSS_INFO_SYNC_MODE, 0, NULL}, + {BSS_INFO_RA, 0, NULL}, + {BSS_INFO_MAX_NUM, 0, NULL}, +}; + +static int __mt7615_mcu_set_bss_info(struct mt7615_dev *dev, + struct bss_info *bss_info) +{ + struct req_hdr { + u8 bss_idx; + u8 rsv0; + __le16 tlv_num; + u8 is_tlv_append; + u8 rsv1[3]; + } __packed req_hdr = {0}; + struct sk_buff *skb; + u16 tlv_num = 0; + u32 size = 0; + int i; + + for (i = 0; i < BSS_INFO_MAX_NUM; i++) + if ((BIT(bss_info_tag_handler[i].tag) & bss_info->feature) && + bss_info_tag_handler[i].handler) { + tlv_num++; + size += bss_info_tag_handler[i].len; + } + + skb = mt7615_mcu_msg_alloc(NULL, sizeof(req_hdr) + size); + + req_hdr.bss_idx = bss_info->bss_idx; + req_hdr.tlv_num = cpu_to_le16(tlv_num); + req_hdr.is_tlv_append = tlv_num ? 1 : 0; + + memcpy(skb_put(skb, sizeof(req_hdr)), &req_hdr, sizeof(req_hdr)); + + for (i = 0; i < BSS_INFO_MAX_NUM; i++) + if ((BIT(bss_info_tag_handler[i].tag) & bss_info->feature) && + bss_info_tag_handler[i].handler) + bss_info_tag_handler[i].handler(dev, bss_info, skb); + + return mt7615_mcu_msg_send(dev, skb, MCU_EXT_CMD_BSS_INFO_UPDATE, + MCU_Q_SET, MCU_S2D_H2N, NULL); +} + +static void bss_info_convert_vif_type(enum nl80211_iftype type, + u32 *network_type, u32 *conn_type) +{ + switch (type) { + case NL80211_IFTYPE_AP: + if (network_type) + *network_type = NETWORK_INFRA; + if (conn_type) + *conn_type = CONNECTION_INFRA_AP; + break; + case NL80211_IFTYPE_STATION: + if (network_type) + *network_type = NETWORK_INFRA; + if (conn_type) + *conn_type = CONNECTION_INFRA_STA; + break; + default: + WARN_ON(1); + break; + }; +} + +int mt7615_mcu_set_bss_info(struct mt7615_dev *dev, struct ieee80211_vif *vif, + int en) +{ + struct mt7615_vif *mvif = (struct mt7615_vif *)vif->drv_priv; + struct bss_info bss_info = {0}; + u8 bmc_tx_wlan_idx = 0; + u32 network_type = 0, conn_type = 0; + + if (vif->type == NL80211_IFTYPE_AP) { + bmc_tx_wlan_idx = mvif->sta.wcid.idx; + } else if (vif->type == NL80211_IFTYPE_STATION) { + /* find the unicast entry for sta mode bmc tx */ + struct ieee80211_sta *ap_sta; + struct mt7615_sta *msta; + + rcu_read_lock(); + + ap_sta = ieee80211_find_sta(vif, vif->bss_conf.bssid); + if (!ap_sta) { + rcu_read_unlock(); + return -EINVAL; + } + + msta = (struct mt7615_sta *)ap_sta->drv_priv; + bmc_tx_wlan_idx = msta->wcid.idx; + + rcu_read_unlock(); + } else { + WARN_ON(1); + } + + bss_info_convert_vif_type(vif->type, &network_type, &conn_type); + + bss_info.bss_idx = mvif->idx; + memcpy(bss_info.bssid, vif->bss_conf.bssid, ETH_ALEN); + bss_info.omac_idx = mvif->omac_idx; + bss_info.band_idx = mvif->band_idx; + bss_info.bmc_tx_wlan_idx = bmc_tx_wlan_idx; + bss_info.wmm_idx = mvif->wmm_idx; + bss_info.network_type = network_type; + bss_info.conn_type = conn_type; + bss_info.bcn_interval = vif->bss_conf.beacon_int; + bss_info.dtim_period = vif->bss_conf.dtim_period; + bss_info.enable = en; + bss_info.feature = BIT(BSS_INFO_BASIC); + if (en) { + bss_info.feature |= BIT(BSS_INFO_OMAC); + if (mvif->omac_idx > EXT_BSSID_START) + bss_info.feature |= BIT(BSS_INFO_EXT_BSS); + } + + return __mt7615_mcu_set_bss_info(dev, &bss_info); +} + +static int __mt7615_mcu_set_wtbl(struct mt7615_dev *dev, int wlan_idx, + int operation, void *buf, int buf_len) +{ + struct req_hdr { + u8 wlan_idx; + u8 operation; + __le16 tlv_num; + u8 rsv[4]; + } __packed req_hdr = {0}; + struct tlv { + __le16 tag; + __le16 len; + u8 buf[0]; + } __packed; + struct sk_buff *skb; + u16 tlv_num = 0; + int offset = 0; + + while (offset < buf_len) { + struct tlv *tlv = (struct tlv *)((u8 *)buf + offset); + + tlv_num++; + offset += tlv->len; + } + + skb = mt7615_mcu_msg_alloc(NULL, sizeof(req_hdr) + buf_len); + + req_hdr.wlan_idx = wlan_idx; + req_hdr.operation = operation; + req_hdr.tlv_num = cpu_to_le16(tlv_num); + + memcpy(skb_put(skb, sizeof(req_hdr)), &req_hdr, sizeof(req_hdr)); + + if (buf && buf_len) + memcpy(skb_put(skb, buf_len), buf, buf_len); + + return mt7615_mcu_msg_send(dev, skb, MCU_EXT_CMD_WTBL_UPDATE, + MCU_Q_SET, MCU_S2D_H2N, NULL); +} + +static enum mt7615_cipher_type +mt7615_get_key_info(struct ieee80211_key_conf *key, u8 *key_data) +{ + if (!key || key->keylen > 32) + return MT_CIPHER_NONE; + + memcpy(key_data, key->key, key->keylen); + + switch (key->cipher) { + case WLAN_CIPHER_SUITE_WEP40: + return MT_CIPHER_WEP40; + case WLAN_CIPHER_SUITE_WEP104: + return MT_CIPHER_WEP104; + case WLAN_CIPHER_SUITE_TKIP: + /* Rx/Tx MIC keys are swapped */ + memcpy(key_data + 16, key->key + 24, 8); + memcpy(key_data + 24, key->key + 16, 8); + return MT_CIPHER_TKIP; + case WLAN_CIPHER_SUITE_CCMP: + return MT_CIPHER_AES_CCMP; + case WLAN_CIPHER_SUITE_CCMP_256: + return MT_CIPHER_CCMP_256; + case WLAN_CIPHER_SUITE_GCMP: + return MT_CIPHER_GCMP; + case WLAN_CIPHER_SUITE_GCMP_256: + return MT_CIPHER_GCMP_256; + case WLAN_CIPHER_SUITE_SMS4: + return MT_CIPHER_WAPI; + default: + return MT_CIPHER_NONE; + } +} + +int mt7615_mcu_set_wtbl_key(struct mt7615_dev *dev, int wcid, + struct ieee80211_key_conf *key, + enum set_key_cmd cmd) +{ + struct wtbl_sec_key wtbl_sec_key = {0}; + int buf_len = sizeof(struct wtbl_sec_key); + u8 cipher; + + wtbl_sec_key.tag = cpu_to_le16(WTBL_SEC_KEY); + wtbl_sec_key.len = cpu_to_le16(buf_len); + wtbl_sec_key.add = cmd; + + if (cmd == SET_KEY) { + cipher = mt7615_get_key_info(key, wtbl_sec_key.key_material); + if (cipher == MT_CIPHER_NONE && key) + return -EOPNOTSUPP; + + wtbl_sec_key.cipher_id = cipher; + wtbl_sec_key.key_id = key->keyidx; + wtbl_sec_key.key_len = key->keylen; + } else { + wtbl_sec_key.key_len = sizeof(wtbl_sec_key.key_material); + } + + return __mt7615_mcu_set_wtbl(dev, wcid, WTBL_SET, &wtbl_sec_key, + buf_len); +} + +int mt7615_mcu_add_wtbl_bmc(struct mt7615_dev *dev, struct ieee80211_vif *vif) +{ + struct mt7615_vif *mvif = (struct mt7615_vif *)vif->drv_priv; + struct wtbl_generic *wtbl_generic; + struct wtbl_rx *wtbl_rx; + int buf_len, ret; + u8 *buf; + + buf = kzalloc(MT7615_WTBL_UPDATE_MAX_SIZE, GFP_KERNEL); + if (!buf) + return -ENOMEM; + + wtbl_generic = (struct wtbl_generic *)buf; + buf_len = sizeof(*wtbl_generic); + wtbl_generic->tag = cpu_to_le16(WTBL_GENERIC); + wtbl_generic->len = cpu_to_le16(buf_len); + eth_broadcast_addr(wtbl_generic->peer_addr); + wtbl_generic->muar_idx = 0xe; + + wtbl_rx = (struct wtbl_rx *)(buf + buf_len); + buf_len += sizeof(*wtbl_rx); + wtbl_rx->tag = cpu_to_le16(WTBL_RX); + wtbl_rx->len = cpu_to_le16(sizeof(*wtbl_rx)); + wtbl_rx->rca1 = 1; + wtbl_rx->rca2 = 1; + wtbl_rx->rv = 1; + + ret = __mt7615_mcu_set_wtbl(dev, mvif->sta.wcid.idx, + WTBL_RESET_AND_SET, buf, buf_len); + + kfree(buf); + return ret; +} + +int mt7615_mcu_del_wtbl_bmc(struct mt7615_dev *dev, struct ieee80211_vif *vif) +{ + struct mt7615_vif *mvif = (struct mt7615_vif *)vif->drv_priv; + + return __mt7615_mcu_set_wtbl(dev, mvif->sta.wcid.idx, + WTBL_RESET_AND_SET, NULL, 0); +} + +int mt7615_mcu_add_wtbl(struct mt7615_dev *dev, struct ieee80211_vif *vif, + struct ieee80211_sta *sta) +{ + struct mt7615_vif *mvif = (struct mt7615_vif *)vif->drv_priv; + struct mt7615_sta *msta = (struct mt7615_sta *)sta->drv_priv; + struct wtbl_generic *wtbl_generic; + struct wtbl_rx *wtbl_rx; + int buf_len, ret; + u8 *buf; + + buf = kzalloc(MT7615_WTBL_UPDATE_MAX_SIZE, GFP_KERNEL); + if (!buf) + return -ENOMEM; + + wtbl_generic = (struct wtbl_generic *)buf; + buf_len = sizeof(*wtbl_generic); + wtbl_generic->tag = cpu_to_le16(WTBL_GENERIC); + wtbl_generic->len = cpu_to_le16(buf_len); + memcpy(wtbl_generic->peer_addr, sta->addr, ETH_ALEN); + wtbl_generic->muar_idx = mvif->omac_idx; + wtbl_generic->qos = sta->wme; + wtbl_generic->partial_aid = cpu_to_le16(sta->aid); + + wtbl_rx = (struct wtbl_rx *)(buf + buf_len); + buf_len += sizeof(*wtbl_rx); + wtbl_rx->tag = cpu_to_le16(WTBL_RX); + wtbl_rx->len = cpu_to_le16(sizeof(*wtbl_rx)); + wtbl_rx->rca1 = (vif->type == NL80211_IFTYPE_AP) ? 0 : 1; + wtbl_rx->rca2 = 1; + wtbl_rx->rv = 1; + + ret = __mt7615_mcu_set_wtbl(dev, msta->wcid.idx, + WTBL_RESET_AND_SET, buf, buf_len); + + kfree(buf); + return ret; +} + +int mt7615_mcu_del_wtbl(struct mt7615_dev *dev, struct ieee80211_vif *vif, + struct ieee80211_sta *sta) +{ + struct mt7615_sta *msta = (struct mt7615_sta *)sta->drv_priv; + + return __mt7615_mcu_set_wtbl(dev, msta->wcid.idx, + WTBL_RESET_AND_SET, NULL, 0); +} + +int mt7615_mcu_del_wtbl_all(struct mt7615_dev *dev) +{ + return __mt7615_mcu_set_wtbl(dev, 0, WTBL_RESET_ALL, NULL, 0); +} + +static int __mt7615_mcu_set_sta_rec(struct mt7615_dev *dev, int bss_idx, + int wlan_idx, int muar_idx, void *buf, + int buf_len) +{ + struct req_hdr { + u8 bss_idx; + u8 wlan_idx; + __le16 tlv_num; + u8 is_tlv_append; + u8 muar_idx; + u8 rsv[2]; + } __packed req_hdr = {0}; + struct tlv { + __le16 tag; + __le16 len; + u8 buf[0]; + } __packed; + struct sk_buff *skb; + u16 tlv_num = 0; + int offset = 0; + + while (offset < buf_len) { + struct tlv *tlv = (struct tlv *)((u8 *)buf + offset); + + tlv_num++; + offset += tlv->len; + } + + skb = mt7615_mcu_msg_alloc(NULL, sizeof(req_hdr) + buf_len); + + req_hdr.bss_idx = bss_idx; + req_hdr.wlan_idx = wlan_idx; + req_hdr.tlv_num = cpu_to_le16(tlv_num); + req_hdr.is_tlv_append = tlv_num ? 1 : 0; + req_hdr.muar_idx = muar_idx; + + memcpy(skb_put(skb, sizeof(req_hdr)), &req_hdr, sizeof(req_hdr)); + + if (buf && buf_len) + memcpy(skb_put(skb, buf_len), buf, buf_len); + + return mt7615_mcu_msg_send(dev, skb, MCU_EXT_CMD_STA_REC_UPDATE, + MCU_Q_SET, MCU_S2D_H2N, NULL); +} + +int mt7615_mcu_set_sta_rec_bmc(struct mt7615_dev *dev, + struct ieee80211_vif *vif, bool en) +{ + struct mt7615_vif *mvif = (struct mt7615_vif *)vif->drv_priv; + struct sta_rec_basic sta_rec_basic = {0}; + int buf_len = sizeof(struct sta_rec_basic); + + sta_rec_basic.tag = cpu_to_le16(STA_REC_BASIC); + sta_rec_basic.len = cpu_to_le16(buf_len); + sta_rec_basic.conn_type = cpu_to_le32(CONNECTION_INFRA_BC); + eth_broadcast_addr(sta_rec_basic.peer_addr); + if (en) { + sta_rec_basic.conn_state = CONN_STATE_PORT_SECURE; + sta_rec_basic.extra_info = + cpu_to_le16(EXTRA_INFO_VER | EXTRA_INFO_NEW); + } else { + sta_rec_basic.conn_state = CONN_STATE_DISCONNECT; + sta_rec_basic.extra_info = cpu_to_le16(EXTRA_INFO_VER); + } + + return __mt7615_mcu_set_sta_rec(dev, mvif->idx, mvif->sta.wcid.idx, + mvif->omac_idx, &sta_rec_basic, + buf_len); +} + +static void sta_rec_convert_vif_type(enum nl80211_iftype type, u32 *conn_type) +{ + switch (type) { + case NL80211_IFTYPE_AP: + if (conn_type) + *conn_type = CONNECTION_INFRA_STA; + break; + case NL80211_IFTYPE_STATION: + if (conn_type) + *conn_type = CONNECTION_INFRA_AP; + break; + default: + WARN_ON(1); + break; + }; +} + +int mt7615_mcu_set_sta_rec(struct mt7615_dev *dev, struct ieee80211_vif *vif, + struct ieee80211_sta *sta, bool en) +{ + struct mt7615_vif *mvif = (struct mt7615_vif *)vif->drv_priv; + struct mt7615_sta *msta = (struct mt7615_sta *)sta->drv_priv; + struct sta_rec_basic sta_rec_basic = {0}; + int buf_len = sizeof(struct sta_rec_basic); + u32 conn_type = 0; + + sta_rec_convert_vif_type(vif->type, &conn_type); + + sta_rec_basic.tag = cpu_to_le16(STA_REC_BASIC); + sta_rec_basic.len = cpu_to_le16(buf_len); + sta_rec_basic.conn_type = cpu_to_le32(conn_type); + sta_rec_basic.qos = sta->wme; + sta_rec_basic.aid = cpu_to_le16(sta->aid); + memcpy(sta_rec_basic.peer_addr, sta->addr, ETH_ALEN); + + if (en) { + sta_rec_basic.conn_state = CONN_STATE_PORT_SECURE; + sta_rec_basic.extra_info = + cpu_to_le16(EXTRA_INFO_VER | EXTRA_INFO_NEW); + } else { + sta_rec_basic.conn_state = CONN_STATE_DISCONNECT; + sta_rec_basic.extra_info = cpu_to_le16(EXTRA_INFO_VER); + } + + return __mt7615_mcu_set_sta_rec(dev, mvif->idx, msta->wcid.idx, + mvif->omac_idx, &sta_rec_basic, + buf_len); +} + +int mt7615_mcu_set_bcn(struct mt7615_dev *dev, struct ieee80211_vif *vif, + int en) +{ + struct req { + u8 omac_idx; + u8 enable; + u8 wlan_idx; + u8 band_idx; + u8 pkt_type; + u8 need_pre_tbtt_int; + __le16 csa_ie_pos; + __le16 pkt_len; + __le16 tim_ie_pos; + u8 pkt[512]; + u8 csa_cnt; + /* bss color change */ + u8 bcc_cnt; + __le16 bcc_ie_pos; + } __packed req = {0}; + struct mt7615_vif *mvif = (struct mt7615_vif *)vif->drv_priv; + struct mt76_wcid *wcid = &dev->mt76.global_wcid; + struct sk_buff *skb; + u16 tim_off, tim_len; + + skb = ieee80211_beacon_get_tim(mt76_hw(dev), vif, &tim_off, &tim_len); + + if (!skb) + return -EINVAL; + + if (skb->len > 512 - MT_TXD_SIZE) { + dev_err(dev->mt76.dev, "Bcn size limit exceed\n"); + dev_kfree_skb(skb); + return -EINVAL; + } + + mt7615_mac_write_txwi(dev, (__le32 *)(req.pkt), skb, wcid, NULL, + 0, NULL); + memcpy(req.pkt + MT_TXD_SIZE, skb->data, skb->len); + dev_kfree_skb(skb); + + req.omac_idx = mvif->omac_idx; + req.enable = en; + req.wlan_idx = wcid->idx; + req.band_idx = mvif->band_idx; + /* pky_type: 0 for bcn, 1 for tim */ + req.pkt_type = 0; + req.pkt_len = cpu_to_le16(MT_TXD_SIZE + skb->len); + req.tim_ie_pos = cpu_to_le16(MT_TXD_SIZE + tim_off); + + skb = mt7615_mcu_msg_alloc(&req, sizeof(req)); + + return mt7615_mcu_msg_send(dev, skb, MCU_EXT_CMD_BCN_OFFLOAD, + MCU_Q_SET, MCU_S2D_H2N, NULL); +} + +int mt7615_mcu_set_channel(struct mt7615_dev *dev) +{ + struct cfg80211_chan_def *chdef = &dev->mt76.chandef; + struct { + u8 control_chan; + u8 center_chan; + u8 bw; + u8 tx_streams; + u8 rx_streams_mask; + u8 switch_reason; + u8 band_idx; + /* for 80+80 only */ + u8 center_chan2; + __le16 cac_case; + u8 channel_band; + u8 rsv0; + __le32 outband_freq; + u8 txpower_drop; + u8 rsv1[3]; + u8 txpower_sku[53]; + u8 rsv2[3]; + } req = {0}; + struct sk_buff *skb; + int ret; + + req.control_chan = chdef->chan->hw_value; + req.center_chan = ieee80211_frequency_to_channel(chdef->center_freq1); + req.tx_streams = (dev->mt76.chainmask >> 8) & 0xf; + req.rx_streams_mask = dev->mt76.antenna_mask; + req.switch_reason = CH_SWITCH_NORMAL; + req.band_idx = 0; + req.center_chan2 = ieee80211_frequency_to_channel(chdef->center_freq2); + req.txpower_drop = 0; + + switch (dev->mt76.chandef.width) { + case NL80211_CHAN_WIDTH_40: + req.bw = CMD_CBW_40MHZ; + break; + case NL80211_CHAN_WIDTH_80: + req.bw = CMD_CBW_80MHZ; + break; + case NL80211_CHAN_WIDTH_80P80: + req.bw = CMD_CBW_8080MHZ; + break; + case NL80211_CHAN_WIDTH_160: + req.bw = CMD_CBW_160MHZ; + break; + case NL80211_CHAN_WIDTH_5: + req.bw = CMD_CBW_5MHZ; + break; + case NL80211_CHAN_WIDTH_10: + req.bw = CMD_CBW_10MHZ; + break; + case NL80211_CHAN_WIDTH_20_NOHT: + case NL80211_CHAN_WIDTH_20: + default: + req.bw = CMD_CBW_20MHZ; + } + + memset(req.txpower_sku, 0x3f, 49); + + skb = mt7615_mcu_msg_alloc(&req, sizeof(req)); + ret = mt7615_mcu_msg_send(dev, skb, MCU_EXT_CMD_CHANNEL_SWITCH, + MCU_Q_SET, MCU_S2D_H2N, NULL); + if (ret) + return ret; + + skb = mt7615_mcu_msg_alloc(&req, sizeof(req)); + return mt7615_mcu_msg_send(dev, skb, MCU_EXT_CMD_SET_RX_PATH, + MCU_Q_SET, MCU_S2D_H2N, NULL); +} + +int mt7615_mcu_set_ht_cap(struct mt7615_dev *dev, struct ieee80211_vif *vif, + struct ieee80211_sta *sta) +{ + struct mt7615_sta *msta = (struct mt7615_sta *)sta->drv_priv; + struct mt7615_vif *mvif = (struct mt7615_vif *)vif->drv_priv; + struct wtbl_ht *wtbl_ht; + struct wtbl_raw *wtbl_raw; + struct sta_rec_ht *sta_rec_ht; + int buf_len, ret; + u32 msk, val = 0; + u8 *buf; + + buf = kzalloc(MT7615_WTBL_UPDATE_MAX_SIZE, GFP_KERNEL); + if (!buf) + return -ENOMEM; + + /* ht basic */ + buf_len = sizeof(*wtbl_ht); + wtbl_ht = (struct wtbl_ht *)buf; + wtbl_ht->tag = cpu_to_le16(WTBL_HT); + wtbl_ht->len = cpu_to_le16(sizeof(*wtbl_ht)); + wtbl_ht->ht = 1; + wtbl_ht->ldpc = sta->ht_cap.cap & IEEE80211_HT_CAP_LDPC_CODING; + wtbl_ht->af = sta->ht_cap.ampdu_factor; + wtbl_ht->mm = sta->ht_cap.ampdu_density; + + if (sta->ht_cap.cap & IEEE80211_HT_CAP_SGI_20) + val |= MT_WTBL_W5_SHORT_GI_20; + if (sta->ht_cap.cap & IEEE80211_HT_CAP_SGI_40) + val |= MT_WTBL_W5_SHORT_GI_40; + + /* vht basic */ + if (sta->vht_cap.vht_supported) { + struct wtbl_vht *wtbl_vht; + + wtbl_vht = (struct wtbl_vht *)(buf + buf_len); + buf_len += sizeof(*wtbl_vht); + wtbl_vht->tag = cpu_to_le16(WTBL_VHT); + wtbl_vht->len = cpu_to_le16(sizeof(*wtbl_vht)); + wtbl_vht->ldpc = sta->vht_cap.cap & IEEE80211_VHT_CAP_RXLDPC; + wtbl_vht->vht = 1; + + if (sta->vht_cap.cap & IEEE80211_VHT_CAP_SHORT_GI_80) + val |= MT_WTBL_W5_SHORT_GI_80; + if (sta->vht_cap.cap & IEEE80211_VHT_CAP_SHORT_GI_160) + val |= MT_WTBL_W5_SHORT_GI_160; + } + + /* smps */ + if (sta->smps_mode == IEEE80211_SMPS_DYNAMIC) { + struct wtbl_smps *wtbl_smps; + + wtbl_smps = (struct wtbl_smps *)(buf + buf_len); + buf_len += sizeof(*wtbl_smps); + wtbl_smps->tag = cpu_to_le16(WTBL_SMPS); + wtbl_smps->len = cpu_to_le16(sizeof(*wtbl_smps)); + wtbl_smps->smps = 1; + } + + /* sgi */ + msk = MT_WTBL_W5_SHORT_GI_20 | MT_WTBL_W5_SHORT_GI_40 | + MT_WTBL_W5_SHORT_GI_80 | MT_WTBL_W5_SHORT_GI_160; + + wtbl_raw = (struct wtbl_raw *)(buf + buf_len); + buf_len += sizeof(*wtbl_raw); + wtbl_raw->tag = cpu_to_le16(WTBL_RAW_DATA); + wtbl_raw->len = cpu_to_le16(sizeof(*wtbl_raw)); + wtbl_raw->wtbl_idx = 1; + wtbl_raw->dw = 5; + wtbl_raw->msk = cpu_to_le32(~msk); + wtbl_raw->val = cpu_to_le32(val); + + ret = __mt7615_mcu_set_wtbl(dev, msta->wcid.idx, WTBL_SET, buf, + buf_len); + if (ret) { + kfree(buf); + return ret; + } + + memset(buf, 0, MT7615_WTBL_UPDATE_MAX_SIZE); + + buf_len = sizeof(*sta_rec_ht); + sta_rec_ht = (struct sta_rec_ht *)buf; + sta_rec_ht->tag = cpu_to_le16(STA_REC_HT); + sta_rec_ht->len = cpu_to_le16(sizeof(*sta_rec_ht)); + sta_rec_ht->ht_cap = cpu_to_le16(sta->ht_cap.cap); + + if (sta->vht_cap.vht_supported) { + struct sta_rec_vht *sta_rec_vht; + + sta_rec_vht = (struct sta_rec_vht *)(buf + buf_len); + buf_len += sizeof(*sta_rec_vht); + sta_rec_vht->tag = cpu_to_le16(STA_REC_VHT); + sta_rec_vht->len = cpu_to_le16(sizeof(*sta_rec_vht)); + sta_rec_vht->vht_cap = cpu_to_le32(sta->vht_cap.cap); + sta_rec_vht->vht_rx_mcs_map = + cpu_to_le16(sta->vht_cap.vht_mcs.rx_mcs_map); + sta_rec_vht->vht_tx_mcs_map = + cpu_to_le16(sta->vht_cap.vht_mcs.tx_mcs_map); + } + + ret = __mt7615_mcu_set_sta_rec(dev, mvif->idx, msta->wcid.idx, + mvif->omac_idx, buf, buf_len); + kfree(buf); + return ret; +} + +int mt7615_mcu_set_tx_ba(struct mt7615_dev *dev, + struct ieee80211_ampdu_params *params, + bool add) +{ + struct ieee80211_sta *sta = params->sta; + struct mt7615_sta *msta = (struct mt7615_sta *)sta->drv_priv; + struct mt7615_vif *mvif = msta->vif; + u8 ba_range[8] = {4, 8, 12, 24, 36, 48, 54, 64}; + u16 tid = params->tid; + u16 ba_size = params->buf_size; + u16 ssn = params->ssn; + struct wtbl_ba wtbl_ba = {0}; + struct sta_rec_ba sta_rec_ba = {0}; + int ret, buf_len; + + buf_len = sizeof(struct wtbl_ba); + + wtbl_ba.tag = cpu_to_le16(WTBL_BA); + wtbl_ba.len = cpu_to_le16(buf_len); + wtbl_ba.tid = tid; + wtbl_ba.ba_type = MT_BA_TYPE_ORIGINATOR; + + if (add) { + u8 idx; + + for (idx = 7; idx > 0; idx--) { + if (ba_size >= ba_range[idx]) + break; + } + + wtbl_ba.sn = cpu_to_le16(ssn); + wtbl_ba.ba_en = 1; + wtbl_ba.ba_winsize_idx = idx; + } + + ret = __mt7615_mcu_set_wtbl(dev, msta->wcid.idx, WTBL_SET, &wtbl_ba, + buf_len); + if (ret) + return ret; + + buf_len = sizeof(struct sta_rec_ba); + + sta_rec_ba.tag = cpu_to_le16(STA_REC_BA); + sta_rec_ba.len = cpu_to_le16(buf_len); + sta_rec_ba.tid = tid; + sta_rec_ba.ba_type = MT_BA_TYPE_ORIGINATOR; + sta_rec_ba.amsdu = params->amsdu; + sta_rec_ba.ba_en = add << tid; + sta_rec_ba.ssn = cpu_to_le16(ssn); + sta_rec_ba.winsize = cpu_to_le16(ba_size); + + return __mt7615_mcu_set_sta_rec(dev, mvif->idx, msta->wcid.idx, + mvif->omac_idx, &sta_rec_ba, buf_len); +} + +int mt7615_mcu_set_rx_ba(struct mt7615_dev *dev, + struct ieee80211_ampdu_params *params, + bool add) +{ + struct ieee80211_sta *sta = params->sta; + struct mt7615_sta *msta = (struct mt7615_sta *)sta->drv_priv; + struct mt7615_vif *mvif = msta->vif; + u16 tid = params->tid; + struct wtbl_ba wtbl_ba = {0}; + struct sta_rec_ba sta_rec_ba = {0}; + int ret, buf_len; + + buf_len = sizeof(struct sta_rec_ba); + + sta_rec_ba.tag = cpu_to_le16(STA_REC_BA); + sta_rec_ba.len = cpu_to_le16(buf_len); + sta_rec_ba.tid = tid; + sta_rec_ba.ba_type = MT_BA_TYPE_RECIPIENT; + sta_rec_ba.amsdu = params->amsdu; + sta_rec_ba.ba_en = add << tid; + sta_rec_ba.ssn = cpu_to_le16(params->ssn); + sta_rec_ba.winsize = cpu_to_le16(params->buf_size); + + ret = __mt7615_mcu_set_sta_rec(dev, mvif->idx, msta->wcid.idx, + mvif->omac_idx, &sta_rec_ba, buf_len); + if (ret || !add) + return ret; + + buf_len = sizeof(struct wtbl_ba); + + wtbl_ba.tag = cpu_to_le16(WTBL_BA); + wtbl_ba.len = cpu_to_le16(buf_len); + wtbl_ba.tid = tid; + wtbl_ba.ba_type = MT_BA_TYPE_RECIPIENT; + memcpy(wtbl_ba.peer_addr, sta->addr, ETH_ALEN); + wtbl_ba.rst_ba_tid = tid; + wtbl_ba.rst_ba_sel = RST_BA_MAC_TID_MATCH; + wtbl_ba.rst_ba_sb = 1; + + return __mt7615_mcu_set_wtbl(dev, msta->wcid.idx, WTBL_SET, + &wtbl_ba, buf_len); +} + +void mt7615_mcu_set_rates(struct mt7615_dev *dev, struct mt7615_sta *sta, + struct ieee80211_tx_rate *probe_rate, + struct ieee80211_tx_rate *rates) +{ + int wcid = sta->wcid.idx; + u32 addr = MT_WTBL_BASE + wcid * MT_WTBL_ENTRY_SIZE; + bool stbc = false; + int n_rates = sta->n_rates; + u8 bw, bw_prev, bw_idx = 0; + u16 val[4]; + u16 probe_val; + u32 w5, w27; + int i; + + if (!mt76_poll(dev, MT_WTBL_UPDATE, MT_WTBL_UPDATE_BUSY, 0, 5000)) + return; + + for (i = n_rates; i < 4; i++) + rates[i] = rates[n_rates - 1]; + + val[0] = mt7615_mac_tx_rate_val(dev, &rates[0], stbc, &bw); + bw_prev = bw; + + if (probe_rate) { + probe_val = mt7615_mac_tx_rate_val(dev, probe_rate, stbc, &bw); + if (bw) + bw_idx = 1; + else + bw_prev = 0; + } else { + probe_val = val[0]; + } + + val[1] = mt7615_mac_tx_rate_val(dev, &rates[1], stbc, &bw); + if (bw_prev) { + bw_idx = 3; + bw_prev = bw; + } + + val[2] = mt7615_mac_tx_rate_val(dev, &rates[2], stbc, &bw); + if (bw_prev) { + bw_idx = 5; + bw_prev = bw; + } + + val[3] = mt7615_mac_tx_rate_val(dev, &rates[3], stbc, &bw); + if (bw_prev) + bw_idx = 7; + + w27 = mt76_rr(dev, addr + 27 * 4); + w27 &= ~MT_WTBL_W27_CC_BW_SEL; + w27 |= FIELD_PREP(MT_WTBL_W27_CC_BW_SEL, bw); + + w5 = mt76_rr(dev, addr + 5 * 4); + w5 &= ~(MT_WTBL_W5_BW_CAP | MT_WTBL_W5_CHANGE_BW_RATE); + w5 |= FIELD_PREP(MT_WTBL_W5_BW_CAP, bw) | + FIELD_PREP(MT_WTBL_W5_CHANGE_BW_RATE, bw_idx ? bw_idx - 1 : 7); + + mt76_wr(dev, MT_WTBL_RIUCR0, w5); + + mt76_wr(dev, MT_WTBL_RIUCR1, + FIELD_PREP(MT_WTBL_RIUCR1_RATE0, probe_val) | + FIELD_PREP(MT_WTBL_RIUCR1_RATE1, val[0]) | + FIELD_PREP(MT_WTBL_RIUCR1_RATE2_LO, val[0])); + + mt76_wr(dev, MT_WTBL_RIUCR2, + FIELD_PREP(MT_WTBL_RIUCR2_RATE2_HI, val[0] >> 8) | + FIELD_PREP(MT_WTBL_RIUCR2_RATE3, val[1]) | + FIELD_PREP(MT_WTBL_RIUCR2_RATE4, val[1]) | + FIELD_PREP(MT_WTBL_RIUCR2_RATE5_LO, val[2])); + + mt76_wr(dev, MT_WTBL_RIUCR3, + FIELD_PREP(MT_WTBL_RIUCR3_RATE5_HI, val[2] >> 4) | + FIELD_PREP(MT_WTBL_RIUCR3_RATE6, val[2]) | + FIELD_PREP(MT_WTBL_RIUCR3_RATE7, val[3])); + + mt76_wr(dev, MT_WTBL_UPDATE, + FIELD_PREP(MT_WTBL_UPDATE_WLAN_IDX, wcid) | + MT_WTBL_UPDATE_RATE_UPDATE | + MT_WTBL_UPDATE_TX_COUNT_CLEAR); + + mt76_wr(dev, addr + 27 * 4, w27); + + if (!(sta->wcid.tx_info & MT_WCID_TX_INFO_SET)) + mt76_poll(dev, MT_WTBL_UPDATE, MT_WTBL_UPDATE_BUSY, 0, 5000); + + sta->rate_count = 2 * MT7615_RATE_RETRY * n_rates; + sta->wcid.tx_info |= MT_WCID_TX_INFO_SET; +} diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/mcu.h b/drivers/net/wireless/mediatek/mt76/mt7615/mcu.h new file mode 100644 index 0000000000000000000000000000000000000000..9455f8fa475d03370c30c2fd2c1580fc30c5f9c9 --- /dev/null +++ b/drivers/net/wireless/mediatek/mt76/mt7615/mcu.h @@ -0,0 +1,520 @@ +/* SPDX-License-Identifier: ISC */ +/* Copyright (C) 2019 MediaTek Inc. */ + +#ifndef __MT7615_MCU_H +#define __MT7615_MCU_H + +struct mt7615_mcu_txd { + __le32 txd[8]; + + __le16 len; + __le16 pq_id; + + u8 cid; + u8 pkt_type; + u8 set_query; /* FW don't care */ + u8 seq; + + u8 uc_d2b0_rev; + u8 ext_cid; + u8 s2d_index; + u8 ext_cid_ack; + + u32 reserved[5]; +} __packed __aligned(4); + +struct mt7615_mcu_rxd { + __le32 rxd[4]; + + __le16 len; + __le16 pkt_type_id; + + u8 eid; + u8 seq; + __le16 __rsv; + + u8 ext_eid; + u8 __rsv1[2]; + u8 s2d_index; +}; + +#define MCU_PQ_ID(p, q) (((p) << 15) | ((q) << 10)) +#define MCU_PKT_ID 0xa0 + +enum { + MCU_Q_QUERY, + MCU_Q_SET, + MCU_Q_RESERVED, + MCU_Q_NA +}; + +enum { + MCU_S2D_H2N, + MCU_S2D_C2N, + MCU_S2D_H2C, + MCU_S2D_H2CN +}; + +enum { + MCU_CMD_TARGET_ADDRESS_LEN_REQ = 0x01, + MCU_CMD_FW_START_REQ = 0x02, + MCU_CMD_INIT_ACCESS_REG = 0x3, + MCU_CMD_PATCH_START_REQ = 0x05, + MCU_CMD_PATCH_FINISH_REQ = 0x07, + MCU_CMD_PATCH_SEM_CONTROL = 0x10, + MCU_CMD_EXT_CID = 0xED, + MCU_CMD_FW_SCATTER = 0xEE, + MCU_CMD_RESTART_DL_REQ = 0xEF, +}; + +enum { + MCU_EXT_CMD_PM_STATE_CTRL = 0x07, + MCU_EXT_CMD_CHANNEL_SWITCH = 0x08, + MCU_EXT_CMD_EFUSE_BUFFER_MODE = 0x21, + MCU_EXT_CMD_STA_REC_UPDATE = 0x25, + MCU_EXT_CMD_BSS_INFO_UPDATE = 0x26, + MCU_EXT_CMD_EDCA_UPDATE = 0x27, + MCU_EXT_CMD_DEV_INFO_UPDATE = 0x2A, + MCU_EXT_CMD_WTBL_UPDATE = 0x32, + MCU_EXT_CMD_PROTECT_CTRL = 0x3e, + MCU_EXT_CMD_MAC_INIT_CTRL = 0x46, + MCU_EXT_CMD_BCN_OFFLOAD = 0x49, + MCU_EXT_CMD_SET_RX_PATH = 0x4e, +}; + +enum { + PATCH_SEM_RELEASE = 0x0, + PATCH_SEM_GET = 0x1 +}; + +enum { + PATCH_NOT_DL_SEM_FAIL = 0x0, + PATCH_IS_DL = 0x1, + PATCH_NOT_DL_SEM_SUCCESS = 0x2, + PATCH_REL_SEM_SUCCESS = 0x3 +}; + +enum { + FW_STATE_INITIAL = 0, + FW_STATE_FW_DOWNLOAD = 1, + FW_STATE_NORMAL_OPERATION = 2, + FW_STATE_NORMAL_TRX = 3, + FW_STATE_CR4_RDY = 7 +}; + +#define STA_TYPE_STA BIT(0) +#define STA_TYPE_AP BIT(1) +#define STA_TYPE_ADHOC BIT(2) +#define STA_TYPE_TDLS BIT(3) +#define STA_TYPE_WDS BIT(4) +#define STA_TYPE_BC BIT(5) + +#define NETWORK_INFRA BIT(16) +#define NETWORK_P2P BIT(17) +#define NETWORK_IBSS BIT(18) +#define NETWORK_MESH BIT(19) +#define NETWORK_BOW BIT(20) +#define NETWORK_WDS BIT(21) + +#define CONNECTION_INFRA_STA (STA_TYPE_STA | NETWORK_INFRA) +#define CONNECTION_INFRA_AP (STA_TYPE_AP | NETWORK_INFRA) +#define CONNECTION_P2P_GC (STA_TYPE_STA | NETWORK_P2P) +#define CONNECTION_P2P_GO (STA_TYPE_AP | NETWORK_P2P) +#define CONNECTION_MESH_STA (STA_TYPE_STA | NETWORK_MESH) +#define CONNECTION_MESH_AP (STA_TYPE_AP | NETWORK_MESH) +#define CONNECTION_IBSS_ADHOC (STA_TYPE_ADHOC | NETWORK_IBSS) +#define CONNECTION_TDLS (STA_TYPE_STA | NETWORK_INFRA | STA_TYPE_TDLS) +#define CONNECTION_WDS (STA_TYPE_WDS | NETWORK_WDS) +#define CONNECTION_INFRA_BC (STA_TYPE_BC | NETWORK_INFRA) + +#define CONN_STATE_DISCONNECT 0 +#define CONN_STATE_CONNECT 1 +#define CONN_STATE_PORT_SECURE 2 + +struct dev_info { + u8 omac_idx; + u8 omac_addr[ETH_ALEN]; + u8 band_idx; + u8 enable; + u32 feature; +}; + +enum { + DEV_INFO_ACTIVE, + DEV_INFO_MAX_NUM +}; + +struct bss_info { + u8 bss_idx; + u8 bssid[ETH_ALEN]; + u8 omac_idx; + u8 band_idx; + u8 bmc_tx_wlan_idx; /* for bmc tx (sta mode use uc entry) */ + u8 wmm_idx; + u32 network_type; + u32 conn_type; + u16 bcn_interval; + u8 dtim_period; + u8 enable; + u32 feature; +}; + +struct bss_info_tag_handler { + u32 tag; + u32 len; + void (*handler)(struct mt7615_dev *dev, + struct bss_info *bss_info, struct sk_buff *skb); +}; + +struct bss_info_omac { + __le16 tag; + __le16 len; + u8 hw_bss_idx; + u8 omac_idx; + u8 band_idx; + u8 rsv0; + __le32 conn_type; + u32 rsv1; +} __packed; + +struct bss_info_basic { + __le16 tag; + __le16 len; + __le32 network_type; + u8 active; + u8 rsv0; + __le16 bcn_interval; + u8 bssid[ETH_ALEN]; + u8 wmm_idx; + u8 dtim_period; + u8 bmc_tx_wlan_idx; + u8 cipher; /* not used */ + u8 phymode; /* not used */ + u8 rsv1[5]; +} __packed; + +struct bss_info_rf_ch { + __le16 tag; + __le16 len; + u8 pri_ch; + u8 central_ch0; + u8 central_ch1; + u8 bw; +} __packed; + +struct bss_info_ext_bss { + __le16 tag; + __le16 len; + __le32 mbss_tsf_offset; /* in unit of us */ + u8 rsv[8]; +} __packed; + +enum { + BSS_INFO_OMAC, + BSS_INFO_BASIC, + BSS_INFO_RF_CH, /* optional, for BT/LTE coex */ + BSS_INFO_PM, /* sta only */ + BSS_INFO_UAPSD, /* sta only */ + BSS_INFO_ROAM_DETECTION, /* obsoleted */ + BSS_INFO_LQ_RM, /* obsoleted */ + BSS_INFO_EXT_BSS, + BSS_INFO_BMC_INFO, /* for bmc rate control in CR4 */ + BSS_INFO_SYNC_MODE, /* obsoleted */ + BSS_INFO_RA, + BSS_INFO_MAX_NUM +}; + +enum { + WTBL_RESET_AND_SET = 1, + WTBL_SET, + WTBL_QUERY, + WTBL_RESET_ALL +}; + +struct wtbl_generic { + __le16 tag; + __le16 len; + u8 peer_addr[ETH_ALEN]; + u8 muar_idx; + u8 skip_tx; + u8 cf_ack; + u8 qos; + u8 mesh; + u8 adm; + __le16 partial_aid; + u8 baf_en; + u8 aad_om; +} __packed; + +struct wtbl_rx { + __le16 tag; + __le16 len; + u8 rcid; + u8 rca1; + u8 rca2; + u8 rv; + u8 rsv[4]; +} __packed; + +struct wtbl_ht { + __le16 tag; + __le16 len; + u8 ht; + u8 ldpc; + u8 af; + u8 mm; + u8 rsv[4]; +} __packed; + +struct wtbl_vht { + __le16 tag; + __le16 len; + u8 ldpc; + u8 dyn_bw; + u8 vht; + u8 txop_ps; + u8 rsv[4]; +} __packed; + +struct wtbl_tx_ps { + __le16 tag; + __le16 len; + u8 txps; + u8 rsv[3]; +} __packed; + +struct wtbl_hdr_trans { + __le16 tag; + __le16 len; + u8 to_ds; + u8 from_ds; + u8 disable_rx_trans; + u8 rsv; +} __packed; + +enum mt7615_cipher_type { + MT_CIPHER_NONE, + MT_CIPHER_WEP40, + MT_CIPHER_TKIP, + MT_CIPHER_TKIP_NO_MIC, + MT_CIPHER_AES_CCMP, + MT_CIPHER_WEP104, + MT_CIPHER_BIP_CMAC_128, + MT_CIPHER_WEP128, + MT_CIPHER_WAPI, + MT_CIPHER_CCMP_256 = 10, + MT_CIPHER_GCMP, + MT_CIPHER_GCMP_256, +}; + +struct wtbl_sec_key { + __le16 tag; + __le16 len; + u8 add; /* 0: add, 1: remove */ + u8 rkv; + u8 ikv; + u8 cipher_id; + u8 key_id; + u8 key_len; + u8 rsv[2]; + u8 key_material[32]; +} __packed; + +enum { + MT_BA_TYPE_INVALID, + MT_BA_TYPE_ORIGINATOR, + MT_BA_TYPE_RECIPIENT +}; + +enum { + RST_BA_MAC_TID_MATCH, + RST_BA_MAC_MATCH, + RST_BA_NO_MATCH +}; + +struct wtbl_ba { + __le16 tag; + __le16 len; + /* common */ + u8 tid; + u8 ba_type; + u8 rsv0[2]; + /* originator only */ + __le16 sn; + u8 ba_en; + u8 ba_winsize_idx; + __le16 ba_winsize; + /* recipient only */ + u8 peer_addr[ETH_ALEN]; + u8 rst_ba_tid; + u8 rst_ba_sel; + u8 rst_ba_sb; + u8 band_idx; + u8 rsv1[4]; +} __packed; + +struct wtbl_bf { + __le16 tag; + __le16 len; + u8 ibf; + u8 ebf; + u8 ibf_vht; + u8 ebf_vht; + u8 gid; + u8 pfmu_idx; + u8 rsv[2]; +} __packed; + +struct wtbl_smps { + __le16 tag; + __le16 len; + u8 smps; + u8 rsv[3]; +} __packed; + +struct wtbl_pn { + __le16 tag; + __le16 len; + u8 pn[6]; + u8 rsv[2]; +} __packed; + +struct wtbl_spe { + __le16 tag; + __le16 len; + u8 spe_idx; + u8 rsv[3]; +} __packed; + +struct wtbl_raw { + __le16 tag; + __le16 len; + u8 wtbl_idx; + u8 dw; + u8 rsv[2]; + __le32 msk; + __le32 val; +} __packed; + +#define MT7615_WTBL_UPDATE_MAX_SIZE (sizeof(struct wtbl_generic) + \ + sizeof(struct wtbl_rx) + \ + sizeof(struct wtbl_ht) + \ + sizeof(struct wtbl_vht) + \ + sizeof(struct wtbl_tx_ps) + \ + sizeof(struct wtbl_hdr_trans) + \ + sizeof(struct wtbl_sec_key) + \ + sizeof(struct wtbl_ba) + \ + sizeof(struct wtbl_bf) + \ + sizeof(struct wtbl_smps) + \ + sizeof(struct wtbl_pn) + \ + sizeof(struct wtbl_spe)) + +enum { + WTBL_GENERIC, + WTBL_RX, + WTBL_HT, + WTBL_VHT, + WTBL_PEER_PS, /* not used */ + WTBL_TX_PS, + WTBL_HDR_TRANS, + WTBL_SEC_KEY, + WTBL_BA, + WTBL_RDG, /* obsoleted */ + WTBL_PROTECT, /* not used */ + WTBL_CLEAR, /* not used */ + WTBL_BF, + WTBL_SMPS, + WTBL_RAW_DATA, /* debug only */ + WTBL_PN, + WTBL_SPE, + WTBL_MAX_NUM +}; + +struct sta_rec_basic { + __le16 tag; + __le16 len; + __le32 conn_type; + u8 conn_state; + u8 qos; + __le16 aid; + u8 peer_addr[ETH_ALEN]; +#define EXTRA_INFO_VER BIT(0) +#define EXTRA_INFO_NEW BIT(1) + __le16 extra_info; +} __packed; + +struct sta_rec_ht { + __le16 tag; + __le16 len; + __le16 ht_cap; + u16 rsv; +} __packed; + +struct sta_rec_vht { + __le16 tag; + __le16 len; + __le32 vht_cap; + __le16 vht_rx_mcs_map; + __le16 vht_tx_mcs_map; +} __packed; + +struct sta_rec_ba { + __le16 tag; + __le16 len; + u8 tid; + u8 ba_type; + u8 amsdu; + u8 ba_en; + __le16 ssn; + __le16 winsize; +} __packed; + +#define MT7615_STA_REC_UPDATE_MAX_SIZE (sizeof(struct sta_rec_basic) + \ + sizeof(struct sta_rec_ht) + \ + sizeof(struct sta_rec_vht)) + +enum { + STA_REC_BASIC, + STA_REC_RA, + STA_REC_RA_CMM_INFO, + STA_REC_RA_UPDATE, + STA_REC_BF, + STA_REC_AMSDU, /* for CR4 */ + STA_REC_BA, + STA_REC_RED, /* not used */ + STA_REC_TX_PROC, /* for hdr trans and CSO in CR4 */ + STA_REC_HT, + STA_REC_VHT, + STA_REC_APPS, + STA_REC_MAX_NUM +}; + +enum { + CMD_CBW_20MHZ, + CMD_CBW_40MHZ, + CMD_CBW_80MHZ, + CMD_CBW_160MHZ, + CMD_CBW_10MHZ, + CMD_CBW_5MHZ, + CMD_CBW_8080MHZ +}; + +enum { + CH_SWITCH_NORMAL = 0, + CH_SWITCH_SCAN = 3, + CH_SWITCH_MCC = 4, + CH_SWITCH_DFS = 5, + CH_SWITCH_BACKGROUND_SCAN_START = 6, + CH_SWITCH_BACKGROUND_SCAN_RUNNING = 7, + CH_SWITCH_BACKGROUND_SCAN_STOP = 8, + CH_SWITCH_SCAN_BYPASS_DPD = 9 +}; + +static inline struct sk_buff * +mt7615_mcu_msg_alloc(const void *data, int len) +{ + return mt76_mcu_msg_alloc(data, sizeof(struct mt7615_mcu_txd), + len, 0); +} + +#endif diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/mt7615.h b/drivers/net/wireless/mediatek/mt76/mt7615/mt7615.h new file mode 100644 index 0000000000000000000000000000000000000000..895c2904d7ebf6059b5c87df469e350a3b560235 --- /dev/null +++ b/drivers/net/wireless/mediatek/mt76/mt7615/mt7615.h @@ -0,0 +1,195 @@ +/* SPDX-License-Identifier: ISC */ +/* Copyright (C) 2019 MediaTek Inc. */ + +#ifndef __MT7615_H +#define __MT7615_H + +#include +#include +#include "../mt76.h" +#include "regs.h" + +#define MT7615_MAX_INTERFACES 4 +#define MT7615_WTBL_SIZE 128 +#define MT7615_WTBL_RESERVED (MT7615_WTBL_SIZE - 1) +#define MT7615_WTBL_STA (MT7615_WTBL_RESERVED - \ + MT7615_MAX_INTERFACES) + +#define MT7615_WATCHDOG_TIME 100 /* ms */ +#define MT7615_RATE_RETRY 2 + +#define MT7615_TX_RING_SIZE 1024 +#define MT7615_TX_MCU_RING_SIZE 128 +#define MT7615_TX_FWDL_RING_SIZE 128 + +#define MT7615_RX_RING_SIZE 1024 +#define MT7615_RX_MCU_RING_SIZE 512 + +#define MT7615_FIRMWARE_CR4 "mt7615_cr4.bin" +#define MT7615_FIRMWARE_N9 "mt7615_n9.bin" +#define MT7615_ROM_PATCH "mt7615_rom_patch.bin" + +#define MT7615_EEPROM_SIZE 1024 +#define MT7615_TOKEN_SIZE 4096 + +struct mt7615_vif; +struct mt7615_sta; + +enum mt7615_hw_txq_id { + MT7615_TXQ_MAIN, + MT7615_TXQ_EXT, + MT7615_TXQ_MCU, + MT7615_TXQ_FWDL, +}; + +struct mt7615_sta { + struct mt76_wcid wcid; /* must be first */ + + struct mt7615_vif *vif; + + struct ieee80211_tx_rate rates[8]; + u8 rate_count; + u8 n_rates; + + u8 rate_probe; +}; + +struct mt7615_vif { + u8 idx; + u8 omac_idx; + u8 band_idx; + u8 wmm_idx; + + struct mt7615_sta sta; +}; + +struct mt7615_dev { + struct mt76_dev mt76; /* must be first */ + u32 vif_mask; + u32 omac_mask; + + spinlock_t token_lock; + struct idr token; +}; + +enum { + HW_BSSID_0 = 0x0, + HW_BSSID_1, + HW_BSSID_2, + HW_BSSID_3, + HW_BSSID_MAX, + EXT_BSSID_START = 0x10, + EXT_BSSID_1, + EXT_BSSID_2, + EXT_BSSID_3, + EXT_BSSID_4, + EXT_BSSID_5, + EXT_BSSID_6, + EXT_BSSID_7, + EXT_BSSID_8, + EXT_BSSID_9, + EXT_BSSID_10, + EXT_BSSID_11, + EXT_BSSID_12, + EXT_BSSID_13, + EXT_BSSID_14, + EXT_BSSID_15, + EXT_BSSID_END +}; + +extern const struct ieee80211_ops mt7615_ops; +extern struct pci_driver mt7615_pci_driver; + +u32 mt7615_reg_map(struct mt7615_dev *dev, u32 addr); + +int mt7615_register_device(struct mt7615_dev *dev); +void mt7615_unregister_device(struct mt7615_dev *dev); +int mt7615_eeprom_init(struct mt7615_dev *dev); +int mt7615_dma_init(struct mt7615_dev *dev); +void mt7615_dma_cleanup(struct mt7615_dev *dev); +int mt7615_mcu_init(struct mt7615_dev *dev); +int mt7615_mcu_set_dev_info(struct mt7615_dev *dev, struct ieee80211_vif *vif, + int en); +int mt7615_mcu_set_bss_info(struct mt7615_dev *dev, struct ieee80211_vif *vif, + int en); +int mt7615_mcu_set_wtbl_key(struct mt7615_dev *dev, int wcid, + struct ieee80211_key_conf *key, + enum set_key_cmd cmd); +void mt7615_mcu_set_rates(struct mt7615_dev *dev, struct mt7615_sta *sta, + struct ieee80211_tx_rate *probe_rate, + struct ieee80211_tx_rate *rates); +int mt7615_mcu_add_wtbl_bmc(struct mt7615_dev *dev, struct ieee80211_vif *vif); +int mt7615_mcu_del_wtbl_bmc(struct mt7615_dev *dev, struct ieee80211_vif *vif); +int mt7615_mcu_add_wtbl(struct mt7615_dev *dev, struct ieee80211_vif *vif, + struct ieee80211_sta *sta); +int mt7615_mcu_del_wtbl(struct mt7615_dev *dev, struct ieee80211_vif *vif, + struct ieee80211_sta *sta); +int mt7615_mcu_del_wtbl_all(struct mt7615_dev *dev); +int mt7615_mcu_set_sta_rec_bmc(struct mt7615_dev *dev, + struct ieee80211_vif *vif, bool en); +int mt7615_mcu_set_sta_rec(struct mt7615_dev *dev, struct ieee80211_vif *vif, + struct ieee80211_sta *sta, bool en); +int mt7615_mcu_set_bcn(struct mt7615_dev *dev, struct ieee80211_vif *vif, + int en); +int mt7615_mcu_set_channel(struct mt7615_dev *dev); +int mt7615_mcu_set_wmm(struct mt7615_dev *dev, u8 queue, + const struct ieee80211_tx_queue_params *params); +int mt7615_mcu_set_tx_ba(struct mt7615_dev *dev, + struct ieee80211_ampdu_params *params, + bool add); +int mt7615_mcu_set_rx_ba(struct mt7615_dev *dev, + struct ieee80211_ampdu_params *params, + bool add); +int mt7615_mcu_set_ht_cap(struct mt7615_dev *dev, struct ieee80211_vif *vif, + struct ieee80211_sta *sta); + +static inline void mt7615_irq_enable(struct mt7615_dev *dev, u32 mask) +{ + mt76_set_irq_mask(&dev->mt76, MT_INT_MASK_CSR, 0, mask); +} + +static inline void mt7615_irq_disable(struct mt7615_dev *dev, u32 mask) +{ + mt76_set_irq_mask(&dev->mt76, MT_INT_MASK_CSR, mask, 0); +} + +u16 mt7615_mac_tx_rate_val(struct mt7615_dev *dev, + const struct ieee80211_tx_rate *rate, + bool stbc, u8 *bw); +int mt7615_mac_write_txwi(struct mt7615_dev *dev, __le32 *txwi, + struct sk_buff *skb, struct mt76_wcid *wcid, + struct ieee80211_sta *sta, int pid, + struct ieee80211_key_conf *key); +int mt7615_mac_fill_rx(struct mt7615_dev *dev, struct sk_buff *skb); +void mt7615_mac_add_txs(struct mt7615_dev *dev, void *data); +void mt7615_mac_tx_free(struct mt7615_dev *dev, struct sk_buff *skb); + +int mt7615_mcu_set_eeprom(struct mt7615_dev *dev); +int mt7615_mcu_init_mac(struct mt7615_dev *dev); +int mt7615_mcu_set_rts_thresh(struct mt7615_dev *dev, u32 val); +int mt7615_mcu_ctrl_pm_state(struct mt7615_dev *dev, int enter); +void mt7615_mcu_exit(struct mt7615_dev *dev); + +int mt7615_tx_prepare_skb(struct mt76_dev *mdev, void *txwi_ptr, + enum mt76_txq_id qid, struct mt76_wcid *wcid, + struct ieee80211_sta *sta, + struct mt76_tx_info *tx_info); + +void mt7615_tx_complete_skb(struct mt76_dev *mdev, enum mt76_txq_id qid, + struct mt76_queue_entry *e); + +void mt7615_queue_rx_skb(struct mt76_dev *mdev, enum mt76_rxq_id q, + struct sk_buff *skb); +void mt7615_rx_poll_complete(struct mt76_dev *mdev, enum mt76_rxq_id q); +void mt7615_sta_ps(struct mt76_dev *mdev, struct ieee80211_sta *sta, bool ps); +int mt7615_sta_add(struct mt76_dev *mdev, struct ieee80211_vif *vif, + struct ieee80211_sta *sta); +void mt7615_sta_assoc(struct mt76_dev *mdev, struct ieee80211_vif *vif, + struct ieee80211_sta *sta); +void mt7615_sta_remove(struct mt76_dev *mdev, struct ieee80211_vif *vif, + struct ieee80211_sta *sta); +void mt7615_mac_work(struct work_struct *work); +void mt7615_txp_skb_unmap(struct mt76_dev *dev, + struct mt76_txwi_cache *txwi); + +#endif diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/pci.c b/drivers/net/wireless/mediatek/mt76/mt7615/pci.c new file mode 100644 index 0000000000000000000000000000000000000000..11122bd2d72781a49ab32fb064bf09087c55b2ce --- /dev/null +++ b/drivers/net/wireless/mediatek/mt76/mt7615/pci.c @@ -0,0 +1,150 @@ +// SPDX-License-Identifier: ISC +/* Copyright (C) 2019 MediaTek Inc. + * + * Author: Ryder Lee + * Felix Fietkau + */ + +#include +#include +#include + +#include "mt7615.h" +#include "mac.h" + +static const struct pci_device_id mt7615_pci_device_table[] = { + { PCI_DEVICE(0x14c3, 0x7615) }, + { }, +}; + +u32 mt7615_reg_map(struct mt7615_dev *dev, u32 addr) +{ + u32 base = addr & MT_MCU_PCIE_REMAP_2_BASE; + u32 offset = addr & MT_MCU_PCIE_REMAP_2_OFFSET; + + mt76_wr(dev, MT_MCU_PCIE_REMAP_2, base); + + return MT_PCIE_REMAP_BASE_2 + offset; +} + +void mt7615_rx_poll_complete(struct mt76_dev *mdev, enum mt76_rxq_id q) +{ + struct mt7615_dev *dev = container_of(mdev, struct mt7615_dev, mt76); + + mt7615_irq_enable(dev, MT_INT_RX_DONE(q)); +} + +irqreturn_t mt7615_irq_handler(int irq, void *dev_instance) +{ + struct mt7615_dev *dev = dev_instance; + u32 intr; + + intr = mt76_rr(dev, MT_INT_SOURCE_CSR); + mt76_wr(dev, MT_INT_SOURCE_CSR, intr); + + if (!test_bit(MT76_STATE_INITIALIZED, &dev->mt76.state)) + return IRQ_NONE; + + intr &= dev->mt76.mmio.irqmask; + + if (intr & MT_INT_TX_DONE_ALL) { + mt7615_irq_disable(dev, MT_INT_TX_DONE_ALL); + tasklet_schedule(&dev->mt76.tx_tasklet); + } + + if (intr & MT_INT_RX_DONE(0)) { + mt7615_irq_disable(dev, MT_INT_RX_DONE(0)); + napi_schedule(&dev->mt76.napi[0]); + } + + if (intr & MT_INT_RX_DONE(1)) { + mt7615_irq_disable(dev, MT_INT_RX_DONE(1)); + napi_schedule(&dev->mt76.napi[1]); + } + + return IRQ_HANDLED; +} + +static int mt7615_pci_probe(struct pci_dev *pdev, + const struct pci_device_id *id) +{ + static const struct mt76_driver_ops drv_ops = { + /* txwi_size = txd size + txp size */ + .txwi_size = MT_TXD_SIZE + sizeof(struct mt7615_txp), + .txwi_flags = MT_TXWI_NO_FREE, + .tx_prepare_skb = mt7615_tx_prepare_skb, + .tx_complete_skb = mt7615_tx_complete_skb, + .rx_skb = mt7615_queue_rx_skb, + .rx_poll_complete = mt7615_rx_poll_complete, + .sta_ps = mt7615_sta_ps, + .sta_add = mt7615_sta_add, + .sta_assoc = mt7615_sta_assoc, + .sta_remove = mt7615_sta_remove, + }; + struct mt7615_dev *dev; + struct mt76_dev *mdev; + int ret; + + ret = pcim_enable_device(pdev); + if (ret) + return ret; + + ret = pcim_iomap_regions(pdev, BIT(0), pci_name(pdev)); + if (ret) + return ret; + + pci_set_master(pdev); + + ret = pci_set_dma_mask(pdev, DMA_BIT_MASK(32)); + if (ret) + return ret; + + mdev = mt76_alloc_device(&pdev->dev, sizeof(*dev), &mt7615_ops, + &drv_ops); + if (!mdev) + return -ENOMEM; + + dev = container_of(mdev, struct mt7615_dev, mt76); + mt76_mmio_init(&dev->mt76, pcim_iomap_table(pdev)[0]); + + mdev->rev = (mt76_rr(dev, MT_HW_CHIPID) << 16) | + (mt76_rr(dev, MT_HW_REV) & 0xff); + dev_dbg(mdev->dev, "ASIC revision: %04x\n", mdev->rev); + + ret = devm_request_irq(mdev->dev, pdev->irq, mt7615_irq_handler, + IRQF_SHARED, KBUILD_MODNAME, dev); + if (ret) + goto error; + + ret = mt7615_register_device(dev); + if (ret) + goto error; + + return 0; +error: + ieee80211_free_hw(mt76_hw(dev)); + return ret; +} + +static void mt7615_pci_remove(struct pci_dev *pdev) +{ + struct mt76_dev *mdev = pci_get_drvdata(pdev); + struct mt7615_dev *dev = container_of(mdev, struct mt7615_dev, mt76); + + mt7615_unregister_device(dev); +} + +struct pci_driver mt7615_pci_driver = { + .name = KBUILD_MODNAME, + .id_table = mt7615_pci_device_table, + .probe = mt7615_pci_probe, + .remove = mt7615_pci_remove, +}; + +module_pci_driver(mt7615_pci_driver); + +MODULE_DEVICE_TABLE(pci, mt7615_pci_device_table); +MODULE_FIRMWARE(MT7615_FIRMWARE_CR4); +MODULE_FIRMWARE(MT7615_FIRMWARE_N9); +MODULE_FIRMWARE(MT7615_ROM_PATCH); +MODULE_LICENSE("Dual BSD/GPL"); diff --git a/drivers/net/wireless/mediatek/mt76/mt7615/regs.h b/drivers/net/wireless/mediatek/mt76/mt7615/regs.h new file mode 100644 index 0000000000000000000000000000000000000000..70e5ace33cc393d3ccf8c94a5f01e597968f65b7 --- /dev/null +++ b/drivers/net/wireless/mediatek/mt76/mt7615/regs.h @@ -0,0 +1,203 @@ +/* SPDX-License-Identifier: ISC */ +/* Copyright (C) 2019 MediaTek Inc. */ + +#ifndef __MT7615_REGS_H +#define __MT7615_REGS_H + +#define MT_HW_REV 0x1000 +#define MT_HW_CHIPID 0x1008 +#define MT_TOP_MISC2 0x1134 +#define MT_TOP_MISC2_FW_STATE GENMASK(2, 0) + +#define MT_MCU_BASE 0x2000 +#define MT_MCU(ofs) (MT_MCU_BASE + (ofs)) + +#define MT_MCU_PCIE_REMAP_1 MT_MCU(0x500) +#define MT_MCU_PCIE_REMAP_1_OFFSET GENMASK(17, 0) +#define MT_MCU_PCIE_REMAP_1_BASE GENMASK(31, 18) +#define MT_PCIE_REMAP_BASE_1 0x40000 + +#define MT_MCU_PCIE_REMAP_2 MT_MCU(0x504) +#define MT_MCU_PCIE_REMAP_2_OFFSET GENMASK(18, 0) +#define MT_MCU_PCIE_REMAP_2_BASE GENMASK(31, 19) +#define MT_PCIE_REMAP_BASE_2 0x80000 + +#define MT_HIF_BASE 0x4000 +#define MT_HIF(ofs) (MT_HIF_BASE + (ofs)) + +#define MT_CFG_LPCR_HOST MT_HIF(0x1f0) +#define MT_CFG_LPCR_HOST_FW_OWN BIT(0) +#define MT_CFG_LPCR_HOST_DRV_OWN BIT(1) + +#define MT_INT_SOURCE_CSR MT_HIF(0x200) +#define MT_INT_MASK_CSR MT_HIF(0x204) +#define MT_DELAY_INT_CFG MT_HIF(0x210) + +#define MT_INT_RX_DONE(_n) BIT(_n) +#define MT_INT_RX_DONE_ALL GENMASK(1, 0) +#define MT_INT_TX_DONE_ALL GENMASK(7, 4) +#define MT_INT_TX_DONE(_n) BIT((_n) + 4) + +#define MT_WPDMA_GLO_CFG MT_HIF(0x208) +#define MT_WPDMA_GLO_CFG_TX_DMA_EN BIT(0) +#define MT_WPDMA_GLO_CFG_TX_DMA_BUSY BIT(1) +#define MT_WPDMA_GLO_CFG_RX_DMA_EN BIT(2) +#define MT_WPDMA_GLO_CFG_RX_DMA_BUSY BIT(3) +#define MT_WPDMA_GLO_CFG_DMA_BURST_SIZE GENMASK(5, 4) +#define MT_WPDMA_GLO_CFG_TX_WRITEBACK_DONE BIT(6) +#define MT_WPDMA_GLO_CFG_BIG_ENDIAN BIT(7) +#define MT_WPDMA_GLO_CFG_TX_BT_SIZE_BIT0 BIT(9) +#define MT_WPDMA_GLO_CFG_MULTI_DMA_EN GENMASK(11, 10) +#define MT_WPDMA_GLO_CFG_FIFO_LITTLE_ENDIAN BIT(12) +#define MT_WPDMA_GLO_CFG_TX_BT_SIZE_BIT21 GENMASK(23, 22) +#define MT_WPDMA_GLO_CFG_SW_RESET BIT(24) +#define MT_WPDMA_GLO_CFG_FIRST_TOKEN_ONLY BIT(26) +#define MT_WPDMA_GLO_CFG_OMIT_TX_INFO BIT(28) + +#define MT_WPDMA_RST_IDX MT_HIF(0x20c) + +#define MT_TX_RING_BASE MT_HIF(0x300) +#define MT_RX_RING_BASE MT_HIF(0x400) + +#define MT_WPDMA_GLO_CFG1 MT_HIF(0x500) +#define MT_WPDMA_TX_PRE_CFG MT_HIF(0x510) +#define MT_WPDMA_RX_PRE_CFG MT_HIF(0x520) +#define MT_WPDMA_ABT_CFG MT_HIF(0x530) +#define MT_WPDMA_ABT_CFG1 MT_HIF(0x534) + +#define MT_WF_PHY_BASE 0x10000 +#define MT_WF_PHY(ofs) (MT_WF_PHY_BASE + (ofs)) + +#define MT_WF_PHY_WF2_RFCTRL0 MT_WF_PHY(0x1900) +#define MT_WF_PHY_WF2_RFCTRL0_LPBCN_EN BIT(9) + +#define MT_WF_CFG_BASE 0x20200 +#define MT_WF_CFG(ofs) (MT_WF_CFG_BASE + (ofs)) + +#define MT_CFG_CCR MT_WF_CFG(0x000) +#define MT_CFG_CCR_MAC_D1_1X_GC_EN BIT(24) +#define MT_CFG_CCR_MAC_D0_1X_GC_EN BIT(25) +#define MT_CFG_CCR_MAC_D1_2X_GC_EN BIT(30) +#define MT_CFG_CCR_MAC_D0_2X_GC_EN BIT(31) + +#define MT_WF_AGG_BASE 0x20a00 +#define MT_WF_AGG(ofs) (MT_WF_AGG_BASE + (ofs)) + +#define MT_AGG_ARCR MT_WF_AGG(0x010) +#define MT_AGG_ARCR_INIT_RATE1 BIT(0) +#define MT_AGG_ARCR_RTS_RATE_THR GENMASK(12, 8) +#define MT_AGG_ARCR_RATE_DOWN_RATIO GENMASK(17, 16) +#define MT_AGG_ARCR_RATE_DOWN_RATIO_EN BIT(19) +#define MT_AGG_ARCR_RATE_UP_EXTRA_TH GENMASK(22, 20) + +#define MT_AGG_ARUCR MT_WF_AGG(0x018) +#define MT_AGG_ARDCR MT_WF_AGG(0x01c) +#define MT_AGG_ARxCR_LIMIT_SHIFT(_n) (4 * (_n)) +#define MT_AGG_ARxCR_LIMIT(_n) GENMASK(2 + \ + MT_AGG_ARxCR_LIMIT_SHIFT(_n), \ + MT_AGG_ARxCR_LIMIT_SHIFT(_n)) + +#define MT_AGG_SCR MT_WF_AGG(0x0fc) +#define MT_AGG_SCR_NLNAV_MID_PTEC_DIS BIT(3) + +#define MT_WF_TMAC_BASE 0x21000 +#define MT_WF_TMAC(ofs) (MT_WF_TMAC_BASE + (ofs)) + +#define MT_TMAC_CTCR0 MT_WF_TMAC(0x0f4) +#define MT_TMAC_CTCR0_INS_DDLMT_REFTIME GENMASK(5, 0) +#define MT_TMAC_CTCR0_INS_DDLMT_DENSITY GENMASK(15, 12) +#define MT_TMAC_CTCR0_INS_DDLMT_EN BIT(17) +#define MT_TMAC_CTCR0_INS_DDLMT_VHT_SMPDU_EN BIT(18) + +#define MT_WF_RMAC_BASE 0x21200 +#define MT_WF_RMAC(ofs) (MT_WF_RMAC_BASE + (ofs)) + +#define MT_WF_RFCR MT_WF_RMAC(0x000) +#define MT_WF_RFCR_DROP_STBC_MULTI BIT(0) +#define MT_WF_RFCR_DROP_FCSFAIL BIT(1) +#define MT_WF_RFCR_DROP_VERSION BIT(3) +#define MT_WF_RFCR_DROP_PROBEREQ BIT(4) +#define MT_WF_RFCR_DROP_MCAST BIT(5) +#define MT_WF_RFCR_DROP_BCAST BIT(6) +#define MT_WF_RFCR_DROP_MCAST_FILTERED BIT(7) +#define MT_WF_RFCR_DROP_A3_MAC BIT(8) +#define MT_WF_RFCR_DROP_A3_BSSID BIT(9) +#define MT_WF_RFCR_DROP_A2_BSSID BIT(10) +#define MT_WF_RFCR_DROP_OTHER_BEACON BIT(11) +#define MT_WF_RFCR_DROP_FRAME_REPORT BIT(12) +#define MT_WF_RFCR_DROP_CTL_RSV BIT(13) +#define MT_WF_RFCR_DROP_CTS BIT(14) +#define MT_WF_RFCR_DROP_RTS BIT(15) +#define MT_WF_RFCR_DROP_DUPLICATE BIT(16) +#define MT_WF_RFCR_DROP_OTHER_BSS BIT(17) +#define MT_WF_RFCR_DROP_OTHER_UC BIT(18) +#define MT_WF_RFCR_DROP_OTHER_TIM BIT(19) +#define MT_WF_RFCR_DROP_NDPA BIT(20) +#define MT_WF_RFCR_DROP_UNWANTED_CTL BIT(21) + +#define MT_WF_DMA_BASE 0x21800 +#define MT_WF_DMA(ofs) (MT_WF_DMA_BASE + (ofs)) + +#define MT_DMA_DCR0 MT_WF_DMA(0x000) +#define MT_DMA_DCR0_MAX_RX_LEN GENMASK(15, 2) +#define MT_DMA_DCR0_RX_VEC_DROP BIT(17) + +#define MT_WTBL_BASE 0x30000 +#define MT_WTBL_ENTRY_SIZE 256 + +#define MT_WTBL_OFF_BASE 0x23400 +#define MT_WTBL_OFF(n) (MT_WTBL_OFF_BASE + (n)) + +#define MT_WTBL_UPDATE MT_WTBL_OFF(0x030) +#define MT_WTBL_UPDATE_WLAN_IDX GENMASK(7, 0) +#define MT_WTBL_UPDATE_RATE_UPDATE BIT(13) +#define MT_WTBL_UPDATE_TX_COUNT_CLEAR BIT(14) +#define MT_WTBL_UPDATE_BUSY BIT(31) + +#define MT_WTBL_ON_BASE 0x23000 +#define MT_WTBL_ON(_n) (MT_WTBL_ON_BASE + (_n)) + +#define MT_WTBL_RIUCR0 MT_WTBL_ON(0x020) + +#define MT_WTBL_RIUCR1 MT_WTBL_ON(0x024) +#define MT_WTBL_RIUCR1_RATE0 GENMASK(11, 0) +#define MT_WTBL_RIUCR1_RATE1 GENMASK(23, 12) +#define MT_WTBL_RIUCR1_RATE2_LO GENMASK(31, 24) + +#define MT_WTBL_RIUCR2 MT_WTBL_ON(0x028) +#define MT_WTBL_RIUCR2_RATE2_HI GENMASK(3, 0) +#define MT_WTBL_RIUCR2_RATE3 GENMASK(15, 4) +#define MT_WTBL_RIUCR2_RATE4 GENMASK(27, 16) +#define MT_WTBL_RIUCR2_RATE5_LO GENMASK(31, 28) + +#define MT_WTBL_RIUCR3 MT_WTBL_ON(0x02c) +#define MT_WTBL_RIUCR3_RATE5_HI GENMASK(7, 0) +#define MT_WTBL_RIUCR3_RATE6 GENMASK(19, 8) +#define MT_WTBL_RIUCR3_RATE7 GENMASK(31, 20) + +#define MT_WTBL_W5_CHANGE_BW_RATE GENMASK(7, 5) +#define MT_WTBL_W5_SHORT_GI_20 BIT(8) +#define MT_WTBL_W5_SHORT_GI_40 BIT(9) +#define MT_WTBL_W5_SHORT_GI_80 BIT(10) +#define MT_WTBL_W5_SHORT_GI_160 BIT(11) +#define MT_WTBL_W5_BW_CAP GENMASK(13, 12) +#define MT_WTBL_W27_CC_BW_SEL GENMASK(6, 5) + +#define MT_EFUSE_BASE 0x81070000 +#define MT_EFUSE_BASE_CTRL 0x000 +#define MT_EFUSE_BASE_CTRL_EMPTY BIT(30) + +#define MT_EFUSE_CTRL 0x008 +#define MT_EFUSE_CTRL_AOUT GENMASK(5, 0) +#define MT_EFUSE_CTRL_MODE GENMASK(7, 6) +#define MT_EFUSE_CTRL_LDO_OFF_TIME GENMASK(13, 8) +#define MT_EFUSE_CTRL_LDO_ON_TIME GENMASK(15, 14) +#define MT_EFUSE_CTRL_AIN GENMASK(25, 16) +#define MT_EFUSE_CTRL_VALID BIT(29) +#define MT_EFUSE_CTRL_KICK BIT(30) +#define MT_EFUSE_CTRL_SEL BIT(31) + +#define MT_EFUSE_WDATA(_i) (0x010 + ((_i) * 4)) +#define MT_EFUSE_RDATA(_i) (0x030 + ((_i) * 4)) + +#endif diff --git a/drivers/net/wireless/mediatek/mt76/mt76x0/init.c b/drivers/net/wireless/mediatek/mt76/mt76x0/init.c index bcb72e019fd2b8c435a960ffbf8345f9d18d45b4..57e46d57b44948f18583a894984e5e5bee995650 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x0/init.c +++ b/drivers/net/wireless/mediatek/mt76/mt76x0/init.c @@ -259,7 +259,6 @@ int mt76x0_init_hardware(struct mt76x02_dev *dev) return ret; mt76x0_phy_init(dev); - mt76x02_init_beacon_config(dev); return 0; } @@ -281,6 +280,7 @@ mt76x0_init_txpower(struct mt76x02_dev *dev, mt76x0_get_power_info(dev, chan, &tp); chan->max_power = (mt76x02_get_max_rate_power(&t) + tp) / 2; + chan->orig_mpwr = chan->max_power; } } diff --git a/drivers/net/wireless/mediatek/mt76/mt76x0/main.c b/drivers/net/wireless/mediatek/mt76/mt76x0/main.c index fee16ab21edb20c0dfa2459ed45dba304b83428c..691984037f98183782db707b8021cec3cf169e8e 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x0/main.c +++ b/drivers/net/wireless/mediatek/mt76/mt76x0/main.c @@ -22,10 +22,9 @@ mt76x0_set_channel(struct mt76x02_dev *dev, struct cfg80211_chan_def *chandef) int ret; cancel_delayed_work_sync(&dev->cal_work); - if (mt76_is_mmio(dev)) { - tasklet_disable(&dev->pre_tbtt_tasklet); + dev->beacon_ops->pre_tbtt_enable(dev, false); + if (mt76_is_mmio(dev)) tasklet_disable(&dev->dfs_pd.dfs_tasklet); - } mt76_set_channel(&dev->mt76); ret = mt76x0_phy_set_channel(dev, chandef); @@ -38,9 +37,10 @@ mt76x0_set_channel(struct mt76x02_dev *dev, struct cfg80211_chan_def *chandef) if (mt76_is_mmio(dev)) { mt76x02_dfs_init_params(dev); - tasklet_enable(&dev->pre_tbtt_tasklet); tasklet_enable(&dev->dfs_pd.dfs_tasklet); } + dev->beacon_ops->pre_tbtt_enable(dev, true); + mt76_txq_schedule_all(&dev->mt76); return ret; diff --git a/drivers/net/wireless/mediatek/mt76/mt76x0/pci.c b/drivers/net/wireless/mediatek/mt76/mt76x0/pci.c index f302162036d0032f6eb3de51b78e07a47e55a886..4585e1b756c2096d93321e9d3aa8da1f2ff25d49 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x0/pci.c +++ b/drivers/net/wireless/mediatek/mt76/mt76x0/pci.c @@ -25,25 +25,21 @@ static int mt76x0e_start(struct ieee80211_hw *hw) { struct mt76x02_dev *dev = hw->priv; - mutex_lock(&dev->mt76.mutex); - mt76x02_mac_start(dev); mt76x0_phy_calibrate(dev, true); - ieee80211_queue_delayed_work(dev->mt76.hw, &dev->mac_work, + ieee80211_queue_delayed_work(dev->mt76.hw, &dev->mt76.mac_work, MT_MAC_WORK_INTERVAL); ieee80211_queue_delayed_work(dev->mt76.hw, &dev->cal_work, MT_CALIBRATE_INTERVAL); set_bit(MT76_STATE_RUNNING, &dev->mt76.state); - mutex_unlock(&dev->mt76.mutex); - return 0; } static void mt76x0e_stop_hw(struct mt76x02_dev *dev) { cancel_delayed_work_sync(&dev->cal_work); - cancel_delayed_work_sync(&dev->mac_work); + cancel_delayed_work_sync(&dev->mt76.mac_work); if (!mt76_poll(dev, MT_WPDMA_GLO_CFG, MT_WPDMA_GLO_CFG_TX_DMA_BUSY, 0, 1000)) @@ -62,10 +58,8 @@ static void mt76x0e_stop(struct ieee80211_hw *hw) { struct mt76x02_dev *dev = hw->priv; - mutex_lock(&dev->mt76.mutex); clear_bit(MT76_STATE_RUNNING, &dev->mt76.state); mt76x0e_stop_hw(dev); - mutex_unlock(&dev->mt76.mutex); } static void @@ -74,13 +68,6 @@ mt76x0e_flush(struct ieee80211_hw *hw, struct ieee80211_vif *vif, { } -static int -mt76x0e_set_tim(struct ieee80211_hw *hw, struct ieee80211_sta *sta, - bool set) -{ - return 0; -} - static const struct ieee80211_ops mt76x0e_ops = { .tx = mt76x02_tx, .start = mt76x0e_start, @@ -101,7 +88,7 @@ static const struct ieee80211_ops mt76x0e_ops = { .get_survey = mt76_get_survey, .get_txpower = mt76_get_txpower, .flush = mt76x0e_flush, - .set_tim = mt76x0e_set_tim, + .set_tim = mt76_set_tim, .release_buffered_frames = mt76_release_buffered_frames, .set_coverage_class = mt76x02_set_coverage_class, .set_rts_threshold = mt76x02_set_rts_threshold, @@ -128,6 +115,8 @@ static int mt76x0e_register_device(struct mt76x02_dev *dev) if (err < 0) return err; + mt76x02e_init_beacon_config(dev); + if (mt76_chip(&dev->mt76) == 0x7610) { u16 val; @@ -164,6 +153,7 @@ mt76x0e_probe(struct pci_dev *pdev, const struct pci_device_id *id) { static const struct mt76_driver_ops drv_ops = { .txwi_size = sizeof(struct mt76x02_txwi), + .tx_aligned4_skbs = true, .update_survey = mt76x02_update_channel, .tx_prepare_skb = mt76x02_tx_prepare_skb, .tx_complete_skb = mt76x02_tx_complete_skb, @@ -223,7 +213,7 @@ error: static void mt76x0e_cleanup(struct mt76x02_dev *dev) { clear_bit(MT76_STATE_INITIALIZED, &dev->mt76.state); - tasklet_disable(&dev->pre_tbtt_tasklet); + tasklet_disable(&dev->mt76.pre_tbtt_tasklet); mt76x0_chip_onoff(dev, false, false); mt76x0e_stop_hw(dev); mt76x02_dma_cleanup(dev); @@ -238,7 +228,7 @@ mt76x0e_remove(struct pci_dev *pdev) mt76_unregister_device(mdev); mt76x0e_cleanup(dev); - ieee80211_free_hw(mdev->hw); + mt76_free_device(mdev); } static const struct pci_device_id mt76x0e_device_table[] = { diff --git a/drivers/net/wireless/mediatek/mt76/mt76x0/usb.c b/drivers/net/wireless/mediatek/mt76/mt76x0/usb.c index e5a06f74a6f701419f703844f4e73fba6307a627..7c38ec4418db23c35cba39e9e5ad55b8118eb22b 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x0/usb.c +++ b/drivers/net/wireless/mediatek/mt76/mt76x0/usb.c @@ -81,20 +81,19 @@ static void mt76x0u_cleanup(struct mt76x02_dev *dev) mt76u_queues_deinit(&dev->mt76); } -static void mt76x0u_mac_stop(struct mt76x02_dev *dev) +static void mt76x0u_stop(struct ieee80211_hw *hw) { + struct mt76x02_dev *dev = hw->priv; + clear_bit(MT76_STATE_RUNNING, &dev->mt76.state); cancel_delayed_work_sync(&dev->cal_work); - cancel_delayed_work_sync(&dev->mac_work); - mt76u_stop_stat_wk(&dev->mt76); + cancel_delayed_work_sync(&dev->mt76.mac_work); + mt76u_stop_tx(&dev->mt76); + mt76x02u_exit_beacon_config(dev); if (test_bit(MT76_REMOVED, &dev->mt76.state)) return; - mt76_clear(dev, MT_BEACON_TIME_CFG, MT_BEACON_TIME_CFG_TIMER_EN | - MT_BEACON_TIME_CFG_SYNC_MODE | MT_BEACON_TIME_CFG_TBTT_EN | - MT_BEACON_TIME_CFG_BEACON_TX); - if (!mt76_poll(dev, MT_USB_DMA_CFG, MT_USB_DMA_CFG_TX_BUSY, 0, 1000)) dev_warn(dev->mt76.dev, "TX DMA did not stop\n"); @@ -109,31 +108,17 @@ static int mt76x0u_start(struct ieee80211_hw *hw) struct mt76x02_dev *dev = hw->priv; int ret; - mutex_lock(&dev->mt76.mutex); - ret = mt76x0_mac_start(dev); if (ret) - goto out; + return ret; mt76x0_phy_calibrate(dev, true); - ieee80211_queue_delayed_work(dev->mt76.hw, &dev->mac_work, + ieee80211_queue_delayed_work(dev->mt76.hw, &dev->mt76.mac_work, MT_MAC_WORK_INTERVAL); ieee80211_queue_delayed_work(dev->mt76.hw, &dev->cal_work, MT_CALIBRATE_INTERVAL); set_bit(MT76_STATE_RUNNING, &dev->mt76.state); - -out: - mutex_unlock(&dev->mt76.mutex); - return ret; -} - -static void mt76x0u_stop(struct ieee80211_hw *hw) -{ - struct mt76x02_dev *dev = hw->priv; - - mutex_lock(&dev->mt76.mutex); - mt76x0u_mac_stop(dev); - mutex_unlock(&dev->mt76.mutex); + return 0; } static const struct ieee80211_ops mt76x0u_ops = { @@ -155,6 +140,8 @@ static const struct ieee80211_ops mt76x0u_ops = { .set_rts_threshold = mt76x02_set_rts_threshold, .wake_tx_queue = mt76_wake_tx_queue, .get_txpower = mt76_get_txpower, + .set_tim = mt76_set_tim, + .release_buffered_frames = mt76_release_buffered_frames, }; static int mt76x0u_init_hardware(struct mt76x02_dev *dev) @@ -175,6 +162,8 @@ static int mt76x0u_init_hardware(struct mt76x02_dev *dev) if (err < 0) return err; + mt76x02u_init_beacon_config(dev); + mt76_rmw(dev, MT_US_CYC_CFG, MT_US_CYC_CNT, 0x1e); mt76_wr(dev, MT_TXOP_CTRL_CFG, FIELD_PREP(MT_TXOP_TRUN_EN, 0x3f) | @@ -223,6 +212,7 @@ static int mt76x0u_probe(struct usb_interface *usb_intf, .tx_complete_skb = mt76x02u_tx_complete_skb, .tx_status_data = mt76x02_tx_status_data, .rx_skb = mt76x02_queue_rx_skb, + .sta_ps = mt76x02_sta_ps, .sta_add = mt76x02_sta_add, .sta_remove = mt76x02_sta_remove, }; @@ -232,7 +222,7 @@ static int mt76x0u_probe(struct usb_interface *usb_intf, u32 mac_rev; int ret; - mdev = mt76_alloc_device(&usb_intf->dev, sizeof(*dev), &mt76x0u_ops, + mdev = mt76_alloc_device(&usb_dev->dev, sizeof(*dev), &mt76x0u_ops, &drv_ops); if (!mdev) return -ENOMEM; @@ -311,8 +301,7 @@ static int __maybe_unused mt76x0_suspend(struct usb_interface *usb_intf, { struct mt76x02_dev *dev = usb_get_intfdata(usb_intf); - mt76u_stop_queues(&dev->mt76); - mt76x0u_mac_stop(dev); + mt76u_stop_rx(&dev->mt76); clear_bit(MT76_STATE_MCU_RUNNING, &dev->mt76.state); mt76x0_chip_onoff(dev, false, false); @@ -322,16 +311,12 @@ static int __maybe_unused mt76x0_suspend(struct usb_interface *usb_intf, static int __maybe_unused mt76x0_resume(struct usb_interface *usb_intf) { struct mt76x02_dev *dev = usb_get_intfdata(usb_intf); - struct mt76_usb *usb = &dev->mt76.usb; int ret; - ret = mt76u_submit_rx_buffers(&dev->mt76); + ret = mt76u_resume_rx(&dev->mt76); if (ret < 0) goto err; - tasklet_enable(&usb->rx_tasklet); - tasklet_enable(&usb->tx_tasklet); - ret = mt76x0u_init_hardware(dev); if (ret) goto err; diff --git a/drivers/net/wireless/mediatek/mt76/mt76x02.h b/drivers/net/wireless/mediatek/mt76/mt76x02.h index 07061eb4d1e1b3ef97b7131791af79a0962b6360..687bd14b2d776037bad703672624d8a60e45f1de 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x02.h +++ b/drivers/net/wireless/mediatek/mt76/mt76x02.h @@ -68,6 +68,13 @@ struct mt76x02_calibration { s8 tssi_dc; }; +struct mt76x02_beacon_ops { + unsigned int nslots; + unsigned int slot_size; + void (*pre_tbtt_enable) (struct mt76x02_dev *, bool); + void (*beacon_enable) (struct mt76x02_dev *, bool); +}; + struct mt76x02_dev { struct mt76_dev mt76; /* must be first */ @@ -79,23 +86,25 @@ struct mt76x02_dev { u8 txdone_seq; DECLARE_KFIFO_PTR(txstatus_fifo, struct mt76x02_tx_status); + spinlock_t txstatus_fifo_lock; struct sk_buff *rx_head; - struct tasklet_struct tx_tasklet; - struct tasklet_struct pre_tbtt_tasklet; + struct napi_struct tx_napi; struct delayed_work cal_work; - struct delayed_work mac_work; struct delayed_work wdt_work; + struct hrtimer pre_tbtt_timer; + struct work_struct pre_tbtt_work; + + const struct mt76x02_beacon_ops *beacon_ops; + u32 aggr_stats[32]; struct sk_buff *beacons[8]; - u8 beacon_mask; u8 beacon_data_mask; u8 tbtt_count; - u16 beacon_int; u32 tx_hang_reset; u8 tx_hang_check; @@ -163,7 +172,6 @@ void mt76x02_set_tx_ackto(struct mt76x02_dev *dev); void mt76x02_set_coverage_class(struct ieee80211_hw *hw, s16 coverage_class); int mt76x02_set_rts_threshold(struct ieee80211_hw *hw, u32 val); -int mt76x02_insert_hdr_pad(struct sk_buff *skb); void mt76x02_remove_hdr_pad(struct sk_buff *skb, int len); bool mt76x02_tx_status_data(struct mt76_dev *mdev, u8 *update); void mt76x02_queue_rx_skb(struct mt76_dev *mdev, enum mt76_rxq_id q, @@ -173,9 +181,9 @@ irqreturn_t mt76x02_irq_handler(int irq, void *dev_instance); void mt76x02_tx(struct ieee80211_hw *hw, struct ieee80211_tx_control *control, struct sk_buff *skb); int mt76x02_tx_prepare_skb(struct mt76_dev *mdev, void *txwi, - struct sk_buff *skb, struct mt76_queue *q, - struct mt76_wcid *wcid, struct ieee80211_sta *sta, - u32 *tx_info); + enum mt76_txq_id qid, struct mt76_wcid *wcid, + struct ieee80211_sta *sta, + struct mt76_tx_info *tx_info); void mt76x02_sw_scan(struct ieee80211_hw *hw, struct ieee80211_vif *vif, const u8 *mac); void mt76x02_sw_scan_complete(struct ieee80211_hw *hw, @@ -185,9 +193,19 @@ void mt76x02_bss_info_changed(struct ieee80211_hw *hw, struct ieee80211_vif *vif, struct ieee80211_bss_conf *info, u32 changed); -extern const u16 mt76x02_beacon_offsets[16]; +struct beacon_bc_data { + struct mt76x02_dev *dev; + struct sk_buff_head q; + struct sk_buff *tail[8]; +}; void mt76x02_init_beacon_config(struct mt76x02_dev *dev); -void mt76x02_set_irq_mask(struct mt76x02_dev *dev, u32 clear, u32 set); +void mt76x02e_init_beacon_config(struct mt76x02_dev *dev); +void mt76x02_resync_beacon_timer(struct mt76x02_dev *dev); +void mt76x02_update_beacon_iter(void *priv, u8 *mac, struct ieee80211_vif *vif); +void mt76x02_enqueue_buffered_bc(struct mt76x02_dev *dev, + struct beacon_bc_data *data, + int max_nframes); + void mt76x02_mac_start(struct mt76x02_dev *dev); void mt76x02_init_debugfs(struct mt76x02_dev *dev); @@ -208,12 +226,12 @@ static inline bool is_mt76x2(struct mt76x02_dev *dev) static inline void mt76x02_irq_enable(struct mt76x02_dev *dev, u32 mask) { - mt76x02_set_irq_mask(dev, 0, mask); + mt76_set_irq_mask(&dev->mt76, MT_INT_MASK_CSR, 0, mask); } static inline void mt76x02_irq_disable(struct mt76x02_dev *dev, u32 mask) { - mt76x02_set_irq_mask(dev, mask, 0); + mt76_set_irq_mask(&dev->mt76, MT_INT_MASK_CSR, mask, 0); } static inline bool diff --git a/drivers/net/wireless/mediatek/mt76/mt76x02_beacon.c b/drivers/net/wireless/mediatek/mt76/mt76x02_beacon.c new file mode 100644 index 0000000000000000000000000000000000000000..e196b9c0a6861545facb602d744171662edb48f8 --- /dev/null +++ b/drivers/net/wireless/mediatek/mt76/mt76x02_beacon.c @@ -0,0 +1,286 @@ +/* + * Copyright (C) 2016 Felix Fietkau + * Copyright (C) 2018 Lorenzo Bianconi + * Copyright (C) 2018 Stanislaw Gruszka + * + * Permission to use, copy, modify, and/or distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +#include "mt76x02.h" + +static void mt76x02_set_beacon_offsets(struct mt76x02_dev *dev) +{ + u32 regs[4] = {}; + u16 val; + int i; + + for (i = 0; i < dev->beacon_ops->nslots; i++) { + val = i * dev->beacon_ops->slot_size; + regs[i / 4] |= (val / 64) << (8 * (i % 4)); + } + + for (i = 0; i < 4; i++) + mt76_wr(dev, MT_BCN_OFFSET(i), regs[i]); +} + +static int +mt76x02_write_beacon(struct mt76x02_dev *dev, int offset, struct sk_buff *skb) +{ + int beacon_len = dev->beacon_ops->slot_size; + struct mt76x02_txwi txwi; + + if (WARN_ON_ONCE(beacon_len < skb->len + sizeof(struct mt76x02_txwi))) + return -ENOSPC; + + mt76x02_mac_write_txwi(dev, &txwi, skb, NULL, NULL, skb->len); + + mt76_wr_copy(dev, offset, &txwi, sizeof(txwi)); + offset += sizeof(txwi); + + mt76_wr_copy(dev, offset, skb->data, skb->len); + return 0; +} + +static int +__mt76x02_mac_set_beacon(struct mt76x02_dev *dev, u8 bcn_idx, + struct sk_buff *skb) +{ + int beacon_len = dev->beacon_ops->slot_size; + int beacon_addr = MT_BEACON_BASE + (beacon_len * bcn_idx); + int ret = 0; + int i; + + /* Prevent corrupt transmissions during update */ + mt76_set(dev, MT_BCN_BYPASS_MASK, BIT(bcn_idx)); + + if (skb) { + ret = mt76x02_write_beacon(dev, beacon_addr, skb); + if (!ret) + dev->beacon_data_mask |= BIT(bcn_idx); + } else { + dev->beacon_data_mask &= ~BIT(bcn_idx); + for (i = 0; i < beacon_len; i += 4) + mt76_wr(dev, beacon_addr + i, 0); + } + + mt76_wr(dev, MT_BCN_BYPASS_MASK, 0xff00 | ~dev->beacon_data_mask); + + return ret; +} + +int mt76x02_mac_set_beacon(struct mt76x02_dev *dev, u8 vif_idx, + struct sk_buff *skb) +{ + bool force_update = false; + int bcn_idx = 0; + int i; + + for (i = 0; i < ARRAY_SIZE(dev->beacons); i++) { + if (vif_idx == i) { + force_update = !!dev->beacons[i] ^ !!skb; + + if (dev->beacons[i]) + dev_kfree_skb(dev->beacons[i]); + + dev->beacons[i] = skb; + __mt76x02_mac_set_beacon(dev, bcn_idx, skb); + } else if (force_update && dev->beacons[i]) { + __mt76x02_mac_set_beacon(dev, bcn_idx, + dev->beacons[i]); + } + + bcn_idx += !!dev->beacons[i]; + } + + for (i = bcn_idx; i < ARRAY_SIZE(dev->beacons); i++) { + if (!(dev->beacon_data_mask & BIT(i))) + break; + + __mt76x02_mac_set_beacon(dev, i, NULL); + } + + mt76_rmw_field(dev, MT_MAC_BSSID_DW1, MT_MAC_BSSID_DW1_MBEACON_N, + bcn_idx - 1); + return 0; +} +EXPORT_SYMBOL_GPL(mt76x02_mac_set_beacon); + +static void +__mt76x02_mac_set_beacon_enable(struct mt76x02_dev *dev, u8 vif_idx, + bool val, struct sk_buff *skb) +{ + u8 old_mask = dev->mt76.beacon_mask; + bool en; + u32 reg; + + if (val) { + dev->mt76.beacon_mask |= BIT(vif_idx); + if (skb) + mt76x02_mac_set_beacon(dev, vif_idx, skb); + } else { + dev->mt76.beacon_mask &= ~BIT(vif_idx); + mt76x02_mac_set_beacon(dev, vif_idx, NULL); + } + + if (!!old_mask == !!dev->mt76.beacon_mask) + return; + + en = dev->mt76.beacon_mask; + + reg = MT_BEACON_TIME_CFG_BEACON_TX | + MT_BEACON_TIME_CFG_TBTT_EN | + MT_BEACON_TIME_CFG_TIMER_EN; + mt76_rmw(dev, MT_BEACON_TIME_CFG, reg, reg * en); + + dev->beacon_ops->beacon_enable(dev, en); +} + +void mt76x02_mac_set_beacon_enable(struct mt76x02_dev *dev, + struct ieee80211_vif *vif, bool val) +{ + u8 vif_idx = ((struct mt76x02_vif *)vif->drv_priv)->idx; + struct sk_buff *skb = NULL; + + dev->beacon_ops->pre_tbtt_enable(dev, false); + + if (mt76_is_usb(dev)) + skb = ieee80211_beacon_get(mt76_hw(dev), vif); + + if (!dev->mt76.beacon_mask) + dev->tbtt_count = 0; + + __mt76x02_mac_set_beacon_enable(dev, vif_idx, val, skb); + + dev->beacon_ops->pre_tbtt_enable(dev, true); +} + +void +mt76x02_resync_beacon_timer(struct mt76x02_dev *dev) +{ + u32 timer_val = dev->mt76.beacon_int << 4; + + dev->tbtt_count++; + + /* + * Beacon timer drifts by 1us every tick, the timer is configured + * in 1/16 TU (64us) units. + */ + if (dev->tbtt_count < 63) + return; + + /* + * The updated beacon interval takes effect after two TBTT, because + * at this point the original interval has already been loaded into + * the next TBTT_TIMER value + */ + if (dev->tbtt_count == 63) + timer_val -= 1; + + mt76_rmw_field(dev, MT_BEACON_TIME_CFG, + MT_BEACON_TIME_CFG_INTVAL, timer_val); + + if (dev->tbtt_count >= 64) { + dev->tbtt_count = 0; + return; + } +} +EXPORT_SYMBOL_GPL(mt76x02_resync_beacon_timer); + +void +mt76x02_update_beacon_iter(void *priv, u8 *mac, struct ieee80211_vif *vif) +{ + struct mt76x02_dev *dev = (struct mt76x02_dev *)priv; + struct mt76x02_vif *mvif = (struct mt76x02_vif *)vif->drv_priv; + struct sk_buff *skb = NULL; + + if (!(dev->mt76.beacon_mask & BIT(mvif->idx))) + return; + + skb = ieee80211_beacon_get(mt76_hw(dev), vif); + if (!skb) + return; + + mt76x02_mac_set_beacon(dev, mvif->idx, skb); +} +EXPORT_SYMBOL_GPL(mt76x02_update_beacon_iter); + +static void +mt76x02_add_buffered_bc(void *priv, u8 *mac, struct ieee80211_vif *vif) +{ + struct beacon_bc_data *data = priv; + struct mt76x02_dev *dev = data->dev; + struct mt76x02_vif *mvif = (struct mt76x02_vif *)vif->drv_priv; + struct ieee80211_tx_info *info; + struct sk_buff *skb; + + if (!(dev->mt76.beacon_mask & BIT(mvif->idx))) + return; + + skb = ieee80211_get_buffered_bc(mt76_hw(dev), vif); + if (!skb) + return; + + info = IEEE80211_SKB_CB(skb); + info->control.vif = vif; + info->flags |= IEEE80211_TX_CTL_ASSIGN_SEQ; + mt76_skb_set_moredata(skb, true); + __skb_queue_tail(&data->q, skb); + data->tail[mvif->idx] = skb; +} + +void +mt76x02_enqueue_buffered_bc(struct mt76x02_dev *dev, struct beacon_bc_data *data, + int max_nframes) +{ + int i, nframes; + + data->dev = dev; + __skb_queue_head_init(&data->q); + + do { + nframes = skb_queue_len(&data->q); + ieee80211_iterate_active_interfaces_atomic(mt76_hw(dev), + IEEE80211_IFACE_ITER_RESUME_ALL, + mt76x02_add_buffered_bc, data); + } while (nframes != skb_queue_len(&data->q) && + skb_queue_len(&data->q) < max_nframes); + + if (!skb_queue_len(&data->q)) + return; + + for (i = 0; i < ARRAY_SIZE(data->tail); i++) { + if (!data->tail[i]) + continue; + mt76_skb_set_moredata(data->tail[i], false); + } +} +EXPORT_SYMBOL_GPL(mt76x02_enqueue_buffered_bc); + +void mt76x02_init_beacon_config(struct mt76x02_dev *dev) +{ + int i; + + mt76_clear(dev, MT_BEACON_TIME_CFG, (MT_BEACON_TIME_CFG_TIMER_EN | + MT_BEACON_TIME_CFG_TBTT_EN | + MT_BEACON_TIME_CFG_BEACON_TX)); + mt76_set(dev, MT_BEACON_TIME_CFG, MT_BEACON_TIME_CFG_SYNC_MODE); + mt76_wr(dev, MT_BCN_BYPASS_MASK, 0xffff); + + for (i = 0; i < 8; i++) + mt76x02_mac_set_beacon(dev, i, NULL); + + mt76x02_set_beacon_offsets(dev); +} +EXPORT_SYMBOL_GPL(mt76x02_init_beacon_config); + + diff --git a/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c b/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c index 4fe5a83ca5a41713d894a4210fe5ef0d68e47e17..56510a1a843a8aaf787ad0deb4c63f6f15c59a77 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c +++ b/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c @@ -218,10 +218,17 @@ mt76x02_mac_tx_rate_val(struct mt76x02_dev *dev, void mt76x02_mac_wcid_set_rate(struct mt76x02_dev *dev, struct mt76_wcid *wcid, const struct ieee80211_tx_rate *rate) { - spin_lock_bh(&dev->mt76.lock); - wcid->tx_rate = mt76x02_mac_tx_rate_val(dev, rate, &wcid->tx_rate_nss); - wcid->tx_rate_set = true; - spin_unlock_bh(&dev->mt76.lock); + s8 max_txpwr_adj = mt76x02_tx_get_max_txpwr_adj(dev, rate); + __le16 rateval; + u32 tx_info; + s8 nss; + + rateval = mt76x02_mac_tx_rate_val(dev, rate, &nss); + tx_info = FIELD_PREP(MT_WCID_TX_INFO_RATE, rateval) | + FIELD_PREP(MT_WCID_TX_INFO_NSS, nss) | + FIELD_PREP(MT_WCID_TX_INFO_TXPWR_ADJ, max_txpwr_adj) | + MT_WCID_TX_INFO_SET; + wcid->tx_info = tx_info; } void mt76x02_mac_set_short_preamble(struct mt76x02_dev *dev, bool enable) @@ -323,6 +330,7 @@ void mt76x02_mac_write_txwi(struct mt76x02_dev *dev, struct mt76x02_txwi *txwi, struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb); struct ieee80211_tx_rate *rate = &info->control.rates[0]; struct ieee80211_key_conf *key = info->control.hw_key; + u32 wcid_tx_info; u16 rate_ht_mask = FIELD_PREP(MT_RXWI_RATE_PHY, BIT(1) | BIT(2)); u16 txwi_flags = 0; u8 nss; @@ -357,16 +365,16 @@ void mt76x02_mac_write_txwi(struct mt76x02_dev *dev, struct mt76x02_txwi *txwi, txwi->eiv = *((__le32 *)&ccmp_pn[4]); } - spin_lock_bh(&dev->mt76.lock); if (wcid && (rate->idx < 0 || !rate->count)) { - txwi->rate = wcid->tx_rate; - max_txpwr_adj = wcid->max_txpwr_adj; - nss = wcid->tx_rate_nss; + wcid_tx_info = wcid->tx_info; + txwi->rate = FIELD_GET(MT_WCID_TX_INFO_RATE, wcid_tx_info); + max_txpwr_adj = FIELD_GET(MT_WCID_TX_INFO_TXPWR_ADJ, + wcid_tx_info); + nss = FIELD_GET(MT_WCID_TX_INFO_NSS, wcid_tx_info); } else { txwi->rate = mt76x02_mac_tx_rate_val(dev, rate, &nss); max_txpwr_adj = mt76x02_tx_get_max_txpwr_adj(dev, rate); } - spin_unlock_bh(&dev->mt76.lock); txpwr_adj = mt76x02_tx_get_txpwr_adj(dev, dev->mt76.txpower_conf, max_txpwr_adj); @@ -731,7 +739,6 @@ int mt76x02_mac_process_rx(struct mt76x02_dev *dev, struct sk_buff *skb, void mt76x02_mac_poll_tx_status(struct mt76x02_dev *dev, bool irq) { struct mt76x02_tx_status stat = {}; - unsigned long flags; u8 update = 1; bool ret; @@ -741,9 +748,11 @@ void mt76x02_mac_poll_tx_status(struct mt76x02_dev *dev, bool irq) trace_mac_txstat_poll(dev); while (!irq || !kfifo_is_full(&dev->txstatus_fifo)) { - spin_lock_irqsave(&dev->mt76.mmio.irq_lock, flags); + if (!spin_trylock(&dev->txstatus_fifo_lock)) + break; + ret = mt76x02_mac_load_tx_status(dev, &stat); - spin_unlock_irqrestore(&dev->mt76.mmio.irq_lock, flags); + spin_unlock(&dev->txstatus_fifo_lock); if (!ret) break; @@ -757,11 +766,12 @@ void mt76x02_mac_poll_tx_status(struct mt76x02_dev *dev, bool irq) } } -void mt76x02_tx_complete_skb(struct mt76_dev *mdev, struct mt76_queue *q, - struct mt76_queue_entry *e, bool flush) +void mt76x02_tx_complete_skb(struct mt76_dev *mdev, enum mt76_txq_id qid, + struct mt76_queue_entry *e) { struct mt76x02_dev *dev = container_of(mdev, struct mt76x02_dev, mt76); struct mt76x02_txwi *txwi; + u8 *txwi_ptr; if (!e->txwi) { dev_kfree_skb_any(e->skb); @@ -770,7 +780,8 @@ void mt76x02_tx_complete_skb(struct mt76_dev *mdev, struct mt76_queue *q, mt76x02_mac_poll_tx_status(dev, false); - txwi = (struct mt76x02_txwi *) &e->txwi->txwi; + txwi_ptr = mt76_get_txwi_ptr(mdev, e->txwi); + txwi = (struct mt76x02_txwi *)txwi_ptr; trace_mac_txdone_add(dev, txwi->wcid, txwi->pktid); mt76_tx_complete_skb(mdev, e->skb); @@ -1021,7 +1032,7 @@ static void mt76x02_edcca_check(struct mt76x02_dev *dev) void mt76x02_mac_work(struct work_struct *work) { struct mt76x02_dev *dev = container_of(work, struct mt76x02_dev, - mac_work.work); + mt76.mac_work.work); int i, idx; mutex_lock(&dev->mt76.mutex); @@ -1034,7 +1045,7 @@ void mt76x02_mac_work(struct work_struct *work) dev->aggr_stats[idx++] += val >> 16; } - if (!dev->beacon_mask) + if (!dev->mt76.beacon_mask) mt76x02_check_mac_err(dev); if (dev->ed_monitor) @@ -1044,7 +1055,7 @@ void mt76x02_mac_work(struct work_struct *work) mt76_tx_status_check(&dev->mt76, NULL, false); - ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mac_work, + ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mt76.mac_work, MT_MAC_WORK_INTERVAL); } @@ -1055,141 +1066,3 @@ void mt76x02_mac_set_bssid(struct mt76x02_dev *dev, u8 idx, const u8 *addr) mt76_rmw_field(dev, MT_MAC_APC_BSSID_H(idx), MT_MAC_APC_BSSID_H_ADDR, get_unaligned_le16(addr + 4)); } - -static int -mt76x02_write_beacon(struct mt76x02_dev *dev, int offset, struct sk_buff *skb) -{ - int beacon_len = mt76x02_beacon_offsets[1] - mt76x02_beacon_offsets[0]; - struct mt76x02_txwi txwi; - - if (WARN_ON_ONCE(beacon_len < skb->len + sizeof(struct mt76x02_txwi))) - return -ENOSPC; - - mt76x02_mac_write_txwi(dev, &txwi, skb, NULL, NULL, skb->len); - - mt76_wr_copy(dev, offset, &txwi, sizeof(txwi)); - offset += sizeof(txwi); - - mt76_wr_copy(dev, offset, skb->data, skb->len); - return 0; -} - -static int -__mt76x02_mac_set_beacon(struct mt76x02_dev *dev, u8 bcn_idx, - struct sk_buff *skb) -{ - int beacon_len = mt76x02_beacon_offsets[1] - mt76x02_beacon_offsets[0]; - int beacon_addr = mt76x02_beacon_offsets[bcn_idx]; - int ret = 0; - int i; - - /* Prevent corrupt transmissions during update */ - mt76_set(dev, MT_BCN_BYPASS_MASK, BIT(bcn_idx)); - - if (skb) { - ret = mt76x02_write_beacon(dev, beacon_addr, skb); - if (!ret) - dev->beacon_data_mask |= BIT(bcn_idx); - } else { - dev->beacon_data_mask &= ~BIT(bcn_idx); - for (i = 0; i < beacon_len; i += 4) - mt76_wr(dev, beacon_addr + i, 0); - } - - mt76_wr(dev, MT_BCN_BYPASS_MASK, 0xff00 | ~dev->beacon_data_mask); - - return ret; -} - -int mt76x02_mac_set_beacon(struct mt76x02_dev *dev, u8 vif_idx, - struct sk_buff *skb) -{ - bool force_update = false; - int bcn_idx = 0; - int i; - - for (i = 0; i < ARRAY_SIZE(dev->beacons); i++) { - if (vif_idx == i) { - force_update = !!dev->beacons[i] ^ !!skb; - - if (dev->beacons[i]) - dev_kfree_skb(dev->beacons[i]); - - dev->beacons[i] = skb; - __mt76x02_mac_set_beacon(dev, bcn_idx, skb); - } else if (force_update && dev->beacons[i]) { - __mt76x02_mac_set_beacon(dev, bcn_idx, - dev->beacons[i]); - } - - bcn_idx += !!dev->beacons[i]; - } - - for (i = bcn_idx; i < ARRAY_SIZE(dev->beacons); i++) { - if (!(dev->beacon_data_mask & BIT(i))) - break; - - __mt76x02_mac_set_beacon(dev, i, NULL); - } - - mt76_rmw_field(dev, MT_MAC_BSSID_DW1, MT_MAC_BSSID_DW1_MBEACON_N, - bcn_idx - 1); - return 0; -} - -static void -__mt76x02_mac_set_beacon_enable(struct mt76x02_dev *dev, u8 vif_idx, - bool val, struct sk_buff *skb) -{ - u8 old_mask = dev->beacon_mask; - bool en; - u32 reg; - - if (val) { - dev->beacon_mask |= BIT(vif_idx); - if (skb) - mt76x02_mac_set_beacon(dev, vif_idx, skb); - } else { - dev->beacon_mask &= ~BIT(vif_idx); - mt76x02_mac_set_beacon(dev, vif_idx, NULL); - } - - if (!!old_mask == !!dev->beacon_mask) - return; - - en = dev->beacon_mask; - - reg = MT_BEACON_TIME_CFG_BEACON_TX | - MT_BEACON_TIME_CFG_TBTT_EN | - MT_BEACON_TIME_CFG_TIMER_EN; - mt76_rmw(dev, MT_BEACON_TIME_CFG, reg, reg * en); - - if (mt76_is_usb(dev)) - return; - - mt76_rmw_field(dev, MT_INT_TIMER_EN, MT_INT_TIMER_EN_PRE_TBTT_EN, en); - if (en) - mt76x02_irq_enable(dev, MT_INT_PRE_TBTT | MT_INT_TBTT); - else - mt76x02_irq_disable(dev, MT_INT_PRE_TBTT | MT_INT_TBTT); -} - -void mt76x02_mac_set_beacon_enable(struct mt76x02_dev *dev, - struct ieee80211_vif *vif, bool val) -{ - u8 vif_idx = ((struct mt76x02_vif *)vif->drv_priv)->idx; - struct sk_buff *skb = NULL; - - if (mt76_is_mmio(dev)) - tasklet_disable(&dev->pre_tbtt_tasklet); - else if (val) - skb = ieee80211_beacon_get(mt76_hw(dev), vif); - - if (!dev->beacon_mask) - dev->tbtt_count = 0; - - __mt76x02_mac_set_beacon_enable(dev, vif_idx, val, skb); - - if (mt76_is_mmio(dev)) - tasklet_enable(&dev->pre_tbtt_tasklet); -} diff --git a/drivers/net/wireless/mediatek/mt76/mt76x02_mac.h b/drivers/net/wireless/mediatek/mt76/mt76x02_mac.h index caeeef96c42faf74ccf9bb15cc60f4ffe865e9f4..e4a9e0d0924b76dcf278cda3356d5690faa5b4d3 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x02_mac.h +++ b/drivers/net/wireless/mediatek/mt76/mt76x02_mac.h @@ -198,8 +198,8 @@ void mt76x02_mac_write_txwi(struct mt76x02_dev *dev, struct mt76x02_txwi *txwi, struct sk_buff *skb, struct mt76_wcid *wcid, struct ieee80211_sta *sta, int len); void mt76x02_mac_poll_tx_status(struct mt76x02_dev *dev, bool irq); -void mt76x02_tx_complete_skb(struct mt76_dev *mdev, struct mt76_queue *q, - struct mt76_queue_entry *e, bool flush); +void mt76x02_tx_complete_skb(struct mt76_dev *mdev, enum mt76_txq_id qid, + struct mt76_queue_entry *e); void mt76x02_update_channel(struct mt76_dev *mdev); void mt76x02_mac_work(struct work_struct *work); diff --git a/drivers/net/wireless/mediatek/mt76/mt76x02_mmio.c b/drivers/net/wireless/mediatek/mt76/mt76x02_mmio.c index daaed1220147ea914c32f13f32f04bd5a360d7fb..7b7163bc3b62355c025c37b2f3f95c62b5fdfef3 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x02_mmio.c +++ b/drivers/net/wireless/mediatek/mt76/mt76x02_mmio.c @@ -22,96 +22,18 @@ #include "mt76x02_mcu.h" #include "mt76x02_trace.h" -struct beacon_bc_data { - struct mt76x02_dev *dev; - struct sk_buff_head q; - struct sk_buff *tail[8]; -}; - -static void -mt76x02_update_beacon_iter(void *priv, u8 *mac, struct ieee80211_vif *vif) -{ - struct mt76x02_dev *dev = (struct mt76x02_dev *)priv; - struct mt76x02_vif *mvif = (struct mt76x02_vif *)vif->drv_priv; - struct sk_buff *skb = NULL; - - if (!(dev->beacon_mask & BIT(mvif->idx))) - return; - - skb = ieee80211_beacon_get(mt76_hw(dev), vif); - if (!skb) - return; - - mt76x02_mac_set_beacon(dev, mvif->idx, skb); -} - -static void -mt76x02_add_buffered_bc(void *priv, u8 *mac, struct ieee80211_vif *vif) -{ - struct beacon_bc_data *data = priv; - struct mt76x02_dev *dev = data->dev; - struct mt76x02_vif *mvif = (struct mt76x02_vif *)vif->drv_priv; - struct ieee80211_tx_info *info; - struct sk_buff *skb; - - if (!(dev->beacon_mask & BIT(mvif->idx))) - return; - - skb = ieee80211_get_buffered_bc(mt76_hw(dev), vif); - if (!skb) - return; - - info = IEEE80211_SKB_CB(skb); - info->control.vif = vif; - info->flags |= IEEE80211_TX_CTL_ASSIGN_SEQ; - mt76_skb_set_moredata(skb, true); - __skb_queue_tail(&data->q, skb); - data->tail[mvif->idx] = skb; -} - -static void -mt76x02_resync_beacon_timer(struct mt76x02_dev *dev) -{ - u32 timer_val = dev->beacon_int << 4; - - dev->tbtt_count++; - - /* - * Beacon timer drifts by 1us every tick, the timer is configured - * in 1/16 TU (64us) units. - */ - if (dev->tbtt_count < 63) - return; - - /* - * The updated beacon interval takes effect after two TBTT, because - * at this point the original interval has already been loaded into - * the next TBTT_TIMER value - */ - if (dev->tbtt_count == 63) - timer_val -= 1; - - mt76_rmw_field(dev, MT_BEACON_TIME_CFG, - MT_BEACON_TIME_CFG_INTVAL, timer_val); - - if (dev->tbtt_count >= 64) { - dev->tbtt_count = 0; - return; - } -} - static void mt76x02_pre_tbtt_tasklet(unsigned long arg) { struct mt76x02_dev *dev = (struct mt76x02_dev *)arg; - struct mt76_queue *q = &dev->mt76.q_tx[MT_TXQ_PSD]; + struct mt76_queue *q = dev->mt76.q_tx[MT_TXQ_PSD].q; struct beacon_bc_data data = {}; struct sk_buff *skb; - int i, nframes; + int i; - mt76x02_resync_beacon_timer(dev); + if (mt76_hw(dev)->conf.flags & IEEE80211_CONF_OFFCHANNEL) + return; - data.dev = dev; - __skb_queue_head_init(&data.q); + mt76x02_resync_beacon_timer(dev); ieee80211_iterate_active_interfaces_atomic(mt76_hw(dev), IEEE80211_IFACE_ITER_RESUME_ALL, @@ -122,13 +44,7 @@ static void mt76x02_pre_tbtt_tasklet(unsigned long arg) if (dev->mt76.csa_complete) return; - do { - nframes = skb_queue_len(&data.q); - ieee80211_iterate_active_interfaces_atomic(mt76_hw(dev), - IEEE80211_IFACE_ITER_RESUME_ALL, - mt76x02_add_buffered_bc, &data); - } while (nframes != skb_queue_len(&data.q) && - skb_queue_len(&data.q) < 8); + mt76x02_enqueue_buffered_bc(dev, &data, 8); if (!skb_queue_len(&data.q)) return; @@ -146,25 +62,67 @@ static void mt76x02_pre_tbtt_tasklet(unsigned long arg) struct ieee80211_vif *vif = info->control.vif; struct mt76x02_vif *mvif = (struct mt76x02_vif *)vif->drv_priv; - mt76_dma_tx_queue_skb(&dev->mt76, q, skb, &mvif->group_wcid, - NULL); + mt76_tx_queue_skb(dev, MT_TXQ_PSD, skb, &mvif->group_wcid, + NULL); } spin_unlock_bh(&q->lock); } +static void mt76x02e_pre_tbtt_enable(struct mt76x02_dev *dev, bool en) +{ + if (en) + tasklet_enable(&dev->mt76.pre_tbtt_tasklet); + else + tasklet_disable(&dev->mt76.pre_tbtt_tasklet); +} + +static void mt76x02e_beacon_enable(struct mt76x02_dev *dev, bool en) +{ + mt76_rmw_field(dev, MT_INT_TIMER_EN, MT_INT_TIMER_EN_PRE_TBTT_EN, en); + if (en) + mt76x02_irq_enable(dev, MT_INT_PRE_TBTT | MT_INT_TBTT); + else + mt76x02_irq_disable(dev, MT_INT_PRE_TBTT | MT_INT_TBTT); +} + +void mt76x02e_init_beacon_config(struct mt76x02_dev *dev) +{ + static const struct mt76x02_beacon_ops beacon_ops = { + .nslots = 8, + .slot_size = 1024, + .pre_tbtt_enable = mt76x02e_pre_tbtt_enable, + .beacon_enable = mt76x02e_beacon_enable, + }; + + dev->beacon_ops = &beacon_ops; + + /* Fire a pre-TBTT interrupt 8 ms before TBTT */ + mt76_rmw_field(dev, MT_INT_TIMER_CFG, MT_INT_TIMER_CFG_PRE_TBTT, 8 << 4); + mt76_rmw_field(dev, MT_INT_TIMER_CFG, MT_INT_TIMER_CFG_GP_TIMER, + MT_DFS_GP_INTERVAL); + mt76_wr(dev, MT_INT_TIMER_EN, 0); + + mt76x02_init_beacon_config(dev); +} +EXPORT_SYMBOL_GPL(mt76x02e_init_beacon_config); + static int -mt76x02_init_tx_queue(struct mt76x02_dev *dev, struct mt76_queue *q, +mt76x02_init_tx_queue(struct mt76x02_dev *dev, struct mt76_sw_queue *q, int idx, int n_desc) { - int ret; + struct mt76_queue *hwq; + int err; - q->regs = dev->mt76.mmio.regs + MT_TX_RING_BASE + idx * MT_RING_SIZE; - q->ndesc = n_desc; - q->hw_idx = idx; + hwq = devm_kzalloc(dev->mt76.dev, sizeof(*hwq), GFP_KERNEL); + if (!hwq) + return -ENOMEM; - ret = mt76_queue_alloc(dev, q); - if (ret) - return ret; + err = mt76_queue_alloc(dev, hwq, idx, n_desc, 0, MT_TX_RING_BASE); + if (err < 0) + return err; + + INIT_LIST_HEAD(&q->swq); + q->q = hwq; mt76x02_irq_enable(dev, MT_INT_TX_DONE(idx)); @@ -175,15 +133,12 @@ static int mt76x02_init_rx_queue(struct mt76x02_dev *dev, struct mt76_queue *q, int idx, int n_desc, int bufsize) { - int ret; - - q->regs = dev->mt76.mmio.regs + MT_RX_RING_BASE + idx * MT_RING_SIZE; - q->ndesc = n_desc; - q->buf_size = bufsize; + int err; - ret = mt76_queue_alloc(dev, q); - if (ret) - return ret; + err = mt76_queue_alloc(dev, q, idx, n_desc, bufsize, + MT_RX_RING_BASE); + if (err < 0) + return err; mt76x02_irq_enable(dev, MT_INT_RX_DONE(idx)); @@ -202,15 +157,32 @@ static void mt76x02_process_tx_status_fifo(struct mt76x02_dev *dev) static void mt76x02_tx_tasklet(unsigned long data) { struct mt76x02_dev *dev = (struct mt76x02_dev *)data; - int i; + mt76x02_mac_poll_tx_status(dev, false); mt76x02_process_tx_status_fifo(dev); + mt76_txq_schedule_all(&dev->mt76); +} + +static int mt76x02_poll_tx(struct napi_struct *napi, int budget) +{ + struct mt76x02_dev *dev = container_of(napi, struct mt76x02_dev, tx_napi); + int i; + + mt76x02_mac_poll_tx_status(dev, false); + for (i = MT_TXQ_MCU; i >= 0; i--) mt76_queue_tx_cleanup(dev, i, false); - mt76x02_mac_poll_tx_status(dev, false); - mt76x02_irq_enable(dev, MT_INT_TX_DONE_ALL); + if (napi_complete_done(napi, 0)) + mt76x02_irq_enable(dev, MT_INT_TX_DONE_ALL); + + for (i = MT_TXQ_MCU; i >= 0; i--) + mt76_queue_tx_cleanup(dev, i, false); + + tasklet_schedule(&dev->mt76.tx_tasklet); + + return 0; } int mt76x02_dma_init(struct mt76x02_dev *dev) @@ -220,7 +192,6 @@ int mt76x02_dma_init(struct mt76x02_dev *dev) struct mt76_queue *q; void *status_fifo; - BUILD_BUG_ON(sizeof(t->txwi) < sizeof(struct mt76x02_txwi)); BUILD_BUG_ON(sizeof(struct mt76x02_rxwi) > MT_RX_HEADROOM); fifo_size = roundup_pow_of_two(32 * sizeof(struct mt76x02_tx_status)); @@ -228,10 +199,12 @@ int mt76x02_dma_init(struct mt76x02_dev *dev) if (!status_fifo) return -ENOMEM; - tasklet_init(&dev->tx_tasklet, mt76x02_tx_tasklet, (unsigned long) dev); - tasklet_init(&dev->pre_tbtt_tasklet, mt76x02_pre_tbtt_tasklet, + tasklet_init(&dev->mt76.tx_tasklet, mt76x02_tx_tasklet, + (unsigned long) dev); + tasklet_init(&dev->mt76.pre_tbtt_tasklet, mt76x02_pre_tbtt_tasklet, (unsigned long)dev); + spin_lock_init(&dev->txstatus_fifo_lock); kfifo_init(&dev->txstatus_fifo, status_fifo, fifo_size); mt76_dma_attach(&dev->mt76); @@ -268,7 +241,15 @@ int mt76x02_dma_init(struct mt76x02_dev *dev) if (ret) return ret; - return mt76_init_queues(dev); + ret = mt76_init_queues(dev); + if (ret) + return ret; + + netif_tx_napi_add(&dev->mt76.napi_dev, &dev->tx_napi, mt76x02_poll_tx, + NAPI_POLL_WEIGHT); + napi_enable(&dev->tx_napi); + + return 0; } EXPORT_SYMBOL_GPL(mt76x02_dma_init); @@ -296,11 +277,6 @@ irqreturn_t mt76x02_irq_handler(int irq, void *dev_instance) intr &= dev->mt76.mmio.irqmask; - if (intr & MT_INT_TX_DONE_ALL) { - mt76x02_irq_disable(dev, MT_INT_TX_DONE_ALL); - tasklet_schedule(&dev->tx_tasklet); - } - if (intr & MT_INT_RX_DONE(0)) { mt76x02_irq_disable(dev, MT_INT_RX_DONE(0)); napi_schedule(&dev->mt76.napi[0]); @@ -312,19 +288,22 @@ irqreturn_t mt76x02_irq_handler(int irq, void *dev_instance) } if (intr & MT_INT_PRE_TBTT) - tasklet_schedule(&dev->pre_tbtt_tasklet); + tasklet_schedule(&dev->mt76.pre_tbtt_tasklet); /* send buffered multicast frames now */ if (intr & MT_INT_TBTT) { if (dev->mt76.csa_complete) mt76_csa_finish(&dev->mt76); else - mt76_queue_kick(dev, &dev->mt76.q_tx[MT_TXQ_PSD]); + mt76_queue_kick(dev, dev->mt76.q_tx[MT_TXQ_PSD].q); } - if (intr & MT_INT_TX_STAT) { + if (intr & MT_INT_TX_STAT) mt76x02_mac_poll_tx_status(dev, true); - tasklet_schedule(&dev->tx_tasklet); + + if (intr & (MT_INT_TX_STAT | MT_INT_TX_DONE_ALL)) { + mt76x02_irq_disable(dev, MT_INT_TX_DONE_ALL); + napi_schedule(&dev->tx_napi); } if (intr & MT_INT_GPTIMER) { @@ -336,18 +315,6 @@ irqreturn_t mt76x02_irq_handler(int irq, void *dev_instance) } EXPORT_SYMBOL_GPL(mt76x02_irq_handler); -void mt76x02_set_irq_mask(struct mt76x02_dev *dev, u32 clear, u32 set) -{ - unsigned long flags; - - spin_lock_irqsave(&dev->mt76.mmio.irq_lock, flags); - dev->mt76.mmio.irqmask &= ~clear; - dev->mt76.mmio.irqmask |= set; - mt76_wr(dev, MT_INT_MASK_CSR, dev->mt76.mmio.irqmask); - spin_unlock_irqrestore(&dev->mt76.mmio.irq_lock, flags); -} -EXPORT_SYMBOL_GPL(mt76x02_set_irq_mask); - static void mt76x02_dma_enable(struct mt76x02_dev *dev) { u32 val; @@ -366,7 +333,8 @@ static void mt76x02_dma_enable(struct mt76x02_dev *dev) void mt76x02_dma_cleanup(struct mt76x02_dev *dev) { - tasklet_kill(&dev->tx_tasklet); + tasklet_kill(&dev->mt76.tx_tasklet); + netif_napi_del(&dev->tx_napi); mt76_dma_cleanup(&dev->mt76); } EXPORT_SYMBOL_GPL(mt76x02_dma_cleanup); @@ -403,13 +371,13 @@ static bool mt76x02_tx_hang(struct mt76x02_dev *dev) int i; for (i = 0; i < 4; i++) { - q = &dev->mt76.q_tx[i]; + q = dev->mt76.q_tx[i].q; if (!q->queued) continue; prev_dma_idx = dev->mt76.tx_dma_idx[i]; - dma_idx = ioread32(&q->regs->dma_idx); + dma_idx = readl(&q->regs->dma_idx); dev->mt76.tx_dma_idx[i] = dma_idx; if (prev_dma_idx == dma_idx) @@ -472,7 +440,7 @@ static void mt76x02_reset_state(struct mt76x02_dev *dev) } dev->vif_mask = 0; - dev->beacon_mask = 0; + dev->mt76.beacon_mask = 0; } static void mt76x02_watchdog_reset(struct mt76x02_dev *dev) @@ -484,8 +452,9 @@ static void mt76x02_watchdog_reset(struct mt76x02_dev *dev) ieee80211_stop_queues(dev->mt76.hw); set_bit(MT76_RESET, &dev->mt76.state); - tasklet_disable(&dev->pre_tbtt_tasklet); - tasklet_disable(&dev->tx_tasklet); + tasklet_disable(&dev->mt76.pre_tbtt_tasklet); + tasklet_disable(&dev->mt76.tx_tasklet); + napi_disable(&dev->tx_napi); for (i = 0; i < ARRAY_SIZE(dev->mt76.napi); i++) napi_disable(&dev->mt76.napi[i]); @@ -495,7 +464,7 @@ static void mt76x02_watchdog_reset(struct mt76x02_dev *dev) if (restart) mt76x02_reset_state(dev); - if (dev->beacon_mask) + if (dev->mt76.beacon_mask) mt76_clear(dev, MT_BEACON_TIME_CFG, MT_BEACON_TIME_CFG_BEACON_TX | MT_BEACON_TIME_CFG_TBTT_EN); @@ -514,7 +483,7 @@ static void mt76x02_watchdog_reset(struct mt76x02_dev *dev) mt76_set(dev, 0x734, 0x3); if (restart) - dev->mt76.mcu_ops->mcu_restart(&dev->mt76); + mt76_mcu_restart(dev); for (i = 0; i < ARRAY_SIZE(dev->mt76.q_tx); i++) mt76_queue_tx_cleanup(dev, i, true); @@ -527,7 +496,7 @@ static void mt76x02_watchdog_reset(struct mt76x02_dev *dev) if (dev->ed_monitor) mt76_set(dev, MT_TXOP_CTRL_CFG, MT_TXOP_ED_CCA_EN); - if (dev->beacon_mask && !restart) + if (dev->mt76.beacon_mask && !restart) mt76_set(dev, MT_BEACON_TIME_CFG, MT_BEACON_TIME_CFG_BEACON_TX | MT_BEACON_TIME_CFG_TBTT_EN); @@ -538,10 +507,11 @@ static void mt76x02_watchdog_reset(struct mt76x02_dev *dev) clear_bit(MT76_RESET, &dev->mt76.state); - tasklet_enable(&dev->tx_tasklet); - tasklet_schedule(&dev->tx_tasklet); + tasklet_enable(&dev->mt76.tx_tasklet); + napi_enable(&dev->tx_napi); + napi_schedule(&dev->tx_napi); - tasklet_enable(&dev->pre_tbtt_tasklet); + tasklet_enable(&dev->mt76.pre_tbtt_tasklet); for (i = 0; i < ARRAY_SIZE(dev->mt76.napi); i++) { napi_enable(&dev->mt76.napi[i]); diff --git a/drivers/net/wireless/mediatek/mt76/mt76x02_regs.h b/drivers/net/wireless/mediatek/mt76/mt76x02_regs.h index 7401cb94fb72382e858dda9211c3c3daac77e666..2ce05b543dff18dcfcca1881208ae72aca71bb0a 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x02_regs.h +++ b/drivers/net/wireless/mediatek/mt76/mt76x02_regs.h @@ -356,7 +356,10 @@ #define MT_BEACON_TIME_CFG_TSF_COMP GENMASK(31, 24) #define MT_TBTT_SYNC_CFG 0x1118 -#define MT_TBTT_TIMER_CFG 0x1124 +#define MT_TSF_TIMER_DW0 0x111c +#define MT_TSF_TIMER_DW1 0x1120 +#define MT_TBTT_TIMER 0x1124 +#define MT_TBTT_TIMER_VAL GENMASK(16, 0) #define MT_INT_TIMER_CFG 0x1128 #define MT_INT_TIMER_CFG_PRE_TBTT GENMASK(15, 0) diff --git a/drivers/net/wireless/mediatek/mt76/mt76x02_txrx.c b/drivers/net/wireless/mediatek/mt76/mt76x02_txrx.c index 94f47248c59f252e9a4228c39e7161d488a70556..cf7abd9b7d2ea11c3a0acb2e401566d8dc1ea1d3 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x02_txrx.c +++ b/drivers/net/wireless/mediatek/mt76/mt76x02_txrx.c @@ -147,36 +147,33 @@ bool mt76x02_tx_status_data(struct mt76_dev *mdev, u8 *update) EXPORT_SYMBOL_GPL(mt76x02_tx_status_data); int mt76x02_tx_prepare_skb(struct mt76_dev *mdev, void *txwi_ptr, - struct sk_buff *skb, struct mt76_queue *q, - struct mt76_wcid *wcid, struct ieee80211_sta *sta, - u32 *tx_info) + enum mt76_txq_id qid, struct mt76_wcid *wcid, + struct ieee80211_sta *sta, + struct mt76_tx_info *tx_info) { struct mt76x02_dev *dev = container_of(mdev, struct mt76x02_dev, mt76); + struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)tx_info->skb->data; struct mt76x02_txwi *txwi = txwi_ptr; - int qsel = MT_QSEL_EDCA; - int pid; - int ret; + int hdrlen, len, pid, qsel = MT_QSEL_EDCA; - if (q == &dev->mt76.q_tx[MT_TXQ_PSD] && wcid && wcid->idx < 128) + if (qid == MT_TXQ_PSD && wcid && wcid->idx < 128) mt76x02_mac_wcid_set_drop(dev, wcid->idx, false); - mt76x02_mac_write_txwi(dev, txwi, skb, wcid, sta, skb->len); + hdrlen = ieee80211_hdrlen(hdr->frame_control); + len = tx_info->skb->len - (hdrlen & 2); + mt76x02_mac_write_txwi(dev, txwi, tx_info->skb, wcid, sta, len); - pid = mt76_tx_status_skb_add(mdev, wcid, skb); + pid = mt76_tx_status_skb_add(mdev, wcid, tx_info->skb); txwi->pktid = pid; - ret = mt76x02_insert_hdr_pad(skb); - if (ret < 0) - return ret; - if (pid >= MT_PACKET_ID_FIRST) qsel = MT_QSEL_MGMT; - *tx_info = FIELD_PREP(MT_TXD_INFO_QSEL, qsel) | - MT_TXD_INFO_80211; + tx_info->info = FIELD_PREP(MT_TXD_INFO_QSEL, qsel) | + MT_TXD_INFO_80211; if (!wcid || wcid->hw_key_idx == 0xff || wcid->sw_iv) - *tx_info |= MT_TXD_INFO_WIV; + tx_info->info |= MT_TXD_INFO_WIV; return 0; } diff --git a/drivers/net/wireless/mediatek/mt76/mt76x02_usb.h b/drivers/net/wireless/mediatek/mt76/mt76x02_usb.h index 0126e51d77ed4ca8476bd2fc2f2d04cd39f75c67..7b53f9e57f2937f5a5de64d3a38d0c923e8fdca3 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x02_usb.h +++ b/drivers/net/wireless/mediatek/mt76/mt76x02_usb.h @@ -26,9 +26,11 @@ int mt76x02u_mcu_fw_send_data(struct mt76x02_dev *dev, const void *data, int mt76x02u_skb_dma_info(struct sk_buff *skb, int port, u32 flags); int mt76x02u_tx_prepare_skb(struct mt76_dev *mdev, void *data, - struct sk_buff *skb, struct mt76_queue *q, - struct mt76_wcid *wcid, struct ieee80211_sta *sta, - u32 *tx_info); -void mt76x02u_tx_complete_skb(struct mt76_dev *mdev, struct mt76_queue *q, - struct mt76_queue_entry *e, bool flush); + enum mt76_txq_id qid, struct mt76_wcid *wcid, + struct ieee80211_sta *sta, + struct mt76_tx_info *tx_info); +void mt76x02u_tx_complete_skb(struct mt76_dev *mdev, enum mt76_txq_id qid, + struct mt76_queue_entry *e); +void mt76x02u_init_beacon_config(struct mt76x02_dev *dev); +void mt76x02u_exit_beacon_config(struct mt76x02_dev *dev); #endif /* __MT76x02_USB_H */ diff --git a/drivers/net/wireless/mediatek/mt76/mt76x02_usb_core.c b/drivers/net/wireless/mediatek/mt76/mt76x02_usb_core.c index 6fb52b596d421753ff24f247225136b30c15150b..6b89f7eab26c9ebbe53859880a753943e9fdb03b 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x02_usb_core.c +++ b/drivers/net/wireless/mediatek/mt76/mt76x02_usb_core.c @@ -26,8 +26,8 @@ static void mt76x02u_remove_dma_hdr(struct sk_buff *skb) mt76x02_remove_hdr_pad(skb, 2); } -void mt76x02u_tx_complete_skb(struct mt76_dev *mdev, struct mt76_queue *q, - struct mt76_queue_entry *e, bool flush) +void mt76x02u_tx_complete_skb(struct mt76_dev *mdev, enum mt76_txq_id qid, + struct mt76_queue_entry *e) { mt76x02u_remove_dma_hdr(e->skb); mt76_tx_complete_skb(mdev, e->skb); @@ -72,27 +72,26 @@ int mt76x02u_skb_dma_info(struct sk_buff *skb, int port, u32 flags) } int mt76x02u_tx_prepare_skb(struct mt76_dev *mdev, void *data, - struct sk_buff *skb, struct mt76_queue *q, - struct mt76_wcid *wcid, struct ieee80211_sta *sta, - u32 *tx_info) + enum mt76_txq_id qid, struct mt76_wcid *wcid, + struct ieee80211_sta *sta, + struct mt76_tx_info *tx_info) { struct mt76x02_dev *dev = container_of(mdev, struct mt76x02_dev, mt76); + int pid, len = tx_info->skb->len, ep = q2ep(mdev->q_tx[qid].q->hw_idx); struct mt76x02_txwi *txwi; enum mt76_qsel qsel; - int len = skb->len; u32 flags; - int pid; - mt76x02_insert_hdr_pad(skb); + mt76_insert_hdr_pad(tx_info->skb); - txwi = (struct mt76x02_txwi *)(skb->data - sizeof(struct mt76x02_txwi)); - mt76x02_mac_write_txwi(dev, txwi, skb, wcid, sta, len); - skb_push(skb, sizeof(struct mt76x02_txwi)); + txwi = (struct mt76x02_txwi *)(tx_info->skb->data - sizeof(*txwi)); + mt76x02_mac_write_txwi(dev, txwi, tx_info->skb, wcid, sta, len); + skb_push(tx_info->skb, sizeof(*txwi)); - pid = mt76_tx_status_skb_add(mdev, wcid, skb); + pid = mt76_tx_status_skb_add(mdev, wcid, tx_info->skb); txwi->pktid = pid; - if (pid >= MT_PACKET_ID_FIRST || q2ep(q->hw_idx) == MT_EP_OUT_HCCA) + if (pid >= MT_PACKET_ID_FIRST || ep == MT_EP_OUT_HCCA) qsel = MT_QSEL_MGMT; else qsel = MT_QSEL_EDCA; @@ -102,6 +101,167 @@ int mt76x02u_tx_prepare_skb(struct mt76_dev *mdev, void *data, if (!wcid || wcid->hw_key_idx == 0xff || wcid->sw_iv) flags |= MT_TXD_INFO_WIV; - return mt76x02u_skb_dma_info(skb, WLAN_PORT, flags); + return mt76x02u_skb_dma_info(tx_info->skb, WLAN_PORT, flags); } EXPORT_SYMBOL_GPL(mt76x02u_tx_prepare_skb); + +/* Trigger pre-TBTT event 8 ms before TBTT */ +#define PRE_TBTT_USEC 8000 + +/* Beacon SRAM memory is limited to 8kB. We need to send PS buffered frames + * (which can be 1500 bytes big) via beacon memory. That make limit of number + * of slots to 5. TODO: dynamically calculate offsets in beacon SRAM. + */ +#define N_BCN_SLOTS 5 + +static void mt76x02u_start_pre_tbtt_timer(struct mt76x02_dev *dev) +{ + u64 time; + u32 tbtt; + + /* Get remaining TBTT in usec */ + tbtt = mt76_get_field(dev, MT_TBTT_TIMER, MT_TBTT_TIMER_VAL); + tbtt *= 32; + + if (tbtt <= PRE_TBTT_USEC) { + queue_work(system_highpri_wq, &dev->pre_tbtt_work); + return; + } + + time = (tbtt - PRE_TBTT_USEC) * 1000ull; + hrtimer_start(&dev->pre_tbtt_timer, time, HRTIMER_MODE_REL); +} + +static void mt76x02u_restart_pre_tbtt_timer(struct mt76x02_dev *dev) +{ + u32 tbtt, dw0, dw1; + u64 tsf, time; + + /* Get remaining TBTT in usec */ + tbtt = mt76_get_field(dev, MT_TBTT_TIMER, MT_TBTT_TIMER_VAL); + tbtt *= 32; + + dw0 = mt76_rr(dev, MT_TSF_TIMER_DW0); + dw1 = mt76_rr(dev, MT_TSF_TIMER_DW1); + tsf = (u64)dw0 << 32 | dw1; + dev_dbg(dev->mt76.dev, "TSF: %llu us TBTT %u us\n", tsf, tbtt); + + /* Convert beacon interval in TU (1024 usec) to nsec */ + time = ((1000000000ull * dev->mt76.beacon_int) >> 10); + + /* Adjust time to trigger hrtimer 8ms before TBTT */ + if (tbtt < PRE_TBTT_USEC) + time -= (PRE_TBTT_USEC - tbtt) * 1000ull; + else + time += (tbtt - PRE_TBTT_USEC) * 1000ull; + + hrtimer_start(&dev->pre_tbtt_timer, time, HRTIMER_MODE_REL); +} + +static void mt76x02u_stop_pre_tbtt_timer(struct mt76x02_dev *dev) +{ + do { + hrtimer_cancel(&dev->pre_tbtt_timer); + cancel_work_sync(&dev->pre_tbtt_work); + /* Timer can be rearmed by work. */ + } while (hrtimer_active(&dev->pre_tbtt_timer)); +} + +static void mt76x02u_pre_tbtt_work(struct work_struct *work) +{ + struct mt76x02_dev *dev = + container_of(work, struct mt76x02_dev, pre_tbtt_work); + struct beacon_bc_data data = {}; + struct sk_buff *skb; + int i, nbeacons; + + if (!dev->mt76.beacon_mask) + return; + + if (mt76_hw(dev)->conf.flags & IEEE80211_CONF_OFFCHANNEL) + return; + + mt76x02_resync_beacon_timer(dev); + + ieee80211_iterate_active_interfaces(mt76_hw(dev), + IEEE80211_IFACE_ITER_RESUME_ALL, + mt76x02_update_beacon_iter, dev); + + nbeacons = hweight8(dev->mt76.beacon_mask); + mt76x02_enqueue_buffered_bc(dev, &data, N_BCN_SLOTS - nbeacons); + + for (i = nbeacons; i < N_BCN_SLOTS; i++) { + skb = __skb_dequeue(&data.q); + mt76x02_mac_set_beacon(dev, i, skb); + } + + mt76x02u_restart_pre_tbtt_timer(dev); +} + +static enum hrtimer_restart mt76x02u_pre_tbtt_interrupt(struct hrtimer *timer) +{ + struct mt76x02_dev *dev = + container_of(timer, struct mt76x02_dev, pre_tbtt_timer); + + queue_work(system_highpri_wq, &dev->pre_tbtt_work); + + return HRTIMER_NORESTART; +} + +static void mt76x02u_pre_tbtt_enable(struct mt76x02_dev *dev, bool en) +{ + if (en && dev->mt76.beacon_mask && + !hrtimer_active(&dev->pre_tbtt_timer)) + mt76x02u_start_pre_tbtt_timer(dev); + if (!en) + mt76x02u_stop_pre_tbtt_timer(dev); +} + +static void mt76x02u_beacon_enable(struct mt76x02_dev *dev, bool en) +{ + int i; + + if (WARN_ON_ONCE(!dev->mt76.beacon_int)) + return; + + if (en) { + mt76x02u_start_pre_tbtt_timer(dev); + } else { + /* Timer is already stopped, only clean up + * PS buffered frames if any. + */ + for (i = 0; i < N_BCN_SLOTS; i++) + mt76x02_mac_set_beacon(dev, i, NULL); + } +} + +void mt76x02u_init_beacon_config(struct mt76x02_dev *dev) +{ + static const struct mt76x02_beacon_ops beacon_ops = { + .nslots = N_BCN_SLOTS, + .slot_size = (8192 / N_BCN_SLOTS) & ~63, + .pre_tbtt_enable = mt76x02u_pre_tbtt_enable, + .beacon_enable = mt76x02u_beacon_enable, + }; + dev->beacon_ops = &beacon_ops; + + hrtimer_init(&dev->pre_tbtt_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL); + dev->pre_tbtt_timer.function = mt76x02u_pre_tbtt_interrupt; + INIT_WORK(&dev->pre_tbtt_work, mt76x02u_pre_tbtt_work); + + mt76x02_init_beacon_config(dev); +} +EXPORT_SYMBOL_GPL(mt76x02u_init_beacon_config); + +void mt76x02u_exit_beacon_config(struct mt76x02_dev *dev) +{ + if (!test_bit(MT76_REMOVED, &dev->mt76.state)) + mt76_clear(dev, MT_BEACON_TIME_CFG, + MT_BEACON_TIME_CFG_TIMER_EN | + MT_BEACON_TIME_CFG_SYNC_MODE | + MT_BEACON_TIME_CFG_TBTT_EN | + MT_BEACON_TIME_CFG_BEACON_TX); + + mt76x02u_stop_pre_tbtt_timer(dev); +} +EXPORT_SYMBOL_GPL(mt76x02u_exit_beacon_config); diff --git a/drivers/net/wireless/mediatek/mt76/mt76x02_util.c b/drivers/net/wireless/mediatek/mt76/mt76x02_util.c index cd072ac614f76847b86618a4f40c87af171f5147..ad5323447ed429a5c43a2ccc1f0d344deb5aa445 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x02_util.c +++ b/drivers/net/wireless/mediatek/mt76/mt76x02_util.c @@ -132,7 +132,7 @@ void mt76x02_init_device(struct mt76x02_dev *dev) struct ieee80211_hw *hw = mt76_hw(dev); struct wiphy *wiphy = hw->wiphy; - INIT_DELAYED_WORK(&dev->mac_work, mt76x02_mac_work); + INIT_DELAYED_WORK(&dev->mt76.mac_work, mt76x02_mac_work); hw->queues = 4; hw->max_rates = 1; @@ -142,6 +142,7 @@ void mt76x02_init_device(struct mt76x02_dev *dev) wiphy->interface_modes = BIT(NL80211_IFTYPE_STATION) | + BIT(NL80211_IFTYPE_AP) | #ifdef CONFIG_MAC80211_MESH BIT(NL80211_IFTYPE_MESH_POINT) | #endif @@ -158,7 +159,6 @@ void mt76x02_init_device(struct mt76x02_dev *dev) wiphy->reg_notifier = mt76x02_regd_notifier; wiphy->iface_combinations = mt76x02_if_comb; wiphy->n_iface_combinations = ARRAY_SIZE(mt76x02_if_comb); - wiphy->interface_modes |= BIT(NL80211_IFTYPE_AP); wiphy->flags |= WIPHY_FLAG_HAS_CHANNEL_SWITCH; /* init led callbacks */ @@ -378,7 +378,7 @@ int mt76x02_ampdu_action(struct ieee80211_hw *hw, struct ieee80211_vif *vif, ieee80211_send_bar(vif, sta->addr, tid, mtxq->agg_ssn); break; case IEEE80211_AMPDU_TX_START: - mtxq->agg_ssn = *ssn << 4; + mtxq->agg_ssn = IEEE80211_SN_TO_SEQ(*ssn); ieee80211_start_tx_ba_cb_irqsafe(vif, sta->addr, tid); break; case IEEE80211_AMPDU_TX_STOP_CONT: @@ -424,6 +424,16 @@ int mt76x02_set_key(struct ieee80211_hw *hw, enum set_key_cmd cmd, !(key->flags & IEEE80211_KEY_FLAG_PAIRWISE)) return -EOPNOTSUPP; + /* + * In USB AP mode, broadcast/multicast frames are setup in beacon + * data registers and sent via HW beacons engine, they require to + * be already encrypted. + */ + if (mt76_is_usb(dev) && + vif->type == NL80211_IFTYPE_AP && + !(key->flags & IEEE80211_KEY_FLAG_PAIRWISE)) + return -EOPNOTSUPP; + msta = sta ? (struct mt76x02_sta *) sta->drv_priv : NULL; wcid = msta ? &msta->wcid : &mvif->group_wcid; @@ -465,7 +475,7 @@ int mt76x02_conf_tx(struct ieee80211_hw *hw, struct ieee80211_vif *vif, u8 cw_min = 5, cw_max = 10, qid; u32 val; - qid = dev->mt76.q_tx[queue].hw_idx; + qid = dev->mt76.q_tx[queue].q->hw_idx; if (params->cw_min) cw_min = fls(params->cw_min); @@ -562,26 +572,9 @@ void mt76x02_sta_rate_tbl_update(struct ieee80211_hw *hw, rate.idx = rates->rate[0].idx; rate.flags = rates->rate[0].flags; mt76x02_mac_wcid_set_rate(dev, &msta->wcid, &rate); - msta->wcid.max_txpwr_adj = mt76x02_tx_get_max_txpwr_adj(dev, &rate); } EXPORT_SYMBOL_GPL(mt76x02_sta_rate_tbl_update); -int mt76x02_insert_hdr_pad(struct sk_buff *skb) -{ - int len = ieee80211_get_hdrlen_from_skb(skb); - - if (len % 4 == 0) - return 0; - - skb_push(skb, 2); - memmove(skb->data, skb->data + 2, len); - - skb->data[len] = 0; - skb->data[len + 1] = 0; - return 2; -} -EXPORT_SYMBOL_GPL(mt76x02_insert_hdr_pad); - void mt76x02_remove_hdr_pad(struct sk_buff *skb, int len) { int hdrlen; @@ -600,8 +593,6 @@ void mt76x02_sw_scan(struct ieee80211_hw *hw, struct ieee80211_vif *vif, { struct mt76x02_dev *dev = hw->priv; - if (mt76_is_mmio(dev)) - tasklet_disable(&dev->pre_tbtt_tasklet); set_bit(MT76_SCANNING, &dev->mt76.state); } EXPORT_SYMBOL_GPL(mt76x02_sw_scan); @@ -612,9 +603,6 @@ void mt76x02_sw_scan_complete(struct ieee80211_hw *hw, struct mt76x02_dev *dev = hw->priv; clear_bit(MT76_SCANNING, &dev->mt76.state); - if (mt76_is_mmio(dev)) - tasklet_enable(&dev->pre_tbtt_tasklet); - if (dev->cal.gain_init_done) { /* Restore AGC gain and resume calibration after scanning. */ dev->cal.low_gain = -1; @@ -631,72 +619,11 @@ void mt76x02_sta_ps(struct mt76_dev *mdev, struct ieee80211_sta *sta, int idx = msta->wcid.idx; mt76_stop_tx_queues(&dev->mt76, sta, true); - mt76x02_mac_wcid_set_drop(dev, idx, ps); + if (mt76_is_mmio(dev)) + mt76x02_mac_wcid_set_drop(dev, idx, ps); } EXPORT_SYMBOL_GPL(mt76x02_sta_ps); -const u16 mt76x02_beacon_offsets[16] = { - /* 1024 byte per beacon */ - 0xc000, - 0xc400, - 0xc800, - 0xcc00, - 0xd000, - 0xd400, - 0xd800, - 0xdc00, - /* BSS idx 8-15 not used for beacons */ - 0xc000, - 0xc000, - 0xc000, - 0xc000, - 0xc000, - 0xc000, - 0xc000, - 0xc000, -}; - -static void mt76x02_set_beacon_offsets(struct mt76x02_dev *dev) -{ - u16 val, base = MT_BEACON_BASE; - u32 regs[4] = {}; - int i; - - for (i = 0; i < 16; i++) { - val = mt76x02_beacon_offsets[i] - base; - regs[i / 4] |= (val / 64) << (8 * (i % 4)); - } - - for (i = 0; i < 4; i++) - mt76_wr(dev, MT_BCN_OFFSET(i), regs[i]); -} - -void mt76x02_init_beacon_config(struct mt76x02_dev *dev) -{ - int i; - - if (mt76_is_mmio(dev)) { - /* Fire a pre-TBTT interrupt 8 ms before TBTT */ - mt76_rmw_field(dev, MT_INT_TIMER_CFG, MT_INT_TIMER_CFG_PRE_TBTT, - 8 << 4); - mt76_rmw_field(dev, MT_INT_TIMER_CFG, MT_INT_TIMER_CFG_GP_TIMER, - MT_DFS_GP_INTERVAL); - mt76_wr(dev, MT_INT_TIMER_EN, 0); - } - - mt76_clear(dev, MT_BEACON_TIME_CFG, (MT_BEACON_TIME_CFG_TIMER_EN | - MT_BEACON_TIME_CFG_TBTT_EN | - MT_BEACON_TIME_CFG_BEACON_TX)); - mt76_set(dev, MT_BEACON_TIME_CFG, MT_BEACON_TIME_CFG_SYNC_MODE); - mt76_wr(dev, MT_BCN_BYPASS_MASK, 0xffff); - - for (i = 0; i < 8; i++) - mt76x02_mac_set_beacon(dev, i, NULL); - - mt76x02_set_beacon_offsets(dev); -} -EXPORT_SYMBOL_GPL(mt76x02_init_beacon_config); - void mt76x02_bss_info_changed(struct ieee80211_hw *hw, struct ieee80211_vif *vif, struct ieee80211_bss_conf *info, @@ -718,7 +645,7 @@ void mt76x02_bss_info_changed(struct ieee80211_hw *hw, mt76_rmw_field(dev, MT_BEACON_TIME_CFG, MT_BEACON_TIME_CFG_INTVAL, info->beacon_int << 4); - dev->beacon_int = info->beacon_int; + dev->mt76.beacon_int = info->beacon_int; } if (changed & BSS_CHANGED_BEACON_ENABLED) diff --git a/drivers/net/wireless/mediatek/mt76/mt76x2/init.c b/drivers/net/wireless/mediatek/mt76/mt76x2/init.c index a30ef2c5a9db0433cbb696f6ab6f3e8b1811cb5d..c6078e90ca4340961bcee4259de33c787bd103a6 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x2/init.c +++ b/drivers/net/wireless/mediatek/mt76/mt76x2/init.c @@ -165,27 +165,21 @@ void mt76x2_init_txpower(struct mt76x02_dev *dev, struct ieee80211_channel *chan; struct mt76x2_tx_power_info txp; struct mt76_rate_power t = {}; - int target_power; int i; for (i = 0; i < sband->n_channels; i++) { chan = &sband->channels[i]; mt76x2_get_power_info(dev, &txp, chan); - - target_power = max_t(int, (txp.chain[0].target_power + - txp.chain[0].delta), - (txp.chain[1].target_power + - txp.chain[1].delta)); - mt76x2_get_rate_power(dev, &t, chan); chan->max_power = mt76x02_get_max_rate_power(&t) + - target_power; - chan->max_power /= 2; + txp.target_power; + chan->max_power = DIV_ROUND_UP(chan->max_power, 2); /* convert to combined output power on 2x2 devices */ chan->max_power += 3; + chan->orig_mpwr = chan->max_power; } } EXPORT_SYMBOL_GPL(mt76x2_init_txpower); diff --git a/drivers/net/wireless/mediatek/mt76/mt76x2/pci.c b/drivers/net/wireless/mediatek/mt76/mt76x2/pci.c index 6274655e1f7e41daf9567cc69b4af0aab26c7015..e84d5c5911ea82147abc4a5ec7b50c431e0b31da 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x2/pci.c +++ b/drivers/net/wireless/mediatek/mt76/mt76x2/pci.c @@ -32,6 +32,7 @@ mt76pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) { static const struct mt76_driver_ops drv_ops = { .txwi_size = sizeof(struct mt76x02_txwi), + .tx_aligned4_skbs = true, .update_survey = mt76x02_update_channel, .tx_prepare_skb = mt76x02_tx_prepare_skb, .tx_complete_skb = mt76x02_tx_complete_skb, @@ -106,7 +107,7 @@ mt76pci_remove(struct pci_dev *pdev) mt76_unregister_device(mdev); mt76x2_cleanup(dev); - ieee80211_free_hw(mdev->hw); + mt76_free_device(mdev); } MODULE_DEVICE_TABLE(pci, mt76pci_device_table); diff --git a/drivers/net/wireless/mediatek/mt76/mt76x2/pci_init.c b/drivers/net/wireless/mediatek/mt76/mt76x2/pci_init.c index d3927a13e92e91068344e431176a3b42ef6ff444..71aea2832644180b7ebda2b3eb9ddafdbdeb0894 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x2/pci_init.c +++ b/drivers/net/wireless/mediatek/mt76/mt76x2/pci_init.c @@ -120,7 +120,7 @@ int mt76x2_mac_reset(struct mt76x02_dev *dev, bool hard) mt76_clear(dev, MT_FCE_L2_STUFF, MT_FCE_L2_STUFF_WR_MPDU_LEN_EN); mt76x02_mac_setaddr(dev, macaddr); - mt76x02_init_beacon_config(dev); + mt76x02e_init_beacon_config(dev); if (!hard) return 0; @@ -291,7 +291,7 @@ static int mt76x2_init_hardware(struct mt76x02_dev *dev) void mt76x2_stop_hardware(struct mt76x02_dev *dev) { cancel_delayed_work_sync(&dev->cal_work); - cancel_delayed_work_sync(&dev->mac_work); + cancel_delayed_work_sync(&dev->mt76.mac_work); cancel_delayed_work_sync(&dev->wdt_work); mt76x02_mcu_set_radio_state(dev, false); mt76x2_mac_stop(dev, false); @@ -300,7 +300,7 @@ void mt76x2_stop_hardware(struct mt76x02_dev *dev) void mt76x2_cleanup(struct mt76x02_dev *dev) { tasklet_disable(&dev->dfs_pd.dfs_tasklet); - tasklet_disable(&dev->pre_tbtt_tasklet); + tasklet_disable(&dev->mt76.pre_tbtt_tasklet); mt76x2_stop_hardware(dev); mt76x02_dma_cleanup(dev); mt76x02_mcu_cleanup(dev); diff --git a/drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c b/drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c index 878ce92405ed854cf251642fd092c5fd0c066c49..e416eee6a306cc5b5f51fc3458a3ffda3d5cdd4f 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c +++ b/drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c @@ -22,26 +22,21 @@ mt76x2_start(struct ieee80211_hw *hw) struct mt76x02_dev *dev = hw->priv; int ret; - mutex_lock(&dev->mt76.mutex); - ret = mt76x2_mac_start(dev); if (ret) - goto out; + return ret; ret = mt76x2_phy_start(dev); if (ret) - goto out; + return ret; - ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mac_work, + ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mt76.mac_work, MT_MAC_WORK_INTERVAL); ieee80211_queue_delayed_work(mt76_hw(dev), &dev->wdt_work, MT_WATCHDOG_TIME); set_bit(MT76_STATE_RUNNING, &dev->mt76.state); - -out: - mutex_unlock(&dev->mt76.mutex); - return ret; + return 0; } static void @@ -49,10 +44,8 @@ mt76x2_stop(struct ieee80211_hw *hw) { struct mt76x02_dev *dev = hw->priv; - mutex_lock(&dev->mt76.mutex); clear_bit(MT76_STATE_RUNNING, &dev->mt76.state); mt76x2_stop_hardware(dev); - mutex_unlock(&dev->mt76.mutex); } static int @@ -66,7 +59,7 @@ mt76x2_set_channel(struct mt76x02_dev *dev, struct cfg80211_chan_def *chandef) mt76_set_channel(&dev->mt76); - tasklet_disable(&dev->pre_tbtt_tasklet); + tasklet_disable(&dev->mt76.pre_tbtt_tasklet); tasklet_disable(&dev->dfs_pd.dfs_tasklet); mt76x2_mac_stop(dev, true); @@ -80,7 +73,7 @@ mt76x2_set_channel(struct mt76x02_dev *dev, struct cfg80211_chan_def *chandef) mt76x2_mac_resume(dev); tasklet_enable(&dev->dfs_pd.dfs_tasklet); - tasklet_enable(&dev->pre_tbtt_tasklet); + tasklet_enable(&dev->mt76.pre_tbtt_tasklet); clear_bit(MT76_RESET, &dev->mt76.state); @@ -135,12 +128,6 @@ mt76x2_flush(struct ieee80211_hw *hw, struct ieee80211_vif *vif, { } -static int -mt76x2_set_tim(struct ieee80211_hw *hw, struct ieee80211_sta *sta, bool set) -{ - return 0; -} - static int mt76x2_set_antenna(struct ieee80211_hw *hw, u32 tx_ant, u32 rx_ant) { @@ -197,7 +184,7 @@ const struct ieee80211_ops mt76x2_ops = { .release_buffered_frames = mt76_release_buffered_frames, .set_coverage_class = mt76x02_set_coverage_class, .get_survey = mt76_get_survey, - .set_tim = mt76x2_set_tim, + .set_tim = mt76_set_tim, .set_antenna = mt76x2_set_antenna, .get_antenna = mt76x2_get_antenna, .set_rts_threshold = mt76x02_set_rts_threshold, diff --git a/drivers/net/wireless/mediatek/mt76/mt76x2/phy.c b/drivers/net/wireless/mediatek/mt76/mt76x2/phy.c index 769a9b9720442c5d303e9a65519a8fe1de166089..cdedf95ca4f5313de3a8dfc0aba8cb30b13b3108 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x2/phy.c +++ b/drivers/net/wireless/mediatek/mt76/mt76x2/phy.c @@ -161,12 +161,12 @@ void mt76x2_phy_set_txpower(struct mt76x02_dev *dev) delta = txp.delta_bw80; mt76x2_get_rate_power(dev, &t, chan); - mt76x02_add_rate_power_offset(&t, txp.chain[0].target_power); + mt76x02_add_rate_power_offset(&t, txp.target_power + delta); mt76x02_limit_rate_power(&t, dev->mt76.txpower_conf); dev->mt76.txpower_cur = mt76x02_get_max_rate_power(&t); base_power = mt76x2_get_min_rate_power(&t); - delta += base_power - txp.chain[0].target_power; + delta = base_power - txp.target_power; txp_0 = txp.chain[0].target_power + txp.chain[0].delta + delta; txp_1 = txp.chain[1].target_power + txp.chain[1].delta + delta; @@ -182,7 +182,7 @@ void mt76x2_phy_set_txpower(struct mt76x02_dev *dev) } mt76x02_add_rate_power_offset(&t, -base_power); - dev->target_power = txp.chain[0].target_power; + dev->target_power = txp.target_power; dev->target_power_delta[0] = txp_0 - txp.chain[0].target_power; dev->target_power_delta[1] = txp_1 - txp.chain[0].target_power; dev->mt76.rate_power = t; diff --git a/drivers/net/wireless/mediatek/mt76/mt76x2/usb.c b/drivers/net/wireless/mediatek/mt76/mt76x2/usb.c index ac0f13d4629963cea77e5ab113caf3e11a33275a..7a994a783510225df54eeb4863001c8ddca16a50 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x2/usb.c +++ b/drivers/net/wireless/mediatek/mt76/mt76x2/usb.c @@ -40,6 +40,7 @@ static int mt76x2u_probe(struct usb_interface *intf, .tx_complete_skb = mt76x02u_tx_complete_skb, .tx_status_data = mt76x02_tx_status_data, .rx_skb = mt76x02_queue_rx_skb, + .sta_ps = mt76x02_sta_ps, .sta_add = mt76x02_sta_add, .sta_remove = mt76x02_sta_remove, }; @@ -48,7 +49,7 @@ static int mt76x2u_probe(struct usb_interface *intf, struct mt76_dev *mdev; int err; - mdev = mt76_alloc_device(&intf->dev, sizeof(*dev), &mt76x2u_ops, + mdev = mt76_alloc_device(&udev->dev, sizeof(*dev), &mt76x2u_ops, &drv_ops); if (!mdev) return -ENOMEM; @@ -58,6 +59,8 @@ static int mt76x2u_probe(struct usb_interface *intf, udev = usb_get_dev(udev); usb_reset_device(udev); + usb_set_intfdata(intf, dev); + mt76x02u_init_mcu(mdev); err = mt76u_init(mdev, intf); if (err < 0) @@ -104,8 +107,7 @@ static int __maybe_unused mt76x2u_suspend(struct usb_interface *intf, { struct mt76x02_dev *dev = usb_get_intfdata(intf); - mt76u_stop_queues(&dev->mt76); - mt76x2u_stop_hw(dev); + mt76u_stop_rx(&dev->mt76); return 0; } @@ -113,16 +115,12 @@ static int __maybe_unused mt76x2u_suspend(struct usb_interface *intf, static int __maybe_unused mt76x2u_resume(struct usb_interface *intf) { struct mt76x02_dev *dev = usb_get_intfdata(intf); - struct mt76_usb *usb = &dev->mt76.usb; int err; - err = mt76u_submit_rx_buffers(&dev->mt76); + err = mt76u_resume_rx(&dev->mt76); if (err < 0) goto err; - tasklet_enable(&usb->rx_tasklet); - tasklet_enable(&usb->tx_tasklet); - err = mt76x2u_init_hardware(dev); if (err < 0) goto err; diff --git a/drivers/net/wireless/mediatek/mt76/mt76x2/usb_init.c b/drivers/net/wireless/mediatek/mt76/mt76x2/usb_init.c index 1da90e58d942a723f3df7ec433896d618d9212be..f2c57d5b87f939598a5deea6081480e962abe5d3 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x2/usb_init.c +++ b/drivers/net/wireless/mediatek/mt76/mt76x2/usb_init.c @@ -183,7 +183,7 @@ int mt76x2u_init_hardware(struct mt76x02_dev *dev) mt76x02_mac_shared_key_setup(dev, i, k, NULL); } - mt76x02_init_beacon_config(dev); + mt76x02u_init_beacon_config(dev); mt76_rmw(dev, MT_US_CYC_CFG, MT_US_CYC_CNT, 0x1e); mt76_wr(dev, MT_TXOP_CTRL_CFG, 0x583f); @@ -244,9 +244,8 @@ fail: void mt76x2u_stop_hw(struct mt76x02_dev *dev) { - mt76u_stop_stat_wk(&dev->mt76); cancel_delayed_work_sync(&dev->cal_work); - cancel_delayed_work_sync(&dev->mac_work); + cancel_delayed_work_sync(&dev->mt76.mac_work); mt76x2u_mac_stop(dev); } diff --git a/drivers/net/wireless/mediatek/mt76/mt76x2/usb_main.c b/drivers/net/wireless/mediatek/mt76/mt76x2/usb_main.c index 2ac78e4dc41a8d86d20ffd911472e509a8e3a30e..97bcf6494ec1754b1748e9ede4740c6bfc7f78a5 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x2/usb_main.c +++ b/drivers/net/wireless/mediatek/mt76/mt76x2/usb_main.c @@ -21,29 +21,24 @@ static int mt76x2u_start(struct ieee80211_hw *hw) struct mt76x02_dev *dev = hw->priv; int ret; - mutex_lock(&dev->mt76.mutex); - ret = mt76x2u_mac_start(dev); if (ret) - goto out; + return ret; - ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mac_work, + ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mt76.mac_work, MT_MAC_WORK_INTERVAL); set_bit(MT76_STATE_RUNNING, &dev->mt76.state); -out: - mutex_unlock(&dev->mt76.mutex); - return ret; + return 0; } static void mt76x2u_stop(struct ieee80211_hw *hw) { struct mt76x02_dev *dev = hw->priv; - mutex_lock(&dev->mt76.mutex); clear_bit(MT76_STATE_RUNNING, &dev->mt76.state); + mt76u_stop_tx(&dev->mt76); mt76x2u_stop_hw(dev); - mutex_unlock(&dev->mt76.mutex); } static int @@ -57,6 +52,8 @@ mt76x2u_set_channel(struct mt76x02_dev *dev, mt76_set_channel(&dev->mt76); + dev->beacon_ops->pre_tbtt_enable(dev, false); + mt76x2_mac_stop(dev, false); err = mt76x2u_phy_set_channel(dev, chandef); @@ -64,6 +61,8 @@ mt76x2u_set_channel(struct mt76x02_dev *dev, mt76x2_mac_resume(dev); mt76x02_edcca_init(dev, true); + dev->beacon_ops->pre_tbtt_enable(dev, true); + clear_bit(MT76_RESET, &dev->mt76.state); mt76_txq_schedule_all(&dev->mt76); @@ -125,4 +124,6 @@ const struct ieee80211_ops mt76x2u_ops = { .sw_scan_complete = mt76x02_sw_scan_complete, .sta_rate_tbl_update = mt76x02_sta_rate_tbl_update, .get_txpower = mt76_get_txpower, + .set_tim = mt76_set_tim, + .release_buffered_frames = mt76_release_buffered_frames, }; diff --git a/drivers/net/wireless/mediatek/mt76/tx.c b/drivers/net/wireless/mediatek/mt76/tx.c index 2585df5123350ba8adf52cf14ef6133de7e03f77..5397827668b91a8caaec8471c099c56c5ae4d18f 100644 --- a/drivers/net/wireless/mediatek/mt76/tx.c +++ b/drivers/net/wireless/mediatek/mt76/tx.c @@ -21,15 +21,17 @@ mt76_alloc_txwi(struct mt76_dev *dev) { struct mt76_txwi_cache *t; dma_addr_t addr; + u8 *txwi; int size; - size = (sizeof(*t) + L1_CACHE_BYTES - 1) & ~(L1_CACHE_BYTES - 1); - t = devm_kzalloc(dev->dev, size, GFP_ATOMIC); - if (!t) + size = L1_CACHE_ALIGN(dev->drv->txwi_size + sizeof(*t)); + txwi = devm_kzalloc(dev->dev, size, GFP_ATOMIC); + if (!txwi) return NULL; - addr = dma_map_single(dev->dev, &t->txwi, sizeof(t->txwi), + addr = dma_map_single(dev->dev, txwi, dev->drv->txwi_size, DMA_TO_DEVICE); + t = (struct mt76_txwi_cache *)(txwi + dev->drv->txwi_size); t->dma_addr = addr; return t; @@ -72,13 +74,14 @@ mt76_put_txwi(struct mt76_dev *dev, struct mt76_txwi_cache *t) list_add(&t->list, &dev->txwi_cache); spin_unlock_bh(&dev->lock); } +EXPORT_SYMBOL_GPL(mt76_put_txwi); void mt76_tx_free(struct mt76_dev *dev) { struct mt76_txwi_cache *t; while ((t = __mt76_get_txwi(dev)) != NULL) - dma_unmap_single(dev->dev, t->dma_addr, sizeof(t->txwi), + dma_unmap_single(dev->dev, t->dma_addr, dev->drv->txwi_size, DMA_TO_DEVICE); } @@ -266,7 +269,7 @@ mt76_tx(struct mt76_dev *dev, struct ieee80211_sta *sta, skb_set_queue_mapping(skb, qid); } - if (!wcid->tx_rate_set) + if (!(wcid->tx_info & MT_WCID_TX_INFO_SET)) ieee80211_get_tx_rates(info->control.vif, sta, skb, info->control.rates, 1); @@ -283,10 +286,10 @@ mt76_tx(struct mt76_dev *dev, struct ieee80211_sta *sta, mt76_check_agg_ssn(mtxq, skb); } - q = &dev->q_tx[qid]; + q = dev->q_tx[qid].q; spin_lock_bh(&q->lock); - dev->queue_ops->tx_queue_skb(dev, q, skb, wcid, sta); + dev->queue_ops->tx_queue_skb(dev, qid, skb, wcid, sta); dev->queue_ops->kick(dev, q); if (q->queued > q->ndesc - 8 && !q->stopped) { @@ -327,7 +330,6 @@ mt76_queue_ps_skb(struct mt76_dev *dev, struct ieee80211_sta *sta, { struct mt76_wcid *wcid = (struct mt76_wcid *) sta->drv_priv; struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb); - struct mt76_queue *hwq = &dev->q_tx[MT_TXQ_PSD]; info->control.flags |= IEEE80211_TX_CTRL_PS_RESPONSE; if (last) @@ -335,7 +337,7 @@ mt76_queue_ps_skb(struct mt76_dev *dev, struct ieee80211_sta *sta, IEEE80211_TX_CTL_REQ_TX_STATUS; mt76_skb_set_moredata(skb, !last); - dev->queue_ops->tx_queue_skb(dev, hwq, skb, wcid, sta); + dev->queue_ops->tx_queue_skb(dev, MT_TXQ_PSD, skb, wcid, sta); } void @@ -346,7 +348,7 @@ mt76_release_buffered_frames(struct ieee80211_hw *hw, struct ieee80211_sta *sta, { struct mt76_dev *dev = hw->priv; struct sk_buff *last_skb = NULL; - struct mt76_queue *hwq = &dev->q_tx[MT_TXQ_PSD]; + struct mt76_queue *hwq = dev->q_tx[MT_TXQ_PSD].q; int i; spin_lock_bh(&hwq->lock); @@ -386,12 +388,14 @@ mt76_release_buffered_frames(struct ieee80211_hw *hw, struct ieee80211_sta *sta, EXPORT_SYMBOL_GPL(mt76_release_buffered_frames); static int -mt76_txq_send_burst(struct mt76_dev *dev, struct mt76_queue *hwq, +mt76_txq_send_burst(struct mt76_dev *dev, struct mt76_sw_queue *sq, struct mt76_txq *mtxq, bool *empty) { struct ieee80211_txq *txq = mtxq_to_txq(mtxq); - struct ieee80211_tx_info *info; + enum mt76_txq_id qid = mt76_txq_get_qid(txq); struct mt76_wcid *wcid = mtxq->wcid; + struct mt76_queue *hwq = sq->q; + struct ieee80211_tx_info *info; struct sk_buff *skb; int n_frames = 1, limit; struct ieee80211_tx_rate tx_rate; @@ -411,7 +415,7 @@ mt76_txq_send_burst(struct mt76_dev *dev, struct mt76_queue *hwq, } info = IEEE80211_SKB_CB(skb); - if (!wcid->tx_rate_set) + if (!(wcid->tx_info & MT_WCID_TX_INFO_SET)) ieee80211_get_tx_rates(txq->vif, txq->sta, skb, info->control.rates, 1); tx_rate = info->control.rates[0]; @@ -423,7 +427,7 @@ mt76_txq_send_burst(struct mt76_dev *dev, struct mt76_queue *hwq, if (ampdu) mt76_check_agg_ssn(mtxq, skb); - idx = dev->queue_ops->tx_queue_skb(dev, hwq, skb, wcid, txq->sta); + idx = dev->queue_ops->tx_queue_skb(dev, qid, skb, wcid, txq->sta); if (idx < 0) return idx; @@ -458,7 +462,7 @@ mt76_txq_send_burst(struct mt76_dev *dev, struct mt76_queue *hwq, if (cur_ampdu) mt76_check_agg_ssn(mtxq, skb); - idx = dev->queue_ops->tx_queue_skb(dev, hwq, skb, wcid, + idx = dev->queue_ops->tx_queue_skb(dev, qid, skb, wcid, txq->sta); if (idx < 0) return idx; @@ -467,8 +471,9 @@ mt76_txq_send_burst(struct mt76_dev *dev, struct mt76_queue *hwq, } while (n_frames < limit); if (!probe) { - hwq->swq_queued++; + hwq->entry[idx].qid = sq - dev->q_tx; hwq->entry[idx].schedule = true; + sq->swq_queued++; } dev->queue_ops->kick(dev, hwq); @@ -477,22 +482,37 @@ mt76_txq_send_burst(struct mt76_dev *dev, struct mt76_queue *hwq, } static int -mt76_txq_schedule_list(struct mt76_dev *dev, struct mt76_queue *hwq) +mt76_txq_schedule_list(struct mt76_dev *dev, enum mt76_txq_id qid) { - struct mt76_txq *mtxq, *mtxq_last; - int len = 0; + struct mt76_sw_queue *sq = &dev->q_tx[qid]; + struct mt76_queue *hwq = sq->q; + struct ieee80211_txq *txq; + struct mt76_txq *mtxq; + struct mt76_wcid *wcid; + int ret = 0; -restart: - mtxq_last = list_last_entry(&hwq->swq, struct mt76_txq, list); - while (!list_empty(&hwq->swq)) { + spin_lock_bh(&hwq->lock); + while (1) { bool empty = false; - int cur; + + if (sq->swq_queued >= 4) + break; if (test_bit(MT76_OFFCHANNEL, &dev->state) || - test_bit(MT76_RESET, &dev->state)) - return -EBUSY; + test_bit(MT76_RESET, &dev->state)) { + ret = -EBUSY; + break; + } + + txq = ieee80211_next_txq(dev->hw, qid); + if (!txq) + break; + + mtxq = (struct mt76_txq *)txq->drv_priv; + wcid = mtxq->wcid; + if (wcid && test_bit(MT_WCID_FLAG_PS, &wcid->flags)) + continue; - mtxq = list_first_entry(&hwq->swq, struct mt76_txq, list); if (mtxq->send_bar && mtxq->aggr) { struct ieee80211_txq *txq = mtxq_to_txq(mtxq); struct ieee80211_sta *sta = txq->sta; @@ -504,38 +524,37 @@ restart: spin_unlock_bh(&hwq->lock); ieee80211_send_bar(vif, sta->addr, tid, agg_ssn); spin_lock_bh(&hwq->lock); - goto restart; } - list_del_init(&mtxq->list); - - cur = mt76_txq_send_burst(dev, hwq, mtxq, &empty); - if (!empty) - list_add_tail(&mtxq->list, &hwq->swq); - - if (cur < 0) - return cur; - - len += cur; - - if (mtxq == mtxq_last) - break; + ret += mt76_txq_send_burst(dev, sq, mtxq, &empty); + if (skb_queue_empty(&mtxq->retry_q)) + empty = true; + ieee80211_return_txq(dev->hw, txq, !empty); } + spin_unlock_bh(&hwq->lock); - return len; + return ret; } -void mt76_txq_schedule(struct mt76_dev *dev, struct mt76_queue *hwq) +void mt76_txq_schedule(struct mt76_dev *dev, enum mt76_txq_id qid) { + struct mt76_sw_queue *sq = &dev->q_tx[qid]; int len; + if (qid >= 4) + return; + + if (sq->swq_queued >= 4) + return; + rcu_read_lock(); - do { - if (hwq->swq_queued >= 4 || list_empty(&hwq->swq)) - break; - len = mt76_txq_schedule_list(dev, hwq); + do { + ieee80211_txq_schedule_start(dev->hw, qid); + len = mt76_txq_schedule_list(dev, qid); + ieee80211_txq_schedule_end(dev->hw, qid); } while (len > 0); + rcu_read_unlock(); } EXPORT_SYMBOL_GPL(mt76_txq_schedule); @@ -544,13 +563,8 @@ void mt76_txq_schedule_all(struct mt76_dev *dev) { int i; - for (i = 0; i <= MT_TXQ_BK; i++) { - struct mt76_queue *q = &dev->q_tx[i]; - - spin_lock_bh(&q->lock); - mt76_txq_schedule(dev, q); - spin_unlock_bh(&q->lock); - } + for (i = 0; i <= MT_TXQ_BK; i++) + mt76_txq_schedule(dev, i); } EXPORT_SYMBOL_GPL(mt76_txq_schedule_all); @@ -561,18 +575,18 @@ void mt76_stop_tx_queues(struct mt76_dev *dev, struct ieee80211_sta *sta, for (i = 0; i < ARRAY_SIZE(sta->txq); i++) { struct ieee80211_txq *txq = sta->txq[i]; + struct mt76_queue *hwq; struct mt76_txq *mtxq; if (!txq) continue; mtxq = (struct mt76_txq *)txq->drv_priv; + hwq = mtxq->swq->q; - spin_lock_bh(&mtxq->hwq->lock); + spin_lock_bh(&hwq->lock); mtxq->send_bar = mtxq->aggr && send_bar; - if (!list_empty(&mtxq->list)) - list_del_init(&mtxq->list); - spin_unlock_bh(&mtxq->hwq->lock); + spin_unlock_bh(&hwq->lock); } } EXPORT_SYMBOL_GPL(mt76_stop_tx_queues); @@ -580,36 +594,23 @@ EXPORT_SYMBOL_GPL(mt76_stop_tx_queues); void mt76_wake_tx_queue(struct ieee80211_hw *hw, struct ieee80211_txq *txq) { struct mt76_dev *dev = hw->priv; - struct mt76_txq *mtxq = (struct mt76_txq *) txq->drv_priv; - struct mt76_queue *hwq = mtxq->hwq; if (!test_bit(MT76_STATE_RUNNING, &dev->state)) return; - spin_lock_bh(&hwq->lock); - if (list_empty(&mtxq->list)) - list_add_tail(&mtxq->list, &hwq->swq); - mt76_txq_schedule(dev, hwq); - spin_unlock_bh(&hwq->lock); + tasklet_schedule(&dev->tx_tasklet); } EXPORT_SYMBOL_GPL(mt76_wake_tx_queue); void mt76_txq_remove(struct mt76_dev *dev, struct ieee80211_txq *txq) { struct mt76_txq *mtxq; - struct mt76_queue *hwq; struct sk_buff *skb; if (!txq) return; mtxq = (struct mt76_txq *) txq->drv_priv; - hwq = mtxq->hwq; - - spin_lock_bh(&hwq->lock); - if (!list_empty(&mtxq->list)) - list_del_init(&mtxq->list); - spin_unlock_bh(&hwq->lock); while ((skb = skb_dequeue(&mtxq->retry_q)) != NULL) ieee80211_free_txskb(dev->hw, skb); @@ -620,10 +621,9 @@ void mt76_txq_init(struct mt76_dev *dev, struct ieee80211_txq *txq) { struct mt76_txq *mtxq = (struct mt76_txq *) txq->drv_priv; - INIT_LIST_HEAD(&mtxq->list); skb_queue_head_init(&mtxq->retry_q); - mtxq->hwq = &dev->q_tx[mt76_txq_get_qid(txq)]; + mtxq->swq = &dev->q_tx[mt76_txq_get_qid(txq)]; } EXPORT_SYMBOL_GPL(mt76_txq_init); diff --git a/drivers/net/wireless/mediatek/mt76/usb.c b/drivers/net/wireless/mediatek/mt76/usb.c index 4c1abd4924054c6f377ca062ad98ca5bb8446dc7..bbaa1365bbda2b07e42f5c4dab34087318dc2039 100644 --- a/drivers/net/wireless/mediatek/mt76/usb.c +++ b/drivers/net/wireless/mediatek/mt76/usb.c @@ -31,8 +31,7 @@ static int __mt76u_vendor_request(struct mt76_dev *dev, u8 req, u8 req_type, u16 val, u16 offset, void *buf, size_t len) { - struct usb_interface *intf = to_usb_interface(dev->dev); - struct usb_device *udev = interface_to_usbdev(intf); + struct usb_device *udev = to_usb_device(dev->dev); unsigned int pipe; int i, ret; @@ -247,8 +246,7 @@ mt76u_rd_rp(struct mt76_dev *dev, u32 base, static bool mt76u_check_sg(struct mt76_dev *dev) { - struct usb_interface *intf = to_usb_interface(dev->dev); - struct usb_device *udev = interface_to_usbdev(intf); + struct usb_device *udev = to_usb_device(dev->dev); return (!disable_usb_sg && udev->bus->sg_tablesize > 0 && (udev->bus->no_sg_constraint || @@ -285,28 +283,24 @@ mt76u_set_endpoints(struct usb_interface *intf, } static int -mt76u_fill_rx_sg(struct mt76_dev *dev, struct mt76u_buf *buf, - int nsgs, int len, int sglen) +mt76u_fill_rx_sg(struct mt76_dev *dev, struct mt76_queue *q, struct urb *urb, + int nsgs, gfp_t gfp) { - struct mt76_queue *q = &dev->q_rx[MT_RXQ_MAIN]; - struct urb *urb = buf->urb; int i; - spin_lock_bh(&q->rx_page_lock); for (i = 0; i < nsgs; i++) { struct page *page; void *data; int offset; - data = page_frag_alloc(&q->rx_page, len, GFP_ATOMIC); + data = page_frag_alloc(&q->rx_page, q->buf_size, gfp); if (!data) break; page = virt_to_head_page(data); offset = data - page_address(page); - sg_set_page(&urb->sg[i], page, sglen, offset); + sg_set_page(&urb->sg[i], page, q->buf_size, offset); } - spin_unlock_bh(&q->rx_page_lock); if (i < nsgs) { int j; @@ -317,72 +311,78 @@ mt76u_fill_rx_sg(struct mt76_dev *dev, struct mt76u_buf *buf, } urb->num_sgs = max_t(int, i, urb->num_sgs); - buf->len = urb->num_sgs * sglen, + urb->transfer_buffer_length = urb->num_sgs * q->buf_size, sg_init_marker(urb->sg, urb->num_sgs); return i ? : -ENOMEM; } static int -mt76u_refill_rx(struct mt76_dev *dev, struct mt76_queue *q, - struct mt76u_buf *buf, int nsgs, gfp_t gfp) +mt76u_refill_rx(struct mt76_dev *dev, struct urb *urb, int nsgs, gfp_t gfp) { + struct mt76_queue *q = &dev->q_rx[MT_RXQ_MAIN]; + if (dev->usb.sg_en) { - return mt76u_fill_rx_sg(dev, buf, nsgs, q->buf_size, - SKB_WITH_OVERHEAD(q->buf_size)); + return mt76u_fill_rx_sg(dev, q, urb, nsgs, gfp); } else { - buf->buf = page_frag_alloc(&q->rx_page, q->buf_size, gfp); - return buf->buf ? 0 : -ENOMEM; + urb->transfer_buffer_length = q->buf_size; + urb->transfer_buffer = page_frag_alloc(&q->rx_page, + q->buf_size, gfp); + return urb->transfer_buffer ? 0 : -ENOMEM; } } static int -mt76u_buf_alloc(struct mt76_dev *dev, struct mt76u_buf *buf) +mt76u_urb_alloc(struct mt76_dev *dev, struct mt76_queue_entry *e) { - struct mt76_queue *q = &dev->q_rx[MT_RXQ_MAIN]; + unsigned int size = sizeof(struct urb); - buf->len = SKB_WITH_OVERHEAD(q->buf_size); - buf->dev = dev; + if (dev->usb.sg_en) + size += MT_SG_MAX_SIZE * sizeof(struct scatterlist); - buf->urb = usb_alloc_urb(0, GFP_KERNEL); - if (!buf->urb) + e->urb = kzalloc(size, GFP_KERNEL); + if (!e->urb) return -ENOMEM; - if (dev->usb.sg_en) { - buf->urb->sg = devm_kcalloc(dev->dev, MT_SG_MAX_SIZE, - sizeof(*buf->urb->sg), - GFP_KERNEL); - if (!buf->urb->sg) - return -ENOMEM; + usb_init_urb(e->urb); - sg_init_table(buf->urb->sg, MT_SG_MAX_SIZE); - } + if (dev->usb.sg_en) + e->urb->sg = (struct scatterlist *)(e->urb + 1); - return mt76u_refill_rx(dev, q, buf, MT_SG_MAX_SIZE, GFP_KERNEL); + return 0; } -static void mt76u_buf_free(struct mt76u_buf *buf) +static int +mt76u_rx_urb_alloc(struct mt76_dev *dev, struct mt76_queue_entry *e) +{ + int err; + + err = mt76u_urb_alloc(dev, e); + if (err) + return err; + + return mt76u_refill_rx(dev, e->urb, MT_SG_MAX_SIZE, GFP_KERNEL); +} + +static void mt76u_urb_free(struct urb *urb) { - struct urb *urb = buf->urb; int i; for (i = 0; i < urb->num_sgs; i++) skb_free_frag(sg_virt(&urb->sg[i])); - if (buf->buf) - skb_free_frag(buf->buf); + if (urb->transfer_buffer) + skb_free_frag(urb->transfer_buffer); - usb_free_urb(buf->urb); + usb_free_urb(urb); } static void mt76u_fill_bulk_urb(struct mt76_dev *dev, int dir, int index, - struct mt76u_buf *buf, usb_complete_t complete_fn, + struct urb *urb, usb_complete_t complete_fn, void *context) { - struct usb_interface *intf = to_usb_interface(dev->dev); - struct usb_device *udev = interface_to_usbdev(intf); - u8 *data = buf->urb->num_sgs ? NULL : buf->buf; + struct usb_device *udev = to_usb_device(dev->dev); unsigned int pipe; if (dir == USB_DIR_IN) @@ -390,37 +390,28 @@ mt76u_fill_bulk_urb(struct mt76_dev *dev, int dir, int index, else pipe = usb_sndbulkpipe(udev, dev->usb.out_ep[index]); - usb_fill_bulk_urb(buf->urb, udev, pipe, data, buf->len, - complete_fn, context); + urb->dev = udev; + urb->pipe = pipe; + urb->complete = complete_fn; + urb->context = context; } -static int -mt76u_submit_buf(struct mt76_dev *dev, int dir, int index, - struct mt76u_buf *buf, gfp_t gfp, - usb_complete_t complete_fn, void *context) +static inline struct urb * +mt76u_get_next_rx_entry(struct mt76_dev *dev) { - mt76u_fill_bulk_urb(dev, dir, index, buf, complete_fn, - context); - trace_submit_urb(dev, buf->urb); - - return usb_submit_urb(buf->urb, gfp); -} - -static inline struct mt76u_buf -*mt76u_get_next_rx_entry(struct mt76_queue *q) -{ - struct mt76u_buf *buf = NULL; + struct mt76_queue *q = &dev->q_rx[MT_RXQ_MAIN]; + struct urb *urb = NULL; unsigned long flags; spin_lock_irqsave(&q->lock, flags); if (q->queued > 0) { - buf = &q->entry[q->head].ubuf; + urb = q->entry[q->head].urb; q->head = (q->head + 1) % q->ndesc; q->queued--; } spin_unlock_irqrestore(&q->lock, flags); - return buf; + return urb; } static int mt76u_get_rx_entry_len(u8 *data, u32 data_len) @@ -439,12 +430,12 @@ static int mt76u_get_rx_entry_len(u8 *data, u32 data_len) } static int -mt76u_process_rx_entry(struct mt76_dev *dev, struct mt76u_buf *buf) +mt76u_process_rx_entry(struct mt76_dev *dev, struct urb *urb) { struct mt76_queue *q = &dev->q_rx[MT_RXQ_MAIN]; - struct urb *urb = buf->urb; - u8 *data = urb->num_sgs ? sg_virt(&urb->sg[0]) : buf->buf; - int data_len, len, nsgs = 1; + u8 *data = urb->num_sgs ? sg_virt(&urb->sg[0]) : urb->transfer_buffer; + int data_len = urb->num_sgs ? urb->sg[0].length : urb->actual_length; + int len, nsgs = 1; struct sk_buff *skb; if (!test_bit(MT76_STATE_INITIALIZED, &dev->state)) @@ -454,10 +445,11 @@ mt76u_process_rx_entry(struct mt76_dev *dev, struct mt76u_buf *buf) if (len < 0) return 0; - data_len = urb->num_sgs ? urb->sg[0].length : buf->len; data_len = min_t(int, len, data_len - MT_DMA_HDR_LEN); - if (MT_DMA_HDR_LEN + data_len > SKB_WITH_OVERHEAD(q->buf_size)) + if (MT_DMA_HDR_LEN + data_len > SKB_WITH_OVERHEAD(q->buf_size)) { + dev_err_ratelimited(dev->dev, "rx data too big %d\n", data_len); return 0; + } skb = build_skb(data, q->buf_size); if (!skb) @@ -503,7 +495,7 @@ static void mt76u_complete_rx(struct urb *urb) } spin_lock_irqsave(&q->lock, flags); - if (WARN_ONCE(q->entry[q->tail].ubuf.urb != urb, "rx urb mismatch")) + if (WARN_ONCE(q->entry[q->tail].urb != urb, "rx urb mismatch")) goto out; q->tail = (q->tail + 1) % q->ndesc; @@ -513,37 +505,43 @@ out: spin_unlock_irqrestore(&q->lock, flags); } +static int +mt76u_submit_rx_buf(struct mt76_dev *dev, struct urb *urb) +{ + mt76u_fill_bulk_urb(dev, USB_DIR_IN, MT_EP_IN_PKT_RX, urb, + mt76u_complete_rx, dev); + trace_submit_urb(dev, urb); + + return usb_submit_urb(urb, GFP_ATOMIC); +} + static void mt76u_rx_tasklet(unsigned long data) { struct mt76_dev *dev = (struct mt76_dev *)data; - struct mt76_queue *q = &dev->q_rx[MT_RXQ_MAIN]; - struct mt76u_buf *buf; + struct urb *urb; int err, count; rcu_read_lock(); while (true) { - buf = mt76u_get_next_rx_entry(q); - if (!buf) + urb = mt76u_get_next_rx_entry(dev); + if (!urb) break; - count = mt76u_process_rx_entry(dev, buf); + count = mt76u_process_rx_entry(dev, urb); if (count > 0) { - err = mt76u_refill_rx(dev, q, buf, count, - GFP_ATOMIC); + err = mt76u_refill_rx(dev, urb, count, GFP_ATOMIC); if (err < 0) break; } - mt76u_submit_buf(dev, USB_DIR_IN, MT_EP_IN_PKT_RX, - buf, GFP_ATOMIC, - mt76u_complete_rx, dev); + mt76u_submit_rx_buf(dev, urb); } mt76_rx_poll_complete(dev, MT_RXQ_MAIN, NULL); rcu_read_unlock(); } -int mt76u_submit_rx_buffers(struct mt76_dev *dev) +static int mt76u_submit_rx_buffers(struct mt76_dev *dev) { struct mt76_queue *q = &dev->q_rx[MT_RXQ_MAIN]; unsigned long flags; @@ -551,9 +549,7 @@ int mt76u_submit_rx_buffers(struct mt76_dev *dev) spin_lock_irqsave(&q->lock, flags); for (i = 0; i < q->ndesc; i++) { - err = mt76u_submit_buf(dev, USB_DIR_IN, MT_EP_IN_PKT_RX, - &q->entry[i].ubuf, GFP_ATOMIC, - mt76u_complete_rx, dev); + err = mt76u_submit_rx_buf(dev, q->entry[i].urb); if (err < 0) break; } @@ -563,7 +559,6 @@ int mt76u_submit_rx_buffers(struct mt76_dev *dev) return err; } -EXPORT_SYMBOL_GPL(mt76u_submit_rx_buffers); static int mt76u_alloc_rx(struct mt76_dev *dev) { @@ -575,7 +570,6 @@ static int mt76u_alloc_rx(struct mt76_dev *dev) if (!usb->mcu.data) return -ENOMEM; - spin_lock_init(&q->rx_page_lock); spin_lock_init(&q->lock); q->entry = devm_kcalloc(dev->dev, MT_NUM_RX_ENTRIES, sizeof(*q->entry), @@ -586,7 +580,7 @@ static int mt76u_alloc_rx(struct mt76_dev *dev) q->buf_size = dev->usb.sg_en ? MT_RX_BUF_SIZE : PAGE_SIZE; q->ndesc = MT_NUM_RX_ENTRIES; for (i = 0; i < q->ndesc; i++) { - err = mt76u_buf_alloc(dev, &q->entry[i].ubuf); + err = mt76u_rx_urb_alloc(dev, &q->entry[i]); if (err < 0) return err; } @@ -601,60 +595,76 @@ static void mt76u_free_rx(struct mt76_dev *dev) int i; for (i = 0; i < q->ndesc; i++) - mt76u_buf_free(&q->entry[i].ubuf); + mt76u_urb_free(q->entry[i].urb); - spin_lock_bh(&q->rx_page_lock); if (!q->rx_page.va) - goto out; + return; page = virt_to_page(q->rx_page.va); __page_frag_cache_drain(page, q->rx_page.pagecnt_bias); memset(&q->rx_page, 0, sizeof(q->rx_page)); -out: - spin_unlock_bh(&q->rx_page_lock); } -static void mt76u_stop_rx(struct mt76_dev *dev) +void mt76u_stop_rx(struct mt76_dev *dev) { struct mt76_queue *q = &dev->q_rx[MT_RXQ_MAIN]; int i; for (i = 0; i < q->ndesc; i++) - usb_kill_urb(q->entry[i].ubuf.urb); + usb_poison_urb(q->entry[i].urb); + + tasklet_kill(&dev->usb.rx_tasklet); } +EXPORT_SYMBOL_GPL(mt76u_stop_rx); + +int mt76u_resume_rx(struct mt76_dev *dev) +{ + struct mt76_queue *q = &dev->q_rx[MT_RXQ_MAIN]; + int i; + + for (i = 0; i < q->ndesc; i++) + usb_unpoison_urb(q->entry[i].urb); + + return mt76u_submit_rx_buffers(dev); +} +EXPORT_SYMBOL_GPL(mt76u_resume_rx); static void mt76u_tx_tasklet(unsigned long data) { struct mt76_dev *dev = (struct mt76_dev *)data; struct mt76_queue_entry entry; - struct mt76u_buf *buf; + struct mt76_sw_queue *sq; struct mt76_queue *q; bool wake; int i; for (i = 0; i < IEEE80211_NUM_ACS; i++) { - q = &dev->q_tx[i]; + u32 n_dequeued = 0, n_sw_dequeued = 0; - spin_lock_bh(&q->lock); - while (true) { - buf = &q->entry[q->head].ubuf; - if (!buf->done || !q->queued) + sq = &dev->q_tx[i]; + q = sq->q; + + while (q->queued > n_dequeued) { + if (!q->entry[q->head].done) break; if (q->entry[q->head].schedule) { q->entry[q->head].schedule = false; - q->swq_queued--; + n_sw_dequeued++; } entry = q->entry[q->head]; + q->entry[q->head].done = false; q->head = (q->head + 1) % q->ndesc; - q->queued--; + n_dequeued++; - spin_unlock_bh(&q->lock); - dev->drv->tx_complete_skb(dev, q, &entry, false); - spin_lock_bh(&q->lock); + dev->drv->tx_complete_skb(dev, i, &entry); } - mt76_txq_schedule(dev, q); + + spin_lock_bh(&q->lock); + + sq->swq_queued -= n_sw_dequeued; + q->queued -= n_dequeued; wake = q->stopped && q->queued < q->ndesc - 8; if (wake) @@ -665,6 +675,8 @@ static void mt76u_tx_tasklet(unsigned long data) spin_unlock_bh(&q->lock); + mt76_txq_schedule(dev, i); + if (!test_and_set_bit(MT76_READING_STATS, &dev->state)) ieee80211_queue_delayed_work(dev->hw, &dev->usb.stat_work, @@ -703,34 +715,43 @@ static void mt76u_tx_status_data(struct work_struct *work) static void mt76u_complete_tx(struct urb *urb) { - struct mt76u_buf *buf = urb->context; - struct mt76_dev *dev = buf->dev; + struct mt76_dev *dev = dev_get_drvdata(&urb->dev->dev); + struct mt76_queue_entry *e = urb->context; if (mt76u_urb_error(urb)) dev_err(dev->dev, "tx urb failed: %d\n", urb->status); - buf->done = true; + e->done = true; - tasklet_schedule(&dev->usb.tx_tasklet); + tasklet_schedule(&dev->tx_tasklet); } static int -mt76u_tx_build_sg(struct mt76_dev *dev, struct sk_buff *skb, - struct urb *urb) +mt76u_tx_setup_buffers(struct mt76_dev *dev, struct sk_buff *skb, + struct urb *urb) { - if (!dev->usb.sg_en) - return 0; + urb->transfer_buffer_length = skb->len; - sg_init_table(urb->sg, MT_SG_MAX_SIZE); - urb->num_sgs = skb_to_sgvec(skb, urb->sg, 0, skb->len); - return urb->num_sgs; + if (!dev->usb.sg_en) { + urb->transfer_buffer = skb->data; + return 0; + } else { + sg_init_table(urb->sg, MT_SG_MAX_SIZE); + urb->num_sgs = skb_to_sgvec(skb, urb->sg, 0, skb->len); + if (urb->num_sgs == 0) + return -ENOMEM; + return urb->num_sgs; + } } static int -mt76u_tx_queue_skb(struct mt76_dev *dev, struct mt76_queue *q, +mt76u_tx_queue_skb(struct mt76_dev *dev, enum mt76_txq_id qid, struct sk_buff *skb, struct mt76_wcid *wcid, struct ieee80211_sta *sta) { - struct mt76u_buf *buf; + struct mt76_queue *q = dev->q_tx[qid].q; + struct mt76_tx_info tx_info = { + .skb = skb, + }; u16 idx = q->tail; int err; @@ -738,24 +759,20 @@ mt76u_tx_queue_skb(struct mt76_dev *dev, struct mt76_queue *q, return -ENOSPC; skb->prev = skb->next = NULL; - err = dev->drv->tx_prepare_skb(dev, NULL, skb, q, wcid, sta, NULL); + err = dev->drv->tx_prepare_skb(dev, NULL, qid, wcid, sta, &tx_info); if (err < 0) return err; - buf = &q->entry[idx].ubuf; - buf->buf = skb->data; - buf->len = skb->len; - buf->done = false; - - err = mt76u_tx_build_sg(dev, skb, buf->urb); + err = mt76u_tx_setup_buffers(dev, tx_info.skb, q->entry[idx].urb); if (err < 0) return err; mt76u_fill_bulk_urb(dev, USB_DIR_OUT, q2ep(q->hw_idx), - buf, mt76u_complete_tx, buf); + q->entry[idx].urb, mt76u_complete_tx, + &q->entry[idx]); q->tail = (q->tail + 1) % q->ndesc; - q->entry[idx].skb = skb; + q->entry[idx].skb = tx_info.skb; q->queued++; return idx; @@ -763,14 +780,14 @@ mt76u_tx_queue_skb(struct mt76_dev *dev, struct mt76_queue *q, static void mt76u_tx_kick(struct mt76_dev *dev, struct mt76_queue *q) { - struct mt76u_buf *buf; + struct urb *urb; int err; while (q->first != q->tail) { - buf = &q->entry[q->first].ubuf; + urb = q->entry[q->first].urb; - trace_submit_urb(dev, buf->urb); - err = usb_submit_urb(buf->urb, GFP_ATOMIC); + trace_submit_urb(dev, urb); + err = usb_submit_urb(urb, GFP_ATOMIC); if (err < 0) { if (err == -ENODEV) set_bit(MT76_REMOVED, &dev->state); @@ -785,15 +802,24 @@ static void mt76u_tx_kick(struct mt76_dev *dev, struct mt76_queue *q) static int mt76u_alloc_tx(struct mt76_dev *dev) { - struct mt76u_buf *buf; struct mt76_queue *q; - int i, j; + int i, j, err; + + for (i = 0; i <= MT_TXQ_PSD; i++) { + INIT_LIST_HEAD(&dev->q_tx[i].swq); + + if (i >= IEEE80211_NUM_ACS) { + dev->q_tx[i].q = dev->q_tx[0].q; + continue; + } + + q = devm_kzalloc(dev->dev, sizeof(*q), GFP_KERNEL); + if (!q) + return -ENOMEM; - for (i = 0; i < IEEE80211_NUM_ACS; i++) { - q = &dev->q_tx[i]; spin_lock_init(&q->lock); - INIT_LIST_HEAD(&q->swq); q->hw_idx = mt76_ac_to_hwq(i); + dev->q_tx[i].q = q; q->entry = devm_kcalloc(dev->dev, MT_NUM_TX_ENTRIES, sizeof(*q->entry), @@ -803,22 +829,9 @@ static int mt76u_alloc_tx(struct mt76_dev *dev) q->ndesc = MT_NUM_TX_ENTRIES; for (j = 0; j < q->ndesc; j++) { - buf = &q->entry[j].ubuf; - buf->dev = dev; - - buf->urb = usb_alloc_urb(0, GFP_KERNEL); - if (!buf->urb) - return -ENOMEM; - - if (dev->usb.sg_en) { - size_t size = MT_SG_MAX_SIZE * - sizeof(struct scatterlist); - - buf->urb->sg = devm_kzalloc(dev->dev, size, - GFP_KERNEL); - if (!buf->urb->sg) - return -ENOMEM; - } + err = mt76u_urb_alloc(dev, &q->entry[j]); + if (err < 0) + return err; } } return 0; @@ -830,44 +843,60 @@ static void mt76u_free_tx(struct mt76_dev *dev) int i, j; for (i = 0; i < IEEE80211_NUM_ACS; i++) { - q = &dev->q_tx[i]; + q = dev->q_tx[i].q; for (j = 0; j < q->ndesc; j++) - usb_free_urb(q->entry[j].ubuf.urb); + usb_free_urb(q->entry[j].urb); } } -static void mt76u_stop_tx(struct mt76_dev *dev) +void mt76u_stop_tx(struct mt76_dev *dev) { + struct mt76_queue_entry entry; struct mt76_queue *q; - int i, j; + int i, j, ret; - for (i = 0; i < IEEE80211_NUM_ACS; i++) { - q = &dev->q_tx[i]; - for (j = 0; j < q->ndesc; j++) - usb_kill_urb(q->entry[j].ubuf.urb); - } -} + ret = wait_event_timeout(dev->tx_wait, !mt76_has_tx_pending(dev), HZ/5); + if (!ret) { + dev_err(dev->dev, "timed out waiting for pending tx\n"); -void mt76u_stop_queues(struct mt76_dev *dev) -{ - tasklet_disable(&dev->usb.rx_tasklet); - tasklet_disable(&dev->usb.tx_tasklet); + for (i = 0; i < IEEE80211_NUM_ACS; i++) { + q = dev->q_tx[i].q; + for (j = 0; j < q->ndesc; j++) + usb_kill_urb(q->entry[j].urb); + } - mt76u_stop_rx(dev); - mt76u_stop_tx(dev); -} -EXPORT_SYMBOL_GPL(mt76u_stop_queues); + tasklet_kill(&dev->tx_tasklet); + + /* On device removal we maight queue skb's, but mt76u_tx_kick() + * will fail to submit urb, cleanup those skb's manually. + */ + for (i = 0; i < IEEE80211_NUM_ACS; i++) { + q = dev->q_tx[i].q; + + /* Assure we are in sync with killed tasklet. */ + spin_lock_bh(&q->lock); + while (q->queued) { + entry = q->entry[q->head]; + q->head = (q->head + 1) % q->ndesc; + q->queued--; + + dev->drv->tx_complete_skb(dev, i, &entry); + } + spin_unlock_bh(&q->lock); + } + } -void mt76u_stop_stat_wk(struct mt76_dev *dev) -{ cancel_delayed_work_sync(&dev->usb.stat_work); clear_bit(MT76_READING_STATS, &dev->state); + + mt76_tx_status_check(dev, NULL, true); } -EXPORT_SYMBOL_GPL(mt76u_stop_stat_wk); +EXPORT_SYMBOL_GPL(mt76u_stop_tx); void mt76u_queues_deinit(struct mt76_dev *dev) { - mt76u_stop_queues(dev); + mt76u_stop_rx(dev); + mt76u_stop_tx(dev); mt76u_free_rx(dev); mt76u_free_tx(dev); @@ -906,7 +935,7 @@ int mt76u_init(struct mt76_dev *dev, struct mt76_usb *usb = &dev->usb; tasklet_init(&usb->rx_tasklet, mt76u_rx_tasklet, (unsigned long)dev); - tasklet_init(&usb->tx_tasklet, mt76u_tx_tasklet, (unsigned long)dev); + tasklet_init(&dev->tx_tasklet, mt76u_tx_tasklet, (unsigned long)dev); INIT_DELAYED_WORK(&usb->stat_work, mt76u_tx_status_data); skb_queue_head_init(&dev->rx_skb[MT_RXQ_MAIN]); diff --git a/drivers/net/wireless/quantenna/qtnfmac/bus.h b/drivers/net/wireless/quantenna/qtnfmac/bus.h index 14b569b6d1b5be62f28b1c094c5a4bb7e0d29dd7..7cea08f71838e7844076a5324ae79a115996089d 100644 --- a/drivers/net/wireless/quantenna/qtnfmac/bus.h +++ b/drivers/net/wireless/quantenna/qtnfmac/bus.h @@ -13,12 +13,11 @@ #define QTNF_MAX_MAC 3 enum qtnf_fw_state { - QTNF_FW_STATE_RESET, - QTNF_FW_STATE_FW_DNLD_DONE, + QTNF_FW_STATE_DETACHED, QTNF_FW_STATE_BOOT_DONE, QTNF_FW_STATE_ACTIVE, - QTNF_FW_STATE_DETACHED, - QTNF_FW_STATE_EP_DEAD, + QTNF_FW_STATE_RUNNING, + QTNF_FW_STATE_DEAD, }; struct qtnf_bus; @@ -50,6 +49,7 @@ struct qtnf_bus { struct napi_struct mux_napi; struct net_device mux_dev; struct workqueue_struct *workqueue; + struct workqueue_struct *hprio_workqueue; struct work_struct fw_work; struct work_struct event_work; struct mutex bus_lock; /* lock during command/event processing */ @@ -58,6 +58,23 @@ struct qtnf_bus { char bus_priv[0] __aligned(sizeof(void *)); }; +static inline bool qtnf_fw_is_up(struct qtnf_bus *bus) +{ + enum qtnf_fw_state state = bus->fw_state; + + return ((state == QTNF_FW_STATE_ACTIVE) || + (state == QTNF_FW_STATE_RUNNING)); +} + +static inline bool qtnf_fw_is_attached(struct qtnf_bus *bus) +{ + enum qtnf_fw_state state = bus->fw_state; + + return ((state == QTNF_FW_STATE_ACTIVE) || + (state == QTNF_FW_STATE_RUNNING) || + (state == QTNF_FW_STATE_DEAD)); +} + static inline void *get_bus_priv(struct qtnf_bus *bus) { if (WARN(!bus, "qtnfmac: invalid bus pointer")) diff --git a/drivers/net/wireless/quantenna/qtnfmac/cfg80211.c b/drivers/net/wireless/quantenna/qtnfmac/cfg80211.c index dcb0991432f4ef657866747089f1c343319c154b..d90016125dfcacb5ff57f8abaeace331c94de7dc 100644 --- a/drivers/net/wireless/quantenna/qtnfmac/cfg80211.c +++ b/drivers/net/wireless/quantenna/qtnfmac/cfg80211.c @@ -144,6 +144,7 @@ int qtnf_del_virtual_intf(struct wiphy *wiphy, struct wireless_dev *wdev) { struct net_device *netdev = wdev->netdev; struct qtnf_vif *vif; + struct sk_buff *skb; if (WARN_ON(!netdev)) return -EFAULT; @@ -157,6 +158,11 @@ int qtnf_del_virtual_intf(struct wiphy *wiphy, struct wireless_dev *wdev) if (netif_carrier_ok(netdev)) netif_carrier_off(netdev); + while ((skb = skb_dequeue(&vif->high_pri_tx_queue))) + dev_kfree_skb_any(skb); + + cancel_work_sync(&vif->high_pri_tx_work); + if (netdev->reg_state == NETREG_REGISTERED) unregister_netdevice(netdev); @@ -424,13 +430,13 @@ qtnf_mgmt_tx(struct wiphy *wiphy, struct wireless_dev *wdev, *cookie = short_cookie; if (params->offchan) - flags |= QLINK_MGMT_FRAME_TX_FLAG_OFFCHAN; + flags |= QLINK_FRAME_TX_FLAG_OFFCHAN; if (params->no_cck) - flags |= QLINK_MGMT_FRAME_TX_FLAG_NO_CCK; + flags |= QLINK_FRAME_TX_FLAG_NO_CCK; if (params->dont_wait_for_ack) - flags |= QLINK_MGMT_FRAME_TX_FLAG_ACK_NOWAIT; + flags |= QLINK_FRAME_TX_FLAG_ACK_NOWAIT; /* If channel is not specified, pass "freq = 0" to tell device * firmware to use current channel. @@ -445,9 +451,8 @@ qtnf_mgmt_tx(struct wiphy *wiphy, struct wireless_dev *wdev, le16_to_cpu(mgmt_frame->frame_control), mgmt_frame->da, params->len, short_cookie, flags); - return qtnf_cmd_send_mgmt_frame(vif, short_cookie, flags, - freq, - params->buf, params->len); + return qtnf_cmd_send_frame(vif, short_cookie, flags, + freq, params->buf, params->len); } static int @@ -993,53 +998,31 @@ static struct cfg80211_ops qtn_cfg80211_ops = { #endif }; -static void qtnf_cfg80211_reg_notifier(struct wiphy *wiphy_in, +static void qtnf_cfg80211_reg_notifier(struct wiphy *wiphy, struct regulatory_request *req) { - struct qtnf_wmac *mac = wiphy_priv(wiphy_in); - struct qtnf_bus *bus = mac->bus; - struct wiphy *wiphy; - unsigned int mac_idx; + struct qtnf_wmac *mac = wiphy_priv(wiphy); enum nl80211_band band; int ret; pr_debug("MAC%u: initiator=%d alpha=%c%c\n", mac->macid, req->initiator, req->alpha2[0], req->alpha2[1]); - ret = qtnf_cmd_reg_notify(bus, req); + ret = qtnf_cmd_reg_notify(mac, req, qtnf_mac_slave_radar_get(wiphy)); if (ret) { - if (ret == -EOPNOTSUPP) { - pr_warn("reg update not supported\n"); - } else if (ret == -EALREADY) { - pr_info("regulatory domain is already set to %c%c", - req->alpha2[0], req->alpha2[1]); - } else { - pr_err("failed to update reg domain to %c%c\n", - req->alpha2[0], req->alpha2[1]); - } - + pr_err("MAC%u: failed to update region to %c%c: %d\n", + mac->macid, req->alpha2[0], req->alpha2[1], ret); return; } - for (mac_idx = 0; mac_idx < QTNF_MAX_MAC; ++mac_idx) { - if (!(bus->hw_info.mac_bitmap & (1 << mac_idx))) + for (band = 0; band < NUM_NL80211_BANDS; ++band) { + if (!wiphy->bands[band]) continue; - mac = bus->mac[mac_idx]; - if (!mac) - continue; - - wiphy = priv_to_wiphy(mac); - - for (band = 0; band < NUM_NL80211_BANDS; ++band) { - if (!wiphy->bands[band]) - continue; - - ret = qtnf_cmd_band_info_get(mac, wiphy->bands[band]); - if (ret) - pr_err("failed to get chan info for mac %u band %u\n", - mac_idx, band); - } + ret = qtnf_cmd_band_info_get(mac, wiphy->bands[band]); + if (ret) + pr_err("MAC%u: failed to update band %u\n", + mac->macid, band); } } @@ -1095,6 +1078,7 @@ int qtnf_wiphy_register(struct qtnf_hw_info *hw_info, struct qtnf_wmac *mac) struct wiphy *wiphy = priv_to_wiphy(mac); struct qtnf_mac_info *macinfo = &mac->macinfo; int ret; + bool regdomain_is_known; if (!wiphy) { pr_err("invalid wiphy pointer\n"); @@ -1127,7 +1111,8 @@ int qtnf_wiphy_register(struct qtnf_hw_info *hw_info, struct qtnf_wmac *mac) WIPHY_FLAG_AP_PROBE_RESP_OFFLOAD | WIPHY_FLAG_AP_UAPSD | WIPHY_FLAG_HAS_CHANNEL_SWITCH | - WIPHY_FLAG_4ADDR_STATION; + WIPHY_FLAG_4ADDR_STATION | + WIPHY_FLAG_NETNS_OK; wiphy->flags &= ~WIPHY_FLAG_PS_ON_BY_DEFAULT; if (hw_info->hw_capab & QLINK_HW_CAPAB_DFS_OFFLOAD) @@ -1166,11 +1151,19 @@ int qtnf_wiphy_register(struct qtnf_hw_info *hw_info, struct qtnf_wmac *mac) wiphy->wowlan = macinfo->wowlan; #endif + regdomain_is_known = isalpha(mac->rd->alpha2[0]) && + isalpha(mac->rd->alpha2[1]); + if (hw_info->hw_capab & QLINK_HW_CAPAB_REG_UPDATE) { - wiphy->regulatory_flags |= REGULATORY_STRICT_REG | - REGULATORY_CUSTOM_REG; wiphy->reg_notifier = qtnf_cfg80211_reg_notifier; - wiphy_apply_custom_regulatory(wiphy, hw_info->rd); + + if (mac->rd->alpha2[0] == '9' && mac->rd->alpha2[1] == '9') { + wiphy->regulatory_flags |= REGULATORY_CUSTOM_REG | + REGULATORY_STRICT_REG; + wiphy_apply_custom_regulatory(wiphy, mac->rd); + } else if (regdomain_is_known) { + wiphy->regulatory_flags |= REGULATORY_STRICT_REG; + } } else { wiphy->regulatory_flags |= REGULATORY_WIPHY_SELF_MANAGED; } @@ -1193,10 +1186,9 @@ int qtnf_wiphy_register(struct qtnf_hw_info *hw_info, struct qtnf_wmac *mac) goto out; if (wiphy->regulatory_flags & REGULATORY_WIPHY_SELF_MANAGED) - ret = regulatory_set_wiphy_regd(wiphy, hw_info->rd); - else if (isalpha(hw_info->rd->alpha2[0]) && - isalpha(hw_info->rd->alpha2[1])) - ret = regulatory_hint(wiphy, hw_info->rd->alpha2); + ret = regulatory_set_wiphy_regd(wiphy, mac->rd); + else if (regdomain_is_known) + ret = regulatory_hint(wiphy, mac->rd->alpha2); out: return ret; diff --git a/drivers/net/wireless/quantenna/qtnfmac/commands.c b/drivers/net/wireless/quantenna/qtnfmac/commands.c index 85a2a58f4c168bf6ece7845e1a701f3570cab0ed..459f6b81d2eb027b05cfbcf78bf64d5fd2d6e8c5 100644 --- a/drivers/net/wireless/quantenna/qtnfmac/commands.c +++ b/drivers/net/wireless/quantenna/qtnfmac/commands.c @@ -11,6 +11,13 @@ #include "bus.h" #include "commands.h" +#define QTNF_SCAN_TIME_AUTO 0 + +/* Let device itself to select best values for current conditions */ +#define QTNF_SCAN_DWELL_ACTIVE_DEFAULT QTNF_SCAN_TIME_AUTO +#define QTNF_SCAN_DWELL_PASSIVE_DEFAULT QTNF_SCAN_TIME_AUTO +#define QTNF_SCAN_SAMPLE_DURATION_DEFAULT QTNF_SCAN_TIME_AUTO + static int qtnf_cmd_check_reply_header(const struct qlink_resp *resp, u16 cmd_id, u8 mac_id, u8 vif_id, size_t resp_size) @@ -89,8 +96,7 @@ static int qtnf_cmd_send_with_reply(struct qtnf_bus *bus, pr_debug("VIF%u.%u cmd=0x%.4X\n", mac_id, vif_id, cmd_id); - if (bus->fw_state != QTNF_FW_STATE_ACTIVE && - cmd_id != QLINK_CMD_FW_INIT) { + if (!qtnf_fw_is_up(bus) && cmd_id != QLINK_CMD_FW_INIT) { pr_warn("VIF%u.%u: drop cmd 0x%.4X in fw state %d\n", mac_id, vif_id, cmd_id, bus->fw_state); dev_kfree_skb(cmd_skb); @@ -177,14 +183,6 @@ static void qtnf_cmd_tlv_ie_set_add(struct sk_buff *cmd_skb, u8 frame_type, memcpy(tlv->ie_data, buf, len); } -static inline size_t qtnf_cmd_acl_data_size(const struct cfg80211_acl_data *acl) -{ - size_t size = sizeof(struct qlink_acl_data) + - acl->n_acl_entries * sizeof(struct qlink_mac_address); - - return size; -} - static bool qtnf_cmd_start_ap_can_fit(const struct qtnf_vif *vif, const struct cfg80211_ap_settings *s) { @@ -203,7 +201,7 @@ static bool qtnf_cmd_start_ap_can_fit(const struct qtnf_vif *vif, if (s->acl) len += sizeof(struct qlink_tlv_hdr) + - qtnf_cmd_acl_data_size(s->acl); + struct_size(s->acl, mac_addrs, s->acl->n_acl_entries); if (len > (sizeof(struct qlink_cmd) + QTNF_MAX_CMD_BUF_SIZE)) { pr_err("VIF%u.%u: can not fit AP settings: %u\n", @@ -310,7 +308,8 @@ int qtnf_cmd_send_start_ap(struct qtnf_vif *vif, } if (s->acl) { - size_t acl_size = qtnf_cmd_acl_data_size(s->acl); + size_t acl_size = struct_size(s->acl, mac_addrs, + s->acl->n_acl_entries); struct qlink_tlv_hdr *tlv = skb_put(cmd_skb, sizeof(*tlv) + acl_size); @@ -382,11 +381,11 @@ out: return ret; } -int qtnf_cmd_send_mgmt_frame(struct qtnf_vif *vif, u32 cookie, u16 flags, - u16 freq, const u8 *buf, size_t len) +int qtnf_cmd_send_frame(struct qtnf_vif *vif, u32 cookie, u16 flags, + u16 freq, const u8 *buf, size_t len) { struct sk_buff *cmd_skb; - struct qlink_cmd_mgmt_frame_tx *cmd; + struct qlink_cmd_frame_tx *cmd; int ret; if (sizeof(*cmd) + len > QTNF_MAX_CMD_BUF_SIZE) { @@ -396,14 +395,14 @@ int qtnf_cmd_send_mgmt_frame(struct qtnf_vif *vif, u32 cookie, u16 flags, } cmd_skb = qtnf_cmd_alloc_new_cmdskb(vif->mac->macid, vif->vifid, - QLINK_CMD_SEND_MGMT_FRAME, + QLINK_CMD_SEND_FRAME, sizeof(*cmd)); if (!cmd_skb) return -ENOMEM; qtnf_bus_lock(vif->mac->bus); - cmd = (struct qlink_cmd_mgmt_frame_tx *)cmd_skb->data; + cmd = (struct qlink_cmd_frame_tx *)cmd_skb->data; cmd->cookie = cpu_to_le32(cookie); cmd->freq = cpu_to_le16(freq); cmd->flags = cpu_to_le16(flags); @@ -786,8 +785,25 @@ int qtnf_cmd_send_change_intf_type(struct qtnf_vif *vif, int use4addr, u8 *mac_addr) { - return qtnf_cmd_send_add_change_intf(vif, iftype, use4addr, mac_addr, - QLINK_CMD_CHANGE_INTF); + int ret; + + ret = qtnf_cmd_send_add_change_intf(vif, iftype, use4addr, mac_addr, + QLINK_CMD_CHANGE_INTF); + + /* Regulatory settings may be different for different interface types */ + if (ret == 0 && vif->wdev.iftype != iftype) { + enum nl80211_band band; + struct wiphy *wiphy = priv_to_wiphy(vif->mac); + + for (band = 0; band < NUM_NL80211_BANDS; ++band) { + if (!wiphy->bands[band]) + continue; + + qtnf_cmd_band_info_get(vif->mac, wiphy->bands[band]); + } + } + + return ret; } int qtnf_cmd_send_del_intf(struct qtnf_vif *vif) @@ -831,55 +847,6 @@ out: return ret; } -static u32 qtnf_cmd_resp_reg_rule_flags_parse(u32 qflags) -{ - u32 flags = 0; - - if (qflags & QLINK_RRF_NO_OFDM) - flags |= NL80211_RRF_NO_OFDM; - - if (qflags & QLINK_RRF_NO_CCK) - flags |= NL80211_RRF_NO_CCK; - - if (qflags & QLINK_RRF_NO_INDOOR) - flags |= NL80211_RRF_NO_INDOOR; - - if (qflags & QLINK_RRF_NO_OUTDOOR) - flags |= NL80211_RRF_NO_OUTDOOR; - - if (qflags & QLINK_RRF_DFS) - flags |= NL80211_RRF_DFS; - - if (qflags & QLINK_RRF_PTP_ONLY) - flags |= NL80211_RRF_PTP_ONLY; - - if (qflags & QLINK_RRF_PTMP_ONLY) - flags |= NL80211_RRF_PTMP_ONLY; - - if (qflags & QLINK_RRF_NO_IR) - flags |= NL80211_RRF_NO_IR; - - if (qflags & QLINK_RRF_AUTO_BW) - flags |= NL80211_RRF_AUTO_BW; - - if (qflags & QLINK_RRF_IR_CONCURRENT) - flags |= NL80211_RRF_IR_CONCURRENT; - - if (qflags & QLINK_RRF_NO_HT40MINUS) - flags |= NL80211_RRF_NO_HT40MINUS; - - if (qflags & QLINK_RRF_NO_HT40PLUS) - flags |= NL80211_RRF_NO_HT40PLUS; - - if (qflags & QLINK_RRF_NO_80MHZ) - flags |= NL80211_RRF_NO_80MHZ; - - if (qflags & QLINK_RRF_NO_160MHZ) - flags |= NL80211_RRF_NO_160MHZ; - - return flags; -} - static int qtnf_cmd_resp_proc_hw_info(struct qtnf_bus *bus, const struct qlink_resp_get_hw_info *resp, @@ -887,7 +854,6 @@ qtnf_cmd_resp_proc_hw_info(struct qtnf_bus *bus, { struct qtnf_hw_info *hwinfo = &bus->hw_info; const struct qlink_tlv_hdr *tlv; - const struct qlink_tlv_reg_rule *tlv_rule; const char *bld_name = NULL; const char *bld_rev = NULL; const char *bld_type = NULL; @@ -898,19 +864,8 @@ qtnf_cmd_resp_proc_hw_info(struct qtnf_bus *bus, const char *calibration_ver = NULL; const char *uboot_ver = NULL; u32 hw_ver = 0; - struct ieee80211_reg_rule *rule; u16 tlv_type; u16 tlv_value_len; - unsigned int rule_idx = 0; - - if (WARN_ON(resp->n_reg_rules > NL80211_MAX_SUPP_REG_RULES)) - return -E2BIG; - - hwinfo->rd = kzalloc(struct_size(hwinfo->rd, reg_rules, - resp->n_reg_rules), GFP_KERNEL); - - if (!hwinfo->rd) - return -ENOMEM; hwinfo->num_mac = resp->num_mac; hwinfo->mac_bitmap = resp->mac_bitmap; @@ -919,30 +874,11 @@ qtnf_cmd_resp_proc_hw_info(struct qtnf_bus *bus, hwinfo->total_tx_chain = resp->total_tx_chain; hwinfo->total_rx_chain = resp->total_rx_chain; hwinfo->hw_capab = le32_to_cpu(resp->hw_capab); - hwinfo->rd->n_reg_rules = resp->n_reg_rules; - hwinfo->rd->alpha2[0] = resp->alpha2[0]; - hwinfo->rd->alpha2[1] = resp->alpha2[1]; bld_tmstamp = le32_to_cpu(resp->bld_tmstamp); plat_id = le32_to_cpu(resp->plat_id); hw_ver = le32_to_cpu(resp->hw_ver); - switch (resp->dfs_region) { - case QLINK_DFS_FCC: - hwinfo->rd->dfs_region = NL80211_DFS_FCC; - break; - case QLINK_DFS_ETSI: - hwinfo->rd->dfs_region = NL80211_DFS_ETSI; - break; - case QLINK_DFS_JP: - hwinfo->rd->dfs_region = NL80211_DFS_JP; - break; - case QLINK_DFS_UNSET: - default: - hwinfo->rd->dfs_region = NL80211_DFS_UNSET; - break; - } - tlv = (const struct qlink_tlv_hdr *)resp->info; while (info_len >= sizeof(*tlv)) { @@ -956,37 +892,6 @@ qtnf_cmd_resp_proc_hw_info(struct qtnf_bus *bus, } switch (tlv_type) { - case QTN_TLV_ID_REG_RULE: - if (rule_idx >= resp->n_reg_rules) { - pr_warn("unexpected number of rules: %u\n", - resp->n_reg_rules); - return -EINVAL; - } - - if (tlv_value_len != sizeof(*tlv_rule) - sizeof(*tlv)) { - pr_warn("malformed TLV 0x%.2X; LEN: %u\n", - tlv_type, tlv_value_len); - return -EINVAL; - } - - tlv_rule = (const struct qlink_tlv_reg_rule *)tlv; - rule = &hwinfo->rd->reg_rules[rule_idx++]; - - rule->freq_range.start_freq_khz = - le32_to_cpu(tlv_rule->start_freq_khz); - rule->freq_range.end_freq_khz = - le32_to_cpu(tlv_rule->end_freq_khz); - rule->freq_range.max_bandwidth_khz = - le32_to_cpu(tlv_rule->max_bandwidth_khz); - rule->power_rule.max_antenna_gain = - le32_to_cpu(tlv_rule->max_antenna_gain); - rule->power_rule.max_eirp = - le32_to_cpu(tlv_rule->max_eirp); - rule->dfs_cac_ms = - le32_to_cpu(tlv_rule->dfs_cac_ms); - rule->flags = qtnf_cmd_resp_reg_rule_flags_parse( - le32_to_cpu(tlv_rule->flags)); - break; case QTN_TLV_ID_BUILD_NAME: bld_name = (const void *)tlv->val; break; @@ -1019,17 +924,8 @@ qtnf_cmd_resp_proc_hw_info(struct qtnf_bus *bus, tlv = (struct qlink_tlv_hdr *)(tlv->val + tlv_value_len); } - if (rule_idx != resp->n_reg_rules) { - pr_warn("unexpected number of rules: expected %u got %u\n", - resp->n_reg_rules, rule_idx); - kfree(hwinfo->rd); - hwinfo->rd = NULL; - return -EINVAL; - } - - pr_info("fw_version=%d, MACs map %#x, alpha2=\"%c%c\", chains Tx=%u Rx=%u, capab=0x%x\n", + pr_info("fw_version=%d, MACs map %#x, chains Tx=%u Rx=%u, capab=0x%x\n", hwinfo->fw_ver, hwinfo->mac_bitmap, - hwinfo->rd->alpha2[0], hwinfo->rd->alpha2[1], hwinfo->total_tx_chain, hwinfo->total_rx_chain, hwinfo->hw_capab); @@ -1042,7 +938,7 @@ qtnf_cmd_resp_proc_hw_info(struct qtnf_bus *bus, "\nHardware ID: %s" \ "\nCalibration version: %s" \ "\nU-Boot version: %s" \ - "\nHardware version: 0x%08x", + "\nHardware version: 0x%08x\n", bld_name, bld_rev, bld_type, bld_label, (unsigned long)bld_tmstamp, (unsigned long)plat_id, @@ -1085,9 +981,12 @@ qtnf_parse_wowlan_info(struct qtnf_wmac *mac, } } -static int qtnf_parse_variable_mac_info(struct qtnf_wmac *mac, - const u8 *tlv_buf, size_t tlv_buf_size) +static int +qtnf_parse_variable_mac_info(struct qtnf_wmac *mac, + const struct qlink_resp_get_mac_info *resp, + size_t tlv_buf_size) { + const u8 *tlv_buf = resp->var_info; struct ieee80211_iface_combination *comb = NULL; size_t n_comb = 0; struct ieee80211_iface_limit *limits; @@ -1105,6 +1004,38 @@ static int qtnf_parse_variable_mac_info(struct qtnf_wmac *mac, u8 ext_capa_len = 0; u8 ext_capa_mask_len = 0; int i = 0; + struct ieee80211_reg_rule *rule; + unsigned int rule_idx = 0; + const struct qlink_tlv_reg_rule *tlv_rule; + + if (WARN_ON(resp->n_reg_rules > NL80211_MAX_SUPP_REG_RULES)) + return -E2BIG; + + mac->rd = kzalloc(sizeof(*mac->rd) + + sizeof(struct ieee80211_reg_rule) * + resp->n_reg_rules, GFP_KERNEL); + if (!mac->rd) + return -ENOMEM; + + mac->rd->n_reg_rules = resp->n_reg_rules; + mac->rd->alpha2[0] = resp->alpha2[0]; + mac->rd->alpha2[1] = resp->alpha2[1]; + + switch (resp->dfs_region) { + case QLINK_DFS_FCC: + mac->rd->dfs_region = NL80211_DFS_FCC; + break; + case QLINK_DFS_ETSI: + mac->rd->dfs_region = NL80211_DFS_ETSI; + break; + case QLINK_DFS_JP: + mac->rd->dfs_region = NL80211_DFS_JP; + break; + case QLINK_DFS_UNSET: + default: + mac->rd->dfs_region = NL80211_DFS_UNSET; + break; + } tlv = (const struct qlink_tlv_hdr *)tlv_buf; while (tlv_buf_size >= sizeof(struct qlink_tlv_hdr)) { @@ -1225,6 +1156,23 @@ static int qtnf_parse_variable_mac_info(struct qtnf_wmac *mac, mac->macinfo.wowlan = NULL; qtnf_parse_wowlan_info(mac, wowlan); break; + case QTN_TLV_ID_REG_RULE: + if (rule_idx >= resp->n_reg_rules) { + pr_warn("unexpected number of rules: %u\n", + resp->n_reg_rules); + return -EINVAL; + } + + if (tlv_value_len != sizeof(*tlv_rule) - sizeof(*tlv)) { + pr_warn("malformed TLV 0x%.2X; LEN: %u\n", + tlv_type, tlv_value_len); + return -EINVAL; + } + + tlv_rule = (const struct qlink_tlv_reg_rule *)tlv; + rule = &mac->rd->reg_rules[rule_idx++]; + qlink_utils_regrule_q2nl(rule, tlv_rule); + break; default: pr_warn("MAC%u: unknown TLV type %u\n", mac->macid, tlv_type); @@ -1253,6 +1201,12 @@ static int qtnf_parse_variable_mac_info(struct qtnf_wmac *mac, return -EINVAL; } + if (rule_idx != resp->n_reg_rules) { + pr_warn("unexpected number of rules: expected %u got %u\n", + resp->n_reg_rules, rule_idx); + return -EINVAL; + } + if (ext_capa_len > 0) { ext_capa = kmemdup(ext_capa, ext_capa_len, GFP_KERNEL); if (!ext_capa) @@ -1663,7 +1617,7 @@ int qtnf_cmd_get_mac_info(struct qtnf_wmac *mac) resp = (const struct qlink_resp_get_mac_info *)resp_skb->data; qtnf_cmd_resp_proc_mac_info(mac, resp); - ret = qtnf_parse_variable_mac_info(mac, resp->var_info, var_data_len); + ret = qtnf_parse_variable_mac_info(mac, resp, var_data_len); out: qtnf_bus_unlock(mac->bus); @@ -1709,21 +1663,7 @@ int qtnf_cmd_band_info_get(struct qtnf_wmac *mac, struct qlink_resp_band_info_get *resp; size_t info_len = 0; int ret = 0; - u8 qband; - - switch (band->band) { - case NL80211_BAND_2GHZ: - qband = QLINK_BAND_2GHZ; - break; - case NL80211_BAND_5GHZ: - qband = QLINK_BAND_5GHZ; - break; - case NL80211_BAND_60GHZ: - qband = QLINK_BAND_60GHZ; - break; - default: - return -EINVAL; - } + u8 qband = qlink_utils_band_cfg2q(band->band); cmd_skb = qtnf_cmd_alloc_new_cmdskb(mac->macid, 0, QLINK_CMD_BAND_INFO_GET, @@ -2107,22 +2047,23 @@ out: static void qtnf_cmd_channel_tlv_add(struct sk_buff *cmd_skb, const struct ieee80211_channel *sc) { - struct qlink_tlv_channel *qchan; - u32 flags = 0; - - qchan = skb_put_zero(cmd_skb, sizeof(*qchan)); - qchan->hdr.type = cpu_to_le16(QTN_TLV_ID_CHANNEL); - qchan->hdr.len = cpu_to_le16(sizeof(*qchan) - sizeof(qchan->hdr)); - qchan->chan.center_freq = cpu_to_le16(sc->center_freq); - qchan->chan.hw_value = cpu_to_le16(sc->hw_value); - - if (sc->flags & IEEE80211_CHAN_NO_IR) - flags |= QLINK_CHAN_NO_IR; - - if (sc->flags & IEEE80211_CHAN_RADAR) - flags |= QLINK_CHAN_RADAR; - - qchan->chan.flags = cpu_to_le32(flags); + struct qlink_tlv_channel *tlv; + struct qlink_channel *qch; + + tlv = skb_put_zero(cmd_skb, sizeof(*tlv)); + qch = &tlv->chan; + tlv->hdr.type = cpu_to_le16(QTN_TLV_ID_CHANNEL); + tlv->hdr.len = cpu_to_le16(sizeof(*qch)); + + qch->center_freq = cpu_to_le16(sc->center_freq); + qch->hw_value = cpu_to_le16(sc->hw_value); + qch->band = qlink_utils_band_cfg2q(sc->band); + qch->max_power = sc->max_power; + qch->max_reg_power = sc->max_reg_power; + qch->max_antenna_gain = sc->max_antenna_gain; + qch->beacon_found = sc->beacon_found; + qch->dfs_state = qlink_utils_dfs_state_cfg2q(sc->dfs_state); + qch->flags = cpu_to_le32(qlink_utils_chflags_cfg2q(sc->flags)); } static void qtnf_cmd_randmac_tlv_add(struct sk_buff *cmd_skb, @@ -2141,6 +2082,35 @@ static void qtnf_cmd_randmac_tlv_add(struct sk_buff *cmd_skb, memcpy(randmac->mac_addr_mask, mac_addr_mask, ETH_ALEN); } +static void qtnf_cmd_scan_set_dwell(struct qtnf_wmac *mac, + struct sk_buff *cmd_skb) +{ + struct cfg80211_scan_request *scan_req = mac->scan_req; + u16 dwell_active = QTNF_SCAN_DWELL_ACTIVE_DEFAULT; + u16 dwell_passive = QTNF_SCAN_DWELL_PASSIVE_DEFAULT; + u16 duration = QTNF_SCAN_SAMPLE_DURATION_DEFAULT; + + if (scan_req->duration) { + dwell_active = scan_req->duration; + dwell_passive = scan_req->duration; + } + + pr_debug("MAC%u: %s scan dwell active=%u, passive=%u, duration=%u\n", + mac->macid, + scan_req->duration_mandatory ? "mandatory" : "max", + dwell_active, dwell_passive, duration); + + qtnf_cmd_skb_put_tlv_u16(cmd_skb, + QTN_TLV_ID_SCAN_DWELL_ACTIVE, + dwell_active); + qtnf_cmd_skb_put_tlv_u16(cmd_skb, + QTN_TLV_ID_SCAN_DWELL_PASSIVE, + dwell_passive); + qtnf_cmd_skb_put_tlv_u16(cmd_skb, + QTN_TLV_ID_SCAN_SAMPLE_DURATION, + duration); +} + int qtnf_cmd_send_scan(struct qtnf_wmac *mac) { struct sk_buff *cmd_skb; @@ -2192,6 +2162,8 @@ int qtnf_cmd_send_scan(struct qtnf_wmac *mac) } } + qtnf_cmd_scan_set_dwell(mac, cmd_skb); + if (scan_req->flags & NL80211_SCAN_FLAG_RANDOM_ADDR) { pr_debug("MAC%u: scan with random addr=%pM, mask=%pM\n", mac->macid, @@ -2207,15 +2179,6 @@ int qtnf_cmd_send_scan(struct qtnf_wmac *mac) qtnf_cmd_skb_put_tlv_tag(cmd_skb, QTN_TLV_ID_SCAN_FLUSH); } - if (scan_req->duration) { - pr_debug("MAC%u: %s scan duration %u\n", mac->macid, - scan_req->duration_mandatory ? "mandatory" : "max", - scan_req->duration); - - qtnf_cmd_skb_put_tlv_u16(cmd_skb, QTN_TLV_ID_SCAN_DWELL, - scan_req->duration); - } - ret = qtnf_cmd_send(mac->bus, cmd_skb); if (ret) goto out; @@ -2404,13 +2367,18 @@ out: return ret; } -int qtnf_cmd_reg_notify(struct qtnf_bus *bus, struct regulatory_request *req) +int qtnf_cmd_reg_notify(struct qtnf_wmac *mac, struct regulatory_request *req, + bool slave_radar) { + struct wiphy *wiphy = priv_to_wiphy(mac); + struct qtnf_bus *bus = mac->bus; struct sk_buff *cmd_skb; int ret; struct qlink_cmd_reg_notify *cmd; + enum nl80211_band band; + const struct ieee80211_supported_band *cfg_band; - cmd_skb = qtnf_cmd_alloc_new_cmdskb(QLINK_MACID_RSVD, QLINK_VIFID_RSVD, + cmd_skb = qtnf_cmd_alloc_new_cmdskb(mac->macid, QLINK_VIFID_RSVD, QLINK_CMD_REG_NOTIFY, sizeof(*cmd)); if (!cmd_skb) @@ -2447,12 +2415,41 @@ int qtnf_cmd_reg_notify(struct qtnf_bus *bus, struct regulatory_request *req) break; } + switch (req->dfs_region) { + case NL80211_DFS_FCC: + cmd->dfs_region = QLINK_DFS_FCC; + break; + case NL80211_DFS_ETSI: + cmd->dfs_region = QLINK_DFS_ETSI; + break; + case NL80211_DFS_JP: + cmd->dfs_region = QLINK_DFS_JP; + break; + default: + cmd->dfs_region = QLINK_DFS_UNSET; + break; + } + + cmd->slave_radar = slave_radar; + cmd->num_channels = 0; + + for (band = 0; band < NUM_NL80211_BANDS; band++) { + unsigned int i; + + cfg_band = wiphy->bands[band]; + if (!cfg_band) + continue; + + cmd->num_channels += cfg_band->n_channels; + + for (i = 0; i < cfg_band->n_channels; ++i) { + qtnf_cmd_channel_tlv_add(cmd_skb, + &cfg_band->channels[i]); + } + } + qtnf_bus_lock(bus); ret = qtnf_cmd_send(bus, cmd_skb); - if (ret) - goto out; - -out: qtnf_bus_unlock(bus); return ret; @@ -2592,7 +2589,7 @@ int qtnf_cmd_set_mac_acl(const struct qtnf_vif *vif, struct qtnf_bus *bus = vif->mac->bus; struct sk_buff *cmd_skb; struct qlink_tlv_hdr *tlv; - size_t acl_size = qtnf_cmd_acl_data_size(params); + size_t acl_size = struct_size(params, mac_addrs, params->n_acl_entries); int ret; cmd_skb = qtnf_cmd_alloc_new_cmdskb(vif->mac->macid, vif->vifid, diff --git a/drivers/net/wireless/quantenna/qtnfmac/commands.h b/drivers/net/wireless/quantenna/qtnfmac/commands.h index 64f0b9dc8a1440ae90e5e3998336bfd546901b85..88d7a3cd90d21bac487b06c5d8078e5032f1803a 100644 --- a/drivers/net/wireless/quantenna/qtnfmac/commands.h +++ b/drivers/net/wireless/quantenna/qtnfmac/commands.h @@ -27,8 +27,8 @@ int qtnf_cmd_send_start_ap(struct qtnf_vif *vif, const struct cfg80211_ap_settings *s); int qtnf_cmd_send_stop_ap(struct qtnf_vif *vif); int qtnf_cmd_send_register_mgmt(struct qtnf_vif *vif, u16 frame_type, bool reg); -int qtnf_cmd_send_mgmt_frame(struct qtnf_vif *vif, u32 cookie, u16 flags, - u16 freq, const u8 *buf, size_t len); +int qtnf_cmd_send_frame(struct qtnf_vif *vif, u32 cookie, u16 flags, + u16 freq, const u8 *buf, size_t len); int qtnf_cmd_send_mgmt_set_appie(struct qtnf_vif *vif, u8 frame_type, const u8 *buf, size_t len); int qtnf_cmd_get_sta_info(struct qtnf_vif *vif, const u8 *sta_mac, @@ -57,7 +57,8 @@ int qtnf_cmd_send_disconnect(struct qtnf_vif *vif, u16 reason_code); int qtnf_cmd_send_updown_intf(struct qtnf_vif *vif, bool up); -int qtnf_cmd_reg_notify(struct qtnf_bus *bus, struct regulatory_request *req); +int qtnf_cmd_reg_notify(struct qtnf_wmac *mac, struct regulatory_request *req, + bool slave_radar); int qtnf_cmd_get_chan_stats(struct qtnf_wmac *mac, u16 channel, struct qtnf_chan_stats *stats); int qtnf_cmd_send_chan_switch(struct qtnf_vif *vif, diff --git a/drivers/net/wireless/quantenna/qtnfmac/core.c b/drivers/net/wireless/quantenna/qtnfmac/core.c index ee1b75fda1dd2c1b7a0125a46838d8cdf862c14e..8d699cc03d26e895daa83cd9bd268f4bb652b9aa 100644 --- a/drivers/net/wireless/quantenna/qtnfmac/core.c +++ b/drivers/net/wireless/quantenna/qtnfmac/core.c @@ -16,6 +16,12 @@ #define QTNF_DMP_MAX_LEN 48 #define QTNF_PRIMARY_VIF_IDX 0 +static bool slave_radar = true; +module_param(slave_radar, bool, 0644); +MODULE_PARM_DESC(slave_radar, "set 0 to disable radar detection in slave mode"); + +static struct dentry *qtnf_debugfs_dir; + struct qtnf_frame_meta_info { u8 magic_s; u8 ifidx; @@ -368,6 +374,23 @@ static void qtnf_mac_scan_timeout(struct work_struct *work) qtnf_mac_scan_finish(mac, true); } +static void qtnf_vif_send_data_high_pri(struct work_struct *work) +{ + struct qtnf_vif *vif = + container_of(work, struct qtnf_vif, high_pri_tx_work); + struct sk_buff *skb; + + if (!vif->netdev || + vif->wdev.iftype == NL80211_IFTYPE_UNSPECIFIED) + return; + + while ((skb = skb_dequeue(&vif->high_pri_tx_queue))) { + qtnf_cmd_send_frame(vif, 0, QLINK_FRAME_TX_FLAG_8023, + 0, skb->data, skb->len); + dev_kfree_skb_any(skb); + } +} + static struct qtnf_wmac *qtnf_core_mac_alloc(struct qtnf_bus *bus, unsigned int macid) { @@ -395,7 +418,8 @@ static struct qtnf_wmac *qtnf_core_mac_alloc(struct qtnf_bus *bus, vif->mac = mac; vif->vifid = i; qtnf_sta_list_init(&vif->sta_list); - + INIT_WORK(&vif->high_pri_tx_work, qtnf_vif_send_data_high_pri); + skb_queue_head_init(&vif->high_pri_tx_queue); vif->stats64 = netdev_alloc_pcpu_stats(struct pcpu_sw_netstats); if (!vif->stats64) pr_warn("VIF%u.%u: per cpu stats allocation failed\n", @@ -408,6 +432,11 @@ static struct qtnf_wmac *qtnf_core_mac_alloc(struct qtnf_bus *bus, return mac; } +bool qtnf_mac_slave_radar_get(struct wiphy *wiphy) +{ + return slave_radar; +} + static const struct ethtool_ops qtnf_ethtool_ops = { .get_drvinfo = cfg80211_get_drvinfo, }; @@ -499,6 +528,8 @@ static void qtnf_core_mac_detach(struct qtnf_bus *bus, unsigned int macid) qtnf_mac_iface_comb_free(mac); qtnf_mac_ext_caps_free(mac); kfree(mac->macinfo.wowlan); + kfree(mac->rd); + mac->rd = NULL; wiphy_free(wiphy); bus->mac[macid] = NULL; } @@ -587,8 +618,6 @@ int qtnf_core_attach(struct qtnf_bus *bus) int ret; qtnf_trans_init(bus); - - bus->fw_state = QTNF_FW_STATE_BOOT_DONE; qtnf_bus_data_rx_start(bus); bus->workqueue = alloc_ordered_workqueue("QTNF_BUS", 0); @@ -598,6 +627,13 @@ int qtnf_core_attach(struct qtnf_bus *bus) goto error; } + bus->hprio_workqueue = alloc_workqueue("QTNF_HPRI", WQ_HIGHPRI, 0); + if (!bus->hprio_workqueue) { + pr_err("failed to alloc high prio workqueue\n"); + ret = -ENOMEM; + goto error; + } + INIT_WORK(&bus->event_work, qtnf_event_work_handler); ret = qtnf_cmd_send_init_fw(bus); @@ -607,7 +643,6 @@ int qtnf_core_attach(struct qtnf_bus *bus) } bus->fw_state = QTNF_FW_STATE_ACTIVE; - ret = qtnf_cmd_get_hw_info(bus); if (ret) { pr_err("failed to get HW info: %d\n", ret); @@ -637,11 +672,11 @@ int qtnf_core_attach(struct qtnf_bus *bus) } } + bus->fw_state = QTNF_FW_STATE_RUNNING; return 0; error: qtnf_core_detach(bus); - return ret; } EXPORT_SYMBOL_GPL(qtnf_core_attach); @@ -655,7 +690,7 @@ void qtnf_core_detach(struct qtnf_bus *bus) for (macid = 0; macid < QTNF_MAX_MAC; macid++) qtnf_core_mac_detach(bus, macid); - if (bus->fw_state == QTNF_FW_STATE_ACTIVE) + if (qtnf_fw_is_up(bus)) qtnf_cmd_send_deinit_fw(bus); bus->fw_state = QTNF_FW_STATE_DETACHED; @@ -663,10 +698,14 @@ void qtnf_core_detach(struct qtnf_bus *bus) if (bus->workqueue) { flush_workqueue(bus->workqueue); destroy_workqueue(bus->workqueue); + bus->workqueue = NULL; } - kfree(bus->hw_info.rd); - bus->hw_info.rd = NULL; + if (bus->hprio_workqueue) { + flush_workqueue(bus->hprio_workqueue); + destroy_workqueue(bus->hprio_workqueue); + bus->hprio_workqueue = NULL; + } qtnf_trans_free(bus); } @@ -684,6 +723,9 @@ struct net_device *qtnf_classify_skb(struct qtnf_bus *bus, struct sk_buff *skb) struct qtnf_wmac *mac; struct qtnf_vif *vif; + if (unlikely(bus->fw_state != QTNF_FW_STATE_RUNNING)) + return NULL; + meta = (struct qtnf_frame_meta_info *) (skb_tail_pointer(skb) - sizeof(*meta)); @@ -799,6 +841,39 @@ void qtnf_update_tx_stats(struct net_device *ndev, const struct sk_buff *skb) } EXPORT_SYMBOL_GPL(qtnf_update_tx_stats); +void qtnf_packet_send_hi_pri(struct sk_buff *skb) +{ + struct qtnf_vif *vif = qtnf_netdev_get_priv(skb->dev); + + skb_queue_tail(&vif->high_pri_tx_queue, skb); + queue_work(vif->mac->bus->hprio_workqueue, &vif->high_pri_tx_work); +} +EXPORT_SYMBOL_GPL(qtnf_packet_send_hi_pri); + +struct dentry *qtnf_get_debugfs_dir(void) +{ + return qtnf_debugfs_dir; +} +EXPORT_SYMBOL_GPL(qtnf_get_debugfs_dir); + +static int __init qtnf_core_register(void) +{ + qtnf_debugfs_dir = debugfs_create_dir(KBUILD_MODNAME, NULL); + + if (IS_ERR(qtnf_debugfs_dir)) + qtnf_debugfs_dir = NULL; + + return 0; +} + +static void __exit qtnf_core_exit(void) +{ + debugfs_remove(qtnf_debugfs_dir); +} + +module_init(qtnf_core_register); +module_exit(qtnf_core_exit); + MODULE_AUTHOR("Quantenna Communications"); MODULE_DESCRIPTION("Quantenna 802.11 wireless LAN FullMAC driver."); MODULE_LICENSE("GPL"); diff --git a/drivers/net/wireless/quantenna/qtnfmac/core.h b/drivers/net/wireless/quantenna/qtnfmac/core.h index a31cff46e9640faa706813d7a407577d9d7b2730..322858df600c73f43d65167139631de33b9e7fb0 100644 --- a/drivers/net/wireless/quantenna/qtnfmac/core.h +++ b/drivers/net/wireless/quantenna/qtnfmac/core.h @@ -63,6 +63,8 @@ struct qtnf_vif { struct qtnf_wmac *mac; struct work_struct reset_work; + struct work_struct high_pri_tx_work; + struct sk_buff_head high_pri_tx_queue; struct qtnf_sta_list sta_list; unsigned long cons_tx_timeout_cnt; int generation; @@ -112,6 +114,7 @@ struct qtnf_wmac { struct cfg80211_scan_request *scan_req; struct mutex mac_lock; /* lock during wmac speicific ops */ struct delayed_work scan_timeout; + struct ieee80211_regdomain *rd; }; struct qtnf_hw_info { @@ -120,7 +123,6 @@ struct qtnf_hw_info { u8 mac_bitmap; u32 fw_ver; u32 hw_capab; - struct ieee80211_regdomain *rd; u8 total_tx_chain; u8 total_rx_chain; char fw_version[ETHTOOL_FWVERS_LEN]; @@ -132,6 +134,7 @@ struct qtnf_vif *qtnf_mac_get_free_vif(struct qtnf_wmac *mac); struct qtnf_vif *qtnf_mac_get_base_vif(struct qtnf_wmac *mac); void qtnf_mac_iface_comb_free(struct qtnf_wmac *mac); void qtnf_mac_ext_caps_free(struct qtnf_wmac *mac); +bool qtnf_mac_slave_radar_get(struct wiphy *wiphy); struct wiphy *qtnf_wiphy_allocate(struct qtnf_bus *bus); int qtnf_core_net_attach(struct qtnf_wmac *mac, struct qtnf_vif *priv, const char *name, unsigned char name_assign_type); @@ -149,6 +152,8 @@ void qtnf_virtual_intf_cleanup(struct net_device *ndev); void qtnf_netdev_updown(struct net_device *ndev, bool up); void qtnf_scan_done(struct qtnf_wmac *mac, bool aborted); +void qtnf_packet_send_hi_pri(struct sk_buff *skb); +struct dentry *qtnf_get_debugfs_dir(void); static inline struct qtnf_vif *qtnf_netdev_get_priv(struct net_device *dev) { diff --git a/drivers/net/wireless/quantenna/qtnfmac/debug.c b/drivers/net/wireless/quantenna/qtnfmac/debug.c index 598ece753a4b8f70fa064b052839ecfcdb131a5c..2d3574c1f10efe9e3c212ef6c72e1f8e0064a028 100644 --- a/drivers/net/wireless/quantenna/qtnfmac/debug.c +++ b/drivers/net/wireless/quantenna/qtnfmac/debug.c @@ -5,7 +5,9 @@ void qtnf_debugfs_init(struct qtnf_bus *bus, const char *name) { - bus->dbg_dir = debugfs_create_dir(name, NULL); + struct dentry *parent = qtnf_get_debugfs_dir(); + + bus->dbg_dir = debugfs_create_dir(name, parent); } void qtnf_debugfs_remove(struct qtnf_bus *bus) diff --git a/drivers/net/wireless/quantenna/qtnfmac/event.c b/drivers/net/wireless/quantenna/qtnfmac/event.c index 6c1b886339acf7f7fc85edf366dbcf9bf41c243e..b57c8c18a8d01b0d2939dfa427002aea1335a17e 100644 --- a/drivers/net/wireless/quantenna/qtnfmac/event.c +++ b/drivers/net/wireless/quantenna/qtnfmac/event.c @@ -493,14 +493,20 @@ qtnf_event_handle_freq_change(struct qtnf_wmac *mac, for (i = 0; i < QTNF_MAX_INTF; i++) { vif = &mac->iflist[i]; + if (vif->wdev.iftype == NL80211_IFTYPE_UNSPECIFIED) continue; - if (vif->netdev) { - mutex_lock(&vif->wdev.mtx); - cfg80211_ch_switch_notify(vif->netdev, &chandef); - mutex_unlock(&vif->wdev.mtx); - } + if (vif->wdev.iftype == NL80211_IFTYPE_STATION && + !vif->wdev.current_bss) + continue; + + if (!vif->netdev) + continue; + + mutex_lock(&vif->wdev.mtx); + cfg80211_ch_switch_notify(vif->netdev, &chandef); + mutex_unlock(&vif->wdev.mtx); } return 0; diff --git a/drivers/net/wireless/quantenna/qtnfmac/pcie/pcie.c b/drivers/net/wireless/quantenna/qtnfmac/pcie/pcie.c index c3a32effa6f09a84e118c8e12995b1c86300fb60..e4e9344b6982c8cf40235702dce16f019325c468 100644 --- a/drivers/net/wireless/quantenna/qtnfmac/pcie/pcie.c +++ b/drivers/net/wireless/quantenna/qtnfmac/pcie/pcie.c @@ -56,7 +56,7 @@ int qtnf_pcie_control_tx(struct qtnf_bus *bus, struct sk_buff *skb) if (ret == -ETIMEDOUT) { pr_err("EP firmware is dead\n"); - bus->fw_state = QTNF_FW_STATE_EP_DEAD; + bus->fw_state = QTNF_FW_STATE_DEAD; } return ret; @@ -128,32 +128,22 @@ static int qtnf_dbg_shm_stats(struct seq_file *s, void *data) return 0; } -void qtnf_pcie_fw_boot_done(struct qtnf_bus *bus, bool boot_success) +int qtnf_pcie_fw_boot_done(struct qtnf_bus *bus) { - struct qtnf_pcie_bus_priv *priv = get_bus_priv(bus); - struct pci_dev *pdev = priv->pdev; int ret; - if (boot_success) { - bus->fw_state = QTNF_FW_STATE_FW_DNLD_DONE; - - ret = qtnf_core_attach(bus); - if (ret) { - pr_err("failed to attach core\n"); - boot_success = false; - } - } - - if (boot_success) { + bus->fw_state = QTNF_FW_STATE_BOOT_DONE; + ret = qtnf_core_attach(bus); + if (ret) { + pr_err("failed to attach core\n"); + } else { qtnf_debugfs_init(bus, DRV_NAME); qtnf_debugfs_add_entry(bus, "mps", qtnf_dbg_mps_show); qtnf_debugfs_add_entry(bus, "msi_enabled", qtnf_dbg_msi_show); qtnf_debugfs_add_entry(bus, "shm_stats", qtnf_dbg_shm_stats); - } else { - bus->fw_state = QTNF_FW_STATE_DETACHED; } - put_device(&pdev->dev); + return ret; } static void qtnf_tune_pcie_mps(struct pci_dev *pdev) @@ -344,7 +334,7 @@ static int qtnf_pcie_probe(struct pci_dev *pdev, const struct pci_device_id *id) pcie_priv = get_bus_priv(bus); pci_set_drvdata(pdev, bus); bus->dev = &pdev->dev; - bus->fw_state = QTNF_FW_STATE_RESET; + bus->fw_state = QTNF_FW_STATE_DETACHED; pcie_priv->pdev = pdev; pcie_priv->tx_stopped = 0; pcie_priv->rx_bd_num = rx_bd_size_param; @@ -364,6 +354,7 @@ static int qtnf_pcie_probe(struct pci_dev *pdev, const struct pci_device_id *id) pcie_priv->pcie_irq_count = 0; pcie_priv->tx_reclaim_done = 0; pcie_priv->tx_reclaim_req = 0; + pcie_priv->tx_eapol = 0; pcie_priv->workqueue = create_singlethread_workqueue("QTNF_PCIE"); if (!pcie_priv->workqueue) { @@ -419,8 +410,7 @@ static void qtnf_pcie_remove(struct pci_dev *dev) cancel_work_sync(&bus->fw_work); - if (bus->fw_state == QTNF_FW_STATE_ACTIVE || - bus->fw_state == QTNF_FW_STATE_EP_DEAD) + if (qtnf_fw_is_attached(bus)) qtnf_core_detach(bus); netif_napi_del(&bus->mux_napi); diff --git a/drivers/net/wireless/quantenna/qtnfmac/pcie/pcie_priv.h b/drivers/net/wireless/quantenna/qtnfmac/pcie/pcie_priv.h index bbc074e1f34d17ea1895e20b980db23ff0e1317b..5e8b9cb68419831602719f56d917ae24645011e4 100644 --- a/drivers/net/wireless/quantenna/qtnfmac/pcie/pcie_priv.h +++ b/drivers/net/wireless/quantenna/qtnfmac/pcie/pcie_priv.h @@ -62,6 +62,7 @@ struct qtnf_pcie_bus_priv { u32 tx_done_count; u32 tx_reclaim_done; u32 tx_reclaim_req; + u32 tx_eapol; u8 msi_enabled; u8 tx_stopped; @@ -70,7 +71,7 @@ struct qtnf_pcie_bus_priv { int qtnf_pcie_control_tx(struct qtnf_bus *bus, struct sk_buff *skb); int qtnf_pcie_alloc_skb_array(struct qtnf_pcie_bus_priv *priv); -void qtnf_pcie_fw_boot_done(struct qtnf_bus *bus, bool boot_success); +int qtnf_pcie_fw_boot_done(struct qtnf_bus *bus); void qtnf_pcie_init_shm_ipc(struct qtnf_pcie_bus_priv *priv, struct qtnf_shm_ipc_region __iomem *ipc_tx_reg, struct qtnf_shm_ipc_region __iomem *ipc_rx_reg, diff --git a/drivers/net/wireless/quantenna/qtnfmac/pcie/pearl_pcie.c b/drivers/net/wireless/quantenna/qtnfmac/pcie/pearl_pcie.c index 1f5facbb89057741837701afb9f75bd90067fa38..3aa3714d4dfd25a5c2cbd11fe88cb81b6c41f986 100644 --- a/drivers/net/wireless/quantenna/qtnfmac/pcie/pearl_pcie.c +++ b/drivers/net/wireless/quantenna/qtnfmac/pcie/pearl_pcie.c @@ -980,12 +980,11 @@ static void qtnf_pearl_fw_work_handler(struct work_struct *work) { struct qtnf_bus *bus = container_of(work, struct qtnf_bus, fw_work); struct qtnf_pcie_pearl_state *ps = (void *)get_bus_priv(bus); + u32 state = QTN_RC_FW_LOADRDY | QTN_RC_FW_QLINK; + const char *fwname = QTN_PCI_PEARL_FW_NAME; struct pci_dev *pdev = ps->base.pdev; const struct firmware *fw; int ret; - u32 state = QTN_RC_FW_LOADRDY | QTN_RC_FW_QLINK; - const char *fwname = QTN_PCI_PEARL_FW_NAME; - bool fw_boot_success = false; if (ps->base.flashboot) { state |= QTN_RC_FW_FLASHBOOT; @@ -1031,23 +1030,23 @@ static void qtnf_pearl_fw_work_handler(struct work_struct *work) goto fw_load_exit; } - pr_info("firmware is up and running\n"); - if (qtnf_poll_state(&ps->bda->bda_ep_state, QTN_EP_FW_QLINK_DONE, QTN_FW_QLINK_TIMEOUT_MS)) { pr_err("firmware runtime failure\n"); goto fw_load_exit; } - fw_boot_success = true; + pr_info("firmware is up and running\n"); -fw_load_exit: - qtnf_pcie_fw_boot_done(bus, fw_boot_success); + ret = qtnf_pcie_fw_boot_done(bus); + if (ret) + goto fw_load_exit; - if (fw_boot_success) { - qtnf_debugfs_add_entry(bus, "hdp_stats", qtnf_dbg_hdp_stats); - qtnf_debugfs_add_entry(bus, "irq_stats", qtnf_dbg_irq_stats); - } + qtnf_debugfs_add_entry(bus, "hdp_stats", qtnf_dbg_hdp_stats); + qtnf_debugfs_add_entry(bus, "irq_stats", qtnf_dbg_irq_stats); + +fw_load_exit: + put_device(&pdev->dev); } static void qtnf_pearl_reclaim_tasklet_fn(unsigned long data) diff --git a/drivers/net/wireless/quantenna/qtnfmac/pcie/topaz_pcie.c b/drivers/net/wireless/quantenna/qtnfmac/pcie/topaz_pcie.c index cbcda57105f353f3a819f04147ffc999409ea8b7..9a4380ed7f1b214383b92dd3de405ef2e39aa930 100644 --- a/drivers/net/wireless/quantenna/qtnfmac/pcie/topaz_pcie.c +++ b/drivers/net/wireless/quantenna/qtnfmac/pcie/topaz_pcie.c @@ -498,6 +498,13 @@ static int qtnf_pcie_data_tx(struct qtnf_bus *bus, struct sk_buff *skb) int len; int i; + if (unlikely(skb->protocol == htons(ETH_P_PAE))) { + qtnf_packet_send_hi_pri(skb); + qtnf_update_tx_stats(skb->dev, skb); + priv->tx_eapol++; + return NETDEV_TX_OK; + } + spin_lock_irqsave(&priv->tx_lock, flags); if (!qtnf_tx_queue_ready(ts)) { @@ -761,6 +768,7 @@ static int qtnf_dbg_pkt_stats(struct seq_file *s, void *data) seq_printf(s, "tx_done_count(%u)\n", priv->tx_done_count); seq_printf(s, "tx_reclaim_done(%u)\n", priv->tx_reclaim_done); seq_printf(s, "tx_reclaim_req(%u)\n", priv->tx_reclaim_req); + seq_printf(s, "tx_eapol(%u)\n", priv->tx_eapol); seq_printf(s, "tx_bd_r_index(%u)\n", priv->tx_bd_r_index); seq_printf(s, "tx_done_index(%u)\n", tx_done_index); @@ -1023,8 +1031,9 @@ static void qtnf_topaz_fw_work_handler(struct work_struct *work) { struct qtnf_bus *bus = container_of(work, struct qtnf_bus, fw_work); struct qtnf_pcie_topaz_state *ts = (void *)get_bus_priv(bus); - int ret; int bootloader_needed = readl(&ts->bda->bda_flags) & QTN_BDA_XMIT_UBOOT; + struct pci_dev *pdev = ts->base.pdev; + int ret; qtnf_set_state(&ts->bda->bda_bootstate, QTN_BDA_FW_TARGET_BOOT); @@ -1073,19 +1082,23 @@ static void qtnf_topaz_fw_work_handler(struct work_struct *work) } } + ret = qtnf_post_init_ep(ts); + if (ret) { + pr_err("FW runtime failure\n"); + goto fw_load_exit; + } + pr_info("firmware is up and running\n"); - ret = qtnf_post_init_ep(ts); + ret = qtnf_pcie_fw_boot_done(bus); if (ret) - pr_err("FW runtime failure\n"); + goto fw_load_exit; -fw_load_exit: - qtnf_pcie_fw_boot_done(bus, ret ? false : true); + qtnf_debugfs_add_entry(bus, "pkt_stats", qtnf_dbg_pkt_stats); + qtnf_debugfs_add_entry(bus, "irq_stats", qtnf_dbg_irq_stats); - if (ret == 0) { - qtnf_debugfs_add_entry(bus, "pkt_stats", qtnf_dbg_pkt_stats); - qtnf_debugfs_add_entry(bus, "irq_stats", qtnf_dbg_irq_stats); - } +fw_load_exit: + put_device(&pdev->dev); } static void qtnf_reclaim_tasklet_fn(unsigned long data) diff --git a/drivers/net/wireless/quantenna/qtnfmac/qlink.h b/drivers/net/wireless/quantenna/qtnfmac/qlink.h index 7798edcf7980250e75ff3c28d582b95e580517a1..8a3c6344fa8ed5ab20124bb0d1667771e7457387 100644 --- a/drivers/net/wireless/quantenna/qtnfmac/qlink.h +++ b/drivers/net/wireless/quantenna/qtnfmac/qlink.h @@ -6,7 +6,7 @@ #include -#define QLINK_PROTO_VER 13 +#define QLINK_PROTO_VER 15 #define QLINK_MACID_RSVD 0xFF #define QLINK_VIFID_RSVD 0xFF @@ -206,6 +206,8 @@ struct qlink_sta_info_state { * execution status (one of &enum qlink_cmd_result). Reply message * may also contain data payload specific to the command type. * + * @QLINK_CMD_SEND_FRAME: send specified frame over the air; firmware will + * encapsulate 802.3 packet into 802.11 frame automatically. * @QLINK_CMD_BAND_INFO_GET: for the specified MAC and specified band, get * the band's description including number of operational channels and * info on each channel, HT/VHT capabilities, supported rates etc. @@ -220,7 +222,7 @@ enum qlink_cmd_type { QLINK_CMD_FW_INIT = 0x0001, QLINK_CMD_FW_DEINIT = 0x0002, QLINK_CMD_REGISTER_MGMT = 0x0003, - QLINK_CMD_SEND_MGMT_FRAME = 0x0004, + QLINK_CMD_SEND_FRAME = 0x0004, QLINK_CMD_MGMT_SET_APPIE = 0x0005, QLINK_CMD_PHY_PARAMS_GET = 0x0011, QLINK_CMD_PHY_PARAMS_SET = 0x0012, @@ -321,22 +323,26 @@ struct qlink_cmd_mgmt_frame_register { u8 do_register; } __packed; -enum qlink_mgmt_frame_tx_flags { - QLINK_MGMT_FRAME_TX_FLAG_NONE = 0, - QLINK_MGMT_FRAME_TX_FLAG_OFFCHAN = BIT(0), - QLINK_MGMT_FRAME_TX_FLAG_NO_CCK = BIT(1), - QLINK_MGMT_FRAME_TX_FLAG_ACK_NOWAIT = BIT(2), +/** + * @QLINK_FRAME_TX_FLAG_8023: frame has a 802.3 header; if not set, frame + * is a 802.11 encapsulated. + */ +enum qlink_frame_tx_flags { + QLINK_FRAME_TX_FLAG_OFFCHAN = BIT(0), + QLINK_FRAME_TX_FLAG_NO_CCK = BIT(1), + QLINK_FRAME_TX_FLAG_ACK_NOWAIT = BIT(2), + QLINK_FRAME_TX_FLAG_8023 = BIT(3), }; /** - * struct qlink_cmd_mgmt_frame_tx - data for QLINK_CMD_SEND_MGMT_FRAME command + * struct qlink_cmd_frame_tx - data for QLINK_CMD_SEND_FRAME command * * @cookie: opaque request identifier. * @freq: Frequency to use for frame transmission. - * @flags: Transmission flags, one of &enum qlink_mgmt_frame_tx_flags. + * @flags: Transmission flags, one of &enum qlink_frame_tx_flags. * @frame_data: frame to transmit. */ -struct qlink_cmd_mgmt_frame_tx { +struct qlink_cmd_frame_tx { struct qlink_cmd chdr; __le32 cookie; __le16 freq; @@ -580,12 +586,22 @@ enum qlink_user_reg_hint_type { * @initiator: which entity sent the request, one of &enum qlink_reg_initiator. * @user_reg_hint_type: type of hint for QLINK_REGDOM_SET_BY_USER request, one * of &enum qlink_user_reg_hint_type. + * @num_channels: number of &struct qlink_tlv_channel in a variable portion of a + * payload. + * @slave_radar: whether slave device should enable radar detection. + * @dfs_region: one of &enum qlink_dfs_regions. + * @info: variable portion of regulatory notifier callback. */ struct qlink_cmd_reg_notify { struct qlink_cmd chdr; u8 alpha2[2]; u8 initiator; u8 user_reg_hint_type; + u8 num_channels; + u8 dfs_region; + u8 slave_radar; + u8 rsvd[1]; + u8 info[0]; } __packed; /** @@ -764,6 +780,18 @@ struct qlink_resp { u8 vifid; } __packed; +/** + * enum qlink_dfs_regions - regulatory DFS regions + * + * Corresponds to &enum nl80211_dfs_regions. + */ +enum qlink_dfs_regions { + QLINK_DFS_UNSET = 0, + QLINK_DFS_FCC = 1, + QLINK_DFS_ETSI = 2, + QLINK_DFS_JP = 3, +}; + /** * struct qlink_resp_get_mac_info - response for QLINK_CMD_MAC_INFO command * @@ -779,6 +807,10 @@ struct qlink_resp { * @bands_cap: wireless bands WMAC can operate in, bitmap of &enum qlink_band. * @max_ap_assoc_sta: Maximum number of associations supported by WMAC. * @radar_detect_widths: bitmask of channels BW for which WMAC can detect radar. + * @alpha2: country code ID firmware is configured to. + * @n_reg_rules: number of regulatory rules TLVs in variable portion of the + * message. + * @dfs_region: regulatory DFS region, one of &enum qlink_dfs_regions. * @var_info: variable-length WMAC info data. */ struct qlink_resp_get_mac_info { @@ -792,22 +824,13 @@ struct qlink_resp_get_mac_info { __le16 radar_detect_widths; __le32 max_acl_mac_addrs; u8 bands_cap; + u8 alpha2[2]; + u8 n_reg_rules; + u8 dfs_region; u8 rsvd[1]; u8 var_info[0]; } __packed; -/** - * enum qlink_dfs_regions - regulatory DFS regions - * - * Corresponds to &enum nl80211_dfs_regions. - */ -enum qlink_dfs_regions { - QLINK_DFS_UNSET = 0, - QLINK_DFS_FCC = 1, - QLINK_DFS_ETSI = 2, - QLINK_DFS_JP = 3, -}; - /** * struct qlink_resp_get_hw_info - response for QLINK_CMD_GET_HW_INFO command * @@ -820,11 +843,7 @@ enum qlink_dfs_regions { * @mac_bitmap: Bitmap of MAC IDs that are active and can be used in firmware. * @total_tx_chains: total number of transmit chains used by device. * @total_rx_chains: total number of receive chains. - * @alpha2: country code ID firmware is configured to. - * @n_reg_rules: number of regulatory rules TLVs in variable portion of the - * message. - * @dfs_region: regulatory DFS region, one of @enum qlink_dfs_region. - * @info: variable-length HW info, can contain QTN_TLV_ID_REG_RULE. + * @info: variable-length HW info. */ struct qlink_resp_get_hw_info { struct qlink_resp rhdr; @@ -838,9 +857,6 @@ struct qlink_resp_get_hw_info { u8 mac_bitmap; u8 total_tx_chain; u8 total_rx_chain; - u8 alpha2[2]; - u8 n_reg_rules; - u8 dfs_region; u8 info[0]; } __packed; @@ -1148,6 +1164,13 @@ struct qlink_event_external_auth { * carried by QTN_TLV_ID_STA_STATS_MAP. * @QTN_TLV_ID_MAX_SCAN_SSIDS: maximum number of SSIDs the device can scan * for in any given scan. + * @QTN_TLV_ID_SCAN_DWELL_ACTIVE: time spent on a single channel for an active + * scan. + * @QTN_TLV_ID_SCAN_DWELL_PASSIVE: time spent on a single channel for a passive + * scan. + * @QTN_TLV_ID_SCAN_SAMPLE_DURATION: total duration of sampling a single channel + * during a scan including off-channel dwell time and operating channel + * time. */ enum qlink_tlv_id { QTN_TLV_ID_FRAG_THRESH = 0x0201, @@ -1180,7 +1203,9 @@ enum qlink_tlv_id { QTN_TLV_ID_WOWLAN_CAPAB = 0x0410, QTN_TLV_ID_WOWLAN_PATTERN = 0x0411, QTN_TLV_ID_SCAN_FLUSH = 0x0412, - QTN_TLV_ID_SCAN_DWELL = 0x0413, + QTN_TLV_ID_SCAN_DWELL_ACTIVE = 0x0413, + QTN_TLV_ID_SCAN_DWELL_PASSIVE = 0x0416, + QTN_TLV_ID_SCAN_SAMPLE_DURATION = 0x0417, }; struct qlink_tlv_hdr { diff --git a/drivers/net/wireless/quantenna/qtnfmac/qlink_util.c b/drivers/net/wireless/quantenna/qtnfmac/qlink_util.c index 72bfd17cb687cfead142a814b78b8027d5839fca..1a972bce7b8bdc5c39ddd1ead1fba3b071aedfcd 100644 --- a/drivers/net/wireless/quantenna/qtnfmac/qlink_util.c +++ b/drivers/net/wireless/quantenna/qtnfmac/qlink_util.c @@ -182,3 +182,120 @@ void qlink_acl_data_cfg2q(const struct cfg80211_acl_data *acl, memcpy(qacl->mac_addrs, acl->mac_addrs, acl->n_acl_entries * sizeof(*qacl->mac_addrs)); } + +enum qlink_band qlink_utils_band_cfg2q(enum nl80211_band band) +{ + switch (band) { + case NL80211_BAND_2GHZ: + return QLINK_BAND_2GHZ; + case NL80211_BAND_5GHZ: + return QLINK_BAND_5GHZ; + case NL80211_BAND_60GHZ: + return QLINK_BAND_60GHZ; + default: + return -EINVAL; + } +} + +enum qlink_dfs_state qlink_utils_dfs_state_cfg2q(enum nl80211_dfs_state state) +{ + switch (state) { + case NL80211_DFS_USABLE: + return QLINK_DFS_USABLE; + case NL80211_DFS_AVAILABLE: + return QLINK_DFS_AVAILABLE; + case NL80211_DFS_UNAVAILABLE: + default: + return QLINK_DFS_UNAVAILABLE; + } +} + +u32 qlink_utils_chflags_cfg2q(u32 cfgflags) +{ + u32 flags = 0; + + if (cfgflags & IEEE80211_CHAN_DISABLED) + flags |= QLINK_CHAN_DISABLED; + + if (cfgflags & IEEE80211_CHAN_NO_IR) + flags |= QLINK_CHAN_NO_IR; + + if (cfgflags & IEEE80211_CHAN_RADAR) + flags |= QLINK_CHAN_RADAR; + + if (cfgflags & IEEE80211_CHAN_NO_HT40PLUS) + flags |= QLINK_CHAN_NO_HT40PLUS; + + if (cfgflags & IEEE80211_CHAN_NO_HT40MINUS) + flags |= QLINK_CHAN_NO_HT40MINUS; + + if (cfgflags & IEEE80211_CHAN_NO_80MHZ) + flags |= QLINK_CHAN_NO_80MHZ; + + if (cfgflags & IEEE80211_CHAN_NO_160MHZ) + flags |= QLINK_CHAN_NO_160MHZ; + + return flags; +} + +static u32 qtnf_reg_rule_flags_parse(u32 qflags) +{ + u32 flags = 0; + + if (qflags & QLINK_RRF_NO_OFDM) + flags |= NL80211_RRF_NO_OFDM; + + if (qflags & QLINK_RRF_NO_CCK) + flags |= NL80211_RRF_NO_CCK; + + if (qflags & QLINK_RRF_NO_INDOOR) + flags |= NL80211_RRF_NO_INDOOR; + + if (qflags & QLINK_RRF_NO_OUTDOOR) + flags |= NL80211_RRF_NO_OUTDOOR; + + if (qflags & QLINK_RRF_DFS) + flags |= NL80211_RRF_DFS; + + if (qflags & QLINK_RRF_PTP_ONLY) + flags |= NL80211_RRF_PTP_ONLY; + + if (qflags & QLINK_RRF_PTMP_ONLY) + flags |= NL80211_RRF_PTMP_ONLY; + + if (qflags & QLINK_RRF_NO_IR) + flags |= NL80211_RRF_NO_IR; + + if (qflags & QLINK_RRF_AUTO_BW) + flags |= NL80211_RRF_AUTO_BW; + + if (qflags & QLINK_RRF_IR_CONCURRENT) + flags |= NL80211_RRF_IR_CONCURRENT; + + if (qflags & QLINK_RRF_NO_HT40MINUS) + flags |= NL80211_RRF_NO_HT40MINUS; + + if (qflags & QLINK_RRF_NO_HT40PLUS) + flags |= NL80211_RRF_NO_HT40PLUS; + + if (qflags & QLINK_RRF_NO_80MHZ) + flags |= NL80211_RRF_NO_80MHZ; + + if (qflags & QLINK_RRF_NO_160MHZ) + flags |= NL80211_RRF_NO_160MHZ; + + return flags; +} + +void qlink_utils_regrule_q2nl(struct ieee80211_reg_rule *rule, + const struct qlink_tlv_reg_rule *tlv) +{ + rule->freq_range.start_freq_khz = le32_to_cpu(tlv->start_freq_khz); + rule->freq_range.end_freq_khz = le32_to_cpu(tlv->end_freq_khz); + rule->freq_range.max_bandwidth_khz = + le32_to_cpu(tlv->max_bandwidth_khz); + rule->power_rule.max_antenna_gain = le32_to_cpu(tlv->max_antenna_gain); + rule->power_rule.max_eirp = le32_to_cpu(tlv->max_eirp); + rule->dfs_cac_ms = le32_to_cpu(tlv->dfs_cac_ms); + rule->flags = qtnf_reg_rule_flags_parse(le32_to_cpu(tlv->flags)); +} diff --git a/drivers/net/wireless/quantenna/qtnfmac/qlink_util.h b/drivers/net/wireless/quantenna/qtnfmac/qlink_util.h index 781ea7fe79f211d4beb224021f38612be7ced232..f873beed2ae75b34539d50a1ef190c03dcb4c160 100644 --- a/drivers/net/wireless/quantenna/qtnfmac/qlink_util.h +++ b/drivers/net/wireless/quantenna/qtnfmac/qlink_util.h @@ -79,5 +79,10 @@ bool qtnf_utils_is_bit_set(const u8 *arr, unsigned int bit, unsigned int arr_max_len); void qlink_acl_data_cfg2q(const struct cfg80211_acl_data *acl, struct qlink_acl_data *qacl); +enum qlink_band qlink_utils_band_cfg2q(enum nl80211_band band); +enum qlink_dfs_state qlink_utils_dfs_state_cfg2q(enum nl80211_dfs_state state); +u32 qlink_utils_chflags_cfg2q(u32 cfgflags); +void qlink_utils_regrule_q2nl(struct ieee80211_reg_rule *rule, + const struct qlink_tlv_reg_rule *tlv_rule); #endif /* _QTN_FMAC_QLINK_UTIL_H_ */ diff --git a/drivers/net/wireless/ralink/rt2x00/rt2800.h b/drivers/net/wireless/ralink/rt2x00/rt2800.h index b05ed2f3025a7734c3826856cef1ee76dc1f2906..06c38bafd2cafb0e9477d2ab5ada2ed65108f531 100644 --- a/drivers/net/wireless/ralink/rt2x00/rt2800.h +++ b/drivers/net/wireless/ralink/rt2x00/rt2800.h @@ -48,7 +48,8 @@ * RF2853 2.4G/5G 3T3R * RF3320 2.4G 1T1R(RT3350/RT3370/RT3390) * RF3322 2.4G 2T2R(RT3352/RT3371/RT3372/RT3391/RT3392) - * RF3053 2.4G/5G 3T3R(RT3883/RT3563/RT3573/RT3593/RT3662) + * RF3053 2.4G/5G 3T3R(RT3563/RT3573/RT3593) + * RF3853 2.4G/5G 3T3R(RT3883/RT3662) * RF5592 2.4G/5G 2T2R * RF3070 2.4G 1T1R * RF5360 2.4G 1T1R @@ -72,6 +73,7 @@ #define RF5592 0x000f #define RF3070 0x3070 #define RF3290 0x3290 +#define RF3853 0x3853 #define RF5350 0x5350 #define RF5360 0x5360 #define RF5362 0x5362 @@ -1725,6 +1727,20 @@ /* bits for new 2T devices */ #define TX_PWR_CFG_9B_STBC_MCS7 FIELD32(0x000000ff) +/* + * TX_TXBF_CFG: + */ +#define TX_TXBF_CFG_0 0x138c +#define TX_TXBF_CFG_1 0x13a4 +#define TX_TXBF_CFG_2 0x13a8 +#define TX_TXBF_CFG_3 0x13ac + +/* + * TX_FBK_CFG_3S: + */ +#define TX_FBK_CFG_3S_0 0x13c4 +#define TX_FBK_CFG_3S_1 0x13c8 + /* * RX_FILTER_CFG: RX configuration register. */ @@ -2296,6 +2312,7 @@ struct mac_iveiv_entry { /* * RFCSR 2: */ +#define RFCSR2_RESCAL_BP FIELD8(0x40) #define RFCSR2_RESCAL_EN FIELD8(0x80) #define RFCSR2_RX2_EN_MT7620 FIELD8(0x02) #define RFCSR2_TX2_EN_MT7620 FIELD8(0x20) diff --git a/drivers/net/wireless/ralink/rt2x00/rt2800lib.c b/drivers/net/wireless/ralink/rt2x00/rt2800lib.c index a03b5284a050952448f806d3e74cc164bea5a4a1..c8f2bf1243fdb40a0061d2a8fc8ae879af94a47c 100644 --- a/drivers/net/wireless/ralink/rt2x00/rt2800lib.c +++ b/drivers/net/wireless/ralink/rt2x00/rt2800lib.c @@ -381,7 +381,8 @@ static unsigned int rt2800_eeprom_word_index(struct rt2x00_dev *rt2x00dev, wiphy_name(rt2x00dev->hw->wiphy), word)) return 0; - if (rt2x00_rt(rt2x00dev, RT3593)) + if (rt2x00_rt(rt2x00dev, RT3593) || + rt2x00_rt(rt2x00dev, RT3883)) map = rt2800_eeprom_map_ext; else map = rt2800_eeprom_map; @@ -590,6 +591,7 @@ void rt2800_get_txwi_rxwi_size(struct rt2x00_dev *rt2x00dev, { switch (rt2x00dev->chip.rt) { case RT3593: + case RT3883: *txwi_size = TXWI_DESC_SIZE_4WORDS; *rxwi_size = RXWI_DESC_SIZE_5WORDS; break; @@ -1100,7 +1102,7 @@ void rt2800_txdone_entry(struct queue_entry *entry, u32 status, __le32 *txwi, } EXPORT_SYMBOL_GPL(rt2800_txdone_entry); -void rt2800_txdone(struct rt2x00_dev *rt2x00dev) +void rt2800_txdone(struct rt2x00_dev *rt2x00dev, unsigned int quota) { struct data_queue *queue; struct queue_entry *entry; @@ -1108,7 +1110,7 @@ void rt2800_txdone(struct rt2x00_dev *rt2x00dev) u8 qid; bool match; - while (kfifo_get(&rt2x00dev->txstatus_fifo, ®)) { + while (quota-- > 0 && kfifo_get(&rt2x00dev->txstatus_fifo, ®)) { /* * TX_STA_FIFO_PID_QUEUE is a 2-bit field, thus qid is * guaranteed to be one of the TX QIDs . @@ -1164,15 +1166,6 @@ bool rt2800_txstatus_timeout(struct rt2x00_dev *rt2x00dev) struct data_queue *queue; struct queue_entry *entry; - if (!test_bit(DEVICE_STATE_FLUSHING, &rt2x00dev->flags)) { - unsigned long tout = msecs_to_jiffies(1000); - - if (time_before(jiffies, rt2x00dev->last_nostatus_check + tout)) - return false; - } - - rt2x00dev->last_nostatus_check = jiffies; - tx_queue_for_each(rt2x00dev, queue) { entry = rt2x00queue_get_entry(queue, Q_INDEX_DONE); if (rt2800_entry_txstatus_timeout(rt2x00dev, entry)) @@ -1183,6 +1176,23 @@ bool rt2800_txstatus_timeout(struct rt2x00_dev *rt2x00dev) } EXPORT_SYMBOL_GPL(rt2800_txstatus_timeout); +/* + * test if there is an entry in any TX queue for which DMA is done + * but the TX status has not been returned yet + */ +bool rt2800_txstatus_pending(struct rt2x00_dev *rt2x00dev) +{ + struct data_queue *queue; + + tx_queue_for_each(rt2x00dev, queue) { + if (rt2x00queue_get_entry(queue, Q_INDEX_DMA_DONE) != + rt2x00queue_get_entry(queue, Q_INDEX_DONE)) + return true; + } + return false; +} +EXPORT_SYMBOL_GPL(rt2800_txstatus_pending); + void rt2800_txdone_nostatus(struct rt2x00_dev *rt2x00dev) { struct data_queue *queue; @@ -2172,7 +2182,8 @@ void rt2800_config_ant(struct rt2x00_dev *rt2x00dev, struct antenna_setup *ant) rt2800_bbp_write(rt2x00dev, 3, r3); rt2800_bbp_write(rt2x00dev, 1, r1); - if (rt2x00_rt(rt2x00dev, RT3593)) { + if (rt2x00_rt(rt2x00dev, RT3593) || + rt2x00_rt(rt2x00dev, RT3883)) { if (ant->rx_chain_num == 1) rt2800_bbp_write(rt2x00dev, 86, 0x00); else @@ -2194,7 +2205,8 @@ static void rt2800_config_lna_gain(struct rt2x00_dev *rt2x00dev, eeprom = rt2800_eeprom_read(rt2x00dev, EEPROM_LNA); lna_gain = rt2x00_get_field16(eeprom, EEPROM_LNA_A0); } else if (libconf->rf.channel <= 128) { - if (rt2x00_rt(rt2x00dev, RT3593)) { + if (rt2x00_rt(rt2x00dev, RT3593) || + rt2x00_rt(rt2x00dev, RT3883)) { eeprom = rt2800_eeprom_read(rt2x00dev, EEPROM_EXT_LNA2); lna_gain = rt2x00_get_field16(eeprom, EEPROM_EXT_LNA2_A1); @@ -2204,7 +2216,8 @@ static void rt2800_config_lna_gain(struct rt2x00_dev *rt2x00dev, EEPROM_RSSI_BG2_LNA_A1); } } else { - if (rt2x00_rt(rt2x00dev, RT3593)) { + if (rt2x00_rt(rt2x00dev, RT3593) || + rt2x00_rt(rt2x00dev, RT3883)) { eeprom = rt2800_eeprom_read(rt2x00dev, EEPROM_EXT_LNA2); lna_gain = rt2x00_get_field16(eeprom, EEPROM_EXT_LNA2_A2); @@ -2872,6 +2885,211 @@ static void rt2800_config_channel_rf3053(struct rt2x00_dev *rt2x00dev, } } +static void rt2800_config_channel_rf3853(struct rt2x00_dev *rt2x00dev, + struct ieee80211_conf *conf, + struct rf_channel *rf, + struct channel_info *info) +{ + u8 rfcsr; + u8 bbp; + u8 pwr1, pwr2, pwr3; + + const bool txbf_enabled = false; /* TODO */ + + /* TODO: add band selection */ + + if (rf->channel <= 14) + rt2800_rfcsr_write(rt2x00dev, 6, 0x40); + else if (rf->channel < 132) + rt2800_rfcsr_write(rt2x00dev, 6, 0x80); + else + rt2800_rfcsr_write(rt2x00dev, 6, 0x40); + + rt2800_rfcsr_write(rt2x00dev, 8, rf->rf1); + rt2800_rfcsr_write(rt2x00dev, 9, rf->rf3); + + if (rf->channel <= 14) + rt2800_rfcsr_write(rt2x00dev, 11, 0x46); + else + rt2800_rfcsr_write(rt2x00dev, 11, 0x48); + + if (rf->channel <= 14) + rt2800_rfcsr_write(rt2x00dev, 12, 0x1a); + else + rt2800_rfcsr_write(rt2x00dev, 12, 0x52); + + rt2800_rfcsr_write(rt2x00dev, 13, 0x12); + + rfcsr = rt2800_rfcsr_read(rt2x00dev, 1); + rt2x00_set_field8(&rfcsr, RFCSR1_RX0_PD, 0); + rt2x00_set_field8(&rfcsr, RFCSR1_TX0_PD, 0); + rt2x00_set_field8(&rfcsr, RFCSR1_RX1_PD, 0); + rt2x00_set_field8(&rfcsr, RFCSR1_TX1_PD, 0); + rt2x00_set_field8(&rfcsr, RFCSR1_RX2_PD, 0); + rt2x00_set_field8(&rfcsr, RFCSR1_TX2_PD, 0); + rt2x00_set_field8(&rfcsr, RFCSR1_RF_BLOCK_EN, 1); + rt2x00_set_field8(&rfcsr, RFCSR1_PLL_PD, 1); + + switch (rt2x00dev->default_ant.tx_chain_num) { + case 3: + rt2x00_set_field8(&rfcsr, RFCSR1_TX2_PD, 1); + /* fallthrough */ + case 2: + rt2x00_set_field8(&rfcsr, RFCSR1_TX1_PD, 1); + /* fallthrough */ + case 1: + rt2x00_set_field8(&rfcsr, RFCSR1_TX0_PD, 1); + break; + } + + switch (rt2x00dev->default_ant.rx_chain_num) { + case 3: + rt2x00_set_field8(&rfcsr, RFCSR1_RX2_PD, 1); + /* fallthrough */ + case 2: + rt2x00_set_field8(&rfcsr, RFCSR1_RX1_PD, 1); + /* fallthrough */ + case 1: + rt2x00_set_field8(&rfcsr, RFCSR1_RX0_PD, 1); + break; + } + rt2800_rfcsr_write(rt2x00dev, 1, rfcsr); + + rt2800_freq_cal_mode1(rt2x00dev); + + rfcsr = rt2800_rfcsr_read(rt2x00dev, 30); + if (!conf_is_ht40(conf)) + rfcsr &= ~(0x06); + else + rfcsr |= 0x06; + rt2800_rfcsr_write(rt2x00dev, 30, rfcsr); + + if (rf->channel <= 14) + rt2800_rfcsr_write(rt2x00dev, 31, 0xa0); + else + rt2800_rfcsr_write(rt2x00dev, 31, 0x80); + + if (conf_is_ht40(conf)) + rt2800_rfcsr_write(rt2x00dev, 32, 0x80); + else + rt2800_rfcsr_write(rt2x00dev, 32, 0xd8); + + if (rf->channel <= 14) + rt2800_rfcsr_write(rt2x00dev, 34, 0x3c); + else + rt2800_rfcsr_write(rt2x00dev, 34, 0x20); + + /* loopback RF_BS */ + rfcsr = rt2800_rfcsr_read(rt2x00dev, 36); + if (rf->channel <= 14) + rt2x00_set_field8(&rfcsr, RFCSR36_RF_BS, 1); + else + rt2x00_set_field8(&rfcsr, RFCSR36_RF_BS, 0); + rt2800_rfcsr_write(rt2x00dev, 36, rfcsr); + + if (rf->channel <= 14) + rfcsr = 0x23; + else if (rf->channel < 100) + rfcsr = 0x36; + else if (rf->channel < 132) + rfcsr = 0x32; + else + rfcsr = 0x30; + + if (txbf_enabled) + rfcsr |= 0x40; + + rt2800_rfcsr_write(rt2x00dev, 39, rfcsr); + + if (rf->channel <= 14) + rt2800_rfcsr_write(rt2x00dev, 44, 0x93); + else + rt2800_rfcsr_write(rt2x00dev, 44, 0x9b); + + if (rf->channel <= 14) + rfcsr = 0xbb; + else if (rf->channel < 100) + rfcsr = 0xeb; + else if (rf->channel < 132) + rfcsr = 0xb3; + else + rfcsr = 0x9b; + rt2800_rfcsr_write(rt2x00dev, 45, rfcsr); + + if (rf->channel <= 14) + rfcsr = 0x8e; + else + rfcsr = 0x8a; + + if (txbf_enabled) + rfcsr |= 0x20; + + rt2800_rfcsr_write(rt2x00dev, 49, rfcsr); + + rt2800_rfcsr_write(rt2x00dev, 50, 0x86); + + rfcsr = rt2800_rfcsr_read(rt2x00dev, 51); + if (rf->channel <= 14) + rt2800_rfcsr_write(rt2x00dev, 51, 0x75); + else + rt2800_rfcsr_write(rt2x00dev, 51, 0x51); + + rfcsr = rt2800_rfcsr_read(rt2x00dev, 52); + if (rf->channel <= 14) + rt2800_rfcsr_write(rt2x00dev, 52, 0x45); + else + rt2800_rfcsr_write(rt2x00dev, 52, 0x05); + + if (rf->channel <= 14) { + pwr1 = info->default_power1 & 0x1f; + pwr2 = info->default_power2 & 0x1f; + pwr3 = info->default_power3 & 0x1f; + } else { + pwr1 = 0x48 | ((info->default_power1 & 0x18) << 1) | + (info->default_power1 & 0x7); + pwr2 = 0x48 | ((info->default_power2 & 0x18) << 1) | + (info->default_power2 & 0x7); + pwr3 = 0x48 | ((info->default_power3 & 0x18) << 1) | + (info->default_power3 & 0x7); + } + + rt2800_rfcsr_write(rt2x00dev, 53, pwr1); + rt2800_rfcsr_write(rt2x00dev, 54, pwr2); + rt2800_rfcsr_write(rt2x00dev, 55, pwr3); + + rt2x00_dbg(rt2x00dev, "Channel:%d, pwr1:%02x, pwr2:%02x, pwr3:%02x\n", + rf->channel, pwr1, pwr2, pwr3); + + bbp = (info->default_power1 >> 5) | + ((info->default_power2 & 0xe0) >> 1); + rt2800_bbp_write(rt2x00dev, 109, bbp); + + bbp = rt2800_bbp_read(rt2x00dev, 110); + bbp &= 0x0f; + bbp |= (info->default_power3 & 0xe0) >> 1; + rt2800_bbp_write(rt2x00dev, 110, bbp); + + rfcsr = rt2800_rfcsr_read(rt2x00dev, 57); + if (rf->channel <= 14) + rt2800_rfcsr_write(rt2x00dev, 57, 0x6e); + else + rt2800_rfcsr_write(rt2x00dev, 57, 0x3e); + + /* Enable RF tuning */ + rfcsr = rt2800_rfcsr_read(rt2x00dev, 3); + rt2x00_set_field8(&rfcsr, RFCSR3_VCOCAL_EN, 1); + rt2800_rfcsr_write(rt2x00dev, 3, rfcsr); + + udelay(2000); + + bbp = rt2800_bbp_read(rt2x00dev, 49); + /* clear update flag */ + rt2800_bbp_write(rt2x00dev, 49, bbp & 0xfe); + rt2800_bbp_write(rt2x00dev, 49, bbp); + + /* TODO: add calibration for TxBF */ +} + #define POWER_BOUND 0x27 #define POWER_BOUND_5G 0x2b @@ -3675,19 +3893,51 @@ static char rt2800_txpower_to_dev(struct rt2x00_dev *rt2x00dev, unsigned int channel, char txpower) { - if (rt2x00_rt(rt2x00dev, RT3593)) + if (rt2x00_rt(rt2x00dev, RT3593) || + rt2x00_rt(rt2x00dev, RT3883)) txpower = rt2x00_get_field8(txpower, EEPROM_TXPOWER_ALC); if (channel <= 14) return clamp_t(char, txpower, MIN_G_TXPOWER, MAX_G_TXPOWER); - if (rt2x00_rt(rt2x00dev, RT3593)) + if (rt2x00_rt(rt2x00dev, RT3593) || + rt2x00_rt(rt2x00dev, RT3883)) return clamp_t(char, txpower, MIN_A_TXPOWER_3593, MAX_A_TXPOWER_3593); else return clamp_t(char, txpower, MIN_A_TXPOWER, MAX_A_TXPOWER); } +static void rt3883_bbp_adjust(struct rt2x00_dev *rt2x00dev, + struct rf_channel *rf) +{ + u8 bbp; + + bbp = (rf->channel > 14) ? 0x48 : 0x38; + rt2800_bbp_write_with_rx_chain(rt2x00dev, 66, bbp); + + rt2800_bbp_write(rt2x00dev, 69, 0x12); + + if (rf->channel <= 14) { + rt2800_bbp_write(rt2x00dev, 70, 0x0a); + } else { + /* Disable CCK packet detection */ + rt2800_bbp_write(rt2x00dev, 70, 0x00); + } + + rt2800_bbp_write(rt2x00dev, 73, 0x10); + + if (rf->channel > 14) { + rt2800_bbp_write(rt2x00dev, 62, 0x1d); + rt2800_bbp_write(rt2x00dev, 63, 0x1d); + rt2800_bbp_write(rt2x00dev, 64, 0x1d); + } else { + rt2800_bbp_write(rt2x00dev, 62, 0x2d); + rt2800_bbp_write(rt2x00dev, 63, 0x2d); + rt2800_bbp_write(rt2x00dev, 64, 0x2d); + } +} + static void rt2800_config_channel(struct rt2x00_dev *rt2x00dev, struct ieee80211_conf *conf, struct rf_channel *rf, @@ -3706,6 +3956,12 @@ static void rt2800_config_channel(struct rt2x00_dev *rt2x00dev, rt2800_txpower_to_dev(rt2x00dev, rf->channel, info->default_power3); + switch (rt2x00dev->chip.rt) { + case RT3883: + rt3883_bbp_adjust(rt2x00dev, rf); + break; + } + switch (rt2x00dev->chip.rf) { case RF2020: case RF3020: @@ -3726,6 +3982,9 @@ static void rt2800_config_channel(struct rt2x00_dev *rt2x00dev, case RF3322: rt2800_config_channel_rf3322(rt2x00dev, conf, rf, info); break; + case RF3853: + rt2800_config_channel_rf3853(rt2x00dev, conf, rf, info); + break; case RF3070: case RF5350: case RF5360: @@ -3807,6 +4066,15 @@ static void rt2800_config_channel(struct rt2x00_dev *rt2x00dev, rt2800_bbp_write(rt2x00dev, 63, 0x37 - rt2x00dev->lna_gain); rt2800_bbp_write(rt2x00dev, 64, 0x37 - rt2x00dev->lna_gain); rt2800_bbp_write(rt2x00dev, 77, 0x98); + } else if (rt2x00_rt(rt2x00dev, RT3883)) { + rt2800_bbp_write(rt2x00dev, 62, 0x37 - rt2x00dev->lna_gain); + rt2800_bbp_write(rt2x00dev, 63, 0x37 - rt2x00dev->lna_gain); + rt2800_bbp_write(rt2x00dev, 64, 0x37 - rt2x00dev->lna_gain); + + if (rt2x00dev->default_ant.rx_chain_num > 1) + rt2800_bbp_write(rt2x00dev, 86, 0x46); + else + rt2800_bbp_write(rt2x00dev, 86, 0); } else { rt2800_bbp_write(rt2x00dev, 62, 0x37 - rt2x00dev->lna_gain); rt2800_bbp_write(rt2x00dev, 63, 0x37 - rt2x00dev->lna_gain); @@ -3819,6 +4087,7 @@ static void rt2800_config_channel(struct rt2x00_dev *rt2x00dev, !rt2x00_rt(rt2x00dev, RT5392) && !rt2x00_rt(rt2x00dev, RT6352)) { if (rt2x00_has_cap_external_lna_bg(rt2x00dev)) { + rt2800_bbp_write(rt2x00dev, 82, 0x62); rt2800_bbp_write(rt2x00dev, 82, 0x62); rt2800_bbp_write(rt2x00dev, 75, 0x46); } else { @@ -3828,19 +4097,22 @@ static void rt2800_config_channel(struct rt2x00_dev *rt2x00dev, rt2800_bbp_write(rt2x00dev, 82, 0x84); rt2800_bbp_write(rt2x00dev, 75, 0x50); } - if (rt2x00_rt(rt2x00dev, RT3593)) + if (rt2x00_rt(rt2x00dev, RT3593) || + rt2x00_rt(rt2x00dev, RT3883)) rt2800_bbp_write(rt2x00dev, 83, 0x8a); } } else { if (rt2x00_rt(rt2x00dev, RT3572)) rt2800_bbp_write(rt2x00dev, 82, 0x94); - else if (rt2x00_rt(rt2x00dev, RT3593)) + else if (rt2x00_rt(rt2x00dev, RT3593) || + rt2x00_rt(rt2x00dev, RT3883)) rt2800_bbp_write(rt2x00dev, 82, 0x82); else if (!rt2x00_rt(rt2x00dev, RT6352)) rt2800_bbp_write(rt2x00dev, 82, 0xf2); - if (rt2x00_rt(rt2x00dev, RT3593)) + if (rt2x00_rt(rt2x00dev, RT3593) || + rt2x00_rt(rt2x00dev, RT3883)) rt2800_bbp_write(rt2x00dev, 83, 0x9a); if (rt2x00_has_cap_external_lna_a(rt2x00dev)) @@ -3976,6 +4248,23 @@ static void rt2800_config_channel(struct rt2x00_dev *rt2x00dev, usleep_range(1000, 1500); } + if (rt2x00_rt(rt2x00dev, RT3883)) { + if (!conf_is_ht40(conf)) + rt2800_bbp_write(rt2x00dev, 105, 0x34); + else + rt2800_bbp_write(rt2x00dev, 105, 0x04); + + /* AGC init */ + if (rf->channel <= 14) + reg = 0x2e + rt2x00dev->lna_gain; + else + reg = 0x20 + ((rt2x00dev->lna_gain * 5) / 3); + + rt2800_bbp_write_with_rx_chain(rt2x00dev, 66, reg); + + usleep_range(1000, 1500); + } + if (rt2x00_rt(rt2x00dev, RT5592) || rt2x00_rt(rt2x00dev, RT6352)) { reg = 0x10; if (!conf_is_ht40(conf)) { @@ -4235,6 +4524,9 @@ static u8 rt2800_compensate_txpower(struct rt2x00_dev *rt2x00dev, int is_rate_b, if (rt2x00_rt(rt2x00dev, RT3593)) return min_t(u8, txpower, 0xc); + if (rt2x00_rt(rt2x00dev, RT3883)) + return min_t(u8, txpower, 0xf); + if (rt2x00_has_cap_power_limit(rt2x00dev)) { /* * Check if eirp txpower exceed txpower_limit. @@ -4996,7 +5288,8 @@ static void rt2800_config_txpower(struct rt2x00_dev *rt2x00dev, struct ieee80211_channel *chan, int power_level) { - if (rt2x00_rt(rt2x00dev, RT3593)) + if (rt2x00_rt(rt2x00dev, RT3593) || + rt2x00_rt(rt2x00dev, RT3883)) rt2800_config_txpower_rt3593(rt2x00dev, chan, power_level); else if (rt2x00_rt(rt2x00dev, RT6352)) rt2800_config_txpower_rt6352(rt2x00dev, chan, power_level); @@ -5043,6 +5336,7 @@ void rt2800_vco_calibration(struct rt2x00_dev *rt2x00dev) case RF3053: case RF3070: case RF3290: + case RF3853: case RF5350: case RF5360: case RF5362: @@ -5243,7 +5537,8 @@ static u8 rt2800_get_default_vgc(struct rt2x00_dev *rt2x00dev) else vgc = 0x2e + rt2x00dev->lna_gain; } else { /* 5GHZ band */ - if (rt2x00_rt(rt2x00dev, RT3593)) + if (rt2x00_rt(rt2x00dev, RT3593) || + rt2x00_rt(rt2x00dev, RT3883)) vgc = 0x20 + (rt2x00dev->lna_gain * 5) / 3; else if (rt2x00_rt(rt2x00dev, RT5592)) vgc = 0x24 + (2 * rt2x00dev->lna_gain); @@ -5263,7 +5558,8 @@ static inline void rt2800_set_vgc(struct rt2x00_dev *rt2x00dev, { if (qual->vgc_level != vgc_level) { if (rt2x00_rt(rt2x00dev, RT3572) || - rt2x00_rt(rt2x00dev, RT3593)) { + rt2x00_rt(rt2x00dev, RT3593) || + rt2x00_rt(rt2x00dev, RT3883)) { rt2800_bbp_write_with_rx_chain(rt2x00dev, 66, vgc_level); } else if (rt2x00_rt(rt2x00dev, RT5592)) { @@ -5310,6 +5606,11 @@ void rt2800_link_tuner(struct rt2x00_dev *rt2x00dev, struct link_qual *qual, } break; + case RT3883: + if (qual->rssi > -65) + vgc += 0x10; + break; + case RT5592: if (qual->rssi > -65) vgc += 0x20; @@ -5462,6 +5763,12 @@ static int rt2800_init_registers(struct rt2x00_dev *rt2x00dev) rt2800_register_write(rt2x00dev, TX_SW_CFG2, 0x00000000); } + } else if (rt2x00_rt(rt2x00dev, RT3883)) { + rt2800_register_write(rt2x00dev, TX_SW_CFG0, 0x00000402); + rt2800_register_write(rt2x00dev, TX_SW_CFG1, 0x00000000); + rt2800_register_write(rt2x00dev, TX_SW_CFG2, 0x00040000); + rt2800_register_write(rt2x00dev, TX_TXBF_CFG_0, 0x8000fc21); + rt2800_register_write(rt2x00dev, TX_TXBF_CFG_3, 0x00009c40); } else if (rt2x00_rt(rt2x00dev, RT5390) || rt2x00_rt(rt2x00dev, RT5392) || rt2x00_rt(rt2x00dev, RT6352)) { @@ -5675,6 +5982,11 @@ static int rt2800_init_registers(struct rt2x00_dev *rt2x00dev) reg = rt2x00_rt(rt2x00dev, RT5592) ? 0x00000082 : 0x00000002; rt2800_register_write(rt2x00dev, TXOP_HLDR_ET, reg); + if (rt2x00_rt(rt2x00dev, RT3883)) { + rt2800_register_write(rt2x00dev, TX_FBK_CFG_3S_0, 0x12111008); + rt2800_register_write(rt2x00dev, TX_FBK_CFG_3S_1, 0x16151413); + } + reg = rt2800_register_read(rt2x00dev, TX_RTS_CFG); rt2x00_set_field32(®, TX_RTS_CFG_AUTO_RTS_RETRY_LIMIT, 7); rt2x00_set_field32(®, TX_RTS_CFG_RTS_THRES, @@ -6291,6 +6603,47 @@ static void rt2800_init_bbp_3593(struct rt2x00_dev *rt2x00dev) rt2800_bbp_write(rt2x00dev, 103, 0xc0); } +static void rt2800_init_bbp_3883(struct rt2x00_dev *rt2x00dev) +{ + rt2800_init_bbp_early(rt2x00dev); + + rt2800_bbp_write(rt2x00dev, 4, 0x50); + rt2800_bbp_write(rt2x00dev, 47, 0x48); + + rt2800_bbp_write(rt2x00dev, 86, 0x46); + rt2800_bbp_write(rt2x00dev, 88, 0x90); + + rt2800_bbp_write(rt2x00dev, 92, 0x02); + + rt2800_bbp_write(rt2x00dev, 103, 0xc0); + rt2800_bbp_write(rt2x00dev, 104, 0x92); + rt2800_bbp_write(rt2x00dev, 105, 0x34); + rt2800_bbp_write(rt2x00dev, 106, 0x12); + rt2800_bbp_write(rt2x00dev, 120, 0x50); + rt2800_bbp_write(rt2x00dev, 137, 0x0f); + rt2800_bbp_write(rt2x00dev, 163, 0x9d); + + /* Set ITxBF timeout to 0x9C40=1000msec */ + rt2800_bbp_write(rt2x00dev, 179, 0x02); + rt2800_bbp_write(rt2x00dev, 180, 0x00); + rt2800_bbp_write(rt2x00dev, 182, 0x40); + rt2800_bbp_write(rt2x00dev, 180, 0x01); + rt2800_bbp_write(rt2x00dev, 182, 0x9c); + + rt2800_bbp_write(rt2x00dev, 179, 0x00); + + /* Reprogram the inband interface to put right values in RXWI */ + rt2800_bbp_write(rt2x00dev, 142, 0x04); + rt2800_bbp_write(rt2x00dev, 143, 0x3b); + rt2800_bbp_write(rt2x00dev, 142, 0x06); + rt2800_bbp_write(rt2x00dev, 143, 0xa0); + rt2800_bbp_write(rt2x00dev, 142, 0x07); + rt2800_bbp_write(rt2x00dev, 143, 0xa1); + rt2800_bbp_write(rt2x00dev, 142, 0x08); + rt2800_bbp_write(rt2x00dev, 143, 0xa2); + rt2800_bbp_write(rt2x00dev, 148, 0xc8); +} + static void rt2800_init_bbp_53xx(struct rt2x00_dev *rt2x00dev) { int ant, div_mode; @@ -6735,6 +7088,9 @@ static void rt2800_init_bbp(struct rt2x00_dev *rt2x00dev) case RT3593: rt2800_init_bbp_3593(rt2x00dev); return; + case RT3883: + rt2800_init_bbp_3883(rt2x00dev); + return; case RT5390: case RT5392: rt2800_init_bbp_53xx(rt2x00dev); @@ -7606,6 +7962,144 @@ static void rt2800_init_rfcsr_5350(struct rt2x00_dev *rt2x00dev) rt2800_rfcsr_write(rt2x00dev, 63, 0x00); } +static void rt2800_init_rfcsr_3883(struct rt2x00_dev *rt2x00dev) +{ + u8 rfcsr; + + /* TODO: get the actual ECO value from the SoC */ + const unsigned int eco = 5; + + rt2800_rf_init_calibration(rt2x00dev, 2); + + rt2800_rfcsr_write(rt2x00dev, 0, 0xe0); + rt2800_rfcsr_write(rt2x00dev, 1, 0x03); + rt2800_rfcsr_write(rt2x00dev, 2, 0x50); + rt2800_rfcsr_write(rt2x00dev, 3, 0x20); + rt2800_rfcsr_write(rt2x00dev, 4, 0x00); + rt2800_rfcsr_write(rt2x00dev, 5, 0x00); + rt2800_rfcsr_write(rt2x00dev, 6, 0x40); + rt2800_rfcsr_write(rt2x00dev, 7, 0x00); + rt2800_rfcsr_write(rt2x00dev, 8, 0x5b); + rt2800_rfcsr_write(rt2x00dev, 9, 0x08); + rt2800_rfcsr_write(rt2x00dev, 10, 0xd3); + rt2800_rfcsr_write(rt2x00dev, 11, 0x48); + rt2800_rfcsr_write(rt2x00dev, 12, 0x1a); + rt2800_rfcsr_write(rt2x00dev, 13, 0x12); + rt2800_rfcsr_write(rt2x00dev, 14, 0x00); + rt2800_rfcsr_write(rt2x00dev, 15, 0x00); + rt2800_rfcsr_write(rt2x00dev, 16, 0x00); + + /* RFCSR 17 will be initialized later based on the + * frequency offset stored in the EEPROM + */ + + rt2800_rfcsr_write(rt2x00dev, 18, 0x40); + rt2800_rfcsr_write(rt2x00dev, 19, 0x00); + rt2800_rfcsr_write(rt2x00dev, 20, 0x00); + rt2800_rfcsr_write(rt2x00dev, 21, 0x00); + rt2800_rfcsr_write(rt2x00dev, 22, 0x20); + rt2800_rfcsr_write(rt2x00dev, 23, 0xc0); + rt2800_rfcsr_write(rt2x00dev, 24, 0x00); + rt2800_rfcsr_write(rt2x00dev, 25, 0x00); + rt2800_rfcsr_write(rt2x00dev, 26, 0x00); + rt2800_rfcsr_write(rt2x00dev, 27, 0x00); + rt2800_rfcsr_write(rt2x00dev, 28, 0x00); + rt2800_rfcsr_write(rt2x00dev, 29, 0x00); + rt2800_rfcsr_write(rt2x00dev, 30, 0x10); + rt2800_rfcsr_write(rt2x00dev, 31, 0x80); + rt2800_rfcsr_write(rt2x00dev, 32, 0x80); + rt2800_rfcsr_write(rt2x00dev, 33, 0x00); + rt2800_rfcsr_write(rt2x00dev, 34, 0x20); + rt2800_rfcsr_write(rt2x00dev, 35, 0x00); + rt2800_rfcsr_write(rt2x00dev, 36, 0x00); + rt2800_rfcsr_write(rt2x00dev, 37, 0x00); + rt2800_rfcsr_write(rt2x00dev, 38, 0x86); + rt2800_rfcsr_write(rt2x00dev, 39, 0x23); + rt2800_rfcsr_write(rt2x00dev, 40, 0x00); + rt2800_rfcsr_write(rt2x00dev, 41, 0x00); + rt2800_rfcsr_write(rt2x00dev, 42, 0x00); + rt2800_rfcsr_write(rt2x00dev, 43, 0x00); + rt2800_rfcsr_write(rt2x00dev, 44, 0x93); + rt2800_rfcsr_write(rt2x00dev, 45, 0xbb); + rt2800_rfcsr_write(rt2x00dev, 46, 0x60); + rt2800_rfcsr_write(rt2x00dev, 47, 0x00); + rt2800_rfcsr_write(rt2x00dev, 48, 0x00); + rt2800_rfcsr_write(rt2x00dev, 49, 0x8e); + rt2800_rfcsr_write(rt2x00dev, 50, 0x86); + rt2800_rfcsr_write(rt2x00dev, 51, 0x51); + rt2800_rfcsr_write(rt2x00dev, 52, 0x05); + rt2800_rfcsr_write(rt2x00dev, 53, 0x76); + rt2800_rfcsr_write(rt2x00dev, 54, 0x76); + rt2800_rfcsr_write(rt2x00dev, 55, 0x76); + rt2800_rfcsr_write(rt2x00dev, 56, 0xdb); + rt2800_rfcsr_write(rt2x00dev, 57, 0x3e); + rt2800_rfcsr_write(rt2x00dev, 58, 0x00); + rt2800_rfcsr_write(rt2x00dev, 59, 0x00); + rt2800_rfcsr_write(rt2x00dev, 60, 0x00); + rt2800_rfcsr_write(rt2x00dev, 61, 0x00); + rt2800_rfcsr_write(rt2x00dev, 62, 0x00); + rt2800_rfcsr_write(rt2x00dev, 63, 0x00); + + /* TODO: rx filter calibration? */ + + rt2800_bbp_write(rt2x00dev, 137, 0x0f); + + rt2800_bbp_write(rt2x00dev, 163, 0x9d); + + rt2800_bbp_write(rt2x00dev, 105, 0x05); + + rt2800_bbp_write(rt2x00dev, 179, 0x02); + rt2800_bbp_write(rt2x00dev, 180, 0x00); + rt2800_bbp_write(rt2x00dev, 182, 0x40); + rt2800_bbp_write(rt2x00dev, 180, 0x01); + rt2800_bbp_write(rt2x00dev, 182, 0x9c); + + rt2800_bbp_write(rt2x00dev, 179, 0x00); + + rt2800_bbp_write(rt2x00dev, 142, 0x04); + rt2800_bbp_write(rt2x00dev, 143, 0x3b); + rt2800_bbp_write(rt2x00dev, 142, 0x06); + rt2800_bbp_write(rt2x00dev, 143, 0xa0); + rt2800_bbp_write(rt2x00dev, 142, 0x07); + rt2800_bbp_write(rt2x00dev, 143, 0xa1); + rt2800_bbp_write(rt2x00dev, 142, 0x08); + rt2800_bbp_write(rt2x00dev, 143, 0xa2); + rt2800_bbp_write(rt2x00dev, 148, 0xc8); + + if (eco == 5) { + rt2800_rfcsr_write(rt2x00dev, 32, 0xd8); + rt2800_rfcsr_write(rt2x00dev, 33, 0x32); + } + + rfcsr = rt2800_rfcsr_read(rt2x00dev, 2); + rt2x00_set_field8(&rfcsr, RFCSR2_RESCAL_BP, 0); + rt2x00_set_field8(&rfcsr, RFCSR2_RESCAL_EN, 1); + rt2800_rfcsr_write(rt2x00dev, 2, rfcsr); + msleep(1); + rt2x00_set_field8(&rfcsr, RFCSR2_RESCAL_EN, 0); + rt2800_rfcsr_write(rt2x00dev, 2, rfcsr); + + rfcsr = rt2800_rfcsr_read(rt2x00dev, 1); + rt2x00_set_field8(&rfcsr, RFCSR1_RF_BLOCK_EN, 1); + rt2800_rfcsr_write(rt2x00dev, 1, rfcsr); + + rfcsr = rt2800_rfcsr_read(rt2x00dev, 6); + rfcsr |= 0xc0; + rt2800_rfcsr_write(rt2x00dev, 6, rfcsr); + + rfcsr = rt2800_rfcsr_read(rt2x00dev, 22); + rfcsr |= 0x20; + rt2800_rfcsr_write(rt2x00dev, 22, rfcsr); + + rfcsr = rt2800_rfcsr_read(rt2x00dev, 46); + rfcsr |= 0x20; + rt2800_rfcsr_write(rt2x00dev, 46, rfcsr); + + rfcsr = rt2800_rfcsr_read(rt2x00dev, 20); + rfcsr &= ~0xee; + rt2800_rfcsr_write(rt2x00dev, 20, rfcsr); +} + static void rt2800_init_rfcsr_5390(struct rt2x00_dev *rt2x00dev) { rt2800_rf_init_calibration(rt2x00dev, 2); @@ -8448,6 +8942,9 @@ static void rt2800_init_rfcsr(struct rt2x00_dev *rt2x00dev) case RT3390: rt2800_init_rfcsr_3390(rt2x00dev); break; + case RT3883: + rt2800_init_rfcsr_3883(rt2x00dev); + break; case RT3572: rt2800_init_rfcsr_3572(rt2x00dev); break; @@ -8653,7 +9150,8 @@ static u8 rt2800_get_txmixer_gain_24g(struct rt2x00_dev *rt2x00dev) { u16 word; - if (rt2x00_rt(rt2x00dev, RT3593)) + if (rt2x00_rt(rt2x00dev, RT3593) || + rt2x00_rt(rt2x00dev, RT3883)) return 0; word = rt2800_eeprom_read(rt2x00dev, EEPROM_TXMIXER_GAIN_BG); @@ -8667,7 +9165,8 @@ static u8 rt2800_get_txmixer_gain_5g(struct rt2x00_dev *rt2x00dev) { u16 word; - if (rt2x00_rt(rt2x00dev, RT3593)) + if (rt2x00_rt(rt2x00dev, RT3593) || + rt2x00_rt(rt2x00dev, RT3883)) return 0; word = rt2800_eeprom_read(rt2x00dev, EEPROM_TXMIXER_GAIN_A); @@ -8773,7 +9272,8 @@ static int rt2800_validate_eeprom(struct rt2x00_dev *rt2x00dev) word = rt2800_eeprom_read(rt2x00dev, EEPROM_RSSI_BG2); if (abs(rt2x00_get_field16(word, EEPROM_RSSI_BG2_OFFSET2)) > 10) rt2x00_set_field16(&word, EEPROM_RSSI_BG2_OFFSET2, 0); - if (!rt2x00_rt(rt2x00dev, RT3593)) { + if (!rt2x00_rt(rt2x00dev, RT3593) && + !rt2x00_rt(rt2x00dev, RT3883)) { if (rt2x00_get_field16(word, EEPROM_RSSI_BG2_LNA_A1) == 0x00 || rt2x00_get_field16(word, EEPROM_RSSI_BG2_LNA_A1) == 0xff) rt2x00_set_field16(&word, EEPROM_RSSI_BG2_LNA_A1, @@ -8793,7 +9293,8 @@ static int rt2800_validate_eeprom(struct rt2x00_dev *rt2x00dev) word = rt2800_eeprom_read(rt2x00dev, EEPROM_RSSI_A2); if (abs(rt2x00_get_field16(word, EEPROM_RSSI_A2_OFFSET2)) > 10) rt2x00_set_field16(&word, EEPROM_RSSI_A2_OFFSET2, 0); - if (!rt2x00_rt(rt2x00dev, RT3593)) { + if (!rt2x00_rt(rt2x00dev, RT3593) && + !rt2x00_rt(rt2x00dev, RT3883)) { if (rt2x00_get_field16(word, EEPROM_RSSI_A2_LNA_A2) == 0x00 || rt2x00_get_field16(word, EEPROM_RSSI_A2_LNA_A2) == 0xff) rt2x00_set_field16(&word, EEPROM_RSSI_A2_LNA_A2, @@ -8801,7 +9302,8 @@ static int rt2800_validate_eeprom(struct rt2x00_dev *rt2x00dev) } rt2800_eeprom_write(rt2x00dev, EEPROM_RSSI_A2, word); - if (rt2x00_rt(rt2x00dev, RT3593)) { + if (rt2x00_rt(rt2x00dev, RT3593) || + rt2x00_rt(rt2x00dev, RT3883)) { word = rt2800_eeprom_read(rt2x00dev, EEPROM_EXT_LNA2); if (rt2x00_get_field16(word, EEPROM_EXT_LNA2_A1) == 0x00 || rt2x00_get_field16(word, EEPROM_EXT_LNA2_A1) == 0xff) @@ -8840,6 +9342,8 @@ static int rt2800_init_eeprom(struct rt2x00_dev *rt2x00dev) rf = rt2800_eeprom_read(rt2x00dev, EEPROM_CHIP_ID); else if (rt2x00_rt(rt2x00dev, RT3352)) rf = RF3322; + else if (rt2x00_rt(rt2x00dev, RT3883)) + rf = RF3853; else if (rt2x00_rt(rt2x00dev, RT5350)) rf = RF5350; else @@ -8860,6 +9364,7 @@ static int rt2800_init_eeprom(struct rt2x00_dev *rt2x00dev) case RF3290: case RF3320: case RF3322: + case RF3853: case RF5350: case RF5360: case RF5362: @@ -9146,6 +9651,66 @@ static const struct rf_channel rf_vals_3x_xtal20[] = { {14, 0xF0, 2, 0x18}, }; +static const struct rf_channel rf_vals_3853[] = { + {1, 241, 6, 2}, + {2, 241, 6, 7}, + {3, 242, 6, 2}, + {4, 242, 6, 7}, + {5, 243, 6, 2}, + {6, 243, 6, 7}, + {7, 244, 6, 2}, + {8, 244, 6, 7}, + {9, 245, 6, 2}, + {10, 245, 6, 7}, + {11, 246, 6, 2}, + {12, 246, 6, 7}, + {13, 247, 6, 2}, + {14, 248, 6, 4}, + + {36, 0x56, 8, 4}, + {38, 0x56, 8, 6}, + {40, 0x56, 8, 8}, + {44, 0x57, 8, 0}, + {46, 0x57, 8, 2}, + {48, 0x57, 8, 4}, + {52, 0x57, 8, 8}, + {54, 0x57, 8, 10}, + {56, 0x58, 8, 0}, + {60, 0x58, 8, 4}, + {62, 0x58, 8, 6}, + {64, 0x58, 8, 8}, + + {100, 0x5b, 8, 8}, + {102, 0x5b, 8, 10}, + {104, 0x5c, 8, 0}, + {108, 0x5c, 8, 4}, + {110, 0x5c, 8, 6}, + {112, 0x5c, 8, 8}, + {114, 0x5c, 8, 10}, + {116, 0x5d, 8, 0}, + {118, 0x5d, 8, 2}, + {120, 0x5d, 8, 4}, + {124, 0x5d, 8, 8}, + {126, 0x5d, 8, 10}, + {128, 0x5e, 8, 0}, + {132, 0x5e, 8, 4}, + {134, 0x5e, 8, 6}, + {136, 0x5e, 8, 8}, + {140, 0x5f, 8, 0}, + + {149, 0x5f, 8, 9}, + {151, 0x5f, 8, 11}, + {153, 0x60, 8, 1}, + {157, 0x60, 8, 5}, + {159, 0x60, 8, 7}, + {161, 0x60, 8, 9}, + {165, 0x61, 8, 1}, + {167, 0x61, 8, 3}, + {169, 0x61, 8, 5}, + {171, 0x61, 8, 7}, + {173, 0x61, 8, 9}, +}; + static const struct rf_channel rf_vals_5592_xtal20[] = { /* Channel, N, K, mod, R */ {1, 482, 4, 10, 3}, @@ -9409,6 +9974,11 @@ static int rt2800_probe_hw_mode(struct rt2x00_dev *rt2x00dev) spec->channels = rf_vals_3x; break; + case RF3853: + spec->num_channels = ARRAY_SIZE(rf_vals_3853); + spec->channels = rf_vals_3853; + break; + case RF5592: reg = rt2800_register_read(rt2x00dev, MAC_DEBUG_INDEX); if (rt2x00_get_field32(reg, MAC_DEBUG_INDEX_XTAL)) { @@ -9528,6 +10098,7 @@ static int rt2800_probe_hw_mode(struct rt2x00_dev *rt2x00dev) case RF3053: case RF3070: case RF3290: + case RF3853: case RF5350: case RF5360: case RF5362: @@ -9570,6 +10141,7 @@ static int rt2800_probe_rt(struct rt2x00_dev *rt2x00dev) case RT3390: case RT3572: case RT3593: + case RT3883: case RT5350: case RT5390: case RT5392: diff --git a/drivers/net/wireless/ralink/rt2x00/rt2800lib.h b/drivers/net/wireless/ralink/rt2x00/rt2800lib.h index 0dff2c7b30105133122f60cac3417153f5eecc68..759eab2b70c3062ea7f8b1d08bc874efeb1b0690 100644 --- a/drivers/net/wireless/ralink/rt2x00/rt2800lib.h +++ b/drivers/net/wireless/ralink/rt2x00/rt2800lib.h @@ -195,9 +195,10 @@ void rt2800_process_rxwi(struct queue_entry *entry, struct rxdone_entry_desc *tx void rt2800_txdone_entry(struct queue_entry *entry, u32 status, __le32 *txwi, bool match); -void rt2800_txdone(struct rt2x00_dev *rt2x00dev); +void rt2800_txdone(struct rt2x00_dev *rt2x00dev, unsigned int quota); void rt2800_txdone_nostatus(struct rt2x00_dev *rt2x00dev); bool rt2800_txstatus_timeout(struct rt2x00_dev *rt2x00dev); +bool rt2800_txstatus_pending(struct rt2x00_dev *rt2x00dev); void rt2800_write_beacon(struct queue_entry *entry, struct txentry_desc *txdesc); void rt2800_clear_beacon(struct queue_entry *entry); diff --git a/drivers/net/wireless/ralink/rt2x00/rt2800mmio.c b/drivers/net/wireless/ralink/rt2x00/rt2800mmio.c index ddb88cfeace21816c5490fba5a1f07b8a1787d6d..ecc4c9332ec7d2a7e49cba7691d299c302af521b 100644 --- a/drivers/net/wireless/ralink/rt2x00/rt2800mmio.c +++ b/drivers/net/wireless/ralink/rt2x00/rt2800mmio.c @@ -255,26 +255,12 @@ void rt2800mmio_autowake_tasklet(unsigned long data) } EXPORT_SYMBOL_GPL(rt2800mmio_autowake_tasklet); -static void rt2800mmio_txdone(struct rt2x00_dev *rt2x00dev) -{ - bool timeout = false; - - while (!kfifo_is_empty(&rt2x00dev->txstatus_fifo) || - (timeout = rt2800_txstatus_timeout(rt2x00dev))) { - - rt2800_txdone(rt2x00dev); - - if (timeout) - rt2800_txdone_nostatus(rt2x00dev); - } -} - -static bool rt2800mmio_fetch_txstatus(struct rt2x00_dev *rt2x00dev) +static void rt2800mmio_fetch_txstatus(struct rt2x00_dev *rt2x00dev) { u32 status; - bool more = false; + unsigned long flags; - /* FIXEME: rewrite this comment + /* * The TX_FIFO_STATUS interrupt needs special care. We should * read TX_STA_FIFO but we should do it immediately as otherwise * the register can overflow and we would lose status reports. @@ -285,34 +271,32 @@ static bool rt2800mmio_fetch_txstatus(struct rt2x00_dev *rt2x00dev) * because we can schedule the tasklet multiple times (when the * interrupt fires again during tx status processing). * - * txstatus tasklet is called with INT_SOURCE_CSR_TX_FIFO_STATUS - * disabled so have only one producer and one consumer - we don't - * need to lock the kfifo. + * We also read statuses from tx status timeout timer, use + * lock to prevent concurent writes to fifo. */ + + spin_lock_irqsave(&rt2x00dev->irqmask_lock, flags); + while (!kfifo_is_full(&rt2x00dev->txstatus_fifo)) { status = rt2x00mmio_register_read(rt2x00dev, TX_STA_FIFO); if (!rt2x00_get_field32(status, TX_STA_FIFO_VALID)) break; kfifo_put(&rt2x00dev->txstatus_fifo, status); - more = true; } - return more; + spin_unlock_irqrestore(&rt2x00dev->irqmask_lock, flags); } void rt2800mmio_txstatus_tasklet(unsigned long data) { struct rt2x00_dev *rt2x00dev = (struct rt2x00_dev *)data; - do { - rt2800mmio_txdone(rt2x00dev); + rt2800_txdone(rt2x00dev, 16); - } while (rt2800mmio_fetch_txstatus(rt2x00dev)); + if (!kfifo_is_empty(&rt2x00dev->txstatus_fifo)) + tasklet_schedule(&rt2x00dev->txstatus_tasklet); - if (test_bit(DEVICE_STATE_ENABLED_RADIO, &rt2x00dev->flags)) - rt2800mmio_enable_interrupt(rt2x00dev, - INT_SOURCE_CSR_TX_FIFO_STATUS); } EXPORT_SYMBOL_GPL(rt2800mmio_txstatus_tasklet); @@ -339,8 +323,10 @@ irqreturn_t rt2800mmio_interrupt(int irq, void *dev_instance) mask = ~reg; if (rt2x00_get_field32(reg, INT_SOURCE_CSR_TX_FIFO_STATUS)) { + rt2x00_set_field32(&mask, INT_MASK_CSR_TX_FIFO_STATUS, 1); rt2800mmio_fetch_txstatus(rt2x00dev); - tasklet_schedule(&rt2x00dev->txstatus_tasklet); + if (!kfifo_is_empty(&rt2x00dev->txstatus_fifo)) + tasklet_schedule(&rt2x00dev->txstatus_tasklet); } if (rt2x00_get_field32(reg, INT_SOURCE_CSR_PRE_TBTT)) @@ -440,6 +426,9 @@ void rt2800mmio_start_queue(struct data_queue *queue) } EXPORT_SYMBOL_GPL(rt2800mmio_start_queue); +/* 200 ms */ +#define TXSTATUS_TIMEOUT 200000000 + void rt2800mmio_kick_queue(struct data_queue *queue) { struct rt2x00_dev *rt2x00dev = queue->rt2x00dev; @@ -454,6 +443,8 @@ void rt2800mmio_kick_queue(struct data_queue *queue) entry = rt2x00queue_get_entry(queue, Q_INDEX); rt2x00mmio_register_write(rt2x00dev, TX_CTX_IDX(queue->qid), entry->entry_idx); + hrtimer_start(&rt2x00dev->txstatus_timer, + TXSTATUS_TIMEOUT, HRTIMER_MODE_REL); break; case QID_MGMT: entry = rt2x00queue_get_entry(queue, Q_INDEX); @@ -498,11 +489,8 @@ void rt2800mmio_flush_queue(struct data_queue *queue, bool drop) * For TX queues schedule completion tasklet to catch * tx status timeouts, othewise just wait. */ - if (tx_queue) { - tasklet_disable(&rt2x00dev->txstatus_tasklet); - rt2800mmio_txdone(rt2x00dev); - tasklet_enable(&rt2x00dev->txstatus_tasklet); - } + if (tx_queue) + queue_work(rt2x00dev->workqueue, &rt2x00dev->txdone_work); /* * Wait for a little while to give the driver @@ -640,6 +628,10 @@ void rt2800mmio_clear_entry(struct queue_entry *entry) word = rt2x00_desc_read(entry_priv->desc, 1); rt2x00_set_field32(&word, TXD_W1_DMA_DONE, 1); rt2x00_desc_write(entry_priv->desc, 1, word); + + /* If last entry stop txstatus timer */ + if (entry->queue->length == 1) + hrtimer_cancel(&rt2x00dev->txstatus_timer); } } EXPORT_SYMBOL_GPL(rt2800mmio_clear_entry); @@ -772,6 +764,70 @@ int rt2800mmio_enable_radio(struct rt2x00_dev *rt2x00dev) } EXPORT_SYMBOL_GPL(rt2800mmio_enable_radio); +static void rt2800mmio_work_txdone(struct work_struct *work) +{ + struct rt2x00_dev *rt2x00dev = + container_of(work, struct rt2x00_dev, txdone_work); + + if (!test_bit(DEVICE_STATE_ENABLED_RADIO, &rt2x00dev->flags)) + return; + + while (!kfifo_is_empty(&rt2x00dev->txstatus_fifo) || + rt2800_txstatus_timeout(rt2x00dev)) { + + tasklet_disable(&rt2x00dev->txstatus_tasklet); + rt2800_txdone(rt2x00dev, UINT_MAX); + rt2800_txdone_nostatus(rt2x00dev); + tasklet_enable(&rt2x00dev->txstatus_tasklet); + } + + if (rt2800_txstatus_pending(rt2x00dev)) + hrtimer_start(&rt2x00dev->txstatus_timer, + TXSTATUS_TIMEOUT, HRTIMER_MODE_REL); +} + +static enum hrtimer_restart rt2800mmio_tx_sta_fifo_timeout(struct hrtimer *timer) +{ + struct rt2x00_dev *rt2x00dev = + container_of(timer, struct rt2x00_dev, txstatus_timer); + + if (!test_bit(DEVICE_STATE_ENABLED_RADIO, &rt2x00dev->flags)) + goto out; + + if (!rt2800_txstatus_pending(rt2x00dev)) + goto out; + + rt2800mmio_fetch_txstatus(rt2x00dev); + if (!kfifo_is_empty(&rt2x00dev->txstatus_fifo)) + tasklet_schedule(&rt2x00dev->txstatus_tasklet); + else + queue_work(rt2x00dev->workqueue, &rt2x00dev->txdone_work); +out: + return HRTIMER_NORESTART; +} + +int rt2800mmio_probe_hw(struct rt2x00_dev *rt2x00dev) +{ + int retval; + + retval = rt2800_probe_hw(rt2x00dev); + if (retval) + return retval; + + /* + * Set txstatus timer function. + */ + rt2x00dev->txstatus_timer.function = rt2800mmio_tx_sta_fifo_timeout; + + /* + * Overwrite TX done handler + */ + INIT_WORK(&rt2x00dev->txdone_work, rt2800mmio_work_txdone); + + return 0; +} +EXPORT_SYMBOL_GPL(rt2800mmio_probe_hw); + MODULE_AUTHOR(DRV_PROJECT); MODULE_VERSION(DRV_VERSION); MODULE_DESCRIPTION("rt2800 MMIO library"); diff --git a/drivers/net/wireless/ralink/rt2x00/rt2800mmio.h b/drivers/net/wireless/ralink/rt2x00/rt2800mmio.h index 3a513273f4149f9d97312c62ebc866cc33b95ef0..ca58e6c3a4e5de11f00871b4b46fbbc01e4eff8b 100644 --- a/drivers/net/wireless/ralink/rt2x00/rt2800mmio.h +++ b/drivers/net/wireless/ralink/rt2x00/rt2800mmio.h @@ -153,6 +153,7 @@ void rt2800mmio_stop_queue(struct data_queue *queue); void rt2800mmio_queue_init(struct data_queue *queue); /* Initialization functions */ +int rt2800mmio_probe_hw(struct rt2x00_dev *rt2x00dev); bool rt2800mmio_get_entry_state(struct queue_entry *entry); void rt2800mmio_clear_entry(struct queue_entry *entry); int rt2800mmio_init_queues(struct rt2x00_dev *rt2x00dev); diff --git a/drivers/net/wireless/ralink/rt2x00/rt2800pci.c b/drivers/net/wireless/ralink/rt2x00/rt2800pci.c index 0291441ac54827793c45c4d700df3fcb1b3cd835..43e1b1ee96bf44b749d06a354f26908ce393d239 100644 --- a/drivers/net/wireless/ralink/rt2x00/rt2800pci.c +++ b/drivers/net/wireless/ralink/rt2x00/rt2800pci.c @@ -346,7 +346,7 @@ static const struct rt2x00lib_ops rt2800pci_rt2x00_ops = { .tbtt_tasklet = rt2800mmio_tbtt_tasklet, .rxdone_tasklet = rt2800mmio_rxdone_tasklet, .autowake_tasklet = rt2800mmio_autowake_tasklet, - .probe_hw = rt2800_probe_hw, + .probe_hw = rt2800mmio_probe_hw, .get_firmware_name = rt2800pci_get_firmware_name, .check_firmware = rt2800_check_firmware, .load_firmware = rt2800_load_firmware, diff --git a/drivers/net/wireless/ralink/rt2x00/rt2800soc.c b/drivers/net/wireless/ralink/rt2x00/rt2800soc.c index a502816214ab0b53cecf749c42c6c1c2440c8812..4e9e38771a56bcdc248ef2e6139d84d1779e6429 100644 --- a/drivers/net/wireless/ralink/rt2x00/rt2800soc.c +++ b/drivers/net/wireless/ralink/rt2x00/rt2800soc.c @@ -51,9 +51,16 @@ static bool rt2800soc_hwcrypt_disabled(struct rt2x00_dev *rt2x00dev) static void rt2800soc_disable_radio(struct rt2x00_dev *rt2x00dev) { + u32 reg; + rt2800_disable_radio(rt2x00dev); rt2x00mmio_register_write(rt2x00dev, PWR_PIN_CFG, 0); - rt2x00mmio_register_write(rt2x00dev, TX_PIN_CFG, 0); + + reg = 0; + if (rt2x00_rt(rt2x00dev, RT3883)) + rt2x00_set_field32(®, TX_PIN_CFG_RFTR_EN, 1); + + rt2x00mmio_register_write(rt2x00dev, TX_PIN_CFG, reg); } static int rt2800soc_set_device_state(struct rt2x00_dev *rt2x00dev, @@ -185,7 +192,7 @@ static const struct rt2x00lib_ops rt2800soc_rt2x00_ops = { .tbtt_tasklet = rt2800mmio_tbtt_tasklet, .rxdone_tasklet = rt2800mmio_rxdone_tasklet, .autowake_tasklet = rt2800mmio_autowake_tasklet, - .probe_hw = rt2800_probe_hw, + .probe_hw = rt2800mmio_probe_hw, .get_firmware_name = rt2800soc_get_firmware_name, .check_firmware = rt2800soc_check_firmware, .load_firmware = rt2800soc_load_firmware, @@ -203,7 +210,7 @@ static const struct rt2x00lib_ops rt2800soc_rt2x00_ops = { .start_queue = rt2800mmio_start_queue, .kick_queue = rt2800mmio_kick_queue, .stop_queue = rt2800mmio_stop_queue, - .flush_queue = rt2x00mmio_flush_queue, + .flush_queue = rt2800mmio_flush_queue, .write_tx_desc = rt2800mmio_write_tx_desc, .write_tx_data = rt2800_write_tx_data, .write_beacon = rt2800_write_beacon, diff --git a/drivers/net/wireless/ralink/rt2x00/rt2800usb.c b/drivers/net/wireless/ralink/rt2x00/rt2800usb.c index 19eabf16147bfe0642938894a065d61c4d9809e3..b5f75df9b563c5a2bf81190598ebbce277e6054c 100644 --- a/drivers/net/wireless/ralink/rt2x00/rt2800usb.c +++ b/drivers/net/wireless/ralink/rt2x00/rt2800usb.c @@ -100,22 +100,6 @@ static void rt2800usb_stop_queue(struct data_queue *queue) } } -/* - * test if there is an entry in any TX queue for which DMA is done - * but the TX status has not been returned yet - */ -static bool rt2800usb_txstatus_pending(struct rt2x00_dev *rt2x00dev) -{ - struct data_queue *queue; - - tx_queue_for_each(rt2x00dev, queue) { - if (rt2x00queue_get_entry(queue, Q_INDEX_DMA_DONE) != - rt2x00queue_get_entry(queue, Q_INDEX_DONE)) - return true; - } - return false; -} - #define TXSTATUS_READ_INTERVAL 1000000 static bool rt2800usb_tx_sta_fifo_read_completed(struct rt2x00_dev *rt2x00dev, @@ -145,7 +129,7 @@ static bool rt2800usb_tx_sta_fifo_read_completed(struct rt2x00_dev *rt2x00dev, if (rt2800_txstatus_timeout(rt2x00dev)) queue_work(rt2x00dev->workqueue, &rt2x00dev->txdone_work); - if (rt2800usb_txstatus_pending(rt2x00dev)) { + if (rt2800_txstatus_pending(rt2x00dev)) { /* Read register after 1 ms */ hrtimer_start(&rt2x00dev->txstatus_timer, TXSTATUS_READ_INTERVAL, @@ -160,7 +144,7 @@ stop_reading: * clear_bit someone could do rt2x00usb_interrupt_txdone, so recheck * here again if status reading is needed. */ - if (rt2800usb_txstatus_pending(rt2x00dev) && + if (rt2800_txstatus_pending(rt2x00dev) && !test_and_set_bit(TX_STATUS_READING, &rt2x00dev->flags)) return true; else @@ -480,7 +464,7 @@ static void rt2800usb_work_txdone(struct work_struct *work) while (!kfifo_is_empty(&rt2x00dev->txstatus_fifo) || rt2800_txstatus_timeout(rt2x00dev)) { - rt2800_txdone(rt2x00dev); + rt2800_txdone(rt2x00dev, UINT_MAX); rt2800_txdone_nostatus(rt2x00dev); @@ -489,7 +473,7 @@ static void rt2800usb_work_txdone(struct work_struct *work) * if the medium is busy, thus the TX_STA_FIFO entry is * also delayed -> use a timer to retrieve it. */ - if (rt2800usb_txstatus_pending(rt2x00dev)) + if (rt2800_txstatus_pending(rt2x00dev)) rt2800usb_async_read_tx_status(rt2x00dev); } } @@ -562,13 +546,13 @@ static void rt2800usb_fill_rxdone(struct queue_entry *entry, * stripped it from the frame. Signal this to mac80211. */ rxdesc->flags |= RX_FLAG_MMIC_STRIPPED; - + if (rxdesc->cipher_status == RX_CRYPTO_SUCCESS) { rxdesc->flags |= RX_FLAG_DECRYPTED; } else if (rxdesc->cipher_status == RX_CRYPTO_FAIL_MIC) { /* * In order to check the Michael Mic, the packet must have - * been decrypted. Mac80211 doesnt check the MMIC failure + * been decrypted. Mac80211 doesnt check the MMIC failure * flag to initiate MMIC countermeasures if the decoded flag * has not been set. */ diff --git a/drivers/net/wireless/ralink/rt2x00/rt2x00.h b/drivers/net/wireless/ralink/rt2x00/rt2x00.h index 50b92ca92bd75c33d783ed9bfdf0f01f7d5ce0ae..9c6ef0ca932bf81880d9fd2747dceb268c0f7fba 100644 --- a/drivers/net/wireless/ralink/rt2x00/rt2x00.h +++ b/drivers/net/wireless/ralink/rt2x00/rt2x00.h @@ -69,10 +69,10 @@ printk(KERN_ERR KBUILD_MODNAME ": %s: Error - " fmt, \ __func__, ##__VA_ARGS__) #define rt2x00_err(dev, fmt, ...) \ - wiphy_err((dev)->hw->wiphy, "%s: Error - " fmt, \ + wiphy_err_ratelimited((dev)->hw->wiphy, "%s: Error - " fmt, \ __func__, ##__VA_ARGS__) #define rt2x00_warn(dev, fmt, ...) \ - wiphy_warn((dev)->hw->wiphy, "%s: Warning - " fmt, \ + wiphy_warn_ratelimited((dev)->hw->wiphy, "%s: Warning - " fmt, \ __func__, ##__VA_ARGS__) #define rt2x00_info(dev, fmt, ...) \ wiphy_info((dev)->hw->wiphy, "%s: Info - " fmt, \ @@ -980,8 +980,6 @@ struct rt2x00_dev { */ DECLARE_KFIFO_PTR(txstatus_fifo, u32); - unsigned long last_nostatus_check; - /* * Timer to ensure tx status reports are read (rt2800usb). */ @@ -1016,6 +1014,7 @@ struct rt2x00_dev { unsigned int extra_tx_headroom; struct usb_anchor *anchor; + unsigned int num_proto_errs; /* Clock for System On Chip devices. */ struct clk *clk; diff --git a/drivers/net/wireless/ralink/rt2x00/rt2x00dev.c b/drivers/net/wireless/ralink/rt2x00/rt2x00dev.c index 357c0941aaad4748fd5b69871359f5cb0475216f..1b08b01db27bc519dc7afc7d9c9efb646b490a3a 100644 --- a/drivers/net/wireless/ralink/rt2x00/rt2x00dev.c +++ b/drivers/net/wireless/ralink/rt2x00/rt2x00dev.c @@ -1007,7 +1007,7 @@ void rt2x00lib_set_mac_address(struct rt2x00_dev *rt2x00dev, u8 *eeprom_mac_addr const char *mac_addr; mac_addr = of_get_mac_address(rt2x00dev->dev->of_node); - if (mac_addr) + if (!IS_ERR(mac_addr)) ether_addr_copy(eeprom_mac_addr, mac_addr); if (!is_valid_ether_addr(eeprom_mac_addr)) { @@ -1391,6 +1391,8 @@ int rt2x00lib_probe_dev(struct rt2x00_dev *rt2x00dev) mutex_init(&rt2x00dev->conf_mutex); INIT_LIST_HEAD(&rt2x00dev->bar_list); spin_lock_init(&rt2x00dev->bar_list_lock); + hrtimer_init(&rt2x00dev->txstatus_timer, CLOCK_MONOTONIC, + HRTIMER_MODE_REL); set_bit(DEVICE_STATE_PRESENT, &rt2x00dev->flags); @@ -1515,6 +1517,8 @@ void rt2x00lib_remove_dev(struct rt2x00_dev *rt2x00dev) cancel_delayed_work_sync(&rt2x00dev->autowakeup_work); cancel_work_sync(&rt2x00dev->sleep_work); + hrtimer_cancel(&rt2x00dev->txstatus_timer); + /* * Kill the tx status tasklet. */ diff --git a/drivers/net/wireless/ralink/rt2x00/rt2x00mmio.h b/drivers/net/wireless/ralink/rt2x00/rt2x00mmio.h index 184a4148b2f8ff619915557bff928e17d93539e3..03e6cdb0b5a4ebb51051759a87344522d4f6c501 100644 --- a/drivers/net/wireless/ralink/rt2x00/rt2x00mmio.h +++ b/drivers/net/wireless/ralink/rt2x00/rt2x00mmio.h @@ -80,8 +80,6 @@ int rt2x00mmio_regbusy_read(struct rt2x00_dev *rt2x00dev, * * @desc: Pointer to device descriptor * @desc_dma: DMA pointer to &desc. - * @data: Pointer to device's entry memory. - * @data_dma: DMA pointer to &data. */ struct queue_entry_priv_mmio { __le32 *desc; diff --git a/drivers/net/wireless/ralink/rt2x00/rt2x00queue.c b/drivers/net/wireless/ralink/rt2x00/rt2x00queue.c index 4834b4eb0206408093a54d47b2a6a5831aa75674..03b206440208673f9902523c3d1246271f6aee30 100644 --- a/drivers/net/wireless/ralink/rt2x00/rt2x00queue.c +++ b/drivers/net/wireless/ralink/rt2x00/rt2x00queue.c @@ -674,7 +674,7 @@ int rt2x00queue_write_tx_frame(struct data_queue *queue, struct sk_buff *skb, spin_lock(&queue->tx_lock); if (unlikely(rt2x00queue_full(queue))) { - rt2x00_err(queue->rt2x00dev, "Dropping frame due to full tx queue %d\n", + rt2x00_dbg(queue->rt2x00dev, "Dropping frame due to full tx queue %d\n", queue->qid); ret = -ENOBUFS; goto out; @@ -1042,7 +1042,6 @@ void rt2x00queue_start_queues(struct rt2x00_dev *rt2x00dev) */ tx_queue_for_each(rt2x00dev, queue) rt2x00queue_start_queue(queue); - rt2x00dev->last_nostatus_check = jiffies; rt2x00queue_start_queue(rt2x00dev->rx); } diff --git a/drivers/net/wireless/ralink/rt2x00/rt2x00queue.h b/drivers/net/wireless/ralink/rt2x00/rt2x00queue.h index a15bae29917b96f1f8f2413ff9122fdb924b6676..20113f861b9e2c7dfe9d33cf890125d5c0628982 100644 --- a/drivers/net/wireless/ralink/rt2x00/rt2x00queue.h +++ b/drivers/net/wireless/ralink/rt2x00/rt2x00queue.h @@ -361,7 +361,6 @@ enum queue_entry_flags { ENTRY_DATA_PENDING, ENTRY_DATA_IO_FAILED, ENTRY_DATA_STATUS_PENDING, - ENTRY_DATA_STATUS_SET, }; /** @@ -387,8 +386,6 @@ struct queue_entry { unsigned int entry_idx; - u32 status; - void *priv_data; }; diff --git a/drivers/net/wireless/ralink/rt2x00/rt2x00usb.c b/drivers/net/wireless/ralink/rt2x00/rt2x00usb.c index 086aad22743dc6892ad25395275ae90df0c001fb..9cdd7f2c92b5f5c06ddf0884bfef38156ba6b22b 100644 --- a/drivers/net/wireless/ralink/rt2x00/rt2x00usb.c +++ b/drivers/net/wireless/ralink/rt2x00/rt2x00usb.c @@ -31,6 +31,22 @@ #include "rt2x00.h" #include "rt2x00usb.h" +static bool rt2x00usb_check_usb_error(struct rt2x00_dev *rt2x00dev, int status) +{ + if (status == -ENODEV || status == -ENOENT) + return true; + + if (status == -EPROTO || status == -ETIMEDOUT) + rt2x00dev->num_proto_errs++; + else + rt2x00dev->num_proto_errs = 0; + + if (rt2x00dev->num_proto_errs > 3) + return true; + + return false; +} + /* * Interfacing with the HW. */ @@ -57,7 +73,7 @@ int rt2x00usb_vendor_request(struct rt2x00_dev *rt2x00dev, if (status >= 0) return 0; - if (status == -ENODEV || status == -ENOENT) { + if (rt2x00usb_check_usb_error(rt2x00dev, status)) { /* Device has disappeared. */ clear_bit(DEVICE_STATE_PRESENT, &rt2x00dev->flags); break; @@ -321,7 +337,7 @@ static bool rt2x00usb_kick_tx_entry(struct queue_entry *entry, void *data) status = usb_submit_urb(entry_priv->urb, GFP_ATOMIC); if (status) { - if (status == -ENODEV || status == -ENOENT) + if (rt2x00usb_check_usb_error(rt2x00dev, status)) clear_bit(DEVICE_STATE_PRESENT, &rt2x00dev->flags); set_bit(ENTRY_DATA_IO_FAILED, &entry->flags); rt2x00lib_dmadone(entry); @@ -410,7 +426,7 @@ static bool rt2x00usb_kick_rx_entry(struct queue_entry *entry, void *data) status = usb_submit_urb(entry_priv->urb, GFP_ATOMIC); if (status) { - if (status == -ENODEV || status == -ENOENT) + if (rt2x00usb_check_usb_error(rt2x00dev, status)) clear_bit(DEVICE_STATE_PRESENT, &rt2x00dev->flags); set_bit(ENTRY_DATA_IO_FAILED, &entry->flags); rt2x00lib_dmadone(entry); diff --git a/drivers/net/wireless/ray_cs.c b/drivers/net/wireless/ray_cs.c index 44a943d18b84c83f24214c73d2f02a14896e8ef5..ee4d8106bba59cf9d6466d33e124eb241488d27a 100644 --- a/drivers/net/wireless/ray_cs.c +++ b/drivers/net/wireless/ray_cs.c @@ -2795,6 +2795,8 @@ static int __init init_ray_cs(void) rc = pcmcia_register_driver(&ray_driver); pr_debug("raylink init_module register_pcmcia_driver returns 0x%x\n", rc); + if (rc) + return rc; #ifdef CONFIG_PROC_FS proc_mkdir("driver/ray_cs", NULL); @@ -2818,11 +2820,7 @@ static void __exit exit_ray_cs(void) pr_debug("ray_cs: cleanup_module\n"); #ifdef CONFIG_PROC_FS - remove_proc_entry("driver/ray_cs/ray_cs", NULL); - remove_proc_entry("driver/ray_cs/essid", NULL); - remove_proc_entry("driver/ray_cs/net_type", NULL); - remove_proc_entry("driver/ray_cs/translate", NULL); - remove_proc_entry("driver/ray_cs", NULL); + remove_proc_subtree("driver/ray_cs", NULL); #endif pcmcia_unregister_driver(&ray_driver); diff --git a/drivers/net/wireless/realtek/Kconfig b/drivers/net/wireless/realtek/Kconfig index 3db988e689d788642eb8c5f65375c2fc4ed5414a..9189fd67257816fb4b1fb66eb5cabee3f59698f5 100644 --- a/drivers/net/wireless/realtek/Kconfig +++ b/drivers/net/wireless/realtek/Kconfig @@ -14,5 +14,6 @@ if WLAN_VENDOR_REALTEK source "drivers/net/wireless/realtek/rtl818x/Kconfig" source "drivers/net/wireless/realtek/rtlwifi/Kconfig" source "drivers/net/wireless/realtek/rtl8xxxu/Kconfig" +source "drivers/net/wireless/realtek/rtw88/Kconfig" endif # WLAN_VENDOR_REALTEK diff --git a/drivers/net/wireless/realtek/Makefile b/drivers/net/wireless/realtek/Makefile index 9c78deb5eea94f71320563b37a8b73f9e6a815f2..118af9963d6163c581e1bcae1cb0b9e08df0c94a 100644 --- a/drivers/net/wireless/realtek/Makefile +++ b/drivers/net/wireless/realtek/Makefile @@ -6,4 +6,5 @@ obj-$(CONFIG_RTL8180) += rtl818x/ obj-$(CONFIG_RTL8187) += rtl818x/ obj-$(CONFIG_RTLWIFI) += rtlwifi/ obj-$(CONFIG_RTL8XXXU) += rtl8xxxu/ +obj-$(CONFIG_RTW88) += rtw88/ diff --git a/drivers/net/wireless/realtek/rtlwifi/base.c b/drivers/net/wireless/realtek/rtlwifi/base.c index 217d2a7a43c74934744933dd0bf9f15bed7d4639..ac746c322554b393676f79f214b9fccdfddb3fda 100644 --- a/drivers/net/wireless/realtek/rtlwifi/base.c +++ b/drivers/net/wireless/realtek/rtlwifi/base.c @@ -448,6 +448,11 @@ static void _rtl_init_deferred_work(struct ieee80211_hw *hw) /* <2> work queue */ rtlpriv->works.hw = hw; rtlpriv->works.rtl_wq = alloc_workqueue("%s", 0, 0, rtlpriv->cfg->name); + if (unlikely(!rtlpriv->works.rtl_wq)) { + pr_err("Failed to allocate work queue\n"); + return; + } + INIT_DELAYED_WORK(&rtlpriv->works.watchdog_wq, (void *)rtl_watchdog_wq_callback); INIT_DELAYED_WORK(&rtlpriv->works.ips_nic_off_wq, diff --git a/drivers/net/wireless/realtek/rtlwifi/pci.c b/drivers/net/wireless/realtek/rtlwifi/pci.c index 48ca52102cef89b398630f02078e4b696708dba1..4055e0ab75bac7688548336398a4da99e1f60f59 100644 --- a/drivers/net/wireless/realtek/rtlwifi/pci.c +++ b/drivers/net/wireless/realtek/rtlwifi/pci.c @@ -499,16 +499,16 @@ static void _rtl_pci_tx_chk_waitq(struct ieee80211_hw *hw) memset(&tcb_desc, 0, sizeof(struct rtl_tcb_desc)); - spin_lock_bh(&rtlpriv->locks.waitq_lock); + spin_lock(&rtlpriv->locks.waitq_lock); if (!skb_queue_empty(&mac->skb_waitq[tid]) && (ring->entries - skb_queue_len(&ring->queue) > rtlhal->max_earlymode_num)) { skb = skb_dequeue(&mac->skb_waitq[tid]); } else { - spin_unlock_bh(&rtlpriv->locks.waitq_lock); + spin_unlock(&rtlpriv->locks.waitq_lock); break; } - spin_unlock_bh(&rtlpriv->locks.waitq_lock); + spin_unlock(&rtlpriv->locks.waitq_lock); /* Some macaddr can't do early mode. like * multicast/broadcast/no_qos data diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8188ee/fw.c b/drivers/net/wireless/realtek/rtlwifi/rtl8188ee/fw.c index 203e7b574e84549acdc24b569d772b9749ffe557..e2e0bfbc24fe245f6d3ed818618bb88a5436b25b 100644 --- a/drivers/net/wireless/realtek/rtlwifi/rtl8188ee/fw.c +++ b/drivers/net/wireless/realtek/rtlwifi/rtl8188ee/fw.c @@ -600,6 +600,8 @@ void rtl88e_set_fw_rsvdpagepkt(struct ieee80211_hw *hw, bool b_dl_finished) u1rsvdpageloc, 3); skb = dev_alloc_skb(totalpacketlen); + if (!skb) + return; skb_put_data(skb, &reserved_page_packet, totalpacketlen); rtstatus = rtl_cmd_send_packet(hw, skb); diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8188ee/trx.c b/drivers/net/wireless/realtek/rtlwifi/rtl8188ee/trx.c index 106011a2482728fdcd2e04567624e789af630808..483dc8bdc5551579e929d0f23fd56b9883f901ef 100644 --- a/drivers/net/wireless/realtek/rtlwifi/rtl8188ee/trx.c +++ b/drivers/net/wireless/realtek/rtlwifi/rtl8188ee/trx.c @@ -372,8 +372,9 @@ bool rtl88ee_rx_query_desc(struct ieee80211_hw *hw, struct rtl_priv *rtlpriv = rtl_priv(hw); struct rx_fwinfo_88e *p_drvinfo; struct ieee80211_hdr *hdr; - + u8 wake_match; u32 phystatus = GET_RX_DESC_PHYST(pdesc); + status->packet_report_type = (u8)GET_RX_STATUS_DESC_RPT_SEL(pdesc); if (status->packet_report_type == TX_REPORT2) status->length = (u16)GET_RX_RPT2_DESC_PKT_LEN(pdesc); @@ -399,18 +400,18 @@ bool rtl88ee_rx_query_desc(struct ieee80211_hw *hw, status->is_cck = RTL8188_RX_HAL_IS_CCK_RATE(status->rate); status->macid = GET_RX_DESC_MACID(pdesc); - if (GET_RX_STATUS_DESC_MAGIC_MATCH(pdesc)) - status->wake_match = BIT(2); + if (GET_RX_STATUS_DESC_PATTERN_MATCH(pdesc)) + wake_match = BIT(2); else if (GET_RX_STATUS_DESC_MAGIC_MATCH(pdesc)) - status->wake_match = BIT(1); + wake_match = BIT(1); else if (GET_RX_STATUS_DESC_UNICAST_MATCH(pdesc)) - status->wake_match = BIT(0); + wake_match = BIT(0); else - status->wake_match = 0; - if (status->wake_match) + wake_match = 0; + if (wake_match) RT_TRACE(rtlpriv, COMP_RXDESC, DBG_LOUD, "GGGGGGGGGGGGGet Wakeup Packet!! WakeMatch=%d\n", - status->wake_match); + wake_match); rx_status->freq = hw->conf.chandef.chan->center_freq; rx_status->band = hw->conf.chandef.chan->band; diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8192c/fw_common.c b/drivers/net/wireless/realtek/rtlwifi/rtl8192c/fw_common.c index 18c76990a089840cb1b9982ffbaa9b59c4970ea2..86b1b88cc4ed8093d8bb6194279fab07f35011bf 100644 --- a/drivers/net/wireless/realtek/rtlwifi/rtl8192c/fw_common.c +++ b/drivers/net/wireless/realtek/rtlwifi/rtl8192c/fw_common.c @@ -623,6 +623,8 @@ void rtl92c_set_fw_rsvdpagepkt(struct ieee80211_hw *hw, u1rsvdpageloc, 3); skb = dev_alloc_skb(totalpacketlen); + if (!skb) + return; skb_put_data(skb, &reserved_page_packet, totalpacketlen); if (cmd_send_packet) diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8192cu/rf.c b/drivers/net/wireless/realtek/rtlwifi/rtl8192cu/rf.c index f3a336e0ea98e977f0b3dd5dfd498101872d162e..d259794a308b9f6c1872e4030577db4d9726fcf4 100644 --- a/drivers/net/wireless/realtek/rtlwifi/rtl8192cu/rf.c +++ b/drivers/net/wireless/realtek/rtlwifi/rtl8192cu/rf.c @@ -42,12 +42,9 @@ void rtl92cu_phy_rf6052_set_cck_txpower(struct ieee80211_hw *hw, struct rtl_mac *mac = rtl_mac(rtl_priv(hw)); struct rtl_efuse *rtlefuse = rtl_efuse(rtl_priv(hw)); u32 tx_agc[2] = { 0, 0 }, tmpval = 0; - bool turbo_scanoff = false; u8 idx1, idx2; u8 *ptr; - if ((rtlefuse->eeprom_regulatory != 0) || (rtlefuse->external_pa)) - turbo_scanoff = true; if (mac->act_scanning) { tx_agc[RF90_PATH_A] = 0x3f3f3f3f; tx_agc[RF90_PATH_B] = 0x3f3f3f3f; diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/fw.c b/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/fw.c index 7c5b54b71a92f26cdabf9b1ba108024a8db65fd8..67305ce915ec4d5673709ce2f77af12f88884d64 100644 --- a/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/fw.c +++ b/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/fw.c @@ -744,6 +744,8 @@ void rtl92ee_set_fw_rsvdpagepkt(struct ieee80211_hw *hw, bool b_dl_finished) u1rsvdpageloc, 3); skb = dev_alloc_skb(totalpacketlen); + if (!skb) + return; skb_put_data(skb, &reserved_page_packet, totalpacketlen); rtstatus = rtl_cmd_send_packet(hw, skb); diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/trx.c b/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/trx.c index 09cf8180e4ff7d9645ca1f946e6e99d8f9c25c94..d297cfc0fd2bb0dfa383a10b80d3e6b8f85019a3 100644 --- a/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/trx.c +++ b/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/trx.c @@ -331,6 +331,7 @@ bool rtl92ee_rx_query_desc(struct ieee80211_hw *hw, struct rx_fwinfo *p_drvinfo; struct ieee80211_hdr *hdr; u32 phystatus = GET_RX_DESC_PHYST(pdesc); + u8 wake_match; if (GET_RX_STATUS_DESC_RPT_SEL(pdesc) == 0) status->packet_report_type = NORMAL_RX; @@ -350,18 +351,18 @@ bool rtl92ee_rx_query_desc(struct ieee80211_hw *hw, status->is_cck = RTL92EE_RX_HAL_IS_CCK_RATE(status->rate); status->macid = GET_RX_DESC_MACID(pdesc); - if (GET_RX_STATUS_DESC_MAGIC_MATCH(pdesc)) - status->wake_match = BIT(2); + if (GET_RX_STATUS_DESC_PATTERN_MATCH(pdesc)) + wake_match = BIT(2); else if (GET_RX_STATUS_DESC_MAGIC_MATCH(pdesc)) - status->wake_match = BIT(1); + wake_match = BIT(1); else if (GET_RX_STATUS_DESC_UNICAST_MATCH(pdesc)) - status->wake_match = BIT(0); + wake_match = BIT(0); else - status->wake_match = 0; - if (status->wake_match) + wake_match = 0; + if (wake_match) RT_TRACE(rtlpriv, COMP_RXDESC, DBG_LOUD, "GGGGGGGGGGGGGet Wakeup Packet!! WakeMatch=%d\n", - status->wake_match); + wake_match); rx_status->freq = hw->conf.chandef.chan->center_freq; rx_status->band = hw->conf.chandef.chan->band; diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8723ae/dm.c b/drivers/net/wireless/realtek/rtlwifi/rtl8723ae/dm.c index 514891ea2c648d162f537917f2f6df43a2c18de3..d8260c7afe09e62bdde33bf22fb9299f06244b7b 100644 --- a/drivers/net/wireless/realtek/rtlwifi/rtl8723ae/dm.c +++ b/drivers/net/wireless/realtek/rtlwifi/rtl8723ae/dm.c @@ -663,7 +663,7 @@ void rtl8723e_dm_init_rate_adaptive_mask(struct ieee80211_hw *hw) } -void rtl8723e_dm_refresh_rate_adaptive_mask(struct ieee80211_hw *hw) +static void rtl8723e_dm_refresh_rate_adaptive_mask(struct ieee80211_hw *hw) { struct rtl_priv *rtlpriv = rtl_priv(hw); struct rtl_hal *rtlhal = rtl_hal(rtl_priv(hw)); diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8723ae/fw.c b/drivers/net/wireless/realtek/rtlwifi/rtl8723ae/fw.c index be451a6f7dbe5cad54b0beb49f390904487b32c3..33481232fad01c8cdb842ab85dabff9e5a718044 100644 --- a/drivers/net/wireless/realtek/rtlwifi/rtl8723ae/fw.c +++ b/drivers/net/wireless/realtek/rtlwifi/rtl8723ae/fw.c @@ -448,6 +448,8 @@ void rtl8723e_set_fw_rsvdpagepkt(struct ieee80211_hw *hw, bool b_dl_finished) u1rsvdpageloc, 3); skb = dev_alloc_skb(totalpacketlen); + if (!skb) + return; skb_put_data(skb, &reserved_page_packet, totalpacketlen); rtstatus = rtl_cmd_send_packet(hw, skb); diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8723ae/hw.c b/drivers/net/wireless/realtek/rtlwifi/rtl8723ae/hw.c index 6bab162e1bb8d75e0562b20d3d518912f3c75c78..655460f61bbc341db842d602e10bc58d66e1562b 100644 --- a/drivers/net/wireless/realtek/rtlwifi/rtl8723ae/hw.c +++ b/drivers/net/wireless/realtek/rtlwifi/rtl8723ae/hw.c @@ -1675,6 +1675,7 @@ static void _rtl8723e_read_adapter_info(struct ieee80211_hw *hw, rtlhal->oem_id = RT_CID_819X_LENOVO; break; } + break; case 0x1025: rtlhal->oem_id = RT_CID_819X_ACER; break; diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8723be/fw.c b/drivers/net/wireless/realtek/rtlwifi/rtl8723be/fw.c index 4d7fa27f55caad0f5eb20f9e14cf61d583e70779..aa56058af56ef1e1ebb62d30bb3732deed4a910c 100644 --- a/drivers/net/wireless/realtek/rtlwifi/rtl8723be/fw.c +++ b/drivers/net/wireless/realtek/rtlwifi/rtl8723be/fw.c @@ -562,6 +562,8 @@ void rtl8723be_set_fw_rsvdpagepkt(struct ieee80211_hw *hw, u1rsvdpageloc, sizeof(u1rsvdpageloc)); skb = dev_alloc_skb(totalpacketlen); + if (!skb) + return; skb_put_data(skb, &reserved_page_packet, totalpacketlen); rtstatus = rtl_cmd_send_packet(hw, skb); diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8723be/trx.c b/drivers/net/wireless/realtek/rtlwifi/rtl8723be/trx.c index 9ada9a06c6ea0297662d19ec4d3366c30f8d00f7..d87ba03fe78f66a3f8641880f7a9552849297e21 100644 --- a/drivers/net/wireless/realtek/rtlwifi/rtl8723be/trx.c +++ b/drivers/net/wireless/realtek/rtlwifi/rtl8723be/trx.c @@ -300,7 +300,7 @@ bool rtl8723be_rx_query_desc(struct ieee80211_hw *hw, struct rtl_priv *rtlpriv = rtl_priv(hw); struct rx_fwinfo_8723be *p_drvinfo; struct ieee80211_hdr *hdr; - + u8 wake_match; u32 phystatus = GET_RX_DESC_PHYST(pdesc); status->length = (u16)GET_RX_DESC_PKT_LEN(pdesc); @@ -329,18 +329,18 @@ bool rtl8723be_rx_query_desc(struct ieee80211_hw *hw, status->packet_report_type = NORMAL_RX; - if (GET_RX_STATUS_DESC_MAGIC_MATCH(pdesc)) - status->wake_match = BIT(2); + if (GET_RX_STATUS_DESC_PATTERN_MATCH(pdesc)) + wake_match = BIT(2); else if (GET_RX_STATUS_DESC_MAGIC_MATCH(pdesc)) - status->wake_match = BIT(1); + wake_match = BIT(1); else if (GET_RX_STATUS_DESC_UNICAST_MATCH(pdesc)) - status->wake_match = BIT(0); + wake_match = BIT(0); else - status->wake_match = 0; - if (status->wake_match) + wake_match = 0; + if (wake_match) RT_TRACE(rtlpriv, COMP_RXDESC, DBG_LOUD, "GGGGGGGGGGGGGet Wakeup Packet!! WakeMatch=%d\n", - status->wake_match); + wake_match); rx_status->freq = hw->conf.chandef.chan->center_freq; rx_status->band = hw->conf.chandef.chan->band; diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/fw.c b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/fw.c index dc0eb692088f634f52765e8c9d2a68f939084a16..fe32d397d287558d206c412215b21904c3f1ce28 100644 --- a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/fw.c +++ b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/fw.c @@ -1623,6 +1623,8 @@ out: &reserved_page_packet_8812[0], totalpacketlen); skb = dev_alloc_skb(totalpacketlen); + if (!skb) + return; skb_put_data(skb, &reserved_page_packet_8812, totalpacketlen); rtstatus = rtl_cmd_send_packet(hw, skb); @@ -1759,6 +1761,8 @@ out: &reserved_page_packet_8821[0], totalpacketlen); skb = dev_alloc_skb(totalpacketlen); + if (!skb) + return; skb_put_data(skb, &reserved_page_packet_8821, totalpacketlen); rtstatus = rtl_cmd_send_packet(hw, skb); diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/trx.c b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/trx.c index db5e628b17edefa70f7de2585b02b183306d84b9..7b6faf38e09c7b35cd2b482c4e2f767fb2341712 100644 --- a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/trx.c +++ b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/trx.c @@ -436,7 +436,7 @@ bool rtl8821ae_rx_query_desc(struct ieee80211_hw *hw, struct rtl_priv *rtlpriv = rtl_priv(hw); struct rx_fwinfo_8821ae *p_drvinfo; struct ieee80211_hdr *hdr; - + u8 wake_match; u32 phystatus = GET_RX_DESC_PHYST(pdesc); status->length = (u16)GET_RX_DESC_PKT_LEN(pdesc); @@ -473,18 +473,18 @@ bool rtl8821ae_rx_query_desc(struct ieee80211_hw *hw, status->packet_report_type = NORMAL_RX; if (GET_RX_STATUS_DESC_PATTERN_MATCH(pdesc)) - status->wake_match = BIT(2); + wake_match = BIT(2); else if (GET_RX_STATUS_DESC_MAGIC_MATCH(pdesc)) - status->wake_match = BIT(1); + wake_match = BIT(1); else if (GET_RX_STATUS_DESC_UNICAST_MATCH(pdesc)) - status->wake_match = BIT(0); + wake_match = BIT(0); else - status->wake_match = 0; + wake_match = 0; - if (status->wake_match) + if (wake_match) RT_TRACE(rtlpriv, COMP_RXDESC, DBG_LOUD, "GGGGGGGGGGGGGet Wakeup Packet!! WakeMatch=%d\n", - status->wake_match); + wake_match); rx_status->freq = hw->conf.chandef.chan->center_freq; rx_status->band = hw->conf.chandef.chan->band; diff --git a/drivers/net/wireless/realtek/rtlwifi/wifi.h b/drivers/net/wireless/realtek/rtlwifi/wifi.h index e32e9ffa3192024c038a8f251587a7fe5dade21d..518aaa87536141b4e4915cae909bad1400f73236 100644 --- a/drivers/net/wireless/realtek/rtlwifi/wifi.h +++ b/drivers/net/wireless/realtek/rtlwifi/wifi.h @@ -2138,7 +2138,6 @@ struct rtl_stats { u8 packet_report_type; u32 macid; - u8 wake_match; u32 bt_rx_rssi_percentage; u32 macid_valid_entry[2]; }; diff --git a/drivers/net/wireless/realtek/rtw88/Kconfig b/drivers/net/wireless/realtek/rtw88/Kconfig new file mode 100644 index 0000000000000000000000000000000000000000..55b1bf3dd9b6676a4385771aadebda0b10fbe438 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/Kconfig @@ -0,0 +1,54 @@ +menuconfig RTW88 + tristate "Realtek 802.11ac wireless chips support" + depends on MAC80211 + help + This module adds support for mac80211-based wireless drivers that + enables Realtek IEEE 802.11ac wireless chipsets. + + If you choose to build a module, it'll be called rtw88. + +if RTW88 + +config RTW88_CORE + tristate + +config RTW88_PCI + tristate + +config RTW88_8822BE + bool "Realtek 8822BE PCI wireless network adapter" + depends on PCI + select RTW88_CORE + select RTW88_PCI + help + Select this option will enable support for 8822BE chipset + + 802.11ac PCIe wireless network adapter + +config RTW88_8822CE + bool "Realtek 8822CE PCI wireless network adapter" + depends on PCI + select RTW88_CORE + select RTW88_PCI + help + Select this option will enable support for 8822CE chipset + + 802.11ac PCIe wireless network adapter + +config RTW88_DEBUG + bool "Realtek rtw88 debug support" + depends on RTW88_CORE + help + Enable debug support + + If unsure, say Y to simplify debug problems + +config RTW88_DEBUGFS + bool "Realtek rtw88 debugfs support" + depends on RTW88_CORE + help + Enable debug support + + If unsure, say Y to simplify debug problems + +endif diff --git a/drivers/net/wireless/realtek/rtw88/Makefile b/drivers/net/wireless/realtek/rtw88/Makefile new file mode 100644 index 0000000000000000000000000000000000000000..e0bfefd154af8079d8467db1853a5cea4823afb5 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/Makefile @@ -0,0 +1,22 @@ +# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause + +obj-$(CONFIG_RTW88_CORE) += rtw88.o +rtw88-y += main.o \ + mac80211.o \ + util.o \ + debug.o \ + tx.o \ + rx.o \ + mac.o \ + phy.o \ + efuse.o \ + fw.o \ + ps.o \ + sec.o \ + regd.o + +rtw88-$(CONFIG_RTW88_8822BE) += rtw8822b.o rtw8822b_table.o +rtw88-$(CONFIG_RTW88_8822CE) += rtw8822c.o rtw8822c_table.o + +obj-$(CONFIG_RTW88_PCI) += rtwpci.o +rtwpci-objs := pci.o diff --git a/drivers/net/wireless/realtek/rtw88/debug.c b/drivers/net/wireless/realtek/rtw88/debug.c new file mode 100644 index 0000000000000000000000000000000000000000..f0ae26018f972818d79097b0d937559f02e5362d --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/debug.c @@ -0,0 +1,637 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#include +#include +#include "main.h" +#include "sec.h" +#include "fw.h" +#include "debug.h" + +#ifdef CONFIG_RTW88_DEBUGFS + +struct rtw_debugfs_priv { + struct rtw_dev *rtwdev; + int (*cb_read)(struct seq_file *m, void *v); + ssize_t (*cb_write)(struct file *filp, const char __user *buffer, + size_t count, loff_t *loff); + union { + u32 cb_data; + u8 *buf; + struct { + u32 page_offset; + u32 page_num; + } rsvd_page; + struct { + u8 rf_path; + u32 rf_addr; + u32 rf_mask; + }; + struct { + u32 addr; + u32 len; + } read_reg; + }; +}; + +static int rtw_debugfs_single_show(struct seq_file *m, void *v) +{ + struct rtw_debugfs_priv *debugfs_priv = m->private; + + return debugfs_priv->cb_read(m, v); +} + +static ssize_t rtw_debugfs_common_write(struct file *filp, + const char __user *buffer, + size_t count, loff_t *loff) +{ + struct rtw_debugfs_priv *debugfs_priv = filp->private_data; + + return debugfs_priv->cb_write(filp, buffer, count, loff); +} + +static ssize_t rtw_debugfs_single_write(struct file *filp, + const char __user *buffer, + size_t count, loff_t *loff) +{ + struct seq_file *seqpriv = (struct seq_file *)filp->private_data; + struct rtw_debugfs_priv *debugfs_priv = seqpriv->private; + + return debugfs_priv->cb_write(filp, buffer, count, loff); +} + +static int rtw_debugfs_single_open_rw(struct inode *inode, struct file *filp) +{ + return single_open(filp, rtw_debugfs_single_show, inode->i_private); +} + +static int rtw_debugfs_close(struct inode *inode, struct file *filp) +{ + return 0; +} + +static const struct file_operations file_ops_single_r = { + .owner = THIS_MODULE, + .open = rtw_debugfs_single_open_rw, + .read = seq_read, + .llseek = seq_lseek, + .release = seq_release, +}; + +static const struct file_operations file_ops_single_rw = { + .owner = THIS_MODULE, + .open = rtw_debugfs_single_open_rw, + .release = single_release, + .read = seq_read, + .llseek = seq_lseek, + .write = rtw_debugfs_single_write, +}; + +static const struct file_operations file_ops_common_write = { + .owner = THIS_MODULE, + .write = rtw_debugfs_common_write, + .open = simple_open, + .release = rtw_debugfs_close, +}; + +static int rtw_debugfs_get_read_reg(struct seq_file *m, void *v) +{ + struct rtw_debugfs_priv *debugfs_priv = m->private; + struct rtw_dev *rtwdev = debugfs_priv->rtwdev; + u32 val, len, addr; + + len = debugfs_priv->read_reg.len; + addr = debugfs_priv->read_reg.addr; + switch (len) { + case 1: + val = rtw_read8(rtwdev, addr); + seq_printf(m, "reg 0x%03x: 0x%02x\n", addr, val); + break; + case 2: + val = rtw_read16(rtwdev, addr); + seq_printf(m, "reg 0x%03x: 0x%04x\n", addr, val); + break; + case 4: + val = rtw_read32(rtwdev, addr); + seq_printf(m, "reg 0x%03x: 0x%08x\n", addr, val); + break; + } + return 0; +} + +static int rtw_debugfs_get_rf_read(struct seq_file *m, void *v) +{ + struct rtw_debugfs_priv *debugfs_priv = m->private; + struct rtw_dev *rtwdev = debugfs_priv->rtwdev; + u32 val, addr, mask; + u8 path; + + path = debugfs_priv->rf_path; + addr = debugfs_priv->rf_addr; + mask = debugfs_priv->rf_mask; + + val = rtw_read_rf(rtwdev, path, addr, mask); + + seq_printf(m, "rf_read path:%d addr:0x%08x mask:0x%08x val=0x%08x\n", + path, addr, mask, val); + + return 0; +} + +static int rtw_debugfs_copy_from_user(char tmp[], int size, + const char __user *buffer, size_t count, + int num) +{ + int tmp_len; + + if (count < num) + return -EFAULT; + + tmp_len = (count > size - 1 ? size - 1 : count); + + if (!buffer || copy_from_user(tmp, buffer, tmp_len)) + return count; + + tmp[tmp_len] = '\0'; + + return 0; +} + +static ssize_t rtw_debugfs_set_read_reg(struct file *filp, + const char __user *buffer, + size_t count, loff_t *loff) +{ + struct seq_file *seqpriv = (struct seq_file *)filp->private_data; + struct rtw_debugfs_priv *debugfs_priv = seqpriv->private; + struct rtw_dev *rtwdev = debugfs_priv->rtwdev; + char tmp[32 + 1]; + u32 addr, len; + int num; + + rtw_debugfs_copy_from_user(tmp, sizeof(tmp), buffer, count, 2); + + num = sscanf(tmp, "%x %x", &addr, &len); + + if (num != 2) + return count; + + if (len != 1 && len != 2 && len != 4) { + rtw_warn(rtwdev, "read reg setting wrong len\n"); + return -EINVAL; + } + debugfs_priv->read_reg.addr = addr; + debugfs_priv->read_reg.len = len; + + return count; +} + +static int rtw_debugfs_get_dump_cam(struct seq_file *m, void *v) +{ + struct rtw_debugfs_priv *debugfs_priv = m->private; + struct rtw_dev *rtwdev = debugfs_priv->rtwdev; + u32 val, command; + u32 hw_key_idx = debugfs_priv->cb_data << RTW_SEC_CAM_ENTRY_SHIFT; + u32 read_cmd = RTW_SEC_CMD_POLLING; + int i; + + seq_printf(m, "cam entry%d\n", debugfs_priv->cb_data); + seq_puts(m, "0x0 0x1 0x2 0x3 "); + seq_puts(m, "0x4 0x5\n"); + mutex_lock(&rtwdev->mutex); + for (i = 0; i <= 5; i++) { + command = read_cmd | (hw_key_idx + i); + rtw_write32(rtwdev, RTW_SEC_CMD_REG, command); + val = rtw_read32(rtwdev, RTW_SEC_READ_REG); + seq_printf(m, "%8.8x", val); + if (i < 2) + seq_puts(m, " "); + } + seq_puts(m, "\n"); + mutex_unlock(&rtwdev->mutex); + return 0; +} + +static int rtw_debugfs_get_rsvd_page(struct seq_file *m, void *v) +{ + struct rtw_debugfs_priv *debugfs_priv = m->private; + struct rtw_dev *rtwdev = debugfs_priv->rtwdev; + u8 page_size = rtwdev->chip->page_size; + u32 buf_size = debugfs_priv->rsvd_page.page_num * page_size; + u32 offset = debugfs_priv->rsvd_page.page_offset * page_size; + u8 *buf; + int i; + int ret; + + buf = vzalloc(buf_size); + if (!buf) + return -ENOMEM; + + ret = rtw_dump_drv_rsvd_page(rtwdev, offset, buf_size, (u32 *)buf); + if (ret) { + rtw_err(rtwdev, "failed to dump rsvd page\n"); + vfree(buf); + return ret; + } + + for (i = 0 ; i < buf_size ; i += 8) { + if (i % page_size == 0) + seq_printf(m, "PAGE %d\n", (i + offset) / page_size); + seq_printf(m, "%2.2x %2.2x %2.2x %2.2x %2.2x %2.2x %2.2x %2.2x\n", + *(buf + i), *(buf + i + 1), + *(buf + i + 2), *(buf + i + 3), + *(buf + i + 4), *(buf + i + 5), + *(buf + i + 6), *(buf + i + 7)); + } + vfree(buf); + + return 0; +} + +static ssize_t rtw_debugfs_set_rsvd_page(struct file *filp, + const char __user *buffer, + size_t count, loff_t *loff) +{ + struct seq_file *seqpriv = (struct seq_file *)filp->private_data; + struct rtw_debugfs_priv *debugfs_priv = seqpriv->private; + struct rtw_dev *rtwdev = debugfs_priv->rtwdev; + char tmp[32 + 1]; + u32 offset, page_num; + int num; + + rtw_debugfs_copy_from_user(tmp, sizeof(tmp), buffer, count, 2); + + num = sscanf(tmp, "%d %d", &offset, &page_num); + + if (num != 2) { + rtw_warn(rtwdev, "invalid arguments\n"); + return num; + } + + debugfs_priv->rsvd_page.page_offset = offset; + debugfs_priv->rsvd_page.page_num = page_num; + + return count; +} + +static ssize_t rtw_debugfs_set_single_input(struct file *filp, + const char __user *buffer, + size_t count, loff_t *loff) +{ + struct seq_file *seqpriv = (struct seq_file *)filp->private_data; + struct rtw_debugfs_priv *debugfs_priv = seqpriv->private; + struct rtw_dev *rtwdev = debugfs_priv->rtwdev; + char tmp[32 + 1]; + u32 input; + int num; + + rtw_debugfs_copy_from_user(tmp, sizeof(tmp), buffer, count, 1); + + num = kstrtoint(tmp, 0, &input); + + if (num) { + rtw_warn(rtwdev, "kstrtoint failed\n"); + return num; + } + + debugfs_priv->cb_data = input; + + return count; +} + +static ssize_t rtw_debugfs_set_write_reg(struct file *filp, + const char __user *buffer, + size_t count, loff_t *loff) +{ + struct rtw_debugfs_priv *debugfs_priv = filp->private_data; + struct rtw_dev *rtwdev = debugfs_priv->rtwdev; + char tmp[32 + 1]; + u32 addr, val, len; + int num; + + rtw_debugfs_copy_from_user(tmp, sizeof(tmp), buffer, count, 3); + + /* write BB/MAC register */ + num = sscanf(tmp, "%x %x %x", &addr, &val, &len); + + if (num != 3) + return count; + + switch (len) { + case 1: + rtw_dbg(rtwdev, RTW_DBG_DEBUGFS, + "reg write8 0x%03x: 0x%08x\n", addr, val); + rtw_write8(rtwdev, addr, (u8)val); + break; + case 2: + rtw_dbg(rtwdev, RTW_DBG_DEBUGFS, + "reg write16 0x%03x: 0x%08x\n", addr, val); + rtw_write16(rtwdev, addr, (u16)val); + break; + case 4: + rtw_dbg(rtwdev, RTW_DBG_DEBUGFS, + "reg write32 0x%03x: 0x%08x\n", addr, val); + rtw_write32(rtwdev, addr, (u32)val); + break; + default: + rtw_dbg(rtwdev, RTW_DBG_DEBUGFS, + "error write length = %d\n", len); + break; + } + + return count; +} + +static ssize_t rtw_debugfs_set_rf_write(struct file *filp, + const char __user *buffer, + size_t count, loff_t *loff) +{ + struct rtw_debugfs_priv *debugfs_priv = filp->private_data; + struct rtw_dev *rtwdev = debugfs_priv->rtwdev; + char tmp[32 + 1]; + u32 path, addr, mask, val; + int num; + + rtw_debugfs_copy_from_user(tmp, sizeof(tmp), buffer, count, 4); + + num = sscanf(tmp, "%x %x %x %x", &path, &addr, &mask, &val); + + if (num != 4) { + rtw_warn(rtwdev, "invalid args, [path] [addr] [mask] [val]\n"); + return count; + } + + rtw_write_rf(rtwdev, path, addr, mask, val); + rtw_dbg(rtwdev, RTW_DBG_DEBUGFS, + "write_rf path:%d addr:0x%08x mask:0x%08x, val:0x%08x\n", + path, addr, mask, val); + + return count; +} + +static ssize_t rtw_debugfs_set_rf_read(struct file *filp, + const char __user *buffer, + size_t count, loff_t *loff) +{ + struct seq_file *seqpriv = (struct seq_file *)filp->private_data; + struct rtw_debugfs_priv *debugfs_priv = seqpriv->private; + struct rtw_dev *rtwdev = debugfs_priv->rtwdev; + char tmp[32 + 1]; + u32 path, addr, mask; + int num; + + rtw_debugfs_copy_from_user(tmp, sizeof(tmp), buffer, count, 3); + + num = sscanf(tmp, "%x %x %x", &path, &addr, &mask); + + if (num != 3) { + rtw_warn(rtwdev, "invalid args, [path] [addr] [mask] [val]\n"); + return count; + } + + debugfs_priv->rf_path = path; + debugfs_priv->rf_addr = addr; + debugfs_priv->rf_mask = mask; + + return count; +} + +static int rtw_debug_get_mac_page(struct seq_file *m, void *v) +{ + struct rtw_debugfs_priv *debugfs_priv = m->private; + struct rtw_dev *rtwdev = debugfs_priv->rtwdev; + u32 val; + u32 page = debugfs_priv->cb_data; + int i, n; + int max = 0xff; + + val = rtw_read32(rtwdev, debugfs_priv->cb_data); + for (n = 0; n <= max; ) { + seq_printf(m, "\n%8.8x ", n + page); + for (i = 0; i < 4 && n <= max; i++, n += 4) + seq_printf(m, "%8.8x ", + rtw_read32(rtwdev, (page | n))); + } + seq_puts(m, "\n"); + return 0; +} + +static int rtw_debug_get_bb_page(struct seq_file *m, void *v) +{ + struct rtw_debugfs_priv *debugfs_priv = m->private; + struct rtw_dev *rtwdev = debugfs_priv->rtwdev; + u32 val; + u32 page = debugfs_priv->cb_data; + int i, n; + int max = 0xff; + + val = rtw_read32(rtwdev, debugfs_priv->cb_data); + for (n = 0; n <= max; ) { + seq_printf(m, "\n%8.8x ", n + page); + for (i = 0; i < 4 && n <= max; i++, n += 4) + seq_printf(m, "%8.8x ", + rtw_read32(rtwdev, (page | n))); + } + seq_puts(m, "\n"); + return 0; +} + +static int rtw_debug_get_rf_dump(struct seq_file *m, void *v) +{ + struct rtw_debugfs_priv *debugfs_priv = m->private; + struct rtw_dev *rtwdev = debugfs_priv->rtwdev; + u32 addr, offset, data; + u8 path; + + for (path = 0; path < rtwdev->hal.rf_path_num; path++) { + seq_printf(m, "RF path:%d\n", path); + for (addr = 0; addr < 0x100; addr += 4) { + seq_printf(m, "%8.8x ", addr); + for (offset = 0; offset < 4; offset++) { + data = rtw_read_rf(rtwdev, path, addr + offset, + 0xffffffff); + seq_printf(m, "%8.8x ", data); + } + seq_puts(m, "\n"); + } + seq_puts(m, "\n"); + } + + return 0; +} + +#define rtw_debug_impl_mac(page, addr) \ +static struct rtw_debugfs_priv rtw_debug_priv_mac_ ##page = { \ + .cb_read = rtw_debug_get_mac_page, \ + .cb_data = addr, \ +} + +rtw_debug_impl_mac(0, 0x0000); +rtw_debug_impl_mac(1, 0x0100); +rtw_debug_impl_mac(2, 0x0200); +rtw_debug_impl_mac(3, 0x0300); +rtw_debug_impl_mac(4, 0x0400); +rtw_debug_impl_mac(5, 0x0500); +rtw_debug_impl_mac(6, 0x0600); +rtw_debug_impl_mac(7, 0x0700); +rtw_debug_impl_mac(10, 0x1000); +rtw_debug_impl_mac(11, 0x1100); +rtw_debug_impl_mac(12, 0x1200); +rtw_debug_impl_mac(13, 0x1300); +rtw_debug_impl_mac(14, 0x1400); +rtw_debug_impl_mac(15, 0x1500); +rtw_debug_impl_mac(16, 0x1600); +rtw_debug_impl_mac(17, 0x1700); + +#define rtw_debug_impl_bb(page, addr) \ +static struct rtw_debugfs_priv rtw_debug_priv_bb_ ##page = { \ + .cb_read = rtw_debug_get_bb_page, \ + .cb_data = addr, \ +} + +rtw_debug_impl_bb(8, 0x0800); +rtw_debug_impl_bb(9, 0x0900); +rtw_debug_impl_bb(a, 0x0a00); +rtw_debug_impl_bb(b, 0x0b00); +rtw_debug_impl_bb(c, 0x0c00); +rtw_debug_impl_bb(d, 0x0d00); +rtw_debug_impl_bb(e, 0x0e00); +rtw_debug_impl_bb(f, 0x0f00); +rtw_debug_impl_bb(18, 0x1800); +rtw_debug_impl_bb(19, 0x1900); +rtw_debug_impl_bb(1a, 0x1a00); +rtw_debug_impl_bb(1b, 0x1b00); +rtw_debug_impl_bb(1c, 0x1c00); +rtw_debug_impl_bb(1d, 0x1d00); +rtw_debug_impl_bb(1e, 0x1e00); +rtw_debug_impl_bb(1f, 0x1f00); +rtw_debug_impl_bb(2c, 0x2c00); +rtw_debug_impl_bb(2d, 0x2d00); +rtw_debug_impl_bb(40, 0x4000); +rtw_debug_impl_bb(41, 0x4100); + +static struct rtw_debugfs_priv rtw_debug_priv_rf_dump = { + .cb_read = rtw_debug_get_rf_dump, +}; + +static struct rtw_debugfs_priv rtw_debug_priv_write_reg = { + .cb_write = rtw_debugfs_set_write_reg, +}; + +static struct rtw_debugfs_priv rtw_debug_priv_rf_write = { + .cb_write = rtw_debugfs_set_rf_write, +}; + +static struct rtw_debugfs_priv rtw_debug_priv_rf_read = { + .cb_write = rtw_debugfs_set_rf_read, + .cb_read = rtw_debugfs_get_rf_read, +}; + +static struct rtw_debugfs_priv rtw_debug_priv_read_reg = { + .cb_write = rtw_debugfs_set_read_reg, + .cb_read = rtw_debugfs_get_read_reg, +}; + +static struct rtw_debugfs_priv rtw_debug_priv_dump_cam = { + .cb_write = rtw_debugfs_set_single_input, + .cb_read = rtw_debugfs_get_dump_cam, +}; + +static struct rtw_debugfs_priv rtw_debug_priv_rsvd_page = { + .cb_write = rtw_debugfs_set_rsvd_page, + .cb_read = rtw_debugfs_get_rsvd_page, +}; + +#define rtw_debugfs_add_core(name, mode, fopname, parent) \ + do { \ + rtw_debug_priv_ ##name.rtwdev = rtwdev; \ + if (!debugfs_create_file(#name, mode, \ + parent, &rtw_debug_priv_ ##name,\ + &file_ops_ ##fopname)) \ + pr_debug("Unable to initialize debugfs:%s\n", \ + #name); \ + } while (0) + +#define rtw_debugfs_add_w(name) \ + rtw_debugfs_add_core(name, S_IFREG | 0222, common_write, debugfs_topdir) +#define rtw_debugfs_add_rw(name) \ + rtw_debugfs_add_core(name, S_IFREG | 0666, single_rw, debugfs_topdir) +#define rtw_debugfs_add_r(name) \ + rtw_debugfs_add_core(name, S_IFREG | 0444, single_r, debugfs_topdir) + +void rtw_debugfs_init(struct rtw_dev *rtwdev) +{ + struct dentry *debugfs_topdir = rtwdev->debugfs; + + debugfs_topdir = debugfs_create_dir("rtw88", + rtwdev->hw->wiphy->debugfsdir); + rtw_debugfs_add_w(write_reg); + rtw_debugfs_add_rw(read_reg); + rtw_debugfs_add_w(rf_write); + rtw_debugfs_add_rw(rf_read); + rtw_debugfs_add_rw(dump_cam); + rtw_debugfs_add_rw(rsvd_page); + rtw_debugfs_add_r(mac_0); + rtw_debugfs_add_r(mac_1); + rtw_debugfs_add_r(mac_2); + rtw_debugfs_add_r(mac_3); + rtw_debugfs_add_r(mac_4); + rtw_debugfs_add_r(mac_5); + rtw_debugfs_add_r(mac_6); + rtw_debugfs_add_r(mac_7); + rtw_debugfs_add_r(bb_8); + rtw_debugfs_add_r(bb_9); + rtw_debugfs_add_r(bb_a); + rtw_debugfs_add_r(bb_b); + rtw_debugfs_add_r(bb_c); + rtw_debugfs_add_r(bb_d); + rtw_debugfs_add_r(bb_e); + rtw_debugfs_add_r(bb_f); + rtw_debugfs_add_r(mac_10); + rtw_debugfs_add_r(mac_11); + rtw_debugfs_add_r(mac_12); + rtw_debugfs_add_r(mac_13); + rtw_debugfs_add_r(mac_14); + rtw_debugfs_add_r(mac_15); + rtw_debugfs_add_r(mac_16); + rtw_debugfs_add_r(mac_17); + rtw_debugfs_add_r(bb_18); + rtw_debugfs_add_r(bb_19); + rtw_debugfs_add_r(bb_1a); + rtw_debugfs_add_r(bb_1b); + rtw_debugfs_add_r(bb_1c); + rtw_debugfs_add_r(bb_1d); + rtw_debugfs_add_r(bb_1e); + rtw_debugfs_add_r(bb_1f); + if (rtwdev->chip->id == RTW_CHIP_TYPE_8822C) { + rtw_debugfs_add_r(bb_2c); + rtw_debugfs_add_r(bb_2d); + rtw_debugfs_add_r(bb_40); + rtw_debugfs_add_r(bb_41); + } + rtw_debugfs_add_r(rf_dump); +} + +#endif /* CONFIG_RTW88_DEBUGFS */ + +#ifdef CONFIG_RTW88_DEBUG + +void __rtw_dbg(struct rtw_dev *rtwdev, enum rtw_debug_mask mask, + const char *fmt, ...) +{ + struct va_format vaf = { + .fmt = fmt, + }; + va_list args; + + va_start(args, fmt); + vaf.va = &args; + + if (rtw_debug_mask & mask) + dev_printk(KERN_DEBUG, rtwdev->dev, "%pV", &vaf); + + va_end(args); +} +EXPORT_SYMBOL(__rtw_dbg); + +#endif /* CONFIG_RTW88_DEBUG */ diff --git a/drivers/net/wireless/realtek/rtw88/debug.h b/drivers/net/wireless/realtek/rtw88/debug.h new file mode 100644 index 0000000000000000000000000000000000000000..45851cbbd2abf941358293c94724fca3ce3ea3c1 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/debug.h @@ -0,0 +1,52 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#ifndef __RTW_DEBUG_H +#define __RTW_DEBUG_H + +enum rtw_debug_mask { + RTW_DBG_PCI = 0x00000001, + RTW_DBG_TX = 0x00000002, + RTW_DBG_RX = 0x00000004, + RTW_DBG_PHY = 0x00000008, + RTW_DBG_FW = 0x00000010, + RTW_DBG_EFUSE = 0x00000020, + RTW_DBG_COEX = 0x00000040, + RTW_DBG_RFK = 0x00000080, + RTW_DBG_REGD = 0x00000100, + RTW_DBG_DEBUGFS = 0x00000200, + + RTW_DBG_ALL = 0xffffffff +}; + +#ifdef CONFIG_RTW88_DEBUGFS + +void rtw_debugfs_init(struct rtw_dev *rtwdev); + +#else + +static inline void rtw_debugfs_init(struct rtw_dev *rtwdev) {} + +#endif /* CONFIG_RTW88_DEBUGFS */ + +#ifdef CONFIG_RTW88_DEBUG + +__printf(3, 4) +void __rtw_dbg(struct rtw_dev *rtwdev, enum rtw_debug_mask mask, + const char *fmt, ...); + +#define rtw_dbg(rtwdev, a...) __rtw_dbg(rtwdev, ##a) + +#else + +static inline void rtw_dbg(struct rtw_dev *rtwdev, enum rtw_debug_mask mask, + const char *fmt, ...) {} + +#endif /* CONFIG_RTW88_DEBUG */ + +#define rtw_info(rtwdev, a...) dev_info(rtwdev->dev, ##a) +#define rtw_warn(rtwdev, a...) dev_warn(rtwdev->dev, ##a) +#define rtw_err(rtwdev, a...) dev_err(rtwdev->dev, ##a) + +#endif diff --git a/drivers/net/wireless/realtek/rtw88/efuse.c b/drivers/net/wireless/realtek/rtw88/efuse.c new file mode 100644 index 0000000000000000000000000000000000000000..212c8376a8c98688f7ca91a52509d6973c6b7a49 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/efuse.c @@ -0,0 +1,160 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#include "main.h" +#include "efuse.h" +#include "reg.h" +#include "debug.h" + +#define RTW_EFUSE_BANK_WIFI 0x0 + +static void switch_efuse_bank(struct rtw_dev *rtwdev) +{ + rtw_write32_mask(rtwdev, REG_LDO_EFUSE_CTRL, BIT_MASK_EFUSE_BANK_SEL, + RTW_EFUSE_BANK_WIFI); +} + +#define invalid_efuse_header(hdr1, hdr2) \ + ((hdr1) == 0xff || (((hdr1) & 0x1f) == 0xf && (hdr2) == 0xff)) +#define invalid_efuse_content(word_en, i) \ + (((word_en) & BIT(i)) != 0x0) +#define get_efuse_blk_idx_2_byte(hdr1, hdr2) \ + ((((hdr2) & 0xf0) >> 1) | (((hdr1) >> 5) & 0x07)) +#define get_efuse_blk_idx_1_byte(hdr1) \ + (((hdr1) & 0xf0) >> 4) +#define block_idx_to_logical_idx(blk_idx, i) \ + (((blk_idx) << 3) + ((i) << 1)) + +/* efuse header format + * + * | 7 5 4 0 | 7 4 3 0 | 15 8 7 0 | + * block[2:0] 0 1111 block[6:3] word_en[3:0] byte0 byte1 + * | header 1 (optional) | header 2 | word N | + * + * word_en: 4 bits each word. 0 -> write; 1 -> not write + * N: 1~4, depends on word_en + */ +static int rtw_dump_logical_efuse_map(struct rtw_dev *rtwdev, u8 *phy_map, + u8 *log_map) +{ + u32 physical_size = rtwdev->efuse.physical_size; + u32 protect_size = rtwdev->efuse.protect_size; + u32 logical_size = rtwdev->efuse.logical_size; + u32 phy_idx, log_idx; + u8 hdr1, hdr2; + u8 blk_idx; + u8 word_en; + int i; + + for (phy_idx = 0; phy_idx < physical_size - protect_size;) { + hdr1 = phy_map[phy_idx]; + hdr2 = phy_map[phy_idx + 1]; + if (invalid_efuse_header(hdr1, hdr2)) + break; + + if ((hdr1 & 0x1f) == 0xf) { + /* 2-byte header format */ + blk_idx = get_efuse_blk_idx_2_byte(hdr1, hdr2); + word_en = hdr2 & 0xf; + phy_idx += 2; + } else { + /* 1-byte header format */ + blk_idx = get_efuse_blk_idx_1_byte(hdr1); + word_en = hdr1 & 0xf; + phy_idx += 1; + } + + for (i = 0; i < 4; i++) { + if (invalid_efuse_content(word_en, i)) + continue; + + log_idx = block_idx_to_logical_idx(blk_idx, i); + if (phy_idx + 1 > physical_size - protect_size || + log_idx + 1 > logical_size) + return -EINVAL; + + log_map[log_idx] = phy_map[phy_idx]; + log_map[log_idx + 1] = phy_map[phy_idx + 1]; + phy_idx += 2; + } + } + return 0; +} + +static int rtw_dump_physical_efuse_map(struct rtw_dev *rtwdev, u8 *map) +{ + struct rtw_chip_info *chip = rtwdev->chip; + u32 size = rtwdev->efuse.physical_size; + u32 efuse_ctl; + u32 addr; + u32 cnt; + + switch_efuse_bank(rtwdev); + + /* disable 2.5V LDO */ + chip->ops->cfg_ldo25(rtwdev, false); + + efuse_ctl = rtw_read32(rtwdev, REG_EFUSE_CTRL); + + for (addr = 0; addr < size; addr++) { + efuse_ctl &= ~(BIT_MASK_EF_DATA | BITS_EF_ADDR); + efuse_ctl |= (addr & BIT_MASK_EF_ADDR) << BIT_SHIFT_EF_ADDR; + rtw_write32(rtwdev, REG_EFUSE_CTRL, efuse_ctl & (~BIT_EF_FLAG)); + + cnt = 1000000; + do { + udelay(1); + efuse_ctl = rtw_read32(rtwdev, REG_EFUSE_CTRL); + if (--cnt == 0) + return -EBUSY; + } while (!(efuse_ctl & BIT_EF_FLAG)); + + *(map + addr) = (u8)(efuse_ctl & BIT_MASK_EF_DATA); + } + + return 0; +} + +int rtw_parse_efuse_map(struct rtw_dev *rtwdev) +{ + struct rtw_chip_info *chip = rtwdev->chip; + struct rtw_efuse *efuse = &rtwdev->efuse; + u32 phy_size = efuse->physical_size; + u32 log_size = efuse->logical_size; + u8 *phy_map = NULL; + u8 *log_map = NULL; + int ret = 0; + + phy_map = kmalloc(phy_size, GFP_KERNEL); + log_map = kmalloc(log_size, GFP_KERNEL); + if (!phy_map || !log_map) { + ret = -ENOMEM; + goto out_free; + } + + ret = rtw_dump_physical_efuse_map(rtwdev, phy_map); + if (ret) { + rtw_err(rtwdev, "failed to dump efuse physical map\n"); + goto out_free; + } + + memset(log_map, 0xff, log_size); + ret = rtw_dump_logical_efuse_map(rtwdev, phy_map, log_map); + if (ret) { + rtw_err(rtwdev, "failed to dump efuse logical map\n"); + goto out_free; + } + + ret = chip->ops->read_efuse(rtwdev, log_map); + if (ret) { + rtw_err(rtwdev, "failed to read efuse map\n"); + goto out_free; + } + +out_free: + kfree(log_map); + kfree(phy_map); + + return ret; +} diff --git a/drivers/net/wireless/realtek/rtw88/efuse.h b/drivers/net/wireless/realtek/rtw88/efuse.h new file mode 100644 index 0000000000000000000000000000000000000000..115bbe85946ac6c643926c725d9bda49da03ff9a --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/efuse.h @@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#ifndef __RTW_EFUSE_H__ +#define __RTW_EFUSE_H__ + +#define EFUSE_HW_CAP_IGNORE 0 +#define EFUSE_HW_CAP_PTCL_VHT 3 +#define EFUSE_HW_CAP_SUPP_BW80 7 +#define EFUSE_HW_CAP_SUPP_BW40 6 + +#define GET_EFUSE_HW_CAP_HCI(hw_cap) \ + le32_get_bits(*((__le32 *)(hw_cap) + 0x01), GENMASK(3, 0)) +#define GET_EFUSE_HW_CAP_BW(hw_cap) \ + le32_get_bits(*((__le32 *)(hw_cap) + 0x01), GENMASK(18, 16)) +#define GET_EFUSE_HW_CAP_NSS(hw_cap) \ + le32_get_bits(*((__le32 *)(hw_cap) + 0x01), GENMASK(20, 19)) +#define GET_EFUSE_HW_CAP_ANT_NUM(hw_cap) \ + le32_get_bits(*((__le32 *)(hw_cap) + 0x01), GENMASK(23, 21)) +#define GET_EFUSE_HW_CAP_PTCL(hw_cap) \ + le32_get_bits(*((__le32 *)(hw_cap) + 0x01), GENMASK(27, 26)) + +int rtw_parse_efuse_map(struct rtw_dev *rtwdev); + +#endif diff --git a/drivers/net/wireless/realtek/rtw88/fw.c b/drivers/net/wireless/realtek/rtw88/fw.c new file mode 100644 index 0000000000000000000000000000000000000000..cf4265cda22468c9b99d2e14c7c3de8647d92849 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/fw.c @@ -0,0 +1,633 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#include "main.h" +#include "fw.h" +#include "tx.h" +#include "reg.h" +#include "debug.h" + +void rtw_fw_c2h_cmd_handle_ext(struct rtw_dev *rtwdev, struct sk_buff *skb) +{ + struct rtw_c2h_cmd *c2h; + u8 sub_cmd_id; + + c2h = get_c2h_from_skb(skb); + sub_cmd_id = c2h->payload[0]; + + switch (sub_cmd_id) { + case C2H_CCX_RPT: + rtw_tx_report_handle(rtwdev, skb); + break; + default: + break; + } +} + +void rtw_fw_c2h_cmd_handle(struct rtw_dev *rtwdev, struct sk_buff *skb) +{ + struct rtw_c2h_cmd *c2h; + u32 pkt_offset; + u8 len; + + pkt_offset = *((u32 *)skb->cb); + c2h = (struct rtw_c2h_cmd *)(skb->data + pkt_offset); + len = skb->len - pkt_offset - 2; + + rtw_dbg(rtwdev, RTW_DBG_FW, "recv C2H, id=0x%02x, seq=0x%02x, len=%d\n", + c2h->id, c2h->seq, len); + + switch (c2h->id) { + case C2H_HALMAC: + rtw_fw_c2h_cmd_handle_ext(rtwdev, skb); + break; + default: + break; + } +} + +void rtw_fw_send_h2c_command(struct rtw_dev *rtwdev, u8 *h2c) +{ + u8 box; + u8 box_state; + u32 box_reg, box_ex_reg; + u32 h2c_wait; + int idx; + + rtw_dbg(rtwdev, RTW_DBG_FW, + "send H2C content %02x%02x%02x%02x %02x%02x%02x%02x\n", + h2c[3], h2c[2], h2c[1], h2c[0], + h2c[7], h2c[6], h2c[5], h2c[4]); + + spin_lock(&rtwdev->h2c.lock); + + box = rtwdev->h2c.last_box_num; + switch (box) { + case 0: + box_reg = REG_HMEBOX0; + box_ex_reg = REG_HMEBOX0_EX; + break; + case 1: + box_reg = REG_HMEBOX1; + box_ex_reg = REG_HMEBOX1_EX; + break; + case 2: + box_reg = REG_HMEBOX2; + box_ex_reg = REG_HMEBOX2_EX; + break; + case 3: + box_reg = REG_HMEBOX3; + box_ex_reg = REG_HMEBOX3_EX; + break; + default: + WARN(1, "invalid h2c mail box number\n"); + goto out; + } + + h2c_wait = 20; + do { + box_state = rtw_read8(rtwdev, REG_HMETFR); + } while ((box_state >> box) & 0x1 && --h2c_wait > 0); + + if (!h2c_wait) { + rtw_err(rtwdev, "failed to send h2c command\n"); + goto out; + } + + for (idx = 0; idx < 4; idx++) + rtw_write8(rtwdev, box_reg + idx, h2c[idx]); + for (idx = 0; idx < 4; idx++) + rtw_write8(rtwdev, box_ex_reg + idx, h2c[idx + 4]); + + if (++rtwdev->h2c.last_box_num >= 4) + rtwdev->h2c.last_box_num = 0; + +out: + spin_unlock(&rtwdev->h2c.lock); +} + +static void rtw_fw_send_h2c_packet(struct rtw_dev *rtwdev, u8 *h2c_pkt) +{ + int ret; + + spin_lock(&rtwdev->h2c.lock); + + FW_OFFLOAD_H2C_SET_SEQ_NUM(h2c_pkt, rtwdev->h2c.seq); + ret = rtw_hci_write_data_h2c(rtwdev, h2c_pkt, H2C_PKT_SIZE); + if (ret) + rtw_err(rtwdev, "failed to send h2c packet\n"); + rtwdev->h2c.seq++; + + spin_unlock(&rtwdev->h2c.lock); +} + +void +rtw_fw_send_general_info(struct rtw_dev *rtwdev) +{ + struct rtw_fifo_conf *fifo = &rtwdev->fifo; + u8 h2c_pkt[H2C_PKT_SIZE] = {0}; + u16 total_size = H2C_PKT_HDR_SIZE + 4; + + rtw_h2c_pkt_set_header(h2c_pkt, H2C_PKT_GENERAL_INFO); + + SET_PKT_H2C_TOTAL_LEN(h2c_pkt, total_size); + + GENERAL_INFO_SET_FW_TX_BOUNDARY(h2c_pkt, + fifo->rsvd_fw_txbuf_addr - + fifo->rsvd_boundary); + + rtw_fw_send_h2c_packet(rtwdev, h2c_pkt); +} + +void +rtw_fw_send_phydm_info(struct rtw_dev *rtwdev) +{ + struct rtw_hal *hal = &rtwdev->hal; + struct rtw_efuse *efuse = &rtwdev->efuse; + u8 h2c_pkt[H2C_PKT_SIZE] = {0}; + u16 total_size = H2C_PKT_HDR_SIZE + 8; + u8 fw_rf_type = 0; + + if (hal->rf_type == RF_1T1R) + fw_rf_type = FW_RF_1T1R; + else if (hal->rf_type == RF_2T2R) + fw_rf_type = FW_RF_2T2R; + + rtw_h2c_pkt_set_header(h2c_pkt, H2C_PKT_PHYDM_INFO); + + SET_PKT_H2C_TOTAL_LEN(h2c_pkt, total_size); + PHYDM_INFO_SET_REF_TYPE(h2c_pkt, efuse->rfe_option); + PHYDM_INFO_SET_RF_TYPE(h2c_pkt, fw_rf_type); + PHYDM_INFO_SET_CUT_VER(h2c_pkt, hal->cut_version); + PHYDM_INFO_SET_RX_ANT_STATUS(h2c_pkt, hal->antenna_tx); + PHYDM_INFO_SET_TX_ANT_STATUS(h2c_pkt, hal->antenna_rx); + + rtw_fw_send_h2c_packet(rtwdev, h2c_pkt); +} + +void rtw_fw_do_iqk(struct rtw_dev *rtwdev, struct rtw_iqk_para *para) +{ + u8 h2c_pkt[H2C_PKT_SIZE] = {0}; + u16 total_size = H2C_PKT_HDR_SIZE + 1; + + rtw_h2c_pkt_set_header(h2c_pkt, H2C_PKT_IQK); + SET_PKT_H2C_TOTAL_LEN(h2c_pkt, total_size); + IQK_SET_CLEAR(h2c_pkt, para->clear); + IQK_SET_SEGMENT_IQK(h2c_pkt, para->segment_iqk); + + rtw_fw_send_h2c_packet(rtwdev, h2c_pkt); +} + +void rtw_fw_send_rssi_info(struct rtw_dev *rtwdev, struct rtw_sta_info *si) +{ + u8 h2c_pkt[H2C_PKT_SIZE] = {0}; + u8 rssi = ewma_rssi_read(&si->avg_rssi); + bool stbc_en = si->stbc_en ? true : false; + + SET_H2C_CMD_ID_CLASS(h2c_pkt, H2C_CMD_RSSI_MONITOR); + + SET_RSSI_INFO_MACID(h2c_pkt, si->mac_id); + SET_RSSI_INFO_RSSI(h2c_pkt, rssi); + SET_RSSI_INFO_STBC(h2c_pkt, stbc_en); + + rtw_fw_send_h2c_command(rtwdev, h2c_pkt); +} + +void rtw_fw_send_ra_info(struct rtw_dev *rtwdev, struct rtw_sta_info *si) +{ + u8 h2c_pkt[H2C_PKT_SIZE] = {0}; + bool no_update = si->updated; + bool disable_pt = true; + + SET_H2C_CMD_ID_CLASS(h2c_pkt, H2C_CMD_RA_INFO); + + SET_RA_INFO_MACID(h2c_pkt, si->mac_id); + SET_RA_INFO_RATE_ID(h2c_pkt, si->rate_id); + SET_RA_INFO_INIT_RA_LVL(h2c_pkt, si->init_ra_lv); + SET_RA_INFO_SGI_EN(h2c_pkt, si->sgi_enable); + SET_RA_INFO_BW_MODE(h2c_pkt, si->bw_mode); + SET_RA_INFO_LDPC(h2c_pkt, si->ldpc_en); + SET_RA_INFO_NO_UPDATE(h2c_pkt, no_update); + SET_RA_INFO_VHT_EN(h2c_pkt, si->vht_enable); + SET_RA_INFO_DIS_PT(h2c_pkt, disable_pt); + SET_RA_INFO_RA_MASK0(h2c_pkt, (si->ra_mask & 0xff)); + SET_RA_INFO_RA_MASK1(h2c_pkt, (si->ra_mask & 0xff00) >> 8); + SET_RA_INFO_RA_MASK2(h2c_pkt, (si->ra_mask & 0xff0000) >> 16); + SET_RA_INFO_RA_MASK3(h2c_pkt, (si->ra_mask & 0xff000000) >> 24); + + si->init_ra_lv = 0; + si->updated = true; + + rtw_fw_send_h2c_command(rtwdev, h2c_pkt); +} + +void rtw_fw_media_status_report(struct rtw_dev *rtwdev, u8 mac_id, bool connect) +{ + u8 h2c_pkt[H2C_PKT_SIZE] = {0}; + + SET_H2C_CMD_ID_CLASS(h2c_pkt, H2C_CMD_MEDIA_STATUS_RPT); + MEDIA_STATUS_RPT_SET_OP_MODE(h2c_pkt, connect); + MEDIA_STATUS_RPT_SET_MACID(h2c_pkt, mac_id); + + rtw_fw_send_h2c_command(rtwdev, h2c_pkt); +} + +void rtw_fw_set_pwr_mode(struct rtw_dev *rtwdev) +{ + struct rtw_lps_conf *conf = &rtwdev->lps_conf; + u8 h2c_pkt[H2C_PKT_SIZE] = {0}; + + SET_H2C_CMD_ID_CLASS(h2c_pkt, H2C_CMD_SET_PWR_MODE); + + SET_PWR_MODE_SET_MODE(h2c_pkt, conf->mode); + SET_PWR_MODE_SET_RLBM(h2c_pkt, conf->rlbm); + SET_PWR_MODE_SET_SMART_PS(h2c_pkt, conf->smart_ps); + SET_PWR_MODE_SET_AWAKE_INTERVAL(h2c_pkt, conf->awake_interval); + SET_PWR_MODE_SET_PORT_ID(h2c_pkt, conf->port_id); + SET_PWR_MODE_SET_PWR_STATE(h2c_pkt, conf->state); + + rtw_fw_send_h2c_command(rtwdev, h2c_pkt); +} + +static u8 rtw_get_rsvd_page_location(struct rtw_dev *rtwdev, + enum rtw_rsvd_packet_type type) +{ + struct rtw_rsvd_page *rsvd_pkt; + u8 location = 0; + + list_for_each_entry(rsvd_pkt, &rtwdev->rsvd_page_list, list) { + if (type == rsvd_pkt->type) + location = rsvd_pkt->page; + } + + return location; +} + +void rtw_send_rsvd_page_h2c(struct rtw_dev *rtwdev) +{ + u8 h2c_pkt[H2C_PKT_SIZE] = {0}; + u8 location = 0; + + SET_H2C_CMD_ID_CLASS(h2c_pkt, H2C_CMD_RSVD_PAGE); + + location = rtw_get_rsvd_page_location(rtwdev, RSVD_PROBE_RESP); + *(h2c_pkt + 1) = location; + rtw_dbg(rtwdev, RTW_DBG_FW, "RSVD_PROBE_RESP loc: %d\n", location); + + location = rtw_get_rsvd_page_location(rtwdev, RSVD_PS_POLL); + *(h2c_pkt + 2) = location; + rtw_dbg(rtwdev, RTW_DBG_FW, "RSVD_PS_POLL loc: %d\n", location); + + location = rtw_get_rsvd_page_location(rtwdev, RSVD_NULL); + *(h2c_pkt + 3) = location; + rtw_dbg(rtwdev, RTW_DBG_FW, "RSVD_NULL loc: %d\n", location); + + location = rtw_get_rsvd_page_location(rtwdev, RSVD_QOS_NULL); + *(h2c_pkt + 4) = location; + rtw_dbg(rtwdev, RTW_DBG_FW, "RSVD_QOS_NULL loc: %d\n", location); + + rtw_fw_send_h2c_command(rtwdev, h2c_pkt); +} + +static struct sk_buff * +rtw_beacon_get(struct ieee80211_hw *hw, struct ieee80211_vif *vif) +{ + struct sk_buff *skb_new; + + if (vif->type != NL80211_IFTYPE_AP && + vif->type != NL80211_IFTYPE_ADHOC && + !ieee80211_vif_is_mesh(vif)) { + skb_new = alloc_skb(1, GFP_KERNEL); + if (!skb_new) + return NULL; + skb_put(skb_new, 1); + } else { + skb_new = ieee80211_beacon_get(hw, vif); + } + + return skb_new; +} + +static struct sk_buff *rtw_get_rsvd_page_skb(struct ieee80211_hw *hw, + struct ieee80211_vif *vif, + enum rtw_rsvd_packet_type type) +{ + struct sk_buff *skb_new; + + switch (type) { + case RSVD_BEACON: + skb_new = rtw_beacon_get(hw, vif); + break; + case RSVD_PS_POLL: + skb_new = ieee80211_pspoll_get(hw, vif); + break; + case RSVD_PROBE_RESP: + skb_new = ieee80211_proberesp_get(hw, vif); + break; + case RSVD_NULL: + skb_new = ieee80211_nullfunc_get(hw, vif, false); + break; + case RSVD_QOS_NULL: + skb_new = ieee80211_nullfunc_get(hw, vif, true); + break; + default: + return NULL; + } + + if (!skb_new) + return NULL; + + return skb_new; +} + +static void rtw_fill_rsvd_page_desc(struct rtw_dev *rtwdev, struct sk_buff *skb) +{ + struct rtw_tx_pkt_info pkt_info; + struct rtw_chip_info *chip = rtwdev->chip; + u8 *pkt_desc; + + memset(&pkt_info, 0, sizeof(pkt_info)); + rtw_rsvd_page_pkt_info_update(rtwdev, &pkt_info, skb); + pkt_desc = skb_push(skb, chip->tx_pkt_desc_sz); + memset(pkt_desc, 0, chip->tx_pkt_desc_sz); + rtw_tx_fill_tx_desc(&pkt_info, skb); +} + +static inline u8 rtw_len_to_page(unsigned int len, u8 page_size) +{ + return DIV_ROUND_UP(len, page_size); +} + +static void rtw_rsvd_page_list_to_buf(struct rtw_dev *rtwdev, u8 page_size, + u8 page_margin, u32 page, u8 *buf, + struct rtw_rsvd_page *rsvd_pkt) +{ + struct sk_buff *skb = rsvd_pkt->skb; + + if (rsvd_pkt->add_txdesc) + rtw_fill_rsvd_page_desc(rtwdev, skb); + + if (page >= 1) + memcpy(buf + page_margin + page_size * (page - 1), + skb->data, skb->len); + else + memcpy(buf, skb->data, skb->len); +} + +void rtw_add_rsvd_page(struct rtw_dev *rtwdev, enum rtw_rsvd_packet_type type, + bool txdesc) +{ + struct rtw_rsvd_page *rsvd_pkt; + + lockdep_assert_held(&rtwdev->mutex); + + list_for_each_entry(rsvd_pkt, &rtwdev->rsvd_page_list, list) { + if (rsvd_pkt->type == type) + return; + } + + rsvd_pkt = kmalloc(sizeof(*rsvd_pkt), GFP_KERNEL); + if (!rsvd_pkt) + return; + + rsvd_pkt->type = type; + rsvd_pkt->add_txdesc = txdesc; + list_add_tail(&rsvd_pkt->list, &rtwdev->rsvd_page_list); +} + +void rtw_reset_rsvd_page(struct rtw_dev *rtwdev) +{ + struct rtw_rsvd_page *rsvd_pkt, *tmp; + + lockdep_assert_held(&rtwdev->mutex); + + list_for_each_entry_safe(rsvd_pkt, tmp, &rtwdev->rsvd_page_list, list) { + if (rsvd_pkt->type == RSVD_BEACON) + continue; + list_del(&rsvd_pkt->list); + kfree(rsvd_pkt); + } +} + +int rtw_fw_write_data_rsvd_page(struct rtw_dev *rtwdev, u16 pg_addr, + u8 *buf, u32 size) +{ + u8 bckp[2]; + u8 val; + u16 rsvd_pg_head; + int ret; + + lockdep_assert_held(&rtwdev->mutex); + + if (!size) + return -EINVAL; + + pg_addr &= BIT_MASK_BCN_HEAD_1_V1; + rtw_write16(rtwdev, REG_FIFOPAGE_CTRL_2, pg_addr | BIT_BCN_VALID_V1); + + val = rtw_read8(rtwdev, REG_CR + 1); + bckp[0] = val; + val |= BIT_ENSWBCN >> 8; + rtw_write8(rtwdev, REG_CR + 1, val); + + val = rtw_read8(rtwdev, REG_FWHW_TXQ_CTRL + 2); + bckp[1] = val; + val &= ~(BIT_EN_BCNQ_DL >> 16); + rtw_write8(rtwdev, REG_FWHW_TXQ_CTRL + 2, val); + + ret = rtw_hci_write_data_rsvd_page(rtwdev, buf, size); + if (ret) { + rtw_err(rtwdev, "failed to write data to rsvd page\n"); + goto restore; + } + + if (!check_hw_ready(rtwdev, REG_FIFOPAGE_CTRL_2, BIT_BCN_VALID_V1, 1)) { + rtw_err(rtwdev, "error beacon valid\n"); + ret = -EBUSY; + } + +restore: + rsvd_pg_head = rtwdev->fifo.rsvd_boundary; + rtw_write16(rtwdev, REG_FIFOPAGE_CTRL_2, + rsvd_pg_head | BIT_BCN_VALID_V1); + rtw_write8(rtwdev, REG_FWHW_TXQ_CTRL + 2, bckp[1]); + rtw_write8(rtwdev, REG_CR + 1, bckp[0]); + + return ret; +} + +static int rtw_download_drv_rsvd_page(struct rtw_dev *rtwdev, u8 *buf, u32 size) +{ + u32 pg_size; + u32 pg_num = 0; + u16 pg_addr = 0; + + pg_size = rtwdev->chip->page_size; + pg_num = size / pg_size + ((size & (pg_size - 1)) ? 1 : 0); + if (pg_num > rtwdev->fifo.rsvd_drv_pg_num) + return -ENOMEM; + + pg_addr = rtwdev->fifo.rsvd_drv_addr; + + return rtw_fw_write_data_rsvd_page(rtwdev, pg_addr, buf, size); +} + +static u8 *rtw_build_rsvd_page(struct rtw_dev *rtwdev, + struct ieee80211_vif *vif, u32 *size) +{ + struct ieee80211_hw *hw = rtwdev->hw; + struct rtw_chip_info *chip = rtwdev->chip; + struct sk_buff *iter; + struct rtw_rsvd_page *rsvd_pkt; + u32 page = 0; + u8 total_page = 0; + u8 page_size, page_margin, tx_desc_sz; + u8 *buf; + + page_size = chip->page_size; + tx_desc_sz = chip->tx_pkt_desc_sz; + page_margin = page_size - tx_desc_sz; + + list_for_each_entry(rsvd_pkt, &rtwdev->rsvd_page_list, list) { + iter = rtw_get_rsvd_page_skb(hw, vif, rsvd_pkt->type); + if (!iter) { + rtw_err(rtwdev, "fail to build rsvd packet\n"); + goto release_skb; + } + rsvd_pkt->skb = iter; + rsvd_pkt->page = total_page; + if (rsvd_pkt->add_txdesc) + total_page += rtw_len_to_page(iter->len + tx_desc_sz, + page_size); + else + total_page += rtw_len_to_page(iter->len, page_size); + } + + if (total_page > rtwdev->fifo.rsvd_drv_pg_num) { + rtw_err(rtwdev, "rsvd page over size: %d\n", total_page); + goto release_skb; + } + + *size = (total_page - 1) * page_size + page_margin; + buf = kzalloc(*size, GFP_KERNEL); + if (!buf) + goto release_skb; + + list_for_each_entry(rsvd_pkt, &rtwdev->rsvd_page_list, list) { + rtw_rsvd_page_list_to_buf(rtwdev, page_size, page_margin, + page, buf, rsvd_pkt); + page += rtw_len_to_page(rsvd_pkt->skb->len, page_size); + } + list_for_each_entry(rsvd_pkt, &rtwdev->rsvd_page_list, list) + kfree_skb(rsvd_pkt->skb); + + return buf; + +release_skb: + list_for_each_entry(rsvd_pkt, &rtwdev->rsvd_page_list, list) + kfree_skb(rsvd_pkt->skb); + + return NULL; +} + +static int +rtw_download_beacon(struct rtw_dev *rtwdev, struct ieee80211_vif *vif) +{ + struct ieee80211_hw *hw = rtwdev->hw; + struct sk_buff *skb; + int ret = 0; + + skb = rtw_beacon_get(hw, vif); + if (!skb) { + rtw_err(rtwdev, "failed to get beacon skb\n"); + ret = -ENOMEM; + goto out; + } + + ret = rtw_download_drv_rsvd_page(rtwdev, skb->data, skb->len); + if (ret) + rtw_err(rtwdev, "failed to download drv rsvd page\n"); + + dev_kfree_skb(skb); + +out: + return ret; +} + +int rtw_fw_download_rsvd_page(struct rtw_dev *rtwdev, struct ieee80211_vif *vif) +{ + u8 *buf; + u32 size; + int ret; + + buf = rtw_build_rsvd_page(rtwdev, vif, &size); + if (!buf) { + rtw_err(rtwdev, "failed to build rsvd page pkt\n"); + return -ENOMEM; + } + + ret = rtw_download_drv_rsvd_page(rtwdev, buf, size); + if (ret) { + rtw_err(rtwdev, "failed to download drv rsvd page\n"); + goto free; + } + + ret = rtw_download_beacon(rtwdev, vif); + if (ret) { + rtw_err(rtwdev, "failed to download beacon\n"); + goto free; + } + +free: + kfree(buf); + + return ret; +} + +int rtw_dump_drv_rsvd_page(struct rtw_dev *rtwdev, + u32 offset, u32 size, u32 *buf) +{ + struct rtw_fifo_conf *fifo = &rtwdev->fifo; + u32 residue, i; + u16 start_pg; + u16 idx = 0; + u16 ctl; + u8 rcr; + + if (size & 0x3) { + rtw_warn(rtwdev, "should be 4-byte aligned\n"); + return -EINVAL; + } + + offset += fifo->rsvd_boundary << TX_PAGE_SIZE_SHIFT; + residue = offset & (FIFO_PAGE_SIZE - 1); + start_pg = offset >> FIFO_PAGE_SIZE_SHIFT; + start_pg += RSVD_PAGE_START_ADDR; + + rcr = rtw_read8(rtwdev, REG_RCR + 2); + ctl = rtw_read16(rtwdev, REG_PKTBUF_DBG_CTRL) & 0xf000; + + /* disable rx clock gate */ + rtw_write8(rtwdev, REG_RCR, rcr | BIT(3)); + + do { + rtw_write16(rtwdev, REG_PKTBUF_DBG_CTRL, start_pg | ctl); + + for (i = FIFO_DUMP_ADDR + residue; + i < FIFO_DUMP_ADDR + FIFO_PAGE_SIZE; i += 4) { + buf[idx++] = rtw_read32(rtwdev, i); + size -= 4; + if (size == 0) + goto out; + } + + residue = 0; + start_pg++; + } while (size); + +out: + rtw_write16(rtwdev, REG_PKTBUF_DBG_CTRL, ctl); + rtw_write8(rtwdev, REG_RCR + 2, rcr); + return 0; +} diff --git a/drivers/net/wireless/realtek/rtw88/fw.h b/drivers/net/wireless/realtek/rtw88/fw.h new file mode 100644 index 0000000000000000000000000000000000000000..703466393ecb6fb9989d43dc3c1731fde0a50e50 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/fw.h @@ -0,0 +1,222 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#ifndef __RTW_FW_H_ +#define __RTW_FW_H_ + +#define H2C_PKT_SIZE 32 +#define H2C_PKT_HDR_SIZE 8 + +/* FW bin information */ +#define FW_HDR_SIZE 64 +#define FW_HDR_CHKSUM_SIZE 8 +#define FW_HDR_VERSION 4 +#define FW_HDR_SUBVERSION 6 +#define FW_HDR_SUBINDEX 7 +#define FW_HDR_MONTH 16 +#define FW_HDR_DATE 17 +#define FW_HDR_HOUR 18 +#define FW_HDR_MIN 19 +#define FW_HDR_YEAR 20 +#define FW_HDR_MEM_USAGE 24 +#define FW_HDR_H2C_FMT_VER 28 +#define FW_HDR_DMEM_ADDR 32 +#define FW_HDR_DMEM_SIZE 36 +#define FW_HDR_IMEM_SIZE 48 +#define FW_HDR_EMEM_SIZE 52 +#define FW_HDR_EMEM_ADDR 56 +#define FW_HDR_IMEM_ADDR 60 + +#define FIFO_PAGE_SIZE_SHIFT 12 +#define FIFO_PAGE_SIZE 4096 +#define RSVD_PAGE_START_ADDR 0x780 +#define FIFO_DUMP_ADDR 0x8000 + +enum rtw_c2h_cmd_id { + C2H_BT_INFO = 0x09, + C2H_HW_FEATURE_REPORT = 0x19, + C2H_HW_FEATURE_DUMP = 0xfd, + C2H_HALMAC = 0xff, +}; + +enum rtw_c2h_cmd_id_ext { + C2H_CCX_RPT = 0x0f, +}; + +struct rtw_c2h_cmd { + u8 id; + u8 seq; + u8 payload[0]; +} __packed; + +enum rtw_rsvd_packet_type { + RSVD_BEACON, + RSVD_PS_POLL, + RSVD_PROBE_RESP, + RSVD_NULL, + RSVD_QOS_NULL, +}; + +enum rtw_fw_rf_type { + FW_RF_1T2R = 0, + FW_RF_2T4R = 1, + FW_RF_2T2R = 2, + FW_RF_2T3R = 3, + FW_RF_1T1R = 4, + FW_RF_2T2R_GREEN = 5, + FW_RF_3T3R = 6, + FW_RF_3T4R = 7, + FW_RF_4T4R = 8, + FW_RF_MAX_TYPE = 0xF, +}; + +struct rtw_iqk_para { + u8 clear; + u8 segment_iqk; +}; + +struct rtw_rsvd_page { + struct list_head list; + struct sk_buff *skb; + enum rtw_rsvd_packet_type type; + u8 page; + bool add_txdesc; +}; + +/* C2H */ +#define GET_CCX_REPORT_SEQNUM(c2h_payload) (c2h_payload[8] & 0xfc) +#define GET_CCX_REPORT_STATUS(c2h_payload) (c2h_payload[9] & 0xc0) + +/* PKT H2C */ +#define H2C_PKT_CMD_ID 0xFF +#define H2C_PKT_CATEGORY 0x01 + +#define H2C_PKT_GENERAL_INFO 0x0D +#define H2C_PKT_PHYDM_INFO 0x11 +#define H2C_PKT_IQK 0x0E + +#define SET_PKT_H2C_CATEGORY(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(6, 0)) +#define SET_PKT_H2C_CMD_ID(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(15, 8)) +#define SET_PKT_H2C_SUB_CMD_ID(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(31, 16)) +#define SET_PKT_H2C_TOTAL_LEN(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x01, value, GENMASK(15, 0)) + +static inline void rtw_h2c_pkt_set_header(u8 *h2c_pkt, u8 sub_id) +{ + SET_PKT_H2C_CATEGORY(h2c_pkt, H2C_PKT_CATEGORY); + SET_PKT_H2C_CMD_ID(h2c_pkt, H2C_PKT_CMD_ID); + SET_PKT_H2C_SUB_CMD_ID(h2c_pkt, sub_id); +} + +#define FW_OFFLOAD_H2C_SET_SEQ_NUM(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x01, value, GENMASK(31, 16)) +#define GENERAL_INFO_SET_FW_TX_BOUNDARY(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x02, value, GENMASK(23, 16)) + +#define PHYDM_INFO_SET_REF_TYPE(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x02, value, GENMASK(7, 0)) +#define PHYDM_INFO_SET_RF_TYPE(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x02, value, GENMASK(15, 8)) +#define PHYDM_INFO_SET_CUT_VER(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x02, value, GENMASK(23, 16)) +#define PHYDM_INFO_SET_RX_ANT_STATUS(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x02, value, GENMASK(27, 24)) +#define PHYDM_INFO_SET_TX_ANT_STATUS(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x02, value, GENMASK(31, 28)) +#define IQK_SET_CLEAR(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x02, value, BIT(0)) +#define IQK_SET_SEGMENT_IQK(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x02, value, BIT(1)) + +/* Command H2C */ +#define H2C_CMD_RSVD_PAGE 0x0 +#define H2C_CMD_MEDIA_STATUS_RPT 0x01 +#define H2C_CMD_SET_PWR_MODE 0x20 +#define H2C_CMD_RA_INFO 0x40 +#define H2C_CMD_RSSI_MONITOR 0x42 + +#define SET_H2C_CMD_ID_CLASS(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(7, 0)) + +#define MEDIA_STATUS_RPT_SET_OP_MODE(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, BIT(8)) +#define MEDIA_STATUS_RPT_SET_MACID(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(23, 16)) + +#define SET_PWR_MODE_SET_MODE(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(14, 8)) +#define SET_PWR_MODE_SET_RLBM(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(19, 16)) +#define SET_PWR_MODE_SET_SMART_PS(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(23, 20)) +#define SET_PWR_MODE_SET_AWAKE_INTERVAL(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(31, 24)) +#define SET_PWR_MODE_SET_PORT_ID(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x01, value, GENMASK(7, 5)) +#define SET_PWR_MODE_SET_PWR_STATE(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x01, value, GENMASK(15, 8)) +#define SET_RSSI_INFO_MACID(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(15, 8)) +#define SET_RSSI_INFO_RSSI(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(31, 24)) +#define SET_RSSI_INFO_STBC(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x01, value, BIT(1)) +#define SET_RA_INFO_MACID(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(15, 8)) +#define SET_RA_INFO_RATE_ID(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(20, 16)) +#define SET_RA_INFO_INIT_RA_LVL(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(22, 21)) +#define SET_RA_INFO_SGI_EN(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, BIT(23)) +#define SET_RA_INFO_BW_MODE(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(25, 24)) +#define SET_RA_INFO_LDPC(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, BIT(26)) +#define SET_RA_INFO_NO_UPDATE(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, BIT(27)) +#define SET_RA_INFO_VHT_EN(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, GENMASK(29, 28)) +#define SET_RA_INFO_DIS_PT(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x00, value, BIT(30)) +#define SET_RA_INFO_RA_MASK0(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x01, value, GENMASK(7, 0)) +#define SET_RA_INFO_RA_MASK1(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x01, value, GENMASK(15, 8)) +#define SET_RA_INFO_RA_MASK2(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x01, value, GENMASK(23, 16)) +#define SET_RA_INFO_RA_MASK3(h2c_pkt, value) \ + le32p_replace_bits((__le32 *)(h2c_pkt) + 0x01, value, GENMASK(31, 24)) + +static inline struct rtw_c2h_cmd *get_c2h_from_skb(struct sk_buff *skb) +{ + u32 pkt_offset; + + pkt_offset = *((u32 *)skb->cb); + return (struct rtw_c2h_cmd *)(skb->data + pkt_offset); +} + +void rtw_fw_c2h_cmd_handle(struct rtw_dev *rtwdev, struct sk_buff *skb); +void rtw_fw_send_general_info(struct rtw_dev *rtwdev); +void rtw_fw_send_phydm_info(struct rtw_dev *rtwdev); + +void rtw_fw_do_iqk(struct rtw_dev *rtwdev, struct rtw_iqk_para *para); +void rtw_fw_set_pwr_mode(struct rtw_dev *rtwdev); +void rtw_fw_send_rssi_info(struct rtw_dev *rtwdev, struct rtw_sta_info *si); +void rtw_fw_send_ra_info(struct rtw_dev *rtwdev, struct rtw_sta_info *si); +void rtw_fw_media_status_report(struct rtw_dev *rtwdev, u8 mac_id, bool conn); +void rtw_add_rsvd_page(struct rtw_dev *rtwdev, enum rtw_rsvd_packet_type type, + bool txdesc); +int rtw_fw_write_data_rsvd_page(struct rtw_dev *rtwdev, u16 pg_addr, + u8 *buf, u32 size); +void rtw_reset_rsvd_page(struct rtw_dev *rtwdev); +int rtw_fw_download_rsvd_page(struct rtw_dev *rtwdev, + struct ieee80211_vif *vif); +void rtw_send_rsvd_page_h2c(struct rtw_dev *rtwdev); +int rtw_dump_drv_rsvd_page(struct rtw_dev *rtwdev, + u32 offset, u32 size, u32 *buf); +#endif diff --git a/drivers/net/wireless/realtek/rtw88/hci.h b/drivers/net/wireless/realtek/rtw88/hci.h new file mode 100644 index 0000000000000000000000000000000000000000..2676582a85a001630af5699ee3eb32a35bebab4f --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/hci.h @@ -0,0 +1,211 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#ifndef __RTW_HCI_H__ +#define __RTW_HCI_H__ + +/* ops for PCI, USB and SDIO */ +struct rtw_hci_ops { + int (*tx)(struct rtw_dev *rtwdev, + struct rtw_tx_pkt_info *pkt_info, + struct sk_buff *skb); + int (*setup)(struct rtw_dev *rtwdev); + int (*start)(struct rtw_dev *rtwdev); + void (*stop)(struct rtw_dev *rtwdev); + + int (*write_data_rsvd_page)(struct rtw_dev *rtwdev, u8 *buf, u32 size); + int (*write_data_h2c)(struct rtw_dev *rtwdev, u8 *buf, u32 size); + + u8 (*read8)(struct rtw_dev *rtwdev, u32 addr); + u16 (*read16)(struct rtw_dev *rtwdev, u32 addr); + u32 (*read32)(struct rtw_dev *rtwdev, u32 addr); + void (*write8)(struct rtw_dev *rtwdev, u32 addr, u8 val); + void (*write16)(struct rtw_dev *rtwdev, u32 addr, u16 val); + void (*write32)(struct rtw_dev *rtwdev, u32 addr, u32 val); +}; + +static inline int rtw_hci_tx(struct rtw_dev *rtwdev, + struct rtw_tx_pkt_info *pkt_info, + struct sk_buff *skb) +{ + return rtwdev->hci.ops->tx(rtwdev, pkt_info, skb); +} + +static inline int rtw_hci_setup(struct rtw_dev *rtwdev) +{ + return rtwdev->hci.ops->setup(rtwdev); +} + +static inline int rtw_hci_start(struct rtw_dev *rtwdev) +{ + return rtwdev->hci.ops->start(rtwdev); +} + +static inline void rtw_hci_stop(struct rtw_dev *rtwdev) +{ + rtwdev->hci.ops->stop(rtwdev); +} + +static inline int +rtw_hci_write_data_rsvd_page(struct rtw_dev *rtwdev, u8 *buf, u32 size) +{ + return rtwdev->hci.ops->write_data_rsvd_page(rtwdev, buf, size); +} + +static inline int +rtw_hci_write_data_h2c(struct rtw_dev *rtwdev, u8 *buf, u32 size) +{ + return rtwdev->hci.ops->write_data_h2c(rtwdev, buf, size); +} + +static inline u8 rtw_read8(struct rtw_dev *rtwdev, u32 addr) +{ + return rtwdev->hci.ops->read8(rtwdev, addr); +} + +static inline u16 rtw_read16(struct rtw_dev *rtwdev, u32 addr) +{ + return rtwdev->hci.ops->read16(rtwdev, addr); +} + +static inline u32 rtw_read32(struct rtw_dev *rtwdev, u32 addr) +{ + return rtwdev->hci.ops->read32(rtwdev, addr); +} + +static inline void rtw_write8(struct rtw_dev *rtwdev, u32 addr, u8 val) +{ + rtwdev->hci.ops->write8(rtwdev, addr, val); +} + +static inline void rtw_write16(struct rtw_dev *rtwdev, u32 addr, u16 val) +{ + rtwdev->hci.ops->write16(rtwdev, addr, val); +} + +static inline void rtw_write32(struct rtw_dev *rtwdev, u32 addr, u32 val) +{ + rtwdev->hci.ops->write32(rtwdev, addr, val); +} + +static inline void rtw_write8_set(struct rtw_dev *rtwdev, u32 addr, u8 bit) +{ + u8 val; + + val = rtw_read8(rtwdev, addr); + rtw_write8(rtwdev, addr, val | bit); +} + +static inline void rtw_writ16_set(struct rtw_dev *rtwdev, u32 addr, u16 bit) +{ + u16 val; + + val = rtw_read16(rtwdev, addr); + rtw_write16(rtwdev, addr, val | bit); +} + +static inline void rtw_write32_set(struct rtw_dev *rtwdev, u32 addr, u32 bit) +{ + u32 val; + + val = rtw_read32(rtwdev, addr); + rtw_write32(rtwdev, addr, val | bit); +} + +static inline void rtw_write8_clr(struct rtw_dev *rtwdev, u32 addr, u8 bit) +{ + u8 val; + + val = rtw_read8(rtwdev, addr); + rtw_write8(rtwdev, addr, val & ~bit); +} + +static inline void rtw_write16_clr(struct rtw_dev *rtwdev, u32 addr, u16 bit) +{ + u16 val; + + val = rtw_read16(rtwdev, addr); + rtw_write16(rtwdev, addr, val & ~bit); +} + +static inline void rtw_write32_clr(struct rtw_dev *rtwdev, u32 addr, u32 bit) +{ + u32 val; + + val = rtw_read32(rtwdev, addr); + rtw_write32(rtwdev, addr, val & ~bit); +} + +static inline u32 +rtw_read_rf(struct rtw_dev *rtwdev, enum rtw_rf_path rf_path, + u32 addr, u32 mask) +{ + unsigned long flags; + u32 val; + + spin_lock_irqsave(&rtwdev->rf_lock, flags); + val = rtwdev->chip->ops->read_rf(rtwdev, rf_path, addr, mask); + spin_unlock_irqrestore(&rtwdev->rf_lock, flags); + + return val; +} + +static inline void +rtw_write_rf(struct rtw_dev *rtwdev, enum rtw_rf_path rf_path, + u32 addr, u32 mask, u32 data) +{ + unsigned long flags; + + spin_lock_irqsave(&rtwdev->rf_lock, flags); + rtwdev->chip->ops->write_rf(rtwdev, rf_path, addr, mask, data); + spin_unlock_irqrestore(&rtwdev->rf_lock, flags); +} + +static inline u32 +rtw_read32_mask(struct rtw_dev *rtwdev, u32 addr, u32 mask) +{ + u32 shift = __ffs(mask); + u32 orig; + u32 ret; + + orig = rtw_read32(rtwdev, addr); + ret = (orig & mask) >> shift; + + return ret; +} + +static inline void +rtw_write32_mask(struct rtw_dev *rtwdev, u32 addr, u32 mask, u32 data) +{ + u32 shift = __ffs(mask); + u32 orig; + u32 set; + + WARN(addr & 0x3, "should be 4-byte aligned, addr = 0x%08x\n", addr); + + orig = rtw_read32(rtwdev, addr); + set = (orig & ~mask) | ((data << shift) & mask); + rtw_write32(rtwdev, addr, set); +} + +static inline void +rtw_write8_mask(struct rtw_dev *rtwdev, u32 addr, u32 mask, u8 data) +{ + u32 shift; + u8 orig, set; + + mask &= 0xff; + shift = __ffs(mask); + + orig = rtw_read8(rtwdev, addr); + set = (orig & ~mask) | ((data << shift) & mask); + rtw_write8(rtwdev, addr, set); +} + +static inline enum rtw_hci_type rtw_hci_type(struct rtw_dev *rtwdev) +{ + return rtwdev->hci.type; +} + +#endif diff --git a/drivers/net/wireless/realtek/rtw88/mac.c b/drivers/net/wireless/realtek/rtw88/mac.c new file mode 100644 index 0000000000000000000000000000000000000000..25a923bc63662fb6ed6af4df290570d43c777a08 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/mac.c @@ -0,0 +1,965 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#include "main.h" +#include "mac.h" +#include "reg.h" +#include "fw.h" +#include "debug.h" + +void rtw_set_channel_mac(struct rtw_dev *rtwdev, u8 channel, u8 bw, + u8 primary_ch_idx) +{ + u8 txsc40 = 0, txsc20 = 0; + u32 value32; + u8 value8; + + txsc20 = primary_ch_idx; + if (txsc20 == 1 || txsc20 == 3) + txsc40 = 9; + else + txsc40 = 10; + rtw_write8(rtwdev, REG_DATA_SC, + BIT_TXSC_20M(txsc20) | BIT_TXSC_40M(txsc40)); + + value32 = rtw_read32(rtwdev, REG_WMAC_TRXPTCL_CTL); + value32 &= ~BIT_RFMOD; + switch (bw) { + case RTW_CHANNEL_WIDTH_80: + value32 |= BIT_RFMOD_80M; + break; + case RTW_CHANNEL_WIDTH_40: + value32 |= BIT_RFMOD_40M; + break; + case RTW_CHANNEL_WIDTH_20: + default: + break; + } + rtw_write32(rtwdev, REG_WMAC_TRXPTCL_CTL, value32); + + value32 = rtw_read32(rtwdev, REG_AFE_CTRL1) & ~(BIT_MAC_CLK_SEL); + value32 |= (MAC_CLK_HW_DEF_80M << BIT_SHIFT_MAC_CLK_SEL); + rtw_write32(rtwdev, REG_AFE_CTRL1, value32); + + rtw_write8(rtwdev, REG_USTIME_TSF, MAC_CLK_SPEED); + rtw_write8(rtwdev, REG_USTIME_EDCA, MAC_CLK_SPEED); + + value8 = rtw_read8(rtwdev, REG_CCK_CHECK); + value8 = value8 & ~BIT_CHECK_CCK_EN; + if (channel > 35) + value8 |= BIT_CHECK_CCK_EN; + rtw_write8(rtwdev, REG_CCK_CHECK, value8); +} + +static int rtw_mac_pre_system_cfg(struct rtw_dev *rtwdev) +{ + u32 value32; + u8 value8; + + rtw_write8(rtwdev, REG_RSV_CTRL, 0); + + switch (rtw_hci_type(rtwdev)) { + case RTW_HCI_TYPE_PCIE: + rtw_write32_set(rtwdev, REG_HCI_OPT_CTRL, BIT_BT_DIG_CLK_EN); + break; + case RTW_HCI_TYPE_USB: + break; + default: + return -EINVAL; + } + + /* config PIN Mux */ + value32 = rtw_read32(rtwdev, REG_PAD_CTRL1); + value32 |= BIT_PAPE_WLBT_SEL | BIT_LNAON_WLBT_SEL; + rtw_write32(rtwdev, REG_PAD_CTRL1, value32); + + value32 = rtw_read32(rtwdev, REG_LED_CFG); + value32 &= ~(BIT_PAPE_SEL_EN | BIT_LNAON_SEL_EN); + rtw_write32(rtwdev, REG_LED_CFG, value32); + + value32 = rtw_read32(rtwdev, REG_GPIO_MUXCFG); + value32 |= BIT_WLRFE_4_5_EN; + rtw_write32(rtwdev, REG_GPIO_MUXCFG, value32); + + /* disable BB/RF */ + value8 = rtw_read8(rtwdev, REG_SYS_FUNC_EN); + value8 &= ~(BIT_FEN_BB_RSTB | BIT_FEN_BB_GLB_RST); + rtw_write8(rtwdev, REG_SYS_FUNC_EN, value8); + + value8 = rtw_read8(rtwdev, REG_RF_CTRL); + value8 &= ~(BIT_RF_SDM_RSTB | BIT_RF_RSTB | BIT_RF_EN); + rtw_write8(rtwdev, REG_RF_CTRL, value8); + + value32 = rtw_read32(rtwdev, REG_WLRF1); + value32 &= ~BIT_WLRF1_BBRF_EN; + rtw_write32(rtwdev, REG_WLRF1, value32); + + return 0; +} + +static int rtw_pwr_cmd_polling(struct rtw_dev *rtwdev, + struct rtw_pwr_seq_cmd *cmd) +{ + u8 value; + u8 flag = 0; + u32 offset; + u32 cnt = RTW_PWR_POLLING_CNT; + + if (cmd->base == RTW_PWR_ADDR_SDIO) + offset = cmd->offset | SDIO_LOCAL_OFFSET; + else + offset = cmd->offset; + + do { + cnt--; + value = rtw_read8(rtwdev, offset); + value &= cmd->mask; + if (value == (cmd->value & cmd->mask)) + return 0; + if (cnt == 0) { + if (rtw_hci_type(rtwdev) == RTW_HCI_TYPE_PCIE && + flag == 0) { + value = rtw_read8(rtwdev, REG_SYS_PW_CTRL); + value |= BIT(3); + rtw_write8(rtwdev, REG_SYS_PW_CTRL, value); + value &= ~BIT(3); + rtw_write8(rtwdev, REG_SYS_PW_CTRL, value); + cnt = RTW_PWR_POLLING_CNT; + flag = 1; + } else { + return -EBUSY; + } + } else { + udelay(50); + } + } while (1); +} + +static int rtw_sub_pwr_seq_parser(struct rtw_dev *rtwdev, u8 intf_mask, + u8 cut_mask, struct rtw_pwr_seq_cmd *cmd) +{ + struct rtw_pwr_seq_cmd *cur_cmd; + u32 offset; + u8 value; + + for (cur_cmd = cmd; cur_cmd->cmd != RTW_PWR_CMD_END; cur_cmd++) { + if (!(cur_cmd->intf_mask & intf_mask) || + !(cur_cmd->cut_mask & cut_mask)) + continue; + + switch (cur_cmd->cmd) { + case RTW_PWR_CMD_WRITE: + offset = cur_cmd->offset; + + if (cur_cmd->base == RTW_PWR_ADDR_SDIO) + offset |= SDIO_LOCAL_OFFSET; + + value = rtw_read8(rtwdev, offset); + value &= ~cur_cmd->mask; + value |= (cur_cmd->value & cur_cmd->mask); + rtw_write8(rtwdev, offset, value); + break; + case RTW_PWR_CMD_POLLING: + if (rtw_pwr_cmd_polling(rtwdev, cur_cmd)) + return -EBUSY; + break; + case RTW_PWR_CMD_DELAY: + if (cur_cmd->value == RTW_PWR_DELAY_US) + udelay(cur_cmd->offset); + else + mdelay(cur_cmd->offset); + break; + case RTW_PWR_CMD_READ: + break; + default: + return -EINVAL; + } + } + + return 0; +} + +static int rtw_pwr_seq_parser(struct rtw_dev *rtwdev, + struct rtw_pwr_seq_cmd **cmd_seq) +{ + u8 cut_mask; + u8 intf_mask; + u8 cut; + u32 idx = 0; + struct rtw_pwr_seq_cmd *cmd; + int ret; + + cut = rtwdev->hal.cut_version; + cut_mask = cut_version_to_mask(cut); + switch (rtw_hci_type(rtwdev)) { + case RTW_HCI_TYPE_PCIE: + intf_mask = BIT(2); + break; + case RTW_HCI_TYPE_USB: + intf_mask = BIT(1); + break; + default: + return -EINVAL; + } + + do { + cmd = cmd_seq[idx]; + if (!cmd) + break; + + ret = rtw_sub_pwr_seq_parser(rtwdev, intf_mask, cut_mask, cmd); + if (ret) + return -EBUSY; + + idx++; + } while (1); + + return 0; +} + +static int rtw_mac_power_switch(struct rtw_dev *rtwdev, bool pwr_on) +{ + struct rtw_chip_info *chip = rtwdev->chip; + struct rtw_pwr_seq_cmd **pwr_seq; + u8 rpwm; + bool cur_pwr; + + rpwm = rtw_read8(rtwdev, rtwdev->hci.rpwm_addr); + + /* Check FW still exist or not */ + if (rtw_read16(rtwdev, REG_MCUFW_CTRL) == 0xC078) { + rpwm = (rpwm ^ BIT_RPWM_TOGGLE) & BIT_RPWM_TOGGLE; + rtw_write8(rtwdev, rtwdev->hci.rpwm_addr, rpwm); + } + + if (rtw_read8(rtwdev, REG_CR) == 0xea) + cur_pwr = false; + else if (rtw_hci_type(rtwdev) == RTW_HCI_TYPE_USB && + (rtw_read8(rtwdev, REG_SYS_STATUS1 + 1) & BIT(0))) + cur_pwr = false; + else + cur_pwr = true; + + if (pwr_on && cur_pwr) + return -EALREADY; + + pwr_seq = pwr_on ? chip->pwr_on_seq : chip->pwr_off_seq; + if (rtw_pwr_seq_parser(rtwdev, pwr_seq)) + return -EINVAL; + + return 0; +} + +static int rtw_mac_init_system_cfg(struct rtw_dev *rtwdev) +{ + u8 sys_func_en = rtwdev->chip->sys_func_en; + u8 value8; + u32 value, tmp; + + value = rtw_read32(rtwdev, REG_CPU_DMEM_CON); + value |= BIT_WL_PLATFORM_RST | BIT_DDMA_EN; + rtw_write32(rtwdev, REG_CPU_DMEM_CON, value); + + rtw_write8(rtwdev, REG_SYS_FUNC_EN + 1, sys_func_en); + value8 = (rtw_read8(rtwdev, REG_CR_EXT + 3) & 0xF0) | 0x0C; + rtw_write8(rtwdev, REG_CR_EXT + 3, value8); + + /* disable boot-from-flash for driver's DL FW */ + tmp = rtw_read32(rtwdev, REG_MCUFW_CTRL); + if (tmp & BIT_BOOT_FSPI_EN) { + rtw_write32(rtwdev, REG_MCUFW_CTRL, tmp & (~BIT_BOOT_FSPI_EN)); + value = rtw_read32(rtwdev, REG_GPIO_MUXCFG) & (~BIT_FSPI_EN); + rtw_write32(rtwdev, REG_GPIO_MUXCFG, value); + } + + return 0; +} + +int rtw_mac_power_on(struct rtw_dev *rtwdev) +{ + int ret = 0; + + ret = rtw_mac_pre_system_cfg(rtwdev); + if (ret) + goto err; + + ret = rtw_mac_power_switch(rtwdev, true); + if (ret) + goto err; + + ret = rtw_mac_init_system_cfg(rtwdev); + if (ret) + goto err; + + return 0; + +err: + rtw_err(rtwdev, "mac power on failed"); + return ret; +} + +void rtw_mac_power_off(struct rtw_dev *rtwdev) +{ + rtw_mac_power_switch(rtwdev, false); +} + +static bool check_firmware_size(const u8 *data, u32 size) +{ + u32 dmem_size; + u32 imem_size; + u32 emem_size; + u32 real_size; + + dmem_size = le32_to_cpu(*((__le32 *)(data + FW_HDR_DMEM_SIZE))); + imem_size = le32_to_cpu(*((__le32 *)(data + FW_HDR_IMEM_SIZE))); + emem_size = ((*(data + FW_HDR_MEM_USAGE)) & BIT(4)) ? + le32_to_cpu(*((__le32 *)(data + FW_HDR_EMEM_SIZE))) : 0; + + dmem_size += FW_HDR_CHKSUM_SIZE; + imem_size += FW_HDR_CHKSUM_SIZE; + emem_size += emem_size ? FW_HDR_CHKSUM_SIZE : 0; + real_size = FW_HDR_SIZE + dmem_size + imem_size + emem_size; + if (real_size != size) + return false; + + return true; +} + +static void wlan_cpu_enable(struct rtw_dev *rtwdev, bool enable) +{ + if (enable) { + /* cpu io interface enable */ + rtw_write8_set(rtwdev, REG_RSV_CTRL + 1, BIT_WLMCU_IOIF); + + /* cpu enable */ + rtw_write8_set(rtwdev, REG_SYS_FUNC_EN + 1, BIT_FEN_CPUEN); + } else { + /* cpu io interface disable */ + rtw_write8_clr(rtwdev, REG_SYS_FUNC_EN + 1, BIT_FEN_CPUEN); + + /* cpu disable */ + rtw_write8_clr(rtwdev, REG_RSV_CTRL + 1, BIT_WLMCU_IOIF); + } +} + +#define DLFW_RESTORE_REG_NUM 6 + +static void download_firmware_reg_backup(struct rtw_dev *rtwdev, + struct rtw_backup_info *bckp) +{ + u8 tmp; + u8 bckp_idx = 0; + + /* set HIQ to hi priority */ + bckp[bckp_idx].len = 1; + bckp[bckp_idx].reg = REG_TXDMA_PQ_MAP + 1; + bckp[bckp_idx].val = rtw_read8(rtwdev, REG_TXDMA_PQ_MAP + 1); + bckp_idx++; + tmp = RTW_DMA_MAPPING_HIGH << 6; + rtw_write8(rtwdev, REG_TXDMA_PQ_MAP + 1, tmp); + + /* DLFW only use HIQ, map HIQ to hi priority */ + bckp[bckp_idx].len = 1; + bckp[bckp_idx].reg = REG_CR; + bckp[bckp_idx].val = rtw_read8(rtwdev, REG_CR); + bckp_idx++; + bckp[bckp_idx].len = 4; + bckp[bckp_idx].reg = REG_H2CQ_CSR; + bckp[bckp_idx].val = BIT_H2CQ_FULL; + bckp_idx++; + tmp = BIT_HCI_TXDMA_EN | BIT_TXDMA_EN; + rtw_write8(rtwdev, REG_CR, tmp); + rtw_write32(rtwdev, REG_H2CQ_CSR, BIT_H2CQ_FULL); + + /* Config hi priority queue and public priority queue page number */ + bckp[bckp_idx].len = 2; + bckp[bckp_idx].reg = REG_FIFOPAGE_INFO_1; + bckp[bckp_idx].val = rtw_read16(rtwdev, REG_FIFOPAGE_INFO_1); + bckp_idx++; + bckp[bckp_idx].len = 4; + bckp[bckp_idx].reg = REG_RQPN_CTRL_2; + bckp[bckp_idx].val = rtw_read32(rtwdev, REG_RQPN_CTRL_2) | BIT_LD_RQPN; + bckp_idx++; + rtw_write16(rtwdev, REG_FIFOPAGE_INFO_1, 0x200); + rtw_write32(rtwdev, REG_RQPN_CTRL_2, bckp[bckp_idx - 1].val); + + /* Disable beacon related functions */ + tmp = rtw_read8(rtwdev, REG_BCN_CTRL); + bckp[bckp_idx].len = 1; + bckp[bckp_idx].reg = REG_BCN_CTRL; + bckp[bckp_idx].val = tmp; + bckp_idx++; + tmp = (u8)((tmp & (~BIT_EN_BCN_FUNCTION)) | BIT_DIS_TSF_UDT); + rtw_write8(rtwdev, REG_BCN_CTRL, tmp); + + WARN(bckp_idx != DLFW_RESTORE_REG_NUM, "wrong backup number\n"); +} + +static void download_firmware_reset_platform(struct rtw_dev *rtwdev) +{ + rtw_write8_clr(rtwdev, REG_CPU_DMEM_CON + 2, BIT_WL_PLATFORM_RST >> 16); + rtw_write8_clr(rtwdev, REG_SYS_CLK_CTRL + 1, BIT_CPU_CLK_EN >> 8); + rtw_write8_set(rtwdev, REG_CPU_DMEM_CON + 2, BIT_WL_PLATFORM_RST >> 16); + rtw_write8_set(rtwdev, REG_SYS_CLK_CTRL + 1, BIT_CPU_CLK_EN >> 8); +} + +static void download_firmware_reg_restore(struct rtw_dev *rtwdev, + struct rtw_backup_info *bckp, + u8 bckp_num) +{ + rtw_restore_reg(rtwdev, bckp, bckp_num); +} + +#define TX_DESC_SIZE 48 + +static int send_firmware_pkt_rsvd_page(struct rtw_dev *rtwdev, u16 pg_addr, + const u8 *data, u32 size) +{ + u8 *buf; + int ret; + + buf = kmemdup(data, size, GFP_KERNEL); + if (!buf) + return -ENOMEM; + + ret = rtw_fw_write_data_rsvd_page(rtwdev, pg_addr, buf, size); + kfree(buf); + return ret; +} + +static int +send_firmware_pkt(struct rtw_dev *rtwdev, u16 pg_addr, const u8 *data, u32 size) +{ + int ret; + + if (rtw_hci_type(rtwdev) == RTW_HCI_TYPE_USB && + !((size + TX_DESC_SIZE) & (512 - 1))) + size += 1; + + ret = send_firmware_pkt_rsvd_page(rtwdev, pg_addr, data, size); + if (ret) + rtw_err(rtwdev, "failed to download rsvd page\n"); + + return ret; +} + +static int +iddma_enable(struct rtw_dev *rtwdev, u32 src, u32 dst, u32 ctrl) +{ + rtw_write32(rtwdev, REG_DDMA_CH0SA, src); + rtw_write32(rtwdev, REG_DDMA_CH0DA, dst); + rtw_write32(rtwdev, REG_DDMA_CH0CTRL, ctrl); + + if (!check_hw_ready(rtwdev, REG_DDMA_CH0CTRL, BIT_DDMACH0_OWN, 0)) + return -EBUSY; + + return 0; +} + +static int iddma_download_firmware(struct rtw_dev *rtwdev, u32 src, u32 dst, + u32 len, u8 first) +{ + u32 ch0_ctrl = BIT_DDMACH0_CHKSUM_EN | BIT_DDMACH0_OWN; + + if (!check_hw_ready(rtwdev, REG_DDMA_CH0CTRL, BIT_DDMACH0_OWN, 0)) + return -EBUSY; + + ch0_ctrl |= len & BIT_MASK_DDMACH0_DLEN; + if (!first) + ch0_ctrl |= BIT_DDMACH0_CHKSUM_CONT; + + if (iddma_enable(rtwdev, src, dst, ch0_ctrl)) + return -EBUSY; + + return 0; +} + +static bool +check_fw_checksum(struct rtw_dev *rtwdev, u32 addr) +{ + u8 fw_ctrl; + + fw_ctrl = rtw_read8(rtwdev, REG_MCUFW_CTRL); + + if (rtw_read32(rtwdev, REG_DDMA_CH0CTRL) & BIT_DDMACH0_CHKSUM_STS) { + if (addr < OCPBASE_DMEM_88XX) { + fw_ctrl |= BIT_IMEM_DW_OK; + fw_ctrl &= ~BIT_IMEM_CHKSUM_OK; + rtw_write8(rtwdev, REG_MCUFW_CTRL, fw_ctrl); + } else { + fw_ctrl |= BIT_DMEM_DW_OK; + fw_ctrl &= ~BIT_DMEM_CHKSUM_OK; + rtw_write8(rtwdev, REG_MCUFW_CTRL, fw_ctrl); + } + + rtw_err(rtwdev, "invalid fw checksum\n"); + + return false; + } + + if (addr < OCPBASE_DMEM_88XX) { + fw_ctrl |= (BIT_IMEM_DW_OK | BIT_IMEM_CHKSUM_OK); + rtw_write8(rtwdev, REG_MCUFW_CTRL, fw_ctrl); + } else { + fw_ctrl |= (BIT_DMEM_DW_OK | BIT_DMEM_CHKSUM_OK); + rtw_write8(rtwdev, REG_MCUFW_CTRL, fw_ctrl); + } + + return true; +} + +static int +download_firmware_to_mem(struct rtw_dev *rtwdev, const u8 *data, + u32 src, u32 dst, u32 size) +{ + struct rtw_chip_info *chip = rtwdev->chip; + u32 desc_size = chip->tx_pkt_desc_sz; + u8 first_part; + u32 mem_offset; + u32 residue_size; + u32 pkt_size; + u32 max_size = 0x1000; + u32 val; + int ret; + + mem_offset = 0; + first_part = 1; + residue_size = size; + + val = rtw_read32(rtwdev, REG_DDMA_CH0CTRL); + val |= BIT_DDMACH0_RESET_CHKSUM_STS; + rtw_write32(rtwdev, REG_DDMA_CH0CTRL, val); + + while (residue_size) { + if (residue_size >= max_size) + pkt_size = max_size; + else + pkt_size = residue_size; + + ret = send_firmware_pkt(rtwdev, (u16)(src >> 7), + data + mem_offset, pkt_size); + if (ret) + return ret; + + ret = iddma_download_firmware(rtwdev, OCPBASE_TXBUF_88XX + + src + desc_size, + dst + mem_offset, pkt_size, + first_part); + if (ret) + return ret; + + first_part = 0; + mem_offset += pkt_size; + residue_size -= pkt_size; + } + + if (!check_fw_checksum(rtwdev, dst)) + return -EINVAL; + + return 0; +} + +static void update_firmware_info(struct rtw_dev *rtwdev, + struct rtw_fw_state *fw) +{ + const u8 *data = fw->firmware->data; + + fw->h2c_version = + le16_to_cpu(*((__le16 *)(data + FW_HDR_H2C_FMT_VER))); + fw->version = + le16_to_cpu(*((__le16 *)(data + FW_HDR_VERSION))); + fw->sub_version = *(data + FW_HDR_SUBVERSION); + fw->sub_index = *(data + FW_HDR_SUBINDEX); + + rtw_dbg(rtwdev, RTW_DBG_FW, "fw h2c version: %x\n", fw->h2c_version); + rtw_dbg(rtwdev, RTW_DBG_FW, "fw version: %x\n", fw->version); + rtw_dbg(rtwdev, RTW_DBG_FW, "fw sub version: %x\n", fw->sub_version); + rtw_dbg(rtwdev, RTW_DBG_FW, "fw sub index: %x\n", fw->sub_index); +} + +static int +start_download_firmware(struct rtw_dev *rtwdev, const u8 *data, u32 size) +{ + const u8 *cur_fw; + u16 val; + u32 imem_size; + u32 dmem_size; + u32 emem_size; + u32 addr; + int ret; + + dmem_size = le32_to_cpu(*((__le32 *)(data + FW_HDR_DMEM_SIZE))); + imem_size = le32_to_cpu(*((__le32 *)(data + FW_HDR_IMEM_SIZE))); + emem_size = ((*(data + FW_HDR_MEM_USAGE)) & BIT(4)) ? + le32_to_cpu(*((__le32 *)(data + FW_HDR_EMEM_SIZE))) : 0; + dmem_size += FW_HDR_CHKSUM_SIZE; + imem_size += FW_HDR_CHKSUM_SIZE; + emem_size += emem_size ? FW_HDR_CHKSUM_SIZE : 0; + + val = (u16)(rtw_read16(rtwdev, REG_MCUFW_CTRL) & 0x3800); + val |= BIT_MCUFWDL_EN; + rtw_write16(rtwdev, REG_MCUFW_CTRL, val); + + cur_fw = data + FW_HDR_SIZE; + addr = le32_to_cpu(*((__le32 *)(data + FW_HDR_DMEM_ADDR))); + addr &= ~BIT(31); + ret = download_firmware_to_mem(rtwdev, cur_fw, 0, addr, dmem_size); + if (ret) + return ret; + + cur_fw = data + FW_HDR_SIZE + dmem_size; + addr = le32_to_cpu(*((__le32 *)(data + FW_HDR_IMEM_ADDR))); + addr &= ~BIT(31); + ret = download_firmware_to_mem(rtwdev, cur_fw, 0, addr, imem_size); + if (ret) + return ret; + + if (emem_size) { + cur_fw = data + FW_HDR_SIZE + dmem_size + imem_size; + addr = le32_to_cpu(*((__le32 *)(data + FW_HDR_EMEM_ADDR))); + addr &= ~BIT(31); + ret = download_firmware_to_mem(rtwdev, cur_fw, 0, addr, + emem_size); + if (ret) + return ret; + } + + return 0; +} + +static int download_firmware_validate(struct rtw_dev *rtwdev) +{ + u32 fw_key; + + if (!check_hw_ready(rtwdev, REG_MCUFW_CTRL, FW_READY_MASK, FW_READY)) { + fw_key = rtw_read32(rtwdev, REG_FW_DBG7) & FW_KEY_MASK; + if (fw_key == ILLEGAL_KEY_GROUP) + rtw_err(rtwdev, "invalid fw key\n"); + return -EINVAL; + } + + return 0; +} + +static void download_firmware_end_flow(struct rtw_dev *rtwdev) +{ + u16 fw_ctrl; + + rtw_write32(rtwdev, REG_TXDMA_STATUS, BTI_PAGE_OVF); + + /* Check IMEM & DMEM checksum is OK or not */ + fw_ctrl = rtw_read16(rtwdev, REG_MCUFW_CTRL); + if ((fw_ctrl & BIT_CHECK_SUM_OK) != BIT_CHECK_SUM_OK) + return; + + fw_ctrl = (fw_ctrl | BIT_FW_DW_RDY) & ~BIT_MCUFWDL_EN; + rtw_write16(rtwdev, REG_MCUFW_CTRL, fw_ctrl); +} + +int rtw_download_firmware(struct rtw_dev *rtwdev, struct rtw_fw_state *fw) +{ + struct rtw_backup_info bckp[DLFW_RESTORE_REG_NUM]; + const u8 *data = fw->firmware->data; + u32 size = fw->firmware->size; + u32 ltecoex_bckp; + int ret; + + if (!check_firmware_size(data, size)) + return -EINVAL; + + if (!ltecoex_read_reg(rtwdev, 0x38, <ecoex_bckp)) + return -EBUSY; + + wlan_cpu_enable(rtwdev, false); + + download_firmware_reg_backup(rtwdev, bckp); + download_firmware_reset_platform(rtwdev); + + ret = start_download_firmware(rtwdev, data, size); + if (ret) + goto dlfw_fail; + + download_firmware_reg_restore(rtwdev, bckp, DLFW_RESTORE_REG_NUM); + + download_firmware_end_flow(rtwdev); + + wlan_cpu_enable(rtwdev, true); + + if (!ltecoex_reg_write(rtwdev, 0x38, ltecoex_bckp)) + return -EBUSY; + + ret = download_firmware_validate(rtwdev); + if (ret) + goto dlfw_fail; + + update_firmware_info(rtwdev, fw); + + /* reset desc and index */ + rtw_hci_setup(rtwdev); + + rtwdev->h2c.last_box_num = 0; + rtwdev->h2c.seq = 0; + + rtw_fw_send_general_info(rtwdev); + rtw_fw_send_phydm_info(rtwdev); + + rtw_flag_set(rtwdev, RTW_FLAG_FW_RUNNING); + + return 0; + +dlfw_fail: + /* Disable FWDL_EN */ + rtw_write8_clr(rtwdev, REG_MCUFW_CTRL, BIT_MCUFWDL_EN); + rtw_write8_set(rtwdev, REG_SYS_FUNC_EN + 1, BIT_FEN_CPUEN); + + return ret; +} + +static int txdma_queue_mapping(struct rtw_dev *rtwdev) +{ + struct rtw_chip_info *chip = rtwdev->chip; + struct rtw_rqpn *rqpn = NULL; + u16 txdma_pq_map = 0; + + switch (rtw_hci_type(rtwdev)) { + case RTW_HCI_TYPE_PCIE: + rqpn = &chip->rqpn_table[1]; + break; + case RTW_HCI_TYPE_USB: + if (rtwdev->hci.bulkout_num == 2) + rqpn = &chip->rqpn_table[2]; + else if (rtwdev->hci.bulkout_num == 3) + rqpn = &chip->rqpn_table[3]; + else if (rtwdev->hci.bulkout_num == 4) + rqpn = &chip->rqpn_table[4]; + else + return -EINVAL; + break; + default: + return -EINVAL; + } + + txdma_pq_map |= BIT_TXDMA_HIQ_MAP(rqpn->dma_map_hi); + txdma_pq_map |= BIT_TXDMA_MGQ_MAP(rqpn->dma_map_mg); + txdma_pq_map |= BIT_TXDMA_BKQ_MAP(rqpn->dma_map_bk); + txdma_pq_map |= BIT_TXDMA_BEQ_MAP(rqpn->dma_map_be); + txdma_pq_map |= BIT_TXDMA_VIQ_MAP(rqpn->dma_map_vi); + txdma_pq_map |= BIT_TXDMA_VOQ_MAP(rqpn->dma_map_vo); + rtw_write16(rtwdev, REG_TXDMA_PQ_MAP, txdma_pq_map); + + rtw_write8(rtwdev, REG_CR, 0); + rtw_write8(rtwdev, REG_CR, MAC_TRX_ENABLE); + rtw_write32(rtwdev, REG_H2CQ_CSR, BIT_H2CQ_FULL); + + return 0; +} + +static int set_trx_fifo_info(struct rtw_dev *rtwdev) +{ + struct rtw_fifo_conf *fifo = &rtwdev->fifo; + struct rtw_chip_info *chip = rtwdev->chip; + u16 cur_pg_addr; + u8 csi_buf_pg_num = chip->csi_buf_pg_num; + + /* config rsvd page num */ + fifo->rsvd_drv_pg_num = 8; + fifo->txff_pg_num = chip->txff_size >> 7; + fifo->rsvd_pg_num = fifo->rsvd_drv_pg_num + + RSVD_PG_H2C_EXTRAINFO_NUM + + RSVD_PG_H2C_STATICINFO_NUM + + RSVD_PG_H2CQ_NUM + + RSVD_PG_CPU_INSTRUCTION_NUM + + RSVD_PG_FW_TXBUF_NUM + + csi_buf_pg_num; + + if (fifo->rsvd_pg_num > fifo->txff_pg_num) + return -ENOMEM; + + fifo->acq_pg_num = fifo->txff_pg_num - fifo->rsvd_pg_num; + fifo->rsvd_boundary = fifo->txff_pg_num - fifo->rsvd_pg_num; + + cur_pg_addr = fifo->txff_pg_num; + cur_pg_addr -= csi_buf_pg_num; + fifo->rsvd_csibuf_addr = cur_pg_addr; + cur_pg_addr -= RSVD_PG_FW_TXBUF_NUM; + fifo->rsvd_fw_txbuf_addr = cur_pg_addr; + cur_pg_addr -= RSVD_PG_CPU_INSTRUCTION_NUM; + fifo->rsvd_cpu_instr_addr = cur_pg_addr; + cur_pg_addr -= RSVD_PG_H2CQ_NUM; + fifo->rsvd_h2cq_addr = cur_pg_addr; + cur_pg_addr -= RSVD_PG_H2C_STATICINFO_NUM; + fifo->rsvd_h2c_sta_info_addr = cur_pg_addr; + cur_pg_addr -= RSVD_PG_H2C_EXTRAINFO_NUM; + fifo->rsvd_h2c_info_addr = cur_pg_addr; + cur_pg_addr -= fifo->rsvd_drv_pg_num; + fifo->rsvd_drv_addr = cur_pg_addr; + + if (fifo->rsvd_boundary != fifo->rsvd_drv_addr) { + rtw_err(rtwdev, "wrong rsvd driver address\n"); + return -EINVAL; + } + + return 0; +} + +static int priority_queue_cfg(struct rtw_dev *rtwdev) +{ + struct rtw_fifo_conf *fifo = &rtwdev->fifo; + struct rtw_chip_info *chip = rtwdev->chip; + struct rtw_page_table *pg_tbl = NULL; + u16 pubq_num; + int ret; + + ret = set_trx_fifo_info(rtwdev); + if (ret) + return ret; + + switch (rtw_hci_type(rtwdev)) { + case RTW_HCI_TYPE_PCIE: + pg_tbl = &chip->page_table[1]; + break; + case RTW_HCI_TYPE_USB: + if (rtwdev->hci.bulkout_num == 2) + pg_tbl = &chip->page_table[2]; + else if (rtwdev->hci.bulkout_num == 3) + pg_tbl = &chip->page_table[3]; + else if (rtwdev->hci.bulkout_num == 4) + pg_tbl = &chip->page_table[4]; + else + return -EINVAL; + break; + default: + return -EINVAL; + } + + pubq_num = fifo->acq_pg_num - pg_tbl->hq_num - pg_tbl->lq_num - + pg_tbl->nq_num - pg_tbl->exq_num - pg_tbl->gapq_num; + rtw_write16(rtwdev, REG_FIFOPAGE_INFO_1, pg_tbl->hq_num); + rtw_write16(rtwdev, REG_FIFOPAGE_INFO_2, pg_tbl->lq_num); + rtw_write16(rtwdev, REG_FIFOPAGE_INFO_3, pg_tbl->nq_num); + rtw_write16(rtwdev, REG_FIFOPAGE_INFO_4, pg_tbl->exq_num); + rtw_write16(rtwdev, REG_FIFOPAGE_INFO_5, pubq_num); + rtw_write32_set(rtwdev, REG_RQPN_CTRL_2, BIT_LD_RQPN); + + rtw_write16(rtwdev, REG_FIFOPAGE_CTRL_2, fifo->rsvd_boundary); + rtw_write8_set(rtwdev, REG_FWHW_TXQ_CTRL + 2, BIT_EN_WR_FREE_TAIL >> 16); + + rtw_write16(rtwdev, REG_BCNQ_BDNY_V1, fifo->rsvd_boundary); + rtw_write16(rtwdev, REG_FIFOPAGE_CTRL_2 + 2, fifo->rsvd_boundary); + rtw_write16(rtwdev, REG_BCNQ1_BDNY_V1, fifo->rsvd_boundary); + rtw_write32(rtwdev, REG_RXFF_BNDY, chip->rxff_size - C2H_PKT_BUF - 1); + rtw_write8_set(rtwdev, REG_AUTO_LLT_V1, BIT_AUTO_INIT_LLT_V1); + + if (!check_hw_ready(rtwdev, REG_AUTO_LLT_V1, BIT_AUTO_INIT_LLT_V1, 0)) + return -EBUSY; + + rtw_write8(rtwdev, REG_CR + 3, 0); + + return 0; +} + +static int init_h2c(struct rtw_dev *rtwdev) +{ + struct rtw_fifo_conf *fifo = &rtwdev->fifo; + u8 value8; + u32 value32; + u32 h2cq_addr; + u32 h2cq_size; + u32 h2cq_free; + u32 wp, rp; + + h2cq_addr = fifo->rsvd_h2cq_addr << TX_PAGE_SIZE_SHIFT; + h2cq_size = RSVD_PG_H2CQ_NUM << TX_PAGE_SIZE_SHIFT; + + value32 = rtw_read32(rtwdev, REG_H2C_HEAD); + value32 = (value32 & 0xFFFC0000) | h2cq_addr; + rtw_write32(rtwdev, REG_H2C_HEAD, value32); + + value32 = rtw_read32(rtwdev, REG_H2C_READ_ADDR); + value32 = (value32 & 0xFFFC0000) | h2cq_addr; + rtw_write32(rtwdev, REG_H2C_READ_ADDR, value32); + + value32 = rtw_read32(rtwdev, REG_H2C_TAIL); + value32 &= 0xFFFC0000; + value32 |= (h2cq_addr + h2cq_size); + rtw_write32(rtwdev, REG_H2C_TAIL, value32); + + value8 = rtw_read8(rtwdev, REG_H2C_INFO); + value8 = (u8)((value8 & 0xFC) | 0x01); + rtw_write8(rtwdev, REG_H2C_INFO, value8); + + value8 = rtw_read8(rtwdev, REG_H2C_INFO); + value8 = (u8)((value8 & 0xFB) | 0x04); + rtw_write8(rtwdev, REG_H2C_INFO, value8); + + value8 = rtw_read8(rtwdev, REG_TXDMA_OFFSET_CHK + 1); + value8 = (u8)((value8 & 0x7f) | 0x80); + rtw_write8(rtwdev, REG_TXDMA_OFFSET_CHK + 1, value8); + + wp = rtw_read32(rtwdev, REG_H2C_PKT_WRITEADDR) & 0x3FFFF; + rp = rtw_read32(rtwdev, REG_H2C_PKT_READADDR) & 0x3FFFF; + h2cq_free = wp >= rp ? h2cq_size - (wp - rp) : rp - wp; + + if (h2cq_size != h2cq_free) { + rtw_err(rtwdev, "H2C queue mismatch\n"); + return -EINVAL; + } + + return 0; +} + +static int rtw_init_trx_cfg(struct rtw_dev *rtwdev) +{ + int ret; + + ret = txdma_queue_mapping(rtwdev); + if (ret) + return ret; + + ret = priority_queue_cfg(rtwdev); + if (ret) + return ret; + + ret = init_h2c(rtwdev); + if (ret) + return ret; + + return 0; +} + +static int rtw_drv_info_cfg(struct rtw_dev *rtwdev) +{ + u8 value8; + + rtw_write8(rtwdev, REG_RX_DRVINFO_SZ, PHY_STATUS_SIZE); + value8 = rtw_read8(rtwdev, REG_TRXFF_BNDY + 1); + value8 &= 0xF0; + /* For rxdesc len = 0 issue */ + value8 |= 0xF; + rtw_write8(rtwdev, REG_TRXFF_BNDY + 1, value8); + rtw_write32_set(rtwdev, REG_RCR, BIT_APP_PHYSTS); + rtw_write32_clr(rtwdev, REG_WMAC_OPTION_FUNCTION + 4, BIT(8) | BIT(9)); + + return 0; +} + +int rtw_mac_init(struct rtw_dev *rtwdev) +{ + struct rtw_chip_info *chip = rtwdev->chip; + int ret; + + ret = rtw_init_trx_cfg(rtwdev); + if (ret) + return ret; + + ret = chip->ops->mac_init(rtwdev); + if (ret) + return ret; + + ret = rtw_drv_info_cfg(rtwdev); + if (ret) + return ret; + + return 0; +} diff --git a/drivers/net/wireless/realtek/rtw88/mac.h b/drivers/net/wireless/realtek/rtw88/mac.h new file mode 100644 index 0000000000000000000000000000000000000000..efe6f731f240e6e9ce9f159335a236247418110c --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/mac.h @@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#ifndef __RTW_MAC_H__ +#define __RTW_MAC_H__ + +#define RTW_HW_PORT_NUM 5 +#define cut_version_to_mask(cut) (0x1 << ((cut) + 1)) +#define SDIO_LOCAL_OFFSET 0x10250000 +#define DDMA_POLLING_COUNT 1000 +#define C2H_PKT_BUF 256 +#define PHY_STATUS_SIZE 4 +#define ILLEGAL_KEY_GROUP 0xFAAAAA00 + +/* HW memory address */ +#define OCPBASE_TXBUF_88XX 0x18780000 +#define OCPBASE_DMEM_88XX 0x00200000 +#define OCPBASE_EMEM_88XX 0x00100000 + +#define RSVD_PG_DRV_NUM 16 +#define RSVD_PG_H2C_EXTRAINFO_NUM 24 +#define RSVD_PG_H2C_STATICINFO_NUM 8 +#define RSVD_PG_H2CQ_NUM 8 +#define RSVD_PG_CPU_INSTRUCTION_NUM 0 +#define RSVD_PG_FW_TXBUF_NUM 4 + +void rtw_set_channel_mac(struct rtw_dev *rtwdev, u8 channel, u8 bw, + u8 primary_ch_idx); +int rtw_mac_power_on(struct rtw_dev *rtwdev); +void rtw_mac_power_off(struct rtw_dev *rtwdev); +int rtw_download_firmware(struct rtw_dev *rtwdev, struct rtw_fw_state *fw); +int rtw_mac_init(struct rtw_dev *rtwdev); + +#endif diff --git a/drivers/net/wireless/realtek/rtw88/mac80211.c b/drivers/net/wireless/realtek/rtw88/mac80211.c new file mode 100644 index 0000000000000000000000000000000000000000..abded63f138d4e21d3f0c63b46b0fc69684a1e96 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/mac80211.c @@ -0,0 +1,481 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#include "main.h" +#include "sec.h" +#include "tx.h" +#include "fw.h" +#include "mac.h" +#include "ps.h" +#include "reg.h" +#include "debug.h" + +static void rtw_ops_tx(struct ieee80211_hw *hw, + struct ieee80211_tx_control *control, + struct sk_buff *skb) +{ + struct rtw_dev *rtwdev = hw->priv; + struct rtw_tx_pkt_info pkt_info = {0}; + + if (!rtw_flag_check(rtwdev, RTW_FLAG_RUNNING)) + goto out; + + rtw_tx_pkt_info_update(rtwdev, &pkt_info, control, skb); + if (rtw_hci_tx(rtwdev, &pkt_info, skb)) + goto out; + + return; + +out: + ieee80211_free_txskb(hw, skb); +} + +static int rtw_ops_start(struct ieee80211_hw *hw) +{ + struct rtw_dev *rtwdev = hw->priv; + int ret; + + mutex_lock(&rtwdev->mutex); + ret = rtw_core_start(rtwdev); + mutex_unlock(&rtwdev->mutex); + + return ret; +} + +static void rtw_ops_stop(struct ieee80211_hw *hw) +{ + struct rtw_dev *rtwdev = hw->priv; + + mutex_lock(&rtwdev->mutex); + rtw_core_stop(rtwdev); + mutex_unlock(&rtwdev->mutex); +} + +static int rtw_ops_config(struct ieee80211_hw *hw, u32 changed) +{ + struct rtw_dev *rtwdev = hw->priv; + int ret = 0; + + mutex_lock(&rtwdev->mutex); + + if (changed & IEEE80211_CONF_CHANGE_IDLE) { + if (hw->conf.flags & IEEE80211_CONF_IDLE) { + rtw_enter_ips(rtwdev); + } else { + ret = rtw_leave_ips(rtwdev); + if (ret) { + rtw_err(rtwdev, "failed to leave idle state\n"); + goto out; + } + } + } + + if (changed & IEEE80211_CONF_CHANGE_CHANNEL) + rtw_set_channel(rtwdev); + +out: + mutex_unlock(&rtwdev->mutex); + return ret; +} + +static const struct rtw_vif_port rtw_vif_port[] = { + [0] = { + .mac_addr = {.addr = 0x0610}, + .bssid = {.addr = 0x0618}, + .net_type = {.addr = 0x0100, .mask = 0x30000}, + .aid = {.addr = 0x06a8, .mask = 0x7ff}, + }, + [1] = { + .mac_addr = {.addr = 0x0700}, + .bssid = {.addr = 0x0708}, + .net_type = {.addr = 0x0100, .mask = 0xc0000}, + .aid = {.addr = 0x0710, .mask = 0x7ff}, + }, + [2] = { + .mac_addr = {.addr = 0x1620}, + .bssid = {.addr = 0x1628}, + .net_type = {.addr = 0x1100, .mask = 0x3}, + .aid = {.addr = 0x1600, .mask = 0x7ff}, + }, + [3] = { + .mac_addr = {.addr = 0x1630}, + .bssid = {.addr = 0x1638}, + .net_type = {.addr = 0x1100, .mask = 0xc}, + .aid = {.addr = 0x1604, .mask = 0x7ff}, + }, + [4] = { + .mac_addr = {.addr = 0x1640}, + .bssid = {.addr = 0x1648}, + .net_type = {.addr = 0x1100, .mask = 0x30}, + .aid = {.addr = 0x1608, .mask = 0x7ff}, + }, +}; + +static int rtw_ops_add_interface(struct ieee80211_hw *hw, + struct ieee80211_vif *vif) +{ + struct rtw_dev *rtwdev = hw->priv; + struct rtw_vif *rtwvif = (struct rtw_vif *)vif->drv_priv; + enum rtw_net_type net_type; + u32 config = 0; + u8 port = 0; + + rtwvif->port = port; + rtwvif->vif = vif; + rtwvif->stats.tx_unicast = 0; + rtwvif->stats.rx_unicast = 0; + rtwvif->stats.tx_cnt = 0; + rtwvif->stats.rx_cnt = 0; + rtwvif->in_lps = false; + rtwvif->conf = &rtw_vif_port[port]; + + mutex_lock(&rtwdev->mutex); + + switch (vif->type) { + case NL80211_IFTYPE_AP: + case NL80211_IFTYPE_MESH_POINT: + net_type = RTW_NET_AP_MODE; + break; + case NL80211_IFTYPE_ADHOC: + net_type = RTW_NET_AD_HOC; + break; + case NL80211_IFTYPE_STATION: + default: + net_type = RTW_NET_NO_LINK; + break; + } + + ether_addr_copy(rtwvif->mac_addr, vif->addr); + config |= PORT_SET_MAC_ADDR; + rtwvif->net_type = net_type; + config |= PORT_SET_NET_TYPE; + rtw_vif_port_config(rtwdev, rtwvif, config); + + mutex_unlock(&rtwdev->mutex); + + rtw_info(rtwdev, "start vif %pM on port %d\n", vif->addr, rtwvif->port); + return 0; +} + +static void rtw_ops_remove_interface(struct ieee80211_hw *hw, + struct ieee80211_vif *vif) +{ + struct rtw_dev *rtwdev = hw->priv; + struct rtw_vif *rtwvif = (struct rtw_vif *)vif->drv_priv; + u32 config = 0; + + rtw_info(rtwdev, "stop vif %pM on port %d\n", vif->addr, rtwvif->port); + + mutex_lock(&rtwdev->mutex); + + eth_zero_addr(rtwvif->mac_addr); + config |= PORT_SET_MAC_ADDR; + rtwvif->net_type = RTW_NET_NO_LINK; + config |= PORT_SET_NET_TYPE; + rtw_vif_port_config(rtwdev, rtwvif, config); + + mutex_unlock(&rtwdev->mutex); +} + +static void rtw_ops_configure_filter(struct ieee80211_hw *hw, + unsigned int changed_flags, + unsigned int *new_flags, + u64 multicast) +{ + struct rtw_dev *rtwdev = hw->priv; + + *new_flags &= FIF_ALLMULTI | FIF_OTHER_BSS | FIF_FCSFAIL | + FIF_BCN_PRBRESP_PROMISC; + + mutex_lock(&rtwdev->mutex); + + if (changed_flags & FIF_ALLMULTI) { + if (*new_flags & FIF_ALLMULTI) + rtwdev->hal.rcr |= BIT_AM | BIT_AB; + else + rtwdev->hal.rcr &= ~(BIT_AM | BIT_AB); + } + if (changed_flags & FIF_FCSFAIL) { + if (*new_flags & FIF_FCSFAIL) + rtwdev->hal.rcr |= BIT_ACRC32; + else + rtwdev->hal.rcr &= ~(BIT_ACRC32); + } + if (changed_flags & FIF_OTHER_BSS) { + if (*new_flags & FIF_OTHER_BSS) + rtwdev->hal.rcr |= BIT_AAP; + else + rtwdev->hal.rcr &= ~(BIT_AAP); + } + if (changed_flags & FIF_BCN_PRBRESP_PROMISC) { + if (*new_flags & FIF_BCN_PRBRESP_PROMISC) + rtwdev->hal.rcr &= ~(BIT_CBSSID_BCN | BIT_CBSSID_DATA); + else + rtwdev->hal.rcr |= BIT_CBSSID_BCN; + } + + rtw_dbg(rtwdev, RTW_DBG_RX, + "config rx filter, changed=0x%08x, new=0x%08x, rcr=0x%08x\n", + changed_flags, *new_flags, rtwdev->hal.rcr); + + rtw_write32(rtwdev, REG_RCR, rtwdev->hal.rcr); + + mutex_unlock(&rtwdev->mutex); +} + +static void rtw_ops_bss_info_changed(struct ieee80211_hw *hw, + struct ieee80211_vif *vif, + struct ieee80211_bss_conf *conf, + u32 changed) +{ + struct rtw_dev *rtwdev = hw->priv; + struct rtw_vif *rtwvif = (struct rtw_vif *)vif->drv_priv; + u32 config = 0; + + mutex_lock(&rtwdev->mutex); + + if (changed & BSS_CHANGED_ASSOC) { + struct rtw_chip_info *chip = rtwdev->chip; + enum rtw_net_type net_type; + + if (conf->assoc) { + net_type = RTW_NET_MGD_LINKED; + chip->ops->do_iqk(rtwdev); + + rtwvif->aid = conf->aid; + rtw_add_rsvd_page(rtwdev, RSVD_PS_POLL, true); + rtw_add_rsvd_page(rtwdev, RSVD_QOS_NULL, true); + rtw_add_rsvd_page(rtwdev, RSVD_NULL, true); + rtw_fw_download_rsvd_page(rtwdev, vif); + rtw_send_rsvd_page_h2c(rtwdev); + } else { + net_type = RTW_NET_NO_LINK; + rtwvif->aid = 0; + rtw_reset_rsvd_page(rtwdev); + } + + rtwvif->net_type = net_type; + config |= PORT_SET_NET_TYPE; + config |= PORT_SET_AID; + } + + if (changed & BSS_CHANGED_BSSID) { + ether_addr_copy(rtwvif->bssid, conf->bssid); + config |= PORT_SET_BSSID; + } + + if (changed & BSS_CHANGED_BEACON) + rtw_fw_download_rsvd_page(rtwdev, vif); + + rtw_vif_port_config(rtwdev, rtwvif, config); + + mutex_unlock(&rtwdev->mutex); +} + +static u8 rtw_acquire_macid(struct rtw_dev *rtwdev) +{ + unsigned long mac_id; + + mac_id = find_first_zero_bit(rtwdev->mac_id_map, RTW_MAX_MAC_ID_NUM); + if (mac_id < RTW_MAX_MAC_ID_NUM) + set_bit(mac_id, rtwdev->mac_id_map); + + return mac_id; +} + +static void rtw_release_macid(struct rtw_dev *rtwdev, u8 mac_id) +{ + clear_bit(mac_id, rtwdev->mac_id_map); +} + +static int rtw_ops_sta_add(struct ieee80211_hw *hw, + struct ieee80211_vif *vif, + struct ieee80211_sta *sta) +{ + struct rtw_dev *rtwdev = hw->priv; + struct rtw_sta_info *si = (struct rtw_sta_info *)sta->drv_priv; + int ret = 0; + + mutex_lock(&rtwdev->mutex); + + si->mac_id = rtw_acquire_macid(rtwdev); + if (si->mac_id >= RTW_MAX_MAC_ID_NUM) { + ret = -ENOSPC; + goto out; + } + + si->sta = sta; + si->vif = vif; + si->init_ra_lv = 1; + ewma_rssi_init(&si->avg_rssi); + + rtw_update_sta_info(rtwdev, si); + rtw_fw_media_status_report(rtwdev, si->mac_id, true); + + rtwdev->sta_cnt++; + + rtw_info(rtwdev, "sta %pM joined with macid %d\n", + sta->addr, si->mac_id); + +out: + mutex_unlock(&rtwdev->mutex); + return ret; +} + +static int rtw_ops_sta_remove(struct ieee80211_hw *hw, + struct ieee80211_vif *vif, + struct ieee80211_sta *sta) +{ + struct rtw_dev *rtwdev = hw->priv; + struct rtw_sta_info *si = (struct rtw_sta_info *)sta->drv_priv; + + mutex_lock(&rtwdev->mutex); + + rtw_release_macid(rtwdev, si->mac_id); + rtw_fw_media_status_report(rtwdev, si->mac_id, false); + + rtwdev->sta_cnt--; + + rtw_info(rtwdev, "sta %pM with macid %d left\n", + sta->addr, si->mac_id); + + mutex_unlock(&rtwdev->mutex); + return 0; +} + +static int rtw_ops_set_key(struct ieee80211_hw *hw, enum set_key_cmd cmd, + struct ieee80211_vif *vif, struct ieee80211_sta *sta, + struct ieee80211_key_conf *key) +{ + struct rtw_dev *rtwdev = hw->priv; + struct rtw_sec_desc *sec = &rtwdev->sec; + u8 hw_key_type; + u8 hw_key_idx; + int ret = 0; + + switch (key->cipher) { + case WLAN_CIPHER_SUITE_WEP40: + hw_key_type = RTW_CAM_WEP40; + break; + case WLAN_CIPHER_SUITE_WEP104: + hw_key_type = RTW_CAM_WEP104; + break; + case WLAN_CIPHER_SUITE_TKIP: + hw_key_type = RTW_CAM_TKIP; + key->flags |= IEEE80211_KEY_FLAG_GENERATE_MMIC; + break; + case WLAN_CIPHER_SUITE_CCMP: + hw_key_type = RTW_CAM_AES; + key->flags |= IEEE80211_KEY_FLAG_SW_MGMT_TX; + break; + case WLAN_CIPHER_SUITE_AES_CMAC: + case WLAN_CIPHER_SUITE_BIP_CMAC_256: + case WLAN_CIPHER_SUITE_BIP_GMAC_128: + case WLAN_CIPHER_SUITE_BIP_GMAC_256: + /* suppress error messages */ + return -EOPNOTSUPP; + default: + return -ENOTSUPP; + } + + mutex_lock(&rtwdev->mutex); + + if (key->flags & IEEE80211_KEY_FLAG_PAIRWISE) { + hw_key_idx = rtw_sec_get_free_cam(sec); + } else { + /* multiple interfaces? */ + hw_key_idx = key->keyidx; + } + + if (hw_key_idx > sec->total_cam_num) { + ret = -ENOSPC; + goto out; + } + + switch (cmd) { + case SET_KEY: + /* need sw generated IV */ + key->flags |= IEEE80211_KEY_FLAG_GENERATE_IV; + key->hw_key_idx = hw_key_idx; + rtw_sec_write_cam(rtwdev, sec, sta, key, + hw_key_type, hw_key_idx); + break; + case DISABLE_KEY: + rtw_sec_clear_cam(rtwdev, sec, key->hw_key_idx); + break; + } + +out: + mutex_unlock(&rtwdev->mutex); + + return ret; +} + +static int rtw_ops_ampdu_action(struct ieee80211_hw *hw, + struct ieee80211_vif *vif, + struct ieee80211_ampdu_params *params) +{ + struct ieee80211_sta *sta = params->sta; + u16 tid = params->tid; + + switch (params->action) { + case IEEE80211_AMPDU_TX_START: + ieee80211_start_tx_ba_cb_irqsafe(vif, sta->addr, tid); + break; + case IEEE80211_AMPDU_TX_STOP_CONT: + case IEEE80211_AMPDU_TX_STOP_FLUSH: + case IEEE80211_AMPDU_TX_STOP_FLUSH_CONT: + ieee80211_stop_tx_ba_cb_irqsafe(vif, sta->addr, tid); + break; + case IEEE80211_AMPDU_TX_OPERATIONAL: + case IEEE80211_AMPDU_RX_START: + case IEEE80211_AMPDU_RX_STOP: + break; + default: + WARN_ON(1); + return -ENOTSUPP; + } + + return 0; +} + +static void rtw_ops_sw_scan_start(struct ieee80211_hw *hw, + struct ieee80211_vif *vif, + const u8 *mac_addr) +{ + struct rtw_dev *rtwdev = hw->priv; + struct rtw_vif *rtwvif = (struct rtw_vif *)vif->drv_priv; + + rtw_leave_lps(rtwdev, rtwvif); + + rtw_flag_set(rtwdev, RTW_FLAG_DIG_DISABLE); + rtw_flag_set(rtwdev, RTW_FLAG_SCANNING); +} + +static void rtw_ops_sw_scan_complete(struct ieee80211_hw *hw, + struct ieee80211_vif *vif) +{ + struct rtw_dev *rtwdev = hw->priv; + + rtw_flag_clear(rtwdev, RTW_FLAG_SCANNING); + rtw_flag_clear(rtwdev, RTW_FLAG_DIG_DISABLE); +} + +const struct ieee80211_ops rtw_ops = { + .tx = rtw_ops_tx, + .start = rtw_ops_start, + .stop = rtw_ops_stop, + .config = rtw_ops_config, + .add_interface = rtw_ops_add_interface, + .remove_interface = rtw_ops_remove_interface, + .configure_filter = rtw_ops_configure_filter, + .bss_info_changed = rtw_ops_bss_info_changed, + .sta_add = rtw_ops_sta_add, + .sta_remove = rtw_ops_sta_remove, + .set_key = rtw_ops_set_key, + .ampdu_action = rtw_ops_ampdu_action, + .sw_scan_start = rtw_ops_sw_scan_start, + .sw_scan_complete = rtw_ops_sw_scan_complete, +}; +EXPORT_SYMBOL(rtw_ops); diff --git a/drivers/net/wireless/realtek/rtw88/main.c b/drivers/net/wireless/realtek/rtw88/main.c new file mode 100644 index 0000000000000000000000000000000000000000..f447361f75734f37baa9a0fba35789535641c40b --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/main.c @@ -0,0 +1,1211 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#include "main.h" +#include "regd.h" +#include "fw.h" +#include "ps.h" +#include "sec.h" +#include "mac.h" +#include "phy.h" +#include "reg.h" +#include "efuse.h" +#include "debug.h" + +static bool rtw_fw_support_lps; +unsigned int rtw_debug_mask; +EXPORT_SYMBOL(rtw_debug_mask); + +module_param_named(support_lps, rtw_fw_support_lps, bool, 0644); +module_param_named(debug_mask, rtw_debug_mask, uint, 0644); + +MODULE_PARM_DESC(support_lps, "Set Y to enable LPS support"); +MODULE_PARM_DESC(debug_mask, "Debugging mask"); + +static struct ieee80211_channel rtw_channeltable_2g[] = { + {.center_freq = 2412, .hw_value = 1,}, + {.center_freq = 2417, .hw_value = 2,}, + {.center_freq = 2422, .hw_value = 3,}, + {.center_freq = 2427, .hw_value = 4,}, + {.center_freq = 2432, .hw_value = 5,}, + {.center_freq = 2437, .hw_value = 6,}, + {.center_freq = 2442, .hw_value = 7,}, + {.center_freq = 2447, .hw_value = 8,}, + {.center_freq = 2452, .hw_value = 9,}, + {.center_freq = 2457, .hw_value = 10,}, + {.center_freq = 2462, .hw_value = 11,}, + {.center_freq = 2467, .hw_value = 12,}, + {.center_freq = 2472, .hw_value = 13,}, + {.center_freq = 2484, .hw_value = 14,}, +}; + +static struct ieee80211_channel rtw_channeltable_5g[] = { + {.center_freq = 5180, .hw_value = 36,}, + {.center_freq = 5200, .hw_value = 40,}, + {.center_freq = 5220, .hw_value = 44,}, + {.center_freq = 5240, .hw_value = 48,}, + {.center_freq = 5260, .hw_value = 52,}, + {.center_freq = 5280, .hw_value = 56,}, + {.center_freq = 5300, .hw_value = 60,}, + {.center_freq = 5320, .hw_value = 64,}, + {.center_freq = 5500, .hw_value = 100,}, + {.center_freq = 5520, .hw_value = 104,}, + {.center_freq = 5540, .hw_value = 108,}, + {.center_freq = 5560, .hw_value = 112,}, + {.center_freq = 5580, .hw_value = 116,}, + {.center_freq = 5600, .hw_value = 120,}, + {.center_freq = 5620, .hw_value = 124,}, + {.center_freq = 5640, .hw_value = 128,}, + {.center_freq = 5660, .hw_value = 132,}, + {.center_freq = 5680, .hw_value = 136,}, + {.center_freq = 5700, .hw_value = 140,}, + {.center_freq = 5745, .hw_value = 149,}, + {.center_freq = 5765, .hw_value = 153,}, + {.center_freq = 5785, .hw_value = 157,}, + {.center_freq = 5805, .hw_value = 161,}, + {.center_freq = 5825, .hw_value = 165, + .flags = IEEE80211_CHAN_NO_HT40MINUS}, +}; + +static struct ieee80211_rate rtw_ratetable[] = { + {.bitrate = 10, .hw_value = 0x00,}, + {.bitrate = 20, .hw_value = 0x01,}, + {.bitrate = 55, .hw_value = 0x02,}, + {.bitrate = 110, .hw_value = 0x03,}, + {.bitrate = 60, .hw_value = 0x04,}, + {.bitrate = 90, .hw_value = 0x05,}, + {.bitrate = 120, .hw_value = 0x06,}, + {.bitrate = 180, .hw_value = 0x07,}, + {.bitrate = 240, .hw_value = 0x08,}, + {.bitrate = 360, .hw_value = 0x09,}, + {.bitrate = 480, .hw_value = 0x0a,}, + {.bitrate = 540, .hw_value = 0x0b,}, +}; + +static struct ieee80211_supported_band rtw_band_2ghz = { + .band = NL80211_BAND_2GHZ, + + .channels = rtw_channeltable_2g, + .n_channels = ARRAY_SIZE(rtw_channeltable_2g), + + .bitrates = rtw_ratetable, + .n_bitrates = ARRAY_SIZE(rtw_ratetable), + + .ht_cap = {0}, + .vht_cap = {0}, +}; + +static struct ieee80211_supported_band rtw_band_5ghz = { + .band = NL80211_BAND_5GHZ, + + .channels = rtw_channeltable_5g, + .n_channels = ARRAY_SIZE(rtw_channeltable_5g), + + /* 5G has no CCK rates */ + .bitrates = rtw_ratetable + 4, + .n_bitrates = ARRAY_SIZE(rtw_ratetable) - 4, + + .ht_cap = {0}, + .vht_cap = {0}, +}; + +struct rtw_watch_dog_iter_data { + struct rtw_vif *rtwvif; + bool active; + u8 assoc_cnt; +}; + +static void rtw_vif_watch_dog_iter(void *data, u8 *mac, + struct ieee80211_vif *vif) +{ + struct rtw_watch_dog_iter_data *iter_data = data; + struct rtw_vif *rtwvif = (struct rtw_vif *)vif->drv_priv; + + if (vif->type == NL80211_IFTYPE_STATION) { + if (vif->bss_conf.assoc) { + iter_data->assoc_cnt++; + iter_data->rtwvif = rtwvif; + } + if (rtwvif->stats.tx_cnt > RTW_LPS_THRESHOLD || + rtwvif->stats.rx_cnt > RTW_LPS_THRESHOLD) + iter_data->active = true; + } else { + /* only STATION mode can enter lps */ + iter_data->active = true; + } + + rtwvif->stats.tx_unicast = 0; + rtwvif->stats.rx_unicast = 0; + rtwvif->stats.tx_cnt = 0; + rtwvif->stats.rx_cnt = 0; +} + +/* process TX/RX statistics periodically for hardware, + * the information helps hardware to enhance performance + */ +static void rtw_watch_dog_work(struct work_struct *work) +{ + struct rtw_dev *rtwdev = container_of(work, struct rtw_dev, + watch_dog_work.work); + struct rtw_watch_dog_iter_data data = {}; + + if (!rtw_flag_check(rtwdev, RTW_FLAG_RUNNING)) + return; + + ieee80211_queue_delayed_work(rtwdev->hw, &rtwdev->watch_dog_work, + RTW_WATCH_DOG_DELAY_TIME); + + /* reset tx/rx statictics */ + rtwdev->stats.tx_unicast = 0; + rtwdev->stats.rx_unicast = 0; + rtwdev->stats.tx_cnt = 0; + rtwdev->stats.rx_cnt = 0; + + rtw_iterate_vifs(rtwdev, rtw_vif_watch_dog_iter, &data); + + /* fw supports only one station associated to enter lps, if there are + * more than two stations associated to the AP, then we can not enter + * lps, because fw does not handle the overlapped beacon interval + */ + if (rtw_fw_support_lps && + data.rtwvif && !data.active && data.assoc_cnt == 1) + rtw_enter_lps(rtwdev, data.rtwvif); + + if (rtw_flag_check(rtwdev, RTW_FLAG_SCANNING)) + return; + + rtw_phy_dynamic_mechanism(rtwdev); + + rtwdev->watch_dog_cnt++; +} + +static void rtw_c2h_work(struct work_struct *work) +{ + struct rtw_dev *rtwdev = container_of(work, struct rtw_dev, c2h_work); + struct sk_buff *skb, *tmp; + + skb_queue_walk_safe(&rtwdev->c2h_queue, skb, tmp) { + skb_unlink(skb, &rtwdev->c2h_queue); + rtw_fw_c2h_cmd_handle(rtwdev, skb); + dev_kfree_skb_any(skb); + } +} + +void rtw_get_channel_params(struct cfg80211_chan_def *chandef, + struct rtw_channel_params *chan_params) +{ + struct ieee80211_channel *channel = chandef->chan; + enum nl80211_chan_width width = chandef->width; + u32 primary_freq, center_freq; + u8 center_chan; + u8 bandwidth = RTW_CHANNEL_WIDTH_20; + u8 primary_chan_idx = 0; + + center_chan = channel->hw_value; + primary_freq = channel->center_freq; + center_freq = chandef->center_freq1; + + switch (width) { + case NL80211_CHAN_WIDTH_20_NOHT: + case NL80211_CHAN_WIDTH_20: + bandwidth = RTW_CHANNEL_WIDTH_20; + primary_chan_idx = 0; + break; + case NL80211_CHAN_WIDTH_40: + bandwidth = RTW_CHANNEL_WIDTH_40; + if (primary_freq > center_freq) { + primary_chan_idx = 1; + center_chan -= 2; + } else { + primary_chan_idx = 2; + center_chan += 2; + } + break; + case NL80211_CHAN_WIDTH_80: + bandwidth = RTW_CHANNEL_WIDTH_80; + if (primary_freq > center_freq) { + if (primary_freq - center_freq == 10) { + primary_chan_idx = 1; + center_chan -= 2; + } else { + primary_chan_idx = 3; + center_chan -= 6; + } + } else { + if (center_freq - primary_freq == 10) { + primary_chan_idx = 2; + center_chan += 2; + } else { + primary_chan_idx = 4; + center_chan += 6; + } + } + break; + default: + center_chan = 0; + break; + } + + chan_params->center_chan = center_chan; + chan_params->bandwidth = bandwidth; + chan_params->primary_chan_idx = primary_chan_idx; +} + +void rtw_set_channel(struct rtw_dev *rtwdev) +{ + struct ieee80211_hw *hw = rtwdev->hw; + struct rtw_hal *hal = &rtwdev->hal; + struct rtw_chip_info *chip = rtwdev->chip; + struct rtw_channel_params ch_param; + u8 center_chan, bandwidth, primary_chan_idx; + + rtw_get_channel_params(&hw->conf.chandef, &ch_param); + if (WARN(ch_param.center_chan == 0, "Invalid channel\n")) + return; + + center_chan = ch_param.center_chan; + bandwidth = ch_param.bandwidth; + primary_chan_idx = ch_param.primary_chan_idx; + + hal->current_band_width = bandwidth; + hal->current_channel = center_chan; + hal->current_band_type = center_chan > 14 ? RTW_BAND_5G : RTW_BAND_2G; + chip->ops->set_channel(rtwdev, center_chan, bandwidth, primary_chan_idx); + + rtw_phy_set_tx_power_level(rtwdev, center_chan); +} + +static void rtw_vif_write_addr(struct rtw_dev *rtwdev, u32 start, u8 *addr) +{ + int i; + + for (i = 0; i < ETH_ALEN; i++) + rtw_write8(rtwdev, start + i, addr[i]); +} + +void rtw_vif_port_config(struct rtw_dev *rtwdev, + struct rtw_vif *rtwvif, + u32 config) +{ + u32 addr, mask; + + if (config & PORT_SET_MAC_ADDR) { + addr = rtwvif->conf->mac_addr.addr; + rtw_vif_write_addr(rtwdev, addr, rtwvif->mac_addr); + } + if (config & PORT_SET_BSSID) { + addr = rtwvif->conf->bssid.addr; + rtw_vif_write_addr(rtwdev, addr, rtwvif->bssid); + } + if (config & PORT_SET_NET_TYPE) { + addr = rtwvif->conf->net_type.addr; + mask = rtwvif->conf->net_type.mask; + rtw_write32_mask(rtwdev, addr, mask, rtwvif->net_type); + } + if (config & PORT_SET_AID) { + addr = rtwvif->conf->aid.addr; + mask = rtwvif->conf->aid.mask; + rtw_write32_mask(rtwdev, addr, mask, rtwvif->aid); + } +} + +static u8 hw_bw_cap_to_bitamp(u8 bw_cap) +{ + u8 bw = 0; + + switch (bw_cap) { + case EFUSE_HW_CAP_IGNORE: + case EFUSE_HW_CAP_SUPP_BW80: + bw |= BIT(RTW_CHANNEL_WIDTH_80); + /* fall through */ + case EFUSE_HW_CAP_SUPP_BW40: + bw |= BIT(RTW_CHANNEL_WIDTH_40); + /* fall through */ + default: + bw |= BIT(RTW_CHANNEL_WIDTH_20); + break; + } + + return bw; +} + +static void rtw_hw_config_rf_ant_num(struct rtw_dev *rtwdev, u8 hw_ant_num) +{ + struct rtw_hal *hal = &rtwdev->hal; + + if (hw_ant_num == EFUSE_HW_CAP_IGNORE || + hw_ant_num >= hal->rf_path_num) + return; + + switch (hw_ant_num) { + case 1: + hal->rf_type = RF_1T1R; + hal->rf_path_num = 1; + hal->antenna_tx = BB_PATH_A; + hal->antenna_rx = BB_PATH_A; + break; + default: + WARN(1, "invalid hw configuration from efuse\n"); + break; + } +} + +static u64 get_vht_ra_mask(struct ieee80211_sta *sta) +{ + u64 ra_mask = 0; + u16 mcs_map = le16_to_cpu(sta->vht_cap.vht_mcs.rx_mcs_map); + u8 vht_mcs_cap; + int i, nss; + + /* 4SS, every two bits for MCS7/8/9 */ + for (i = 0, nss = 12; i < 4; i++, mcs_map >>= 2, nss += 10) { + vht_mcs_cap = mcs_map & 0x3; + switch (vht_mcs_cap) { + case 2: /* MCS9 */ + ra_mask |= 0x3ffULL << nss; + break; + case 1: /* MCS8 */ + ra_mask |= 0x1ffULL << nss; + break; + case 0: /* MCS7 */ + ra_mask |= 0x0ffULL << nss; + break; + default: + break; + } + } + + return ra_mask; +} + +static u8 get_rate_id(u8 wireless_set, enum rtw_bandwidth bw_mode, u8 tx_num) +{ + u8 rate_id = 0; + + switch (wireless_set) { + case WIRELESS_CCK: + rate_id = RTW_RATEID_B_20M; + break; + case WIRELESS_OFDM: + rate_id = RTW_RATEID_G; + break; + case WIRELESS_CCK | WIRELESS_OFDM: + rate_id = RTW_RATEID_BG; + break; + case WIRELESS_OFDM | WIRELESS_HT: + if (tx_num == 1) + rate_id = RTW_RATEID_GN_N1SS; + else if (tx_num == 2) + rate_id = RTW_RATEID_GN_N2SS; + else if (tx_num == 3) + rate_id = RTW_RATEID_ARFR5_N_3SS; + break; + case WIRELESS_CCK | WIRELESS_OFDM | WIRELESS_HT: + if (bw_mode == RTW_CHANNEL_WIDTH_40) { + if (tx_num == 1) + rate_id = RTW_RATEID_BGN_40M_1SS; + else if (tx_num == 2) + rate_id = RTW_RATEID_BGN_40M_2SS; + else if (tx_num == 3) + rate_id = RTW_RATEID_ARFR5_N_3SS; + else if (tx_num == 4) + rate_id = RTW_RATEID_ARFR7_N_4SS; + } else { + if (tx_num == 1) + rate_id = RTW_RATEID_BGN_20M_1SS; + else if (tx_num == 2) + rate_id = RTW_RATEID_BGN_20M_2SS; + else if (tx_num == 3) + rate_id = RTW_RATEID_ARFR5_N_3SS; + else if (tx_num == 4) + rate_id = RTW_RATEID_ARFR7_N_4SS; + } + break; + case WIRELESS_OFDM | WIRELESS_VHT: + if (tx_num == 1) + rate_id = RTW_RATEID_ARFR1_AC_1SS; + else if (tx_num == 2) + rate_id = RTW_RATEID_ARFR0_AC_2SS; + else if (tx_num == 3) + rate_id = RTW_RATEID_ARFR4_AC_3SS; + else if (tx_num == 4) + rate_id = RTW_RATEID_ARFR6_AC_4SS; + break; + case WIRELESS_CCK | WIRELESS_OFDM | WIRELESS_VHT: + if (bw_mode >= RTW_CHANNEL_WIDTH_80) { + if (tx_num == 1) + rate_id = RTW_RATEID_ARFR1_AC_1SS; + else if (tx_num == 2) + rate_id = RTW_RATEID_ARFR0_AC_2SS; + else if (tx_num == 3) + rate_id = RTW_RATEID_ARFR4_AC_3SS; + else if (tx_num == 4) + rate_id = RTW_RATEID_ARFR6_AC_4SS; + } else { + if (tx_num == 1) + rate_id = RTW_RATEID_ARFR2_AC_2G_1SS; + else if (tx_num == 2) + rate_id = RTW_RATEID_ARFR3_AC_2G_2SS; + else if (tx_num == 3) + rate_id = RTW_RATEID_ARFR4_AC_3SS; + else if (tx_num == 4) + rate_id = RTW_RATEID_ARFR6_AC_4SS; + } + break; + default: + break; + } + + return rate_id; +} + +#define RA_MASK_CCK_RATES 0x0000f +#define RA_MASK_OFDM_RATES 0x00ff0 +#define RA_MASK_HT_RATES_1SS (0xff000ULL << 0) +#define RA_MASK_HT_RATES_2SS (0xff000ULL << 8) +#define RA_MASK_HT_RATES_3SS (0xff000ULL << 16) +#define RA_MASK_HT_RATES (RA_MASK_HT_RATES_1SS | \ + RA_MASK_HT_RATES_2SS | \ + RA_MASK_HT_RATES_3SS) +#define RA_MASK_VHT_RATES_1SS (0x3ff000ULL << 0) +#define RA_MASK_VHT_RATES_2SS (0x3ff000ULL << 10) +#define RA_MASK_VHT_RATES_3SS (0x3ff000ULL << 20) +#define RA_MASK_VHT_RATES (RA_MASK_VHT_RATES_1SS | \ + RA_MASK_VHT_RATES_2SS | \ + RA_MASK_VHT_RATES_3SS) +#define RA_MASK_CCK_IN_HT 0x00005 +#define RA_MASK_CCK_IN_VHT 0x00005 +#define RA_MASK_OFDM_IN_VHT 0x00010 +#define RA_MASK_OFDM_IN_HT_2G 0x00010 +#define RA_MASK_OFDM_IN_HT_5G 0x00030 + +void rtw_update_sta_info(struct rtw_dev *rtwdev, struct rtw_sta_info *si) +{ + struct ieee80211_sta *sta = si->sta; + struct rtw_efuse *efuse = &rtwdev->efuse; + struct rtw_hal *hal = &rtwdev->hal; + u8 rssi_level; + u8 wireless_set; + u8 bw_mode; + u8 rate_id; + u8 rf_type = RF_1T1R; + u8 stbc_en = 0; + u8 ldpc_en = 0; + u8 tx_num = 1; + u64 ra_mask = 0; + bool is_vht_enable = false; + bool is_support_sgi = false; + + if (sta->vht_cap.vht_supported) { + is_vht_enable = true; + ra_mask |= get_vht_ra_mask(sta); + if (sta->vht_cap.cap & IEEE80211_VHT_CAP_RXSTBC_MASK) + stbc_en = VHT_STBC_EN; + if (sta->vht_cap.cap & IEEE80211_VHT_CAP_RXLDPC) + ldpc_en = VHT_LDPC_EN; + if (sta->vht_cap.cap & IEEE80211_VHT_CAP_SHORT_GI_80) + is_support_sgi = true; + } else if (sta->ht_cap.ht_supported) { + ra_mask |= (sta->ht_cap.mcs.rx_mask[NL80211_BAND_5GHZ] << 20) | + (sta->ht_cap.mcs.rx_mask[NL80211_BAND_2GHZ] << 12); + if (sta->ht_cap.cap & IEEE80211_HT_CAP_RX_STBC) + stbc_en = HT_STBC_EN; + if (sta->ht_cap.cap & IEEE80211_HT_CAP_LDPC_CODING) + ldpc_en = HT_LDPC_EN; + if (sta->ht_cap.cap & IEEE80211_HT_CAP_SGI_20 || + sta->ht_cap.cap & IEEE80211_HT_CAP_SGI_40) + is_support_sgi = true; + } + + if (hal->current_band_type == RTW_BAND_5G) { + ra_mask |= (u64)sta->supp_rates[NL80211_BAND_5GHZ] << 4; + if (sta->vht_cap.vht_supported) { + ra_mask &= RA_MASK_VHT_RATES | RA_MASK_OFDM_IN_VHT; + wireless_set = WIRELESS_OFDM | WIRELESS_VHT; + } else if (sta->ht_cap.ht_supported) { + ra_mask &= RA_MASK_HT_RATES | RA_MASK_OFDM_IN_HT_5G; + wireless_set = WIRELESS_OFDM | WIRELESS_HT; + } else { + wireless_set = WIRELESS_OFDM; + } + } else if (hal->current_band_type == RTW_BAND_2G) { + ra_mask |= sta->supp_rates[NL80211_BAND_2GHZ]; + if (sta->vht_cap.vht_supported) { + ra_mask &= RA_MASK_VHT_RATES | RA_MASK_CCK_IN_VHT | + RA_MASK_OFDM_IN_VHT; + wireless_set = WIRELESS_CCK | WIRELESS_OFDM | + WIRELESS_HT | WIRELESS_VHT; + } else if (sta->ht_cap.ht_supported) { + ra_mask &= RA_MASK_HT_RATES | RA_MASK_CCK_IN_HT | + RA_MASK_OFDM_IN_HT_2G; + wireless_set = WIRELESS_CCK | WIRELESS_OFDM | + WIRELESS_HT; + } else if (sta->supp_rates[0] <= 0xf) { + wireless_set = WIRELESS_CCK; + } else { + wireless_set = WIRELESS_CCK | WIRELESS_OFDM; + } + } else { + rtw_err(rtwdev, "Unknown band type\n"); + wireless_set = 0; + } + + if (efuse->hw_cap.nss == 1) { + ra_mask &= RA_MASK_VHT_RATES_1SS; + ra_mask &= RA_MASK_HT_RATES_1SS; + } + + switch (sta->bandwidth) { + case IEEE80211_STA_RX_BW_80: + bw_mode = RTW_CHANNEL_WIDTH_80; + break; + case IEEE80211_STA_RX_BW_40: + bw_mode = RTW_CHANNEL_WIDTH_40; + break; + default: + bw_mode = RTW_CHANNEL_WIDTH_20; + break; + } + + if (sta->vht_cap.vht_supported && ra_mask & 0xffc00000) { + tx_num = 2; + rf_type = RF_2T2R; + } else if (sta->ht_cap.ht_supported && ra_mask & 0xfff00000) { + tx_num = 2; + rf_type = RF_2T2R; + } + + rate_id = get_rate_id(wireless_set, bw_mode, tx_num); + + if (wireless_set != WIRELESS_CCK) { + rssi_level = si->rssi_level; + if (rssi_level == 0) + ra_mask &= 0xffffffffffffffffULL; + else if (rssi_level == 1) + ra_mask &= 0xfffffffffffffff0ULL; + else if (rssi_level == 2) + ra_mask &= 0xffffffffffffefe0ULL; + else if (rssi_level == 3) + ra_mask &= 0xffffffffffffcfc0ULL; + else if (rssi_level == 4) + ra_mask &= 0xffffffffffff8f80ULL; + else if (rssi_level >= 5) + ra_mask &= 0xffffffffffff0f00ULL; + } + + si->bw_mode = bw_mode; + si->stbc_en = stbc_en; + si->ldpc_en = ldpc_en; + si->rf_type = rf_type; + si->wireless_set = wireless_set; + si->sgi_enable = is_support_sgi; + si->vht_enable = is_vht_enable; + si->ra_mask = ra_mask; + si->rate_id = rate_id; + + rtw_fw_send_ra_info(rtwdev, si); +} + +static int rtw_power_on(struct rtw_dev *rtwdev) +{ + struct rtw_chip_info *chip = rtwdev->chip; + struct rtw_fw_state *fw = &rtwdev->fw; + int ret; + + ret = rtw_hci_setup(rtwdev); + if (ret) { + rtw_err(rtwdev, "failed to setup hci\n"); + goto err; + } + + /* power on MAC before firmware downloaded */ + ret = rtw_mac_power_on(rtwdev); + if (ret) { + rtw_err(rtwdev, "failed to power on mac\n"); + goto err; + } + + wait_for_completion(&fw->completion); + if (!fw->firmware) { + ret = -EINVAL; + rtw_err(rtwdev, "failed to load firmware\n"); + goto err; + } + + ret = rtw_download_firmware(rtwdev, fw); + if (ret) { + rtw_err(rtwdev, "failed to download firmware\n"); + goto err_off; + } + + /* config mac after firmware downloaded */ + ret = rtw_mac_init(rtwdev); + if (ret) { + rtw_err(rtwdev, "failed to configure mac\n"); + goto err_off; + } + + chip->ops->phy_set_param(rtwdev); + + ret = rtw_hci_start(rtwdev); + if (ret) { + rtw_err(rtwdev, "failed to start hci\n"); + goto err_off; + } + + return 0; + +err_off: + rtw_mac_power_off(rtwdev); + +err: + return ret; +} + +int rtw_core_start(struct rtw_dev *rtwdev) +{ + int ret; + + ret = rtw_power_on(rtwdev); + if (ret) + return ret; + + rtw_sec_enable_sec_engine(rtwdev); + + /* rcr reset after powered on */ + rtw_write32(rtwdev, REG_RCR, rtwdev->hal.rcr); + + ieee80211_queue_delayed_work(rtwdev->hw, &rtwdev->watch_dog_work, + RTW_WATCH_DOG_DELAY_TIME); + + rtw_flag_set(rtwdev, RTW_FLAG_RUNNING); + + return 0; +} + +static void rtw_power_off(struct rtw_dev *rtwdev) +{ + rtwdev->hci.ops->stop(rtwdev); + rtw_mac_power_off(rtwdev); +} + +void rtw_core_stop(struct rtw_dev *rtwdev) +{ + rtw_flag_clear(rtwdev, RTW_FLAG_RUNNING); + rtw_flag_clear(rtwdev, RTW_FLAG_FW_RUNNING); + + cancel_delayed_work_sync(&rtwdev->watch_dog_work); + + rtw_power_off(rtwdev); +} + +static void rtw_init_ht_cap(struct rtw_dev *rtwdev, + struct ieee80211_sta_ht_cap *ht_cap) +{ + struct rtw_efuse *efuse = &rtwdev->efuse; + + ht_cap->ht_supported = true; + ht_cap->cap = 0; + ht_cap->cap |= IEEE80211_HT_CAP_SGI_20 | + IEEE80211_HT_CAP_MAX_AMSDU | + IEEE80211_HT_CAP_LDPC_CODING | + (1 << IEEE80211_HT_CAP_RX_STBC_SHIFT); + if (efuse->hw_cap.bw & BIT(RTW_CHANNEL_WIDTH_40)) + ht_cap->cap |= IEEE80211_HT_CAP_SUP_WIDTH_20_40 | + IEEE80211_HT_CAP_DSSSCCK40 | + IEEE80211_HT_CAP_SGI_40; + ht_cap->ampdu_factor = IEEE80211_HT_MAX_AMPDU_64K; + ht_cap->ampdu_density = IEEE80211_HT_MPDU_DENSITY_16; + ht_cap->mcs.tx_params = IEEE80211_HT_MCS_TX_DEFINED; + if (efuse->hw_cap.nss > 1) { + ht_cap->mcs.rx_mask[0] = 0xFF; + ht_cap->mcs.rx_mask[1] = 0xFF; + ht_cap->mcs.rx_mask[4] = 0x01; + ht_cap->mcs.rx_highest = cpu_to_le16(300); + } else { + ht_cap->mcs.rx_mask[0] = 0xFF; + ht_cap->mcs.rx_mask[1] = 0x00; + ht_cap->mcs.rx_mask[4] = 0x01; + ht_cap->mcs.rx_highest = cpu_to_le16(150); + } +} + +static void rtw_init_vht_cap(struct rtw_dev *rtwdev, + struct ieee80211_sta_vht_cap *vht_cap) +{ + struct rtw_efuse *efuse = &rtwdev->efuse; + u16 mcs_map; + __le16 highest; + + if (efuse->hw_cap.ptcl != EFUSE_HW_CAP_IGNORE && + efuse->hw_cap.ptcl != EFUSE_HW_CAP_PTCL_VHT) + return; + + vht_cap->vht_supported = true; + vht_cap->cap = IEEE80211_VHT_CAP_MAX_MPDU_LENGTH_11454 | + IEEE80211_VHT_CAP_RXLDPC | + IEEE80211_VHT_CAP_SHORT_GI_80 | + IEEE80211_VHT_CAP_TXSTBC | + IEEE80211_VHT_CAP_RXSTBC_1 | + IEEE80211_VHT_CAP_HTC_VHT | + IEEE80211_VHT_CAP_MAX_A_MPDU_LENGTH_EXPONENT_MASK | + 0; + mcs_map = IEEE80211_VHT_MCS_SUPPORT_0_9 << 0 | + IEEE80211_VHT_MCS_NOT_SUPPORTED << 4 | + IEEE80211_VHT_MCS_NOT_SUPPORTED << 6 | + IEEE80211_VHT_MCS_NOT_SUPPORTED << 8 | + IEEE80211_VHT_MCS_NOT_SUPPORTED << 10 | + IEEE80211_VHT_MCS_NOT_SUPPORTED << 12 | + IEEE80211_VHT_MCS_NOT_SUPPORTED << 14; + if (efuse->hw_cap.nss > 1) { + highest = cpu_to_le16(780); + mcs_map |= IEEE80211_VHT_MCS_SUPPORT_0_9 << 2; + } else { + highest = cpu_to_le16(390); + mcs_map |= IEEE80211_VHT_MCS_NOT_SUPPORTED << 2; + } + + vht_cap->vht_mcs.rx_mcs_map = cpu_to_le16(mcs_map); + vht_cap->vht_mcs.tx_mcs_map = cpu_to_le16(mcs_map); + vht_cap->vht_mcs.rx_highest = highest; + vht_cap->vht_mcs.tx_highest = highest; +} + +static void rtw_set_supported_band(struct ieee80211_hw *hw, + struct rtw_chip_info *chip) +{ + struct rtw_dev *rtwdev = hw->priv; + struct ieee80211_supported_band *sband; + + if (chip->band & RTW_BAND_2G) { + sband = kmemdup(&rtw_band_2ghz, sizeof(*sband), GFP_KERNEL); + if (!sband) + goto err_out; + if (chip->ht_supported) + rtw_init_ht_cap(rtwdev, &sband->ht_cap); + hw->wiphy->bands[NL80211_BAND_2GHZ] = sband; + } + + if (chip->band & RTW_BAND_5G) { + sband = kmemdup(&rtw_band_5ghz, sizeof(*sband), GFP_KERNEL); + if (!sband) + goto err_out; + if (chip->ht_supported) + rtw_init_ht_cap(rtwdev, &sband->ht_cap); + if (chip->vht_supported) + rtw_init_vht_cap(rtwdev, &sband->vht_cap); + hw->wiphy->bands[NL80211_BAND_5GHZ] = sband; + } + + return; + +err_out: + rtw_err(rtwdev, "failed to set supported band\n"); + kfree(sband); +} + +static void rtw_unset_supported_band(struct ieee80211_hw *hw, + struct rtw_chip_info *chip) +{ + kfree(hw->wiphy->bands[NL80211_BAND_2GHZ]); + kfree(hw->wiphy->bands[NL80211_BAND_5GHZ]); +} + +static void rtw_load_firmware_cb(const struct firmware *firmware, void *context) +{ + struct rtw_dev *rtwdev = context; + struct rtw_fw_state *fw = &rtwdev->fw; + + if (!firmware) + rtw_err(rtwdev, "failed to request firmware\n"); + + fw->firmware = firmware; + complete_all(&fw->completion); +} + +static int rtw_load_firmware(struct rtw_dev *rtwdev, const char *fw_name) +{ + struct rtw_fw_state *fw = &rtwdev->fw; + int ret; + + init_completion(&fw->completion); + + ret = request_firmware_nowait(THIS_MODULE, true, fw_name, rtwdev->dev, + GFP_KERNEL, rtwdev, rtw_load_firmware_cb); + if (ret) { + rtw_err(rtwdev, "async firmware request failed\n"); + return ret; + } + + return 0; +} + +static int rtw_chip_parameter_setup(struct rtw_dev *rtwdev) +{ + struct rtw_chip_info *chip = rtwdev->chip; + struct rtw_hal *hal = &rtwdev->hal; + struct rtw_efuse *efuse = &rtwdev->efuse; + u32 wl_bt_pwr_ctrl; + int ret = 0; + + switch (rtw_hci_type(rtwdev)) { + case RTW_HCI_TYPE_PCIE: + rtwdev->hci.rpwm_addr = 0x03d9; + break; + default: + rtw_err(rtwdev, "unsupported hci type\n"); + return -EINVAL; + } + + wl_bt_pwr_ctrl = rtw_read32(rtwdev, REG_WL_BT_PWR_CTRL); + if (wl_bt_pwr_ctrl & BIT_BT_FUNC_EN) + rtwdev->efuse.btcoex = true; + hal->chip_version = rtw_read32(rtwdev, REG_SYS_CFG1); + hal->fab_version = BIT_GET_VENDOR_ID(hal->chip_version) >> 2; + hal->cut_version = BIT_GET_CHIP_VER(hal->chip_version); + hal->mp_chip = (hal->chip_version & BIT_RTL_ID) ? 0 : 1; + if (hal->chip_version & BIT_RF_TYPE_ID) { + hal->rf_type = RF_2T2R; + hal->rf_path_num = 2; + hal->antenna_tx = BB_PATH_AB; + hal->antenna_rx = BB_PATH_AB; + } else { + hal->rf_type = RF_1T1R; + hal->rf_path_num = 1; + hal->antenna_tx = BB_PATH_A; + hal->antenna_rx = BB_PATH_A; + } + + if (hal->fab_version == 2) + hal->fab_version = 1; + else if (hal->fab_version == 1) + hal->fab_version = 2; + + efuse->physical_size = chip->phy_efuse_size; + efuse->logical_size = chip->log_efuse_size; + efuse->protect_size = chip->ptct_efuse_size; + + /* default use ack */ + rtwdev->hal.rcr |= BIT_VHT_DACK; + + return ret; +} + +static int rtw_chip_efuse_enable(struct rtw_dev *rtwdev) +{ + struct rtw_fw_state *fw = &rtwdev->fw; + int ret; + + ret = rtw_hci_setup(rtwdev); + if (ret) { + rtw_err(rtwdev, "failed to setup hci\n"); + goto err; + } + + ret = rtw_mac_power_on(rtwdev); + if (ret) { + rtw_err(rtwdev, "failed to power on mac\n"); + goto err; + } + + rtw_write8(rtwdev, REG_C2HEVT, C2H_HW_FEATURE_DUMP); + + wait_for_completion(&fw->completion); + if (!fw->firmware) { + ret = -EINVAL; + rtw_err(rtwdev, "failed to load firmware\n"); + goto err; + } + + ret = rtw_download_firmware(rtwdev, fw); + if (ret) { + rtw_err(rtwdev, "failed to download firmware\n"); + goto err_off; + } + + return 0; + +err_off: + rtw_mac_power_off(rtwdev); + +err: + return ret; +} + +static int rtw_dump_hw_feature(struct rtw_dev *rtwdev) +{ + struct rtw_efuse *efuse = &rtwdev->efuse; + u8 hw_feature[HW_FEATURE_LEN]; + u8 id; + u8 bw; + int i; + + id = rtw_read8(rtwdev, REG_C2HEVT); + if (id != C2H_HW_FEATURE_REPORT) { + rtw_err(rtwdev, "failed to read hw feature report\n"); + return -EBUSY; + } + + for (i = 0; i < HW_FEATURE_LEN; i++) + hw_feature[i] = rtw_read8(rtwdev, REG_C2HEVT + 2 + i); + + rtw_write8(rtwdev, REG_C2HEVT, 0); + + bw = GET_EFUSE_HW_CAP_BW(hw_feature); + efuse->hw_cap.bw = hw_bw_cap_to_bitamp(bw); + efuse->hw_cap.hci = GET_EFUSE_HW_CAP_HCI(hw_feature); + efuse->hw_cap.nss = GET_EFUSE_HW_CAP_NSS(hw_feature); + efuse->hw_cap.ptcl = GET_EFUSE_HW_CAP_PTCL(hw_feature); + efuse->hw_cap.ant_num = GET_EFUSE_HW_CAP_ANT_NUM(hw_feature); + + rtw_hw_config_rf_ant_num(rtwdev, efuse->hw_cap.ant_num); + + if (efuse->hw_cap.nss == EFUSE_HW_CAP_IGNORE) + efuse->hw_cap.nss = rtwdev->hal.rf_path_num; + + rtw_dbg(rtwdev, RTW_DBG_EFUSE, + "hw cap: hci=0x%02x, bw=0x%02x, ptcl=0x%02x, ant_num=%d, nss=%d\n", + efuse->hw_cap.hci, efuse->hw_cap.bw, efuse->hw_cap.ptcl, + efuse->hw_cap.ant_num, efuse->hw_cap.nss); + + return 0; +} + +static void rtw_chip_efuse_disable(struct rtw_dev *rtwdev) +{ + rtw_hci_stop(rtwdev); + rtw_mac_power_off(rtwdev); +} + +static int rtw_chip_efuse_info_setup(struct rtw_dev *rtwdev) +{ + struct rtw_efuse *efuse = &rtwdev->efuse; + int ret; + + mutex_lock(&rtwdev->mutex); + + /* power on mac to read efuse */ + ret = rtw_chip_efuse_enable(rtwdev); + if (ret) + goto out; + + ret = rtw_parse_efuse_map(rtwdev); + if (ret) + goto out; + + ret = rtw_dump_hw_feature(rtwdev); + if (ret) + goto out; + + ret = rtw_check_supported_rfe(rtwdev); + if (ret) + goto out; + + if (efuse->crystal_cap == 0xff) + efuse->crystal_cap = 0; + if (efuse->pa_type_2g == 0xff) + efuse->pa_type_2g = 0; + if (efuse->pa_type_5g == 0xff) + efuse->pa_type_5g = 0; + if (efuse->lna_type_2g == 0xff) + efuse->lna_type_2g = 0; + if (efuse->lna_type_5g == 0xff) + efuse->lna_type_5g = 0; + if (efuse->channel_plan == 0xff) + efuse->channel_plan = 0x7f; + if (efuse->bt_setting & BIT(0)) + efuse->share_ant = true; + if (efuse->regd == 0xff) + efuse->regd = 0; + + efuse->ext_pa_2g = efuse->pa_type_2g & BIT(4) ? 1 : 0; + efuse->ext_lna_2g = efuse->lna_type_2g & BIT(3) ? 1 : 0; + efuse->ext_pa_5g = efuse->pa_type_5g & BIT(0) ? 1 : 0; + efuse->ext_lna_2g = efuse->lna_type_5g & BIT(3) ? 1 : 0; + + rtw_chip_efuse_disable(rtwdev); + +out: + mutex_unlock(&rtwdev->mutex); + return ret; +} + +static int rtw_chip_board_info_setup(struct rtw_dev *rtwdev) +{ + struct rtw_hal *hal = &rtwdev->hal; + const struct rtw_rfe_def *rfe_def = rtw_get_rfe_def(rtwdev); + + if (!rfe_def) + return -ENODEV; + + rtw_phy_setup_phy_cond(rtwdev, 0); + + rtw_hw_init_tx_power(hal); + rtw_load_table(rtwdev, rfe_def->phy_pg_tbl); + rtw_load_table(rtwdev, rfe_def->txpwr_lmt_tbl); + rtw_phy_tx_power_by_rate_config(hal); + rtw_phy_tx_power_limit_config(hal); + + return 0; +} + +int rtw_chip_info_setup(struct rtw_dev *rtwdev) +{ + int ret; + + ret = rtw_chip_parameter_setup(rtwdev); + if (ret) { + rtw_err(rtwdev, "failed to setup chip parameters\n"); + goto err_out; + } + + ret = rtw_chip_efuse_info_setup(rtwdev); + if (ret) { + rtw_err(rtwdev, "failed to setup chip efuse info\n"); + goto err_out; + } + + ret = rtw_chip_board_info_setup(rtwdev); + if (ret) { + rtw_err(rtwdev, "failed to setup chip board info\n"); + goto err_out; + } + + return 0; + +err_out: + return ret; +} +EXPORT_SYMBOL(rtw_chip_info_setup); + +int rtw_core_init(struct rtw_dev *rtwdev) +{ + int ret; + + INIT_LIST_HEAD(&rtwdev->rsvd_page_list); + + timer_setup(&rtwdev->tx_report.purge_timer, + rtw_tx_report_purge_timer, 0); + + INIT_DELAYED_WORK(&rtwdev->watch_dog_work, rtw_watch_dog_work); + INIT_DELAYED_WORK(&rtwdev->lps_work, rtw_lps_work); + INIT_WORK(&rtwdev->c2h_work, rtw_c2h_work); + skb_queue_head_init(&rtwdev->c2h_queue); + skb_queue_head_init(&rtwdev->tx_report.queue); + + spin_lock_init(&rtwdev->dm_lock); + spin_lock_init(&rtwdev->rf_lock); + spin_lock_init(&rtwdev->h2c.lock); + spin_lock_init(&rtwdev->tx_report.q_lock); + + mutex_init(&rtwdev->mutex); + mutex_init(&rtwdev->hal.tx_power_mutex); + + rtwdev->sec.total_cam_num = 32; + rtwdev->hal.current_channel = 1; + set_bit(RTW_BC_MC_MACID, rtwdev->mac_id_map); + + mutex_lock(&rtwdev->mutex); + rtw_add_rsvd_page(rtwdev, RSVD_BEACON, false); + mutex_unlock(&rtwdev->mutex); + + /* default rx filter setting */ + rtwdev->hal.rcr = BIT_APP_FCS | BIT_APP_MIC | BIT_APP_ICV | + BIT_HTC_LOC_CTRL | BIT_APP_PHYSTS | + BIT_AB | BIT_AM | BIT_APM; + + ret = rtw_load_firmware(rtwdev, rtwdev->chip->fw_name); + if (ret) { + rtw_warn(rtwdev, "no firmware loaded\n"); + return ret; + } + + return 0; +} +EXPORT_SYMBOL(rtw_core_init); + +void rtw_core_deinit(struct rtw_dev *rtwdev) +{ + struct rtw_fw_state *fw = &rtwdev->fw; + struct rtw_rsvd_page *rsvd_pkt, *tmp; + unsigned long flags; + + if (fw->firmware) + release_firmware(fw->firmware); + + spin_lock_irqsave(&rtwdev->tx_report.q_lock, flags); + skb_queue_purge(&rtwdev->tx_report.queue); + spin_unlock_irqrestore(&rtwdev->tx_report.q_lock, flags); + + list_for_each_entry_safe(rsvd_pkt, tmp, &rtwdev->rsvd_page_list, list) { + list_del(&rsvd_pkt->list); + kfree(rsvd_pkt); + } + + mutex_destroy(&rtwdev->mutex); + mutex_destroy(&rtwdev->hal.tx_power_mutex); +} +EXPORT_SYMBOL(rtw_core_deinit); + +int rtw_register_hw(struct rtw_dev *rtwdev, struct ieee80211_hw *hw) +{ + int max_tx_headroom = 0; + int ret; + + /* TODO: USB & SDIO may need extra room? */ + max_tx_headroom = rtwdev->chip->tx_pkt_desc_sz; + + hw->extra_tx_headroom = max_tx_headroom; + hw->queues = IEEE80211_NUM_ACS; + hw->sta_data_size = sizeof(struct rtw_sta_info); + hw->vif_data_size = sizeof(struct rtw_vif); + + ieee80211_hw_set(hw, SIGNAL_DBM); + ieee80211_hw_set(hw, RX_INCLUDES_FCS); + ieee80211_hw_set(hw, AMPDU_AGGREGATION); + ieee80211_hw_set(hw, MFP_CAPABLE); + ieee80211_hw_set(hw, REPORTS_TX_ACK_STATUS); + ieee80211_hw_set(hw, SUPPORTS_PS); + ieee80211_hw_set(hw, SUPPORTS_DYNAMIC_PS); + + hw->wiphy->interface_modes = BIT(NL80211_IFTYPE_STATION) | + BIT(NL80211_IFTYPE_AP) | + BIT(NL80211_IFTYPE_ADHOC) | + BIT(NL80211_IFTYPE_MESH_POINT); + + hw->wiphy->flags |= WIPHY_FLAG_SUPPORTS_TDLS | + WIPHY_FLAG_TDLS_EXTERNAL_SETUP; + + rtw_set_supported_band(hw, rtwdev->chip); + SET_IEEE80211_PERM_ADDR(hw, rtwdev->efuse.addr); + + rtw_regd_init(rtwdev, rtw_regd_notifier); + + ret = ieee80211_register_hw(hw); + if (ret) { + rtw_err(rtwdev, "failed to register hw\n"); + return ret; + } + + if (regulatory_hint(hw->wiphy, rtwdev->regd.alpha2)) + rtw_err(rtwdev, "regulatory_hint fail\n"); + + rtw_debugfs_init(rtwdev); + + return 0; +} +EXPORT_SYMBOL(rtw_register_hw); + +void rtw_unregister_hw(struct rtw_dev *rtwdev, struct ieee80211_hw *hw) +{ + struct rtw_chip_info *chip = rtwdev->chip; + + ieee80211_unregister_hw(hw); + rtw_unset_supported_band(hw, chip); +} +EXPORT_SYMBOL(rtw_unregister_hw); + +MODULE_AUTHOR("Realtek Corporation"); +MODULE_DESCRIPTION("Realtek 802.11ac wireless core module"); +MODULE_LICENSE("Dual BSD/GPL"); diff --git a/drivers/net/wireless/realtek/rtw88/main.h b/drivers/net/wireless/realtek/rtw88/main.h new file mode 100644 index 0000000000000000000000000000000000000000..00fc77fb9b5497c59fc3c8454ee550a2019c70a6 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/main.h @@ -0,0 +1,1104 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#ifndef __RTK_MAIN_H_ +#define __RTK_MAIN_H_ + +#include +#include +#include +#include +#include +#include + +#include "util.h" + +#define RTW_MAX_MAC_ID_NUM 32 +#define RTW_MAX_SEC_CAM_NUM 32 + +#define RTW_WATCH_DOG_DELAY_TIME round_jiffies_relative(HZ * 2) + +#define RFREG_MASK 0xfffff +#define INV_RF_DATA 0xffffffff +#define TX_PAGE_SIZE_SHIFT 7 + +#define RTW_CHANNEL_WIDTH_MAX 3 +#define RTW_RF_PATH_MAX 4 +#define HW_FEATURE_LEN 13 + +extern unsigned int rtw_debug_mask; +extern const struct ieee80211_ops rtw_ops; +extern struct rtw_chip_info rtw8822b_hw_spec; +extern struct rtw_chip_info rtw8822c_hw_spec; + +#define RTW_MAX_CHANNEL_NUM_2G 14 +#define RTW_MAX_CHANNEL_NUM_5G 49 + +struct rtw_dev; + +enum rtw_hci_type { + RTW_HCI_TYPE_PCIE, + RTW_HCI_TYPE_USB, + RTW_HCI_TYPE_SDIO, + + RTW_HCI_TYPE_UNDEFINE, +}; + +struct rtw_hci { + struct rtw_hci_ops *ops; + enum rtw_hci_type type; + + u32 rpwm_addr; + + u8 bulkout_num; +}; + +enum rtw_supported_band { + RTW_BAND_2G = 1 << 0, + RTW_BAND_5G = 1 << 1, + RTW_BAND_60G = 1 << 2, + + RTW_BAND_MAX, +}; + +enum rtw_bandwidth { + RTW_CHANNEL_WIDTH_20 = 0, + RTW_CHANNEL_WIDTH_40 = 1, + RTW_CHANNEL_WIDTH_80 = 2, + RTW_CHANNEL_WIDTH_160 = 3, + RTW_CHANNEL_WIDTH_80_80 = 4, + RTW_CHANNEL_WIDTH_5 = 5, + RTW_CHANNEL_WIDTH_10 = 6, +}; + +enum rtw_net_type { + RTW_NET_NO_LINK = 0, + RTW_NET_AD_HOC = 1, + RTW_NET_MGD_LINKED = 2, + RTW_NET_AP_MODE = 3, +}; + +enum rtw_rf_type { + RF_1T1R = 0, + RF_1T2R = 1, + RF_2T2R = 2, + RF_2T3R = 3, + RF_2T4R = 4, + RF_3T3R = 5, + RF_3T4R = 6, + RF_4T4R = 7, + RF_TYPE_MAX, +}; + +enum rtw_rf_path { + RF_PATH_A = 0, + RF_PATH_B = 1, + RF_PATH_C = 2, + RF_PATH_D = 3, +}; + +enum rtw_bb_path { + BB_PATH_A = BIT(0), + BB_PATH_B = BIT(1), + BB_PATH_C = BIT(2), + BB_PATH_D = BIT(3), + + BB_PATH_AB = (BB_PATH_A | BB_PATH_B), + BB_PATH_AC = (BB_PATH_A | BB_PATH_C), + BB_PATH_AD = (BB_PATH_A | BB_PATH_D), + BB_PATH_BC = (BB_PATH_B | BB_PATH_C), + BB_PATH_BD = (BB_PATH_B | BB_PATH_D), + BB_PATH_CD = (BB_PATH_C | BB_PATH_D), + + BB_PATH_ABC = (BB_PATH_A | BB_PATH_B | BB_PATH_C), + BB_PATH_ABD = (BB_PATH_A | BB_PATH_B | BB_PATH_D), + BB_PATH_ACD = (BB_PATH_A | BB_PATH_C | BB_PATH_D), + BB_PATH_BCD = (BB_PATH_B | BB_PATH_C | BB_PATH_D), + + BB_PATH_ABCD = (BB_PATH_A | BB_PATH_B | BB_PATH_C | BB_PATH_D), +}; + +enum rtw_rate_section { + RTW_RATE_SECTION_CCK = 0, + RTW_RATE_SECTION_OFDM, + RTW_RATE_SECTION_HT_1S, + RTW_RATE_SECTION_HT_2S, + RTW_RATE_SECTION_VHT_1S, + RTW_RATE_SECTION_VHT_2S, + + /* keep last */ + RTW_RATE_SECTION_MAX, +}; + +enum rtw_wireless_set { + WIRELESS_CCK = 0x00000001, + WIRELESS_OFDM = 0x00000002, + WIRELESS_HT = 0x00000004, + WIRELESS_VHT = 0x00000008, +}; + +#define HT_STBC_EN BIT(0) +#define VHT_STBC_EN BIT(1) +#define HT_LDPC_EN BIT(0) +#define VHT_LDPC_EN BIT(1) + +enum rtw_chip_type { + RTW_CHIP_TYPE_8822B, + RTW_CHIP_TYPE_8822C, +}; + +enum rtw_tx_queue_type { + /* the order of AC queues matters */ + RTW_TX_QUEUE_BK = 0x0, + RTW_TX_QUEUE_BE = 0x1, + RTW_TX_QUEUE_VI = 0x2, + RTW_TX_QUEUE_VO = 0x3, + + RTW_TX_QUEUE_BCN = 0x4, + RTW_TX_QUEUE_MGMT = 0x5, + RTW_TX_QUEUE_HI0 = 0x6, + RTW_TX_QUEUE_H2C = 0x7, + /* keep it last */ + RTK_MAX_TX_QUEUE_NUM +}; + +enum rtw_rx_queue_type { + RTW_RX_QUEUE_MPDU = 0x0, + RTW_RX_QUEUE_C2H = 0x1, + /* keep it last */ + RTK_MAX_RX_QUEUE_NUM +}; + +enum rtw_rate_index { + RTW_RATEID_BGN_40M_2SS = 0, + RTW_RATEID_BGN_40M_1SS = 1, + RTW_RATEID_BGN_20M_2SS = 2, + RTW_RATEID_BGN_20M_1SS = 3, + RTW_RATEID_GN_N2SS = 4, + RTW_RATEID_GN_N1SS = 5, + RTW_RATEID_BG = 6, + RTW_RATEID_G = 7, + RTW_RATEID_B_20M = 8, + RTW_RATEID_ARFR0_AC_2SS = 9, + RTW_RATEID_ARFR1_AC_1SS = 10, + RTW_RATEID_ARFR2_AC_2G_1SS = 11, + RTW_RATEID_ARFR3_AC_2G_2SS = 12, + RTW_RATEID_ARFR4_AC_3SS = 13, + RTW_RATEID_ARFR5_N_3SS = 14, + RTW_RATEID_ARFR7_N_4SS = 15, + RTW_RATEID_ARFR6_AC_4SS = 16 +}; + +enum rtw_trx_desc_rate { + DESC_RATE1M = 0x00, + DESC_RATE2M = 0x01, + DESC_RATE5_5M = 0x02, + DESC_RATE11M = 0x03, + + DESC_RATE6M = 0x04, + DESC_RATE9M = 0x05, + DESC_RATE12M = 0x06, + DESC_RATE18M = 0x07, + DESC_RATE24M = 0x08, + DESC_RATE36M = 0x09, + DESC_RATE48M = 0x0a, + DESC_RATE54M = 0x0b, + + DESC_RATEMCS0 = 0x0c, + DESC_RATEMCS1 = 0x0d, + DESC_RATEMCS2 = 0x0e, + DESC_RATEMCS3 = 0x0f, + DESC_RATEMCS4 = 0x10, + DESC_RATEMCS5 = 0x11, + DESC_RATEMCS6 = 0x12, + DESC_RATEMCS7 = 0x13, + DESC_RATEMCS8 = 0x14, + DESC_RATEMCS9 = 0x15, + DESC_RATEMCS10 = 0x16, + DESC_RATEMCS11 = 0x17, + DESC_RATEMCS12 = 0x18, + DESC_RATEMCS13 = 0x19, + DESC_RATEMCS14 = 0x1a, + DESC_RATEMCS15 = 0x1b, + DESC_RATEMCS16 = 0x1c, + DESC_RATEMCS17 = 0x1d, + DESC_RATEMCS18 = 0x1e, + DESC_RATEMCS19 = 0x1f, + DESC_RATEMCS20 = 0x20, + DESC_RATEMCS21 = 0x21, + DESC_RATEMCS22 = 0x22, + DESC_RATEMCS23 = 0x23, + DESC_RATEMCS24 = 0x24, + DESC_RATEMCS25 = 0x25, + DESC_RATEMCS26 = 0x26, + DESC_RATEMCS27 = 0x27, + DESC_RATEMCS28 = 0x28, + DESC_RATEMCS29 = 0x29, + DESC_RATEMCS30 = 0x2a, + DESC_RATEMCS31 = 0x2b, + + DESC_RATEVHT1SS_MCS0 = 0x2c, + DESC_RATEVHT1SS_MCS1 = 0x2d, + DESC_RATEVHT1SS_MCS2 = 0x2e, + DESC_RATEVHT1SS_MCS3 = 0x2f, + DESC_RATEVHT1SS_MCS4 = 0x30, + DESC_RATEVHT1SS_MCS5 = 0x31, + DESC_RATEVHT1SS_MCS6 = 0x32, + DESC_RATEVHT1SS_MCS7 = 0x33, + DESC_RATEVHT1SS_MCS8 = 0x34, + DESC_RATEVHT1SS_MCS9 = 0x35, + + DESC_RATEVHT2SS_MCS0 = 0x36, + DESC_RATEVHT2SS_MCS1 = 0x37, + DESC_RATEVHT2SS_MCS2 = 0x38, + DESC_RATEVHT2SS_MCS3 = 0x39, + DESC_RATEVHT2SS_MCS4 = 0x3a, + DESC_RATEVHT2SS_MCS5 = 0x3b, + DESC_RATEVHT2SS_MCS6 = 0x3c, + DESC_RATEVHT2SS_MCS7 = 0x3d, + DESC_RATEVHT2SS_MCS8 = 0x3e, + DESC_RATEVHT2SS_MCS9 = 0x3f, + + DESC_RATEVHT3SS_MCS0 = 0x40, + DESC_RATEVHT3SS_MCS1 = 0x41, + DESC_RATEVHT3SS_MCS2 = 0x42, + DESC_RATEVHT3SS_MCS3 = 0x43, + DESC_RATEVHT3SS_MCS4 = 0x44, + DESC_RATEVHT3SS_MCS5 = 0x45, + DESC_RATEVHT3SS_MCS6 = 0x46, + DESC_RATEVHT3SS_MCS7 = 0x47, + DESC_RATEVHT3SS_MCS8 = 0x48, + DESC_RATEVHT3SS_MCS9 = 0x49, + + DESC_RATEVHT4SS_MCS0 = 0x4a, + DESC_RATEVHT4SS_MCS1 = 0x4b, + DESC_RATEVHT4SS_MCS2 = 0x4c, + DESC_RATEVHT4SS_MCS3 = 0x4d, + DESC_RATEVHT4SS_MCS4 = 0x4e, + DESC_RATEVHT4SS_MCS5 = 0x4f, + DESC_RATEVHT4SS_MCS6 = 0x50, + DESC_RATEVHT4SS_MCS7 = 0x51, + DESC_RATEVHT4SS_MCS8 = 0x52, + DESC_RATEVHT4SS_MCS9 = 0x53, + + DESC_RATE_MAX, +}; + +enum rtw_regulatory_domains { + RTW_REGD_FCC = 0, + RTW_REGD_MKK = 1, + RTW_REGD_ETSI = 2, + RTW_REGD_WW = 3, + + RTW_REGD_MAX +}; + +enum rtw_flags { + RTW_FLAG_RUNNING, + RTW_FLAG_FW_RUNNING, + RTW_FLAG_SCANNING, + RTW_FLAG_INACTIVE_PS, + RTW_FLAG_LEISURE_PS, + RTW_FLAG_DIG_DISABLE, + + NUM_OF_RTW_FLAGS, +}; + +/* the power index is represented by differences, which cck-1s & ht40-1s are + * the base values, so for 1s's differences, there are only ht20 & ofdm + */ +struct rtw_2g_1s_pwr_idx_diff { +#ifdef __LITTLE_ENDIAN + s8 ofdm:4; + s8 bw20:4; +#else + s8 bw20:4; + s8 ofdm:4; +#endif +} __packed; + +struct rtw_2g_ns_pwr_idx_diff { +#ifdef __LITTLE_ENDIAN + s8 bw20:4; + s8 bw40:4; + s8 cck:4; + s8 ofdm:4; +#else + s8 ofdm:4; + s8 cck:4; + s8 bw40:4; + s8 bw20:4; +#endif +} __packed; + +struct rtw_2g_txpwr_idx { + u8 cck_base[6]; + u8 bw40_base[5]; + struct rtw_2g_1s_pwr_idx_diff ht_1s_diff; + struct rtw_2g_ns_pwr_idx_diff ht_2s_diff; + struct rtw_2g_ns_pwr_idx_diff ht_3s_diff; + struct rtw_2g_ns_pwr_idx_diff ht_4s_diff; +}; + +struct rtw_5g_ht_1s_pwr_idx_diff { +#ifdef __LITTLE_ENDIAN + s8 ofdm:4; + s8 bw20:4; +#else + s8 bw20:4; + s8 ofdm:4; +#endif +} __packed; + +struct rtw_5g_ht_ns_pwr_idx_diff { +#ifdef __LITTLE_ENDIAN + s8 bw20:4; + s8 bw40:4; +#else + s8 bw40:4; + s8 bw20:4; +#endif +} __packed; + +struct rtw_5g_ofdm_ns_pwr_idx_diff { +#ifdef __LITTLE_ENDIAN + s8 ofdm_3s:4; + s8 ofdm_2s:4; + s8 ofdm_4s:4; + s8 res:4; +#else + s8 res:4; + s8 ofdm_4s:4; + s8 ofdm_2s:4; + s8 ofdm_3s:4; +#endif +} __packed; + +struct rtw_5g_vht_ns_pwr_idx_diff { +#ifdef __LITTLE_ENDIAN + s8 bw160:4; + s8 bw80:4; +#else + s8 bw80:4; + s8 bw160:4; +#endif +} __packed; + +struct rtw_5g_txpwr_idx { + u8 bw40_base[14]; + struct rtw_5g_ht_1s_pwr_idx_diff ht_1s_diff; + struct rtw_5g_ht_ns_pwr_idx_diff ht_2s_diff; + struct rtw_5g_ht_ns_pwr_idx_diff ht_3s_diff; + struct rtw_5g_ht_ns_pwr_idx_diff ht_4s_diff; + struct rtw_5g_ofdm_ns_pwr_idx_diff ofdm_diff; + struct rtw_5g_vht_ns_pwr_idx_diff vht_1s_diff; + struct rtw_5g_vht_ns_pwr_idx_diff vht_2s_diff; + struct rtw_5g_vht_ns_pwr_idx_diff vht_3s_diff; + struct rtw_5g_vht_ns_pwr_idx_diff vht_4s_diff; +}; + +struct rtw_txpwr_idx { + struct rtw_2g_txpwr_idx pwr_idx_2g; + struct rtw_5g_txpwr_idx pwr_idx_5g; +}; + +struct rtw_timer_list { + struct timer_list timer; + void (*function)(void *data); + void *args; +}; + +struct rtw_channel_params { + u8 center_chan; + u8 bandwidth; + u8 primary_chan_idx; +}; + +struct rtw_hw_reg { + u32 addr; + u32 mask; +}; + +struct rtw_backup_info { + u8 len; + u32 reg; + u32 val; +}; + +enum rtw_vif_port_set { + PORT_SET_MAC_ADDR = BIT(0), + PORT_SET_BSSID = BIT(1), + PORT_SET_NET_TYPE = BIT(2), + PORT_SET_AID = BIT(3), +}; + +struct rtw_vif_port { + struct rtw_hw_reg mac_addr; + struct rtw_hw_reg bssid; + struct rtw_hw_reg net_type; + struct rtw_hw_reg aid; +}; + +struct rtw_tx_pkt_info { + u32 tx_pkt_size; + u8 offset; + u8 pkt_offset; + u8 mac_id; + u8 rate_id; + u8 rate; + u8 qsel; + u8 bw; + u8 sec_type; + u8 sn; + bool ampdu_en; + u8 ampdu_factor; + u8 ampdu_density; + u16 seq; + bool stbc; + bool ldpc; + bool dis_rate_fallback; + bool bmc; + bool use_rate; + bool ls; + bool fs; + bool short_gi; + bool report; +}; + +struct rtw_rx_pkt_stat { + bool phy_status; + bool icv_err; + bool crc_err; + bool decrypted; + bool is_c2h; + + s32 signal_power; + u16 pkt_len; + u8 bw; + u8 drv_info_sz; + u8 shift; + u8 rate; + u8 mac_id; + u8 cam_id; + u8 ppdu_cnt; + u32 tsf_low; + s8 rx_power[RTW_RF_PATH_MAX]; + u8 rssi; + u8 rxsc; + struct rtw_sta_info *si; + struct ieee80211_vif *vif; +}; + +struct rtw_traffic_stats { + /* units in bytes */ + u64 tx_unicast; + u64 rx_unicast; + + /* count for packets */ + u64 tx_cnt; + u64 rx_cnt; + + /* units in Mbps */ + u32 tx_throughput; + u32 rx_throughput; +}; + +enum rtw_lps_mode { + RTW_MODE_ACTIVE = 0, + RTW_MODE_LPS = 1, + RTW_MODE_WMM_PS = 2, +}; + +enum rtw_pwr_state { + RTW_RF_OFF = 0x0, + RTW_RF_ON = 0x4, + RTW_ALL_ON = 0xc, +}; + +struct rtw_lps_conf { + /* the interface to enter lps */ + struct rtw_vif *rtwvif; + enum rtw_lps_mode mode; + enum rtw_pwr_state state; + u8 awake_interval; + u8 rlbm; + u8 smart_ps; + u8 port_id; +}; + +enum rtw_hw_key_type { + RTW_CAM_NONE = 0, + RTW_CAM_WEP40 = 1, + RTW_CAM_TKIP = 2, + RTW_CAM_AES = 4, + RTW_CAM_WEP104 = 5, +}; + +struct rtw_cam_entry { + bool valid; + bool group; + u8 addr[ETH_ALEN]; + u8 hw_key_type; + struct ieee80211_key_conf *key; +}; + +struct rtw_sec_desc { + /* search strategy */ + bool default_key_search; + + u32 total_cam_num; + struct rtw_cam_entry cam_table[RTW_MAX_SEC_CAM_NUM]; + DECLARE_BITMAP(cam_map, RTW_MAX_SEC_CAM_NUM); +}; + +struct rtw_tx_report { + /* protect the tx report queue */ + spinlock_t q_lock; + struct sk_buff_head queue; + atomic_t sn; + struct timer_list purge_timer; +}; + +#define RTW_BC_MC_MACID 1 +DECLARE_EWMA(rssi, 10, 16); + +struct rtw_sta_info { + struct ieee80211_sta *sta; + struct ieee80211_vif *vif; + + struct ewma_rssi avg_rssi; + u8 rssi_level; + + u8 mac_id; + u8 rate_id; + enum rtw_bandwidth bw_mode; + enum rtw_rf_type rf_type; + enum rtw_wireless_set wireless_set; + u8 stbc_en:2; + u8 ldpc_en:2; + bool sgi_enable; + bool vht_enable; + bool updated; + u8 init_ra_lv; + u64 ra_mask; +}; + +struct rtw_vif { + struct ieee80211_vif *vif; + enum rtw_net_type net_type; + u16 aid; + u8 mac_addr[ETH_ALEN]; + u8 bssid[ETH_ALEN]; + u8 port; + const struct rtw_vif_port *conf; + + struct rtw_traffic_stats stats; + bool in_lps; +}; + +struct rtw_regulatory { + char alpha2[2]; + u8 chplan; + u8 txpwr_regd; +}; + +struct rtw_chip_ops { + int (*mac_init)(struct rtw_dev *rtwdev); + int (*read_efuse)(struct rtw_dev *rtwdev, u8 *map); + void (*phy_set_param)(struct rtw_dev *rtwdev); + void (*set_channel)(struct rtw_dev *rtwdev, u8 channel, + u8 bandwidth, u8 primary_chan_idx); + void (*query_rx_desc)(struct rtw_dev *rtwdev, u8 *rx_desc, + struct rtw_rx_pkt_stat *pkt_stat, + struct ieee80211_rx_status *rx_status); + u32 (*read_rf)(struct rtw_dev *rtwdev, enum rtw_rf_path rf_path, + u32 addr, u32 mask); + bool (*write_rf)(struct rtw_dev *rtwdev, enum rtw_rf_path rf_path, + u32 addr, u32 mask, u32 data); + void (*set_tx_power_index)(struct rtw_dev *rtwdev); + int (*rsvd_page_dump)(struct rtw_dev *rtwdev, u8 *buf, u32 offset, + u32 size); + void (*set_antenna)(struct rtw_dev *rtwdev, u8 antenna_tx, + u8 antenna_rx); + void (*cfg_ldo25)(struct rtw_dev *rtwdev, bool enable); + void (*false_alarm_statistics)(struct rtw_dev *rtwdev); + void (*do_iqk)(struct rtw_dev *rtwdev); +}; + +#define RTW_PWR_POLLING_CNT 20000 + +#define RTW_PWR_CMD_READ 0x00 +#define RTW_PWR_CMD_WRITE 0x01 +#define RTW_PWR_CMD_POLLING 0x02 +#define RTW_PWR_CMD_DELAY 0x03 +#define RTW_PWR_CMD_END 0x04 + +/* define the base address of each block */ +#define RTW_PWR_ADDR_MAC 0x00 +#define RTW_PWR_ADDR_USB 0x01 +#define RTW_PWR_ADDR_PCIE 0x02 +#define RTW_PWR_ADDR_SDIO 0x03 + +#define RTW_PWR_INTF_SDIO_MSK BIT(0) +#define RTW_PWR_INTF_USB_MSK BIT(1) +#define RTW_PWR_INTF_PCI_MSK BIT(2) +#define RTW_PWR_INTF_ALL_MSK (BIT(0) | BIT(1) | BIT(2) | BIT(3)) + +#define RTW_PWR_CUT_A_MSK BIT(1) +#define RTW_PWR_CUT_B_MSK BIT(2) +#define RTW_PWR_CUT_C_MSK BIT(3) +#define RTW_PWR_CUT_D_MSK BIT(4) +#define RTW_PWR_CUT_E_MSK BIT(5) +#define RTW_PWR_CUT_F_MSK BIT(6) +#define RTW_PWR_CUT_G_MSK BIT(7) +#define RTW_PWR_CUT_ALL_MSK 0xFF + +enum rtw_pwr_seq_cmd_delay_unit { + RTW_PWR_DELAY_US, + RTW_PWR_DELAY_MS, +}; + +struct rtw_pwr_seq_cmd { + u16 offset; + u8 cut_mask; + u8 intf_mask; + u8 base:4; + u8 cmd:4; + u8 mask; + u8 value; +}; + +enum rtw_chip_ver { + RTW_CHIP_VER_CUT_A = 0x00, + RTW_CHIP_VER_CUT_B = 0x01, + RTW_CHIP_VER_CUT_C = 0x02, + RTW_CHIP_VER_CUT_D = 0x03, + RTW_CHIP_VER_CUT_E = 0x04, + RTW_CHIP_VER_CUT_F = 0x05, + RTW_CHIP_VER_CUT_G = 0x06, +}; + +#define RTW_INTF_PHY_PLATFORM_ALL 0 + +enum rtw_intf_phy_cut { + RTW_INTF_PHY_CUT_A = BIT(0), + RTW_INTF_PHY_CUT_B = BIT(1), + RTW_INTF_PHY_CUT_C = BIT(2), + RTW_INTF_PHY_CUT_D = BIT(3), + RTW_INTF_PHY_CUT_E = BIT(4), + RTW_INTF_PHY_CUT_F = BIT(5), + RTW_INTF_PHY_CUT_G = BIT(6), + RTW_INTF_PHY_CUT_ALL = 0xFFFF, +}; + +enum rtw_ip_sel { + RTW_IP_SEL_PHY = 0, + RTW_IP_SEL_MAC = 1, + RTW_IP_SEL_DBI = 2, + + RTW_IP_SEL_UNDEF = 0xFFFF +}; + +enum rtw_pq_map_id { + RTW_PQ_MAP_VO = 0x0, + RTW_PQ_MAP_VI = 0x1, + RTW_PQ_MAP_BE = 0x2, + RTW_PQ_MAP_BK = 0x3, + RTW_PQ_MAP_MG = 0x4, + RTW_PQ_MAP_HI = 0x5, + RTW_PQ_MAP_NUM = 0x6, + + RTW_PQ_MAP_UNDEF, +}; + +enum rtw_dma_mapping { + RTW_DMA_MAPPING_EXTRA = 0, + RTW_DMA_MAPPING_LOW = 1, + RTW_DMA_MAPPING_NORMAL = 2, + RTW_DMA_MAPPING_HIGH = 3, + + RTW_DMA_MAPPING_UNDEF, +}; + +struct rtw_rqpn { + enum rtw_dma_mapping dma_map_vo; + enum rtw_dma_mapping dma_map_vi; + enum rtw_dma_mapping dma_map_be; + enum rtw_dma_mapping dma_map_bk; + enum rtw_dma_mapping dma_map_mg; + enum rtw_dma_mapping dma_map_hi; +}; + +struct rtw_page_table { + u16 hq_num; + u16 nq_num; + u16 lq_num; + u16 exq_num; + u16 gapq_num; +}; + +struct rtw_intf_phy_para { + u16 offset; + u16 value; + u16 ip_sel; + u16 cut_mask; + u16 platform; +}; + +struct rtw_intf_phy_para_table { + struct rtw_intf_phy_para *usb2_para; + struct rtw_intf_phy_para *usb3_para; + struct rtw_intf_phy_para *gen1_para; + struct rtw_intf_phy_para *gen2_para; + u8 n_usb2_para; + u8 n_usb3_para; + u8 n_gen1_para; + u8 n_gen2_para; +}; + +struct rtw_table { + const void *data; + const u32 size; + void (*parse)(struct rtw_dev *rtwdev, const struct rtw_table *tbl); + void (*do_cfg)(struct rtw_dev *rtwdev, const struct rtw_table *tbl, + u32 addr, u32 data); + enum rtw_rf_path rf_path; +}; + +static inline void rtw_load_table(struct rtw_dev *rtwdev, + const struct rtw_table *tbl) +{ + (*tbl->parse)(rtwdev, tbl); +} + +enum rtw_rfe_fem { + RTW_RFE_IFEM, + RTW_RFE_EFEM, + RTW_RFE_IFEM2G_EFEM5G, + RTW_RFE_NUM, +}; + +struct rtw_rfe_def { + const struct rtw_table *phy_pg_tbl; + const struct rtw_table *txpwr_lmt_tbl; +}; + +#define RTW_DEF_RFE(chip, bb_pg, pwrlmt) { \ + .phy_pg_tbl = &rtw ## chip ## _bb_pg_type ## bb_pg ## _tbl, \ + .txpwr_lmt_tbl = &rtw ## chip ## _txpwr_lmt_type ## pwrlmt ## _tbl, \ + } + +/* hardware configuration for each IC */ +struct rtw_chip_info { + struct rtw_chip_ops *ops; + u8 id; + + const char *fw_name; + u8 tx_pkt_desc_sz; + u8 tx_buf_desc_sz; + u8 rx_pkt_desc_sz; + u8 rx_buf_desc_sz; + u32 phy_efuse_size; + u32 log_efuse_size; + u32 ptct_efuse_size; + u32 txff_size; + u32 rxff_size; + u8 band; + u8 page_size; + u8 csi_buf_pg_num; + u8 dig_max; + u8 dig_min; + u8 txgi_factor; + bool is_pwr_by_rate_dec; + u8 max_power_index; + + bool ht_supported; + bool vht_supported; + + /* init values */ + u8 sys_func_en; + struct rtw_pwr_seq_cmd **pwr_on_seq; + struct rtw_pwr_seq_cmd **pwr_off_seq; + struct rtw_rqpn *rqpn_table; + struct rtw_page_table *page_table; + struct rtw_intf_phy_para_table *intf_table; + + struct rtw_hw_reg *dig; + u32 rf_base_addr[2]; + u32 rf_sipi_addr[2]; + + const struct rtw_table *mac_tbl; + const struct rtw_table *agc_tbl; + const struct rtw_table *bb_tbl; + const struct rtw_table *rf_tbl[RTW_RF_PATH_MAX]; + const struct rtw_table *rfk_init_tbl; + + const struct rtw_rfe_def *rfe_defs; + u32 rfe_defs_size; +}; + +struct rtw_dm_info { + u32 cck_fa_cnt; + u32 ofdm_fa_cnt; + u32 total_fa_cnt; + u8 min_rssi; + u8 pre_min_rssi; + u16 fa_history[4]; + u8 igi_history[4]; + u8 igi_bitmap; + bool damping; + u8 damping_cnt; + u8 damping_rssi; + + u8 cck_gi_u_bnd; + u8 cck_gi_l_bnd; +}; + +struct rtw_efuse { + u32 size; + u32 physical_size; + u32 logical_size; + u32 protect_size; + + u8 addr[ETH_ALEN]; + u8 channel_plan; + u8 country_code[2]; + u8 rfe_option; + u8 thermal_meter; + u8 crystal_cap; + u8 ant_div_cfg; + u8 ant_div_type; + u8 regd; + + u8 lna_type_2g; + u8 lna_type_5g; + u8 glna_type; + u8 alna_type; + bool ext_lna_2g; + bool ext_lna_5g; + u8 pa_type_2g; + u8 pa_type_5g; + u8 gpa_type; + u8 apa_type; + bool ext_pa_2g; + bool ext_pa_5g; + + bool btcoex; + /* bt share antenna with wifi */ + bool share_ant; + u8 bt_setting; + + struct { + u8 hci; + u8 bw; + u8 ptcl; + u8 nss; + u8 ant_num; + } hw_cap; + + struct rtw_txpwr_idx txpwr_idx_table[4]; +}; + +struct rtw_phy_cond { +#ifdef __LITTLE_ENDIAN + u32 rfe:8; + u32 intf:4; + u32 pkg:4; + u32 plat:4; + u32 intf_rsvd:4; + u32 cut:4; + u32 branch:2; + u32 neg:1; + u32 pos:1; +#else + u32 pos:1; + u32 neg:1; + u32 branch:2; + u32 cut:4; + u32 intf_rsvd:4; + u32 plat:4; + u32 pkg:4; + u32 intf:4; + u32 rfe:8; +#endif + /* for intf:4 */ + #define INTF_PCIE BIT(0) + #define INTF_USB BIT(1) + #define INTF_SDIO BIT(2) + /* for branch:2 */ + #define BRANCH_IF 0 + #define BRANCH_ELIF 1 + #define BRANCH_ELSE 2 + #define BRANCH_ENDIF 3 +}; + +struct rtw_fifo_conf { + /* tx fifo information */ + u16 rsvd_boundary; + u16 rsvd_pg_num; + u16 rsvd_drv_pg_num; + u16 txff_pg_num; + u16 acq_pg_num; + u16 rsvd_drv_addr; + u16 rsvd_h2c_info_addr; + u16 rsvd_h2c_sta_info_addr; + u16 rsvd_h2cq_addr; + u16 rsvd_cpu_instr_addr; + u16 rsvd_fw_txbuf_addr; + u16 rsvd_csibuf_addr; + enum rtw_dma_mapping pq_map[RTW_PQ_MAP_NUM]; +}; + +struct rtw_fw_state { + const struct firmware *firmware; + struct completion completion; + u16 version; + u8 sub_version; + u8 sub_index; + u16 h2c_version; +}; + +struct rtw_hal { + u32 rcr; + + u32 chip_version; + u8 fab_version; + u8 cut_version; + u8 mp_chip; + u8 oem_id; + struct rtw_phy_cond phy_cond; + + u8 ps_mode; + u8 current_channel; + u8 current_band_width; + u8 current_band_type; + u8 sec_ch_offset; + u8 rf_type; + u8 rf_path_num; + u8 antenna_tx; + u8 antenna_rx; + + /* protect tx power section */ + struct mutex tx_power_mutex; + s8 tx_pwr_by_rate_offset_2g[RTW_RF_PATH_MAX] + [DESC_RATE_MAX]; + s8 tx_pwr_by_rate_offset_5g[RTW_RF_PATH_MAX] + [DESC_RATE_MAX]; + s8 tx_pwr_by_rate_base_2g[RTW_RF_PATH_MAX] + [RTW_RATE_SECTION_MAX]; + s8 tx_pwr_by_rate_base_5g[RTW_RF_PATH_MAX] + [RTW_RATE_SECTION_MAX]; + s8 tx_pwr_limit_2g[RTW_REGD_MAX] + [RTW_CHANNEL_WIDTH_MAX] + [RTW_RATE_SECTION_MAX] + [RTW_MAX_CHANNEL_NUM_2G]; + s8 tx_pwr_limit_5g[RTW_REGD_MAX] + [RTW_CHANNEL_WIDTH_MAX] + [RTW_RATE_SECTION_MAX] + [RTW_MAX_CHANNEL_NUM_5G]; + s8 tx_pwr_tbl[RTW_RF_PATH_MAX] + [DESC_RATE_MAX]; +}; + +struct rtw_dev { + struct ieee80211_hw *hw; + struct device *dev; + + struct rtw_hci hci; + + struct rtw_chip_info *chip; + struct rtw_hal hal; + struct rtw_fifo_conf fifo; + struct rtw_fw_state fw; + struct rtw_efuse efuse; + struct rtw_sec_desc sec; + struct rtw_traffic_stats stats; + struct rtw_regulatory regd; + + struct rtw_dm_info dm_info; + + /* ensures exclusive access from mac80211 callbacks */ + struct mutex mutex; + + /* lock for dm to use */ + spinlock_t dm_lock; + + /* read/write rf register */ + spinlock_t rf_lock; + + /* watch dog every 2 sec */ + struct delayed_work watch_dog_work; + u32 watch_dog_cnt; + + struct list_head rsvd_page_list; + + /* c2h cmd queue & handler work */ + struct sk_buff_head c2h_queue; + struct work_struct c2h_work; + + struct rtw_tx_report tx_report; + + struct { + /* incicate the mail box to use with fw */ + u8 last_box_num; + /* protect to send h2c to fw */ + spinlock_t lock; + u32 seq; + } h2c; + + /* lps power state & handler work */ + struct rtw_lps_conf lps_conf; + struct delayed_work lps_work; + + struct dentry *debugfs; + + u8 sta_cnt; + + DECLARE_BITMAP(mac_id_map, RTW_MAX_MAC_ID_NUM); + DECLARE_BITMAP(flags, NUM_OF_RTW_FLAGS); + + u8 mp_mode; + + /* hci related data, must be last */ + u8 priv[0] __aligned(sizeof(void *)); +}; + +#include "hci.h" + +static inline bool rtw_flag_check(struct rtw_dev *rtwdev, enum rtw_flags flag) +{ + return test_bit(flag, rtwdev->flags); +} + +static inline void rtw_flag_clear(struct rtw_dev *rtwdev, enum rtw_flags flag) +{ + clear_bit(flag, rtwdev->flags); +} + +static inline void rtw_flag_set(struct rtw_dev *rtwdev, enum rtw_flags flag) +{ + set_bit(flag, rtwdev->flags); +} + +void rtw_get_channel_params(struct cfg80211_chan_def *chandef, + struct rtw_channel_params *ch_param); +bool check_hw_ready(struct rtw_dev *rtwdev, u32 addr, u32 mask, u32 target); +bool ltecoex_read_reg(struct rtw_dev *rtwdev, u16 offset, u32 *val); +bool ltecoex_reg_write(struct rtw_dev *rtwdev, u16 offset, u32 value); +void rtw_restore_reg(struct rtw_dev *rtwdev, + struct rtw_backup_info *bckp, u32 num); +void rtw_set_channel(struct rtw_dev *rtwdev); +void rtw_vif_port_config(struct rtw_dev *rtwdev, struct rtw_vif *rtwvif, + u32 config); +void rtw_tx_report_purge_timer(struct timer_list *t); +void rtw_update_sta_info(struct rtw_dev *rtwdev, struct rtw_sta_info *si); +int rtw_core_start(struct rtw_dev *rtwdev); +void rtw_core_stop(struct rtw_dev *rtwdev); +int rtw_chip_info_setup(struct rtw_dev *rtwdev); +int rtw_core_init(struct rtw_dev *rtwdev); +void rtw_core_deinit(struct rtw_dev *rtwdev); +int rtw_register_hw(struct rtw_dev *rtwdev, struct ieee80211_hw *hw); +void rtw_unregister_hw(struct rtw_dev *rtwdev, struct ieee80211_hw *hw); + +#endif diff --git a/drivers/net/wireless/realtek/rtw88/pci.c b/drivers/net/wireless/realtek/rtw88/pci.c new file mode 100644 index 0000000000000000000000000000000000000000..cfe05ba7280de520a51f1505cc82279f38cac4d2 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/pci.c @@ -0,0 +1,1211 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#include +#include +#include "main.h" +#include "pci.h" +#include "tx.h" +#include "rx.h" +#include "debug.h" + +static u32 rtw_pci_tx_queue_idx_addr[] = { + [RTW_TX_QUEUE_BK] = RTK_PCI_TXBD_IDX_BKQ, + [RTW_TX_QUEUE_BE] = RTK_PCI_TXBD_IDX_BEQ, + [RTW_TX_QUEUE_VI] = RTK_PCI_TXBD_IDX_VIQ, + [RTW_TX_QUEUE_VO] = RTK_PCI_TXBD_IDX_VOQ, + [RTW_TX_QUEUE_MGMT] = RTK_PCI_TXBD_IDX_MGMTQ, + [RTW_TX_QUEUE_HI0] = RTK_PCI_TXBD_IDX_HI0Q, + [RTW_TX_QUEUE_H2C] = RTK_PCI_TXBD_IDX_H2CQ, +}; + +static u8 rtw_pci_get_tx_qsel(struct sk_buff *skb, u8 queue) +{ + switch (queue) { + case RTW_TX_QUEUE_BCN: + return TX_DESC_QSEL_BEACON; + case RTW_TX_QUEUE_H2C: + return TX_DESC_QSEL_H2C; + case RTW_TX_QUEUE_MGMT: + return TX_DESC_QSEL_MGMT; + case RTW_TX_QUEUE_HI0: + return TX_DESC_QSEL_HIGH; + default: + return skb->priority; + } +}; + +static u8 rtw_pci_read8(struct rtw_dev *rtwdev, u32 addr) +{ + struct rtw_pci *rtwpci = (struct rtw_pci *)rtwdev->priv; + + return readb(rtwpci->mmap + addr); +} + +static u16 rtw_pci_read16(struct rtw_dev *rtwdev, u32 addr) +{ + struct rtw_pci *rtwpci = (struct rtw_pci *)rtwdev->priv; + + return readw(rtwpci->mmap + addr); +} + +static u32 rtw_pci_read32(struct rtw_dev *rtwdev, u32 addr) +{ + struct rtw_pci *rtwpci = (struct rtw_pci *)rtwdev->priv; + + return readl(rtwpci->mmap + addr); +} + +static void rtw_pci_write8(struct rtw_dev *rtwdev, u32 addr, u8 val) +{ + struct rtw_pci *rtwpci = (struct rtw_pci *)rtwdev->priv; + + writeb(val, rtwpci->mmap + addr); +} + +static void rtw_pci_write16(struct rtw_dev *rtwdev, u32 addr, u16 val) +{ + struct rtw_pci *rtwpci = (struct rtw_pci *)rtwdev->priv; + + writew(val, rtwpci->mmap + addr); +} + +static void rtw_pci_write32(struct rtw_dev *rtwdev, u32 addr, u32 val) +{ + struct rtw_pci *rtwpci = (struct rtw_pci *)rtwdev->priv; + + writel(val, rtwpci->mmap + addr); +} + +static inline void *rtw_pci_get_tx_desc(struct rtw_pci_tx_ring *tx_ring, u8 idx) +{ + int offset = tx_ring->r.desc_size * idx; + + return tx_ring->r.head + offset; +} + +static void rtw_pci_free_tx_ring(struct rtw_dev *rtwdev, + struct rtw_pci_tx_ring *tx_ring) +{ + struct pci_dev *pdev = to_pci_dev(rtwdev->dev); + struct rtw_pci_tx_data *tx_data; + struct sk_buff *skb, *tmp; + dma_addr_t dma; + u8 *head = tx_ring->r.head; + u32 len = tx_ring->r.len; + int ring_sz = len * tx_ring->r.desc_size; + + /* free every skb remained in tx list */ + skb_queue_walk_safe(&tx_ring->queue, skb, tmp) { + __skb_unlink(skb, &tx_ring->queue); + tx_data = rtw_pci_get_tx_data(skb); + dma = tx_data->dma; + + pci_unmap_single(pdev, dma, skb->len, PCI_DMA_TODEVICE); + dev_kfree_skb_any(skb); + } + + /* free the ring itself */ + pci_free_consistent(pdev, ring_sz, head, tx_ring->r.dma); + tx_ring->r.head = NULL; +} + +static void rtw_pci_free_rx_ring(struct rtw_dev *rtwdev, + struct rtw_pci_rx_ring *rx_ring) +{ + struct pci_dev *pdev = to_pci_dev(rtwdev->dev); + struct sk_buff *skb; + dma_addr_t dma; + u8 *head = rx_ring->r.head; + int buf_sz = RTK_PCI_RX_BUF_SIZE; + int ring_sz = rx_ring->r.desc_size * rx_ring->r.len; + int i; + + for (i = 0; i < rx_ring->r.len; i++) { + skb = rx_ring->buf[i]; + if (!skb) + continue; + + dma = *((dma_addr_t *)skb->cb); + pci_unmap_single(pdev, dma, buf_sz, PCI_DMA_FROMDEVICE); + dev_kfree_skb(skb); + rx_ring->buf[i] = NULL; + } + + pci_free_consistent(pdev, ring_sz, head, rx_ring->r.dma); +} + +static void rtw_pci_free_trx_ring(struct rtw_dev *rtwdev) +{ + struct rtw_pci *rtwpci = (struct rtw_pci *)rtwdev->priv; + struct rtw_pci_tx_ring *tx_ring; + struct rtw_pci_rx_ring *rx_ring; + int i; + + for (i = 0; i < RTK_MAX_TX_QUEUE_NUM; i++) { + tx_ring = &rtwpci->tx_rings[i]; + rtw_pci_free_tx_ring(rtwdev, tx_ring); + } + + for (i = 0; i < RTK_MAX_RX_QUEUE_NUM; i++) { + rx_ring = &rtwpci->rx_rings[i]; + rtw_pci_free_rx_ring(rtwdev, rx_ring); + } +} + +static int rtw_pci_init_tx_ring(struct rtw_dev *rtwdev, + struct rtw_pci_tx_ring *tx_ring, + u8 desc_size, u32 len) +{ + struct pci_dev *pdev = to_pci_dev(rtwdev->dev); + int ring_sz = desc_size * len; + dma_addr_t dma; + u8 *head; + + head = pci_zalloc_consistent(pdev, ring_sz, &dma); + if (!head) { + rtw_err(rtwdev, "failed to allocate tx ring\n"); + return -ENOMEM; + } + + skb_queue_head_init(&tx_ring->queue); + tx_ring->r.head = head; + tx_ring->r.dma = dma; + tx_ring->r.len = len; + tx_ring->r.desc_size = desc_size; + tx_ring->r.wp = 0; + tx_ring->r.rp = 0; + + return 0; +} + +static int rtw_pci_reset_rx_desc(struct rtw_dev *rtwdev, struct sk_buff *skb, + struct rtw_pci_rx_ring *rx_ring, + u32 idx, u32 desc_sz) +{ + struct pci_dev *pdev = to_pci_dev(rtwdev->dev); + struct rtw_pci_rx_buffer_desc *buf_desc; + int buf_sz = RTK_PCI_RX_BUF_SIZE; + dma_addr_t dma; + + if (!skb) + return -EINVAL; + + dma = pci_map_single(pdev, skb->data, buf_sz, PCI_DMA_FROMDEVICE); + if (pci_dma_mapping_error(pdev, dma)) + return -EBUSY; + + *((dma_addr_t *)skb->cb) = dma; + buf_desc = (struct rtw_pci_rx_buffer_desc *)(rx_ring->r.head + + idx * desc_sz); + memset(buf_desc, 0, sizeof(*buf_desc)); + buf_desc->buf_size = cpu_to_le16(RTK_PCI_RX_BUF_SIZE); + buf_desc->dma = cpu_to_le32(dma); + + return 0; +} + +static int rtw_pci_init_rx_ring(struct rtw_dev *rtwdev, + struct rtw_pci_rx_ring *rx_ring, + u8 desc_size, u32 len) +{ + struct pci_dev *pdev = to_pci_dev(rtwdev->dev); + struct sk_buff *skb = NULL; + dma_addr_t dma; + u8 *head; + int ring_sz = desc_size * len; + int buf_sz = RTK_PCI_RX_BUF_SIZE; + int i, allocated; + int ret = 0; + + head = pci_zalloc_consistent(pdev, ring_sz, &dma); + if (!head) { + rtw_err(rtwdev, "failed to allocate rx ring\n"); + return -ENOMEM; + } + rx_ring->r.head = head; + + for (i = 0; i < len; i++) { + skb = dev_alloc_skb(buf_sz); + if (!skb) { + allocated = i; + ret = -ENOMEM; + goto err_out; + } + + memset(skb->data, 0, buf_sz); + rx_ring->buf[i] = skb; + ret = rtw_pci_reset_rx_desc(rtwdev, skb, rx_ring, i, desc_size); + if (ret) { + allocated = i; + dev_kfree_skb_any(skb); + goto err_out; + } + } + + rx_ring->r.dma = dma; + rx_ring->r.len = len; + rx_ring->r.desc_size = desc_size; + rx_ring->r.wp = 0; + rx_ring->r.rp = 0; + + return 0; + +err_out: + for (i = 0; i < allocated; i++) { + skb = rx_ring->buf[i]; + if (!skb) + continue; + dma = *((dma_addr_t *)skb->cb); + pci_unmap_single(pdev, dma, buf_sz, PCI_DMA_FROMDEVICE); + dev_kfree_skb_any(skb); + rx_ring->buf[i] = NULL; + } + pci_free_consistent(pdev, ring_sz, head, dma); + + rtw_err(rtwdev, "failed to init rx buffer\n"); + + return ret; +} + +static int rtw_pci_init_trx_ring(struct rtw_dev *rtwdev) +{ + struct rtw_pci *rtwpci = (struct rtw_pci *)rtwdev->priv; + struct rtw_pci_tx_ring *tx_ring; + struct rtw_pci_rx_ring *rx_ring; + struct rtw_chip_info *chip = rtwdev->chip; + int i = 0, j = 0, tx_alloced = 0, rx_alloced = 0; + int tx_desc_size, rx_desc_size; + u32 len; + int ret; + + tx_desc_size = chip->tx_buf_desc_sz; + + for (i = 0; i < RTK_MAX_TX_QUEUE_NUM; i++) { + tx_ring = &rtwpci->tx_rings[i]; + len = max_num_of_tx_queue(i); + ret = rtw_pci_init_tx_ring(rtwdev, tx_ring, tx_desc_size, len); + if (ret) + goto out; + } + + rx_desc_size = chip->rx_buf_desc_sz; + + for (j = 0; j < RTK_MAX_RX_QUEUE_NUM; j++) { + rx_ring = &rtwpci->rx_rings[j]; + ret = rtw_pci_init_rx_ring(rtwdev, rx_ring, rx_desc_size, + RTK_MAX_RX_DESC_NUM); + if (ret) + goto out; + } + + return 0; + +out: + tx_alloced = i; + for (i = 0; i < tx_alloced; i++) { + tx_ring = &rtwpci->tx_rings[i]; + rtw_pci_free_tx_ring(rtwdev, tx_ring); + } + + rx_alloced = j; + for (j = 0; j < rx_alloced; j++) { + rx_ring = &rtwpci->rx_rings[j]; + rtw_pci_free_rx_ring(rtwdev, rx_ring); + } + + return ret; +} + +static void rtw_pci_deinit(struct rtw_dev *rtwdev) +{ + rtw_pci_free_trx_ring(rtwdev); +} + +static int rtw_pci_init(struct rtw_dev *rtwdev) +{ + struct rtw_pci *rtwpci = (struct rtw_pci *)rtwdev->priv; + int ret = 0; + + rtwpci->irq_mask[0] = IMR_HIGHDOK | + IMR_MGNTDOK | + IMR_BKDOK | + IMR_BEDOK | + IMR_VIDOK | + IMR_VODOK | + IMR_ROK | + IMR_BCNDMAINT_E | + 0; + rtwpci->irq_mask[1] = IMR_TXFOVW | + 0; + rtwpci->irq_mask[3] = IMR_H2CDOK | + 0; + spin_lock_init(&rtwpci->irq_lock); + ret = rtw_pci_init_trx_ring(rtwdev); + + return ret; +} + +static void rtw_pci_reset_buf_desc(struct rtw_dev *rtwdev) +{ + struct rtw_pci *rtwpci = (struct rtw_pci *)rtwdev->priv; + u32 len; + u8 tmp; + dma_addr_t dma; + + tmp = rtw_read8(rtwdev, RTK_PCI_CTRL + 3); + rtw_write8(rtwdev, RTK_PCI_CTRL + 3, tmp | 0xf7); + + dma = rtwpci->tx_rings[RTW_TX_QUEUE_BCN].r.dma; + rtw_write32(rtwdev, RTK_PCI_TXBD_DESA_BCNQ, dma); + + len = rtwpci->tx_rings[RTW_TX_QUEUE_H2C].r.len; + dma = rtwpci->tx_rings[RTW_TX_QUEUE_H2C].r.dma; + rtwpci->tx_rings[RTW_TX_QUEUE_H2C].r.rp = 0; + rtwpci->tx_rings[RTW_TX_QUEUE_H2C].r.wp = 0; + rtw_write16(rtwdev, RTK_PCI_TXBD_NUM_H2CQ, len); + rtw_write32(rtwdev, RTK_PCI_TXBD_DESA_H2CQ, dma); + + len = rtwpci->tx_rings[RTW_TX_QUEUE_BK].r.len; + dma = rtwpci->tx_rings[RTW_TX_QUEUE_BK].r.dma; + rtwpci->tx_rings[RTW_TX_QUEUE_BK].r.rp = 0; + rtwpci->tx_rings[RTW_TX_QUEUE_BK].r.wp = 0; + rtw_write16(rtwdev, RTK_PCI_TXBD_NUM_BKQ, len); + rtw_write32(rtwdev, RTK_PCI_TXBD_DESA_BKQ, dma); + + len = rtwpci->tx_rings[RTW_TX_QUEUE_BE].r.len; + dma = rtwpci->tx_rings[RTW_TX_QUEUE_BE].r.dma; + rtwpci->tx_rings[RTW_TX_QUEUE_BE].r.rp = 0; + rtwpci->tx_rings[RTW_TX_QUEUE_BE].r.wp = 0; + rtw_write16(rtwdev, RTK_PCI_TXBD_NUM_BEQ, len); + rtw_write32(rtwdev, RTK_PCI_TXBD_DESA_BEQ, dma); + + len = rtwpci->tx_rings[RTW_TX_QUEUE_VO].r.len; + dma = rtwpci->tx_rings[RTW_TX_QUEUE_VO].r.dma; + rtwpci->tx_rings[RTW_TX_QUEUE_VO].r.rp = 0; + rtwpci->tx_rings[RTW_TX_QUEUE_VO].r.wp = 0; + rtw_write16(rtwdev, RTK_PCI_TXBD_NUM_VOQ, len); + rtw_write32(rtwdev, RTK_PCI_TXBD_DESA_VOQ, dma); + + len = rtwpci->tx_rings[RTW_TX_QUEUE_VI].r.len; + dma = rtwpci->tx_rings[RTW_TX_QUEUE_VI].r.dma; + rtwpci->tx_rings[RTW_TX_QUEUE_VI].r.rp = 0; + rtwpci->tx_rings[RTW_TX_QUEUE_VI].r.wp = 0; + rtw_write16(rtwdev, RTK_PCI_TXBD_NUM_VIQ, len); + rtw_write32(rtwdev, RTK_PCI_TXBD_DESA_VIQ, dma); + + len = rtwpci->tx_rings[RTW_TX_QUEUE_MGMT].r.len; + dma = rtwpci->tx_rings[RTW_TX_QUEUE_MGMT].r.dma; + rtwpci->tx_rings[RTW_TX_QUEUE_MGMT].r.rp = 0; + rtwpci->tx_rings[RTW_TX_QUEUE_MGMT].r.wp = 0; + rtw_write16(rtwdev, RTK_PCI_TXBD_NUM_MGMTQ, len); + rtw_write32(rtwdev, RTK_PCI_TXBD_DESA_MGMTQ, dma); + + len = rtwpci->tx_rings[RTW_TX_QUEUE_HI0].r.len; + dma = rtwpci->tx_rings[RTW_TX_QUEUE_HI0].r.dma; + rtwpci->tx_rings[RTW_TX_QUEUE_HI0].r.rp = 0; + rtwpci->tx_rings[RTW_TX_QUEUE_HI0].r.wp = 0; + rtw_write16(rtwdev, RTK_PCI_TXBD_NUM_HI0Q, len); + rtw_write32(rtwdev, RTK_PCI_TXBD_DESA_HI0Q, dma); + + len = rtwpci->rx_rings[RTW_RX_QUEUE_MPDU].r.len; + dma = rtwpci->rx_rings[RTW_RX_QUEUE_MPDU].r.dma; + rtwpci->rx_rings[RTW_RX_QUEUE_MPDU].r.rp = 0; + rtwpci->rx_rings[RTW_RX_QUEUE_MPDU].r.wp = 0; + rtw_write16(rtwdev, RTK_PCI_RXBD_NUM_MPDUQ, len & 0xfff); + rtw_write32(rtwdev, RTK_PCI_RXBD_DESA_MPDUQ, dma); + + /* reset read/write point */ + rtw_write32(rtwdev, RTK_PCI_TXBD_RWPTR_CLR, 0xffffffff); + + /* rest H2C Queue index */ + rtw_write32_set(rtwdev, RTK_PCI_TXBD_H2CQ_CSR, BIT_CLR_H2CQ_HOST_IDX); + rtw_write32_set(rtwdev, RTK_PCI_TXBD_H2CQ_CSR, BIT_CLR_H2CQ_HW_IDX); +} + +static void rtw_pci_reset_trx_ring(struct rtw_dev *rtwdev) +{ + rtw_pci_reset_buf_desc(rtwdev); +} + +static void rtw_pci_enable_interrupt(struct rtw_dev *rtwdev, + struct rtw_pci *rtwpci) +{ + rtw_write32(rtwdev, RTK_PCI_HIMR0, rtwpci->irq_mask[0]); + rtw_write32(rtwdev, RTK_PCI_HIMR1, rtwpci->irq_mask[1]); + rtw_write32(rtwdev, RTK_PCI_HIMR3, rtwpci->irq_mask[3]); + rtwpci->irq_enabled = true; +} + +static void rtw_pci_disable_interrupt(struct rtw_dev *rtwdev, + struct rtw_pci *rtwpci) +{ + rtw_write32(rtwdev, RTK_PCI_HIMR0, 0); + rtw_write32(rtwdev, RTK_PCI_HIMR1, 0); + rtw_write32(rtwdev, RTK_PCI_HIMR3, 0); + rtwpci->irq_enabled = false; +} + +static int rtw_pci_setup(struct rtw_dev *rtwdev) +{ + rtw_pci_reset_trx_ring(rtwdev); + + return 0; +} + +static void rtw_pci_dma_reset(struct rtw_dev *rtwdev, struct rtw_pci *rtwpci) +{ + /* reset dma and rx tag */ + rtw_write32_set(rtwdev, RTK_PCI_CTRL, + BIT_RST_TRXDMA_INTF | BIT_RX_TAG_EN); + rtwpci->rx_tag = 0; +} + +static int rtw_pci_start(struct rtw_dev *rtwdev) +{ + struct rtw_pci *rtwpci = (struct rtw_pci *)rtwdev->priv; + unsigned long flags; + + rtw_pci_dma_reset(rtwdev, rtwpci); + + spin_lock_irqsave(&rtwpci->irq_lock, flags); + rtw_pci_enable_interrupt(rtwdev, rtwpci); + spin_unlock_irqrestore(&rtwpci->irq_lock, flags); + + return 0; +} + +static void rtw_pci_stop(struct rtw_dev *rtwdev) +{ + struct rtw_pci *rtwpci = (struct rtw_pci *)rtwdev->priv; + unsigned long flags; + + spin_lock_irqsave(&rtwpci->irq_lock, flags); + rtw_pci_disable_interrupt(rtwdev, rtwpci); + spin_unlock_irqrestore(&rtwpci->irq_lock, flags); +} + +static u8 ac_to_hwq[] = { + [0] = RTW_TX_QUEUE_VO, + [1] = RTW_TX_QUEUE_VI, + [2] = RTW_TX_QUEUE_BE, + [3] = RTW_TX_QUEUE_BK, +}; + +static u8 rtw_hw_queue_mapping(struct sk_buff *skb) +{ + struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)skb->data; + __le16 fc = hdr->frame_control; + u8 q_mapping = skb_get_queue_mapping(skb); + u8 queue; + + if (unlikely(ieee80211_is_beacon(fc))) + queue = RTW_TX_QUEUE_BCN; + else if (unlikely(ieee80211_is_mgmt(fc) || ieee80211_is_ctl(fc))) + queue = RTW_TX_QUEUE_MGMT; + else + queue = ac_to_hwq[q_mapping]; + + return queue; +} + +static void rtw_pci_release_rsvd_page(struct rtw_pci *rtwpci, + struct rtw_pci_tx_ring *ring) +{ + struct sk_buff *prev = skb_dequeue(&ring->queue); + struct rtw_pci_tx_data *tx_data; + dma_addr_t dma; + + if (!prev) + return; + + tx_data = rtw_pci_get_tx_data(prev); + dma = tx_data->dma; + pci_unmap_single(rtwpci->pdev, dma, prev->len, + PCI_DMA_TODEVICE); + dev_kfree_skb_any(prev); +} + +static void rtw_pci_dma_check(struct rtw_dev *rtwdev, + struct rtw_pci_rx_ring *rx_ring, + u32 idx) +{ + struct rtw_pci *rtwpci = (struct rtw_pci *)rtwdev->priv; + struct rtw_chip_info *chip = rtwdev->chip; + struct rtw_pci_rx_buffer_desc *buf_desc; + u32 desc_sz = chip->rx_buf_desc_sz; + u16 total_pkt_size; + + buf_desc = (struct rtw_pci_rx_buffer_desc *)(rx_ring->r.head + + idx * desc_sz); + total_pkt_size = le16_to_cpu(buf_desc->total_pkt_size); + + /* rx tag mismatch, throw a warning */ + if (total_pkt_size != rtwpci->rx_tag) + rtw_warn(rtwdev, "pci bus timeout, check dma status\n"); + + rtwpci->rx_tag = (rtwpci->rx_tag + 1) % RX_TAG_MAX; +} + +static int rtw_pci_xmit(struct rtw_dev *rtwdev, + struct rtw_tx_pkt_info *pkt_info, + struct sk_buff *skb, u8 queue) +{ + struct rtw_pci *rtwpci = (struct rtw_pci *)rtwdev->priv; + struct rtw_chip_info *chip = rtwdev->chip; + struct rtw_pci_tx_ring *ring; + struct rtw_pci_tx_data *tx_data; + dma_addr_t dma; + u32 tx_pkt_desc_sz = chip->tx_pkt_desc_sz; + u32 tx_buf_desc_sz = chip->tx_buf_desc_sz; + u32 size; + u32 psb_len; + u8 *pkt_desc; + struct rtw_pci_tx_buffer_desc *buf_desc; + u32 bd_idx; + + ring = &rtwpci->tx_rings[queue]; + + size = skb->len; + + if (queue == RTW_TX_QUEUE_BCN) + rtw_pci_release_rsvd_page(rtwpci, ring); + else if (!avail_desc(ring->r.wp, ring->r.rp, ring->r.len)) + return -ENOSPC; + + pkt_desc = skb_push(skb, chip->tx_pkt_desc_sz); + memset(pkt_desc, 0, tx_pkt_desc_sz); + pkt_info->qsel = rtw_pci_get_tx_qsel(skb, queue); + rtw_tx_fill_tx_desc(pkt_info, skb); + dma = pci_map_single(rtwpci->pdev, skb->data, skb->len, + PCI_DMA_TODEVICE); + if (pci_dma_mapping_error(rtwpci->pdev, dma)) + return -EBUSY; + + /* after this we got dma mapped, there is no way back */ + buf_desc = get_tx_buffer_desc(ring, tx_buf_desc_sz); + memset(buf_desc, 0, tx_buf_desc_sz); + psb_len = (skb->len - 1) / 128 + 1; + if (queue == RTW_TX_QUEUE_BCN) + psb_len |= 1 << RTK_PCI_TXBD_OWN_OFFSET; + + buf_desc[0].psb_len = cpu_to_le16(psb_len); + buf_desc[0].buf_size = cpu_to_le16(tx_pkt_desc_sz); + buf_desc[0].dma = cpu_to_le32(dma); + buf_desc[1].buf_size = cpu_to_le16(size); + buf_desc[1].dma = cpu_to_le32(dma + tx_pkt_desc_sz); + + tx_data = rtw_pci_get_tx_data(skb); + tx_data->dma = dma; + tx_data->sn = pkt_info->sn; + skb_queue_tail(&ring->queue, skb); + + /* kick off tx queue */ + if (queue != RTW_TX_QUEUE_BCN) { + if (++ring->r.wp >= ring->r.len) + ring->r.wp = 0; + bd_idx = rtw_pci_tx_queue_idx_addr[queue]; + rtw_write16(rtwdev, bd_idx, ring->r.wp & 0xfff); + } else { + u32 reg_bcn_work; + + reg_bcn_work = rtw_read8(rtwdev, RTK_PCI_TXBD_BCN_WORK); + reg_bcn_work |= BIT_PCI_BCNQ_FLAG; + rtw_write8(rtwdev, RTK_PCI_TXBD_BCN_WORK, reg_bcn_work); + } + + return 0; +} + +static int rtw_pci_write_data_rsvd_page(struct rtw_dev *rtwdev, u8 *buf, + u32 size) +{ + struct sk_buff *skb; + struct rtw_tx_pkt_info pkt_info; + u32 tx_pkt_desc_sz; + u32 length; + + tx_pkt_desc_sz = rtwdev->chip->tx_pkt_desc_sz; + length = size + tx_pkt_desc_sz; + skb = dev_alloc_skb(length); + if (!skb) + return -ENOMEM; + + skb_reserve(skb, tx_pkt_desc_sz); + memcpy((u8 *)skb_put(skb, size), buf, size); + memset(&pkt_info, 0, sizeof(pkt_info)); + pkt_info.tx_pkt_size = size; + pkt_info.offset = tx_pkt_desc_sz; + + return rtw_pci_xmit(rtwdev, &pkt_info, skb, RTW_TX_QUEUE_BCN); +} + +static int rtw_pci_write_data_h2c(struct rtw_dev *rtwdev, u8 *buf, u32 size) +{ + struct sk_buff *skb; + struct rtw_tx_pkt_info pkt_info; + u32 tx_pkt_desc_sz; + u32 length; + + tx_pkt_desc_sz = rtwdev->chip->tx_pkt_desc_sz; + length = size + tx_pkt_desc_sz; + skb = dev_alloc_skb(length); + if (!skb) + return -ENOMEM; + + skb_reserve(skb, tx_pkt_desc_sz); + memcpy((u8 *)skb_put(skb, size), buf, size); + memset(&pkt_info, 0, sizeof(pkt_info)); + pkt_info.tx_pkt_size = size; + + return rtw_pci_xmit(rtwdev, &pkt_info, skb, RTW_TX_QUEUE_H2C); +} + +static int rtw_pci_tx(struct rtw_dev *rtwdev, + struct rtw_tx_pkt_info *pkt_info, + struct sk_buff *skb) +{ + struct rtw_pci *rtwpci = (struct rtw_pci *)rtwdev->priv; + struct rtw_pci_tx_ring *ring; + u8 queue = rtw_hw_queue_mapping(skb); + int ret; + + ret = rtw_pci_xmit(rtwdev, pkt_info, skb, queue); + if (ret) + return ret; + + ring = &rtwpci->tx_rings[queue]; + if (avail_desc(ring->r.wp, ring->r.rp, ring->r.len) < 2) { + ieee80211_stop_queue(rtwdev->hw, skb_get_queue_mapping(skb)); + ring->queue_stopped = true; + } + + return 0; +} + +static void rtw_pci_tx_isr(struct rtw_dev *rtwdev, struct rtw_pci *rtwpci, + u8 hw_queue) +{ + struct ieee80211_hw *hw = rtwdev->hw; + struct ieee80211_tx_info *info; + struct rtw_pci_tx_ring *ring; + struct rtw_pci_tx_data *tx_data; + struct sk_buff *skb; + u32 count; + u32 bd_idx_addr; + u32 bd_idx, cur_rp; + u16 q_map; + + ring = &rtwpci->tx_rings[hw_queue]; + + bd_idx_addr = rtw_pci_tx_queue_idx_addr[hw_queue]; + bd_idx = rtw_read32(rtwdev, bd_idx_addr); + cur_rp = bd_idx >> 16; + cur_rp &= 0xfff; + if (cur_rp >= ring->r.rp) + count = cur_rp - ring->r.rp; + else + count = ring->r.len - (ring->r.rp - cur_rp); + + while (count--) { + skb = skb_dequeue(&ring->queue); + tx_data = rtw_pci_get_tx_data(skb); + pci_unmap_single(rtwpci->pdev, tx_data->dma, skb->len, + PCI_DMA_TODEVICE); + + /* just free command packets from host to card */ + if (hw_queue == RTW_TX_QUEUE_H2C) { + dev_kfree_skb_irq(skb); + continue; + } + + if (ring->queue_stopped && + avail_desc(ring->r.wp, ring->r.rp, ring->r.len) > 4) { + q_map = skb_get_queue_mapping(skb); + ieee80211_wake_queue(hw, q_map); + ring->queue_stopped = false; + } + + skb_pull(skb, rtwdev->chip->tx_pkt_desc_sz); + + info = IEEE80211_SKB_CB(skb); + + /* enqueue to wait for tx report */ + if (info->flags & IEEE80211_TX_CTL_REQ_TX_STATUS) { + rtw_tx_report_enqueue(rtwdev, skb, tx_data->sn); + continue; + } + + /* always ACK for others, then they won't be marked as drop */ + if (info->flags & IEEE80211_TX_CTL_NO_ACK) + info->flags |= IEEE80211_TX_STAT_NOACK_TRANSMITTED; + else + info->flags |= IEEE80211_TX_STAT_ACK; + + ieee80211_tx_info_clear_status(info); + ieee80211_tx_status_irqsafe(hw, skb); + } + + ring->r.rp = cur_rp; +} + +static void rtw_pci_rx_isr(struct rtw_dev *rtwdev, struct rtw_pci *rtwpci, + u8 hw_queue) +{ + struct rtw_chip_info *chip = rtwdev->chip; + struct rtw_pci_rx_ring *ring; + struct rtw_rx_pkt_stat pkt_stat; + struct ieee80211_rx_status rx_status; + struct sk_buff *skb, *new; + u32 cur_wp, cur_rp, tmp; + u32 count; + u32 pkt_offset; + u32 pkt_desc_sz = chip->rx_pkt_desc_sz; + u32 buf_desc_sz = chip->rx_buf_desc_sz; + u8 *rx_desc; + dma_addr_t dma; + + ring = &rtwpci->rx_rings[RTW_RX_QUEUE_MPDU]; + + tmp = rtw_read32(rtwdev, RTK_PCI_RXBD_IDX_MPDUQ); + cur_wp = tmp >> 16; + cur_wp &= 0xfff; + if (cur_wp >= ring->r.wp) + count = cur_wp - ring->r.wp; + else + count = ring->r.len - (ring->r.wp - cur_wp); + + cur_rp = ring->r.rp; + while (count--) { + rtw_pci_dma_check(rtwdev, ring, cur_rp); + skb = ring->buf[cur_rp]; + dma = *((dma_addr_t *)skb->cb); + pci_unmap_single(rtwpci->pdev, dma, RTK_PCI_RX_BUF_SIZE, + PCI_DMA_FROMDEVICE); + rx_desc = skb->data; + chip->ops->query_rx_desc(rtwdev, rx_desc, &pkt_stat, &rx_status); + + /* offset from rx_desc to payload */ + pkt_offset = pkt_desc_sz + pkt_stat.drv_info_sz + + pkt_stat.shift; + + if (pkt_stat.is_c2h) { + /* keep rx_desc, halmac needs it */ + skb_put(skb, pkt_stat.pkt_len + pkt_offset); + + /* pass offset for further operation */ + *((u32 *)skb->cb) = pkt_offset; + skb_queue_tail(&rtwdev->c2h_queue, skb); + ieee80211_queue_work(rtwdev->hw, &rtwdev->c2h_work); + } else { + /* remove rx_desc, maybe use skb_pull? */ + skb_put(skb, pkt_stat.pkt_len); + skb_reserve(skb, pkt_offset); + + /* alloc a smaller skb to mac80211 */ + new = dev_alloc_skb(pkt_stat.pkt_len); + if (!new) { + new = skb; + } else { + skb_put_data(new, skb->data, skb->len); + dev_kfree_skb_any(skb); + } + /* TODO: merge into rx.c */ + rtw_rx_stats(rtwdev, pkt_stat.vif, skb); + memcpy(new->cb, &rx_status, sizeof(rx_status)); + ieee80211_rx_irqsafe(rtwdev->hw, new); + } + + /* skb delivered to mac80211, alloc a new one in rx ring */ + new = dev_alloc_skb(RTK_PCI_RX_BUF_SIZE); + if (WARN(!new, "rx routine starvation\n")) + return; + + ring->buf[cur_rp] = new; + rtw_pci_reset_rx_desc(rtwdev, new, ring, cur_rp, buf_desc_sz); + + /* host read next element in ring */ + if (++cur_rp >= ring->r.len) + cur_rp = 0; + } + + ring->r.rp = cur_rp; + ring->r.wp = cur_wp; + rtw_write16(rtwdev, RTK_PCI_RXBD_IDX_MPDUQ, ring->r.rp); +} + +static void rtw_pci_irq_recognized(struct rtw_dev *rtwdev, + struct rtw_pci *rtwpci, u32 *irq_status) +{ + irq_status[0] = rtw_read32(rtwdev, RTK_PCI_HISR0); + irq_status[1] = rtw_read32(rtwdev, RTK_PCI_HISR1); + irq_status[3] = rtw_read32(rtwdev, RTK_PCI_HISR3); + irq_status[0] &= rtwpci->irq_mask[0]; + irq_status[1] &= rtwpci->irq_mask[1]; + irq_status[3] &= rtwpci->irq_mask[3]; + rtw_write32(rtwdev, RTK_PCI_HISR0, irq_status[0]); + rtw_write32(rtwdev, RTK_PCI_HISR1, irq_status[1]); + rtw_write32(rtwdev, RTK_PCI_HISR3, irq_status[3]); +} + +static irqreturn_t rtw_pci_interrupt_handler(int irq, void *dev) +{ + struct rtw_dev *rtwdev = dev; + struct rtw_pci *rtwpci = (struct rtw_pci *)rtwdev->priv; + u32 irq_status[4]; + + spin_lock(&rtwpci->irq_lock); + if (!rtwpci->irq_enabled) + goto out; + + rtw_pci_irq_recognized(rtwdev, rtwpci, irq_status); + + if (irq_status[0] & IMR_MGNTDOK) + rtw_pci_tx_isr(rtwdev, rtwpci, RTW_TX_QUEUE_MGMT); + if (irq_status[0] & IMR_HIGHDOK) + rtw_pci_tx_isr(rtwdev, rtwpci, RTW_TX_QUEUE_HI0); + if (irq_status[0] & IMR_BEDOK) + rtw_pci_tx_isr(rtwdev, rtwpci, RTW_TX_QUEUE_BE); + if (irq_status[0] & IMR_BKDOK) + rtw_pci_tx_isr(rtwdev, rtwpci, RTW_TX_QUEUE_BK); + if (irq_status[0] & IMR_VODOK) + rtw_pci_tx_isr(rtwdev, rtwpci, RTW_TX_QUEUE_VO); + if (irq_status[0] & IMR_VIDOK) + rtw_pci_tx_isr(rtwdev, rtwpci, RTW_TX_QUEUE_VI); + if (irq_status[3] & IMR_H2CDOK) + rtw_pci_tx_isr(rtwdev, rtwpci, RTW_TX_QUEUE_H2C); + if (irq_status[0] & IMR_ROK) + rtw_pci_rx_isr(rtwdev, rtwpci, RTW_RX_QUEUE_MPDU); + +out: + spin_unlock(&rtwpci->irq_lock); + + return IRQ_HANDLED; +} + +static int rtw_pci_io_mapping(struct rtw_dev *rtwdev, + struct pci_dev *pdev) +{ + struct rtw_pci *rtwpci = (struct rtw_pci *)rtwdev->priv; + unsigned long len; + u8 bar_id = 2; + int ret; + + ret = pci_request_regions(pdev, KBUILD_MODNAME); + if (ret) { + rtw_err(rtwdev, "failed to request pci regions\n"); + return ret; + } + + len = pci_resource_len(pdev, bar_id); + rtwpci->mmap = pci_iomap(pdev, bar_id, len); + if (!rtwpci->mmap) { + rtw_err(rtwdev, "failed to map pci memory\n"); + return -ENOMEM; + } + + return 0; +} + +static void rtw_pci_io_unmapping(struct rtw_dev *rtwdev, + struct pci_dev *pdev) +{ + struct rtw_pci *rtwpci = (struct rtw_pci *)rtwdev->priv; + + if (rtwpci->mmap) { + pci_iounmap(pdev, rtwpci->mmap); + pci_release_regions(pdev); + } +} + +static void rtw_dbi_write8(struct rtw_dev *rtwdev, u16 addr, u8 data) +{ + u16 write_addr; + u16 remainder = addr & 0x3; + u8 flag; + u8 cnt = 20; + + write_addr = ((addr & 0x0ffc) | (BIT(0) << (remainder + 12))); + rtw_write8(rtwdev, REG_DBI_WDATA_V1 + remainder, data); + rtw_write16(rtwdev, REG_DBI_FLAG_V1, write_addr); + rtw_write8(rtwdev, REG_DBI_FLAG_V1 + 2, 0x01); + + flag = rtw_read8(rtwdev, REG_DBI_FLAG_V1 + 2); + while (flag && (cnt != 0)) { + udelay(10); + flag = rtw_read8(rtwdev, REG_DBI_FLAG_V1 + 2); + cnt--; + } + + WARN(flag, "DBI write fail\n"); +} + +static void rtw_mdio_write(struct rtw_dev *rtwdev, u8 addr, u16 data, bool g1) +{ + u8 page; + u8 wflag; + u8 cnt; + + rtw_write16(rtwdev, REG_MDIO_V1, data); + + page = addr < 0x20 ? 0 : 1; + page += g1 ? 0 : 2; + rtw_write8(rtwdev, REG_PCIE_MIX_CFG, addr & 0x1f); + rtw_write8(rtwdev, REG_PCIE_MIX_CFG + 3, page); + + rtw_write32_mask(rtwdev, REG_PCIE_MIX_CFG, BIT_MDIO_WFLAG_V1, 1); + wflag = rtw_read32_mask(rtwdev, REG_PCIE_MIX_CFG, BIT_MDIO_WFLAG_V1); + + cnt = 20; + while (wflag && (cnt != 0)) { + udelay(10); + wflag = rtw_read32_mask(rtwdev, REG_PCIE_MIX_CFG, + BIT_MDIO_WFLAG_V1); + cnt--; + } + + WARN(wflag, "MDIO write fail\n"); +} + +static void rtw_pci_phy_cfg(struct rtw_dev *rtwdev) +{ + struct rtw_chip_info *chip = rtwdev->chip; + struct rtw_intf_phy_para *para; + u16 cut; + u16 value; + u16 offset; + u16 ip_sel; + int i; + + cut = BIT(0) << rtwdev->hal.cut_version; + + for (i = 0; i < chip->intf_table->n_gen1_para; i++) { + para = &chip->intf_table->gen1_para[i]; + if (!(para->cut_mask & cut)) + continue; + if (para->offset == 0xffff) + break; + offset = para->offset; + value = para->value; + ip_sel = para->ip_sel; + if (para->ip_sel == RTW_IP_SEL_PHY) + rtw_mdio_write(rtwdev, offset, value, true); + else + rtw_dbi_write8(rtwdev, offset, value); + } + + for (i = 0; i < chip->intf_table->n_gen2_para; i++) { + para = &chip->intf_table->gen2_para[i]; + if (!(para->cut_mask & cut)) + continue; + if (para->offset == 0xffff) + break; + offset = para->offset; + value = para->value; + ip_sel = para->ip_sel; + if (para->ip_sel == RTW_IP_SEL_PHY) + rtw_mdio_write(rtwdev, offset, value, false); + else + rtw_dbi_write8(rtwdev, offset, value); + } +} + +static int rtw_pci_claim(struct rtw_dev *rtwdev, struct pci_dev *pdev) +{ + int ret; + + ret = pci_enable_device(pdev); + if (ret) { + rtw_err(rtwdev, "failed to enable pci device\n"); + return ret; + } + + pci_set_master(pdev); + pci_set_drvdata(pdev, rtwdev->hw); + SET_IEEE80211_DEV(rtwdev->hw, &pdev->dev); + + return 0; +} + +static void rtw_pci_declaim(struct rtw_dev *rtwdev, struct pci_dev *pdev) +{ + pci_clear_master(pdev); + pci_disable_device(pdev); +} + +static int rtw_pci_setup_resource(struct rtw_dev *rtwdev, struct pci_dev *pdev) +{ + struct rtw_pci *rtwpci; + int ret; + + rtwpci = (struct rtw_pci *)rtwdev->priv; + rtwpci->pdev = pdev; + + /* after this driver can access to hw registers */ + ret = rtw_pci_io_mapping(rtwdev, pdev); + if (ret) { + rtw_err(rtwdev, "failed to request pci io region\n"); + goto err_out; + } + + ret = rtw_pci_init(rtwdev); + if (ret) { + rtw_err(rtwdev, "failed to allocate pci resources\n"); + goto err_io_unmap; + } + + rtw_pci_phy_cfg(rtwdev); + + return 0; + +err_io_unmap: + rtw_pci_io_unmapping(rtwdev, pdev); + +err_out: + return ret; +} + +static void rtw_pci_destroy(struct rtw_dev *rtwdev, struct pci_dev *pdev) +{ + rtw_pci_deinit(rtwdev); + rtw_pci_io_unmapping(rtwdev, pdev); +} + +static struct rtw_hci_ops rtw_pci_ops = { + .tx = rtw_pci_tx, + .setup = rtw_pci_setup, + .start = rtw_pci_start, + .stop = rtw_pci_stop, + + .read8 = rtw_pci_read8, + .read16 = rtw_pci_read16, + .read32 = rtw_pci_read32, + .write8 = rtw_pci_write8, + .write16 = rtw_pci_write16, + .write32 = rtw_pci_write32, + .write_data_rsvd_page = rtw_pci_write_data_rsvd_page, + .write_data_h2c = rtw_pci_write_data_h2c, +}; + +static int rtw_pci_probe(struct pci_dev *pdev, + const struct pci_device_id *id) +{ + struct ieee80211_hw *hw; + struct rtw_dev *rtwdev; + int drv_data_size; + int ret; + + drv_data_size = sizeof(struct rtw_dev) + sizeof(struct rtw_pci); + hw = ieee80211_alloc_hw(drv_data_size, &rtw_ops); + if (!hw) { + dev_err(&pdev->dev, "failed to allocate hw\n"); + return -ENOMEM; + } + + rtwdev = hw->priv; + rtwdev->hw = hw; + rtwdev->dev = &pdev->dev; + rtwdev->chip = (struct rtw_chip_info *)id->driver_data; + rtwdev->hci.ops = &rtw_pci_ops; + rtwdev->hci.type = RTW_HCI_TYPE_PCIE; + + ret = rtw_core_init(rtwdev); + if (ret) + goto err_release_hw; + + rtw_dbg(rtwdev, RTW_DBG_PCI, + "rtw88 pci probe: vendor=0x%4.04X device=0x%4.04X rev=%d\n", + pdev->vendor, pdev->device, pdev->revision); + + ret = rtw_pci_claim(rtwdev, pdev); + if (ret) { + rtw_err(rtwdev, "failed to claim pci device\n"); + goto err_deinit_core; + } + + ret = rtw_pci_setup_resource(rtwdev, pdev); + if (ret) { + rtw_err(rtwdev, "failed to setup pci resources\n"); + goto err_pci_declaim; + } + + ret = rtw_chip_info_setup(rtwdev); + if (ret) { + rtw_err(rtwdev, "failed to setup chip information\n"); + goto err_destroy_pci; + } + + ret = rtw_register_hw(rtwdev, hw); + if (ret) { + rtw_err(rtwdev, "failed to register hw\n"); + goto err_destroy_pci; + } + + ret = request_irq(pdev->irq, &rtw_pci_interrupt_handler, + IRQF_SHARED, KBUILD_MODNAME, rtwdev); + if (ret) { + ieee80211_unregister_hw(hw); + goto err_destroy_pci; + } + + return 0; + +err_destroy_pci: + rtw_pci_destroy(rtwdev, pdev); + +err_pci_declaim: + rtw_pci_declaim(rtwdev, pdev); + +err_deinit_core: + rtw_core_deinit(rtwdev); + +err_release_hw: + ieee80211_free_hw(hw); + + return ret; +} + +static void rtw_pci_remove(struct pci_dev *pdev) +{ + struct ieee80211_hw *hw = pci_get_drvdata(pdev); + struct rtw_dev *rtwdev; + struct rtw_pci *rtwpci; + + if (!hw) + return; + + rtwdev = hw->priv; + rtwpci = (struct rtw_pci *)rtwdev->priv; + + rtw_unregister_hw(rtwdev, hw); + rtw_pci_disable_interrupt(rtwdev, rtwpci); + rtw_pci_destroy(rtwdev, pdev); + rtw_pci_declaim(rtwdev, pdev); + free_irq(rtwpci->pdev->irq, rtwdev); + rtw_core_deinit(rtwdev); + ieee80211_free_hw(hw); +} + +static const struct pci_device_id rtw_pci_id_table[] = { +#ifdef CONFIG_RTW88_8822BE + { RTK_PCI_DEVICE(PCI_VENDOR_ID_REALTEK, 0xB822, rtw8822b_hw_spec) }, +#endif +#ifdef CONFIG_RTW88_8822CE + { RTK_PCI_DEVICE(PCI_VENDOR_ID_REALTEK, 0xC822, rtw8822c_hw_spec) }, +#endif + {}, +}; +MODULE_DEVICE_TABLE(pci, rtw_pci_id_table); + +static struct pci_driver rtw_pci_driver = { + .name = "rtw_pci", + .id_table = rtw_pci_id_table, + .probe = rtw_pci_probe, + .remove = rtw_pci_remove, +}; +module_pci_driver(rtw_pci_driver); + +MODULE_AUTHOR("Realtek Corporation"); +MODULE_DESCRIPTION("Realtek 802.11ac wireless PCI driver"); +MODULE_LICENSE("Dual BSD/GPL"); diff --git a/drivers/net/wireless/realtek/rtw88/pci.h b/drivers/net/wireless/realtek/rtw88/pci.h new file mode 100644 index 0000000000000000000000000000000000000000..87824a4caba98d7aade7f4fe2cf93764ef4f1328 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/pci.h @@ -0,0 +1,237 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#ifndef __RTK_PCI_H_ +#define __RTK_PCI_H_ + +#define RTK_PCI_DEVICE(vend, dev, hw_config) \ + PCI_DEVICE(vend, dev), \ + .driver_data = (kernel_ulong_t)&(hw_config), + +#define RTK_DEFAULT_TX_DESC_NUM 128 +#define RTK_BEQ_TX_DESC_NUM 256 + +#define RTK_MAX_RX_DESC_NUM 512 +/* 8K + rx desc size */ +#define RTK_PCI_RX_BUF_SIZE (8192 + 24) + +#define RTK_PCI_CTRL 0x300 +#define BIT_RST_TRXDMA_INTF BIT(20) +#define BIT_RX_TAG_EN BIT(15) +#define REG_DBI_WDATA_V1 0x03E8 +#define REG_DBI_FLAG_V1 0x03F0 +#define REG_MDIO_V1 0x03F4 +#define REG_PCIE_MIX_CFG 0x03F8 +#define BIT_MDIO_WFLAG_V1 BIT(5) + +#define BIT_PCI_BCNQ_FLAG BIT(4) +#define RTK_PCI_TXBD_DESA_BCNQ 0x308 +#define RTK_PCI_TXBD_DESA_H2CQ 0x1320 +#define RTK_PCI_TXBD_DESA_MGMTQ 0x310 +#define RTK_PCI_TXBD_DESA_BKQ 0x330 +#define RTK_PCI_TXBD_DESA_BEQ 0x328 +#define RTK_PCI_TXBD_DESA_VIQ 0x320 +#define RTK_PCI_TXBD_DESA_VOQ 0x318 +#define RTK_PCI_TXBD_DESA_HI0Q 0x340 +#define RTK_PCI_RXBD_DESA_MPDUQ 0x338 + +/* BCNQ is specialized for rsvd page, does not need to specify a number */ +#define RTK_PCI_TXBD_NUM_H2CQ 0x1328 +#define RTK_PCI_TXBD_NUM_MGMTQ 0x380 +#define RTK_PCI_TXBD_NUM_BKQ 0x38A +#define RTK_PCI_TXBD_NUM_BEQ 0x388 +#define RTK_PCI_TXBD_NUM_VIQ 0x386 +#define RTK_PCI_TXBD_NUM_VOQ 0x384 +#define RTK_PCI_TXBD_NUM_HI0Q 0x38C +#define RTK_PCI_RXBD_NUM_MPDUQ 0x382 +#define RTK_PCI_TXBD_IDX_H2CQ 0x132C +#define RTK_PCI_TXBD_IDX_MGMTQ 0x3B0 +#define RTK_PCI_TXBD_IDX_BKQ 0x3AC +#define RTK_PCI_TXBD_IDX_BEQ 0x3A8 +#define RTK_PCI_TXBD_IDX_VIQ 0x3A4 +#define RTK_PCI_TXBD_IDX_VOQ 0x3A0 +#define RTK_PCI_TXBD_IDX_HI0Q 0x3B8 +#define RTK_PCI_RXBD_IDX_MPDUQ 0x3B4 + +#define RTK_PCI_TXBD_RWPTR_CLR 0x39C +#define RTK_PCI_TXBD_H2CQ_CSR 0x1330 + +#define BIT_CLR_H2CQ_HOST_IDX BIT(16) +#define BIT_CLR_H2CQ_HW_IDX BIT(8) + +#define RTK_PCI_HIMR0 0x0B0 +#define RTK_PCI_HISR0 0x0B4 +#define RTK_PCI_HIMR1 0x0B8 +#define RTK_PCI_HISR1 0x0BC +#define RTK_PCI_HIMR2 0x10B0 +#define RTK_PCI_HISR2 0x10B4 +#define RTK_PCI_HIMR3 0x10B8 +#define RTK_PCI_HISR3 0x10BC +/* IMR 0 */ +#define IMR_TIMER2 BIT(31) +#define IMR_TIMER1 BIT(30) +#define IMR_PSTIMEOUT BIT(29) +#define IMR_GTINT4 BIT(28) +#define IMR_GTINT3 BIT(27) +#define IMR_TBDER BIT(26) +#define IMR_TBDOK BIT(25) +#define IMR_TSF_BIT32_TOGGLE BIT(24) +#define IMR_BCNDMAINT0 BIT(20) +#define IMR_BCNDOK0 BIT(16) +#define IMR_HSISR_IND_ON_INT BIT(15) +#define IMR_BCNDMAINT_E BIT(14) +#define IMR_ATIMEND BIT(12) +#define IMR_HISR1_IND_INT BIT(11) +#define IMR_C2HCMD BIT(10) +#define IMR_CPWM2 BIT(9) +#define IMR_CPWM BIT(8) +#define IMR_HIGHDOK BIT(7) +#define IMR_MGNTDOK BIT(6) +#define IMR_BKDOK BIT(5) +#define IMR_BEDOK BIT(4) +#define IMR_VIDOK BIT(3) +#define IMR_VODOK BIT(2) +#define IMR_RDU BIT(1) +#define IMR_ROK BIT(0) +/* IMR 1 */ +#define IMR_TXFIFO_TH_INT BIT(30) +#define IMR_BTON_STS_UPDATE BIT(29) +#define IMR_MCUERR BIT(28) +#define IMR_BCNDMAINT7 BIT(27) +#define IMR_BCNDMAINT6 BIT(26) +#define IMR_BCNDMAINT5 BIT(25) +#define IMR_BCNDMAINT4 BIT(24) +#define IMR_BCNDMAINT3 BIT(23) +#define IMR_BCNDMAINT2 BIT(22) +#define IMR_BCNDMAINT1 BIT(21) +#define IMR_BCNDOK7 BIT(20) +#define IMR_BCNDOK6 BIT(19) +#define IMR_BCNDOK5 BIT(18) +#define IMR_BCNDOK4 BIT(17) +#define IMR_BCNDOK3 BIT(16) +#define IMR_BCNDOK2 BIT(15) +#define IMR_BCNDOK1 BIT(14) +#define IMR_ATIMEND_E BIT(13) +#define IMR_ATIMEND BIT(12) +#define IMR_TXERR BIT(11) +#define IMR_RXERR BIT(10) +#define IMR_TXFOVW BIT(9) +#define IMR_RXFOVW BIT(8) +#define IMR_CPU_MGQ_TXDONE BIT(5) +#define IMR_PS_TIMER_C BIT(4) +#define IMR_PS_TIMER_B BIT(3) +#define IMR_PS_TIMER_A BIT(2) +#define IMR_CPUMGQ_TX_TIMER BIT(1) +/* IMR 3 */ +#define IMR_H2CDOK BIT(16) + +/* one element is reserved to know if the ring is closed */ +static inline int avail_desc(u32 wp, u32 rp, u32 len) +{ + if (rp > wp) + return rp - wp - 1; + else + return len - wp + rp - 1; +} + +#define RTK_PCI_TXBD_OWN_OFFSET 15 +#define RTK_PCI_TXBD_BCN_WORK 0x383 + +struct rtw_pci_tx_buffer_desc { + __le16 buf_size; + __le16 psb_len; + __le32 dma; +}; + +struct rtw_pci_tx_data { + dma_addr_t dma; + u8 sn; +}; + +struct rtw_pci_ring { + u8 *head; + dma_addr_t dma; + + u8 desc_size; + + u32 len; + u32 wp; + u32 rp; +}; + +struct rtw_pci_tx_ring { + struct rtw_pci_ring r; + struct sk_buff_head queue; + bool queue_stopped; +}; + +struct rtw_pci_rx_buffer_desc { + __le16 buf_size; + __le16 total_pkt_size; + __le32 dma; +}; + +struct rtw_pci_rx_ring { + struct rtw_pci_ring r; + struct sk_buff *buf[RTK_MAX_RX_DESC_NUM]; +}; + +#define RX_TAG_MAX 8192 + +struct rtw_pci { + struct pci_dev *pdev; + + /* used for pci interrupt */ + spinlock_t irq_lock; + u32 irq_mask[4]; + bool irq_enabled; + + u16 rx_tag; + struct rtw_pci_tx_ring tx_rings[RTK_MAX_TX_QUEUE_NUM]; + struct rtw_pci_rx_ring rx_rings[RTK_MAX_RX_QUEUE_NUM]; + + void __iomem *mmap; +}; + +static u32 max_num_of_tx_queue(u8 queue) +{ + u32 max_num; + + switch (queue) { + case RTW_TX_QUEUE_BE: + max_num = RTK_BEQ_TX_DESC_NUM; + break; + case RTW_TX_QUEUE_BCN: + max_num = 1; + break; + default: + max_num = RTK_DEFAULT_TX_DESC_NUM; + break; + } + + return max_num; +} + +static inline struct +rtw_pci_tx_data *rtw_pci_get_tx_data(struct sk_buff *skb) +{ + struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb); + + BUILD_BUG_ON(sizeof(struct rtw_pci_tx_data) > + sizeof(info->status.status_driver_data)); + + return (struct rtw_pci_tx_data *)info->status.status_driver_data; +} + +static inline +struct rtw_pci_tx_buffer_desc *get_tx_buffer_desc(struct rtw_pci_tx_ring *ring, + u32 size) +{ + u8 *buf_desc; + + buf_desc = ring->r.head + ring->r.wp * size; + return (struct rtw_pci_tx_buffer_desc *)buf_desc; +} + +#endif diff --git a/drivers/net/wireless/realtek/rtw88/phy.c b/drivers/net/wireless/realtek/rtw88/phy.c new file mode 100644 index 0000000000000000000000000000000000000000..4381b360b5b5358cc49b93b1aebe6cbaf304f4dd --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/phy.c @@ -0,0 +1,1727 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#include + +#include "main.h" +#include "reg.h" +#include "fw.h" +#include "phy.h" +#include "debug.h" + +struct phy_cfg_pair { + u32 addr; + u32 data; +}; + +union phy_table_tile { + struct rtw_phy_cond cond; + struct phy_cfg_pair cfg; +}; + +struct phy_pg_cfg_pair { + u32 band; + u32 rf_path; + u32 tx_num; + u32 addr; + u32 bitmask; + u32 data; +}; + +struct txpwr_lmt_cfg_pair { + u8 regd; + u8 band; + u8 bw; + u8 rs; + u8 ch; + s8 txpwr_lmt; +}; + +static const u32 db_invert_table[12][8] = { + {10, 13, 16, 20, + 25, 32, 40, 50}, + {64, 80, 101, 128, + 160, 201, 256, 318}, + {401, 505, 635, 800, + 1007, 1268, 1596, 2010}, + {316, 398, 501, 631, + 794, 1000, 1259, 1585}, + {1995, 2512, 3162, 3981, + 5012, 6310, 7943, 10000}, + {12589, 15849, 19953, 25119, + 31623, 39811, 50119, 63098}, + {79433, 100000, 125893, 158489, + 199526, 251189, 316228, 398107}, + {501187, 630957, 794328, 1000000, + 1258925, 1584893, 1995262, 2511886}, + {3162278, 3981072, 5011872, 6309573, + 7943282, 1000000, 12589254, 15848932}, + {19952623, 25118864, 31622777, 39810717, + 50118723, 63095734, 79432823, 100000000}, + {125892541, 158489319, 199526232, 251188643, + 316227766, 398107171, 501187234, 630957345}, + {794328235, 1000000000, 1258925412, 1584893192, + 1995262315, 2511886432U, 3162277660U, 3981071706U} +}; + +enum rtw_phy_band_type { + PHY_BAND_2G = 0, + PHY_BAND_5G = 1, +}; + +void rtw_phy_init(struct rtw_dev *rtwdev) +{ + struct rtw_chip_info *chip = rtwdev->chip; + struct rtw_dm_info *dm_info = &rtwdev->dm_info; + u32 addr, mask; + + dm_info->fa_history[3] = 0; + dm_info->fa_history[2] = 0; + dm_info->fa_history[1] = 0; + dm_info->fa_history[0] = 0; + dm_info->igi_bitmap = 0; + dm_info->igi_history[3] = 0; + dm_info->igi_history[2] = 0; + dm_info->igi_history[1] = 0; + + addr = chip->dig[0].addr; + mask = chip->dig[0].mask; + dm_info->igi_history[0] = rtw_read32_mask(rtwdev, addr, mask); +} + +void rtw_phy_dig_write(struct rtw_dev *rtwdev, u8 igi) +{ + struct rtw_chip_info *chip = rtwdev->chip; + struct rtw_hal *hal = &rtwdev->hal; + u32 addr, mask; + u8 path; + + for (path = 0; path < hal->rf_path_num; path++) { + addr = chip->dig[path].addr; + mask = chip->dig[path].mask; + rtw_write32_mask(rtwdev, addr, mask, igi); + } +} + +static void rtw_phy_stat_false_alarm(struct rtw_dev *rtwdev) +{ + struct rtw_chip_info *chip = rtwdev->chip; + + chip->ops->false_alarm_statistics(rtwdev); +} + +#define RA_FLOOR_TABLE_SIZE 7 +#define RA_FLOOR_UP_GAP 3 + +static u8 rtw_phy_get_rssi_level(u8 old_level, u8 rssi) +{ + u8 table[RA_FLOOR_TABLE_SIZE] = {20, 34, 38, 42, 46, 50, 100}; + u8 new_level = 0; + int i; + + for (i = 0; i < RA_FLOOR_TABLE_SIZE; i++) + if (i >= old_level) + table[i] += RA_FLOOR_UP_GAP; + + for (i = 0; i < RA_FLOOR_TABLE_SIZE; i++) { + if (rssi < table[i]) { + new_level = i; + break; + } + } + + return new_level; +} + +struct rtw_phy_stat_iter_data { + struct rtw_dev *rtwdev; + u8 min_rssi; +}; + +static void rtw_phy_stat_rssi_iter(void *data, struct ieee80211_sta *sta) +{ + struct rtw_phy_stat_iter_data *iter_data = data; + struct rtw_dev *rtwdev = iter_data->rtwdev; + struct rtw_sta_info *si = (struct rtw_sta_info *)sta->drv_priv; + u8 rssi, rssi_level; + + rssi = ewma_rssi_read(&si->avg_rssi); + rssi_level = rtw_phy_get_rssi_level(si->rssi_level, rssi); + + rtw_fw_send_rssi_info(rtwdev, si); + + iter_data->min_rssi = min_t(u8, rssi, iter_data->min_rssi); +} + +static void rtw_phy_stat_rssi(struct rtw_dev *rtwdev) +{ + struct rtw_dm_info *dm_info = &rtwdev->dm_info; + struct rtw_phy_stat_iter_data data = {}; + + data.rtwdev = rtwdev; + data.min_rssi = U8_MAX; + rtw_iterate_stas_atomic(rtwdev, rtw_phy_stat_rssi_iter, &data); + + dm_info->pre_min_rssi = dm_info->min_rssi; + dm_info->min_rssi = data.min_rssi; +} + +static void rtw_phy_statistics(struct rtw_dev *rtwdev) +{ + rtw_phy_stat_rssi(rtwdev); + rtw_phy_stat_false_alarm(rtwdev); +} + +#define DIG_PERF_FA_TH_LOW 250 +#define DIG_PERF_FA_TH_HIGH 500 +#define DIG_PERF_FA_TH_EXTRA_HIGH 750 +#define DIG_PERF_MAX 0x5a +#define DIG_PERF_MID 0x40 +#define DIG_CVRG_FA_TH_LOW 2000 +#define DIG_CVRG_FA_TH_HIGH 4000 +#define DIG_CVRG_FA_TH_EXTRA_HIGH 5000 +#define DIG_CVRG_MAX 0x2a +#define DIG_CVRG_MID 0x26 +#define DIG_CVRG_MIN 0x1c +#define DIG_RSSI_GAIN_OFFSET 15 + +static bool +rtw_phy_dig_check_damping(struct rtw_dm_info *dm_info) +{ + u16 fa_lo = DIG_PERF_FA_TH_LOW; + u16 fa_hi = DIG_PERF_FA_TH_HIGH; + u16 *fa_history; + u8 *igi_history; + u8 damping_rssi; + u8 min_rssi; + u8 diff; + u8 igi_bitmap; + bool damping = false; + + min_rssi = dm_info->min_rssi; + if (dm_info->damping) { + damping_rssi = dm_info->damping_rssi; + diff = min_rssi > damping_rssi ? min_rssi - damping_rssi : + damping_rssi - min_rssi; + if (diff > 3 || dm_info->damping_cnt++ > 20) { + dm_info->damping = false; + return false; + } + + return true; + } + + igi_history = dm_info->igi_history; + fa_history = dm_info->fa_history; + igi_bitmap = dm_info->igi_bitmap & 0xf; + switch (igi_bitmap) { + case 5: + /* down -> up -> down -> up */ + if (igi_history[0] > igi_history[1] && + igi_history[2] > igi_history[3] && + igi_history[0] - igi_history[1] >= 2 && + igi_history[2] - igi_history[3] >= 2 && + fa_history[0] > fa_hi && fa_history[1] < fa_lo && + fa_history[2] > fa_hi && fa_history[3] < fa_lo) + damping = true; + break; + case 9: + /* up -> down -> down -> up */ + if (igi_history[0] > igi_history[1] && + igi_history[3] > igi_history[2] && + igi_history[0] - igi_history[1] >= 4 && + igi_history[3] - igi_history[2] >= 2 && + fa_history[0] > fa_hi && fa_history[1] < fa_lo && + fa_history[2] < fa_lo && fa_history[3] > fa_hi) + damping = true; + break; + default: + return false; + } + + if (damping) { + dm_info->damping = true; + dm_info->damping_cnt = 0; + dm_info->damping_rssi = min_rssi; + } + + return damping; +} + +static void rtw_phy_dig_get_boundary(struct rtw_dm_info *dm_info, + u8 *upper, u8 *lower, bool linked) +{ + u8 dig_max, dig_min, dig_mid; + u8 min_rssi; + + if (linked) { + dig_max = DIG_PERF_MAX; + dig_mid = DIG_PERF_MID; + /* 22B=0x1c, 22C=0x20 */ + dig_min = 0x1c; + min_rssi = max_t(u8, dm_info->min_rssi, dig_min); + } else { + dig_max = DIG_CVRG_MAX; + dig_mid = DIG_CVRG_MID; + dig_min = DIG_CVRG_MIN; + min_rssi = dig_min; + } + + /* DIG MAX should be bounded by minimum RSSI with offset +15 */ + dig_max = min_t(u8, dig_max, min_rssi + DIG_RSSI_GAIN_OFFSET); + + *lower = clamp_t(u8, min_rssi, dig_min, dig_mid); + *upper = clamp_t(u8, *lower + DIG_RSSI_GAIN_OFFSET, dig_min, dig_max); +} + +static void rtw_phy_dig_get_threshold(struct rtw_dm_info *dm_info, + u16 *fa_th, u8 *step, bool linked) +{ + u8 min_rssi, pre_min_rssi; + + min_rssi = dm_info->min_rssi; + pre_min_rssi = dm_info->pre_min_rssi; + step[0] = 4; + step[1] = 3; + step[2] = 2; + + if (linked) { + fa_th[0] = DIG_PERF_FA_TH_EXTRA_HIGH; + fa_th[1] = DIG_PERF_FA_TH_HIGH; + fa_th[2] = DIG_PERF_FA_TH_LOW; + if (pre_min_rssi > min_rssi) { + step[0] = 6; + step[1] = 4; + step[2] = 2; + } + } else { + fa_th[0] = DIG_CVRG_FA_TH_EXTRA_HIGH; + fa_th[1] = DIG_CVRG_FA_TH_HIGH; + fa_th[2] = DIG_CVRG_FA_TH_LOW; + } +} + +static void rtw_phy_dig_recorder(struct rtw_dm_info *dm_info, u8 igi, u16 fa) +{ + u8 *igi_history; + u16 *fa_history; + u8 igi_bitmap; + bool up; + + igi_bitmap = dm_info->igi_bitmap << 1 & 0xfe; + igi_history = dm_info->igi_history; + fa_history = dm_info->fa_history; + + up = igi > igi_history[0]; + igi_bitmap |= up; + + igi_history[3] = igi_history[2]; + igi_history[2] = igi_history[1]; + igi_history[1] = igi_history[0]; + igi_history[0] = igi; + + fa_history[3] = fa_history[2]; + fa_history[2] = fa_history[1]; + fa_history[1] = fa_history[0]; + fa_history[0] = fa; + + dm_info->igi_bitmap = igi_bitmap; +} + +static void rtw_phy_dig(struct rtw_dev *rtwdev) +{ + struct rtw_dm_info *dm_info = &rtwdev->dm_info; + u8 upper_bound, lower_bound; + u8 pre_igi, cur_igi; + u16 fa_th[3], fa_cnt; + u8 level; + u8 step[3]; + bool linked; + + if (rtw_flag_check(rtwdev, RTW_FLAG_DIG_DISABLE)) + return; + + if (rtw_phy_dig_check_damping(dm_info)) + return; + + linked = !!rtwdev->sta_cnt; + + fa_cnt = dm_info->total_fa_cnt; + pre_igi = dm_info->igi_history[0]; + + rtw_phy_dig_get_threshold(dm_info, fa_th, step, linked); + + /* test the false alarm count from the highest threshold level first, + * and increase it by corresponding step size + * + * note that the step size is offset by -2, compensate it afterall + */ + cur_igi = pre_igi; + for (level = 0; level < 3; level++) { + if (fa_cnt > fa_th[level]) { + cur_igi += step[level]; + break; + } + } + cur_igi -= 2; + + /* calculate the upper/lower bound by the minimum rssi we have among + * the peers connected with us, meanwhile make sure the igi value does + * not beyond the hardware limitation + */ + rtw_phy_dig_get_boundary(dm_info, &upper_bound, &lower_bound, linked); + cur_igi = clamp_t(u8, cur_igi, lower_bound, upper_bound); + + /* record current igi value and false alarm statistics for further + * damping checks, and record the trend of igi values + */ + rtw_phy_dig_recorder(dm_info, cur_igi, fa_cnt); + + if (cur_igi != pre_igi) + rtw_phy_dig_write(rtwdev, cur_igi); +} + +static void rtw_phy_ra_info_update_iter(void *data, struct ieee80211_sta *sta) +{ + struct rtw_dev *rtwdev = data; + struct rtw_sta_info *si = (struct rtw_sta_info *)sta->drv_priv; + + rtw_update_sta_info(rtwdev, si); +} + +static void rtw_phy_ra_info_update(struct rtw_dev *rtwdev) +{ + if (rtwdev->watch_dog_cnt & 0x3) + return; + + rtw_iterate_stas_atomic(rtwdev, rtw_phy_ra_info_update_iter, rtwdev); +} + +void rtw_phy_dynamic_mechanism(struct rtw_dev *rtwdev) +{ + /* for further calculation */ + rtw_phy_statistics(rtwdev); + rtw_phy_dig(rtwdev); + rtw_phy_ra_info_update(rtwdev); +} + +#define FRAC_BITS 3 + +static u8 rtw_phy_power_2_db(s8 power) +{ + if (power <= -100 || power >= 20) + return 0; + else if (power >= 0) + return 100; + else + return 100 + power; +} + +static u64 rtw_phy_db_2_linear(u8 power_db) +{ + u8 i, j; + u64 linear; + + /* 1dB ~ 96dB */ + i = (power_db - 1) >> 3; + j = (power_db - 1) - (i << 3); + + linear = db_invert_table[i][j]; + linear = i > 2 ? linear << FRAC_BITS : linear; + + return linear; +} + +static u8 rtw_phy_linear_2_db(u64 linear) +{ + u8 i; + u8 j; + u32 dB; + + if (linear >= db_invert_table[11][7]) + return 96; /* maximum 96 dB */ + + for (i = 0; i < 12; i++) { + if (i <= 2 && (linear << FRAC_BITS) <= db_invert_table[i][7]) + break; + else if (i > 2 && linear <= db_invert_table[i][7]) + break; + } + + for (j = 0; j < 8; j++) { + if (i <= 2 && (linear << FRAC_BITS) <= db_invert_table[i][j]) + break; + else if (i > 2 && linear <= db_invert_table[i][j]) + break; + } + + if (j == 0 && i == 0) + goto end; + + if (j == 0) { + if (i != 3) { + if (db_invert_table[i][0] - linear > + linear - db_invert_table[i - 1][7]) { + i = i - 1; + j = 7; + } + } else { + if (db_invert_table[3][0] - linear > + linear - db_invert_table[2][7]) { + i = 2; + j = 7; + } + } + } else { + if (db_invert_table[i][j] - linear > + linear - db_invert_table[i][j - 1]) { + j = j - 1; + } + } +end: + dB = (i << 3) + j + 1; + + return dB; +} + +u8 rtw_phy_rf_power_2_rssi(s8 *rf_power, u8 path_num) +{ + s8 power; + u8 power_db; + u64 linear; + u64 sum = 0; + u8 path; + + for (path = 0; path < path_num; path++) { + power = rf_power[path]; + power_db = rtw_phy_power_2_db(power); + linear = rtw_phy_db_2_linear(power_db); + sum += linear; + } + + sum = (sum + (1 << (FRAC_BITS - 1))) >> FRAC_BITS; + switch (path_num) { + case 2: + sum >>= 1; + break; + case 3: + sum = ((sum) + ((sum) << 1) + ((sum) << 3)) >> 5; + break; + case 4: + sum >>= 2; + break; + default: + break; + } + + return rtw_phy_linear_2_db(sum); +} + +u32 rtw_phy_read_rf(struct rtw_dev *rtwdev, enum rtw_rf_path rf_path, + u32 addr, u32 mask) +{ + struct rtw_hal *hal = &rtwdev->hal; + struct rtw_chip_info *chip = rtwdev->chip; + const u32 *base_addr = chip->rf_base_addr; + u32 val, direct_addr; + + if (rf_path >= hal->rf_path_num) { + rtw_err(rtwdev, "unsupported rf path (%d)\n", rf_path); + return INV_RF_DATA; + } + + addr &= 0xff; + direct_addr = base_addr[rf_path] + (addr << 2); + mask &= RFREG_MASK; + + val = rtw_read32_mask(rtwdev, direct_addr, mask); + + return val; +} + +bool rtw_phy_write_rf_reg_sipi(struct rtw_dev *rtwdev, enum rtw_rf_path rf_path, + u32 addr, u32 mask, u32 data) +{ + struct rtw_hal *hal = &rtwdev->hal; + struct rtw_chip_info *chip = rtwdev->chip; + u32 *sipi_addr = chip->rf_sipi_addr; + u32 data_and_addr; + u32 old_data = 0; + u32 shift; + + if (rf_path >= hal->rf_path_num) { + rtw_err(rtwdev, "unsupported rf path (%d)\n", rf_path); + return false; + } + + addr &= 0xff; + mask &= RFREG_MASK; + + if (mask != RFREG_MASK) { + old_data = rtw_phy_read_rf(rtwdev, rf_path, addr, RFREG_MASK); + + if (old_data == INV_RF_DATA) { + rtw_err(rtwdev, "Write fail, rf is disabled\n"); + return false; + } + + shift = __ffs(mask); + data = ((old_data) & (~mask)) | (data << shift); + } + + data_and_addr = ((addr << 20) | (data & 0x000fffff)) & 0x0fffffff; + + rtw_write32(rtwdev, sipi_addr[rf_path], data_and_addr); + + udelay(13); + + return true; +} + +bool rtw_phy_write_rf_reg(struct rtw_dev *rtwdev, enum rtw_rf_path rf_path, + u32 addr, u32 mask, u32 data) +{ + struct rtw_hal *hal = &rtwdev->hal; + struct rtw_chip_info *chip = rtwdev->chip; + const u32 *base_addr = chip->rf_base_addr; + u32 direct_addr; + + if (rf_path >= hal->rf_path_num) { + rtw_err(rtwdev, "unsupported rf path (%d)\n", rf_path); + return false; + } + + addr &= 0xff; + direct_addr = base_addr[rf_path] + (addr << 2); + mask &= RFREG_MASK; + + rtw_write32_mask(rtwdev, REG_RSV_CTRL, BITS_RFC_DIRECT, DISABLE_PI); + rtw_write32_mask(rtwdev, REG_WLRF1, BITS_RFC_DIRECT, DISABLE_PI); + rtw_write32_mask(rtwdev, direct_addr, mask, data); + + udelay(1); + + rtw_write32_mask(rtwdev, REG_RSV_CTRL, BITS_RFC_DIRECT, ENABLE_PI); + rtw_write32_mask(rtwdev, REG_WLRF1, BITS_RFC_DIRECT, ENABLE_PI); + + return true; +} + +bool rtw_phy_write_rf_reg_mix(struct rtw_dev *rtwdev, enum rtw_rf_path rf_path, + u32 addr, u32 mask, u32 data) +{ + if (addr != 0x00) + return rtw_phy_write_rf_reg(rtwdev, rf_path, addr, mask, data); + + return rtw_phy_write_rf_reg_sipi(rtwdev, rf_path, addr, mask, data); +} + +void rtw_phy_setup_phy_cond(struct rtw_dev *rtwdev, u32 pkg) +{ + struct rtw_hal *hal = &rtwdev->hal; + struct rtw_efuse *efuse = &rtwdev->efuse; + struct rtw_phy_cond cond = {0}; + + cond.cut = hal->cut_version ? hal->cut_version : 15; + cond.pkg = pkg ? pkg : 15; + cond.plat = 0x04; + cond.rfe = efuse->rfe_option; + + switch (rtw_hci_type(rtwdev)) { + case RTW_HCI_TYPE_USB: + cond.intf = INTF_USB; + break; + case RTW_HCI_TYPE_SDIO: + cond.intf = INTF_SDIO; + break; + case RTW_HCI_TYPE_PCIE: + default: + cond.intf = INTF_PCIE; + break; + } + + hal->phy_cond = cond; + + rtw_dbg(rtwdev, RTW_DBG_PHY, "phy cond=0x%08x\n", *((u32 *)&hal->phy_cond)); +} + +static bool check_positive(struct rtw_dev *rtwdev, struct rtw_phy_cond cond) +{ + struct rtw_hal *hal = &rtwdev->hal; + struct rtw_phy_cond drv_cond = hal->phy_cond; + + if (cond.cut && cond.cut != drv_cond.cut) + return false; + + if (cond.pkg && cond.pkg != drv_cond.pkg) + return false; + + if (cond.intf && cond.intf != drv_cond.intf) + return false; + + if (cond.rfe != drv_cond.rfe) + return false; + + return true; +} + +void rtw_parse_tbl_phy_cond(struct rtw_dev *rtwdev, const struct rtw_table *tbl) +{ + const union phy_table_tile *p = tbl->data; + const union phy_table_tile *end = p + tbl->size / 2; + struct rtw_phy_cond pos_cond = {0}; + bool is_matched = true, is_skipped = false; + + BUILD_BUG_ON(sizeof(union phy_table_tile) != sizeof(struct phy_cfg_pair)); + + for (; p < end; p++) { + if (p->cond.pos) { + switch (p->cond.branch) { + case BRANCH_ENDIF: + is_matched = true; + is_skipped = false; + break; + case BRANCH_ELSE: + is_matched = is_skipped ? false : true; + break; + case BRANCH_IF: + case BRANCH_ELIF: + default: + pos_cond = p->cond; + break; + } + } else if (p->cond.neg) { + if (!is_skipped) { + if (check_positive(rtwdev, pos_cond)) { + is_matched = true; + is_skipped = true; + } else { + is_matched = false; + is_skipped = false; + } + } else { + is_matched = false; + } + } else if (is_matched) { + (*tbl->do_cfg)(rtwdev, tbl, p->cfg.addr, p->cfg.data); + } + } +} + +void rtw_parse_tbl_bb_pg(struct rtw_dev *rtwdev, const struct rtw_table *tbl) +{ + const struct phy_pg_cfg_pair *p = tbl->data; + const struct phy_pg_cfg_pair *end = p + tbl->size / 6; + + BUILD_BUG_ON(sizeof(struct phy_pg_cfg_pair) != sizeof(u32) * 6); + + for (; p < end; p++) { + if (p->addr == 0xfe || p->addr == 0xffe) { + msleep(50); + continue; + } + phy_store_tx_power_by_rate(rtwdev, p->band, p->rf_path, + p->tx_num, p->addr, p->bitmask, + p->data); + } +} + +void rtw_parse_tbl_txpwr_lmt(struct rtw_dev *rtwdev, + const struct rtw_table *tbl) +{ + const struct txpwr_lmt_cfg_pair *p = tbl->data; + const struct txpwr_lmt_cfg_pair *end = p + tbl->size / 6; + + BUILD_BUG_ON(sizeof(struct txpwr_lmt_cfg_pair) != sizeof(u8) * 6); + + for (; p < end; p++) { + phy_set_tx_power_limit(rtwdev, p->regd, p->band, + p->bw, p->rs, + p->ch, p->txpwr_lmt); + } +} + +void rtw_phy_cfg_mac(struct rtw_dev *rtwdev, const struct rtw_table *tbl, + u32 addr, u32 data) +{ + rtw_write8(rtwdev, addr, data); +} + +void rtw_phy_cfg_agc(struct rtw_dev *rtwdev, const struct rtw_table *tbl, + u32 addr, u32 data) +{ + rtw_write32(rtwdev, addr, data); +} + +void rtw_phy_cfg_bb(struct rtw_dev *rtwdev, const struct rtw_table *tbl, + u32 addr, u32 data) +{ + if (addr == 0xfe) + msleep(50); + else if (addr == 0xfd) + mdelay(5); + else if (addr == 0xfc) + mdelay(1); + else if (addr == 0xfb) + usleep_range(50, 60); + else if (addr == 0xfa) + udelay(5); + else if (addr == 0xf9) + udelay(1); + else + rtw_write32(rtwdev, addr, data); +} + +void rtw_phy_cfg_rf(struct rtw_dev *rtwdev, const struct rtw_table *tbl, + u32 addr, u32 data) +{ + if (addr == 0xffe) { + msleep(50); + } else if (addr == 0xfe) { + usleep_range(100, 110); + } else { + rtw_write_rf(rtwdev, tbl->rf_path, addr, RFREG_MASK, data); + udelay(1); + } +} + +static void rtw_load_rfk_table(struct rtw_dev *rtwdev) +{ + struct rtw_chip_info *chip = rtwdev->chip; + + if (!chip->rfk_init_tbl) + return; + + rtw_load_table(rtwdev, chip->rfk_init_tbl); +} + +void rtw_phy_load_tables(struct rtw_dev *rtwdev) +{ + struct rtw_chip_info *chip = rtwdev->chip; + u8 rf_path; + + rtw_load_table(rtwdev, chip->mac_tbl); + rtw_load_table(rtwdev, chip->bb_tbl); + rtw_load_table(rtwdev, chip->agc_tbl); + rtw_load_rfk_table(rtwdev); + + for (rf_path = 0; rf_path < rtwdev->hal.rf_path_num; rf_path++) { + const struct rtw_table *tbl; + + tbl = chip->rf_tbl[rf_path]; + rtw_load_table(rtwdev, tbl); + } +} + +#define bcd_to_dec_pwr_by_rate(val, i) bcd2bin(val >> (i * 8)) + +#define RTW_MAX_POWER_INDEX 0x3F + +u8 rtw_cck_rates[] = { DESC_RATE1M, DESC_RATE2M, DESC_RATE5_5M, DESC_RATE11M }; +u8 rtw_ofdm_rates[] = { + DESC_RATE6M, DESC_RATE9M, DESC_RATE12M, + DESC_RATE18M, DESC_RATE24M, DESC_RATE36M, + DESC_RATE48M, DESC_RATE54M +}; +u8 rtw_ht_1s_rates[] = { + DESC_RATEMCS0, DESC_RATEMCS1, DESC_RATEMCS2, + DESC_RATEMCS3, DESC_RATEMCS4, DESC_RATEMCS5, + DESC_RATEMCS6, DESC_RATEMCS7 +}; +u8 rtw_ht_2s_rates[] = { + DESC_RATEMCS8, DESC_RATEMCS9, DESC_RATEMCS10, + DESC_RATEMCS11, DESC_RATEMCS12, DESC_RATEMCS13, + DESC_RATEMCS14, DESC_RATEMCS15 +}; +u8 rtw_vht_1s_rates[] = { + DESC_RATEVHT1SS_MCS0, DESC_RATEVHT1SS_MCS1, + DESC_RATEVHT1SS_MCS2, DESC_RATEVHT1SS_MCS3, + DESC_RATEVHT1SS_MCS4, DESC_RATEVHT1SS_MCS5, + DESC_RATEVHT1SS_MCS6, DESC_RATEVHT1SS_MCS7, + DESC_RATEVHT1SS_MCS8, DESC_RATEVHT1SS_MCS9 +}; +u8 rtw_vht_2s_rates[] = { + DESC_RATEVHT2SS_MCS0, DESC_RATEVHT2SS_MCS1, + DESC_RATEVHT2SS_MCS2, DESC_RATEVHT2SS_MCS3, + DESC_RATEVHT2SS_MCS4, DESC_RATEVHT2SS_MCS5, + DESC_RATEVHT2SS_MCS6, DESC_RATEVHT2SS_MCS7, + DESC_RATEVHT2SS_MCS8, DESC_RATEVHT2SS_MCS9 +}; +u8 rtw_cck_size = ARRAY_SIZE(rtw_cck_rates); +u8 rtw_ofdm_size = ARRAY_SIZE(rtw_ofdm_rates); +u8 rtw_ht_1s_size = ARRAY_SIZE(rtw_ht_1s_rates); +u8 rtw_ht_2s_size = ARRAY_SIZE(rtw_ht_2s_rates); +u8 rtw_vht_1s_size = ARRAY_SIZE(rtw_vht_1s_rates); +u8 rtw_vht_2s_size = ARRAY_SIZE(rtw_vht_2s_rates); +u8 *rtw_rate_section[RTW_RATE_SECTION_MAX] = { + rtw_cck_rates, rtw_ofdm_rates, + rtw_ht_1s_rates, rtw_ht_2s_rates, + rtw_vht_1s_rates, rtw_vht_2s_rates +}; +u8 rtw_rate_size[RTW_RATE_SECTION_MAX] = { + ARRAY_SIZE(rtw_cck_rates), + ARRAY_SIZE(rtw_ofdm_rates), + ARRAY_SIZE(rtw_ht_1s_rates), + ARRAY_SIZE(rtw_ht_2s_rates), + ARRAY_SIZE(rtw_vht_1s_rates), + ARRAY_SIZE(rtw_vht_2s_rates) +}; + +static const u8 rtw_channel_idx_5g[RTW_MAX_CHANNEL_NUM_5G] = { + 36, 38, 40, 42, 44, 46, 48, /* Band 1 */ + 52, 54, 56, 58, 60, 62, 64, /* Band 2 */ + 100, 102, 104, 106, 108, 110, 112, /* Band 3 */ + 116, 118, 120, 122, 124, 126, 128, /* Band 3 */ + 132, 134, 136, 138, 140, 142, 144, /* Band 3 */ + 149, 151, 153, 155, 157, 159, 161, /* Band 4 */ + 165, 167, 169, 171, 173, 175, 177}; /* Band 4 */ + +static int rtw_channel_to_idx(u8 band, u8 channel) +{ + int ch_idx; + u8 n_channel; + + if (band == PHY_BAND_2G) { + ch_idx = channel - 1; + n_channel = RTW_MAX_CHANNEL_NUM_2G; + } else if (band == PHY_BAND_5G) { + n_channel = RTW_MAX_CHANNEL_NUM_5G; + for (ch_idx = 0; ch_idx < n_channel; ch_idx++) + if (rtw_channel_idx_5g[ch_idx] == channel) + break; + } else { + return -1; + } + + if (ch_idx >= n_channel) + return -1; + + return ch_idx; +} + +static u8 rtw_get_channel_group(u8 channel) +{ + switch (channel) { + default: + WARN_ON(1); + /* fall through */ + case 1: + case 2: + case 36: + case 38: + case 40: + case 42: + return 0; + case 3: + case 4: + case 5: + case 44: + case 46: + case 48: + case 50: + return 1; + case 6: + case 7: + case 8: + case 52: + case 54: + case 56: + case 58: + return 2; + case 9: + case 10: + case 11: + case 60: + case 62: + case 64: + return 3; + case 12: + case 13: + case 100: + case 102: + case 104: + case 106: + return 4; + case 14: + case 108: + case 110: + case 112: + case 114: + return 5; + case 116: + case 118: + case 120: + case 122: + return 6; + case 124: + case 126: + case 128: + case 130: + return 7; + case 132: + case 134: + case 136: + case 138: + return 8; + case 140: + case 142: + case 144: + return 9; + case 149: + case 151: + case 153: + case 155: + return 10; + case 157: + case 159: + case 161: + return 11; + case 165: + case 167: + case 169: + case 171: + return 12; + case 173: + case 175: + case 177: + return 13; + } +} + +static u8 phy_get_2g_tx_power_index(struct rtw_dev *rtwdev, + struct rtw_2g_txpwr_idx *pwr_idx_2g, + enum rtw_bandwidth bandwidth, + u8 rate, u8 group) +{ + struct rtw_chip_info *chip = rtwdev->chip; + u8 tx_power; + bool mcs_rate; + bool above_2ss; + u8 factor = chip->txgi_factor; + + if (rate <= DESC_RATE11M) + tx_power = pwr_idx_2g->cck_base[group]; + else + tx_power = pwr_idx_2g->bw40_base[group]; + + if (rate >= DESC_RATE6M && rate <= DESC_RATE54M) + tx_power += pwr_idx_2g->ht_1s_diff.ofdm * factor; + + mcs_rate = (rate >= DESC_RATEMCS0 && rate <= DESC_RATEMCS15) || + (rate >= DESC_RATEVHT1SS_MCS0 && + rate <= DESC_RATEVHT2SS_MCS9); + above_2ss = (rate >= DESC_RATEMCS8 && rate <= DESC_RATEMCS15) || + (rate >= DESC_RATEVHT2SS_MCS0); + + if (!mcs_rate) + return tx_power; + + switch (bandwidth) { + default: + WARN_ON(1); + /* fall through */ + case RTW_CHANNEL_WIDTH_20: + tx_power += pwr_idx_2g->ht_1s_diff.bw20 * factor; + if (above_2ss) + tx_power += pwr_idx_2g->ht_2s_diff.bw20 * factor; + break; + case RTW_CHANNEL_WIDTH_40: + /* bw40 is the base power */ + if (above_2ss) + tx_power += pwr_idx_2g->ht_2s_diff.bw40 * factor; + break; + } + + return tx_power; +} + +static u8 phy_get_5g_tx_power_index(struct rtw_dev *rtwdev, + struct rtw_5g_txpwr_idx *pwr_idx_5g, + enum rtw_bandwidth bandwidth, + u8 rate, u8 group) +{ + struct rtw_chip_info *chip = rtwdev->chip; + u8 tx_power; + u8 upper, lower; + bool mcs_rate; + bool above_2ss; + u8 factor = chip->txgi_factor; + + tx_power = pwr_idx_5g->bw40_base[group]; + + mcs_rate = (rate >= DESC_RATEMCS0 && rate <= DESC_RATEMCS15) || + (rate >= DESC_RATEVHT1SS_MCS0 && + rate <= DESC_RATEVHT2SS_MCS9); + above_2ss = (rate >= DESC_RATEMCS8 && rate <= DESC_RATEMCS15) || + (rate >= DESC_RATEVHT2SS_MCS0); + + if (!mcs_rate) { + tx_power += pwr_idx_5g->ht_1s_diff.ofdm * factor; + return tx_power; + } + + switch (bandwidth) { + default: + WARN_ON(1); + /* fall through */ + case RTW_CHANNEL_WIDTH_20: + tx_power += pwr_idx_5g->ht_1s_diff.bw20 * factor; + if (above_2ss) + tx_power += pwr_idx_5g->ht_2s_diff.bw20 * factor; + break; + case RTW_CHANNEL_WIDTH_40: + /* bw40 is the base power */ + if (above_2ss) + tx_power += pwr_idx_5g->ht_2s_diff.bw40 * factor; + break; + case RTW_CHANNEL_WIDTH_80: + /* the base idx of bw80 is the average of bw40+/bw40- */ + lower = pwr_idx_5g->bw40_base[group]; + upper = pwr_idx_5g->bw40_base[group + 1]; + + tx_power = (lower + upper) / 2; + tx_power += pwr_idx_5g->vht_1s_diff.bw80 * factor; + if (above_2ss) + tx_power += pwr_idx_5g->vht_2s_diff.bw80 * factor; + break; + } + + return tx_power; +} + +/* set tx power level by path for each rates, note that the order of the rates + * are *very* important, bacause 8822B/8821C combines every four bytes of tx + * power index into a four-byte power index register, and calls set_tx_agc to + * write these values into hardware + */ +static +void phy_set_tx_power_level_by_path(struct rtw_dev *rtwdev, u8 ch, u8 path) +{ + struct rtw_hal *hal = &rtwdev->hal; + u8 rs; + + /* do not need cck rates if we are not in 2.4G */ + if (hal->current_band_type == RTW_BAND_2G) + rs = RTW_RATE_SECTION_CCK; + else + rs = RTW_RATE_SECTION_OFDM; + + for (; rs < RTW_RATE_SECTION_MAX; rs++) + phy_set_tx_power_index_by_rs(rtwdev, ch, path, rs); +} + +void rtw_phy_set_tx_power_level(struct rtw_dev *rtwdev, u8 channel) +{ + struct rtw_chip_info *chip = rtwdev->chip; + struct rtw_hal *hal = &rtwdev->hal; + u8 path; + + mutex_lock(&hal->tx_power_mutex); + + for (path = 0; path < hal->rf_path_num; path++) + phy_set_tx_power_level_by_path(rtwdev, channel, path); + + chip->ops->set_tx_power_index(rtwdev); + mutex_unlock(&hal->tx_power_mutex); +} + +s8 phy_get_tx_power_limit(struct rtw_dev *rtwdev, u8 band, + enum rtw_bandwidth bandwidth, u8 rf_path, + u8 rate, u8 channel, u8 regd); + +static +u8 phy_get_tx_power_index(void *adapter, u8 rf_path, u8 rate, + enum rtw_bandwidth bandwidth, u8 channel, u8 regd) +{ + struct rtw_dev *rtwdev = adapter; + struct rtw_hal *hal = &rtwdev->hal; + struct rtw_txpwr_idx *pwr_idx; + u8 tx_power; + u8 group; + u8 band; + s8 offset, limit; + + pwr_idx = &rtwdev->efuse.txpwr_idx_table[rf_path]; + group = rtw_get_channel_group(channel); + + /* base power index for 2.4G/5G */ + if (channel <= 14) { + band = PHY_BAND_2G; + tx_power = phy_get_2g_tx_power_index(rtwdev, + &pwr_idx->pwr_idx_2g, + bandwidth, rate, group); + offset = hal->tx_pwr_by_rate_offset_2g[rf_path][rate]; + } else { + band = PHY_BAND_5G; + tx_power = phy_get_5g_tx_power_index(rtwdev, + &pwr_idx->pwr_idx_5g, + bandwidth, rate, group); + offset = hal->tx_pwr_by_rate_offset_5g[rf_path][rate]; + } + + limit = phy_get_tx_power_limit(rtwdev, band, bandwidth, rf_path, + rate, channel, regd); + + if (offset > limit) + offset = limit; + + tx_power += offset; + + if (tx_power > rtwdev->chip->max_power_index) + tx_power = rtwdev->chip->max_power_index; + + return tx_power; +} + +void phy_set_tx_power_index_by_rs(void *adapter, u8 ch, u8 path, u8 rs) +{ + struct rtw_dev *rtwdev = adapter; + struct rtw_hal *hal = &rtwdev->hal; + u8 regd = rtwdev->regd.txpwr_regd; + u8 *rates; + u8 size; + u8 rate; + u8 pwr_idx; + u8 bw; + int i; + + if (rs >= RTW_RATE_SECTION_MAX) + return; + + rates = rtw_rate_section[rs]; + size = rtw_rate_size[rs]; + bw = hal->current_band_width; + for (i = 0; i < size; i++) { + rate = rates[i]; + pwr_idx = phy_get_tx_power_index(adapter, path, rate, bw, ch, + regd); + hal->tx_pwr_tbl[path][rate] = pwr_idx; + } +} + +static u8 tbl_to_dec_pwr_by_rate(struct rtw_dev *rtwdev, u32 hex, u8 i) +{ + if (rtwdev->chip->is_pwr_by_rate_dec) + return bcd_to_dec_pwr_by_rate(hex, i); + else + return (hex >> (i * 8)) & 0xFF; +} + +static void phy_get_rate_values_of_txpwr_by_rate(struct rtw_dev *rtwdev, + u32 addr, u32 mask, + u32 val, u8 *rate, + u8 *pwr_by_rate, u8 *rate_num) +{ + int i; + + switch (addr) { + case 0xE00: + case 0x830: + rate[0] = DESC_RATE6M; + rate[1] = DESC_RATE9M; + rate[2] = DESC_RATE12M; + rate[3] = DESC_RATE18M; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xE04: + case 0x834: + rate[0] = DESC_RATE24M; + rate[1] = DESC_RATE36M; + rate[2] = DESC_RATE48M; + rate[3] = DESC_RATE54M; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xE08: + rate[0] = DESC_RATE1M; + pwr_by_rate[0] = bcd_to_dec_pwr_by_rate(val, 1); + *rate_num = 1; + break; + case 0x86C: + if (mask == 0xffffff00) { + rate[0] = DESC_RATE2M; + rate[1] = DESC_RATE5_5M; + rate[2] = DESC_RATE11M; + for (i = 1; i < 4; ++i) + pwr_by_rate[i - 1] = + tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 3; + } else if (mask == 0x000000ff) { + rate[0] = DESC_RATE11M; + pwr_by_rate[0] = bcd_to_dec_pwr_by_rate(val, 0); + *rate_num = 1; + } + break; + case 0xE10: + case 0x83C: + rate[0] = DESC_RATEMCS0; + rate[1] = DESC_RATEMCS1; + rate[2] = DESC_RATEMCS2; + rate[3] = DESC_RATEMCS3; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xE14: + case 0x848: + rate[0] = DESC_RATEMCS4; + rate[1] = DESC_RATEMCS5; + rate[2] = DESC_RATEMCS6; + rate[3] = DESC_RATEMCS7; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xE18: + case 0x84C: + rate[0] = DESC_RATEMCS8; + rate[1] = DESC_RATEMCS9; + rate[2] = DESC_RATEMCS10; + rate[3] = DESC_RATEMCS11; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xE1C: + case 0x868: + rate[0] = DESC_RATEMCS12; + rate[1] = DESC_RATEMCS13; + rate[2] = DESC_RATEMCS14; + rate[3] = DESC_RATEMCS15; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + + break; + case 0x838: + rate[0] = DESC_RATE1M; + rate[1] = DESC_RATE2M; + rate[2] = DESC_RATE5_5M; + for (i = 1; i < 4; ++i) + pwr_by_rate[i - 1] = tbl_to_dec_pwr_by_rate(rtwdev, + val, i); + *rate_num = 3; + break; + case 0xC20: + case 0xE20: + case 0x1820: + case 0x1A20: + rate[0] = DESC_RATE1M; + rate[1] = DESC_RATE2M; + rate[2] = DESC_RATE5_5M; + rate[3] = DESC_RATE11M; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xC24: + case 0xE24: + case 0x1824: + case 0x1A24: + rate[0] = DESC_RATE6M; + rate[1] = DESC_RATE9M; + rate[2] = DESC_RATE12M; + rate[3] = DESC_RATE18M; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xC28: + case 0xE28: + case 0x1828: + case 0x1A28: + rate[0] = DESC_RATE24M; + rate[1] = DESC_RATE36M; + rate[2] = DESC_RATE48M; + rate[3] = DESC_RATE54M; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xC2C: + case 0xE2C: + case 0x182C: + case 0x1A2C: + rate[0] = DESC_RATEMCS0; + rate[1] = DESC_RATEMCS1; + rate[2] = DESC_RATEMCS2; + rate[3] = DESC_RATEMCS3; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xC30: + case 0xE30: + case 0x1830: + case 0x1A30: + rate[0] = DESC_RATEMCS4; + rate[1] = DESC_RATEMCS5; + rate[2] = DESC_RATEMCS6; + rate[3] = DESC_RATEMCS7; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xC34: + case 0xE34: + case 0x1834: + case 0x1A34: + rate[0] = DESC_RATEMCS8; + rate[1] = DESC_RATEMCS9; + rate[2] = DESC_RATEMCS10; + rate[3] = DESC_RATEMCS11; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xC38: + case 0xE38: + case 0x1838: + case 0x1A38: + rate[0] = DESC_RATEMCS12; + rate[1] = DESC_RATEMCS13; + rate[2] = DESC_RATEMCS14; + rate[3] = DESC_RATEMCS15; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xC3C: + case 0xE3C: + case 0x183C: + case 0x1A3C: + rate[0] = DESC_RATEVHT1SS_MCS0; + rate[1] = DESC_RATEVHT1SS_MCS1; + rate[2] = DESC_RATEVHT1SS_MCS2; + rate[3] = DESC_RATEVHT1SS_MCS3; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xC40: + case 0xE40: + case 0x1840: + case 0x1A40: + rate[0] = DESC_RATEVHT1SS_MCS4; + rate[1] = DESC_RATEVHT1SS_MCS5; + rate[2] = DESC_RATEVHT1SS_MCS6; + rate[3] = DESC_RATEVHT1SS_MCS7; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xC44: + case 0xE44: + case 0x1844: + case 0x1A44: + rate[0] = DESC_RATEVHT1SS_MCS8; + rate[1] = DESC_RATEVHT1SS_MCS9; + rate[2] = DESC_RATEVHT2SS_MCS0; + rate[3] = DESC_RATEVHT2SS_MCS1; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xC48: + case 0xE48: + case 0x1848: + case 0x1A48: + rate[0] = DESC_RATEVHT2SS_MCS2; + rate[1] = DESC_RATEVHT2SS_MCS3; + rate[2] = DESC_RATEVHT2SS_MCS4; + rate[3] = DESC_RATEVHT2SS_MCS5; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xC4C: + case 0xE4C: + case 0x184C: + case 0x1A4C: + rate[0] = DESC_RATEVHT2SS_MCS6; + rate[1] = DESC_RATEVHT2SS_MCS7; + rate[2] = DESC_RATEVHT2SS_MCS8; + rate[3] = DESC_RATEVHT2SS_MCS9; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xCD8: + case 0xED8: + case 0x18D8: + case 0x1AD8: + rate[0] = DESC_RATEMCS16; + rate[1] = DESC_RATEMCS17; + rate[2] = DESC_RATEMCS18; + rate[3] = DESC_RATEMCS19; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xCDC: + case 0xEDC: + case 0x18DC: + case 0x1ADC: + rate[0] = DESC_RATEMCS20; + rate[1] = DESC_RATEMCS21; + rate[2] = DESC_RATEMCS22; + rate[3] = DESC_RATEMCS23; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xCE0: + case 0xEE0: + case 0x18E0: + case 0x1AE0: + rate[0] = DESC_RATEVHT3SS_MCS0; + rate[1] = DESC_RATEVHT3SS_MCS1; + rate[2] = DESC_RATEVHT3SS_MCS2; + rate[3] = DESC_RATEVHT3SS_MCS3; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xCE4: + case 0xEE4: + case 0x18E4: + case 0x1AE4: + rate[0] = DESC_RATEVHT3SS_MCS4; + rate[1] = DESC_RATEVHT3SS_MCS5; + rate[2] = DESC_RATEVHT3SS_MCS6; + rate[3] = DESC_RATEVHT3SS_MCS7; + for (i = 0; i < 4; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 4; + break; + case 0xCE8: + case 0xEE8: + case 0x18E8: + case 0x1AE8: + rate[0] = DESC_RATEVHT3SS_MCS8; + rate[1] = DESC_RATEVHT3SS_MCS9; + for (i = 0; i < 2; ++i) + pwr_by_rate[i] = tbl_to_dec_pwr_by_rate(rtwdev, val, i); + *rate_num = 2; + break; + default: + rtw_warn(rtwdev, "invalid tx power index addr 0x%08x\n", addr); + break; + } +} + +void phy_store_tx_power_by_rate(void *adapter, u32 band, u32 rfpath, u32 txnum, + u32 regaddr, u32 bitmask, u32 data) +{ + struct rtw_dev *rtwdev = adapter; + struct rtw_hal *hal = &rtwdev->hal; + u8 rate_num = 0; + u8 rate; + u8 rates[RTW_RF_PATH_MAX] = {0}; + s8 offset; + s8 pwr_by_rate[RTW_RF_PATH_MAX] = {0}; + int i; + + phy_get_rate_values_of_txpwr_by_rate(rtwdev, regaddr, bitmask, data, + rates, pwr_by_rate, &rate_num); + + if (WARN_ON(rfpath >= RTW_RF_PATH_MAX || + (band != PHY_BAND_2G && band != PHY_BAND_5G) || + rate_num > RTW_RF_PATH_MAX)) + return; + + for (i = 0; i < rate_num; i++) { + offset = pwr_by_rate[i]; + rate = rates[i]; + if (band == PHY_BAND_2G) + hal->tx_pwr_by_rate_offset_2g[rfpath][rate] = offset; + else if (band == PHY_BAND_5G) + hal->tx_pwr_by_rate_offset_5g[rfpath][rate] = offset; + else + continue; + } +} + +static +void phy_tx_power_by_rate_config_by_path(struct rtw_hal *hal, u8 path, + u8 rs, u8 size, u8 *rates) +{ + u8 rate; + u8 base_idx, rate_idx; + s8 base_2g, base_5g; + + if (rs >= RTW_RATE_SECTION_VHT_1S) + base_idx = rates[size - 3]; + else + base_idx = rates[size - 1]; + base_2g = hal->tx_pwr_by_rate_offset_2g[path][base_idx]; + base_5g = hal->tx_pwr_by_rate_offset_5g[path][base_idx]; + hal->tx_pwr_by_rate_base_2g[path][rs] = base_2g; + hal->tx_pwr_by_rate_base_5g[path][rs] = base_5g; + for (rate = 0; rate < size; rate++) { + rate_idx = rates[rate]; + hal->tx_pwr_by_rate_offset_2g[path][rate_idx] -= base_2g; + hal->tx_pwr_by_rate_offset_5g[path][rate_idx] -= base_5g; + } +} + +void rtw_phy_tx_power_by_rate_config(struct rtw_hal *hal) +{ + u8 path; + + for (path = 0; path < RTW_RF_PATH_MAX; path++) { + phy_tx_power_by_rate_config_by_path(hal, path, + RTW_RATE_SECTION_CCK, + rtw_cck_size, rtw_cck_rates); + phy_tx_power_by_rate_config_by_path(hal, path, + RTW_RATE_SECTION_OFDM, + rtw_ofdm_size, rtw_ofdm_rates); + phy_tx_power_by_rate_config_by_path(hal, path, + RTW_RATE_SECTION_HT_1S, + rtw_ht_1s_size, rtw_ht_1s_rates); + phy_tx_power_by_rate_config_by_path(hal, path, + RTW_RATE_SECTION_HT_2S, + rtw_ht_2s_size, rtw_ht_2s_rates); + phy_tx_power_by_rate_config_by_path(hal, path, + RTW_RATE_SECTION_VHT_1S, + rtw_vht_1s_size, rtw_vht_1s_rates); + phy_tx_power_by_rate_config_by_path(hal, path, + RTW_RATE_SECTION_VHT_2S, + rtw_vht_2s_size, rtw_vht_2s_rates); + } +} + +static void +phy_tx_power_limit_config(struct rtw_hal *hal, u8 regd, u8 bw, u8 rs) +{ + s8 base, orig; + u8 ch; + + for (ch = 0; ch < RTW_MAX_CHANNEL_NUM_2G; ch++) { + base = hal->tx_pwr_by_rate_base_2g[0][rs]; + orig = hal->tx_pwr_limit_2g[regd][bw][rs][ch]; + hal->tx_pwr_limit_2g[regd][bw][rs][ch] -= base; + } + + for (ch = 0; ch < RTW_MAX_CHANNEL_NUM_5G; ch++) { + base = hal->tx_pwr_by_rate_base_5g[0][rs]; + hal->tx_pwr_limit_5g[regd][bw][rs][ch] -= base; + } +} + +void rtw_phy_tx_power_limit_config(struct rtw_hal *hal) +{ + u8 regd, bw, rs; + + for (regd = 0; regd < RTW_REGD_MAX; regd++) + for (bw = 0; bw < RTW_CHANNEL_WIDTH_MAX; bw++) + for (rs = 0; rs < RTW_RATE_SECTION_MAX; rs++) + phy_tx_power_limit_config(hal, regd, bw, rs); +} + +static s8 get_tx_power_limit(struct rtw_hal *hal, u8 bw, u8 rs, u8 ch, u8 regd) +{ + if (regd > RTW_REGD_WW) + return RTW_MAX_POWER_INDEX; + + return hal->tx_pwr_limit_2g[regd][bw][rs][ch]; +} + +s8 phy_get_tx_power_limit(struct rtw_dev *rtwdev, u8 band, + enum rtw_bandwidth bw, u8 rf_path, + u8 rate, u8 channel, u8 regd) +{ + struct rtw_hal *hal = &rtwdev->hal; + s8 power_limit; + u8 rs; + int ch_idx; + + if (rate >= DESC_RATE1M && rate <= DESC_RATE11M) + rs = RTW_RATE_SECTION_CCK; + else if (rate >= DESC_RATE6M && rate <= DESC_RATE54M) + rs = RTW_RATE_SECTION_OFDM; + else if (rate >= DESC_RATEMCS0 && rate <= DESC_RATEMCS7) + rs = RTW_RATE_SECTION_HT_1S; + else if (rate >= DESC_RATEMCS8 && rate <= DESC_RATEMCS15) + rs = RTW_RATE_SECTION_HT_2S; + else if (rate >= DESC_RATEVHT1SS_MCS0 && rate <= DESC_RATEVHT1SS_MCS9) + rs = RTW_RATE_SECTION_VHT_1S; + else if (rate >= DESC_RATEVHT2SS_MCS0 && rate <= DESC_RATEVHT2SS_MCS9) + rs = RTW_RATE_SECTION_VHT_2S; + else + goto err; + + ch_idx = rtw_channel_to_idx(band, channel); + if (ch_idx < 0) + goto err; + + power_limit = get_tx_power_limit(hal, bw, rs, ch_idx, regd); + + return power_limit; + +err: + WARN(1, "invalid arguments, band=%d, bw=%d, path=%d, rate=%d, ch=%d\n", + band, bw, rf_path, rate, channel); + return RTW_MAX_POWER_INDEX; +} + +void phy_set_tx_power_limit(struct rtw_dev *rtwdev, u8 regd, u8 band, + u8 bw, u8 rs, u8 ch, s8 pwr_limit) +{ + struct rtw_hal *hal = &rtwdev->hal; + int ch_idx; + + pwr_limit = clamp_t(s8, pwr_limit, + -RTW_MAX_POWER_INDEX, RTW_MAX_POWER_INDEX); + ch_idx = rtw_channel_to_idx(band, ch); + + if (regd >= RTW_REGD_MAX || bw >= RTW_CHANNEL_WIDTH_MAX || + rs >= RTW_RATE_SECTION_MAX || ch_idx < 0) { + WARN(1, + "wrong txpwr_lmt regd=%u, band=%u bw=%u, rs=%u, ch_idx=%u, pwr_limit=%d\n", + regd, band, bw, rs, ch_idx, pwr_limit); + return; + } + + if (band == PHY_BAND_2G) + hal->tx_pwr_limit_2g[regd][bw][rs][ch_idx] = pwr_limit; + else if (band == PHY_BAND_5G) + hal->tx_pwr_limit_5g[regd][bw][rs][ch_idx] = pwr_limit; +} + +static +void rtw_hw_tx_power_limit_init(struct rtw_hal *hal, u8 regd, u8 bw, u8 rs) +{ + u8 ch; + + /* 2.4G channels */ + for (ch = 0; ch < RTW_MAX_CHANNEL_NUM_2G; ch++) + hal->tx_pwr_limit_2g[regd][bw][rs][ch] = RTW_MAX_POWER_INDEX; + + /* 5G channels */ + for (ch = 0; ch < RTW_MAX_CHANNEL_NUM_5G; ch++) + hal->tx_pwr_limit_5g[regd][bw][rs][ch] = RTW_MAX_POWER_INDEX; +} + +void rtw_hw_init_tx_power(struct rtw_hal *hal) +{ + u8 regd, path, rate, rs, bw; + + /* init tx power by rate offset */ + for (path = 0; path < RTW_RF_PATH_MAX; path++) { + for (rate = 0; rate < DESC_RATE_MAX; rate++) { + hal->tx_pwr_by_rate_offset_2g[path][rate] = 0; + hal->tx_pwr_by_rate_offset_5g[path][rate] = 0; + } + } + + /* init tx power limit */ + for (regd = 0; regd < RTW_REGD_MAX; regd++) + for (bw = 0; bw < RTW_CHANNEL_WIDTH_MAX; bw++) + for (rs = 0; rs < RTW_RATE_SECTION_MAX; rs++) + rtw_hw_tx_power_limit_init(hal, regd, bw, rs); +} diff --git a/drivers/net/wireless/realtek/rtw88/phy.h b/drivers/net/wireless/realtek/rtw88/phy.h new file mode 100644 index 0000000000000000000000000000000000000000..ec03a2051e52bdb4371514197d10bb4c81acf9a3 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/phy.h @@ -0,0 +1,134 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#ifndef __RTW_PHY_H_ +#define __RTW_PHY_H_ + +#include "debug.h" + +extern u8 rtw_cck_rates[]; +extern u8 rtw_ofdm_rates[]; +extern u8 rtw_ht_1s_rates[]; +extern u8 rtw_ht_2s_rates[]; +extern u8 rtw_vht_1s_rates[]; +extern u8 rtw_vht_2s_rates[]; +extern u8 *rtw_rate_section[]; +extern u8 rtw_rate_size[]; + +void rtw_phy_init(struct rtw_dev *rtwdev); +void rtw_phy_dynamic_mechanism(struct rtw_dev *rtwdev); +u8 rtw_phy_rf_power_2_rssi(s8 *rf_power, u8 path_num); +u32 rtw_phy_read_rf(struct rtw_dev *rtwdev, enum rtw_rf_path rf_path, + u32 addr, u32 mask); +bool rtw_phy_write_rf_reg_sipi(struct rtw_dev *rtwdev, enum rtw_rf_path rf_path, + u32 addr, u32 mask, u32 data); +bool rtw_phy_write_rf_reg(struct rtw_dev *rtwdev, enum rtw_rf_path rf_path, + u32 addr, u32 mask, u32 data); +bool rtw_phy_write_rf_reg_mix(struct rtw_dev *rtwdev, enum rtw_rf_path rf_path, + u32 addr, u32 mask, u32 data); +void phy_store_tx_power_by_rate(void *adapter, u32 band, u32 rfpath, u32 txnum, + u32 regaddr, u32 bitmask, u32 data); +void phy_set_tx_power_limit(struct rtw_dev *rtwdev, u8 regd, u8 band, + u8 bw, u8 rs, u8 ch, s8 pwr_limit); +void phy_set_tx_power_index_by_rs(void *adapter, u8 ch, u8 path, u8 rs); +void rtw_phy_setup_phy_cond(struct rtw_dev *rtwdev, u32 pkg); +void rtw_parse_tbl_phy_cond(struct rtw_dev *rtwdev, const struct rtw_table *tbl); +void rtw_parse_tbl_bb_pg(struct rtw_dev *rtwdev, const struct rtw_table *tbl); +void rtw_parse_tbl_txpwr_lmt(struct rtw_dev *rtwdev, const struct rtw_table *tbl); +void rtw_phy_cfg_mac(struct rtw_dev *rtwdev, const struct rtw_table *tbl, + u32 addr, u32 data); +void rtw_phy_cfg_agc(struct rtw_dev *rtwdev, const struct rtw_table *tbl, + u32 addr, u32 data); +void rtw_phy_cfg_bb(struct rtw_dev *rtwdev, const struct rtw_table *tbl, + u32 addr, u32 data); +void rtw_phy_cfg_rf(struct rtw_dev *rtwdev, const struct rtw_table *tbl, + u32 addr, u32 data); +void rtw_hw_init_tx_power(struct rtw_hal *hal); +void rtw_phy_load_tables(struct rtw_dev *rtwdev); +void rtw_phy_set_tx_power_level(struct rtw_dev *rtwdev, u8 channel); +void rtw_phy_tx_power_by_rate_config(struct rtw_hal *hal); +void rtw_phy_tx_power_limit_config(struct rtw_hal *hal); + +#define RTW_DECL_TABLE_PHY_COND_CORE(name, cfg, path) \ +const struct rtw_table name ## _tbl = { \ + .data = name, \ + .size = ARRAY_SIZE(name), \ + .parse = rtw_parse_tbl_phy_cond, \ + .do_cfg = cfg, \ + .rf_path = path, \ +} + +#define RTW_DECL_TABLE_PHY_COND(name, cfg) \ + RTW_DECL_TABLE_PHY_COND_CORE(name, cfg, 0) + +#define RTW_DECL_TABLE_RF_RADIO(name, path) \ + RTW_DECL_TABLE_PHY_COND_CORE(name, rtw_phy_cfg_rf, RF_PATH_ ## path) + +#define RTW_DECL_TABLE_BB_PG(name) \ +const struct rtw_table name ## _tbl = { \ + .data = name, \ + .size = ARRAY_SIZE(name), \ + .parse = rtw_parse_tbl_bb_pg, \ +} + +#define RTW_DECL_TABLE_TXPWR_LMT(name) \ +const struct rtw_table name ## _tbl = { \ + .data = name, \ + .size = ARRAY_SIZE(name), \ + .parse = rtw_parse_tbl_txpwr_lmt, \ +} + +static inline const struct rtw_rfe_def *rtw_get_rfe_def(struct rtw_dev *rtwdev) +{ + struct rtw_chip_info *chip = rtwdev->chip; + struct rtw_efuse *efuse = &rtwdev->efuse; + const struct rtw_rfe_def *rfe_def = NULL; + + if (chip->rfe_defs_size == 0) + return NULL; + + if (efuse->rfe_option < chip->rfe_defs_size) + rfe_def = &chip->rfe_defs[efuse->rfe_option]; + + rtw_dbg(rtwdev, RTW_DBG_PHY, "use rfe_def[%d]\n", efuse->rfe_option); + return rfe_def; +} + +static inline int rtw_check_supported_rfe(struct rtw_dev *rtwdev) +{ + const struct rtw_rfe_def *rfe_def = rtw_get_rfe_def(rtwdev); + + if (!rfe_def || !rfe_def->phy_pg_tbl || !rfe_def->txpwr_lmt_tbl) { + rtw_err(rtwdev, "rfe %d isn't supported\n", + rtwdev->efuse.rfe_option); + return -ENODEV; + } + + return 0; +} + +void rtw_phy_dig_write(struct rtw_dev *rtwdev, u8 igi); + +#define MASKBYTE0 0xff +#define MASKBYTE1 0xff00 +#define MASKBYTE2 0xff0000 +#define MASKBYTE3 0xff000000 +#define MASKHWORD 0xffff0000 +#define MASKLWORD 0x0000ffff +#define MASKDWORD 0xffffffff +#define RFREG_MASK 0xfffff + +#define MASK7BITS 0x7f +#define MASK12BITS 0xfff +#define MASKH4BITS 0xf0000000 +#define MASK20BITS 0xfffff +#define MASK24BITS 0xffffff + +#define MASKH3BYTES 0xffffff00 +#define MASKL3BYTES 0x00ffffff +#define MASKBYTE2HIGHNIBBLE 0x00f00000 +#define MASKBYTE3LOWNIBBLE 0x0f000000 +#define MASKL3BYTES 0x00ffffff + +#endif diff --git a/drivers/net/wireless/realtek/rtw88/ps.c b/drivers/net/wireless/realtek/rtw88/ps.c new file mode 100644 index 0000000000000000000000000000000000000000..607bfa4317d926bbda3cef1c65cca6aefc1ae239 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/ps.c @@ -0,0 +1,166 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#include "main.h" +#include "fw.h" +#include "ps.h" +#include "mac.h" +#include "debug.h" + +static int rtw_ips_pwr_up(struct rtw_dev *rtwdev) +{ + int ret; + + ret = rtw_core_start(rtwdev); + if (ret) + rtw_err(rtwdev, "leave idle state failed\n"); + + rtw_set_channel(rtwdev); + rtw_flag_clear(rtwdev, RTW_FLAG_INACTIVE_PS); + + return ret; +} + +int rtw_enter_ips(struct rtw_dev *rtwdev) +{ + rtw_flag_set(rtwdev, RTW_FLAG_INACTIVE_PS); + + rtw_core_stop(rtwdev); + + return 0; +} + +static void rtw_restore_port_cfg_iter(void *data, u8 *mac, + struct ieee80211_vif *vif) +{ + struct rtw_dev *rtwdev = data; + struct rtw_vif *rtwvif = (struct rtw_vif *)vif->drv_priv; + u32 config = ~0; + + rtw_vif_port_config(rtwdev, rtwvif, config); +} + +int rtw_leave_ips(struct rtw_dev *rtwdev) +{ + int ret; + + ret = rtw_ips_pwr_up(rtwdev); + if (ret) { + rtw_err(rtwdev, "failed to leave ips state\n"); + return ret; + } + + rtw_iterate_vifs_atomic(rtwdev, rtw_restore_port_cfg_iter, rtwdev); + + return 0; +} + +static void rtw_leave_lps_core(struct rtw_dev *rtwdev) +{ + struct rtw_lps_conf *conf = &rtwdev->lps_conf; + + conf->state = RTW_ALL_ON; + conf->awake_interval = 1; + conf->rlbm = 0; + conf->smart_ps = 0; + + rtw_fw_set_pwr_mode(rtwdev); + rtw_flag_clear(rtwdev, RTW_FLAG_LEISURE_PS); +} + +static void rtw_enter_lps_core(struct rtw_dev *rtwdev) +{ + struct rtw_lps_conf *conf = &rtwdev->lps_conf; + + conf->state = RTW_RF_OFF; + conf->awake_interval = 1; + conf->rlbm = 1; + conf->smart_ps = 2; + + rtw_fw_set_pwr_mode(rtwdev); + rtw_flag_set(rtwdev, RTW_FLAG_LEISURE_PS); +} + +void rtw_lps_work(struct work_struct *work) +{ + struct rtw_dev *rtwdev = container_of(work, struct rtw_dev, + lps_work.work); + struct rtw_lps_conf *conf = &rtwdev->lps_conf; + struct rtw_vif *rtwvif = conf->rtwvif; + + if (WARN_ON(!rtwvif)) + return; + + if (conf->mode == RTW_MODE_LPS) + rtw_enter_lps_core(rtwdev); + else + rtw_leave_lps_core(rtwdev); +} + +void rtw_enter_lps_irqsafe(struct rtw_dev *rtwdev, struct rtw_vif *rtwvif) +{ + struct rtw_lps_conf *conf = &rtwdev->lps_conf; + + if (rtwvif->in_lps) + return; + + conf->mode = RTW_MODE_LPS; + conf->rtwvif = rtwvif; + rtwvif->in_lps = true; + + ieee80211_queue_delayed_work(rtwdev->hw, &rtwdev->lps_work, 0); +} + +void rtw_leave_lps_irqsafe(struct rtw_dev *rtwdev, struct rtw_vif *rtwvif) +{ + struct rtw_lps_conf *conf = &rtwdev->lps_conf; + + if (!rtwvif->in_lps) + return; + + conf->mode = RTW_MODE_ACTIVE; + conf->rtwvif = rtwvif; + rtwvif->in_lps = false; + + ieee80211_queue_delayed_work(rtwdev->hw, &rtwdev->lps_work, 0); +} + +bool rtw_in_lps(struct rtw_dev *rtwdev) +{ + return rtw_flag_check(rtwdev, RTW_FLAG_LEISURE_PS); +} + +void rtw_enter_lps(struct rtw_dev *rtwdev, struct rtw_vif *rtwvif) +{ + struct rtw_lps_conf *conf = &rtwdev->lps_conf; + + if (WARN_ON(!rtwvif)) + return; + + if (rtwvif->in_lps) + return; + + conf->mode = RTW_MODE_LPS; + conf->rtwvif = rtwvif; + rtwvif->in_lps = true; + + rtw_enter_lps_core(rtwdev); +} + +void rtw_leave_lps(struct rtw_dev *rtwdev, struct rtw_vif *rtwvif) +{ + struct rtw_lps_conf *conf = &rtwdev->lps_conf; + + if (WARN_ON(!rtwvif)) + return; + + if (!rtwvif->in_lps) + return; + + conf->mode = RTW_MODE_ACTIVE; + conf->rtwvif = rtwvif; + rtwvif->in_lps = false; + + rtw_leave_lps_core(rtwdev); +} diff --git a/drivers/net/wireless/realtek/rtw88/ps.h b/drivers/net/wireless/realtek/rtw88/ps.h new file mode 100644 index 0000000000000000000000000000000000000000..09e57405dc1b919e2ad311bcb1a23c4e23849c74 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/ps.h @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#ifndef __RTW_PS_H_ +#define __RTW_PS_H_ + +#define RTW_LPS_THRESHOLD 2 + +int rtw_enter_ips(struct rtw_dev *rtwdev); +int rtw_leave_ips(struct rtw_dev *rtwdev); + +void rtw_lps_work(struct work_struct *work); +void rtw_enter_lps_irqsafe(struct rtw_dev *rtwdev, struct rtw_vif *rtwvif); +void rtw_leave_lps_irqsafe(struct rtw_dev *rtwdev, struct rtw_vif *rtwvif); +void rtw_enter_lps(struct rtw_dev *rtwdev, struct rtw_vif *rtwvif); +void rtw_leave_lps(struct rtw_dev *rtwdev, struct rtw_vif *rtwvif); +bool rtw_in_lps(struct rtw_dev *rtwdev); + +#endif diff --git a/drivers/net/wireless/realtek/rtw88/reg.h b/drivers/net/wireless/realtek/rtw88/reg.h new file mode 100644 index 0000000000000000000000000000000000000000..e2628f05812c6572b57f18b9facf7fca8a8649b3 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/reg.h @@ -0,0 +1,421 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#ifndef __RTW_REG_DEF_H__ +#define __RTW_REG_DEF_H__ + +#define REG_SYS_FUNC_EN 0x0002 +#define BIT_FEN_CPUEN BIT(2) +#define BIT_FEN_BB_GLB_RST BIT(1) +#define BIT_FEN_BB_RSTB BIT(0) +#define REG_SYS_PW_CTRL 0x0004 +#define REG_SYS_CLK_CTRL 0x0008 +#define BIT_CPU_CLK_EN BIT(14) + +#define REG_RSV_CTRL 0x001C +#define DISABLE_PI 0x3 +#define ENABLE_PI 0x2 +#define BITS_RFC_DIRECT (BIT(31) | BIT(30)) +#define BIT_WLMCU_IOIF BIT(0) +#define REG_RF_CTRL 0x001F +#define BIT_RF_SDM_RSTB BIT(2) +#define BIT_RF_RSTB BIT(1) +#define BIT_RF_EN BIT(0) + +#define REG_AFE_CTRL1 0x0024 +#define BIT_MAC_CLK_SEL (BIT(20) | BIT(21)) +#define REG_EFUSE_CTRL 0x0030 +#define BIT_EF_FLAG BIT(31) +#define BIT_SHIFT_EF_ADDR 8 +#define BIT_MASK_EF_ADDR 0x3ff +#define BIT_MASK_EF_DATA 0xff +#define BITS_EF_ADDR (BIT_MASK_EF_ADDR << BIT_SHIFT_EF_ADDR) + +#define REG_LDO_EFUSE_CTRL 0x0034 +#define BIT_MASK_EFUSE_BANK_SEL (BIT(8) | BIT(9)) + +#define REG_GPIO_MUXCFG 0x0040 +#define BIT_FSPI_EN BIT(19) +#define BIT_WLRFE_4_5_EN BIT(2) + +#define REG_LED_CFG 0x004C +#define BIT_LNAON_SEL_EN BIT(26) +#define BIT_PAPE_SEL_EN BIT(25) +#define REG_PAD_CTRL1 0x0064 +#define BIT_PAPE_WLBT_SEL BIT(29) +#define BIT_LNAON_WLBT_SEL BIT(28) +#define REG_WL_BT_PWR_CTRL 0x0068 +#define BIT_BT_FUNC_EN BIT(18) +#define BIT_BT_DIG_CLK_EN BIT(8) +#define REG_HCI_OPT_CTRL 0x0074 + +#define REG_MCUFW_CTRL 0x0080 +#define BIT_ANA_PORT_EN BIT(22) +#define BIT_MAC_PORT_EN BIT(21) +#define BIT_BOOT_FSPI_EN BIT(20) +#define BIT_FW_INIT_RDY BIT(15) +#define BIT_FW_DW_RDY BIT(14) +#define BIT_RPWM_TOGGLE BIT(7) +#define BIT_DMEM_CHKSUM_OK BIT(6) +#define BIT_DMEM_DW_OK BIT(5) +#define BIT_IMEM_CHKSUM_OK BIT(4) +#define BIT_IMEM_DW_OK BIT(3) +#define BIT_IMEM_BOOT_LOAD_CHECKSUM_OK BIT(2) +#define BIT_MCUFWDL_EN BIT(0) +#define BIT_CHECK_SUM_OK (BIT(4) | BIT(6)) +#define FW_READY (BIT_FW_INIT_RDY | BIT_FW_DW_RDY | \ + BIT_IMEM_DW_OK | BIT_DMEM_DW_OK | \ + BIT_CHECK_SUM_OK) +#define FW_READY_MASK 0xffff + +#define REG_WLRF1 0x00EC +#define REG_SYS_CFG1 0x00F0 +#define BIT_RTL_ID BIT(23) +#define BIT_RF_TYPE_ID BIT(27) +#define BIT_SHIFT_VENDOR_ID 16 +#define BIT_MASK_VENDOR_ID 0xf +#define BIT_VENDOR_ID(x) (((x) & BIT_MASK_VENDOR_ID) << BIT_SHIFT_VENDOR_ID) +#define BITS_VENDOR_ID (BIT_MASK_VENDOR_ID << BIT_SHIFT_VENDOR_ID) +#define BIT_CLEAR_VENDOR_ID(x) ((x) & (~BITS_VENDOR_ID)) +#define BIT_GET_VENDOR_ID(x) (((x) >> BIT_SHIFT_VENDOR_ID) & BIT_MASK_VENDOR_ID) +#define BIT_SHIFT_CHIP_VER 12 +#define BIT_MASK_CHIP_VER 0xf +#define BIT_CHIP_VER(x) (((x) & BIT_MASK_CHIP_VER) << BIT_SHIFT_CHIP_VER) +#define BITS_CHIP_VER (BIT_MASK_CHIP_VER << BIT_SHIFT_CHIP_VER) +#define BIT_CLEAR_CHIP_VER(x) ((x) & (~BITS_CHIP_VER)) +#define BIT_GET_CHIP_VER(x) (((x) >> BIT_SHIFT_CHIP_VER) & BIT_MASK_CHIP_VER) +#define REG_SYS_STATUS1 0x00F4 +#define REG_SYS_STATUS2 0x00F8 +#define REG_SYS_CFG2 0x00FC +#define REG_WLRF1 0x00EC +#define BIT_WLRF1_BBRF_EN (BIT(24) | BIT(25) | BIT(26)) +#define REG_CR 0x0100 +#define BIT_32K_CAL_TMR_EN BIT(10) +#define BIT_MAC_SEC_EN BIT(9) +#define BIT_ENSWBCN BIT(8) +#define BIT_MACRXEN BIT(7) +#define BIT_MACTXEN BIT(6) +#define BIT_SCHEDULE_EN BIT(5) +#define BIT_PROTOCOL_EN BIT(4) +#define BIT_RXDMA_EN BIT(3) +#define BIT_TXDMA_EN BIT(2) +#define BIT_HCI_RXDMA_EN BIT(1) +#define BIT_HCI_TXDMA_EN BIT(0) +#define MAC_TRX_ENABLE (BIT_HCI_TXDMA_EN | BIT_HCI_RXDMA_EN | BIT_TXDMA_EN | \ + BIT_RXDMA_EN | BIT_PROTOCOL_EN | BIT_SCHEDULE_EN | \ + BIT_MACTXEN | BIT_MACRXEN) +#define BIT_SHIFT_TXDMA_VOQ_MAP 4 +#define BIT_MASK_TXDMA_VOQ_MAP 0x3 +#define BIT_TXDMA_VOQ_MAP(x) \ + (((x) & BIT_MASK_TXDMA_VOQ_MAP) << BIT_SHIFT_TXDMA_VOQ_MAP) +#define BIT_SHIFT_TXDMA_VIQ_MAP 6 +#define BIT_MASK_TXDMA_VIQ_MAP 0x3 +#define BIT_TXDMA_VIQ_MAP(x) \ + (((x) & BIT_MASK_TXDMA_VIQ_MAP) << BIT_SHIFT_TXDMA_VIQ_MAP) +#define REG_TXDMA_PQ_MAP 0x010C +#define BIT_SHIFT_TXDMA_BEQ_MAP 8 +#define BIT_MASK_TXDMA_BEQ_MAP 0x3 +#define BIT_TXDMA_BEQ_MAP(x) \ + (((x) & BIT_MASK_TXDMA_BEQ_MAP) << BIT_SHIFT_TXDMA_BEQ_MAP) +#define BIT_SHIFT_TXDMA_BKQ_MAP 10 +#define BIT_MASK_TXDMA_BKQ_MAP 0x3 +#define BIT_TXDMA_BKQ_MAP(x) \ + (((x) & BIT_MASK_TXDMA_BKQ_MAP) << BIT_SHIFT_TXDMA_BKQ_MAP) +#define BIT_SHIFT_TXDMA_MGQ_MAP 12 +#define BIT_MASK_TXDMA_MGQ_MAP 0x3 +#define BIT_TXDMA_MGQ_MAP(x) \ + (((x) & BIT_MASK_TXDMA_MGQ_MAP) << BIT_SHIFT_TXDMA_MGQ_MAP) +#define BIT_SHIFT_TXDMA_HIQ_MAP 14 +#define BIT_MASK_TXDMA_HIQ_MAP 0x3 +#define BIT_TXDMA_HIQ_MAP(x) \ + (((x) & BIT_MASK_TXDMA_HIQ_MAP) << BIT_SHIFT_TXDMA_HIQ_MAP) +#define BIT_SHIFT_TXSC_40M 4 +#define BIT_MASK_TXSC_40M 0xf +#define BIT_TXSC_40M(x) \ + (((x) & BIT_MASK_TXSC_40M) << BIT_SHIFT_TXSC_40M) +#define BIT_SHIFT_TXSC_20M 0 +#define BIT_MASK_TXSC_20M 0xf +#define BIT_TXSC_20M(x) \ + (((x) & BIT_MASK_TXSC_20M) << BIT_SHIFT_TXSC_20M) +#define BIT_SHIFT_MAC_CLK_SEL 20 +#define MAC_CLK_HW_DEF_80M 0 +#define MAC_CLK_HW_DEF_40M 1 +#define MAC_CLK_HW_DEF_20M 2 +#define MAC_CLK_SPEED 80 + +#define REG_CR 0x0100 +#define REG_TRXFF_BNDY 0x0114 +#define REG_RXFF_BNDY 0x011C +#define REG_PKTBUF_DBG_CTRL 0x0140 +#define REG_C2HEVT 0x01A0 +#define REG_HMETFR 0x01CC +#define REG_HMEBOX0 0x01D0 +#define REG_HMEBOX1 0x01D4 +#define REG_HMEBOX2 0x01D8 +#define REG_HMEBOX3 0x01DC +#define REG_HMEBOX0_EX 0x01F0 +#define REG_HMEBOX1_EX 0x01F4 +#define REG_HMEBOX2_EX 0x01F8 +#define REG_HMEBOX3_EX 0x01FC + +#define REG_FIFOPAGE_CTRL_2 0x0204 +#define BIT_BCN_VALID_V1 BIT(15) +#define BIT_MASK_BCN_HEAD_1_V1 0xfff +#define REG_AUTO_LLT_V1 0x0208 +#define BIT_AUTO_INIT_LLT_V1 BIT(0) +#define REG_TXDMA_OFFSET_CHK 0x020C +#define REG_TXDMA_STATUS 0x0210 +#define BTI_PAGE_OVF BIT(2) +#define REG_RQPN_CTRL_1 0x0228 +#define REG_RQPN_CTRL_2 0x022C +#define BIT_LD_RQPN BIT(31) +#define REG_FIFOPAGE_INFO_1 0x0230 +#define REG_FIFOPAGE_INFO_2 0x0234 +#define REG_FIFOPAGE_INFO_3 0x0238 +#define REG_FIFOPAGE_INFO_4 0x023C +#define REG_FIFOPAGE_INFO_5 0x0240 +#define REG_H2C_HEAD 0x0244 +#define REG_H2C_TAIL 0x0248 +#define REG_H2C_READ_ADDR 0x024C +#define REG_H2C_INFO 0x0254 + +#define REG_FWHW_TXQ_CTRL 0x0420 +#define BIT_EN_BCNQ_DL BIT(22) +#define BIT_EN_WR_FREE_TAIL BIT(20) +#define REG_BCNQ_BDNY_V1 0x0424 +#define REG_LIFETIME_EN 0x0426 +#define BIT_BA_PARSER_EN BIT(5) +#define REG_SPEC_SIFS 0x0428 +#define REG_DARFRC 0x0430 +#define REG_DARFRCH 0x0434 +#define REG_RARFRCH 0x043C +#define REG_ARFR0 0x0444 +#define REG_ARFRH0 0x0448 +#define REG_ARFR1_V1 0x044C +#define REG_ARFRH1_V1 0x0450 +#define REG_CCK_CHECK 0x0454 +#define BIT_CHECK_CCK_EN BIT(7) +#define REG_AMPDU_MAX_TIME_V1 0x0455 +#define REG_BCNQ1_BDNY_V1 0x0456 +#define REG_TX_HANG_CTRL 0x045E +#define BIT_EN_EOF_V1 BIT(2) +#define REG_DATA_SC 0x0483 +#define REG_ARFR4 0x049C +#define REG_ARFRH4 0x04A0 +#define REG_ARFR5 0x04A4 +#define REG_ARFRH5 0x04A8 +#define REG_SW_AMPDU_BURST_MODE_CTRL 0x04BC +#define BIT_PRE_TX_CMD BIT(6) +#define REG_PROT_MODE_CTRL 0x04C8 +#define REG_BAR_MODE_CTRL 0x04CC +#define REG_PRECNT_CTRL 0x04E5 +#define BIT_EN_PRECNT BIT(11) + +#define REG_EDCA_VO_PARAM 0x0500 +#define REG_EDCA_VI_PARAM 0x0504 +#define REG_EDCA_BE_PARAM 0x0508 +#define REG_EDCA_BK_PARAM 0x050C +#define REG_PIFS 0x0512 +#define REG_SIFS 0x0514 +#define BIT_SHIFT_SIFS_OFDM_CTX 8 +#define BIT_SHIFT_SIFS_CCK_TRX 16 +#define BIT_SHIFT_SIFS_OFDM_TRX 24 +#define REG_SLOT 0x051B +#define REG_TX_PTCL_CTRL 0x0520 +#define BIT_SIFS_BK_EN BIT(12) +#define REG_TXPAUSE 0x0522 +#define REG_RD_CTRL 0x0524 +#define BIT_DIS_TXOP_CFE BIT(10) +#define BIT_DIS_LSIG_CFE BIT(9) +#define BIT_DIS_STBC_CFE BIT(8) +#define REG_TBTT_PROHIBIT 0x0540 +#define BIT_SHIFT_TBTT_HOLD_TIME_AP 8 +#define REG_RD_NAV_NXT 0x0544 +#define REG_BCN_CTRL 0x0550 +#define BIT_DIS_TSF_UDT BIT(4) +#define BIT_EN_BCN_FUNCTION BIT(3) +#define REG_BCN_CTRL_CLINT0 0x0551 +#define REG_DRVERLYINT 0x0558 +#define REG_BCNDMATIM 0x0559 +#define REG_USTIME_TSF 0x055C +#define REG_BCN_MAX_ERR 0x055D +#define REG_RXTSF_OFFSET_CCK 0x055E +#define REG_MISC_CTRL 0x0577 +#define BIT_EN_FREE_CNT BIT(3) +#define BIT_DIS_SECOND_CCA (BIT(0) | BIT(1)) +#define REG_TIMER0_SRC_SEL 0x05B4 +#define BIT_TSFT_SEL_TIMER0 (BIT(4) | BIT(5) | BIT(6)) + +#define REG_TCR 0x0604 +#define REG_RCR 0x0608 +#define BIT_APP_FCS BIT(31) +#define BIT_APP_MIC BIT(30) +#define BIT_APP_ICV BIT(29) +#define BIT_APP_PHYSTS BIT(28) +#define BIT_APP_BASSN BIT(27) +#define BIT_VHT_DACK BIT(26) +#define BIT_TCPOFLD_EN BIT(25) +#define BIT_ENMBID BIT(24) +#define BIT_LSIGEN BIT(23) +#define BIT_MFBEN BIT(22) +#define BIT_DISCHKPPDLLEN BIT(21) +#define BIT_PKTCTL_DLEN BIT(20) +#define BIT_TIM_PARSER_EN BIT(18) +#define BIT_BC_MD_EN BIT(17) +#define BIT_UC_MD_EN BIT(16) +#define BIT_RXSK_PERPKT BIT(15) +#define BIT_HTC_LOC_CTRL BIT(14) +#define BIT_RPFM_CAM_ENABLE BIT(12) +#define BIT_TA_BCN BIT(11) +#define BIT_DISDECMYPKT BIT(10) +#define BIT_AICV BIT(9) +#define BIT_ACRC32 BIT(8) +#define BIT_CBSSID_BCN BIT(7) +#define BIT_CBSSID_DATA BIT(6) +#define BIT_APWRMGT BIT(5) +#define BIT_ADD3 BIT(4) +#define BIT_AB BIT(3) +#define BIT_AM BIT(2) +#define BIT_APM BIT(1) +#define BIT_AAP BIT(0) +#define REG_RX_PKT_LIMIT 0x060C +#define REG_RX_DRVINFO_SZ 0x060F +#define BIT_APP_PHYSTS BIT(28) +#define REG_USTIME_EDCA 0x0638 +#define REG_ACKTO_CCK 0x0639 +#define REG_RESP_SIFS_CCK 0x063C +#define REG_RESP_SIFS_OFDM 0x063E +#define REG_ACKTO 0x0640 +#define REG_EIFS 0x0642 +#define REG_NAV_CTRL 0x0650 +#define REG_WMAC_TRXPTCL_CTL 0x0668 +#define BIT_RFMOD (BIT(7) | BIT(8)) +#define BIT_RFMOD_80M BIT(8) +#define BIT_RFMOD_40M BIT(7) +#define REG_WMAC_TRXPTCL_CTL_H 0x066C +#define REG_RXFLTMAP0 0x06A0 +#define REG_RXFLTMAP1 0x06A2 +#define REG_RXFLTMAP2 0x06A4 +#define REG_BBPSF_CTRL 0x06DC + +#define REG_WMAC_OPTION_FUNCTION 0x07D0 +#define REG_WMAC_OPTION_FUNCTION_1 0x07D4 + +#define REG_ANAPAR_XTAL_0 0x1040 +#define REG_CPU_DMEM_CON 0x1080 +#define BIT_WL_PLATFORM_RST BIT(16) +#define BIT_WL_SECURITY_CLK BIT(15) +#define BIT_DDMA_EN BIT(8) + +#define REG_H2C_PKT_READADDR 0x10D0 +#define REG_H2C_PKT_WRITEADDR 0x10D4 +#define REG_FW_DBG7 0x10FC +#define FW_KEY_MASK 0xffffff00 + +#define REG_CR_EXT 0x1100 + +#define REG_DDMA_CH0SA 0x1200 +#define REG_DDMA_CH0DA 0x1204 +#define REG_DDMA_CH0CTRL 0x1208 +#define BIT_DDMACH0_OWN BIT(31) +#define BIT_DDMACH0_CHKSUM_EN BIT(29) +#define BIT_DDMACH0_CHKSUM_STS BIT(27) +#define BIT_DDMACH0_RESET_CHKSUM_STS BIT(25) +#define BIT_DDMACH0_CHKSUM_CONT BIT(24) +#define BIT_MASK_DDMACH0_DLEN 0x3ffff + +#define REG_H2CQ_CSR 0x1330 +#define BIT_H2CQ_FULL BIT(31) +#define REG_FAST_EDCA_VOVI_SETTING 0x1448 +#define REG_FAST_EDCA_BEBK_SETTING 0x144C + +#define REG_RXPSF_CTRL 0x1610 +#define BIT_RXGCK_FIFOTHR_EN BIT(28) + +#define BIT_SHIFT_RXGCK_VHT_FIFOTHR 26 +#define BIT_MASK_RXGCK_VHT_FIFOTHR 0x3 +#define BIT_RXGCK_VHT_FIFOTHR(x) \ + (((x) & BIT_MASK_RXGCK_VHT_FIFOTHR) << BIT_SHIFT_RXGCK_VHT_FIFOTHR) +#define BITS_RXGCK_VHT_FIFOTHR \ + (BIT_MASK_RXGCK_VHT_FIFOTHR << BIT_SHIFT_RXGCK_VHT_FIFOTHR) + +#define BIT_SHIFT_RXGCK_HT_FIFOTHR 24 +#define BIT_MASK_RXGCK_HT_FIFOTHR 0x3 +#define BIT_RXGCK_HT_FIFOTHR(x) \ + (((x) & BIT_MASK_RXGCK_HT_FIFOTHR) << BIT_SHIFT_RXGCK_HT_FIFOTHR) +#define BITS_RXGCK_HT_FIFOTHR \ + (BIT_MASK_RXGCK_HT_FIFOTHR << BIT_SHIFT_RXGCK_HT_FIFOTHR) + +#define BIT_SHIFT_RXGCK_OFDM_FIFOTHR 22 +#define BIT_MASK_RXGCK_OFDM_FIFOTHR 0x3 +#define BIT_RXGCK_OFDM_FIFOTHR(x) \ + (((x) & BIT_MASK_RXGCK_OFDM_FIFOTHR) << BIT_SHIFT_RXGCK_OFDM_FIFOTHR) +#define BITS_RXGCK_OFDM_FIFOTHR \ + (BIT_MASK_RXGCK_OFDM_FIFOTHR << BIT_SHIFT_RXGCK_OFDM_FIFOTHR) + +#define BIT_SHIFT_RXGCK_CCK_FIFOTHR 20 +#define BIT_MASK_RXGCK_CCK_FIFOTHR 0x3 +#define BIT_RXGCK_CCK_FIFOTHR(x) \ + (((x) & BIT_MASK_RXGCK_CCK_FIFOTHR) << BIT_SHIFT_RXGCK_CCK_FIFOTHR) +#define BITS_RXGCK_CCK_FIFOTHR \ + (BIT_MASK_RXGCK_CCK_FIFOTHR << BIT_SHIFT_RXGCK_CCK_FIFOTHR) + +#define BIT_RXGCK_OFDMCCA_EN BIT(16) + +#define BIT_SHIFT_RXPSF_PKTLENTHR 13 +#define BIT_MASK_RXPSF_PKTLENTHR 0x7 +#define BIT_RXPSF_PKTLENTHR(x) \ + (((x) & BIT_MASK_RXPSF_PKTLENTHR) << BIT_SHIFT_RXPSF_PKTLENTHR) +#define BITS_RXPSF_PKTLENTHR \ + (BIT_MASK_RXPSF_PKTLENTHR << BIT_SHIFT_RXPSF_PKTLENTHR) +#define BIT_CLEAR_RXPSF_PKTLENTHR(x) ((x) & (~BITS_RXPSF_PKTLENTHR)) +#define BIT_SET_RXPSF_PKTLENTHR(x, v) \ + (BIT_CLEAR_RXPSF_PKTLENTHR(x) | BIT_RXPSF_PKTLENTHR(v)) + +#define BIT_RXPSF_CTRLEN BIT(12) +#define BIT_RXPSF_VHTCHKEN BIT(11) +#define BIT_RXPSF_HTCHKEN BIT(10) +#define BIT_RXPSF_OFDMCHKEN BIT(9) +#define BIT_RXPSF_CCKCHKEN BIT(8) +#define BIT_RXPSF_OFDMRST BIT(7) +#define BIT_RXPSF_CCKRST BIT(6) +#define BIT_RXPSF_MHCHKEN BIT(5) +#define BIT_RXPSF_CONT_ERRCHKEN BIT(4) +#define BIT_RXPSF_ALL_ERRCHKEN BIT(3) + +#define BIT_SHIFT_RXPSF_ERRTHR 0 +#define BIT_MASK_RXPSF_ERRTHR 0x7 +#define BIT_RXPSF_ERRTHR(x) \ + (((x) & BIT_MASK_RXPSF_ERRTHR) << BIT_SHIFT_RXPSF_ERRTHR) +#define BITS_RXPSF_ERRTHR (BIT_MASK_RXPSF_ERRTHR << BIT_SHIFT_RXPSF_ERRTHR) +#define BIT_CLEAR_RXPSF_ERRTHR(x) ((x) & (~BITS_RXPSF_ERRTHR)) +#define BIT_GET_RXPSF_ERRTHR(x) \ + (((x) >> BIT_SHIFT_RXPSF_ERRTHR) & BIT_MASK_RXPSF_ERRTHR) +#define BIT_SET_RXPSF_ERRTHR(x, v) \ + (BIT_CLEAR_RXPSF_ERRTHR(x) | BIT_RXPSF_ERRTHR(v)) + +#define REG_RXPSF_TYPE_CTRL 0x1614 +#define REG_GENERAL_OPTION 0x1664 +#define BIT_DUMMY_FCS_READY_MASK_EN BIT(9) + +#define REG_WL2LTECOEX_INDIRECT_ACCESS_CTRL_V1 0x1700 +#define REG_WL2LTECOEX_INDIRECT_ACCESS_WRITE_DATA_V1 0x1704 +#define REG_WL2LTECOEX_INDIRECT_ACCESS_READ_DATA_V1 0x1708 +#define LTECOEX_READY BIT(29) +#define LTECOEX_ACCESS_CTRL REG_WL2LTECOEX_INDIRECT_ACCESS_CTRL_V1 +#define LTECOEX_WRITE_DATA REG_WL2LTECOEX_INDIRECT_ACCESS_WRITE_DATA_V1 +#define LTECOEX_READ_DATA REG_WL2LTECOEX_INDIRECT_ACCESS_READ_DATA_V1 + +#define RF_DTXLOK 0x08 +#define RF_CFGCH 0x18 +#define RF_LUTWA 0x33 +#define RF_LUTWD1 0x3e +#define RF_LUTWD0 0x3f +#define RF_XTALX2 0xb8 +#define RF_MALSEL 0xbe +#define RF_LUTDBG 0xdf +#define RF_LUTWE2 0xee +#define RF_LUTWE 0xef + +#endif diff --git a/drivers/net/wireless/realtek/rtw88/regd.c b/drivers/net/wireless/realtek/rtw88/regd.c new file mode 100644 index 0000000000000000000000000000000000000000..e7750a833a8e20eabe80e65b9fe9d71ce6464e2f --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/regd.c @@ -0,0 +1,391 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#include "main.h" +#include "regd.h" +#include "debug.h" +#include "phy.h" + +#define COUNTRY_CHPLAN_ENT(_alpha2, _chplan, _txpwr_regd) \ + {.alpha2 = (_alpha2), \ + .chplan = (_chplan), \ + .txpwr_regd = (_txpwr_regd) \ + } + +/* If country code is not correctly defined in efuse, + * use worldwide country code and txpwr regd. + */ +static const struct rtw_regulatory rtw_defined_chplan = + COUNTRY_CHPLAN_ENT("00", RTW_CHPLAN_REALTEK_DEFINE, RTW_REGD_WW); + +static const struct rtw_regulatory all_chplan_map[] = { + COUNTRY_CHPLAN_ENT("AD", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("AE", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("AF", RTW_CHPLAN_ETSI1_ETSI4, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("AG", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("AI", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("AL", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("AM", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("AN", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("AO", RTW_CHPLAN_WORLD_ETSI6, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("AQ", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("AR", RTW_CHPLAN_FCC2_FCC7, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("AS", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("AT", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("AU", RTW_CHPLAN_WORLD_ACMA1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("AW", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("AZ", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("BA", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("BB", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("BD", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("BE", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("BF", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("BG", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("BH", RTW_CHPLAN_WORLD_ETSI6, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("BI", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("BJ", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("BN", RTW_CHPLAN_WORLD_ETSI6, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("BO", RTW_CHPLAN_WORLD_FCC7, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("BR", RTW_CHPLAN_FCC2_FCC1, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("BS", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("BW", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("BY", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("BZ", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("CA", RTW_CHPLAN_IC1_IC2, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("CC", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("CD", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("CF", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("CG", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("CH", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("CI", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("CK", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("CL", RTW_CHPLAN_WORLD_CHILE1, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("CM", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("CN", RTW_CHPLAN_WORLD_ETSI7, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("CO", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("CR", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("CV", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("CX", RTW_CHPLAN_WORLD_ACMA1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("CY", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("CZ", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("DE", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("DJ", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("DK", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("DM", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("DO", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("DZ", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("EC", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("EE", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("EG", RTW_CHPLAN_WORLD_ETSI6, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("EH", RTW_CHPLAN_WORLD_ETSI6, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("ER", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("ES", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("ET", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("FI", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("FJ", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("FK", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("FM", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("FO", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("FR", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("GA", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("GB", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("GD", RTW_CHPLAN_FCC1_FCC7, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("GE", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("GF", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("GG", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("GH", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("GI", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("GL", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("GM", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("GN", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("GP", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("GQ", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("GR", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("GS", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("GT", RTW_CHPLAN_FCC2_FCC7, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("GU", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("GW", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("GY", RTW_CHPLAN_FCC1_NCC3, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("HK", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("HM", RTW_CHPLAN_WORLD_ACMA1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("HN", RTW_CHPLAN_WORLD_FCC5, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("HR", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("HT", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("HU", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("ID", RTW_CHPLAN_ETSI1_ETSI12, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("IE", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("IL", RTW_CHPLAN_WORLD_ETSI6, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("IM", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("IN", RTW_CHPLAN_WORLD_ETSI7, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("IQ", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("IR", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("IS", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("IT", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("JE", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("JM", RTW_CHPLAN_WORLD_ETSI10, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("JO", RTW_CHPLAN_WORLD_ETSI8, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("JP", RTW_CHPLAN_MKK1_MKK1, RTW_REGD_MKK), + COUNTRY_CHPLAN_ENT("KE", RTW_CHPLAN_WORLD_ETSI6, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("KG", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("KH", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("KI", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("KN", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("KR", RTW_CHPLAN_KCC1_KCC2, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("KW", RTW_CHPLAN_WORLD_ETSI6, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("KY", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("KZ", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("LA", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("LB", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("LC", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("LI", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("LK", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("LR", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("LS", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("LT", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("LU", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("LV", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("LY", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("MA", RTW_CHPLAN_WORLD_ETSI6, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("MC", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("MD", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("ME", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("MF", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("MG", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("MH", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("MK", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("ML", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("MM", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("MN", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("MO", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("MP", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("MQ", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("MR", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("MS", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("MT", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("MU", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("MV", RTW_CHPLAN_WORLD_ETSI6, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("MW", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("MX", RTW_CHPLAN_FCC2_FCC7, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("MY", RTW_CHPLAN_WORLD_ETSI20, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("MZ", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("NA", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("NC", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("NE", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("NF", RTW_CHPLAN_WORLD_ACMA1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("NG", RTW_CHPLAN_WORLD_ETSI20, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("NI", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("NL", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("NO", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("NP", RTW_CHPLAN_WORLD_ETSI6, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("NR", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("NU", RTW_CHPLAN_WORLD_ACMA1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("NZ", RTW_CHPLAN_WORLD_ACMA1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("OM", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("PA", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("PE", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("PF", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("PG", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("PH", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("PK", RTW_CHPLAN_WORLD_ETSI10, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("PL", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("PM", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("PR", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("PT", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("PW", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("PY", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("QA", RTW_CHPLAN_WORLD_ETSI10, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("RE", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("RO", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("RS", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("RU", RTW_CHPLAN_WORLD_ETSI14, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("RW", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("SA", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("SB", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("SC", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("SE", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("SG", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("SH", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("SI", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("SJ", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("SK", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("SL", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("SM", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("SN", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("SO", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("SR", RTW_CHPLAN_FCC2_FCC17, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("ST", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("SV", RTW_CHPLAN_WORLD_FCC3, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("SX", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("SZ", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("TC", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("TD", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("TF", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("TG", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("TH", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("TJ", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("TK", RTW_CHPLAN_WORLD_ACMA1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("TM", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("TN", RTW_CHPLAN_WORLD_ETSI6, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("TO", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("TR", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("TT", RTW_CHPLAN_ETSI1_ETSI4, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("TW", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("TZ", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("UA", RTW_CHPLAN_WORLD_ETSI3, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("UG", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("US", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("UY", RTW_CHPLAN_WORLD_FCC3, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("UZ", RTW_CHPLAN_WORLD_ETSI6, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("VA", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("VC", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("VE", RTW_CHPLAN_WORLD_FCC3, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("VI", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("VN", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("VU", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("WF", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("WS", RTW_CHPLAN_FCC2_FCC11, RTW_REGD_FCC), + COUNTRY_CHPLAN_ENT("YE", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("YT", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("ZA", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("ZM", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), + COUNTRY_CHPLAN_ENT("ZW", RTW_CHPLAN_WORLD_ETSI1, RTW_REGD_ETSI), +}; + +static void rtw_regd_apply_beaconing_flags(struct wiphy *wiphy, + enum nl80211_reg_initiator initiator) +{ + enum nl80211_band band; + struct ieee80211_supported_band *sband; + const struct ieee80211_reg_rule *reg_rule; + struct ieee80211_channel *ch; + unsigned int i; + + for (band = 0; band < NUM_NL80211_BANDS; band++) { + if (!wiphy->bands[band]) + continue; + + sband = wiphy->bands[band]; + for (i = 0; i < sband->n_channels; i++) { + ch = &sband->channels[i]; + + reg_rule = freq_reg_info(wiphy, + MHZ_TO_KHZ(ch->center_freq)); + if (IS_ERR(reg_rule)) + continue; + + ch->flags &= ~IEEE80211_CHAN_DISABLED; + + if (!(reg_rule->flags & NL80211_RRF_NO_IR)) + ch->flags &= ~IEEE80211_CHAN_NO_IR; + } + } +} + +static void rtw_regd_apply_hw_cap_flags(struct wiphy *wiphy) +{ + struct ieee80211_hw *hw = wiphy_to_ieee80211_hw(wiphy); + struct ieee80211_supported_band *sband; + struct ieee80211_channel *ch; + struct rtw_dev *rtwdev = hw->priv; + struct rtw_efuse *efuse = &rtwdev->efuse; + int i; + + if (efuse->hw_cap.bw & BIT(RTW_CHANNEL_WIDTH_80)) + return; + + sband = wiphy->bands[NL80211_BAND_2GHZ]; + if (!sband) + goto out_5g; + + for (i = 0; i < sband->n_channels; i++) { + ch = &sband->channels[i]; + ch->flags |= IEEE80211_CHAN_NO_80MHZ; + } + +out_5g: + sband = wiphy->bands[NL80211_BAND_5GHZ]; + if (!sband) + return; + + for (i = 0; i < sband->n_channels; i++) { + ch = &sband->channels[i]; + ch->flags |= IEEE80211_CHAN_NO_80MHZ; + } +} + +static void rtw_regd_apply_world_flags(struct wiphy *wiphy, + enum nl80211_reg_initiator initiator) +{ + rtw_regd_apply_beaconing_flags(wiphy, initiator); +} + +static struct rtw_regulatory rtw_regd_find_reg_by_name(char *alpha2) +{ + unsigned int i; + + for (i = 0; i < ARRAY_SIZE(all_chplan_map); i++) { + if (!memcmp(all_chplan_map[i].alpha2, alpha2, 2)) + return all_chplan_map[i]; + } + + return rtw_defined_chplan; +} + +static int rtw_regd_notifier_apply(struct rtw_dev *rtwdev, + struct wiphy *wiphy, + struct regulatory_request *request) +{ + if (request->initiator == NL80211_REGDOM_SET_BY_USER) + return 0; + rtwdev->regd = rtw_regd_find_reg_by_name(request->alpha2); + rtw_regd_apply_world_flags(wiphy, request->initiator); + + return 0; +} + +static int +rtw_regd_init_wiphy(struct rtw_regulatory *reg, struct wiphy *wiphy, + void (*reg_notifier)(struct wiphy *wiphy, + struct regulatory_request *request)) +{ + wiphy->reg_notifier = reg_notifier; + + wiphy->regulatory_flags &= ~REGULATORY_CUSTOM_REG; + wiphy->regulatory_flags &= ~REGULATORY_STRICT_REG; + wiphy->regulatory_flags &= ~REGULATORY_DISABLE_BEACON_HINTS; + + rtw_regd_apply_hw_cap_flags(wiphy); + + return 0; +} + +int rtw_regd_init(struct rtw_dev *rtwdev, + void (*reg_notifier)(struct wiphy *wiphy, + struct regulatory_request *request)) +{ + struct wiphy *wiphy = rtwdev->hw->wiphy; + + if (!wiphy) + return -EINVAL; + + rtwdev->regd = rtw_regd_find_reg_by_name(rtwdev->efuse.country_code); + rtw_regd_init_wiphy(&rtwdev->regd, wiphy, reg_notifier); + + return 0; +} + +void rtw_regd_notifier(struct wiphy *wiphy, struct regulatory_request *request) +{ + struct ieee80211_hw *hw = wiphy_to_ieee80211_hw(wiphy); + struct rtw_dev *rtwdev = hw->priv; + struct rtw_hal *hal = &rtwdev->hal; + + rtw_regd_notifier_apply(rtwdev, wiphy, request); + rtw_dbg(rtwdev, RTW_DBG_REGD, + "get alpha2 %c%c from initiator %d, mapping to chplan 0x%x, txregd %d\n", + request->alpha2[0], request->alpha2[1], request->initiator, + rtwdev->regd.chplan, rtwdev->regd.txpwr_regd); + + rtw_phy_set_tx_power_level(rtwdev, hal->current_channel); +} diff --git a/drivers/net/wireless/realtek/rtw88/regd.h b/drivers/net/wireless/realtek/rtw88/regd.h new file mode 100644 index 0000000000000000000000000000000000000000..7784bb6d3ba7cef7178b8fd36538aaa487b2f5af --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/regd.h @@ -0,0 +1,67 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#ifndef __RTW_REGD_H_ +#define __RTW_REGD_H_ + +#define IEEE80211_CHAN_NO_IBSS IEEE80211_CHAN_NO_IR +#define IEEE80211_CHAN_PASSIVE_SCAN IEEE80211_CHAN_NO_IR +enum rtw_chplan_id { + RTW_CHPLAN_WORLD_ETSI1 = 0x26, + RTW_CHPLAN_MKK1_MKK1 = 0x27, + RTW_CHPLAN_IC1_IC2 = 0x2B, + RTW_CHPLAN_WORLD_CHILE1 = 0x2D, + RTW_CHPLAN_WORLD_FCC3 = 0x30, + RTW_CHPLAN_WORLD_FCC5 = 0x32, + RTW_CHPLAN_FCC1_FCC7 = 0x34, + RTW_CHPLAN_WORLD_ETSI3 = 0x36, + RTW_CHPLAN_ETSI1_ETSI12 = 0x3D, + RTW_CHPLAN_KCC1_KCC2 = 0x3E, + RTW_CHPLAN_ETSI1_ETSI4 = 0x42, + RTW_CHPLAN_FCC1_NCC3 = 0x44, + RTW_CHPLAN_WORLD_ACMA1 = 0x45, + RTW_CHPLAN_WORLD_ETSI6 = 0x47, + RTW_CHPLAN_WORLD_ETSI7 = 0x48, + RTW_CHPLAN_WORLD_ETSI8 = 0x49, + RTW_CHPLAN_WORLD_ETSI10 = 0x51, + RTW_CHPLAN_WORLD_ETSI14 = 0x59, + RTW_CHPLAN_FCC2_FCC7 = 0x61, + RTW_CHPLAN_FCC2_FCC1 = 0x62, + RTW_CHPLAN_WORLD_FCC7 = 0x73, + RTW_CHPLAN_FCC2_FCC17 = 0x74, + RTW_CHPLAN_WORLD_ETSI20 = 0x75, + RTW_CHPLAN_FCC2_FCC11 = 0x76, + RTW_CHPLAN_REALTEK_DEFINE = 0x7f, +}; + +struct country_code_to_enum_rd { + u16 countrycode; + const char *iso_name; +}; + +enum country_code_type { + COUNTRY_CODE_FCC = 0, + COUNTRY_CODE_IC = 1, + COUNTRY_CODE_ETSI = 2, + COUNTRY_CODE_SPAIN = 3, + COUNTRY_CODE_FRANCE = 4, + COUNTRY_CODE_MKK = 5, + COUNTRY_CODE_MKK1 = 6, + COUNTRY_CODE_ISRAEL = 7, + COUNTRY_CODE_TELEC = 8, + COUNTRY_CODE_MIC = 9, + COUNTRY_CODE_GLOBAL_DOMAIN = 10, + COUNTRY_CODE_WORLD_WIDE_13 = 11, + COUNTRY_CODE_TELEC_NETGEAR = 12, + COUNTRY_CODE_WORLD_WIDE_13_5G_ALL = 13, + + /* new channel plan above this */ + COUNTRY_CODE_MAX +}; + +int rtw_regd_init(struct rtw_dev *rtwdev, + void (*reg_notifier)(struct wiphy *wiphy, + struct regulatory_request *request)); +void rtw_regd_notifier(struct wiphy *wiphy, struct regulatory_request *request); +#endif diff --git a/drivers/net/wireless/realtek/rtw88/rtw8822b.c b/drivers/net/wireless/realtek/rtw88/rtw8822b.c new file mode 100644 index 0000000000000000000000000000000000000000..1172f6c0605b328a32aafe8c055cf64ac87329d0 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/rtw8822b.c @@ -0,0 +1,1594 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#include "main.h" +#include "fw.h" +#include "tx.h" +#include "rx.h" +#include "phy.h" +#include "rtw8822b.h" +#include "rtw8822b_table.h" +#include "mac.h" +#include "reg.h" +#include "debug.h" + +static void rtw8822b_config_trx_mode(struct rtw_dev *rtwdev, u8 tx_path, + u8 rx_path, bool is_tx2_path); + +static void rtw8822be_efuse_parsing(struct rtw_efuse *efuse, + struct rtw8822b_efuse *map) +{ + ether_addr_copy(efuse->addr, map->e.mac_addr); +} + +static int rtw8822b_read_efuse(struct rtw_dev *rtwdev, u8 *log_map) +{ + struct rtw_efuse *efuse = &rtwdev->efuse; + struct rtw8822b_efuse *map; + int i; + + map = (struct rtw8822b_efuse *)log_map; + + efuse->rfe_option = map->rfe_option; + efuse->crystal_cap = map->xtal_k; + efuse->pa_type_2g = map->pa_type; + efuse->pa_type_5g = map->pa_type; + efuse->lna_type_2g = map->lna_type_2g[0]; + efuse->lna_type_5g = map->lna_type_5g[0]; + efuse->channel_plan = map->channel_plan; + efuse->country_code[0] = map->country_code[0]; + efuse->country_code[1] = map->country_code[1]; + efuse->bt_setting = map->rf_bt_setting; + efuse->regd = map->rf_board_option & 0x7; + + for (i = 0; i < 4; i++) + efuse->txpwr_idx_table[i] = map->txpwr_idx_table[i]; + + switch (rtw_hci_type(rtwdev)) { + case RTW_HCI_TYPE_PCIE: + rtw8822be_efuse_parsing(efuse, map); + break; + default: + /* unsupported now */ + return -ENOTSUPP; + } + + return 0; +} + +static void rtw8822b_phy_rfe_init(struct rtw_dev *rtwdev) +{ + /* chip top mux */ + rtw_write32_mask(rtwdev, 0x64, BIT(29) | BIT(28), 0x3); + rtw_write32_mask(rtwdev, 0x4c, BIT(26) | BIT(25), 0x0); + rtw_write32_mask(rtwdev, 0x40, BIT(2), 0x1); + + /* from s0 or s1 */ + rtw_write32_mask(rtwdev, 0x1990, 0x3f, 0x30); + rtw_write32_mask(rtwdev, 0x1990, (BIT(11) | BIT(10)), 0x3); + + /* input or output */ + rtw_write32_mask(rtwdev, 0x974, 0x3f, 0x3f); + rtw_write32_mask(rtwdev, 0x974, (BIT(11) | BIT(10)), 0x3); +} + +static void rtw8822b_phy_set_param(struct rtw_dev *rtwdev) +{ + struct rtw_hal *hal = &rtwdev->hal; + u8 crystal_cap; + bool is_tx2_path; + + /* power on BB/RF domain */ + rtw_write8_set(rtwdev, REG_SYS_FUNC_EN, + BIT_FEN_BB_RSTB | BIT_FEN_BB_GLB_RST); + rtw_write8_set(rtwdev, REG_RF_CTRL, + BIT_RF_EN | BIT_RF_RSTB | BIT_RF_SDM_RSTB); + rtw_write32_set(rtwdev, REG_WLRF1, BIT_WLRF1_BBRF_EN); + + /* pre init before header files config */ + rtw_write32_clr(rtwdev, REG_RXPSEL, BIT_RX_PSEL_RST); + + rtw_phy_load_tables(rtwdev); + + crystal_cap = rtwdev->efuse.crystal_cap & 0x3F; + rtw_write32_mask(rtwdev, 0x24, 0x7e000000, crystal_cap); + rtw_write32_mask(rtwdev, 0x28, 0x7e, crystal_cap); + + /* post init after header files config */ + rtw_write32_set(rtwdev, REG_RXPSEL, BIT_RX_PSEL_RST); + + is_tx2_path = false; + rtw8822b_config_trx_mode(rtwdev, hal->antenna_tx, hal->antenna_rx, + is_tx2_path); + rtw_phy_init(rtwdev); + + rtw8822b_phy_rfe_init(rtwdev); + + /* wifi path controller */ + rtw_write32_mask(rtwdev, 0x70, 0x4000000, 1); + /* BB control */ + rtw_write32_mask(rtwdev, 0x4c, 0x01800000, 0x2); + /* antenna mux switch */ + rtw_write8(rtwdev, 0x974, 0xff); + rtw_write32_mask(rtwdev, 0x1990, 0x300, 0); + rtw_write32_mask(rtwdev, 0xcbc, 0x80000, 0x0); + /* SW control */ + rtw_write8(rtwdev, 0xcb4, 0x77); + /* switch to WL side controller and gnt_wl gnt_bt debug signal */ + rtw_write32_mask(rtwdev, 0x70, 0xff000000, 0x0e); + /* gnt_wl = 1, gnt_bt = 0 */ + rtw_write32(rtwdev, 0x1704, 0x7700); + rtw_write32(rtwdev, 0x1700, 0xc00f0038); + /* switch for WL 2G */ + rtw_write8(rtwdev, 0xcbd, 0x2); +} + +#define WLAN_SLOT_TIME 0x09 +#define WLAN_PIFS_TIME 0x19 +#define WLAN_SIFS_CCK_CONT_TX 0xA +#define WLAN_SIFS_OFDM_CONT_TX 0xE +#define WLAN_SIFS_CCK_TRX 0x10 +#define WLAN_SIFS_OFDM_TRX 0x10 +#define WLAN_VO_TXOP_LIMIT 0x186 /* unit : 32us */ +#define WLAN_VI_TXOP_LIMIT 0x3BC /* unit : 32us */ +#define WLAN_RDG_NAV 0x05 +#define WLAN_TXOP_NAV 0x1B +#define WLAN_CCK_RX_TSF 0x30 +#define WLAN_OFDM_RX_TSF 0x30 +#define WLAN_TBTT_PROHIBIT 0x04 /* unit : 32us */ +#define WLAN_TBTT_HOLD_TIME 0x064 /* unit : 32us */ +#define WLAN_DRV_EARLY_INT 0x04 +#define WLAN_BCN_DMA_TIME 0x02 + +#define WLAN_RX_FILTER0 0x0FFFFFFF +#define WLAN_RX_FILTER2 0xFFFF +#define WLAN_RCR_CFG 0xE400220E +#define WLAN_RXPKT_MAX_SZ 12288 +#define WLAN_RXPKT_MAX_SZ_512 (WLAN_RXPKT_MAX_SZ >> 9) + +#define WLAN_AMPDU_MAX_TIME 0x70 +#define WLAN_RTS_LEN_TH 0xFF +#define WLAN_RTS_TX_TIME_TH 0x08 +#define WLAN_MAX_AGG_PKT_LIMIT 0x20 +#define WLAN_RTS_MAX_AGG_PKT_LIMIT 0x20 +#define FAST_EDCA_VO_TH 0x06 +#define FAST_EDCA_VI_TH 0x06 +#define FAST_EDCA_BE_TH 0x06 +#define FAST_EDCA_BK_TH 0x06 +#define WLAN_BAR_RETRY_LIMIT 0x01 +#define WLAN_RA_TRY_RATE_AGG_LIMIT 0x08 + +#define WLAN_TX_FUNC_CFG1 0x30 +#define WLAN_TX_FUNC_CFG2 0x30 +#define WLAN_MAC_OPT_NORM_FUNC1 0x98 +#define WLAN_MAC_OPT_LB_FUNC1 0x80 +#define WLAN_MAC_OPT_FUNC2 0x30810041 + +#define WLAN_SIFS_CFG (WLAN_SIFS_CCK_CONT_TX | \ + (WLAN_SIFS_OFDM_CONT_TX << BIT_SHIFT_SIFS_OFDM_CTX) | \ + (WLAN_SIFS_CCK_TRX << BIT_SHIFT_SIFS_CCK_TRX) | \ + (WLAN_SIFS_OFDM_TRX << BIT_SHIFT_SIFS_OFDM_TRX)) + +#define WLAN_TBTT_TIME (WLAN_TBTT_PROHIBIT |\ + (WLAN_TBTT_HOLD_TIME << BIT_SHIFT_TBTT_HOLD_TIME_AP)) + +#define WLAN_NAV_CFG (WLAN_RDG_NAV | (WLAN_TXOP_NAV << 16)) +#define WLAN_RX_TSF_CFG (WLAN_CCK_RX_TSF | (WLAN_OFDM_RX_TSF) << 8) + +static int rtw8822b_mac_init(struct rtw_dev *rtwdev) +{ + u32 value32; + + /* protocol configuration */ + rtw_write8_clr(rtwdev, REG_SW_AMPDU_BURST_MODE_CTRL, BIT_PRE_TX_CMD); + rtw_write8(rtwdev, REG_AMPDU_MAX_TIME_V1, WLAN_AMPDU_MAX_TIME); + rtw_write8_set(rtwdev, REG_TX_HANG_CTRL, BIT_EN_EOF_V1); + value32 = WLAN_RTS_LEN_TH | (WLAN_RTS_TX_TIME_TH << 8) | + (WLAN_MAX_AGG_PKT_LIMIT << 16) | + (WLAN_RTS_MAX_AGG_PKT_LIMIT << 24); + rtw_write32(rtwdev, REG_PROT_MODE_CTRL, value32); + rtw_write16(rtwdev, REG_BAR_MODE_CTRL + 2, + WLAN_BAR_RETRY_LIMIT | WLAN_RA_TRY_RATE_AGG_LIMIT << 8); + rtw_write8(rtwdev, REG_FAST_EDCA_VOVI_SETTING, FAST_EDCA_VO_TH); + rtw_write8(rtwdev, REG_FAST_EDCA_VOVI_SETTING + 2, FAST_EDCA_VI_TH); + rtw_write8(rtwdev, REG_FAST_EDCA_BEBK_SETTING, FAST_EDCA_BE_TH); + rtw_write8(rtwdev, REG_FAST_EDCA_BEBK_SETTING + 2, FAST_EDCA_BK_TH); + /* EDCA configuration */ + rtw_write8_clr(rtwdev, REG_TIMER0_SRC_SEL, BIT_TSFT_SEL_TIMER0); + rtw_write16(rtwdev, REG_TXPAUSE, 0x0000); + rtw_write8(rtwdev, REG_SLOT, WLAN_SLOT_TIME); + rtw_write8(rtwdev, REG_PIFS, WLAN_PIFS_TIME); + rtw_write32(rtwdev, REG_SIFS, WLAN_SIFS_CFG); + rtw_write16(rtwdev, REG_EDCA_VO_PARAM + 2, WLAN_VO_TXOP_LIMIT); + rtw_write16(rtwdev, REG_EDCA_VI_PARAM + 2, WLAN_VI_TXOP_LIMIT); + rtw_write32(rtwdev, REG_RD_NAV_NXT, WLAN_NAV_CFG); + rtw_write16(rtwdev, REG_RXTSF_OFFSET_CCK, WLAN_RX_TSF_CFG); + /* Set beacon cotnrol - enable TSF and other related functions */ + rtw_write8_set(rtwdev, REG_BCN_CTRL, BIT_EN_BCN_FUNCTION); + /* Set send beacon related registers */ + rtw_write32(rtwdev, REG_TBTT_PROHIBIT, WLAN_TBTT_TIME); + rtw_write8(rtwdev, REG_DRVERLYINT, WLAN_DRV_EARLY_INT); + rtw_write8(rtwdev, REG_BCNDMATIM, WLAN_BCN_DMA_TIME); + rtw_write8_clr(rtwdev, REG_TX_PTCL_CTRL + 1, BIT_SIFS_BK_EN >> 8); + /* WMAC configuration */ + rtw_write32(rtwdev, REG_RXFLTMAP0, WLAN_RX_FILTER0); + rtw_write16(rtwdev, REG_RXFLTMAP2, WLAN_RX_FILTER2); + rtw_write32(rtwdev, REG_RCR, WLAN_RCR_CFG); + rtw_write8(rtwdev, REG_RX_PKT_LIMIT, WLAN_RXPKT_MAX_SZ_512); + rtw_write8(rtwdev, REG_TCR + 2, WLAN_TX_FUNC_CFG2); + rtw_write8(rtwdev, REG_TCR + 1, WLAN_TX_FUNC_CFG1); + rtw_write32(rtwdev, REG_WMAC_OPTION_FUNCTION + 8, WLAN_MAC_OPT_FUNC2); + rtw_write8(rtwdev, REG_WMAC_OPTION_FUNCTION + 4, WLAN_MAC_OPT_NORM_FUNC1); + + return 0; +} + +static void rtw8822b_set_channel_rfe_efem(struct rtw_dev *rtwdev, u8 channel) +{ + struct rtw_hal *hal = &rtwdev->hal; + bool is_channel_2g = (channel <= 14) ? true : false; + + if (is_channel_2g) { + rtw_write32s_mask(rtwdev, REG_RFESEL0, 0xffffff, 0x705770); + rtw_write32s_mask(rtwdev, REG_RFESEL8, MASKBYTE1, 0x57); + rtw_write32s_mask(rtwdev, REG_RFECTL, BIT(4), 0); + } else { + rtw_write32s_mask(rtwdev, REG_RFESEL0, 0xffffff, 0x177517); + rtw_write32s_mask(rtwdev, REG_RFESEL8, MASKBYTE1, 0x75); + rtw_write32s_mask(rtwdev, REG_RFECTL, BIT(5), 0); + } + + rtw_write32s_mask(rtwdev, REG_RFEINV, BIT(11) | BIT(10) | 0x3f, 0x0); + + if (hal->antenna_rx == BB_PATH_AB || + hal->antenna_tx == BB_PATH_AB) { + /* 2TX or 2RX */ + rtw_write32s_mask(rtwdev, REG_TRSW, MASKLWORD, 0xa501); + } else if (hal->antenna_rx == hal->antenna_tx) { + /* TXA+RXA or TXB+RXB */ + rtw_write32s_mask(rtwdev, REG_TRSW, MASKLWORD, 0xa500); + } else { + /* TXB+RXA or TXA+RXB */ + rtw_write32s_mask(rtwdev, REG_TRSW, MASKLWORD, 0xa005); + } +} + +static void rtw8822b_set_channel_rfe_ifem(struct rtw_dev *rtwdev, u8 channel) +{ + struct rtw_hal *hal = &rtwdev->hal; + bool is_channel_2g = (channel <= 14) ? true : false; + + if (is_channel_2g) { + /* signal source */ + rtw_write32s_mask(rtwdev, REG_RFESEL0, 0xffffff, 0x745774); + rtw_write32s_mask(rtwdev, REG_RFESEL8, MASKBYTE1, 0x57); + } else { + /* signal source */ + rtw_write32s_mask(rtwdev, REG_RFESEL0, 0xffffff, 0x477547); + rtw_write32s_mask(rtwdev, REG_RFESEL8, MASKBYTE1, 0x75); + } + + rtw_write32s_mask(rtwdev, REG_RFEINV, BIT(11) | BIT(10) | 0x3f, 0x0); + + if (is_channel_2g) { + if (hal->antenna_rx == BB_PATH_AB || + hal->antenna_tx == BB_PATH_AB) { + /* 2TX or 2RX */ + rtw_write32s_mask(rtwdev, REG_TRSW, MASKLWORD, 0xa501); + } else if (hal->antenna_rx == hal->antenna_tx) { + /* TXA+RXA or TXB+RXB */ + rtw_write32s_mask(rtwdev, REG_TRSW, MASKLWORD, 0xa500); + } else { + /* TXB+RXA or TXA+RXB */ + rtw_write32s_mask(rtwdev, REG_TRSW, MASKLWORD, 0xa005); + } + } else { + rtw_write32s_mask(rtwdev, REG_TRSW, MASKLWORD, 0xa5a5); + } +} + +enum { + CCUT_IDX_1R_2G, + CCUT_IDX_2R_2G, + CCUT_IDX_1R_5G, + CCUT_IDX_2R_5G, + CCUT_IDX_NR, +}; + +struct cca_ccut { + u32 reg82c[CCUT_IDX_NR]; + u32 reg830[CCUT_IDX_NR]; + u32 reg838[CCUT_IDX_NR]; +}; + +static const struct cca_ccut cca_ifem_ccut = { + {0x75C97010, 0x75C97010, 0x75C97010, 0x75C97010}, /*Reg82C*/ + {0x79a0eaaa, 0x79A0EAAC, 0x79a0eaaa, 0x79a0eaaa}, /*Reg830*/ + {0x87765541, 0x87746341, 0x87765541, 0x87746341}, /*Reg838*/ +}; + +static const struct cca_ccut cca_efem_ccut = { + {0x75B86010, 0x75B76010, 0x75B86010, 0x75B76010}, /*Reg82C*/ + {0x79A0EAA8, 0x79A0EAAC, 0x79A0EAA8, 0x79a0eaaa}, /*Reg830*/ + {0x87766451, 0x87766431, 0x87766451, 0x87766431}, /*Reg838*/ +}; + +static const struct cca_ccut cca_ifem_ccut_ext = { + {0x75da8010, 0x75da8010, 0x75da8010, 0x75da8010}, /*Reg82C*/ + {0x79a0eaaa, 0x97A0EAAC, 0x79a0eaaa, 0x79a0eaaa}, /*Reg830*/ + {0x87765541, 0x86666341, 0x87765561, 0x86666361}, /*Reg838*/ +}; + +static void rtw8822b_get_cca_val(const struct cca_ccut *cca_ccut, u8 col, + u32 *reg82c, u32 *reg830, u32 *reg838) +{ + *reg82c = cca_ccut->reg82c[col]; + *reg830 = cca_ccut->reg830[col]; + *reg838 = cca_ccut->reg838[col]; +} + +struct rtw8822b_rfe_info { + const struct cca_ccut *cca_ccut_2g; + const struct cca_ccut *cca_ccut_5g; + enum rtw_rfe_fem fem; + bool ifem_ext; + void (*rtw_set_channel_rfe)(struct rtw_dev *rtwdev, u8 channel); +}; + +#define I2GE5G_CCUT(set_ch) { \ + .cca_ccut_2g = &cca_ifem_ccut, \ + .cca_ccut_5g = &cca_efem_ccut, \ + .fem = RTW_RFE_IFEM2G_EFEM5G, \ + .ifem_ext = false, \ + .rtw_set_channel_rfe = &rtw8822b_set_channel_rfe_ ## set_ch, \ + } +#define IFEM_EXT_CCUT(set_ch) { \ + .cca_ccut_2g = &cca_ifem_ccut_ext, \ + .cca_ccut_5g = &cca_ifem_ccut_ext, \ + .fem = RTW_RFE_IFEM, \ + .ifem_ext = true, \ + .rtw_set_channel_rfe = &rtw8822b_set_channel_rfe_ ## set_ch, \ + } + +static const struct rtw8822b_rfe_info rtw8822b_rfe_info[] = { + [2] = I2GE5G_CCUT(efem), + [5] = IFEM_EXT_CCUT(ifem), +}; + +static void rtw8822b_set_channel_cca(struct rtw_dev *rtwdev, u8 channel, u8 bw, + const struct rtw8822b_rfe_info *rfe_info) +{ + struct rtw_hal *hal = &rtwdev->hal; + struct rtw_efuse *efuse = &rtwdev->efuse; + const struct cca_ccut *cca_ccut; + u8 col; + u32 reg82c, reg830, reg838; + bool is_efem_cca = false, is_ifem_cca = false, is_rfe_type = false; + + if (channel <= 14) { + cca_ccut = rfe_info->cca_ccut_2g; + + if (hal->antenna_rx == BB_PATH_A || + hal->antenna_rx == BB_PATH_B) + col = CCUT_IDX_1R_2G; + else + col = CCUT_IDX_2R_2G; + } else { + cca_ccut = rfe_info->cca_ccut_5g; + + if (hal->antenna_rx == BB_PATH_A || + hal->antenna_rx == BB_PATH_B) + col = CCUT_IDX_1R_5G; + else + col = CCUT_IDX_2R_5G; + } + + rtw8822b_get_cca_val(cca_ccut, col, ®82c, ®830, ®838); + + switch (rfe_info->fem) { + case RTW_RFE_IFEM: + default: + is_ifem_cca = true; + if (rfe_info->ifem_ext) + is_rfe_type = true; + break; + case RTW_RFE_EFEM: + is_efem_cca = true; + break; + case RTW_RFE_IFEM2G_EFEM5G: + if (channel <= 14) + is_ifem_cca = true; + else + is_efem_cca = true; + break; + } + + if (is_ifem_cca) { + if ((hal->cut_version == RTW_CHIP_VER_CUT_B && + (col == CCUT_IDX_2R_2G || col == CCUT_IDX_2R_5G) && + bw == RTW_CHANNEL_WIDTH_40) || + (!is_rfe_type && col == CCUT_IDX_2R_5G && + bw == RTW_CHANNEL_WIDTH_40) || + (efuse->rfe_option == 5 && col == CCUT_IDX_2R_5G)) + reg830 = 0x79a0ea28; + } + + rtw_write32_mask(rtwdev, REG_CCASEL, MASKDWORD, reg82c); + rtw_write32_mask(rtwdev, REG_PDMFTH, MASKDWORD, reg830); + rtw_write32_mask(rtwdev, REG_CCA2ND, MASKDWORD, reg838); + + if (is_efem_cca && !(hal->cut_version == RTW_CHIP_VER_CUT_B)) + rtw_write32_mask(rtwdev, REG_L1WT, MASKDWORD, 0x9194b2b9); + + if (bw == RTW_CHANNEL_WIDTH_20 && + ((channel >= 52 && channel <= 64) || + (channel >= 100 && channel <= 144))) + rtw_write32_mask(rtwdev, REG_CCA2ND, 0xf0, 0x4); +} + +static const u8 low_band[15] = {0x7, 0x6, 0x6, 0x5, 0x0, 0x0, 0x7, 0xff, 0x6, + 0x5, 0x0, 0x0, 0x7, 0x6, 0x6}; +static const u8 middle_band[23] = {0x6, 0x5, 0x0, 0x0, 0x7, 0x6, 0x6, 0xff, 0x0, + 0x0, 0x7, 0x6, 0x6, 0x5, 0x0, 0xff, 0x7, 0x6, + 0x6, 0x5, 0x0, 0x0, 0x7}; +static const u8 high_band[15] = {0x5, 0x5, 0x0, 0x7, 0x7, 0x6, 0x5, 0xff, 0x0, + 0x7, 0x7, 0x6, 0x5, 0x5, 0x0}; + +static void rtw8822b_set_channel_rf(struct rtw_dev *rtwdev, u8 channel, u8 bw) +{ +#define RF18_BAND_MASK (BIT(16) | BIT(9) | BIT(8)) +#define RF18_BAND_2G (0) +#define RF18_BAND_5G (BIT(16) | BIT(8)) +#define RF18_CHANNEL_MASK (MASKBYTE0) +#define RF18_RFSI_MASK (BIT(18) | BIT(17)) +#define RF18_RFSI_GE_CH80 (BIT(17)) +#define RF18_RFSI_GT_CH144 (BIT(18)) +#define RF18_BW_MASK (BIT(11) | BIT(10)) +#define RF18_BW_20M (BIT(11) | BIT(10)) +#define RF18_BW_40M (BIT(11)) +#define RF18_BW_80M (BIT(10)) +#define RFBE_MASK (BIT(17) | BIT(16) | BIT(15)) + + struct rtw_hal *hal = &rtwdev->hal; + u32 rf_reg18, rf_reg_be; + + rf_reg18 = rtw_read_rf(rtwdev, RF_PATH_A, 0x18, RFREG_MASK); + + rf_reg18 &= ~(RF18_BAND_MASK | RF18_CHANNEL_MASK | RF18_RFSI_MASK | + RF18_BW_MASK); + + rf_reg18 |= (channel <= 14 ? RF18_BAND_2G : RF18_BAND_5G); + rf_reg18 |= (channel & RF18_CHANNEL_MASK); + if (channel > 144) + rf_reg18 |= RF18_RFSI_GT_CH144; + else if (channel >= 80) + rf_reg18 |= RF18_RFSI_GE_CH80; + + switch (bw) { + case RTW_CHANNEL_WIDTH_5: + case RTW_CHANNEL_WIDTH_10: + case RTW_CHANNEL_WIDTH_20: + default: + rf_reg18 |= RF18_BW_20M; + break; + case RTW_CHANNEL_WIDTH_40: + rf_reg18 |= RF18_BW_40M; + break; + case RTW_CHANNEL_WIDTH_80: + rf_reg18 |= RF18_BW_80M; + break; + } + + if (channel <= 14) + rf_reg_be = 0x0; + else if (channel >= 36 && channel <= 64) + rf_reg_be = low_band[(channel - 36) >> 1]; + else if (channel >= 100 && channel <= 144) + rf_reg_be = middle_band[(channel - 100) >> 1]; + else if (channel >= 149 && channel <= 177) + rf_reg_be = high_band[(channel - 149) >> 1]; + else + goto err; + + rtw_write_rf(rtwdev, RF_PATH_A, RF_MALSEL, RFBE_MASK, rf_reg_be); + + /* need to set 0xdf[18]=1 before writing RF18 when channel 144 */ + if (channel == 144) + rtw_write_rf(rtwdev, RF_PATH_A, RF_LUTDBG, BIT(18), 0x1); + else + rtw_write_rf(rtwdev, RF_PATH_A, RF_LUTDBG, BIT(18), 0x0); + + rtw_write_rf(rtwdev, RF_PATH_A, 0x18, RFREG_MASK, rf_reg18); + if (hal->rf_type > RF_1T1R) + rtw_write_rf(rtwdev, RF_PATH_B, 0x18, RFREG_MASK, rf_reg18); + + rtw_write_rf(rtwdev, RF_PATH_A, RF_XTALX2, BIT(19), 0); + rtw_write_rf(rtwdev, RF_PATH_A, RF_XTALX2, BIT(19), 1); + + return; + +err: + WARN_ON(1); +} + +static void rtw8822b_toggle_igi(struct rtw_dev *rtwdev) +{ + struct rtw_hal *hal = &rtwdev->hal; + u32 igi; + + igi = rtw_read32_mask(rtwdev, REG_RXIGI_A, 0x7f); + rtw_write32_mask(rtwdev, REG_RXIGI_A, 0x7f, igi - 2); + rtw_write32_mask(rtwdev, REG_RXIGI_A, 0x7f, igi); + rtw_write32_mask(rtwdev, REG_RXIGI_B, 0x7f, igi - 2); + rtw_write32_mask(rtwdev, REG_RXIGI_B, 0x7f, igi); + + rtw_write32_mask(rtwdev, REG_RXPSEL, MASKBYTE0, 0x0); + rtw_write32_mask(rtwdev, REG_RXPSEL, MASKBYTE0, + hal->antenna_rx | (hal->antenna_rx << 4)); +} + +static void rtw8822b_set_channel_rxdfir(struct rtw_dev *rtwdev, u8 bw) +{ + if (bw == RTW_CHANNEL_WIDTH_40) { + /* RX DFIR for BW40 */ + rtw_write32_mask(rtwdev, REG_ACBB0, BIT(29) | BIT(28), 0x1); + rtw_write32_mask(rtwdev, REG_ACBBRXFIR, BIT(29) | BIT(28), 0x0); + rtw_write32s_mask(rtwdev, REG_TXDFIR, BIT(31), 0x0); + } else if (bw == RTW_CHANNEL_WIDTH_80) { + /* RX DFIR for BW80 */ + rtw_write32_mask(rtwdev, REG_ACBB0, BIT(29) | BIT(28), 0x2); + rtw_write32_mask(rtwdev, REG_ACBBRXFIR, BIT(29) | BIT(28), 0x1); + rtw_write32s_mask(rtwdev, REG_TXDFIR, BIT(31), 0x0); + } else { + /* RX DFIR for BW20, BW10 and BW5*/ + rtw_write32_mask(rtwdev, REG_ACBB0, BIT(29) | BIT(28), 0x2); + rtw_write32_mask(rtwdev, REG_ACBBRXFIR, BIT(29) | BIT(28), 0x2); + rtw_write32s_mask(rtwdev, REG_TXDFIR, BIT(31), 0x1); + } +} + +static void rtw8822b_set_channel_bb(struct rtw_dev *rtwdev, u8 channel, u8 bw, + u8 primary_ch_idx) +{ + struct rtw_efuse *efuse = &rtwdev->efuse; + u8 rfe_option = efuse->rfe_option; + u32 val32; + + if (channel <= 14) { + rtw_write32_mask(rtwdev, REG_RXPSEL, BIT(28), 0x1); + rtw_write32_mask(rtwdev, REG_CCK_CHECK, BIT(7), 0x0); + rtw_write32_mask(rtwdev, REG_ENTXCCK, BIT(18), 0x0); + rtw_write32_mask(rtwdev, REG_RXCCAMSK, 0x0000FC00, 15); + + rtw_write32_mask(rtwdev, REG_ACGG2TBL, 0x1f, 0x0); + rtw_write32_mask(rtwdev, REG_CLKTRK, 0x1ffe0000, 0x96a); + if (channel == 14) { + rtw_write32_mask(rtwdev, REG_TXSF2, MASKDWORD, 0x00006577); + rtw_write32_mask(rtwdev, REG_TXSF6, MASKLWORD, 0x0000); + } else { + rtw_write32_mask(rtwdev, REG_TXSF2, MASKDWORD, 0x384f6577); + rtw_write32_mask(rtwdev, REG_TXSF6, MASKLWORD, 0x1525); + } + + rtw_write32_mask(rtwdev, REG_RFEINV, 0x300, 0x2); + } else if (channel > 35) { + rtw_write32_mask(rtwdev, REG_ENTXCCK, BIT(18), 0x1); + rtw_write32_mask(rtwdev, REG_CCK_CHECK, BIT(7), 0x1); + rtw_write32_mask(rtwdev, REG_RXPSEL, BIT(28), 0x0); + rtw_write32_mask(rtwdev, REG_RXCCAMSK, 0x0000FC00, 34); + + if (channel >= 36 && channel <= 64) + rtw_write32_mask(rtwdev, REG_ACGG2TBL, 0x1f, 0x1); + else if (channel >= 100 && channel <= 144) + rtw_write32_mask(rtwdev, REG_ACGG2TBL, 0x1f, 0x2); + else if (channel >= 149) + rtw_write32_mask(rtwdev, REG_ACGG2TBL, 0x1f, 0x3); + + if (channel >= 36 && channel <= 48) + rtw_write32_mask(rtwdev, REG_CLKTRK, 0x1ffe0000, 0x494); + else if (channel >= 52 && channel <= 64) + rtw_write32_mask(rtwdev, REG_CLKTRK, 0x1ffe0000, 0x453); + else if (channel >= 100 && channel <= 116) + rtw_write32_mask(rtwdev, REG_CLKTRK, 0x1ffe0000, 0x452); + else if (channel >= 118 && channel <= 177) + rtw_write32_mask(rtwdev, REG_CLKTRK, 0x1ffe0000, 0x412); + + rtw_write32_mask(rtwdev, 0xcbc, 0x300, 0x1); + } + + switch (bw) { + case RTW_CHANNEL_WIDTH_20: + default: + val32 = rtw_read32_mask(rtwdev, REG_ADCCLK, MASKDWORD); + val32 &= 0xFFCFFC00; + val32 |= (RTW_CHANNEL_WIDTH_20); + rtw_write32_mask(rtwdev, REG_ADCCLK, MASKDWORD, val32); + + rtw_write32_mask(rtwdev, REG_ADC160, BIT(30), 0x1); + break; + case RTW_CHANNEL_WIDTH_40: + if (primary_ch_idx == 1) + rtw_write32_set(rtwdev, REG_RXSB, BIT(4)); + else + rtw_write32_clr(rtwdev, REG_RXSB, BIT(4)); + + val32 = rtw_read32_mask(rtwdev, REG_ADCCLK, MASKDWORD); + val32 &= 0xFF3FF300; + val32 |= (((primary_ch_idx & 0xf) << 2) | RTW_CHANNEL_WIDTH_40); + rtw_write32_mask(rtwdev, REG_ADCCLK, MASKDWORD, val32); + + rtw_write32_mask(rtwdev, REG_ADC160, BIT(30), 0x1); + break; + case RTW_CHANNEL_WIDTH_80: + val32 = rtw_read32_mask(rtwdev, REG_ADCCLK, MASKDWORD); + val32 &= 0xFCEFCF00; + val32 |= (((primary_ch_idx & 0xf) << 2) | RTW_CHANNEL_WIDTH_80); + rtw_write32_mask(rtwdev, REG_ADCCLK, MASKDWORD, val32); + + rtw_write32_mask(rtwdev, REG_ADC160, BIT(30), 0x1); + + if (rfe_option == 2) { + rtw_write32_mask(rtwdev, REG_L1PKWT, 0x0000f000, 0x6); + rtw_write32_mask(rtwdev, REG_ADC40, BIT(10), 0x1); + } + break; + case RTW_CHANNEL_WIDTH_5: + val32 = rtw_read32_mask(rtwdev, REG_ADCCLK, MASKDWORD); + val32 &= 0xEFEEFE00; + val32 |= ((BIT(6) | RTW_CHANNEL_WIDTH_20)); + rtw_write32_mask(rtwdev, REG_ADCCLK, MASKDWORD, val32); + + rtw_write32_mask(rtwdev, REG_ADC160, BIT(30), 0x0); + rtw_write32_mask(rtwdev, REG_ADC40, BIT(31), 0x1); + break; + case RTW_CHANNEL_WIDTH_10: + val32 = rtw_read32_mask(rtwdev, REG_ADCCLK, MASKDWORD); + val32 &= 0xEFFEFF00; + val32 |= ((BIT(7) | RTW_CHANNEL_WIDTH_20)); + rtw_write32_mask(rtwdev, REG_ADCCLK, MASKDWORD, val32); + + rtw_write32_mask(rtwdev, REG_ADC160, BIT(30), 0x0); + rtw_write32_mask(rtwdev, REG_ADC40, BIT(31), 0x1); + break; + } +} + +static void rtw8822b_set_channel(struct rtw_dev *rtwdev, u8 channel, u8 bw, + u8 primary_chan_idx) +{ + struct rtw_efuse *efuse = &rtwdev->efuse; + const struct rtw8822b_rfe_info *rfe_info; + + if (WARN(efuse->rfe_option >= ARRAY_SIZE(rtw8822b_rfe_info), + "rfe_option %d is out of boundary\n", efuse->rfe_option)) + return; + + rfe_info = &rtw8822b_rfe_info[efuse->rfe_option]; + + rtw8822b_set_channel_bb(rtwdev, channel, bw, primary_chan_idx); + rtw_set_channel_mac(rtwdev, channel, bw, primary_chan_idx); + rtw8822b_set_channel_rf(rtwdev, channel, bw); + rtw8822b_set_channel_rxdfir(rtwdev, bw); + rtw8822b_toggle_igi(rtwdev); + rtw8822b_set_channel_cca(rtwdev, channel, bw, rfe_info); + (*rfe_info->rtw_set_channel_rfe)(rtwdev, channel); +} + +static void rtw8822b_config_trx_mode(struct rtw_dev *rtwdev, u8 tx_path, + u8 rx_path, bool is_tx2_path) +{ + struct rtw_efuse *efuse = &rtwdev->efuse; + const struct rtw8822b_rfe_info *rfe_info; + u8 ch = rtwdev->hal.current_channel; + u8 tx_path_sel, rx_path_sel; + int counter; + + if (WARN(efuse->rfe_option >= ARRAY_SIZE(rtw8822b_rfe_info), + "rfe_option %d is out of boundary\n", efuse->rfe_option)) + return; + + rfe_info = &rtw8822b_rfe_info[efuse->rfe_option]; + + if ((tx_path | rx_path) & BB_PATH_A) + rtw_write32_mask(rtwdev, REG_AGCTR_A, MASKLWORD, 0x3231); + else + rtw_write32_mask(rtwdev, REG_AGCTR_A, MASKLWORD, 0x1111); + + if ((tx_path | rx_path) & BB_PATH_B) + rtw_write32_mask(rtwdev, REG_AGCTR_B, MASKLWORD, 0x3231); + else + rtw_write32_mask(rtwdev, REG_AGCTR_B, MASKLWORD, 0x1111); + + rtw_write32_mask(rtwdev, REG_CDDTXP, (BIT(19) | BIT(18)), 0x3); + rtw_write32_mask(rtwdev, REG_TXPSEL, (BIT(29) | BIT(28)), 0x1); + rtw_write32_mask(rtwdev, REG_TXPSEL, BIT(30), 0x1); + + if (tx_path & BB_PATH_A) { + rtw_write32_mask(rtwdev, REG_CDDTXP, 0xfff00000, 0x001); + rtw_write32_mask(rtwdev, REG_ADCINI, 0xf0000000, 0x8); + } else if (tx_path & BB_PATH_B) { + rtw_write32_mask(rtwdev, REG_CDDTXP, 0xfff00000, 0x002); + rtw_write32_mask(rtwdev, REG_ADCINI, 0xf0000000, 0x4); + } + + if (tx_path == BB_PATH_A || tx_path == BB_PATH_B) + rtw_write32_mask(rtwdev, REG_TXPSEL1, 0xfff0, 0x01); + else + rtw_write32_mask(rtwdev, REG_TXPSEL1, 0xfff0, 0x43); + + tx_path_sel = (tx_path << 4) | tx_path; + rtw_write32_mask(rtwdev, REG_TXPSEL, MASKBYTE0, tx_path_sel); + + if (tx_path != BB_PATH_A && tx_path != BB_PATH_B) { + if (is_tx2_path || rtwdev->mp_mode) { + rtw_write32_mask(rtwdev, REG_CDDTXP, 0xfff00000, 0x043); + rtw_write32_mask(rtwdev, REG_ADCINI, 0xf0000000, 0xc); + } + } + + rtw_write32_mask(rtwdev, REG_RXDESC, BIT(22), 0x0); + rtw_write32_mask(rtwdev, REG_RXDESC, BIT(18), 0x0); + + if (rx_path & BB_PATH_A) + rtw_write32_mask(rtwdev, REG_ADCINI, 0x0f000000, 0x0); + else if (rx_path & BB_PATH_B) + rtw_write32_mask(rtwdev, REG_ADCINI, 0x0f000000, 0x5); + + rx_path_sel = (rx_path << 4) | rx_path; + rtw_write32_mask(rtwdev, REG_RXPSEL, MASKBYTE0, rx_path_sel); + + if (rx_path == BB_PATH_A || rx_path == BB_PATH_B) { + rtw_write32_mask(rtwdev, REG_ANTWT, BIT(16), 0x0); + rtw_write32_mask(rtwdev, REG_HTSTFWT, BIT(28), 0x0); + rtw_write32_mask(rtwdev, REG_MRC, BIT(23), 0x0); + } else { + rtw_write32_mask(rtwdev, REG_ANTWT, BIT(16), 0x1); + rtw_write32_mask(rtwdev, REG_HTSTFWT, BIT(28), 0x1); + rtw_write32_mask(rtwdev, REG_MRC, BIT(23), 0x1); + } + + for (counter = 100; counter > 0; counter--) { + u32 rf_reg33; + + rtw_write_rf(rtwdev, RF_PATH_A, RF_LUTWE, RFREG_MASK, 0x80000); + rtw_write_rf(rtwdev, RF_PATH_A, RF_LUTWA, RFREG_MASK, 0x00001); + + udelay(2); + rf_reg33 = rtw_read_rf(rtwdev, RF_PATH_A, 0x33, RFREG_MASK); + + if (rf_reg33 == 0x00001) + break; + } + + if (WARN(counter <= 0, "write RF mode table fail\n")) + return; + + rtw_write_rf(rtwdev, RF_PATH_A, RF_LUTWE, RFREG_MASK, 0x80000); + rtw_write_rf(rtwdev, RF_PATH_A, RF_LUTWA, RFREG_MASK, 0x00001); + rtw_write_rf(rtwdev, RF_PATH_A, RF_LUTWD1, RFREG_MASK, 0x00034); + rtw_write_rf(rtwdev, RF_PATH_A, RF_LUTWD0, RFREG_MASK, 0x4080c); + rtw_write_rf(rtwdev, RF_PATH_A, RF_LUTWE, RFREG_MASK, 0x00000); + rtw_write_rf(rtwdev, RF_PATH_A, RF_LUTWE, RFREG_MASK, 0x00000); + + rtw8822b_toggle_igi(rtwdev); + rtw8822b_set_channel_cca(rtwdev, 1, RTW_CHANNEL_WIDTH_20, rfe_info); + (*rfe_info->rtw_set_channel_rfe)(rtwdev, ch); +} + +static void query_phy_status_page0(struct rtw_dev *rtwdev, u8 *phy_status, + struct rtw_rx_pkt_stat *pkt_stat) +{ + s8 min_rx_power = -120; + u8 pwdb = GET_PHY_STAT_P0_PWDB(phy_status); + + pkt_stat->rx_power[RF_PATH_A] = pwdb - 110; + pkt_stat->rssi = rtw_phy_rf_power_2_rssi(pkt_stat->rx_power, 1); + pkt_stat->bw = RTW_CHANNEL_WIDTH_20; + pkt_stat->signal_power = max(pkt_stat->rx_power[RF_PATH_A], + min_rx_power); +} + +static void query_phy_status_page1(struct rtw_dev *rtwdev, u8 *phy_status, + struct rtw_rx_pkt_stat *pkt_stat) +{ + u8 rxsc, bw; + s8 min_rx_power = -120; + + if (pkt_stat->rate > DESC_RATE11M && pkt_stat->rate < DESC_RATEMCS0) + rxsc = GET_PHY_STAT_P1_L_RXSC(phy_status); + else + rxsc = GET_PHY_STAT_P1_HT_RXSC(phy_status); + + if (rxsc >= 1 && rxsc <= 8) + bw = RTW_CHANNEL_WIDTH_20; + else if (rxsc >= 9 && rxsc <= 12) + bw = RTW_CHANNEL_WIDTH_40; + else if (rxsc >= 13) + bw = RTW_CHANNEL_WIDTH_80; + else + bw = GET_PHY_STAT_P1_RF_MODE(phy_status); + + pkt_stat->rx_power[RF_PATH_A] = GET_PHY_STAT_P1_PWDB_A(phy_status) - 110; + pkt_stat->rx_power[RF_PATH_B] = GET_PHY_STAT_P1_PWDB_B(phy_status) - 110; + pkt_stat->rssi = rtw_phy_rf_power_2_rssi(pkt_stat->rx_power, 2); + pkt_stat->bw = bw; + pkt_stat->signal_power = max3(pkt_stat->rx_power[RF_PATH_A], + pkt_stat->rx_power[RF_PATH_B], + min_rx_power); +} + +static void query_phy_status(struct rtw_dev *rtwdev, u8 *phy_status, + struct rtw_rx_pkt_stat *pkt_stat) +{ + u8 page; + + page = *phy_status & 0xf; + + switch (page) { + case 0: + query_phy_status_page0(rtwdev, phy_status, pkt_stat); + break; + case 1: + query_phy_status_page1(rtwdev, phy_status, pkt_stat); + break; + default: + rtw_warn(rtwdev, "unused phy status page (%d)\n", page); + return; + } +} + +static void rtw8822b_query_rx_desc(struct rtw_dev *rtwdev, u8 *rx_desc, + struct rtw_rx_pkt_stat *pkt_stat, + struct ieee80211_rx_status *rx_status) +{ + struct ieee80211_hdr *hdr; + u32 desc_sz = rtwdev->chip->rx_pkt_desc_sz; + u8 *phy_status = NULL; + + memset(pkt_stat, 0, sizeof(*pkt_stat)); + + pkt_stat->phy_status = GET_RX_DESC_PHYST(rx_desc); + pkt_stat->icv_err = GET_RX_DESC_ICV_ERR(rx_desc); + pkt_stat->crc_err = GET_RX_DESC_CRC32(rx_desc); + pkt_stat->decrypted = !GET_RX_DESC_SWDEC(rx_desc); + pkt_stat->is_c2h = GET_RX_DESC_C2H(rx_desc); + pkt_stat->pkt_len = GET_RX_DESC_PKT_LEN(rx_desc); + pkt_stat->drv_info_sz = GET_RX_DESC_DRV_INFO_SIZE(rx_desc); + pkt_stat->shift = GET_RX_DESC_SHIFT(rx_desc); + pkt_stat->rate = GET_RX_DESC_RX_RATE(rx_desc); + pkt_stat->cam_id = GET_RX_DESC_MACID(rx_desc); + pkt_stat->ppdu_cnt = GET_RX_DESC_PPDU_CNT(rx_desc); + pkt_stat->tsf_low = GET_RX_DESC_TSFL(rx_desc); + + /* drv_info_sz is in unit of 8-bytes */ + pkt_stat->drv_info_sz *= 8; + + /* c2h cmd pkt's rx/phy status is not interested */ + if (pkt_stat->is_c2h) + return; + + hdr = (struct ieee80211_hdr *)(rx_desc + desc_sz + pkt_stat->shift + + pkt_stat->drv_info_sz); + if (pkt_stat->phy_status) { + phy_status = rx_desc + desc_sz + pkt_stat->shift; + query_phy_status(rtwdev, phy_status, pkt_stat); + } + + rtw_rx_fill_rx_status(rtwdev, pkt_stat, hdr, rx_status, phy_status); +} + +static void +rtw8822b_set_tx_power_index_by_rate(struct rtw_dev *rtwdev, u8 path, u8 rs) +{ + struct rtw_hal *hal = &rtwdev->hal; + static const u32 offset_txagc[2] = {0x1d00, 0x1d80}; + static u32 phy_pwr_idx; + u8 rate, rate_idx, pwr_index, shift; + int j; + + for (j = 0; j < rtw_rate_size[rs]; j++) { + rate = rtw_rate_section[rs][j]; + pwr_index = hal->tx_pwr_tbl[path][rate]; + shift = rate & 0x3; + phy_pwr_idx |= ((u32)pwr_index << (shift * 8)); + if (shift == 0x3) { + rate_idx = rate & 0xfc; + rtw_write32(rtwdev, offset_txagc[path] + rate_idx, + phy_pwr_idx); + phy_pwr_idx = 0; + } + } +} + +static void rtw8822b_set_tx_power_index(struct rtw_dev *rtwdev) +{ + struct rtw_hal *hal = &rtwdev->hal; + int rs, path; + + for (path = 0; path < hal->rf_path_num; path++) { + for (rs = 0; rs < RTW_RATE_SECTION_MAX; rs++) + rtw8822b_set_tx_power_index_by_rate(rtwdev, path, rs); + } +} + +static bool rtw8822b_check_rf_path(u8 antenna) +{ + switch (antenna) { + case BB_PATH_A: + case BB_PATH_B: + case BB_PATH_AB: + return true; + default: + return false; + } +} + +static void rtw8822b_set_antenna(struct rtw_dev *rtwdev, u8 antenna_tx, + u8 antenna_rx) +{ + struct rtw_hal *hal = &rtwdev->hal; + + rtw_dbg(rtwdev, RTW_DBG_PHY, "config RF path, tx=0x%x rx=0x%x\n", + antenna_tx, antenna_rx); + + if (!rtw8822b_check_rf_path(antenna_tx)) { + rtw_info(rtwdev, "unsupport tx path, set to default path ab\n"); + antenna_tx = BB_PATH_AB; + } + if (!rtw8822b_check_rf_path(antenna_rx)) { + rtw_info(rtwdev, "unsupport rx path, set to default path ab\n"); + antenna_rx = BB_PATH_AB; + } + hal->antenna_tx = antenna_tx; + hal->antenna_rx = antenna_rx; + rtw8822b_config_trx_mode(rtwdev, antenna_tx, antenna_rx, false); +} + +static void rtw8822b_cfg_ldo25(struct rtw_dev *rtwdev, bool enable) +{ + u8 ldo_pwr; + + ldo_pwr = rtw_read8(rtwdev, REG_LDO_EFUSE_CTRL + 3); + ldo_pwr = enable ? ldo_pwr | BIT(7) : ldo_pwr & ~BIT(7); + rtw_write8(rtwdev, REG_LDO_EFUSE_CTRL + 3, ldo_pwr); +} + +static void rtw8822b_false_alarm_statistics(struct rtw_dev *rtwdev) +{ + struct rtw_dm_info *dm_info = &rtwdev->dm_info; + u32 cck_enable; + u32 cck_fa_cnt; + u32 ofdm_fa_cnt; + + cck_enable = rtw_read32(rtwdev, 0x808) & BIT(28); + cck_fa_cnt = rtw_read16(rtwdev, 0xa5c); + ofdm_fa_cnt = rtw_read16(rtwdev, 0xf48); + + dm_info->cck_fa_cnt = cck_fa_cnt; + dm_info->ofdm_fa_cnt = ofdm_fa_cnt; + dm_info->total_fa_cnt = ofdm_fa_cnt; + dm_info->total_fa_cnt += cck_enable ? cck_fa_cnt : 0; + + rtw_write32_set(rtwdev, 0x9a4, BIT(17)); + rtw_write32_clr(rtwdev, 0x9a4, BIT(17)); + rtw_write32_clr(rtwdev, 0xa2c, BIT(15)); + rtw_write32_set(rtwdev, 0xa2c, BIT(15)); + rtw_write32_set(rtwdev, 0xb58, BIT(0)); + rtw_write32_clr(rtwdev, 0xb58, BIT(0)); +} + +static void rtw8822b_do_iqk(struct rtw_dev *rtwdev) +{ + static int do_iqk_cnt; + struct rtw_iqk_para para = {.clear = 0, .segment_iqk = 0}; + u32 rf_reg, iqk_fail_mask; + int counter; + bool reload; + + rtw_fw_do_iqk(rtwdev, ¶); + + for (counter = 0; counter < 300; counter++) { + rf_reg = rtw_read_rf(rtwdev, RF_PATH_A, RF_DTXLOK, RFREG_MASK); + if (rf_reg == 0xabcde) + break; + msleep(20); + } + rtw_write_rf(rtwdev, RF_PATH_A, RF_DTXLOK, RFREG_MASK, 0x0); + + reload = !!rtw_read32_mask(rtwdev, REG_IQKFAILMSK, BIT(16)); + iqk_fail_mask = rtw_read32_mask(rtwdev, REG_IQKFAILMSK, GENMASK(0, 7)); + rtw_dbg(rtwdev, RTW_DBG_PHY, + "iqk counter=%d reload=%d do_iqk_cnt=%d n_iqk_fail(mask)=0x%02x\n", + counter, reload, ++do_iqk_cnt, iqk_fail_mask); +} + +static struct rtw_pwr_seq_cmd trans_carddis_to_cardemu_8822b[] = { + {0x0086, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_SDIO, + RTW_PWR_CMD_WRITE, BIT(0), 0}, + {0x0086, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_SDIO, + RTW_PWR_CMD_POLLING, BIT(1), BIT(1)}, + {0x004A, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(0), 0}, + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(3) | BIT(4) | BIT(7), 0}, + {0x0300, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_PCI_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, 0xFF, 0}, + {0x0301, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_PCI_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, 0xFF, 0}, + {0xFFFF, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + 0, + RTW_PWR_CMD_END, 0, 0}, +}; + +static struct rtw_pwr_seq_cmd trans_cardemu_to_act_8822b[] = { + {0x0012, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(1), 0}, + {0x0012, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(0), BIT(0)}, + {0x0020, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK | RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(0), BIT(0)}, + {0x0001, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK | RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_DELAY, 1, RTW_PWR_DELAY_MS}, + {0x0000, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK | RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(5), 0}, + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, (BIT(4) | BIT(3) | BIT(2)), 0}, + {0x0075, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_PCI_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(0), BIT(0)}, + {0x0006, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_POLLING, BIT(1), BIT(1)}, + {0x0075, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_PCI_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(0), 0}, + {0xFF1A, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, 0xFF, 0}, + {0x0006, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(0), BIT(0)}, + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(7), 0}, + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, (BIT(4) | BIT(3)), 0}, + {0x10C3, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(0), BIT(0)}, + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(0), BIT(0)}, + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_POLLING, BIT(0), 0}, + {0x0020, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(3), BIT(3)}, + {0x10A8, + RTW_PWR_CUT_C_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, 0xFF, 0}, + {0x10A9, + RTW_PWR_CUT_C_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, 0xFF, 0xef}, + {0x10AA, + RTW_PWR_CUT_C_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, 0xFF, 0x0c}, + {0x0068, + RTW_PWR_CUT_C_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(4), BIT(4)}, + {0x0029, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, 0xFF, 0xF9}, + {0x0024, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(2), 0}, + {0x0074, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_PCI_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(5), BIT(5)}, + {0x00AF, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(5), BIT(5)}, + {0xFFFF, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + 0, + RTW_PWR_CMD_END, 0, 0}, +}; + +static struct rtw_pwr_seq_cmd trans_act_to_cardemu_8822b[] = { + {0x0003, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(2), 0}, + {0x0093, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(3), 0}, + {0x001F, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, 0xFF, 0}, + {0x00EF, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, 0xFF, 0}, + {0xFF1A, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, 0xFF, 0x30}, + {0x0049, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(1), 0}, + {0x0006, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(0), BIT(0)}, + {0x0002, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(1), 0}, + {0x10C3, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(0), 0}, + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(1), BIT(1)}, + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_POLLING, BIT(1), 0}, + {0x0020, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(3), 0}, + {0x0000, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK | RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(5), BIT(5)}, + {0xFFFF, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + 0, + RTW_PWR_CMD_END, 0, 0}, +}; + +static struct rtw_pwr_seq_cmd trans_cardemu_to_carddis_8822b[] = { + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(7), BIT(7)}, + {0x0007, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK | RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, 0xFF, 0x20}, + {0x0067, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(5), 0}, + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_PCI_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(2), BIT(2)}, + {0x004A, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(0), 0}, + {0x0067, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(5), 0}, + {0x0067, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(4), 0}, + {0x004F, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(0), 0}, + {0x0067, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(1), 0}, + {0x0046, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(6), BIT(6)}, + {0x0067, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(2), 0}, + {0x0046, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(7), BIT(7)}, + {0x0062, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(4), BIT(4)}, + {0x0081, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(7) | BIT(6), 0}, + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK | RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(3) | BIT(4), BIT(3)}, + {0x0086, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_SDIO, + RTW_PWR_CMD_WRITE, BIT(0), BIT(0)}, + {0x0086, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_SDIO, + RTW_PWR_CMD_POLLING, BIT(1), 0}, + {0x0090, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK | RTW_PWR_INTF_PCI_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(1), 0}, + {0x0044, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_SDIO, + RTW_PWR_CMD_WRITE, 0xFF, 0}, + {0x0040, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_SDIO, + RTW_PWR_CMD_WRITE, 0xFF, 0x90}, + {0x0041, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_SDIO, + RTW_PWR_CMD_WRITE, 0xFF, 0x00}, + {0x0042, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_SDIO, + RTW_PWR_CMD_WRITE, 0xFF, 0x04}, + {0xFFFF, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + 0, + RTW_PWR_CMD_END, 0, 0}, +}; + +static struct rtw_pwr_seq_cmd *card_enable_flow_8822b[] = { + trans_carddis_to_cardemu_8822b, + trans_cardemu_to_act_8822b, + NULL +}; + +static struct rtw_pwr_seq_cmd *card_disable_flow_8822b[] = { + trans_act_to_cardemu_8822b, + trans_cardemu_to_carddis_8822b, + NULL +}; + +static struct rtw_intf_phy_para usb2_param_8822b[] = { + {0xFFFF, 0x00, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_ALL, + RTW_INTF_PHY_PLATFORM_ALL}, +}; + +static struct rtw_intf_phy_para usb3_param_8822b[] = { + {0x0001, 0xA841, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_D, + RTW_INTF_PHY_PLATFORM_ALL}, + {0xFFFF, 0x0000, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_ALL, + RTW_INTF_PHY_PLATFORM_ALL}, +}; + +static struct rtw_intf_phy_para pcie_gen1_param_8822b[] = { + {0x0001, 0xA841, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_C, + RTW_INTF_PHY_PLATFORM_ALL}, + {0x0002, 0x60C6, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_C, + RTW_INTF_PHY_PLATFORM_ALL}, + {0x0008, 0x3596, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_C, + RTW_INTF_PHY_PLATFORM_ALL}, + {0x0009, 0x321C, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_C, + RTW_INTF_PHY_PLATFORM_ALL}, + {0x000A, 0x9623, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_C, + RTW_INTF_PHY_PLATFORM_ALL}, + {0x0020, 0x94FF, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_C, + RTW_INTF_PHY_PLATFORM_ALL}, + {0x0021, 0xFFCF, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_C, + RTW_INTF_PHY_PLATFORM_ALL}, + {0x0026, 0xC006, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_C, + RTW_INTF_PHY_PLATFORM_ALL}, + {0x0029, 0xFF0E, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_C, + RTW_INTF_PHY_PLATFORM_ALL}, + {0x002A, 0x1840, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_C, + RTW_INTF_PHY_PLATFORM_ALL}, + {0xFFFF, 0x0000, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_ALL, + RTW_INTF_PHY_PLATFORM_ALL}, +}; + +static struct rtw_intf_phy_para pcie_gen2_param_8822b[] = { + {0x0001, 0xA841, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_C, + RTW_INTF_PHY_PLATFORM_ALL}, + {0x0002, 0x60C6, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_C, + RTW_INTF_PHY_PLATFORM_ALL}, + {0x0008, 0x3597, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_C, + RTW_INTF_PHY_PLATFORM_ALL}, + {0x0009, 0x321C, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_C, + RTW_INTF_PHY_PLATFORM_ALL}, + {0x000A, 0x9623, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_C, + RTW_INTF_PHY_PLATFORM_ALL}, + {0x0020, 0x94FF, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_C, + RTW_INTF_PHY_PLATFORM_ALL}, + {0x0021, 0xFFCF, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_C, + RTW_INTF_PHY_PLATFORM_ALL}, + {0x0026, 0xC006, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_C, + RTW_INTF_PHY_PLATFORM_ALL}, + {0x0029, 0xFF0E, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_C, + RTW_INTF_PHY_PLATFORM_ALL}, + {0x002A, 0x3040, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_C, + RTW_INTF_PHY_PLATFORM_ALL}, + {0xFFFF, 0x0000, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_ALL, + RTW_INTF_PHY_PLATFORM_ALL}, +}; + +static struct rtw_intf_phy_para_table phy_para_table_8822b = { + .usb2_para = usb2_param_8822b, + .usb3_para = usb3_param_8822b, + .gen1_para = pcie_gen1_param_8822b, + .gen2_para = pcie_gen2_param_8822b, + .n_usb2_para = ARRAY_SIZE(usb2_param_8822b), + .n_usb3_para = ARRAY_SIZE(usb2_param_8822b), + .n_gen1_para = ARRAY_SIZE(pcie_gen1_param_8822b), + .n_gen2_para = ARRAY_SIZE(pcie_gen2_param_8822b), +}; + +static const struct rtw_rfe_def rtw8822b_rfe_defs[] = { + [2] = RTW_DEF_RFE(8822b, 2, 2), + [5] = RTW_DEF_RFE(8822b, 5, 5), +}; + +static struct rtw_hw_reg rtw8822b_dig[] = { + [0] = { .addr = 0xc50, .mask = 0x7f }, + [1] = { .addr = 0xe50, .mask = 0x7f }, +}; + +static struct rtw_page_table page_table_8822b[] = { + {64, 64, 64, 64, 1}, + {64, 64, 64, 64, 1}, + {64, 64, 0, 0, 1}, + {64, 64, 64, 0, 1}, + {64, 64, 64, 64, 1}, +}; + +static struct rtw_rqpn rqpn_table_8822b[] = { + {RTW_DMA_MAPPING_NORMAL, RTW_DMA_MAPPING_NORMAL, + RTW_DMA_MAPPING_LOW, RTW_DMA_MAPPING_LOW, + RTW_DMA_MAPPING_EXTRA, RTW_DMA_MAPPING_HIGH}, + {RTW_DMA_MAPPING_NORMAL, RTW_DMA_MAPPING_NORMAL, + RTW_DMA_MAPPING_LOW, RTW_DMA_MAPPING_LOW, + RTW_DMA_MAPPING_EXTRA, RTW_DMA_MAPPING_HIGH}, + {RTW_DMA_MAPPING_NORMAL, RTW_DMA_MAPPING_NORMAL, + RTW_DMA_MAPPING_NORMAL, RTW_DMA_MAPPING_HIGH, + RTW_DMA_MAPPING_HIGH, RTW_DMA_MAPPING_HIGH}, + {RTW_DMA_MAPPING_NORMAL, RTW_DMA_MAPPING_NORMAL, + RTW_DMA_MAPPING_LOW, RTW_DMA_MAPPING_LOW, + RTW_DMA_MAPPING_HIGH, RTW_DMA_MAPPING_HIGH}, + {RTW_DMA_MAPPING_NORMAL, RTW_DMA_MAPPING_NORMAL, + RTW_DMA_MAPPING_LOW, RTW_DMA_MAPPING_LOW, + RTW_DMA_MAPPING_EXTRA, RTW_DMA_MAPPING_HIGH}, +}; + +static struct rtw_chip_ops rtw8822b_ops = { + .phy_set_param = rtw8822b_phy_set_param, + .read_efuse = rtw8822b_read_efuse, + .query_rx_desc = rtw8822b_query_rx_desc, + .set_channel = rtw8822b_set_channel, + .mac_init = rtw8822b_mac_init, + .read_rf = rtw_phy_read_rf, + .write_rf = rtw_phy_write_rf_reg_sipi, + .set_tx_power_index = rtw8822b_set_tx_power_index, + .set_antenna = rtw8822b_set_antenna, + .cfg_ldo25 = rtw8822b_cfg_ldo25, + .false_alarm_statistics = rtw8822b_false_alarm_statistics, + .do_iqk = rtw8822b_do_iqk, +}; + +struct rtw_chip_info rtw8822b_hw_spec = { + .ops = &rtw8822b_ops, + .id = RTW_CHIP_TYPE_8822B, + .fw_name = "rtw88/rtw8822b_fw.bin", + .tx_pkt_desc_sz = 48, + .tx_buf_desc_sz = 16, + .rx_pkt_desc_sz = 24, + .rx_buf_desc_sz = 8, + .phy_efuse_size = 1024, + .log_efuse_size = 768, + .ptct_efuse_size = 96, + .txff_size = 262144, + .rxff_size = 24576, + .txgi_factor = 1, + .is_pwr_by_rate_dec = true, + .max_power_index = 0x3f, + .csi_buf_pg_num = 0, + .band = RTW_BAND_2G | RTW_BAND_5G, + .page_size = 128, + .dig_min = 0x1c, + .ht_supported = true, + .vht_supported = true, + .sys_func_en = 0xDC, + .pwr_on_seq = card_enable_flow_8822b, + .pwr_off_seq = card_disable_flow_8822b, + .page_table = page_table_8822b, + .rqpn_table = rqpn_table_8822b, + .intf_table = &phy_para_table_8822b, + .dig = rtw8822b_dig, + .rf_base_addr = {0x2800, 0x2c00}, + .rf_sipi_addr = {0xc90, 0xe90}, + .mac_tbl = &rtw8822b_mac_tbl, + .agc_tbl = &rtw8822b_agc_tbl, + .bb_tbl = &rtw8822b_bb_tbl, + .rf_tbl = {&rtw8822b_rf_a_tbl, &rtw8822b_rf_b_tbl}, + .rfe_defs = rtw8822b_rfe_defs, + .rfe_defs_size = ARRAY_SIZE(rtw8822b_rfe_defs), +}; +EXPORT_SYMBOL(rtw8822b_hw_spec); + +MODULE_FIRMWARE("rtw88/rtw8822b_fw.bin"); diff --git a/drivers/net/wireless/realtek/rtw88/rtw8822b.h b/drivers/net/wireless/realtek/rtw88/rtw8822b.h new file mode 100644 index 0000000000000000000000000000000000000000..0cb93d7d4cfd92ec4a5c902c55964d86a7d15016 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/rtw8822b.h @@ -0,0 +1,170 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#ifndef __RTW8822B_H__ +#define __RTW8822B_H__ + +#include + +#define RCR_VHT_ACK BIT(26) + +struct rtw8822bu_efuse { + u8 res4[4]; /* 0xd0 */ + u8 usb_optional_function; + u8 res5[0x1e]; + u8 res6[2]; + u8 serial[0x0b]; /* 0xf5 */ + u8 vid; /* 0x100 */ + u8 res7; + u8 pid; + u8 res8[4]; + u8 mac_addr[ETH_ALEN]; /* 0x107 */ + u8 res9[2]; + u8 vendor_name[0x07]; + u8 res10[2]; + u8 device_name[0x14]; + u8 res11[0xcf]; + u8 package_type; /* 0x1fb */ + u8 res12[0x4]; +}; + +struct rtw8822be_efuse { + u8 mac_addr[ETH_ALEN]; /* 0xd0 */ + u8 vender_id[2]; + u8 device_id[2]; + u8 sub_vender_id[2]; + u8 sub_device_id[2]; + u8 pmc[2]; + u8 exp_device_cap[2]; + u8 msi_cap; + u8 ltr_cap; /* 0xe3 */ + u8 exp_link_control[2]; + u8 link_cap[4]; + u8 link_control[2]; + u8 serial_number[8]; + u8 res0:2; /* 0xf4 */ + u8 ltr_en:1; + u8 res1:2; + u8 obff:2; + u8 res2:3; + u8 obff_cap:2; + u8 res3:4; + u8 res4[3]; + u8 class_code[3]; + u8 pci_pm_L1_2_supp:1; + u8 pci_pm_L1_1_supp:1; + u8 aspm_pm_L1_2_supp:1; + u8 aspm_pm_L1_1_supp:1; + u8 L1_pm_substates_supp:1; + u8 res5:3; + u8 port_common_mode_restore_time; + u8 port_t_power_on_scale:2; + u8 res6:1; + u8 port_t_power_on_value:5; + u8 res7; +}; + +struct rtw8822b_efuse { + __le16 rtl_id; + u8 res0[0x0e]; + + /* power index for four RF paths */ + struct rtw_txpwr_idx txpwr_idx_table[4]; + + u8 channel_plan; /* 0xb8 */ + u8 xtal_k; + u8 thermal_meter; + u8 iqk_lck; + u8 pa_type; /* 0xbc */ + u8 lna_type_2g[2]; /* 0xbd */ + u8 lna_type_5g[2]; + u8 rf_board_option; + u8 rf_feature_option; + u8 rf_bt_setting; + u8 eeprom_version; + u8 eeprom_customer_id; + u8 tx_bb_swing_setting_2g; + u8 tx_bb_swing_setting_5g; + u8 tx_pwr_calibrate_rate; + u8 rf_antenna_option; /* 0xc9 */ + u8 rfe_option; + u8 country_code[2]; + u8 res[3]; + union { + struct rtw8822bu_efuse u; + struct rtw8822be_efuse e; + }; +}; + +static inline void +_rtw_write32s_mask(struct rtw_dev *rtwdev, u32 addr, u32 mask, u32 data) +{ + /* 0xC00-0xCFF and 0xE00-0xEFF have the same layout */ + rtw_write32_mask(rtwdev, addr, mask, data); + rtw_write32_mask(rtwdev, addr + 0x200, mask, data); +} + +#define rtw_write32s_mask(rtwdev, addr, mask, data) \ + do { \ + BUILD_BUG_ON((addr) < 0xC00 || (addr) >= 0xD00); \ + \ + _rtw_write32s_mask(rtwdev, addr, mask, data); \ + } while (0) + +/* phy status page0 */ +#define GET_PHY_STAT_P0_PWDB(phy_stat) \ + le32_get_bits(*((__le32 *)(phy_stat) + 0x00), GENMASK(15, 8)) + +/* phy status page1 */ +#define GET_PHY_STAT_P1_PWDB_A(phy_stat) \ + le32_get_bits(*((__le32 *)(phy_stat) + 0x00), GENMASK(15, 8)) +#define GET_PHY_STAT_P1_PWDB_B(phy_stat) \ + le32_get_bits(*((__le32 *)(phy_stat) + 0x00), GENMASK(23, 16)) +#define GET_PHY_STAT_P1_RF_MODE(phy_stat) \ + le32_get_bits(*((__le32 *)(phy_stat) + 0x03), GENMASK(29, 28)) +#define GET_PHY_STAT_P1_L_RXSC(phy_stat) \ + le32_get_bits(*((__le32 *)(phy_stat) + 0x01), GENMASK(11, 8)) +#define GET_PHY_STAT_P1_HT_RXSC(phy_stat) \ + le32_get_bits(*((__le32 *)(phy_stat) + 0x01), GENMASK(15, 12)) + +#define REG_HTSTFWT 0x800 +#define REG_RXPSEL 0x808 +#define BIT_RX_PSEL_RST (BIT(28) | BIT(29)) +#define REG_TXPSEL 0x80c +#define REG_RXCCAMSK 0x814 +#define REG_CCASEL 0x82c +#define REG_PDMFTH 0x830 +#define REG_CCA2ND 0x838 +#define REG_L1WT 0x83c +#define REG_L1PKWT 0x840 +#define REG_MRC 0x850 +#define REG_CLKTRK 0x860 +#define REG_ADCCLK 0x8ac +#define REG_ADC160 0x8c4 +#define REG_ADC40 0x8c8 +#define REG_CDDTXP 0x93c +#define REG_TXPSEL1 0x940 +#define REG_ACBB0 0x948 +#define REG_ACBBRXFIR 0x94c +#define REG_ACGG2TBL 0x958 +#define REG_RXSB 0xa00 +#define REG_ADCINI 0xa04 +#define REG_TXSF2 0xa24 +#define REG_TXSF6 0xa28 +#define REG_RXDESC 0xa2c +#define REG_ENTXCCK 0xa80 +#define REG_AGCTR_A 0xc08 +#define REG_TXDFIR 0xc20 +#define REG_RXIGI_A 0xc50 +#define REG_TRSW 0xca0 +#define REG_RFESEL0 0xcb0 +#define REG_RFESEL8 0xcb4 +#define REG_RFECTL 0xcb8 +#define REG_RFEINV 0xcbc +#define REG_AGCTR_B 0xe08 +#define REG_RXIGI_B 0xe50 +#define REG_ANTWT 0x1904 +#define REG_IQKFAILMSK 0x1bf0 + +#endif diff --git a/drivers/net/wireless/realtek/rtw88/rtw8822b_table.c b/drivers/net/wireless/realtek/rtw88/rtw8822b_table.c new file mode 100644 index 0000000000000000000000000000000000000000..2d2dfb495ce1f6ed11e390912fe9cf6ab4646fbb --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/rtw8822b_table.c @@ -0,0 +1,20783 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#include "main.h" +#include "phy.h" +#include "rtw8822b_table.h" + +static const u32 rtw8822b_mac[] = { + 0x029, 0x000000F9, + 0x420, 0x00000080, + 0x421, 0x0000001F, + 0x428, 0x0000000A, + 0x429, 0x00000010, + 0x430, 0x00000000, + 0x431, 0x00000000, + 0x432, 0x00000000, + 0x433, 0x00000001, + 0x434, 0x00000004, + 0x435, 0x00000005, + 0x436, 0x00000007, + 0x437, 0x00000008, + 0x43C, 0x00000004, + 0x43D, 0x00000005, + 0x43E, 0x00000007, + 0x43F, 0x00000008, + 0x440, 0x0000005D, + 0x441, 0x00000001, + 0x442, 0x00000000, + 0x444, 0x00000010, + 0x445, 0x000000F0, + 0x446, 0x00000001, + 0x447, 0x000000FE, + 0x448, 0x00000000, + 0x449, 0x00000000, + 0x44A, 0x00000000, + 0x44B, 0x00000040, + 0x44C, 0x00000010, + 0x44D, 0x000000F0, + 0x44E, 0x0000003F, + 0x44F, 0x00000000, + 0x450, 0x00000000, + 0x451, 0x00000000, + 0x452, 0x00000000, + 0x453, 0x00000040, + 0x455, 0x00000070, + 0x45E, 0x00000004, + 0x49C, 0x00000010, + 0x49D, 0x000000F0, + 0x49E, 0x00000000, + 0x49F, 0x00000006, + 0x4A0, 0x000000E0, + 0x4A1, 0x00000003, + 0x4A2, 0x00000000, + 0x4A3, 0x00000040, + 0x4A4, 0x00000015, + 0x4A5, 0x000000F0, + 0x4A6, 0x00000000, + 0x4A7, 0x00000006, + 0x4A8, 0x000000E0, + 0x4A9, 0x00000000, + 0x4AA, 0x00000000, + 0x4AB, 0x00000000, + 0x7DA, 0x00000008, + 0x1448, 0x00000006, + 0x144A, 0x00000006, + 0x144C, 0x00000006, + 0x144E, 0x00000006, + 0x4C8, 0x000000FF, + 0x4C9, 0x00000008, + 0x4CA, 0x00000020, + 0x4CB, 0x00000020, + 0x4CC, 0x000000FF, + 0x4CD, 0x000000FF, + 0x4CE, 0x00000001, + 0x4CF, 0x00000008, + 0x500, 0x00000026, + 0x501, 0x000000A2, + 0x502, 0x0000002F, + 0x503, 0x00000000, + 0x504, 0x00000028, + 0x505, 0x000000A3, + 0x506, 0x0000005E, + 0x507, 0x00000000, + 0x508, 0x0000002B, + 0x509, 0x000000A4, + 0x50A, 0x0000005E, + 0x50B, 0x00000000, + 0x50C, 0x0000004F, + 0x50D, 0x000000A4, + 0x50E, 0x00000000, + 0x50F, 0x00000000, + 0x512, 0x0000001C, + 0x514, 0x0000000A, + 0x516, 0x0000000A, + 0x521, 0x0000002F, + 0x525, 0x0000004F, + 0x551, 0x00000010, + 0x559, 0x00000002, + 0x55C, 0x00000050, + 0x55D, 0x000000FF, + 0x577, 0x0000000B, + 0x5BE, 0x00000064, + 0x605, 0x00000030, + 0x608, 0x0000000E, + 0x609, 0x00000022, + 0x60C, 0x00000018, + 0x6A0, 0x000000FF, + 0x6A1, 0x000000FF, + 0x6A2, 0x000000FF, + 0x6A3, 0x000000FF, + 0x6A4, 0x000000FF, + 0x6A5, 0x000000FF, + 0x6DE, 0x00000084, + 0x620, 0x000000FF, + 0x621, 0x000000FF, + 0x622, 0x000000FF, + 0x623, 0x000000FF, + 0x624, 0x000000FF, + 0x625, 0x000000FF, + 0x626, 0x000000FF, + 0x627, 0x000000FF, + 0x638, 0x00000050, + 0x63C, 0x0000000A, + 0x63D, 0x0000000A, + 0x63E, 0x0000000E, + 0x63F, 0x0000000E, + 0x640, 0x00000040, + 0x642, 0x00000040, + 0x643, 0x00000000, + 0x652, 0x000000C8, + 0x66E, 0x00000005, + 0x718, 0x00000040, + 0x7D4, 0x00000098, +}; + +RTW_DECL_TABLE_PHY_COND(rtw8822b_mac, rtw_phy_cfg_mac); + +static const u32 rtw8822b_agc[] = { + 0x80000000, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000003, + 0x81C, 0xFE000003, + 0x81C, 0xFD020003, + 0x81C, 0xFC040003, + 0x81C, 0xFB060003, + 0x81C, 0xFA080003, + 0x81C, 0xF90A0003, + 0x81C, 0xF80C0003, + 0x81C, 0xF70E0003, + 0x81C, 0xF6100003, + 0x81C, 0xF5120003, + 0x81C, 0xF4140003, + 0x81C, 0xF3160003, + 0x81C, 0xF2180003, + 0x81C, 0xF11A0003, + 0x81C, 0xF01C0003, + 0x81C, 0xEF1E0003, + 0x81C, 0xEE200003, + 0x81C, 0xED220003, + 0x81C, 0xEC240003, + 0x81C, 0xEB260003, + 0x81C, 0xEA280003, + 0x81C, 0xE92A0003, + 0x81C, 0xE82C0003, + 0x81C, 0xE72E0003, + 0x81C, 0xE6300003, + 0x81C, 0xE5320003, + 0x81C, 0xC8340003, + 0x81C, 0xC7360003, + 0x81C, 0xC6380003, + 0x81C, 0xC53A0003, + 0x81C, 0xC43C0003, + 0x81C, 0xC33E0003, + 0x81C, 0xC2400003, + 0x81C, 0xC1420003, + 0x81C, 0xC0440003, + 0x81C, 0xA3460003, + 0x81C, 0xA2480003, + 0x81C, 0xA14A0003, + 0x81C, 0xA04C0003, + 0x81C, 0x824E0003, + 0x81C, 0x81500003, + 0x81C, 0x80520003, + 0x81C, 0x64540003, + 0x81C, 0x63560003, + 0x81C, 0x62580003, + 0x81C, 0x445A0003, + 0x81C, 0x435C0003, + 0x81C, 0x425E0003, + 0x81C, 0x41600003, + 0x81C, 0x40620003, + 0x81C, 0x05640003, + 0x81C, 0x04660003, + 0x81C, 0x03680003, + 0x81C, 0x026A0003, + 0x81C, 0x016C0003, + 0x81C, 0x006E0003, + 0x81C, 0x00700003, + 0x81C, 0x00720003, + 0x81C, 0x00740003, + 0x81C, 0x00760003, + 0x81C, 0x00780003, + 0x81C, 0x007A0003, + 0x81C, 0x007C0003, + 0x81C, 0x007E0003, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000003, + 0x81C, 0xF5000003, + 0x81C, 0xF4020003, + 0x81C, 0xF3040003, + 0x81C, 0xF2060003, + 0x81C, 0xF1080003, + 0x81C, 0xF00A0003, + 0x81C, 0xEF0C0003, + 0x81C, 0xEE0E0003, + 0x81C, 0xED100003, + 0x81C, 0xEC120003, + 0x81C, 0xEB140003, + 0x81C, 0xEA160003, + 0x81C, 0xE9180003, + 0x81C, 0xE81A0003, + 0x81C, 0xE71C0003, + 0x81C, 0xE61E0003, + 0x81C, 0xE5200003, + 0x81C, 0xE4220003, + 0x81C, 0xE3240003, + 0x81C, 0xE2260003, + 0x81C, 0xE1280003, + 0x81C, 0xE02A0003, + 0x81C, 0xC32C0003, + 0x81C, 0xC22E0003, + 0x81C, 0xC1300003, + 0x81C, 0xC0320003, + 0x81C, 0xA4340003, + 0x81C, 0xA3360003, + 0x81C, 0xA2380003, + 0x81C, 0xA13A0003, + 0x81C, 0xA03C0003, + 0x81C, 0x823E0003, + 0x81C, 0x81400003, + 0x81C, 0x80420003, + 0x81C, 0x64440003, + 0x81C, 0x63460003, + 0x81C, 0x62480003, + 0x81C, 0x614A0003, + 0x81C, 0x604C0003, + 0x81C, 0x454E0003, + 0x81C, 0x44500003, + 0x81C, 0x43520003, + 0x81C, 0x42540003, + 0x81C, 0x41560003, + 0x81C, 0x40580003, + 0x81C, 0x055A0003, + 0x81C, 0x045C0003, + 0x81C, 0x035E0003, + 0x81C, 0x02600003, + 0x81C, 0x01620003, + 0x81C, 0x00640003, + 0x81C, 0x00660003, + 0x81C, 0x00680003, + 0x81C, 0x006A0003, + 0x81C, 0x006C0003, + 0x81C, 0x006E0003, + 0x81C, 0x00700003, + 0x81C, 0x00720003, + 0x81C, 0x00740003, + 0x81C, 0x00760003, + 0x81C, 0x00780003, + 0x81C, 0x007A0003, + 0x81C, 0x007C0003, + 0x81C, 0x007E0003, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000003, + 0x81C, 0xFD000003, + 0x81C, 0xFC020003, + 0x81C, 0xFB040003, + 0x81C, 0xFA060003, + 0x81C, 0xF9080003, + 0x81C, 0xF80A0003, + 0x81C, 0xF70C0003, + 0x81C, 0xF60E0003, + 0x81C, 0xF5100003, + 0x81C, 0xF4120003, + 0x81C, 0xF3140003, + 0x81C, 0xF2160003, + 0x81C, 0xF1180003, + 0x81C, 0xF01A0003, + 0x81C, 0xEF1C0003, + 0x81C, 0xEE1E0003, + 0x81C, 0xED200003, + 0x81C, 0xEC220003, + 0x81C, 0xEB240003, + 0x81C, 0xEA260003, + 0x81C, 0xE9280003, + 0x81C, 0xE82A0003, + 0x81C, 0xE72C0003, + 0x81C, 0xE62E0003, + 0x81C, 0xE5300003, + 0x81C, 0xC8320003, + 0x81C, 0xC7340003, + 0x81C, 0xC6360003, + 0x81C, 0xC5380003, + 0x81C, 0xC43A0003, + 0x81C, 0xC33C0003, + 0x81C, 0xC23E0003, + 0x81C, 0xC1400003, + 0x81C, 0xC0420003, + 0x81C, 0xA5440003, + 0x81C, 0xA4460003, + 0x81C, 0xA3480003, + 0x81C, 0xA24A0003, + 0x81C, 0xA14C0003, + 0x81C, 0x834E0003, + 0x81C, 0x82500003, + 0x81C, 0x81520003, + 0x81C, 0x80540003, + 0x81C, 0x65560003, + 0x81C, 0x64580003, + 0x81C, 0x635A0003, + 0x81C, 0x625C0003, + 0x81C, 0x435E0003, + 0x81C, 0x42600003, + 0x81C, 0x41620003, + 0x81C, 0x40640003, + 0x81C, 0x06660003, + 0x81C, 0x05680003, + 0x81C, 0x046A0003, + 0x81C, 0x036C0003, + 0x81C, 0x026E0003, + 0x81C, 0x01700003, + 0x81C, 0x00720003, + 0x81C, 0x00740003, + 0x81C, 0x00760003, + 0x81C, 0x00780003, + 0x81C, 0x007A0003, + 0x81C, 0x007C0003, + 0x81C, 0x007E0003, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000003, + 0x81C, 0xFD000003, + 0x81C, 0xFC020003, + 0x81C, 0xFB040003, + 0x81C, 0xFA060003, + 0x81C, 0xF9080003, + 0x81C, 0xF80A0003, + 0x81C, 0xF70C0003, + 0x81C, 0xF60E0003, + 0x81C, 0xF5100003, + 0x81C, 0xF4120003, + 0x81C, 0xF3140003, + 0x81C, 0xF2160003, + 0x81C, 0xF1180003, + 0x81C, 0xF01A0003, + 0x81C, 0xEF1C0003, + 0x81C, 0xEE1E0003, + 0x81C, 0xED200003, + 0x81C, 0xEC220003, + 0x81C, 0xEB240003, + 0x81C, 0xEA260003, + 0x81C, 0xE9280003, + 0x81C, 0xE82A0003, + 0x81C, 0xE72C0003, + 0x81C, 0xE62E0003, + 0x81C, 0xE5300003, + 0x81C, 0xC8320003, + 0x81C, 0xC7340003, + 0x81C, 0xC6360003, + 0x81C, 0xC5380003, + 0x81C, 0xC43A0003, + 0x81C, 0xC33C0003, + 0x81C, 0xC23E0003, + 0x81C, 0xC1400003, + 0x81C, 0xC0420003, + 0x81C, 0xA5440003, + 0x81C, 0xA4460003, + 0x81C, 0xA3480003, + 0x81C, 0xA24A0003, + 0x81C, 0xA14C0003, + 0x81C, 0x834E0003, + 0x81C, 0x82500003, + 0x81C, 0x81520003, + 0x81C, 0x80540003, + 0x81C, 0x65560003, + 0x81C, 0x64580003, + 0x81C, 0x635A0003, + 0x81C, 0x625C0003, + 0x81C, 0x435E0003, + 0x81C, 0x42600003, + 0x81C, 0x41620003, + 0x81C, 0x40640003, + 0x81C, 0x06660003, + 0x81C, 0x05680003, + 0x81C, 0x046A0003, + 0x81C, 0x036C0003, + 0x81C, 0x026E0003, + 0x81C, 0x01700003, + 0x81C, 0x00720003, + 0x81C, 0x00740003, + 0x81C, 0x00760003, + 0x81C, 0x00780003, + 0x81C, 0x007A0003, + 0x81C, 0x007C0003, + 0x81C, 0x007E0003, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000003, + 0x81C, 0xDC000003, + 0x81C, 0xDB020003, + 0x81C, 0xDA040003, + 0x81C, 0xD9060003, + 0x81C, 0xD8080003, + 0x81C, 0xD70A0003, + 0x81C, 0xD60C0003, + 0x81C, 0xD50E0003, + 0x81C, 0xD4100003, + 0x81C, 0xD3120003, + 0x81C, 0xD2140003, + 0x81C, 0xD1160003, + 0x81C, 0xD0180003, + 0x81C, 0xB41A0003, + 0x81C, 0xB31C0003, + 0x81C, 0xB21E0003, + 0x81C, 0xB1200003, + 0x81C, 0xB0220003, + 0x81C, 0xAF240003, + 0x81C, 0xAE260003, + 0x81C, 0xAD280003, + 0x81C, 0xAC2A0003, + 0x81C, 0xAB2C0003, + 0x81C, 0x8C2E0003, + 0x81C, 0x8B300003, + 0x81C, 0x8A320003, + 0x81C, 0x89340003, + 0x81C, 0x88360003, + 0x81C, 0x87380003, + 0x81C, 0x863A0003, + 0x81C, 0x853C0003, + 0x81C, 0x693E0003, + 0x81C, 0x68400003, + 0x81C, 0x67420003, + 0x81C, 0x66440003, + 0x81C, 0x65460003, + 0x81C, 0x48480003, + 0x81C, 0x474A0003, + 0x81C, 0x464C0003, + 0x81C, 0x454E0003, + 0x81C, 0x44500003, + 0x81C, 0x43520003, + 0x81C, 0x27540003, + 0x81C, 0x26560003, + 0x81C, 0x25580003, + 0x81C, 0x245A0003, + 0x81C, 0x235C0003, + 0x81C, 0x045E0003, + 0x81C, 0x03600003, + 0x81C, 0x02620003, + 0x81C, 0x01640003, + 0x81C, 0x00660003, + 0x81C, 0x00680003, + 0x81C, 0x006A0003, + 0x81C, 0x006C0003, + 0x81C, 0x006E0003, + 0x81C, 0x00700003, + 0x81C, 0x00720003, + 0x81C, 0x00740003, + 0x81C, 0x00760003, + 0x81C, 0x00780003, + 0x81C, 0x007A0003, + 0x81C, 0x007C0003, + 0x81C, 0x007E0003, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000003, + 0x81C, 0xFD000003, + 0x81C, 0xFC020003, + 0x81C, 0xFB040003, + 0x81C, 0xFA060003, + 0x81C, 0xF9080003, + 0x81C, 0xF80A0003, + 0x81C, 0xF70C0003, + 0x81C, 0xF60E0003, + 0x81C, 0xF5100003, + 0x81C, 0xF4120003, + 0x81C, 0xF3140003, + 0x81C, 0xF2160003, + 0x81C, 0xF1180003, + 0x81C, 0xF01A0003, + 0x81C, 0xEF1C0003, + 0x81C, 0xEE1E0003, + 0x81C, 0xED200003, + 0x81C, 0xEC220003, + 0x81C, 0xEB240003, + 0x81C, 0xEA260003, + 0x81C, 0xE9280003, + 0x81C, 0xE82A0003, + 0x81C, 0xE72C0003, + 0x81C, 0xE62E0003, + 0x81C, 0xE5300003, + 0x81C, 0xC8320003, + 0x81C, 0xC7340003, + 0x81C, 0xC6360003, + 0x81C, 0xC5380003, + 0x81C, 0xC43A0003, + 0x81C, 0xC33C0003, + 0x81C, 0xC23E0003, + 0x81C, 0xC1400003, + 0x81C, 0xC0420003, + 0x81C, 0xA5440003, + 0x81C, 0xA4460003, + 0x81C, 0xA3480003, + 0x81C, 0xA24A0003, + 0x81C, 0xA14C0003, + 0x81C, 0x834E0003, + 0x81C, 0x82500003, + 0x81C, 0x81520003, + 0x81C, 0x80540003, + 0x81C, 0x65560003, + 0x81C, 0x64580003, + 0x81C, 0x635A0003, + 0x81C, 0x625C0003, + 0x81C, 0x435E0003, + 0x81C, 0x42600003, + 0x81C, 0x41620003, + 0x81C, 0x40640003, + 0x81C, 0x06660003, + 0x81C, 0x05680003, + 0x81C, 0x046A0003, + 0x81C, 0x036C0003, + 0x81C, 0x026E0003, + 0x81C, 0x01700003, + 0x81C, 0x00720003, + 0x81C, 0x00740003, + 0x81C, 0x00760003, + 0x81C, 0x00780003, + 0x81C, 0x007A0003, + 0x81C, 0x007C0003, + 0x81C, 0x007E0003, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000003, + 0x81C, 0xF5000003, + 0x81C, 0xF4020003, + 0x81C, 0xF3040003, + 0x81C, 0xF2060003, + 0x81C, 0xF1080003, + 0x81C, 0xF00A0003, + 0x81C, 0xEF0C0003, + 0x81C, 0xEE0E0003, + 0x81C, 0xED100003, + 0x81C, 0xEC120003, + 0x81C, 0xEB140003, + 0x81C, 0xEA160003, + 0x81C, 0xE9180003, + 0x81C, 0xE81A0003, + 0x81C, 0xE71C0003, + 0x81C, 0xE61E0003, + 0x81C, 0xE5200003, + 0x81C, 0xE4220003, + 0x81C, 0xE3240003, + 0x81C, 0xE2260003, + 0x81C, 0xE1280003, + 0x81C, 0xE02A0003, + 0x81C, 0xC32C0003, + 0x81C, 0xC22E0003, + 0x81C, 0xC1300003, + 0x81C, 0xC0320003, + 0x81C, 0xA4340003, + 0x81C, 0xA3360003, + 0x81C, 0xA2380003, + 0x81C, 0xA13A0003, + 0x81C, 0xA03C0003, + 0x81C, 0x823E0003, + 0x81C, 0x81400003, + 0x81C, 0x80420003, + 0x81C, 0x64440003, + 0x81C, 0x63460003, + 0x81C, 0x62480003, + 0x81C, 0x614A0003, + 0x81C, 0x604C0003, + 0x81C, 0x454E0003, + 0x81C, 0x44500003, + 0x81C, 0x43520003, + 0x81C, 0x42540003, + 0x81C, 0x41560003, + 0x81C, 0x40580003, + 0x81C, 0x055A0003, + 0x81C, 0x045C0003, + 0x81C, 0x035E0003, + 0x81C, 0x02600003, + 0x81C, 0x01620003, + 0x81C, 0x00640003, + 0x81C, 0x00660003, + 0x81C, 0x00680003, + 0x81C, 0x006A0003, + 0x81C, 0x006C0003, + 0x81C, 0x006E0003, + 0x81C, 0x00700003, + 0x81C, 0x00720003, + 0x81C, 0x00740003, + 0x81C, 0x00760003, + 0x81C, 0x00780003, + 0x81C, 0x007A0003, + 0x81C, 0x007C0003, + 0x81C, 0x007E0003, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000003, + 0x81C, 0xF5000003, + 0x81C, 0xF4020003, + 0x81C, 0xF3040003, + 0x81C, 0xF2060003, + 0x81C, 0xF1080003, + 0x81C, 0xF00A0003, + 0x81C, 0xEF0C0003, + 0x81C, 0xEE0E0003, + 0x81C, 0xED100003, + 0x81C, 0xEC120003, + 0x81C, 0xEB140003, + 0x81C, 0xEA160003, + 0x81C, 0xE9180003, + 0x81C, 0xE81A0003, + 0x81C, 0xE71C0003, + 0x81C, 0xE61E0003, + 0x81C, 0xE5200003, + 0x81C, 0xE4220003, + 0x81C, 0xE3240003, + 0x81C, 0xE2260003, + 0x81C, 0xE1280003, + 0x81C, 0xE02A0003, + 0x81C, 0xC32C0003, + 0x81C, 0xC22E0003, + 0x81C, 0xC1300003, + 0x81C, 0xC0320003, + 0x81C, 0xA4340003, + 0x81C, 0xA3360003, + 0x81C, 0xA2380003, + 0x81C, 0xA13A0003, + 0x81C, 0xA03C0003, + 0x81C, 0x823E0003, + 0x81C, 0x81400003, + 0x81C, 0x80420003, + 0x81C, 0x64440003, + 0x81C, 0x63460003, + 0x81C, 0x62480003, + 0x81C, 0x614A0003, + 0x81C, 0x604C0003, + 0x81C, 0x454E0003, + 0x81C, 0x44500003, + 0x81C, 0x43520003, + 0x81C, 0x42540003, + 0x81C, 0x41560003, + 0x81C, 0x40580003, + 0x81C, 0x055A0003, + 0x81C, 0x045C0003, + 0x81C, 0x035E0003, + 0x81C, 0x02600003, + 0x81C, 0x01620003, + 0x81C, 0x00640003, + 0x81C, 0x00660003, + 0x81C, 0x00680003, + 0x81C, 0x006A0003, + 0x81C, 0x006C0003, + 0x81C, 0x006E0003, + 0x81C, 0x00700003, + 0x81C, 0x00720003, + 0x81C, 0x00740003, + 0x81C, 0x00760003, + 0x81C, 0x00780003, + 0x81C, 0x007A0003, + 0x81C, 0x007C0003, + 0x81C, 0x007E0003, + 0x90000008, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000003, + 0x81C, 0xFE000003, + 0x81C, 0xFD020003, + 0x81C, 0xFC040003, + 0x81C, 0xFB060003, + 0x81C, 0xFA080003, + 0x81C, 0xF90A0003, + 0x81C, 0xF80C0003, + 0x81C, 0xF70E0003, + 0x81C, 0xF6100003, + 0x81C, 0xF5120003, + 0x81C, 0xF4140003, + 0x81C, 0xF3160003, + 0x81C, 0xF2180003, + 0x81C, 0xF11A0003, + 0x81C, 0xF01C0003, + 0x81C, 0xEF1E0003, + 0x81C, 0xEE200003, + 0x81C, 0xED220003, + 0x81C, 0xEC240003, + 0x81C, 0xEB260003, + 0x81C, 0xEA280003, + 0x81C, 0xE92A0003, + 0x81C, 0xE82C0003, + 0x81C, 0xE72E0003, + 0x81C, 0xE6300003, + 0x81C, 0xE5320003, + 0x81C, 0xC8340003, + 0x81C, 0xC7360003, + 0x81C, 0xC6380003, + 0x81C, 0xC53A0003, + 0x81C, 0xC43C0003, + 0x81C, 0xC33E0003, + 0x81C, 0xC2400003, + 0x81C, 0xC1420003, + 0x81C, 0xC0440003, + 0x81C, 0xA3460003, + 0x81C, 0xA2480003, + 0x81C, 0xA14A0003, + 0x81C, 0xA04C0003, + 0x81C, 0x824E0003, + 0x81C, 0x81500003, + 0x81C, 0x80520003, + 0x81C, 0x64540003, + 0x81C, 0x63560003, + 0x81C, 0x62580003, + 0x81C, 0x445A0003, + 0x81C, 0x435C0003, + 0x81C, 0x425E0003, + 0x81C, 0x41600003, + 0x81C, 0x40620003, + 0x81C, 0x05640003, + 0x81C, 0x04660003, + 0x81C, 0x03680003, + 0x81C, 0x026A0003, + 0x81C, 0x016C0003, + 0x81C, 0x006E0003, + 0x81C, 0x00700003, + 0x81C, 0x00720003, + 0x81C, 0x00740003, + 0x81C, 0x00760003, + 0x81C, 0x00780003, + 0x81C, 0x007A0003, + 0x81C, 0x007C0003, + 0x81C, 0x007E0003, + 0x90000009, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000003, + 0x81C, 0xF5000003, + 0x81C, 0xF4020003, + 0x81C, 0xF3040003, + 0x81C, 0xF2060003, + 0x81C, 0xF1080003, + 0x81C, 0xF00A0003, + 0x81C, 0xEF0C0003, + 0x81C, 0xEE0E0003, + 0x81C, 0xED100003, + 0x81C, 0xEC120003, + 0x81C, 0xEB140003, + 0x81C, 0xEA160003, + 0x81C, 0xE9180003, + 0x81C, 0xE81A0003, + 0x81C, 0xE71C0003, + 0x81C, 0xE61E0003, + 0x81C, 0xE5200003, + 0x81C, 0xE4220003, + 0x81C, 0xE3240003, + 0x81C, 0xE2260003, + 0x81C, 0xE1280003, + 0x81C, 0xE02A0003, + 0x81C, 0xC32C0003, + 0x81C, 0xC22E0003, + 0x81C, 0xC1300003, + 0x81C, 0xC0320003, + 0x81C, 0xA4340003, + 0x81C, 0xA3360003, + 0x81C, 0xA2380003, + 0x81C, 0xA13A0003, + 0x81C, 0xA03C0003, + 0x81C, 0x823E0003, + 0x81C, 0x81400003, + 0x81C, 0x80420003, + 0x81C, 0x64440003, + 0x81C, 0x63460003, + 0x81C, 0x62480003, + 0x81C, 0x614A0003, + 0x81C, 0x604C0003, + 0x81C, 0x454E0003, + 0x81C, 0x44500003, + 0x81C, 0x43520003, + 0x81C, 0x42540003, + 0x81C, 0x41560003, + 0x81C, 0x40580003, + 0x81C, 0x055A0003, + 0x81C, 0x045C0003, + 0x81C, 0x035E0003, + 0x81C, 0x02600003, + 0x81C, 0x01620003, + 0x81C, 0x00640003, + 0x81C, 0x00660003, + 0x81C, 0x00680003, + 0x81C, 0x006A0003, + 0x81C, 0x006C0003, + 0x81C, 0x006E0003, + 0x81C, 0x00700003, + 0x81C, 0x00720003, + 0x81C, 0x00740003, + 0x81C, 0x00760003, + 0x81C, 0x00780003, + 0x81C, 0x007A0003, + 0x81C, 0x007C0003, + 0x81C, 0x007E0003, + 0x9000000a, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000003, + 0x81C, 0xFE000003, + 0x81C, 0xFD020003, + 0x81C, 0xFC040003, + 0x81C, 0xFB060003, + 0x81C, 0xFA080003, + 0x81C, 0xF90A0003, + 0x81C, 0xF80C0003, + 0x81C, 0xF70E0003, + 0x81C, 0xF6100003, + 0x81C, 0xF5120003, + 0x81C, 0xF4140003, + 0x81C, 0xF3160003, + 0x81C, 0xF2180003, + 0x81C, 0xF11A0003, + 0x81C, 0xF01C0003, + 0x81C, 0xEF1E0003, + 0x81C, 0xEE200003, + 0x81C, 0xED220003, + 0x81C, 0xEC240003, + 0x81C, 0xEB260003, + 0x81C, 0xEA280003, + 0x81C, 0xE92A0003, + 0x81C, 0xE82C0003, + 0x81C, 0xE72E0003, + 0x81C, 0xE6300003, + 0x81C, 0xE5320003, + 0x81C, 0xC8340003, + 0x81C, 0xC7360003, + 0x81C, 0xC6380003, + 0x81C, 0xC53A0003, + 0x81C, 0xC43C0003, + 0x81C, 0xC33E0003, + 0x81C, 0xC2400003, + 0x81C, 0xC1420003, + 0x81C, 0xC0440003, + 0x81C, 0xA3460003, + 0x81C, 0xA2480003, + 0x81C, 0xA14A0003, + 0x81C, 0xA04C0003, + 0x81C, 0x824E0003, + 0x81C, 0x81500003, + 0x81C, 0x80520003, + 0x81C, 0x64540003, + 0x81C, 0x63560003, + 0x81C, 0x62580003, + 0x81C, 0x445A0003, + 0x81C, 0x435C0003, + 0x81C, 0x425E0003, + 0x81C, 0x41600003, + 0x81C, 0x40620003, + 0x81C, 0x05640003, + 0x81C, 0x04660003, + 0x81C, 0x03680003, + 0x81C, 0x026A0003, + 0x81C, 0x016C0003, + 0x81C, 0x006E0003, + 0x81C, 0x00700003, + 0x81C, 0x00720003, + 0x81C, 0x00740003, + 0x81C, 0x00760003, + 0x81C, 0x00780003, + 0x81C, 0x007A0003, + 0x81C, 0x007C0003, + 0x81C, 0x007E0003, + 0x9000000b, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000003, + 0x81C, 0xF5000003, + 0x81C, 0xF4020003, + 0x81C, 0xF3040003, + 0x81C, 0xF2060003, + 0x81C, 0xF1080003, + 0x81C, 0xF00A0003, + 0x81C, 0xEF0C0003, + 0x81C, 0xEE0E0003, + 0x81C, 0xED100003, + 0x81C, 0xEC120003, + 0x81C, 0xEB140003, + 0x81C, 0xEA160003, + 0x81C, 0xE9180003, + 0x81C, 0xE81A0003, + 0x81C, 0xE71C0003, + 0x81C, 0xE61E0003, + 0x81C, 0xE5200003, + 0x81C, 0xE4220003, + 0x81C, 0xE3240003, + 0x81C, 0xE2260003, + 0x81C, 0xE1280003, + 0x81C, 0xE02A0003, + 0x81C, 0xC32C0003, + 0x81C, 0xC22E0003, + 0x81C, 0xC1300003, + 0x81C, 0xC0320003, + 0x81C, 0xA4340003, + 0x81C, 0xA3360003, + 0x81C, 0xA2380003, + 0x81C, 0xA13A0003, + 0x81C, 0xA03C0003, + 0x81C, 0x823E0003, + 0x81C, 0x81400003, + 0x81C, 0x80420003, + 0x81C, 0x64440003, + 0x81C, 0x63460003, + 0x81C, 0x62480003, + 0x81C, 0x614A0003, + 0x81C, 0x604C0003, + 0x81C, 0x454E0003, + 0x81C, 0x44500003, + 0x81C, 0x43520003, + 0x81C, 0x42540003, + 0x81C, 0x41560003, + 0x81C, 0x40580003, + 0x81C, 0x055A0003, + 0x81C, 0x045C0003, + 0x81C, 0x035E0003, + 0x81C, 0x02600003, + 0x81C, 0x01620003, + 0x81C, 0x00640003, + 0x81C, 0x00660003, + 0x81C, 0x00680003, + 0x81C, 0x006A0003, + 0x81C, 0x006C0003, + 0x81C, 0x006E0003, + 0x81C, 0x00700003, + 0x81C, 0x00720003, + 0x81C, 0x00740003, + 0x81C, 0x00760003, + 0x81C, 0x00780003, + 0x81C, 0x007A0003, + 0x81C, 0x007C0003, + 0x81C, 0x007E0003, + 0x9000000c, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000003, + 0x81C, 0xFD000003, + 0x81C, 0xFC020003, + 0x81C, 0xFB040003, + 0x81C, 0xFA060003, + 0x81C, 0xF9080003, + 0x81C, 0xF80A0003, + 0x81C, 0xF70C0003, + 0x81C, 0xF60E0003, + 0x81C, 0xF5100003, + 0x81C, 0xF4120003, + 0x81C, 0xF3140003, + 0x81C, 0xF2160003, + 0x81C, 0xF1180003, + 0x81C, 0xF01A0003, + 0x81C, 0xEF1C0003, + 0x81C, 0xEE1E0003, + 0x81C, 0xED200003, + 0x81C, 0xEC220003, + 0x81C, 0xEB240003, + 0x81C, 0xEA260003, + 0x81C, 0xE9280003, + 0x81C, 0xE82A0003, + 0x81C, 0xE72C0003, + 0x81C, 0xE62E0003, + 0x81C, 0xE5300003, + 0x81C, 0xC8320003, + 0x81C, 0xC7340003, + 0x81C, 0xC6360003, + 0x81C, 0xC5380003, + 0x81C, 0xC43A0003, + 0x81C, 0xC33C0003, + 0x81C, 0xC23E0003, + 0x81C, 0xC1400003, + 0x81C, 0xC0420003, + 0x81C, 0xA5440003, + 0x81C, 0xA4460003, + 0x81C, 0xA3480003, + 0x81C, 0xA24A0003, + 0x81C, 0xA14C0003, + 0x81C, 0x834E0003, + 0x81C, 0x82500003, + 0x81C, 0x81520003, + 0x81C, 0x80540003, + 0x81C, 0x65560003, + 0x81C, 0x64580003, + 0x81C, 0x635A0003, + 0x81C, 0x625C0003, + 0x81C, 0x435E0003, + 0x81C, 0x42600003, + 0x81C, 0x41620003, + 0x81C, 0x40640003, + 0x81C, 0x06660003, + 0x81C, 0x05680003, + 0x81C, 0x046A0003, + 0x81C, 0x036C0003, + 0x81C, 0x026E0003, + 0x81C, 0x01700003, + 0x81C, 0x00720003, + 0x81C, 0x00740003, + 0x81C, 0x00760003, + 0x81C, 0x00780003, + 0x81C, 0x007A0003, + 0x81C, 0x007C0003, + 0x81C, 0x007E0003, + 0x9000000d, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000003, + 0x81C, 0xFE000003, + 0x81C, 0xFD020003, + 0x81C, 0xFC040003, + 0x81C, 0xFB060003, + 0x81C, 0xFA080003, + 0x81C, 0xF90A0003, + 0x81C, 0xF80C0003, + 0x81C, 0xF70E0003, + 0x81C, 0xF6100003, + 0x81C, 0xF5120003, + 0x81C, 0xF4140003, + 0x81C, 0xF3160003, + 0x81C, 0xF2180003, + 0x81C, 0xF11A0003, + 0x81C, 0xF01C0003, + 0x81C, 0xEF1E0003, + 0x81C, 0xEE200003, + 0x81C, 0xED220003, + 0x81C, 0xEC240003, + 0x81C, 0xEB260003, + 0x81C, 0xEA280003, + 0x81C, 0xE92A0003, + 0x81C, 0xE82C0003, + 0x81C, 0xE72E0003, + 0x81C, 0xE6300003, + 0x81C, 0xE5320003, + 0x81C, 0xC8340003, + 0x81C, 0xC7360003, + 0x81C, 0xC6380003, + 0x81C, 0xC53A0003, + 0x81C, 0xC43C0003, + 0x81C, 0xC33E0003, + 0x81C, 0xC2400003, + 0x81C, 0xC1420003, + 0x81C, 0xC0440003, + 0x81C, 0xA3460003, + 0x81C, 0xA2480003, + 0x81C, 0xA14A0003, + 0x81C, 0xA04C0003, + 0x81C, 0x824E0003, + 0x81C, 0x81500003, + 0x81C, 0x80520003, + 0x81C, 0x64540003, + 0x81C, 0x63560003, + 0x81C, 0x62580003, + 0x81C, 0x445A0003, + 0x81C, 0x435C0003, + 0x81C, 0x425E0003, + 0x81C, 0x41600003, + 0x81C, 0x40620003, + 0x81C, 0x05640003, + 0x81C, 0x04660003, + 0x81C, 0x03680003, + 0x81C, 0x026A0003, + 0x81C, 0x016C0003, + 0x81C, 0x006E0003, + 0x81C, 0x00700003, + 0x81C, 0x00720003, + 0x81C, 0x00740003, + 0x81C, 0x00760003, + 0x81C, 0x00780003, + 0x81C, 0x007A0003, + 0x81C, 0x007C0003, + 0x81C, 0x007E0003, + 0x9000000e, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000003, + 0x81C, 0xFE000003, + 0x81C, 0xFD020003, + 0x81C, 0xFC040003, + 0x81C, 0xFB060003, + 0x81C, 0xFA080003, + 0x81C, 0xF90A0003, + 0x81C, 0xF80C0003, + 0x81C, 0xF70E0003, + 0x81C, 0xF6100003, + 0x81C, 0xF5120003, + 0x81C, 0xF4140003, + 0x81C, 0xF3160003, + 0x81C, 0xF2180003, + 0x81C, 0xF11A0003, + 0x81C, 0xF01C0003, + 0x81C, 0xEF1E0003, + 0x81C, 0xEE200003, + 0x81C, 0xED220003, + 0x81C, 0xEC240003, + 0x81C, 0xEB260003, + 0x81C, 0xEA280003, + 0x81C, 0xE92A0003, + 0x81C, 0xE82C0003, + 0x81C, 0xE72E0003, + 0x81C, 0xE6300003, + 0x81C, 0xE5320003, + 0x81C, 0xC8340003, + 0x81C, 0xC7360003, + 0x81C, 0xC6380003, + 0x81C, 0xC53A0003, + 0x81C, 0xC43C0003, + 0x81C, 0xC33E0003, + 0x81C, 0xC2400003, + 0x81C, 0xC1420003, + 0x81C, 0xC0440003, + 0x81C, 0xA3460003, + 0x81C, 0xA2480003, + 0x81C, 0xA14A0003, + 0x81C, 0xA04C0003, + 0x81C, 0x824E0003, + 0x81C, 0x81500003, + 0x81C, 0x80520003, + 0x81C, 0x64540003, + 0x81C, 0x63560003, + 0x81C, 0x62580003, + 0x81C, 0x445A0003, + 0x81C, 0x435C0003, + 0x81C, 0x425E0003, + 0x81C, 0x41600003, + 0x81C, 0x40620003, + 0x81C, 0x05640003, + 0x81C, 0x04660003, + 0x81C, 0x03680003, + 0x81C, 0x026A0003, + 0x81C, 0x016C0003, + 0x81C, 0x006E0003, + 0x81C, 0x00700003, + 0x81C, 0x00720003, + 0x81C, 0x00740003, + 0x81C, 0x00760003, + 0x81C, 0x00780003, + 0x81C, 0x007A0003, + 0x81C, 0x007C0003, + 0x81C, 0x007E0003, + 0x9000000f, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000003, + 0x81C, 0xFC000003, + 0x81C, 0xFB020003, + 0x81C, 0xFA040003, + 0x81C, 0xF9060003, + 0x81C, 0xF8080003, + 0x81C, 0xF70A0003, + 0x81C, 0xF60C0003, + 0x81C, 0xF50E0003, + 0x81C, 0xF4100003, + 0x81C, 0xF3120003, + 0x81C, 0xF2140003, + 0x81C, 0xF1160003, + 0x81C, 0xF0180003, + 0x81C, 0xEF1A0003, + 0x81C, 0xEE1C0003, + 0x81C, 0xED1E0003, + 0x81C, 0xEC200003, + 0x81C, 0xEB220003, + 0x81C, 0xEA240003, + 0x81C, 0xE9260003, + 0x81C, 0xE8280003, + 0x81C, 0xE72A0003, + 0x81C, 0xE62C0003, + 0x81C, 0xE52E0003, + 0x81C, 0xC8300003, + 0x81C, 0xC7320003, + 0x81C, 0xC6340003, + 0x81C, 0xC5360003, + 0x81C, 0xC4380003, + 0x81C, 0xC33A0003, + 0x81C, 0xC23C0003, + 0x81C, 0xC13E0003, + 0x81C, 0xA4400003, + 0x81C, 0xA3420003, + 0x81C, 0xA2440003, + 0x81C, 0xA1460003, + 0x81C, 0xA0480003, + 0x81C, 0x684A0003, + 0x81C, 0x674C0003, + 0x81C, 0x664E0003, + 0x81C, 0x65500003, + 0x81C, 0x64520003, + 0x81C, 0x63540003, + 0x81C, 0x44560003, + 0x81C, 0x43580003, + 0x81C, 0x425A0003, + 0x81C, 0x415C0003, + 0x81C, 0x405E0003, + 0x81C, 0x23600003, + 0x81C, 0x22620003, + 0x81C, 0x21640003, + 0x81C, 0x03660003, + 0x81C, 0x02680003, + 0x81C, 0x016A0003, + 0x81C, 0x006C0003, + 0x81C, 0x006E0003, + 0x81C, 0x00700003, + 0x81C, 0x00720003, + 0x81C, 0x00740003, + 0x81C, 0x00760003, + 0x81C, 0x00780003, + 0x81C, 0x007A0003, + 0x81C, 0x007C0003, + 0x81C, 0x007E0003, + 0x90000010, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000003, + 0x81C, 0xFD000003, + 0x81C, 0xFC020003, + 0x81C, 0xFB040003, + 0x81C, 0xFA060003, + 0x81C, 0xF9080003, + 0x81C, 0xF80A0003, + 0x81C, 0xF70C0003, + 0x81C, 0xF60E0003, + 0x81C, 0xF5100003, + 0x81C, 0xF4120003, + 0x81C, 0xF3140003, + 0x81C, 0xF2160003, + 0x81C, 0xF1180003, + 0x81C, 0xF01A0003, + 0x81C, 0xEF1C0003, + 0x81C, 0xEE1E0003, + 0x81C, 0xED200003, + 0x81C, 0xEC220003, + 0x81C, 0xEB240003, + 0x81C, 0xEA260003, + 0x81C, 0xE9280003, + 0x81C, 0xE82A0003, + 0x81C, 0xE72C0003, + 0x81C, 0xE62E0003, + 0x81C, 0xE5300003, + 0x81C, 0xC8320003, + 0x81C, 0xC7340003, + 0x81C, 0xC6360003, + 0x81C, 0xC5380003, + 0x81C, 0xC43A0003, + 0x81C, 0xC33C0003, + 0x81C, 0xC23E0003, + 0x81C, 0xC1400003, + 0x81C, 0xC0420003, + 0x81C, 0xA5440003, + 0x81C, 0xA4460003, + 0x81C, 0xA3480003, + 0x81C, 0xA24A0003, + 0x81C, 0xA14C0003, + 0x81C, 0x834E0003, + 0x81C, 0x82500003, + 0x81C, 0x81520003, + 0x81C, 0x80540003, + 0x81C, 0x65560003, + 0x81C, 0x64580003, + 0x81C, 0x635A0003, + 0x81C, 0x625C0003, + 0x81C, 0x435E0003, + 0x81C, 0x42600003, + 0x81C, 0x41620003, + 0x81C, 0x40640003, + 0x81C, 0x06660003, + 0x81C, 0x05680003, + 0x81C, 0x046A0003, + 0x81C, 0x036C0003, + 0x81C, 0x026E0003, + 0x81C, 0x01700003, + 0x81C, 0x00720003, + 0x81C, 0x00740003, + 0x81C, 0x00760003, + 0x81C, 0x00780003, + 0x81C, 0x007A0003, + 0x81C, 0x007C0003, + 0x81C, 0x007E0003, + 0x90000012, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000003, + 0x81C, 0xDC000003, + 0x81C, 0xDB020003, + 0x81C, 0xDA040003, + 0x81C, 0xD9060003, + 0x81C, 0xD8080003, + 0x81C, 0xD70A0003, + 0x81C, 0xD60C0003, + 0x81C, 0xD50E0003, + 0x81C, 0xD4100003, + 0x81C, 0xD3120003, + 0x81C, 0xD2140003, + 0x81C, 0xD1160003, + 0x81C, 0xD0180003, + 0x81C, 0xB41A0003, + 0x81C, 0xB31C0003, + 0x81C, 0xB21E0003, + 0x81C, 0xB1200003, + 0x81C, 0xB0220003, + 0x81C, 0xAF240003, + 0x81C, 0xAE260003, + 0x81C, 0xAD280003, + 0x81C, 0xAC2A0003, + 0x81C, 0xAB2C0003, + 0x81C, 0x8C2E0003, + 0x81C, 0x8B300003, + 0x81C, 0x8A320003, + 0x81C, 0x89340003, + 0x81C, 0x88360003, + 0x81C, 0x87380003, + 0x81C, 0x863A0003, + 0x81C, 0x853C0003, + 0x81C, 0x693E0003, + 0x81C, 0x68400003, + 0x81C, 0x67420003, + 0x81C, 0x66440003, + 0x81C, 0x65460003, + 0x81C, 0x48480003, + 0x81C, 0x474A0003, + 0x81C, 0x464C0003, + 0x81C, 0x454E0003, + 0x81C, 0x44500003, + 0x81C, 0x43520003, + 0x81C, 0x27540003, + 0x81C, 0x26560003, + 0x81C, 0x25580003, + 0x81C, 0x245A0003, + 0x81C, 0x235C0003, + 0x81C, 0x045E0003, + 0x81C, 0x03600003, + 0x81C, 0x02620003, + 0x81C, 0x01640003, + 0x81C, 0x00660003, + 0x81C, 0x00680003, + 0x81C, 0x006A0003, + 0x81C, 0x006C0003, + 0x81C, 0x006E0003, + 0x81C, 0x00700003, + 0x81C, 0x00720003, + 0x81C, 0x00740003, + 0x81C, 0x00760003, + 0x81C, 0x00780003, + 0x81C, 0x007A0003, + 0x81C, 0x007C0003, + 0x81C, 0x007E0003, + 0xA0000000, 0x00000000, + 0x81C, 0xFF000003, + 0x81C, 0xFE000003, + 0x81C, 0xFD020003, + 0x81C, 0xFC040003, + 0x81C, 0xFB060003, + 0x81C, 0xFA080003, + 0x81C, 0xF90A0003, + 0x81C, 0xF80C0003, + 0x81C, 0xF70E0003, + 0x81C, 0xF6100003, + 0x81C, 0xF5120003, + 0x81C, 0xF4140003, + 0x81C, 0xF3160003, + 0x81C, 0xF2180003, + 0x81C, 0xF11A0003, + 0x81C, 0xF01C0003, + 0x81C, 0xEF1E0003, + 0x81C, 0xEE200003, + 0x81C, 0xED220003, + 0x81C, 0xEC240003, + 0x81C, 0xEB260003, + 0x81C, 0xEA280003, + 0x81C, 0xE92A0003, + 0x81C, 0xE82C0003, + 0x81C, 0xE72E0003, + 0x81C, 0xE6300003, + 0x81C, 0xE5320003, + 0x81C, 0xC8340003, + 0x81C, 0xC7360003, + 0x81C, 0xC6380003, + 0x81C, 0xC53A0003, + 0x81C, 0xC43C0003, + 0x81C, 0xC33E0003, + 0x81C, 0xC2400003, + 0x81C, 0xC1420003, + 0x81C, 0xC0440003, + 0x81C, 0xA3460003, + 0x81C, 0xA2480003, + 0x81C, 0xA14A0003, + 0x81C, 0xA04C0003, + 0x81C, 0x824E0003, + 0x81C, 0x81500003, + 0x81C, 0x80520003, + 0x81C, 0x64540003, + 0x81C, 0x63560003, + 0x81C, 0x62580003, + 0x81C, 0x445A0003, + 0x81C, 0x435C0003, + 0x81C, 0x425E0003, + 0x81C, 0x41600003, + 0x81C, 0x40620003, + 0x81C, 0x05640003, + 0x81C, 0x04660003, + 0x81C, 0x03680003, + 0x81C, 0x026A0003, + 0x81C, 0x016C0003, + 0x81C, 0x006E0003, + 0x81C, 0x00700003, + 0x81C, 0x00720003, + 0x81C, 0x00740003, + 0x81C, 0x00760003, + 0x81C, 0x00780003, + 0x81C, 0x007A0003, + 0x81C, 0x007C0003, + 0x81C, 0x007E0003, + 0xB0000000, 0x00000000, + 0x80000000, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFD000103, + 0x81C, 0xFC020103, + 0x81C, 0xFB040103, + 0x81C, 0xFA060103, + 0x81C, 0xF9080103, + 0x81C, 0xF80A0103, + 0x81C, 0xF70C0103, + 0x81C, 0xF60E0103, + 0x81C, 0xF5100103, + 0x81C, 0xF4120103, + 0x81C, 0xF3140103, + 0x81C, 0xF2160103, + 0x81C, 0xF1180103, + 0x81C, 0xF01A0103, + 0x81C, 0xEE1C0103, + 0x81C, 0xED1E0103, + 0x81C, 0xEC200103, + 0x81C, 0xEB220103, + 0x81C, 0xEA240103, + 0x81C, 0xE9260103, + 0x81C, 0xE8280103, + 0x81C, 0xE72A0103, + 0x81C, 0xE62C0103, + 0x81C, 0xE52E0103, + 0x81C, 0xE4300103, + 0x81C, 0xE3320103, + 0x81C, 0xE2340103, + 0x81C, 0xC5360103, + 0x81C, 0xC4380103, + 0x81C, 0xC33A0103, + 0x81C, 0xC23C0103, + 0x81C, 0xA53E0103, + 0x81C, 0xA4400103, + 0x81C, 0xA3420103, + 0x81C, 0xA2440103, + 0x81C, 0xA1460103, + 0x81C, 0x83480103, + 0x81C, 0x824A0103, + 0x81C, 0x814C0103, + 0x81C, 0x804E0103, + 0x81C, 0x63500103, + 0x81C, 0x62520103, + 0x81C, 0x61540103, + 0x81C, 0x43560103, + 0x81C, 0x42580103, + 0x81C, 0x415A0103, + 0x81C, 0x405C0103, + 0x81C, 0x225E0103, + 0x81C, 0x21600103, + 0x81C, 0x20620103, + 0x81C, 0x03640103, + 0x81C, 0x02660103, + 0x81C, 0x01680103, + 0x81C, 0x006A0103, + 0x81C, 0x006C0103, + 0x81C, 0x006E0103, + 0x81C, 0x00700103, + 0x81C, 0x00720103, + 0x81C, 0x00740103, + 0x81C, 0x00760103, + 0x81C, 0x00780103, + 0x81C, 0x007A0103, + 0x81C, 0x007C0103, + 0x81C, 0x007E0103, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF8000103, + 0x81C, 0xF7020103, + 0x81C, 0xF6040103, + 0x81C, 0xF5060103, + 0x81C, 0xF4080103, + 0x81C, 0xF30A0103, + 0x81C, 0xF20C0103, + 0x81C, 0xF10E0103, + 0x81C, 0xF0100103, + 0x81C, 0xEF120103, + 0x81C, 0xEE140103, + 0x81C, 0xED160103, + 0x81C, 0xEC180103, + 0x81C, 0xEB1A0103, + 0x81C, 0xEA1C0103, + 0x81C, 0xE91E0103, + 0x81C, 0xE8200103, + 0x81C, 0xE7220103, + 0x81C, 0xE6240103, + 0x81C, 0xE5260103, + 0x81C, 0xE4280103, + 0x81C, 0xE32A0103, + 0x81C, 0xC32C0103, + 0x81C, 0xC22E0103, + 0x81C, 0xC1300103, + 0x81C, 0xC0320103, + 0x81C, 0xA3340103, + 0x81C, 0xA2360103, + 0x81C, 0xA1380103, + 0x81C, 0xA03A0103, + 0x81C, 0x823C0103, + 0x81C, 0x813E0103, + 0x81C, 0x80400103, + 0x81C, 0x63420103, + 0x81C, 0x62440103, + 0x81C, 0x61460103, + 0x81C, 0x60480103, + 0x81C, 0x424A0103, + 0x81C, 0x414C0103, + 0x81C, 0x404E0103, + 0x81C, 0x06500103, + 0x81C, 0x05520103, + 0x81C, 0x04540103, + 0x81C, 0x03560103, + 0x81C, 0x02580103, + 0x81C, 0x015A0103, + 0x81C, 0x005C0103, + 0x81C, 0x005E0103, + 0x81C, 0x00600103, + 0x81C, 0x00620103, + 0x81C, 0x00640103, + 0x81C, 0x00660103, + 0x81C, 0x00680103, + 0x81C, 0x006A0103, + 0x81C, 0x006C0103, + 0x81C, 0x006E0103, + 0x81C, 0x00700103, + 0x81C, 0x00720103, + 0x81C, 0x00740103, + 0x81C, 0x00760103, + 0x81C, 0x00780103, + 0x81C, 0x007A0103, + 0x81C, 0x007C0103, + 0x81C, 0x007E0103, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF8000103, + 0x81C, 0xF7020103, + 0x81C, 0xF6040103, + 0x81C, 0xF5060103, + 0x81C, 0xF4080103, + 0x81C, 0xF30A0103, + 0x81C, 0xF20C0103, + 0x81C, 0xF10E0103, + 0x81C, 0xF0100103, + 0x81C, 0xEF120103, + 0x81C, 0xEE140103, + 0x81C, 0xED160103, + 0x81C, 0xEC180103, + 0x81C, 0xEB1A0103, + 0x81C, 0xEA1C0103, + 0x81C, 0xE91E0103, + 0x81C, 0xE8200103, + 0x81C, 0xE7220103, + 0x81C, 0xE6240103, + 0x81C, 0xE5260103, + 0x81C, 0xE4280103, + 0x81C, 0xE32A0103, + 0x81C, 0xC32C0103, + 0x81C, 0xC22E0103, + 0x81C, 0xC1300103, + 0x81C, 0xC0320103, + 0x81C, 0xA3340103, + 0x81C, 0xA2360103, + 0x81C, 0xA1380103, + 0x81C, 0xA03A0103, + 0x81C, 0x823C0103, + 0x81C, 0x813E0103, + 0x81C, 0x80400103, + 0x81C, 0x63420103, + 0x81C, 0x62440103, + 0x81C, 0x61460103, + 0x81C, 0x60480103, + 0x81C, 0x424A0103, + 0x81C, 0x414C0103, + 0x81C, 0x404E0103, + 0x81C, 0x22500103, + 0x81C, 0x21520103, + 0x81C, 0x20540103, + 0x81C, 0x03560103, + 0x81C, 0x02580103, + 0x81C, 0x015A0103, + 0x81C, 0x005C0103, + 0x81C, 0x005E0103, + 0x81C, 0x00600103, + 0x81C, 0x00620103, + 0x81C, 0x00640103, + 0x81C, 0x00660103, + 0x81C, 0x00680103, + 0x81C, 0x006A0103, + 0x81C, 0x006C0103, + 0x81C, 0x006E0103, + 0x81C, 0x00700103, + 0x81C, 0x00720103, + 0x81C, 0x00740103, + 0x81C, 0x00760103, + 0x81C, 0x00780103, + 0x81C, 0x007A0103, + 0x81C, 0x007C0103, + 0x81C, 0x007E0103, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFE000103, + 0x81C, 0xFD020103, + 0x81C, 0xFC040103, + 0x81C, 0xFB060103, + 0x81C, 0xFA080103, + 0x81C, 0xF90A0103, + 0x81C, 0xF80C0103, + 0x81C, 0xF70E0103, + 0x81C, 0xF6100103, + 0x81C, 0xF5120103, + 0x81C, 0xF4140103, + 0x81C, 0xF3160103, + 0x81C, 0xF2180103, + 0x81C, 0xF11A0103, + 0x81C, 0xF01C0103, + 0x81C, 0xEF1E0103, + 0x81C, 0xEE200103, + 0x81C, 0xED220103, + 0x81C, 0xEC240103, + 0x81C, 0xEB260103, + 0x81C, 0xEA280103, + 0x81C, 0xE92A0103, + 0x81C, 0xE82C0103, + 0x81C, 0xE72E0103, + 0x81C, 0xE6300103, + 0x81C, 0xE5320103, + 0x81C, 0xE4340103, + 0x81C, 0xE3360103, + 0x81C, 0xC6380103, + 0x81C, 0xC53A0103, + 0x81C, 0xC43C0103, + 0x81C, 0xC33E0103, + 0x81C, 0xA5400103, + 0x81C, 0xA4420103, + 0x81C, 0xA3440103, + 0x81C, 0xA2460103, + 0x81C, 0xA1480103, + 0x81C, 0xA04A0103, + 0x81C, 0x824C0103, + 0x81C, 0x814E0103, + 0x81C, 0x80500103, + 0x81C, 0x64520103, + 0x81C, 0x63540103, + 0x81C, 0x62560103, + 0x81C, 0x61580103, + 0x81C, 0x605A0103, + 0x81C, 0x235C0103, + 0x81C, 0x225E0103, + 0x81C, 0x21600103, + 0x81C, 0x20620103, + 0x81C, 0x03640103, + 0x81C, 0x02660103, + 0x81C, 0x01680103, + 0x81C, 0x006A0103, + 0x81C, 0x006C0103, + 0x81C, 0x006E0103, + 0x81C, 0x00700103, + 0x81C, 0x00720103, + 0x81C, 0x00740103, + 0x81C, 0x00760103, + 0x81C, 0x00780103, + 0x81C, 0x007A0103, + 0x81C, 0x007C0103, + 0x81C, 0x007E0103, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF8000103, + 0x81C, 0xF7020103, + 0x81C, 0xF6040103, + 0x81C, 0xF5060103, + 0x81C, 0xF4080103, + 0x81C, 0xF30A0103, + 0x81C, 0xF20C0103, + 0x81C, 0xF10E0103, + 0x81C, 0xF0100103, + 0x81C, 0xEF120103, + 0x81C, 0xEE140103, + 0x81C, 0xED160103, + 0x81C, 0xEC180103, + 0x81C, 0xEB1A0103, + 0x81C, 0xEA1C0103, + 0x81C, 0xE91E0103, + 0x81C, 0xE8200103, + 0x81C, 0xE7220103, + 0x81C, 0xE6240103, + 0x81C, 0xE5260103, + 0x81C, 0xE4280103, + 0x81C, 0xE32A0103, + 0x81C, 0xC32C0103, + 0x81C, 0xC22E0103, + 0x81C, 0xC1300103, + 0x81C, 0xC0320103, + 0x81C, 0xA3340103, + 0x81C, 0xA2360103, + 0x81C, 0xA1380103, + 0x81C, 0xA03A0103, + 0x81C, 0x823C0103, + 0x81C, 0x813E0103, + 0x81C, 0x80400103, + 0x81C, 0x63420103, + 0x81C, 0x62440103, + 0x81C, 0x61460103, + 0x81C, 0x60480103, + 0x81C, 0x424A0103, + 0x81C, 0x414C0103, + 0x81C, 0x404E0103, + 0x81C, 0x22500103, + 0x81C, 0x21520103, + 0x81C, 0x20540103, + 0x81C, 0x03560103, + 0x81C, 0x02580103, + 0x81C, 0x015A0103, + 0x81C, 0x005C0103, + 0x81C, 0x005E0103, + 0x81C, 0x00600103, + 0x81C, 0x00620103, + 0x81C, 0x00640103, + 0x81C, 0x00660103, + 0x81C, 0x00680103, + 0x81C, 0x006A0103, + 0x81C, 0x006C0103, + 0x81C, 0x006E0103, + 0x81C, 0x00700103, + 0x81C, 0x00720103, + 0x81C, 0x00740103, + 0x81C, 0x00760103, + 0x81C, 0x00780103, + 0x81C, 0x007A0103, + 0x81C, 0x007C0103, + 0x81C, 0x007E0103, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFD000103, + 0x81C, 0xFC020103, + 0x81C, 0xFB040103, + 0x81C, 0xFA060103, + 0x81C, 0xF9080103, + 0x81C, 0xF80A0103, + 0x81C, 0xF70C0103, + 0x81C, 0xF60E0103, + 0x81C, 0xF5100103, + 0x81C, 0xF4120103, + 0x81C, 0xF3140103, + 0x81C, 0xF2160103, + 0x81C, 0xF1180103, + 0x81C, 0xF01A0103, + 0x81C, 0xEF1C0103, + 0x81C, 0xEE1E0103, + 0x81C, 0xED200103, + 0x81C, 0xEC220103, + 0x81C, 0xEB240103, + 0x81C, 0xEA260103, + 0x81C, 0xE9280103, + 0x81C, 0xE82A0103, + 0x81C, 0xE72C0103, + 0x81C, 0xE62E0103, + 0x81C, 0xE5300103, + 0x81C, 0xE4320103, + 0x81C, 0xE3340103, + 0x81C, 0xE2360103, + 0x81C, 0xC5380103, + 0x81C, 0xC43A0103, + 0x81C, 0xC33C0103, + 0x81C, 0xC23E0103, + 0x81C, 0xA5400103, + 0x81C, 0xA4420103, + 0x81C, 0xA3440103, + 0x81C, 0xA2460103, + 0x81C, 0xA1480103, + 0x81C, 0x834A0103, + 0x81C, 0x824C0103, + 0x81C, 0x814E0103, + 0x81C, 0x64500103, + 0x81C, 0x63520103, + 0x81C, 0x62540103, + 0x81C, 0x61560103, + 0x81C, 0x42580103, + 0x81C, 0x415A0103, + 0x81C, 0x405C0103, + 0x81C, 0x065E0103, + 0x81C, 0x05600103, + 0x81C, 0x04620103, + 0x81C, 0x03640103, + 0x81C, 0x02660103, + 0x81C, 0x01680103, + 0x81C, 0x006A0103, + 0x81C, 0x006C0103, + 0x81C, 0x006E0103, + 0x81C, 0x00700103, + 0x81C, 0x00720103, + 0x81C, 0x00740103, + 0x81C, 0x00760103, + 0x81C, 0x00780103, + 0x81C, 0x007A0103, + 0x81C, 0x007C0103, + 0x81C, 0x007E0103, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFA000103, + 0x81C, 0xF9020103, + 0x81C, 0xF8040103, + 0x81C, 0xF7060103, + 0x81C, 0xF6080103, + 0x81C, 0xF50A0103, + 0x81C, 0xF40C0103, + 0x81C, 0xF30E0103, + 0x81C, 0xF2100103, + 0x81C, 0xF1120103, + 0x81C, 0xF0140103, + 0x81C, 0xEF160103, + 0x81C, 0xEE180103, + 0x81C, 0xED1A0103, + 0x81C, 0xEC1C0103, + 0x81C, 0xEB1E0103, + 0x81C, 0xEA200103, + 0x81C, 0xE9220103, + 0x81C, 0xE8240103, + 0x81C, 0xE7260103, + 0x81C, 0xE6280103, + 0x81C, 0xE52A0103, + 0x81C, 0xC42C0103, + 0x81C, 0xC32E0103, + 0x81C, 0xC2300103, + 0x81C, 0xC1320103, + 0x81C, 0xA4340103, + 0x81C, 0xA3360103, + 0x81C, 0xA2380103, + 0x81C, 0xA13A0103, + 0x81C, 0x833C0103, + 0x81C, 0x823E0103, + 0x81C, 0x81400103, + 0x81C, 0x63420103, + 0x81C, 0x62440103, + 0x81C, 0x61460103, + 0x81C, 0x60480103, + 0x81C, 0x424A0103, + 0x81C, 0x414C0103, + 0x81C, 0x404E0103, + 0x81C, 0x22500103, + 0x81C, 0x21520103, + 0x81C, 0x20540103, + 0x81C, 0x03560103, + 0x81C, 0x02580103, + 0x81C, 0x015A0103, + 0x81C, 0x005C0103, + 0x81C, 0x005E0103, + 0x81C, 0x00600103, + 0x81C, 0x00620103, + 0x81C, 0x00640103, + 0x81C, 0x00660103, + 0x81C, 0x00680103, + 0x81C, 0x006A0103, + 0x81C, 0x006C0103, + 0x81C, 0x006E0103, + 0x81C, 0x00700103, + 0x81C, 0x00720103, + 0x81C, 0x00740103, + 0x81C, 0x00760103, + 0x81C, 0x00780103, + 0x81C, 0x007A0103, + 0x81C, 0x007C0103, + 0x81C, 0x007E0103, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF8000103, + 0x81C, 0xF7020103, + 0x81C, 0xF6040103, + 0x81C, 0xF5060103, + 0x81C, 0xF4080103, + 0x81C, 0xF30A0103, + 0x81C, 0xF20C0103, + 0x81C, 0xF10E0103, + 0x81C, 0xF0100103, + 0x81C, 0xEF120103, + 0x81C, 0xEE140103, + 0x81C, 0xED160103, + 0x81C, 0xEC180103, + 0x81C, 0xEB1A0103, + 0x81C, 0xEA1C0103, + 0x81C, 0xE91E0103, + 0x81C, 0xE8200103, + 0x81C, 0xE7220103, + 0x81C, 0xE6240103, + 0x81C, 0xE5260103, + 0x81C, 0xE4280103, + 0x81C, 0xE32A0103, + 0x81C, 0xE22C0103, + 0x81C, 0xC32E0103, + 0x81C, 0xC2300103, + 0x81C, 0xC1320103, + 0x81C, 0xA3340103, + 0x81C, 0xA2360103, + 0x81C, 0xA1380103, + 0x81C, 0xA03A0103, + 0x81C, 0x823C0103, + 0x81C, 0x813E0103, + 0x81C, 0x80400103, + 0x81C, 0x64420103, + 0x81C, 0x63440103, + 0x81C, 0x62460103, + 0x81C, 0x61480103, + 0x81C, 0x434A0103, + 0x81C, 0x424C0103, + 0x81C, 0x414E0103, + 0x81C, 0x40500103, + 0x81C, 0x22520103, + 0x81C, 0x21540103, + 0x81C, 0x20560103, + 0x81C, 0x04580103, + 0x81C, 0x035A0103, + 0x81C, 0x025C0103, + 0x81C, 0x015E0103, + 0x81C, 0x00600103, + 0x81C, 0x00620103, + 0x81C, 0x00640103, + 0x81C, 0x00660103, + 0x81C, 0x00680103, + 0x81C, 0x006A0103, + 0x81C, 0x006C0103, + 0x81C, 0x006E0103, + 0x81C, 0x00700103, + 0x81C, 0x00720103, + 0x81C, 0x00740103, + 0x81C, 0x00760103, + 0x81C, 0x00780103, + 0x81C, 0x007A0103, + 0x81C, 0x007C0103, + 0x81C, 0x007E0103, + 0x90000008, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFD000103, + 0x81C, 0xFC020103, + 0x81C, 0xFB040103, + 0x81C, 0xFA060103, + 0x81C, 0xF9080103, + 0x81C, 0xF80A0103, + 0x81C, 0xF70C0103, + 0x81C, 0xF60E0103, + 0x81C, 0xF5100103, + 0x81C, 0xF4120103, + 0x81C, 0xF3140103, + 0x81C, 0xF2160103, + 0x81C, 0xF1180103, + 0x81C, 0xF01A0103, + 0x81C, 0xEF1C0103, + 0x81C, 0xEE1E0103, + 0x81C, 0xED200103, + 0x81C, 0xEC220103, + 0x81C, 0xEB240103, + 0x81C, 0xEA260103, + 0x81C, 0xE9280103, + 0x81C, 0xE82A0103, + 0x81C, 0xE72C0103, + 0x81C, 0xE62E0103, + 0x81C, 0xE5300103, + 0x81C, 0xE4320103, + 0x81C, 0xE3340103, + 0x81C, 0xC6360103, + 0x81C, 0xC5380103, + 0x81C, 0xC43A0103, + 0x81C, 0xC33C0103, + 0x81C, 0xC23E0103, + 0x81C, 0xA5400103, + 0x81C, 0xA4420103, + 0x81C, 0xA3440103, + 0x81C, 0xA2460103, + 0x81C, 0xA1480103, + 0x81C, 0x834A0103, + 0x81C, 0x824C0103, + 0x81C, 0x814E0103, + 0x81C, 0x63500103, + 0x81C, 0x62520103, + 0x81C, 0x61540103, + 0x81C, 0x43560103, + 0x81C, 0x42580103, + 0x81C, 0x245A0103, + 0x81C, 0x235C0103, + 0x81C, 0x225E0103, + 0x81C, 0x21600103, + 0x81C, 0x04620103, + 0x81C, 0x03640103, + 0x81C, 0x02660103, + 0x81C, 0x01680103, + 0x81C, 0x006A0103, + 0x81C, 0x006C0103, + 0x81C, 0x006E0103, + 0x81C, 0x00700103, + 0x81C, 0x00720103, + 0x81C, 0x00740103, + 0x81C, 0x00760103, + 0x81C, 0x00780103, + 0x81C, 0x007A0103, + 0x81C, 0x007C0103, + 0x81C, 0x007E0103, + 0x90000009, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF8000103, + 0x81C, 0xF7020103, + 0x81C, 0xF6040103, + 0x81C, 0xF5060103, + 0x81C, 0xF4080103, + 0x81C, 0xF30A0103, + 0x81C, 0xF20C0103, + 0x81C, 0xF10E0103, + 0x81C, 0xF0100103, + 0x81C, 0xEF120103, + 0x81C, 0xEE140103, + 0x81C, 0xED160103, + 0x81C, 0xEC180103, + 0x81C, 0xEB1A0103, + 0x81C, 0xEA1C0103, + 0x81C, 0xE91E0103, + 0x81C, 0xE8200103, + 0x81C, 0xE7220103, + 0x81C, 0xE6240103, + 0x81C, 0xE5260103, + 0x81C, 0xE4280103, + 0x81C, 0xE32A0103, + 0x81C, 0xE22C0103, + 0x81C, 0xC32E0103, + 0x81C, 0xC2300103, + 0x81C, 0xC1320103, + 0x81C, 0xA3340103, + 0x81C, 0xA2360103, + 0x81C, 0xA1380103, + 0x81C, 0xA03A0103, + 0x81C, 0x823C0103, + 0x81C, 0x813E0103, + 0x81C, 0x80400103, + 0x81C, 0x64420103, + 0x81C, 0x63440103, + 0x81C, 0x62460103, + 0x81C, 0x61480103, + 0x81C, 0x434A0103, + 0x81C, 0x424C0103, + 0x81C, 0x414E0103, + 0x81C, 0x40500103, + 0x81C, 0x22520103, + 0x81C, 0x21540103, + 0x81C, 0x20560103, + 0x81C, 0x04580103, + 0x81C, 0x035A0103, + 0x81C, 0x025C0103, + 0x81C, 0x015E0103, + 0x81C, 0x00600103, + 0x81C, 0x00620103, + 0x81C, 0x00640103, + 0x81C, 0x00660103, + 0x81C, 0x00680103, + 0x81C, 0x006A0103, + 0x81C, 0x006C0103, + 0x81C, 0x006E0103, + 0x81C, 0x00700103, + 0x81C, 0x00720103, + 0x81C, 0x00740103, + 0x81C, 0x00760103, + 0x81C, 0x00780103, + 0x81C, 0x007A0103, + 0x81C, 0x007C0103, + 0x81C, 0x007E0103, + 0x9000000a, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFD000103, + 0x81C, 0xFC020103, + 0x81C, 0xFB040103, + 0x81C, 0xFA060103, + 0x81C, 0xF9080103, + 0x81C, 0xF80A0103, + 0x81C, 0xF70C0103, + 0x81C, 0xF60E0103, + 0x81C, 0xF5100103, + 0x81C, 0xF4120103, + 0x81C, 0xF3140103, + 0x81C, 0xF2160103, + 0x81C, 0xF1180103, + 0x81C, 0xF01A0103, + 0x81C, 0xEE1C0103, + 0x81C, 0xED1E0103, + 0x81C, 0xEC200103, + 0x81C, 0xEB220103, + 0x81C, 0xEA240103, + 0x81C, 0xE9260103, + 0x81C, 0xE8280103, + 0x81C, 0xE72A0103, + 0x81C, 0xE62C0103, + 0x81C, 0xE52E0103, + 0x81C, 0xE4300103, + 0x81C, 0xE3320103, + 0x81C, 0xE2340103, + 0x81C, 0xC5360103, + 0x81C, 0xC4380103, + 0x81C, 0xC33A0103, + 0x81C, 0xC23C0103, + 0x81C, 0xA53E0103, + 0x81C, 0xA4400103, + 0x81C, 0xA3420103, + 0x81C, 0xA2440103, + 0x81C, 0xA1460103, + 0x81C, 0x83480103, + 0x81C, 0x824A0103, + 0x81C, 0x814C0103, + 0x81C, 0x804E0103, + 0x81C, 0x63500103, + 0x81C, 0x62520103, + 0x81C, 0x61540103, + 0x81C, 0x43560103, + 0x81C, 0x42580103, + 0x81C, 0x415A0103, + 0x81C, 0x405C0103, + 0x81C, 0x225E0103, + 0x81C, 0x21600103, + 0x81C, 0x20620103, + 0x81C, 0x03640103, + 0x81C, 0x02660103, + 0x81C, 0x01680103, + 0x81C, 0x006A0103, + 0x81C, 0x006C0103, + 0x81C, 0x006E0103, + 0x81C, 0x00700103, + 0x81C, 0x00720103, + 0x81C, 0x00740103, + 0x81C, 0x00760103, + 0x81C, 0x00780103, + 0x81C, 0x007A0103, + 0x81C, 0x007C0103, + 0x81C, 0x007E0103, + 0x9000000b, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF9000103, + 0x81C, 0xF8020103, + 0x81C, 0xF7040103, + 0x81C, 0xF6060103, + 0x81C, 0xF5080103, + 0x81C, 0xF40A0103, + 0x81C, 0xF30C0103, + 0x81C, 0xF20E0103, + 0x81C, 0xF1100103, + 0x81C, 0xF0120103, + 0x81C, 0xEF140103, + 0x81C, 0xEE160103, + 0x81C, 0xED180103, + 0x81C, 0xEC1A0103, + 0x81C, 0xEB1C0103, + 0x81C, 0xEA1E0103, + 0x81C, 0xE9200103, + 0x81C, 0xE8220103, + 0x81C, 0xE7240103, + 0x81C, 0xE6260103, + 0x81C, 0xE5280103, + 0x81C, 0xE42A0103, + 0x81C, 0xE32C0103, + 0x81C, 0xC32E0103, + 0x81C, 0xC2300103, + 0x81C, 0xC1320103, + 0x81C, 0xA4340103, + 0x81C, 0xA3360103, + 0x81C, 0xA2380103, + 0x81C, 0xA13A0103, + 0x81C, 0xA03C0103, + 0x81C, 0x823E0103, + 0x81C, 0x81400103, + 0x81C, 0x80420103, + 0x81C, 0x63440103, + 0x81C, 0x62460103, + 0x81C, 0x61480103, + 0x81C, 0x604A0103, + 0x81C, 0x244C0103, + 0x81C, 0x234E0103, + 0x81C, 0x22500103, + 0x81C, 0x21520103, + 0x81C, 0x20540103, + 0x81C, 0x05560103, + 0x81C, 0x04580103, + 0x81C, 0x035A0103, + 0x81C, 0x025C0103, + 0x81C, 0x015E0103, + 0x81C, 0x00600103, + 0x81C, 0x00620103, + 0x81C, 0x00640103, + 0x81C, 0x00660103, + 0x81C, 0x00680103, + 0x81C, 0x006A0103, + 0x81C, 0x006C0103, + 0x81C, 0x006E0103, + 0x81C, 0x00700103, + 0x81C, 0x00720103, + 0x81C, 0x00740103, + 0x81C, 0x00760103, + 0x81C, 0x00780103, + 0x81C, 0x007A0103, + 0x81C, 0x007C0103, + 0x81C, 0x007E0103, + 0x9000000c, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFE000103, + 0x81C, 0xFD020103, + 0x81C, 0xFC040103, + 0x81C, 0xFB060103, + 0x81C, 0xFA080103, + 0x81C, 0xF90A0103, + 0x81C, 0xF80C0103, + 0x81C, 0xF70E0103, + 0x81C, 0xF6100103, + 0x81C, 0xF5120103, + 0x81C, 0xF4140103, + 0x81C, 0xF3160103, + 0x81C, 0xF2180103, + 0x81C, 0xF11A0103, + 0x81C, 0xF01C0103, + 0x81C, 0xEF1E0103, + 0x81C, 0xEE200103, + 0x81C, 0xED220103, + 0x81C, 0xEC240103, + 0x81C, 0xEB260103, + 0x81C, 0xEA280103, + 0x81C, 0xE92A0103, + 0x81C, 0xE82C0103, + 0x81C, 0xE72E0103, + 0x81C, 0xE6300103, + 0x81C, 0xE5320103, + 0x81C, 0xE4340103, + 0x81C, 0xE3360103, + 0x81C, 0xC6380103, + 0x81C, 0xC53A0103, + 0x81C, 0xC43C0103, + 0x81C, 0xC33E0103, + 0x81C, 0xA5400103, + 0x81C, 0xA4420103, + 0x81C, 0xA3440103, + 0x81C, 0xA2460103, + 0x81C, 0xA1480103, + 0x81C, 0xA04A0103, + 0x81C, 0x824C0103, + 0x81C, 0x814E0103, + 0x81C, 0x80500103, + 0x81C, 0x64520103, + 0x81C, 0x63540103, + 0x81C, 0x62560103, + 0x81C, 0x61580103, + 0x81C, 0x605A0103, + 0x81C, 0x235C0103, + 0x81C, 0x225E0103, + 0x81C, 0x21600103, + 0x81C, 0x20620103, + 0x81C, 0x03640103, + 0x81C, 0x02660103, + 0x81C, 0x01680103, + 0x81C, 0x006A0103, + 0x81C, 0x006C0103, + 0x81C, 0x006E0103, + 0x81C, 0x00700103, + 0x81C, 0x00720103, + 0x81C, 0x00740103, + 0x81C, 0x00760103, + 0x81C, 0x00780103, + 0x81C, 0x007A0103, + 0x81C, 0x007C0103, + 0x81C, 0x007E0103, + 0x9000000d, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000103, + 0x81C, 0xFB020103, + 0x81C, 0xFA040103, + 0x81C, 0xF9060103, + 0x81C, 0xF8080103, + 0x81C, 0xF70A0103, + 0x81C, 0xF60C0103, + 0x81C, 0xF50E0103, + 0x81C, 0xF4100103, + 0x81C, 0xF3120103, + 0x81C, 0xF2140103, + 0x81C, 0xF1160103, + 0x81C, 0xF0180103, + 0x81C, 0xEE1A0103, + 0x81C, 0xED1C0103, + 0x81C, 0xEC1E0103, + 0x81C, 0xEB200103, + 0x81C, 0xEA220103, + 0x81C, 0xE9240103, + 0x81C, 0xE8260103, + 0x81C, 0xE7280103, + 0x81C, 0xE62A0103, + 0x81C, 0xE52C0103, + 0x81C, 0xE42E0103, + 0x81C, 0xE3300103, + 0x81C, 0xE2320103, + 0x81C, 0xE1340103, + 0x81C, 0xC5360103, + 0x81C, 0xC4380103, + 0x81C, 0xC33A0103, + 0x81C, 0xC23C0103, + 0x81C, 0xA53E0103, + 0x81C, 0xA4400103, + 0x81C, 0xA3420103, + 0x81C, 0xA2440103, + 0x81C, 0xA1460103, + 0x81C, 0x83480103, + 0x81C, 0x824A0103, + 0x81C, 0x814C0103, + 0x81C, 0x804E0103, + 0x81C, 0x63500103, + 0x81C, 0x62520103, + 0x81C, 0x61540103, + 0x81C, 0x43560103, + 0x81C, 0x42580103, + 0x81C, 0x415A0103, + 0x81C, 0x405C0103, + 0x81C, 0x225E0103, + 0x81C, 0x21600103, + 0x81C, 0x20620103, + 0x81C, 0x03640103, + 0x81C, 0x02660103, + 0x81C, 0x01680103, + 0x81C, 0x006A0103, + 0x81C, 0x006C0103, + 0x81C, 0x006E0103, + 0x81C, 0x00700103, + 0x81C, 0x00720103, + 0x81C, 0x00740103, + 0x81C, 0x00760103, + 0x81C, 0x00780103, + 0x81C, 0x007A0103, + 0x81C, 0x007C0103, + 0x81C, 0x007E0103, + 0x9000000e, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFD000103, + 0x81C, 0xFC020103, + 0x81C, 0xFB040103, + 0x81C, 0xFA060103, + 0x81C, 0xF9080103, + 0x81C, 0xF80A0103, + 0x81C, 0xF70C0103, + 0x81C, 0xF60E0103, + 0x81C, 0xF5100103, + 0x81C, 0xF4120103, + 0x81C, 0xF3140103, + 0x81C, 0xF2160103, + 0x81C, 0xF1180103, + 0x81C, 0xF01A0103, + 0x81C, 0xEE1C0103, + 0x81C, 0xED1E0103, + 0x81C, 0xEC200103, + 0x81C, 0xEB220103, + 0x81C, 0xEA240103, + 0x81C, 0xE9260103, + 0x81C, 0xE8280103, + 0x81C, 0xE72A0103, + 0x81C, 0xE62C0103, + 0x81C, 0xE52E0103, + 0x81C, 0xE4300103, + 0x81C, 0xE3320103, + 0x81C, 0xE2340103, + 0x81C, 0xC5360103, + 0x81C, 0xC4380103, + 0x81C, 0xC33A0103, + 0x81C, 0xC23C0103, + 0x81C, 0xA53E0103, + 0x81C, 0xA4400103, + 0x81C, 0xA3420103, + 0x81C, 0xA2440103, + 0x81C, 0xA1460103, + 0x81C, 0x83480103, + 0x81C, 0x824A0103, + 0x81C, 0x814C0103, + 0x81C, 0x804E0103, + 0x81C, 0x63500103, + 0x81C, 0x62520103, + 0x81C, 0x61540103, + 0x81C, 0x43560103, + 0x81C, 0x42580103, + 0x81C, 0x415A0103, + 0x81C, 0x405C0103, + 0x81C, 0x225E0103, + 0x81C, 0x21600103, + 0x81C, 0x20620103, + 0x81C, 0x03640103, + 0x81C, 0x02660103, + 0x81C, 0x01680103, + 0x81C, 0x006A0103, + 0x81C, 0x006C0103, + 0x81C, 0x006E0103, + 0x81C, 0x00700103, + 0x81C, 0x00720103, + 0x81C, 0x00740103, + 0x81C, 0x00760103, + 0x81C, 0x00780103, + 0x81C, 0x007A0103, + 0x81C, 0x007C0103, + 0x81C, 0x007E0103, + 0x9000000f, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000103, + 0x81C, 0xFB020103, + 0x81C, 0xFA040103, + 0x81C, 0xF9060103, + 0x81C, 0xF8080103, + 0x81C, 0xF70A0103, + 0x81C, 0xF60C0103, + 0x81C, 0xF50E0103, + 0x81C, 0xF4100103, + 0x81C, 0xF3120103, + 0x81C, 0xF2140103, + 0x81C, 0xF1160103, + 0x81C, 0xF0180103, + 0x81C, 0xEF1A0103, + 0x81C, 0xEE1C0103, + 0x81C, 0xED1E0103, + 0x81C, 0xEC200103, + 0x81C, 0xEB220103, + 0x81C, 0xEA240103, + 0x81C, 0xE9260103, + 0x81C, 0xE8280103, + 0x81C, 0xE72A0103, + 0x81C, 0xE62C0103, + 0x81C, 0xE52E0103, + 0x81C, 0xE4300103, + 0x81C, 0xE3320103, + 0x81C, 0xE2340103, + 0x81C, 0xE1360103, + 0x81C, 0xC3380103, + 0x81C, 0xC23A0103, + 0x81C, 0xC13C0103, + 0x81C, 0xC03E0103, + 0x81C, 0xA4400103, + 0x81C, 0xA3420103, + 0x81C, 0xA2440103, + 0x81C, 0xA1460103, + 0x81C, 0x82480103, + 0x81C, 0x814A0103, + 0x81C, 0x804C0103, + 0x81C, 0x634E0103, + 0x81C, 0x62500103, + 0x81C, 0x61520103, + 0x81C, 0x42540103, + 0x81C, 0x41560103, + 0x81C, 0x24580103, + 0x81C, 0x235A0103, + 0x81C, 0x225C0103, + 0x81C, 0x215E0103, + 0x81C, 0x20600103, + 0x81C, 0x03620103, + 0x81C, 0x02640103, + 0x81C, 0x01660103, + 0x81C, 0x00680103, + 0x81C, 0x006A0103, + 0x81C, 0x006C0103, + 0x81C, 0x006E0103, + 0x81C, 0x00700103, + 0x81C, 0x00720103, + 0x81C, 0x00740103, + 0x81C, 0x00760103, + 0x81C, 0x00780103, + 0x81C, 0x007A0103, + 0x81C, 0x007C0103, + 0x81C, 0x007E0103, + 0x90000010, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFE000103, + 0x81C, 0xFD020103, + 0x81C, 0xFC040103, + 0x81C, 0xFB060103, + 0x81C, 0xFA080103, + 0x81C, 0xF90A0103, + 0x81C, 0xF80C0103, + 0x81C, 0xF70E0103, + 0x81C, 0xF6100103, + 0x81C, 0xF5120103, + 0x81C, 0xF4140103, + 0x81C, 0xF3160103, + 0x81C, 0xF2180103, + 0x81C, 0xF11A0103, + 0x81C, 0xF01C0103, + 0x81C, 0xEF1E0103, + 0x81C, 0xEE200103, + 0x81C, 0xED220103, + 0x81C, 0xEC240103, + 0x81C, 0xEB260103, + 0x81C, 0xEA280103, + 0x81C, 0xE92A0103, + 0x81C, 0xE82C0103, + 0x81C, 0xE72E0103, + 0x81C, 0xE6300103, + 0x81C, 0xE5320103, + 0x81C, 0xE4340103, + 0x81C, 0xE3360103, + 0x81C, 0xC6380103, + 0x81C, 0xC53A0103, + 0x81C, 0xC43C0103, + 0x81C, 0xC33E0103, + 0x81C, 0xA5400103, + 0x81C, 0xA4420103, + 0x81C, 0xA3440103, + 0x81C, 0xA2460103, + 0x81C, 0xA1480103, + 0x81C, 0xA04A0103, + 0x81C, 0x824C0103, + 0x81C, 0x814E0103, + 0x81C, 0x80500103, + 0x81C, 0x64520103, + 0x81C, 0x63540103, + 0x81C, 0x62560103, + 0x81C, 0x61580103, + 0x81C, 0x605A0103, + 0x81C, 0x235C0103, + 0x81C, 0x225E0103, + 0x81C, 0x21600103, + 0x81C, 0x20620103, + 0x81C, 0x03640103, + 0x81C, 0x02660103, + 0x81C, 0x01680103, + 0x81C, 0x006A0103, + 0x81C, 0x006C0103, + 0x81C, 0x006E0103, + 0x81C, 0x00700103, + 0x81C, 0x00720103, + 0x81C, 0x00740103, + 0x81C, 0x00760103, + 0x81C, 0x00780103, + 0x81C, 0x007A0103, + 0x81C, 0x007C0103, + 0x81C, 0x007E0103, + 0x90000012, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF8000103, + 0x81C, 0xF7020103, + 0x81C, 0xF6040103, + 0x81C, 0xF5060103, + 0x81C, 0xF4080103, + 0x81C, 0xF30A0103, + 0x81C, 0xF20C0103, + 0x81C, 0xF10E0103, + 0x81C, 0xF0100103, + 0x81C, 0xEF120103, + 0x81C, 0xEE140103, + 0x81C, 0xED160103, + 0x81C, 0xEC180103, + 0x81C, 0xEB1A0103, + 0x81C, 0xEA1C0103, + 0x81C, 0xE91E0103, + 0x81C, 0xE8200103, + 0x81C, 0xE7220103, + 0x81C, 0xE6240103, + 0x81C, 0xE5260103, + 0x81C, 0xE4280103, + 0x81C, 0xE32A0103, + 0x81C, 0xC32C0103, + 0x81C, 0xC22E0103, + 0x81C, 0xC1300103, + 0x81C, 0xC0320103, + 0x81C, 0xA3340103, + 0x81C, 0xA2360103, + 0x81C, 0xA1380103, + 0x81C, 0xA03A0103, + 0x81C, 0x823C0103, + 0x81C, 0x813E0103, + 0x81C, 0x80400103, + 0x81C, 0x63420103, + 0x81C, 0x62440103, + 0x81C, 0x61460103, + 0x81C, 0x60480103, + 0x81C, 0x424A0103, + 0x81C, 0x414C0103, + 0x81C, 0x404E0103, + 0x81C, 0x22500103, + 0x81C, 0x21520103, + 0x81C, 0x20540103, + 0x81C, 0x03560103, + 0x81C, 0x02580103, + 0x81C, 0x015A0103, + 0x81C, 0x005C0103, + 0x81C, 0x005E0103, + 0x81C, 0x00600103, + 0x81C, 0x00620103, + 0x81C, 0x00640103, + 0x81C, 0x00660103, + 0x81C, 0x00680103, + 0x81C, 0x006A0103, + 0x81C, 0x006C0103, + 0x81C, 0x006E0103, + 0x81C, 0x00700103, + 0x81C, 0x00720103, + 0x81C, 0x00740103, + 0x81C, 0x00760103, + 0x81C, 0x00780103, + 0x81C, 0x007A0103, + 0x81C, 0x007C0103, + 0x81C, 0x007E0103, + 0xA0000000, 0x00000000, + 0x81C, 0xFE000103, + 0x81C, 0xFD020103, + 0x81C, 0xFC040103, + 0x81C, 0xFB060103, + 0x81C, 0xFA080103, + 0x81C, 0xF90A0103, + 0x81C, 0xF80C0103, + 0x81C, 0xF70E0103, + 0x81C, 0xF6100103, + 0x81C, 0xF5120103, + 0x81C, 0xF4140103, + 0x81C, 0xF3160103, + 0x81C, 0xF2180103, + 0x81C, 0xF11A0103, + 0x81C, 0xF01C0103, + 0x81C, 0xEF1E0103, + 0x81C, 0xEE200103, + 0x81C, 0xED220103, + 0x81C, 0xEC240103, + 0x81C, 0xEB260103, + 0x81C, 0xEA280103, + 0x81C, 0xE92A0103, + 0x81C, 0xE82C0103, + 0x81C, 0xE72E0103, + 0x81C, 0xE6300103, + 0x81C, 0xE5320103, + 0x81C, 0xE4340103, + 0x81C, 0xE3360103, + 0x81C, 0xC6380103, + 0x81C, 0xC53A0103, + 0x81C, 0xC43C0103, + 0x81C, 0xC33E0103, + 0x81C, 0xA5400103, + 0x81C, 0xA4420103, + 0x81C, 0xA3440103, + 0x81C, 0xA2460103, + 0x81C, 0xA1480103, + 0x81C, 0xA04A0103, + 0x81C, 0x824C0103, + 0x81C, 0x814E0103, + 0x81C, 0x80500103, + 0x81C, 0x64520103, + 0x81C, 0x63540103, + 0x81C, 0x62560103, + 0x81C, 0x61580103, + 0x81C, 0x605A0103, + 0x81C, 0x235C0103, + 0x81C, 0x225E0103, + 0x81C, 0x21600103, + 0x81C, 0x20620103, + 0x81C, 0x03640103, + 0x81C, 0x02660103, + 0x81C, 0x01680103, + 0x81C, 0x006A0103, + 0x81C, 0x006C0103, + 0x81C, 0x006E0103, + 0x81C, 0x00700103, + 0x81C, 0x00720103, + 0x81C, 0x00740103, + 0x81C, 0x00760103, + 0x81C, 0x00780103, + 0x81C, 0x007A0103, + 0x81C, 0x007C0103, + 0x81C, 0x007E0103, + 0xB0000000, 0x00000000, + 0x80000000, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000203, + 0x81C, 0xFB020203, + 0x81C, 0xFA040203, + 0x81C, 0xF9060203, + 0x81C, 0xF8080203, + 0x81C, 0xF70A0203, + 0x81C, 0xF60C0203, + 0x81C, 0xF50E0203, + 0x81C, 0xF4100203, + 0x81C, 0xF3120203, + 0x81C, 0xF2140203, + 0x81C, 0xF1160203, + 0x81C, 0xF0180203, + 0x81C, 0xEE1A0203, + 0x81C, 0xED1C0203, + 0x81C, 0xEC1E0203, + 0x81C, 0xEB200203, + 0x81C, 0xEA220203, + 0x81C, 0xE9240203, + 0x81C, 0xE8260203, + 0x81C, 0xE7280203, + 0x81C, 0xE62A0203, + 0x81C, 0xE52C0203, + 0x81C, 0xE42E0203, + 0x81C, 0xE3300203, + 0x81C, 0xE2320203, + 0x81C, 0xC6340203, + 0x81C, 0xC5360203, + 0x81C, 0xC4380203, + 0x81C, 0xC33A0203, + 0x81C, 0xA63C0203, + 0x81C, 0xA53E0203, + 0x81C, 0xA4400203, + 0x81C, 0xA3420203, + 0x81C, 0xA2440203, + 0x81C, 0xA1460203, + 0x81C, 0x83480203, + 0x81C, 0x824A0203, + 0x81C, 0x814C0203, + 0x81C, 0x804E0203, + 0x81C, 0x63500203, + 0x81C, 0x62520203, + 0x81C, 0x61540203, + 0x81C, 0x42560203, + 0x81C, 0x41580203, + 0x81C, 0x405A0203, + 0x81C, 0x225C0203, + 0x81C, 0x215E0203, + 0x81C, 0x20600203, + 0x81C, 0x04620203, + 0x81C, 0x03640203, + 0x81C, 0x02660203, + 0x81C, 0x01680203, + 0x81C, 0x006A0203, + 0x81C, 0x006C0203, + 0x81C, 0x006E0203, + 0x81C, 0x00700203, + 0x81C, 0x00720203, + 0x81C, 0x00740203, + 0x81C, 0x00760203, + 0x81C, 0x00780203, + 0x81C, 0x007A0203, + 0x81C, 0x007C0203, + 0x81C, 0x007E0203, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF7000203, + 0x81C, 0xF6020203, + 0x81C, 0xF5040203, + 0x81C, 0xF4060203, + 0x81C, 0xF3080203, + 0x81C, 0xF20A0203, + 0x81C, 0xF10C0203, + 0x81C, 0xF00E0203, + 0x81C, 0xEF100203, + 0x81C, 0xEE120203, + 0x81C, 0xED140203, + 0x81C, 0xEC160203, + 0x81C, 0xEB180203, + 0x81C, 0xEA1A0203, + 0x81C, 0xE91C0203, + 0x81C, 0xE81E0203, + 0x81C, 0xE7200203, + 0x81C, 0xE6220203, + 0x81C, 0xE5240203, + 0x81C, 0xE4260203, + 0x81C, 0xE3280203, + 0x81C, 0xC42A0203, + 0x81C, 0xC32C0203, + 0x81C, 0xC22E0203, + 0x81C, 0xC1300203, + 0x81C, 0xC0320203, + 0x81C, 0xA3340203, + 0x81C, 0xA2360203, + 0x81C, 0xA1380203, + 0x81C, 0xA03A0203, + 0x81C, 0x823C0203, + 0x81C, 0x813E0203, + 0x81C, 0x80400203, + 0x81C, 0x63420203, + 0x81C, 0x62440203, + 0x81C, 0x61460203, + 0x81C, 0x60480203, + 0x81C, 0x424A0203, + 0x81C, 0x414C0203, + 0x81C, 0x404E0203, + 0x81C, 0x06500203, + 0x81C, 0x05520203, + 0x81C, 0x04540203, + 0x81C, 0x03560203, + 0x81C, 0x02580203, + 0x81C, 0x015A0203, + 0x81C, 0x005C0203, + 0x81C, 0x005E0203, + 0x81C, 0x00600203, + 0x81C, 0x00620203, + 0x81C, 0x00640203, + 0x81C, 0x00660203, + 0x81C, 0x00680203, + 0x81C, 0x006A0203, + 0x81C, 0x006C0203, + 0x81C, 0x006E0203, + 0x81C, 0x00700203, + 0x81C, 0x00720203, + 0x81C, 0x00740203, + 0x81C, 0x00760203, + 0x81C, 0x00780203, + 0x81C, 0x007A0203, + 0x81C, 0x007C0203, + 0x81C, 0x007E0203, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF7000203, + 0x81C, 0xF6020203, + 0x81C, 0xF5040203, + 0x81C, 0xF4060203, + 0x81C, 0xF3080203, + 0x81C, 0xF20A0203, + 0x81C, 0xF10C0203, + 0x81C, 0xF00E0203, + 0x81C, 0xEF100203, + 0x81C, 0xEE120203, + 0x81C, 0xED140203, + 0x81C, 0xEC160203, + 0x81C, 0xEB180203, + 0x81C, 0xEA1A0203, + 0x81C, 0xE91C0203, + 0x81C, 0xE81E0203, + 0x81C, 0xE7200203, + 0x81C, 0xE6220203, + 0x81C, 0xE5240203, + 0x81C, 0xE4260203, + 0x81C, 0xE3280203, + 0x81C, 0xC42A0203, + 0x81C, 0xC32C0203, + 0x81C, 0xC22E0203, + 0x81C, 0xC1300203, + 0x81C, 0xC0320203, + 0x81C, 0xA3340203, + 0x81C, 0xA2360203, + 0x81C, 0xA1380203, + 0x81C, 0xA03A0203, + 0x81C, 0x823C0203, + 0x81C, 0x813E0203, + 0x81C, 0x80400203, + 0x81C, 0x64420203, + 0x81C, 0x63440203, + 0x81C, 0x62460203, + 0x81C, 0x61480203, + 0x81C, 0x604A0203, + 0x81C, 0x414C0203, + 0x81C, 0x404E0203, + 0x81C, 0x22500203, + 0x81C, 0x21520203, + 0x81C, 0x20540203, + 0x81C, 0x03560203, + 0x81C, 0x02580203, + 0x81C, 0x015A0203, + 0x81C, 0x005C0203, + 0x81C, 0x005E0203, + 0x81C, 0x00600203, + 0x81C, 0x00620203, + 0x81C, 0x00640203, + 0x81C, 0x00660203, + 0x81C, 0x00680203, + 0x81C, 0x006A0203, + 0x81C, 0x006C0203, + 0x81C, 0x006E0203, + 0x81C, 0x00700203, + 0x81C, 0x00720203, + 0x81C, 0x00740203, + 0x81C, 0x00760203, + 0x81C, 0x00780203, + 0x81C, 0x007A0203, + 0x81C, 0x007C0203, + 0x81C, 0x007E0203, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000203, + 0x81C, 0xFB020203, + 0x81C, 0xFA040203, + 0x81C, 0xF9060203, + 0x81C, 0xF8080203, + 0x81C, 0xF70A0203, + 0x81C, 0xF60C0203, + 0x81C, 0xF50E0203, + 0x81C, 0xF4100203, + 0x81C, 0xF3120203, + 0x81C, 0xF2140203, + 0x81C, 0xF1160203, + 0x81C, 0xF0180203, + 0x81C, 0xEF1A0203, + 0x81C, 0xEE1C0203, + 0x81C, 0xED1E0203, + 0x81C, 0xEC200203, + 0x81C, 0xEB220203, + 0x81C, 0xEA240203, + 0x81C, 0xE9260203, + 0x81C, 0xE8280203, + 0x81C, 0xE72A0203, + 0x81C, 0xE62C0203, + 0x81C, 0xE52E0203, + 0x81C, 0xE4300203, + 0x81C, 0xE3320203, + 0x81C, 0xE2340203, + 0x81C, 0xC6360203, + 0x81C, 0xC5380203, + 0x81C, 0xC43A0203, + 0x81C, 0xC33C0203, + 0x81C, 0xA63E0203, + 0x81C, 0xA5400203, + 0x81C, 0xA4420203, + 0x81C, 0xA3440203, + 0x81C, 0xA2460203, + 0x81C, 0xA1480203, + 0x81C, 0x834A0203, + 0x81C, 0x824C0203, + 0x81C, 0x814E0203, + 0x81C, 0x64500203, + 0x81C, 0x63520203, + 0x81C, 0x62540203, + 0x81C, 0x61560203, + 0x81C, 0x60580203, + 0x81C, 0x405A0203, + 0x81C, 0x215C0203, + 0x81C, 0x205E0203, + 0x81C, 0x03600203, + 0x81C, 0x02620203, + 0x81C, 0x01640203, + 0x81C, 0x00660203, + 0x81C, 0x00680203, + 0x81C, 0x006A0203, + 0x81C, 0x006C0203, + 0x81C, 0x006E0203, + 0x81C, 0x00700203, + 0x81C, 0x00720203, + 0x81C, 0x00740203, + 0x81C, 0x00760203, + 0x81C, 0x00780203, + 0x81C, 0x007A0203, + 0x81C, 0x007C0203, + 0x81C, 0x007E0203, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF7000203, + 0x81C, 0xF6020203, + 0x81C, 0xF5040203, + 0x81C, 0xF4060203, + 0x81C, 0xF3080203, + 0x81C, 0xF20A0203, + 0x81C, 0xF10C0203, + 0x81C, 0xF00E0203, + 0x81C, 0xEF100203, + 0x81C, 0xEE120203, + 0x81C, 0xED140203, + 0x81C, 0xEC160203, + 0x81C, 0xEB180203, + 0x81C, 0xEA1A0203, + 0x81C, 0xE91C0203, + 0x81C, 0xE81E0203, + 0x81C, 0xE7200203, + 0x81C, 0xE6220203, + 0x81C, 0xE5240203, + 0x81C, 0xE4260203, + 0x81C, 0xE3280203, + 0x81C, 0xC42A0203, + 0x81C, 0xC32C0203, + 0x81C, 0xC22E0203, + 0x81C, 0xC1300203, + 0x81C, 0xC0320203, + 0x81C, 0xA3340203, + 0x81C, 0xA2360203, + 0x81C, 0xA1380203, + 0x81C, 0xA03A0203, + 0x81C, 0x823C0203, + 0x81C, 0x813E0203, + 0x81C, 0x80400203, + 0x81C, 0x64420203, + 0x81C, 0x63440203, + 0x81C, 0x62460203, + 0x81C, 0x61480203, + 0x81C, 0x604A0203, + 0x81C, 0x414C0203, + 0x81C, 0x404E0203, + 0x81C, 0x22500203, + 0x81C, 0x21520203, + 0x81C, 0x20540203, + 0x81C, 0x03560203, + 0x81C, 0x02580203, + 0x81C, 0x015A0203, + 0x81C, 0x005C0203, + 0x81C, 0x005E0203, + 0x81C, 0x00600203, + 0x81C, 0x00620203, + 0x81C, 0x00640203, + 0x81C, 0x00660203, + 0x81C, 0x00680203, + 0x81C, 0x006A0203, + 0x81C, 0x006C0203, + 0x81C, 0x006E0203, + 0x81C, 0x00700203, + 0x81C, 0x00720203, + 0x81C, 0x00740203, + 0x81C, 0x00760203, + 0x81C, 0x00780203, + 0x81C, 0x007A0203, + 0x81C, 0x007C0203, + 0x81C, 0x007E0203, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000203, + 0x81C, 0xFB020203, + 0x81C, 0xFA040203, + 0x81C, 0xF9060203, + 0x81C, 0xF8080203, + 0x81C, 0xF70A0203, + 0x81C, 0xF60C0203, + 0x81C, 0xF50E0203, + 0x81C, 0xF4100203, + 0x81C, 0xF3120203, + 0x81C, 0xF2140203, + 0x81C, 0xF1160203, + 0x81C, 0xF0180203, + 0x81C, 0xEF1A0203, + 0x81C, 0xEE1C0203, + 0x81C, 0xED1E0203, + 0x81C, 0xEC200203, + 0x81C, 0xEB220203, + 0x81C, 0xEA240203, + 0x81C, 0xE9260203, + 0x81C, 0xE8280203, + 0x81C, 0xE72A0203, + 0x81C, 0xE62C0203, + 0x81C, 0xE52E0203, + 0x81C, 0xE4300203, + 0x81C, 0xE3320203, + 0x81C, 0xE2340203, + 0x81C, 0xE1360203, + 0x81C, 0xC5380203, + 0x81C, 0xC43A0203, + 0x81C, 0xC33C0203, + 0x81C, 0xC23E0203, + 0x81C, 0xC1400203, + 0x81C, 0xA3420203, + 0x81C, 0xA2440203, + 0x81C, 0xA1460203, + 0x81C, 0xA0480203, + 0x81C, 0x834A0203, + 0x81C, 0x824C0203, + 0x81C, 0x814E0203, + 0x81C, 0x64500203, + 0x81C, 0x63520203, + 0x81C, 0x62540203, + 0x81C, 0x61560203, + 0x81C, 0x25580203, + 0x81C, 0x245A0203, + 0x81C, 0x235C0203, + 0x81C, 0x225E0203, + 0x81C, 0x21600203, + 0x81C, 0x04620203, + 0x81C, 0x03640203, + 0x81C, 0x02660203, + 0x81C, 0x01680203, + 0x81C, 0x006A0203, + 0x81C, 0x006C0203, + 0x81C, 0x006E0203, + 0x81C, 0x00700203, + 0x81C, 0x00720203, + 0x81C, 0x00740203, + 0x81C, 0x00760203, + 0x81C, 0x00780203, + 0x81C, 0x007A0203, + 0x81C, 0x007C0203, + 0x81C, 0x007E0203, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF9000203, + 0x81C, 0xF8020203, + 0x81C, 0xF7040203, + 0x81C, 0xF6060203, + 0x81C, 0xF5080203, + 0x81C, 0xF40A0203, + 0x81C, 0xF30C0203, + 0x81C, 0xF20E0203, + 0x81C, 0xF1100203, + 0x81C, 0xF0120203, + 0x81C, 0xEF140203, + 0x81C, 0xEE160203, + 0x81C, 0xED180203, + 0x81C, 0xEC1A0203, + 0x81C, 0xEB1C0203, + 0x81C, 0xEA1E0203, + 0x81C, 0xE9200203, + 0x81C, 0xE8220203, + 0x81C, 0xE7240203, + 0x81C, 0xE6260203, + 0x81C, 0xE5280203, + 0x81C, 0xC42A0203, + 0x81C, 0xC32C0203, + 0x81C, 0xC22E0203, + 0x81C, 0xC1300203, + 0x81C, 0xC0320203, + 0x81C, 0xA3340203, + 0x81C, 0xA2360203, + 0x81C, 0xA1380203, + 0x81C, 0xA03A0203, + 0x81C, 0x823C0203, + 0x81C, 0x813E0203, + 0x81C, 0x80400203, + 0x81C, 0x64420203, + 0x81C, 0x63440203, + 0x81C, 0x62460203, + 0x81C, 0x61480203, + 0x81C, 0x604A0203, + 0x81C, 0x414C0203, + 0x81C, 0x404E0203, + 0x81C, 0x22500203, + 0x81C, 0x21520203, + 0x81C, 0x20540203, + 0x81C, 0x03560203, + 0x81C, 0x02580203, + 0x81C, 0x015A0203, + 0x81C, 0x005C0203, + 0x81C, 0x005E0203, + 0x81C, 0x00600203, + 0x81C, 0x00620203, + 0x81C, 0x00640203, + 0x81C, 0x00660203, + 0x81C, 0x00680203, + 0x81C, 0x006A0203, + 0x81C, 0x006C0203, + 0x81C, 0x006E0203, + 0x81C, 0x00700203, + 0x81C, 0x00720203, + 0x81C, 0x00740203, + 0x81C, 0x00760203, + 0x81C, 0x00780203, + 0x81C, 0x007A0203, + 0x81C, 0x007C0203, + 0x81C, 0x007E0203, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF8000203, + 0x81C, 0xF7020203, + 0x81C, 0xF6040203, + 0x81C, 0xF5060203, + 0x81C, 0xF4080203, + 0x81C, 0xF30A0203, + 0x81C, 0xF20C0203, + 0x81C, 0xF10E0203, + 0x81C, 0xF0100203, + 0x81C, 0xEF120203, + 0x81C, 0xEE140203, + 0x81C, 0xED160203, + 0x81C, 0xEC180203, + 0x81C, 0xEB1A0203, + 0x81C, 0xEA1C0203, + 0x81C, 0xE91E0203, + 0x81C, 0xE8200203, + 0x81C, 0xE7220203, + 0x81C, 0xE6240203, + 0x81C, 0xE5260203, + 0x81C, 0xE4280203, + 0x81C, 0xE32A0203, + 0x81C, 0xC42C0203, + 0x81C, 0xC32E0203, + 0x81C, 0xC2300203, + 0x81C, 0xC1320203, + 0x81C, 0xA3340203, + 0x81C, 0xA2360203, + 0x81C, 0xA1380203, + 0x81C, 0xA03A0203, + 0x81C, 0x823C0203, + 0x81C, 0x813E0203, + 0x81C, 0x80400203, + 0x81C, 0x65420203, + 0x81C, 0x64440203, + 0x81C, 0x63460203, + 0x81C, 0x62480203, + 0x81C, 0x614A0203, + 0x81C, 0x424C0203, + 0x81C, 0x414E0203, + 0x81C, 0x40500203, + 0x81C, 0x22520203, + 0x81C, 0x21540203, + 0x81C, 0x20560203, + 0x81C, 0x04580203, + 0x81C, 0x035A0203, + 0x81C, 0x025C0203, + 0x81C, 0x015E0203, + 0x81C, 0x00600203, + 0x81C, 0x00620203, + 0x81C, 0x00640203, + 0x81C, 0x00660203, + 0x81C, 0x00680203, + 0x81C, 0x006A0203, + 0x81C, 0x006C0203, + 0x81C, 0x006E0203, + 0x81C, 0x00700203, + 0x81C, 0x00720203, + 0x81C, 0x00740203, + 0x81C, 0x00760203, + 0x81C, 0x00780203, + 0x81C, 0x007A0203, + 0x81C, 0x007C0203, + 0x81C, 0x007E0203, + 0x90000008, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFB000203, + 0x81C, 0xFA020203, + 0x81C, 0xF9040203, + 0x81C, 0xF8060203, + 0x81C, 0xF7080203, + 0x81C, 0xF60A0203, + 0x81C, 0xF50C0203, + 0x81C, 0xF40E0203, + 0x81C, 0xF3100203, + 0x81C, 0xF2120203, + 0x81C, 0xF1140203, + 0x81C, 0xF0160203, + 0x81C, 0xEF180203, + 0x81C, 0xEE1A0203, + 0x81C, 0xED1C0203, + 0x81C, 0xEC1E0203, + 0x81C, 0xEB200203, + 0x81C, 0xEA220203, + 0x81C, 0xE9240203, + 0x81C, 0xE8260203, + 0x81C, 0xE7280203, + 0x81C, 0xE62A0203, + 0x81C, 0xE52C0203, + 0x81C, 0xE42E0203, + 0x81C, 0xE3300203, + 0x81C, 0xE2320203, + 0x81C, 0xC6340203, + 0x81C, 0xC5360203, + 0x81C, 0xC4380203, + 0x81C, 0xC33A0203, + 0x81C, 0xC23C0203, + 0x81C, 0xC13E0203, + 0x81C, 0xC0400203, + 0x81C, 0xA3420203, + 0x81C, 0xA2440203, + 0x81C, 0xA1460203, + 0x81C, 0xA0480203, + 0x81C, 0x824A0203, + 0x81C, 0x814C0203, + 0x81C, 0x804E0203, + 0x81C, 0x63500203, + 0x81C, 0x62520203, + 0x81C, 0x61540203, + 0x81C, 0x60560203, + 0x81C, 0x24580203, + 0x81C, 0x235A0203, + 0x81C, 0x225C0203, + 0x81C, 0x215E0203, + 0x81C, 0x20600203, + 0x81C, 0x03620203, + 0x81C, 0x02640203, + 0x81C, 0x01660203, + 0x81C, 0x00680203, + 0x81C, 0x006A0203, + 0x81C, 0x006C0203, + 0x81C, 0x006E0203, + 0x81C, 0x00700203, + 0x81C, 0x00720203, + 0x81C, 0x00740203, + 0x81C, 0x00760203, + 0x81C, 0x00780203, + 0x81C, 0x007A0203, + 0x81C, 0x007C0203, + 0x81C, 0x007E0203, + 0x90000009, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF8000203, + 0x81C, 0xF7020203, + 0x81C, 0xF6040203, + 0x81C, 0xF5060203, + 0x81C, 0xF4080203, + 0x81C, 0xF30A0203, + 0x81C, 0xF20C0203, + 0x81C, 0xF10E0203, + 0x81C, 0xF0100203, + 0x81C, 0xEF120203, + 0x81C, 0xEE140203, + 0x81C, 0xED160203, + 0x81C, 0xEC180203, + 0x81C, 0xEB1A0203, + 0x81C, 0xEA1C0203, + 0x81C, 0xE91E0203, + 0x81C, 0xE8200203, + 0x81C, 0xE7220203, + 0x81C, 0xE6240203, + 0x81C, 0xE5260203, + 0x81C, 0xE4280203, + 0x81C, 0xE32A0203, + 0x81C, 0xC42C0203, + 0x81C, 0xC32E0203, + 0x81C, 0xC2300203, + 0x81C, 0xC1320203, + 0x81C, 0xA3340203, + 0x81C, 0xA2360203, + 0x81C, 0xA1380203, + 0x81C, 0xA03A0203, + 0x81C, 0x823C0203, + 0x81C, 0x813E0203, + 0x81C, 0x80400203, + 0x81C, 0x65420203, + 0x81C, 0x64440203, + 0x81C, 0x63460203, + 0x81C, 0x62480203, + 0x81C, 0x614A0203, + 0x81C, 0x424C0203, + 0x81C, 0x414E0203, + 0x81C, 0x40500203, + 0x81C, 0x22520203, + 0x81C, 0x21540203, + 0x81C, 0x20560203, + 0x81C, 0x04580203, + 0x81C, 0x035A0203, + 0x81C, 0x025C0203, + 0x81C, 0x015E0203, + 0x81C, 0x00600203, + 0x81C, 0x00620203, + 0x81C, 0x00640203, + 0x81C, 0x00660203, + 0x81C, 0x00680203, + 0x81C, 0x006A0203, + 0x81C, 0x006C0203, + 0x81C, 0x006E0203, + 0x81C, 0x00700203, + 0x81C, 0x00720203, + 0x81C, 0x00740203, + 0x81C, 0x00760203, + 0x81C, 0x00780203, + 0x81C, 0x007A0203, + 0x81C, 0x007C0203, + 0x81C, 0x007E0203, + 0x9000000a, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000203, + 0x81C, 0xFB020203, + 0x81C, 0xFA040203, + 0x81C, 0xF9060203, + 0x81C, 0xF8080203, + 0x81C, 0xF70A0203, + 0x81C, 0xF60C0203, + 0x81C, 0xF50E0203, + 0x81C, 0xF4100203, + 0x81C, 0xF3120203, + 0x81C, 0xF2140203, + 0x81C, 0xF1160203, + 0x81C, 0xF0180203, + 0x81C, 0xEE1A0203, + 0x81C, 0xED1C0203, + 0x81C, 0xEC1E0203, + 0x81C, 0xEB200203, + 0x81C, 0xEA220203, + 0x81C, 0xE9240203, + 0x81C, 0xE8260203, + 0x81C, 0xE7280203, + 0x81C, 0xE62A0203, + 0x81C, 0xE52C0203, + 0x81C, 0xE42E0203, + 0x81C, 0xE3300203, + 0x81C, 0xE2320203, + 0x81C, 0xC6340203, + 0x81C, 0xC5360203, + 0x81C, 0xC4380203, + 0x81C, 0xC33A0203, + 0x81C, 0xA63C0203, + 0x81C, 0xA53E0203, + 0x81C, 0xA4400203, + 0x81C, 0xA3420203, + 0x81C, 0xA2440203, + 0x81C, 0xA1460203, + 0x81C, 0x83480203, + 0x81C, 0x824A0203, + 0x81C, 0x814C0203, + 0x81C, 0x804E0203, + 0x81C, 0x63500203, + 0x81C, 0x62520203, + 0x81C, 0x61540203, + 0x81C, 0x42560203, + 0x81C, 0x41580203, + 0x81C, 0x405A0203, + 0x81C, 0x225C0203, + 0x81C, 0x215E0203, + 0x81C, 0x20600203, + 0x81C, 0x04620203, + 0x81C, 0x03640203, + 0x81C, 0x02660203, + 0x81C, 0x01680203, + 0x81C, 0x006A0203, + 0x81C, 0x006C0203, + 0x81C, 0x006E0203, + 0x81C, 0x00700203, + 0x81C, 0x00720203, + 0x81C, 0x00740203, + 0x81C, 0x00760203, + 0x81C, 0x00780203, + 0x81C, 0x007A0203, + 0x81C, 0x007C0203, + 0x81C, 0x007E0203, + 0x9000000b, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF9000203, + 0x81C, 0xF8020203, + 0x81C, 0xF7040203, + 0x81C, 0xF6060203, + 0x81C, 0xF5080203, + 0x81C, 0xF40A0203, + 0x81C, 0xF30C0203, + 0x81C, 0xF20E0203, + 0x81C, 0xF1100203, + 0x81C, 0xF0120203, + 0x81C, 0xEF140203, + 0x81C, 0xEE160203, + 0x81C, 0xED180203, + 0x81C, 0xEC1A0203, + 0x81C, 0xEB1C0203, + 0x81C, 0xEA1E0203, + 0x81C, 0xE9200203, + 0x81C, 0xE8220203, + 0x81C, 0xE7240203, + 0x81C, 0xE6260203, + 0x81C, 0xE5280203, + 0x81C, 0xE42A0203, + 0x81C, 0xC42C0203, + 0x81C, 0xC32E0203, + 0x81C, 0xC2300203, + 0x81C, 0xC1320203, + 0x81C, 0xA3340203, + 0x81C, 0xA2360203, + 0x81C, 0xA1380203, + 0x81C, 0xA03A0203, + 0x81C, 0x823C0203, + 0x81C, 0x813E0203, + 0x81C, 0x80400203, + 0x81C, 0x64420203, + 0x81C, 0x63440203, + 0x81C, 0x62460203, + 0x81C, 0x61480203, + 0x81C, 0x604A0203, + 0x81C, 0x244C0203, + 0x81C, 0x234E0203, + 0x81C, 0x22500203, + 0x81C, 0x21520203, + 0x81C, 0x20540203, + 0x81C, 0x05560203, + 0x81C, 0x04580203, + 0x81C, 0x035A0203, + 0x81C, 0x025C0203, + 0x81C, 0x015E0203, + 0x81C, 0x00600203, + 0x81C, 0x00620203, + 0x81C, 0x00640203, + 0x81C, 0x00660203, + 0x81C, 0x00680203, + 0x81C, 0x006A0203, + 0x81C, 0x006C0203, + 0x81C, 0x006E0203, + 0x81C, 0x00700203, + 0x81C, 0x00720203, + 0x81C, 0x00740203, + 0x81C, 0x00760203, + 0x81C, 0x00780203, + 0x81C, 0x007A0203, + 0x81C, 0x007C0203, + 0x81C, 0x007E0203, + 0x9000000c, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000203, + 0x81C, 0xFB020203, + 0x81C, 0xFA040203, + 0x81C, 0xF9060203, + 0x81C, 0xF8080203, + 0x81C, 0xF70A0203, + 0x81C, 0xF60C0203, + 0x81C, 0xF50E0203, + 0x81C, 0xF4100203, + 0x81C, 0xF3120203, + 0x81C, 0xF2140203, + 0x81C, 0xF1160203, + 0x81C, 0xF0180203, + 0x81C, 0xEF1A0203, + 0x81C, 0xEE1C0203, + 0x81C, 0xED1E0203, + 0x81C, 0xEC200203, + 0x81C, 0xEB220203, + 0x81C, 0xEA240203, + 0x81C, 0xE9260203, + 0x81C, 0xE8280203, + 0x81C, 0xE72A0203, + 0x81C, 0xE62C0203, + 0x81C, 0xE52E0203, + 0x81C, 0xE4300203, + 0x81C, 0xE3320203, + 0x81C, 0xE2340203, + 0x81C, 0xC6360203, + 0x81C, 0xC5380203, + 0x81C, 0xC43A0203, + 0x81C, 0xC33C0203, + 0x81C, 0xA63E0203, + 0x81C, 0xA5400203, + 0x81C, 0xA4420203, + 0x81C, 0xA3440203, + 0x81C, 0xA2460203, + 0x81C, 0xA1480203, + 0x81C, 0x834A0203, + 0x81C, 0x824C0203, + 0x81C, 0x814E0203, + 0x81C, 0x64500203, + 0x81C, 0x63520203, + 0x81C, 0x62540203, + 0x81C, 0x61560203, + 0x81C, 0x60580203, + 0x81C, 0x405A0203, + 0x81C, 0x215C0203, + 0x81C, 0x205E0203, + 0x81C, 0x03600203, + 0x81C, 0x02620203, + 0x81C, 0x01640203, + 0x81C, 0x00660203, + 0x81C, 0x00680203, + 0x81C, 0x006A0203, + 0x81C, 0x006C0203, + 0x81C, 0x006E0203, + 0x81C, 0x00700203, + 0x81C, 0x00720203, + 0x81C, 0x00740203, + 0x81C, 0x00760203, + 0x81C, 0x00780203, + 0x81C, 0x007A0203, + 0x81C, 0x007C0203, + 0x81C, 0x007E0203, + 0x9000000d, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000203, + 0x81C, 0xFB020203, + 0x81C, 0xFA040203, + 0x81C, 0xF9060203, + 0x81C, 0xF8080203, + 0x81C, 0xF70A0203, + 0x81C, 0xF60C0203, + 0x81C, 0xF50E0203, + 0x81C, 0xF4100203, + 0x81C, 0xF3120203, + 0x81C, 0xF2140203, + 0x81C, 0xF1160203, + 0x81C, 0xF0180203, + 0x81C, 0xEE1A0203, + 0x81C, 0xED1C0203, + 0x81C, 0xEC1E0203, + 0x81C, 0xEB200203, + 0x81C, 0xEA220203, + 0x81C, 0xE9240203, + 0x81C, 0xE8260203, + 0x81C, 0xE7280203, + 0x81C, 0xE62A0203, + 0x81C, 0xE52C0203, + 0x81C, 0xE42E0203, + 0x81C, 0xE3300203, + 0x81C, 0xE2320203, + 0x81C, 0xC6340203, + 0x81C, 0xC5360203, + 0x81C, 0xC4380203, + 0x81C, 0xC33A0203, + 0x81C, 0xA63C0203, + 0x81C, 0xA53E0203, + 0x81C, 0xA4400203, + 0x81C, 0xA3420203, + 0x81C, 0xA2440203, + 0x81C, 0xA1460203, + 0x81C, 0x83480203, + 0x81C, 0x824A0203, + 0x81C, 0x814C0203, + 0x81C, 0x804E0203, + 0x81C, 0x63500203, + 0x81C, 0x62520203, + 0x81C, 0x61540203, + 0x81C, 0x42560203, + 0x81C, 0x41580203, + 0x81C, 0x405A0203, + 0x81C, 0x225C0203, + 0x81C, 0x215E0203, + 0x81C, 0x20600203, + 0x81C, 0x04620203, + 0x81C, 0x03640203, + 0x81C, 0x02660203, + 0x81C, 0x01680203, + 0x81C, 0x006A0203, + 0x81C, 0x006C0203, + 0x81C, 0x006E0203, + 0x81C, 0x00700203, + 0x81C, 0x00720203, + 0x81C, 0x00740203, + 0x81C, 0x00760203, + 0x81C, 0x00780203, + 0x81C, 0x007A0203, + 0x81C, 0x007C0203, + 0x81C, 0x007E0203, + 0x9000000e, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000203, + 0x81C, 0xFB020203, + 0x81C, 0xFA040203, + 0x81C, 0xF9060203, + 0x81C, 0xF8080203, + 0x81C, 0xF70A0203, + 0x81C, 0xF60C0203, + 0x81C, 0xF50E0203, + 0x81C, 0xF4100203, + 0x81C, 0xF3120203, + 0x81C, 0xF2140203, + 0x81C, 0xF1160203, + 0x81C, 0xF0180203, + 0x81C, 0xEE1A0203, + 0x81C, 0xED1C0203, + 0x81C, 0xEC1E0203, + 0x81C, 0xEB200203, + 0x81C, 0xEA220203, + 0x81C, 0xE9240203, + 0x81C, 0xE8260203, + 0x81C, 0xE7280203, + 0x81C, 0xE62A0203, + 0x81C, 0xE52C0203, + 0x81C, 0xE42E0203, + 0x81C, 0xE3300203, + 0x81C, 0xE2320203, + 0x81C, 0xC6340203, + 0x81C, 0xC5360203, + 0x81C, 0xC4380203, + 0x81C, 0xC33A0203, + 0x81C, 0xA63C0203, + 0x81C, 0xA53E0203, + 0x81C, 0xA4400203, + 0x81C, 0xA3420203, + 0x81C, 0xA2440203, + 0x81C, 0xA1460203, + 0x81C, 0x83480203, + 0x81C, 0x824A0203, + 0x81C, 0x814C0203, + 0x81C, 0x804E0203, + 0x81C, 0x63500203, + 0x81C, 0x62520203, + 0x81C, 0x61540203, + 0x81C, 0x42560203, + 0x81C, 0x41580203, + 0x81C, 0x405A0203, + 0x81C, 0x225C0203, + 0x81C, 0x215E0203, + 0x81C, 0x20600203, + 0x81C, 0x04620203, + 0x81C, 0x03640203, + 0x81C, 0x02660203, + 0x81C, 0x01680203, + 0x81C, 0x006A0203, + 0x81C, 0x006C0203, + 0x81C, 0x006E0203, + 0x81C, 0x00700203, + 0x81C, 0x00720203, + 0x81C, 0x00740203, + 0x81C, 0x00760203, + 0x81C, 0x00780203, + 0x81C, 0x007A0203, + 0x81C, 0x007C0203, + 0x81C, 0x007E0203, + 0x9000000f, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000203, + 0x81C, 0xFB020203, + 0x81C, 0xFA040203, + 0x81C, 0xF9060203, + 0x81C, 0xF8080203, + 0x81C, 0xF70A0203, + 0x81C, 0xF60C0203, + 0x81C, 0xF50E0203, + 0x81C, 0xF4100203, + 0x81C, 0xF3120203, + 0x81C, 0xF2140203, + 0x81C, 0xF1160203, + 0x81C, 0xF0180203, + 0x81C, 0xEF1A0203, + 0x81C, 0xEE1C0203, + 0x81C, 0xED1E0203, + 0x81C, 0xEC200203, + 0x81C, 0xEB220203, + 0x81C, 0xEA240203, + 0x81C, 0xE9260203, + 0x81C, 0xE8280203, + 0x81C, 0xE72A0203, + 0x81C, 0xE62C0203, + 0x81C, 0xE52E0203, + 0x81C, 0xE4300203, + 0x81C, 0xE3320203, + 0x81C, 0xE2340203, + 0x81C, 0xE1360203, + 0x81C, 0xE0380203, + 0x81C, 0xC33A0203, + 0x81C, 0xC23C0203, + 0x81C, 0xC13E0203, + 0x81C, 0xA3400203, + 0x81C, 0xA2420203, + 0x81C, 0xA1440203, + 0x81C, 0xA0460203, + 0x81C, 0x83480203, + 0x81C, 0x824A0203, + 0x81C, 0x814C0203, + 0x81C, 0x644E0203, + 0x81C, 0x63500203, + 0x81C, 0x62520203, + 0x81C, 0x61540203, + 0x81C, 0x42560203, + 0x81C, 0x41580203, + 0x81C, 0x235A0203, + 0x81C, 0x225C0203, + 0x81C, 0x215E0203, + 0x81C, 0x04600203, + 0x81C, 0x03620203, + 0x81C, 0x02640203, + 0x81C, 0x01660203, + 0x81C, 0x00680203, + 0x81C, 0x006A0203, + 0x81C, 0x006C0203, + 0x81C, 0x006E0203, + 0x81C, 0x00700203, + 0x81C, 0x00720203, + 0x81C, 0x00740203, + 0x81C, 0x00760203, + 0x81C, 0x00780203, + 0x81C, 0x007A0203, + 0x81C, 0x007C0203, + 0x81C, 0x007E0203, + 0x90000010, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000203, + 0x81C, 0xFB020203, + 0x81C, 0xFA040203, + 0x81C, 0xF9060203, + 0x81C, 0xF8080203, + 0x81C, 0xF70A0203, + 0x81C, 0xF60C0203, + 0x81C, 0xF50E0203, + 0x81C, 0xF4100203, + 0x81C, 0xF3120203, + 0x81C, 0xF2140203, + 0x81C, 0xF1160203, + 0x81C, 0xF0180203, + 0x81C, 0xEF1A0203, + 0x81C, 0xEE1C0203, + 0x81C, 0xED1E0203, + 0x81C, 0xEC200203, + 0x81C, 0xEB220203, + 0x81C, 0xEA240203, + 0x81C, 0xE9260203, + 0x81C, 0xE8280203, + 0x81C, 0xE72A0203, + 0x81C, 0xE62C0203, + 0x81C, 0xE52E0203, + 0x81C, 0xE4300203, + 0x81C, 0xE3320203, + 0x81C, 0xE2340203, + 0x81C, 0xC6360203, + 0x81C, 0xC5380203, + 0x81C, 0xC43A0203, + 0x81C, 0xC33C0203, + 0x81C, 0xA63E0203, + 0x81C, 0xA5400203, + 0x81C, 0xA4420203, + 0x81C, 0xA3440203, + 0x81C, 0xA2460203, + 0x81C, 0xA1480203, + 0x81C, 0x834A0203, + 0x81C, 0x824C0203, + 0x81C, 0x814E0203, + 0x81C, 0x64500203, + 0x81C, 0x63520203, + 0x81C, 0x62540203, + 0x81C, 0x61560203, + 0x81C, 0x60580203, + 0x81C, 0x405A0203, + 0x81C, 0x215C0203, + 0x81C, 0x205E0203, + 0x81C, 0x03600203, + 0x81C, 0x02620203, + 0x81C, 0x01640203, + 0x81C, 0x00660203, + 0x81C, 0x00680203, + 0x81C, 0x006A0203, + 0x81C, 0x006C0203, + 0x81C, 0x006E0203, + 0x81C, 0x00700203, + 0x81C, 0x00720203, + 0x81C, 0x00740203, + 0x81C, 0x00760203, + 0x81C, 0x00780203, + 0x81C, 0x007A0203, + 0x81C, 0x007C0203, + 0x81C, 0x007E0203, + 0x90000012, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF7000203, + 0x81C, 0xF6020203, + 0x81C, 0xF5040203, + 0x81C, 0xF4060203, + 0x81C, 0xF3080203, + 0x81C, 0xF20A0203, + 0x81C, 0xF10C0203, + 0x81C, 0xF00E0203, + 0x81C, 0xEF100203, + 0x81C, 0xEE120203, + 0x81C, 0xED140203, + 0x81C, 0xEC160203, + 0x81C, 0xEB180203, + 0x81C, 0xEA1A0203, + 0x81C, 0xE91C0203, + 0x81C, 0xE81E0203, + 0x81C, 0xE7200203, + 0x81C, 0xE6220203, + 0x81C, 0xE5240203, + 0x81C, 0xE4260203, + 0x81C, 0xE3280203, + 0x81C, 0xC42A0203, + 0x81C, 0xC32C0203, + 0x81C, 0xC22E0203, + 0x81C, 0xC1300203, + 0x81C, 0xC0320203, + 0x81C, 0xA3340203, + 0x81C, 0xA2360203, + 0x81C, 0xA1380203, + 0x81C, 0xA03A0203, + 0x81C, 0x823C0203, + 0x81C, 0x813E0203, + 0x81C, 0x80400203, + 0x81C, 0x64420203, + 0x81C, 0x63440203, + 0x81C, 0x62460203, + 0x81C, 0x61480203, + 0x81C, 0x604A0203, + 0x81C, 0x414C0203, + 0x81C, 0x404E0203, + 0x81C, 0x22500203, + 0x81C, 0x21520203, + 0x81C, 0x20540203, + 0x81C, 0x03560203, + 0x81C, 0x02580203, + 0x81C, 0x015A0203, + 0x81C, 0x005C0203, + 0x81C, 0x005E0203, + 0x81C, 0x00600203, + 0x81C, 0x00620203, + 0x81C, 0x00640203, + 0x81C, 0x00660203, + 0x81C, 0x00680203, + 0x81C, 0x006A0203, + 0x81C, 0x006C0203, + 0x81C, 0x006E0203, + 0x81C, 0x00700203, + 0x81C, 0x00720203, + 0x81C, 0x00740203, + 0x81C, 0x00760203, + 0x81C, 0x00780203, + 0x81C, 0x007A0203, + 0x81C, 0x007C0203, + 0x81C, 0x007E0203, + 0xA0000000, 0x00000000, + 0x81C, 0xFD000203, + 0x81C, 0xFC020203, + 0x81C, 0xFB040203, + 0x81C, 0xFA060203, + 0x81C, 0xF9080203, + 0x81C, 0xF80A0203, + 0x81C, 0xF70C0203, + 0x81C, 0xF60E0203, + 0x81C, 0xF5100203, + 0x81C, 0xF4120203, + 0x81C, 0xF3140203, + 0x81C, 0xF2160203, + 0x81C, 0xF1180203, + 0x81C, 0xF01A0203, + 0x81C, 0xEF1C0203, + 0x81C, 0xEE1E0203, + 0x81C, 0xED200203, + 0x81C, 0xEC220203, + 0x81C, 0xEB240203, + 0x81C, 0xEA260203, + 0x81C, 0xE9280203, + 0x81C, 0xE82A0203, + 0x81C, 0xE72C0203, + 0x81C, 0xE62E0203, + 0x81C, 0xE5300203, + 0x81C, 0xE4320203, + 0x81C, 0xE3340203, + 0x81C, 0xC6360203, + 0x81C, 0xC5380203, + 0x81C, 0xC43A0203, + 0x81C, 0xC33C0203, + 0x81C, 0xA63E0203, + 0x81C, 0xA5400203, + 0x81C, 0xA4420203, + 0x81C, 0xA3440203, + 0x81C, 0xA2460203, + 0x81C, 0xA1480203, + 0x81C, 0x834A0203, + 0x81C, 0x824C0203, + 0x81C, 0x814E0203, + 0x81C, 0x64500203, + 0x81C, 0x63520203, + 0x81C, 0x62540203, + 0x81C, 0x61560203, + 0x81C, 0x60580203, + 0x81C, 0x235A0203, + 0x81C, 0x225C0203, + 0x81C, 0x215E0203, + 0x81C, 0x20600203, + 0x81C, 0x03620203, + 0x81C, 0x02640203, + 0x81C, 0x01660203, + 0x81C, 0x00680203, + 0x81C, 0x006A0203, + 0x81C, 0x006C0203, + 0x81C, 0x006E0203, + 0x81C, 0x00700203, + 0x81C, 0x00720203, + 0x81C, 0x00740203, + 0x81C, 0x00760203, + 0x81C, 0x00780203, + 0x81C, 0x007A0203, + 0x81C, 0x007C0203, + 0x81C, 0x007E0203, + 0xB0000000, 0x00000000, + 0x80000000, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000303, + 0x81C, 0xFB020303, + 0x81C, 0xFA040303, + 0x81C, 0xF9060303, + 0x81C, 0xF8080303, + 0x81C, 0xF70A0303, + 0x81C, 0xF60C0303, + 0x81C, 0xF50E0303, + 0x81C, 0xF4100303, + 0x81C, 0xF3120303, + 0x81C, 0xF2140303, + 0x81C, 0xF1160303, + 0x81C, 0xEF180303, + 0x81C, 0xEE1A0303, + 0x81C, 0xED1C0303, + 0x81C, 0xEC1E0303, + 0x81C, 0xEB200303, + 0x81C, 0xEA220303, + 0x81C, 0xE9240303, + 0x81C, 0xE8260303, + 0x81C, 0xE7280303, + 0x81C, 0xE62A0303, + 0x81C, 0xE52C0303, + 0x81C, 0xE42E0303, + 0x81C, 0xE3300303, + 0x81C, 0xE2320303, + 0x81C, 0xC6340303, + 0x81C, 0xC5360303, + 0x81C, 0xC4380303, + 0x81C, 0xC33A0303, + 0x81C, 0xA63C0303, + 0x81C, 0xA53E0303, + 0x81C, 0xA4400303, + 0x81C, 0xA3420303, + 0x81C, 0xA2440303, + 0x81C, 0xA1460303, + 0x81C, 0x83480303, + 0x81C, 0x824A0303, + 0x81C, 0x814C0303, + 0x81C, 0x804E0303, + 0x81C, 0x63500303, + 0x81C, 0x62520303, + 0x81C, 0x61540303, + 0x81C, 0x42560303, + 0x81C, 0x41580303, + 0x81C, 0x405A0303, + 0x81C, 0x225C0303, + 0x81C, 0x215E0303, + 0x81C, 0x20600303, + 0x81C, 0x04620303, + 0x81C, 0x03640303, + 0x81C, 0x02660303, + 0x81C, 0x01680303, + 0x81C, 0x006A0303, + 0x81C, 0x006C0303, + 0x81C, 0x006E0303, + 0x81C, 0x00700303, + 0x81C, 0x00720303, + 0x81C, 0x00740303, + 0x81C, 0x00760303, + 0x81C, 0x00780303, + 0x81C, 0x007A0303, + 0x81C, 0x007C0303, + 0x81C, 0x007E0303, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF7000303, + 0x81C, 0xF6020303, + 0x81C, 0xF5040303, + 0x81C, 0xF4060303, + 0x81C, 0xF3080303, + 0x81C, 0xF20A0303, + 0x81C, 0xF10C0303, + 0x81C, 0xF00E0303, + 0x81C, 0xEF100303, + 0x81C, 0xEE120303, + 0x81C, 0xED140303, + 0x81C, 0xEC160303, + 0x81C, 0xEB180303, + 0x81C, 0xEA1A0303, + 0x81C, 0xE91C0303, + 0x81C, 0xCA1E0303, + 0x81C, 0xC9200303, + 0x81C, 0xC8220303, + 0x81C, 0xC7240303, + 0x81C, 0xC6260303, + 0x81C, 0xC5280303, + 0x81C, 0xC42A0303, + 0x81C, 0xC32C0303, + 0x81C, 0xC22E0303, + 0x81C, 0xC1300303, + 0x81C, 0xA4320303, + 0x81C, 0xA3340303, + 0x81C, 0xA2360303, + 0x81C, 0xA1380303, + 0x81C, 0xA03A0303, + 0x81C, 0x823C0303, + 0x81C, 0x813E0303, + 0x81C, 0x80400303, + 0x81C, 0x64420303, + 0x81C, 0x63440303, + 0x81C, 0x62460303, + 0x81C, 0x61480303, + 0x81C, 0x604A0303, + 0x81C, 0x414C0303, + 0x81C, 0x404E0303, + 0x81C, 0x06500303, + 0x81C, 0x05520303, + 0x81C, 0x04540303, + 0x81C, 0x03560303, + 0x81C, 0x02580303, + 0x81C, 0x015A0303, + 0x81C, 0x005C0303, + 0x81C, 0x005E0303, + 0x81C, 0x00600303, + 0x81C, 0x00620303, + 0x81C, 0x00640303, + 0x81C, 0x00660303, + 0x81C, 0x00680303, + 0x81C, 0x006A0303, + 0x81C, 0x006C0303, + 0x81C, 0x006E0303, + 0x81C, 0x00700303, + 0x81C, 0x00720303, + 0x81C, 0x00740303, + 0x81C, 0x00760303, + 0x81C, 0x00780303, + 0x81C, 0x007A0303, + 0x81C, 0x007C0303, + 0x81C, 0x007E0303, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF7000303, + 0x81C, 0xF6020303, + 0x81C, 0xF5040303, + 0x81C, 0xF4060303, + 0x81C, 0xF3080303, + 0x81C, 0xF20A0303, + 0x81C, 0xF10C0303, + 0x81C, 0xF00E0303, + 0x81C, 0xEF100303, + 0x81C, 0xEE120303, + 0x81C, 0xED140303, + 0x81C, 0xEC160303, + 0x81C, 0xEB180303, + 0x81C, 0xEA1A0303, + 0x81C, 0xE91C0303, + 0x81C, 0xCA1E0303, + 0x81C, 0xC9200303, + 0x81C, 0xC8220303, + 0x81C, 0xC7240303, + 0x81C, 0xC6260303, + 0x81C, 0xC5280303, + 0x81C, 0xC42A0303, + 0x81C, 0xC32C0303, + 0x81C, 0xC22E0303, + 0x81C, 0xC1300303, + 0x81C, 0xA4320303, + 0x81C, 0xA3340303, + 0x81C, 0xA2360303, + 0x81C, 0xA1380303, + 0x81C, 0xA03A0303, + 0x81C, 0x823C0303, + 0x81C, 0x813E0303, + 0x81C, 0x80400303, + 0x81C, 0x64420303, + 0x81C, 0x63440303, + 0x81C, 0x62460303, + 0x81C, 0x61480303, + 0x81C, 0x604A0303, + 0x81C, 0x414C0303, + 0x81C, 0x404E0303, + 0x81C, 0x22500303, + 0x81C, 0x21520303, + 0x81C, 0x20540303, + 0x81C, 0x03560303, + 0x81C, 0x02580303, + 0x81C, 0x015A0303, + 0x81C, 0x005C0303, + 0x81C, 0x005E0303, + 0x81C, 0x00600303, + 0x81C, 0x00620303, + 0x81C, 0x00640303, + 0x81C, 0x00660303, + 0x81C, 0x00680303, + 0x81C, 0x006A0303, + 0x81C, 0x006C0303, + 0x81C, 0x006E0303, + 0x81C, 0x00700303, + 0x81C, 0x00720303, + 0x81C, 0x00740303, + 0x81C, 0x00760303, + 0x81C, 0x00780303, + 0x81C, 0x007A0303, + 0x81C, 0x007C0303, + 0x81C, 0x007E0303, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000303, + 0x81C, 0xFB020303, + 0x81C, 0xFA040303, + 0x81C, 0xF9060303, + 0x81C, 0xF8080303, + 0x81C, 0xF70A0303, + 0x81C, 0xF60C0303, + 0x81C, 0xF50E0303, + 0x81C, 0xF4100303, + 0x81C, 0xF3120303, + 0x81C, 0xF2140303, + 0x81C, 0xF1160303, + 0x81C, 0xF0180303, + 0x81C, 0xEF1A0303, + 0x81C, 0xEE1C0303, + 0x81C, 0xED1E0303, + 0x81C, 0xEC200303, + 0x81C, 0xEB220303, + 0x81C, 0xEA240303, + 0x81C, 0xE9260303, + 0x81C, 0xE8280303, + 0x81C, 0xE72A0303, + 0x81C, 0xE62C0303, + 0x81C, 0xE52E0303, + 0x81C, 0xE4300303, + 0x81C, 0xE3320303, + 0x81C, 0xE2340303, + 0x81C, 0xC6360303, + 0x81C, 0xC5380303, + 0x81C, 0xC43A0303, + 0x81C, 0xC33C0303, + 0x81C, 0xA63E0303, + 0x81C, 0xA5400303, + 0x81C, 0xA4420303, + 0x81C, 0xA3440303, + 0x81C, 0xA2460303, + 0x81C, 0x84480303, + 0x81C, 0x834A0303, + 0x81C, 0x824C0303, + 0x81C, 0x814E0303, + 0x81C, 0x80500303, + 0x81C, 0x63520303, + 0x81C, 0x62540303, + 0x81C, 0x61560303, + 0x81C, 0x60580303, + 0x81C, 0x225A0303, + 0x81C, 0x055C0303, + 0x81C, 0x045E0303, + 0x81C, 0x03600303, + 0x81C, 0x02620303, + 0x81C, 0x01640303, + 0x81C, 0x00660303, + 0x81C, 0x00680303, + 0x81C, 0x006A0303, + 0x81C, 0x006C0303, + 0x81C, 0x006E0303, + 0x81C, 0x00700303, + 0x81C, 0x00720303, + 0x81C, 0x00740303, + 0x81C, 0x00760303, + 0x81C, 0x00780303, + 0x81C, 0x007A0303, + 0x81C, 0x007C0303, + 0x81C, 0x007E0303, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF7000303, + 0x81C, 0xF6020303, + 0x81C, 0xF5040303, + 0x81C, 0xF4060303, + 0x81C, 0xF3080303, + 0x81C, 0xF20A0303, + 0x81C, 0xF10C0303, + 0x81C, 0xF00E0303, + 0x81C, 0xEF100303, + 0x81C, 0xEE120303, + 0x81C, 0xED140303, + 0x81C, 0xEC160303, + 0x81C, 0xEB180303, + 0x81C, 0xEA1A0303, + 0x81C, 0xE91C0303, + 0x81C, 0xCA1E0303, + 0x81C, 0xC9200303, + 0x81C, 0xC8220303, + 0x81C, 0xC7240303, + 0x81C, 0xC6260303, + 0x81C, 0xC5280303, + 0x81C, 0xC42A0303, + 0x81C, 0xC32C0303, + 0x81C, 0xC22E0303, + 0x81C, 0xC1300303, + 0x81C, 0xA4320303, + 0x81C, 0xA3340303, + 0x81C, 0xA2360303, + 0x81C, 0xA1380303, + 0x81C, 0xA03A0303, + 0x81C, 0x823C0303, + 0x81C, 0x813E0303, + 0x81C, 0x80400303, + 0x81C, 0x64420303, + 0x81C, 0x63440303, + 0x81C, 0x62460303, + 0x81C, 0x61480303, + 0x81C, 0x604A0303, + 0x81C, 0x414C0303, + 0x81C, 0x404E0303, + 0x81C, 0x22500303, + 0x81C, 0x21520303, + 0x81C, 0x20540303, + 0x81C, 0x03560303, + 0x81C, 0x02580303, + 0x81C, 0x015A0303, + 0x81C, 0x005C0303, + 0x81C, 0x005E0303, + 0x81C, 0x00600303, + 0x81C, 0x00620303, + 0x81C, 0x00640303, + 0x81C, 0x00660303, + 0x81C, 0x00680303, + 0x81C, 0x006A0303, + 0x81C, 0x006C0303, + 0x81C, 0x006E0303, + 0x81C, 0x00700303, + 0x81C, 0x00720303, + 0x81C, 0x00740303, + 0x81C, 0x00760303, + 0x81C, 0x00780303, + 0x81C, 0x007A0303, + 0x81C, 0x007C0303, + 0x81C, 0x007E0303, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFB000303, + 0x81C, 0xFA020303, + 0x81C, 0xF9040303, + 0x81C, 0xF8060303, + 0x81C, 0xF7080303, + 0x81C, 0xF60A0303, + 0x81C, 0xF50C0303, + 0x81C, 0xF40E0303, + 0x81C, 0xF3100303, + 0x81C, 0xF2120303, + 0x81C, 0xF1140303, + 0x81C, 0xF0160303, + 0x81C, 0xEF180303, + 0x81C, 0xEE1A0303, + 0x81C, 0xED1C0303, + 0x81C, 0xEC1E0303, + 0x81C, 0xEB200303, + 0x81C, 0xEA220303, + 0x81C, 0xE9240303, + 0x81C, 0xE8260303, + 0x81C, 0xE7280303, + 0x81C, 0xE62A0303, + 0x81C, 0xE52C0303, + 0x81C, 0xE42E0303, + 0x81C, 0xE3300303, + 0x81C, 0xE2320303, + 0x81C, 0xE1340303, + 0x81C, 0xC5360303, + 0x81C, 0xC4380303, + 0x81C, 0xC33A0303, + 0x81C, 0xC23C0303, + 0x81C, 0xC13E0303, + 0x81C, 0xA4400303, + 0x81C, 0xA3420303, + 0x81C, 0xA2440303, + 0x81C, 0xA1460303, + 0x81C, 0x83480303, + 0x81C, 0x824A0303, + 0x81C, 0x814C0303, + 0x81C, 0x804E0303, + 0x81C, 0x64500303, + 0x81C, 0x63520303, + 0x81C, 0x62540303, + 0x81C, 0x61560303, + 0x81C, 0x60580303, + 0x81C, 0x235A0303, + 0x81C, 0x225C0303, + 0x81C, 0x215E0303, + 0x81C, 0x20600303, + 0x81C, 0x04620303, + 0x81C, 0x03640303, + 0x81C, 0x02660303, + 0x81C, 0x01680303, + 0x81C, 0x006A0303, + 0x81C, 0x006C0303, + 0x81C, 0x006E0303, + 0x81C, 0x00700303, + 0x81C, 0x00720303, + 0x81C, 0x00740303, + 0x81C, 0x00760303, + 0x81C, 0x00780303, + 0x81C, 0x007A0303, + 0x81C, 0x007C0303, + 0x81C, 0x007E0303, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF9000303, + 0x81C, 0xF8020303, + 0x81C, 0xF7040303, + 0x81C, 0xF6060303, + 0x81C, 0xF5080303, + 0x81C, 0xF40A0303, + 0x81C, 0xF30C0303, + 0x81C, 0xF20E0303, + 0x81C, 0xF1100303, + 0x81C, 0xF0120303, + 0x81C, 0xEF140303, + 0x81C, 0xEE160303, + 0x81C, 0xED180303, + 0x81C, 0xEC1A0303, + 0x81C, 0xEB1C0303, + 0x81C, 0xEA1E0303, + 0x81C, 0xC9200303, + 0x81C, 0xC8220303, + 0x81C, 0xC7240303, + 0x81C, 0xC6260303, + 0x81C, 0xC5280303, + 0x81C, 0xC42A0303, + 0x81C, 0xC32C0303, + 0x81C, 0xC22E0303, + 0x81C, 0xC1300303, + 0x81C, 0xC0320303, + 0x81C, 0xA3340303, + 0x81C, 0xA2360303, + 0x81C, 0xA1380303, + 0x81C, 0xA03A0303, + 0x81C, 0x823C0303, + 0x81C, 0x813E0303, + 0x81C, 0x80400303, + 0x81C, 0x64420303, + 0x81C, 0x63440303, + 0x81C, 0x62460303, + 0x81C, 0x61480303, + 0x81C, 0x604A0303, + 0x81C, 0x414C0303, + 0x81C, 0x404E0303, + 0x81C, 0x22500303, + 0x81C, 0x21520303, + 0x81C, 0x20540303, + 0x81C, 0x03560303, + 0x81C, 0x02580303, + 0x81C, 0x015A0303, + 0x81C, 0x005C0303, + 0x81C, 0x005E0303, + 0x81C, 0x00600303, + 0x81C, 0x00620303, + 0x81C, 0x00640303, + 0x81C, 0x00660303, + 0x81C, 0x00680303, + 0x81C, 0x006A0303, + 0x81C, 0x006C0303, + 0x81C, 0x006E0303, + 0x81C, 0x00700303, + 0x81C, 0x00720303, + 0x81C, 0x00740303, + 0x81C, 0x00760303, + 0x81C, 0x00780303, + 0x81C, 0x007A0303, + 0x81C, 0x007C0303, + 0x81C, 0x007E0303, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF8000303, + 0x81C, 0xF7020303, + 0x81C, 0xF6040303, + 0x81C, 0xF5060303, + 0x81C, 0xF4080303, + 0x81C, 0xF30A0303, + 0x81C, 0xF20C0303, + 0x81C, 0xF10E0303, + 0x81C, 0xF0100303, + 0x81C, 0xEF120303, + 0x81C, 0xEE140303, + 0x81C, 0xED160303, + 0x81C, 0xEC180303, + 0x81C, 0xEB1A0303, + 0x81C, 0xEA1C0303, + 0x81C, 0xE91E0303, + 0x81C, 0xCA200303, + 0x81C, 0xC9220303, + 0x81C, 0xC8240303, + 0x81C, 0xC7260303, + 0x81C, 0xC6280303, + 0x81C, 0xC52A0303, + 0x81C, 0xC42C0303, + 0x81C, 0xC32E0303, + 0x81C, 0xC2300303, + 0x81C, 0xC1320303, + 0x81C, 0xA3340303, + 0x81C, 0xA2360303, + 0x81C, 0xA1380303, + 0x81C, 0xA03A0303, + 0x81C, 0x823C0303, + 0x81C, 0x813E0303, + 0x81C, 0x80400303, + 0x81C, 0x65420303, + 0x81C, 0x64440303, + 0x81C, 0x63460303, + 0x81C, 0x62480303, + 0x81C, 0x614A0303, + 0x81C, 0x424C0303, + 0x81C, 0x414E0303, + 0x81C, 0x40500303, + 0x81C, 0x22520303, + 0x81C, 0x21540303, + 0x81C, 0x20560303, + 0x81C, 0x04580303, + 0x81C, 0x035A0303, + 0x81C, 0x025C0303, + 0x81C, 0x015E0303, + 0x81C, 0x00600303, + 0x81C, 0x00620303, + 0x81C, 0x00640303, + 0x81C, 0x00660303, + 0x81C, 0x00680303, + 0x81C, 0x006A0303, + 0x81C, 0x006C0303, + 0x81C, 0x006E0303, + 0x81C, 0x00700303, + 0x81C, 0x00720303, + 0x81C, 0x00740303, + 0x81C, 0x00760303, + 0x81C, 0x00780303, + 0x81C, 0x007A0303, + 0x81C, 0x007C0303, + 0x81C, 0x007E0303, + 0x90000008, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFB000303, + 0x81C, 0xFA020303, + 0x81C, 0xF9040303, + 0x81C, 0xF8060303, + 0x81C, 0xF7080303, + 0x81C, 0xF60A0303, + 0x81C, 0xF50C0303, + 0x81C, 0xF40E0303, + 0x81C, 0xF3100303, + 0x81C, 0xF2120303, + 0x81C, 0xF1140303, + 0x81C, 0xF0160303, + 0x81C, 0xEF180303, + 0x81C, 0xEE1A0303, + 0x81C, 0xED1C0303, + 0x81C, 0xEC1E0303, + 0x81C, 0xEB200303, + 0x81C, 0xEA220303, + 0x81C, 0xE9240303, + 0x81C, 0xE8260303, + 0x81C, 0xE7280303, + 0x81C, 0xE62A0303, + 0x81C, 0xE52C0303, + 0x81C, 0xE42E0303, + 0x81C, 0xE3300303, + 0x81C, 0xE2320303, + 0x81C, 0xC6340303, + 0x81C, 0xC5360303, + 0x81C, 0xC4380303, + 0x81C, 0xC33A0303, + 0x81C, 0xC23C0303, + 0x81C, 0xC13E0303, + 0x81C, 0xA4400303, + 0x81C, 0xA3420303, + 0x81C, 0xA2440303, + 0x81C, 0xA1460303, + 0x81C, 0x83480303, + 0x81C, 0x824A0303, + 0x81C, 0x814C0303, + 0x81C, 0x804E0303, + 0x81C, 0x63500303, + 0x81C, 0x62520303, + 0x81C, 0x43540303, + 0x81C, 0x42560303, + 0x81C, 0x41580303, + 0x81C, 0x235A0303, + 0x81C, 0x225C0303, + 0x81C, 0x215E0303, + 0x81C, 0x20600303, + 0x81C, 0x04620303, + 0x81C, 0x03640303, + 0x81C, 0x02660303, + 0x81C, 0x01680303, + 0x81C, 0x006A0303, + 0x81C, 0x006C0303, + 0x81C, 0x006E0303, + 0x81C, 0x00700303, + 0x81C, 0x00720303, + 0x81C, 0x00740303, + 0x81C, 0x00760303, + 0x81C, 0x00780303, + 0x81C, 0x007A0303, + 0x81C, 0x007C0303, + 0x81C, 0x007E0303, + 0x90000009, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF8000303, + 0x81C, 0xF7020303, + 0x81C, 0xF6040303, + 0x81C, 0xF5060303, + 0x81C, 0xF4080303, + 0x81C, 0xF30A0303, + 0x81C, 0xF20C0303, + 0x81C, 0xF10E0303, + 0x81C, 0xF0100303, + 0x81C, 0xEF120303, + 0x81C, 0xEE140303, + 0x81C, 0xED160303, + 0x81C, 0xEC180303, + 0x81C, 0xEB1A0303, + 0x81C, 0xEA1C0303, + 0x81C, 0xE91E0303, + 0x81C, 0xCA200303, + 0x81C, 0xC9220303, + 0x81C, 0xC8240303, + 0x81C, 0xC7260303, + 0x81C, 0xC6280303, + 0x81C, 0xC52A0303, + 0x81C, 0xC42C0303, + 0x81C, 0xC32E0303, + 0x81C, 0xC2300303, + 0x81C, 0xC1320303, + 0x81C, 0xA3340303, + 0x81C, 0xA2360303, + 0x81C, 0xA1380303, + 0x81C, 0xA03A0303, + 0x81C, 0x823C0303, + 0x81C, 0x813E0303, + 0x81C, 0x80400303, + 0x81C, 0x65420303, + 0x81C, 0x64440303, + 0x81C, 0x63460303, + 0x81C, 0x62480303, + 0x81C, 0x614A0303, + 0x81C, 0x424C0303, + 0x81C, 0x414E0303, + 0x81C, 0x40500303, + 0x81C, 0x22520303, + 0x81C, 0x21540303, + 0x81C, 0x20560303, + 0x81C, 0x04580303, + 0x81C, 0x035A0303, + 0x81C, 0x025C0303, + 0x81C, 0x015E0303, + 0x81C, 0x00600303, + 0x81C, 0x00620303, + 0x81C, 0x00640303, + 0x81C, 0x00660303, + 0x81C, 0x00680303, + 0x81C, 0x006A0303, + 0x81C, 0x006C0303, + 0x81C, 0x006E0303, + 0x81C, 0x00700303, + 0x81C, 0x00720303, + 0x81C, 0x00740303, + 0x81C, 0x00760303, + 0x81C, 0x00780303, + 0x81C, 0x007A0303, + 0x81C, 0x007C0303, + 0x81C, 0x007E0303, + 0x9000000a, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000303, + 0x81C, 0xFB020303, + 0x81C, 0xFA040303, + 0x81C, 0xF9060303, + 0x81C, 0xF8080303, + 0x81C, 0xF70A0303, + 0x81C, 0xF60C0303, + 0x81C, 0xF50E0303, + 0x81C, 0xF4100303, + 0x81C, 0xF3120303, + 0x81C, 0xF2140303, + 0x81C, 0xF1160303, + 0x81C, 0xEF180303, + 0x81C, 0xEE1A0303, + 0x81C, 0xED1C0303, + 0x81C, 0xEC1E0303, + 0x81C, 0xEB200303, + 0x81C, 0xEA220303, + 0x81C, 0xE9240303, + 0x81C, 0xE8260303, + 0x81C, 0xE7280303, + 0x81C, 0xE62A0303, + 0x81C, 0xE52C0303, + 0x81C, 0xE42E0303, + 0x81C, 0xE3300303, + 0x81C, 0xE2320303, + 0x81C, 0xC6340303, + 0x81C, 0xC5360303, + 0x81C, 0xC4380303, + 0x81C, 0xC33A0303, + 0x81C, 0xA63C0303, + 0x81C, 0xA53E0303, + 0x81C, 0xA4400303, + 0x81C, 0xA3420303, + 0x81C, 0xA2440303, + 0x81C, 0xA1460303, + 0x81C, 0x83480303, + 0x81C, 0x824A0303, + 0x81C, 0x814C0303, + 0x81C, 0x804E0303, + 0x81C, 0x63500303, + 0x81C, 0x62520303, + 0x81C, 0x61540303, + 0x81C, 0x42560303, + 0x81C, 0x41580303, + 0x81C, 0x405A0303, + 0x81C, 0x225C0303, + 0x81C, 0x215E0303, + 0x81C, 0x20600303, + 0x81C, 0x04620303, + 0x81C, 0x03640303, + 0x81C, 0x02660303, + 0x81C, 0x01680303, + 0x81C, 0x006A0303, + 0x81C, 0x006C0303, + 0x81C, 0x006E0303, + 0x81C, 0x00700303, + 0x81C, 0x00720303, + 0x81C, 0x00740303, + 0x81C, 0x00760303, + 0x81C, 0x00780303, + 0x81C, 0x007A0303, + 0x81C, 0x007C0303, + 0x81C, 0x007E0303, + 0x9000000b, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF8000303, + 0x81C, 0xF7020303, + 0x81C, 0xF6040303, + 0x81C, 0xF5060303, + 0x81C, 0xF4080303, + 0x81C, 0xF30A0303, + 0x81C, 0xF20C0303, + 0x81C, 0xF10E0303, + 0x81C, 0xF0100303, + 0x81C, 0xEF120303, + 0x81C, 0xEE140303, + 0x81C, 0xED160303, + 0x81C, 0xEC180303, + 0x81C, 0xEB1A0303, + 0x81C, 0xEA1C0303, + 0x81C, 0xE91E0303, + 0x81C, 0xCA200303, + 0x81C, 0xC9220303, + 0x81C, 0xC8240303, + 0x81C, 0xC7260303, + 0x81C, 0xC6280303, + 0x81C, 0xC52A0303, + 0x81C, 0xC42C0303, + 0x81C, 0xC32E0303, + 0x81C, 0xC2300303, + 0x81C, 0xC1320303, + 0x81C, 0xA3340303, + 0x81C, 0xA2360303, + 0x81C, 0xA1380303, + 0x81C, 0xA03A0303, + 0x81C, 0x823C0303, + 0x81C, 0x813E0303, + 0x81C, 0x80400303, + 0x81C, 0x64420303, + 0x81C, 0x63440303, + 0x81C, 0x62460303, + 0x81C, 0x61480303, + 0x81C, 0x604A0303, + 0x81C, 0x234C0303, + 0x81C, 0x224E0303, + 0x81C, 0x21500303, + 0x81C, 0x20520303, + 0x81C, 0x06540303, + 0x81C, 0x05560303, + 0x81C, 0x04580303, + 0x81C, 0x035A0303, + 0x81C, 0x025C0303, + 0x81C, 0x015E0303, + 0x81C, 0x00600303, + 0x81C, 0x00620303, + 0x81C, 0x00640303, + 0x81C, 0x00660303, + 0x81C, 0x00680303, + 0x81C, 0x006A0303, + 0x81C, 0x006C0303, + 0x81C, 0x006E0303, + 0x81C, 0x00700303, + 0x81C, 0x00720303, + 0x81C, 0x00740303, + 0x81C, 0x00760303, + 0x81C, 0x00780303, + 0x81C, 0x007A0303, + 0x81C, 0x007C0303, + 0x81C, 0x007E0303, + 0x9000000c, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000303, + 0x81C, 0xFB020303, + 0x81C, 0xFA040303, + 0x81C, 0xF9060303, + 0x81C, 0xF8080303, + 0x81C, 0xF70A0303, + 0x81C, 0xF60C0303, + 0x81C, 0xF50E0303, + 0x81C, 0xF4100303, + 0x81C, 0xF3120303, + 0x81C, 0xF2140303, + 0x81C, 0xF1160303, + 0x81C, 0xF0180303, + 0x81C, 0xEF1A0303, + 0x81C, 0xEE1C0303, + 0x81C, 0xED1E0303, + 0x81C, 0xEC200303, + 0x81C, 0xEB220303, + 0x81C, 0xEA240303, + 0x81C, 0xE9260303, + 0x81C, 0xE8280303, + 0x81C, 0xE72A0303, + 0x81C, 0xE62C0303, + 0x81C, 0xE52E0303, + 0x81C, 0xE4300303, + 0x81C, 0xE3320303, + 0x81C, 0xE2340303, + 0x81C, 0xC6360303, + 0x81C, 0xC5380303, + 0x81C, 0xC43A0303, + 0x81C, 0xC33C0303, + 0x81C, 0xA63E0303, + 0x81C, 0xA5400303, + 0x81C, 0xA4420303, + 0x81C, 0xA3440303, + 0x81C, 0xA2460303, + 0x81C, 0x84480303, + 0x81C, 0x834A0303, + 0x81C, 0x824C0303, + 0x81C, 0x814E0303, + 0x81C, 0x80500303, + 0x81C, 0x63520303, + 0x81C, 0x62540303, + 0x81C, 0x61560303, + 0x81C, 0x60580303, + 0x81C, 0x225A0303, + 0x81C, 0x055C0303, + 0x81C, 0x045E0303, + 0x81C, 0x03600303, + 0x81C, 0x02620303, + 0x81C, 0x01640303, + 0x81C, 0x00660303, + 0x81C, 0x00680303, + 0x81C, 0x006A0303, + 0x81C, 0x006C0303, + 0x81C, 0x006E0303, + 0x81C, 0x00700303, + 0x81C, 0x00720303, + 0x81C, 0x00740303, + 0x81C, 0x00760303, + 0x81C, 0x00780303, + 0x81C, 0x007A0303, + 0x81C, 0x007C0303, + 0x81C, 0x007E0303, + 0x9000000d, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFB000303, + 0x81C, 0xFA020303, + 0x81C, 0xF9040303, + 0x81C, 0xF8060303, + 0x81C, 0xF7080303, + 0x81C, 0xF60A0303, + 0x81C, 0xF50C0303, + 0x81C, 0xF40E0303, + 0x81C, 0xF3100303, + 0x81C, 0xF2120303, + 0x81C, 0xF1140303, + 0x81C, 0xEF160303, + 0x81C, 0xEE180303, + 0x81C, 0xED1A0303, + 0x81C, 0xEC1C0303, + 0x81C, 0xEB1E0303, + 0x81C, 0xEA200303, + 0x81C, 0xE9220303, + 0x81C, 0xE8240303, + 0x81C, 0xE7260303, + 0x81C, 0xE6280303, + 0x81C, 0xE52A0303, + 0x81C, 0xE42C0303, + 0x81C, 0xE32E0303, + 0x81C, 0xE2300303, + 0x81C, 0xE1320303, + 0x81C, 0xC6340303, + 0x81C, 0xC5360303, + 0x81C, 0xC4380303, + 0x81C, 0xC33A0303, + 0x81C, 0xA63C0303, + 0x81C, 0xA53E0303, + 0x81C, 0xA4400303, + 0x81C, 0xA3420303, + 0x81C, 0xA2440303, + 0x81C, 0xA1460303, + 0x81C, 0x83480303, + 0x81C, 0x824A0303, + 0x81C, 0x814C0303, + 0x81C, 0x804E0303, + 0x81C, 0x63500303, + 0x81C, 0x62520303, + 0x81C, 0x61540303, + 0x81C, 0x42560303, + 0x81C, 0x41580303, + 0x81C, 0x405A0303, + 0x81C, 0x225C0303, + 0x81C, 0x215E0303, + 0x81C, 0x20600303, + 0x81C, 0x04620303, + 0x81C, 0x03640303, + 0x81C, 0x02660303, + 0x81C, 0x01680303, + 0x81C, 0x006A0303, + 0x81C, 0x006C0303, + 0x81C, 0x006E0303, + 0x81C, 0x00700303, + 0x81C, 0x00720303, + 0x81C, 0x00740303, + 0x81C, 0x00760303, + 0x81C, 0x00780303, + 0x81C, 0x007A0303, + 0x81C, 0x007C0303, + 0x81C, 0x007E0303, + 0x9000000e, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFB000303, + 0x81C, 0xFA020303, + 0x81C, 0xF9040303, + 0x81C, 0xF8060303, + 0x81C, 0xF7080303, + 0x81C, 0xF60A0303, + 0x81C, 0xF50C0303, + 0x81C, 0xF40E0303, + 0x81C, 0xF3100303, + 0x81C, 0xF2120303, + 0x81C, 0xF1140303, + 0x81C, 0xEF160303, + 0x81C, 0xEE180303, + 0x81C, 0xED1A0303, + 0x81C, 0xEC1C0303, + 0x81C, 0xEB1E0303, + 0x81C, 0xEA200303, + 0x81C, 0xE9220303, + 0x81C, 0xE8240303, + 0x81C, 0xE7260303, + 0x81C, 0xE6280303, + 0x81C, 0xE52A0303, + 0x81C, 0xE42C0303, + 0x81C, 0xE32E0303, + 0x81C, 0xE2300303, + 0x81C, 0xE1320303, + 0x81C, 0xC6340303, + 0x81C, 0xC5360303, + 0x81C, 0xC4380303, + 0x81C, 0xC33A0303, + 0x81C, 0xA63C0303, + 0x81C, 0xA53E0303, + 0x81C, 0xA4400303, + 0x81C, 0xA3420303, + 0x81C, 0xA2440303, + 0x81C, 0xA1460303, + 0x81C, 0x83480303, + 0x81C, 0x824A0303, + 0x81C, 0x814C0303, + 0x81C, 0x804E0303, + 0x81C, 0x63500303, + 0x81C, 0x62520303, + 0x81C, 0x61540303, + 0x81C, 0x42560303, + 0x81C, 0x41580303, + 0x81C, 0x405A0303, + 0x81C, 0x225C0303, + 0x81C, 0x215E0303, + 0x81C, 0x20600303, + 0x81C, 0x04620303, + 0x81C, 0x03640303, + 0x81C, 0x02660303, + 0x81C, 0x01680303, + 0x81C, 0x006A0303, + 0x81C, 0x006C0303, + 0x81C, 0x006E0303, + 0x81C, 0x00700303, + 0x81C, 0x00720303, + 0x81C, 0x00740303, + 0x81C, 0x00760303, + 0x81C, 0x00780303, + 0x81C, 0x007A0303, + 0x81C, 0x007C0303, + 0x81C, 0x007E0303, + 0x9000000f, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFB000303, + 0x81C, 0xFA020303, + 0x81C, 0xF9040303, + 0x81C, 0xF8060303, + 0x81C, 0xF7080303, + 0x81C, 0xF60A0303, + 0x81C, 0xF50C0303, + 0x81C, 0xF40E0303, + 0x81C, 0xF3100303, + 0x81C, 0xF2120303, + 0x81C, 0xF1140303, + 0x81C, 0xF0160303, + 0x81C, 0xEF180303, + 0x81C, 0xEE1A0303, + 0x81C, 0xED1C0303, + 0x81C, 0xEC1E0303, + 0x81C, 0xEB200303, + 0x81C, 0xEA220303, + 0x81C, 0xE9240303, + 0x81C, 0xE8260303, + 0x81C, 0xE7280303, + 0x81C, 0xE62A0303, + 0x81C, 0xE52C0303, + 0x81C, 0xE42E0303, + 0x81C, 0xE3300303, + 0x81C, 0xE2320303, + 0x81C, 0xE1340303, + 0x81C, 0xE0360303, + 0x81C, 0xC3380303, + 0x81C, 0xC23A0303, + 0x81C, 0xC13C0303, + 0x81C, 0xC03E0303, + 0x81C, 0xA3400303, + 0x81C, 0xA2420303, + 0x81C, 0xA1440303, + 0x81C, 0xA0460303, + 0x81C, 0x83480303, + 0x81C, 0x824A0303, + 0x81C, 0x814C0303, + 0x81C, 0x644E0303, + 0x81C, 0x63500303, + 0x81C, 0x62520303, + 0x81C, 0x61540303, + 0x81C, 0x24560303, + 0x81C, 0x23580303, + 0x81C, 0x225A0303, + 0x81C, 0x215C0303, + 0x81C, 0x055E0303, + 0x81C, 0x04600303, + 0x81C, 0x03620303, + 0x81C, 0x02640303, + 0x81C, 0x01660303, + 0x81C, 0x00680303, + 0x81C, 0x006A0303, + 0x81C, 0x006C0303, + 0x81C, 0x006E0303, + 0x81C, 0x00700303, + 0x81C, 0x00720303, + 0x81C, 0x00740303, + 0x81C, 0x00760303, + 0x81C, 0x00780303, + 0x81C, 0x007A0303, + 0x81C, 0x007C0303, + 0x81C, 0x007E0303, + 0x90000010, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000303, + 0x81C, 0xFB020303, + 0x81C, 0xFA040303, + 0x81C, 0xF9060303, + 0x81C, 0xF8080303, + 0x81C, 0xF70A0303, + 0x81C, 0xF60C0303, + 0x81C, 0xF50E0303, + 0x81C, 0xF4100303, + 0x81C, 0xF3120303, + 0x81C, 0xF2140303, + 0x81C, 0xF1160303, + 0x81C, 0xF0180303, + 0x81C, 0xEF1A0303, + 0x81C, 0xEE1C0303, + 0x81C, 0xED1E0303, + 0x81C, 0xEC200303, + 0x81C, 0xEB220303, + 0x81C, 0xEA240303, + 0x81C, 0xE9260303, + 0x81C, 0xE8280303, + 0x81C, 0xE72A0303, + 0x81C, 0xE62C0303, + 0x81C, 0xE52E0303, + 0x81C, 0xE4300303, + 0x81C, 0xE3320303, + 0x81C, 0xE2340303, + 0x81C, 0xC6360303, + 0x81C, 0xC5380303, + 0x81C, 0xC43A0303, + 0x81C, 0xC33C0303, + 0x81C, 0xA63E0303, + 0x81C, 0xA5400303, + 0x81C, 0xA4420303, + 0x81C, 0xA3440303, + 0x81C, 0xA2460303, + 0x81C, 0x84480303, + 0x81C, 0x834A0303, + 0x81C, 0x824C0303, + 0x81C, 0x814E0303, + 0x81C, 0x80500303, + 0x81C, 0x63520303, + 0x81C, 0x62540303, + 0x81C, 0x61560303, + 0x81C, 0x60580303, + 0x81C, 0x225A0303, + 0x81C, 0x055C0303, + 0x81C, 0x045E0303, + 0x81C, 0x03600303, + 0x81C, 0x02620303, + 0x81C, 0x01640303, + 0x81C, 0x00660303, + 0x81C, 0x00680303, + 0x81C, 0x006A0303, + 0x81C, 0x006C0303, + 0x81C, 0x006E0303, + 0x81C, 0x00700303, + 0x81C, 0x00720303, + 0x81C, 0x00740303, + 0x81C, 0x00760303, + 0x81C, 0x00780303, + 0x81C, 0x007A0303, + 0x81C, 0x007C0303, + 0x81C, 0x007E0303, + 0x90000012, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF7000303, + 0x81C, 0xF6020303, + 0x81C, 0xF5040303, + 0x81C, 0xF4060303, + 0x81C, 0xF3080303, + 0x81C, 0xF20A0303, + 0x81C, 0xF10C0303, + 0x81C, 0xF00E0303, + 0x81C, 0xEF100303, + 0x81C, 0xEE120303, + 0x81C, 0xED140303, + 0x81C, 0xEC160303, + 0x81C, 0xEB180303, + 0x81C, 0xEA1A0303, + 0x81C, 0xE91C0303, + 0x81C, 0xCA1E0303, + 0x81C, 0xC9200303, + 0x81C, 0xC8220303, + 0x81C, 0xC7240303, + 0x81C, 0xC6260303, + 0x81C, 0xC5280303, + 0x81C, 0xC42A0303, + 0x81C, 0xC32C0303, + 0x81C, 0xC22E0303, + 0x81C, 0xC1300303, + 0x81C, 0xA4320303, + 0x81C, 0xA3340303, + 0x81C, 0xA2360303, + 0x81C, 0xA1380303, + 0x81C, 0xA03A0303, + 0x81C, 0x823C0303, + 0x81C, 0x813E0303, + 0x81C, 0x80400303, + 0x81C, 0x64420303, + 0x81C, 0x63440303, + 0x81C, 0x62460303, + 0x81C, 0x61480303, + 0x81C, 0x604A0303, + 0x81C, 0x414C0303, + 0x81C, 0x404E0303, + 0x81C, 0x22500303, + 0x81C, 0x21520303, + 0x81C, 0x20540303, + 0x81C, 0x03560303, + 0x81C, 0x02580303, + 0x81C, 0x015A0303, + 0x81C, 0x005C0303, + 0x81C, 0x005E0303, + 0x81C, 0x00600303, + 0x81C, 0x00620303, + 0x81C, 0x00640303, + 0x81C, 0x00660303, + 0x81C, 0x00680303, + 0x81C, 0x006A0303, + 0x81C, 0x006C0303, + 0x81C, 0x006E0303, + 0x81C, 0x00700303, + 0x81C, 0x00720303, + 0x81C, 0x00740303, + 0x81C, 0x00760303, + 0x81C, 0x00780303, + 0x81C, 0x007A0303, + 0x81C, 0x007C0303, + 0x81C, 0x007E0303, + 0xA0000000, 0x00000000, + 0x81C, 0xFC000303, + 0x81C, 0xFB020303, + 0x81C, 0xFA040303, + 0x81C, 0xF9060303, + 0x81C, 0xF8080303, + 0x81C, 0xF70A0303, + 0x81C, 0xF60C0303, + 0x81C, 0xF50E0303, + 0x81C, 0xF4100303, + 0x81C, 0xF3120303, + 0x81C, 0xF2140303, + 0x81C, 0xF1160303, + 0x81C, 0xF0180303, + 0x81C, 0xEF1A0303, + 0x81C, 0xEE1C0303, + 0x81C, 0xED1E0303, + 0x81C, 0xEC200303, + 0x81C, 0xEB220303, + 0x81C, 0xEA240303, + 0x81C, 0xE9260303, + 0x81C, 0xE8280303, + 0x81C, 0xE72A0303, + 0x81C, 0xE62C0303, + 0x81C, 0xE52E0303, + 0x81C, 0xE4300303, + 0x81C, 0xE3320303, + 0x81C, 0xE2340303, + 0x81C, 0xC6360303, + 0x81C, 0xC5380303, + 0x81C, 0xC43A0303, + 0x81C, 0xC33C0303, + 0x81C, 0xA63E0303, + 0x81C, 0xA5400303, + 0x81C, 0xA4420303, + 0x81C, 0xA3440303, + 0x81C, 0xA2460303, + 0x81C, 0x84480303, + 0x81C, 0x834A0303, + 0x81C, 0x824C0303, + 0x81C, 0x814E0303, + 0x81C, 0x80500303, + 0x81C, 0x63520303, + 0x81C, 0x62540303, + 0x81C, 0x61560303, + 0x81C, 0x60580303, + 0x81C, 0x235A0303, + 0x81C, 0x225C0303, + 0x81C, 0x215E0303, + 0x81C, 0x20600303, + 0x81C, 0x03620303, + 0x81C, 0x02640303, + 0x81C, 0x01660303, + 0x81C, 0x00680303, + 0x81C, 0x006A0303, + 0x81C, 0x006C0303, + 0x81C, 0x006E0303, + 0x81C, 0x00700303, + 0x81C, 0x00720303, + 0x81C, 0x00740303, + 0x81C, 0x00760303, + 0x81C, 0x00780303, + 0x81C, 0x007A0303, + 0x81C, 0x007C0303, + 0x81C, 0x007E0303, + 0xB0000000, 0x00000000, + 0x80000000, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000403, + 0x81C, 0xFF000403, + 0x81C, 0xFF020403, + 0x81C, 0xFE040403, + 0x81C, 0xFD060403, + 0x81C, 0xFC080403, + 0x81C, 0xFB0A0403, + 0x81C, 0xFA0C0403, + 0x81C, 0xF90E0403, + 0x81C, 0xF8100403, + 0x81C, 0xF7120403, + 0x81C, 0xF6140403, + 0x81C, 0xF5160403, + 0x81C, 0xF4180403, + 0x81C, 0xF31A0403, + 0x81C, 0xF21C0403, + 0x81C, 0xD51E0403, + 0x81C, 0xD4200403, + 0x81C, 0xD3220403, + 0x81C, 0xD2240403, + 0x81C, 0xB6260403, + 0x81C, 0xB5280403, + 0x81C, 0xB42A0403, + 0x81C, 0xB32C0403, + 0x81C, 0xB22E0403, + 0x81C, 0xB1300403, + 0x81C, 0xB0320403, + 0x81C, 0xAF340403, + 0x81C, 0xAE360403, + 0x81C, 0xAD380403, + 0x81C, 0xAC3A0403, + 0x81C, 0xAB3C0403, + 0x81C, 0xAA3E0403, + 0x81C, 0xA9400403, + 0x81C, 0xA8420403, + 0x81C, 0xA7440403, + 0x81C, 0xA6460403, + 0x81C, 0xA5480403, + 0x81C, 0xA44A0403, + 0x81C, 0xA34C0403, + 0x81C, 0x854E0403, + 0x81C, 0x84500403, + 0x81C, 0x83520403, + 0x81C, 0x82540403, + 0x81C, 0x81560403, + 0x81C, 0x80580403, + 0x81C, 0x485A0403, + 0x81C, 0x475C0403, + 0x81C, 0x465E0403, + 0x81C, 0x45600403, + 0x81C, 0x44620403, + 0x81C, 0x0A640403, + 0x81C, 0x09660403, + 0x81C, 0x08680403, + 0x81C, 0x076A0403, + 0x81C, 0x066C0403, + 0x81C, 0x056E0403, + 0x81C, 0x04700403, + 0x81C, 0x03720403, + 0x81C, 0x02740403, + 0x81C, 0x01760403, + 0x81C, 0x00780403, + 0x81C, 0x007A0403, + 0x81C, 0x007C0403, + 0x81C, 0x007E0403, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000403, + 0x81C, 0xF5000403, + 0x81C, 0xF4020403, + 0x81C, 0xF3040403, + 0x81C, 0xF2060403, + 0x81C, 0xF1080403, + 0x81C, 0xF00A0403, + 0x81C, 0xEF0C0403, + 0x81C, 0xEE0E0403, + 0x81C, 0xED100403, + 0x81C, 0xEC120403, + 0x81C, 0xEB140403, + 0x81C, 0xEA160403, + 0x81C, 0xE9180403, + 0x81C, 0xE81A0403, + 0x81C, 0xE71C0403, + 0x81C, 0xE61E0403, + 0x81C, 0xE5200403, + 0x81C, 0xE4220403, + 0x81C, 0xE3240403, + 0x81C, 0xE2260403, + 0x81C, 0xE1280403, + 0x81C, 0xE02A0403, + 0x81C, 0xC32C0403, + 0x81C, 0xC22E0403, + 0x81C, 0xC1300403, + 0x81C, 0xC0320403, + 0x81C, 0xA4340403, + 0x81C, 0xA3360403, + 0x81C, 0xA2380403, + 0x81C, 0xA13A0403, + 0x81C, 0xA03C0403, + 0x81C, 0x823E0403, + 0x81C, 0x81400403, + 0x81C, 0x80420403, + 0x81C, 0x64440403, + 0x81C, 0x63460403, + 0x81C, 0x62480403, + 0x81C, 0x614A0403, + 0x81C, 0x604C0403, + 0x81C, 0x454E0403, + 0x81C, 0x44500403, + 0x81C, 0x43520403, + 0x81C, 0x42540403, + 0x81C, 0x41560403, + 0x81C, 0x40580403, + 0x81C, 0x055A0403, + 0x81C, 0x045C0403, + 0x81C, 0x035E0403, + 0x81C, 0x02600403, + 0x81C, 0x01620403, + 0x81C, 0x00640403, + 0x81C, 0x00660403, + 0x81C, 0x00680403, + 0x81C, 0x006A0403, + 0x81C, 0x006C0403, + 0x81C, 0x006E0403, + 0x81C, 0x00700403, + 0x81C, 0x00720403, + 0x81C, 0x00740403, + 0x81C, 0x00760403, + 0x81C, 0x00780403, + 0x81C, 0x007A0403, + 0x81C, 0x007C0403, + 0x81C, 0x007E0403, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000403, + 0x81C, 0xFF000403, + 0x81C, 0xFF020403, + 0x81C, 0xFE040403, + 0x81C, 0xFD060403, + 0x81C, 0xFC080403, + 0x81C, 0xFB0A0403, + 0x81C, 0xFA0C0403, + 0x81C, 0xF90E0403, + 0x81C, 0xF8100403, + 0x81C, 0xF7120403, + 0x81C, 0xF6140403, + 0x81C, 0xF5160403, + 0x81C, 0xF4180403, + 0x81C, 0xF31A0403, + 0x81C, 0xF21C0403, + 0x81C, 0xD51E0403, + 0x81C, 0xD4200403, + 0x81C, 0xD3220403, + 0x81C, 0xD2240403, + 0x81C, 0xB6260403, + 0x81C, 0xB5280403, + 0x81C, 0xB42A0403, + 0x81C, 0xB32C0403, + 0x81C, 0xB22E0403, + 0x81C, 0xB1300403, + 0x81C, 0xB0320403, + 0x81C, 0xAF340403, + 0x81C, 0xAE360403, + 0x81C, 0xAD380403, + 0x81C, 0xAC3A0403, + 0x81C, 0xAB3C0403, + 0x81C, 0xAA3E0403, + 0x81C, 0xA9400403, + 0x81C, 0xA8420403, + 0x81C, 0xA7440403, + 0x81C, 0xA6460403, + 0x81C, 0xA5480403, + 0x81C, 0xA44A0403, + 0x81C, 0xA34C0403, + 0x81C, 0x854E0403, + 0x81C, 0x84500403, + 0x81C, 0x83520403, + 0x81C, 0x82540403, + 0x81C, 0x81560403, + 0x81C, 0x80580403, + 0x81C, 0x485A0403, + 0x81C, 0x475C0403, + 0x81C, 0x465E0403, + 0x81C, 0x45600403, + 0x81C, 0x44620403, + 0x81C, 0x0A640403, + 0x81C, 0x09660403, + 0x81C, 0x08680403, + 0x81C, 0x076A0403, + 0x81C, 0x066C0403, + 0x81C, 0x056E0403, + 0x81C, 0x04700403, + 0x81C, 0x03720403, + 0x81C, 0x02740403, + 0x81C, 0x01760403, + 0x81C, 0x00780403, + 0x81C, 0x007A0403, + 0x81C, 0x007C0403, + 0x81C, 0x007E0403, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000403, + 0x81C, 0xFF000403, + 0x81C, 0xFF020403, + 0x81C, 0xFE040403, + 0x81C, 0xFD060403, + 0x81C, 0xFC080403, + 0x81C, 0xFB0A0403, + 0x81C, 0xFA0C0403, + 0x81C, 0xF90E0403, + 0x81C, 0xF8100403, + 0x81C, 0xF7120403, + 0x81C, 0xF6140403, + 0x81C, 0xF5160403, + 0x81C, 0xF4180403, + 0x81C, 0xF31A0403, + 0x81C, 0xF21C0403, + 0x81C, 0xD51E0403, + 0x81C, 0xD4200403, + 0x81C, 0xD3220403, + 0x81C, 0xD2240403, + 0x81C, 0xB6260403, + 0x81C, 0xB5280403, + 0x81C, 0xB42A0403, + 0x81C, 0xB32C0403, + 0x81C, 0xB22E0403, + 0x81C, 0xB1300403, + 0x81C, 0xB0320403, + 0x81C, 0xAF340403, + 0x81C, 0xAE360403, + 0x81C, 0xAD380403, + 0x81C, 0xAC3A0403, + 0x81C, 0xAB3C0403, + 0x81C, 0xAA3E0403, + 0x81C, 0xA9400403, + 0x81C, 0xA8420403, + 0x81C, 0xA7440403, + 0x81C, 0xA6460403, + 0x81C, 0xA5480403, + 0x81C, 0xA44A0403, + 0x81C, 0xA34C0403, + 0x81C, 0x854E0403, + 0x81C, 0x84500403, + 0x81C, 0x83520403, + 0x81C, 0x82540403, + 0x81C, 0x81560403, + 0x81C, 0x80580403, + 0x81C, 0x485A0403, + 0x81C, 0x475C0403, + 0x81C, 0x465E0403, + 0x81C, 0x45600403, + 0x81C, 0x44620403, + 0x81C, 0x0A640403, + 0x81C, 0x09660403, + 0x81C, 0x08680403, + 0x81C, 0x076A0403, + 0x81C, 0x066C0403, + 0x81C, 0x056E0403, + 0x81C, 0x04700403, + 0x81C, 0x03720403, + 0x81C, 0x02740403, + 0x81C, 0x01760403, + 0x81C, 0x00780403, + 0x81C, 0x007A0403, + 0x81C, 0x007C0403, + 0x81C, 0x007E0403, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000403, + 0x81C, 0xF6000403, + 0x81C, 0xF5020403, + 0x81C, 0xF4040403, + 0x81C, 0xF3060403, + 0x81C, 0xF2080403, + 0x81C, 0xF10A0403, + 0x81C, 0xF00C0403, + 0x81C, 0xEF0E0403, + 0x81C, 0xD6100403, + 0x81C, 0xD5120403, + 0x81C, 0xD4140403, + 0x81C, 0xD3160403, + 0x81C, 0xD2180403, + 0x81C, 0xD11A0403, + 0x81C, 0xD01C0403, + 0x81C, 0xCF1E0403, + 0x81C, 0x95200403, + 0x81C, 0x94220403, + 0x81C, 0x93240403, + 0x81C, 0x92260403, + 0x81C, 0x91280403, + 0x81C, 0x902A0403, + 0x81C, 0x8F2C0403, + 0x81C, 0x8E2E0403, + 0x81C, 0x8D300403, + 0x81C, 0x8C320403, + 0x81C, 0x8B340403, + 0x81C, 0x8A360403, + 0x81C, 0x89380403, + 0x81C, 0x883A0403, + 0x81C, 0x873C0403, + 0x81C, 0x863E0403, + 0x81C, 0x68400403, + 0x81C, 0x67420403, + 0x81C, 0x66440403, + 0x81C, 0x65460403, + 0x81C, 0x64480403, + 0x81C, 0x634A0403, + 0x81C, 0x484C0403, + 0x81C, 0x474E0403, + 0x81C, 0x46500403, + 0x81C, 0x45520403, + 0x81C, 0x44540403, + 0x81C, 0x27560403, + 0x81C, 0x26580403, + 0x81C, 0x255A0403, + 0x81C, 0x245C0403, + 0x81C, 0x235E0403, + 0x81C, 0x04600403, + 0x81C, 0x03620403, + 0x81C, 0x02640403, + 0x81C, 0x01660403, + 0x81C, 0x00680403, + 0x81C, 0x006A0403, + 0x81C, 0x006C0403, + 0x81C, 0x006E0403, + 0x81C, 0x00700403, + 0x81C, 0x00720403, + 0x81C, 0x00740403, + 0x81C, 0x00760403, + 0x81C, 0x00780403, + 0x81C, 0x007A0403, + 0x81C, 0x007C0403, + 0x81C, 0x007E0403, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000403, + 0x81C, 0xFF000403, + 0x81C, 0xFF020403, + 0x81C, 0xFE040403, + 0x81C, 0xFD060403, + 0x81C, 0xFC080403, + 0x81C, 0xFB0A0403, + 0x81C, 0xFA0C0403, + 0x81C, 0xF90E0403, + 0x81C, 0xF8100403, + 0x81C, 0xF7120403, + 0x81C, 0xF6140403, + 0x81C, 0xF5160403, + 0x81C, 0xF4180403, + 0x81C, 0xF31A0403, + 0x81C, 0xF21C0403, + 0x81C, 0xD51E0403, + 0x81C, 0xD4200403, + 0x81C, 0xD3220403, + 0x81C, 0xD2240403, + 0x81C, 0xB6260403, + 0x81C, 0xB5280403, + 0x81C, 0xB42A0403, + 0x81C, 0xB32C0403, + 0x81C, 0xB22E0403, + 0x81C, 0xB1300403, + 0x81C, 0xB0320403, + 0x81C, 0xAF340403, + 0x81C, 0xAE360403, + 0x81C, 0xAD380403, + 0x81C, 0xAC3A0403, + 0x81C, 0xAB3C0403, + 0x81C, 0xAA3E0403, + 0x81C, 0xA9400403, + 0x81C, 0xA8420403, + 0x81C, 0xA7440403, + 0x81C, 0xA6460403, + 0x81C, 0xA5480403, + 0x81C, 0xA44A0403, + 0x81C, 0xA34C0403, + 0x81C, 0x854E0403, + 0x81C, 0x84500403, + 0x81C, 0x83520403, + 0x81C, 0x82540403, + 0x81C, 0x81560403, + 0x81C, 0x80580403, + 0x81C, 0x485A0403, + 0x81C, 0x475C0403, + 0x81C, 0x465E0403, + 0x81C, 0x45600403, + 0x81C, 0x44620403, + 0x81C, 0x0A640403, + 0x81C, 0x09660403, + 0x81C, 0x08680403, + 0x81C, 0x076A0403, + 0x81C, 0x066C0403, + 0x81C, 0x056E0403, + 0x81C, 0x04700403, + 0x81C, 0x03720403, + 0x81C, 0x02740403, + 0x81C, 0x01760403, + 0x81C, 0x00780403, + 0x81C, 0x007A0403, + 0x81C, 0x007C0403, + 0x81C, 0x007E0403, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000403, + 0x81C, 0xF5000403, + 0x81C, 0xF4020403, + 0x81C, 0xF3040403, + 0x81C, 0xF2060403, + 0x81C, 0xF1080403, + 0x81C, 0xF00A0403, + 0x81C, 0xEF0C0403, + 0x81C, 0xEE0E0403, + 0x81C, 0xED100403, + 0x81C, 0xEC120403, + 0x81C, 0xEB140403, + 0x81C, 0xEA160403, + 0x81C, 0xE9180403, + 0x81C, 0xE81A0403, + 0x81C, 0xE71C0403, + 0x81C, 0xE61E0403, + 0x81C, 0xE5200403, + 0x81C, 0xE4220403, + 0x81C, 0xE3240403, + 0x81C, 0xE2260403, + 0x81C, 0xE1280403, + 0x81C, 0xE02A0403, + 0x81C, 0xC32C0403, + 0x81C, 0xC22E0403, + 0x81C, 0xC1300403, + 0x81C, 0xC0320403, + 0x81C, 0xA4340403, + 0x81C, 0xA3360403, + 0x81C, 0xA2380403, + 0x81C, 0xA13A0403, + 0x81C, 0xA03C0403, + 0x81C, 0x823E0403, + 0x81C, 0x81400403, + 0x81C, 0x80420403, + 0x81C, 0x64440403, + 0x81C, 0x63460403, + 0x81C, 0x62480403, + 0x81C, 0x614A0403, + 0x81C, 0x604C0403, + 0x81C, 0x454E0403, + 0x81C, 0x44500403, + 0x81C, 0x43520403, + 0x81C, 0x42540403, + 0x81C, 0x41560403, + 0x81C, 0x40580403, + 0x81C, 0x055A0403, + 0x81C, 0x045C0403, + 0x81C, 0x035E0403, + 0x81C, 0x02600403, + 0x81C, 0x01620403, + 0x81C, 0x00640403, + 0x81C, 0x00660403, + 0x81C, 0x00680403, + 0x81C, 0x006A0403, + 0x81C, 0x006C0403, + 0x81C, 0x006E0403, + 0x81C, 0x00700403, + 0x81C, 0x00720403, + 0x81C, 0x00740403, + 0x81C, 0x00760403, + 0x81C, 0x00780403, + 0x81C, 0x007A0403, + 0x81C, 0x007C0403, + 0x81C, 0x007E0403, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000403, + 0x81C, 0xF5000403, + 0x81C, 0xF4020403, + 0x81C, 0xF3040403, + 0x81C, 0xF2060403, + 0x81C, 0xF1080403, + 0x81C, 0xF00A0403, + 0x81C, 0xEF0C0403, + 0x81C, 0xEE0E0403, + 0x81C, 0xED100403, + 0x81C, 0xEC120403, + 0x81C, 0xEB140403, + 0x81C, 0xEA160403, + 0x81C, 0xE9180403, + 0x81C, 0xE81A0403, + 0x81C, 0xE71C0403, + 0x81C, 0xE61E0403, + 0x81C, 0xE5200403, + 0x81C, 0xE4220403, + 0x81C, 0xE3240403, + 0x81C, 0xE2260403, + 0x81C, 0xE1280403, + 0x81C, 0xE02A0403, + 0x81C, 0xC32C0403, + 0x81C, 0xC22E0403, + 0x81C, 0xC1300403, + 0x81C, 0xC0320403, + 0x81C, 0xA4340403, + 0x81C, 0xA3360403, + 0x81C, 0xA2380403, + 0x81C, 0xA13A0403, + 0x81C, 0xA03C0403, + 0x81C, 0x823E0403, + 0x81C, 0x81400403, + 0x81C, 0x80420403, + 0x81C, 0x64440403, + 0x81C, 0x63460403, + 0x81C, 0x62480403, + 0x81C, 0x614A0403, + 0x81C, 0x604C0403, + 0x81C, 0x454E0403, + 0x81C, 0x44500403, + 0x81C, 0x43520403, + 0x81C, 0x42540403, + 0x81C, 0x41560403, + 0x81C, 0x40580403, + 0x81C, 0x055A0403, + 0x81C, 0x045C0403, + 0x81C, 0x035E0403, + 0x81C, 0x02600403, + 0x81C, 0x01620403, + 0x81C, 0x00640403, + 0x81C, 0x00660403, + 0x81C, 0x00680403, + 0x81C, 0x006A0403, + 0x81C, 0x006C0403, + 0x81C, 0x006E0403, + 0x81C, 0x00700403, + 0x81C, 0x00720403, + 0x81C, 0x00740403, + 0x81C, 0x00760403, + 0x81C, 0x00780403, + 0x81C, 0x007A0403, + 0x81C, 0x007C0403, + 0x81C, 0x007E0403, + 0x90000008, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000403, + 0x81C, 0xFF000403, + 0x81C, 0xFF020403, + 0x81C, 0xFE040403, + 0x81C, 0xFD060403, + 0x81C, 0xFC080403, + 0x81C, 0xFB0A0403, + 0x81C, 0xFA0C0403, + 0x81C, 0xF90E0403, + 0x81C, 0xF8100403, + 0x81C, 0xF7120403, + 0x81C, 0xF6140403, + 0x81C, 0xF5160403, + 0x81C, 0xF4180403, + 0x81C, 0xF31A0403, + 0x81C, 0xF21C0403, + 0x81C, 0xD51E0403, + 0x81C, 0xD4200403, + 0x81C, 0xD3220403, + 0x81C, 0xD2240403, + 0x81C, 0xB6260403, + 0x81C, 0xB5280403, + 0x81C, 0xB42A0403, + 0x81C, 0xB32C0403, + 0x81C, 0xB22E0403, + 0x81C, 0xB1300403, + 0x81C, 0xB0320403, + 0x81C, 0xAF340403, + 0x81C, 0xAE360403, + 0x81C, 0xAD380403, + 0x81C, 0xAC3A0403, + 0x81C, 0xAB3C0403, + 0x81C, 0xAA3E0403, + 0x81C, 0xA9400403, + 0x81C, 0xA8420403, + 0x81C, 0xA7440403, + 0x81C, 0xA6460403, + 0x81C, 0xA5480403, + 0x81C, 0xA44A0403, + 0x81C, 0xA34C0403, + 0x81C, 0x854E0403, + 0x81C, 0x84500403, + 0x81C, 0x83520403, + 0x81C, 0x82540403, + 0x81C, 0x81560403, + 0x81C, 0x80580403, + 0x81C, 0x485A0403, + 0x81C, 0x475C0403, + 0x81C, 0x465E0403, + 0x81C, 0x45600403, + 0x81C, 0x44620403, + 0x81C, 0x0A640403, + 0x81C, 0x09660403, + 0x81C, 0x08680403, + 0x81C, 0x076A0403, + 0x81C, 0x066C0403, + 0x81C, 0x056E0403, + 0x81C, 0x04700403, + 0x81C, 0x03720403, + 0x81C, 0x02740403, + 0x81C, 0x01760403, + 0x81C, 0x00780403, + 0x81C, 0x007A0403, + 0x81C, 0x007C0403, + 0x81C, 0x007E0403, + 0x90000009, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000403, + 0x81C, 0xF5000403, + 0x81C, 0xF4020403, + 0x81C, 0xF3040403, + 0x81C, 0xF2060403, + 0x81C, 0xF1080403, + 0x81C, 0xF00A0403, + 0x81C, 0xEF0C0403, + 0x81C, 0xEE0E0403, + 0x81C, 0xED100403, + 0x81C, 0xEC120403, + 0x81C, 0xEB140403, + 0x81C, 0xEA160403, + 0x81C, 0xE9180403, + 0x81C, 0xE81A0403, + 0x81C, 0xE71C0403, + 0x81C, 0xE61E0403, + 0x81C, 0xE5200403, + 0x81C, 0xE4220403, + 0x81C, 0xE3240403, + 0x81C, 0xE2260403, + 0x81C, 0xE1280403, + 0x81C, 0xE02A0403, + 0x81C, 0xC32C0403, + 0x81C, 0xC22E0403, + 0x81C, 0xC1300403, + 0x81C, 0xC0320403, + 0x81C, 0xA4340403, + 0x81C, 0xA3360403, + 0x81C, 0xA2380403, + 0x81C, 0xA13A0403, + 0x81C, 0xA03C0403, + 0x81C, 0x823E0403, + 0x81C, 0x81400403, + 0x81C, 0x80420403, + 0x81C, 0x64440403, + 0x81C, 0x63460403, + 0x81C, 0x62480403, + 0x81C, 0x614A0403, + 0x81C, 0x604C0403, + 0x81C, 0x454E0403, + 0x81C, 0x44500403, + 0x81C, 0x43520403, + 0x81C, 0x42540403, + 0x81C, 0x41560403, + 0x81C, 0x40580403, + 0x81C, 0x055A0403, + 0x81C, 0x045C0403, + 0x81C, 0x035E0403, + 0x81C, 0x02600403, + 0x81C, 0x01620403, + 0x81C, 0x00640403, + 0x81C, 0x00660403, + 0x81C, 0x00680403, + 0x81C, 0x006A0403, + 0x81C, 0x006C0403, + 0x81C, 0x006E0403, + 0x81C, 0x00700403, + 0x81C, 0x00720403, + 0x81C, 0x00740403, + 0x81C, 0x00760403, + 0x81C, 0x00780403, + 0x81C, 0x007A0403, + 0x81C, 0x007C0403, + 0x81C, 0x007E0403, + 0x9000000a, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000403, + 0x81C, 0xFF000403, + 0x81C, 0xFF020403, + 0x81C, 0xFE040403, + 0x81C, 0xFD060403, + 0x81C, 0xFC080403, + 0x81C, 0xFB0A0403, + 0x81C, 0xFA0C0403, + 0x81C, 0xF90E0403, + 0x81C, 0xF8100403, + 0x81C, 0xF7120403, + 0x81C, 0xF6140403, + 0x81C, 0xF5160403, + 0x81C, 0xF4180403, + 0x81C, 0xF31A0403, + 0x81C, 0xF21C0403, + 0x81C, 0xD51E0403, + 0x81C, 0xD4200403, + 0x81C, 0xD3220403, + 0x81C, 0xD2240403, + 0x81C, 0xB6260403, + 0x81C, 0xB5280403, + 0x81C, 0xB42A0403, + 0x81C, 0xB32C0403, + 0x81C, 0xB22E0403, + 0x81C, 0xB1300403, + 0x81C, 0xB0320403, + 0x81C, 0xAF340403, + 0x81C, 0xAE360403, + 0x81C, 0xAD380403, + 0x81C, 0xAC3A0403, + 0x81C, 0xAB3C0403, + 0x81C, 0xAA3E0403, + 0x81C, 0xA9400403, + 0x81C, 0xA8420403, + 0x81C, 0xA7440403, + 0x81C, 0xA6460403, + 0x81C, 0xA5480403, + 0x81C, 0xA44A0403, + 0x81C, 0xA34C0403, + 0x81C, 0x854E0403, + 0x81C, 0x84500403, + 0x81C, 0x83520403, + 0x81C, 0x82540403, + 0x81C, 0x81560403, + 0x81C, 0x80580403, + 0x81C, 0x485A0403, + 0x81C, 0x475C0403, + 0x81C, 0x465E0403, + 0x81C, 0x45600403, + 0x81C, 0x44620403, + 0x81C, 0x0A640403, + 0x81C, 0x09660403, + 0x81C, 0x08680403, + 0x81C, 0x076A0403, + 0x81C, 0x066C0403, + 0x81C, 0x056E0403, + 0x81C, 0x04700403, + 0x81C, 0x03720403, + 0x81C, 0x02740403, + 0x81C, 0x01760403, + 0x81C, 0x00780403, + 0x81C, 0x007A0403, + 0x81C, 0x007C0403, + 0x81C, 0x007E0403, + 0x9000000b, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000403, + 0x81C, 0xF5000403, + 0x81C, 0xF4020403, + 0x81C, 0xF3040403, + 0x81C, 0xF2060403, + 0x81C, 0xF1080403, + 0x81C, 0xF00A0403, + 0x81C, 0xEF0C0403, + 0x81C, 0xEE0E0403, + 0x81C, 0xED100403, + 0x81C, 0xEC120403, + 0x81C, 0xEB140403, + 0x81C, 0xEA160403, + 0x81C, 0xE9180403, + 0x81C, 0xE81A0403, + 0x81C, 0xE71C0403, + 0x81C, 0xE61E0403, + 0x81C, 0xE5200403, + 0x81C, 0xE4220403, + 0x81C, 0xE3240403, + 0x81C, 0xE2260403, + 0x81C, 0xE1280403, + 0x81C, 0xE02A0403, + 0x81C, 0xC32C0403, + 0x81C, 0xC22E0403, + 0x81C, 0xC1300403, + 0x81C, 0xC0320403, + 0x81C, 0xA4340403, + 0x81C, 0xA3360403, + 0x81C, 0xA2380403, + 0x81C, 0xA13A0403, + 0x81C, 0xA03C0403, + 0x81C, 0x823E0403, + 0x81C, 0x81400403, + 0x81C, 0x80420403, + 0x81C, 0x64440403, + 0x81C, 0x63460403, + 0x81C, 0x62480403, + 0x81C, 0x614A0403, + 0x81C, 0x604C0403, + 0x81C, 0x454E0403, + 0x81C, 0x44500403, + 0x81C, 0x43520403, + 0x81C, 0x42540403, + 0x81C, 0x41560403, + 0x81C, 0x40580403, + 0x81C, 0x055A0403, + 0x81C, 0x045C0403, + 0x81C, 0x035E0403, + 0x81C, 0x02600403, + 0x81C, 0x01620403, + 0x81C, 0x00640403, + 0x81C, 0x00660403, + 0x81C, 0x00680403, + 0x81C, 0x006A0403, + 0x81C, 0x006C0403, + 0x81C, 0x006E0403, + 0x81C, 0x00700403, + 0x81C, 0x00720403, + 0x81C, 0x00740403, + 0x81C, 0x00760403, + 0x81C, 0x00780403, + 0x81C, 0x007A0403, + 0x81C, 0x007C0403, + 0x81C, 0x007E0403, + 0x9000000c, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000403, + 0x81C, 0xFF000403, + 0x81C, 0xFF020403, + 0x81C, 0xFE040403, + 0x81C, 0xFD060403, + 0x81C, 0xFC080403, + 0x81C, 0xFB0A0403, + 0x81C, 0xFA0C0403, + 0x81C, 0xF90E0403, + 0x81C, 0xF8100403, + 0x81C, 0xF7120403, + 0x81C, 0xF6140403, + 0x81C, 0xF5160403, + 0x81C, 0xF4180403, + 0x81C, 0xF31A0403, + 0x81C, 0xF21C0403, + 0x81C, 0xD51E0403, + 0x81C, 0xD4200403, + 0x81C, 0xD3220403, + 0x81C, 0xD2240403, + 0x81C, 0xB6260403, + 0x81C, 0xB5280403, + 0x81C, 0xB42A0403, + 0x81C, 0xB32C0403, + 0x81C, 0xB22E0403, + 0x81C, 0xB1300403, + 0x81C, 0xB0320403, + 0x81C, 0xAF340403, + 0x81C, 0xAE360403, + 0x81C, 0xAD380403, + 0x81C, 0xAC3A0403, + 0x81C, 0xAB3C0403, + 0x81C, 0xAA3E0403, + 0x81C, 0xA9400403, + 0x81C, 0xA8420403, + 0x81C, 0xA7440403, + 0x81C, 0xA6460403, + 0x81C, 0xA5480403, + 0x81C, 0xA44A0403, + 0x81C, 0xA34C0403, + 0x81C, 0x854E0403, + 0x81C, 0x84500403, + 0x81C, 0x83520403, + 0x81C, 0x82540403, + 0x81C, 0x81560403, + 0x81C, 0x80580403, + 0x81C, 0x485A0403, + 0x81C, 0x475C0403, + 0x81C, 0x465E0403, + 0x81C, 0x45600403, + 0x81C, 0x44620403, + 0x81C, 0x0A640403, + 0x81C, 0x09660403, + 0x81C, 0x08680403, + 0x81C, 0x076A0403, + 0x81C, 0x066C0403, + 0x81C, 0x056E0403, + 0x81C, 0x04700403, + 0x81C, 0x03720403, + 0x81C, 0x02740403, + 0x81C, 0x01760403, + 0x81C, 0x00780403, + 0x81C, 0x007A0403, + 0x81C, 0x007C0403, + 0x81C, 0x007E0403, + 0x9000000d, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000403, + 0x81C, 0xFF000403, + 0x81C, 0xFF020403, + 0x81C, 0xFE040403, + 0x81C, 0xFD060403, + 0x81C, 0xFC080403, + 0x81C, 0xFB0A0403, + 0x81C, 0xFA0C0403, + 0x81C, 0xF90E0403, + 0x81C, 0xF8100403, + 0x81C, 0xF7120403, + 0x81C, 0xF6140403, + 0x81C, 0xF5160403, + 0x81C, 0xF4180403, + 0x81C, 0xF31A0403, + 0x81C, 0xF21C0403, + 0x81C, 0xD51E0403, + 0x81C, 0xD4200403, + 0x81C, 0xD3220403, + 0x81C, 0xD2240403, + 0x81C, 0xB6260403, + 0x81C, 0xB5280403, + 0x81C, 0xB42A0403, + 0x81C, 0xB32C0403, + 0x81C, 0xB22E0403, + 0x81C, 0xB1300403, + 0x81C, 0xB0320403, + 0x81C, 0xAF340403, + 0x81C, 0xAE360403, + 0x81C, 0xAD380403, + 0x81C, 0xAC3A0403, + 0x81C, 0xAB3C0403, + 0x81C, 0xAA3E0403, + 0x81C, 0xA9400403, + 0x81C, 0xA8420403, + 0x81C, 0xA7440403, + 0x81C, 0xA6460403, + 0x81C, 0xA5480403, + 0x81C, 0xA44A0403, + 0x81C, 0xA34C0403, + 0x81C, 0x854E0403, + 0x81C, 0x84500403, + 0x81C, 0x83520403, + 0x81C, 0x82540403, + 0x81C, 0x81560403, + 0x81C, 0x80580403, + 0x81C, 0x485A0403, + 0x81C, 0x475C0403, + 0x81C, 0x465E0403, + 0x81C, 0x45600403, + 0x81C, 0x44620403, + 0x81C, 0x0A640403, + 0x81C, 0x09660403, + 0x81C, 0x08680403, + 0x81C, 0x076A0403, + 0x81C, 0x066C0403, + 0x81C, 0x056E0403, + 0x81C, 0x04700403, + 0x81C, 0x03720403, + 0x81C, 0x02740403, + 0x81C, 0x01760403, + 0x81C, 0x00780403, + 0x81C, 0x007A0403, + 0x81C, 0x007C0403, + 0x81C, 0x007E0403, + 0x9000000e, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000403, + 0x81C, 0xFF000403, + 0x81C, 0xFF020403, + 0x81C, 0xFE040403, + 0x81C, 0xFD060403, + 0x81C, 0xFC080403, + 0x81C, 0xFB0A0403, + 0x81C, 0xFA0C0403, + 0x81C, 0xF90E0403, + 0x81C, 0xF8100403, + 0x81C, 0xF7120403, + 0x81C, 0xF6140403, + 0x81C, 0xF5160403, + 0x81C, 0xF4180403, + 0x81C, 0xF31A0403, + 0x81C, 0xF21C0403, + 0x81C, 0xD51E0403, + 0x81C, 0xD4200403, + 0x81C, 0xD3220403, + 0x81C, 0xD2240403, + 0x81C, 0xB6260403, + 0x81C, 0xB5280403, + 0x81C, 0xB42A0403, + 0x81C, 0xB32C0403, + 0x81C, 0xB22E0403, + 0x81C, 0xB1300403, + 0x81C, 0xB0320403, + 0x81C, 0xAF340403, + 0x81C, 0xAE360403, + 0x81C, 0xAD380403, + 0x81C, 0xAC3A0403, + 0x81C, 0xAB3C0403, + 0x81C, 0xAA3E0403, + 0x81C, 0xA9400403, + 0x81C, 0xA8420403, + 0x81C, 0xA7440403, + 0x81C, 0xA6460403, + 0x81C, 0xA5480403, + 0x81C, 0xA44A0403, + 0x81C, 0xA34C0403, + 0x81C, 0x854E0403, + 0x81C, 0x84500403, + 0x81C, 0x83520403, + 0x81C, 0x82540403, + 0x81C, 0x81560403, + 0x81C, 0x80580403, + 0x81C, 0x485A0403, + 0x81C, 0x475C0403, + 0x81C, 0x465E0403, + 0x81C, 0x45600403, + 0x81C, 0x44620403, + 0x81C, 0x0A640403, + 0x81C, 0x09660403, + 0x81C, 0x08680403, + 0x81C, 0x076A0403, + 0x81C, 0x066C0403, + 0x81C, 0x056E0403, + 0x81C, 0x04700403, + 0x81C, 0x03720403, + 0x81C, 0x02740403, + 0x81C, 0x01760403, + 0x81C, 0x00780403, + 0x81C, 0x007A0403, + 0x81C, 0x007C0403, + 0x81C, 0x007E0403, + 0x9000000f, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000403, + 0x81C, 0xFF000403, + 0x81C, 0xFF020403, + 0x81C, 0xFE040403, + 0x81C, 0xFD060403, + 0x81C, 0xFC080403, + 0x81C, 0xFB0A0403, + 0x81C, 0xFA0C0403, + 0x81C, 0xF90E0403, + 0x81C, 0xF8100403, + 0x81C, 0xF7120403, + 0x81C, 0xF6140403, + 0x81C, 0xF5160403, + 0x81C, 0xF4180403, + 0x81C, 0xF31A0403, + 0x81C, 0xF21C0403, + 0x81C, 0xD51E0403, + 0x81C, 0xD4200403, + 0x81C, 0xD3220403, + 0x81C, 0xD2240403, + 0x81C, 0xB6260403, + 0x81C, 0xB5280403, + 0x81C, 0xB42A0403, + 0x81C, 0xB32C0403, + 0x81C, 0xB22E0403, + 0x81C, 0xB1300403, + 0x81C, 0xB0320403, + 0x81C, 0xAF340403, + 0x81C, 0xAE360403, + 0x81C, 0xAD380403, + 0x81C, 0xAC3A0403, + 0x81C, 0xAB3C0403, + 0x81C, 0xAA3E0403, + 0x81C, 0xA9400403, + 0x81C, 0xA8420403, + 0x81C, 0xA7440403, + 0x81C, 0xA6460403, + 0x81C, 0xA5480403, + 0x81C, 0xA44A0403, + 0x81C, 0xA34C0403, + 0x81C, 0x854E0403, + 0x81C, 0x84500403, + 0x81C, 0x83520403, + 0x81C, 0x82540403, + 0x81C, 0x81560403, + 0x81C, 0x80580403, + 0x81C, 0x485A0403, + 0x81C, 0x475C0403, + 0x81C, 0x465E0403, + 0x81C, 0x45600403, + 0x81C, 0x44620403, + 0x81C, 0x0A640403, + 0x81C, 0x09660403, + 0x81C, 0x08680403, + 0x81C, 0x076A0403, + 0x81C, 0x066C0403, + 0x81C, 0x056E0403, + 0x81C, 0x04700403, + 0x81C, 0x03720403, + 0x81C, 0x02740403, + 0x81C, 0x01760403, + 0x81C, 0x00780403, + 0x81C, 0x007A0403, + 0x81C, 0x007C0403, + 0x81C, 0x007E0403, + 0x90000010, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000403, + 0x81C, 0xFF000403, + 0x81C, 0xFF020403, + 0x81C, 0xFE040403, + 0x81C, 0xFD060403, + 0x81C, 0xFC080403, + 0x81C, 0xFB0A0403, + 0x81C, 0xFA0C0403, + 0x81C, 0xF90E0403, + 0x81C, 0xF8100403, + 0x81C, 0xF7120403, + 0x81C, 0xF6140403, + 0x81C, 0xF5160403, + 0x81C, 0xF4180403, + 0x81C, 0xF31A0403, + 0x81C, 0xF21C0403, + 0x81C, 0xD51E0403, + 0x81C, 0xD4200403, + 0x81C, 0xD3220403, + 0x81C, 0xD2240403, + 0x81C, 0xB6260403, + 0x81C, 0xB5280403, + 0x81C, 0xB42A0403, + 0x81C, 0xB32C0403, + 0x81C, 0xB22E0403, + 0x81C, 0xB1300403, + 0x81C, 0xB0320403, + 0x81C, 0xAF340403, + 0x81C, 0xAE360403, + 0x81C, 0xAD380403, + 0x81C, 0xAC3A0403, + 0x81C, 0xAB3C0403, + 0x81C, 0xAA3E0403, + 0x81C, 0xA9400403, + 0x81C, 0xA8420403, + 0x81C, 0xA7440403, + 0x81C, 0xA6460403, + 0x81C, 0xA5480403, + 0x81C, 0xA44A0403, + 0x81C, 0xA34C0403, + 0x81C, 0x854E0403, + 0x81C, 0x84500403, + 0x81C, 0x83520403, + 0x81C, 0x82540403, + 0x81C, 0x81560403, + 0x81C, 0x80580403, + 0x81C, 0x485A0403, + 0x81C, 0x475C0403, + 0x81C, 0x465E0403, + 0x81C, 0x45600403, + 0x81C, 0x44620403, + 0x81C, 0x0A640403, + 0x81C, 0x09660403, + 0x81C, 0x08680403, + 0x81C, 0x076A0403, + 0x81C, 0x066C0403, + 0x81C, 0x056E0403, + 0x81C, 0x04700403, + 0x81C, 0x03720403, + 0x81C, 0x02740403, + 0x81C, 0x01760403, + 0x81C, 0x00780403, + 0x81C, 0x007A0403, + 0x81C, 0x007C0403, + 0x81C, 0x007E0403, + 0x90000012, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFF000403, + 0x81C, 0xF6000403, + 0x81C, 0xF5020403, + 0x81C, 0xF4040403, + 0x81C, 0xF3060403, + 0x81C, 0xF2080403, + 0x81C, 0xF10A0403, + 0x81C, 0xF00C0403, + 0x81C, 0xEF0E0403, + 0x81C, 0xD6100403, + 0x81C, 0xD5120403, + 0x81C, 0xD4140403, + 0x81C, 0xD3160403, + 0x81C, 0xD2180403, + 0x81C, 0xD11A0403, + 0x81C, 0xD01C0403, + 0x81C, 0xCF1E0403, + 0x81C, 0x95200403, + 0x81C, 0x94220403, + 0x81C, 0x93240403, + 0x81C, 0x92260403, + 0x81C, 0x91280403, + 0x81C, 0x902A0403, + 0x81C, 0x8F2C0403, + 0x81C, 0x8E2E0403, + 0x81C, 0x8D300403, + 0x81C, 0x8C320403, + 0x81C, 0x8B340403, + 0x81C, 0x8A360403, + 0x81C, 0x89380403, + 0x81C, 0x883A0403, + 0x81C, 0x873C0403, + 0x81C, 0x863E0403, + 0x81C, 0x68400403, + 0x81C, 0x67420403, + 0x81C, 0x66440403, + 0x81C, 0x65460403, + 0x81C, 0x64480403, + 0x81C, 0x634A0403, + 0x81C, 0x484C0403, + 0x81C, 0x474E0403, + 0x81C, 0x46500403, + 0x81C, 0x45520403, + 0x81C, 0x44540403, + 0x81C, 0x27560403, + 0x81C, 0x26580403, + 0x81C, 0x255A0403, + 0x81C, 0x245C0403, + 0x81C, 0x235E0403, + 0x81C, 0x04600403, + 0x81C, 0x03620403, + 0x81C, 0x02640403, + 0x81C, 0x01660403, + 0x81C, 0x00680403, + 0x81C, 0x006A0403, + 0x81C, 0x006C0403, + 0x81C, 0x006E0403, + 0x81C, 0x00700403, + 0x81C, 0x00720403, + 0x81C, 0x00740403, + 0x81C, 0x00760403, + 0x81C, 0x00780403, + 0x81C, 0x007A0403, + 0x81C, 0x007C0403, + 0x81C, 0x007E0403, + 0xA0000000, 0x00000000, + 0x81C, 0xFF000403, + 0x81C, 0xFF000403, + 0x81C, 0xFF020403, + 0x81C, 0xFE040403, + 0x81C, 0xFD060403, + 0x81C, 0xFC080403, + 0x81C, 0xFB0A0403, + 0x81C, 0xFA0C0403, + 0x81C, 0xF90E0403, + 0x81C, 0xF8100403, + 0x81C, 0xF7120403, + 0x81C, 0xF6140403, + 0x81C, 0xF5160403, + 0x81C, 0xF4180403, + 0x81C, 0xF31A0403, + 0x81C, 0xF21C0403, + 0x81C, 0xD51E0403, + 0x81C, 0xD4200403, + 0x81C, 0xD3220403, + 0x81C, 0xD2240403, + 0x81C, 0xB6260403, + 0x81C, 0xB5280403, + 0x81C, 0xB42A0403, + 0x81C, 0xB32C0403, + 0x81C, 0xB22E0403, + 0x81C, 0xB1300403, + 0x81C, 0xB0320403, + 0x81C, 0xAF340403, + 0x81C, 0xAE360403, + 0x81C, 0xAD380403, + 0x81C, 0xAC3A0403, + 0x81C, 0xAB3C0403, + 0x81C, 0xAA3E0403, + 0x81C, 0xA9400403, + 0x81C, 0xA8420403, + 0x81C, 0xA7440403, + 0x81C, 0xA6460403, + 0x81C, 0xA5480403, + 0x81C, 0xA44A0403, + 0x81C, 0xA34C0403, + 0x81C, 0x854E0403, + 0x81C, 0x84500403, + 0x81C, 0x83520403, + 0x81C, 0x82540403, + 0x81C, 0x81560403, + 0x81C, 0x80580403, + 0x81C, 0x485A0403, + 0x81C, 0x475C0403, + 0x81C, 0x465E0403, + 0x81C, 0x45600403, + 0x81C, 0x44620403, + 0x81C, 0x0A640403, + 0x81C, 0x09660403, + 0x81C, 0x08680403, + 0x81C, 0x076A0403, + 0x81C, 0x066C0403, + 0x81C, 0x056E0403, + 0x81C, 0x04700403, + 0x81C, 0x03720403, + 0x81C, 0x02740403, + 0x81C, 0x01760403, + 0x81C, 0x00780403, + 0x81C, 0x007A0403, + 0x81C, 0x007C0403, + 0x81C, 0x007E0403, + 0xB0000000, 0x00000000, + 0x80000000, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFD000503, + 0x81C, 0xFC020503, + 0x81C, 0xFB040503, + 0x81C, 0xFA060503, + 0x81C, 0xF9080503, + 0x81C, 0xF80A0503, + 0x81C, 0xF70C0503, + 0x81C, 0xF60E0503, + 0x81C, 0xF5100503, + 0x81C, 0xF4120503, + 0x81C, 0xF3140503, + 0x81C, 0xF2160503, + 0x81C, 0xF1180503, + 0x81C, 0xF01A0503, + 0x81C, 0xEE1C0503, + 0x81C, 0xED1E0503, + 0x81C, 0xEC200503, + 0x81C, 0xEB220503, + 0x81C, 0xEA240503, + 0x81C, 0xE9260503, + 0x81C, 0xE8280503, + 0x81C, 0xE72A0503, + 0x81C, 0xE62C0503, + 0x81C, 0xE52E0503, + 0x81C, 0xE4300503, + 0x81C, 0xE3320503, + 0x81C, 0xE2340503, + 0x81C, 0xC5360503, + 0x81C, 0xC4380503, + 0x81C, 0xC33A0503, + 0x81C, 0xC23C0503, + 0x81C, 0xA53E0503, + 0x81C, 0xA4400503, + 0x81C, 0xA3420503, + 0x81C, 0xA2440503, + 0x81C, 0xA1460503, + 0x81C, 0x83480503, + 0x81C, 0x824A0503, + 0x81C, 0x814C0503, + 0x81C, 0x804E0503, + 0x81C, 0x63500503, + 0x81C, 0x62520503, + 0x81C, 0x61540503, + 0x81C, 0x43560503, + 0x81C, 0x42580503, + 0x81C, 0x415A0503, + 0x81C, 0x405C0503, + 0x81C, 0x225E0503, + 0x81C, 0x21600503, + 0x81C, 0x20620503, + 0x81C, 0x03640503, + 0x81C, 0x02660503, + 0x81C, 0x01680503, + 0x81C, 0x006A0503, + 0x81C, 0x006C0503, + 0x81C, 0x006E0503, + 0x81C, 0x00700503, + 0x81C, 0x00720503, + 0x81C, 0x00740503, + 0x81C, 0x00760503, + 0x81C, 0x00780503, + 0x81C, 0x007A0503, + 0x81C, 0x007C0503, + 0x81C, 0x007E0503, + 0x81C, 0x007E0503, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xBE000503, + 0x81C, 0xBD020503, + 0x81C, 0xBC040503, + 0x81C, 0xBB060503, + 0x81C, 0xBA080503, + 0x81C, 0xB90A0503, + 0x81C, 0xB80C0503, + 0x81C, 0xB70E0503, + 0x81C, 0xB6100503, + 0x81C, 0xB5120503, + 0x81C, 0xB4140503, + 0x81C, 0xB3160503, + 0x81C, 0xB2180503, + 0x81C, 0xB11A0503, + 0x81C, 0xB01C0503, + 0x81C, 0xAF1E0503, + 0x81C, 0xAE200503, + 0x81C, 0xAD220503, + 0x81C, 0xAC240503, + 0x81C, 0xAB260503, + 0x81C, 0x8D280503, + 0x81C, 0x8C2A0503, + 0x81C, 0x8B2C0503, + 0x81C, 0x8A2E0503, + 0x81C, 0x89300503, + 0x81C, 0x88320503, + 0x81C, 0x6A340503, + 0x81C, 0x69360503, + 0x81C, 0x68380503, + 0x81C, 0x673A0503, + 0x81C, 0x663C0503, + 0x81C, 0x653E0503, + 0x81C, 0x64400503, + 0x81C, 0x63420503, + 0x81C, 0x62440503, + 0x81C, 0x61460503, + 0x81C, 0x60480503, + 0x81C, 0x424A0503, + 0x81C, 0x414C0503, + 0x81C, 0x404E0503, + 0x81C, 0x06500503, + 0x81C, 0x05520503, + 0x81C, 0x04540503, + 0x81C, 0x03560503, + 0x81C, 0x02580503, + 0x81C, 0x015A0503, + 0x81C, 0x005C0503, + 0x81C, 0x005E0503, + 0x81C, 0x00600503, + 0x81C, 0x00620503, + 0x81C, 0x00640503, + 0x81C, 0x00660503, + 0x81C, 0x00680503, + 0x81C, 0x006A0503, + 0x81C, 0x006C0503, + 0x81C, 0x006E0503, + 0x81C, 0x00700503, + 0x81C, 0x00720503, + 0x81C, 0x00740503, + 0x81C, 0x00760503, + 0x81C, 0x00780503, + 0x81C, 0x007A0503, + 0x81C, 0x007C0503, + 0x81C, 0x007E0503, + 0x81C, 0x007C0503, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF8000503, + 0x81C, 0xF7020503, + 0x81C, 0xF6040503, + 0x81C, 0xF5060503, + 0x81C, 0xF4080503, + 0x81C, 0xF30A0503, + 0x81C, 0xF20C0503, + 0x81C, 0xF10E0503, + 0x81C, 0xF0100503, + 0x81C, 0xEF120503, + 0x81C, 0xEE140503, + 0x81C, 0xED160503, + 0x81C, 0xEC180503, + 0x81C, 0xEB1A0503, + 0x81C, 0xEA1C0503, + 0x81C, 0xE91E0503, + 0x81C, 0xE8200503, + 0x81C, 0xE7220503, + 0x81C, 0xE6240503, + 0x81C, 0xE5260503, + 0x81C, 0xE4280503, + 0x81C, 0xE32A0503, + 0x81C, 0xC32C0503, + 0x81C, 0xC22E0503, + 0x81C, 0xC1300503, + 0x81C, 0xC0320503, + 0x81C, 0xA3340503, + 0x81C, 0xA2360503, + 0x81C, 0xA1380503, + 0x81C, 0xA03A0503, + 0x81C, 0x823C0503, + 0x81C, 0x813E0503, + 0x81C, 0x80400503, + 0x81C, 0x63420503, + 0x81C, 0x62440503, + 0x81C, 0x61460503, + 0x81C, 0x60480503, + 0x81C, 0x424A0503, + 0x81C, 0x414C0503, + 0x81C, 0x404E0503, + 0x81C, 0x22500503, + 0x81C, 0x21520503, + 0x81C, 0x20540503, + 0x81C, 0x03560503, + 0x81C, 0x02580503, + 0x81C, 0x015A0503, + 0x81C, 0x005C0503, + 0x81C, 0x005E0503, + 0x81C, 0x00600503, + 0x81C, 0x00620503, + 0x81C, 0x00640503, + 0x81C, 0x00660503, + 0x81C, 0x00680503, + 0x81C, 0x006A0503, + 0x81C, 0x006C0503, + 0x81C, 0x006E0503, + 0x81C, 0x00700503, + 0x81C, 0x00720503, + 0x81C, 0x00740503, + 0x81C, 0x00760503, + 0x81C, 0x00780503, + 0x81C, 0x007A0503, + 0x81C, 0x007C0503, + 0x81C, 0x007E0503, + 0x81C, 0x007E0503, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFE000503, + 0x81C, 0xFD020503, + 0x81C, 0xFC040503, + 0x81C, 0xFB060503, + 0x81C, 0xFA080503, + 0x81C, 0xF90A0503, + 0x81C, 0xF80C0503, + 0x81C, 0xF70E0503, + 0x81C, 0xF6100503, + 0x81C, 0xF5120503, + 0x81C, 0xF4140503, + 0x81C, 0xF3160503, + 0x81C, 0xF2180503, + 0x81C, 0xF11A0503, + 0x81C, 0xF01C0503, + 0x81C, 0xEF1E0503, + 0x81C, 0xEE200503, + 0x81C, 0xED220503, + 0x81C, 0xEC240503, + 0x81C, 0xEB260503, + 0x81C, 0xEA280503, + 0x81C, 0xE92A0503, + 0x81C, 0xE82C0503, + 0x81C, 0xE72E0503, + 0x81C, 0xE6300503, + 0x81C, 0xE5320503, + 0x81C, 0xE4340503, + 0x81C, 0xE3360503, + 0x81C, 0xC6380503, + 0x81C, 0xC53A0503, + 0x81C, 0xC43C0503, + 0x81C, 0xC33E0503, + 0x81C, 0xA5400503, + 0x81C, 0xA4420503, + 0x81C, 0xA3440503, + 0x81C, 0xA2460503, + 0x81C, 0xA1480503, + 0x81C, 0xA04A0503, + 0x81C, 0x824C0503, + 0x81C, 0x814E0503, + 0x81C, 0x80500503, + 0x81C, 0x64520503, + 0x81C, 0x63540503, + 0x81C, 0x62560503, + 0x81C, 0x61580503, + 0x81C, 0x605A0503, + 0x81C, 0x235C0503, + 0x81C, 0x225E0503, + 0x81C, 0x21600503, + 0x81C, 0x20620503, + 0x81C, 0x03640503, + 0x81C, 0x02660503, + 0x81C, 0x01680503, + 0x81C, 0x006A0503, + 0x81C, 0x006C0503, + 0x81C, 0x006E0503, + 0x81C, 0x00700503, + 0x81C, 0x00720503, + 0x81C, 0x00740503, + 0x81C, 0x00760503, + 0x81C, 0x00780503, + 0x81C, 0x007A0503, + 0x81C, 0x007C0503, + 0x81C, 0x007E0503, + 0x81C, 0x007E0503, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF8000503, + 0x81C, 0xF7020503, + 0x81C, 0xF6040503, + 0x81C, 0xF5060503, + 0x81C, 0xF4080503, + 0x81C, 0xF30A0503, + 0x81C, 0xF20C0503, + 0x81C, 0xF10E0503, + 0x81C, 0xF0100503, + 0x81C, 0xEF120503, + 0x81C, 0xEE140503, + 0x81C, 0xED160503, + 0x81C, 0xEC180503, + 0x81C, 0xEB1A0503, + 0x81C, 0xEA1C0503, + 0x81C, 0xE91E0503, + 0x81C, 0xE8200503, + 0x81C, 0xE7220503, + 0x81C, 0xE6240503, + 0x81C, 0xE5260503, + 0x81C, 0xE4280503, + 0x81C, 0xE32A0503, + 0x81C, 0xC32C0503, + 0x81C, 0xC22E0503, + 0x81C, 0xC1300503, + 0x81C, 0xC0320503, + 0x81C, 0xA3340503, + 0x81C, 0xA2360503, + 0x81C, 0xA1380503, + 0x81C, 0xA03A0503, + 0x81C, 0x823C0503, + 0x81C, 0x813E0503, + 0x81C, 0x80400503, + 0x81C, 0x63420503, + 0x81C, 0x62440503, + 0x81C, 0x61460503, + 0x81C, 0x60480503, + 0x81C, 0x424A0503, + 0x81C, 0x414C0503, + 0x81C, 0x404E0503, + 0x81C, 0x22500503, + 0x81C, 0x21520503, + 0x81C, 0x20540503, + 0x81C, 0x03560503, + 0x81C, 0x02580503, + 0x81C, 0x015A0503, + 0x81C, 0x005C0503, + 0x81C, 0x005E0503, + 0x81C, 0x00600503, + 0x81C, 0x00620503, + 0x81C, 0x00640503, + 0x81C, 0x00660503, + 0x81C, 0x00680503, + 0x81C, 0x006A0503, + 0x81C, 0x006C0503, + 0x81C, 0x006E0503, + 0x81C, 0x00700503, + 0x81C, 0x00720503, + 0x81C, 0x00740503, + 0x81C, 0x00760503, + 0x81C, 0x00780503, + 0x81C, 0x007A0503, + 0x81C, 0x007C0503, + 0x81C, 0x007E0503, + 0x81C, 0x007E0503, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFD000503, + 0x81C, 0xFC020503, + 0x81C, 0xFB040503, + 0x81C, 0xFA060503, + 0x81C, 0xF9080503, + 0x81C, 0xF80A0503, + 0x81C, 0xF70C0503, + 0x81C, 0xF60E0503, + 0x81C, 0xF5100503, + 0x81C, 0xF4120503, + 0x81C, 0xF3140503, + 0x81C, 0xF2160503, + 0x81C, 0xF1180503, + 0x81C, 0xF01A0503, + 0x81C, 0xEF1C0503, + 0x81C, 0xEE1E0503, + 0x81C, 0xED200503, + 0x81C, 0xEC220503, + 0x81C, 0xEB240503, + 0x81C, 0xEA260503, + 0x81C, 0xE9280503, + 0x81C, 0xE82A0503, + 0x81C, 0xE72C0503, + 0x81C, 0xE62E0503, + 0x81C, 0xE5300503, + 0x81C, 0xE4320503, + 0x81C, 0xE3340503, + 0x81C, 0xE2360503, + 0x81C, 0xC5380503, + 0x81C, 0xC43A0503, + 0x81C, 0xC33C0503, + 0x81C, 0xC23E0503, + 0x81C, 0xA5400503, + 0x81C, 0xA4420503, + 0x81C, 0xA3440503, + 0x81C, 0xA2460503, + 0x81C, 0xA1480503, + 0x81C, 0x834A0503, + 0x81C, 0x824C0503, + 0x81C, 0x814E0503, + 0x81C, 0x64500503, + 0x81C, 0x63520503, + 0x81C, 0x62540503, + 0x81C, 0x61560503, + 0x81C, 0x42580503, + 0x81C, 0x415A0503, + 0x81C, 0x405C0503, + 0x81C, 0x065E0503, + 0x81C, 0x05600503, + 0x81C, 0x04620503, + 0x81C, 0x03640503, + 0x81C, 0x02660503, + 0x81C, 0x01680503, + 0x81C, 0x006A0503, + 0x81C, 0x006C0503, + 0x81C, 0x006E0503, + 0x81C, 0x00700503, + 0x81C, 0x00720503, + 0x81C, 0x00740503, + 0x81C, 0x00760503, + 0x81C, 0x00780503, + 0x81C, 0x007A0503, + 0x81C, 0x007C0503, + 0x81C, 0x007E0503, + 0x81C, 0x007E0503, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFA000503, + 0x81C, 0xF9020503, + 0x81C, 0xF8040503, + 0x81C, 0xF7060503, + 0x81C, 0xF6080503, + 0x81C, 0xF50A0503, + 0x81C, 0xF40C0503, + 0x81C, 0xF30E0503, + 0x81C, 0xF2100503, + 0x81C, 0xF1120503, + 0x81C, 0xF0140503, + 0x81C, 0xEF160503, + 0x81C, 0xEE180503, + 0x81C, 0xED1A0503, + 0x81C, 0xEC1C0503, + 0x81C, 0xEB1E0503, + 0x81C, 0xEA200503, + 0x81C, 0xE9220503, + 0x81C, 0xE8240503, + 0x81C, 0xE7260503, + 0x81C, 0xE6280503, + 0x81C, 0xE52A0503, + 0x81C, 0xC42C0503, + 0x81C, 0xC32E0503, + 0x81C, 0xC2300503, + 0x81C, 0xC1320503, + 0x81C, 0xA4340503, + 0x81C, 0xA3360503, + 0x81C, 0xA2380503, + 0x81C, 0xA13A0503, + 0x81C, 0x833C0503, + 0x81C, 0x823E0503, + 0x81C, 0x81400503, + 0x81C, 0x63420503, + 0x81C, 0x62440503, + 0x81C, 0x61460503, + 0x81C, 0x60480503, + 0x81C, 0x424A0503, + 0x81C, 0x414C0503, + 0x81C, 0x404E0503, + 0x81C, 0x22500503, + 0x81C, 0x21520503, + 0x81C, 0x20540503, + 0x81C, 0x03560503, + 0x81C, 0x02580503, + 0x81C, 0x015A0503, + 0x81C, 0x005C0503, + 0x81C, 0x005E0503, + 0x81C, 0x00600503, + 0x81C, 0x00620503, + 0x81C, 0x00640503, + 0x81C, 0x00660503, + 0x81C, 0x00680503, + 0x81C, 0x006A0503, + 0x81C, 0x006C0503, + 0x81C, 0x006E0503, + 0x81C, 0x00700503, + 0x81C, 0x00720503, + 0x81C, 0x00740503, + 0x81C, 0x00760503, + 0x81C, 0x00780503, + 0x81C, 0x007A0503, + 0x81C, 0x007C0503, + 0x81C, 0x007E0503, + 0x81C, 0x007E0503, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xBF000503, + 0x81C, 0xBE020503, + 0x81C, 0xBD040503, + 0x81C, 0xBC060503, + 0x81C, 0xBB080503, + 0x81C, 0xBA0A0503, + 0x81C, 0xB90C0503, + 0x81C, 0xB80E0503, + 0x81C, 0xB7100503, + 0x81C, 0xB6120503, + 0x81C, 0xB5140503, + 0x81C, 0xB4160503, + 0x81C, 0xB3180503, + 0x81C, 0xB21A0503, + 0x81C, 0xB11C0503, + 0x81C, 0x931E0503, + 0x81C, 0x92200503, + 0x81C, 0x91220503, + 0x81C, 0x90240503, + 0x81C, 0x8F260503, + 0x81C, 0x8E280503, + 0x81C, 0x8D2A0503, + 0x81C, 0x8C2C0503, + 0x81C, 0x8B2E0503, + 0x81C, 0x8A300503, + 0x81C, 0x89320503, + 0x81C, 0x88340503, + 0x81C, 0x6A360503, + 0x81C, 0x69380503, + 0x81C, 0x683A0503, + 0x81C, 0x673C0503, + 0x81C, 0x663E0503, + 0x81C, 0x65400503, + 0x81C, 0x64420503, + 0x81C, 0x63440503, + 0x81C, 0x62460503, + 0x81C, 0x61480503, + 0x81C, 0x604A0503, + 0x81C, 0x424C0503, + 0x81C, 0x414E0503, + 0x81C, 0x40500503, + 0x81C, 0x06520503, + 0x81C, 0x05540503, + 0x81C, 0x04560503, + 0x81C, 0x03580503, + 0x81C, 0x025A0503, + 0x81C, 0x015C0503, + 0x81C, 0x005E0503, + 0x81C, 0x00600503, + 0x81C, 0x00620503, + 0x81C, 0x00640503, + 0x81C, 0x00660503, + 0x81C, 0x00680503, + 0x81C, 0x006A0503, + 0x81C, 0x006C0503, + 0x81C, 0x006E0503, + 0x81C, 0x00700503, + 0x81C, 0x00720503, + 0x81C, 0x00740503, + 0x81C, 0x00760503, + 0x81C, 0x00780503, + 0x81C, 0x007A0503, + 0x81C, 0x007C0503, + 0x81C, 0x007E0503, + 0x81C, 0x007E0503, + 0x90000008, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFD000503, + 0x81C, 0xFC020503, + 0x81C, 0xFB040503, + 0x81C, 0xFA060503, + 0x81C, 0xF9080503, + 0x81C, 0xF80A0503, + 0x81C, 0xF70C0503, + 0x81C, 0xF60E0503, + 0x81C, 0xF5100503, + 0x81C, 0xF4120503, + 0x81C, 0xF3140503, + 0x81C, 0xF2160503, + 0x81C, 0xF1180503, + 0x81C, 0xF01A0503, + 0x81C, 0xEF1C0503, + 0x81C, 0xEE1E0503, + 0x81C, 0xED200503, + 0x81C, 0xEC220503, + 0x81C, 0xEB240503, + 0x81C, 0xEA260503, + 0x81C, 0xE9280503, + 0x81C, 0xE82A0503, + 0x81C, 0xE72C0503, + 0x81C, 0xE62E0503, + 0x81C, 0xE5300503, + 0x81C, 0xE4320503, + 0x81C, 0xE3340503, + 0x81C, 0xC6360503, + 0x81C, 0xC5380503, + 0x81C, 0xC43A0503, + 0x81C, 0xC33C0503, + 0x81C, 0xC23E0503, + 0x81C, 0xA5400503, + 0x81C, 0xA4420503, + 0x81C, 0xA3440503, + 0x81C, 0xA2460503, + 0x81C, 0xA1480503, + 0x81C, 0x834A0503, + 0x81C, 0x824C0503, + 0x81C, 0x814E0503, + 0x81C, 0x63500503, + 0x81C, 0x62520503, + 0x81C, 0x61540503, + 0x81C, 0x43560503, + 0x81C, 0x42580503, + 0x81C, 0x245A0503, + 0x81C, 0x235C0503, + 0x81C, 0x225E0503, + 0x81C, 0x21600503, + 0x81C, 0x04620503, + 0x81C, 0x03640503, + 0x81C, 0x02660503, + 0x81C, 0x01680503, + 0x81C, 0x006A0503, + 0x81C, 0x006C0503, + 0x81C, 0x006E0503, + 0x81C, 0x00700503, + 0x81C, 0x00720503, + 0x81C, 0x00740503, + 0x81C, 0x00760503, + 0x81C, 0x00780503, + 0x81C, 0x007A0503, + 0x81C, 0x007C0503, + 0x81C, 0x007E0503, + 0x81C, 0x007E0503, + 0x90000009, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF8000503, + 0x81C, 0xF7020503, + 0x81C, 0xF6040503, + 0x81C, 0xF5060503, + 0x81C, 0xF4080503, + 0x81C, 0xF30A0503, + 0x81C, 0xF20C0503, + 0x81C, 0xF10E0503, + 0x81C, 0xF0100503, + 0x81C, 0xEF120503, + 0x81C, 0xEE140503, + 0x81C, 0xED160503, + 0x81C, 0xEC180503, + 0x81C, 0xEB1A0503, + 0x81C, 0xEA1C0503, + 0x81C, 0xE91E0503, + 0x81C, 0xE8200503, + 0x81C, 0xE7220503, + 0x81C, 0xE6240503, + 0x81C, 0xE5260503, + 0x81C, 0xE4280503, + 0x81C, 0xE32A0503, + 0x81C, 0xE22C0503, + 0x81C, 0xC32E0503, + 0x81C, 0xC2300503, + 0x81C, 0xC1320503, + 0x81C, 0xA3340503, + 0x81C, 0xA2360503, + 0x81C, 0xA1380503, + 0x81C, 0xA03A0503, + 0x81C, 0x823C0503, + 0x81C, 0x813E0503, + 0x81C, 0x80400503, + 0x81C, 0x64420503, + 0x81C, 0x63440503, + 0x81C, 0x62460503, + 0x81C, 0x61480503, + 0x81C, 0x434A0503, + 0x81C, 0x424C0503, + 0x81C, 0x414E0503, + 0x81C, 0x40500503, + 0x81C, 0x22520503, + 0x81C, 0x21540503, + 0x81C, 0x20560503, + 0x81C, 0x04580503, + 0x81C, 0x035A0503, + 0x81C, 0x025C0503, + 0x81C, 0x015E0503, + 0x81C, 0x00600503, + 0x81C, 0x00620503, + 0x81C, 0x00640503, + 0x81C, 0x00660503, + 0x81C, 0x00680503, + 0x81C, 0x006A0503, + 0x81C, 0x006C0503, + 0x81C, 0x006E0503, + 0x81C, 0x00700503, + 0x81C, 0x00720503, + 0x81C, 0x00740503, + 0x81C, 0x00760503, + 0x81C, 0x00780503, + 0x81C, 0x007A0503, + 0x81C, 0x007C0503, + 0x81C, 0x007E0503, + 0x81C, 0x007E0503, + 0x9000000a, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFD000503, + 0x81C, 0xFC020503, + 0x81C, 0xFB040503, + 0x81C, 0xFA060503, + 0x81C, 0xF9080503, + 0x81C, 0xF80A0503, + 0x81C, 0xF70C0503, + 0x81C, 0xF60E0503, + 0x81C, 0xF5100503, + 0x81C, 0xF4120503, + 0x81C, 0xF3140503, + 0x81C, 0xF2160503, + 0x81C, 0xF1180503, + 0x81C, 0xF01A0503, + 0x81C, 0xEE1C0503, + 0x81C, 0xED1E0503, + 0x81C, 0xEC200503, + 0x81C, 0xEB220503, + 0x81C, 0xEA240503, + 0x81C, 0xE9260503, + 0x81C, 0xE8280503, + 0x81C, 0xE72A0503, + 0x81C, 0xE62C0503, + 0x81C, 0xE52E0503, + 0x81C, 0xE4300503, + 0x81C, 0xE3320503, + 0x81C, 0xE2340503, + 0x81C, 0xC5360503, + 0x81C, 0xC4380503, + 0x81C, 0xC33A0503, + 0x81C, 0xC23C0503, + 0x81C, 0xA53E0503, + 0x81C, 0xA4400503, + 0x81C, 0xA3420503, + 0x81C, 0xA2440503, + 0x81C, 0xA1460503, + 0x81C, 0x83480503, + 0x81C, 0x824A0503, + 0x81C, 0x814C0503, + 0x81C, 0x804E0503, + 0x81C, 0x63500503, + 0x81C, 0x62520503, + 0x81C, 0x61540503, + 0x81C, 0x43560503, + 0x81C, 0x42580503, + 0x81C, 0x415A0503, + 0x81C, 0x405C0503, + 0x81C, 0x225E0503, + 0x81C, 0x21600503, + 0x81C, 0x20620503, + 0x81C, 0x03640503, + 0x81C, 0x02660503, + 0x81C, 0x01680503, + 0x81C, 0x006A0503, + 0x81C, 0x006C0503, + 0x81C, 0x006E0503, + 0x81C, 0x00700503, + 0x81C, 0x00720503, + 0x81C, 0x00740503, + 0x81C, 0x00760503, + 0x81C, 0x00780503, + 0x81C, 0x007A0503, + 0x81C, 0x007C0503, + 0x81C, 0x007E0503, + 0x81C, 0x007E0503, + 0x9000000b, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF9000503, + 0x81C, 0xF8020503, + 0x81C, 0xF7040503, + 0x81C, 0xF6060503, + 0x81C, 0xF5080503, + 0x81C, 0xF40A0503, + 0x81C, 0xF30C0503, + 0x81C, 0xF20E0503, + 0x81C, 0xF1100503, + 0x81C, 0xF0120503, + 0x81C, 0xEF140503, + 0x81C, 0xEE160503, + 0x81C, 0xED180503, + 0x81C, 0xEC1A0503, + 0x81C, 0xEB1C0503, + 0x81C, 0xEA1E0503, + 0x81C, 0xE9200503, + 0x81C, 0xE8220503, + 0x81C, 0xE7240503, + 0x81C, 0xE6260503, + 0x81C, 0xE5280503, + 0x81C, 0xE42A0503, + 0x81C, 0xE32C0503, + 0x81C, 0xC32E0503, + 0x81C, 0xC2300503, + 0x81C, 0xC1320503, + 0x81C, 0xA4340503, + 0x81C, 0xA3360503, + 0x81C, 0xA2380503, + 0x81C, 0xA13A0503, + 0x81C, 0xA03C0503, + 0x81C, 0x823E0503, + 0x81C, 0x81400503, + 0x81C, 0x80420503, + 0x81C, 0x63440503, + 0x81C, 0x62460503, + 0x81C, 0x61480503, + 0x81C, 0x604A0503, + 0x81C, 0x244C0503, + 0x81C, 0x234E0503, + 0x81C, 0x22500503, + 0x81C, 0x21520503, + 0x81C, 0x20540503, + 0x81C, 0x05560503, + 0x81C, 0x04580503, + 0x81C, 0x035A0503, + 0x81C, 0x025C0503, + 0x81C, 0x015E0503, + 0x81C, 0x00600503, + 0x81C, 0x00620503, + 0x81C, 0x00640503, + 0x81C, 0x00660503, + 0x81C, 0x00680503, + 0x81C, 0x006A0503, + 0x81C, 0x006C0503, + 0x81C, 0x006E0503, + 0x81C, 0x00700503, + 0x81C, 0x00720503, + 0x81C, 0x00740503, + 0x81C, 0x00760503, + 0x81C, 0x00780503, + 0x81C, 0x007A0503, + 0x81C, 0x007C0503, + 0x81C, 0x007E0503, + 0x81C, 0x007E0503, + 0x9000000c, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFE000503, + 0x81C, 0xFD020503, + 0x81C, 0xFC040503, + 0x81C, 0xFB060503, + 0x81C, 0xFA080503, + 0x81C, 0xF90A0503, + 0x81C, 0xF80C0503, + 0x81C, 0xF70E0503, + 0x81C, 0xF6100503, + 0x81C, 0xF5120503, + 0x81C, 0xF4140503, + 0x81C, 0xF3160503, + 0x81C, 0xF2180503, + 0x81C, 0xF11A0503, + 0x81C, 0xF01C0503, + 0x81C, 0xEF1E0503, + 0x81C, 0xEE200503, + 0x81C, 0xED220503, + 0x81C, 0xEC240503, + 0x81C, 0xEB260503, + 0x81C, 0xEA280503, + 0x81C, 0xE92A0503, + 0x81C, 0xE82C0503, + 0x81C, 0xE72E0503, + 0x81C, 0xE6300503, + 0x81C, 0xE5320503, + 0x81C, 0xE4340503, + 0x81C, 0xE3360503, + 0x81C, 0xC6380503, + 0x81C, 0xC53A0503, + 0x81C, 0xC43C0503, + 0x81C, 0xC33E0503, + 0x81C, 0xA5400503, + 0x81C, 0xA4420503, + 0x81C, 0xA3440503, + 0x81C, 0xA2460503, + 0x81C, 0xA1480503, + 0x81C, 0xA04A0503, + 0x81C, 0x824C0503, + 0x81C, 0x814E0503, + 0x81C, 0x80500503, + 0x81C, 0x64520503, + 0x81C, 0x63540503, + 0x81C, 0x62560503, + 0x81C, 0x61580503, + 0x81C, 0x605A0503, + 0x81C, 0x235C0503, + 0x81C, 0x225E0503, + 0x81C, 0x21600503, + 0x81C, 0x20620503, + 0x81C, 0x03640503, + 0x81C, 0x02660503, + 0x81C, 0x01680503, + 0x81C, 0x006A0503, + 0x81C, 0x006C0503, + 0x81C, 0x006E0503, + 0x81C, 0x00700503, + 0x81C, 0x00720503, + 0x81C, 0x00740503, + 0x81C, 0x00760503, + 0x81C, 0x00780503, + 0x81C, 0x007A0503, + 0x81C, 0x007C0503, + 0x81C, 0x007E0503, + 0x81C, 0x007E0503, + 0x9000000d, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFD000503, + 0x81C, 0xFC020503, + 0x81C, 0xFB040503, + 0x81C, 0xFA060503, + 0x81C, 0xF9080503, + 0x81C, 0xF80A0503, + 0x81C, 0xF70C0503, + 0x81C, 0xF60E0503, + 0x81C, 0xF5100503, + 0x81C, 0xF4120503, + 0x81C, 0xF3140503, + 0x81C, 0xF2160503, + 0x81C, 0xF1180503, + 0x81C, 0xF01A0503, + 0x81C, 0xEE1C0503, + 0x81C, 0xED1E0503, + 0x81C, 0xEC200503, + 0x81C, 0xEB220503, + 0x81C, 0xEA240503, + 0x81C, 0xE9260503, + 0x81C, 0xE8280503, + 0x81C, 0xE72A0503, + 0x81C, 0xE62C0503, + 0x81C, 0xE52E0503, + 0x81C, 0xE4300503, + 0x81C, 0xE3320503, + 0x81C, 0xE2340503, + 0x81C, 0xC5360503, + 0x81C, 0xC4380503, + 0x81C, 0xC33A0503, + 0x81C, 0xC23C0503, + 0x81C, 0xA53E0503, + 0x81C, 0xA4400503, + 0x81C, 0xA3420503, + 0x81C, 0xA2440503, + 0x81C, 0xA1460503, + 0x81C, 0x83480503, + 0x81C, 0x824A0503, + 0x81C, 0x814C0503, + 0x81C, 0x804E0503, + 0x81C, 0x63500503, + 0x81C, 0x62520503, + 0x81C, 0x61540503, + 0x81C, 0x43560503, + 0x81C, 0x42580503, + 0x81C, 0x415A0503, + 0x81C, 0x405C0503, + 0x81C, 0x225E0503, + 0x81C, 0x21600503, + 0x81C, 0x20620503, + 0x81C, 0x03640503, + 0x81C, 0x02660503, + 0x81C, 0x01680503, + 0x81C, 0x006A0503, + 0x81C, 0x006C0503, + 0x81C, 0x006E0503, + 0x81C, 0x00700503, + 0x81C, 0x00720503, + 0x81C, 0x00740503, + 0x81C, 0x00760503, + 0x81C, 0x00780503, + 0x81C, 0x007A0503, + 0x81C, 0x007C0503, + 0x81C, 0x007E0503, + 0x81C, 0x007E0503, + 0x9000000e, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFD000503, + 0x81C, 0xFC020503, + 0x81C, 0xFB040503, + 0x81C, 0xFA060503, + 0x81C, 0xF9080503, + 0x81C, 0xF80A0503, + 0x81C, 0xF70C0503, + 0x81C, 0xF60E0503, + 0x81C, 0xF5100503, + 0x81C, 0xF4120503, + 0x81C, 0xF3140503, + 0x81C, 0xF2160503, + 0x81C, 0xF1180503, + 0x81C, 0xF01A0503, + 0x81C, 0xEE1C0503, + 0x81C, 0xED1E0503, + 0x81C, 0xEC200503, + 0x81C, 0xEB220503, + 0x81C, 0xEA240503, + 0x81C, 0xE9260503, + 0x81C, 0xE8280503, + 0x81C, 0xE72A0503, + 0x81C, 0xE62C0503, + 0x81C, 0xE52E0503, + 0x81C, 0xE4300503, + 0x81C, 0xE3320503, + 0x81C, 0xE2340503, + 0x81C, 0xC5360503, + 0x81C, 0xC4380503, + 0x81C, 0xC33A0503, + 0x81C, 0xC23C0503, + 0x81C, 0xA53E0503, + 0x81C, 0xA4400503, + 0x81C, 0xA3420503, + 0x81C, 0xA2440503, + 0x81C, 0xA1460503, + 0x81C, 0x83480503, + 0x81C, 0x824A0503, + 0x81C, 0x814C0503, + 0x81C, 0x804E0503, + 0x81C, 0x63500503, + 0x81C, 0x62520503, + 0x81C, 0x61540503, + 0x81C, 0x43560503, + 0x81C, 0x42580503, + 0x81C, 0x415A0503, + 0x81C, 0x405C0503, + 0x81C, 0x225E0503, + 0x81C, 0x21600503, + 0x81C, 0x20620503, + 0x81C, 0x03640503, + 0x81C, 0x02660503, + 0x81C, 0x01680503, + 0x81C, 0x006A0503, + 0x81C, 0x006C0503, + 0x81C, 0x006E0503, + 0x81C, 0x00700503, + 0x81C, 0x00720503, + 0x81C, 0x00740503, + 0x81C, 0x00760503, + 0x81C, 0x00780503, + 0x81C, 0x007A0503, + 0x81C, 0x007C0503, + 0x81C, 0x007E0503, + 0x81C, 0x007E0503, + 0x9000000f, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xBF000503, + 0x81C, 0xBF020503, + 0x81C, 0xBF040503, + 0x81C, 0xBF060503, + 0x81C, 0xBF080503, + 0x81C, 0xBF0A0503, + 0x81C, 0xBE0C0503, + 0x81C, 0xBD0E0503, + 0x81C, 0xBC100503, + 0x81C, 0xBB120503, + 0x81C, 0xBA140503, + 0x81C, 0xB9160503, + 0x81C, 0xB8180503, + 0x81C, 0xB71A0503, + 0x81C, 0xB61C0503, + 0x81C, 0xB51E0503, + 0x81C, 0xB2200503, + 0x81C, 0xB3220503, + 0x81C, 0xB2240503, + 0x81C, 0xB1260503, + 0x81C, 0xB0280503, + 0x81C, 0xAF2A0503, + 0x81C, 0xAE2C0503, + 0x81C, 0xAD2E0503, + 0x81C, 0xAC300503, + 0x81C, 0xAB320503, + 0x81C, 0xAA340503, + 0x81C, 0xC6360503, + 0x81C, 0xC5380503, + 0x81C, 0xC43A0503, + 0x81C, 0xC33C0503, + 0x81C, 0x883E0503, + 0x81C, 0x87400503, + 0x81C, 0x86420503, + 0x81C, 0x85440503, + 0x81C, 0x84460503, + 0x81C, 0x83480503, + 0x81C, 0x674A0503, + 0x81C, 0x664C0503, + 0x81C, 0x654E0503, + 0x81C, 0x64500503, + 0x81C, 0x27520503, + 0x81C, 0x26540503, + 0x81C, 0x25560503, + 0x81C, 0x24580503, + 0x81C, 0x235A0503, + 0x81C, 0x225C0503, + 0x81C, 0x215E0503, + 0x81C, 0x20600503, + 0x81C, 0x03620503, + 0x81C, 0x02640503, + 0x81C, 0x01660503, + 0x81C, 0x00680503, + 0x81C, 0x006A0503, + 0x81C, 0x006C0503, + 0x81C, 0x006E0503, + 0x81C, 0x00700503, + 0x81C, 0x00720503, + 0x81C, 0x00740503, + 0x81C, 0x00760503, + 0x81C, 0x00780503, + 0x81C, 0x007A0503, + 0x81C, 0x007C0503, + 0x81C, 0x007E0503, + 0x81C, 0x007E0503, + 0x90000010, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFE000403, + 0x81C, 0xFD000503, + 0x81C, 0xFC020503, + 0x81C, 0xFB040503, + 0x81C, 0xFA060503, + 0x81C, 0xF9080503, + 0x81C, 0xF80A0503, + 0x81C, 0xF70C0503, + 0x81C, 0xF60E0503, + 0x81C, 0xF5100503, + 0x81C, 0xF4120503, + 0x81C, 0xF3140503, + 0x81C, 0xF2160503, + 0x81C, 0xF1180503, + 0x81C, 0xF01A0503, + 0x81C, 0xEF1C0503, + 0x81C, 0xEE1E0503, + 0x81C, 0xED200503, + 0x81C, 0xEC220503, + 0x81C, 0xEB240503, + 0x81C, 0xEA260503, + 0x81C, 0xE9280503, + 0x81C, 0xE82A0503, + 0x81C, 0xE72C0503, + 0x81C, 0xE62E0503, + 0x81C, 0xE5300503, + 0x81C, 0xE4320503, + 0x81C, 0xE3340503, + 0x81C, 0xC6360503, + 0x81C, 0xC5380503, + 0x81C, 0xC43A0503, + 0x81C, 0xC33C0503, + 0x81C, 0xA53E0503, + 0x81C, 0xA4400503, + 0x81C, 0xA3420503, + 0x81C, 0xA2440503, + 0x81C, 0xA1460503, + 0x81C, 0xA0480503, + 0x81C, 0x824A0503, + 0x81C, 0x814C0503, + 0x81C, 0x804E0503, + 0x81C, 0x64500503, + 0x81C, 0x63520503, + 0x81C, 0x62540503, + 0x81C, 0x61560503, + 0x81C, 0x60580503, + 0x81C, 0x235A0503, + 0x81C, 0x225C0503, + 0x81C, 0x215E0503, + 0x81C, 0x20600503, + 0x81C, 0x03620503, + 0x81C, 0x02640503, + 0x81C, 0x01660503, + 0x81C, 0x00680503, + 0x81C, 0x006A0503, + 0x81C, 0x006C0503, + 0x81C, 0x006E0503, + 0x81C, 0x00700503, + 0x81C, 0x00720503, + 0x81C, 0x00740503, + 0x81C, 0x00760503, + 0x81C, 0x00780503, + 0x81C, 0x007A0503, + 0x81C, 0x007C0503, + 0x81C, 0x007E0503, + 0x90000012, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF8000503, + 0x81C, 0xF7020503, + 0x81C, 0xF6040503, + 0x81C, 0xF5060503, + 0x81C, 0xF4080503, + 0x81C, 0xF30A0503, + 0x81C, 0xF20C0503, + 0x81C, 0xF10E0503, + 0x81C, 0xF0100503, + 0x81C, 0xEF120503, + 0x81C, 0xEE140503, + 0x81C, 0xED160503, + 0x81C, 0xEC180503, + 0x81C, 0xEB1A0503, + 0x81C, 0xEA1C0503, + 0x81C, 0xE91E0503, + 0x81C, 0xE8200503, + 0x81C, 0xE7220503, + 0x81C, 0xE6240503, + 0x81C, 0xE5260503, + 0x81C, 0xE4280503, + 0x81C, 0xE32A0503, + 0x81C, 0xC32C0503, + 0x81C, 0xC22E0503, + 0x81C, 0xC1300503, + 0x81C, 0xC0320503, + 0x81C, 0xA3340503, + 0x81C, 0xA2360503, + 0x81C, 0xA1380503, + 0x81C, 0xA03A0503, + 0x81C, 0x823C0503, + 0x81C, 0x813E0503, + 0x81C, 0x80400503, + 0x81C, 0x63420503, + 0x81C, 0x62440503, + 0x81C, 0x61460503, + 0x81C, 0x60480503, + 0x81C, 0x424A0503, + 0x81C, 0x414C0503, + 0x81C, 0x404E0503, + 0x81C, 0x22500503, + 0x81C, 0x21520503, + 0x81C, 0x20540503, + 0x81C, 0x03560503, + 0x81C, 0x02580503, + 0x81C, 0x015A0503, + 0x81C, 0x005C0503, + 0x81C, 0x005E0503, + 0x81C, 0x00600503, + 0x81C, 0x00620503, + 0x81C, 0x00640503, + 0x81C, 0x00660503, + 0x81C, 0x00680503, + 0x81C, 0x006A0503, + 0x81C, 0x006C0503, + 0x81C, 0x006E0503, + 0x81C, 0x00700503, + 0x81C, 0x00720503, + 0x81C, 0x00740503, + 0x81C, 0x00760503, + 0x81C, 0x00780503, + 0x81C, 0x007A0503, + 0x81C, 0x007C0503, + 0x81C, 0x007E0503, + 0x81C, 0x007E0503, + 0xA0000000, 0x00000000, + 0x81C, 0xFE000503, + 0x81C, 0xFD020503, + 0x81C, 0xFC040503, + 0x81C, 0xFB060503, + 0x81C, 0xFA080503, + 0x81C, 0xF90A0503, + 0x81C, 0xF80C0503, + 0x81C, 0xF70E0503, + 0x81C, 0xF6100503, + 0x81C, 0xF5120503, + 0x81C, 0xF4140503, + 0x81C, 0xF3160503, + 0x81C, 0xF2180503, + 0x81C, 0xF11A0503, + 0x81C, 0xF01C0503, + 0x81C, 0xEF1E0503, + 0x81C, 0xEE200503, + 0x81C, 0xED220503, + 0x81C, 0xEC240503, + 0x81C, 0xEB260503, + 0x81C, 0xEA280503, + 0x81C, 0xE92A0503, + 0x81C, 0xE82C0503, + 0x81C, 0xE72E0503, + 0x81C, 0xE6300503, + 0x81C, 0xE5320503, + 0x81C, 0xE4340503, + 0x81C, 0xE3360503, + 0x81C, 0xC6380503, + 0x81C, 0xC53A0503, + 0x81C, 0xC43C0503, + 0x81C, 0xC33E0503, + 0x81C, 0xA5400503, + 0x81C, 0xA4420503, + 0x81C, 0xA3440503, + 0x81C, 0xA2460503, + 0x81C, 0xA1480503, + 0x81C, 0xA04A0503, + 0x81C, 0x824C0503, + 0x81C, 0x814E0503, + 0x81C, 0x80500503, + 0x81C, 0x64520503, + 0x81C, 0x63540503, + 0x81C, 0x62560503, + 0x81C, 0x61580503, + 0x81C, 0x605A0503, + 0x81C, 0x235C0503, + 0x81C, 0x225E0503, + 0x81C, 0x21600503, + 0x81C, 0x20620503, + 0x81C, 0x03640503, + 0x81C, 0x02660503, + 0x81C, 0x01680503, + 0x81C, 0x006A0503, + 0x81C, 0x006C0503, + 0x81C, 0x006E0503, + 0x81C, 0x00700503, + 0x81C, 0x00720503, + 0x81C, 0x00740503, + 0x81C, 0x00760503, + 0x81C, 0x00780503, + 0x81C, 0x007A0503, + 0x81C, 0x007C0503, + 0x81C, 0x007E0503, + 0x81C, 0x007E0503, + 0xB0000000, 0x00000000, + 0x80000000, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000603, + 0x81C, 0xFB020603, + 0x81C, 0xFA040603, + 0x81C, 0xF9060603, + 0x81C, 0xF8080603, + 0x81C, 0xF70A0603, + 0x81C, 0xF60C0603, + 0x81C, 0xF50E0603, + 0x81C, 0xF4100603, + 0x81C, 0xF3120603, + 0x81C, 0xF2140603, + 0x81C, 0xF1160603, + 0x81C, 0xF0180603, + 0x81C, 0xEE1A0603, + 0x81C, 0xED1C0603, + 0x81C, 0xEC1E0603, + 0x81C, 0xEB200603, + 0x81C, 0xEA220603, + 0x81C, 0xE9240603, + 0x81C, 0xE8260603, + 0x81C, 0xE7280603, + 0x81C, 0xE62A0603, + 0x81C, 0xE52C0603, + 0x81C, 0xE42E0603, + 0x81C, 0xE3300603, + 0x81C, 0xE2320603, + 0x81C, 0xC6340603, + 0x81C, 0xC5360603, + 0x81C, 0xC4380603, + 0x81C, 0xC33A0603, + 0x81C, 0xA63C0603, + 0x81C, 0xA53E0603, + 0x81C, 0xA4400603, + 0x81C, 0xA3420603, + 0x81C, 0xA2440603, + 0x81C, 0xA1460603, + 0x81C, 0x83480603, + 0x81C, 0x824A0603, + 0x81C, 0x814C0603, + 0x81C, 0x804E0603, + 0x81C, 0x63500603, + 0x81C, 0x62520603, + 0x81C, 0x61540603, + 0x81C, 0x42560603, + 0x81C, 0x41580603, + 0x81C, 0x405A0603, + 0x81C, 0x225C0603, + 0x81C, 0x215E0603, + 0x81C, 0x20600603, + 0x81C, 0x04620603, + 0x81C, 0x03640603, + 0x81C, 0x02660603, + 0x81C, 0x01680603, + 0x81C, 0x006A0603, + 0x81C, 0x006C0603, + 0x81C, 0x006E0603, + 0x81C, 0x00700603, + 0x81C, 0x00720603, + 0x81C, 0x00740603, + 0x81C, 0x00760603, + 0x81C, 0x00780603, + 0x81C, 0x007A0603, + 0x81C, 0x007C0603, + 0x81C, 0x007E0603, + 0x81C, 0x007E0603, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xBD000603, + 0x81C, 0xBC020603, + 0x81C, 0xBB040603, + 0x81C, 0xBA060603, + 0x81C, 0xB9080603, + 0x81C, 0xB80A0603, + 0x81C, 0xB70C0603, + 0x81C, 0xB60E0603, + 0x81C, 0xB5100603, + 0x81C, 0xB4120603, + 0x81C, 0xB3140603, + 0x81C, 0xB2160603, + 0x81C, 0xB1180603, + 0x81C, 0xB01A0603, + 0x81C, 0xAF1C0603, + 0x81C, 0xAE1E0603, + 0x81C, 0xAD200603, + 0x81C, 0x8F220603, + 0x81C, 0x8E240603, + 0x81C, 0x8D260603, + 0x81C, 0x8C280603, + 0x81C, 0x8B2A0603, + 0x81C, 0x8A2C0603, + 0x81C, 0x892E0603, + 0x81C, 0x88300603, + 0x81C, 0x6B320603, + 0x81C, 0x6A340603, + 0x81C, 0x69360603, + 0x81C, 0x68380603, + 0x81C, 0x673A0603, + 0x81C, 0x663C0603, + 0x81C, 0x653E0603, + 0x81C, 0x64400603, + 0x81C, 0x63420603, + 0x81C, 0x62440603, + 0x81C, 0x61460603, + 0x81C, 0x60480603, + 0x81C, 0x424A0603, + 0x81C, 0x414C0603, + 0x81C, 0x404E0603, + 0x81C, 0x06500603, + 0x81C, 0x05520603, + 0x81C, 0x04540603, + 0x81C, 0x03560603, + 0x81C, 0x02580603, + 0x81C, 0x015A0603, + 0x81C, 0x005C0603, + 0x81C, 0x005E0603, + 0x81C, 0x00600603, + 0x81C, 0x00620603, + 0x81C, 0x00640603, + 0x81C, 0x00660603, + 0x81C, 0x00680603, + 0x81C, 0x006A0603, + 0x81C, 0x006C0603, + 0x81C, 0x006E0603, + 0x81C, 0x00700603, + 0x81C, 0x00720603, + 0x81C, 0x00740603, + 0x81C, 0x00760603, + 0x81C, 0x00780603, + 0x81C, 0x007A0603, + 0x81C, 0x007C0603, + 0x81C, 0x007E0603, + 0x81C, 0x007C0603, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF7000603, + 0x81C, 0xF6020603, + 0x81C, 0xF5040603, + 0x81C, 0xF4060603, + 0x81C, 0xF3080603, + 0x81C, 0xF20A0603, + 0x81C, 0xF10C0603, + 0x81C, 0xF00E0603, + 0x81C, 0xEF100603, + 0x81C, 0xEE120603, + 0x81C, 0xED140603, + 0x81C, 0xEC160603, + 0x81C, 0xEB180603, + 0x81C, 0xEA1A0603, + 0x81C, 0xE91C0603, + 0x81C, 0xE81E0603, + 0x81C, 0xE7200603, + 0x81C, 0xE6220603, + 0x81C, 0xE5240603, + 0x81C, 0xE4260603, + 0x81C, 0xE3280603, + 0x81C, 0xC42A0603, + 0x81C, 0xC32C0603, + 0x81C, 0xC22E0603, + 0x81C, 0xC1300603, + 0x81C, 0xC0320603, + 0x81C, 0xA3340603, + 0x81C, 0xA2360603, + 0x81C, 0xA1380603, + 0x81C, 0xA03A0603, + 0x81C, 0x823C0603, + 0x81C, 0x813E0603, + 0x81C, 0x80400603, + 0x81C, 0x64420603, + 0x81C, 0x63440603, + 0x81C, 0x62460603, + 0x81C, 0x61480603, + 0x81C, 0x604A0603, + 0x81C, 0x414C0603, + 0x81C, 0x404E0603, + 0x81C, 0x22500603, + 0x81C, 0x21520603, + 0x81C, 0x20540603, + 0x81C, 0x03560603, + 0x81C, 0x02580603, + 0x81C, 0x015A0603, + 0x81C, 0x005C0603, + 0x81C, 0x005E0603, + 0x81C, 0x00600603, + 0x81C, 0x00620603, + 0x81C, 0x00640603, + 0x81C, 0x00660603, + 0x81C, 0x00680603, + 0x81C, 0x006A0603, + 0x81C, 0x006C0603, + 0x81C, 0x006E0603, + 0x81C, 0x00700603, + 0x81C, 0x00720603, + 0x81C, 0x00740603, + 0x81C, 0x00760603, + 0x81C, 0x00780603, + 0x81C, 0x007A0603, + 0x81C, 0x007C0603, + 0x81C, 0x007E0603, + 0x81C, 0x007E0603, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000603, + 0x81C, 0xFB020603, + 0x81C, 0xFA040603, + 0x81C, 0xF9060603, + 0x81C, 0xF8080603, + 0x81C, 0xF70A0603, + 0x81C, 0xF60C0603, + 0x81C, 0xF50E0603, + 0x81C, 0xF4100603, + 0x81C, 0xF3120603, + 0x81C, 0xF2140603, + 0x81C, 0xF1160603, + 0x81C, 0xF0180603, + 0x81C, 0xEF1A0603, + 0x81C, 0xEE1C0603, + 0x81C, 0xED1E0603, + 0x81C, 0xEC200603, + 0x81C, 0xEB220603, + 0x81C, 0xEA240603, + 0x81C, 0xE9260603, + 0x81C, 0xE8280603, + 0x81C, 0xE72A0603, + 0x81C, 0xE62C0603, + 0x81C, 0xE52E0603, + 0x81C, 0xE4300603, + 0x81C, 0xE3320603, + 0x81C, 0xE2340603, + 0x81C, 0xC6360603, + 0x81C, 0xC5380603, + 0x81C, 0xC43A0603, + 0x81C, 0xC33C0603, + 0x81C, 0xA63E0603, + 0x81C, 0xA5400603, + 0x81C, 0xA4420603, + 0x81C, 0xA3440603, + 0x81C, 0xA2460603, + 0x81C, 0xA1480603, + 0x81C, 0x834A0603, + 0x81C, 0x824C0603, + 0x81C, 0x814E0603, + 0x81C, 0x64500603, + 0x81C, 0x63520603, + 0x81C, 0x62540603, + 0x81C, 0x61560603, + 0x81C, 0x60580603, + 0x81C, 0x405A0603, + 0x81C, 0x215C0603, + 0x81C, 0x205E0603, + 0x81C, 0x03600603, + 0x81C, 0x02620603, + 0x81C, 0x01640603, + 0x81C, 0x00660603, + 0x81C, 0x00680603, + 0x81C, 0x006A0603, + 0x81C, 0x006C0603, + 0x81C, 0x006E0603, + 0x81C, 0x00700603, + 0x81C, 0x00720603, + 0x81C, 0x00740603, + 0x81C, 0x00760603, + 0x81C, 0x00780603, + 0x81C, 0x007A0603, + 0x81C, 0x007C0603, + 0x81C, 0x007E0603, + 0x81C, 0x007E0603, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF7000603, + 0x81C, 0xF6020603, + 0x81C, 0xF5040603, + 0x81C, 0xF4060603, + 0x81C, 0xF3080603, + 0x81C, 0xF20A0603, + 0x81C, 0xF10C0603, + 0x81C, 0xF00E0603, + 0x81C, 0xEF100603, + 0x81C, 0xEE120603, + 0x81C, 0xED140603, + 0x81C, 0xEC160603, + 0x81C, 0xEB180603, + 0x81C, 0xEA1A0603, + 0x81C, 0xE91C0603, + 0x81C, 0xE81E0603, + 0x81C, 0xE7200603, + 0x81C, 0xE6220603, + 0x81C, 0xE5240603, + 0x81C, 0xE4260603, + 0x81C, 0xE3280603, + 0x81C, 0xC42A0603, + 0x81C, 0xC32C0603, + 0x81C, 0xC22E0603, + 0x81C, 0xC1300603, + 0x81C, 0xC0320603, + 0x81C, 0xA3340603, + 0x81C, 0xA2360603, + 0x81C, 0xA1380603, + 0x81C, 0xA03A0603, + 0x81C, 0x823C0603, + 0x81C, 0x813E0603, + 0x81C, 0x80400603, + 0x81C, 0x64420603, + 0x81C, 0x63440603, + 0x81C, 0x62460603, + 0x81C, 0x61480603, + 0x81C, 0x604A0603, + 0x81C, 0x414C0603, + 0x81C, 0x404E0603, + 0x81C, 0x22500603, + 0x81C, 0x21520603, + 0x81C, 0x20540603, + 0x81C, 0x03560603, + 0x81C, 0x02580603, + 0x81C, 0x015A0603, + 0x81C, 0x005C0603, + 0x81C, 0x005E0603, + 0x81C, 0x00600603, + 0x81C, 0x00620603, + 0x81C, 0x00640603, + 0x81C, 0x00660603, + 0x81C, 0x00680603, + 0x81C, 0x006A0603, + 0x81C, 0x006C0603, + 0x81C, 0x006E0603, + 0x81C, 0x00700603, + 0x81C, 0x00720603, + 0x81C, 0x00740603, + 0x81C, 0x00760603, + 0x81C, 0x00780603, + 0x81C, 0x007A0603, + 0x81C, 0x007C0603, + 0x81C, 0x007E0603, + 0x81C, 0x007E0603, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000603, + 0x81C, 0xFB020603, + 0x81C, 0xFA040603, + 0x81C, 0xF9060603, + 0x81C, 0xF8080603, + 0x81C, 0xF70A0603, + 0x81C, 0xF60C0603, + 0x81C, 0xF50E0603, + 0x81C, 0xF4100603, + 0x81C, 0xF3120603, + 0x81C, 0xF2140603, + 0x81C, 0xF1160603, + 0x81C, 0xF0180603, + 0x81C, 0xEF1A0603, + 0x81C, 0xEE1C0603, + 0x81C, 0xED1E0603, + 0x81C, 0xEC200603, + 0x81C, 0xEB220603, + 0x81C, 0xEA240603, + 0x81C, 0xE9260603, + 0x81C, 0xE8280603, + 0x81C, 0xE72A0603, + 0x81C, 0xE62C0603, + 0x81C, 0xE52E0603, + 0x81C, 0xE4300603, + 0x81C, 0xE3320603, + 0x81C, 0xE2340603, + 0x81C, 0xE1360603, + 0x81C, 0xC5380603, + 0x81C, 0xC43A0603, + 0x81C, 0xC33C0603, + 0x81C, 0xC23E0603, + 0x81C, 0xC1400603, + 0x81C, 0xA3420603, + 0x81C, 0xA2440603, + 0x81C, 0xA1460603, + 0x81C, 0xA0480603, + 0x81C, 0x834A0603, + 0x81C, 0x824C0603, + 0x81C, 0x814E0603, + 0x81C, 0x64500603, + 0x81C, 0x63520603, + 0x81C, 0x62540603, + 0x81C, 0x61560603, + 0x81C, 0x25580603, + 0x81C, 0x245A0603, + 0x81C, 0x235C0603, + 0x81C, 0x225E0603, + 0x81C, 0x21600603, + 0x81C, 0x04620603, + 0x81C, 0x03640603, + 0x81C, 0x02660603, + 0x81C, 0x01680603, + 0x81C, 0x006A0603, + 0x81C, 0x006C0603, + 0x81C, 0x006E0603, + 0x81C, 0x00700603, + 0x81C, 0x00720603, + 0x81C, 0x00740603, + 0x81C, 0x00760603, + 0x81C, 0x00780603, + 0x81C, 0x007A0603, + 0x81C, 0x007C0603, + 0x81C, 0x007E0603, + 0x81C, 0x007E0603, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF9000603, + 0x81C, 0xF8020603, + 0x81C, 0xF7040603, + 0x81C, 0xF6060603, + 0x81C, 0xF5080603, + 0x81C, 0xF40A0603, + 0x81C, 0xF30C0603, + 0x81C, 0xF20E0603, + 0x81C, 0xF1100603, + 0x81C, 0xF0120603, + 0x81C, 0xEF140603, + 0x81C, 0xEE160603, + 0x81C, 0xED180603, + 0x81C, 0xEC1A0603, + 0x81C, 0xEB1C0603, + 0x81C, 0xEA1E0603, + 0x81C, 0xE9200603, + 0x81C, 0xE8220603, + 0x81C, 0xE7240603, + 0x81C, 0xE6260603, + 0x81C, 0xE5280603, + 0x81C, 0xC42A0603, + 0x81C, 0xC32C0603, + 0x81C, 0xC22E0603, + 0x81C, 0xC1300603, + 0x81C, 0xC0320603, + 0x81C, 0xA3340603, + 0x81C, 0xA2360603, + 0x81C, 0xA1380603, + 0x81C, 0xA03A0603, + 0x81C, 0x823C0603, + 0x81C, 0x813E0603, + 0x81C, 0x80400603, + 0x81C, 0x64420603, + 0x81C, 0x63440603, + 0x81C, 0x62460603, + 0x81C, 0x61480603, + 0x81C, 0x604A0603, + 0x81C, 0x414C0603, + 0x81C, 0x404E0603, + 0x81C, 0x22500603, + 0x81C, 0x21520603, + 0x81C, 0x20540603, + 0x81C, 0x03560603, + 0x81C, 0x02580603, + 0x81C, 0x015A0603, + 0x81C, 0x005C0603, + 0x81C, 0x005E0603, + 0x81C, 0x00600603, + 0x81C, 0x00620603, + 0x81C, 0x00640603, + 0x81C, 0x00660603, + 0x81C, 0x00680603, + 0x81C, 0x006A0603, + 0x81C, 0x006C0603, + 0x81C, 0x006E0603, + 0x81C, 0x00700603, + 0x81C, 0x00720603, + 0x81C, 0x00740603, + 0x81C, 0x00760603, + 0x81C, 0x00780603, + 0x81C, 0x007A0603, + 0x81C, 0x007C0603, + 0x81C, 0x007E0603, + 0x81C, 0x007E0603, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xBE000603, + 0x81C, 0xBD020603, + 0x81C, 0xBC040603, + 0x81C, 0xBB060603, + 0x81C, 0xBA080603, + 0x81C, 0xB90A0603, + 0x81C, 0xB80C0603, + 0x81C, 0xB70E0603, + 0x81C, 0xB6100603, + 0x81C, 0xB5120603, + 0x81C, 0xB4140603, + 0x81C, 0xB3160603, + 0x81C, 0xB2180603, + 0x81C, 0xB11A0603, + 0x81C, 0xB01C0603, + 0x81C, 0x921E0603, + 0x81C, 0x91200603, + 0x81C, 0x90220603, + 0x81C, 0x8F240603, + 0x81C, 0x8E260603, + 0x81C, 0x8D280603, + 0x81C, 0x8C2A0603, + 0x81C, 0x8B2C0603, + 0x81C, 0x8A2E0603, + 0x81C, 0x89300603, + 0x81C, 0x88320603, + 0x81C, 0x6B340603, + 0x81C, 0x6A360603, + 0x81C, 0x69380603, + 0x81C, 0x683A0603, + 0x81C, 0x673C0603, + 0x81C, 0x663E0603, + 0x81C, 0x65400603, + 0x81C, 0x64420603, + 0x81C, 0x63440603, + 0x81C, 0x62460603, + 0x81C, 0x61480603, + 0x81C, 0x604A0603, + 0x81C, 0x424C0603, + 0x81C, 0x414E0603, + 0x81C, 0x40500603, + 0x81C, 0x06520603, + 0x81C, 0x05540603, + 0x81C, 0x04560603, + 0x81C, 0x03580603, + 0x81C, 0x025A0603, + 0x81C, 0x015C0603, + 0x81C, 0x005E0603, + 0x81C, 0x00600603, + 0x81C, 0x00620603, + 0x81C, 0x00640603, + 0x81C, 0x00660603, + 0x81C, 0x00680603, + 0x81C, 0x006A0603, + 0x81C, 0x006C0603, + 0x81C, 0x006E0603, + 0x81C, 0x00700603, + 0x81C, 0x00720603, + 0x81C, 0x00740603, + 0x81C, 0x00760603, + 0x81C, 0x00780603, + 0x81C, 0x007A0603, + 0x81C, 0x007C0603, + 0x81C, 0x007E0603, + 0x81C, 0x007E0603, + 0x90000008, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFB000603, + 0x81C, 0xFA020603, + 0x81C, 0xF9040603, + 0x81C, 0xF8060603, + 0x81C, 0xF7080603, + 0x81C, 0xF60A0603, + 0x81C, 0xF50C0603, + 0x81C, 0xF40E0603, + 0x81C, 0xF3100603, + 0x81C, 0xF2120603, + 0x81C, 0xF1140603, + 0x81C, 0xF0160603, + 0x81C, 0xEF180603, + 0x81C, 0xEE1A0603, + 0x81C, 0xED1C0603, + 0x81C, 0xEC1E0603, + 0x81C, 0xEB200603, + 0x81C, 0xEA220603, + 0x81C, 0xE9240603, + 0x81C, 0xE8260603, + 0x81C, 0xE7280603, + 0x81C, 0xE62A0603, + 0x81C, 0xE52C0603, + 0x81C, 0xE42E0603, + 0x81C, 0xE3300603, + 0x81C, 0xE2320603, + 0x81C, 0xC6340603, + 0x81C, 0xC5360603, + 0x81C, 0xC4380603, + 0x81C, 0xC33A0603, + 0x81C, 0xC23C0603, + 0x81C, 0xC13E0603, + 0x81C, 0xC0400603, + 0x81C, 0xA3420603, + 0x81C, 0xA2440603, + 0x81C, 0xA1460603, + 0x81C, 0xA0480603, + 0x81C, 0x824A0603, + 0x81C, 0x814C0603, + 0x81C, 0x804E0603, + 0x81C, 0x63500603, + 0x81C, 0x62520603, + 0x81C, 0x61540603, + 0x81C, 0x60560603, + 0x81C, 0x24580603, + 0x81C, 0x235A0603, + 0x81C, 0x225C0603, + 0x81C, 0x215E0603, + 0x81C, 0x20600603, + 0x81C, 0x03620603, + 0x81C, 0x02640603, + 0x81C, 0x01660603, + 0x81C, 0x00680603, + 0x81C, 0x006A0603, + 0x81C, 0x006C0603, + 0x81C, 0x006E0603, + 0x81C, 0x00700603, + 0x81C, 0x00720603, + 0x81C, 0x00740603, + 0x81C, 0x00760603, + 0x81C, 0x00780603, + 0x81C, 0x007A0603, + 0x81C, 0x007C0603, + 0x81C, 0x007E0603, + 0x81C, 0x007E0603, + 0x90000009, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF8000603, + 0x81C, 0xF7020603, + 0x81C, 0xF6040603, + 0x81C, 0xF5060603, + 0x81C, 0xF4080603, + 0x81C, 0xF30A0603, + 0x81C, 0xF20C0603, + 0x81C, 0xF10E0603, + 0x81C, 0xF0100603, + 0x81C, 0xEF120603, + 0x81C, 0xEE140603, + 0x81C, 0xED160603, + 0x81C, 0xEC180603, + 0x81C, 0xEB1A0603, + 0x81C, 0xEA1C0603, + 0x81C, 0xE91E0603, + 0x81C, 0xE8200603, + 0x81C, 0xE7220603, + 0x81C, 0xE6240603, + 0x81C, 0xE5260603, + 0x81C, 0xE4280603, + 0x81C, 0xE32A0603, + 0x81C, 0xC42C0603, + 0x81C, 0xC32E0603, + 0x81C, 0xC2300603, + 0x81C, 0xC1320603, + 0x81C, 0xA3340603, + 0x81C, 0xA2360603, + 0x81C, 0xA1380603, + 0x81C, 0xA03A0603, + 0x81C, 0x823C0603, + 0x81C, 0x813E0603, + 0x81C, 0x80400603, + 0x81C, 0x65420603, + 0x81C, 0x64440603, + 0x81C, 0x63460603, + 0x81C, 0x62480603, + 0x81C, 0x614A0603, + 0x81C, 0x424C0603, + 0x81C, 0x414E0603, + 0x81C, 0x40500603, + 0x81C, 0x22520603, + 0x81C, 0x21540603, + 0x81C, 0x20560603, + 0x81C, 0x04580603, + 0x81C, 0x035A0603, + 0x81C, 0x025C0603, + 0x81C, 0x015E0603, + 0x81C, 0x00600603, + 0x81C, 0x00620603, + 0x81C, 0x00640603, + 0x81C, 0x00660603, + 0x81C, 0x00680603, + 0x81C, 0x006A0603, + 0x81C, 0x006C0603, + 0x81C, 0x006E0603, + 0x81C, 0x00700603, + 0x81C, 0x00720603, + 0x81C, 0x00740603, + 0x81C, 0x00760603, + 0x81C, 0x00780603, + 0x81C, 0x007A0603, + 0x81C, 0x007C0603, + 0x81C, 0x007E0603, + 0x81C, 0x007E0603, + 0x9000000a, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000603, + 0x81C, 0xFB020603, + 0x81C, 0xFA040603, + 0x81C, 0xF9060603, + 0x81C, 0xF8080603, + 0x81C, 0xF70A0603, + 0x81C, 0xF60C0603, + 0x81C, 0xF50E0603, + 0x81C, 0xF4100603, + 0x81C, 0xF3120603, + 0x81C, 0xF2140603, + 0x81C, 0xF1160603, + 0x81C, 0xF0180603, + 0x81C, 0xEE1A0603, + 0x81C, 0xED1C0603, + 0x81C, 0xEC1E0603, + 0x81C, 0xEB200603, + 0x81C, 0xEA220603, + 0x81C, 0xE9240603, + 0x81C, 0xE8260603, + 0x81C, 0xE7280603, + 0x81C, 0xE62A0603, + 0x81C, 0xE52C0603, + 0x81C, 0xE42E0603, + 0x81C, 0xE3300603, + 0x81C, 0xE2320603, + 0x81C, 0xC6340603, + 0x81C, 0xC5360603, + 0x81C, 0xC4380603, + 0x81C, 0xC33A0603, + 0x81C, 0xA63C0603, + 0x81C, 0xA53E0603, + 0x81C, 0xA4400603, + 0x81C, 0xA3420603, + 0x81C, 0xA2440603, + 0x81C, 0xA1460603, + 0x81C, 0x83480603, + 0x81C, 0x824A0603, + 0x81C, 0x814C0603, + 0x81C, 0x804E0603, + 0x81C, 0x63500603, + 0x81C, 0x62520603, + 0x81C, 0x61540603, + 0x81C, 0x42560603, + 0x81C, 0x41580603, + 0x81C, 0x405A0603, + 0x81C, 0x225C0603, + 0x81C, 0x215E0603, + 0x81C, 0x20600603, + 0x81C, 0x04620603, + 0x81C, 0x03640603, + 0x81C, 0x02660603, + 0x81C, 0x01680603, + 0x81C, 0x006A0603, + 0x81C, 0x006C0603, + 0x81C, 0x006E0603, + 0x81C, 0x00700603, + 0x81C, 0x00720603, + 0x81C, 0x00740603, + 0x81C, 0x00760603, + 0x81C, 0x00780603, + 0x81C, 0x007A0603, + 0x81C, 0x007C0603, + 0x81C, 0x007E0603, + 0x81C, 0x007E0603, + 0x9000000b, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF9000603, + 0x81C, 0xF8020603, + 0x81C, 0xF7040603, + 0x81C, 0xF6060603, + 0x81C, 0xF5080603, + 0x81C, 0xF40A0603, + 0x81C, 0xF30C0603, + 0x81C, 0xF20E0603, + 0x81C, 0xF1100603, + 0x81C, 0xF0120603, + 0x81C, 0xEF140603, + 0x81C, 0xEE160603, + 0x81C, 0xED180603, + 0x81C, 0xEC1A0603, + 0x81C, 0xEB1C0603, + 0x81C, 0xEA1E0603, + 0x81C, 0xE9200603, + 0x81C, 0xE8220603, + 0x81C, 0xE7240603, + 0x81C, 0xE6260603, + 0x81C, 0xE5280603, + 0x81C, 0xE42A0603, + 0x81C, 0xC42C0603, + 0x81C, 0xC32E0603, + 0x81C, 0xC2300603, + 0x81C, 0xC1320603, + 0x81C, 0xA3340603, + 0x81C, 0xA2360603, + 0x81C, 0xA1380603, + 0x81C, 0xA03A0603, + 0x81C, 0x823C0603, + 0x81C, 0x813E0603, + 0x81C, 0x80400603, + 0x81C, 0x64420603, + 0x81C, 0x63440603, + 0x81C, 0x62460603, + 0x81C, 0x61480603, + 0x81C, 0x604A0603, + 0x81C, 0x244C0603, + 0x81C, 0x234E0603, + 0x81C, 0x22500603, + 0x81C, 0x21520603, + 0x81C, 0x20540603, + 0x81C, 0x05560603, + 0x81C, 0x04580603, + 0x81C, 0x035A0603, + 0x81C, 0x025C0603, + 0x81C, 0x015E0603, + 0x81C, 0x00600603, + 0x81C, 0x00620603, + 0x81C, 0x00640603, + 0x81C, 0x00660603, + 0x81C, 0x00680603, + 0x81C, 0x006A0603, + 0x81C, 0x006C0603, + 0x81C, 0x006E0603, + 0x81C, 0x00700603, + 0x81C, 0x00720603, + 0x81C, 0x00740603, + 0x81C, 0x00760603, + 0x81C, 0x00780603, + 0x81C, 0x007A0603, + 0x81C, 0x007C0603, + 0x81C, 0x007E0603, + 0x81C, 0x007E0603, + 0x9000000c, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000603, + 0x81C, 0xFB020603, + 0x81C, 0xFA040603, + 0x81C, 0xF9060603, + 0x81C, 0xF8080603, + 0x81C, 0xF70A0603, + 0x81C, 0xF60C0603, + 0x81C, 0xF50E0603, + 0x81C, 0xF4100603, + 0x81C, 0xF3120603, + 0x81C, 0xF2140603, + 0x81C, 0xF1160603, + 0x81C, 0xF0180603, + 0x81C, 0xEF1A0603, + 0x81C, 0xEE1C0603, + 0x81C, 0xED1E0603, + 0x81C, 0xEC200603, + 0x81C, 0xEB220603, + 0x81C, 0xEA240603, + 0x81C, 0xE9260603, + 0x81C, 0xE8280603, + 0x81C, 0xE72A0603, + 0x81C, 0xE62C0603, + 0x81C, 0xE52E0603, + 0x81C, 0xE4300603, + 0x81C, 0xE3320603, + 0x81C, 0xE2340603, + 0x81C, 0xC6360603, + 0x81C, 0xC5380603, + 0x81C, 0xC43A0603, + 0x81C, 0xC33C0603, + 0x81C, 0xA63E0603, + 0x81C, 0xA5400603, + 0x81C, 0xA4420603, + 0x81C, 0xA3440603, + 0x81C, 0xA2460603, + 0x81C, 0xA1480603, + 0x81C, 0x834A0603, + 0x81C, 0x824C0603, + 0x81C, 0x814E0603, + 0x81C, 0x64500603, + 0x81C, 0x63520603, + 0x81C, 0x62540603, + 0x81C, 0x61560603, + 0x81C, 0x60580603, + 0x81C, 0x405A0603, + 0x81C, 0x215C0603, + 0x81C, 0x205E0603, + 0x81C, 0x03600603, + 0x81C, 0x02620603, + 0x81C, 0x01640603, + 0x81C, 0x00660603, + 0x81C, 0x00680603, + 0x81C, 0x006A0603, + 0x81C, 0x006C0603, + 0x81C, 0x006E0603, + 0x81C, 0x00700603, + 0x81C, 0x00720603, + 0x81C, 0x00740603, + 0x81C, 0x00760603, + 0x81C, 0x00780603, + 0x81C, 0x007A0603, + 0x81C, 0x007C0603, + 0x81C, 0x007E0603, + 0x81C, 0x007E0603, + 0x9000000d, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000603, + 0x81C, 0xFB020603, + 0x81C, 0xFA040603, + 0x81C, 0xF9060603, + 0x81C, 0xF8080603, + 0x81C, 0xF70A0603, + 0x81C, 0xF60C0603, + 0x81C, 0xF50E0603, + 0x81C, 0xF4100603, + 0x81C, 0xF3120603, + 0x81C, 0xF2140603, + 0x81C, 0xF1160603, + 0x81C, 0xF0180603, + 0x81C, 0xEE1A0603, + 0x81C, 0xED1C0603, + 0x81C, 0xEC1E0603, + 0x81C, 0xEB200603, + 0x81C, 0xEA220603, + 0x81C, 0xE9240603, + 0x81C, 0xE8260603, + 0x81C, 0xE7280603, + 0x81C, 0xE62A0603, + 0x81C, 0xE52C0603, + 0x81C, 0xE42E0603, + 0x81C, 0xE3300603, + 0x81C, 0xE2320603, + 0x81C, 0xC6340603, + 0x81C, 0xC5360603, + 0x81C, 0xC4380603, + 0x81C, 0xC33A0603, + 0x81C, 0xA63C0603, + 0x81C, 0xA53E0603, + 0x81C, 0xA4400603, + 0x81C, 0xA3420603, + 0x81C, 0xA2440603, + 0x81C, 0xA1460603, + 0x81C, 0x83480603, + 0x81C, 0x824A0603, + 0x81C, 0x814C0603, + 0x81C, 0x804E0603, + 0x81C, 0x63500603, + 0x81C, 0x62520603, + 0x81C, 0x61540603, + 0x81C, 0x42560603, + 0x81C, 0x41580603, + 0x81C, 0x405A0603, + 0x81C, 0x225C0603, + 0x81C, 0x215E0603, + 0x81C, 0x20600603, + 0x81C, 0x04620603, + 0x81C, 0x03640603, + 0x81C, 0x02660603, + 0x81C, 0x01680603, + 0x81C, 0x006A0603, + 0x81C, 0x006C0603, + 0x81C, 0x006E0603, + 0x81C, 0x00700603, + 0x81C, 0x00720603, + 0x81C, 0x00740603, + 0x81C, 0x00760603, + 0x81C, 0x00780603, + 0x81C, 0x007A0603, + 0x81C, 0x007C0603, + 0x81C, 0x007E0603, + 0x81C, 0x007E0603, + 0x9000000e, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000603, + 0x81C, 0xFB020603, + 0x81C, 0xFA040603, + 0x81C, 0xF9060603, + 0x81C, 0xF8080603, + 0x81C, 0xF70A0603, + 0x81C, 0xF60C0603, + 0x81C, 0xF50E0603, + 0x81C, 0xF4100603, + 0x81C, 0xF3120603, + 0x81C, 0xF2140603, + 0x81C, 0xF1160603, + 0x81C, 0xF0180603, + 0x81C, 0xEE1A0603, + 0x81C, 0xED1C0603, + 0x81C, 0xEC1E0603, + 0x81C, 0xEB200603, + 0x81C, 0xEA220603, + 0x81C, 0xE9240603, + 0x81C, 0xE8260603, + 0x81C, 0xE7280603, + 0x81C, 0xE62A0603, + 0x81C, 0xE52C0603, + 0x81C, 0xE42E0603, + 0x81C, 0xE3300603, + 0x81C, 0xE2320603, + 0x81C, 0xC6340603, + 0x81C, 0xC5360603, + 0x81C, 0xC4380603, + 0x81C, 0xC33A0603, + 0x81C, 0xA63C0603, + 0x81C, 0xA53E0603, + 0x81C, 0xA4400603, + 0x81C, 0xA3420603, + 0x81C, 0xA2440603, + 0x81C, 0xA1460603, + 0x81C, 0x83480603, + 0x81C, 0x824A0603, + 0x81C, 0x814C0603, + 0x81C, 0x804E0603, + 0x81C, 0x63500603, + 0x81C, 0x62520603, + 0x81C, 0x61540603, + 0x81C, 0x42560603, + 0x81C, 0x41580603, + 0x81C, 0x405A0603, + 0x81C, 0x225C0603, + 0x81C, 0x215E0603, + 0x81C, 0x20600603, + 0x81C, 0x04620603, + 0x81C, 0x03640603, + 0x81C, 0x02660603, + 0x81C, 0x01680603, + 0x81C, 0x006A0603, + 0x81C, 0x006C0603, + 0x81C, 0x006E0603, + 0x81C, 0x00700603, + 0x81C, 0x00720603, + 0x81C, 0x00740603, + 0x81C, 0x00760603, + 0x81C, 0x00780603, + 0x81C, 0x007A0603, + 0x81C, 0x007C0603, + 0x81C, 0x007E0603, + 0x81C, 0x007E0603, + 0x9000000f, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xBF000603, + 0x81C, 0xBF020603, + 0x81C, 0xBF040603, + 0x81C, 0xBF060603, + 0x81C, 0xBF080603, + 0x81C, 0xBE0A0603, + 0x81C, 0xBD0C0603, + 0x81C, 0xBC0E0603, + 0x81C, 0xBB100603, + 0x81C, 0xBA120603, + 0x81C, 0xB9140603, + 0x81C, 0xB8160603, + 0x81C, 0xB7180603, + 0x81C, 0xB61A0603, + 0x81C, 0xB51C0603, + 0x81C, 0xB41E0603, + 0x81C, 0xB1200603, + 0x81C, 0xB2220603, + 0x81C, 0xB1240603, + 0x81C, 0xB0260603, + 0x81C, 0xAF280603, + 0x81C, 0xAE2A0603, + 0x81C, 0xAD2C0603, + 0x81C, 0xAC2E0603, + 0x81C, 0xAB300603, + 0x81C, 0xAA320603, + 0x81C, 0xC6340603, + 0x81C, 0xC5360603, + 0x81C, 0xC4380603, + 0x81C, 0xC33A0603, + 0x81C, 0x883C0603, + 0x81C, 0x873E0603, + 0x81C, 0x86400603, + 0x81C, 0x85420603, + 0x81C, 0x84440603, + 0x81C, 0x83460603, + 0x81C, 0x67480603, + 0x81C, 0x664A0603, + 0x81C, 0x654C0603, + 0x81C, 0x644E0603, + 0x81C, 0x27500603, + 0x81C, 0x26520603, + 0x81C, 0x25540603, + 0x81C, 0x24560603, + 0x81C, 0x23580603, + 0x81C, 0x225A0603, + 0x81C, 0x215C0603, + 0x81C, 0x205E0603, + 0x81C, 0x03600603, + 0x81C, 0x02620603, + 0x81C, 0x01640603, + 0x81C, 0x00660603, + 0x81C, 0x00680603, + 0x81C, 0x006A0603, + 0x81C, 0x006C0603, + 0x81C, 0x006E0603, + 0x81C, 0x00700603, + 0x81C, 0x00720603, + 0x81C, 0x00740603, + 0x81C, 0x00760603, + 0x81C, 0x00780603, + 0x81C, 0x007A0603, + 0x81C, 0x007C0603, + 0x81C, 0x007E0603, + 0x81C, 0x007E0603, + 0x90000010, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000403, + 0x81C, 0xFB000603, + 0x81C, 0xFA020603, + 0x81C, 0xF9040603, + 0x81C, 0xF8060603, + 0x81C, 0xF7080603, + 0x81C, 0xF60A0603, + 0x81C, 0xF50C0603, + 0x81C, 0xF40E0603, + 0x81C, 0xF3100603, + 0x81C, 0xF2120603, + 0x81C, 0xF1140603, + 0x81C, 0xF0160603, + 0x81C, 0xEF180603, + 0x81C, 0xEE1A0603, + 0x81C, 0xED1C0603, + 0x81C, 0xEC1E0603, + 0x81C, 0xEB200603, + 0x81C, 0xEA220603, + 0x81C, 0xE9240603, + 0x81C, 0xE8260603, + 0x81C, 0xE7280603, + 0x81C, 0xE62A0603, + 0x81C, 0xE52C0603, + 0x81C, 0xE42E0603, + 0x81C, 0xE3300603, + 0x81C, 0xE2320603, + 0x81C, 0xC6340603, + 0x81C, 0xC5360603, + 0x81C, 0xC4380603, + 0x81C, 0xC33A0603, + 0x81C, 0xA63C0603, + 0x81C, 0xA53E0603, + 0x81C, 0xA4400603, + 0x81C, 0xA3420603, + 0x81C, 0xA2440603, + 0x81C, 0xA1460603, + 0x81C, 0x83480603, + 0x81C, 0x824A0603, + 0x81C, 0x814C0603, + 0x81C, 0x644E0603, + 0x81C, 0x63500603, + 0x81C, 0x62520603, + 0x81C, 0x61540603, + 0x81C, 0x60560603, + 0x81C, 0x40580603, + 0x81C, 0x215A0603, + 0x81C, 0x205C0603, + 0x81C, 0x035E0603, + 0x81C, 0x02600603, + 0x81C, 0x01620603, + 0x81C, 0x00640603, + 0x81C, 0x00660603, + 0x81C, 0x00680603, + 0x81C, 0x006A0603, + 0x81C, 0x006C0603, + 0x81C, 0x006E0603, + 0x81C, 0x00700603, + 0x81C, 0x00720603, + 0x81C, 0x00740603, + 0x81C, 0x00760603, + 0x81C, 0x00780603, + 0x81C, 0x007A0603, + 0x81C, 0x007C0603, + 0x81C, 0x007E0603, + 0x90000012, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF7000603, + 0x81C, 0xF6020603, + 0x81C, 0xF5040603, + 0x81C, 0xF4060603, + 0x81C, 0xF3080603, + 0x81C, 0xF20A0603, + 0x81C, 0xF10C0603, + 0x81C, 0xF00E0603, + 0x81C, 0xEF100603, + 0x81C, 0xEE120603, + 0x81C, 0xED140603, + 0x81C, 0xEC160603, + 0x81C, 0xEB180603, + 0x81C, 0xEA1A0603, + 0x81C, 0xE91C0603, + 0x81C, 0xE81E0603, + 0x81C, 0xE7200603, + 0x81C, 0xE6220603, + 0x81C, 0xE5240603, + 0x81C, 0xE4260603, + 0x81C, 0xE3280603, + 0x81C, 0xC42A0603, + 0x81C, 0xC32C0603, + 0x81C, 0xC22E0603, + 0x81C, 0xC1300603, + 0x81C, 0xC0320603, + 0x81C, 0xA3340603, + 0x81C, 0xA2360603, + 0x81C, 0xA1380603, + 0x81C, 0xA03A0603, + 0x81C, 0x823C0603, + 0x81C, 0x813E0603, + 0x81C, 0x80400603, + 0x81C, 0x64420603, + 0x81C, 0x63440603, + 0x81C, 0x62460603, + 0x81C, 0x61480603, + 0x81C, 0x604A0603, + 0x81C, 0x414C0603, + 0x81C, 0x404E0603, + 0x81C, 0x22500603, + 0x81C, 0x21520603, + 0x81C, 0x20540603, + 0x81C, 0x03560603, + 0x81C, 0x02580603, + 0x81C, 0x015A0603, + 0x81C, 0x005C0603, + 0x81C, 0x005E0603, + 0x81C, 0x00600603, + 0x81C, 0x00620603, + 0x81C, 0x00640603, + 0x81C, 0x00660603, + 0x81C, 0x00680603, + 0x81C, 0x006A0603, + 0x81C, 0x006C0603, + 0x81C, 0x006E0603, + 0x81C, 0x00700603, + 0x81C, 0x00720603, + 0x81C, 0x00740603, + 0x81C, 0x00760603, + 0x81C, 0x00780603, + 0x81C, 0x007A0603, + 0x81C, 0x007C0603, + 0x81C, 0x007E0603, + 0x81C, 0x007E0603, + 0xA0000000, 0x00000000, + 0x81C, 0xFD000603, + 0x81C, 0xFC020603, + 0x81C, 0xFB040603, + 0x81C, 0xFA060603, + 0x81C, 0xF9080603, + 0x81C, 0xF80A0603, + 0x81C, 0xF70C0603, + 0x81C, 0xF60E0603, + 0x81C, 0xF5100603, + 0x81C, 0xF4120603, + 0x81C, 0xF3140603, + 0x81C, 0xF2160603, + 0x81C, 0xF1180603, + 0x81C, 0xF01A0603, + 0x81C, 0xEF1C0603, + 0x81C, 0xEE1E0603, + 0x81C, 0xED200603, + 0x81C, 0xEC220603, + 0x81C, 0xEB240603, + 0x81C, 0xEA260603, + 0x81C, 0xE9280603, + 0x81C, 0xE82A0603, + 0x81C, 0xE72C0603, + 0x81C, 0xE62E0603, + 0x81C, 0xE5300603, + 0x81C, 0xE4320603, + 0x81C, 0xE3340603, + 0x81C, 0xC6360603, + 0x81C, 0xC5380603, + 0x81C, 0xC43A0603, + 0x81C, 0xC33C0603, + 0x81C, 0xA63E0603, + 0x81C, 0xA5400603, + 0x81C, 0xA4420603, + 0x81C, 0xA3440603, + 0x81C, 0xA2460603, + 0x81C, 0xA1480603, + 0x81C, 0x834A0603, + 0x81C, 0x824C0603, + 0x81C, 0x814E0603, + 0x81C, 0x64500603, + 0x81C, 0x63520603, + 0x81C, 0x62540603, + 0x81C, 0x61560603, + 0x81C, 0x60580603, + 0x81C, 0x235A0603, + 0x81C, 0x225C0603, + 0x81C, 0x215E0603, + 0x81C, 0x20600603, + 0x81C, 0x03620603, + 0x81C, 0x02640603, + 0x81C, 0x01660603, + 0x81C, 0x00680603, + 0x81C, 0x006A0603, + 0x81C, 0x006C0603, + 0x81C, 0x006E0603, + 0x81C, 0x00700603, + 0x81C, 0x00720603, + 0x81C, 0x00740603, + 0x81C, 0x00760603, + 0x81C, 0x00780603, + 0x81C, 0x007A0603, + 0x81C, 0x007C0603, + 0x81C, 0x007E0603, + 0x81C, 0x007E0603, + 0xB0000000, 0x00000000, + 0x80000000, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000703, + 0x81C, 0xFB020703, + 0x81C, 0xFA040703, + 0x81C, 0xF9060703, + 0x81C, 0xF8080703, + 0x81C, 0xF70A0703, + 0x81C, 0xF60C0703, + 0x81C, 0xF50E0703, + 0x81C, 0xF4100703, + 0x81C, 0xF3120703, + 0x81C, 0xF2140703, + 0x81C, 0xF1160703, + 0x81C, 0xEF180703, + 0x81C, 0xEE1A0703, + 0x81C, 0xED1C0703, + 0x81C, 0xEC1E0703, + 0x81C, 0xEB200703, + 0x81C, 0xEA220703, + 0x81C, 0xE9240703, + 0x81C, 0xE8260703, + 0x81C, 0xE7280703, + 0x81C, 0xE62A0703, + 0x81C, 0xE52C0703, + 0x81C, 0xE42E0703, + 0x81C, 0xE3300703, + 0x81C, 0xE2320703, + 0x81C, 0xC6340703, + 0x81C, 0xC5360703, + 0x81C, 0xC4380703, + 0x81C, 0xC33A0703, + 0x81C, 0xA63C0703, + 0x81C, 0xA53E0703, + 0x81C, 0xA4400703, + 0x81C, 0xA3420703, + 0x81C, 0xA2440703, + 0x81C, 0xA1460703, + 0x81C, 0x83480703, + 0x81C, 0x824A0703, + 0x81C, 0x814C0703, + 0x81C, 0x804E0703, + 0x81C, 0x63500703, + 0x81C, 0x62520703, + 0x81C, 0x61540703, + 0x81C, 0x42560703, + 0x81C, 0x41580703, + 0x81C, 0x405A0703, + 0x81C, 0x225C0703, + 0x81C, 0x215E0703, + 0x81C, 0x20600703, + 0x81C, 0x04620703, + 0x81C, 0x03640703, + 0x81C, 0x02660703, + 0x81C, 0x01680703, + 0x81C, 0x006A0703, + 0x81C, 0x006C0703, + 0x81C, 0x006E0703, + 0x81C, 0x00700703, + 0x81C, 0x00720703, + 0x81C, 0x00740703, + 0x81C, 0x00760703, + 0x81C, 0x00780703, + 0x81C, 0x007A0703, + 0x81C, 0x007C0703, + 0x81C, 0x007E0703, + 0x81C, 0x007E0703, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xBD000703, + 0x81C, 0xBC020703, + 0x81C, 0xBB040703, + 0x81C, 0xBA060703, + 0x81C, 0xB9080703, + 0x81C, 0xB80A0703, + 0x81C, 0xB70C0703, + 0x81C, 0xB60E0703, + 0x81C, 0xB5100703, + 0x81C, 0xB4120703, + 0x81C, 0xB3140703, + 0x81C, 0xB2160703, + 0x81C, 0xB1180703, + 0x81C, 0xB01A0703, + 0x81C, 0xAF1C0703, + 0x81C, 0xAE1E0703, + 0x81C, 0xAD200703, + 0x81C, 0xAC220703, + 0x81C, 0x8E240703, + 0x81C, 0x8D260703, + 0x81C, 0x8C280703, + 0x81C, 0x6F2A0703, + 0x81C, 0x6E2C0703, + 0x81C, 0x6D2E0703, + 0x81C, 0x6C300703, + 0x81C, 0x6B320703, + 0x81C, 0x6A340703, + 0x81C, 0x69360703, + 0x81C, 0x68380703, + 0x81C, 0x673A0703, + 0x81C, 0x663C0703, + 0x81C, 0x653E0703, + 0x81C, 0x64400703, + 0x81C, 0x63420703, + 0x81C, 0x62440703, + 0x81C, 0x61460703, + 0x81C, 0x60480703, + 0x81C, 0x424A0703, + 0x81C, 0x414C0703, + 0x81C, 0x404E0703, + 0x81C, 0x06500703, + 0x81C, 0x05520703, + 0x81C, 0x04540703, + 0x81C, 0x03560703, + 0x81C, 0x02580703, + 0x81C, 0x015A0703, + 0x81C, 0x005C0703, + 0x81C, 0x005E0703, + 0x81C, 0x00600703, + 0x81C, 0x00620703, + 0x81C, 0x00640703, + 0x81C, 0x00660703, + 0x81C, 0x00680703, + 0x81C, 0x006A0703, + 0x81C, 0x006C0703, + 0x81C, 0x006E0703, + 0x81C, 0x00700703, + 0x81C, 0x00720703, + 0x81C, 0x00740703, + 0x81C, 0x00760703, + 0x81C, 0x00780703, + 0x81C, 0x007A0703, + 0x81C, 0x007C0703, + 0x81C, 0x007E0703, + 0x81C, 0x007C0703, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF7000703, + 0x81C, 0xF6020703, + 0x81C, 0xF5040703, + 0x81C, 0xF4060703, + 0x81C, 0xF3080703, + 0x81C, 0xF20A0703, + 0x81C, 0xF10C0703, + 0x81C, 0xF00E0703, + 0x81C, 0xEF100703, + 0x81C, 0xEE120703, + 0x81C, 0xED140703, + 0x81C, 0xEC160703, + 0x81C, 0xEB180703, + 0x81C, 0xEA1A0703, + 0x81C, 0xE91C0703, + 0x81C, 0xCA1E0703, + 0x81C, 0xC9200703, + 0x81C, 0xC8220703, + 0x81C, 0xC7240703, + 0x81C, 0xC6260703, + 0x81C, 0xC5280703, + 0x81C, 0xC42A0703, + 0x81C, 0xC32C0703, + 0x81C, 0xC22E0703, + 0x81C, 0xC1300703, + 0x81C, 0xA4320703, + 0x81C, 0xA3340703, + 0x81C, 0xA2360703, + 0x81C, 0xA1380703, + 0x81C, 0xA03A0703, + 0x81C, 0x823C0703, + 0x81C, 0x813E0703, + 0x81C, 0x80400703, + 0x81C, 0x64420703, + 0x81C, 0x63440703, + 0x81C, 0x62460703, + 0x81C, 0x61480703, + 0x81C, 0x604A0703, + 0x81C, 0x414C0703, + 0x81C, 0x404E0703, + 0x81C, 0x22500703, + 0x81C, 0x21520703, + 0x81C, 0x20540703, + 0x81C, 0x03560703, + 0x81C, 0x02580703, + 0x81C, 0x015A0703, + 0x81C, 0x005C0703, + 0x81C, 0x005E0703, + 0x81C, 0x00600703, + 0x81C, 0x00620703, + 0x81C, 0x00640703, + 0x81C, 0x00660703, + 0x81C, 0x00680703, + 0x81C, 0x006A0703, + 0x81C, 0x006C0703, + 0x81C, 0x006E0703, + 0x81C, 0x00700703, + 0x81C, 0x00720703, + 0x81C, 0x00740703, + 0x81C, 0x00760703, + 0x81C, 0x00780703, + 0x81C, 0x007A0703, + 0x81C, 0x007C0703, + 0x81C, 0x007E0703, + 0x81C, 0x007E0703, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000703, + 0x81C, 0xFB020703, + 0x81C, 0xFA040703, + 0x81C, 0xF9060703, + 0x81C, 0xF8080703, + 0x81C, 0xF70A0703, + 0x81C, 0xF60C0703, + 0x81C, 0xF50E0703, + 0x81C, 0xF4100703, + 0x81C, 0xF3120703, + 0x81C, 0xF2140703, + 0x81C, 0xF1160703, + 0x81C, 0xF0180703, + 0x81C, 0xEF1A0703, + 0x81C, 0xEE1C0703, + 0x81C, 0xED1E0703, + 0x81C, 0xEC200703, + 0x81C, 0xEB220703, + 0x81C, 0xEA240703, + 0x81C, 0xE9260703, + 0x81C, 0xE8280703, + 0x81C, 0xE72A0703, + 0x81C, 0xE62C0703, + 0x81C, 0xE52E0703, + 0x81C, 0xE4300703, + 0x81C, 0xE3320703, + 0x81C, 0xE2340703, + 0x81C, 0xC6360703, + 0x81C, 0xC5380703, + 0x81C, 0xC43A0703, + 0x81C, 0xC33C0703, + 0x81C, 0xA63E0703, + 0x81C, 0xA5400703, + 0x81C, 0xA4420703, + 0x81C, 0xA3440703, + 0x81C, 0xA2460703, + 0x81C, 0x84480703, + 0x81C, 0x834A0703, + 0x81C, 0x824C0703, + 0x81C, 0x814E0703, + 0x81C, 0x80500703, + 0x81C, 0x63520703, + 0x81C, 0x62540703, + 0x81C, 0x61560703, + 0x81C, 0x60580703, + 0x81C, 0x225A0703, + 0x81C, 0x055C0703, + 0x81C, 0x045E0703, + 0x81C, 0x03600703, + 0x81C, 0x02620703, + 0x81C, 0x01640703, + 0x81C, 0x00660703, + 0x81C, 0x00680703, + 0x81C, 0x006A0703, + 0x81C, 0x006C0703, + 0x81C, 0x006E0703, + 0x81C, 0x00700703, + 0x81C, 0x00720703, + 0x81C, 0x00740703, + 0x81C, 0x00760703, + 0x81C, 0x00780703, + 0x81C, 0x007A0703, + 0x81C, 0x007C0703, + 0x81C, 0x007E0703, + 0x81C, 0x007E0703, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF7000703, + 0x81C, 0xF6020703, + 0x81C, 0xF5040703, + 0x81C, 0xF4060703, + 0x81C, 0xF3080703, + 0x81C, 0xF20A0703, + 0x81C, 0xF10C0703, + 0x81C, 0xF00E0703, + 0x81C, 0xEF100703, + 0x81C, 0xEE120703, + 0x81C, 0xED140703, + 0x81C, 0xEC160703, + 0x81C, 0xEB180703, + 0x81C, 0xEA1A0703, + 0x81C, 0xE91C0703, + 0x81C, 0xCA1E0703, + 0x81C, 0xC9200703, + 0x81C, 0xC8220703, + 0x81C, 0xC7240703, + 0x81C, 0xC6260703, + 0x81C, 0xC5280703, + 0x81C, 0xC42A0703, + 0x81C, 0xC32C0703, + 0x81C, 0xC22E0703, + 0x81C, 0xC1300703, + 0x81C, 0xA4320703, + 0x81C, 0xA3340703, + 0x81C, 0xA2360703, + 0x81C, 0xA1380703, + 0x81C, 0xA03A0703, + 0x81C, 0x823C0703, + 0x81C, 0x813E0703, + 0x81C, 0x80400703, + 0x81C, 0x64420703, + 0x81C, 0x63440703, + 0x81C, 0x62460703, + 0x81C, 0x61480703, + 0x81C, 0x604A0703, + 0x81C, 0x414C0703, + 0x81C, 0x404E0703, + 0x81C, 0x22500703, + 0x81C, 0x21520703, + 0x81C, 0x20540703, + 0x81C, 0x03560703, + 0x81C, 0x02580703, + 0x81C, 0x015A0703, + 0x81C, 0x005C0703, + 0x81C, 0x005E0703, + 0x81C, 0x00600703, + 0x81C, 0x00620703, + 0x81C, 0x00640703, + 0x81C, 0x00660703, + 0x81C, 0x00680703, + 0x81C, 0x006A0703, + 0x81C, 0x006C0703, + 0x81C, 0x006E0703, + 0x81C, 0x00700703, + 0x81C, 0x00720703, + 0x81C, 0x00740703, + 0x81C, 0x00760703, + 0x81C, 0x00780703, + 0x81C, 0x007A0703, + 0x81C, 0x007C0703, + 0x81C, 0x007E0703, + 0x81C, 0x007E0703, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFB000703, + 0x81C, 0xFA020703, + 0x81C, 0xF9040703, + 0x81C, 0xF8060703, + 0x81C, 0xF7080703, + 0x81C, 0xF60A0703, + 0x81C, 0xF50C0703, + 0x81C, 0xF40E0703, + 0x81C, 0xF3100703, + 0x81C, 0xF2120703, + 0x81C, 0xF1140703, + 0x81C, 0xF0160703, + 0x81C, 0xEF180703, + 0x81C, 0xEE1A0703, + 0x81C, 0xED1C0703, + 0x81C, 0xEC1E0703, + 0x81C, 0xEB200703, + 0x81C, 0xEA220703, + 0x81C, 0xE9240703, + 0x81C, 0xE8260703, + 0x81C, 0xE7280703, + 0x81C, 0xE62A0703, + 0x81C, 0xE52C0703, + 0x81C, 0xE42E0703, + 0x81C, 0xE3300703, + 0x81C, 0xE2320703, + 0x81C, 0xE1340703, + 0x81C, 0xC5360703, + 0x81C, 0xC4380703, + 0x81C, 0xC33A0703, + 0x81C, 0xC23C0703, + 0x81C, 0xC13E0703, + 0x81C, 0xA4400703, + 0x81C, 0xA3420703, + 0x81C, 0xA2440703, + 0x81C, 0xA1460703, + 0x81C, 0x83480703, + 0x81C, 0x824A0703, + 0x81C, 0x814C0703, + 0x81C, 0x804E0703, + 0x81C, 0x64500703, + 0x81C, 0x63520703, + 0x81C, 0x62540703, + 0x81C, 0x61560703, + 0x81C, 0x60580703, + 0x81C, 0x235A0703, + 0x81C, 0x225C0703, + 0x81C, 0x215E0703, + 0x81C, 0x20600703, + 0x81C, 0x04620703, + 0x81C, 0x03640703, + 0x81C, 0x02660703, + 0x81C, 0x01680703, + 0x81C, 0x006A0703, + 0x81C, 0x006C0703, + 0x81C, 0x006E0703, + 0x81C, 0x00700703, + 0x81C, 0x00720703, + 0x81C, 0x00740703, + 0x81C, 0x00760703, + 0x81C, 0x00780703, + 0x81C, 0x007A0703, + 0x81C, 0x007C0703, + 0x81C, 0x007E0703, + 0x81C, 0x007E0703, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF9000703, + 0x81C, 0xF8020703, + 0x81C, 0xF7040703, + 0x81C, 0xF6060703, + 0x81C, 0xF5080703, + 0x81C, 0xF40A0703, + 0x81C, 0xF30C0703, + 0x81C, 0xF20E0703, + 0x81C, 0xF1100703, + 0x81C, 0xF0120703, + 0x81C, 0xEF140703, + 0x81C, 0xEE160703, + 0x81C, 0xED180703, + 0x81C, 0xEC1A0703, + 0x81C, 0xEB1C0703, + 0x81C, 0xEA1E0703, + 0x81C, 0xC9200703, + 0x81C, 0xC8220703, + 0x81C, 0xC7240703, + 0x81C, 0xC6260703, + 0x81C, 0xC5280703, + 0x81C, 0xC42A0703, + 0x81C, 0xC32C0703, + 0x81C, 0xC22E0703, + 0x81C, 0xC1300703, + 0x81C, 0xC0320703, + 0x81C, 0xA3340703, + 0x81C, 0xA2360703, + 0x81C, 0xA1380703, + 0x81C, 0xA03A0703, + 0x81C, 0x823C0703, + 0x81C, 0x813E0703, + 0x81C, 0x80400703, + 0x81C, 0x64420703, + 0x81C, 0x63440703, + 0x81C, 0x62460703, + 0x81C, 0x61480703, + 0x81C, 0x604A0703, + 0x81C, 0x414C0703, + 0x81C, 0x404E0703, + 0x81C, 0x22500703, + 0x81C, 0x21520703, + 0x81C, 0x20540703, + 0x81C, 0x03560703, + 0x81C, 0x02580703, + 0x81C, 0x015A0703, + 0x81C, 0x005C0703, + 0x81C, 0x005E0703, + 0x81C, 0x00600703, + 0x81C, 0x00620703, + 0x81C, 0x00640703, + 0x81C, 0x00660703, + 0x81C, 0x00680703, + 0x81C, 0x006A0703, + 0x81C, 0x006C0703, + 0x81C, 0x006E0703, + 0x81C, 0x00700703, + 0x81C, 0x00720703, + 0x81C, 0x00740703, + 0x81C, 0x00760703, + 0x81C, 0x00780703, + 0x81C, 0x007A0703, + 0x81C, 0x007C0703, + 0x81C, 0x007E0703, + 0x81C, 0x007E0703, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xBE000703, + 0x81C, 0xBD020703, + 0x81C, 0xBC040703, + 0x81C, 0xBB060703, + 0x81C, 0xBA080703, + 0x81C, 0xB90A0703, + 0x81C, 0xB80C0703, + 0x81C, 0xB70E0703, + 0x81C, 0xB6100703, + 0x81C, 0xB5120703, + 0x81C, 0xB4140703, + 0x81C, 0xB3160703, + 0x81C, 0xB2180703, + 0x81C, 0xB11A0703, + 0x81C, 0xB01C0703, + 0x81C, 0x921E0703, + 0x81C, 0x91200703, + 0x81C, 0x90220703, + 0x81C, 0x8F240703, + 0x81C, 0x8E260703, + 0x81C, 0x8D280703, + 0x81C, 0x8C2A0703, + 0x81C, 0x6F2C0703, + 0x81C, 0x6E2E0703, + 0x81C, 0x6D300703, + 0x81C, 0x6C320703, + 0x81C, 0x6B340703, + 0x81C, 0x6A360703, + 0x81C, 0x69380703, + 0x81C, 0x683A0703, + 0x81C, 0x673C0703, + 0x81C, 0x663E0703, + 0x81C, 0x65400703, + 0x81C, 0x64420703, + 0x81C, 0x63440703, + 0x81C, 0x62460703, + 0x81C, 0x61480703, + 0x81C, 0x604A0703, + 0x81C, 0x424C0703, + 0x81C, 0x414E0703, + 0x81C, 0x40500703, + 0x81C, 0x06520703, + 0x81C, 0x05540703, + 0x81C, 0x04560703, + 0x81C, 0x03580703, + 0x81C, 0x025A0703, + 0x81C, 0x015C0703, + 0x81C, 0x005E0703, + 0x81C, 0x00600703, + 0x81C, 0x00620703, + 0x81C, 0x00640703, + 0x81C, 0x00660703, + 0x81C, 0x00680703, + 0x81C, 0x006A0703, + 0x81C, 0x006C0703, + 0x81C, 0x006E0703, + 0x81C, 0x00700703, + 0x81C, 0x00720703, + 0x81C, 0x00740703, + 0x81C, 0x00760703, + 0x81C, 0x00780703, + 0x81C, 0x007A0703, + 0x81C, 0x007C0703, + 0x81C, 0x007E0703, + 0x81C, 0x007E0703, + 0x90000008, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFB000703, + 0x81C, 0xFA020703, + 0x81C, 0xF9040703, + 0x81C, 0xF8060703, + 0x81C, 0xF7080703, + 0x81C, 0xF60A0703, + 0x81C, 0xF50C0703, + 0x81C, 0xF40E0703, + 0x81C, 0xF3100703, + 0x81C, 0xF2120703, + 0x81C, 0xF1140703, + 0x81C, 0xF0160703, + 0x81C, 0xEF180703, + 0x81C, 0xEE1A0703, + 0x81C, 0xED1C0703, + 0x81C, 0xEC1E0703, + 0x81C, 0xEB200703, + 0x81C, 0xEA220703, + 0x81C, 0xE9240703, + 0x81C, 0xE8260703, + 0x81C, 0xE7280703, + 0x81C, 0xE62A0703, + 0x81C, 0xE52C0703, + 0x81C, 0xE42E0703, + 0x81C, 0xE3300703, + 0x81C, 0xE2320703, + 0x81C, 0xC6340703, + 0x81C, 0xC5360703, + 0x81C, 0xC4380703, + 0x81C, 0xC33A0703, + 0x81C, 0xC23C0703, + 0x81C, 0xC13E0703, + 0x81C, 0xA4400703, + 0x81C, 0xA3420703, + 0x81C, 0xA2440703, + 0x81C, 0xA1460703, + 0x81C, 0x83480703, + 0x81C, 0x824A0703, + 0x81C, 0x814C0703, + 0x81C, 0x804E0703, + 0x81C, 0x63500703, + 0x81C, 0x62520703, + 0x81C, 0x43540703, + 0x81C, 0x42560703, + 0x81C, 0x41580703, + 0x81C, 0x235A0703, + 0x81C, 0x225C0703, + 0x81C, 0x215E0703, + 0x81C, 0x20600703, + 0x81C, 0x04620703, + 0x81C, 0x03640703, + 0x81C, 0x02660703, + 0x81C, 0x01680703, + 0x81C, 0x006A0703, + 0x81C, 0x006C0703, + 0x81C, 0x006E0703, + 0x81C, 0x00700703, + 0x81C, 0x00720703, + 0x81C, 0x00740703, + 0x81C, 0x00760703, + 0x81C, 0x00780703, + 0x81C, 0x007A0703, + 0x81C, 0x007C0703, + 0x81C, 0x007E0703, + 0x81C, 0x007E0703, + 0x90000009, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF8000703, + 0x81C, 0xF7020703, + 0x81C, 0xF6040703, + 0x81C, 0xF5060703, + 0x81C, 0xF4080703, + 0x81C, 0xF30A0703, + 0x81C, 0xF20C0703, + 0x81C, 0xF10E0703, + 0x81C, 0xF0100703, + 0x81C, 0xEF120703, + 0x81C, 0xEE140703, + 0x81C, 0xED160703, + 0x81C, 0xEC180703, + 0x81C, 0xEB1A0703, + 0x81C, 0xEA1C0703, + 0x81C, 0xE91E0703, + 0x81C, 0xCA200703, + 0x81C, 0xC9220703, + 0x81C, 0xC8240703, + 0x81C, 0xC7260703, + 0x81C, 0xC6280703, + 0x81C, 0xC52A0703, + 0x81C, 0xC42C0703, + 0x81C, 0xC32E0703, + 0x81C, 0xC2300703, + 0x81C, 0xC1320703, + 0x81C, 0xA3340703, + 0x81C, 0xA2360703, + 0x81C, 0xA1380703, + 0x81C, 0xA03A0703, + 0x81C, 0x823C0703, + 0x81C, 0x813E0703, + 0x81C, 0x80400703, + 0x81C, 0x65420703, + 0x81C, 0x64440703, + 0x81C, 0x63460703, + 0x81C, 0x62480703, + 0x81C, 0x614A0703, + 0x81C, 0x424C0703, + 0x81C, 0x414E0703, + 0x81C, 0x40500703, + 0x81C, 0x22520703, + 0x81C, 0x21540703, + 0x81C, 0x20560703, + 0x81C, 0x04580703, + 0x81C, 0x035A0703, + 0x81C, 0x025C0703, + 0x81C, 0x015E0703, + 0x81C, 0x00600703, + 0x81C, 0x00620703, + 0x81C, 0x00640703, + 0x81C, 0x00660703, + 0x81C, 0x00680703, + 0x81C, 0x006A0703, + 0x81C, 0x006C0703, + 0x81C, 0x006E0703, + 0x81C, 0x00700703, + 0x81C, 0x00720703, + 0x81C, 0x00740703, + 0x81C, 0x00760703, + 0x81C, 0x00780703, + 0x81C, 0x007A0703, + 0x81C, 0x007C0703, + 0x81C, 0x007E0703, + 0x81C, 0x007E0703, + 0x9000000a, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000703, + 0x81C, 0xFB020703, + 0x81C, 0xFA040703, + 0x81C, 0xF9060703, + 0x81C, 0xF8080703, + 0x81C, 0xF70A0703, + 0x81C, 0xF60C0703, + 0x81C, 0xF50E0703, + 0x81C, 0xF4100703, + 0x81C, 0xF3120703, + 0x81C, 0xF2140703, + 0x81C, 0xF1160703, + 0x81C, 0xEF180703, + 0x81C, 0xEE1A0703, + 0x81C, 0xED1C0703, + 0x81C, 0xEC1E0703, + 0x81C, 0xEB200703, + 0x81C, 0xEA220703, + 0x81C, 0xE9240703, + 0x81C, 0xE8260703, + 0x81C, 0xE7280703, + 0x81C, 0xE62A0703, + 0x81C, 0xE52C0703, + 0x81C, 0xE42E0703, + 0x81C, 0xE3300703, + 0x81C, 0xE2320703, + 0x81C, 0xC6340703, + 0x81C, 0xC5360703, + 0x81C, 0xC4380703, + 0x81C, 0xC33A0703, + 0x81C, 0xA63C0703, + 0x81C, 0xA53E0703, + 0x81C, 0xA4400703, + 0x81C, 0xA3420703, + 0x81C, 0xA2440703, + 0x81C, 0xA1460703, + 0x81C, 0x83480703, + 0x81C, 0x824A0703, + 0x81C, 0x814C0703, + 0x81C, 0x804E0703, + 0x81C, 0x63500703, + 0x81C, 0x62520703, + 0x81C, 0x61540703, + 0x81C, 0x42560703, + 0x81C, 0x41580703, + 0x81C, 0x405A0703, + 0x81C, 0x225C0703, + 0x81C, 0x215E0703, + 0x81C, 0x20600703, + 0x81C, 0x04620703, + 0x81C, 0x03640703, + 0x81C, 0x02660703, + 0x81C, 0x01680703, + 0x81C, 0x006A0703, + 0x81C, 0x006C0703, + 0x81C, 0x006E0703, + 0x81C, 0x00700703, + 0x81C, 0x00720703, + 0x81C, 0x00740703, + 0x81C, 0x00760703, + 0x81C, 0x00780703, + 0x81C, 0x007A0703, + 0x81C, 0x007C0703, + 0x81C, 0x007E0703, + 0x81C, 0x007E0703, + 0x9000000b, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF8000703, + 0x81C, 0xF7020703, + 0x81C, 0xF6040703, + 0x81C, 0xF5060703, + 0x81C, 0xF4080703, + 0x81C, 0xF30A0703, + 0x81C, 0xF20C0703, + 0x81C, 0xF10E0703, + 0x81C, 0xF0100703, + 0x81C, 0xEF120703, + 0x81C, 0xEE140703, + 0x81C, 0xED160703, + 0x81C, 0xEC180703, + 0x81C, 0xEB1A0703, + 0x81C, 0xEA1C0703, + 0x81C, 0xE91E0703, + 0x81C, 0xCA200703, + 0x81C, 0xC9220703, + 0x81C, 0xC8240703, + 0x81C, 0xC7260703, + 0x81C, 0xC6280703, + 0x81C, 0xC52A0703, + 0x81C, 0xC42C0703, + 0x81C, 0xC32E0703, + 0x81C, 0xC2300703, + 0x81C, 0xC1320703, + 0x81C, 0xA3340703, + 0x81C, 0xA2360703, + 0x81C, 0xA1380703, + 0x81C, 0xA03A0703, + 0x81C, 0x823C0703, + 0x81C, 0x813E0703, + 0x81C, 0x80400703, + 0x81C, 0x64420703, + 0x81C, 0x63440703, + 0x81C, 0x62460703, + 0x81C, 0x61480703, + 0x81C, 0x604A0703, + 0x81C, 0x234C0703, + 0x81C, 0x224E0703, + 0x81C, 0x21500703, + 0x81C, 0x20520703, + 0x81C, 0x06540703, + 0x81C, 0x05560703, + 0x81C, 0x04580703, + 0x81C, 0x035A0703, + 0x81C, 0x025C0703, + 0x81C, 0x015E0703, + 0x81C, 0x00600703, + 0x81C, 0x00620703, + 0x81C, 0x00640703, + 0x81C, 0x00660703, + 0x81C, 0x00680703, + 0x81C, 0x006A0703, + 0x81C, 0x006C0703, + 0x81C, 0x006E0703, + 0x81C, 0x00700703, + 0x81C, 0x00720703, + 0x81C, 0x00740703, + 0x81C, 0x00760703, + 0x81C, 0x00780703, + 0x81C, 0x007A0703, + 0x81C, 0x007C0703, + 0x81C, 0x007E0703, + 0x81C, 0x007E0703, + 0x9000000c, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000703, + 0x81C, 0xFB020703, + 0x81C, 0xFA040703, + 0x81C, 0xF9060703, + 0x81C, 0xF8080703, + 0x81C, 0xF70A0703, + 0x81C, 0xF60C0703, + 0x81C, 0xF50E0703, + 0x81C, 0xF4100703, + 0x81C, 0xF3120703, + 0x81C, 0xF2140703, + 0x81C, 0xF1160703, + 0x81C, 0xF0180703, + 0x81C, 0xEF1A0703, + 0x81C, 0xEE1C0703, + 0x81C, 0xED1E0703, + 0x81C, 0xEC200703, + 0x81C, 0xEB220703, + 0x81C, 0xEA240703, + 0x81C, 0xE9260703, + 0x81C, 0xE8280703, + 0x81C, 0xE72A0703, + 0x81C, 0xE62C0703, + 0x81C, 0xE52E0703, + 0x81C, 0xE4300703, + 0x81C, 0xE3320703, + 0x81C, 0xE2340703, + 0x81C, 0xC6360703, + 0x81C, 0xC5380703, + 0x81C, 0xC43A0703, + 0x81C, 0xC33C0703, + 0x81C, 0xA63E0703, + 0x81C, 0xA5400703, + 0x81C, 0xA4420703, + 0x81C, 0xA3440703, + 0x81C, 0xA2460703, + 0x81C, 0x84480703, + 0x81C, 0x834A0703, + 0x81C, 0x824C0703, + 0x81C, 0x814E0703, + 0x81C, 0x80500703, + 0x81C, 0x63520703, + 0x81C, 0x62540703, + 0x81C, 0x61560703, + 0x81C, 0x60580703, + 0x81C, 0x225A0703, + 0x81C, 0x055C0703, + 0x81C, 0x045E0703, + 0x81C, 0x03600703, + 0x81C, 0x02620703, + 0x81C, 0x01640703, + 0x81C, 0x00660703, + 0x81C, 0x00680703, + 0x81C, 0x006A0703, + 0x81C, 0x006C0703, + 0x81C, 0x006E0703, + 0x81C, 0x00700703, + 0x81C, 0x00720703, + 0x81C, 0x00740703, + 0x81C, 0x00760703, + 0x81C, 0x00780703, + 0x81C, 0x007A0703, + 0x81C, 0x007C0703, + 0x81C, 0x007E0703, + 0x81C, 0x007E0703, + 0x9000000d, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFB000703, + 0x81C, 0xFA020703, + 0x81C, 0xF9040703, + 0x81C, 0xF8060703, + 0x81C, 0xF7080703, + 0x81C, 0xF60A0703, + 0x81C, 0xF50C0703, + 0x81C, 0xF40E0703, + 0x81C, 0xF3100703, + 0x81C, 0xF2120703, + 0x81C, 0xF1140703, + 0x81C, 0xEF160703, + 0x81C, 0xEE180703, + 0x81C, 0xED1A0703, + 0x81C, 0xEC1C0703, + 0x81C, 0xEB1E0703, + 0x81C, 0xEA200703, + 0x81C, 0xE9220703, + 0x81C, 0xE8240703, + 0x81C, 0xE7260703, + 0x81C, 0xE6280703, + 0x81C, 0xE52A0703, + 0x81C, 0xE42C0703, + 0x81C, 0xE32E0703, + 0x81C, 0xE2300703, + 0x81C, 0xE1320703, + 0x81C, 0xC6340703, + 0x81C, 0xC5360703, + 0x81C, 0xC4380703, + 0x81C, 0xC33A0703, + 0x81C, 0xA63C0703, + 0x81C, 0xA53E0703, + 0x81C, 0xA4400703, + 0x81C, 0xA3420703, + 0x81C, 0xA2440703, + 0x81C, 0xA1460703, + 0x81C, 0x83480703, + 0x81C, 0x824A0703, + 0x81C, 0x814C0703, + 0x81C, 0x804E0703, + 0x81C, 0x63500703, + 0x81C, 0x62520703, + 0x81C, 0x61540703, + 0x81C, 0x42560703, + 0x81C, 0x41580703, + 0x81C, 0x405A0703, + 0x81C, 0x225C0703, + 0x81C, 0x215E0703, + 0x81C, 0x20600703, + 0x81C, 0x04620703, + 0x81C, 0x03640703, + 0x81C, 0x02660703, + 0x81C, 0x01680703, + 0x81C, 0x006A0703, + 0x81C, 0x006C0703, + 0x81C, 0x006E0703, + 0x81C, 0x00700703, + 0x81C, 0x00720703, + 0x81C, 0x00740703, + 0x81C, 0x00760703, + 0x81C, 0x00780703, + 0x81C, 0x007A0703, + 0x81C, 0x007C0703, + 0x81C, 0x007E0703, + 0x81C, 0x007E0703, + 0x9000000e, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFB000703, + 0x81C, 0xFA020703, + 0x81C, 0xF9040703, + 0x81C, 0xF8060703, + 0x81C, 0xF7080703, + 0x81C, 0xF60A0703, + 0x81C, 0xF50C0703, + 0x81C, 0xF40E0703, + 0x81C, 0xF3100703, + 0x81C, 0xF2120703, + 0x81C, 0xF1140703, + 0x81C, 0xEF160703, + 0x81C, 0xEE180703, + 0x81C, 0xED1A0703, + 0x81C, 0xEC1C0703, + 0x81C, 0xEB1E0703, + 0x81C, 0xEA200703, + 0x81C, 0xE9220703, + 0x81C, 0xE8240703, + 0x81C, 0xE7260703, + 0x81C, 0xE6280703, + 0x81C, 0xE52A0703, + 0x81C, 0xE42C0703, + 0x81C, 0xE32E0703, + 0x81C, 0xE2300703, + 0x81C, 0xE1320703, + 0x81C, 0xC6340703, + 0x81C, 0xC5360703, + 0x81C, 0xC4380703, + 0x81C, 0xC33A0703, + 0x81C, 0xA63C0703, + 0x81C, 0xA53E0703, + 0x81C, 0xA4400703, + 0x81C, 0xA3420703, + 0x81C, 0xA2440703, + 0x81C, 0xA1460703, + 0x81C, 0x83480703, + 0x81C, 0x824A0703, + 0x81C, 0x814C0703, + 0x81C, 0x804E0703, + 0x81C, 0x63500703, + 0x81C, 0x62520703, + 0x81C, 0x61540703, + 0x81C, 0x42560703, + 0x81C, 0x41580703, + 0x81C, 0x405A0703, + 0x81C, 0x225C0703, + 0x81C, 0x215E0703, + 0x81C, 0x20600703, + 0x81C, 0x04620703, + 0x81C, 0x03640703, + 0x81C, 0x02660703, + 0x81C, 0x01680703, + 0x81C, 0x006A0703, + 0x81C, 0x006C0703, + 0x81C, 0x006E0703, + 0x81C, 0x00700703, + 0x81C, 0x00720703, + 0x81C, 0x00740703, + 0x81C, 0x00760703, + 0x81C, 0x00780703, + 0x81C, 0x007A0703, + 0x81C, 0x007C0703, + 0x81C, 0x007E0703, + 0x81C, 0x007E0703, + 0x9000000f, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xBF000703, + 0x81C, 0xBF020703, + 0x81C, 0xBF040703, + 0x81C, 0xBF060703, + 0x81C, 0xBF080703, + 0x81C, 0xBE0A0703, + 0x81C, 0xBD0C0703, + 0x81C, 0xBC0E0703, + 0x81C, 0xBB100703, + 0x81C, 0xBA120703, + 0x81C, 0xB9140703, + 0x81C, 0xB8160703, + 0x81C, 0xB7180703, + 0x81C, 0xB61A0703, + 0x81C, 0xB51C0703, + 0x81C, 0xB41E0703, + 0x81C, 0xB1200703, + 0x81C, 0xB2220703, + 0x81C, 0xB1240703, + 0x81C, 0xB0260703, + 0x81C, 0xAF280703, + 0x81C, 0xAE2A0703, + 0x81C, 0xAD2C0703, + 0x81C, 0xAC2E0703, + 0x81C, 0xAB300703, + 0x81C, 0xAA320703, + 0x81C, 0xC6340703, + 0x81C, 0xC5360703, + 0x81C, 0xC4380703, + 0x81C, 0xC33A0703, + 0x81C, 0x883C0703, + 0x81C, 0x873E0703, + 0x81C, 0x86400703, + 0x81C, 0x85420703, + 0x81C, 0x84440703, + 0x81C, 0x83460703, + 0x81C, 0x67480703, + 0x81C, 0x664A0703, + 0x81C, 0x654C0703, + 0x81C, 0x644E0703, + 0x81C, 0x27500703, + 0x81C, 0x26520703, + 0x81C, 0x25540703, + 0x81C, 0x24560703, + 0x81C, 0x23580703, + 0x81C, 0x225A0703, + 0x81C, 0x215C0703, + 0x81C, 0x205E0703, + 0x81C, 0x03600703, + 0x81C, 0x02620703, + 0x81C, 0x01640703, + 0x81C, 0x00660703, + 0x81C, 0x00680703, + 0x81C, 0x006A0703, + 0x81C, 0x006C0703, + 0x81C, 0x006E0703, + 0x81C, 0x00700703, + 0x81C, 0x00720703, + 0x81C, 0x00740703, + 0x81C, 0x00760703, + 0x81C, 0x00780703, + 0x81C, 0x007A0703, + 0x81C, 0x007C0703, + 0x81C, 0x007E0703, + 0x81C, 0x007E0703, + 0x90000010, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xFC000403, + 0x81C, 0xFB000703, + 0x81C, 0xFA020703, + 0x81C, 0xF9040703, + 0x81C, 0xF8060703, + 0x81C, 0xF7080703, + 0x81C, 0xF60A0703, + 0x81C, 0xF50C0703, + 0x81C, 0xF40E0703, + 0x81C, 0xF3100703, + 0x81C, 0xF2120703, + 0x81C, 0xF1140703, + 0x81C, 0xF0160703, + 0x81C, 0xEF180703, + 0x81C, 0xEE1A0703, + 0x81C, 0xED1C0703, + 0x81C, 0xEC1E0703, + 0x81C, 0xEB200703, + 0x81C, 0xEA220703, + 0x81C, 0xE9240703, + 0x81C, 0xE8260703, + 0x81C, 0xE7280703, + 0x81C, 0xE62A0703, + 0x81C, 0xE52C0703, + 0x81C, 0xE42E0703, + 0x81C, 0xE3300703, + 0x81C, 0xE2320703, + 0x81C, 0xC6340703, + 0x81C, 0xC5360703, + 0x81C, 0xC4380703, + 0x81C, 0xC33A0703, + 0x81C, 0xA63C0703, + 0x81C, 0xA53E0703, + 0x81C, 0xA4400703, + 0x81C, 0xA3420703, + 0x81C, 0xA2440703, + 0x81C, 0x84460703, + 0x81C, 0x83480703, + 0x81C, 0x824A0703, + 0x81C, 0x814C0703, + 0x81C, 0x804E0703, + 0x81C, 0x63500703, + 0x81C, 0x62520703, + 0x81C, 0x61540703, + 0x81C, 0x60560703, + 0x81C, 0x22580703, + 0x81C, 0x055A0703, + 0x81C, 0x045C0703, + 0x81C, 0x035E0703, + 0x81C, 0x02600703, + 0x81C, 0x01620703, + 0x81C, 0x00640703, + 0x81C, 0x00660703, + 0x81C, 0x00680703, + 0x81C, 0x006A0703, + 0x81C, 0x006C0703, + 0x81C, 0x006E0703, + 0x81C, 0x00700703, + 0x81C, 0x00720703, + 0x81C, 0x00740703, + 0x81C, 0x00760703, + 0x81C, 0x00780703, + 0x81C, 0x007A0703, + 0x81C, 0x007C0703, + 0x81C, 0x007E0703, + 0x90000012, 0x00000000, 0x40000000, 0x00000000, + 0x81C, 0xF7000703, + 0x81C, 0xF6020703, + 0x81C, 0xF5040703, + 0x81C, 0xF4060703, + 0x81C, 0xF3080703, + 0x81C, 0xF20A0703, + 0x81C, 0xF10C0703, + 0x81C, 0xF00E0703, + 0x81C, 0xEF100703, + 0x81C, 0xEE120703, + 0x81C, 0xED140703, + 0x81C, 0xEC160703, + 0x81C, 0xEB180703, + 0x81C, 0xEA1A0703, + 0x81C, 0xE91C0703, + 0x81C, 0xCA1E0703, + 0x81C, 0xC9200703, + 0x81C, 0xC8220703, + 0x81C, 0xC7240703, + 0x81C, 0xC6260703, + 0x81C, 0xC5280703, + 0x81C, 0xC42A0703, + 0x81C, 0xC32C0703, + 0x81C, 0xC22E0703, + 0x81C, 0xC1300703, + 0x81C, 0xA4320703, + 0x81C, 0xA3340703, + 0x81C, 0xA2360703, + 0x81C, 0xA1380703, + 0x81C, 0xA03A0703, + 0x81C, 0x823C0703, + 0x81C, 0x813E0703, + 0x81C, 0x80400703, + 0x81C, 0x64420703, + 0x81C, 0x63440703, + 0x81C, 0x62460703, + 0x81C, 0x61480703, + 0x81C, 0x604A0703, + 0x81C, 0x414C0703, + 0x81C, 0x404E0703, + 0x81C, 0x22500703, + 0x81C, 0x21520703, + 0x81C, 0x20540703, + 0x81C, 0x03560703, + 0x81C, 0x02580703, + 0x81C, 0x015A0703, + 0x81C, 0x005C0703, + 0x81C, 0x005E0703, + 0x81C, 0x00600703, + 0x81C, 0x00620703, + 0x81C, 0x00640703, + 0x81C, 0x00660703, + 0x81C, 0x00680703, + 0x81C, 0x006A0703, + 0x81C, 0x006C0703, + 0x81C, 0x006E0703, + 0x81C, 0x00700703, + 0x81C, 0x00720703, + 0x81C, 0x00740703, + 0x81C, 0x00760703, + 0x81C, 0x00780703, + 0x81C, 0x007A0703, + 0x81C, 0x007C0703, + 0x81C, 0x007E0703, + 0x81C, 0x007E0703, + 0xA0000000, 0x00000000, + 0x81C, 0xFC000703, + 0x81C, 0xFB020703, + 0x81C, 0xFA040703, + 0x81C, 0xF9060703, + 0x81C, 0xF8080703, + 0x81C, 0xF70A0703, + 0x81C, 0xF60C0703, + 0x81C, 0xF50E0703, + 0x81C, 0xF4100703, + 0x81C, 0xF3120703, + 0x81C, 0xF2140703, + 0x81C, 0xF1160703, + 0x81C, 0xF0180703, + 0x81C, 0xEF1A0703, + 0x81C, 0xEE1C0703, + 0x81C, 0xED1E0703, + 0x81C, 0xEC200703, + 0x81C, 0xEB220703, + 0x81C, 0xEA240703, + 0x81C, 0xE9260703, + 0x81C, 0xE8280703, + 0x81C, 0xE72A0703, + 0x81C, 0xE62C0703, + 0x81C, 0xE52E0703, + 0x81C, 0xE4300703, + 0x81C, 0xE3320703, + 0x81C, 0xE2340703, + 0x81C, 0xC6360703, + 0x81C, 0xC5380703, + 0x81C, 0xC43A0703, + 0x81C, 0xC33C0703, + 0x81C, 0xA63E0703, + 0x81C, 0xA5400703, + 0x81C, 0xA4420703, + 0x81C, 0xA3440703, + 0x81C, 0xA2460703, + 0x81C, 0x84480703, + 0x81C, 0x834A0703, + 0x81C, 0x824C0703, + 0x81C, 0x814E0703, + 0x81C, 0x80500703, + 0x81C, 0x63520703, + 0x81C, 0x62540703, + 0x81C, 0x61560703, + 0x81C, 0x60580703, + 0x81C, 0x235A0703, + 0x81C, 0x225C0703, + 0x81C, 0x215E0703, + 0x81C, 0x20600703, + 0x81C, 0x03620703, + 0x81C, 0x02640703, + 0x81C, 0x01660703, + 0x81C, 0x00680703, + 0x81C, 0x006A0703, + 0x81C, 0x006C0703, + 0x81C, 0x006E0703, + 0x81C, 0x00700703, + 0x81C, 0x00720703, + 0x81C, 0x00740703, + 0x81C, 0x00760703, + 0x81C, 0x00780703, + 0x81C, 0x007A0703, + 0x81C, 0x007C0703, + 0x81C, 0x007E0703, + 0x81C, 0x007E0703, + 0xB0000000, 0x00000000, + 0x80000000, 0x00000000, 0x40000000, 0x00000000, + 0xC50, 0x00000022, + 0xC50, 0x00000020, + 0xE50, 0x00000022, + 0xE50, 0x00000020, + 0x9000000d, 0x00000000, 0x40000000, 0x00000000, + 0xC50, 0x00000022, + 0xC50, 0x00000020, + 0xE50, 0x00000022, + 0xE50, 0x00000020, + 0x9000000e, 0x00000000, 0x40000000, 0x00000000, + 0xC50, 0x00000022, + 0xC50, 0x00000020, + 0xE50, 0x00000022, + 0xE50, 0x00000020, + 0xA0000000, 0x00000000, + 0xC50, 0x00000022, + 0xC50, 0x00000020, + 0xE50, 0x00000022, + 0xE50, 0x00000020, + 0xB0000000, 0x00000000, + +}; + +RTW_DECL_TABLE_PHY_COND(rtw8822b_agc, rtw_phy_cfg_agc); + +static const u32 rtw8822b_bb[] = { + 0x800, 0x9020D010, + 0x804, 0x800181A0, + 0x808, 0x0E028233, + 0x80C, 0x10000013, + 0x810, 0x22101243, + 0x814, 0x020C3D11, + 0x818, 0x84A10385, + 0x81C, 0x1E1E081F, + 0x820, 0x0001AAAA, + 0x824, 0x00030FE0, + 0x828, 0x0000CCCC, + 0x82C, 0x75CB7010, + 0x830, 0x79A0EAAA, + 0x834, 0x072E6986, + 0x838, 0x87766441, + 0x83C, 0x9194B2B7, + 0x840, 0x171750E0, + 0x844, 0x4D3D7CDB, + 0x848, 0x4AD0408B, + 0x84C, 0x6AFBF7A5, + 0x850, 0x28A74706, + 0x854, 0x0001520C, + 0x858, 0x4060C000, + 0x85C, 0x74010160, + 0x860, 0x68A7C321, + 0x864, 0x79F27032, + 0x868, 0x8CA7A314, + 0x86C, 0x778C2878, + 0x870, 0x77777777, + 0x874, 0x27612C2E, + 0x878, 0xC0003152, + 0x87C, 0x5C8FC000, + 0x880, 0x00000000, + 0x884, 0x00000000, + 0x888, 0x00000000, + 0x88C, 0x00000000, + 0x890, 0x00000000, + 0x894, 0x00000000, + 0x898, 0x00000000, + 0x89C, 0x00000000, + 0x8A0, 0x00000013, + 0x8A4, 0x7F7F7F7F, + 0x8A8, 0x2202033E, + 0x8AC, 0xF00F000A, + 0x8B0, 0x00000600, + 0x8B4, 0x000FC080, + 0x8B8, 0xEC0057F7, + 0x8BC, 0xACB520A3, + 0x8C0, 0xFFE04020, + 0x8C4, 0x47C00000, + 0x8C8, 0x000251A5, + 0x8CC, 0x08108492, + 0x8D0, 0x0000B800, + 0x8D4, 0x860308A0, + 0x8D8, 0x29095612, + 0x8DC, 0x00000000, + 0x8E0, 0x32D16777, + 0x8E4, 0x4C098935, + 0x8E8, 0xFFFFC42C, + 0x8EC, 0x99999999, + 0x8F0, 0x00009999, + 0x8F4, 0x00D80FA1, + 0x8F8, 0x40000080, + 0x8FC, 0x00000130, + 0x900, 0x00800000, + 0x904, 0x00000000, + 0x908, 0x00000000, + 0x90C, 0xD3000000, + 0x910, 0x0000FC00, + 0x914, 0xC6380000, + 0x918, 0x1C1028C0, + 0x91C, 0x64B11A1C, + 0x920, 0xE0767233, + 0x924, 0x855A2500, + 0x928, 0x4AB0E4E4, + 0x92C, 0xFFFEB200, + 0x930, 0xFFFFFFFE, + 0x934, 0x001FFFFF, + 0x938, 0x00008480, + 0x93C, 0xE41C0642, + 0x940, 0x0E470430, + 0x944, 0x00000000, + 0x948, 0xAC000000, + 0x94C, 0x10000083, + 0x950, 0x32010080, + 0x954, 0x84510080, + 0x958, 0x00000001, + 0x95C, 0x04248000, + 0x960, 0x00000000, + 0x964, 0x00000000, + 0x968, 0x00000000, + 0x96C, 0x00000000, + 0x970, 0x00001FFF, + 0x974, 0x44000FFF, + 0x978, 0x00000000, + 0x97C, 0x00000000, + 0x980, 0x00000000, + 0x984, 0x00000000, + 0x988, 0x00000000, + 0x98C, 0x43440000, + 0x990, 0x27100000, + 0x994, 0xFFFF0100, + 0x998, 0xFFFFFF5C, + 0x99C, 0xFFFFFFFF, + 0x9A0, 0x000000FF, + 0x9A4, 0x80000088, + 0x9A8, 0x0C2F0000, + 0x9AC, 0x01560000, + 0x9B0, 0x70000000, + 0x9B4, 0x00000000, + 0x9B8, 0x00000000, + 0x9BC, 0x00000000, + 0x9C0, 0x00000000, + 0x9C4, 0x00000000, + 0x9C8, 0x00000000, + 0x9CC, 0x00000000, + 0x9D0, 0x00000000, + 0x9D4, 0x00000000, + 0x9D8, 0x00000000, + 0x9DC, 0x00000000, + 0x9E0, 0x00000000, + 0x9E4, 0x02000402, + 0x9E8, 0x000022D4, + 0x9EC, 0x00000000, + 0x9F0, 0x00010080, + 0x9F4, 0x00000000, + 0x9F8, 0x00000000, + 0x9FC, 0xEFFFF7F7, + 0xA00, 0x00D047C8, + 0xA04, 0x81FF800C, + 0xA08, 0x8C838300, + 0xA0C, 0x2E20100F, + 0xA10, 0x9500BB78, + 0xA14, 0x1114D028, + 0xA18, 0x00881117, + 0xA1C, 0x89140F00, + 0xA20, 0x84880000, + 0xA24, 0x384F6577, + 0xA28, 0x00001525, + 0xA2C, 0x00920000, + 0xA70, 0x101FFF00, + 0xA74, 0x00000148, + 0xA78, 0x00000900, + 0xA7C, 0x225B0606, + 0xA80, 0x218675B2, + 0xA84, 0x80208C00, + 0xA88, 0x040C0000, + 0xA8C, 0x12345678, + 0xA90, 0xABCDEF00, + 0xA94, 0x001B1B89, + 0xA98, 0x030A0000, + 0xA9C, 0x00060000, + 0xAA0, 0x00000000, + 0xAA4, 0x0004000F, + 0xAA8, 0x00000200, + 0xB00, 0xE1000440, + 0xB04, 0x00800000, + 0xB08, 0xFF02030B, + 0xB0C, 0x01EAA406, + 0xB10, 0x00030690, + 0xB14, 0x006000FA, + 0xB18, 0x00000002, + 0xB1C, 0x00000002, + 0xB20, 0x4B00001F, + 0xB24, 0x4E8E3E40, + 0xB28, 0x03020100, + 0xB2C, 0x07060504, + 0xB30, 0x0B0A0908, + 0xB34, 0x0F0E0D0C, + 0xB38, 0x13121110, + 0xB3C, 0x0000003A, + 0xB40, 0x00000000, + 0xB44, 0x80000000, + 0xB48, 0x3F0000FA, + 0xB4C, 0x88C80020, + 0xB50, 0x00000000, + 0xB54, 0x00004241, + 0xB58, 0xE0008208, + 0xB5C, 0x41EFFFF9, + 0xB60, 0x00000000, + 0xB64, 0x00200063, + 0xB68, 0x0000003A, + 0xB6C, 0x00000102, + 0xB70, 0x4E6D1870, + 0xB74, 0x03020100, + 0xB78, 0x07060504, + 0xB7C, 0x0B0A0908, + 0xB80, 0x0F0E0D0C, + 0xB84, 0x13121110, + 0xB88, 0x00000000, + 0xB8C, 0x00000000, + 0xC00, 0x00000007, + 0xC04, 0x00000020, + 0xC08, 0x60403231, + 0xC0C, 0x00012345, + 0xC10, 0x00000100, + 0xC14, 0x01000000, + 0xC18, 0x00000000, + 0xC1C, 0x40040053, + 0xC20, 0x40020103, + 0xC24, 0x00000000, + 0xC28, 0x00000000, + 0xC2C, 0x00000000, + 0xC30, 0x00000000, + 0xC34, 0x00000000, + 0xC38, 0x00000000, + 0xC3C, 0x00000000, + 0xC40, 0x00000000, + 0xC44, 0x00000000, + 0xC48, 0x00000000, + 0xC4C, 0x00000000, + 0xC50, 0x00000020, + 0xC54, 0x00000000, + 0xC58, 0xD8020402, + 0xC5C, 0xDE000120, + 0xC68, 0x5979993F, + 0xC6C, 0x0000122A, + 0xC70, 0x99795979, + 0xC74, 0x99795979, + 0xC78, 0x99799979, + 0xC7C, 0x99791979, + 0xC80, 0x19791979, + 0xC84, 0x19791979, + 0xC88, 0x00000000, + 0xC8C, 0x07000000, + 0xC94, 0x01000100, + 0xC98, 0x201C8000, + 0xC9C, 0x00000000, + 0xCA0, 0x0000A555, + 0xCA4, 0x08040201, + 0xCA8, 0x80402010, + 0xCAC, 0x00000000, + 0xCB0, 0x77777777, + 0xCB4, 0x00007777, + 0xCB8, 0x00000000, + 0xCBC, 0x00000000, + 0xCC0, 0x00000000, + 0xCC4, 0x00000000, + 0xCC8, 0x00000000, + 0xCCC, 0x00000000, + 0xCD0, 0x00000000, + 0xCD4, 0x00000000, + 0xCD8, 0x00000000, + 0xCDC, 0x00000000, + 0xCE0, 0x00000000, + 0xCE4, 0x00000000, + 0xCE8, 0x00000000, + 0xCEC, 0x00000000, + 0xE00, 0x00000007, + 0xE04, 0x00000020, + 0xE08, 0x60403231, + 0xE0C, 0x00012345, + 0xE10, 0x00000100, + 0xE14, 0x01000000, + 0xE18, 0x00000000, + 0xE1C, 0x40040053, + 0xE20, 0x40020103, + 0xE24, 0x00000000, + 0xE28, 0x00000000, + 0xE2C, 0x00000000, + 0xE30, 0x00000000, + 0xE34, 0x00000000, + 0xE38, 0x00000000, + 0xE3C, 0x00000000, + 0xE40, 0x00000000, + 0xE44, 0x00000000, + 0xE48, 0x00000000, + 0xE4C, 0x00000000, + 0xE50, 0x00000020, + 0xE54, 0x00000000, + 0xE58, 0xD8120402, + 0xE5C, 0xDE000120, + 0xE68, 0x5979993F, + 0xE6C, 0x0000122A, + 0xE70, 0x99795979, + 0xE74, 0x99795979, + 0xE78, 0x99799979, + 0xE7C, 0x99791979, + 0xE80, 0x19791979, + 0xE84, 0x19791979, + 0xE88, 0x00000000, + 0xE8C, 0x07000000, + 0xE94, 0x01000100, + 0xE98, 0x201C8000, + 0xE9C, 0x00000000, + 0xEA0, 0x0000A555, + 0xEA4, 0x08040201, + 0xEA8, 0x80402010, + 0xEAC, 0x00000000, + 0xEB0, 0x77777777, + 0xEB4, 0x00007777, + 0xEB8, 0x00000000, + 0xEBC, 0x00000000, + 0xEC0, 0x00000000, + 0xEC4, 0x00000000, + 0xEC8, 0x00000000, + 0xECC, 0x00000000, + 0xED0, 0x00000000, + 0xED4, 0x00000000, + 0xED8, 0x00000000, + 0xEDC, 0x00000000, + 0xEE0, 0x00000000, + 0xEE4, 0x00000000, + 0xEE8, 0x00000000, + 0xEEC, 0x00000000, + 0x1900, 0x00000000, + 0x1904, 0x00238000, + 0x1908, 0x00000000, + 0x190C, 0x00000000, + 0x1910, 0x00000000, + 0x1914, 0x00000000, + 0x1918, 0x00000000, + 0x191C, 0x00000000, + 0x1920, 0x00000000, + 0x1924, 0x00000000, + 0x1928, 0x00000000, + 0x192C, 0x00000000, + 0x1930, 0x00000000, + 0x1934, 0x00000000, + 0x1938, 0x00000000, + 0x193C, 0x00000000, + 0x1940, 0x00000000, + 0x1944, 0x00000000, + 0x1948, 0x00000000, + 0x194C, 0x00000000, + 0x1950, 0x00000000, + 0x1954, 0x00000000, + 0x1958, 0x00000000, + 0x195C, 0x00000000, + 0x1960, 0x00000000, + 0x1964, 0x00000000, + 0x1968, 0x00000000, + 0x196C, 0x00000000, + 0x1970, 0x00000000, + 0x1974, 0x00000000, + 0x1978, 0x00000000, + 0x197C, 0x00000000, + 0x1980, 0x00000000, + 0x1984, 0x03000000, + 0x1988, 0x21401E88, + 0x198C, 0x00004000, + 0x1990, 0x00000000, + 0x1994, 0x00000000, + 0x1998, 0x00000053, + 0x199C, 0x00000000, + 0x19A0, 0x00000000, + 0x19A4, 0x00000000, + 0x19A8, 0x00000000, + 0x19AC, 0x0E47E47F, + 0x19B0, 0x00000000, + 0x19B4, 0x0E47E47F, + 0x19B8, 0x00000000, + 0x19BC, 0x00000000, + 0x19C0, 0x00000000, + 0x19C4, 0x00000000, + 0x19C8, 0x00000000, + 0x19CC, 0x00000000, + 0x19D0, 0x00000000, + 0x19D4, 0xAAAAAAAA, + 0x19D8, 0x00000AAA, + 0x19DC, 0x133E0F37, + 0x19E0, 0x00000000, + 0x19E4, 0x00000000, + 0x19E8, 0x00000000, + 0x19EC, 0x00000000, + 0x19F0, 0x00000000, + 0x19F4, 0x00000000, + 0x19F8, 0x01A00000, + 0x19FC, 0x00000000, + 0x1C00, 0x00000100, + 0x1C04, 0x01000000, + 0x1C08, 0x00000100, + 0x1C0C, 0x01000000, + 0x1C10, 0x00000100, + 0x1C14, 0x01000000, + 0x1C18, 0x00000100, + 0x1C1C, 0x01000000, + 0x1C20, 0x00000100, + 0x1C24, 0x01000000, + 0x1C28, 0x00000100, + 0x1C2C, 0x01000000, + 0x1C30, 0x00000100, + 0x1C34, 0x01000000, + 0x1C38, 0x00000000, + 0x1C3C, 0x00000000, + 0x1C40, 0x000C0100, + 0x1C44, 0x000000F3, + 0x1C48, 0x1A8249A8, + 0x1C4C, 0x1461C826, + 0x1C50, 0x0001469E, + 0x1C54, 0x58D158D1, + 0x1C58, 0x04490088, + 0x1C5C, 0x04004400, + 0x1C60, 0x00000000, + 0x1C64, 0x04004400, + 0x1C68, 0x00000100, + 0x1C6C, 0x01000000, + 0x1C70, 0x00000100, + 0x1C74, 0x01000000, + 0x1C78, 0x00000000, + 0x1C7C, 0x00000010, + 0x1C80, 0x5FFF5FFF, + 0x1C84, 0x5FFF5FFF, + 0x1C88, 0x5FFF5FFF, + 0x1C8C, 0x5FFF5FFF, + 0x1C90, 0x5FFF5FFF, + 0x1C94, 0x5FFF5FFF, + 0x1C98, 0x5FFF5FFF, + 0x1C9C, 0x5FFF5FFF, + 0x1CA0, 0x00000100, + 0x1CA4, 0x01000000, + 0x1CA8, 0x00000100, + 0x1CAC, 0x5FFF5FFF, + 0x1CB0, 0x00000100, + 0x1CB4, 0x01000000, + 0x1CB8, 0x00000000, + 0x1CBC, 0x00000000, + 0x1CC0, 0x00000100, + 0x1CC4, 0x01000000, + 0x1CC8, 0x00000100, + 0x1CCC, 0x01000000, + 0x1CD0, 0x00000100, + 0x1CD4, 0x01000000, + 0x1CD8, 0x00000100, + 0x1CDC, 0x01000000, + 0x1CE0, 0x00000100, + 0x1CE4, 0x01000000, + 0x1CE8, 0x00000100, + 0x1CEC, 0x01000000, + 0x1CF0, 0x00000100, + 0x1CF4, 0x01000000, + 0x1CF8, 0x00000000, + 0x1CFC, 0x00000000, + 0xC60, 0x70038040, + 0xC60, 0x70038040, + 0xC60, 0x70146040, + 0xC60, 0x70246040, + 0xC60, 0x70346040, + 0xC60, 0x70446040, + 0xC60, 0x70532040, + 0xC60, 0x70646040, + 0xC60, 0x70738040, + 0xC60, 0x70838040, + 0xC60, 0x70938040, + 0xC60, 0x70A38040, + 0xC60, 0x70B36040, + 0xC60, 0x70C06040, + 0xC60, 0x70D06040, + 0xC60, 0x70E76040, + 0xC60, 0x70F06040, + 0xE60, 0x70038040, + 0xE60, 0x70038040, + 0xE60, 0x70146040, + 0xE60, 0x70246040, + 0xE60, 0x70346040, + 0xE60, 0x70446040, + 0xE60, 0x70532040, + 0xE60, 0x70646040, + 0xE60, 0x70738040, + 0xE60, 0x70838040, + 0xE60, 0x70938040, + 0xE60, 0x70A38040, + 0xE60, 0x70B36040, + 0xE60, 0x70C06040, + 0xE60, 0x70D06040, + 0xE60, 0x70E76040, + 0xE60, 0x70F06040, + 0xC64, 0x00800000, + 0xC64, 0x08800001, + 0xC64, 0x00800002, + 0xC64, 0x00800003, + 0xC64, 0x00800004, + 0xC64, 0x00800005, + 0xC64, 0x00800006, + 0xC64, 0x08800007, + 0xC64, 0x00004000, + 0xE64, 0x00800000, + 0xE64, 0x08800001, + 0xE64, 0x00800002, + 0xE64, 0x00800003, + 0xE64, 0x00800004, + 0xE64, 0x00800005, + 0xE64, 0x00800006, + 0xE64, 0x08800007, + 0xE64, 0x00004000, + 0x1B00, 0xF8000008, + 0x1B00, 0xF80A7008, + 0x1B00, 0xF8015008, + 0x1B00, 0xF8000008, + 0x1B04, 0xE24629D2, + 0x1B08, 0x00000080, + 0x1B0C, 0x00000000, + 0x1B10, 0x00011C00, + 0x1B14, 0x00000000, + 0x1B18, 0x00292903, + 0x1B1C, 0xA2193C32, + 0x1B20, 0x01840008, + 0x1B24, 0x01860008, + 0x1B28, 0x80060300, + 0x1B2C, 0x00000003, + 0x1B30, 0x20000000, + 0x1B34, 0x00000800, + 0x1B3C, 0x20000000, + 0x1BC0, 0x01000000, + 0x1BCC, 0x00000000, + 0x1B00, 0xF800000A, + 0x1B1C, 0xA2193C32, + 0x1B20, 0x01840008, + 0x1B24, 0x01860008, + 0x1B28, 0x80060300, + 0x1B2C, 0x00000003, + 0x1B30, 0x20000000, + 0x1B34, 0x00000800, + 0x1B3C, 0x20000000, + 0x1BC0, 0x01000000, + 0x1BCC, 0x00000000, + 0x1B00, 0xF8000000, + 0x1B80, 0x00000007, + 0x1B80, 0x090A0005, + 0x1B80, 0x090A0007, + 0x1B80, 0x0FFE0015, + 0x1B80, 0x0FFE0017, + 0x1B80, 0x00220025, + 0x1B80, 0x00220027, + 0x1B80, 0x00040035, + 0x1B80, 0x00040037, + 0x1B80, 0x05C00045, + 0x1B80, 0x05C00047, + 0x1B80, 0x00070055, + 0x1B80, 0x00070057, + 0x1B80, 0x64000065, + 0x1B80, 0x64000067, + 0x1B80, 0x00020075, + 0x1B80, 0x00020077, + 0x1B80, 0x00080085, + 0x1B80, 0x00080087, + 0x1B80, 0x80000095, + 0x1B80, 0x80000097, + 0x1B80, 0x090800A5, + 0x1B80, 0x090800A7, + 0x1B80, 0x0F0200B5, + 0x1B80, 0x0F0200B7, + 0x1B80, 0x002200C5, + 0x1B80, 0x002200C7, + 0x1B80, 0x000400D5, + 0x1B80, 0x000400D7, + 0x1B80, 0x05C000E5, + 0x1B80, 0x05C000E7, + 0x1B80, 0x000700F5, + 0x1B80, 0x000700F7, + 0x1B80, 0x64020105, + 0x1B80, 0x64020107, + 0x1B80, 0x00020115, + 0x1B80, 0x00020117, + 0x1B80, 0x00040125, + 0x1B80, 0x00040127, + 0x1B80, 0x4A000135, + 0x1B80, 0x4A000137, + 0x1B80, 0x4B040145, + 0x1B80, 0x4B040147, + 0x1B80, 0x85030155, + 0x1B80, 0x85030157, + 0x1B80, 0x40090165, + 0x1B80, 0x40090167, + 0x1B80, 0xE0280175, + 0x1B80, 0xE0280177, + 0x1B80, 0x4B050185, + 0x1B80, 0x4B050187, + 0x1B80, 0x86030195, + 0x1B80, 0x86030197, + 0x1B80, 0x400B01A5, + 0x1B80, 0x400B01A7, + 0x1B80, 0xE02801B5, + 0x1B80, 0xE02801B7, + 0x1B80, 0x4B0001C5, + 0x1B80, 0x4B0001C7, + 0x1B80, 0x000701D5, + 0x1B80, 0x000701D7, + 0x1B80, 0x4C0001E5, + 0x1B80, 0x4C0001E7, + 0x1B80, 0x000401F5, + 0x1B80, 0x000401F7, + 0x1B80, 0x4D040205, + 0x1B80, 0x4D040207, + 0x1B80, 0x2EF00215, + 0x1B80, 0x2EF00217, + 0x1B80, 0x00000225, + 0x1B80, 0x00000227, + 0x1B80, 0x20810235, + 0x1B80, 0x20810237, + 0x1B80, 0x23450245, + 0x1B80, 0x23450247, + 0x1B80, 0x4D000255, + 0x1B80, 0x4D000257, + 0x1B80, 0x00040265, + 0x1B80, 0x00040267, + 0x1B80, 0x30000275, + 0x1B80, 0x30000277, + 0x1B80, 0xE1D80285, + 0x1B80, 0xE1D80287, + 0x1B80, 0xF0110295, + 0x1B80, 0xF0110297, + 0x1B80, 0xF11102A5, + 0x1B80, 0xF11102A7, + 0x1B80, 0xF21102B5, + 0x1B80, 0xF21102B7, + 0x1B80, 0xF31102C5, + 0x1B80, 0xF31102C7, + 0x1B80, 0xF41102D5, + 0x1B80, 0xF41102D7, + 0x1B80, 0xF51102E5, + 0x1B80, 0xF51102E7, + 0x1B80, 0xF61102F5, + 0x1B80, 0xF61102F7, + 0x1B80, 0xF7110305, + 0x1B80, 0xF7110307, + 0x1B80, 0xF8110315, + 0x1B80, 0xF8110317, + 0x1B80, 0xF9110325, + 0x1B80, 0xF9110327, + 0x1B80, 0xFA110335, + 0x1B80, 0xFA110337, + 0x1B80, 0xFB110345, + 0x1B80, 0xFB110347, + 0x1B80, 0xFC110355, + 0x1B80, 0xFC110357, + 0x1B80, 0xFD110365, + 0x1B80, 0xFD110367, + 0x1B80, 0xFE110375, + 0x1B80, 0xFE110377, + 0x1B80, 0xFF110385, + 0x1B80, 0xFF110387, + 0x1B80, 0x00010395, + 0x1B80, 0x00010397, + 0x1B80, 0x305103A5, + 0x1B80, 0x305103A7, + 0x1B80, 0x306903B5, + 0x1B80, 0x306903B7, + 0x1B80, 0x30B403C5, + 0x1B80, 0x30B403C7, + 0x1B80, 0x30B703D5, + 0x1B80, 0x30B703D7, + 0x1B80, 0x306B03E5, + 0x1B80, 0x306B03E7, + 0x1B80, 0x307603F5, + 0x1B80, 0x307603F7, + 0x1B80, 0x30810405, + 0x1B80, 0x30810407, + 0x1B80, 0x30C10415, + 0x1B80, 0x30C10417, + 0x1B80, 0x30BB0425, + 0x1B80, 0x30BB0427, + 0x1B80, 0x30CF0435, + 0x1B80, 0x30CF0437, + 0x1B80, 0x30DA0445, + 0x1B80, 0x30DA0447, + 0x1B80, 0x30E50455, + 0x1B80, 0x30E50457, + 0x1B80, 0x304A0465, + 0x1B80, 0x304A0467, + 0x1B80, 0x31140475, + 0x1B80, 0x31140477, + 0x1B80, 0x31250485, + 0x1B80, 0x31250487, + 0x1B80, 0x313A0495, + 0x1B80, 0x313A0497, + 0x1B80, 0x4D0404A5, + 0x1B80, 0x4D0404A7, + 0x1B80, 0x2EF004B5, + 0x1B80, 0x2EF004B7, + 0x1B80, 0x000004C5, + 0x1B80, 0x000004C7, + 0x1B80, 0x208104D5, + 0x1B80, 0x208104D7, + 0x1B80, 0xA3B504E5, + 0x1B80, 0xA3B504E7, + 0x1B80, 0x4D0004F5, + 0x1B80, 0x4D0004F7, + 0x1B80, 0x30000505, + 0x1B80, 0x30000507, + 0x1B80, 0xE1650515, + 0x1B80, 0xE1650517, + 0x1B80, 0x4D040525, + 0x1B80, 0x4D040527, + 0x1B80, 0x20800535, + 0x1B80, 0x20800537, + 0x1B80, 0x00000545, + 0x1B80, 0x00000547, + 0x1B80, 0x4D000555, + 0x1B80, 0x4D000557, + 0x1B80, 0x55070565, + 0x1B80, 0x55070567, + 0x1B80, 0xE15D0575, + 0x1B80, 0xE15D0577, + 0x1B80, 0xE15D0585, + 0x1B80, 0xE15D0587, + 0x1B80, 0x4D040595, + 0x1B80, 0x4D040597, + 0x1B80, 0x208805A5, + 0x1B80, 0x208805A7, + 0x1B80, 0x020005B5, + 0x1B80, 0x020005B7, + 0x1B80, 0x4D0005C5, + 0x1B80, 0x4D0005C7, + 0x1B80, 0x550F05D5, + 0x1B80, 0x550F05D7, + 0x1B80, 0xE15D05E5, + 0x1B80, 0xE15D05E7, + 0x1B80, 0x4F0205F5, + 0x1B80, 0x4F0205F7, + 0x1B80, 0x4E000605, + 0x1B80, 0x4E000607, + 0x1B80, 0x53020615, + 0x1B80, 0x53020617, + 0x1B80, 0x52010625, + 0x1B80, 0x52010627, + 0x1B80, 0xE1610635, + 0x1B80, 0xE1610637, + 0x1B80, 0x4D080645, + 0x1B80, 0x4D080647, + 0x1B80, 0x57100655, + 0x1B80, 0x57100657, + 0x1B80, 0x57000665, + 0x1B80, 0x57000667, + 0x1B80, 0x4D000675, + 0x1B80, 0x4D000677, + 0x1B80, 0x00010685, + 0x1B80, 0x00010687, + 0x1B80, 0xE1650695, + 0x1B80, 0xE1650697, + 0x1B80, 0x000106A5, + 0x1B80, 0x000106A7, + 0x1B80, 0x308B06B5, + 0x1B80, 0x308B06B7, + 0x1B80, 0x002306C5, + 0x1B80, 0x002306C7, + 0x1B80, 0xE1CB06D5, + 0x1B80, 0xE1CB06D7, + 0x1B80, 0x000206E5, + 0x1B80, 0x000206E7, + 0x1B80, 0x54E906F5, + 0x1B80, 0x54E906F7, + 0x1B80, 0x0BA60705, + 0x1B80, 0x0BA60707, + 0x1B80, 0x00230715, + 0x1B80, 0x00230717, + 0x1B80, 0xE1CB0725, + 0x1B80, 0xE1CB0727, + 0x1B80, 0x00020735, + 0x1B80, 0x00020737, + 0x1B80, 0x4D300745, + 0x1B80, 0x4D300747, + 0x1B80, 0x30A40755, + 0x1B80, 0x30A40757, + 0x1B80, 0x30870765, + 0x1B80, 0x30870767, + 0x1B80, 0x00220775, + 0x1B80, 0x00220777, + 0x1B80, 0xE1CB0785, + 0x1B80, 0xE1CB0787, + 0x1B80, 0x00020795, + 0x1B80, 0x00020797, + 0x1B80, 0x54E807A5, + 0x1B80, 0x54E807A7, + 0x1B80, 0x0BA607B5, + 0x1B80, 0x0BA607B7, + 0x1B80, 0x002207C5, + 0x1B80, 0x002207C7, + 0x1B80, 0xE1CB07D5, + 0x1B80, 0xE1CB07D7, + 0x1B80, 0x000207E5, + 0x1B80, 0x000207E7, + 0x1B80, 0x4D3007F5, + 0x1B80, 0x4D3007F7, + 0x1B80, 0x30A40805, + 0x1B80, 0x30A40807, + 0x1B80, 0x63F10815, + 0x1B80, 0x63F10817, + 0x1B80, 0xE1650825, + 0x1B80, 0xE1650827, + 0x1B80, 0xE1CB0835, + 0x1B80, 0xE1CB0837, + 0x1B80, 0x63F40845, + 0x1B80, 0x63F40847, + 0x1B80, 0xE1650855, + 0x1B80, 0xE1650857, + 0x1B80, 0xE1CB0865, + 0x1B80, 0xE1CB0867, + 0x1B80, 0x0BA80875, + 0x1B80, 0x0BA80877, + 0x1B80, 0x63F80885, + 0x1B80, 0x63F80887, + 0x1B80, 0xE1650895, + 0x1B80, 0xE1650897, + 0x1B80, 0xE1CB08A5, + 0x1B80, 0xE1CB08A7, + 0x1B80, 0x0BA908B5, + 0x1B80, 0x0BA908B7, + 0x1B80, 0x63FC08C5, + 0x1B80, 0x63FC08C7, + 0x1B80, 0xE16508D5, + 0x1B80, 0xE16508D7, + 0x1B80, 0xE1CB08E5, + 0x1B80, 0xE1CB08E7, + 0x1B80, 0x63FF08F5, + 0x1B80, 0x63FF08F7, + 0x1B80, 0xE1650905, + 0x1B80, 0xE1650907, + 0x1B80, 0xE1CB0915, + 0x1B80, 0xE1CB0917, + 0x1B80, 0x63000925, + 0x1B80, 0x63000927, + 0x1B80, 0xE1650935, + 0x1B80, 0xE1650937, + 0x1B80, 0xE1CB0945, + 0x1B80, 0xE1CB0947, + 0x1B80, 0x63030955, + 0x1B80, 0x63030957, + 0x1B80, 0xE1650965, + 0x1B80, 0xE1650967, + 0x1B80, 0xE1CB0975, + 0x1B80, 0xE1CB0977, + 0x1B80, 0xF4D40985, + 0x1B80, 0xF4D40987, + 0x1B80, 0x63070995, + 0x1B80, 0x63070997, + 0x1B80, 0xE16509A5, + 0x1B80, 0xE16509A7, + 0x1B80, 0xE1CB09B5, + 0x1B80, 0xE1CB09B7, + 0x1B80, 0xF5DB09C5, + 0x1B80, 0xF5DB09C7, + 0x1B80, 0x630B09D5, + 0x1B80, 0x630B09D7, + 0x1B80, 0xE16509E5, + 0x1B80, 0xE16509E7, + 0x1B80, 0xE1CB09F5, + 0x1B80, 0xE1CB09F7, + 0x1B80, 0x630E0A05, + 0x1B80, 0x630E0A07, + 0x1B80, 0xE1650A15, + 0x1B80, 0xE1650A17, + 0x1B80, 0xE1CB0A25, + 0x1B80, 0xE1CB0A27, + 0x1B80, 0x4D300A35, + 0x1B80, 0x4D300A37, + 0x1B80, 0x55010A45, + 0x1B80, 0x55010A47, + 0x1B80, 0x57040A55, + 0x1B80, 0x57040A57, + 0x1B80, 0x57000A65, + 0x1B80, 0x57000A67, + 0x1B80, 0x96000A75, + 0x1B80, 0x96000A77, + 0x1B80, 0x57080A85, + 0x1B80, 0x57080A87, + 0x1B80, 0x57000A95, + 0x1B80, 0x57000A97, + 0x1B80, 0x95000AA5, + 0x1B80, 0x95000AA7, + 0x1B80, 0x4D000AB5, + 0x1B80, 0x4D000AB7, + 0x1B80, 0x6C070AC5, + 0x1B80, 0x6C070AC7, + 0x1B80, 0x7B200AD5, + 0x1B80, 0x7B200AD7, + 0x1B80, 0x7A000AE5, + 0x1B80, 0x7A000AE7, + 0x1B80, 0x79000AF5, + 0x1B80, 0x79000AF7, + 0x1B80, 0x7F200B05, + 0x1B80, 0x7F200B07, + 0x1B80, 0x7E000B15, + 0x1B80, 0x7E000B17, + 0x1B80, 0x7D000B25, + 0x1B80, 0x7D000B27, + 0x1B80, 0x00010B35, + 0x1B80, 0x00010B37, + 0x1B80, 0x62850B45, + 0x1B80, 0x62850B47, + 0x1B80, 0xE1650B55, + 0x1B80, 0xE1650B57, + 0x1B80, 0x00010B65, + 0x1B80, 0x00010B67, + 0x1B80, 0x5C320B75, + 0x1B80, 0x5C320B77, + 0x1B80, 0xE1C70B85, + 0x1B80, 0xE1C70B87, + 0x1B80, 0xE1930B95, + 0x1B80, 0xE1930B97, + 0x1B80, 0x00010BA5, + 0x1B80, 0x00010BA7, + 0x1B80, 0x5C320BB5, + 0x1B80, 0x5C320BB7, + 0x1B80, 0x63F40BC5, + 0x1B80, 0x63F40BC7, + 0x1B80, 0x62850BD5, + 0x1B80, 0x62850BD7, + 0x1B80, 0x0BB00BE5, + 0x1B80, 0x0BB00BE7, + 0x1B80, 0xE1650BF5, + 0x1B80, 0xE1650BF7, + 0x1B80, 0xE1CB0C05, + 0x1B80, 0xE1CB0C07, + 0x1B80, 0x5C320C15, + 0x1B80, 0x5C320C17, + 0x1B80, 0x63FC0C25, + 0x1B80, 0x63FC0C27, + 0x1B80, 0x62850C35, + 0x1B80, 0x62850C37, + 0x1B80, 0x0BB10C45, + 0x1B80, 0x0BB10C47, + 0x1B80, 0xE1650C55, + 0x1B80, 0xE1650C57, + 0x1B80, 0xE1CB0C65, + 0x1B80, 0xE1CB0C67, + 0x1B80, 0x63030C75, + 0x1B80, 0x63030C77, + 0x1B80, 0xE1650C85, + 0x1B80, 0xE1650C87, + 0x1B80, 0xE1CB0C95, + 0x1B80, 0xE1CB0C97, + 0x1B80, 0xF7040CA5, + 0x1B80, 0xF7040CA7, + 0x1B80, 0x630B0CB5, + 0x1B80, 0x630B0CB7, + 0x1B80, 0xE1650CC5, + 0x1B80, 0xE1650CC7, + 0x1B80, 0xE1CB0CD5, + 0x1B80, 0xE1CB0CD7, + 0x1B80, 0x00010CE5, + 0x1B80, 0x00010CE7, + 0x1B80, 0x30F30CF5, + 0x1B80, 0x30F30CF7, + 0x1B80, 0x00230D05, + 0x1B80, 0x00230D07, + 0x1B80, 0xE1D00D15, + 0x1B80, 0xE1D00D17, + 0x1B80, 0x00020D25, + 0x1B80, 0x00020D27, + 0x1B80, 0x54E90D35, + 0x1B80, 0x54E90D37, + 0x1B80, 0x0BA60D45, + 0x1B80, 0x0BA60D47, + 0x1B80, 0x00230D55, + 0x1B80, 0x00230D57, + 0x1B80, 0xE1D00D65, + 0x1B80, 0xE1D00D67, + 0x1B80, 0x00020D75, + 0x1B80, 0x00020D77, + 0x1B80, 0x4D100D85, + 0x1B80, 0x4D100D87, + 0x1B80, 0x30A40D95, + 0x1B80, 0x30A40D97, + 0x1B80, 0x30ED0DA5, + 0x1B80, 0x30ED0DA7, + 0x1B80, 0x00220DB5, + 0x1B80, 0x00220DB7, + 0x1B80, 0xE1D00DC5, + 0x1B80, 0xE1D00DC7, + 0x1B80, 0x00020DD5, + 0x1B80, 0x00020DD7, + 0x1B80, 0x54E80DE5, + 0x1B80, 0x54E80DE7, + 0x1B80, 0x0BA60DF5, + 0x1B80, 0x0BA60DF7, + 0x1B80, 0x00220E05, + 0x1B80, 0x00220E07, + 0x1B80, 0xE1D00E15, + 0x1B80, 0xE1D00E17, + 0x1B80, 0x00020E25, + 0x1B80, 0x00020E27, + 0x1B80, 0x4D100E35, + 0x1B80, 0x4D100E37, + 0x1B80, 0x30A40E45, + 0x1B80, 0x30A40E47, + 0x1B80, 0x5C320E55, + 0x1B80, 0x5C320E57, + 0x1B80, 0x54F00E65, + 0x1B80, 0x54F00E67, + 0x1B80, 0x67F10E75, + 0x1B80, 0x67F10E77, + 0x1B80, 0xE1930E85, + 0x1B80, 0xE1930E87, + 0x1B80, 0xE1D00E95, + 0x1B80, 0xE1D00E97, + 0x1B80, 0x67F40EA5, + 0x1B80, 0x67F40EA7, + 0x1B80, 0xE1930EB5, + 0x1B80, 0xE1930EB7, + 0x1B80, 0xE1D00EC5, + 0x1B80, 0xE1D00EC7, + 0x1B80, 0x5C320ED5, + 0x1B80, 0x5C320ED7, + 0x1B80, 0x54F10EE5, + 0x1B80, 0x54F10EE7, + 0x1B80, 0x0BA80EF5, + 0x1B80, 0x0BA80EF7, + 0x1B80, 0x67F80F05, + 0x1B80, 0x67F80F07, + 0x1B80, 0xE1930F15, + 0x1B80, 0xE1930F17, + 0x1B80, 0xE1D00F25, + 0x1B80, 0xE1D00F27, + 0x1B80, 0x5C320F35, + 0x1B80, 0x5C320F37, + 0x1B80, 0x54F10F45, + 0x1B80, 0x54F10F47, + 0x1B80, 0x0BA90F55, + 0x1B80, 0x0BA90F57, + 0x1B80, 0x67FC0F65, + 0x1B80, 0x67FC0F67, + 0x1B80, 0xE1930F75, + 0x1B80, 0xE1930F77, + 0x1B80, 0xE1D00F85, + 0x1B80, 0xE1D00F87, + 0x1B80, 0x67FF0F95, + 0x1B80, 0x67FF0F97, + 0x1B80, 0xE1930FA5, + 0x1B80, 0xE1930FA7, + 0x1B80, 0xE1D00FB5, + 0x1B80, 0xE1D00FB7, + 0x1B80, 0x5C320FC5, + 0x1B80, 0x5C320FC7, + 0x1B80, 0x54F20FD5, + 0x1B80, 0x54F20FD7, + 0x1B80, 0x67000FE5, + 0x1B80, 0x67000FE7, + 0x1B80, 0xE1930FF5, + 0x1B80, 0xE1930FF7, + 0x1B80, 0xE1D01005, + 0x1B80, 0xE1D01007, + 0x1B80, 0x67031015, + 0x1B80, 0x67031017, + 0x1B80, 0xE1931025, + 0x1B80, 0xE1931027, + 0x1B80, 0xE1D01035, + 0x1B80, 0xE1D01037, + 0x1B80, 0xF9CC1045, + 0x1B80, 0xF9CC1047, + 0x1B80, 0x67071055, + 0x1B80, 0x67071057, + 0x1B80, 0xE1931065, + 0x1B80, 0xE1931067, + 0x1B80, 0xE1D01075, + 0x1B80, 0xE1D01077, + 0x1B80, 0xFAD31085, + 0x1B80, 0xFAD31087, + 0x1B80, 0x5C321095, + 0x1B80, 0x5C321097, + 0x1B80, 0x54F310A5, + 0x1B80, 0x54F310A7, + 0x1B80, 0x670B10B5, + 0x1B80, 0x670B10B7, + 0x1B80, 0xE19310C5, + 0x1B80, 0xE19310C7, + 0x1B80, 0xE1D010D5, + 0x1B80, 0xE1D010D7, + 0x1B80, 0x670E10E5, + 0x1B80, 0x670E10E7, + 0x1B80, 0xE19310F5, + 0x1B80, 0xE19310F7, + 0x1B80, 0xE1D01105, + 0x1B80, 0xE1D01107, + 0x1B80, 0x4D101115, + 0x1B80, 0x4D101117, + 0x1B80, 0x30A41125, + 0x1B80, 0x30A41127, + 0x1B80, 0x00011135, + 0x1B80, 0x00011137, + 0x1B80, 0x6C001145, + 0x1B80, 0x6C001147, + 0x1B80, 0x00061155, + 0x1B80, 0x00061157, + 0x1B80, 0x53001165, + 0x1B80, 0x53001167, + 0x1B80, 0x57F71175, + 0x1B80, 0x57F71177, + 0x1B80, 0x58211185, + 0x1B80, 0x58211187, + 0x1B80, 0x592E1195, + 0x1B80, 0x592E1197, + 0x1B80, 0x5A3811A5, + 0x1B80, 0x5A3811A7, + 0x1B80, 0x5B4111B5, + 0x1B80, 0x5B4111B7, + 0x1B80, 0x000711C5, + 0x1B80, 0x000711C7, + 0x1B80, 0x5C0011D5, + 0x1B80, 0x5C0011D7, + 0x1B80, 0x4B0011E5, + 0x1B80, 0x4B0011E7, + 0x1B80, 0x4E8F11F5, + 0x1B80, 0x4E8F11F7, + 0x1B80, 0x4F151205, + 0x1B80, 0x4F151207, + 0x1B80, 0x00041215, + 0x1B80, 0x00041217, + 0x1B80, 0xE1B51225, + 0x1B80, 0xE1B51227, + 0x1B80, 0xAB001235, + 0x1B80, 0xAB001237, + 0x1B80, 0x00011245, + 0x1B80, 0x00011247, + 0x1B80, 0x6C001255, + 0x1B80, 0x6C001257, + 0x1B80, 0x00061265, + 0x1B80, 0x00061267, + 0x1B80, 0x53001275, + 0x1B80, 0x53001277, + 0x1B80, 0x57F71285, + 0x1B80, 0x57F71287, + 0x1B80, 0x58211295, + 0x1B80, 0x58211297, + 0x1B80, 0x592E12A5, + 0x1B80, 0x592E12A7, + 0x1B80, 0x5A3812B5, + 0x1B80, 0x5A3812B7, + 0x1B80, 0x5B4112C5, + 0x1B80, 0x5B4112C7, + 0x1B80, 0x000712D5, + 0x1B80, 0x000712D7, + 0x1B80, 0x5C0012E5, + 0x1B80, 0x5C0012E7, + 0x1B80, 0x4B4012F5, + 0x1B80, 0x4B4012F7, + 0x1B80, 0x4E971305, + 0x1B80, 0x4E971307, + 0x1B80, 0x4F111315, + 0x1B80, 0x4F111317, + 0x1B80, 0x00041325, + 0x1B80, 0x00041327, + 0x1B80, 0xE1B51335, + 0x1B80, 0xE1B51337, + 0x1B80, 0xAB001345, + 0x1B80, 0xAB001347, + 0x1B80, 0x8B001355, + 0x1B80, 0x8B001357, + 0x1B80, 0xAB001365, + 0x1B80, 0xAB001367, + 0x1B80, 0x8A191375, + 0x1B80, 0x8A191377, + 0x1B80, 0x301D1385, + 0x1B80, 0x301D1387, + 0x1B80, 0x00011395, + 0x1B80, 0x00011397, + 0x1B80, 0x6C0113A5, + 0x1B80, 0x6C0113A7, + 0x1B80, 0x000613B5, + 0x1B80, 0x000613B7, + 0x1B80, 0x530113C5, + 0x1B80, 0x530113C7, + 0x1B80, 0x57F713D5, + 0x1B80, 0x57F713D7, + 0x1B80, 0x582113E5, + 0x1B80, 0x582113E7, + 0x1B80, 0x592E13F5, + 0x1B80, 0x592E13F7, + 0x1B80, 0x5A381405, + 0x1B80, 0x5A381407, + 0x1B80, 0x5B411415, + 0x1B80, 0x5B411417, + 0x1B80, 0x00071425, + 0x1B80, 0x00071427, + 0x1B80, 0x5C001435, + 0x1B80, 0x5C001437, + 0x1B80, 0x4B001445, + 0x1B80, 0x4B001447, + 0x1B80, 0x4E871455, + 0x1B80, 0x4E871457, + 0x1B80, 0x4F111465, + 0x1B80, 0x4F111467, + 0x1B80, 0x00041475, + 0x1B80, 0x00041477, + 0x1B80, 0xE1B51485, + 0x1B80, 0xE1B51487, + 0x1B80, 0xAB001495, + 0x1B80, 0xAB001497, + 0x1B80, 0x000614A5, + 0x1B80, 0x000614A7, + 0x1B80, 0x577714B5, + 0x1B80, 0x577714B7, + 0x1B80, 0x000714C5, + 0x1B80, 0x000714C7, + 0x1B80, 0x4E8614D5, + 0x1B80, 0x4E8614D7, + 0x1B80, 0x000414E5, + 0x1B80, 0x000414E7, + 0x1B80, 0x000114F5, + 0x1B80, 0x000114F7, + 0x1B80, 0x00011505, + 0x1B80, 0x00011507, + 0x1B80, 0x7B241515, + 0x1B80, 0x7B241517, + 0x1B80, 0x7A401525, + 0x1B80, 0x7A401527, + 0x1B80, 0x79001535, + 0x1B80, 0x79001537, + 0x1B80, 0x55031545, + 0x1B80, 0x55031547, + 0x1B80, 0x315D1555, + 0x1B80, 0x315D1557, + 0x1B80, 0x7B1C1565, + 0x1B80, 0x7B1C1567, + 0x1B80, 0x7A401575, + 0x1B80, 0x7A401577, + 0x1B80, 0x550B1585, + 0x1B80, 0x550B1587, + 0x1B80, 0x315D1595, + 0x1B80, 0x315D1597, + 0x1B80, 0x7B2015A5, + 0x1B80, 0x7B2015A7, + 0x1B80, 0x7A0015B5, + 0x1B80, 0x7A0015B7, + 0x1B80, 0x551315C5, + 0x1B80, 0x551315C7, + 0x1B80, 0x740115D5, + 0x1B80, 0x740115D7, + 0x1B80, 0x740015E5, + 0x1B80, 0x740015E7, + 0x1B80, 0x8E0015F5, + 0x1B80, 0x8E0015F7, + 0x1B80, 0x00011605, + 0x1B80, 0x00011607, + 0x1B80, 0x57021615, + 0x1B80, 0x57021617, + 0x1B80, 0x57001625, + 0x1B80, 0x57001627, + 0x1B80, 0x97001635, + 0x1B80, 0x97001637, + 0x1B80, 0x00011645, + 0x1B80, 0x00011647, + 0x1B80, 0x4F781655, + 0x1B80, 0x4F781657, + 0x1B80, 0x53881665, + 0x1B80, 0x53881667, + 0x1B80, 0xE1731675, + 0x1B80, 0xE1731677, + 0x1B80, 0x54801685, + 0x1B80, 0x54801687, + 0x1B80, 0x54001695, + 0x1B80, 0x54001697, + 0x1B80, 0xE17316A5, + 0x1B80, 0xE17316A7, + 0x1B80, 0x548116B5, + 0x1B80, 0x548116B7, + 0x1B80, 0x540016C5, + 0x1B80, 0x540016C7, + 0x1B80, 0xE17316D5, + 0x1B80, 0xE17316D7, + 0x1B80, 0x548216E5, + 0x1B80, 0x548216E7, + 0x1B80, 0x540016F5, + 0x1B80, 0x540016F7, + 0x1B80, 0xE17E1705, + 0x1B80, 0xE17E1707, + 0x1B80, 0xBF1D1715, + 0x1B80, 0xBF1D1717, + 0x1B80, 0x301D1725, + 0x1B80, 0x301D1727, + 0x1B80, 0xE1511735, + 0x1B80, 0xE1511737, + 0x1B80, 0xE1561745, + 0x1B80, 0xE1561747, + 0x1B80, 0xE15A1755, + 0x1B80, 0xE15A1757, + 0x1B80, 0xE1611765, + 0x1B80, 0xE1611767, + 0x1B80, 0xE1C71775, + 0x1B80, 0xE1C71777, + 0x1B80, 0x55131785, + 0x1B80, 0x55131787, + 0x1B80, 0xE15D1795, + 0x1B80, 0xE15D1797, + 0x1B80, 0x551517A5, + 0x1B80, 0x551517A7, + 0x1B80, 0xE16117B5, + 0x1B80, 0xE16117B7, + 0x1B80, 0xE1C717C5, + 0x1B80, 0xE1C717C7, + 0x1B80, 0x000117D5, + 0x1B80, 0x000117D7, + 0x1B80, 0x54BF17E5, + 0x1B80, 0x54BF17E7, + 0x1B80, 0x54C017F5, + 0x1B80, 0x54C017F7, + 0x1B80, 0x54A31805, + 0x1B80, 0x54A31807, + 0x1B80, 0x54C11815, + 0x1B80, 0x54C11817, + 0x1B80, 0x54A41825, + 0x1B80, 0x54A41827, + 0x1B80, 0x4C181835, + 0x1B80, 0x4C181837, + 0x1B80, 0xBF071845, + 0x1B80, 0xBF071847, + 0x1B80, 0x54C21855, + 0x1B80, 0x54C21857, + 0x1B80, 0x54A41865, + 0x1B80, 0x54A41867, + 0x1B80, 0xBF041875, + 0x1B80, 0xBF041877, + 0x1B80, 0x54C11885, + 0x1B80, 0x54C11887, + 0x1B80, 0x54A31895, + 0x1B80, 0x54A31897, + 0x1B80, 0xBF0118A5, + 0x1B80, 0xBF0118A7, + 0x1B80, 0xE1D518B5, + 0x1B80, 0xE1D518B7, + 0x1B80, 0x54DF18C5, + 0x1B80, 0x54DF18C7, + 0x1B80, 0x000118D5, + 0x1B80, 0x000118D7, + 0x1B80, 0x54BF18E5, + 0x1B80, 0x54BF18E7, + 0x1B80, 0x54E518F5, + 0x1B80, 0x54E518F7, + 0x1B80, 0x050A1905, + 0x1B80, 0x050A1907, + 0x1B80, 0x54DF1915, + 0x1B80, 0x54DF1917, + 0x1B80, 0x00011925, + 0x1B80, 0x00011927, + 0x1B80, 0x7F201935, + 0x1B80, 0x7F201937, + 0x1B80, 0x7E001945, + 0x1B80, 0x7E001947, + 0x1B80, 0x7D001955, + 0x1B80, 0x7D001957, + 0x1B80, 0x55011965, + 0x1B80, 0x55011967, + 0x1B80, 0x5C311975, + 0x1B80, 0x5C311977, + 0x1B80, 0xE15D1985, + 0x1B80, 0xE15D1987, + 0x1B80, 0xE1611995, + 0x1B80, 0xE1611997, + 0x1B80, 0x548019A5, + 0x1B80, 0x548019A7, + 0x1B80, 0x540019B5, + 0x1B80, 0x540019B7, + 0x1B80, 0xE15D19C5, + 0x1B80, 0xE15D19C7, + 0x1B80, 0xE16119D5, + 0x1B80, 0xE16119D7, + 0x1B80, 0x548119E5, + 0x1B80, 0x548119E7, + 0x1B80, 0x540019F5, + 0x1B80, 0x540019F7, + 0x1B80, 0xE15D1A05, + 0x1B80, 0xE15D1A07, + 0x1B80, 0xE1611A15, + 0x1B80, 0xE1611A17, + 0x1B80, 0x54821A25, + 0x1B80, 0x54821A27, + 0x1B80, 0x54001A35, + 0x1B80, 0x54001A37, + 0x1B80, 0xE17E1A45, + 0x1B80, 0xE17E1A47, + 0x1B80, 0xBFE91A55, + 0x1B80, 0xBFE91A57, + 0x1B80, 0x301D1A65, + 0x1B80, 0x301D1A67, + 0x1B80, 0x00231A75, + 0x1B80, 0x00231A77, + 0x1B80, 0x7B201A85, + 0x1B80, 0x7B201A87, + 0x1B80, 0x7A001A95, + 0x1B80, 0x7A001A97, + 0x1B80, 0x79001AA5, + 0x1B80, 0x79001AA7, + 0x1B80, 0xE1CB1AB5, + 0x1B80, 0xE1CB1AB7, + 0x1B80, 0x00021AC5, + 0x1B80, 0x00021AC7, + 0x1B80, 0x00011AD5, + 0x1B80, 0x00011AD7, + 0x1B80, 0x00221AE5, + 0x1B80, 0x00221AE7, + 0x1B80, 0x7B201AF5, + 0x1B80, 0x7B201AF7, + 0x1B80, 0x7A001B05, + 0x1B80, 0x7A001B07, + 0x1B80, 0x79001B15, + 0x1B80, 0x79001B17, + 0x1B80, 0xE1CB1B25, + 0x1B80, 0xE1CB1B27, + 0x1B80, 0x00021B35, + 0x1B80, 0x00021B37, + 0x1B80, 0x00011B45, + 0x1B80, 0x00011B47, + 0x1B80, 0x74021B55, + 0x1B80, 0x74021B57, + 0x1B80, 0x003F1B65, + 0x1B80, 0x003F1B67, + 0x1B80, 0x74001B75, + 0x1B80, 0x74001B77, + 0x1B80, 0x00021B85, + 0x1B80, 0x00021B87, + 0x1B80, 0x00011B95, + 0x1B80, 0x00011B97, + 0x1B80, 0x4D041BA5, + 0x1B80, 0x4D041BA7, + 0x1B80, 0x2EF81BB5, + 0x1B80, 0x2EF81BB7, + 0x1B80, 0x00001BC5, + 0x1B80, 0x00001BC7, + 0x1B80, 0x23301BD5, + 0x1B80, 0x23301BD7, + 0x1B80, 0x00241BE5, + 0x1B80, 0x00241BE7, + 0x1B80, 0x23E01BF5, + 0x1B80, 0x23E01BF7, + 0x1B80, 0x003F1C05, + 0x1B80, 0x003F1C07, + 0x1B80, 0x23FC1C15, + 0x1B80, 0x23FC1C17, + 0x1B80, 0xBFCE1C25, + 0x1B80, 0xBFCE1C27, + 0x1B80, 0x2EF01C35, + 0x1B80, 0x2EF01C37, + 0x1B80, 0x00001C45, + 0x1B80, 0x00001C47, + 0x1B80, 0x4D001C55, + 0x1B80, 0x4D001C57, + 0x1B80, 0x00011C65, + 0x1B80, 0x00011C67, + 0x1B80, 0x549F1C75, + 0x1B80, 0x549F1C77, + 0x1B80, 0x54FF1C85, + 0x1B80, 0x54FF1C87, + 0x1B80, 0x54001C95, + 0x1B80, 0x54001C97, + 0x1B80, 0x00011CA5, + 0x1B80, 0x00011CA7, + 0x1B80, 0x5C311CB5, + 0x1B80, 0x5C311CB7, + 0x1B80, 0x07141CC5, + 0x1B80, 0x07141CC7, + 0x1B80, 0x54001CD5, + 0x1B80, 0x54001CD7, + 0x1B80, 0x5C321CE5, + 0x1B80, 0x5C321CE7, + 0x1B80, 0x00011CF5, + 0x1B80, 0x00011CF7, + 0x1B80, 0x5C321D05, + 0x1B80, 0x5C321D07, + 0x1B80, 0x07141D15, + 0x1B80, 0x07141D17, + 0x1B80, 0x54001D25, + 0x1B80, 0x54001D27, + 0x1B80, 0x5C311D35, + 0x1B80, 0x5C311D37, + 0x1B80, 0x00011D45, + 0x1B80, 0x00011D47, + 0x1B80, 0x4C981D55, + 0x1B80, 0x4C981D57, + 0x1B80, 0x4C181D65, + 0x1B80, 0x4C181D67, + 0x1B80, 0x00011D75, + 0x1B80, 0x00011D77, + 0x1B80, 0x5C321D85, + 0x1B80, 0x5C321D87, + 0x1B80, 0x62841D95, + 0x1B80, 0x62841D97, + 0x1B80, 0x66861DA5, + 0x1B80, 0x66861DA7, + 0x1B80, 0x6C031DB5, + 0x1B80, 0x6C031DB7, + 0x1B80, 0x7B201DC5, + 0x1B80, 0x7B201DC7, + 0x1B80, 0x7A001DD5, + 0x1B80, 0x7A001DD7, + 0x1B80, 0x79001DE5, + 0x1B80, 0x79001DE7, + 0x1B80, 0x7F201DF5, + 0x1B80, 0x7F201DF7, + 0x1B80, 0x7E001E05, + 0x1B80, 0x7E001E07, + 0x1B80, 0x7D001E15, + 0x1B80, 0x7D001E17, + 0x1B80, 0x09011E25, + 0x1B80, 0x09011E27, + 0x1B80, 0x0C011E35, + 0x1B80, 0x0C011E37, + 0x1B80, 0x0BA61E45, + 0x1B80, 0x0BA61E47, + 0x1B80, 0x00011E55, + 0x1B80, 0x00011E57, + 0x1B80, 0x00000006, + 0x1B80, 0x00000002, +}; + +RTW_DECL_TABLE_PHY_COND(rtw8822b_bb, rtw_phy_cfg_bb); + +static const u32 rtw8822b_bb_pg_type2[] = { + 0, 0, 0, 0x00000c20, 0xffffffff, 0x32343638, + 0, 0, 0, 0x00000c24, 0xffffffff, 0x36384042, + 0, 0, 0, 0x00000c28, 0xffffffff, 0x28303234, + 0, 0, 0, 0x00000c2c, 0xffffffff, 0x34363840, + 0, 0, 0, 0x00000c30, 0xffffffff, 0x26283032, + 0, 0, 1, 0x00000c34, 0xffffffff, 0x34363840, + 0, 0, 1, 0x00000c38, 0xffffffff, 0x26283032, + 0, 0, 0, 0x00000c3c, 0xffffffff, 0x34363840, + 0, 0, 0, 0x00000c40, 0xffffffff, 0x26283032, + 0, 0, 0, 0x00000c44, 0xffffffff, 0x38402224, + 0, 0, 1, 0x00000c48, 0xffffffff, 0x30323436, + 0, 0, 1, 0x00000c4c, 0xffffffff, 0x22242628, + 0, 1, 0, 0x00000e20, 0xffffffff, 0x32343638, + 0, 1, 0, 0x00000e24, 0xffffffff, 0x36384042, + 0, 1, 0, 0x00000e28, 0xffffffff, 0x28303234, + 0, 1, 0, 0x00000e2c, 0xffffffff, 0x34363840, + 0, 1, 0, 0x00000e30, 0xffffffff, 0x26283032, + 0, 1, 1, 0x00000e34, 0xffffffff, 0x34363840, + 0, 1, 1, 0x00000e38, 0xffffffff, 0x26283032, + 0, 1, 0, 0x00000e3c, 0xffffffff, 0x34363840, + 0, 1, 0, 0x00000e40, 0xffffffff, 0x26283032, + 0, 1, 0, 0x00000e44, 0xffffffff, 0x38402224, + 0, 1, 1, 0x00000e48, 0xffffffff, 0x30323436, + 0, 1, 1, 0x00000e4c, 0xffffffff, 0x22242628, + 1, 0, 0, 0x00000c24, 0xffffffff, 0x40424446, + 1, 0, 0, 0x00000c28, 0xffffffff, 0x32343638, + 1, 0, 0, 0x00000c2c, 0xffffffff, 0x38404244, + 1, 0, 0, 0x00000c30, 0xffffffff, 0x30323436, + 1, 0, 1, 0x00000c34, 0xffffffff, 0x38404244, + 1, 0, 1, 0x00000c38, 0xffffffff, 0x30323436, + 1, 0, 0, 0x00000c3c, 0xffffffff, 0x38404244, + 1, 0, 0, 0x00000c40, 0xffffffff, 0x30323436, + 1, 0, 0, 0x00000c44, 0xffffffff, 0x42442628, + 1, 0, 1, 0x00000c48, 0xffffffff, 0x34363840, + 1, 0, 1, 0x00000c4c, 0xffffffff, 0x26283032, + 1, 1, 0, 0x00000e24, 0xffffffff, 0x40424446, + 1, 1, 0, 0x00000e28, 0xffffffff, 0x32343638, + 1, 1, 0, 0x00000e2c, 0xffffffff, 0x38404244, + 1, 1, 0, 0x00000e30, 0xffffffff, 0x30323436, + 1, 1, 1, 0x00000e34, 0xffffffff, 0x38404244, + 1, 1, 1, 0x00000e38, 0xffffffff, 0x30323436, + 1, 1, 0, 0x00000e3c, 0xffffffff, 0x38404244, + 1, 1, 0, 0x00000e40, 0xffffffff, 0x30323436, + 1, 1, 0, 0x00000e44, 0xffffffff, 0x42442628, + 1, 1, 1, 0x00000e48, 0xffffffff, 0x34363840, + 1, 1, 1, 0x00000e4c, 0xffffffff, 0x26283032 +}; + +RTW_DECL_TABLE_BB_PG(rtw8822b_bb_pg_type2); + +static const u32 rtw8822b_bb_pg_type5[] = { + 0, 0, 0, 0x00000c20, 0xffffffff, 0x32343638, + 0, 0, 0, 0x00000c24, 0xffffffff, 0x36384042, + 0, 0, 0, 0x00000c28, 0xffffffff, 0x28303234, + 0, 0, 0, 0x00000c2c, 0xffffffff, 0x34363840, + 0, 0, 0, 0x00000c30, 0xffffffff, 0x26283032, + 0, 0, 1, 0x00000c34, 0xffffffff, 0x34363840, + 0, 0, 1, 0x00000c38, 0xffffffff, 0x26283032, + 0, 0, 0, 0x00000c3c, 0xffffffff, 0x34363840, + 0, 0, 0, 0x00000c40, 0xffffffff, 0x26283032, + 0, 0, 0, 0x00000c44, 0xffffffff, 0x38402224, + 0, 0, 1, 0x00000c48, 0xffffffff, 0x30323436, + 0, 0, 1, 0x00000c4c, 0xffffffff, 0x22242628, + 0, 1, 0, 0x00000e20, 0xffffffff, 0x32343638, + 0, 1, 0, 0x00000e24, 0xffffffff, 0x36384042, + 0, 1, 0, 0x00000e28, 0xffffffff, 0x28303234, + 0, 1, 0, 0x00000e2c, 0xffffffff, 0x34363840, + 0, 1, 0, 0x00000e30, 0xffffffff, 0x26283032, + 0, 1, 1, 0x00000e34, 0xffffffff, 0x34363840, + 0, 1, 1, 0x00000e38, 0xffffffff, 0x26283032, + 0, 1, 0, 0x00000e3c, 0xffffffff, 0x34363840, + 0, 1, 0, 0x00000e40, 0xffffffff, 0x26283032, + 0, 1, 0, 0x00000e44, 0xffffffff, 0x38402224, + 0, 1, 1, 0x00000e48, 0xffffffff, 0x30323436, + 0, 1, 1, 0x00000e4c, 0xffffffff, 0x22242628, + 1, 0, 0, 0x00000c24, 0xffffffff, 0x34363840, + 1, 0, 0, 0x00000c28, 0xffffffff, 0x26283032, + 1, 0, 0, 0x00000c2c, 0xffffffff, 0x32343638, + 1, 0, 0, 0x00000c30, 0xffffffff, 0x24262830, + 1, 0, 1, 0x00000c34, 0xffffffff, 0x32343638, + 1, 0, 1, 0x00000c38, 0xffffffff, 0x24262830, + 1, 0, 0, 0x00000c3c, 0xffffffff, 0x32343638, + 1, 0, 0, 0x00000c40, 0xffffffff, 0x24262830, + 1, 0, 0, 0x00000c44, 0xffffffff, 0x36382022, + 1, 0, 1, 0x00000c48, 0xffffffff, 0x28303234, + 1, 0, 1, 0x00000c4c, 0xffffffff, 0x20222426, + 1, 1, 0, 0x00000e24, 0xffffffff, 0x34363840, + 1, 1, 0, 0x00000e28, 0xffffffff, 0x26283032, + 1, 1, 0, 0x00000e2c, 0xffffffff, 0x32343638, + 1, 1, 0, 0x00000e30, 0xffffffff, 0x24262830, + 1, 1, 1, 0x00000e34, 0xffffffff, 0x32343638, + 1, 1, 1, 0x00000e38, 0xffffffff, 0x24262830, + 1, 1, 0, 0x00000e3c, 0xffffffff, 0x32343638, + 1, 1, 0, 0x00000e40, 0xffffffff, 0x24262830, + 1, 1, 0, 0x00000e44, 0xffffffff, 0x36382022, + 1, 1, 1, 0x00000e48, 0xffffffff, 0x28303234, + 1, 1, 1, 0x00000e4c, 0xffffffff, 0x20222426 +}; + +RTW_DECL_TABLE_BB_PG(rtw8822b_bb_pg_type5); + +static const u32 rtw8822b_rf_a[] = { + 0x000, 0x00030000, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x0004002D, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x0004002D, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x0004002D, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x0004002D, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x0004002D, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x0004002D, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x0004002D, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x0004002D, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x0004002D, + 0xA0000000, 0x00000000, + 0x001, 0x00040029, + 0xB0000000, 0x00000000, + 0x018, 0x00010D24, + 0x0EF, 0x00080000, + 0x033, 0x00000002, + 0x03E, 0x0000003F, + 0x8300000c, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000D0F4E, + 0xA0000000, 0x00000000, + 0x03F, 0x000C0F4E, + 0xB0000000, 0x00000000, + 0x033, 0x00000001, + 0x03E, 0x00000034, + 0x03F, 0x0004080E, + 0x0EF, 0x00080000, + 0x0DF, 0x00002449, + 0x033, 0x00000024, + 0x03E, 0x0000003F, + 0x03F, 0x00060FDE, + 0x0EF, 0x00000000, + 0x0EF, 0x00080000, + 0x033, 0x00000025, + 0x03E, 0x00000037, + 0x03F, 0x0007EFCE, + 0x0EF, 0x00000000, + 0x0EF, 0x00080000, + 0x033, 0x00000026, + 0x03E, 0x00000037, + 0x03F, 0x000DEFCE, + 0x0EF, 0x00000000, + 0x07F, 0x00000000, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FF0F8, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FF0F8, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FB0F8, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FB0F8, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FB0F8, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FB0F8, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FF0F8, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FF0F8, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FB0F8, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FF0F8, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FF0F8, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FB0F8, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FB0F8, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FF0F8, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FF0F8, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FB0F8, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FB0F8, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FB0F8, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FB0F8, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FF0F8, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FB0F8, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FB0F8, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FB0F8, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FB0F8, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FF0F8, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x0B0, 0x000FF0F8, + 0xA0000000, 0x00000000, + 0x0B0, 0x000FF0F8, + 0xB0000000, 0x00000000, + 0x0B1, 0x0007DBE4, + 0x0B2, 0x000225D1, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x000FC760, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x000FC760, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x000FC760, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x0007C330, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x000FC760, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x000FC760, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x000FC760, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x000FC760, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x0003C360, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x000FC760, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x000FC760, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x000FC760, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x000FC760, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x000FC760, + 0xA0000000, 0x00000000, + 0x0B3, 0x000FC760, + 0xB0000000, 0x00000000, + 0x0B4, 0x00099DD0, + 0x0B5, 0x000400FC, + 0x0B6, 0x000187F0, + 0x0B7, 0x00030018, + 0x0B8, 0x00080800, + 0x0B9, 0x00000000, + 0x0BA, 0x00008000, + 0x0BB, 0x00000000, + 0x0BC, 0x00040030, + 0x0BD, 0x00000000, + 0x0BE, 0x00000000, + 0x0BF, 0x00000000, + 0x0C0, 0x00000000, + 0x0C1, 0x00000000, + 0x0C2, 0x00000000, + 0x0C3, 0x00000000, + 0x0C4, 0x00002402, + 0x0C5, 0x00000009, + 0x0C6, 0x00040299, + 0x0C7, 0x00055555, + 0x0C8, 0x0000C16C, + 0x0C9, 0x0001C146, + 0x0CA, 0x00000000, + 0x0CB, 0x00000000, + 0x0CC, 0x00000000, + 0x0CD, 0x00000000, + 0x0CE, 0x00090C00, + 0x0CF, 0x0006D200, + 0x0DF, 0x00000009, + 0x018, 0x00010524, + 0x089, 0x00000207, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FF186, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FF186, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FF186, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FE186, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FE186, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FF186, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FF186, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FF186, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FF186, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FF186, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FF186, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FE186, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FF186, + 0xA0000000, 0x00000000, + 0x08A, 0x000FF186, + 0xB0000000, 0x00000000, + 0x08B, 0x00061E3C, + 0x08C, 0x000112C7, + 0x08D, 0x000F4988, + 0x08E, 0x00064D40, + 0x0EF, 0x00020000, + 0x033, 0x00000007, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004080, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0xA0000000, 0x00000000, + 0x03E, 0x00004000, + 0xB0000000, 0x00000000, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C0006, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000DFF86, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C0006, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000DFF86, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C0006, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C0006, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C0006, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0xA0000000, 0x00000000, + 0x03F, 0x000C3186, + 0xB0000000, 0x00000000, + 0x033, 0x00000006, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004080, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004080, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004080, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004080, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004080, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004080, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004080, + 0xA0000000, 0x00000000, + 0x03E, 0x00004080, + 0xB0000000, 0x00000000, + 0x03F, 0x000C3186, + 0x033, 0x00000005, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x000040C8, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x000040C8, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x000040C8, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x000040C8, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x000040C8, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x000040C8, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004084, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x000040C8, + 0xA0000000, 0x00000000, + 0x03E, 0x000040C8, + 0xB0000000, 0x00000000, + 0x03F, 0x000C3186, + 0x033, 0x00000004, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004190, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004190, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004190, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004190, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004190, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004190, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004108, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004190, + 0xA0000000, 0x00000000, + 0x03E, 0x00004190, + 0xB0000000, 0x00000000, + 0x03F, 0x000C3186, + 0x033, 0x00000003, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004998, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004998, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004998, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004998, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004998, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004998, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x0000490C, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004998, + 0xA0000000, 0x00000000, + 0x03E, 0x00004998, + 0xB0000000, 0x00000000, + 0x03F, 0x000C3186, + 0x033, 0x00000002, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00005840, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00005840, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00005840, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00005840, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00005840, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00005840, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00005E00, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00005840, + 0xA0000000, 0x00000000, + 0x03E, 0x00005840, + 0xB0000000, 0x00000000, + 0x03F, 0x000C3186, + 0x033, 0x00000001, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x000058C2, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x000058C2, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x000058C2, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x000058C2, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x000058C2, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x000058C2, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00005862, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x000058C2, + 0xA0000000, 0x00000000, + 0x03E, 0x000058C2, + 0xB0000000, 0x00000000, + 0x03F, 0x000C3186, + 0x033, 0x00000000, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00005930, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00005930, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00005930, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00005930, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00005930, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00005930, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00005948, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00005930, + 0xA0000000, 0x00000000, + 0x03E, 0x00005930, + 0xB0000000, 0x00000000, + 0x03F, 0x000C3186, + 0x033, 0x0000000F, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004080, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004080, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0xA0000000, 0x00000000, + 0x03E, 0x00004000, + 0xB0000000, 0x00000000, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C0006, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000DFF86, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000DFF86, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C0006, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C0006, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C0006, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C0006, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0xA0000000, 0x00000000, + 0x03F, 0x000C3186, + 0xB0000000, 0x00000000, + 0x033, 0x0000000E, + 0x03E, 0x00004080, + 0x03F, 0x000C3186, + 0x033, 0x0000000D, + 0x03E, 0x000040C8, + 0x03F, 0x000C3186, + 0x033, 0x0000000C, + 0x03E, 0x00004190, + 0x03F, 0x000C3186, + 0x033, 0x0000000B, + 0x03E, 0x00004998, + 0x03F, 0x000C3186, + 0x033, 0x0000000A, + 0x03E, 0x00005840, + 0x03F, 0x000C3186, + 0x033, 0x00000009, + 0x03E, 0x000058C2, + 0x03F, 0x000C3186, + 0x033, 0x00000008, + 0x03E, 0x00005930, + 0x03F, 0x000C3186, + 0x033, 0x00000017, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004080, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004080, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0xA0000000, 0x00000000, + 0x03E, 0x00004000, + 0xB0000000, 0x00000000, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C0006, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C0006, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000DFF86, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C0006, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C0006, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C0006, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000C3186, + 0xA0000000, 0x00000000, + 0x03F, 0x000C3186, + 0xB0000000, 0x00000000, + 0x033, 0x00000016, + 0x03E, 0x00004080, + 0x03F, 0x000C3186, + 0x033, 0x00000015, + 0x03E, 0x000040C8, + 0x03F, 0x000C3186, + 0x033, 0x00000014, + 0x03E, 0x00004190, + 0x03F, 0x000C3186, + 0x033, 0x00000013, + 0x03E, 0x00004998, + 0x03F, 0x000C3186, + 0x033, 0x00000012, + 0x03E, 0x00005840, + 0x03F, 0x000C3186, + 0x033, 0x00000011, + 0x03E, 0x000058C2, + 0x03F, 0x000C3186, + 0x033, 0x00000010, + 0x03E, 0x00005930, + 0x03F, 0x000C3186, + 0x0EF, 0x00000000, + 0x0EF, 0x00004000, + 0x033, 0x00000000, + 0x03F, 0x0000000A, + 0x033, 0x00000001, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000005, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000006, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000005, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000005, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000005, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000005, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0xA0000000, 0x00000000, + 0x03F, 0x00000005, + 0xB0000000, 0x00000000, + 0x033, 0x00000002, + 0x03F, 0x00000000, + 0x0EF, 0x00000000, + 0x018, 0x00000401, + 0x084, 0x00001209, + 0x086, 0x000001A0, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x087, 0x00068080, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x087, 0x00068080, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x087, 0x00068080, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x087, 0x00068080, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x087, 0x00068080, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x087, 0x00068080, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x087, 0x00068080, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x087, 0x00068080, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x087, 0x00068080, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x087, 0x00068080, + 0xA0000000, 0x00000000, + 0x087, 0x000E8180, + 0xB0000000, 0x00000000, + 0x088, 0x00070020, + 0x0DE, 0x00000010, + 0x0EF, 0x00008000, + 0x033, 0x0000000F, + 0x03F, 0x0000003C, + 0x033, 0x0000000E, + 0x03F, 0x00000038, + 0x033, 0x0000000D, + 0x03F, 0x00000030, + 0x033, 0x0000000C, + 0x03F, 0x00000028, + 0x033, 0x0000000B, + 0x03F, 0x00000020, + 0x033, 0x0000000A, + 0x03F, 0x00000018, + 0x033, 0x00000009, + 0x03F, 0x00000010, + 0x033, 0x00000008, + 0x03F, 0x00000008, + 0x033, 0x00000007, + 0x03F, 0x0000003C, + 0x033, 0x00000006, + 0x03F, 0x00000038, + 0x033, 0x00000005, + 0x03F, 0x00000030, + 0x033, 0x00000004, + 0x03F, 0x00000028, + 0x033, 0x00000003, + 0x03F, 0x00000020, + 0x033, 0x00000002, + 0x03F, 0x00000018, + 0x033, 0x00000001, + 0x03F, 0x00000010, + 0x033, 0x00000000, + 0x03F, 0x00000008, + 0x0EF, 0x00000000, + 0x0B8, 0x00080A00, + 0x0FE, 0x00000000, + 0x0B0, 0x000FF0FA, + 0x0FE, 0x00000000, + 0x0FE, 0x00000000, + 0x0CA, 0x00080000, + 0x0FE, 0x00000000, + 0x0C9, 0x0001C141, + 0x0FE, 0x00000000, + 0x0FE, 0x00000000, + 0x0B0, 0x000FF0F8, + 0x018, 0x00018D24, + 0xFFE, 0x00000000, + 0xFFE, 0x00000000, + 0xFFE, 0x00000000, + 0xFFE, 0x00000000, + 0x018, 0x00010D24, + 0x01B, 0x00075A40, + 0x0EE, 0x00000002, + 0x033, 0x00000000, + 0x03F, 0x00000004, + 0x033, 0x00000001, + 0x03F, 0x00000004, + 0x033, 0x00000002, + 0x03F, 0x00000004, + 0x033, 0x00000003, + 0x03F, 0x00000004, + 0x033, 0x00000004, + 0x03F, 0x00000004, + 0x033, 0x00000005, + 0x03F, 0x00000006, + 0x033, 0x00000006, + 0x03F, 0x00000004, + 0x033, 0x00000007, + 0x03F, 0x00000000, + 0x0EE, 0x00000000, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D3D1, + 0x062, 0x0000D3A2, + 0x063, 0x00000002, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D4A0, + 0x062, 0x0000D203, + 0x063, 0x00000062, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D2A1, + 0x062, 0x0000D3A2, + 0x063, 0x00000062, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D301, + 0x062, 0x0000D303, + 0x063, 0x00000002, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D2A1, + 0x062, 0x0000D3A2, + 0x063, 0x00000062, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D301, + 0x062, 0x0000D303, + 0x063, 0x00000002, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D4A0, + 0x062, 0x0000D203, + 0x063, 0x00000062, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D4A0, + 0x062, 0x0000D203, + 0x063, 0x00000062, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D301, + 0x062, 0x0000D303, + 0x063, 0x00000002, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D3D1, + 0x062, 0x0000D3A2, + 0x063, 0x00000002, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D3D1, + 0x062, 0x0000D3A2, + 0x063, 0x00000002, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D4A0, + 0x062, 0x0000D203, + 0x063, 0x00000062, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D301, + 0x062, 0x0000D303, + 0x063, 0x00000002, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D3D1, + 0x062, 0x0000D3A2, + 0x063, 0x00000002, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D3D1, + 0x062, 0x0000D3A2, + 0x063, 0x00000002, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D301, + 0x062, 0x0000D303, + 0x063, 0x00000002, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D301, + 0x062, 0x0000D303, + 0x063, 0x00000002, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D301, + 0x062, 0x0000D303, + 0x063, 0x00000002, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D2A1, + 0x062, 0x0000D3A2, + 0x063, 0x00000062, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D4A0, + 0x062, 0x0000D203, + 0x063, 0x00000062, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D2A1, + 0x062, 0x0000D3A2, + 0x063, 0x00000062, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D301, + 0x062, 0x0000D303, + 0x063, 0x00000002, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D2A1, + 0x062, 0x0000D3A2, + 0x063, 0x00000062, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D301, + 0x062, 0x0000D303, + 0x063, 0x00000002, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D4A0, + 0x062, 0x0000D203, + 0x063, 0x00000062, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D4A0, + 0x062, 0x0000D203, + 0x063, 0x00000062, + 0xA0000000, 0x00000000, + 0x061, 0x0005D3D0, + 0x062, 0x0000D303, + 0x063, 0x00000002, + 0xB0000000, 0x00000000, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A3, + 0x030, 0x000093A3, + 0x030, 0x0000A3A3, + 0x030, 0x0000B3A3, + 0x0EF, 0x00000000, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x000004A3, + 0x030, 0x000014A3, + 0x030, 0x000024A3, + 0x030, 0x000034A3, + 0x030, 0x000044A3, + 0x030, 0x000054A3, + 0x030, 0x000064A3, + 0x030, 0x000074A3, + 0x030, 0x000084A3, + 0x030, 0x000094A3, + 0x030, 0x0000A4A3, + 0x030, 0x0000B4A3, + 0x0EF, 0x00000000, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x000002A6, + 0x030, 0x000012A6, + 0x030, 0x000022A6, + 0x030, 0x000032A6, + 0x030, 0x000042A6, + 0x030, 0x000052A6, + 0x030, 0x000062A6, + 0x030, 0x000072A6, + 0x030, 0x000082A6, + 0x030, 0x000092A6, + 0x030, 0x0000A2A6, + 0x030, 0x0000B2A6, + 0x0EF, 0x00000000, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x00000303, + 0x030, 0x00001303, + 0x030, 0x00002303, + 0x030, 0x00003303, + 0x030, 0x000043A4, + 0x030, 0x000053A4, + 0x030, 0x000063A4, + 0x030, 0x000073A4, + 0x030, 0x00008365, + 0x030, 0x00009365, + 0x030, 0x0000A365, + 0x030, 0x0000B365, + 0x0EF, 0x00000000, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x000002A6, + 0x030, 0x000012A6, + 0x030, 0x000022A6, + 0x030, 0x000032A6, + 0x030, 0x000042A6, + 0x030, 0x000052A6, + 0x030, 0x000062A6, + 0x030, 0x000072A6, + 0x030, 0x000082A6, + 0x030, 0x000092A6, + 0x030, 0x0000A2A6, + 0x030, 0x0000B2A6, + 0x0EF, 0x00000000, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x000003A3, + 0x030, 0x000013A3, + 0x030, 0x000023A3, + 0x030, 0x000033A3, + 0x030, 0x00004355, + 0x030, 0x00005355, + 0x030, 0x00006355, + 0x030, 0x00007355, + 0x030, 0x00008315, + 0x030, 0x00009315, + 0x030, 0x0000A315, + 0x030, 0x0000B315, + 0x0EF, 0x00000000, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x000004A3, + 0x030, 0x000014A3, + 0x030, 0x000024A3, + 0x030, 0x000034A3, + 0x030, 0x000044A3, + 0x030, 0x000054A3, + 0x030, 0x000064A3, + 0x030, 0x000074A3, + 0x030, 0x000084A3, + 0x030, 0x000094A3, + 0x030, 0x0000A4A3, + 0x030, 0x0000B4A3, + 0x0EF, 0x00000000, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x000004A3, + 0x030, 0x000014A3, + 0x030, 0x000024A3, + 0x030, 0x000034A3, + 0x030, 0x000044A3, + 0x030, 0x000054A3, + 0x030, 0x000064A3, + 0x030, 0x000074A3, + 0x030, 0x000084A3, + 0x030, 0x000094A3, + 0x030, 0x0000A4A3, + 0x030, 0x0000B4A3, + 0x0EF, 0x00000000, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x00000384, + 0x030, 0x00001384, + 0x030, 0x00002384, + 0x030, 0x00003384, + 0x030, 0x00004425, + 0x030, 0x00005425, + 0x030, 0x00006425, + 0x030, 0x00007425, + 0x030, 0x000084A6, + 0x030, 0x000094A6, + 0x030, 0x0000A4A6, + 0x030, 0x0000B4A6, + 0x0EF, 0x00000000, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x00000463, + 0x030, 0x00001463, + 0x030, 0x00002463, + 0x030, 0x00003463, + 0x030, 0x00004545, + 0x030, 0x00005545, + 0x030, 0x00006545, + 0x030, 0x00007545, + 0x030, 0x00008565, + 0x030, 0x00009565, + 0x030, 0x0000A565, + 0x030, 0x0000B565, + 0x0EF, 0x00000000, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A3, + 0x030, 0x000093A3, + 0x030, 0x0000A3A3, + 0x030, 0x0000B3A3, + 0x0EF, 0x00000000, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x000004A3, + 0x030, 0x000014A3, + 0x030, 0x000024A3, + 0x030, 0x000034A3, + 0x030, 0x000044A3, + 0x030, 0x000054A3, + 0x030, 0x000064A3, + 0x030, 0x000074A3, + 0x030, 0x000084A3, + 0x030, 0x000094A3, + 0x030, 0x0000A4A3, + 0x030, 0x0000B4A3, + 0x0EF, 0x00000000, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x00000443, + 0x030, 0x00001443, + 0x030, 0x00002443, + 0x030, 0x00003443, + 0x030, 0x000043A4, + 0x030, 0x000053A4, + 0x030, 0x000063A4, + 0x030, 0x000073A4, + 0x030, 0x00008365, + 0x030, 0x00009365, + 0x030, 0x0000A365, + 0x030, 0x0000B365, + 0x0EF, 0x00000000, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x00000443, + 0x030, 0x00001443, + 0x030, 0x00002443, + 0x030, 0x00003443, + 0x030, 0x00004483, + 0x030, 0x00005483, + 0x030, 0x00006483, + 0x030, 0x00007483, + 0x030, 0x000084A4, + 0x030, 0x000094A4, + 0x030, 0x0000A4A4, + 0x030, 0x0000B4A4, + 0x0EF, 0x00000000, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x00000361, + 0x030, 0x00001361, + 0x030, 0x00002361, + 0x030, 0x00003361, + 0x030, 0x00004443, + 0x030, 0x00005443, + 0x030, 0x00006443, + 0x030, 0x00007443, + 0x030, 0x00008424, + 0x030, 0x00009424, + 0x030, 0x0000A424, + 0x030, 0x0000B424, + 0x0EF, 0x00000000, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x00000334, + 0x030, 0x00001334, + 0x030, 0x00002334, + 0x030, 0x00003334, + 0x030, 0x000043A4, + 0x030, 0x000053A4, + 0x030, 0x000063A4, + 0x030, 0x000073A4, + 0x030, 0x00008365, + 0x030, 0x00009365, + 0x030, 0x0000A365, + 0x030, 0x0000B365, + 0x0EF, 0x00000000, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x00000403, + 0x030, 0x00001403, + 0x030, 0x00002403, + 0x030, 0x00003403, + 0x030, 0x000044A2, + 0x030, 0x000054A2, + 0x030, 0x000064A2, + 0x030, 0x000074A2, + 0x030, 0x000083A3, + 0x030, 0x000093A3, + 0x030, 0x0000A3A3, + 0x030, 0x0000B3A3, + 0x0EF, 0x00000000, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x000003A3, + 0x030, 0x000013A3, + 0x030, 0x000023A3, + 0x030, 0x000033A3, + 0x030, 0x000043A4, + 0x030, 0x000053A4, + 0x030, 0x000063A4, + 0x030, 0x000073A4, + 0x030, 0x00008365, + 0x030, 0x00009365, + 0x030, 0x0000A365, + 0x030, 0x0000B365, + 0x0EF, 0x00000000, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x000002A6, + 0x030, 0x000012A6, + 0x030, 0x000022A6, + 0x030, 0x000032A6, + 0x030, 0x000042A6, + 0x030, 0x000052A6, + 0x030, 0x000062A6, + 0x030, 0x000072A6, + 0x030, 0x000082A6, + 0x030, 0x000092A6, + 0x030, 0x0000A2A6, + 0x030, 0x0000B2A6, + 0x0EF, 0x00000000, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x000004A0, + 0x030, 0x000014A0, + 0x030, 0x000024A0, + 0x030, 0x000034A0, + 0x030, 0x000044A0, + 0x030, 0x000054A0, + 0x030, 0x000064A0, + 0x030, 0x000074A0, + 0x030, 0x000084A0, + 0x030, 0x000094A0, + 0x030, 0x0000A4A0, + 0x030, 0x0000B4A0, + 0x0EF, 0x00000000, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x000002A1, + 0x030, 0x000012A1, + 0x030, 0x000022A1, + 0x030, 0x000032A1, + 0x030, 0x000042A1, + 0x030, 0x000052A1, + 0x030, 0x000062A1, + 0x030, 0x000072A1, + 0x030, 0x000082A1, + 0x030, 0x000092A1, + 0x030, 0x0000A2A1, + 0x030, 0x0000B2A1, + 0x0EF, 0x00000000, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x000003A0, + 0x030, 0x000013A0, + 0x030, 0x000023A0, + 0x030, 0x000033A0, + 0x030, 0x000043A1, + 0x030, 0x000053A1, + 0x030, 0x000063A1, + 0x030, 0x000073A1, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x0EF, 0x00000000, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x000002A1, + 0x030, 0x000012A1, + 0x030, 0x000022A1, + 0x030, 0x000032A1, + 0x030, 0x000042A1, + 0x030, 0x000052A1, + 0x030, 0x000062A1, + 0x030, 0x000072A1, + 0x030, 0x000082A1, + 0x030, 0x000092A1, + 0x030, 0x0000A2A1, + 0x030, 0x0000B2A1, + 0x0EF, 0x00000000, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x000003A0, + 0x030, 0x000013A0, + 0x030, 0x000023A0, + 0x030, 0x000033A0, + 0x030, 0x00004430, + 0x030, 0x00005430, + 0x030, 0x00006430, + 0x030, 0x00007430, + 0x030, 0x00008372, + 0x030, 0x00009372, + 0x030, 0x0000A372, + 0x030, 0x0000B372, + 0x0EF, 0x00000000, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x000004A0, + 0x030, 0x000014A0, + 0x030, 0x000024A0, + 0x030, 0x000034A0, + 0x030, 0x000044A0, + 0x030, 0x000054A0, + 0x030, 0x000064A0, + 0x030, 0x000074A0, + 0x030, 0x000084A0, + 0x030, 0x000094A0, + 0x030, 0x0000A4A0, + 0x030, 0x0000B4A0, + 0x0EF, 0x00000000, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x000004A0, + 0x030, 0x000014A0, + 0x030, 0x000024A0, + 0x030, 0x000034A0, + 0x030, 0x000044A0, + 0x030, 0x000054A0, + 0x030, 0x000064A0, + 0x030, 0x000074A0, + 0x030, 0x000084A0, + 0x030, 0x000094A0, + 0x030, 0x0000A4A0, + 0x030, 0x0000B4A0, + 0x0EF, 0x00000000, + 0xA0000000, 0x00000000, + 0x0EF, 0x00000200, + 0x030, 0x000003D0, + 0x030, 0x000013D0, + 0x030, 0x000023D0, + 0x030, 0x000033D0, + 0x030, 0x000043D0, + 0x030, 0x000053D0, + 0x030, 0x000063D0, + 0x030, 0x000073D0, + 0x030, 0x000083D0, + 0x030, 0x000093D0, + 0x030, 0x0000A3D0, + 0x030, 0x0000B3D0, + 0x0EF, 0x00000000, + 0xB0000000, 0x00000000, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x00000203, + 0x030, 0x00001203, + 0x030, 0x00002203, + 0x030, 0x00003203, + 0x030, 0x00004203, + 0x030, 0x00005203, + 0x030, 0x00006203, + 0x030, 0x00007203, + 0x030, 0x00008203, + 0x030, 0x00009203, + 0x030, 0x0000A203, + 0x030, 0x0000B203, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x00000203, + 0x030, 0x00001203, + 0x030, 0x00002203, + 0x030, 0x00003203, + 0x030, 0x00004203, + 0x030, 0x00005203, + 0x030, 0x00006203, + 0x030, 0x00007203, + 0x030, 0x00008203, + 0x030, 0x00009203, + 0x030, 0x0000A203, + 0x030, 0x0000B203, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x00000203, + 0x030, 0x00001203, + 0x030, 0x00002203, + 0x030, 0x00003203, + 0x030, 0x00004203, + 0x030, 0x00005203, + 0x030, 0x00006203, + 0x030, 0x00007203, + 0x030, 0x00008203, + 0x030, 0x00009203, + 0x030, 0x0000A203, + 0x030, 0x0000B203, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x000003A3, + 0x030, 0x000013A3, + 0x030, 0x000023A3, + 0x030, 0x000033A3, + 0x030, 0x000043A3, + 0x030, 0x000053A3, + 0x030, 0x000063A3, + 0x030, 0x000073A3, + 0x030, 0x000083A3, + 0x030, 0x000093A3, + 0x030, 0x0000A3A3, + 0x030, 0x0000B3A3, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x00000203, + 0x030, 0x00001203, + 0x030, 0x00002203, + 0x030, 0x00003203, + 0x030, 0x00004203, + 0x030, 0x00005203, + 0x030, 0x00006203, + 0x030, 0x00007203, + 0x030, 0x00008203, + 0x030, 0x00009203, + 0x030, 0x0000A203, + 0x030, 0x0000B203, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x00000203, + 0x030, 0x00001203, + 0x030, 0x00002203, + 0x030, 0x00003203, + 0x030, 0x00004203, + 0x030, 0x00005203, + 0x030, 0x00006203, + 0x030, 0x00007203, + 0x030, 0x00008203, + 0x030, 0x00009203, + 0x030, 0x0000A203, + 0x030, 0x0000B203, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x00000203, + 0x030, 0x00001203, + 0x030, 0x00002203, + 0x030, 0x00003203, + 0x030, 0x00004203, + 0x030, 0x00005203, + 0x030, 0x00006203, + 0x030, 0x00007203, + 0x030, 0x00008203, + 0x030, 0x00009203, + 0x030, 0x0000A203, + 0x030, 0x0000B203, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x00000203, + 0x030, 0x00001203, + 0x030, 0x00002203, + 0x030, 0x00003203, + 0x030, 0x00004203, + 0x030, 0x00005203, + 0x030, 0x00006203, + 0x030, 0x00007203, + 0x030, 0x00008203, + 0x030, 0x00009203, + 0x030, 0x0000A203, + 0x030, 0x0000B203, + 0xA0000000, 0x00000000, + 0x0EF, 0x00000080, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000764, + 0x030, 0x00001632, + 0x030, 0x00002421, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004777, + 0x030, 0x00005777, + 0x030, 0x00006777, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000777, + 0x030, 0x00001442, + 0x030, 0x00002222, + 0x030, 0x00004777, + 0x030, 0x00005777, + 0x030, 0x00006777, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004777, + 0x030, 0x00005777, + 0x030, 0x00006777, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000776, + 0x030, 0x00001455, + 0x030, 0x00002335, + 0x030, 0x00004777, + 0x030, 0x00005777, + 0x030, 0x00006777, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000660, + 0x030, 0x00001443, + 0x030, 0x00002221, + 0x030, 0x00004777, + 0x030, 0x00005777, + 0x030, 0x00006777, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000764, + 0x030, 0x00001632, + 0x030, 0x00002421, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000764, + 0x030, 0x00001632, + 0x030, 0x00002421, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000767, + 0x030, 0x00001442, + 0x030, 0x00002222, + 0x030, 0x00004777, + 0x030, 0x00005777, + 0x030, 0x00006777, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000765, + 0x030, 0x00001632, + 0x030, 0x00002451, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000764, + 0x030, 0x00001632, + 0x030, 0x00002421, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000777, + 0x030, 0x00001454, + 0x030, 0x00002224, + 0x030, 0x00004777, + 0x030, 0x00005777, + 0x030, 0x00006777, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000777, + 0x030, 0x00001442, + 0x030, 0x00002222, + 0x030, 0x00004777, + 0x030, 0x00005777, + 0x030, 0x00006777, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000777, + 0x030, 0x00001442, + 0x030, 0x00002222, + 0x030, 0x00004777, + 0x030, 0x00005777, + 0x030, 0x00006777, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004777, + 0x030, 0x00005777, + 0x030, 0x00006777, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000775, + 0x030, 0x00001422, + 0x030, 0x00002210, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000775, + 0x030, 0x00001343, + 0x030, 0x00002210, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0xA0000000, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000764, + 0x030, 0x00001632, + 0x030, 0x00002421, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x0EF, 0x00000800, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000C09, + 0x033, 0x00000021, + 0x03F, 0x00000C0C, + 0x033, 0x00000022, + 0x03F, 0x00000C0F, + 0x033, 0x00000023, + 0x03F, 0x00000C2C, + 0x033, 0x00000024, + 0x03F, 0x00000C2F, + 0x033, 0x00000025, + 0x03F, 0x00000C8A, + 0x033, 0x00000026, + 0x03F, 0x00000C8D, + 0x033, 0x00000027, + 0x03F, 0x00000C90, + 0x033, 0x00000028, + 0x03F, 0x00000CD0, + 0x033, 0x00000029, + 0x03F, 0x00000CF2, + 0x033, 0x0000002A, + 0x03F, 0x00000CF5, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000007, + 0x033, 0x00000021, + 0x03F, 0x0000000A, + 0x033, 0x00000022, + 0x03F, 0x0000000D, + 0x033, 0x00000023, + 0x03F, 0x0000002A, + 0x033, 0x00000024, + 0x03F, 0x0000002D, + 0x033, 0x00000025, + 0x03F, 0x00000030, + 0x033, 0x00000026, + 0x03F, 0x0000006D, + 0x033, 0x00000027, + 0x03F, 0x00000070, + 0x033, 0x00000028, + 0x03F, 0x000000ED, + 0x033, 0x00000029, + 0x03F, 0x000000F0, + 0x033, 0x0000002A, + 0x03F, 0x000000F3, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000005, + 0x033, 0x00000021, + 0x03F, 0x00000008, + 0x033, 0x00000022, + 0x03F, 0x0000000B, + 0x033, 0x00000023, + 0x03F, 0x0000000E, + 0x033, 0x00000024, + 0x03F, 0x0000002B, + 0x033, 0x00000025, + 0x03F, 0x0000002E, + 0x033, 0x00000026, + 0x03F, 0x0000006B, + 0x033, 0x00000027, + 0x03F, 0x0000006E, + 0x033, 0x00000028, + 0x03F, 0x00000071, + 0x033, 0x00000029, + 0x03F, 0x00000074, + 0x033, 0x0000002A, + 0x03F, 0x00000077, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000429, + 0x033, 0x00000021, + 0x03F, 0x00000828, + 0x033, 0x00000022, + 0x03F, 0x00000847, + 0x033, 0x00000023, + 0x03F, 0x0000084A, + 0x033, 0x00000024, + 0x03F, 0x00000C4B, + 0x033, 0x00000025, + 0x03F, 0x00000C6C, + 0x033, 0x00000026, + 0x03F, 0x00000C8D, + 0x033, 0x00000027, + 0x03F, 0x00000CAF, + 0x033, 0x00000028, + 0x03F, 0x00000CD1, + 0x033, 0x00000029, + 0x03F, 0x00000CF3, + 0x033, 0x0000002A, + 0x03F, 0x00000CF6, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000005, + 0x033, 0x00000021, + 0x03F, 0x00000008, + 0x033, 0x00000022, + 0x03F, 0x0000000B, + 0x033, 0x00000023, + 0x03F, 0x0000000E, + 0x033, 0x00000024, + 0x03F, 0x0000002B, + 0x033, 0x00000025, + 0x03F, 0x0000002E, + 0x033, 0x00000026, + 0x03F, 0x0000006B, + 0x033, 0x00000027, + 0x03F, 0x0000006E, + 0x033, 0x00000028, + 0x03F, 0x00000071, + 0x033, 0x00000029, + 0x03F, 0x00000074, + 0x033, 0x0000002A, + 0x03F, 0x00000077, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x0000042B, + 0x033, 0x00000021, + 0x03F, 0x0000082A, + 0x033, 0x00000022, + 0x03F, 0x00000849, + 0x033, 0x00000023, + 0x03F, 0x0000084C, + 0x033, 0x00000024, + 0x03F, 0x00000C4C, + 0x033, 0x00000025, + 0x03F, 0x00000C6C, + 0x033, 0x00000026, + 0x03F, 0x00000CAC, + 0x033, 0x00000027, + 0x03F, 0x00000CED, + 0x033, 0x00000028, + 0x03F, 0x00000CF0, + 0x033, 0x00000029, + 0x03F, 0x00000CF3, + 0x033, 0x0000002A, + 0x03F, 0x00000CF6, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000007, + 0x033, 0x00000021, + 0x03F, 0x0000000A, + 0x033, 0x00000022, + 0x03F, 0x0000000D, + 0x033, 0x00000023, + 0x03F, 0x0000002A, + 0x033, 0x00000024, + 0x03F, 0x0000002D, + 0x033, 0x00000025, + 0x03F, 0x00000030, + 0x033, 0x00000026, + 0x03F, 0x0000006D, + 0x033, 0x00000027, + 0x03F, 0x00000070, + 0x033, 0x00000028, + 0x03F, 0x000000ED, + 0x033, 0x00000029, + 0x03F, 0x000000F0, + 0x033, 0x0000002A, + 0x03F, 0x000000F3, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000007, + 0x033, 0x00000021, + 0x03F, 0x0000000A, + 0x033, 0x00000022, + 0x03F, 0x0000000D, + 0x033, 0x00000023, + 0x03F, 0x0000002A, + 0x033, 0x00000024, + 0x03F, 0x0000002D, + 0x033, 0x00000025, + 0x03F, 0x00000030, + 0x033, 0x00000026, + 0x03F, 0x0000006D, + 0x033, 0x00000027, + 0x03F, 0x00000070, + 0x033, 0x00000028, + 0x03F, 0x000000ED, + 0x033, 0x00000029, + 0x03F, 0x000000F0, + 0x033, 0x0000002A, + 0x03F, 0x000000F3, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000C0C, + 0x033, 0x00000021, + 0x03F, 0x00000C29, + 0x033, 0x00000022, + 0x03F, 0x00000C2C, + 0x033, 0x00000023, + 0x03F, 0x00000C69, + 0x033, 0x00000024, + 0x03F, 0x00000CA8, + 0x033, 0x00000025, + 0x03F, 0x00000CE8, + 0x033, 0x00000026, + 0x03F, 0x00000CEB, + 0x033, 0x00000027, + 0x03F, 0x00000CEE, + 0x033, 0x00000028, + 0x03F, 0x00000CF1, + 0x033, 0x00000029, + 0x03F, 0x00000CF4, + 0x033, 0x0000002A, + 0x03F, 0x00000CF7, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000C09, + 0x033, 0x00000021, + 0x03F, 0x00000C0C, + 0x033, 0x00000022, + 0x03F, 0x00000C0F, + 0x033, 0x00000023, + 0x03F, 0x00000C2C, + 0x033, 0x00000024, + 0x03F, 0x00000C2F, + 0x033, 0x00000025, + 0x03F, 0x00000C8A, + 0x033, 0x00000026, + 0x03F, 0x00000C8D, + 0x033, 0x00000027, + 0x03F, 0x00000C90, + 0x033, 0x00000028, + 0x03F, 0x00000CD0, + 0x033, 0x00000029, + 0x03F, 0x00000CF2, + 0x033, 0x0000002A, + 0x03F, 0x00000CF5, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000C09, + 0x033, 0x00000021, + 0x03F, 0x00000C0C, + 0x033, 0x00000022, + 0x03F, 0x00000C0F, + 0x033, 0x00000023, + 0x03F, 0x00000C2C, + 0x033, 0x00000024, + 0x03F, 0x00000C2F, + 0x033, 0x00000025, + 0x03F, 0x00000C8A, + 0x033, 0x00000026, + 0x03F, 0x00000C8D, + 0x033, 0x00000027, + 0x03F, 0x00000C90, + 0x033, 0x00000028, + 0x03F, 0x00000CD0, + 0x033, 0x00000029, + 0x03F, 0x00000CF2, + 0x033, 0x0000002A, + 0x03F, 0x00000CF5, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000007, + 0x033, 0x00000021, + 0x03F, 0x0000000A, + 0x033, 0x00000022, + 0x03F, 0x0000000D, + 0x033, 0x00000023, + 0x03F, 0x0000002A, + 0x033, 0x00000024, + 0x03F, 0x0000002D, + 0x033, 0x00000025, + 0x03F, 0x00000030, + 0x033, 0x00000026, + 0x03F, 0x0000006D, + 0x033, 0x00000027, + 0x03F, 0x00000070, + 0x033, 0x00000028, + 0x03F, 0x000000ED, + 0x033, 0x00000029, + 0x03F, 0x000000F0, + 0x033, 0x0000002A, + 0x03F, 0x000000F3, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000429, + 0x033, 0x00000021, + 0x03F, 0x00000828, + 0x033, 0x00000022, + 0x03F, 0x00000847, + 0x033, 0x00000023, + 0x03F, 0x0000084A, + 0x033, 0x00000024, + 0x03F, 0x00000C4B, + 0x033, 0x00000025, + 0x03F, 0x00000CE5, + 0x033, 0x00000026, + 0x03F, 0x00000CE8, + 0x033, 0x00000027, + 0x03F, 0x00000CEB, + 0x033, 0x00000028, + 0x03F, 0x00000CEE, + 0x033, 0x00000029, + 0x03F, 0x00000CF1, + 0x033, 0x0000002A, + 0x03F, 0x00000CF4, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000C0B, + 0x033, 0x00000021, + 0x03F, 0x00000C0E, + 0x033, 0x00000022, + 0x03F, 0x00000C2B, + 0x033, 0x00000023, + 0x03F, 0x00000C2E, + 0x033, 0x00000024, + 0x03F, 0x00000C89, + 0x033, 0x00000025, + 0x03F, 0x00000CE8, + 0x033, 0x00000026, + 0x03F, 0x00000CEB, + 0x033, 0x00000027, + 0x03F, 0x00000CEE, + 0x033, 0x00000028, + 0x03F, 0x00000CF1, + 0x033, 0x00000029, + 0x03F, 0x00000CF4, + 0x033, 0x0000002A, + 0x03F, 0x00000CF7, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000C09, + 0x033, 0x00000021, + 0x03F, 0x00000C0C, + 0x033, 0x00000022, + 0x03F, 0x00000C0F, + 0x033, 0x00000023, + 0x03F, 0x00000C2C, + 0x033, 0x00000024, + 0x03F, 0x00000C2F, + 0x033, 0x00000025, + 0x03F, 0x00000C8A, + 0x033, 0x00000026, + 0x03F, 0x00000C8D, + 0x033, 0x00000027, + 0x03F, 0x00000C90, + 0x033, 0x00000028, + 0x03F, 0x00000CD0, + 0x033, 0x00000029, + 0x03F, 0x00000CF2, + 0x033, 0x0000002A, + 0x03F, 0x00000CF5, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000429, + 0x033, 0x00000021, + 0x03F, 0x00000828, + 0x033, 0x00000022, + 0x03F, 0x00000847, + 0x033, 0x00000023, + 0x03F, 0x0000084A, + 0x033, 0x00000024, + 0x03F, 0x0000086A, + 0x033, 0x00000025, + 0x03F, 0x0000086D, + 0x033, 0x00000026, + 0x03F, 0x00000870, + 0x033, 0x00000027, + 0x03F, 0x00000891, + 0x033, 0x00000028, + 0x03F, 0x00000894, + 0x033, 0x00000029, + 0x03F, 0x000008B5, + 0x033, 0x0000002A, + 0x03F, 0x000008F5, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000429, + 0x033, 0x00000021, + 0x03F, 0x00000828, + 0x033, 0x00000022, + 0x03F, 0x00000847, + 0x033, 0x00000023, + 0x03F, 0x0000084A, + 0x033, 0x00000024, + 0x03F, 0x00000C4B, + 0x033, 0x00000025, + 0x03F, 0x00000C6C, + 0x033, 0x00000026, + 0x03F, 0x00000C8D, + 0x033, 0x00000027, + 0x03F, 0x00000CAF, + 0x033, 0x00000028, + 0x03F, 0x00000CD1, + 0x033, 0x00000029, + 0x03F, 0x00000CF3, + 0x033, 0x0000002A, + 0x03F, 0x00000CF6, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000429, + 0x033, 0x00000021, + 0x03F, 0x00000828, + 0x033, 0x00000022, + 0x03F, 0x00000847, + 0x033, 0x00000023, + 0x03F, 0x0000084A, + 0x033, 0x00000024, + 0x03F, 0x00000C4B, + 0x033, 0x00000025, + 0x03F, 0x00000C6C, + 0x033, 0x00000026, + 0x03F, 0x00000C8D, + 0x033, 0x00000027, + 0x03F, 0x00000CAF, + 0x033, 0x00000028, + 0x03F, 0x00000CD1, + 0x033, 0x00000029, + 0x03F, 0x00000CF3, + 0x033, 0x0000002A, + 0x03F, 0x00000CF6, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000005, + 0x033, 0x00000021, + 0x03F, 0x00000008, + 0x033, 0x00000022, + 0x03F, 0x0000000B, + 0x033, 0x00000023, + 0x03F, 0x0000000E, + 0x033, 0x00000024, + 0x03F, 0x0000002B, + 0x033, 0x00000025, + 0x03F, 0x0000002E, + 0x033, 0x00000026, + 0x03F, 0x0000006B, + 0x033, 0x00000027, + 0x03F, 0x0000006E, + 0x033, 0x00000028, + 0x03F, 0x00000071, + 0x033, 0x00000029, + 0x03F, 0x00000074, + 0x033, 0x0000002A, + 0x03F, 0x00000077, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000007, + 0x033, 0x00000021, + 0x03F, 0x0000000A, + 0x033, 0x00000022, + 0x03F, 0x0000000D, + 0x033, 0x00000023, + 0x03F, 0x0000002A, + 0x033, 0x00000024, + 0x03F, 0x0000002D, + 0x033, 0x00000025, + 0x03F, 0x00000030, + 0x033, 0x00000026, + 0x03F, 0x0000006D, + 0x033, 0x00000027, + 0x03F, 0x00000070, + 0x033, 0x00000028, + 0x03F, 0x000000ED, + 0x033, 0x00000029, + 0x03F, 0x000000F0, + 0x033, 0x0000002A, + 0x03F, 0x000000F3, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000005, + 0x033, 0x00000021, + 0x03F, 0x00000008, + 0x033, 0x00000022, + 0x03F, 0x0000000B, + 0x033, 0x00000023, + 0x03F, 0x0000000E, + 0x033, 0x00000024, + 0x03F, 0x0000002B, + 0x033, 0x00000025, + 0x03F, 0x00000068, + 0x033, 0x00000026, + 0x03F, 0x0000006B, + 0x033, 0x00000027, + 0x03F, 0x0000006E, + 0x033, 0x00000028, + 0x03F, 0x00000071, + 0x033, 0x00000029, + 0x03F, 0x00000074, + 0x033, 0x0000002A, + 0x03F, 0x00000077, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x0000042B, + 0x033, 0x00000021, + 0x03F, 0x0000082A, + 0x033, 0x00000022, + 0x03F, 0x00000849, + 0x033, 0x00000023, + 0x03F, 0x0000084C, + 0x033, 0x00000024, + 0x03F, 0x00000C4C, + 0x033, 0x00000025, + 0x03F, 0x00000C8A, + 0x033, 0x00000026, + 0x03F, 0x00000C8D, + 0x033, 0x00000027, + 0x03F, 0x00000CEB, + 0x033, 0x00000028, + 0x03F, 0x00000CEE, + 0x033, 0x00000029, + 0x03F, 0x00000CF1, + 0x033, 0x0000002A, + 0x03F, 0x00000CF4, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000005, + 0x033, 0x00000021, + 0x03F, 0x00000008, + 0x033, 0x00000022, + 0x03F, 0x0000000B, + 0x033, 0x00000023, + 0x03F, 0x0000000E, + 0x033, 0x00000024, + 0x03F, 0x0000002B, + 0x033, 0x00000025, + 0x03F, 0x00000068, + 0x033, 0x00000026, + 0x03F, 0x0000006B, + 0x033, 0x00000027, + 0x03F, 0x0000006E, + 0x033, 0x00000028, + 0x03F, 0x00000071, + 0x033, 0x00000029, + 0x03F, 0x00000074, + 0x033, 0x0000002A, + 0x03F, 0x00000077, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x0000042B, + 0x033, 0x00000021, + 0x03F, 0x0000082A, + 0x033, 0x00000022, + 0x03F, 0x00000849, + 0x033, 0x00000023, + 0x03F, 0x0000084C, + 0x033, 0x00000024, + 0x03F, 0x00000C4C, + 0x033, 0x00000025, + 0x03F, 0x00000C8A, + 0x033, 0x00000026, + 0x03F, 0x00000C8D, + 0x033, 0x00000027, + 0x03F, 0x00000CEB, + 0x033, 0x00000028, + 0x03F, 0x00000CEE, + 0x033, 0x00000029, + 0x03F, 0x00000CF1, + 0x033, 0x0000002A, + 0x03F, 0x00000CF4, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000007, + 0x033, 0x00000021, + 0x03F, 0x0000000A, + 0x033, 0x00000022, + 0x03F, 0x0000000D, + 0x033, 0x00000023, + 0x03F, 0x0000002A, + 0x033, 0x00000024, + 0x03F, 0x0000002D, + 0x033, 0x00000025, + 0x03F, 0x00000030, + 0x033, 0x00000026, + 0x03F, 0x0000006D, + 0x033, 0x00000027, + 0x03F, 0x00000070, + 0x033, 0x00000028, + 0x03F, 0x000000ED, + 0x033, 0x00000029, + 0x03F, 0x000000F0, + 0x033, 0x0000002A, + 0x03F, 0x000000F3, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000007, + 0x033, 0x00000021, + 0x03F, 0x0000000A, + 0x033, 0x00000022, + 0x03F, 0x0000000D, + 0x033, 0x00000023, + 0x03F, 0x0000002A, + 0x033, 0x00000024, + 0x03F, 0x0000002D, + 0x033, 0x00000025, + 0x03F, 0x00000030, + 0x033, 0x00000026, + 0x03F, 0x0000006D, + 0x033, 0x00000027, + 0x03F, 0x00000070, + 0x033, 0x00000028, + 0x03F, 0x000000ED, + 0x033, 0x00000029, + 0x03F, 0x000000F0, + 0x033, 0x0000002A, + 0x03F, 0x000000F3, + 0xA0000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000C09, + 0x033, 0x00000021, + 0x03F, 0x00000C0C, + 0x033, 0x00000022, + 0x03F, 0x00000C0F, + 0x033, 0x00000023, + 0x03F, 0x00000C2C, + 0x033, 0x00000024, + 0x03F, 0x00000C2F, + 0x033, 0x00000025, + 0x03F, 0x00000C8A, + 0x033, 0x00000026, + 0x03F, 0x00000C8D, + 0x033, 0x00000027, + 0x03F, 0x00000C90, + 0x033, 0x00000028, + 0x03F, 0x00000CD0, + 0x033, 0x00000029, + 0x03F, 0x00000CF2, + 0x033, 0x0000002A, + 0x03F, 0x00000CF5, + 0xB0000000, 0x00000000, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000C0A, + 0x033, 0x00000061, + 0x03F, 0x00000C0D, + 0x033, 0x00000062, + 0x03F, 0x00000C2A, + 0x033, 0x00000063, + 0x03F, 0x00000C2D, + 0x033, 0x00000064, + 0x03F, 0x00000C6A, + 0x033, 0x00000065, + 0x03F, 0x00000CAA, + 0x033, 0x00000066, + 0x03F, 0x00000CAD, + 0x033, 0x00000067, + 0x03F, 0x00000CB0, + 0x033, 0x00000068, + 0x03F, 0x00000CF1, + 0x033, 0x00000069, + 0x03F, 0x00000CF4, + 0x033, 0x0000006A, + 0x03F, 0x00000CF7, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000007, + 0x033, 0x00000061, + 0x03F, 0x0000000A, + 0x033, 0x00000062, + 0x03F, 0x0000000D, + 0x033, 0x00000063, + 0x03F, 0x0000002A, + 0x033, 0x00000064, + 0x03F, 0x0000002D, + 0x033, 0x00000065, + 0x03F, 0x00000030, + 0x033, 0x00000066, + 0x03F, 0x0000006D, + 0x033, 0x00000067, + 0x03F, 0x00000070, + 0x033, 0x00000068, + 0x03F, 0x000000ED, + 0x033, 0x00000069, + 0x03F, 0x000000F0, + 0x033, 0x0000006A, + 0x03F, 0x000000F3, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000005, + 0x033, 0x00000061, + 0x03F, 0x00000008, + 0x033, 0x00000062, + 0x03F, 0x0000000B, + 0x033, 0x00000063, + 0x03F, 0x0000000E, + 0x033, 0x00000064, + 0x03F, 0x0000002B, + 0x033, 0x00000065, + 0x03F, 0x0000002E, + 0x033, 0x00000066, + 0x03F, 0x0000006B, + 0x033, 0x00000067, + 0x03F, 0x0000006E, + 0x033, 0x00000068, + 0x03F, 0x00000071, + 0x033, 0x00000069, + 0x03F, 0x00000074, + 0x033, 0x0000006A, + 0x03F, 0x00000077, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000429, + 0x033, 0x00000061, + 0x03F, 0x00000828, + 0x033, 0x00000062, + 0x03F, 0x00000847, + 0x033, 0x00000063, + 0x03F, 0x0000084A, + 0x033, 0x00000064, + 0x03F, 0x00000C4B, + 0x033, 0x00000065, + 0x03F, 0x00000C6C, + 0x033, 0x00000066, + 0x03F, 0x00000C8D, + 0x033, 0x00000067, + 0x03F, 0x00000CAF, + 0x033, 0x00000068, + 0x03F, 0x00000CD1, + 0x033, 0x00000069, + 0x03F, 0x00000CF3, + 0x033, 0x0000006A, + 0x03F, 0x00000CF6, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000005, + 0x033, 0x00000061, + 0x03F, 0x00000008, + 0x033, 0x00000062, + 0x03F, 0x0000000B, + 0x033, 0x00000063, + 0x03F, 0x0000000E, + 0x033, 0x00000064, + 0x03F, 0x0000002B, + 0x033, 0x00000065, + 0x03F, 0x0000002E, + 0x033, 0x00000066, + 0x03F, 0x0000006B, + 0x033, 0x00000067, + 0x03F, 0x0000006E, + 0x033, 0x00000068, + 0x03F, 0x00000071, + 0x033, 0x00000069, + 0x03F, 0x00000074, + 0x033, 0x0000006A, + 0x03F, 0x00000077, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x0000042A, + 0x033, 0x00000061, + 0x03F, 0x00000829, + 0x033, 0x00000062, + 0x03F, 0x00000848, + 0x033, 0x00000063, + 0x03F, 0x0000084B, + 0x033, 0x00000064, + 0x03F, 0x00000C4B, + 0x033, 0x00000065, + 0x03F, 0x00000C6C, + 0x033, 0x00000066, + 0x03F, 0x00000CAC, + 0x033, 0x00000067, + 0x03F, 0x00000CED, + 0x033, 0x00000068, + 0x03F, 0x00000CF0, + 0x033, 0x00000069, + 0x03F, 0x00000CF3, + 0x033, 0x0000006A, + 0x03F, 0x00000CF6, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000007, + 0x033, 0x00000061, + 0x03F, 0x0000000A, + 0x033, 0x00000062, + 0x03F, 0x0000000D, + 0x033, 0x00000063, + 0x03F, 0x0000002A, + 0x033, 0x00000064, + 0x03F, 0x0000002D, + 0x033, 0x00000065, + 0x03F, 0x00000030, + 0x033, 0x00000066, + 0x03F, 0x0000006D, + 0x033, 0x00000067, + 0x03F, 0x00000070, + 0x033, 0x00000068, + 0x03F, 0x000000ED, + 0x033, 0x00000069, + 0x03F, 0x000000F0, + 0x033, 0x0000006A, + 0x03F, 0x000000F3, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000007, + 0x033, 0x00000061, + 0x03F, 0x0000000A, + 0x033, 0x00000062, + 0x03F, 0x0000000D, + 0x033, 0x00000063, + 0x03F, 0x0000002A, + 0x033, 0x00000064, + 0x03F, 0x0000002D, + 0x033, 0x00000065, + 0x03F, 0x00000030, + 0x033, 0x00000066, + 0x03F, 0x0000006D, + 0x033, 0x00000067, + 0x03F, 0x00000070, + 0x033, 0x00000068, + 0x03F, 0x000000ED, + 0x033, 0x00000069, + 0x03F, 0x000000F0, + 0x033, 0x0000006A, + 0x03F, 0x000000F3, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x0000080B, + 0x033, 0x00000061, + 0x03F, 0x0000080E, + 0x033, 0x00000062, + 0x03F, 0x00000848, + 0x033, 0x00000063, + 0x03F, 0x00000869, + 0x033, 0x00000064, + 0x03F, 0x000008A9, + 0x033, 0x00000065, + 0x03F, 0x00000CE8, + 0x033, 0x00000066, + 0x03F, 0x00000CEB, + 0x033, 0x00000067, + 0x03F, 0x00000CEE, + 0x033, 0x00000068, + 0x03F, 0x00000CF1, + 0x033, 0x00000069, + 0x03F, 0x00000CF4, + 0x033, 0x0000006A, + 0x03F, 0x00000CF7, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000C0A, + 0x033, 0x00000061, + 0x03F, 0x00000C0D, + 0x033, 0x00000062, + 0x03F, 0x00000C2A, + 0x033, 0x00000063, + 0x03F, 0x00000C2D, + 0x033, 0x00000064, + 0x03F, 0x00000C6A, + 0x033, 0x00000065, + 0x03F, 0x00000CAA, + 0x033, 0x00000066, + 0x03F, 0x00000CAD, + 0x033, 0x00000067, + 0x03F, 0x00000CB0, + 0x033, 0x00000068, + 0x03F, 0x00000CF1, + 0x033, 0x00000069, + 0x03F, 0x00000CF4, + 0x033, 0x0000006A, + 0x03F, 0x00000CF7, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000C0A, + 0x033, 0x00000061, + 0x03F, 0x00000C0D, + 0x033, 0x00000062, + 0x03F, 0x00000C2A, + 0x033, 0x00000063, + 0x03F, 0x00000C2D, + 0x033, 0x00000064, + 0x03F, 0x00000C6A, + 0x033, 0x00000065, + 0x03F, 0x00000CAA, + 0x033, 0x00000066, + 0x03F, 0x00000CAD, + 0x033, 0x00000067, + 0x03F, 0x00000CB0, + 0x033, 0x00000068, + 0x03F, 0x00000CF1, + 0x033, 0x00000069, + 0x03F, 0x00000CF4, + 0x033, 0x0000006A, + 0x03F, 0x00000CF7, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000005, + 0x033, 0x00000061, + 0x03F, 0x00000008, + 0x033, 0x00000062, + 0x03F, 0x0000000B, + 0x033, 0x00000063, + 0x03F, 0x0000000E, + 0x033, 0x00000064, + 0x03F, 0x0000002B, + 0x033, 0x00000065, + 0x03F, 0x00000068, + 0x033, 0x00000066, + 0x03F, 0x0000006B, + 0x033, 0x00000067, + 0x03F, 0x0000006E, + 0x033, 0x00000068, + 0x03F, 0x00000071, + 0x033, 0x00000069, + 0x03F, 0x00000074, + 0x033, 0x0000006A, + 0x03F, 0x00000077, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000429, + 0x033, 0x00000061, + 0x03F, 0x00000828, + 0x033, 0x00000062, + 0x03F, 0x00000847, + 0x033, 0x00000063, + 0x03F, 0x0000084A, + 0x033, 0x00000064, + 0x03F, 0x00000C4B, + 0x033, 0x00000065, + 0x03F, 0x00000CE5, + 0x033, 0x00000066, + 0x03F, 0x00000CE8, + 0x033, 0x00000067, + 0x03F, 0x00000CEB, + 0x033, 0x00000068, + 0x03F, 0x00000CEE, + 0x033, 0x00000069, + 0x03F, 0x00000CF1, + 0x033, 0x0000006A, + 0x03F, 0x00000CF4, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000C0A, + 0x033, 0x00000061, + 0x03F, 0x00000C0D, + 0x033, 0x00000062, + 0x03F, 0x00000C10, + 0x033, 0x00000063, + 0x03F, 0x00000C4A, + 0x033, 0x00000064, + 0x03F, 0x00000C4D, + 0x033, 0x00000065, + 0x03F, 0x00000CC9, + 0x033, 0x00000066, + 0x03F, 0x00000CEB, + 0x033, 0x00000067, + 0x03F, 0x00000CEE, + 0x033, 0x00000068, + 0x03F, 0x00000CF1, + 0x033, 0x00000069, + 0x03F, 0x00000CF4, + 0x033, 0x0000006A, + 0x03F, 0x00000CF7, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000C0A, + 0x033, 0x00000061, + 0x03F, 0x00000C0D, + 0x033, 0x00000062, + 0x03F, 0x00000C2A, + 0x033, 0x00000063, + 0x03F, 0x00000C2D, + 0x033, 0x00000064, + 0x03F, 0x00000C6A, + 0x033, 0x00000065, + 0x03F, 0x00000CAA, + 0x033, 0x00000066, + 0x03F, 0x00000CAD, + 0x033, 0x00000067, + 0x03F, 0x00000CB0, + 0x033, 0x00000068, + 0x03F, 0x00000CF1, + 0x033, 0x00000069, + 0x03F, 0x00000CF4, + 0x033, 0x0000006A, + 0x03F, 0x00000CF7, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000429, + 0x033, 0x00000061, + 0x03F, 0x00000828, + 0x033, 0x00000062, + 0x03F, 0x00000847, + 0x033, 0x00000063, + 0x03F, 0x0000084A, + 0x033, 0x00000064, + 0x03F, 0x0000086A, + 0x033, 0x00000065, + 0x03F, 0x0000086D, + 0x033, 0x00000066, + 0x03F, 0x00000870, + 0x033, 0x00000067, + 0x03F, 0x00000891, + 0x033, 0x00000068, + 0x03F, 0x00000894, + 0x033, 0x00000069, + 0x03F, 0x000008B5, + 0x033, 0x0000006A, + 0x03F, 0x000008F5, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000429, + 0x033, 0x00000061, + 0x03F, 0x00000828, + 0x033, 0x00000062, + 0x03F, 0x00000847, + 0x033, 0x00000063, + 0x03F, 0x0000084A, + 0x033, 0x00000064, + 0x03F, 0x00000C4B, + 0x033, 0x00000065, + 0x03F, 0x00000C6C, + 0x033, 0x00000066, + 0x03F, 0x00000C8D, + 0x033, 0x00000067, + 0x03F, 0x00000CAF, + 0x033, 0x00000068, + 0x03F, 0x00000CD1, + 0x033, 0x00000069, + 0x03F, 0x00000CF3, + 0x033, 0x0000006A, + 0x03F, 0x00000CF6, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000429, + 0x033, 0x00000061, + 0x03F, 0x00000828, + 0x033, 0x00000062, + 0x03F, 0x00000847, + 0x033, 0x00000063, + 0x03F, 0x0000084A, + 0x033, 0x00000064, + 0x03F, 0x00000C4B, + 0x033, 0x00000065, + 0x03F, 0x00000C6C, + 0x033, 0x00000066, + 0x03F, 0x00000C8D, + 0x033, 0x00000067, + 0x03F, 0x00000CAF, + 0x033, 0x00000068, + 0x03F, 0x00000CD1, + 0x033, 0x00000069, + 0x03F, 0x00000CF3, + 0x033, 0x0000006A, + 0x03F, 0x00000CF6, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000005, + 0x033, 0x00000061, + 0x03F, 0x00000008, + 0x033, 0x00000062, + 0x03F, 0x0000000B, + 0x033, 0x00000063, + 0x03F, 0x0000000E, + 0x033, 0x00000064, + 0x03F, 0x0000002B, + 0x033, 0x00000065, + 0x03F, 0x0000002E, + 0x033, 0x00000066, + 0x03F, 0x0000006B, + 0x033, 0x00000067, + 0x03F, 0x0000006E, + 0x033, 0x00000068, + 0x03F, 0x00000071, + 0x033, 0x00000069, + 0x03F, 0x00000074, + 0x033, 0x0000006A, + 0x03F, 0x00000077, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000007, + 0x033, 0x00000061, + 0x03F, 0x0000000A, + 0x033, 0x00000062, + 0x03F, 0x0000000D, + 0x033, 0x00000063, + 0x03F, 0x0000002A, + 0x033, 0x00000064, + 0x03F, 0x0000002D, + 0x033, 0x00000065, + 0x03F, 0x00000030, + 0x033, 0x00000066, + 0x03F, 0x0000006D, + 0x033, 0x00000067, + 0x03F, 0x00000070, + 0x033, 0x00000068, + 0x03F, 0x000000ED, + 0x033, 0x00000069, + 0x03F, 0x000000F0, + 0x033, 0x0000006A, + 0x03F, 0x000000F3, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000005, + 0x033, 0x00000061, + 0x03F, 0x00000008, + 0x033, 0x00000062, + 0x03F, 0x0000000B, + 0x033, 0x00000063, + 0x03F, 0x0000000E, + 0x033, 0x00000064, + 0x03F, 0x0000002B, + 0x033, 0x00000065, + 0x03F, 0x00000068, + 0x033, 0x00000066, + 0x03F, 0x0000006B, + 0x033, 0x00000067, + 0x03F, 0x0000006E, + 0x033, 0x00000068, + 0x03F, 0x00000071, + 0x033, 0x00000069, + 0x03F, 0x00000074, + 0x033, 0x0000006A, + 0x03F, 0x00000077, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x0000042C, + 0x033, 0x00000061, + 0x03F, 0x0000082B, + 0x033, 0x00000062, + 0x03F, 0x0000084A, + 0x033, 0x00000063, + 0x03F, 0x0000084D, + 0x033, 0x00000064, + 0x03F, 0x00000C4D, + 0x033, 0x00000065, + 0x03F, 0x00000C8B, + 0x033, 0x00000066, + 0x03F, 0x00000C8E, + 0x033, 0x00000067, + 0x03F, 0x00000CEC, + 0x033, 0x00000068, + 0x03F, 0x00000CEF, + 0x033, 0x00000069, + 0x03F, 0x00000CF2, + 0x033, 0x0000006A, + 0x03F, 0x00000CF5, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000005, + 0x033, 0x00000061, + 0x03F, 0x00000008, + 0x033, 0x00000062, + 0x03F, 0x0000000B, + 0x033, 0x00000063, + 0x03F, 0x0000000E, + 0x033, 0x00000064, + 0x03F, 0x0000002B, + 0x033, 0x00000065, + 0x03F, 0x00000068, + 0x033, 0x00000066, + 0x03F, 0x0000006B, + 0x033, 0x00000067, + 0x03F, 0x0000006E, + 0x033, 0x00000068, + 0x03F, 0x00000071, + 0x033, 0x00000069, + 0x03F, 0x00000074, + 0x033, 0x0000006A, + 0x03F, 0x00000077, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x0000042C, + 0x033, 0x00000061, + 0x03F, 0x0000082B, + 0x033, 0x00000062, + 0x03F, 0x0000084A, + 0x033, 0x00000063, + 0x03F, 0x0000084D, + 0x033, 0x00000064, + 0x03F, 0x00000C4D, + 0x033, 0x00000065, + 0x03F, 0x00000C8B, + 0x033, 0x00000066, + 0x03F, 0x00000C8E, + 0x033, 0x00000067, + 0x03F, 0x00000CEC, + 0x033, 0x00000068, + 0x03F, 0x00000CEF, + 0x033, 0x00000069, + 0x03F, 0x00000CF2, + 0x033, 0x0000006A, + 0x03F, 0x00000CF5, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000007, + 0x033, 0x00000061, + 0x03F, 0x0000000A, + 0x033, 0x00000062, + 0x03F, 0x0000000D, + 0x033, 0x00000063, + 0x03F, 0x0000002A, + 0x033, 0x00000064, + 0x03F, 0x0000002D, + 0x033, 0x00000065, + 0x03F, 0x00000030, + 0x033, 0x00000066, + 0x03F, 0x0000006D, + 0x033, 0x00000067, + 0x03F, 0x00000070, + 0x033, 0x00000068, + 0x03F, 0x000000ED, + 0x033, 0x00000069, + 0x03F, 0x000000F0, + 0x033, 0x0000006A, + 0x03F, 0x000000F3, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000007, + 0x033, 0x00000061, + 0x03F, 0x0000000A, + 0x033, 0x00000062, + 0x03F, 0x0000000D, + 0x033, 0x00000063, + 0x03F, 0x0000002A, + 0x033, 0x00000064, + 0x03F, 0x0000002D, + 0x033, 0x00000065, + 0x03F, 0x00000030, + 0x033, 0x00000066, + 0x03F, 0x0000006D, + 0x033, 0x00000067, + 0x03F, 0x00000070, + 0x033, 0x00000068, + 0x03F, 0x000000ED, + 0x033, 0x00000069, + 0x03F, 0x000000F0, + 0x033, 0x0000006A, + 0x03F, 0x000000F3, + 0xA0000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000C0A, + 0x033, 0x00000061, + 0x03F, 0x00000C0D, + 0x033, 0x00000062, + 0x03F, 0x00000C2A, + 0x033, 0x00000063, + 0x03F, 0x00000C2D, + 0x033, 0x00000064, + 0x03F, 0x00000C6A, + 0x033, 0x00000065, + 0x03F, 0x00000CAA, + 0x033, 0x00000066, + 0x03F, 0x00000CAD, + 0x033, 0x00000067, + 0x03F, 0x00000CB0, + 0x033, 0x00000068, + 0x03F, 0x00000CF1, + 0x033, 0x00000069, + 0x03F, 0x00000CF4, + 0x033, 0x0000006A, + 0x03F, 0x00000CF7, + 0xB0000000, 0x00000000, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000C09, + 0x033, 0x000000A1, + 0x03F, 0x00000C0C, + 0x033, 0x000000A2, + 0x03F, 0x00000C0F, + 0x033, 0x000000A3, + 0x03F, 0x00000C2C, + 0x033, 0x000000A4, + 0x03F, 0x00000C2F, + 0x033, 0x000000A5, + 0x03F, 0x00000C8A, + 0x033, 0x000000A6, + 0x03F, 0x00000C8D, + 0x033, 0x000000A7, + 0x03F, 0x00000C90, + 0x033, 0x000000A8, + 0x03F, 0x00000CEF, + 0x033, 0x000000A9, + 0x03F, 0x00000CF2, + 0x033, 0x000000AA, + 0x03F, 0x00000CF5, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000007, + 0x033, 0x000000A1, + 0x03F, 0x0000000A, + 0x033, 0x000000A2, + 0x03F, 0x0000000D, + 0x033, 0x000000A3, + 0x03F, 0x0000002A, + 0x033, 0x000000A4, + 0x03F, 0x0000002D, + 0x033, 0x000000A5, + 0x03F, 0x00000030, + 0x033, 0x000000A6, + 0x03F, 0x0000006D, + 0x033, 0x000000A7, + 0x03F, 0x00000070, + 0x033, 0x000000A8, + 0x03F, 0x000000ED, + 0x033, 0x000000A9, + 0x03F, 0x000000F0, + 0x033, 0x000000AA, + 0x03F, 0x000000F3, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000005, + 0x033, 0x000000A1, + 0x03F, 0x00000008, + 0x033, 0x000000A2, + 0x03F, 0x0000000B, + 0x033, 0x000000A3, + 0x03F, 0x0000000E, + 0x033, 0x000000A4, + 0x03F, 0x0000002B, + 0x033, 0x000000A5, + 0x03F, 0x0000002E, + 0x033, 0x000000A6, + 0x03F, 0x00000031, + 0x033, 0x000000A7, + 0x03F, 0x00000034, + 0x033, 0x000000A8, + 0x03F, 0x00000053, + 0x033, 0x000000A9, + 0x03F, 0x00000056, + 0x033, 0x000000AA, + 0x03F, 0x000000D1, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000429, + 0x033, 0x000000A1, + 0x03F, 0x00000828, + 0x033, 0x000000A2, + 0x03F, 0x00000847, + 0x033, 0x000000A3, + 0x03F, 0x0000084A, + 0x033, 0x000000A4, + 0x03F, 0x00000C4B, + 0x033, 0x000000A5, + 0x03F, 0x00000C6C, + 0x033, 0x000000A6, + 0x03F, 0x00000C8D, + 0x033, 0x000000A7, + 0x03F, 0x00000CAF, + 0x033, 0x000000A8, + 0x03F, 0x00000CD1, + 0x033, 0x000000A9, + 0x03F, 0x00000CF3, + 0x033, 0x000000AA, + 0x03F, 0x00000CF6, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000005, + 0x033, 0x000000A1, + 0x03F, 0x00000008, + 0x033, 0x000000A2, + 0x03F, 0x0000000B, + 0x033, 0x000000A3, + 0x03F, 0x0000000E, + 0x033, 0x000000A4, + 0x03F, 0x0000002B, + 0x033, 0x000000A5, + 0x03F, 0x0000002E, + 0x033, 0x000000A6, + 0x03F, 0x00000031, + 0x033, 0x000000A7, + 0x03F, 0x00000034, + 0x033, 0x000000A8, + 0x03F, 0x00000053, + 0x033, 0x000000A9, + 0x03F, 0x00000056, + 0x033, 0x000000AA, + 0x03F, 0x000000D1, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000429, + 0x033, 0x000000A1, + 0x03F, 0x00000828, + 0x033, 0x000000A2, + 0x03F, 0x00000847, + 0x033, 0x000000A3, + 0x03F, 0x0000084A, + 0x033, 0x000000A4, + 0x03F, 0x00000C4B, + 0x033, 0x000000A5, + 0x03F, 0x00000C6C, + 0x033, 0x000000A6, + 0x03F, 0x00000CAC, + 0x033, 0x000000A7, + 0x03F, 0x00000CED, + 0x033, 0x000000A8, + 0x03F, 0x00000CF0, + 0x033, 0x000000A9, + 0x03F, 0x00000CF3, + 0x033, 0x000000AA, + 0x03F, 0x00000CF6, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000007, + 0x033, 0x000000A1, + 0x03F, 0x0000000A, + 0x033, 0x000000A2, + 0x03F, 0x0000000D, + 0x033, 0x000000A3, + 0x03F, 0x0000002A, + 0x033, 0x000000A4, + 0x03F, 0x0000002D, + 0x033, 0x000000A5, + 0x03F, 0x00000030, + 0x033, 0x000000A6, + 0x03F, 0x0000006D, + 0x033, 0x000000A7, + 0x03F, 0x00000070, + 0x033, 0x000000A8, + 0x03F, 0x000000ED, + 0x033, 0x000000A9, + 0x03F, 0x000000F0, + 0x033, 0x000000AA, + 0x03F, 0x000000F3, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000007, + 0x033, 0x000000A1, + 0x03F, 0x0000000A, + 0x033, 0x000000A2, + 0x03F, 0x0000000D, + 0x033, 0x000000A3, + 0x03F, 0x0000002A, + 0x033, 0x000000A4, + 0x03F, 0x0000002D, + 0x033, 0x000000A5, + 0x03F, 0x00000030, + 0x033, 0x000000A6, + 0x03F, 0x0000006D, + 0x033, 0x000000A7, + 0x03F, 0x00000070, + 0x033, 0x000000A8, + 0x03F, 0x000000ED, + 0x033, 0x000000A9, + 0x03F, 0x000000F0, + 0x033, 0x000000AA, + 0x03F, 0x000000F3, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000C0A, + 0x033, 0x000000A1, + 0x03F, 0x00000C0D, + 0x033, 0x000000A2, + 0x03F, 0x00000C2A, + 0x033, 0x000000A3, + 0x03F, 0x00000C2D, + 0x033, 0x000000A4, + 0x03F, 0x00000C6A, + 0x033, 0x000000A5, + 0x03F, 0x00000CE8, + 0x033, 0x000000A6, + 0x03F, 0x00000CEB, + 0x033, 0x000000A7, + 0x03F, 0x00000CEE, + 0x033, 0x000000A8, + 0x03F, 0x00000CF1, + 0x033, 0x000000A9, + 0x03F, 0x00000CF4, + 0x033, 0x000000AA, + 0x03F, 0x00000CF7, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000C09, + 0x033, 0x000000A1, + 0x03F, 0x00000C0C, + 0x033, 0x000000A2, + 0x03F, 0x00000C0F, + 0x033, 0x000000A3, + 0x03F, 0x00000C2C, + 0x033, 0x000000A4, + 0x03F, 0x00000C2F, + 0x033, 0x000000A5, + 0x03F, 0x00000C8A, + 0x033, 0x000000A6, + 0x03F, 0x00000C8D, + 0x033, 0x000000A7, + 0x03F, 0x00000C90, + 0x033, 0x000000A8, + 0x03F, 0x00000CEF, + 0x033, 0x000000A9, + 0x03F, 0x00000CF2, + 0x033, 0x000000AA, + 0x03F, 0x00000CF5, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000C09, + 0x033, 0x000000A1, + 0x03F, 0x00000C0C, + 0x033, 0x000000A2, + 0x03F, 0x00000C0F, + 0x033, 0x000000A3, + 0x03F, 0x00000C2C, + 0x033, 0x000000A4, + 0x03F, 0x00000C2F, + 0x033, 0x000000A5, + 0x03F, 0x00000C8A, + 0x033, 0x000000A6, + 0x03F, 0x00000C8D, + 0x033, 0x000000A7, + 0x03F, 0x00000C90, + 0x033, 0x000000A8, + 0x03F, 0x00000CEF, + 0x033, 0x000000A9, + 0x03F, 0x00000CF2, + 0x033, 0x000000AA, + 0x03F, 0x00000CF5, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000007, + 0x033, 0x000000A1, + 0x03F, 0x0000000A, + 0x033, 0x000000A2, + 0x03F, 0x0000000D, + 0x033, 0x000000A3, + 0x03F, 0x0000002A, + 0x033, 0x000000A4, + 0x03F, 0x0000002D, + 0x033, 0x000000A5, + 0x03F, 0x00000030, + 0x033, 0x000000A6, + 0x03F, 0x0000006D, + 0x033, 0x000000A7, + 0x03F, 0x00000070, + 0x033, 0x000000A8, + 0x03F, 0x000000ED, + 0x033, 0x000000A9, + 0x03F, 0x000000F0, + 0x033, 0x000000AA, + 0x03F, 0x000000F3, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000429, + 0x033, 0x000000A1, + 0x03F, 0x00000828, + 0x033, 0x000000A2, + 0x03F, 0x00000847, + 0x033, 0x000000A3, + 0x03F, 0x0000084A, + 0x033, 0x000000A4, + 0x03F, 0x00000C4B, + 0x033, 0x000000A5, + 0x03F, 0x00000CE5, + 0x033, 0x000000A6, + 0x03F, 0x00000CE8, + 0x033, 0x000000A7, + 0x03F, 0x00000CEB, + 0x033, 0x000000A8, + 0x03F, 0x00000CEE, + 0x033, 0x000000A9, + 0x03F, 0x00000CF1, + 0x033, 0x000000AA, + 0x03F, 0x00000CF4, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000824, + 0x033, 0x000000A1, + 0x03F, 0x00000827, + 0x033, 0x000000A2, + 0x03F, 0x0000082A, + 0x033, 0x000000A3, + 0x03F, 0x0000082D, + 0x033, 0x000000A4, + 0x03F, 0x00000C68, + 0x033, 0x000000A5, + 0x03F, 0x00000C6B, + 0x033, 0x000000A6, + 0x03F, 0x00000CCA, + 0x033, 0x000000A7, + 0x03F, 0x00000CCD, + 0x033, 0x000000A8, + 0x03F, 0x00000CEF, + 0x033, 0x000000A9, + 0x03F, 0x00000CF2, + 0x033, 0x000000AA, + 0x03F, 0x00000CF5, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000C08, + 0x033, 0x000000A1, + 0x03F, 0x00000C0B, + 0x033, 0x000000A2, + 0x03F, 0x00000C0E, + 0x033, 0x000000A3, + 0x03F, 0x00000C2B, + 0x033, 0x000000A4, + 0x03F, 0x00000C2E, + 0x033, 0x000000A5, + 0x03F, 0x00000C31, + 0x033, 0x000000A6, + 0x03F, 0x00000CCA, + 0x033, 0x000000A7, + 0x03F, 0x00000CCD, + 0x033, 0x000000A8, + 0x03F, 0x00000CEF, + 0x033, 0x000000A9, + 0x03F, 0x00000CF2, + 0x033, 0x000000AA, + 0x03F, 0x00000CF5, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000429, + 0x033, 0x000000A1, + 0x03F, 0x00000828, + 0x033, 0x000000A2, + 0x03F, 0x00000847, + 0x033, 0x000000A3, + 0x03F, 0x0000084A, + 0x033, 0x000000A4, + 0x03F, 0x0000086A, + 0x033, 0x000000A5, + 0x03F, 0x0000086D, + 0x033, 0x000000A6, + 0x03F, 0x00000870, + 0x033, 0x000000A7, + 0x03F, 0x00000891, + 0x033, 0x000000A8, + 0x03F, 0x00000894, + 0x033, 0x000000A9, + 0x03F, 0x000008B5, + 0x033, 0x000000AA, + 0x03F, 0x000008F5, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000429, + 0x033, 0x000000A1, + 0x03F, 0x00000828, + 0x033, 0x000000A2, + 0x03F, 0x00000847, + 0x033, 0x000000A3, + 0x03F, 0x0000084A, + 0x033, 0x000000A4, + 0x03F, 0x00000C4B, + 0x033, 0x000000A5, + 0x03F, 0x00000C6C, + 0x033, 0x000000A6, + 0x03F, 0x00000C8D, + 0x033, 0x000000A7, + 0x03F, 0x00000CAF, + 0x033, 0x000000A8, + 0x03F, 0x00000CD1, + 0x033, 0x000000A9, + 0x03F, 0x00000CF3, + 0x033, 0x000000AA, + 0x03F, 0x00000CF6, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000429, + 0x033, 0x000000A1, + 0x03F, 0x00000828, + 0x033, 0x000000A2, + 0x03F, 0x00000847, + 0x033, 0x000000A3, + 0x03F, 0x0000084A, + 0x033, 0x000000A4, + 0x03F, 0x00000C4B, + 0x033, 0x000000A5, + 0x03F, 0x00000C6C, + 0x033, 0x000000A6, + 0x03F, 0x00000C8D, + 0x033, 0x000000A7, + 0x03F, 0x00000CAF, + 0x033, 0x000000A8, + 0x03F, 0x00000CD1, + 0x033, 0x000000A9, + 0x03F, 0x00000CF3, + 0x033, 0x000000AA, + 0x03F, 0x00000CF6, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000005, + 0x033, 0x000000A1, + 0x03F, 0x00000008, + 0x033, 0x000000A2, + 0x03F, 0x0000000B, + 0x033, 0x000000A3, + 0x03F, 0x0000000E, + 0x033, 0x000000A4, + 0x03F, 0x0000002B, + 0x033, 0x000000A5, + 0x03F, 0x0000002E, + 0x033, 0x000000A6, + 0x03F, 0x00000031, + 0x033, 0x000000A7, + 0x03F, 0x00000034, + 0x033, 0x000000A8, + 0x03F, 0x00000053, + 0x033, 0x000000A9, + 0x03F, 0x00000056, + 0x033, 0x000000AA, + 0x03F, 0x000000D1, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000007, + 0x033, 0x000000A1, + 0x03F, 0x0000000A, + 0x033, 0x000000A2, + 0x03F, 0x0000000D, + 0x033, 0x000000A3, + 0x03F, 0x0000002A, + 0x033, 0x000000A4, + 0x03F, 0x0000002D, + 0x033, 0x000000A5, + 0x03F, 0x00000030, + 0x033, 0x000000A6, + 0x03F, 0x0000006D, + 0x033, 0x000000A7, + 0x03F, 0x00000070, + 0x033, 0x000000A8, + 0x03F, 0x000000ED, + 0x033, 0x000000A9, + 0x03F, 0x000000F0, + 0x033, 0x000000AA, + 0x03F, 0x000000F3, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000005, + 0x033, 0x000000A1, + 0x03F, 0x00000008, + 0x033, 0x000000A2, + 0x03F, 0x0000000B, + 0x033, 0x000000A3, + 0x03F, 0x0000000E, + 0x033, 0x000000A4, + 0x03F, 0x00000047, + 0x033, 0x000000A5, + 0x03F, 0x0000004A, + 0x033, 0x000000A6, + 0x03F, 0x0000004D, + 0x033, 0x000000A7, + 0x03F, 0x00000050, + 0x033, 0x000000A8, + 0x03F, 0x00000053, + 0x033, 0x000000A9, + 0x03F, 0x00000056, + 0x033, 0x000000AA, + 0x03F, 0x00000094, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x0000042A, + 0x033, 0x000000A1, + 0x03F, 0x00000829, + 0x033, 0x000000A2, + 0x03F, 0x00000848, + 0x033, 0x000000A3, + 0x03F, 0x0000084B, + 0x033, 0x000000A4, + 0x03F, 0x00000C4C, + 0x033, 0x000000A5, + 0x03F, 0x00000C8A, + 0x033, 0x000000A6, + 0x03F, 0x00000C8D, + 0x033, 0x000000A7, + 0x03F, 0x00000CEB, + 0x033, 0x000000A8, + 0x03F, 0x00000CEE, + 0x033, 0x000000A9, + 0x03F, 0x00000CF1, + 0x033, 0x000000AA, + 0x03F, 0x00000CF4, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000005, + 0x033, 0x000000A1, + 0x03F, 0x00000008, + 0x033, 0x000000A2, + 0x03F, 0x0000000B, + 0x033, 0x000000A3, + 0x03F, 0x0000000E, + 0x033, 0x000000A4, + 0x03F, 0x00000047, + 0x033, 0x000000A5, + 0x03F, 0x0000004A, + 0x033, 0x000000A6, + 0x03F, 0x0000004D, + 0x033, 0x000000A7, + 0x03F, 0x00000050, + 0x033, 0x000000A8, + 0x03F, 0x00000053, + 0x033, 0x000000A9, + 0x03F, 0x00000056, + 0x033, 0x000000AA, + 0x03F, 0x00000094, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x0000042A, + 0x033, 0x000000A1, + 0x03F, 0x00000829, + 0x033, 0x000000A2, + 0x03F, 0x00000848, + 0x033, 0x000000A3, + 0x03F, 0x0000084B, + 0x033, 0x000000A4, + 0x03F, 0x00000C4C, + 0x033, 0x000000A5, + 0x03F, 0x00000C8A, + 0x033, 0x000000A6, + 0x03F, 0x00000C8D, + 0x033, 0x000000A7, + 0x03F, 0x00000CEB, + 0x033, 0x000000A8, + 0x03F, 0x00000CEE, + 0x033, 0x000000A9, + 0x03F, 0x00000CF1, + 0x033, 0x000000AA, + 0x03F, 0x00000CF4, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000007, + 0x033, 0x000000A1, + 0x03F, 0x0000000A, + 0x033, 0x000000A2, + 0x03F, 0x0000000D, + 0x033, 0x000000A3, + 0x03F, 0x0000002A, + 0x033, 0x000000A4, + 0x03F, 0x0000002D, + 0x033, 0x000000A5, + 0x03F, 0x00000030, + 0x033, 0x000000A6, + 0x03F, 0x0000006D, + 0x033, 0x000000A7, + 0x03F, 0x00000070, + 0x033, 0x000000A8, + 0x03F, 0x000000ED, + 0x033, 0x000000A9, + 0x03F, 0x000000F0, + 0x033, 0x000000AA, + 0x03F, 0x000000F3, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000007, + 0x033, 0x000000A1, + 0x03F, 0x0000000A, + 0x033, 0x000000A2, + 0x03F, 0x0000000D, + 0x033, 0x000000A3, + 0x03F, 0x0000002A, + 0x033, 0x000000A4, + 0x03F, 0x0000002D, + 0x033, 0x000000A5, + 0x03F, 0x00000030, + 0x033, 0x000000A6, + 0x03F, 0x0000006D, + 0x033, 0x000000A7, + 0x03F, 0x00000070, + 0x033, 0x000000A8, + 0x03F, 0x000000ED, + 0x033, 0x000000A9, + 0x03F, 0x000000F0, + 0x033, 0x000000AA, + 0x03F, 0x000000F3, + 0xA0000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000C09, + 0x033, 0x000000A1, + 0x03F, 0x00000C0C, + 0x033, 0x000000A2, + 0x03F, 0x00000C0F, + 0x033, 0x000000A3, + 0x03F, 0x00000C2C, + 0x033, 0x000000A4, + 0x03F, 0x00000C2F, + 0x033, 0x000000A5, + 0x03F, 0x00000C8A, + 0x033, 0x000000A6, + 0x03F, 0x00000C8D, + 0x033, 0x000000A7, + 0x03F, 0x00000C90, + 0x033, 0x000000A8, + 0x03F, 0x00000CEF, + 0x033, 0x000000A9, + 0x03F, 0x00000CF2, + 0x033, 0x000000AA, + 0x03F, 0x00000CF5, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x0EF, 0x00000400, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0000047C, + 0x033, 0x00000001, + 0x03F, 0x0000047C, + 0x033, 0x00000002, + 0x03F, 0x0000047C, + 0x033, 0x00000003, + 0x03F, 0x0000047C, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0000047C, + 0x033, 0x00000001, + 0x03F, 0x0000047C, + 0x033, 0x00000002, + 0x03F, 0x0000047C, + 0x033, 0x00000003, + 0x03F, 0x0000047C, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0000047C, + 0x033, 0x00000001, + 0x03F, 0x0000047C, + 0x033, 0x00000002, + 0x03F, 0x0000047C, + 0x033, 0x00000003, + 0x03F, 0x0000047C, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0000047C, + 0x033, 0x00000001, + 0x03F, 0x0000047C, + 0x033, 0x00000002, + 0x03F, 0x0000047C, + 0x033, 0x00000003, + 0x03F, 0x0000047C, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0000047C, + 0x033, 0x00000001, + 0x03F, 0x0000047C, + 0x033, 0x00000002, + 0x03F, 0x0000047C, + 0x033, 0x00000003, + 0x03F, 0x0000047C, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x000008BB, + 0x033, 0x00000001, + 0x03F, 0x000008BB, + 0x033, 0x00000002, + 0x03F, 0x000008BB, + 0x033, 0x00000003, + 0x03F, 0x000008BB, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0000047C, + 0x033, 0x00000001, + 0x03F, 0x0000047C, + 0x033, 0x00000002, + 0x03F, 0x0000047C, + 0x033, 0x00000003, + 0x03F, 0x0000047C, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0000047C, + 0x033, 0x00000001, + 0x03F, 0x0000047C, + 0x033, 0x00000002, + 0x03F, 0x0000047C, + 0x033, 0x00000003, + 0x03F, 0x0000047C, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0000047C, + 0x033, 0x00000001, + 0x03F, 0x0000047C, + 0x033, 0x00000002, + 0x03F, 0x0000047C, + 0x033, 0x00000003, + 0x03F, 0x0000047C, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0000047C, + 0x033, 0x00000001, + 0x03F, 0x0000047C, + 0x033, 0x00000002, + 0x03F, 0x0000047C, + 0x033, 0x00000003, + 0x03F, 0x0000047C, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0000047C, + 0x033, 0x00000001, + 0x03F, 0x0000047C, + 0x033, 0x00000002, + 0x03F, 0x0000047C, + 0x033, 0x00000003, + 0x03F, 0x0000047C, + 0xA0000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x000004BB, + 0x033, 0x00000001, + 0x03F, 0x000004BB, + 0x033, 0x00000002, + 0x03F, 0x000004BB, + 0x033, 0x00000003, + 0x03F, 0x000004BB, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x0EF, 0x00000100, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00001726, + 0x033, 0x00000001, + 0x03F, 0x00001726, + 0x033, 0x00000002, + 0x03F, 0x00001726, + 0x033, 0x00000003, + 0x03F, 0x00001726, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00001726, + 0x033, 0x00000001, + 0x03F, 0x00001726, + 0x033, 0x00000002, + 0x03F, 0x00001726, + 0x033, 0x00000003, + 0x03F, 0x00001726, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00001726, + 0x033, 0x00000001, + 0x03F, 0x00001726, + 0x033, 0x00000002, + 0x03F, 0x00001726, + 0x033, 0x00000003, + 0x03F, 0x00001726, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00001726, + 0x033, 0x00000001, + 0x03F, 0x00001726, + 0x033, 0x00000002, + 0x03F, 0x00001726, + 0x033, 0x00000003, + 0x03F, 0x00001726, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00001726, + 0x033, 0x00000001, + 0x03F, 0x00001726, + 0x033, 0x00000002, + 0x03F, 0x00001726, + 0x033, 0x00000003, + 0x03F, 0x00001726, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00001726, + 0x033, 0x00000001, + 0x03F, 0x00001726, + 0x033, 0x00000002, + 0x03F, 0x00001726, + 0x033, 0x00000003, + 0x03F, 0x00001726, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00001726, + 0x033, 0x00000001, + 0x03F, 0x00001726, + 0x033, 0x00000002, + 0x03F, 0x00001726, + 0x033, 0x00000003, + 0x03F, 0x00001726, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00001726, + 0x033, 0x00000001, + 0x03F, 0x00001726, + 0x033, 0x00000002, + 0x03F, 0x00001726, + 0x033, 0x00000003, + 0x03F, 0x00001726, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00001726, + 0x033, 0x00000001, + 0x03F, 0x00001726, + 0x033, 0x00000002, + 0x03F, 0x00001726, + 0x033, 0x00000003, + 0x03F, 0x00001726, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00001726, + 0x033, 0x00000001, + 0x03F, 0x00001726, + 0x033, 0x00000002, + 0x03F, 0x00001726, + 0x033, 0x00000003, + 0x03F, 0x00001726, + 0xA0000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000F34, + 0x033, 0x00000001, + 0x03F, 0x00000F34, + 0x033, 0x00000002, + 0x03F, 0x00000F34, + 0x033, 0x00000003, + 0x03F, 0x00000F34, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x081, 0x0000F400, + 0x087, 0x00016040, + 0x051, 0x00000808, + 0x052, 0x00098002, + 0x053, 0x0000FA47, + 0x054, 0x00058032, + 0x056, 0x00051000, + 0x057, 0x0000CE0A, + 0x058, 0x00082030, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x081, 0x0000F400, + 0x087, 0x00016040, + 0x051, 0x00000808, + 0x052, 0x00098002, + 0x053, 0x0000FA47, + 0x054, 0x00058032, + 0x056, 0x00051000, + 0x057, 0x0000CE0A, + 0x058, 0x00082030, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x081, 0x0000F400, + 0x087, 0x00016040, + 0x051, 0x00000808, + 0x052, 0x00098002, + 0x053, 0x0000FA47, + 0x054, 0x00058032, + 0x056, 0x00051000, + 0x057, 0x0000CE0A, + 0x058, 0x00082030, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x081, 0x0000F400, + 0x087, 0x00016040, + 0x051, 0x00000808, + 0x052, 0x00098002, + 0x053, 0x0000FA47, + 0x054, 0x00058032, + 0x056, 0x00051000, + 0x057, 0x0000CE0A, + 0x058, 0x00082030, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x081, 0x0000F400, + 0x087, 0x00016040, + 0x051, 0x00000808, + 0x052, 0x00098002, + 0x053, 0x0000FA47, + 0x054, 0x00058032, + 0x056, 0x00051000, + 0x057, 0x0000CE0A, + 0x058, 0x00082030, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x081, 0x0000F400, + 0x087, 0x00016040, + 0x051, 0x00000808, + 0x052, 0x00098002, + 0x053, 0x0000FA47, + 0x054, 0x00058032, + 0x056, 0x00051000, + 0x057, 0x0000CE0A, + 0x058, 0x00082030, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x081, 0x0000F400, + 0x087, 0x00016040, + 0x051, 0x00000808, + 0x052, 0x00098002, + 0x053, 0x0000FA47, + 0x054, 0x00058032, + 0x056, 0x00051000, + 0x057, 0x0000CE0A, + 0x058, 0x00082030, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x081, 0x0000F400, + 0x087, 0x00016040, + 0x051, 0x00000808, + 0x052, 0x00098002, + 0x053, 0x0000FA47, + 0x054, 0x00058032, + 0x056, 0x00051000, + 0x057, 0x0000CE0A, + 0x058, 0x00082030, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x081, 0x0000F400, + 0x087, 0x00016040, + 0x051, 0x00000808, + 0x052, 0x00098002, + 0x053, 0x0000FA47, + 0x054, 0x00058032, + 0x056, 0x00051000, + 0x057, 0x0000CE0A, + 0x058, 0x00082030, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x081, 0x0000F400, + 0x087, 0x00016040, + 0x051, 0x00000808, + 0x052, 0x00098002, + 0x053, 0x0000FA47, + 0x054, 0x00058032, + 0x056, 0x00051000, + 0x057, 0x0000CE0A, + 0x058, 0x00082030, + 0xA0000000, 0x00000000, + 0x081, 0x0000F000, + 0x087, 0x00016040, + 0x051, 0x00000C00, + 0x052, 0x0007C241, + 0x053, 0x0001C069, + 0x054, 0x00078032, + 0x057, 0x0000CE0A, + 0x058, 0x00058750, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000800, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000003, + 0x033, 0x00000001, + 0x03F, 0x00000006, + 0x033, 0x00000002, + 0x03F, 0x00000009, + 0x033, 0x00000003, + 0x03F, 0x00000026, + 0x033, 0x00000004, + 0x03F, 0x00000029, + 0x033, 0x00000005, + 0x03F, 0x0000002C, + 0x033, 0x00000006, + 0x03F, 0x0000002F, + 0x033, 0x00000007, + 0x03F, 0x00000033, + 0x033, 0x00000008, + 0x03F, 0x00000036, + 0x033, 0x00000009, + 0x03F, 0x00000039, + 0x033, 0x0000000A, + 0x03F, 0x0000003C, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000002, + 0x033, 0x00000001, + 0x03F, 0x00000005, + 0x033, 0x00000002, + 0x03F, 0x00000008, + 0x033, 0x00000003, + 0x03F, 0x0000000B, + 0x033, 0x00000004, + 0x03F, 0x0000000E, + 0x033, 0x00000005, + 0x03F, 0x0000002B, + 0x033, 0x00000006, + 0x03F, 0x0000002E, + 0x033, 0x00000007, + 0x03F, 0x00000031, + 0x033, 0x00000008, + 0x03F, 0x0000006E, + 0x033, 0x00000009, + 0x03F, 0x00000071, + 0x033, 0x0000000A, + 0x03F, 0x00000074, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000003, + 0x033, 0x00000001, + 0x03F, 0x00000006, + 0x033, 0x00000002, + 0x03F, 0x00000009, + 0x033, 0x00000003, + 0x03F, 0x00000026, + 0x033, 0x00000004, + 0x03F, 0x00000029, + 0x033, 0x00000005, + 0x03F, 0x0000002C, + 0x033, 0x00000006, + 0x03F, 0x0000002F, + 0x033, 0x00000007, + 0x03F, 0x00000033, + 0x033, 0x00000008, + 0x03F, 0x00000036, + 0x033, 0x00000009, + 0x03F, 0x00000039, + 0x033, 0x0000000A, + 0x03F, 0x0000003C, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000003, + 0x033, 0x00000001, + 0x03F, 0x00000006, + 0x033, 0x00000002, + 0x03F, 0x00000009, + 0x033, 0x00000003, + 0x03F, 0x00000026, + 0x033, 0x00000004, + 0x03F, 0x00000029, + 0x033, 0x00000005, + 0x03F, 0x0000002C, + 0x033, 0x00000006, + 0x03F, 0x0000002F, + 0x033, 0x00000007, + 0x03F, 0x00000033, + 0x033, 0x00000008, + 0x03F, 0x00000036, + 0x033, 0x00000009, + 0x03F, 0x00000039, + 0x033, 0x0000000A, + 0x03F, 0x0000003C, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000003, + 0x033, 0x00000001, + 0x03F, 0x00000006, + 0x033, 0x00000002, + 0x03F, 0x00000009, + 0x033, 0x00000003, + 0x03F, 0x00000026, + 0x033, 0x00000004, + 0x03F, 0x00000029, + 0x033, 0x00000005, + 0x03F, 0x0000002C, + 0x033, 0x00000006, + 0x03F, 0x0000002F, + 0x033, 0x00000007, + 0x03F, 0x00000033, + 0x033, 0x00000008, + 0x03F, 0x00000036, + 0x033, 0x00000009, + 0x03F, 0x00000039, + 0x033, 0x0000000A, + 0x03F, 0x0000003C, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000002, + 0x033, 0x00000001, + 0x03F, 0x00000005, + 0x033, 0x00000002, + 0x03F, 0x00000008, + 0x033, 0x00000003, + 0x03F, 0x0000000B, + 0x033, 0x00000004, + 0x03F, 0x0000000E, + 0x033, 0x00000005, + 0x03F, 0x0000002B, + 0x033, 0x00000006, + 0x03F, 0x0000002E, + 0x033, 0x00000007, + 0x03F, 0x00000031, + 0x033, 0x00000008, + 0x03F, 0x0000006E, + 0x033, 0x00000009, + 0x03F, 0x00000071, + 0x033, 0x0000000A, + 0x03F, 0x00000074, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000003, + 0x033, 0x00000001, + 0x03F, 0x00000006, + 0x033, 0x00000002, + 0x03F, 0x00000009, + 0x033, 0x00000003, + 0x03F, 0x00000026, + 0x033, 0x00000004, + 0x03F, 0x00000029, + 0x033, 0x00000005, + 0x03F, 0x0000002C, + 0x033, 0x00000006, + 0x03F, 0x0000002F, + 0x033, 0x00000007, + 0x03F, 0x00000033, + 0x033, 0x00000008, + 0x03F, 0x00000036, + 0x033, 0x00000009, + 0x03F, 0x00000039, + 0x033, 0x0000000A, + 0x03F, 0x0000003C, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000003, + 0x033, 0x00000001, + 0x03F, 0x00000006, + 0x033, 0x00000002, + 0x03F, 0x00000009, + 0x033, 0x00000003, + 0x03F, 0x00000026, + 0x033, 0x00000004, + 0x03F, 0x00000029, + 0x033, 0x00000005, + 0x03F, 0x0000002C, + 0x033, 0x00000006, + 0x03F, 0x0000002F, + 0x033, 0x00000007, + 0x03F, 0x00000033, + 0x033, 0x00000008, + 0x03F, 0x00000036, + 0x033, 0x00000009, + 0x03F, 0x00000039, + 0x033, 0x0000000A, + 0x03F, 0x0000003C, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000003, + 0x033, 0x00000001, + 0x03F, 0x00000006, + 0x033, 0x00000002, + 0x03F, 0x00000009, + 0x033, 0x00000003, + 0x03F, 0x00000026, + 0x033, 0x00000004, + 0x03F, 0x00000029, + 0x033, 0x00000005, + 0x03F, 0x0000002C, + 0x033, 0x00000006, + 0x03F, 0x0000002F, + 0x033, 0x00000007, + 0x03F, 0x00000033, + 0x033, 0x00000008, + 0x03F, 0x00000036, + 0x033, 0x00000009, + 0x03F, 0x00000039, + 0x033, 0x0000000A, + 0x03F, 0x0000003C, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000003, + 0x033, 0x00000001, + 0x03F, 0x00000006, + 0x033, 0x00000002, + 0x03F, 0x00000009, + 0x033, 0x00000003, + 0x03F, 0x00000026, + 0x033, 0x00000004, + 0x03F, 0x00000029, + 0x033, 0x00000005, + 0x03F, 0x0000002C, + 0x033, 0x00000006, + 0x03F, 0x0000002F, + 0x033, 0x00000007, + 0x03F, 0x00000033, + 0x033, 0x00000008, + 0x03F, 0x00000036, + 0x033, 0x00000009, + 0x03F, 0x00000039, + 0x033, 0x0000000A, + 0x03F, 0x0000003C, + 0xA0000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0005142C, + 0x033, 0x00000001, + 0x03F, 0x0005144B, + 0x033, 0x00000002, + 0x03F, 0x0005144E, + 0x033, 0x00000003, + 0x03F, 0x00051C69, + 0x033, 0x00000004, + 0x03F, 0x00051C6C, + 0x033, 0x00000005, + 0x03F, 0x00051C6F, + 0x033, 0x00000006, + 0x03F, 0x00051CEB, + 0x033, 0x00000007, + 0x03F, 0x00051CEE, + 0x033, 0x00000008, + 0x03F, 0x00051CF1, + 0x033, 0x00000009, + 0x03F, 0x00051CF4, + 0x033, 0x0000000A, + 0x03F, 0x00051CF7, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x0EF, 0x00000010, + 0x033, 0x00000000, + 0x008, 0x0009C060, + 0x033, 0x00000001, + 0x008, 0x0009C060, + 0x0EF, 0x00000000, + 0x033, 0x000000A2, + 0x0EF, 0x00080000, + 0x03E, 0x0000593F, + 0x8300000c, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000D0F4F, + 0xA0000000, 0x00000000, + 0x03F, 0x000C0F4F, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x033, 0x000000A3, + 0x0EF, 0x00080000, + 0x03E, 0x00005934, + 0x03F, 0x0005AFCF, + 0x0EF, 0x00000000, + 0x83000002, 0x00000000, 0x40000000, 0x00000000, + 0x0CE, 0x00094400, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x0CE, 0x00094400, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x0CE, 0x00094400, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x0CE, 0x00094400, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x0CE, 0x00094400, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x0CE, 0x00094400, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x0CE, 0x00094400, + 0xA0000000, 0x00000000, + 0x0CE, 0x00094C00, + 0xB0000000, 0x00000000, + 0x83000002, 0x00000000, 0x40000000, 0x00000000, + 0x0CF, 0x00072F00, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x0CF, 0x00072F00, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x0CF, 0x00072F00, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x0CF, 0x00072F00, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x0CF, 0x00064700, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x0CF, 0x00072F00, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x0CF, 0x00072F00, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x0CF, 0x00072F00, + 0xA0000000, 0x00000000, + 0x0CF, 0x00064700, + 0xB0000000, 0x00000000, + 0x83000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000004, + 0x033, 0x00000000, + 0x03F, 0x00000056, + 0x033, 0x00000001, + 0x03F, 0x000000D6, + 0x0EF, 0x00000000, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000004, + 0x033, 0x00000000, + 0x03F, 0x00000056, + 0x033, 0x00000001, + 0x03F, 0x000000D6, + 0x0EF, 0x00000000, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000004, + 0x033, 0x00000000, + 0x03F, 0x00000056, + 0x033, 0x00000001, + 0x03F, 0x000000D6, + 0x0EF, 0x00000000, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000004, + 0x033, 0x00000000, + 0x03F, 0x00000056, + 0x033, 0x00000001, + 0x03F, 0x000000D6, + 0x0EF, 0x00000000, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000004, + 0x033, 0x00000000, + 0x03F, 0x00000096, + 0x033, 0x00000001, + 0x03F, 0x000000D6, + 0x0EF, 0x00000000, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000004, + 0x033, 0x00000000, + 0x03F, 0x00000056, + 0x033, 0x00000001, + 0x03F, 0x00000056, + 0x0EF, 0x00000000, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000004, + 0x033, 0x00000000, + 0x03F, 0x00000056, + 0x033, 0x00000001, + 0x03F, 0x000000D6, + 0x0EF, 0x00000000, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000004, + 0x033, 0x00000000, + 0x03F, 0x00000056, + 0x033, 0x00000001, + 0x03F, 0x000000D6, + 0x0EF, 0x00000000, + 0xA0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000096, + 0x033, 0x00000001, + 0x03F, 0x000000D6, + 0x0EF, 0x00000000, + 0xB0000000, 0x00000000, + 0x0B0, 0x000FF0FC, + 0x0C4, 0x00081402, + 0x0CC, 0x00082000, +}; + +RTW_DECL_TABLE_RF_RADIO(rtw8822b_rf_a, A); + +static const u32 rtw8822b_rf_b[] = { + 0x000, 0x00030000, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x0004002D, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x0004002D, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x0004002D, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x0004002D, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x0004002D, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x0004002D, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x0004002D, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x0004002D, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x00040029, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x0004002D, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x001, 0x0004002D, + 0xA0000000, 0x00000000, + 0x001, 0x00040029, + 0xB0000000, 0x00000000, + 0x018, 0x00010D24, + 0x0EF, 0x00080000, + 0x033, 0x00000002, + 0x03E, 0x0000003F, + 0x8300000c, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000D0F4E, + 0xA0000000, 0x00000000, + 0x03F, 0x000C0F4E, + 0xB0000000, 0x00000000, + 0x033, 0x00000001, + 0x03E, 0x00000034, + 0x03F, 0x0004080E, + 0x0EF, 0x00080000, + 0x0DF, 0x00002449, + 0x033, 0x00000024, + 0x03E, 0x0000003F, + 0x03F, 0x00060FDE, + 0x0EF, 0x00000000, + 0x0EF, 0x00080000, + 0x033, 0x00000025, + 0x03E, 0x00000037, + 0x03F, 0x0007EFCE, + 0x0EF, 0x00000000, + 0x0EF, 0x00080000, + 0x033, 0x00000026, + 0x03E, 0x00000037, + 0x03F, 0x000DEFCE, + 0x0EF, 0x00000000, + 0x0DF, 0x00000009, + 0x018, 0x00010524, + 0x089, 0x00000207, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FF186, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FF186, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FE186, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FF186, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FF186, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FF186, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FE186, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x08A, 0x000FF186, + 0xA0000000, 0x00000000, + 0x08A, 0x000FF186, + 0xB0000000, 0x00000000, + 0x08B, 0x00061E3C, + 0x08C, 0x000112C7, + 0x08D, 0x000F4988, + 0x08E, 0x00064D40, + 0x0EF, 0x00020000, + 0x033, 0x00000007, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C3186, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C0006, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C0006, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004080, + 0x03F, 0x000C3186, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C0006, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C3186, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C0006, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C3186, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C3186, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C0006, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C0006, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C0006, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004080, + 0x03F, 0x000C3186, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0xA0000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C3186, + 0xB0000000, 0x00000000, + 0x033, 0x00000006, + 0x03E, 0x00004080, + 0x03F, 0x000C3186, + 0x033, 0x00000005, + 0x03E, 0x000040C8, + 0x03F, 0x000C3186, + 0x033, 0x00000004, + 0x03E, 0x00004190, + 0x03F, 0x000C3186, + 0x033, 0x00000003, + 0x03E, 0x00004998, + 0x03F, 0x000C3186, + 0x033, 0x00000002, + 0x03E, 0x00005840, + 0x03F, 0x000C3186, + 0x033, 0x00000001, + 0x03E, 0x000058C2, + 0x03F, 0x000C3186, + 0x033, 0x00000000, + 0x03E, 0x00005930, + 0x03F, 0x000C3186, + 0x033, 0x0000000F, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C3186, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C0006, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C3186, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004080, + 0x03F, 0x000C3186, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C0006, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C3186, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C0006, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C3186, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C3186, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C0006, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C0006, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C0006, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004080, + 0x03F, 0x000C3186, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0xA0000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C3186, + 0xB0000000, 0x00000000, + 0x033, 0x0000000E, + 0x03E, 0x00004080, + 0x03F, 0x000C3186, + 0x033, 0x0000000D, + 0x8300000f, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x000040D0, + 0xA0000000, 0x00000000, + 0x03E, 0x000040C8, + 0xB0000000, 0x00000000, + 0x03F, 0x000C3186, + 0x033, 0x0000000C, + 0x03E, 0x00004190, + 0x03F, 0x000C3186, + 0x033, 0x0000000B, + 0x03E, 0x00004998, + 0x03F, 0x000C3186, + 0x033, 0x0000000A, + 0x03E, 0x00005840, + 0x03F, 0x000C3186, + 0x033, 0x00000009, + 0x03E, 0x000058C2, + 0x03F, 0x000C3186, + 0x033, 0x00000008, + 0x03E, 0x00005930, + 0x03F, 0x000C3186, + 0x033, 0x00000017, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C3186, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C0006, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000DFF86, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004080, + 0x03F, 0x000C3186, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C3186, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C3186, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C0006, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000DFF86, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C3186, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C0006, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C0006, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C0006, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004080, + 0x03F, 0x000C3186, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x00004040, + 0x03F, 0x000C3186, + 0xA0000000, 0x00000000, + 0x03E, 0x00004000, + 0x03F, 0x000C3186, + 0xB0000000, 0x00000000, + 0x033, 0x00000016, + 0x03E, 0x00004080, + 0x03F, 0x000C3186, + 0x033, 0x00000015, + 0x8300000f, 0x00000000, 0x40000000, 0x00000000, + 0x03E, 0x000040D0, + 0xA0000000, 0x00000000, + 0x03E, 0x000040C8, + 0xB0000000, 0x00000000, + 0x03F, 0x000C3186, + 0x033, 0x00000014, + 0x03E, 0x00004190, + 0x03F, 0x000C3186, + 0x033, 0x00000013, + 0x03E, 0x00004998, + 0x03F, 0x000C3186, + 0x033, 0x00000012, + 0x03E, 0x00005840, + 0x03F, 0x000C3186, + 0x033, 0x00000011, + 0x03E, 0x000058C2, + 0x03F, 0x000C3186, + 0x033, 0x00000010, + 0x03E, 0x00005930, + 0x03F, 0x000C3186, + 0x0EF, 0x00000000, + 0x0EF, 0x00004000, + 0x033, 0x00000000, + 0x03F, 0x0000000A, + 0x033, 0x00000001, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000005, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000002, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000005, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000005, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000005, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00000000, + 0xA0000000, 0x00000000, + 0x03F, 0x00000005, + 0xB0000000, 0x00000000, + 0x033, 0x00000002, + 0x03F, 0x00000000, + 0x0EF, 0x00000000, + 0x018, 0x00000401, + 0x084, 0x00001209, + 0x086, 0x000001A0, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x087, 0x00068080, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x087, 0x00068080, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x087, 0x00068080, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x087, 0x00068080, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x087, 0x00068080, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x087, 0x00068080, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x087, 0x00068080, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x087, 0x00068080, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x087, 0x00068080, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x087, 0x00068080, + 0xA0000000, 0x00000000, + 0x087, 0x000E8180, + 0xB0000000, 0x00000000, + 0x088, 0x00070020, + 0x0DE, 0x00000010, + 0x0EF, 0x00008000, + 0x033, 0x0000000F, + 0x03F, 0x0000003C, + 0x033, 0x0000000E, + 0x03F, 0x00000038, + 0x033, 0x0000000D, + 0x03F, 0x00000030, + 0x033, 0x0000000C, + 0x03F, 0x00000028, + 0x033, 0x0000000B, + 0x03F, 0x00000020, + 0x033, 0x0000000A, + 0x03F, 0x00000018, + 0x033, 0x00000009, + 0x03F, 0x00000010, + 0x033, 0x00000008, + 0x03F, 0x00000008, + 0x033, 0x00000007, + 0x03F, 0x0000003C, + 0x033, 0x00000006, + 0x03F, 0x00000038, + 0x033, 0x00000005, + 0x03F, 0x00000030, + 0x033, 0x00000004, + 0x03F, 0x00000028, + 0x033, 0x00000003, + 0x03F, 0x00000020, + 0x033, 0x00000002, + 0x03F, 0x00000018, + 0x033, 0x00000001, + 0x03F, 0x00000010, + 0x033, 0x00000000, + 0x03F, 0x00000008, + 0x0EF, 0x00000000, + 0x018, 0x00018D24, + 0xFFE, 0x00000000, + 0xFFE, 0x00000000, + 0xFFE, 0x00000000, + 0xFFE, 0x00000000, + 0x018, 0x00010D24, + 0x01B, 0x00075A40, + 0x0EE, 0x00000002, + 0x033, 0x00000000, + 0x03F, 0x00000004, + 0x033, 0x00000001, + 0x03F, 0x00000004, + 0x033, 0x00000002, + 0x03F, 0x00000004, + 0x033, 0x00000003, + 0x03F, 0x00000004, + 0x033, 0x00000004, + 0x03F, 0x00000004, + 0x033, 0x00000005, + 0x03F, 0x00000006, + 0x033, 0x00000006, + 0x03F, 0x00000004, + 0x033, 0x00000007, + 0x03F, 0x00000000, + 0x0EE, 0x00000000, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D3D1, + 0x062, 0x0000D3A2, + 0x063, 0x00000002, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D4A0, + 0x062, 0x0000D203, + 0x063, 0x00000062, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D2A1, + 0x062, 0x0000D3A2, + 0x063, 0x00000062, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D2A1, + 0x062, 0x0000D3A2, + 0x063, 0x00000002, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D2A1, + 0x062, 0x0000D3A2, + 0x063, 0x00000062, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D2A1, + 0x062, 0x0000D3A2, + 0x063, 0x00000002, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D4A0, + 0x062, 0x0000D203, + 0x063, 0x00000062, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D4A0, + 0x062, 0x0000D203, + 0x063, 0x00000062, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D2A1, + 0x062, 0x0000D3A2, + 0x063, 0x00000002, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D3D1, + 0x062, 0x0000D3A2, + 0x063, 0x00000002, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D3D1, + 0x062, 0x0000D3A2, + 0x063, 0x00000002, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D4A0, + 0x062, 0x0000D203, + 0x063, 0x00000062, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D2A1, + 0x062, 0x0000D3A2, + 0x063, 0x00000002, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D3D1, + 0x062, 0x0000D3A2, + 0x063, 0x00000002, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D3D1, + 0x062, 0x0000D3A2, + 0x063, 0x00000002, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D2A1, + 0x062, 0x0000D3A2, + 0x063, 0x00000002, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D2A1, + 0x062, 0x0000D3A2, + 0x063, 0x00000002, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D2A1, + 0x062, 0x0000D3A2, + 0x063, 0x00000002, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D2A1, + 0x062, 0x0000D3A2, + 0x063, 0x00000062, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D4A0, + 0x062, 0x0000D203, + 0x063, 0x00000062, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D2A1, + 0x062, 0x0000D3A2, + 0x063, 0x00000062, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D2A1, + 0x062, 0x0000D3A2, + 0x063, 0x00000002, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D2A1, + 0x062, 0x0000D3A2, + 0x063, 0x00000062, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D2A1, + 0x062, 0x0000D3A2, + 0x063, 0x00000002, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D4A0, + 0x062, 0x0000D203, + 0x063, 0x00000062, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x061, 0x0005D4A0, + 0x062, 0x0000D203, + 0x063, 0x00000062, + 0xA0000000, 0x00000000, + 0x061, 0x0005D3D0, + 0x062, 0x0000D303, + 0x063, 0x00000002, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000200, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A3, + 0x030, 0x000053A3, + 0x030, 0x000063A3, + 0x030, 0x000073A3, + 0x030, 0x000083A3, + 0x030, 0x000093A3, + 0x030, 0x0000A3A3, + 0x030, 0x0000B3A3, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000004A3, + 0x030, 0x000014A3, + 0x030, 0x000024A3, + 0x030, 0x000034A3, + 0x030, 0x000044A3, + 0x030, 0x000054A3, + 0x030, 0x000064A3, + 0x030, 0x000074A3, + 0x030, 0x000084A3, + 0x030, 0x000094A3, + 0x030, 0x0000A4A3, + 0x030, 0x0000B4A3, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000002A6, + 0x030, 0x000012A6, + 0x030, 0x000022A6, + 0x030, 0x000032A6, + 0x030, 0x000042A6, + 0x030, 0x000052A6, + 0x030, 0x000062A6, + 0x030, 0x000072A6, + 0x030, 0x000082A6, + 0x030, 0x000092A6, + 0x030, 0x0000A2A6, + 0x030, 0x0000B2A6, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000303, + 0x030, 0x00001303, + 0x030, 0x00002303, + 0x030, 0x00003303, + 0x030, 0x000043A4, + 0x030, 0x000053A4, + 0x030, 0x000063A4, + 0x030, 0x000073A4, + 0x030, 0x00008365, + 0x030, 0x00009365, + 0x030, 0x0000A365, + 0x030, 0x0000B365, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000002A6, + 0x030, 0x000012A6, + 0x030, 0x000022A6, + 0x030, 0x000032A6, + 0x030, 0x000042A6, + 0x030, 0x000052A6, + 0x030, 0x000062A6, + 0x030, 0x000072A6, + 0x030, 0x000082A6, + 0x030, 0x000092A6, + 0x030, 0x0000A2A6, + 0x030, 0x0000B2A6, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000004A4, + 0x030, 0x000014A4, + 0x030, 0x000024A4, + 0x030, 0x000034A4, + 0x030, 0x000043A4, + 0x030, 0x000053A4, + 0x030, 0x000063A4, + 0x030, 0x000073A4, + 0x030, 0x000083A5, + 0x030, 0x000093A5, + 0x030, 0x0000A3A5, + 0x030, 0x0000B3A5, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000004A3, + 0x030, 0x000014A3, + 0x030, 0x000024A3, + 0x030, 0x000034A3, + 0x030, 0x000044A3, + 0x030, 0x000054A3, + 0x030, 0x000064A3, + 0x030, 0x000074A3, + 0x030, 0x000084A3, + 0x030, 0x000094A3, + 0x030, 0x0000A4A3, + 0x030, 0x0000B4A3, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000004A3, + 0x030, 0x000014A3, + 0x030, 0x000024A3, + 0x030, 0x000034A3, + 0x030, 0x000044A3, + 0x030, 0x000054A3, + 0x030, 0x000064A3, + 0x030, 0x000074A3, + 0x030, 0x000084A3, + 0x030, 0x000094A3, + 0x030, 0x0000A4A3, + 0x030, 0x0000B4A3, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000002F4, + 0x030, 0x000012F4, + 0x030, 0x000022F4, + 0x030, 0x000032F4, + 0x030, 0x00004365, + 0x030, 0x00005365, + 0x030, 0x00006365, + 0x030, 0x00007365, + 0x030, 0x000082A4, + 0x030, 0x000092A4, + 0x030, 0x0000A2A4, + 0x030, 0x0000B2A4, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000382, + 0x030, 0x00001382, + 0x030, 0x00002382, + 0x030, 0x00003382, + 0x030, 0x00004445, + 0x030, 0x00005445, + 0x030, 0x00006445, + 0x030, 0x00007445, + 0x030, 0x00008425, + 0x030, 0x00009425, + 0x030, 0x0000A425, + 0x030, 0x0000B425, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A3, + 0x030, 0x000053A3, + 0x030, 0x000063A3, + 0x030, 0x000073A3, + 0x030, 0x000083A3, + 0x030, 0x000093A3, + 0x030, 0x0000A3A3, + 0x030, 0x0000B3A3, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000004A3, + 0x030, 0x000014A3, + 0x030, 0x000024A3, + 0x030, 0x000034A3, + 0x030, 0x000044A3, + 0x030, 0x000054A3, + 0x030, 0x000064A3, + 0x030, 0x000074A3, + 0x030, 0x000084A3, + 0x030, 0x000094A3, + 0x030, 0x0000A4A3, + 0x030, 0x0000B4A3, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000443, + 0x030, 0x00001443, + 0x030, 0x00002443, + 0x030, 0x00003443, + 0x030, 0x000043A4, + 0x030, 0x000053A4, + 0x030, 0x000063A4, + 0x030, 0x000073A4, + 0x030, 0x00008365, + 0x030, 0x00009365, + 0x030, 0x0000A365, + 0x030, 0x0000B365, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000343, + 0x030, 0x00001343, + 0x030, 0x00002343, + 0x030, 0x00003343, + 0x030, 0x00004483, + 0x030, 0x00005483, + 0x030, 0x00006483, + 0x030, 0x00007483, + 0x030, 0x000083A4, + 0x030, 0x000093A4, + 0x030, 0x0000A3A4, + 0x030, 0x0000B3A4, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x00004423, + 0x030, 0x00005423, + 0x030, 0x00006423, + 0x030, 0x00007423, + 0x030, 0x00008324, + 0x030, 0x00009324, + 0x030, 0x0000A324, + 0x030, 0x0000B324, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000303, + 0x030, 0x00001303, + 0x030, 0x00002303, + 0x030, 0x00003303, + 0x030, 0x000043A4, + 0x030, 0x000053A4, + 0x030, 0x000063A4, + 0x030, 0x000073A4, + 0x030, 0x00008365, + 0x030, 0x00009365, + 0x030, 0x0000A365, + 0x030, 0x0000B365, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000403, + 0x030, 0x00001403, + 0x030, 0x00002403, + 0x030, 0x00003403, + 0x030, 0x000043A4, + 0x030, 0x000053A4, + 0x030, 0x000063A4, + 0x030, 0x000073A4, + 0x030, 0x000083A3, + 0x030, 0x000093A3, + 0x030, 0x0000A3A3, + 0x030, 0x0000B3A3, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A3, + 0x030, 0x000013A3, + 0x030, 0x000023A3, + 0x030, 0x000033A3, + 0x030, 0x000043A4, + 0x030, 0x000053A4, + 0x030, 0x000063A4, + 0x030, 0x000073A4, + 0x030, 0x00008365, + 0x030, 0x00009365, + 0x030, 0x0000A365, + 0x030, 0x0000B365, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000002A6, + 0x030, 0x000012A6, + 0x030, 0x000022A6, + 0x030, 0x000032A6, + 0x030, 0x000042A6, + 0x030, 0x000052A6, + 0x030, 0x000062A6, + 0x030, 0x000072A6, + 0x030, 0x000082A6, + 0x030, 0x000092A6, + 0x030, 0x0000A2A6, + 0x030, 0x0000B2A6, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000004A0, + 0x030, 0x000014A0, + 0x030, 0x000024A0, + 0x030, 0x000034A0, + 0x030, 0x000044A0, + 0x030, 0x000054A0, + 0x030, 0x000064A0, + 0x030, 0x000074A0, + 0x030, 0x000084A0, + 0x030, 0x000094A0, + 0x030, 0x0000A4A0, + 0x030, 0x0000B4A0, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000002A1, + 0x030, 0x000012A1, + 0x030, 0x000022A1, + 0x030, 0x000032A1, + 0x030, 0x000042A1, + 0x030, 0x000052A1, + 0x030, 0x000062A1, + 0x030, 0x000072A1, + 0x030, 0x000082A1, + 0x030, 0x000092A1, + 0x030, 0x0000A2A1, + 0x030, 0x0000B2A1, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000004A0, + 0x030, 0x000014A0, + 0x030, 0x000024A0, + 0x030, 0x000034A0, + 0x030, 0x000043A1, + 0x030, 0x000053A1, + 0x030, 0x000063A1, + 0x030, 0x000073A1, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000002A1, + 0x030, 0x000012A1, + 0x030, 0x000022A1, + 0x030, 0x000032A1, + 0x030, 0x000042A1, + 0x030, 0x000052A1, + 0x030, 0x000062A1, + 0x030, 0x000072A1, + 0x030, 0x000082A1, + 0x030, 0x000092A1, + 0x030, 0x0000A2A1, + 0x030, 0x0000B2A1, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000004A1, + 0x030, 0x000014A1, + 0x030, 0x000024A1, + 0x030, 0x000034A1, + 0x030, 0x000043A1, + 0x030, 0x000053A1, + 0x030, 0x000063A1, + 0x030, 0x000073A1, + 0x030, 0x000083A1, + 0x030, 0x000093A1, + 0x030, 0x0000A3A1, + 0x030, 0x0000B3A1, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000004A0, + 0x030, 0x000014A0, + 0x030, 0x000024A0, + 0x030, 0x000034A0, + 0x030, 0x000044A0, + 0x030, 0x000054A0, + 0x030, 0x000064A0, + 0x030, 0x000074A0, + 0x030, 0x000084A0, + 0x030, 0x000094A0, + 0x030, 0x0000A4A0, + 0x030, 0x0000B4A0, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000004A0, + 0x030, 0x000014A0, + 0x030, 0x000024A0, + 0x030, 0x000034A0, + 0x030, 0x000044A0, + 0x030, 0x000054A0, + 0x030, 0x000064A0, + 0x030, 0x000074A0, + 0x030, 0x000084A0, + 0x030, 0x000094A0, + 0x030, 0x0000A4A0, + 0x030, 0x0000B4A0, + 0xA0000000, 0x00000000, + 0x030, 0x000002D0, + 0x030, 0x000012D0, + 0x030, 0x000022D0, + 0x030, 0x000032D0, + 0x030, 0x000042D0, + 0x030, 0x000052D0, + 0x030, 0x000062D0, + 0x030, 0x000072D0, + 0x030, 0x000082D0, + 0x030, 0x000092D0, + 0x030, 0x0000A2D0, + 0x030, 0x0000B2D0, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x0EF, 0x00000080, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000203, + 0x030, 0x00001203, + 0x030, 0x00002203, + 0x030, 0x00003203, + 0x030, 0x00004203, + 0x030, 0x00005203, + 0x030, 0x00006203, + 0x030, 0x00007203, + 0x030, 0x00008203, + 0x030, 0x00009203, + 0x030, 0x0000A203, + 0x030, 0x0000B203, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000203, + 0x030, 0x00001203, + 0x030, 0x00002203, + 0x030, 0x00003203, + 0x030, 0x00004203, + 0x030, 0x00005203, + 0x030, 0x00006203, + 0x030, 0x00007203, + 0x030, 0x00008203, + 0x030, 0x00009203, + 0x030, 0x0000A203, + 0x030, 0x0000B203, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000203, + 0x030, 0x00001203, + 0x030, 0x00002203, + 0x030, 0x00003203, + 0x030, 0x00004203, + 0x030, 0x00005203, + 0x030, 0x00006203, + 0x030, 0x00007203, + 0x030, 0x00008203, + 0x030, 0x00009203, + 0x030, 0x0000A203, + 0x030, 0x0000B203, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A3, + 0x030, 0x000013A3, + 0x030, 0x000023A3, + 0x030, 0x000033A3, + 0x030, 0x000043A4, + 0x030, 0x000053A4, + 0x030, 0x000063A4, + 0x030, 0x000073A4, + 0x030, 0x000083A3, + 0x030, 0x000093A3, + 0x030, 0x0000A3A3, + 0x030, 0x0000B3A3, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000203, + 0x030, 0x00001203, + 0x030, 0x00002203, + 0x030, 0x00003203, + 0x030, 0x00004203, + 0x030, 0x00005203, + 0x030, 0x00006203, + 0x030, 0x00007203, + 0x030, 0x00008203, + 0x030, 0x00009203, + 0x030, 0x0000A203, + 0x030, 0x0000B203, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000203, + 0x030, 0x00001203, + 0x030, 0x00002203, + 0x030, 0x00003203, + 0x030, 0x00004203, + 0x030, 0x00005203, + 0x030, 0x00006203, + 0x030, 0x00007203, + 0x030, 0x00008203, + 0x030, 0x00009203, + 0x030, 0x0000A203, + 0x030, 0x0000B203, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000203, + 0x030, 0x00001203, + 0x030, 0x00002203, + 0x030, 0x00003203, + 0x030, 0x00004203, + 0x030, 0x00005203, + 0x030, 0x00006203, + 0x030, 0x00007203, + 0x030, 0x00008203, + 0x030, 0x00009203, + 0x030, 0x0000A203, + 0x030, 0x0000B203, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000203, + 0x030, 0x00001203, + 0x030, 0x00002203, + 0x030, 0x00003203, + 0x030, 0x00004203, + 0x030, 0x00005203, + 0x030, 0x00006203, + 0x030, 0x00007203, + 0x030, 0x00008203, + 0x030, 0x00009203, + 0x030, 0x0000A203, + 0x030, 0x0000B203, + 0xA0000000, 0x00000000, + 0x030, 0x000003A2, + 0x030, 0x000013A2, + 0x030, 0x000023A2, + 0x030, 0x000033A2, + 0x030, 0x000043A2, + 0x030, 0x000053A2, + 0x030, 0x000063A2, + 0x030, 0x000073A2, + 0x030, 0x000083A2, + 0x030, 0x000093A2, + 0x030, 0x0000A3A2, + 0x030, 0x0000B3A2, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x0EF, 0x00000040, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000764, + 0x030, 0x00001632, + 0x030, 0x00002421, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004777, + 0x030, 0x00005777, + 0x030, 0x00006777, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000777, + 0x030, 0x00001442, + 0x030, 0x00002222, + 0x030, 0x00004777, + 0x030, 0x00005777, + 0x030, 0x00006777, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004777, + 0x030, 0x00005777, + 0x030, 0x00006777, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000764, + 0x030, 0x00001452, + 0x030, 0x00002220, + 0x030, 0x00004777, + 0x030, 0x00005777, + 0x030, 0x00006777, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000660, + 0x030, 0x00001341, + 0x030, 0x00002220, + 0x030, 0x00004777, + 0x030, 0x00005777, + 0x030, 0x00006777, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000764, + 0x030, 0x00001632, + 0x030, 0x00002421, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000764, + 0x030, 0x00001632, + 0x030, 0x00002421, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000767, + 0x030, 0x00001442, + 0x030, 0x00002222, + 0x030, 0x00004777, + 0x030, 0x00005777, + 0x030, 0x00006777, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000765, + 0x030, 0x00001632, + 0x030, 0x00002451, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000764, + 0x030, 0x00001632, + 0x030, 0x00002421, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000777, + 0x030, 0x00001442, + 0x030, 0x00002222, + 0x030, 0x00004777, + 0x030, 0x00005777, + 0x030, 0x00006777, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000776, + 0x030, 0x00001442, + 0x030, 0x00002222, + 0x030, 0x00004777, + 0x030, 0x00005777, + 0x030, 0x00006777, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000777, + 0x030, 0x00001442, + 0x030, 0x00002222, + 0x030, 0x00004777, + 0x030, 0x00005777, + 0x030, 0x00006777, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004777, + 0x030, 0x00005777, + 0x030, 0x00006777, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000775, + 0x030, 0x00001422, + 0x030, 0x00002210, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000775, + 0x030, 0x00001222, + 0x030, 0x00002210, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000645, + 0x030, 0x00001333, + 0x030, 0x00002011, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0xA0000000, 0x00000000, + 0x030, 0x00000764, + 0x030, 0x00001632, + 0x030, 0x00002421, + 0x030, 0x00004000, + 0x030, 0x00005000, + 0x030, 0x00006000, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x0EF, 0x00000800, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000C09, + 0x033, 0x00000021, + 0x03F, 0x00000C0C, + 0x033, 0x00000022, + 0x03F, 0x00000C0F, + 0x033, 0x00000023, + 0x03F, 0x00000C2C, + 0x033, 0x00000024, + 0x03F, 0x00000C2F, + 0x033, 0x00000025, + 0x03F, 0x00000C8A, + 0x033, 0x00000026, + 0x03F, 0x00000C8D, + 0x033, 0x00000027, + 0x03F, 0x00000C90, + 0x033, 0x00000028, + 0x03F, 0x00000CD0, + 0x033, 0x00000029, + 0x03F, 0x00000CF2, + 0x033, 0x0000002A, + 0x03F, 0x00000CF5, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000007, + 0x033, 0x00000021, + 0x03F, 0x0000000A, + 0x033, 0x00000022, + 0x03F, 0x0000000D, + 0x033, 0x00000023, + 0x03F, 0x0000002A, + 0x033, 0x00000024, + 0x03F, 0x0000002D, + 0x033, 0x00000025, + 0x03F, 0x00000030, + 0x033, 0x00000026, + 0x03F, 0x0000006D, + 0x033, 0x00000027, + 0x03F, 0x00000070, + 0x033, 0x00000028, + 0x03F, 0x000000ED, + 0x033, 0x00000029, + 0x03F, 0x000000F0, + 0x033, 0x0000002A, + 0x03F, 0x000000F3, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000005, + 0x033, 0x00000021, + 0x03F, 0x00000008, + 0x033, 0x00000022, + 0x03F, 0x0000000B, + 0x033, 0x00000023, + 0x03F, 0x0000000E, + 0x033, 0x00000024, + 0x03F, 0x0000002B, + 0x033, 0x00000025, + 0x03F, 0x0000002E, + 0x033, 0x00000026, + 0x03F, 0x0000006B, + 0x033, 0x00000027, + 0x03F, 0x0000006E, + 0x033, 0x00000028, + 0x03F, 0x00000071, + 0x033, 0x00000029, + 0x03F, 0x00000074, + 0x033, 0x0000002A, + 0x03F, 0x00000077, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000429, + 0x033, 0x00000021, + 0x03F, 0x00000828, + 0x033, 0x00000022, + 0x03F, 0x00000847, + 0x033, 0x00000023, + 0x03F, 0x0000084A, + 0x033, 0x00000024, + 0x03F, 0x00000C4B, + 0x033, 0x00000025, + 0x03F, 0x00000C6C, + 0x033, 0x00000026, + 0x03F, 0x00000C8D, + 0x033, 0x00000027, + 0x03F, 0x00000CAF, + 0x033, 0x00000028, + 0x03F, 0x00000CD1, + 0x033, 0x00000029, + 0x03F, 0x00000CF3, + 0x033, 0x0000002A, + 0x03F, 0x00000CF6, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000005, + 0x033, 0x00000021, + 0x03F, 0x00000008, + 0x033, 0x00000022, + 0x03F, 0x0000000B, + 0x033, 0x00000023, + 0x03F, 0x0000000E, + 0x033, 0x00000024, + 0x03F, 0x0000002B, + 0x033, 0x00000025, + 0x03F, 0x0000002E, + 0x033, 0x00000026, + 0x03F, 0x0000006B, + 0x033, 0x00000027, + 0x03F, 0x0000006E, + 0x033, 0x00000028, + 0x03F, 0x00000071, + 0x033, 0x00000029, + 0x03F, 0x00000074, + 0x033, 0x0000002A, + 0x03F, 0x00000077, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x0000042C, + 0x033, 0x00000021, + 0x03F, 0x0000082B, + 0x033, 0x00000022, + 0x03F, 0x0000084A, + 0x033, 0x00000023, + 0x03F, 0x0000084D, + 0x033, 0x00000024, + 0x03F, 0x00000C4E, + 0x033, 0x00000025, + 0x03F, 0x00000C6E, + 0x033, 0x00000026, + 0x03F, 0x00000CAD, + 0x033, 0x00000027, + 0x03F, 0x00000CED, + 0x033, 0x00000028, + 0x03F, 0x00000CF0, + 0x033, 0x00000029, + 0x03F, 0x00000CF3, + 0x033, 0x0000002A, + 0x03F, 0x00000CF6, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000007, + 0x033, 0x00000021, + 0x03F, 0x0000000A, + 0x033, 0x00000022, + 0x03F, 0x0000000D, + 0x033, 0x00000023, + 0x03F, 0x0000002A, + 0x033, 0x00000024, + 0x03F, 0x0000002D, + 0x033, 0x00000025, + 0x03F, 0x00000030, + 0x033, 0x00000026, + 0x03F, 0x0000006D, + 0x033, 0x00000027, + 0x03F, 0x00000070, + 0x033, 0x00000028, + 0x03F, 0x000000ED, + 0x033, 0x00000029, + 0x03F, 0x000000F0, + 0x033, 0x0000002A, + 0x03F, 0x000000F3, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000007, + 0x033, 0x00000021, + 0x03F, 0x0000000A, + 0x033, 0x00000022, + 0x03F, 0x0000000D, + 0x033, 0x00000023, + 0x03F, 0x0000002A, + 0x033, 0x00000024, + 0x03F, 0x0000002D, + 0x033, 0x00000025, + 0x03F, 0x00000030, + 0x033, 0x00000026, + 0x03F, 0x0000006D, + 0x033, 0x00000027, + 0x03F, 0x00000070, + 0x033, 0x00000028, + 0x03F, 0x000000ED, + 0x033, 0x00000029, + 0x03F, 0x000000F0, + 0x033, 0x0000002A, + 0x03F, 0x000000F3, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000828, + 0x033, 0x00000021, + 0x03F, 0x0000082B, + 0x033, 0x00000022, + 0x03F, 0x00000868, + 0x033, 0x00000023, + 0x03F, 0x00000889, + 0x033, 0x00000024, + 0x03F, 0x000008AA, + 0x033, 0x00000025, + 0x03F, 0x00000CE8, + 0x033, 0x00000026, + 0x03F, 0x00000CEB, + 0x033, 0x00000027, + 0x03F, 0x00000CEE, + 0x033, 0x00000028, + 0x03F, 0x00000CF1, + 0x033, 0x00000029, + 0x03F, 0x00000CF4, + 0x033, 0x0000002A, + 0x03F, 0x00000CF7, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000C09, + 0x033, 0x00000021, + 0x03F, 0x00000C0C, + 0x033, 0x00000022, + 0x03F, 0x00000C0F, + 0x033, 0x00000023, + 0x03F, 0x00000C2C, + 0x033, 0x00000024, + 0x03F, 0x00000C2F, + 0x033, 0x00000025, + 0x03F, 0x00000C8A, + 0x033, 0x00000026, + 0x03F, 0x00000C8D, + 0x033, 0x00000027, + 0x03F, 0x00000C90, + 0x033, 0x00000028, + 0x03F, 0x00000CD0, + 0x033, 0x00000029, + 0x03F, 0x00000CF2, + 0x033, 0x0000002A, + 0x03F, 0x00000CF5, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000C09, + 0x033, 0x00000021, + 0x03F, 0x00000C0C, + 0x033, 0x00000022, + 0x03F, 0x00000C0F, + 0x033, 0x00000023, + 0x03F, 0x00000C2C, + 0x033, 0x00000024, + 0x03F, 0x00000C2F, + 0x033, 0x00000025, + 0x03F, 0x00000C8A, + 0x033, 0x00000026, + 0x03F, 0x00000C8D, + 0x033, 0x00000027, + 0x03F, 0x00000C90, + 0x033, 0x00000028, + 0x03F, 0x00000CD0, + 0x033, 0x00000029, + 0x03F, 0x00000CF2, + 0x033, 0x0000002A, + 0x03F, 0x00000CF5, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000007, + 0x033, 0x00000021, + 0x03F, 0x0000000A, + 0x033, 0x00000022, + 0x03F, 0x0000000D, + 0x033, 0x00000023, + 0x03F, 0x0000002A, + 0x033, 0x00000024, + 0x03F, 0x0000002D, + 0x033, 0x00000025, + 0x03F, 0x00000030, + 0x033, 0x00000026, + 0x03F, 0x0000006D, + 0x033, 0x00000027, + 0x03F, 0x00000070, + 0x033, 0x00000028, + 0x03F, 0x000000ED, + 0x033, 0x00000029, + 0x03F, 0x000000F0, + 0x033, 0x0000002A, + 0x03F, 0x000000F3, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000429, + 0x033, 0x00000021, + 0x03F, 0x00000828, + 0x033, 0x00000022, + 0x03F, 0x00000847, + 0x033, 0x00000023, + 0x03F, 0x0000084A, + 0x033, 0x00000024, + 0x03F, 0x00000C4B, + 0x033, 0x00000025, + 0x03F, 0x00000CE5, + 0x033, 0x00000026, + 0x03F, 0x00000CE8, + 0x033, 0x00000027, + 0x03F, 0x00000CEB, + 0x033, 0x00000028, + 0x03F, 0x00000CEE, + 0x033, 0x00000029, + 0x03F, 0x00000CF1, + 0x033, 0x0000002A, + 0x03F, 0x00000CF4, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000C25, + 0x033, 0x00000021, + 0x03F, 0x00000C28, + 0x033, 0x00000022, + 0x03F, 0x00000C2B, + 0x033, 0x00000023, + 0x03F, 0x00000C68, + 0x033, 0x00000024, + 0x03F, 0x00000C6B, + 0x033, 0x00000025, + 0x03F, 0x00000C6E, + 0x033, 0x00000026, + 0x03F, 0x00000CEB, + 0x033, 0x00000027, + 0x03F, 0x00000CEE, + 0x033, 0x00000028, + 0x03F, 0x00000CF1, + 0x033, 0x00000029, + 0x03F, 0x00000CF4, + 0x033, 0x0000002A, + 0x03F, 0x00000CF7, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000C09, + 0x033, 0x00000021, + 0x03F, 0x00000C0C, + 0x033, 0x00000022, + 0x03F, 0x00000C0F, + 0x033, 0x00000023, + 0x03F, 0x00000C2C, + 0x033, 0x00000024, + 0x03F, 0x00000C2F, + 0x033, 0x00000025, + 0x03F, 0x00000C8A, + 0x033, 0x00000026, + 0x03F, 0x00000C8D, + 0x033, 0x00000027, + 0x03F, 0x00000C90, + 0x033, 0x00000028, + 0x03F, 0x00000CD0, + 0x033, 0x00000029, + 0x03F, 0x00000CF2, + 0x033, 0x0000002A, + 0x03F, 0x00000CF5, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000429, + 0x033, 0x00000021, + 0x03F, 0x00000828, + 0x033, 0x00000022, + 0x03F, 0x00000847, + 0x033, 0x00000023, + 0x03F, 0x0000084A, + 0x033, 0x00000024, + 0x03F, 0x00000C4B, + 0x033, 0x00000025, + 0x03F, 0x00000C8A, + 0x033, 0x00000026, + 0x03F, 0x00000CEA, + 0x033, 0x00000027, + 0x03F, 0x00000CED, + 0x033, 0x00000028, + 0x03F, 0x00000CF0, + 0x033, 0x00000029, + 0x03F, 0x00000CF3, + 0x033, 0x0000002A, + 0x03F, 0x00000CF6, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000429, + 0x033, 0x00000021, + 0x03F, 0x00000828, + 0x033, 0x00000022, + 0x03F, 0x00000847, + 0x033, 0x00000023, + 0x03F, 0x0000084A, + 0x033, 0x00000024, + 0x03F, 0x00000C4B, + 0x033, 0x00000025, + 0x03F, 0x00000C6C, + 0x033, 0x00000026, + 0x03F, 0x00000C8D, + 0x033, 0x00000027, + 0x03F, 0x00000CAF, + 0x033, 0x00000028, + 0x03F, 0x00000CD1, + 0x033, 0x00000029, + 0x03F, 0x00000CF3, + 0x033, 0x0000002A, + 0x03F, 0x00000CF6, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000429, + 0x033, 0x00000021, + 0x03F, 0x00000828, + 0x033, 0x00000022, + 0x03F, 0x00000847, + 0x033, 0x00000023, + 0x03F, 0x0000084A, + 0x033, 0x00000024, + 0x03F, 0x00000C4B, + 0x033, 0x00000025, + 0x03F, 0x00000C6C, + 0x033, 0x00000026, + 0x03F, 0x00000C8D, + 0x033, 0x00000027, + 0x03F, 0x00000CAF, + 0x033, 0x00000028, + 0x03F, 0x00000CD1, + 0x033, 0x00000029, + 0x03F, 0x00000CF3, + 0x033, 0x0000002A, + 0x03F, 0x00000CF6, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000005, + 0x033, 0x00000021, + 0x03F, 0x00000008, + 0x033, 0x00000022, + 0x03F, 0x0000000B, + 0x033, 0x00000023, + 0x03F, 0x0000000E, + 0x033, 0x00000024, + 0x03F, 0x0000002B, + 0x033, 0x00000025, + 0x03F, 0x0000002E, + 0x033, 0x00000026, + 0x03F, 0x0000006B, + 0x033, 0x00000027, + 0x03F, 0x0000006E, + 0x033, 0x00000028, + 0x03F, 0x00000071, + 0x033, 0x00000029, + 0x03F, 0x00000074, + 0x033, 0x0000002A, + 0x03F, 0x00000077, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000007, + 0x033, 0x00000021, + 0x03F, 0x0000000A, + 0x033, 0x00000022, + 0x03F, 0x0000000D, + 0x033, 0x00000023, + 0x03F, 0x0000002A, + 0x033, 0x00000024, + 0x03F, 0x0000002D, + 0x033, 0x00000025, + 0x03F, 0x00000030, + 0x033, 0x00000026, + 0x03F, 0x0000006D, + 0x033, 0x00000027, + 0x03F, 0x00000070, + 0x033, 0x00000028, + 0x03F, 0x000000ED, + 0x033, 0x00000029, + 0x03F, 0x000000F0, + 0x033, 0x0000002A, + 0x03F, 0x000000F3, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000005, + 0x033, 0x00000021, + 0x03F, 0x00000008, + 0x033, 0x00000022, + 0x03F, 0x0000000B, + 0x033, 0x00000023, + 0x03F, 0x0000000E, + 0x033, 0x00000024, + 0x03F, 0x0000002B, + 0x033, 0x00000025, + 0x03F, 0x00000068, + 0x033, 0x00000026, + 0x03F, 0x0000006B, + 0x033, 0x00000027, + 0x03F, 0x0000006E, + 0x033, 0x00000028, + 0x03F, 0x00000071, + 0x033, 0x00000029, + 0x03F, 0x00000074, + 0x033, 0x0000002A, + 0x03F, 0x00000077, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x0000042B, + 0x033, 0x00000021, + 0x03F, 0x0000082A, + 0x033, 0x00000022, + 0x03F, 0x00000849, + 0x033, 0x00000023, + 0x03F, 0x0000084C, + 0x033, 0x00000024, + 0x03F, 0x00000C4C, + 0x033, 0x00000025, + 0x03F, 0x00000C8A, + 0x033, 0x00000026, + 0x03F, 0x00000C8D, + 0x033, 0x00000027, + 0x03F, 0x00000CEB, + 0x033, 0x00000028, + 0x03F, 0x00000CEE, + 0x033, 0x00000029, + 0x03F, 0x00000CF1, + 0x033, 0x0000002A, + 0x03F, 0x00000CF4, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000005, + 0x033, 0x00000021, + 0x03F, 0x00000008, + 0x033, 0x00000022, + 0x03F, 0x0000000B, + 0x033, 0x00000023, + 0x03F, 0x0000000E, + 0x033, 0x00000024, + 0x03F, 0x0000002B, + 0x033, 0x00000025, + 0x03F, 0x00000068, + 0x033, 0x00000026, + 0x03F, 0x0000006B, + 0x033, 0x00000027, + 0x03F, 0x0000006E, + 0x033, 0x00000028, + 0x03F, 0x00000071, + 0x033, 0x00000029, + 0x03F, 0x00000074, + 0x033, 0x0000002A, + 0x03F, 0x00000077, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x0000042B, + 0x033, 0x00000021, + 0x03F, 0x0000082A, + 0x033, 0x00000022, + 0x03F, 0x00000849, + 0x033, 0x00000023, + 0x03F, 0x0000084C, + 0x033, 0x00000024, + 0x03F, 0x00000C4C, + 0x033, 0x00000025, + 0x03F, 0x00000C8A, + 0x033, 0x00000026, + 0x03F, 0x00000C8D, + 0x033, 0x00000027, + 0x03F, 0x00000CEB, + 0x033, 0x00000028, + 0x03F, 0x00000CEE, + 0x033, 0x00000029, + 0x03F, 0x00000CF1, + 0x033, 0x0000002A, + 0x03F, 0x00000CF4, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000007, + 0x033, 0x00000021, + 0x03F, 0x0000000A, + 0x033, 0x00000022, + 0x03F, 0x0000000D, + 0x033, 0x00000023, + 0x03F, 0x0000002A, + 0x033, 0x00000024, + 0x03F, 0x0000002D, + 0x033, 0x00000025, + 0x03F, 0x00000030, + 0x033, 0x00000026, + 0x03F, 0x0000006D, + 0x033, 0x00000027, + 0x03F, 0x00000070, + 0x033, 0x00000028, + 0x03F, 0x000000ED, + 0x033, 0x00000029, + 0x03F, 0x000000F0, + 0x033, 0x0000002A, + 0x03F, 0x000000F3, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000007, + 0x033, 0x00000021, + 0x03F, 0x0000000A, + 0x033, 0x00000022, + 0x03F, 0x0000000D, + 0x033, 0x00000023, + 0x03F, 0x0000002A, + 0x033, 0x00000024, + 0x03F, 0x0000002D, + 0x033, 0x00000025, + 0x03F, 0x00000030, + 0x033, 0x00000026, + 0x03F, 0x0000006D, + 0x033, 0x00000027, + 0x03F, 0x00000070, + 0x033, 0x00000028, + 0x03F, 0x000000ED, + 0x033, 0x00000029, + 0x03F, 0x000000F0, + 0x033, 0x0000002A, + 0x03F, 0x000000F3, + 0xA0000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000C09, + 0x033, 0x00000021, + 0x03F, 0x00000C0C, + 0x033, 0x00000022, + 0x03F, 0x00000C0F, + 0x033, 0x00000023, + 0x03F, 0x00000C2C, + 0x033, 0x00000024, + 0x03F, 0x00000C2F, + 0x033, 0x00000025, + 0x03F, 0x00000C8A, + 0x033, 0x00000026, + 0x03F, 0x00000C8D, + 0x033, 0x00000027, + 0x03F, 0x00000C90, + 0x033, 0x00000028, + 0x03F, 0x00000CD0, + 0x033, 0x00000029, + 0x03F, 0x00000CF2, + 0x033, 0x0000002A, + 0x03F, 0x00000CF5, + 0xB0000000, 0x00000000, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000C0A, + 0x033, 0x00000061, + 0x03F, 0x00000C0D, + 0x033, 0x00000062, + 0x03F, 0x00000C2A, + 0x033, 0x00000063, + 0x03F, 0x00000C2D, + 0x033, 0x00000064, + 0x03F, 0x00000C6A, + 0x033, 0x00000065, + 0x03F, 0x00000CAA, + 0x033, 0x00000066, + 0x03F, 0x00000CAD, + 0x033, 0x00000067, + 0x03F, 0x00000CB0, + 0x033, 0x00000068, + 0x03F, 0x00000CF1, + 0x033, 0x00000069, + 0x03F, 0x00000CF4, + 0x033, 0x0000006A, + 0x03F, 0x00000CF7, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000007, + 0x033, 0x00000061, + 0x03F, 0x0000000A, + 0x033, 0x00000062, + 0x03F, 0x0000000D, + 0x033, 0x00000063, + 0x03F, 0x0000002A, + 0x033, 0x00000064, + 0x03F, 0x0000002D, + 0x033, 0x00000065, + 0x03F, 0x00000030, + 0x033, 0x00000066, + 0x03F, 0x0000006D, + 0x033, 0x00000067, + 0x03F, 0x00000070, + 0x033, 0x00000068, + 0x03F, 0x000000ED, + 0x033, 0x00000069, + 0x03F, 0x000000F0, + 0x033, 0x0000006A, + 0x03F, 0x000000F3, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000005, + 0x033, 0x00000061, + 0x03F, 0x00000008, + 0x033, 0x00000062, + 0x03F, 0x0000000B, + 0x033, 0x00000063, + 0x03F, 0x0000000E, + 0x033, 0x00000064, + 0x03F, 0x0000002B, + 0x033, 0x00000065, + 0x03F, 0x0000002E, + 0x033, 0x00000066, + 0x03F, 0x0000006B, + 0x033, 0x00000067, + 0x03F, 0x0000006E, + 0x033, 0x00000068, + 0x03F, 0x00000071, + 0x033, 0x00000069, + 0x03F, 0x00000074, + 0x033, 0x0000006A, + 0x03F, 0x00000077, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000429, + 0x033, 0x00000061, + 0x03F, 0x00000828, + 0x033, 0x00000062, + 0x03F, 0x00000847, + 0x033, 0x00000063, + 0x03F, 0x0000084A, + 0x033, 0x00000064, + 0x03F, 0x00000C4B, + 0x033, 0x00000065, + 0x03F, 0x00000C6C, + 0x033, 0x00000066, + 0x03F, 0x00000C8D, + 0x033, 0x00000067, + 0x03F, 0x00000CAF, + 0x033, 0x00000068, + 0x03F, 0x00000CD1, + 0x033, 0x00000069, + 0x03F, 0x00000CF3, + 0x033, 0x0000006A, + 0x03F, 0x00000CF6, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000005, + 0x033, 0x00000061, + 0x03F, 0x00000008, + 0x033, 0x00000062, + 0x03F, 0x0000000B, + 0x033, 0x00000063, + 0x03F, 0x0000000E, + 0x033, 0x00000064, + 0x03F, 0x0000002B, + 0x033, 0x00000065, + 0x03F, 0x0000002E, + 0x033, 0x00000066, + 0x03F, 0x0000006B, + 0x033, 0x00000067, + 0x03F, 0x0000006E, + 0x033, 0x00000068, + 0x03F, 0x00000071, + 0x033, 0x00000069, + 0x03F, 0x00000074, + 0x033, 0x0000006A, + 0x03F, 0x00000077, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x0000042A, + 0x033, 0x00000061, + 0x03F, 0x00000829, + 0x033, 0x00000062, + 0x03F, 0x00000848, + 0x033, 0x00000063, + 0x03F, 0x0000084B, + 0x033, 0x00000064, + 0x03F, 0x00000C4B, + 0x033, 0x00000065, + 0x03F, 0x00000C6C, + 0x033, 0x00000066, + 0x03F, 0x00000CAC, + 0x033, 0x00000067, + 0x03F, 0x00000CED, + 0x033, 0x00000068, + 0x03F, 0x00000CF0, + 0x033, 0x00000069, + 0x03F, 0x00000CF3, + 0x033, 0x0000006A, + 0x03F, 0x00000CF6, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000007, + 0x033, 0x00000061, + 0x03F, 0x0000000A, + 0x033, 0x00000062, + 0x03F, 0x0000000D, + 0x033, 0x00000063, + 0x03F, 0x0000002A, + 0x033, 0x00000064, + 0x03F, 0x0000002D, + 0x033, 0x00000065, + 0x03F, 0x00000030, + 0x033, 0x00000066, + 0x03F, 0x0000006D, + 0x033, 0x00000067, + 0x03F, 0x00000070, + 0x033, 0x00000068, + 0x03F, 0x000000ED, + 0x033, 0x00000069, + 0x03F, 0x000000F0, + 0x033, 0x0000006A, + 0x03F, 0x000000F3, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000007, + 0x033, 0x00000061, + 0x03F, 0x0000000A, + 0x033, 0x00000062, + 0x03F, 0x0000000D, + 0x033, 0x00000063, + 0x03F, 0x0000002A, + 0x033, 0x00000064, + 0x03F, 0x0000002D, + 0x033, 0x00000065, + 0x03F, 0x00000030, + 0x033, 0x00000066, + 0x03F, 0x0000006D, + 0x033, 0x00000067, + 0x03F, 0x00000070, + 0x033, 0x00000068, + 0x03F, 0x000000ED, + 0x033, 0x00000069, + 0x03F, 0x000000F0, + 0x033, 0x0000006A, + 0x03F, 0x000000F3, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000842, + 0x033, 0x00000061, + 0x03F, 0x00000845, + 0x033, 0x00000062, + 0x03F, 0x00000866, + 0x033, 0x00000063, + 0x03F, 0x000008A6, + 0x033, 0x00000064, + 0x03F, 0x000008C8, + 0x033, 0x00000065, + 0x03F, 0x00000CE8, + 0x033, 0x00000066, + 0x03F, 0x00000CEB, + 0x033, 0x00000067, + 0x03F, 0x00000CEE, + 0x033, 0x00000068, + 0x03F, 0x00000CF1, + 0x033, 0x00000069, + 0x03F, 0x00000CF4, + 0x033, 0x0000006A, + 0x03F, 0x00000CF7, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000C0A, + 0x033, 0x00000061, + 0x03F, 0x00000C0D, + 0x033, 0x00000062, + 0x03F, 0x00000C2A, + 0x033, 0x00000063, + 0x03F, 0x00000C2D, + 0x033, 0x00000064, + 0x03F, 0x00000C6A, + 0x033, 0x00000065, + 0x03F, 0x00000CAA, + 0x033, 0x00000066, + 0x03F, 0x00000CAD, + 0x033, 0x00000067, + 0x03F, 0x00000CB0, + 0x033, 0x00000068, + 0x03F, 0x00000CF1, + 0x033, 0x00000069, + 0x03F, 0x00000CF4, + 0x033, 0x0000006A, + 0x03F, 0x00000CF7, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000C0A, + 0x033, 0x00000061, + 0x03F, 0x00000C0D, + 0x033, 0x00000062, + 0x03F, 0x00000C2A, + 0x033, 0x00000063, + 0x03F, 0x00000C2D, + 0x033, 0x00000064, + 0x03F, 0x00000C6A, + 0x033, 0x00000065, + 0x03F, 0x00000CAA, + 0x033, 0x00000066, + 0x03F, 0x00000CAD, + 0x033, 0x00000067, + 0x03F, 0x00000CB0, + 0x033, 0x00000068, + 0x03F, 0x00000CF1, + 0x033, 0x00000069, + 0x03F, 0x00000CF4, + 0x033, 0x0000006A, + 0x03F, 0x00000CF7, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000005, + 0x033, 0x00000061, + 0x03F, 0x00000008, + 0x033, 0x00000062, + 0x03F, 0x0000000B, + 0x033, 0x00000063, + 0x03F, 0x0000000E, + 0x033, 0x00000064, + 0x03F, 0x0000002B, + 0x033, 0x00000065, + 0x03F, 0x00000068, + 0x033, 0x00000066, + 0x03F, 0x0000006B, + 0x033, 0x00000067, + 0x03F, 0x0000006E, + 0x033, 0x00000068, + 0x03F, 0x00000071, + 0x033, 0x00000069, + 0x03F, 0x00000074, + 0x033, 0x0000006A, + 0x03F, 0x00000077, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000429, + 0x033, 0x00000061, + 0x03F, 0x00000828, + 0x033, 0x00000062, + 0x03F, 0x00000847, + 0x033, 0x00000063, + 0x03F, 0x0000084A, + 0x033, 0x00000064, + 0x03F, 0x00000C4B, + 0x033, 0x00000065, + 0x03F, 0x00000CE5, + 0x033, 0x00000066, + 0x03F, 0x00000CE8, + 0x033, 0x00000067, + 0x03F, 0x00000CEB, + 0x033, 0x00000068, + 0x03F, 0x00000CEE, + 0x033, 0x00000069, + 0x03F, 0x00000CF1, + 0x033, 0x0000006A, + 0x03F, 0x00000CF4, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000C0A, + 0x033, 0x00000061, + 0x03F, 0x00000C0D, + 0x033, 0x00000062, + 0x03F, 0x00000C10, + 0x033, 0x00000063, + 0x03F, 0x00000C4A, + 0x033, 0x00000064, + 0x03F, 0x00000C4D, + 0x033, 0x00000065, + 0x03F, 0x00000CC9, + 0x033, 0x00000066, + 0x03F, 0x00000CEB, + 0x033, 0x00000067, + 0x03F, 0x00000CEE, + 0x033, 0x00000068, + 0x03F, 0x00000CF1, + 0x033, 0x00000069, + 0x03F, 0x00000CF4, + 0x033, 0x0000006A, + 0x03F, 0x00000CF7, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000C0A, + 0x033, 0x00000061, + 0x03F, 0x00000C0D, + 0x033, 0x00000062, + 0x03F, 0x00000C2A, + 0x033, 0x00000063, + 0x03F, 0x00000C2D, + 0x033, 0x00000064, + 0x03F, 0x00000C6A, + 0x033, 0x00000065, + 0x03F, 0x00000CAA, + 0x033, 0x00000066, + 0x03F, 0x00000CAD, + 0x033, 0x00000067, + 0x03F, 0x00000CB0, + 0x033, 0x00000068, + 0x03F, 0x00000CF1, + 0x033, 0x00000069, + 0x03F, 0x00000CF4, + 0x033, 0x0000006A, + 0x03F, 0x00000CF7, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000429, + 0x033, 0x00000061, + 0x03F, 0x00000828, + 0x033, 0x00000062, + 0x03F, 0x00000847, + 0x033, 0x00000063, + 0x03F, 0x0000084A, + 0x033, 0x00000064, + 0x03F, 0x00000C4B, + 0x033, 0x00000065, + 0x03F, 0x00000C8A, + 0x033, 0x00000066, + 0x03F, 0x00000CEA, + 0x033, 0x00000067, + 0x03F, 0x00000CED, + 0x033, 0x00000068, + 0x03F, 0x00000CF0, + 0x033, 0x00000069, + 0x03F, 0x00000CF3, + 0x033, 0x0000006A, + 0x03F, 0x00000CF6, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000429, + 0x033, 0x00000061, + 0x03F, 0x00000828, + 0x033, 0x00000062, + 0x03F, 0x00000847, + 0x033, 0x00000063, + 0x03F, 0x0000084A, + 0x033, 0x00000064, + 0x03F, 0x00000C4B, + 0x033, 0x00000065, + 0x03F, 0x00000C6C, + 0x033, 0x00000066, + 0x03F, 0x00000C8D, + 0x033, 0x00000067, + 0x03F, 0x00000CAF, + 0x033, 0x00000068, + 0x03F, 0x00000CD1, + 0x033, 0x00000069, + 0x03F, 0x00000CF3, + 0x033, 0x0000006A, + 0x03F, 0x00000CF6, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000429, + 0x033, 0x00000061, + 0x03F, 0x00000828, + 0x033, 0x00000062, + 0x03F, 0x00000847, + 0x033, 0x00000063, + 0x03F, 0x0000084A, + 0x033, 0x00000064, + 0x03F, 0x00000C4B, + 0x033, 0x00000065, + 0x03F, 0x00000C6C, + 0x033, 0x00000066, + 0x03F, 0x00000C8D, + 0x033, 0x00000067, + 0x03F, 0x00000CAF, + 0x033, 0x00000068, + 0x03F, 0x00000CD1, + 0x033, 0x00000069, + 0x03F, 0x00000CF3, + 0x033, 0x0000006A, + 0x03F, 0x00000CF6, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000005, + 0x033, 0x00000061, + 0x03F, 0x00000008, + 0x033, 0x00000062, + 0x03F, 0x0000000B, + 0x033, 0x00000063, + 0x03F, 0x0000000E, + 0x033, 0x00000064, + 0x03F, 0x0000002B, + 0x033, 0x00000065, + 0x03F, 0x0000002E, + 0x033, 0x00000066, + 0x03F, 0x0000006B, + 0x033, 0x00000067, + 0x03F, 0x0000006E, + 0x033, 0x00000068, + 0x03F, 0x00000071, + 0x033, 0x00000069, + 0x03F, 0x00000074, + 0x033, 0x0000006A, + 0x03F, 0x00000077, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000007, + 0x033, 0x00000061, + 0x03F, 0x0000000A, + 0x033, 0x00000062, + 0x03F, 0x0000000D, + 0x033, 0x00000063, + 0x03F, 0x0000002A, + 0x033, 0x00000064, + 0x03F, 0x0000002D, + 0x033, 0x00000065, + 0x03F, 0x00000030, + 0x033, 0x00000066, + 0x03F, 0x0000006D, + 0x033, 0x00000067, + 0x03F, 0x00000070, + 0x033, 0x00000068, + 0x03F, 0x000000ED, + 0x033, 0x00000069, + 0x03F, 0x000000F0, + 0x033, 0x0000006A, + 0x03F, 0x000000F3, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000005, + 0x033, 0x00000061, + 0x03F, 0x00000008, + 0x033, 0x00000062, + 0x03F, 0x0000000B, + 0x033, 0x00000063, + 0x03F, 0x0000000E, + 0x033, 0x00000064, + 0x03F, 0x0000002B, + 0x033, 0x00000065, + 0x03F, 0x00000068, + 0x033, 0x00000066, + 0x03F, 0x0000006B, + 0x033, 0x00000067, + 0x03F, 0x0000006E, + 0x033, 0x00000068, + 0x03F, 0x00000071, + 0x033, 0x00000069, + 0x03F, 0x00000074, + 0x033, 0x0000006A, + 0x03F, 0x00000077, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x0000042C, + 0x033, 0x00000061, + 0x03F, 0x0000082B, + 0x033, 0x00000062, + 0x03F, 0x0000084A, + 0x033, 0x00000063, + 0x03F, 0x0000084D, + 0x033, 0x00000064, + 0x03F, 0x00000C4E, + 0x033, 0x00000065, + 0x03F, 0x00000C8C, + 0x033, 0x00000066, + 0x03F, 0x00000C8F, + 0x033, 0x00000067, + 0x03F, 0x00000CEC, + 0x033, 0x00000068, + 0x03F, 0x00000CEF, + 0x033, 0x00000069, + 0x03F, 0x00000CF2, + 0x033, 0x0000006A, + 0x03F, 0x00000CF5, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000005, + 0x033, 0x00000061, + 0x03F, 0x00000008, + 0x033, 0x00000062, + 0x03F, 0x0000000B, + 0x033, 0x00000063, + 0x03F, 0x0000000E, + 0x033, 0x00000064, + 0x03F, 0x0000002B, + 0x033, 0x00000065, + 0x03F, 0x00000068, + 0x033, 0x00000066, + 0x03F, 0x0000006B, + 0x033, 0x00000067, + 0x03F, 0x0000006E, + 0x033, 0x00000068, + 0x03F, 0x00000071, + 0x033, 0x00000069, + 0x03F, 0x00000074, + 0x033, 0x0000006A, + 0x03F, 0x00000077, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x0000042C, + 0x033, 0x00000061, + 0x03F, 0x0000082B, + 0x033, 0x00000062, + 0x03F, 0x0000084A, + 0x033, 0x00000063, + 0x03F, 0x0000084D, + 0x033, 0x00000064, + 0x03F, 0x00000C4E, + 0x033, 0x00000065, + 0x03F, 0x00000C8C, + 0x033, 0x00000066, + 0x03F, 0x00000C8F, + 0x033, 0x00000067, + 0x03F, 0x00000CEC, + 0x033, 0x00000068, + 0x03F, 0x00000CEF, + 0x033, 0x00000069, + 0x03F, 0x00000CF2, + 0x033, 0x0000006A, + 0x03F, 0x00000CF5, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000007, + 0x033, 0x00000061, + 0x03F, 0x0000000A, + 0x033, 0x00000062, + 0x03F, 0x0000000D, + 0x033, 0x00000063, + 0x03F, 0x0000002A, + 0x033, 0x00000064, + 0x03F, 0x0000002D, + 0x033, 0x00000065, + 0x03F, 0x00000030, + 0x033, 0x00000066, + 0x03F, 0x0000006D, + 0x033, 0x00000067, + 0x03F, 0x00000070, + 0x033, 0x00000068, + 0x03F, 0x000000ED, + 0x033, 0x00000069, + 0x03F, 0x000000F0, + 0x033, 0x0000006A, + 0x03F, 0x000000F3, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000007, + 0x033, 0x00000061, + 0x03F, 0x0000000A, + 0x033, 0x00000062, + 0x03F, 0x0000000D, + 0x033, 0x00000063, + 0x03F, 0x0000002A, + 0x033, 0x00000064, + 0x03F, 0x0000002D, + 0x033, 0x00000065, + 0x03F, 0x00000030, + 0x033, 0x00000066, + 0x03F, 0x0000006D, + 0x033, 0x00000067, + 0x03F, 0x00000070, + 0x033, 0x00000068, + 0x03F, 0x000000ED, + 0x033, 0x00000069, + 0x03F, 0x000000F0, + 0x033, 0x0000006A, + 0x03F, 0x000000F3, + 0xA0000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000C0A, + 0x033, 0x00000061, + 0x03F, 0x00000C0D, + 0x033, 0x00000062, + 0x03F, 0x00000C2A, + 0x033, 0x00000063, + 0x03F, 0x00000C2D, + 0x033, 0x00000064, + 0x03F, 0x00000C6A, + 0x033, 0x00000065, + 0x03F, 0x00000CAA, + 0x033, 0x00000066, + 0x03F, 0x00000CAD, + 0x033, 0x00000067, + 0x03F, 0x00000CB0, + 0x033, 0x00000068, + 0x03F, 0x00000CF1, + 0x033, 0x00000069, + 0x03F, 0x00000CF4, + 0x033, 0x0000006A, + 0x03F, 0x00000CF7, + 0xB0000000, 0x00000000, + 0x83000000, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000C09, + 0x033, 0x000000A1, + 0x03F, 0x00000C0C, + 0x033, 0x000000A2, + 0x03F, 0x00000C0F, + 0x033, 0x000000A3, + 0x03F, 0x00000C2C, + 0x033, 0x000000A4, + 0x03F, 0x00000C2F, + 0x033, 0x000000A5, + 0x03F, 0x00000C8A, + 0x033, 0x000000A6, + 0x03F, 0x00000C8D, + 0x033, 0x000000A7, + 0x03F, 0x00000C90, + 0x033, 0x000000A8, + 0x03F, 0x00000CEF, + 0x033, 0x000000A9, + 0x03F, 0x00000CF2, + 0x033, 0x000000AA, + 0x03F, 0x00000CF5, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000007, + 0x033, 0x000000A1, + 0x03F, 0x0000000A, + 0x033, 0x000000A2, + 0x03F, 0x0000000D, + 0x033, 0x000000A3, + 0x03F, 0x0000002A, + 0x033, 0x000000A4, + 0x03F, 0x0000002D, + 0x033, 0x000000A5, + 0x03F, 0x00000030, + 0x033, 0x000000A6, + 0x03F, 0x0000006D, + 0x033, 0x000000A7, + 0x03F, 0x00000070, + 0x033, 0x000000A8, + 0x03F, 0x000000ED, + 0x033, 0x000000A9, + 0x03F, 0x000000F0, + 0x033, 0x000000AA, + 0x03F, 0x000000F3, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000005, + 0x033, 0x000000A1, + 0x03F, 0x00000008, + 0x033, 0x000000A2, + 0x03F, 0x0000000B, + 0x033, 0x000000A3, + 0x03F, 0x0000000E, + 0x033, 0x000000A4, + 0x03F, 0x0000002B, + 0x033, 0x000000A5, + 0x03F, 0x0000002E, + 0x033, 0x000000A6, + 0x03F, 0x00000031, + 0x033, 0x000000A7, + 0x03F, 0x00000034, + 0x033, 0x000000A8, + 0x03F, 0x00000053, + 0x033, 0x000000A9, + 0x03F, 0x00000056, + 0x033, 0x000000AA, + 0x03F, 0x000000D1, + 0x93000003, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000429, + 0x033, 0x000000A1, + 0x03F, 0x00000828, + 0x033, 0x000000A2, + 0x03F, 0x00000847, + 0x033, 0x000000A3, + 0x03F, 0x0000084A, + 0x033, 0x000000A4, + 0x03F, 0x00000C4B, + 0x033, 0x000000A5, + 0x03F, 0x00000C6C, + 0x033, 0x000000A6, + 0x03F, 0x00000C8D, + 0x033, 0x000000A7, + 0x03F, 0x00000CAF, + 0x033, 0x000000A8, + 0x03F, 0x00000CD1, + 0x033, 0x000000A9, + 0x03F, 0x00000CF3, + 0x033, 0x000000AA, + 0x03F, 0x00000CF6, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000005, + 0x033, 0x000000A1, + 0x03F, 0x00000008, + 0x033, 0x000000A2, + 0x03F, 0x0000000B, + 0x033, 0x000000A3, + 0x03F, 0x0000000E, + 0x033, 0x000000A4, + 0x03F, 0x0000002B, + 0x033, 0x000000A5, + 0x03F, 0x0000002E, + 0x033, 0x000000A6, + 0x03F, 0x00000031, + 0x033, 0x000000A7, + 0x03F, 0x00000034, + 0x033, 0x000000A8, + 0x03F, 0x00000053, + 0x033, 0x000000A9, + 0x03F, 0x00000056, + 0x033, 0x000000AA, + 0x03F, 0x000000D1, + 0x93000005, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x0000042A, + 0x033, 0x000000A1, + 0x03F, 0x00000829, + 0x033, 0x000000A2, + 0x03F, 0x00000848, + 0x033, 0x000000A3, + 0x03F, 0x0000084B, + 0x033, 0x000000A4, + 0x03F, 0x00000C4C, + 0x033, 0x000000A5, + 0x03F, 0x00000C6C, + 0x033, 0x000000A6, + 0x03F, 0x00000CAC, + 0x033, 0x000000A7, + 0x03F, 0x00000CED, + 0x033, 0x000000A8, + 0x03F, 0x00000CF0, + 0x033, 0x000000A9, + 0x03F, 0x00000CF3, + 0x033, 0x000000AA, + 0x03F, 0x00000CF6, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000007, + 0x033, 0x000000A1, + 0x03F, 0x0000000A, + 0x033, 0x000000A2, + 0x03F, 0x0000000D, + 0x033, 0x000000A3, + 0x03F, 0x0000002A, + 0x033, 0x000000A4, + 0x03F, 0x0000002D, + 0x033, 0x000000A5, + 0x03F, 0x00000030, + 0x033, 0x000000A6, + 0x03F, 0x0000006D, + 0x033, 0x000000A7, + 0x03F, 0x00000070, + 0x033, 0x000000A8, + 0x03F, 0x000000ED, + 0x033, 0x000000A9, + 0x03F, 0x000000F0, + 0x033, 0x000000AA, + 0x03F, 0x000000F3, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000007, + 0x033, 0x000000A1, + 0x03F, 0x0000000A, + 0x033, 0x000000A2, + 0x03F, 0x0000000D, + 0x033, 0x000000A3, + 0x03F, 0x0000002A, + 0x033, 0x000000A4, + 0x03F, 0x0000002D, + 0x033, 0x000000A5, + 0x03F, 0x00000030, + 0x033, 0x000000A6, + 0x03F, 0x0000006D, + 0x033, 0x000000A7, + 0x03F, 0x00000070, + 0x033, 0x000000A8, + 0x03F, 0x000000ED, + 0x033, 0x000000A9, + 0x03F, 0x000000F0, + 0x033, 0x000000AA, + 0x03F, 0x000000F3, + 0x93000008, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000826, + 0x033, 0x000000A1, + 0x03F, 0x00000829, + 0x033, 0x000000A2, + 0x03F, 0x0000082C, + 0x033, 0x000000A3, + 0x03F, 0x0000082F, + 0x033, 0x000000A4, + 0x03F, 0x0000086C, + 0x033, 0x000000A5, + 0x03F, 0x00000CE8, + 0x033, 0x000000A6, + 0x03F, 0x00000CEB, + 0x033, 0x000000A7, + 0x03F, 0x00000CEE, + 0x033, 0x000000A8, + 0x03F, 0x00000CF1, + 0x033, 0x000000A9, + 0x03F, 0x00000CF4, + 0x033, 0x000000AA, + 0x03F, 0x00000CF7, + 0x93000009, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000C09, + 0x033, 0x000000A1, + 0x03F, 0x00000C0C, + 0x033, 0x000000A2, + 0x03F, 0x00000C0F, + 0x033, 0x000000A3, + 0x03F, 0x00000C2C, + 0x033, 0x000000A4, + 0x03F, 0x00000C2F, + 0x033, 0x000000A5, + 0x03F, 0x00000C8A, + 0x033, 0x000000A6, + 0x03F, 0x00000C8D, + 0x033, 0x000000A7, + 0x03F, 0x00000C90, + 0x033, 0x000000A8, + 0x03F, 0x00000CEF, + 0x033, 0x000000A9, + 0x03F, 0x00000CF2, + 0x033, 0x000000AA, + 0x03F, 0x00000CF5, + 0x9300000a, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000C09, + 0x033, 0x000000A1, + 0x03F, 0x00000C0C, + 0x033, 0x000000A2, + 0x03F, 0x00000C0F, + 0x033, 0x000000A3, + 0x03F, 0x00000C2C, + 0x033, 0x000000A4, + 0x03F, 0x00000C2F, + 0x033, 0x000000A5, + 0x03F, 0x00000C8A, + 0x033, 0x000000A6, + 0x03F, 0x00000C8D, + 0x033, 0x000000A7, + 0x03F, 0x00000C90, + 0x033, 0x000000A8, + 0x03F, 0x00000CEF, + 0x033, 0x000000A9, + 0x03F, 0x00000CF2, + 0x033, 0x000000AA, + 0x03F, 0x00000CF5, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000007, + 0x033, 0x000000A1, + 0x03F, 0x0000000A, + 0x033, 0x000000A2, + 0x03F, 0x0000000D, + 0x033, 0x000000A3, + 0x03F, 0x0000002A, + 0x033, 0x000000A4, + 0x03F, 0x0000002D, + 0x033, 0x000000A5, + 0x03F, 0x00000030, + 0x033, 0x000000A6, + 0x03F, 0x0000006D, + 0x033, 0x000000A7, + 0x03F, 0x00000070, + 0x033, 0x000000A8, + 0x03F, 0x000000ED, + 0x033, 0x000000A9, + 0x03F, 0x000000F0, + 0x033, 0x000000AA, + 0x03F, 0x000000F3, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000429, + 0x033, 0x000000A1, + 0x03F, 0x00000828, + 0x033, 0x000000A2, + 0x03F, 0x00000847, + 0x033, 0x000000A3, + 0x03F, 0x0000084A, + 0x033, 0x000000A4, + 0x03F, 0x00000C4B, + 0x033, 0x000000A5, + 0x03F, 0x00000CE5, + 0x033, 0x000000A6, + 0x03F, 0x00000CE8, + 0x033, 0x000000A7, + 0x03F, 0x00000CEB, + 0x033, 0x000000A8, + 0x03F, 0x00000CEE, + 0x033, 0x000000A9, + 0x03F, 0x00000CF1, + 0x033, 0x000000AA, + 0x03F, 0x00000CF4, + 0x9300000d, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x0000080A, + 0x033, 0x000000A1, + 0x03F, 0x0000080D, + 0x033, 0x000000A2, + 0x03F, 0x00000810, + 0x033, 0x000000A3, + 0x03F, 0x00000868, + 0x033, 0x000000A4, + 0x03F, 0x00000C68, + 0x033, 0x000000A5, + 0x03F, 0x00000C6B, + 0x033, 0x000000A6, + 0x03F, 0x00000CAB, + 0x033, 0x000000A7, + 0x03F, 0x00000CAE, + 0x033, 0x000000A8, + 0x03F, 0x00000CEF, + 0x033, 0x000000A9, + 0x03F, 0x00000CF2, + 0x033, 0x000000AA, + 0x03F, 0x00000CF5, + 0x9300000e, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000C08, + 0x033, 0x000000A1, + 0x03F, 0x00000C0B, + 0x033, 0x000000A2, + 0x03F, 0x00000C0E, + 0x033, 0x000000A3, + 0x03F, 0x00000C2B, + 0x033, 0x000000A4, + 0x03F, 0x00000C2E, + 0x033, 0x000000A5, + 0x03F, 0x00000C31, + 0x033, 0x000000A6, + 0x03F, 0x00000CAB, + 0x033, 0x000000A7, + 0x03F, 0x00000CAE, + 0x033, 0x000000A8, + 0x03F, 0x00000CEF, + 0x033, 0x000000A9, + 0x03F, 0x00000CF2, + 0x033, 0x000000AA, + 0x03F, 0x00000CF5, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000429, + 0x033, 0x000000A1, + 0x03F, 0x00000828, + 0x033, 0x000000A2, + 0x03F, 0x00000847, + 0x033, 0x000000A3, + 0x03F, 0x0000084A, + 0x033, 0x000000A4, + 0x03F, 0x00000C4B, + 0x033, 0x000000A5, + 0x03F, 0x00000C8A, + 0x033, 0x000000A6, + 0x03F, 0x00000CEA, + 0x033, 0x000000A7, + 0x03F, 0x00000CED, + 0x033, 0x000000A8, + 0x03F, 0x00000CF0, + 0x033, 0x000000A9, + 0x03F, 0x00000CF3, + 0x033, 0x000000AA, + 0x03F, 0x00000CF6, + 0x93000010, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000429, + 0x033, 0x000000A1, + 0x03F, 0x00000828, + 0x033, 0x000000A2, + 0x03F, 0x00000847, + 0x033, 0x000000A3, + 0x03F, 0x0000084A, + 0x033, 0x000000A4, + 0x03F, 0x00000C4B, + 0x033, 0x000000A5, + 0x03F, 0x00000C6C, + 0x033, 0x000000A6, + 0x03F, 0x00000C8D, + 0x033, 0x000000A7, + 0x03F, 0x00000CAF, + 0x033, 0x000000A8, + 0x03F, 0x00000CD1, + 0x033, 0x000000A9, + 0x03F, 0x00000CF3, + 0x033, 0x000000AA, + 0x03F, 0x00000CF6, + 0x93000011, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000429, + 0x033, 0x000000A1, + 0x03F, 0x00000828, + 0x033, 0x000000A2, + 0x03F, 0x00000847, + 0x033, 0x000000A3, + 0x03F, 0x0000084A, + 0x033, 0x000000A4, + 0x03F, 0x00000C4B, + 0x033, 0x000000A5, + 0x03F, 0x00000C6C, + 0x033, 0x000000A6, + 0x03F, 0x00000C8D, + 0x033, 0x000000A7, + 0x03F, 0x00000CAF, + 0x033, 0x000000A8, + 0x03F, 0x00000CD1, + 0x033, 0x000000A9, + 0x03F, 0x00000CF3, + 0x033, 0x000000AA, + 0x03F, 0x00000CF6, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000005, + 0x033, 0x000000A1, + 0x03F, 0x00000008, + 0x033, 0x000000A2, + 0x03F, 0x0000000B, + 0x033, 0x000000A3, + 0x03F, 0x0000000E, + 0x033, 0x000000A4, + 0x03F, 0x0000002B, + 0x033, 0x000000A5, + 0x03F, 0x0000002E, + 0x033, 0x000000A6, + 0x03F, 0x00000031, + 0x033, 0x000000A7, + 0x03F, 0x00000034, + 0x033, 0x000000A8, + 0x03F, 0x00000053, + 0x033, 0x000000A9, + 0x03F, 0x00000056, + 0x033, 0x000000AA, + 0x03F, 0x000000D1, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000007, + 0x033, 0x000000A1, + 0x03F, 0x0000000A, + 0x033, 0x000000A2, + 0x03F, 0x0000000D, + 0x033, 0x000000A3, + 0x03F, 0x0000002A, + 0x033, 0x000000A4, + 0x03F, 0x0000002D, + 0x033, 0x000000A5, + 0x03F, 0x00000030, + 0x033, 0x000000A6, + 0x03F, 0x0000006D, + 0x033, 0x000000A7, + 0x03F, 0x00000070, + 0x033, 0x000000A8, + 0x03F, 0x000000ED, + 0x033, 0x000000A9, + 0x03F, 0x000000F0, + 0x033, 0x000000AA, + 0x03F, 0x000000F3, + 0x90000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000005, + 0x033, 0x000000A1, + 0x03F, 0x00000008, + 0x033, 0x000000A2, + 0x03F, 0x0000000B, + 0x033, 0x000000A3, + 0x03F, 0x0000000E, + 0x033, 0x000000A4, + 0x03F, 0x00000047, + 0x033, 0x000000A5, + 0x03F, 0x0000004A, + 0x033, 0x000000A6, + 0x03F, 0x0000004D, + 0x033, 0x000000A7, + 0x03F, 0x00000050, + 0x033, 0x000000A8, + 0x03F, 0x00000053, + 0x033, 0x000000A9, + 0x03F, 0x00000056, + 0x033, 0x000000AA, + 0x03F, 0x00000094, + 0x90000003, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x0000042A, + 0x033, 0x000000A1, + 0x03F, 0x00000829, + 0x033, 0x000000A2, + 0x03F, 0x00000848, + 0x033, 0x000000A3, + 0x03F, 0x0000084B, + 0x033, 0x000000A4, + 0x03F, 0x00000C4C, + 0x033, 0x000000A5, + 0x03F, 0x00000C8A, + 0x033, 0x000000A6, + 0x03F, 0x00000C8D, + 0x033, 0x000000A7, + 0x03F, 0x00000CEC, + 0x033, 0x000000A8, + 0x03F, 0x00000CEF, + 0x033, 0x000000A9, + 0x03F, 0x00000CF2, + 0x033, 0x000000AA, + 0x03F, 0x00000CF5, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000005, + 0x033, 0x000000A1, + 0x03F, 0x00000008, + 0x033, 0x000000A2, + 0x03F, 0x0000000B, + 0x033, 0x000000A3, + 0x03F, 0x0000000E, + 0x033, 0x000000A4, + 0x03F, 0x00000047, + 0x033, 0x000000A5, + 0x03F, 0x0000004A, + 0x033, 0x000000A6, + 0x03F, 0x0000004D, + 0x033, 0x000000A7, + 0x03F, 0x00000050, + 0x033, 0x000000A8, + 0x03F, 0x00000053, + 0x033, 0x000000A9, + 0x03F, 0x00000056, + 0x033, 0x000000AA, + 0x03F, 0x00000094, + 0x90000005, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x0000042A, + 0x033, 0x000000A1, + 0x03F, 0x00000829, + 0x033, 0x000000A2, + 0x03F, 0x00000848, + 0x033, 0x000000A3, + 0x03F, 0x0000084B, + 0x033, 0x000000A4, + 0x03F, 0x00000C4C, + 0x033, 0x000000A5, + 0x03F, 0x00000C8A, + 0x033, 0x000000A6, + 0x03F, 0x00000C8D, + 0x033, 0x000000A7, + 0x03F, 0x00000CEC, + 0x033, 0x000000A8, + 0x03F, 0x00000CEF, + 0x033, 0x000000A9, + 0x03F, 0x00000CF2, + 0x033, 0x000000AA, + 0x03F, 0x00000CF5, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000007, + 0x033, 0x000000A1, + 0x03F, 0x0000000A, + 0x033, 0x000000A2, + 0x03F, 0x0000000D, + 0x033, 0x000000A3, + 0x03F, 0x0000002A, + 0x033, 0x000000A4, + 0x03F, 0x0000002D, + 0x033, 0x000000A5, + 0x03F, 0x00000030, + 0x033, 0x000000A6, + 0x03F, 0x0000006D, + 0x033, 0x000000A7, + 0x03F, 0x00000070, + 0x033, 0x000000A8, + 0x03F, 0x000000ED, + 0x033, 0x000000A9, + 0x03F, 0x000000F0, + 0x033, 0x000000AA, + 0x03F, 0x000000F3, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000007, + 0x033, 0x000000A1, + 0x03F, 0x0000000A, + 0x033, 0x000000A2, + 0x03F, 0x0000000D, + 0x033, 0x000000A3, + 0x03F, 0x0000002A, + 0x033, 0x000000A4, + 0x03F, 0x0000002D, + 0x033, 0x000000A5, + 0x03F, 0x00000030, + 0x033, 0x000000A6, + 0x03F, 0x0000006D, + 0x033, 0x000000A7, + 0x03F, 0x00000070, + 0x033, 0x000000A8, + 0x03F, 0x000000ED, + 0x033, 0x000000A9, + 0x03F, 0x000000F0, + 0x033, 0x000000AA, + 0x03F, 0x000000F3, + 0xA0000000, 0x00000000, + 0x033, 0x000000A0, + 0x03F, 0x00000C09, + 0x033, 0x000000A1, + 0x03F, 0x00000C0C, + 0x033, 0x000000A2, + 0x03F, 0x00000C0F, + 0x033, 0x000000A3, + 0x03F, 0x00000C2C, + 0x033, 0x000000A4, + 0x03F, 0x00000C2F, + 0x033, 0x000000A5, + 0x03F, 0x00000C8A, + 0x033, 0x000000A6, + 0x03F, 0x00000C8D, + 0x033, 0x000000A7, + 0x03F, 0x00000C90, + 0x033, 0x000000A8, + 0x03F, 0x00000CEF, + 0x033, 0x000000A9, + 0x03F, 0x00000CF2, + 0x033, 0x000000AA, + 0x03F, 0x00000CF5, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x0EF, 0x00000400, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0000265A, + 0x033, 0x00000001, + 0x03F, 0x0000265A, + 0x033, 0x00000002, + 0x03F, 0x0000265A, + 0x033, 0x00000003, + 0x03F, 0x0000265A, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0000265A, + 0x033, 0x00000001, + 0x03F, 0x0000265A, + 0x033, 0x00000002, + 0x03F, 0x0000265A, + 0x033, 0x00000003, + 0x03F, 0x0000265A, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0000265A, + 0x033, 0x00000001, + 0x03F, 0x0000265A, + 0x033, 0x00000002, + 0x03F, 0x0000265A, + 0x033, 0x00000003, + 0x03F, 0x0000265A, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0000265A, + 0x033, 0x00000001, + 0x03F, 0x0000265A, + 0x033, 0x00000002, + 0x03F, 0x0000265A, + 0x033, 0x00000003, + 0x03F, 0x0000265A, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0000265A, + 0x033, 0x00000001, + 0x03F, 0x0000265A, + 0x033, 0x00000002, + 0x03F, 0x0000265A, + 0x033, 0x00000003, + 0x03F, 0x0000265A, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x000004FB, + 0x033, 0x00000001, + 0x03F, 0x000004FB, + 0x033, 0x00000002, + 0x03F, 0x000004FB, + 0x033, 0x00000003, + 0x03F, 0x000004FB, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0000265A, + 0x033, 0x00000001, + 0x03F, 0x0000265A, + 0x033, 0x00000002, + 0x03F, 0x0000265A, + 0x033, 0x00000003, + 0x03F, 0x0000265A, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0000265A, + 0x033, 0x00000001, + 0x03F, 0x0000265A, + 0x033, 0x00000002, + 0x03F, 0x0000265A, + 0x033, 0x00000003, + 0x03F, 0x0000265A, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0000265A, + 0x033, 0x00000001, + 0x03F, 0x0000265A, + 0x033, 0x00000002, + 0x03F, 0x0000265A, + 0x033, 0x00000003, + 0x03F, 0x0000265A, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0000265A, + 0x033, 0x00000001, + 0x03F, 0x0000265A, + 0x033, 0x00000002, + 0x03F, 0x0000265A, + 0x033, 0x00000003, + 0x03F, 0x0000265A, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0000265A, + 0x033, 0x00000001, + 0x03F, 0x0000265A, + 0x033, 0x00000002, + 0x03F, 0x0000265A, + 0x033, 0x00000003, + 0x03F, 0x0000265A, + 0xA0000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x000004BB, + 0x033, 0x00000001, + 0x03F, 0x000004BB, + 0x033, 0x00000002, + 0x03F, 0x000004BB, + 0x033, 0x00000003, + 0x03F, 0x000004BB, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x0EF, 0x00000100, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000745, + 0x033, 0x00000001, + 0x03F, 0x00000745, + 0x033, 0x00000002, + 0x03F, 0x00000745, + 0x033, 0x00000003, + 0x03F, 0x00000745, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000745, + 0x033, 0x00000001, + 0x03F, 0x00000745, + 0x033, 0x00000002, + 0x03F, 0x00000745, + 0x033, 0x00000003, + 0x03F, 0x00000745, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000745, + 0x033, 0x00000001, + 0x03F, 0x00000745, + 0x033, 0x00000002, + 0x03F, 0x00000745, + 0x033, 0x00000003, + 0x03F, 0x00000745, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000745, + 0x033, 0x00000001, + 0x03F, 0x00000745, + 0x033, 0x00000002, + 0x03F, 0x00000745, + 0x033, 0x00000003, + 0x03F, 0x00000745, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000745, + 0x033, 0x00000001, + 0x03F, 0x00000745, + 0x033, 0x00000002, + 0x03F, 0x00000745, + 0x033, 0x00000003, + 0x03F, 0x00000745, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000745, + 0x033, 0x00000001, + 0x03F, 0x00000745, + 0x033, 0x00000002, + 0x03F, 0x00000745, + 0x033, 0x00000003, + 0x03F, 0x00000745, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000745, + 0x033, 0x00000001, + 0x03F, 0x00000745, + 0x033, 0x00000002, + 0x03F, 0x00000745, + 0x033, 0x00000003, + 0x03F, 0x00000745, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000745, + 0x033, 0x00000001, + 0x03F, 0x00000745, + 0x033, 0x00000002, + 0x03F, 0x00000745, + 0x033, 0x00000003, + 0x03F, 0x00000745, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000745, + 0x033, 0x00000001, + 0x03F, 0x00000745, + 0x033, 0x00000002, + 0x03F, 0x00000745, + 0x033, 0x00000003, + 0x03F, 0x00000745, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000745, + 0x033, 0x00000001, + 0x03F, 0x00000745, + 0x033, 0x00000002, + 0x03F, 0x00000745, + 0x033, 0x00000003, + 0x03F, 0x00000745, + 0xA0000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000F34, + 0x033, 0x00000001, + 0x03F, 0x00000F34, + 0x033, 0x00000002, + 0x03F, 0x00000F34, + 0x033, 0x00000003, + 0x03F, 0x00000F34, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x081, 0x0000F400, + 0x087, 0x00016040, + 0x051, 0x00000808, + 0x052, 0x00098002, + 0x053, 0x0000FA47, + 0x054, 0x00058032, + 0x056, 0x00051000, + 0x057, 0x0000CE0A, + 0x058, 0x00082030, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x081, 0x0000F400, + 0x087, 0x00016040, + 0x051, 0x00000808, + 0x052, 0x00098002, + 0x053, 0x0000FA47, + 0x054, 0x00058032, + 0x056, 0x00051000, + 0x057, 0x0000CE0A, + 0x058, 0x00082030, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x081, 0x0000F400, + 0x087, 0x00016040, + 0x051, 0x00000808, + 0x052, 0x00098002, + 0x053, 0x0000FA47, + 0x054, 0x00058032, + 0x056, 0x00051000, + 0x057, 0x0000CE0A, + 0x058, 0x00082030, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x081, 0x0000F400, + 0x087, 0x00016040, + 0x051, 0x00000808, + 0x052, 0x00098002, + 0x053, 0x0000FA47, + 0x054, 0x00058032, + 0x056, 0x00051000, + 0x057, 0x0000CE0A, + 0x058, 0x00082030, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x081, 0x0000F400, + 0x087, 0x00016040, + 0x051, 0x00000808, + 0x052, 0x00098002, + 0x053, 0x0000FA47, + 0x054, 0x00058032, + 0x056, 0x00051000, + 0x057, 0x0000CE0A, + 0x058, 0x00082030, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x081, 0x0000F400, + 0x087, 0x00016040, + 0x051, 0x00000808, + 0x052, 0x00098002, + 0x053, 0x0000FA47, + 0x054, 0x00058032, + 0x056, 0x00051000, + 0x057, 0x0000CE0A, + 0x058, 0x00082030, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x081, 0x0000F400, + 0x087, 0x00016040, + 0x051, 0x00000808, + 0x052, 0x00098002, + 0x053, 0x0000FA47, + 0x054, 0x00058032, + 0x056, 0x00051000, + 0x057, 0x0000CE0A, + 0x058, 0x00082030, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x081, 0x0000F400, + 0x087, 0x00016040, + 0x051, 0x00000808, + 0x052, 0x00098002, + 0x053, 0x0000FA47, + 0x054, 0x00058032, + 0x056, 0x00051000, + 0x057, 0x0000CE0A, + 0x058, 0x00082030, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x081, 0x0000F400, + 0x087, 0x00016040, + 0x051, 0x00000808, + 0x052, 0x00098002, + 0x053, 0x0000FA47, + 0x054, 0x00058032, + 0x056, 0x00051000, + 0x057, 0x0000CE0A, + 0x058, 0x00082030, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x081, 0x0000F400, + 0x087, 0x00016040, + 0x051, 0x00000808, + 0x052, 0x00098002, + 0x053, 0x0000FA47, + 0x054, 0x00058032, + 0x056, 0x00051000, + 0x057, 0x0000CE0A, + 0x058, 0x00082030, + 0xA0000000, 0x00000000, + 0x081, 0x0000F000, + 0x087, 0x00016040, + 0x051, 0x00000C00, + 0x052, 0x0007C241, + 0x053, 0x0001C069, + 0x054, 0x00078032, + 0x057, 0x0000CE0A, + 0x058, 0x00058750, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000800, + 0x83000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000003, + 0x033, 0x00000001, + 0x03F, 0x00000006, + 0x033, 0x00000002, + 0x03F, 0x00000009, + 0x033, 0x00000003, + 0x03F, 0x00000026, + 0x033, 0x00000004, + 0x03F, 0x00000029, + 0x033, 0x00000005, + 0x03F, 0x0000002C, + 0x033, 0x00000006, + 0x03F, 0x0000002F, + 0x033, 0x00000007, + 0x03F, 0x00000033, + 0x033, 0x00000008, + 0x03F, 0x00000036, + 0x033, 0x00000009, + 0x03F, 0x00000039, + 0x033, 0x0000000A, + 0x03F, 0x0000003C, + 0x93000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000002, + 0x033, 0x00000001, + 0x03F, 0x00000005, + 0x033, 0x00000002, + 0x03F, 0x00000008, + 0x033, 0x00000003, + 0x03F, 0x0000000B, + 0x033, 0x00000004, + 0x03F, 0x0000000E, + 0x033, 0x00000005, + 0x03F, 0x0000002B, + 0x033, 0x00000006, + 0x03F, 0x0000002E, + 0x033, 0x00000007, + 0x03F, 0x00000031, + 0x033, 0x00000008, + 0x03F, 0x0000006E, + 0x033, 0x00000009, + 0x03F, 0x00000071, + 0x033, 0x0000000A, + 0x03F, 0x00000074, + 0x93000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000003, + 0x033, 0x00000001, + 0x03F, 0x00000006, + 0x033, 0x00000002, + 0x03F, 0x00000009, + 0x033, 0x00000003, + 0x03F, 0x00000026, + 0x033, 0x00000004, + 0x03F, 0x00000029, + 0x033, 0x00000005, + 0x03F, 0x0000002C, + 0x033, 0x00000006, + 0x03F, 0x0000002F, + 0x033, 0x00000007, + 0x03F, 0x00000033, + 0x033, 0x00000008, + 0x03F, 0x00000036, + 0x033, 0x00000009, + 0x03F, 0x00000039, + 0x033, 0x0000000A, + 0x03F, 0x0000003C, + 0x93000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000003, + 0x033, 0x00000001, + 0x03F, 0x00000006, + 0x033, 0x00000002, + 0x03F, 0x00000009, + 0x033, 0x00000003, + 0x03F, 0x00000026, + 0x033, 0x00000004, + 0x03F, 0x00000029, + 0x033, 0x00000005, + 0x03F, 0x0000002C, + 0x033, 0x00000006, + 0x03F, 0x0000002F, + 0x033, 0x00000007, + 0x03F, 0x00000033, + 0x033, 0x00000008, + 0x03F, 0x00000036, + 0x033, 0x00000009, + 0x03F, 0x00000039, + 0x033, 0x0000000A, + 0x03F, 0x0000003C, + 0x9300000b, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000003, + 0x033, 0x00000001, + 0x03F, 0x00000006, + 0x033, 0x00000002, + 0x03F, 0x00000009, + 0x033, 0x00000003, + 0x03F, 0x00000026, + 0x033, 0x00000004, + 0x03F, 0x00000029, + 0x033, 0x00000005, + 0x03F, 0x0000002C, + 0x033, 0x00000006, + 0x03F, 0x0000002F, + 0x033, 0x00000007, + 0x03F, 0x00000033, + 0x033, 0x00000008, + 0x03F, 0x00000036, + 0x033, 0x00000009, + 0x03F, 0x00000039, + 0x033, 0x0000000A, + 0x03F, 0x0000003C, + 0x9300000c, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0005142C, + 0x033, 0x00000001, + 0x03F, 0x0005142F, + 0x033, 0x00000002, + 0x03F, 0x00051432, + 0x033, 0x00000003, + 0x03F, 0x00051CA5, + 0x033, 0x00000004, + 0x03F, 0x00051CA8, + 0x033, 0x00000005, + 0x03F, 0x00051CAB, + 0x033, 0x00000006, + 0x03F, 0x00051CEB, + 0x033, 0x00000007, + 0x03F, 0x00051CEE, + 0x033, 0x00000008, + 0x03F, 0x00051CF1, + 0x033, 0x00000009, + 0x03F, 0x00051CF4, + 0x033, 0x0000000A, + 0x03F, 0x00051CF7, + 0x9300000f, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0005142C, + 0x033, 0x00000001, + 0x03F, 0x0005144B, + 0x033, 0x00000002, + 0x03F, 0x00051868, + 0x033, 0x00000003, + 0x03F, 0x0005186B, + 0x033, 0x00000004, + 0x03F, 0x0005186E, + 0x033, 0x00000005, + 0x03F, 0x00051871, + 0x033, 0x00000006, + 0x03F, 0x00051874, + 0x033, 0x00000007, + 0x03F, 0x00051895, + 0x033, 0x00000008, + 0x03F, 0x000518B6, + 0x033, 0x00000009, + 0x03F, 0x000518F6, + 0x033, 0x0000000A, + 0x03F, 0x00051CF7, + 0x93000012, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000002, + 0x033, 0x00000001, + 0x03F, 0x00000005, + 0x033, 0x00000002, + 0x03F, 0x00000008, + 0x033, 0x00000003, + 0x03F, 0x0000000B, + 0x033, 0x00000004, + 0x03F, 0x0000000E, + 0x033, 0x00000005, + 0x03F, 0x0000002B, + 0x033, 0x00000006, + 0x03F, 0x0000002E, + 0x033, 0x00000007, + 0x03F, 0x00000031, + 0x033, 0x00000008, + 0x03F, 0x0000006E, + 0x033, 0x00000009, + 0x03F, 0x00000071, + 0x033, 0x0000000A, + 0x03F, 0x00000074, + 0x90000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000003, + 0x033, 0x00000001, + 0x03F, 0x00000006, + 0x033, 0x00000002, + 0x03F, 0x00000009, + 0x033, 0x00000003, + 0x03F, 0x00000026, + 0x033, 0x00000004, + 0x03F, 0x00000029, + 0x033, 0x00000005, + 0x03F, 0x0000002C, + 0x033, 0x00000006, + 0x03F, 0x0000002F, + 0x033, 0x00000007, + 0x03F, 0x00000033, + 0x033, 0x00000008, + 0x03F, 0x00000036, + 0x033, 0x00000009, + 0x03F, 0x00000039, + 0x033, 0x0000000A, + 0x03F, 0x0000003C, + 0x90000004, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000003, + 0x033, 0x00000001, + 0x03F, 0x00000006, + 0x033, 0x00000002, + 0x03F, 0x00000009, + 0x033, 0x00000003, + 0x03F, 0x00000026, + 0x033, 0x00000004, + 0x03F, 0x00000029, + 0x033, 0x00000005, + 0x03F, 0x0000002C, + 0x033, 0x00000006, + 0x03F, 0x0000002F, + 0x033, 0x00000007, + 0x03F, 0x00000033, + 0x033, 0x00000008, + 0x03F, 0x00000036, + 0x033, 0x00000009, + 0x03F, 0x00000039, + 0x033, 0x0000000A, + 0x03F, 0x0000003C, + 0x90000006, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000003, + 0x033, 0x00000001, + 0x03F, 0x00000006, + 0x033, 0x00000002, + 0x03F, 0x00000009, + 0x033, 0x00000003, + 0x03F, 0x00000026, + 0x033, 0x00000004, + 0x03F, 0x00000029, + 0x033, 0x00000005, + 0x03F, 0x0000002C, + 0x033, 0x00000006, + 0x03F, 0x0000002F, + 0x033, 0x00000007, + 0x03F, 0x00000033, + 0x033, 0x00000008, + 0x03F, 0x00000036, + 0x033, 0x00000009, + 0x03F, 0x00000039, + 0x033, 0x0000000A, + 0x03F, 0x0000003C, + 0x90000007, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x00000003, + 0x033, 0x00000001, + 0x03F, 0x00000006, + 0x033, 0x00000002, + 0x03F, 0x00000009, + 0x033, 0x00000003, + 0x03F, 0x00000026, + 0x033, 0x00000004, + 0x03F, 0x00000029, + 0x033, 0x00000005, + 0x03F, 0x0000002C, + 0x033, 0x00000006, + 0x03F, 0x0000002F, + 0x033, 0x00000007, + 0x03F, 0x00000033, + 0x033, 0x00000008, + 0x03F, 0x00000036, + 0x033, 0x00000009, + 0x03F, 0x00000039, + 0x033, 0x0000000A, + 0x03F, 0x0000003C, + 0xA0000000, 0x00000000, + 0x033, 0x00000000, + 0x03F, 0x0005142C, + 0x033, 0x00000001, + 0x03F, 0x0005142F, + 0x033, 0x00000002, + 0x03F, 0x00051432, + 0x033, 0x00000003, + 0x03F, 0x00051C87, + 0x033, 0x00000004, + 0x03F, 0x00051C8A, + 0x033, 0x00000005, + 0x03F, 0x00051C8D, + 0x033, 0x00000006, + 0x03F, 0x00051CEB, + 0x033, 0x00000007, + 0x03F, 0x00051CEE, + 0x033, 0x00000008, + 0x03F, 0x00051CF1, + 0x033, 0x00000009, + 0x03F, 0x00051CF4, + 0x033, 0x0000000A, + 0x03F, 0x00051CF7, + 0xB0000000, 0x00000000, + 0x8300000c, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000000, + 0xA0000000, 0x00000000, + 0x0EF, 0x00000000, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000010, + 0x033, 0x00000000, + 0x008, 0x0009C060, + 0x033, 0x00000001, + 0x008, 0x0009C060, + 0x0EF, 0x00000000, + 0x033, 0x000000A2, + 0x0EF, 0x00080000, + 0x03E, 0x0000593F, + 0x8300000c, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x000D0F4F, + 0xA0000000, 0x00000000, + 0x03F, 0x000C0F4F, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x033, 0x000000A3, + 0x0EF, 0x00080000, + 0x03E, 0x00005934, + 0x03F, 0x0005AFCF, + 0x0EF, 0x00000000, +}; + +RTW_DECL_TABLE_RF_RADIO(rtw8822b_rf_b, B); + +static const u8 rtw8822b_txpwr_lmt_type2[] = { + 0, 0, 0, 0, 1, 32, 2, 0, 0, 0, 1, 28, 1, 0, 0, 0, 1, 30, + 0, 0, 0, 0, 2, 32, 2, 0, 0, 0, 2, 28, 1, 0, 0, 0, 2, 30, + 0, 0, 0, 0, 3, 32, 2, 0, 0, 0, 3, 28, 1, 0, 0, 0, 3, 30, + 0, 0, 0, 0, 4, 32, 2, 0, 0, 0, 4, 28, 1, 0, 0, 0, 4, 30, + 0, 0, 0, 0, 5, 32, 2, 0, 0, 0, 5, 28, 1, 0, 0, 0, 5, 30, + 0, 0, 0, 0, 6, 32, 2, 0, 0, 0, 6, 28, 1, 0, 0, 0, 6, 30, + 0, 0, 0, 0, 7, 32, 2, 0, 0, 0, 7, 28, 1, 0, 0, 0, 7, 30, + 0, 0, 0, 0, 8, 32, 2, 0, 0, 0, 8, 28, 1, 0, 0, 0, 8, 30, + 0, 0, 0, 0, 9, 32, 2, 0, 0, 0, 9, 28, 1, 0, 0, 0, 9, 30, + 0, 0, 0, 0, 10, 32, 2, 0, 0, 0, 10, 28, 1, 0, 0, 0, 10, 30, + 0, 0, 0, 0, 11, 32, 2, 0, 0, 0, 11, 28, 1, 0, 0, 0, 11, 30, + 0, 0, 0, 0, 12, 26, 2, 0, 0, 0, 12, 28, 1, 0, 0, 0, 12, 30, + 0, 0, 0, 0, 13, 20, 2, 0, 0, 0, 13, 28, 1, 0, 0, 0, 13, 28, + 0, 0, 0, 0, 14, 63, 2, 0, 0, 0, 14, 63, 1, 0, 0, 0, 14, 32, + 0, 0, 0, 1, 1, 26, 2, 0, 0, 1, 1, 30, 1, 0, 0, 1, 1, 34, + 0, 0, 0, 1, 2, 30, 2, 0, 0, 1, 2, 30, 1, 0, 0, 1, 2, 34, + 0, 0, 0, 1, 3, 32, 2, 0, 0, 1, 3, 30, 1, 0, 0, 1, 3, 34, + 0, 0, 0, 1, 4, 34, 2, 0, 0, 1, 4, 30, 1, 0, 0, 1, 4, 34, + 0, 0, 0, 1, 5, 34, 2, 0, 0, 1, 5, 30, 1, 0, 0, 1, 5, 34, + 0, 0, 0, 1, 6, 34, 2, 0, 0, 1, 6, 30, 1, 0, 0, 1, 6, 34, + 0, 0, 0, 1, 7, 34, 2, 0, 0, 1, 7, 30, 1, 0, 0, 1, 7, 34, + 0, 0, 0, 1, 8, 34, 2, 0, 0, 1, 8, 30, 1, 0, 0, 1, 8, 34, + 0, 0, 0, 1, 9, 32, 2, 0, 0, 1, 9, 30, 1, 0, 0, 1, 9, 34, + 0, 0, 0, 1, 10, 30, 2, 0, 0, 1, 10, 30, 1, 0, 0, 1, 10, 34, + 0, 0, 0, 1, 11, 28, 2, 0, 0, 1, 11, 30, 1, 0, 0, 1, 11, 34, + 0, 0, 0, 1, 12, 22, 2, 0, 0, 1, 12, 30, 1, 0, 0, 1, 12, 34, + 0, 0, 0, 1, 13, 14, 2, 0, 0, 1, 13, 30, 1, 0, 0, 1, 13, 34, + 0, 0, 0, 1, 14, 63, 2, 0, 0, 1, 14, 63, 1, 0, 0, 1, 14, 63, + 0, 0, 0, 2, 1, 26, 2, 0, 0, 2, 1, 30, 1, 0, 0, 2, 1, 34, + 0, 0, 0, 2, 2, 30, 2, 0, 0, 2, 2, 30, 1, 0, 0, 2, 2, 34, + 0, 0, 0, 2, 3, 32, 2, 0, 0, 2, 3, 30, 1, 0, 0, 2, 3, 34, + 0, 0, 0, 2, 4, 34, 2, 0, 0, 2, 4, 30, 1, 0, 0, 2, 4, 34, + 0, 0, 0, 2, 5, 34, 2, 0, 0, 2, 5, 30, 1, 0, 0, 2, 5, 34, + 0, 0, 0, 2, 6, 34, 2, 0, 0, 2, 6, 30, 1, 0, 0, 2, 6, 34, + 0, 0, 0, 2, 7, 34, 2, 0, 0, 2, 7, 30, 1, 0, 0, 2, 7, 34, + 0, 0, 0, 2, 8, 34, 2, 0, 0, 2, 8, 30, 1, 0, 0, 2, 8, 34, + 0, 0, 0, 2, 9, 32, 2, 0, 0, 2, 9, 30, 1, 0, 0, 2, 9, 34, + 0, 0, 0, 2, 10, 30, 2, 0, 0, 2, 10, 30, 1, 0, 0, 2, 10, 34, + 0, 0, 0, 2, 11, 26, 2, 0, 0, 2, 11, 30, 1, 0, 0, 2, 11, 34, + 0, 0, 0, 2, 12, 20, 2, 0, 0, 2, 12, 30, 1, 0, 0, 2, 12, 34, + 0, 0, 0, 2, 13, 14, 2, 0, 0, 2, 13, 30, 1, 0, 0, 2, 13, 34, + 0, 0, 0, 2, 14, 63, 2, 0, 0, 2, 14, 63, 1, 0, 0, 2, 14, 63, + 0, 0, 0, 3, 1, 26, 2, 0, 0, 3, 1, 18, 1, 0, 0, 3, 1, 30, + 0, 0, 0, 3, 2, 28, 2, 0, 0, 3, 2, 18, 1, 0, 0, 3, 2, 30, + 0, 0, 0, 3, 3, 30, 2, 0, 0, 3, 3, 18, 1, 0, 0, 3, 3, 30, + 0, 0, 0, 3, 4, 30, 2, 0, 0, 3, 4, 18, 1, 0, 0, 3, 4, 30, + 0, 0, 0, 3, 5, 32, 2, 0, 0, 3, 5, 18, 1, 0, 0, 3, 5, 30, + 0, 0, 0, 3, 6, 32, 2, 0, 0, 3, 6, 18, 1, 0, 0, 3, 6, 30, + 0, 0, 0, 3, 7, 32, 2, 0, 0, 3, 7, 18, 1, 0, 0, 3, 7, 30, + 0, 0, 0, 3, 8, 30, 2, 0, 0, 3, 8, 18, 1, 0, 0, 3, 8, 30, + 0, 0, 0, 3, 9, 30, 2, 0, 0, 3, 9, 18, 1, 0, 0, 3, 9, 30, + 0, 0, 0, 3, 10, 28, 2, 0, 0, 3, 10, 18, 1, 0, 0, 3, 10, 30, + 0, 0, 0, 3, 11, 26, 2, 0, 0, 3, 11, 18, 1, 0, 0, 3, 11, 30, + 0, 0, 0, 3, 12, 20, 2, 0, 0, 3, 12, 18, 1, 0, 0, 3, 12, 30, + 0, 0, 0, 3, 13, 14, 2, 0, 0, 3, 13, 18, 1, 0, 0, 3, 13, 30, + 0, 0, 0, 3, 14, 63, 2, 0, 0, 3, 14, 63, 1, 0, 0, 3, 14, 63, + 0, 0, 1, 2, 1, 63, 2, 0, 1, 2, 1, 63, 1, 0, 1, 2, 1, 63, + 0, 0, 1, 2, 2, 63, 2, 0, 1, 2, 2, 63, 1, 0, 1, 2, 2, 63, + 0, 0, 1, 2, 3, 26, 2, 0, 1, 2, 3, 30, 1, 0, 1, 2, 3, 34, + 0, 0, 1, 2, 4, 26, 2, 0, 1, 2, 4, 30, 1, 0, 1, 2, 4, 34, + 0, 0, 1, 2, 5, 30, 2, 0, 1, 2, 5, 30, 1, 0, 1, 2, 5, 34, + 0, 0, 1, 2, 6, 32, 2, 0, 1, 2, 6, 30, 1, 0, 1, 2, 6, 34, + 0, 0, 1, 2, 7, 30, 2, 0, 1, 2, 7, 30, 1, 0, 1, 2, 7, 34, + 0, 0, 1, 2, 8, 26, 2, 0, 1, 2, 8, 30, 1, 0, 1, 2, 8, 34, + 0, 0, 1, 2, 9, 26, 2, 0, 1, 2, 9, 30, 1, 0, 1, 2, 9, 34, + 0, 0, 1, 2, 10, 20, 2, 0, 1, 2, 10, 30, 1, 0, 1, 2, 10, 34, + 0, 0, 1, 2, 11, 14, 2, 0, 1, 2, 11, 30, 1, 0, 1, 2, 11, 34, + 0, 0, 1, 2, 12, 63, 2, 0, 1, 2, 12, 63, 1, 0, 1, 2, 12, 63, + 0, 0, 1, 2, 13, 63, 2, 0, 1, 2, 13, 63, 1, 0, 1, 2, 13, 63, + 0, 0, 1, 2, 14, 63, 2, 0, 1, 2, 14, 63, 1, 0, 1, 2, 14, 63, + 0, 0, 1, 3, 1, 63, 2, 0, 1, 3, 1, 63, 1, 0, 1, 3, 1, 63, + 0, 0, 1, 3, 2, 63, 2, 0, 1, 3, 2, 63, 1, 0, 1, 3, 2, 63, + 0, 0, 1, 3, 3, 24, 2, 0, 1, 3, 3, 18, 1, 0, 1, 3, 3, 30, + 0, 0, 1, 3, 4, 24, 2, 0, 1, 3, 4, 18, 1, 0, 1, 3, 4, 30, + 0, 0, 1, 3, 5, 26, 2, 0, 1, 3, 5, 18, 1, 0, 1, 3, 5, 30, + 0, 0, 1, 3, 6, 28, 2, 0, 1, 3, 6, 18, 1, 0, 1, 3, 6, 30, + 0, 0, 1, 3, 7, 26, 2, 0, 1, 3, 7, 18, 1, 0, 1, 3, 7, 30, + 0, 0, 1, 3, 8, 26, 2, 0, 1, 3, 8, 18, 1, 0, 1, 3, 8, 30, + 0, 0, 1, 3, 9, 26, 2, 0, 1, 3, 9, 18, 1, 0, 1, 3, 9, 30, + 0, 0, 1, 3, 10, 20, 2, 0, 1, 3, 10, 18, 1, 0, 1, 3, 10, 30, + 0, 0, 1, 3, 11, 14, 2, 0, 1, 3, 11, 18, 1, 0, 1, 3, 11, 30, + 0, 0, 1, 3, 12, 63, 2, 0, 1, 3, 12, 63, 1, 0, 1, 3, 12, 63, + 0, 0, 1, 3, 13, 63, 2, 0, 1, 3, 13, 63, 1, 0, 1, 3, 13, 63, + 0, 0, 1, 3, 14, 63, 2, 0, 1, 3, 14, 63, 1, 0, 1, 3, 14, 63, + 0, 1, 0, 1, 36, 36, 2, 1, 0, 1, 36, 32, 1, 1, 0, 1, 36, 30, + 0, 1, 0, 1, 40, 38, 2, 1, 0, 1, 40, 32, 1, 1, 0, 1, 40, 30, + 0, 1, 0, 1, 44, 38, 2, 1, 0, 1, 44, 32, 1, 1, 0, 1, 44, 30, + 0, 1, 0, 1, 48, 38, 2, 1, 0, 1, 48, 32, 1, 1, 0, 1, 48, 30, + 0, 1, 0, 1, 52, 38, 2, 1, 0, 1, 52, 32, 1, 1, 0, 1, 52, 28, + 0, 1, 0, 1, 56, 38, 2, 1, 0, 1, 56, 32, 1, 1, 0, 1, 56, 28, + 0, 1, 0, 1, 60, 38, 2, 1, 0, 1, 60, 32, 1, 1, 0, 1, 60, 28, + 0, 1, 0, 1, 64, 34, 2, 1, 0, 1, 64, 32, 1, 1, 0, 1, 64, 28, + 0, 1, 0, 1, 100, 32, 2, 1, 0, 1, 100, 32, 1, 1, 0, 1, 100, 32, + 0, 1, 0, 1, 104, 38, 2, 1, 0, 1, 104, 32, 1, 1, 0, 1, 104, 32, + 0, 1, 0, 1, 108, 38, 2, 1, 0, 1, 108, 32, 1, 1, 0, 1, 108, 32, + 0, 1, 0, 1, 112, 38, 2, 1, 0, 1, 112, 32, 1, 1, 0, 1, 112, 32, + 0, 1, 0, 1, 116, 38, 2, 1, 0, 1, 116, 32, 1, 1, 0, 1, 116, 32, + 0, 1, 0, 1, 120, 38, 2, 1, 0, 1, 120, 32, 1, 1, 0, 1, 120, 32, + 0, 1, 0, 1, 124, 38, 2, 1, 0, 1, 124, 32, 1, 1, 0, 1, 124, 32, + 0, 1, 0, 1, 128, 38, 2, 1, 0, 1, 128, 32, 1, 1, 0, 1, 128, 32, + 0, 1, 0, 1, 132, 38, 2, 1, 0, 1, 132, 32, 1, 1, 0, 1, 132, 32, + 0, 1, 0, 1, 136, 38, 2, 1, 0, 1, 136, 32, 1, 1, 0, 1, 136, 32, + 0, 1, 0, 1, 140, 34, 2, 1, 0, 1, 140, 32, 1, 1, 0, 1, 140, 32, + 0, 1, 0, 1, 144, 34, 2, 1, 0, 1, 144, 32, 1, 1, 0, 1, 144, 63, + 0, 1, 0, 1, 149, 38, 2, 1, 0, 1, 149, 63, 1, 1, 0, 1, 149, 63, + 0, 1, 0, 1, 153, 38, 2, 1, 0, 1, 153, 63, 1, 1, 0, 1, 153, 63, + 0, 1, 0, 1, 157, 38, 2, 1, 0, 1, 157, 63, 1, 1, 0, 1, 157, 63, + 0, 1, 0, 1, 161, 38, 2, 1, 0, 1, 161, 63, 1, 1, 0, 1, 161, 63, + 0, 1, 0, 1, 165, 38, 2, 1, 0, 1, 165, 63, 1, 1, 0, 1, 165, 63, + 0, 1, 0, 2, 36, 36, 2, 1, 0, 2, 36, 32, 1, 1, 0, 2, 36, 28, + 0, 1, 0, 2, 40, 38, 2, 1, 0, 2, 40, 32, 1, 1, 0, 2, 40, 28, + 0, 1, 0, 2, 44, 38, 2, 1, 0, 2, 44, 32, 1, 1, 0, 2, 44, 28, + 0, 1, 0, 2, 48, 38, 2, 1, 0, 2, 48, 32, 1, 1, 0, 2, 48, 28, + 0, 1, 0, 2, 52, 38, 2, 1, 0, 2, 52, 32, 1, 1, 0, 2, 52, 28, + 0, 1, 0, 2, 56, 38, 2, 1, 0, 2, 56, 32, 1, 1, 0, 2, 56, 28, + 0, 1, 0, 2, 60, 38, 2, 1, 0, 2, 60, 32, 1, 1, 0, 2, 60, 28, + 0, 1, 0, 2, 64, 34, 2, 1, 0, 2, 64, 32, 1, 1, 0, 2, 64, 28, + 0, 1, 0, 2, 100, 32, 2, 1, 0, 2, 100, 32, 1, 1, 0, 2, 100, 32, + 0, 1, 0, 2, 104, 38, 2, 1, 0, 2, 104, 32, 1, 1, 0, 2, 104, 32, + 0, 1, 0, 2, 108, 38, 2, 1, 0, 2, 108, 32, 1, 1, 0, 2, 108, 32, + 0, 1, 0, 2, 112, 38, 2, 1, 0, 2, 112, 32, 1, 1, 0, 2, 112, 32, + 0, 1, 0, 2, 116, 38, 2, 1, 0, 2, 116, 32, 1, 1, 0, 2, 116, 32, + 0, 1, 0, 2, 120, 38, 2, 1, 0, 2, 120, 32, 1, 1, 0, 2, 120, 32, + 0, 1, 0, 2, 124, 38, 2, 1, 0, 2, 124, 32, 1, 1, 0, 2, 124, 32, + 0, 1, 0, 2, 128, 38, 2, 1, 0, 2, 128, 32, 1, 1, 0, 2, 128, 32, + 0, 1, 0, 2, 132, 38, 2, 1, 0, 2, 132, 32, 1, 1, 0, 2, 132, 32, + 0, 1, 0, 2, 136, 38, 2, 1, 0, 2, 136, 32, 1, 1, 0, 2, 136, 32, + 0, 1, 0, 2, 140, 32, 2, 1, 0, 2, 140, 32, 1, 1, 0, 2, 140, 32, + 0, 1, 0, 2, 144, 26, 2, 1, 0, 2, 144, 63, 1, 1, 0, 2, 144, 63, + 0, 1, 0, 2, 149, 38, 2, 1, 0, 2, 149, 63, 1, 1, 0, 2, 149, 63, + 0, 1, 0, 2, 153, 38, 2, 1, 0, 2, 153, 63, 1, 1, 0, 2, 153, 63, + 0, 1, 0, 2, 157, 38, 2, 1, 0, 2, 157, 63, 1, 1, 0, 2, 157, 63, + 0, 1, 0, 2, 161, 38, 2, 1, 0, 2, 161, 63, 1, 1, 0, 2, 161, 63, + 0, 1, 0, 2, 165, 38, 2, 1, 0, 2, 165, 63, 1, 1, 0, 2, 165, 63, + 0, 1, 0, 3, 36, 34, 2, 1, 0, 3, 36, 20, 1, 1, 0, 3, 36, 22, + 0, 1, 0, 3, 40, 36, 2, 1, 0, 3, 40, 20, 1, 1, 0, 3, 40, 22, + 0, 1, 0, 3, 44, 36, 2, 1, 0, 3, 44, 20, 1, 1, 0, 3, 44, 22, + 0, 1, 0, 3, 48, 36, 2, 1, 0, 3, 48, 20, 1, 1, 0, 3, 48, 22, + 0, 1, 0, 3, 52, 36, 2, 1, 0, 3, 52, 20, 1, 1, 0, 3, 52, 22, + 0, 1, 0, 3, 56, 36, 2, 1, 0, 3, 56, 20, 1, 1, 0, 3, 56, 22, + 0, 1, 0, 3, 60, 36, 2, 1, 0, 3, 60, 20, 1, 1, 0, 3, 60, 22, + 0, 1, 0, 3, 64, 34, 2, 1, 0, 3, 64, 20, 1, 1, 0, 3, 64, 22, + 0, 1, 0, 3, 100, 32, 2, 1, 0, 3, 100, 20, 1, 1, 0, 3, 100, 30, + 0, 1, 0, 3, 104, 36, 2, 1, 0, 3, 104, 20, 1, 1, 0, 3, 104, 30, + 0, 1, 0, 3, 108, 38, 2, 1, 0, 3, 108, 20, 1, 1, 0, 3, 108, 30, + 0, 1, 0, 3, 112, 38, 2, 1, 0, 3, 112, 20, 1, 1, 0, 3, 112, 30, + 0, 1, 0, 3, 116, 38, 2, 1, 0, 3, 116, 20, 1, 1, 0, 3, 116, 30, + 0, 1, 0, 3, 120, 38, 2, 1, 0, 3, 120, 20, 1, 1, 0, 3, 120, 30, + 0, 1, 0, 3, 124, 38, 2, 1, 0, 3, 124, 20, 1, 1, 0, 3, 124, 30, + 0, 1, 0, 3, 128, 38, 2, 1, 0, 3, 128, 20, 1, 1, 0, 3, 128, 30, + 0, 1, 0, 3, 132, 38, 2, 1, 0, 3, 132, 20, 1, 1, 0, 3, 132, 30, + 0, 1, 0, 3, 136, 36, 2, 1, 0, 3, 136, 20, 1, 1, 0, 3, 136, 30, + 0, 1, 0, 3, 140, 32, 2, 1, 0, 3, 140, 20, 1, 1, 0, 3, 140, 30, + 0, 1, 0, 3, 144, 26, 2, 1, 0, 3, 144, 63, 1, 1, 0, 3, 144, 63, + 0, 1, 0, 3, 149, 38, 2, 1, 0, 3, 149, 63, 1, 1, 0, 3, 149, 63, + 0, 1, 0, 3, 153, 38, 2, 1, 0, 3, 153, 63, 1, 1, 0, 3, 153, 63, + 0, 1, 0, 3, 157, 38, 2, 1, 0, 3, 157, 63, 1, 1, 0, 3, 157, 63, + 0, 1, 0, 3, 161, 38, 2, 1, 0, 3, 161, 63, 1, 1, 0, 3, 161, 63, + 0, 1, 0, 3, 165, 38, 2, 1, 0, 3, 165, 63, 1, 1, 0, 3, 165, 63, + 0, 1, 1, 2, 38, 28, 2, 1, 1, 2, 38, 30, 1, 1, 1, 2, 38, 30, + 0, 1, 1, 2, 46, 36, 2, 1, 1, 2, 46, 30, 1, 1, 1, 2, 46, 30, + 0, 1, 1, 2, 54, 36, 2, 1, 1, 2, 54, 30, 1, 1, 1, 2, 54, 30, + 0, 1, 1, 2, 62, 30, 2, 1, 1, 2, 62, 30, 1, 1, 1, 2, 62, 30, + 0, 1, 1, 2, 102, 30, 2, 1, 1, 2, 102, 30, 1, 1, 1, 2, 102, 30, + 0, 1, 1, 2, 110, 36, 2, 1, 1, 2, 110, 30, 1, 1, 1, 2, 110, 30, + 0, 1, 1, 2, 118, 36, 2, 1, 1, 2, 118, 30, 1, 1, 1, 2, 118, 30, + 0, 1, 1, 2, 126, 36, 2, 1, 1, 2, 126, 30, 1, 1, 1, 2, 126, 30, + 0, 1, 1, 2, 134, 36, 2, 1, 1, 2, 134, 30, 1, 1, 1, 2, 134, 30, + 0, 1, 1, 2, 142, 30, 2, 1, 1, 2, 142, 63, 1, 1, 1, 2, 142, 63, + 0, 1, 1, 2, 151, 36, 2, 1, 1, 2, 151, 63, 1, 1, 1, 2, 151, 63, + 0, 1, 1, 2, 159, 36, 2, 1, 1, 2, 159, 63, 1, 1, 1, 2, 159, 63, + 0, 1, 1, 3, 38, 26, 2, 1, 1, 3, 38, 20, 1, 1, 1, 3, 38, 22, + 0, 1, 1, 3, 46, 36, 2, 1, 1, 3, 46, 20, 1, 1, 1, 3, 46, 22, + 0, 1, 1, 3, 54, 36, 2, 1, 1, 3, 54, 20, 1, 1, 1, 3, 54, 22, + 0, 1, 1, 3, 62, 28, 2, 1, 1, 3, 62, 20, 1, 1, 1, 3, 62, 22, + 0, 1, 1, 3, 102, 28, 2, 1, 1, 3, 102, 20, 1, 1, 1, 3, 102, 30, + 0, 1, 1, 3, 110, 36, 2, 1, 1, 3, 110, 20, 1, 1, 1, 3, 110, 30, + 0, 1, 1, 3, 118, 36, 2, 1, 1, 3, 118, 20, 1, 1, 1, 3, 118, 30, + 0, 1, 1, 3, 126, 36, 2, 1, 1, 3, 126, 20, 1, 1, 1, 3, 126, 30, + 0, 1, 1, 3, 134, 36, 2, 1, 1, 3, 134, 20, 1, 1, 1, 3, 134, 30, + 0, 1, 1, 3, 142, 30, 2, 1, 1, 3, 142, 63, 1, 1, 1, 3, 142, 63, + 0, 1, 1, 3, 151, 36, 2, 1, 1, 3, 151, 63, 1, 1, 1, 3, 151, 63, + 0, 1, 1, 3, 159, 36, 2, 1, 1, 3, 159, 63, 1, 1, 1, 3, 159, 63, + 0, 1, 2, 4, 42, 26, 2, 1, 2, 4, 42, 30, 1, 1, 2, 4, 42, 28, + 0, 1, 2, 4, 58, 26, 2, 1, 2, 4, 58, 30, 1, 1, 2, 4, 58, 28, + 0, 1, 2, 4, 106, 26, 2, 1, 2, 4, 106, 30, 1, 1, 2, 4, 106, 30, + 0, 1, 2, 4, 122, 36, 2, 1, 2, 4, 122, 30, 1, 1, 2, 4, 122, 30, + 0, 1, 2, 4, 138, 36, 2, 1, 2, 4, 138, 63, 1, 1, 2, 4, 138, 63, + 0, 1, 2, 4, 155, 36, 2, 1, 2, 4, 155, 63, 1, 1, 2, 4, 155, 63, + 0, 1, 2, 5, 42, 24, 2, 1, 2, 5, 42, 20, 1, 1, 2, 5, 42, 22, + 0, 1, 2, 5, 58, 24, 2, 1, 2, 5, 58, 20, 1, 1, 2, 5, 58, 22, + 0, 1, 2, 5, 106, 26, 2, 1, 2, 5, 106, 20, 1, 1, 2, 5, 106, 30, + 0, 1, 2, 5, 122, 36, 2, 1, 2, 5, 122, 20, 1, 1, 2, 5, 122, 30, + 0, 1, 2, 5, 138, 36, 2, 1, 2, 5, 138, 63, 1, 1, 2, 5, 138, 63, + 0, 1, 2, 5, 155, 36, 2, 1, 2, 5, 155, 63, 1, 1, 2, 5, 155, 63 +}; + +RTW_DECL_TABLE_TXPWR_LMT(rtw8822b_txpwr_lmt_type2); + +static const u8 rtw8822b_txpwr_lmt_type5[] = { + 0, 0, 0, 0, 1, 32, 2, 0, 0, 0, 1, 28, 1, 0, 0, 0, 1, 30, + 0, 0, 0, 0, 2, 32, 2, 0, 0, 0, 2, 28, 1, 0, 0, 0, 2, 30, + 0, 0, 0, 0, 3, 32, 2, 0, 0, 0, 3, 28, 1, 0, 0, 0, 3, 30, + 0, 0, 0, 0, 4, 32, 2, 0, 0, 0, 4, 28, 1, 0, 0, 0, 4, 30, + 0, 0, 0, 0, 5, 32, 2, 0, 0, 0, 5, 28, 1, 0, 0, 0, 5, 30, + 0, 0, 0, 0, 6, 32, 2, 0, 0, 0, 6, 28, 1, 0, 0, 0, 6, 30, + 0, 0, 0, 0, 7, 32, 2, 0, 0, 0, 7, 28, 1, 0, 0, 0, 7, 30, + 0, 0, 0, 0, 8, 32, 2, 0, 0, 0, 8, 28, 1, 0, 0, 0, 8, 30, + 0, 0, 0, 0, 9, 32, 2, 0, 0, 0, 9, 28, 1, 0, 0, 0, 9, 30, + 0, 0, 0, 0, 10, 32, 2, 0, 0, 0, 10, 28, 1, 0, 0, 0, 10, 30, + 0, 0, 0, 0, 11, 32, 2, 0, 0, 0, 11, 28, 1, 0, 0, 0, 11, 30, + 0, 0, 0, 0, 12, 26, 2, 0, 0, 0, 12, 28, 1, 0, 0, 0, 12, 30, + 0, 0, 0, 0, 13, 20, 2, 0, 0, 0, 13, 28, 1, 0, 0, 0, 13, 28, + 0, 0, 0, 0, 14, 63, 2, 0, 0, 0, 14, 63, 1, 0, 0, 0, 14, 32, + 0, 0, 0, 1, 1, 26, 2, 0, 0, 1, 1, 30, 1, 0, 0, 1, 1, 34, + 0, 0, 0, 1, 2, 30, 2, 0, 0, 1, 2, 30, 1, 0, 0, 1, 2, 34, + 0, 0, 0, 1, 3, 32, 2, 0, 0, 1, 3, 30, 1, 0, 0, 1, 3, 34, + 0, 0, 0, 1, 4, 34, 2, 0, 0, 1, 4, 30, 1, 0, 0, 1, 4, 34, + 0, 0, 0, 1, 5, 34, 2, 0, 0, 1, 5, 30, 1, 0, 0, 1, 5, 34, + 0, 0, 0, 1, 6, 34, 2, 0, 0, 1, 6, 30, 1, 0, 0, 1, 6, 34, + 0, 0, 0, 1, 7, 34, 2, 0, 0, 1, 7, 30, 1, 0, 0, 1, 7, 34, + 0, 0, 0, 1, 8, 34, 2, 0, 0, 1, 8, 30, 1, 0, 0, 1, 8, 34, + 0, 0, 0, 1, 9, 32, 2, 0, 0, 1, 9, 30, 1, 0, 0, 1, 9, 34, + 0, 0, 0, 1, 10, 30, 2, 0, 0, 1, 10, 30, 1, 0, 0, 1, 10, 34, + 0, 0, 0, 1, 11, 28, 2, 0, 0, 1, 11, 30, 1, 0, 0, 1, 11, 34, + 0, 0, 0, 1, 12, 22, 2, 0, 0, 1, 12, 30, 1, 0, 0, 1, 12, 34, + 0, 0, 0, 1, 13, 14, 2, 0, 0, 1, 13, 30, 1, 0, 0, 1, 13, 34, + 0, 0, 0, 1, 14, 63, 2, 0, 0, 1, 14, 63, 1, 0, 0, 1, 14, 63, + 0, 0, 0, 2, 1, 26, 2, 0, 0, 2, 1, 30, 1, 0, 0, 2, 1, 34, + 0, 0, 0, 2, 2, 30, 2, 0, 0, 2, 2, 30, 1, 0, 0, 2, 2, 34, + 0, 0, 0, 2, 3, 32, 2, 0, 0, 2, 3, 30, 1, 0, 0, 2, 3, 34, + 0, 0, 0, 2, 4, 34, 2, 0, 0, 2, 4, 30, 1, 0, 0, 2, 4, 34, + 0, 0, 0, 2, 5, 34, 2, 0, 0, 2, 5, 30, 1, 0, 0, 2, 5, 34, + 0, 0, 0, 2, 6, 34, 2, 0, 0, 2, 6, 30, 1, 0, 0, 2, 6, 34, + 0, 0, 0, 2, 7, 34, 2, 0, 0, 2, 7, 30, 1, 0, 0, 2, 7, 34, + 0, 0, 0, 2, 8, 34, 2, 0, 0, 2, 8, 30, 1, 0, 0, 2, 8, 34, + 0, 0, 0, 2, 9, 32, 2, 0, 0, 2, 9, 30, 1, 0, 0, 2, 9, 34, + 0, 0, 0, 2, 10, 30, 2, 0, 0, 2, 10, 30, 1, 0, 0, 2, 10, 34, + 0, 0, 0, 2, 11, 26, 2, 0, 0, 2, 11, 30, 1, 0, 0, 2, 11, 34, + 0, 0, 0, 2, 12, 20, 2, 0, 0, 2, 12, 30, 1, 0, 0, 2, 12, 34, + 0, 0, 0, 2, 13, 14, 2, 0, 0, 2, 13, 30, 1, 0, 0, 2, 13, 34, + 0, 0, 0, 2, 14, 63, 2, 0, 0, 2, 14, 63, 1, 0, 0, 2, 14, 63, + 0, 0, 0, 3, 1, 26, 2, 0, 0, 3, 1, 18, 1, 0, 0, 3, 1, 30, + 0, 0, 0, 3, 2, 28, 2, 0, 0, 3, 2, 18, 1, 0, 0, 3, 2, 30, + 0, 0, 0, 3, 3, 30, 2, 0, 0, 3, 3, 18, 1, 0, 0, 3, 3, 30, + 0, 0, 0, 3, 4, 30, 2, 0, 0, 3, 4, 18, 1, 0, 0, 3, 4, 30, + 0, 0, 0, 3, 5, 32, 2, 0, 0, 3, 5, 18, 1, 0, 0, 3, 5, 30, + 0, 0, 0, 3, 6, 32, 2, 0, 0, 3, 6, 18, 1, 0, 0, 3, 6, 30, + 0, 0, 0, 3, 7, 32, 2, 0, 0, 3, 7, 18, 1, 0, 0, 3, 7, 30, + 0, 0, 0, 3, 8, 30, 2, 0, 0, 3, 8, 18, 1, 0, 0, 3, 8, 30, + 0, 0, 0, 3, 9, 30, 2, 0, 0, 3, 9, 18, 1, 0, 0, 3, 9, 30, + 0, 0, 0, 3, 10, 28, 2, 0, 0, 3, 10, 18, 1, 0, 0, 3, 10, 30, + 0, 0, 0, 3, 11, 26, 2, 0, 0, 3, 11, 18, 1, 0, 0, 3, 11, 30, + 0, 0, 0, 3, 12, 20, 2, 0, 0, 3, 12, 18, 1, 0, 0, 3, 12, 30, + 0, 0, 0, 3, 13, 14, 2, 0, 0, 3, 13, 18, 1, 0, 0, 3, 13, 30, + 0, 0, 0, 3, 14, 63, 2, 0, 0, 3, 14, 63, 1, 0, 0, 3, 14, 63, + 0, 0, 1, 2, 1, 63, 2, 0, 1, 2, 1, 63, 1, 0, 1, 2, 1, 63, + 0, 0, 1, 2, 2, 63, 2, 0, 1, 2, 2, 63, 1, 0, 1, 2, 2, 63, + 0, 0, 1, 2, 3, 26, 2, 0, 1, 2, 3, 30, 1, 0, 1, 2, 3, 34, + 0, 0, 1, 2, 4, 26, 2, 0, 1, 2, 4, 30, 1, 0, 1, 2, 4, 34, + 0, 0, 1, 2, 5, 30, 2, 0, 1, 2, 5, 30, 1, 0, 1, 2, 5, 34, + 0, 0, 1, 2, 6, 32, 2, 0, 1, 2, 6, 30, 1, 0, 1, 2, 6, 34, + 0, 0, 1, 2, 7, 30, 2, 0, 1, 2, 7, 30, 1, 0, 1, 2, 7, 34, + 0, 0, 1, 2, 8, 26, 2, 0, 1, 2, 8, 30, 1, 0, 1, 2, 8, 34, + 0, 0, 1, 2, 9, 26, 2, 0, 1, 2, 9, 30, 1, 0, 1, 2, 9, 34, + 0, 0, 1, 2, 10, 20, 2, 0, 1, 2, 10, 30, 1, 0, 1, 2, 10, 34, + 0, 0, 1, 2, 11, 14, 2, 0, 1, 2, 11, 30, 1, 0, 1, 2, 11, 34, + 0, 0, 1, 2, 12, 63, 2, 0, 1, 2, 12, 63, 1, 0, 1, 2, 12, 63, + 0, 0, 1, 2, 13, 63, 2, 0, 1, 2, 13, 63, 1, 0, 1, 2, 13, 63, + 0, 0, 1, 2, 14, 63, 2, 0, 1, 2, 14, 63, 1, 0, 1, 2, 14, 63, + 0, 0, 1, 3, 1, 63, 2, 0, 1, 3, 1, 63, 1, 0, 1, 3, 1, 63, + 0, 0, 1, 3, 2, 63, 2, 0, 1, 3, 2, 63, 1, 0, 1, 3, 2, 63, + 0, 0, 1, 3, 3, 24, 2, 0, 1, 3, 3, 18, 1, 0, 1, 3, 3, 30, + 0, 0, 1, 3, 4, 24, 2, 0, 1, 3, 4, 18, 1, 0, 1, 3, 4, 30, + 0, 0, 1, 3, 5, 26, 2, 0, 1, 3, 5, 18, 1, 0, 1, 3, 5, 30, + 0, 0, 1, 3, 6, 28, 2, 0, 1, 3, 6, 18, 1, 0, 1, 3, 6, 30, + 0, 0, 1, 3, 7, 26, 2, 0, 1, 3, 7, 18, 1, 0, 1, 3, 7, 30, + 0, 0, 1, 3, 8, 26, 2, 0, 1, 3, 8, 18, 1, 0, 1, 3, 8, 30, + 0, 0, 1, 3, 9, 26, 2, 0, 1, 3, 9, 18, 1, 0, 1, 3, 9, 30, + 0, 0, 1, 3, 10, 20, 2, 0, 1, 3, 10, 18, 1, 0, 1, 3, 10, 30, + 0, 0, 1, 3, 11, 14, 2, 0, 1, 3, 11, 18, 1, 0, 1, 3, 11, 30, + 0, 0, 1, 3, 12, 63, 2, 0, 1, 3, 12, 63, 1, 0, 1, 3, 12, 63, + 0, 0, 1, 3, 13, 63, 2, 0, 1, 3, 13, 63, 1, 0, 1, 3, 13, 63, + 0, 0, 1, 3, 14, 63, 2, 0, 1, 3, 14, 63, 1, 0, 1, 3, 14, 63, + 0, 1, 0, 1, 36, 30, 2, 1, 0, 1, 36, 32, 1, 1, 0, 1, 36, 30, + 0, 1, 0, 1, 40, 32, 2, 1, 0, 1, 40, 32, 1, 1, 0, 1, 40, 30, + 0, 1, 0, 1, 44, 32, 2, 1, 0, 1, 44, 32, 1, 1, 0, 1, 44, 30, + 0, 1, 0, 1, 48, 32, 2, 1, 0, 1, 48, 32, 1, 1, 0, 1, 48, 30, + 0, 1, 0, 1, 52, 32, 2, 1, 0, 1, 52, 32, 1, 1, 0, 1, 52, 28, + 0, 1, 0, 1, 56, 32, 2, 1, 0, 1, 56, 32, 1, 1, 0, 1, 56, 28, + 0, 1, 0, 1, 60, 32, 2, 1, 0, 1, 60, 32, 1, 1, 0, 1, 60, 28, + 0, 1, 0, 1, 64, 28, 2, 1, 0, 1, 64, 32, 1, 1, 0, 1, 64, 28, + 0, 1, 0, 1, 100, 26, 2, 1, 0, 1, 100, 32, 1, 1, 0, 1, 100, 32, + 0, 1, 0, 1, 104, 32, 2, 1, 0, 1, 104, 32, 1, 1, 0, 1, 104, 32, + 0, 1, 0, 1, 108, 32, 2, 1, 0, 1, 108, 32, 1, 1, 0, 1, 108, 32, + 0, 1, 0, 1, 112, 32, 2, 1, 0, 1, 112, 32, 1, 1, 0, 1, 112, 32, + 0, 1, 0, 1, 116, 32, 2, 1, 0, 1, 116, 32, 1, 1, 0, 1, 116, 32, + 0, 1, 0, 1, 120, 32, 2, 1, 0, 1, 120, 32, 1, 1, 0, 1, 120, 32, + 0, 1, 0, 1, 124, 32, 2, 1, 0, 1, 124, 32, 1, 1, 0, 1, 124, 32, + 0, 1, 0, 1, 128, 32, 2, 1, 0, 1, 128, 32, 1, 1, 0, 1, 128, 32, + 0, 1, 0, 1, 132, 32, 2, 1, 0, 1, 132, 32, 1, 1, 0, 1, 132, 32, + 0, 1, 0, 1, 136, 32, 2, 1, 0, 1, 136, 32, 1, 1, 0, 1, 136, 32, + 0, 1, 0, 1, 140, 28, 2, 1, 0, 1, 140, 32, 1, 1, 0, 1, 140, 32, + 0, 1, 0, 1, 144, 28, 2, 1, 0, 1, 144, 63, 1, 1, 0, 1, 144, 63, + 0, 1, 0, 1, 149, 32, 2, 1, 0, 1, 149, 63, 1, 1, 0, 1, 149, 63, + 0, 1, 0, 1, 153, 32, 2, 1, 0, 1, 153, 63, 1, 1, 0, 1, 153, 63, + 0, 1, 0, 1, 157, 32, 2, 1, 0, 1, 157, 63, 1, 1, 0, 1, 157, 63, + 0, 1, 0, 1, 161, 32, 2, 1, 0, 1, 161, 63, 1, 1, 0, 1, 161, 63, + 0, 1, 0, 1, 165, 32, 2, 1, 0, 1, 165, 63, 1, 1, 0, 1, 165, 63, + 0, 1, 0, 2, 36, 30, 2, 1, 0, 2, 36, 32, 1, 1, 0, 2, 36, 28, + 0, 1, 0, 2, 40, 32, 2, 1, 0, 2, 40, 32, 1, 1, 0, 2, 40, 28, + 0, 1, 0, 2, 44, 32, 2, 1, 0, 2, 44, 32, 1, 1, 0, 2, 44, 28, + 0, 1, 0, 2, 48, 32, 2, 1, 0, 2, 48, 32, 1, 1, 0, 2, 48, 28, + 0, 1, 0, 2, 52, 32, 2, 1, 0, 2, 52, 32, 1, 1, 0, 2, 52, 28, + 0, 1, 0, 2, 56, 32, 2, 1, 0, 2, 56, 32, 1, 1, 0, 2, 56, 28, + 0, 1, 0, 2, 60, 32, 2, 1, 0, 2, 60, 32, 1, 1, 0, 2, 60, 28, + 0, 1, 0, 2, 64, 28, 2, 1, 0, 2, 64, 32, 1, 1, 0, 2, 64, 28, + 0, 1, 0, 2, 100, 26, 2, 1, 0, 2, 100, 32, 1, 1, 0, 2, 100, 32, + 0, 1, 0, 2, 104, 32, 2, 1, 0, 2, 104, 32, 1, 1, 0, 2, 104, 32, + 0, 1, 0, 2, 108, 32, 2, 1, 0, 2, 108, 32, 1, 1, 0, 2, 108, 32, + 0, 1, 0, 2, 112, 32, 2, 1, 0, 2, 112, 32, 1, 1, 0, 2, 112, 32, + 0, 1, 0, 2, 116, 32, 2, 1, 0, 2, 116, 32, 1, 1, 0, 2, 116, 32, + 0, 1, 0, 2, 120, 32, 2, 1, 0, 2, 120, 32, 1, 1, 0, 2, 120, 32, + 0, 1, 0, 2, 124, 32, 2, 1, 0, 2, 124, 32, 1, 1, 0, 2, 124, 32, + 0, 1, 0, 2, 128, 32, 2, 1, 0, 2, 128, 32, 1, 1, 0, 2, 128, 32, + 0, 1, 0, 2, 132, 32, 2, 1, 0, 2, 132, 32, 1, 1, 0, 2, 132, 32, + 0, 1, 0, 2, 136, 32, 2, 1, 0, 2, 136, 32, 1, 1, 0, 2, 136, 32, + 0, 1, 0, 2, 140, 26, 2, 1, 0, 2, 140, 32, 1, 1, 0, 2, 140, 32, + 0, 1, 0, 2, 144, 26, 2, 1, 0, 2, 144, 63, 1, 1, 0, 2, 144, 63, + 0, 1, 0, 2, 149, 32, 2, 1, 0, 2, 149, 63, 1, 1, 0, 2, 149, 63, + 0, 1, 0, 2, 153, 32, 2, 1, 0, 2, 153, 63, 1, 1, 0, 2, 153, 63, + 0, 1, 0, 2, 157, 32, 2, 1, 0, 2, 157, 63, 1, 1, 0, 2, 157, 63, + 0, 1, 0, 2, 161, 32, 2, 1, 0, 2, 161, 63, 1, 1, 0, 2, 161, 63, + 0, 1, 0, 2, 165, 32, 2, 1, 0, 2, 165, 63, 1, 1, 0, 2, 165, 63, + 0, 1, 0, 3, 36, 28, 2, 1, 0, 3, 36, 20, 1, 1, 0, 3, 36, 22, + 0, 1, 0, 3, 40, 30, 2, 1, 0, 3, 40, 20, 1, 1, 0, 3, 40, 22, + 0, 1, 0, 3, 44, 30, 2, 1, 0, 3, 44, 20, 1, 1, 0, 3, 44, 22, + 0, 1, 0, 3, 48, 30, 2, 1, 0, 3, 48, 20, 1, 1, 0, 3, 48, 22, + 0, 1, 0, 3, 52, 30, 2, 1, 0, 3, 52, 20, 1, 1, 0, 3, 52, 22, + 0, 1, 0, 3, 56, 30, 2, 1, 0, 3, 56, 20, 1, 1, 0, 3, 56, 22, + 0, 1, 0, 3, 60, 30, 2, 1, 0, 3, 60, 20, 1, 1, 0, 3, 60, 22, + 0, 1, 0, 3, 64, 28, 2, 1, 0, 3, 64, 20, 1, 1, 0, 3, 64, 22, + 0, 1, 0, 3, 100, 26, 2, 1, 0, 3, 100, 20, 1, 1, 0, 3, 100, 30, + 0, 1, 0, 3, 104, 30, 2, 1, 0, 3, 104, 20, 1, 1, 0, 3, 104, 30, + 0, 1, 0, 3, 108, 32, 2, 1, 0, 3, 108, 20, 1, 1, 0, 3, 108, 30, + 0, 1, 0, 3, 112, 32, 2, 1, 0, 3, 112, 20, 1, 1, 0, 3, 112, 30, + 0, 1, 0, 3, 116, 32, 2, 1, 0, 3, 116, 20, 1, 1, 0, 3, 116, 30, + 0, 1, 0, 3, 120, 32, 2, 1, 0, 3, 120, 20, 1, 1, 0, 3, 120, 30, + 0, 1, 0, 3, 124, 32, 2, 1, 0, 3, 124, 20, 1, 1, 0, 3, 124, 30, + 0, 1, 0, 3, 128, 32, 2, 1, 0, 3, 128, 20, 1, 1, 0, 3, 128, 30, + 0, 1, 0, 3, 132, 32, 2, 1, 0, 3, 132, 20, 1, 1, 0, 3, 132, 30, + 0, 1, 0, 3, 136, 30, 2, 1, 0, 3, 136, 20, 1, 1, 0, 3, 136, 30, + 0, 1, 0, 3, 140, 26, 2, 1, 0, 3, 140, 20, 1, 1, 0, 3, 140, 30, + 0, 1, 0, 3, 144, 26, 2, 1, 0, 3, 144, 63, 1, 1, 0, 3, 144, 63, + 0, 1, 0, 3, 149, 32, 2, 1, 0, 3, 149, 63, 1, 1, 0, 3, 149, 63, + 0, 1, 0, 3, 153, 32, 2, 1, 0, 3, 153, 63, 1, 1, 0, 3, 153, 63, + 0, 1, 0, 3, 157, 32, 2, 1, 0, 3, 157, 63, 1, 1, 0, 3, 157, 63, + 0, 1, 0, 3, 161, 32, 2, 1, 0, 3, 161, 63, 1, 1, 0, 3, 161, 63, + 0, 1, 0, 3, 165, 32, 2, 1, 0, 3, 165, 63, 1, 1, 0, 3, 165, 63, + 0, 1, 1, 2, 38, 22, 2, 1, 1, 2, 38, 30, 1, 1, 1, 2, 38, 30, + 0, 1, 1, 2, 46, 30, 2, 1, 1, 2, 46, 30, 1, 1, 1, 2, 46, 30, + 0, 1, 1, 2, 54, 30, 2, 1, 1, 2, 54, 30, 1, 1, 1, 2, 54, 30, + 0, 1, 1, 2, 62, 24, 2, 1, 1, 2, 62, 30, 1, 1, 1, 2, 62, 30, + 0, 1, 1, 2, 102, 24, 2, 1, 1, 2, 102, 30, 1, 1, 1, 2, 102, 30, + 0, 1, 1, 2, 110, 30, 2, 1, 1, 2, 110, 30, 1, 1, 1, 2, 110, 30, + 0, 1, 1, 2, 118, 30, 2, 1, 1, 2, 118, 30, 1, 1, 1, 2, 118, 30, + 0, 1, 1, 2, 126, 30, 2, 1, 1, 2, 126, 30, 1, 1, 1, 2, 126, 30, + 0, 1, 1, 2, 134, 30, 2, 1, 1, 2, 134, 30, 1, 1, 1, 2, 134, 30, + 0, 1, 1, 2, 142, 30, 2, 1, 1, 2, 142, 63, 1, 1, 1, 2, 142, 63, + 0, 1, 1, 2, 151, 30, 2, 1, 1, 2, 151, 63, 1, 1, 1, 2, 151, 63, + 0, 1, 1, 2, 159, 30, 2, 1, 1, 2, 159, 63, 1, 1, 1, 2, 159, 63, + 0, 1, 1, 3, 38, 20, 2, 1, 1, 3, 38, 20, 1, 1, 1, 3, 38, 22, + 0, 1, 1, 3, 46, 30, 2, 1, 1, 3, 46, 20, 1, 1, 1, 3, 46, 22, + 0, 1, 1, 3, 54, 30, 2, 1, 1, 3, 54, 20, 1, 1, 1, 3, 54, 22, + 0, 1, 1, 3, 62, 22, 2, 1, 1, 3, 62, 20, 1, 1, 1, 3, 62, 22, + 0, 1, 1, 3, 102, 22, 2, 1, 1, 3, 102, 20, 1, 1, 1, 3, 102, 30, + 0, 1, 1, 3, 110, 30, 2, 1, 1, 3, 110, 20, 1, 1, 1, 3, 110, 30, + 0, 1, 1, 3, 118, 30, 2, 1, 1, 3, 118, 20, 1, 1, 1, 3, 118, 30, + 0, 1, 1, 3, 126, 30, 2, 1, 1, 3, 126, 20, 1, 1, 1, 3, 126, 30, + 0, 1, 1, 3, 134, 30, 2, 1, 1, 3, 134, 20, 1, 1, 1, 3, 134, 30, + 0, 1, 1, 3, 142, 30, 2, 1, 1, 3, 142, 63, 1, 1, 1, 3, 142, 63, + 0, 1, 1, 3, 151, 30, 2, 1, 1, 3, 151, 63, 1, 1, 1, 3, 151, 63, + 0, 1, 1, 3, 159, 30, 2, 1, 1, 3, 159, 63, 1, 1, 1, 3, 159, 63, + 0, 1, 2, 4, 42, 20, 2, 1, 2, 4, 42, 30, 1, 1, 2, 4, 42, 28, + 0, 1, 2, 4, 58, 20, 2, 1, 2, 4, 58, 30, 1, 1, 2, 4, 58, 28, + 0, 1, 2, 4, 106, 20, 2, 1, 2, 4, 106, 30, 1, 1, 2, 4, 106, 30, + 0, 1, 2, 4, 122, 30, 2, 1, 2, 4, 122, 30, 1, 1, 2, 4, 122, 30, + 0, 1, 2, 4, 138, 30, 2, 1, 2, 4, 138, 63, 1, 1, 2, 4, 138, 63, + 0, 1, 2, 4, 155, 30, 2, 1, 2, 4, 155, 63, 1, 1, 2, 4, 155, 63, + 0, 1, 2, 5, 42, 18, 2, 1, 2, 5, 42, 20, 1, 1, 2, 5, 42, 22, + 0, 1, 2, 5, 58, 18, 2, 1, 2, 5, 58, 20, 1, 1, 2, 5, 58, 22, + 0, 1, 2, 5, 106, 20, 2, 1, 2, 5, 106, 20, 1, 1, 2, 5, 106, 30, + 0, 1, 2, 5, 122, 30, 2, 1, 2, 5, 122, 20, 1, 1, 2, 5, 122, 30, + 0, 1, 2, 5, 138, 30, 2, 1, 2, 5, 138, 63, 1, 1, 2, 5, 138, 63, + 0, 1, 2, 5, 155, 30, 2, 1, 2, 5, 155, 63, 1, 1, 2, 5, 155, 63, +}; + +RTW_DECL_TABLE_TXPWR_LMT(rtw8822b_txpwr_lmt_type5); diff --git a/drivers/net/wireless/realtek/rtw88/rtw8822b_table.h b/drivers/net/wireless/realtek/rtw88/rtw8822b_table.h new file mode 100644 index 0000000000000000000000000000000000000000..d4c268889368c9e002282b9232af5d69760a9b1b --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/rtw8822b_table.h @@ -0,0 +1,18 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#ifndef __RTW8822B_TABLE_H__ +#define __RTW8822B_TABLE_H__ + +extern const struct rtw_table rtw8822b_mac_tbl; +extern const struct rtw_table rtw8822b_agc_tbl; +extern const struct rtw_table rtw8822b_bb_tbl; +extern const struct rtw_table rtw8822b_bb_pg_type2_tbl; +extern const struct rtw_table rtw8822b_bb_pg_type5_tbl; +extern const struct rtw_table rtw8822b_rf_a_tbl; +extern const struct rtw_table rtw8822b_rf_b_tbl; +extern const struct rtw_table rtw8822b_txpwr_lmt_type2_tbl; +extern const struct rtw_table rtw8822b_txpwr_lmt_type5_tbl; + +#endif diff --git a/drivers/net/wireless/realtek/rtw88/rtw8822c.c b/drivers/net/wireless/realtek/rtw88/rtw8822c.c new file mode 100644 index 0000000000000000000000000000000000000000..b4f7242e5aa312a74f1a5d969acde829dfffcf71 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/rtw8822c.c @@ -0,0 +1,1890 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#include "main.h" +#include "fw.h" +#include "tx.h" +#include "rx.h" +#include "phy.h" +#include "rtw8822c.h" +#include "rtw8822c_table.h" +#include "mac.h" +#include "reg.h" +#include "debug.h" + +static void rtw8822c_config_trx_mode(struct rtw_dev *rtwdev, u8 tx_path, + u8 rx_path, bool is_tx2_path); + +static void rtw8822ce_efuse_parsing(struct rtw_efuse *efuse, + struct rtw8822c_efuse *map) +{ + ether_addr_copy(efuse->addr, map->e.mac_addr); +} + +static int rtw8822c_read_efuse(struct rtw_dev *rtwdev, u8 *log_map) +{ + struct rtw_efuse *efuse = &rtwdev->efuse; + struct rtw8822c_efuse *map; + int i; + + map = (struct rtw8822c_efuse *)log_map; + + efuse->rfe_option = map->rfe_option; + efuse->crystal_cap = map->xtal_k; + efuse->channel_plan = map->channel_plan; + efuse->country_code[0] = map->country_code[0]; + efuse->country_code[1] = map->country_code[1]; + efuse->bt_setting = map->rf_bt_setting; + efuse->regd = map->rf_board_option & 0x7; + + for (i = 0; i < 4; i++) + efuse->txpwr_idx_table[i] = map->txpwr_idx_table[i]; + + switch (rtw_hci_type(rtwdev)) { + case RTW_HCI_TYPE_PCIE: + rtw8822ce_efuse_parsing(efuse, map); + break; + default: + /* unsupported now */ + return -ENOTSUPP; + } + + return 0; +} + +static void rtw8822c_header_file_init(struct rtw_dev *rtwdev, bool pre) +{ + rtw_write32_set(rtwdev, REG_3WIRE, BIT_3WIRE_TX_EN | BIT_3WIRE_RX_EN); + rtw_write32_set(rtwdev, REG_3WIRE, BIT_3WIRE_PI_ON); + rtw_write32_set(rtwdev, REG_3WIRE2, BIT_3WIRE_TX_EN | BIT_3WIRE_RX_EN); + rtw_write32_set(rtwdev, REG_3WIRE2, BIT_3WIRE_PI_ON); + + if (pre) + rtw_write32_clr(rtwdev, REG_ENCCK, BIT_CCK_OFDM_BLK_EN); + else + rtw_write32_set(rtwdev, REG_ENCCK, BIT_CCK_OFDM_BLK_EN); +} + +static void rtw8822c_dac_backup_reg(struct rtw_dev *rtwdev, + struct rtw_backup_info *backup, + struct rtw_backup_info *backup_rf) +{ + u32 path, i; + u32 val; + u32 reg; + u32 rf_addr[DACK_RF_8822C] = {0x8f}; + u32 addrs[DACK_REG_8822C] = {0x180c, 0x1810, 0x410c, 0x4110, + 0x1c3c, 0x1c24, 0x1d70, 0x9b4, + 0x1a00, 0x1a14, 0x1d58, 0x1c38, + 0x1e24, 0x1e28, 0x1860, 0x4160}; + + for (i = 0; i < DACK_REG_8822C; i++) { + backup[i].len = 4; + backup[i].reg = addrs[i]; + backup[i].val = rtw_read32(rtwdev, addrs[i]); + } + + for (path = 0; path < DACK_PATH_8822C; path++) { + for (i = 0; i < DACK_RF_8822C; i++) { + reg = rf_addr[i]; + val = rtw_read_rf(rtwdev, path, reg, RFREG_MASK); + backup_rf[path * i + i].reg = reg; + backup_rf[path * i + i].val = val; + } + } +} + +static void rtw8822c_dac_restore_reg(struct rtw_dev *rtwdev, + struct rtw_backup_info *backup, + struct rtw_backup_info *backup_rf) +{ + u32 path, i; + u32 val; + u32 reg; + + rtw_restore_reg(rtwdev, backup, DACK_REG_8822C); + + for (path = 0; path < DACK_PATH_8822C; path++) { + for (i = 0; i < DACK_RF_8822C; i++) { + val = backup_rf[path * i + i].val; + reg = backup_rf[path * i + i].reg; + rtw_write_rf(rtwdev, path, reg, RFREG_MASK, val); + } + } +} + +static void rtw8822c_rf_minmax_cmp(struct rtw_dev *rtwdev, u32 value, + u32 *min, u32 *max) +{ + if (value >= 0x200) { + if (*min >= 0x200) { + if (*min > value) + *min = value; + } else { + *min = value; + } + if (*max >= 0x200) { + if (*max < value) + *max = value; + } + } else { + if (*min < 0x200) { + if (*min > value) + *min = value; + } + + if (*max >= 0x200) { + *max = value; + } else { + if (*max < value) + *max = value; + } + } +} + +static void swap_u32(u32 *v1, u32 *v2) +{ + u32 tmp; + + tmp = *v1; + *v1 = *v2; + *v2 = tmp; +} + +static void __rtw8822c_dac_iq_sort(struct rtw_dev *rtwdev, u32 *v1, u32 *v2) +{ + if (*v1 >= 0x200 && *v2 >= 0x200) { + if (*v1 > *v2) + swap_u32(v1, v2); + } else if (*v1 < 0x200 && *v2 < 0x200) { + if (*v1 > *v2) + swap_u32(v1, v2); + } else if (*v1 < 0x200 && *v2 >= 0x200) { + swap_u32(v1, v2); + } +} + +static void rtw8822c_dac_iq_sort(struct rtw_dev *rtwdev, u32 *iv, u32 *qv) +{ + u32 i, j; + + for (i = 0; i < DACK_SN_8822C - 1; i++) { + for (j = 0; j < (DACK_SN_8822C - 1 - i) ; j++) { + __rtw8822c_dac_iq_sort(rtwdev, &iv[j], &iv[j + 1]); + __rtw8822c_dac_iq_sort(rtwdev, &qv[j], &qv[j + 1]); + } + } +} + +static void rtw8822c_dac_iq_offset(struct rtw_dev *rtwdev, u32 *vec, u32 *val) +{ + u32 p, m, t, i; + + m = 0; + p = 0; + for (i = 10; i < DACK_SN_8822C - 10; i++) { + if (vec[i] > 0x200) + m = (0x400 - vec[i]) + m; + else + p = vec[i] + p; + } + + if (p > m) { + t = p - m; + t = t / (DACK_SN_8822C - 20); + } else { + t = m - p; + t = t / (DACK_SN_8822C - 20); + if (t != 0x0) + t = 0x400 - t; + } + + *val = t; +} + +static u32 rtw8822c_get_path_base_addr(u8 path) +{ + u32 base_addr; + + switch (path) { + case RF_PATH_A: + base_addr = 0x1800; + break; + case RF_PATH_B: + base_addr = 0x4100; + break; + default: + WARN_ON(1); + return -1; + } + + return base_addr; +} + +static bool rtw8822c_dac_iq_check(struct rtw_dev *rtwdev, u32 value) +{ + bool ret = true; + + if ((value >= 0x200 && (0x400 - value) > 0x64) || + (value < 0x200 && value > 0x64)) { + ret = false; + rtw_dbg(rtwdev, RTW_DBG_RFK, "[DACK] Error overflow\n"); + } + + return ret; +} + +static void rtw8822c_dac_cal_iq_sample(struct rtw_dev *rtwdev, u32 *iv, u32 *qv) +{ + u32 temp; + int i = 0, cnt = 0; + + while (i < DACK_SN_8822C && cnt < 10000) { + cnt++; + temp = rtw_read32_mask(rtwdev, 0x2dbc, 0x3fffff); + iv[i] = (temp & 0x3ff000) >> 12; + qv[i] = temp & 0x3ff; + + if (rtw8822c_dac_iq_check(rtwdev, iv[i]) && + rtw8822c_dac_iq_check(rtwdev, qv[i])) + i++; + } +} + +static void rtw8822c_dac_cal_iq_search(struct rtw_dev *rtwdev, + u32 *iv, u32 *qv, + u32 *i_value, u32 *q_value) +{ + u32 i_max = 0, q_max = 0, i_min = 0, q_min = 0; + u32 i_delta, q_delta; + u32 temp; + int i, cnt = 0; + + do { + i_min = iv[0]; + i_max = iv[0]; + q_min = qv[0]; + q_max = qv[0]; + for (i = 0; i < DACK_SN_8822C; i++) { + rtw8822c_rf_minmax_cmp(rtwdev, iv[i], &i_min, &i_max); + rtw8822c_rf_minmax_cmp(rtwdev, qv[i], &q_min, &q_max); + } + + if (i_max < 0x200 && i_min < 0x200) + i_delta = i_max - i_min; + else if (i_max >= 0x200 && i_min >= 0x200) + i_delta = i_max - i_min; + else + i_delta = i_max + (0x400 - i_min); + + if (q_max < 0x200 && q_min < 0x200) + q_delta = q_max - q_min; + else if (q_max >= 0x200 && q_min >= 0x200) + q_delta = q_max - q_min; + else + q_delta = q_max + (0x400 - q_min); + + rtw_dbg(rtwdev, RTW_DBG_RFK, + "[DACK] i: min=0x%08x, max=0x%08x, delta=0x%08x\n", + i_min, i_max, i_delta); + rtw_dbg(rtwdev, RTW_DBG_RFK, + "[DACK] q: min=0x%08x, max=0x%08x, delta=0x%08x\n", + q_min, q_max, q_delta); + + rtw8822c_dac_iq_sort(rtwdev, iv, qv); + + if (i_delta > 5 || q_delta > 5) { + temp = rtw_read32_mask(rtwdev, 0x2dbc, 0x3fffff); + iv[0] = (temp & 0x3ff000) >> 12; + qv[0] = temp & 0x3ff; + temp = rtw_read32_mask(rtwdev, 0x2dbc, 0x3fffff); + iv[DACK_SN_8822C - 1] = (temp & 0x3ff000) >> 12; + qv[DACK_SN_8822C - 1] = temp & 0x3ff; + } else { + break; + } + } while (cnt++ < 100); + + rtw8822c_dac_iq_offset(rtwdev, iv, i_value); + rtw8822c_dac_iq_offset(rtwdev, qv, q_value); +} + +static void rtw8822c_dac_cal_rf_mode(struct rtw_dev *rtwdev, + u32 *i_value, u32 *q_value) +{ + u32 iv[DACK_SN_8822C], qv[DACK_SN_8822C]; + u32 rf_a, rf_b; + + mdelay(10); + + rf_a = rtw_read_rf(rtwdev, RF_PATH_A, 0x0, RFREG_MASK); + rf_b = rtw_read_rf(rtwdev, RF_PATH_B, 0x0, RFREG_MASK); + + rtw_dbg(rtwdev, RTW_DBG_RFK, "[DACK] RF path-A=0x%05x\n", rf_a); + rtw_dbg(rtwdev, RTW_DBG_RFK, "[DACK] RF path-B=0x%05x\n", rf_b); + + rtw8822c_dac_cal_iq_sample(rtwdev, iv, qv); + rtw8822c_dac_cal_iq_search(rtwdev, iv, qv, i_value, q_value); +} + +static void rtw8822c_dac_bb_setting(struct rtw_dev *rtwdev) +{ + rtw_write32_mask(rtwdev, 0x1d58, 0xff8, 0x1ff); + rtw_write32_mask(rtwdev, 0x1a00, 0x3, 0x2); + rtw_write32_mask(rtwdev, 0x1a14, 0x300, 0x3); + rtw_write32(rtwdev, 0x1d70, 0x7e7e7e7e); + rtw_write32_mask(rtwdev, 0x180c, 0x3, 0x0); + rtw_write32_mask(rtwdev, 0x410c, 0x3, 0x0); + rtw_write32(rtwdev, 0x1b00, 0x00000008); + rtw_write8(rtwdev, 0x1bcc, 0x3f); + rtw_write32(rtwdev, 0x1b00, 0x0000000a); + rtw_write8(rtwdev, 0x1bcc, 0x3f); + rtw_write32_mask(rtwdev, 0x1e24, BIT(31), 0x0); + rtw_write32_mask(rtwdev, 0x1e28, 0xf, 0x3); +} + +static void rtw8822c_dac_cal_adc(struct rtw_dev *rtwdev, + u8 path, u32 *adc_ic, u32 *adc_qc) +{ + u32 ic = 0, qc = 0, temp = 0; + u32 base_addr; + u32 path_sel; + int i; + + rtw_dbg(rtwdev, RTW_DBG_RFK, "[DACK] ADCK path(%d)\n", path); + + base_addr = rtw8822c_get_path_base_addr(path); + switch (path) { + case RF_PATH_A: + path_sel = 0xa0000; + break; + case RF_PATH_B: + path_sel = 0x80000; + break; + default: + WARN_ON(1); + return; + } + + /* ADCK step1 */ + rtw_write32_mask(rtwdev, base_addr + 0x30, BIT(30), 0x0); + if (path == RF_PATH_B) + rtw_write32(rtwdev, base_addr + 0x30, 0x30db8041); + rtw_write32(rtwdev, base_addr + 0x60, 0xf0040ff0); + rtw_write32(rtwdev, base_addr + 0x0c, 0xdff00220); + rtw_write32(rtwdev, base_addr + 0x10, 0x02dd08c4); + rtw_write32(rtwdev, base_addr + 0x0c, 0x10000260); + rtw_write_rf(rtwdev, RF_PATH_A, 0x0, RFREG_MASK, 0x10000); + rtw_write_rf(rtwdev, RF_PATH_B, 0x0, RFREG_MASK, 0x10000); + for (i = 0; i < 10; i++) { + rtw_dbg(rtwdev, RTW_DBG_RFK, "[DACK] ADCK count=%d\n", i); + rtw_write32(rtwdev, 0x1c3c, path_sel + 0x8003); + rtw_write32(rtwdev, 0x1c24, 0x00010002); + rtw8822c_dac_cal_rf_mode(rtwdev, &ic, &qc); + rtw_dbg(rtwdev, RTW_DBG_RFK, + "[DACK] before: i=0x%x, q=0x%x\n", ic, qc); + + /* compensation value */ + if (ic != 0x0) { + ic = 0x400 - ic; + *adc_ic = ic; + } + if (qc != 0x0) { + qc = 0x400 - qc; + *adc_qc = qc; + } + temp = (ic & 0x3ff) | ((qc & 0x3ff) << 10); + rtw_write32(rtwdev, base_addr + 0x68, temp); + rtw_dbg(rtwdev, RTW_DBG_RFK, "[DACK] ADCK 0x%08x=0x08%x\n", + base_addr + 0x68, temp); + /* check ADC DC offset */ + rtw_write32(rtwdev, 0x1c3c, path_sel + 0x8103); + rtw8822c_dac_cal_rf_mode(rtwdev, &ic, &qc); + rtw_dbg(rtwdev, RTW_DBG_RFK, + "[DACK] after: i=0x%08x, q=0x%08x\n", ic, qc); + if (ic >= 0x200) + ic = 0x400 - ic; + if (qc >= 0x200) + qc = 0x400 - qc; + if (ic < 5 && qc < 5) + break; + } + + /* ADCK step2 */ + rtw_write32(rtwdev, 0x1c3c, 0x00000003); + rtw_write32(rtwdev, base_addr + 0x0c, 0x10000260); + rtw_write32(rtwdev, base_addr + 0x10, 0x02d508c4); + + /* release pull low switch on IQ path */ + rtw_write_rf(rtwdev, path, 0x8f, BIT(13), 0x1); +} + +static void rtw8822c_dac_cal_step1(struct rtw_dev *rtwdev, u8 path) +{ + u32 base_addr; + + base_addr = rtw8822c_get_path_base_addr(path); + + rtw_write32(rtwdev, base_addr + 0x0c, 0xdff00220); + if (path == RF_PATH_A) { + rtw_write32(rtwdev, base_addr + 0x60, 0xf0040ff0); + rtw_write32(rtwdev, 0x1c38, 0xffffffff); + } + rtw_write32(rtwdev, base_addr + 0x10, 0x02d508c5); + rtw_write32(rtwdev, 0x9b4, 0xdb66db00); + rtw_write32(rtwdev, base_addr + 0xb0, 0x0a11fb88); + rtw_write32(rtwdev, base_addr + 0xbc, 0x0008ff81); + rtw_write32(rtwdev, base_addr + 0xc0, 0x0003d208); + rtw_write32(rtwdev, base_addr + 0xcc, 0x0a11fb88); + rtw_write32(rtwdev, base_addr + 0xd8, 0x0008ff81); + rtw_write32(rtwdev, base_addr + 0xdc, 0x0003d208); + rtw_write32(rtwdev, base_addr + 0xb8, 0x60000000); + mdelay(2); + rtw_write32(rtwdev, base_addr + 0xbc, 0x000aff8d); + mdelay(2); + rtw_write32(rtwdev, base_addr + 0xb0, 0x0a11fb89); + rtw_write32(rtwdev, base_addr + 0xcc, 0x0a11fb89); + mdelay(1); + rtw_write32(rtwdev, base_addr + 0xb8, 0x62000000); + mdelay(20); + rtw_write32(rtwdev, base_addr + 0xd4, 0x62000000); + mdelay(20); + rtw_write32(rtwdev, base_addr + 0xb8, 0x02000000); + mdelay(20); + rtw_write32(rtwdev, base_addr + 0xbc, 0x0008ff87); + rtw_write32(rtwdev, 0x9b4, 0xdb6db600); + rtw_write32(rtwdev, base_addr + 0x10, 0x02d508c5); + rtw_write32(rtwdev, base_addr + 0xbc, 0x0008ff87); + rtw_write32(rtwdev, base_addr + 0x60, 0xf0000000); +} + +static void rtw8822c_dac_cal_step2(struct rtw_dev *rtwdev, + u8 path, u32 *ic_out, u32 *qc_out) +{ + u32 base_addr; + u32 ic, qc, ic_in, qc_in; + + base_addr = rtw8822c_get_path_base_addr(path); + rtw_write32_mask(rtwdev, base_addr + 0xbc, 0xf0000000, 0x0); + rtw_write32_mask(rtwdev, base_addr + 0xc0, 0xf, 0x8); + rtw_write32_mask(rtwdev, base_addr + 0xd8, 0xf0000000, 0x0); + rtw_write32_mask(rtwdev, base_addr + 0xdc, 0xf, 0x8); + + rtw_write32(rtwdev, 0x1b00, 0x00000008); + rtw_write8(rtwdev, 0x1bcc, 0x03f); + rtw_write32(rtwdev, base_addr + 0x0c, 0xdff00220); + rtw_write32(rtwdev, base_addr + 0x10, 0x02d508c5); + rtw_write32(rtwdev, 0x1c3c, 0x00088103); + + rtw8822c_dac_cal_rf_mode(rtwdev, &ic_in, &qc_in); + ic = ic_in; + qc = qc_in; + + /* compensation value */ + if (ic != 0x0) + ic = 0x400 - ic; + if (qc != 0x0) + qc = 0x400 - qc; + if (ic < 0x300) { + ic = ic * 2 * 6 / 5; + ic = ic + 0x80; + } else { + ic = (0x400 - ic) * 2 * 6 / 5; + ic = 0x7f - ic; + } + if (qc < 0x300) { + qc = qc * 2 * 6 / 5; + qc = qc + 0x80; + } else { + qc = (0x400 - qc) * 2 * 6 / 5; + qc = 0x7f - qc; + } + + *ic_out = ic; + *qc_out = qc; + + rtw_dbg(rtwdev, RTW_DBG_RFK, "[DACK] before i=0x%x, q=0x%x\n", ic_in, qc_in); + rtw_dbg(rtwdev, RTW_DBG_RFK, "[DACK] after i=0x%x, q=0x%x\n", ic, qc); +} + +static void rtw8822c_dac_cal_step3(struct rtw_dev *rtwdev, u8 path, + u32 adc_ic, u32 adc_qc, + u32 *ic_in, u32 *qc_in, + u32 *i_out, u32 *q_out) +{ + u32 base_addr; + u32 ic, qc; + u32 temp; + + base_addr = rtw8822c_get_path_base_addr(path); + ic = *ic_in; + qc = *qc_in; + + rtw_write32(rtwdev, base_addr + 0x0c, 0xdff00220); + rtw_write32(rtwdev, base_addr + 0x10, 0x02d508c5); + rtw_write32(rtwdev, 0x9b4, 0xdb66db00); + rtw_write32(rtwdev, base_addr + 0xb0, 0x0a11fb88); + rtw_write32(rtwdev, base_addr + 0xbc, 0xc008ff81); + rtw_write32(rtwdev, base_addr + 0xc0, 0x0003d208); + rtw_write32_mask(rtwdev, base_addr + 0xbc, 0xf0000000, ic & 0xf); + rtw_write32_mask(rtwdev, base_addr + 0xc0, 0xf, (ic & 0xf0) >> 4); + rtw_write32(rtwdev, base_addr + 0xcc, 0x0a11fb88); + rtw_write32(rtwdev, base_addr + 0xd8, 0xe008ff81); + rtw_write32(rtwdev, base_addr + 0xdc, 0x0003d208); + rtw_write32_mask(rtwdev, base_addr + 0xd8, 0xf0000000, qc & 0xf); + rtw_write32_mask(rtwdev, base_addr + 0xdc, 0xf, (qc & 0xf0) >> 4); + rtw_write32(rtwdev, base_addr + 0xb8, 0x60000000); + mdelay(2); + rtw_write32_mask(rtwdev, base_addr + 0xbc, 0xe, 0x6); + mdelay(2); + rtw_write32(rtwdev, base_addr + 0xb0, 0x0a11fb89); + rtw_write32(rtwdev, base_addr + 0xcc, 0x0a11fb89); + mdelay(1); + rtw_write32(rtwdev, base_addr + 0xb8, 0x62000000); + mdelay(20); + rtw_write32(rtwdev, base_addr + 0xd4, 0x62000000); + mdelay(20); + rtw_write32(rtwdev, base_addr + 0xb8, 0x02000000); + mdelay(20); + rtw_write32_mask(rtwdev, base_addr + 0xbc, 0xe, 0x3); + rtw_write32(rtwdev, 0x9b4, 0xdb6db600); + + /* check DAC DC offset */ + temp = ((adc_ic + 0x10) & 0x3ff) | (((adc_qc + 0x10) & 0x3ff) << 10); + rtw_write32(rtwdev, base_addr + 0x68, temp); + rtw_write32(rtwdev, base_addr + 0x10, 0x02d508c5); + rtw_write32(rtwdev, base_addr + 0x60, 0xf0000000); + rtw8822c_dac_cal_rf_mode(rtwdev, &ic, &qc); + if (ic >= 0x10) + ic = ic - 0x10; + else + ic = 0x400 - (0x10 - ic); + + if (qc >= 0x10) + qc = qc - 0x10; + else + qc = 0x400 - (0x10 - qc); + + *i_out = ic; + *q_out = qc; + + if (ic >= 0x200) + ic = 0x400 - ic; + if (qc >= 0x200) + qc = 0x400 - qc; + + *ic_in = ic; + *qc_in = qc; + + rtw_dbg(rtwdev, RTW_DBG_RFK, + "[DACK] after DACK i=0x%x, q=0x%x\n", *i_out, *q_out); +} + +static void rtw8822c_dac_cal_step4(struct rtw_dev *rtwdev, u8 path) +{ + u32 base_addr = rtw8822c_get_path_base_addr(path); + + rtw_write32(rtwdev, base_addr + 0x68, 0x0); + rtw_write32(rtwdev, base_addr + 0x10, 0x02d508c4); + rtw_write32_mask(rtwdev, base_addr + 0xbc, 0x1, 0x0); + rtw_write32_mask(rtwdev, base_addr + 0x30, BIT(30), 0x1); +} + +static void rtw8822c_rf_dac_cal(struct rtw_dev *rtwdev) +{ + struct rtw_backup_info backup_rf[DACK_RF_8822C * DACK_PATH_8822C]; + struct rtw_backup_info backup[DACK_REG_8822C]; + u32 ic = 0, qc = 0, i; + u32 i_a = 0x0, q_a = 0x0, i_b = 0x0, q_b = 0x0; + u32 ic_a = 0x0, qc_a = 0x0, ic_b = 0x0, qc_b = 0x0; + u32 adc_ic_a = 0x0, adc_qc_a = 0x0, adc_ic_b = 0x0, adc_qc_b = 0x0; + + rtw8822c_dac_backup_reg(rtwdev, backup, backup_rf); + + rtw8822c_dac_bb_setting(rtwdev); + + /* path-A */ + rtw8822c_dac_cal_adc(rtwdev, RF_PATH_A, &adc_ic_a, &adc_qc_a); + for (i = 0; i < 10; i++) { + rtw8822c_dac_cal_step1(rtwdev, RF_PATH_A); + rtw8822c_dac_cal_step2(rtwdev, RF_PATH_A, &ic, &qc); + ic_a = ic; + qc_a = qc; + + rtw8822c_dac_cal_step3(rtwdev, RF_PATH_A, adc_ic_a, adc_qc_a, + &ic, &qc, &i_a, &q_a); + + if (ic < 5 && qc < 5) + break; + } + rtw8822c_dac_cal_step4(rtwdev, RF_PATH_A); + + /* path-B */ + rtw8822c_dac_cal_adc(rtwdev, RF_PATH_B, &adc_ic_b, &adc_qc_b); + for (i = 0; i < 10; i++) { + rtw8822c_dac_cal_step1(rtwdev, RF_PATH_B); + rtw8822c_dac_cal_step2(rtwdev, RF_PATH_B, &ic, &qc); + ic_b = ic; + qc_b = qc; + + rtw8822c_dac_cal_step3(rtwdev, RF_PATH_B, adc_ic_b, adc_qc_b, + &ic, &qc, &i_b, &q_b); + + if (ic < 5 && qc < 5) + break; + } + rtw8822c_dac_cal_step4(rtwdev, RF_PATH_B); + + rtw_write32(rtwdev, 0x1b00, 0x00000008); + rtw_write32_mask(rtwdev, 0x4130, BIT(30), 0x1); + rtw_write8(rtwdev, 0x1bcc, 0x0); + rtw_write32(rtwdev, 0x1b00, 0x0000000a); + rtw_write8(rtwdev, 0x1bcc, 0x0); + + rtw8822c_dac_restore_reg(rtwdev, backup, backup_rf); + + rtw_dbg(rtwdev, RTW_DBG_RFK, "[DACK] path A: ic=0x%x, qc=0x%x\n", ic_a, qc_a); + rtw_dbg(rtwdev, RTW_DBG_RFK, "[DACK] path B: ic=0x%x, qc=0x%x\n", ic_b, qc_b); + rtw_dbg(rtwdev, RTW_DBG_RFK, "[DACK] path A: i=0x%x, q=0x%x\n", i_a, q_a); + rtw_dbg(rtwdev, RTW_DBG_RFK, "[DACK] path B: i=0x%x, q=0x%x\n", i_b, q_b); +} + +static void rtw8822c_rf_x2_check(struct rtw_dev *rtwdev) +{ + u8 x2k_busy; + + mdelay(1); + x2k_busy = rtw_read_rf(rtwdev, RF_PATH_A, 0xb8, BIT(15)); + if (x2k_busy == 1) { + rtw_write_rf(rtwdev, RF_PATH_A, 0xb8, RFREG_MASK, 0xC4440); + rtw_write_rf(rtwdev, RF_PATH_A, 0xba, RFREG_MASK, 0x6840D); + rtw_write_rf(rtwdev, RF_PATH_A, 0xb8, RFREG_MASK, 0x80440); + mdelay(1); + } +} + +static void rtw8822c_rf_init(struct rtw_dev *rtwdev) +{ + rtw8822c_rf_dac_cal(rtwdev); + rtw8822c_rf_x2_check(rtwdev); +} + +static void rtw8822c_phy_set_param(struct rtw_dev *rtwdev) +{ + struct rtw_dm_info *dm_info = &rtwdev->dm_info; + struct rtw_hal *hal = &rtwdev->hal; + u8 crystal_cap; + u8 cck_gi_u_bnd_msb = 0; + u8 cck_gi_u_bnd_lsb = 0; + u8 cck_gi_l_bnd_msb = 0; + u8 cck_gi_l_bnd_lsb = 0; + bool is_tx2_path; + + /* power on BB/RF domain */ + rtw_write8_set(rtwdev, REG_SYS_FUNC_EN, + BIT_FEN_BB_GLB_RST | BIT_FEN_BB_RSTB); + rtw_write8_set(rtwdev, REG_RF_CTRL, + BIT_RF_EN | BIT_RF_RSTB | BIT_RF_SDM_RSTB); + rtw_write32_set(rtwdev, REG_WLRF1, BIT_WLRF1_BBRF_EN); + + /* pre init before header files config */ + rtw8822c_header_file_init(rtwdev, true); + + rtw_phy_load_tables(rtwdev); + + crystal_cap = rtwdev->efuse.crystal_cap & 0x7f; + rtw_write32_mask(rtwdev, REG_ANAPAR_XTAL_0, 0xfffc00, + crystal_cap | (crystal_cap << 7)); + + /* post init after header files config */ + rtw8822c_header_file_init(rtwdev, false); + + is_tx2_path = false; + rtw8822c_config_trx_mode(rtwdev, hal->antenna_tx, hal->antenna_rx, + is_tx2_path); + rtw_phy_init(rtwdev); + + cck_gi_u_bnd_msb = (u8)rtw_read32_mask(rtwdev, 0x1a98, 0xc000); + cck_gi_u_bnd_lsb = (u8)rtw_read32_mask(rtwdev, 0x1aa8, 0xf0000); + cck_gi_l_bnd_msb = (u8)rtw_read32_mask(rtwdev, 0x1a98, 0xc0); + cck_gi_l_bnd_lsb = (u8)rtw_read32_mask(rtwdev, 0x1a70, 0x0f000000); + + dm_info->cck_gi_u_bnd = ((cck_gi_u_bnd_msb << 4) | (cck_gi_u_bnd_lsb)); + dm_info->cck_gi_l_bnd = ((cck_gi_l_bnd_msb << 4) | (cck_gi_l_bnd_lsb)); + + rtw8822c_rf_init(rtwdev); + /* wifi path controller */ + rtw_write32_mask(rtwdev, 0x70, 0xff000000, 0x0e); + rtw_write32_mask(rtwdev, 0x1704, 0xffffffff, 0x7700); + rtw_write32_mask(rtwdev, 0x1700, 0xffffffff, 0xc00f0038); + rtw_write32_mask(rtwdev, 0x6c0, 0xffffffff, 0xaaaaaaaa); + rtw_write32_mask(rtwdev, 0x6c4, 0xffffffff, 0xaaaaaaaa); +} + +#define WLAN_TXQ_RPT_EN 0x1F +#define WLAN_SLOT_TIME 0x09 +#define WLAN_PIFS_TIME 0x1C +#define WLAN_SIFS_CCK_CONT_TX 0x0A +#define WLAN_SIFS_OFDM_CONT_TX 0x0E +#define WLAN_SIFS_CCK_TRX 0x0A +#define WLAN_SIFS_OFDM_TRX 0x10 +#define WLAN_NAV_MAX 0xC8 +#define WLAN_RDG_NAV 0x05 +#define WLAN_TXOP_NAV 0x1B +#define WLAN_CCK_RX_TSF 0x30 +#define WLAN_OFDM_RX_TSF 0x30 +#define WLAN_TBTT_PROHIBIT 0x04 /* unit : 32us */ +#define WLAN_TBTT_HOLD_TIME 0x064 /* unit : 32us */ +#define WLAN_DRV_EARLY_INT 0x04 +#define WLAN_BCN_CTRL_CLT0 0x10 +#define WLAN_BCN_DMA_TIME 0x02 +#define WLAN_BCN_MAX_ERR 0xFF +#define WLAN_SIFS_CCK_DUR_TUNE 0x0A +#define WLAN_SIFS_OFDM_DUR_TUNE 0x10 +#define WLAN_SIFS_CCK_CTX 0x0A +#define WLAN_SIFS_CCK_IRX 0x0A +#define WLAN_SIFS_OFDM_CTX 0x0E +#define WLAN_SIFS_OFDM_IRX 0x0E +#define WLAN_EIFS_DUR_TUNE 0x40 +#define WLAN_EDCA_VO_PARAM 0x002FA226 +#define WLAN_EDCA_VI_PARAM 0x005EA328 +#define WLAN_EDCA_BE_PARAM 0x005EA42B +#define WLAN_EDCA_BK_PARAM 0x0000A44F + +#define WLAN_RX_FILTER0 0xFFFFFFFF +#define WLAN_RX_FILTER2 0xFFFF +#define WLAN_RCR_CFG 0xE400220E +#define WLAN_RXPKT_MAX_SZ 12288 +#define WLAN_RXPKT_MAX_SZ_512 (WLAN_RXPKT_MAX_SZ >> 9) + +#define WLAN_AMPDU_MAX_TIME 0x70 +#define WLAN_RTS_LEN_TH 0xFF +#define WLAN_RTS_TX_TIME_TH 0x08 +#define WLAN_MAX_AGG_PKT_LIMIT 0x20 +#define WLAN_RTS_MAX_AGG_PKT_LIMIT 0x20 +#define WLAN_PRE_TXCNT_TIME_TH 0x1E0 +#define FAST_EDCA_VO_TH 0x06 +#define FAST_EDCA_VI_TH 0x06 +#define FAST_EDCA_BE_TH 0x06 +#define FAST_EDCA_BK_TH 0x06 +#define WLAN_BAR_RETRY_LIMIT 0x01 +#define WLAN_BAR_ACK_TYPE 0x05 +#define WLAN_RA_TRY_RATE_AGG_LIMIT 0x08 +#define WLAN_RESP_TXRATE 0x84 +#define WLAN_ACK_TO 0x21 +#define WLAN_ACK_TO_CCK 0x6A +#define WLAN_DATA_RATE_FB_CNT_1_4 0x01000000 +#define WLAN_DATA_RATE_FB_CNT_5_8 0x08070504 +#define WLAN_RTS_RATE_FB_CNT_5_8 0x08070504 +#define WLAN_DATA_RATE_FB_RATE0 0xFE01F010 +#define WLAN_DATA_RATE_FB_RATE0_H 0x40000000 +#define WLAN_RTS_RATE_FB_RATE1 0x003FF010 +#define WLAN_RTS_RATE_FB_RATE1_H 0x40000000 +#define WLAN_RTS_RATE_FB_RATE4 0x0600F010 +#define WLAN_RTS_RATE_FB_RATE4_H 0x400003E0 +#define WLAN_RTS_RATE_FB_RATE5 0x0600F015 +#define WLAN_RTS_RATE_FB_RATE5_H 0x000000E0 + +#define WLAN_TX_FUNC_CFG1 0x30 +#define WLAN_TX_FUNC_CFG2 0x30 +#define WLAN_MAC_OPT_NORM_FUNC1 0x98 +#define WLAN_MAC_OPT_LB_FUNC1 0x80 +#define WLAN_MAC_OPT_FUNC2 0x30810041 + +#define WLAN_SIFS_CFG (WLAN_SIFS_CCK_CONT_TX | \ + (WLAN_SIFS_OFDM_CONT_TX << BIT_SHIFT_SIFS_OFDM_CTX) | \ + (WLAN_SIFS_CCK_TRX << BIT_SHIFT_SIFS_CCK_TRX) | \ + (WLAN_SIFS_OFDM_TRX << BIT_SHIFT_SIFS_OFDM_TRX)) + +#define WLAN_SIFS_DUR_TUNE (WLAN_SIFS_CCK_DUR_TUNE | \ + (WLAN_SIFS_OFDM_DUR_TUNE << 8)) + +#define WLAN_TBTT_TIME (WLAN_TBTT_PROHIBIT |\ + (WLAN_TBTT_HOLD_TIME << BIT_SHIFT_TBTT_HOLD_TIME_AP)) + +#define WLAN_NAV_CFG (WLAN_RDG_NAV | (WLAN_TXOP_NAV << 16)) +#define WLAN_RX_TSF_CFG (WLAN_CCK_RX_TSF | (WLAN_OFDM_RX_TSF) << 8) + +#define MAC_CLK_SPEED 80 /* 80M */ +#define EFUSE_PCB_INFO_OFFSET 0xCA + +static int rtw8822c_mac_init(struct rtw_dev *rtwdev) +{ + u8 value8; + u16 value16; + u32 value32; + u16 pre_txcnt; + + /* txq control */ + value8 = rtw_read8(rtwdev, REG_FWHW_TXQ_CTRL); + value8 |= (BIT(7) & ~BIT(1) & ~BIT(2)); + rtw_write8(rtwdev, REG_FWHW_TXQ_CTRL, value8); + rtw_write8(rtwdev, REG_FWHW_TXQ_CTRL + 1, WLAN_TXQ_RPT_EN); + /* sifs control */ + rtw_write16(rtwdev, REG_SPEC_SIFS, WLAN_SIFS_DUR_TUNE); + rtw_write32(rtwdev, REG_SIFS, WLAN_SIFS_CFG); + rtw_write16(rtwdev, REG_RESP_SIFS_CCK, + WLAN_SIFS_CCK_CTX | WLAN_SIFS_CCK_IRX << 8); + rtw_write16(rtwdev, REG_RESP_SIFS_OFDM, + WLAN_SIFS_OFDM_CTX | WLAN_SIFS_OFDM_IRX << 8); + /* rate fallback control */ + rtw_write32(rtwdev, REG_DARFRC, WLAN_DATA_RATE_FB_CNT_1_4); + rtw_write32(rtwdev, REG_DARFRCH, WLAN_DATA_RATE_FB_CNT_5_8); + rtw_write32(rtwdev, REG_RARFRCH, WLAN_RTS_RATE_FB_CNT_5_8); + rtw_write32(rtwdev, REG_ARFR0, WLAN_DATA_RATE_FB_RATE0); + rtw_write32(rtwdev, REG_ARFRH0, WLAN_DATA_RATE_FB_RATE0_H); + rtw_write32(rtwdev, REG_ARFR1_V1, WLAN_RTS_RATE_FB_RATE1); + rtw_write32(rtwdev, REG_ARFRH1_V1, WLAN_RTS_RATE_FB_RATE1_H); + rtw_write32(rtwdev, REG_ARFR4, WLAN_RTS_RATE_FB_RATE4); + rtw_write32(rtwdev, REG_ARFRH4, WLAN_RTS_RATE_FB_RATE4_H); + rtw_write32(rtwdev, REG_ARFR5, WLAN_RTS_RATE_FB_RATE5); + rtw_write32(rtwdev, REG_ARFRH5, WLAN_RTS_RATE_FB_RATE5_H); + /* protocol configuration */ + rtw_write8(rtwdev, REG_AMPDU_MAX_TIME_V1, WLAN_AMPDU_MAX_TIME); + rtw_write8_set(rtwdev, REG_TX_HANG_CTRL, BIT_EN_EOF_V1); + pre_txcnt = WLAN_PRE_TXCNT_TIME_TH | BIT_EN_PRECNT; + rtw_write8(rtwdev, REG_PRECNT_CTRL, (u8)(pre_txcnt & 0xFF)); + rtw_write8(rtwdev, REG_PRECNT_CTRL + 1, (u8)(pre_txcnt >> 8)); + value32 = WLAN_RTS_LEN_TH | (WLAN_RTS_TX_TIME_TH << 8) | + (WLAN_MAX_AGG_PKT_LIMIT << 16) | + (WLAN_RTS_MAX_AGG_PKT_LIMIT << 24); + rtw_write32(rtwdev, REG_PROT_MODE_CTRL, value32); + rtw_write16(rtwdev, REG_BAR_MODE_CTRL + 2, + WLAN_BAR_RETRY_LIMIT | WLAN_RA_TRY_RATE_AGG_LIMIT << 8); + rtw_write8(rtwdev, REG_FAST_EDCA_VOVI_SETTING, FAST_EDCA_VO_TH); + rtw_write8(rtwdev, REG_FAST_EDCA_VOVI_SETTING + 2, FAST_EDCA_VI_TH); + rtw_write8(rtwdev, REG_FAST_EDCA_BEBK_SETTING, FAST_EDCA_BE_TH); + rtw_write8(rtwdev, REG_FAST_EDCA_BEBK_SETTING + 2, FAST_EDCA_BK_TH); + /* close BA parser */ + rtw_write8_clr(rtwdev, REG_LIFETIME_EN, BIT_BA_PARSER_EN); + rtw_write32_clr(rtwdev, REG_RRSR, BITS_RRSR_RSC); + + /* EDCA configuration */ + rtw_write32(rtwdev, REG_EDCA_VO_PARAM, WLAN_EDCA_VO_PARAM); + rtw_write32(rtwdev, REG_EDCA_VI_PARAM, WLAN_EDCA_VI_PARAM); + rtw_write32(rtwdev, REG_EDCA_BE_PARAM, WLAN_EDCA_BE_PARAM); + rtw_write32(rtwdev, REG_EDCA_BK_PARAM, WLAN_EDCA_BK_PARAM); + rtw_write8(rtwdev, REG_PIFS, WLAN_PIFS_TIME); + rtw_write8_clr(rtwdev, REG_TX_PTCL_CTRL + 1, BIT_SIFS_BK_EN >> 8); + rtw_write8_set(rtwdev, REG_RD_CTRL + 1, + (BIT_DIS_TXOP_CFE | BIT_DIS_LSIG_CFE | + BIT_DIS_STBC_CFE) >> 8); + + /* MAC clock configuration */ + rtw_write32_clr(rtwdev, REG_AFE_CTRL1, BIT_MAC_CLK_SEL); + rtw_write8(rtwdev, REG_USTIME_TSF, MAC_CLK_SPEED); + rtw_write8(rtwdev, REG_USTIME_EDCA, MAC_CLK_SPEED); + + rtw_write8_set(rtwdev, REG_MISC_CTRL, + BIT_EN_FREE_CNT | BIT_DIS_SECOND_CCA); + rtw_write8_clr(rtwdev, REG_TIMER0_SRC_SEL, BIT_TSFT_SEL_TIMER0); + rtw_write16(rtwdev, REG_TXPAUSE, 0x0000); + rtw_write8(rtwdev, REG_SLOT, WLAN_SLOT_TIME); + rtw_write32(rtwdev, REG_RD_NAV_NXT, WLAN_NAV_CFG); + rtw_write16(rtwdev, REG_RXTSF_OFFSET_CCK, WLAN_RX_TSF_CFG); + /* Set beacon cotnrol - enable TSF and other related functions */ + rtw_write8_set(rtwdev, REG_BCN_CTRL, BIT_EN_BCN_FUNCTION); + /* Set send beacon related registers */ + rtw_write32(rtwdev, REG_TBTT_PROHIBIT, WLAN_TBTT_TIME); + rtw_write8(rtwdev, REG_DRVERLYINT, WLAN_DRV_EARLY_INT); + rtw_write8(rtwdev, REG_BCN_CTRL_CLINT0, WLAN_BCN_CTRL_CLT0); + rtw_write8(rtwdev, REG_BCNDMATIM, WLAN_BCN_DMA_TIME); + rtw_write8(rtwdev, REG_BCN_MAX_ERR, WLAN_BCN_MAX_ERR); + + /* WMAC configuration */ + rtw_write8(rtwdev, REG_BBPSF_CTRL + 2, WLAN_RESP_TXRATE); + rtw_write8(rtwdev, REG_ACKTO, WLAN_ACK_TO); + rtw_write8(rtwdev, REG_ACKTO_CCK, WLAN_ACK_TO_CCK); + rtw_write16(rtwdev, REG_EIFS, WLAN_EIFS_DUR_TUNE); + rtw_write8(rtwdev, REG_NAV_CTRL + 2, WLAN_NAV_MAX); + rtw_write8(rtwdev, REG_WMAC_TRXPTCL_CTL_H + 2, WLAN_BAR_ACK_TYPE); + rtw_write32(rtwdev, REG_RXFLTMAP0, WLAN_RX_FILTER0); + rtw_write16(rtwdev, REG_RXFLTMAP2, WLAN_RX_FILTER2); + rtw_write32(rtwdev, REG_RCR, WLAN_RCR_CFG); + rtw_write8(rtwdev, REG_RX_PKT_LIMIT, WLAN_RXPKT_MAX_SZ_512); + rtw_write8(rtwdev, REG_TCR + 2, WLAN_TX_FUNC_CFG2); + rtw_write8(rtwdev, REG_TCR + 1, WLAN_TX_FUNC_CFG1); + rtw_write32_set(rtwdev, REG_GENERAL_OPTION, BIT_DUMMY_FCS_READY_MASK_EN); + rtw_write32(rtwdev, REG_WMAC_OPTION_FUNCTION + 8, WLAN_MAC_OPT_FUNC2); + rtw_write8(rtwdev, REG_WMAC_OPTION_FUNCTION_1, WLAN_MAC_OPT_NORM_FUNC1); + + /* init low power */ + value16 = rtw_read16(rtwdev, REG_RXPSF_CTRL + 2) & 0xF00F; + value16 |= (BIT_RXGCK_VHT_FIFOTHR(1) | BIT_RXGCK_HT_FIFOTHR(1) | + BIT_RXGCK_OFDM_FIFOTHR(1) | BIT_RXGCK_CCK_FIFOTHR(1)) >> 16; + rtw_write16(rtwdev, REG_RXPSF_CTRL + 2, value16); + value16 = 0; + value16 = BIT_SET_RXPSF_PKTLENTHR(value16, 1); + value16 |= BIT_RXPSF_CTRLEN | BIT_RXPSF_VHTCHKEN | BIT_RXPSF_HTCHKEN + | BIT_RXPSF_OFDMCHKEN | BIT_RXPSF_CCKCHKEN + | BIT_RXPSF_OFDMRST; + rtw_write16(rtwdev, REG_RXPSF_CTRL, value16); + rtw_write32(rtwdev, REG_RXPSF_TYPE_CTRL, 0xFFFFFFFF); + /* rx ignore configuration */ + value16 = rtw_read16(rtwdev, REG_RXPSF_CTRL); + value16 &= ~(BIT_RXPSF_MHCHKEN | BIT_RXPSF_CCKRST | + BIT_RXPSF_CONT_ERRCHKEN); + value16 = BIT_SET_RXPSF_ERRTHR(value16, 0x07); + rtw_write16(rtwdev, REG_RXPSF_CTRL, value16); + + return 0; +} + +static void rtw8822c_set_channel_rf(struct rtw_dev *rtwdev, u8 channel, u8 bw) +{ +#define RF18_BAND_MASK (BIT(16) | BIT(9) | BIT(8)) +#define RF18_BAND_2G (0) +#define RF18_BAND_5G (BIT(16) | BIT(8)) +#define RF18_CHANNEL_MASK (MASKBYTE0) +#define RF18_RFSI_MASK (BIT(18) | BIT(17)) +#define RF18_RFSI_GE_CH80 (BIT(17)) +#define RF18_RFSI_GT_CH140 (BIT(18)) +#define RF18_BW_MASK (BIT(13) | BIT(12)) +#define RF18_BW_20M (BIT(13) | BIT(12)) +#define RF18_BW_40M (BIT(13)) +#define RF18_BW_80M (BIT(12)) + + u32 rf_reg18 = 0; + u32 rf_rxbb = 0; + + rf_reg18 = rtw_read_rf(rtwdev, RF_PATH_A, 0x18, RFREG_MASK); + + rf_reg18 &= ~(RF18_BAND_MASK | RF18_CHANNEL_MASK | RF18_RFSI_MASK | + RF18_BW_MASK); + + rf_reg18 |= (channel <= 14 ? RF18_BAND_2G : RF18_BAND_5G); + rf_reg18 |= (channel & RF18_CHANNEL_MASK); + if (channel > 144) + rf_reg18 |= RF18_RFSI_GT_CH140; + else if (channel >= 80) + rf_reg18 |= RF18_RFSI_GE_CH80; + + switch (bw) { + case RTW_CHANNEL_WIDTH_5: + case RTW_CHANNEL_WIDTH_10: + case RTW_CHANNEL_WIDTH_20: + default: + rf_reg18 |= RF18_BW_20M; + rf_rxbb = 0x18; + break; + case RTW_CHANNEL_WIDTH_40: + /* RF bandwidth */ + rf_reg18 |= RF18_BW_40M; + rf_rxbb = 0x10; + break; + case RTW_CHANNEL_WIDTH_80: + rf_reg18 |= RF18_BW_80M; + rf_rxbb = 0x8; + break; + } + + rtw_write_rf(rtwdev, RF_PATH_A, RF_LUTWE2, 0x04, 0x01); + rtw_write_rf(rtwdev, RF_PATH_A, RF_LUTWA, 0x1f, 0x12); + rtw_write_rf(rtwdev, RF_PATH_A, RF_LUTWD0, 0xfffff, rf_rxbb); + rtw_write_rf(rtwdev, RF_PATH_A, RF_LUTWE2, 0x04, 0x00); + + rtw_write_rf(rtwdev, RF_PATH_B, RF_LUTWE2, 0x04, 0x01); + rtw_write_rf(rtwdev, RF_PATH_B, RF_LUTWA, 0x1f, 0x12); + rtw_write_rf(rtwdev, RF_PATH_B, RF_LUTWD0, 0xfffff, rf_rxbb); + rtw_write_rf(rtwdev, RF_PATH_B, RF_LUTWE2, 0x04, 0x00); + + rtw_write_rf(rtwdev, RF_PATH_A, RF_CFGCH, RFREG_MASK, rf_reg18); + rtw_write_rf(rtwdev, RF_PATH_B, RF_CFGCH, RFREG_MASK, rf_reg18); +} + +static void rtw8822c_toggle_igi(struct rtw_dev *rtwdev) +{ + u32 igi; + + igi = rtw_read32_mask(rtwdev, REG_RXIGI, 0x7f); + rtw_write32_mask(rtwdev, REG_RXIGI, 0x7f, igi - 2); + rtw_write32_mask(rtwdev, REG_RXIGI, 0x7f00, igi - 2); + rtw_write32_mask(rtwdev, REG_RXIGI, 0x7f, igi); + rtw_write32_mask(rtwdev, REG_RXIGI, 0x7f00, igi); +} + +static void rtw8822c_set_channel_bb(struct rtw_dev *rtwdev, u8 channel, u8 bw, + u8 primary_ch_idx) +{ + if (channel <= 14) { + rtw_write32_clr(rtwdev, REG_BGCTRL, BITS_RX_IQ_WEIGHT); + rtw_write32_mask(rtwdev, REG_RXCCKSEL, 0xf0000000, 0x8); + rtw_write32_set(rtwdev, REG_TXF4, BIT(20)); + rtw_write32_clr(rtwdev, REG_CCK_CHECK, BIT_CHECK_CCK_EN); + rtw_write32_clr(rtwdev, REG_CCKTXONLY, BIT_BB_CCK_CHECK_EN); + rtw_write32_mask(rtwdev, REG_CCAMSK, 0x3F000000, 0xF); + + rtw_write32_mask(rtwdev, REG_RXAGCCTL0, 0x1f0, 0x0); + rtw_write32_mask(rtwdev, REG_RXAGCCTL, 0x1f0, 0x0); + if (channel == 13 || channel == 14) + rtw_write32_mask(rtwdev, REG_SCOTRK, 0xfff, 0x969); + else if (channel == 11 || channel == 12) + rtw_write32_mask(rtwdev, REG_SCOTRK, 0xfff, 0x96a); + else + rtw_write32_mask(rtwdev, REG_SCOTRK, 0xfff, 0x9aa); + if (channel == 14) { + rtw_write32_mask(rtwdev, REG_TXF0, MASKHWORD, 0x3da0); + rtw_write32_mask(rtwdev, REG_TXF1, MASKDWORD, + 0x4962c931); + rtw_write32_mask(rtwdev, REG_TXF2, MASKLWORD, 0x6aa3); + rtw_write32_mask(rtwdev, REG_TXF3, MASKHWORD, 0xaa7b); + rtw_write32_mask(rtwdev, REG_TXF4, MASKLWORD, 0xf3d7); + rtw_write32_mask(rtwdev, REG_TXF5, MASKDWORD, 0x0); + rtw_write32_mask(rtwdev, REG_TXF6, MASKDWORD, + 0xff012455); + rtw_write32_mask(rtwdev, REG_TXF7, MASKDWORD, 0xffff); + } else { + rtw_write32_mask(rtwdev, REG_TXF0, MASKHWORD, 0x5284); + rtw_write32_mask(rtwdev, REG_TXF1, MASKDWORD, + 0x3e18fec8); + rtw_write32_mask(rtwdev, REG_TXF2, MASKLWORD, 0x0a88); + rtw_write32_mask(rtwdev, REG_TXF3, MASKHWORD, 0xacc4); + rtw_write32_mask(rtwdev, REG_TXF4, MASKLWORD, 0xc8b2); + rtw_write32_mask(rtwdev, REG_TXF5, MASKDWORD, + 0x00faf0de); + rtw_write32_mask(rtwdev, REG_TXF6, MASKDWORD, + 0x00122344); + rtw_write32_mask(rtwdev, REG_TXF7, MASKDWORD, + 0x0fffffff); + } + if (channel == 13) + rtw_write32_mask(rtwdev, REG_TXDFIR0, 0x70, 0x3); + else + rtw_write32_mask(rtwdev, REG_TXDFIR0, 0x70, 0x1); + } else if (channel > 35) { + rtw_write32_set(rtwdev, REG_CCKTXONLY, BIT_BB_CCK_CHECK_EN); + rtw_write32_set(rtwdev, REG_CCK_CHECK, BIT_CHECK_CCK_EN); + rtw_write32_set(rtwdev, REG_BGCTRL, BITS_RX_IQ_WEIGHT); + rtw_write32_clr(rtwdev, REG_TXF4, BIT(20)); + rtw_write32_mask(rtwdev, REG_RXCCKSEL, 0xf0000000, 0x0); + rtw_write32_mask(rtwdev, REG_CCAMSK, 0x3F000000, 0x22); + rtw_write32_mask(rtwdev, REG_TXDFIR0, 0x70, 0x3); + if (channel >= 36 && channel <= 64) { + rtw_write32_mask(rtwdev, REG_RXAGCCTL0, 0x1f0, 0x1); + rtw_write32_mask(rtwdev, REG_RXAGCCTL, 0x1f0, 0x1); + } else if (channel >= 100 && channel <= 144) { + rtw_write32_mask(rtwdev, REG_RXAGCCTL0, 0x1f0, 0x2); + rtw_write32_mask(rtwdev, REG_RXAGCCTL, 0x1f0, 0x2); + } else if (channel >= 149) { + rtw_write32_mask(rtwdev, REG_RXAGCCTL0, 0x1f0, 0x3); + rtw_write32_mask(rtwdev, REG_RXAGCCTL, 0x1f0, 0x3); + } + + if (channel >= 36 && channel <= 51) + rtw_write32_mask(rtwdev, REG_SCOTRK, 0xfff, 0x494); + else if (channel >= 52 && channel <= 55) + rtw_write32_mask(rtwdev, REG_SCOTRK, 0xfff, 0x493); + else if (channel >= 56 && channel <= 111) + rtw_write32_mask(rtwdev, REG_SCOTRK, 0xfff, 0x453); + else if (channel >= 112 && channel <= 119) + rtw_write32_mask(rtwdev, REG_SCOTRK, 0xfff, 0x452); + else if (channel >= 120 && channel <= 172) + rtw_write32_mask(rtwdev, REG_SCOTRK, 0xfff, 0x412); + else if (channel >= 173 && channel <= 177) + rtw_write32_mask(rtwdev, REG_SCOTRK, 0xfff, 0x411); + } + + switch (bw) { + case RTW_CHANNEL_WIDTH_20: + rtw_write32_mask(rtwdev, REG_DFIRBW, 0x3FF0, 0x19B); + rtw_write32_mask(rtwdev, REG_TXBWCTL, 0xf, 0x0); + rtw_write32_mask(rtwdev, REG_TXBWCTL, 0xffc0, 0x0); + rtw_write32_mask(rtwdev, REG_TXCLK, 0x700, 0x7); + rtw_write32_mask(rtwdev, REG_TXCLK, 0x700000, 0x6); + break; + case RTW_CHANNEL_WIDTH_40: + rtw_write32_mask(rtwdev, REG_CCKSB, BIT(4), + (primary_ch_idx == 1 ? 1 : 0)); + rtw_write32_mask(rtwdev, REG_TXBWCTL, 0xf, 0x5); + rtw_write32_mask(rtwdev, REG_TXBWCTL, 0xc0, 0x0); + rtw_write32_mask(rtwdev, REG_TXBWCTL, 0xff00, + (primary_ch_idx | (primary_ch_idx << 4))); + break; + case RTW_CHANNEL_WIDTH_80: + rtw_write32_mask(rtwdev, REG_TXBWCTL, 0xf, 0xa); + rtw_write32_mask(rtwdev, REG_TXBWCTL, 0xc0, 0x0); + rtw_write32_mask(rtwdev, REG_TXBWCTL, 0xff00, + (primary_ch_idx | (primary_ch_idx << 4))); + break; + case RTW_CHANNEL_WIDTH_5: + rtw_write32_mask(rtwdev, REG_DFIRBW, 0x3FF0, 0x2AB); + rtw_write32_mask(rtwdev, REG_TXBWCTL, 0xf, 0x0); + rtw_write32_mask(rtwdev, REG_TXBWCTL, 0xffc0, 0x1); + rtw_write32_mask(rtwdev, REG_TXCLK, 0x700, 0x4); + rtw_write32_mask(rtwdev, REG_TXCLK, 0x700000, 0x4); + break; + case RTW_CHANNEL_WIDTH_10: + rtw_write32_mask(rtwdev, REG_DFIRBW, 0x3FF0, 0x2AB); + rtw_write32_mask(rtwdev, REG_TXBWCTL, 0xf, 0x0); + rtw_write32_mask(rtwdev, REG_TXBWCTL, 0xffc0, 0x2); + rtw_write32_mask(rtwdev, REG_TXCLK, 0x700, 0x6); + rtw_write32_mask(rtwdev, REG_TXCLK, 0x700000, 0x5); + break; + } +} + +static void rtw8822c_set_channel(struct rtw_dev *rtwdev, u8 channel, u8 bw, + u8 primary_chan_idx) +{ + rtw8822c_set_channel_bb(rtwdev, channel, bw, primary_chan_idx); + rtw_set_channel_mac(rtwdev, channel, bw, primary_chan_idx); + rtw8822c_set_channel_rf(rtwdev, channel, bw); + rtw8822c_toggle_igi(rtwdev); +} + +static void rtw8822c_config_cck_rx_path(struct rtw_dev *rtwdev, u8 rx_path) +{ + if (rx_path == BB_PATH_A || rx_path == BB_PATH_B) { + rtw_write32_mask(rtwdev, REG_CCANRX, 0x00060000, 0x0); + rtw_write32_mask(rtwdev, REG_CCANRX, 0x00600000, 0x0); + } else if (rx_path == BB_PATH_AB) { + rtw_write32_mask(rtwdev, REG_CCANRX, 0x00600000, 0x1); + rtw_write32_mask(rtwdev, REG_CCANRX, 0x00060000, 0x1); + } + + if (rx_path == BB_PATH_A) + rtw_write32_mask(rtwdev, REG_RXCCKSEL, 0x0f000000, 0x0); + else if (rx_path == BB_PATH_B) + rtw_write32_mask(rtwdev, REG_RXCCKSEL, 0x0f000000, 0x5); + else if (rx_path == BB_PATH_AB) + rtw_write32_mask(rtwdev, REG_RXCCKSEL, 0x0f000000, 0x1); +} + +static void rtw8822c_config_ofdm_rx_path(struct rtw_dev *rtwdev, u8 rx_path) +{ + if (rx_path == BB_PATH_A || rx_path == BB_PATH_B) { + rtw_write32_mask(rtwdev, REG_RXFNCTL, 0x300, 0x0); + rtw_write32_mask(rtwdev, REG_RXFNCTL, 0x600000, 0x0); + rtw_write32_mask(rtwdev, REG_AGCSWSH, BIT(17), 0x0); + rtw_write32_mask(rtwdev, REG_ANTWTPD, BIT(20), 0x0); + rtw_write32_mask(rtwdev, REG_MRCM, BIT(24), 0x0); + } else if (rx_path == BB_PATH_AB) { + rtw_write32_mask(rtwdev, REG_RXFNCTL, 0x300, 0x1); + rtw_write32_mask(rtwdev, REG_RXFNCTL, 0x600000, 0x1); + rtw_write32_mask(rtwdev, REG_AGCSWSH, BIT(17), 0x1); + rtw_write32_mask(rtwdev, REG_ANTWTPD, BIT(20), 0x1); + rtw_write32_mask(rtwdev, REG_MRCM, BIT(24), 0x1); + } + + rtw_write32_mask(rtwdev, 0x824, 0x0f000000, rx_path); + rtw_write32_mask(rtwdev, 0x824, 0x000f0000, rx_path); +} + +static void rtw8822c_config_rx_path(struct rtw_dev *rtwdev, u8 rx_path) +{ + rtw8822c_config_cck_rx_path(rtwdev, rx_path); + rtw8822c_config_ofdm_rx_path(rtwdev, rx_path); +} + +static void rtw8822c_config_cck_tx_path(struct rtw_dev *rtwdev, u8 tx_path, + bool is_tx2_path) +{ + if (tx_path == BB_PATH_A) { + rtw_write32_mask(rtwdev, REG_RXCCKSEL, 0xf0000000, 0x8); + } else if (tx_path == BB_PATH_B) { + rtw_write32_mask(rtwdev, REG_RXCCKSEL, 0xf0000000, 0x4); + } else { + if (is_tx2_path) + rtw_write32_mask(rtwdev, REG_RXCCKSEL, 0xf0000000, 0xc); + else + rtw_write32_mask(rtwdev, REG_RXCCKSEL, 0xf0000000, 0x8); + } +} + +static void rtw8822c_config_ofdm_tx_path(struct rtw_dev *rtwdev, u8 tx_path, + bool is_tx2_path) +{ + if (tx_path == BB_PATH_A) { + rtw_write32_mask(rtwdev, REG_ANTMAP0, 0xff, 0x11); + rtw_write32_mask(rtwdev, REG_TXLGMAP, 0xff, 0x0); + } else if (tx_path == BB_PATH_B) { + rtw_write32_mask(rtwdev, REG_ANTMAP0, 0xff, 0x12); + rtw_write32_mask(rtwdev, REG_TXLGMAP, 0xff, 0x0); + } else { + if (is_tx2_path) { + rtw_write32_mask(rtwdev, REG_ANTMAP0, 0xff, 0x33); + rtw_write32_mask(rtwdev, REG_TXLGMAP, 0xffff, 0x0404); + } else { + rtw_write32_mask(rtwdev, REG_ANTMAP0, 0xff, 0x31); + rtw_write32_mask(rtwdev, REG_TXLGMAP, 0xffff, 0x0400); + } + } +} + +static void rtw8822c_config_tx_path(struct rtw_dev *rtwdev, u8 tx_path, + bool is_tx2_path) +{ + rtw8822c_config_cck_tx_path(rtwdev, tx_path, is_tx2_path); + rtw8822c_config_ofdm_tx_path(rtwdev, tx_path, is_tx2_path); +} + +static void rtw8822c_config_trx_mode(struct rtw_dev *rtwdev, u8 tx_path, + u8 rx_path, bool is_tx2_path) +{ + if ((tx_path | rx_path) & BB_PATH_A) + rtw_write32_mask(rtwdev, REG_ORITXCODE, MASK20BITS, 0x33312); + else + rtw_write32_mask(rtwdev, REG_ORITXCODE, MASK20BITS, 0x11111); + if ((tx_path | rx_path) & BB_PATH_B) + rtw_write32_mask(rtwdev, REG_ORITXCODE2, MASK20BITS, 0x33312); + else + rtw_write32_mask(rtwdev, REG_ORITXCODE2, MASK20BITS, 0x11111); + + rtw8822c_config_rx_path(rtwdev, rx_path); + rtw8822c_config_tx_path(rtwdev, tx_path, is_tx2_path); + + rtw8822c_toggle_igi(rtwdev); +} + +static void query_phy_status_page0(struct rtw_dev *rtwdev, u8 *phy_status, + struct rtw_rx_pkt_stat *pkt_stat) +{ + struct rtw_dm_info *dm_info = &rtwdev->dm_info; + u8 l_bnd, u_bnd; + u8 gain_a, gain_b; + s8 rx_power[RTW_RF_PATH_MAX]; + s8 min_rx_power = -120; + + rx_power[RF_PATH_A] = GET_PHY_STAT_P0_PWDB_A(phy_status); + rx_power[RF_PATH_B] = GET_PHY_STAT_P0_PWDB_B(phy_status); + l_bnd = dm_info->cck_gi_l_bnd; + u_bnd = dm_info->cck_gi_u_bnd; + gain_a = GET_PHY_STAT_P0_GAIN_A(phy_status); + gain_b = GET_PHY_STAT_P0_GAIN_B(phy_status); + if (gain_a < l_bnd) + rx_power[RF_PATH_A] += (l_bnd - gain_a) << 1; + else if (gain_a > u_bnd) + rx_power[RF_PATH_A] -= (gain_a - u_bnd) << 1; + if (gain_b < l_bnd) + rx_power[RF_PATH_A] += (l_bnd - gain_b) << 1; + else if (gain_b > u_bnd) + rx_power[RF_PATH_A] -= (gain_b - u_bnd) << 1; + + rx_power[RF_PATH_A] -= 110; + rx_power[RF_PATH_B] -= 110; + + pkt_stat->rx_power[RF_PATH_A] = max3(rx_power[RF_PATH_A], + rx_power[RF_PATH_B], + min_rx_power); + pkt_stat->rssi = rtw_phy_rf_power_2_rssi(pkt_stat->rx_power, 1); + pkt_stat->bw = RTW_CHANNEL_WIDTH_20; + pkt_stat->signal_power = max(pkt_stat->rx_power[RF_PATH_A], + min_rx_power); +} + +static void query_phy_status_page1(struct rtw_dev *rtwdev, u8 *phy_status, + struct rtw_rx_pkt_stat *pkt_stat) +{ + u8 rxsc, bw; + s8 min_rx_power = -120; + + if (pkt_stat->rate > DESC_RATE11M && pkt_stat->rate < DESC_RATEMCS0) + rxsc = GET_PHY_STAT_P1_L_RXSC(phy_status); + else + rxsc = GET_PHY_STAT_P1_HT_RXSC(phy_status); + + if (rxsc >= 9 && rxsc <= 12) + bw = RTW_CHANNEL_WIDTH_40; + else if (rxsc >= 13) + bw = RTW_CHANNEL_WIDTH_80; + else + bw = RTW_CHANNEL_WIDTH_20; + + pkt_stat->rx_power[RF_PATH_A] = GET_PHY_STAT_P1_PWDB_A(phy_status) - 110; + pkt_stat->rx_power[RF_PATH_B] = GET_PHY_STAT_P1_PWDB_B(phy_status) - 110; + pkt_stat->rssi = rtw_phy_rf_power_2_rssi(pkt_stat->rx_power, 2); + pkt_stat->bw = bw; + pkt_stat->signal_power = max3(pkt_stat->rx_power[RF_PATH_A], + pkt_stat->rx_power[RF_PATH_B], + min_rx_power); +} + +static void query_phy_status(struct rtw_dev *rtwdev, u8 *phy_status, + struct rtw_rx_pkt_stat *pkt_stat) +{ + u8 page; + + page = *phy_status & 0xf; + + switch (page) { + case 0: + query_phy_status_page0(rtwdev, phy_status, pkt_stat); + break; + case 1: + query_phy_status_page1(rtwdev, phy_status, pkt_stat); + break; + default: + rtw_warn(rtwdev, "unused phy status page (%d)\n", page); + return; + } +} + +static void rtw8822c_query_rx_desc(struct rtw_dev *rtwdev, u8 *rx_desc, + struct rtw_rx_pkt_stat *pkt_stat, + struct ieee80211_rx_status *rx_status) +{ + struct ieee80211_hdr *hdr; + u32 desc_sz = rtwdev->chip->rx_pkt_desc_sz; + u8 *phy_status = NULL; + + memset(pkt_stat, 0, sizeof(*pkt_stat)); + + pkt_stat->phy_status = GET_RX_DESC_PHYST(rx_desc); + pkt_stat->icv_err = GET_RX_DESC_ICV_ERR(rx_desc); + pkt_stat->crc_err = GET_RX_DESC_CRC32(rx_desc); + pkt_stat->decrypted = !GET_RX_DESC_SWDEC(rx_desc); + pkt_stat->is_c2h = GET_RX_DESC_C2H(rx_desc); + pkt_stat->pkt_len = GET_RX_DESC_PKT_LEN(rx_desc); + pkt_stat->drv_info_sz = GET_RX_DESC_DRV_INFO_SIZE(rx_desc); + pkt_stat->shift = GET_RX_DESC_SHIFT(rx_desc); + pkt_stat->rate = GET_RX_DESC_RX_RATE(rx_desc); + pkt_stat->cam_id = GET_RX_DESC_MACID(rx_desc); + pkt_stat->ppdu_cnt = GET_RX_DESC_PPDU_CNT(rx_desc); + pkt_stat->tsf_low = GET_RX_DESC_TSFL(rx_desc); + + /* drv_info_sz is in unit of 8-bytes */ + pkt_stat->drv_info_sz *= 8; + + /* c2h cmd pkt's rx/phy status is not interested */ + if (pkt_stat->is_c2h) + return; + + hdr = (struct ieee80211_hdr *)(rx_desc + desc_sz + pkt_stat->shift + + pkt_stat->drv_info_sz); + if (pkt_stat->phy_status) { + phy_status = rx_desc + desc_sz + pkt_stat->shift; + query_phy_status(rtwdev, phy_status, pkt_stat); + } + + rtw_rx_fill_rx_status(rtwdev, pkt_stat, hdr, rx_status, phy_status); +} + +static void +rtw8822c_set_write_tx_power_ref(struct rtw_dev *rtwdev, u8 *tx_pwr_ref_cck, + u8 *tx_pwr_ref_ofdm) +{ + struct rtw_hal *hal = &rtwdev->hal; + u32 txref_cck[2] = {0x18a0, 0x41a0}; + u32 txref_ofdm[2] = {0x18e8, 0x41e8}; + u8 path; + + for (path = 0; path < hal->rf_path_num; path++) { + rtw_write32_mask(rtwdev, 0x1c90, BIT(15), 0); + rtw_write32_mask(rtwdev, txref_cck[path], 0x7f0000, + tx_pwr_ref_cck[path]); + } + for (path = 0; path < hal->rf_path_num; path++) { + rtw_write32_mask(rtwdev, 0x1c90, BIT(15), 0); + rtw_write32_mask(rtwdev, txref_ofdm[path], 0x1fc00, + tx_pwr_ref_ofdm[path]); + } +} + +static void rtw8822c_set_tx_power_diff(struct rtw_dev *rtwdev, u8 rate, + s8 *diff_idx) +{ + u32 offset_txagc = 0x3a00; + u8 rate_idx = rate & 0xfc; + u8 pwr_idx[4]; + u32 phy_pwr_idx; + int i; + + for (i = 0; i < 4; i++) + pwr_idx[i] = diff_idx[i] & 0x7f; + + phy_pwr_idx = pwr_idx[0] | + (pwr_idx[1] << 8) | + (pwr_idx[2] << 16) | + (pwr_idx[3] << 24); + + rtw_write32_mask(rtwdev, 0x1c90, BIT(15), 0x0); + rtw_write32_mask(rtwdev, offset_txagc + rate_idx, MASKDWORD, + phy_pwr_idx); +} + +static void rtw8822c_set_tx_power_index(struct rtw_dev *rtwdev) +{ + struct rtw_hal *hal = &rtwdev->hal; + u8 rs, rate, j; + u8 pwr_ref_cck[2] = {hal->tx_pwr_tbl[RF_PATH_A][DESC_RATE11M], + hal->tx_pwr_tbl[RF_PATH_B][DESC_RATE11M]}; + u8 pwr_ref_ofdm[2] = {hal->tx_pwr_tbl[RF_PATH_A][DESC_RATEMCS7], + hal->tx_pwr_tbl[RF_PATH_B][DESC_RATEMCS7]}; + s8 diff_a, diff_b; + u8 pwr_a, pwr_b; + s8 diff_idx[4]; + + rtw8822c_set_write_tx_power_ref(rtwdev, pwr_ref_cck, pwr_ref_ofdm); + for (rs = 0; rs < RTW_RATE_SECTION_MAX; rs++) { + for (j = 0; j < rtw_rate_size[rs]; j++) { + rate = rtw_rate_section[rs][j]; + pwr_a = hal->tx_pwr_tbl[RF_PATH_A][rate]; + pwr_b = hal->tx_pwr_tbl[RF_PATH_B][rate]; + if (rs == 0) { + diff_a = (s8)pwr_a - (s8)pwr_ref_cck[0]; + diff_b = (s8)pwr_b - (s8)pwr_ref_cck[1]; + } else { + diff_a = (s8)pwr_a - (s8)pwr_ref_ofdm[0]; + diff_b = (s8)pwr_b - (s8)pwr_ref_ofdm[1]; + } + diff_idx[rate % 4] = min(diff_a, diff_b); + if (rate % 4 == 3) + rtw8822c_set_tx_power_diff(rtwdev, rate - 3, + diff_idx); + } + } +} + +static void rtw8822c_cfg_ldo25(struct rtw_dev *rtwdev, bool enable) +{ + u8 ldo_pwr; + + ldo_pwr = rtw_read8(rtwdev, REG_ANAPARLDO_POW_MAC); + ldo_pwr = enable ? ldo_pwr | BIT_LDOE25_PON : ldo_pwr & ~BIT_LDOE25_PON; + rtw_write8(rtwdev, REG_ANAPARLDO_POW_MAC, ldo_pwr); +} + +static void rtw8822c_false_alarm_statistics(struct rtw_dev *rtwdev) +{ + struct rtw_dm_info *dm_info = &rtwdev->dm_info; + u32 cck_enable; + u32 cck_fa_cnt; + u32 ofdm_fa_cnt; + u32 ofdm_tx_counter; + + cck_enable = rtw_read32(rtwdev, REG_ENCCK) & BIT_CCK_BLK_EN; + cck_fa_cnt = rtw_read16(rtwdev, REG_CCK_FACNT); + ofdm_fa_cnt = rtw_read16(rtwdev, REG_OFDM_FACNT); + ofdm_tx_counter = rtw_read16(rtwdev, REG_OFDM_TXCNT); + ofdm_fa_cnt -= ofdm_tx_counter; + + dm_info->cck_fa_cnt = cck_fa_cnt; + dm_info->ofdm_fa_cnt = ofdm_fa_cnt; + dm_info->total_fa_cnt = ofdm_fa_cnt; + dm_info->total_fa_cnt += cck_enable ? cck_fa_cnt : 0; + + rtw_write32_mask(rtwdev, REG_CCANRX, BIT_CCK_FA_RST, 0); + rtw_write32_mask(rtwdev, REG_CCANRX, BIT_CCK_FA_RST, 2); + rtw_write32_mask(rtwdev, REG_CCANRX, BIT_OFDM_FA_RST, 0); + rtw_write32_mask(rtwdev, REG_CCANRX, BIT_OFDM_FA_RST, 2); + rtw_write32_set(rtwdev, REG_CNT_CTRL, BIT_ALL_CNT_RST); + rtw_write32_clr(rtwdev, REG_CNT_CTRL, BIT_ALL_CNT_RST); +} + +static void rtw8822c_do_iqk(struct rtw_dev *rtwdev) +{ +} + +static struct rtw_pwr_seq_cmd trans_carddis_to_cardemu_8822c[] = { + {0x0086, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_SDIO, + RTW_PWR_CMD_WRITE, BIT(0), 0}, + {0x0086, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_SDIO, + RTW_PWR_CMD_POLLING, BIT(1), BIT(1)}, + {0x002E, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(2), BIT(2)}, + {0x002D, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(0), 0}, + {0x007F, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(7), 0}, + {0x004A, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(0), 0}, + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(3) | BIT(4) | BIT(7), 0}, + {0xFFFF, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + 0, + RTW_PWR_CMD_END, 0, 0}, +}; + +static struct rtw_pwr_seq_cmd trans_cardemu_to_act_8822c[] = { + {0x0000, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK | RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(5), 0}, + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, (BIT(4) | BIT(3) | BIT(2)), 0}, + {0x0075, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_PCI_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(0), BIT(0)}, + {0x0006, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_POLLING, BIT(1), BIT(1)}, + {0x0075, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_PCI_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(0), 0}, + {0xFF1A, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, 0xFF, 0}, + {0x002E, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(3), 0}, + {0x0006, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(0), BIT(0)}, + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(7), 0}, + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, (BIT(4) | BIT(3)), 0}, + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(0), BIT(0)}, + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_POLLING, BIT(0), 0}, + {0x0074, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_PCI_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(5), BIT(5)}, + {0x0071, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_PCI_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(4), 0}, + {0x0062, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_PCI_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, (BIT(7) | BIT(6) | BIT(5)), + (BIT(7) | BIT(6) | BIT(5))}, + {0x0061, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_PCI_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, (BIT(7) | BIT(6) | BIT(5)), 0}, + {0x001F, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, (BIT(7) | BIT(6)), BIT(7)}, + {0x00EF, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, (BIT(7) | BIT(6)), BIT(7)}, + {0x1045, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(4), BIT(4)}, + {0x0010, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(2), BIT(2)}, + {0xFFFF, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + 0, + RTW_PWR_CMD_END, 0, 0}, +}; + +static struct rtw_pwr_seq_cmd trans_act_to_cardemu_8822c[] = { + {0x0093, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(3), 0}, + {0x001F, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, 0xFF, 0}, + {0x00EF, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, 0xFF, 0}, + {0x1045, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(4), 0}, + {0xFF1A, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, 0xFF, 0x30}, + {0x0049, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(1), 0}, + {0x0006, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(0), BIT(0)}, + {0x0002, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(1), 0}, + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(1), BIT(1)}, + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_POLLING, BIT(1), 0}, + {0x0000, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK | RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(5), BIT(5)}, + {0xFFFF, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + 0, + RTW_PWR_CMD_END, 0, 0}, +}; + +static struct rtw_pwr_seq_cmd trans_cardemu_to_carddis_8822c[] = { + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(7), BIT(7)}, + {0x0007, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK | RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, 0xFF, 0x00}, + {0x0067, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(5), 0}, + {0x004A, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(0), 0}, + {0x0081, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(7) | BIT(6), 0}, + {0x0090, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(1), 0}, + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_USB_MSK | RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(3) | BIT(4), BIT(3)}, + {0x0005, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_PCI_MSK, + RTW_PWR_ADDR_MAC, + RTW_PWR_CMD_WRITE, BIT(2), BIT(2)}, + {0x0086, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_SDIO_MSK, + RTW_PWR_ADDR_SDIO, + RTW_PWR_CMD_WRITE, BIT(0), BIT(0)}, + {0xFFFF, + RTW_PWR_CUT_ALL_MSK, + RTW_PWR_INTF_ALL_MSK, + 0, + RTW_PWR_CMD_END, 0, 0}, +}; + +static struct rtw_pwr_seq_cmd *card_enable_flow_8822c[] = { + trans_carddis_to_cardemu_8822c, + trans_cardemu_to_act_8822c, + NULL +}; + +static struct rtw_pwr_seq_cmd *card_disable_flow_8822c[] = { + trans_act_to_cardemu_8822c, + trans_cardemu_to_carddis_8822c, + NULL +}; + +static struct rtw_intf_phy_para usb2_param_8822c[] = { + {0xFFFF, 0x00, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_ALL, + RTW_INTF_PHY_PLATFORM_ALL}, +}; + +static struct rtw_intf_phy_para usb3_param_8822c[] = { + {0xFFFF, 0x0000, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_ALL, + RTW_INTF_PHY_PLATFORM_ALL}, +}; + +static struct rtw_intf_phy_para pcie_gen1_param_8822c[] = { + {0xFFFF, 0x0000, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_ALL, + RTW_INTF_PHY_PLATFORM_ALL}, +}; + +static struct rtw_intf_phy_para pcie_gen2_param_8822c[] = { + {0xFFFF, 0x0000, + RTW_IP_SEL_PHY, + RTW_INTF_PHY_CUT_ALL, + RTW_INTF_PHY_PLATFORM_ALL}, +}; + +static struct rtw_intf_phy_para_table phy_para_table_8822c = { + .usb2_para = usb2_param_8822c, + .usb3_para = usb3_param_8822c, + .gen1_para = pcie_gen1_param_8822c, + .gen2_para = pcie_gen2_param_8822c, + .n_usb2_para = ARRAY_SIZE(usb2_param_8822c), + .n_usb3_para = ARRAY_SIZE(usb2_param_8822c), + .n_gen1_para = ARRAY_SIZE(pcie_gen1_param_8822c), + .n_gen2_para = ARRAY_SIZE(pcie_gen2_param_8822c), +}; + +static const struct rtw_rfe_def rtw8822c_rfe_defs[] = { + [0] = RTW_DEF_RFE(8822c, 0, 0), + [1] = RTW_DEF_RFE(8822c, 0, 0), + [2] = RTW_DEF_RFE(8822c, 0, 0), +}; + +static struct rtw_hw_reg rtw8822c_dig[] = { + [0] = { .addr = 0x1d70, .mask = 0x7f }, + [1] = { .addr = 0x1d70, .mask = 0x7f00 }, +}; + +static struct rtw_page_table page_table_8822c[] = { + {64, 64, 64, 64, 1}, + {64, 64, 64, 64, 1}, + {64, 64, 0, 0, 1}, + {64, 64, 64, 0, 1}, + {64, 64, 64, 64, 1}, +}; + +static struct rtw_rqpn rqpn_table_8822c[] = { + {RTW_DMA_MAPPING_NORMAL, RTW_DMA_MAPPING_NORMAL, + RTW_DMA_MAPPING_LOW, RTW_DMA_MAPPING_LOW, + RTW_DMA_MAPPING_EXTRA, RTW_DMA_MAPPING_HIGH}, + {RTW_DMA_MAPPING_NORMAL, RTW_DMA_MAPPING_NORMAL, + RTW_DMA_MAPPING_LOW, RTW_DMA_MAPPING_LOW, + RTW_DMA_MAPPING_EXTRA, RTW_DMA_MAPPING_HIGH}, + {RTW_DMA_MAPPING_NORMAL, RTW_DMA_MAPPING_NORMAL, + RTW_DMA_MAPPING_NORMAL, RTW_DMA_MAPPING_HIGH, + RTW_DMA_MAPPING_HIGH, RTW_DMA_MAPPING_HIGH}, + {RTW_DMA_MAPPING_NORMAL, RTW_DMA_MAPPING_NORMAL, + RTW_DMA_MAPPING_LOW, RTW_DMA_MAPPING_LOW, + RTW_DMA_MAPPING_HIGH, RTW_DMA_MAPPING_HIGH}, + {RTW_DMA_MAPPING_NORMAL, RTW_DMA_MAPPING_NORMAL, + RTW_DMA_MAPPING_LOW, RTW_DMA_MAPPING_LOW, + RTW_DMA_MAPPING_EXTRA, RTW_DMA_MAPPING_HIGH}, +}; + +static struct rtw_chip_ops rtw8822c_ops = { + .phy_set_param = rtw8822c_phy_set_param, + .read_efuse = rtw8822c_read_efuse, + .query_rx_desc = rtw8822c_query_rx_desc, + .set_channel = rtw8822c_set_channel, + .mac_init = rtw8822c_mac_init, + .read_rf = rtw_phy_read_rf, + .write_rf = rtw_phy_write_rf_reg_mix, + .set_tx_power_index = rtw8822c_set_tx_power_index, + .cfg_ldo25 = rtw8822c_cfg_ldo25, + .false_alarm_statistics = rtw8822c_false_alarm_statistics, + .do_iqk = rtw8822c_do_iqk, +}; + +struct rtw_chip_info rtw8822c_hw_spec = { + .ops = &rtw8822c_ops, + .id = RTW_CHIP_TYPE_8822C, + .fw_name = "rtw88/rtw8822c_fw.bin", + .tx_pkt_desc_sz = 48, + .tx_buf_desc_sz = 16, + .rx_pkt_desc_sz = 24, + .rx_buf_desc_sz = 8, + .phy_efuse_size = 512, + .log_efuse_size = 768, + .ptct_efuse_size = 124, + .txff_size = 262144, + .rxff_size = 24576, + .txgi_factor = 2, + .is_pwr_by_rate_dec = false, + .max_power_index = 0x7f, + .csi_buf_pg_num = 50, + .band = RTW_BAND_2G | RTW_BAND_5G, + .page_size = 128, + .dig_min = 0x20, + .ht_supported = true, + .vht_supported = true, + .sys_func_en = 0xD8, + .pwr_on_seq = card_enable_flow_8822c, + .pwr_off_seq = card_disable_flow_8822c, + .page_table = page_table_8822c, + .rqpn_table = rqpn_table_8822c, + .intf_table = &phy_para_table_8822c, + .dig = rtw8822c_dig, + .rf_base_addr = {0x3c00, 0x4c00}, + .rf_sipi_addr = {0x1808, 0x4108}, + .mac_tbl = &rtw8822c_mac_tbl, + .agc_tbl = &rtw8822c_agc_tbl, + .bb_tbl = &rtw8822c_bb_tbl, + .rfk_init_tbl = &rtw8822c_array_mp_cal_init_tbl, + .rf_tbl = {&rtw8822c_rf_a_tbl, &rtw8822c_rf_b_tbl}, + .rfe_defs = rtw8822c_rfe_defs, + .rfe_defs_size = ARRAY_SIZE(rtw8822c_rfe_defs), +}; +EXPORT_SYMBOL(rtw8822c_hw_spec); + +MODULE_FIRMWARE("rtw88/rtw8822c_fw.bin"); diff --git a/drivers/net/wireless/realtek/rtw88/rtw8822c.h b/drivers/net/wireless/realtek/rtw88/rtw8822c.h new file mode 100644 index 0000000000000000000000000000000000000000..d3bd9850baa0b806530ca06a05e385f637d55a3e --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/rtw8822c.h @@ -0,0 +1,186 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#ifndef __RTW8822C_H__ +#define __RTW8822C_H__ + +#include + +struct rtw8822cu_efuse { + u8 res0[0x30]; /* 0x120 */ + u8 vid[2]; /* 0x150 */ + u8 pid[2]; + u8 res1[3]; + u8 mac_addr[ETH_ALEN]; /* 0x157 */ + u8 res2[0x3d]; +}; + +struct rtw8822ce_efuse { + u8 mac_addr[ETH_ALEN]; /* 0x120 */ + u8 vender_id[2]; + u8 device_id[2]; + u8 sub_vender_id[2]; + u8 sub_device_id[2]; + u8 pmc[2]; + u8 exp_device_cap[2]; + u8 msi_cap; + u8 ltr_cap; /* 0x133 */ + u8 exp_link_control[2]; + u8 link_cap[4]; + u8 link_control[2]; + u8 serial_number[8]; + u8 res0:2; /* 0x144 */ + u8 ltr_en:1; + u8 res1:2; + u8 obff:2; + u8 res2:3; + u8 obff_cap:2; + u8 res3:4; + u8 class_code[3]; + u8 res4; + u8 pci_pm_L1_2_supp:1; + u8 pci_pm_L1_1_supp:1; + u8 aspm_pm_L1_2_supp:1; + u8 aspm_pm_L1_1_supp:1; + u8 L1_pm_substates_supp:1; + u8 res5:3; + u8 port_common_mode_restore_time; + u8 port_t_power_on_scale:2; + u8 res6:1; + u8 port_t_power_on_value:5; + u8 res7; +}; + +struct rtw8822c_efuse { + __le16 rtl_id; + u8 res0[0x0e]; + + /* power index for four RF paths */ + struct rtw_txpwr_idx txpwr_idx_table[4]; + + u8 channel_plan; /* 0xb8 */ + u8 xtal_k; + u8 res1; + u8 iqk_lck; + u8 res2[5]; /* 0xbc */ + u8 rf_board_option; + u8 rf_feature_option; + u8 rf_bt_setting; + u8 eeprom_version; + u8 eeprom_customer_id; + u8 tx_bb_swing_setting_2g; + u8 tx_bb_swing_setting_5g; + u8 tx_pwr_calibrate_rate; + u8 rf_antenna_option; /* 0xc9 */ + u8 rfe_option; + u8 country_code[2]; + u8 res3[3]; + u8 path_a_thermal; /* 0xd0 */ + u8 path_b_thermal; + u8 res4[2]; + u8 rx_gain_gap_2g_ofdm; + u8 res5; + u8 rx_gain_gap_2g_cck; + u8 res6; + u8 rx_gain_gap_5gl; + u8 res7; + u8 rx_gain_gap_5gm; + u8 res8; + u8 rx_gain_gap_5gh; + u8 res9; + u8 res10[0x42]; + union { + struct rtw8822cu_efuse u; + struct rtw8822ce_efuse e; + }; +}; + +#define DACK_PATH_8822C 2 +#define DACK_REG_8822C 16 +#define DACK_RF_8822C 1 +#define DACK_SN_8822C 100 + +/* phy status page0 */ +#define GET_PHY_STAT_P0_PWDB_A(phy_stat) \ + le32_get_bits(*((__le32 *)(phy_stat) + 0x00), GENMASK(15, 8)) +#define GET_PHY_STAT_P0_PWDB_B(phy_stat) \ + le32_get_bits(*((__le32 *)(phy_stat) + 0x04), GENMASK(7, 0)) +#define GET_PHY_STAT_P0_GAIN_A(phy_stat) \ + le32_get_bits(*((__le32 *)(phy_stat) + 0x00), GENMASK(21, 16)) +#define GET_PHY_STAT_P0_GAIN_B(phy_stat) \ + le32_get_bits(*((__le32 *)(phy_stat) + 0x04), GENMASK(29, 24)) + +/* phy status page1 */ +#define GET_PHY_STAT_P1_PWDB_A(phy_stat) \ + le32_get_bits(*((__le32 *)(phy_stat) + 0x00), GENMASK(15, 8)) +#define GET_PHY_STAT_P1_PWDB_B(phy_stat) \ + le32_get_bits(*((__le32 *)(phy_stat) + 0x00), GENMASK(23, 16)) +#define GET_PHY_STAT_P1_L_RXSC(phy_stat) \ + le32_get_bits(*((__le32 *)(phy_stat) + 0x01), GENMASK(11, 8)) +#define GET_PHY_STAT_P1_HT_RXSC(phy_stat) \ + le32_get_bits(*((__le32 *)(phy_stat) + 0x01), GENMASK(15, 12)) + +#define REG_ANAPARLDO_POW_MAC 0x0029 +#define BIT_LDOE25_PON BIT(0) +#define REG_RRSR 0x0440 +#define BITS_RRSR_RSC (BIT(21) | BIT(22)) + +#define REG_TXDFIR0 0x808 +#define REG_DFIRBW 0x810 +#define REG_ANTMAP0 0x820 +#define REG_ANTMAP 0x824 +#define REG_DYMPRITH 0x86c +#define REG_DYMENTH0 0x870 +#define REG_DYMENTH 0x874 +#define REG_DYMTHMIN 0x8a4 +#define REG_TXBWCTL 0x9b0 +#define REG_TXCLK 0x9b4 +#define REG_SCOTRK 0xc30 +#define REG_MRCM 0xc38 +#define REG_AGCSWSH 0xc44 +#define REG_ANTWTPD 0xc54 +#define REG_ORITXCODE 0x1800 +#define REG_3WIRE 0x180c +#define BIT_3WIRE_TX_EN BIT(0) +#define BIT_3WIRE_RX_EN BIT(1) +#define BIT_3WIRE_PI_ON BIT(28) +#define REG_RXAGCCTL0 0x18ac +#define REG_CCKSB 0x1a00 +#define REG_RXCCKSEL 0x1a04 +#define REG_BGCTRL 0x1a14 +#define BITS_RX_IQ_WEIGHT (BIT(8) | BIT(9)) +#define REG_TXF0 0x1a20 +#define REG_TXF1 0x1a24 +#define REG_TXF2 0x1a28 +#define REG_CCANRX 0x1a2c +#define BIT_CCK_FA_RST (BIT(14) | BIT(15)) +#define BIT_OFDM_FA_RST (BIT(12) | BIT(13)) +#define REG_CCK_FACNT 0x1a5c +#define REG_CCKTXONLY 0x1a80 +#define BIT_BB_CCK_CHECK_EN BIT(18) +#define REG_TXF3 0x1a98 +#define REG_TXF4 0x1a9c +#define REG_TXF5 0x1aa0 +#define REG_TXF6 0x1aac +#define REG_TXF7 0x1ab0 +#define REG_TXANT 0x1c28 +#define REG_ENCCK 0x1c3c +#define BIT_CCK_BLK_EN BIT(1) +#define BIT_CCK_OFDM_BLK_EN (BIT(0) | BIT(1)) +#define REG_CCAMSK 0x1c80 +#define REG_RXFNCTL 0x1d30 +#define REG_RXIGI 0x1d70 +#define REG_ENFN 0x1e24 +#define REG_TXANTSEG 0x1e28 +#define REG_TXLGMAP 0x1e2c +#define REG_CCKPATH 0x1e5c +#define REG_CNT_CTRL 0x1eb4 +#define BIT_ALL_CNT_RST BIT(25) +#define REG_OFDM_FACNT 0x2d00 +#define REG_OFDM_TXCNT 0x2de0 +#define REG_ORITXCODE2 0x4100 +#define REG_3WIRE2 0x410c +#define REG_RXAGCCTL 0x41ac + +#endif diff --git a/drivers/net/wireless/realtek/rtw88/rtw8822c_table.c b/drivers/net/wireless/realtek/rtw88/rtw8822c_table.c new file mode 100644 index 0000000000000000000000000000000000000000..49044f510c6cf06ebe126ac87eee2d724361563e --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/rtw8822c_table.c @@ -0,0 +1,11753 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#include "main.h" +#include "phy.h" +#include "rtw8822c_table.h" + +static const u32 rtw8822c_mac[] = { +}; + +RTW_DECL_TABLE_PHY_COND(rtw8822c_mac, rtw_phy_cfg_mac); + +static const u32 rtw8822c_agc[] = { + 0x1D90, 0x300001FF, + 0x1D90, 0x300101FF, + 0x1D90, 0x300201FE, + 0x1D90, 0x300301FD, + 0x1D90, 0x300401FC, + 0x1D90, 0x300501FB, + 0x1D90, 0x300601FA, + 0x1D90, 0x300701F9, + 0x1D90, 0x300801F8, + 0x1D90, 0x300901F7, + 0x1D90, 0x300A01F6, + 0x1D90, 0x300B01F5, + 0x1D90, 0x300C01F4, + 0x1D90, 0x300D01F3, + 0x1D90, 0x300E01F2, + 0x1D90, 0x300F01F1, + 0x1D90, 0x301001F0, + 0x1D90, 0x301101EF, + 0x1D90, 0x301201EE, + 0x1D90, 0x301301ED, + 0x1D90, 0x301401EC, + 0x1D90, 0x301501EB, + 0x1D90, 0x30160192, + 0x1D90, 0x30170191, + 0x1D90, 0x30180190, + 0x1D90, 0x3019018F, + 0x1D90, 0x301A018E, + 0x1D90, 0x301B018D, + 0x1D90, 0x301C018C, + 0x1D90, 0x301D018B, + 0x1D90, 0x301E018A, + 0x1D90, 0x301F0189, + 0x1D90, 0x30200188, + 0x1D90, 0x30210187, + 0x1D90, 0x30220186, + 0x1D90, 0x30230185, + 0x1D90, 0x3024014B, + 0x1D90, 0x3025014A, + 0x1D90, 0x30260149, + 0x1D90, 0x30270148, + 0x1D90, 0x30280147, + 0x1D90, 0x30290146, + 0x1D90, 0x302A0145, + 0x1D90, 0x302B0144, + 0x1D90, 0x302C0143, + 0x1D90, 0x302D0142, + 0x1D90, 0x302E00C8, + 0x1D90, 0x302F00C7, + 0x1D90, 0x303000C6, + 0x1D90, 0x303100C5, + 0x1D90, 0x303200C4, + 0x1D90, 0x30330088, + 0x1D90, 0x30340087, + 0x1D90, 0x30350086, + 0x1D90, 0x30360045, + 0x1D90, 0x30370044, + 0x1D90, 0x30380043, + 0x1D90, 0x30390023, + 0x1D90, 0x303A0022, + 0x1D90, 0x303B0021, + 0x1D90, 0x303C0020, + 0x1D90, 0x303D0002, + 0x1D90, 0x303E0001, + 0x1D90, 0x303F0000, + 0x1D90, 0x304000FF, + 0x1D90, 0x304100FF, + 0x1D90, 0x304200FF, + 0x1D90, 0x304300FF, + 0x1D90, 0x304400FE, + 0x1D90, 0x304500FD, + 0x1D90, 0x304600FC, + 0x1D90, 0x304700FB, + 0x1D90, 0x304800FA, + 0x1D90, 0x304900F9, + 0x1D90, 0x304A00F8, + 0x1D90, 0x304B00F7, + 0x1D90, 0x304C00F6, + 0x1D90, 0x304D00F5, + 0x1D90, 0x304E00F4, + 0x1D90, 0x304F00F3, + 0x1D90, 0x305000F2, + 0x1D90, 0x305100F1, + 0x1D90, 0x305200F0, + 0x1D90, 0x305300EF, + 0x1D90, 0x305400EE, + 0x1D90, 0x305500ED, + 0x1D90, 0x305600EC, + 0x1D90, 0x305700EB, + 0x1D90, 0x305800EA, + 0x1D90, 0x305900E9, + 0x1D90, 0x305A00E8, + 0x1D90, 0x305B00E7, + 0x1D90, 0x305C00E6, + 0x1D90, 0x305D00C7, + 0x1D90, 0x305E00C6, + 0x1D90, 0x305F00C5, + 0x1D90, 0x306000C4, + 0x1D90, 0x306100C3, + 0x1D90, 0x306200C2, + 0x1D90, 0x306300A4, + 0x1D90, 0x306400A3, + 0x1D90, 0x306500A2, + 0x1D90, 0x30660086, + 0x1D90, 0x30670085, + 0x1D90, 0x30680084, + 0x1D90, 0x30690083, + 0x1D90, 0x306A0082, + 0x1D90, 0x306B0069, + 0x1D90, 0x306C0068, + 0x1D90, 0x306D0067, + 0x1D90, 0x306E0066, + 0x1D90, 0x306F0065, + 0x1D90, 0x30700064, + 0x1D90, 0x30710063, + 0x1D90, 0x30720044, + 0x1D90, 0x30730043, + 0x1D90, 0x30740042, + 0x1D90, 0x30750025, + 0x1D90, 0x30760024, + 0x1D90, 0x30770023, + 0x1D90, 0x30780022, + 0x1D90, 0x30790021, + 0x1D90, 0x307A0020, + 0x1D90, 0x307B0003, + 0x1D90, 0x307C0002, + 0x1D90, 0x307D0001, + 0x1D90, 0x307E0000, + 0x1D90, 0x307F0000, + 0x1D90, 0x308000FF, + 0x1D90, 0x308100FF, + 0x1D90, 0x308200FF, + 0x1D90, 0x308300FF, + 0x1D90, 0x308400FE, + 0x1D90, 0x308500FD, + 0x1D90, 0x308600FC, + 0x1D90, 0x308700FB, + 0x1D90, 0x308800FA, + 0x1D90, 0x308900F9, + 0x1D90, 0x308A00F8, + 0x1D90, 0x308B00F7, + 0x1D90, 0x308C00F6, + 0x1D90, 0x308D00F5, + 0x1D90, 0x308E00F4, + 0x1D90, 0x308F00F3, + 0x1D90, 0x309000F2, + 0x1D90, 0x309100F1, + 0x1D90, 0x309200F0, + 0x1D90, 0x309300EF, + 0x1D90, 0x309400EE, + 0x1D90, 0x309500ED, + 0x1D90, 0x309600EC, + 0x1D90, 0x309700EB, + 0x1D90, 0x309800EA, + 0x1D90, 0x309900E9, + 0x1D90, 0x309A00E8, + 0x1D90, 0x309B00E7, + 0x1D90, 0x309C00E6, + 0x1D90, 0x309D00C7, + 0x1D90, 0x309E00C6, + 0x1D90, 0x309F00C5, + 0x1D90, 0x30A000C4, + 0x1D90, 0x30A100C3, + 0x1D90, 0x30A200C2, + 0x1D90, 0x30A300A4, + 0x1D90, 0x30A400A3, + 0x1D90, 0x30A500A2, + 0x1D90, 0x30A60086, + 0x1D90, 0x30A70085, + 0x1D90, 0x30A80084, + 0x1D90, 0x30A90083, + 0x1D90, 0x30AA0082, + 0x1D90, 0x30AB0069, + 0x1D90, 0x30AC0068, + 0x1D90, 0x30AD0067, + 0x1D90, 0x30AE0066, + 0x1D90, 0x30AF0065, + 0x1D90, 0x30B00064, + 0x1D90, 0x30B10063, + 0x1D90, 0x30B20044, + 0x1D90, 0x30B30043, + 0x1D90, 0x30B40042, + 0x1D90, 0x30B50025, + 0x1D90, 0x30B60024, + 0x1D90, 0x30B70023, + 0x1D90, 0x30B80022, + 0x1D90, 0x30B90021, + 0x1D90, 0x30BA0020, + 0x1D90, 0x30BB0003, + 0x1D90, 0x30BC0002, + 0x1D90, 0x30BD0001, + 0x1D90, 0x30BE0000, + 0x1D90, 0x30BF0000, + 0x1D90, 0x30C000FF, + 0x1D90, 0x30C100FF, + 0x1D90, 0x30C200FF, + 0x1D90, 0x30C300FF, + 0x1D90, 0x30C400FE, + 0x1D90, 0x30C500FD, + 0x1D90, 0x30C600FC, + 0x1D90, 0x30C700FB, + 0x1D90, 0x30C800FA, + 0x1D90, 0x30C900F9, + 0x1D90, 0x30CA00F8, + 0x1D90, 0x30CB00F7, + 0x1D90, 0x30CC00F6, + 0x1D90, 0x30CD00F5, + 0x1D90, 0x30CE00F4, + 0x1D90, 0x30CF00F3, + 0x1D90, 0x30D000F2, + 0x1D90, 0x30D100F1, + 0x1D90, 0x30D200F0, + 0x1D90, 0x30D300EF, + 0x1D90, 0x30D400EE, + 0x1D90, 0x30D500ED, + 0x1D90, 0x30D600EC, + 0x1D90, 0x30D700EB, + 0x1D90, 0x30D800EA, + 0x1D90, 0x30D900E9, + 0x1D90, 0x30DA00E8, + 0x1D90, 0x30DB00E7, + 0x1D90, 0x30DC00E6, + 0x1D90, 0x30DD00C7, + 0x1D90, 0x30DE00C6, + 0x1D90, 0x30DF00C5, + 0x1D90, 0x30E000C4, + 0x1D90, 0x30E100C3, + 0x1D90, 0x30E200C2, + 0x1D90, 0x30E300A4, + 0x1D90, 0x30E400A3, + 0x1D90, 0x30E500A2, + 0x1D90, 0x30E60086, + 0x1D90, 0x30E70085, + 0x1D90, 0x30E80084, + 0x1D90, 0x30E90083, + 0x1D90, 0x30EA0082, + 0x1D90, 0x30EB0069, + 0x1D90, 0x30EC0068, + 0x1D90, 0x30ED0067, + 0x1D90, 0x30EE0066, + 0x1D90, 0x30EF0065, + 0x1D90, 0x30F00064, + 0x1D90, 0x30F10063, + 0x1D90, 0x30F20044, + 0x1D90, 0x30F30043, + 0x1D90, 0x30F40042, + 0x1D90, 0x30F50025, + 0x1D90, 0x30F60024, + 0x1D90, 0x30F70023, + 0x1D90, 0x30F80022, + 0x1D90, 0x30F90021, + 0x1D90, 0x30FA0020, + 0x1D90, 0x30FB0003, + 0x1D90, 0x30FC0002, + 0x1D90, 0x30FD0001, + 0x1D90, 0x30FE0000, + 0x1D90, 0x30FF0000, + 0x1D90, 0x310001FF, + 0x1D90, 0x310101FF, + 0x1D90, 0x310201FF, + 0x1D90, 0x310301FF, + 0x1D90, 0x310401FF, + 0x1D90, 0x310501FF, + 0x1D90, 0x310601FF, + 0x1D90, 0x310701FF, + 0x1D90, 0x310801FF, + 0x1D90, 0x310901FE, + 0x1D90, 0x310A01FD, + 0x1D90, 0x310B01FC, + 0x1D90, 0x310C01FB, + 0x1D90, 0x310D01FA, + 0x1D90, 0x310E01F9, + 0x1D90, 0x310F01F8, + 0x1D90, 0x311001F7, + 0x1D90, 0x311101F6, + 0x1D90, 0x311201F5, + 0x1D90, 0x311301F4, + 0x1D90, 0x311401F3, + 0x1D90, 0x311501F2, + 0x1D90, 0x311601F1, + 0x1D90, 0x311701F0, + 0x1D90, 0x311801EF, + 0x1D90, 0x311901EE, + 0x1D90, 0x311A01ED, + 0x1D90, 0x311B01EC, + 0x1D90, 0x311C01EB, + 0x1D90, 0x311D0192, + 0x1D90, 0x311E0191, + 0x1D90, 0x311F0190, + 0x1D90, 0x3120018F, + 0x1D90, 0x3121018E, + 0x1D90, 0x3122018D, + 0x1D90, 0x3123018C, + 0x1D90, 0x3124018B, + 0x1D90, 0x3125018A, + 0x1D90, 0x31260189, + 0x1D90, 0x31270188, + 0x1D90, 0x31280187, + 0x1D90, 0x31290186, + 0x1D90, 0x312A0185, + 0x1D90, 0x312B0149, + 0x1D90, 0x312C0148, + 0x1D90, 0x312D0147, + 0x1D90, 0x312E0146, + 0x1D90, 0x312F0145, + 0x1D90, 0x31300144, + 0x1D90, 0x31310143, + 0x1D90, 0x31320142, + 0x1D90, 0x31330141, + 0x1D90, 0x31340140, + 0x1D90, 0x313500C7, + 0x1D90, 0x313600C6, + 0x1D90, 0x313700C5, + 0x1D90, 0x313800C4, + 0x1D90, 0x313900C3, + 0x1D90, 0x313A0088, + 0x1D90, 0x313B0087, + 0x1D90, 0x313C0086, + 0x1D90, 0x313D0045, + 0x1D90, 0x313E0044, + 0x1D90, 0x313F0043, + 0x1D90, 0x314001FF, + 0x1D90, 0x314101FF, + 0x1D90, 0x314201FF, + 0x1D90, 0x314301FF, + 0x1D90, 0x314401FF, + 0x1D90, 0x314501FF, + 0x1D90, 0x314601FF, + 0x1D90, 0x314701FE, + 0x1D90, 0x314801FD, + 0x1D90, 0x314901FC, + 0x1D90, 0x314A01FB, + 0x1D90, 0x314B01FA, + 0x1D90, 0x314C01F9, + 0x1D90, 0x314D01F8, + 0x1D90, 0x314E01F7, + 0x1D90, 0x314F01F6, + 0x1D90, 0x315001F5, + 0x1D90, 0x315101F4, + 0x1D90, 0x315201F3, + 0x1D90, 0x315301F2, + 0x1D90, 0x315401F1, + 0x1D90, 0x315501F0, + 0x1D90, 0x315601EF, + 0x1D90, 0x315701EE, + 0x1D90, 0x315801ED, + 0x1D90, 0x315901EC, + 0x1D90, 0x315A01EB, + 0x1D90, 0x315B01EA, + 0x1D90, 0x315C01E9, + 0x1D90, 0x315D018F, + 0x1D90, 0x315E018E, + 0x1D90, 0x315F018D, + 0x1D90, 0x3160018C, + 0x1D90, 0x3161018B, + 0x1D90, 0x3162018A, + 0x1D90, 0x31630189, + 0x1D90, 0x31640188, + 0x1D90, 0x31650187, + 0x1D90, 0x31660186, + 0x1D90, 0x31670185, + 0x1D90, 0x31680184, + 0x1D90, 0x31690183, + 0x1D90, 0x316A0182, + 0x1D90, 0x316B0149, + 0x1D90, 0x316C0148, + 0x1D90, 0x316D0147, + 0x1D90, 0x316E0146, + 0x1D90, 0x316F0145, + 0x1D90, 0x31700144, + 0x1D90, 0x31710143, + 0x1D90, 0x31720142, + 0x1D90, 0x31730141, + 0x1D90, 0x31740140, + 0x1D90, 0x317500C7, + 0x1D90, 0x317600C6, + 0x1D90, 0x317700C5, + 0x1D90, 0x317800C4, + 0x1D90, 0x317900C3, + 0x1D90, 0x317A0088, + 0x1D90, 0x317B0087, + 0x1D90, 0x317C0086, + 0x1D90, 0x317D0045, + 0x1D90, 0x317E0044, + 0x1D90, 0x317F0043, + 0x1D90, 0x318001FE, + 0x1D90, 0x318101FD, + 0x1D90, 0x318201FC, + 0x1D90, 0x318301FB, + 0x1D90, 0x318401FA, + 0x1D90, 0x318501F9, + 0x1D90, 0x318601F8, + 0x1D90, 0x318701F7, + 0x1D90, 0x318801F6, + 0x1D90, 0x318901F5, + 0x1D90, 0x318A01F4, + 0x1D90, 0x318B01F3, + 0x1D90, 0x318C01F2, + 0x1D90, 0x318D01F1, + 0x1D90, 0x318E01F0, + 0x1D90, 0x318F01EF, + 0x1D90, 0x319001EE, + 0x1D90, 0x319101ED, + 0x1D90, 0x319201EC, + 0x1D90, 0x319301EB, + 0x1D90, 0x319401EA, + 0x1D90, 0x319501E9, + 0x1D90, 0x3196018F, + 0x1D90, 0x3197018E, + 0x1D90, 0x3198018D, + 0x1D90, 0x3199018C, + 0x1D90, 0x319A018B, + 0x1D90, 0x319B018A, + 0x1D90, 0x319C0189, + 0x1D90, 0x319D0188, + 0x1D90, 0x319E0187, + 0x1D90, 0x319F0186, + 0x1D90, 0x31A00185, + 0x1D90, 0x31A10184, + 0x1D90, 0x31A20183, + 0x1D90, 0x31A30182, + 0x1D90, 0x31A40149, + 0x1D90, 0x31A50148, + 0x1D90, 0x31A60147, + 0x1D90, 0x31A70146, + 0x1D90, 0x31A80145, + 0x1D90, 0x31A90144, + 0x1D90, 0x31AA0143, + 0x1D90, 0x31AB0142, + 0x1D90, 0x31AC0141, + 0x1D90, 0x31AD0140, + 0x1D90, 0x31AE00C7, + 0x1D90, 0x31AF00C6, + 0x1D90, 0x31B000C5, + 0x1D90, 0x31B100C4, + 0x1D90, 0x31B200C3, + 0x1D90, 0x31B30088, + 0x1D90, 0x31B40087, + 0x1D90, 0x31B50086, + 0x1D90, 0x31B60045, + 0x1D90, 0x31B70044, + 0x1D90, 0x31B80043, + 0x1D90, 0x31B90023, + 0x1D90, 0x31BA0022, + 0x1D90, 0x31BB0021, + 0x1D90, 0x31BC0020, + 0x1D90, 0x31BD0002, + 0x1D90, 0x31BE0001, + 0x1D90, 0x31BF0000, + 0x1D70, 0x22222222, + 0x1D70, 0x20202020, +}; + +RTW_DECL_TABLE_PHY_COND(rtw8822c_agc, rtw_phy_cfg_agc); + +static const u32 rtw8822c_bb[] = { + 0x1D0C, 0x00410000, + 0x1C3C, 0x01038040, + 0x1C90, 0x00E49708, + 0x800, 0x00000000, + 0x804, 0xD6300000, + 0x808, 0x60956093, + 0x80C, 0x00000025, + 0x810, 0x11B019B0, + 0x814, 0x00904080, + 0x818, 0xC30056F1, + 0x81C, 0x00050000, + 0x820, 0x11111133, + 0x824, 0xC3C3CCC4, + 0x828, 0x30FB186C, + 0x82C, 0x185D6556, + 0x830, 0x1751145B, + 0x834, 0x776995D7, + 0x838, 0x74777A7D, + 0x83C, 0xF9AA9982, + 0x840, 0x89AA9ABB, + 0x844, 0x0DEEDDC1, + 0x848, 0xCDEEDEFF, + 0x84C, 0xFFFF5555, + 0x850, 0x6F7A727D, + 0x854, 0x6C776F7A, + 0x858, 0x6F7A6C77, + 0x85C, 0x69746974, + 0x860, 0x6F7A6C77, + 0x864, 0x6C776C77, + 0x868, 0x727D6F7A, + 0x86C, 0x69D7B196, + 0x870, 0x1A6D769B, + 0x874, 0x55823917, + 0x878, 0x00C025BD, + 0x87C, 0x4140557D, + 0x880, 0x9A1D9D47, + 0x884, 0x1DE7134F, + 0x888, 0x2857A857, + 0x88C, 0x520E8A24, + 0x890, 0x8F628C44, + 0x894, 0x72745F43, + 0x898, 0x03F02F0D, + 0x89C, 0x5DB6886F, + 0x8A0, 0x07DC309F, + 0x8A4, 0x09412495, + 0x8A8, 0x222222A9, + 0x8AC, 0x89628C44, + 0x8B0, 0x72745F43, + 0x8B4, 0x03F02F0D, + 0x8B8, 0x55B6886F, + 0x8BC, 0x07D0309F, + 0x8C0, 0x70404023, + 0x8C4, 0x00440001, + 0x8C8, 0x7A7A2E26, + 0x8CC, 0x25297777, + 0x8D0, 0x6CEB6DCE, + 0x8D4, 0x0005A632, + 0x8D8, 0x00000000, + 0x8DC, 0x00000000, + 0x8E0, 0x00000000, + 0x8E4, 0x00000000, + 0x8E8, 0x00000000, + 0x8EC, 0x00000000, + 0x8F0, 0x00000000, + 0x8F4, 0x00000000, + 0x8F8, 0x25239843, + 0x900, 0x00000000, + 0x904, 0x00000000, + 0x908, 0x000008CB, + 0x90C, 0x00000000, + 0x910, 0x00000000, + 0x914, 0x20000000, + 0x918, 0x20000000, + 0x91C, 0x20000000, + 0x920, 0x20000000, + 0x924, 0x00000000, + 0x928, 0x0000003A, + 0x92C, 0x0000003A, + 0x930, 0x0000003A, + 0x934, 0x0000003A, + 0x938, 0x0000000F, + 0x93C, 0x00000000, + 0x940, 0x4E1F3E81, + 0x944, 0x4E1F3E81, + 0x948, 0x4E1F3E81, + 0x94C, 0x4E1F3E81, + 0x950, 0x03020100, + 0x954, 0x07060504, + 0x958, 0x0B0A0908, + 0x95C, 0x0F0E0D0C, + 0x960, 0x13121110, + 0x964, 0x17161514, + 0x968, 0x03020100, + 0x96C, 0x07060504, + 0x970, 0x0B0A0908, + 0x974, 0x0F0E0D0C, + 0x978, 0x13121110, + 0x97C, 0x17161514, + 0x980, 0x03020100, + 0x984, 0x07060504, + 0x988, 0x0B0A0908, + 0x98C, 0x0F0E0D0C, + 0x990, 0x13121110, + 0x994, 0x17161514, + 0x998, 0x03020100, + 0x99C, 0x07060504, + 0x9A0, 0x0B0A0908, + 0x9A4, 0x0F0E0D0C, + 0x9A8, 0x13121110, + 0x9AC, 0x17161514, + 0x9B0, 0x00002200, + 0x9B4, 0xDB6FFF00, + 0x9B8, 0x00400064, + 0x9BC, 0x00000000, + 0x9C0, 0x01010101, + 0x9C4, 0x00640064, + 0x9C8, 0x00640064, + 0x9CC, 0x00007777, + 0x9D0, 0x00000000, + 0x9D4, 0x00000000, + 0x9D8, 0x00000000, + 0x9DC, 0x00000000, + 0x9E0, 0x00000000, + 0x9E4, 0x00000000, + 0x9E8, 0x00000000, + 0x9EC, 0x00000000, + 0x9F0, 0x100024E0, + 0x9F4, 0x00000000, + 0x9F8, 0x00000000, + 0xA00, 0x02001208, + 0xA04, 0x00000000, + 0xA08, 0x00000000, + 0xA0C, 0x00000000, + 0xA10, 0x00000000, + 0xA14, 0x00000000, + 0xA18, 0x00000000, + 0xA1C, 0x00000000, + 0xA20, 0xEB31B333, + 0xA24, 0x00275485, + 0xA28, 0x00166366, + 0xA2C, 0x00275485, + 0xA30, 0x00166366, + 0xA34, 0x00275485, + 0xA38, 0x00200400, + 0xA3C, 0x00200400, + 0xA40, 0xB35DC5BD, + 0xA44, 0x3033BEBD, + 0xA48, 0x2A521254, + 0xA4C, 0xA2733345, + 0xA50, 0x617BE003, + 0xA54, 0x50000968, + 0xA58, 0x00020000, + 0xA5C, 0x01000000, + 0xA60, 0x02000000, + 0xA64, 0x03000000, + 0xA68, 0x00020000, + 0xA6C, 0x00000000, + 0xA70, 0x00000000, + 0xA74, 0x00000000, + 0xA78, 0x00000000, + 0xA7C, 0x00000000, + 0xA80, 0x00000000, + 0xA84, 0x00000000, + 0xA88, 0x00000000, + 0xA8C, 0x00000000, + 0xA90, 0x00000000, + 0xA94, 0x00000000, + 0xA98, 0x00000000, + 0xA9C, 0x00000000, + 0xAA0, 0x00000000, + 0xAA4, 0x00000000, + 0xAA8, 0x00000000, + 0xAAC, 0x00000000, + 0xAB0, 0x00000000, + 0xAB4, 0x00000000, + 0xAB8, 0x00000000, + 0xABC, 0x00000000, + 0xAC0, 0x00000000, + 0xAC4, 0x00000000, + 0xAC8, 0x00000000, + 0xACC, 0x00000000, + 0xAD0, 0x00000000, + 0xAD4, 0x00000000, + 0xAD8, 0x00000000, + 0xADC, 0x00000000, + 0xAE0, 0x00000000, + 0xAE4, 0x00000000, + 0xAE8, 0x00000000, + 0xAEC, 0x00000000, + 0xAF0, 0x00000000, + 0xAF4, 0x00000000, + 0xAF8, 0x00000000, + 0xB00, 0x00000000, + 0xB04, 0x00000000, + 0xB08, 0x00000000, + 0xB0C, 0x00000000, + 0xB10, 0x00000000, + 0xB14, 0x00000000, + 0xB18, 0x00000000, + 0xB1C, 0x00000000, + 0xB20, 0x00000000, + 0xB24, 0x00000000, + 0xB28, 0x00000000, + 0xB2C, 0x00000000, + 0xB30, 0x00000000, + 0xB34, 0x00000000, + 0xB38, 0x00000000, + 0xB3C, 0x00000000, + 0xB40, 0x00000000, + 0xB44, 0x00000000, + 0xB48, 0x00000000, + 0xB4C, 0x00000000, + 0xB50, 0x00000000, + 0xB54, 0x00000000, + 0xB58, 0x00060100, + 0xB5C, 0x00000000, + 0xB60, 0x00000000, + 0xB64, 0x00000000, + 0xB68, 0x00000000, + 0xB6C, 0x00000000, + 0xB70, 0x00000000, + 0xB74, 0x00000000, + 0xB78, 0x00000000, + 0xB7C, 0x00000000, + 0xB80, 0x00000000, + 0xB84, 0x00000000, + 0xB88, 0x00000000, + 0xB8C, 0x00000000, + 0xB90, 0x00000000, + 0xB94, 0x00000000, + 0xB98, 0x00000000, + 0xB9C, 0x00000000, + 0xBA0, 0x00000000, + 0xBA4, 0x00000000, + 0xBA8, 0x00000000, + 0xBAC, 0x00000000, + 0xBB0, 0x00000000, + 0xBB4, 0x00000000, + 0xBB8, 0x00000000, + 0xBBC, 0x00000000, + 0xBC0, 0x00000000, + 0xBC4, 0x00000000, + 0xBC8, 0x00000000, + 0xBCC, 0x00000000, + 0xBD0, 0x00000000, + 0xBD4, 0x00000000, + 0xBD8, 0x00000000, + 0xBDC, 0x00000000, + 0xBE0, 0x00000000, + 0xBE4, 0x00000000, + 0xBE8, 0x00000000, + 0xBEC, 0x00000000, + 0xBF0, 0x00000000, + 0xBF4, 0x00000000, + 0xBF8, 0x00000000, + 0xC00, 0x1C8BA0D6, + 0xC04, 0x00000001, + 0xC08, 0x00000000, + 0xC0C, 0x02F1D8B7, + 0xC10, 0x000000B0, + 0xC14, 0x0000D891, + 0xC18, 0x00087672, + 0xC1C, 0x15260000, + 0xC20, 0x00000000, + 0xC24, 0x40600000, + 0xC28, 0x06400F76, + 0xC2C, 0xE30020E1, + 0xC30, 0x140C9494, + 0xC34, 0x00A04946, + 0xC38, 0x011D4820, + 0xC3C, 0x168DB61B, + 0xC40, 0x009C50F8, + 0xC44, 0x2013BAD1, + 0xC48, 0xFFFFF7CC, + 0xC4C, 0xA000FFFF, + 0xC50, 0x20D0F800, + 0xC54, 0x941A0200, + 0xC58, 0x18380111, + 0xC5C, 0x006E01B8, + 0xC60, 0x2CA5555B, + 0xC64, 0x0210005F, + 0xC68, 0x039A5300, + 0xC6C, 0x0265C2BA, + 0xC70, 0x000CEB21, + 0xC74, 0x0E149CA1, + 0xC78, 0x1AB4956B, + 0xC7C, 0x00000ABF, + 0xC80, 0xC02A8799, + 0xC84, 0x06C636C6, + 0xC88, 0x08090202, + 0xC8C, 0x00204048, + 0xC90, 0x00F85F85, + 0xC94, 0x00000F85, + 0xC98, 0x58385858, + 0xC9C, 0x18382838, + 0xCA0, 0x00002838, + 0xCA4, 0x3A253A3A, + 0xCA8, 0x10251A25, + 0xCAC, 0x00001025, + 0xCB0, 0x3A133A3A, + 0xCB4, 0x08130D13, + 0xCB8, 0x00000813, + 0xCBC, 0x001F1066, + 0xCC0, 0x88A00400, + 0xCC4, 0x00200400, + 0xCC8, 0x0B200400, + 0xCCC, 0x00600400, + 0xCD0, 0x00000092, + 0xCD4, 0x22220000, + 0xCD8, 0x22222222, + 0xCDC, 0x22222222, + 0xCE0, 0x22222222, + 0xCE4, 0x22222222, + 0xCE8, 0x00002222, + 0xCEC, 0x00000000, + 0xCF0, 0x00000000, + 0xCF4, 0x00000000, + 0xCF8, 0x00000000, + 0xD00, 0x1083A10A, + 0xD04, 0x0EC42948, + 0xD08, 0x10852108, + 0xD0C, 0x0CC41D08, + 0xD10, 0x108620EC, + 0xD14, 0x0CA42108, + 0xD18, 0x107620E8, + 0xD1C, 0x0E742108, + 0xD20, 0x0E8618C8, + 0xD24, 0x00000108, + 0xD28, 0x288C224C, + 0xD2C, 0x11C6320C, + 0xD30, 0x30CEBD98, + 0xD34, 0x10C31908, + 0xD38, 0x310A318C, + 0xD3C, 0x18C41D08, + 0xD40, 0x28CC4190, + 0xD44, 0x19062108, + 0xD48, 0x294A5A17, + 0xD4C, 0x00000108, + 0xD50, 0x10A3A908, + 0xD54, 0x10842148, + 0xD58, 0x14C5314A, + 0xD5C, 0x1086258C, + 0xD60, 0x10A42948, + 0xD64, 0x10842108, + 0xD68, 0x08C42108, + 0xD6C, 0x10842148, + 0xD70, 0x08822084, + 0xD74, 0x10841D04, + 0xD78, 0x08421088, + 0xD7C, 0x1083A104, + 0xD80, 0x10842108, + 0xD84, 0x1085294A, + 0xD88, 0x08822104, + 0xD8C, 0x10852948, + 0xD90, 0x08421084, + 0xD94, 0x10852104, + 0xD98, 0x08421084, + 0xD9C, 0x10863184, + 0xDA0, 0x1083B10A, + 0xDA4, 0x10842148, + 0xDA8, 0x1984718C, + 0xDAC, 0x108C33AF, + 0xDB0, 0x00000000, + 0xDB4, 0x00000000, + 0xDB8, 0x00000000, + 0xDBC, 0x00000000, + 0xDC0, 0x00000000, + 0xDC4, 0x00000000, + 0xDC8, 0x00000000, + 0xDCC, 0x00000000, + 0xDD0, 0x00000000, + 0xDD4, 0x00000000, + 0xDD8, 0x00000000, + 0xDDC, 0x00000000, + 0xDE0, 0x00000000, + 0xDE4, 0x00000000, + 0xDE8, 0x00000000, + 0xDEC, 0x00000000, + 0xDF0, 0x00000000, + 0xDF4, 0x00000000, + 0xDF8, 0x00000000, + 0x1800, 0x00033312, + 0x1804, 0x00033312, + 0x180C, 0x17F40060, + 0x1810, 0x62F508C4, + 0x1814, 0x506AA5B4, + 0x1818, 0x000014FF, + 0x181C, 0x00000000, + 0x1820, 0x02D508CC, + 0x1824, 0x506AA5B4, + 0x1828, 0x000004FD, + 0x182C, 0x00000000, + 0x1834, 0x00000000, + 0x1838, 0x20000000, + 0x183C, 0x00000000, + 0x1840, 0x00000000, + 0x1844, 0x00000000, + 0x1848, 0x00000000, + 0x184C, 0x00000000, + 0x1850, 0x00000000, + 0x1854, 0x00000000, + 0x1858, 0x00000000, + 0x185C, 0x00000000, + 0x1860, 0xF0040FF8, + 0x1864, 0x7F000000, + 0x1868, 0x00000000, + 0x186C, 0x0000FF00, + 0x1870, 0x00000000, + 0x1874, 0x00000000, + 0x1878, 0x00000000, + 0x187C, 0x00000000, + 0x1880, 0x00000000, + 0x1884, 0x02B00000, + 0x1888, 0x00000000, + 0x188C, 0x00000000, + 0x1890, 0x00000000, + 0x1894, 0x00000000, + 0x1898, 0x00000000, + 0x18A0, 0x00510000, + 0x18A4, 0x183C1F7F, + 0x18A8, 0x0A02C99A, + 0x18AC, 0x00004200, + 0x18B0, 0x0809FB08, + 0x18B0, 0x0809FB09, + 0x18B4, 0x00000000, + 0x18B8, 0x00000000, + 0x18BC, 0x00C3FF80, + 0x18C0, 0x0002D100, + 0x18C4, 0x00000004, + 0x18C8, 0x001FFFE0, + 0x18CC, 0x0809FB08, + 0x18CC, 0x0809FB09, + 0x18D0, 0x00000000, + 0x18D4, 0x00000000, + 0x18D8, 0x00C3FF80, + 0x18DC, 0x0002D100, + 0x18E0, 0x00000004, + 0x18E4, 0x001FFFE0, + 0x18E8, 0x00800000, + 0x18EC, 0x1EC08000, + 0x18F0, 0x7F000064, + 0x18F4, 0x1F7DE75C, + 0x18F8, 0x7F7F7F7F, + 0x18FC, 0x7F7F7F7F, + 0x1900, 0xA7A7A7A7, + 0x1904, 0x95959595, + 0x1908, 0x00777788, + 0x190C, 0x77776666, + 0x1910, 0x00033333, + 0x1914, 0xAAAC875A, + 0x1918, 0x2AA2A8A2, + 0x191C, 0x2AAAA8A2, + 0x1920, 0x00878766, + 0x1924, 0x000C4924, + 0x1928, 0x5669B6C0, + 0x192C, 0x00409190, + 0x1930, 0xB85C0492, + 0x1934, 0x00B4A298, + 0x1938, 0x00030151, + 0x193C, 0x0058C618, + 0x1940, 0x41000000, + 0x1944, 0x00000BCB, + 0x1948, 0xAAAAAAAA, + 0x194C, 0x00B99999, + 0x1950, 0x88886665, + 0x1954, 0x08888888, + 0x1958, 0x00000618, + 0x195C, 0x00000000, + 0x1960, 0x00000000, + 0x1964, 0x00000000, + 0x1968, 0x00000000, + 0x196C, 0x00000000, + 0x1970, 0x00000000, + 0x1974, 0x00000000, + 0x1978, 0x00000000, + 0x197C, 0x00000000, + 0x1980, 0x00000000, + 0x1984, 0x00000000, + 0x1988, 0x00000000, + 0x198C, 0x00000000, + 0x1990, 0x00000000, + 0x1994, 0x00000000, + 0x1998, 0x00000000, + 0x199C, 0x00000000, + 0x19A0, 0x00000000, + 0x19A4, 0x00000000, + 0x19A8, 0x00000000, + 0x19AC, 0x00000000, + 0x19B0, 0x00000000, + 0x19B4, 0x00000000, + 0x19B8, 0x00000000, + 0x19BC, 0x00000000, + 0x19C0, 0x00000000, + 0x19C4, 0x00000000, + 0x19C8, 0x00000000, + 0x19CC, 0x00000000, + 0x19D0, 0x00000000, + 0x19D4, 0x00000000, + 0x19D8, 0x00000000, + 0x19DC, 0x00000000, + 0x19E0, 0x00000000, + 0x19E4, 0x00000000, + 0x19E8, 0x00000000, + 0x19EC, 0x00000000, + 0x19F0, 0x00000000, + 0x19F4, 0x00000000, + 0x19F8, 0x00000000, + 0x1C00, 0x00000000, + 0x1C04, 0x00000000, + 0x1C08, 0x00000000, + 0x1C0C, 0x00000000, + 0x1C10, 0x00000000, + 0x1C14, 0x00000000, + 0x1C18, 0x00000000, + 0x1C1C, 0x00000000, + 0x1C20, 0x03C23F00, + 0x1C24, 0xF101F002, + 0x1C28, 0x0FFE0010, + 0x1C2C, 0x453090FF, + 0x1C30, 0xFE0090FE, + 0x1C34, 0xE4E42000, + 0x1C38, 0xFFA1005E, + 0x1C40, 0x8F588837, + 0x1C44, 0x04400300, + 0x1C48, 0x00000000, + 0x1C4C, 0x00000200, + 0x1C50, 0x8E588837, + 0x1C54, 0x04400300, + 0x1C58, 0x00000000, + 0x1C5C, 0xFFFFFFFF, + 0x1C60, 0x0F030032, + 0x1C64, 0x360F0000, + 0x1C68, 0x007F0000, + 0x1C6C, 0x00010000, + 0x1C70, 0x00037FFE, + 0x1C74, 0x00000000, + 0x1C78, 0x00020000, + 0x1C7C, 0x00310000, + 0x1C80, 0x0E38E000, + 0x1C84, 0x245120D4, + 0x1C88, 0xC8400483, + 0x1C8C, 0x40005A20, + 0x1C94, 0x00000000, + 0x1C98, 0x00000000, + 0x1C9C, 0x00000000, + 0x1CA0, 0x00000000, + 0x1CA4, 0x20000000, + 0x1CA8, 0x0E000000, + 0x1CAC, 0xE424A2CC, + 0x1CB0, 0x00000000, + 0x1CB4, 0x00000000, + 0x1CB8, 0x24800000, + 0x1CBC, 0x60004800, + 0x1CC0, 0x24800000, + 0x1CC4, 0x60004800, + 0x1CC8, 0xF0444900, + 0x1CCC, 0x030300F1, + 0x1CD0, 0x0F000000, + 0x1CD4, 0x02024B00, + 0x1CD8, 0x04000000, + 0x1CDC, 0x10000000, + 0x1CE0, 0x60000000, + 0x1CE4, 0x00000000, + 0x1CE8, 0xC0000000, + 0x1CEC, 0x00000000, + 0x1CF0, 0x00000000, + 0x1CF4, 0xE4000000, + 0x1CF8, 0x00000000, + 0x1D00, 0x00000000, + 0x1D04, 0x08A3C000, + 0x1D08, 0xA0000000, + 0x1D10, 0x08B5BBBB, + 0x1D14, 0x77777777, + 0x1D18, 0x99999999, + 0x1D1C, 0x99999999, + 0x1D20, 0x000081E0, + 0x1D24, 0x00000000, + 0x1D28, 0x00000000, + 0x1D2C, 0xC0000000, + 0x1D30, 0x50009C00, + 0x1D34, 0x00000000, + 0x1D38, 0x00000000, + 0x1D3C, 0xF8000000, + 0x1D40, 0x00000000, + 0x1D44, 0x74740000, + 0x1D48, 0x14147474, + 0x1D4C, 0x00FFFF14, + 0x1D50, 0x00000000, + 0x1D54, 0x03A00000, + 0x1D58, 0x80800000, + 0x1D5C, 0x00000000, + 0x1D60, 0x00000000, + 0x1D64, 0x88000000, + 0x1D68, 0x00000000, + 0x1D6C, 0x666D8001, + 0x1D70, 0x20202020, + 0x1D74, 0x4E4E4E4E, + 0x1D78, 0x18189818, + 0x1D7C, 0x0005A000, + 0x1D80, 0x00080000, + 0x1D84, 0x00080000, + 0x1D88, 0x000000EF, + 0x1D8C, 0x0C0C0C0C, + 0x1D90, 0x103F003F, + 0x1D94, 0x00000000, + 0x1D98, 0x00000000, + 0x1D9C, 0x00000000, + 0x1DA0, 0x00000000, + 0x1DA4, 0x00000000, + 0x1DA8, 0x00000000, + 0x1DAC, 0x00000000, + 0x1DB0, 0x00000000, + 0x1DB4, 0x00000000, + 0x1DB8, 0x00000000, + 0x1DBC, 0x00000000, + 0x1DC0, 0x00000000, + 0x1DC4, 0x00000000, + 0x1DC8, 0x00000000, + 0x1DCC, 0x00000000, + 0x1DD0, 0x00000000, + 0x1DD4, 0x00000000, + 0x1DD8, 0x00000000, + 0x1DDC, 0x1FDF0000, + 0x1DE0, 0x01010000, + 0x1DE4, 0x05210123, + 0x1DE8, 0xFFFF4848, + 0x1DEC, 0x00000000, + 0x1DF0, 0x00000000, + 0x1DF4, 0x80000002, + 0x1DF8, 0x00000000, + 0x1E00, 0x00000000, + 0x1E04, 0x00000000, + 0x1E08, 0x00000000, + 0x1E0C, 0x00000000, + 0x1E10, 0x00000000, + 0x1E14, 0x00000000, + 0x1E18, 0x00000000, + 0x1E1C, 0x00000000, + 0x1E20, 0x00000000, + 0x1E24, 0x80003000, + 0x1E28, 0x000CC0C3, + 0x1E2C, 0xE4E40404, + 0x1E30, 0xE4E4E4E4, + 0x1E34, 0xF3001234, + 0x1E38, 0x00000000, + 0x1E3C, 0x00000000, + 0x1E40, 0x00000000, + 0x1E44, 0x00000000, + 0x1E48, 0x00000000, + 0x1E4C, 0x00000000, + 0x1E50, 0x00000000, + 0x1E54, 0x00000000, + 0x1E58, 0x00000000, + 0x1E5C, 0xC1000000, + 0x1E60, 0x00000000, + 0x1E64, 0xF3A00001, + 0x1E68, 0x0028846E, + 0x1E6C, 0x40274906, + 0x1E70, 0x00001000, + 0x1E74, 0x00000000, + 0x1E78, 0x00000000, + 0x1E7C, 0x00000000, + 0x1E80, 0x00000000, + 0x1E84, 0x00000000, + 0x1E84, 0x40000000, + 0x1E84, 0x41000000, + 0x1E84, 0x42000000, + 0x1E84, 0x43000000, + 0x1E84, 0x44000000, + 0x1E84, 0x45000000, + 0x1E84, 0x46000000, + 0x1E84, 0x47000000, + 0x1E84, 0x48000000, + 0x1E84, 0x49000000, + 0x1E84, 0x4A000000, + 0x1E84, 0x4B000000, + 0x1E84, 0x4C000000, + 0x1E84, 0x4D000000, + 0x1E84, 0x4E000000, + 0x1E84, 0x4F000000, + 0x1E84, 0x50000000, + 0x1E84, 0x51000000, + 0x1E84, 0x52000000, + 0x1E84, 0x53000000, + 0x1E84, 0x54000000, + 0x1E84, 0x55000000, + 0x1E84, 0x56000000, + 0x1E84, 0x57000000, + 0x1E84, 0x58000000, + 0x1E84, 0x59000000, + 0x1E84, 0x5A000000, + 0x1E84, 0x5B000000, + 0x1E84, 0x5C000000, + 0x1E84, 0x5D000000, + 0x1E84, 0x5E000000, + 0x1E84, 0x5F000000, + 0x1E84, 0x60000000, + 0x1E84, 0x61000000, + 0x1E84, 0x62000000, + 0x1E84, 0x63000000, + 0x1E84, 0x64000000, + 0x1E84, 0x65000000, + 0x1E84, 0x66000000, + 0x1E84, 0x67000000, + 0x1E84, 0x68000000, + 0x1E84, 0x69000000, + 0x1E84, 0x6A000000, + 0x1E84, 0x6B000000, + 0x1E84, 0x6C000000, + 0x1E84, 0x6D000000, + 0x1E84, 0x6E000000, + 0x1E84, 0x6F000000, + 0x1E84, 0x70000000, + 0x1E84, 0x71000000, + 0x1E84, 0x72000000, + 0x1E84, 0x73000000, + 0x1E84, 0x74000000, + 0x1E84, 0x75000000, + 0x1E84, 0x76000000, + 0x1E84, 0x77000000, + 0x1E84, 0x78000000, + 0x1E84, 0x79000000, + 0x1E84, 0x7A000000, + 0x1E84, 0x7B000000, + 0x1E84, 0x7C000000, + 0x1E84, 0x7D000000, + 0x1E84, 0x7E000000, + 0x1E84, 0x7F000000, + 0x1E84, 0x80000000, + 0x1E84, 0x00000000, + 0x1E88, 0x0200FC1C, + 0x1E8C, 0x00000000, + 0x1E90, 0x00000000, + 0x1E94, 0x04000000, + 0x1E98, 0x00000000, + 0x1E9C, 0x00000000, + 0x1EA0, 0x00000000, + 0x1EA4, 0x00000000, + 0x1EA8, 0xAA464646, + 0x1EAC, 0x01800030, + 0x1EB0, 0x00003002, + 0x1EB4, 0x31800002, + 0x1EB8, 0x00000000, + 0x1EBC, 0x00000000, + 0x1EC0, 0x00000000, + 0x1EC4, 0x00000000, + 0x1EC8, 0x00000000, + 0x1ECC, 0x00000000, + 0x1ED0, 0x00000000, + 0x1ED4, 0x8000000A, + 0x1ED8, 0x800B03E8, + 0x1EDC, 0x83E90FFF, + 0x1EE0, 0x8000FFFF, + 0x1EE4, 0x70000000, + 0x1EE8, 0x00000000, + 0x1EEC, 0x0280A933, + 0x1EF0, 0x00000A80, + 0x1EF4, 0x00001266, + 0x1EF8, 0x01000100, + 0x3A00, 0x0004080C, + 0x3A04, 0x1C202428, + 0x3A08, 0x0C101418, + 0x3A0C, 0x181C2024, + 0x3A10, 0x080C1014, + 0x3A14, 0x181C2024, + 0x3A18, 0x080C1014, + 0x3A1C, 0x00000000, + 0x3A20, 0x00000000, + 0x3A24, 0x00000000, + 0x3A28, 0x00000000, + 0x3A2C, 0x181C2024, + 0x3A30, 0x080C1014, + 0x3A34, 0x20240004, + 0x3A38, 0x1014181C, + 0x3A3C, 0x0004080C, + 0x3A40, 0x00000000, + 0x3A44, 0x00000000, + 0x3A48, 0x00000000, + 0x3A4C, 0x00000000, + 0x3A50, 0x00000000, + 0x3A54, 0x00000000, + 0x3A58, 0x00000000, + 0x3A5C, 0x00000000, + 0x3A60, 0x00000000, + 0x3A64, 0x00000000, + 0x3A68, 0x00000000, + 0x3A6C, 0x00000000, + 0x3A70, 0x00000000, + 0x3A74, 0x00000000, + 0x3A78, 0x00000000, + 0x3A7C, 0x00000000, + 0x3A80, 0x00000000, + 0x3A84, 0x00000000, + 0x3A88, 0x00000000, + 0x3A8C, 0x00000000, + 0x3A90, 0x00000000, + 0x3A94, 0x00000000, + 0x3A98, 0x00000000, + 0x3A9C, 0x00000000, + 0x3AA0, 0x00000000, + 0x3AA4, 0x00000000, + 0x4000, 0xA6A6A6A6, + 0x4004, 0x95959595, + 0x4008, 0x00777777, + 0x400C, 0x77776666, + 0x4010, 0x00033333, + 0x4014, 0xAAAC875A, + 0x4018, 0x2AA2A8A2, + 0x401C, 0x2AAAA8A2, + 0x4020, 0x00878766, + 0x4024, 0x000C4924, + 0x4028, 0x5669B6C0, + 0x402C, 0x00409190, + 0x4030, 0xB85C0492, + 0x4034, 0x00B4A298, + 0x4038, 0x00030151, + 0x403C, 0x0058C618, + 0x4040, 0x41000000, + 0x4044, 0x00000BCB, + 0x4048, 0xAAAAAAAA, + 0x404C, 0x00B98989, + 0x4050, 0x88886665, + 0x4054, 0x08888888, + 0x4058, 0x00000618, + 0x405C, 0x00000000, + 0x4060, 0x00000000, + 0x4064, 0x00000000, + 0x4068, 0x00000000, + 0x406C, 0x00000000, + 0x4070, 0x00000000, + 0x4074, 0x00000000, + 0x4078, 0x00000000, + 0x407C, 0x00000000, + 0x4080, 0x00000000, + 0x4084, 0x00000000, + 0x4088, 0x00000000, + 0x408C, 0x00000000, + 0x4090, 0x00000000, + 0x4094, 0x00000000, + 0x4098, 0x00000000, + 0x409C, 0x00000000, + 0x40A0, 0x00000000, + 0x40A4, 0x00000000, + 0x40A8, 0x00000000, + 0x40AC, 0x00000000, + 0x40B0, 0x00000000, + 0x40B4, 0x00000000, + 0x40B8, 0x00000000, + 0x40BC, 0x00000000, + 0x40C0, 0x00000000, + 0x40C4, 0x00000000, + 0x40C8, 0x00000000, + 0x40CC, 0x00000000, + 0x40D0, 0x00000000, + 0x40D4, 0x00000000, + 0x40D8, 0x00000000, + 0x40DC, 0x00000000, + 0x40E0, 0x00000000, + 0x40E4, 0x00000000, + 0x40E8, 0x00000000, + 0x40EC, 0x00000000, + 0x40F0, 0x00000000, + 0x40F4, 0x00000000, + 0x40F8, 0x00000000, + 0x4100, 0x00033312, + 0x4104, 0x00033312, + 0x410C, 0x17F40060, + 0x4110, 0x62D508C4, + 0x4114, 0x506AA5B4, + 0x4118, 0x000014FF, + 0x411C, 0x00000000, + 0x4120, 0x02D508CC, + 0x4124, 0x506AA5B4, + 0x4128, 0x000004FD, + 0x412C, 0x00000000, + 0x4134, 0x00000000, + 0x4138, 0x20000000, + 0x413C, 0x00000000, + 0x4140, 0x00000000, + 0x4144, 0x00000000, + 0x4148, 0x00000000, + 0x414C, 0x00000000, + 0x4150, 0x00000000, + 0x4154, 0x00000000, + 0x4158, 0x00000000, + 0x415C, 0x00000000, + 0x4160, 0xF0040FF8, + 0x4164, 0x7F000000, + 0x4168, 0x00000000, + 0x416C, 0x00008000, + 0x4170, 0x00000000, + 0x4174, 0x00000000, + 0x4178, 0x00000000, + 0x417C, 0x00000000, + 0x4180, 0x00000000, + 0x4184, 0x02B00000, + 0x4188, 0x00000000, + 0x418C, 0x00000000, + 0x4190, 0x00000000, + 0x4194, 0x00000000, + 0x4198, 0x00000000, + 0x41A0, 0x00510000, + 0x41A4, 0x183C1F7F, + 0x41A8, 0x1402C99A, + 0x41AC, 0x00004200, + 0x41B0, 0x0809FB08, + 0x41B0, 0x0809FB09, + 0x41B4, 0x00000000, + 0x41B8, 0x00000000, + 0x41BC, 0x00C3FF80, + 0x41C0, 0x0002D100, + 0x41C4, 0x00000004, + 0x41C8, 0x001FFFE0, + 0x41CC, 0x0809FB08, + 0x41CC, 0x0809FB09, + 0x41D0, 0x00000000, + 0x41D4, 0x00000000, + 0x41D8, 0x00C3FF80, + 0x41DC, 0x0002D100, + 0x41E0, 0x00000004, + 0x41E4, 0x001FFFE0, + 0x41E8, 0x00000200, + 0x41EC, 0x1E008000, + 0x41F0, 0x7F000064, + 0x41F4, 0x1F7DE75C, + 0x41F8, 0x7F7F7F7F, + 0x41FC, 0x7F7F7F7F, + 0x1830, 0x700B8001, + 0x1830, 0x700B8001, + 0x1830, 0x70144001, + 0x1830, 0x70244001, + 0x1830, 0x70344001, + 0x1830, 0x70444001, + 0x1830, 0x705B8001, + 0x1830, 0x70644001, + 0x1830, 0x707B8001, + 0x1830, 0x708B8001, + 0x1830, 0x709B8001, + 0x1830, 0x70AB8001, + 0x1830, 0x70BB8001, + 0x1830, 0x70CB8001, + 0x1830, 0x70DB8001, + 0x1830, 0x70EB8001, + 0x1830, 0x70FB8001, + 0x1830, 0x70FB8001, + 0x4130, 0x700B8001, + 0x4130, 0x700B8001, + 0x4130, 0x70144001, + 0x4130, 0x70244001, + 0x4130, 0x70344001, + 0x4130, 0x70444001, + 0x4130, 0x705B8001, + 0x4130, 0x70644001, + 0x4130, 0x707B8001, + 0x4130, 0x708B8001, + 0x4130, 0x709B8001, + 0x4130, 0x70AB8001, + 0x4130, 0x70BB8001, + 0x4130, 0x70CB8001, + 0x4130, 0x70DB8001, + 0x4130, 0x70EB8001, + 0x4130, 0x70FB8001, + 0x4130, 0x70FB8001, + 0x1A00, 0x00D047C8, + 0x1A04, 0xC0000008, + 0x1A08, 0x88838300, + 0x1A0C, 0x2E20100F, + 0x1A10, 0x9500BB78, + 0x1A14, 0x111440A8, + 0x1A18, 0x00881117, + 0x1A1C, 0x89140F00, + 0x1A20, 0x52840000, + 0x1A24, 0x3E18FEC8, + 0x1A28, 0x00150A88, + 0x1A2C, 0x12988000, + 0x1A30, 0x10114007, + 0x1A34, 0x1011C007, + 0x1A38, 0x00000000, + 0x1A3C, 0x00000000, + 0x1A40, 0x00000000, + 0x1A44, 0x00000000, + 0x1A48, 0x000C0000, + 0x1A4C, 0xB00000C0, + 0x1A50, 0x22040700, + 0x1A54, 0x09003000, + 0x1A58, 0x00000881, + 0x1A5C, 0x00000128, + 0x1A60, 0x85830000, + 0x1A64, 0x00000128, + 0x1A68, 0x00222211, + 0x1A6C, 0x00000000, + 0x1A70, 0x00008000, + 0x1A74, 0x00000048, + 0x1A78, 0x000089F0, + 0x1A7C, 0x225B0606, + 0x1A80, 0x208A7532, + 0x1A84, 0x85200200, + 0x1A88, 0x048C0000, + 0x1A8C, 0x00000000, + 0x1A90, 0x00000000, + 0x1A94, 0x00000000, + 0x1A98, 0xACC4C040, + 0x1A9C, 0x0016C8B2, + 0x1AA0, 0x00FAF0DE, + 0x1AA4, 0x00020000, + 0x1AA8, 0xBA0F0004, + 0x1AAC, 0x00122344, + 0x1AB0, 0x0FFFFFFF, + 0x1AB4, 0x0F201402, + 0x1AB8, 0x00000000, + 0x1ABC, 0xC2008080, + 0x1AC0, 0x54D0A742, + 0x1AC4, 0x00000000, + 0x1AC8, 0x00000807, + 0x1ACC, 0x00000707, + 0x1AD0, 0xA33529AD, + 0x1AD4, 0x0D8D8452, + 0x1AD8, 0x08024024, + 0x1ADC, 0x000DB001, + 0x1AE0, 0x00600391, + 0x1AE4, 0x08000080, + 0x1AE8, 0x00000002, + 0x1AEC, 0x00000000, + 0x1AF0, 0x00000000, + 0x1AF4, 0x00000000, + 0x1AF8, 0x00000000, + 0x1AFC, 0x00000000, + 0x1D0C, 0x00400000, + 0x1D0C, 0x00410000, + 0x1EE8, 0x00000003, + 0xC0C, 0x02F1D8BF, + 0x1D94, 0x40000000, + 0x1D94, 0x40010000, + 0x1D94, 0x40020000, + 0x1D94, 0x40030000, + 0x1D94, 0x40040000, + 0x1D94, 0x40050000, + 0x1D94, 0x40060000, + 0x1D94, 0x40070000, + 0x1D94, 0x40080000, + 0x1D94, 0x40090000, + 0x1D94, 0x400A0000, + 0x1D94, 0x400B0000, + 0x1D94, 0x400C0000, + 0x1D94, 0x400D0000, + 0x1D94, 0x400E0000, + 0x1D94, 0x400F0000, + 0x1D94, 0x40100000, + 0x1D94, 0x40110000, + 0x1D94, 0x40120000, + 0x1D94, 0x40130000, + 0x1D94, 0x40140000, + 0x1D94, 0x40150000, + 0x1D94, 0x40160000, + 0x1D94, 0x40170000, + 0x1D94, 0x40180000, + 0x1D94, 0x40190000, + 0x1D94, 0x401A0000, + 0x1D94, 0x401B0000, + 0x1D94, 0x401C0000, + 0x1D94, 0x401D0000, + 0x1D94, 0x401E0000, + 0x1D94, 0x401F0000, + 0x1D94, 0x40200000, + 0x1D94, 0x40210000, + 0x1D94, 0x40220000, + 0x1D94, 0x40230000, + 0x1D94, 0x40240000, + 0x1D94, 0x40250000, + 0x1D94, 0x40260000, + 0x1D94, 0x40270000, + 0x1D94, 0x40280000, + 0x1D94, 0x40290000, + 0x1D94, 0x402A0000, + 0x1D94, 0x402B0000, + 0x1D94, 0x402C0000, + 0x1D94, 0x402D0000, + 0x1D94, 0x402E0000, + 0x1D94, 0x402F0000, + 0x1D94, 0x40300000, + 0x1D94, 0x40310000, + 0x1D94, 0x40320000, + 0x1D94, 0x40330000, + 0x1D94, 0x40340000, + 0x1D94, 0x40350000, + 0x1D94, 0x40360000, + 0x1D94, 0x40370000, + 0x1D94, 0x40380000, + 0x1D94, 0x40390000, + 0x1D94, 0x403A0000, + 0x1D94, 0x403B0000, + 0x1D94, 0x403C0000, + 0x1D94, 0x403D0000, + 0x1D94, 0x403E0000, + 0x1D94, 0x403F0000, + 0x1D94, 0x40400000, + 0x1D94, 0x40410000, + 0x1D94, 0x40420000, + 0x1D94, 0x40430000, + 0x1D94, 0x40440000, + 0x1D94, 0x40450000, + 0x1D94, 0x40460000, + 0x1D94, 0x40470000, + 0x1D94, 0x40480000, + 0x1D94, 0x40490000, + 0x1D94, 0x404A0000, + 0x1D94, 0x404B0000, + 0x1D94, 0x404C0000, + 0x1D94, 0x404D0000, + 0x1D94, 0x404E0000, + 0x1D94, 0x404F0000, + 0x1D94, 0x40500000, + 0x1D94, 0x40510000, + 0x1D94, 0x40520000, + 0x1D94, 0x40530000, + 0x1D94, 0x40540000, + 0x1D94, 0x40550000, + 0x1D94, 0x40560000, + 0x1D94, 0x40570000, + 0x1D94, 0x40580000, + 0x1D94, 0x40590000, + 0x1D94, 0x405A0000, + 0x1D94, 0x405B0000, + 0x1D94, 0x405C0000, + 0x1D94, 0x405D0000, + 0x1D94, 0x405E0000, + 0x1D94, 0x405F0000, + 0x1D94, 0x40600000, + 0x1D94, 0x40610000, + 0x1D94, 0x40620000, + 0x1D94, 0x40630000, + 0x1D94, 0x40640000, + 0x1D94, 0x40650000, + 0x1D94, 0x40660000, + 0x1D94, 0x40670000, + 0x1D94, 0x40680000, + 0x1D94, 0x40690000, + 0x1D94, 0x406A0000, + 0x1D94, 0x406B0000, + 0x1D94, 0x406C0000, + 0x1D94, 0x406D0000, + 0x1D94, 0x406E0000, + 0x1D94, 0x406F0000, + 0x1D94, 0x40700000, + 0x1D94, 0x40710000, + 0x1D94, 0x40720000, + 0x1D94, 0x40730000, + 0x1D94, 0x40740000, + 0x1D94, 0x40750000, + 0x1D94, 0x40760000, + 0x1D94, 0x40770000, + 0x1D94, 0x40780000, + 0x1D94, 0x40790000, + 0x1D94, 0x407A0000, + 0x1D94, 0x407B0000, + 0x1D94, 0x407C0000, + 0x1D94, 0x407D0000, + 0x1D94, 0x407E0000, + 0x1D94, 0x407F0000, + 0x1D94, 0x40800000, + 0x1D94, 0x40810000, + 0x1D94, 0x40820000, + 0x1D94, 0x40830000, + 0x1D94, 0x40840000, + 0x1D94, 0x40850000, + 0x1D94, 0x40860000, + 0x1D94, 0x40870000, + 0x1D94, 0x40880000, + 0x1D94, 0x40890000, + 0x1D94, 0x408A0000, + 0x1D94, 0x408B0000, + 0x1D94, 0x408C0000, + 0x1D94, 0x408D0000, + 0x1D94, 0x408E0000, + 0x1D94, 0x408F0000, + 0x1D94, 0x40900000, + 0x1D94, 0x40910000, + 0x1D94, 0x40920000, + 0x1D94, 0x40930000, + 0x1D94, 0x40940000, + 0x1D94, 0x40950000, + 0x1D94, 0x40960000, + 0x1D94, 0x40970000, + 0x1D94, 0x40980000, + 0x1D94, 0x40990000, + 0x1D94, 0x409A0000, + 0x1D94, 0x409B0000, + 0x1D94, 0x409C0000, + 0x1D94, 0x409D0000, + 0x1D94, 0x409E0000, + 0x1D94, 0x409F0000, + 0x1D94, 0x40A00000, + 0x1D94, 0x40A10000, + 0x1D94, 0x40A20000, + 0x1D94, 0x40A30000, + 0x1D94, 0x40A40000, + 0x1D94, 0x40A50000, + 0x1D94, 0x40A60000, + 0x1D94, 0x40A70000, + 0x1D94, 0x40A80000, + 0x1D94, 0x40A90000, + 0x1D94, 0x40AA0000, + 0x1D94, 0x40AB0000, + 0x1D94, 0x40AC0000, + 0x1D94, 0x40AD0000, + 0x1D94, 0x40AE0000, + 0x1D94, 0x40AF0000, + 0x1D94, 0x40B00000, + 0x1D94, 0x40B10000, + 0x1D94, 0x40B20000, + 0x1D94, 0x40B30000, + 0x1D94, 0x40B40000, + 0x1D94, 0x40B50000, + 0x1D94, 0x40B60000, + 0x1D94, 0x40B70000, + 0x1D94, 0x40B80000, + 0x1D94, 0x40B90000, + 0x1D94, 0x40BA0000, + 0x1D94, 0x40BB0000, + 0x1D94, 0x40BC0000, + 0x1D94, 0x40BD0000, + 0x1D94, 0x40BE0000, + 0x1D94, 0x40BF0000, + 0x1D94, 0x40C00000, + 0x1D94, 0x40C10000, + 0x1D94, 0x40C20000, + 0x1D94, 0x40C30000, + 0x1D94, 0x40C40000, + 0x1D94, 0x40C50000, + 0x1D94, 0x40C60000, + 0x1D94, 0x40C70000, + 0x1D94, 0x40C80000, + 0x1D94, 0x40C90000, + 0x1D94, 0x40CA0000, + 0x1D94, 0x40CB0000, + 0x1D94, 0x40CC0000, + 0x1D94, 0x40CD0000, + 0x1D94, 0x40CE0000, + 0x1D94, 0x40CF0000, + 0x1D94, 0x40D00000, + 0x1D94, 0x40D10000, + 0x1D94, 0x40D20000, + 0x1D94, 0x40D30000, + 0x1D94, 0x40D40000, + 0x1D94, 0x40D50000, + 0x1D94, 0x40D60000, + 0x1D94, 0x40D70000, + 0x1D94, 0x40D80000, + 0x1D94, 0x40D90000, + 0x1D94, 0x40DA0000, + 0x1D94, 0x40DB0000, + 0x1D94, 0x40DC0000, + 0x1D94, 0x40DD0000, + 0x1D94, 0x40DE0000, + 0x1D94, 0x40DF0000, + 0x1D94, 0x40E00000, + 0x1D94, 0x40E10000, + 0x1D94, 0x40E20000, + 0x1D94, 0x40E30000, + 0x1D94, 0x40E40000, + 0x1D94, 0x40E50000, + 0x1D94, 0x40E60000, + 0x1D94, 0x40E70000, + 0x1D94, 0x40E80000, + 0x1D94, 0x40E90000, + 0x1D94, 0x40EA0000, + 0x1D94, 0x40EB0000, + 0x1D94, 0x40EC0000, + 0x1D94, 0x40ED0000, + 0x1D94, 0x40EE0000, + 0x1D94, 0x40EF0000, + 0x1D94, 0x40F00000, + 0x1D94, 0x40F10000, + 0x1D94, 0x40F20000, + 0x1D94, 0x40F30000, + 0x1D94, 0x40F40000, + 0x1D94, 0x40F50000, + 0x1D94, 0x40F60000, + 0x1D94, 0x40F70000, + 0x1D94, 0x40F80000, + 0x1D94, 0x40F90000, + 0x1D94, 0x40FA0000, + 0x1D94, 0x40FB0000, + 0x1D94, 0x40FC0000, + 0x1D94, 0x40FD0000, + 0x1D94, 0x40FE0000, + 0x1D94, 0x40FF0000, + 0xC0C, 0x02F1D8B7, + 0x1EE8, 0x00000000, +}; + +RTW_DECL_TABLE_PHY_COND(rtw8822c_bb, rtw_phy_cfg_bb); + +static const u32 rtw8822c_bb_pg_type0[] = { + 0, 0, 0, 0x00000c20, 0xffffffff, 0x484c5054, + 0, 0, 0, 0x00000c24, 0xffffffff, 0x54585c60, + 0, 0, 0, 0x00000c28, 0xffffffff, 0x44484c50, + 0, 0, 0, 0x00000c2c, 0xffffffff, 0x5054585c, + 0, 0, 0, 0x00000c30, 0xffffffff, 0x4044484c, + 0, 0, 1, 0x00000c34, 0xffffffff, 0x5054585c, + 0, 0, 1, 0x00000c38, 0xffffffff, 0x4044484c, + 0, 0, 0, 0x00000c3c, 0xffffffff, 0x5054585c, + 0, 0, 0, 0x00000c40, 0xffffffff, 0x4044484c, + 0, 0, 0, 0x00000c44, 0xffffffff, 0x585c383c, + 0, 0, 1, 0x00000c48, 0xffffffff, 0x484c5054, + 0, 0, 1, 0x00000c4c, 0xffffffff, 0x383c4044, + 0, 1, 0, 0x00000e20, 0xffffffff, 0x484c5054, + 0, 1, 0, 0x00000e24, 0xffffffff, 0x54585c60, + 0, 1, 0, 0x00000e28, 0xffffffff, 0x44484c50, + 0, 1, 0, 0x00000e2c, 0xffffffff, 0x5054585c, + 0, 1, 0, 0x00000e30, 0xffffffff, 0x4044484c, + 0, 1, 1, 0x00000e34, 0xffffffff, 0x5054585c, + 0, 1, 1, 0x00000e38, 0xffffffff, 0x4044484c, + 0, 1, 0, 0x00000e3c, 0xffffffff, 0x5054585c, + 0, 1, 0, 0x00000e40, 0xffffffff, 0x4044484c, + 0, 1, 0, 0x00000e44, 0xffffffff, 0x585c383c, + 0, 1, 1, 0x00000e48, 0xffffffff, 0x484c5054, + 0, 1, 1, 0x00000e4c, 0xffffffff, 0x383c4044, + 1, 0, 0, 0x00000c24, 0xffffffff, 0x54585c60, + 1, 0, 0, 0x00000c28, 0xffffffff, 0x44484c50, + 1, 0, 0, 0x00000c2c, 0xffffffff, 0x5054585c, + 1, 0, 0, 0x00000c30, 0xffffffff, 0x4044484c, + 1, 0, 1, 0x00000c34, 0xffffffff, 0x5054585c, + 1, 0, 1, 0x00000c38, 0xffffffff, 0x4044484c, + 1, 0, 0, 0x00000c3c, 0xffffffff, 0x5054585c, + 1, 0, 0, 0x00000c40, 0xffffffff, 0x4044484c, + 1, 0, 0, 0x00000c44, 0xffffffff, 0x585c383c, + 1, 0, 1, 0x00000c48, 0xffffffff, 0x484c5054, + 1, 0, 1, 0x00000c4c, 0xffffffff, 0x383c4044, + 1, 1, 0, 0x00000e24, 0xffffffff, 0x54585c60, + 1, 1, 0, 0x00000e28, 0xffffffff, 0x44484c50, + 1, 1, 0, 0x00000e2c, 0xffffffff, 0x5054585c, + 1, 1, 0, 0x00000e30, 0xffffffff, 0x4044484c, + 1, 1, 1, 0x00000e34, 0xffffffff, 0x5054585c, + 1, 1, 1, 0x00000e38, 0xffffffff, 0x4044484c, + 1, 1, 0, 0x00000e3c, 0xffffffff, 0x5054585c, + 1, 1, 0, 0x00000e40, 0xffffffff, 0x4044484c, + 1, 1, 0, 0x00000e44, 0xffffffff, 0x585c383c, + 1, 1, 1, 0x00000e48, 0xffffffff, 0x484c5054, + 1, 1, 1, 0x00000e4c, 0xffffffff, 0x383c4044 +}; + +RTW_DECL_TABLE_BB_PG(rtw8822c_bb_pg_type0); + +static const u32 rtw8822c_rf_a[] = { + 0x000, 0x00030000, + 0x018, 0x00013124, + 0x093, 0x0008483F, + 0x0DE, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x08E, 0x000B9140, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x08E, 0x000B9140, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x08E, 0x000A5540, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x08E, 0x000A5540, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x08E, 0x000A5540, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x08E, 0x000A5540, + 0xA0000000, 0x00000000, + 0x08E, 0x000A5540, + 0xB0000000, 0x00000000, + 0x081, 0x0000FC01, + 0x081, 0x0002FC01, + 0x081, 0x0003FC01, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x085, 0x0006A06C, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x085, 0x0006A06C, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x085, 0x0006A06C, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x085, 0x0006A06C, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x085, 0x0006A06C, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x085, 0x0006A06C, + 0xA0000000, 0x00000000, + 0x085, 0x0006A06C, + 0xB0000000, 0x00000000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EE, 0x00000010, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000002, + 0x03F, 0x0000002A, + 0x0EE, 0x00000000, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EE, 0x00000010, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000002, + 0x03F, 0x0000002A, + 0x0EE, 0x00000000, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EE, 0x00000010, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000002, + 0x03F, 0x0000002A, + 0x0EE, 0x00000000, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EE, 0x00000010, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000002, + 0x03F, 0x0000002A, + 0x0EE, 0x00000000, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EE, 0x00000010, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000002, + 0x03F, 0x0000002A, + 0x0EE, 0x00000000, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EE, 0x00000010, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000002, + 0x03F, 0x0000002A, + 0x0EE, 0x00000000, + 0xA0000000, 0x00000000, + 0x0EE, 0x00000010, + 0x033, 0x00000001, + 0x03F, 0x0000003F, + 0x033, 0x00000001, + 0x03F, 0x0000003F, + 0x033, 0x00000002, + 0x03F, 0x0000003F, + 0x0EE, 0x00000000, + 0xB0000000, 0x00000000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00010000, + 0x033, 0x0000000F, + 0x03F, 0x000773C0, + 0x033, 0x0000000E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000000D, + 0x03F, 0x000773E8, + 0x033, 0x0000000C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000000B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000000A, + 0x03F, 0x000002A8, + 0x033, 0x00000009, + 0x03F, 0x00000280, + 0x033, 0x00000008, + 0x03F, 0x000FF280, + 0x033, 0x00000007, + 0x03F, 0x00000200, + 0x033, 0x00000006, + 0x03F, 0x000001C0, + 0x033, 0x00000005, + 0x03F, 0x00000180, + 0x033, 0x00000004, + 0x03F, 0x00000040, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00010000, + 0x033, 0x0000000F, + 0x03F, 0x000773C0, + 0x033, 0x0000000E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000000D, + 0x03F, 0x000773E8, + 0x033, 0x0000000C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000000B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000000A, + 0x03F, 0x000002A8, + 0x033, 0x00000009, + 0x03F, 0x00000280, + 0x033, 0x00000008, + 0x03F, 0x000FF280, + 0x033, 0x00000007, + 0x03F, 0x00000200, + 0x033, 0x00000006, + 0x03F, 0x000001C0, + 0x033, 0x00000005, + 0x03F, 0x00000180, + 0x033, 0x00000004, + 0x03F, 0x00000040, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00010000, + 0x033, 0x0000000F, + 0x03F, 0x000773C0, + 0x033, 0x0000000E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000000D, + 0x03F, 0x000773E8, + 0x033, 0x0000000C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000000B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000000A, + 0x03F, 0x000002A8, + 0x033, 0x00000009, + 0x03F, 0x00000280, + 0x033, 0x00000008, + 0x03F, 0x000FF280, + 0x033, 0x00000007, + 0x03F, 0x00000200, + 0x033, 0x00000006, + 0x03F, 0x000001C0, + 0x033, 0x00000005, + 0x03F, 0x00000180, + 0x033, 0x00000004, + 0x03F, 0x00000040, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00010000, + 0x033, 0x0000000F, + 0x03F, 0x000773C0, + 0x033, 0x0000000E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000000D, + 0x03F, 0x000773E8, + 0x033, 0x0000000C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000000B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000000A, + 0x03F, 0x000002A8, + 0x033, 0x00000009, + 0x03F, 0x00000280, + 0x033, 0x00000008, + 0x03F, 0x000FF280, + 0x033, 0x00000007, + 0x03F, 0x00000200, + 0x033, 0x00000006, + 0x03F, 0x000001C0, + 0x033, 0x00000005, + 0x03F, 0x00000180, + 0x033, 0x00000004, + 0x03F, 0x00000040, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00010000, + 0x033, 0x0000000F, + 0x03F, 0x000773C0, + 0x033, 0x0000000E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000000D, + 0x03F, 0x000773E8, + 0x033, 0x0000000C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000000B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000000A, + 0x03F, 0x000002A8, + 0x033, 0x00000009, + 0x03F, 0x00000280, + 0x033, 0x00000008, + 0x03F, 0x000FF280, + 0x033, 0x00000007, + 0x03F, 0x00000200, + 0x033, 0x00000006, + 0x03F, 0x000001C0, + 0x033, 0x00000005, + 0x03F, 0x00000180, + 0x033, 0x00000004, + 0x03F, 0x00000040, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00010000, + 0x033, 0x0000000F, + 0x03F, 0x000773C0, + 0x033, 0x0000000E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000000D, + 0x03F, 0x000773E8, + 0x033, 0x0000000C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000000B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000000A, + 0x03F, 0x000002A8, + 0x033, 0x00000009, + 0x03F, 0x00000280, + 0x033, 0x00000008, + 0x03F, 0x000FF280, + 0x033, 0x00000007, + 0x03F, 0x00000200, + 0x033, 0x00000006, + 0x03F, 0x000001C0, + 0x033, 0x00000005, + 0x03F, 0x00000180, + 0x033, 0x00000004, + 0x03F, 0x00000040, + 0xA0000000, 0x00000000, + 0x0EF, 0x00010000, + 0x033, 0x0000000F, + 0x03F, 0x000773E8, + 0x033, 0x0000000E, + 0x03F, 0x000FF3A0, + 0x033, 0x0000000D, + 0x03F, 0x00000380, + 0x033, 0x0000000C, + 0x03F, 0x000FF380, + 0x033, 0x0000000B, + 0x03F, 0x00000300, + 0x033, 0x0000000A, + 0x03F, 0x000002A8, + 0x033, 0x00000009, + 0x03F, 0x00000280, + 0x033, 0x00000008, + 0x03F, 0x000FF280, + 0x033, 0x00000007, + 0x03F, 0x00000200, + 0x033, 0x00000006, + 0x03F, 0x000001C0, + 0x033, 0x00000005, + 0x03F, 0x00000180, + 0x033, 0x00000004, + 0x03F, 0x00000040, + 0xB0000000, 0x00000000, + 0x033, 0x00000003, + 0x03F, 0x00000000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000001F, + 0x03F, 0x000773C0, + 0x033, 0x0000001E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000001D, + 0x03F, 0x000773E8, + 0x033, 0x0000001C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000001B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000001A, + 0x03F, 0x000002A8, + 0x033, 0x00000019, + 0x03F, 0x00000280, + 0x033, 0x00000018, + 0x03F, 0x000FF280, + 0x033, 0x00000017, + 0x03F, 0x00000200, + 0x033, 0x00000016, + 0x03F, 0x000001C0, + 0x033, 0x00000015, + 0x03F, 0x00000180, + 0x033, 0x00000014, + 0x03F, 0x00000040, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000001F, + 0x03F, 0x000773C0, + 0x033, 0x0000001E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000001D, + 0x03F, 0x000773E8, + 0x033, 0x0000001C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000001B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000001A, + 0x03F, 0x000002A8, + 0x033, 0x00000019, + 0x03F, 0x00000280, + 0x033, 0x00000018, + 0x03F, 0x000FF280, + 0x033, 0x00000017, + 0x03F, 0x00000200, + 0x033, 0x00000016, + 0x03F, 0x000001C0, + 0x033, 0x00000015, + 0x03F, 0x00000180, + 0x033, 0x00000014, + 0x03F, 0x00000040, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000001F, + 0x03F, 0x000773C0, + 0x033, 0x0000001E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000001D, + 0x03F, 0x000773E8, + 0x033, 0x0000001C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000001B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000001A, + 0x03F, 0x000002A8, + 0x033, 0x00000019, + 0x03F, 0x00000280, + 0x033, 0x00000018, + 0x03F, 0x000FF280, + 0x033, 0x00000017, + 0x03F, 0x00000200, + 0x033, 0x00000016, + 0x03F, 0x000001C0, + 0x033, 0x00000015, + 0x03F, 0x00000180, + 0x033, 0x00000014, + 0x03F, 0x00000040, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000001F, + 0x03F, 0x000773C0, + 0x033, 0x0000001E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000001D, + 0x03F, 0x000773E8, + 0x033, 0x0000001C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000001B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000001A, + 0x03F, 0x000002A8, + 0x033, 0x00000019, + 0x03F, 0x00000280, + 0x033, 0x00000018, + 0x03F, 0x000FF280, + 0x033, 0x00000017, + 0x03F, 0x00000200, + 0x033, 0x00000016, + 0x03F, 0x000001C0, + 0x033, 0x00000015, + 0x03F, 0x00000180, + 0x033, 0x00000014, + 0x03F, 0x00000040, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000001F, + 0x03F, 0x000773C0, + 0x033, 0x0000001E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000001D, + 0x03F, 0x000773E8, + 0x033, 0x0000001C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000001B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000001A, + 0x03F, 0x000002A8, + 0x033, 0x00000019, + 0x03F, 0x00000280, + 0x033, 0x00000018, + 0x03F, 0x000FF280, + 0x033, 0x00000017, + 0x03F, 0x00000200, + 0x033, 0x00000016, + 0x03F, 0x000001C0, + 0x033, 0x00000015, + 0x03F, 0x00000180, + 0x033, 0x00000014, + 0x03F, 0x00000040, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000001F, + 0x03F, 0x000773C0, + 0x033, 0x0000001E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000001D, + 0x03F, 0x000773E8, + 0x033, 0x0000001C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000001B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000001A, + 0x03F, 0x000002A8, + 0x033, 0x00000019, + 0x03F, 0x00000280, + 0x033, 0x00000018, + 0x03F, 0x000FF280, + 0x033, 0x00000017, + 0x03F, 0x00000200, + 0x033, 0x00000016, + 0x03F, 0x000001C0, + 0x033, 0x00000015, + 0x03F, 0x00000180, + 0x033, 0x00000014, + 0x03F, 0x00000040, + 0xA0000000, 0x00000000, + 0x033, 0x0000001F, + 0x03F, 0x000773E8, + 0x033, 0x0000001E, + 0x03F, 0x000FF3A0, + 0x033, 0x0000001D, + 0x03F, 0x00000380, + 0x033, 0x0000001C, + 0x03F, 0x000FF380, + 0x033, 0x0000001B, + 0x03F, 0x00000300, + 0x033, 0x0000001A, + 0x03F, 0x000002A8, + 0x033, 0x00000019, + 0x03F, 0x00000280, + 0x033, 0x00000018, + 0x03F, 0x000FF280, + 0x033, 0x00000017, + 0x03F, 0x00000200, + 0x033, 0x00000016, + 0x03F, 0x000001C0, + 0x033, 0x00000015, + 0x03F, 0x00000180, + 0x033, 0x00000014, + 0x03F, 0x00000040, + 0xB0000000, 0x00000000, + 0x033, 0x00000013, + 0x03F, 0x00000000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000002F, + 0x03F, 0x000773C0, + 0x033, 0x0000002E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000002D, + 0x03F, 0x000773E8, + 0x033, 0x0000002C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000002B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000002A, + 0x03F, 0x000002A8, + 0x033, 0x00000029, + 0x03F, 0x00000280, + 0x033, 0x00000028, + 0x03F, 0x000FF280, + 0x033, 0x00000027, + 0x03F, 0x00000200, + 0x033, 0x00000026, + 0x03F, 0x000001C0, + 0x033, 0x00000025, + 0x03F, 0x00000180, + 0x033, 0x00000024, + 0x03F, 0x00000040, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000002F, + 0x03F, 0x000773C0, + 0x033, 0x0000002E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000002D, + 0x03F, 0x000773E8, + 0x033, 0x0000002C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000002B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000002A, + 0x03F, 0x000002A8, + 0x033, 0x00000029, + 0x03F, 0x00000280, + 0x033, 0x00000028, + 0x03F, 0x000FF280, + 0x033, 0x00000027, + 0x03F, 0x00000200, + 0x033, 0x00000026, + 0x03F, 0x000001C0, + 0x033, 0x00000025, + 0x03F, 0x00000180, + 0x033, 0x00000024, + 0x03F, 0x00000040, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000002F, + 0x03F, 0x000773C0, + 0x033, 0x0000002E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000002D, + 0x03F, 0x000773E8, + 0x033, 0x0000002C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000002B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000002A, + 0x03F, 0x000002A8, + 0x033, 0x00000029, + 0x03F, 0x00000280, + 0x033, 0x00000028, + 0x03F, 0x000FF280, + 0x033, 0x00000027, + 0x03F, 0x00000200, + 0x033, 0x00000026, + 0x03F, 0x000001C0, + 0x033, 0x00000025, + 0x03F, 0x00000180, + 0x033, 0x00000024, + 0x03F, 0x00000040, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000002F, + 0x03F, 0x000773C0, + 0x033, 0x0000002E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000002D, + 0x03F, 0x000773E8, + 0x033, 0x0000002C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000002B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000002A, + 0x03F, 0x000002A8, + 0x033, 0x00000029, + 0x03F, 0x00000280, + 0x033, 0x00000028, + 0x03F, 0x000FF280, + 0x033, 0x00000027, + 0x03F, 0x00000200, + 0x033, 0x00000026, + 0x03F, 0x000001C0, + 0x033, 0x00000025, + 0x03F, 0x00000180, + 0x033, 0x00000024, + 0x03F, 0x00000040, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000002F, + 0x03F, 0x000773C0, + 0x033, 0x0000002E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000002D, + 0x03F, 0x000773E8, + 0x033, 0x0000002C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000002B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000002A, + 0x03F, 0x000002A8, + 0x033, 0x00000029, + 0x03F, 0x00000280, + 0x033, 0x00000028, + 0x03F, 0x000FF280, + 0x033, 0x00000027, + 0x03F, 0x00000200, + 0x033, 0x00000026, + 0x03F, 0x000001C0, + 0x033, 0x00000025, + 0x03F, 0x00000180, + 0x033, 0x00000024, + 0x03F, 0x00000040, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000002F, + 0x03F, 0x000773C0, + 0x033, 0x0000002E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000002D, + 0x03F, 0x000773E8, + 0x033, 0x0000002C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000002B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000002A, + 0x03F, 0x000002A8, + 0x033, 0x00000029, + 0x03F, 0x00000280, + 0x033, 0x00000028, + 0x03F, 0x000FF280, + 0x033, 0x00000027, + 0x03F, 0x00000200, + 0x033, 0x00000026, + 0x03F, 0x000001C0, + 0x033, 0x00000025, + 0x03F, 0x00000180, + 0x033, 0x00000024, + 0x03F, 0x00000040, + 0xA0000000, 0x00000000, + 0x033, 0x0000002F, + 0x03F, 0x000773E8, + 0x033, 0x0000002E, + 0x03F, 0x000FF3A0, + 0x033, 0x0000002D, + 0x03F, 0x00000380, + 0x033, 0x0000002C, + 0x03F, 0x000FF380, + 0x033, 0x0000002B, + 0x03F, 0x00000300, + 0x033, 0x0000002A, + 0x03F, 0x000002A8, + 0x033, 0x00000029, + 0x03F, 0x00000280, + 0x033, 0x00000028, + 0x03F, 0x000FF280, + 0x033, 0x00000027, + 0x03F, 0x00000200, + 0x033, 0x00000026, + 0x03F, 0x000001C0, + 0x033, 0x00000025, + 0x03F, 0x00000180, + 0x033, 0x00000024, + 0x03F, 0x00000040, + 0xB0000000, 0x00000000, + 0x033, 0x00000023, + 0x03F, 0x00000000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000003F, + 0x03F, 0x000773C0, + 0x033, 0x0000003E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000003D, + 0x03F, 0x000773E8, + 0x033, 0x0000003C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000003B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000003A, + 0x03F, 0x000002A8, + 0x033, 0x00000039, + 0x03F, 0x00000280, + 0x033, 0x00000038, + 0x03F, 0x000FF280, + 0x033, 0x00000037, + 0x03F, 0x00000200, + 0x033, 0x00000036, + 0x03F, 0x000001C0, + 0x033, 0x00000035, + 0x03F, 0x00000180, + 0x033, 0x00000034, + 0x03F, 0x00000040, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000003F, + 0x03F, 0x000773C0, + 0x033, 0x0000003E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000003D, + 0x03F, 0x000773E8, + 0x033, 0x0000003C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000003B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000003A, + 0x03F, 0x000002A8, + 0x033, 0x00000039, + 0x03F, 0x00000280, + 0x033, 0x00000038, + 0x03F, 0x000FF280, + 0x033, 0x00000037, + 0x03F, 0x00000200, + 0x033, 0x00000036, + 0x03F, 0x000001C0, + 0x033, 0x00000035, + 0x03F, 0x00000180, + 0x033, 0x00000034, + 0x03F, 0x00000040, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000003F, + 0x03F, 0x000773C0, + 0x033, 0x0000003E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000003D, + 0x03F, 0x000773E8, + 0x033, 0x0000003C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000003B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000003A, + 0x03F, 0x000002A8, + 0x033, 0x00000039, + 0x03F, 0x00000280, + 0x033, 0x00000038, + 0x03F, 0x000FF280, + 0x033, 0x00000037, + 0x03F, 0x00000200, + 0x033, 0x00000036, + 0x03F, 0x000001C0, + 0x033, 0x00000035, + 0x03F, 0x00000180, + 0x033, 0x00000034, + 0x03F, 0x00000040, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000003F, + 0x03F, 0x000773C0, + 0x033, 0x0000003E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000003D, + 0x03F, 0x000773E8, + 0x033, 0x0000003C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000003B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000003A, + 0x03F, 0x000002A8, + 0x033, 0x00000039, + 0x03F, 0x00000280, + 0x033, 0x00000038, + 0x03F, 0x000FF280, + 0x033, 0x00000037, + 0x03F, 0x00000200, + 0x033, 0x00000036, + 0x03F, 0x000001C0, + 0x033, 0x00000035, + 0x03F, 0x00000180, + 0x033, 0x00000034, + 0x03F, 0x00000040, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000003F, + 0x03F, 0x000773C0, + 0x033, 0x0000003E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000003D, + 0x03F, 0x000773E8, + 0x033, 0x0000003C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000003B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000003A, + 0x03F, 0x000002A8, + 0x033, 0x00000039, + 0x03F, 0x00000280, + 0x033, 0x00000038, + 0x03F, 0x000FF280, + 0x033, 0x00000037, + 0x03F, 0x00000200, + 0x033, 0x00000036, + 0x03F, 0x000001C0, + 0x033, 0x00000035, + 0x03F, 0x00000180, + 0x033, 0x00000034, + 0x03F, 0x00000040, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000003F, + 0x03F, 0x000773C0, + 0x033, 0x0000003E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000003D, + 0x03F, 0x000773E8, + 0x033, 0x0000003C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000003B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000003A, + 0x03F, 0x000002A8, + 0x033, 0x00000039, + 0x03F, 0x00000280, + 0x033, 0x00000038, + 0x03F, 0x000FF280, + 0x033, 0x00000037, + 0x03F, 0x00000200, + 0x033, 0x00000036, + 0x03F, 0x000001C0, + 0x033, 0x00000035, + 0x03F, 0x00000180, + 0x033, 0x00000034, + 0x03F, 0x00000040, + 0xA0000000, 0x00000000, + 0x033, 0x0000003F, + 0x03F, 0x000773E8, + 0x033, 0x0000003E, + 0x03F, 0x000FF3A0, + 0x033, 0x0000003D, + 0x03F, 0x00000380, + 0x033, 0x0000003C, + 0x03F, 0x000FF380, + 0x033, 0x0000003B, + 0x03F, 0x00000300, + 0x033, 0x0000003A, + 0x03F, 0x000002A8, + 0x033, 0x00000039, + 0x03F, 0x00000280, + 0x033, 0x00000038, + 0x03F, 0x000FF280, + 0x033, 0x00000037, + 0x03F, 0x00000200, + 0x033, 0x00000036, + 0x03F, 0x000001C0, + 0x033, 0x00000035, + 0x03F, 0x00000180, + 0x033, 0x00000034, + 0x03F, 0x00000040, + 0xB0000000, 0x00000000, + 0x033, 0x00000033, + 0x03F, 0x00000000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000004F, + 0x03F, 0x000773C0, + 0x033, 0x0000004E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000004D, + 0x03F, 0x000773E8, + 0x033, 0x0000004C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000004B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000004A, + 0x03F, 0x000002A8, + 0x033, 0x00000049, + 0x03F, 0x00000280, + 0x033, 0x00000048, + 0x03F, 0x000FF280, + 0x033, 0x00000047, + 0x03F, 0x00000200, + 0x033, 0x00000046, + 0x03F, 0x000001C0, + 0x033, 0x00000045, + 0x03F, 0x00000180, + 0x033, 0x00000044, + 0x03F, 0x00000040, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000004F, + 0x03F, 0x000773C0, + 0x033, 0x0000004E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000004D, + 0x03F, 0x000773E8, + 0x033, 0x0000004C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000004B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000004A, + 0x03F, 0x000002A8, + 0x033, 0x00000049, + 0x03F, 0x00000280, + 0x033, 0x00000048, + 0x03F, 0x000FF280, + 0x033, 0x00000047, + 0x03F, 0x00000200, + 0x033, 0x00000046, + 0x03F, 0x000001C0, + 0x033, 0x00000045, + 0x03F, 0x00000180, + 0x033, 0x00000044, + 0x03F, 0x00000040, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000004F, + 0x03F, 0x000773C0, + 0x033, 0x0000004E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000004D, + 0x03F, 0x000773E8, + 0x033, 0x0000004C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000004B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000004A, + 0x03F, 0x000002A8, + 0x033, 0x00000049, + 0x03F, 0x00000280, + 0x033, 0x00000048, + 0x03F, 0x000FF280, + 0x033, 0x00000047, + 0x03F, 0x00000200, + 0x033, 0x00000046, + 0x03F, 0x000001C0, + 0x033, 0x00000045, + 0x03F, 0x00000180, + 0x033, 0x00000044, + 0x03F, 0x00000040, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000004F, + 0x03F, 0x000773C0, + 0x033, 0x0000004E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000004D, + 0x03F, 0x000773E8, + 0x033, 0x0000004C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000004B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000004A, + 0x03F, 0x000002A8, + 0x033, 0x00000049, + 0x03F, 0x00000280, + 0x033, 0x00000048, + 0x03F, 0x000FF280, + 0x033, 0x00000047, + 0x03F, 0x00000200, + 0x033, 0x00000046, + 0x03F, 0x000001C0, + 0x033, 0x00000045, + 0x03F, 0x00000180, + 0x033, 0x00000044, + 0x03F, 0x00000040, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000004F, + 0x03F, 0x000773C0, + 0x033, 0x0000004E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000004D, + 0x03F, 0x000773E8, + 0x033, 0x0000004C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000004B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000004A, + 0x03F, 0x000002A8, + 0x033, 0x00000049, + 0x03F, 0x00000280, + 0x033, 0x00000048, + 0x03F, 0x000FF280, + 0x033, 0x00000047, + 0x03F, 0x00000200, + 0x033, 0x00000046, + 0x03F, 0x000001C0, + 0x033, 0x00000045, + 0x03F, 0x00000180, + 0x033, 0x00000044, + 0x03F, 0x00000040, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000004F, + 0x03F, 0x000773C0, + 0x033, 0x0000004E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000004D, + 0x03F, 0x000773E8, + 0x033, 0x0000004C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000004B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000004A, + 0x03F, 0x000002A8, + 0x033, 0x00000049, + 0x03F, 0x00000280, + 0x033, 0x00000048, + 0x03F, 0x000FF280, + 0x033, 0x00000047, + 0x03F, 0x00000200, + 0x033, 0x00000046, + 0x03F, 0x000001C0, + 0x033, 0x00000045, + 0x03F, 0x00000180, + 0x033, 0x00000044, + 0x03F, 0x00000040, + 0xA0000000, 0x00000000, + 0x033, 0x0000004F, + 0x03F, 0x000773E8, + 0x033, 0x0000004E, + 0x03F, 0x000FF3A0, + 0x033, 0x0000004D, + 0x03F, 0x00000380, + 0x033, 0x0000004C, + 0x03F, 0x000FF380, + 0x033, 0x0000004B, + 0x03F, 0x00000300, + 0x033, 0x0000004A, + 0x03F, 0x000002A8, + 0x033, 0x00000049, + 0x03F, 0x00000280, + 0x033, 0x00000048, + 0x03F, 0x000FF280, + 0x033, 0x00000047, + 0x03F, 0x00000200, + 0x033, 0x00000046, + 0x03F, 0x000001C0, + 0x033, 0x00000045, + 0x03F, 0x00000180, + 0x033, 0x00000044, + 0x03F, 0x00000040, + 0xB0000000, 0x00000000, + 0x033, 0x00000043, + 0x03F, 0x00000000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000005F, + 0x03F, 0x000773C0, + 0x033, 0x0000005E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000005D, + 0x03F, 0x000773E8, + 0x033, 0x0000005C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000005B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000005A, + 0x03F, 0x000002A8, + 0x033, 0x00000059, + 0x03F, 0x00000280, + 0x033, 0x00000058, + 0x03F, 0x000FF280, + 0x033, 0x00000057, + 0x03F, 0x00000200, + 0x033, 0x00000056, + 0x03F, 0x000001C0, + 0x033, 0x00000055, + 0x03F, 0x00000180, + 0x033, 0x00000054, + 0x03F, 0x00000040, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000005F, + 0x03F, 0x000773C0, + 0x033, 0x0000005E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000005D, + 0x03F, 0x000773E8, + 0x033, 0x0000005C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000005B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000005A, + 0x03F, 0x000002A8, + 0x033, 0x00000059, + 0x03F, 0x00000280, + 0x033, 0x00000058, + 0x03F, 0x000FF280, + 0x033, 0x00000057, + 0x03F, 0x00000200, + 0x033, 0x00000056, + 0x03F, 0x000001C0, + 0x033, 0x00000055, + 0x03F, 0x00000180, + 0x033, 0x00000054, + 0x03F, 0x00000040, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000005F, + 0x03F, 0x000773C0, + 0x033, 0x0000005E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000005D, + 0x03F, 0x000773E8, + 0x033, 0x0000005C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000005B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000005A, + 0x03F, 0x000002A8, + 0x033, 0x00000059, + 0x03F, 0x00000280, + 0x033, 0x00000058, + 0x03F, 0x000FF280, + 0x033, 0x00000057, + 0x03F, 0x00000200, + 0x033, 0x00000056, + 0x03F, 0x000001C0, + 0x033, 0x00000055, + 0x03F, 0x00000180, + 0x033, 0x00000054, + 0x03F, 0x00000040, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000005F, + 0x03F, 0x000773C0, + 0x033, 0x0000005E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000005D, + 0x03F, 0x000773E8, + 0x033, 0x0000005C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000005B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000005A, + 0x03F, 0x000002A8, + 0x033, 0x00000059, + 0x03F, 0x00000280, + 0x033, 0x00000058, + 0x03F, 0x000FF280, + 0x033, 0x00000057, + 0x03F, 0x00000200, + 0x033, 0x00000056, + 0x03F, 0x000001C0, + 0x033, 0x00000055, + 0x03F, 0x00000180, + 0x033, 0x00000054, + 0x03F, 0x00000040, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000005F, + 0x03F, 0x000773C0, + 0x033, 0x0000005E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000005D, + 0x03F, 0x000773E8, + 0x033, 0x0000005C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000005B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000005A, + 0x03F, 0x000002A8, + 0x033, 0x00000059, + 0x03F, 0x00000280, + 0x033, 0x00000058, + 0x03F, 0x000FF280, + 0x033, 0x00000057, + 0x03F, 0x00000200, + 0x033, 0x00000056, + 0x03F, 0x000001C0, + 0x033, 0x00000055, + 0x03F, 0x00000180, + 0x033, 0x00000054, + 0x03F, 0x00000040, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000005F, + 0x03F, 0x000773C0, + 0x033, 0x0000005E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000005D, + 0x03F, 0x000773E8, + 0x033, 0x0000005C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000005B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000005A, + 0x03F, 0x000002A8, + 0x033, 0x00000059, + 0x03F, 0x00000280, + 0x033, 0x00000058, + 0x03F, 0x000FF280, + 0x033, 0x00000057, + 0x03F, 0x00000200, + 0x033, 0x00000056, + 0x03F, 0x000001C0, + 0x033, 0x00000055, + 0x03F, 0x00000180, + 0x033, 0x00000054, + 0x03F, 0x00000040, + 0xA0000000, 0x00000000, + 0x033, 0x0000005F, + 0x03F, 0x000773E8, + 0x033, 0x0000005E, + 0x03F, 0x000FF3A0, + 0x033, 0x0000005D, + 0x03F, 0x00000380, + 0x033, 0x0000005C, + 0x03F, 0x000FF380, + 0x033, 0x0000005B, + 0x03F, 0x00000300, + 0x033, 0x0000005A, + 0x03F, 0x000002A8, + 0x033, 0x00000059, + 0x03F, 0x00000280, + 0x033, 0x00000058, + 0x03F, 0x000FF280, + 0x033, 0x00000057, + 0x03F, 0x00000200, + 0x033, 0x00000056, + 0x03F, 0x000001C0, + 0x033, 0x00000055, + 0x03F, 0x00000180, + 0x033, 0x00000054, + 0x03F, 0x00000040, + 0xB0000000, 0x00000000, + 0x033, 0x00000053, + 0x03F, 0x00000000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000000, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000000, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000000, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000000, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000000, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000000, + 0xA0000000, 0x00000000, + 0x0EF, 0x00000000, + 0xB0000000, 0x00000000, + 0x08A, 0x000E7DE3, + 0x08B, 0x0008FE00, + 0x0EE, 0x00000008, + 0x033, 0x00000000, + 0x03F, 0x00000023, + 0x033, 0x00000001, + 0x03F, 0x00000023, + 0x0EE, 0x00000000, + 0x0EF, 0x00004000, + 0x033, 0x00000000, + 0x03F, 0x0000000F, + 0x033, 0x00000002, + 0x03F, 0x00000000, + 0x0EF, 0x00000000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00020000, + 0x033, 0x00000000, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000001, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000002, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000003, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000004, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000005, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000006, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000007, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000008, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000009, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000000A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000000B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000000C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000000D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000000E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000000F, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000010, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000011, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000012, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000013, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000014, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000015, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000016, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000017, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000018, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000019, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000001A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000001B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000001C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000001D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000001E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000001F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000020, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000021, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000022, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000023, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000024, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000025, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000026, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000027, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000028, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000029, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000002A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000002B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000002C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000002D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000002E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000002F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x0EF, 0x00000000, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00020000, + 0x033, 0x00000000, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000001, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000002, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000003, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000004, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000005, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000006, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000007, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000008, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000009, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000000A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000000B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000000C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000000D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000000E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000000F, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000010, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000011, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000012, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000013, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000014, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000015, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000016, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000017, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000018, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000019, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000001A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000001B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000001C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000001D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000001E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000001F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000020, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000021, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000022, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000023, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000024, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000025, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000026, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000027, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000028, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000029, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000002A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000002B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000002C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000002D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000002E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000002F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x0EF, 0x00000000, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00020000, + 0x033, 0x00000000, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000001, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000002, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000003, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000004, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000005, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000006, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000007, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000008, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000009, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000000A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000000B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000000C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000000D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000000E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000000F, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000010, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000011, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000012, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000013, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000014, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000015, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000016, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000017, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000018, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000019, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000001A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000001B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000001C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000001D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000001E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000001F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000020, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000021, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000022, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000023, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000024, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000025, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000026, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000027, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000028, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000029, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000002A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000002B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000002C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000002D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000002E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000002F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x0EF, 0x00000000, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00020000, + 0x033, 0x00000000, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000001, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000002, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000003, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000004, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000005, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000006, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000007, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000008, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000009, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000000A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000000B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000000C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000000D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000000E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000000F, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000010, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000011, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000012, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000013, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000014, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000015, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000016, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000017, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000018, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000019, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000001A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000001B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000001C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000001D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000001E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000001F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000020, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000021, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000022, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000023, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000024, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000025, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000026, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000027, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000028, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000029, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000002A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000002B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000002C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000002D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000002E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000002F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x0EF, 0x00000000, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00020000, + 0x033, 0x00000000, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000001, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000002, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000003, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000004, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000005, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000006, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000007, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000008, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000009, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000000A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000000B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000000C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000000D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000000E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000000F, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000010, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000011, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000012, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000013, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000014, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000015, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000016, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000017, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000018, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000019, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000001A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000001B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000001C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000001D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000001E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000001F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000020, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000021, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000022, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000023, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000024, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000025, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000026, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000027, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000028, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000029, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000002A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000002B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000002C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000002D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000002E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000002F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x0EF, 0x00000000, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00020000, + 0x033, 0x00000000, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000001, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000002, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000003, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000004, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000005, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000006, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000007, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000008, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000009, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000000A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000000B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000000C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000000D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000000E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000000F, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000010, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000011, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000012, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000013, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000014, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000015, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000016, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000017, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000018, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000019, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000001A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000001B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000001C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000001D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000001E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000001F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000020, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000021, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000022, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000023, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000024, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000025, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000026, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000027, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000028, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000029, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000002A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000002B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000002C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000002D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000002E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000002F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x0EF, 0x00000000, + 0xA0000000, 0x00000000, + 0x0EF, 0x00020000, + 0x033, 0x00000000, + 0x03E, 0x00001910, + 0x03F, 0x00020000, + 0x033, 0x00000001, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000002, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000003, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000004, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000005, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000006, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000007, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000008, + 0x03E, 0x00001910, + 0x03F, 0x00020000, + 0x033, 0x00000009, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000000A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000000B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000000C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000000D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000000E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000000F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000010, + 0x03E, 0x00001910, + 0x03F, 0x00020000, + 0x033, 0x00000011, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000012, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000013, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000014, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000015, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000016, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000017, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000018, + 0x03E, 0x00001910, + 0x03F, 0x00020000, + 0x033, 0x00000019, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000001A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000001B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000001C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000001D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000001E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000001F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000020, + 0x03E, 0x00001910, + 0x03F, 0x00020000, + 0x033, 0x00000021, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000022, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000023, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000024, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000025, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000026, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000027, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000028, + 0x03E, 0x00001910, + 0x03F, 0x00020000, + 0x033, 0x00000029, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000002A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000002B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000002C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000002D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000002E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000002F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x0EF, 0x00000000, + 0xB0000000, 0x00000000, + 0x0FE, 0x00000000, + 0x01B, 0x00003A40, + 0x061, 0x0000D233, + 0x062, 0x0004D232, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x063, 0x00000002, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x063, 0x00000002, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x063, 0x00000002, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x063, 0x00000002, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x063, 0x00000002, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x063, 0x00000002, + 0xA0000000, 0x00000000, + 0x063, 0x00000C02, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000200, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000237, + 0x030, 0x00001237, + 0x030, 0x00002237, + 0x030, 0x00003237, + 0x030, 0x00004207, + 0x030, 0x00005237, + 0x030, 0x00006237, + 0x030, 0x00007237, + 0x030, 0x00008207, + 0x030, 0x00009237, + 0x030, 0x0000A237, + 0x030, 0x0000B237, + 0x030, 0x0000C237, + 0x030, 0x0000D237, + 0x030, 0x0000E207, + 0x030, 0x0000F237, + 0x030, 0x00010237, + 0x030, 0x00011237, + 0x030, 0x00012207, + 0x030, 0x00013237, + 0x030, 0x00014237, + 0x030, 0x00015237, + 0x030, 0x00016207, + 0x030, 0x00017237, + 0x030, 0x00018207, + 0x030, 0x00019237, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000237, + 0x030, 0x00001237, + 0x030, 0x00002237, + 0x030, 0x00003237, + 0x030, 0x00004207, + 0x030, 0x00005237, + 0x030, 0x00006237, + 0x030, 0x00007237, + 0x030, 0x00008207, + 0x030, 0x00009237, + 0x030, 0x0000A237, + 0x030, 0x0000B237, + 0x030, 0x0000C237, + 0x030, 0x0000D237, + 0x030, 0x0000E207, + 0x030, 0x0000F237, + 0x030, 0x00010237, + 0x030, 0x00011237, + 0x030, 0x00012207, + 0x030, 0x00013237, + 0x030, 0x00014237, + 0x030, 0x00015237, + 0x030, 0x00016207, + 0x030, 0x00017237, + 0x030, 0x00018207, + 0x030, 0x00019237, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000237, + 0x030, 0x00001237, + 0x030, 0x00002237, + 0x030, 0x00003237, + 0x030, 0x00004207, + 0x030, 0x00005237, + 0x030, 0x00006237, + 0x030, 0x00007237, + 0x030, 0x00008207, + 0x030, 0x00009237, + 0x030, 0x0000A237, + 0x030, 0x0000B237, + 0x030, 0x0000C237, + 0x030, 0x0000D237, + 0x030, 0x0000E207, + 0x030, 0x0000F237, + 0x030, 0x00010237, + 0x030, 0x00011237, + 0x030, 0x00012207, + 0x030, 0x00013237, + 0x030, 0x00014237, + 0x030, 0x00015237, + 0x030, 0x00016207, + 0x030, 0x00017237, + 0x030, 0x00018207, + 0x030, 0x00019237, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000237, + 0x030, 0x00001237, + 0x030, 0x00002237, + 0x030, 0x00003237, + 0x030, 0x00004207, + 0x030, 0x00005237, + 0x030, 0x00006237, + 0x030, 0x00007237, + 0x030, 0x00008207, + 0x030, 0x00009237, + 0x030, 0x0000A237, + 0x030, 0x0000B237, + 0x030, 0x0000C237, + 0x030, 0x0000D237, + 0x030, 0x0000E207, + 0x030, 0x0000F237, + 0x030, 0x00010237, + 0x030, 0x00011237, + 0x030, 0x00012207, + 0x030, 0x00013237, + 0x030, 0x00014237, + 0x030, 0x00015237, + 0x030, 0x00016207, + 0x030, 0x00017237, + 0x030, 0x00018207, + 0x030, 0x00019237, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000237, + 0x030, 0x00001237, + 0x030, 0x00002237, + 0x030, 0x00003237, + 0x030, 0x00004207, + 0x030, 0x00005237, + 0x030, 0x00006237, + 0x030, 0x00007237, + 0x030, 0x00008207, + 0x030, 0x00009237, + 0x030, 0x0000A237, + 0x030, 0x0000B237, + 0x030, 0x0000C237, + 0x030, 0x0000D237, + 0x030, 0x0000E207, + 0x030, 0x0000F237, + 0x030, 0x00010237, + 0x030, 0x00011237, + 0x030, 0x00012207, + 0x030, 0x00013237, + 0x030, 0x00014237, + 0x030, 0x00015237, + 0x030, 0x00016207, + 0x030, 0x00017237, + 0x030, 0x00018207, + 0x030, 0x00019237, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000237, + 0x030, 0x00001237, + 0x030, 0x00002237, + 0x030, 0x00003237, + 0x030, 0x00004207, + 0x030, 0x00005237, + 0x030, 0x00006237, + 0x030, 0x00007237, + 0x030, 0x00008207, + 0x030, 0x00009237, + 0x030, 0x0000A237, + 0x030, 0x0000B237, + 0x030, 0x0000C237, + 0x030, 0x0000D237, + 0x030, 0x0000E207, + 0x030, 0x0000F237, + 0x030, 0x00010237, + 0x030, 0x00011237, + 0x030, 0x00012207, + 0x030, 0x00013237, + 0x030, 0x00014237, + 0x030, 0x00015237, + 0x030, 0x00016207, + 0x030, 0x00017237, + 0x030, 0x00018207, + 0x030, 0x00019237, + 0xA0000000, 0x00000000, + 0x030, 0x00000233, + 0x030, 0x00001233, + 0x030, 0x00002233, + 0x030, 0x00003233, + 0x030, 0x00004203, + 0x030, 0x00005233, + 0x030, 0x00006233, + 0x030, 0x00007233, + 0x030, 0x00008203, + 0x030, 0x00009233, + 0x030, 0x0000A233, + 0x030, 0x0000B233, + 0x030, 0x0000C233, + 0x030, 0x0000D233, + 0x030, 0x0000E203, + 0x030, 0x0000F233, + 0x030, 0x00010233, + 0x030, 0x00011233, + 0x030, 0x00012203, + 0x030, 0x00013233, + 0x030, 0x00014233, + 0x030, 0x00015233, + 0x030, 0x00016203, + 0x030, 0x00017233, + 0x030, 0x00018203, + 0x030, 0x00019233, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x0EF, 0x00000080, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000334, + 0x030, 0x00001334, + 0x030, 0x00002334, + 0x030, 0x00003334, + 0x030, 0x00004334, + 0x030, 0x00005334, + 0x030, 0x00006334, + 0x030, 0x00007334, + 0x030, 0x00008334, + 0x030, 0x00009334, + 0x030, 0x0000A334, + 0x030, 0x0000B334, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000334, + 0x030, 0x00001334, + 0x030, 0x00002334, + 0x030, 0x00003334, + 0x030, 0x00004334, + 0x030, 0x00005334, + 0x030, 0x00006334, + 0x030, 0x00007334, + 0x030, 0x00008334, + 0x030, 0x00009334, + 0x030, 0x0000A334, + 0x030, 0x0000B334, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000334, + 0x030, 0x00001334, + 0x030, 0x00002334, + 0x030, 0x00003334, + 0x030, 0x00004334, + 0x030, 0x00005334, + 0x030, 0x00006334, + 0x030, 0x00007334, + 0x030, 0x00008334, + 0x030, 0x00009334, + 0x030, 0x0000A334, + 0x030, 0x0000B334, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000334, + 0x030, 0x00001334, + 0x030, 0x00002334, + 0x030, 0x00003334, + 0x030, 0x00004334, + 0x030, 0x00005334, + 0x030, 0x00006334, + 0x030, 0x00007334, + 0x030, 0x00008334, + 0x030, 0x00009334, + 0x030, 0x0000A334, + 0x030, 0x0000B334, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000334, + 0x030, 0x00001334, + 0x030, 0x00002334, + 0x030, 0x00003334, + 0x030, 0x00004334, + 0x030, 0x00005334, + 0x030, 0x00006334, + 0x030, 0x00007334, + 0x030, 0x00008334, + 0x030, 0x00009334, + 0x030, 0x0000A334, + 0x030, 0x0000B334, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000334, + 0x030, 0x00001334, + 0x030, 0x00002334, + 0x030, 0x00003334, + 0x030, 0x00004334, + 0x030, 0x00005334, + 0x030, 0x00006334, + 0x030, 0x00007334, + 0x030, 0x00008334, + 0x030, 0x00009334, + 0x030, 0x0000A334, + 0x030, 0x0000B334, + 0xA0000000, 0x00000000, + 0x030, 0x00000232, + 0x030, 0x00001232, + 0x030, 0x00002232, + 0x030, 0x00003232, + 0x030, 0x00004232, + 0x030, 0x00005232, + 0x030, 0x00006232, + 0x030, 0x00007232, + 0x030, 0x00008232, + 0x030, 0x00009232, + 0x030, 0x0000A232, + 0x030, 0x0000B232, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000770, + 0x030, 0x00001770, + 0x030, 0x00002440, + 0x030, 0x00003440, + 0x030, 0x00004330, + 0x030, 0x00005330, + 0x030, 0x00008770, + 0x030, 0x0000A440, + 0x030, 0x0000C330, + 0x0EF, 0x00000000, + 0x0EE, 0x00010000, + 0x033, 0x00000200, + 0x03F, 0x0000006A, + 0x033, 0x00000201, + 0x03F, 0x0000006D, + 0x033, 0x00000202, + 0x03F, 0x0000046A, + 0x033, 0x00000203, + 0x03F, 0x0000086A, + 0x033, 0x00000204, + 0x03F, 0x00000C89, + 0x033, 0x00000205, + 0x03F, 0x00000CE8, + 0x033, 0x00000206, + 0x03F, 0x00000CEB, + 0x033, 0x00000207, + 0x03F, 0x00000CEE, + 0x033, 0x00000208, + 0x03F, 0x00000CF1, + 0x033, 0x00000209, + 0x03F, 0x00000CF4, + 0x033, 0x0000020A, + 0x03F, 0x00000CF7, + 0x033, 0x00000280, + 0x03F, 0x0000006A, + 0x033, 0x00000281, + 0x03F, 0x0000006D, + 0x033, 0x00000282, + 0x03F, 0x0000046A, + 0x033, 0x00000283, + 0x03F, 0x0000086A, + 0x033, 0x00000284, + 0x03F, 0x00000C89, + 0x033, 0x00000285, + 0x03F, 0x00000CE8, + 0x033, 0x00000286, + 0x03F, 0x00000CEB, + 0x033, 0x00000287, + 0x03F, 0x00000CEE, + 0x033, 0x00000288, + 0x03F, 0x00000CF1, + 0x033, 0x00000289, + 0x03F, 0x00000CF4, + 0x033, 0x0000028A, + 0x03F, 0x00000CF7, + 0x033, 0x00000300, + 0x03F, 0x0000006A, + 0x033, 0x00000301, + 0x03F, 0x0000006D, + 0x033, 0x00000302, + 0x03F, 0x0000046A, + 0x033, 0x00000303, + 0x03F, 0x0000086A, + 0x033, 0x00000304, + 0x03F, 0x00000C89, + 0x033, 0x00000305, + 0x03F, 0x00000CE8, + 0x033, 0x00000306, + 0x03F, 0x00000CEB, + 0x033, 0x00000307, + 0x03F, 0x00000CEE, + 0x033, 0x00000308, + 0x03F, 0x00000CF1, + 0x033, 0x00000309, + 0x03F, 0x00000CF4, + 0x033, 0x0000030A, + 0x03F, 0x00000CF7, + 0x0EE, 0x00000000, + 0x051, 0x0003C800, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x052, 0x000902CA, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x052, 0x000902CA, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x052, 0x000902CA, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x052, 0x000902CA, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x052, 0x000902CA, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x052, 0x000902CA, + 0xA0000000, 0x00000000, + 0x052, 0x000942CA, + 0xB0000000, 0x00000000, + 0x053, 0x000090F9, + 0x054, 0x00088000, + 0x057, 0x0004C80A, + 0x0EF, 0x00000020, + 0x033, 0x00000000, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000001, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000002, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00030246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00030246, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000003, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000004, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000005, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00030246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00030246, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000006, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000007, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000008, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00030246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00030246, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000009, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x0000000A, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x0000000B, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00030246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00030246, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x0000000C, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x0000000D, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x0000000E, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00010E46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00030246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00030246, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x0000000F, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000010, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000011, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00030246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00030246, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000012, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000013, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000014, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00031E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00031E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000015, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000016, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000017, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00031E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00031E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000018, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000019, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x0000001A, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00031E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00031E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x0000001B, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x0000001C, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x0000001D, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00031E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00031E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x0000001E, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x0000001F, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000020, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00031E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00031E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000021, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000022, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000023, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00031E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00031E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000024, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000025, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000026, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00031E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00031E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000027, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000028, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x00000029, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00031E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00031E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x033, 0x0000002A, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0000EA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00021E46, + 0xA0000000, 0x00000000, + 0x03F, 0x00002A46, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x0EE, 0x00010000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000468, + 0x033, 0x00000061, + 0x03F, 0x00000868, + 0x033, 0x00000062, + 0x03F, 0x00000909, + 0x033, 0x00000063, + 0x03F, 0x00000D0A, + 0x033, 0x00000064, + 0x03F, 0x00000D4A, + 0x033, 0x00000065, + 0x03F, 0x00000D8B, + 0x033, 0x00000066, + 0x03F, 0x00000DEB, + 0x033, 0x00000067, + 0x03F, 0x00000DEE, + 0x033, 0x00000068, + 0x03F, 0x00000DF1, + 0x033, 0x00000069, + 0x03F, 0x00000DF4, + 0x033, 0x0000006A, + 0x03F, 0x00000DF7, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000468, + 0x033, 0x00000061, + 0x03F, 0x00000868, + 0x033, 0x00000062, + 0x03F, 0x00000909, + 0x033, 0x00000063, + 0x03F, 0x00000D0A, + 0x033, 0x00000064, + 0x03F, 0x00000D4A, + 0x033, 0x00000065, + 0x03F, 0x00000D8B, + 0x033, 0x00000066, + 0x03F, 0x00000DEB, + 0x033, 0x00000067, + 0x03F, 0x00000DEE, + 0x033, 0x00000068, + 0x03F, 0x00000DF1, + 0x033, 0x00000069, + 0x03F, 0x00000DF4, + 0x033, 0x0000006A, + 0x03F, 0x00000DF7, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000468, + 0x033, 0x00000061, + 0x03F, 0x00000868, + 0x033, 0x00000062, + 0x03F, 0x00000909, + 0x033, 0x00000063, + 0x03F, 0x00000D0A, + 0x033, 0x00000064, + 0x03F, 0x00000D4A, + 0x033, 0x00000065, + 0x03F, 0x00000D8B, + 0x033, 0x00000066, + 0x03F, 0x00000DEB, + 0x033, 0x00000067, + 0x03F, 0x00000DEE, + 0x033, 0x00000068, + 0x03F, 0x00000DF1, + 0x033, 0x00000069, + 0x03F, 0x00000DF4, + 0x033, 0x0000006A, + 0x03F, 0x00000DF7, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000468, + 0x033, 0x00000061, + 0x03F, 0x00000868, + 0x033, 0x00000062, + 0x03F, 0x00000909, + 0x033, 0x00000063, + 0x03F, 0x00000D0A, + 0x033, 0x00000064, + 0x03F, 0x00000D4A, + 0x033, 0x00000065, + 0x03F, 0x00000D8B, + 0x033, 0x00000066, + 0x03F, 0x00000DEB, + 0x033, 0x00000067, + 0x03F, 0x00000DEE, + 0x033, 0x00000068, + 0x03F, 0x00000DF1, + 0x033, 0x00000069, + 0x03F, 0x00000DF4, + 0x033, 0x0000006A, + 0x03F, 0x00000DF7, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000468, + 0x033, 0x00000061, + 0x03F, 0x00000868, + 0x033, 0x00000062, + 0x03F, 0x00000909, + 0x033, 0x00000063, + 0x03F, 0x00000D0A, + 0x033, 0x00000064, + 0x03F, 0x00000D4A, + 0x033, 0x00000065, + 0x03F, 0x00000D8B, + 0x033, 0x00000066, + 0x03F, 0x00000DEB, + 0x033, 0x00000067, + 0x03F, 0x00000DEE, + 0x033, 0x00000068, + 0x03F, 0x00000DF1, + 0x033, 0x00000069, + 0x03F, 0x00000DF4, + 0x033, 0x0000006A, + 0x03F, 0x00000DF7, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000468, + 0x033, 0x00000061, + 0x03F, 0x00000868, + 0x033, 0x00000062, + 0x03F, 0x00000909, + 0x033, 0x00000063, + 0x03F, 0x00000D0A, + 0x033, 0x00000064, + 0x03F, 0x00000D4A, + 0x033, 0x00000065, + 0x03F, 0x00000D8B, + 0x033, 0x00000066, + 0x03F, 0x00000DEB, + 0x033, 0x00000067, + 0x03F, 0x00000DEE, + 0x033, 0x00000068, + 0x03F, 0x00000DF1, + 0x033, 0x00000069, + 0x03F, 0x00000DF4, + 0x033, 0x0000006A, + 0x03F, 0x00000DF7, + 0xA0000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000487, + 0x033, 0x00000061, + 0x03F, 0x00000887, + 0x033, 0x00000062, + 0x03F, 0x00000947, + 0x033, 0x00000063, + 0x03F, 0x00000D48, + 0x033, 0x00000064, + 0x03F, 0x00000D88, + 0x033, 0x00000065, + 0x03F, 0x00000DE8, + 0x033, 0x00000066, + 0x03F, 0x00000DEB, + 0x033, 0x00000067, + 0x03F, 0x00000DEE, + 0x033, 0x00000068, + 0x03F, 0x00000DF1, + 0x033, 0x00000069, + 0x03F, 0x00000DF4, + 0x033, 0x0000006A, + 0x03F, 0x00000DF7, + 0xB0000000, 0x00000000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000468, + 0x033, 0x00000021, + 0x03F, 0x00000868, + 0x033, 0x00000022, + 0x03F, 0x00000909, + 0x033, 0x00000023, + 0x03F, 0x00000D0A, + 0x033, 0x00000024, + 0x03F, 0x00000D4A, + 0x033, 0x00000025, + 0x03F, 0x00000D8B, + 0x033, 0x00000026, + 0x03F, 0x00000DEB, + 0x033, 0x00000027, + 0x03F, 0x00000DEE, + 0x033, 0x00000028, + 0x03F, 0x00000DF1, + 0x033, 0x00000029, + 0x03F, 0x00000DF4, + 0x033, 0x0000002A, + 0x03F, 0x00000DF7, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000468, + 0x033, 0x00000021, + 0x03F, 0x00000868, + 0x033, 0x00000022, + 0x03F, 0x00000909, + 0x033, 0x00000023, + 0x03F, 0x00000D0A, + 0x033, 0x00000024, + 0x03F, 0x00000D4A, + 0x033, 0x00000025, + 0x03F, 0x00000D8B, + 0x033, 0x00000026, + 0x03F, 0x00000DEB, + 0x033, 0x00000027, + 0x03F, 0x00000DEE, + 0x033, 0x00000028, + 0x03F, 0x00000DF1, + 0x033, 0x00000029, + 0x03F, 0x00000DF4, + 0x033, 0x0000002A, + 0x03F, 0x00000DF7, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000468, + 0x033, 0x00000021, + 0x03F, 0x00000868, + 0x033, 0x00000022, + 0x03F, 0x00000909, + 0x033, 0x00000023, + 0x03F, 0x00000D0A, + 0x033, 0x00000024, + 0x03F, 0x00000D4A, + 0x033, 0x00000025, + 0x03F, 0x00000D8B, + 0x033, 0x00000026, + 0x03F, 0x00000DEB, + 0x033, 0x00000027, + 0x03F, 0x00000DEE, + 0x033, 0x00000028, + 0x03F, 0x00000DF1, + 0x033, 0x00000029, + 0x03F, 0x00000DF4, + 0x033, 0x0000002A, + 0x03F, 0x00000DF7, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000468, + 0x033, 0x00000021, + 0x03F, 0x00000868, + 0x033, 0x00000022, + 0x03F, 0x00000909, + 0x033, 0x00000023, + 0x03F, 0x00000D0A, + 0x033, 0x00000024, + 0x03F, 0x00000D4A, + 0x033, 0x00000025, + 0x03F, 0x00000D8B, + 0x033, 0x00000026, + 0x03F, 0x00000DEB, + 0x033, 0x00000027, + 0x03F, 0x00000DEE, + 0x033, 0x00000028, + 0x03F, 0x00000DF1, + 0x033, 0x00000029, + 0x03F, 0x00000DF4, + 0x033, 0x0000002A, + 0x03F, 0x00000DF7, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000468, + 0x033, 0x00000021, + 0x03F, 0x00000868, + 0x033, 0x00000022, + 0x03F, 0x00000909, + 0x033, 0x00000023, + 0x03F, 0x00000D0A, + 0x033, 0x00000024, + 0x03F, 0x00000D4A, + 0x033, 0x00000025, + 0x03F, 0x00000D8B, + 0x033, 0x00000026, + 0x03F, 0x00000DEB, + 0x033, 0x00000027, + 0x03F, 0x00000DEE, + 0x033, 0x00000028, + 0x03F, 0x00000DF1, + 0x033, 0x00000029, + 0x03F, 0x00000DF4, + 0x033, 0x0000002A, + 0x03F, 0x00000DF7, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000468, + 0x033, 0x00000021, + 0x03F, 0x00000868, + 0x033, 0x00000022, + 0x03F, 0x00000909, + 0x033, 0x00000023, + 0x03F, 0x00000D0A, + 0x033, 0x00000024, + 0x03F, 0x00000D4A, + 0x033, 0x00000025, + 0x03F, 0x00000D8B, + 0x033, 0x00000026, + 0x03F, 0x00000DEB, + 0x033, 0x00000027, + 0x03F, 0x00000DEE, + 0x033, 0x00000028, + 0x03F, 0x00000DF1, + 0x033, 0x00000029, + 0x03F, 0x00000DF4, + 0x033, 0x0000002A, + 0x03F, 0x00000DF7, + 0xA0000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000487, + 0x033, 0x00000021, + 0x03F, 0x00000887, + 0x033, 0x00000022, + 0x03F, 0x00000947, + 0x033, 0x00000023, + 0x03F, 0x00000D48, + 0x033, 0x00000024, + 0x03F, 0x00000D88, + 0x033, 0x00000025, + 0x03F, 0x00000DE8, + 0x033, 0x00000026, + 0x03F, 0x00000DEB, + 0x033, 0x00000027, + 0x03F, 0x00000DEE, + 0x033, 0x00000028, + 0x03F, 0x00000DF1, + 0x033, 0x00000029, + 0x03F, 0x00000DF4, + 0x033, 0x0000002A, + 0x03F, 0x00000DF7, + 0xB0000000, 0x00000000, + 0x0EE, 0x00000000, + 0x05C, 0x000FCC00, + 0x067, 0x0000A505, + 0x0D3, 0x00000542, + 0x043, 0x00005000, + 0x07F, 0x00000000, + 0x0B0, 0x0001F0FC, + 0x0B1, 0x0007DBE4, + 0x0B2, 0x00022400, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x0007C760, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x0007C760, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x0007C760, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x0007C760, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x0007C760, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x0007C760, + 0xA0000000, 0x00000000, + 0x0B3, 0x0007C760, + 0xB0000000, 0x00000000, + 0x0B4, 0x00099D40, + 0x0B5, 0x0004103F, + 0x0B6, 0x000187F8, + 0x0B7, 0x00030018, + 0x0BC, 0x00000008, + 0x0D3, 0x00000542, + 0x0DD, 0x00000500, + 0x0BB, 0x00040010, + 0x0B0, 0x0001F0FA, + 0x0FE, 0x00000000, + 0x0CA, 0x00080000, + 0x0CA, 0x00080001, + 0x0FE, 0x00000000, + 0x0B0, 0x0001F0F8, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x0007C700, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x0007C700, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x0007C700, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x0007C700, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x0007C700, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x0007C700, + 0xA0000000, 0x00000000, + 0x0B3, 0x0007C700, + 0xB0000000, 0x00000000, + 0x018, 0x0001B124, + 0xFFE, 0x00000000, + 0xFFE, 0x00000000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x0007C760, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x0007C760, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x0007C760, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x0007C760, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x0007C760, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x0B3, 0x0007C760, + 0xA0000000, 0x00000000, + 0x0B3, 0x0007C760, + 0xB0000000, 0x00000000, + 0x018, 0x00013124, + 0x0CC, 0x0000F000, + 0x0CD, 0x00089600, + 0x018, 0x00013108, + 0x0FE, 0x00000000, + 0x0B8, 0x000C0440, + 0x0BA, 0x000E840D, + 0x0FE, 0x00000000, + 0x018, 0x00013124, + 0x059, 0x000A0000, + 0x05A, 0x00060000, + 0x05B, 0x00014000, + 0x0ED, 0x00000008, + 0x033, 0x00000001, + 0x03F, 0x0000000F, + 0x0ED, 0x00000000, + 0x0EE, 0x00000002, + 0x033, 0x00000017, + 0x03F, 0x0000003F, + 0x033, 0x00000018, + 0x03F, 0x0000003F, + 0x033, 0x00000019, + 0x03F, 0x00000000, + 0x033, 0x0000001A, + 0x03F, 0x0000003F, + 0x033, 0x0000001B, + 0x03F, 0x0000003F, + 0x033, 0x0000001C, + 0x03F, 0x0000003F, + 0x0EE, 0x00000000, + 0x0ED, 0x00000200, + 0x033, 0x00000000, + 0x03F, 0x000F45A4, + 0x033, 0x00000001, + 0x03F, 0x000F49A4, + 0x033, 0x00000002, + 0x03F, 0x000F49A4, + 0x033, 0x00000003, + 0x03F, 0x000F69A4, + 0x033, 0x00000004, + 0x03F, 0x000F69A4, + 0x033, 0x00000005, + 0x03F, 0x000F69A4, + 0x033, 0x00000006, + 0x03F, 0x000F6DA4, + 0x033, 0x00000007, + 0x03F, 0x000F6DA4, + 0x033, 0x00000008, + 0x03F, 0x000F6DA4, + 0x033, 0x00000009, + 0x03F, 0x000F8DA4, + 0x033, 0x0000000A, + 0x03F, 0x000F8DA4, + 0x033, 0x0000000B, + 0x03F, 0x000F8DA4, + 0x033, 0x0000000C, + 0x03F, 0x000F91A4, + 0x033, 0x0000000D, + 0x03F, 0x000F91A4, + 0x033, 0x0000000E, + 0x03F, 0x000F91A4, + 0x033, 0x0000000F, + 0x03F, 0x000FB1A4, + 0x033, 0x00000010, + 0x03F, 0x000FB1A4, + 0x033, 0x00000011, + 0x03F, 0x000FB1A4, + 0x033, 0x00000012, + 0x03F, 0x000FB5A4, + 0x033, 0x00000013, + 0x03F, 0x000FB5A4, + 0x033, 0x00000014, + 0x03F, 0x000FD9A4, + 0x033, 0x00000015, + 0x03F, 0x000FD9A4, + 0x033, 0x00000016, + 0x03F, 0x000FF9A4, + 0x033, 0x00000017, + 0x03F, 0x000FF9A4, + 0x033, 0x00000018, + 0x03F, 0x000FFDA4, + 0x033, 0x00000019, + 0x03F, 0x000FFDA4, + 0x033, 0x0000001A, + 0x03F, 0x000FFDA4, + 0x0ED, 0x00000000, + 0x092, 0x00084800, + 0x092, 0x00084801, + 0x0FE, 0x00000000, + 0x0FE, 0x00000000, + 0x0FE, 0x00000000, + 0x0FE, 0x00000000, + 0x092, 0x00084800, + 0x08F, 0x0000182C, + 0x088, 0x0004326B, + 0x019, 0x00000005, +}; + +RTW_DECL_TABLE_RF_RADIO(rtw8822c_rf_a, A); + +static const u32 rtw8822c_rf_b[] = { + 0x000, 0x00030000, + 0x018, 0x00013124, + 0x093, 0x0008483F, + 0x0EF, 0x00080000, + 0x033, 0x00000001, + 0x03F, 0x00091020, + 0x0EF, 0x00000000, + 0x0DE, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x08E, 0x000B9140, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x08E, 0x000B9140, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x08E, 0x000A5540, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x08E, 0x000A5540, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x08E, 0x000A5540, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x08E, 0x000A5540, + 0xA0000000, 0x00000000, + 0x08E, 0x000A5540, + 0xB0000000, 0x00000000, + 0x081, 0x0000FC01, + 0x081, 0x0002FC01, + 0x081, 0x0003FC01, + 0x085, 0x0006A06C, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EE, 0x00000010, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000002, + 0x03F, 0x0000002A, + 0x0EE, 0x00000000, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EE, 0x00000010, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000002, + 0x03F, 0x0000002A, + 0x0EE, 0x00000000, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EE, 0x00000010, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000002, + 0x03F, 0x0000002A, + 0x0EE, 0x00000000, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EE, 0x00000010, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000002, + 0x03F, 0x0000002A, + 0x0EE, 0x00000000, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EE, 0x00000010, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000002, + 0x03F, 0x0000002A, + 0x0EE, 0x00000000, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EE, 0x00000010, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000001, + 0x03F, 0x0000002A, + 0x033, 0x00000002, + 0x03F, 0x0000002A, + 0x0EE, 0x00000000, + 0xA0000000, 0x00000000, + 0x0EE, 0x00000010, + 0x033, 0x00000001, + 0x03F, 0x0000003F, + 0x033, 0x00000001, + 0x03F, 0x0000003F, + 0x033, 0x00000002, + 0x03F, 0x0000003F, + 0x0EE, 0x00000000, + 0xB0000000, 0x00000000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00010000, + 0x033, 0x0000000F, + 0x03F, 0x000773C0, + 0x033, 0x0000000E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000000D, + 0x03F, 0x000773E8, + 0x033, 0x0000000C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000000B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000000A, + 0x03F, 0x000002A8, + 0x033, 0x00000009, + 0x03F, 0x00000280, + 0x033, 0x00000008, + 0x03F, 0x000FF280, + 0x033, 0x00000007, + 0x03F, 0x00000200, + 0x033, 0x00000006, + 0x03F, 0x000001C0, + 0x033, 0x00000005, + 0x03F, 0x00000180, + 0x033, 0x00000004, + 0x03F, 0x00000040, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00010000, + 0x033, 0x0000000F, + 0x03F, 0x000773C0, + 0x033, 0x0000000E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000000D, + 0x03F, 0x000773E8, + 0x033, 0x0000000C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000000B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000000A, + 0x03F, 0x000002A8, + 0x033, 0x00000009, + 0x03F, 0x00000280, + 0x033, 0x00000008, + 0x03F, 0x000FF280, + 0x033, 0x00000007, + 0x03F, 0x00000200, + 0x033, 0x00000006, + 0x03F, 0x000001C0, + 0x033, 0x00000005, + 0x03F, 0x00000180, + 0x033, 0x00000004, + 0x03F, 0x00000040, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00010000, + 0x033, 0x0000000F, + 0x03F, 0x000773C0, + 0x033, 0x0000000E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000000D, + 0x03F, 0x000773E8, + 0x033, 0x0000000C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000000B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000000A, + 0x03F, 0x000002A8, + 0x033, 0x00000009, + 0x03F, 0x00000280, + 0x033, 0x00000008, + 0x03F, 0x000FF280, + 0x033, 0x00000007, + 0x03F, 0x00000200, + 0x033, 0x00000006, + 0x03F, 0x000001C0, + 0x033, 0x00000005, + 0x03F, 0x00000180, + 0x033, 0x00000004, + 0x03F, 0x00000040, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00010000, + 0x033, 0x0000000F, + 0x03F, 0x000773C0, + 0x033, 0x0000000E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000000D, + 0x03F, 0x000773E8, + 0x033, 0x0000000C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000000B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000000A, + 0x03F, 0x000002A8, + 0x033, 0x00000009, + 0x03F, 0x00000280, + 0x033, 0x00000008, + 0x03F, 0x000FF280, + 0x033, 0x00000007, + 0x03F, 0x00000200, + 0x033, 0x00000006, + 0x03F, 0x000001C0, + 0x033, 0x00000005, + 0x03F, 0x00000180, + 0x033, 0x00000004, + 0x03F, 0x00000040, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00010000, + 0x033, 0x0000000F, + 0x03F, 0x000773C0, + 0x033, 0x0000000E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000000D, + 0x03F, 0x000773E8, + 0x033, 0x0000000C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000000B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000000A, + 0x03F, 0x000002A8, + 0x033, 0x00000009, + 0x03F, 0x00000280, + 0x033, 0x00000008, + 0x03F, 0x000FF280, + 0x033, 0x00000007, + 0x03F, 0x00000200, + 0x033, 0x00000006, + 0x03F, 0x000001C0, + 0x033, 0x00000005, + 0x03F, 0x00000180, + 0x033, 0x00000004, + 0x03F, 0x00000040, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00010000, + 0x033, 0x0000000F, + 0x03F, 0x000773C0, + 0x033, 0x0000000E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000000D, + 0x03F, 0x000773E8, + 0x033, 0x0000000C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000000B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000000A, + 0x03F, 0x000002A8, + 0x033, 0x00000009, + 0x03F, 0x00000280, + 0x033, 0x00000008, + 0x03F, 0x000FF280, + 0x033, 0x00000007, + 0x03F, 0x00000200, + 0x033, 0x00000006, + 0x03F, 0x000001C0, + 0x033, 0x00000005, + 0x03F, 0x00000180, + 0x033, 0x00000004, + 0x03F, 0x00000040, + 0xA0000000, 0x00000000, + 0x0EF, 0x00010000, + 0x033, 0x0000000F, + 0x03F, 0x000773E8, + 0x033, 0x0000000E, + 0x03F, 0x000FF3A0, + 0x033, 0x0000000D, + 0x03F, 0x00000380, + 0x033, 0x0000000C, + 0x03F, 0x000FF380, + 0x033, 0x0000000B, + 0x03F, 0x00000300, + 0x033, 0x0000000A, + 0x03F, 0x000002A8, + 0x033, 0x00000009, + 0x03F, 0x00000280, + 0x033, 0x00000008, + 0x03F, 0x000FF280, + 0x033, 0x00000007, + 0x03F, 0x00000200, + 0x033, 0x00000006, + 0x03F, 0x000001C0, + 0x033, 0x00000005, + 0x03F, 0x00000180, + 0x033, 0x00000004, + 0x03F, 0x00000040, + 0xB0000000, 0x00000000, + 0x033, 0x00000003, + 0x03F, 0x00000000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000001F, + 0x03F, 0x000773C0, + 0x033, 0x0000001E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000001D, + 0x03F, 0x000773E8, + 0x033, 0x0000001C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000001B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000001A, + 0x03F, 0x000002A8, + 0x033, 0x00000019, + 0x03F, 0x00000280, + 0x033, 0x00000018, + 0x03F, 0x000FF280, + 0x033, 0x00000017, + 0x03F, 0x00000200, + 0x033, 0x00000016, + 0x03F, 0x000001C0, + 0x033, 0x00000015, + 0x03F, 0x00000180, + 0x033, 0x00000014, + 0x03F, 0x00000040, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000001F, + 0x03F, 0x000773C0, + 0x033, 0x0000001E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000001D, + 0x03F, 0x000773E8, + 0x033, 0x0000001C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000001B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000001A, + 0x03F, 0x000002A8, + 0x033, 0x00000019, + 0x03F, 0x00000280, + 0x033, 0x00000018, + 0x03F, 0x000FF280, + 0x033, 0x00000017, + 0x03F, 0x00000200, + 0x033, 0x00000016, + 0x03F, 0x000001C0, + 0x033, 0x00000015, + 0x03F, 0x00000180, + 0x033, 0x00000014, + 0x03F, 0x00000040, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000001F, + 0x03F, 0x000773C0, + 0x033, 0x0000001E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000001D, + 0x03F, 0x000773E8, + 0x033, 0x0000001C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000001B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000001A, + 0x03F, 0x000002A8, + 0x033, 0x00000019, + 0x03F, 0x00000280, + 0x033, 0x00000018, + 0x03F, 0x000FF280, + 0x033, 0x00000017, + 0x03F, 0x00000200, + 0x033, 0x00000016, + 0x03F, 0x000001C0, + 0x033, 0x00000015, + 0x03F, 0x00000180, + 0x033, 0x00000014, + 0x03F, 0x00000040, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000001F, + 0x03F, 0x000773C0, + 0x033, 0x0000001E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000001D, + 0x03F, 0x000773E8, + 0x033, 0x0000001C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000001B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000001A, + 0x03F, 0x000002A8, + 0x033, 0x00000019, + 0x03F, 0x00000280, + 0x033, 0x00000018, + 0x03F, 0x000FF280, + 0x033, 0x00000017, + 0x03F, 0x00000200, + 0x033, 0x00000016, + 0x03F, 0x000001C0, + 0x033, 0x00000015, + 0x03F, 0x00000180, + 0x033, 0x00000014, + 0x03F, 0x00000040, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000001F, + 0x03F, 0x000773C0, + 0x033, 0x0000001E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000001D, + 0x03F, 0x000773E8, + 0x033, 0x0000001C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000001B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000001A, + 0x03F, 0x000002A8, + 0x033, 0x00000019, + 0x03F, 0x00000280, + 0x033, 0x00000018, + 0x03F, 0x000FF280, + 0x033, 0x00000017, + 0x03F, 0x00000200, + 0x033, 0x00000016, + 0x03F, 0x000001C0, + 0x033, 0x00000015, + 0x03F, 0x00000180, + 0x033, 0x00000014, + 0x03F, 0x00000040, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000001F, + 0x03F, 0x000773C0, + 0x033, 0x0000001E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000001D, + 0x03F, 0x000773E8, + 0x033, 0x0000001C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000001B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000001A, + 0x03F, 0x000002A8, + 0x033, 0x00000019, + 0x03F, 0x00000280, + 0x033, 0x00000018, + 0x03F, 0x000FF280, + 0x033, 0x00000017, + 0x03F, 0x00000200, + 0x033, 0x00000016, + 0x03F, 0x000001C0, + 0x033, 0x00000015, + 0x03F, 0x00000180, + 0x033, 0x00000014, + 0x03F, 0x00000040, + 0xA0000000, 0x00000000, + 0x033, 0x0000001F, + 0x03F, 0x000773E8, + 0x033, 0x0000001E, + 0x03F, 0x000FF3A0, + 0x033, 0x0000001D, + 0x03F, 0x00000380, + 0x033, 0x0000001C, + 0x03F, 0x000FF380, + 0x033, 0x0000001B, + 0x03F, 0x00000300, + 0x033, 0x0000001A, + 0x03F, 0x000002A8, + 0x033, 0x00000019, + 0x03F, 0x00000280, + 0x033, 0x00000018, + 0x03F, 0x000FF280, + 0x033, 0x00000017, + 0x03F, 0x00000200, + 0x033, 0x00000016, + 0x03F, 0x000001C0, + 0x033, 0x00000015, + 0x03F, 0x00000180, + 0x033, 0x00000014, + 0x03F, 0x00000040, + 0xB0000000, 0x00000000, + 0x033, 0x00000013, + 0x03F, 0x00000000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000002F, + 0x03F, 0x000773C0, + 0x033, 0x0000002E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000002D, + 0x03F, 0x000773E8, + 0x033, 0x0000002C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000002B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000002A, + 0x03F, 0x000002A8, + 0x033, 0x00000029, + 0x03F, 0x00000280, + 0x033, 0x00000028, + 0x03F, 0x000FF280, + 0x033, 0x00000027, + 0x03F, 0x00000200, + 0x033, 0x00000026, + 0x03F, 0x000001C0, + 0x033, 0x00000025, + 0x03F, 0x00000180, + 0x033, 0x00000024, + 0x03F, 0x00000040, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000002F, + 0x03F, 0x000773C0, + 0x033, 0x0000002E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000002D, + 0x03F, 0x000773E8, + 0x033, 0x0000002C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000002B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000002A, + 0x03F, 0x000002A8, + 0x033, 0x00000029, + 0x03F, 0x00000280, + 0x033, 0x00000028, + 0x03F, 0x000FF280, + 0x033, 0x00000027, + 0x03F, 0x00000200, + 0x033, 0x00000026, + 0x03F, 0x000001C0, + 0x033, 0x00000025, + 0x03F, 0x00000180, + 0x033, 0x00000024, + 0x03F, 0x00000040, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000002F, + 0x03F, 0x000773C0, + 0x033, 0x0000002E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000002D, + 0x03F, 0x000773E8, + 0x033, 0x0000002C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000002B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000002A, + 0x03F, 0x000002A8, + 0x033, 0x00000029, + 0x03F, 0x00000280, + 0x033, 0x00000028, + 0x03F, 0x000FF280, + 0x033, 0x00000027, + 0x03F, 0x00000200, + 0x033, 0x00000026, + 0x03F, 0x000001C0, + 0x033, 0x00000025, + 0x03F, 0x00000180, + 0x033, 0x00000024, + 0x03F, 0x00000040, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000002F, + 0x03F, 0x000773C0, + 0x033, 0x0000002E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000002D, + 0x03F, 0x000773E8, + 0x033, 0x0000002C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000002B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000002A, + 0x03F, 0x000002A8, + 0x033, 0x00000029, + 0x03F, 0x00000280, + 0x033, 0x00000028, + 0x03F, 0x000FF280, + 0x033, 0x00000027, + 0x03F, 0x00000200, + 0x033, 0x00000026, + 0x03F, 0x000001C0, + 0x033, 0x00000025, + 0x03F, 0x00000180, + 0x033, 0x00000024, + 0x03F, 0x00000040, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000002F, + 0x03F, 0x000773C0, + 0x033, 0x0000002E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000002D, + 0x03F, 0x000773E8, + 0x033, 0x0000002C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000002B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000002A, + 0x03F, 0x000002A8, + 0x033, 0x00000029, + 0x03F, 0x00000280, + 0x033, 0x00000028, + 0x03F, 0x000FF280, + 0x033, 0x00000027, + 0x03F, 0x00000200, + 0x033, 0x00000026, + 0x03F, 0x000001C0, + 0x033, 0x00000025, + 0x03F, 0x00000180, + 0x033, 0x00000024, + 0x03F, 0x00000040, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000002F, + 0x03F, 0x000773C0, + 0x033, 0x0000002E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000002D, + 0x03F, 0x000773E8, + 0x033, 0x0000002C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000002B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000002A, + 0x03F, 0x000002A8, + 0x033, 0x00000029, + 0x03F, 0x00000280, + 0x033, 0x00000028, + 0x03F, 0x000FF280, + 0x033, 0x00000027, + 0x03F, 0x00000200, + 0x033, 0x00000026, + 0x03F, 0x000001C0, + 0x033, 0x00000025, + 0x03F, 0x00000180, + 0x033, 0x00000024, + 0x03F, 0x00000040, + 0xA0000000, 0x00000000, + 0x033, 0x0000002F, + 0x03F, 0x000773E8, + 0x033, 0x0000002E, + 0x03F, 0x000FF3A0, + 0x033, 0x0000002D, + 0x03F, 0x00000380, + 0x033, 0x0000002C, + 0x03F, 0x000FF380, + 0x033, 0x0000002B, + 0x03F, 0x00000300, + 0x033, 0x0000002A, + 0x03F, 0x000002A8, + 0x033, 0x00000029, + 0x03F, 0x00000280, + 0x033, 0x00000028, + 0x03F, 0x000FF280, + 0x033, 0x00000027, + 0x03F, 0x00000200, + 0x033, 0x00000026, + 0x03F, 0x000001C0, + 0x033, 0x00000025, + 0x03F, 0x00000180, + 0x033, 0x00000024, + 0x03F, 0x00000040, + 0xB0000000, 0x00000000, + 0x033, 0x00000023, + 0x03F, 0x00000000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000003F, + 0x03F, 0x000773C0, + 0x033, 0x0000003E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000003D, + 0x03F, 0x000773E8, + 0x033, 0x0000003C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000003B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000003A, + 0x03F, 0x000002A8, + 0x033, 0x00000039, + 0x03F, 0x00000280, + 0x033, 0x00000038, + 0x03F, 0x000FF280, + 0x033, 0x00000037, + 0x03F, 0x00000200, + 0x033, 0x00000036, + 0x03F, 0x000001C0, + 0x033, 0x00000035, + 0x03F, 0x00000180, + 0x033, 0x00000034, + 0x03F, 0x00000040, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000003F, + 0x03F, 0x000773C0, + 0x033, 0x0000003E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000003D, + 0x03F, 0x000773E8, + 0x033, 0x0000003C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000003B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000003A, + 0x03F, 0x000002A8, + 0x033, 0x00000039, + 0x03F, 0x00000280, + 0x033, 0x00000038, + 0x03F, 0x000FF280, + 0x033, 0x00000037, + 0x03F, 0x00000200, + 0x033, 0x00000036, + 0x03F, 0x000001C0, + 0x033, 0x00000035, + 0x03F, 0x00000180, + 0x033, 0x00000034, + 0x03F, 0x00000040, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000003F, + 0x03F, 0x000773C0, + 0x033, 0x0000003E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000003D, + 0x03F, 0x000773E8, + 0x033, 0x0000003C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000003B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000003A, + 0x03F, 0x000002A8, + 0x033, 0x00000039, + 0x03F, 0x00000280, + 0x033, 0x00000038, + 0x03F, 0x000FF280, + 0x033, 0x00000037, + 0x03F, 0x00000200, + 0x033, 0x00000036, + 0x03F, 0x000001C0, + 0x033, 0x00000035, + 0x03F, 0x00000180, + 0x033, 0x00000034, + 0x03F, 0x00000040, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000003F, + 0x03F, 0x000773C0, + 0x033, 0x0000003E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000003D, + 0x03F, 0x000773E8, + 0x033, 0x0000003C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000003B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000003A, + 0x03F, 0x000002A8, + 0x033, 0x00000039, + 0x03F, 0x00000280, + 0x033, 0x00000038, + 0x03F, 0x000FF280, + 0x033, 0x00000037, + 0x03F, 0x00000200, + 0x033, 0x00000036, + 0x03F, 0x000001C0, + 0x033, 0x00000035, + 0x03F, 0x00000180, + 0x033, 0x00000034, + 0x03F, 0x00000040, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000003F, + 0x03F, 0x000773C0, + 0x033, 0x0000003E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000003D, + 0x03F, 0x000773E8, + 0x033, 0x0000003C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000003B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000003A, + 0x03F, 0x000002A8, + 0x033, 0x00000039, + 0x03F, 0x00000280, + 0x033, 0x00000038, + 0x03F, 0x000FF280, + 0x033, 0x00000037, + 0x03F, 0x00000200, + 0x033, 0x00000036, + 0x03F, 0x000001C0, + 0x033, 0x00000035, + 0x03F, 0x00000180, + 0x033, 0x00000034, + 0x03F, 0x00000040, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000003F, + 0x03F, 0x000773C0, + 0x033, 0x0000003E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000003D, + 0x03F, 0x000773E8, + 0x033, 0x0000003C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000003B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000003A, + 0x03F, 0x000002A8, + 0x033, 0x00000039, + 0x03F, 0x00000280, + 0x033, 0x00000038, + 0x03F, 0x000FF280, + 0x033, 0x00000037, + 0x03F, 0x00000200, + 0x033, 0x00000036, + 0x03F, 0x000001C0, + 0x033, 0x00000035, + 0x03F, 0x00000180, + 0x033, 0x00000034, + 0x03F, 0x00000040, + 0xA0000000, 0x00000000, + 0x033, 0x0000003F, + 0x03F, 0x000773E8, + 0x033, 0x0000003E, + 0x03F, 0x000FF3A0, + 0x033, 0x0000003D, + 0x03F, 0x00000380, + 0x033, 0x0000003C, + 0x03F, 0x000FF380, + 0x033, 0x0000003B, + 0x03F, 0x00000300, + 0x033, 0x0000003A, + 0x03F, 0x000002A8, + 0x033, 0x00000039, + 0x03F, 0x00000280, + 0x033, 0x00000038, + 0x03F, 0x000FF280, + 0x033, 0x00000037, + 0x03F, 0x00000200, + 0x033, 0x00000036, + 0x03F, 0x000001C0, + 0x033, 0x00000035, + 0x03F, 0x00000180, + 0x033, 0x00000034, + 0x03F, 0x00000040, + 0xB0000000, 0x00000000, + 0x033, 0x00000033, + 0x03F, 0x00000000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000004F, + 0x03F, 0x000773C0, + 0x033, 0x0000004E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000004D, + 0x03F, 0x000773E8, + 0x033, 0x0000004C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000004B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000004A, + 0x03F, 0x000002A8, + 0x033, 0x00000049, + 0x03F, 0x00000280, + 0x033, 0x00000048, + 0x03F, 0x000FF280, + 0x033, 0x00000047, + 0x03F, 0x00000200, + 0x033, 0x00000046, + 0x03F, 0x000001C0, + 0x033, 0x00000045, + 0x03F, 0x00000180, + 0x033, 0x00000044, + 0x03F, 0x00000040, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000004F, + 0x03F, 0x000773C0, + 0x033, 0x0000004E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000004D, + 0x03F, 0x000773E8, + 0x033, 0x0000004C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000004B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000004A, + 0x03F, 0x000002A8, + 0x033, 0x00000049, + 0x03F, 0x00000280, + 0x033, 0x00000048, + 0x03F, 0x000FF280, + 0x033, 0x00000047, + 0x03F, 0x00000200, + 0x033, 0x00000046, + 0x03F, 0x000001C0, + 0x033, 0x00000045, + 0x03F, 0x00000180, + 0x033, 0x00000044, + 0x03F, 0x00000040, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000004F, + 0x03F, 0x000773C0, + 0x033, 0x0000004E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000004D, + 0x03F, 0x000773E8, + 0x033, 0x0000004C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000004B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000004A, + 0x03F, 0x000002A8, + 0x033, 0x00000049, + 0x03F, 0x00000280, + 0x033, 0x00000048, + 0x03F, 0x000FF280, + 0x033, 0x00000047, + 0x03F, 0x00000200, + 0x033, 0x00000046, + 0x03F, 0x000001C0, + 0x033, 0x00000045, + 0x03F, 0x00000180, + 0x033, 0x00000044, + 0x03F, 0x00000040, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000004F, + 0x03F, 0x000773C0, + 0x033, 0x0000004E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000004D, + 0x03F, 0x000773E8, + 0x033, 0x0000004C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000004B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000004A, + 0x03F, 0x000002A8, + 0x033, 0x00000049, + 0x03F, 0x00000280, + 0x033, 0x00000048, + 0x03F, 0x000FF280, + 0x033, 0x00000047, + 0x03F, 0x00000200, + 0x033, 0x00000046, + 0x03F, 0x000001C0, + 0x033, 0x00000045, + 0x03F, 0x00000180, + 0x033, 0x00000044, + 0x03F, 0x00000040, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000004F, + 0x03F, 0x000773C0, + 0x033, 0x0000004E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000004D, + 0x03F, 0x000773E8, + 0x033, 0x0000004C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000004B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000004A, + 0x03F, 0x000002A8, + 0x033, 0x00000049, + 0x03F, 0x00000280, + 0x033, 0x00000048, + 0x03F, 0x000FF280, + 0x033, 0x00000047, + 0x03F, 0x00000200, + 0x033, 0x00000046, + 0x03F, 0x000001C0, + 0x033, 0x00000045, + 0x03F, 0x00000180, + 0x033, 0x00000044, + 0x03F, 0x00000040, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000004F, + 0x03F, 0x000773C0, + 0x033, 0x0000004E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000004D, + 0x03F, 0x000773E8, + 0x033, 0x0000004C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000004B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000004A, + 0x03F, 0x000002A8, + 0x033, 0x00000049, + 0x03F, 0x00000280, + 0x033, 0x00000048, + 0x03F, 0x000FF280, + 0x033, 0x00000047, + 0x03F, 0x00000200, + 0x033, 0x00000046, + 0x03F, 0x000001C0, + 0x033, 0x00000045, + 0x03F, 0x00000180, + 0x033, 0x00000044, + 0x03F, 0x00000040, + 0xA0000000, 0x00000000, + 0x033, 0x0000004F, + 0x03F, 0x000773E8, + 0x033, 0x0000004E, + 0x03F, 0x000FF3A0, + 0x033, 0x0000004D, + 0x03F, 0x00000380, + 0x033, 0x0000004C, + 0x03F, 0x000FF380, + 0x033, 0x0000004B, + 0x03F, 0x00000300, + 0x033, 0x0000004A, + 0x03F, 0x000002A8, + 0x033, 0x00000049, + 0x03F, 0x00000280, + 0x033, 0x00000048, + 0x03F, 0x000FF280, + 0x033, 0x00000047, + 0x03F, 0x00000200, + 0x033, 0x00000046, + 0x03F, 0x000001C0, + 0x033, 0x00000045, + 0x03F, 0x00000180, + 0x033, 0x00000044, + 0x03F, 0x00000040, + 0xB0000000, 0x00000000, + 0x033, 0x00000043, + 0x03F, 0x00000000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000005F, + 0x03F, 0x000773C0, + 0x033, 0x0000005E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000005D, + 0x03F, 0x000773E8, + 0x033, 0x0000005C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000005B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000005A, + 0x03F, 0x000002A8, + 0x033, 0x00000059, + 0x03F, 0x00000280, + 0x033, 0x00000058, + 0x03F, 0x000FF280, + 0x033, 0x00000057, + 0x03F, 0x00000200, + 0x033, 0x00000056, + 0x03F, 0x000001C0, + 0x033, 0x00000055, + 0x03F, 0x00000180, + 0x033, 0x00000054, + 0x03F, 0x00000040, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000005F, + 0x03F, 0x000773C0, + 0x033, 0x0000005E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000005D, + 0x03F, 0x000773E8, + 0x033, 0x0000005C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000005B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000005A, + 0x03F, 0x000002A8, + 0x033, 0x00000059, + 0x03F, 0x00000280, + 0x033, 0x00000058, + 0x03F, 0x000FF280, + 0x033, 0x00000057, + 0x03F, 0x00000200, + 0x033, 0x00000056, + 0x03F, 0x000001C0, + 0x033, 0x00000055, + 0x03F, 0x00000180, + 0x033, 0x00000054, + 0x03F, 0x00000040, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000005F, + 0x03F, 0x000773C0, + 0x033, 0x0000005E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000005D, + 0x03F, 0x000773E8, + 0x033, 0x0000005C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000005B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000005A, + 0x03F, 0x000002A8, + 0x033, 0x00000059, + 0x03F, 0x00000280, + 0x033, 0x00000058, + 0x03F, 0x000FF280, + 0x033, 0x00000057, + 0x03F, 0x00000200, + 0x033, 0x00000056, + 0x03F, 0x000001C0, + 0x033, 0x00000055, + 0x03F, 0x00000180, + 0x033, 0x00000054, + 0x03F, 0x00000040, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000005F, + 0x03F, 0x000773C0, + 0x033, 0x0000005E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000005D, + 0x03F, 0x000773E8, + 0x033, 0x0000005C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000005B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000005A, + 0x03F, 0x000002A8, + 0x033, 0x00000059, + 0x03F, 0x00000280, + 0x033, 0x00000058, + 0x03F, 0x000FF280, + 0x033, 0x00000057, + 0x03F, 0x00000200, + 0x033, 0x00000056, + 0x03F, 0x000001C0, + 0x033, 0x00000055, + 0x03F, 0x00000180, + 0x033, 0x00000054, + 0x03F, 0x00000040, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000005F, + 0x03F, 0x000773C0, + 0x033, 0x0000005E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000005D, + 0x03F, 0x000773E8, + 0x033, 0x0000005C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000005B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000005A, + 0x03F, 0x000002A8, + 0x033, 0x00000059, + 0x03F, 0x00000280, + 0x033, 0x00000058, + 0x03F, 0x000FF280, + 0x033, 0x00000057, + 0x03F, 0x00000200, + 0x033, 0x00000056, + 0x03F, 0x000001C0, + 0x033, 0x00000055, + 0x03F, 0x00000180, + 0x033, 0x00000054, + 0x03F, 0x00000040, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x0000005F, + 0x03F, 0x000773C0, + 0x033, 0x0000005E, + 0x03F, 0x000FF3C0, + 0x033, 0x0000005D, + 0x03F, 0x000773E8, + 0x033, 0x0000005C, + 0x03F, 0x000FF3E8, + 0x033, 0x0000005B, + 0x03F, 0x000FF3A0, + 0x033, 0x0000005A, + 0x03F, 0x000002A8, + 0x033, 0x00000059, + 0x03F, 0x00000280, + 0x033, 0x00000058, + 0x03F, 0x000FF280, + 0x033, 0x00000057, + 0x03F, 0x00000200, + 0x033, 0x00000056, + 0x03F, 0x000001C0, + 0x033, 0x00000055, + 0x03F, 0x00000180, + 0x033, 0x00000054, + 0x03F, 0x00000040, + 0xA0000000, 0x00000000, + 0x033, 0x0000005F, + 0x03F, 0x000773E8, + 0x033, 0x0000005E, + 0x03F, 0x000FF3A0, + 0x033, 0x0000005D, + 0x03F, 0x00000380, + 0x033, 0x0000005C, + 0x03F, 0x000FF380, + 0x033, 0x0000005B, + 0x03F, 0x00000300, + 0x033, 0x0000005A, + 0x03F, 0x000002A8, + 0x033, 0x00000059, + 0x03F, 0x00000280, + 0x033, 0x00000058, + 0x03F, 0x000FF280, + 0x033, 0x00000057, + 0x03F, 0x00000200, + 0x033, 0x00000056, + 0x03F, 0x000001C0, + 0x033, 0x00000055, + 0x03F, 0x00000180, + 0x033, 0x00000054, + 0x03F, 0x00000040, + 0xB0000000, 0x00000000, + 0x033, 0x00000053, + 0x03F, 0x00000000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000000, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000000, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000000, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000000, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000000, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00000000, + 0xA0000000, 0x00000000, + 0x0EF, 0x00000000, + 0xB0000000, 0x00000000, + 0x08A, 0x000E7DE3, + 0x08B, 0x0008FE00, + 0x0EE, 0x00000008, + 0x033, 0x00000000, + 0x03F, 0x00000023, + 0x033, 0x00000001, + 0x03F, 0x00000023, + 0x0EE, 0x00000000, + 0x0EF, 0x00004000, + 0x033, 0x00000000, + 0x03F, 0x0000000F, + 0x033, 0x00000002, + 0x03F, 0x00000000, + 0x0EF, 0x00000000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00020000, + 0x033, 0x00000000, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000001, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000002, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000003, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000004, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000005, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000006, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000007, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000008, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000009, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000000A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000000B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000000C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000000D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000000E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000000F, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000010, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000011, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000012, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000013, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000014, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000015, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000016, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000017, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000018, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000019, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000001A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000001B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000001C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000001D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000001E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000001F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000020, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000021, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000022, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000023, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000024, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000025, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000026, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000027, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000028, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000029, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000002A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000002B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000002C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000002D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000002E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000002F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x0EF, 0x00000000, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00020000, + 0x033, 0x00000000, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000001, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000002, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000003, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000004, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000005, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000006, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000007, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000008, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000009, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000000A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000000B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000000C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000000D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000000E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000000F, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000010, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000011, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000012, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000013, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000014, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000015, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000016, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000017, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000018, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000019, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000001A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000001B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000001C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000001D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000001E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000001F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000020, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000021, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000022, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000023, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000024, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000025, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000026, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000027, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000028, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000029, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000002A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000002B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000002C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000002D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000002E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000002F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x0EF, 0x00000000, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00020000, + 0x033, 0x00000000, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000001, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000002, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000003, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000004, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000005, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000006, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000007, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000008, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000009, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000000A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000000B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000000C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000000D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000000E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000000F, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000010, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000011, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000012, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000013, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000014, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000015, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000016, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000017, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000018, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000019, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000001A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000001B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000001C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000001D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000001E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000001F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000020, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000021, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000022, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000023, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000024, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000025, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000026, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000027, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000028, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000029, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000002A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000002B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000002C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000002D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000002E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000002F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x0EF, 0x00000000, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00020000, + 0x033, 0x00000000, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000001, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000002, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000003, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000004, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000005, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000006, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000007, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000008, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000009, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000000A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000000B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000000C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000000D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000000E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000000F, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000010, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000011, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000012, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000013, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000014, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000015, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000016, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000017, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000018, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000019, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000001A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000001B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000001C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000001D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000001E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000001F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000020, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000021, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000022, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000023, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000024, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000025, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000026, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000027, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000028, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000029, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000002A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000002B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000002C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000002D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000002E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000002F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x0EF, 0x00000000, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00020000, + 0x033, 0x00000000, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000001, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000002, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000003, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000004, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000005, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000006, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000007, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000008, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000009, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000000A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000000B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000000C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000000D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000000E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000000F, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000010, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000011, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000012, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000013, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000014, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000015, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000016, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000017, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000018, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000019, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000001A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000001B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000001C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000001D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000001E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000001F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000020, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000021, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000022, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000023, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000024, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000025, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000026, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000027, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000028, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000029, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000002A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000002B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000002C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000002D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000002E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000002F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x0EF, 0x00000000, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x0EF, 0x00020000, + 0x033, 0x00000000, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000001, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000002, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000003, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000004, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000005, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000006, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000007, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000008, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000009, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000000A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000000B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000000C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000000D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000000E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000000F, + 0x03E, 0x00000000, + 0x03F, 0x0002F81C, + 0x033, 0x00000010, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000011, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000012, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000013, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000014, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000015, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000016, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000017, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000018, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000019, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000001A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000001B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000001C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000001D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000001E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000001F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000020, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000021, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000022, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000023, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000024, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000025, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000026, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000027, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000028, + 0x03E, 0x00001C86, + 0x03F, 0x00020000, + 0x033, 0x00000029, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000002A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000002B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000002C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000002D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000002E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000002F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x0EF, 0x00000000, + 0xA0000000, 0x00000000, + 0x0EF, 0x00020000, + 0x033, 0x00000000, + 0x03E, 0x00001910, + 0x03F, 0x00020000, + 0x033, 0x00000001, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000002, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000003, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000004, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000005, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000006, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000007, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000008, + 0x03E, 0x00001910, + 0x03F, 0x00020000, + 0x033, 0x00000009, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000000A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000000B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000000C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000000D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000000E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000000F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000010, + 0x03E, 0x00001910, + 0x03F, 0x00020000, + 0x033, 0x00000011, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000012, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000013, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000014, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000015, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000016, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000017, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000018, + 0x03E, 0x00001910, + 0x03F, 0x00020000, + 0x033, 0x00000019, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000001A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000001B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000001C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000001D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000001E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000001F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000020, + 0x03E, 0x00001910, + 0x03F, 0x00020000, + 0x033, 0x00000021, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x00000022, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x00000023, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x00000024, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x00000025, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x00000026, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x00000027, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x033, 0x00000028, + 0x03E, 0x00001910, + 0x03F, 0x00020000, + 0x033, 0x00000029, + 0x03E, 0x00001C02, + 0x03F, 0x00020000, + 0x033, 0x0000002A, + 0x03E, 0x00000F02, + 0x03F, 0x00020000, + 0x033, 0x0000002B, + 0x03E, 0x00000F00, + 0x03F, 0x00020000, + 0x033, 0x0000002C, + 0x03E, 0x00000086, + 0x03F, 0x00020000, + 0x033, 0x0000002D, + 0x03E, 0x00000002, + 0x03F, 0x00020000, + 0x033, 0x0000002E, + 0x03E, 0x00000000, + 0x03F, 0x00020000, + 0x033, 0x0000002F, + 0x03E, 0x00000000, + 0x03F, 0x0002C010, + 0x0EF, 0x00000000, + 0xB0000000, 0x00000000, + 0x0FE, 0x00000000, + 0x01B, 0x00003A40, + 0x061, 0x0000D233, + 0x062, 0x0004D232, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x063, 0x00000002, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x063, 0x00000002, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x063, 0x00000002, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x063, 0x00000002, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x063, 0x00000002, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x063, 0x00000002, + 0xA0000000, 0x00000000, + 0x063, 0x00000C02, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000200, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000237, + 0x030, 0x00001237, + 0x030, 0x00002237, + 0x030, 0x00003237, + 0x030, 0x00004207, + 0x030, 0x00005237, + 0x030, 0x00006237, + 0x030, 0x00007237, + 0x030, 0x00008207, + 0x030, 0x00009237, + 0x030, 0x0000A237, + 0x030, 0x0000B237, + 0x030, 0x0000C237, + 0x030, 0x0000D237, + 0x030, 0x0000E207, + 0x030, 0x0000F237, + 0x030, 0x00010237, + 0x030, 0x00011237, + 0x030, 0x00012207, + 0x030, 0x00013237, + 0x030, 0x00014237, + 0x030, 0x00015237, + 0x030, 0x00016207, + 0x030, 0x00017237, + 0x030, 0x00018207, + 0x030, 0x00019237, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000237, + 0x030, 0x00001237, + 0x030, 0x00002237, + 0x030, 0x00003237, + 0x030, 0x00004207, + 0x030, 0x00005237, + 0x030, 0x00006237, + 0x030, 0x00007237, + 0x030, 0x00008207, + 0x030, 0x00009237, + 0x030, 0x0000A237, + 0x030, 0x0000B237, + 0x030, 0x0000C237, + 0x030, 0x0000D237, + 0x030, 0x0000E207, + 0x030, 0x0000F237, + 0x030, 0x00010237, + 0x030, 0x00011237, + 0x030, 0x00012207, + 0x030, 0x00013237, + 0x030, 0x00014237, + 0x030, 0x00015237, + 0x030, 0x00016207, + 0x030, 0x00017237, + 0x030, 0x00018207, + 0x030, 0x00019237, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000237, + 0x030, 0x00001237, + 0x030, 0x00002237, + 0x030, 0x00003237, + 0x030, 0x00004207, + 0x030, 0x00005237, + 0x030, 0x00006237, + 0x030, 0x00007237, + 0x030, 0x00008207, + 0x030, 0x00009237, + 0x030, 0x0000A237, + 0x030, 0x0000B237, + 0x030, 0x0000C237, + 0x030, 0x0000D237, + 0x030, 0x0000E207, + 0x030, 0x0000F237, + 0x030, 0x00010237, + 0x030, 0x00011237, + 0x030, 0x00012207, + 0x030, 0x00013237, + 0x030, 0x00014237, + 0x030, 0x00015237, + 0x030, 0x00016207, + 0x030, 0x00017237, + 0x030, 0x00018207, + 0x030, 0x00019237, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000237, + 0x030, 0x00001237, + 0x030, 0x00002237, + 0x030, 0x00003237, + 0x030, 0x00004207, + 0x030, 0x00005237, + 0x030, 0x00006237, + 0x030, 0x00007237, + 0x030, 0x00008207, + 0x030, 0x00009237, + 0x030, 0x0000A237, + 0x030, 0x0000B237, + 0x030, 0x0000C237, + 0x030, 0x0000D237, + 0x030, 0x0000E207, + 0x030, 0x0000F237, + 0x030, 0x00010237, + 0x030, 0x00011237, + 0x030, 0x00012207, + 0x030, 0x00013237, + 0x030, 0x00014237, + 0x030, 0x00015237, + 0x030, 0x00016207, + 0x030, 0x00017237, + 0x030, 0x00018207, + 0x030, 0x00019237, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000237, + 0x030, 0x00001237, + 0x030, 0x00002237, + 0x030, 0x00003237, + 0x030, 0x00004207, + 0x030, 0x00005237, + 0x030, 0x00006237, + 0x030, 0x00007237, + 0x030, 0x00008207, + 0x030, 0x00009237, + 0x030, 0x0000A237, + 0x030, 0x0000B237, + 0x030, 0x0000C237, + 0x030, 0x0000D237, + 0x030, 0x0000E207, + 0x030, 0x0000F237, + 0x030, 0x00010237, + 0x030, 0x00011237, + 0x030, 0x00012207, + 0x030, 0x00013237, + 0x030, 0x00014237, + 0x030, 0x00015237, + 0x030, 0x00016207, + 0x030, 0x00017237, + 0x030, 0x00018207, + 0x030, 0x00019237, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000237, + 0x030, 0x00001237, + 0x030, 0x00002237, + 0x030, 0x00003237, + 0x030, 0x00004207, + 0x030, 0x00005237, + 0x030, 0x00006237, + 0x030, 0x00007237, + 0x030, 0x00008207, + 0x030, 0x00009237, + 0x030, 0x0000A237, + 0x030, 0x0000B237, + 0x030, 0x0000C237, + 0x030, 0x0000D237, + 0x030, 0x0000E207, + 0x030, 0x0000F237, + 0x030, 0x00010237, + 0x030, 0x00011237, + 0x030, 0x00012207, + 0x030, 0x00013237, + 0x030, 0x00014237, + 0x030, 0x00015237, + 0x030, 0x00016207, + 0x030, 0x00017237, + 0x030, 0x00018207, + 0x030, 0x00019237, + 0xA0000000, 0x00000000, + 0x030, 0x00000233, + 0x030, 0x00001233, + 0x030, 0x00002233, + 0x030, 0x00003233, + 0x030, 0x00004203, + 0x030, 0x00005233, + 0x030, 0x00006233, + 0x030, 0x00007233, + 0x030, 0x00008203, + 0x030, 0x00009233, + 0x030, 0x0000A233, + 0x030, 0x0000B233, + 0x030, 0x0000C233, + 0x030, 0x0000D233, + 0x030, 0x0000E203, + 0x030, 0x0000F233, + 0x030, 0x00010233, + 0x030, 0x00011233, + 0x030, 0x00012203, + 0x030, 0x00013233, + 0x030, 0x00014233, + 0x030, 0x00015233, + 0x030, 0x00016203, + 0x030, 0x00017233, + 0x030, 0x00018203, + 0x030, 0x00019233, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x0EF, 0x00000080, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000334, + 0x030, 0x00001334, + 0x030, 0x00002334, + 0x030, 0x00003334, + 0x030, 0x00004334, + 0x030, 0x00005334, + 0x030, 0x00006334, + 0x030, 0x00007334, + 0x030, 0x00008334, + 0x030, 0x00009334, + 0x030, 0x0000A334, + 0x030, 0x0000B334, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000334, + 0x030, 0x00001334, + 0x030, 0x00002334, + 0x030, 0x00003334, + 0x030, 0x00004334, + 0x030, 0x00005334, + 0x030, 0x00006334, + 0x030, 0x00007334, + 0x030, 0x00008334, + 0x030, 0x00009334, + 0x030, 0x0000A334, + 0x030, 0x0000B334, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000334, + 0x030, 0x00001334, + 0x030, 0x00002334, + 0x030, 0x00003334, + 0x030, 0x00004334, + 0x030, 0x00005334, + 0x030, 0x00006334, + 0x030, 0x00007334, + 0x030, 0x00008334, + 0x030, 0x00009334, + 0x030, 0x0000A334, + 0x030, 0x0000B334, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000334, + 0x030, 0x00001334, + 0x030, 0x00002334, + 0x030, 0x00003334, + 0x030, 0x00004334, + 0x030, 0x00005334, + 0x030, 0x00006334, + 0x030, 0x00007334, + 0x030, 0x00008334, + 0x030, 0x00009334, + 0x030, 0x0000A334, + 0x030, 0x0000B334, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000334, + 0x030, 0x00001334, + 0x030, 0x00002334, + 0x030, 0x00003334, + 0x030, 0x00004334, + 0x030, 0x00005334, + 0x030, 0x00006334, + 0x030, 0x00007334, + 0x030, 0x00008334, + 0x030, 0x00009334, + 0x030, 0x0000A334, + 0x030, 0x0000B334, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x030, 0x00000334, + 0x030, 0x00001334, + 0x030, 0x00002334, + 0x030, 0x00003334, + 0x030, 0x00004334, + 0x030, 0x00005334, + 0x030, 0x00006334, + 0x030, 0x00007334, + 0x030, 0x00008334, + 0x030, 0x00009334, + 0x030, 0x0000A334, + 0x030, 0x0000B334, + 0xA0000000, 0x00000000, + 0x030, 0x00000232, + 0x030, 0x00001232, + 0x030, 0x00002232, + 0x030, 0x00003232, + 0x030, 0x00004232, + 0x030, 0x00005232, + 0x030, 0x00006232, + 0x030, 0x00007232, + 0x030, 0x00008232, + 0x030, 0x00009232, + 0x030, 0x0000A232, + 0x030, 0x0000B232, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x0EF, 0x00000040, + 0x030, 0x00000770, + 0x030, 0x00001770, + 0x030, 0x00002440, + 0x030, 0x00003440, + 0x030, 0x00004330, + 0x030, 0x00005330, + 0x030, 0x00008770, + 0x030, 0x0000A440, + 0x030, 0x0000C330, + 0x0EF, 0x00000000, + 0x0EE, 0x00010000, + 0x033, 0x00000200, + 0x03F, 0x0000006A, + 0x033, 0x00000201, + 0x03F, 0x0000006D, + 0x033, 0x00000202, + 0x03F, 0x0000046A, + 0x033, 0x00000203, + 0x03F, 0x0000086A, + 0x033, 0x00000204, + 0x03F, 0x00000C89, + 0x033, 0x00000205, + 0x03F, 0x00000CE8, + 0x033, 0x00000206, + 0x03F, 0x00000CEB, + 0x033, 0x00000207, + 0x03F, 0x00000CEE, + 0x033, 0x00000208, + 0x03F, 0x00000CF1, + 0x033, 0x00000209, + 0x03F, 0x00000CF4, + 0x033, 0x0000020A, + 0x03F, 0x00000CF7, + 0x033, 0x00000280, + 0x03F, 0x0000006A, + 0x033, 0x00000281, + 0x03F, 0x0000006D, + 0x033, 0x00000282, + 0x03F, 0x0000046A, + 0x033, 0x00000283, + 0x03F, 0x0000086A, + 0x033, 0x00000284, + 0x03F, 0x00000C89, + 0x033, 0x00000285, + 0x03F, 0x00000CE8, + 0x033, 0x00000286, + 0x03F, 0x00000CEB, + 0x033, 0x00000287, + 0x03F, 0x00000CEE, + 0x033, 0x00000288, + 0x03F, 0x00000CF1, + 0x033, 0x00000289, + 0x03F, 0x00000CF4, + 0x033, 0x0000028A, + 0x03F, 0x00000CF7, + 0x033, 0x00000300, + 0x03F, 0x0000006A, + 0x033, 0x00000301, + 0x03F, 0x0000006D, + 0x033, 0x00000302, + 0x03F, 0x0000046A, + 0x033, 0x00000303, + 0x03F, 0x0000086A, + 0x033, 0x00000304, + 0x03F, 0x00000C89, + 0x033, 0x00000305, + 0x03F, 0x00000CE8, + 0x033, 0x00000306, + 0x03F, 0x00000CEB, + 0x033, 0x00000307, + 0x03F, 0x00000CEE, + 0x033, 0x00000308, + 0x03F, 0x00000CF1, + 0x033, 0x00000309, + 0x03F, 0x00000CF4, + 0x033, 0x0000030A, + 0x03F, 0x00000CF7, + 0x0EE, 0x00000000, + 0x051, 0x0003C800, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x052, 0x000902CA, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x052, 0x000902CA, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x052, 0x000902CA, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x052, 0x000902CA, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x052, 0x000902CA, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x052, 0x000902CA, + 0xA0000000, 0x00000000, + 0x052, 0x000942C0, + 0xB0000000, 0x00000000, + 0x053, 0x000090F9, + 0x054, 0x00088000, + 0x057, 0x0004C80A, + 0x0EF, 0x00000020, + 0x033, 0x00000000, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x0000C246, + 0xB0000000, 0x00000000, + 0x033, 0x00000001, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x0000C246, + 0xB0000000, 0x00000000, + 0x033, 0x00000002, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002C246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002C246, + 0xA0000000, 0x00000000, + 0x03F, 0x0000C246, + 0xB0000000, 0x00000000, + 0x033, 0x00000003, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x0000C246, + 0xB0000000, 0x00000000, + 0x033, 0x00000004, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x0000C246, + 0xB0000000, 0x00000000, + 0x033, 0x00000005, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002C246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002C246, + 0xA0000000, 0x00000000, + 0x03F, 0x0000C246, + 0xB0000000, 0x00000000, + 0x033, 0x00000006, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x0000C246, + 0xB0000000, 0x00000000, + 0x033, 0x00000007, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x0000C246, + 0xB0000000, 0x00000000, + 0x033, 0x00000008, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002C246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002C246, + 0xA0000000, 0x00000000, + 0x03F, 0x0000C246, + 0xB0000000, 0x00000000, + 0x033, 0x00000009, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x0000000A, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x0000000B, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002C246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002C246, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x0000000C, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x0000000D, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x0000000E, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002C246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002C246, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x0000000F, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x00000010, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x00000011, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x00024246, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002C246, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002C246, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x00000012, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x00000013, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x00000014, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x00000015, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x00000016, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x00000017, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x00000018, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x00000019, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x0000001A, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x0000001B, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x0000001C, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x0000001D, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x0000001E, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x0000001F, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x00000020, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x00000021, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x00000022, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x00000023, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x00000024, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x00000025, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x00000026, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x00000027, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x00000028, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x00000029, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0002CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x033, 0x0000002A, + 0x03E, 0x00000020, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x03F, 0x0001CA46, + 0xA0000000, 0x00000000, + 0x03F, 0x00008E46, + 0xB0000000, 0x00000000, + 0x0EF, 0x00000000, + 0x0EE, 0x00010000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000468, + 0x033, 0x00000061, + 0x03F, 0x00000868, + 0x033, 0x00000062, + 0x03F, 0x00000909, + 0x033, 0x00000063, + 0x03F, 0x00000D0A, + 0x033, 0x00000064, + 0x03F, 0x00000D4A, + 0x033, 0x00000065, + 0x03F, 0x00000D8B, + 0x033, 0x00000066, + 0x03F, 0x00000DEB, + 0x033, 0x00000067, + 0x03F, 0x00000DEE, + 0x033, 0x00000068, + 0x03F, 0x00000DF1, + 0x033, 0x00000069, + 0x03F, 0x00000DF4, + 0x033, 0x0000006A, + 0x03F, 0x00000DF7, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000468, + 0x033, 0x00000061, + 0x03F, 0x00000868, + 0x033, 0x00000062, + 0x03F, 0x00000909, + 0x033, 0x00000063, + 0x03F, 0x00000D0A, + 0x033, 0x00000064, + 0x03F, 0x00000D4A, + 0x033, 0x00000065, + 0x03F, 0x00000D8B, + 0x033, 0x00000066, + 0x03F, 0x00000DEB, + 0x033, 0x00000067, + 0x03F, 0x00000DEE, + 0x033, 0x00000068, + 0x03F, 0x00000DF1, + 0x033, 0x00000069, + 0x03F, 0x00000DF4, + 0x033, 0x0000006A, + 0x03F, 0x00000DF7, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000468, + 0x033, 0x00000061, + 0x03F, 0x00000868, + 0x033, 0x00000062, + 0x03F, 0x00000909, + 0x033, 0x00000063, + 0x03F, 0x00000D0A, + 0x033, 0x00000064, + 0x03F, 0x00000D4A, + 0x033, 0x00000065, + 0x03F, 0x00000D8B, + 0x033, 0x00000066, + 0x03F, 0x00000DEB, + 0x033, 0x00000067, + 0x03F, 0x00000DEE, + 0x033, 0x00000068, + 0x03F, 0x00000DF1, + 0x033, 0x00000069, + 0x03F, 0x00000DF4, + 0x033, 0x0000006A, + 0x03F, 0x00000DF7, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000468, + 0x033, 0x00000061, + 0x03F, 0x00000868, + 0x033, 0x00000062, + 0x03F, 0x00000909, + 0x033, 0x00000063, + 0x03F, 0x00000D0A, + 0x033, 0x00000064, + 0x03F, 0x00000D4A, + 0x033, 0x00000065, + 0x03F, 0x00000D8B, + 0x033, 0x00000066, + 0x03F, 0x00000DEB, + 0x033, 0x00000067, + 0x03F, 0x00000DEE, + 0x033, 0x00000068, + 0x03F, 0x00000DF1, + 0x033, 0x00000069, + 0x03F, 0x00000DF4, + 0x033, 0x0000006A, + 0x03F, 0x00000DF7, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000468, + 0x033, 0x00000061, + 0x03F, 0x00000868, + 0x033, 0x00000062, + 0x03F, 0x00000909, + 0x033, 0x00000063, + 0x03F, 0x00000D0A, + 0x033, 0x00000064, + 0x03F, 0x00000D4A, + 0x033, 0x00000065, + 0x03F, 0x00000D8B, + 0x033, 0x00000066, + 0x03F, 0x00000DEB, + 0x033, 0x00000067, + 0x03F, 0x00000DEE, + 0x033, 0x00000068, + 0x03F, 0x00000DF1, + 0x033, 0x00000069, + 0x03F, 0x00000DF4, + 0x033, 0x0000006A, + 0x03F, 0x00000DF7, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000468, + 0x033, 0x00000061, + 0x03F, 0x00000868, + 0x033, 0x00000062, + 0x03F, 0x00000909, + 0x033, 0x00000063, + 0x03F, 0x00000D0A, + 0x033, 0x00000064, + 0x03F, 0x00000D4A, + 0x033, 0x00000065, + 0x03F, 0x00000D8B, + 0x033, 0x00000066, + 0x03F, 0x00000DEB, + 0x033, 0x00000067, + 0x03F, 0x00000DEE, + 0x033, 0x00000068, + 0x03F, 0x00000DF1, + 0x033, 0x00000069, + 0x03F, 0x00000DF4, + 0x033, 0x0000006A, + 0x03F, 0x00000DF7, + 0xA0000000, 0x00000000, + 0x033, 0x00000060, + 0x03F, 0x00000487, + 0x033, 0x00000061, + 0x03F, 0x00000887, + 0x033, 0x00000062, + 0x03F, 0x00000947, + 0x033, 0x00000063, + 0x03F, 0x00000D48, + 0x033, 0x00000064, + 0x03F, 0x00000D88, + 0x033, 0x00000065, + 0x03F, 0x00000DE8, + 0x033, 0x00000066, + 0x03F, 0x00000DEB, + 0x033, 0x00000067, + 0x03F, 0x00000DEE, + 0x033, 0x00000068, + 0x03F, 0x00000DF1, + 0x033, 0x00000069, + 0x03F, 0x00000DF4, + 0x033, 0x0000006A, + 0x03F, 0x00000DF7, + 0xB0000000, 0x00000000, + 0x81000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000468, + 0x033, 0x00000021, + 0x03F, 0x00000868, + 0x033, 0x00000022, + 0x03F, 0x00000909, + 0x033, 0x00000023, + 0x03F, 0x00000D0A, + 0x033, 0x00000024, + 0x03F, 0x00000D4A, + 0x033, 0x00000025, + 0x03F, 0x00000D8B, + 0x033, 0x00000026, + 0x03F, 0x00000DEB, + 0x033, 0x00000027, + 0x03F, 0x00000DEE, + 0x033, 0x00000028, + 0x03F, 0x00000DF1, + 0x033, 0x00000029, + 0x03F, 0x00000DF4, + 0x033, 0x0000002A, + 0x03F, 0x00000DF7, + 0x91000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000468, + 0x033, 0x00000021, + 0x03F, 0x00000868, + 0x033, 0x00000022, + 0x03F, 0x00000909, + 0x033, 0x00000023, + 0x03F, 0x00000D0A, + 0x033, 0x00000024, + 0x03F, 0x00000D4A, + 0x033, 0x00000025, + 0x03F, 0x00000D8B, + 0x033, 0x00000026, + 0x03F, 0x00000DEB, + 0x033, 0x00000027, + 0x03F, 0x00000DEE, + 0x033, 0x00000028, + 0x03F, 0x00000DF1, + 0x033, 0x00000029, + 0x03F, 0x00000DF4, + 0x033, 0x0000002A, + 0x03F, 0x00000DF7, + 0x92000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000468, + 0x033, 0x00000021, + 0x03F, 0x00000868, + 0x033, 0x00000022, + 0x03F, 0x00000909, + 0x033, 0x00000023, + 0x03F, 0x00000D0A, + 0x033, 0x00000024, + 0x03F, 0x00000D4A, + 0x033, 0x00000025, + 0x03F, 0x00000D8B, + 0x033, 0x00000026, + 0x03F, 0x00000DEB, + 0x033, 0x00000027, + 0x03F, 0x00000DEE, + 0x033, 0x00000028, + 0x03F, 0x00000DF1, + 0x033, 0x00000029, + 0x03F, 0x00000DF4, + 0x033, 0x0000002A, + 0x03F, 0x00000DF7, + 0x92000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000468, + 0x033, 0x00000021, + 0x03F, 0x00000868, + 0x033, 0x00000022, + 0x03F, 0x00000909, + 0x033, 0x00000023, + 0x03F, 0x00000D0A, + 0x033, 0x00000024, + 0x03F, 0x00000D4A, + 0x033, 0x00000025, + 0x03F, 0x00000D8B, + 0x033, 0x00000026, + 0x03F, 0x00000DEB, + 0x033, 0x00000027, + 0x03F, 0x00000DEE, + 0x033, 0x00000028, + 0x03F, 0x00000DF1, + 0x033, 0x00000029, + 0x03F, 0x00000DF4, + 0x033, 0x0000002A, + 0x03F, 0x00000DF7, + 0x93000001, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000468, + 0x033, 0x00000021, + 0x03F, 0x00000868, + 0x033, 0x00000022, + 0x03F, 0x00000909, + 0x033, 0x00000023, + 0x03F, 0x00000D0A, + 0x033, 0x00000024, + 0x03F, 0x00000D4A, + 0x033, 0x00000025, + 0x03F, 0x00000D8B, + 0x033, 0x00000026, + 0x03F, 0x00000DEB, + 0x033, 0x00000027, + 0x03F, 0x00000DEE, + 0x033, 0x00000028, + 0x03F, 0x00000DF1, + 0x033, 0x00000029, + 0x03F, 0x00000DF4, + 0x033, 0x0000002A, + 0x03F, 0x00000DF7, + 0x93000002, 0x00000000, 0x40000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000468, + 0x033, 0x00000021, + 0x03F, 0x00000868, + 0x033, 0x00000022, + 0x03F, 0x00000909, + 0x033, 0x00000023, + 0x03F, 0x00000D0A, + 0x033, 0x00000024, + 0x03F, 0x00000D4A, + 0x033, 0x00000025, + 0x03F, 0x00000D8B, + 0x033, 0x00000026, + 0x03F, 0x00000DEB, + 0x033, 0x00000027, + 0x03F, 0x00000DEE, + 0x033, 0x00000028, + 0x03F, 0x00000DF1, + 0x033, 0x00000029, + 0x03F, 0x00000DF4, + 0x033, 0x0000002A, + 0x03F, 0x00000DF7, + 0xA0000000, 0x00000000, + 0x033, 0x00000020, + 0x03F, 0x00000487, + 0x033, 0x00000021, + 0x03F, 0x00000887, + 0x033, 0x00000022, + 0x03F, 0x00000947, + 0x033, 0x00000023, + 0x03F, 0x00000D48, + 0x033, 0x00000024, + 0x03F, 0x00000D88, + 0x033, 0x00000025, + 0x03F, 0x00000DE8, + 0x033, 0x00000026, + 0x03F, 0x00000DEB, + 0x033, 0x00000027, + 0x03F, 0x00000DEE, + 0x033, 0x00000028, + 0x03F, 0x00000DF1, + 0x033, 0x00000029, + 0x03F, 0x00000DF4, + 0x033, 0x0000002A, + 0x03F, 0x00000DF7, + 0xB0000000, 0x00000000, + 0x0EE, 0x00000000, + 0x05C, 0x000FCC00, + 0x067, 0x0000A505, + 0x0D3, 0x00000542, + 0x043, 0x00005000, + 0x059, 0x000A0000, + 0x05A, 0x00060000, + 0x05B, 0x00014000, + 0x001, 0x00040000, + 0x0EE, 0x00000002, + 0x033, 0x00000017, + 0x03F, 0x0000003F, + 0x033, 0x00000018, + 0x03F, 0x0000003F, + 0x033, 0x00000019, + 0x03F, 0x00000000, + 0x033, 0x0000001A, + 0x03F, 0x0000003F, + 0x033, 0x0000001B, + 0x03F, 0x0000003F, + 0x033, 0x0000001C, + 0x03F, 0x0000003F, + 0x0EE, 0x00000000, + 0x092, 0x00084800, + 0x092, 0x00084801, + 0x0FE, 0x00000000, + 0x0FE, 0x00000000, + 0x0FE, 0x00000000, + 0x0FE, 0x00000000, + 0x092, 0x00084800, + 0x08F, 0x0000182C, + 0x088, 0x0004326B, + 0x019, 0x00000005, +}; + +RTW_DECL_TABLE_RF_RADIO(rtw8822c_rf_b, B); + +static const u8 rtw8822c_txpwr_lmt_type0[] = { + 0, 0, 0, 0, 1, 72, 2, 0, 0, 0, 1, 60, + 0, 0, 0, 0, 2, 72, 2, 0, 0, 0, 2, 60, + 0, 0, 0, 0, 3, 76, 2, 0, 0, 0, 3, 60, + 0, 0, 0, 0, 4, 76, 2, 0, 0, 0, 4, 60, + 0, 0, 0, 0, 5, 76, 2, 0, 0, 0, 5, 60, + 0, 0, 0, 0, 6, 76, 2, 0, 0, 0, 6, 60, + 0, 0, 0, 0, 7, 76, 2, 0, 0, 0, 7, 60, + 0, 0, 0, 0, 8, 76, 2, 0, 0, 0, 8, 60, + 0, 0, 0, 0, 9, 76, 2, 0, 0, 0, 9, 60, + 0, 0, 0, 0, 10, 72, 2, 0, 0, 0, 10, 60, + 0, 0, 0, 0, 11, 72, 2, 0, 0, 0, 11, 60, + 0, 0, 0, 0, 12, 52, 2, 0, 0, 0, 12, 60, + 0, 0, 0, 0, 13, 48, 2, 0, 0, 0, 13, 60, + 0, 0, 0, 0, 14, 127, 2, 0, 0, 0, 14, 127, + 0, 0, 0, 1, 1, 52, 2, 0, 0, 1, 1, 60, + 0, 0, 0, 1, 2, 60, 2, 0, 0, 1, 2, 60, + 0, 0, 0, 1, 3, 64, 2, 0, 0, 1, 3, 60, + 0, 0, 0, 1, 4, 68, 2, 0, 0, 1, 4, 60, + 0, 0, 0, 1, 5, 76, 2, 0, 0, 1, 5, 60, + 0, 0, 0, 1, 6, 76, 2, 0, 0, 1, 6, 60, + 0, 0, 0, 1, 7, 76, 2, 0, 0, 1, 7, 60, + 0, 0, 0, 1, 8, 68, 2, 0, 0, 1, 8, 60, + 0, 0, 0, 1, 9, 64, 2, 0, 0, 1, 9, 60, + 0, 0, 0, 1, 10, 60, 2, 0, 0, 1, 10, 60, + 0, 0, 0, 1, 11, 52, 2, 0, 0, 1, 11, 60, + 0, 0, 0, 1, 12, 40, 2, 0, 0, 1, 12, 60, + 0, 0, 0, 1, 13, 28, 2, 0, 0, 1, 13, 60, + 0, 0, 0, 1, 14, 127, 2, 0, 0, 1, 14, 127, + 0, 0, 0, 2, 1, 52, 2, 0, 0, 2, 1, 60, + 0, 0, 0, 2, 2, 60, 2, 0, 0, 2, 2, 60, + 0, 0, 0, 2, 3, 64, 2, 0, 0, 2, 3, 60, + 0, 0, 0, 2, 4, 68, 2, 0, 0, 2, 4, 60, + 0, 0, 0, 2, 5, 76, 2, 0, 0, 2, 5, 60, + 0, 0, 0, 2, 6, 76, 2, 0, 0, 2, 6, 60, + 0, 0, 0, 2, 7, 76, 2, 0, 0, 2, 7, 60, + 0, 0, 0, 2, 8, 68, 2, 0, 0, 2, 8, 60, + 0, 0, 0, 2, 9, 64, 2, 0, 0, 2, 9, 60, + 0, 0, 0, 2, 10, 60, 2, 0, 0, 2, 10, 60, + 0, 0, 0, 2, 11, 52, 2, 0, 0, 2, 11, 60, + 0, 0, 0, 2, 12, 40, 2, 0, 0, 2, 12, 60, + 0, 0, 0, 2, 13, 28, 2, 0, 0, 2, 13, 60, + 0, 0, 0, 2, 14, 127, 2, 0, 0, 2, 14, 127, + 0, 0, 0, 3, 1, 52, 2, 0, 0, 3, 1, 36, + 0, 0, 0, 3, 2, 60, 2, 0, 0, 3, 2, 36, + 0, 0, 0, 3, 3, 64, 2, 0, 0, 3, 3, 36, + 0, 0, 0, 3, 4, 68, 2, 0, 0, 3, 4, 36, + 0, 0, 0, 3, 5, 76, 2, 0, 0, 3, 5, 36, + 0, 0, 0, 3, 6, 76, 2, 0, 0, 3, 6, 36, + 0, 0, 0, 3, 7, 76, 2, 0, 0, 3, 7, 36, + 0, 0, 0, 3, 8, 68, 2, 0, 0, 3, 8, 36, + 0, 0, 0, 3, 9, 64, 2, 0, 0, 3, 9, 36, + 0, 0, 0, 3, 10, 60, 2, 0, 0, 3, 10, 36, + 0, 0, 0, 3, 11, 52, 2, 0, 0, 3, 11, 36, + 0, 0, 0, 3, 12, 40, 2, 0, 0, 3, 12, 36, + 0, 0, 0, 3, 13, 28, 2, 0, 0, 3, 13, 36, + 0, 0, 0, 3, 14, 127, 2, 0, 0, 3, 14, 127, + 0, 0, 1, 2, 1, 127, 2, 0, 1, 2, 1, 127, + 0, 0, 1, 2, 2, 127, 2, 0, 1, 2, 2, 127, + 0, 0, 1, 2, 3, 52, 2, 0, 1, 2, 3, 60, + 0, 0, 1, 2, 4, 52, 2, 0, 1, 2, 4, 60, + 0, 0, 1, 2, 5, 60, 2, 0, 1, 2, 5, 60, + 0, 0, 1, 2, 6, 64, 2, 0, 1, 2, 6, 60, + 0, 0, 1, 2, 7, 60, 2, 0, 1, 2, 7, 60, + 0, 0, 1, 2, 8, 52, 2, 0, 1, 2, 8, 60, + 0, 0, 1, 2, 9, 52, 2, 0, 1, 2, 9, 60, + 0, 0, 1, 2, 10, 40, 2, 0, 1, 2, 10, 60, + 0, 0, 1, 2, 11, 28, 2, 0, 1, 2, 11, 60, + 0, 0, 1, 2, 12, 127, 2, 0, 1, 2, 12, 127, + 0, 0, 1, 2, 13, 127, 2, 0, 1, 2, 13, 127, + 0, 0, 1, 2, 14, 127, 2, 0, 1, 2, 14, 127, + 0, 0, 1, 3, 1, 127, 2, 0, 1, 3, 1, 127, + 0, 0, 1, 3, 2, 127, 2, 0, 1, 3, 2, 127, + 0, 0, 1, 3, 3, 48, 2, 0, 1, 3, 3, 36, + 0, 0, 1, 3, 4, 48, 2, 0, 1, 3, 4, 36, + 0, 0, 1, 3, 5, 60, 2, 0, 1, 3, 5, 36, + 0, 0, 1, 3, 6, 64, 2, 0, 1, 3, 6, 36, + 0, 0, 1, 3, 7, 60, 2, 0, 1, 3, 7, 36, + 0, 0, 1, 3, 8, 52, 2, 0, 1, 3, 8, 36, + 0, 0, 1, 3, 9, 52, 2, 0, 1, 3, 9, 36, + 0, 0, 1, 3, 10, 40, 2, 0, 1, 3, 10, 36, + 0, 0, 1, 3, 11, 26, 2, 0, 1, 3, 11, 36, + 0, 0, 1, 3, 12, 127, 2, 0, 1, 3, 12, 127, + 0, 0, 1, 3, 13, 127, 2, 0, 1, 3, 13, 127, + 0, 0, 1, 3, 14, 127, 2, 0, 1, 3, 14, 127, + 0, 1, 0, 1, 36, 74, 2, 1, 0, 1, 36, 62, + 0, 1, 0, 1, 40, 80, 2, 1, 0, 1, 40, 62, + 0, 1, 0, 1, 44, 80, 2, 1, 0, 1, 44, 62, + 0, 1, 0, 1, 48, 80, 2, 1, 0, 1, 48, 62, + 0, 1, 0, 1, 52, 80, 2, 1, 0, 1, 52, 62, + 0, 1, 0, 1, 56, 80, 2, 1, 0, 1, 56, 62, + 0, 1, 0, 1, 60, 80, 2, 1, 0, 1, 60, 62, + 0, 1, 0, 1, 64, 74, 2, 1, 0, 1, 64, 62, + 0, 1, 0, 1, 100, 72, 2, 1, 0, 1, 100, 62, + 0, 1, 0, 1, 104, 80, 2, 1, 0, 1, 104, 62, + 0, 1, 0, 1, 108, 80, 2, 1, 0, 1, 108, 62, + 0, 1, 0, 1, 112, 80, 2, 1, 0, 1, 112, 62, + 0, 1, 0, 1, 116, 80, 2, 1, 0, 1, 116, 62, + 0, 1, 0, 1, 120, 80, 2, 1, 0, 1, 120, 62, + 0, 1, 0, 1, 124, 80, 2, 1, 0, 1, 124, 62, + 0, 1, 0, 1, 128, 80, 2, 1, 0, 1, 128, 62, + 0, 1, 0, 1, 132, 80, 2, 1, 0, 1, 132, 62, + 0, 1, 0, 1, 136, 80, 2, 1, 0, 1, 136, 62, + 0, 1, 0, 1, 140, 72, 2, 1, 0, 1, 140, 62, + 0, 1, 0, 1, 144, 80, 2, 1, 0, 1, 144, 127, + 0, 1, 0, 1, 149, 80, 2, 1, 0, 1, 149, 127, + 0, 1, 0, 1, 153, 80, 2, 1, 0, 1, 153, 127, + 0, 1, 0, 1, 157, 80, 2, 1, 0, 1, 157, 127, + 0, 1, 0, 1, 161, 80, 2, 1, 0, 1, 161, 127, + 0, 1, 0, 1, 165, 80, 2, 1, 0, 1, 165, 127, + 0, 1, 0, 2, 36, 72, 2, 1, 0, 2, 36, 62, + 0, 1, 0, 2, 40, 80, 2, 1, 0, 2, 40, 62, + 0, 1, 0, 2, 44, 80, 2, 1, 0, 2, 44, 62, + 0, 1, 0, 2, 48, 80, 2, 1, 0, 2, 48, 62, + 0, 1, 0, 2, 52, 80, 2, 1, 0, 2, 52, 62, + 0, 1, 0, 2, 56, 80, 2, 1, 0, 2, 56, 62, + 0, 1, 0, 2, 60, 80, 2, 1, 0, 2, 60, 62, + 0, 1, 0, 2, 64, 74, 2, 1, 0, 2, 64, 62, + 0, 1, 0, 2, 100, 70, 2, 1, 0, 2, 100, 62, + 0, 1, 0, 2, 104, 80, 2, 1, 0, 2, 104, 62, + 0, 1, 0, 2, 108, 80, 2, 1, 0, 2, 108, 62, + 0, 1, 0, 2, 112, 80, 2, 1, 0, 2, 112, 62, + 0, 1, 0, 2, 116, 80, 2, 1, 0, 2, 116, 62, + 0, 1, 0, 2, 120, 80, 2, 1, 0, 2, 120, 62, + 0, 1, 0, 2, 124, 80, 2, 1, 0, 2, 124, 62, + 0, 1, 0, 2, 128, 80, 2, 1, 0, 2, 128, 62, + 0, 1, 0, 2, 132, 80, 2, 1, 0, 2, 132, 62, + 0, 1, 0, 2, 136, 80, 2, 1, 0, 2, 136, 62, + 0, 1, 0, 2, 140, 70, 2, 1, 0, 2, 140, 62, + 0, 1, 0, 2, 144, 80, 2, 1, 0, 2, 144, 127, + 0, 1, 0, 2, 149, 80, 2, 1, 0, 2, 149, 127, + 0, 1, 0, 2, 153, 80, 2, 1, 0, 2, 153, 127, + 0, 1, 0, 2, 157, 80, 2, 1, 0, 2, 157, 127, + 0, 1, 0, 2, 161, 80, 2, 1, 0, 2, 161, 127, + 0, 1, 0, 2, 165, 80, 2, 1, 0, 2, 165, 127, + 0, 1, 0, 3, 36, 68, 2, 1, 0, 3, 36, 38, + 0, 1, 0, 3, 40, 68, 2, 1, 0, 3, 40, 38, + 0, 1, 0, 3, 44, 68, 2, 1, 0, 3, 44, 38, + 0, 1, 0, 3, 48, 68, 2, 1, 0, 3, 48, 38, + 0, 1, 0, 3, 52, 68, 2, 1, 0, 3, 52, 38, + 0, 1, 0, 3, 56, 68, 2, 1, 0, 3, 56, 38, + 0, 1, 0, 3, 60, 66, 2, 1, 0, 3, 60, 38, + 0, 1, 0, 3, 64, 68, 2, 1, 0, 3, 64, 38, + 0, 1, 0, 3, 100, 60, 2, 1, 0, 3, 100, 38, + 0, 1, 0, 3, 104, 68, 2, 1, 0, 3, 104, 38, + 0, 1, 0, 3, 108, 68, 2, 1, 0, 3, 108, 38, + 0, 1, 0, 3, 112, 68, 2, 1, 0, 3, 112, 38, + 0, 1, 0, 3, 116, 68, 2, 1, 0, 3, 116, 38, + 0, 1, 0, 3, 120, 68, 2, 1, 0, 3, 120, 38, + 0, 1, 0, 3, 124, 68, 2, 1, 0, 3, 124, 38, + 0, 1, 0, 3, 128, 68, 2, 1, 0, 3, 128, 38, + 0, 1, 0, 3, 132, 68, 2, 1, 0, 3, 132, 38, + 0, 1, 0, 3, 136, 68, 2, 1, 0, 3, 136, 38, + 0, 1, 0, 3, 140, 60, 2, 1, 0, 3, 140, 38, + 0, 1, 0, 3, 144, 68, 2, 1, 0, 3, 144, 127, + 0, 1, 0, 3, 149, 80, 2, 1, 0, 3, 149, 127, + 0, 1, 0, 3, 153, 80, 2, 1, 0, 3, 153, 127, + 0, 1, 0, 3, 157, 80, 2, 1, 0, 3, 157, 127, + 0, 1, 0, 3, 161, 80, 2, 1, 0, 3, 161, 127, + 0, 1, 0, 3, 165, 80, 2, 1, 0, 3, 165, 127, + 0, 1, 1, 2, 38, 66, 2, 1, 1, 2, 38, 64, + 0, 1, 1, 2, 46, 72, 2, 1, 1, 2, 46, 64, + 0, 1, 1, 2, 54, 72, 2, 1, 1, 2, 54, 64, + 0, 1, 1, 2, 62, 64, 2, 1, 1, 2, 62, 64, + 0, 1, 1, 2, 102, 58, 2, 1, 1, 2, 102, 64, + 0, 1, 1, 2, 110, 74, 2, 1, 1, 2, 110, 64, + 0, 1, 1, 2, 118, 74, 2, 1, 1, 2, 118, 64, + 0, 1, 1, 2, 126, 74, 2, 1, 1, 2, 126, 64, + 0, 1, 1, 2, 134, 74, 2, 1, 1, 2, 134, 64, + 0, 1, 1, 2, 142, 74, 2, 1, 1, 2, 142, 127, + 0, 1, 1, 2, 151, 74, 2, 1, 1, 2, 151, 127, + 0, 1, 1, 2, 159, 74, 2, 1, 1, 2, 159, 127, + 0, 1, 1, 3, 38, 60, 2, 1, 1, 3, 38, 40, + 0, 1, 1, 3, 46, 68, 2, 1, 1, 3, 46, 40, + 0, 1, 1, 3, 54, 68, 2, 1, 1, 3, 54, 40, + 0, 1, 1, 3, 62, 58, 2, 1, 1, 3, 62, 40, + 0, 1, 1, 3, 102, 54, 2, 1, 1, 3, 102, 40, + 0, 1, 1, 3, 110, 68, 2, 1, 1, 3, 110, 40, + 0, 1, 1, 3, 118, 68, 2, 1, 1, 3, 118, 40, + 0, 1, 1, 3, 126, 68, 2, 1, 1, 3, 126, 40, + 0, 1, 1, 3, 134, 68, 2, 1, 1, 3, 134, 40, + 0, 1, 1, 3, 142, 68, 2, 1, 1, 3, 142, 127, + 0, 1, 1, 3, 151, 74, 2, 1, 1, 3, 151, 127, + 0, 1, 1, 3, 159, 74, 2, 1, 1, 3, 159, 127, + 0, 1, 2, 4, 42, 64, 2, 1, 2, 4, 42, 64, + 0, 1, 2, 4, 58, 62, 2, 1, 2, 4, 58, 64, + 0, 1, 2, 4, 106, 58, 2, 1, 2, 4, 106, 64, + 0, 1, 2, 4, 122, 72, 2, 1, 2, 4, 122, 64, + 0, 1, 2, 4, 138, 72, 2, 1, 2, 4, 138, 127, + 0, 1, 2, 4, 155, 72, 2, 1, 2, 4, 155, 127, + 0, 1, 2, 5, 42, 54, 2, 1, 2, 5, 42, 40, + 0, 1, 2, 5, 58, 52, 2, 1, 2, 5, 58, 40, + 0, 1, 2, 5, 106, 50, 2, 1, 2, 5, 106, 40, + 0, 1, 2, 5, 122, 66, 2, 1, 2, 5, 122, 40, + 0, 1, 2, 5, 138, 66, 2, 1, 2, 5, 138, 127, + 0, 1, 2, 5, 155, 62, 2, 1, 2, 5, 155, 127 +}; + +RTW_DECL_TABLE_TXPWR_LMT(rtw8822c_txpwr_lmt_type0); + +static const u32 rtw8822c_array_mp_cal_init[] = { + 0x1b00, 0x00000008, + 0x1b00, 0x00A70008, + 0x1b00, 0x00150008, + 0x1b00, 0x00000008, + 0x1b04, 0xE2462952, + 0x1b08, 0x00000080, + 0x1b0c, 0x00000000, + 0x1b10, 0x00010C00, + 0x1b14, 0x00000000, + 0x1b18, 0x00292903, + 0x1b1c, 0xA218FC32, + 0x1b20, 0x01040008, + 0x1b24, 0x00060008, + 0x1b28, 0x00060300, + 0x1b2C, 0x00180018, + 0x1b30, 0x40000000, + 0x1b34, 0x00000800, + 0x1b38, 0x40000000, + 0x1b3C, 0x40000000, + 0x1b98, 0x00000000, + 0x1b9c, 0x00000000, + 0x1bc0, 0x01000000, + 0x1bcc, 0x00000000, + 0x1be4, 0x00000000, + 0x1bec, 0x40000000, + 0x1b40, 0x40000000, + 0x1b44, 0x20004064, + 0x1b48, 0x0005002D, + 0x1b4c, 0x00000000, + 0x1b60, 0x1F100000, + 0x1b64, 0x12000000, + 0x1b4c, 0x00000000, + 0x1b4c, 0x008a0000, + 0x1b50, 0x000003BE, + 0x1b4c, 0x018a0000, + 0x1b50, 0x0000057A, + 0x1b4c, 0x028a0000, + 0x1b50, 0x000006C8, + 0x1b4c, 0x038a0000, + 0x1b50, 0x000007E0, + 0x1b4c, 0x048a0000, + 0x1b50, 0x000008D5, + 0x1b4c, 0x058a0000, + 0x1b50, 0x000009B2, + 0x1b4c, 0x068a0000, + 0x1b50, 0x00000A7D, + 0x1b4c, 0x078a0000, + 0x1b50, 0x00000B3A, + 0x1b4c, 0x088a0000, + 0x1b50, 0x00000BEB, + 0x1b4c, 0x098a0000, + 0x1b50, 0x00000C92, + 0x1b4c, 0x0A8a0000, + 0x1b50, 0x00000D31, + 0x1b4c, 0x0B8a0000, + 0x1b50, 0x00000DC9, + 0x1b4c, 0x0C8a0000, + 0x1b50, 0x00000E5A, + 0x1b4c, 0x0D8a0000, + 0x1b50, 0x00000EE6, + 0x1b4c, 0x0E8a0000, + 0x1b50, 0x00000F6D, + 0x1b4c, 0x0F8a0000, + 0x1b50, 0x00000FF0, + 0x1b4c, 0x108a0000, + 0x1b50, 0x0000106F, + 0x1b4c, 0x118a0000, + 0x1b50, 0x000010E9, + 0x1b4c, 0x128a0000, + 0x1b50, 0x00001161, + 0x1b4c, 0x138a0000, + 0x1b50, 0x000011D5, + 0x1b4c, 0x148a0000, + 0x1b50, 0x00001247, + 0x1b4c, 0x158a0000, + 0x1b50, 0x000012B5, + 0x1b4c, 0x168a0000, + 0x1b50, 0x00001322, + 0x1b4c, 0x178a0000, + 0x1b50, 0x0000138B, + 0x1b4c, 0x188a0000, + 0x1b50, 0x000013F3, + 0x1b4c, 0x198a0000, + 0x1b50, 0x00001459, + 0x1b4c, 0x1A8a0000, + 0x1b50, 0x000014BD, + 0x1b4c, 0x1B8a0000, + 0x1b50, 0x0000151E, + 0x1b4c, 0x1C8a0000, + 0x1b50, 0x0000157F, + 0x1b4c, 0x1D8a0000, + 0x1b50, 0x000015DD, + 0x1b4c, 0x1E8a0000, + 0x1b50, 0x0000163A, + 0x1b4c, 0x1F8a0000, + 0x1b50, 0x00001695, + 0x1b4c, 0x208a0000, + 0x1b50, 0x000016EF, + 0x1b4c, 0x218a0000, + 0x1b50, 0x00001748, + 0x1b4c, 0x228a0000, + 0x1b50, 0x0000179F, + 0x1b4c, 0x238a0000, + 0x1b50, 0x000017F5, + 0x1b4c, 0x248a0000, + 0x1b50, 0x0000184A, + 0x1b4c, 0x258a0000, + 0x1b50, 0x0000189E, + 0x1b4c, 0x268a0000, + 0x1b50, 0x000018F1, + 0x1b4c, 0x278a0000, + 0x1b50, 0x00001942, + 0x1b4c, 0x288a0000, + 0x1b50, 0x00001993, + 0x1b4c, 0x298a0000, + 0x1b50, 0x000019E2, + 0x1b4c, 0x2A8a0000, + 0x1b50, 0x00001A31, + 0x1b4c, 0x2B8a0000, + 0x1b50, 0x00001A7F, + 0x1b4c, 0x2C8a0000, + 0x1b50, 0x00001ACC, + 0x1b4c, 0x2D8a0000, + 0x1b50, 0x00001B18, + 0x1b4c, 0x2E8a0000, + 0x1b50, 0x00001B63, + 0x1b4c, 0x2F8a0000, + 0x1b50, 0x00001BAD, + 0x1b4c, 0x308a0000, + 0x1b50, 0x00001BF7, + 0x1b4c, 0x318a0000, + 0x1b50, 0x00001C40, + 0x1b4c, 0x328a0000, + 0x1b50, 0x00001C88, + 0x1b4c, 0x338a0000, + 0x1b50, 0x00001CCF, + 0x1b4c, 0x348a0000, + 0x1b50, 0x00001D16, + 0x1b4c, 0x358a0000, + 0x1b50, 0x00001D5C, + 0x1b4c, 0x368a0000, + 0x1b50, 0x00001DA2, + 0x1b4c, 0x378a0000, + 0x1b50, 0x00001DE6, + 0x1b4c, 0x388a0000, + 0x1b50, 0x00001E2B, + 0x1b4c, 0x398a0000, + 0x1b50, 0x00001E6E, + 0x1b4c, 0x3A8a0000, + 0x1b50, 0x00001EB1, + 0x1b4c, 0x3B8a0000, + 0x1b50, 0x00001EF4, + 0x1b4c, 0x3C8a0000, + 0x1b50, 0x00001F35, + 0x1b4c, 0x3D8a0000, + 0x1b50, 0x00001F77, + 0x1b4c, 0x3E8a0000, + 0x1b50, 0x00001FB8, + 0x1b4c, 0x3F8a0000, + 0x1b50, 0x00001FF8, + 0x1b4c, 0x00000000, + 0x1b50, 0x00000000, + 0x1b58, 0x00890000, + 0x1b5C, 0x3C6B3FFF, + 0x1b58, 0x02890000, + 0x1b5C, 0x35D9390A, + 0x1b58, 0x04890000, + 0x1b5C, 0x2FFE32D6, + 0x1b58, 0x06890000, + 0x1b5C, 0x2AC62D4F, + 0x1b58, 0x08890000, + 0x1b5C, 0x261F2862, + 0x1b58, 0x0A890000, + 0x1b5C, 0x21FA23FD, + 0x1b58, 0x0C890000, + 0x1b5C, 0x1E482013, + 0x1b58, 0x0E890000, + 0x1b5C, 0x1AFD1C96, + 0x1b58, 0x10890000, + 0x1b5C, 0x180E197B, + 0x1b58, 0x12890000, + 0x1b5C, 0x157016B5, + 0x1b58, 0x14890000, + 0x1b5C, 0x131B143D, + 0x1b58, 0x16890000, + 0x1b5C, 0x1107120A, + 0x1b58, 0x18890000, + 0x1b5C, 0x0F2D1013, + 0x1b58, 0x1A890000, + 0x1b5C, 0x0D870E54, + 0x1b58, 0x1C890000, + 0x1b5C, 0x0C0E0CC5, + 0x1b58, 0x1E890000, + 0x1b5C, 0x0ABF0B62, + 0x1b58, 0x20890000, + 0x1b5C, 0x09930A25, + 0x1b58, 0x22890000, + 0x1b5C, 0x0889090A, + 0x1b58, 0x24890000, + 0x1b5C, 0x079B080F, + 0x1b58, 0x26890000, + 0x1b5C, 0x06C7072E, + 0x1b58, 0x28890000, + 0x1b5C, 0x060B0666, + 0x1b58, 0x2A890000, + 0x1b5C, 0x056305B4, + 0x1b58, 0x2C890000, + 0x1b5C, 0x04CD0515, + 0x1b58, 0x2E890000, + 0x1b5C, 0x04470488, + 0x1b58, 0x30890000, + 0x1b5C, 0x03D0040A, + 0x1b58, 0x32890000, + 0x1b5C, 0x03660399, + 0x1b58, 0x34890000, + 0x1b5C, 0x03070335, + 0x1b58, 0x36890000, + 0x1b5C, 0x02B302DC, + 0x1b58, 0x38890000, + 0x1b5C, 0x0268028C, + 0x1b58, 0x3A890000, + 0x1b5C, 0x02250245, + 0x1b58, 0x3C890000, + 0x1b5C, 0x01E90206, + 0x1b58, 0x3E890000, + 0x1b5C, 0x01B401CE, + 0x1b58, 0x40890000, + 0x1b5C, 0x0185019C, + 0x1b58, 0x42890000, + 0x1b5C, 0x015A016F, + 0x1b58, 0x44890000, + 0x1b5C, 0x01350147, + 0x1b58, 0x46890000, + 0x1b5C, 0x01130123, + 0x1b58, 0x48890000, + 0x1b5C, 0x00F50104, + 0x1b58, 0x4A890000, + 0x1b5C, 0x00DA00E7, + 0x1b58, 0x4C890000, + 0x1b5C, 0x00C300CE, + 0x1b58, 0x4E890000, + 0x1b5C, 0x00AE00B8, + 0x1b58, 0x50890000, + 0x1b5C, 0x009B00A4, + 0x1b58, 0x52890000, + 0x1b5C, 0x008A0092, + 0x1b58, 0x54890000, + 0x1b5C, 0x007B0082, + 0x1b58, 0x56890000, + 0x1b5C, 0x006E0074, + 0x1b58, 0x58890000, + 0x1b5C, 0x00620067, + 0x1b58, 0x5A890000, + 0x1b5C, 0x0057005C, + 0x1b58, 0x5C890000, + 0x1b5C, 0x004E0052, + 0x1b58, 0x5E890000, + 0x1b5C, 0x00450049, + 0x1b58, 0x60890000, + 0x1b5C, 0x003E0041, + 0x1b58, 0x62890000, + 0x1b5C, 0x0037003A, + 0x1b58, 0x62010000, + 0x1b00, 0x0000000A, + 0x1b00, 0x00A7000A, + 0x1b00, 0x0015000A, + 0x1b00, 0x0000000A, + 0x1b04, 0xE2462952, + 0x1b08, 0x00000080, + 0x1b0c, 0x00000000, + 0x1b10, 0x00010C00, + 0x1b14, 0x00000000, + 0x1b18, 0x00292903, + 0x1b1c, 0xA218FC32, + 0x1b20, 0x01040008, + 0x1b24, 0x00060008, + 0x1b28, 0x00060300, + 0x1b2C, 0x00180018, + 0x1b30, 0x40000000, + 0x1b34, 0x00000800, + 0x1b38, 0x40000000, + 0x1b3C, 0x40000000, + 0x1b98, 0x00000000, + 0x1b9c, 0x00000000, + 0x1bc0, 0x01000000, + 0x1bcc, 0x00000000, + 0x1be4, 0x00000000, + 0x1bec, 0x40000000, + 0x1b60, 0x1F100000, + 0x1b64, 0x12000000, + 0x1b58, 0x00890000, + 0x1b5C, 0x3C6B3FFF, + 0x1b58, 0x02890000, + 0x1b5C, 0x35D9390A, + 0x1b58, 0x04890000, + 0x1b5C, 0x2FFE32D6, + 0x1b58, 0x06890000, + 0x1b5C, 0x2AC62D4F, + 0x1b58, 0x08890000, + 0x1b5C, 0x261F2862, + 0x1b58, 0x0A890000, + 0x1b5C, 0x21FA23FD, + 0x1b58, 0x0C890000, + 0x1b5C, 0x1E482013, + 0x1b58, 0x0E890000, + 0x1b5C, 0x1AFD1C96, + 0x1b58, 0x10890000, + 0x1b5C, 0x180E197B, + 0x1b58, 0x12890000, + 0x1b5C, 0x157016B5, + 0x1b58, 0x14890000, + 0x1b5C, 0x131B143D, + 0x1b58, 0x16890000, + 0x1b5C, 0x1107120A, + 0x1b58, 0x18890000, + 0x1b5C, 0x0F2D1013, + 0x1b58, 0x1A890000, + 0x1b5C, 0x0D870E54, + 0x1b58, 0x1C890000, + 0x1b5C, 0x0C0E0CC5, + 0x1b58, 0x1E890000, + 0x1b5C, 0x0ABF0B62, + 0x1b58, 0x20890000, + 0x1b5C, 0x09930A25, + 0x1b58, 0x22890000, + 0x1b5C, 0x0889090A, + 0x1b58, 0x24890000, + 0x1b5C, 0x079B080F, + 0x1b58, 0x26890000, + 0x1b5C, 0x06C7072E, + 0x1b58, 0x28890000, + 0x1b5C, 0x060B0666, + 0x1b58, 0x2A890000, + 0x1b5C, 0x056305B4, + 0x1b58, 0x2C890000, + 0x1b5C, 0x04CD0515, + 0x1b58, 0x2E890000, + 0x1b5C, 0x04470488, + 0x1b58, 0x30890000, + 0x1b5C, 0x03D0040A, + 0x1b58, 0x32890000, + 0x1b5C, 0x03660399, + 0x1b58, 0x34890000, + 0x1b5C, 0x03070335, + 0x1b58, 0x36890000, + 0x1b5C, 0x02B302DC, + 0x1b58, 0x38890000, + 0x1b5C, 0x0268028C, + 0x1b58, 0x3A890000, + 0x1b5C, 0x02250245, + 0x1b58, 0x3C890000, + 0x1b5C, 0x01E90206, + 0x1b58, 0x3E890000, + 0x1b5C, 0x01B401CE, + 0x1b58, 0x40890000, + 0x1b5C, 0x0185019C, + 0x1b58, 0x42890000, + 0x1b5C, 0x015A016F, + 0x1b58, 0x44890000, + 0x1b5C, 0x01350147, + 0x1b58, 0x46890000, + 0x1b5C, 0x01130123, + 0x1b58, 0x48890000, + 0x1b5C, 0x00F50104, + 0x1b58, 0x4A890000, + 0x1b5C, 0x00DA00E7, + 0x1b58, 0x4C890000, + 0x1b5C, 0x00C300CE, + 0x1b58, 0x4E890000, + 0x1b5C, 0x00AE00B8, + 0x1b58, 0x50890000, + 0x1b5C, 0x009B00A4, + 0x1b58, 0x52890000, + 0x1b5C, 0x008A0092, + 0x1b58, 0x54890000, + 0x1b5C, 0x007B0082, + 0x1b58, 0x56890000, + 0x1b5C, 0x006E0074, + 0x1b58, 0x58890000, + 0x1b5C, 0x00620067, + 0x1b58, 0x5A890000, + 0x1b5C, 0x0057005C, + 0x1b58, 0x5C890000, + 0x1b5C, 0x004E0052, + 0x1b58, 0x5E890000, + 0x1b5C, 0x00450049, + 0x1b58, 0x60890000, + 0x1b5C, 0x003E0041, + 0x1b58, 0x62890000, + 0x1b5C, 0x0037003A, + 0x1b58, 0x62010000, + 0x1b00, 0x0000000C, + 0x1bd4, 0x000000F0, + 0x1bb8, 0x20202020, + 0x1bbc, 0x20202020, + 0x1bc0, 0x20202020, + 0x1bc4, 0x20202020, + 0x1bc8, 0x04040404, + 0x1bcc, 0x04040404, + 0x1bd0, 0x04040404, + 0x1bd8, 0x04040404, + 0x1bdc, 0x20202020, + 0x1be0, 0x04040404, + 0x1be4, 0x77472F17, + 0x1be8, 0xEFBFA78F, + 0x1bec, 0x00000000, + 0x1bf0, 0x1F1F1939, + 0x1b04, 0x0000005B, + 0x1b08, 0xB000C000, + 0x1b5c, 0x0000005B, + 0x1b60, 0xB000C000, + 0x1bb4, 0x20000000, + 0x1b00, 0x00000008, + 0x1b80, 0x00000007, + 0x1b80, 0x00080005, + 0x1b80, 0x00080007, + 0x1b80, 0x80000015, + 0x1b80, 0x80000017, + 0x1b80, 0x09080025, + 0x1b80, 0x09080027, + 0x1b80, 0x0f020035, + 0x1b80, 0x0f020037, + 0x1b80, 0x00220045, + 0x1b80, 0x00220047, + 0x1b80, 0x00040055, + 0x1b80, 0x00040057, + 0x1b80, 0x05c00065, + 0x1b80, 0x05c00067, + 0x1b80, 0x00070075, + 0x1b80, 0x00070077, + 0x1b80, 0x64020085, + 0x1b80, 0x64020087, + 0x1b80, 0x00020095, + 0x1b80, 0x00020097, + 0x1b80, 0x000400a5, + 0x1b80, 0x000400a7, + 0x1b80, 0x4a0000b5, + 0x1b80, 0x4a0000b7, + 0x1b80, 0x4b0400c5, + 0x1b80, 0x4b0400c7, + 0x1b80, 0x860300d5, + 0x1b80, 0x860300d7, + 0x1b80, 0x400900e5, + 0x1b80, 0x400900e7, + 0x1b80, 0xe02700f5, + 0x1b80, 0xe02700f7, + 0x1b80, 0x4b050105, + 0x1b80, 0x4b050107, + 0x1b80, 0x87030115, + 0x1b80, 0x87030117, + 0x1b80, 0x400b0125, + 0x1b80, 0x400b0127, + 0x1b80, 0xe0270135, + 0x1b80, 0xe0270137, + 0x1b80, 0x4b060145, + 0x1b80, 0x4b060147, + 0x1b80, 0x88030155, + 0x1b80, 0x88030157, + 0x1b80, 0x400d0165, + 0x1b80, 0x400d0167, + 0x1b80, 0xe0270175, + 0x1b80, 0xe0270177, + 0x1b80, 0x4b000185, + 0x1b80, 0x4b000187, + 0x1b80, 0x00070195, + 0x1b80, 0x00070197, + 0x1b80, 0x4c0001a5, + 0x1b80, 0x4c0001a7, + 0x1b80, 0x000401b5, + 0x1b80, 0x000401b7, + 0x1b80, 0x400801c5, + 0x1b80, 0x400801c7, + 0x1b80, 0x505501d5, + 0x1b80, 0x505501d7, + 0x1b80, 0x090a01e5, + 0x1b80, 0x090a01e7, + 0x1b80, 0x0ffe01f5, + 0x1b80, 0x0ffe01f7, + 0x1b80, 0x00220205, + 0x1b80, 0x00220207, + 0x1b80, 0x00040215, + 0x1b80, 0x00040217, + 0x1b80, 0x05c00225, + 0x1b80, 0x05c00227, + 0x1b80, 0x00070235, + 0x1b80, 0x00070237, + 0x1b80, 0x64000245, + 0x1b80, 0x64000247, + 0x1b80, 0x00020255, + 0x1b80, 0x00020257, + 0x1b80, 0x30000265, + 0x1b80, 0x30000267, + 0x1b80, 0xa5100275, + 0x1b80, 0xa5100277, + 0x1b80, 0xe3520285, + 0x1b80, 0xe3520287, + 0x1b80, 0xf01d0295, + 0x1b80, 0xf01d0297, + 0x1b80, 0xf11d02a5, + 0x1b80, 0xf11d02a7, + 0x1b80, 0xf21d02b5, + 0x1b80, 0xf21d02b7, + 0x1b80, 0xf31d02c5, + 0x1b80, 0xf31d02c7, + 0x1b80, 0xf41d02d5, + 0x1b80, 0xf41d02d7, + 0x1b80, 0xf51d02e5, + 0x1b80, 0xf51d02e7, + 0x1b80, 0xf61d02f5, + 0x1b80, 0xf61d02f7, + 0x1b80, 0xf71d0305, + 0x1b80, 0xf71d0307, + 0x1b80, 0xf81d0315, + 0x1b80, 0xf81d0317, + 0x1b80, 0xf91d0325, + 0x1b80, 0xf91d0327, + 0x1b80, 0xfa1d0335, + 0x1b80, 0xfa1d0337, + 0x1b80, 0xfb1d0345, + 0x1b80, 0xfb1d0347, + 0x1b80, 0xfc1d0355, + 0x1b80, 0xfc1d0357, + 0x1b80, 0xfd1d0365, + 0x1b80, 0xfd1d0367, + 0x1b80, 0xf21d0375, + 0x1b80, 0xf21d0377, + 0x1b80, 0xf31d0385, + 0x1b80, 0xf31d0387, + 0x1b80, 0xf41d0395, + 0x1b80, 0xf41d0397, + 0x1b80, 0xf51d03a5, + 0x1b80, 0xf51d03a7, + 0x1b80, 0xf61d03b5, + 0x1b80, 0xf61d03b7, + 0x1b80, 0xf71d03c5, + 0x1b80, 0xf71d03c7, + 0x1b80, 0xf81d03d5, + 0x1b80, 0xf81d03d7, + 0x1b80, 0xf91d03e5, + 0x1b80, 0xf91d03e7, + 0x1b80, 0xfa1d03f5, + 0x1b80, 0xfa1d03f7, + 0x1b80, 0xfb1d0405, + 0x1b80, 0xfb1d0407, + 0x1b80, 0xfc1d0415, + 0x1b80, 0xfc1d0417, + 0x1b80, 0xfd1d0425, + 0x1b80, 0xfd1d0427, + 0x1b80, 0xfe1d0435, + 0x1b80, 0xfe1d0437, + 0x1b80, 0xff1d0445, + 0x1b80, 0xff1d0447, + 0x1b80, 0x00010455, + 0x1b80, 0x00010457, + 0x1b80, 0x30620465, + 0x1b80, 0x30620467, + 0x1b80, 0x307a0475, + 0x1b80, 0x307a0477, + 0x1b80, 0x307c0485, + 0x1b80, 0x307c0487, + 0x1b80, 0x30eb0495, + 0x1b80, 0x30eb0497, + 0x1b80, 0x308004a5, + 0x1b80, 0x308004a7, + 0x1b80, 0x308c04b5, + 0x1b80, 0x308c04b7, + 0x1b80, 0x309804c5, + 0x1b80, 0x309804c7, + 0x1b80, 0x307f04d5, + 0x1b80, 0x307f04d7, + 0x1b80, 0x308b04e5, + 0x1b80, 0x308b04e7, + 0x1b80, 0x309704f5, + 0x1b80, 0x309704f7, + 0x1b80, 0x30ef0505, + 0x1b80, 0x30ef0507, + 0x1b80, 0x30fa0515, + 0x1b80, 0x30fa0517, + 0x1b80, 0x31050525, + 0x1b80, 0x31050527, + 0x1b80, 0x316a0535, + 0x1b80, 0x316a0537, + 0x1b80, 0x307a0545, + 0x1b80, 0x307a0547, + 0x1b80, 0x30e90555, + 0x1b80, 0x30e90557, + 0x1b80, 0x31870565, + 0x1b80, 0x31870567, + 0x1b80, 0x31a00575, + 0x1b80, 0x31a00577, + 0x1b80, 0x31ba0585, + 0x1b80, 0x31ba0587, + 0x1b80, 0x31c20595, + 0x1b80, 0x31c20597, + 0x1b80, 0x31ca05a5, + 0x1b80, 0x31ca05a7, + 0x1b80, 0x31d205b5, + 0x1b80, 0x31d205b7, + 0x1b80, 0x31da05c5, + 0x1b80, 0x31da05c7, + 0x1b80, 0x31e905d5, + 0x1b80, 0x31e905d7, + 0x1b80, 0x31f805e5, + 0x1b80, 0x31f805e7, + 0x1b80, 0x31fe05f5, + 0x1b80, 0x31fe05f7, + 0x1b80, 0x32040605, + 0x1b80, 0x32040607, + 0x1b80, 0x320a0615, + 0x1b80, 0x320a0617, + 0x1b80, 0xe2eb0625, + 0x1b80, 0xe2eb0627, + 0x1b80, 0x4d040635, + 0x1b80, 0x4d040637, + 0x1b80, 0x20800645, + 0x1b80, 0x20800647, + 0x1b80, 0x00000655, + 0x1b80, 0x00000657, + 0x1b80, 0x4d000665, + 0x1b80, 0x4d000667, + 0x1b80, 0x55070675, + 0x1b80, 0x55070677, + 0x1b80, 0xe2e30685, + 0x1b80, 0xe2e30687, + 0x1b80, 0xe2e30695, + 0x1b80, 0xe2e30697, + 0x1b80, 0x4d0406a5, + 0x1b80, 0x4d0406a7, + 0x1b80, 0x208806b5, + 0x1b80, 0x208806b7, + 0x1b80, 0x020006c5, + 0x1b80, 0x020006c7, + 0x1b80, 0x4d0006d5, + 0x1b80, 0x4d0006d7, + 0x1b80, 0x550f06e5, + 0x1b80, 0x550f06e7, + 0x1b80, 0xe2e306f5, + 0x1b80, 0xe2e306f7, + 0x1b80, 0x4f020705, + 0x1b80, 0x4f020707, + 0x1b80, 0x4e000715, + 0x1b80, 0x4e000717, + 0x1b80, 0x53020725, + 0x1b80, 0x53020727, + 0x1b80, 0x52010735, + 0x1b80, 0x52010737, + 0x1b80, 0xe2e70745, + 0x1b80, 0xe2e70747, + 0x1b80, 0x4d080755, + 0x1b80, 0x4d080757, + 0x1b80, 0x57100765, + 0x1b80, 0x57100767, + 0x1b80, 0x57000775, + 0x1b80, 0x57000777, + 0x1b80, 0x4d000785, + 0x1b80, 0x4d000787, + 0x1b80, 0x00010795, + 0x1b80, 0x00010797, + 0x1b80, 0xe2eb07a5, + 0x1b80, 0xe2eb07a7, + 0x1b80, 0x000107b5, + 0x1b80, 0x000107b7, + 0x1b80, 0x620607c5, + 0x1b80, 0x620607c7, + 0x1b80, 0xe2eb07d5, + 0x1b80, 0xe2eb07d7, + 0x1b80, 0x000107e5, + 0x1b80, 0x000107e7, + 0x1b80, 0x620607f5, + 0x1b80, 0x620607f7, + 0x1b80, 0x30ad0805, + 0x1b80, 0x30ad0807, + 0x1b80, 0x00260815, + 0x1b80, 0x00260817, + 0x1b80, 0xe3450825, + 0x1b80, 0xe3450827, + 0x1b80, 0x00020835, + 0x1b80, 0x00020837, + 0x1b80, 0x54ec0845, + 0x1b80, 0x54ec0847, + 0x1b80, 0x0ba60855, + 0x1b80, 0x0ba60857, + 0x1b80, 0x00260865, + 0x1b80, 0x00260867, + 0x1b80, 0xe3450875, + 0x1b80, 0xe3450877, + 0x1b80, 0x00020885, + 0x1b80, 0x00020887, + 0x1b80, 0x63c30895, + 0x1b80, 0x63c30897, + 0x1b80, 0x30d908a5, + 0x1b80, 0x30d908a7, + 0x1b80, 0x620608b5, + 0x1b80, 0x620608b7, + 0x1b80, 0x30a508c5, + 0x1b80, 0x30a508c7, + 0x1b80, 0x002408d5, + 0x1b80, 0x002408d7, + 0x1b80, 0xe34508e5, + 0x1b80, 0xe34508e7, + 0x1b80, 0x000208f5, + 0x1b80, 0x000208f7, + 0x1b80, 0x54ea0905, + 0x1b80, 0x54ea0907, + 0x1b80, 0x0ba60915, + 0x1b80, 0x0ba60917, + 0x1b80, 0x00240925, + 0x1b80, 0x00240927, + 0x1b80, 0xe3450935, + 0x1b80, 0xe3450937, + 0x1b80, 0x00020945, + 0x1b80, 0x00020947, + 0x1b80, 0x63c30955, + 0x1b80, 0x63c30957, + 0x1b80, 0x30d90965, + 0x1b80, 0x30d90967, + 0x1b80, 0x62060975, + 0x1b80, 0x62060977, + 0x1b80, 0x6c100985, + 0x1b80, 0x6c100987, + 0x1b80, 0x6d0f0995, + 0x1b80, 0x6d0f0997, + 0x1b80, 0xe2eb09a5, + 0x1b80, 0xe2eb09a7, + 0x1b80, 0xe34509b5, + 0x1b80, 0xe34509b7, + 0x1b80, 0x6c2409c5, + 0x1b80, 0x6c2409c7, + 0x1b80, 0xe2eb09d5, + 0x1b80, 0xe2eb09d7, + 0x1b80, 0xe34509e5, + 0x1b80, 0xe34509e7, + 0x1b80, 0x6c4409f5, + 0x1b80, 0x6c4409f7, + 0x1b80, 0xe2eb0a05, + 0x1b80, 0xe2eb0a07, + 0x1b80, 0xe3450a15, + 0x1b80, 0xe3450a17, + 0x1b80, 0x6c640a25, + 0x1b80, 0x6c640a27, + 0x1b80, 0xe2eb0a35, + 0x1b80, 0xe2eb0a37, + 0x1b80, 0xe3450a45, + 0x1b80, 0xe3450a47, + 0x1b80, 0x0baa0a55, + 0x1b80, 0x0baa0a57, + 0x1b80, 0x6c840a65, + 0x1b80, 0x6c840a67, + 0x1b80, 0x6d0f0a75, + 0x1b80, 0x6d0f0a77, + 0x1b80, 0xe2eb0a85, + 0x1b80, 0xe2eb0a87, + 0x1b80, 0xe3450a95, + 0x1b80, 0xe3450a97, + 0x1b80, 0x6ca40aa5, + 0x1b80, 0x6ca40aa7, + 0x1b80, 0xe2eb0ab5, + 0x1b80, 0xe2eb0ab7, + 0x1b80, 0xe3450ac5, + 0x1b80, 0xe3450ac7, + 0x1b80, 0x0bac0ad5, + 0x1b80, 0x0bac0ad7, + 0x1b80, 0x6cc40ae5, + 0x1b80, 0x6cc40ae7, + 0x1b80, 0x6d0f0af5, + 0x1b80, 0x6d0f0af7, + 0x1b80, 0xe2eb0b05, + 0x1b80, 0xe2eb0b07, + 0x1b80, 0xe3450b15, + 0x1b80, 0xe3450b17, + 0x1b80, 0x6ce40b25, + 0x1b80, 0x6ce40b27, + 0x1b80, 0xe2eb0b35, + 0x1b80, 0xe2eb0b37, + 0x1b80, 0xe3450b45, + 0x1b80, 0xe3450b47, + 0x1b80, 0x6cf40b55, + 0x1b80, 0x6cf40b57, + 0x1b80, 0xe2eb0b65, + 0x1b80, 0xe2eb0b67, + 0x1b80, 0xe3450b75, + 0x1b80, 0xe3450b77, + 0x1b80, 0x6c0c0b85, + 0x1b80, 0x6c0c0b87, + 0x1b80, 0x6d000b95, + 0x1b80, 0x6d000b97, + 0x1b80, 0xe2eb0ba5, + 0x1b80, 0xe2eb0ba7, + 0x1b80, 0xe3450bb5, + 0x1b80, 0xe3450bb7, + 0x1b80, 0x6c1c0bc5, + 0x1b80, 0x6c1c0bc7, + 0x1b80, 0xe2eb0bd5, + 0x1b80, 0xe2eb0bd7, + 0x1b80, 0xe3450be5, + 0x1b80, 0xe3450be7, + 0x1b80, 0x6c3c0bf5, + 0x1b80, 0x6c3c0bf7, + 0x1b80, 0xe2eb0c05, + 0x1b80, 0xe2eb0c07, + 0x1b80, 0xe3450c15, + 0x1b80, 0xe3450c17, + 0x1b80, 0xf4bf0c25, + 0x1b80, 0xf4bf0c27, + 0x1b80, 0xf7be0c35, + 0x1b80, 0xf7be0c37, + 0x1b80, 0x6c5c0c45, + 0x1b80, 0x6c5c0c47, + 0x1b80, 0xe2eb0c55, + 0x1b80, 0xe2eb0c57, + 0x1b80, 0xe3450c65, + 0x1b80, 0xe3450c67, + 0x1b80, 0x6c7c0c75, + 0x1b80, 0x6c7c0c77, + 0x1b80, 0xe2eb0c85, + 0x1b80, 0xe2eb0c87, + 0x1b80, 0xe3450c95, + 0x1b80, 0xe3450c97, + 0x1b80, 0xf5c30ca5, + 0x1b80, 0xf5c30ca7, + 0x1b80, 0xf8c20cb5, + 0x1b80, 0xf8c20cb7, + 0x1b80, 0x6c9c0cc5, + 0x1b80, 0x6c9c0cc7, + 0x1b80, 0xe2eb0cd5, + 0x1b80, 0xe2eb0cd7, + 0x1b80, 0xe3450ce5, + 0x1b80, 0xe3450ce7, + 0x1b80, 0x6cbc0cf5, + 0x1b80, 0x6cbc0cf7, + 0x1b80, 0xe2eb0d05, + 0x1b80, 0xe2eb0d07, + 0x1b80, 0xe3450d15, + 0x1b80, 0xe3450d17, + 0x1b80, 0x6cdc0d25, + 0x1b80, 0x6cdc0d27, + 0x1b80, 0xe2eb0d35, + 0x1b80, 0xe2eb0d37, + 0x1b80, 0xe3450d45, + 0x1b80, 0xe3450d47, + 0x1b80, 0x6cf00d55, + 0x1b80, 0x6cf00d57, + 0x1b80, 0xe2eb0d65, + 0x1b80, 0xe2eb0d67, + 0x1b80, 0xe3450d75, + 0x1b80, 0xe3450d77, + 0x1b80, 0x63c30d85, + 0x1b80, 0x63c30d87, + 0x1b80, 0x55010d95, + 0x1b80, 0x55010d97, + 0x1b80, 0x57040da5, + 0x1b80, 0x57040da7, + 0x1b80, 0x57000db5, + 0x1b80, 0x57000db7, + 0x1b80, 0x96000dc5, + 0x1b80, 0x96000dc7, + 0x1b80, 0x57080dd5, + 0x1b80, 0x57080dd7, + 0x1b80, 0x57000de5, + 0x1b80, 0x57000de7, + 0x1b80, 0x95000df5, + 0x1b80, 0x95000df7, + 0x1b80, 0x4d000e05, + 0x1b80, 0x4d000e07, + 0x1b80, 0x63050e15, + 0x1b80, 0x63050e17, + 0x1b80, 0x7b400e25, + 0x1b80, 0x7b400e27, + 0x1b80, 0x7a000e35, + 0x1b80, 0x7a000e37, + 0x1b80, 0x79000e45, + 0x1b80, 0x79000e47, + 0x1b80, 0x7f400e55, + 0x1b80, 0x7f400e57, + 0x1b80, 0x7e000e65, + 0x1b80, 0x7e000e67, + 0x1b80, 0x7d000e75, + 0x1b80, 0x7d000e77, + 0x1b80, 0x00010e85, + 0x1b80, 0x00010e87, + 0x1b80, 0xe3170e95, + 0x1b80, 0xe3170e97, + 0x1b80, 0x00010ea5, + 0x1b80, 0x00010ea7, + 0x1b80, 0x5c320eb5, + 0x1b80, 0x5c320eb7, + 0x1b80, 0xe3410ec5, + 0x1b80, 0xe3410ec7, + 0x1b80, 0xe3170ed5, + 0x1b80, 0xe3170ed7, + 0x1b80, 0x00010ee5, + 0x1b80, 0x00010ee7, + 0x1b80, 0x31260ef5, + 0x1b80, 0x31260ef7, + 0x1b80, 0x00260f05, + 0x1b80, 0x00260f07, + 0x1b80, 0xe34a0f15, + 0x1b80, 0xe34a0f17, + 0x1b80, 0x00020f25, + 0x1b80, 0x00020f27, + 0x1b80, 0x54ec0f35, + 0x1b80, 0x54ec0f37, + 0x1b80, 0x0ba60f45, + 0x1b80, 0x0ba60f47, + 0x1b80, 0x00260f55, + 0x1b80, 0x00260f57, + 0x1b80, 0xe34a0f65, + 0x1b80, 0xe34a0f67, + 0x1b80, 0x00020f75, + 0x1b80, 0x00020f77, + 0x1b80, 0x63830f85, + 0x1b80, 0x63830f87, + 0x1b80, 0x30d90f95, + 0x1b80, 0x30d90f97, + 0x1b80, 0x311a0fa5, + 0x1b80, 0x311a0fa7, + 0x1b80, 0x00240fb5, + 0x1b80, 0x00240fb7, + 0x1b80, 0xe34a0fc5, + 0x1b80, 0xe34a0fc7, + 0x1b80, 0x00020fd5, + 0x1b80, 0x00020fd7, + 0x1b80, 0x54ea0fe5, + 0x1b80, 0x54ea0fe7, + 0x1b80, 0x0ba60ff5, + 0x1b80, 0x0ba60ff7, + 0x1b80, 0x00241005, + 0x1b80, 0x00241007, + 0x1b80, 0xe34a1015, + 0x1b80, 0xe34a1017, + 0x1b80, 0x00021025, + 0x1b80, 0x00021027, + 0x1b80, 0x63831035, + 0x1b80, 0x63831037, + 0x1b80, 0x30d91045, + 0x1b80, 0x30d91047, + 0x1b80, 0x5c321055, + 0x1b80, 0x5c321057, + 0x1b80, 0x54e61065, + 0x1b80, 0x54e61067, + 0x1b80, 0x6e101075, + 0x1b80, 0x6e101077, + 0x1b80, 0x6f0f1085, + 0x1b80, 0x6f0f1087, + 0x1b80, 0xe3171095, + 0x1b80, 0xe3171097, + 0x1b80, 0xe34a10a5, + 0x1b80, 0xe34a10a7, + 0x1b80, 0x5c3210b5, + 0x1b80, 0x5c3210b7, + 0x1b80, 0x54e710c5, + 0x1b80, 0x54e710c7, + 0x1b80, 0x6e2410d5, + 0x1b80, 0x6e2410d7, + 0x1b80, 0xe31710e5, + 0x1b80, 0xe31710e7, + 0x1b80, 0xe34a10f5, + 0x1b80, 0xe34a10f7, + 0x1b80, 0x5c321105, + 0x1b80, 0x5c321107, + 0x1b80, 0x54e81115, + 0x1b80, 0x54e81117, + 0x1b80, 0x6e441125, + 0x1b80, 0x6e441127, + 0x1b80, 0xe3171135, + 0x1b80, 0xe3171137, + 0x1b80, 0xe34a1145, + 0x1b80, 0xe34a1147, + 0x1b80, 0x5c321155, + 0x1b80, 0x5c321157, + 0x1b80, 0x54e91165, + 0x1b80, 0x54e91167, + 0x1b80, 0x6e641175, + 0x1b80, 0x6e641177, + 0x1b80, 0xe3171185, + 0x1b80, 0xe3171187, + 0x1b80, 0xe34a1195, + 0x1b80, 0xe34a1197, + 0x1b80, 0x5c3211a5, + 0x1b80, 0x5c3211a7, + 0x1b80, 0x54ea11b5, + 0x1b80, 0x54ea11b7, + 0x1b80, 0x0baa11c5, + 0x1b80, 0x0baa11c7, + 0x1b80, 0x6e8411d5, + 0x1b80, 0x6e8411d7, + 0x1b80, 0x6f0f11e5, + 0x1b80, 0x6f0f11e7, + 0x1b80, 0xe31711f5, + 0x1b80, 0xe31711f7, + 0x1b80, 0xe34a1205, + 0x1b80, 0xe34a1207, + 0x1b80, 0x5c321215, + 0x1b80, 0x5c321217, + 0x1b80, 0x54eb1225, + 0x1b80, 0x54eb1227, + 0x1b80, 0x6ea41235, + 0x1b80, 0x6ea41237, + 0x1b80, 0xe3171245, + 0x1b80, 0xe3171247, + 0x1b80, 0xe34a1255, + 0x1b80, 0xe34a1257, + 0x1b80, 0x5c321265, + 0x1b80, 0x5c321267, + 0x1b80, 0x54ec1275, + 0x1b80, 0x54ec1277, + 0x1b80, 0x0bac1285, + 0x1b80, 0x0bac1287, + 0x1b80, 0x6ec41295, + 0x1b80, 0x6ec41297, + 0x1b80, 0x6f0f12a5, + 0x1b80, 0x6f0f12a7, + 0x1b80, 0xe31712b5, + 0x1b80, 0xe31712b7, + 0x1b80, 0xe34a12c5, + 0x1b80, 0xe34a12c7, + 0x1b80, 0x5c3212d5, + 0x1b80, 0x5c3212d7, + 0x1b80, 0x54ed12e5, + 0x1b80, 0x54ed12e7, + 0x1b80, 0x6ee412f5, + 0x1b80, 0x6ee412f7, + 0x1b80, 0xe3171305, + 0x1b80, 0xe3171307, + 0x1b80, 0xe34a1315, + 0x1b80, 0xe34a1317, + 0x1b80, 0x5c321325, + 0x1b80, 0x5c321327, + 0x1b80, 0x54ee1335, + 0x1b80, 0x54ee1337, + 0x1b80, 0x6ef41345, + 0x1b80, 0x6ef41347, + 0x1b80, 0xe3171355, + 0x1b80, 0xe3171357, + 0x1b80, 0xe34a1365, + 0x1b80, 0xe34a1367, + 0x1b80, 0x5c321375, + 0x1b80, 0x5c321377, + 0x1b80, 0x54ef1385, + 0x1b80, 0x54ef1387, + 0x1b80, 0x6e0c1395, + 0x1b80, 0x6e0c1397, + 0x1b80, 0x6f0013a5, + 0x1b80, 0x6f0013a7, + 0x1b80, 0xe31713b5, + 0x1b80, 0xe31713b7, + 0x1b80, 0xe34a13c5, + 0x1b80, 0xe34a13c7, + 0x1b80, 0x5c3213d5, + 0x1b80, 0x5c3213d7, + 0x1b80, 0x54f013e5, + 0x1b80, 0x54f013e7, + 0x1b80, 0x6e1c13f5, + 0x1b80, 0x6e1c13f7, + 0x1b80, 0xe3171405, + 0x1b80, 0xe3171407, + 0x1b80, 0xe34a1415, + 0x1b80, 0xe34a1417, + 0x1b80, 0x5c321425, + 0x1b80, 0x5c321427, + 0x1b80, 0x54f11435, + 0x1b80, 0x54f11437, + 0x1b80, 0x6e3c1445, + 0x1b80, 0x6e3c1447, + 0x1b80, 0xe3171455, + 0x1b80, 0xe3171457, + 0x1b80, 0xe34a1465, + 0x1b80, 0xe34a1467, + 0x1b80, 0xfaa91475, + 0x1b80, 0xfaa91477, + 0x1b80, 0x5c321485, + 0x1b80, 0x5c321487, + 0x1b80, 0x54f21495, + 0x1b80, 0x54f21497, + 0x1b80, 0x6e5c14a5, + 0x1b80, 0x6e5c14a7, + 0x1b80, 0xe31714b5, + 0x1b80, 0xe31714b7, + 0x1b80, 0xe34a14c5, + 0x1b80, 0xe34a14c7, + 0x1b80, 0x5c3214d5, + 0x1b80, 0x5c3214d7, + 0x1b80, 0x54f314e5, + 0x1b80, 0x54f314e7, + 0x1b80, 0x6e7c14f5, + 0x1b80, 0x6e7c14f7, + 0x1b80, 0xe3171505, + 0x1b80, 0xe3171507, + 0x1b80, 0xe34a1515, + 0x1b80, 0xe34a1517, + 0x1b80, 0xfba91525, + 0x1b80, 0xfba91527, + 0x1b80, 0x5c321535, + 0x1b80, 0x5c321537, + 0x1b80, 0x54f41545, + 0x1b80, 0x54f41547, + 0x1b80, 0x6e9c1555, + 0x1b80, 0x6e9c1557, + 0x1b80, 0xe3171565, + 0x1b80, 0xe3171567, + 0x1b80, 0xe34a1575, + 0x1b80, 0xe34a1577, + 0x1b80, 0x5c321585, + 0x1b80, 0x5c321587, + 0x1b80, 0x54f51595, + 0x1b80, 0x54f51597, + 0x1b80, 0x6ebc15a5, + 0x1b80, 0x6ebc15a7, + 0x1b80, 0xe31715b5, + 0x1b80, 0xe31715b7, + 0x1b80, 0xe34a15c5, + 0x1b80, 0xe34a15c7, + 0x1b80, 0x5c3215d5, + 0x1b80, 0x5c3215d7, + 0x1b80, 0x54f615e5, + 0x1b80, 0x54f615e7, + 0x1b80, 0x6edc15f5, + 0x1b80, 0x6edc15f7, + 0x1b80, 0xe3171605, + 0x1b80, 0xe3171607, + 0x1b80, 0xe34a1615, + 0x1b80, 0xe34a1617, + 0x1b80, 0x5c321625, + 0x1b80, 0x5c321627, + 0x1b80, 0x54f71635, + 0x1b80, 0x54f71637, + 0x1b80, 0x6ef01645, + 0x1b80, 0x6ef01647, + 0x1b80, 0xe3171655, + 0x1b80, 0xe3171657, + 0x1b80, 0xe34a1665, + 0x1b80, 0xe34a1667, + 0x1b80, 0x63831675, + 0x1b80, 0x63831677, + 0x1b80, 0x30d91685, + 0x1b80, 0x30d91687, + 0x1b80, 0x00011695, + 0x1b80, 0x00011697, + 0x1b80, 0x000416a5, + 0x1b80, 0x000416a7, + 0x1b80, 0x550116b5, + 0x1b80, 0x550116b7, + 0x1b80, 0x5c3116c5, + 0x1b80, 0x5c3116c7, + 0x1b80, 0x5f8216d5, + 0x1b80, 0x5f8216d7, + 0x1b80, 0x660516e5, + 0x1b80, 0x660516e7, + 0x1b80, 0x000616f5, + 0x1b80, 0x000616f7, + 0x1b80, 0x5d801705, + 0x1b80, 0x5d801707, + 0x1b80, 0x09001715, + 0x1b80, 0x09001717, + 0x1b80, 0x0a011725, + 0x1b80, 0x0a011727, + 0x1b80, 0x0b401735, + 0x1b80, 0x0b401737, + 0x1b80, 0x0d001745, + 0x1b80, 0x0d001747, + 0x1b80, 0x0f011755, + 0x1b80, 0x0f011757, + 0x1b80, 0x002a1765, + 0x1b80, 0x002a1767, + 0x1b80, 0x055a1775, + 0x1b80, 0x055a1777, + 0x1b80, 0x05db1785, + 0x1b80, 0x05db1787, + 0x1b80, 0xe3351795, + 0x1b80, 0xe3351797, + 0x1b80, 0xe2e317a5, + 0x1b80, 0xe2e317a7, + 0x1b80, 0x000617b5, + 0x1b80, 0x000617b7, + 0x1b80, 0x06da17c5, + 0x1b80, 0x06da17c7, + 0x1b80, 0x07db17d5, + 0x1b80, 0x07db17d7, + 0x1b80, 0xe33517e5, + 0x1b80, 0xe33517e7, + 0x1b80, 0xe2e317f5, + 0x1b80, 0xe2e317f7, + 0x1b80, 0xe32c1805, + 0x1b80, 0xe32c1807, + 0x1b80, 0x00021815, + 0x1b80, 0x00021817, + 0x1b80, 0xe3311825, + 0x1b80, 0xe3311827, + 0x1b80, 0x5d001835, + 0x1b80, 0x5d001837, + 0x1b80, 0x00041845, + 0x1b80, 0x00041847, + 0x1b80, 0x5fa21855, + 0x1b80, 0x5fa21857, + 0x1b80, 0x00011865, + 0x1b80, 0x00011867, + 0x1b80, 0xe2571875, + 0x1b80, 0xe2571877, + 0x1b80, 0x74081885, + 0x1b80, 0x74081887, + 0x1b80, 0xe2a11895, + 0x1b80, 0xe2a11897, + 0x1b80, 0xe28318a5, + 0x1b80, 0xe28318a7, + 0x1b80, 0xe2c118b5, + 0x1b80, 0xe2c118b7, + 0x1b80, 0xb90018c5, + 0x1b80, 0xb90018c7, + 0x1b80, 0x990018d5, + 0x1b80, 0x990018d7, + 0x1b80, 0x000618e5, + 0x1b80, 0x000618e7, + 0x1b80, 0x770018f5, + 0x1b80, 0x770018f7, + 0x1b80, 0x00041905, + 0x1b80, 0x00041907, + 0x1b80, 0x49041915, + 0x1b80, 0x49041917, + 0x1b80, 0x4bb01925, + 0x1b80, 0x4bb01927, + 0x1b80, 0x00061935, + 0x1b80, 0x00061937, + 0x1b80, 0x75041945, + 0x1b80, 0x75041947, + 0x1b80, 0x77081955, + 0x1b80, 0x77081957, + 0x1b80, 0x00071965, + 0x1b80, 0x00071967, + 0x1b80, 0x77101975, + 0x1b80, 0x77101977, + 0x1b80, 0x00041985, + 0x1b80, 0x00041987, + 0x1b80, 0x44801995, + 0x1b80, 0x44801997, + 0x1b80, 0x45ff19a5, + 0x1b80, 0x45ff19a7, + 0x1b80, 0x463f19b5, + 0x1b80, 0x463f19b7, + 0x1b80, 0x473119c5, + 0x1b80, 0x473119c7, + 0x1b80, 0x400819d5, + 0x1b80, 0x400819d7, + 0x1b80, 0xe23e19e5, + 0x1b80, 0xe23e19e7, + 0x1b80, 0x000119f5, + 0x1b80, 0x000119f7, + 0x1b80, 0xe2571a05, + 0x1b80, 0xe2571a07, + 0x1b80, 0x74081a15, + 0x1b80, 0x74081a17, + 0x1b80, 0xe2b11a25, + 0x1b80, 0xe2b11a27, + 0x1b80, 0xe2831a35, + 0x1b80, 0xe2831a37, + 0x1b80, 0xe2c71a45, + 0x1b80, 0xe2c71a47, + 0x1b80, 0xb9001a55, + 0x1b80, 0xb9001a57, + 0x1b80, 0x99001a65, + 0x1b80, 0x99001a67, + 0x1b80, 0x00061a75, + 0x1b80, 0x00061a77, + 0x1b80, 0x77001a85, + 0x1b80, 0x77001a87, + 0x1b80, 0x00051a95, + 0x1b80, 0x00051a97, + 0x1b80, 0x61041aa5, + 0x1b80, 0x61041aa7, + 0x1b80, 0x63b01ab5, + 0x1b80, 0x63b01ab7, + 0x1b80, 0x00061ac5, + 0x1b80, 0x00061ac7, + 0x1b80, 0x75081ad5, + 0x1b80, 0x75081ad7, + 0x1b80, 0x77081ae5, + 0x1b80, 0x77081ae7, + 0x1b80, 0x00071af5, + 0x1b80, 0x00071af7, + 0x1b80, 0x77201b05, + 0x1b80, 0x77201b07, + 0x1b80, 0x00051b15, + 0x1b80, 0x00051b17, + 0x1b80, 0x5c801b25, + 0x1b80, 0x5c801b27, + 0x1b80, 0x5dff1b35, + 0x1b80, 0x5dff1b37, + 0x1b80, 0x5e3f1b45, + 0x1b80, 0x5e3f1b47, + 0x1b80, 0x5f311b55, + 0x1b80, 0x5f311b57, + 0x1b80, 0x00041b65, + 0x1b80, 0x00041b67, + 0x1b80, 0x400a1b75, + 0x1b80, 0x400a1b77, + 0x1b80, 0xe23e1b85, + 0x1b80, 0xe23e1b87, + 0x1b80, 0x00011b95, + 0x1b80, 0x00011b97, + 0x1b80, 0xe2571ba5, + 0x1b80, 0xe2571ba7, + 0x1b80, 0x74081bb5, + 0x1b80, 0x74081bb7, + 0x1b80, 0xe2a11bc5, + 0x1b80, 0xe2a11bc7, + 0x1b80, 0xe2831bd5, + 0x1b80, 0xe2831bd7, + 0x1b80, 0xe2c11be5, + 0x1b80, 0xe2c11be7, + 0x1b80, 0xe2cd1bf5, + 0x1b80, 0xe2cd1bf7, + 0x1b80, 0xe2101c05, + 0x1b80, 0xe2101c07, + 0x1b80, 0x00011c15, + 0x1b80, 0x00011c17, + 0x1b80, 0xe2571c25, + 0x1b80, 0xe2571c27, + 0x1b80, 0x74081c35, + 0x1b80, 0x74081c37, + 0x1b80, 0xe2b11c45, + 0x1b80, 0xe2b11c47, + 0x1b80, 0xe2831c55, + 0x1b80, 0xe2831c57, + 0x1b80, 0xe2c71c65, + 0x1b80, 0xe2c71c67, + 0x1b80, 0xe2cd1c75, + 0x1b80, 0xe2cd1c77, + 0x1b80, 0xe2261c85, + 0x1b80, 0xe2261c87, + 0x1b80, 0x00011c95, + 0x1b80, 0x00011c97, + 0x1b80, 0xe26d1ca5, + 0x1b80, 0xe26d1ca7, + 0x1b80, 0x74001cb5, + 0x1b80, 0x74001cb7, + 0x1b80, 0xe2a11cc5, + 0x1b80, 0xe2a11cc7, + 0x1b80, 0xe2921cd5, + 0x1b80, 0xe2921cd7, + 0x1b80, 0xe2c11ce5, + 0x1b80, 0xe2c11ce7, + 0x1b80, 0xe2cd1cf5, + 0x1b80, 0xe2cd1cf7, + 0x1b80, 0xe2101d05, + 0x1b80, 0xe2101d07, + 0x1b80, 0x00011d15, + 0x1b80, 0x00011d17, + 0x1b80, 0xe26d1d25, + 0x1b80, 0xe26d1d27, + 0x1b80, 0x74001d35, + 0x1b80, 0x74001d37, + 0x1b80, 0xe2b11d45, + 0x1b80, 0xe2b11d47, + 0x1b80, 0xe2921d55, + 0x1b80, 0xe2921d57, + 0x1b80, 0xe2c71d65, + 0x1b80, 0xe2c71d67, + 0x1b80, 0xe2cd1d75, + 0x1b80, 0xe2cd1d77, + 0x1b80, 0xe2261d85, + 0x1b80, 0xe2261d87, + 0x1b80, 0x00011d95, + 0x1b80, 0x00011d97, + 0x1b80, 0x00041da5, + 0x1b80, 0x00041da7, + 0x1b80, 0x445b1db5, + 0x1b80, 0x445b1db7, + 0x1b80, 0x47b01dc5, + 0x1b80, 0x47b01dc7, + 0x1b80, 0x47301dd5, + 0x1b80, 0x47301dd7, + 0x1b80, 0x47001de5, + 0x1b80, 0x47001de7, + 0x1b80, 0x00061df5, + 0x1b80, 0x00061df7, + 0x1b80, 0x77081e05, + 0x1b80, 0x77081e07, + 0x1b80, 0x00041e15, + 0x1b80, 0x00041e17, + 0x1b80, 0x49401e25, + 0x1b80, 0x49401e27, + 0x1b80, 0x4bb01e35, + 0x1b80, 0x4bb01e37, + 0x1b80, 0x00071e45, + 0x1b80, 0x00071e47, + 0x1b80, 0x54401e55, + 0x1b80, 0x54401e57, + 0x1b80, 0x00041e65, + 0x1b80, 0x00041e67, + 0x1b80, 0x40081e75, + 0x1b80, 0x40081e77, + 0x1b80, 0x00011e85, + 0x1b80, 0x00011e87, + 0x1b80, 0x00051e95, + 0x1b80, 0x00051e97, + 0x1b80, 0x5c5b1ea5, + 0x1b80, 0x5c5b1ea7, + 0x1b80, 0x5fb01eb5, + 0x1b80, 0x5fb01eb7, + 0x1b80, 0x5f301ec5, + 0x1b80, 0x5f301ec7, + 0x1b80, 0x5f001ed5, + 0x1b80, 0x5f001ed7, + 0x1b80, 0x00061ee5, + 0x1b80, 0x00061ee7, + 0x1b80, 0x77081ef5, + 0x1b80, 0x77081ef7, + 0x1b80, 0x00051f05, + 0x1b80, 0x00051f07, + 0x1b80, 0x61401f15, + 0x1b80, 0x61401f17, + 0x1b80, 0x63b01f25, + 0x1b80, 0x63b01f27, + 0x1b80, 0x00071f35, + 0x1b80, 0x00071f37, + 0x1b80, 0x54401f45, + 0x1b80, 0x54401f47, + 0x1b80, 0x00041f55, + 0x1b80, 0x00041f57, + 0x1b80, 0x40081f65, + 0x1b80, 0x40081f67, + 0x1b80, 0x00011f75, + 0x1b80, 0x00011f77, + 0x1b80, 0xe2571f85, + 0x1b80, 0xe2571f87, + 0x1b80, 0x74081f95, + 0x1b80, 0x74081f97, + 0x1b80, 0xe2a11fa5, + 0x1b80, 0xe2a11fa7, + 0x1b80, 0x00041fb5, + 0x1b80, 0x00041fb7, + 0x1b80, 0x40081fc5, + 0x1b80, 0x40081fc7, + 0x1b80, 0x00011fd5, + 0x1b80, 0x00011fd7, + 0x1b80, 0xe2571fe5, + 0x1b80, 0xe2571fe7, + 0x1b80, 0x74081ff5, + 0x1b80, 0x74081ff7, + 0x1b80, 0xe2b12005, + 0x1b80, 0xe2b12007, + 0x1b80, 0x00042015, + 0x1b80, 0x00042017, + 0x1b80, 0x40082025, + 0x1b80, 0x40082027, + 0x1b80, 0x00012035, + 0x1b80, 0x00012037, + 0x1b80, 0xe26d2045, + 0x1b80, 0xe26d2047, + 0x1b80, 0x74002055, + 0x1b80, 0x74002057, + 0x1b80, 0xe2a12065, + 0x1b80, 0xe2a12067, + 0x1b80, 0x00042075, + 0x1b80, 0x00042077, + 0x1b80, 0x40082085, + 0x1b80, 0x40082087, + 0x1b80, 0x00012095, + 0x1b80, 0x00012097, + 0x1b80, 0xe26d20a5, + 0x1b80, 0xe26d20a7, + 0x1b80, 0x740020b5, + 0x1b80, 0x740020b7, + 0x1b80, 0xe2b120c5, + 0x1b80, 0xe2b120c7, + 0x1b80, 0x000420d5, + 0x1b80, 0x000420d7, + 0x1b80, 0x400820e5, + 0x1b80, 0x400820e7, + 0x1b80, 0x000120f5, + 0x1b80, 0x000120f7, + 0x1b80, 0x00042105, + 0x1b80, 0x00042107, + 0x1b80, 0x49042115, + 0x1b80, 0x49042117, + 0x1b80, 0x4bb02125, + 0x1b80, 0x4bb02127, + 0x1b80, 0x00062135, + 0x1b80, 0x00062137, + 0x1b80, 0x75042145, + 0x1b80, 0x75042147, + 0x1b80, 0x77082155, + 0x1b80, 0x77082157, + 0x1b80, 0x00042165, + 0x1b80, 0x00042167, + 0x1b80, 0x44802175, + 0x1b80, 0x44802177, + 0x1b80, 0x45ff2185, + 0x1b80, 0x45ff2187, + 0x1b80, 0x463f2195, + 0x1b80, 0x463f2197, + 0x1b80, 0x473121a5, + 0x1b80, 0x473121a7, + 0x1b80, 0x400821b5, + 0x1b80, 0x400821b7, + 0x1b80, 0xe23e21c5, + 0x1b80, 0xe23e21c7, + 0x1b80, 0x000421d5, + 0x1b80, 0x000421d7, + 0x1b80, 0x400c21e5, + 0x1b80, 0x400c21e7, + 0x1b80, 0x000621f5, + 0x1b80, 0x000621f7, + 0x1b80, 0x75002205, + 0x1b80, 0x75002207, + 0x1b80, 0x00042215, + 0x1b80, 0x00042217, + 0x1b80, 0x445b2225, + 0x1b80, 0x445b2227, + 0x1b80, 0x47002235, + 0x1b80, 0x47002237, + 0x1b80, 0x40082245, + 0x1b80, 0x40082247, + 0x1b80, 0x00012255, + 0x1b80, 0x00012257, + 0x1b80, 0x00052265, + 0x1b80, 0x00052267, + 0x1b80, 0x61042275, + 0x1b80, 0x61042277, + 0x1b80, 0x63b02285, + 0x1b80, 0x63b02287, + 0x1b80, 0x00062295, + 0x1b80, 0x00062297, + 0x1b80, 0x750822a5, + 0x1b80, 0x750822a7, + 0x1b80, 0x770822b5, + 0x1b80, 0x770822b7, + 0x1b80, 0x000522c5, + 0x1b80, 0x000522c7, + 0x1b80, 0x5c8022d5, + 0x1b80, 0x5c8022d7, + 0x1b80, 0x5dff22e5, + 0x1b80, 0x5dff22e7, + 0x1b80, 0x5e3f22f5, + 0x1b80, 0x5e3f22f7, + 0x1b80, 0x5f312305, + 0x1b80, 0x5f312307, + 0x1b80, 0x00042315, + 0x1b80, 0x00042317, + 0x1b80, 0x400a2325, + 0x1b80, 0x400a2327, + 0x1b80, 0xe23e2335, + 0x1b80, 0xe23e2337, + 0x1b80, 0x00042345, + 0x1b80, 0x00042347, + 0x1b80, 0x400c2355, + 0x1b80, 0x400c2357, + 0x1b80, 0x00062365, + 0x1b80, 0x00062367, + 0x1b80, 0x75002375, + 0x1b80, 0x75002377, + 0x1b80, 0x00052385, + 0x1b80, 0x00052387, + 0x1b80, 0x5c5b2395, + 0x1b80, 0x5c5b2397, + 0x1b80, 0x5f0023a5, + 0x1b80, 0x5f0023a7, + 0x1b80, 0x000423b5, + 0x1b80, 0x000423b7, + 0x1b80, 0x400823c5, + 0x1b80, 0x400823c7, + 0x1b80, 0x000123d5, + 0x1b80, 0x000123d7, + 0x1b80, 0x000723e5, + 0x1b80, 0x000723e7, + 0x1b80, 0x4c1223f5, + 0x1b80, 0x4c1223f7, + 0x1b80, 0x4e202405, + 0x1b80, 0x4e202407, + 0x1b80, 0x00052415, + 0x1b80, 0x00052417, + 0x1b80, 0x598f2425, + 0x1b80, 0x598f2427, + 0x1b80, 0x40022435, + 0x1b80, 0x40022437, + 0x1b80, 0x4c012445, + 0x1b80, 0x4c012447, + 0x1b80, 0x4c002455, + 0x1b80, 0x4c002457, + 0x1b80, 0xab002465, + 0x1b80, 0xab002467, + 0x1b80, 0x40032475, + 0x1b80, 0x40032477, + 0x1b80, 0x49802485, + 0x1b80, 0x49802487, + 0x1b80, 0x56c02495, + 0x1b80, 0x56c02497, + 0x1b80, 0x540224a5, + 0x1b80, 0x540224a7, + 0x1b80, 0x4c0124b5, + 0x1b80, 0x4c0124b7, + 0x1b80, 0x4c0024c5, + 0x1b80, 0x4c0024c7, + 0x1b80, 0xab0024d5, + 0x1b80, 0xab0024d7, + 0x1b80, 0x540024e5, + 0x1b80, 0x540024e7, + 0x1b80, 0x000724f5, + 0x1b80, 0x000724f7, + 0x1b80, 0x4c002505, + 0x1b80, 0x4c002507, + 0x1b80, 0x4e002515, + 0x1b80, 0x4e002517, + 0x1b80, 0x00052525, + 0x1b80, 0x00052527, + 0x1b80, 0x40042535, + 0x1b80, 0x40042537, + 0x1b80, 0x4c012545, + 0x1b80, 0x4c012547, + 0x1b80, 0x4c002555, + 0x1b80, 0x4c002557, + 0x1b80, 0x00012565, + 0x1b80, 0x00012567, + 0x1b80, 0x00042575, + 0x1b80, 0x00042577, + 0x1b80, 0x44802585, + 0x1b80, 0x44802587, + 0x1b80, 0x4b002595, + 0x1b80, 0x4b002597, + 0x1b80, 0x000525a5, + 0x1b80, 0x000525a7, + 0x1b80, 0x5c8025b5, + 0x1b80, 0x5c8025b7, + 0x1b80, 0x630025c5, + 0x1b80, 0x630025c7, + 0x1b80, 0x000725d5, + 0x1b80, 0x000725d7, + 0x1b80, 0x780c25e5, + 0x1b80, 0x780c25e7, + 0x1b80, 0x791925f5, + 0x1b80, 0x791925f7, + 0x1b80, 0x7a002605, + 0x1b80, 0x7a002607, + 0x1b80, 0x7b822615, + 0x1b80, 0x7b822617, + 0x1b80, 0x7b022625, + 0x1b80, 0x7b022627, + 0x1b80, 0x78142635, + 0x1b80, 0x78142637, + 0x1b80, 0x79ee2645, + 0x1b80, 0x79ee2647, + 0x1b80, 0x7a012655, + 0x1b80, 0x7a012657, + 0x1b80, 0x7b832665, + 0x1b80, 0x7b832667, + 0x1b80, 0x7b032675, + 0x1b80, 0x7b032677, + 0x1b80, 0x78282685, + 0x1b80, 0x78282687, + 0x1b80, 0x79b42695, + 0x1b80, 0x79b42697, + 0x1b80, 0x7a0026a5, + 0x1b80, 0x7a0026a7, + 0x1b80, 0x7b0026b5, + 0x1b80, 0x7b0026b7, + 0x1b80, 0x000126c5, + 0x1b80, 0x000126c7, + 0x1b80, 0x000426d5, + 0x1b80, 0x000426d7, + 0x1b80, 0x448026e5, + 0x1b80, 0x448026e7, + 0x1b80, 0x4b0026f5, + 0x1b80, 0x4b0026f7, + 0x1b80, 0x00052705, + 0x1b80, 0x00052707, + 0x1b80, 0x5c802715, + 0x1b80, 0x5c802717, + 0x1b80, 0x63002725, + 0x1b80, 0x63002727, + 0x1b80, 0x00072735, + 0x1b80, 0x00072737, + 0x1b80, 0x78102745, + 0x1b80, 0x78102747, + 0x1b80, 0x79132755, + 0x1b80, 0x79132757, + 0x1b80, 0x7a002765, + 0x1b80, 0x7a002767, + 0x1b80, 0x7b802775, + 0x1b80, 0x7b802777, + 0x1b80, 0x7b002785, + 0x1b80, 0x7b002787, + 0x1b80, 0x78db2795, + 0x1b80, 0x78db2797, + 0x1b80, 0x790027a5, + 0x1b80, 0x790027a7, + 0x1b80, 0x7a0027b5, + 0x1b80, 0x7a0027b7, + 0x1b80, 0x7b8127c5, + 0x1b80, 0x7b8127c7, + 0x1b80, 0x7b0127d5, + 0x1b80, 0x7b0127d7, + 0x1b80, 0x782827e5, + 0x1b80, 0x782827e7, + 0x1b80, 0x79b427f5, + 0x1b80, 0x79b427f7, + 0x1b80, 0x7a002805, + 0x1b80, 0x7a002807, + 0x1b80, 0x7b002815, + 0x1b80, 0x7b002817, + 0x1b80, 0x00012825, + 0x1b80, 0x00012827, + 0x1b80, 0x00072835, + 0x1b80, 0x00072837, + 0x1b80, 0x783e2845, + 0x1b80, 0x783e2847, + 0x1b80, 0x79f92855, + 0x1b80, 0x79f92857, + 0x1b80, 0x7a012865, + 0x1b80, 0x7a012867, + 0x1b80, 0x7b822875, + 0x1b80, 0x7b822877, + 0x1b80, 0x7b022885, + 0x1b80, 0x7b022887, + 0x1b80, 0x78a92895, + 0x1b80, 0x78a92897, + 0x1b80, 0x79ed28a5, + 0x1b80, 0x79ed28a7, + 0x1b80, 0x7b8328b5, + 0x1b80, 0x7b8328b7, + 0x1b80, 0x7b0328c5, + 0x1b80, 0x7b0328c7, + 0x1b80, 0x782828d5, + 0x1b80, 0x782828d7, + 0x1b80, 0x79b428e5, + 0x1b80, 0x79b428e7, + 0x1b80, 0x7a0028f5, + 0x1b80, 0x7a0028f7, + 0x1b80, 0x7b002905, + 0x1b80, 0x7b002907, + 0x1b80, 0x00012915, + 0x1b80, 0x00012917, + 0x1b80, 0x00072925, + 0x1b80, 0x00072927, + 0x1b80, 0x78ae2935, + 0x1b80, 0x78ae2937, + 0x1b80, 0x79fa2945, + 0x1b80, 0x79fa2947, + 0x1b80, 0x7a012955, + 0x1b80, 0x7a012957, + 0x1b80, 0x7b802965, + 0x1b80, 0x7b802967, + 0x1b80, 0x7b002975, + 0x1b80, 0x7b002977, + 0x1b80, 0x787a2985, + 0x1b80, 0x787a2987, + 0x1b80, 0x79f12995, + 0x1b80, 0x79f12997, + 0x1b80, 0x7b8129a5, + 0x1b80, 0x7b8129a7, + 0x1b80, 0x7b0129b5, + 0x1b80, 0x7b0129b7, + 0x1b80, 0x782829c5, + 0x1b80, 0x782829c7, + 0x1b80, 0x79b429d5, + 0x1b80, 0x79b429d7, + 0x1b80, 0x7a0029e5, + 0x1b80, 0x7a0029e7, + 0x1b80, 0x7b0029f5, + 0x1b80, 0x7b0029f7, + 0x1b80, 0x00012a05, + 0x1b80, 0x00012a07, + 0x1b80, 0x00072a15, + 0x1b80, 0x00072a17, + 0x1b80, 0x75002a25, + 0x1b80, 0x75002a27, + 0x1b80, 0x76022a35, + 0x1b80, 0x76022a37, + 0x1b80, 0x77152a45, + 0x1b80, 0x77152a47, + 0x1b80, 0x00062a55, + 0x1b80, 0x00062a57, + 0x1b80, 0x74002a65, + 0x1b80, 0x74002a67, + 0x1b80, 0x76002a75, + 0x1b80, 0x76002a77, + 0x1b80, 0x77002a85, + 0x1b80, 0x77002a87, + 0x1b80, 0x75102a95, + 0x1b80, 0x75102a97, + 0x1b80, 0x75002aa5, + 0x1b80, 0x75002aa7, + 0x1b80, 0xb3002ab5, + 0x1b80, 0xb3002ab7, + 0x1b80, 0x93002ac5, + 0x1b80, 0x93002ac7, + 0x1b80, 0x00072ad5, + 0x1b80, 0x00072ad7, + 0x1b80, 0x76002ae5, + 0x1b80, 0x76002ae7, + 0x1b80, 0x77002af5, + 0x1b80, 0x77002af7, + 0x1b80, 0x00012b05, + 0x1b80, 0x00012b07, + 0x1b80, 0x00072b15, + 0x1b80, 0x00072b17, + 0x1b80, 0x75002b25, + 0x1b80, 0x75002b27, + 0x1b80, 0x76022b35, + 0x1b80, 0x76022b37, + 0x1b80, 0x77252b45, + 0x1b80, 0x77252b47, + 0x1b80, 0x00062b55, + 0x1b80, 0x00062b57, + 0x1b80, 0x74002b65, + 0x1b80, 0x74002b67, + 0x1b80, 0x76002b75, + 0x1b80, 0x76002b77, + 0x1b80, 0x77012b85, + 0x1b80, 0x77012b87, + 0x1b80, 0x75102b95, + 0x1b80, 0x75102b97, + 0x1b80, 0x75002ba5, + 0x1b80, 0x75002ba7, + 0x1b80, 0xb3002bb5, + 0x1b80, 0xb3002bb7, + 0x1b80, 0x93002bc5, + 0x1b80, 0x93002bc7, + 0x1b80, 0x00072bd5, + 0x1b80, 0x00072bd7, + 0x1b80, 0x76002be5, + 0x1b80, 0x76002be7, + 0x1b80, 0x77002bf5, + 0x1b80, 0x77002bf7, + 0x1b80, 0x00012c05, + 0x1b80, 0x00012c07, + 0x1b80, 0x00042c15, + 0x1b80, 0x00042c17, + 0x1b80, 0x44802c25, + 0x1b80, 0x44802c27, + 0x1b80, 0x47302c35, + 0x1b80, 0x47302c37, + 0x1b80, 0x00062c45, + 0x1b80, 0x00062c47, + 0x1b80, 0x776c2c55, + 0x1b80, 0x776c2c57, + 0x1b80, 0x00012c65, + 0x1b80, 0x00012c67, + 0x1b80, 0x00052c75, + 0x1b80, 0x00052c77, + 0x1b80, 0x5c802c85, + 0x1b80, 0x5c802c87, + 0x1b80, 0x5f302c95, + 0x1b80, 0x5f302c97, + 0x1b80, 0x00062ca5, + 0x1b80, 0x00062ca7, + 0x1b80, 0x776d2cb5, + 0x1b80, 0x776d2cb7, + 0x1b80, 0x00012cc5, + 0x1b80, 0x00012cc7, + 0x1b80, 0xb9002cd5, + 0x1b80, 0xb9002cd7, + 0x1b80, 0x99002ce5, + 0x1b80, 0x99002ce7, + 0x1b80, 0x00062cf5, + 0x1b80, 0x00062cf7, + 0x1b80, 0x77002d05, + 0x1b80, 0x77002d07, + 0x1b80, 0x98052d15, + 0x1b80, 0x98052d17, + 0x1b80, 0x00042d25, + 0x1b80, 0x00042d27, + 0x1b80, 0x40082d35, + 0x1b80, 0x40082d37, + 0x1b80, 0x4a022d45, + 0x1b80, 0x4a022d47, + 0x1b80, 0x30192d55, + 0x1b80, 0x30192d57, + 0x1b80, 0x00012d65, + 0x1b80, 0x00012d67, + 0x1b80, 0x7b482d75, + 0x1b80, 0x7b482d77, + 0x1b80, 0x7a902d85, + 0x1b80, 0x7a902d87, + 0x1b80, 0x79002d95, + 0x1b80, 0x79002d97, + 0x1b80, 0x55032da5, + 0x1b80, 0x55032da7, + 0x1b80, 0x32e32db5, + 0x1b80, 0x32e32db7, + 0x1b80, 0x7b382dc5, + 0x1b80, 0x7b382dc7, + 0x1b80, 0x7a802dd5, + 0x1b80, 0x7a802dd7, + 0x1b80, 0x550b2de5, + 0x1b80, 0x550b2de7, + 0x1b80, 0x32e32df5, + 0x1b80, 0x32e32df7, + 0x1b80, 0x7b402e05, + 0x1b80, 0x7b402e07, + 0x1b80, 0x7a002e15, + 0x1b80, 0x7a002e17, + 0x1b80, 0x55132e25, + 0x1b80, 0x55132e27, + 0x1b80, 0x74012e35, + 0x1b80, 0x74012e37, + 0x1b80, 0x74002e45, + 0x1b80, 0x74002e47, + 0x1b80, 0x8e002e55, + 0x1b80, 0x8e002e57, + 0x1b80, 0x00012e65, + 0x1b80, 0x00012e67, + 0x1b80, 0x57022e75, + 0x1b80, 0x57022e77, + 0x1b80, 0x57002e85, + 0x1b80, 0x57002e87, + 0x1b80, 0x97002e95, + 0x1b80, 0x97002e97, + 0x1b80, 0x00012ea5, + 0x1b80, 0x00012ea7, + 0x1b80, 0x4f782eb5, + 0x1b80, 0x4f782eb7, + 0x1b80, 0x53882ec5, + 0x1b80, 0x53882ec7, + 0x1b80, 0xe2f72ed5, + 0x1b80, 0xe2f72ed7, + 0x1b80, 0x54802ee5, + 0x1b80, 0x54802ee7, + 0x1b80, 0x54002ef5, + 0x1b80, 0x54002ef7, + 0x1b80, 0x54812f05, + 0x1b80, 0x54812f07, + 0x1b80, 0x54002f15, + 0x1b80, 0x54002f17, + 0x1b80, 0x54822f25, + 0x1b80, 0x54822f27, + 0x1b80, 0x54002f35, + 0x1b80, 0x54002f37, + 0x1b80, 0xe3022f45, + 0x1b80, 0xe3022f47, + 0x1b80, 0xbf1d2f55, + 0x1b80, 0xbf1d2f57, + 0x1b80, 0x30192f65, + 0x1b80, 0x30192f67, + 0x1b80, 0xe2d72f75, + 0x1b80, 0xe2d72f77, + 0x1b80, 0xe2dc2f85, + 0x1b80, 0xe2dc2f87, + 0x1b80, 0xe2e02f95, + 0x1b80, 0xe2e02f97, + 0x1b80, 0xe2e72fa5, + 0x1b80, 0xe2e72fa7, + 0x1b80, 0xe3412fb5, + 0x1b80, 0xe3412fb7, + 0x1b80, 0x55132fc5, + 0x1b80, 0x55132fc7, + 0x1b80, 0xe2e32fd5, + 0x1b80, 0xe2e32fd7, + 0x1b80, 0x55152fe5, + 0x1b80, 0x55152fe7, + 0x1b80, 0xe2e72ff5, + 0x1b80, 0xe2e72ff7, + 0x1b80, 0xe3413005, + 0x1b80, 0xe3413007, + 0x1b80, 0x00013015, + 0x1b80, 0x00013017, + 0x1b80, 0x54bf3025, + 0x1b80, 0x54bf3027, + 0x1b80, 0x54c03035, + 0x1b80, 0x54c03037, + 0x1b80, 0x54a33045, + 0x1b80, 0x54a33047, + 0x1b80, 0x54c13055, + 0x1b80, 0x54c13057, + 0x1b80, 0x54a43065, + 0x1b80, 0x54a43067, + 0x1b80, 0x4c183075, + 0x1b80, 0x4c183077, + 0x1b80, 0xbf073085, + 0x1b80, 0xbf073087, + 0x1b80, 0x54c23095, + 0x1b80, 0x54c23097, + 0x1b80, 0x54a430a5, + 0x1b80, 0x54a430a7, + 0x1b80, 0xbf0430b5, + 0x1b80, 0xbf0430b7, + 0x1b80, 0x54c130c5, + 0x1b80, 0x54c130c7, + 0x1b80, 0x54a330d5, + 0x1b80, 0x54a330d7, + 0x1b80, 0xbf0130e5, + 0x1b80, 0xbf0130e7, + 0x1b80, 0xe34f30f5, + 0x1b80, 0xe34f30f7, + 0x1b80, 0x54df3105, + 0x1b80, 0x54df3107, + 0x1b80, 0x00013115, + 0x1b80, 0x00013117, + 0x1b80, 0x54bf3125, + 0x1b80, 0x54bf3127, + 0x1b80, 0x54e53135, + 0x1b80, 0x54e53137, + 0x1b80, 0x050a3145, + 0x1b80, 0x050a3147, + 0x1b80, 0x54df3155, + 0x1b80, 0x54df3157, + 0x1b80, 0x00013165, + 0x1b80, 0x00013167, + 0x1b80, 0x7f403175, + 0x1b80, 0x7f403177, + 0x1b80, 0x7e003185, + 0x1b80, 0x7e003187, + 0x1b80, 0x7d003195, + 0x1b80, 0x7d003197, + 0x1b80, 0x550131a5, + 0x1b80, 0x550131a7, + 0x1b80, 0x5c3131b5, + 0x1b80, 0x5c3131b7, + 0x1b80, 0xe2e331c5, + 0x1b80, 0xe2e331c7, + 0x1b80, 0xe2e731d5, + 0x1b80, 0xe2e731d7, + 0x1b80, 0x548031e5, + 0x1b80, 0x548031e7, + 0x1b80, 0x540031f5, + 0x1b80, 0x540031f7, + 0x1b80, 0x54813205, + 0x1b80, 0x54813207, + 0x1b80, 0x54003215, + 0x1b80, 0x54003217, + 0x1b80, 0x54823225, + 0x1b80, 0x54823227, + 0x1b80, 0x54003235, + 0x1b80, 0x54003237, + 0x1b80, 0xe3023245, + 0x1b80, 0xe3023247, + 0x1b80, 0xbfed3255, + 0x1b80, 0xbfed3257, + 0x1b80, 0x30193265, + 0x1b80, 0x30193267, + 0x1b80, 0x74023275, + 0x1b80, 0x74023277, + 0x1b80, 0x003f3285, + 0x1b80, 0x003f3287, + 0x1b80, 0x74003295, + 0x1b80, 0x74003297, + 0x1b80, 0x000232a5, + 0x1b80, 0x000232a7, + 0x1b80, 0x000132b5, + 0x1b80, 0x000132b7, + 0x1b80, 0x000632c5, + 0x1b80, 0x000632c7, + 0x1b80, 0x5a8032d5, + 0x1b80, 0x5a8032d7, + 0x1b80, 0x5a0032e5, + 0x1b80, 0x5a0032e7, + 0x1b80, 0x920032f5, + 0x1b80, 0x920032f7, + 0x1b80, 0x00013305, + 0x1b80, 0x00013307, + 0x1b80, 0x5b8f3315, + 0x1b80, 0x5b8f3317, + 0x1b80, 0x5b0f3325, + 0x1b80, 0x5b0f3327, + 0x1b80, 0x91003335, + 0x1b80, 0x91003337, + 0x1b80, 0x00013345, + 0x1b80, 0x00013347, + 0x1b80, 0x00063355, + 0x1b80, 0x00063357, + 0x1b80, 0x5d803365, + 0x1b80, 0x5d803367, + 0x1b80, 0x5e563375, + 0x1b80, 0x5e563377, + 0x1b80, 0x00043385, + 0x1b80, 0x00043387, + 0x1b80, 0x4d083395, + 0x1b80, 0x4d083397, + 0x1b80, 0x571033a5, + 0x1b80, 0x571033a7, + 0x1b80, 0x570033b5, + 0x1b80, 0x570033b7, + 0x1b80, 0x4d0033c5, + 0x1b80, 0x4d0033c7, + 0x1b80, 0x000633d5, + 0x1b80, 0x000633d7, + 0x1b80, 0x5d0033e5, + 0x1b80, 0x5d0033e7, + 0x1b80, 0x000433f5, + 0x1b80, 0x000433f7, + 0x1b80, 0x00013405, + 0x1b80, 0x00013407, + 0x1b80, 0x549f3415, + 0x1b80, 0x549f3417, + 0x1b80, 0x54ff3425, + 0x1b80, 0x54ff3427, + 0x1b80, 0x54003435, + 0x1b80, 0x54003437, + 0x1b80, 0x00013445, + 0x1b80, 0x00013447, + 0x1b80, 0x5c313455, + 0x1b80, 0x5c313457, + 0x1b80, 0x07143465, + 0x1b80, 0x07143467, + 0x1b80, 0x54003475, + 0x1b80, 0x54003477, + 0x1b80, 0x5c323485, + 0x1b80, 0x5c323487, + 0x1b80, 0x00013495, + 0x1b80, 0x00013497, + 0x1b80, 0x5c3234a5, + 0x1b80, 0x5c3234a7, + 0x1b80, 0x071434b5, + 0x1b80, 0x071434b7, + 0x1b80, 0x540034c5, + 0x1b80, 0x540034c7, + 0x1b80, 0x5c3134d5, + 0x1b80, 0x5c3134d7, + 0x1b80, 0x000134e5, + 0x1b80, 0x000134e7, + 0x1b80, 0x4c9834f5, + 0x1b80, 0x4c9834f7, + 0x1b80, 0x4c183505, + 0x1b80, 0x4c183507, + 0x1b80, 0x00013515, + 0x1b80, 0x00013517, + 0x1b80, 0x5c323525, + 0x1b80, 0x5c323527, + 0x1b80, 0x62043535, + 0x1b80, 0x62043537, + 0x1b80, 0x63033545, + 0x1b80, 0x63033547, + 0x1b80, 0x66073555, + 0x1b80, 0x66073557, + 0x1b80, 0x7b403565, + 0x1b80, 0x7b403567, + 0x1b80, 0x7a003575, + 0x1b80, 0x7a003577, + 0x1b80, 0x79003585, + 0x1b80, 0x79003587, + 0x1b80, 0x7f403595, + 0x1b80, 0x7f403597, + 0x1b80, 0x7e0035a5, + 0x1b80, 0x7e0035a7, + 0x1b80, 0x7d0035b5, + 0x1b80, 0x7d0035b7, + 0x1b80, 0x090135c5, + 0x1b80, 0x090135c7, + 0x1b80, 0x0c0135d5, + 0x1b80, 0x0c0135d7, + 0x1b80, 0x0ba635e5, + 0x1b80, 0x0ba635e7, + 0x1b80, 0x000135f5, + 0x1b80, 0x000135f7, + 0x1b80, 0x00000006, + 0x1b80, 0x00000002, +}; + +RTW_DECL_TABLE_PHY_COND(rtw8822c_array_mp_cal_init, rtw_phy_cfg_bb); diff --git a/drivers/net/wireless/realtek/rtw88/rtw8822c_table.h b/drivers/net/wireless/realtek/rtw88/rtw8822c_table.h new file mode 100644 index 0000000000000000000000000000000000000000..06e207dd8e5fdc7faed10860905f07faced22d28 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/rtw8822c_table.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#ifndef __RTW8822C_TABLE_H__ +#define __RTW8822C_TABLE_H__ + +extern const struct rtw_table rtw8822c_mac_tbl; +extern const struct rtw_table rtw8822c_agc_tbl; +extern const struct rtw_table rtw8822c_bb_tbl; +extern const struct rtw_table rtw8822c_bb_pg_type0_tbl; +extern const struct rtw_table rtw8822c_rf_a_tbl; +extern const struct rtw_table rtw8822c_rf_b_tbl; +extern const struct rtw_table rtw8822c_txpwr_lmt_type0_tbl; +extern const struct rtw_table rtw8822c_array_mp_cal_init_tbl; + +#endif diff --git a/drivers/net/wireless/realtek/rtw88/rx.c b/drivers/net/wireless/realtek/rtw88/rx.c new file mode 100644 index 0000000000000000000000000000000000000000..4d837f0c6d5f857f33fa491e80236d8f4bdb5bc8 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/rx.c @@ -0,0 +1,151 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#include "main.h" +#include "rx.h" +#include "ps.h" + +void rtw_rx_stats(struct rtw_dev *rtwdev, struct ieee80211_vif *vif, + struct sk_buff *skb) +{ + struct ieee80211_hdr *hdr; + struct rtw_vif *rtwvif; + + hdr = (struct ieee80211_hdr *)skb->data; + + if (!ieee80211_is_data(hdr->frame_control)) + return; + + if (!is_broadcast_ether_addr(hdr->addr1) && + !is_multicast_ether_addr(hdr->addr1)) { + rtwdev->stats.rx_unicast += skb->len; + rtwdev->stats.rx_cnt++; + if (vif) { + rtwvif = (struct rtw_vif *)vif->drv_priv; + rtwvif->stats.rx_unicast += skb->len; + rtwvif->stats.rx_cnt++; + if (rtwvif->stats.rx_cnt > RTW_LPS_THRESHOLD) + rtw_leave_lps_irqsafe(rtwdev, rtwvif); + } + } +} +EXPORT_SYMBOL(rtw_rx_stats); + +struct rtw_rx_addr_match_data { + struct rtw_dev *rtwdev; + struct ieee80211_hdr *hdr; + struct rtw_rx_pkt_stat *pkt_stat; + u8 *bssid; +}; + +static void rtw_rx_addr_match_iter(void *data, u8 *mac, + struct ieee80211_vif *vif) +{ + struct rtw_rx_addr_match_data *iter_data = data; + struct ieee80211_sta *sta; + struct ieee80211_hdr *hdr = iter_data->hdr; + struct rtw_dev *rtwdev = iter_data->rtwdev; + struct rtw_sta_info *si; + struct rtw_rx_pkt_stat *pkt_stat = iter_data->pkt_stat; + u8 *bssid = iter_data->bssid; + + if (ether_addr_equal(vif->bss_conf.bssid, bssid) && + (ether_addr_equal(vif->addr, hdr->addr1) || + ieee80211_is_beacon(hdr->frame_control))) + sta = ieee80211_find_sta_by_ifaddr(rtwdev->hw, hdr->addr2, + vif->addr); + else + return; + + if (!sta) + return; + + si = (struct rtw_sta_info *)sta->drv_priv; + ewma_rssi_add(&si->avg_rssi, pkt_stat->rssi); +} + +static void rtw_rx_addr_match(struct rtw_dev *rtwdev, + struct rtw_rx_pkt_stat *pkt_stat, + struct ieee80211_hdr *hdr) +{ + struct rtw_rx_addr_match_data data = {}; + + if (pkt_stat->crc_err || pkt_stat->icv_err || !pkt_stat->phy_status || + ieee80211_is_ctl(hdr->frame_control)) + return; + + data.rtwdev = rtwdev; + data.hdr = hdr; + data.pkt_stat = pkt_stat; + data.bssid = get_hdr_bssid(hdr); + + rtw_iterate_vifs_atomic(rtwdev, rtw_rx_addr_match_iter, &data); +} + +void rtw_rx_fill_rx_status(struct rtw_dev *rtwdev, + struct rtw_rx_pkt_stat *pkt_stat, + struct ieee80211_hdr *hdr, + struct ieee80211_rx_status *rx_status, + u8 *phy_status) +{ + struct ieee80211_hw *hw = rtwdev->hw; + + memset(rx_status, 0, sizeof(*rx_status)); + rx_status->freq = hw->conf.chandef.chan->center_freq; + rx_status->band = hw->conf.chandef.chan->band; + if (pkt_stat->crc_err) + rx_status->flag |= RX_FLAG_FAILED_FCS_CRC; + if (pkt_stat->decrypted) + rx_status->flag |= RX_FLAG_DECRYPTED; + + if (pkt_stat->rate >= DESC_RATEVHT1SS_MCS0) + rx_status->encoding = RX_ENC_VHT; + else if (pkt_stat->rate >= DESC_RATEMCS0) + rx_status->encoding = RX_ENC_HT; + + if (pkt_stat->rate >= DESC_RATEVHT1SS_MCS0 && + pkt_stat->rate <= DESC_RATEVHT1SS_MCS9) { + rx_status->nss = 1; + rx_status->rate_idx = pkt_stat->rate - DESC_RATEVHT1SS_MCS0; + } else if (pkt_stat->rate >= DESC_RATEVHT2SS_MCS0 && + pkt_stat->rate <= DESC_RATEVHT2SS_MCS9) { + rx_status->nss = 2; + rx_status->rate_idx = pkt_stat->rate - DESC_RATEVHT2SS_MCS0; + } else if (pkt_stat->rate >= DESC_RATEVHT3SS_MCS0 && + pkt_stat->rate <= DESC_RATEVHT3SS_MCS9) { + rx_status->nss = 3; + rx_status->rate_idx = pkt_stat->rate - DESC_RATEVHT3SS_MCS0; + } else if (pkt_stat->rate >= DESC_RATEVHT4SS_MCS0 && + pkt_stat->rate <= DESC_RATEVHT4SS_MCS9) { + rx_status->nss = 4; + rx_status->rate_idx = pkt_stat->rate - DESC_RATEVHT4SS_MCS0; + } else if (pkt_stat->rate >= DESC_RATEMCS0 && + pkt_stat->rate <= DESC_RATEMCS15) { + rx_status->rate_idx = pkt_stat->rate - DESC_RATEMCS0; + } else if (rx_status->band == NL80211_BAND_5GHZ && + pkt_stat->rate >= DESC_RATE6M && + pkt_stat->rate <= DESC_RATE54M) { + rx_status->rate_idx = pkt_stat->rate - DESC_RATE6M; + } else if (rx_status->band == NL80211_BAND_2GHZ && + pkt_stat->rate >= DESC_RATE1M && + pkt_stat->rate <= DESC_RATE54M) { + rx_status->rate_idx = pkt_stat->rate - DESC_RATE1M; + } else { + rx_status->rate_idx = 0; + } + + rx_status->flag |= RX_FLAG_MACTIME_START; + rx_status->mactime = pkt_stat->tsf_low; + + if (pkt_stat->bw == RTW_CHANNEL_WIDTH_80) + rx_status->bw = RATE_INFO_BW_80; + else if (pkt_stat->bw == RTW_CHANNEL_WIDTH_40) + rx_status->bw = RATE_INFO_BW_40; + else + rx_status->bw = RATE_INFO_BW_20; + + rx_status->signal = pkt_stat->signal_power; + + rtw_rx_addr_match(rtwdev, pkt_stat, hdr); +} diff --git a/drivers/net/wireless/realtek/rtw88/rx.h b/drivers/net/wireless/realtek/rtw88/rx.h new file mode 100644 index 0000000000000000000000000000000000000000..383f3b2babc101d59d39db24ba2bfaaa3b025717 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/rx.h @@ -0,0 +1,41 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#ifndef __RTW_RX_H_ +#define __RTW_RX_H_ + +#define GET_RX_DESC_PHYST(rxdesc) \ + le32_get_bits(*((__le32 *)(rxdesc) + 0x00), BIT(26)) +#define GET_RX_DESC_ICV_ERR(rxdesc) \ + le32_get_bits(*((__le32 *)(rxdesc) + 0x00), BIT(15)) +#define GET_RX_DESC_CRC32(rxdesc) \ + le32_get_bits(*((__le32 *)(rxdesc) + 0x00), BIT(14)) +#define GET_RX_DESC_SWDEC(rxdesc) \ + le32_get_bits(*((__le32 *)(rxdesc) + 0x00), BIT(27)) +#define GET_RX_DESC_C2H(rxdesc) \ + le32_get_bits(*((__le32 *)(rxdesc) + 0x02), BIT(28)) +#define GET_RX_DESC_PKT_LEN(rxdesc) \ + le32_get_bits(*((__le32 *)(rxdesc) + 0x00), GENMASK(13, 0)) +#define GET_RX_DESC_DRV_INFO_SIZE(rxdesc) \ + le32_get_bits(*((__le32 *)(rxdesc) + 0x00), GENMASK(19, 16)) +#define GET_RX_DESC_SHIFT(rxdesc) \ + le32_get_bits(*((__le32 *)(rxdesc) + 0x00), GENMASK(25, 24)) +#define GET_RX_DESC_RX_RATE(rxdesc) \ + le32_get_bits(*((__le32 *)(rxdesc) + 0x03), GENMASK(6, 0)) +#define GET_RX_DESC_MACID(rxdesc) \ + le32_get_bits(*((__le32 *)(rxdesc) + 0x01), GENMASK(6, 0)) +#define GET_RX_DESC_PPDU_CNT(rxdesc) \ + le32_get_bits(*((__le32 *)(rxdesc) + 0x02), GENMASK(30, 29)) +#define GET_RX_DESC_TSFL(rxdesc) \ + le32_get_bits(*((__le32 *)(rxdesc) + 0x05), GENMASK(31, 0)) + +void rtw_rx_stats(struct rtw_dev *rtwdev, struct ieee80211_vif *vif, + struct sk_buff *skb); +void rtw_rx_fill_rx_status(struct rtw_dev *rtwdev, + struct rtw_rx_pkt_stat *pkt_stat, + struct ieee80211_hdr *hdr, + struct ieee80211_rx_status *rx_status, + u8 *phy_status); + +#endif diff --git a/drivers/net/wireless/realtek/rtw88/sec.c b/drivers/net/wireless/realtek/rtw88/sec.c new file mode 100644 index 0000000000000000000000000000000000000000..c594fc02804df4a3b0a2b842c3f960a728c4440e --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/sec.c @@ -0,0 +1,120 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#include "main.h" +#include "sec.h" +#include "reg.h" + +int rtw_sec_get_free_cam(struct rtw_sec_desc *sec) +{ + /* if default key search is enabled, the first 4 cam entries + * are used to direct map to group key with its key->key_idx, so + * driver should use cam entries after 4 to install pairwise key + */ + if (sec->default_key_search) + return find_next_zero_bit(sec->cam_map, RTW_MAX_SEC_CAM_NUM, + RTW_SEC_DEFAULT_KEY_NUM); + + return find_first_zero_bit(sec->cam_map, RTW_MAX_SEC_CAM_NUM); +} + +void rtw_sec_write_cam(struct rtw_dev *rtwdev, + struct rtw_sec_desc *sec, + struct ieee80211_sta *sta, + struct ieee80211_key_conf *key, + u8 hw_key_type, u8 hw_key_idx) +{ + struct rtw_cam_entry *cam = &sec->cam_table[hw_key_idx]; + u32 write_cmd; + u32 command; + u32 content; + u32 addr; + int i, j; + + set_bit(hw_key_idx, sec->cam_map); + cam->valid = true; + cam->group = !(key->flags & IEEE80211_KEY_FLAG_PAIRWISE); + cam->hw_key_type = hw_key_type; + cam->key = key; + if (sta) + ether_addr_copy(cam->addr, sta->addr); + else + eth_broadcast_addr(cam->addr); + + write_cmd = RTW_SEC_CMD_WRITE_ENABLE | RTW_SEC_CMD_POLLING; + addr = hw_key_idx << RTW_SEC_CAM_ENTRY_SHIFT; + for (i = 5; i >= 0; i--) { + switch (i) { + case 0: + content = ((key->keyidx & 0x3)) | + ((hw_key_type & 0x7) << 2) | + (cam->group << 6) | + (cam->valid << 15) | + (cam->addr[0] << 16) | + (cam->addr[1] << 24); + break; + case 1: + content = (cam->addr[2]) | + (cam->addr[3] << 8) | + (cam->addr[4] << 16) | + (cam->addr[5] << 24); + break; + default: + j = (i - 2) << 2; + content = (key->key[j]) | + (key->key[j + 1] << 8) | + (key->key[j + 2] << 16) | + (key->key[j + 3] << 24); + break; + } + + command = write_cmd | (addr + i); + rtw_write32(rtwdev, RTW_SEC_WRITE_REG, content); + rtw_write32(rtwdev, RTW_SEC_CMD_REG, command); + } +} + +void rtw_sec_clear_cam(struct rtw_dev *rtwdev, + struct rtw_sec_desc *sec, + u8 hw_key_idx) +{ + struct rtw_cam_entry *cam = &sec->cam_table[hw_key_idx]; + u32 write_cmd; + u32 command; + u32 addr; + + clear_bit(hw_key_idx, sec->cam_map); + cam->valid = false; + cam->key = NULL; + eth_zero_addr(cam->addr); + + write_cmd = RTW_SEC_CMD_WRITE_ENABLE | RTW_SEC_CMD_POLLING; + addr = hw_key_idx << RTW_SEC_CAM_ENTRY_SHIFT; + command = write_cmd | addr; + rtw_write32(rtwdev, RTW_SEC_WRITE_REG, 0); + rtw_write32(rtwdev, RTW_SEC_CMD_REG, command); +} + +void rtw_sec_enable_sec_engine(struct rtw_dev *rtwdev) +{ + struct rtw_sec_desc *sec = &rtwdev->sec; + u16 ctrl_reg; + u16 sec_config; + + /* default use default key search for now */ + sec->default_key_search = true; + + ctrl_reg = rtw_read16(rtwdev, REG_CR); + ctrl_reg |= RTW_SEC_ENGINE_EN; + rtw_write16(rtwdev, REG_CR, ctrl_reg); + + sec_config = rtw_read16(rtwdev, RTW_SEC_CONFIG); + + sec_config |= RTW_SEC_TX_DEC_EN | RTW_SEC_RX_DEC_EN; + if (sec->default_key_search) + sec_config |= RTW_SEC_TX_UNI_USE_DK | RTW_SEC_RX_UNI_USE_DK | + RTW_SEC_TX_BC_USE_DK | RTW_SEC_RX_BC_USE_DK; + + rtw_write16(rtwdev, RTW_SEC_CONFIG, sec_config); +} diff --git a/drivers/net/wireless/realtek/rtw88/sec.h b/drivers/net/wireless/realtek/rtw88/sec.h new file mode 100644 index 0000000000000000000000000000000000000000..8c50a895c7977c6fb94af71cd8d96ecd441bd8dc --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/sec.h @@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#ifndef __RTW_SEC_H_ +#define __RTW_SEC_H_ + +#define RTW_SEC_CMD_REG 0x670 +#define RTW_SEC_WRITE_REG 0x674 +#define RTW_SEC_READ_REG 0x678 +#define RTW_SEC_CONFIG 0x680 + +#define RTW_SEC_CAM_ENTRY_SHIFT 3 +#define RTW_SEC_DEFAULT_KEY_NUM 4 +#define RTW_SEC_CMD_WRITE_ENABLE BIT(16) +#define RTW_SEC_CMD_CLEAR BIT(30) +#define RTW_SEC_CMD_POLLING BIT(31) + +#define RTW_SEC_TX_UNI_USE_DK BIT(0) +#define RTW_SEC_RX_UNI_USE_DK BIT(1) +#define RTW_SEC_TX_DEC_EN BIT(2) +#define RTW_SEC_RX_DEC_EN BIT(3) +#define RTW_SEC_TX_BC_USE_DK BIT(6) +#define RTW_SEC_RX_BC_USE_DK BIT(7) + +#define RTW_SEC_ENGINE_EN BIT(9) + +int rtw_sec_get_free_cam(struct rtw_sec_desc *sec); +void rtw_sec_write_cam(struct rtw_dev *rtwdev, + struct rtw_sec_desc *sec, + struct ieee80211_sta *sta, + struct ieee80211_key_conf *key, + u8 hw_key_type, u8 hw_key_idx); +void rtw_sec_clear_cam(struct rtw_dev *rtwdev, + struct rtw_sec_desc *sec, + u8 hw_key_idx); +void rtw_sec_enable_sec_engine(struct rtw_dev *rtwdev); + +#endif diff --git a/drivers/net/wireless/realtek/rtw88/tx.c b/drivers/net/wireless/realtek/rtw88/tx.c new file mode 100644 index 0000000000000000000000000000000000000000..e32faf8bead92cf2ae7483348423c98b3b15e556 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/tx.c @@ -0,0 +1,367 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#include "main.h" +#include "tx.h" +#include "fw.h" +#include "ps.h" + +static +void rtw_tx_stats(struct rtw_dev *rtwdev, struct ieee80211_vif *vif, + struct sk_buff *skb) +{ + struct ieee80211_hdr *hdr; + struct rtw_vif *rtwvif; + + hdr = (struct ieee80211_hdr *)skb->data; + + if (!ieee80211_is_data(hdr->frame_control)) + return; + + if (!is_broadcast_ether_addr(hdr->addr1) && + !is_multicast_ether_addr(hdr->addr1)) { + rtwdev->stats.tx_unicast += skb->len; + rtwdev->stats.tx_cnt++; + if (vif) { + rtwvif = (struct rtw_vif *)vif->drv_priv; + rtwvif->stats.tx_unicast += skb->len; + rtwvif->stats.tx_cnt++; + if (rtwvif->stats.tx_cnt > RTW_LPS_THRESHOLD) + rtw_leave_lps_irqsafe(rtwdev, rtwvif); + } + } +} + +void rtw_tx_fill_tx_desc(struct rtw_tx_pkt_info *pkt_info, struct sk_buff *skb) +{ + __le32 *txdesc = (__le32 *)skb->data; + + SET_TX_DESC_TXPKTSIZE(txdesc, pkt_info->tx_pkt_size); + SET_TX_DESC_OFFSET(txdesc, pkt_info->offset); + SET_TX_DESC_PKT_OFFSET(txdesc, pkt_info->pkt_offset); + SET_TX_DESC_QSEL(txdesc, pkt_info->qsel); + SET_TX_DESC_BMC(txdesc, pkt_info->bmc); + SET_TX_DESC_RATE_ID(txdesc, pkt_info->rate_id); + SET_TX_DESC_DATARATE(txdesc, pkt_info->rate); + SET_TX_DESC_DISDATAFB(txdesc, pkt_info->dis_rate_fallback); + SET_TX_DESC_USE_RATE(txdesc, pkt_info->use_rate); + SET_TX_DESC_SEC_TYPE(txdesc, pkt_info->sec_type); + SET_TX_DESC_DATA_BW(txdesc, pkt_info->bw); + SET_TX_DESC_SW_SEQ(txdesc, pkt_info->seq); + SET_TX_DESC_MAX_AGG_NUM(txdesc, pkt_info->ampdu_factor); + SET_TX_DESC_AMPDU_DENSITY(txdesc, pkt_info->ampdu_density); + SET_TX_DESC_DATA_STBC(txdesc, pkt_info->stbc); + SET_TX_DESC_DATA_LDPC(txdesc, pkt_info->ldpc); + SET_TX_DESC_AGG_EN(txdesc, pkt_info->ampdu_en); + SET_TX_DESC_LS(txdesc, pkt_info->ls); + SET_TX_DESC_DATA_SHORT(txdesc, pkt_info->short_gi); + SET_TX_DESC_SPE_RPT(txdesc, pkt_info->report); + SET_TX_DESC_SW_DEFINE(txdesc, pkt_info->sn); +} +EXPORT_SYMBOL(rtw_tx_fill_tx_desc); + +static u8 get_tx_ampdu_factor(struct ieee80211_sta *sta) +{ + u8 exp = sta->ht_cap.ampdu_factor; + + /* the least ampdu factor is 8K, and the value in the tx desc is the + * max aggregation num, which represents val * 2 packets can be + * aggregated in an AMPDU, so here we should use 8/2=4 as the base + */ + return (BIT(2) << exp) - 1; +} + +static u8 get_tx_ampdu_density(struct ieee80211_sta *sta) +{ + return sta->ht_cap.ampdu_density; +} + +static u8 get_highest_ht_tx_rate(struct rtw_dev *rtwdev, + struct ieee80211_sta *sta) +{ + u8 rate; + + if (rtwdev->hal.rf_type == RF_2T2R && sta->ht_cap.mcs.rx_mask[1] != 0) + rate = DESC_RATEMCS15; + else + rate = DESC_RATEMCS7; + + return rate; +} + +static u8 get_highest_vht_tx_rate(struct rtw_dev *rtwdev, + struct ieee80211_sta *sta) +{ + struct rtw_efuse *efuse = &rtwdev->efuse; + u8 rate; + u16 tx_mcs_map; + + tx_mcs_map = le16_to_cpu(sta->vht_cap.vht_mcs.tx_mcs_map); + if (efuse->hw_cap.nss == 1) { + switch (tx_mcs_map & 0x3) { + case IEEE80211_VHT_MCS_SUPPORT_0_7: + rate = DESC_RATEVHT1SS_MCS7; + break; + case IEEE80211_VHT_MCS_SUPPORT_0_8: + rate = DESC_RATEVHT1SS_MCS8; + break; + default: + case IEEE80211_VHT_MCS_SUPPORT_0_9: + rate = DESC_RATEVHT1SS_MCS9; + break; + } + } else if (efuse->hw_cap.nss >= 2) { + switch ((tx_mcs_map & 0xc) >> 2) { + case IEEE80211_VHT_MCS_SUPPORT_0_7: + rate = DESC_RATEVHT2SS_MCS7; + break; + case IEEE80211_VHT_MCS_SUPPORT_0_8: + rate = DESC_RATEVHT2SS_MCS8; + break; + default: + case IEEE80211_VHT_MCS_SUPPORT_0_9: + rate = DESC_RATEVHT2SS_MCS9; + break; + } + } else { + rate = DESC_RATEVHT1SS_MCS9; + } + + return rate; +} + +static void rtw_tx_report_enable(struct rtw_dev *rtwdev, + struct rtw_tx_pkt_info *pkt_info) +{ + struct rtw_tx_report *tx_report = &rtwdev->tx_report; + + /* [11:8], reserved, fills with zero + * [7:2], tx report sequence number + * [1:0], firmware use, fills with zero + */ + pkt_info->sn = (atomic_inc_return(&tx_report->sn) << 2) & 0xfc; + pkt_info->report = true; +} + +void rtw_tx_report_purge_timer(struct timer_list *t) +{ + struct rtw_dev *rtwdev = from_timer(rtwdev, t, tx_report.purge_timer); + struct rtw_tx_report *tx_report = &rtwdev->tx_report; + unsigned long flags; + + if (skb_queue_len(&tx_report->queue) == 0) + return; + + WARN(1, "purge skb(s) not reported by firmware\n"); + + spin_lock_irqsave(&tx_report->q_lock, flags); + skb_queue_purge(&tx_report->queue); + spin_unlock_irqrestore(&tx_report->q_lock, flags); +} + +void rtw_tx_report_enqueue(struct rtw_dev *rtwdev, struct sk_buff *skb, u8 sn) +{ + struct rtw_tx_report *tx_report = &rtwdev->tx_report; + unsigned long flags; + u8 *drv_data; + + /* pass sn to tx report handler through driver data */ + drv_data = (u8 *)IEEE80211_SKB_CB(skb)->status.status_driver_data; + *drv_data = sn; + + spin_lock_irqsave(&tx_report->q_lock, flags); + __skb_queue_tail(&tx_report->queue, skb); + spin_unlock_irqrestore(&tx_report->q_lock, flags); + + mod_timer(&tx_report->purge_timer, jiffies + RTW_TX_PROBE_TIMEOUT); +} +EXPORT_SYMBOL(rtw_tx_report_enqueue); + +static void rtw_tx_report_tx_status(struct rtw_dev *rtwdev, + struct sk_buff *skb, bool acked) +{ + struct ieee80211_tx_info *info; + + info = IEEE80211_SKB_CB(skb); + ieee80211_tx_info_clear_status(info); + if (acked) + info->flags |= IEEE80211_TX_STAT_ACK; + else + info->flags &= ~IEEE80211_TX_STAT_ACK; + + ieee80211_tx_status_irqsafe(rtwdev->hw, skb); +} + +void rtw_tx_report_handle(struct rtw_dev *rtwdev, struct sk_buff *skb) +{ + struct rtw_tx_report *tx_report = &rtwdev->tx_report; + struct rtw_c2h_cmd *c2h; + struct sk_buff *cur, *tmp; + unsigned long flags; + u8 sn, st; + u8 *n; + + c2h = get_c2h_from_skb(skb); + + sn = GET_CCX_REPORT_SEQNUM(c2h->payload); + st = GET_CCX_REPORT_STATUS(c2h->payload); + + spin_lock_irqsave(&tx_report->q_lock, flags); + skb_queue_walk_safe(&tx_report->queue, cur, tmp) { + n = (u8 *)IEEE80211_SKB_CB(cur)->status.status_driver_data; + if (*n == sn) { + __skb_unlink(cur, &tx_report->queue); + rtw_tx_report_tx_status(rtwdev, cur, st == 0); + break; + } + } + spin_unlock_irqrestore(&tx_report->q_lock, flags); +} + +static void rtw_tx_mgmt_pkt_info_update(struct rtw_dev *rtwdev, + struct rtw_tx_pkt_info *pkt_info, + struct ieee80211_tx_control *control, + struct sk_buff *skb) +{ + pkt_info->use_rate = true; + pkt_info->rate_id = 6; + pkt_info->dis_rate_fallback = true; +} + +static void rtw_tx_data_pkt_info_update(struct rtw_dev *rtwdev, + struct rtw_tx_pkt_info *pkt_info, + struct ieee80211_tx_control *control, + struct sk_buff *skb) +{ + struct ieee80211_sta *sta = control->sta; + struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)skb->data; + struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb); + struct rtw_sta_info *si; + u16 seq; + u8 ampdu_factor = 0; + u8 ampdu_density = 0; + bool ampdu_en = false; + u8 rate = DESC_RATE6M; + u8 rate_id = 6; + u8 bw = RTW_CHANNEL_WIDTH_20; + bool stbc = false; + bool ldpc = false; + + seq = (le16_to_cpu(hdr->seq_ctrl) & IEEE80211_SCTL_SEQ) >> 4; + + /* for broadcast/multicast, use default values */ + if (!sta) + goto out; + + if (info->flags & IEEE80211_TX_CTL_AMPDU) { + ampdu_en = true; + ampdu_factor = get_tx_ampdu_factor(sta); + ampdu_density = get_tx_ampdu_density(sta); + } + + if (sta->vht_cap.vht_supported) + rate = get_highest_vht_tx_rate(rtwdev, sta); + else if (sta->ht_cap.ht_supported) + rate = get_highest_ht_tx_rate(rtwdev, sta); + else if (sta->supp_rates[0] <= 0xf) + rate = DESC_RATE11M; + else + rate = DESC_RATE54M; + + si = (struct rtw_sta_info *)sta->drv_priv; + + bw = si->bw_mode; + rate_id = si->rate_id; + stbc = si->stbc_en; + ldpc = si->ldpc_en; + +out: + pkt_info->seq = seq; + pkt_info->ampdu_factor = ampdu_factor; + pkt_info->ampdu_density = ampdu_density; + pkt_info->ampdu_en = ampdu_en; + pkt_info->rate = rate; + pkt_info->rate_id = rate_id; + pkt_info->bw = bw; + pkt_info->stbc = stbc; + pkt_info->ldpc = ldpc; +} + +void rtw_tx_pkt_info_update(struct rtw_dev *rtwdev, + struct rtw_tx_pkt_info *pkt_info, + struct ieee80211_tx_control *control, + struct sk_buff *skb) +{ + struct rtw_chip_info *chip = rtwdev->chip; + struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb); + struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)skb->data; + struct rtw_sta_info *si; + struct ieee80211_vif *vif = NULL; + __le16 fc = hdr->frame_control; + u8 sec_type = 0; + bool bmc; + + if (control->sta) { + si = (struct rtw_sta_info *)control->sta->drv_priv; + vif = si->vif; + } + + if (ieee80211_is_mgmt(fc) || ieee80211_is_nullfunc(fc)) + rtw_tx_mgmt_pkt_info_update(rtwdev, pkt_info, control, skb); + else if (ieee80211_is_data(fc)) + rtw_tx_data_pkt_info_update(rtwdev, pkt_info, control, skb); + + if (info->control.hw_key) { + struct ieee80211_key_conf *key = info->control.hw_key; + + switch (key->cipher) { + case WLAN_CIPHER_SUITE_WEP40: + case WLAN_CIPHER_SUITE_WEP104: + case WLAN_CIPHER_SUITE_TKIP: + sec_type = 0x01; + break; + case WLAN_CIPHER_SUITE_CCMP: + sec_type = 0x03; + break; + default: + break; + } + } + + bmc = is_broadcast_ether_addr(hdr->addr1) || + is_multicast_ether_addr(hdr->addr1); + + if (info->flags & IEEE80211_TX_CTL_REQ_TX_STATUS) + rtw_tx_report_enable(rtwdev, pkt_info); + + pkt_info->bmc = bmc; + pkt_info->sec_type = sec_type; + pkt_info->tx_pkt_size = skb->len; + pkt_info->offset = chip->tx_pkt_desc_sz; + pkt_info->qsel = skb->priority; + pkt_info->ls = true; + + /* maybe merge with tx status ? */ + rtw_tx_stats(rtwdev, vif, skb); +} + +void rtw_rsvd_page_pkt_info_update(struct rtw_dev *rtwdev, + struct rtw_tx_pkt_info *pkt_info, + struct sk_buff *skb) +{ + struct rtw_chip_info *chip = rtwdev->chip; + struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)skb->data; + bool bmc; + + bmc = is_broadcast_ether_addr(hdr->addr1) || + is_multicast_ether_addr(hdr->addr1); + pkt_info->use_rate = true; + pkt_info->rate_id = 6; + pkt_info->dis_rate_fallback = true; + pkt_info->bmc = bmc; + pkt_info->tx_pkt_size = skb->len; + pkt_info->offset = chip->tx_pkt_desc_sz; + pkt_info->qsel = skb->priority; + pkt_info->ls = true; +} diff --git a/drivers/net/wireless/realtek/rtw88/tx.h b/drivers/net/wireless/realtek/rtw88/tx.h new file mode 100644 index 0000000000000000000000000000000000000000..8338dbf5557679afa2132ffe639cebf5ea270600 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/tx.h @@ -0,0 +1,89 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#ifndef __RTW_TX_H_ +#define __RTW_TX_H_ + +#define RTK_TX_MAX_AGG_NUM_MASK 0x1f + +#define RTW_TX_PROBE_TIMEOUT msecs_to_jiffies(500) + +#define SET_TX_DESC_TXPKTSIZE(txdesc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x00, value, GENMASK(15, 0)) +#define SET_TX_DESC_OFFSET(txdesc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x00, value, GENMASK(23, 16)) +#define SET_TX_DESC_PKT_OFFSET(txdesc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x01, value, GENMASK(28, 24)) +#define SET_TX_DESC_QSEL(txdesc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x01, value, GENMASK(12, 8)) +#define SET_TX_DESC_BMC(txdesc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x00, value, BIT(24)) +#define SET_TX_DESC_RATE_ID(txdesc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x01, value, GENMASK(20, 16)) +#define SET_TX_DESC_DATARATE(txdesc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x04, value, GENMASK(6, 0)) +#define SET_TX_DESC_DISDATAFB(txdesc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x03, value, BIT(10)) +#define SET_TX_DESC_USE_RATE(txdesc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x03, value, BIT(8)) +#define SET_TX_DESC_SEC_TYPE(txdesc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x01, value, GENMASK(23, 22)) +#define SET_TX_DESC_DATA_BW(txdesc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x05, value, GENMASK(6, 5)) +#define SET_TX_DESC_SW_SEQ(txdesc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x09, value, GENMASK(23, 12)) +#define SET_TX_DESC_MAX_AGG_NUM(txdesc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x03, value, GENMASK(21, 17)) +#define SET_TX_DESC_AMPDU_DENSITY(txdesc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x02, value, GENMASK(22, 20)) +#define SET_TX_DESC_DATA_STBC(txdesc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x05, value, GENMASK(9, 8)) +#define SET_TX_DESC_DATA_LDPC(txdesc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x05, value, BIT(7)) +#define SET_TX_DESC_AGG_EN(txdesc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x02, value, BIT(12)) +#define SET_TX_DESC_LS(txdesc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x00, value, BIT(26)) +#define SET_TX_DESC_DATA_SHORT(txdesc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x05, value, BIT(4)) +#define SET_TX_DESC_SPE_RPT(tx_desc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x02, value, BIT(19)) +#define SET_TX_DESC_SW_DEFINE(tx_desc, value) \ + le32p_replace_bits((__le32 *)(txdesc) + 0x06, value, GENMASK(11, 0)) + +enum rtw_tx_desc_queue_select { + TX_DESC_QSEL_TID0 = 0, + TX_DESC_QSEL_TID1 = 1, + TX_DESC_QSEL_TID2 = 2, + TX_DESC_QSEL_TID3 = 3, + TX_DESC_QSEL_TID4 = 4, + TX_DESC_QSEL_TID5 = 5, + TX_DESC_QSEL_TID6 = 6, + TX_DESC_QSEL_TID7 = 7, + TX_DESC_QSEL_TID8 = 8, + TX_DESC_QSEL_TID9 = 9, + TX_DESC_QSEL_TID10 = 10, + TX_DESC_QSEL_TID11 = 11, + TX_DESC_QSEL_TID12 = 12, + TX_DESC_QSEL_TID13 = 13, + TX_DESC_QSEL_TID14 = 14, + TX_DESC_QSEL_TID15 = 15, + TX_DESC_QSEL_BEACON = 16, + TX_DESC_QSEL_HIGH = 17, + TX_DESC_QSEL_MGMT = 18, + TX_DESC_QSEL_H2C = 19, +}; + +void rtw_tx_pkt_info_update(struct rtw_dev *rtwdev, + struct rtw_tx_pkt_info *pkt_info, + struct ieee80211_tx_control *control, + struct sk_buff *skb); +void rtw_tx_fill_tx_desc(struct rtw_tx_pkt_info *pkt_info, struct sk_buff *skb); +void rtw_tx_report_enqueue(struct rtw_dev *rtwdev, struct sk_buff *skb, u8 sn); +void rtw_tx_report_handle(struct rtw_dev *rtwdev, struct sk_buff *skb); +void rtw_rsvd_page_pkt_info_update(struct rtw_dev *rtwdev, + struct rtw_tx_pkt_info *pkt_info, + struct sk_buff *skb); + +#endif diff --git a/drivers/net/wireless/realtek/rtw88/util.c b/drivers/net/wireless/realtek/rtw88/util.c new file mode 100644 index 0000000000000000000000000000000000000000..212070c2baa89ca8c7ef3b8a949a2357d72a9ba7 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/util.c @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#include "main.h" +#include "util.h" +#include "reg.h" + +bool check_hw_ready(struct rtw_dev *rtwdev, u32 addr, u32 mask, u32 target) +{ + u32 cnt; + + for (cnt = 0; cnt < 1000; cnt++) { + if (rtw_read32_mask(rtwdev, addr, mask) == target) + return true; + + udelay(10); + } + + return false; +} + +bool ltecoex_read_reg(struct rtw_dev *rtwdev, u16 offset, u32 *val) +{ + if (!check_hw_ready(rtwdev, LTECOEX_ACCESS_CTRL, LTECOEX_READY, 1)) + return false; + + rtw_write32(rtwdev, LTECOEX_ACCESS_CTRL, 0x800F0000 | offset); + *val = rtw_read32(rtwdev, LTECOEX_READ_DATA); + + return true; +} + +bool ltecoex_reg_write(struct rtw_dev *rtwdev, u16 offset, u32 value) +{ + if (!check_hw_ready(rtwdev, LTECOEX_ACCESS_CTRL, LTECOEX_READY, 1)) + return false; + + rtw_write32(rtwdev, LTECOEX_WRITE_DATA, value); + rtw_write32(rtwdev, LTECOEX_ACCESS_CTRL, 0xC00F0000 | offset); + + return true; +} + +void rtw_restore_reg(struct rtw_dev *rtwdev, + struct rtw_backup_info *bckp, u32 num) +{ + u8 len; + u32 reg; + u32 val; + int i; + + for (i = 0; i < num; i++, bckp++) { + len = bckp->len; + reg = bckp->reg; + val = bckp->val; + + switch (len) { + case 1: + rtw_write8(rtwdev, reg, (u8)val); + break; + case 2: + rtw_write16(rtwdev, reg, (u16)val); + break; + case 4: + rtw_write32(rtwdev, reg, (u32)val); + break; + default: + break; + } + } +} diff --git a/drivers/net/wireless/realtek/rtw88/util.h b/drivers/net/wireless/realtek/rtw88/util.h new file mode 100644 index 0000000000000000000000000000000000000000..7bd2843b0bce32a69aa31f9cfcc4db0af10b1051 --- /dev/null +++ b/drivers/net/wireless/realtek/rtw88/util.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* Copyright(c) 2018-2019 Realtek Corporation + */ + +#ifndef __RTW_UTIL_H__ +#define __RTW_UTIL_H__ + +struct rtw_dev; + +#define rtw_iterate_vifs(rtwdev, iterator, data) \ + ieee80211_iterate_active_interfaces(rtwdev->hw, \ + IEEE80211_IFACE_ITER_NORMAL, iterator, data) +#define rtw_iterate_vifs_atomic(rtwdev, iterator, data) \ + ieee80211_iterate_active_interfaces_atomic(rtwdev->hw, \ + IEEE80211_IFACE_ITER_NORMAL, iterator, data) +#define rtw_iterate_stas_atomic(rtwdev, iterator, data) \ + ieee80211_iterate_stations_atomic(rtwdev->hw, iterator, data) + +static inline u8 *get_hdr_bssid(struct ieee80211_hdr *hdr) +{ + __le16 fc = hdr->frame_control; + u8 *bssid; + + if (ieee80211_has_tods(fc)) + bssid = hdr->addr1; + else if (ieee80211_has_fromds(fc)) + bssid = hdr->addr2; + else + bssid = hdr->addr3; + + return bssid; +} + +#endif diff --git a/drivers/net/wireless/rndis_wlan.c b/drivers/net/wireless/rndis_wlan.c index 51e4e92d95a0d314c600771f06cb938e6731091b..e07a1152cec1f5f4e7e1a8d167063757d854fcc7 100644 --- a/drivers/net/wireless/rndis_wlan.c +++ b/drivers/net/wireless/rndis_wlan.c @@ -1707,7 +1707,7 @@ static struct ndis_80211_pmkid *get_device_pmkids(struct usbnet *usbdev) int len, ret, max_pmkids; max_pmkids = priv->wdev.wiphy->max_num_pmkids; - len = sizeof(*pmkids) + max_pmkids * sizeof(pmkids->bssid_info[0]); + len = struct_size(pmkids, bssid_info, max_pmkids); pmkids = kzalloc(len, GFP_KERNEL); if (!pmkids) @@ -1740,7 +1740,7 @@ static int set_device_pmkids(struct usbnet *usbdev, int ret, len, num_pmkids; num_pmkids = le32_to_cpu(pmkids->bssid_info_count); - len = sizeof(*pmkids) + num_pmkids * sizeof(pmkids->bssid_info[0]); + len = struct_size(pmkids, bssid_info, num_pmkids); pmkids->length = cpu_to_le32(len); debug_print_pmkids(usbdev, pmkids, __func__); @@ -1761,7 +1761,7 @@ static struct ndis_80211_pmkid *remove_pmkid(struct usbnet *usbdev, struct cfg80211_pmksa *pmksa, int max_pmkids) { - int i, newlen, err; + int i, err; unsigned int count; count = le32_to_cpu(pmkids->bssid_info_count); @@ -1786,9 +1786,7 @@ static struct ndis_80211_pmkid *remove_pmkid(struct usbnet *usbdev, pmkids->bssid_info[i] = pmkids->bssid_info[i + 1]; count--; - newlen = sizeof(*pmkids) + count * sizeof(pmkids->bssid_info[0]); - - pmkids->length = cpu_to_le32(newlen); + pmkids->length = cpu_to_le32(struct_size(pmkids, bssid_info, count)); pmkids->bssid_info_count = cpu_to_le32(count); return pmkids; @@ -1831,7 +1829,7 @@ static struct ndis_80211_pmkid *update_pmkid(struct usbnet *usbdev, } /* add new pmkid */ - newlen = sizeof(*pmkids) + (count + 1) * sizeof(pmkids->bssid_info[0]); + newlen = struct_size(pmkids, bssid_info, count + 1); new_pmkids = krealloc(pmkids, newlen, GFP_KERNEL); if (!new_pmkids) { diff --git a/drivers/net/wireless/rsi/rsi_91x_hal.c b/drivers/net/wireless/rsi/rsi_91x_hal.c index 1dbaab2a96b748a2104224bca61051d6dd7db195..f84250bdb8cf85196bc517eb30c3ddfe263eefa0 100644 --- a/drivers/net/wireless/rsi/rsi_91x_hal.c +++ b/drivers/net/wireless/rsi/rsi_91x_hal.c @@ -31,6 +31,13 @@ static struct ta_metadata metadata_flash_content[] = { }; +static struct ta_metadata metadata[] = {{"pmemdata_dummy", 0x00000000}, + {"rsi/rs9116_wlan.rps", 0x00000000}, + {"rsi/rs9116_wlan_bt_classic.rps", 0x00000000}, + {"rsi/pmemdata_dummy", 0x00000000}, + {"rsi/rs9116_wlan_bt_classic.rps", 0x00000000} +}; + int rsi_send_pkt_to_bus(struct rsi_common *common, struct sk_buff *skb) { struct rsi_hw *adapter = common->priv; @@ -829,21 +836,18 @@ static int auto_fw_upgrade(struct rsi_hw *adapter, u8 *flash_content, return 0; } -static int rsi_load_firmware(struct rsi_hw *adapter) +static int rsi_hal_prepare_fwload(struct rsi_hw *adapter) { - struct rsi_common *common = adapter->priv; struct rsi_host_intf_ops *hif_ops = adapter->host_intf_ops; - const struct firmware *fw_entry = NULL; - u32 regout_val = 0, content_size; - u16 tmp_regout_val = 0; - struct ta_metadata *metadata_p; + u32 regout_val = 0; int status; bl_start_cmd_timer(adapter, BL_CMD_TIMEOUT); while (!adapter->blcmd_timer_expired) { status = hif_ops->master_reg_read(adapter, SWBL_REGOUT, - ®out_val, 2); + ®out_val, + RSI_COMMON_REG_SIZE); if (status < 0) { rsi_dbg(ERR_ZONE, "%s: REGOUT read failed\n", __func__); @@ -865,13 +869,26 @@ static int rsi_load_firmware(struct rsi_hw *adapter) (regout_val & 0xff)); status = hif_ops->master_reg_write(adapter, SWBL_REGOUT, - (REGOUT_INVALID | REGOUT_INVALID << 8), - 2); - if (status < 0) { + (REGOUT_INVALID | + REGOUT_INVALID << 8), + RSI_COMMON_REG_SIZE); + if (status < 0) rsi_dbg(ERR_ZONE, "%s: REGOUT writing failed..\n", __func__); - return status; - } - mdelay(1); + else + rsi_dbg(INFO_ZONE, + "===> Device is ready to load firmware <===\n"); + + return status; +} + +static int rsi_load_9113_firmware(struct rsi_hw *adapter) +{ + struct rsi_common *common = adapter->priv; + const struct firmware *fw_entry = NULL; + u32 content_size; + u16 tmp_regout_val = 0; + struct ta_metadata *metadata_p; + int status; status = bl_cmd(adapter, CONFIG_AUTO_READ_MODE, CMD_PASS, "AUTO_READ_CMD"); @@ -902,13 +919,15 @@ static int rsi_load_firmware(struct rsi_hw *adapter) /* Get the firmware version */ common->lmac_ver.ver.info.fw_ver[0] = - fw_entry->data[LMAC_VER_OFFSET] & 0xFF; + fw_entry->data[LMAC_VER_OFFSET_9113] & 0xFF; common->lmac_ver.ver.info.fw_ver[1] = - fw_entry->data[LMAC_VER_OFFSET + 1] & 0xFF; - common->lmac_ver.major = fw_entry->data[LMAC_VER_OFFSET + 2] & 0xFF; + fw_entry->data[LMAC_VER_OFFSET_9113 + 1] & 0xFF; + common->lmac_ver.major = + fw_entry->data[LMAC_VER_OFFSET_9113 + 2] & 0xFF; common->lmac_ver.release_num = - fw_entry->data[LMAC_VER_OFFSET + 3] & 0xFF; - common->lmac_ver.minor = fw_entry->data[LMAC_VER_OFFSET + 4] & 0xFF; + fw_entry->data[LMAC_VER_OFFSET_9113 + 3] & 0xFF; + common->lmac_ver.minor = + fw_entry->data[LMAC_VER_OFFSET_9113 + 4] & 0xFF; common->lmac_ver.patch_num = 0; rsi_print_version(common); @@ -977,19 +996,161 @@ fail: return status; } +static int rsi_load_9116_firmware(struct rsi_hw *adapter) +{ + struct rsi_common *common = adapter->priv; + struct rsi_host_intf_ops *hif_ops = adapter->host_intf_ops; + const struct firmware *fw_entry; + struct ta_metadata *metadata_p; + u8 *ta_firmware, *fw_p; + struct bootload_ds bootload_ds; + u32 instructions_sz, base_address; + u16 block_size = adapter->block_size; + u32 dest, len; + int status, cnt; + + rsi_dbg(INIT_ZONE, "***** Load 9116 TA Instructions *****\n"); + + if (adapter->rsi_host_intf == RSI_HOST_INTF_USB) { + status = bl_cmd(adapter, POLLING_MODE, CMD_PASS, + "POLLING_MODE"); + if (status < 0) + return status; + } + + status = hif_ops->master_reg_write(adapter, MEM_ACCESS_CTRL_FROM_HOST, + RAM_384K_ACCESS_FROM_TA, + RSI_9116_REG_SIZE); + if (status < 0) { + rsi_dbg(ERR_ZONE, "%s: Unable to access full RAM memory\n", + __func__); + return status; + } + + metadata_p = &metadata[adapter->priv->coex_mode]; + rsi_dbg(INIT_ZONE, "%s: loading file %s\n", __func__, metadata_p->name); + status = request_firmware(&fw_entry, metadata_p->name, adapter->device); + if (status < 0) { + rsi_dbg(ERR_ZONE, "%s: Failed to open file %s\n", + __func__, metadata_p->name); + return status; + } + + ta_firmware = kmemdup(fw_entry->data, fw_entry->size, GFP_KERNEL); + if (!ta_firmware) + goto fail_release_fw; + fw_p = ta_firmware; + instructions_sz = fw_entry->size; + rsi_dbg(INFO_ZONE, "FW Length = %d bytes\n", instructions_sz); + + common->lmac_ver.major = ta_firmware[LMAC_VER_OFFSET_9116]; + common->lmac_ver.minor = ta_firmware[LMAC_VER_OFFSET_9116 + 1]; + common->lmac_ver.release_num = ta_firmware[LMAC_VER_OFFSET_9116 + 2]; + common->lmac_ver.patch_num = ta_firmware[LMAC_VER_OFFSET_9116 + 3]; + common->lmac_ver.ver.info.fw_ver[0] = + ta_firmware[LMAC_VER_OFFSET_9116 + 4]; + + if (instructions_sz % FW_ALIGN_SIZE) + instructions_sz += + (FW_ALIGN_SIZE - (instructions_sz % FW_ALIGN_SIZE)); + rsi_dbg(INFO_ZONE, "instructions_sz : %d\n", instructions_sz); + + if (*(u16 *)fw_p == RSI_9116_FW_MAGIC_WORD) { + memcpy(&bootload_ds, fw_p, sizeof(struct bootload_ds)); + fw_p += le16_to_cpu(bootload_ds.offset); + rsi_dbg(INFO_ZONE, "FW start = %x\n", *(u32 *)fw_p); + + cnt = 0; + do { + rsi_dbg(ERR_ZONE, "%s: Loading chunk %d\n", + __func__, cnt); + + dest = le32_to_cpu(bootload_ds.bl_entry[cnt].dst_addr); + len = le32_to_cpu(bootload_ds.bl_entry[cnt].control) & + RSI_BL_CTRL_LEN_MASK; + rsi_dbg(INFO_ZONE, "length %d destination %x\n", + len, dest); + + status = hif_ops->load_data_master_write(adapter, dest, + len, + block_size, + fw_p); + if (status < 0) { + rsi_dbg(ERR_ZONE, + "Failed to load chunk %d\n", cnt); + break; + } + fw_p += len; + if (le32_to_cpu(bootload_ds.bl_entry[cnt].control) & + RSI_BL_CTRL_LAST_ENTRY) + break; + cnt++; + } while (1); + } else { + base_address = metadata_p->address; + status = hif_ops->load_data_master_write(adapter, + base_address, + instructions_sz, + block_size, + ta_firmware); + } + if (status) { + rsi_dbg(ERR_ZONE, + "%s: Unable to load %s blk\n", + __func__, metadata_p->name); + goto fail_free_fw; + } + + rsi_dbg(INIT_ZONE, "%s: Successfully loaded %s instructions\n", + __func__, metadata_p->name); + + if (adapter->rsi_host_intf == RSI_HOST_INTF_SDIO) { + if (hif_ops->ta_reset(adapter)) + rsi_dbg(ERR_ZONE, "Unable to put ta in reset\n"); + } else { + if (bl_cmd(adapter, JUMP_TO_ZERO_PC, + CMD_PASS, "JUMP_TO_ZERO") < 0) + rsi_dbg(INFO_ZONE, "Jump to zero command failed\n"); + else + rsi_dbg(INFO_ZONE, "Jump to zero command successful\n"); + } + +fail_free_fw: + kfree(ta_firmware); +fail_release_fw: + release_firmware(fw_entry); + + return status; +} + int rsi_hal_device_init(struct rsi_hw *adapter) { struct rsi_common *common = adapter->priv; + int status; switch (adapter->device_model) { case RSI_DEV_9113: - if (rsi_load_firmware(adapter)) { + status = rsi_hal_prepare_fwload(adapter); + if (status < 0) + return status; + if (rsi_load_9113_firmware(adapter)) { rsi_dbg(ERR_ZONE, "%s: Failed to load TA instructions\n", __func__); return -EINVAL; } break; + case RSI_DEV_9116: + status = rsi_hal_prepare_fwload(adapter); + if (status < 0) + return status; + if (rsi_load_9116_firmware(adapter)) { + rsi_dbg(ERR_ZONE, + "%s: Failed to load firmware to 9116 device\n", + __func__); + return -EINVAL; + } + break; default: return -EINVAL; } diff --git a/drivers/net/wireless/rsi/rsi_91x_mac80211.c b/drivers/net/wireless/rsi/rsi_91x_mac80211.c index 831046e760f8a1efb7dd61ac9d5cd9f293022708..49df3bb08d41fbfd58bc6d2b2869ace910a191cf 100644 --- a/drivers/net/wireless/rsi/rsi_91x_mac80211.c +++ b/drivers/net/wireless/rsi/rsi_91x_mac80211.c @@ -188,27 +188,27 @@ bool rsi_is_cipher_wep(struct rsi_common *common) * @adapter: Pointer to the adapter structure. * @band: Operating band to be set. * - * Return: None. + * Return: int - 0 on success, negative error on failure. */ -static void rsi_register_rates_channels(struct rsi_hw *adapter, int band) +static int rsi_register_rates_channels(struct rsi_hw *adapter, int band) { struct ieee80211_supported_band *sbands = &adapter->sbands[band]; void *channels = NULL; if (band == NL80211_BAND_2GHZ) { - channels = kmalloc(sizeof(rsi_2ghz_channels), GFP_KERNEL); - memcpy(channels, - rsi_2ghz_channels, - sizeof(rsi_2ghz_channels)); + channels = kmemdup(rsi_2ghz_channels, sizeof(rsi_2ghz_channels), + GFP_KERNEL); + if (!channels) + return -ENOMEM; sbands->band = NL80211_BAND_2GHZ; sbands->n_channels = ARRAY_SIZE(rsi_2ghz_channels); sbands->bitrates = rsi_rates; sbands->n_bitrates = ARRAY_SIZE(rsi_rates); } else { - channels = kmalloc(sizeof(rsi_5ghz_channels), GFP_KERNEL); - memcpy(channels, - rsi_5ghz_channels, - sizeof(rsi_5ghz_channels)); + channels = kmemdup(rsi_5ghz_channels, sizeof(rsi_5ghz_channels), + GFP_KERNEL); + if (!channels) + return -ENOMEM; sbands->band = NL80211_BAND_5GHZ; sbands->n_channels = ARRAY_SIZE(rsi_5ghz_channels); sbands->bitrates = &rsi_rates[4]; @@ -227,6 +227,7 @@ static void rsi_register_rates_channels(struct rsi_hw *adapter, int band) sbands->ht_cap.mcs.rx_mask[0] = 0xff; sbands->ht_cap.mcs.tx_params = IEEE80211_HT_MCS_TX_DEFINED; /* sbands->ht_cap.mcs.rx_highest = 0x82; */ + return 0; } static int rsi_mac80211_hw_scan_start(struct ieee80211_hw *hw, @@ -2064,11 +2065,16 @@ int rsi_mac80211_attach(struct rsi_common *common) wiphy->available_antennas_rx = 1; wiphy->available_antennas_tx = 1; - rsi_register_rates_channels(adapter, NL80211_BAND_2GHZ); + status = rsi_register_rates_channels(adapter, NL80211_BAND_2GHZ); + if (status) + return status; wiphy->bands[NL80211_BAND_2GHZ] = &adapter->sbands[NL80211_BAND_2GHZ]; if (common->num_supp_bands > 1) { - rsi_register_rates_channels(adapter, NL80211_BAND_5GHZ); + status = rsi_register_rates_channels(adapter, + NL80211_BAND_5GHZ); + if (status) + return status; wiphy->bands[NL80211_BAND_5GHZ] = &adapter->sbands[NL80211_BAND_5GHZ]; } diff --git a/drivers/net/wireless/rsi/rsi_91x_mgmt.c b/drivers/net/wireless/rsi/rsi_91x_mgmt.c index 844f2fac298f8bf12698e364c35b5a888ac16f0a..6c7f26ef6476aecf7d168538b4ff4c42cb4f73fe 100644 --- a/drivers/net/wireless/rsi/rsi_91x_mgmt.c +++ b/drivers/net/wireless/rsi/rsi_91x_mgmt.c @@ -209,6 +209,59 @@ static struct bootup_params boot_params_40 = { .beacon_resedue_alg_en = 0, }; +static struct bootup_params_9116 boot_params_9116_20 = { + .magic_number = cpu_to_le16(LOADED_TOKEN), + .valid = cpu_to_le32(VALID_20), + .device_clk_info_9116 = {{ + .pll_config_9116_g = { + .pll_ctrl_set_reg = cpu_to_le16(0xd518), + .pll_ctrl_clr_reg = cpu_to_le16(0x2ae7), + .pll_modem_conig_reg = cpu_to_le16(0x2000), + .soc_clk_config_reg = cpu_to_le16(0x0c18), + .adc_dac_strm1_config_reg = cpu_to_le16(0x1100), + .adc_dac_strm2_config_reg = cpu_to_le16(0x6600), + }, + .switch_clk_9116_g = { + .switch_clk_info = + cpu_to_le32((RSI_SWITCH_TASS_CLK | + RSI_SWITCH_WLAN_BBP_LMAC_CLK_REG | + RSI_SWITCH_BBP_LMAC_CLK_REG)), + .tass_clock_reg = cpu_to_le32(0x083C0503), + .wlan_bbp_lmac_clk_reg_val = cpu_to_le32(0x01042001), + .zbbt_bbp_lmac_clk_reg_val = cpu_to_le32(0x02010001), + .bbp_lmac_clk_en_val = cpu_to_le32(0x0000003b), + } + }, + }, +}; + +static struct bootup_params_9116 boot_params_9116_40 = { + .magic_number = cpu_to_le16(LOADED_TOKEN), + .valid = cpu_to_le32(VALID_40), + .device_clk_info_9116 = {{ + .pll_config_9116_g = { + .pll_ctrl_set_reg = cpu_to_le16(0xd518), + .pll_ctrl_clr_reg = cpu_to_le16(0x2ae7), + .pll_modem_conig_reg = cpu_to_le16(0x3000), + .soc_clk_config_reg = cpu_to_le16(0x0c18), + .adc_dac_strm1_config_reg = cpu_to_le16(0x0000), + .adc_dac_strm2_config_reg = cpu_to_le16(0x6600), + }, + .switch_clk_9116_g = { + .switch_clk_info = + cpu_to_le32((RSI_SWITCH_TASS_CLK | + RSI_SWITCH_WLAN_BBP_LMAC_CLK_REG | + RSI_SWITCH_BBP_LMAC_CLK_REG | + RSI_MODEM_CLK_160MHZ)), + .tass_clock_reg = cpu_to_le32(0x083C0503), + .wlan_bbp_lmac_clk_reg_val = cpu_to_le32(0x01042002), + .zbbt_bbp_lmac_clk_reg_val = cpu_to_le32(0x04010002), + .bbp_lmac_clk_en_val = cpu_to_le32(0x0000003b), + } + }, + }, +}; + static u16 mcs[] = {13, 26, 39, 52, 78, 104, 117, 130}; /** @@ -235,6 +288,14 @@ static void rsi_set_default_parameters(struct rsi_common *common) common->obm_ant_sel_val = 2; common->beacon_interval = RSI_BEACON_INTERVAL; common->dtim_cnt = RSI_DTIM_COUNT; + common->w9116_features.pll_mode = 0x0; + common->w9116_features.rf_type = 1; + common->w9116_features.wireless_mode = 0; + common->w9116_features.enable_ppe = 0; + common->w9116_features.afe_type = 1; + common->w9116_features.dpd = 0; + common->w9116_features.sifs_tx_enable = 0; + common->w9116_features.ps_options = 0; } void init_bgscan_params(struct rsi_common *common) @@ -363,6 +424,10 @@ static int rsi_load_radio_caps(struct rsi_common *common) } radio_caps->radio_info |= radio_id; + if (adapter->device_model == RSI_DEV_9116 && + common->channel_width == BW_20MHZ) + radio_caps->radio_cfg_info &= ~0x3; + radio_caps->sifs_tx_11n = cpu_to_le16(SIFS_TX_11N_VALUE); radio_caps->sifs_tx_11b = cpu_to_le16(SIFS_TX_11B_VALUE); radio_caps->slot_rx_11n = cpu_to_le16(SHORT_SLOT_VALUE); @@ -378,14 +443,16 @@ static int rsi_load_radio_caps(struct rsi_common *common) } for (ii = 0; ii < NUM_EDCA_QUEUES; ii++) { - radio_caps->qos_params[ii].cont_win_min_q = - cpu_to_le16(common->edca_params[ii].cw_min); - radio_caps->qos_params[ii].cont_win_max_q = - cpu_to_le16(common->edca_params[ii].cw_max); - radio_caps->qos_params[ii].aifsn_val_q = - cpu_to_le16((common->edca_params[ii].aifs) << 8); - radio_caps->qos_params[ii].txop_q = - cpu_to_le16(common->edca_params[ii].txop); + if (common->edca_params[ii].cw_max > 0) { + radio_caps->qos_params[ii].cont_win_min_q = + cpu_to_le16(common->edca_params[ii].cw_min); + radio_caps->qos_params[ii].cont_win_max_q = + cpu_to_le16(common->edca_params[ii].cw_max); + radio_caps->qos_params[ii].aifsn_val_q = + cpu_to_le16(common->edca_params[ii].aifs << 8); + radio_caps->qos_params[ii].txop_q = + cpu_to_le16(common->edca_params[ii].txop); + } } radio_caps->qos_params[BROADCAST_HW_Q].txop_q = cpu_to_le16(0xffff); @@ -893,6 +960,50 @@ static int rsi_load_bootup_params(struct rsi_common *common) return rsi_send_internal_mgmt_frame(common, skb); } +static int rsi_load_9116_bootup_params(struct rsi_common *common) +{ + struct sk_buff *skb; + struct rsi_boot_params_9116 *boot_params; + + rsi_dbg(MGMT_TX_ZONE, "%s: Sending boot params frame\n", __func__); + + skb = dev_alloc_skb(sizeof(struct rsi_boot_params_9116)); + if (!skb) + return -ENOMEM; + memset(skb->data, 0, sizeof(struct rsi_boot_params)); + boot_params = (struct rsi_boot_params_9116 *)skb->data; + + if (common->channel_width == BW_40MHZ) { + memcpy(&boot_params->bootup_params, + &boot_params_9116_40, + sizeof(struct bootup_params_9116)); + rsi_dbg(MGMT_TX_ZONE, "%s: Packet 40MHZ <=== %d\n", __func__, + UMAC_CLK_40BW); + boot_params->umac_clk = cpu_to_le16(UMAC_CLK_40BW); + } else { + memcpy(&boot_params->bootup_params, + &boot_params_9116_20, + sizeof(struct bootup_params_9116)); + if (boot_params_20.valid != cpu_to_le32(VALID_20)) { + boot_params->umac_clk = cpu_to_le16(UMAC_CLK_20BW); + rsi_dbg(MGMT_TX_ZONE, + "%s: Packet 20MHZ <=== %d\n", __func__, + UMAC_CLK_20BW); + } else { + boot_params->umac_clk = cpu_to_le16(UMAC_CLK_40MHZ); + rsi_dbg(MGMT_TX_ZONE, + "%s: Packet 20MHZ <=== %d\n", __func__, + UMAC_CLK_40MHZ); + } + } + rsi_set_len_qno(&boot_params->desc_dword0.len_qno, + sizeof(struct bootup_params_9116), RSI_WIFI_MGMT_Q); + boot_params->desc_dword0.frame_type = BOOTUP_PARAMS_REQUEST; + skb_put(skb, sizeof(struct rsi_boot_params_9116)); + + return rsi_send_internal_mgmt_frame(common, skb); +} + /** * rsi_send_reset_mac() - This function prepares reset MAC request and sends an * internal management frame to indicate it to firmware. @@ -921,6 +1032,11 @@ static int rsi_send_reset_mac(struct rsi_common *common) mgmt_frame->desc_word[1] = cpu_to_le16(RESET_MAC_REQ); mgmt_frame->desc_word[4] = cpu_to_le16(RETRY_COUNT << 8); +#define RSI_9116_DEF_TA_AGGR 3 + if (common->priv->device_model == RSI_DEV_9116) + mgmt_frame->desc_word[3] |= + cpu_to_le16(RSI_9116_DEF_TA_AGGR << 8); + skb_put(skb, FRAME_DESC_SZ); return rsi_send_internal_mgmt_frame(common, skb); @@ -971,7 +1087,10 @@ int rsi_band_check(struct rsi_common *common, } if (common->channel_width != prev_bw) { - status = rsi_load_bootup_params(common); + if (adapter->device_model == RSI_DEV_9116) + status = rsi_load_9116_bootup_params(common); + else + status = rsi_load_bootup_params(common); if (status) return status; @@ -1546,6 +1665,47 @@ int rsi_send_ps_request(struct rsi_hw *adapter, bool enable, return rsi_send_internal_mgmt_frame(common, skb); } +static int rsi_send_w9116_features(struct rsi_common *common) +{ + struct rsi_wlan_9116_features *w9116_features; + u16 frame_len = sizeof(struct rsi_wlan_9116_features); + struct sk_buff *skb; + + rsi_dbg(MGMT_TX_ZONE, + "%s: Sending wlan 9116 features\n", __func__); + + skb = dev_alloc_skb(frame_len); + if (!skb) + return -ENOMEM; + memset(skb->data, 0, frame_len); + + w9116_features = (struct rsi_wlan_9116_features *)skb->data; + + w9116_features->pll_mode = common->w9116_features.pll_mode; + w9116_features->rf_type = common->w9116_features.rf_type; + w9116_features->wireless_mode = common->w9116_features.wireless_mode; + w9116_features->enable_ppe = common->w9116_features.enable_ppe; + w9116_features->afe_type = common->w9116_features.afe_type; + if (common->w9116_features.dpd) + w9116_features->feature_enable |= cpu_to_le32(RSI_DPD); + if (common->w9116_features.sifs_tx_enable) + w9116_features->feature_enable |= + cpu_to_le32(RSI_SIFS_TX_ENABLE); + if (common->w9116_features.ps_options & RSI_DUTY_CYCLING) + w9116_features->feature_enable |= cpu_to_le32(RSI_DUTY_CYCLING); + if (common->w9116_features.ps_options & RSI_END_OF_FRAME) + w9116_features->feature_enable |= cpu_to_le32(RSI_END_OF_FRAME); + w9116_features->feature_enable |= + cpu_to_le32((common->w9116_features.ps_options & ~0x3) << 2); + + rsi_set_len_qno(&w9116_features->desc.desc_dword0.len_qno, + frame_len - FRAME_DESC_SZ, RSI_WIFI_MGMT_Q); + w9116_features->desc.desc_dword0.frame_type = FEATURES_ENABLE; + skb_put(skb, frame_len); + + return rsi_send_internal_mgmt_frame(common, skb); +} + /** * rsi_set_antenna() - This function send antenna configuration request * to device @@ -1766,15 +1926,26 @@ static int rsi_handle_ta_confirm_type(struct rsi_common *common, rsi_dbg(FSM_ZONE, "%s: Boot up params confirm received\n", __func__); if (common->fsm_state == FSM_BOOT_PARAMS_SENT) { - adapter->eeprom.length = (IEEE80211_ADDR_LEN + - WLAN_MAC_MAGIC_WORD_LEN + - WLAN_HOST_MODE_LEN); - adapter->eeprom.offset = WLAN_MAC_EEPROM_ADDR; - if (rsi_eeprom_read(common)) { - common->fsm_state = FSM_CARD_NOT_READY; - goto out; + if (adapter->device_model == RSI_DEV_9116) { + common->band = NL80211_BAND_5GHZ; + common->num_supp_bands = 2; + + if (rsi_send_reset_mac(common)) + goto out; + else + common->fsm_state = FSM_RESET_MAC_SENT; + } else { + adapter->eeprom.length = + (IEEE80211_ADDR_LEN + + WLAN_MAC_MAGIC_WORD_LEN + + WLAN_HOST_MODE_LEN); + adapter->eeprom.offset = WLAN_MAC_EEPROM_ADDR; + if (rsi_eeprom_read(common)) { + common->fsm_state = FSM_CARD_NOT_READY; + goto out; + } + common->fsm_state = FSM_EEPROM_READ_MAC_ADDR; } - common->fsm_state = FSM_EEPROM_READ_MAC_ADDR; } else { rsi_dbg(INFO_ZONE, "%s: Received bootup params cfm in %d state\n", @@ -1853,6 +2024,12 @@ static int rsi_handle_ta_confirm_type(struct rsi_common *common, case RADIO_CAPABILITIES: if (common->fsm_state == FSM_RADIO_CAPS_SENT) { common->rf_reset = 1; + if (adapter->device_model == RSI_DEV_9116 && + rsi_send_w9116_features(common)) { + rsi_dbg(ERR_ZONE, + "Failed to send 9116 features\n"); + goto out; + } if (rsi_program_bb_rf(common)) { goto out; } else { @@ -1925,6 +2102,8 @@ out: int rsi_handle_card_ready(struct rsi_common *common, u8 *msg) { + int status; + switch (common->fsm_state) { case FSM_CARD_NOT_READY: rsi_dbg(INIT_ZONE, "Card ready indication from Common HAL\n"); @@ -1936,14 +2115,29 @@ int rsi_handle_card_ready(struct rsi_common *common, u8 *msg) case FSM_COMMON_DEV_PARAMS_SENT: rsi_dbg(INIT_ZONE, "Card ready indication from WLAN HAL\n"); + if (common->priv->device_model == RSI_DEV_9116) { + if (msg[16] != MAGIC_WORD) { + rsi_dbg(FSM_ZONE, + "%s: [EEPROM_READ] Invalid token\n", + __func__); + common->fsm_state = FSM_CARD_NOT_READY; + return -EINVAL; + } + memcpy(common->mac_addr, &msg[20], ETH_ALEN); + rsi_dbg(INIT_ZONE, "MAC Addr %pM", common->mac_addr); + } /* Get usb buffer status register address */ common->priv->usb_buffer_status_reg = *(u32 *)&msg[8]; rsi_dbg(INFO_ZONE, "USB buffer status register = %x\n", common->priv->usb_buffer_status_reg); - if (rsi_load_bootup_params(common)) { + if (common->priv->device_model == RSI_DEV_9116) + status = rsi_load_9116_bootup_params(common); + else + status = rsi_load_bootup_params(common); + if (status < 0) { common->fsm_state = FSM_CARD_NOT_READY; - return -EINVAL; + return status; } common->fsm_state = FSM_BOOT_PARAMS_SENT; break; diff --git a/drivers/net/wireless/rsi/rsi_91x_sdio.c b/drivers/net/wireless/rsi/rsi_91x_sdio.c index 3430d7a0899e6c510c0e75e7379fe0882c205f2f..f9c67ed473d1f286e4c1acc8d694ba102ebb6b38 100644 --- a/drivers/net/wireless/rsi/rsi_91x_sdio.c +++ b/drivers/net/wireless/rsi/rsi_91x_sdio.c @@ -923,6 +923,70 @@ static int rsi_sdio_reinit_device(struct rsi_hw *adapter) return 0; } +static int rsi_sdio_ta_reset(struct rsi_hw *adapter) +{ + int status; + u32 addr; + u8 *data; + + status = rsi_sdio_master_access_msword(adapter, TA_BASE_ADDR); + if (status < 0) { + rsi_dbg(ERR_ZONE, + "Unable to set ms word to common reg\n"); + return status; + } + + rsi_dbg(INIT_ZONE, "%s: Bring TA out of reset\n", __func__); + put_unaligned_le32(TA_HOLD_THREAD_VALUE, data); + addr = TA_HOLD_THREAD_REG | RSI_SD_REQUEST_MASTER; + status = rsi_sdio_write_register_multiple(adapter, addr, + (u8 *)&data, + RSI_9116_REG_SIZE); + if (status < 0) { + rsi_dbg(ERR_ZONE, "Unable to hold TA threads\n"); + return status; + } + + put_unaligned_le32(TA_SOFT_RST_CLR, data); + addr = TA_SOFT_RESET_REG | RSI_SD_REQUEST_MASTER; + status = rsi_sdio_write_register_multiple(adapter, addr, + (u8 *)&data, + RSI_9116_REG_SIZE); + if (status < 0) { + rsi_dbg(ERR_ZONE, "Unable to get TA out of reset\n"); + return status; + } + + put_unaligned_le32(TA_PC_ZERO, data); + addr = TA_TH0_PC_REG | RSI_SD_REQUEST_MASTER; + status = rsi_sdio_write_register_multiple(adapter, addr, + (u8 *)&data, + RSI_9116_REG_SIZE); + if (status < 0) { + rsi_dbg(ERR_ZONE, "Unable to Reset TA PC value\n"); + return -EINVAL; + } + + put_unaligned_le32(TA_RELEASE_THREAD_VALUE, data); + addr = TA_RELEASE_THREAD_REG | RSI_SD_REQUEST_MASTER; + status = rsi_sdio_write_register_multiple(adapter, addr, + (u8 *)&data, + RSI_9116_REG_SIZE); + if (status < 0) { + rsi_dbg(ERR_ZONE, "Unable to release TA threads\n"); + return status; + } + + status = rsi_sdio_master_access_msword(adapter, MISC_CFG_BASE_ADDR); + if (status < 0) { + rsi_dbg(ERR_ZONE, "Unable to set ms word to common reg\n"); + return status; + } + rsi_dbg(INIT_ZONE, "***** TA Reset done *****\n"); + + return 0; +} + static struct rsi_host_intf_ops sdio_host_intf_ops = { .write_pkt = rsi_sdio_host_intf_write_pkt, .read_pkt = rsi_sdio_host_intf_read_pkt, @@ -933,6 +997,7 @@ static struct rsi_host_intf_ops sdio_host_intf_ops = { .master_reg_write = rsi_sdio_master_reg_write, .load_data_master_write = rsi_sdio_load_data_master_write, .reinit_device = rsi_sdio_reinit_device, + .ta_reset = rsi_sdio_ta_reset, }; /** @@ -949,7 +1014,7 @@ static int rsi_probe(struct sdio_func *pfunction, { struct rsi_hw *adapter; struct rsi_91x_sdiodev *sdev; - int status; + int status = -EINVAL; rsi_dbg(INIT_ZONE, "%s: Init function called\n", __func__); @@ -968,6 +1033,20 @@ static int rsi_probe(struct sdio_func *pfunction, status = -EIO; goto fail_free_adapter; } + + if (pfunction->device == RSI_SDIO_PID_9113) { + rsi_dbg(ERR_ZONE, "%s: 9113 module detected\n", __func__); + adapter->device_model = RSI_DEV_9113; + } else if (pfunction->device == RSI_SDIO_PID_9116) { + rsi_dbg(ERR_ZONE, "%s: 9116 module detected\n", __func__); + adapter->device_model = RSI_DEV_9116; + } else { + rsi_dbg(ERR_ZONE, + "%s: Unsupported RSI device id 0x%x\n", __func__, + pfunction->device); + goto fail_free_adapter; + } + sdev = (struct rsi_91x_sdiodev *)adapter->rsi_dev; rsi_init_event(&sdev->rx_thread.event); status = rsi_create_kthread(adapter->priv, &sdev->rx_thread, @@ -1088,16 +1167,41 @@ static void rsi_reset_chip(struct rsi_hw *adapter) * and any pending dma transfers to rf spi in device to finish. */ msleep(100); - - ulp_read_write(adapter, RSI_ULP_RESET_REG, RSI_ULP_WRITE_0, 32); - ulp_read_write(adapter, RSI_WATCH_DOG_TIMER_1, RSI_ULP_WRITE_2, 32); - ulp_read_write(adapter, RSI_WATCH_DOG_TIMER_2, RSI_ULP_WRITE_0, 32); - ulp_read_write(adapter, RSI_WATCH_DOG_DELAY_TIMER_1, RSI_ULP_WRITE_50, - 32); - ulp_read_write(adapter, RSI_WATCH_DOG_DELAY_TIMER_2, RSI_ULP_WRITE_0, - 32); - ulp_read_write(adapter, RSI_WATCH_DOG_TIMER_ENABLE, - RSI_ULP_TIMER_ENABLE, 32); + if (adapter->device_model != RSI_DEV_9116) { + ulp_read_write(adapter, RSI_ULP_RESET_REG, RSI_ULP_WRITE_0, 32); + ulp_read_write(adapter, + RSI_WATCH_DOG_TIMER_1, RSI_ULP_WRITE_2, 32); + ulp_read_write(adapter, RSI_WATCH_DOG_TIMER_2, RSI_ULP_WRITE_0, + 32); + ulp_read_write(adapter, RSI_WATCH_DOG_DELAY_TIMER_1, + RSI_ULP_WRITE_50, 32); + ulp_read_write(adapter, RSI_WATCH_DOG_DELAY_TIMER_2, + RSI_ULP_WRITE_0, 32); + ulp_read_write(adapter, RSI_WATCH_DOG_TIMER_ENABLE, + RSI_ULP_TIMER_ENABLE, 32); + } else { + if ((rsi_sdio_master_reg_write(adapter, + NWP_WWD_INTERRUPT_TIMER, + NWP_WWD_INT_TIMER_CLKS, + RSI_9116_REG_SIZE)) < 0) { + rsi_dbg(ERR_ZONE, "Failed to write to intr timer\n"); + } + if ((rsi_sdio_master_reg_write(adapter, + NWP_WWD_SYSTEM_RESET_TIMER, + NWP_WWD_SYS_RESET_TIMER_CLKS, + RSI_9116_REG_SIZE)) < 0) { + rsi_dbg(ERR_ZONE, + "Failed to write to system reset timer\n"); + } + if ((rsi_sdio_master_reg_write(adapter, + NWP_WWD_MODE_AND_RSTART, + NWP_WWD_TIMER_DISABLE, + RSI_9116_REG_SIZE)) < 0) { + rsi_dbg(ERR_ZONE, + "Failed to write to mode and restart\n"); + } + rsi_dbg(ERR_ZONE, "***** Watch Dog Reset Successful *****\n"); + } /* This msleep will be sufficient for the ulp * read write operations to complete for chip reset. */ @@ -1415,7 +1519,8 @@ static const struct dev_pm_ops rsi_pm_ops = { #endif static const struct sdio_device_id rsi_dev_table[] = { - { SDIO_DEVICE(RSI_SDIO_VID_9113, RSI_SDIO_PID_9113) }, + { SDIO_DEVICE(RSI_SDIO_VENDOR_ID, RSI_SDIO_PID_9113) }, + { SDIO_DEVICE(RSI_SDIO_VENDOR_ID, RSI_SDIO_PID_9116) }, { /* Blank */}, }; diff --git a/drivers/net/wireless/rsi/rsi_91x_usb.c b/drivers/net/wireless/rsi/rsi_91x_usb.c index ac0ef5ea6ffb981f16a52c50f77dd53156ec972b..f5048d4b8cb6d3d32ba0f44f271023cc15544601 100644 --- a/drivers/net/wireless/rsi/rsi_91x_usb.c +++ b/drivers/net/wireless/rsi/rsi_91x_usb.c @@ -213,7 +213,7 @@ static int rsi_usb_reg_read(struct usb_device *usbdev, */ static int rsi_usb_reg_write(struct usb_device *usbdev, u32 reg, - u16 value, + u32 value, u16 len) { u8 *usb_reg_buf; @@ -226,17 +226,17 @@ static int rsi_usb_reg_write(struct usb_device *usbdev, if (!usb_reg_buf) return status; - usb_reg_buf[0] = (value & 0x00ff); - usb_reg_buf[1] = (value & 0xff00) >> 8; - usb_reg_buf[2] = 0x0; - usb_reg_buf[3] = 0x0; + usb_reg_buf[0] = (cpu_to_le32(value) & 0x00ff); + usb_reg_buf[1] = (cpu_to_le32(value) & 0xff00) >> 8; + usb_reg_buf[2] = (cpu_to_le32(value) & 0x00ff0000) >> 16; + usb_reg_buf[3] = (cpu_to_le32(value) & 0xff000000) >> 24; status = usb_control_msg(usbdev, usb_sndctrlpipe(usbdev, 0), USB_VENDOR_REGISTER_WRITE, RSI_USB_REQ_OUT, - ((reg & 0xffff0000) >> 16), - (reg & 0xffff), + ((cpu_to_le32(reg) & 0xffff0000) >> 16), + (cpu_to_le32(reg) & 0xffff), (void *)usb_reg_buf, len, USB_CTRL_SET_TIMEOUT); @@ -263,8 +263,10 @@ static void rsi_rx_done_handler(struct urb *urb) struct rsi_91x_usbdev *dev = (struct rsi_91x_usbdev *)rx_cb->data; int status = -EINVAL; - if (urb->status) - goto out; + if (urb->status) { + dev_kfree_skb(rx_cb->rx_skb); + return; + } if (urb->actual_length <= 0 || urb->actual_length > rx_cb->rx_skb->len) { @@ -698,26 +700,47 @@ static int rsi_reset_card(struct rsi_hw *adapter) goto fail; } - ret = usb_ulp_read_write(adapter, RSI_WATCH_DOG_TIMER_1, - RSI_ULP_WRITE_2, 32); - if (ret < 0) - goto fail; - ret = usb_ulp_read_write(adapter, RSI_WATCH_DOG_TIMER_2, - RSI_ULP_WRITE_0, 32); - if (ret < 0) - goto fail; - ret = usb_ulp_read_write(adapter, RSI_WATCH_DOG_DELAY_TIMER_1, - RSI_ULP_WRITE_50, 32); - if (ret < 0) - goto fail; - ret = usb_ulp_read_write(adapter, RSI_WATCH_DOG_DELAY_TIMER_2, - RSI_ULP_WRITE_0, 32); - if (ret < 0) - goto fail; - ret = usb_ulp_read_write(adapter, RSI_WATCH_DOG_TIMER_ENABLE, - RSI_ULP_TIMER_ENABLE, 32); - if (ret < 0) - goto fail; + if (adapter->device_model != RSI_DEV_9116) { + ret = usb_ulp_read_write(adapter, RSI_WATCH_DOG_TIMER_1, + RSI_ULP_WRITE_2, 32); + if (ret < 0) + goto fail; + ret = usb_ulp_read_write(adapter, RSI_WATCH_DOG_TIMER_2, + RSI_ULP_WRITE_0, 32); + if (ret < 0) + goto fail; + ret = usb_ulp_read_write(adapter, RSI_WATCH_DOG_DELAY_TIMER_1, + RSI_ULP_WRITE_50, 32); + if (ret < 0) + goto fail; + ret = usb_ulp_read_write(adapter, RSI_WATCH_DOG_DELAY_TIMER_2, + RSI_ULP_WRITE_0, 32); + if (ret < 0) + goto fail; + ret = usb_ulp_read_write(adapter, RSI_WATCH_DOG_TIMER_ENABLE, + RSI_ULP_TIMER_ENABLE, 32); + if (ret < 0) + goto fail; + } else { + if ((rsi_usb_master_reg_write(adapter, + NWP_WWD_INTERRUPT_TIMER, + NWP_WWD_INT_TIMER_CLKS, + RSI_9116_REG_SIZE)) < 0) { + goto fail; + } + if ((rsi_usb_master_reg_write(adapter, + NWP_WWD_SYSTEM_RESET_TIMER, + NWP_WWD_SYS_RESET_TIMER_CLKS, + RSI_9116_REG_SIZE)) < 0) { + goto fail; + } + if ((rsi_usb_master_reg_write(adapter, + NWP_WWD_MODE_AND_RSTART, + NWP_WWD_TIMER_DISABLE, + RSI_9116_REG_SIZE)) < 0) { + goto fail; + } + } rsi_dbg(INFO_ZONE, "Reset card done\n"); return ret; @@ -763,6 +786,18 @@ static int rsi_probe(struct usb_interface *pfunction, rsi_dbg(ERR_ZONE, "%s: Initialized os intf ops\n", __func__); + if (id && id->idProduct == RSI_USB_PID_9113) { + rsi_dbg(INIT_ZONE, "%s: 9113 module detected\n", __func__); + adapter->device_model = RSI_DEV_9113; + } else if (id && id->idProduct == RSI_USB_PID_9116) { + rsi_dbg(INIT_ZONE, "%s: 9116 module detected\n", __func__); + adapter->device_model = RSI_DEV_9116; + } else { + rsi_dbg(ERR_ZONE, "%s: Unsupported RSI device id 0x%x\n", + __func__, id->idProduct); + goto err1; + } + dev = (struct rsi_91x_usbdev *)adapter->rsi_dev; status = rsi_usb_reg_read(dev->usbdev, FW_STATUS_REG, &fw_status, 2); @@ -845,7 +880,8 @@ static int rsi_resume(struct usb_interface *intf) #endif static const struct usb_device_id rsi_dev_table[] = { - { USB_DEVICE(RSI_USB_VID_9113, RSI_USB_PID_9113) }, + { USB_DEVICE(RSI_USB_VENDOR_ID, RSI_USB_PID_9113) }, + { USB_DEVICE(RSI_USB_VENDOR_ID, RSI_USB_PID_9116) }, { /* Blank */}, }; diff --git a/drivers/net/wireless/rsi/rsi_boot_params.h b/drivers/net/wireless/rsi/rsi_boot_params.h index ad903b22440e6f0653149904d6c307bdb7f3cf29..c1cf19d1e376325b1f4855610c4f3205cde318d6 100644 --- a/drivers/net/wireless/rsi/rsi_boot_params.h +++ b/drivers/net/wireless/rsi/rsi_boot_params.h @@ -80,6 +80,15 @@ struct pll_config { struct afepll_info afepll_info_g; } __packed; +struct pll_config_9116 { + __le16 pll_ctrl_set_reg; + __le16 pll_ctrl_clr_reg; + __le16 pll_modem_conig_reg; + __le16 soc_clk_config_reg; + __le16 adc_dac_strm1_config_reg; + __le16 adc_dac_strm2_config_reg; +} __packed; + /* structure to store configs related to UMAC clk programming */ struct switch_clk { __le16 switch_clk_info; @@ -93,11 +102,32 @@ struct switch_clk { __le16 qspi_uart_clock_reg_config; } __packed; +#define RSI_SWITCH_TASS_CLK BIT(0) +#define RSI_SWITCH_QSPI_CLK BIT(1) +#define RSI_SWITCH_SLP_CLK_2_32 BIT(2) +#define RSI_SWITCH_WLAN_BBP_LMAC_CLK_REG BIT(3) +#define RSI_SWITCH_ZBBT_BBP_LMAC_CLK_REG BIT(4) +#define RSI_SWITCH_BBP_LMAC_CLK_REG BIT(5) +#define RSI_MODEM_CLK_160MHZ BIT(6) + +struct switch_clk_9116 { + __le32 switch_clk_info; + __le32 tass_clock_reg; + __le32 wlan_bbp_lmac_clk_reg_val; + __le32 zbbt_bbp_lmac_clk_reg_val; + __le32 bbp_lmac_clk_en_val; +} __packed; + struct device_clk_info { struct pll_config pll_config_g; struct switch_clk switch_clk_g; } __packed; +struct device_clk_info_9116 { + struct pll_config_9116 pll_config_9116_g; + struct switch_clk_9116 switch_clk_9116_g; +} __packed; + struct bootup_params { __le16 magic_number; __le16 crystal_good_time; @@ -127,4 +157,37 @@ struct bootup_params { __le32 max_threshold_to_avoid_sleep; u8 beacon_resedue_alg_en; } __packed; + +struct bootup_params_9116 { + __le16 magic_number; +#define LOADED_TOKEN 0x5AA5 /* Bootup params are installed by host + * or OTP/FLASH (Bootloader) + */ +#define ROM_TOKEN 0x55AA /* Bootup params are taken from ROM + * itself in MCU mode. + */ + __le16 crystal_good_time; + __le32 valid; + __le32 reserved_for_valids; + __le16 bootup_mode_info; +#define BT_COEXIST BIT(0) +#define BOOTUP_MODE (BIT(2) | BIT(1)) +#define CUR_DEV_MODE_9116 (bootup_params_9116.bootup_mode_info >> 1) + __le16 digital_loop_back_params; + __le16 rtls_timestamp_en; + __le16 host_spi_intr_cfg; + struct device_clk_info_9116 device_clk_info_9116[1]; + __le32 buckboost_wakeup_cnt; + __le16 pmu_wakeup_wait; + u8 shutdown_wait_time; + u8 pmu_slp_clkout_sel; + __le32 wdt_prog_value; + __le32 wdt_soc_rst_delay; + __le32 dcdc_operation_mode; + __le32 soc_reset_wait_cnt; + __le32 waiting_time_at_fresh_sleep; + __le32 max_threshold_to_avoid_sleep; + u8 beacon_resedue_alg_en; +} __packed; + #endif diff --git a/drivers/net/wireless/rsi/rsi_hal.h b/drivers/net/wireless/rsi/rsi_hal.h index 327638cdd30b1e746d4afe33309480dcaca8b7ca..46e36df9e8e3ccb4ff14aab41f7d098592a2f308 100644 --- a/drivers/net/wireless/rsi/rsi_hal.h +++ b/drivers/net/wireless/rsi/rsi_hal.h @@ -70,6 +70,21 @@ #define RSI_WATCH_DOG_DELAY_TIMER_2 0x16f #define RSI_WATCH_DOG_TIMER_ENABLE 0x170 +/* Watchdog timer addresses for 9116 */ +#define NWP_AHB_BASE_ADDR 0x41300000 +#define NWP_WWD_INTERRUPT_TIMER (NWP_AHB_BASE_ADDR + 0x300) +#define NWP_WWD_SYSTEM_RESET_TIMER (NWP_AHB_BASE_ADDR + 0x304) +#define NWP_WWD_WINDOW_TIMER (NWP_AHB_BASE_ADDR + 0x308) +#define NWP_WWD_TIMER_SETTINGS (NWP_AHB_BASE_ADDR + 0x30C) +#define NWP_WWD_MODE_AND_RSTART (NWP_AHB_BASE_ADDR + 0x310) +#define NWP_WWD_RESET_BYPASS (NWP_AHB_BASE_ADDR + 0x314) +#define NWP_FSM_INTR_MASK_REG (NWP_AHB_BASE_ADDR + 0x104) + +/* Watchdog timer values */ +#define NWP_WWD_INT_TIMER_CLKS 5 +#define NWP_WWD_SYS_RESET_TIMER_CLKS 4 +#define NWP_WWD_TIMER_DISABLE 0xAA0001 + #define RSI_ULP_WRITE_0 00 #define RSI_ULP_WRITE_2 02 #define RSI_ULP_WRITE_50 50 @@ -113,9 +128,18 @@ #define BBP_INFO_40MHZ 0x6 #define FW_FLASH_OFFSET 0x820 -#define LMAC_VER_OFFSET (FW_FLASH_OFFSET + 0x200) +#define LMAC_VER_OFFSET_9113 (FW_FLASH_OFFSET + 0x200) +#define LMAC_VER_OFFSET_9116 0x22C2 #define MAX_DWORD_ALIGN_BYTES 64 #define RSI_COMMON_REG_SIZE 2 +#define RSI_9116_REG_SIZE 4 +#define FW_ALIGN_SIZE 4 +#define RSI_9116_FW_MAGIC_WORD 0x5aa5 + +#define MEM_ACCESS_CTRL_FROM_HOST 0x41300000 +#define RAM_384K_ACCESS_FROM_TA (BIT(2) | BIT(3) | BIT(4) | BIT(5) | \ + BIT(20) | BIT(21) | BIT(22) | \ + BIT(23) | BIT(24) | BIT(25)) struct bl_header { __le32 flags; @@ -130,6 +154,24 @@ struct ta_metadata { unsigned int address; }; +#define RSI_BL_CTRL_LEN_MASK 0xFFFFFF +#define RSI_BL_CTRL_SPI_32BIT_MODE BIT(27) +#define RSI_BL_CTRL_REL_TA_SOFTRESET BIT(28) +#define RSI_BL_CTRL_START_FROM_ROM_PC BIT(29) +#define RSI_BL_CTRL_SPI_8BIT_MODE BIT(30) +#define RSI_BL_CTRL_LAST_ENTRY BIT(31) +struct bootload_entry { + __le32 control; + __le32 dst_addr; +} __packed; + +struct bootload_ds { + __le16 fixed_pattern; + __le16 offset; + __le32 reserved; + struct bootload_entry bl_entry[7]; +} __packed; + struct rsi_mgmt_desc { __le16 len_qno; u8 frame_type; diff --git a/drivers/net/wireless/rsi/rsi_main.h b/drivers/net/wireless/rsi/rsi_main.h index 35d13f35e9b002261f5dd00502b742cd539a579b..73a19e43106b1efe62eddb72143dcb2a3474bcc3 100644 --- a/drivers/net/wireless/rsi/rsi_main.h +++ b/drivers/net/wireless/rsi/rsi_main.h @@ -111,9 +111,13 @@ extern __printf(2, 3) void rsi_dbg(u32 zone, const char *fmt, ...); #define RSI_WOW_ENABLED BIT(0) #define RSI_WOW_NO_CONNECTION BIT(1) -#define RSI_DEV_9113 1 #define RSI_MAX_RX_PKTS 64 +enum rsi_dev_model { + RSI_DEV_9113 = 0, + RSI_DEV_9116 +}; + struct version_info { u16 major; u16 minor; @@ -215,6 +219,17 @@ enum rsi_dfs_regions { RSI_REGION_WORLD }; +struct rsi_9116_features { + u8 pll_mode; + u8 rf_type; + u8 wireless_mode; + u8 afe_type; + u8 enable_ppe; + u8 dpd; + u32 sifs_tx_enable; + u32 ps_options; +}; + struct rsi_common { struct rsi_hw *priv; struct vif_priv vif_info[RSI_MAX_VIFS]; @@ -310,6 +325,7 @@ struct rsi_common { struct cfg80211_scan_request *hwscan; struct rsi_bgscan_params bgscan; + struct rsi_9116_features w9116_features; u8 bgscan_en; u8 mac_ops_resumed; }; @@ -329,7 +345,7 @@ struct eeprom_read { struct rsi_hw { struct rsi_common *priv; - u8 device_model; + enum rsi_dev_model device_model; struct ieee80211_hw *hw; struct ieee80211_vif *vifs[RSI_MAX_VIFS]; struct ieee80211_tx_queue_params edca_params[NUM_EDCA_QUEUES]; @@ -381,6 +397,7 @@ struct rsi_host_intf_ops { u32 instructions_size, u16 block_size, u8 *fw); int (*reinit_device)(struct rsi_hw *adapter); + int (*ta_reset)(struct rsi_hw *adapter); }; enum rsi_host_intf rsi_get_host_intf(void *priv); diff --git a/drivers/net/wireless/rsi/rsi_mgmt.h b/drivers/net/wireless/rsi/rsi_mgmt.h index ea83faa15c7ec27e4f11ec7d22ad73c0cf487d99..2ce2dcf574412897eb783f8ffb2c92489669262d 100644 --- a/drivers/net/wireless/rsi/rsi_mgmt.h +++ b/drivers/net/wireless/rsi/rsi_mgmt.h @@ -294,6 +294,7 @@ enum cmd_frame_type { COMMON_DEV_CONFIG = 0x28, RADIO_PARAMS_UPDATE = 0x29, WOWLAN_CONFIG_PARAMS = 0x2B, + FEATURES_ENABLE = 0x33, WOWLAN_WAKEUP_REASON = 0xc5 }; @@ -351,6 +352,15 @@ struct rsi_boot_params { struct bootup_params bootup_params; } __packed; +struct rsi_boot_params_9116 { + struct rsi_cmd_desc_dword0 desc_dword0; + struct rsi_cmd_desc_dword1 desc_dword1; + struct rsi_cmd_desc_dword2 desc_dword2; + __le16 reserved; + __le16 umac_clk; + struct bootup_params_9116 bootup_params; +} __packed; + struct rsi_peer_notify { struct rsi_cmd_desc desc; u8 mac_addr[6]; @@ -654,6 +664,22 @@ struct rsi_bgscan_probe { __le16 probe_req_length; } __packed; +#define RSI_DUTY_CYCLING BIT(0) +#define RSI_END_OF_FRAME BIT(1) +#define RSI_SIFS_TX_ENABLE BIT(2) +#define RSI_DPD BIT(3) +struct rsi_wlan_9116_features { + struct rsi_cmd_desc desc; + u8 pll_mode; + u8 rf_type; + u8 wireless_mode; + u8 enable_ppe; + u8 afe_type; + u8 reserved1; + __le16 reserved2; + __le32 feature_enable; +}; + static inline u32 rsi_get_queueno(u8 *addr, u16 offset) { return (le16_to_cpu(*(__le16 *)&addr[offset]) & 0x7000) >> 12; diff --git a/drivers/net/wireless/rsi/rsi_sdio.h b/drivers/net/wireless/rsi/rsi_sdio.h index 66dcd2ec90515b96ac04c2d7fb9c5f7e4044333a..c5cfb6238f73785188a8c9d9471647c8ea3455d5 100644 --- a/drivers/net/wireless/rsi/rsi_sdio.h +++ b/drivers/net/wireless/rsi/rsi_sdio.h @@ -28,8 +28,9 @@ #include #include "rsi_main.h" -#define RSI_SDIO_VID_9113 0x041B +#define RSI_SDIO_VENDOR_ID 0x041B #define RSI_SDIO_PID_9113 0x9330 +#define RSI_SDIO_PID_9116 0x9116 enum sdio_interrupt_type { BUFFER_FULL = 0x0, @@ -91,7 +92,7 @@ enum sdio_interrupt_type { #define TA_SOFT_RST_SET BIT(0) #define TA_PC_ZERO 0 #define TA_HOLD_THREAD_VALUE 0xF -#define TA_RELEASE_THREAD_VALUE cpu_to_le32(0xF) +#define TA_RELEASE_THREAD_VALUE 0xF #define TA_BASE_ADDR 0x2200 #define MISC_CFG_BASE_ADDR 0x4105 diff --git a/drivers/net/wireless/rsi/rsi_usb.h b/drivers/net/wireless/rsi/rsi_usb.h index 5b2eddd1a2ee5f7cc372ad1a16a027d3923b8d5b..8702f434b5699978549144b03203dacf5e016790 100644 --- a/drivers/net/wireless/rsi/rsi_usb.h +++ b/drivers/net/wireless/rsi/rsi_usb.h @@ -22,8 +22,9 @@ #include "rsi_main.h" #include "rsi_common.h" -#define RSI_USB_VID_9113 0x1618 +#define RSI_USB_VENDOR_ID 0x1618 #define RSI_USB_PID_9113 0x9113 +#define RSI_USB_PID_9116 0x9116 #define USB_INTERNAL_REG_1 0x25000 #define RSI_USB_READY_MAGIC_NUM 0xab diff --git a/drivers/net/wireless/st/cw1200/main.c b/drivers/net/wireless/st/cw1200/main.c index 90dc979f260b600c2aad6683122c1b9ba5aa6984..c1608f0bf6d017d9fc5be100dfd5a25ce3054599 100644 --- a/drivers/net/wireless/st/cw1200/main.c +++ b/drivers/net/wireless/st/cw1200/main.c @@ -345,6 +345,11 @@ static struct ieee80211_hw *cw1200_init_common(const u8 *macaddr, mutex_init(&priv->wsm_cmd_mux); mutex_init(&priv->conf_mutex); priv->workqueue = create_singlethread_workqueue("cw1200_wq"); + if (!priv->workqueue) { + ieee80211_free_hw(hw); + return NULL; + } + sema_init(&priv->scan.lock, 1); INIT_WORK(&priv->scan.work, cw1200_scan_work); INIT_DELAYED_WORK(&priv->scan.probe_work, cw1200_probe_work); diff --git a/drivers/net/wireless/ti/wlcore/cmd.c b/drivers/net/wireless/ti/wlcore/cmd.c index 348be0aed97e51be4e8b2bcf9cb9c2565b386cf5..0415a064f6e209d065845a367522b6c59f23a3eb 100644 --- a/drivers/net/wireless/ti/wlcore/cmd.c +++ b/drivers/net/wireless/ti/wlcore/cmd.c @@ -1700,14 +1700,14 @@ void wlcore_set_pending_regdomain_ch(struct wl1271 *wl, u16 channel, ch_bit_idx = wlcore_get_reg_conf_ch_idx(band, channel); if (ch_bit_idx >= 0 && ch_bit_idx <= WL1271_MAX_CHANNELS) - set_bit(ch_bit_idx, (long *)wl->reg_ch_conf_pending); + __set_bit_le(ch_bit_idx, (long *)wl->reg_ch_conf_pending); } int wlcore_cmd_regdomain_config_locked(struct wl1271 *wl) { struct wl12xx_cmd_regdomain_dfs_config *cmd = NULL; int ret = 0, i, b, ch_bit_idx; - u32 tmp_ch_bitmap[2]; + __le32 tmp_ch_bitmap[2] __aligned(sizeof(unsigned long)); struct wiphy *wiphy = wl->hw->wiphy; struct ieee80211_supported_band *band; bool timeout = false; @@ -1717,7 +1717,7 @@ int wlcore_cmd_regdomain_config_locked(struct wl1271 *wl) wl1271_debug(DEBUG_CMD, "cmd reg domain config"); - memset(tmp_ch_bitmap, 0, sizeof(tmp_ch_bitmap)); + memcpy(tmp_ch_bitmap, wl->reg_ch_conf_pending, sizeof(tmp_ch_bitmap)); for (b = NL80211_BAND_2GHZ; b <= NL80211_BAND_5GHZ; b++) { band = wiphy->bands[b]; @@ -1738,13 +1738,10 @@ int wlcore_cmd_regdomain_config_locked(struct wl1271 *wl) if (ch_bit_idx < 0) continue; - set_bit(ch_bit_idx, (long *)tmp_ch_bitmap); + __set_bit_le(ch_bit_idx, (long *)tmp_ch_bitmap); } } - tmp_ch_bitmap[0] |= wl->reg_ch_conf_pending[0]; - tmp_ch_bitmap[1] |= wl->reg_ch_conf_pending[1]; - if (!memcmp(tmp_ch_bitmap, wl->reg_ch_conf_last, sizeof(tmp_ch_bitmap))) goto out; @@ -1754,8 +1751,8 @@ int wlcore_cmd_regdomain_config_locked(struct wl1271 *wl) goto out; } - cmd->ch_bit_map1 = cpu_to_le32(tmp_ch_bitmap[0]); - cmd->ch_bit_map2 = cpu_to_le32(tmp_ch_bitmap[1]); + cmd->ch_bit_map1 = tmp_ch_bitmap[0]; + cmd->ch_bit_map2 = tmp_ch_bitmap[1]; cmd->dfs_region = wl->dfs_region; wl1271_debug(DEBUG_CMD, diff --git a/drivers/net/wireless/ti/wlcore/testmode.c b/drivers/net/wireless/ti/wlcore/testmode.c index dcb2c8b0feb63ae4b1ecb35a0d43ab1fdddf30b9..affefaaea1ea3ad92269c6ff460621905620e9ae 100644 --- a/drivers/net/wireless/ti/wlcore/testmode.c +++ b/drivers/net/wireless/ti/wlcore/testmode.c @@ -372,8 +372,8 @@ int wl1271_tm_cmd(struct ieee80211_hw *hw, struct ieee80211_vif *vif, u32 nla_cmd; int err; - err = nla_parse(tb, WL1271_TM_ATTR_MAX, data, len, wl1271_tm_policy, - NULL); + err = nla_parse_deprecated(tb, WL1271_TM_ATTR_MAX, data, len, + wl1271_tm_policy, NULL); if (err) return err; diff --git a/drivers/net/wireless/ti/wlcore/vendor_cmd.c b/drivers/net/wireless/ti/wlcore/vendor_cmd.c index 7f34ec077ee57000e49fc094042ceb834c680ff2..75756fb8e7b0df7ec00a65c83cee724514ed5be5 100644 --- a/drivers/net/wireless/ti/wlcore/vendor_cmd.c +++ b/drivers/net/wireless/ti/wlcore/vendor_cmd.c @@ -41,8 +41,8 @@ wlcore_vendor_cmd_smart_config_start(struct wiphy *wiphy, if (!data) return -EINVAL; - ret = nla_parse(tb, MAX_WLCORE_VENDOR_ATTR, data, data_len, - wlcore_vendor_attr_policy, NULL); + ret = nla_parse_deprecated(tb, MAX_WLCORE_VENDOR_ATTR, data, data_len, + wlcore_vendor_attr_policy, NULL); if (ret) return ret; @@ -122,8 +122,8 @@ wlcore_vendor_cmd_smart_config_set_group_key(struct wiphy *wiphy, if (!data) return -EINVAL; - ret = nla_parse(tb, MAX_WLCORE_VENDOR_ATTR, data, data_len, - wlcore_vendor_attr_policy, NULL); + ret = nla_parse_deprecated(tb, MAX_WLCORE_VENDOR_ATTR, data, data_len, + wlcore_vendor_attr_policy, NULL); if (ret) return ret; diff --git a/drivers/net/wireless/ti/wlcore/wlcore.h b/drivers/net/wireless/ti/wlcore/wlcore.h index dd14850b06033ebfa014fa5d351ec9bf0e0207fb..870eea3e7a27276935f5cbd38046b10d23b2a635 100644 --- a/drivers/net/wireless/ti/wlcore/wlcore.h +++ b/drivers/net/wireless/ti/wlcore/wlcore.h @@ -320,9 +320,9 @@ struct wl1271 { bool watchdog_recovery; /* Reg domain last configuration */ - u32 reg_ch_conf_last[2] __aligned(8); + DECLARE_BITMAP(reg_ch_conf_last, 64); /* Reg domain pending configuration */ - u32 reg_ch_conf_pending[2]; + DECLARE_BITMAP(reg_ch_conf_pending, 64); /* Pointer that holds DMA-friendly block for the mailbox */ void *mbox; diff --git a/drivers/net/wireless/zydas/zd1211rw/zd_usb.c b/drivers/net/wireless/zydas/zd1211rw/zd_usb.c index c2cda3acd4af37a2ca4130283010e7faba8d3af1..a094d5b3383cfc982f59aa38f53e0f65057613dc 100644 --- a/drivers/net/wireless/zydas/zd1211rw/zd_usb.c +++ b/drivers/net/wireless/zydas/zd1211rw/zd_usb.c @@ -1917,8 +1917,7 @@ int zd_usb_iowrite16v_async(struct zd_usb *usb, const struct zd_ioreq16 *ioreqs, if (!urb) return -ENOMEM; - req_len = sizeof(struct usb_req_write_regs) + - count * sizeof(struct reg_data); + req_len = struct_size(req, reg_writes, count); req = kmalloc(req_len, GFP_KERNEL); if (!req) { r = -ENOMEM; diff --git a/drivers/net/xen-netback/common.h b/drivers/net/xen-netback/common.h index 936c0b3e0ba28ec1f6586a5bab86d403a86d0dd6..05847eb91a1b4a24c29567384d0a1b7094f56235 100644 --- a/drivers/net/xen-netback/common.h +++ b/drivers/net/xen-netback/common.h @@ -248,6 +248,22 @@ struct xenvif_hash { struct xenvif_hash_cache cache; }; +struct backend_info { + struct xenbus_device *dev; + struct xenvif *vif; + + /* This is the state that will be reflected in xenstore when any + * active hotplug script completes. + */ + enum xenbus_state state; + + enum xenbus_state frontend_state; + struct xenbus_watch hotplug_status_watch; + u8 have_hotplug_status_watch:1; + + const char *hotplug_script; +}; + struct xenvif { /* Unique identifier for this interface. */ domid_t domid; @@ -283,6 +299,8 @@ struct xenvif { struct xenbus_watch credit_watch; struct xenbus_watch mcast_ctrl_watch; + struct backend_info *be; + spinlock_t lock; #ifdef CONFIG_DEBUG_FS diff --git a/drivers/net/xen-netback/interface.c b/drivers/net/xen-netback/interface.c index 6da12518e6934ce20ec6a1becfdbbef07f430822..783198844dd766f92658c5ca3d205690a8ac3aad 100644 --- a/drivers/net/xen-netback/interface.c +++ b/drivers/net/xen-netback/interface.c @@ -148,8 +148,7 @@ void xenvif_wake_queue(struct xenvif_queue *queue) } static u16 xenvif_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { struct xenvif *vif = netdev_priv(dev); unsigned int size = vif->hash.size; @@ -162,7 +161,8 @@ static u16 xenvif_select_queue(struct net_device *dev, struct sk_buff *skb, return 0; if (vif->hash.alg == XEN_NETIF_CTRL_HASH_ALGORITHM_NONE) - return fallback(dev, skb, NULL) % dev->real_num_tx_queues; + return netdev_pick_tx(dev, skb, NULL) % + dev->real_num_tx_queues; xenvif_set_skb_hash(vif, skb); diff --git a/drivers/net/xen-netback/xenbus.c b/drivers/net/xen-netback/xenbus.c index 330ddb64930f98f694a40afd72acb8f78515aca9..41c9e8f2e52032d24309f6971e8c947b5b949a93 100644 --- a/drivers/net/xen-netback/xenbus.c +++ b/drivers/net/xen-netback/xenbus.c @@ -22,22 +22,6 @@ #include #include -struct backend_info { - struct xenbus_device *dev; - struct xenvif *vif; - - /* This is the state that will be reflected in xenstore when any - * active hotplug script completes. - */ - enum xenbus_state state; - - enum xenbus_state frontend_state; - struct xenbus_watch hotplug_status_watch; - u8 have_hotplug_status_watch:1; - - const char *hotplug_script; -}; - static int connect_data_rings(struct backend_info *be, struct xenvif_queue *queue); static void connect(struct backend_info *be); @@ -472,6 +456,7 @@ static int backend_create_xenvif(struct backend_info *be) return err; } be->vif = vif; + vif->be = be; kobject_uevent(&dev->dev.kobj, KOBJ_ONLINE); return 0; diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c index c914c24f880bc694d83e98c1e903ebcf7a364e36..8d33970a2950ea824831426cc4ef935dd06e6a6a 100644 --- a/drivers/net/xen-netfront.c +++ b/drivers/net/xen-netfront.c @@ -543,8 +543,7 @@ static int xennet_count_skb_slots(struct sk_buff *skb) } static u16 xennet_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { unsigned int num_queues = dev->real_num_tx_queues; u32 hash; @@ -2038,7 +2037,7 @@ static void netback_changed(struct xenbus_device *dev, case XenbusStateClosed: if (dev->state == XenbusStateClosed) break; - /* Missed the backend's CLOSING state -- fallthrough */ + /* Fall through - Missed the backend's CLOSING state. */ case XenbusStateClosing: xenbus_frontend_closed(dev); break; diff --git a/drivers/nfc/st95hf/core.c b/drivers/nfc/st95hf/core.c index 01acb6e533655d6b6041cbbde43af8c1364aec60..99727a2edda0c9cb138611935634f105bb4a8ab8 100644 --- a/drivers/nfc/st95hf/core.c +++ b/drivers/nfc/st95hf/core.c @@ -781,9 +781,7 @@ static irqreturn_t st95hf_irq_thread_handler(int irq, void *st95hfcontext) int result = 0; int res_len; static bool wtx; - struct device *dev; struct device *spidevice; - struct nfc_digital_dev *nfcddev; struct sk_buff *skb_resp; struct st95hf_context *stcontext = (struct st95hf_context *)st95hfcontext; @@ -828,8 +826,6 @@ static irqreturn_t st95hf_irq_thread_handler(int irq, void *st95hfcontext) goto end; } - dev = &stcontext->nfcdev->dev; - nfcddev = stcontext->ddev; if (skb_resp->data[2] == WTX_REQ_FROM_TAG) { /* Request for new FWT from tag */ result = st95hf_handle_wtx(stcontext, true, skb_resp->data[3]); diff --git a/drivers/of/of_net.c b/drivers/of/of_net.c index d820f3edd4311821696e6045d843024166b83dfc..9649cd53e95574a26a61b65bc587788ab19846e6 100644 --- a/drivers/of/of_net.c +++ b/drivers/of/of_net.c @@ -8,8 +8,10 @@ #include #include #include +#include #include #include +#include /** * of_get_phy_mode - Get phy mode for given device_node @@ -47,12 +49,52 @@ static const void *of_get_mac_addr(struct device_node *np, const char *name) return NULL; } +static const void *of_get_mac_addr_nvmem(struct device_node *np) +{ + int ret; + u8 mac[ETH_ALEN]; + struct property *pp; + struct platform_device *pdev = of_find_device_by_node(np); + + if (!pdev) + return ERR_PTR(-ENODEV); + + ret = nvmem_get_mac_address(&pdev->dev, &mac); + if (ret) + return ERR_PTR(ret); + + pp = devm_kzalloc(&pdev->dev, sizeof(*pp), GFP_KERNEL); + if (!pp) + return ERR_PTR(-ENOMEM); + + pp->name = "nvmem-mac-address"; + pp->length = ETH_ALEN; + pp->value = devm_kmemdup(&pdev->dev, mac, ETH_ALEN, GFP_KERNEL); + if (!pp->value) { + ret = -ENOMEM; + goto free; + } + + ret = of_add_property(np, pp); + if (ret) + goto free; + + return pp->value; +free: + devm_kfree(&pdev->dev, pp->value); + devm_kfree(&pdev->dev, pp); + + return ERR_PTR(ret); +} + /** * Search the device tree for the best MAC address to use. 'mac-address' is * checked first, because that is supposed to contain to "most recent" MAC * address. If that isn't set, then 'local-mac-address' is checked next, - * because that is the default address. If that isn't set, then the obsolete - * 'address' is checked, just in case we're using an old device tree. + * because that is the default address. If that isn't set, then the obsolete + * 'address' is checked, just in case we're using an old device tree. If any + * of the above isn't set, then try to get MAC address from nvmem cell named + * 'mac-address'. * * Note that the 'address' property is supposed to contain a virtual address of * the register set, but some DTS files have redefined that property to be the @@ -64,6 +106,8 @@ static const void *of_get_mac_addr(struct device_node *np, const char *name) * addresses. Some older U-Boots only initialized 'local-mac-address'. In * this case, the real MAC is in 'local-mac-address', and 'mac-address' exists * but is all zeros. + * + * Return: Will be a valid pointer on success and ERR_PTR in case of error. */ const void *of_get_mac_address(struct device_node *np) { @@ -77,6 +121,10 @@ const void *of_get_mac_address(struct device_node *np) if (addr) return addr; - return of_get_mac_addr(np, "address"); + addr = of_get_mac_addr(np, "address"); + if (addr) + return addr; + + return of_get_mac_addr_nvmem(np); } EXPORT_SYMBOL(of_get_mac_address); diff --git a/drivers/s390/net/qeth_core.h b/drivers/s390/net/qeth_core.h index c851cf6e01c43664e0c624fad6b3e23b47746e5a..784a2e76a1b04a45f13399a73244e59493bcb733 100644 --- a/drivers/s390/net/qeth_core.h +++ b/drivers/s390/net/qeth_core.h @@ -10,6 +10,7 @@ #ifndef __QETH_CORE_H__ #define __QETH_CORE_H__ +#include #include #include #include @@ -21,6 +22,7 @@ #include #include #include +#include #include #include @@ -163,6 +165,12 @@ struct qeth_vnicc_info { bool rx_bcast_enabled; }; +static inline int qeth_is_adp_supported(struct qeth_ipa_info *ipa, + enum qeth_ipa_setadp_cmd func) +{ + return (ipa->supported_funcs & func); +} + static inline int qeth_is_ipa_supported(struct qeth_ipa_info *ipa, enum qeth_ipa_funcs func) { @@ -176,9 +184,7 @@ static inline int qeth_is_ipa_enabled(struct qeth_ipa_info *ipa, } #define qeth_adp_supported(c, f) \ - qeth_is_ipa_supported(&c->options.adp, f) -#define qeth_adp_enabled(c, f) \ - qeth_is_ipa_enabled(&c->options.adp, f) + qeth_is_adp_supported(&c->options.adp, f) #define qeth_is_supported(c, f) \ qeth_is_ipa_supported(&c->options.ipa4, f) #define qeth_is_enabled(c, f) \ @@ -217,6 +223,9 @@ static inline int qeth_is_ipa_enabled(struct qeth_ipa_info *ipa, /* QDIO queue and buffer handling */ /*****************************************************************************/ #define QETH_MAX_QUEUES 4 +#define QETH_IQD_MIN_TXQ 2 /* One for ucast, one for mcast. */ +#define QETH_IQD_MCAST_TXQ 0 +#define QETH_IQD_MIN_UCAST_TXQ 1 #define QETH_IN_BUF_SIZE_DEFAULT 65536 #define QETH_IN_BUF_COUNT_DEFAULT 64 #define QETH_IN_BUF_COUNT_HSDEFAULT 128 @@ -365,34 +374,6 @@ enum qeth_header_ids { #define QETH_HDR_EXT_CSUM_TRANSP_REQ 0x20 #define QETH_HDR_EXT_UDP 0x40 /*bit off for TCP*/ -enum qeth_qdio_buffer_states { - /* - * inbound: read out by driver; owned by hardware in order to be filled - * outbound: owned by driver in order to be filled - */ - QETH_QDIO_BUF_EMPTY, - /* - * inbound: filled by hardware; owned by driver in order to be read out - * outbound: filled by driver; owned by hardware in order to be sent - */ - QETH_QDIO_BUF_PRIMED, - /* - * inbound: not applicable - * outbound: identified to be pending in TPQ - */ - QETH_QDIO_BUF_PENDING, - /* - * inbound: not applicable - * outbound: found in completion queue - */ - QETH_QDIO_BUF_IN_CQ, - /* - * inbound: not applicable - * outbound: handled via transfer pending / completion queue - */ - QETH_QDIO_BUF_HANDLED_DELAYED, -}; - enum qeth_qdio_info_states { QETH_QDIO_UNINITIALIZED, QETH_QDIO_ALLOCATED, @@ -424,6 +405,19 @@ struct qeth_qdio_q { int next_buf_to_init; }; +enum qeth_qdio_out_buffer_state { + /* Owned by driver, in order to be filled. */ + QETH_QDIO_BUF_EMPTY, + /* Filled by driver; owned by hardware in order to be sent. */ + QETH_QDIO_BUF_PRIMED, + /* Identified to be pending in TPQ. */ + QETH_QDIO_BUF_PENDING, + /* Found in completion queue. */ + QETH_QDIO_BUF_IN_CQ, + /* Handled via transfer pending / completion queue. */ + QETH_QDIO_BUF_HANDLED_DELAYED, +}; + struct qeth_qdio_out_buffer { struct qdio_buffer *buffer; atomic_t state; @@ -462,7 +456,6 @@ struct qeth_card_stats { u64 rx_errors; u64 rx_dropped; u64 rx_multicast; - u64 tx_errors; }; struct qeth_out_q_stats { @@ -477,6 +470,7 @@ struct qeth_out_q_stats { u64 skbs_linearized_fail; u64 tso_bytes; u64 packing_mode_switch; + u64 stopped; /* rtnl_link_stats64 */ u64 tx_packets; @@ -490,14 +484,12 @@ struct qeth_qdio_out_q { struct qeth_qdio_out_buffer *bufs[QDIO_MAX_BUFFERS_PER_Q]; struct qdio_outbuf_state *bufstates; /* convenience pointer */ struct qeth_out_q_stats stats; - int queue_no; + u8 next_buf_to_fill; + u8 max_elements; + u8 queue_no; + u8 do_pack; struct qeth_card *card; atomic_t state; - int do_pack; - /* - * index of buffer to be filled by driver; state EMPTY or PACKING - */ - int next_buf_to_fill; /* * number of buffers that are currently filled (PRIMED) * -> these buffers are hardware-owned @@ -507,6 +499,11 @@ struct qeth_qdio_out_q { atomic_t set_pci_flags_count; }; +static inline bool qeth_out_queue_is_full(struct qeth_qdio_out_q *queue) +{ + return atomic_read(&queue->used_buffers) >= QDIO_MAX_BUFFERS_PER_Q; +} + struct qeth_qdio_info { atomic_t state; /* input */ @@ -538,7 +535,6 @@ struct qeth_qdio_info { enum qeth_channel_states { CH_STATE_UP, CH_STATE_DOWN, - CH_STATE_ACTIVATING, CH_STATE_HALTED, CH_STATE_STOPPED, CH_STATE_RCD, @@ -585,7 +581,10 @@ struct qeth_cmd_buffer { enum qeth_cmd_buffer_state state; struct qeth_channel *channel; struct qeth_reply *reply; + long timeout; unsigned char *data; + void (*finalize)(struct qeth_card *card, struct qeth_cmd_buffer *iob, + unsigned int length); void (*callback)(struct qeth_card *card, struct qeth_channel *channel, struct qeth_cmd_buffer *iob); }; @@ -610,6 +609,11 @@ struct qeth_channel { int io_buf_no; }; +static inline bool qeth_trylock_channel(struct qeth_channel *channel) +{ + return atomic_cmpxchg(&channel->irq_pending, 0, 1) == 0; +} + /** * OSA card related definitions */ @@ -631,17 +635,15 @@ struct qeth_seqno { __u32 pdu_hdr; __u32 pdu_hdr_ack; __u16 ipa; - __u32 pkt_seqno; }; struct qeth_reply { struct list_head list; - wait_queue_head_t wait_q; + struct completion received; int (*callback)(struct qeth_card *, struct qeth_reply *, unsigned long); u32 seqno; unsigned long offset; - atomic_t received; int rc; void *param; refcount_t refcnt; @@ -663,7 +665,7 @@ struct qeth_card_info { __u16 func_level; char mcl_level[QETH_MCL_LENGTH + 1]; u8 open_when_online:1; - int guestlan; + u8 is_vm_nic:1; int mac_bits; enum qeth_card_types type; enum qeth_link_types link_type; @@ -774,18 +776,19 @@ struct qeth_card { struct qeth_card_options options; struct workqueue_struct *event_wq; + struct workqueue_struct *cmd_wq; wait_queue_head_t wait_q; - spinlock_t mclock; unsigned long active_vlans[BITS_TO_LONGS(VLAN_N_VID)]; DECLARE_HASHTABLE(mac_htable, 4); DECLARE_HASHTABLE(ip_htable, 4); + struct mutex ip_lock; DECLARE_HASHTABLE(ip_mc_htable, 4); + struct work_struct rx_mode_work; struct work_struct kernel_thread_starter; spinlock_t thread_mask_lock; unsigned long thread_start_mask; unsigned long thread_allowed_mask; unsigned long thread_running_mask; - spinlock_t ip_lock; struct qeth_ipato ipato; struct list_head cmd_waiter_list; /* QDIO buffer handling */ @@ -827,6 +830,15 @@ static inline bool qeth_netdev_is_registered(struct net_device *dev) return dev->netdev_ops != NULL; } +static inline u16 qeth_iqd_translate_txq(struct net_device *dev, u16 txq) +{ + if (txq == QETH_IQD_MCAST_TXQ) + return dev->num_tx_queues - 1; + if (txq == dev->num_tx_queues - 1) + return QETH_IQD_MCAST_TXQ; + return txq; +} + static inline void qeth_scrub_qdio_buffer(struct qdio_buffer *buf, unsigned int elements) { @@ -869,6 +881,16 @@ static inline int qeth_get_ip_version(struct sk_buff *skb) } } +static inline int qeth_get_ether_cast_type(struct sk_buff *skb) +{ + u8 *addr = eth_hdr(skb)->h_dest; + + if (is_multicast_ether_addr(addr)) + return is_broadcast_ether_addr(addr) ? RTN_BROADCAST : + RTN_MULTICAST; + return RTN_UNICAST; +} + static inline void qeth_rx_csum(struct qeth_card *card, struct sk_buff *skb, u8 flags) { @@ -922,18 +944,7 @@ static inline int qeth_send_simple_setassparms_v6(struct qeth_card *card, data, QETH_PROT_IPV6); } -int qeth_get_priority_queue(struct qeth_card *card, struct sk_buff *skb, - int ipv); -static inline struct qeth_qdio_out_q *qeth_get_tx_queue(struct qeth_card *card, - struct sk_buff *skb, - int ipv, int cast_type) -{ - if (IS_IQD(card) && cast_type != RTN_UNICAST) - return card->qdio.out_qs[card->qdio.no_out_queues - 1]; - if (!card->qdio.do_prio_queueing) - return card->qdio.out_qs[card->qdio.default_out_queue]; - return card->qdio.out_qs[qeth_get_priority_queue(card, skb, ipv)]; -} +int qeth_get_priority_queue(struct qeth_card *card, struct sk_buff *skb); extern struct qeth_discipline qeth_l2_discipline; extern struct qeth_discipline qeth_l3_discipline; @@ -979,12 +990,10 @@ void qeth_clear_ipacmd_list(struct qeth_card *); int qeth_qdio_clear_card(struct qeth_card *, int); void qeth_clear_working_pool_list(struct qeth_card *); void qeth_clear_cmd_buffers(struct qeth_channel *); -void qeth_clear_qdio_buffers(struct qeth_card *); +void qeth_drain_output_queues(struct qeth_card *card); void qeth_setadp_promisc_mode(struct qeth_card *); int qeth_setadpparms_change_macaddr(struct qeth_card *); void qeth_tx_timeout(struct net_device *); -void qeth_prepare_control_data(struct qeth_card *, int, - struct qeth_cmd_buffer *); void qeth_release_buffer(struct qeth_channel *, struct qeth_cmd_buffer *); void qeth_prepare_ipa_cmd(struct qeth_card *card, struct qeth_cmd_buffer *iob, u16 cmd_length); @@ -1016,6 +1025,8 @@ netdev_features_t qeth_features_check(struct sk_buff *skb, struct net_device *dev, netdev_features_t features); void qeth_get_stats64(struct net_device *dev, struct rtnl_link_stats64 *stats); +u16 qeth_iqd_select_queue(struct net_device *dev, struct sk_buff *skb, + u8 cast_type, struct net_device *sb_dev); int qeth_open(struct net_device *dev); int qeth_stop(struct net_device *dev); diff --git a/drivers/s390/net/qeth_core_main.c b/drivers/s390/net/qeth_core_main.c index 44bd6f04c145da55b1aef66ad147983d74cedf9a..009f2c0ec504247877d433711c33401e11cb0475 100644 --- a/drivers/s390/net/qeth_core_main.c +++ b/drivers/s390/net/qeth_core_main.c @@ -61,13 +61,13 @@ static struct kmem_cache *qeth_qdio_outbuf_cache; static struct device *qeth_core_root_dev; static struct lock_class_key qdio_out_skb_queue_key; -static void qeth_send_control_data_cb(struct qeth_card *card, - struct qeth_channel *channel, - struct qeth_cmd_buffer *iob); +static void qeth_issue_next_read_cb(struct qeth_card *card, + struct qeth_channel *channel, + struct qeth_cmd_buffer *iob); static struct qeth_cmd_buffer *qeth_get_buffer(struct qeth_channel *); static void qeth_free_buffer_pool(struct qeth_card *); static int qeth_qdio_establish(struct qeth_card *); -static void qeth_free_qdio_buffers(struct qeth_card *); +static void qeth_free_qdio_queues(struct qeth_card *card); static void qeth_notify_skbs(struct qeth_qdio_out_q *queue, struct qeth_qdio_out_buffer *buf, enum iucv_tx_notify notification); @@ -85,7 +85,7 @@ static void qeth_close_dev_handler(struct work_struct *work) static const char *qeth_get_cardname(struct qeth_card *card) { - if (card->info.guestlan) { + if (IS_VM_NIC(card)) { switch (card->info.type) { case QETH_CARD_TYPE_OSD: return " Virtual NIC QDIO"; @@ -120,7 +120,7 @@ static const char *qeth_get_cardname(struct qeth_card *card) /* max length to be returned: 14 */ const char *qeth_get_cardname_short(struct qeth_card *card) { - if (card->info.guestlan) { + if (IS_VM_NIC(card)) { switch (card->info.type) { case QETH_CARD_TYPE_OSD: return "Virt.NIC QDIO"; @@ -511,7 +511,9 @@ static int __qeth_issue_next_read(struct qeth_card *card) CARD_DEVID(card)); return -ENOMEM; } + qeth_setup_ccw(channel->ccw, CCW_CMD_READ, QETH_BUFSIZE, iob->data); + iob->callback = qeth_issue_next_read_cb; QETH_CARD_TEXT(card, 6, "noirqpnd"); rc = ccw_device_start(channel->ccwdev, channel->ccw, (addr_t) iob, 0, 0); @@ -542,11 +544,10 @@ static struct qeth_reply *qeth_alloc_reply(struct qeth_card *card) { struct qeth_reply *reply; - reply = kzalloc(sizeof(struct qeth_reply), GFP_ATOMIC); + reply = kzalloc(sizeof(*reply), GFP_KERNEL); if (reply) { refcount_set(&reply->refcnt, 1); - atomic_set(&reply->received, 0); - init_waitqueue_head(&reply->wait_q); + init_completion(&reply->received); } return reply; } @@ -576,10 +577,10 @@ static void qeth_dequeue_reply(struct qeth_card *card, struct qeth_reply *reply) spin_unlock_irq(&card->lock); } -static void qeth_notify_reply(struct qeth_reply *reply) +static void qeth_notify_reply(struct qeth_reply *reply, int reason) { - atomic_inc(&reply->received); - wake_up(&reply->wait_q); + reply->rc = reason; + complete(&reply->received); } static void qeth_issue_ipa_msg(struct qeth_ipa_cmd *cmd, int rc, @@ -664,10 +665,8 @@ void qeth_clear_ipacmd_list(struct qeth_card *card) QETH_CARD_TEXT(card, 4, "clipalst"); spin_lock_irqsave(&card->lock, flags); - list_for_each_entry(reply, &card->cmd_waiter_list, list) { - reply->rc = -EIO; - qeth_notify_reply(reply); - } + list_for_each_entry(reply, &card->cmd_waiter_list, list) + qeth_notify_reply(reply, -EIO); spin_unlock_irqrestore(&card->lock, flags); } EXPORT_SYMBOL_GPL(qeth_clear_ipacmd_list); @@ -675,9 +674,6 @@ EXPORT_SYMBOL_GPL(qeth_clear_ipacmd_list); static int qeth_check_idx_response(struct qeth_card *card, unsigned char *buffer) { - if (!buffer) - return 0; - QETH_DBF_HEX(CTRL, 2, buffer, QETH_DBF_CTRL_LEN); if ((buffer[2] & 0xc0) == 0xc0) { QETH_DBF_MESSAGE(2, "received an IDX TERMINATE with cause code %#04x\n", @@ -704,6 +700,7 @@ static struct qeth_cmd_buffer *__qeth_get_buffer(struct qeth_channel *channel) do { if (channel->iob[index].state == BUF_STATE_FREE) { channel->iob[index].state = BUF_STATE_LOCKED; + channel->iob[index].timeout = QETH_TIMEOUT; channel->io_buf_no = (channel->io_buf_no + 1) % QETH_CMD_BUFFER_NO; memset(channel->iob[index].data, 0, QETH_BUFSIZE); @@ -722,7 +719,7 @@ void qeth_release_buffer(struct qeth_channel *channel, spin_lock_irqsave(&channel->iob_lock, flags); iob->state = BUF_STATE_FREE; - iob->callback = qeth_send_control_data_cb; + iob->callback = NULL; if (iob->reply) { qeth_put_reply(iob->reply); iob->reply = NULL; @@ -743,10 +740,8 @@ static void qeth_cancel_cmd(struct qeth_cmd_buffer *iob, int rc) { struct qeth_reply *reply = iob->reply; - if (reply) { - reply->rc = rc; - qeth_notify_reply(reply); - } + if (reply) + qeth_notify_reply(reply, rc); qeth_release_buffer(iob->channel, iob); } @@ -780,9 +775,9 @@ void qeth_clear_cmd_buffers(struct qeth_channel *channel) } EXPORT_SYMBOL_GPL(qeth_clear_cmd_buffers); -static void qeth_send_control_data_cb(struct qeth_card *card, - struct qeth_channel *channel, - struct qeth_cmd_buffer *iob) +static void qeth_issue_next_read_cb(struct qeth_card *card, + struct qeth_channel *channel, + struct qeth_cmd_buffer *iob) { struct qeth_ipa_cmd *cmd = NULL; struct qeth_reply *reply = NULL; @@ -846,11 +841,8 @@ static void qeth_send_control_data_cb(struct qeth_card *card, } } - if (rc <= 0) { - reply->rc = rc; - qeth_notify_reply(reply); - } - + if (rc <= 0) + qeth_notify_reply(reply, rc); qeth_put_reply(reply); out: @@ -1173,20 +1165,19 @@ static void qeth_clear_output_buffer(struct qeth_qdio_out_q *queue, qeth_release_skbs(buf); - for (i = 0; i < QETH_MAX_BUFFER_ELEMENTS(queue->card); ++i) { + for (i = 0; i < queue->max_elements; ++i) { if (buf->buffer->element[i].addr && buf->is_header[i]) kmem_cache_free(qeth_core_header_cache, buf->buffer->element[i].addr); buf->is_header[i] = 0; } - qeth_scrub_qdio_buffer(buf->buffer, - QETH_MAX_BUFFER_ELEMENTS(queue->card)); + qeth_scrub_qdio_buffer(buf->buffer, queue->max_elements); buf->next_element_to_fill = 0; atomic_set(&buf->state, QETH_QDIO_BUF_EMPTY); } -static void qeth_clear_outq_buffers(struct qeth_qdio_out_q *q, int free) +static void qeth_drain_output_queue(struct qeth_qdio_out_q *q, bool free) { int j; @@ -1202,19 +1193,18 @@ static void qeth_clear_outq_buffers(struct qeth_qdio_out_q *q, int free) } } -void qeth_clear_qdio_buffers(struct qeth_card *card) +void qeth_drain_output_queues(struct qeth_card *card) { int i; QETH_CARD_TEXT(card, 2, "clearqdbf"); /* clear outbound buffers to free skbs */ for (i = 0; i < card->qdio.no_out_queues; ++i) { - if (card->qdio.out_qs[i]) { - qeth_clear_outq_buffers(card->qdio.out_qs[i], 0); - } + if (card->qdio.out_qs[i]) + qeth_drain_output_queue(card->qdio.out_qs[i], false); } } -EXPORT_SYMBOL_GPL(qeth_clear_qdio_buffers); +EXPORT_SYMBOL_GPL(qeth_drain_output_queues); static void qeth_free_buffer_pool(struct qeth_card *card) { @@ -1273,7 +1263,6 @@ static int qeth_setup_channel(struct qeth_channel *channel, bool alloc_buffers) break; channel->iob[cnt].state = BUF_STATE_FREE; channel->iob[cnt].channel = channel; - channel->iob[cnt].callback = qeth_send_control_data_cb; } if (cnt < QETH_CMD_BUFFER_NO) { qeth_clean_channel(channel); @@ -1285,30 +1274,28 @@ static int qeth_setup_channel(struct qeth_channel *channel, bool alloc_buffers) return 0; } -static void qeth_set_single_write_queues(struct qeth_card *card) +static void qeth_osa_set_output_queues(struct qeth_card *card, bool single) { - if ((atomic_read(&card->qdio.state) != QETH_QDIO_UNINITIALIZED) && - (card->qdio.no_out_queues == 4)) - qeth_free_qdio_buffers(card); + unsigned int count = single ? 1 : card->dev->num_tx_queues; - card->qdio.no_out_queues = 1; - if (card->qdio.default_out_queue != 0) - dev_info(&card->gdev->dev, "Priority Queueing not supported\n"); + rtnl_lock(); + netif_set_real_num_tx_queues(card->dev, count); + rtnl_unlock(); - card->qdio.default_out_queue = 0; -} + if (card->qdio.no_out_queues == count) + return; -static void qeth_set_multiple_write_queues(struct qeth_card *card) -{ - if ((atomic_read(&card->qdio.state) != QETH_QDIO_UNINITIALIZED) && - (card->qdio.no_out_queues == 1)) { - qeth_free_qdio_buffers(card); - card->qdio.default_out_queue = 2; - } - card->qdio.no_out_queues = 4; + if (atomic_read(&card->qdio.state) != QETH_QDIO_UNINITIALIZED) + qeth_free_qdio_queues(card); + + if (count == 1) + dev_info(&card->gdev->dev, "Priority Queueing not supported\n"); + + card->qdio.default_out_queue = single ? 0 : QETH_DEFAULT_QUEUE; + card->qdio.no_out_queues = count; } -static void qeth_update_from_chp_desc(struct qeth_card *card) +static int qeth_update_from_chp_desc(struct qeth_card *card) { struct ccw_device *ccwdev; struct channel_path_desc_fmt0 *chp_dsc; @@ -1318,21 +1305,18 @@ static void qeth_update_from_chp_desc(struct qeth_card *card) ccwdev = card->data.ccwdev; chp_dsc = ccw_device_get_chp_desc(ccwdev, 0); if (!chp_dsc) - goto out; + return -ENOMEM; card->info.func_level = 0x4100 + chp_dsc->desc; - if (card->info.type == QETH_CARD_TYPE_IQD) - goto out; - /* CHPP field bit 6 == 1 -> single queue */ - if ((chp_dsc->chpp & 0x02) == 0x02) - qeth_set_single_write_queues(card); - else - qeth_set_multiple_write_queues(card); -out: + if (IS_OSD(card) || IS_OSX(card)) + /* CHPP field bit 6 == 1 -> single queue */ + qeth_osa_set_output_queues(card, chp_dsc->chpp & 0x02); + kfree(chp_dsc); QETH_DBF_TEXT_(SETUP, 2, "nr:%x", card->qdio.no_out_queues); QETH_DBF_TEXT_(SETUP, 2, "lvl:%02x", card->info.func_level); + return 0; } static void qeth_init_qdio_info(struct qeth_card *card) @@ -1341,12 +1325,11 @@ static void qeth_init_qdio_info(struct qeth_card *card) atomic_set(&card->qdio.state, QETH_QDIO_UNINITIALIZED); card->qdio.do_prio_queueing = QETH_PRIOQ_DEFAULT; card->qdio.default_out_queue = QETH_DEFAULT_QUEUE; - card->qdio.no_out_queues = QETH_MAX_QUEUES; /* inbound */ card->qdio.no_in_queues = 1; card->qdio.in_buf_size = QETH_IN_BUF_SIZE_DEFAULT; - if (card->info.type == QETH_CARD_TYPE_IQD) + if (IS_IQD(card)) card->qdio.init_pool.buf_count = QETH_IN_BUF_COUNT_HSDEFAULT; else card->qdio.init_pool.buf_count = QETH_IN_BUF_COUNT_DEFAULT; @@ -1409,9 +1392,7 @@ static void qeth_setup_card(struct qeth_card *card) card->info.type = CARD_RDEV(card)->id.driver_info; card->state = CARD_STATE_DOWN; - spin_lock_init(&card->mclock); spin_lock_init(&card->lock); - spin_lock_init(&card->ip_lock); spin_lock_init(&card->thread_mask_lock); mutex_init(&card->conf_mutex); mutex_init(&card->discipline_mutex); @@ -1451,7 +1432,8 @@ static struct qeth_card *qeth_alloc_card(struct ccwgroup_device *gdev) CARD_WDEV(card) = gdev->cdev[1]; CARD_DDEV(card) = gdev->cdev[2]; - card->event_wq = alloc_ordered_workqueue("%s", 0, dev_name(&gdev->dev)); + card->event_wq = alloc_ordered_workqueue("%s_event", 0, + dev_name(&gdev->dev)); if (!card->event_wq) goto out_wq; if (qeth_setup_channel(&card->read, true)) @@ -1571,7 +1553,7 @@ int qeth_qdio_clear_card(struct qeth_card *card, int use_halt) switch (atomic_cmpxchg(&card->qdio.state, QETH_QDIO_ESTABLISHED, QETH_QDIO_CLEANING)) { case QETH_QDIO_ESTABLISHED: - if (card->info.type == QETH_CARD_TYPE_IQD) + if (IS_IQD(card)) rc = qdio_shutdown(CARD_DDEV(card), QDIO_FLAG_CLEANUP_USING_HALT); else @@ -1644,8 +1626,8 @@ static void qeth_configure_unitaddr(struct qeth_card *card, char *prcd) card->info.chpid = prcd[30]; card->info.unit_addr2 = prcd[31]; card->info.cula = prcd[63]; - card->info.guestlan = ((prcd[0x10] == _ascebc['V']) && - (prcd[0x11] == _ascebc['M'])); + card->info.is_vm_nic = ((prcd[0x10] == _ascebc['V']) && + (prcd[0x11] == _ascebc['M'])); } static enum qeth_discipline_id qeth_vm_detect_layer(struct qeth_card *card) @@ -1709,13 +1691,11 @@ static enum qeth_discipline_id qeth_enforce_discipline(struct qeth_card *card) { enum qeth_discipline_id disc = QETH_DISCIPLINE_UNDETERMINED; - if (card->info.type == QETH_CARD_TYPE_OSM || - card->info.type == QETH_CARD_TYPE_OSN) + if (IS_OSM(card) || IS_OSN(card)) disc = QETH_DISCIPLINE_LAYER2; - else if (card->info.guestlan) - disc = (card->info.type == QETH_CARD_TYPE_IQD) ? - QETH_DISCIPLINE_LAYER3 : - qeth_vm_detect_layer(card); + else if (IS_VM_NIC(card)) + disc = IS_IQD(card) ? QETH_DISCIPLINE_LAYER3 : + qeth_vm_detect_layer(card); switch (disc) { case QETH_DISCIPLINE_LAYER2: @@ -1771,121 +1751,16 @@ static void qeth_init_func_level(struct qeth_card *card) } } -static int qeth_idx_activate_get_answer(struct qeth_card *card, - struct qeth_channel *channel, - void (*reply_cb)(struct qeth_card *, - struct qeth_channel *, - struct qeth_cmd_buffer *)) -{ - struct qeth_cmd_buffer *iob; - int rc; - - QETH_DBF_TEXT(SETUP, 2, "idxanswr"); - iob = qeth_get_buffer(channel); - if (!iob) - return -ENOMEM; - iob->callback = reply_cb; - qeth_setup_ccw(channel->ccw, CCW_CMD_READ, QETH_BUFSIZE, iob->data); - - wait_event(card->wait_q, - atomic_cmpxchg(&channel->irq_pending, 0, 1) == 0); - QETH_DBF_TEXT(SETUP, 6, "noirqpnd"); - spin_lock_irq(get_ccwdev_lock(channel->ccwdev)); - rc = ccw_device_start_timeout(channel->ccwdev, channel->ccw, - (addr_t) iob, 0, 0, QETH_TIMEOUT); - spin_unlock_irq(get_ccwdev_lock(channel->ccwdev)); - - if (rc) { - QETH_DBF_MESSAGE(2, "Error2 in activating channel rc=%d\n", rc); - QETH_DBF_TEXT_(SETUP, 2, "2err%d", rc); - atomic_set(&channel->irq_pending, 0); - qeth_release_buffer(channel, iob); - wake_up(&card->wait_q); - return rc; - } - rc = wait_event_interruptible_timeout(card->wait_q, - channel->state == CH_STATE_UP, QETH_TIMEOUT); - if (rc == -ERESTARTSYS) - return rc; - if (channel->state != CH_STATE_UP) { - rc = -ETIME; - QETH_DBF_TEXT_(SETUP, 2, "3err%d", rc); - } else - rc = 0; - return rc; -} - -static int qeth_idx_activate_channel(struct qeth_card *card, - struct qeth_channel *channel, - void (*reply_cb)(struct qeth_card *, - struct qeth_channel *, - struct qeth_cmd_buffer *)) +static void qeth_idx_finalize_cmd(struct qeth_card *card, + struct qeth_cmd_buffer *iob, + unsigned int length) { - struct qeth_cmd_buffer *iob; - __u16 temp; - __u8 tmp; - int rc; - struct ccw_dev_id temp_devid; - - QETH_DBF_TEXT(SETUP, 2, "idxactch"); + qeth_setup_ccw(iob->channel->ccw, CCW_CMD_WRITE, length, iob->data); - iob = qeth_get_buffer(channel); - if (!iob) - return -ENOMEM; - iob->callback = reply_cb; - qeth_setup_ccw(channel->ccw, CCW_CMD_WRITE, IDX_ACTIVATE_SIZE, - iob->data); - if (channel == &card->write) { - memcpy(iob->data, IDX_ACTIVATE_WRITE, IDX_ACTIVATE_SIZE); - memcpy(QETH_TRANSPORT_HEADER_SEQ_NO(iob->data), - &card->seqno.trans_hdr, QETH_SEQ_NO_LENGTH); + memcpy(QETH_TRANSPORT_HEADER_SEQ_NO(iob->data), &card->seqno.trans_hdr, + QETH_SEQ_NO_LENGTH); + if (iob->channel == &card->write) card->seqno.trans_hdr++; - } else { - memcpy(iob->data, IDX_ACTIVATE_READ, IDX_ACTIVATE_SIZE); - memcpy(QETH_TRANSPORT_HEADER_SEQ_NO(iob->data), - &card->seqno.trans_hdr, QETH_SEQ_NO_LENGTH); - } - tmp = ((u8)card->dev->dev_port) | 0x80; - memcpy(QETH_IDX_ACT_PNO(iob->data), &tmp, 1); - memcpy(QETH_IDX_ACT_ISSUER_RM_TOKEN(iob->data), - &card->token.issuer_rm_w, QETH_MPC_TOKEN_LENGTH); - memcpy(QETH_IDX_ACT_FUNC_LEVEL(iob->data), - &card->info.func_level, sizeof(__u16)); - ccw_device_get_id(CARD_DDEV(card), &temp_devid); - memcpy(QETH_IDX_ACT_QDIO_DEV_CUA(iob->data), &temp_devid.devno, 2); - temp = (card->info.cula << 8) + card->info.unit_addr2; - memcpy(QETH_IDX_ACT_QDIO_DEV_REALADDR(iob->data), &temp, 2); - - wait_event(card->wait_q, - atomic_cmpxchg(&channel->irq_pending, 0, 1) == 0); - QETH_DBF_TEXT(SETUP, 6, "noirqpnd"); - spin_lock_irq(get_ccwdev_lock(channel->ccwdev)); - rc = ccw_device_start_timeout(channel->ccwdev, channel->ccw, - (addr_t) iob, 0, 0, QETH_TIMEOUT); - spin_unlock_irq(get_ccwdev_lock(channel->ccwdev)); - - if (rc) { - QETH_DBF_MESSAGE(2, "Error1 in activating channel. rc=%d\n", - rc); - QETH_DBF_TEXT_(SETUP, 2, "1err%d", rc); - atomic_set(&channel->irq_pending, 0); - qeth_release_buffer(channel, iob); - wake_up(&card->wait_q); - return rc; - } - rc = wait_event_interruptible_timeout(card->wait_q, - channel->state == CH_STATE_ACTIVATING, QETH_TIMEOUT); - if (rc == -ERESTARTSYS) - return rc; - if (channel->state != CH_STATE_ACTIVATING) { - dev_warn(&channel->ccwdev->dev, "The qeth device driver" - " failed to recover an error on the device\n"); - QETH_DBF_MESSAGE(2, "IDX activate timed out on channel %x\n", - CCW_DEVID(channel->ccwdev)); - QETH_DBF_TEXT_(SETUP, 2, "2err%d", -ETIME); - return -ETIME; - } - return qeth_idx_activate_get_answer(card, channel, reply_cb); } static int qeth_peer_func_level(int level) @@ -1897,112 +1772,21 @@ static int qeth_peer_func_level(int level) return level; } -static void qeth_idx_write_cb(struct qeth_card *card, - struct qeth_channel *channel, - struct qeth_cmd_buffer *iob) -{ - __u16 temp; - - QETH_DBF_TEXT(SETUP , 2, "idxwrcb"); - - if (channel->state == CH_STATE_DOWN) { - channel->state = CH_STATE_ACTIVATING; - goto out; - } - - if (!(QETH_IS_IDX_ACT_POS_REPLY(iob->data))) { - if (QETH_IDX_ACT_CAUSE_CODE(iob->data) == QETH_IDX_ACT_ERR_EXCL) - dev_err(&channel->ccwdev->dev, - "The adapter is used exclusively by another " - "host\n"); - else - QETH_DBF_MESSAGE(2, "IDX_ACTIVATE on channel %x: negative reply\n", - CCW_DEVID(channel->ccwdev)); - goto out; - } - memcpy(&temp, QETH_IDX_ACT_FUNC_LEVEL(iob->data), 2); - if ((temp & ~0x0100) != qeth_peer_func_level(card->info.func_level)) { - QETH_DBF_MESSAGE(2, "IDX_ACTIVATE on channel %x: function level mismatch (sent: %#x, received: %#x)\n", - CCW_DEVID(channel->ccwdev), - card->info.func_level, temp); - goto out; - } - channel->state = CH_STATE_UP; -out: - qeth_release_buffer(channel, iob); -} - -static void qeth_idx_read_cb(struct qeth_card *card, - struct qeth_channel *channel, - struct qeth_cmd_buffer *iob) -{ - __u16 temp; - - QETH_DBF_TEXT(SETUP , 2, "idxrdcb"); - if (channel->state == CH_STATE_DOWN) { - channel->state = CH_STATE_ACTIVATING; - goto out; - } - - if (qeth_check_idx_response(card, iob->data)) - goto out; - - if (!(QETH_IS_IDX_ACT_POS_REPLY(iob->data))) { - switch (QETH_IDX_ACT_CAUSE_CODE(iob->data)) { - case QETH_IDX_ACT_ERR_EXCL: - dev_err(&channel->ccwdev->dev, - "The adapter is used exclusively by another " - "host\n"); - break; - case QETH_IDX_ACT_ERR_AUTH: - case QETH_IDX_ACT_ERR_AUTH_USER: - dev_err(&channel->ccwdev->dev, - "Setting the device online failed because of " - "insufficient authorization\n"); - break; - default: - QETH_DBF_MESSAGE(2, "IDX_ACTIVATE on channel %x: negative reply\n", - CCW_DEVID(channel->ccwdev)); - } - QETH_CARD_TEXT_(card, 2, "idxread%c", - QETH_IDX_ACT_CAUSE_CODE(iob->data)); - goto out; - } - - memcpy(&temp, QETH_IDX_ACT_FUNC_LEVEL(iob->data), 2); - if (temp != qeth_peer_func_level(card->info.func_level)) { - QETH_DBF_MESSAGE(2, "IDX_ACTIVATE on channel %x: function level mismatch (sent: %#x, received: %#x)\n", - CCW_DEVID(channel->ccwdev), - card->info.func_level, temp); - goto out; - } - memcpy(&card->token.issuer_rm_r, - QETH_IDX_ACT_ISSUER_RM_TOKEN(iob->data), - QETH_MPC_TOKEN_LENGTH); - memcpy(&card->info.mcl_level[0], - QETH_IDX_REPLY_LEVEL(iob->data), QETH_MCL_LENGTH); - channel->state = CH_STATE_UP; -out: - qeth_release_buffer(channel, iob); -} - -void qeth_prepare_control_data(struct qeth_card *card, int len, - struct qeth_cmd_buffer *iob) +static void qeth_mpc_finalize_cmd(struct qeth_card *card, + struct qeth_cmd_buffer *iob, + unsigned int length) { - qeth_setup_ccw(iob->channel->ccw, CCW_CMD_WRITE, len, iob->data); - iob->callback = qeth_release_buffer_cb; + qeth_idx_finalize_cmd(card, iob, length); - memcpy(QETH_TRANSPORT_HEADER_SEQ_NO(iob->data), - &card->seqno.trans_hdr, QETH_SEQ_NO_LENGTH); - card->seqno.trans_hdr++; memcpy(QETH_PDU_HEADER_SEQ_NO(iob->data), &card->seqno.pdu_hdr, QETH_SEQ_NO_LENGTH); card->seqno.pdu_hdr++; memcpy(QETH_PDU_HEADER_ACK_SEQ_NO(iob->data), &card->seqno.pdu_hdr_ack, QETH_SEQ_NO_LENGTH); - QETH_DBF_HEX(CTRL, 2, iob->data, min(len, QETH_DBF_CTRL_LEN)); + + iob->reply->seqno = QETH_IDX_COMMAND_SEQNO; + iob->callback = qeth_release_buffer_cb; } -EXPORT_SYMBOL_GPL(qeth_prepare_control_data); /** * qeth_send_control_data() - send control command to the card @@ -2035,17 +1819,12 @@ static int qeth_send_control_data(struct qeth_card *card, int len, void *reply_param) { struct qeth_channel *channel = iob->channel; + long timeout = iob->timeout; int rc; struct qeth_reply *reply = NULL; - unsigned long timeout, event_timeout; - struct qeth_ipa_cmd *cmd = NULL; QETH_CARD_TEXT(card, 2, "sendctl"); - if (card->read_or_write_problem) { - qeth_release_buffer(channel, iob); - return -EIO; - } reply = qeth_alloc_reply(card); if (!reply) { qeth_release_buffer(channel, iob); @@ -2058,27 +1837,24 @@ static int qeth_send_control_data(struct qeth_card *card, int len, qeth_get_reply(reply); iob->reply = reply; - while (atomic_cmpxchg(&channel->irq_pending, 0, 1)) ; - - if (IS_IPA(iob->data)) { - cmd = __ipa_cmd(iob); - cmd->hdr.seqno = card->seqno.ipa++; - reply->seqno = cmd->hdr.seqno; - event_timeout = QETH_IPA_TIMEOUT; - } else { - reply->seqno = QETH_IDX_COMMAND_SEQNO; - event_timeout = QETH_TIMEOUT; + timeout = wait_event_interruptible_timeout(card->wait_q, + qeth_trylock_channel(channel), + timeout); + if (timeout <= 0) { + qeth_put_reply(reply); + qeth_release_buffer(channel, iob); + return (timeout == -ERESTARTSYS) ? -EINTR : -ETIME; } - qeth_prepare_control_data(card, len, iob); - qeth_enqueue_reply(card, reply); + iob->finalize(card, iob, len); + QETH_DBF_HEX(CTRL, 2, iob->data, min(len, QETH_DBF_CTRL_LEN)); - timeout = jiffies + event_timeout; + qeth_enqueue_reply(card, reply); QETH_CARD_TEXT(card, 6, "noirqpnd"); spin_lock_irq(get_ccwdev_lock(channel->ccwdev)); rc = ccw_device_start_timeout(channel->ccwdev, channel->ccw, - (addr_t) iob, 0, 0, event_timeout); + (addr_t) iob, 0, 0, timeout); spin_unlock_irq(get_ccwdev_lock(channel->ccwdev)); if (rc) { QETH_DBF_MESSAGE(2, "qeth_send_control_data on device %x: ccw_device_start rc = %i\n", @@ -2092,30 +1868,211 @@ static int qeth_send_control_data(struct qeth_card *card, int len, return rc; } - /* we have only one long running ipassist, since we can ensure - process context of this command we can sleep */ - if (cmd && cmd->hdr.command == IPA_CMD_SETIP && - cmd->hdr.prot_version == QETH_PROT_IPV4) { - if (!wait_event_timeout(reply->wait_q, - atomic_read(&reply->received), event_timeout)) - goto time_err; - } else { - while (!atomic_read(&reply->received)) { - if (time_after(jiffies, timeout)) - goto time_err; - cpu_relax(); - } - } + timeout = wait_for_completion_interruptible_timeout(&reply->received, + timeout); + if (timeout <= 0) + rc = (timeout == -ERESTARTSYS) ? -EINTR : -ETIME; qeth_dequeue_reply(card, reply); - rc = reply->rc; + if (!rc) + rc = reply->rc; qeth_put_reply(reply); return rc; +} -time_err: - qeth_dequeue_reply(card, reply); - qeth_put_reply(reply); - return -ETIME; +static int qeth_idx_check_activate_response(struct qeth_card *card, + struct qeth_channel *channel, + struct qeth_cmd_buffer *iob) +{ + int rc; + + rc = qeth_check_idx_response(card, iob->data); + if (rc) + return rc; + + if (QETH_IS_IDX_ACT_POS_REPLY(iob->data)) + return 0; + + /* negative reply: */ + QETH_DBF_TEXT_(SETUP, 2, "idxneg%c", + QETH_IDX_ACT_CAUSE_CODE(iob->data)); + + switch (QETH_IDX_ACT_CAUSE_CODE(iob->data)) { + case QETH_IDX_ACT_ERR_EXCL: + dev_err(&channel->ccwdev->dev, + "The adapter is used exclusively by another host\n"); + return -EBUSY; + case QETH_IDX_ACT_ERR_AUTH: + case QETH_IDX_ACT_ERR_AUTH_USER: + dev_err(&channel->ccwdev->dev, + "Setting the device online failed because of insufficient authorization\n"); + return -EPERM; + default: + QETH_DBF_MESSAGE(2, "IDX_ACTIVATE on channel %x: negative reply\n", + CCW_DEVID(channel->ccwdev)); + return -EIO; + } +} + +static void qeth_idx_query_read_cb(struct qeth_card *card, + struct qeth_channel *channel, + struct qeth_cmd_buffer *iob) +{ + u16 peer_level; + int rc; + + QETH_DBF_TEXT(SETUP, 2, "idxrdcb"); + + rc = qeth_idx_check_activate_response(card, channel, iob); + if (rc) + goto out; + + memcpy(&peer_level, QETH_IDX_ACT_FUNC_LEVEL(iob->data), 2); + if (peer_level != qeth_peer_func_level(card->info.func_level)) { + QETH_DBF_MESSAGE(2, "IDX_ACTIVATE on channel %x: function level mismatch (sent: %#x, received: %#x)\n", + CCW_DEVID(channel->ccwdev), + card->info.func_level, peer_level); + rc = -EINVAL; + goto out; + } + + memcpy(&card->token.issuer_rm_r, + QETH_IDX_ACT_ISSUER_RM_TOKEN(iob->data), + QETH_MPC_TOKEN_LENGTH); + memcpy(&card->info.mcl_level[0], + QETH_IDX_REPLY_LEVEL(iob->data), QETH_MCL_LENGTH); + +out: + qeth_notify_reply(iob->reply, rc); + qeth_release_buffer(channel, iob); +} + +static void qeth_idx_query_write_cb(struct qeth_card *card, + struct qeth_channel *channel, + struct qeth_cmd_buffer *iob) +{ + u16 peer_level; + int rc; + + QETH_DBF_TEXT(SETUP, 2, "idxwrcb"); + + rc = qeth_idx_check_activate_response(card, channel, iob); + if (rc) + goto out; + + memcpy(&peer_level, QETH_IDX_ACT_FUNC_LEVEL(iob->data), 2); + if ((peer_level & ~0x0100) != + qeth_peer_func_level(card->info.func_level)) { + QETH_DBF_MESSAGE(2, "IDX_ACTIVATE on channel %x: function level mismatch (sent: %#x, received: %#x)\n", + CCW_DEVID(channel->ccwdev), + card->info.func_level, peer_level); + rc = -EINVAL; + } + +out: + qeth_notify_reply(iob->reply, rc); + qeth_release_buffer(channel, iob); +} + +static void qeth_idx_finalize_query_cmd(struct qeth_card *card, + struct qeth_cmd_buffer *iob, + unsigned int length) +{ + qeth_setup_ccw(iob->channel->ccw, CCW_CMD_READ, length, iob->data); +} + +static void qeth_idx_activate_cb(struct qeth_card *card, + struct qeth_channel *channel, + struct qeth_cmd_buffer *iob) +{ + qeth_notify_reply(iob->reply, 0); + qeth_release_buffer(channel, iob); +} + +static void qeth_idx_setup_activate_cmd(struct qeth_card *card, + struct qeth_cmd_buffer *iob) +{ + u16 addr = (card->info.cula << 8) + card->info.unit_addr2; + u8 port = ((u8)card->dev->dev_port) | 0x80; + struct ccw_dev_id dev_id; + + ccw_device_get_id(CARD_DDEV(card), &dev_id); + iob->finalize = qeth_idx_finalize_cmd; + iob->callback = qeth_idx_activate_cb; + + memcpy(QETH_IDX_ACT_PNO(iob->data), &port, 1); + memcpy(QETH_IDX_ACT_ISSUER_RM_TOKEN(iob->data), + &card->token.issuer_rm_w, QETH_MPC_TOKEN_LENGTH); + memcpy(QETH_IDX_ACT_FUNC_LEVEL(iob->data), + &card->info.func_level, 2); + memcpy(QETH_IDX_ACT_QDIO_DEV_CUA(iob->data), &dev_id.devno, 2); + memcpy(QETH_IDX_ACT_QDIO_DEV_REALADDR(iob->data), &addr, 2); +} + +static int qeth_idx_activate_read_channel(struct qeth_card *card) +{ + struct qeth_channel *channel = &card->read; + struct qeth_cmd_buffer *iob; + int rc; + + QETH_DBF_TEXT(SETUP, 2, "idxread"); + + iob = qeth_get_buffer(channel); + if (!iob) + return -ENOMEM; + + memcpy(iob->data, IDX_ACTIVATE_READ, IDX_ACTIVATE_SIZE); + qeth_idx_setup_activate_cmd(card, iob); + + rc = qeth_send_control_data(card, IDX_ACTIVATE_SIZE, iob, NULL, NULL); + if (rc) + return rc; + + iob = qeth_get_buffer(channel); + if (!iob) + return -ENOMEM; + + iob->finalize = qeth_idx_finalize_query_cmd; + iob->callback = qeth_idx_query_read_cb; + rc = qeth_send_control_data(card, QETH_BUFSIZE, iob, NULL, NULL); + if (rc) + return rc; + + channel->state = CH_STATE_UP; + return 0; +} + +static int qeth_idx_activate_write_channel(struct qeth_card *card) +{ + struct qeth_channel *channel = &card->write; + struct qeth_cmd_buffer *iob; + int rc; + + QETH_DBF_TEXT(SETUP, 2, "idxwrite"); + + iob = qeth_get_buffer(channel); + if (!iob) + return -ENOMEM; + + memcpy(iob->data, IDX_ACTIVATE_WRITE, IDX_ACTIVATE_SIZE); + qeth_idx_setup_activate_cmd(card, iob); + + rc = qeth_send_control_data(card, IDX_ACTIVATE_SIZE, iob, NULL, NULL); + if (rc) + return rc; + + iob = qeth_get_buffer(channel); + if (!iob) + return -ENOMEM; + + iob->finalize = qeth_idx_finalize_query_cmd; + iob->callback = qeth_idx_query_write_cb; + rc = qeth_send_control_data(card, QETH_BUFSIZE, iob, NULL, NULL); + if (rc) + return rc; + + channel->state = CH_STATE_UP; + return 0; } static int qeth_cm_enable_cb(struct qeth_card *card, struct qeth_reply *reply, @@ -2140,7 +2097,9 @@ static int qeth_cm_enable(struct qeth_card *card) QETH_DBF_TEXT(SETUP, 2, "cmenable"); iob = qeth_wait_for_buffer(&card->write); + iob->finalize = qeth_mpc_finalize_cmd; memcpy(iob->data, CM_ENABLE, CM_ENABLE_SIZE); + memcpy(QETH_CM_ENABLE_ISSUER_RM_TOKEN(iob->data), &card->token.issuer_rm_r, QETH_MPC_TOKEN_LENGTH); memcpy(QETH_CM_ENABLE_FILTER_TOKEN(iob->data), @@ -2173,7 +2132,9 @@ static int qeth_cm_setup(struct qeth_card *card) QETH_DBF_TEXT(SETUP, 2, "cmsetup"); iob = qeth_wait_for_buffer(&card->write); + iob->finalize = qeth_mpc_finalize_cmd; memcpy(iob->data, CM_SETUP, CM_SETUP_SIZE); + memcpy(QETH_CM_SETUP_DEST_ADDR(iob->data), &card->token.issuer_rm_r, QETH_MPC_TOKEN_LENGTH); memcpy(QETH_CM_SETUP_CONNECTION_TOKEN(iob->data), @@ -2206,7 +2167,7 @@ static int qeth_update_max_mtu(struct qeth_card *card, unsigned int max_mtu) /* adjust RX buffer size to new max MTU: */ card->qdio.in_buf_size = max_mtu + 2 * PAGE_SIZE; if (dev->max_mtu && dev->max_mtu != max_mtu) - qeth_free_qdio_buffers(card); + qeth_free_qdio_queues(card); } else { if (dev->mtu) new_mtu = dev->mtu; @@ -2253,7 +2214,7 @@ static int qeth_ulp_enable_cb(struct qeth_card *card, struct qeth_reply *reply, memcpy(&card->token.ulp_filter_r, QETH_ULP_ENABLE_RESP_FILTER_TOKEN(iob->data), QETH_MPC_TOKEN_LENGTH); - if (card->info.type == QETH_CARD_TYPE_IQD) { + if (IS_IQD(card)) { memcpy(&framesize, QETH_ULP_ENABLE_RESP_MAX_MTU(iob->data), 2); mtu = qeth_get_mtu_outof_framesize(framesize); } else { @@ -2290,6 +2251,7 @@ static int qeth_ulp_enable(struct qeth_card *card) QETH_DBF_TEXT(SETUP, 2, "ulpenabl"); iob = qeth_wait_for_buffer(&card->write); + iob->finalize = qeth_mpc_finalize_cmd; memcpy(iob->data, ULP_ENABLE, ULP_ENABLE_SIZE); *(QETH_ULP_ENABLE_LINKNUM(iob->data)) = (u8) card->dev->dev_port; @@ -2336,6 +2298,7 @@ static int qeth_ulp_setup(struct qeth_card *card) QETH_DBF_TEXT(SETUP, 2, "ulpsetup"); iob = qeth_wait_for_buffer(&card->write); + iob->finalize = qeth_mpc_finalize_cmd; memcpy(iob->data, ULP_SETUP, ULP_SETUP_SIZE); memcpy(QETH_ULP_SETUP_DEST_ADDR(iob->data), @@ -2377,12 +2340,12 @@ static void qeth_free_output_queue(struct qeth_qdio_out_q *q) if (!q) return; - qeth_clear_outq_buffers(q, 1); + qeth_drain_output_queue(q, true); qdio_free_buffers(q->qdio_bufs, QDIO_MAX_BUFFERS_PER_Q); kfree(q); } -static struct qeth_qdio_out_q *qeth_alloc_qdio_out_buf(void) +static struct qeth_qdio_out_q *qeth_alloc_output_queue(void) { struct qeth_qdio_out_q *q = kzalloc(sizeof(*q), GFP_KERNEL); @@ -2396,7 +2359,7 @@ static struct qeth_qdio_out_q *qeth_alloc_qdio_out_buf(void) return q; } -static int qeth_alloc_qdio_buffers(struct qeth_card *card) +static int qeth_alloc_qdio_queues(struct qeth_card *card) { int i, j; @@ -2417,11 +2380,12 @@ static int qeth_alloc_qdio_buffers(struct qeth_card *card) /* outbound */ for (i = 0; i < card->qdio.no_out_queues; ++i) { - card->qdio.out_qs[i] = qeth_alloc_qdio_out_buf(); + card->qdio.out_qs[i] = qeth_alloc_output_queue(); if (!card->qdio.out_qs[i]) goto out_freeoutq; QETH_DBF_TEXT_(SETUP, 2, "outq %i", i); QETH_DBF_HEX(SETUP, 2, &card->qdio.out_qs[i], sizeof(void *)); + card->qdio.out_qs[i]->card = card; card->qdio.out_qs[i]->queue_no = i; /* give outbound qeth_qdio_buffers their qdio_buffers */ for (j = 0; j < QDIO_MAX_BUFFERS_PER_Q; ++j) { @@ -2458,7 +2422,7 @@ out_nomem: return -ENOMEM; } -static void qeth_free_qdio_buffers(struct qeth_card *card) +static void qeth_free_qdio_queues(struct qeth_card *card) { int i, j; @@ -2523,6 +2487,7 @@ static int qeth_dm_act(struct qeth_card *card) QETH_DBF_TEXT(SETUP, 2, "dmact"); iob = qeth_wait_for_buffer(&card->write); + iob->finalize = qeth_mpc_finalize_cmd; memcpy(iob->data, DM_ACT, DM_ACT_SIZE); memcpy(QETH_DM_ACT_DEST_ADDR(iob->data), @@ -2564,7 +2529,7 @@ static int qeth_mpc_initialize(struct qeth_card *card) QETH_DBF_TEXT_(SETUP, 2, "5err%d", rc); goto out_qdio; } - rc = qeth_alloc_qdio_buffers(card); + rc = qeth_alloc_qdio_queues(card); if (rc) { QETH_DBF_TEXT_(SETUP, 2, "5err%d", rc); goto out_qdio; @@ -2572,7 +2537,7 @@ static int qeth_mpc_initialize(struct qeth_card *card) rc = qeth_qdio_establish(card); if (rc) { QETH_DBF_TEXT_(SETUP, 2, "6err%d", rc); - qeth_free_qdio_buffers(card); + qeth_free_qdio_queues(card); goto out_qdio; } rc = qeth_qdio_activate(card); @@ -2588,7 +2553,7 @@ static int qeth_mpc_initialize(struct qeth_card *card) return 0; out_qdio: - qeth_qdio_clear_card(card, card->info.type != QETH_CARD_TYPE_IQD); + qeth_qdio_clear_card(card, !IS_IQD(card)); qdio_free(CARD_DDEV(card)); return rc; } @@ -2611,8 +2576,7 @@ void qeth_print_status_message(struct qeth_card *card) } /* fallthrough */ case QETH_CARD_TYPE_IQD: - if ((card->info.guestlan) || - (card->info.mcl_level[0] & 0x80)) { + if (IS_VM_NIC(card) || (card->info.mcl_level[0] & 0x80)) { card->info.mcl_level[0] = (char) _ebcasc[(__u8) card->info.mcl_level[0]]; card->info.mcl_level[1] = (char) _ebcasc[(__u8) @@ -2733,7 +2697,7 @@ static int qeth_init_input_buffer(struct qeth_card *card, int qeth_init_qdio_queues(struct qeth_card *card) { - int i, j; + unsigned int i; int rc; QETH_DBF_TEXT(SETUP, 2, "initqdqs"); @@ -2762,19 +2726,15 @@ int qeth_init_qdio_queues(struct qeth_card *card) /* outbound queue */ for (i = 0; i < card->qdio.no_out_queues; ++i) { - qdio_reset_buffers(card->qdio.out_qs[i]->qdio_bufs, - QDIO_MAX_BUFFERS_PER_Q); - for (j = 0; j < QDIO_MAX_BUFFERS_PER_Q; ++j) { - qeth_clear_output_buffer(card->qdio.out_qs[i], - card->qdio.out_qs[i]->bufs[j]); - } - card->qdio.out_qs[i]->card = card; - card->qdio.out_qs[i]->next_buf_to_fill = 0; - card->qdio.out_qs[i]->do_pack = 0; - atomic_set(&card->qdio.out_qs[i]->used_buffers, 0); - atomic_set(&card->qdio.out_qs[i]->set_pci_flags_count, 0); - atomic_set(&card->qdio.out_qs[i]->state, - QETH_OUT_Q_UNLOCKED); + struct qeth_qdio_out_q *queue = card->qdio.out_qs[i]; + + qdio_reset_buffers(queue->qdio_bufs, QDIO_MAX_BUFFERS_PER_Q); + queue->max_elements = QETH_MAX_BUFFER_ELEMENTS(card); + queue->next_buf_to_fill = 0; + queue->do_pack = 0; + atomic_set(&queue->used_buffers, 0); + atomic_set(&queue->set_pci_flags_count, 0); + atomic_set(&queue->state, QETH_OUT_Q_UNLOCKED); } return 0; } @@ -2805,12 +2765,26 @@ static void qeth_fill_ipacmd_header(struct qeth_card *card, cmd->hdr.prot_version = prot; } +static void qeth_ipa_finalize_cmd(struct qeth_card *card, + struct qeth_cmd_buffer *iob, + unsigned int length) +{ + qeth_mpc_finalize_cmd(card, iob, length); + + /* override with IPA-specific values: */ + __ipa_cmd(iob)->hdr.seqno = card->seqno.ipa; + iob->reply->seqno = card->seqno.ipa++; +} + void qeth_prepare_ipa_cmd(struct qeth_card *card, struct qeth_cmd_buffer *iob, u16 cmd_length) { u16 total_length = IPA_PDU_HEADER_SIZE + cmd_length; u8 prot_type = qeth_mpc_select_prot_type(card); + iob->finalize = qeth_ipa_finalize_cmd; + iob->timeout = QETH_IPA_TIMEOUT; + memcpy(iob->data, IPA_PDU_HEADER, IPA_PDU_HEADER_SIZE); memcpy(QETH_IPA_PDU_LEN_TOTAL(iob->data), &total_length, 2); memcpy(QETH_IPA_CMD_PROT_TYPE(iob->data), &prot_type, 1); @@ -2866,6 +2840,11 @@ int qeth_send_ipa_cmd(struct qeth_card *card, struct qeth_cmd_buffer *iob, QETH_CARD_TEXT(card, 4, "sendipa"); + if (card->read_or_write_problem) { + qeth_release_buffer(iob->channel, iob); + return -EIO; + } + if (reply_cb == NULL) reply_cb = qeth_send_ipa_cmd_cb; memcpy(&length, QETH_IPA_PDU_LEN_TOTAL(iob->data), 2); @@ -3251,7 +3230,7 @@ static void qeth_handle_send_error(struct qeth_card *card, int sbalf15 = buffer->buffer->element[15].sflags; QETH_CARD_TEXT(card, 6, "hdsnderr"); - if (card->info.type == QETH_CARD_TYPE_IQD) { + if (IS_IQD(card)) { if (sbalf15 == 0) { qdio_err = 0; } else { @@ -3348,7 +3327,7 @@ static void qeth_flush_buffers(struct qeth_qdio_out_q *queue, int index, if (queue->bufstates) queue->bufstates[bidx].user = buf; - if (queue->card->info.type == QETH_CARD_TYPE_IQD) + if (IS_IQD(queue->card)) continue; if (!queue->do_pack) { @@ -3378,11 +3357,9 @@ static void qeth_flush_buffers(struct qeth_qdio_out_q *queue, int index, } QETH_TXQ_STAT_ADD(queue, bufs, count); - netif_trans_update(queue->card->dev); qdio_flags = QDIO_FLAG_SYNC_OUTPUT; if (atomic_read(&queue->set_pci_flags_count)) qdio_flags |= QDIO_FLAG_PCI_OUT; - atomic_add(count, &queue->used_buffers); rc = do_QDIO(CARD_DDEV(queue->card), qdio_flags, queue->queue_no, index, count); if (rc) { @@ -3422,7 +3399,6 @@ static void qeth_check_outbound_queue(struct qeth_qdio_out_q *queue) * do_send_packet. So, we check if there is a * packing buffer to be flushed here. */ - netif_stop_queue(queue->card->dev); index = queue->next_buf_to_fill; q_was_packing = queue->do_pack; /* queue->do_pack may change */ @@ -3467,7 +3443,7 @@ int qeth_configure_cq(struct qeth_card *card, enum qeth_cq cq) goto out; } - qeth_free_qdio_buffers(card); + qeth_free_qdio_queues(card); card->options.cq = cq; rc = 0; } @@ -3493,7 +3469,7 @@ static void qeth_qdio_cq_handler(struct qeth_card *card, unsigned int qdio_err, QETH_CARD_TEXT_(card, 5, "qcqherr%d", qdio_err); if (qdio_err) { - netif_stop_queue(card->dev); + netif_tx_stop_all_queues(card->dev); qeth_schedule_recovery(card); return; } @@ -3549,12 +3525,14 @@ static void qeth_qdio_output_handler(struct ccw_device *ccwdev, struct qeth_card *card = (struct qeth_card *) card_ptr; struct qeth_qdio_out_q *queue = card->qdio.out_qs[__queue]; struct qeth_qdio_out_buffer *buffer; + struct net_device *dev = card->dev; + struct netdev_queue *txq; int i; QETH_CARD_TEXT(card, 6, "qdouhdl"); if (qdio_error & QDIO_ERROR_FATAL) { QETH_CARD_TEXT(card, 2, "achkcond"); - netif_stop_queue(card->dev); + netif_tx_stop_all_queues(dev); qeth_schedule_recovery(card); return; } @@ -3580,7 +3558,7 @@ static void qeth_qdio_output_handler(struct ccw_device *ccwdev, /* prepare the queue slot for re-use: */ qeth_scrub_qdio_buffer(buffer->buffer, - QETH_MAX_BUFFER_ELEMENTS(card)); + queue->max_elements); if (qeth_init_qdio_out_buf(queue, bidx)) { QETH_CARD_TEXT(card, 2, "outofbuf"); qeth_schedule_recovery(card); @@ -3600,33 +3578,32 @@ static void qeth_qdio_output_handler(struct ccw_device *ccwdev, } atomic_sub(count, &queue->used_buffers); /* check if we need to do something on this outbound queue */ - if (card->info.type != QETH_CARD_TYPE_IQD) + if (!IS_IQD(card)) qeth_check_outbound_queue(queue); - netif_wake_queue(queue->card->dev); -} - -/* We cannot use outbound queue 3 for unicast packets on HiperSockets */ -static inline int qeth_cut_iqd_prio(struct qeth_card *card, int queue_num) -{ - if ((card->info.type == QETH_CARD_TYPE_IQD) && (queue_num == 3)) - return 2; - return queue_num; + if (IS_IQD(card)) + __queue = qeth_iqd_translate_txq(dev, __queue); + txq = netdev_get_tx_queue(dev, __queue); + /* xmit may have observed the full-condition, but not yet stopped the + * txq. In which case the code below won't trigger. So before returning, + * xmit will re-check the txq's fill level and wake it up if needed. + */ + if (netif_tx_queue_stopped(txq) && !qeth_out_queue_is_full(queue)) + netif_tx_wake_queue(txq); } /** * Note: Function assumes that we have 4 outbound queues. */ -int qeth_get_priority_queue(struct qeth_card *card, struct sk_buff *skb, - int ipv) +int qeth_get_priority_queue(struct qeth_card *card, struct sk_buff *skb) { - __be16 *tci; + struct vlan_ethhdr *veth = vlan_eth_hdr(skb); u8 tos; switch (card->qdio.do_prio_queueing) { case QETH_PRIO_Q_ING_TOS: case QETH_PRIO_Q_ING_PREC: - switch (ipv) { + switch (qeth_get_ip_version(skb)) { case 4: tos = ipv4_get_dsfield(ip_hdr(skb)); break; @@ -3637,9 +3614,9 @@ int qeth_get_priority_queue(struct qeth_card *card, struct sk_buff *skb, return card->qdio.default_out_queue; } if (card->qdio.do_prio_queueing == QETH_PRIO_Q_ING_PREC) - return qeth_cut_iqd_prio(card, ~tos >> 6 & 3); + return ~tos >> 6 & 3; if (tos & IPTOS_MINCOST) - return qeth_cut_iqd_prio(card, 3); + return 3; if (tos & IPTOS_RELIABILITY) return 2; if (tos & IPTOS_THROUGHPUT) @@ -3650,12 +3627,11 @@ int qeth_get_priority_queue(struct qeth_card *card, struct sk_buff *skb, case QETH_PRIO_Q_ING_SKB: if (skb->priority > 5) return 0; - return qeth_cut_iqd_prio(card, ~skb->priority >> 1 & 3); + return ~skb->priority >> 1 & 3; case QETH_PRIO_Q_ING_VLAN: - tci = &((struct ethhdr *)skb->data)->h_proto; - if (be16_to_cpu(*tci) == ETH_P_8021Q) - return qeth_cut_iqd_prio(card, - ~be16_to_cpu(*(tci + 1)) >> (VLAN_PRIO_SHIFT + 1) & 3); + if (veth->h_vlan_proto == htons(ETH_P_8021Q)) + return ~ntohs(veth->h_vlan_TCI) >> + (VLAN_PRIO_SHIFT + 1) & 3; break; default: break; @@ -3729,8 +3705,8 @@ static int qeth_add_hw_header(struct qeth_qdio_out_q *queue, unsigned int hdr_len, unsigned int proto_len, unsigned int *elements) { - const unsigned int max_elements = QETH_MAX_BUFFER_ELEMENTS(queue->card); const unsigned int contiguous = proto_len ? proto_len : 1; + const unsigned int max_elements = queue->max_elements; unsigned int __elements; addr_t start, end; bool push_ok; @@ -3867,11 +3843,13 @@ static void __qeth_fill_buffer(struct sk_buff *skb, * from qeth_core_header_cache. * @offset: when mapping the skb, start at skb->data + offset * @hd_len: if > 0, build a dedicated header element of this size + * flush: Prepare the buffer to be flushed, regardless of its fill level. */ static int qeth_fill_buffer(struct qeth_qdio_out_q *queue, struct qeth_qdio_out_buffer *buf, struct sk_buff *skb, struct qeth_hdr *hdr, - unsigned int offset, unsigned int hd_len) + unsigned int offset, unsigned int hd_len, + bool flush) { struct qdio_buffer *buffer = buf->buffer; bool is_first_elem = true; @@ -3900,8 +3878,8 @@ static int qeth_fill_buffer(struct qeth_qdio_out_q *queue, QETH_TXQ_STAT_INC(queue, skbs_pack); /* If the buffer still has free elements, keep using it. */ - if (buf->next_element_to_fill < - QETH_MAX_BUFFER_ELEMENTS(queue->card)) + if (!flush && + buf->next_element_to_fill < queue->max_elements) return 0; } @@ -3918,15 +3896,31 @@ static int qeth_do_send_packet_fast(struct qeth_qdio_out_q *queue, { int index = queue->next_buf_to_fill; struct qeth_qdio_out_buffer *buffer = queue->bufs[index]; + struct netdev_queue *txq; + bool stopped = false; - /* - * check if buffer is empty to make sure that we do not 'overtake' - * ourselves and try to fill a buffer that is already primed + /* Just a sanity check, the wake/stop logic should ensure that we always + * get a free buffer. */ if (atomic_read(&buffer->state) != QETH_QDIO_BUF_EMPTY) return -EBUSY; - qeth_fill_buffer(queue, buffer, skb, hdr, offset, hd_len); + + txq = netdev_get_tx_queue(queue->card->dev, skb_get_queue_mapping(skb)); + + if (atomic_inc_return(&queue->used_buffers) >= QDIO_MAX_BUFFERS_PER_Q) { + /* If a TX completion happens right _here_ and misses to wake + * the txq, then our re-check below will catch the race. + */ + QETH_TXQ_STAT_INC(queue, stopped); + netif_tx_stop_queue(txq); + stopped = true; + } + + qeth_fill_buffer(queue, buffer, skb, hdr, offset, hd_len, stopped); qeth_flush_buffers(queue, index, 1); + + if (stopped && !qeth_out_queue_is_full(queue)) + netif_tx_start_queue(txq); return 0; } @@ -3936,6 +3930,8 @@ int qeth_do_send_packet(struct qeth_card *card, struct qeth_qdio_out_q *queue, int elements_needed) { struct qeth_qdio_out_buffer *buffer; + struct netdev_queue *txq; + bool stopped = false; int start_index; int flush_count = 0; int do_pack = 0; @@ -3947,21 +3943,24 @@ int qeth_do_send_packet(struct qeth_card *card, struct qeth_qdio_out_q *queue, QETH_OUT_Q_LOCKED) != QETH_OUT_Q_UNLOCKED); start_index = queue->next_buf_to_fill; buffer = queue->bufs[queue->next_buf_to_fill]; - /* - * check if buffer is empty to make sure that we do not 'overtake' - * ourselves and try to fill a buffer that is already primed + + /* Just a sanity check, the wake/stop logic should ensure that we always + * get a free buffer. */ if (atomic_read(&buffer->state) != QETH_QDIO_BUF_EMPTY) { atomic_set(&queue->state, QETH_OUT_Q_UNLOCKED); return -EBUSY; } + + txq = netdev_get_tx_queue(card->dev, skb_get_queue_mapping(skb)); + /* check if we need to switch packing state of this queue */ qeth_switch_to_packing_if_needed(queue); if (queue->do_pack) { do_pack = 1; /* does packet fit in current buffer? */ - if ((QETH_MAX_BUFFER_ELEMENTS(card) - - buffer->next_element_to_fill) < elements_needed) { + if (buffer->next_element_to_fill + elements_needed > + queue->max_elements) { /* ... no -> set state PRIMED */ atomic_set(&buffer->state, QETH_QDIO_BUF_PRIMED); flush_count++; @@ -3969,8 +3968,8 @@ int qeth_do_send_packet(struct qeth_card *card, struct qeth_qdio_out_q *queue, (queue->next_buf_to_fill + 1) % QDIO_MAX_BUFFERS_PER_Q; buffer = queue->bufs[queue->next_buf_to_fill]; - /* we did a step forward, so check buffer state - * again */ + + /* We stepped forward, so sanity-check again: */ if (atomic_read(&buffer->state) != QETH_QDIO_BUF_EMPTY) { qeth_flush_buffers(queue, start_index, @@ -3983,8 +3982,18 @@ int qeth_do_send_packet(struct qeth_card *card, struct qeth_qdio_out_q *queue, } } - flush_count += qeth_fill_buffer(queue, buffer, skb, hdr, offset, - hd_len); + if (buffer->next_element_to_fill == 0 && + atomic_inc_return(&queue->used_buffers) >= QDIO_MAX_BUFFERS_PER_Q) { + /* If a TX completion happens right _here_ and misses to wake + * the txq, then our re-check below will catch the race. + */ + QETH_TXQ_STAT_INC(queue, stopped); + netif_tx_stop_queue(txq); + stopped = true; + } + + flush_count += qeth_fill_buffer(queue, buffer, skb, hdr, offset, hd_len, + stopped); if (flush_count) qeth_flush_buffers(queue, start_index, flush_count); else if (!atomic_read(&queue->set_pci_flags_count)) @@ -4015,6 +4024,8 @@ out: if (do_pack) QETH_TXQ_STAT_ADD(queue, bufs_pack, flush_count); + if (stopped && !qeth_out_queue_is_full(queue)) + netif_tx_start_queue(txq); return rc; } EXPORT_SYMBOL_GPL(qeth_do_send_packet); @@ -4101,9 +4112,6 @@ int qeth_xmit(struct qeth_card *card, struct sk_buff *skb, } else { if (!push_len) kmem_cache_free(qeth_core_header_cache, hdr); - if (rc == -EBUSY) - /* roll back to ETH header */ - skb_pull(skb, push_len); } return rc; } @@ -4321,9 +4329,8 @@ int qeth_set_access_ctrl_online(struct qeth_card *card, int fallback) QETH_CARD_TEXT(card, 4, "setactlo"); - if ((card->info.type == QETH_CARD_TYPE_OSD || - card->info.type == QETH_CARD_TYPE_OSX) && - qeth_adp_supported(card, IPA_SETADP_SET_ACCESS_CONTROL)) { + if ((IS_OSD(card) || IS_OSX(card)) && + qeth_adp_supported(card, IPA_SETADP_SET_ACCESS_CONTROL)) { rc = qeth_setadpparms_set_access_ctrl(card, card->options.isolation, fallback); if (rc) { @@ -4348,7 +4355,6 @@ void qeth_tx_timeout(struct net_device *dev) card = dev->ml_priv; QETH_CARD_TEXT(card, 4, "txtimeo"); - QETH_CARD_STAT_INC(card, tx_errors); qeth_schedule_recovery(card); } EXPORT_SYMBOL_GPL(qeth_tx_timeout); @@ -4489,7 +4495,7 @@ static int qeth_snmp_command(struct qeth_card *card, char __user *udata) QETH_CARD_TEXT(card, 3, "snmpcmd"); - if (card->info.guestlan) + if (IS_VM_NIC(card)) return -EOPNOTSUPP; if ((!qeth_adp_supported(card, IPA_SETADP_SET_SNMP_CONTROL)) && @@ -4732,14 +4738,6 @@ out: } EXPORT_SYMBOL_GPL(qeth_vm_request_mac); -static int qeth_get_qdio_q_format(struct qeth_card *card) -{ - if (card->info.type == QETH_CARD_TYPE_IQD) - return QDIO_IQDIO_QFMT; - else - return QDIO_QETH_QFMT; -} - static void qeth_determine_capabilities(struct qeth_card *card) { int rc; @@ -4878,7 +4876,8 @@ static int qeth_qdio_establish(struct qeth_card *card) memset(&init_data, 0, sizeof(struct qdio_initialize)); init_data.cdev = CARD_DDEV(card); - init_data.q_format = qeth_get_qdio_q_format(card); + init_data.q_format = IS_IQD(card) ? QDIO_IQDIO_QFMT : + QDIO_QETH_QFMT; init_data.qib_param_field_format = 0; init_data.qib_param_field = qib_param_field; init_data.no_input_qs = card->qdio.no_in_queues; @@ -4890,8 +4889,7 @@ static int qeth_qdio_establish(struct qeth_card *card) init_data.input_sbal_addr_array = in_sbal_ptrs; init_data.output_sbal_addr_array = out_sbal_ptrs; init_data.output_sbal_state_array = card->qdio.out_bufstates; - init_data.scan_threshold = - (card->info.type == QETH_CARD_TYPE_IQD) ? 1 : 32; + init_data.scan_threshold = IS_IQD(card) ? 1 : 32; if (atomic_cmpxchg(&card->qdio.state, QETH_QDIO_ALLOCATED, QETH_QDIO_ESTABLISHED) == QETH_QDIO_ALLOCATED) { @@ -4937,7 +4935,7 @@ static void qeth_core_free_card(struct qeth_card *card) qeth_clean_channel(&card->write); qeth_clean_channel(&card->data); destroy_workqueue(card->event_wq); - qeth_free_qdio_buffers(card); + qeth_free_qdio_queues(card); unregister_service_level(&card->qeth_service_level); dev_set_drvdata(&card->gdev->dev, NULL); kfree(card); @@ -4986,12 +4984,14 @@ int qeth_core_hardsetup_card(struct qeth_card *card, bool *carrier_ok) QETH_DBF_TEXT(SETUP, 2, "hrdsetup"); atomic_set(&card->force_alloc_skb, 0); - qeth_update_from_chp_desc(card); + rc = qeth_update_from_chp_desc(card); + if (rc) + return rc; retry: if (retries < 3) QETH_DBF_MESSAGE(2, "Retrying to do IDX activates on device %x.\n", CARD_DEVID(card)); - rc = qeth_qdio_clear_card(card, card->info.type != QETH_CARD_TYPE_IQD); + rc = qeth_qdio_clear_card(card, !IS_IQD(card)); ccw_device_set_offline(CARD_DDEV(card)); ccw_device_set_offline(CARD_WDEV(card)); ccw_device_set_offline(CARD_RDEV(card)); @@ -5019,8 +5019,9 @@ retriable: qeth_determine_capabilities(card); qeth_init_tokens(card); qeth_init_func_level(card); - rc = qeth_idx_activate_channel(card, &card->read, qeth_idx_read_cb); - if (rc == -ERESTARTSYS) { + + rc = qeth_idx_activate_read_channel(card); + if (rc == -EINTR) { QETH_DBF_TEXT(SETUP, 2, "break2"); return rc; } else if (rc) { @@ -5030,8 +5031,9 @@ retriable: else goto retry; } - rc = qeth_idx_activate_channel(card, &card->write, qeth_idx_write_cb); - if (rc == -ERESTARTSYS) { + + rc = qeth_idx_activate_write_channel(card); + if (rc == -EINTR) { QETH_DBF_TEXT(SETUP, 2, "break3"); return rc; } else if (rc) { @@ -5171,7 +5173,7 @@ struct sk_buff *qeth_core_get_next_skb(struct qeth_card *card, return NULL; if (((skb_len >= card->options.rx_sg_cb) && - (!(card->info.type == QETH_CARD_TYPE_OSN)) && + !IS_OSN(card) && (!atomic_read(&card->force_alloc_skb))) || (card->options.cq == QETH_CQ_ENABLED)) use_rx_sg = 1; @@ -5562,13 +5564,17 @@ static struct net_device *qeth_alloc_netdev(struct qeth_card *card) switch (card->info.type) { case QETH_CARD_TYPE_IQD: - dev = alloc_netdev(0, "hsi%d", NET_NAME_UNKNOWN, ether_setup); + dev = alloc_netdev_mqs(0, "hsi%d", NET_NAME_UNKNOWN, + ether_setup, QETH_MAX_QUEUES, 1); + break; + case QETH_CARD_TYPE_OSM: + dev = alloc_etherdev(0); break; case QETH_CARD_TYPE_OSN: dev = alloc_netdev(0, "osn%d", NET_NAME_UNKNOWN, ether_setup); break; default: - dev = alloc_etherdev(0); + dev = alloc_etherdev_mqs(0, QETH_MAX_QUEUES, 1); } if (!dev) @@ -5590,8 +5596,10 @@ static struct net_device *qeth_alloc_netdev(struct qeth_card *card) dev->priv_flags &= ~IFF_TX_SKB_SHARING; dev->hw_features |= NETIF_F_SG; dev->vlan_features |= NETIF_F_SG; - if (IS_IQD(card)) + if (IS_IQD(card)) { + netif_set_real_num_tx_queues(dev, QETH_IQD_MIN_TXQ); dev->features |= NETIF_F_SG; + } } return dev; @@ -5641,14 +5649,16 @@ static int qeth_core_probe_device(struct ccwgroup_device *gdev) } qeth_setup_card(card); - qeth_update_from_chp_desc(card); - card->dev = qeth_alloc_netdev(card); if (!card->dev) { rc = -ENOMEM; goto err_card; } + card->qdio.no_out_queues = card->dev->num_tx_queues; + rc = qeth_update_from_chp_desc(card); + if (rc) + goto err_chp_desc; qeth_determine_capabilities(card); enforced_disc = qeth_enforce_discipline(card); switch (enforced_disc) { @@ -5661,9 +5671,8 @@ static int qeth_core_probe_device(struct ccwgroup_device *gdev) if (rc) goto err_load; - gdev->dev.type = (card->info.type != QETH_CARD_TYPE_OSN) - ? card->discipline->devtype - : &qeth_osn_devtype; + gdev->dev.type = IS_OSN(card) ? &qeth_osn_devtype : + card->discipline->devtype; rc = card->discipline->setup(card->gdev); if (rc) goto err_disc; @@ -5675,6 +5684,7 @@ static int qeth_core_probe_device(struct ccwgroup_device *gdev) err_disc: qeth_core_free_discipline(card); err_load: +err_chp_desc: free_netdev(card->dev); err_card: qeth_core_free_card(card); @@ -5706,10 +5716,8 @@ static int qeth_core_set_online(struct ccwgroup_device *gdev) enum qeth_discipline_id def_discipline; if (!card->discipline) { - if (card->info.type == QETH_CARD_TYPE_IQD) - def_discipline = QETH_DISCIPLINE_LAYER3; - else - def_discipline = QETH_DISCIPLINE_LAYER2; + def_discipline = IS_IQD(card) ? QETH_DISCIPLINE_LAYER3 : + QETH_DISCIPLINE_LAYER2; rc = qeth_core_load_discipline(card, def_discipline); if (rc) goto err; @@ -5737,7 +5745,7 @@ static void qeth_core_shutdown(struct ccwgroup_device *gdev) if ((gdev->state == CCWGROUP_ONLINE) && card->info.hwtrap) qeth_hw_trap(card, QETH_DIAGS_TRAP_DISARM); qeth_qdio_clear_card(card, 0); - qeth_clear_qdio_buffers(card); + qeth_drain_output_queues(card); qdio_free(CARD_DDEV(card)); } @@ -5837,13 +5845,10 @@ int qeth_do_ioctl(struct net_device *dev, struct ifreq *rq, int cmd) rc = qeth_snmp_command(card, rq->ifr_ifru.ifru_data); break; case SIOC_QETH_GET_CARD_TYPE: - if ((card->info.type == QETH_CARD_TYPE_OSD || - card->info.type == QETH_CARD_TYPE_OSM || - card->info.type == QETH_CARD_TYPE_OSX) && - !card->info.guestlan) + if ((IS_OSD(card) || IS_OSM(card) || IS_OSX(card)) && + !IS_VM_NIC(card)) return 1; - else - return 0; + return 0; case SIOCGMIIPHY: mii_data = if_mii(rq); mii_data->phy_id = 0; @@ -6193,7 +6198,6 @@ void qeth_get_stats64(struct net_device *dev, struct rtnl_link_stats64 *stats) stats->rx_errors = card->stats.rx_errors; stats->rx_dropped = card->stats.rx_dropped; stats->multicast = card->stats.rx_multicast; - stats->tx_errors = card->stats.tx_errors; for (i = 0; i < card->qdio.no_out_queues; i++) { queue = card->qdio.out_qs[i]; @@ -6206,6 +6210,15 @@ void qeth_get_stats64(struct net_device *dev, struct rtnl_link_stats64 *stats) } EXPORT_SYMBOL_GPL(qeth_get_stats64); +u16 qeth_iqd_select_queue(struct net_device *dev, struct sk_buff *skb, + u8 cast_type, struct net_device *sb_dev) +{ + if (cast_type != RTN_UNICAST) + return QETH_IQD_MCAST_TXQ; + return QETH_IQD_MIN_UCAST_TXQ; +} +EXPORT_SYMBOL_GPL(qeth_iqd_select_queue); + int qeth_open(struct net_device *dev) { struct qeth_card *card = dev->ml_priv; @@ -6216,7 +6229,7 @@ int qeth_open(struct net_device *dev) return -EIO; card->data.state = CH_STATE_UP; - netif_start_queue(dev); + netif_tx_start_all_queues(dev); napi_enable(&card->napi); local_bh_disable(); diff --git a/drivers/s390/net/qeth_core_mpc.h b/drivers/s390/net/qeth_core_mpc.h index f8c5d4a9be13708b6aeb3273f7d2531ce464f26d..f5237b7c14c436bbfde90370a63d474a8a102f3a 100644 --- a/drivers/s390/net/qeth_core_mpc.h +++ b/drivers/s390/net/qeth_core_mpc.h @@ -82,7 +82,7 @@ enum qeth_card_types { #define IS_OSM(card) ((card)->info.type == QETH_CARD_TYPE_OSM) #define IS_OSN(card) ((card)->info.type == QETH_CARD_TYPE_OSN) #define IS_OSX(card) ((card)->info.type == QETH_CARD_TYPE_OSX) -#define IS_VM_NIC(card) ((card)->info.guestlan) +#define IS_VM_NIC(card) ((card)->info.is_vm_nic) #define QETH_MPC_DIFINFO_LEN_INDICATES_LINK_TYPE 0x18 /* only the first two bytes are looked at in qeth_get_cardname_short */ diff --git a/drivers/s390/net/qeth_core_sys.c b/drivers/s390/net/qeth_core_sys.c index 56deeb6f7bc0746c13db3b9b0745c1078e6aeb24..9f392497d5708f1d8c2d81c72b9c39ca1dbfe9cc 100644 --- a/drivers/s390/net/qeth_core_sys.c +++ b/drivers/s390/net/qeth_core_sys.c @@ -198,6 +198,9 @@ static ssize_t qeth_dev_prioqing_store(struct device *dev, if (!card) return -EINVAL; + if (IS_IQD(card)) + return -EOPNOTSUPP; + mutex_lock(&card->conf_mutex); if (card->state != CARD_STATE_DOWN) { rc = -EPERM; @@ -239,10 +242,6 @@ static ssize_t qeth_dev_prioqing_store(struct device *dev, card->qdio.do_prio_queueing = QETH_NO_PRIO_QUEUEING; card->qdio.default_out_queue = 2; } else if (sysfs_streq(buf, "no_prio_queueing:3")) { - if (card->info.type == QETH_CARD_TYPE_IQD) { - rc = -EPERM; - goto out; - } card->qdio.do_prio_queueing = QETH_NO_PRIO_QUEUEING; card->qdio.default_out_queue = 3; } else if (sysfs_streq(buf, "no_prio_queueing")) { @@ -480,8 +479,7 @@ static ssize_t qeth_dev_isolation_store(struct device *dev, return -EINVAL; mutex_lock(&card->conf_mutex); - if (card->info.type != QETH_CARD_TYPE_OSD && - card->info.type != QETH_CARD_TYPE_OSX) { + if (!IS_OSD(card) && !IS_OSX(card)) { rc = -EOPNOTSUPP; dev_err(&card->gdev->dev, "Adapter does not " "support QDIO data connection isolation\n"); diff --git a/drivers/s390/net/qeth_ethtool.c b/drivers/s390/net/qeth_ethtool.c index 93a53fed4cf8908698592988691390a4e8ea2203..4166eb29f0bdd9fcbfa94f7d654f81c0028a192c 100644 --- a/drivers/s390/net/qeth_ethtool.c +++ b/drivers/s390/net/qeth_ethtool.c @@ -38,6 +38,7 @@ static const struct qeth_stats txq_stats[] = { QETH_TXQ_STAT("linearized+error skbs", skbs_linearized_fail), QETH_TXQ_STAT("TSO bytes", tso_bytes), QETH_TXQ_STAT("Packing mode switches", packing_mode_switch), + QETH_TXQ_STAT("Queue stopped", stopped), }; static const struct qeth_stats card_stats[] = { @@ -154,6 +155,21 @@ static void qeth_get_drvinfo(struct net_device *dev, CARD_RDEV_ID(card), CARD_WDEV_ID(card), CARD_DDEV_ID(card)); } +static void qeth_get_channels(struct net_device *dev, + struct ethtool_channels *channels) +{ + struct qeth_card *card = dev->ml_priv; + + channels->max_rx = dev->num_rx_queues; + channels->max_tx = card->qdio.no_out_queues; + channels->max_other = 0; + channels->max_combined = 0; + channels->rx_count = dev->real_num_rx_queues; + channels->tx_count = dev->real_num_tx_queues; + channels->other_count = 0; + channels->combined_count = 0; +} + /* Helper function to fill 'advertising' and 'supported' which are the same. */ /* Autoneg and full-duplex are supported and advertised unconditionally. */ /* Always advertise and support all speeds up to specified, and only one */ @@ -359,6 +375,7 @@ const struct ethtool_ops qeth_ethtool_ops = { .get_ethtool_stats = qeth_get_ethtool_stats, .get_sset_count = qeth_get_sset_count, .get_drvinfo = qeth_get_drvinfo, + .get_channels = qeth_get_channels, .get_link_ksettings = qeth_get_link_ksettings, }; diff --git a/drivers/s390/net/qeth_l2_main.c b/drivers/s390/net/qeth_l2_main.c index c3067fd3bd9ee47ad79d106cd3b17067ea91fbf3..218801232ca2609671f12ce4a99ae1dfd04d315f 100644 --- a/drivers/s390/net/qeth_l2_main.c +++ b/drivers/s390/net/qeth_l2_main.c @@ -149,29 +149,16 @@ static int qeth_l2_remove_mac(struct qeth_card *card, u8 *mac) return rc; } -static void qeth_l2_del_all_macs(struct qeth_card *card) +static void qeth_l2_drain_rx_mode_cache(struct qeth_card *card) { struct qeth_mac *mac; struct hlist_node *tmp; int i; - spin_lock_bh(&card->mclock); hash_for_each_safe(card->mac_htable, i, tmp, mac, hnode) { hash_del(&mac->hnode); kfree(mac); } - spin_unlock_bh(&card->mclock); -} - -static int qeth_l2_get_cast_type(struct qeth_card *card, struct sk_buff *skb) -{ - if (card->info.type == QETH_CARD_TYPE_OSN) - return RTN_UNICAST; - if (is_broadcast_ether_addr(skb->data)) - return RTN_BROADCAST; - if (is_multicast_ether_addr(skb->data)) - return RTN_MULTICAST; - return RTN_UNICAST; } static void qeth_l2_fill_header(struct qeth_qdio_out_q *queue, @@ -292,14 +279,16 @@ static void qeth_l2_stop_card(struct qeth_card *card) qeth_set_allowed_threads(card, 0, 1); + cancel_work_sync(&card->rx_mode_work); + qeth_l2_drain_rx_mode_cache(card); + if (card->state == CARD_STATE_SOFTSETUP) { - qeth_l2_del_all_macs(card); qeth_clear_ipacmd_list(card); card->state = CARD_STATE_HARDSETUP; } if (card->state == CARD_STATE_HARDSETUP) { qeth_qdio_clear_card(card, 0); - qeth_clear_qdio_buffers(card); + qeth_drain_output_queues(card); qeth_clear_working_pool_list(card); card->state = CARD_STATE_DOWN; } @@ -334,13 +323,11 @@ static int qeth_l2_process_inbound_buffer(struct qeth_card *card, case QETH_HEADER_TYPE_LAYER2: skb->protocol = eth_type_trans(skb, skb->dev); qeth_rx_csum(card, skb, hdr->hdr.l2.flags[1]); - if (skb->protocol == htons(ETH_P_802_2)) - *((__u32 *)skb->cb) = ++card->seqno.pkt_seqno; len = skb->len; napi_gro_receive(&card->napi, skb); break; case QETH_HEADER_TYPE_OSN: - if (card->info.type == QETH_CARD_TYPE_OSN) { + if (IS_OSN(card)) { skb_push(skb, sizeof(struct qeth_hdr)); skb_copy_to_linear_data(skb, hdr, sizeof(struct qeth_hdr)); @@ -391,8 +378,7 @@ static int qeth_l2_request_initial_mac(struct qeth_card *card) } /* some devices don't support a custom MAC address: */ - if (card->info.type == QETH_CARD_TYPE_OSM || - card->info.type == QETH_CARD_TYPE_OSX) + if (IS_OSM(card) || IS_OSX(card)) return (rc) ? rc : -EADDRNOTAVAIL; eth_hw_addr_random(card->dev); @@ -515,9 +501,11 @@ static void qeth_l2_add_mac(struct qeth_card *card, struct netdev_hw_addr *ha) hash_add(card->mac_htable, &mac->hnode, mac_hash); } -static void qeth_l2_set_rx_mode(struct net_device *dev) +static void qeth_l2_rx_mode_work(struct work_struct *work) { - struct qeth_card *card = dev->ml_priv; + struct qeth_card *card = container_of(work, struct qeth_card, + rx_mode_work); + struct net_device *dev = card->dev; struct netdev_hw_addr *ha; struct qeth_mac *mac; struct hlist_node *tmp; @@ -526,12 +514,12 @@ static void qeth_l2_set_rx_mode(struct net_device *dev) QETH_CARD_TEXT(card, 3, "setmulti"); - spin_lock_bh(&card->mclock); - + netif_addr_lock_bh(dev); netdev_for_each_mc_addr(ha, dev) qeth_l2_add_mac(card, ha); netdev_for_each_uc_addr(ha, dev) qeth_l2_add_mac(card, ha); + netif_addr_unlock_bh(dev); hash_for_each_safe(card->mac_htable, i, tmp, mac, hnode) { switch (mac->disp_flag) { @@ -554,8 +542,6 @@ static void qeth_l2_set_rx_mode(struct net_device *dev) } } - spin_unlock_bh(&card->mclock); - if (qeth_adp_supported(card, IPA_SETADP_SET_PROMISC_MODE)) qeth_setadp_promisc_mode(card); else @@ -586,7 +572,7 @@ static int qeth_l2_xmit_osn(struct qeth_card *card, struct sk_buff *skb, } elements += qeth_count_elements(skb, hd_len); - if (elements > QETH_MAX_BUFFER_ELEMENTS(card)) { + if (elements > queue->max_elements) { rc = -E2BIG; goto out; } @@ -603,37 +589,45 @@ static netdev_tx_t qeth_l2_hard_start_xmit(struct sk_buff *skb, struct net_device *dev) { struct qeth_card *card = dev->ml_priv; - int cast_type = qeth_l2_get_cast_type(card, skb); - int ipv = qeth_get_ip_version(skb); + u16 txq = skb_get_queue_mapping(skb); struct qeth_qdio_out_q *queue; int tx_bytes = skb->len; int rc; - queue = qeth_get_tx_queue(card, skb, ipv, cast_type); - - netif_stop_queue(dev); + if (IS_IQD(card)) + txq = qeth_iqd_translate_txq(dev, txq); + queue = card->qdio.out_qs[txq]; if (IS_OSN(card)) rc = qeth_l2_xmit_osn(card, skb, queue); else - rc = qeth_xmit(card, skb, queue, ipv, cast_type, + rc = qeth_xmit(card, skb, queue, qeth_get_ip_version(skb), + qeth_get_ether_cast_type(skb), qeth_l2_fill_header); if (!rc) { QETH_TXQ_STAT_INC(queue, tx_packets); QETH_TXQ_STAT_ADD(queue, tx_bytes, tx_bytes); - netif_wake_queue(dev); return NETDEV_TX_OK; - } else if (rc == -EBUSY) { - return NETDEV_TX_BUSY; - } /* else fall through */ + } QETH_TXQ_STAT_INC(queue, tx_dropped); kfree_skb(skb); - netif_wake_queue(dev); return NETDEV_TX_OK; } +static u16 qeth_l2_select_queue(struct net_device *dev, struct sk_buff *skb, + struct net_device *sb_dev) +{ + struct qeth_card *card = dev->ml_priv; + + if (IS_IQD(card)) + return qeth_iqd_select_queue(dev, skb, + qeth_get_ether_cast_type(skb), + sb_dev); + return qeth_get_priority_queue(card, skb); +} + static const struct device_type qeth_l2_devtype = { .name = "qeth_layer2", .groups = qeth_l2_attr_groups, @@ -653,6 +647,7 @@ static int qeth_l2_probe_device(struct ccwgroup_device *gdev) } hash_init(card->mac_htable); + INIT_WORK(&card->rx_mode_work, qeth_l2_rx_mode_work); return 0; } @@ -673,12 +668,20 @@ static void qeth_l2_remove_device(struct ccwgroup_device *cgdev) unregister_netdev(card->dev); } +static void qeth_l2_set_rx_mode(struct net_device *dev) +{ + struct qeth_card *card = dev->ml_priv; + + schedule_work(&card->rx_mode_work); +} + static const struct net_device_ops qeth_l2_netdev_ops = { .ndo_open = qeth_open, .ndo_stop = qeth_stop, .ndo_get_stats64 = qeth_get_stats64, .ndo_start_xmit = qeth_l2_hard_start_xmit, .ndo_features_check = qeth_features_check, + .ndo_select_queue = qeth_l2_select_queue, .ndo_validate_addr = qeth_l2_validate_addr, .ndo_set_rx_mode = qeth_l2_set_rx_mode, .ndo_do_ioctl = qeth_do_ioctl, @@ -721,7 +724,7 @@ static int qeth_l2_setup_netdev(struct qeth_card *card, bool carrier_ok) card->dev->features |= NETIF_F_HW_VLAN_CTAG_FILTER; } - if (card->info.type == QETH_CARD_TYPE_OSD && !card->info.guestlan) { + if (IS_OSD(card) && !IS_VM_NIC(card)) { card->dev->features |= NETIF_F_SG; /* OSA 3S and earlier has no RX/TX support */ if (qeth_is_supported(card, IPA_OUTBOUND_CHECKSUM)) { @@ -831,8 +834,7 @@ static int qeth_l2_set_online(struct ccwgroup_device *gdev) /* softsetup */ QETH_DBF_TEXT(SETUP, 2, "softsetp"); - if ((card->info.type == QETH_CARD_TYPE_OSD) || - (card->info.type == QETH_CARD_TYPE_OSX)) { + if (IS_OSD(card) || IS_OSX(card)) { rc = qeth_l2_start_ipassists(card); if (rc) goto out_remove; @@ -1042,13 +1044,13 @@ static int qeth_osn_send_control_data(struct qeth_card *card, int len, QETH_CARD_TEXT(card, 5, "osndctrd"); - wait_event(card->wait_q, - atomic_cmpxchg(&channel->irq_pending, 0, 1) == 0); - qeth_prepare_control_data(card, len, iob); + wait_event(card->wait_q, qeth_trylock_channel(channel)); + iob->finalize(card, iob, len); + QETH_DBF_HEX(CTRL, 2, iob->data, min(len, QETH_DBF_CTRL_LEN)); QETH_CARD_TEXT(card, 6, "osnoirqp"); spin_lock_irq(get_ccwdev_lock(channel->ccwdev)); rc = ccw_device_start_timeout(channel->ccwdev, channel->ccw, - (addr_t) iob, 0, 0, QETH_IPA_TIMEOUT); + (addr_t) iob, 0, 0, iob->timeout); spin_unlock_irq(get_ccwdev_lock(channel->ccwdev)); if (rc) { QETH_DBF_MESSAGE(2, "qeth_osn_send_control_data: " @@ -1456,9 +1458,8 @@ static struct qeth_cmd_buffer *qeth_sbp_build_cmd(struct qeth_card *card, enum qeth_ipa_sbp_cmd sbp_cmd, unsigned int cmd_length) { - enum qeth_ipa_cmds ipa_cmd = (card->info.type == QETH_CARD_TYPE_IQD) ? - IPA_CMD_SETBRIDGEPORT_IQD : - IPA_CMD_SETBRIDGEPORT_OSA; + enum qeth_ipa_cmds ipa_cmd = IS_IQD(card) ? IPA_CMD_SETBRIDGEPORT_IQD : + IPA_CMD_SETBRIDGEPORT_OSA; struct qeth_cmd_buffer *iob; struct qeth_ipa_cmd *cmd; diff --git a/drivers/s390/net/qeth_l3_main.c b/drivers/s390/net/qeth_l3_main.c index 53712cf2640659cb0da642ba1dc05bb03e6567ec..0271833da6a2a42a0c78320318ed3f24f9b0b85f 100644 --- a/drivers/s390/net/qeth_l3_main.c +++ b/drivers/s390/net/qeth_l3_main.c @@ -246,9 +246,9 @@ static int qeth_l3_add_ip(struct qeth_card *card, struct qeth_ipaddr *tmp_addr) */ if (addr->proto == QETH_PROT_IPV4) { addr->in_progress = 1; - spin_unlock_bh(&card->ip_lock); + mutex_unlock(&card->ip_lock); rc = qeth_l3_register_addr_entry(card, addr); - spin_lock_bh(&card->ip_lock); + mutex_lock(&card->ip_lock); addr->in_progress = 0; } else rc = qeth_l3_register_addr_entry(card, addr); @@ -268,6 +268,30 @@ static int qeth_l3_add_ip(struct qeth_card *card, struct qeth_ipaddr *tmp_addr) return rc; } +static int qeth_l3_modify_ip(struct qeth_card *card, struct qeth_ipaddr *addr, + bool add) +{ + int rc; + + mutex_lock(&card->ip_lock); + rc = add ? qeth_l3_add_ip(card, addr) : qeth_l3_delete_ip(card, addr); + mutex_unlock(&card->ip_lock); + + return rc; +} + +static void qeth_l3_drain_rx_mode_cache(struct qeth_card *card) +{ + struct qeth_ipaddr *addr; + struct hlist_node *tmp; + int i; + + hash_for_each_safe(card->ip_mc_htable, i, tmp, addr, hnode) { + hash_del(&addr->hnode); + kfree(addr); + } +} + static void qeth_l3_clear_ip_htable(struct qeth_card *card, int recover) { struct qeth_ipaddr *addr; @@ -276,7 +300,7 @@ static void qeth_l3_clear_ip_htable(struct qeth_card *card, int recover) QETH_CARD_TEXT(card, 4, "clearip"); - spin_lock_bh(&card->ip_lock); + mutex_lock(&card->ip_lock); hash_for_each_safe(card->ip_htable, i, tmp, addr, hnode) { if (!recover) { @@ -287,19 +311,9 @@ static void qeth_l3_clear_ip_htable(struct qeth_card *card, int recover) addr->disp_flag = QETH_DISP_ADDR_ADD; } - spin_unlock_bh(&card->ip_lock); - - spin_lock_bh(&card->mclock); - - hash_for_each_safe(card->ip_mc_htable, i, tmp, addr, hnode) { - hash_del(&addr->hnode); - kfree(addr); - } - - spin_unlock_bh(&card->mclock); - - + mutex_unlock(&card->ip_lock); } + static void qeth_l3_recover_ip(struct qeth_card *card) { struct qeth_ipaddr *addr; @@ -309,15 +323,15 @@ static void qeth_l3_recover_ip(struct qeth_card *card) QETH_CARD_TEXT(card, 4, "recovrip"); - spin_lock_bh(&card->ip_lock); + mutex_lock(&card->ip_lock); hash_for_each_safe(card->ip_htable, i, tmp, addr, hnode) { if (addr->disp_flag == QETH_DISP_ADDR_ADD) { if (addr->proto == QETH_PROT_IPV4) { addr->in_progress = 1; - spin_unlock_bh(&card->ip_lock); + mutex_unlock(&card->ip_lock); rc = qeth_l3_register_addr_entry(card, addr); - spin_lock_bh(&card->ip_lock); + mutex_lock(&card->ip_lock); addr->in_progress = 0; } else rc = qeth_l3_register_addr_entry(card, addr); @@ -333,8 +347,7 @@ static void qeth_l3_recover_ip(struct qeth_card *card) } } - spin_unlock_bh(&card->ip_lock); - + mutex_unlock(&card->ip_lock); } static int qeth_l3_setdelip_cb(struct qeth_card *card, struct qeth_reply *reply, @@ -461,7 +474,7 @@ static int qeth_l3_send_setrouting(struct qeth_card *card, static int qeth_l3_correct_routing_type(struct qeth_card *card, enum qeth_routing_types *type, enum qeth_prot_versions prot) { - if (card->info.type == QETH_CARD_TYPE_IQD) { + if (IS_IQD(card)) { switch (*type) { case NO_ROUTER: case PRIMARY_CONNECTOR: @@ -559,7 +572,7 @@ static void qeth_l3_clear_ipato_list(struct qeth_card *card) { struct qeth_ipato_entry *ipatoe, *tmp; - spin_lock_bh(&card->ip_lock); + mutex_lock(&card->ip_lock); list_for_each_entry_safe(ipatoe, tmp, &card->ipato.entries, entry) { list_del(&ipatoe->entry); @@ -567,7 +580,7 @@ static void qeth_l3_clear_ipato_list(struct qeth_card *card) } qeth_l3_update_ipato(card); - spin_unlock_bh(&card->ip_lock); + mutex_unlock(&card->ip_lock); } int qeth_l3_add_ipato_entry(struct qeth_card *card, @@ -578,7 +591,7 @@ int qeth_l3_add_ipato_entry(struct qeth_card *card, QETH_CARD_TEXT(card, 2, "addipato"); - spin_lock_bh(&card->ip_lock); + mutex_lock(&card->ip_lock); list_for_each_entry(ipatoe, &card->ipato.entries, entry) { if (ipatoe->proto != new->proto) @@ -596,7 +609,7 @@ int qeth_l3_add_ipato_entry(struct qeth_card *card, qeth_l3_update_ipato(card); } - spin_unlock_bh(&card->ip_lock); + mutex_unlock(&card->ip_lock); return rc; } @@ -610,7 +623,7 @@ int qeth_l3_del_ipato_entry(struct qeth_card *card, QETH_CARD_TEXT(card, 2, "delipato"); - spin_lock_bh(&card->ip_lock); + mutex_lock(&card->ip_lock); list_for_each_entry_safe(ipatoe, tmp, &card->ipato.entries, entry) { if (ipatoe->proto != proto) @@ -625,7 +638,7 @@ int qeth_l3_del_ipato_entry(struct qeth_card *card, } } - spin_unlock_bh(&card->ip_lock); + mutex_unlock(&card->ip_lock); return rc; } @@ -634,7 +647,6 @@ int qeth_l3_modify_rxip_vipa(struct qeth_card *card, bool add, const u8 *ip, enum qeth_prot_versions proto) { struct qeth_ipaddr addr; - int rc; qeth_l3_init_ipaddr(&addr, type, proto); if (proto == QETH_PROT_IPV4) @@ -642,16 +654,13 @@ int qeth_l3_modify_rxip_vipa(struct qeth_card *card, bool add, const u8 *ip, else memcpy(&addr.u.a6.addr, ip, 16); - spin_lock_bh(&card->ip_lock); - rc = add ? qeth_l3_add_ip(card, &addr) : qeth_l3_delete_ip(card, &addr); - spin_unlock_bh(&card->ip_lock); - return rc; + return qeth_l3_modify_ip(card, &addr, add); } int qeth_l3_modify_hsuid(struct qeth_card *card, bool add) { struct qeth_ipaddr addr; - int rc, i; + unsigned int i; qeth_l3_init_ipaddr(&addr, QETH_IP_TYPE_NORMAL, QETH_PROT_IPV6); addr.u.a6.addr.s6_addr[0] = 0xfe; @@ -659,10 +668,7 @@ int qeth_l3_modify_hsuid(struct qeth_card *card, bool add) for (i = 0; i < 8; i++) addr.u.a6.addr.s6_addr[8+i] = card->options.hsuid[i]; - spin_lock_bh(&card->ip_lock); - rc = add ? qeth_l3_add_ip(card, &addr) : qeth_l3_delete_ip(card, &addr); - spin_unlock_bh(&card->ip_lock); - return rc; + return qeth_l3_modify_ip(card, &addr, add); } static int qeth_l3_register_addr_entry(struct qeth_card *card, @@ -848,7 +854,7 @@ static int qeth_l3_softsetup_ipv6(struct qeth_card *card) QETH_CARD_TEXT(card, 3, "softipv6"); - if (card->info.type == QETH_CARD_TYPE_IQD) + if (IS_IQD(card)) goto out; rc = qeth_send_simple_setassparms(card, IPA_IPV6, @@ -1374,8 +1380,7 @@ static int qeth_l3_process_inbound_buffer(struct qeth_card *card, switch (hdr->hdr.l3.id) { case QETH_HEADER_TYPE_LAYER3: magic = *(__u16 *)skb->data; - if ((card->info.type == QETH_CARD_TYPE_IQD) && - (magic == ETH_P_AF_IUCV)) { + if (IS_IQD(card) && magic == ETH_P_AF_IUCV) { len = skb->len; dev_hard_header(skb, dev, ETH_P_AF_IUCV, dev->dev_addr, "FAKELL", len); @@ -1413,6 +1418,9 @@ static void qeth_l3_stop_card(struct qeth_card *card) qeth_set_allowed_threads(card, 0, 1); + cancel_work_sync(&card->rx_mode_work); + qeth_l3_drain_rx_mode_cache(card); + if (card->options.sniffer && (card->info.promisc_mode == SET_PROMISC_MODE_ON)) qeth_diags_trace(card, QETH_DIAGS_CMD_TRACE_DISABLE); @@ -1424,7 +1432,7 @@ static void qeth_l3_stop_card(struct qeth_card *card) } if (card->state == CARD_STATE_HARDSETUP) { qeth_qdio_clear_card(card, 0); - qeth_clear_qdio_buffers(card); + qeth_drain_output_queues(card); qeth_clear_working_pool_list(card); card->state = CARD_STATE_DOWN; } @@ -1451,7 +1459,7 @@ qeth_l3_handle_promisc_mode(struct qeth_card *card) (card->info.promisc_mode == SET_PROMISC_MODE_OFF))) return; - if (card->info.guestlan) { /* Guestlan trace */ + if (IS_VM_NIC(card)) { /* Guestlan trace */ if (qeth_adp_supported(card, IPA_SETADP_SET_PROMISC_MODE)) qeth_setadp_promisc_mode(card); } else if (card->options.sniffer && /* HiperSockets trace */ @@ -1466,9 +1474,10 @@ qeth_l3_handle_promisc_mode(struct qeth_card *card) } } -static void qeth_l3_set_rx_mode(struct net_device *dev) +static void qeth_l3_rx_mode_work(struct work_struct *work) { - struct qeth_card *card = dev->ml_priv; + struct qeth_card *card = container_of(work, struct qeth_card, + rx_mode_work); struct qeth_ipaddr *addr; struct hlist_node *tmp; int i, rc; @@ -1476,8 +1485,6 @@ static void qeth_l3_set_rx_mode(struct net_device *dev) QETH_CARD_TEXT(card, 3, "setmulti"); if (!card->options.sniffer) { - spin_lock_bh(&card->mclock); - qeth_l3_add_multicast_ipv4(card); qeth_l3_add_multicast_ipv6(card); @@ -1505,8 +1512,6 @@ static void qeth_l3_set_rx_mode(struct net_device *dev) } } - spin_unlock_bh(&card->mclock); - if (!qeth_adp_supported(card, IPA_SETADP_SET_PROMISC_MODE)) return; } @@ -1551,7 +1556,7 @@ static int qeth_l3_arp_set_no_entries(struct qeth_card *card, int no_entries) * IPA_CMD_ASS_ARP_QUERY_INFO, but not IPA_CMD_ASS_ARP_SET_NO_ENTRIES; * thus we say EOPNOTSUPP for this ARP function */ - if (card->info.guestlan) + if (IS_VM_NIC(card)) return -EOPNOTSUPP; if (!qeth_is_supported(card, IPA_ARP_PROCESSING)) { return -EOPNOTSUPP; @@ -1783,7 +1788,7 @@ static int qeth_l3_arp_modify_entry(struct qeth_card *card, * IPA_CMD_ASS_ARP_QUERY_INFO, but not IPA_CMD_ASS_ARP_ADD_ENTRY; * thus we say EOPNOTSUPP for this ARP function */ - if (card->info.guestlan) + if (IS_VM_NIC(card)) return -EOPNOTSUPP; if (!qeth_is_supported(card, IPA_ARP_PROCESSING)) { return -EOPNOTSUPP; @@ -1816,7 +1821,7 @@ static int qeth_l3_arp_flush_cache(struct qeth_card *card) * IPA_CMD_ASS_ARP_QUERY_INFO, but not IPA_CMD_ASS_ARP_FLUSH_CACHE; * thus we say EOPNOTSUPP for this ARP function */ - if (card->info.guestlan || (card->info.type == QETH_CARD_TYPE_IQD)) + if (IS_VM_NIC(card) || IS_IQD(card)) return -EOPNOTSUPP; if (!qeth_is_supported(card, IPA_ARP_PROCESSING)) { return -EOPNOTSUPP; @@ -1913,13 +1918,7 @@ static int qeth_l3_get_cast_type(struct sk_buff *skb) RTN_MULTICAST : RTN_UNICAST; default: /* ... and MAC address */ - if (ether_addr_equal_64bits(eth_hdr(skb)->h_dest, - skb->dev->broadcast)) - return RTN_BROADCAST; - if (is_multicast_ether_addr(eth_hdr(skb)->h_dest)) - return RTN_MULTICAST; - /* default to unicast */ - return RTN_UNICAST; + return qeth_get_ether_cast_type(skb); } } @@ -1977,19 +1976,14 @@ static void qeth_l3_fill_header(struct qeth_qdio_out_q *queue, hdr->hdr.l3.vlan_id = ntohs(veth->h_vlan_TCI); } + l3_hdr->flags = qeth_l3_cast_type_to_flag(cast_type); + /* OSA only: */ if (!ipv) { - hdr->hdr.l3.flags = QETH_HDR_PASSTHRU; - if (ether_addr_equal_64bits(eth_hdr(skb)->h_dest, - skb->dev->broadcast)) - hdr->hdr.l3.flags |= QETH_CAST_BROADCAST; - else - hdr->hdr.l3.flags |= (cast_type == RTN_MULTICAST) ? - QETH_CAST_MULTICAST : QETH_CAST_UNICAST; + l3_hdr->flags |= QETH_HDR_PASSTHRU; return; } - hdr->hdr.l3.flags = qeth_l3_cast_type_to_flag(cast_type); rcu_read_lock(); if (ipv == 4) { struct rtable *rt = skb_rtable(skb); @@ -2007,7 +2001,7 @@ static void qeth_l3_fill_header(struct qeth_qdio_out_q *queue, l3_hdr->next_hop.ipv6_addr = ipv6_hdr(skb)->daddr; hdr->hdr.l3.flags |= QETH_HDR_IPV6; - if (card->info.type != QETH_CARD_TYPE_IQD) + if (!IS_IQD(card)) hdr->hdr.l3.flags |= QETH_HDR_PASSTHRU; } rcu_read_unlock(); @@ -2030,7 +2024,6 @@ static void qeth_l3_fixup_headers(struct sk_buff *skb) static int qeth_l3_xmit(struct qeth_card *card, struct sk_buff *skb, struct qeth_qdio_out_q *queue, int ipv, int cast_type) { - unsigned char eth_hdr[ETH_HLEN]; unsigned int hw_hdr_len; int rc; @@ -2040,45 +2033,44 @@ static int qeth_l3_xmit(struct qeth_card *card, struct sk_buff *skb, rc = skb_cow_head(skb, hw_hdr_len - ETH_HLEN); if (rc) return rc; - skb_copy_from_linear_data(skb, eth_hdr, ETH_HLEN); skb_pull(skb, ETH_HLEN); qeth_l3_fixup_headers(skb); - rc = qeth_xmit(card, skb, queue, ipv, cast_type, qeth_l3_fill_header); - if (rc == -EBUSY) { - /* roll back to ETH header */ - skb_push(skb, ETH_HLEN); - skb_copy_to_linear_data(skb, eth_hdr, ETH_HLEN); - } - return rc; + return qeth_xmit(card, skb, queue, ipv, cast_type, qeth_l3_fill_header); } static netdev_tx_t qeth_l3_hard_start_xmit(struct sk_buff *skb, struct net_device *dev) { - int cast_type = qeth_l3_get_cast_type(skb); struct qeth_card *card = dev->ml_priv; + u16 txq = skb_get_queue_mapping(skb); int ipv = qeth_get_ip_version(skb); struct qeth_qdio_out_q *queue; int tx_bytes = skb->len; - int rc; - - queue = qeth_get_tx_queue(card, skb, ipv, cast_type); + int cast_type, rc; if (IS_IQD(card)) { + queue = card->qdio.out_qs[qeth_iqd_translate_txq(dev, txq)]; + if (card->options.sniffer) goto tx_drop; if ((card->options.cq != QETH_CQ_ENABLED && !ipv) || (card->options.cq == QETH_CQ_ENABLED && skb->protocol != htons(ETH_P_AF_IUCV))) goto tx_drop; + + if (txq == QETH_IQD_MCAST_TXQ) + cast_type = qeth_l3_get_cast_type(skb); + else + cast_type = RTN_UNICAST; + } else { + queue = card->qdio.out_qs[txq]; + cast_type = qeth_l3_get_cast_type(skb); } if (cast_type == RTN_BROADCAST && !card->info.broadcast_capable) goto tx_drop; - netif_stop_queue(dev); - if (ipv == 4 || IS_IQD(card)) rc = qeth_l3_xmit(card, skb, queue, ipv, cast_type); else @@ -2088,19 +2080,22 @@ static netdev_tx_t qeth_l3_hard_start_xmit(struct sk_buff *skb, if (!rc) { QETH_TXQ_STAT_INC(queue, tx_packets); QETH_TXQ_STAT_ADD(queue, tx_bytes, tx_bytes); - netif_wake_queue(dev); return NETDEV_TX_OK; - } else if (rc == -EBUSY) { - return NETDEV_TX_BUSY; - } /* else fall through */ + } tx_drop: QETH_TXQ_STAT_INC(queue, tx_dropped); kfree_skb(skb); - netif_wake_queue(dev); return NETDEV_TX_OK; } +static void qeth_l3_set_rx_mode(struct net_device *dev) +{ + struct qeth_card *card = dev->ml_priv; + + schedule_work(&card->rx_mode_work); +} + /* * we need NOARP for IPv4 but we want neighbor solicitation for IPv6. Setting * NOARP on the netdevice is no option because it also turns off neighbor @@ -2134,11 +2129,27 @@ static netdev_features_t qeth_l3_osa_features_check(struct sk_buff *skb, return qeth_features_check(skb, dev, features); } +static u16 qeth_l3_iqd_select_queue(struct net_device *dev, struct sk_buff *skb, + struct net_device *sb_dev) +{ + return qeth_iqd_select_queue(dev, skb, qeth_l3_get_cast_type(skb), + sb_dev); +} + +static u16 qeth_l3_osa_select_queue(struct net_device *dev, struct sk_buff *skb, + struct net_device *sb_dev) +{ + struct qeth_card *card = dev->ml_priv; + + return qeth_get_priority_queue(card, skb); +} + static const struct net_device_ops qeth_l3_netdev_ops = { .ndo_open = qeth_open, .ndo_stop = qeth_stop, .ndo_get_stats64 = qeth_get_stats64, .ndo_start_xmit = qeth_l3_hard_start_xmit, + .ndo_select_queue = qeth_l3_iqd_select_queue, .ndo_validate_addr = eth_validate_addr, .ndo_set_rx_mode = qeth_l3_set_rx_mode, .ndo_do_ioctl = qeth_do_ioctl, @@ -2155,6 +2166,7 @@ static const struct net_device_ops qeth_l3_osa_netdev_ops = { .ndo_get_stats64 = qeth_get_stats64, .ndo_start_xmit = qeth_l3_hard_start_xmit, .ndo_features_check = qeth_l3_osa_features_check, + .ndo_select_queue = qeth_l3_osa_select_queue, .ndo_validate_addr = eth_validate_addr, .ndo_set_rx_mode = qeth_l3_set_rx_mode, .ndo_do_ioctl = qeth_do_ioctl, @@ -2171,8 +2183,7 @@ static int qeth_l3_setup_netdev(struct qeth_card *card, bool carrier_ok) unsigned int headroom; int rc; - if (card->info.type == QETH_CARD_TYPE_OSD || - card->info.type == QETH_CARD_TYPE_OSX) { + if (IS_OSD(card) || IS_OSX(card)) { if ((card->info.link_type == QETH_LINK_TYPE_LANE_TR) || (card->info.link_type == QETH_LINK_TYPE_HSTR)) { pr_info("qeth_l3: ignoring TR device\n"); @@ -2186,7 +2197,7 @@ static int qeth_l3_setup_netdev(struct qeth_card *card, bool carrier_ok) if (!(card->info.unique_id & UNIQUE_ID_NOT_BY_CARD)) card->dev->dev_id = card->info.unique_id & 0xffff; - if (!card->info.guestlan) { + if (!IS_VM_NIC(card)) { card->dev->features |= NETIF_F_SG; card->dev->hw_features |= NETIF_F_TSO | NETIF_F_RXCSUM | NETIF_F_IP_CSUM; @@ -2210,7 +2221,7 @@ static int qeth_l3_setup_netdev(struct qeth_card *card, bool carrier_ok) headroom = sizeof(struct qeth_hdr_tso); else headroom = sizeof(struct qeth_hdr) + VLAN_HLEN; - } else if (card->info.type == QETH_CARD_TYPE_IQD) { + } else if (IS_IQD(card)) { card->dev->flags |= IFF_NOARP; card->dev->netdev_ops = &qeth_l3_netdev_ops; headroom = sizeof(struct qeth_hdr) - ETH_HLEN; @@ -2253,14 +2264,22 @@ static int qeth_l3_probe_device(struct ccwgroup_device *gdev) int rc; hash_init(card->ip_htable); + mutex_init(&card->ip_lock); + card->cmd_wq = alloc_ordered_workqueue("%s_cmd", 0, + dev_name(&gdev->dev)); + if (!card->cmd_wq) + return -ENOMEM; if (gdev->dev.type == &qeth_generic_devtype) { rc = qeth_l3_create_device_attributes(&gdev->dev); - if (rc) + if (rc) { + destroy_workqueue(card->cmd_wq); return rc; + } } hash_init(card->ip_mc_htable); + INIT_WORK(&card->rx_mode_work, qeth_l3_rx_mode_work); return 0; } @@ -2280,6 +2299,9 @@ static void qeth_l3_remove_device(struct ccwgroup_device *cgdev) cancel_work_sync(&card->close_dev_work); if (qeth_netdev_is_registered(card->dev)) unregister_netdev(card->dev); + + flush_workqueue(card->cmd_wq); + destroy_workqueue(card->cmd_wq); qeth_l3_clear_ip_htable(card, 0); qeth_l3_clear_ipato_list(card); } @@ -2517,20 +2539,40 @@ static int qeth_l3_handle_ip_event(struct qeth_card *card, { switch (event) { case NETDEV_UP: - spin_lock_bh(&card->ip_lock); - qeth_l3_add_ip(card, addr); - spin_unlock_bh(&card->ip_lock); + qeth_l3_modify_ip(card, addr, true); return NOTIFY_OK; case NETDEV_DOWN: - spin_lock_bh(&card->ip_lock); - qeth_l3_delete_ip(card, addr); - spin_unlock_bh(&card->ip_lock); + qeth_l3_modify_ip(card, addr, false); return NOTIFY_OK; default: return NOTIFY_DONE; } } +struct qeth_l3_ip_event_work { + struct work_struct work; + struct qeth_card *card; + struct qeth_ipaddr addr; +}; + +#define to_ip_work(w) container_of((w), struct qeth_l3_ip_event_work, work) + +static void qeth_l3_add_ip_worker(struct work_struct *work) +{ + struct qeth_l3_ip_event_work *ip_work = to_ip_work(work); + + qeth_l3_modify_ip(ip_work->card, &ip_work->addr, true); + kfree(work); +} + +static void qeth_l3_delete_ip_worker(struct work_struct *work) +{ + struct qeth_l3_ip_event_work *ip_work = to_ip_work(work); + + qeth_l3_modify_ip(ip_work->card, &ip_work->addr, false); + kfree(work); +} + static struct qeth_card *qeth_l3_get_card_from_dev(struct net_device *dev) { if (is_vlan_dev(dev)) @@ -2575,9 +2617,12 @@ static int qeth_l3_ip6_event(struct notifier_block *this, { struct inet6_ifaddr *ifa = (struct inet6_ifaddr *)ptr; struct net_device *dev = ifa->idev->dev; - struct qeth_ipaddr addr; + struct qeth_l3_ip_event_work *ip_work; struct qeth_card *card; + if (event != NETDEV_UP && event != NETDEV_DOWN) + return NOTIFY_DONE; + card = qeth_l3_get_card_from_dev(dev); if (!card) return NOTIFY_DONE; @@ -2585,11 +2630,23 @@ static int qeth_l3_ip6_event(struct notifier_block *this, if (!qeth_is_supported(card, IPA_IPV6)) return NOTIFY_DONE; - qeth_l3_init_ipaddr(&addr, QETH_IP_TYPE_NORMAL, QETH_PROT_IPV6); - addr.u.a6.addr = ifa->addr; - addr.u.a6.pfxlen = ifa->prefix_len; + ip_work = kmalloc(sizeof(*ip_work), GFP_ATOMIC); + if (!ip_work) + return NOTIFY_DONE; - return qeth_l3_handle_ip_event(card, &addr, event); + if (event == NETDEV_UP) + INIT_WORK(&ip_work->work, qeth_l3_add_ip_worker); + else + INIT_WORK(&ip_work->work, qeth_l3_delete_ip_worker); + + ip_work->card = card; + qeth_l3_init_ipaddr(&ip_work->addr, QETH_IP_TYPE_NORMAL, + QETH_PROT_IPV6); + ip_work->addr.u.a6.addr = ifa->addr; + ip_work->addr.u.a6.pfxlen = ifa->prefix_len; + + queue_work(card->cmd_wq, &ip_work->work); + return NOTIFY_OK; } static struct notifier_block qeth_l3_ip6_notifier = { diff --git a/drivers/s390/net/qeth_l3_sys.c b/drivers/s390/net/qeth_l3_sys.c index cff518b0f9043ae4855dfc91fef82c8ede81ceb5..2f73b33c9347e7e8b8cd92991d9edf891f0c7d97 100644 --- a/drivers/s390/net/qeth_l3_sys.c +++ b/drivers/s390/net/qeth_l3_sys.c @@ -206,7 +206,7 @@ static ssize_t qeth_l3_dev_sniffer_store(struct device *dev, if (!card) return -EINVAL; - if (card->info.type != QETH_CARD_TYPE_IQD) + if (!IS_IQD(card)) return -EPERM; if (card->options.cq == QETH_CQ_ENABLED) return -EPERM; @@ -258,7 +258,7 @@ static ssize_t qeth_l3_dev_hsuid_show(struct device *dev, if (!card) return -EINVAL; - if (card->info.type != QETH_CARD_TYPE_IQD) + if (!IS_IQD(card)) return -EPERM; memcpy(tmp_hsuid, card->options.hsuid, sizeof(tmp_hsuid)); @@ -276,7 +276,7 @@ static ssize_t qeth_l3_dev_hsuid_store(struct device *dev, if (!card) return -EINVAL; - if (card->info.type != QETH_CARD_TYPE_IQD) + if (!IS_IQD(card)) return -EPERM; if (card->state != CARD_STATE_DOWN) return -EPERM; @@ -367,9 +367,9 @@ static ssize_t qeth_l3_dev_ipato_enable_store(struct device *dev, if (card->ipato.enabled != enable) { card->ipato.enabled = enable; - spin_lock_bh(&card->ip_lock); + mutex_lock(&card->ip_lock); qeth_l3_update_ipato(card); - spin_unlock_bh(&card->ip_lock); + mutex_unlock(&card->ip_lock); } out: mutex_unlock(&card->conf_mutex); @@ -412,9 +412,9 @@ static ssize_t qeth_l3_dev_ipato_invert4_store(struct device *dev, if (card->ipato.invert4 != invert) { card->ipato.invert4 = invert; - spin_lock_bh(&card->ip_lock); + mutex_lock(&card->ip_lock); qeth_l3_update_ipato(card); - spin_unlock_bh(&card->ip_lock); + mutex_unlock(&card->ip_lock); } out: mutex_unlock(&card->conf_mutex); @@ -436,7 +436,7 @@ static ssize_t qeth_l3_dev_ipato_add_show(char *buf, struct qeth_card *card, entry_len = (proto == QETH_PROT_IPV4)? 12 : 40; /* add strlen for "/\n" */ entry_len += (proto == QETH_PROT_IPV4)? 5 : 6; - spin_lock_bh(&card->ip_lock); + mutex_lock(&card->ip_lock); list_for_each_entry(ipatoe, &card->ipato.entries, entry) { if (ipatoe->proto != proto) continue; @@ -449,7 +449,7 @@ static ssize_t qeth_l3_dev_ipato_add_show(char *buf, struct qeth_card *card, i += snprintf(buf + i, PAGE_SIZE - i, "%s/%i\n", addr_str, ipatoe->mask_bits); } - spin_unlock_bh(&card->ip_lock); + mutex_unlock(&card->ip_lock); i += snprintf(buf + i, PAGE_SIZE - i, "\n"); return i; @@ -598,9 +598,9 @@ static ssize_t qeth_l3_dev_ipato_invert6_store(struct device *dev, if (card->ipato.invert6 != invert) { card->ipato.invert6 = invert; - spin_lock_bh(&card->ip_lock); + mutex_lock(&card->ip_lock); qeth_l3_update_ipato(card); - spin_unlock_bh(&card->ip_lock); + mutex_unlock(&card->ip_lock); } out: mutex_unlock(&card->conf_mutex); @@ -684,7 +684,7 @@ static ssize_t qeth_l3_dev_ip_add_show(struct device *dev, char *buf, entry_len = (proto == QETH_PROT_IPV4)? 12 : 40; entry_len += 2; /* \n + terminator */ - spin_lock_bh(&card->ip_lock); + mutex_lock(&card->ip_lock); hash_for_each(card->ip_htable, i, ipaddr, hnode) { if (ipaddr->proto != proto || ipaddr->type != type) continue; @@ -698,7 +698,7 @@ static ssize_t qeth_l3_dev_ip_add_show(struct device *dev, char *buf, str_len += snprintf(buf + str_len, PAGE_SIZE - str_len, "%s\n", addr_str); } - spin_unlock_bh(&card->ip_lock); + mutex_unlock(&card->ip_lock); str_len += snprintf(buf + str_len, PAGE_SIZE - str_len, "\n"); return str_len; diff --git a/drivers/ssb/bridge_pcmcia_80211.c b/drivers/ssb/bridge_pcmcia_80211.c index f51f150307dfbcab4d6ed3ddad9077e62427e37a..ffa379efff83c7609ecf2be2516e5edd34bc250e 100644 --- a/drivers/ssb/bridge_pcmcia_80211.c +++ b/drivers/ssb/bridge_pcmcia_80211.c @@ -113,16 +113,21 @@ static struct pcmcia_driver ssb_host_pcmcia_driver = { .resume = ssb_host_pcmcia_resume, }; +static int pcmcia_init_failed; + /* * These are not module init/exit functions! * The module_pcmcia_driver() helper cannot be used here. */ int ssb_host_pcmcia_init(void) { - return pcmcia_register_driver(&ssb_host_pcmcia_driver); + pcmcia_init_failed = pcmcia_register_driver(&ssb_host_pcmcia_driver); + + return pcmcia_init_failed; } void ssb_host_pcmcia_exit(void) { - pcmcia_unregister_driver(&ssb_host_pcmcia_driver); + if (!pcmcia_init_failed) + pcmcia_unregister_driver(&ssb_host_pcmcia_driver); } diff --git a/drivers/staging/octeon/ethernet.c b/drivers/staging/octeon/ethernet.c index 986db76705ccc6b5d384f2db08f715739936d942..8847a11c212fe450950e1ea398558a27bcfe9e4b 100644 --- a/drivers/staging/octeon/ethernet.c +++ b/drivers/staging/octeon/ethernet.c @@ -421,7 +421,7 @@ int cvm_oct_common_init(struct net_device *dev) if (priv->of_node) mac = of_get_mac_address(priv->of_node); - if (mac) + if (!IS_ERR_OR_NULL(mac)) ether_addr_copy(dev->dev_addr, mac); else eth_hw_addr_random(dev); diff --git a/drivers/staging/rtl8188eu/os_dep/os_intfs.c b/drivers/staging/rtl8188eu/os_dep/os_intfs.c index 8dde5a40e25334f08d326a209fc90022d7986256..2c088af44c8ba8d98f146a848c81a59076d99461 100644 --- a/drivers/staging/rtl8188eu/os_dep/os_intfs.c +++ b/drivers/staging/rtl8188eu/os_dep/os_intfs.c @@ -245,8 +245,7 @@ static unsigned int rtw_classify8021d(struct sk_buff *skb) } static u16 rtw_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { struct adapter *padapter = rtw_netdev_priv(dev); struct mlme_priv *pmlmepriv = &padapter->mlmepriv; diff --git a/drivers/staging/rtl8723bs/os_dep/os_intfs.c b/drivers/staging/rtl8723bs/os_dep/os_intfs.c index 9021bf51960552099ac819cb85ae2988a5dd60a7..8a9d838af24e825640a98cbfe12bd5c105b1d38e 100644 --- a/drivers/staging/rtl8723bs/os_dep/os_intfs.c +++ b/drivers/staging/rtl8723bs/os_dep/os_intfs.c @@ -401,8 +401,7 @@ static unsigned int rtw_classify8021d(struct sk_buff *skb) static u16 rtw_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { struct adapter *padapter = rtw_netdev_priv(dev); struct mlme_priv *pmlmepriv = &padapter->mlmepriv; diff --git a/drivers/target/target_core_user.c b/drivers/target/target_core_user.c index 9704b135a7bc5f10fee7b37140686485ed24d59b..40b29ca5a98d00d040e0f592da11af438d747abd 100644 --- a/drivers/target/target_core_user.c +++ b/drivers/target/target_core_user.c @@ -441,26 +441,26 @@ static int tcmu_genl_set_features(struct sk_buff *skb, struct genl_info *info) static const struct genl_ops tcmu_genl_ops[] = { { .cmd = TCMU_CMD_SET_FEATURES, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = tcmu_attr_policy, .doit = tcmu_genl_set_features, }, { .cmd = TCMU_CMD_ADDED_DEVICE_DONE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = tcmu_attr_policy, .doit = tcmu_genl_add_dev_done, }, { .cmd = TCMU_CMD_REMOVED_DEVICE_DONE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = tcmu_attr_policy, .doit = tcmu_genl_rm_dev_done, }, { .cmd = TCMU_CMD_RECONFIG_DEVICE_DONE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = tcmu_attr_policy, .doit = tcmu_genl_reconfig_dev_done, }, }; @@ -472,6 +472,7 @@ static struct genl_family tcmu_genl_family __ro_after_init = { .name = "TCM-USER", .version = 2, .maxattr = TCMU_ATTR_MAX, + .policy = tcmu_attr_policy, .mcgrps = tcmu_mcgrps, .n_mcgrps = ARRAY_SIZE(tcmu_mcgrps), .netnsok = true, diff --git a/fs/dlm/netlink.c b/fs/dlm/netlink.c index 43a96c330570d68c6b12b6c4701afad30144312b..d8e27defa89f7977fe5786fe36c68e8fe52a77dd 100644 --- a/fs/dlm/netlink.c +++ b/fs/dlm/netlink.c @@ -68,6 +68,7 @@ static int user_cmd(struct sk_buff *skb, struct genl_info *info) static const struct genl_ops dlm_nl_ops[] = { { .cmd = DLM_CMD_HELLO, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = user_cmd, }, }; diff --git a/fs/proc/proc_sysctl.c b/fs/proc/proc_sysctl.c index 7325baa8f9d474f166c1bbef54b584a028b287fb..c74570736b24c3798a9e2f55fd3f953ac28aa130 100644 --- a/fs/proc/proc_sysctl.c +++ b/fs/proc/proc_sysctl.c @@ -13,6 +13,7 @@ #include #include #include +#include #include "internal.h" static const struct dentry_operations proc_sys_dentry_operations; @@ -569,8 +570,8 @@ static ssize_t proc_sys_call_handler(struct file *filp, void __user *buf, struct inode *inode = file_inode(filp); struct ctl_table_header *head = grab_header(inode); struct ctl_table *table = PROC_I(inode)->sysctl_entry; + void *new_buf = NULL; ssize_t error; - size_t res; if (IS_ERR(head)) return PTR_ERR(head); @@ -588,11 +589,27 @@ static ssize_t proc_sys_call_handler(struct file *filp, void __user *buf, if (!table->proc_handler) goto out; + error = BPF_CGROUP_RUN_PROG_SYSCTL(head, table, write, buf, &count, + ppos, &new_buf); + if (error) + goto out; + /* careful: calling conventions are nasty here */ - res = count; - error = table->proc_handler(table, write, buf, &res, ppos); + if (new_buf) { + mm_segment_t old_fs; + + old_fs = get_fs(); + set_fs(KERNEL_DS); + error = table->proc_handler(table, write, (void __user *)new_buf, + &count, ppos); + set_fs(old_fs); + kfree(new_buf); + } else { + error = table->proc_handler(table, write, buf, &count, ppos); + } + if (!error) - error = res; + error = count; out: sysctl_head_finish(head); diff --git a/include/linux/bpf-cgroup.h b/include/linux/bpf-cgroup.h index a4c644c1c09141f8a992286681bd60433d76f057..cb3c6b3b89c802941edea7f732e2009b00a4b356 100644 --- a/include/linux/bpf-cgroup.h +++ b/include/linux/bpf-cgroup.h @@ -17,6 +17,8 @@ struct bpf_map; struct bpf_prog; struct bpf_sock_ops_kern; struct bpf_cgroup_storage; +struct ctl_table; +struct ctl_table_header; #ifdef CONFIG_CGROUP_BPF @@ -109,6 +111,12 @@ int __cgroup_bpf_run_filter_sock_ops(struct sock *sk, int __cgroup_bpf_check_dev_permission(short dev_type, u32 major, u32 minor, short access, enum bpf_attach_type type); +int __cgroup_bpf_run_filter_sysctl(struct ctl_table_header *head, + struct ctl_table *table, int write, + void __user *buf, size_t *pcount, + loff_t *ppos, void **new_buf, + enum bpf_attach_type type); + static inline enum bpf_cgroup_storage_type cgroup_storage_type( struct bpf_map *map) { @@ -253,6 +261,18 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *map, void *key, \ __ret; \ }) + + +#define BPF_CGROUP_RUN_PROG_SYSCTL(head, table, write, buf, count, pos, nbuf) \ +({ \ + int __ret = 0; \ + if (cgroup_bpf_enabled) \ + __ret = __cgroup_bpf_run_filter_sysctl(head, table, write, \ + buf, count, pos, nbuf, \ + BPF_CGROUP_SYSCTL); \ + __ret; \ +}) + int cgroup_bpf_prog_attach(const union bpf_attr *attr, enum bpf_prog_type ptype, struct bpf_prog *prog); int cgroup_bpf_prog_detach(const union bpf_attr *attr, @@ -321,6 +341,7 @@ static inline int bpf_percpu_cgroup_storage_update(struct bpf_map *map, #define BPF_CGROUP_RUN_PROG_UDP6_SENDMSG_LOCK(sk, uaddr, t_ctx) ({ 0; }) #define BPF_CGROUP_RUN_PROG_SOCK_OPS(sock_ops) ({ 0; }) #define BPF_CGROUP_RUN_PROG_DEVICE_CGROUP(type,major,minor,access) ({ 0; }) +#define BPF_CGROUP_RUN_PROG_SYSCTL(head,table,write,buf,count,pos,nbuf) ({ 0; }) #define for_each_cgroup_storage_type(stype) for (; false; ) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 944ccc310201d461e73d738fe8e971aaf9e1b454..59631dd0777c4392a0781b2d74e66c792a731398 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -57,6 +57,12 @@ struct bpf_map_ops { const struct btf *btf, const struct btf_type *key_type, const struct btf_type *value_type); + + /* Direct value access helpers. */ + int (*map_direct_value_addr)(const struct bpf_map *map, + u64 *imm, u32 off); + int (*map_direct_value_meta)(const struct bpf_map *map, + u64 imm, u32 *off); }; struct bpf_map { @@ -81,7 +87,8 @@ struct bpf_map { struct btf *btf; u32 pages; bool unpriv_array; - /* 51 bytes hole */ + bool frozen; /* write-once */ + /* 48 bytes hole */ /* The 3rd and 4th cacheline with misc members to avoid false sharing * particularly with refcounting. @@ -177,6 +184,7 @@ enum bpf_arg_type { ARG_PTR_TO_MAP_KEY, /* pointer to stack used as map key */ ARG_PTR_TO_MAP_VALUE, /* pointer to stack used as map value */ ARG_PTR_TO_UNINIT_MAP_VALUE, /* pointer to valid memory used to store a map value */ + ARG_PTR_TO_MAP_VALUE_OR_NULL, /* pointer to stack used as map value or NULL */ /* the following constraints used to prototype bpf_memcmp() and other * functions that access data on eBPF program stack @@ -195,6 +203,9 @@ enum bpf_arg_type { ARG_ANYTHING, /* any (initialized) argument is ok */ ARG_PTR_TO_SPIN_LOCK, /* pointer to bpf_spin_lock */ ARG_PTR_TO_SOCK_COMMON, /* pointer to sock_common */ + ARG_PTR_TO_INT, /* pointer to int */ + ARG_PTR_TO_LONG, /* pointer to long */ + ARG_PTR_TO_SOCKET, /* pointer to bpf_sock (fullsock) */ }; /* type of values returned from helper functions */ @@ -205,6 +216,7 @@ enum bpf_return_type { RET_PTR_TO_MAP_VALUE_OR_NULL, /* returns a pointer to map elem value or NULL */ RET_PTR_TO_SOCKET_OR_NULL, /* returns a pointer to a socket or NULL */ RET_PTR_TO_TCP_SOCK_OR_NULL, /* returns a pointer to a tcp_sock or NULL */ + RET_PTR_TO_SOCK_COMMON_OR_NULL, /* returns a pointer to a sock_common or NULL */ }; /* eBPF function prototype used by verifier to allow BPF_CALLs from eBPF programs @@ -262,6 +274,7 @@ enum bpf_reg_type { PTR_TO_SOCK_COMMON_OR_NULL, /* reg points to sock_common or NULL */ PTR_TO_TCP_SOCK, /* reg points to struct tcp_sock */ PTR_TO_TCP_SOCK_OR_NULL, /* reg points to struct tcp_sock or NULL */ + PTR_TO_TP_BUFFER, /* reg points to a writable raw tp's buffer */ }; /* The information passed from prog-specific *_is_valid_access @@ -351,6 +364,7 @@ struct bpf_prog_aux { u32 used_map_cnt; u32 max_ctx_offset; u32 max_pkt_offset; + u32 max_tp_access; u32 stack_depth; u32 id; u32 func_cnt; /* used by non-func prog as the number of func progs */ @@ -420,8 +434,38 @@ struct bpf_array { }; }; +#define BPF_COMPLEXITY_LIMIT_INSNS 1000000 /* yes. 1M insns */ #define MAX_TAIL_CALL_CNT 32 +#define BPF_F_ACCESS_MASK (BPF_F_RDONLY | \ + BPF_F_RDONLY_PROG | \ + BPF_F_WRONLY | \ + BPF_F_WRONLY_PROG) + +#define BPF_MAP_CAN_READ BIT(0) +#define BPF_MAP_CAN_WRITE BIT(1) + +static inline u32 bpf_map_flags_to_cap(struct bpf_map *map) +{ + u32 access_flags = map->map_flags & (BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG); + + /* Combination of BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG is + * not possible. + */ + if (access_flags & BPF_F_RDONLY_PROG) + return BPF_MAP_CAN_READ; + else if (access_flags & BPF_F_WRONLY_PROG) + return BPF_MAP_CAN_WRITE; + else + return BPF_MAP_CAN_READ | BPF_MAP_CAN_WRITE; +} + +static inline bool bpf_map_flags_access_ok(u32 access_flags) +{ + return (access_flags & (BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG)) != + (BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG); +} + struct bpf_event_entry { struct perf_event *event; struct file *perf_file; @@ -445,14 +489,6 @@ typedef u32 (*bpf_convert_ctx_access_t)(enum bpf_access_type type, u64 bpf_event_output(struct bpf_map *map, u64 flags, void *meta, u64 meta_size, void *ctx, u64 ctx_size, bpf_ctx_copy_t ctx_copy); -int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr, - union bpf_attr __user *uattr); -int bpf_prog_test_run_skb(struct bpf_prog *prog, const union bpf_attr *kattr, - union bpf_attr __user *uattr); -int bpf_prog_test_run_flow_dissector(struct bpf_prog *prog, - const union bpf_attr *kattr, - union bpf_attr __user *uattr); - /* an array of programs to be executed under rcu_lock. * * Typical usage: @@ -643,6 +679,13 @@ static inline int bpf_map_attr_numa_node(const union bpf_attr *attr) struct bpf_prog *bpf_prog_get_type_path(const char *name, enum bpf_prog_type type); int array_map_alloc_check(union bpf_attr *attr); +int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr, + union bpf_attr __user *uattr); +int bpf_prog_test_run_skb(struct bpf_prog *prog, const union bpf_attr *kattr, + union bpf_attr __user *uattr); +int bpf_prog_test_run_flow_dissector(struct bpf_prog *prog, + const union bpf_attr *kattr, + union bpf_attr __user *uattr); #else /* !CONFIG_BPF_SYSCALL */ static inline struct bpf_prog *bpf_prog_get(u32 ufd) { @@ -754,6 +797,27 @@ static inline struct bpf_prog *bpf_prog_get_type_path(const char *name, { return ERR_PTR(-EOPNOTSUPP); } + +static inline int bpf_prog_test_run_xdp(struct bpf_prog *prog, + const union bpf_attr *kattr, + union bpf_attr __user *uattr) +{ + return -ENOTSUPP; +} + +static inline int bpf_prog_test_run_skb(struct bpf_prog *prog, + const union bpf_attr *kattr, + union bpf_attr __user *uattr) +{ + return -ENOTSUPP; +} + +static inline int bpf_prog_test_run_flow_dissector(struct bpf_prog *prog, + const union bpf_attr *kattr, + union bpf_attr __user *uattr) +{ + return -ENOTSUPP; +} #endif /* CONFIG_BPF_SYSCALL */ static inline struct bpf_prog *bpf_prog_get_type(u32 ufd, @@ -929,6 +993,8 @@ extern const struct bpf_func_proto bpf_sk_redirect_map_proto; extern const struct bpf_func_proto bpf_spin_lock_proto; extern const struct bpf_func_proto bpf_spin_unlock_proto; extern const struct bpf_func_proto bpf_get_local_storage_proto; +extern const struct bpf_func_proto bpf_strtol_proto; +extern const struct bpf_func_proto bpf_strtoul_proto; /* Shared helpers among cBPF and eBPF. */ void bpf_user_rnd_init_once(void); diff --git a/include/linux/bpf_types.h b/include/linux/bpf_types.h index 08bf2f1fe553c59edd50d6f3b530753843330f60..5a9975678d6fcac4d54138826d2de88d2d6ebf0f 100644 --- a/include/linux/bpf_types.h +++ b/include/linux/bpf_types.h @@ -25,9 +25,11 @@ BPF_PROG_TYPE(BPF_PROG_TYPE_KPROBE, kprobe) BPF_PROG_TYPE(BPF_PROG_TYPE_TRACEPOINT, tracepoint) BPF_PROG_TYPE(BPF_PROG_TYPE_PERF_EVENT, perf_event) BPF_PROG_TYPE(BPF_PROG_TYPE_RAW_TRACEPOINT, raw_tracepoint) +BPF_PROG_TYPE(BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE, raw_tracepoint_writable) #endif #ifdef CONFIG_CGROUP_BPF BPF_PROG_TYPE(BPF_PROG_TYPE_CGROUP_DEVICE, cg_dev) +BPF_PROG_TYPE(BPF_PROG_TYPE_CGROUP_SYSCTL, cg_sysctl) #endif #ifdef CONFIG_BPF_LIRC_MODE2 BPF_PROG_TYPE(BPF_PROG_TYPE_LIRC_MODE2, lirc_mode2) @@ -59,6 +61,7 @@ BPF_MAP_TYPE(BPF_MAP_TYPE_ARRAY_OF_MAPS, array_of_maps_map_ops) BPF_MAP_TYPE(BPF_MAP_TYPE_HASH_OF_MAPS, htab_of_maps_map_ops) #ifdef CONFIG_NET BPF_MAP_TYPE(BPF_MAP_TYPE_DEVMAP, dev_map_ops) +BPF_MAP_TYPE(BPF_MAP_TYPE_SK_STORAGE, sk_storage_map_ops) #if defined(CONFIG_BPF_STREAM_PARSER) BPF_MAP_TYPE(BPF_MAP_TYPE_SOCKMAP, sock_map_ops) BPF_MAP_TYPE(BPF_MAP_TYPE_SOCKHASH, sock_hash_ops) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 7d8228d1c8981d9b73fb72a8953687c1f550eb19..1305ccbd8fe6493c8ccbcaf042dbcbd93ae366de 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -207,6 +207,7 @@ struct bpf_verifier_state { struct bpf_verifier_state_list { struct bpf_verifier_state state; struct bpf_verifier_state_list *next; + int miss_cnt, hit_cnt; }; /* Possible states for alu_state member. */ @@ -223,6 +224,10 @@ struct bpf_insn_aux_data { unsigned long map_state; /* pointer/poison value for maps */ s32 call_imm; /* saved imm field of call insn */ u32 alu_limit; /* limit for add/sub register with pointer */ + struct { + u32 map_index; /* index into used_maps[] */ + u32 map_off; /* offset from value base address */ + }; }; int ctx_field_size; /* the ctx field size for load insn, maybe 0 */ int sanitize_stack_off; /* stack slot to be cleared */ @@ -248,6 +253,12 @@ static inline bool bpf_verifier_log_full(const struct bpf_verifier_log *log) return log->len_used >= log->len_total - 1; } +#define BPF_LOG_LEVEL1 1 +#define BPF_LOG_LEVEL2 2 +#define BPF_LOG_STATS 4 +#define BPF_LOG_LEVEL (BPF_LOG_LEVEL1 | BPF_LOG_LEVEL2) +#define BPF_LOG_MASK (BPF_LOG_LEVEL | BPF_LOG_STATS) + static inline bool bpf_verifier_log_needed(const struct bpf_verifier_log *log) { return log->level && log->ubuf && !bpf_verifier_log_full(log); @@ -274,6 +285,7 @@ struct bpf_verifier_env { bool strict_alignment; /* perform strict pointer alignment checks */ struct bpf_verifier_state *cur_state; /* current verifier state */ struct bpf_verifier_state_list **explored_states; /* search pruning optimization */ + struct bpf_verifier_state_list *free_list; struct bpf_map *used_maps[MAX_USED_MAPS]; /* array of map's used by eBPF program */ u32 used_map_cnt; /* number of used maps */ u32 id_gen; /* used to generate unique reg IDs */ @@ -283,7 +295,27 @@ struct bpf_verifier_env { const struct bpf_line_info *prev_linfo; struct bpf_verifier_log log; struct bpf_subprog_info subprog_info[BPF_MAX_SUBPROGS + 1]; + struct { + int *insn_state; + int *insn_stack; + int cur_stack; + } cfg; u32 subprog_cnt; + /* number of instructions analyzed by the verifier */ + u32 insn_processed; + /* total verification time */ + u64 verification_time; + /* maximum number of verifier states kept in 'branching' instructions */ + u32 max_states_per_insn; + /* total number of allocated verifier states */ + u32 total_states; + /* some states are freed during program analysis. + * this is peak number of states. this number dominates kernel + * memory consumption during verification + */ + u32 peak_states; + /* longest register parentage chain walked for liveness marking */ + u32 longest_mark_read_walk; }; __printf(2, 0) void bpf_verifier_vlog(struct bpf_verifier_log *log, diff --git a/include/linux/btf.h b/include/linux/btf.h index 455d31b55828d2240f3e0f27920be11b75dfee57..64cdf2a23d427cf2af3f2fd264d32aeffd6c582b 100644 --- a/include/linux/btf.h +++ b/include/linux/btf.h @@ -51,6 +51,7 @@ bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s, const struct btf_member *m, u32 expected_offset, u32 expected_size); int btf_find_spin_lock(const struct btf *btf, const struct btf_type *t); +bool btf_type_is_void(const struct btf_type *t); #ifdef CONFIG_BPF_SYSCALL const struct btf_type *btf_type_by_id(const struct btf *btf, u32 type_id); diff --git a/include/linux/dsa/8021q.h b/include/linux/dsa/8021q.h new file mode 100644 index 0000000000000000000000000000000000000000..3911e058647889d029d58a9e0d0101661ac529c3 --- /dev/null +++ b/include/linux/dsa/8021q.h @@ -0,0 +1,76 @@ +/* SPDX-License-Identifier: GPL-2.0 + * Copyright (c) 2019, Vladimir Oltean + */ + +#ifndef _NET_DSA_8021Q_H +#define _NET_DSA_8021Q_H + +#include + +struct dsa_switch; +struct sk_buff; +struct net_device; +struct packet_type; + +#if IS_ENABLED(CONFIG_NET_DSA_TAG_8021Q) + +int dsa_port_setup_8021q_tagging(struct dsa_switch *ds, int index, + bool enabled); + +struct sk_buff *dsa_8021q_xmit(struct sk_buff *skb, struct net_device *netdev, + u16 tpid, u16 tci); + +struct sk_buff *dsa_8021q_rcv(struct sk_buff *skb, struct net_device *netdev, + struct packet_type *pt, u16 *tpid, u16 *tci); + +u16 dsa_8021q_tx_vid(struct dsa_switch *ds, int port); + +u16 dsa_8021q_rx_vid(struct dsa_switch *ds, int port); + +int dsa_8021q_rx_switch_id(u16 vid); + +int dsa_8021q_rx_source_port(u16 vid); + +#else + +int dsa_port_setup_8021q_tagging(struct dsa_switch *ds, int index, + bool enabled) +{ + return 0; +} + +struct sk_buff *dsa_8021q_xmit(struct sk_buff *skb, struct net_device *netdev, + u16 tpid, u16 tci) +{ + return NULL; +} + +struct sk_buff *dsa_8021q_rcv(struct sk_buff *skb, struct net_device *netdev, + struct packet_type *pt, u16 *tpid, u16 *tci) +{ + return NULL; +} + +u16 dsa_8021q_tx_vid(struct dsa_switch *ds, int port) +{ + return 0; +} + +u16 dsa_8021q_rx_vid(struct dsa_switch *ds, int port) +{ + return 0; +} + +int dsa_8021q_rx_switch_id(u16 vid) +{ + return 0; +} + +int dsa_8021q_rx_source_port(u16 vid) +{ + return 0; +} + +#endif /* IS_ENABLED(CONFIG_NET_DSA_TAG_8021Q) */ + +#endif /* _NET_DSA_8021Q_H */ diff --git a/include/linux/dsa/sja1105.h b/include/linux/dsa/sja1105.h new file mode 100644 index 0000000000000000000000000000000000000000..603a02e5a8cbcd2da7e91bf991158aeaed24a40f --- /dev/null +++ b/include/linux/dsa/sja1105.h @@ -0,0 +1,40 @@ +/* SPDX-License-Identifier: GPL-2.0 + * Copyright (c) 2019, Vladimir Oltean + */ + +/* Included by drivers/net/dsa/sja1105/sja1105.h and net/dsa/tag_sja1105.c */ + +#ifndef _NET_DSA_SJA1105_H +#define _NET_DSA_SJA1105_H + +#include +#include +#include + +#define ETH_P_SJA1105 ETH_P_DSA_8021Q + +/* IEEE 802.3 Annex 57A: Slow Protocols PDUs (01:80:C2:xx:xx:xx) */ +#define SJA1105_LINKLOCAL_FILTER_A 0x0180C2000000ull +#define SJA1105_LINKLOCAL_FILTER_A_MASK 0xFFFFFF000000ull +/* IEEE 1588 Annex F: Transport of PTP over Ethernet (01:1B:19:xx:xx:xx) */ +#define SJA1105_LINKLOCAL_FILTER_B 0x011B19000000ull +#define SJA1105_LINKLOCAL_FILTER_B_MASK 0xFFFFFF000000ull + +enum sja1105_frame_type { + SJA1105_FRAME_TYPE_NORMAL = 0, + SJA1105_FRAME_TYPE_LINK_LOCAL, +}; + +struct sja1105_skb_cb { + enum sja1105_frame_type type; +}; + +#define SJA1105_SKB_CB(skb) \ + ((struct sja1105_skb_cb *)DSA_SKB_CB_PRIV(skb)) + +struct sja1105_port { + struct dsa_port *dp; + int mgmt_slot; +}; + +#endif /* _NET_DSA_SJA1105_H */ diff --git a/include/linux/etherdevice.h b/include/linux/etherdevice.h index aa8bfd6f738c7fac838b31de87ec390cf0a81d3c..25c0d049336fec3655a2ffbd228c1f83bc00520e 100644 --- a/include/linux/etherdevice.h +++ b/include/linux/etherdevice.h @@ -33,7 +33,7 @@ struct device; int eth_platform_get_mac_address(struct device *dev, u8 *mac_addr); unsigned char *arch_get_platform_mac_address(void); int nvmem_get_mac_address(struct device *dev, void *addrbuf); -u32 eth_get_headlen(void *data, unsigned int max_len); +u32 eth_get_headlen(const struct net_device *dev, void *data, unsigned int len); __be16 eth_type_trans(struct sk_buff *skb, struct net_device *dev); extern const struct header_ops eth_header_ops; diff --git a/include/linux/filter.h b/include/linux/filter.h index 7d3abde3f183cc2a96b88c89adaa72d7f90cc4af..7148bab96943802557d7cddcd1fd763482647ed2 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -34,6 +34,8 @@ struct bpf_prog_aux; struct xdp_rxq_info; struct xdp_buff; struct sock_reuseport; +struct ctl_table; +struct ctl_table_header; /* ArgX, context and stack frame pointer register positions. Note, * Arg1, Arg2, Arg3, etc are used as argument mappings of function @@ -1167,4 +1169,18 @@ struct bpf_sock_ops_kern { */ }; +struct bpf_sysctl_kern { + struct ctl_table_header *head; + struct ctl_table *table; + void *cur_val; + size_t cur_len; + void *new_val; + size_t new_len; + int new_updated; + int write; + loff_t *ppos; + /* Temporary "register" for indirect stores to ppos. */ + u64 tmp_reg; +}; + #endif /* __LINUX_FILTER_H__ */ diff --git a/include/linux/genl_magic_func.h b/include/linux/genl_magic_func.h index 83f81ac5328256c90d408a2ed345a71c938216ee..6cb82301d8e991229955f03af0291cb4aea2c991 100644 --- a/include/linux/genl_magic_func.h +++ b/include/linux/genl_magic_func.h @@ -233,7 +233,6 @@ const char *CONCAT_(GENL_MAGIC_FAMILY, _genl_cmd_to_str)(__u8 cmd) { \ handler \ .cmd = op_name, \ - .policy = CONCAT_(GENL_MAGIC_FAMILY, _tla_nl_policy), \ }, #define ZZZ_genl_ops CONCAT_(GENL_MAGIC_FAMILY, _genl_ops) @@ -290,7 +289,8 @@ static struct genl_family ZZZ_genl_family __ro_after_init = { #ifdef GENL_MAGIC_FAMILY_HDRSZ .hdrsize = NLA_ALIGN(GENL_MAGIC_FAMILY_HDRSZ), #endif - .maxattr = ARRAY_SIZE(drbd_tla_nl_policy)-1, + .maxattr = ARRAY_SIZE(CONCAT_(GENL_MAGIC_FAMILY, _tla_nl_policy))-1, + .policy = CONCAT_(GENL_MAGIC_FAMILY, _tla_nl_policy), .ops = ZZZ_genl_ops, .n_ops = ARRAY_SIZE(ZZZ_genl_ops), .mcgrps = ZZZ_genl_mcgrps, diff --git a/include/linux/ieee80211.h b/include/linux/ieee80211.h index 48703ec60d06ceb6a936aba457f6475d9c661463..61f0a316c6ac9dff9cdd75d9b358770ea07a375b 100644 --- a/include/linux/ieee80211.h +++ b/include/linux/ieee80211.h @@ -1557,7 +1557,7 @@ struct ieee80211_vht_operation { * struct ieee80211_he_cap_elem - HE capabilities element * * This structure is the "HE capabilities element" fixed fields as - * described in P802.11ax_D3.0 section 9.4.2.237.2 and 9.4.2.237.3 + * described in P802.11ax_D4.0 section 9.4.2.242.2 and 9.4.2.242.3 */ struct ieee80211_he_cap_elem { u8 mac_cap_info[6]; @@ -1619,12 +1619,12 @@ struct ieee80211_he_mcs_nss_supp { * struct ieee80211_he_operation - HE capabilities element * * This structure is the "HE operation element" fields as - * described in P802.11ax_D3.0 section 9.4.2.238 + * described in P802.11ax_D4.0 section 9.4.2.243 */ struct ieee80211_he_operation { __le32 he_oper_params; __le16 he_mcs_nss_set; - /* Optional 0,1,3 or 4 bytes: depends on @he_oper_params */ + /* Optional 0,1,3,4,5,7 or 8 bytes: depends on @he_oper_params */ u8 optional[0]; } __packed; @@ -1632,7 +1632,7 @@ struct ieee80211_he_operation { * struct ieee80211_he_mu_edca_param_ac_rec - MU AC Parameter Record field * * This structure is the "MU AC Parameter Record" fields as - * described in P802.11ax_D2.0 section 9.4.2.240 + * described in P802.11ax_D4.0 section 9.4.2.245 */ struct ieee80211_he_mu_edca_param_ac_rec { u8 aifsn; @@ -1644,7 +1644,7 @@ struct ieee80211_he_mu_edca_param_ac_rec { * struct ieee80211_mu_edca_param_set - MU EDCA Parameter Set element * * This structure is the "MU EDCA Parameter Set element" fields as - * described in P802.11ax_D2.0 section 9.4.2.240 + * described in P802.11ax_D4.0 section 9.4.2.245 */ struct ieee80211_mu_edca_param_set { u8 mu_qos_info; @@ -2026,6 +2026,7 @@ ieee80211_he_ppe_size(u8 ppe_thres_hdr, const u8 *phy_cap_info) #define IEEE80211_HE_OPERATION_VHT_OPER_INFO 0x00004000 #define IEEE80211_HE_OPERATION_CO_HOSTED_BSS 0x00008000 #define IEEE80211_HE_OPERATION_ER_SU_DISABLE 0x00010000 +#define IEEE80211_HE_OPERATION_6GHZ_OP_INFO 0x00020000 #define IEEE80211_HE_OPERATION_BSS_COLOR_MASK 0x3f000000 #define IEEE80211_HE_OPERATION_BSS_COLOR_OFFSET 24 #define IEEE80211_HE_OPERATION_PARTIAL_BSS_COLOR 0x40000000 @@ -2056,6 +2057,8 @@ ieee80211_he_oper_size(const u8 *he_oper_ie) oper_len += 3; if (he_oper_params & IEEE80211_HE_OPERATION_CO_HOSTED_BSS) oper_len++; + if (he_oper_params & IEEE80211_HE_OPERATION_6GHZ_OP_INFO) + oper_len += 4; /* Add the first byte (extension ID) to the total length */ oper_len++; @@ -2487,6 +2490,7 @@ enum ieee80211_eid_ext { WLAN_EID_EXT_HE_MU_EDCA = 38, WLAN_EID_EXT_MAX_CHANNEL_SWITCH_TIME = 52, WLAN_EID_EXT_MULTIPLE_BSSID_CONFIGURATION = 55, + WLAN_EID_EXT_NON_INHERITANCE = 56, }; /* Action category code */ diff --git a/include/linux/if_bridge.h b/include/linux/if_bridge.h index 627b788ba0ff8a31e8e602f1f0b4f2ee117082bd..ef0819ced0fc77cd180c3efa716e9a630f3ffd3d 100644 --- a/include/linux/if_bridge.h +++ b/include/linux/if_bridge.h @@ -56,9 +56,6 @@ struct br_ip_list { extern void brioctl_set(int (*ioctl_hook)(struct net *, unsigned int, void __user *)); -typedef int br_should_route_hook_t(struct sk_buff *skb); -extern br_should_route_hook_t __rcu *br_should_route_hook; - #if IS_ENABLED(CONFIG_BRIDGE) && IS_ENABLED(CONFIG_BRIDGE_IGMP_SNOOPING) int br_multicast_list_adjacent(struct net_device *dev, struct list_head *br_ip_list); diff --git a/include/linux/inetdevice.h b/include/linux/inetdevice.h index a64f21a97369a5a7f8eb35c5a5040d359ffd6681..367dc2a0f84a94bf2c0836d839201288c5e196e4 100644 --- a/include/linux/inetdevice.h +++ b/include/linux/inetdevice.h @@ -237,6 +237,20 @@ static inline struct in_device *__in_dev_get_rtnl(const struct net_device *dev) return rtnl_dereference(dev->ip_ptr); } +/* called with rcu_read_lock or rtnl held */ +static inline bool ip_ignore_linkdown(const struct net_device *dev) +{ + struct in_device *in_dev; + bool rc = false; + + in_dev = rcu_dereference_rtnl(dev->ip_ptr); + if (in_dev && + IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN(in_dev)) + rc = true; + + return rc; +} + static inline struct neigh_parms *__in_dev_arp_parms_get_rcu(const struct net_device *dev) { struct in_device *in_dev = __in_dev_get_rcu(dev); diff --git a/include/linux/jiffies.h b/include/linux/jiffies.h index fa928242567db30769839ac8738be5dc58e372ab..1b6d31da7cbc3dd885e43a0c430b8cac890b13dd 100644 --- a/include/linux/jiffies.h +++ b/include/linux/jiffies.h @@ -297,6 +297,7 @@ static inline u64 jiffies_to_nsecs(const unsigned long j) } extern u64 jiffies64_to_nsecs(u64 j); +extern u64 jiffies64_to_msecs(u64 j); extern unsigned long __msecs_to_jiffies(const unsigned int m); #if HZ <= MSEC_PER_SEC && !(MSEC_PER_SEC % HZ) diff --git a/include/linux/marvell_phy.h b/include/linux/marvell_phy.h index 73d04743a2bbe99a002dc1555e76c19c73c6a9f7..af6b11d4d6737845b3c41fd09eab1a674d2253c5 100644 --- a/include/linux/marvell_phy.h +++ b/include/linux/marvell_phy.h @@ -34,5 +34,6 @@ /* struct phy_device dev_flags definitions */ #define MARVELL_PHY_M1145_FLAGS_RESISTANCE 0x00000001 #define MARVELL_PHY_M1118_DNS323_LEDS 0x00000002 +#define MARVELL_PHY_LED0_LINK_LED1_ACTIVE 0x00000004 #endif /* _MARVELL_PHY_H */ diff --git a/include/linux/mdio.h b/include/linux/mdio.h index 3e99ae3ed87f67260aaca4654312ed1ca9f8809d..9dc16d5705a1e2324960d03e8b791e50ac5a5d40 100644 --- a/include/linux/mdio.h +++ b/include/linux/mdio.h @@ -39,7 +39,8 @@ struct mdio_device { /* Bus address of the MDIO device (0-31) */ int addr; int flags; - struct gpio_desc *reset; + struct gpio_desc *reset_gpio; + struct reset_control *reset_ctrl; unsigned int reset_assert_delay; unsigned int reset_deassert_delay; }; diff --git a/include/linux/mlx5/cq.h b/include/linux/mlx5/cq.h index 612c8c2f2466ab98dd634845180028f207f2fd0f..769326ea1d9b5a32d35bfa64126d18fdbef8a6d6 100644 --- a/include/linux/mlx5/cq.h +++ b/include/linux/mlx5/cq.h @@ -170,7 +170,7 @@ static inline void mlx5_cq_arm(struct mlx5_core_cq *cq, u32 cmd, doorbell[0] = cpu_to_be32(sn << 28 | cmd | ci); doorbell[1] = cpu_to_be32(cq->cqn); - mlx5_write64(doorbell, uar_page + MLX5_CQ_DOORBELL, NULL); + mlx5_write64(doorbell, uar_page + MLX5_CQ_DOORBELL); } static inline void mlx5_cq_hold(struct mlx5_core_cq *cq) diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h index f93a5598b942ff4eddb8dcebb218b8a77d0d7332..fc2b6e807f063f88a15ee0c01e77d5656d888d07 100644 --- a/include/linux/mlx5/device.h +++ b/include/linux/mlx5/device.h @@ -361,6 +361,7 @@ enum { enum { MLX5_GENERAL_SUBTYPE_DELAY_DROP_TIMEOUT = 0x1, + MLX5_GENERAL_SUBTYPE_PCI_POWER_CHANGE_EVENT = 0x5, }; enum { @@ -1001,7 +1002,8 @@ enum { MLX5_MATCH_OUTER_HEADERS = 1 << 0, MLX5_MATCH_MISC_PARAMETERS = 1 << 1, MLX5_MATCH_INNER_HEADERS = 1 << 2, - + MLX5_MATCH_MISC_PARAMETERS_2 = 1 << 3, + MLX5_MATCH_MISC_PARAMETERS_3 = 1 << 4, }; enum { @@ -1045,6 +1047,7 @@ enum mlx5_mpls_supported_fields { }; enum mlx5_flex_parser_protos { + MLX5_FLEX_PROTO_GENEVE = 1 << 3, MLX5_FLEX_PROTO_CW_MPLS_GRE = 1 << 4, MLX5_FLEX_PROTO_CW_MPLS_UDP = 1 << 5, }; @@ -1166,6 +1169,12 @@ enum mlx5_qcam_feature_groups { #define MLX5_CAP_FLOWTABLE_SNIFFER_TX_MAX(mdev, cap) \ MLX5_CAP_FLOWTABLE_MAX(mdev, flow_table_properties_nic_transmit_sniffer.cap) +#define MLX5_CAP_FLOWTABLE_RDMA_RX(mdev, cap) \ + MLX5_CAP_FLOWTABLE(mdev, flow_table_properties_nic_receive_rdma.cap) + +#define MLX5_CAP_FLOWTABLE_RDMA_RX_MAX(mdev, cap) \ + MLX5_CAP_FLOWTABLE_MAX(mdev, flow_table_properties_nic_receive_rdma.cap) + #define MLX5_CAP_ESW_FLOWTABLE(mdev, cap) \ MLX5_GET(flow_table_eswitch_cap, \ mdev->caps.hca_cur[MLX5_CAP_ESWITCH_FLOW_TABLE], cap) diff --git a/include/linux/mlx5/doorbell.h b/include/linux/mlx5/doorbell.h index 0787de28f2fcd6372226d7e28c8920c1d7a8d6dc..5c267707e1df7c3efea359743028013ca13c1ac5 100644 --- a/include/linux/mlx5/doorbell.h +++ b/include/linux/mlx5/doorbell.h @@ -36,46 +36,25 @@ #define MLX5_BF_OFFSET 0x800 #define MLX5_CQ_DOORBELL 0x20 -#if BITS_PER_LONG == 64 /* Assume that we can just write a 64-bit doorbell atomically. s390 * actually doesn't have writeq() but S/390 systems don't even have * PCI so we won't worry about it. + * + * Note that the write is not atomic on 32-bit systems! In contrast to 64-bit + * ones, it requires proper locking. mlx5_write64 doesn't do any locking, so use + * it at your own discretion, protected by some kind of lock on 32 bits. + * + * TODO: use write{q,l}_relaxed() */ -#define MLX5_DECLARE_DOORBELL_LOCK(name) -#define MLX5_INIT_DOORBELL_LOCK(ptr) do { } while (0) -#define MLX5_GET_DOORBELL_LOCK(ptr) (NULL) - -static inline void mlx5_write64(__be32 val[2], void __iomem *dest, - spinlock_t *doorbell_lock) +static inline void mlx5_write64(__be32 val[2], void __iomem *dest) { +#if BITS_PER_LONG == 64 __raw_writeq(*(u64 *)val, dest); -} - #else - -/* Just fall back to a spinlock to protect the doorbell if - * BITS_PER_LONG is 32 -- there's no portable way to do atomic 64-bit - * MMIO writes. - */ - -#define MLX5_DECLARE_DOORBELL_LOCK(name) spinlock_t name; -#define MLX5_INIT_DOORBELL_LOCK(ptr) spin_lock_init(ptr) -#define MLX5_GET_DOORBELL_LOCK(ptr) (ptr) - -static inline void mlx5_write64(__be32 val[2], void __iomem *dest, - spinlock_t *doorbell_lock) -{ - unsigned long flags; - - if (doorbell_lock) - spin_lock_irqsave(doorbell_lock, flags); __raw_writel((__force u32) val[0], dest); __raw_writel((__force u32) val[1], dest + 4); - if (doorbell_lock) - spin_unlock_irqrestore(doorbell_lock, flags); -} - #endif +} #endif /* MLX5_DOORBELL_H */ diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index 0d07296488448b28b608cc7795de638aef185ba6..5a39b323c52e9e82b5c455c1723964d4d98a169f 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -56,7 +56,6 @@ enum { MLX5_BOARD_ID_LEN = 64, - MLX5_MAX_NAME_LEN = 16, }; enum { @@ -133,6 +132,7 @@ enum { MLX5_REG_MTRC_CONF = 0x9041, MLX5_REG_MTRC_STDB = 0x9042, MLX5_REG_MTRC_CTRL = 0x9043, + MLX5_REG_MPEIN = 0x9050, MLX5_REG_MPCNT = 0x9051, MLX5_REG_MTPPS = 0x9053, MLX5_REG_MTPPSE = 0x9054, @@ -512,8 +512,13 @@ struct mlx5_rl_table { struct mlx5_rl_entry *rl_entry; }; +struct mlx5_core_roce { + struct mlx5_flow_table *ft; + struct mlx5_flow_group *fg; + struct mlx5_flow_handle *allow_rule; +}; + struct mlx5_priv { - char name[MLX5_MAX_NAME_LEN]; struct mlx5_eq_table *eq_table; /* pages stuff */ @@ -566,6 +571,7 @@ struct mlx5_priv { struct mlx5_lag *lag; struct mlx5_devcom *devcom; unsigned long pci_dev_data; + struct mlx5_core_roce roce; struct mlx5_fc_stats fc_stats; struct mlx5_rl_table rl_table; @@ -642,6 +648,7 @@ struct mlx5_fw_tracer; struct mlx5_vxlan; struct mlx5_core_dev { + struct device *device; struct pci_dev *pdev; /* sync pci state */ struct mutex pci_status_mutex; @@ -662,6 +669,7 @@ struct mlx5_core_dev { u64 sys_image_guid; phys_addr_t iseg_base; struct mlx5_init_seg __iomem *iseg; + phys_addr_t bar_addr; enum mlx5_device_state state; /* sync interface state */ struct mutex intf_state_mutex; @@ -887,6 +895,7 @@ void mlx5_cmd_mbox_status(void *out, u8 *status, u32 *syndrome); int mlx5_core_get_caps(struct mlx5_core_dev *dev, enum mlx5_cap_type cap_type); int mlx5_cmd_alloc_uar(struct mlx5_core_dev *dev, u32 *uarn); int mlx5_cmd_free_uar(struct mlx5_core_dev *dev, u32 uarn); +void mlx5_health_flush(struct mlx5_core_dev *dev); void mlx5_health_cleanup(struct mlx5_core_dev *dev); int mlx5_health_init(struct mlx5_core_dev *dev); void mlx5_start_health_poll(struct mlx5_core_dev *dev); diff --git a/include/linux/mlx5/eswitch.h b/include/linux/mlx5/eswitch.h index 96d8435421de85810815362c89828e11106785b3..0ca77dd1429c07efb12e17d5a0e2311065ec82c9 100644 --- a/include/linux/mlx5/eswitch.h +++ b/include/linux/mlx5/eswitch.h @@ -35,7 +35,7 @@ struct mlx5_eswitch_rep_if { void (*unload)(struct mlx5_eswitch_rep *rep); void *(*get_proto_dev)(struct mlx5_eswitch_rep *rep); void *priv; - u8 state; + atomic_t state; }; struct mlx5_eswitch_rep { diff --git a/include/linux/mlx5/fs.h b/include/linux/mlx5/fs.h index 9df51da04621bc51cea8bc0c4495869bd73c47e6..e690ba0f965ce1f362345414cda0159390c12472 100644 --- a/include/linux/mlx5/fs.h +++ b/include/linux/mlx5/fs.h @@ -73,6 +73,13 @@ enum mlx5_flow_namespace_type { MLX5_FLOW_NAMESPACE_SNIFFER_RX, MLX5_FLOW_NAMESPACE_SNIFFER_TX, MLX5_FLOW_NAMESPACE_EGRESS, + MLX5_FLOW_NAMESPACE_RDMA_RX, +}; + +enum { + FDB_BYPASS_PATH, + FDB_FAST_PATH, + FDB_SLOW_PATH, }; struct mlx5_flow_table; diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h index 3b83288749c6114b22d866cdac86ec4cf4fac0d2..82612741b29e27000592c7816c214320eb556cdc 100644 --- a/include/linux/mlx5/mlx5_ifc.h +++ b/include/linux/mlx5/mlx5_ifc.h @@ -80,6 +80,19 @@ enum { MLX5_SHARED_RESOURCE_UID = 0xffff, }; +enum { + MLX5_OBJ_TYPE_SW_ICM = 0x0008, +}; + +enum { + MLX5_GENERAL_OBJ_TYPES_CAP_SW_ICM = (1ULL << MLX5_OBJ_TYPE_SW_ICM), + MLX5_GENERAL_OBJ_TYPES_CAP_GENEVE_TLV_OPT = (1ULL << 11), +}; + +enum { + MLX5_OBJ_TYPE_GENEVE_TLV_OPT = 0x000b, +}; + enum { MLX5_CMD_OP_QUERY_HCA_CAP = 0x100, MLX5_CMD_OP_QUERY_ADAPTER = 0x101, @@ -299,7 +312,11 @@ struct mlx5_ifc_flow_table_fields_supported_bits { u8 outer_gre_protocol[0x1]; u8 outer_gre_key[0x1]; u8 outer_vxlan_vni[0x1]; - u8 reserved_at_1a[0x5]; + u8 outer_geneve_vni[0x1]; + u8 outer_geneve_oam[0x1]; + u8 outer_geneve_protocol_type[0x1]; + u8 outer_geneve_opt_len[0x1]; + u8 reserved_at_1e[0x1]; u8 source_eswitch_port[0x1]; u8 inner_dmac[0x1]; @@ -327,7 +344,8 @@ struct mlx5_ifc_flow_table_fields_supported_bits { u8 inner_tcp_flags[0x1]; u8 reserved_at_37[0x9]; - u8 reserved_at_40[0x5]; + u8 geneve_tlv_option_0_data[0x1]; + u8 reserved_at_41[0x4]; u8 outer_first_mpls_over_udp[0x4]; u8 outer_first_mpls_over_gre[0x4]; u8 inner_first_mpls[0x4]; @@ -357,11 +375,14 @@ struct mlx5_ifc_flow_table_prop_layout_bits { u8 pop_vlan_2[0x1]; u8 push_vlan_2[0x1]; u8 reformat_and_vlan_action[0x1]; - u8 reserved_at_10[0x2]; + u8 reserved_at_10[0x1]; + u8 sw_owner[0x1]; u8 reformat_l3_tunnel_to_l2[0x1]; u8 reformat_l2_to_l3_tunnel[0x1]; u8 reformat_and_modify_action[0x1]; - u8 reserved_at_15[0xb]; + u8 reserved_at_15[0x2]; + u8 table_miss_action_domain[0x1]; + u8 reserved_at_18[0x8]; u8 reserved_at_20[0x2]; u8 log_max_ft_size[0x6]; u8 log_max_modify_header_context[0x8]; @@ -469,7 +490,9 @@ struct mlx5_ifc_fte_match_set_misc_bits { u8 vxlan_vni[0x18]; u8 reserved_at_b8[0x8]; - u8 reserved_at_c0[0x20]; + u8 geneve_vni[0x18]; + u8 reserved_at_d8[0x7]; + u8 geneve_oam[0x1]; u8 reserved_at_e0[0xc]; u8 outer_ipv6_flow_label[0x14]; @@ -477,7 +500,11 @@ struct mlx5_ifc_fte_match_set_misc_bits { u8 reserved_at_100[0xc]; u8 inner_ipv6_flow_label[0x14]; - u8 reserved_at_120[0x28]; + u8 reserved_at_120[0xa]; + u8 geneve_opt_len[0x6]; + u8 geneve_protocol_type[0x10]; + + u8 reserved_at_140[0x8]; u8 bth_dst_qp[0x18]; u8 reserved_at_160[0x20]; u8 outer_esp_spi[0x20]; @@ -507,6 +534,12 @@ struct mlx5_ifc_fte_match_set_misc2_bits { u8 reserved_at_1a0[0x60]; }; +struct mlx5_ifc_fte_match_set_misc3_bits { + u8 reserved_at_0[0x120]; + u8 geneve_tlv_option_0_data[0x20]; + u8 reserved_at_140[0xc0]; +}; + struct mlx5_ifc_cmd_pas_bits { u8 pa_h[0x20]; @@ -589,7 +622,7 @@ struct mlx5_ifc_flow_table_nic_cap_bits { struct mlx5_ifc_flow_table_prop_layout_bits flow_table_properties_nic_receive; - u8 reserved_at_400[0x200]; + struct mlx5_ifc_flow_table_prop_layout_bits flow_table_properties_nic_receive_rdma; struct mlx5_ifc_flow_table_prop_layout_bits flow_table_properties_nic_receive_sniffer; @@ -770,7 +803,19 @@ struct mlx5_ifc_device_mem_cap_bits { u8 max_memic_size[0x20]; - u8 reserved_at_c0[0x740]; + u8 steering_sw_icm_start_address[0x40]; + + u8 reserved_at_100[0x8]; + u8 log_header_modify_sw_icm_size[0x8]; + u8 reserved_at_110[0x2]; + u8 log_sw_icm_alloc_granularity[0x6]; + u8 log_steering_sw_icm_size[0x8]; + + u8 reserved_at_120[0x20]; + + u8 header_modify_sw_icm_start_address[0x40]; + + u8 reserved_at_180[0x680]; }; enum { @@ -919,6 +964,7 @@ enum { enum { MLX5_UCTX_CAP_RAW_TX = 1UL << 0, + MLX5_UCTX_CAP_INTERNAL_DEV_RES = 1UL << 1, }; struct mlx5_ifc_cmd_hca_cap_bits { @@ -929,7 +975,9 @@ struct mlx5_ifc_cmd_hca_cap_bits { u8 log_max_srq_sz[0x8]; u8 log_max_qp_sz[0x8]; - u8 reserved_at_90[0xb]; + u8 reserved_at_90[0x8]; + u8 prio_tag_required[0x1]; + u8 reserved_at_99[0x2]; u8 log_max_qp[0x5]; u8 reserved_at_a0[0xb]; @@ -1211,7 +1259,11 @@ struct mlx5_ifc_cmd_hca_cap_bits { u8 num_of_uars_per_page[0x20]; u8 flex_parser_protocols[0x20]; - u8 reserved_at_560[0x20]; + + u8 max_geneve_tlv_options[0x8]; + u8 reserved_at_568[0x3]; + u8 max_geneve_tlv_option_data_len[0x5]; + u8 reserved_at_570[0x10]; u8 reserved_at_580[0x3c]; u8 mini_cqe_resp_stride_index[0x1]; @@ -1247,7 +1299,9 @@ struct mlx5_ifc_cmd_hca_cap_bits { u8 uctx_cap[0x20]; - u8 reserved_at_6c0[0x140]; + u8 reserved_at_6c0[0x4]; + u8 flex_parser_id_geneve_tlv_option_0[0x4]; + u8 reserved_at_6c8[0x138]; }; enum mlx5_flow_destination_type { @@ -1260,6 +1314,12 @@ enum mlx5_flow_destination_type { MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE_NUM = 0x101, }; +enum mlx5_flow_table_miss_action { + MLX5_FLOW_TABLE_MISS_ACTION_DEF, + MLX5_FLOW_TABLE_MISS_ACTION_FWD, + MLX5_FLOW_TABLE_MISS_ACTION_SWITCH_DOMAIN, +}; + struct mlx5_ifc_dest_format_struct_bits { u8 destination_type[0x8]; u8 destination_id[0x18]; @@ -1299,7 +1359,9 @@ struct mlx5_ifc_fte_match_param_bits { struct mlx5_ifc_fte_match_set_misc2_bits misc_parameters_2; - u8 reserved_at_800[0x800]; + struct mlx5_ifc_fte_match_set_misc3_bits misc_parameters_3; + + u8 reserved_at_a00[0x600]; }; enum { @@ -2920,6 +2982,7 @@ enum { MLX5_MKC_ACCESS_MODE_MTT = 0x1, MLX5_MKC_ACCESS_MODE_KLMS = 0x2, MLX5_MKC_ACCESS_MODE_KSM = 0x3, + MLX5_MKC_ACCESS_MODE_SW_ICM = 0x4, MLX5_MKC_ACCESS_MODE_MEMIC = 0x5, }; @@ -4807,6 +4870,7 @@ enum { MLX5_QUERY_FLOW_GROUP_OUT_MATCH_CRITERIA_ENABLE_MISC_PARAMETERS = 0x1, MLX5_QUERY_FLOW_GROUP_OUT_MATCH_CRITERIA_ENABLE_INNER_HEADERS = 0x2, MLX5_QUERY_FLOW_GROUP_IN_MATCH_CRITERIA_ENABLE_MISC_PARAMETERS_2 = 0x3, + MLX5_QUERY_FLOW_GROUP_IN_MATCH_CRITERIA_ENABLE_MISC_PARAMETERS_3 = 0x4, }; struct mlx5_ifc_query_flow_group_out_bits { @@ -5110,6 +5174,7 @@ enum { MLX5_ACTION_IN_FIELD_OUT_DIPV6_31_0 = 0x14, MLX5_ACTION_IN_FIELD_OUT_SIPV4 = 0x15, MLX5_ACTION_IN_FIELD_OUT_DIPV4 = 0x16, + MLX5_ACTION_IN_FIELD_OUT_FIRST_VID = 0x17, MLX5_ACTION_IN_FIELD_OUT_IPV6_HOPLIMIT = 0x47, }; @@ -6874,14 +6939,14 @@ struct mlx5_ifc_create_tis_in_bits { struct mlx5_ifc_create_tir_out_bits { u8 status[0x8]; - u8 reserved_at_8[0x18]; + u8 icm_address_63_40[0x18]; u8 syndrome[0x20]; - u8 reserved_at_40[0x8]; + u8 icm_address_39_32[0x8]; u8 tirn[0x18]; - u8 reserved_at_60[0x20]; + u8 icm_address_31_0[0x20]; }; struct mlx5_ifc_create_tir_in_bits { @@ -8026,6 +8091,52 @@ struct mlx5_ifc_ppcnt_reg_bits { union mlx5_ifc_eth_cntrs_grp_data_layout_auto_bits counter_set; }; +struct mlx5_ifc_mpein_reg_bits { + u8 reserved_at_0[0x2]; + u8 depth[0x6]; + u8 pcie_index[0x8]; + u8 node[0x8]; + u8 reserved_at_18[0x8]; + + u8 capability_mask[0x20]; + + u8 reserved_at_40[0x8]; + u8 link_width_enabled[0x8]; + u8 link_speed_enabled[0x10]; + + u8 lane0_physical_position[0x8]; + u8 link_width_active[0x8]; + u8 link_speed_active[0x10]; + + u8 num_of_pfs[0x10]; + u8 num_of_vfs[0x10]; + + u8 bdf0[0x10]; + u8 reserved_at_b0[0x10]; + + u8 max_read_request_size[0x4]; + u8 max_payload_size[0x4]; + u8 reserved_at_c8[0x5]; + u8 pwr_status[0x3]; + u8 port_type[0x4]; + u8 reserved_at_d4[0xb]; + u8 lane_reversal[0x1]; + + u8 reserved_at_e0[0x14]; + u8 pci_power[0xc]; + + u8 reserved_at_100[0x20]; + + u8 device_status[0x10]; + u8 port_state[0x8]; + u8 reserved_at_138[0x8]; + + u8 reserved_at_140[0x10]; + u8 receiver_detect_result[0x10]; + + u8 reserved_at_160[0x20]; +}; + struct mlx5_ifc_mpcnt_reg_bits { u8 reserved_at_0[0x8]; u8 pcie_index[0x8]; @@ -8345,7 +8456,9 @@ struct mlx5_ifc_pcam_reg_bits { }; struct mlx5_ifc_mcam_enhanced_features_bits { - u8 reserved_at_0[0x74]; + u8 reserved_at_0[0x6e]; + u8 pci_status_and_power[0x1]; + u8 reserved_at_6f[0x5]; u8 mark_tx_action_cnp[0x1]; u8 mark_tx_action_cqe[0x1]; u8 dynamic_tx_overflow[0x1]; @@ -8953,6 +9066,7 @@ union mlx5_ifc_ports_control_registers_document_bits { struct mlx5_ifc_pmtu_reg_bits pmtu_reg; struct mlx5_ifc_ppad_reg_bits ppad_reg; struct mlx5_ifc_ppcnt_reg_bits ppcnt_reg; + struct mlx5_ifc_mpein_reg_bits mpein_reg; struct mlx5_ifc_mpcnt_reg_bits mpcnt_reg; struct mlx5_ifc_pplm_reg_bits pplm_reg; struct mlx5_ifc_pplr_reg_bits pplr_reg; @@ -9442,6 +9556,33 @@ struct mlx5_ifc_uctx_bits { u8 reserved_at_20[0x160]; }; +struct mlx5_ifc_sw_icm_bits { + u8 modify_field_select[0x40]; + + u8 reserved_at_40[0x18]; + u8 log_sw_icm_size[0x8]; + + u8 reserved_at_60[0x20]; + + u8 sw_icm_start_addr[0x40]; + + u8 reserved_at_c0[0x140]; +}; + +struct mlx5_ifc_geneve_tlv_option_bits { + u8 modify_field_select[0x40]; + + u8 reserved_at_40[0x18]; + u8 geneve_option_fte_index[0x8]; + + u8 option_class[0x10]; + u8 option_type[0x8]; + u8 reserved_at_78[0x3]; + u8 option_data_length[0x5]; + + u8 reserved_at_80[0x180]; +}; + struct mlx5_ifc_create_umem_in_bits { u8 opcode[0x10]; u8 uid[0x10]; @@ -9479,6 +9620,16 @@ struct mlx5_ifc_destroy_uctx_in_bits { u8 reserved_at_60[0x20]; }; +struct mlx5_ifc_create_sw_icm_in_bits { + struct mlx5_ifc_general_obj_in_cmd_hdr_bits hdr; + struct mlx5_ifc_sw_icm_bits sw_icm; +}; + +struct mlx5_ifc_create_geneve_tlv_option_in_bits { + struct mlx5_ifc_general_obj_in_cmd_hdr_bits hdr; + struct mlx5_ifc_geneve_tlv_option_bits geneve_tlv_opt; +}; + struct mlx5_ifc_mtrc_string_db_param_bits { u8 string_db_base_address[0x20]; diff --git a/include/linux/mlx5/port.h b/include/linux/mlx5/port.h index 64e78394fc9c3e4169fa8e8fecc28f678c73932b..de9a272c9f3d64f560104d269dcd905ace67cc70 100644 --- a/include/linux/mlx5/port.h +++ b/include/linux/mlx5/port.h @@ -60,6 +60,7 @@ enum mlx5_an_status { #define MLX5_I2C_ADDR_LOW 0x50 #define MLX5_I2C_ADDR_HIGH 0x51 #define MLX5_EEPROM_PAGE_LENGTH 256 +#define MLX5_EEPROM_HIGH_PAGE_LENGTH 128 enum mlx5e_link_mode { MLX5E_1000BASE_CX_SGMII = 0, diff --git a/include/linux/mlx5/qp.h b/include/linux/mlx5/qp.h index 0343c81d4c5f5d7d994718c1f645014241314b2a..3ba4edbd17a6713f160d870bb3cd6465823548c0 100644 --- a/include/linux/mlx5/qp.h +++ b/include/linux/mlx5/qp.h @@ -395,6 +395,7 @@ struct mlx5_wqe_signature_seg { struct mlx5_wqe_inline_seg { __be32 byte_count; + __be32 data[0]; }; enum mlx5_sig_type { diff --git a/include/linux/mlx5/transobj.h b/include/linux/mlx5/transobj.h index a261d5528ff7e5b5382576d746d731175ac42e2c..dc6b1e7cb8c427a5eed859432e62ea0ff5d6ee3f 100644 --- a/include/linux/mlx5/transobj.h +++ b/include/linux/mlx5/transobj.h @@ -50,6 +50,9 @@ int mlx5_core_query_sq(struct mlx5_core_dev *dev, u32 sqn, u32 *out); int mlx5_core_query_sq_state(struct mlx5_core_dev *dev, u32 sqn, u8 *state); int mlx5_core_create_tir(struct mlx5_core_dev *dev, u32 *in, int inlen, u32 *tirn); +int mlx5_core_create_tir_out(struct mlx5_core_dev *dev, + u32 *in, int inlen, + u32 *out, int outlen); int mlx5_core_modify_tir(struct mlx5_core_dev *dev, u32 tirn, u32 *in, int inlen); void mlx5_core_destroy_tir(struct mlx5_core_dev *dev, u32 tirn); diff --git a/include/linux/mlx5/vport.h b/include/linux/mlx5/vport.h index 0eef548b994696c69a2c44d0f9e1babd9c9516b5..3d1c6cdbbba72c1124b199e8117b54244193800c 100644 --- a/include/linux/mlx5/vport.h +++ b/include/linux/mlx5/vport.h @@ -118,10 +118,6 @@ int mlx5_modify_nic_vport_promisc(struct mlx5_core_dev *mdev, int promisc_uc, int promisc_mc, int promisc_all); -int mlx5_query_nic_vport_vlans(struct mlx5_core_dev *dev, - u16 vport, - u16 vlans[], - int *size); int mlx5_modify_nic_vport_vlans(struct mlx5_core_dev *dev, u16 vlans[], int list_size); diff --git a/include/linux/mmc/sdio_func.h b/include/linux/mmc/sdio_func.h index 97ca105347a6c5e608297ae1a3562925dc6f6834..5685805533b57c172f57efd423a223c90ee03991 100644 --- a/include/linux/mmc/sdio_func.h +++ b/include/linux/mmc/sdio_func.h @@ -111,6 +111,18 @@ struct sdio_driver { extern int sdio_register_driver(struct sdio_driver *); extern void sdio_unregister_driver(struct sdio_driver *); +/** + * module_sdio_driver() - Helper macro for registering a SDIO driver + * @__sdio_driver: sdio_driver struct + * + * Helper macro for SDIO drivers which do not do anything special in module + * init/exit. This eliminates a lot of boilerplate. Each module may only + * use this macro once, and calling it replaces module_init() and module_exit() + */ +#define module_sdio_driver(__sdio_driver) \ + module_driver(__sdio_driver, sdio_register_driver, \ + sdio_unregister_driver) + /* * SDIO I/O operations */ diff --git a/include/linux/mmc/sdio_ids.h b/include/linux/mmc/sdio_ids.h index 4332199c71c25f46431f253b7693218bbaa8ae23..d1a5d5df02f5743946bdfec56a969ce0c84f4f32 100644 --- a/include/linux/mmc/sdio_ids.h +++ b/include/linux/mmc/sdio_ids.h @@ -59,6 +59,8 @@ #define SDIO_DEVICE_ID_MARVELL_8797_F0 0x9128 #define SDIO_DEVICE_ID_MARVELL_8887WLAN 0x9134 +#define SDIO_VENDOR_ID_MEDIATEK 0x037a + #define SDIO_VENDOR_ID_SIANO 0x039a #define SDIO_DEVICE_ID_SIANO_NOVA_B0 0x0201 #define SDIO_DEVICE_ID_SIANO_NICE 0x0202 diff --git a/include/linux/net.h b/include/linux/net.h index c606c72311d0e08564b274fba3aaa2d2cb3a6e7c..50bf5206ead6d5a7ac1020a437a1ebac220701b0 100644 --- a/include/linux/net.h +++ b/include/linux/net.h @@ -161,6 +161,8 @@ struct proto_ops { int (*compat_ioctl) (struct socket *sock, unsigned int cmd, unsigned long arg); #endif + int (*gettstamp) (struct socket *sock, void __user *userstamp, + bool timeval, bool time32); int (*listen) (struct socket *sock, int len); int (*shutdown) (struct socket *sock, int flags); int (*setsockopt)(struct socket *sock, int level, diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 324e872c91d15407b6804c297de56cd480b215f8..44b47e9df94ac14ebfb80ead15a2cb97e7ce5b60 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -194,8 +194,8 @@ struct net_device_stats { #ifdef CONFIG_RPS #include -extern struct static_key rps_needed; -extern struct static_key rfs_needed; +extern struct static_key_false rps_needed; +extern struct static_key_false rfs_needed; #endif struct neighbour; @@ -914,34 +914,13 @@ struct xfrmdev_ops { }; #endif -#if IS_ENABLED(CONFIG_TLS_DEVICE) -enum tls_offload_ctx_dir { - TLS_OFFLOAD_CTX_DIR_RX, - TLS_OFFLOAD_CTX_DIR_TX, -}; - -struct tls_crypto_info; -struct tls_context; - -struct tlsdev_ops { - int (*tls_dev_add)(struct net_device *netdev, struct sock *sk, - enum tls_offload_ctx_dir direction, - struct tls_crypto_info *crypto_info, - u32 start_offload_tcp_sn); - void (*tls_dev_del)(struct net_device *netdev, - struct tls_context *ctx, - enum tls_offload_ctx_dir direction); - void (*tls_dev_resync_rx)(struct net_device *netdev, - struct sock *sk, u32 seq, u64 rcd_sn); -}; -#endif - struct dev_ifalias { struct rcu_head rcuhead; char ifalias[]; }; struct devlink; +struct tlsdev_ops; /* * This structure defines the management hooks for network devices. @@ -986,8 +965,7 @@ struct devlink; * those the driver believes to be appropriate. * * u16 (*ndo_select_queue)(struct net_device *dev, struct sk_buff *skb, - * struct net_device *sb_dev, - * select_queue_fallback_t fallback); + * struct net_device *sb_dev); * Called to decide which queue to use when device supports multiple * transmit queues. * @@ -1251,8 +1229,8 @@ struct devlink; * that got dropped are freed/returned via xdp_return_frame(). * Returns negative number, means general error invoking ndo, meaning * no frames were xmit'ed and core-caller will free all frames. - * struct devlink *(*ndo_get_devlink)(struct net_device *dev); - * Get devlink instance associated with a given netdev. + * struct devlink_port *(*ndo_get_devlink_port)(struct net_device *dev); + * Get devlink port instance associated with a given netdev. * Called with a reference on the netdevice and devlink locks only, * rtnl_lock is not held. */ @@ -1268,8 +1246,7 @@ struct net_device_ops { netdev_features_t features); u16 (*ndo_select_queue)(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback); + struct net_device *sb_dev); void (*ndo_change_rx_flags)(struct net_device *dev, int flags); void (*ndo_set_rx_mode)(struct net_device *dev); @@ -1453,7 +1430,7 @@ struct net_device_ops { u32 flags); int (*ndo_xsk_async_xmit)(struct net_device *dev, u32 queue_id); - struct devlink * (*ndo_get_devlink)(struct net_device *dev); + struct devlink_port * (*ndo_get_devlink_port)(struct net_device *dev); }; /** @@ -2155,9 +2132,11 @@ static inline void netdev_for_each_tx_queue(struct net_device *dev, &qdisc_xmit_lock_key); \ } -struct netdev_queue *netdev_pick_tx(struct net_device *dev, - struct sk_buff *skb, - struct net_device *sb_dev); +u16 netdev_pick_tx(struct net_device *dev, struct sk_buff *skb, + struct net_device *sb_dev); +struct netdev_queue *netdev_core_pick_tx(struct net_device *dev, + struct sk_buff *skb, + struct net_device *sb_dev); /* returns the headroom that the master device needs to take in account * when forwarding to this dev @@ -2642,11 +2621,9 @@ void dev_close_many(struct list_head *head, bool unlink); void dev_disable_lro(struct net_device *dev); int dev_loopback_xmit(struct net *net, struct sock *sk, struct sk_buff *newskb); u16 dev_pick_tx_zero(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback); + struct net_device *sb_dev); u16 dev_pick_tx_cpu_id(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback); + struct net_device *sb_dev); int dev_queue_xmit(struct sk_buff *skb); int dev_queue_xmit_accel(struct sk_buff *skb, struct net_device *sb_dev); int dev_direct_xmit(struct sk_buff *skb, u16 queue_id); @@ -2664,14 +2641,6 @@ void netdev_freemem(struct net_device *dev); void synchronize_net(void); int init_dummy_netdev(struct net_device *dev); -DECLARE_PER_CPU(int, xmit_recursion); -#define XMIT_RECURSION_LIMIT 10 - -static inline int dev_recursion_level(void) -{ - return this_cpu_read(xmit_recursion); -} - struct net_device *dev_get_by_index(struct net *net, int ifindex); struct net_device *__dev_get_by_index(struct net *net, int ifindex); struct net_device *dev_get_by_index_rcu(struct net *net, int ifindex); @@ -3020,6 +2989,11 @@ struct softnet_data { #ifdef CONFIG_XFRM_OFFLOAD struct sk_buff_head xfrm_backlog; #endif + /* written and read only by owning cpu: */ + struct { + u16 recursion; + u8 more; + } xmit; #ifdef CONFIG_RPS /* input_queue_head should be written by cpu owning this struct, * and only read by other cpus. Worth using a cache line. @@ -3055,6 +3029,28 @@ static inline void input_queue_tail_incr_save(struct softnet_data *sd, DECLARE_PER_CPU_ALIGNED(struct softnet_data, softnet_data); +static inline int dev_recursion_level(void) +{ + return this_cpu_read(softnet_data.xmit.recursion); +} + +#define XMIT_RECURSION_LIMIT 10 +static inline bool dev_xmit_recursion(void) +{ + return unlikely(__this_cpu_read(softnet_data.xmit.recursion) > + XMIT_RECURSION_LIMIT); +} + +static inline void dev_xmit_recursion_inc(void) +{ + __this_cpu_inc(softnet_data.xmit.recursion); +} + +static inline void dev_xmit_recursion_dec(void) +{ + __this_cpu_dec(softnet_data.xmit.recursion); +} + void __netif_schedule(struct Qdisc *q); void netif_schedule_queue(struct netdev_queue *txq); @@ -4410,10 +4406,15 @@ static inline netdev_tx_t __netdev_start_xmit(const struct net_device_ops *ops, struct sk_buff *skb, struct net_device *dev, bool more) { - skb->xmit_more = more ? 1 : 0; + __this_cpu_write(softnet_data.xmit.more, more); return ops->ndo_start_xmit(skb, dev); } +static inline bool netdev_xmit_more(void) +{ + return __this_cpu_read(softnet_data.xmit.more); +} + static inline netdev_tx_t netdev_start_xmit(struct sk_buff *skb, struct net_device *dev, struct netdev_queue *txq, bool more) { diff --git a/include/linux/netfilter.h b/include/linux/netfilter.h index 72cb19c3db6aa99e1779649c7213b6067fd29cda..996bc247ef6e5ed20722f6e41cd52988a5c10240 100644 --- a/include/linux/netfilter.h +++ b/include/linux/netfilter.h @@ -24,20 +24,36 @@ static inline int NF_DROP_GETERR(int verdict) static inline int nf_inet_addr_cmp(const union nf_inet_addr *a1, const union nf_inet_addr *a2) { +#if defined(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) && BITS_PER_LONG == 64 + const unsigned long *ul1 = (const unsigned long *)a1; + const unsigned long *ul2 = (const unsigned long *)a2; + + return ((ul1[0] ^ ul2[0]) | (ul1[1] ^ ul2[1])) == 0UL; +#else return a1->all[0] == a2->all[0] && a1->all[1] == a2->all[1] && a1->all[2] == a2->all[2] && a1->all[3] == a2->all[3]; +#endif } static inline void nf_inet_addr_mask(const union nf_inet_addr *a1, union nf_inet_addr *result, const union nf_inet_addr *mask) { +#if defined(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) && BITS_PER_LONG == 64 + const unsigned long *ua = (const unsigned long *)a1; + unsigned long *ur = (unsigned long *)result; + const unsigned long *um = (const unsigned long *)mask; + + ur[0] = ua[0] & um[0]; + ur[1] = ua[1] & um[1]; +#else result->all[0] = a1->all[0] & mask->all[0]; result->all[1] = a1->all[1] & mask->all[1]; result->all[2] = a1->all[2] & mask->all[2]; result->all[3] = a1->all[3] & mask->all[3]; +#endif } int netfilter_init(void); @@ -360,7 +376,7 @@ extern struct nf_nat_hook __rcu *nf_nat_hook; static inline void nf_nat_decode_session(struct sk_buff *skb, struct flowi *fl, u_int8_t family) { -#ifdef CONFIG_NF_NAT_NEEDED +#if IS_ENABLED(CONFIG_NF_NAT) struct nf_nat_hook *nat_hook; rcu_read_lock(); diff --git a/include/linux/netfilter/ipset/ip_set.h b/include/linux/netfilter/ipset/ip_set.h index f2e1e6b13ca4ce9ca2723211e14a6596016dd427..e499d170f12d5c8495f057e994670f9f75008546 100644 --- a/include/linux/netfilter/ipset/ip_set.h +++ b/include/linux/netfilter/ipset/ip_set.h @@ -401,33 +401,30 @@ ip_set_get_h16(const struct nlattr *attr) return ntohs(nla_get_be16(attr)); } -#define ipset_nest_start(skb, attr) nla_nest_start(skb, attr | NLA_F_NESTED) -#define ipset_nest_end(skb, start) nla_nest_end(skb, start) - static inline int nla_put_ipaddr4(struct sk_buff *skb, int type, __be32 ipaddr) { - struct nlattr *__nested = ipset_nest_start(skb, type); + struct nlattr *__nested = nla_nest_start(skb, type); int ret; if (!__nested) return -EMSGSIZE; ret = nla_put_in_addr(skb, IPSET_ATTR_IPADDR_IPV4, ipaddr); if (!ret) - ipset_nest_end(skb, __nested); + nla_nest_end(skb, __nested); return ret; } static inline int nla_put_ipaddr6(struct sk_buff *skb, int type, const struct in6_addr *ipaddrptr) { - struct nlattr *__nested = ipset_nest_start(skb, type); + struct nlattr *__nested = nla_nest_start(skb, type); int ret; if (!__nested) return -EMSGSIZE; ret = nla_put_in6_addr(skb, IPSET_ATTR_IPADDR_IPV6, ipaddrptr); if (!ret) - ipset_nest_end(skb, __nested); + nla_nest_end(skb, __nested); return ret; } diff --git a/include/linux/netfilter/nfnetlink_osf.h b/include/linux/netfilter/nfnetlink_osf.h index c6000046c96690808525e6e5f534f9902f8900ce..788613f3693590827514c39c0bdc8712a6bc260c 100644 --- a/include/linux/netfilter/nfnetlink_osf.h +++ b/include/linux/netfilter/nfnetlink_osf.h @@ -21,13 +21,18 @@ struct nf_osf_finger { struct nf_osf_user_finger finger; }; +struct nf_osf_data { + const char *genre; + const char *version; +}; + bool nf_osf_match(const struct sk_buff *skb, u_int8_t family, int hooknum, struct net_device *in, struct net_device *out, const struct nf_osf_info *info, struct net *net, const struct list_head *nf_osf_fingers); -const char *nf_osf_find(const struct sk_buff *skb, - const struct list_head *nf_osf_fingers, - const int ttl_check); +bool nf_osf_find(const struct sk_buff *skb, + const struct list_head *nf_osf_fingers, + const int ttl_check, struct nf_osf_data *data); #endif /* _NFOSF_H */ diff --git a/include/linux/netfilter/x_tables.h b/include/linux/netfilter/x_tables.h index bf384b3eedb8da0628258ccace89d2e19c0f5105..1f852ef7b0983cd67b95bf4e833681803397f0e7 100644 --- a/include/linux/netfilter/x_tables.h +++ b/include/linux/netfilter/x_tables.h @@ -317,7 +317,6 @@ struct xt_table_info *xt_replace_table(struct xt_table *table, int *error); struct xt_match *xt_find_match(u8 af, const char *name, u8 revision); -struct xt_target *xt_find_target(u8 af, const char *name, u8 revision); struct xt_match *xt_request_find_match(u8 af, const char *name, u8 revision); struct xt_target *xt_request_find_target(u8 af, const char *name, u8 revision); int xt_find_revision(u8 af, const char *name, u8 revision, int target, diff --git a/include/linux/netfilter_ipv6.h b/include/linux/netfilter_ipv6.h index 471e9467105b32a820fb13b9659d44be4f29aae3..12113e50265639976f8564b19cdae43e4a232309 100644 --- a/include/linux/netfilter_ipv6.h +++ b/include/linux/netfilter_ipv6.h @@ -87,6 +87,21 @@ static inline int nf_ip6_route(struct net *net, struct dst_entry **dst, } int ip6_route_me_harder(struct net *net, struct sk_buff *skb); + +static inline int nf_ip6_route_me_harder(struct net *net, struct sk_buff *skb) +{ +#if IS_MODULE(CONFIG_IPV6) + const struct nf_ipv6_ops *v6_ops = nf_get_ipv6_ops(); + + if (!v6_ops) + return -EHOSTUNREACH; + + return v6_ops->route_me_harder(net, skb); +#else + return ip6_route_me_harder(net, skb); +#endif +} + __sum16 nf_ip6_checksum(struct sk_buff *skb, unsigned int hook, unsigned int dataoff, u_int8_t protocol); diff --git a/include/linux/packing.h b/include/linux/packing.h new file mode 100644 index 0000000000000000000000000000000000000000..54667735cc67588da572105b6cbe8be6b75e0503 --- /dev/null +++ b/include/linux/packing.h @@ -0,0 +1,49 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2016-2018, NXP Semiconductors + * Copyright (c) 2018-2019, Vladimir Oltean + */ +#ifndef _LINUX_PACKING_H +#define _LINUX_PACKING_H + +#include +#include + +#define QUIRK_MSB_ON_THE_RIGHT BIT(0) +#define QUIRK_LITTLE_ENDIAN BIT(1) +#define QUIRK_LSW32_IS_FIRST BIT(2) + +enum packing_op { + PACK, + UNPACK, +}; + +/** + * packing - Convert numbers (currently u64) between a packed and an unpacked + * format. Unpacked means laid out in memory in the CPU's native + * understanding of integers, while packed means anything else that + * requires translation. + * + * @pbuf: Pointer to a buffer holding the packed value. + * @uval: Pointer to an u64 holding the unpacked value. + * @startbit: The index (in logical notation, compensated for quirks) where + * the packed value starts within pbuf. Must be larger than, or + * equal to, endbit. + * @endbit: The index (in logical notation, compensated for quirks) where + * the packed value ends within pbuf. Must be smaller than, or equal + * to, startbit. + * @op: If PACK, then uval will be treated as const pointer and copied (packed) + * into pbuf, between startbit and endbit. + * If UNPACK, then pbuf will be treated as const pointer and the logical + * value between startbit and endbit will be copied (unpacked) to uval. + * @quirks: A bit mask of QUIRK_LITTLE_ENDIAN, QUIRK_LSW32_IS_FIRST and + * QUIRK_MSB_ON_THE_RIGHT. + * + * Return: 0 on success, EINVAL or ERANGE if called incorrectly. Assuming + * correct usage, return code may be discarded. + * If op is PACK, pbuf is modified. + * If op is UNPACK, uval is modified. + */ +int packing(void *pbuf, u64 *uval, int startbit, int endbit, size_t pbuflen, + enum packing_op op, u8 quirks); + +#endif diff --git a/include/linux/phy.h b/include/linux/phy.h index 34084892a466d6350984b69d07bd39cc9ecdb938..073fb151b5a99c7e51dea743496d4b21e1663ae2 100644 --- a/include/linux/phy.h +++ b/include/linux/phy.h @@ -308,13 +308,7 @@ struct phy_device *mdiobus_scan(struct mii_bus *bus, int addr); * * HALTED: PHY is up, but no polling or interrupts are done. Or * PHY is in an error state. - * - * - phy_start moves to RESUMING - * - * RESUMING: PHY was halted, but now wants to run again. - * - If we are forcing, or aneg is done, timer moves to RUNNING - * - If aneg is not done, timer moves to AN - * - phy_stop moves to HALTED + * - phy_start moves to UP */ enum phy_state { PHY_DOWN = 0, @@ -324,7 +318,6 @@ enum phy_state { PHY_RUNNING, PHY_NOLINK, PHY_FORCING, - PHY_RESUMING }; /** @@ -345,6 +338,7 @@ struct phy_c45_device_ids { * is_c45: Set to true if this phy uses clause 45 addressing. * is_internal: Set to true if this phy is internal to a MAC. * is_pseudo_fixed_link: Set to true if this phy is an Ethernet switch, etc. + * is_gigabit_capable: Set to true if PHY supports 1000Mbps * has_fixups: Set to true if this phy has fixups/quirks. * suspended: Set to true if this phy has been suspended successfully. * sysfs_links: Internal boolean tracking sysfs symbolic links setup/removal. @@ -382,6 +376,7 @@ struct phy_device { unsigned is_c45:1; unsigned is_internal:1; unsigned is_pseudo_fixed_link:1; + unsigned is_gigabit_capable:1; unsigned has_fixups:1; unsigned suspended:1; unsigned sysfs_links:1; @@ -390,6 +385,7 @@ struct phy_device { unsigned autoneg:1; /* The most recently read link state */ unsigned link:1; + unsigned autoneg_complete:1; /* Interrupts are enabled */ unsigned interrupts:1; @@ -1075,6 +1071,7 @@ void phy_attached_info(struct phy_device *phydev); /* Clause 22 PHY */ int genphy_config_init(struct phy_device *phydev); +int genphy_read_abilities(struct phy_device *phydev); int genphy_setup_forced(struct phy_device *phydev); int genphy_restart_aneg(struct phy_device *phydev); int genphy_config_eee_advert(struct phy_device *phydev); @@ -1150,6 +1147,7 @@ void phy_request_interrupt(struct phy_device *phydev); void phy_print_status(struct phy_device *phydev); int phy_set_max_speed(struct phy_device *phydev, u32 max_speed); void phy_remove_link_mode(struct phy_device *phydev, u32 link_mode); +void phy_advertise_supported(struct phy_device *phydev); void phy_support_sym_pause(struct phy_device *phydev); void phy_support_asym_pause(struct phy_device *phydev); void phy_set_sym_pause(struct phy_device *phydev, bool rx, bool tx, diff --git a/include/linux/platform_data/macb.h b/include/linux/platform_data/macb.h index 7815d50c26ff4bf021484cf801ee53be0ce09aaf..2bc51b82295612a092a195cd4a044f25e89fe752 100644 --- a/include/linux/platform_data/macb.h +++ b/include/linux/platform_data/macb.h @@ -12,19 +12,10 @@ /** * struct macb_platform_data - platform data for MACB Ethernet - * @phy_mask: phy mask passed when register the MDIO bus - * within the driver - * @phy_irq_pin: PHY IRQ - * @is_rmii: using RMII interface? - * @rev_eth_addr: reverse Ethernet address byte order * @pclk: platform clock * @hclk: AHB clock */ struct macb_platform_data { - u32 phy_mask; - int phy_irq_pin; - u8 is_rmii; - u8 rev_eth_addr; struct clk *pclk; struct clk *hclk; }; diff --git a/include/linux/platform_data/mv88e6xxx.h b/include/linux/platform_data/mv88e6xxx.h index 963730b44aeaae4141c28a649eef2d3576623dfd..21452a9365e1e2999abcbbac3a63086c55e3ba86 100644 --- a/include/linux/platform_data/mv88e6xxx.h +++ b/include/linux/platform_data/mv88e6xxx.h @@ -13,6 +13,7 @@ struct dsa_mv88e6xxx_pdata { unsigned int enabled_ports; struct net_device *netdev; u32 eeprom_len; + int irq; }; #endif diff --git a/include/linux/platform_data/xilinx-ll-temac.h b/include/linux/platform_data/xilinx-ll-temac.h new file mode 100644 index 0000000000000000000000000000000000000000..368530f9817637125e53c5d6d0282dd9c399c014 --- /dev/null +++ b/include/linux/platform_data/xilinx-ll-temac.h @@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __LINUX_XILINX_LL_TEMAC_H +#define __LINUX_XILINX_LL_TEMAC_H + +#include +#include + +struct ll_temac_platform_data { + bool txcsum; /* Enable/disable TX checksum */ + bool rxcsum; /* Enable/disable RX checksum */ + u8 mac_addr[ETH_ALEN]; /* MAC address (6 bytes) */ + /* Clock frequency for input to MDIO clock generator */ + u32 mdio_clk_freq; + unsigned long long mdio_bus_id; /* Unique id for MDIO bus */ + int phy_addr; /* Address of the PHY to connect to */ + phy_interface_t phy_interface; /* PHY interface mode */ + bool reg_little_endian; /* Little endian TEMAC register access */ + bool dma_little_endian; /* Little endian DMA register access */ + /* Pre-initialized mutex to use for synchronizing indirect + * register access. When using both interfaces of a single + * TEMAC IP block, the same mutex should be passed here, as + * they share the same DCR bus bridge. + */ + struct mutex *indirect_mutex; + /* DMA channel control setup */ + u8 tx_irq_timeout; /* TX Interrupt Delay Time-out */ + u8 tx_irq_count; /* TX Interrupt Coalescing Threshold Count */ + u8 rx_irq_timeout; /* RX Interrupt Delay Time-out */ + u8 rx_irq_count; /* RX Interrupt Coalescing Threshold Count */ +}; + +#endif /* __LINUX_XILINX_LL_TEMAC_H */ diff --git a/include/linux/rhashtable-types.h b/include/linux/rhashtable-types.h index 763d613ce2c2f275037c3b1b5ce4d390394295c6..57467cbf4c5b1e0e8e68e440f1feadc45b1e806e 100644 --- a/include/linux/rhashtable-types.h +++ b/include/linux/rhashtable-types.h @@ -48,7 +48,6 @@ typedef int (*rht_obj_cmpfn_t)(struct rhashtable_compare_arg *arg, * @head_offset: Offset of rhash_head in struct to be hashed * @max_size: Maximum size while expanding * @min_size: Minimum size while shrinking - * @locks_mul: Number of bucket locks to allocate per cpu (default: 32) * @automatic_shrinking: Enable automatic shrinking of tables * @hashfn: Hash function (default: jhash2 if !(key_len % 4), or jhash) * @obj_hashfn: Function to hash object @@ -62,7 +61,6 @@ struct rhashtable_params { unsigned int max_size; u16 min_size; bool automatic_shrinking; - u8 locks_mul; rht_hashfn_t hashfn; rht_obj_hashfn_t obj_hashfn; rht_obj_cmpfn_t obj_cmpfn; diff --git a/include/linux/rhashtable.h b/include/linux/rhashtable.h index ae9c0f71f311ebb60bb8bc9dc82202144125f9ba..f7714d3b46bd03139e9072653a22b8750d5ad6e5 100644 --- a/include/linux/rhashtable.h +++ b/include/linux/rhashtable.h @@ -24,12 +24,27 @@ #include #include #include +#include #include /* + * Objects in an rhashtable have an embedded struct rhash_head + * which is linked into as hash chain from the hash table - or one + * of two or more hash tables when the rhashtable is being resized. * The end of the chain is marked with a special nulls marks which has - * the least significant bit set. + * the least significant bit set but otherwise stores the address of + * the hash bucket. This allows us to be be sure we've found the end + * of the right list. + * The value stored in the hash bucket has BIT(0) used as a lock bit. + * This bit must be atomically set before any changes are made to + * the chain. To avoid dereferencing this pointer without clearing + * the bit first, we use an opaque 'struct rhash_lock_head *' for the + * pointer stored in the bucket. This struct needs to be defined so + * that rcu_dereference() works on it, but it has no content so a + * cast is needed for it to be useful. This ensures it isn't + * used by mistake with clearing the lock bit first. */ +struct rhash_lock_head {}; /* Maximum chain length before rehash * @@ -52,8 +67,6 @@ * @nest: Number of bits of first-level nested table. * @rehash: Current bucket being rehashed * @hash_rnd: Random seed to fold into hash - * @locks_mask: Mask to apply before accessing locks[] - * @locks: Array of spinlocks protecting individual buckets * @walkers: List of active walkers * @rcu: RCU structure for freeing the table * @future_tbl: Table under construction during rehashing @@ -63,31 +76,34 @@ struct bucket_table { unsigned int size; unsigned int nest; - unsigned int rehash; u32 hash_rnd; - unsigned int locks_mask; - spinlock_t *locks; struct list_head walkers; struct rcu_head rcu; struct bucket_table __rcu *future_tbl; - struct rhash_head __rcu *buckets[] ____cacheline_aligned_in_smp; + struct lockdep_map dep_map; + + struct rhash_lock_head __rcu *buckets[] ____cacheline_aligned_in_smp; }; /* * NULLS_MARKER() expects a hash value with the low * bits mostly likely to be significant, and it discards * the msb. - * We git it an address, in which the bottom 2 bits are + * We give it an address, in which the bottom bit is * always 0, and the msb might be significant. * So we shift the address down one bit to align with * expectations and avoid losing a significant bit. + * + * We never store the NULLS_MARKER in the hash table + * itself as we need the lsb for locking. + * Instead we store a NULL */ #define RHT_NULLS_MARKER(ptr) \ ((void *)NULLS_MARKER(((unsigned long) (ptr)) >> 1)) #define INIT_RHT_NULLS_HEAD(ptr) \ - ((ptr) = RHT_NULLS_MARKER(&(ptr))) + ((ptr) = NULL) static inline bool rht_is_a_nulls(const struct rhash_head *ptr) { @@ -207,25 +223,6 @@ static inline bool rht_grow_above_max(const struct rhashtable *ht, return atomic_read(&ht->nelems) >= ht->max_elems; } -/* The bucket lock is selected based on the hash and protects mutations - * on a group of hash buckets. - * - * A maximum of tbl->size/2 bucket locks is allocated. This ensures that - * a single lock always covers both buckets which may both contains - * entries which link to the same bucket of the old table during resizing. - * This allows to simplify the locking as locking the bucket in both - * tables during resize always guarantee protection. - * - * IMPORTANT: When holding the bucket lock of both the old and new table - * during expansions and shrinking, the old bucket lock must always be - * acquired first. - */ -static inline spinlock_t *rht_bucket_lock(const struct bucket_table *tbl, - unsigned int hash) -{ - return &tbl->locks[hash & tbl->locks_mask]; -} - #ifdef CONFIG_PROVE_LOCKING int lockdep_rht_mutex_is_held(struct rhashtable *ht); int lockdep_rht_bucket_is_held(const struct bucket_table *tbl, u32 hash); @@ -264,11 +261,13 @@ void rhashtable_free_and_destroy(struct rhashtable *ht, void *arg); void rhashtable_destroy(struct rhashtable *ht); -struct rhash_head __rcu **rht_bucket_nested(const struct bucket_table *tbl, - unsigned int hash); -struct rhash_head __rcu **rht_bucket_nested_insert(struct rhashtable *ht, - struct bucket_table *tbl, +struct rhash_lock_head __rcu **rht_bucket_nested(const struct bucket_table *tbl, + unsigned int hash); +struct rhash_lock_head __rcu **__rht_bucket_nested(const struct bucket_table *tbl, unsigned int hash); +struct rhash_lock_head __rcu **rht_bucket_nested_insert(struct rhashtable *ht, + struct bucket_table *tbl, + unsigned int hash); #define rht_dereference(p, ht) \ rcu_dereference_protected(p, lockdep_rht_mutex_is_held(ht)) @@ -285,37 +284,136 @@ struct rhash_head __rcu **rht_bucket_nested_insert(struct rhashtable *ht, #define rht_entry(tpos, pos, member) \ ({ tpos = container_of(pos, typeof(*tpos), member); 1; }) -static inline struct rhash_head __rcu *const *rht_bucket( +static inline struct rhash_lock_head __rcu *const *rht_bucket( const struct bucket_table *tbl, unsigned int hash) { return unlikely(tbl->nest) ? rht_bucket_nested(tbl, hash) : &tbl->buckets[hash]; } -static inline struct rhash_head __rcu **rht_bucket_var( +static inline struct rhash_lock_head __rcu **rht_bucket_var( struct bucket_table *tbl, unsigned int hash) { - return unlikely(tbl->nest) ? rht_bucket_nested(tbl, hash) : + return unlikely(tbl->nest) ? __rht_bucket_nested(tbl, hash) : &tbl->buckets[hash]; } -static inline struct rhash_head __rcu **rht_bucket_insert( +static inline struct rhash_lock_head __rcu **rht_bucket_insert( struct rhashtable *ht, struct bucket_table *tbl, unsigned int hash) { return unlikely(tbl->nest) ? rht_bucket_nested_insert(ht, tbl, hash) : &tbl->buckets[hash]; } +/* + * We lock a bucket by setting BIT(0) in the pointer - this is always + * zero in real pointers. The NULLS mark is never stored in the bucket, + * rather we store NULL if the bucket is empty. + * bit_spin_locks do not handle contention well, but the whole point + * of the hashtable design is to achieve minimum per-bucket contention. + * A nested hash table might not have a bucket pointer. In that case + * we cannot get a lock. For remove and replace the bucket cannot be + * interesting and doesn't need locking. + * For insert we allocate the bucket if this is the last bucket_table, + * and then take the lock. + * Sometimes we unlock a bucket by writing a new pointer there. In that + * case we don't need to unlock, but we do need to reset state such as + * local_bh. For that we have rht_assign_unlock(). As rcu_assign_pointer() + * provides the same release semantics that bit_spin_unlock() provides, + * this is safe. + * When we write to a bucket without unlocking, we use rht_assign_locked(). + */ + +static inline void rht_lock(struct bucket_table *tbl, + struct rhash_lock_head **bkt) +{ + local_bh_disable(); + bit_spin_lock(0, (unsigned long *)bkt); + lock_map_acquire(&tbl->dep_map); +} + +static inline void rht_lock_nested(struct bucket_table *tbl, + struct rhash_lock_head **bucket, + unsigned int subclass) +{ + local_bh_disable(); + bit_spin_lock(0, (unsigned long *)bucket); + lock_acquire_exclusive(&tbl->dep_map, subclass, 0, NULL, _THIS_IP_); +} + +static inline void rht_unlock(struct bucket_table *tbl, + struct rhash_lock_head **bkt) +{ + lock_map_release(&tbl->dep_map); + bit_spin_unlock(0, (unsigned long *)bkt); + local_bh_enable(); +} + +/* + * Where 'bkt' is a bucket and might be locked: + * rht_ptr() dereferences that pointer and clears the lock bit. + * rht_ptr_exclusive() dereferences in a context where exclusive + * access is guaranteed, such as when destroying the table. + */ +static inline struct rhash_head *rht_ptr( + struct rhash_lock_head __rcu * const *bkt, + struct bucket_table *tbl, + unsigned int hash) +{ + const struct rhash_lock_head *p = + rht_dereference_bucket_rcu(*bkt, tbl, hash); + + if ((((unsigned long)p) & ~BIT(0)) == 0) + return RHT_NULLS_MARKER(bkt); + return (void *)(((unsigned long)p) & ~BIT(0)); +} + +static inline struct rhash_head *rht_ptr_exclusive( + struct rhash_lock_head __rcu * const *bkt) +{ + const struct rhash_lock_head *p = + rcu_dereference_protected(*bkt, 1); + + if (!p) + return RHT_NULLS_MARKER(bkt); + return (void *)(((unsigned long)p) & ~BIT(0)); +} + +static inline void rht_assign_locked(struct rhash_lock_head __rcu **bkt, + struct rhash_head *obj) +{ + struct rhash_head __rcu **p = (struct rhash_head __rcu **)bkt; + + if (rht_is_a_nulls(obj)) + obj = NULL; + rcu_assign_pointer(*p, (void *)((unsigned long)obj | BIT(0))); +} + +static inline void rht_assign_unlock(struct bucket_table *tbl, + struct rhash_lock_head __rcu **bkt, + struct rhash_head *obj) +{ + struct rhash_head __rcu **p = (struct rhash_head __rcu **)bkt; + + if (rht_is_a_nulls(obj)) + obj = NULL; + lock_map_release(&tbl->dep_map); + rcu_assign_pointer(*p, obj); + preempt_enable(); + __release(bitlock); + local_bh_enable(); +} + /** - * rht_for_each_continue - continue iterating over hash chain + * rht_for_each_from - iterate over hash chain from given head * @pos: the &struct rhash_head to use as a loop cursor. - * @head: the previous &struct rhash_head to continue from + * @head: the &struct rhash_head to start from * @tbl: the &struct bucket_table * @hash: the hash value / bucket index */ -#define rht_for_each_continue(pos, head, tbl, hash) \ - for (pos = rht_dereference_bucket(head, tbl, hash); \ - !rht_is_a_nulls(pos); \ +#define rht_for_each_from(pos, head, tbl, hash) \ + for (pos = head; \ + !rht_is_a_nulls(pos); \ pos = rht_dereference_bucket((pos)->next, tbl, hash)) /** @@ -325,19 +423,20 @@ static inline struct rhash_head __rcu **rht_bucket_insert( * @hash: the hash value / bucket index */ #define rht_for_each(pos, tbl, hash) \ - rht_for_each_continue(pos, *rht_bucket(tbl, hash), tbl, hash) + rht_for_each_from(pos, rht_ptr(rht_bucket(tbl, hash), tbl, hash), \ + tbl, hash) /** - * rht_for_each_entry_continue - continue iterating over hash chain + * rht_for_each_entry_from - iterate over hash chain from given head * @tpos: the type * to use as a loop cursor. * @pos: the &struct rhash_head to use as a loop cursor. - * @head: the previous &struct rhash_head to continue from + * @head: the &struct rhash_head to start from * @tbl: the &struct bucket_table * @hash: the hash value / bucket index * @member: name of the &struct rhash_head within the hashable struct. */ -#define rht_for_each_entry_continue(tpos, pos, head, tbl, hash, member) \ - for (pos = rht_dereference_bucket(head, tbl, hash); \ +#define rht_for_each_entry_from(tpos, pos, head, tbl, hash, member) \ + for (pos = head; \ (!rht_is_a_nulls(pos)) && rht_entry(tpos, pos, member); \ pos = rht_dereference_bucket((pos)->next, tbl, hash)) @@ -350,8 +449,9 @@ static inline struct rhash_head __rcu **rht_bucket_insert( * @member: name of the &struct rhash_head within the hashable struct. */ #define rht_for_each_entry(tpos, pos, tbl, hash, member) \ - rht_for_each_entry_continue(tpos, pos, *rht_bucket(tbl, hash), \ - tbl, hash, member) + rht_for_each_entry_from(tpos, pos, \ + rht_ptr(rht_bucket(tbl, hash), tbl, hash), \ + tbl, hash, member) /** * rht_for_each_entry_safe - safely iterate over hash chain of given type @@ -366,7 +466,7 @@ static inline struct rhash_head __rcu **rht_bucket_insert( * remove the loop cursor from the list. */ #define rht_for_each_entry_safe(tpos, pos, next, tbl, hash, member) \ - for (pos = rht_dereference_bucket(*rht_bucket(tbl, hash), tbl, hash), \ + for (pos = rht_ptr(rht_bucket(tbl, hash), tbl, hash), \ next = !rht_is_a_nulls(pos) ? \ rht_dereference_bucket(pos->next, tbl, hash) : NULL; \ (!rht_is_a_nulls(pos)) && rht_entry(tpos, pos, member); \ @@ -375,9 +475,9 @@ static inline struct rhash_head __rcu **rht_bucket_insert( rht_dereference_bucket(pos->next, tbl, hash) : NULL) /** - * rht_for_each_rcu_continue - continue iterating over rcu hash chain + * rht_for_each_rcu_from - iterate over rcu hash chain from given head * @pos: the &struct rhash_head to use as a loop cursor. - * @head: the previous &struct rhash_head to continue from + * @head: the &struct rhash_head to start from * @tbl: the &struct bucket_table * @hash: the hash value / bucket index * @@ -385,9 +485,9 @@ static inline struct rhash_head __rcu **rht_bucket_insert( * the _rcu mutation primitives such as rhashtable_insert() as long as the * traversal is guarded by rcu_read_lock(). */ -#define rht_for_each_rcu_continue(pos, head, tbl, hash) \ +#define rht_for_each_rcu_from(pos, head, tbl, hash) \ for (({barrier(); }), \ - pos = rht_dereference_bucket_rcu(head, tbl, hash); \ + pos = head; \ !rht_is_a_nulls(pos); \ pos = rcu_dereference_raw(pos->next)) @@ -401,14 +501,17 @@ static inline struct rhash_head __rcu **rht_bucket_insert( * the _rcu mutation primitives such as rhashtable_insert() as long as the * traversal is guarded by rcu_read_lock(). */ -#define rht_for_each_rcu(pos, tbl, hash) \ - rht_for_each_rcu_continue(pos, *rht_bucket(tbl, hash), tbl, hash) +#define rht_for_each_rcu(pos, tbl, hash) \ + for (({barrier(); }), \ + pos = rht_ptr(rht_bucket(tbl, hash), tbl, hash); \ + !rht_is_a_nulls(pos); \ + pos = rcu_dereference_raw(pos->next)) /** - * rht_for_each_entry_rcu_continue - continue iterating over rcu hash chain + * rht_for_each_entry_rcu_from - iterated over rcu hash chain from given head * @tpos: the type * to use as a loop cursor. * @pos: the &struct rhash_head to use as a loop cursor. - * @head: the previous &struct rhash_head to continue from + * @head: the &struct rhash_head to start from * @tbl: the &struct bucket_table * @hash: the hash value / bucket index * @member: name of the &struct rhash_head within the hashable struct. @@ -417,9 +520,9 @@ static inline struct rhash_head __rcu **rht_bucket_insert( * the _rcu mutation primitives such as rhashtable_insert() as long as the * traversal is guarded by rcu_read_lock(). */ -#define rht_for_each_entry_rcu_continue(tpos, pos, head, tbl, hash, member) \ +#define rht_for_each_entry_rcu_from(tpos, pos, head, tbl, hash, member) \ for (({barrier(); }), \ - pos = rht_dereference_bucket_rcu(head, tbl, hash); \ + pos = head; \ (!rht_is_a_nulls(pos)) && rht_entry(tpos, pos, member); \ pos = rht_dereference_bucket_rcu(pos->next, tbl, hash)) @@ -436,8 +539,10 @@ static inline struct rhash_head __rcu **rht_bucket_insert( * traversal is guarded by rcu_read_lock(). */ #define rht_for_each_entry_rcu(tpos, pos, tbl, hash, member) \ - rht_for_each_entry_rcu_continue(tpos, pos, *rht_bucket(tbl, hash), \ - tbl, hash, member) + rht_for_each_entry_rcu_from(tpos, pos, \ + rht_ptr(rht_bucket(tbl, hash), \ + tbl, hash), \ + tbl, hash, member) /** * rhl_for_each_rcu - iterate over rcu hash table list @@ -482,7 +587,7 @@ static inline struct rhash_head *__rhashtable_lookup( .ht = ht, .key = key, }; - struct rhash_head __rcu * const *head; + struct rhash_lock_head __rcu * const *bkt; struct bucket_table *tbl; struct rhash_head *he; unsigned int hash; @@ -490,9 +595,9 @@ static inline struct rhash_head *__rhashtable_lookup( tbl = rht_dereference_rcu(ht->tbl, ht); restart: hash = rht_key_hashfn(ht, tbl, key, params); - head = rht_bucket(tbl, hash); + bkt = rht_bucket(tbl, hash); do { - rht_for_each_rcu_continue(he, *head, tbl, hash) { + rht_for_each_rcu_from(he, rht_ptr(bkt, tbl, hash), tbl, hash) { if (params.obj_cmpfn ? params.obj_cmpfn(&arg, rht_obj(ht, he)) : rhashtable_compare(&arg, rht_obj(ht, he))) @@ -502,7 +607,7 @@ restart: /* An object might have been moved to a different hash chain, * while we walk along it - better check and retry. */ - } while (he != RHT_NULLS_MARKER(head)); + } while (he != RHT_NULLS_MARKER(bkt)); /* Ensure we see any new tables. */ smp_rmb(); @@ -598,10 +703,10 @@ static inline void *__rhashtable_insert_fast( .ht = ht, .key = key, }; + struct rhash_lock_head __rcu **bkt; struct rhash_head __rcu **pprev; struct bucket_table *tbl; struct rhash_head *head; - spinlock_t *lock; unsigned int hash; int elasticity; void *data; @@ -610,23 +715,22 @@ static inline void *__rhashtable_insert_fast( tbl = rht_dereference_rcu(ht->tbl, ht); hash = rht_head_hashfn(ht, tbl, obj, params); - lock = rht_bucket_lock(tbl, hash); - spin_lock_bh(lock); + elasticity = RHT_ELASTICITY; + bkt = rht_bucket_insert(ht, tbl, hash); + data = ERR_PTR(-ENOMEM); + if (!bkt) + goto out; + pprev = NULL; + rht_lock(tbl, bkt); if (unlikely(rcu_access_pointer(tbl->future_tbl))) { slow_path: - spin_unlock_bh(lock); + rht_unlock(tbl, bkt); rcu_read_unlock(); return rhashtable_insert_slow(ht, key, obj); } - elasticity = RHT_ELASTICITY; - pprev = rht_bucket_insert(ht, tbl, hash); - data = ERR_PTR(-ENOMEM); - if (!pprev) - goto out; - - rht_for_each_continue(head, *pprev, tbl, hash) { + rht_for_each_from(head, rht_ptr(bkt, tbl, hash), tbl, hash) { struct rhlist_head *plist; struct rhlist_head *list; @@ -642,7 +746,7 @@ slow_path: data = rht_obj(ht, head); if (!rhlist) - goto out; + goto out_unlock; list = container_of(obj, struct rhlist_head, rhead); @@ -651,9 +755,13 @@ slow_path: RCU_INIT_POINTER(list->next, plist); head = rht_dereference_bucket(head->next, tbl, hash); RCU_INIT_POINTER(list->rhead.next, head); - rcu_assign_pointer(*pprev, obj); - - goto good; + if (pprev) { + rcu_assign_pointer(*pprev, obj); + rht_unlock(tbl, bkt); + } else + rht_assign_unlock(tbl, bkt, obj); + data = NULL; + goto out; } if (elasticity <= 0) @@ -661,12 +769,13 @@ slow_path: data = ERR_PTR(-E2BIG); if (unlikely(rht_grow_above_max(ht, tbl))) - goto out; + goto out_unlock; if (unlikely(rht_grow_above_100(ht, tbl))) goto slow_path; - head = rht_dereference_bucket(*pprev, tbl, hash); + /* Inserting at head of list makes unlocking free. */ + head = rht_ptr(bkt, tbl, hash); RCU_INIT_POINTER(obj->next, head); if (rhlist) { @@ -676,20 +785,21 @@ slow_path: RCU_INIT_POINTER(list->next, NULL); } - rcu_assign_pointer(*pprev, obj); - atomic_inc(&ht->nelems); + rht_assign_unlock(tbl, bkt, obj); + if (rht_grow_above_75(ht, tbl)) schedule_work(&ht->run_work); -good: data = NULL; - out: - spin_unlock_bh(lock); rcu_read_unlock(); return data; + +out_unlock: + rht_unlock(tbl, bkt); + goto out; } /** @@ -698,9 +808,9 @@ out: * @obj: pointer to hash head inside object * @params: hash table parameters * - * Will take a per bucket spinlock to protect against mutual mutations + * Will take the per bucket bitlock to protect against mutual mutations * on the same bucket. Multiple insertions may occur in parallel unless - * they map to the same bucket lock. + * they map to the same bucket. * * It is safe to call this function from atomic context. * @@ -727,9 +837,9 @@ static inline int rhashtable_insert_fast( * @list: pointer to hash list head inside object * @params: hash table parameters * - * Will take a per bucket spinlock to protect against mutual mutations + * Will take the per bucket bitlock to protect against mutual mutations * on the same bucket. Multiple insertions may occur in parallel unless - * they map to the same bucket lock. + * they map to the same bucket. * * It is safe to call this function from atomic context. * @@ -750,9 +860,9 @@ static inline int rhltable_insert_key( * @list: pointer to hash list head inside object * @params: hash table parameters * - * Will take a per bucket spinlock to protect against mutual mutations + * Will take the per bucket bitlock to protect against mutual mutations * on the same bucket. Multiple insertions may occur in parallel unless - * they map to the same bucket lock. + * they map to the same bucket. * * It is safe to call this function from atomic context. * @@ -776,12 +886,6 @@ static inline int rhltable_insert( * @obj: pointer to hash head inside object * @params: hash table parameters * - * Locks down the bucket chain in both the old and new table if a resize - * is in progress to ensure that writers can't remove from the old table - * and can't insert to the new table during the atomic operation of search - * and insertion. Searches for duplicates in both the old and new table if - * a resize is in progress. - * * This lookup function may only be used for fixed key hash table (key_len * parameter set). It will BUG() if used inappropriately. * @@ -837,12 +941,6 @@ static inline void *rhashtable_lookup_get_insert_fast( * @obj: pointer to hash head inside object * @params: hash table parameters * - * Locks down the bucket chain in both the old and new table if a resize - * is in progress to ensure that writers can't remove from the old table - * and can't insert to the new table during the atomic operation of search - * and insertion. Searches for duplicates in both the old and new table if - * a resize is in progress. - * * Lookups may occur in parallel with hashtable mutations and resizing. * * Will trigger an automatic deferred table resizing if residency in the @@ -891,19 +989,20 @@ static inline int __rhashtable_remove_fast_one( struct rhash_head *obj, const struct rhashtable_params params, bool rhlist) { + struct rhash_lock_head __rcu **bkt; struct rhash_head __rcu **pprev; struct rhash_head *he; - spinlock_t * lock; unsigned int hash; int err = -ENOENT; hash = rht_head_hashfn(ht, tbl, obj, params); - lock = rht_bucket_lock(tbl, hash); - - spin_lock_bh(lock); + bkt = rht_bucket_var(tbl, hash); + if (!bkt) + return -ENOENT; + pprev = NULL; + rht_lock(tbl, bkt); - pprev = rht_bucket_var(tbl, hash); - rht_for_each_continue(he, *pprev, tbl, hash) { + rht_for_each_from(he, rht_ptr(bkt, tbl, hash), tbl, hash) { struct rhlist_head *list; list = container_of(he, struct rhlist_head, rhead); @@ -943,12 +1042,17 @@ static inline int __rhashtable_remove_fast_one( } } - rcu_assign_pointer(*pprev, obj); - break; + if (pprev) { + rcu_assign_pointer(*pprev, obj); + rht_unlock(tbl, bkt); + } else { + rht_assign_unlock(tbl, bkt, obj); + } + goto unlocked; } - spin_unlock_bh(lock); - + rht_unlock(tbl, bkt); +unlocked: if (err > 0) { atomic_dec(&ht->nelems); if (unlikely(ht->p.automatic_shrinking && @@ -1037,9 +1141,9 @@ static inline int __rhashtable_replace_fast( struct rhash_head *obj_old, struct rhash_head *obj_new, const struct rhashtable_params params) { + struct rhash_lock_head __rcu **bkt; struct rhash_head __rcu **pprev; struct rhash_head *he; - spinlock_t *lock; unsigned int hash; int err = -ENOENT; @@ -1050,25 +1154,33 @@ static inline int __rhashtable_replace_fast( if (hash != rht_head_hashfn(ht, tbl, obj_new, params)) return -EINVAL; - lock = rht_bucket_lock(tbl, hash); + bkt = rht_bucket_var(tbl, hash); + if (!bkt) + return -ENOENT; - spin_lock_bh(lock); + pprev = NULL; + rht_lock(tbl, bkt); - pprev = rht_bucket_var(tbl, hash); - rht_for_each_continue(he, *pprev, tbl, hash) { + rht_for_each_from(he, rht_ptr(bkt, tbl, hash), tbl, hash) { if (he != obj_old) { pprev = &he->next; continue; } rcu_assign_pointer(obj_new->next, obj_old->next); - rcu_assign_pointer(*pprev, obj_new); + if (pprev) { + rcu_assign_pointer(*pprev, obj_new); + rht_unlock(tbl, bkt); + } else { + rht_assign_unlock(tbl, bkt, obj_new); + } err = 0; - break; + goto unlocked; } - spin_unlock_bh(lock); + rht_unlock(tbl, bkt); +unlocked: return err; } diff --git a/include/linux/siphash.h b/include/linux/siphash.h index fa7a6b9cedbffeac141a157790115288132f43e9..bf21591a9e5e653585c26cb3f3f0857256c0eb89 100644 --- a/include/linux/siphash.h +++ b/include/linux/siphash.h @@ -21,6 +21,11 @@ typedef struct { u64 key[2]; } siphash_key_t; +static inline bool siphash_key_is_zero(const siphash_key_t *key) +{ + return !(key->key[0] | key->key[1]); +} + u64 __siphash_aligned(const void *data, size_t len, const siphash_key_t *key); #ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS u64 __siphash_unaligned(const void *data, size_t len, const siphash_key_t *key); diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 9027a8c4219f9a3b0de149ff2974db50471c328a..6d58fa8a65fde76c2b52c2a88e8b1974b87f304c 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -657,7 +657,6 @@ typedef unsigned char *sk_buff_data_t; * @tc_index: Traffic control index * @hash: the packet hash * @queue_mapping: Queue mapping for multiqueue devices - * @xmit_more: More SKBs are pending for this queue * @pfmemalloc: skbuff was allocated from PFMEMALLOC reserves * @active_extensions: active extensions (skb_ext_id types) * @ndisc_nodetype: router type (from link layer) @@ -764,7 +763,6 @@ struct sk_buff { fclone:2, peeked:1, head_frag:1, - xmit_more:1, pfmemalloc:1; #ifdef CONFIG_SKB_EXTENSIONS __u8 active_extensions; @@ -1044,6 +1042,8 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t priority, int flags, int node); struct sk_buff *__build_skb(void *data, unsigned int frag_size); struct sk_buff *build_skb(void *data, unsigned int frag_size); +struct sk_buff *build_skb_around(struct sk_buff *skb, + void *data, unsigned int frag_size); /** * alloc_skb - allocate a network buffer @@ -1258,11 +1258,19 @@ void skb_flow_dissector_init(struct flow_dissector *flow_dissector, unsigned int key_count); #ifdef CONFIG_NET +int skb_flow_dissector_prog_query(const union bpf_attr *attr, + union bpf_attr __user *uattr); int skb_flow_dissector_bpf_prog_attach(const union bpf_attr *attr, struct bpf_prog *prog); int skb_flow_dissector_bpf_prog_detach(const union bpf_attr *attr); #else +static inline int skb_flow_dissector_prog_query(const union bpf_attr *attr, + union bpf_attr __user *uattr) +{ + return -EOPNOTSUPP; +} + static inline int skb_flow_dissector_bpf_prog_attach(const union bpf_attr *attr, struct bpf_prog *prog) { @@ -1275,12 +1283,12 @@ static inline int skb_flow_dissector_bpf_prog_detach(const union bpf_attr *attr) } #endif -struct bpf_flow_keys; -bool __skb_flow_bpf_dissect(struct bpf_prog *prog, - const struct sk_buff *skb, - struct flow_dissector *flow_dissector, - struct bpf_flow_keys *flow_keys); -bool __skb_flow_dissect(const struct sk_buff *skb, +struct bpf_flow_dissector; +bool bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx, + __be16 proto, int nhoff, int hlen); + +bool __skb_flow_dissect(const struct net *net, + const struct sk_buff *skb, struct flow_dissector *flow_dissector, void *target_container, void *data, __be16 proto, int nhoff, int hlen, @@ -1290,8 +1298,8 @@ static inline bool skb_flow_dissect(const struct sk_buff *skb, struct flow_dissector *flow_dissector, void *target_container, unsigned int flags) { - return __skb_flow_dissect(skb, flow_dissector, target_container, - NULL, 0, 0, 0, flags); + return __skb_flow_dissect(NULL, skb, flow_dissector, + target_container, NULL, 0, 0, 0, flags); } static inline bool skb_flow_dissect_flow_keys(const struct sk_buff *skb, @@ -1299,18 +1307,19 @@ static inline bool skb_flow_dissect_flow_keys(const struct sk_buff *skb, unsigned int flags) { memset(flow, 0, sizeof(*flow)); - return __skb_flow_dissect(skb, &flow_keys_dissector, flow, - NULL, 0, 0, 0, flags); + return __skb_flow_dissect(NULL, skb, &flow_keys_dissector, + flow, NULL, 0, 0, 0, flags); } static inline bool -skb_flow_dissect_flow_keys_basic(const struct sk_buff *skb, +skb_flow_dissect_flow_keys_basic(const struct net *net, + const struct sk_buff *skb, struct flow_keys_basic *flow, void *data, __be16 proto, int nhoff, int hlen, unsigned int flags) { memset(flow, 0, sizeof(*flow)); - return __skb_flow_dissect(skb, &flow_keys_basic_dissector, flow, + return __skb_flow_dissect(net, skb, &flow_keys_basic_dissector, flow, data, proto, nhoff, hlen, flags); } @@ -2102,8 +2111,6 @@ void skb_add_rx_frag(struct sk_buff *skb, int i, struct page *page, int off, void skb_coalesce_rx_frag(struct sk_buff *skb, int i, int size, unsigned int truesize); -#define SKB_PAGE_ASSERT(skb) BUG_ON(skb_shinfo(skb)->nr_frags) -#define SKB_FRAG_ASSERT(skb) BUG_ON(skb_has_frag_list(skb)) #define SKB_LINEAR_ASSERT(skb) BUG_ON(skb_is_nonlinear(skb)) #ifdef NET_SKBUFF_DATA_USES_OFFSET @@ -2490,7 +2497,8 @@ static inline void skb_probe_transport_header(struct sk_buff *skb) if (skb_transport_header_was_set(skb)) return; - if (skb_flow_dissect_flow_keys_basic(skb, &keys, NULL, 0, 0, 0, 0)) + if (skb_flow_dissect_flow_keys_basic(NULL, skb, &keys, + NULL, 0, 0, 0, 0)) skb_set_transport_header(skb, keys.control.thoff); } @@ -3372,17 +3380,17 @@ struct sk_buff *__skb_try_recv_from_queue(struct sock *sk, unsigned int flags, void (*destructor)(struct sock *sk, struct sk_buff *skb), - int *peeked, int *off, int *err, + int *off, int *err, struct sk_buff **last); struct sk_buff *__skb_try_recv_datagram(struct sock *sk, unsigned flags, void (*destructor)(struct sock *sk, struct sk_buff *skb), - int *peeked, int *off, int *err, + int *off, int *err, struct sk_buff **last); struct sk_buff *__skb_recv_datagram(struct sock *sk, unsigned flags, void (*destructor)(struct sock *sk, struct sk_buff *skb), - int *peeked, int *off, int *err); + int *off, int *err); struct sk_buff *skb_recv_datagram(struct sock *sk, unsigned flags, int noblock, int *err); __poll_t datagram_poll(struct file *file, struct socket *sock, diff --git a/include/linux/tracepoint-defs.h b/include/linux/tracepoint-defs.h index 49ba9cde7e4bb5e1b6b66f4f94912a65df971906..b29950a19205e7784dab42924bf3465cb636d76c 100644 --- a/include/linux/tracepoint-defs.h +++ b/include/linux/tracepoint-defs.h @@ -45,6 +45,7 @@ struct bpf_raw_event_map { struct tracepoint *tp; void *bpf_func; u32 num_args; + u32 writable_size; } __aligned(32); #endif diff --git a/include/net/addrconf.h b/include/net/addrconf.h index 269ec27385e91f3ba15710b07e5e6caab21f5169..2f67ae854ff01e24bc57221c400613d096916d3b 100644 --- a/include/net/addrconf.h +++ b/include/net/addrconf.h @@ -238,53 +238,6 @@ bool ipv6_chk_mcast_addr(struct net_device *dev, const struct in6_addr *group, void ipv6_mc_dad_complete(struct inet6_dev *idev); -/* A stub used by vxlan module. This is ugly, ideally these - * symbols should be built into the core kernel. - */ -struct ipv6_stub { - int (*ipv6_sock_mc_join)(struct sock *sk, int ifindex, - const struct in6_addr *addr); - int (*ipv6_sock_mc_drop)(struct sock *sk, int ifindex, - const struct in6_addr *addr); - int (*ipv6_dst_lookup)(struct net *net, struct sock *sk, - struct dst_entry **dst, struct flowi6 *fl6); - int (*ipv6_route_input)(struct sk_buff *skb); - - struct fib6_table *(*fib6_get_table)(struct net *net, u32 id); - struct fib6_info *(*fib6_lookup)(struct net *net, int oif, - struct flowi6 *fl6, int flags); - struct fib6_info *(*fib6_table_lookup)(struct net *net, - struct fib6_table *table, - int oif, struct flowi6 *fl6, - int flags); - struct fib6_info *(*fib6_multipath_select)(const struct net *net, - struct fib6_info *f6i, - struct flowi6 *fl6, int oif, - const struct sk_buff *skb, - int strict); - u32 (*ip6_mtu_from_fib6)(struct fib6_info *f6i, struct in6_addr *daddr, - struct in6_addr *saddr); - - void (*udpv6_encap_enable)(void); - void (*ndisc_send_na)(struct net_device *dev, const struct in6_addr *daddr, - const struct in6_addr *solicited_addr, - bool router, bool solicited, bool override, bool inc_opt); - struct neigh_table *nd_tbl; -}; -extern const struct ipv6_stub *ipv6_stub __read_mostly; - -/* A stub used by bpf helpers. Similarly ugly as ipv6_stub */ -struct ipv6_bpf_stub { - int (*inet6_bind)(struct sock *sk, struct sockaddr *uaddr, int addr_len, - bool force_bind_address_no_port, bool with_lock); - struct sock *(*udp6_lib_lookup)(struct net *net, - const struct in6_addr *saddr, __be16 sport, - const struct in6_addr *daddr, __be16 dport, - int dif, int sdif, struct udp_table *tbl, - struct sk_buff *skb); -}; -extern const struct ipv6_bpf_stub *ipv6_bpf_stub __read_mostly; - /* * identify MLD packets for MLD filter exceptions */ @@ -425,6 +378,14 @@ static inline void in6_dev_hold(struct inet6_dev *idev) refcount_inc(&idev->refcnt); } +/* called with rcu_read_lock held */ +static inline bool ip6_ignore_linkdown(const struct net_device *dev) +{ + const struct inet6_dev *idev = __in6_dev_get(dev); + + return !!idev->cnf.ignore_routes_with_linkdown; +} + void inet6_ifa_finish_destroy(struct inet6_ifaddr *ifp); static inline void in6_ifa_put(struct inet6_ifaddr *ifp) diff --git a/include/net/arp.h b/include/net/arp.h index 977aabfcdc03bfe85602fce1becd15f579993f3d..c8f580a0e6b1f5c0853cda1605336fa8eb90917c 100644 --- a/include/net/arp.h +++ b/include/net/arp.h @@ -18,6 +18,7 @@ static inline u32 arp_hashfn(const void *pkey, const struct net_device *dev, u32 return val * hash_rnd[0]; } +#ifdef CONFIG_INET static inline struct neighbour *__ipv4_neigh_lookup_noref(struct net_device *dev, u32 key) { if (dev->flags & (IFF_LOOPBACK | IFF_POINTOPOINT)) @@ -25,6 +26,13 @@ static inline struct neighbour *__ipv4_neigh_lookup_noref(struct net_device *dev return ___neigh_lookup_noref(&arp_tbl, neigh_key_eq32, arp_hashfn, &key, dev); } +#else +static inline +struct neighbour *__ipv4_neigh_lookup_noref(struct net_device *dev, u32 key) +{ + return NULL; +} +#endif static inline struct neighbour *__ipv4_neigh_lookup(struct net_device *dev, u32 key) { diff --git a/include/net/bluetooth/hci.h b/include/net/bluetooth/hci.h index fbba43e9bef5b9959626cac81248a5ac20e0f384..9a5330eed7944fca7a7b013edf1d5d99c4d93128 100644 --- a/include/net/bluetooth/hci.h +++ b/include/net/bluetooth/hci.h @@ -282,6 +282,7 @@ enum { HCI_FORCE_BREDR_SMP, HCI_FORCE_STATIC_ADDR, HCI_LL_RPA_RESOLUTION, + HCI_CMD_PENDING, __HCI_NUM_FLAGS, }; diff --git a/include/net/bluetooth/hci_core.h b/include/net/bluetooth/hci_core.h index 094e61e07030f5f5418f1aae008db4c00e7ee1f3..05b1b96f4d9e43bd97c2c5a3dd09a697c6dd2a92 100644 --- a/include/net/bluetooth/hci_core.h +++ b/include/net/bluetooth/hci_core.h @@ -190,6 +190,9 @@ struct adv_info { #define HCI_MAX_SHORT_NAME_LENGTH 10 +/* Min encryption key size to match with SMP */ +#define HCI_MIN_ENC_KEY_SIZE 7 + /* Default LE RPA expiry time, 15 minutes */ #define HCI_DEFAULT_RPA_TIMEOUT (15 * 60) diff --git a/include/net/bpf_sk_storage.h b/include/net/bpf_sk_storage.h new file mode 100644 index 0000000000000000000000000000000000000000..b9dcb02e756b29d080ba9343f7a9e24178cfad67 --- /dev/null +++ b/include/net/bpf_sk_storage.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2019 Facebook */ +#ifndef _BPF_SK_STORAGE_H +#define _BPF_SK_STORAGE_H + +struct sock; + +void bpf_sk_storage_free(struct sock *sk); + +extern const struct bpf_func_proto bpf_sk_storage_get_proto; +extern const struct bpf_func_proto bpf_sk_storage_delete_proto; + +#endif /* _BPF_SK_STORAGE_H */ diff --git a/include/net/cfg80211.h b/include/net/cfg80211.h index 13bfeb712d36943cf9c04111a777d39cf08034a9..87dae868707e2a7fb92e483651c4e6cb2a524011 100644 --- a/include/net/cfg80211.h +++ b/include/net/cfg80211.h @@ -485,6 +485,7 @@ struct vif_params { * with the get_key() callback, must be in little endian, * length given by @seq_len. * @seq_len: length of @seq. + * @mode: key install mode (RX_TX, NO_TX or SET_TX) */ struct key_params { const u8 *key; @@ -492,6 +493,7 @@ struct key_params { int key_len; int seq_len; u32 cipher; + enum nl80211_key_mode mode; }; /** @@ -973,6 +975,27 @@ enum station_parameters_apply_mask { STATION_PARAM_APPLY_UAPSD = BIT(0), STATION_PARAM_APPLY_CAPABILITY = BIT(1), STATION_PARAM_APPLY_PLINK_STATE = BIT(2), + STATION_PARAM_APPLY_STA_TXPOWER = BIT(3), +}; + +/** + * struct sta_txpwr - station txpower configuration + * + * Used to configure txpower for station. + * + * @power: tx power (in dBm) to be used for sending data traffic. If tx power + * is not provided, the default per-interface tx power setting will be + * overriding. Driver should be picking up the lowest tx power, either tx + * power per-interface or per-station. + * @type: In particular if TPC %type is NL80211_TX_POWER_LIMITED then tx power + * will be less than or equal to specified from userspace, whereas if TPC + * %type is NL80211_TX_POWER_AUTOMATIC then it indicates default tx power. + * NL80211_TX_POWER_FIXED is not a valid configuration option for + * per peer TPC. + */ +struct sta_txpwr { + s16 power; + enum nl80211_tx_power_setting type; }; /** @@ -1047,6 +1070,7 @@ struct station_parameters { const struct ieee80211_he_cap_elem *he_capa; u8 he_capa_len; u16 airtime_weight; + struct sta_txpwr txpwr; }; /** @@ -1327,6 +1351,7 @@ struct cfg80211_tid_stats { * @fcs_err_count: number of packets (MPDUs) received from this station with * an FCS error. This counter should be incremented only when TA of the * received packet with an FCS error matches the peer MAC address. + * @airtime_link_metric: mesh airtime link metric. */ struct station_info { u64 filled; @@ -1381,6 +1406,8 @@ struct station_info { u32 rx_mpdu_count; u32 fcs_err_count; + + u32 airtime_link_metric; }; #if IS_ENABLED(CONFIG_CFG80211) @@ -1832,11 +1859,19 @@ static inline void get_random_mask_addr(u8 *buf, const u8 *addr, const u8 *mask) * @bssid: BSSID to be matched; may be all-zero BSSID in case of SSID match * or no match (RSSI only) * @rssi_thold: don't report scan results below this threshold (in s32 dBm) + * @per_band_rssi_thold: Minimum rssi threshold for each band to be applied + * for filtering out scan results received. Drivers advertize this support + * of band specific rssi based filtering through the feature capability + * %NL80211_EXT_FEATURE_SCHED_SCAN_BAND_SPECIFIC_RSSI_THOLD. These band + * specific rssi thresholds take precedence over rssi_thold, if specified. + * If not specified for any band, it will be assigned with rssi_thold of + * corresponding matchset. */ struct cfg80211_match_set { struct cfg80211_ssid ssid; u8 bssid[ETH_ALEN]; s32 rssi_thold; + s32 per_band_rssi_thold[NUM_NL80211_BANDS]; }; /** @@ -3099,6 +3134,32 @@ struct cfg80211_pmsr_request { struct cfg80211_pmsr_request_peer peers[]; }; +/** + * struct cfg80211_update_owe_info - OWE Information + * + * This structure provides information needed for the drivers to offload OWE + * (Opportunistic Wireless Encryption) processing to the user space. + * + * Commonly used across update_owe_info request and event interfaces. + * + * @peer: MAC address of the peer device for which the OWE processing + * has to be done. + * @status: status code, %WLAN_STATUS_SUCCESS for successful OWE info + * processing, use %WLAN_STATUS_UNSPECIFIED_FAILURE if user space + * cannot give you the real status code for failures. Used only for + * OWE update request command interface (user space to driver). + * @ie: IEs obtained from the peer or constructed by the user space. These are + * the IEs of the remote peer in the event from the host driver and + * the constructed IEs by the user space in the request interface. + * @ie_len: Length of IEs in octets. + */ +struct cfg80211_update_owe_info { + u8 peer[ETH_ALEN] __aligned(2); + u16 status; + const u8 *ie; + size_t ie_len; +}; + /** * struct cfg80211_ops - backend description for wireless configuration * @@ -3436,6 +3497,13 @@ struct cfg80211_pmsr_request { * Statistics should be cumulative, currently no way to reset is provided. * @start_pmsr: start peer measurement (e.g. FTM) * @abort_pmsr: abort peer measurement + * + * @update_owe_info: Provide updated OWE info to driver. Driver implementing SME + * but offloading OWE processing to the user space will get the updated + * DH IE through this interface. + * + * @probe_mesh_link: Probe direct Mesh peer's link quality by sending data frame + * and overrule HWMP path selection algorithm. */ struct cfg80211_ops { int (*suspend)(struct wiphy *wiphy, struct cfg80211_wowlan *wow); @@ -3750,6 +3818,10 @@ struct cfg80211_ops { struct cfg80211_pmsr_request *request); void (*abort_pmsr)(struct wiphy *wiphy, struct wireless_dev *wdev, struct cfg80211_pmsr_request *request); + int (*update_owe_info)(struct wiphy *wiphy, struct net_device *dev, + struct cfg80211_update_owe_info *owe_info); + int (*probe_mesh_link)(struct wiphy *wiphy, struct net_device *dev, + const u8 *buf, size_t len); }; /* @@ -5491,6 +5563,28 @@ static inline void cfg80211_gen_new_bssid(const u8 *bssid, u8 max_bssid, u64_to_ether_addr(new_bssid_u64, new_bssid); } +/** + * cfg80211_is_element_inherited - returns if element ID should be inherited + * @element: element to check + * @non_inherit_element: non inheritance element + */ +bool cfg80211_is_element_inherited(const struct element *element, + const struct element *non_inherit_element); + +/** + * cfg80211_merge_profile - merges a MBSSID profile if it is split between IEs + * @ie: ies + * @ielen: length of IEs + * @mbssid_elem: current MBSSID element + * @sub_elem: current MBSSID subelement (profile) + * @merged_ie: location of the merged profile + * @max_copy_len: max merged profile length + */ +size_t cfg80211_merge_profile(const u8 *ie, size_t ielen, + const struct element *mbssid_elem, + const struct element *sub_elem, + u8 *merged_ie, size_t max_copy_len); + /** * enum cfg80211_bss_frame_type - frame type that the BSS data came from * @CFG80211_BSS_FTYPE_UNKNOWN: driver doesn't know whether the data is @@ -7213,4 +7307,14 @@ void cfg80211_pmsr_complete(struct wireless_dev *wdev, #define wiphy_WARN(wiphy, format, args...) \ WARN(1, "wiphy: %s\n" format, wiphy_name(wiphy), ##args); +/** + * cfg80211_update_owe_info_event - Notify the peer's OWE info to user space + * @netdev: network device + * @owe_info: peer's owe info + * @gfp: allocation flags + */ +void cfg80211_update_owe_info_event(struct net_device *netdev, + struct cfg80211_update_owe_info *owe_info, + gfp_t gfp); + #endif /* __NET_CFG80211_H */ diff --git a/include/net/compat.h b/include/net/compat.h index 4c6d75612b6c725d821068fe6b0dad22d9536f47..f277653c7e1739fb4209b1244ea8499ca50508b3 100644 --- a/include/net/compat.h +++ b/include/net/compat.h @@ -30,9 +30,6 @@ struct compat_cmsghdr { compat_int_t cmsg_type; }; -int compat_sock_get_timestamp(struct sock *, struct timeval __user *); -int compat_sock_get_timestampns(struct sock *, struct timespec __user *); - #else /* defined(CONFIG_COMPAT) */ /* * To avoid compiler warnings: diff --git a/include/net/devlink.h b/include/net/devlink.h index 63de99e09f046e9bc91f9b77aab0051d8919f943..1c4adfb4195a0f8f60a87c7500aec3ac2ab5b3f8 100644 --- a/include/net/devlink.h +++ b/include/net/devlink.h @@ -16,6 +16,7 @@ #include #include #include +#include #include #include @@ -31,6 +32,7 @@ struct devlink { struct list_head region_list; u32 snapshot_id; struct list_head reporter_list; + struct mutex reporters_lock; /* protects reporter_list */ struct devlink_dpipe_headers *dpipe_headers; const struct devlink_ops *ops; struct device *dev; @@ -40,11 +42,13 @@ struct devlink { }; struct devlink_port_attrs { - bool set; + u8 set:1, + split:1, + switch_port:1; enum devlink_port_flavour flavour; u32 port_number; /* same value as "split group" */ - bool split; u32 split_subport_number; + struct netdev_phys_item_id switch_id; }; struct devlink_port { @@ -53,6 +57,9 @@ struct devlink_port { struct devlink *devlink; unsigned index; bool registered; + spinlock_t type_lock; /* Protects type and type_dev + * pointer consistency. + */ enum devlink_port_type type; enum devlink_port_type desired_type; void *type_dev; @@ -485,13 +492,14 @@ struct devlink_ops { struct devlink_sb_pool_info *pool_info); int (*sb_pool_set)(struct devlink *devlink, unsigned int sb_index, u16 pool_index, u32 size, - enum devlink_sb_threshold_type threshold_type); + enum devlink_sb_threshold_type threshold_type, + struct netlink_ext_ack *extack); int (*sb_port_pool_get)(struct devlink_port *devlink_port, unsigned int sb_index, u16 pool_index, u32 *p_threshold); int (*sb_port_pool_set)(struct devlink_port *devlink_port, unsigned int sb_index, u16 pool_index, - u32 threshold); + u32 threshold, struct netlink_ext_ack *extack); int (*sb_tc_pool_bind_get)(struct devlink_port *devlink_port, unsigned int sb_index, u16 tc_index, @@ -501,7 +509,8 @@ struct devlink_ops { unsigned int sb_index, u16 tc_index, enum devlink_sb_pool_type pool_type, - u16 pool_index, u32 threshold); + u16 pool_index, u32 threshold, + struct netlink_ext_ack *extack); int (*sb_occ_snapshot)(struct devlink *devlink, unsigned int sb_index); int (*sb_occ_max_clear)(struct devlink *devlink, @@ -543,19 +552,25 @@ static inline struct devlink *priv_to_devlink(void *priv) return container_of(priv, struct devlink, priv); } +static inline struct devlink_port * +netdev_to_devlink_port(struct net_device *dev) +{ + if (dev->netdev_ops->ndo_get_devlink_port) + return dev->netdev_ops->ndo_get_devlink_port(dev); + return NULL; +} + static inline struct devlink *netdev_to_devlink(struct net_device *dev) { -#if IS_ENABLED(CONFIG_NET_DEVLINK) - if (dev->netdev_ops->ndo_get_devlink) - return dev->netdev_ops->ndo_get_devlink(dev); -#endif + struct devlink_port *devlink_port = netdev_to_devlink_port(dev); + + if (devlink_port) + return devlink_port->devlink; return NULL; } struct ib_device; -#if IS_ENABLED(CONFIG_NET_DEVLINK) - struct devlink *devlink_alloc(const struct devlink_ops *ops, size_t priv_size); int devlink_register(struct devlink *devlink, struct device *dev); void devlink_unregister(struct devlink *devlink); @@ -572,9 +587,9 @@ void devlink_port_type_clear(struct devlink_port *devlink_port); void devlink_port_attrs_set(struct devlink_port *devlink_port, enum devlink_port_flavour flavour, u32 port_number, bool split, - u32 split_subport_number); -int devlink_port_get_phys_port_name(struct devlink_port *devlink_port, - char *name, size_t len); + u32 split_subport_number, + const unsigned char *switch_id, + unsigned char switch_id_len); int devlink_sb_register(struct devlink *devlink, unsigned int sb_index, u32 size, u16 ingress_pools_count, u16 egress_pools_count, u16 ingress_tc_count, @@ -724,510 +739,43 @@ void devlink_health_reporter_state_update(struct devlink_health_reporter *reporter, enum devlink_health_reporter_state state); +#if IS_ENABLED(CONFIG_NET_DEVLINK) + void devlink_compat_running_version(struct net_device *dev, char *buf, size_t len); int devlink_compat_flash_update(struct net_device *dev, const char *file_name); +int devlink_compat_phys_port_name_get(struct net_device *dev, + char *name, size_t len); +int devlink_compat_switch_id_get(struct net_device *dev, + struct netdev_phys_item_id *ppid); #else -static inline struct devlink *devlink_alloc(const struct devlink_ops *ops, - size_t priv_size) -{ - return kzalloc(sizeof(struct devlink) + priv_size, GFP_KERNEL); -} - -static inline int devlink_register(struct devlink *devlink, struct device *dev) -{ - return 0; -} - -static inline void devlink_unregister(struct devlink *devlink) -{ -} - -static inline void devlink_params_publish(struct devlink *devlink) -{ -} - -static inline void devlink_params_unpublish(struct devlink *devlink) -{ -} - -static inline void devlink_free(struct devlink *devlink) -{ - kfree(devlink); -} - -static inline int devlink_port_register(struct devlink *devlink, - struct devlink_port *devlink_port, - unsigned int port_index) -{ - return 0; -} - -static inline void devlink_port_unregister(struct devlink_port *devlink_port) -{ -} - -static inline void devlink_port_type_eth_set(struct devlink_port *devlink_port, - struct net_device *netdev) -{ -} - -static inline void devlink_port_type_ib_set(struct devlink_port *devlink_port, - struct ib_device *ibdev) -{ -} - -static inline void devlink_port_type_clear(struct devlink_port *devlink_port) -{ -} - -static inline void devlink_port_attrs_set(struct devlink_port *devlink_port, - enum devlink_port_flavour flavour, - u32 port_number, bool split, - u32 split_subport_number) -{ -} - -static inline int -devlink_port_get_phys_port_name(struct devlink_port *devlink_port, - char *name, size_t len) -{ - return -EOPNOTSUPP; -} - -static inline int devlink_sb_register(struct devlink *devlink, - unsigned int sb_index, u32 size, - u16 ingress_pools_count, - u16 egress_pools_count, - u16 ingress_tc_count, - u16 egress_tc_count) -{ - return 0; -} - -static inline void devlink_sb_unregister(struct devlink *devlink, - unsigned int sb_index) -{ -} - -static inline int -devlink_dpipe_table_register(struct devlink *devlink, - const char *table_name, - struct devlink_dpipe_table_ops *table_ops, - void *priv, bool counter_control_extern) -{ - return 0; -} - -static inline void devlink_dpipe_table_unregister(struct devlink *devlink, - const char *table_name) -{ -} - -static inline int devlink_dpipe_headers_register(struct devlink *devlink, - struct devlink_dpipe_headers * - dpipe_headers) -{ - return 0; -} - -static inline void devlink_dpipe_headers_unregister(struct devlink *devlink) -{ -} - -static inline bool devlink_dpipe_table_counter_enabled(struct devlink *devlink, - const char *table_name) -{ - return false; -} - -static inline int -devlink_dpipe_entry_ctx_prepare(struct devlink_dpipe_dump_ctx *dump_ctx) -{ - return 0; -} - -static inline int -devlink_dpipe_entry_ctx_append(struct devlink_dpipe_dump_ctx *dump_ctx, - struct devlink_dpipe_entry *entry) -{ - return 0; -} - -static inline int -devlink_dpipe_entry_ctx_close(struct devlink_dpipe_dump_ctx *dump_ctx) -{ - return 0; -} - -static inline void -devlink_dpipe_entry_clear(struct devlink_dpipe_entry *entry) -{ -} - -static inline int -devlink_dpipe_action_put(struct sk_buff *skb, - struct devlink_dpipe_action *action) -{ - return 0; -} - -static inline int -devlink_dpipe_match_put(struct sk_buff *skb, - struct devlink_dpipe_match *match) -{ - return 0; -} - -static inline int -devlink_resource_register(struct devlink *devlink, - const char *resource_name, - u64 resource_size, - u64 resource_id, - u64 parent_resource_id, - const struct devlink_resource_size_params *size_params) -{ - return 0; -} - static inline void -devlink_resources_unregister(struct devlink *devlink, - struct devlink_resource *resource) -{ -} - -static inline int -devlink_resource_size_get(struct devlink *devlink, u64 resource_id, - u64 *p_resource_size) -{ - return -EOPNOTSUPP; -} - -static inline int -devlink_dpipe_table_resource_set(struct devlink *devlink, - const char *table_name, u64 resource_id, - u64 resource_units) -{ - return -EOPNOTSUPP; -} - -static inline void -devlink_resource_occ_get_register(struct devlink *devlink, - u64 resource_id, - devlink_resource_occ_get_t *occ_get, - void *occ_get_priv) -{ -} - -static inline void -devlink_resource_occ_get_unregister(struct devlink *devlink, - u64 resource_id) -{ -} - -static inline int -devlink_params_register(struct devlink *devlink, - const struct devlink_param *params, - size_t params_count) -{ - return 0; -} - -static inline void -devlink_params_unregister(struct devlink *devlink, - const struct devlink_param *params, - size_t params_count) -{ - -} - -static inline int -devlink_port_params_register(struct devlink_port *devlink_port, - const struct devlink_param *params, - size_t params_count) -{ - return 0; -} - -static inline void -devlink_port_params_unregister(struct devlink_port *devlink_port, - const struct devlink_param *params, - size_t params_count) -{ -} - -static inline int -devlink_param_driverinit_value_get(struct devlink *devlink, u32 param_id, - union devlink_param_value *init_val) +devlink_compat_running_version(struct net_device *dev, char *buf, size_t len) { - return -EOPNOTSUPP; } static inline int -devlink_param_driverinit_value_set(struct devlink *devlink, u32 param_id, - union devlink_param_value init_val) +devlink_compat_flash_update(struct net_device *dev, const char *file_name) { return -EOPNOTSUPP; } static inline int -devlink_port_param_driverinit_value_get(struct devlink_port *devlink_port, - u32 param_id, - union devlink_param_value *init_val) +devlink_compat_phys_port_name_get(struct net_device *dev, + char *name, size_t len) { return -EOPNOTSUPP; } static inline int -devlink_port_param_driverinit_value_set(struct devlink_port *devlink_port, - u32 param_id, - union devlink_param_value init_val) +devlink_compat_switch_id_get(struct net_device *dev, + struct netdev_phys_item_id *ppid) { return -EOPNOTSUPP; } -static inline void -devlink_param_value_changed(struct devlink *devlink, u32 param_id) -{ -} - -static inline void -devlink_port_param_value_changed(struct devlink_port *devlink_port, - u32 param_id) -{ -} - -static inline void -devlink_param_value_str_fill(union devlink_param_value *dst_val, - const char *src) -{ -} - -static inline struct devlink_region * -devlink_region_create(struct devlink *devlink, - const char *region_name, - u32 region_max_snapshots, - u64 region_size) -{ - return NULL; -} - -static inline void -devlink_region_destroy(struct devlink_region *region) -{ -} - -static inline u32 -devlink_region_shapshot_id_get(struct devlink *devlink) -{ - return 0; -} - -static inline int -devlink_region_snapshot_create(struct devlink_region *region, u64 data_len, - u8 *data, u32 snapshot_id, - devlink_snapshot_data_dest_t *data_destructor) -{ - return 0; -} - -static inline int -devlink_info_driver_name_put(struct devlink_info_req *req, const char *name) -{ - return 0; -} - -static inline int -devlink_info_serial_number_put(struct devlink_info_req *req, const char *sn) -{ - return 0; -} - -static inline int -devlink_info_version_fixed_put(struct devlink_info_req *req, - const char *version_name, - const char *version_value) -{ - return 0; -} - -static inline int -devlink_info_version_stored_put(struct devlink_info_req *req, - const char *version_name, - const char *version_value) -{ - return 0; -} - -static inline int -devlink_info_version_running_put(struct devlink_info_req *req, - const char *version_name, - const char *version_value) -{ - return 0; -} - -static inline int -devlink_fmsg_obj_nest_start(struct devlink_fmsg *fmsg) -{ - return 0; -} - -static inline int -devlink_fmsg_obj_nest_end(struct devlink_fmsg *fmsg) -{ - return 0; -} - -static inline int -devlink_fmsg_pair_nest_start(struct devlink_fmsg *fmsg, const char *name) -{ - return 0; -} - -static inline int -devlink_fmsg_pair_nest_end(struct devlink_fmsg *fmsg) -{ - return 0; -} - -static inline int -devlink_fmsg_arr_pair_nest_start(struct devlink_fmsg *fmsg, - const char *name) -{ - return 0; -} - -static inline int -devlink_fmsg_arr_pair_nest_end(struct devlink_fmsg *fmsg) -{ - return 0; -} - -static inline int -devlink_fmsg_bool_put(struct devlink_fmsg *fmsg, bool value) -{ - return 0; -} - -static inline int -devlink_fmsg_u8_put(struct devlink_fmsg *fmsg, u8 value) -{ - return 0; -} - -static inline int -devlink_fmsg_u32_put(struct devlink_fmsg *fmsg, u32 value) -{ - return 0; -} - -static inline int -devlink_fmsg_u64_put(struct devlink_fmsg *fmsg, u64 value) -{ - return 0; -} - -static inline int -devlink_fmsg_string_put(struct devlink_fmsg *fmsg, const char *value) -{ - return 0; -} - -static inline int -devlink_fmsg_binary_put(struct devlink_fmsg *fmsg, const void *value, - u16 value_len) -{ - return 0; -} - -static inline int -devlink_fmsg_bool_pair_put(struct devlink_fmsg *fmsg, const char *name, - bool value) -{ - return 0; -} - -static inline int -devlink_fmsg_u8_pair_put(struct devlink_fmsg *fmsg, const char *name, - u8 value) -{ - return 0; -} - -static inline int -devlink_fmsg_u32_pair_put(struct devlink_fmsg *fmsg, const char *name, - u32 value) -{ - return 0; -} - -static inline int -devlink_fmsg_u64_pair_put(struct devlink_fmsg *fmsg, const char *name, - u64 value) -{ - return 0; -} - -static inline int -devlink_fmsg_string_pair_put(struct devlink_fmsg *fmsg, const char *name, - const char *value) -{ - return 0; -} - -static inline int -devlink_fmsg_binary_pair_put(struct devlink_fmsg *fmsg, const char *name, - const void *value, u16 value_len) -{ - return 0; -} - -static inline struct devlink_health_reporter * -devlink_health_reporter_create(struct devlink *devlink, - const struct devlink_health_reporter_ops *ops, - u64 graceful_period, bool auto_recover, - void *priv) -{ - return NULL; -} - -static inline void -devlink_health_reporter_destroy(struct devlink_health_reporter *reporter) -{ -} - -static inline void * -devlink_health_reporter_priv(struct devlink_health_reporter *reporter) -{ - return NULL; -} - -static inline int -devlink_health_report(struct devlink_health_reporter *reporter, - const char *msg, void *priv_ctx) -{ - return 0; -} - -static inline void -devlink_health_reporter_state_update(struct devlink_health_reporter *reporter, - enum devlink_health_reporter_state state) -{ -} - -static inline void -devlink_compat_running_version(struct net_device *dev, char *buf, size_t len) -{ -} - -static inline int -devlink_compat_flash_update(struct net_device *dev, const char *file_name) -{ - return -EOPNOTSUPP; -} #endif #endif /* _NET_DEVLINK_H_ */ diff --git a/include/net/dsa.h b/include/net/dsa.h index ae480bba11f58e28435bd45c18632d2d5614a661..6aaaadd6a413c9bf1d8600325d171fb69006d3be 100644 --- a/include/net/dsa.h +++ b/include/net/dsa.h @@ -30,20 +30,36 @@ struct phy_device; struct fixed_phy_status; struct phylink_link_state; +#define DSA_TAG_PROTO_NONE_VALUE 0 +#define DSA_TAG_PROTO_BRCM_VALUE 1 +#define DSA_TAG_PROTO_BRCM_PREPEND_VALUE 2 +#define DSA_TAG_PROTO_DSA_VALUE 3 +#define DSA_TAG_PROTO_EDSA_VALUE 4 +#define DSA_TAG_PROTO_GSWIP_VALUE 5 +#define DSA_TAG_PROTO_KSZ9477_VALUE 6 +#define DSA_TAG_PROTO_KSZ9893_VALUE 7 +#define DSA_TAG_PROTO_LAN9303_VALUE 8 +#define DSA_TAG_PROTO_MTK_VALUE 9 +#define DSA_TAG_PROTO_QCA_VALUE 10 +#define DSA_TAG_PROTO_TRAILER_VALUE 11 +#define DSA_TAG_PROTO_8021Q_VALUE 12 +#define DSA_TAG_PROTO_SJA1105_VALUE 13 + enum dsa_tag_protocol { - DSA_TAG_PROTO_NONE = 0, - DSA_TAG_PROTO_BRCM, - DSA_TAG_PROTO_BRCM_PREPEND, - DSA_TAG_PROTO_DSA, - DSA_TAG_PROTO_EDSA, - DSA_TAG_PROTO_GSWIP, - DSA_TAG_PROTO_KSZ9477, - DSA_TAG_PROTO_KSZ9893, - DSA_TAG_PROTO_LAN9303, - DSA_TAG_PROTO_MTK, - DSA_TAG_PROTO_QCA, - DSA_TAG_PROTO_TRAILER, - DSA_TAG_LAST, /* MUST BE LAST */ + DSA_TAG_PROTO_NONE = DSA_TAG_PROTO_NONE_VALUE, + DSA_TAG_PROTO_BRCM = DSA_TAG_PROTO_BRCM_VALUE, + DSA_TAG_PROTO_BRCM_PREPEND = DSA_TAG_PROTO_BRCM_PREPEND_VALUE, + DSA_TAG_PROTO_DSA = DSA_TAG_PROTO_DSA_VALUE, + DSA_TAG_PROTO_EDSA = DSA_TAG_PROTO_EDSA_VALUE, + DSA_TAG_PROTO_GSWIP = DSA_TAG_PROTO_GSWIP_VALUE, + DSA_TAG_PROTO_KSZ9477 = DSA_TAG_PROTO_KSZ9477_VALUE, + DSA_TAG_PROTO_KSZ9893 = DSA_TAG_PROTO_KSZ9893_VALUE, + DSA_TAG_PROTO_LAN9303 = DSA_TAG_PROTO_LAN9303_VALUE, + DSA_TAG_PROTO_MTK = DSA_TAG_PROTO_MTK_VALUE, + DSA_TAG_PROTO_QCA = DSA_TAG_PROTO_QCA_VALUE, + DSA_TAG_PROTO_TRAILER = DSA_TAG_PROTO_TRAILER_VALUE, + DSA_TAG_PROTO_8021Q = DSA_TAG_PROTO_8021Q_VALUE, + DSA_TAG_PROTO_SJA1105 = DSA_TAG_PROTO_SJA1105_VALUE, }; struct packet_type; @@ -55,9 +71,52 @@ struct dsa_device_ops { struct packet_type *pt); int (*flow_dissect)(const struct sk_buff *skb, __be16 *proto, int *offset); + /* Used to determine which traffic should match the DSA filter in + * eth_type_trans, and which, if any, should bypass it and be processed + * as regular on the master net device. + */ + bool (*filter)(const struct sk_buff *skb, struct net_device *dev); unsigned int overhead; + const char *name; + enum dsa_tag_protocol proto; +}; + +#define DSA_TAG_DRIVER_ALIAS "dsa_tag-" +#define MODULE_ALIAS_DSA_TAG_DRIVER(__proto) \ + MODULE_ALIAS(DSA_TAG_DRIVER_ALIAS __stringify(__proto##_VALUE)) + +struct dsa_skb_cb { + struct sk_buff *clone; + bool deferred_xmit; +}; + +struct __dsa_skb_cb { + struct dsa_skb_cb cb; + u8 priv[48 - sizeof(struct dsa_skb_cb)]; }; +#define __DSA_SKB_CB(skb) ((struct __dsa_skb_cb *)((skb)->cb)) + +#define DSA_SKB_CB(skb) ((struct dsa_skb_cb *)((skb)->cb)) + +#define DSA_SKB_CB_COPY(nskb, skb) \ + { *__DSA_SKB_CB(nskb) = *__DSA_SKB_CB(skb); } + +#define DSA_SKB_CB_ZERO(skb) \ + { *__DSA_SKB_CB(skb) = (struct __dsa_skb_cb) {0}; } + +#define DSA_SKB_CB_PRIV(skb) \ + ((void *)(skb)->cb + offsetof(struct __dsa_skb_cb, priv)) + +#define DSA_SKB_CB_CLONE(_clone, _skb) \ + { \ + struct sk_buff *clone = _clone; \ + struct sk_buff *skb = _skb; \ + \ + DSA_SKB_CB_COPY(clone, skb); \ + DSA_SKB_CB(skb)->clone = clone; \ + } + struct dsa_switch_tree { struct list_head list; @@ -128,6 +187,7 @@ struct dsa_port { struct dsa_switch_tree *dst; struct sk_buff *(*rcv)(struct sk_buff *skb, struct net_device *dev, struct packet_type *pt); + bool (*filter)(const struct sk_buff *skb, struct net_device *dev); enum { DSA_PORT_TYPE_UNUSED = 0, @@ -140,12 +200,24 @@ struct dsa_port { unsigned int index; const char *name; const struct dsa_port *cpu_dp; + const char *mac; struct device_node *dn; unsigned int ageing_time; + bool vlan_filtering; u8 stp_state; struct net_device *bridge_dev; struct devlink_port devlink_port; struct phylink *pl; + + struct work_struct xmit_work; + struct sk_buff_head xmit_queue; + + /* + * Give the switch driver somewhere to hang its per-port private data + * structures (accessible from the tagger). + */ + void *priv; + /* * Original copy of the master netdev ethtool_ops */ @@ -208,6 +280,16 @@ struct dsa_switch { /* Number of switch port queues */ unsigned int num_tx_queues; + /* Disallow bridge core from requesting different VLAN awareness + * settings on ports if not hardware-supported + */ + bool vlan_filtering_is_global; + + /* In case vlan_filtering_is_global is set, the VLAN awareness state + * should be retrieved from here and not from the per-port settings. + */ + bool vlan_filtering; + unsigned long *bitmap; unsigned long _bitmap; @@ -275,18 +357,19 @@ static inline unsigned int dsa_upstream_port(struct dsa_switch *ds, int port) return dsa_towards_port(ds, cpu_dp->ds->index, cpu_dp->index); } +static inline bool dsa_port_is_vlan_filtering(const struct dsa_port *dp) +{ + const struct dsa_switch *ds = dp->ds; + + if (ds->vlan_filtering_is_global) + return ds->vlan_filtering; + else + return dp->vlan_filtering; +} + typedef int dsa_fdb_dump_cb_t(const unsigned char *addr, u16 vid, bool is_static, void *data); struct dsa_switch_ops { -#if IS_ENABLED(CONFIG_NET_DSA_LEGACY) - /* - * Legacy probing. - */ - const char *(*probe)(struct device *dsa_dev, - struct device *host_dev, int sw_addr, - void **priv); -#endif - enum dsa_tag_protocol (*get_tag_protocol)(struct dsa_switch *ds, int port); @@ -469,6 +552,12 @@ struct dsa_switch_ops { struct sk_buff *clone, unsigned int type); bool (*port_rxtstamp)(struct dsa_switch *ds, int port, struct sk_buff *skb, unsigned int type); + + /* + * Deferred frame Tx + */ + netdev_tx_t (*port_deferred_xmit)(struct dsa_switch *ds, int port, + struct sk_buff *skb); }; struct dsa_switch_driver { @@ -476,20 +565,6 @@ struct dsa_switch_driver { const struct dsa_switch_ops *ops; }; -#if IS_ENABLED(CONFIG_NET_DSA_LEGACY) -/* Legacy driver registration */ -void register_switch_driver(struct dsa_switch_driver *type); -void unregister_switch_driver(struct dsa_switch_driver *type); -struct mii_bus *dsa_host_dev_to_mii_bus(struct device *dev); - -#else -static inline void register_switch_driver(struct dsa_switch_driver *type) { } -static inline void unregister_switch_driver(struct dsa_switch_driver *type) { } -static inline struct mii_bus *dsa_host_dev_to_mii_bus(struct device *dev) -{ - return NULL; -} -#endif struct net_device *dsa_dev_to_net_device(struct device *dev); /* Keep inline for faster access in hot path */ @@ -501,6 +576,15 @@ static inline bool netdev_uses_dsa(struct net_device *dev) return false; } +static inline bool dsa_can_decode(const struct sk_buff *skb, + struct net_device *dev) +{ +#if IS_ENABLED(CONFIG_NET_DSA) + return !dev->dsa_ptr->filter || dev->dsa_ptr->filter(skb, dev); +#endif + return false; +} + struct dsa_switch *dsa_switch_alloc(struct device *dev, size_t n); void dsa_unregister_switch(struct dsa_switch *ds); int dsa_register_switch(struct dsa_switch *ds); @@ -569,9 +653,76 @@ static inline int call_dsa_notifiers(unsigned long val, struct net_device *dev, #define BRCM_TAG_GET_QUEUE(v) ((v) & 0xff) +netdev_tx_t dsa_enqueue_skb(struct sk_buff *skb, struct net_device *dev); int dsa_port_get_phy_strings(struct dsa_port *dp, uint8_t *data); int dsa_port_get_ethtool_phy_stats(struct dsa_port *dp, uint64_t *data); int dsa_port_get_phy_sset_count(struct dsa_port *dp); void dsa_port_phylink_mac_change(struct dsa_switch *ds, int port, bool up); +struct dsa_tag_driver { + const struct dsa_device_ops *ops; + struct list_head list; + struct module *owner; +}; + +void dsa_tag_drivers_register(struct dsa_tag_driver *dsa_tag_driver_array[], + unsigned int count, + struct module *owner); +void dsa_tag_drivers_unregister(struct dsa_tag_driver *dsa_tag_driver_array[], + unsigned int count); + +#define dsa_tag_driver_module_drivers(__dsa_tag_drivers_array, __count) \ +static int __init dsa_tag_driver_module_init(void) \ +{ \ + dsa_tag_drivers_register(__dsa_tag_drivers_array, __count, \ + THIS_MODULE); \ + return 0; \ +} \ +module_init(dsa_tag_driver_module_init); \ + \ +static void __exit dsa_tag_driver_module_exit(void) \ +{ \ + dsa_tag_drivers_unregister(__dsa_tag_drivers_array, __count); \ +} \ +module_exit(dsa_tag_driver_module_exit) + +/** + * module_dsa_tag_drivers() - Helper macro for registering DSA tag + * drivers + * @__ops_array: Array of tag driver strucutres + * + * Helper macro for DSA tag drivers which do not do anything special + * in module init/exit. Each module may only use this macro once, and + * calling it replaces module_init() and module_exit(). + */ +#define module_dsa_tag_drivers(__ops_array) \ +dsa_tag_driver_module_drivers(__ops_array, ARRAY_SIZE(__ops_array)) + +#define DSA_TAG_DRIVER_NAME(__ops) dsa_tag_driver ## _ ## __ops + +/* Create a static structure we can build a linked list of dsa_tag + * drivers + */ +#define DSA_TAG_DRIVER(__ops) \ +static struct dsa_tag_driver DSA_TAG_DRIVER_NAME(__ops) = { \ + .ops = &__ops, \ +} + +/** + * module_dsa_tag_driver() - Helper macro for registering a single DSA tag + * driver + * @__ops: Single tag driver structures + * + * Helper macro for DSA tag drivers which do not do anything special + * in module init/exit. Each module may only use this macro once, and + * calling it replaces module_init() and module_exit(). + */ +#define module_dsa_tag_driver(__ops) \ +DSA_TAG_DRIVER(__ops); \ + \ +static struct dsa_tag_driver *dsa_tag_driver_array[] = { \ + &DSA_TAG_DRIVER_NAME(__ops) \ +}; \ +module_dsa_tag_drivers(dsa_tag_driver_array) #endif + diff --git a/include/net/dst.h b/include/net/dst.h index 6cf0870414c783720026a151ba94307a2bb4ad30..12b31c602cb0dd648cca27ebd4eed4fa6b07e81e 100644 --- a/include/net/dst.h +++ b/include/net/dst.h @@ -19,17 +19,6 @@ #include #include -#define DST_GC_MIN (HZ/10) -#define DST_GC_INC (HZ/2) -#define DST_GC_MAX (120*HZ) - -/* Each dst_entry has reference count and sits in some parent list(s). - * When it is removed from parent list, it is "freed" (dst_free). - * After this it enters dead state (dst->obsolete > 0) and if its refcnt - * is zero, it can be destroyed immediately, otherwise it is added - * to gc list and garbage collector periodically checks the refcnt. - */ - struct sk_buff; struct dst_entry { diff --git a/include/net/fib_notifier.h b/include/net/fib_notifier.h index c91ec732afd6230a647e461f3d3fe0b336b92235..c49d7bfb5c309c75572e40c8d112a71697c9da95 100644 --- a/include/net/fib_notifier.h +++ b/include/net/fib_notifier.h @@ -2,10 +2,11 @@ #define __NET_FIB_NOTIFIER_H #include -#include #include #include +struct module; + struct fib_notifier_info { struct net *net; int family; diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h index 2b26979efb48fc1400e9bab56b9c3a03aa6e54b0..7c5a8d9a8d2a77eb65d11b5358a4d26a11591bdb 100644 --- a/include/net/flow_dissector.h +++ b/include/net/flow_dissector.h @@ -305,4 +305,11 @@ static inline void *skb_flow_dissector_target(struct flow_dissector *flow_dissec return ((char *)target_container) + flow_dissector->offset[key_id]; } +struct bpf_flow_dissector { + struct bpf_flow_keys *flow_keys; + const struct sk_buff *skb; + void *data; + void *data_end; +}; + #endif diff --git a/include/net/flow_offload.h b/include/net/flow_offload.h index d035183c8d0322e689ae4b285da925cf52b9aa84..6200900434e17cc01b1768496a7341a0a4dacf65 100644 --- a/include/net/flow_offload.h +++ b/include/net/flow_offload.h @@ -118,6 +118,8 @@ enum flow_action_id { FLOW_ACTION_MARK, FLOW_ACTION_WAKE, FLOW_ACTION_QUEUE, + FLOW_ACTION_SAMPLE, + FLOW_ACTION_POLICE, }; /* This is mirroring enum pedit_header_type definition for easy mapping between @@ -157,6 +159,16 @@ struct flow_action_entry { u32 index; u8 vf; } queue; + struct { /* FLOW_ACTION_SAMPLE */ + struct psample_group *psample_group; + u32 rate; + u32 trunc_size; + bool truncate; + } sample; + struct { /* FLOW_ACTION_POLICE */ + s64 burst; + u64 rate_bytes_ps; + } police; }; }; @@ -170,6 +182,17 @@ static inline bool flow_action_has_entries(const struct flow_action *action) return action->num_entries; } +/** + * flow_action_has_one_action() - check if exactly one action is present + * @action: tc filter flow offload action + * + * Returns true if exactly one action is present. + */ +static inline bool flow_offload_has_one_action(const struct flow_action *action) +{ + return action->num_entries == 1; +} + #define flow_action_for_each(__i, __act, __actions) \ for (__i = 0, __act = &(__actions)->entries[0]; __i < (__actions)->num_entries; __act = &(__actions)->entries[++__i]) diff --git a/include/net/fq_impl.h b/include/net/fq_impl.h index be7c0fab34782ee5dce7166c55aa8430ecba3af3..2caa86660ab0502cf2fad524f79fd829374b42dc 100644 --- a/include/net/fq_impl.h +++ b/include/net/fq_impl.h @@ -107,21 +107,23 @@ begin: return skb; } +static u32 fq_flow_idx(struct fq *fq, struct sk_buff *skb) +{ + u32 hash = skb_get_hash_perturb(skb, fq->perturbation); + + return reciprocal_scale(hash, fq->flows_cnt); +} + static struct fq_flow *fq_flow_classify(struct fq *fq, - struct fq_tin *tin, + struct fq_tin *tin, u32 idx, struct sk_buff *skb, fq_flow_get_default_t get_default_func) { struct fq_flow *flow; - u32 hash; - u32 idx; lockdep_assert_held(&fq->lock); - hash = skb_get_hash_perturb(skb, fq->perturbation); - idx = reciprocal_scale(hash, fq->flows_cnt); flow = &fq->flows[idx]; - if (flow->tin && flow->tin != tin) { flow = get_default_func(fq, tin, idx, skb); tin->collisions++; @@ -153,7 +155,7 @@ static void fq_recalc_backlog(struct fq *fq, } static void fq_tin_enqueue(struct fq *fq, - struct fq_tin *tin, + struct fq_tin *tin, u32 idx, struct sk_buff *skb, fq_skb_free_t free_func, fq_flow_get_default_t get_default_func) @@ -163,7 +165,7 @@ static void fq_tin_enqueue(struct fq *fq, lockdep_assert_held(&fq->lock); - flow = fq_flow_classify(fq, tin, skb, get_default_func); + flow = fq_flow_classify(fq, tin, idx, skb, get_default_func); flow->tin = tin; flow->backlog += skb->len; diff --git a/include/net/genetlink.h b/include/net/genetlink.h index aa2e5888f18d60e6b56c8ad4eea2dee1fe770c9b..9292f1c588b7afe73c02c885301c8c706a5053a3 100644 --- a/include/net/genetlink.h +++ b/include/net/genetlink.h @@ -26,6 +26,7 @@ struct genl_info; * @name: name of family * @version: protocol version * @maxattr: maximum number of attributes supported + * @policy: netlink policy * @netnsok: set to true if the family can handle network * namespaces and should be presented in all of them * @parallel_ops: operations can be called in parallel and aren't @@ -56,6 +57,7 @@ struct genl_family { unsigned int maxattr; bool netnsok; bool parallel_ops; + const struct nla_policy *policy; int (*pre_doit)(const struct genl_ops *ops, struct sk_buff *skb, struct genl_info *info); @@ -119,19 +121,23 @@ static inline int genl_err_attr(struct genl_info *info, int err, return err; } +enum genl_validate_flags { + GENL_DONT_VALIDATE_STRICT = BIT(0), + GENL_DONT_VALIDATE_DUMP = BIT(1), + GENL_DONT_VALIDATE_DUMP_STRICT = BIT(2), +}; + /** * struct genl_ops - generic netlink operations * @cmd: command identifier * @internal_flags: flags used by the family * @flags: flags - * @policy: attribute validation policy * @doit: standard command callback * @start: start callback for dumps * @dumpit: callback for dumpers * @done: completion callback for dumps */ struct genl_ops { - const struct nla_policy *policy; int (*doit)(struct sk_buff *skb, struct genl_info *info); int (*start)(struct netlink_callback *cb); @@ -141,6 +147,7 @@ struct genl_ops { u8 cmd; u8 internal_flags; u8 flags; + u8 validate; }; int genl_register_family(struct genl_family *family); @@ -164,6 +171,25 @@ static inline struct nlmsghdr *genlmsg_nlhdr(void *user_hdr) NLMSG_HDRLEN); } +/** + * genlmsg_parse_deprecated - parse attributes of a genetlink message + * @nlh: netlink message header + * @family: genetlink message family + * @tb: destination array with maxtype+1 elements + * @maxtype: maximum attribute type to be expected + * @policy: validation policy + * @extack: extended ACK report struct + */ +static inline int genlmsg_parse_deprecated(const struct nlmsghdr *nlh, + const struct genl_family *family, + struct nlattr *tb[], int maxtype, + const struct nla_policy *policy, + struct netlink_ext_ack *extack) +{ + return __nlmsg_parse(nlh, family->hdrsize + GENL_HDRLEN, tb, maxtype, + policy, NL_VALIDATE_LIBERAL, extack); +} + /** * genlmsg_parse - parse attributes of a genetlink message * @nlh: netlink message header @@ -179,8 +205,8 @@ static inline int genlmsg_parse(const struct nlmsghdr *nlh, const struct nla_policy *policy, struct netlink_ext_ack *extack) { - return nlmsg_parse(nlh, family->hdrsize + GENL_HDRLEN, tb, maxtype, - policy, extack); + return __nlmsg_parse(nlh, family->hdrsize + GENL_HDRLEN, tb, maxtype, + policy, NL_VALIDATE_STRICT, extack); } /** diff --git a/include/net/geneve.h b/include/net/geneve.h index fc6a7e0a874a9ff9da0f4eb023b1394723311b09..bced0b1d9fe4f827a32c2038737e9f6264cabd0b 100644 --- a/include/net/geneve.h +++ b/include/net/geneve.h @@ -4,6 +4,8 @@ #include +#define GENEVE_UDP_PORT 6081 + /* Geneve Header: * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * |Ver| Opt Len |O|C| Rsvd. | Protocol Type | diff --git a/include/net/ife.h b/include/net/ife.h index e117617e3c347af5eec412ebd8cbb2271de22b56..7e2538d8585b15b3ca19d55a548cccfcf1d70c2d 100644 --- a/include/net/ife.h +++ b/include/net/ife.h @@ -4,7 +4,6 @@ #include #include -#include #include #if IS_ENABLED(CONFIG_NET_IFE) diff --git a/include/net/ip.h b/include/net/ip.h index 583526aad1d0ac6ac3681cf9cf56d4496a221c3f..2d3cce7c3e8a925f8f89719d2a8137ce4819edcf 100644 --- a/include/net/ip.h +++ b/include/net/ip.h @@ -38,6 +38,10 @@ #define IPV4_MAX_PMTU 65535U /* RFC 2675, Section 5.1 */ #define IPV4_MIN_MTU 68 /* RFC 791 */ +extern unsigned int sysctl_fib_sync_mem; +extern unsigned int sysctl_fib_sync_mem_min; +extern unsigned int sysctl_fib_sync_mem_max; + struct sock; struct inet_skb_parm { diff --git a/include/net/ip6_fib.h b/include/net/ip6_fib.h index 84097010237cd094dd783d9c2dde4b8a75d8d233..40105738e2f6b8e37adac1ff46879ce6c09381b8 100644 --- a/include/net/ip6_fib.h +++ b/include/net/ip6_fib.h @@ -19,6 +19,7 @@ #include #include #include +#include #include #include #include @@ -50,7 +51,8 @@ struct fib6_config { u32 fc_protocol; u16 fc_type; /* only 8 bits are used */ u16 fc_delete_all_nh : 1, - __unused : 15; + fc_ignore_dev_down:1, + __unused : 14; struct in6_addr fc_dst; struct in6_addr fc_src; @@ -124,13 +126,11 @@ struct rt6_exception { #define FIB6_MAX_DEPTH 5 struct fib6_nh { - struct in6_addr nh_gw; - struct net_device *nh_dev; - struct lwtunnel_state *nh_lwtstate; + struct fib_nh_common nh_common; - unsigned int nh_flags; - atomic_t nh_upper_bound; - int nh_weight; +#ifdef CONFIG_IPV6_ROUTER_PREF + unsigned long last_probe; +#endif }; struct fib6_info { @@ -146,7 +146,7 @@ struct fib6_info { struct list_head fib6_siblings; unsigned int fib6_nsiblings; - atomic_t fib6_ref; + refcount_t fib6_ref; unsigned long expires; struct dst_metrics *fib6_metrics; #define fib6_pmtu fib6_metrics->metrics[RTAX_MTU-1] @@ -159,10 +159,6 @@ struct fib6_info { struct rt6_info * __percpu *rt6i_pcpu; struct rt6_exception_bucket __rcu *rt6i_exception_bucket; -#ifdef CONFIG_IPV6_ROUTER_PREF - unsigned long last_probe; -#endif - u32 fib6_metric; u8 fib6_protocol; u8 fib6_type; @@ -194,6 +190,14 @@ struct rt6_info { unsigned short rt6i_nfheader_len; }; +struct fib6_result { + struct fib6_nh *nh; + struct fib6_info *f6i; + u32 fib6_flags; + u8 fib6_type; + struct rt6_info *rt6; +}; + #define for_each_fib6_node_rt_rcu(fn) \ for (rt = rcu_dereference((fn)->leaf); rt; \ rt = rcu_dereference(rt->fib6_next)) @@ -281,17 +285,17 @@ void fib6_info_destroy_rcu(struct rcu_head *head); static inline void fib6_info_hold(struct fib6_info *f6i) { - atomic_inc(&f6i->fib6_ref); + refcount_inc(&f6i->fib6_ref); } static inline bool fib6_info_hold_safe(struct fib6_info *f6i) { - return atomic_inc_not_zero(&f6i->fib6_ref); + return refcount_inc_not_zero(&f6i->fib6_ref); } static inline void fib6_info_release(struct fib6_info *f6i) { - if (f6i && atomic_dec_and_test(&f6i->fib6_ref)) + if (f6i && refcount_dec_and_test(&f6i->fib6_ref)) call_rcu(&f6i->rcu, fib6_info_destroy_rcu); } @@ -388,18 +392,17 @@ struct dst_entry *fib6_rule_lookup(struct net *net, struct flowi6 *fl6, /* called with rcu lock held; can return error pointer * caller needs to select path */ -struct fib6_info *fib6_lookup(struct net *net, int oif, struct flowi6 *fl6, - int flags); +int fib6_lookup(struct net *net, int oif, struct flowi6 *fl6, + struct fib6_result *res, int flags); /* called with rcu lock held; caller needs to select path */ -struct fib6_info *fib6_table_lookup(struct net *net, struct fib6_table *table, - int oif, struct flowi6 *fl6, int strict); - -struct fib6_info *fib6_multipath_select(const struct net *net, - struct fib6_info *match, - struct flowi6 *fl6, int oif, - const struct sk_buff *skb, int strict); +int fib6_table_lookup(struct net *net, struct fib6_table *table, + int oif, struct flowi6 *fl6, struct fib6_result *res, + int strict); +void fib6_select_path(const struct net *net, struct fib6_result *res, + struct flowi6 *fl6, int oif, bool have_oif_match, + const struct sk_buff *skb, int strict); struct fib6_node *fib6_node_lookup(struct fib6_node *root, const struct in6_addr *daddr, const struct in6_addr *saddr); @@ -440,14 +443,13 @@ void rt6_get_prefsrc(const struct rt6_info *rt, struct in6_addr *addr) static inline struct net_device *fib6_info_nh_dev(const struct fib6_info *f6i) { - return f6i->fib6_nh.nh_dev; + return f6i->fib6_nh.fib_nh_dev; } -static inline -struct lwtunnel_state *fib6_info_nh_lwt(const struct fib6_info *f6i) -{ - return f6i->fib6_nh.nh_lwtstate; -} +int fib6_nh_init(struct net *net, struct fib6_nh *fib6_nh, + struct fib6_config *cfg, gfp_t gfp_flags, + struct netlink_ext_ack *extack); +void fib6_nh_release(struct fib6_nh *fib6_nh); void inet6_rt_notify(int event, struct fib6_info *rt, struct nl_info *info, unsigned int flags); diff --git a/include/net/ip6_route.h b/include/net/ip6_route.h index 7ab119936e6979b6fc20a1beb1a3690384e723c0..4790beaa86e01e026e279fd397578810392919cf 100644 --- a/include/net/ip6_route.h +++ b/include/net/ip6_route.h @@ -68,8 +68,9 @@ static inline bool rt6_need_strict(const struct in6_addr *daddr) static inline bool rt6_qualify_for_ecmp(const struct fib6_info *f6i) { - return (f6i->fib6_flags & (RTF_GATEWAY|RTF_ADDRCONF|RTF_DYNAMIC)) == - RTF_GATEWAY; + /* the RTF_ADDRCONF flag filters out RA's */ + return !(f6i->fib6_flags & RTF_ADDRCONF) && + f6i->fib6_nh.fib_nh_gw_family; } void ip6_route_input(struct sk_buff *skb); @@ -181,7 +182,7 @@ int rt6_dump_route(struct fib6_info *f6i, void *p_arg); void rt6_mtu_change(struct net_device *dev, unsigned int mtu); void rt6_remove_prefsrc(struct inet6_ifaddr *ifp); void rt6_clean_tohost(struct net *net, struct in6_addr *gateway); -void rt6_sync_up(struct net_device *dev, unsigned int nh_flags); +void rt6_sync_up(struct net_device *dev, unsigned char nh_flags); void rt6_disable_ip(struct net_device *dev, unsigned long event); void rt6_sync_down_dev(struct net_device *dev, unsigned long event); void rt6_multipath_rebalance(struct fib6_info *f6i); @@ -274,9 +275,11 @@ static inline struct in6_addr *rt6_nexthop(struct rt6_info *rt, static inline bool rt6_duplicate_nexthop(struct fib6_info *a, struct fib6_info *b) { - return a->fib6_nh.nh_dev == b->fib6_nh.nh_dev && - ipv6_addr_equal(&a->fib6_nh.nh_gw, &b->fib6_nh.nh_gw) && - !lwtunnel_cmp_encap(a->fib6_nh.nh_lwtstate, b->fib6_nh.nh_lwtstate); + struct fib6_nh *nha = &a->fib6_nh, *nhb = &b->fib6_nh; + + return nha->fib_nh_dev == nhb->fib_nh_dev && + ipv6_addr_equal(&nha->fib_nh_gw6, &nhb->fib_nh_gw6) && + !lwtunnel_cmp_encap(nha->fib_nh_lws, nhb->fib_nh_lws); } static inline unsigned int ip6_dst_mtu_forward(const struct dst_entry *dst) @@ -300,8 +303,9 @@ static inline unsigned int ip6_dst_mtu_forward(const struct dst_entry *dst) return mtu; } -u32 ip6_mtu_from_fib6(struct fib6_info *f6i, struct in6_addr *daddr, - struct in6_addr *saddr); +u32 ip6_mtu_from_fib6(const struct fib6_result *res, + const struct in6_addr *daddr, + const struct in6_addr *saddr); struct neighbour *ip6_neigh_lookup(const struct in6_addr *gw, struct net_device *dev, struct sk_buff *skb, diff --git a/include/net/ip_fib.h b/include/net/ip_fib.h index 9c8214d2116d3caacf54853d00fd97ee7557db00..d0e28f4ab0994ae55ca0b1778a29fda0aa2789e8 100644 --- a/include/net/ip_fib.h +++ b/include/net/ip_fib.h @@ -32,10 +32,14 @@ struct fib_config { u8 fc_protocol; u8 fc_scope; u8 fc_type; - /* 3 bytes unused */ + u8 fc_gw_family; + /* 2 bytes unused */ u32 fc_table; __be32 fc_dst; - __be32 fc_gw; + union { + __be32 fc_gw4; + struct in6_addr fc_gw6; + }; int fc_oif; u32 fc_flags; u32 fc_priority; @@ -76,27 +80,49 @@ struct fnhe_hash_bucket { #define FNHE_HASH_SIZE (1 << FNHE_HASH_SHIFT) #define FNHE_RECLAIM_DEPTH 5 +struct fib_nh_common { + struct net_device *nhc_dev; + int nhc_oif; + unsigned char nhc_scope; + u8 nhc_family; + u8 nhc_gw_family; + unsigned char nhc_flags; + struct lwtunnel_state *nhc_lwtstate; + + union { + __be32 ipv4; + struct in6_addr ipv6; + } nhc_gw; + + int nhc_weight; + atomic_t nhc_upper_bound; + + /* v4 specific, but allows fib6_nh with v4 routes */ + struct rtable __rcu * __percpu *nhc_pcpu_rth_output; + struct rtable __rcu *nhc_rth_input; + struct fnhe_hash_bucket __rcu *nhc_exceptions; +}; + struct fib_nh { - struct net_device *nh_dev; + struct fib_nh_common nh_common; struct hlist_node nh_hash; struct fib_info *nh_parent; - unsigned int nh_flags; - unsigned char nh_scope; -#ifdef CONFIG_IP_ROUTE_MULTIPATH - int nh_weight; - atomic_t nh_upper_bound; -#endif #ifdef CONFIG_IP_ROUTE_CLASSID __u32 nh_tclassid; #endif - int nh_oif; - __be32 nh_gw; __be32 nh_saddr; int nh_saddr_genid; - struct rtable __rcu * __percpu *nh_pcpu_rth_output; - struct rtable __rcu *nh_rth_input; - struct fnhe_hash_bucket __rcu *nh_exceptions; - struct lwtunnel_state *nh_lwtstate; +#define fib_nh_family nh_common.nhc_family +#define fib_nh_dev nh_common.nhc_dev +#define fib_nh_oif nh_common.nhc_oif +#define fib_nh_flags nh_common.nhc_flags +#define fib_nh_lws nh_common.nhc_lwtstate +#define fib_nh_scope nh_common.nhc_scope +#define fib_nh_gw_family nh_common.nhc_gw_family +#define fib_nh_gw4 nh_common.nhc_gw.ipv4 +#define fib_nh_gw6 nh_common.nhc_gw.ipv6 +#define fib_nh_weight nh_common.nhc_weight +#define fib_nh_upper_bound nh_common.nhc_upper_bound }; /* @@ -123,9 +149,10 @@ struct fib_info { #define fib_rtt fib_metrics->metrics[RTAX_RTT-1] #define fib_advmss fib_metrics->metrics[RTAX_ADVMSS-1] int fib_nhs; + bool fib_nh_is_v6; struct rcu_head rcu; struct fib_nh fib_nh[0]; -#define fib_dev fib_nh[0].nh_dev +#define fib_dev fib_nh[0].fib_nh_dev }; @@ -135,15 +162,16 @@ struct fib_rule; struct fib_table; struct fib_result { - __be32 prefix; - unsigned char prefixlen; - unsigned char nh_sel; - unsigned char type; - unsigned char scope; - u32 tclassid; - struct fib_info *fi; - struct fib_table *table; - struct hlist_head *fa_head; + __be32 prefix; + unsigned char prefixlen; + unsigned char nh_sel; + unsigned char type; + unsigned char scope; + u32 tclassid; + struct fib_nh_common *nhc; + struct fib_info *fi; + struct fib_table *table; + struct hlist_head *fa_head; }; struct fib_result_nl { @@ -161,11 +189,10 @@ struct fib_result_nl { int err; }; -#ifdef CONFIG_IP_ROUTE_MULTIPATH -#define FIB_RES_NH(res) ((res).fi->fib_nh[(res).nh_sel]) -#else /* CONFIG_IP_ROUTE_MULTIPATH */ -#define FIB_RES_NH(res) ((res).fi->fib_nh[0]) -#endif /* CONFIG_IP_ROUTE_MULTIPATH */ +static inline struct fib_nh_common *fib_info_nhc(struct fib_info *fi, int nhsel) +{ + return &fi->fib_nh[nhsel].nh_common; +} #ifdef CONFIG_IP_MULTIPLE_TABLES #define FIB_TABLE_HASHSZ 256 @@ -174,18 +201,11 @@ struct fib_result_nl { #endif __be32 fib_info_update_nh_saddr(struct net *net, struct fib_nh *nh); +__be32 fib_result_prefsrc(struct net *net, struct fib_result *res); -#define FIB_RES_SADDR(net, res) \ - ((FIB_RES_NH(res).nh_saddr_genid == \ - atomic_read(&(net)->ipv4.dev_addr_genid)) ? \ - FIB_RES_NH(res).nh_saddr : \ - fib_info_update_nh_saddr((net), &FIB_RES_NH(res))) -#define FIB_RES_GW(res) (FIB_RES_NH(res).nh_gw) -#define FIB_RES_DEV(res) (FIB_RES_NH(res).nh_dev) -#define FIB_RES_OIF(res) (FIB_RES_NH(res).nh_oif) - -#define FIB_RES_PREFSRC(net, res) ((res).fi->fib_prefsrc ? : \ - FIB_RES_SADDR(net, res)) +#define FIB_RES_NHC(res) ((res).nhc) +#define FIB_RES_DEV(res) (FIB_RES_NHC(res)->nhc_dev) +#define FIB_RES_OIF(res) (FIB_RES_NHC(res)->nhc_oif) struct fib_entry_notifier_info { struct fib_notifier_info info; /* must be first */ @@ -383,6 +403,8 @@ static inline bool fib4_rules_early_flow_dissect(struct net *net, /* Exported by fib_frontend.c */ extern const struct nla_policy rtm_ipv4_policy[]; void ip_fib_init(void); +int fib_gw_from_via(struct fib_config *cfg, struct nlattr *nla, + struct netlink_ext_ack *extack); __be32 fib_compute_spec_dst(struct sk_buff *skb); bool fib_info_nh_uses_dev(struct fib_info *fi, const struct net_device *dev); int fib_validate_source(struct sk_buff *skb, __be32 src, __be32 dst, @@ -405,7 +427,7 @@ int fib_unmerge(struct net *net); int ip_fib_check_default(__be32 gw, struct net_device *dev); int fib_sync_down_dev(struct net_device *dev, unsigned long event, bool force); int fib_sync_down_addr(struct net_device *dev, __be32 local); -int fib_sync_up(struct net_device *dev, unsigned int nh_flags); +int fib_sync_up(struct net_device *dev, unsigned char nh_flags); void fib_sync_mtu(struct net_device *dev, u32 orig_mtu); #ifdef CONFIG_IP_ROUTE_MULTIPATH @@ -416,6 +438,15 @@ void fib_select_multipath(struct fib_result *res, int hash); void fib_select_path(struct net *net, struct fib_result *res, struct flowi4 *fl4, const struct sk_buff *skb); +int fib_nh_init(struct net *net, struct fib_nh *fib_nh, + struct fib_config *cfg, int nh_weight, + struct netlink_ext_ack *extack); +void fib_nh_release(struct net *net, struct fib_nh *fib_nh); +int fib_nh_common_init(struct fib_nh_common *nhc, struct nlattr *fc_encap, + u16 fc_encap_type, void *cfg, gfp_t gfp_flags, + struct netlink_ext_ack *extack); +void fib_nh_common_release(struct fib_nh_common *nhc); + /* Exported by fib_trie.c */ void fib_trie_init(void); struct fib_table *fib_trie_table(u32 id, struct fib_table *alias); @@ -423,10 +454,12 @@ struct fib_table *fib_trie_table(u32 id, struct fib_table *alias); static inline void fib_combine_itag(u32 *itag, const struct fib_result *res) { #ifdef CONFIG_IP_ROUTE_CLASSID + struct fib_nh_common *nhc = res->nhc; + struct fib_nh *nh = container_of(nhc, struct fib_nh, nh_common); #ifdef CONFIG_IP_MULTIPLE_TABLES u32 rtag; #endif - *itag = FIB_RES_NH(*res).nh_tclassid<<16; + *itag = nh->nh_tclassid << 16; #ifdef CONFIG_IP_MULTIPLE_TABLES rtag = res->tclassid; if (*itag == 0) @@ -467,4 +500,9 @@ u32 ip_mtu_from_fib_result(struct fib_result *res, __be32 daddr); int ip_valid_fib_dump_req(struct net *net, const struct nlmsghdr *nlh, struct fib_dump_filter *filter, struct netlink_callback *cb); + +int fib_nexthop_info(struct sk_buff *skb, const struct fib_nh_common *nh, + unsigned char *flags, bool skip_oif); +int fib_add_nexthop(struct sk_buff *skb, const struct fib_nh_common *nh, + int nh_weight); #endif /* _NET_FIB_H */ diff --git a/include/net/ip_vs.h b/include/net/ip_vs.h index 047f9a5ccaad4a4c28e88838fa8a86e976b46a19..2ac40135b5765ffc373b6322dc79a22d96db275b 100644 --- a/include/net/ip_vs.h +++ b/include/net/ip_vs.h @@ -600,6 +600,9 @@ struct ip_vs_dest_user_kern { /* Address family of addr */ u16 af; + + u16 tun_type; /* tunnel type */ + __be16 tun_port; /* tunnel port */ }; @@ -660,6 +663,8 @@ struct ip_vs_dest { atomic_t conn_flags; /* flags to copy to conn */ atomic_t weight; /* server weight */ atomic_t last_weight; /* server latest weight */ + __u16 tun_type; /* tunnel type */ + __be16 tun_port; /* tunnel port */ refcount_t refcnt; /* reference counter */ struct ip_vs_stats stats; /* statistics */ diff --git a/include/net/ipv6_frag.h b/include/net/ipv6_frag.h index 28aa9b30aeceac9a86ee6754e4b5809be115e947..1f77fb4dc79df6bc4e41d6d2f4d49ace32082ca4 100644 --- a/include/net/ipv6_frag.h +++ b/include/net/ipv6_frag.h @@ -94,7 +94,6 @@ ip6frag_expire_frag_queue(struct net *net, struct frag_queue *fq) goto out; head->dev = dev; - skb_get(head); spin_unlock(&fq->q.lock); icmpv6_send(head, ICMPV6_TIME_EXCEED, ICMPV6_EXC_FRAGTIME, 0); diff --git a/include/net/ipv6_stubs.h b/include/net/ipv6_stubs.h new file mode 100644 index 0000000000000000000000000000000000000000..6c0c4fde16f8902ecb9baf23cbc8db182dc2ab8c --- /dev/null +++ b/include/net/ipv6_stubs.h @@ -0,0 +1,68 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _IPV6_STUBS_H +#define _IPV6_STUBS_H + +#include +#include +#include +#include +#include +#include +#include + +/* structs from net/ip6_fib.h */ +struct fib6_info; +struct fib6_nh; +struct fib6_config; +struct fib6_result; + +/* This is ugly, ideally these symbols should be built + * into the core kernel. + */ +struct ipv6_stub { + int (*ipv6_sock_mc_join)(struct sock *sk, int ifindex, + const struct in6_addr *addr); + int (*ipv6_sock_mc_drop)(struct sock *sk, int ifindex, + const struct in6_addr *addr); + int (*ipv6_dst_lookup)(struct net *net, struct sock *sk, + struct dst_entry **dst, struct flowi6 *fl6); + int (*ipv6_route_input)(struct sk_buff *skb); + + struct fib6_table *(*fib6_get_table)(struct net *net, u32 id); + int (*fib6_lookup)(struct net *net, int oif, struct flowi6 *fl6, + struct fib6_result *res, int flags); + int (*fib6_table_lookup)(struct net *net, struct fib6_table *table, + int oif, struct flowi6 *fl6, + struct fib6_result *res, int flags); + void (*fib6_select_path)(const struct net *net, struct fib6_result *res, + struct flowi6 *fl6, int oif, bool oif_match, + const struct sk_buff *skb, int strict); + u32 (*ip6_mtu_from_fib6)(const struct fib6_result *res, + const struct in6_addr *daddr, + const struct in6_addr *saddr); + + int (*fib6_nh_init)(struct net *net, struct fib6_nh *fib6_nh, + struct fib6_config *cfg, gfp_t gfp_flags, + struct netlink_ext_ack *extack); + void (*fib6_nh_release)(struct fib6_nh *fib6_nh); + void (*udpv6_encap_enable)(void); + void (*ndisc_send_na)(struct net_device *dev, const struct in6_addr *daddr, + const struct in6_addr *solicited_addr, + bool router, bool solicited, bool override, bool inc_opt); + struct neigh_table *nd_tbl; +}; +extern const struct ipv6_stub *ipv6_stub __read_mostly; + +/* A stub used by bpf helpers. Similarly ugly as ipv6_stub */ +struct ipv6_bpf_stub { + int (*inet6_bind)(struct sock *sk, struct sockaddr *uaddr, int addr_len, + bool force_bind_address_no_port, bool with_lock); + struct sock *(*udp6_lib_lookup)(struct net *net, + const struct in6_addr *saddr, __be16 sport, + const struct in6_addr *daddr, __be16 dport, + int dif, int sdif, struct udp_table *tbl, + struct sk_buff *skb); +}; +extern const struct ipv6_bpf_stub *ipv6_bpf_stub __read_mostly; + +#endif diff --git a/include/net/lwtunnel.h b/include/net/lwtunnel.h index 671113bcb2cc20e2b4ff56f4160a9241cebce477..5d6c5b1fc6955fddb69357fc7c4c4a81f9cd694d 100644 --- a/include/net/lwtunnel.h +++ b/include/net/lwtunnel.h @@ -118,8 +118,8 @@ int lwtunnel_build_state(u16 encap_type, unsigned int family, const void *cfg, struct lwtunnel_state **lws, struct netlink_ext_ack *extack); -int lwtunnel_fill_encap(struct sk_buff *skb, - struct lwtunnel_state *lwtstate); +int lwtunnel_fill_encap(struct sk_buff *skb, struct lwtunnel_state *lwtstate, + int encap_attr, int encap_type_attr); int lwtunnel_get_encap_size(struct lwtunnel_state *lwtstate); struct lwtunnel_state *lwtunnel_state_alloc(int hdr_len); int lwtunnel_cmp_encap(struct lwtunnel_state *a, struct lwtunnel_state *b); @@ -219,7 +219,8 @@ static inline int lwtunnel_build_state(u16 encap_type, } static inline int lwtunnel_fill_encap(struct sk_buff *skb, - struct lwtunnel_state *lwtstate) + struct lwtunnel_state *lwtstate, + int encap_attr, int encap_type_attr) { return 0; } diff --git a/include/net/mac80211.h b/include/net/mac80211.h index 112dc18c658f15f79525cae64afa0ea38e2a1159..72080d9d617e2708233659a455bd679f290886b1 100644 --- a/include/net/mac80211.h +++ b/include/net/mac80211.h @@ -807,6 +807,7 @@ enum mac80211_tx_info_flags { * @IEEE80211_TX_CTRL_RATE_INJECT: This frame is injected with rate information * @IEEE80211_TX_CTRL_AMSDU: This frame is an A-MSDU frame * @IEEE80211_TX_CTRL_FAST_XMIT: This frame is going through the fast_xmit path + * @IEEE80211_TX_CTRL_SKIP_MPATH_LOOKUP: This frame skips mesh path lookup * * These flags are used in tx_info->control.flags. */ @@ -816,6 +817,7 @@ enum mac80211_tx_control_flags { IEEE80211_TX_CTRL_RATE_INJECT = BIT(2), IEEE80211_TX_CTRL_AMSDU = BIT(3), IEEE80211_TX_CTRL_FAST_XMIT = BIT(4), + IEEE80211_TX_CTRL_SKIP_MPATH_LOOKUP = BIT(5), }; /* @@ -1697,6 +1699,7 @@ struct wireless_dev *ieee80211_vif_to_wdev(struct ieee80211_vif *vif); * @IEEE80211_KEY_FLAG_PUT_MIC_SPACE: This flag should be set by the driver for * a TKIP key if it only requires MIC space. Do not set together with * @IEEE80211_KEY_FLAG_GENERATE_MMIC on the same key. + * @IEEE80211_KEY_FLAG_NO_AUTO_TX: Key needs explicit Tx activation. */ enum ieee80211_key_flags { IEEE80211_KEY_FLAG_GENERATE_IV_MGMT = BIT(0), @@ -1708,6 +1711,7 @@ enum ieee80211_key_flags { IEEE80211_KEY_FLAG_RX_MGMT = BIT(6), IEEE80211_KEY_FLAG_RESERVE_TAILROOM = BIT(7), IEEE80211_KEY_FLAG_PUT_MIC_SPACE = BIT(8), + IEEE80211_KEY_FLAG_NO_AUTO_TX = BIT(9), }; /** @@ -1887,6 +1891,24 @@ struct ieee80211_sta_rates { } rate[IEEE80211_TX_RATE_TABLE_SIZE]; }; +/** + * struct ieee80211_sta_txpwr - station txpower configuration + * + * Used to configure txpower for station. + * + * @power: indicates the tx power, in dBm, to be used when sending data frames + * to the STA. + * @type: In particular if TPC %type is NL80211_TX_POWER_LIMITED then tx power + * will be less than or equal to specified from userspace, whereas if TPC + * %type is NL80211_TX_POWER_AUTOMATIC then it indicates default tx power. + * NL80211_TX_POWER_FIXED is not a valid configuration option for + * per peer TPC. + */ +struct ieee80211_sta_txpwr { + s16 power; + enum nl80211_tx_power_setting type; +}; + /** * struct ieee80211_sta - station table entry * @@ -1973,6 +1995,7 @@ struct ieee80211_sta { bool support_p2p_ps; u16 max_rc_amsdu_len; u16 max_tid_amsdu_len[IEEE80211_NUM_TIDS]; + struct ieee80211_sta_txpwr txpwr; struct ieee80211_txq *txq[IEEE80211_NUM_TIDS + 1]; @@ -2243,6 +2266,9 @@ struct ieee80211_txq { * @IEEE80211_HW_SUPPORTS_ONLY_HE_MULTI_BSSID: Hardware supports multi BSSID * only for HE APs. Applies if @IEEE80211_HW_SUPPORTS_MULTI_BSSID is set. * + * @IEEE80211_HW_EXT_KEY_ID_NATIVE: Driver and hardware are supporting Extended + * Key ID and can handle two unicast keys per station for Rx and Tx. + * * @NUM_IEEE80211_HW_FLAGS: number of hardware flags, used for sizing arrays */ enum ieee80211_hw_flags { @@ -2294,6 +2320,7 @@ enum ieee80211_hw_flags { IEEE80211_HW_TX_STATUS_NO_AMPDU_LEN, IEEE80211_HW_SUPPORTS_MULTI_BSSID, IEEE80211_HW_SUPPORTS_ONLY_HE_MULTI_BSSID, + IEEE80211_HW_EXT_KEY_ID_NATIVE, /* keep last, obviously */ NUM_IEEE80211_HW_FLAGS @@ -3794,6 +3821,9 @@ struct ieee80211_ops { #endif void (*sta_notify)(struct ieee80211_hw *hw, struct ieee80211_vif *vif, enum sta_notify_cmd, struct ieee80211_sta *sta); + int (*sta_set_txpwr)(struct ieee80211_hw *hw, + struct ieee80211_vif *vif, + struct ieee80211_sta *sta); int (*sta_state)(struct ieee80211_hw *hw, struct ieee80211_vif *vif, struct ieee80211_sta *sta, enum ieee80211_sta_state old_state, diff --git a/include/net/ndisc.h b/include/net/ndisc.h index ddfbb591e2c568cbaf186417b8cb9ab33a4075e2..3661500530432745eb6b13cb20e8d5194a73816e 100644 --- a/include/net/ndisc.h +++ b/include/net/ndisc.h @@ -2,6 +2,8 @@ #ifndef _NDISC_H #define _NDISC_H +#include + /* * ICMP codes for neighbour discovery messages */ @@ -379,6 +381,14 @@ static inline struct neighbour *__ipv6_neigh_lookup_noref(struct net_device *dev return ___neigh_lookup_noref(&nd_tbl, neigh_key_eq128, ndisc_hashfn, pkey, dev); } +static inline +struct neighbour *__ipv6_neigh_lookup_noref_stub(struct net_device *dev, + const void *pkey) +{ + return ___neigh_lookup_noref(ipv6_stub->nd_tbl, neigh_key_eq128, + ndisc_hashfn, pkey, dev); +} + static inline struct neighbour *__ipv6_neigh_lookup(struct net_device *dev, const void *pkey) { struct neighbour *n; @@ -409,6 +419,36 @@ static inline void __ipv6_confirm_neigh(struct net_device *dev, rcu_read_unlock_bh(); } +static inline void __ipv6_confirm_neigh_stub(struct net_device *dev, + const void *pkey) +{ + struct neighbour *n; + + rcu_read_lock_bh(); + n = __ipv6_neigh_lookup_noref_stub(dev, pkey); + if (n) { + unsigned long now = jiffies; + + /* avoid dirtying neighbour */ + if (n->confirmed != now) + n->confirmed = now; + } + rcu_read_unlock_bh(); +} + +/* uses ipv6_stub and is meant for use outside of IPv6 core */ +static inline struct neighbour *ip_neigh_gw6(struct net_device *dev, + const void *addr) +{ + struct neighbour *neigh; + + neigh = __ipv6_neigh_lookup_noref_stub(dev, addr); + if (unlikely(!neigh)) + neigh = __neigh_create(ipv6_stub->nd_tbl, addr, dev, false); + + return neigh; +} + int ndisc_init(void); int ndisc_late_init(void); diff --git a/include/net/neighbour.h b/include/net/neighbour.h index 7c1ab9edba03873ffc9ea029e7925e12051f6b02..50a67bd6a43413bf69e2ad7b7c27e8460b6fb152 100644 --- a/include/net/neighbour.h +++ b/include/net/neighbour.h @@ -205,6 +205,8 @@ struct neigh_table { int (*pconstructor)(struct pneigh_entry *); void (*pdestructor)(struct pneigh_entry *); void (*proxy_redo)(struct sk_buff *skb); + bool (*allow_add)(const struct net_device *dev, + struct netlink_ext_ack *extack); char *id; struct neigh_parms parms; struct list_head parms_list; @@ -498,11 +500,12 @@ static inline int neigh_hh_output(const struct hh_cache *hh, struct sk_buff *skb return dev_queue_xmit(skb); } -static inline int neigh_output(struct neighbour *n, struct sk_buff *skb) +static inline int neigh_output(struct neighbour *n, struct sk_buff *skb, + bool skip_cache) { const struct hh_cache *hh = &n->hh; - if ((n->nud_state & NUD_CONNECTED) && hh->hh_len) + if ((n->nud_state & NUD_CONNECTED) && hh->hh_len && !skip_cache) return neigh_hh_output(hh, skb); else return n->output(n, skb); diff --git a/include/net/netfilter/ipv4/nf_nat_masquerade.h b/include/net/netfilter/ipv4/nf_nat_masquerade.h deleted file mode 100644 index 13d55206bb9fccee2d669aec6a8c09ff92beaf8d..0000000000000000000000000000000000000000 --- a/include/net/netfilter/ipv4/nf_nat_masquerade.h +++ /dev/null @@ -1,15 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _NF_NAT_MASQUERADE_IPV4_H_ -#define _NF_NAT_MASQUERADE_IPV4_H_ - -#include - -unsigned int -nf_nat_masquerade_ipv4(struct sk_buff *skb, unsigned int hooknum, - const struct nf_nat_range2 *range, - const struct net_device *out); - -int nf_nat_masquerade_ipv4_register_notifier(void); -void nf_nat_masquerade_ipv4_unregister_notifier(void); - -#endif /*_NF_NAT_MASQUERADE_IPV4_H_ */ diff --git a/include/net/netfilter/ipv6/nf_nat_masquerade.h b/include/net/netfilter/ipv6/nf_nat_masquerade.h deleted file mode 100644 index 2917bf95c4370db57fcc64a6bb76fd64ad1ef902..0000000000000000000000000000000000000000 --- a/include/net/netfilter/ipv6/nf_nat_masquerade.h +++ /dev/null @@ -1,11 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _NF_NAT_MASQUERADE_IPV6_H_ -#define _NF_NAT_MASQUERADE_IPV6_H_ - -unsigned int -nf_nat_masquerade_ipv6(struct sk_buff *skb, const struct nf_nat_range2 *range, - const struct net_device *out); -int nf_nat_masquerade_ipv6_register_notifier(void); -void nf_nat_masquerade_ipv6_unregister_notifier(void); - -#endif /* _NF_NAT_MASQUERADE_IPV6_H_ */ diff --git a/include/net/netfilter/nf_conntrack_expect.h b/include/net/netfilter/nf_conntrack_expect.h index 006e430d1cdfb7f211e1f655337af7f01367f3e8..93ce6b0daaba9accf8dd270653b51f4a3d815c02 100644 --- a/include/net/netfilter/nf_conntrack_expect.h +++ b/include/net/netfilter/nf_conntrack_expect.h @@ -48,7 +48,7 @@ struct nf_conntrack_expect { /* Expectation class */ unsigned int class; -#ifdef CONFIG_NF_NAT_NEEDED +#if IS_ENABLED(CONFIG_NF_NAT) union nf_inet_addr saved_addr; /* This is the original per-proto part, used to map the * expected connection the way the recipient expects. */ diff --git a/include/net/netfilter/nf_conntrack_helper.h b/include/net/netfilter/nf_conntrack_helper.h index ec52a8dc32fd5029d8c6dac95c3a32eacb2e98d8..44b5a00a9c64c8150c8eea2616a0a5b0e69223f3 100644 --- a/include/net/netfilter/nf_conntrack_helper.h +++ b/include/net/netfilter/nf_conntrack_helper.h @@ -15,6 +15,11 @@ #include #include +#define NF_NAT_HELPER_PREFIX "ip_nat_" +#define NF_NAT_HELPER_NAME(name) NF_NAT_HELPER_PREFIX name +#define MODULE_ALIAS_NF_NAT_HELPER(name) \ + MODULE_ALIAS(NF_NAT_HELPER_NAME(name)) + struct module; enum nf_ct_helper_flags { @@ -54,6 +59,8 @@ struct nf_conntrack_helper { unsigned int queue_num; /* length of userspace private data stored in nf_conn_help->data */ u16 data_len; + /* name of NAT helper module */ + char nat_mod_name[NF_CT_HELPER_NAME_LEN]; }; /* Must be kept in sync with the classes defined by helpers */ @@ -153,4 +160,21 @@ nf_ct_helper_expectfn_find_by_symbol(const void *symbol); extern struct hlist_head *nf_ct_helper_hash; extern unsigned int nf_ct_helper_hsize; +struct nf_conntrack_nat_helper { + struct list_head list; + char mod_name[NF_CT_HELPER_NAME_LEN]; /* module name */ + struct module *module; /* pointer to self */ +}; + +#define NF_CT_NAT_HELPER_INIT(name) \ + { \ + .mod_name = NF_NAT_HELPER_NAME(name), \ + .module = THIS_MODULE \ + } + +void nf_nat_helper_register(struct nf_conntrack_nat_helper *nat); +void nf_nat_helper_unregister(struct nf_conntrack_nat_helper *nat); +int nf_nat_helper_try_module_get(const char *name, u16 l3num, + u8 protonum); +void nf_nat_helper_put(struct nf_conntrack_helper *helper); #endif /*_NF_CONNTRACK_HELPER_H*/ diff --git a/include/net/netfilter/nf_conntrack_timeout.h b/include/net/netfilter/nf_conntrack_timeout.h index 3394d75e1c80380c9a27fcb31fa67e831afb19c7..00a8fbb2d73545c3f05dc748aa42b9e94f51bf3a 100644 --- a/include/net/netfilter/nf_conntrack_timeout.h +++ b/include/net/netfilter/nf_conntrack_timeout.h @@ -88,6 +88,9 @@ static inline unsigned int *nf_ct_timeout_lookup(const struct nf_conn *ct) int nf_conntrack_timeout_init(void); void nf_conntrack_timeout_fini(void); void nf_ct_untimeout(struct net *net, struct nf_ct_timeout *timeout); +int nf_ct_set_timeout(struct net *net, struct nf_conn *ct, u8 l3num, u8 l4num, + const char *timeout_name); +void nf_ct_destroy_timeout(struct nf_conn *ct); #else static inline int nf_conntrack_timeout_init(void) { @@ -98,6 +101,18 @@ static inline void nf_conntrack_timeout_fini(void) { return; } + +static inline int nf_ct_set_timeout(struct net *net, struct nf_conn *ct, + u8 l3num, u8 l4num, + const char *timeout_name) +{ + return -EOPNOTSUPP; +} + +static inline void nf_ct_destroy_timeout(struct nf_conn *ct) +{ + return; +} #endif /* CONFIG_NF_CONNTRACK_TIMEOUT */ #ifdef CONFIG_NF_CONNTRACK_TIMEOUT diff --git a/include/net/netfilter/nf_nat.h b/include/net/netfilter/nf_nat.h index cf332c4e0b325bc06e7926fb5f886b0a3dd07e42..423cda2c65425e43659f2558f67ef4c0fe94dea2 100644 --- a/include/net/netfilter/nf_nat.h +++ b/include/net/netfilter/nf_nat.h @@ -69,9 +69,9 @@ static inline bool nf_nat_oif_changed(unsigned int hooknum, #endif } -int nf_nat_register_fn(struct net *net, const struct nf_hook_ops *ops, +int nf_nat_register_fn(struct net *net, u8 pf, const struct nf_hook_ops *ops, const struct nf_hook_ops *nat_ops, unsigned int ops_count); -void nf_nat_unregister_fn(struct net *net, const struct nf_hook_ops *ops, +void nf_nat_unregister_fn(struct net *net, u8 pf, const struct nf_hook_ops *ops, unsigned int ops_count); unsigned int nf_nat_packet(struct nf_conn *ct, enum ip_conntrack_info ctinfo, @@ -98,6 +98,9 @@ void nf_nat_ipv4_unregister_fn(struct net *net, const struct nf_hook_ops *ops); int nf_nat_ipv6_register_fn(struct net *net, const struct nf_hook_ops *ops); void nf_nat_ipv6_unregister_fn(struct net *net, const struct nf_hook_ops *ops); +int nf_nat_inet_register_fn(struct net *net, const struct nf_hook_ops *ops); +void nf_nat_inet_unregister_fn(struct net *net, const struct nf_hook_ops *ops); + unsigned int nf_nat_inet_fn(void *priv, struct sk_buff *skb, const struct nf_hook_state *state); diff --git a/include/net/netfilter/nf_nat_masquerade.h b/include/net/netfilter/nf_nat_masquerade.h new file mode 100644 index 0000000000000000000000000000000000000000..54a14d643c3490ee0ba5c96d2372dfd66548ae99 --- /dev/null +++ b/include/net/netfilter/nf_nat_masquerade.h @@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _NF_NAT_MASQUERADE_H_ +#define _NF_NAT_MASQUERADE_H_ + +#include + +unsigned int +nf_nat_masquerade_ipv4(struct sk_buff *skb, unsigned int hooknum, + const struct nf_nat_range2 *range, + const struct net_device *out); + +int nf_nat_masquerade_inet_register_notifiers(void); +void nf_nat_masquerade_inet_unregister_notifiers(void); + +unsigned int +nf_nat_masquerade_ipv6(struct sk_buff *skb, const struct nf_nat_range2 *range, + const struct net_device *out); + +#endif /*_NF_NAT_MASQUERADE_H_ */ diff --git a/include/net/netfilter/nf_queue.h b/include/net/netfilter/nf_queue.h index a50a69f5334c8647d129ac1edce909edfc442691..7239105d9d2e27bac3d0714710cee815d082d051 100644 --- a/include/net/netfilter/nf_queue.h +++ b/include/net/netfilter/nf_queue.h @@ -119,4 +119,7 @@ nfqueue_hash(const struct sk_buff *skb, u16 queue, u16 queues_total, u8 family, return queue; } +int nf_queue(struct sk_buff *skb, struct nf_hook_state *state, + const struct nf_hook_entries *entries, unsigned int index, + unsigned int verdict); #endif /* _NF_QUEUE_H */ diff --git a/include/net/netfilter/nf_tables.h b/include/net/netfilter/nf_tables.h index 3e9ab643eedfe3056d74e4b247dfe08a0c153c06..5b8624ae4a27f4b2617b32ab971bfc700835f5e5 100644 --- a/include/net/netfilter/nf_tables.h +++ b/include/net/netfilter/nf_tables.h @@ -2,7 +2,6 @@ #ifndef _NET_NF_TABLES_H #define _NET_NF_TABLES_H -#include #include #include #include @@ -13,6 +12,8 @@ #include #include +struct module; + #define NFT_JUMP_STACK_SIZE 16 struct nft_pktinfo { @@ -475,8 +476,6 @@ void nf_tables_deactivate_set(const struct nft_ctx *ctx, struct nft_set *set, enum nft_trans_phase phase); int nf_tables_bind_set(const struct nft_ctx *ctx, struct nft_set *set, struct nft_set_binding *binding); -void nf_tables_unbind_set(const struct nft_ctx *ctx, struct nft_set *set, - struct nft_set_binding *binding, bool commit); void nf_tables_destroy_set(const struct nft_ctx *ctx, struct nft_set *set); /** @@ -808,23 +807,6 @@ void nft_expr_destroy(const struct nft_ctx *ctx, struct nft_expr *expr); int nft_expr_dump(struct sk_buff *skb, unsigned int attr, const struct nft_expr *expr); -static inline int nft_expr_clone(struct nft_expr *dst, struct nft_expr *src) -{ - int err; - - if (src->ops->clone) { - dst->ops = src->ops; - err = src->ops->clone(dst, src); - if (err < 0) - return err; - } else { - memcpy(dst, src, src->ops->size); - } - - __module_get(src->ops->type->owner); - return 0; -} - /** * struct nft_rule - nf_tables rule * @@ -1411,4 +1393,6 @@ struct nft_trans_flowtable { int __init nft_chain_filter_init(void); void nft_chain_filter_fini(void); +void __init nft_chain_route_init(void); +void nft_chain_route_fini(void); #endif /* _NET_NF_TABLES_H */ diff --git a/include/net/netlink.h b/include/net/netlink.h index 23f27b0b3cef0dc18ee91f25a7314b4f9a79600b..395b4406f4b087ee26d35091c0dcd68ad1d6f06b 100644 --- a/include/net/netlink.h +++ b/include/net/netlink.h @@ -183,6 +183,7 @@ enum { NLA_REJECT, NLA_EXACT_LEN, NLA_EXACT_LEN_WARN, + NLA_MIN_LEN, __NLA_TYPE_MAX, }; @@ -212,6 +213,7 @@ enum nla_policy_validation { * NLA_NUL_STRING Maximum length of string (excluding NUL) * NLA_FLAG Unused * NLA_BINARY Maximum length of attribute payload + * NLA_MIN_LEN Minimum length of attribute payload * NLA_NESTED, * NLA_NESTED_ARRAY Length verification is done by checking len of * nested header (or empty); len field is used if @@ -230,6 +232,7 @@ enum nla_policy_validation { * it is rejected. * NLA_EXACT_LEN_WARN Attribute should have exactly this length, a warning * is logged if it is longer, shorter is rejected. + * NLA_MIN_LEN Minimum length of attribute payload * All other Minimum length of attribute payload * * Meaning of `validation_data' field: @@ -281,7 +284,7 @@ enum nla_policy_validation { * static const struct nla_policy my_policy[ATTR_MAX+1] = { * [ATTR_FOO] = { .type = NLA_U16 }, * [ATTR_BAR] = { .type = NLA_STRING, .len = BARSIZ }, - * [ATTR_BAZ] = { .len = sizeof(struct mystruct) }, + * [ATTR_BAZ] = { .type = NLA_EXACT_LEN, .len = sizeof(struct mystruct) }, * [ATTR_GOO] = { .type = NLA_BITFIELD32, .validation_data = &myvalidflags }, * }; */ @@ -296,12 +299,31 @@ struct nla_policy { }; int (*validate)(const struct nlattr *attr, struct netlink_ext_ack *extack); + /* This entry is special, and used for the attribute at index 0 + * only, and specifies special data about the policy, namely it + * specifies the "boundary type" where strict length validation + * starts for any attribute types >= this value, also, strict + * nesting validation starts here. + * + * Additionally, it means that NLA_UNSPEC is actually NLA_REJECT + * for any types >= this, so need to use NLA_MIN_LEN to get the + * previous pure { .len = xyz } behaviour. The advantage of this + * is that types not specified in the policy will be rejected. + * + * For completely new families it should be set to 1 so that the + * validation is enforced for all attributes. For existing ones + * it should be set at least when new attributes are added to + * the enum used by the policy, and be set to the new value that + * was added to enforce strict validation from thereon. + */ + u16 strict_start_type; }; }; #define NLA_POLICY_EXACT_LEN(_len) { .type = NLA_EXACT_LEN, .len = _len } #define NLA_POLICY_EXACT_LEN_WARN(_len) { .type = NLA_EXACT_LEN_WARN, \ .len = _len } +#define NLA_POLICY_MIN_LEN(_len) { .type = NLA_MIN_LEN, .len = _len } #define NLA_POLICY_ETH_ADDR NLA_POLICY_EXACT_LEN(ETH_ALEN) #define NLA_POLICY_ETH_ADDR_COMPAT NLA_POLICY_EXACT_LEN_WARN(ETH_ALEN) @@ -365,21 +387,52 @@ struct nl_info { bool skip_notify; }; +/** + * enum netlink_validation - netlink message/attribute validation levels + * @NL_VALIDATE_LIBERAL: Old-style "be liberal" validation, not caring about + * extra data at the end of the message, attributes being longer than + * they should be, or unknown attributes being present. + * @NL_VALIDATE_TRAILING: Reject junk data encountered after attribute parsing. + * @NL_VALIDATE_MAXTYPE: Reject attributes > max type; Together with _TRAILING + * this is equivalent to the old nla_parse_strict()/nlmsg_parse_strict(). + * @NL_VALIDATE_UNSPEC: Reject attributes with NLA_UNSPEC in the policy. + * This can safely be set by the kernel when the given policy has no + * NLA_UNSPEC anymore, and can thus be used to ensure policy entries + * are enforced going forward. + * @NL_VALIDATE_STRICT_ATTRS: strict attribute policy parsing (e.g. + * U8, U16, U32 must have exact size, etc.) + * @NL_VALIDATE_NESTED: Check that NLA_F_NESTED is set for NLA_NESTED(_ARRAY) + * and unset for other policies. + */ +enum netlink_validation { + NL_VALIDATE_LIBERAL = 0, + NL_VALIDATE_TRAILING = BIT(0), + NL_VALIDATE_MAXTYPE = BIT(1), + NL_VALIDATE_UNSPEC = BIT(2), + NL_VALIDATE_STRICT_ATTRS = BIT(3), + NL_VALIDATE_NESTED = BIT(4), +}; + +#define NL_VALIDATE_DEPRECATED_STRICT (NL_VALIDATE_TRAILING |\ + NL_VALIDATE_MAXTYPE) +#define NL_VALIDATE_STRICT (NL_VALIDATE_TRAILING |\ + NL_VALIDATE_MAXTYPE |\ + NL_VALIDATE_UNSPEC |\ + NL_VALIDATE_STRICT_ATTRS |\ + NL_VALIDATE_NESTED) + int netlink_rcv_skb(struct sk_buff *skb, int (*cb)(struct sk_buff *, struct nlmsghdr *, struct netlink_ext_ack *)); int nlmsg_notify(struct sock *sk, struct sk_buff *skb, u32 portid, unsigned int group, int report, gfp_t flags); -int nla_validate(const struct nlattr *head, int len, int maxtype, - const struct nla_policy *policy, - struct netlink_ext_ack *extack); -int nla_parse(struct nlattr **tb, int maxtype, const struct nlattr *head, - int len, const struct nla_policy *policy, - struct netlink_ext_ack *extack); -int nla_parse_strict(struct nlattr **tb, int maxtype, const struct nlattr *head, - int len, const struct nla_policy *policy, - struct netlink_ext_ack *extack); +int __nla_validate(const struct nlattr *head, int len, int maxtype, + const struct nla_policy *policy, unsigned int validate, + struct netlink_ext_ack *extack); +int __nla_parse(struct nlattr **tb, int maxtype, const struct nlattr *head, + int len, const struct nla_policy *policy, unsigned int validate, + struct netlink_ext_ack *extack); int nla_policy_len(const struct nla_policy *, int); struct nlattr *nla_find(const struct nlattr *head, int len, int attrtype); size_t nla_strlcpy(char *dst, const struct nlattr *nla, size_t dstsize); @@ -508,42 +561,167 @@ nlmsg_next(const struct nlmsghdr *nlh, int *remaining) } /** - * nlmsg_parse - parse attributes of a netlink message + * nla_parse - Parse a stream of attributes into a tb buffer + * @tb: destination array with maxtype+1 elements + * @maxtype: maximum attribute type to be expected + * @head: head of attribute stream + * @len: length of attribute stream + * @policy: validation policy + * @extack: extended ACK pointer + * + * Parses a stream of attributes and stores a pointer to each attribute in + * the tb array accessible via the attribute type. Attributes with a type + * exceeding maxtype will be rejected, policy must be specified, attributes + * will be validated in the strictest way possible. + * + * Returns 0 on success or a negative error code. + */ +static inline int nla_parse(struct nlattr **tb, int maxtype, + const struct nlattr *head, int len, + const struct nla_policy *policy, + struct netlink_ext_ack *extack) +{ + return __nla_parse(tb, maxtype, head, len, policy, + NL_VALIDATE_STRICT, extack); +} + +/** + * nla_parse_deprecated - Parse a stream of attributes into a tb buffer + * @tb: destination array with maxtype+1 elements + * @maxtype: maximum attribute type to be expected + * @head: head of attribute stream + * @len: length of attribute stream + * @policy: validation policy + * @extack: extended ACK pointer + * + * Parses a stream of attributes and stores a pointer to each attribute in + * the tb array accessible via the attribute type. Attributes with a type + * exceeding maxtype will be ignored and attributes from the policy are not + * always strictly validated (only for new attributes). + * + * Returns 0 on success or a negative error code. + */ +static inline int nla_parse_deprecated(struct nlattr **tb, int maxtype, + const struct nlattr *head, int len, + const struct nla_policy *policy, + struct netlink_ext_ack *extack) +{ + return __nla_parse(tb, maxtype, head, len, policy, + NL_VALIDATE_LIBERAL, extack); +} + +/** + * nla_parse_deprecated_strict - Parse a stream of attributes into a tb buffer + * @tb: destination array with maxtype+1 elements + * @maxtype: maximum attribute type to be expected + * @head: head of attribute stream + * @len: length of attribute stream + * @policy: validation policy + * @extack: extended ACK pointer + * + * Parses a stream of attributes and stores a pointer to each attribute in + * the tb array accessible via the attribute type. Attributes with a type + * exceeding maxtype will be rejected as well as trailing data, but the + * policy is not completely strictly validated (only for new attributes). + * + * Returns 0 on success or a negative error code. + */ +static inline int nla_parse_deprecated_strict(struct nlattr **tb, int maxtype, + const struct nlattr *head, + int len, + const struct nla_policy *policy, + struct netlink_ext_ack *extack) +{ + return __nla_parse(tb, maxtype, head, len, policy, + NL_VALIDATE_DEPRECATED_STRICT, extack); +} + +/** + * __nlmsg_parse - parse attributes of a netlink message * @nlh: netlink message header * @hdrlen: length of family specific header * @tb: destination array with maxtype+1 elements * @maxtype: maximum attribute type to be expected * @policy: validation policy + * @validate: validation strictness * @extack: extended ACK report struct * * See nla_parse() */ -static inline int nlmsg_parse(const struct nlmsghdr *nlh, int hdrlen, - struct nlattr *tb[], int maxtype, - const struct nla_policy *policy, - struct netlink_ext_ack *extack) +static inline int __nlmsg_parse(const struct nlmsghdr *nlh, int hdrlen, + struct nlattr *tb[], int maxtype, + const struct nla_policy *policy, + unsigned int validate, + struct netlink_ext_ack *extack) { if (nlh->nlmsg_len < nlmsg_msg_size(hdrlen)) { NL_SET_ERR_MSG(extack, "Invalid header length"); return -EINVAL; } - return nla_parse(tb, maxtype, nlmsg_attrdata(nlh, hdrlen), - nlmsg_attrlen(nlh, hdrlen), policy, extack); + return __nla_parse(tb, maxtype, nlmsg_attrdata(nlh, hdrlen), + nlmsg_attrlen(nlh, hdrlen), policy, validate, + extack); } -static inline int nlmsg_parse_strict(const struct nlmsghdr *nlh, int hdrlen, - struct nlattr *tb[], int maxtype, - const struct nla_policy *policy, - struct netlink_ext_ack *extack) +/** + * nlmsg_parse - parse attributes of a netlink message + * @nlh: netlink message header + * @hdrlen: length of family specific header + * @tb: destination array with maxtype+1 elements + * @maxtype: maximum attribute type to be expected + * @validate: validation strictness + * @extack: extended ACK report struct + * + * See nla_parse() + */ +static inline int nlmsg_parse(const struct nlmsghdr *nlh, int hdrlen, + struct nlattr *tb[], int maxtype, + const struct nla_policy *policy, + struct netlink_ext_ack *extack) { - if (nlh->nlmsg_len < nlmsg_msg_size(hdrlen)) { - NL_SET_ERR_MSG(extack, "Invalid header length"); - return -EINVAL; - } + return __nla_parse(tb, maxtype, nlmsg_attrdata(nlh, hdrlen), + nlmsg_attrlen(nlh, hdrlen), policy, + NL_VALIDATE_STRICT, extack); +} - return nla_parse_strict(tb, maxtype, nlmsg_attrdata(nlh, hdrlen), - nlmsg_attrlen(nlh, hdrlen), policy, extack); +/** + * nlmsg_parse_deprecated - parse attributes of a netlink message + * @nlh: netlink message header + * @hdrlen: length of family specific header + * @tb: destination array with maxtype+1 elements + * @maxtype: maximum attribute type to be expected + * @extack: extended ACK report struct + * + * See nla_parse_deprecated() + */ +static inline int nlmsg_parse_deprecated(const struct nlmsghdr *nlh, int hdrlen, + struct nlattr *tb[], int maxtype, + const struct nla_policy *policy, + struct netlink_ext_ack *extack) +{ + return __nlmsg_parse(nlh, hdrlen, tb, maxtype, policy, + NL_VALIDATE_LIBERAL, extack); +} + +/** + * nlmsg_parse_deprecated_strict - parse attributes of a netlink message + * @nlh: netlink message header + * @hdrlen: length of family specific header + * @tb: destination array with maxtype+1 elements + * @maxtype: maximum attribute type to be expected + * @extack: extended ACK report struct + * + * See nla_parse_deprecated_strict() + */ +static inline int +nlmsg_parse_deprecated_strict(const struct nlmsghdr *nlh, int hdrlen, + struct nlattr *tb[], int maxtype, + const struct nla_policy *policy, + struct netlink_ext_ack *extack) +{ + return __nlmsg_parse(nlh, hdrlen, tb, maxtype, policy, + NL_VALIDATE_DEPRECATED_STRICT, extack); } /** @@ -562,26 +740,75 @@ static inline struct nlattr *nlmsg_find_attr(const struct nlmsghdr *nlh, } /** - * nlmsg_validate - validate a netlink message including attributes + * nla_validate_deprecated - Validate a stream of attributes + * @head: head of attribute stream + * @len: length of attribute stream + * @maxtype: maximum attribute type to be expected + * @policy: validation policy + * @validate: validation strictness + * @extack: extended ACK report struct + * + * Validates all attributes in the specified attribute stream against the + * specified policy. Validation is done in liberal mode. + * See documenation of struct nla_policy for more details. + * + * Returns 0 on success or a negative error code. + */ +static inline int nla_validate_deprecated(const struct nlattr *head, int len, + int maxtype, + const struct nla_policy *policy, + struct netlink_ext_ack *extack) +{ + return __nla_validate(head, len, maxtype, policy, NL_VALIDATE_LIBERAL, + extack); +} + +/** + * nla_validate - Validate a stream of attributes + * @head: head of attribute stream + * @len: length of attribute stream + * @maxtype: maximum attribute type to be expected + * @policy: validation policy + * @validate: validation strictness + * @extack: extended ACK report struct + * + * Validates all attributes in the specified attribute stream against the + * specified policy. Validation is done in strict mode. + * See documenation of struct nla_policy for more details. + * + * Returns 0 on success or a negative error code. + */ +static inline int nla_validate(const struct nlattr *head, int len, int maxtype, + const struct nla_policy *policy, + struct netlink_ext_ack *extack) +{ + return __nla_validate(head, len, maxtype, policy, NL_VALIDATE_STRICT, + extack); +} + +/** + * nlmsg_validate_deprecated - validate a netlink message including attributes * @nlh: netlinket message header * @hdrlen: length of familiy specific header * @maxtype: maximum attribute type to be expected * @policy: validation policy * @extack: extended ACK report struct */ -static inline int nlmsg_validate(const struct nlmsghdr *nlh, - int hdrlen, int maxtype, - const struct nla_policy *policy, - struct netlink_ext_ack *extack) +static inline int nlmsg_validate_deprecated(const struct nlmsghdr *nlh, + int hdrlen, int maxtype, + const struct nla_policy *policy, + struct netlink_ext_ack *extack) { if (nlh->nlmsg_len < nlmsg_msg_size(hdrlen)) return -EINVAL; - return nla_validate(nlmsg_attrdata(nlh, hdrlen), - nlmsg_attrlen(nlh, hdrlen), maxtype, policy, - extack); + return __nla_validate(nlmsg_attrdata(nlh, hdrlen), + nlmsg_attrlen(nlh, hdrlen), maxtype, + policy, NL_VALIDATE_LIBERAL, extack); } + + /** * nlmsg_report - need to report back to application? * @nlh: netlink message header @@ -909,8 +1136,32 @@ static inline int nla_parse_nested(struct nlattr *tb[], int maxtype, const struct nla_policy *policy, struct netlink_ext_ack *extack) { - return nla_parse(tb, maxtype, nla_data(nla), nla_len(nla), policy, - extack); + if (!(nla->nla_type & NLA_F_NESTED)) { + NL_SET_ERR_MSG_ATTR(extack, nla, "NLA_F_NESTED is missing"); + return -EINVAL; + } + + return __nla_parse(tb, maxtype, nla_data(nla), nla_len(nla), policy, + NL_VALIDATE_STRICT, extack); +} + +/** + * nla_parse_nested_deprecated - parse nested attributes + * @tb: destination array with maxtype+1 elements + * @maxtype: maximum attribute type to be expected + * @nla: attribute containing the nested attributes + * @policy: validation policy + * @extack: extended ACK report struct + * + * See nla_parse_deprecated() + */ +static inline int nla_parse_nested_deprecated(struct nlattr *tb[], int maxtype, + const struct nlattr *nla, + const struct nla_policy *policy, + struct netlink_ext_ack *extack) +{ + return __nla_parse(tb, maxtype, nla_data(nla), nla_len(nla), policy, + NL_VALIDATE_LIBERAL, extack); } /** @@ -1415,13 +1666,18 @@ static inline void *nla_memdup(const struct nlattr *src, gfp_t gfp) } /** - * nla_nest_start - Start a new level of nested attributes + * nla_nest_start_noflag - Start a new level of nested attributes * @skb: socket buffer to add attributes to * @attrtype: attribute type of container * - * Returns the container attribute + * This function exists for backward compatibility to use in APIs which never + * marked their nest attributes with NLA_F_NESTED flag. New APIs should use + * nla_nest_start() which sets the flag. + * + * Returns the container attribute or NULL on error */ -static inline struct nlattr *nla_nest_start(struct sk_buff *skb, int attrtype) +static inline struct nlattr *nla_nest_start_noflag(struct sk_buff *skb, + int attrtype) { struct nlattr *start = (struct nlattr *)skb_tail_pointer(skb); @@ -1431,6 +1687,21 @@ static inline struct nlattr *nla_nest_start(struct sk_buff *skb, int attrtype) return start; } +/** + * nla_nest_start - Start a new level of nested attributes, with NLA_F_NESTED + * @skb: socket buffer to add attributes to + * @attrtype: attribute type of container + * + * Unlike nla_nest_start_noflag(), mark the nest attribute with NLA_F_NESTED + * flag. This is the preferred function to use in new code. + * + * Returns the container attribute or NULL on error + */ +static inline struct nlattr *nla_nest_start(struct sk_buff *skb, int attrtype) +{ + return nla_nest_start_noflag(skb, attrtype | NLA_F_NESTED); +} + /** * nla_nest_end - Finalize nesting of attributes * @skb: socket buffer the attributes are stored in @@ -1465,6 +1736,7 @@ static inline void nla_nest_cancel(struct sk_buff *skb, struct nlattr *start) * @start: container attribute * @maxtype: maximum attribute type to be expected * @policy: validation policy + * @validate: validation strictness * @extack: extended ACK report struct * * Validates all attributes in the nested attribute stream against the @@ -1473,12 +1745,22 @@ static inline void nla_nest_cancel(struct sk_buff *skb, struct nlattr *start) * * Returns 0 on success or a negative error code. */ -static inline int nla_validate_nested(const struct nlattr *start, int maxtype, - const struct nla_policy *policy, - struct netlink_ext_ack *extack) +static inline int __nla_validate_nested(const struct nlattr *start, int maxtype, + const struct nla_policy *policy, + unsigned int validate, + struct netlink_ext_ack *extack) +{ + return __nla_validate(nla_data(start), nla_len(start), maxtype, policy, + validate, extack); +} + +static inline int +nla_validate_nested_deprecated(const struct nlattr *start, int maxtype, + const struct nla_policy *policy, + struct netlink_ext_ack *extack) { - return nla_validate(nla_data(start), nla_len(start), maxtype, policy, - extack); + return __nla_validate_nested(start, maxtype, policy, + NL_VALIDATE_LIBERAL, extack); } /** diff --git a/include/net/netns/conntrack.h b/include/net/netns/conntrack.h index f19b53130bf784f18df2b2894ddfa2ea95f09e32..806454e767bf422893a523eebf96b67d38a532cb 100644 --- a/include/net/netns/conntrack.h +++ b/include/net/netns/conntrack.h @@ -24,9 +24,9 @@ struct nf_generic_net { struct nf_tcp_net { unsigned int timeouts[TCP_CONNTRACK_TIMEOUT_MAX]; - unsigned int tcp_loose; - unsigned int tcp_be_liberal; - unsigned int tcp_max_retrans; + int tcp_loose; + int tcp_be_liberal; + int tcp_max_retrans; }; enum udp_conntrack { diff --git a/include/net/netns/ipv4.h b/include/net/netns/ipv4.h index 104a6669e3445c1ac43fee5fba9060c565005b62..7698460a3dd1e5070e12d406b3ee58834688cdc9 100644 --- a/include/net/netns/ipv4.h +++ b/include/net/netns/ipv4.h @@ -9,6 +9,7 @@ #include #include #include +#include struct tcpm_hash_bucket; struct ctl_table_header; @@ -217,5 +218,6 @@ struct netns_ipv4 { unsigned int ipmr_seq; /* protected by rtnl_mutex */ atomic_t rt_genid; + siphash_key_t ip_id_key; }; #endif diff --git a/include/net/netns/ipv6.h b/include/net/netns/ipv6.h index b028a1dc150ddde3e3535351748cef72af56c1ce..5e61b5a8635d7a01181886e3968d258eb7d74698 100644 --- a/include/net/netns/ipv6.h +++ b/include/net/netns/ipv6.h @@ -8,6 +8,7 @@ #ifndef __NETNS_IPV6_H__ #define __NETNS_IPV6_H__ #include +#include struct ctl_table_header; @@ -33,6 +34,10 @@ struct netns_sysctl_ipv6 { int auto_flowlabels; int icmpv6_time; int icmpv6_echo_ignore_all; + int icmpv6_echo_ignore_multicast; + int icmpv6_echo_ignore_anycast; + DECLARE_BITMAP(icmpv6_ratemask, ICMPV6_MSG_MAX + 1); + unsigned long *icmpv6_ratemask_ptr; int anycast_src_echo_reply; int ip_nonlocal_bind; int fwmark_reflect; diff --git a/include/net/pkt_cls.h b/include/net/pkt_cls.h index d5e7a1af346fe06b6ad8f8e091a3ad8a77e7d399..514e3c80ecc1152474228e0ea53b6037cc18f482 100644 --- a/include/net/pkt_cls.h +++ b/include/net/pkt_cls.h @@ -100,6 +100,11 @@ int tcf_classify(struct sk_buff *skb, const struct tcf_proto *tp, struct tcf_result *res, bool compat_mode); #else +static inline bool tcf_block_shared(struct tcf_block *block) +{ + return false; +} + static inline int tcf_block_get(struct tcf_block **p_block, struct tcf_proto __rcu **p_filter_chain, struct Qdisc *q, @@ -371,30 +376,6 @@ static inline bool tcf_exts_has_actions(struct tcf_exts *exts) #endif } -/** - * tcf_exts_has_one_action - check if exactly one action is present - * @exts: tc filter extensions handle - * - * Returns true if exactly one action is present. - */ -static inline bool tcf_exts_has_one_action(struct tcf_exts *exts) -{ -#ifdef CONFIG_NET_CLS_ACT - return exts->nr_actions == 1; -#else - return false; -#endif -} - -static inline struct tc_action *tcf_exts_first_action(struct tcf_exts *exts) -{ -#ifdef CONFIG_NET_CLS_ACT - return exts->actions[0]; -#else - return NULL; -#endif -} - /** * tcf_exts_exec - execute tc filter extensions * @skb: socket buffer @@ -784,12 +765,14 @@ tc_cls_flower_offload_flow_rule(struct tc_cls_flower_offload *tc_flow_cmd) enum tc_matchall_command { TC_CLSMATCHALL_REPLACE, TC_CLSMATCHALL_DESTROY, + TC_CLSMATCHALL_STATS, }; struct tc_cls_matchall_offload { struct tc_cls_common_offload common; enum tc_matchall_command command; - struct tcf_exts *exts; + struct flow_rule *rule; + struct flow_stats stats; unsigned long cookie; }; diff --git a/include/net/psample.h b/include/net/psample.h index 9b80f814ab043b733ec3efbcb0c30f9e1f690815..37a4df2325b2eff3ad8d17df2c76fde184dd87c1 100644 --- a/include/net/psample.h +++ b/include/net/psample.h @@ -3,7 +3,6 @@ #define __NET_PSAMPLE_H #include -#include #include struct psample_group { diff --git a/include/net/request_sock.h b/include/net/request_sock.h index 21a5243fecd160825deb50fbe8e5bdfc1d4d39dc..9dfd7960d90a1c365df6e191223264b66599f251 100644 --- a/include/net/request_sock.h +++ b/include/net/request_sock.h @@ -106,10 +106,8 @@ reqsk_alloc(const struct request_sock_ops *ops, struct sock *sk_listener, return req; } -static inline void reqsk_free(struct request_sock *req) +static inline void __reqsk_free(struct request_sock *req) { - WARN_ON_ONCE(refcount_read(&req->rsk_refcnt) != 0); - req->rsk_ops->destructor(req); if (req->rsk_listener) sock_put(req->rsk_listener); @@ -117,6 +115,12 @@ static inline void reqsk_free(struct request_sock *req) kmem_cache_free(req->rsk_ops->slab, req); } +static inline void reqsk_free(struct request_sock *req) +{ + WARN_ON_ONCE(refcount_read(&req->rsk_refcnt) != 0); + __reqsk_free(req); +} + static inline void reqsk_put(struct request_sock *req) { if (refcount_dec_and_test(&req->rsk_refcnt)) diff --git a/include/net/route.h b/include/net/route.h index 9883dc82f7233dede86ec2e5cfeddf09cf56d62d..96f6c9ae33c2357b1d5c9877c95d8bcd482c18cb 100644 --- a/include/net/route.h +++ b/include/net/route.h @@ -29,6 +29,8 @@ #include #include #include +#include +#include #include #include #include @@ -55,12 +57,15 @@ struct rtable { unsigned int rt_flags; __u16 rt_type; __u8 rt_is_input; - __u8 rt_uses_gateway; + u8 rt_gw_family; int rt_iif; /* Info on neighbour */ - __be32 rt_gateway; + union { + __be32 rt_gw4; + struct in6_addr rt_gw6; + }; /* Miscellaneous cached information */ u32 rt_mtu_locked:1, @@ -82,8 +87,8 @@ static inline bool rt_is_output_route(const struct rtable *rt) static inline __be32 rt_nexthop(const struct rtable *rt, __be32 daddr) { - if (rt->rt_gateway) - return rt->rt_gateway; + if (rt->rt_gw_family == AF_INET) + return rt->rt_gw4; return daddr; } @@ -347,4 +352,34 @@ static inline int ip4_dst_hoplimit(const struct dst_entry *dst) return hoplimit; } +static inline struct neighbour *ip_neigh_gw4(struct net_device *dev, + __be32 daddr) +{ + struct neighbour *neigh; + + neigh = __ipv4_neigh_lookup_noref(dev, daddr); + if (unlikely(!neigh)) + neigh = __neigh_create(&arp_tbl, &daddr, dev, false); + + return neigh; +} + +static inline struct neighbour *ip_neigh_for_gw(struct rtable *rt, + struct sk_buff *skb, + bool *is_v6gw) +{ + struct net_device *dev = rt->dst.dev; + struct neighbour *neigh; + + if (likely(rt->rt_gw_family == AF_INET)) { + neigh = ip_neigh_gw4(dev, rt->rt_gw4); + } else if (rt->rt_gw_family == AF_INET6) { + neigh = ip_neigh_gw6(dev, &rt->rt_gw6); + *is_v6gw = true; + } else { + neigh = ip_neigh_gw4(dev, ip_hdr(skb)->daddr); + } + return neigh; +} + #endif /* _ROUTE_H */ diff --git a/include/net/nexthop.h b/include/net/rtnh.h similarity index 94% rename from include/net/nexthop.h rename to include/net/rtnh.h index 902ff382a6dcc6e3f0c1a9a244061e5a84264915..aa2cfc508f7c237dc47b68f0de751b5d06c702e6 100644 --- a/include/net/nexthop.h +++ b/include/net/rtnh.h @@ -1,6 +1,6 @@ /* SPDX-License-Identifier: GPL-2.0 */ -#ifndef __NET_NEXTHOP_H -#define __NET_NEXTHOP_H +#ifndef __NET_RTNH_H +#define __NET_RTNH_H #include #include diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h index a2b38b3deeca2096d01d536d9a492047827ff994..21f434f3ac9e34a9a00eb40e490481757b10f660 100644 --- a/include/net/sch_generic.h +++ b/include/net/sch_generic.h @@ -52,10 +52,7 @@ struct qdisc_size_table { struct qdisc_skb_head { struct sk_buff *head; struct sk_buff *tail; - union { - u32 qlen; - atomic_t atomic_qlen; - }; + __u32 qlen; spinlock_t lock; }; @@ -113,6 +110,9 @@ struct Qdisc { spinlock_t busylock ____cacheline_aligned_in_smp; spinlock_t seqlock; + + /* for NOLOCK qdisc, true if there are no enqueued skbs */ + bool empty; struct rcu_head rcu; }; @@ -143,11 +143,24 @@ static inline bool qdisc_is_running(struct Qdisc *qdisc) return (raw_read_seqcount(&qdisc->running) & 1) ? true : false; } +static inline bool qdisc_is_percpu_stats(const struct Qdisc *q) +{ + return q->flags & TCQ_F_CPUSTATS; +} + +static inline bool qdisc_is_empty(const struct Qdisc *qdisc) +{ + if (qdisc_is_percpu_stats(qdisc)) + return qdisc->empty; + return !qdisc->q.qlen; +} + static inline bool qdisc_run_begin(struct Qdisc *qdisc) { if (qdisc->flags & TCQ_F_NOLOCK) { if (!spin_trylock(&qdisc->seqlock)) return false; + qdisc->empty = false; } else if (qdisc_is_running(qdisc)) { return false; } @@ -351,13 +364,10 @@ struct tcf_proto { }; struct qdisc_skb_cb { - union { - struct { - unsigned int pkt_len; - u16 slave_dev_queue_mapping; - u16 tc_classid; - }; - struct bpf_flow_keys *flow_keys; + struct { + unsigned int pkt_len; + u16 slave_dev_queue_mapping; + u16 tc_classid; }; #define QDISC_CB_PRIV_LEN 20 unsigned char data[QDISC_CB_PRIV_LEN]; @@ -470,19 +480,27 @@ static inline void qdisc_cb_private_validate(const struct sk_buff *skb, int sz) BUILD_BUG_ON(sizeof(qcb->data) < sz); } +static inline int qdisc_qlen_cpu(const struct Qdisc *q) +{ + return this_cpu_ptr(q->cpu_qstats)->qlen; +} + static inline int qdisc_qlen(const struct Qdisc *q) { return q->q.qlen; } -static inline u32 qdisc_qlen_sum(const struct Qdisc *q) +static inline int qdisc_qlen_sum(const struct Qdisc *q) { - u32 qlen = q->qstats.qlen; + __u32 qlen = q->qstats.qlen; + int i; - if (q->flags & TCQ_F_NOLOCK) - qlen += atomic_read(&q->q.atomic_qlen); - else + if (qdisc_is_percpu_stats(q)) { + for_each_possible_cpu(i) + qlen += per_cpu_ptr(q->cpu_qstats, i)->qlen; + } else { qlen += q->q.qlen; + } return qlen; } @@ -736,7 +754,7 @@ static inline bool qdisc_all_tx_empty(const struct net_device *dev) struct netdev_queue *txq = netdev_get_tx_queue(dev, i); const struct Qdisc *q = rcu_dereference(txq->qdisc); - if (q->q.qlen) { + if (!qdisc_is_empty(q)) { rcu_read_unlock(); return false; } @@ -806,11 +824,6 @@ static inline int qdisc_enqueue(struct sk_buff *skb, struct Qdisc *sch, return sch->enqueue(skb, sch, to_free); } -static inline bool qdisc_is_percpu_stats(const struct Qdisc *q) -{ - return q->flags & TCQ_F_CPUSTATS; -} - static inline void _bstats_update(struct gnet_stats_basic_packed *bstats, __u64 bytes, __u32 packets) { @@ -878,14 +891,14 @@ static inline void qdisc_qstats_cpu_backlog_inc(struct Qdisc *sch, this_cpu_add(sch->cpu_qstats->backlog, qdisc_pkt_len(skb)); } -static inline void qdisc_qstats_atomic_qlen_inc(struct Qdisc *sch) +static inline void qdisc_qstats_cpu_qlen_inc(struct Qdisc *sch) { - atomic_inc(&sch->q.atomic_qlen); + this_cpu_inc(sch->cpu_qstats->qlen); } -static inline void qdisc_qstats_atomic_qlen_dec(struct Qdisc *sch) +static inline void qdisc_qstats_cpu_qlen_dec(struct Qdisc *sch) { - atomic_dec(&sch->q.atomic_qlen); + this_cpu_dec(sch->cpu_qstats->qlen); } static inline void qdisc_qstats_cpu_requeues_inc(struct Qdisc *sch) @@ -1095,6 +1108,32 @@ static inline struct sk_buff *qdisc_peek_dequeued(struct Qdisc *sch) return skb; } +static inline void qdisc_update_stats_at_dequeue(struct Qdisc *sch, + struct sk_buff *skb) +{ + if (qdisc_is_percpu_stats(sch)) { + qdisc_qstats_cpu_backlog_dec(sch, skb); + qdisc_bstats_cpu_update(sch, skb); + qdisc_qstats_cpu_qlen_dec(sch); + } else { + qdisc_qstats_backlog_dec(sch, skb); + qdisc_bstats_update(sch, skb); + sch->q.qlen--; + } +} + +static inline void qdisc_update_stats_at_enqueue(struct Qdisc *sch, + unsigned int pkt_len) +{ + if (qdisc_is_percpu_stats(sch)) { + qdisc_qstats_cpu_qlen_inc(sch); + this_cpu_add(sch->cpu_qstats->backlog, pkt_len); + } else { + sch->qstats.backlog += pkt_len; + sch->q.qlen++; + } +} + /* use instead of qdisc->dequeue() for all qdiscs queried with ->peek() */ static inline struct sk_buff *qdisc_dequeue_peeked(struct Qdisc *sch) { @@ -1102,8 +1141,13 @@ static inline struct sk_buff *qdisc_dequeue_peeked(struct Qdisc *sch) if (skb) { skb = __skb_dequeue(&sch->gso_skb); - qdisc_qstats_backlog_dec(sch, skb); - sch->q.qlen--; + if (qdisc_is_percpu_stats(sch)) { + qdisc_qstats_cpu_backlog_dec(sch, skb); + qdisc_qstats_cpu_qlen_dec(sch); + } else { + qdisc_qstats_backlog_dec(sch, skb); + sch->q.qlen--; + } } else { skb = sch->dequeue(sch); } diff --git a/include/net/sctp/sctp.h b/include/net/sctp/sctp.h index 1d13ec3f2707e5777d564aaf15690f10a15b05b6..eefdfa5abf6e088aaf6a27d4113007c19d6d2f8f 100644 --- a/include/net/sctp/sctp.h +++ b/include/net/sctp/sctp.h @@ -421,7 +421,7 @@ static inline void sctp_skb_set_owner_r(struct sk_buff *skb, struct sock *sk) /* * This mimics the behavior of skb_set_owner_r */ - sk->sk_forward_alloc -= event->rmem_len; + sk_mem_charge(sk, event->rmem_len); } /* Tests if the list has one and only one entry. */ diff --git a/include/net/sctp/ulpqueue.h b/include/net/sctp/ulpqueue.h index bb0ecba3db2b35e7d6539cfc7133e8b7b63c2207..f4ac7117ff2918868ea97e384cbc2087a1a968d8 100644 --- a/include/net/sctp/ulpqueue.h +++ b/include/net/sctp/ulpqueue.h @@ -59,7 +59,7 @@ void sctp_ulpq_free(struct sctp_ulpq *); int sctp_ulpq_tail_data(struct sctp_ulpq *, struct sctp_chunk *, gfp_t); /* Add a new event for propagation to the ULP. */ -int sctp_ulpq_tail_event(struct sctp_ulpq *, struct sctp_ulpevent *ev); +int sctp_ulpq_tail_event(struct sctp_ulpq *, struct sk_buff_head *skb_list); /* Renege previously received chunks. */ void sctp_ulpq_renege(struct sctp_ulpq *, struct sctp_chunk *, gfp_t); diff --git a/include/net/sock.h b/include/net/sock.h index 341f8bafa0cf585fc72e5819054f1b2f15a8e338..4d208c0f9c14b7c8f5975dcc69e13d89d1880df5 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -236,6 +236,8 @@ struct sock_common { /* public: */ }; +struct bpf_sk_storage; + /** * struct sock - network layer representation of sockets * @__sk_common: shared layout with inet_timewait_sock @@ -368,6 +370,7 @@ struct sock { atomic_t sk_drops; int sk_rcvlowat; struct sk_buff_head sk_error_queue; + struct sk_buff *sk_rx_skb_cache; struct sk_buff_head sk_receive_queue; /* * The backlog queue is special, it is always used with @@ -414,6 +417,7 @@ struct sock { struct sk_buff *sk_send_head; struct rb_root tcp_rtx_queue; }; + struct sk_buff *sk_tx_skb_cache; struct sk_buff_head sk_write_queue; __s32 sk_peek_off; int sk_write_pending; @@ -508,6 +512,9 @@ struct sock { #endif void (*sk_destruct)(struct sock *sk); struct sock_reuseport __rcu *sk_reuseport_cb; +#ifdef CONFIG_BPF_SYSCALL + struct bpf_sk_storage __rcu *sk_bpf_storage; +#endif struct rcu_head sk_rcu; }; @@ -966,7 +973,7 @@ static inline void sock_rps_record_flow_hash(__u32 hash) static inline void sock_rps_record_flow(const struct sock *sk) { #ifdef CONFIG_RPS - if (static_key_false(&rfs_needed)) { + if (static_branch_unlikely(&rfs_needed)) { /* Reading sk->sk_rxhash might incur an expensive cache line * miss. * @@ -1466,6 +1473,11 @@ static inline void sk_wmem_free_skb(struct sock *sk, struct sk_buff *skb) sock_set_flag(sk, SOCK_QUEUE_SHRUNK); sk->sk_wmem_queued -= skb->truesize; sk_mem_uncharge(sk, skb->truesize); + if (!sk->sk_tx_skb_cache) { + skb_zcopy_clear(skb, true); + sk->sk_tx_skb_cache = skb; + return; + } __kfree_skb(skb); } @@ -1607,6 +1619,8 @@ int sock_setsockopt(struct socket *sock, int level, int op, int sock_getsockopt(struct socket *sock, int level, int op, char __user *optval, int __user *optlen); +int sock_gettstamp(struct socket *sock, void __user *userstamp, + bool timeval, bool time32); struct sk_buff *sock_alloc_send_skb(struct sock *sk, unsigned long size, int noblock, int *errcode); struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, @@ -2427,6 +2441,15 @@ static inline void skb_setup_tx_timestamp(struct sk_buff *skb, __u16 tsflags) static inline void sk_eat_skb(struct sock *sk, struct sk_buff *skb) { __skb_unlink(skb, &sk->sk_receive_queue); + if ( +#ifdef CONFIG_RPS + !static_branch_unlikely(&rps_needed) && +#endif + !sk->sk_rx_skb_cache) { + sk->sk_rx_skb_cache = skb; + skb_orphan(skb); + return; + } __kfree_skb(skb); } @@ -2487,8 +2510,6 @@ static inline bool sk_listener(const struct sock *sk) } void sock_enable_timestamp(struct sock *sk, int flag); -int sock_get_timestamp(struct sock *, struct timeval __user *); -int sock_get_timestampns(struct sock *, struct timespec __user *); int sock_recv_errqueue(struct sock *sk, struct msghdr *msg, int len, int level, int type); diff --git a/include/net/tc_act/tc_ife.h b/include/net/tc_act/tc_ife.h index 86d13b01b39d52cbfcee49e81f8f4cd2fbf0eaf2..c7f24a2da1cad46ade04d5103a8883cebff034e9 100644 --- a/include/net/tc_act/tc_ife.h +++ b/include/net/tc_act/tc_ife.h @@ -5,7 +5,8 @@ #include #include #include -#include + +struct module; struct tcf_ife_params { u8 eth_dst[ETH_ALEN]; diff --git a/include/net/tc_act/tc_police.h b/include/net/tc_act/tc_police.h new file mode 100644 index 0000000000000000000000000000000000000000..8b9ef366426256ee83943f9da05bd1c301ffbf23 --- /dev/null +++ b/include/net/tc_act/tc_police.h @@ -0,0 +1,70 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __NET_TC_POLICE_H +#define __NET_TC_POLICE_H + +#include + +struct tcf_police_params { + int tcfp_result; + u32 tcfp_ewma_rate; + s64 tcfp_burst; + u32 tcfp_mtu; + s64 tcfp_mtu_ptoks; + struct psched_ratecfg rate; + bool rate_present; + struct psched_ratecfg peak; + bool peak_present; + struct rcu_head rcu; +}; + +struct tcf_police { + struct tc_action common; + struct tcf_police_params __rcu *params; + + spinlock_t tcfp_lock ____cacheline_aligned_in_smp; + s64 tcfp_toks; + s64 tcfp_ptoks; + s64 tcfp_t_c; +}; + +#define to_police(pc) ((struct tcf_police *)pc) + +/* old policer structure from before tc actions */ +struct tc_police_compat { + u32 index; + int action; + u32 limit; + u32 burst; + u32 mtu; + struct tc_ratespec rate; + struct tc_ratespec peakrate; +}; + +static inline bool is_tcf_police(const struct tc_action *act) +{ +#ifdef CONFIG_NET_CLS_ACT + if (act->ops && act->ops->id == TCA_ID_POLICE) + return true; +#endif + return false; +} + +static inline u64 tcf_police_rate_bytes_ps(const struct tc_action *act) +{ + struct tcf_police *police = to_police(act); + struct tcf_police_params *params; + + params = rcu_dereference_bh(police->params); + return params->rate.rate_bytes_ps; +} + +static inline s64 tcf_police_tcfp_burst(const struct tc_action *act) +{ + struct tcf_police *police = to_police(act); + struct tcf_police_params *params; + + params = rcu_dereference_bh(police->params); + return params->tcfp_burst; +} + +#endif /* __NET_TC_POLICE_H */ diff --git a/include/net/tcp.h b/include/net/tcp.h index 68ee02523b873db037aa9d6707edb14592e763fd..7cf1181630a3a1fb36501a25bff8efb67c1184e5 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -1314,7 +1314,7 @@ static inline void tcp_update_wl(struct tcp_sock *tp, u32 seq) static inline __sum16 tcp_v4_check(int len, __be32 saddr, __be32 daddr, __wsum base) { - return csum_tcpudp_magic(saddr,daddr,len,IPPROTO_TCP,base); + return csum_tcpudp_magic(saddr, daddr, len, IPPROTO_TCP, base); } static inline bool tcp_checksum_complete(struct sk_buff *skb) diff --git a/include/net/tls.h b/include/net/tls.h index 5934246b2c6f4bafbe318fdddfacb328a2b9bf5c..39ea62f0c1f6aef57a3cef401b55959a5ac30f85 100644 --- a/include/net/tls.h +++ b/include/net/tls.h @@ -60,6 +60,17 @@ #define TLS_AAD_SPACE_SIZE 13 #define TLS_DEVICE_NAME_MAX 32 +#define MAX_IV_SIZE 16 + +/* For AES-CCM, the full 16-bytes of IV is made of '4' fields of given sizes. + * + * IV[16] = b0[1] || implicit nonce[4] || explicit nonce[8] || length[3] + * + * The field 'length' is encoded in field 'b0' as '(length width - 1)'. + * Hence b0 contains (3 - 1) = 2. + */ +#define TLS_AES_CCM_IV_B0_BYTE 2 + /* * This structure defines the routines for Inline TLS driver. * The following routines are optional and filled with a @@ -123,8 +134,7 @@ struct tls_rec { struct scatterlist sg_content_type; char aad_space[TLS_AAD_SPACE_SIZE]; - u8 iv_data[TLS_CIPHER_AES_GCM_128_IV_SIZE + - TLS_CIPHER_AES_GCM_128_SALT_SIZE]; + u8 iv_data[MAX_IV_SIZE]; struct aead_request aead_req; u8 aead_req_ctx[]; }; @@ -219,6 +229,7 @@ struct tls_prot_info { u16 tag_size; u16 overhead_size; u16 iv_size; + u16 salt_size; u16 rec_seq_size; u16 aad_size; u16 tail_size; @@ -266,6 +277,23 @@ struct tls_context { void (*unhash)(struct sock *sk); }; +enum tls_offload_ctx_dir { + TLS_OFFLOAD_CTX_DIR_RX, + TLS_OFFLOAD_CTX_DIR_TX, +}; + +struct tlsdev_ops { + int (*tls_dev_add)(struct net_device *netdev, struct sock *sk, + enum tls_offload_ctx_dir direction, + struct tls_crypto_info *crypto_info, + u32 start_offload_tcp_sn); + void (*tls_dev_del)(struct net_device *netdev, + struct tls_context *ctx, + enum tls_offload_ctx_dir direction); + void (*tls_dev_resync_rx)(struct net_device *netdev, + struct sock *sk, u32 seq, u64 rcd_sn); +}; + struct tls_offload_context_rx { /* sw must be the first member of tls_offload_context_rx */ struct tls_sw_context_rx sw; @@ -306,7 +334,6 @@ int tls_set_device_offload(struct sock *sk, struct tls_context *ctx); int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size); int tls_device_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags); -void tls_device_sk_destruct(struct sock *sk); void tls_device_free_resources_tx(struct sock *sk); void tls_device_init(void); void tls_device_cleanup(void); @@ -325,7 +352,6 @@ static inline u32 tls_record_start_seq(struct tls_record_info *rec) return rec->end_seq - rec->len; } -void tls_sk_destruct(struct sock *sk, struct tls_context *ctx); int tls_push_sg(struct sock *sk, struct tls_context *ctx, struct scatterlist *sg, u16 first_offset, int flags); @@ -536,7 +562,7 @@ static inline void tls_offload_rx_resync_request(struct sock *sk, __be32 seq) struct tls_context *tls_ctx = tls_get_ctx(sk); struct tls_offload_context_rx *rx_ctx = tls_offload_ctx_rx(tls_ctx); - atomic64_set(&rx_ctx->resync_req, ((((uint64_t)seq) << 32) | 1)); + atomic64_set(&rx_ctx->resync_req, ((u64)ntohl(seq) << 32) | 1); } diff --git a/include/net/udp.h b/include/net/udp.h index fd6d948755c81a5a48838bc4439a778e87542304..d8ce937bc3952b646e2db0bbdcfdf699cc5f7aa1 100644 --- a/include/net/udp.h +++ b/include/net/udp.h @@ -269,13 +269,13 @@ void skb_consume_udp(struct sock *sk, struct sk_buff *skb, int len); int __udp_enqueue_schedule_skb(struct sock *sk, struct sk_buff *skb); void udp_skb_destructor(struct sock *sk, struct sk_buff *skb); struct sk_buff *__skb_recv_udp(struct sock *sk, unsigned int flags, - int noblock, int *peeked, int *off, int *err); + int noblock, int *off, int *err); static inline struct sk_buff *skb_recv_udp(struct sock *sk, unsigned int flags, int noblock, int *err) { - int peeked, off = 0; + int off = 0; - return __skb_recv_udp(sk, flags, noblock, &peeked, &off, err); + return __skb_recv_udp(sk, flags, noblock, &off, err); } int udp_v4_early_demux(struct sk_buff *skb); diff --git a/include/net/udp_tunnel.h b/include/net/udp_tunnel.h index b8137953fea31a377b3f5cc4aae99db932ad1ed4..4b1f95e083070ae6b4ff81eb1b5fd2433a93e282 100644 --- a/include/net/udp_tunnel.h +++ b/include/net/udp_tunnel.h @@ -7,7 +7,7 @@ #if IS_ENABLED(CONFIG_IPV6) #include -#include +#include #endif struct udp_port_cfg { diff --git a/include/net/vxlan.h b/include/net/vxlan.h index 00254a58824bd0abcae412388db307bbe8c5e8d5..83b5999a25876464fe8e000f3a5aa980c961ed9e 100644 --- a/include/net/vxlan.h +++ b/include/net/vxlan.h @@ -8,6 +8,8 @@ #include #include +#define IANA_VXLAN_UDP_PORT 4789 + /* VXLAN protocol (RFC 7348) header: * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * |R|R|R|R|I|R|R|R| Reserved | diff --git a/include/net/xfrm.h b/include/net/xfrm.h index 99f722c4d8044dca73f13f3a27376a20cda6181f..a2907873ed561bfdcf2d8504a4b69f3f7e8d7921 100644 --- a/include/net/xfrm.h +++ b/include/net/xfrm.h @@ -132,6 +132,17 @@ struct xfrm_state_offload { u8 flags; }; +struct xfrm_mode { + u8 encap; + u8 family; + u8 flags; +}; + +/* Flags for xfrm_mode. */ +enum { + XFRM_MODE_FLAG_TUNNEL = 1, +}; + /* Full description of state of transformer. */ struct xfrm_state { possible_net_t xs_net; @@ -234,9 +245,9 @@ struct xfrm_state { /* Reference to data common to all the instances of this * transformer. */ const struct xfrm_type *type; - struct xfrm_mode *inner_mode; - struct xfrm_mode *inner_mode_iaf; - struct xfrm_mode *outer_mode; + struct xfrm_mode inner_mode; + struct xfrm_mode inner_mode_iaf; + struct xfrm_mode outer_mode; const struct xfrm_type_offload *type_offload; @@ -316,13 +327,6 @@ struct xfrm_policy_afinfo { xfrm_address_t *saddr, xfrm_address_t *daddr, u32 mark); - void (*decode_session)(struct sk_buff *skb, - struct flowi *fl, - int reverse); - int (*get_tos)(const struct flowi *fl); - int (*init_path)(struct xfrm_dst *path, - struct dst_entry *dst, - int nfheader_len); int (*fill_dst)(struct xfrm_dst *xdst, struct net_device *dev, const struct flowi *fl); @@ -348,7 +352,6 @@ struct xfrm_state_afinfo { struct module *owner; const struct xfrm_type *type_map[IPPROTO_MAX]; const struct xfrm_type_offload *type_offload_map[IPPROTO_MAX]; - struct xfrm_mode *mode_map[XFRM_MODE_MAX]; int (*init_flags)(struct xfrm_state *x); void (*init_tempsel)(struct xfrm_selector *sel, @@ -423,78 +426,6 @@ struct xfrm_type_offload { int xfrm_register_type_offload(const struct xfrm_type_offload *type, unsigned short family); int xfrm_unregister_type_offload(const struct xfrm_type_offload *type, unsigned short family); -struct xfrm_mode { - /* - * Remove encapsulation header. - * - * The IP header will be moved over the top of the encapsulation - * header. - * - * On entry, the transport header shall point to where the IP header - * should be and the network header shall be set to where the IP - * header currently is. skb->data shall point to the start of the - * payload. - */ - int (*input2)(struct xfrm_state *x, struct sk_buff *skb); - - /* - * This is the actual input entry point. - * - * For transport mode and equivalent this would be identical to - * input2 (which does not need to be set). While tunnel mode - * and equivalent would set this to the tunnel encapsulation function - * xfrm4_prepare_input that would in turn call input2. - */ - int (*input)(struct xfrm_state *x, struct sk_buff *skb); - - /* - * Add encapsulation header. - * - * On exit, the transport header will be set to the start of the - * encapsulation header to be filled in by x->type->output and - * the mac header will be set to the nextheader (protocol for - * IPv4) field of the extension header directly preceding the - * encapsulation header, or in its absence, that of the top IP - * header. The value of the network header will always point - * to the top IP header while skb->data will point to the payload. - */ - int (*output2)(struct xfrm_state *x,struct sk_buff *skb); - - /* - * This is the actual output entry point. - * - * For transport mode and equivalent this would be identical to - * output2 (which does not need to be set). While tunnel mode - * and equivalent would set this to a tunnel encapsulation function - * (xfrm4_prepare_output or xfrm6_prepare_output) that would in turn - * call output2. - */ - int (*output)(struct xfrm_state *x, struct sk_buff *skb); - - /* - * Adjust pointers into the packet and do GSO segmentation. - */ - struct sk_buff *(*gso_segment)(struct xfrm_state *x, struct sk_buff *skb, netdev_features_t features); - - /* - * Adjust pointers into the packet when IPsec is done at layer2. - */ - void (*xmit)(struct xfrm_state *x, struct sk_buff *skb); - - struct xfrm_state_afinfo *afinfo; - struct module *owner; - unsigned int encap; - int flags; -}; - -/* Flags for xfrm_mode. */ -enum { - XFRM_MODE_FLAG_TUNNEL = 1, -}; - -int xfrm_register_mode(struct xfrm_mode *mode, int family); -int xfrm_unregister_mode(struct xfrm_mode *mode, int family); - static inline int xfrm_af2proto(unsigned int family) { switch(family) { @@ -507,13 +438,13 @@ static inline int xfrm_af2proto(unsigned int family) } } -static inline struct xfrm_mode *xfrm_ip2inner_mode(struct xfrm_state *x, int ipproto) +static inline const struct xfrm_mode *xfrm_ip2inner_mode(struct xfrm_state *x, int ipproto) { if ((ipproto == IPPROTO_IPIP && x->props.family == AF_INET) || (ipproto == IPPROTO_IPV6 && x->props.family == AF_INET6)) - return x->inner_mode; + return &x->inner_mode; else - return x->inner_mode_iaf; + return &x->inner_mode_iaf; } struct xfrm_tmpl { @@ -1623,7 +1554,6 @@ int xfrm_init_replay(struct xfrm_state *x); int xfrm_state_mtu(struct xfrm_state *x, int mtu); int __xfrm_init_state(struct xfrm_state *x, bool init_replay, bool offload); int xfrm_init_state(struct xfrm_state *x); -int xfrm_prepare_input(struct xfrm_state *x, struct sk_buff *skb); int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type); int xfrm_input_resume(struct sk_buff *skb, int nexthdr); int xfrm_trans_queue(struct sk_buff *skb, @@ -1631,7 +1561,11 @@ int xfrm_trans_queue(struct sk_buff *skb, struct sk_buff *)); int xfrm_output_resume(struct sk_buff *skb, int err); int xfrm_output(struct sock *sk, struct sk_buff *skb); -int xfrm_inner_extract_output(struct xfrm_state *x, struct sk_buff *skb); + +#if IS_ENABLED(CONFIG_NET_PKTGEN) +int pktgen_xfrm_outer_mode_output(struct xfrm_state *x, struct sk_buff *skb); +#endif + void xfrm_local_error(struct sk_buff *skb, int mtu); int xfrm4_extract_header(struct sk_buff *skb); int xfrm4_extract_input(struct xfrm_state *x, struct sk_buff *skb); @@ -1650,10 +1584,8 @@ static inline int xfrm4_rcv_spi(struct sk_buff *skb, int nexthdr, __be32 spi) } int xfrm4_extract_output(struct xfrm_state *x, struct sk_buff *skb); -int xfrm4_prepare_output(struct xfrm_state *x, struct sk_buff *skb); int xfrm4_output(struct net *net, struct sock *sk, struct sk_buff *skb); int xfrm4_output_finish(struct sock *sk, struct sk_buff *skb); -int xfrm4_rcv_cb(struct sk_buff *skb, u8 protocol, int err); int xfrm4_protocol_register(struct xfrm4_protocol *handler, unsigned char protocol); int xfrm4_protocol_deregister(struct xfrm4_protocol *handler, unsigned char protocol); int xfrm4_tunnel_register(struct xfrm_tunnel *handler, unsigned short family); @@ -1669,7 +1601,6 @@ int xfrm6_rcv(struct sk_buff *skb); int xfrm6_input_addr(struct sk_buff *skb, xfrm_address_t *daddr, xfrm_address_t *saddr, u8 proto); void xfrm6_local_error(struct sk_buff *skb, u32 mtu); -int xfrm6_rcv_cb(struct sk_buff *skb, u8 protocol, int err); int xfrm6_protocol_register(struct xfrm6_protocol *handler, unsigned char protocol); int xfrm6_protocol_deregister(struct xfrm6_protocol *handler, unsigned char protocol); int xfrm6_tunnel_register(struct xfrm6_tunnel *handler, unsigned short family); @@ -1677,7 +1608,6 @@ int xfrm6_tunnel_deregister(struct xfrm6_tunnel *handler, unsigned short family) __be32 xfrm6_tunnel_alloc_spi(struct net *net, xfrm_address_t *saddr); __be32 xfrm6_tunnel_spi_lookup(struct net *net, const xfrm_address_t *saddr); int xfrm6_extract_output(struct xfrm_state *x, struct sk_buff *skb); -int xfrm6_prepare_output(struct xfrm_state *x, struct sk_buff *skb); int xfrm6_output(struct net *net, struct sock *sk, struct sk_buff *skb); int xfrm6_output_finish(struct sock *sk, struct sk_buff *skb); int xfrm6_find_1stfragopt(struct xfrm_state *x, struct sk_buff *skb, @@ -2069,7 +1999,7 @@ static inline int xfrm_tunnel_check(struct sk_buff *skb, struct xfrm_state *x, tunnel = true; break; } - if (tunnel && !(x->outer_mode->flags & XFRM_MODE_FLAG_TUNNEL)) + if (tunnel && !(x->outer_mode.flags & XFRM_MODE_FLAG_TUNNEL)) return -EINVAL; return 0; diff --git a/include/trace/bpf_probe.h b/include/trace/bpf_probe.h index 505dae0bed80bd83373811c60ce4ac60d54ee8af..d6e556c0a0852df0c357361dfbb642bdbf3e58bf 100644 --- a/include/trace/bpf_probe.h +++ b/include/trace/bpf_probe.h @@ -69,8 +69,7 @@ __bpf_trace_##call(void *__data, proto) \ * to make sure that if the tracepoint handling changes, the * bpf probe will fail to compile unless it too is updated. */ -#undef DEFINE_EVENT -#define DEFINE_EVENT(template, call, proto, args) \ +#define __DEFINE_EVENT(template, call, proto, args, size) \ static inline void bpf_test_probe_##call(void) \ { \ check_trace_callback_type_##call(__bpf_trace_##template); \ @@ -81,12 +80,36 @@ __bpf_trace_tp_map_##call = { \ .tp = &__tracepoint_##call, \ .bpf_func = (void *)__bpf_trace_##template, \ .num_args = COUNT_ARGS(args), \ + .writable_size = size, \ }; +#define FIRST(x, ...) x + +#undef DEFINE_EVENT_WRITABLE +#define DEFINE_EVENT_WRITABLE(template, call, proto, args, size) \ +static inline void bpf_test_buffer_##call(void) \ +{ \ + /* BUILD_BUG_ON() is ignored if the code is completely eliminated, but \ + * BUILD_BUG_ON_ZERO() uses a different mechanism that is not \ + * dead-code-eliminated. \ + */ \ + FIRST(proto); \ + (void)BUILD_BUG_ON_ZERO(size != sizeof(*FIRST(args))); \ +} \ +__DEFINE_EVENT(template, call, PARAMS(proto), PARAMS(args), size) + +#undef DEFINE_EVENT +#define DEFINE_EVENT(template, call, proto, args) \ + __DEFINE_EVENT(template, call, PARAMS(proto), PARAMS(args), 0) #undef DEFINE_EVENT_PRINT #define DEFINE_EVENT_PRINT(template, name, proto, args, print) \ DEFINE_EVENT(template, name, PARAMS(proto), PARAMS(args)) #include TRACE_INCLUDE(TRACE_INCLUDE_FILE) + +#undef DEFINE_EVENT_WRITABLE +#undef __DEFINE_EVENT +#undef FIRST + #endif /* CONFIG_BPF_EVENTS */ diff --git a/include/trace/events/bpf_test_run.h b/include/trace/events/bpf_test_run.h new file mode 100644 index 0000000000000000000000000000000000000000..265447e3f71abeea68de239f722c6428fe5b8289 --- /dev/null +++ b/include/trace/events/bpf_test_run.h @@ -0,0 +1,50 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM bpf_test_run + +#if !defined(_TRACE_BPF_TEST_RUN_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_BPF_TEST_RUN_H + +#include + +DECLARE_EVENT_CLASS(bpf_test_finish, + + TP_PROTO(int *err), + + TP_ARGS(err), + + TP_STRUCT__entry( + __field(int, err) + ), + + TP_fast_assign( + __entry->err = *err; + ), + + TP_printk("bpf_test_finish with err=%d", __entry->err) +); + +#ifdef DEFINE_EVENT_WRITABLE +#undef BPF_TEST_RUN_DEFINE_EVENT +#define BPF_TEST_RUN_DEFINE_EVENT(template, call, proto, args, size) \ + DEFINE_EVENT_WRITABLE(template, call, PARAMS(proto), \ + PARAMS(args), size) +#else +#undef BPF_TEST_RUN_DEFINE_EVENT +#define BPF_TEST_RUN_DEFINE_EVENT(template, call, proto, args, size) \ + DEFINE_EVENT(template, call, PARAMS(proto), PARAMS(args)) +#endif + +BPF_TEST_RUN_DEFINE_EVENT(bpf_test_finish, bpf_test_finish, + + TP_PROTO(int *err), + + TP_ARGS(err), + + sizeof(int) +); + +#endif + +/* This part must be outside protection */ +#include diff --git a/include/trace/events/fib.h b/include/trace/events/fib.h index 6271bab63bfb99558ca2895a16c5f3061a9f5b75..6f2a4dc35e37eae4d302055d195ac2fc2b67089b 100644 --- a/include/trace/events/fib.h +++ b/include/trace/events/fib.h @@ -13,9 +13,9 @@ TRACE_EVENT(fib_table_lookup, TP_PROTO(u32 tb_id, const struct flowi4 *flp, - const struct fib_nh *nh, int err), + const struct fib_nh_common *nhc, int err), - TP_ARGS(tb_id, flp, nh, err), + TP_ARGS(tb_id, flp, nhc, err), TP_STRUCT__entry( __field( u32, tb_id ) @@ -28,14 +28,17 @@ TRACE_EVENT(fib_table_lookup, __field( __u8, flags ) __array( __u8, src, 4 ) __array( __u8, dst, 4 ) - __array( __u8, gw, 4 ) - __array( __u8, saddr, 4 ) + __array( __u8, gw4, 4 ) + __array( __u8, gw6, 16 ) __field( u16, sport ) __field( u16, dport ) __dynamic_array(char, name, IFNAMSIZ ) ), TP_fast_assign( + struct in6_addr in6_zero = {}; + struct net_device *dev; + struct in6_addr *in6; __be32 *p32; __entry->tb_id = tb_id; @@ -62,30 +65,37 @@ TRACE_EVENT(fib_table_lookup, __entry->dport = 0; } - if (nh) { - p32 = (__be32 *) __entry->saddr; - *p32 = nh->nh_saddr; + dev = nhc ? nhc->nhc_dev : NULL; + __assign_str(name, dev ? dev->name : "-"); - p32 = (__be32 *) __entry->gw; - *p32 = nh->nh_gw; + if (nhc) { + if (nhc->nhc_gw_family == AF_INET) { + p32 = (__be32 *) __entry->gw4; + *p32 = nhc->nhc_gw.ipv4; - __assign_str(name, nh->nh_dev ? nh->nh_dev->name : "-"); - } else { - p32 = (__be32 *) __entry->saddr; - *p32 = 0; + in6 = (struct in6_addr *)__entry->gw6; + *in6 = in6_zero; + } else if (nhc->nhc_gw_family == AF_INET6) { + p32 = (__be32 *) __entry->gw4; + *p32 = 0; - p32 = (__be32 *) __entry->gw; + in6 = (struct in6_addr *)__entry->gw6; + *in6 = nhc->nhc_gw.ipv6; + } + } else { + p32 = (__be32 *) __entry->gw4; *p32 = 0; - __assign_str(name, "-"); + in6 = (struct in6_addr *)__entry->gw6; + *in6 = in6_zero; } ), - TP_printk("table %u oif %d iif %d proto %u %pI4/%u -> %pI4/%u tos %d scope %d flags %x ==> dev %s gw %pI4 src %pI4 err %d", + TP_printk("table %u oif %d iif %d proto %u %pI4/%u -> %pI4/%u tos %d scope %d flags %x ==> dev %s gw %pI4/%pI6c err %d", __entry->tb_id, __entry->oif, __entry->iif, __entry->proto, __entry->src, __entry->sport, __entry->dst, __entry->dport, __entry->tos, __entry->scope, __entry->flags, - __get_str(name), __entry->gw, __entry->saddr, __entry->err) + __get_str(name), __entry->gw4, __entry->gw6, __entry->err) ); #endif /* _TRACE_FIB_H */ diff --git a/include/trace/events/fib6.h b/include/trace/events/fib6.h index b088b54d699cedad66577fcadd34e7778032639f..c6abdcc77c1276df9a947af435295b1d4e9cd766 100644 --- a/include/trace/events/fib6.h +++ b/include/trace/events/fib6.h @@ -12,10 +12,10 @@ TRACE_EVENT(fib6_table_lookup, - TP_PROTO(const struct net *net, const struct fib6_info *f6i, + TP_PROTO(const struct net *net, const struct fib6_result *res, struct fib6_table *table, const struct flowi6 *flp), - TP_ARGS(net, f6i, table, flp), + TP_ARGS(net, res, table, flp), TP_STRUCT__entry( __field( u32, tb_id ) @@ -39,7 +39,7 @@ TRACE_EVENT(fib6_table_lookup, struct in6_addr *in6; __entry->tb_id = table->tb6_id; - __entry->err = ip6_rt_type_to_error(f6i->fib6_type); + __entry->err = ip6_rt_type_to_error(res->fib6_type); __entry->oif = flp->flowi6_oif; __entry->iif = flp->flowi6_iif; __entry->tos = ip6_tclass(flp->flowlabel); @@ -62,20 +62,20 @@ TRACE_EVENT(fib6_table_lookup, __entry->dport = 0; } - if (f6i->fib6_nh.nh_dev) { - __assign_str(name, f6i->fib6_nh.nh_dev); + if (res->nh && res->nh->fib_nh_dev) { + __assign_str(name, res->nh->fib_nh_dev); } else { __assign_str(name, "-"); } - if (f6i == net->ipv6.fib6_null_entry) { + if (res->f6i == net->ipv6.fib6_null_entry) { struct in6_addr in6_zero = {}; in6 = (struct in6_addr *)__entry->gw; *in6 = in6_zero; - } else if (f6i) { + } else if (res->nh) { in6 = (struct in6_addr *)__entry->gw; - *in6 = f6i->fib6_nh.nh_gw; + *in6 = res->nh->fib_nh_gw6; } ), diff --git a/include/trace/events/mlxsw.h b/include/trace/events/mlxsw.h index 6a4cfaef33a2122d00dedb10447959d7fcc640be..19a25ed323a506f92c9867bcf259f5c74a2b1e27 100644 --- a/include/trace/events/mlxsw.h +++ b/include/trace/events/mlxsw.h @@ -93,7 +93,7 @@ TRACE_EVENT(mlxsw_sp_acl_tcam_vregion_migrate_end, __entry->mlxsw_sp, __entry->vregion) ); -TRACE_EVENT(mlxsw_sp_acl_tcam_vregion_rehash_dis, +TRACE_EVENT(mlxsw_sp_acl_tcam_vregion_rehash_rollback_failed, TP_PROTO(const struct mlxsw_sp *mlxsw_sp, const struct mlxsw_sp_acl_tcam_vregion *vregion), diff --git a/include/trace/events/nbd.h b/include/trace/events/nbd.h new file mode 100644 index 0000000000000000000000000000000000000000..9849956f34d85fbb264cf64e99131c4885d7b99c --- /dev/null +++ b/include/trace/events/nbd.h @@ -0,0 +1,107 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM nbd + +#if !defined(_TRACE_NBD_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_NBD_H + +#include + +DECLARE_EVENT_CLASS(nbd_transport_event, + + TP_PROTO(struct request *req, u64 handle), + + TP_ARGS(req, handle), + + TP_STRUCT__entry( + __field(struct request *, req) + __field(u64, handle) + ), + + TP_fast_assign( + __entry->req = req; + __entry->handle = handle; + ), + + TP_printk( + "nbd transport event: request %p, handle 0x%016llx", + __entry->req, + __entry->handle + ) +); + +DEFINE_EVENT(nbd_transport_event, nbd_header_sent, + + TP_PROTO(struct request *req, u64 handle), + + TP_ARGS(req, handle) +); + +DEFINE_EVENT(nbd_transport_event, nbd_payload_sent, + + TP_PROTO(struct request *req, u64 handle), + + TP_ARGS(req, handle) +); + +DEFINE_EVENT(nbd_transport_event, nbd_header_received, + + TP_PROTO(struct request *req, u64 handle), + + TP_ARGS(req, handle) +); + +DEFINE_EVENT(nbd_transport_event, nbd_payload_received, + + TP_PROTO(struct request *req, u64 handle), + + TP_ARGS(req, handle) +); + +DECLARE_EVENT_CLASS(nbd_send_request, + + TP_PROTO(struct nbd_request *nbd_request, int index, + struct request *rq), + + TP_ARGS(nbd_request, index, rq), + + TP_STRUCT__entry( + __field(struct nbd_request *, nbd_request) + __field(u64, dev_index) + __field(struct request *, request) + ), + + TP_fast_assign( + __entry->nbd_request = 0; + __entry->dev_index = index; + __entry->request = rq; + ), + + TP_printk("nbd%lld: request %p", __entry->dev_index, __entry->request) +); + +#ifdef DEFINE_EVENT_WRITABLE +#undef NBD_DEFINE_EVENT +#define NBD_DEFINE_EVENT(template, call, proto, args, size) \ + DEFINE_EVENT_WRITABLE(template, call, PARAMS(proto), \ + PARAMS(args), size) +#else +#undef NBD_DEFINE_EVENT +#define NBD_DEFINE_EVENT(template, call, proto, args, size) \ + DEFINE_EVENT(template, call, PARAMS(proto), PARAMS(args)) +#endif + +NBD_DEFINE_EVENT(nbd_send_request, nbd_send_request, + + TP_PROTO(struct nbd_request *nbd_request, int index, + struct request *rq), + + TP_ARGS(nbd_request, index, rq), + + sizeof(struct nbd_request) +); + +#endif + +/* This part must be outside protection */ +#include diff --git a/include/trace/events/net.h b/include/trace/events/net.h index 1efd7d9b25fecdb27ec98e61367b1954d4a72c78..2399073c3afc603bd303960302cf42f5bb38044a 100644 --- a/include/trace/events/net.h +++ b/include/trace/events/net.h @@ -95,6 +95,29 @@ TRACE_EVENT(net_dev_xmit, __get_str(name), __entry->skbaddr, __entry->len, __entry->rc) ); +TRACE_EVENT(net_dev_xmit_timeout, + + TP_PROTO(struct net_device *dev, + int queue_index), + + TP_ARGS(dev, queue_index), + + TP_STRUCT__entry( + __string( name, dev->name ) + __string( driver, netdev_drivername(dev)) + __field( int, queue_index ) + ), + + TP_fast_assign( + __assign_str(name, dev->name); + __assign_str(driver, netdev_drivername(dev)); + __entry->queue_index = queue_index; + ), + + TP_printk("dev=%s driver=%s queue=%d", + __get_str(name), __get_str(driver), __entry->queue_index) +); + DECLARE_EVENT_CLASS(net_dev_template, TP_PROTO(struct sk_buff *skb), diff --git a/include/uapi/asm-generic/sockios.h b/include/uapi/asm-generic/sockios.h index 64f658c7cec2a90710394ca8b827bfe257bebc2e..44fa3ed7048322bf072e4ed21945f5ed9415e318 100644 --- a/include/uapi/asm-generic/sockios.h +++ b/include/uapi/asm-generic/sockios.h @@ -8,7 +8,7 @@ #define FIOGETOWN 0x8903 #define SIOCGPGRP 0x8904 #define SIOCATMARK 0x8905 -#define SIOCGSTAMP 0x8906 /* Get stamp (timeval) */ -#define SIOCGSTAMPNS 0x8907 /* Get stamp (timespec) */ +#define SIOCGSTAMP_OLD 0x8906 /* Get stamp (timeval) */ +#define SIOCGSTAMPNS_OLD 0x8907 /* Get stamp (timespec) */ #endif /* __ASM_GENERIC_SOCKIOS_H */ diff --git a/include/uapi/linux/batadv_packet.h b/include/uapi/linux/batadv_packet.h index c99336f4eefe9e3f06b1e7838d4a3a462daf2d0b..4ebc2135e950fc5ec7057d247ee922a625a62903 100644 --- a/include/uapi/linux/batadv_packet.h +++ b/include/uapi/linux/batadv_packet.h @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _UAPI_LINUX_BATADV_PACKET_H_ diff --git a/include/uapi/linux/batman_adv.h b/include/uapi/linux/batman_adv.h index 305bf316dd03ba0fc335930706c58319d0bd6a3e..67f4636758af9bf6f667a1d2f7dcdf42e71170b1 100644 --- a/include/uapi/linux/batman_adv.h +++ b/include/uapi/linux/batman_adv.h @@ -2,24 +2,6 @@ /* Copyright (C) 2016-2019 B.A.T.M.A.N. contributors: * * Matthias Schiffer - * - * Permission is hereby granted, free of charge, to any person obtaining a - * copy of this software and associated documentation files (the "Software"), - * to deal in the Software without restriction, including without limitation - * the rights to use, copy, modify, merge, publish, distribute, sublicense, - * and/or sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following conditions: - * - * The above copyright notice and this permission notice shall be included in - * all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR - * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, - * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE - * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER - * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING - * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER - * DEALINGS IN THE SOFTWARE. */ #ifndef _UAPI_LINUX_BATMAN_ADV_H_ @@ -491,6 +473,13 @@ enum batadv_nl_attrs { */ BATADV_ATTR_THROUGHPUT_OVERRIDE, + /** + * @BATADV_ATTR_MULTICAST_FANOUT: defines the maximum number of packet + * copies that may be generated for a multicast-to-unicast conversion. + * Once this limit is exceeded distribution will fall back to broadcast. + */ + BATADV_ATTR_MULTICAST_FANOUT, + /* add attributes above here, update the policy in netlink.c */ /** diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 929c8e537a14a517c0a3c7ca5b6b15353d622c30..72336bac7573368e7bce974626b68569c46eaabb 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -105,6 +105,7 @@ enum bpf_cmd { BPF_BTF_GET_FD_BY_ID, BPF_TASK_FD_QUERY, BPF_MAP_LOOKUP_AND_DELETE_ELEM, + BPF_MAP_FREEZE, }; enum bpf_map_type { @@ -132,6 +133,7 @@ enum bpf_map_type { BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE, BPF_MAP_TYPE_QUEUE, BPF_MAP_TYPE_STACK, + BPF_MAP_TYPE_SK_STORAGE, }; /* Note that tracing related programs such as @@ -166,6 +168,8 @@ enum bpf_prog_type { BPF_PROG_TYPE_LIRC_MODE2, BPF_PROG_TYPE_SK_REUSEPORT, BPF_PROG_TYPE_FLOW_DISSECTOR, + BPF_PROG_TYPE_CGROUP_SYSCTL, + BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE, }; enum bpf_attach_type { @@ -187,6 +191,7 @@ enum bpf_attach_type { BPF_CGROUP_UDP6_SENDMSG, BPF_LIRC_MODE2, BPF_FLOW_DISSECTOR, + BPF_CGROUP_SYSCTL, __MAX_BPF_ATTACH_TYPE }; @@ -255,8 +260,19 @@ enum bpf_attach_type { */ #define BPF_F_ANY_ALIGNMENT (1U << 1) -/* when bpf_ldimm64->src_reg == BPF_PSEUDO_MAP_FD, bpf_ldimm64->imm == fd */ +/* When BPF ldimm64's insn[0].src_reg != 0 then this can have + * two extensions: + * + * insn[0].src_reg: BPF_PSEUDO_MAP_FD BPF_PSEUDO_MAP_VALUE + * insn[0].imm: map fd map fd + * insn[1].imm: 0 offset into value + * insn[0].off: 0 0 + * insn[1].off: 0 0 + * ldimm64 rewrite: address of map address of map[0]+offset + * verifier type: CONST_PTR_TO_MAP PTR_TO_MAP_VALUE + */ #define BPF_PSEUDO_MAP_FD 1 +#define BPF_PSEUDO_MAP_VALUE 2 /* when bpf_call->src_reg == BPF_PSEUDO_CALL, bpf_call->imm == pc-relative * offset to another bpf function @@ -283,7 +299,7 @@ enum bpf_attach_type { #define BPF_OBJ_NAME_LEN 16U -/* Flags for accessing BPF object */ +/* Flags for accessing BPF object from syscall side. */ #define BPF_F_RDONLY (1U << 3) #define BPF_F_WRONLY (1U << 4) @@ -293,6 +309,10 @@ enum bpf_attach_type { /* Zero-initialize hash function seed. This should only be used for testing. */ #define BPF_F_ZERO_SEED (1U << 6) +/* Flags for accessing BPF object from program side. */ +#define BPF_F_RDONLY_PROG (1U << 7) +#define BPF_F_WRONLY_PROG (1U << 8) + /* flags for BPF_PROG_QUERY */ #define BPF_F_QUERY_EFFECTIVE (1U << 0) @@ -396,6 +416,13 @@ union bpf_attr { __aligned_u64 data_out; __u32 repeat; __u32 duration; + __u32 ctx_size_in; /* input: len of ctx_in */ + __u32 ctx_size_out; /* input/output: len of ctx_out + * returns ENOSPC if ctx_out + * is too small. + */ + __aligned_u64 ctx_in; + __aligned_u64 ctx_out; } test; struct { /* anonymous struct used by BPF_*_GET_*_ID */ @@ -1478,13 +1505,31 @@ union bpf_attr { * Grow or shrink the room for data in the packet associated to * *skb* by *len_diff*, and according to the selected *mode*. * - * There is a single supported mode at this time: + * There are two supported modes at this time: + * + * * **BPF_ADJ_ROOM_MAC**: Adjust room at the mac layer + * (room space is added or removed below the layer 2 header). * * * **BPF_ADJ_ROOM_NET**: Adjust room at the network layer * (room space is added or removed below the layer 3 header). * - * All values for *flags* are reserved for future usage, and must - * be left at zero. + * The following flags are supported at this time: + * + * * **BPF_F_ADJ_ROOM_FIXED_GSO**: Do not adjust gso_size. + * Adjusting mss in this way is not allowed for datagrams. + * + * * **BPF_F_ADJ_ROOM_ENCAP_L3_IPV4 **: + * * **BPF_F_ADJ_ROOM_ENCAP_L3_IPV6 **: + * Any new space is reserved to hold a tunnel header. + * Configure skb offsets and other fields accordingly. + * + * * **BPF_F_ADJ_ROOM_ENCAP_L4_GRE **: + * * **BPF_F_ADJ_ROOM_ENCAP_L4_UDP **: + * Use with ENCAP_L3 flags to further specify the tunnel type. + * + * * **BPF_F_ADJ_ROOM_ENCAP_L2(len) **: + * Use with ENCAP_L3/L4 flags to further specify the tunnel + * type; **len** is the length of the inner MAC header. * * A call to this helper is susceptible to change the underlaying * packet buffer. Therefore, at load time, all checks on pointers @@ -1694,12 +1739,19 @@ union bpf_attr { * error if an eBPF program tries to set a callback that is not * supported in the current kernel. * - * The supported callback values that *argval* can combine are: + * *argval* is a flag array which can combine these flags: * * * **BPF_SOCK_OPS_RTO_CB_FLAG** (retransmission time out) * * **BPF_SOCK_OPS_RETRANS_CB_FLAG** (retransmission) * * **BPF_SOCK_OPS_STATE_CB_FLAG** (TCP state change) * + * Therefore, this function can be used to clear a callback flag by + * setting the appropriate bit to zero. e.g. to disable the RTO + * callback: + * + * **bpf_sock_ops_cb_flags_set(bpf_sock,** + * **bpf_sock->bpf_sock_ops_cb_flags & ~BPF_SOCK_OPS_RTO_CB_FLAG)** + * * Here are some examples of where one could call such eBPF * program: * @@ -2431,6 +2483,190 @@ union bpf_attr { * Return * A **struct bpf_sock** pointer on success, or **NULL** in * case of failure. + * + * struct bpf_sock *bpf_skc_lookup_tcp(void *ctx, struct bpf_sock_tuple *tuple, u32 tuple_size, u64 netns, u64 flags) + * Description + * Look for TCP socket matching *tuple*, optionally in a child + * network namespace *netns*. The return value must be checked, + * and if non-**NULL**, released via **bpf_sk_release**\ (). + * + * This function is identical to bpf_sk_lookup_tcp, except that it + * also returns timewait or request sockets. Use bpf_sk_fullsock + * or bpf_tcp_socket to access the full structure. + * + * This helper is available only if the kernel was compiled with + * **CONFIG_NET** configuration option. + * Return + * Pointer to **struct bpf_sock**, or **NULL** in case of failure. + * For sockets with reuseport option, the **struct bpf_sock** + * result is from **reuse->socks**\ [] using the hash of the tuple. + * + * int bpf_tcp_check_syncookie(struct bpf_sock *sk, void *iph, u32 iph_len, struct tcphdr *th, u32 th_len) + * Description + * Check whether iph and th contain a valid SYN cookie ACK for + * the listening socket in sk. + * + * iph points to the start of the IPv4 or IPv6 header, while + * iph_len contains sizeof(struct iphdr) or sizeof(struct ip6hdr). + * + * th points to the start of the TCP header, while th_len contains + * sizeof(struct tcphdr). + * + * Return + * 0 if iph and th are a valid SYN cookie ACK, or a negative error + * otherwise. + * + * int bpf_sysctl_get_name(struct bpf_sysctl *ctx, char *buf, size_t buf_len, u64 flags) + * Description + * Get name of sysctl in /proc/sys/ and copy it into provided by + * program buffer *buf* of size *buf_len*. + * + * The buffer is always NUL terminated, unless it's zero-sized. + * + * If *flags* is zero, full name (e.g. "net/ipv4/tcp_mem") is + * copied. Use **BPF_F_SYSCTL_BASE_NAME** flag to copy base name + * only (e.g. "tcp_mem"). + * Return + * Number of character copied (not including the trailing NUL). + * + * **-E2BIG** if the buffer wasn't big enough (*buf* will contain + * truncated name in this case). + * + * int bpf_sysctl_get_current_value(struct bpf_sysctl *ctx, char *buf, size_t buf_len) + * Description + * Get current value of sysctl as it is presented in /proc/sys + * (incl. newline, etc), and copy it as a string into provided + * by program buffer *buf* of size *buf_len*. + * + * The whole value is copied, no matter what file position user + * space issued e.g. sys_read at. + * + * The buffer is always NUL terminated, unless it's zero-sized. + * Return + * Number of character copied (not including the trailing NUL). + * + * **-E2BIG** if the buffer wasn't big enough (*buf* will contain + * truncated name in this case). + * + * **-EINVAL** if current value was unavailable, e.g. because + * sysctl is uninitialized and read returns -EIO for it. + * + * int bpf_sysctl_get_new_value(struct bpf_sysctl *ctx, char *buf, size_t buf_len) + * Description + * Get new value being written by user space to sysctl (before + * the actual write happens) and copy it as a string into + * provided by program buffer *buf* of size *buf_len*. + * + * User space may write new value at file position > 0. + * + * The buffer is always NUL terminated, unless it's zero-sized. + * Return + * Number of character copied (not including the trailing NUL). + * + * **-E2BIG** if the buffer wasn't big enough (*buf* will contain + * truncated name in this case). + * + * **-EINVAL** if sysctl is being read. + * + * int bpf_sysctl_set_new_value(struct bpf_sysctl *ctx, const char *buf, size_t buf_len) + * Description + * Override new value being written by user space to sysctl with + * value provided by program in buffer *buf* of size *buf_len*. + * + * *buf* should contain a string in same form as provided by user + * space on sysctl write. + * + * User space may write new value at file position > 0. To override + * the whole sysctl value file position should be set to zero. + * Return + * 0 on success. + * + * **-E2BIG** if the *buf_len* is too big. + * + * **-EINVAL** if sysctl is being read. + * + * int bpf_strtol(const char *buf, size_t buf_len, u64 flags, long *res) + * Description + * Convert the initial part of the string from buffer *buf* of + * size *buf_len* to a long integer according to the given base + * and save the result in *res*. + * + * The string may begin with an arbitrary amount of white space + * (as determined by isspace(3)) followed by a single optional '-' + * sign. + * + * Five least significant bits of *flags* encode base, other bits + * are currently unused. + * + * Base must be either 8, 10, 16 or 0 to detect it automatically + * similar to user space strtol(3). + * Return + * Number of characters consumed on success. Must be positive but + * no more than buf_len. + * + * **-EINVAL** if no valid digits were found or unsupported base + * was provided. + * + * **-ERANGE** if resulting value was out of range. + * + * int bpf_strtoul(const char *buf, size_t buf_len, u64 flags, unsigned long *res) + * Description + * Convert the initial part of the string from buffer *buf* of + * size *buf_len* to an unsigned long integer according to the + * given base and save the result in *res*. + * + * The string may begin with an arbitrary amount of white space + * (as determined by isspace(3)). + * + * Five least significant bits of *flags* encode base, other bits + * are currently unused. + * + * Base must be either 8, 10, 16 or 0 to detect it automatically + * similar to user space strtoul(3). + * Return + * Number of characters consumed on success. Must be positive but + * no more than buf_len. + * + * **-EINVAL** if no valid digits were found or unsupported base + * was provided. + * + * **-ERANGE** if resulting value was out of range. + * + * void *bpf_sk_storage_get(struct bpf_map *map, struct bpf_sock *sk, void *value, u64 flags) + * Description + * Get a bpf-local-storage from a sk. + * + * Logically, it could be thought of getting the value from + * a *map* with *sk* as the **key**. From this + * perspective, the usage is not much different from + * **bpf_map_lookup_elem(map, &sk)** except this + * helper enforces the key must be a **bpf_fullsock()** + * and the map must be a BPF_MAP_TYPE_SK_STORAGE also. + * + * Underneath, the value is stored locally at *sk* instead of + * the map. The *map* is used as the bpf-local-storage **type**. + * The bpf-local-storage **type** (i.e. the *map*) is searched + * against all bpf-local-storages residing at sk. + * + * An optional *flags* (BPF_SK_STORAGE_GET_F_CREATE) can be + * used such that a new bpf-local-storage will be + * created if one does not exist. *value* can be used + * together with BPF_SK_STORAGE_GET_F_CREATE to specify + * the initial value of a bpf-local-storage. If *value* is + * NULL, the new bpf-local-storage will be zero initialized. + * Return + * A bpf-local-storage pointer is returned on success. + * + * **NULL** if not found or there was an error in adding + * a new bpf-local-storage. + * + * int bpf_sk_storage_delete(struct bpf_map *map, struct bpf_sock *sk) + * Description + * Delete a bpf-local-storage from a sk. + * Return + * 0 on success. + * + * **-ENOENT** if the bpf-local-storage cannot be found. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -2531,7 +2767,17 @@ union bpf_attr { FN(sk_fullsock), \ FN(tcp_sock), \ FN(skb_ecn_set_ce), \ - FN(get_listener_sock), + FN(get_listener_sock), \ + FN(skc_lookup_tcp), \ + FN(tcp_check_syncookie), \ + FN(sysctl_get_name), \ + FN(sysctl_get_current_value), \ + FN(sysctl_get_new_value), \ + FN(sysctl_set_new_value), \ + FN(strtol), \ + FN(strtoul), \ + FN(sk_storage_get), \ + FN(sk_storage_delete), /* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call @@ -2590,9 +2836,30 @@ enum bpf_func_id { /* Current network namespace */ #define BPF_F_CURRENT_NETNS (-1L) +/* BPF_FUNC_skb_adjust_room flags. */ +#define BPF_F_ADJ_ROOM_FIXED_GSO (1ULL << 0) + +#define BPF_ADJ_ROOM_ENCAP_L2_MASK 0xff +#define BPF_ADJ_ROOM_ENCAP_L2_SHIFT 56 + +#define BPF_F_ADJ_ROOM_ENCAP_L3_IPV4 (1ULL << 1) +#define BPF_F_ADJ_ROOM_ENCAP_L3_IPV6 (1ULL << 2) +#define BPF_F_ADJ_ROOM_ENCAP_L4_GRE (1ULL << 3) +#define BPF_F_ADJ_ROOM_ENCAP_L4_UDP (1ULL << 4) +#define BPF_F_ADJ_ROOM_ENCAP_L2(len) (((__u64)len & \ + BPF_ADJ_ROOM_ENCAP_L2_MASK) \ + << BPF_ADJ_ROOM_ENCAP_L2_SHIFT) + +/* BPF_FUNC_sysctl_get_name flags. */ +#define BPF_F_SYSCTL_BASE_NAME (1ULL << 0) + +/* BPF_FUNC_sk_storage_get flags */ +#define BPF_SK_STORAGE_GET_F_CREATE (1ULL << 0) + /* Mode for BPF_FUNC_skb_adjust_room helper. */ enum bpf_adj_room_mode { BPF_ADJ_ROOM_NET, + BPF_ADJ_ROOM_MAC, }; /* Mode for BPF_FUNC_skb_load_bytes_relative helper. */ @@ -3218,4 +3485,14 @@ struct bpf_line_info { struct bpf_spin_lock { __u32 val; }; + +struct bpf_sysctl { + __u32 write; /* Sysctl is being read (= 0) or written (= 1). + * Allows 1,2,4-byte read, but no write. + */ + __u32 file_pos; /* Sysctl file position to read from, write to. + * Allows 1,2,4-byte read an 4-byte write. + */ +}; + #endif /* _UAPI__LINUX_BPF_H__ */ diff --git a/include/uapi/linux/btf.h b/include/uapi/linux/btf.h index 7b7475ef2f175c9279915aa8d9eb014845533bd9..9310652ca4f96bb56c6f7f561332bcff01511f3a 100644 --- a/include/uapi/linux/btf.h +++ b/include/uapi/linux/btf.h @@ -39,11 +39,11 @@ struct btf_type { * struct, union and fwd */ __u32 info; - /* "size" is used by INT, ENUM, STRUCT and UNION. + /* "size" is used by INT, ENUM, STRUCT, UNION and DATASEC. * "size" tells the size of the type it is describing. * * "type" is used by PTR, TYPEDEF, VOLATILE, CONST, RESTRICT, - * FUNC and FUNC_PROTO. + * FUNC, FUNC_PROTO and VAR. * "type" is a type_id referring to another type. */ union { @@ -70,8 +70,10 @@ struct btf_type { #define BTF_KIND_RESTRICT 11 /* Restrict */ #define BTF_KIND_FUNC 12 /* Function */ #define BTF_KIND_FUNC_PROTO 13 /* Function Proto */ -#define BTF_KIND_MAX 13 -#define NR_BTF_KINDS 14 +#define BTF_KIND_VAR 14 /* Variable */ +#define BTF_KIND_DATASEC 15 /* Section */ +#define BTF_KIND_MAX BTF_KIND_DATASEC +#define NR_BTF_KINDS (BTF_KIND_MAX + 1) /* For some specific BTF_KIND, "struct btf_type" is immediately * followed by extra data. @@ -138,4 +140,26 @@ struct btf_param { __u32 type; }; +enum { + BTF_VAR_STATIC = 0, + BTF_VAR_GLOBAL_ALLOCATED, +}; + +/* BTF_KIND_VAR is followed by a single "struct btf_var" to describe + * additional information related to the variable such as its linkage. + */ +struct btf_var { + __u32 linkage; +}; + +/* BTF_KIND_DATASEC is followed by multiple "struct btf_var_secinfo" + * to describe all BTF_KIND_VAR types it contains along with it's + * in-section offset as well as size. + */ +struct btf_var_secinfo { + __u32 type; + __u32 offset; + __u32 size; +}; + #endif /* _UAPI__LINUX_BTF_H__ */ diff --git a/include/uapi/linux/ethtool.h b/include/uapi/linux/ethtool.h index d473e5ed044c71df4e734f91c96f828952834dc3..3534ce157ae96670620adbbed93fb4641d13b680 100644 --- a/include/uapi/linux/ethtool.h +++ b/include/uapi/linux/ethtool.h @@ -252,9 +252,17 @@ struct ethtool_tunable { #define DOWNSHIFT_DEV_DEFAULT_COUNT 0xff #define DOWNSHIFT_DEV_DISABLE 0 +/* Time in msecs after which link is reported as down + * 0 = lowest time supported by the PHY + * 0xff = off, link down detection according to standard + */ +#define ETHTOOL_PHY_FAST_LINK_DOWN_ON 0 +#define ETHTOOL_PHY_FAST_LINK_DOWN_OFF 0xff + enum phy_tunable_id { ETHTOOL_PHY_ID_UNSPEC, ETHTOOL_PHY_DOWNSHIFT, + ETHTOOL_PHY_FAST_LINK_DOWN, /* * Add your fresh new phy tunable attribute above and remember to update * phy_tunable_strings[] in net/core/ethtool.c @@ -1704,6 +1712,9 @@ static inline int ethtool_validate_duplex(__u8 duplex) #define ETH_MODULE_SFF_8436 0x4 #define ETH_MODULE_SFF_8436_LEN 256 +#define ETH_MODULE_SFF_8636_MAX_LEN 640 +#define ETH_MODULE_SFF_8436_MAX_LEN 640 + /* Reset flags */ /* The reset() operation must clear the flags for the components which * were actually reset. On successful return, the flags indicate the diff --git a/include/uapi/linux/fou.h b/include/uapi/linux/fou.h index f2ea833a2812561d7b10517a39e695589e8b236b..87c2c9f0880304e20202350ea164b9293e085d50 100644 --- a/include/uapi/linux/fou.h +++ b/include/uapi/linux/fou.h @@ -16,6 +16,12 @@ enum { FOU_ATTR_IPPROTO, /* u8 */ FOU_ATTR_TYPE, /* u8 */ FOU_ATTR_REMCSUM_NOPARTIAL, /* flag */ + FOU_ATTR_LOCAL_V4, /* u32 */ + FOU_ATTR_LOCAL_V6, /* in6_addr */ + FOU_ATTR_PEER_V4, /* u32 */ + FOU_ATTR_PEER_V6, /* in6_addr */ + FOU_ATTR_PEER_PORT, /* u16 */ + FOU_ATTR_IFINDEX, /* s32 */ __FOU_ATTR_MAX, }; diff --git a/include/uapi/linux/icmpv6.h b/include/uapi/linux/icmpv6.h index 325395f56bfa61aaf733788bd43424e7462ddf2d..2622b5a3e6163a56faa4deb92ee92c8cb2321716 100644 --- a/include/uapi/linux/icmpv6.h +++ b/include/uapi/linux/icmpv6.h @@ -90,6 +90,8 @@ struct icmp6hdr { #define ICMPV6_TIME_EXCEED 3 #define ICMPV6_PARAMPROB 4 +#define ICMPV6_ERRMSG_MAX 127 + #define ICMPV6_INFOMSG_MASK 0x80 #define ICMPV6_ECHO_REQUEST 128 @@ -110,6 +112,8 @@ struct icmp6hdr { #define ICMPV6_MRDISC_ADV 151 +#define ICMPV6_MSG_MAX 255 + /* * Codes for Destination Unreachable */ diff --git a/include/uapi/linux/if_ether.h b/include/uapi/linux/if_ether.h index 3a45b4ad71a3083a16085d69e2891e31dddcdaa8..3158ba672b72ba24b97a5e7f341389c629202c11 100644 --- a/include/uapi/linux/if_ether.h +++ b/include/uapi/linux/if_ether.h @@ -109,6 +109,7 @@ #define ETH_P_QINQ2 0x9200 /* deprecated QinQ VLAN [ NOT AN OFFICIALLY REGISTERED ID ] */ #define ETH_P_QINQ3 0x9300 /* deprecated QinQ VLAN [ NOT AN OFFICIALLY REGISTERED ID ] */ #define ETH_P_EDSA 0xDADA /* Ethertype DSA [ NOT AN OFFICIALLY REGISTERED ID ] */ +#define ETH_P_DSA_8021Q 0xDADB /* Fake VLAN Header for DSA [ NOT AN OFFICIALLY REGISTERED ID ] */ #define ETH_P_IFE 0xED3E /* ForCES inter-FE LFB type */ #define ETH_P_AF_IUCV 0xFBFB /* IBM af_iucv [ NOT AN OFFICIALLY REGISTERED ID ] */ diff --git a/include/uapi/linux/if_tun.h b/include/uapi/linux/if_tun.h index 23a6753b37df3a0a241c2986aa932301bef02b1f..454ae31b93c7a4b4aeedddf7ceb5274377b39a58 100644 --- a/include/uapi/linux/if_tun.h +++ b/include/uapi/linux/if_tun.h @@ -60,6 +60,7 @@ #define TUNSETSTEERINGEBPF _IOR('T', 224, int) #define TUNSETFILTEREBPF _IOR('T', 225, int) #define TUNSETCARRIER _IOW('T', 226, int) +#define TUNGETDEVNETNS _IO('T', 227) /* TUNSETIFF ifr flags */ #define IFF_TUN 0x0001 diff --git a/include/uapi/linux/if_vlan.h b/include/uapi/linux/if_vlan.h index 7a0e8bd65b6b826a76b3c8b47fe7eddce7f18d6a..90a2c89afc8fe02b83406f72c51f76faa2890493 100644 --- a/include/uapi/linux/if_vlan.h +++ b/include/uapi/linux/if_vlan.h @@ -32,10 +32,11 @@ enum vlan_ioctl_cmds { }; enum vlan_flags { - VLAN_FLAG_REORDER_HDR = 0x1, - VLAN_FLAG_GVRP = 0x2, - VLAN_FLAG_LOOSE_BINDING = 0x4, - VLAN_FLAG_MVRP = 0x8, + VLAN_FLAG_REORDER_HDR = 0x1, + VLAN_FLAG_GVRP = 0x2, + VLAN_FLAG_LOOSE_BINDING = 0x4, + VLAN_FLAG_MVRP = 0x8, + VLAN_FLAG_BRIDGE_BINDING = 0x10, }; enum vlan_name_types { diff --git a/include/uapi/linux/ip_vs.h b/include/uapi/linux/ip_vs.h index 1c916b2f89dc8a31a330cb0979fc1ced992e51c6..e34f436fc79d35f7d5930e11ef5176bb088ecd9b 100644 --- a/include/uapi/linux/ip_vs.h +++ b/include/uapi/linux/ip_vs.h @@ -124,6 +124,13 @@ #define IP_VS_PEDATA_MAXLEN 255 +/* Tunnel types */ +enum { + IP_VS_CONN_F_TUNNEL_TYPE_IPIP = 0, /* IPIP */ + IP_VS_CONN_F_TUNNEL_TYPE_GUE, /* GUE */ + IP_VS_CONN_F_TUNNEL_TYPE_MAX, +}; + /* * The struct ip_vs_service_user and struct ip_vs_dest_user are * used to set IPVS rules through setsockopt. @@ -392,6 +399,10 @@ enum { IPVS_DEST_ATTR_STATS64, /* nested attribute for dest stats */ + IPVS_DEST_ATTR_TUN_TYPE, /* tunnel type */ + + IPVS_DEST_ATTR_TUN_PORT, /* tunnel port */ + __IPVS_DEST_ATTR_MAX, }; diff --git a/include/uapi/linux/netfilter/nf_tables.h b/include/uapi/linux/netfilter/nf_tables.h index a66c8de006cc0396afceb1b77db9634e4f2c5e56..f0cf7b0f4f352719056e52db790df82e6512bd14 100644 --- a/include/uapi/linux/netfilter/nf_tables.h +++ b/include/uapi/linux/netfilter/nf_tables.h @@ -967,6 +967,7 @@ enum nft_socket_keys { * @NFT_CT_SRC_IP6: conntrack layer 3 protocol source (IPv6 address) * @NFT_CT_DST_IP6: conntrack layer 3 protocol destination (IPv6 address) * @NFT_CT_TIMEOUT: connection tracking timeout policy assigned to conntrack + * @NFT_CT_ID: conntrack id */ enum nft_ct_keys { NFT_CT_STATE, @@ -993,6 +994,7 @@ enum nft_ct_keys { NFT_CT_SRC_IP6, NFT_CT_DST_IP6, NFT_CT_TIMEOUT, + NFT_CT_ID, __NFT_CT_MAX }; #define NFT_CT_MAX (__NFT_CT_MAX - 1) @@ -1522,15 +1524,21 @@ enum nft_flowtable_hook_attributes { * * @NFTA_OSF_DREG: destination register (NLA_U32: nft_registers) * @NFTA_OSF_TTL: Value of the TTL osf option (NLA_U8) + * @NFTA_OSF_FLAGS: flags (NLA_U32) */ enum nft_osf_attributes { NFTA_OSF_UNSPEC, NFTA_OSF_DREG, NFTA_OSF_TTL, + NFTA_OSF_FLAGS, __NFTA_OSF_MAX, }; #define NFTA_OSF_MAX (__NFTA_OSF_MAX - 1) +enum nft_osf_flags { + NFT_OSF_F_VERSION = (1 << 0), +}; + /** * enum nft_device_attributes - nf_tables device netlink attributes * diff --git a/include/uapi/linux/nl80211.h b/include/uapi/linux/nl80211.h index dd4f86ee286ec052a9ec2f4bb5d1f39bce55589b..6f09d1500960d98eb7cc35bfe9970e25aad728d9 100644 --- a/include/uapi/linux/nl80211.h +++ b/include/uapi/linux/nl80211.h @@ -11,7 +11,7 @@ * Copyright 2008 Jouni Malinen * Copyright 2008 Colin McCabe * Copyright 2015-2017 Intel Deutschland GmbH - * Copyright (C) 2018 Intel Corporation + * Copyright (C) 2018-2019 Intel Corporation * * Permission to use, copy, modify, and/or distribute this software for any * purpose with or without fee is hereby granted, provided that the above @@ -1065,6 +1065,26 @@ * indicated by %NL80211_ATTR_WIPHY_FREQ and other attributes * determining the width and type. * + * @NL80211_CMD_UPDATE_OWE_INFO: This interface allows the host driver to + * offload OWE processing to user space. This intends to support + * OWE AKM by the host drivers that implement SME but rely + * on the user space for the cryptographic/DH IE processing in AP mode. + * + * @NL80211_CMD_PROBE_MESH_LINK: The requirement for mesh link metric + * refreshing, is that from one mesh point we be able to send some data + * frames to other mesh points which are not currently selected as a + * primary traffic path, but which are only 1 hop away. The absence of + * the primary path to the chosen node makes it necessary to apply some + * form of marking on a chosen packet stream so that the packets can be + * properly steered to the selected node for testing, and not by the + * regular mesh path lookup. Further, the packets must be of type data + * so that the rate control (often embedded in firmware) is used for + * rate selection. + * + * Here attribute %NL80211_ATTR_MAC is used to specify connected mesh + * peer MAC address and %NL80211_ATTR_FRAME is used to specify the frame + * content. The frame is ethernet data. + * * @NL80211_CMD_MAX: highest used command number * @__NL80211_CMD_AFTER_LAST: internal use */ @@ -1285,6 +1305,10 @@ enum nl80211_commands { NL80211_CMD_NOTIFY_RADAR, + NL80211_CMD_UPDATE_OWE_INFO, + + NL80211_CMD_PROBE_MESH_LINK, + /* add new commands above here */ /* used to define NL80211_CMD_MAX below */ @@ -2308,6 +2332,15 @@ enum nl80211_commands { * @NL80211_ATTR_AIRTIME_WEIGHT: Station's weight when scheduled by the airtime * scheduler. * + * @NL80211_ATTR_STA_TX_POWER_SETTING: Transmit power setting type (u8) for + * station associated with the AP. See &enum nl80211_tx_power_setting for + * possible values. + * @NL80211_ATTR_STA_TX_POWER: Transmit power level (s16) in dBm units. This + * allows to set Tx power for a station. If this attribute is not included, + * the default per-interface tx power setting will be overriding. Driver + * should be picking up the lowest tx power, either tx power per-interface + * or per-station. + * * @NUM_NL80211_ATTR: total number of nl80211_attrs available * @NL80211_ATTR_MAX: highest attribute number currently defined * @__NL80211_ATTR_AFTER_LAST: internal use @@ -2758,6 +2791,8 @@ enum nl80211_attrs { NL80211_ATTR_PEER_MEASUREMENTS, NL80211_ATTR_AIRTIME_WEIGHT, + NL80211_ATTR_STA_TX_POWER_SETTING, + NL80211_ATTR_STA_TX_POWER, /* add attributes here, update the policy in nl80211.c */ @@ -2802,7 +2837,7 @@ enum nl80211_attrs { #define NL80211_MAX_SUPP_RATES 32 #define NL80211_MAX_SUPP_HT_RATES 77 -#define NL80211_MAX_SUPP_REG_RULES 64 +#define NL80211_MAX_SUPP_REG_RULES 128 #define NL80211_TKIP_DATA_OFFSET_ENCR_KEY 0 #define NL80211_TKIP_DATA_OFFSET_TX_MIC_KEY 16 #define NL80211_TKIP_DATA_OFFSET_RX_MIC_KEY 24 @@ -3139,6 +3174,7 @@ enum nl80211_sta_bss_param { * @NL80211_STA_INFO_TX_DURATION: aggregate PPDU duration for all frames * sent to the station (u64, usec) * @NL80211_STA_INFO_AIRTIME_WEIGHT: current airtime weight for station (u16) + * @NL80211_STA_INFO_AIRTIME_LINK_METRIC: airtime link metric for mesh station * @__NL80211_STA_INFO_AFTER_LAST: internal * @NL80211_STA_INFO_MAX: highest possible station info attribute */ @@ -3184,6 +3220,7 @@ enum nl80211_sta_info { NL80211_STA_INFO_CONNECTED_TO_GATE, NL80211_STA_INFO_TX_DURATION, NL80211_STA_INFO_AIRTIME_WEIGHT, + NL80211_STA_INFO_AIRTIME_LINK_METRIC, /* keep last */ __NL80211_STA_INFO_AFTER_LAST, @@ -3638,6 +3675,14 @@ enum nl80211_reg_rule_attr { * value as specified by &struct nl80211_bss_select_rssi_adjust. * @NL80211_SCHED_SCAN_MATCH_ATTR_BSSID: BSSID to be used for matching * (this cannot be used together with SSID). + * @NL80211_SCHED_SCAN_MATCH_PER_BAND_RSSI: Nested attribute that carries the + * band specific minimum rssi thresholds for the bands defined in + * enum nl80211_band. The minimum rssi threshold value(s32) specific to a + * band shall be encapsulated in attribute with type value equals to one + * of the NL80211_BAND_* defined in enum nl80211_band. For example, the + * minimum rssi threshold value for 2.4GHZ band shall be encapsulated + * within an attribute of type NL80211_BAND_2GHZ. And one or more of such + * attributes will be nested within this attribute. * @NL80211_SCHED_SCAN_MATCH_ATTR_MAX: highest scheduled scan filter * attribute number currently defined * @__NL80211_SCHED_SCAN_MATCH_ATTR_AFTER_LAST: internal use @@ -3650,6 +3695,7 @@ enum nl80211_sched_scan_match_attr { NL80211_SCHED_SCAN_MATCH_ATTR_RELATIVE_RSSI, NL80211_SCHED_SCAN_MATCH_ATTR_RSSI_ADJUST, NL80211_SCHED_SCAN_MATCH_ATTR_BSSID, + NL80211_SCHED_SCAN_MATCH_PER_BAND_RSSI, /* keep last */ __NL80211_SCHED_SCAN_MATCH_ATTR_AFTER_LAST, @@ -4134,6 +4180,27 @@ enum nl80211_channel_type { NL80211_CHAN_HT40PLUS }; +/** + * enum nl80211_key_mode - Key mode + * + * @NL80211_KEY_RX_TX: (Default) + * Key can be used for Rx and Tx immediately + * + * The following modes can only be selected for unicast keys and when the + * driver supports @NL80211_EXT_FEATURE_EXT_KEY_ID: + * + * @NL80211_KEY_NO_TX: Only allowed in combination with @NL80211_CMD_NEW_KEY: + * Unicast key can only be used for Rx, Tx not allowed, yet + * @NL80211_KEY_SET_TX: Only allowed in combination with @NL80211_CMD_SET_KEY: + * The unicast key identified by idx and mac is cleared for Tx and becomes + * the preferred Tx key for the station. + */ +enum nl80211_key_mode { + NL80211_KEY_RX_TX, + NL80211_KEY_NO_TX, + NL80211_KEY_SET_TX +}; + /** * enum nl80211_chan_width - channel width definitions * @@ -4377,6 +4444,9 @@ enum nl80211_key_default_types { * @NL80211_KEY_DEFAULT_TYPES: A nested attribute containing flags * attributes, specifying what a key should be set as default as. * See &enum nl80211_key_default_types. + * @NL80211_KEY_MODE: the mode from enum nl80211_key_mode. + * Defaults to @NL80211_KEY_RX_TX. + * * @__NL80211_KEY_AFTER_LAST: internal * @NL80211_KEY_MAX: highest key attribute */ @@ -4390,6 +4460,7 @@ enum nl80211_key_attributes { NL80211_KEY_DEFAULT_MGMT, NL80211_KEY_TYPE, NL80211_KEY_DEFAULT_TYPES, + NL80211_KEY_MODE, /* keep last */ __NL80211_KEY_AFTER_LAST, @@ -5335,6 +5406,8 @@ enum nl80211_feature_flags { * able to rekey an in-use key correctly. Userspace must not rekey PTK keys * if this flag is not set. Ignoring this can leak clear text packets and/or * freeze the connection. + * @NL80211_EXT_FEATURE_EXT_KEY_ID: Driver supports "Extended Key ID for + * Individually Addressed Frames" from IEEE802.11-2016. * * @NL80211_EXT_FEATURE_AIRTIME_FAIRNESS: Driver supports getting airtime * fairness for transmitted packets and has enabled airtime fairness @@ -5343,6 +5416,12 @@ enum nl80211_feature_flags { * @NL80211_EXT_FEATURE_AP_PMKSA_CACHING: Driver/device supports PMKSA caching * (set/del PMKSA operations) in AP mode. * + * @NL80211_EXT_FEATURE_SCHED_SCAN_BAND_SPECIFIC_RSSI_THOLD: Driver supports + * filtering of sched scan results using band specific RSSI thresholds. + * + * @NL80211_EXT_FEATURE_STA_TX_PWR: This driver supports controlling tx power + * to a station. + * * @NUM_NL80211_EXT_FEATURES: number of extended features. * @MAX_NL80211_EXT_FEATURES: highest extended feature index. */ @@ -5384,6 +5463,9 @@ enum nl80211_ext_feature_index { NL80211_EXT_FEATURE_ENABLE_FTM_RESPONDER, NL80211_EXT_FEATURE_AIRTIME_FAIRNESS, NL80211_EXT_FEATURE_AP_PMKSA_CACHING, + NL80211_EXT_FEATURE_SCHED_SCAN_BAND_SPECIFIC_RSSI_THOLD, + NL80211_EXT_FEATURE_EXT_KEY_ID, + NL80211_EXT_FEATURE_STA_TX_PWR, /* add new features before the definition below */ NUM_NL80211_EXT_FEATURES, diff --git a/include/uapi/linux/openvswitch.h b/include/uapi/linux/openvswitch.h index dbe0cbe4f1b72b9e0c4751791614e98b0825989f..f271f1ec50aed08cb8f0e52fdd9ae8752b8c3b83 100644 --- a/include/uapi/linux/openvswitch.h +++ b/include/uapi/linux/openvswitch.h @@ -364,6 +364,7 @@ enum ovs_tunnel_key_attr { OVS_TUNNEL_KEY_ATTR_IPV6_DST, /* struct in6_addr dst IPv6 address. */ OVS_TUNNEL_KEY_ATTR_PAD, OVS_TUNNEL_KEY_ATTR_ERSPAN_OPTS, /* struct erspan_metadata */ + OVS_TUNNEL_KEY_ATTR_IPV4_INFO_BRIDGE, /* No argument. IPV4_INFO_BRIDGE mode.*/ __OVS_TUNNEL_KEY_ATTR_MAX }; @@ -734,6 +735,7 @@ struct ovs_action_hash { * be received on NFNLGRP_CONNTRACK_NEW and NFNLGRP_CONNTRACK_DESTROY groups, * respectively. Remaining bits control the changes for which an event is * delivered on the NFNLGRP_CONNTRACK_UPDATE group. + * @OVS_CT_ATTR_TIMEOUT: Variable length string defining conntrack timeout. */ enum ovs_ct_attr { OVS_CT_ATTR_UNSPEC, @@ -746,6 +748,8 @@ enum ovs_ct_attr { OVS_CT_ATTR_NAT, /* Nested OVS_NAT_ATTR_* */ OVS_CT_ATTR_FORCE_COMMIT, /* No argument */ OVS_CT_ATTR_EVENTMASK, /* u32 mask of IPCT_* events. */ + OVS_CT_ATTR_TIMEOUT, /* Associate timeout with this connection for + * fine-grain timeout tuning. */ __OVS_CT_ATTR_MAX }; @@ -798,6 +802,44 @@ struct ovs_action_push_eth { struct ovs_key_ethernet addresses; }; +/* + * enum ovs_check_pkt_len_attr - Attributes for %OVS_ACTION_ATTR_CHECK_PKT_LEN. + * + * @OVS_CHECK_PKT_LEN_ATTR_PKT_LEN: u16 Packet length to check for. + * @OVS_CHECK_PKT_LEN_ATTR_ACTIONS_IF_GREATER: Nested OVS_ACTION_ATTR_* + * actions to apply if the packer length is greater than the specified + * length in the attr - OVS_CHECK_PKT_LEN_ATTR_PKT_LEN. + * @OVS_CHECK_PKT_LEN_ATTR_ACTIONS_IF_LESS_EQUAL - Nested OVS_ACTION_ATTR_* + * actions to apply if the packer length is lesser or equal to the specified + * length in the attr - OVS_CHECK_PKT_LEN_ATTR_PKT_LEN. + */ +enum ovs_check_pkt_len_attr { + OVS_CHECK_PKT_LEN_ATTR_UNSPEC, + OVS_CHECK_PKT_LEN_ATTR_PKT_LEN, + OVS_CHECK_PKT_LEN_ATTR_ACTIONS_IF_GREATER, + OVS_CHECK_PKT_LEN_ATTR_ACTIONS_IF_LESS_EQUAL, + __OVS_CHECK_PKT_LEN_ATTR_MAX, + +#ifdef __KERNEL__ + OVS_CHECK_PKT_LEN_ATTR_ARG /* struct check_pkt_len_arg */ +#endif +}; + +#define OVS_CHECK_PKT_LEN_ATTR_MAX (__OVS_CHECK_PKT_LEN_ATTR_MAX - 1) + +#ifdef __KERNEL__ +struct check_pkt_len_arg { + u16 pkt_len; /* Same value as OVS_CHECK_PKT_LEN_ATTR_PKT_LEN'. */ + bool exec_for_greater; /* When true, actions in IF_GREATER will + * not change flow keys. False otherwise. + */ + bool exec_for_lesser_equal; /* When true, actions in IF_LESS_EQUAL + * will not change flow keys. False + * otherwise. + */ +}; +#endif + /** * enum ovs_action_attr - Action types. * @@ -842,6 +884,9 @@ struct ovs_action_push_eth { * packet, or modify the packet (e.g., change the DSCP field). * @OVS_ACTION_ATTR_CLONE: make a copy of the packet and execute a list of * actions without affecting the original packet and key. + * @OVS_ACTION_ATTR_CHECK_PKT_LEN: Check the packet length and execute a set + * of actions if greater than the specified packet length, else execute + * another set of actions. * * Only a single header can be set with a single %OVS_ACTION_ATTR_SET. Not all * fields within a header are modifiable, e.g. the IPv4 protocol and fragment @@ -876,6 +921,7 @@ enum ovs_action_attr { OVS_ACTION_ATTR_POP_NSH, /* No argument. */ OVS_ACTION_ATTR_METER, /* u32 meter ID. */ OVS_ACTION_ATTR_CLONE, /* Nested OVS_CLONE_ATTR_*. */ + OVS_ACTION_ATTR_CHECK_PKT_LEN, /* Nested OVS_CHECK_PKT_LEN_ATTR_*. */ __OVS_ACTION_ATTR_MAX, /* Nothing past this will be accepted * from userspace. */ diff --git a/include/uapi/linux/pkt_sched.h b/include/uapi/linux/pkt_sched.h index 7ee74c3474bff4b39b11fc48cb6154fdb4b6aad0..8b2f993cbb773dcfb90dadc306cb98d5a66083fb 100644 --- a/include/uapi/linux/pkt_sched.h +++ b/include/uapi/linux/pkt_sched.h @@ -1148,6 +1148,16 @@ enum { #define TCA_TAPRIO_SCHED_MAX (__TCA_TAPRIO_SCHED_MAX - 1) +/* The format for the admin sched (dump only): + * [TCA_TAPRIO_SCHED_ADMIN_SCHED] + * [TCA_TAPRIO_ATTR_SCHED_BASE_TIME] + * [TCA_TAPRIO_ATTR_SCHED_ENTRY_LIST] + * [TCA_TAPRIO_ATTR_SCHED_ENTRY] + * [TCA_TAPRIO_ATTR_SCHED_ENTRY_CMD] + * [TCA_TAPRIO_ATTR_SCHED_ENTRY_GATES] + * [TCA_TAPRIO_ATTR_SCHED_ENTRY_INTERVAL] + */ + enum { TCA_TAPRIO_ATTR_UNSPEC, TCA_TAPRIO_ATTR_PRIOMAP, /* struct tc_mqprio_qopt */ @@ -1156,6 +1166,9 @@ enum { TCA_TAPRIO_ATTR_SCHED_SINGLE_ENTRY, /* single entry */ TCA_TAPRIO_ATTR_SCHED_CLOCKID, /* s32 */ TCA_TAPRIO_PAD, + TCA_TAPRIO_ATTR_ADMIN_SCHED, /* The admin sched, only used in dump */ + TCA_TAPRIO_ATTR_SCHED_CYCLE_TIME, /* s64 */ + TCA_TAPRIO_ATTR_SCHED_CYCLE_TIME_EXTENSION, /* s64 */ __TCA_TAPRIO_ATTR_MAX, }; diff --git a/include/uapi/linux/sockios.h b/include/uapi/linux/sockios.h index d393e9ed396426125152a0583a6837274e111e0f..7d1bccbbef781f2c0064b568ee22dba135b5b87e 100644 --- a/include/uapi/linux/sockios.h +++ b/include/uapi/linux/sockios.h @@ -19,6 +19,7 @@ #ifndef _LINUX_SOCKIOS_H #define _LINUX_SOCKIOS_H +#include #include /* Linux-specific socket ioctls */ @@ -27,6 +28,26 @@ #define SOCK_IOC_TYPE 0x89 +/* + * the timeval/timespec data structure layout is defined by libc, + * so we need to cover both possible versions on 32-bit. + */ +/* Get stamp (timeval) */ +#define SIOCGSTAMP_NEW _IOR(SOCK_IOC_TYPE, 0x06, long long[2]) +/* Get stamp (timespec) */ +#define SIOCGSTAMPNS_NEW _IOR(SOCK_IOC_TYPE, 0x07, long long[2]) + +#if __BITS_PER_LONG == 64 || (defined(__x86_64__) && defined(__ILP32__)) +/* on 64-bit and x32, avoid the ?: operator */ +#define SIOCGSTAMP SIOCGSTAMP_OLD +#define SIOCGSTAMPNS SIOCGSTAMPNS_OLD +#else +#define SIOCGSTAMP ((sizeof(struct timeval)) == 8 ? \ + SIOCGSTAMP_OLD : SIOCGSTAMP_NEW) +#define SIOCGSTAMPNS ((sizeof(struct timespec)) == 8 ? \ + SIOCGSTAMPNS_OLD : SIOCGSTAMPNS_NEW) +#endif + /* Routing table calls. */ #define SIOCADDRT 0x890B /* add routing table entry */ #define SIOCDELRT 0x890C /* delete routing table entry */ diff --git a/include/uapi/linux/tcp.h b/include/uapi/linux/tcp.h index 8bb6cc5f3235696ad264623b19440dd20f1019c7..b521464ea9627f922d2cc4b4ded68d1bc95d602e 100644 --- a/include/uapi/linux/tcp.h +++ b/include/uapi/linux/tcp.h @@ -160,15 +160,42 @@ enum { #define TCPI_OPT_ECN_SEEN 16 /* we received at least one packet with ECT */ #define TCPI_OPT_SYN_DATA 32 /* SYN-ACK acked data in SYN sent or rcvd */ +/* + * Sender's congestion state indicating normal or abnormal situations + * in the last round of packets sent. The state is driven by the ACK + * information and timer events. + */ enum tcp_ca_state { + /* + * Nothing bad has been observed recently. + * No apparent reordering, packet loss, or ECN marks. + */ TCP_CA_Open = 0, #define TCPF_CA_Open (1<max_entries == 0 || attr->key_size != 4 || attr->value_size == 0 || attr->map_flags & ~ARRAY_CREATE_FLAG_MASK || + !bpf_map_flags_access_ok(attr->map_flags) || (percpu && numa_node != NUMA_NO_NODE)) return -EINVAL; @@ -160,6 +161,36 @@ static void *array_map_lookup_elem(struct bpf_map *map, void *key) return array->value + array->elem_size * (index & array->index_mask); } +static int array_map_direct_value_addr(const struct bpf_map *map, u64 *imm, + u32 off) +{ + struct bpf_array *array = container_of(map, struct bpf_array, map); + + if (map->max_entries != 1) + return -ENOTSUPP; + if (off >= map->value_size) + return -EINVAL; + + *imm = (unsigned long)array->value; + return 0; +} + +static int array_map_direct_value_meta(const struct bpf_map *map, u64 imm, + u32 *off) +{ + struct bpf_array *array = container_of(map, struct bpf_array, map); + u64 base = (unsigned long)array->value; + u64 range = array->elem_size; + + if (map->max_entries != 1) + return -ENOTSUPP; + if (imm < base || imm >= base + range) + return -ENOENT; + + *off = imm - base; + return 0; +} + /* emit BPF instructions equivalent to C code of array_map_lookup_elem() */ static u32 array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf) { @@ -360,7 +391,8 @@ static void array_map_seq_show_elem(struct bpf_map *map, void *key, return; } - seq_printf(m, "%u: ", *(u32 *)key); + if (map->btf_key_type_id) + seq_printf(m, "%u: ", *(u32 *)key); btf_type_seq_show(map->btf, map->btf_value_type_id, value, m); seq_puts(m, "\n"); @@ -397,6 +429,18 @@ static int array_map_check_btf(const struct bpf_map *map, { u32 int_data; + /* One exception for keyless BTF: .bss/.data/.rodata map */ + if (btf_type_is_void(key_type)) { + if (map->map_type != BPF_MAP_TYPE_ARRAY || + map->max_entries != 1) + return -EINVAL; + + if (BTF_INFO_KIND(value_type->info) != BTF_KIND_DATASEC) + return -EINVAL; + + return 0; + } + if (BTF_INFO_KIND(key_type->info) != BTF_KIND_INT) return -EINVAL; @@ -419,6 +463,8 @@ const struct bpf_map_ops array_map_ops = { .map_update_elem = array_map_update_elem, .map_delete_elem = array_map_delete_elem, .map_gen_lookup = array_map_gen_lookup, + .map_direct_value_addr = array_map_direct_value_addr, + .map_direct_value_meta = array_map_direct_value_meta, .map_seq_show_elem = array_map_seq_show_elem, .map_check_btf = array_map_check_btf, }; @@ -440,6 +486,9 @@ static int fd_array_map_alloc_check(union bpf_attr *attr) /* only file descriptors can be stored in this type of map */ if (attr->value_size != sizeof(u32)) return -EINVAL; + /* Program read-only/write-only not supported for special maps yet. */ + if (attr->map_flags & (BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG)) + return -EINVAL; return array_map_alloc_check(attr); } diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index bd3921b1514b0b24bc9600f04190702badb4897f..cad09858a5f25beabb53aaaefeb1b4ad5a7a44f1 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -185,6 +185,16 @@ i < btf_type_vlen(struct_type); \ i++, member++) +#define for_each_vsi(i, struct_type, member) \ + for (i = 0, member = btf_type_var_secinfo(struct_type); \ + i < btf_type_vlen(struct_type); \ + i++, member++) + +#define for_each_vsi_from(i, from, struct_type, member) \ + for (i = from, member = btf_type_var_secinfo(struct_type) + from; \ + i < btf_type_vlen(struct_type); \ + i++, member++) + static DEFINE_IDR(btf_idr); static DEFINE_SPINLOCK(btf_idr_lock); @@ -262,6 +272,8 @@ static const char * const btf_kind_str[NR_BTF_KINDS] = { [BTF_KIND_RESTRICT] = "RESTRICT", [BTF_KIND_FUNC] = "FUNC", [BTF_KIND_FUNC_PROTO] = "FUNC_PROTO", + [BTF_KIND_VAR] = "VAR", + [BTF_KIND_DATASEC] = "DATASEC", }; struct btf_kind_operations { @@ -314,7 +326,7 @@ static bool btf_type_is_modifier(const struct btf_type *t) return false; } -static bool btf_type_is_void(const struct btf_type *t) +bool btf_type_is_void(const struct btf_type *t) { return t == &btf_void; } @@ -375,13 +387,36 @@ static bool btf_type_is_int(const struct btf_type *t) return BTF_INFO_KIND(t->info) == BTF_KIND_INT; } +static bool btf_type_is_var(const struct btf_type *t) +{ + return BTF_INFO_KIND(t->info) == BTF_KIND_VAR; +} + +static bool btf_type_is_datasec(const struct btf_type *t) +{ + return BTF_INFO_KIND(t->info) == BTF_KIND_DATASEC; +} + +/* Types that act only as a source, not sink or intermediate + * type when resolving. + */ +static bool btf_type_is_resolve_source_only(const struct btf_type *t) +{ + return btf_type_is_var(t) || + btf_type_is_datasec(t); +} + /* What types need to be resolved? * * btf_type_is_modifier() is an obvious one. * * btf_type_is_struct() because its member refers to * another type (through member->type). - + * + * btf_type_is_var() because the variable refers to + * another type. btf_type_is_datasec() holds multiple + * btf_type_is_var() types that need resolving. + * * btf_type_is_array() because its element (array->type) * refers to another type. Array can be thought of a * special case of struct while array just has the same @@ -390,9 +425,11 @@ static bool btf_type_is_int(const struct btf_type *t) static bool btf_type_needs_resolve(const struct btf_type *t) { return btf_type_is_modifier(t) || - btf_type_is_ptr(t) || - btf_type_is_struct(t) || - btf_type_is_array(t); + btf_type_is_ptr(t) || + btf_type_is_struct(t) || + btf_type_is_array(t) || + btf_type_is_var(t) || + btf_type_is_datasec(t); } /* t->size can be used */ @@ -403,6 +440,7 @@ static bool btf_type_has_size(const struct btf_type *t) case BTF_KIND_STRUCT: case BTF_KIND_UNION: case BTF_KIND_ENUM: + case BTF_KIND_DATASEC: return true; } @@ -467,6 +505,16 @@ static const struct btf_enum *btf_type_enum(const struct btf_type *t) return (const struct btf_enum *)(t + 1); } +static const struct btf_var *btf_type_var(const struct btf_type *t) +{ + return (const struct btf_var *)(t + 1); +} + +static const struct btf_var_secinfo *btf_type_var_secinfo(const struct btf_type *t) +{ + return (const struct btf_var_secinfo *)(t + 1); +} + static const struct btf_kind_operations *btf_type_ops(const struct btf_type *t) { return kind_ops[BTF_INFO_KIND(t->info)]; @@ -478,23 +526,31 @@ static bool btf_name_offset_valid(const struct btf *btf, u32 offset) offset < btf->hdr.str_len; } -/* Only C-style identifier is permitted. This can be relaxed if - * necessary. - */ -static bool btf_name_valid_identifier(const struct btf *btf, u32 offset) +static bool __btf_name_char_ok(char c, bool first, bool dot_ok) +{ + if ((first ? !isalpha(c) : + !isalnum(c)) && + c != '_' && + ((c == '.' && !dot_ok) || + c != '.')) + return false; + return true; +} + +static bool __btf_name_valid(const struct btf *btf, u32 offset, bool dot_ok) { /* offset must be valid */ const char *src = &btf->strings[offset]; const char *src_limit; - if (!isalpha(*src) && *src != '_') + if (!__btf_name_char_ok(*src, true, dot_ok)) return false; /* set a limit on identifier length */ src_limit = src + KSYM_NAME_LEN; src++; while (*src && src < src_limit) { - if (!isalnum(*src) && *src != '_') + if (!__btf_name_char_ok(*src, false, dot_ok)) return false; src++; } @@ -502,6 +558,19 @@ static bool btf_name_valid_identifier(const struct btf *btf, u32 offset) return !*src; } +/* Only C-style identifier is permitted. This can be relaxed if + * necessary. + */ +static bool btf_name_valid_identifier(const struct btf *btf, u32 offset) +{ + return __btf_name_valid(btf, offset, false); +} + +static bool btf_name_valid_section(const struct btf *btf, u32 offset) +{ + return __btf_name_valid(btf, offset, true); +} + static const char *__btf_name_by_offset(const struct btf *btf, u32 offset) { if (!offset) @@ -697,6 +766,32 @@ static void btf_verifier_log_member(struct btf_verifier_env *env, __btf_verifier_log(log, "\n"); } +__printf(4, 5) +static void btf_verifier_log_vsi(struct btf_verifier_env *env, + const struct btf_type *datasec_type, + const struct btf_var_secinfo *vsi, + const char *fmt, ...) +{ + struct bpf_verifier_log *log = &env->log; + va_list args; + + if (!bpf_verifier_log_needed(log)) + return; + if (env->phase != CHECK_META) + btf_verifier_log_type(env, datasec_type, NULL); + + __btf_verifier_log(log, "\t type_id=%u offset=%u size=%u", + vsi->type, vsi->offset, vsi->size); + if (fmt && *fmt) { + __btf_verifier_log(log, " "); + va_start(args, fmt); + bpf_verifier_vlog(log, fmt, args); + va_end(args); + } + + __btf_verifier_log(log, "\n"); +} + static void btf_verifier_log_hdr(struct btf_verifier_env *env, u32 btf_data_size) { @@ -974,7 +1069,8 @@ const struct btf_type *btf_type_id_size(const struct btf *btf, } else if (btf_type_is_ptr(size_type)) { size = sizeof(void *); } else { - if (WARN_ON_ONCE(!btf_type_is_modifier(size_type))) + if (WARN_ON_ONCE(!btf_type_is_modifier(size_type) && + !btf_type_is_var(size_type))) return NULL; size = btf->resolved_sizes[size_type_id]; @@ -1509,7 +1605,7 @@ static int btf_modifier_resolve(struct btf_verifier_env *env, u32 next_type_size = 0; next_type = btf_type_by_id(btf, next_type_id); - if (!next_type) { + if (!next_type || btf_type_is_resolve_source_only(next_type)) { btf_verifier_log_type(env, v->t, "Invalid type_id"); return -EINVAL; } @@ -1542,6 +1638,53 @@ static int btf_modifier_resolve(struct btf_verifier_env *env, return 0; } +static int btf_var_resolve(struct btf_verifier_env *env, + const struct resolve_vertex *v) +{ + const struct btf_type *next_type; + const struct btf_type *t = v->t; + u32 next_type_id = t->type; + struct btf *btf = env->btf; + u32 next_type_size; + + next_type = btf_type_by_id(btf, next_type_id); + if (!next_type || btf_type_is_resolve_source_only(next_type)) { + btf_verifier_log_type(env, v->t, "Invalid type_id"); + return -EINVAL; + } + + if (!env_type_is_resolve_sink(env, next_type) && + !env_type_is_resolved(env, next_type_id)) + return env_stack_push(env, next_type, next_type_id); + + if (btf_type_is_modifier(next_type)) { + const struct btf_type *resolved_type; + u32 resolved_type_id; + + resolved_type_id = next_type_id; + resolved_type = btf_type_id_resolve(btf, &resolved_type_id); + + if (btf_type_is_ptr(resolved_type) && + !env_type_is_resolve_sink(env, resolved_type) && + !env_type_is_resolved(env, resolved_type_id)) + return env_stack_push(env, resolved_type, + resolved_type_id); + } + + /* We must resolve to something concrete at this point, no + * forward types or similar that would resolve to size of + * zero is allowed. + */ + if (!btf_type_id_size(btf, &next_type_id, &next_type_size)) { + btf_verifier_log_type(env, v->t, "Invalid type_id"); + return -EINVAL; + } + + env_stack_pop_resolved(env, next_type_id, next_type_size); + + return 0; +} + static int btf_ptr_resolve(struct btf_verifier_env *env, const struct resolve_vertex *v) { @@ -1551,7 +1694,7 @@ static int btf_ptr_resolve(struct btf_verifier_env *env, struct btf *btf = env->btf; next_type = btf_type_by_id(btf, next_type_id); - if (!next_type) { + if (!next_type || btf_type_is_resolve_source_only(next_type)) { btf_verifier_log_type(env, v->t, "Invalid type_id"); return -EINVAL; } @@ -1609,6 +1752,15 @@ static void btf_modifier_seq_show(const struct btf *btf, btf_type_ops(t)->seq_show(btf, t, type_id, data, bits_offset, m); } +static void btf_var_seq_show(const struct btf *btf, const struct btf_type *t, + u32 type_id, void *data, u8 bits_offset, + struct seq_file *m) +{ + t = btf_type_id_resolve(btf, &type_id); + + btf_type_ops(t)->seq_show(btf, t, type_id, data, bits_offset, m); +} + static void btf_ptr_seq_show(const struct btf *btf, const struct btf_type *t, u32 type_id, void *data, u8 bits_offset, struct seq_file *m) @@ -1776,7 +1928,8 @@ static int btf_array_resolve(struct btf_verifier_env *env, /* Check array->index_type */ index_type_id = array->index_type; index_type = btf_type_by_id(btf, index_type_id); - if (btf_type_nosize_or_null(index_type)) { + if (btf_type_is_resolve_source_only(index_type) || + btf_type_nosize_or_null(index_type)) { btf_verifier_log_type(env, v->t, "Invalid index"); return -EINVAL; } @@ -1795,7 +1948,8 @@ static int btf_array_resolve(struct btf_verifier_env *env, /* Check array->type */ elem_type_id = array->type; elem_type = btf_type_by_id(btf, elem_type_id); - if (btf_type_nosize_or_null(elem_type)) { + if (btf_type_is_resolve_source_only(elem_type) || + btf_type_nosize_or_null(elem_type)) { btf_verifier_log_type(env, v->t, "Invalid elem"); return -EINVAL; @@ -2016,7 +2170,8 @@ static int btf_struct_resolve(struct btf_verifier_env *env, const struct btf_type *member_type = btf_type_by_id(env->btf, member_type_id); - if (btf_type_nosize_or_null(member_type)) { + if (btf_type_is_resolve_source_only(member_type) || + btf_type_nosize_or_null(member_type)) { btf_verifier_log_member(env, v->t, member, "Invalid member"); return -EINVAL; @@ -2411,6 +2566,222 @@ static struct btf_kind_operations func_ops = { .seq_show = btf_df_seq_show, }; +static s32 btf_var_check_meta(struct btf_verifier_env *env, + const struct btf_type *t, + u32 meta_left) +{ + const struct btf_var *var; + u32 meta_needed = sizeof(*var); + + if (meta_left < meta_needed) { + btf_verifier_log_basic(env, t, + "meta_left:%u meta_needed:%u", + meta_left, meta_needed); + return -EINVAL; + } + + if (btf_type_vlen(t)) { + btf_verifier_log_type(env, t, "vlen != 0"); + return -EINVAL; + } + + if (btf_type_kflag(t)) { + btf_verifier_log_type(env, t, "Invalid btf_info kind_flag"); + return -EINVAL; + } + + if (!t->name_off || + !__btf_name_valid(env->btf, t->name_off, true)) { + btf_verifier_log_type(env, t, "Invalid name"); + return -EINVAL; + } + + /* A var cannot be in type void */ + if (!t->type || !BTF_TYPE_ID_VALID(t->type)) { + btf_verifier_log_type(env, t, "Invalid type_id"); + return -EINVAL; + } + + var = btf_type_var(t); + if (var->linkage != BTF_VAR_STATIC && + var->linkage != BTF_VAR_GLOBAL_ALLOCATED) { + btf_verifier_log_type(env, t, "Linkage not supported"); + return -EINVAL; + } + + btf_verifier_log_type(env, t, NULL); + + return meta_needed; +} + +static void btf_var_log(struct btf_verifier_env *env, const struct btf_type *t) +{ + const struct btf_var *var = btf_type_var(t); + + btf_verifier_log(env, "type_id=%u linkage=%u", t->type, var->linkage); +} + +static const struct btf_kind_operations var_ops = { + .check_meta = btf_var_check_meta, + .resolve = btf_var_resolve, + .check_member = btf_df_check_member, + .check_kflag_member = btf_df_check_kflag_member, + .log_details = btf_var_log, + .seq_show = btf_var_seq_show, +}; + +static s32 btf_datasec_check_meta(struct btf_verifier_env *env, + const struct btf_type *t, + u32 meta_left) +{ + const struct btf_var_secinfo *vsi; + u64 last_vsi_end_off = 0, sum = 0; + u32 i, meta_needed; + + meta_needed = btf_type_vlen(t) * sizeof(*vsi); + if (meta_left < meta_needed) { + btf_verifier_log_basic(env, t, + "meta_left:%u meta_needed:%u", + meta_left, meta_needed); + return -EINVAL; + } + + if (!btf_type_vlen(t)) { + btf_verifier_log_type(env, t, "vlen == 0"); + return -EINVAL; + } + + if (!t->size) { + btf_verifier_log_type(env, t, "size == 0"); + return -EINVAL; + } + + if (btf_type_kflag(t)) { + btf_verifier_log_type(env, t, "Invalid btf_info kind_flag"); + return -EINVAL; + } + + if (!t->name_off || + !btf_name_valid_section(env->btf, t->name_off)) { + btf_verifier_log_type(env, t, "Invalid name"); + return -EINVAL; + } + + btf_verifier_log_type(env, t, NULL); + + for_each_vsi(i, t, vsi) { + /* A var cannot be in type void */ + if (!vsi->type || !BTF_TYPE_ID_VALID(vsi->type)) { + btf_verifier_log_vsi(env, t, vsi, + "Invalid type_id"); + return -EINVAL; + } + + if (vsi->offset < last_vsi_end_off || vsi->offset >= t->size) { + btf_verifier_log_vsi(env, t, vsi, + "Invalid offset"); + return -EINVAL; + } + + if (!vsi->size || vsi->size > t->size) { + btf_verifier_log_vsi(env, t, vsi, + "Invalid size"); + return -EINVAL; + } + + last_vsi_end_off = vsi->offset + vsi->size; + if (last_vsi_end_off > t->size) { + btf_verifier_log_vsi(env, t, vsi, + "Invalid offset+size"); + return -EINVAL; + } + + btf_verifier_log_vsi(env, t, vsi, NULL); + sum += vsi->size; + } + + if (t->size < sum) { + btf_verifier_log_type(env, t, "Invalid btf_info size"); + return -EINVAL; + } + + return meta_needed; +} + +static int btf_datasec_resolve(struct btf_verifier_env *env, + const struct resolve_vertex *v) +{ + const struct btf_var_secinfo *vsi; + struct btf *btf = env->btf; + u16 i; + + for_each_vsi_from(i, v->next_member, v->t, vsi) { + u32 var_type_id = vsi->type, type_id, type_size = 0; + const struct btf_type *var_type = btf_type_by_id(env->btf, + var_type_id); + if (!var_type || !btf_type_is_var(var_type)) { + btf_verifier_log_vsi(env, v->t, vsi, + "Not a VAR kind member"); + return -EINVAL; + } + + if (!env_type_is_resolve_sink(env, var_type) && + !env_type_is_resolved(env, var_type_id)) { + env_stack_set_next_member(env, i + 1); + return env_stack_push(env, var_type, var_type_id); + } + + type_id = var_type->type; + if (!btf_type_id_size(btf, &type_id, &type_size)) { + btf_verifier_log_vsi(env, v->t, vsi, "Invalid type"); + return -EINVAL; + } + + if (vsi->size < type_size) { + btf_verifier_log_vsi(env, v->t, vsi, "Invalid size"); + return -EINVAL; + } + } + + env_stack_pop_resolved(env, 0, 0); + return 0; +} + +static void btf_datasec_log(struct btf_verifier_env *env, + const struct btf_type *t) +{ + btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t)); +} + +static void btf_datasec_seq_show(const struct btf *btf, + const struct btf_type *t, u32 type_id, + void *data, u8 bits_offset, + struct seq_file *m) +{ + const struct btf_var_secinfo *vsi; + const struct btf_type *var; + u32 i; + + seq_printf(m, "section (\"%s\") = {", __btf_name_by_offset(btf, t->name_off)); + for_each_vsi(i, t, vsi) { + var = btf_type_by_id(btf, vsi->type); + if (i) + seq_puts(m, ","); + btf_type_ops(var)->seq_show(btf, var, vsi->type, + data + vsi->offset, bits_offset, m); + } + seq_puts(m, "}"); +} + +static const struct btf_kind_operations datasec_ops = { + .check_meta = btf_datasec_check_meta, + .resolve = btf_datasec_resolve, + .check_member = btf_df_check_member, + .check_kflag_member = btf_df_check_kflag_member, + .log_details = btf_datasec_log, + .seq_show = btf_datasec_seq_show, +}; + static int btf_func_proto_check(struct btf_verifier_env *env, const struct btf_type *t) { @@ -2542,6 +2913,8 @@ static const struct btf_kind_operations * const kind_ops[NR_BTF_KINDS] = { [BTF_KIND_RESTRICT] = &modifier_ops, [BTF_KIND_FUNC] = &func_ops, [BTF_KIND_FUNC_PROTO] = &func_proto_ops, + [BTF_KIND_VAR] = &var_ops, + [BTF_KIND_DATASEC] = &datasec_ops, }; static s32 btf_check_meta(struct btf_verifier_env *env, @@ -2622,13 +2995,17 @@ static bool btf_resolve_valid(struct btf_verifier_env *env, if (!env_type_is_resolved(env, type_id)) return false; - if (btf_type_is_struct(t)) + if (btf_type_is_struct(t) || btf_type_is_datasec(t)) return !btf->resolved_ids[type_id] && - !btf->resolved_sizes[type_id]; + !btf->resolved_sizes[type_id]; - if (btf_type_is_modifier(t) || btf_type_is_ptr(t)) { + if (btf_type_is_modifier(t) || btf_type_is_ptr(t) || + btf_type_is_var(t)) { t = btf_type_id_resolve(btf, &type_id); - return t && !btf_type_is_modifier(t); + return t && + !btf_type_is_modifier(t) && + !btf_type_is_var(t) && + !btf_type_is_datasec(t); } if (btf_type_is_array(t)) { diff --git a/kernel/bpf/cgroup.c b/kernel/bpf/cgroup.c index 4e807973aa80cebd8a2bc03ff51a9890cbad5dfe..fcde0f7b25854882c42e76f901406cbf4a1718a9 100644 --- a/kernel/bpf/cgroup.c +++ b/kernel/bpf/cgroup.c @@ -11,7 +11,10 @@ #include #include #include +#include #include +#include +#include #include #include #include @@ -701,7 +704,7 @@ int __cgroup_bpf_check_dev_permission(short dev_type, u32 major, u32 minor, EXPORT_SYMBOL(__cgroup_bpf_check_dev_permission); static const struct bpf_func_proto * -cgroup_dev_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) +cgroup_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) { switch (func_id) { case BPF_FUNC_map_lookup_elem: @@ -710,6 +713,12 @@ cgroup_dev_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_map_update_elem_proto; case BPF_FUNC_map_delete_elem: return &bpf_map_delete_elem_proto; + case BPF_FUNC_map_push_elem: + return &bpf_map_push_elem_proto; + case BPF_FUNC_map_pop_elem: + return &bpf_map_pop_elem_proto; + case BPF_FUNC_map_peek_elem: + return &bpf_map_peek_elem_proto; case BPF_FUNC_get_current_uid_gid: return &bpf_get_current_uid_gid_proto; case BPF_FUNC_get_local_storage: @@ -725,6 +734,12 @@ cgroup_dev_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) } } +static const struct bpf_func_proto * +cgroup_dev_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) +{ + return cgroup_base_func_proto(func_id, prog); +} + static bool cgroup_dev_is_valid_access(int off, int size, enum bpf_access_type type, const struct bpf_prog *prog, @@ -762,3 +777,356 @@ const struct bpf_verifier_ops cg_dev_verifier_ops = { .get_func_proto = cgroup_dev_func_proto, .is_valid_access = cgroup_dev_is_valid_access, }; + +/** + * __cgroup_bpf_run_filter_sysctl - Run a program on sysctl + * + * @head: sysctl table header + * @table: sysctl table + * @write: sysctl is being read (= 0) or written (= 1) + * @buf: pointer to buffer passed by user space + * @pcount: value-result argument: value is size of buffer pointed to by @buf, + * result is size of @new_buf if program set new value, initial value + * otherwise + * @ppos: value-result argument: value is position at which read from or write + * to sysctl is happening, result is new position if program overrode it, + * initial value otherwise + * @new_buf: pointer to pointer to new buffer that will be allocated if program + * overrides new value provided by user space on sysctl write + * NOTE: it's caller responsibility to free *new_buf if it was set + * @type: type of program to be executed + * + * Program is run when sysctl is being accessed, either read or written, and + * can allow or deny such access. + * + * This function will return %-EPERM if an attached program is found and + * returned value != 1 during execution. In all other cases 0 is returned. + */ +int __cgroup_bpf_run_filter_sysctl(struct ctl_table_header *head, + struct ctl_table *table, int write, + void __user *buf, size_t *pcount, + loff_t *ppos, void **new_buf, + enum bpf_attach_type type) +{ + struct bpf_sysctl_kern ctx = { + .head = head, + .table = table, + .write = write, + .ppos = ppos, + .cur_val = NULL, + .cur_len = PAGE_SIZE, + .new_val = NULL, + .new_len = 0, + .new_updated = 0, + }; + struct cgroup *cgrp; + int ret; + + ctx.cur_val = kmalloc_track_caller(ctx.cur_len, GFP_KERNEL); + if (ctx.cur_val) { + mm_segment_t old_fs; + loff_t pos = 0; + + old_fs = get_fs(); + set_fs(KERNEL_DS); + if (table->proc_handler(table, 0, (void __user *)ctx.cur_val, + &ctx.cur_len, &pos)) { + /* Let BPF program decide how to proceed. */ + ctx.cur_len = 0; + } + set_fs(old_fs); + } else { + /* Let BPF program decide how to proceed. */ + ctx.cur_len = 0; + } + + if (write && buf && *pcount) { + /* BPF program should be able to override new value with a + * buffer bigger than provided by user. + */ + ctx.new_val = kmalloc_track_caller(PAGE_SIZE, GFP_KERNEL); + ctx.new_len = min_t(size_t, PAGE_SIZE, *pcount); + if (!ctx.new_val || + copy_from_user(ctx.new_val, buf, ctx.new_len)) + /* Let BPF program decide how to proceed. */ + ctx.new_len = 0; + } + + rcu_read_lock(); + cgrp = task_dfl_cgroup(current); + ret = BPF_PROG_RUN_ARRAY(cgrp->bpf.effective[type], &ctx, BPF_PROG_RUN); + rcu_read_unlock(); + + kfree(ctx.cur_val); + + if (ret == 1 && ctx.new_updated) { + *new_buf = ctx.new_val; + *pcount = ctx.new_len; + } else { + kfree(ctx.new_val); + } + + return ret == 1 ? 0 : -EPERM; +} +EXPORT_SYMBOL(__cgroup_bpf_run_filter_sysctl); + +static ssize_t sysctl_cpy_dir(const struct ctl_dir *dir, char **bufp, + size_t *lenp) +{ + ssize_t tmp_ret = 0, ret; + + if (dir->header.parent) { + tmp_ret = sysctl_cpy_dir(dir->header.parent, bufp, lenp); + if (tmp_ret < 0) + return tmp_ret; + } + + ret = strscpy(*bufp, dir->header.ctl_table[0].procname, *lenp); + if (ret < 0) + return ret; + *bufp += ret; + *lenp -= ret; + ret += tmp_ret; + + /* Avoid leading slash. */ + if (!ret) + return ret; + + tmp_ret = strscpy(*bufp, "/", *lenp); + if (tmp_ret < 0) + return tmp_ret; + *bufp += tmp_ret; + *lenp -= tmp_ret; + + return ret + tmp_ret; +} + +BPF_CALL_4(bpf_sysctl_get_name, struct bpf_sysctl_kern *, ctx, char *, buf, + size_t, buf_len, u64, flags) +{ + ssize_t tmp_ret = 0, ret; + + if (!buf) + return -EINVAL; + + if (!(flags & BPF_F_SYSCTL_BASE_NAME)) { + if (!ctx->head) + return -EINVAL; + tmp_ret = sysctl_cpy_dir(ctx->head->parent, &buf, &buf_len); + if (tmp_ret < 0) + return tmp_ret; + } + + ret = strscpy(buf, ctx->table->procname, buf_len); + + return ret < 0 ? ret : tmp_ret + ret; +} + +static const struct bpf_func_proto bpf_sysctl_get_name_proto = { + .func = bpf_sysctl_get_name, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_PTR_TO_CTX, + .arg2_type = ARG_PTR_TO_MEM, + .arg3_type = ARG_CONST_SIZE, + .arg4_type = ARG_ANYTHING, +}; + +static int copy_sysctl_value(char *dst, size_t dst_len, char *src, + size_t src_len) +{ + if (!dst) + return -EINVAL; + + if (!dst_len) + return -E2BIG; + + if (!src || !src_len) { + memset(dst, 0, dst_len); + return -EINVAL; + } + + memcpy(dst, src, min(dst_len, src_len)); + + if (dst_len > src_len) { + memset(dst + src_len, '\0', dst_len - src_len); + return src_len; + } + + dst[dst_len - 1] = '\0'; + + return -E2BIG; +} + +BPF_CALL_3(bpf_sysctl_get_current_value, struct bpf_sysctl_kern *, ctx, + char *, buf, size_t, buf_len) +{ + return copy_sysctl_value(buf, buf_len, ctx->cur_val, ctx->cur_len); +} + +static const struct bpf_func_proto bpf_sysctl_get_current_value_proto = { + .func = bpf_sysctl_get_current_value, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_PTR_TO_CTX, + .arg2_type = ARG_PTR_TO_UNINIT_MEM, + .arg3_type = ARG_CONST_SIZE, +}; + +BPF_CALL_3(bpf_sysctl_get_new_value, struct bpf_sysctl_kern *, ctx, char *, buf, + size_t, buf_len) +{ + if (!ctx->write) { + if (buf && buf_len) + memset(buf, '\0', buf_len); + return -EINVAL; + } + return copy_sysctl_value(buf, buf_len, ctx->new_val, ctx->new_len); +} + +static const struct bpf_func_proto bpf_sysctl_get_new_value_proto = { + .func = bpf_sysctl_get_new_value, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_PTR_TO_CTX, + .arg2_type = ARG_PTR_TO_UNINIT_MEM, + .arg3_type = ARG_CONST_SIZE, +}; + +BPF_CALL_3(bpf_sysctl_set_new_value, struct bpf_sysctl_kern *, ctx, + const char *, buf, size_t, buf_len) +{ + if (!ctx->write || !ctx->new_val || !ctx->new_len || !buf || !buf_len) + return -EINVAL; + + if (buf_len > PAGE_SIZE - 1) + return -E2BIG; + + memcpy(ctx->new_val, buf, buf_len); + ctx->new_len = buf_len; + ctx->new_updated = 1; + + return 0; +} + +static const struct bpf_func_proto bpf_sysctl_set_new_value_proto = { + .func = bpf_sysctl_set_new_value, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_PTR_TO_CTX, + .arg2_type = ARG_PTR_TO_MEM, + .arg3_type = ARG_CONST_SIZE, +}; + +static const struct bpf_func_proto * +sysctl_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) +{ + switch (func_id) { + case BPF_FUNC_strtol: + return &bpf_strtol_proto; + case BPF_FUNC_strtoul: + return &bpf_strtoul_proto; + case BPF_FUNC_sysctl_get_name: + return &bpf_sysctl_get_name_proto; + case BPF_FUNC_sysctl_get_current_value: + return &bpf_sysctl_get_current_value_proto; + case BPF_FUNC_sysctl_get_new_value: + return &bpf_sysctl_get_new_value_proto; + case BPF_FUNC_sysctl_set_new_value: + return &bpf_sysctl_set_new_value_proto; + default: + return cgroup_base_func_proto(func_id, prog); + } +} + +static bool sysctl_is_valid_access(int off, int size, enum bpf_access_type type, + const struct bpf_prog *prog, + struct bpf_insn_access_aux *info) +{ + const int size_default = sizeof(__u32); + + if (off < 0 || off + size > sizeof(struct bpf_sysctl) || off % size) + return false; + + switch (off) { + case offsetof(struct bpf_sysctl, write): + if (type != BPF_READ) + return false; + bpf_ctx_record_field_size(info, size_default); + return bpf_ctx_narrow_access_ok(off, size, size_default); + case offsetof(struct bpf_sysctl, file_pos): + if (type == BPF_READ) { + bpf_ctx_record_field_size(info, size_default); + return bpf_ctx_narrow_access_ok(off, size, size_default); + } else { + return size == size_default; + } + default: + return false; + } +} + +static u32 sysctl_convert_ctx_access(enum bpf_access_type type, + const struct bpf_insn *si, + struct bpf_insn *insn_buf, + struct bpf_prog *prog, u32 *target_size) +{ + struct bpf_insn *insn = insn_buf; + + switch (si->off) { + case offsetof(struct bpf_sysctl, write): + *insn++ = BPF_LDX_MEM( + BPF_SIZE(si->code), si->dst_reg, si->src_reg, + bpf_target_off(struct bpf_sysctl_kern, write, + FIELD_SIZEOF(struct bpf_sysctl_kern, + write), + target_size)); + break; + case offsetof(struct bpf_sysctl, file_pos): + /* ppos is a pointer so it should be accessed via indirect + * loads and stores. Also for stores additional temporary + * register is used since neither src_reg nor dst_reg can be + * overridden. + */ + if (type == BPF_WRITE) { + int treg = BPF_REG_9; + + if (si->src_reg == treg || si->dst_reg == treg) + --treg; + if (si->src_reg == treg || si->dst_reg == treg) + --treg; + *insn++ = BPF_STX_MEM( + BPF_DW, si->dst_reg, treg, + offsetof(struct bpf_sysctl_kern, tmp_reg)); + *insn++ = BPF_LDX_MEM( + BPF_FIELD_SIZEOF(struct bpf_sysctl_kern, ppos), + treg, si->dst_reg, + offsetof(struct bpf_sysctl_kern, ppos)); + *insn++ = BPF_STX_MEM( + BPF_SIZEOF(u32), treg, si->src_reg, 0); + *insn++ = BPF_LDX_MEM( + BPF_DW, treg, si->dst_reg, + offsetof(struct bpf_sysctl_kern, tmp_reg)); + } else { + *insn++ = BPF_LDX_MEM( + BPF_FIELD_SIZEOF(struct bpf_sysctl_kern, ppos), + si->dst_reg, si->src_reg, + offsetof(struct bpf_sysctl_kern, ppos)); + *insn++ = BPF_LDX_MEM( + BPF_SIZE(si->code), si->dst_reg, si->dst_reg, 0); + } + *target_size = sizeof(u32); + break; + } + + return insn - insn_buf; +} + +const struct bpf_verifier_ops cg_sysctl_verifier_ops = { + .get_func_proto = sysctl_func_proto, + .is_valid_access = sysctl_is_valid_access, + .convert_ctx_access = sysctl_convert_ctx_access, +}; + +const struct bpf_prog_ops cg_sysctl_prog_ops = { +}; diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index c605397c79f03050a22fa0e03b170e5c6a55f5a6..3ba56e73c90e87db4dde46ca4ed26b147537ec96 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -292,7 +292,8 @@ int bpf_prog_calc_tag(struct bpf_prog *fp) dst[i] = fp->insnsi[i]; if (!was_ld_map && dst[i].code == (BPF_LD | BPF_IMM | BPF_DW) && - dst[i].src_reg == BPF_PSEUDO_MAP_FD) { + (dst[i].src_reg == BPF_PSEUDO_MAP_FD || + dst[i].src_reg == BPF_PSEUDO_MAP_VALUE)) { was_ld_map = true; dst[i].imm = 0; } else if (was_ld_map && @@ -438,6 +439,7 @@ struct bpf_prog *bpf_patch_insn_single(struct bpf_prog *prog, u32 off, u32 insn_adj_cnt, insn_rest, insn_delta = len - 1; const u32 cnt_max = S16_MAX; struct bpf_prog *prog_adj; + int err; /* Since our patchlet doesn't expand the image, we're done. */ if (insn_delta == 0) { @@ -453,8 +455,8 @@ struct bpf_prog *bpf_patch_insn_single(struct bpf_prog *prog, u32 off, * we afterwards may not fail anymore. */ if (insn_adj_cnt > cnt_max && - bpf_adj_branches(prog, off, off + 1, off + len, true)) - return NULL; + (err = bpf_adj_branches(prog, off, off + 1, off + len, true))) + return ERR_PTR(err); /* Several new instructions need to be inserted. Make room * for them. Likely, there's no need for a new allocation as @@ -463,7 +465,7 @@ struct bpf_prog *bpf_patch_insn_single(struct bpf_prog *prog, u32 off, prog_adj = bpf_prog_realloc(prog, bpf_prog_size(insn_adj_cnt), GFP_USER); if (!prog_adj) - return NULL; + return ERR_PTR(-ENOMEM); prog_adj->len = insn_adj_cnt; @@ -1095,13 +1097,13 @@ struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *prog) continue; tmp = bpf_patch_insn_single(clone, i, insn_buff, rewritten); - if (!tmp) { + if (IS_ERR(tmp)) { /* Patching may have repointed aux->prog during * realloc from the original one, so we need to * fix it up here on error. */ bpf_jit_prog_release_other(prog, clone); - return ERR_PTR(-ENOMEM); + return tmp; } clone = tmp; diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index 3c18260403dde1df951448c600b6ef9ac61f5635..cf727d77c6c6be3cfb3ae656c7061aa0709ac207 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -160,12 +160,12 @@ static void cpu_map_kthread_stop(struct work_struct *work) } static struct sk_buff *cpu_map_build_skb(struct bpf_cpu_map_entry *rcpu, - struct xdp_frame *xdpf) + struct xdp_frame *xdpf, + struct sk_buff *skb) { unsigned int hard_start_headroom; unsigned int frame_size; void *pkt_data_start; - struct sk_buff *skb; /* Part of headroom was reserved to xdpf */ hard_start_headroom = sizeof(struct xdp_frame) + xdpf->headroom; @@ -191,8 +191,8 @@ static struct sk_buff *cpu_map_build_skb(struct bpf_cpu_map_entry *rcpu, SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); pkt_data_start = xdpf->data - hard_start_headroom; - skb = build_skb(pkt_data_start, frame_size); - if (!skb) + skb = build_skb_around(skb, pkt_data_start, frame_size); + if (unlikely(!skb)) return NULL; skb_reserve(skb, hard_start_headroom); @@ -240,6 +240,8 @@ static void put_cpu_map_entry(struct bpf_cpu_map_entry *rcpu) } } +#define CPUMAP_BATCH 8 + static int cpu_map_kthread_run(void *data) { struct bpf_cpu_map_entry *rcpu = data; @@ -252,8 +254,11 @@ static int cpu_map_kthread_run(void *data) * kthread_stop signal until queue is empty. */ while (!kthread_should_stop() || !__ptr_ring_empty(rcpu->queue)) { - unsigned int processed = 0, drops = 0, sched = 0; - struct xdp_frame *xdpf; + unsigned int drops = 0, sched = 0; + void *frames[CPUMAP_BATCH]; + void *skbs[CPUMAP_BATCH]; + gfp_t gfp = __GFP_ZERO | GFP_ATOMIC; + int i, n, m; /* Release CPU reschedule checks */ if (__ptr_ring_empty(rcpu->queue)) { @@ -269,18 +274,38 @@ static int cpu_map_kthread_run(void *data) sched = cond_resched(); } - /* Process packets in rcpu->queue */ - local_bh_disable(); /* * The bpf_cpu_map_entry is single consumer, with this * kthread CPU pinned. Lockless access to ptr_ring * consume side valid as no-resize allowed of queue. */ - while ((xdpf = __ptr_ring_consume(rcpu->queue))) { - struct sk_buff *skb; + n = ptr_ring_consume_batched(rcpu->queue, frames, CPUMAP_BATCH); + + for (i = 0; i < n; i++) { + void *f = frames[i]; + struct page *page = virt_to_page(f); + + /* Bring struct page memory area to curr CPU. Read by + * build_skb_around via page_is_pfmemalloc(), and when + * freed written by page_frag_free call. + */ + prefetchw(page); + } + + m = kmem_cache_alloc_bulk(skbuff_head_cache, gfp, n, skbs); + if (unlikely(m == 0)) { + for (i = 0; i < n; i++) + skbs[i] = NULL; /* effect: xdp_return_frame */ + drops = n; + } + + local_bh_disable(); + for (i = 0; i < n; i++) { + struct xdp_frame *xdpf = frames[i]; + struct sk_buff *skb = skbs[i]; int ret; - skb = cpu_map_build_skb(rcpu, xdpf); + skb = cpu_map_build_skb(rcpu, xdpf, skb); if (!skb) { xdp_return_frame(xdpf); continue; @@ -290,13 +315,9 @@ static int cpu_map_kthread_run(void *data) ret = netif_receive_skb_core(skb); if (ret == NET_RX_DROP) drops++; - - /* Limit BH-disable period */ - if (++processed == 8) - break; } /* Feedback loop via tracepoint */ - trace_xdp_cpumap_kthread(rcpu->map_id, processed, drops, sched); + trace_xdp_cpumap_kthread(rcpu->map_id, n, drops, sched); local_bh_enable(); /* resched point, may call do_softirq() */ } diff --git a/kernel/bpf/disasm.c b/kernel/bpf/disasm.c index de73f55e42fd4922ae04c6d4384b93937ef91cff..d9ce383c0f9ce24d9be2546cad822bcb52d0e719 100644 --- a/kernel/bpf/disasm.c +++ b/kernel/bpf/disasm.c @@ -205,10 +205,11 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, * part of the ldimm64 insn is accessible. */ u64 imm = ((u64)(insn + 1)->imm << 32) | (u32)insn->imm; - bool map_ptr = insn->src_reg == BPF_PSEUDO_MAP_FD; + bool is_ptr = insn->src_reg == BPF_PSEUDO_MAP_FD || + insn->src_reg == BPF_PSEUDO_MAP_VALUE; char tmp[64]; - if (map_ptr && !allow_ptr_leaks) + if (is_ptr && !allow_ptr_leaks) imm = 0; verbose(cbs->private_data, "(%02x) r%d = %s\n", diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index fed15cf94dca65431f7ee233514d80d24cb5420b..192d32e77db3f05c9507ef43592e70d6db1df14b 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -23,7 +23,7 @@ #define HTAB_CREATE_FLAG_MASK \ (BPF_F_NO_PREALLOC | BPF_F_NO_COMMON_LRU | BPF_F_NUMA_NODE | \ - BPF_F_RDONLY | BPF_F_WRONLY | BPF_F_ZERO_SEED) + BPF_F_ACCESS_MASK | BPF_F_ZERO_SEED) struct bucket { struct hlist_nulls_head head; @@ -262,8 +262,8 @@ static int htab_map_alloc_check(union bpf_attr *attr) /* Guard against local DoS, and discourage production use. */ return -EPERM; - if (attr->map_flags & ~HTAB_CREATE_FLAG_MASK) - /* reserved bits should not be used */ + if (attr->map_flags & ~HTAB_CREATE_FLAG_MASK || + !bpf_map_flags_access_ok(attr->map_flags)) return -EINVAL; if (!lru && percpu_lru) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index a411fc17d265e1c7809d263df74d8881ff0d006b..4266ffde07caed42a2c704355665a4f5f7948007 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -18,6 +18,9 @@ #include #include #include +#include + +#include "../../lib/kstrtox.h" /* If kernel subsystem is allowing eBPF programs to call this function, * inside its own verifier_ops->get_func_proto() callback it should return @@ -363,4 +366,132 @@ const struct bpf_func_proto bpf_get_local_storage_proto = { .arg2_type = ARG_ANYTHING, }; #endif + +#define BPF_STRTOX_BASE_MASK 0x1F + +static int __bpf_strtoull(const char *buf, size_t buf_len, u64 flags, + unsigned long long *res, bool *is_negative) +{ + unsigned int base = flags & BPF_STRTOX_BASE_MASK; + const char *cur_buf = buf; + size_t cur_len = buf_len; + unsigned int consumed; + size_t val_len; + char str[64]; + + if (!buf || !buf_len || !res || !is_negative) + return -EINVAL; + + if (base != 0 && base != 8 && base != 10 && base != 16) + return -EINVAL; + + if (flags & ~BPF_STRTOX_BASE_MASK) + return -EINVAL; + + while (cur_buf < buf + buf_len && isspace(*cur_buf)) + ++cur_buf; + + *is_negative = (cur_buf < buf + buf_len && *cur_buf == '-'); + if (*is_negative) + ++cur_buf; + + consumed = cur_buf - buf; + cur_len -= consumed; + if (!cur_len) + return -EINVAL; + + cur_len = min(cur_len, sizeof(str) - 1); + memcpy(str, cur_buf, cur_len); + str[cur_len] = '\0'; + cur_buf = str; + + cur_buf = _parse_integer_fixup_radix(cur_buf, &base); + val_len = _parse_integer(cur_buf, base, res); + + if (val_len & KSTRTOX_OVERFLOW) + return -ERANGE; + + if (val_len == 0) + return -EINVAL; + + cur_buf += val_len; + consumed += cur_buf - str; + + return consumed; +} + +static int __bpf_strtoll(const char *buf, size_t buf_len, u64 flags, + long long *res) +{ + unsigned long long _res; + bool is_negative; + int err; + + err = __bpf_strtoull(buf, buf_len, flags, &_res, &is_negative); + if (err < 0) + return err; + if (is_negative) { + if ((long long)-_res > 0) + return -ERANGE; + *res = -_res; + } else { + if ((long long)_res < 0) + return -ERANGE; + *res = _res; + } + return err; +} + +BPF_CALL_4(bpf_strtol, const char *, buf, size_t, buf_len, u64, flags, + long *, res) +{ + long long _res; + int err; + + err = __bpf_strtoll(buf, buf_len, flags, &_res); + if (err < 0) + return err; + if (_res != (long)_res) + return -ERANGE; + *res = _res; + return err; +} + +const struct bpf_func_proto bpf_strtol_proto = { + .func = bpf_strtol, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_PTR_TO_MEM, + .arg2_type = ARG_CONST_SIZE, + .arg3_type = ARG_ANYTHING, + .arg4_type = ARG_PTR_TO_LONG, +}; + +BPF_CALL_4(bpf_strtoul, const char *, buf, size_t, buf_len, u64, flags, + unsigned long *, res) +{ + unsigned long long _res; + bool is_negative; + int err; + + err = __bpf_strtoull(buf, buf_len, flags, &_res, &is_negative); + if (err < 0) + return err; + if (is_negative) + return -EINVAL; + if (_res != (unsigned long)_res) + return -ERANGE; + *res = _res; + return err; +} + +const struct bpf_func_proto bpf_strtoul_proto = { + .func = bpf_strtoul, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_PTR_TO_MEM, + .arg2_type = ARG_CONST_SIZE, + .arg3_type = ARG_ANYTHING, + .arg4_type = ARG_PTR_TO_LONG, +}; #endif diff --git a/kernel/bpf/local_storage.c b/kernel/bpf/local_storage.c index 6b572e2de7fbee76434bb1a3d723394d7e0b3679..980e8f1f6cb5d0d53889714a953ca1a2abaa0462 100644 --- a/kernel/bpf/local_storage.c +++ b/kernel/bpf/local_storage.c @@ -14,7 +14,7 @@ DEFINE_PER_CPU(struct bpf_cgroup_storage*, bpf_cgroup_storage[MAX_BPF_CGROUP_STO #ifdef CONFIG_CGROUP_BPF #define LOCAL_STORAGE_CREATE_FLAG_MASK \ - (BPF_F_NUMA_NODE | BPF_F_RDONLY | BPF_F_WRONLY) + (BPF_F_NUMA_NODE | BPF_F_ACCESS_MASK) struct bpf_cgroup_storage_map { struct bpf_map map; @@ -282,8 +282,8 @@ static struct bpf_map *cgroup_storage_map_alloc(union bpf_attr *attr) if (attr->value_size > PAGE_SIZE) return ERR_PTR(-E2BIG); - if (attr->map_flags & ~LOCAL_STORAGE_CREATE_FLAG_MASK) - /* reserved bits should not be used */ + if (attr->map_flags & ~LOCAL_STORAGE_CREATE_FLAG_MASK || + !bpf_map_flags_access_ok(attr->map_flags)) return ERR_PTR(-EINVAL); if (attr->max_entries) diff --git a/kernel/bpf/lpm_trie.c b/kernel/bpf/lpm_trie.c index 93a5cbbde421c346e72b10cd56e04a13888d9e7b..e61630c2e50b28a3e342c4684b9be8e8141784a9 100644 --- a/kernel/bpf/lpm_trie.c +++ b/kernel/bpf/lpm_trie.c @@ -538,7 +538,7 @@ out: #define LPM_KEY_SIZE_MIN LPM_KEY_SIZE(LPM_DATA_SIZE_MIN) #define LPM_CREATE_FLAG_MASK (BPF_F_NO_PREALLOC | BPF_F_NUMA_NODE | \ - BPF_F_RDONLY | BPF_F_WRONLY) + BPF_F_ACCESS_MASK) static struct bpf_map *trie_alloc(union bpf_attr *attr) { @@ -553,6 +553,7 @@ static struct bpf_map *trie_alloc(union bpf_attr *attr) if (attr->max_entries == 0 || !(attr->map_flags & BPF_F_NO_PREALLOC) || attr->map_flags & ~LPM_CREATE_FLAG_MASK || + !bpf_map_flags_access_ok(attr->map_flags) || attr->key_size < LPM_KEY_SIZE_MIN || attr->key_size > LPM_KEY_SIZE_MAX || attr->value_size < LPM_VAL_SIZE_MIN || diff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c index b384ea9f3254987f1caa16dff0780900720d93ca..0b140d2368896d2d154bc0d0fa90b82aa238061c 100644 --- a/kernel/bpf/queue_stack_maps.c +++ b/kernel/bpf/queue_stack_maps.c @@ -11,8 +11,7 @@ #include "percpu_freelist.h" #define QUEUE_STACK_CREATE_FLAG_MASK \ - (BPF_F_NUMA_NODE | BPF_F_RDONLY | BPF_F_WRONLY) - + (BPF_F_NUMA_NODE | BPF_F_ACCESS_MASK) struct bpf_queue_stack { struct bpf_map map; @@ -52,7 +51,8 @@ static int queue_stack_map_alloc_check(union bpf_attr *attr) /* check sanity of attributes */ if (attr->max_entries == 0 || attr->key_size != 0 || attr->value_size == 0 || - attr->map_flags & ~QUEUE_STACK_CREATE_FLAG_MASK) + attr->map_flags & ~QUEUE_STACK_CREATE_FLAG_MASK || + !bpf_map_flags_access_ok(attr->map_flags)) return -EINVAL; if (attr->value_size > KMALLOC_MAX_SIZE) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index afca36f53c492718820ecacdb588af585dbb50e4..ad3ccf82f31dd01480e662eb489baa99925d688b 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -166,13 +166,25 @@ void bpf_map_area_free(void *area) kvfree(area); } +static u32 bpf_map_flags_retain_permanent(u32 flags) +{ + /* Some map creation flags are not tied to the map object but + * rather to the map fd instead, so they have no meaning upon + * map object inspection since multiple file descriptors with + * different (access) properties can exist here. Thus, given + * this has zero meaning for the map itself, lets clear these + * from here. + */ + return flags & ~(BPF_F_RDONLY | BPF_F_WRONLY); +} + void bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr) { map->map_type = attr->map_type; map->key_size = attr->key_size; map->value_size = attr->value_size; map->max_entries = attr->max_entries; - map->map_flags = attr->map_flags; + map->map_flags = bpf_map_flags_retain_permanent(attr->map_flags); map->numa_node = bpf_map_attr_numa_node(attr); } @@ -343,6 +355,18 @@ static int bpf_map_release(struct inode *inode, struct file *filp) return 0; } +static fmode_t map_get_sys_perms(struct bpf_map *map, struct fd f) +{ + fmode_t mode = f.file->f_mode; + + /* Our file permissions may have been overridden by global + * map permissions facing syscall side. + */ + if (READ_ONCE(map->frozen)) + mode &= ~FMODE_CAN_WRITE; + return mode; +} + #ifdef CONFIG_PROC_FS static void bpf_map_show_fdinfo(struct seq_file *m, struct file *filp) { @@ -364,14 +388,16 @@ static void bpf_map_show_fdinfo(struct seq_file *m, struct file *filp) "max_entries:\t%u\n" "map_flags:\t%#x\n" "memlock:\t%llu\n" - "map_id:\t%u\n", + "map_id:\t%u\n" + "frozen:\t%u\n", map->map_type, map->key_size, map->value_size, map->max_entries, map->map_flags, map->pages * 1ULL << PAGE_SHIFT, - map->id); + map->id, + READ_ONCE(map->frozen)); if (owner_prog_type) { seq_printf(m, "owner_prog_type:\t%u\n", @@ -448,10 +474,10 @@ static int bpf_obj_name_cpy(char *dst, const char *src) const char *end = src + BPF_OBJ_NAME_LEN; memset(dst, 0, BPF_OBJ_NAME_LEN); - - /* Copy all isalnum() and '_' char */ + /* Copy all isalnum(), '_' and '.' chars. */ while (src < end && *src) { - if (!isalnum(*src) && *src != '_') + if (!isalnum(*src) && + *src != '_' && *src != '.') return -EINVAL; *dst++ = *src++; } @@ -478,9 +504,16 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf, u32 key_size, value_size; int ret = 0; - key_type = btf_type_id_size(btf, &btf_key_id, &key_size); - if (!key_type || key_size != map->key_size) - return -EINVAL; + /* Some maps allow key to be unspecified. */ + if (btf_key_id) { + key_type = btf_type_id_size(btf, &btf_key_id, &key_size); + if (!key_type || key_size != map->key_size) + return -EINVAL; + } else { + key_type = btf_type_by_id(btf, 0); + if (!map->ops->map_check_btf) + return -EINVAL; + } value_type = btf_type_id_size(btf, &btf_value_id, &value_size); if (!value_type || value_size != map->value_size) @@ -489,9 +522,12 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf, map->spin_lock_off = btf_find_spin_lock(btf, value_type); if (map_value_has_spin_lock(map)) { + if (map->map_flags & BPF_F_RDONLY_PROG) + return -EACCES; if (map->map_type != BPF_MAP_TYPE_HASH && map->map_type != BPF_MAP_TYPE_ARRAY && - map->map_type != BPF_MAP_TYPE_CGROUP_STORAGE) + map->map_type != BPF_MAP_TYPE_CGROUP_STORAGE && + map->map_type != BPF_MAP_TYPE_SK_STORAGE) return -ENOTSUPP; if (map->spin_lock_off + sizeof(struct bpf_spin_lock) > map->value_size) { @@ -545,7 +581,7 @@ static int map_create(union bpf_attr *attr) if (attr->btf_key_type_id || attr->btf_value_type_id) { struct btf *btf; - if (!attr->btf_key_type_id || !attr->btf_value_type_id) { + if (!attr->btf_value_type_id) { err = -EINVAL; goto free_map_nouncharge; } @@ -713,8 +749,7 @@ static int map_lookup_elem(union bpf_attr *attr) map = __bpf_map_get(f); if (IS_ERR(map)) return PTR_ERR(map); - - if (!(f.file->f_mode & FMODE_CAN_READ)) { + if (!(map_get_sys_perms(map, f) & FMODE_CAN_READ)) { err = -EPERM; goto err_put; } @@ -843,8 +878,7 @@ static int map_update_elem(union bpf_attr *attr) map = __bpf_map_get(f); if (IS_ERR(map)) return PTR_ERR(map); - - if (!(f.file->f_mode & FMODE_CAN_WRITE)) { + if (!(map_get_sys_perms(map, f) & FMODE_CAN_WRITE)) { err = -EPERM; goto err_put; } @@ -955,8 +989,7 @@ static int map_delete_elem(union bpf_attr *attr) map = __bpf_map_get(f); if (IS_ERR(map)) return PTR_ERR(map); - - if (!(f.file->f_mode & FMODE_CAN_WRITE)) { + if (!(map_get_sys_perms(map, f) & FMODE_CAN_WRITE)) { err = -EPERM; goto err_put; } @@ -1007,8 +1040,7 @@ static int map_get_next_key(union bpf_attr *attr) map = __bpf_map_get(f); if (IS_ERR(map)) return PTR_ERR(map); - - if (!(f.file->f_mode & FMODE_CAN_READ)) { + if (!(map_get_sys_perms(map, f) & FMODE_CAN_READ)) { err = -EPERM; goto err_put; } @@ -1075,8 +1107,7 @@ static int map_lookup_and_delete_elem(union bpf_attr *attr) map = __bpf_map_get(f); if (IS_ERR(map)) return PTR_ERR(map); - - if (!(f.file->f_mode & FMODE_CAN_WRITE)) { + if (!(map_get_sys_perms(map, f) & FMODE_CAN_WRITE)) { err = -EPERM; goto err_put; } @@ -1118,6 +1149,36 @@ err_put: return err; } +#define BPF_MAP_FREEZE_LAST_FIELD map_fd + +static int map_freeze(const union bpf_attr *attr) +{ + int err = 0, ufd = attr->map_fd; + struct bpf_map *map; + struct fd f; + + if (CHECK_ATTR(BPF_MAP_FREEZE)) + return -EINVAL; + + f = fdget(ufd); + map = __bpf_map_get(f); + if (IS_ERR(map)) + return PTR_ERR(map); + if (READ_ONCE(map->frozen)) { + err = -EBUSY; + goto err_put; + } + if (!capable(CAP_SYS_ADMIN)) { + err = -EPERM; + goto err_put; + } + + WRITE_ONCE(map->frozen, true); +err_put: + fdput(f); + return err; +} + static const struct bpf_prog_ops * const bpf_prog_types[] = { #define BPF_PROG_TYPE(_id, _name) \ [_id] = & _name ## _prog_ops, @@ -1557,7 +1618,8 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr) /* eBPF programs must be GPL compatible to use GPL-ed functions */ is_gpl = license_is_gpl_compatible(license); - if (attr->insn_cnt == 0 || attr->insn_cnt > BPF_MAXINSNS) + if (attr->insn_cnt == 0 || + attr->insn_cnt > (capable(CAP_SYS_ADMIN) ? BPF_COMPLEXITY_LIMIT_INSNS : BPF_MAXINSNS)) return -E2BIG; if (type != BPF_PROG_TYPE_SOCKET_FILTER && type != BPF_PROG_TYPE_CGROUP_SKB && @@ -1728,12 +1790,16 @@ static int bpf_raw_tracepoint_open(const union bpf_attr *attr) } raw_tp->btp = btp; - prog = bpf_prog_get_type(attr->raw_tracepoint.prog_fd, - BPF_PROG_TYPE_RAW_TRACEPOINT); + prog = bpf_prog_get(attr->raw_tracepoint.prog_fd); if (IS_ERR(prog)) { err = PTR_ERR(prog); goto out_free_tp; } + if (prog->type != BPF_PROG_TYPE_RAW_TRACEPOINT && + prog->type != BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE) { + err = -EINVAL; + goto out_put_prog; + } err = bpf_probe_register(raw_tp->btp, prog); if (err) @@ -1827,6 +1893,9 @@ static int bpf_prog_attach(const union bpf_attr *attr) case BPF_FLOW_DISSECTOR: ptype = BPF_PROG_TYPE_FLOW_DISSECTOR; break; + case BPF_CGROUP_SYSCTL: + ptype = BPF_PROG_TYPE_CGROUP_SYSCTL; + break; default: return -EINVAL; } @@ -1905,6 +1974,9 @@ static int bpf_prog_detach(const union bpf_attr *attr) return lirc_prog_detach(attr); case BPF_FLOW_DISSECTOR: return skb_flow_dissector_bpf_prog_detach(attr); + case BPF_CGROUP_SYSCTL: + ptype = BPF_PROG_TYPE_CGROUP_SYSCTL; + break; default: return -EINVAL; } @@ -1938,9 +2010,12 @@ static int bpf_prog_query(const union bpf_attr *attr, case BPF_CGROUP_UDP6_SENDMSG: case BPF_CGROUP_SOCK_OPS: case BPF_CGROUP_DEVICE: + case BPF_CGROUP_SYSCTL: break; case BPF_LIRC_MODE2: return lirc_prog_query(attr, uattr); + case BPF_FLOW_DISSECTOR: + return skb_flow_dissector_prog_query(attr, uattr); default: return -EINVAL; } @@ -1948,7 +2023,7 @@ static int bpf_prog_query(const union bpf_attr *attr, return cgroup_bpf_prog_query(attr, uattr); } -#define BPF_PROG_TEST_RUN_LAST_FIELD test.duration +#define BPF_PROG_TEST_RUN_LAST_FIELD test.ctx_out static int bpf_prog_test_run(const union bpf_attr *attr, union bpf_attr __user *uattr) @@ -1961,6 +2036,14 @@ static int bpf_prog_test_run(const union bpf_attr *attr, if (CHECK_ATTR(BPF_PROG_TEST_RUN)) return -EINVAL; + if ((attr->test.ctx_size_in && !attr->test.ctx_in) || + (!attr->test.ctx_size_in && attr->test.ctx_in)) + return -EINVAL; + + if ((attr->test.ctx_size_out && !attr->test.ctx_out) || + (!attr->test.ctx_size_out && attr->test.ctx_out)) + return -EINVAL; + prog = bpf_prog_get(attr->test.prog_fd); if (IS_ERR(prog)) return PTR_ERR(prog); @@ -2071,13 +2154,26 @@ static int bpf_map_get_fd_by_id(const union bpf_attr *attr) } static const struct bpf_map *bpf_map_from_imm(const struct bpf_prog *prog, - unsigned long addr) + unsigned long addr, u32 *off, + u32 *type) { + const struct bpf_map *map; int i; - for (i = 0; i < prog->aux->used_map_cnt; i++) - if (prog->aux->used_maps[i] == (void *)addr) - return prog->aux->used_maps[i]; + for (i = 0, *off = 0; i < prog->aux->used_map_cnt; i++) { + map = prog->aux->used_maps[i]; + if (map == (void *)addr) { + *type = BPF_PSEUDO_MAP_FD; + return map; + } + if (!map->ops->map_direct_value_meta) + continue; + if (!map->ops->map_direct_value_meta(map, addr, off)) { + *type = BPF_PSEUDO_MAP_VALUE; + return map; + } + } + return NULL; } @@ -2085,6 +2181,7 @@ static struct bpf_insn *bpf_insn_prepare_dump(const struct bpf_prog *prog) { const struct bpf_map *map; struct bpf_insn *insns; + u32 off, type; u64 imm; int i; @@ -2112,11 +2209,11 @@ static struct bpf_insn *bpf_insn_prepare_dump(const struct bpf_prog *prog) continue; imm = ((u64)insns[i + 1].imm << 32) | (u32)insns[i].imm; - map = bpf_map_from_imm(prog, imm); + map = bpf_map_from_imm(prog, imm, &off, &type); if (map) { - insns[i].src_reg = BPF_PSEUDO_MAP_FD; + insns[i].src_reg = type; insns[i].imm = map->id; - insns[i + 1].imm = 0; + insns[i + 1].imm = off; continue; } } @@ -2706,6 +2803,9 @@ SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, siz case BPF_MAP_GET_NEXT_KEY: err = map_get_next_key(&attr); break; + case BPF_MAP_FREEZE: + err = map_freeze(&attr); + break; case BPF_PROG_LOAD: err = bpf_prog_load(&attr, uattr); break; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 09d5d972c9ff20c9fe69ca4ffbbbb185a998b56d..7b05e8938d5cc69192b768a78108223d8322010c 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -176,7 +176,6 @@ struct bpf_verifier_stack_elem { struct bpf_verifier_stack_elem *next; }; -#define BPF_COMPLEXITY_LIMIT_INSNS 131072 #define BPF_COMPLEXITY_LIMIT_STACK 1024 #define BPF_COMPLEXITY_LIMIT_STATES 64 @@ -377,7 +376,8 @@ static bool is_release_function(enum bpf_func_id func_id) static bool is_acquire_function(enum bpf_func_id func_id) { return func_id == BPF_FUNC_sk_lookup_tcp || - func_id == BPF_FUNC_sk_lookup_udp; + func_id == BPF_FUNC_sk_lookup_udp || + func_id == BPF_FUNC_skc_lookup_tcp; } static bool is_ptr_cast_function(enum bpf_func_id func_id) @@ -405,6 +405,7 @@ static const char * const reg_type_str[] = { [PTR_TO_SOCK_COMMON_OR_NULL] = "sock_common_or_null", [PTR_TO_TCP_SOCK] = "tcp_sock", [PTR_TO_TCP_SOCK_OR_NULL] = "tcp_sock_or_null", + [PTR_TO_TP_BUFFER] = "tp_buffer", }; static char slot_type_char[] = { @@ -1091,7 +1092,7 @@ static int check_subprogs(struct bpf_verifier_env *env) */ subprog[env->subprog_cnt].start = insn_cnt; - if (env->log.level > 1) + if (env->log.level & BPF_LOG_LEVEL2) for (i = 0; i < env->subprog_cnt; i++) verbose(env, "func#%d @%d\n", i, subprog[i].start); @@ -1138,6 +1139,7 @@ static int mark_reg_read(struct bpf_verifier_env *env, struct bpf_reg_state *parent) { bool writes = parent == state->parent; /* Observe write marks */ + int cnt = 0; while (parent) { /* if read wasn't screened by an earlier write ... */ @@ -1149,12 +1151,25 @@ static int mark_reg_read(struct bpf_verifier_env *env, parent->var_off.value, parent->off); return -EFAULT; } + if (parent->live & REG_LIVE_READ) + /* The parentage chain never changes and + * this parent was already marked as LIVE_READ. + * There is no need to keep walking the chain again and + * keep re-marking all parents as LIVE_READ. + * This case happens when the same register is read + * multiple times without writes into it in-between. + */ + break; /* ... then we depend on parent's value */ parent->live |= REG_LIVE_READ; state = parent; parent = state->parent; writes = true; + cnt++; } + + if (env->longest_mark_read_walk < cnt) + env->longest_mark_read_walk = cnt; return 0; } @@ -1163,30 +1178,32 @@ static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, { struct bpf_verifier_state *vstate = env->cur_state; struct bpf_func_state *state = vstate->frame[vstate->curframe]; - struct bpf_reg_state *regs = state->regs; + struct bpf_reg_state *reg, *regs = state->regs; if (regno >= MAX_BPF_REG) { verbose(env, "R%d is invalid\n", regno); return -EINVAL; } + reg = ®s[regno]; if (t == SRC_OP) { /* check whether register used as source operand can be read */ - if (regs[regno].type == NOT_INIT) { + if (reg->type == NOT_INIT) { verbose(env, "R%d !read_ok\n", regno); return -EACCES; } /* We don't need to worry about FP liveness because it's read-only */ - if (regno != BPF_REG_FP) - return mark_reg_read(env, ®s[regno], - regs[regno].parent); + if (regno == BPF_REG_FP) + return 0; + + return mark_reg_read(env, reg, reg->parent); } else { /* check whether register used as dest operand can be written to */ if (regno == BPF_REG_FP) { verbose(env, "frame pointer is read only\n"); return -EACCES; } - regs[regno].live |= REG_LIVE_WRITTEN; + reg->live |= REG_LIVE_WRITTEN; if (t == DST_OP) mark_reg_unknown(env, regs, regno); } @@ -1412,7 +1429,7 @@ static int check_stack_access(struct bpf_verifier_env *env, char tn_buf[48]; tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); - verbose(env, "variable stack access var_off=%s off=%d size=%d", + verbose(env, "variable stack access var_off=%s off=%d size=%d\n", tn_buf, off, size); return -EACCES; } @@ -1425,6 +1442,28 @@ static int check_stack_access(struct bpf_verifier_env *env, return 0; } +static int check_map_access_type(struct bpf_verifier_env *env, u32 regno, + int off, int size, enum bpf_access_type type) +{ + struct bpf_reg_state *regs = cur_regs(env); + struct bpf_map *map = regs[regno].map_ptr; + u32 cap = bpf_map_flags_to_cap(map); + + if (type == BPF_WRITE && !(cap & BPF_MAP_CAN_WRITE)) { + verbose(env, "write into map forbidden, value_size=%d off=%d size=%d\n", + map->value_size, off, size); + return -EACCES; + } + + if (type == BPF_READ && !(cap & BPF_MAP_CAN_READ)) { + verbose(env, "read from map forbidden, value_size=%d off=%d size=%d\n", + map->value_size, off, size); + return -EACCES; + } + + return 0; +} + /* check read/write into map element returned by bpf_map_lookup_elem() */ static int __check_map_access(struct bpf_verifier_env *env, u32 regno, int off, int size, bool zero_size_allowed) @@ -1454,7 +1493,7 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno, * need to try adding each of min_value and max_value to off * to make sure our theoretical access will be safe. */ - if (env->log.level) + if (env->log.level & BPF_LOG_LEVEL) print_verifier_state(env, state); /* The minimum value is only important with signed @@ -1955,6 +1994,32 @@ static int check_ctx_reg(struct bpf_verifier_env *env, return 0; } +static int check_tp_buffer_access(struct bpf_verifier_env *env, + const struct bpf_reg_state *reg, + int regno, int off, int size) +{ + if (off < 0) { + verbose(env, + "R%d invalid tracepoint buffer access: off=%d, size=%d", + regno, off, size); + return -EACCES; + } + if (!tnum_is_const(reg->var_off) || reg->var_off.value) { + char tn_buf[48]; + + tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); + verbose(env, + "R%d invalid variable buffer offset: off=%d, var_off=%s", + regno, off, tn_buf); + return -EACCES; + } + if (off + size > env->prog->aux->max_tp_access) + env->prog->aux->max_tp_access = off + size; + + return 0; +} + + /* truncate register to smaller size (in bytes) * must be called with size < BPF_REG_SIZE */ @@ -2011,7 +2076,9 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn verbose(env, "R%d leaks addr into map\n", value_regno); return -EACCES; } - + err = check_map_access_type(env, regno, off, size, t); + if (err) + return err; err = check_map_access(env, regno, off, size, false); if (!err && t == BPF_READ && value_regno >= 0) mark_reg_unknown(env, regs, value_regno); @@ -2097,6 +2164,10 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn err = check_sock_access(env, insn_idx, regno, off, size, t); if (!err && value_regno >= 0) mark_reg_unknown(env, regs, value_regno); + } else if (reg->type == PTR_TO_TP_BUFFER) { + err = check_tp_buffer_access(env, reg, regno, off, size); + if (!err && t == BPF_READ && value_regno >= 0) + mark_reg_unknown(env, regs, value_regno); } else { verbose(env, "R%d invalid mem access '%s'\n", regno, reg_type_str[reg->type]); @@ -2157,6 +2228,29 @@ static int check_xadd(struct bpf_verifier_env *env, int insn_idx, struct bpf_ins BPF_SIZE(insn->code), BPF_WRITE, -1, true); } +static int __check_stack_boundary(struct bpf_verifier_env *env, u32 regno, + int off, int access_size, + bool zero_size_allowed) +{ + struct bpf_reg_state *reg = reg_state(env, regno); + + if (off >= 0 || off < -MAX_BPF_STACK || off + access_size > 0 || + access_size < 0 || (access_size == 0 && !zero_size_allowed)) { + if (tnum_is_const(reg->var_off)) { + verbose(env, "invalid stack type R%d off=%d access_size=%d\n", + regno, off, access_size); + } else { + char tn_buf[48]; + + tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); + verbose(env, "invalid stack type R%d var_off=%s access_size=%d\n", + regno, tn_buf, access_size); + } + return -EACCES; + } + return 0; +} + /* when register 'regno' is passed into function that will read 'access_size' * bytes from that pointer, make sure that it's within stack boundary * and all elements of stack are initialized. @@ -2169,7 +2263,7 @@ static int check_stack_boundary(struct bpf_verifier_env *env, int regno, { struct bpf_reg_state *reg = reg_state(env, regno); struct bpf_func_state *state = func(env, reg); - int off, i, slot, spi; + int err, min_off, max_off, i, slot, spi; if (reg->type != PTR_TO_STACK) { /* Allow zero-byte read from NULL, regardless of pointer type */ @@ -2183,21 +2277,57 @@ static int check_stack_boundary(struct bpf_verifier_env *env, int regno, return -EACCES; } - /* Only allow fixed-offset stack reads */ - if (!tnum_is_const(reg->var_off)) { - char tn_buf[48]; + if (tnum_is_const(reg->var_off)) { + min_off = max_off = reg->var_off.value + reg->off; + err = __check_stack_boundary(env, regno, min_off, access_size, + zero_size_allowed); + if (err) + return err; + } else { + /* Variable offset is prohibited for unprivileged mode for + * simplicity since it requires corresponding support in + * Spectre masking for stack ALU. + * See also retrieve_ptr_limit(). + */ + if (!env->allow_ptr_leaks) { + char tn_buf[48]; - tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); - verbose(env, "invalid variable stack read R%d var_off=%s\n", - regno, tn_buf); - return -EACCES; - } - off = reg->off + reg->var_off.value; - if (off >= 0 || off < -MAX_BPF_STACK || off + access_size > 0 || - access_size < 0 || (access_size == 0 && !zero_size_allowed)) { - verbose(env, "invalid stack type R%d off=%d access_size=%d\n", - regno, off, access_size); - return -EACCES; + tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); + verbose(env, "R%d indirect variable offset stack access prohibited for !root, var_off=%s\n", + regno, tn_buf); + return -EACCES; + } + /* Only initialized buffer on stack is allowed to be accessed + * with variable offset. With uninitialized buffer it's hard to + * guarantee that whole memory is marked as initialized on + * helper return since specific bounds are unknown what may + * cause uninitialized stack leaking. + */ + if (meta && meta->raw_mode) + meta = NULL; + + if (reg->smax_value >= BPF_MAX_VAR_OFF || + reg->smax_value <= -BPF_MAX_VAR_OFF) { + verbose(env, "R%d unbounded indirect variable offset stack access\n", + regno); + return -EACCES; + } + min_off = reg->smin_value + reg->off; + max_off = reg->smax_value + reg->off; + err = __check_stack_boundary(env, regno, min_off, access_size, + zero_size_allowed); + if (err) { + verbose(env, "R%d min value is outside of stack bound\n", + regno); + return err; + } + err = __check_stack_boundary(env, regno, max_off, access_size, + zero_size_allowed); + if (err) { + verbose(env, "R%d max value is outside of stack bound\n", + regno); + return err; + } } if (meta && meta->raw_mode) { @@ -2206,10 +2336,10 @@ static int check_stack_boundary(struct bpf_verifier_env *env, int regno, return 0; } - for (i = 0; i < access_size; i++) { + for (i = min_off; i < max_off + access_size; i++) { u8 *stype; - slot = -(off + i) - 1; + slot = -i - 1; spi = slot / BPF_REG_SIZE; if (state->allocated_stack <= slot) goto err; @@ -2222,8 +2352,16 @@ static int check_stack_boundary(struct bpf_verifier_env *env, int regno, goto mark; } err: - verbose(env, "invalid indirect read from stack off %d+%d size %d\n", - off, i, access_size); + if (tnum_is_const(reg->var_off)) { + verbose(env, "invalid indirect read from stack off %d+%d size %d\n", + min_off, i - min_off, access_size); + } else { + char tn_buf[48]; + + tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); + verbose(env, "invalid indirect read from stack var_off %s+%d size %d\n", + tn_buf, i - min_off, access_size); + } return -EACCES; mark: /* reading any byte out of 8-byte 'spill_slot' will cause @@ -2232,7 +2370,7 @@ mark: mark_reg_read(env, &state->stack[spi].spilled_ptr, state->stack[spi].spilled_ptr.parent); } - return update_stack_depth(env, state, off); + return update_stack_depth(env, state, min_off); } static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, @@ -2247,6 +2385,10 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, return check_packet_access(env, regno, reg->off, access_size, zero_size_allowed); case PTR_TO_MAP_VALUE: + if (check_map_access_type(env, regno, reg->off, access_size, + meta && meta->raw_mode ? BPF_WRITE : + BPF_READ)) + return -EACCES; return check_map_access(env, regno, reg->off, access_size, zero_size_allowed); default: /* scalar_value|ptr_to_stack or invalid ptr */ @@ -2353,6 +2495,22 @@ static bool arg_type_is_mem_size(enum bpf_arg_type type) type == ARG_CONST_SIZE_OR_ZERO; } +static bool arg_type_is_int_ptr(enum bpf_arg_type type) +{ + return type == ARG_PTR_TO_INT || + type == ARG_PTR_TO_LONG; +} + +static int int_ptr_type_to_size(enum bpf_arg_type type) +{ + if (type == ARG_PTR_TO_INT) + return sizeof(u32); + else if (type == ARG_PTR_TO_LONG) + return sizeof(u64); + + return -EINVAL; +} + static int check_func_arg(struct bpf_verifier_env *env, u32 regno, enum bpf_arg_type arg_type, struct bpf_call_arg_meta *meta) @@ -2385,10 +2543,15 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, if (arg_type == ARG_PTR_TO_MAP_KEY || arg_type == ARG_PTR_TO_MAP_VALUE || - arg_type == ARG_PTR_TO_UNINIT_MAP_VALUE) { + arg_type == ARG_PTR_TO_UNINIT_MAP_VALUE || + arg_type == ARG_PTR_TO_MAP_VALUE_OR_NULL) { expected_type = PTR_TO_STACK; - if (!type_is_pkt_pointer(type) && type != PTR_TO_MAP_VALUE && - type != expected_type) + if (register_is_null(reg) && + arg_type == ARG_PTR_TO_MAP_VALUE_OR_NULL) + /* final test in check_stack_boundary() */; + else if (!type_is_pkt_pointer(type) && + type != PTR_TO_MAP_VALUE && + type != expected_type) goto err_type; } else if (arg_type == ARG_CONST_SIZE || arg_type == ARG_CONST_SIZE_OR_ZERO) { @@ -2420,6 +2583,10 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, } meta->ref_obj_id = reg->ref_obj_id; } + } else if (arg_type == ARG_PTR_TO_SOCKET) { + expected_type = PTR_TO_SOCKET; + if (type != expected_type) + goto err_type; } else if (arg_type == ARG_PTR_TO_SPIN_LOCK) { if (meta->func_id == BPF_FUNC_spin_lock) { if (process_spin_lock(env, regno, true)) @@ -2445,6 +2612,12 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, type != expected_type) goto err_type; meta->raw_mode = arg_type == ARG_PTR_TO_UNINIT_MEM; + } else if (arg_type_is_int_ptr(arg_type)) { + expected_type = PTR_TO_STACK; + if (!type_is_pkt_pointer(type) && + type != PTR_TO_MAP_VALUE && + type != expected_type) + goto err_type; } else { verbose(env, "unsupported arg_type %d\n", arg_type); return -EFAULT; @@ -2471,6 +2644,8 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, meta->map_ptr->key_size, false, NULL); } else if (arg_type == ARG_PTR_TO_MAP_VALUE || + (arg_type == ARG_PTR_TO_MAP_VALUE_OR_NULL && + !register_is_null(reg)) || arg_type == ARG_PTR_TO_UNINIT_MAP_VALUE) { /* bpf_map_xxx(..., map_ptr, ..., value) call: * check [value, value + map->value_size) validity @@ -2526,6 +2701,13 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, err = check_helper_mem_access(env, regno - 1, reg->umax_value, zero_size_allowed, meta); + } else if (arg_type_is_int_ptr(arg_type)) { + int size = int_ptr_type_to_size(arg_type); + + err = check_helper_mem_access(env, regno, size, false, meta); + if (err) + return err; + err = check_ptr_alignment(env, reg, 0, size, true); } return err; @@ -2613,6 +2795,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, func_id != BPF_FUNC_map_push_elem) goto error; break; + case BPF_MAP_TYPE_SK_STORAGE: + if (func_id != BPF_FUNC_sk_storage_get && + func_id != BPF_FUNC_sk_storage_delete) + goto error; + break; default: break; } @@ -2676,6 +2863,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, map->map_type != BPF_MAP_TYPE_STACK) goto error; break; + case BPF_FUNC_sk_storage_get: + case BPF_FUNC_sk_storage_delete: + if (map->map_type != BPF_MAP_TYPE_SK_STORAGE) + goto error; + break; default: break; } @@ -2905,7 +3097,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, /* and go analyze first insn of the callee */ *insn_idx = target_insn; - if (env->log.level) { + if (env->log.level & BPF_LOG_LEVEL) { verbose(env, "caller:\n"); print_verifier_state(env, caller); verbose(env, "callee:\n"); @@ -2945,7 +3137,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) return err; *insn_idx = callee->callsite + 1; - if (env->log.level) { + if (env->log.level & BPF_LOG_LEVEL) { verbose(env, "returning from callee:\n"); print_verifier_state(env, callee); verbose(env, "to caller at %d:\n", *insn_idx); @@ -2979,6 +3171,7 @@ record_func_map(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, int func_id, int insn_idx) { struct bpf_insn_aux_data *aux = &env->insn_aux_data[insn_idx]; + struct bpf_map *map = meta->map_ptr; if (func_id != BPF_FUNC_tail_call && func_id != BPF_FUNC_map_lookup_elem && @@ -2989,11 +3182,24 @@ record_func_map(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, func_id != BPF_FUNC_map_peek_elem) return 0; - if (meta->map_ptr == NULL) { + if (map == NULL) { verbose(env, "kernel subsystem misconfigured verifier\n"); return -EINVAL; } + /* In case of read-only, some additional restrictions + * need to be applied in order to prevent altering the + * state of the map from program side. + */ + if ((map->map_flags & BPF_F_RDONLY_PROG) && + (func_id == BPF_FUNC_map_delete_elem || + func_id == BPF_FUNC_map_update_elem || + func_id == BPF_FUNC_map_push_elem || + func_id == BPF_FUNC_map_pop_elem)) { + verbose(env, "write into map forbidden\n"); + return -EACCES; + } + if (!BPF_MAP_PTR(aux->map_state)) bpf_map_ptr_store(aux, meta->map_ptr, meta->map_ptr->unpriv_array); @@ -3157,19 +3363,11 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn } else if (fn->ret_type == RET_PTR_TO_SOCKET_OR_NULL) { mark_reg_known_zero(env, regs, BPF_REG_0); regs[BPF_REG_0].type = PTR_TO_SOCKET_OR_NULL; - if (is_acquire_function(func_id)) { - int id = acquire_reference_state(env, insn_idx); - - if (id < 0) - return id; - /* For mark_ptr_or_null_reg() */ - regs[BPF_REG_0].id = id; - /* For release_reference() */ - regs[BPF_REG_0].ref_obj_id = id; - } else { - /* For mark_ptr_or_null_reg() */ - regs[BPF_REG_0].id = ++env->id_gen; - } + regs[BPF_REG_0].id = ++env->id_gen; + } else if (fn->ret_type == RET_PTR_TO_SOCK_COMMON_OR_NULL) { + mark_reg_known_zero(env, regs, BPF_REG_0); + regs[BPF_REG_0].type = PTR_TO_SOCK_COMMON_OR_NULL; + regs[BPF_REG_0].id = ++env->id_gen; } else if (fn->ret_type == RET_PTR_TO_TCP_SOCK_OR_NULL) { mark_reg_known_zero(env, regs, BPF_REG_0); regs[BPF_REG_0].type = PTR_TO_TCP_SOCK_OR_NULL; @@ -3180,9 +3378,19 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn return -EINVAL; } - if (is_ptr_cast_function(func_id)) + if (is_ptr_cast_function(func_id)) { /* For release_reference() */ regs[BPF_REG_0].ref_obj_id = meta.ref_obj_id; + } else if (is_acquire_function(func_id)) { + int id = acquire_reference_state(env, insn_idx); + + if (id < 0) + return id; + /* For mark_ptr_or_null_reg() */ + regs[BPF_REG_0].id = id; + /* For release_reference() */ + regs[BPF_REG_0].ref_obj_id = id; + } do_refine_retval_range(regs, fn->ret_type, func_id, &meta); @@ -3282,6 +3490,9 @@ static int retrieve_ptr_limit(const struct bpf_reg_state *ptr_reg, switch (ptr_reg->type) { case PTR_TO_STACK: + /* Indirect variable offset stack access is prohibited in + * unprivileged mode so it's not handled here. + */ off = ptr_reg->off + ptr_reg->var_off.value; if (mask_to_left) *ptr_limit = MAX_BPF_STACK + off; @@ -4982,23 +5193,17 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env, insn->dst_reg); return -EACCES; } - if (env->log.level) + if (env->log.level & BPF_LOG_LEVEL) print_verifier_state(env, this_branch->frame[this_branch->curframe]); return 0; } -/* return the map pointer stored inside BPF_LD_IMM64 instruction */ -static struct bpf_map *ld_imm64_to_map_ptr(struct bpf_insn *insn) -{ - u64 imm64 = ((u64) (u32) insn[0].imm) | ((u64) (u32) insn[1].imm) << 32; - - return (struct bpf_map *) (unsigned long) imm64; -} - /* verify BPF_LD_IMM64 instruction */ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn) { + struct bpf_insn_aux_data *aux = cur_aux(env); struct bpf_reg_state *regs = cur_regs(env); + struct bpf_map *map; int err; if (BPF_SIZE(insn->code) != BPF_DW) { @@ -5022,11 +5227,22 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn) return 0; } - /* replace_map_fd_with_map_ptr() should have caught bad ld_imm64 */ - BUG_ON(insn->src_reg != BPF_PSEUDO_MAP_FD); + map = env->used_maps[aux->map_index]; + mark_reg_known_zero(env, regs, insn->dst_reg); + regs[insn->dst_reg].map_ptr = map; + + if (insn->src_reg == BPF_PSEUDO_MAP_VALUE) { + regs[insn->dst_reg].type = PTR_TO_MAP_VALUE; + regs[insn->dst_reg].off = aux->map_off; + if (map_value_has_spin_lock(map)) + regs[insn->dst_reg].id = ++env->id_gen; + } else if (insn->src_reg == BPF_PSEUDO_MAP_FD) { + regs[insn->dst_reg].type = CONST_PTR_TO_MAP; + } else { + verbose(env, "bpf verifier is misconfigured\n"); + return -EINVAL; + } - regs[insn->dst_reg].type = CONST_PTR_TO_MAP; - regs[insn->dst_reg].map_ptr = ld_imm64_to_map_ptr(insn); return 0; } @@ -5150,6 +5366,7 @@ static int check_return_code(struct bpf_verifier_env *env) case BPF_PROG_TYPE_CGROUP_SOCK_ADDR: case BPF_PROG_TYPE_SOCK_OPS: case BPF_PROG_TYPE_CGROUP_DEVICE: + case BPF_PROG_TYPE_CGROUP_SYSCTL: break; default: return 0; @@ -5220,10 +5437,6 @@ enum { #define STATE_LIST_MARK ((struct bpf_verifier_state_list *) -1L) -static int *insn_stack; /* stack of insns to process */ -static int cur_stack; /* current stack index */ -static int *insn_state; - /* t, w, e - match pseudo-code above: * t - index of current instruction * w - next instruction @@ -5231,6 +5444,9 @@ static int *insn_state; */ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env) { + int *insn_stack = env->cfg.insn_stack; + int *insn_state = env->cfg.insn_state; + if (e == FALLTHROUGH && insn_state[t] >= (DISCOVERED | FALLTHROUGH)) return 0; @@ -5251,9 +5467,9 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env) /* tree-edge */ insn_state[t] = DISCOVERED | e; insn_state[w] = DISCOVERED; - if (cur_stack >= env->prog->len) + if (env->cfg.cur_stack >= env->prog->len) return -E2BIG; - insn_stack[cur_stack++] = w; + insn_stack[env->cfg.cur_stack++] = w; return 1; } else if ((insn_state[w] & 0xF0) == DISCOVERED) { verbose_linfo(env, t, "%d: ", t); @@ -5277,27 +5493,28 @@ static int check_cfg(struct bpf_verifier_env *env) { struct bpf_insn *insns = env->prog->insnsi; int insn_cnt = env->prog->len; + int *insn_stack, *insn_state; int ret = 0; int i, t; - insn_state = kcalloc(insn_cnt, sizeof(int), GFP_KERNEL); + insn_state = env->cfg.insn_state = kvcalloc(insn_cnt, sizeof(int), GFP_KERNEL); if (!insn_state) return -ENOMEM; - insn_stack = kcalloc(insn_cnt, sizeof(int), GFP_KERNEL); + insn_stack = env->cfg.insn_stack = kvcalloc(insn_cnt, sizeof(int), GFP_KERNEL); if (!insn_stack) { - kfree(insn_state); + kvfree(insn_state); return -ENOMEM; } insn_state[0] = DISCOVERED; /* mark 1st insn as discovered */ insn_stack[0] = 0; /* 0 is the first instruction */ - cur_stack = 1; + env->cfg.cur_stack = 1; peek_stack: - if (cur_stack == 0) + if (env->cfg.cur_stack == 0) goto check_state; - t = insn_stack[cur_stack - 1]; + t = insn_stack[env->cfg.cur_stack - 1]; if (BPF_CLASS(insns[t].code) == BPF_JMP || BPF_CLASS(insns[t].code) == BPF_JMP32) { @@ -5366,7 +5583,7 @@ peek_stack: mark_explored: insn_state[t] = EXPLORED; - if (cur_stack-- <= 0) { + if (env->cfg.cur_stack-- <= 0) { verbose(env, "pop stack internal bug\n"); ret = -EFAULT; goto err_free; @@ -5384,8 +5601,9 @@ check_state: ret = 0; /* cfg looks good */ err_free: - kfree(insn_state); - kfree(insn_stack); + kvfree(insn_state); + kvfree(insn_stack); + env->cfg.insn_state = env->cfg.insn_stack = NULL; return ret; } @@ -6074,6 +6292,22 @@ static bool states_equal(struct bpf_verifier_env *env, return true; } +static int propagate_liveness_reg(struct bpf_verifier_env *env, + struct bpf_reg_state *reg, + struct bpf_reg_state *parent_reg) +{ + int err; + + if (parent_reg->live & REG_LIVE_READ || !(reg->live & REG_LIVE_READ)) + return 0; + + err = mark_reg_read(env, reg, parent_reg); + if (err) + return err; + + return 0; +} + /* A write screens off any subsequent reads; but write marks come from the * straight-line code between a state and its parent. When we arrive at an * equivalent state (jump target or such) we didn't arrive by the straight-line @@ -6085,8 +6319,9 @@ static int propagate_liveness(struct bpf_verifier_env *env, const struct bpf_verifier_state *vstate, struct bpf_verifier_state *vparent) { - int i, frame, err = 0; + struct bpf_reg_state *state_reg, *parent_reg; struct bpf_func_state *state, *parent; + int i, frame, err = 0; if (vparent->curframe != vstate->curframe) { WARN(1, "propagate_live: parent frame %d current frame %d\n", @@ -6096,30 +6331,27 @@ static int propagate_liveness(struct bpf_verifier_env *env, /* Propagate read liveness of registers... */ BUILD_BUG_ON(BPF_REG_FP + 1 != MAX_BPF_REG); for (frame = 0; frame <= vstate->curframe; frame++) { + parent = vparent->frame[frame]; + state = vstate->frame[frame]; + parent_reg = parent->regs; + state_reg = state->regs; /* We don't need to worry about FP liveness, it's read-only */ for (i = frame < vstate->curframe ? BPF_REG_6 : 0; i < BPF_REG_FP; i++) { - if (vparent->frame[frame]->regs[i].live & REG_LIVE_READ) - continue; - if (vstate->frame[frame]->regs[i].live & REG_LIVE_READ) { - err = mark_reg_read(env, &vstate->frame[frame]->regs[i], - &vparent->frame[frame]->regs[i]); - if (err) - return err; - } + err = propagate_liveness_reg(env, &state_reg[i], + &parent_reg[i]); + if (err) + return err; } - } - /* ... and stack slots */ - for (frame = 0; frame <= vstate->curframe; frame++) { - state = vstate->frame[frame]; - parent = vparent->frame[frame]; + /* Propagate stack slots. */ for (i = 0; i < state->allocated_stack / BPF_REG_SIZE && i < parent->allocated_stack / BPF_REG_SIZE; i++) { - if (parent->stack[i].spilled_ptr.live & REG_LIVE_READ) - continue; - if (state->stack[i].spilled_ptr.live & REG_LIVE_READ) - mark_reg_read(env, &state->stack[i].spilled_ptr, - &parent->stack[i].spilled_ptr); + parent_reg = &parent->stack[i].spilled_ptr; + state_reg = &state->stack[i].spilled_ptr; + err = propagate_liveness_reg(env, state_reg, + parent_reg); + if (err) + return err; } } return err; @@ -6128,11 +6360,13 @@ static int propagate_liveness(struct bpf_verifier_env *env, static int is_state_visited(struct bpf_verifier_env *env, int insn_idx) { struct bpf_verifier_state_list *new_sl; - struct bpf_verifier_state_list *sl; + struct bpf_verifier_state_list *sl, **pprev; struct bpf_verifier_state *cur = env->cur_state, *new; int i, j, err, states_cnt = 0; - sl = env->explored_states[insn_idx]; + pprev = &env->explored_states[insn_idx]; + sl = *pprev; + if (!sl) /* this 'insn_idx' instruction wasn't marked, so we will not * be doing state search here @@ -6143,6 +6377,7 @@ static int is_state_visited(struct bpf_verifier_env *env, int insn_idx) while (sl != STATE_LIST_MARK) { if (states_equal(env, &sl->state, cur)) { + sl->hit_cnt++; /* reached equivalent register/stack state, * prune the search. * Registers read by the continuation are read by us. @@ -6158,10 +6393,40 @@ static int is_state_visited(struct bpf_verifier_env *env, int insn_idx) return err; return 1; } - sl = sl->next; states_cnt++; + sl->miss_cnt++; + /* heuristic to determine whether this state is beneficial + * to keep checking from state equivalence point of view. + * Higher numbers increase max_states_per_insn and verification time, + * but do not meaningfully decrease insn_processed. + */ + if (sl->miss_cnt > sl->hit_cnt * 3 + 3) { + /* the state is unlikely to be useful. Remove it to + * speed up verification + */ + *pprev = sl->next; + if (sl->state.frame[0]->regs[0].live & REG_LIVE_DONE) { + free_verifier_state(&sl->state, false); + kfree(sl); + env->peak_states--; + } else { + /* cannot free this state, since parentage chain may + * walk it later. Add it for free_list instead to + * be freed at the end of verification + */ + sl->next = env->free_list; + env->free_list = sl; + } + sl = *pprev; + continue; + } + pprev = &sl->next; + sl = *pprev; } + if (env->max_states_per_insn < states_cnt) + env->max_states_per_insn = states_cnt; + if (!env->allow_ptr_leaks && states_cnt > BPF_COMPLEXITY_LIMIT_STATES) return 0; @@ -6175,6 +6440,8 @@ static int is_state_visited(struct bpf_verifier_env *env, int insn_idx) new_sl = kzalloc(sizeof(struct bpf_verifier_state_list), GFP_KERNEL); if (!new_sl) return -ENOMEM; + env->total_states++; + env->peak_states++; /* add new state to the head of linked list */ new = &new_sl->state; @@ -6259,8 +6526,7 @@ static int do_check(struct bpf_verifier_env *env) struct bpf_verifier_state *state; struct bpf_insn *insns = env->prog->insnsi; struct bpf_reg_state *regs; - int insn_cnt = env->prog->len, i; - int insn_processed = 0; + int insn_cnt = env->prog->len; bool do_print_state = false; env->prev_linfo = NULL; @@ -6295,10 +6561,10 @@ static int do_check(struct bpf_verifier_env *env) insn = &insns[env->insn_idx]; class = BPF_CLASS(insn->code); - if (++insn_processed > BPF_COMPLEXITY_LIMIT_INSNS) { + if (++env->insn_processed > BPF_COMPLEXITY_LIMIT_INSNS) { verbose(env, "BPF program is too large. Processed %d insn\n", - insn_processed); + env->insn_processed); return -E2BIG; } @@ -6307,7 +6573,7 @@ static int do_check(struct bpf_verifier_env *env) return err; if (err == 1) { /* found equivalent state, can prune the search */ - if (env->log.level) { + if (env->log.level & BPF_LOG_LEVEL) { if (do_print_state) verbose(env, "\nfrom %d to %d%s: safe\n", env->prev_insn_idx, env->insn_idx, @@ -6325,8 +6591,9 @@ static int do_check(struct bpf_verifier_env *env) if (need_resched()) cond_resched(); - if (env->log.level > 1 || (env->log.level && do_print_state)) { - if (env->log.level > 1) + if (env->log.level & BPF_LOG_LEVEL2 || + (env->log.level & BPF_LOG_LEVEL && do_print_state)) { + if (env->log.level & BPF_LOG_LEVEL2) verbose(env, "%d:", env->insn_idx); else verbose(env, "\nfrom %d to %d%s:", @@ -6337,7 +6604,7 @@ static int do_check(struct bpf_verifier_env *env) do_print_state = false; } - if (env->log.level) { + if (env->log.level & BPF_LOG_LEVEL) { const struct bpf_insn_cbs cbs = { .cb_print = verbose, .private_data = env, @@ -6602,16 +6869,6 @@ process_bpf_exit: env->insn_idx++; } - verbose(env, "processed %d insns (limit %d), stack depth ", - insn_processed, BPF_COMPLEXITY_LIMIT_INSNS); - for (i = 0; i < env->subprog_cnt; i++) { - u32 depth = env->subprog_info[i].stack_depth; - - verbose(env, "%d", depth); - if (i + 1 < env->subprog_cnt) - verbose(env, "+"); - } - verbose(env, "\n"); env->prog->aux->stack_depth = env->subprog_info[0].stack_depth; return 0; } @@ -6709,8 +6966,10 @@ static int replace_map_fd_with_map_ptr(struct bpf_verifier_env *env) } if (insn[0].code == (BPF_LD | BPF_IMM | BPF_DW)) { + struct bpf_insn_aux_data *aux; struct bpf_map *map; struct fd f; + u64 addr; if (i == insn_cnt - 1 || insn[1].code != 0 || insn[1].dst_reg != 0 || insn[1].src_reg != 0 || @@ -6719,13 +6978,19 @@ static int replace_map_fd_with_map_ptr(struct bpf_verifier_env *env) return -EINVAL; } - if (insn->src_reg == 0) + if (insn[0].src_reg == 0) /* valid generic load 64-bit imm */ goto next_insn; - if (insn[0].src_reg != BPF_PSEUDO_MAP_FD || - insn[1].imm != 0) { - verbose(env, "unrecognized bpf_ld_imm64 insn\n"); + /* In final convert_pseudo_ld_imm64() step, this is + * converted into regular 64-bit imm load insn. + */ + if ((insn[0].src_reg != BPF_PSEUDO_MAP_FD && + insn[0].src_reg != BPF_PSEUDO_MAP_VALUE) || + (insn[0].src_reg == BPF_PSEUDO_MAP_FD && + insn[1].imm != 0)) { + verbose(env, + "unrecognized bpf_ld_imm64 insn\n"); return -EINVAL; } @@ -6743,16 +7008,47 @@ static int replace_map_fd_with_map_ptr(struct bpf_verifier_env *env) return err; } - /* store map pointer inside BPF_LD_IMM64 instruction */ - insn[0].imm = (u32) (unsigned long) map; - insn[1].imm = ((u64) (unsigned long) map) >> 32; + aux = &env->insn_aux_data[i]; + if (insn->src_reg == BPF_PSEUDO_MAP_FD) { + addr = (unsigned long)map; + } else { + u32 off = insn[1].imm; + + if (off >= BPF_MAX_VAR_OFF) { + verbose(env, "direct value offset of %u is not allowed\n", off); + fdput(f); + return -EINVAL; + } + + if (!map->ops->map_direct_value_addr) { + verbose(env, "no direct value access support for this map type\n"); + fdput(f); + return -EINVAL; + } + + err = map->ops->map_direct_value_addr(map, &addr, off); + if (err) { + verbose(env, "invalid access to map value pointer, value_size=%u off=%u\n", + map->value_size, off); + fdput(f); + return err; + } + + aux->map_off = off; + addr += off; + } + + insn[0].imm = (u32)addr; + insn[1].imm = addr >> 32; /* check whether we recorded this map already */ - for (j = 0; j < env->used_map_cnt; j++) + for (j = 0; j < env->used_map_cnt; j++) { if (env->used_maps[j] == map) { + aux->map_index = j; fdput(f); goto next_insn; } + } if (env->used_map_cnt >= MAX_USED_MAPS) { fdput(f); @@ -6769,6 +7065,8 @@ static int replace_map_fd_with_map_ptr(struct bpf_verifier_env *env) fdput(f); return PTR_ERR(map); } + + aux->map_index = env->used_map_cnt; env->used_maps[env->used_map_cnt++] = map; if (bpf_map_is_cgroup_storage(map) && @@ -6874,8 +7172,13 @@ static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 of struct bpf_prog *new_prog; new_prog = bpf_patch_insn_single(env->prog, off, patch, len); - if (!new_prog) + if (IS_ERR(new_prog)) { + if (PTR_ERR(new_prog) == -ERANGE) + verbose(env, + "insn %d cannot be patched due to 16-bit range\n", + env->insn_aux_data[off].orig_idx); return NULL; + } if (adjust_insn_aux_data(env, new_prog->len, off, len)) return NULL; adjust_subprog_starts(env, off, len); @@ -7413,9 +7716,8 @@ static int jit_subprogs(struct bpf_verifier_env *env) insn->src_reg != BPF_PSEUDO_CALL) continue; subprog = insn->off; - insn->imm = (u64 (*)(u64, u64, u64, u64, u64)) - func[subprog]->bpf_func - - __bpf_call_base; + insn->imm = BPF_CAST_CALL(func[subprog]->bpf_func) - + __bpf_call_base; } /* we use the aux data to keep a list of the start addresses @@ -7817,6 +8119,14 @@ static void free_states(struct bpf_verifier_env *env) struct bpf_verifier_state_list *sl, *sln; int i; + sl = env->free_list; + while (sl) { + sln = sl->next; + free_verifier_state(&sl->state, false); + kfree(sl); + sl = sln; + } + if (!env->explored_states) return; @@ -7832,12 +8142,37 @@ static void free_states(struct bpf_verifier_env *env) } } - kfree(env->explored_states); + kvfree(env->explored_states); +} + +static void print_verification_stats(struct bpf_verifier_env *env) +{ + int i; + + if (env->log.level & BPF_LOG_STATS) { + verbose(env, "verification time %lld usec\n", + div_u64(env->verification_time, 1000)); + verbose(env, "stack depth "); + for (i = 0; i < env->subprog_cnt; i++) { + u32 depth = env->subprog_info[i].stack_depth; + + verbose(env, "%d", depth); + if (i + 1 < env->subprog_cnt) + verbose(env, "+"); + } + verbose(env, "\n"); + } + verbose(env, "processed %d insns (limit %d) max_states_per_insn %d " + "total_states %d peak_states %d mark_read %d\n", + env->insn_processed, BPF_COMPLEXITY_LIMIT_INSNS, + env->max_states_per_insn, env->total_states, + env->peak_states, env->longest_mark_read_walk); } int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, union bpf_attr __user *uattr) { + u64 start_time = ktime_get_ns(); struct bpf_verifier_env *env; struct bpf_verifier_log *log; int i, len, ret = -EINVAL; @@ -7865,9 +8200,11 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, env->insn_aux_data[i].orig_idx = i; env->prog = *prog; env->ops = bpf_verifier_ops[env->prog->type]; + is_priv = capable(CAP_SYS_ADMIN); /* grab the mutex to protect few globals used by verifier */ - mutex_lock(&bpf_verifier_lock); + if (!is_priv) + mutex_lock(&bpf_verifier_lock); if (attr->log_level || attr->log_buf || attr->log_size) { /* user requested verbose verifier output @@ -7879,8 +8216,8 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, ret = -EINVAL; /* log attributes have to be sane */ - if (log->len_total < 128 || log->len_total > UINT_MAX >> 8 || - !log->level || !log->ubuf) + if (log->len_total < 128 || log->len_total > UINT_MAX >> 2 || + !log->level || !log->ubuf || log->level & ~BPF_LOG_MASK) goto err_unlock; } @@ -7890,7 +8227,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, if (attr->prog_flags & BPF_F_ANY_ALIGNMENT) env->strict_alignment = false; - is_priv = capable(CAP_SYS_ADMIN); env->allow_ptr_leaks = is_priv; ret = replace_map_fd_with_map_ptr(env); @@ -7903,7 +8239,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, goto skip_full_check; } - env->explored_states = kcalloc(env->prog->len, + env->explored_states = kvcalloc(env->prog->len, sizeof(struct bpf_verifier_state_list *), GFP_USER); ret = -ENOMEM; @@ -7961,6 +8297,9 @@ skip_full_check: if (ret == 0) ret = fixup_call_args(env); + env->verification_time = ktime_get_ns() - start_time; + print_verification_stats(env); + if (log->level && bpf_verifier_log_full(log)) ret = -ENOSPC; if (log->level && !log->ubuf) { @@ -8000,7 +8339,8 @@ err_release_maps: release_maps(env); *prog = env->prog; err_unlock: - mutex_unlock(&bpf_verifier_lock); + if (!is_priv) + mutex_unlock(&bpf_verifier_lock); vfree(env->insn_aux_data); err_free_env: kfree(env); diff --git a/kernel/sysctl.c b/kernel/sysctl.c index c9ec050bcf46126286dba7d122b3a7240de6de65..599510a3355e1336609c4237e1a566905c7a117d 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -3326,6 +3326,11 @@ int proc_doulongvec_ms_jiffies_minmax(struct ctl_table *table, int write, return -ENOSYS; } +int proc_do_large_bitmap(struct ctl_table *table, int write, + void __user *buffer, size_t *lenp, loff_t *ppos) +{ + return -ENOSYS; +} #endif /* CONFIG_PROC_SYSCTL */ @@ -3366,3 +3371,4 @@ EXPORT_SYMBOL(proc_dointvec_ms_jiffies); EXPORT_SYMBOL(proc_dostring); EXPORT_SYMBOL(proc_doulongvec_minmax); EXPORT_SYMBOL(proc_doulongvec_ms_jiffies_minmax); +EXPORT_SYMBOL(proc_do_large_bitmap); diff --git a/kernel/taskstats.c b/kernel/taskstats.c index 4e62a4a8fa9168225c2756512fdaaa51670b49f4..5f852b8f59f72b9c0b19f105ab3289e3ea0a0809 100644 --- a/kernel/taskstats.c +++ b/kernel/taskstats.c @@ -375,7 +375,7 @@ static struct taskstats *mk_reply(struct sk_buff *skb, int type, u32 pid) ? TASKSTATS_TYPE_AGGR_PID : TASKSTATS_TYPE_AGGR_TGID; - na = nla_nest_start(skb, aggr); + na = nla_nest_start_noflag(skb, aggr); if (!na) goto err; @@ -649,17 +649,41 @@ err: static const struct genl_ops taskstats_ops[] = { { .cmd = TASKSTATS_CMD_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = taskstats_user_cmd, - .policy = taskstats_cmd_get_policy, - .flags = GENL_ADMIN_PERM, + /* policy enforced later */ + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_HASPOL, }, { .cmd = CGROUPSTATS_CMD_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = cgroupstats_user_cmd, - .policy = cgroupstats_cmd_get_policy, + /* policy enforced later */ + .flags = GENL_CMD_CAP_HASPOL, }, }; +static int taskstats_pre_doit(const struct genl_ops *ops, struct sk_buff *skb, + struct genl_info *info) +{ + const struct nla_policy *policy = NULL; + + switch (ops->cmd) { + case TASKSTATS_CMD_GET: + policy = taskstats_cmd_get_policy; + break; + case CGROUPSTATS_CMD_GET: + policy = cgroupstats_cmd_get_policy; + break; + default: + return -EINVAL; + } + + return nlmsg_validate_deprecated(info->nlhdr, GENL_HDRLEN, + TASKSTATS_CMD_ATTR_MAX, policy, + info->extack); +} + static struct genl_family family __ro_after_init = { .name = TASKSTATS_GENL_NAME, .version = TASKSTATS_GENL_VERSION, @@ -667,6 +691,7 @@ static struct genl_family family __ro_after_init = { .module = THIS_MODULE, .ops = taskstats_ops, .n_ops = ARRAY_SIZE(taskstats_ops), + .pre_doit = taskstats_pre_doit, }; /* Needed early in initialization */ diff --git a/kernel/time/time.c b/kernel/time/time.c index 86656bbac232e659d6ba4f38242bc774b3a6cbde..7f7d6914ddd5ebd5a029a082bc4afc134593f4f9 100644 --- a/kernel/time/time.c +++ b/kernel/time/time.c @@ -783,6 +783,16 @@ u64 jiffies64_to_nsecs(u64 j) } EXPORT_SYMBOL(jiffies64_to_nsecs); +u64 jiffies64_to_msecs(const u64 j) +{ +#if HZ <= MSEC_PER_SEC && !(MSEC_PER_SEC % HZ) + return (MSEC_PER_SEC / HZ) * j; +#else + return div_u64(j * HZ_TO_MSEC_NUM, HZ_TO_MSEC_DEN); +#endif +} +EXPORT_SYMBOL(jiffies64_to_msecs); + /** * nsecs_to_jiffies64 - Convert nsecs in u64 to jiffies64 * diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 94b0e37d90ef7852886a62da2a03a3a25e798db0..b496ffdf5f36452b01dd49228a3214d2e01afcf7 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -577,6 +577,12 @@ tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_map_update_elem_proto; case BPF_FUNC_map_delete_elem: return &bpf_map_delete_elem_proto; + case BPF_FUNC_map_push_elem: + return &bpf_map_push_elem_proto; + case BPF_FUNC_map_pop_elem: + return &bpf_map_pop_elem_proto; + case BPF_FUNC_map_peek_elem: + return &bpf_map_peek_elem_proto; case BPF_FUNC_probe_read: return &bpf_probe_read_proto; case BPF_FUNC_ktime_get_ns: @@ -917,6 +923,27 @@ const struct bpf_verifier_ops raw_tracepoint_verifier_ops = { const struct bpf_prog_ops raw_tracepoint_prog_ops = { }; +static bool raw_tp_writable_prog_is_valid_access(int off, int size, + enum bpf_access_type type, + const struct bpf_prog *prog, + struct bpf_insn_access_aux *info) +{ + if (off == 0) { + if (size != sizeof(u64) || type != BPF_READ) + return false; + info->reg_type = PTR_TO_TP_BUFFER; + } + return raw_tp_prog_is_valid_access(off, size, type, prog, info); +} + +const struct bpf_verifier_ops raw_tracepoint_writable_verifier_ops = { + .get_func_proto = raw_tp_prog_func_proto, + .is_valid_access = raw_tp_writable_prog_is_valid_access, +}; + +const struct bpf_prog_ops raw_tracepoint_writable_prog_ops = { +}; + static bool pe_prog_is_valid_access(int off, int size, enum bpf_access_type type, const struct bpf_prog *prog, struct bpf_insn_access_aux *info) @@ -1206,6 +1233,9 @@ static int __bpf_probe_register(struct bpf_raw_event_map *btp, struct bpf_prog * if (prog->aux->max_ctx_offset > btp->num_args * sizeof(u64)) return -EINVAL; + if (prog->aux->max_tp_access > btp->writable_size) + return -EINVAL; + return tracepoint_probe_register(tp, (void *)btp->bpf_func, prog); } diff --git a/lib/Kconfig b/lib/Kconfig index e86975bfca6ab72e3518954413ae97ad44e334bd..f323b85ad11cb311426806148c300d918697564d 100644 --- a/lib/Kconfig +++ b/lib/Kconfig @@ -18,6 +18,23 @@ config RAID6_PQ_BENCHMARK Benchmark all available RAID6 PQ functions on init and choose the fastest one. +config PACKING + bool "Generic bitfield packing and unpacking" + default n + help + This option provides the packing() helper function, which permits + converting bitfields between a CPU-usable representation and a + memory representation that can have any combination of these quirks: + - Is little endian (bytes are reversed within a 32-bit group) + - The least-significant 32-bit word comes first (within a 64-bit + group) + - The most significant bit of a byte is at its right (bit 0 of a + register description is numerically 2^7). + Drivers may use these helpers to match the bit indices as described + in the data sheets of the peripherals they are in control of. + + When in doubt, say N. + config BITREVERSE tristate diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 971c6c70891e47412ff5cc6267c86f7b4e396959..d695ec1477f39e86f7c6c74bf7f1021a6776879c 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -219,6 +219,14 @@ config DEBUG_INFO_DWARF4 But it significantly improves the success of resolving variables in gdb on optimized code. +config DEBUG_INFO_BTF + bool "Generate BTF typeinfo" + depends on DEBUG_INFO + help + Generate deduplicated BTF type information from DWARF debug info. + Turning this on expects presence of pahole tool, which will convert + DWARF type info into equivalent deduplicated BTF type info. + config GDB_SCRIPTS bool "Provide GDB scripts for kernel debugging" depends on DEBUG_INFO diff --git a/lib/Makefile b/lib/Makefile index 07506e3891a084d2ac4017bb8c2cec0e5e57f315..83d7df2661ffa1bcf8bd1f0db2711a4533a0b720 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -120,6 +120,7 @@ obj-$(CONFIG_DEBUG_LIST) += list_debug.o obj-$(CONFIG_DEBUG_OBJECTS) += debugobjects.o obj-$(CONFIG_BITREVERSE) += bitrev.o +obj-$(CONFIG_PACKING) += packing.o obj-$(CONFIG_RATIONAL) += rational.o obj-$(CONFIG_CRC_CCITT) += crc-ccitt.o obj-$(CONFIG_CRC16) += crc16.o diff --git a/lib/nlattr.c b/lib/nlattr.c index d26de6156b97db73803cb617bcd81fd7415ac7b1..cace9b307781034df0a63b3edfe8e55c135b2ca9 100644 --- a/lib/nlattr.c +++ b/lib/nlattr.c @@ -69,7 +69,8 @@ static int validate_nla_bitfield32(const struct nlattr *nla, static int nla_validate_array(const struct nlattr *head, int len, int maxtype, const struct nla_policy *policy, - struct netlink_ext_ack *extack) + struct netlink_ext_ack *extack, + unsigned int validate) { const struct nlattr *entry; int rem; @@ -86,8 +87,8 @@ static int nla_validate_array(const struct nlattr *head, int len, int maxtype, return -ERANGE; } - ret = nla_validate(nla_data(entry), nla_len(entry), - maxtype, policy, extack); + ret = __nla_validate(nla_data(entry), nla_len(entry), + maxtype, policy, validate, extack); if (ret < 0) return ret; } @@ -154,13 +155,17 @@ static int nla_validate_int_range(const struct nla_policy *pt, } static int validate_nla(const struct nlattr *nla, int maxtype, - const struct nla_policy *policy, + const struct nla_policy *policy, unsigned int validate, struct netlink_ext_ack *extack) { + u16 strict_start_type = policy[0].strict_start_type; const struct nla_policy *pt; int minlen = 0, attrlen = nla_len(nla), type = nla_type(nla); int err = -ERANGE; + if (strict_start_type && type >= strict_start_type) + validate |= NL_VALIDATE_STRICT; + if (type <= 0 || type > maxtype) return 0; @@ -172,6 +177,26 @@ static int validate_nla(const struct nlattr *nla, int maxtype, (pt->type == NLA_EXACT_LEN_WARN && attrlen != pt->len)) { pr_warn_ratelimited("netlink: '%s': attribute type %d has an invalid length.\n", current->comm, type); + if (validate & NL_VALIDATE_STRICT_ATTRS) { + NL_SET_ERR_MSG_ATTR(extack, nla, + "invalid attribute length"); + return -EINVAL; + } + } + + if (validate & NL_VALIDATE_NESTED) { + if ((pt->type == NLA_NESTED || pt->type == NLA_NESTED_ARRAY) && + !(nla->nla_type & NLA_F_NESTED)) { + NL_SET_ERR_MSG_ATTR(extack, nla, + "NLA_F_NESTED is missing"); + return -EINVAL; + } + if (pt->type != NLA_NESTED && pt->type != NLA_NESTED_ARRAY && + pt->type != NLA_UNSPEC && (nla->nla_type & NLA_F_NESTED)) { + NL_SET_ERR_MSG_ATTR(extack, nla, + "NLA_F_NESTED not expected"); + return -EINVAL; + } } switch (pt->type) { @@ -244,8 +269,9 @@ static int validate_nla(const struct nlattr *nla, int maxtype, if (attrlen < NLA_HDRLEN) goto out_err; if (pt->validation_data) { - err = nla_validate(nla_data(nla), nla_len(nla), pt->len, - pt->validation_data, extack); + err = __nla_validate(nla_data(nla), nla_len(nla), pt->len, + pt->validation_data, validate, + extack); if (err < 0) { /* * return directly to preserve the inner @@ -268,7 +294,7 @@ static int validate_nla(const struct nlattr *nla, int maxtype, err = nla_validate_array(nla_data(nla), nla_len(nla), pt->len, pt->validation_data, - extack); + extack, validate); if (err < 0) { /* * return directly to preserve the inner @@ -278,10 +304,23 @@ static int validate_nla(const struct nlattr *nla, int maxtype, } } break; + + case NLA_UNSPEC: + if (validate & NL_VALIDATE_UNSPEC) { + NL_SET_ERR_MSG_ATTR(extack, nla, + "Unsupported attribute"); + return -EINVAL; + } + /* fall through */ + case NLA_MIN_LEN: + if (attrlen < pt->len) + goto out_err; + break; + default: if (pt->len) minlen = pt->len; - else if (pt->type != NLA_UNSPEC) + else minlen = nla_attr_minlen[pt->type]; if (attrlen < minlen) @@ -315,37 +354,76 @@ out_err: return err; } +static int __nla_validate_parse(const struct nlattr *head, int len, int maxtype, + const struct nla_policy *policy, + unsigned int validate, + struct netlink_ext_ack *extack, + struct nlattr **tb) +{ + const struct nlattr *nla; + int rem; + + if (tb) + memset(tb, 0, sizeof(struct nlattr *) * (maxtype + 1)); + + nla_for_each_attr(nla, head, len, rem) { + u16 type = nla_type(nla); + + if (type == 0 || type > maxtype) { + if (validate & NL_VALIDATE_MAXTYPE) { + NL_SET_ERR_MSG_ATTR(extack, nla, + "Unknown attribute type"); + return -EINVAL; + } + continue; + } + if (policy) { + int err = validate_nla(nla, maxtype, policy, + validate, extack); + + if (err < 0) + return err; + } + + if (tb) + tb[type] = (struct nlattr *)nla; + } + + if (unlikely(rem > 0)) { + pr_warn_ratelimited("netlink: %d bytes leftover after parsing attributes in process `%s'.\n", + rem, current->comm); + NL_SET_ERR_MSG(extack, "bytes leftover after parsing attributes"); + if (validate & NL_VALIDATE_TRAILING) + return -EINVAL; + } + + return 0; +} + /** - * nla_validate - Validate a stream of attributes + * __nla_validate - Validate a stream of attributes * @head: head of attribute stream * @len: length of attribute stream * @maxtype: maximum attribute type to be expected * @policy: validation policy + * @validate: validation strictness * @extack: extended ACK report struct * * Validates all attributes in the specified attribute stream against the - * specified policy. Attributes with a type exceeding maxtype will be - * ignored. See documenation of struct nla_policy for more details. + * specified policy. Validation depends on the validate flags passed, see + * &enum netlink_validation for more details on that. + * See documenation of struct nla_policy for more details. * * Returns 0 on success or a negative error code. */ -int nla_validate(const struct nlattr *head, int len, int maxtype, - const struct nla_policy *policy, - struct netlink_ext_ack *extack) +int __nla_validate(const struct nlattr *head, int len, int maxtype, + const struct nla_policy *policy, unsigned int validate, + struct netlink_ext_ack *extack) { - const struct nlattr *nla; - int rem; - - nla_for_each_attr(nla, head, len, rem) { - int err = validate_nla(nla, maxtype, policy, extack); - - if (err < 0) - return err; - } - - return 0; + return __nla_validate_parse(head, len, maxtype, policy, validate, + extack, NULL); } -EXPORT_SYMBOL(nla_validate); +EXPORT_SYMBOL(__nla_validate); /** * nla_policy_len - Determin the max. length of a policy @@ -377,76 +455,30 @@ nla_policy_len(const struct nla_policy *p, int n) EXPORT_SYMBOL(nla_policy_len); /** - * nla_parse - Parse a stream of attributes into a tb buffer + * __nla_parse - Parse a stream of attributes into a tb buffer * @tb: destination array with maxtype+1 elements * @maxtype: maximum attribute type to be expected * @head: head of attribute stream * @len: length of attribute stream * @policy: validation policy + * @validate: validation strictness + * @extack: extended ACK pointer * * Parses a stream of attributes and stores a pointer to each attribute in - * the tb array accessible via the attribute type. Attributes with a type - * exceeding maxtype will be silently ignored for backwards compatibility - * reasons. policy may be set to NULL if no validation is required. + * the tb array accessible via the attribute type. + * Validation is controlled by the @validate parameter. * * Returns 0 on success or a negative error code. */ -static int __nla_parse(struct nlattr **tb, int maxtype, - const struct nlattr *head, int len, - bool strict, const struct nla_policy *policy, - struct netlink_ext_ack *extack) -{ - const struct nlattr *nla; - int rem; - - memset(tb, 0, sizeof(struct nlattr *) * (maxtype + 1)); - - nla_for_each_attr(nla, head, len, rem) { - u16 type = nla_type(nla); - - if (type == 0 || type > maxtype) { - if (strict) { - NL_SET_ERR_MSG(extack, "Unknown attribute type"); - return -EINVAL; - } - continue; - } - if (policy) { - int err = validate_nla(nla, maxtype, policy, extack); - - if (err < 0) - return err; - } - - tb[type] = (struct nlattr *)nla; - } - - if (unlikely(rem > 0)) { - pr_warn_ratelimited("netlink: %d bytes leftover after parsing attributes in process `%s'.\n", - rem, current->comm); - NL_SET_ERR_MSG(extack, "bytes leftover after parsing attributes"); - if (strict) - return -EINVAL; - } - - return 0; -} - -int nla_parse(struct nlattr **tb, int maxtype, const struct nlattr *head, - int len, const struct nla_policy *policy, - struct netlink_ext_ack *extack) -{ - return __nla_parse(tb, maxtype, head, len, false, policy, extack); -} -EXPORT_SYMBOL(nla_parse); - -int nla_parse_strict(struct nlattr **tb, int maxtype, const struct nlattr *head, - int len, const struct nla_policy *policy, - struct netlink_ext_ack *extack) +int __nla_parse(struct nlattr **tb, int maxtype, + const struct nlattr *head, int len, + const struct nla_policy *policy, unsigned int validate, + struct netlink_ext_ack *extack) { - return __nla_parse(tb, maxtype, head, len, true, policy, extack); + return __nla_validate_parse(head, len, maxtype, policy, validate, + extack, tb); } -EXPORT_SYMBOL(nla_parse_strict); +EXPORT_SYMBOL(__nla_parse); /** * nla_find - Find a specific attribute in a stream of attributes diff --git a/lib/packing.c b/lib/packing.c new file mode 100644 index 0000000000000000000000000000000000000000..50d1e9f2f5a77338dad83d3951c3d5a75947924a --- /dev/null +++ b/lib/packing.c @@ -0,0 +1,213 @@ +// SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0 +/* Copyright (c) 2016-2018, NXP Semiconductors + * Copyright (c) 2018-2019, Vladimir Oltean + */ +#include +#include +#include +#include +#include + +static int get_le_offset(int offset) +{ + int closest_multiple_of_4; + + closest_multiple_of_4 = (offset / 4) * 4; + offset -= closest_multiple_of_4; + return closest_multiple_of_4 + (3 - offset); +} + +static int get_reverse_lsw32_offset(int offset, size_t len) +{ + int closest_multiple_of_4; + int word_index; + + word_index = offset / 4; + closest_multiple_of_4 = word_index * 4; + offset -= closest_multiple_of_4; + word_index = (len / 4) - word_index - 1; + return word_index * 4 + offset; +} + +static u64 bit_reverse(u64 val, unsigned int width) +{ + u64 new_val = 0; + unsigned int bit; + unsigned int i; + + for (i = 0; i < width; i++) { + bit = (val & (1 << i)) != 0; + new_val |= (bit << (width - i - 1)); + } + return new_val; +} + +static void adjust_for_msb_right_quirk(u64 *to_write, int *box_start_bit, + int *box_end_bit, u8 *box_mask) +{ + int box_bit_width = *box_start_bit - *box_end_bit + 1; + int new_box_start_bit, new_box_end_bit; + + *to_write >>= *box_end_bit; + *to_write = bit_reverse(*to_write, box_bit_width); + *to_write <<= *box_end_bit; + + new_box_end_bit = box_bit_width - *box_start_bit - 1; + new_box_start_bit = box_bit_width - *box_end_bit - 1; + *box_mask = GENMASK_ULL(new_box_start_bit, new_box_end_bit); + *box_start_bit = new_box_start_bit; + *box_end_bit = new_box_end_bit; +} + +/** + * packing - Convert numbers (currently u64) between a packed and an unpacked + * format. Unpacked means laid out in memory in the CPU's native + * understanding of integers, while packed means anything else that + * requires translation. + * + * @pbuf: Pointer to a buffer holding the packed value. + * @uval: Pointer to an u64 holding the unpacked value. + * @startbit: The index (in logical notation, compensated for quirks) where + * the packed value starts within pbuf. Must be larger than, or + * equal to, endbit. + * @endbit: The index (in logical notation, compensated for quirks) where + * the packed value ends within pbuf. Must be smaller than, or equal + * to, startbit. + * @op: If PACK, then uval will be treated as const pointer and copied (packed) + * into pbuf, between startbit and endbit. + * If UNPACK, then pbuf will be treated as const pointer and the logical + * value between startbit and endbit will be copied (unpacked) to uval. + * @quirks: A bit mask of QUIRK_LITTLE_ENDIAN, QUIRK_LSW32_IS_FIRST and + * QUIRK_MSB_ON_THE_RIGHT. + * + * Return: 0 on success, EINVAL or ERANGE if called incorrectly. Assuming + * correct usage, return code may be discarded. + * If op is PACK, pbuf is modified. + * If op is UNPACK, uval is modified. + */ +int packing(void *pbuf, u64 *uval, int startbit, int endbit, size_t pbuflen, + enum packing_op op, u8 quirks) +{ + /* Number of bits for storing "uval" + * also width of the field to access in the pbuf + */ + u64 value_width; + /* Logical byte indices corresponding to the + * start and end of the field. + */ + int plogical_first_u8, plogical_last_u8, box; + + /* startbit is expected to be larger than endbit */ + if (startbit < endbit) + /* Invalid function call */ + return -EINVAL; + + value_width = startbit - endbit + 1; + if (value_width > 64) + return -ERANGE; + + /* Check if "uval" fits in "value_width" bits. + * If value_width is 64, the check will fail, but any + * 64-bit uval will surely fit. + */ + if (op == PACK && value_width < 64 && (*uval >= (1ull << value_width))) + /* Cannot store "uval" inside "value_width" bits. + * Truncating "uval" is most certainly not desirable, + * so simply erroring out is appropriate. + */ + return -ERANGE; + + /* Initialize parameter */ + if (op == UNPACK) + *uval = 0; + + /* Iterate through an idealistic view of the pbuf as an u64 with + * no quirks, u8 by u8 (aligned at u8 boundaries), from high to low + * logical bit significance. "box" denotes the current logical u8. + */ + plogical_first_u8 = startbit / 8; + plogical_last_u8 = endbit / 8; + + for (box = plogical_first_u8; box >= plogical_last_u8; box--) { + /* Bit indices into the currently accessed 8-bit box */ + int box_start_bit, box_end_bit, box_addr; + u8 box_mask; + /* Corresponding bits from the unpacked u64 parameter */ + int proj_start_bit, proj_end_bit; + u64 proj_mask; + + /* This u8 may need to be accessed in its entirety + * (from bit 7 to bit 0), or not, depending on the + * input arguments startbit and endbit. + */ + if (box == plogical_first_u8) + box_start_bit = startbit % 8; + else + box_start_bit = 7; + if (box == plogical_last_u8) + box_end_bit = endbit % 8; + else + box_end_bit = 0; + + /* We have determined the box bit start and end. + * Now we calculate where this (masked) u8 box would fit + * in the unpacked (CPU-readable) u64 - the u8 box's + * projection onto the unpacked u64. Though the + * box is u8, the projection is u64 because it may fall + * anywhere within the unpacked u64. + */ + proj_start_bit = ((box * 8) + box_start_bit) - endbit; + proj_end_bit = ((box * 8) + box_end_bit) - endbit; + proj_mask = GENMASK_ULL(proj_start_bit, proj_end_bit); + box_mask = GENMASK_ULL(box_start_bit, box_end_bit); + + /* Determine the offset of the u8 box inside the pbuf, + * adjusted for quirks. The adjusted box_addr will be used for + * effective addressing inside the pbuf (so it's not + * logical any longer). + */ + box_addr = pbuflen - box - 1; + if (quirks & QUIRK_LITTLE_ENDIAN) + box_addr = get_le_offset(box_addr); + if (quirks & QUIRK_LSW32_IS_FIRST) + box_addr = get_reverse_lsw32_offset(box_addr, + pbuflen); + + if (op == UNPACK) { + u64 pval; + + /* Read from pbuf, write to uval */ + pval = ((u8 *)pbuf)[box_addr] & box_mask; + if (quirks & QUIRK_MSB_ON_THE_RIGHT) + adjust_for_msb_right_quirk(&pval, + &box_start_bit, + &box_end_bit, + &box_mask); + + pval >>= box_end_bit; + pval <<= proj_end_bit; + *uval &= ~proj_mask; + *uval |= pval; + } else { + u64 pval; + + /* Write to pbuf, read from uval */ + pval = (*uval) & proj_mask; + pval >>= proj_end_bit; + if (quirks & QUIRK_MSB_ON_THE_RIGHT) + adjust_for_msb_right_quirk(&pval, + &box_start_bit, + &box_end_bit, + &box_mask); + + pval <<= box_end_bit; + ((u8 *)pbuf)[box_addr] &= ~box_mask; + ((u8 *)pbuf)[box_addr] |= pval; + } + } + return 0; +} +EXPORT_SYMBOL(packing); + +MODULE_LICENSE("GPL v2"); +MODULE_DESCRIPTION("Generic bitfield packing and unpacking"); diff --git a/lib/rhashtable.c b/lib/rhashtable.c index 97f59abc3e92583917769f232bdd9f7d5e67520f..6529fe1b45c13421d69665326ba620e7c39d01e0 100644 --- a/lib/rhashtable.c +++ b/lib/rhashtable.c @@ -31,11 +31,10 @@ #define HASH_DEFAULT_SIZE 64UL #define HASH_MIN_SIZE 4U -#define BUCKET_LOCKS_PER_CPU 32UL union nested_table { union nested_table __rcu *table; - struct rhash_head __rcu *bucket; + struct rhash_lock_head __rcu *bucket; }; static u32 head_hashfn(struct rhashtable *ht, @@ -56,9 +55,11 @@ EXPORT_SYMBOL_GPL(lockdep_rht_mutex_is_held); int lockdep_rht_bucket_is_held(const struct bucket_table *tbl, u32 hash) { - spinlock_t *lock = rht_bucket_lock(tbl, hash); - - return (debug_locks) ? lockdep_is_held(lock) : 1; + if (!debug_locks) + return 1; + if (unlikely(tbl->nest)) + return 1; + return bit_spin_is_locked(0, (unsigned long *)&tbl->buckets[hash]); } EXPORT_SYMBOL_GPL(lockdep_rht_bucket_is_held); #else @@ -104,7 +105,6 @@ static void bucket_table_free(const struct bucket_table *tbl) if (tbl->nest) nested_bucket_table_free(tbl); - free_bucket_spinlocks(tbl->locks); kvfree(tbl); } @@ -131,9 +131,11 @@ static union nested_table *nested_table_alloc(struct rhashtable *ht, INIT_RHT_NULLS_HEAD(ntbl[i].bucket); } - rcu_assign_pointer(*prev, ntbl); - - return ntbl; + if (cmpxchg(prev, NULL, ntbl) == NULL) + return ntbl; + /* Raced with another thread. */ + kfree(ntbl); + return rcu_dereference(*prev); } static struct bucket_table *nested_bucket_table_alloc(struct rhashtable *ht, @@ -169,11 +171,11 @@ static struct bucket_table *bucket_table_alloc(struct rhashtable *ht, gfp_t gfp) { struct bucket_table *tbl = NULL; - size_t size, max_locks; + size_t size; int i; + static struct lock_class_key __key; - size = sizeof(*tbl) + nbuckets * sizeof(tbl->buckets[0]); - tbl = kvzalloc(size, gfp); + tbl = kvzalloc(struct_size(tbl, buckets, nbuckets), gfp); size = nbuckets; @@ -185,18 +187,11 @@ static struct bucket_table *bucket_table_alloc(struct rhashtable *ht, if (tbl == NULL) return NULL; - tbl->size = size; - - max_locks = size >> 1; - if (tbl->nest) - max_locks = min_t(size_t, max_locks, 1U << tbl->nest); + lockdep_init_map(&tbl->dep_map, "rhashtable_bucket", &__key, 0); - if (alloc_bucket_spinlocks(&tbl->locks, &tbl->locks_mask, max_locks, - ht->p.locks_mul, gfp) < 0) { - bucket_table_free(tbl); - return NULL; - } + tbl->size = size; + rcu_head_init(&tbl->rcu); INIT_LIST_HEAD(&tbl->walkers); tbl->hash_rnd = get_random_u32(); @@ -220,14 +215,15 @@ static struct bucket_table *rhashtable_last_table(struct rhashtable *ht, return new_tbl; } -static int rhashtable_rehash_one(struct rhashtable *ht, unsigned int old_hash) +static int rhashtable_rehash_one(struct rhashtable *ht, + struct rhash_lock_head __rcu **bkt, + unsigned int old_hash) { struct bucket_table *old_tbl = rht_dereference(ht->tbl, ht); struct bucket_table *new_tbl = rhashtable_last_table(ht, old_tbl); - struct rhash_head __rcu **pprev = rht_bucket_var(old_tbl, old_hash); int err = -EAGAIN; struct rhash_head *head, *next, *entry; - spinlock_t *new_bucket_lock; + struct rhash_head __rcu **pprev = NULL; unsigned int new_hash; if (new_tbl->nest) @@ -235,7 +231,8 @@ static int rhashtable_rehash_one(struct rhashtable *ht, unsigned int old_hash) err = -ENOENT; - rht_for_each(entry, old_tbl, old_hash) { + rht_for_each_from(entry, rht_ptr(bkt, old_tbl, old_hash), + old_tbl, old_hash) { err = 0; next = rht_dereference_bucket(entry->next, old_tbl, old_hash); @@ -250,18 +247,19 @@ static int rhashtable_rehash_one(struct rhashtable *ht, unsigned int old_hash) new_hash = head_hashfn(ht, new_tbl, entry); - new_bucket_lock = rht_bucket_lock(new_tbl, new_hash); + rht_lock_nested(new_tbl, &new_tbl->buckets[new_hash], SINGLE_DEPTH_NESTING); - spin_lock_nested(new_bucket_lock, SINGLE_DEPTH_NESTING); - head = rht_dereference_bucket(new_tbl->buckets[new_hash], - new_tbl, new_hash); + head = rht_ptr(new_tbl->buckets + new_hash, new_tbl, new_hash); RCU_INIT_POINTER(entry->next, head); - rcu_assign_pointer(new_tbl->buckets[new_hash], entry); - spin_unlock(new_bucket_lock); + rht_assign_unlock(new_tbl, &new_tbl->buckets[new_hash], entry); - rcu_assign_pointer(*pprev, next); + if (pprev) + rcu_assign_pointer(*pprev, next); + else + /* Need to preserved the bit lock. */ + rht_assign_locked(bkt, next); out: return err; @@ -271,20 +269,19 @@ static int rhashtable_rehash_chain(struct rhashtable *ht, unsigned int old_hash) { struct bucket_table *old_tbl = rht_dereference(ht->tbl, ht); - spinlock_t *old_bucket_lock; + struct rhash_lock_head __rcu **bkt = rht_bucket_var(old_tbl, old_hash); int err; - old_bucket_lock = rht_bucket_lock(old_tbl, old_hash); + if (!bkt) + return 0; + rht_lock(old_tbl, bkt); - spin_lock_bh(old_bucket_lock); - while (!(err = rhashtable_rehash_one(ht, old_hash))) + while (!(err = rhashtable_rehash_one(ht, bkt, old_hash))) ; - if (err == -ENOENT) { - old_tbl->rehash++; + if (err == -ENOENT) err = 0; - } - spin_unlock_bh(old_bucket_lock); + rht_unlock(old_tbl, bkt); return err; } @@ -330,13 +327,16 @@ static int rhashtable_rehash_table(struct rhashtable *ht) spin_lock(&ht->lock); list_for_each_entry(walker, &old_tbl->walkers, list) walker->tbl = NULL; - spin_unlock(&ht->lock); /* Wait for readers. All new readers will see the new * table, and thus no references to the old table will * remain. + * We do this inside the locked region so that + * rhashtable_walk_stop() can use rcu_head_after_call_rcu() + * to check if it should not re-link the table. */ call_rcu(&old_tbl->rcu, bucket_table_free_rcu); + spin_unlock(&ht->lock); return rht_dereference(new_tbl->future_tbl, ht) ? -EAGAIN : 0; } @@ -478,6 +478,7 @@ fail: } static void *rhashtable_lookup_one(struct rhashtable *ht, + struct rhash_lock_head __rcu **bkt, struct bucket_table *tbl, unsigned int hash, const void *key, struct rhash_head *obj) { @@ -485,13 +486,12 @@ static void *rhashtable_lookup_one(struct rhashtable *ht, .ht = ht, .key = key, }; - struct rhash_head __rcu **pprev; + struct rhash_head __rcu **pprev = NULL; struct rhash_head *head; int elasticity; elasticity = RHT_ELASTICITY; - pprev = rht_bucket_var(tbl, hash); - rht_for_each_continue(head, *pprev, tbl, hash) { + rht_for_each_from(head, rht_ptr(bkt, tbl, hash), tbl, hash) { struct rhlist_head *list; struct rhlist_head *plist; @@ -513,7 +513,11 @@ static void *rhashtable_lookup_one(struct rhashtable *ht, RCU_INIT_POINTER(list->next, plist); head = rht_dereference_bucket(head->next, tbl, hash); RCU_INIT_POINTER(list->rhead.next, head); - rcu_assign_pointer(*pprev, obj); + if (pprev) + rcu_assign_pointer(*pprev, obj); + else + /* Need to preserve the bit lock */ + rht_assign_locked(bkt, obj); return NULL; } @@ -525,12 +529,12 @@ static void *rhashtable_lookup_one(struct rhashtable *ht, } static struct bucket_table *rhashtable_insert_one(struct rhashtable *ht, + struct rhash_lock_head __rcu **bkt, struct bucket_table *tbl, unsigned int hash, struct rhash_head *obj, void *data) { - struct rhash_head __rcu **pprev; struct bucket_table *new_tbl; struct rhash_head *head; @@ -553,11 +557,7 @@ static struct bucket_table *rhashtable_insert_one(struct rhashtable *ht, if (unlikely(rht_grow_above_100(ht, tbl))) return ERR_PTR(-EAGAIN); - pprev = rht_bucket_insert(ht, tbl, hash); - if (!pprev) - return ERR_PTR(-ENOMEM); - - head = rht_dereference_bucket(*pprev, tbl, hash); + head = rht_ptr(bkt, tbl, hash); RCU_INIT_POINTER(obj->next, head); if (ht->rhlist) { @@ -567,7 +567,10 @@ static struct bucket_table *rhashtable_insert_one(struct rhashtable *ht, RCU_INIT_POINTER(list->next, NULL); } - rcu_assign_pointer(*pprev, obj); + /* bkt is always the head of the list, so it holds + * the lock, which we need to preserve + */ + rht_assign_locked(bkt, obj); atomic_inc(&ht->nelems); if (rht_grow_above_75(ht, tbl)) @@ -581,47 +584,35 @@ static void *rhashtable_try_insert(struct rhashtable *ht, const void *key, { struct bucket_table *new_tbl; struct bucket_table *tbl; + struct rhash_lock_head __rcu **bkt; unsigned int hash; - spinlock_t *lock; void *data; - tbl = rcu_dereference(ht->tbl); - - /* All insertions must grab the oldest table containing - * the hashed bucket that is yet to be rehashed. - */ - for (;;) { - hash = rht_head_hashfn(ht, tbl, obj, ht->p); - lock = rht_bucket_lock(tbl, hash); - spin_lock_bh(lock); - - if (tbl->rehash <= hash) - break; - - spin_unlock_bh(lock); - tbl = rht_dereference_rcu(tbl->future_tbl, ht); - } - - data = rhashtable_lookup_one(ht, tbl, hash, key, obj); - new_tbl = rhashtable_insert_one(ht, tbl, hash, obj, data); - if (PTR_ERR(new_tbl) != -EEXIST) - data = ERR_CAST(new_tbl); + new_tbl = rcu_dereference(ht->tbl); - while (!IS_ERR_OR_NULL(new_tbl)) { + do { tbl = new_tbl; hash = rht_head_hashfn(ht, tbl, obj, ht->p); - spin_lock_nested(rht_bucket_lock(tbl, hash), - SINGLE_DEPTH_NESTING); - - data = rhashtable_lookup_one(ht, tbl, hash, key, obj); - new_tbl = rhashtable_insert_one(ht, tbl, hash, obj, data); - if (PTR_ERR(new_tbl) != -EEXIST) - data = ERR_CAST(new_tbl); - - spin_unlock(rht_bucket_lock(tbl, hash)); - } - - spin_unlock_bh(lock); + if (rcu_access_pointer(tbl->future_tbl)) + /* Failure is OK */ + bkt = rht_bucket_var(tbl, hash); + else + bkt = rht_bucket_insert(ht, tbl, hash); + if (bkt == NULL) { + new_tbl = rht_dereference_rcu(tbl->future_tbl, ht); + data = ERR_PTR(-EAGAIN); + } else { + rht_lock(tbl, bkt); + data = rhashtable_lookup_one(ht, bkt, tbl, + hash, key, obj); + new_tbl = rhashtable_insert_one(ht, bkt, tbl, + hash, obj, data); + if (PTR_ERR(new_tbl) != -EEXIST) + data = ERR_CAST(new_tbl); + + rht_unlock(tbl, bkt); + } + } while (!IS_ERR_OR_NULL(new_tbl)); if (PTR_ERR(data) == -EAGAIN) data = ERR_PTR(rhashtable_insert_rehash(ht, tbl) ?: @@ -943,10 +934,11 @@ void rhashtable_walk_stop(struct rhashtable_iter *iter) ht = iter->ht; spin_lock(&ht->lock); - if (tbl->rehash < tbl->size) - list_add(&iter->walker.list, &tbl->walkers); - else + if (rcu_head_after_call_rcu(&tbl->rcu, bucket_table_free_rcu)) + /* This bucket table is being freed, don't re-link it. */ iter->walker.tbl = NULL; + else + list_add(&iter->walker.list, &tbl->walkers); spin_unlock(&ht->lock); out: @@ -1046,11 +1038,6 @@ int rhashtable_init(struct rhashtable *ht, size = rounded_hashtable_size(&ht->p); - if (params->locks_mul) - ht->p.locks_mul = roundup_pow_of_two(params->locks_mul); - else - ht->p.locks_mul = BUCKET_LOCKS_PER_CPU; - ht->key_len = ht->p.key_len; if (!params->hashfn) { ht->p.hashfn = jhash; @@ -1152,7 +1139,7 @@ restart: struct rhash_head *pos, *next; cond_resched(); - for (pos = rht_dereference(*rht_bucket(tbl, i), ht), + for (pos = rht_ptr_exclusive(rht_bucket(tbl, i)), next = !rht_is_a_nulls(pos) ? rht_dereference(pos->next, ht) : NULL; !rht_is_a_nulls(pos); @@ -1179,11 +1166,10 @@ void rhashtable_destroy(struct rhashtable *ht) } EXPORT_SYMBOL_GPL(rhashtable_destroy); -struct rhash_head __rcu **rht_bucket_nested(const struct bucket_table *tbl, - unsigned int hash) +struct rhash_lock_head __rcu **__rht_bucket_nested(const struct bucket_table *tbl, + unsigned int hash) { const unsigned int shift = PAGE_SHIFT - ilog2(sizeof(void *)); - static struct rhash_head __rcu *rhnull; unsigned int index = hash & ((1 << tbl->nest) - 1); unsigned int size = tbl->size >> tbl->nest; unsigned int subhash = hash; @@ -1201,20 +1187,28 @@ struct rhash_head __rcu **rht_bucket_nested(const struct bucket_table *tbl, subhash >>= shift; } - if (!ntbl) { - if (!rhnull) - INIT_RHT_NULLS_HEAD(rhnull); - return &rhnull; - } + if (!ntbl) + return NULL; return &ntbl[subhash].bucket; } +EXPORT_SYMBOL_GPL(__rht_bucket_nested); + +struct rhash_lock_head __rcu **rht_bucket_nested(const struct bucket_table *tbl, + unsigned int hash) +{ + static struct rhash_lock_head __rcu *rhnull; + + if (!rhnull) + INIT_RHT_NULLS_HEAD(rhnull); + return __rht_bucket_nested(tbl, hash) ?: &rhnull; +} EXPORT_SYMBOL_GPL(rht_bucket_nested); -struct rhash_head __rcu **rht_bucket_nested_insert(struct rhashtable *ht, - struct bucket_table *tbl, - unsigned int hash) +struct rhash_lock_head __rcu **rht_bucket_nested_insert(struct rhashtable *ht, + struct bucket_table *tbl, + unsigned int hash) { const unsigned int shift = PAGE_SHIFT - ilog2(sizeof(void *)); unsigned int index = hash & ((1 << tbl->nest) - 1); diff --git a/lib/test_rhashtable.c b/lib/test_rhashtable.c index 3bd2e91bfc297245c85713fe55637fc7b3b26961..084fe5a6ac57cd9bed1101f0e8acf16a4d521ce2 100644 --- a/lib/test_rhashtable.c +++ b/lib/test_rhashtable.c @@ -500,7 +500,7 @@ static unsigned int __init print_ht(struct rhltable *rhlt) struct rhash_head *pos, *next; struct test_obj_rhl *p; - pos = rht_dereference(tbl->buckets[i], ht); + pos = rht_ptr_exclusive(tbl->buckets + i); next = !rht_is_a_nulls(pos) ? rht_dereference(pos->next, ht) : NULL; if (!rht_is_a_nulls(pos)) { diff --git a/net/6lowpan/nhc.c b/net/6lowpan/nhc.c index 4fa2fdda174d0139e669bd282ab4362aea41f512..9e56fb98f33cf0682991774293f77749a272478c 100644 --- a/net/6lowpan/nhc.c +++ b/net/6lowpan/nhc.c @@ -18,7 +18,7 @@ #include "nhc.h" static struct rb_root rb_root = RB_ROOT; -static struct lowpan_nhc *lowpan_nexthdr_nhcs[NEXTHDR_MAX]; +static struct lowpan_nhc *lowpan_nexthdr_nhcs[NEXTHDR_MAX + 1]; static DEFINE_SPINLOCK(lowpan_nhc_lock); static int lowpan_nhc_insert(struct lowpan_nhc *nhc) diff --git a/net/8021q/vlan.c b/net/8021q/vlan.c index dc4411165e437a2f6e962d7466612eec9706631d..1f99678751dff24d06a2e27a80d0d74d2232ca63 100644 --- a/net/8021q/vlan.c +++ b/net/8021q/vlan.c @@ -75,6 +75,14 @@ static int vlan_group_prealloc_vid(struct vlan_group *vg, return 0; } +static void vlan_stacked_transfer_operstate(const struct net_device *rootdev, + struct net_device *dev, + struct vlan_dev_priv *vlan) +{ + if (!(vlan->flags & VLAN_FLAG_BRIDGE_BINDING)) + netif_stacked_transfer_operstate(rootdev, dev); +} + void unregister_vlan_dev(struct net_device *dev, struct list_head *head) { struct vlan_dev_priv *vlan = vlan_dev_priv(dev); @@ -180,7 +188,7 @@ int register_vlan_dev(struct net_device *dev, struct netlink_ext_ack *extack) /* Account for reference in struct vlan_dev_priv */ dev_hold(real_dev); - netif_stacked_transfer_operstate(real_dev, dev); + vlan_stacked_transfer_operstate(real_dev, dev, vlan); linkwatch_fire_event(dev); /* _MUST_ call rfc2863_policy() */ /* So, got the sucker initialized, now lets place @@ -399,7 +407,8 @@ static int vlan_device_event(struct notifier_block *unused, unsigned long event, case NETDEV_CHANGE: /* Propagate real device state to vlan devices */ vlan_group_for_each_dev(grp, i, vlandev) - netif_stacked_transfer_operstate(dev, vlandev); + vlan_stacked_transfer_operstate(dev, vlandev, + vlan_dev_priv(vlandev)); break; case NETDEV_CHANGEADDR: @@ -446,7 +455,8 @@ static int vlan_device_event(struct notifier_block *unused, unsigned long event, dev_close_many(&close_list, false); list_for_each_entry_safe(vlandev, tmp, &close_list, close_list) { - netif_stacked_transfer_operstate(dev, vlandev); + vlan_stacked_transfer_operstate(dev, vlandev, + vlan_dev_priv(vlandev)); list_del_init(&vlandev->close_list); } list_del(&close_list); @@ -463,7 +473,7 @@ static int vlan_device_event(struct notifier_block *unused, unsigned long event, if (!(vlan->flags & VLAN_FLAG_LOOSE_BINDING)) dev_change_flags(vlandev, flgs | IFF_UP, extack); - netif_stacked_transfer_operstate(dev, vlandev); + vlan_stacked_transfer_operstate(dev, vlandev, vlan); } break; diff --git a/net/8021q/vlan_dev.c b/net/8021q/vlan_dev.c index 8d77b6ee4477df71bc466c057338ea4ae62dcd88..f044ae56a3132624127d4100cd8f9dc3e4ed6b05 100644 --- a/net/8021q/vlan_dev.c +++ b/net/8021q/vlan_dev.c @@ -223,7 +223,8 @@ int vlan_dev_change_flags(const struct net_device *dev, u32 flags, u32 mask) u32 old_flags = vlan->flags; if (mask & ~(VLAN_FLAG_REORDER_HDR | VLAN_FLAG_GVRP | - VLAN_FLAG_LOOSE_BINDING | VLAN_FLAG_MVRP)) + VLAN_FLAG_LOOSE_BINDING | VLAN_FLAG_MVRP | + VLAN_FLAG_BRIDGE_BINDING)) return -EINVAL; vlan->flags = (old_flags & ~mask) | (flags & mask); @@ -296,7 +297,8 @@ static int vlan_dev_open(struct net_device *dev) if (vlan->flags & VLAN_FLAG_MVRP) vlan_mvrp_request_join(dev); - if (netif_carrier_ok(real_dev)) + if (netif_carrier_ok(real_dev) && + !(vlan->flags & VLAN_FLAG_BRIDGE_BINDING)) netif_carrier_on(dev); return 0; @@ -326,7 +328,8 @@ static int vlan_dev_stop(struct net_device *dev) if (!ether_addr_equal(dev->dev_addr, real_dev->dev_addr)) dev_uc_del(real_dev, dev->dev_addr); - netif_carrier_off(dev); + if (!(vlan->flags & VLAN_FLAG_BRIDGE_BINDING)) + netif_carrier_off(dev); return 0; } @@ -550,7 +553,8 @@ static const struct net_device_ops vlan_netdev_ops; static int vlan_dev_init(struct net_device *dev) { - struct net_device *real_dev = vlan_dev_priv(dev)->real_dev; + struct vlan_dev_priv *vlan = vlan_dev_priv(dev); + struct net_device *real_dev = vlan->real_dev; netif_carrier_off(dev); @@ -561,6 +565,9 @@ static int vlan_dev_init(struct net_device *dev) (1<<__LINK_STATE_DORMANT))) | (1<<__LINK_STATE_PRESENT); + if (vlan->flags & VLAN_FLAG_BRIDGE_BINDING) + dev->state |= (1 << __LINK_STATE_NOCARRIER); + dev->hw_features = NETIF_F_HW_CSUM | NETIF_F_SG | NETIF_F_FRAGLIST | NETIF_F_GSO_SOFTWARE | NETIF_F_GSO_ENCAP_ALL | @@ -591,8 +598,7 @@ static int vlan_dev_init(struct net_device *dev) #endif dev->needed_headroom = real_dev->needed_headroom; - if (vlan_hw_offload_capable(real_dev->features, - vlan_dev_priv(dev)->vlan_proto)) { + if (vlan_hw_offload_capable(real_dev->features, vlan->vlan_proto)) { dev->header_ops = &vlan_passthru_header_ops; dev->hard_header_len = real_dev->hard_header_len; } else { @@ -606,8 +612,8 @@ static int vlan_dev_init(struct net_device *dev) vlan_dev_set_lockdep_class(dev, vlan_dev_get_lock_subclass(dev)); - vlan_dev_priv(dev)->vlan_pcpu_stats = netdev_alloc_pcpu_stats(struct vlan_pcpu_stats); - if (!vlan_dev_priv(dev)->vlan_pcpu_stats) + vlan->vlan_pcpu_stats = netdev_alloc_pcpu_stats(struct vlan_pcpu_stats); + if (!vlan->vlan_pcpu_stats) return -ENOMEM; return 0; diff --git a/net/8021q/vlan_netlink.c b/net/8021q/vlan_netlink.c index 9b60c1e399e27e7f0eb300f30479e1f0553079d5..24eebbc9236458a15d26777aca32e2fac3a919c4 100644 --- a/net/8021q/vlan_netlink.c +++ b/net/8021q/vlan_netlink.c @@ -35,8 +35,8 @@ static inline int vlan_validate_qos_map(struct nlattr *attr) { if (!attr) return 0; - return nla_validate_nested(attr, IFLA_VLAN_QOS_MAX, vlan_map_policy, - NULL); + return nla_validate_nested_deprecated(attr, IFLA_VLAN_QOS_MAX, + vlan_map_policy, NULL); } static int vlan_validate(struct nlattr *tb[], struct nlattr *data[], @@ -84,7 +84,8 @@ static int vlan_validate(struct nlattr *tb[], struct nlattr *data[], flags = nla_data(data[IFLA_VLAN_FLAGS]); if ((flags->flags & flags->mask) & ~(VLAN_FLAG_REORDER_HDR | VLAN_FLAG_GVRP | - VLAN_FLAG_LOOSE_BINDING | VLAN_FLAG_MVRP)) { + VLAN_FLAG_LOOSE_BINDING | VLAN_FLAG_MVRP | + VLAN_FLAG_BRIDGE_BINDING)) { NL_SET_ERR_MSG_MOD(extack, "Invalid VLAN flags"); return -EINVAL; } @@ -226,7 +227,7 @@ static int vlan_fill_info(struct sk_buff *skb, const struct net_device *dev) goto nla_put_failure; } if (vlan->nr_ingress_mappings) { - nest = nla_nest_start(skb, IFLA_VLAN_INGRESS_QOS); + nest = nla_nest_start_noflag(skb, IFLA_VLAN_INGRESS_QOS); if (nest == NULL) goto nla_put_failure; @@ -244,7 +245,7 @@ static int vlan_fill_info(struct sk_buff *skb, const struct net_device *dev) } if (vlan->nr_egress_mappings) { - nest = nla_nest_start(skb, IFLA_VLAN_EGRESS_QOS); + nest = nla_nest_start_noflag(skb, IFLA_VLAN_EGRESS_QOS); if (nest == NULL) goto nla_put_failure; diff --git a/net/Kconfig b/net/Kconfig index 1efe1f9ee492ea55f1227f2449f03883d318e810..3e8fdd688329cbbfd06063452a10a71583b7d49e 100644 --- a/net/Kconfig +++ b/net/Kconfig @@ -429,11 +429,8 @@ config NET_SOCK_MSG with the help of BPF programs. config NET_DEVLINK - bool "Network physical/parent device Netlink interface" - help - Network physical/parent device Netlink interface provides - infrastructure to support access to physical chip-wide config and - monitoring. + bool + default n config PAGE_POOL bool diff --git a/net/appletalk/ddp.c b/net/appletalk/ddp.c index dbe8b1993be9eec3c84e2be09602a5f471ed3f34..a2555023c654617029d3e0fed30ca7e4e75632ff 100644 --- a/net/appletalk/ddp.c +++ b/net/appletalk/ddp.c @@ -1806,12 +1806,6 @@ static int atalk_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) rc = put_user(amount, (int __user *)argp); break; } - case SIOCGSTAMP: - rc = sock_get_timestamp(sk, argp); - break; - case SIOCGSTAMPNS: - rc = sock_get_timestampns(sk, argp); - break; /* Routing */ case SIOCADDRT: case SIOCDELRT: @@ -1871,6 +1865,7 @@ static const struct proto_ops atalk_dgram_ops = { .getname = atalk_getname, .poll = datagram_poll, .ioctl = atalk_ioctl, + .gettstamp = sock_gettstamp, #ifdef CONFIG_COMPAT .compat_ioctl = atalk_compat_ioctl, #endif diff --git a/net/atm/clip.c b/net/atm/clip.c index d795b9c5aea4a4e35021d9db2e10254036df55fe..b9e67e589a7b43eeda8268f329adf607e167e80f 100644 --- a/net/atm/clip.c +++ b/net/atm/clip.c @@ -345,8 +345,8 @@ static netdev_tx_t clip_start_xmit(struct sk_buff *skb, return NETDEV_TX_OK; } rt = (struct rtable *) dst; - if (rt->rt_gateway) - daddr = &rt->rt_gateway; + if (rt->rt_gw_family == AF_INET) + daddr = &rt->rt_gw4; else daddr = &ip_hdr(skb)->daddr; n = dst_neigh_lookup(dst, daddr); diff --git a/net/atm/ioctl.c b/net/atm/ioctl.c index 2ff0e5e470e3d43647bdd6a3b30bdc87bfbf2511..d955b683aa7c468eb4be08cffb735e40d329be62 100644 --- a/net/atm/ioctl.c +++ b/net/atm/ioctl.c @@ -81,22 +81,6 @@ static int do_vcc_ioctl(struct socket *sock, unsigned int cmd, (int __user *)argp) ? -EFAULT : 0; goto done; } - case SIOCGSTAMP: /* borrowed from IP */ -#ifdef CONFIG_COMPAT - if (compat) - error = compat_sock_get_timestamp(sk, argp); - else -#endif - error = sock_get_timestamp(sk, argp); - goto done; - case SIOCGSTAMPNS: /* borrowed from IP */ -#ifdef CONFIG_COMPAT - if (compat) - error = compat_sock_get_timestampns(sk, argp); - else -#endif - error = sock_get_timestampns(sk, argp); - goto done; case ATM_SETSC: net_warn_ratelimited("ATM_SETSC is obsolete; used by %s:%d\n", current->comm, task_pid_nr(current)); diff --git a/net/atm/lec.c b/net/atm/lec.c index ad4f829193f053c8a0c0846f1e9f619617dcd18e..a0311493b01b0494aadd873c1fb245d389c97d3b 100644 --- a/net/atm/lec.c +++ b/net/atm/lec.c @@ -726,9 +726,7 @@ static int lecd_attach(struct atm_vcc *vcc, int arg) struct lec_priv *priv; if (arg < 0) - i = 0; - else - i = arg; + arg = 0; if (arg >= MAX_LEC_ITF) return -EINVAL; i = array_index_nospec(arg, MAX_LEC_ITF); diff --git a/net/atm/pvc.c b/net/atm/pvc.c index 2cb10af16afcf8eeb925bfe1aab33e839821109a..02bd2a436bdf9ec5f3c235ad390bc99109f16bad 100644 --- a/net/atm/pvc.c +++ b/net/atm/pvc.c @@ -118,6 +118,7 @@ static const struct proto_ops pvc_proto_ops = { #ifdef CONFIG_COMPAT .compat_ioctl = vcc_compat_ioctl, #endif + .gettstamp = sock_gettstamp, .listen = sock_no_listen, .shutdown = pvc_shutdown, .setsockopt = pvc_setsockopt, diff --git a/net/atm/svc.c b/net/atm/svc.c index 2f91b766ac423c97a0b9c1fd340222e31b17eefa..908cbb8654f532548b20f166492a2e02de6324d6 100644 --- a/net/atm/svc.c +++ b/net/atm/svc.c @@ -641,6 +641,7 @@ static const struct proto_ops svc_proto_ops = { #ifdef CONFIG_COMPAT .compat_ioctl = svc_compat_ioctl, #endif + .gettstamp = sock_gettstamp, .listen = svc_listen, .shutdown = svc_shutdown, .setsockopt = svc_setsockopt, diff --git a/net/ax25/af_ax25.c b/net/ax25/af_ax25.c index 5d01edf8d819e89aec03df70e9f3bcf6340314cb..012c0b6fc4f6cc583e2f936468ba5dc67d9a5bfd 100644 --- a/net/ax25/af_ax25.c +++ b/net/ax25/af_ax25.c @@ -1714,14 +1714,6 @@ static int ax25_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) break; } - case SIOCGSTAMP: - res = sock_get_timestamp(sk, argp); - break; - - case SIOCGSTAMPNS: - res = sock_get_timestampns(sk, argp); - break; - case SIOCAX25ADDUID: /* Add a uid to the uid/call map table */ case SIOCAX25DELUID: /* Delete a uid from the uid/call map table */ case SIOCAX25GETUID: { @@ -1888,8 +1880,8 @@ static int ax25_info_show(struct seq_file *seq, void *v) * magic dev src_addr dest_addr,digi1,digi2,.. st vs vr va t1 t1 t2 t2 t3 t3 idle idle n2 n2 rtt window paclen Snd-Q Rcv-Q inode */ - seq_printf(seq, "%8.8lx %s %s%s ", - (long) ax25, + seq_printf(seq, "%p %s %s%s ", + ax25, ax25->ax25_dev == NULL? "???" : ax25->ax25_dev->dev->name, ax2asc(buf, &ax25->source_addr), ax25->iamdigi? "*":""); @@ -1950,6 +1942,7 @@ static const struct proto_ops ax25_proto_ops = { .getname = ax25_getname, .poll = datagram_poll, .ioctl = ax25_ioctl, + .gettstamp = sock_gettstamp, .listen = ax25_listen, .shutdown = ax25_shutdown, .setsockopt = ax25_setsockopt, diff --git a/net/batman-adv/Kconfig b/net/batman-adv/Kconfig index a31db5e9ac8e7186cccc49b186364c5d781bd8f2..a3d188dfbe75a4a94d69e5036ee22c306e8b085f 100644 --- a/net/batman-adv/Kconfig +++ b/net/batman-adv/Kconfig @@ -2,18 +2,6 @@ # Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: # # Marek Lindner, Simon Wunderlich -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of version 2 of the GNU General Public -# License as published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, see . # # B.A.T.M.A.N meshing protocol @@ -109,6 +97,18 @@ config BATMAN_ADV_DEBUG buffer. The output is controlled via the batadv netdev specific log_level setting. +config BATMAN_ADV_SYSFS + bool "batman-adv sysfs entries" + depends on BATMAN_ADV + default y + help + Say Y here if you want to enable batman-adv device configuration and + status interface through sysfs attributes. It is replaced by the + batadv generic netlink family but still used by various userspace + tools and scripts. + + If unsure, say Y. + config BATMAN_ADV_TRACING bool "B.A.T.M.A.N. tracing support" depends on BATMAN_ADV diff --git a/net/batman-adv/Makefile b/net/batman-adv/Makefile index a887ecc3efa124d6728859493510bb612323915a..fd63e116d9ff7ccc59f613d18707f80b21055b39 100644 --- a/net/batman-adv/Makefile +++ b/net/batman-adv/Makefile @@ -2,19 +2,6 @@ # Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: # # Marek Lindner, Simon Wunderlich -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of version 2 of the GNU General Public -# License as published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, see . -# obj-$(CONFIG_BATMAN_ADV) += batman-adv.o batman-adv-y += bat_algo.o @@ -41,7 +28,7 @@ batman-adv-y += originator.o batman-adv-y += routing.o batman-adv-y += send.o batman-adv-y += soft-interface.o -batman-adv-y += sysfs.o +batman-adv-$(CONFIG_BATMAN_ADV_SYSFS) += sysfs.o batman-adv-$(CONFIG_BATMAN_ADV_TRACING) += trace.o batman-adv-y += tp_meter.o batman-adv-y += translation-table.o diff --git a/net/batman-adv/bat_algo.c b/net/batman-adv/bat_algo.c index 7b7e15641fef4d1f12f7fd71753c8038e868b196..fa39eaaab9d7524d1c1c81c3f7f075777e102c7d 100644 --- a/net/batman-adv/bat_algo.c +++ b/net/batman-adv/bat_algo.c @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "main.h" diff --git a/net/batman-adv/bat_algo.h b/net/batman-adv/bat_algo.h index 25e7bb51928ccabd3e7aa0f5b31278556aed717a..cb7d57d16c9dd6425b4b32ab02764732803ad28a 100644 --- a/net/batman-adv/bat_algo.h +++ b/net/batman-adv/bat_algo.h @@ -2,18 +2,6 @@ /* Copyright (C) 2011-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Linus Lüssing - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_BAT_ALGO_H_ diff --git a/net/batman-adv/bat_iv_ogm.c b/net/batman-adv/bat_iv_ogm.c index de61091af6668734709ab5fc46a40358eff54b82..bd4138ddf7e09a0020d9842d603dc98f21e225c7 100644 --- a/net/batman-adv/bat_iv_ogm.c +++ b/net/batman-adv/bat_iv_ogm.c @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "bat_iv_ogm.h" diff --git a/net/batman-adv/bat_iv_ogm.h b/net/batman-adv/bat_iv_ogm.h index 785f6666273c58cfc628b3bf000171a654b892ad..c7a9ba305bfc8a18ae40d4db93ea4b66ecd7356c 100644 --- a/net/batman-adv/bat_iv_ogm.h +++ b/net/batman-adv/bat_iv_ogm.h @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_BAT_IV_OGM_H_ diff --git a/net/batman-adv/bat_v.c b/net/batman-adv/bat_v.c index 445594ed58afe55ef8284509b6ac856ba0604745..231b4aab4d8de491c6a687db0680e46c677ed514 100644 --- a/net/batman-adv/bat_v.c +++ b/net/batman-adv/bat_v.c @@ -2,18 +2,6 @@ /* Copyright (C) 2013-2019 B.A.T.M.A.N. contributors: * * Linus Lüssing, Marek Lindner - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "bat_v.h" diff --git a/net/batman-adv/bat_v.h b/net/batman-adv/bat_v.h index 465a4fc23354753cf58a08e3fab237b3a903fd3a..37833db098e631b09b2a8b2d8e134c5d6ba6e619 100644 --- a/net/batman-adv/bat_v.h +++ b/net/batman-adv/bat_v.h @@ -2,18 +2,6 @@ /* Copyright (C) 2011-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Linus Lüssing - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_BAT_V_H_ diff --git a/net/batman-adv/bat_v_elp.c b/net/batman-adv/bat_v_elp.c index d5df0114f08ac5331dcd46699551556ba60a4b80..2614a9caee008539cc489b71dabdc36ac0ae3752 100644 --- a/net/batman-adv/bat_v_elp.c +++ b/net/batman-adv/bat_v_elp.c @@ -2,18 +2,6 @@ /* Copyright (C) 2011-2019 B.A.T.M.A.N. contributors: * * Linus Lüssing, Marek Lindner - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "bat_v_elp.h" diff --git a/net/batman-adv/bat_v_elp.h b/net/batman-adv/bat_v_elp.h index 75f189ee4a1c5bea61aa4471292042937652e5ff..bb3d40f73bfebf1f0940b716709c75269444f3e4 100644 --- a/net/batman-adv/bat_v_elp.h +++ b/net/batman-adv/bat_v_elp.h @@ -2,18 +2,6 @@ /* Copyright (C) 2013-2019 B.A.T.M.A.N. contributors: * * Linus Lüssing, Marek Lindner - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_BAT_V_ELP_H_ diff --git a/net/batman-adv/bat_v_ogm.c b/net/batman-adv/bat_v_ogm.c index c9698ad41854cf4dd5ef2cd773d5dd239f457fa3..fad95ef64e01a9670ed66f9a81658718a14fc716 100644 --- a/net/batman-adv/bat_v_ogm.c +++ b/net/batman-adv/bat_v_ogm.c @@ -2,18 +2,6 @@ /* Copyright (C) 2013-2019 B.A.T.M.A.N. contributors: * * Antonio Quartulli - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "bat_v_ogm.h" diff --git a/net/batman-adv/bat_v_ogm.h b/net/batman-adv/bat_v_ogm.h index f67cf7ee06b281110811abc149d741be8968437d..616bf2ea8755742b688f07a1166b7cb98bf50dd4 100644 --- a/net/batman-adv/bat_v_ogm.h +++ b/net/batman-adv/bat_v_ogm.h @@ -2,18 +2,6 @@ /* Copyright (C) 2013-2019 B.A.T.M.A.N. contributors: * * Antonio Quartulli - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_BAT_V_OGM_H_ diff --git a/net/batman-adv/bitarray.c b/net/batman-adv/bitarray.c index 63e134e763e311f8edb030a77c9ead48d5e232d5..7f04a6acf14e5d0b638c3d091f1c1b7566c27b84 100644 --- a/net/batman-adv/bitarray.c +++ b/net/batman-adv/bitarray.c @@ -2,18 +2,6 @@ /* Copyright (C) 2006-2019 B.A.T.M.A.N. contributors: * * Simon Wunderlich, Marek Lindner - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "bitarray.h" diff --git a/net/batman-adv/bitarray.h b/net/batman-adv/bitarray.h index f3a05ad9afad9b07a3d05ed097f5ee37eb35e34a..84ad2d2b6ac931435d91c4b0decab24034b2e8d5 100644 --- a/net/batman-adv/bitarray.h +++ b/net/batman-adv/bitarray.h @@ -2,18 +2,6 @@ /* Copyright (C) 2006-2019 B.A.T.M.A.N. contributors: * * Simon Wunderlich, Marek Lindner - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_BITARRAY_H_ diff --git a/net/batman-adv/bridge_loop_avoidance.c b/net/batman-adv/bridge_loop_avoidance.c index 4fb01108e5f534b8a055edface0717f950ccdacc..663a53b6d36e65508b4296d86cecdd808c653836 100644 --- a/net/batman-adv/bridge_loop_avoidance.c +++ b/net/batman-adv/bridge_loop_avoidance.c @@ -2,18 +2,6 @@ /* Copyright (C) 2011-2019 B.A.T.M.A.N. contributors: * * Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "bridge_loop_avoidance.h" @@ -59,7 +47,6 @@ #include "netlink.h" #include "originator.h" #include "soft-interface.h" -#include "sysfs.h" #include "translation-table.h" static const u8 batadv_announce_mac[4] = {0x43, 0x05, 0x43, 0x05}; diff --git a/net/batman-adv/bridge_loop_avoidance.h b/net/batman-adv/bridge_loop_avoidance.h index 31771c751efb84009c7816b1ad3b02f88fe001f2..012d72c8d06463d64ed2374b0a9963577a5c25ab 100644 --- a/net/batman-adv/bridge_loop_avoidance.h +++ b/net/batman-adv/bridge_loop_avoidance.h @@ -2,18 +2,6 @@ /* Copyright (C) 2011-2019 B.A.T.M.A.N. contributors: * * Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_BLA_H_ diff --git a/net/batman-adv/debugfs.c b/net/batman-adv/debugfs.c index 3b9d1ad2f467dd12226adf6e12b709a9906fd865..d38d70ccdd5ae8aab3afa87b38c7f498a9d5f826 100644 --- a/net/batman-adv/debugfs.c +++ b/net/batman-adv/debugfs.c @@ -2,18 +2,6 @@ /* Copyright (C) 2010-2019 B.A.T.M.A.N. contributors: * * Marek Lindner - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "debugfs.h" diff --git a/net/batman-adv/debugfs.h b/net/batman-adv/debugfs.h index c0b8694041ece9397900506814304fac673a4f7e..7fac680cf74056a749760c00e9607adee7272ae4 100644 --- a/net/batman-adv/debugfs.h +++ b/net/batman-adv/debugfs.h @@ -2,18 +2,6 @@ /* Copyright (C) 2010-2019 B.A.T.M.A.N. contributors: * * Marek Lindner - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_DEBUGFS_H_ diff --git a/net/batman-adv/distributed-arp-table.c b/net/batman-adv/distributed-arp-table.c index 310a4f353008d55ddf5e64e1bd4d98b0daa6a3d8..b0af3a11d4069cb7e44419d4494f20dd653dbca8 100644 --- a/net/batman-adv/distributed-arp-table.c +++ b/net/batman-adv/distributed-arp-table.c @@ -2,18 +2,6 @@ /* Copyright (C) 2011-2019 B.A.T.M.A.N. contributors: * * Antonio Quartulli - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "distributed-arp-table.h" @@ -667,7 +655,7 @@ batadv_dat_select_candidates(struct batadv_priv *bat_priv, __be32 ip_dst, } /** - * batadv_dat_send_data() - send a payload to the selected candidates + * batadv_dat_forward_data() - copy and send payload to the selected candidates * @bat_priv: the bat priv with all the soft interface information * @skb: payload to send * @ip: the DHT key @@ -680,9 +668,9 @@ batadv_dat_select_candidates(struct batadv_priv *bat_priv, __be32 ip_dst, * Return: true if the packet is sent to at least one candidate, false * otherwise. */ -static bool batadv_dat_send_data(struct batadv_priv *bat_priv, - struct sk_buff *skb, __be32 ip, - unsigned short vid, int packet_subtype) +static bool batadv_dat_forward_data(struct batadv_priv *bat_priv, + struct sk_buff *skb, __be32 ip, + unsigned short vid, int packet_subtype) { int i; bool ret = false; @@ -1277,8 +1265,8 @@ bool batadv_dat_snoop_outgoing_arp_request(struct batadv_priv *bat_priv, ret = true; } else { /* Send the request to the DHT */ - ret = batadv_dat_send_data(bat_priv, skb, ip_dst, vid, - BATADV_P_DAT_DHT_GET); + ret = batadv_dat_forward_data(bat_priv, skb, ip_dst, vid, + BATADV_P_DAT_DHT_GET); } out: if (dat_entry) @@ -1392,8 +1380,10 @@ void batadv_dat_snoop_outgoing_arp_reply(struct batadv_priv *bat_priv, /* Send the ARP reply to the candidates for both the IP addresses that * the node obtained from the ARP reply */ - batadv_dat_send_data(bat_priv, skb, ip_src, vid, BATADV_P_DAT_DHT_PUT); - batadv_dat_send_data(bat_priv, skb, ip_dst, vid, BATADV_P_DAT_DHT_PUT); + batadv_dat_forward_data(bat_priv, skb, ip_src, vid, + BATADV_P_DAT_DHT_PUT); + batadv_dat_forward_data(bat_priv, skb, ip_dst, vid, + BATADV_P_DAT_DHT_PUT); } /** @@ -1444,7 +1434,6 @@ bool batadv_dat_snoop_incoming_arp_reply(struct batadv_priv *bat_priv, hw_src, &ip_src, hw_dst, &ip_dst, dat_entry->mac_addr, &dat_entry->ip); dropped = true; - goto out; } /* Update our internal cache with both the IP addresses the node got @@ -1453,6 +1442,9 @@ bool batadv_dat_snoop_incoming_arp_reply(struct batadv_priv *bat_priv, batadv_dat_entry_add(bat_priv, ip_src, hw_src, vid); batadv_dat_entry_add(bat_priv, ip_dst, hw_dst, vid); + if (dropped) + goto out; + /* If BLA is enabled, only forward ARP replies if we have claimed the * source of the ARP reply or if no one else of the same backbone has * already claimed that client. This prevents that different gateways @@ -1708,8 +1700,10 @@ static void batadv_dat_put_dhcp(struct batadv_priv *bat_priv, u8 *chaddr, batadv_dat_entry_add(bat_priv, yiaddr, chaddr, vid); batadv_dat_entry_add(bat_priv, ip_dst, hw_dst, vid); - batadv_dat_send_data(bat_priv, skb, yiaddr, vid, BATADV_P_DAT_DHT_PUT); - batadv_dat_send_data(bat_priv, skb, ip_dst, vid, BATADV_P_DAT_DHT_PUT); + batadv_dat_forward_data(bat_priv, skb, yiaddr, vid, + BATADV_P_DAT_DHT_PUT); + batadv_dat_forward_data(bat_priv, skb, ip_dst, vid, + BATADV_P_DAT_DHT_PUT); consume_skb(skb); diff --git a/net/batman-adv/distributed-arp-table.h b/net/batman-adv/distributed-arp-table.h index 68c0ff321acde30eb6bb162d80bb4918771cb00a..110c27447d706ea43894e44167a529c384576059 100644 --- a/net/batman-adv/distributed-arp-table.h +++ b/net/batman-adv/distributed-arp-table.h @@ -2,18 +2,6 @@ /* Copyright (C) 2011-2019 B.A.T.M.A.N. contributors: * * Antonio Quartulli - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_DISTRIBUTED_ARP_TABLE_H_ diff --git a/net/batman-adv/fragmentation.c b/net/batman-adv/fragmentation.c index b506d15b823032fb051ab96d49a37b39c5f27074..385fccdcf69d05d63f59d024dfb721b1e8f1c8b9 100644 --- a/net/batman-adv/fragmentation.c +++ b/net/batman-adv/fragmentation.c @@ -2,18 +2,6 @@ /* Copyright (C) 2013-2019 B.A.T.M.A.N. contributors: * * Martin Hundebøll - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "fragmentation.h" diff --git a/net/batman-adv/fragmentation.h b/net/batman-adv/fragmentation.h index abdac26579bfb7fc0d9563bea22fcb9f04bc5323..d6074ba2ada7ee7ad06ada719a4d3b74b31cbd84 100644 --- a/net/batman-adv/fragmentation.h +++ b/net/batman-adv/fragmentation.h @@ -2,18 +2,6 @@ /* Copyright (C) 2013-2019 B.A.T.M.A.N. contributors: * * Martin Hundebøll - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_FRAGMENTATION_H_ diff --git a/net/batman-adv/gateway_client.c b/net/batman-adv/gateway_client.c index f5811f61aa927e80ca5cbecd04b19b8c6d93e3d0..47df4c678988613f7410acb96889558eabdf396d 100644 --- a/net/batman-adv/gateway_client.c +++ b/net/batman-adv/gateway_client.c @@ -2,18 +2,6 @@ /* Copyright (C) 2009-2019 B.A.T.M.A.N. contributors: * * Marek Lindner - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "gateway_client.h" @@ -53,7 +41,6 @@ #include "originator.h" #include "routing.h" #include "soft-interface.h" -#include "sysfs.h" #include "translation-table.h" /* These are the offsets of the "hw type" and "hw address length" in the dhcp diff --git a/net/batman-adv/gateway_client.h b/net/batman-adv/gateway_client.h index b5732c8be81a30d9eb0573c48eaf03ca58410cb0..0e14026feebd3117d64eb8e2fce5ab1d900f8e81 100644 --- a/net/batman-adv/gateway_client.h +++ b/net/batman-adv/gateway_client.h @@ -2,18 +2,6 @@ /* Copyright (C) 2009-2019 B.A.T.M.A.N. contributors: * * Marek Lindner - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_GATEWAY_CLIENT_H_ diff --git a/net/batman-adv/gateway_common.c b/net/batman-adv/gateway_common.c index e064de45e22cdbb575c093b3ae5ec33926e7f6a8..dac097f9be03866be5f571724c30088d9cad0879 100644 --- a/net/batman-adv/gateway_common.c +++ b/net/batman-adv/gateway_common.c @@ -2,18 +2,6 @@ /* Copyright (C) 2009-2019 B.A.T.M.A.N. contributors: * * Marek Lindner - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "gateway_common.h" diff --git a/net/batman-adv/gateway_common.h b/net/batman-adv/gateway_common.h index 128467a0fb895484c810502c5f86e4094de81748..5cf50736c635f996169cc980cb0690a358f119a2 100644 --- a/net/batman-adv/gateway_common.h +++ b/net/batman-adv/gateway_common.h @@ -2,18 +2,6 @@ /* Copyright (C) 2009-2019 B.A.T.M.A.N. contributors: * * Marek Lindner - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_GATEWAY_COMMON_H_ diff --git a/net/batman-adv/hard-interface.c b/net/batman-adv/hard-interface.c index 96ef7c70b4d92ef249e0b233c1f2a0c9bec6452c..79d1731b83066c60f9aef958d2bc343233bce67a 100644 --- a/net/batman-adv/hard-interface.c +++ b/net/batman-adv/hard-interface.c @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "hard-interface.h" diff --git a/net/batman-adv/hard-interface.h b/net/batman-adv/hard-interface.h index 48de28c8340150fad752f49334a67c4d25bc0f27..c8ef6aa0e86500c73f644a39e02fd376775271f4 100644 --- a/net/batman-adv/hard-interface.h +++ b/net/batman-adv/hard-interface.h @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_HARD_INTERFACE_H_ diff --git a/net/batman-adv/hash.c b/net/batman-adv/hash.c index 56a08ce193d507b856828756fb3fc54c72efca0b..a9d4e176f4deefc7ecb734f73a3f4c3c810aecd4 100644 --- a/net/batman-adv/hash.c +++ b/net/batman-adv/hash.c @@ -2,18 +2,6 @@ /* Copyright (C) 2006-2019 B.A.T.M.A.N. contributors: * * Simon Wunderlich, Marek Lindner - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "hash.h" diff --git a/net/batman-adv/hash.h b/net/batman-adv/hash.h index 37507b6d40068aa4f5f0fa2ebee3a497a45487ff..ceef171f7f98d02b12ef099d6fdc1f9df2209553 100644 --- a/net/batman-adv/hash.h +++ b/net/batman-adv/hash.h @@ -2,18 +2,6 @@ /* Copyright (C) 2006-2019 B.A.T.M.A.N. contributors: * * Simon Wunderlich, Marek Lindner - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_HASH_H_ diff --git a/net/batman-adv/icmp_socket.c b/net/batman-adv/icmp_socket.c index 3ff32125f4b5756e393ba70e0e2410fdd9bf422d..0a91c8661357d4ddbea1ba20dcd0df67b8ba5a97 100644 --- a/net/batman-adv/icmp_socket.c +++ b/net/batman-adv/icmp_socket.c @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "icmp_socket.h" diff --git a/net/batman-adv/icmp_socket.h b/net/batman-adv/icmp_socket.h index 5f8926522ff0d74e4482b927810aab407158cb7a..35eecbfd2e65bbbe11b26066fefeaf2111468534 100644 --- a/net/batman-adv/icmp_socket.h +++ b/net/batman-adv/icmp_socket.h @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_ICMP_SOCKET_H_ diff --git a/net/batman-adv/log.c b/net/batman-adv/log.c index e8ff13598c089e48f03e52021b24a361a7626782..f79ebd5b46e95b3b6de717c7ea1ecf44e5c96051 100644 --- a/net/batman-adv/log.c +++ b/net/batman-adv/log.c @@ -2,18 +2,6 @@ /* Copyright (C) 2010-2019 B.A.T.M.A.N. contributors: * * Marek Lindner - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "log.h" diff --git a/net/batman-adv/log.h b/net/batman-adv/log.h index 660e9bcc85a289295b3ab095f4dc4061e45101e1..5504637e63d80ae0ff81b2803e4d7b0fb4dbf005 100644 --- a/net/batman-adv/log.h +++ b/net/batman-adv/log.h @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_LOG_H_ diff --git a/net/batman-adv/main.c b/net/batman-adv/main.c index 75750870cf048e6beb39fdf311883449a9c73731..dabcaff87e3499b416e762da173f0def4cdeac86 100644 --- a/net/batman-adv/main.c +++ b/net/batman-adv/main.c @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "main.h" @@ -22,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -31,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -40,6 +30,7 @@ #include #include #include +#include #include #include #include @@ -84,6 +75,22 @@ struct workqueue_struct *batadv_event_workqueue; static void batadv_recv_handler_init(void); +#define BATADV_UEV_TYPE_VAR "BATTYPE=" +#define BATADV_UEV_ACTION_VAR "BATACTION=" +#define BATADV_UEV_DATA_VAR "BATDATA=" + +static char *batadv_uev_action_str[] = { + "add", + "del", + "change", + "loopdetect", +}; + +static char *batadv_uev_type_str[] = { + "gw", + "bla", +}; + static int __init batadv_init(void) { int ret; @@ -678,6 +685,60 @@ bool batadv_vlan_ap_isola_get(struct batadv_priv *bat_priv, unsigned short vid) return ap_isolation_enabled; } +/** + * batadv_throw_uevent() - Send an uevent with batman-adv specific env data + * @bat_priv: the bat priv with all the soft interface information + * @type: subsystem type of event. Stored in uevent's BATTYPE + * @action: action type of event. Stored in uevent's BATACTION + * @data: string with additional information to the event (ignored for + * BATADV_UEV_DEL). Stored in uevent's BATDATA + * + * Return: 0 on success or negative error number in case of failure + */ +int batadv_throw_uevent(struct batadv_priv *bat_priv, enum batadv_uev_type type, + enum batadv_uev_action action, const char *data) +{ + int ret = -ENOMEM; + struct kobject *bat_kobj; + char *uevent_env[4] = { NULL, NULL, NULL, NULL }; + + bat_kobj = &bat_priv->soft_iface->dev.kobj; + + uevent_env[0] = kasprintf(GFP_ATOMIC, + "%s%s", BATADV_UEV_TYPE_VAR, + batadv_uev_type_str[type]); + if (!uevent_env[0]) + goto out; + + uevent_env[1] = kasprintf(GFP_ATOMIC, + "%s%s", BATADV_UEV_ACTION_VAR, + batadv_uev_action_str[action]); + if (!uevent_env[1]) + goto out; + + /* If the event is DEL, ignore the data field */ + if (action != BATADV_UEV_DEL) { + uevent_env[2] = kasprintf(GFP_ATOMIC, + "%s%s", BATADV_UEV_DATA_VAR, data); + if (!uevent_env[2]) + goto out; + } + + ret = kobject_uevent_env(bat_kobj, KOBJ_CHANGE, uevent_env); +out: + kfree(uevent_env[0]); + kfree(uevent_env[1]); + kfree(uevent_env[2]); + + if (ret) + batadv_dbg(BATADV_DBG_BATMAN, bat_priv, + "Impossible to send uevent for (%s,%s,%s) event (err: %d)\n", + batadv_uev_type_str[type], + batadv_uev_action_str[action], + (action == BATADV_UEV_DEL ? "NULL" : data), ret); + return ret; +} + module_init(batadv_init); module_exit(batadv_exit); diff --git a/net/batman-adv/main.h b/net/batman-adv/main.h index 3ed669d7dc6ba93c04eefd1dc2ba38766ccd7ea2..f827e441025ff11df10945781e0da72ac4794b70 100644 --- a/net/batman-adv/main.h +++ b/net/batman-adv/main.h @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_MAIN_H_ @@ -394,5 +382,7 @@ static inline void batadv_add_counter(struct batadv_priv *bat_priv, size_t idx, unsigned short batadv_get_vid(struct sk_buff *skb, size_t header_len); bool batadv_vlan_ap_isola_get(struct batadv_priv *bat_priv, unsigned short vid); +int batadv_throw_uevent(struct batadv_priv *bat_priv, enum batadv_uev_type type, + enum batadv_uev_action action, const char *data); #endif /* _NET_BATMAN_ADV_MAIN_H_ */ diff --git a/net/batman-adv/multicast.c b/net/batman-adv/multicast.c index f91b1b6265cfea6d0f4d056defe30eeed452270e..3feb9435b715b66be8b67c4649a415e18a669278 100644 --- a/net/batman-adv/multicast.c +++ b/net/batman-adv/multicast.c @@ -2,18 +2,6 @@ /* Copyright (C) 2014-2019 B.A.T.M.A.N. contributors: * * Linus Lüssing - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "multicast.h" @@ -66,6 +54,7 @@ #include "hash.h" #include "log.h" #include "netlink.h" +#include "send.h" #include "soft-interface.h" #include "translation-table.h" #include "tvlv.h" @@ -991,6 +980,7 @@ batadv_mcast_forw_mode(struct batadv_priv *bat_priv, struct sk_buff *skb, { int ret, tt_count, ip_count, unsnoop_count, total_count; bool is_unsnoopable = false; + unsigned int mcast_fanout; struct ethhdr *ethhdr; ret = batadv_mcast_forw_mode_check(bat_priv, skb, &is_unsnoopable); @@ -1025,8 +1015,203 @@ batadv_mcast_forw_mode(struct batadv_priv *bat_priv, struct sk_buff *skb, case 0: return BATADV_FORW_NONE; default: - return BATADV_FORW_ALL; + mcast_fanout = atomic_read(&bat_priv->multicast_fanout); + + if (!unsnoop_count && total_count <= mcast_fanout) + return BATADV_FORW_SOME; } + + return BATADV_FORW_ALL; +} + +/** + * batadv_mcast_forw_tt() - forwards a packet to multicast listeners + * @bat_priv: the bat priv with all the soft interface information + * @skb: the multicast packet to transmit + * @vid: the vlan identifier + * + * Sends copies of a frame with multicast destination to any multicast + * listener registered in the translation table. A transmission is performed + * via a batman-adv unicast packet for each such destination node. + * + * Return: NET_XMIT_DROP on memory allocation failure, NET_XMIT_SUCCESS + * otherwise. + */ +static int +batadv_mcast_forw_tt(struct batadv_priv *bat_priv, struct sk_buff *skb, + unsigned short vid) +{ + int ret = NET_XMIT_SUCCESS; + struct sk_buff *newskb; + + struct batadv_tt_orig_list_entry *orig_entry; + + struct batadv_tt_global_entry *tt_global; + const u8 *addr = eth_hdr(skb)->h_dest; + + tt_global = batadv_tt_global_hash_find(bat_priv, addr, vid); + if (!tt_global) + goto out; + + rcu_read_lock(); + hlist_for_each_entry_rcu(orig_entry, &tt_global->orig_list, list) { + newskb = skb_copy(skb, GFP_ATOMIC); + if (!newskb) { + ret = NET_XMIT_DROP; + break; + } + + batadv_send_skb_unicast(bat_priv, newskb, BATADV_UNICAST, 0, + orig_entry->orig_node, vid); + } + rcu_read_unlock(); + + batadv_tt_global_entry_put(tt_global); + +out: + return ret; +} + +/** + * batadv_mcast_forw_want_all_ipv4() - forward to nodes with want-all-ipv4 + * @bat_priv: the bat priv with all the soft interface information + * @skb: the multicast packet to transmit + * @vid: the vlan identifier + * + * Sends copies of a frame with multicast destination to any node with a + * BATADV_MCAST_WANT_ALL_IPV4 flag set. A transmission is performed via a + * batman-adv unicast packet for each such destination node. + * + * Return: NET_XMIT_DROP on memory allocation failure, NET_XMIT_SUCCESS + * otherwise. + */ +static int +batadv_mcast_forw_want_all_ipv4(struct batadv_priv *bat_priv, + struct sk_buff *skb, unsigned short vid) +{ + struct batadv_orig_node *orig_node; + int ret = NET_XMIT_SUCCESS; + struct sk_buff *newskb; + + rcu_read_lock(); + hlist_for_each_entry_rcu(orig_node, + &bat_priv->mcast.want_all_ipv4_list, + mcast_want_all_ipv4_node) { + newskb = skb_copy(skb, GFP_ATOMIC); + if (!newskb) { + ret = NET_XMIT_DROP; + break; + } + + batadv_send_skb_unicast(bat_priv, newskb, BATADV_UNICAST, 0, + orig_node, vid); + } + rcu_read_unlock(); + return ret; +} + +/** + * batadv_mcast_forw_want_all_ipv6() - forward to nodes with want-all-ipv6 + * @bat_priv: the bat priv with all the soft interface information + * @skb: The multicast packet to transmit + * @vid: the vlan identifier + * + * Sends copies of a frame with multicast destination to any node with a + * BATADV_MCAST_WANT_ALL_IPV6 flag set. A transmission is performed via a + * batman-adv unicast packet for each such destination node. + * + * Return: NET_XMIT_DROP on memory allocation failure, NET_XMIT_SUCCESS + * otherwise. + */ +static int +batadv_mcast_forw_want_all_ipv6(struct batadv_priv *bat_priv, + struct sk_buff *skb, unsigned short vid) +{ + struct batadv_orig_node *orig_node; + int ret = NET_XMIT_SUCCESS; + struct sk_buff *newskb; + + rcu_read_lock(); + hlist_for_each_entry_rcu(orig_node, + &bat_priv->mcast.want_all_ipv6_list, + mcast_want_all_ipv6_node) { + newskb = skb_copy(skb, GFP_ATOMIC); + if (!newskb) { + ret = NET_XMIT_DROP; + break; + } + + batadv_send_skb_unicast(bat_priv, newskb, BATADV_UNICAST, 0, + orig_node, vid); + } + rcu_read_unlock(); + return ret; +} + +/** + * batadv_mcast_forw_want_all() - forward packet to nodes in a want-all list + * @bat_priv: the bat priv with all the soft interface information + * @skb: the multicast packet to transmit + * @vid: the vlan identifier + * + * Sends copies of a frame with multicast destination to any node with a + * BATADV_MCAST_WANT_ALL_IPV4 or BATADV_MCAST_WANT_ALL_IPV6 flag set. A + * transmission is performed via a batman-adv unicast packet for each such + * destination node. + * + * Return: NET_XMIT_DROP on memory allocation failure or if the protocol family + * is neither IPv4 nor IPv6. NET_XMIT_SUCCESS otherwise. + */ +static int +batadv_mcast_forw_want_all(struct batadv_priv *bat_priv, + struct sk_buff *skb, unsigned short vid) +{ + switch (ntohs(eth_hdr(skb)->h_proto)) { + case ETH_P_IP: + return batadv_mcast_forw_want_all_ipv4(bat_priv, skb, vid); + case ETH_P_IPV6: + return batadv_mcast_forw_want_all_ipv6(bat_priv, skb, vid); + default: + /* we shouldn't be here... */ + return NET_XMIT_DROP; + } +} + +/** + * batadv_mcast_forw_send() - send packet to any detected multicast recpient + * @bat_priv: the bat priv with all the soft interface information + * @skb: the multicast packet to transmit + * @vid: the vlan identifier + * + * Sends copies of a frame with multicast destination to any node that signaled + * interest in it, that is either via the translation table or the according + * want-all flags. A transmission is performed via a batman-adv unicast packet + * for each such destination node. + * + * The given skb is consumed/freed. + * + * Return: NET_XMIT_DROP on memory allocation failure or if the protocol family + * is neither IPv4 nor IPv6. NET_XMIT_SUCCESS otherwise. + */ +int batadv_mcast_forw_send(struct batadv_priv *bat_priv, struct sk_buff *skb, + unsigned short vid) +{ + int ret; + + ret = batadv_mcast_forw_tt(bat_priv, skb, vid); + if (ret != NET_XMIT_SUCCESS) { + kfree_skb(skb); + return ret; + } + + ret = batadv_mcast_forw_want_all(bat_priv, skb, vid); + if (ret != NET_XMIT_SUCCESS) { + kfree_skb(skb); + return ret; + } + + consume_skb(skb); + return ret; } /** diff --git a/net/batman-adv/multicast.h b/net/batman-adv/multicast.h index 466013fe88af50e9298e4e445d8f31147420f5c5..653b9b76fabe68ee7dbe038ef86b7e631848c18a 100644 --- a/net/batman-adv/multicast.h +++ b/net/batman-adv/multicast.h @@ -2,18 +2,6 @@ /* Copyright (C) 2014-2019 B.A.T.M.A.N. contributors: * * Linus Lüssing - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_MULTICAST_H_ @@ -35,6 +23,13 @@ enum batadv_forw_mode { */ BATADV_FORW_ALL, + /** + * @BATADV_FORW_SOME: forward the packet to some nodes (currently via + * a multicast-to-unicast conversion and the BATMAN unicast routing + * protocol) + */ + BATADV_FORW_SOME, + /** * @BATADV_FORW_SINGLE: forward the packet to a single node (currently * via the BATMAN unicast routing protocol) @@ -51,6 +46,9 @@ enum batadv_forw_mode batadv_mcast_forw_mode(struct batadv_priv *bat_priv, struct sk_buff *skb, struct batadv_orig_node **mcast_single_orig); +int batadv_mcast_forw_send(struct batadv_priv *bat_priv, struct sk_buff *skb, + unsigned short vid); + void batadv_mcast_init(struct batadv_priv *bat_priv); int batadv_mcast_flags_seq_print_text(struct seq_file *seq, void *offset); @@ -73,6 +71,14 @@ batadv_mcast_forw_mode(struct batadv_priv *bat_priv, struct sk_buff *skb, return BATADV_FORW_ALL; } +static inline int +batadv_mcast_forw_send(struct batadv_priv *bat_priv, struct sk_buff *skb, + unsigned short vid) +{ + kfree_skb(skb); + return NET_XMIT_DROP; +} + static inline int batadv_mcast_init(struct batadv_priv *bat_priv) { return 0; diff --git a/net/batman-adv/netlink.c b/net/batman-adv/netlink.c index 67a58da2e6a0ceb263b8e10472c4009fee8d3dca..a67720fad46ca496c932c0306e2f7ec4ed496fc9 100644 --- a/net/batman-adv/netlink.c +++ b/net/batman-adv/netlink.c @@ -2,18 +2,6 @@ /* Copyright (C) 2016-2019 B.A.T.M.A.N. contributors: * * Matthias Schiffer - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "netlink.h" @@ -157,6 +145,7 @@ static const struct nla_policy batadv_netlink_policy[NUM_BATADV_ATTR] = { [BATADV_ATTR_HOP_PENALTY] = { .type = NLA_U8 }, [BATADV_ATTR_LOG_LEVEL] = { .type = NLA_U32 }, [BATADV_ATTR_MULTICAST_FORCEFLOOD_ENABLED] = { .type = NLA_U8 }, + [BATADV_ATTR_MULTICAST_FANOUT] = { .type = NLA_U32 }, [BATADV_ATTR_NETWORK_CODING_ENABLED] = { .type = NLA_U8 }, [BATADV_ATTR_ORIG_INTERVAL] = { .type = NLA_U32 }, [BATADV_ATTR_ELP_INTERVAL] = { .type = NLA_U32 }, @@ -353,6 +342,10 @@ static int batadv_netlink_mesh_fill(struct sk_buff *msg, if (nla_put_u8(msg, BATADV_ATTR_MULTICAST_FORCEFLOOD_ENABLED, !atomic_read(&bat_priv->multicast_mode))) goto nla_put_failure; + + if (nla_put_u32(msg, BATADV_ATTR_MULTICAST_FANOUT, + atomic_read(&bat_priv->multicast_fanout))) + goto nla_put_failure; #endif /* CONFIG_BATMAN_ADV_MCAST */ #ifdef CONFIG_BATMAN_ADV_NC @@ -592,6 +585,12 @@ static int batadv_netlink_set_mesh(struct sk_buff *skb, struct genl_info *info) atomic_set(&bat_priv->multicast_mode, !nla_get_u8(attr)); } + + if (info->attrs[BATADV_ATTR_MULTICAST_FANOUT]) { + attr = info->attrs[BATADV_ATTR_MULTICAST_FANOUT]; + + atomic_set(&bat_priv->multicast_fanout, nla_get_u32(attr)); + } #endif /* CONFIG_BATMAN_ADV_MCAST */ #ifdef CONFIG_BATMAN_ADV_NC @@ -1344,35 +1343,35 @@ static void batadv_post_doit(const struct genl_ops *ops, struct sk_buff *skb, static const struct genl_ops batadv_netlink_ops[] = { { .cmd = BATADV_CMD_GET_MESH, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, /* can be retrieved by unprivileged users */ - .policy = batadv_netlink_policy, .doit = batadv_netlink_get_mesh, .internal_flags = BATADV_FLAG_NEED_MESH, }, { .cmd = BATADV_CMD_TP_METER, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = batadv_netlink_policy, .doit = batadv_netlink_tp_meter_start, .internal_flags = BATADV_FLAG_NEED_MESH, }, { .cmd = BATADV_CMD_TP_METER_CANCEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = batadv_netlink_policy, .doit = batadv_netlink_tp_meter_cancel, .internal_flags = BATADV_FLAG_NEED_MESH, }, { .cmd = BATADV_CMD_GET_ROUTING_ALGOS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = batadv_netlink_policy, .dumpit = batadv_algo_dump, }, { .cmd = BATADV_CMD_GET_HARDIF, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, /* can be retrieved by unprivileged users */ - .policy = batadv_netlink_policy, .dumpit = batadv_netlink_dump_hardif, .doit = batadv_netlink_get_hardif, .internal_flags = BATADV_FLAG_NEED_MESH | @@ -1380,85 +1379,85 @@ static const struct genl_ops batadv_netlink_ops[] = { }, { .cmd = BATADV_CMD_GET_TRANSTABLE_LOCAL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = batadv_netlink_policy, .dumpit = batadv_tt_local_dump, }, { .cmd = BATADV_CMD_GET_TRANSTABLE_GLOBAL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = batadv_netlink_policy, .dumpit = batadv_tt_global_dump, }, { .cmd = BATADV_CMD_GET_ORIGINATORS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = batadv_netlink_policy, .dumpit = batadv_orig_dump, }, { .cmd = BATADV_CMD_GET_NEIGHBORS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = batadv_netlink_policy, .dumpit = batadv_hardif_neigh_dump, }, { .cmd = BATADV_CMD_GET_GATEWAYS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = batadv_netlink_policy, .dumpit = batadv_gw_dump, }, { .cmd = BATADV_CMD_GET_BLA_CLAIM, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = batadv_netlink_policy, .dumpit = batadv_bla_claim_dump, }, { .cmd = BATADV_CMD_GET_BLA_BACKBONE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = batadv_netlink_policy, .dumpit = batadv_bla_backbone_dump, }, { .cmd = BATADV_CMD_GET_DAT_CACHE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = batadv_netlink_policy, .dumpit = batadv_dat_cache_dump, }, { .cmd = BATADV_CMD_GET_MCAST_FLAGS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = batadv_netlink_policy, .dumpit = batadv_mcast_flags_dump, }, { .cmd = BATADV_CMD_SET_MESH, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = batadv_netlink_policy, .doit = batadv_netlink_set_mesh, .internal_flags = BATADV_FLAG_NEED_MESH, }, { .cmd = BATADV_CMD_SET_HARDIF, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = batadv_netlink_policy, .doit = batadv_netlink_set_hardif, .internal_flags = BATADV_FLAG_NEED_MESH | BATADV_FLAG_NEED_HARDIF, }, { .cmd = BATADV_CMD_GET_VLAN, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, /* can be retrieved by unprivileged users */ - .policy = batadv_netlink_policy, .doit = batadv_netlink_get_vlan, .internal_flags = BATADV_FLAG_NEED_MESH | BATADV_FLAG_NEED_VLAN, }, { .cmd = BATADV_CMD_SET_VLAN, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = batadv_netlink_policy, .doit = batadv_netlink_set_vlan, .internal_flags = BATADV_FLAG_NEED_MESH | BATADV_FLAG_NEED_VLAN, @@ -1470,6 +1469,7 @@ struct genl_family batadv_netlink_family __ro_after_init = { .name = BATADV_NL_NAME, .version = 1, .maxattr = BATADV_ATTR_MAX, + .policy = batadv_netlink_policy, .netnsok = true, .pre_doit = batadv_pre_doit, .post_doit = batadv_post_doit, diff --git a/net/batman-adv/netlink.h b/net/batman-adv/netlink.h index 7273368544fc2c1ed7ee3d385faf4205086c11d3..d1e0681b87438c3d9c52bddbbec7dc27c1830df6 100644 --- a/net/batman-adv/netlink.h +++ b/net/batman-adv/netlink.h @@ -2,18 +2,6 @@ /* Copyright (C) 2016-2019 B.A.T.M.A.N. contributors: * * Matthias Schiffer - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_NETLINK_H_ diff --git a/net/batman-adv/network-coding.c b/net/batman-adv/network-coding.c index 278762bd94c65d82d5c5e4363a8d82a9d60c4924..c5e7906045f3c62c052400c44a18bca0a38499ba 100644 --- a/net/batman-adv/network-coding.c +++ b/net/batman-adv/network-coding.c @@ -2,18 +2,6 @@ /* Copyright (C) 2012-2019 B.A.T.M.A.N. contributors: * * Martin Hundebøll, Jeppe Ledet-Pedersen - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "network-coding.h" diff --git a/net/batman-adv/network-coding.h b/net/batman-adv/network-coding.h index 96ef0a511fc7b4913e23a08da01e0a887e2049dc..74f56113a5d06f0afaa02706c7318a8d6762527e 100644 --- a/net/batman-adv/network-coding.h +++ b/net/batman-adv/network-coding.h @@ -2,18 +2,6 @@ /* Copyright (C) 2012-2019 B.A.T.M.A.N. contributors: * * Martin Hundebøll, Jeppe Ledet-Pedersen - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_NETWORK_CODING_H_ diff --git a/net/batman-adv/originator.c b/net/batman-adv/originator.c index e5cdf89ef63c292d8f6c3f797702bf4cf6c3f450..45db798a7297d9f6d4be2d9c061bd2f37377c120 100644 --- a/net/batman-adv/originator.c +++ b/net/batman-adv/originator.c @@ -2,18 +2,6 @@ /* Copyright (C) 2009-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "originator.h" diff --git a/net/batman-adv/originator.h b/net/batman-adv/originator.h index dca1e4a34ec6c96b6e7a8265a943fa354473f5d3..3829e26f9c5d99332a7bcb837ba491f51e616ca3 100644 --- a/net/batman-adv/originator.h +++ b/net/batman-adv/originator.h @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_ORIGINATOR_H_ diff --git a/net/batman-adv/routing.c b/net/batman-adv/routing.c index cae0e5dd0768bdc4241915b8e2ed09b096caba14..f0f864820dead3e7811a6b0d522ec84954d9aec2 100644 --- a/net/batman-adv/routing.c +++ b/net/batman-adv/routing.c @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "routing.h" diff --git a/net/batman-adv/routing.h b/net/batman-adv/routing.h index 0102d69d345cf5737fde69793e4d52a47ac89ded..b96c6d06d188e7e3d67f70b71764898ff4f01a9c 100644 --- a/net/batman-adv/routing.h +++ b/net/batman-adv/routing.h @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_ROUTING_H_ diff --git a/net/batman-adv/send.c b/net/batman-adv/send.c index 66a8b3e445012248a9ffcb8f635b74dac10f0190..3ce5f7bad3698f89c8190aa2d31329e0c947092a 100644 --- a/net/batman-adv/send.c +++ b/net/batman-adv/send.c @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "send.h" diff --git a/net/batman-adv/send.h b/net/batman-adv/send.h index 1f6132922e6061793a76de6b89f2d872ab1dc131..5921ee4e107c643bee9255ce2b3f47e5edf8a622 100644 --- a/net/batman-adv/send.h +++ b/net/batman-adv/send.h @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_SEND_H_ diff --git a/net/batman-adv/soft-interface.c b/net/batman-adv/soft-interface.c index 2e367230376b932f61fde553c85c2c46e7095111..a7677e1d000fbc6ab60466cf5fc4f1750014268e 100644 --- a/net/batman-adv/soft-interface.c +++ b/net/batman-adv/soft-interface.c @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "soft-interface.h" @@ -209,7 +197,7 @@ static netdev_tx_t batadv_interface_tx(struct sk_buff *skb, unsigned short vid; u32 seqno; int gw_mode; - enum batadv_forw_mode forw_mode; + enum batadv_forw_mode forw_mode = BATADV_FORW_SINGLE; struct batadv_orig_node *mcast_single_orig = NULL; int network_offset = ETH_HLEN; __be16 proto; @@ -317,7 +305,8 @@ send: if (forw_mode == BATADV_FORW_NONE) goto dropped; - if (forw_mode == BATADV_FORW_SINGLE) + if (forw_mode == BATADV_FORW_SINGLE || + forw_mode == BATADV_FORW_SOME) do_bcast = false; } } @@ -377,6 +366,8 @@ send: ret = batadv_send_skb_unicast(bat_priv, skb, BATADV_UNICAST, 0, mcast_single_orig, vid); + } else if (forw_mode == BATADV_FORW_SOME) { + ret = batadv_mcast_forw_send(bat_priv, skb, vid); } else { if (batadv_dat_snoop_outgoing_arp_request(bat_priv, skb)) @@ -818,6 +809,7 @@ static int batadv_softif_init_late(struct net_device *dev) bat_priv->mcast.querier_ipv6.shadowing = false; bat_priv->mcast.flags = BATADV_NO_FLAGS; atomic_set(&bat_priv->multicast_mode, 1); + atomic_set(&bat_priv->multicast_fanout, 16); atomic_set(&bat_priv->mcast.num_want_all_unsnoopables, 0); atomic_set(&bat_priv->mcast.num_want_all_ipv4, 0); atomic_set(&bat_priv->mcast.num_want_all_ipv6, 0); diff --git a/net/batman-adv/soft-interface.h b/net/batman-adv/soft-interface.h index 538bb661878cdd1115d8670925246110d85b46ef..275442a7acb6d901653dc98aaf0abad410caf639 100644 --- a/net/batman-adv/soft-interface.h +++ b/net/batman-adv/soft-interface.h @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_SOFT_INTERFACE_H_ diff --git a/net/batman-adv/sysfs.c b/net/batman-adv/sysfs.c index 208655cf67179b5c2bbcfb8330993b42ca70109f..80fc3253c3368e3cc356176c5ba961542de0a8c9 100644 --- a/net/batman-adv/sysfs.c +++ b/net/batman-adv/sysfs.c @@ -2,23 +2,12 @@ /* Copyright (C) 2010-2019 B.A.T.M.A.N. contributors: * * Marek Lindner - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "sysfs.h" #include "main.h" +#include #include #include #include @@ -34,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -52,6 +42,16 @@ #include "network-coding.h" #include "soft-interface.h" +/** + * batadv_sysfs_deprecated() - Log use of deprecated batadv sysfs access + * @attr: attribute which was accessed + */ +static void batadv_sysfs_deprecated(struct attribute *attr) +{ + pr_warn_ratelimited(DEPRECATED "%s (pid %d) Use of sysfs file \"%s\".\nUse batadv genl family instead", + current->comm, task_pid_nr(current), attr->name); +} + static struct net_device *batadv_kobj_to_netdev(struct kobject *obj) { struct device *dev = container_of(obj->parent, struct device, kobj); @@ -114,22 +114,6 @@ batadv_kobj_to_vlan(struct batadv_priv *bat_priv, struct kobject *obj) return vlan; } -#define BATADV_UEV_TYPE_VAR "BATTYPE=" -#define BATADV_UEV_ACTION_VAR "BATACTION=" -#define BATADV_UEV_DATA_VAR "BATDATA=" - -static char *batadv_uev_action_str[] = { - "add", - "del", - "change", - "loopdetect", -}; - -static char *batadv_uev_type_str[] = { - "gw", - "bla", -}; - /* Use this, if you have customized show and store functions for vlan attrs */ #define BATADV_ATTR_VLAN(_name, _mode, _show, _store) \ struct batadv_attribute batadv_attr_vlan_##_name = { \ @@ -157,6 +141,7 @@ ssize_t batadv_store_##_name(struct kobject *kobj, \ struct batadv_priv *bat_priv = netdev_priv(net_dev); \ ssize_t length; \ \ + batadv_sysfs_deprecated(attr); \ length = __batadv_store_bool_attr(buff, count, _post_func, attr,\ &bat_priv->_name, net_dev); \ \ @@ -171,6 +156,7 @@ ssize_t batadv_show_##_name(struct kobject *kobj, \ { \ struct batadv_priv *bat_priv = batadv_kobj_to_batpriv(kobj); \ \ + batadv_sysfs_deprecated(attr); \ return sprintf(buff, "%s\n", \ atomic_read(&bat_priv->_name) == 0 ? \ "disabled" : "enabled"); \ @@ -194,6 +180,7 @@ ssize_t batadv_store_##_name(struct kobject *kobj, \ struct batadv_priv *bat_priv = netdev_priv(net_dev); \ ssize_t length; \ \ + batadv_sysfs_deprecated(attr); \ length = __batadv_store_uint_attr(buff, count, _min, _max, \ _post_func, attr, \ &bat_priv->_var, net_dev, \ @@ -210,6 +197,7 @@ ssize_t batadv_show_##_name(struct kobject *kobj, \ { \ struct batadv_priv *bat_priv = batadv_kobj_to_batpriv(kobj); \ \ + batadv_sysfs_deprecated(attr); \ return sprintf(buff, "%i\n", atomic_read(&bat_priv->_var)); \ } \ @@ -234,6 +222,7 @@ ssize_t batadv_store_vlan_##_name(struct kobject *kobj, \ attr, &vlan->_name, \ bat_priv->soft_iface); \ \ + batadv_sysfs_deprecated(attr); \ if (vlan->vid) \ batadv_netlink_notify_vlan(bat_priv, vlan); \ else \ @@ -254,6 +243,7 @@ ssize_t batadv_show_vlan_##_name(struct kobject *kobj, \ atomic_read(&vlan->_name) == 0 ? \ "disabled" : "enabled"); \ \ + batadv_sysfs_deprecated(attr); \ batadv_softif_vlan_put(vlan); \ return res; \ } @@ -275,6 +265,7 @@ ssize_t batadv_store_##_name(struct kobject *kobj, \ struct batadv_priv *bat_priv; \ ssize_t length; \ \ + batadv_sysfs_deprecated(attr); \ hard_iface = batadv_hardif_get_by_netdev(net_dev); \ if (!hard_iface) \ return 0; \ @@ -302,6 +293,7 @@ ssize_t batadv_show_##_name(struct kobject *kobj, \ struct batadv_hard_iface *hard_iface; \ ssize_t length; \ \ + batadv_sysfs_deprecated(attr); \ hard_iface = batadv_hardif_get_by_netdev(net_dev); \ if (!hard_iface) \ return 0; \ @@ -446,6 +438,7 @@ static ssize_t batadv_show_bat_algo(struct kobject *kobj, { struct batadv_priv *bat_priv = batadv_kobj_to_batpriv(kobj); + batadv_sysfs_deprecated(attr); return sprintf(buff, "%s\n", bat_priv->algo_ops->name); } @@ -462,6 +455,8 @@ static ssize_t batadv_show_gw_mode(struct kobject *kobj, struct attribute *attr, struct batadv_priv *bat_priv = batadv_kobj_to_batpriv(kobj); int bytes_written; + batadv_sysfs_deprecated(attr); + /* GW mode is not available if the routing algorithm in use does not * implement the GW API */ @@ -496,6 +491,8 @@ static ssize_t batadv_store_gw_mode(struct kobject *kobj, char *curr_gw_mode_str; int gw_mode_tmp = -1; + batadv_sysfs_deprecated(attr); + /* toggling GW mode is allowed only if the routing algorithm in use * provides the GW API */ @@ -570,6 +567,8 @@ static ssize_t batadv_show_gw_sel_class(struct kobject *kobj, { struct batadv_priv *bat_priv = batadv_kobj_to_batpriv(kobj); + batadv_sysfs_deprecated(attr); + /* GW selection class is not available if the routing algorithm in use * does not implement the GW API */ @@ -590,6 +589,8 @@ static ssize_t batadv_store_gw_sel_class(struct kobject *kobj, struct batadv_priv *bat_priv = batadv_kobj_to_batpriv(kobj); ssize_t length; + batadv_sysfs_deprecated(attr); + /* setting the GW selection class is allowed only if the routing * algorithm in use implements the GW API */ @@ -620,6 +621,8 @@ static ssize_t batadv_show_gw_bwidth(struct kobject *kobj, struct batadv_priv *bat_priv = batadv_kobj_to_batpriv(kobj); u32 down, up; + batadv_sysfs_deprecated(attr); + down = atomic_read(&bat_priv->gw.bandwidth_down); up = atomic_read(&bat_priv->gw.bandwidth_up); @@ -635,6 +638,8 @@ static ssize_t batadv_store_gw_bwidth(struct kobject *kobj, struct net_device *net_dev = batadv_kobj_to_netdev(kobj); ssize_t length; + batadv_sysfs_deprecated(attr); + if (buff[count - 1] == '\n') buff[count - 1] = '\0'; @@ -659,6 +664,7 @@ static ssize_t batadv_show_isolation_mark(struct kobject *kobj, { struct batadv_priv *bat_priv = batadv_kobj_to_batpriv(kobj); + batadv_sysfs_deprecated(attr); return sprintf(buff, "%#.8x/%#.8x\n", bat_priv->isolation_mark, bat_priv->isolation_mark_mask); } @@ -682,6 +688,8 @@ static ssize_t batadv_store_isolation_mark(struct kobject *kobj, u32 mark, mask; char *mask_ptr; + batadv_sysfs_deprecated(attr); + /* parse the mask if it has been specified, otherwise assume the mask is * the biggest possible */ @@ -937,6 +945,8 @@ static ssize_t batadv_show_mesh_iface(struct kobject *kobj, ssize_t length; const char *ifname; + batadv_sysfs_deprecated(attr); + hard_iface = batadv_hardif_get_by_netdev(net_dev); if (!hard_iface) return 0; @@ -1041,6 +1051,8 @@ static ssize_t batadv_store_mesh_iface(struct kobject *kobj, struct net_device *net_dev = batadv_kobj_to_netdev(kobj); struct batadv_store_mesh_work *store_work; + batadv_sysfs_deprecated(attr); + if (buff[count - 1] == '\n') buff[count - 1] = '\0'; @@ -1072,6 +1084,8 @@ static ssize_t batadv_show_iface_status(struct kobject *kobj, struct batadv_hard_iface *hard_iface; ssize_t length; + batadv_sysfs_deprecated(attr); + hard_iface = batadv_hardif_get_by_netdev(net_dev); if (!hard_iface) return 0; @@ -1123,6 +1137,8 @@ static ssize_t batadv_store_throughput_override(struct kobject *kobj, u32 old_tp_override; bool ret; + batadv_sysfs_deprecated(attr); + hard_iface = batadv_hardif_get_by_netdev(net_dev); if (!hard_iface) return -EINVAL; @@ -1165,6 +1181,8 @@ static ssize_t batadv_show_throughput_override(struct kobject *kobj, struct batadv_hard_iface *hard_iface; u32 tp_override; + batadv_sysfs_deprecated(attr); + hard_iface = batadv_hardif_get_by_netdev(net_dev); if (!hard_iface) return -EINVAL; @@ -1250,57 +1268,3 @@ void batadv_sysfs_del_hardif(struct kobject **hardif_obj) kobject_put(*hardif_obj); *hardif_obj = NULL; } - -/** - * batadv_throw_uevent() - Send an uevent with batman-adv specific env data - * @bat_priv: the bat priv with all the soft interface information - * @type: subsystem type of event. Stored in uevent's BATTYPE - * @action: action type of event. Stored in uevent's BATACTION - * @data: string with additional information to the event (ignored for - * BATADV_UEV_DEL). Stored in uevent's BATDATA - * - * Return: 0 on success or negative error number in case of failure - */ -int batadv_throw_uevent(struct batadv_priv *bat_priv, enum batadv_uev_type type, - enum batadv_uev_action action, const char *data) -{ - int ret = -ENOMEM; - struct kobject *bat_kobj; - char *uevent_env[4] = { NULL, NULL, NULL, NULL }; - - bat_kobj = &bat_priv->soft_iface->dev.kobj; - - uevent_env[0] = kasprintf(GFP_ATOMIC, - "%s%s", BATADV_UEV_TYPE_VAR, - batadv_uev_type_str[type]); - if (!uevent_env[0]) - goto out; - - uevent_env[1] = kasprintf(GFP_ATOMIC, - "%s%s", BATADV_UEV_ACTION_VAR, - batadv_uev_action_str[action]); - if (!uevent_env[1]) - goto out; - - /* If the event is DEL, ignore the data field */ - if (action != BATADV_UEV_DEL) { - uevent_env[2] = kasprintf(GFP_ATOMIC, - "%s%s", BATADV_UEV_DATA_VAR, data); - if (!uevent_env[2]) - goto out; - } - - ret = kobject_uevent_env(bat_kobj, KOBJ_CHANGE, uevent_env); -out: - kfree(uevent_env[0]); - kfree(uevent_env[1]); - kfree(uevent_env[2]); - - if (ret) - batadv_dbg(BATADV_DBG_BATMAN, bat_priv, - "Impossible to send uevent for (%s,%s,%s) event (err: %d)\n", - batadv_uev_type_str[type], - batadv_uev_action_str[action], - (action == BATADV_UEV_DEL ? "NULL" : data), ret); - return ret; -} diff --git a/net/batman-adv/sysfs.h b/net/batman-adv/sysfs.h index 705ffbe763f47968e3649b871728b71b0eeed908..83fa808b187137827406a3bc41c77ff0c23863db 100644 --- a/net/batman-adv/sysfs.h +++ b/net/batman-adv/sysfs.h @@ -2,18 +2,6 @@ /* Copyright (C) 2010-2019 B.A.T.M.A.N. contributors: * * Marek Lindner - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_SYSFS_H_ @@ -57,6 +45,8 @@ struct batadv_attribute { char *buf, size_t count); }; +#ifdef CONFIG_BATMAN_ADV_SYSFS + int batadv_sysfs_add_meshif(struct net_device *dev); void batadv_sysfs_del_meshif(struct net_device *dev); int batadv_sysfs_add_hardif(struct kobject **hardif_obj, @@ -66,7 +56,39 @@ int batadv_sysfs_add_vlan(struct net_device *dev, struct batadv_softif_vlan *vlan); void batadv_sysfs_del_vlan(struct batadv_priv *bat_priv, struct batadv_softif_vlan *vlan); -int batadv_throw_uevent(struct batadv_priv *bat_priv, enum batadv_uev_type type, - enum batadv_uev_action action, const char *data); + +#else + +static inline int batadv_sysfs_add_meshif(struct net_device *dev) +{ + return 0; +} + +static inline void batadv_sysfs_del_meshif(struct net_device *dev) +{ +} + +static inline int batadv_sysfs_add_hardif(struct kobject **hardif_obj, + struct net_device *dev) +{ + return 0; +} + +static inline void batadv_sysfs_del_hardif(struct kobject **hardif_obj) +{ +} + +static inline int batadv_sysfs_add_vlan(struct net_device *dev, + struct batadv_softif_vlan *vlan) +{ + return 0; +} + +static inline void batadv_sysfs_del_vlan(struct batadv_priv *bat_priv, + struct batadv_softif_vlan *vlan) +{ +} + +#endif #endif /* _NET_BATMAN_ADV_SYSFS_H_ */ diff --git a/net/batman-adv/tp_meter.c b/net/batman-adv/tp_meter.c index 500109bbd551314317d8982a419b3388dd872e28..8203921462499106e910276cd846d0b989382d41 100644 --- a/net/batman-adv/tp_meter.c +++ b/net/batman-adv/tp_meter.c @@ -2,18 +2,6 @@ /* Copyright (C) 2012-2019 B.A.T.M.A.N. contributors: * * Edo Monticelli, Antonio Quartulli - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "tp_meter.h" diff --git a/net/batman-adv/tp_meter.h b/net/batman-adv/tp_meter.h index 6b4d0f733896ac47210c360157aa1590d6337c9f..604b3799c972046bf10fb37ab4f2712ca73d1981 100644 --- a/net/batman-adv/tp_meter.h +++ b/net/batman-adv/tp_meter.h @@ -2,18 +2,6 @@ /* Copyright (C) 2012-2019 B.A.T.M.A.N. contributors: * * Edo Monticelli, Antonio Quartulli - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_TP_METER_H_ diff --git a/net/batman-adv/trace.c b/net/batman-adv/trace.c index f77c917ed20d175b7d95e4c60c7f49d7bf3a3b12..3cedd2c36528605036e7da7a4a526fc6d5621f78 100644 --- a/net/batman-adv/trace.c +++ b/net/batman-adv/trace.c @@ -2,18 +2,6 @@ /* Copyright (C) 2010-2019 B.A.T.M.A.N. contributors: * * Sven Eckelmann - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #define CREATE_TRACE_POINTS diff --git a/net/batman-adv/trace.h b/net/batman-adv/trace.h index 5e55790514008d949abd6b57f1ec25b4c61892c2..d8f764521c0b2849c7e37c1a920e4eb59e59866c 100644 --- a/net/batman-adv/trace.h +++ b/net/batman-adv/trace.h @@ -2,18 +2,6 @@ /* Copyright (C) 2010-2019 B.A.T.M.A.N. contributors: * * Sven Eckelmann - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #if !defined(_NET_BATMAN_ADV_TRACE_H_) || defined(TRACE_HEADER_MULTI_READ) diff --git a/net/batman-adv/translation-table.c b/net/batman-adv/translation-table.c index 26c4e2493ddfbfdea26a9b45409821834010a913..1ddfd5e011eecb807d7bb9192ff012a8ed1f923e 100644 --- a/net/batman-adv/translation-table.c +++ b/net/batman-adv/translation-table.c @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich, Antonio Quartulli - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "translation-table.h" @@ -205,7 +193,7 @@ batadv_tt_local_hash_find(struct batadv_priv *bat_priv, const u8 *addr, * Return: a pointer to the corresponding tt_global_entry struct if the client * is found, NULL otherwise. */ -static struct batadv_tt_global_entry * +struct batadv_tt_global_entry * batadv_tt_global_hash_find(struct batadv_priv *bat_priv, const u8 *addr, unsigned short vid) { @@ -300,8 +288,7 @@ static void batadv_tt_global_entry_release(struct kref *ref) * possibly release it * @tt_global_entry: tt_global_entry to be free'd */ -static void -batadv_tt_global_entry_put(struct batadv_tt_global_entry *tt_global_entry) +void batadv_tt_global_entry_put(struct batadv_tt_global_entry *tt_global_entry) { kref_put(&tt_global_entry->common.refcount, batadv_tt_global_entry_release); diff --git a/net/batman-adv/translation-table.h b/net/batman-adv/translation-table.h index 61bca75e5911a0214a64913aabf09a62a0eecd3f..c8c48d62a430333ae7db63561f08e59dac6292f3 100644 --- a/net/batman-adv/translation-table.h +++ b/net/batman-adv/translation-table.h @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich, Antonio Quartulli - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_TRANSLATION_TABLE_H_ @@ -41,6 +29,10 @@ int batadv_tt_global_dump(struct sk_buff *msg, struct netlink_callback *cb); void batadv_tt_global_del_orig(struct batadv_priv *bat_priv, struct batadv_orig_node *orig_node, s32 match_vid, const char *message); +struct batadv_tt_global_entry * +batadv_tt_global_hash_find(struct batadv_priv *bat_priv, const u8 *addr, + unsigned short vid); +void batadv_tt_global_entry_put(struct batadv_tt_global_entry *tt_global_entry); int batadv_tt_global_hash_count(struct batadv_priv *bat_priv, const u8 *addr, unsigned short vid); struct batadv_orig_node *batadv_transtable_search(struct batadv_priv *bat_priv, diff --git a/net/batman-adv/tvlv.c b/net/batman-adv/tvlv.c index 7e947b01919d8324ce22d9d52708d2e538a0c978..aae63f0d21eb5cb68a41ad705ee4cb31d46e0ed5 100644 --- a/net/batman-adv/tvlv.c +++ b/net/batman-adv/tvlv.c @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #include "main.h" diff --git a/net/batman-adv/tvlv.h b/net/batman-adv/tvlv.h index c0f033b1acb8829e25c83cf7861e0e894747e88d..114ac01e06afa333cbb98771198e792bbd8fefa2 100644 --- a/net/batman-adv/tvlv.h +++ b/net/batman-adv/tvlv.h @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_TVLV_H_ diff --git a/net/batman-adv/types.h b/net/batman-adv/types.h index a21b34ed6548f4017ce3bf58949f2d758a9c84a8..357ca119329a58efa77534c3397e08411bd5f578 100644 --- a/net/batman-adv/types.h +++ b/net/batman-adv/types.h @@ -2,18 +2,6 @@ /* Copyright (C) 2007-2019 B.A.T.M.A.N. contributors: * * Marek Lindner, Simon Wunderlich - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of version 2 of the GNU General Public - * License as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . */ #ifndef _NET_BATMAN_ADV_TYPES_H_ @@ -1565,6 +1553,12 @@ struct batadv_priv { * node's sender/originating side */ atomic_t multicast_mode; + + /** + * @multicast_fanout: Maximum number of packet copies to generate for a + * multicast-to-unicast conversion + */ + atomic_t multicast_fanout; #endif /** @orig_interval: OGM broadcast interval in milliseconds */ diff --git a/net/bluetooth/af_bluetooth.c b/net/bluetooth/af_bluetooth.c index 8d12198eaa9490daf54d1dbc9fd52a79108967bb..94ddf19998c7ee7098b52c9ef37f3e0296089fab 100644 --- a/net/bluetooth/af_bluetooth.c +++ b/net/bluetooth/af_bluetooth.c @@ -521,14 +521,6 @@ int bt_sock_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) err = put_user(amount, (int __user *) arg); break; - case SIOCGSTAMP: - err = sock_get_timestamp(sk, (struct timeval __user *) arg); - break; - - case SIOCGSTAMPNS: - err = sock_get_timestampns(sk, (struct timespec __user *) arg); - break; - default: err = -ENOIOCTLCMD; break; diff --git a/net/bluetooth/hci_conn.c b/net/bluetooth/hci_conn.c index bd4978ce8c45330219e823fad140b7b586622c92..3cf0764d5793f42c55b39caf79a7e15734fbc40d 100644 --- a/net/bluetooth/hci_conn.c +++ b/net/bluetooth/hci_conn.c @@ -1276,6 +1276,14 @@ int hci_conn_check_link_mode(struct hci_conn *conn) !test_bit(HCI_CONN_ENCRYPT, &conn->flags)) return 0; + /* The minimum encryption key size needs to be enforced by the + * host stack before establishing any L2CAP connections. The + * specification in theory allows a minimum of 1, but to align + * BR/EDR and LE transports, a minimum of 7 is chosen. + */ + if (conn->enc_key_size < HCI_MIN_ENC_KEY_SIZE) + return 0; + return 1; } diff --git a/net/bluetooth/hci_core.c b/net/bluetooth/hci_core.c index d6b2540ba7f8b3d78c02f238fd5a89abc25dd8c2..b81bf53c5ac4a564491e71fcbdec4474903eaed5 100644 --- a/net/bluetooth/hci_core.c +++ b/net/bluetooth/hci_core.c @@ -1460,8 +1460,6 @@ static int hci_dev_do_open(struct hci_dev *hdev) hdev->set_bdaddr) ret = hdev->set_bdaddr(hdev, &hdev->public_addr); - else - ret = -EADDRNOTAVAIL; } setup_failed: @@ -4383,6 +4381,9 @@ void hci_req_cmd_complete(struct hci_dev *hdev, u16 opcode, u8 status, return; } + /* If we reach this point this event matches the last command sent */ + hci_dev_clear_flag(hdev, HCI_CMD_PENDING); + /* If the command succeeded and there's still more commands in * this request the request is not yet complete. */ @@ -4493,6 +4494,8 @@ static void hci_cmd_work(struct work_struct *work) hdev->sent_cmd = skb_clone(skb, GFP_KERNEL); if (hdev->sent_cmd) { + if (hci_req_status_pend(hdev)) + hci_dev_set_flag(hdev, HCI_CMD_PENDING); atomic_dec(&hdev->cmd_cnt); hci_send_frame(hdev, skb); if (test_bit(HCI_RESET, &hdev->flags)) diff --git a/net/bluetooth/hci_event.c b/net/bluetooth/hci_event.c index 609fd6871c5ad2f060bd191fb3f04cc785bb5392..9e4fcf406d9cd773e02b62998eb1cf387e3e622d 100644 --- a/net/bluetooth/hci_event.c +++ b/net/bluetooth/hci_event.c @@ -3404,6 +3404,12 @@ static void hci_cmd_complete_evt(struct hci_dev *hdev, struct sk_buff *skb, hci_req_cmd_complete(hdev, *opcode, *status, req_complete, req_complete_skb); + if (hci_dev_test_flag(hdev, HCI_CMD_PENDING)) { + bt_dev_err(hdev, + "unexpected event for opcode 0x%4.4x", *opcode); + return; + } + if (atomic_read(&hdev->cmd_cnt) && !skb_queue_empty(&hdev->cmd_q)) queue_work(hdev->workqueue, &hdev->cmd_work); } @@ -3511,6 +3517,12 @@ static void hci_cmd_status_evt(struct hci_dev *hdev, struct sk_buff *skb, hci_req_cmd_complete(hdev, *opcode, ev->status, req_complete, req_complete_skb); + if (hci_dev_test_flag(hdev, HCI_CMD_PENDING)) { + bt_dev_err(hdev, + "unexpected event for opcode 0x%4.4x", *opcode); + return; + } + if (atomic_read(&hdev->cmd_cnt) && !skb_queue_empty(&hdev->cmd_q)) queue_work(hdev->workqueue, &hdev->cmd_work); } @@ -5433,7 +5445,7 @@ static void hci_le_ext_adv_report_evt(struct hci_dev *hdev, struct sk_buff *skb) ev->data, ev->length); } - ptr += sizeof(*ev) + ev->length + 1; + ptr += sizeof(*ev) + ev->length; } hci_dev_unlock(hdev); diff --git a/net/bluetooth/hci_request.c b/net/bluetooth/hci_request.c index ca73d36cc149460e8d3eb08b9b291b1a7a396ef3..e9a95ed654915dd976f5a6bcb81af9dc764ff9ba 100644 --- a/net/bluetooth/hci_request.c +++ b/net/bluetooth/hci_request.c @@ -46,6 +46,11 @@ void hci_req_purge(struct hci_request *req) skb_queue_purge(&req->cmd_q); } +bool hci_req_status_pend(struct hci_dev *hdev) +{ + return hdev->req_status == HCI_REQ_PEND; +} + static int req_run(struct hci_request *req, hci_req_complete_t complete, hci_req_complete_skb_t complete_skb) { diff --git a/net/bluetooth/hci_request.h b/net/bluetooth/hci_request.h index 692cc8b133682ab749b73cf34fe0f5da4c6276b4..55b2050cc9ff0557f3aec21573ad97af57dc76f4 100644 --- a/net/bluetooth/hci_request.h +++ b/net/bluetooth/hci_request.h @@ -37,6 +37,7 @@ struct hci_request { void hci_req_init(struct hci_request *req, struct hci_dev *hdev); void hci_req_purge(struct hci_request *req); +bool hci_req_status_pend(struct hci_dev *hdev); int hci_req_run(struct hci_request *req, hci_req_complete_t complete); int hci_req_run_skb(struct hci_request *req, hci_req_complete_skb_t complete); void hci_req_add(struct hci_request *req, u16 opcode, u32 plen, diff --git a/net/bluetooth/hidp/sock.c b/net/bluetooth/hidp/sock.c index 9f85a1943be9b6b6ed06a61fe4a830b233f82ec5..2151913892ce8705d093a96fa65c3888e8a6dbb1 100644 --- a/net/bluetooth/hidp/sock.c +++ b/net/bluetooth/hidp/sock.c @@ -75,6 +75,7 @@ static int do_hidp_sock_ioctl(struct socket *sock, unsigned int cmd, void __user sockfd_put(csock); return err; } + ca.name[sizeof(ca.name)-1] = 0; err = hidp_connection_add(&ca, csock, isock); if (!err && copy_to_user(argp, &ca, sizeof(ca))) diff --git a/net/bluetooth/l2cap_core.c b/net/bluetooth/l2cap_core.c index f17e393b43b47078a62f3a30d17595293c65264b..b53acd6c9a3df8bcd1da3f92bb27308ec19ac0f2 100644 --- a/net/bluetooth/l2cap_core.c +++ b/net/bluetooth/l2cap_core.c @@ -510,12 +510,12 @@ void l2cap_chan_set_defaults(struct l2cap_chan *chan) } EXPORT_SYMBOL_GPL(l2cap_chan_set_defaults); -static void l2cap_le_flowctl_init(struct l2cap_chan *chan) +static void l2cap_le_flowctl_init(struct l2cap_chan *chan, u16 tx_credits) { chan->sdu = NULL; chan->sdu_last_frag = NULL; chan->sdu_len = 0; - chan->tx_credits = 0; + chan->tx_credits = tx_credits; /* Derive MPS from connection MTU to stop HCI fragmentation */ chan->mps = min_t(u16, chan->imtu, chan->conn->mtu - L2CAP_HDR_SIZE); /* Give enough credits for a full packet */ @@ -1281,7 +1281,7 @@ static void l2cap_le_connect(struct l2cap_chan *chan) if (test_and_set_bit(FLAG_LE_CONN_REQ_SENT, &chan->flags)) return; - l2cap_le_flowctl_init(chan); + l2cap_le_flowctl_init(chan, 0); req.psm = chan->psm; req.scid = cpu_to_le16(chan->scid); @@ -5532,11 +5532,10 @@ static int l2cap_le_connect_req(struct l2cap_conn *conn, chan->dcid = scid; chan->omtu = mtu; chan->remote_mps = mps; - chan->tx_credits = __le16_to_cpu(req->credits); __l2cap_chan_add(conn, chan); - l2cap_le_flowctl_init(chan); + l2cap_le_flowctl_init(chan, __le16_to_cpu(req->credits)); dcid = chan->scid; credits = chan->rx_credits; diff --git a/net/bluetooth/l2cap_sock.c b/net/bluetooth/l2cap_sock.c index a3a2cd55e23a9caa288663e69243db71cdf211fc..a7be8b59b3c280de738ec808534635265ee35169 100644 --- a/net/bluetooth/l2cap_sock.c +++ b/net/bluetooth/l2cap_sock.c @@ -791,10 +791,13 @@ static int l2cap_sock_setsockopt(struct socket *sock, int level, int optname, conn = chan->conn; - /*change security for LE channels */ + /* change security for LE channels */ if (chan->scid == L2CAP_CID_ATT) { - if (smp_conn_security(conn->hcon, sec.level)) + if (smp_conn_security(conn->hcon, sec.level)) { + err = -EINVAL; break; + } + set_bit(FLAG_PENDING_SECURITY, &chan->flags); sk->sk_state = BT_CONFIG; chan->state = BT_CONFIG; @@ -1655,6 +1658,7 @@ static const struct proto_ops l2cap_sock_ops = { .recvmsg = l2cap_sock_recvmsg, .poll = bt_sock_poll, .ioctl = bt_sock_ioctl, + .gettstamp = sock_gettstamp, .mmap = sock_no_mmap, .socketpair = sock_no_socketpair, .shutdown = l2cap_sock_shutdown, diff --git a/net/bluetooth/mgmt.c b/net/bluetooth/mgmt.c index 2457f408d17d2f599402ddb1b9fa242664f313cf..150114e33b20f22c6da3bc25d4a2b9a7ffa5f81e 100644 --- a/net/bluetooth/mgmt.c +++ b/net/bluetooth/mgmt.c @@ -2301,8 +2301,7 @@ static int load_link_keys(struct sock *sk, struct hci_dev *hdev, void *data, MGMT_STATUS_INVALID_PARAMS); } - expected_len = sizeof(*cp) + key_count * - sizeof(struct mgmt_link_key_info); + expected_len = struct_size(cp, keys, key_count); if (expected_len != len) { bt_dev_err(hdev, "load_link_keys: expected %u bytes, got %u bytes", expected_len, len); @@ -5030,7 +5029,7 @@ static int load_irks(struct sock *sk, struct hci_dev *hdev, void *cp_data, MGMT_STATUS_INVALID_PARAMS); } - expected_len = sizeof(*cp) + irk_count * sizeof(struct mgmt_irk_info); + expected_len = struct_size(cp, irks, irk_count); if (expected_len != len) { bt_dev_err(hdev, "load_irks: expected %u bytes, got %u bytes", expected_len, len); @@ -5112,8 +5111,7 @@ static int load_long_term_keys(struct sock *sk, struct hci_dev *hdev, MGMT_STATUS_INVALID_PARAMS); } - expected_len = sizeof(*cp) + key_count * - sizeof(struct mgmt_ltk_info); + expected_len = struct_size(cp, keys, key_count); if (expected_len != len) { bt_dev_err(hdev, "load_keys: expected %u bytes, got %u bytes", expected_len, len); @@ -5847,8 +5845,7 @@ static int load_conn_param(struct sock *sk, struct hci_dev *hdev, void *data, MGMT_STATUS_INVALID_PARAMS); } - expected_len = sizeof(*cp) + param_count * - sizeof(struct mgmt_conn_param); + expected_len = struct_size(cp, params, param_count); if (expected_len != len) { bt_dev_err(hdev, "load_conn_param: expected %u bytes, got %u bytes", expected_len, len); diff --git a/net/bluetooth/rfcomm/sock.c b/net/bluetooth/rfcomm/sock.c index b1f49fcc04780e15d27b5daab6a95ec6385c8b39..90bb53aa4beed9a20488ffb842837123eda716a1 100644 --- a/net/bluetooth/rfcomm/sock.c +++ b/net/bluetooth/rfcomm/sock.c @@ -1039,6 +1039,7 @@ static const struct proto_ops rfcomm_sock_ops = { .setsockopt = rfcomm_sock_setsockopt, .getsockopt = rfcomm_sock_getsockopt, .ioctl = rfcomm_sock_ioctl, + .gettstamp = sock_gettstamp, .poll = bt_sock_poll, .socketpair = sock_no_socketpair, .mmap = sock_no_mmap diff --git a/net/bluetooth/sco.c b/net/bluetooth/sco.c index d892b7c3cc42a05e10053832d7bd4d969f019e46..b91d6b440fdf6c424d919a8cd6d26b3081e72944 100644 --- a/net/bluetooth/sco.c +++ b/net/bluetooth/sco.c @@ -1190,6 +1190,7 @@ static const struct proto_ops sco_sock_ops = { .recvmsg = sco_sock_recvmsg, .poll = bt_sock_poll, .ioctl = bt_sock_ioctl, + .gettstamp = sock_gettstamp, .mmap = sock_no_mmap, .socketpair = sock_no_socketpair, .shutdown = sco_sock_shutdown, diff --git a/net/bpf/Makefile b/net/bpf/Makefile index 27b2992a06925240a8118e8182e8c7f048a73bbf..b0ca361742e43bc71d822d79bf064a952a159677 100644 --- a/net/bpf/Makefile +++ b/net/bpf/Makefile @@ -1 +1 @@ -obj-y := test_run.o +obj-$(CONFIG_BPF_SYSCALL) := test_run.o diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c index fab142b796ef28e509b862af3d02523283e1f8ee..33e0dc168c16aa24fc7b9c42e17582b316b10472 100644 --- a/net/bpf/test_run.c +++ b/net/bpf/test_run.c @@ -10,9 +10,13 @@ #include #include #include +#include #include #include +#define CREATE_TRACE_POINTS +#include + static int bpf_test_run(struct bpf_prog *prog, void *ctx, u32 repeat, u32 *retval, u32 *time) { @@ -100,6 +104,7 @@ static int bpf_test_finish(const union bpf_attr *kattr, if (err != -ENOSPC) err = 0; out: + trace_bpf_test_finish(&err); return err; } @@ -123,12 +128,126 @@ static void *bpf_test_init(const union bpf_attr *kattr, u32 size, return data; } +static void *bpf_ctx_init(const union bpf_attr *kattr, u32 max_size) +{ + void __user *data_in = u64_to_user_ptr(kattr->test.ctx_in); + void __user *data_out = u64_to_user_ptr(kattr->test.ctx_out); + u32 size = kattr->test.ctx_size_in; + void *data; + int err; + + if (!data_in && !data_out) + return NULL; + + data = kzalloc(max_size, GFP_USER); + if (!data) + return ERR_PTR(-ENOMEM); + + if (data_in) { + err = bpf_check_uarg_tail_zero(data_in, max_size, size); + if (err) { + kfree(data); + return ERR_PTR(err); + } + + size = min_t(u32, max_size, size); + if (copy_from_user(data, data_in, size)) { + kfree(data); + return ERR_PTR(-EFAULT); + } + } + return data; +} + +static int bpf_ctx_finish(const union bpf_attr *kattr, + union bpf_attr __user *uattr, const void *data, + u32 size) +{ + void __user *data_out = u64_to_user_ptr(kattr->test.ctx_out); + int err = -EFAULT; + u32 copy_size = size; + + if (!data || !data_out) + return 0; + + if (copy_size > kattr->test.ctx_size_out) { + copy_size = kattr->test.ctx_size_out; + err = -ENOSPC; + } + + if (copy_to_user(data_out, data, copy_size)) + goto out; + if (copy_to_user(&uattr->test.ctx_size_out, &size, sizeof(size))) + goto out; + if (err != -ENOSPC) + err = 0; +out: + return err; +} + +/** + * range_is_zero - test whether buffer is initialized + * @buf: buffer to check + * @from: check from this position + * @to: check up until (excluding) this position + * + * This function returns true if the there is a non-zero byte + * in the buf in the range [from,to). + */ +static inline bool range_is_zero(void *buf, size_t from, size_t to) +{ + return !memchr_inv((u8 *)buf + from, 0, to - from); +} + +static int convert___skb_to_skb(struct sk_buff *skb, struct __sk_buff *__skb) +{ + struct qdisc_skb_cb *cb = (struct qdisc_skb_cb *)skb->cb; + + if (!__skb) + return 0; + + /* make sure the fields we don't use are zeroed */ + if (!range_is_zero(__skb, 0, offsetof(struct __sk_buff, priority))) + return -EINVAL; + + /* priority is allowed */ + + if (!range_is_zero(__skb, offsetof(struct __sk_buff, priority) + + FIELD_SIZEOF(struct __sk_buff, priority), + offsetof(struct __sk_buff, cb))) + return -EINVAL; + + /* cb is allowed */ + + if (!range_is_zero(__skb, offsetof(struct __sk_buff, cb) + + FIELD_SIZEOF(struct __sk_buff, cb), + sizeof(struct __sk_buff))) + return -EINVAL; + + skb->priority = __skb->priority; + memcpy(&cb->data, __skb->cb, QDISC_CB_PRIV_LEN); + + return 0; +} + +static void convert_skb_to___skb(struct sk_buff *skb, struct __sk_buff *__skb) +{ + struct qdisc_skb_cb *cb = (struct qdisc_skb_cb *)skb->cb; + + if (!__skb) + return; + + __skb->priority = skb->priority; + memcpy(__skb->cb, &cb->data, QDISC_CB_PRIV_LEN); +} + int bpf_prog_test_run_skb(struct bpf_prog *prog, const union bpf_attr *kattr, union bpf_attr __user *uattr) { bool is_l2 = false, is_direct_pkt_access = false; u32 size = kattr->test.data_size_in; u32 repeat = kattr->test.repeat; + struct __sk_buff *ctx = NULL; u32 retval, duration; int hh_len = ETH_HLEN; struct sk_buff *skb; @@ -141,6 +260,12 @@ int bpf_prog_test_run_skb(struct bpf_prog *prog, const union bpf_attr *kattr, if (IS_ERR(data)) return PTR_ERR(data); + ctx = bpf_ctx_init(kattr, sizeof(struct __sk_buff)); + if (IS_ERR(ctx)) { + kfree(data); + return PTR_ERR(ctx); + } + switch (prog->type) { case BPF_PROG_TYPE_SCHED_CLS: case BPF_PROG_TYPE_SCHED_ACT: @@ -158,6 +283,7 @@ int bpf_prog_test_run_skb(struct bpf_prog *prog, const union bpf_attr *kattr, sk = kzalloc(sizeof(struct sock), GFP_USER); if (!sk) { kfree(data); + kfree(ctx); return -ENOMEM; } sock_net_set(sk, current->nsproxy->net_ns); @@ -166,6 +292,7 @@ int bpf_prog_test_run_skb(struct bpf_prog *prog, const union bpf_attr *kattr, skb = build_skb(data, 0); if (!skb) { kfree(data); + kfree(ctx); kfree(sk); return -ENOMEM; } @@ -180,32 +307,38 @@ int bpf_prog_test_run_skb(struct bpf_prog *prog, const union bpf_attr *kattr, __skb_push(skb, hh_len); if (is_direct_pkt_access) bpf_compute_data_pointers(skb); + ret = convert___skb_to_skb(skb, ctx); + if (ret) + goto out; ret = bpf_test_run(prog, skb, repeat, &retval, &duration); - if (ret) { - kfree_skb(skb); - kfree(sk); - return ret; - } + if (ret) + goto out; if (!is_l2) { if (skb_headroom(skb) < hh_len) { int nhead = HH_DATA_ALIGN(hh_len - skb_headroom(skb)); if (pskb_expand_head(skb, nhead, 0, GFP_USER)) { - kfree_skb(skb); - kfree(sk); - return -ENOMEM; + ret = -ENOMEM; + goto out; } } memset(__skb_push(skb, hh_len), 0, hh_len); } + convert_skb_to___skb(skb, ctx); size = skb->len; /* bpf program can never convert linear skb to non-linear */ if (WARN_ON_ONCE(skb_is_nonlinear(skb))) size = skb_headlen(skb); ret = bpf_test_finish(kattr, uattr, skb->data, size, retval, duration); + if (!ret) + ret = bpf_ctx_finish(kattr, uattr, ctx, + sizeof(struct __sk_buff)); +out: kfree_skb(skb); + bpf_sk_storage_free(sk); kfree(sk); + kfree(ctx); return ret; } @@ -220,6 +353,9 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr, void *data; int ret; + if (kattr->test.ctx_in || kattr->test.ctx_out) + return -EINVAL; + data = bpf_test_init(kattr, size, XDP_PACKET_HEADROOM + NET_IP_ALIGN, 0); if (IS_ERR(data)) return PTR_ERR(data); @@ -249,13 +385,12 @@ int bpf_prog_test_run_flow_dissector(struct bpf_prog *prog, union bpf_attr __user *uattr) { u32 size = kattr->test.data_size_in; + struct bpf_flow_dissector ctx = {}; u32 repeat = kattr->test.repeat; struct bpf_flow_keys flow_keys; u64 time_start, time_spent = 0; - struct bpf_skb_data_end *cb; + const struct ethhdr *eth; u32 retval, duration; - struct sk_buff *skb; - struct sock *sk; void *data; int ret; u32 i; @@ -263,46 +398,31 @@ int bpf_prog_test_run_flow_dissector(struct bpf_prog *prog, if (prog->type != BPF_PROG_TYPE_FLOW_DISSECTOR) return -EINVAL; - data = bpf_test_init(kattr, size, NET_SKB_PAD + NET_IP_ALIGN, - SKB_DATA_ALIGN(sizeof(struct skb_shared_info))); - if (IS_ERR(data)) - return PTR_ERR(data); - - sk = kzalloc(sizeof(*sk), GFP_USER); - if (!sk) { - kfree(data); - return -ENOMEM; - } - sock_net_set(sk, current->nsproxy->net_ns); - sock_init_data(NULL, sk); + if (kattr->test.ctx_in || kattr->test.ctx_out) + return -EINVAL; - skb = build_skb(data, 0); - if (!skb) { - kfree(data); - kfree(sk); - return -ENOMEM; - } - skb->sk = sk; + if (size < ETH_HLEN) + return -EINVAL; - skb_reserve(skb, NET_SKB_PAD + NET_IP_ALIGN); - __skb_put(skb, size); - skb->protocol = eth_type_trans(skb, - current->nsproxy->net_ns->loopback_dev); - skb_reset_network_header(skb); + data = bpf_test_init(kattr, size, 0, 0); + if (IS_ERR(data)) + return PTR_ERR(data); - cb = (struct bpf_skb_data_end *)skb->cb; - cb->qdisc_cb.flow_keys = &flow_keys; + eth = (struct ethhdr *)data; if (!repeat) repeat = 1; + ctx.flow_keys = &flow_keys; + ctx.data = data; + ctx.data_end = (__u8 *)data + size; + rcu_read_lock(); preempt_disable(); time_start = ktime_get_ns(); for (i = 0; i < repeat; i++) { - retval = __skb_flow_bpf_dissect(prog, skb, - &flow_keys_dissector, - &flow_keys); + retval = bpf_flow_dissect(prog, &ctx, eth->h_proto, ETH_HLEN, + size); if (signal_pending(current)) { preempt_enable(); @@ -335,7 +455,6 @@ int bpf_prog_test_run_flow_dissector(struct bpf_prog *prog, retval, duration); out: - kfree_skb(skb); - kfree(sk); + kfree(data); return ret; } diff --git a/net/bridge/br.c b/net/bridge/br.c index a5174e5001d86a5b16faadd519167820284961a9..3c8e4b38f0548fa0eb4fcab7faba97b7affd64b8 100644 --- a/net/bridge/br.c +++ b/net/bridge/br.c @@ -40,10 +40,13 @@ static int br_device_event(struct notifier_block *unused, unsigned long event, v bool changed_addr; int err; - /* register of bridge completed, add sysfs entries */ - if ((dev->priv_flags & IFF_EBRIDGE) && event == NETDEV_REGISTER) { - br_sysfs_addbr(dev); - return NOTIFY_DONE; + if (dev->priv_flags & IFF_EBRIDGE) { + if (event == NETDEV_REGISTER) { + /* register of bridge completed, add sysfs entries */ + br_sysfs_addbr(dev); + return NOTIFY_DONE; + } + br_vlan_bridge_event(dev, event, ptr); } /* not a port of a bridge */ @@ -126,6 +129,9 @@ static int br_device_event(struct notifier_block *unused, unsigned long event, v break; } + if (event != NETDEV_UNREGISTER) + br_vlan_port_event(p, event); + /* Events that may cause spanning tree to refresh */ if (!notified && (event == NETDEV_CHANGEADDR || event == NETDEV_UP || event == NETDEV_CHANGE || event == NETDEV_DOWN)) diff --git a/net/bridge/br_arp_nd_proxy.c b/net/bridge/br_arp_nd_proxy.c index 6b78e63517193af21d65efc750c6ca83bc92d9e1..15116752365a601ab3fe424cfdcaf0167bade8e8 100644 --- a/net/bridge/br_arp_nd_proxy.c +++ b/net/bridge/br_arp_nd_proxy.c @@ -21,6 +21,7 @@ #include #include #include +#include #if IS_ENABLED(CONFIG_IPV6) #include #endif @@ -130,7 +131,7 @@ void br_do_proxy_suppress_arp(struct sk_buff *skb, struct net_bridge *br, u8 *arpptr, *sha; __be32 sip, tip; - BR_INPUT_SKB_CB(skb)->proxyarp_replied = false; + BR_INPUT_SKB_CB(skb)->proxyarp_replied = 0; if ((dev->flags & IFF_NOARP) || !pskb_may_pull(skb, arp_hdr_len(dev))) @@ -160,7 +161,7 @@ void br_do_proxy_suppress_arp(struct sk_buff *skb, struct net_bridge *br, return; if (ipv4_is_zeronet(sip) || sip == tip) { /* prevent flooding to neigh suppress ports */ - BR_INPUT_SKB_CB(skb)->proxyarp_replied = true; + BR_INPUT_SKB_CB(skb)->proxyarp_replied = 1; return; } } @@ -180,7 +181,7 @@ void br_do_proxy_suppress_arp(struct sk_buff *skb, struct net_bridge *br, /* its our local ip, so don't proxy reply * and don't forward to neigh suppress ports */ - BR_INPUT_SKB_CB(skb)->proxyarp_replied = true; + BR_INPUT_SKB_CB(skb)->proxyarp_replied = 1; return; } @@ -216,7 +217,7 @@ void br_do_proxy_suppress_arp(struct sk_buff *skb, struct net_bridge *br, */ if (replied || br_opt_get(br, BROPT_NEIGH_SUPPRESS_ENABLED)) - BR_INPUT_SKB_CB(skb)->proxyarp_replied = true; + BR_INPUT_SKB_CB(skb)->proxyarp_replied = 1; } neigh_release(n); @@ -392,7 +393,7 @@ void br_do_suppress_nd(struct sk_buff *skb, struct net_bridge *br, struct ipv6hdr *iphdr; struct neighbour *n; - BR_INPUT_SKB_CB(skb)->proxyarp_replied = false; + BR_INPUT_SKB_CB(skb)->proxyarp_replied = 0; if (p && (p->flags & BR_NEIGH_SUPPRESS)) return; @@ -400,7 +401,7 @@ void br_do_suppress_nd(struct sk_buff *skb, struct net_bridge *br, if (msg->icmph.icmp6_type == NDISC_NEIGHBOUR_ADVERTISEMENT && !msg->icmph.icmp6_solicited) { /* prevent flooding to neigh suppress ports */ - BR_INPUT_SKB_CB(skb)->proxyarp_replied = true; + BR_INPUT_SKB_CB(skb)->proxyarp_replied = 1; return; } @@ -413,7 +414,7 @@ void br_do_suppress_nd(struct sk_buff *skb, struct net_bridge *br, if (ipv6_addr_any(saddr) || !ipv6_addr_cmp(saddr, daddr)) { /* prevent flooding to neigh suppress ports */ - BR_INPUT_SKB_CB(skb)->proxyarp_replied = true; + BR_INPUT_SKB_CB(skb)->proxyarp_replied = 1; return; } @@ -431,7 +432,7 @@ void br_do_suppress_nd(struct sk_buff *skb, struct net_bridge *br, /* its our own ip, so don't proxy reply * and don't forward to arp suppress ports */ - BR_INPUT_SKB_CB(skb)->proxyarp_replied = true; + BR_INPUT_SKB_CB(skb)->proxyarp_replied = 1; return; } @@ -464,7 +465,7 @@ void br_do_suppress_nd(struct sk_buff *skb, struct net_bridge *br, */ if (replied || br_opt_get(br, BROPT_NEIGH_SUPPRESS_ENABLED)) - BR_INPUT_SKB_CB(skb)->proxyarp_replied = true; + BR_INPUT_SKB_CB(skb)->proxyarp_replied = 1; } neigh_release(n); } diff --git a/net/bridge/br_fdb.c b/net/bridge/br_fdb.c index 00573cc46c98b4358679b178d772a28501bd3bd5..b1c91f66d79c4e3e1beca4d091a7db7b5ffe3b34 100644 --- a/net/bridge/br_fdb.c +++ b/net/bridge/br_fdb.c @@ -33,7 +33,6 @@ static const struct rhashtable_params br_fdb_rht_params = { .key_offset = offsetof(struct net_bridge_fdb_entry, key), .key_len = sizeof(struct net_bridge_fdb_key), .automatic_shrinking = true, - .locks_mul = 1, }; static struct kmem_cache *br_fdb_cache __read_mostly; diff --git a/net/bridge/br_forward.c b/net/bridge/br_forward.c index 48ddc60b4fbdece571592bc86689860707ddf6ba..82225b8b54f5989a0441a740515e71ea25614bd4 100644 --- a/net/bridge/br_forward.c +++ b/net/bridge/br_forward.c @@ -173,6 +173,7 @@ static struct net_bridge_port *maybe_deliver( struct net_bridge_port *prev, struct net_bridge_port *p, struct sk_buff *skb, bool local_orig) { + u8 igmp_type = br_multicast_igmp_type(skb); int err; if (!should_deliver(p, skb)) @@ -184,8 +185,9 @@ static struct net_bridge_port *maybe_deliver( err = deliver_clone(prev, skb, local_orig); if (err) return ERR_PTR(err); - out: + br_multicast_count(p->br, p, skb, igmp_type, BR_MCAST_DIR_TX); + return p; } @@ -193,7 +195,6 @@ out: void br_flood(struct net_bridge *br, struct sk_buff *skb, enum br_pkt_type pkt_type, bool local_rcv, bool local_orig) { - u8 igmp_type = br_multicast_igmp_type(skb); struct net_bridge_port *prev = NULL; struct net_bridge_port *p; @@ -226,9 +227,6 @@ void br_flood(struct net_bridge *br, struct sk_buff *skb, prev = maybe_deliver(prev, p, skb, local_orig); if (IS_ERR(prev)) goto out; - if (prev == p) - br_multicast_count(p->br, p, skb, igmp_type, - BR_MCAST_DIR_TX); } if (!prev) @@ -277,7 +275,6 @@ void br_multicast_flood(struct net_bridge_mdb_entry *mdst, bool local_rcv, bool local_orig) { struct net_device *dev = BR_INPUT_SKB_CB(skb)->brdev; - u8 igmp_type = br_multicast_igmp_type(skb); struct net_bridge *br = netdev_priv(dev); struct net_bridge_port *prev = NULL; struct net_bridge_port_group *p; @@ -304,13 +301,9 @@ void br_multicast_flood(struct net_bridge_mdb_entry *mdst, } prev = maybe_deliver(prev, port, skb, local_orig); -delivered: if (IS_ERR(prev)) goto out; - if (prev == port) - br_multicast_count(port->br, port, skb, igmp_type, - BR_MCAST_DIR_TX); - +delivered: if ((unsigned long)lport >= (unsigned long)port) p = rcu_dereference(p->next); if ((unsigned long)rport >= (unsigned long)port) diff --git a/net/bridge/br_if.c b/net/bridge/br_if.c index 41f0a696a65f4362a4e1507024dcff0ce1a9208e..4a9aaa3fac8fdfa39a66a6712baa3c1e35722d80 100644 --- a/net/bridge/br_if.c +++ b/net/bridge/br_if.c @@ -179,7 +179,7 @@ int nbp_backup_change(struct net_bridge_port *p, ASSERT_RTNL(); if (backup_dev) { - if (!br_port_exists(backup_dev)) + if (!netif_is_bridge_port(backup_dev)) return -ENOENT; backup_p = br_port_get_rtnl(backup_dev); diff --git a/net/bridge/br_input.c b/net/bridge/br_input.c index ba303ee99b9b59762e724072d0f66907f46235b2..014af7efef251871fdbf2fbca2a60714b1a3bc6f 100644 --- a/net/bridge/br_input.c +++ b/net/bridge/br_input.c @@ -16,6 +16,9 @@ #include #include #include +#ifdef CONFIG_NETFILTER_FAMILY_BRIDGE +#include +#endif #include #include #include @@ -23,10 +26,6 @@ #include "br_private.h" #include "br_private_tunnel.h" -/* Hook for brouter */ -br_should_route_hook_t __rcu *br_should_route_hook __read_mostly; -EXPORT_SYMBOL(br_should_route_hook); - static int br_netif_receive_skb(struct net *net, struct sock *sk, struct sk_buff *skb) { @@ -203,6 +202,59 @@ static int br_handle_local_finish(struct net *net, struct sock *sk, struct sk_bu return 1; } +static int nf_hook_bridge_pre(struct sk_buff *skb, struct sk_buff **pskb) +{ +#ifdef CONFIG_NETFILTER_FAMILY_BRIDGE + struct nf_hook_entries *e = NULL; + struct nf_hook_state state; + unsigned int verdict, i; + struct net *net; + int ret; + + net = dev_net(skb->dev); +#ifdef HAVE_JUMP_LABEL + if (!static_key_false(&nf_hooks_needed[NFPROTO_BRIDGE][NF_BR_PRE_ROUTING])) + goto frame_finish; +#endif + + e = rcu_dereference(net->nf.hooks_bridge[NF_BR_PRE_ROUTING]); + if (!e) + goto frame_finish; + + nf_hook_state_init(&state, NF_BR_PRE_ROUTING, + NFPROTO_BRIDGE, skb->dev, NULL, NULL, + net, br_handle_frame_finish); + + for (i = 0; i < e->num_hook_entries; i++) { + verdict = nf_hook_entry_hookfn(&e->hooks[i], skb, &state); + switch (verdict & NF_VERDICT_MASK) { + case NF_ACCEPT: + if (BR_INPUT_SKB_CB(skb)->br_netfilter_broute) { + *pskb = skb; + return RX_HANDLER_PASS; + } + break; + case NF_DROP: + kfree_skb(skb); + return RX_HANDLER_CONSUMED; + case NF_QUEUE: + ret = nf_queue(skb, &state, e, i, verdict); + if (ret == 1) + continue; + return RX_HANDLER_CONSUMED; + default: /* STOLEN */ + return RX_HANDLER_CONSUMED; + } + } +frame_finish: + net = dev_net(skb->dev); + br_handle_frame_finish(net, NULL, skb); +#else + br_handle_frame_finish(dev_net(skb->dev), NULL, skb); +#endif + return RX_HANDLER_CONSUMED; +} + /* * Return NULL if skb is handled * note: already called with rcu_read_lock @@ -212,7 +264,6 @@ rx_handler_result_t br_handle_frame(struct sk_buff **pskb) struct net_bridge_port *p; struct sk_buff *skb = *pskb; const unsigned char *dest = eth_hdr(skb)->h_dest; - br_should_route_hook_t *rhook; if (unlikely(skb->pkt_type == PACKET_LOOPBACK)) return RX_HANDLER_PASS; @@ -224,6 +275,8 @@ rx_handler_result_t br_handle_frame(struct sk_buff **pskb) if (!skb) return RX_HANDLER_CONSUMED; + memset(skb->cb, 0, sizeof(struct br_input_skb_cb)); + p = br_port_get_rcu(skb->dev); if (p->flags & BR_VLAN_TUNNEL) { if (br_handle_ingress_vlan_tunnel(skb, p, @@ -294,23 +347,11 @@ rx_handler_result_t br_handle_frame(struct sk_buff **pskb) forward: switch (p->state) { case BR_STATE_FORWARDING: - rhook = rcu_dereference(br_should_route_hook); - if (rhook) { - if ((*rhook)(skb)) { - *pskb = skb; - return RX_HANDLER_PASS; - } - dest = eth_hdr(skb)->h_dest; - } - /* fall through */ case BR_STATE_LEARNING: if (ether_addr_equal(p->br->dev->dev_addr, dest)) skb->pkt_type = PACKET_HOST; - NF_HOOK(NFPROTO_BRIDGE, NF_BR_PRE_ROUTING, - dev_net(skb->dev), NULL, skb, skb->dev, NULL, - br_handle_frame_finish); - break; + return nf_hook_bridge_pre(skb, pskb); default: drop: kfree_skb(skb); diff --git a/net/bridge/br_mdb.c b/net/bridge/br_mdb.c index f69c8d91dc8189773223080c4c8c8f8ef0924a36..bf6acd34234daa8a1da4aff928c169837bcf58c1 100644 --- a/net/bridge/br_mdb.c +++ b/net/bridge/br_mdb.c @@ -26,14 +26,14 @@ static int br_rports_fill_info(struct sk_buff *skb, struct netlink_callback *cb, if (!br->multicast_router || hlist_empty(&br->router_list)) return 0; - nest = nla_nest_start(skb, MDBA_ROUTER); + nest = nla_nest_start_noflag(skb, MDBA_ROUTER); if (nest == NULL) return -EMSGSIZE; hlist_for_each_entry_rcu(p, &br->router_list, rlist) { if (!p) continue; - port_nest = nla_nest_start(skb, MDBA_ROUTER_PORT); + port_nest = nla_nest_start_noflag(skb, MDBA_ROUTER_PORT); if (!port_nest) goto fail; if (nla_put_nohdr(skb, sizeof(u32), &p->dev->ifindex) || @@ -86,7 +86,7 @@ static int br_mdb_fill_info(struct sk_buff *skb, struct netlink_callback *cb, if (!br_opt_get(br, BROPT_MULTICAST_ENABLED)) return 0; - nest = nla_nest_start(skb, MDBA_MDB); + nest = nla_nest_start_noflag(skb, MDBA_MDB); if (nest == NULL) return -EMSGSIZE; @@ -98,7 +98,7 @@ static int br_mdb_fill_info(struct sk_buff *skb, struct netlink_callback *cb, if (idx < s_idx) goto skip; - nest2 = nla_nest_start(skb, MDBA_MDB_ENTRY); + nest2 = nla_nest_start_noflag(skb, MDBA_MDB_ENTRY); if (!nest2) { err = -EMSGSIZE; break; @@ -124,7 +124,8 @@ static int br_mdb_fill_info(struct sk_buff *skb, struct netlink_callback *cb, e.addr.u.ip6 = p->addr.u.ip6; #endif e.addr.proto = p->addr.proto; - nest_ent = nla_nest_start(skb, MDBA_MDB_ENTRY_INFO); + nest_ent = nla_nest_start_noflag(skb, + MDBA_MDB_ENTRY_INFO); if (!nest_ent) { nla_nest_cancel(skb, nest2); err = -EMSGSIZE; @@ -248,10 +249,10 @@ static int nlmsg_populate_mdb_fill(struct sk_buff *skb, memset(bpm, 0, sizeof(*bpm)); bpm->family = AF_BRIDGE; bpm->ifindex = dev->ifindex; - nest = nla_nest_start(skb, MDBA_MDB); + nest = nla_nest_start_noflag(skb, MDBA_MDB); if (nest == NULL) goto cancel; - nest2 = nla_nest_start(skb, MDBA_MDB_ENTRY); + nest2 = nla_nest_start_noflag(skb, MDBA_MDB_ENTRY); if (nest2 == NULL) goto end; @@ -444,7 +445,7 @@ static int nlmsg_populate_rtr_fill(struct sk_buff *skb, memset(bpm, 0, sizeof(*bpm)); bpm->family = AF_BRIDGE; bpm->ifindex = dev->ifindex; - nest = nla_nest_start(skb, MDBA_ROUTER); + nest = nla_nest_start_noflag(skb, MDBA_ROUTER); if (!nest) goto cancel; @@ -529,8 +530,8 @@ static int br_mdb_parse(struct sk_buff *skb, struct nlmsghdr *nlh, struct net_device *dev; int err; - err = nlmsg_parse(nlh, sizeof(*bpm), tb, MDBA_SET_ENTRY_MAX, NULL, - NULL); + err = nlmsg_parse_deprecated(nlh, sizeof(*bpm), tb, + MDBA_SET_ENTRY_MAX, NULL, NULL); if (err < 0) return err; diff --git a/net/bridge/br_multicast.c b/net/bridge/br_multicast.c index 45e7f4173bbafe7e59e2ea514a7bccbee8456c79..c2a30f79a9d013de41e90e969324cb19075dfbfe 100644 --- a/net/bridge/br_multicast.c +++ b/net/bridge/br_multicast.c @@ -44,7 +44,6 @@ static const struct rhashtable_params br_mdb_rht_params = { .key_offset = offsetof(struct net_bridge_mdb_entry, addr), .key_len = sizeof(struct br_ip), .automatic_shrinking = true, - .locks_mul = 1, }; static void br_multicast_start_querier(struct net_bridge *br, @@ -65,23 +64,6 @@ static void br_ip6_multicast_leave_group(struct net_bridge *br, __u16 vid, const unsigned char *src); #endif -static inline int br_ip_equal(const struct br_ip *a, const struct br_ip *b) -{ - if (a->proto != b->proto) - return 0; - if (a->vid != b->vid) - return 0; - switch (a->proto) { - case htons(ETH_P_IP): - return a->u.ip4 == b->u.ip4; -#if IS_ENABLED(CONFIG_IPV6) - case htons(ETH_P_IPV6): - return ipv6_addr_equal(&a->u.ip6, &b->u.ip6); -#endif - } - return 0; -} - static struct net_bridge_mdb_entry *br_mdb_ip_get_rcu(struct net_bridge *br, struct br_ip *dst) { @@ -517,7 +499,7 @@ struct net_bridge_port_group *br_multicast_new_port_group( if (src) memcpy(p->eth_addr, src, ETH_ALEN); else - memset(p->eth_addr, 0xff, ETH_ALEN); + eth_broadcast_addr(p->eth_addr); return p; } @@ -2194,7 +2176,7 @@ int br_multicast_list_adjacent(struct net_device *dev, int count = 0; rcu_read_lock(); - if (!br_ip_list || !br_port_exists(dev)) + if (!br_ip_list || !netif_is_bridge_port(dev)) goto unlock; port = br_port_get_rcu(dev); @@ -2241,7 +2223,7 @@ bool br_multicast_has_querier_anywhere(struct net_device *dev, int proto) bool ret = false; rcu_read_lock(); - if (!br_port_exists(dev)) + if (!netif_is_bridge_port(dev)) goto unlock; port = br_port_get_rcu(dev); @@ -2277,7 +2259,7 @@ bool br_multicast_has_querier_adjacent(struct net_device *dev, int proto) bool ret = false; rcu_read_lock(); - if (!br_port_exists(dev)) + if (!netif_is_bridge_port(dev)) goto unlock; port = br_port_get_rcu(dev); diff --git a/net/bridge/br_netlink.c b/net/bridge/br_netlink.c index 7104cf13da840d21cca1a63d3c1551ff9bbd9076..a5acad29cd4f1df4cc8e228b2e6e78c82676200d 100644 --- a/net/bridge/br_netlink.c +++ b/net/bridge/br_netlink.c @@ -102,7 +102,7 @@ static size_t br_get_link_af_size_filtered(const struct net_device *dev, size_t vinfo_sz = 0; rcu_read_lock(); - if (br_port_exists(dev)) { + if (netif_is_bridge_port(dev)) { p = br_port_get_rcu(dev); vg = nbp_vlan_group_rcu(p); } else if (dev->priv_flags & IFF_EBRIDGE) { @@ -413,9 +413,9 @@ static int br_fill_ifinfo(struct sk_buff *skb, goto nla_put_failure; if (event == RTM_NEWLINK && port) { - struct nlattr *nest - = nla_nest_start(skb, IFLA_PROTINFO | NLA_F_NESTED); + struct nlattr *nest; + nest = nla_nest_start(skb, IFLA_PROTINFO); if (nest == NULL || br_port_fill_attrs(skb, port) < 0) goto nla_put_failure; nla_nest_end(skb, nest); @@ -439,7 +439,7 @@ static int br_fill_ifinfo(struct sk_buff *skb, rcu_read_unlock(); goto done; } - af = nla_nest_start(skb, IFLA_AF_SPEC); + af = nla_nest_start_noflag(skb, IFLA_AF_SPEC); if (!af) { rcu_read_unlock(); goto nla_put_failure; @@ -880,8 +880,10 @@ int br_setlink(struct net_device *dev, struct nlmsghdr *nlh, u16 flags, if (p && protinfo) { if (protinfo->nla_type & NLA_F_NESTED) { - err = nla_parse_nested(tb, IFLA_BRPORT_MAX, protinfo, - br_port_policy, NULL); + err = nla_parse_nested_deprecated(tb, IFLA_BRPORT_MAX, + protinfo, + br_port_policy, + NULL); if (err) return err; @@ -1569,7 +1571,7 @@ static int br_fill_linkxstats(struct sk_buff *skb, return -EINVAL; } - nest = nla_nest_start(skb, LINK_XSTATS_TYPE_BRIDGE); + nest = nla_nest_start_noflag(skb, LINK_XSTATS_TYPE_BRIDGE); if (!nest) return -EMSGSIZE; diff --git a/net/bridge/br_netlink_tunnel.c b/net/bridge/br_netlink_tunnel.c index da8cb99fd2591d15b77a8fb98540a2c9b034f6c5..34629d558709e2de843605520a5d399d8f2de8ad 100644 --- a/net/bridge/br_netlink_tunnel.c +++ b/net/bridge/br_netlink_tunnel.c @@ -97,7 +97,7 @@ static int br_fill_vlan_tinfo(struct sk_buff *skb, u16 vid, __be32 tid = tunnel_id_to_key32(tunnel_id); struct nlattr *tmap; - tmap = nla_nest_start(skb, IFLA_BRIDGE_VLAN_TUNNEL_INFO); + tmap = nla_nest_start_noflag(skb, IFLA_BRIDGE_VLAN_TUNNEL_INFO); if (!tmap) return -EMSGSIZE; if (nla_put_u32(skb, IFLA_BRIDGE_VLAN_TUNNEL_ID, @@ -230,8 +230,8 @@ int br_parse_vlan_tunnel_info(struct nlattr *attr, memset(tinfo, 0, sizeof(*tinfo)); - err = nla_parse_nested(tb, IFLA_BRIDGE_VLAN_TUNNEL_MAX, attr, - vlan_tunnel_policy, NULL); + err = nla_parse_nested_deprecated(tb, IFLA_BRIDGE_VLAN_TUNNEL_MAX, + attr, vlan_tunnel_policy, NULL); if (err < 0) return err; diff --git a/net/bridge/br_private.h b/net/bridge/br_private.h index 00deef7fc1f303656b8f7c7ce7b094ba6194944a..334a8c496b50ca96211157491a95ba7eafdf9742 100644 --- a/net/bridge/br_private.h +++ b/net/bridge/br_private.h @@ -288,8 +288,6 @@ struct net_bridge_port { #define br_auto_port(p) ((p)->flags & BR_AUTO_MASK) #define br_promisc_port(p) ((p)->flags & BR_PROMISC) -#define br_port_exists(dev) (dev->priv_flags & IFF_BRIDGE_PORT) - static inline struct net_bridge_port *br_port_get_rcu(const struct net_device *dev) { return rcu_dereference(dev->rx_handler_data); @@ -297,13 +295,13 @@ static inline struct net_bridge_port *br_port_get_rcu(const struct net_device *d static inline struct net_bridge_port *br_port_get_rtnl(const struct net_device *dev) { - return br_port_exists(dev) ? + return netif_is_bridge_port(dev) ? rtnl_dereference(dev->rx_handler_data) : NULL; } static inline struct net_bridge_port *br_port_get_rtnl_rcu(const struct net_device *dev) { - return br_port_exists(dev) ? + return netif_is_bridge_port(dev) ? rcu_dereference_rtnl(dev->rx_handler_data) : NULL; } @@ -323,6 +321,7 @@ enum net_bridge_opts { BROPT_MTU_SET_BY_USER, BROPT_VLAN_STATS_PER_PORT, BROPT_NO_LL_LEARN, + BROPT_VLAN_BRIDGE_BINDING, }; struct net_bridge { @@ -427,15 +426,16 @@ struct br_input_skb_cb { struct net_device *brdev; #ifdef CONFIG_BRIDGE_IGMP_SNOOPING - int igmp; - int mrouters_only; + u8 igmp; + u8 mrouters_only:1; #endif - - bool proxyarp_replied; - bool src_port_isolated; - + u8 proxyarp_replied:1; + u8 src_port_isolated:1; #ifdef CONFIG_BRIDGE_VLAN_FILTERING - bool vlan_filtered; + u8 vlan_filtered:1; +#endif +#ifdef CONFIG_NETFILTER_FAMILY_BRIDGE + u8 br_netfilter_broute:1; #endif #ifdef CONFIG_NET_SWITCHDEV @@ -896,6 +896,9 @@ int nbp_vlan_init(struct net_bridge_port *port, struct netlink_ext_ack *extack); int nbp_get_num_vlan_infos(struct net_bridge_port *p, u32 filter_mask); void br_vlan_get_stats(const struct net_bridge_vlan *v, struct br_vlan_stats *stats); +void br_vlan_port_event(struct net_bridge_port *p, unsigned long event); +void br_vlan_bridge_event(struct net_device *dev, unsigned long event, + void *ptr); static inline struct net_bridge_vlan_group *br_vlan_group( const struct net_bridge *br) @@ -1079,6 +1082,16 @@ static inline void br_vlan_get_stats(const struct net_bridge_vlan *v, struct br_vlan_stats *stats) { } + +static inline void br_vlan_port_event(struct net_bridge_port *p, + unsigned long event) +{ +} + +static inline void br_vlan_bridge_event(struct net_device *dev, + unsigned long event, void *ptr) +{ +} #endif struct nf_br_ops { diff --git a/net/bridge/br_stp_if.c b/net/bridge/br_stp_if.c index 808e2b914015fa4dd69e7736054c20e3c9e1aaba..8d65ae5210e09b7e732feaf2d60d5014de33a7e5 100644 --- a/net/bridge/br_stp_if.c +++ b/net/bridge/br_stp_if.c @@ -117,7 +117,8 @@ void br_stp_disable_port(struct net_bridge_port *p) del_timer(&p->forward_delay_timer); del_timer(&p->hold_timer); - br_fdb_delete_by_port(br, p, 0, 0); + if (!rcu_access_pointer(p->backup_port)) + br_fdb_delete_by_port(br, p, 0, 0); br_multicast_disable_port(p); br_configuration_update(br); diff --git a/net/bridge/br_vlan.c b/net/bridge/br_vlan.c index 96abf8feb9dc6c2e01a39fa1f9827fc4da1bb1b7..2db63997f313533662f5280f3afca5a2bd3ae22a 100644 --- a/net/bridge/br_vlan.c +++ b/net/bridge/br_vlan.c @@ -7,6 +7,8 @@ #include "br_private.h" #include "br_private_tunnel.h" +static void nbp_vlan_set_vlan_dev_state(struct net_bridge_port *p, u16 vid); + static inline int br_vlan_cmp(struct rhashtable_compare_arg *arg, const void *ptr) { @@ -21,7 +23,6 @@ static const struct rhashtable_params br_vlan_rht_params = { .key_offset = offsetof(struct net_bridge_vlan, vid), .key_len = sizeof(u16), .nelem_hint = 3, - .locks_mul = 1, .max_size = VLAN_N_VID, .obj_cmpfn = br_vlan_cmp, .automatic_shrinking = true, @@ -294,6 +295,9 @@ static int __vlan_add(struct net_bridge_vlan *v, u16 flags, __vlan_add_list(v); __vlan_add_flags(v, flags); + + if (p) + nbp_vlan_set_vlan_dev_state(p, v->vid); out: return err; @@ -358,6 +362,7 @@ static int __vlan_del(struct net_bridge_vlan *v) rhashtable_remove_fast(&vg->vlan_hash, &v->vnode, br_vlan_rht_params); __vlan_del_list(v); + nbp_vlan_set_vlan_dev_state(p, v->vid); call_rcu(&v->rcu, nbp_vlan_rcu_free); } @@ -1265,3 +1270,211 @@ int br_vlan_get_info(const struct net_device *dev, u16 vid, return 0; } EXPORT_SYMBOL_GPL(br_vlan_get_info); + +static int br_vlan_is_bind_vlan_dev(const struct net_device *dev) +{ + return is_vlan_dev(dev) && + !!(vlan_dev_priv(dev)->flags & VLAN_FLAG_BRIDGE_BINDING); +} + +static int br_vlan_is_bind_vlan_dev_fn(struct net_device *dev, + __always_unused void *data) +{ + return br_vlan_is_bind_vlan_dev(dev); +} + +static bool br_vlan_has_upper_bind_vlan_dev(struct net_device *dev) +{ + int found; + + rcu_read_lock(); + found = netdev_walk_all_upper_dev_rcu(dev, br_vlan_is_bind_vlan_dev_fn, + NULL); + rcu_read_unlock(); + + return !!found; +} + +struct br_vlan_bind_walk_data { + u16 vid; + struct net_device *result; +}; + +static int br_vlan_match_bind_vlan_dev_fn(struct net_device *dev, + void *data_in) +{ + struct br_vlan_bind_walk_data *data = data_in; + int found = 0; + + if (br_vlan_is_bind_vlan_dev(dev) && + vlan_dev_priv(dev)->vlan_id == data->vid) { + data->result = dev; + found = 1; + } + + return found; +} + +static struct net_device * +br_vlan_get_upper_bind_vlan_dev(struct net_device *dev, u16 vid) +{ + struct br_vlan_bind_walk_data data = { + .vid = vid, + }; + + rcu_read_lock(); + netdev_walk_all_upper_dev_rcu(dev, br_vlan_match_bind_vlan_dev_fn, + &data); + rcu_read_unlock(); + + return data.result; +} + +static bool br_vlan_is_dev_up(const struct net_device *dev) +{ + return !!(dev->flags & IFF_UP) && netif_oper_up(dev); +} + +static void br_vlan_set_vlan_dev_state(const struct net_bridge *br, + struct net_device *vlan_dev) +{ + u16 vid = vlan_dev_priv(vlan_dev)->vlan_id; + struct net_bridge_vlan_group *vg; + struct net_bridge_port *p; + bool has_carrier = false; + + if (!netif_carrier_ok(br->dev)) { + netif_carrier_off(vlan_dev); + return; + } + + list_for_each_entry(p, &br->port_list, list) { + vg = nbp_vlan_group(p); + if (br_vlan_find(vg, vid) && br_vlan_is_dev_up(p->dev)) { + has_carrier = true; + break; + } + } + + if (has_carrier) + netif_carrier_on(vlan_dev); + else + netif_carrier_off(vlan_dev); +} + +static void br_vlan_set_all_vlan_dev_state(struct net_bridge_port *p) +{ + struct net_bridge_vlan_group *vg = nbp_vlan_group(p); + struct net_bridge_vlan *vlan; + struct net_device *vlan_dev; + + list_for_each_entry(vlan, &vg->vlan_list, vlist) { + vlan_dev = br_vlan_get_upper_bind_vlan_dev(p->br->dev, + vlan->vid); + if (vlan_dev) { + if (br_vlan_is_dev_up(p->dev)) { + if (netif_carrier_ok(p->br->dev)) + netif_carrier_on(vlan_dev); + } else { + br_vlan_set_vlan_dev_state(p->br, vlan_dev); + } + } + } +} + +static void br_vlan_upper_change(struct net_device *dev, + struct net_device *upper_dev, + bool linking) +{ + struct net_bridge *br = netdev_priv(dev); + + if (!br_vlan_is_bind_vlan_dev(upper_dev)) + return; + + if (linking) { + br_vlan_set_vlan_dev_state(br, upper_dev); + br_opt_toggle(br, BROPT_VLAN_BRIDGE_BINDING, true); + } else { + br_opt_toggle(br, BROPT_VLAN_BRIDGE_BINDING, + br_vlan_has_upper_bind_vlan_dev(dev)); + } +} + +struct br_vlan_link_state_walk_data { + struct net_bridge *br; +}; + +static int br_vlan_link_state_change_fn(struct net_device *vlan_dev, + void *data_in) +{ + struct br_vlan_link_state_walk_data *data = data_in; + + if (br_vlan_is_bind_vlan_dev(vlan_dev)) + br_vlan_set_vlan_dev_state(data->br, vlan_dev); + + return 0; +} + +static void br_vlan_link_state_change(struct net_device *dev, + struct net_bridge *br) +{ + struct br_vlan_link_state_walk_data data = { + .br = br + }; + + rcu_read_lock(); + netdev_walk_all_upper_dev_rcu(dev, br_vlan_link_state_change_fn, + &data); + rcu_read_unlock(); +} + +/* Must be protected by RTNL. */ +static void nbp_vlan_set_vlan_dev_state(struct net_bridge_port *p, u16 vid) +{ + struct net_device *vlan_dev; + + if (!br_opt_get(p->br, BROPT_VLAN_BRIDGE_BINDING)) + return; + + vlan_dev = br_vlan_get_upper_bind_vlan_dev(p->br->dev, vid); + if (vlan_dev) + br_vlan_set_vlan_dev_state(p->br, vlan_dev); +} + +/* Must be protected by RTNL. */ +void br_vlan_bridge_event(struct net_device *dev, unsigned long event, + void *ptr) +{ + struct netdev_notifier_changeupper_info *info; + struct net_bridge *br; + + switch (event) { + case NETDEV_CHANGEUPPER: + info = ptr; + br_vlan_upper_change(dev, info->upper_dev, info->linking); + break; + + case NETDEV_CHANGE: + case NETDEV_UP: + br = netdev_priv(dev); + if (!br_opt_get(br, BROPT_VLAN_BRIDGE_BINDING)) + return; + br_vlan_link_state_change(dev, br); + break; + } +} + +/* Must be protected by RTNL. */ +void br_vlan_port_event(struct net_bridge_port *p, unsigned long event) +{ + if (!br_opt_get(p->br, BROPT_VLAN_BRIDGE_BINDING)) + return; + + switch (event) { + case NETDEV_CHANGE: + case NETDEV_DOWN: + case NETDEV_UP: + br_vlan_set_all_vlan_dev_state(p); + break; + } +} diff --git a/net/bridge/br_vlan_tunnel.c b/net/bridge/br_vlan_tunnel.c index 6d2c4eed2dc892b15a99f4741dc8639b25a63e32..75815186366976a95329d7f5a93959b7eb9f9632 100644 --- a/net/bridge/br_vlan_tunnel.c +++ b/net/bridge/br_vlan_tunnel.c @@ -34,7 +34,6 @@ static const struct rhashtable_params br_vlan_tunnel_rht_params = { .key_offset = offsetof(struct net_bridge_vlan, tinfo.tunnel_id), .key_len = sizeof(__be64), .nelem_hint = 3, - .locks_mul = 1, .obj_cmpfn = br_vlan_tunid_cmp, .automatic_shrinking = true, }; diff --git a/net/bridge/netfilter/ebtable_broute.c b/net/bridge/netfilter/ebtable_broute.c index 276b60262981c95a9fccd508e8d8123212d535de..ec2652a459da87664ec6847c2fbad6516e3eafac 100644 --- a/net/bridge/netfilter/ebtable_broute.c +++ b/net/bridge/netfilter/ebtable_broute.c @@ -15,6 +15,8 @@ #include #include +#include "../br_private.h" + /* EBT_ACCEPT means the frame will be bridged * EBT_DROP means the frame will be routed */ @@ -48,30 +50,63 @@ static const struct ebt_table broute_table = { .me = THIS_MODULE, }; -static int ebt_broute(struct sk_buff *skb) +static unsigned int ebt_broute(void *priv, struct sk_buff *skb, + const struct nf_hook_state *s) { + struct net_bridge_port *p = br_port_get_rcu(skb->dev); struct nf_hook_state state; + unsigned char *dest; int ret; + if (!p || p->state != BR_STATE_FORWARDING) + return NF_ACCEPT; + nf_hook_state_init(&state, NF_BR_BROUTING, - NFPROTO_BRIDGE, skb->dev, NULL, NULL, - dev_net(skb->dev), NULL); + NFPROTO_BRIDGE, s->in, NULL, NULL, + s->net, NULL); ret = ebt_do_table(skb, &state, state.net->xt.broute_table); - if (ret == NF_DROP) - return 1; /* route it */ - return 0; /* bridge it */ + + if (ret != NF_DROP) + return ret; + + /* DROP in ebtables -t broute means that the + * skb should be routed, not bridged. + * This is awkward, but can't be changed for compatibility + * reasons. + * + * We map DROP to ACCEPT and set the ->br_netfilter_broute flag. + */ + BR_INPUT_SKB_CB(skb)->br_netfilter_broute = 1; + + /* undo PACKET_HOST mangling done in br_input in case the dst + * address matches the logical bridge but not the port. + */ + dest = eth_hdr(skb)->h_dest; + if (skb->pkt_type == PACKET_HOST && + !ether_addr_equal(skb->dev->dev_addr, dest) && + ether_addr_equal(p->br->dev->dev_addr, dest)) + skb->pkt_type = PACKET_OTHERHOST; + + return NF_ACCEPT; } +static const struct nf_hook_ops ebt_ops_broute = { + .hook = ebt_broute, + .pf = NFPROTO_BRIDGE, + .hooknum = NF_BR_PRE_ROUTING, + .priority = NF_BR_PRI_FIRST, +}; + static int __net_init broute_net_init(struct net *net) { - return ebt_register_table(net, &broute_table, NULL, + return ebt_register_table(net, &broute_table, &ebt_ops_broute, &net->xt.broute_table); } static void __net_exit broute_net_exit(struct net *net) { - ebt_unregister_table(net, net->xt.broute_table, NULL); + ebt_unregister_table(net, net->xt.broute_table, &ebt_ops_broute); } static struct pernet_operations broute_net_ops = { @@ -81,21 +116,11 @@ static struct pernet_operations broute_net_ops = { static int __init ebtable_broute_init(void) { - int ret; - - ret = register_pernet_subsys(&broute_net_ops); - if (ret < 0) - return ret; - /* see br_input.c */ - RCU_INIT_POINTER(br_should_route_hook, - (br_should_route_hook_t *)ebt_broute); - return 0; + return register_pernet_subsys(&broute_net_ops); } static void __exit ebtable_broute_fini(void) { - RCU_INIT_POINTER(br_should_route_hook, NULL); - synchronize_net(); unregister_pernet_subsys(&broute_net_ops); } diff --git a/net/bridge/netfilter/ebtables.c b/net/bridge/netfilter/ebtables.c index 3cad01ac64e4a2a5ebcafa394d04cc4415c7ddab..4e0091311d40227694704db18b42e6911544cfc5 100644 --- a/net/bridge/netfilter/ebtables.c +++ b/net/bridge/netfilter/ebtables.c @@ -1221,10 +1221,6 @@ int ebt_register_table(struct net *net, const struct ebt_table *input_table, mutex_unlock(&ebt_mutex); WRITE_ONCE(*res, table); - - if (!ops) - return 0; - ret = nf_register_net_hooks(net, ops, hweight32(table->valid_hooks)); if (ret) { __ebt_unregister_table(net, table); @@ -1248,8 +1244,7 @@ out: void ebt_unregister_table(struct net *net, struct ebt_table *table, const struct nf_hook_ops *ops) { - if (ops) - nf_unregister_net_hooks(net, ops, hweight32(table->valid_hooks)); + nf_unregister_net_hooks(net, ops, hweight32(table->valid_hooks)); __ebt_unregister_table(net, table); } diff --git a/net/caif/caif_dev.c b/net/caif/caif_dev.c index 711d7156efd8bc94b449d0e8066eedd4fe0d5747..6c6e01963aaccd256c887d9c2aa68107de469f08 100644 --- a/net/caif/caif_dev.c +++ b/net/caif/caif_dev.c @@ -186,15 +186,19 @@ static int transmit(struct cflayer *layer, struct cfpkt *pkt) goto noxoff; if (likely(!netif_queue_stopped(caifd->netdev))) { + struct Qdisc *sch; + /* If we run with a TX queue, check if the queue is too long*/ txq = netdev_get_tx_queue(skb->dev, 0); - qlen = qdisc_qlen(rcu_dereference_bh(txq->qdisc)); - - if (likely(qlen == 0)) + sch = rcu_dereference_bh(txq->qdisc); + if (likely(qdisc_is_empty(sch))) goto noxoff; + /* can check for explicit qdisc len value only !NOLOCK, + * always set flow off otherwise + */ high = (caifd->netdev->tx_queue_len * q_high) / 100; - if (likely(qlen < high)) + if (!(sch->flags & TCQ_F_NOLOCK) && likely(sch->q.qlen < high)) goto noxoff; } diff --git a/net/can/af_can.c b/net/can/af_can.c index 1684ba5b51eb6c2f38a382b07e869d92d7907740..e8fd5dc1780aeb0a5b2f6b888052938c931913a5 100644 --- a/net/can/af_can.c +++ b/net/can/af_can.c @@ -89,13 +89,7 @@ static atomic_t skbcounter = ATOMIC_INIT(0); int can_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) { - struct sock *sk = sock->sk; - switch (cmd) { - - case SIOCGSTAMP: - return sock_get_timestamp(sk, (struct timeval __user *)arg); - default: return -ENOIOCTLCMD; } diff --git a/net/can/bcm.c b/net/can/bcm.c index 79bb8afa9c0c0990729a781acc7220a78ee317af..a34ee52f19eae510a50caeb2a95f71e0b346f2c0 100644 --- a/net/can/bcm.c +++ b/net/can/bcm.c @@ -1689,6 +1689,7 @@ static const struct proto_ops bcm_ops = { .getname = sock_no_getname, .poll = datagram_poll, .ioctl = can_ioctl, /* use can_ioctl() from af_can.c */ + .gettstamp = sock_gettstamp, .listen = sock_no_listen, .shutdown = sock_no_shutdown, .setsockopt = sock_no_setsockopt, diff --git a/net/can/gw.c b/net/can/gw.c index 53859346dc9a92f76daf35e4daf68a15ef1c2c9b..5275ddf580bc7d64139e323eca15e276357bfdd1 100644 --- a/net/can/gw.c +++ b/net/can/gw.c @@ -662,8 +662,8 @@ static int cgw_parse_attr(struct nlmsghdr *nlh, struct cf_mod *mod, /* initialize modification & checksum data space */ memset(mod, 0, sizeof(*mod)); - err = nlmsg_parse(nlh, sizeof(struct rtcanmsg), tb, CGW_MAX, - cgw_policy, NULL); + err = nlmsg_parse_deprecated(nlh, sizeof(struct rtcanmsg), tb, + CGW_MAX, cgw_policy, NULL); if (err < 0) return err; diff --git a/net/can/raw.c b/net/can/raw.c index c70207537488ff3c4f6183a725cf6334d2d042eb..afcbff063a67b2144f44a88856814560facc9ac7 100644 --- a/net/can/raw.c +++ b/net/can/raw.c @@ -846,6 +846,7 @@ static const struct proto_ops raw_ops = { .getname = raw_getname, .poll = datagram_poll, .ioctl = can_ioctl, /* use can_ioctl() from af_can.c */ + .gettstamp = sock_gettstamp, .listen = sock_no_listen, .shutdown = sock_no_shutdown, .setsockopt = raw_setsockopt, diff --git a/net/compat.c b/net/compat.c index eeea5eb71639cbe14a4029369eabd058df47439e..a031bd33309234d2c42aebea62d151d51514380b 100644 --- a/net/compat.c +++ b/net/compat.c @@ -395,63 +395,6 @@ COMPAT_SYSCALL_DEFINE5(setsockopt, int, fd, int, level, int, optname, return __compat_sys_setsockopt(fd, level, optname, optval, optlen); } -int compat_sock_get_timestamp(struct sock *sk, struct timeval __user *userstamp) -{ - struct compat_timeval __user *ctv; - int err; - struct timeval tv; - - if (COMPAT_USE_64BIT_TIME) - return sock_get_timestamp(sk, userstamp); - - ctv = (struct compat_timeval __user *) userstamp; - err = -ENOENT; - sock_enable_timestamp(sk, SOCK_TIMESTAMP); - tv = ktime_to_timeval(sock_read_timestamp(sk)); - - if (tv.tv_sec == -1) - return err; - if (tv.tv_sec == 0) { - ktime_t kt = ktime_get_real(); - sock_write_timestamp(sk, kt); - tv = ktime_to_timeval(kt); - } - err = 0; - if (put_user(tv.tv_sec, &ctv->tv_sec) || - put_user(tv.tv_usec, &ctv->tv_usec)) - err = -EFAULT; - return err; -} -EXPORT_SYMBOL(compat_sock_get_timestamp); - -int compat_sock_get_timestampns(struct sock *sk, struct timespec __user *userstamp) -{ - struct compat_timespec __user *ctv; - int err; - struct timespec ts; - - if (COMPAT_USE_64BIT_TIME) - return sock_get_timestampns (sk, userstamp); - - ctv = (struct compat_timespec __user *) userstamp; - err = -ENOENT; - sock_enable_timestamp(sk, SOCK_TIMESTAMP); - ts = ktime_to_timespec(sock_read_timestamp(sk)); - if (ts.tv_sec == -1) - return err; - if (ts.tv_sec == 0) { - ktime_t kt = ktime_get_real(); - sock_write_timestamp(sk, kt); - ts = ktime_to_timespec(kt); - } - err = 0; - if (put_user(ts.tv_sec, &ctv->tv_sec) || - put_user(ts.tv_nsec, &ctv->tv_nsec)) - err = -EFAULT; - return err; -} -EXPORT_SYMBOL(compat_sock_get_timestampns); - static int __compat_sys_getsockopt(int fd, int level, int optname, char __user *optval, int __user *optlen) diff --git a/net/core/Makefile b/net/core/Makefile index f97d6254e5640f92635f33981e81e614b91a39ec..a104dc8faafcc584a1ceed725b3b25a3f703b75a 100644 --- a/net/core/Makefile +++ b/net/core/Makefile @@ -34,3 +34,4 @@ obj-$(CONFIG_HWBM) += hwbm.o obj-$(CONFIG_NET_DEVLINK) += devlink.o obj-$(CONFIG_GRO_CELLS) += gro_cells.o obj-$(CONFIG_FAILOVER) += failover.o +obj-$(CONFIG_BPF_SYSCALL) += bpf_sk_storage.o diff --git a/net/core/bpf_sk_storage.c b/net/core/bpf_sk_storage.c new file mode 100644 index 0000000000000000000000000000000000000000..cc9597a877707f3de6e7bbfdec3d3c7ae9f56fbc --- /dev/null +++ b/net/core/bpf_sk_storage.c @@ -0,0 +1,804 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2019 Facebook */ +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static atomic_t cache_idx; + +struct bucket { + struct hlist_head list; + raw_spinlock_t lock; +}; + +/* Thp map is not the primary owner of a bpf_sk_storage_elem. + * Instead, the sk->sk_bpf_storage is. + * + * The map (bpf_sk_storage_map) is for two purposes + * 1. Define the size of the "sk local storage". It is + * the map's value_size. + * + * 2. Maintain a list to keep track of all elems such + * that they can be cleaned up during the map destruction. + * + * When a bpf local storage is being looked up for a + * particular sk, the "bpf_map" pointer is actually used + * as the "key" to search in the list of elem in + * sk->sk_bpf_storage. + * + * Hence, consider sk->sk_bpf_storage is the mini-map + * with the "bpf_map" pointer as the searching key. + */ +struct bpf_sk_storage_map { + struct bpf_map map; + /* Lookup elem does not require accessing the map. + * + * Updating/Deleting requires a bucket lock to + * link/unlink the elem from the map. Having + * multiple buckets to improve contention. + */ + struct bucket *buckets; + u32 bucket_log; + u16 elem_size; + u16 cache_idx; +}; + +struct bpf_sk_storage_data { + /* smap is used as the searching key when looking up + * from sk->sk_bpf_storage. + * + * Put it in the same cacheline as the data to minimize + * the number of cachelines access during the cache hit case. + */ + struct bpf_sk_storage_map __rcu *smap; + u8 data[0] __aligned(8); +}; + +/* Linked to bpf_sk_storage and bpf_sk_storage_map */ +struct bpf_sk_storage_elem { + struct hlist_node map_node; /* Linked to bpf_sk_storage_map */ + struct hlist_node snode; /* Linked to bpf_sk_storage */ + struct bpf_sk_storage __rcu *sk_storage; + struct rcu_head rcu; + /* 8 bytes hole */ + /* The data is stored in aother cacheline to minimize + * the number of cachelines access during a cache hit. + */ + struct bpf_sk_storage_data sdata ____cacheline_aligned; +}; + +#define SELEM(_SDATA) container_of((_SDATA), struct bpf_sk_storage_elem, sdata) +#define SDATA(_SELEM) (&(_SELEM)->sdata) +#define BPF_SK_STORAGE_CACHE_SIZE 16 + +struct bpf_sk_storage { + struct bpf_sk_storage_data __rcu *cache[BPF_SK_STORAGE_CACHE_SIZE]; + struct hlist_head list; /* List of bpf_sk_storage_elem */ + struct sock *sk; /* The sk that owns the the above "list" of + * bpf_sk_storage_elem. + */ + struct rcu_head rcu; + raw_spinlock_t lock; /* Protect adding/removing from the "list" */ +}; + +static struct bucket *select_bucket(struct bpf_sk_storage_map *smap, + struct bpf_sk_storage_elem *selem) +{ + return &smap->buckets[hash_ptr(selem, smap->bucket_log)]; +} + +static int omem_charge(struct sock *sk, unsigned int size) +{ + /* same check as in sock_kmalloc() */ + if (size <= sysctl_optmem_max && + atomic_read(&sk->sk_omem_alloc) + size < sysctl_optmem_max) { + atomic_add(size, &sk->sk_omem_alloc); + return 0; + } + + return -ENOMEM; +} + +static bool selem_linked_to_sk(const struct bpf_sk_storage_elem *selem) +{ + return !hlist_unhashed(&selem->snode); +} + +static bool selem_linked_to_map(const struct bpf_sk_storage_elem *selem) +{ + return !hlist_unhashed(&selem->map_node); +} + +static struct bpf_sk_storage_elem *selem_alloc(struct bpf_sk_storage_map *smap, + struct sock *sk, void *value, + bool charge_omem) +{ + struct bpf_sk_storage_elem *selem; + + if (charge_omem && omem_charge(sk, smap->elem_size)) + return NULL; + + selem = kzalloc(smap->elem_size, GFP_ATOMIC | __GFP_NOWARN); + if (selem) { + if (value) + memcpy(SDATA(selem)->data, value, smap->map.value_size); + return selem; + } + + if (charge_omem) + atomic_sub(smap->elem_size, &sk->sk_omem_alloc); + + return NULL; +} + +/* sk_storage->lock must be held and selem->sk_storage == sk_storage. + * The caller must ensure selem->smap is still valid to be + * dereferenced for its smap->elem_size and smap->cache_idx. + */ +static bool __selem_unlink_sk(struct bpf_sk_storage *sk_storage, + struct bpf_sk_storage_elem *selem, + bool uncharge_omem) +{ + struct bpf_sk_storage_map *smap; + bool free_sk_storage; + struct sock *sk; + + smap = rcu_dereference(SDATA(selem)->smap); + sk = sk_storage->sk; + + /* All uncharging on sk->sk_omem_alloc must be done first. + * sk may be freed once the last selem is unlinked from sk_storage. + */ + if (uncharge_omem) + atomic_sub(smap->elem_size, &sk->sk_omem_alloc); + + free_sk_storage = hlist_is_singular_node(&selem->snode, + &sk_storage->list); + if (free_sk_storage) { + atomic_sub(sizeof(struct bpf_sk_storage), &sk->sk_omem_alloc); + sk_storage->sk = NULL; + /* After this RCU_INIT, sk may be freed and cannot be used */ + RCU_INIT_POINTER(sk->sk_bpf_storage, NULL); + + /* sk_storage is not freed now. sk_storage->lock is + * still held and raw_spin_unlock_bh(&sk_storage->lock) + * will be done by the caller. + * + * Although the unlock will be done under + * rcu_read_lock(), it is more intutivie to + * read if kfree_rcu(sk_storage, rcu) is done + * after the raw_spin_unlock_bh(&sk_storage->lock). + * + * Hence, a "bool free_sk_storage" is returned + * to the caller which then calls the kfree_rcu() + * after unlock. + */ + } + hlist_del_init_rcu(&selem->snode); + if (rcu_access_pointer(sk_storage->cache[smap->cache_idx]) == + SDATA(selem)) + RCU_INIT_POINTER(sk_storage->cache[smap->cache_idx], NULL); + + kfree_rcu(selem, rcu); + + return free_sk_storage; +} + +static void selem_unlink_sk(struct bpf_sk_storage_elem *selem) +{ + struct bpf_sk_storage *sk_storage; + bool free_sk_storage = false; + + if (unlikely(!selem_linked_to_sk(selem))) + /* selem has already been unlinked from sk */ + return; + + sk_storage = rcu_dereference(selem->sk_storage); + raw_spin_lock_bh(&sk_storage->lock); + if (likely(selem_linked_to_sk(selem))) + free_sk_storage = __selem_unlink_sk(sk_storage, selem, true); + raw_spin_unlock_bh(&sk_storage->lock); + + if (free_sk_storage) + kfree_rcu(sk_storage, rcu); +} + +/* sk_storage->lock must be held and sk_storage->list cannot be empty */ +static void __selem_link_sk(struct bpf_sk_storage *sk_storage, + struct bpf_sk_storage_elem *selem) +{ + RCU_INIT_POINTER(selem->sk_storage, sk_storage); + hlist_add_head(&selem->snode, &sk_storage->list); +} + +static void selem_unlink_map(struct bpf_sk_storage_elem *selem) +{ + struct bpf_sk_storage_map *smap; + struct bucket *b; + + if (unlikely(!selem_linked_to_map(selem))) + /* selem has already be unlinked from smap */ + return; + + smap = rcu_dereference(SDATA(selem)->smap); + b = select_bucket(smap, selem); + raw_spin_lock_bh(&b->lock); + if (likely(selem_linked_to_map(selem))) + hlist_del_init_rcu(&selem->map_node); + raw_spin_unlock_bh(&b->lock); +} + +static void selem_link_map(struct bpf_sk_storage_map *smap, + struct bpf_sk_storage_elem *selem) +{ + struct bucket *b = select_bucket(smap, selem); + + raw_spin_lock_bh(&b->lock); + RCU_INIT_POINTER(SDATA(selem)->smap, smap); + hlist_add_head_rcu(&selem->map_node, &b->list); + raw_spin_unlock_bh(&b->lock); +} + +static void selem_unlink(struct bpf_sk_storage_elem *selem) +{ + /* Always unlink from map before unlinking from sk_storage + * because selem will be freed after successfully unlinked from + * the sk_storage. + */ + selem_unlink_map(selem); + selem_unlink_sk(selem); +} + +static struct bpf_sk_storage_data * +__sk_storage_lookup(struct bpf_sk_storage *sk_storage, + struct bpf_sk_storage_map *smap, + bool cacheit_lockit) +{ + struct bpf_sk_storage_data *sdata; + struct bpf_sk_storage_elem *selem; + + /* Fast path (cache hit) */ + sdata = rcu_dereference(sk_storage->cache[smap->cache_idx]); + if (sdata && rcu_access_pointer(sdata->smap) == smap) + return sdata; + + /* Slow path (cache miss) */ + hlist_for_each_entry_rcu(selem, &sk_storage->list, snode) + if (rcu_access_pointer(SDATA(selem)->smap) == smap) + break; + + if (!selem) + return NULL; + + sdata = SDATA(selem); + if (cacheit_lockit) { + /* spinlock is needed to avoid racing with the + * parallel delete. Otherwise, publishing an already + * deleted sdata to the cache will become a use-after-free + * problem in the next __sk_storage_lookup(). + */ + raw_spin_lock_bh(&sk_storage->lock); + if (selem_linked_to_sk(selem)) + rcu_assign_pointer(sk_storage->cache[smap->cache_idx], + sdata); + raw_spin_unlock_bh(&sk_storage->lock); + } + + return sdata; +} + +static struct bpf_sk_storage_data * +sk_storage_lookup(struct sock *sk, struct bpf_map *map, bool cacheit_lockit) +{ + struct bpf_sk_storage *sk_storage; + struct bpf_sk_storage_map *smap; + + sk_storage = rcu_dereference(sk->sk_bpf_storage); + if (!sk_storage) + return NULL; + + smap = (struct bpf_sk_storage_map *)map; + return __sk_storage_lookup(sk_storage, smap, cacheit_lockit); +} + +static int check_flags(const struct bpf_sk_storage_data *old_sdata, + u64 map_flags) +{ + if (old_sdata && (map_flags & ~BPF_F_LOCK) == BPF_NOEXIST) + /* elem already exists */ + return -EEXIST; + + if (!old_sdata && (map_flags & ~BPF_F_LOCK) == BPF_EXIST) + /* elem doesn't exist, cannot update it */ + return -ENOENT; + + return 0; +} + +static int sk_storage_alloc(struct sock *sk, + struct bpf_sk_storage_map *smap, + struct bpf_sk_storage_elem *first_selem) +{ + struct bpf_sk_storage *prev_sk_storage, *sk_storage; + int err; + + err = omem_charge(sk, sizeof(*sk_storage)); + if (err) + return err; + + sk_storage = kzalloc(sizeof(*sk_storage), GFP_ATOMIC | __GFP_NOWARN); + if (!sk_storage) { + err = -ENOMEM; + goto uncharge; + } + INIT_HLIST_HEAD(&sk_storage->list); + raw_spin_lock_init(&sk_storage->lock); + sk_storage->sk = sk; + + __selem_link_sk(sk_storage, first_selem); + selem_link_map(smap, first_selem); + /* Publish sk_storage to sk. sk->sk_lock cannot be acquired. + * Hence, atomic ops is used to set sk->sk_bpf_storage + * from NULL to the newly allocated sk_storage ptr. + * + * From now on, the sk->sk_bpf_storage pointer is protected + * by the sk_storage->lock. Hence, when freeing + * the sk->sk_bpf_storage, the sk_storage->lock must + * be held before setting sk->sk_bpf_storage to NULL. + */ + prev_sk_storage = cmpxchg((struct bpf_sk_storage **)&sk->sk_bpf_storage, + NULL, sk_storage); + if (unlikely(prev_sk_storage)) { + selem_unlink_map(first_selem); + err = -EAGAIN; + goto uncharge; + + /* Note that even first_selem was linked to smap's + * bucket->list, first_selem can be freed immediately + * (instead of kfree_rcu) because + * bpf_sk_storage_map_free() does a + * synchronize_rcu() before walking the bucket->list. + * Hence, no one is accessing selem from the + * bucket->list under rcu_read_lock(). + */ + } + + return 0; + +uncharge: + kfree(sk_storage); + atomic_sub(sizeof(*sk_storage), &sk->sk_omem_alloc); + return err; +} + +/* sk cannot be going away because it is linking new elem + * to sk->sk_bpf_storage. (i.e. sk->sk_refcnt cannot be 0). + * Otherwise, it will become a leak (and other memory issues + * during map destruction). + */ +static struct bpf_sk_storage_data *sk_storage_update(struct sock *sk, + struct bpf_map *map, + void *value, + u64 map_flags) +{ + struct bpf_sk_storage_data *old_sdata = NULL; + struct bpf_sk_storage_elem *selem; + struct bpf_sk_storage *sk_storage; + struct bpf_sk_storage_map *smap; + int err; + + /* BPF_EXIST and BPF_NOEXIST cannot be both set */ + if (unlikely((map_flags & ~BPF_F_LOCK) > BPF_EXIST) || + /* BPF_F_LOCK can only be used in a value with spin_lock */ + unlikely((map_flags & BPF_F_LOCK) && !map_value_has_spin_lock(map))) + return ERR_PTR(-EINVAL); + + smap = (struct bpf_sk_storage_map *)map; + sk_storage = rcu_dereference(sk->sk_bpf_storage); + if (!sk_storage || hlist_empty(&sk_storage->list)) { + /* Very first elem for this sk */ + err = check_flags(NULL, map_flags); + if (err) + return ERR_PTR(err); + + selem = selem_alloc(smap, sk, value, true); + if (!selem) + return ERR_PTR(-ENOMEM); + + err = sk_storage_alloc(sk, smap, selem); + if (err) { + kfree(selem); + atomic_sub(smap->elem_size, &sk->sk_omem_alloc); + return ERR_PTR(err); + } + + return SDATA(selem); + } + + if ((map_flags & BPF_F_LOCK) && !(map_flags & BPF_NOEXIST)) { + /* Hoping to find an old_sdata to do inline update + * such that it can avoid taking the sk_storage->lock + * and changing the lists. + */ + old_sdata = __sk_storage_lookup(sk_storage, smap, false); + err = check_flags(old_sdata, map_flags); + if (err) + return ERR_PTR(err); + if (old_sdata && selem_linked_to_sk(SELEM(old_sdata))) { + copy_map_value_locked(map, old_sdata->data, + value, false); + return old_sdata; + } + } + + raw_spin_lock_bh(&sk_storage->lock); + + /* Recheck sk_storage->list under sk_storage->lock */ + if (unlikely(hlist_empty(&sk_storage->list))) { + /* A parallel del is happening and sk_storage is going + * away. It has just been checked before, so very + * unlikely. Return instead of retry to keep things + * simple. + */ + err = -EAGAIN; + goto unlock_err; + } + + old_sdata = __sk_storage_lookup(sk_storage, smap, false); + err = check_flags(old_sdata, map_flags); + if (err) + goto unlock_err; + + if (old_sdata && (map_flags & BPF_F_LOCK)) { + copy_map_value_locked(map, old_sdata->data, value, false); + selem = SELEM(old_sdata); + goto unlock; + } + + /* sk_storage->lock is held. Hence, we are sure + * we can unlink and uncharge the old_sdata successfully + * later. Hence, instead of charging the new selem now + * and then uncharge the old selem later (which may cause + * a potential but unnecessary charge failure), avoid taking + * a charge at all here (the "!old_sdata" check) and the + * old_sdata will not be uncharged later during __selem_unlink_sk(). + */ + selem = selem_alloc(smap, sk, value, !old_sdata); + if (!selem) { + err = -ENOMEM; + goto unlock_err; + } + + /* First, link the new selem to the map */ + selem_link_map(smap, selem); + + /* Second, link (and publish) the new selem to sk_storage */ + __selem_link_sk(sk_storage, selem); + + /* Third, remove old selem, SELEM(old_sdata) */ + if (old_sdata) { + selem_unlink_map(SELEM(old_sdata)); + __selem_unlink_sk(sk_storage, SELEM(old_sdata), false); + } + +unlock: + raw_spin_unlock_bh(&sk_storage->lock); + return SDATA(selem); + +unlock_err: + raw_spin_unlock_bh(&sk_storage->lock); + return ERR_PTR(err); +} + +static int sk_storage_delete(struct sock *sk, struct bpf_map *map) +{ + struct bpf_sk_storage_data *sdata; + + sdata = sk_storage_lookup(sk, map, false); + if (!sdata) + return -ENOENT; + + selem_unlink(SELEM(sdata)); + + return 0; +} + +/* Called by __sk_destruct() */ +void bpf_sk_storage_free(struct sock *sk) +{ + struct bpf_sk_storage_elem *selem; + struct bpf_sk_storage *sk_storage; + bool free_sk_storage = false; + struct hlist_node *n; + + rcu_read_lock(); + sk_storage = rcu_dereference(sk->sk_bpf_storage); + if (!sk_storage) { + rcu_read_unlock(); + return; + } + + /* Netiher the bpf_prog nor the bpf-map's syscall + * could be modifying the sk_storage->list now. + * Thus, no elem can be added-to or deleted-from the + * sk_storage->list by the bpf_prog or by the bpf-map's syscall. + * + * It is racing with bpf_sk_storage_map_free() alone + * when unlinking elem from the sk_storage->list and + * the map's bucket->list. + */ + raw_spin_lock_bh(&sk_storage->lock); + hlist_for_each_entry_safe(selem, n, &sk_storage->list, snode) { + /* Always unlink from map before unlinking from + * sk_storage. + */ + selem_unlink_map(selem); + free_sk_storage = __selem_unlink_sk(sk_storage, selem, true); + } + raw_spin_unlock_bh(&sk_storage->lock); + rcu_read_unlock(); + + if (free_sk_storage) + kfree_rcu(sk_storage, rcu); +} + +static void bpf_sk_storage_map_free(struct bpf_map *map) +{ + struct bpf_sk_storage_elem *selem; + struct bpf_sk_storage_map *smap; + struct bucket *b; + unsigned int i; + + smap = (struct bpf_sk_storage_map *)map; + + synchronize_rcu(); + + /* bpf prog and the userspace can no longer access this map + * now. No new selem (of this map) can be added + * to the sk->sk_bpf_storage or to the map bucket's list. + * + * The elem of this map can be cleaned up here + * or + * by bpf_sk_storage_free() during __sk_destruct(). + */ + for (i = 0; i < (1U << smap->bucket_log); i++) { + b = &smap->buckets[i]; + + rcu_read_lock(); + /* No one is adding to b->list now */ + while ((selem = hlist_entry_safe(rcu_dereference_raw(hlist_first_rcu(&b->list)), + struct bpf_sk_storage_elem, + map_node))) { + selem_unlink(selem); + cond_resched_rcu(); + } + rcu_read_unlock(); + } + + /* bpf_sk_storage_free() may still need to access the map. + * e.g. bpf_sk_storage_free() has unlinked selem from the map + * which then made the above while((selem = ...)) loop + * exited immediately. + * + * However, the bpf_sk_storage_free() still needs to access + * the smap->elem_size to do the uncharging in + * __selem_unlink_sk(). + * + * Hence, wait another rcu grace period for the + * bpf_sk_storage_free() to finish. + */ + synchronize_rcu(); + + kvfree(smap->buckets); + kfree(map); +} + +static int bpf_sk_storage_map_alloc_check(union bpf_attr *attr) +{ + if (attr->map_flags != BPF_F_NO_PREALLOC || attr->max_entries || + attr->key_size != sizeof(int) || !attr->value_size || + /* Enforce BTF for userspace sk dumping */ + !attr->btf_key_type_id || !attr->btf_value_type_id) + return -EINVAL; + + if (!capable(CAP_SYS_ADMIN)) + return -EPERM; + + if (attr->value_size >= KMALLOC_MAX_SIZE - + MAX_BPF_STACK - sizeof(struct bpf_sk_storage_elem) || + /* U16_MAX is much more than enough for sk local storage + * considering a tcp_sock is ~2k. + */ + attr->value_size > U16_MAX - sizeof(struct bpf_sk_storage_elem)) + return -E2BIG; + + return 0; +} + +static struct bpf_map *bpf_sk_storage_map_alloc(union bpf_attr *attr) +{ + struct bpf_sk_storage_map *smap; + unsigned int i; + u32 nbuckets; + u64 cost; + + smap = kzalloc(sizeof(*smap), GFP_USER | __GFP_NOWARN); + if (!smap) + return ERR_PTR(-ENOMEM); + bpf_map_init_from_attr(&smap->map, attr); + + smap->bucket_log = ilog2(roundup_pow_of_two(num_possible_cpus())); + nbuckets = 1U << smap->bucket_log; + smap->buckets = kvcalloc(sizeof(*smap->buckets), nbuckets, + GFP_USER | __GFP_NOWARN); + if (!smap->buckets) { + kfree(smap); + return ERR_PTR(-ENOMEM); + } + cost = sizeof(*smap->buckets) * nbuckets + sizeof(*smap); + + for (i = 0; i < nbuckets; i++) { + INIT_HLIST_HEAD(&smap->buckets[i].list); + raw_spin_lock_init(&smap->buckets[i].lock); + } + + smap->elem_size = sizeof(struct bpf_sk_storage_elem) + attr->value_size; + smap->cache_idx = (unsigned int)atomic_inc_return(&cache_idx) % + BPF_SK_STORAGE_CACHE_SIZE; + smap->map.pages = round_up(cost, PAGE_SIZE) >> PAGE_SHIFT; + + return &smap->map; +} + +static int notsupp_get_next_key(struct bpf_map *map, void *key, + void *next_key) +{ + return -ENOTSUPP; +} + +static int bpf_sk_storage_map_check_btf(const struct bpf_map *map, + const struct btf *btf, + const struct btf_type *key_type, + const struct btf_type *value_type) +{ + u32 int_data; + + if (BTF_INFO_KIND(key_type->info) != BTF_KIND_INT) + return -EINVAL; + + int_data = *(u32 *)(key_type + 1); + if (BTF_INT_BITS(int_data) != 32 || BTF_INT_OFFSET(int_data)) + return -EINVAL; + + return 0; +} + +static void *bpf_fd_sk_storage_lookup_elem(struct bpf_map *map, void *key) +{ + struct bpf_sk_storage_data *sdata; + struct socket *sock; + int fd, err; + + fd = *(int *)key; + sock = sockfd_lookup(fd, &err); + if (sock) { + sdata = sk_storage_lookup(sock->sk, map, true); + sockfd_put(sock); + return sdata ? sdata->data : NULL; + } + + return ERR_PTR(err); +} + +static int bpf_fd_sk_storage_update_elem(struct bpf_map *map, void *key, + void *value, u64 map_flags) +{ + struct bpf_sk_storage_data *sdata; + struct socket *sock; + int fd, err; + + fd = *(int *)key; + sock = sockfd_lookup(fd, &err); + if (sock) { + sdata = sk_storage_update(sock->sk, map, value, map_flags); + sockfd_put(sock); + return PTR_ERR_OR_ZERO(sdata); + } + + return err; +} + +static int bpf_fd_sk_storage_delete_elem(struct bpf_map *map, void *key) +{ + struct socket *sock; + int fd, err; + + fd = *(int *)key; + sock = sockfd_lookup(fd, &err); + if (sock) { + err = sk_storage_delete(sock->sk, map); + sockfd_put(sock); + return err; + } + + return err; +} + +BPF_CALL_4(bpf_sk_storage_get, struct bpf_map *, map, struct sock *, sk, + void *, value, u64, flags) +{ + struct bpf_sk_storage_data *sdata; + + if (flags > BPF_SK_STORAGE_GET_F_CREATE) + return (unsigned long)NULL; + + sdata = sk_storage_lookup(sk, map, true); + if (sdata) + return (unsigned long)sdata->data; + + if (flags == BPF_SK_STORAGE_GET_F_CREATE && + /* Cannot add new elem to a going away sk. + * Otherwise, the new elem may become a leak + * (and also other memory issues during map + * destruction). + */ + refcount_inc_not_zero(&sk->sk_refcnt)) { + sdata = sk_storage_update(sk, map, value, BPF_NOEXIST); + /* sk must be a fullsock (guaranteed by verifier), + * so sock_gen_put() is unnecessary. + */ + sock_put(sk); + return IS_ERR(sdata) ? + (unsigned long)NULL : (unsigned long)sdata->data; + } + + return (unsigned long)NULL; +} + +BPF_CALL_2(bpf_sk_storage_delete, struct bpf_map *, map, struct sock *, sk) +{ + if (refcount_inc_not_zero(&sk->sk_refcnt)) { + int err; + + err = sk_storage_delete(sk, map); + sock_put(sk); + return err; + } + + return -ENOENT; +} + +const struct bpf_map_ops sk_storage_map_ops = { + .map_alloc_check = bpf_sk_storage_map_alloc_check, + .map_alloc = bpf_sk_storage_map_alloc, + .map_free = bpf_sk_storage_map_free, + .map_get_next_key = notsupp_get_next_key, + .map_lookup_elem = bpf_fd_sk_storage_lookup_elem, + .map_update_elem = bpf_fd_sk_storage_update_elem, + .map_delete_elem = bpf_fd_sk_storage_delete_elem, + .map_check_btf = bpf_sk_storage_map_check_btf, +}; + +const struct bpf_func_proto bpf_sk_storage_get_proto = { + .func = bpf_sk_storage_get, + .gpl_only = false, + .ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL, + .arg1_type = ARG_CONST_MAP_PTR, + .arg2_type = ARG_PTR_TO_SOCKET, + .arg3_type = ARG_PTR_TO_MAP_VALUE_OR_NULL, + .arg4_type = ARG_ANYTHING, +}; + +const struct bpf_func_proto bpf_sk_storage_delete_proto = { + .func = bpf_sk_storage_delete, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_CONST_MAP_PTR, + .arg2_type = ARG_PTR_TO_SOCKET, +}; diff --git a/net/core/datagram.c b/net/core/datagram.c index e657289db4ac44b023ce40ca9959185a6a33cb22..45a162ef5e02570189b006492ad8cd65c33c1990 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -61,6 +61,8 @@ #include #include +#include "datagram.h" + /* * Is a socket 'connection oriented' ? */ @@ -165,7 +167,7 @@ struct sk_buff *__skb_try_recv_from_queue(struct sock *sk, unsigned int flags, void (*destructor)(struct sock *sk, struct sk_buff *skb), - int *peeked, int *off, int *err, + int *off, int *err, struct sk_buff **last) { bool peek_at_off = false; @@ -192,7 +194,6 @@ struct sk_buff *__skb_try_recv_from_queue(struct sock *sk, return NULL; } } - *peeked = 1; refcount_inc(&skb->users); } else { __skb_unlink(skb, queue); @@ -210,7 +211,6 @@ struct sk_buff *__skb_try_recv_from_queue(struct sock *sk, * @sk: socket * @flags: MSG\_ flags * @destructor: invoked under the receive lock on successful dequeue - * @peeked: returns non-zero if this packet has been seen before * @off: an offset in bytes to peek skb from. Returns an offset * within an skb where data actually starts * @err: error code returned @@ -244,7 +244,7 @@ struct sk_buff *__skb_try_recv_from_queue(struct sock *sk, struct sk_buff *__skb_try_recv_datagram(struct sock *sk, unsigned int flags, void (*destructor)(struct sock *sk, struct sk_buff *skb), - int *peeked, int *off, int *err, + int *off, int *err, struct sk_buff **last) { struct sk_buff_head *queue = &sk->sk_receive_queue; @@ -258,7 +258,6 @@ struct sk_buff *__skb_try_recv_datagram(struct sock *sk, unsigned int flags, if (error) goto no_packet; - *peeked = 0; do { /* Again only user level code calls this function, so nothing * interrupt level will suddenly eat the receive_queue. @@ -268,7 +267,7 @@ struct sk_buff *__skb_try_recv_datagram(struct sock *sk, unsigned int flags, */ spin_lock_irqsave(&queue->lock, cpu_flags); skb = __skb_try_recv_from_queue(sk, queue, flags, destructor, - peeked, off, &error, last); + off, &error, last); spin_unlock_irqrestore(&queue->lock, cpu_flags); if (error) goto no_packet; @@ -292,7 +291,7 @@ EXPORT_SYMBOL(__skb_try_recv_datagram); struct sk_buff *__skb_recv_datagram(struct sock *sk, unsigned int flags, void (*destructor)(struct sock *sk, struct sk_buff *skb), - int *peeked, int *off, int *err) + int *off, int *err) { struct sk_buff *skb, *last; long timeo; @@ -300,8 +299,8 @@ struct sk_buff *__skb_recv_datagram(struct sock *sk, unsigned int flags, timeo = sock_rcvtimeo(sk, flags & MSG_DONTWAIT); do { - skb = __skb_try_recv_datagram(sk, flags, destructor, peeked, - off, err, &last); + skb = __skb_try_recv_datagram(sk, flags, destructor, off, err, + &last); if (skb) return skb; @@ -317,10 +316,10 @@ EXPORT_SYMBOL(__skb_recv_datagram); struct sk_buff *skb_recv_datagram(struct sock *sk, unsigned int flags, int noblock, int *err) { - int peeked, off = 0; + int off = 0; return __skb_recv_datagram(sk, flags | (noblock ? MSG_DONTWAIT : 0), - NULL, &peeked, &off, err); + NULL, &off, err); } EXPORT_SYMBOL(skb_recv_datagram); @@ -408,10 +407,10 @@ int skb_kill_datagram(struct sock *sk, struct sk_buff *skb, unsigned int flags) } EXPORT_SYMBOL(skb_kill_datagram); -int __skb_datagram_iter(const struct sk_buff *skb, int offset, - struct iov_iter *to, int len, bool fault_short, - size_t (*cb)(const void *, size_t, void *, struct iov_iter *), - void *data) +static int __skb_datagram_iter(const struct sk_buff *skb, int offset, + struct iov_iter *to, int len, bool fault_short, + size_t (*cb)(const void *, size_t, void *, + struct iov_iter *), void *data) { int start = skb_headlen(skb); int i, copy = start - offset, start_off = offset, n; diff --git a/net/core/datagram.h b/net/core/datagram.h new file mode 100644 index 0000000000000000000000000000000000000000..bcfb75bfa3b2dfff26808bba0ba10936b39a68b0 --- /dev/null +++ b/net/core/datagram.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _NET_CORE_DATAGRAM_H_ +#define _NET_CORE_DATAGRAM_H_ + +#include + +struct sock; +struct sk_buff; +struct iov_iter; + +int __zerocopy_sg_from_iter(struct sock *sk, struct sk_buff *skb, + struct iov_iter *from, size_t length); + +#endif /* _NET_CORE_DATAGRAM_H_ */ diff --git a/net/core/dev.c b/net/core/dev.c index f409406254ddf2e204676bb8bdfb95d0cb3a0e71..108ac8137b9be8bc7041b3e49e09c25eeb891d8f 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -131,7 +131,6 @@ #include #include #include -#include #include #include #include @@ -146,6 +145,7 @@ #include #include #include +#include #include "net-sysfs.h" @@ -3482,6 +3482,15 @@ static inline int __dev_xmit_skb(struct sk_buff *skb, struct Qdisc *q, if (unlikely(test_bit(__QDISC_STATE_DEACTIVATED, &q->state))) { __qdisc_drop(skb, &to_free); rc = NET_XMIT_DROP; + } else if ((q->flags & TCQ_F_CAN_BYPASS) && q->empty && + qdisc_run_begin(q)) { + qdisc_bstats_cpu_update(q, skb); + + if (sch_direct_xmit(skb, q, dev, txq, NULL, true)) + __qdisc_run(q); + + qdisc_run_end(q); + rc = NET_XMIT_SUCCESS; } else { rc = q->enqueue(skb, q, &to_free) & NET_XMIT_MASK; qdisc_run(q); @@ -3570,9 +3579,6 @@ static void skb_update_prio(struct sk_buff *skb) #define skb_update_prio(skb) #endif -DEFINE_PER_CPU(int, xmit_recursion); -EXPORT_SYMBOL(xmit_recursion); - /** * dev_loopback_xmit - loop back @skb * @net: network namespace this loopback is happening in @@ -3703,23 +3709,21 @@ get_cpus_map: } u16 dev_pick_tx_zero(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { return 0; } EXPORT_SYMBOL(dev_pick_tx_zero); u16 dev_pick_tx_cpu_id(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { return (u16)raw_smp_processor_id() % dev->real_num_tx_queues; } EXPORT_SYMBOL(dev_pick_tx_cpu_id); -static u16 __netdev_pick_tx(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev) +u16 netdev_pick_tx(struct net_device *dev, struct sk_buff *skb, + struct net_device *sb_dev) { struct sock *sk = skb->sk; int queue_index = sk_tx_queue_get(sk); @@ -3743,10 +3747,11 @@ static u16 __netdev_pick_tx(struct net_device *dev, struct sk_buff *skb, return queue_index; } +EXPORT_SYMBOL(netdev_pick_tx); -struct netdev_queue *netdev_pick_tx(struct net_device *dev, - struct sk_buff *skb, - struct net_device *sb_dev) +struct netdev_queue *netdev_core_pick_tx(struct net_device *dev, + struct sk_buff *skb, + struct net_device *sb_dev) { int queue_index = 0; @@ -3761,10 +3766,9 @@ struct netdev_queue *netdev_pick_tx(struct net_device *dev, const struct net_device_ops *ops = dev->netdev_ops; if (ops->ndo_select_queue) - queue_index = ops->ndo_select_queue(dev, skb, sb_dev, - __netdev_pick_tx); + queue_index = ops->ndo_select_queue(dev, skb, sb_dev); else - queue_index = __netdev_pick_tx(dev, skb, sb_dev); + queue_index = netdev_pick_tx(dev, skb, sb_dev); queue_index = netdev_cap_txqueue(dev, queue_index); } @@ -3838,7 +3842,7 @@ static int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev) else skb_dst_force(skb); - txq = netdev_pick_tx(dev, skb, sb_dev); + txq = netdev_core_pick_tx(dev, skb, sb_dev); q = rcu_dereference_bh(txq->qdisc); trace_net_dev_queue(skb); @@ -3863,8 +3867,7 @@ static int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev) int cpu = smp_processor_id(); /* ok because BHs are off */ if (txq->xmit_lock_owner != cpu) { - if (unlikely(__this_cpu_read(xmit_recursion) > - XMIT_RECURSION_LIMIT)) + if (dev_xmit_recursion()) goto recursion_alert; skb = validate_xmit_skb(skb, dev, &again); @@ -3874,9 +3877,9 @@ static int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev) HARD_TX_LOCK(dev, txq, cpu); if (!netif_xmit_stopped(txq)) { - __this_cpu_inc(xmit_recursion); + dev_xmit_recursion_inc(); skb = dev_hard_start_xmit(skb, dev, txq, &rc); - __this_cpu_dec(xmit_recursion); + dev_xmit_recursion_dec(); if (dev_xmit_complete(rc)) { HARD_TX_UNLOCK(dev, txq); goto out; @@ -3989,9 +3992,9 @@ EXPORT_SYMBOL(rps_sock_flow_table); u32 rps_cpu_mask __read_mostly; EXPORT_SYMBOL(rps_cpu_mask); -struct static_key rps_needed __read_mostly; +struct static_key_false rps_needed __read_mostly; EXPORT_SYMBOL(rps_needed); -struct static_key rfs_needed __read_mostly; +struct static_key_false rfs_needed __read_mostly; EXPORT_SYMBOL(rfs_needed); static struct rps_dev_flow * @@ -4443,7 +4446,7 @@ void generic_xdp_tx(struct sk_buff *skb, struct bpf_prog *xdp_prog) bool free_skb = true; int cpu, rc; - txq = netdev_pick_tx(dev, skb, NULL); + txq = netdev_core_pick_tx(dev, skb, NULL); cpu = smp_processor_id(); HARD_TX_LOCK(dev, txq, cpu); if (!netif_xmit_stopped(txq)) { @@ -4517,7 +4520,7 @@ static int netif_rx_internal(struct sk_buff *skb) } #ifdef CONFIG_RPS - if (static_key_false(&rps_needed)) { + if (static_branch_unlikely(&rps_needed)) { struct rps_dev_flow voidflow, *rflow = &voidflow; int cpu; @@ -4984,7 +4987,8 @@ static int __netif_receive_skb_one_core(struct sk_buff *skb, bool pfmemalloc) ret = __netif_receive_skb_core(skb, pfmemalloc, &pt_prev); if (pt_prev) - ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev); + ret = INDIRECT_CALL_INET(pt_prev->func, ipv6_rcv, ip_rcv, skb, + skb->dev, pt_prev, orig_dev); return ret; } @@ -5030,7 +5034,8 @@ static inline void __netif_receive_skb_list_ptype(struct list_head *head, else list_for_each_entry_safe(skb, next, head, list) { skb_list_del_init(skb); - pt_prev->func(skb, skb->dev, pt_prev, orig_dev); + INDIRECT_CALL_INET(pt_prev->func, ipv6_rcv, ip_rcv, skb, + skb->dev, pt_prev, orig_dev); } } @@ -5188,7 +5193,7 @@ static int netif_receive_skb_internal(struct sk_buff *skb) rcu_read_lock(); #ifdef CONFIG_RPS - if (static_key_false(&rps_needed)) { + if (static_branch_unlikely(&rps_needed)) { struct rps_dev_flow voidflow, *rflow = &voidflow; int cpu = get_rps_cpu(skb->dev, skb, &rflow); @@ -5236,7 +5241,7 @@ static void netif_receive_skb_list_internal(struct list_head *head) rcu_read_lock(); #ifdef CONFIG_RPS - if (static_key_false(&rps_needed)) { + if (static_branch_unlikely(&rps_needed)) { list_for_each_entry_safe(skb, next, head, list) { struct rps_dev_flow voidflow, *rflow = &voidflow; int cpu = get_rps_cpu(skb->dev, skb, &rflow); @@ -7886,10 +7891,14 @@ int dev_get_phys_port_name(struct net_device *dev, char *name, size_t len) { const struct net_device_ops *ops = dev->netdev_ops; + int err; - if (!ops->ndo_get_phys_port_name) - return -EOPNOTSUPP; - return ops->ndo_get_phys_port_name(dev, name, len); + if (ops->ndo_get_phys_port_name) { + err = ops->ndo_get_phys_port_name(dev, name, len); + if (err != -EOPNOTSUPP) + return err; + } + return devlink_compat_phys_port_name_get(dev, name, len); } EXPORT_SYMBOL(dev_get_phys_port_name); @@ -7909,14 +7918,21 @@ int dev_get_port_parent_id(struct net_device *dev, struct netdev_phys_item_id first = { }; struct net_device *lower_dev; struct list_head *iter; - int err = -EOPNOTSUPP; + int err; - if (ops->ndo_get_port_parent_id) - return ops->ndo_get_port_parent_id(dev, ppid); + if (ops->ndo_get_port_parent_id) { + err = ops->ndo_get_port_parent_id(dev, ppid); + if (err != -EOPNOTSUPP) + return err; + } - if (!recurse) + err = devlink_compat_switch_id_get(dev, ppid); + if (!err || err != -EOPNOTSUPP) return err; + if (!recurse) + return -EOPNOTSUPP; + netdev_for_each_lower_dev(dev, lower_dev, iter) { err = dev_get_port_parent_id(lower_dev, ppid, recurse); if (err) diff --git a/net/core/dev_ioctl.c b/net/core/dev_ioctl.c index 31380fd5a4e2ace4736384c3f182381768e3e0d5..5163d900bb4fb1057e273ed667e9457f05b33c93 100644 --- a/net/core/dev_ioctl.c +++ b/net/core/dev_ioctl.c @@ -366,7 +366,8 @@ EXPORT_SYMBOL(dev_load); * dev_ioctl - network device ioctl * @net: the applicable net namespace * @cmd: command to issue - * @arg: pointer to a struct ifreq in user space + * @ifr: pointer to a struct ifreq in user space + * @need_copyout: whether or not copy_to_user() should be called * * Issue ioctl functions to devices. This is normally called by the * user space syscall interfaces but can sometimes be useful for diff --git a/net/core/devlink.c b/net/core/devlink.c index da0a29f30885d1f54b975ecfa4f83b47c68be543..d43bc52b8840d76b77ff44f46c961dea74d5c72c 100644 --- a/net/core/devlink.c +++ b/net/core/devlink.c @@ -19,6 +19,8 @@ #include #include #include +#include +#include #include #include #include @@ -543,12 +545,14 @@ static int devlink_nl_port_fill(struct sk_buff *msg, struct devlink *devlink, goto nla_put_failure; if (nla_put_u32(msg, DEVLINK_ATTR_PORT_INDEX, devlink_port->index)) goto nla_put_failure; + + spin_lock(&devlink_port->type_lock); if (nla_put_u16(msg, DEVLINK_ATTR_PORT_TYPE, devlink_port->type)) - goto nla_put_failure; + goto nla_put_failure_type_locked; if (devlink_port->desired_type != DEVLINK_PORT_TYPE_NOTSET && nla_put_u16(msg, DEVLINK_ATTR_PORT_DESIRED_TYPE, devlink_port->desired_type)) - goto nla_put_failure; + goto nla_put_failure_type_locked; if (devlink_port->type == DEVLINK_PORT_TYPE_ETH) { struct net_device *netdev = devlink_port->type_dev; @@ -557,7 +561,7 @@ static int devlink_nl_port_fill(struct sk_buff *msg, struct devlink *devlink, netdev->ifindex) || nla_put_string(msg, DEVLINK_ATTR_PORT_NETDEV_NAME, netdev->name))) - goto nla_put_failure; + goto nla_put_failure_type_locked; } if (devlink_port->type == DEVLINK_PORT_TYPE_IB) { struct ib_device *ibdev = devlink_port->type_dev; @@ -565,14 +569,17 @@ static int devlink_nl_port_fill(struct sk_buff *msg, struct devlink *devlink, if (ibdev && nla_put_string(msg, DEVLINK_ATTR_PORT_IBDEV_NAME, ibdev->name)) - goto nla_put_failure; + goto nla_put_failure_type_locked; } + spin_unlock(&devlink_port->type_lock); if (devlink_nl_port_attrs_put(msg, devlink_port)) goto nla_put_failure; genlmsg_end(msg, hdr); return 0; +nla_put_failure_type_locked: + spin_unlock(&devlink_port->type_lock); nla_put_failure: genlmsg_cancel(msg, hdr); return -EMSGSIZE; @@ -1041,14 +1048,15 @@ out: static int devlink_sb_pool_set(struct devlink *devlink, unsigned int sb_index, u16 pool_index, u32 size, - enum devlink_sb_threshold_type threshold_type) + enum devlink_sb_threshold_type threshold_type, + struct netlink_ext_ack *extack) { const struct devlink_ops *ops = devlink->ops; if (ops->sb_pool_set) return ops->sb_pool_set(devlink, sb_index, pool_index, - size, threshold_type); + size, threshold_type, extack); return -EOPNOTSUPP; } @@ -1076,7 +1084,8 @@ static int devlink_nl_cmd_sb_pool_set_doit(struct sk_buff *skb, size = nla_get_u32(info->attrs[DEVLINK_ATTR_SB_POOL_SIZE]); return devlink_sb_pool_set(devlink, devlink_sb->index, - pool_index, size, threshold_type); + pool_index, size, threshold_type, + info->extack); } static int devlink_nl_sb_port_pool_fill(struct sk_buff *msg, @@ -1237,14 +1246,15 @@ out: static int devlink_sb_port_pool_set(struct devlink_port *devlink_port, unsigned int sb_index, u16 pool_index, - u32 threshold) + u32 threshold, + struct netlink_ext_ack *extack) { const struct devlink_ops *ops = devlink_port->devlink->ops; if (ops->sb_port_pool_set) return ops->sb_port_pool_set(devlink_port, sb_index, - pool_index, threshold); + pool_index, threshold, extack); return -EOPNOTSUPP; } @@ -1267,7 +1277,7 @@ static int devlink_nl_cmd_sb_port_pool_set_doit(struct sk_buff *skb, threshold = nla_get_u32(info->attrs[DEVLINK_ATTR_SB_THRESHOLD]); return devlink_sb_port_pool_set(devlink_port, devlink_sb->index, - pool_index, threshold); + pool_index, threshold, info->extack); } static int @@ -1466,7 +1476,8 @@ out: static int devlink_sb_tc_pool_bind_set(struct devlink_port *devlink_port, unsigned int sb_index, u16 tc_index, enum devlink_sb_pool_type pool_type, - u16 pool_index, u32 threshold) + u16 pool_index, u32 threshold, + struct netlink_ext_ack *extack) { const struct devlink_ops *ops = devlink_port->devlink->ops; @@ -1474,7 +1485,7 @@ static int devlink_sb_tc_pool_bind_set(struct devlink_port *devlink_port, if (ops->sb_tc_pool_bind_set) return ops->sb_tc_pool_bind_set(devlink_port, sb_index, tc_index, pool_type, - pool_index, threshold); + pool_index, threshold, extack); return -EOPNOTSUPP; } @@ -1509,7 +1520,7 @@ static int devlink_nl_cmd_sb_tc_pool_bind_set_doit(struct sk_buff *skb, threshold = nla_get_u32(info->attrs[DEVLINK_ATTR_SB_THRESHOLD]); return devlink_sb_tc_pool_bind_set(devlink_port, devlink_sb->index, tc_index, pool_type, - pool_index, threshold); + pool_index, threshold, info->extack); } static int devlink_nl_cmd_sb_occ_snapshot_doit(struct sk_buff *skb, @@ -1661,7 +1672,7 @@ int devlink_dpipe_match_put(struct sk_buff *skb, struct devlink_dpipe_field *field = &header->fields[match->field_id]; struct nlattr *match_attr; - match_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_MATCH); + match_attr = nla_nest_start_noflag(skb, DEVLINK_ATTR_DPIPE_MATCH); if (!match_attr) return -EMSGSIZE; @@ -1686,7 +1697,8 @@ static int devlink_dpipe_matches_put(struct devlink_dpipe_table *table, { struct nlattr *matches_attr; - matches_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_TABLE_MATCHES); + matches_attr = nla_nest_start_noflag(skb, + DEVLINK_ATTR_DPIPE_TABLE_MATCHES); if (!matches_attr) return -EMSGSIZE; @@ -1708,7 +1720,7 @@ int devlink_dpipe_action_put(struct sk_buff *skb, struct devlink_dpipe_field *field = &header->fields[action->field_id]; struct nlattr *action_attr; - action_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_ACTION); + action_attr = nla_nest_start_noflag(skb, DEVLINK_ATTR_DPIPE_ACTION); if (!action_attr) return -EMSGSIZE; @@ -1733,7 +1745,8 @@ static int devlink_dpipe_actions_put(struct devlink_dpipe_table *table, { struct nlattr *actions_attr; - actions_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_TABLE_ACTIONS); + actions_attr = nla_nest_start_noflag(skb, + DEVLINK_ATTR_DPIPE_TABLE_ACTIONS); if (!actions_attr) return -EMSGSIZE; @@ -1755,7 +1768,7 @@ static int devlink_dpipe_table_put(struct sk_buff *skb, u64 table_size; table_size = table->table_ops->size_get(table->priv); - table_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_TABLE); + table_attr = nla_nest_start_noflag(skb, DEVLINK_ATTR_DPIPE_TABLE); if (!table_attr) return -EMSGSIZE; @@ -1835,7 +1848,7 @@ start_again: if (devlink_nl_put_handle(skb, devlink)) goto nla_put_failure; - tables_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_TABLES); + tables_attr = nla_nest_start_noflag(skb, DEVLINK_ATTR_DPIPE_TABLES); if (!tables_attr) goto nla_put_failure; @@ -1936,8 +1949,8 @@ static int devlink_dpipe_action_values_put(struct sk_buff *skb, int err; for (i = 0; i < values_count; i++) { - action_attr = nla_nest_start(skb, - DEVLINK_ATTR_DPIPE_ACTION_VALUE); + action_attr = nla_nest_start_noflag(skb, + DEVLINK_ATTR_DPIPE_ACTION_VALUE); if (!action_attr) return -EMSGSIZE; err = devlink_dpipe_action_value_put(skb, &values[i]); @@ -1973,8 +1986,8 @@ static int devlink_dpipe_match_values_put(struct sk_buff *skb, int err; for (i = 0; i < values_count; i++) { - match_attr = nla_nest_start(skb, - DEVLINK_ATTR_DPIPE_MATCH_VALUE); + match_attr = nla_nest_start_noflag(skb, + DEVLINK_ATTR_DPIPE_MATCH_VALUE); if (!match_attr) return -EMSGSIZE; err = devlink_dpipe_match_value_put(skb, &values[i]); @@ -1995,7 +2008,7 @@ static int devlink_dpipe_entry_put(struct sk_buff *skb, struct nlattr *entry_attr, *matches_attr, *actions_attr; int err; - entry_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_ENTRY); + entry_attr = nla_nest_start_noflag(skb, DEVLINK_ATTR_DPIPE_ENTRY); if (!entry_attr) return -EMSGSIZE; @@ -2007,8 +2020,8 @@ static int devlink_dpipe_entry_put(struct sk_buff *skb, entry->counter, DEVLINK_ATTR_PAD)) goto nla_put_failure; - matches_attr = nla_nest_start(skb, - DEVLINK_ATTR_DPIPE_ENTRY_MATCH_VALUES); + matches_attr = nla_nest_start_noflag(skb, + DEVLINK_ATTR_DPIPE_ENTRY_MATCH_VALUES); if (!matches_attr) goto nla_put_failure; @@ -2020,8 +2033,8 @@ static int devlink_dpipe_entry_put(struct sk_buff *skb, } nla_nest_end(skb, matches_attr); - actions_attr = nla_nest_start(skb, - DEVLINK_ATTR_DPIPE_ENTRY_ACTION_VALUES); + actions_attr = nla_nest_start_noflag(skb, + DEVLINK_ATTR_DPIPE_ENTRY_ACTION_VALUES); if (!actions_attr) goto nla_put_failure; @@ -2078,8 +2091,8 @@ int devlink_dpipe_entry_ctx_prepare(struct devlink_dpipe_dump_ctx *dump_ctx) devlink = dump_ctx->info->user_ptr[0]; if (devlink_nl_put_handle(dump_ctx->skb, devlink)) goto nla_put_failure; - dump_ctx->nest = nla_nest_start(dump_ctx->skb, - DEVLINK_ATTR_DPIPE_ENTRIES); + dump_ctx->nest = nla_nest_start_noflag(dump_ctx->skb, + DEVLINK_ATTR_DPIPE_ENTRIES); if (!dump_ctx->nest) goto nla_put_failure; return 0; @@ -2189,7 +2202,8 @@ static int devlink_dpipe_fields_put(struct sk_buff *skb, for (i = 0; i < header->fields_count; i++) { field = &header->fields[i]; - field_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_FIELD); + field_attr = nla_nest_start_noflag(skb, + DEVLINK_ATTR_DPIPE_FIELD); if (!field_attr) return -EMSGSIZE; if (nla_put_string(skb, DEVLINK_ATTR_DPIPE_FIELD_NAME, field->name) || @@ -2212,7 +2226,7 @@ static int devlink_dpipe_header_put(struct sk_buff *skb, struct nlattr *fields_attr, *header_attr; int err; - header_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_HEADER); + header_attr = nla_nest_start_noflag(skb, DEVLINK_ATTR_DPIPE_HEADER); if (!header_attr) return -EMSGSIZE; @@ -2221,7 +2235,8 @@ static int devlink_dpipe_header_put(struct sk_buff *skb, nla_put_u8(skb, DEVLINK_ATTR_DPIPE_HEADER_GLOBAL, header->global)) goto nla_put_failure; - fields_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_HEADER_FIELDS); + fields_attr = nla_nest_start_noflag(skb, + DEVLINK_ATTR_DPIPE_HEADER_FIELDS); if (!fields_attr) goto nla_put_failure; @@ -2268,7 +2283,7 @@ start_again: if (devlink_nl_put_handle(skb, devlink)) goto nla_put_failure; - headers_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_HEADERS); + headers_attr = nla_nest_start_noflag(skb, DEVLINK_ATTR_DPIPE_HEADERS); if (!headers_attr) goto nla_put_failure; @@ -2492,7 +2507,7 @@ static int devlink_resource_put(struct devlink *devlink, struct sk_buff *skb, struct nlattr *child_resource_attr; struct nlattr *resource_attr; - resource_attr = nla_nest_start(skb, DEVLINK_ATTR_RESOURCE); + resource_attr = nla_nest_start_noflag(skb, DEVLINK_ATTR_RESOURCE); if (!resource_attr) return -EMSGSIZE; @@ -2516,7 +2531,8 @@ static int devlink_resource_put(struct devlink *devlink, struct sk_buff *skb, resource->size_valid)) goto nla_put_failure; - child_resource_attr = nla_nest_start(skb, DEVLINK_ATTR_RESOURCE_LIST); + child_resource_attr = nla_nest_start_noflag(skb, + DEVLINK_ATTR_RESOURCE_LIST); if (!child_resource_attr) goto nla_put_failure; @@ -2567,7 +2583,8 @@ start_again: if (devlink_nl_put_handle(skb, devlink)) goto nla_put_failure; - resources_attr = nla_nest_start(skb, DEVLINK_ATTR_RESOURCE_LIST); + resources_attr = nla_nest_start_noflag(skb, + DEVLINK_ATTR_RESOURCE_LIST); if (!resources_attr) goto nla_put_failure; @@ -2821,7 +2838,8 @@ devlink_nl_param_value_fill_one(struct sk_buff *msg, { struct nlattr *param_value_attr; - param_value_attr = nla_nest_start(msg, DEVLINK_ATTR_PARAM_VALUE); + param_value_attr = nla_nest_start_noflag(msg, + DEVLINK_ATTR_PARAM_VALUE); if (!param_value_attr) goto nla_put_failure; @@ -2912,7 +2930,7 @@ static int devlink_nl_param_fill(struct sk_buff *msg, struct devlink *devlink, if (nla_put_u32(msg, DEVLINK_ATTR_PORT_INDEX, port_index)) goto genlmsg_cancel; - param_attr = nla_nest_start(msg, DEVLINK_ATTR_PARAM); + param_attr = nla_nest_start_noflag(msg, DEVLINK_ATTR_PARAM); if (!param_attr) goto genlmsg_cancel; if (nla_put_string(msg, DEVLINK_ATTR_PARAM_NAME, param->name)) @@ -2926,7 +2944,8 @@ static int devlink_nl_param_fill(struct sk_buff *msg, struct devlink *devlink, if (nla_put_u8(msg, DEVLINK_ATTR_PARAM_TYPE, nla_type)) goto param_nest_cancel; - param_values_list = nla_nest_start(msg, DEVLINK_ATTR_PARAM_VALUES_LIST); + param_values_list = nla_nest_start_noflag(msg, + DEVLINK_ATTR_PARAM_VALUES_LIST); if (!param_values_list) goto param_nest_cancel; @@ -3326,7 +3345,7 @@ static int devlink_nl_region_snapshot_id_put(struct sk_buff *msg, struct nlattr *snap_attr; int err; - snap_attr = nla_nest_start(msg, DEVLINK_ATTR_REGION_SNAPSHOT); + snap_attr = nla_nest_start_noflag(msg, DEVLINK_ATTR_REGION_SNAPSHOT); if (!snap_attr) return -EINVAL; @@ -3350,7 +3369,8 @@ static int devlink_nl_region_snapshots_id_put(struct sk_buff *msg, struct nlattr *snapshots_attr; int err; - snapshots_attr = nla_nest_start(msg, DEVLINK_ATTR_REGION_SNAPSHOTS); + snapshots_attr = nla_nest_start_noflag(msg, + DEVLINK_ATTR_REGION_SNAPSHOTS); if (!snapshots_attr) return -EINVAL; @@ -3566,7 +3586,7 @@ static int devlink_nl_cmd_region_read_chunk_fill(struct sk_buff *msg, struct nlattr *chunk_attr; int err; - chunk_attr = nla_nest_start(msg, DEVLINK_ATTR_REGION_CHUNK); + chunk_attr = nla_nest_start_noflag(msg, DEVLINK_ATTR_REGION_CHUNK); if (!chunk_attr) return -EINVAL; @@ -3640,7 +3660,6 @@ static int devlink_nl_cmd_region_read_dumpit(struct sk_buff *skb, struct netlink_callback *cb) { u64 ret_offset, start_offset, end_offset = 0; - const struct genl_ops *ops = cb->data; struct devlink_region *region; struct nlattr *chunks_attr; const char *region_name; @@ -3656,8 +3675,10 @@ static int devlink_nl_cmd_region_read_dumpit(struct sk_buff *skb, if (!attrs) return -ENOMEM; - err = nlmsg_parse(cb->nlh, GENL_HDRLEN + devlink_nl_family.hdrsize, - attrs, DEVLINK_ATTR_MAX, ops->policy, cb->extack); + err = nlmsg_parse_deprecated(cb->nlh, + GENL_HDRLEN + devlink_nl_family.hdrsize, + attrs, DEVLINK_ATTR_MAX, + devlink_nl_family.policy, cb->extack); if (err) goto out_free; @@ -3699,7 +3720,7 @@ static int devlink_nl_cmd_region_read_dumpit(struct sk_buff *skb, if (err) goto nla_put_failure; - chunks_attr = nla_nest_start(skb, DEVLINK_ATTR_REGION_CHUNKS); + chunks_attr = nla_nest_start_noflag(skb, DEVLINK_ATTR_REGION_CHUNKS); if (!chunks_attr) { err = -EMSGSIZE; goto nla_put_failure; @@ -3775,7 +3796,7 @@ static int devlink_info_version_put(struct devlink_info_req *req, int attr, struct nlattr *nest; int err; - nest = nla_nest_start(req->msg, attr); + nest = nla_nest_start_noflag(req->msg, attr); if (!nest) return -EMSGSIZE; @@ -4303,7 +4324,7 @@ devlink_fmsg_prepare_skb(struct devlink_fmsg *fmsg, struct sk_buff *skb, int i = 0; int err; - fmsg_nlattr = nla_nest_start(skb, DEVLINK_ATTR_FMSG); + fmsg_nlattr = nla_nest_start_noflag(skb, DEVLINK_ATTR_FMSG); if (!fmsg_nlattr) return -EMSGSIZE; @@ -4412,6 +4433,7 @@ struct devlink_health_reporter { u64 error_count; u64 recovery_count; u64 last_recovery_ts; + refcount_t refcount; }; void * @@ -4427,6 +4449,7 @@ devlink_health_reporter_find_by_name(struct devlink *devlink, { struct devlink_health_reporter *reporter; + lockdep_assert_held(&devlink->reporters_lock); list_for_each_entry(reporter, &devlink->reporter_list, list) if (!strcmp(reporter->ops->name, reporter_name)) return reporter; @@ -4450,7 +4473,7 @@ devlink_health_reporter_create(struct devlink *devlink, { struct devlink_health_reporter *reporter; - mutex_lock(&devlink->lock); + mutex_lock(&devlink->reporters_lock); if (devlink_health_reporter_find_by_name(devlink, ops->name)) { reporter = ERR_PTR(-EEXIST); goto unlock; @@ -4474,9 +4497,10 @@ devlink_health_reporter_create(struct devlink *devlink, reporter->graceful_period = graceful_period; reporter->auto_recover = auto_recover; mutex_init(&reporter->dump_lock); + refcount_set(&reporter->refcount, 1); list_add_tail(&reporter->list, &devlink->reporter_list); unlock: - mutex_unlock(&devlink->lock); + mutex_unlock(&devlink->reporters_lock); return reporter; } EXPORT_SYMBOL_GPL(devlink_health_reporter_create); @@ -4489,9 +4513,12 @@ EXPORT_SYMBOL_GPL(devlink_health_reporter_create); void devlink_health_reporter_destroy(struct devlink_health_reporter *reporter) { - mutex_lock(&reporter->devlink->lock); + mutex_lock(&reporter->devlink->reporters_lock); list_del(&reporter->list); - mutex_unlock(&reporter->devlink->lock); + mutex_unlock(&reporter->devlink->reporters_lock); + while (refcount_read(&reporter->refcount) > 1) + msleep(100); + mutex_destroy(&reporter->dump_lock); if (reporter->dump_fmsg) devlink_fmsg_free(reporter->dump_fmsg); kfree(reporter); @@ -4627,6 +4654,7 @@ static struct devlink_health_reporter * devlink_health_reporter_get_from_info(struct devlink *devlink, struct genl_info *info) { + struct devlink_health_reporter *reporter; char *reporter_name; if (!info->attrs[DEVLINK_ATTR_HEALTH_REPORTER_NAME]) @@ -4634,7 +4662,18 @@ devlink_health_reporter_get_from_info(struct devlink *devlink, reporter_name = nla_data(info->attrs[DEVLINK_ATTR_HEALTH_REPORTER_NAME]); - return devlink_health_reporter_find_by_name(devlink, reporter_name); + mutex_lock(&devlink->reporters_lock); + reporter = devlink_health_reporter_find_by_name(devlink, reporter_name); + if (reporter) + refcount_inc(&reporter->refcount); + mutex_unlock(&devlink->reporters_lock); + return reporter; +} + +static void +devlink_health_reporter_put(struct devlink_health_reporter *reporter) +{ + refcount_dec(&reporter->refcount); } static int @@ -4654,7 +4693,8 @@ devlink_nl_health_reporter_fill(struct sk_buff *msg, if (devlink_nl_put_handle(msg, devlink)) goto genlmsg_cancel; - reporter_attr = nla_nest_start(msg, DEVLINK_ATTR_HEALTH_REPORTER); + reporter_attr = nla_nest_start_noflag(msg, + DEVLINK_ATTR_HEALTH_REPORTER); if (!reporter_attr) goto genlmsg_cancel; if (nla_put_string(msg, DEVLINK_ATTR_HEALTH_REPORTER_NAME, @@ -4708,8 +4748,10 @@ static int devlink_nl_cmd_health_reporter_get_doit(struct sk_buff *skb, return -EINVAL; msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL); - if (!msg) - return -ENOMEM; + if (!msg) { + err = -ENOMEM; + goto out; + } err = devlink_nl_health_reporter_fill(msg, devlink, reporter, DEVLINK_CMD_HEALTH_REPORTER_GET, @@ -4717,10 +4759,13 @@ static int devlink_nl_cmd_health_reporter_get_doit(struct sk_buff *skb, 0); if (err) { nlmsg_free(msg); - return err; + goto out; } - return genlmsg_reply(msg, info); + err = genlmsg_reply(msg, info); +out: + devlink_health_reporter_put(reporter); + return err; } static int @@ -4737,7 +4782,7 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg, list_for_each_entry(devlink, &devlink_list, list) { if (!net_eq(devlink_net(devlink), sock_net(msg->sk))) continue; - mutex_lock(&devlink->lock); + mutex_lock(&devlink->reporters_lock); list_for_each_entry(reporter, &devlink->reporter_list, list) { if (idx < start) { @@ -4751,12 +4796,12 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg, cb->nlh->nlmsg_seq, NLM_F_MULTI); if (err) { - mutex_unlock(&devlink->lock); + mutex_unlock(&devlink->reporters_lock); goto out; } idx++; } - mutex_unlock(&devlink->lock); + mutex_unlock(&devlink->reporters_lock); } out: mutex_unlock(&devlink_mutex); @@ -4771,6 +4816,7 @@ devlink_nl_cmd_health_reporter_set_doit(struct sk_buff *skb, { struct devlink *devlink = info->user_ptr[0]; struct devlink_health_reporter *reporter; + int err; reporter = devlink_health_reporter_get_from_info(devlink, info); if (!reporter) @@ -4778,8 +4824,10 @@ devlink_nl_cmd_health_reporter_set_doit(struct sk_buff *skb, if (!reporter->ops->recover && (info->attrs[DEVLINK_ATTR_HEALTH_REPORTER_GRACEFUL_PERIOD] || - info->attrs[DEVLINK_ATTR_HEALTH_REPORTER_AUTO_RECOVER])) - return -EOPNOTSUPP; + info->attrs[DEVLINK_ATTR_HEALTH_REPORTER_AUTO_RECOVER])) { + err = -EOPNOTSUPP; + goto out; + } if (info->attrs[DEVLINK_ATTR_HEALTH_REPORTER_GRACEFUL_PERIOD]) reporter->graceful_period = @@ -4789,7 +4837,11 @@ devlink_nl_cmd_health_reporter_set_doit(struct sk_buff *skb, reporter->auto_recover = nla_get_u8(info->attrs[DEVLINK_ATTR_HEALTH_REPORTER_AUTO_RECOVER]); + devlink_health_reporter_put(reporter); return 0; +out: + devlink_health_reporter_put(reporter); + return err; } static int devlink_nl_cmd_health_reporter_recover_doit(struct sk_buff *skb, @@ -4797,12 +4849,16 @@ static int devlink_nl_cmd_health_reporter_recover_doit(struct sk_buff *skb, { struct devlink *devlink = info->user_ptr[0]; struct devlink_health_reporter *reporter; + int err; reporter = devlink_health_reporter_get_from_info(devlink, info); if (!reporter) return -EINVAL; - return devlink_health_reporter_recover(reporter, NULL); + err = devlink_health_reporter_recover(reporter, NULL); + + devlink_health_reporter_put(reporter); + return err; } static int devlink_nl_cmd_health_reporter_diagnose_doit(struct sk_buff *skb, @@ -4817,12 +4873,16 @@ static int devlink_nl_cmd_health_reporter_diagnose_doit(struct sk_buff *skb, if (!reporter) return -EINVAL; - if (!reporter->ops->diagnose) + if (!reporter->ops->diagnose) { + devlink_health_reporter_put(reporter); return -EOPNOTSUPP; + } fmsg = devlink_fmsg_alloc(); - if (!fmsg) + if (!fmsg) { + devlink_health_reporter_put(reporter); return -ENOMEM; + } err = devlink_fmsg_obj_nest_start(fmsg); if (err) @@ -4841,6 +4901,7 @@ static int devlink_nl_cmd_health_reporter_diagnose_doit(struct sk_buff *skb, out: devlink_fmsg_free(fmsg); + devlink_health_reporter_put(reporter); return err; } @@ -4855,8 +4916,10 @@ static int devlink_nl_cmd_health_reporter_dump_get_doit(struct sk_buff *skb, if (!reporter) return -EINVAL; - if (!reporter->ops->dump) + if (!reporter->ops->dump) { + devlink_health_reporter_put(reporter); return -EOPNOTSUPP; + } mutex_lock(&reporter->dump_lock); err = devlink_health_do_dump(reporter, NULL); @@ -4868,6 +4931,7 @@ static int devlink_nl_cmd_health_reporter_dump_get_doit(struct sk_buff *skb, out: mutex_unlock(&reporter->dump_lock); + devlink_health_reporter_put(reporter); return err; } @@ -4882,12 +4946,15 @@ devlink_nl_cmd_health_reporter_dump_clear_doit(struct sk_buff *skb, if (!reporter) return -EINVAL; - if (!reporter->ops->dump) + if (!reporter->ops->dump) { + devlink_health_reporter_put(reporter); return -EOPNOTSUPP; + } mutex_lock(&reporter->dump_lock); devlink_health_dump_clear(reporter); mutex_unlock(&reporter->dump_lock); + devlink_health_reporter_put(reporter); return 0; } @@ -4926,293 +4993,297 @@ static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = { static const struct genl_ops devlink_nl_ops[] = { { .cmd = DEVLINK_CMD_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_get_doit, .dumpit = devlink_nl_cmd_get_dumpit, - .policy = devlink_nl_policy, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, /* can be retrieved by unprivileged users */ }, { .cmd = DEVLINK_CMD_PORT_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_port_get_doit, .dumpit = devlink_nl_cmd_port_get_dumpit, - .policy = devlink_nl_policy, .internal_flags = DEVLINK_NL_FLAG_NEED_PORT, /* can be retrieved by unprivileged users */ }, { .cmd = DEVLINK_CMD_PORT_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_port_set_doit, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_PORT, }, { .cmd = DEVLINK_CMD_PORT_SPLIT, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_port_split_doit, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK | DEVLINK_NL_FLAG_NO_LOCK, }, { .cmd = DEVLINK_CMD_PORT_UNSPLIT, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_port_unsplit_doit, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK | DEVLINK_NL_FLAG_NO_LOCK, }, { .cmd = DEVLINK_CMD_SB_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_sb_get_doit, .dumpit = devlink_nl_cmd_sb_get_dumpit, - .policy = devlink_nl_policy, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK | DEVLINK_NL_FLAG_NEED_SB, /* can be retrieved by unprivileged users */ }, { .cmd = DEVLINK_CMD_SB_POOL_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_sb_pool_get_doit, .dumpit = devlink_nl_cmd_sb_pool_get_dumpit, - .policy = devlink_nl_policy, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK | DEVLINK_NL_FLAG_NEED_SB, /* can be retrieved by unprivileged users */ }, { .cmd = DEVLINK_CMD_SB_POOL_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_sb_pool_set_doit, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK | DEVLINK_NL_FLAG_NEED_SB, }, { .cmd = DEVLINK_CMD_SB_PORT_POOL_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_sb_port_pool_get_doit, .dumpit = devlink_nl_cmd_sb_port_pool_get_dumpit, - .policy = devlink_nl_policy, .internal_flags = DEVLINK_NL_FLAG_NEED_PORT | DEVLINK_NL_FLAG_NEED_SB, /* can be retrieved by unprivileged users */ }, { .cmd = DEVLINK_CMD_SB_PORT_POOL_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_sb_port_pool_set_doit, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_PORT | DEVLINK_NL_FLAG_NEED_SB, }, { .cmd = DEVLINK_CMD_SB_TC_POOL_BIND_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_sb_tc_pool_bind_get_doit, .dumpit = devlink_nl_cmd_sb_tc_pool_bind_get_dumpit, - .policy = devlink_nl_policy, .internal_flags = DEVLINK_NL_FLAG_NEED_PORT | DEVLINK_NL_FLAG_NEED_SB, /* can be retrieved by unprivileged users */ }, { .cmd = DEVLINK_CMD_SB_TC_POOL_BIND_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_sb_tc_pool_bind_set_doit, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_PORT | DEVLINK_NL_FLAG_NEED_SB, }, { .cmd = DEVLINK_CMD_SB_OCC_SNAPSHOT, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_sb_occ_snapshot_doit, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK | DEVLINK_NL_FLAG_NEED_SB, }, { .cmd = DEVLINK_CMD_SB_OCC_MAX_CLEAR, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_sb_occ_max_clear_doit, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK | DEVLINK_NL_FLAG_NEED_SB, }, { .cmd = DEVLINK_CMD_ESWITCH_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_eswitch_get_doit, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, }, { .cmd = DEVLINK_CMD_ESWITCH_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_eswitch_set_doit, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK | DEVLINK_NL_FLAG_NO_LOCK, }, { .cmd = DEVLINK_CMD_DPIPE_TABLE_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_dpipe_table_get, - .policy = devlink_nl_policy, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, /* can be retrieved by unprivileged users */ }, { .cmd = DEVLINK_CMD_DPIPE_ENTRIES_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_dpipe_entries_get, - .policy = devlink_nl_policy, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, /* can be retrieved by unprivileged users */ }, { .cmd = DEVLINK_CMD_DPIPE_HEADERS_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_dpipe_headers_get, - .policy = devlink_nl_policy, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, /* can be retrieved by unprivileged users */ }, { .cmd = DEVLINK_CMD_DPIPE_TABLE_COUNTERS_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_dpipe_table_counters_set, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, }, { .cmd = DEVLINK_CMD_RESOURCE_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_resource_set, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, }, { .cmd = DEVLINK_CMD_RESOURCE_DUMP, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_resource_dump, - .policy = devlink_nl_policy, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, /* can be retrieved by unprivileged users */ }, { .cmd = DEVLINK_CMD_RELOAD, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_reload, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK | DEVLINK_NL_FLAG_NO_LOCK, }, { .cmd = DEVLINK_CMD_PARAM_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_param_get_doit, .dumpit = devlink_nl_cmd_param_get_dumpit, - .policy = devlink_nl_policy, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, /* can be retrieved by unprivileged users */ }, { .cmd = DEVLINK_CMD_PARAM_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_param_set_doit, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, }, { .cmd = DEVLINK_CMD_PORT_PARAM_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_port_param_get_doit, .dumpit = devlink_nl_cmd_port_param_get_dumpit, - .policy = devlink_nl_policy, .internal_flags = DEVLINK_NL_FLAG_NEED_PORT, /* can be retrieved by unprivileged users */ }, { .cmd = DEVLINK_CMD_PORT_PARAM_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_port_param_set_doit, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_PORT, }, { .cmd = DEVLINK_CMD_REGION_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_region_get_doit, .dumpit = devlink_nl_cmd_region_get_dumpit, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, }, { .cmd = DEVLINK_CMD_REGION_DEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_region_del, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, }, { .cmd = DEVLINK_CMD_REGION_READ, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .dumpit = devlink_nl_cmd_region_read_dumpit, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, }, { .cmd = DEVLINK_CMD_INFO_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_info_get_doit, .dumpit = devlink_nl_cmd_info_get_dumpit, - .policy = devlink_nl_policy, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, /* can be retrieved by unprivileged users */ }, { .cmd = DEVLINK_CMD_HEALTH_REPORTER_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_health_reporter_get_doit, .dumpit = devlink_nl_cmd_health_reporter_get_dumpit, - .policy = devlink_nl_policy, - .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, + .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK | + DEVLINK_NL_FLAG_NO_LOCK, /* can be retrieved by unprivileged users */ }, { .cmd = DEVLINK_CMD_HEALTH_REPORTER_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_health_reporter_set_doit, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, - .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, + .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK | + DEVLINK_NL_FLAG_NO_LOCK, }, { .cmd = DEVLINK_CMD_HEALTH_REPORTER_RECOVER, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_health_reporter_recover_doit, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, - .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, + .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK | + DEVLINK_NL_FLAG_NO_LOCK, }, { .cmd = DEVLINK_CMD_HEALTH_REPORTER_DIAGNOSE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_health_reporter_diagnose_doit, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, - .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, + .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK | + DEVLINK_NL_FLAG_NO_LOCK, }, { .cmd = DEVLINK_CMD_HEALTH_REPORTER_DUMP_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_health_reporter_dump_get_doit, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK | DEVLINK_NL_FLAG_NO_LOCK, }, { .cmd = DEVLINK_CMD_HEALTH_REPORTER_DUMP_CLEAR, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_health_reporter_dump_clear_doit, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK | DEVLINK_NL_FLAG_NO_LOCK, }, { .cmd = DEVLINK_CMD_FLASH_UPDATE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = devlink_nl_cmd_flash_update, - .policy = devlink_nl_policy, .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, }, @@ -5222,6 +5293,7 @@ static struct genl_family devlink_nl_family __ro_after_init = { .name = DEVLINK_GENL_NAME, .version = DEVLINK_GENL_VERSION, .maxattr = DEVLINK_ATTR_MAX, + .policy = devlink_nl_policy, .netnsok = true, .pre_doit = devlink_nl_pre_doit, .post_doit = devlink_nl_post_doit, @@ -5261,6 +5333,7 @@ struct devlink *devlink_alloc(const struct devlink_ops *ops, size_t priv_size) INIT_LIST_HEAD(&devlink->region_list); INIT_LIST_HEAD(&devlink->reporter_list); mutex_init(&devlink->lock); + mutex_init(&devlink->reporters_lock); return devlink; } EXPORT_SYMBOL_GPL(devlink_alloc); @@ -5303,6 +5376,8 @@ EXPORT_SYMBOL_GPL(devlink_unregister); */ void devlink_free(struct devlink *devlink) { + mutex_destroy(&devlink->reporters_lock); + mutex_destroy(&devlink->lock); WARN_ON(!list_empty(&devlink->reporter_list)); WARN_ON(!list_empty(&devlink->region_list)); WARN_ON(!list_empty(&devlink->param_list)); @@ -5340,6 +5415,7 @@ int devlink_port_register(struct devlink *devlink, devlink_port->devlink = devlink; devlink_port->index = port_index; devlink_port->registered = true; + spin_lock_init(&devlink_port->type_lock); list_add_tail(&devlink_port->list, &devlink->port_list); INIT_LIST_HEAD(&devlink_port->param_list); mutex_unlock(&devlink->lock); @@ -5368,8 +5444,12 @@ static void __devlink_port_type_set(struct devlink_port *devlink_port, enum devlink_port_type type, void *type_dev) { + if (WARN_ON(!devlink_port->registered)) + return; + spin_lock(&devlink_port->type_lock); devlink_port->type = type; devlink_port->type_dev = type_dev; + spin_unlock(&devlink_port->type_lock); devlink_port_notify(devlink_port, DEVLINK_CMD_PORT_NEW); } @@ -5382,8 +5462,39 @@ static void __devlink_port_type_set(struct devlink_port *devlink_port, void devlink_port_type_eth_set(struct devlink_port *devlink_port, struct net_device *netdev) { - return __devlink_port_type_set(devlink_port, - DEVLINK_PORT_TYPE_ETH, netdev); + const struct net_device_ops *ops = netdev->netdev_ops; + + /* If driver registers devlink port, it should set devlink port + * attributes accordingly so the compat functions are called + * and the original ops are not used. + */ + if (ops->ndo_get_phys_port_name) { + /* Some drivers use the same set of ndos for netdevs + * that have devlink_port registered and also for + * those who don't. Make sure that ndo_get_phys_port_name + * returns -EOPNOTSUPP here in case it is defined. + * Warn if not. + */ + char name[IFNAMSIZ]; + int err; + + err = ops->ndo_get_phys_port_name(netdev, name, sizeof(name)); + WARN_ON(err != -EOPNOTSUPP); + } + if (ops->ndo_get_port_parent_id) { + /* Some drivers use the same set of ndos for netdevs + * that have devlink_port registered and also for + * those who don't. Make sure that ndo_get_port_parent_id + * returns -EOPNOTSUPP here in case it is defined. + * Warn if not. + */ + struct netdev_phys_item_id ppid; + int err; + + err = ops->ndo_get_port_parent_id(netdev, &ppid); + WARN_ON(err != -EOPNOTSUPP); + } + __devlink_port_type_set(devlink_port, DEVLINK_PORT_TYPE_ETH, netdev); } EXPORT_SYMBOL_GPL(devlink_port_type_eth_set); @@ -5396,8 +5507,7 @@ EXPORT_SYMBOL_GPL(devlink_port_type_eth_set); void devlink_port_type_ib_set(struct devlink_port *devlink_port, struct ib_device *ibdev) { - return __devlink_port_type_set(devlink_port, - DEVLINK_PORT_TYPE_IB, ibdev); + __devlink_port_type_set(devlink_port, DEVLINK_PORT_TYPE_IB, ibdev); } EXPORT_SYMBOL_GPL(devlink_port_type_ib_set); @@ -5408,8 +5518,7 @@ EXPORT_SYMBOL_GPL(devlink_port_type_ib_set); */ void devlink_port_type_clear(struct devlink_port *devlink_port) { - return __devlink_port_type_set(devlink_port, - DEVLINK_PORT_TYPE_NOTSET, NULL); + __devlink_port_type_set(devlink_port, DEVLINK_PORT_TYPE_NOTSET, NULL); } EXPORT_SYMBOL_GPL(devlink_port_type_clear); @@ -5423,25 +5532,40 @@ EXPORT_SYMBOL_GPL(devlink_port_type_clear); * @split: indicates if this is split port * @split_subport_number: if the port is split, this is the number * of subport. + * @switch_id: if the port is part of switch, this is buffer with ID, + * otwerwise this is NULL + * @switch_id_len: length of the switch_id buffer */ void devlink_port_attrs_set(struct devlink_port *devlink_port, enum devlink_port_flavour flavour, u32 port_number, bool split, - u32 split_subport_number) + u32 split_subport_number, + const unsigned char *switch_id, + unsigned char switch_id_len) { struct devlink_port_attrs *attrs = &devlink_port->attrs; + if (WARN_ON(devlink_port->registered)) + return; attrs->set = true; attrs->flavour = flavour; attrs->port_number = port_number; attrs->split = split; attrs->split_subport_number = split_subport_number; - devlink_port_notify(devlink_port, DEVLINK_CMD_PORT_NEW); + if (switch_id) { + attrs->switch_port = true; + if (WARN_ON(switch_id_len > MAX_PHYS_ITEM_ID_LEN)) + switch_id_len = MAX_PHYS_ITEM_ID_LEN; + memcpy(attrs->switch_id.id, switch_id, switch_id_len); + attrs->switch_id.id_len = switch_id_len; + } else { + attrs->switch_port = false; + } } EXPORT_SYMBOL_GPL(devlink_port_attrs_set); -int devlink_port_get_phys_port_name(struct devlink_port *devlink_port, - char *name, size_t len) +static int __devlink_port_phys_port_name_get(struct devlink_port *devlink_port, + char *name, size_t len) { struct devlink_port_attrs *attrs = &devlink_port->attrs; int n = 0; @@ -5471,7 +5595,6 @@ int devlink_port_get_phys_port_name(struct devlink_port *devlink_port, return 0; } -EXPORT_SYMBOL_GPL(devlink_port_get_phys_port_name); int devlink_sb_register(struct devlink *devlink, unsigned int sb_index, u32 size, u16 ingress_pools_count, @@ -6447,17 +6570,15 @@ void devlink_compat_running_version(struct net_device *dev, dev_hold(dev); rtnl_unlock(); - mutex_lock(&devlink_mutex); devlink = netdev_to_devlink(dev); if (!devlink || !devlink->ops->info_get) - goto unlock_list; + goto out; mutex_lock(&devlink->lock); __devlink_compat_running_version(devlink, buf, len); mutex_unlock(&devlink->lock); -unlock_list: - mutex_unlock(&devlink_mutex); +out: rtnl_lock(); dev_put(dev); } @@ -6465,28 +6586,65 @@ unlock_list: int devlink_compat_flash_update(struct net_device *dev, const char *file_name) { struct devlink *devlink; - int ret = -EOPNOTSUPP; + int ret; dev_hold(dev); rtnl_unlock(); - mutex_lock(&devlink_mutex); devlink = netdev_to_devlink(dev); - if (!devlink || !devlink->ops->flash_update) - goto unlock_list; + if (!devlink || !devlink->ops->flash_update) { + ret = -EOPNOTSUPP; + goto out; + } mutex_lock(&devlink->lock); ret = devlink->ops->flash_update(devlink, file_name, NULL, NULL); mutex_unlock(&devlink->lock); -unlock_list: - mutex_unlock(&devlink_mutex); +out: rtnl_lock(); dev_put(dev); return ret; } +int devlink_compat_phys_port_name_get(struct net_device *dev, + char *name, size_t len) +{ + struct devlink_port *devlink_port; + + /* RTNL mutex is held here which ensures that devlink_port + * instance cannot disappear in the middle. No need to take + * any devlink lock as only permanent values are accessed. + */ + ASSERT_RTNL(); + + devlink_port = netdev_to_devlink_port(dev); + if (!devlink_port) + return -EOPNOTSUPP; + + return __devlink_port_phys_port_name_get(devlink_port, name, len); +} + +int devlink_compat_switch_id_get(struct net_device *dev, + struct netdev_phys_item_id *ppid) +{ + struct devlink_port *devlink_port; + + /* RTNL mutex is held here which ensures that devlink_port + * instance cannot disappear in the middle. No need to take + * any devlink lock as only permanent values are accessed. + */ + ASSERT_RTNL(); + devlink_port = netdev_to_devlink_port(dev); + if (!devlink_port || !devlink_port->attrs.switch_port) + return -EOPNOTSUPP; + + memcpy(ppid, &devlink_port->attrs.switch_id, sizeof(*ppid)); + + return 0; +} + static int __init devlink_init(void) { return genl_register_family(&devlink_nl_family); diff --git a/net/core/drop_monitor.c b/net/core/drop_monitor.c index c7785efeea577594b8e5ed0a74fb7b8b924ebae2..d4ce0542acfa9b67094f9f5e1d46bbe97cc224be 100644 --- a/net/core/drop_monitor.c +++ b/net/core/drop_monitor.c @@ -355,14 +355,17 @@ out: static const struct genl_ops dropmon_ops[] = { { .cmd = NET_DM_CMD_CONFIG, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = net_dm_cmd_config, }, { .cmd = NET_DM_CMD_START, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = net_dm_cmd_trace, }, { .cmd = NET_DM_CMD_STOP, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = net_dm_cmd_trace, }, }; diff --git a/net/core/dst.c b/net/core/dst.c index a263309df115ee5157f3cd6c7bec88c8ce02fe09..1f13d90cd0e40628915c4e1e3b793c4ea1c5b448 100644 --- a/net/core/dst.c +++ b/net/core/dst.c @@ -26,23 +26,6 @@ #include #include -/* - * Theory of operations: - * 1) We use a list, protected by a spinlock, to add - * new entries from both BH and non-BH context. - * 2) In order to keep spinlock held for a small delay, - * we use a second list where are stored long lived - * entries, that are handled by the garbage collect thread - * fired by a workqueue. - * 3) This list is guarded by a mutex, - * so that the gc_task and dst_dev_event() can be synchronized. - */ - -/* - * We want to keep lock & list close together - * to dirty as few cache lines as possible in __dst_free(). - * As this is not a very strong hint, we dont force an alignment on SMP. - */ int dst_discard_out(struct net *net, struct sock *sk, struct sk_buff *skb) { kfree_skb(skb); diff --git a/net/core/ethtool.c b/net/core/ethtool.c index 36ed619faf3641ae4d40edc7385b003de7500764..4a593853cbf283832d57ff085690d9110a1eee1e 100644 --- a/net/core/ethtool.c +++ b/net/core/ethtool.c @@ -136,6 +136,7 @@ static const char phy_tunable_strings[__ETHTOOL_PHY_TUNABLE_COUNT][ETH_GSTRING_LEN] = { [ETHTOOL_ID_UNSPEC] = "Unspec", [ETHTOOL_PHY_DOWNSHIFT] = "phy-downshift", + [ETHTOOL_PHY_FAST_LINK_DOWN] = "phy-fast-link-down", }; static int ethtool_get_features(struct net_device *dev, void __user *useraddr) @@ -2446,6 +2447,7 @@ static int ethtool_phy_tunable_valid(const struct ethtool_tunable *tuna) { switch (tuna->id) { case ETHTOOL_PHY_DOWNSHIFT: + case ETHTOOL_PHY_FAST_LINK_DOWN: if (tuna->len != sizeof(u8) || tuna->type_id != ETHTOOL_TUNABLE_U8) return -EINVAL; diff --git a/net/core/fib_rules.c b/net/core/fib_rules.c index ffbb827723a236a6b187a10a33886561727ae0be..18f8dd8329ede7cd78a9abe9c49f1b8c0e881a6b 100644 --- a/net/core/fib_rules.c +++ b/net/core/fib_rules.c @@ -746,7 +746,8 @@ int fib_nl_newrule(struct sk_buff *skb, struct nlmsghdr *nlh, goto errout; } - err = nlmsg_parse(nlh, sizeof(*frh), tb, FRA_MAX, ops->policy, extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*frh), tb, FRA_MAX, + ops->policy, extack); if (err < 0) { NL_SET_ERR_MSG(extack, "Error parsing msg"); goto errout; @@ -853,7 +854,8 @@ int fib_nl_delrule(struct sk_buff *skb, struct nlmsghdr *nlh, goto errout; } - err = nlmsg_parse(nlh, sizeof(*frh), tb, FRA_MAX, ops->policy, extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*frh), tb, FRA_MAX, + ops->policy, extack); if (err < 0) { NL_SET_ERR_MSG(extack, "Error parsing msg"); goto errout; diff --git a/net/core/filter.c b/net/core/filter.c index 27e61ffd903931c45f0a3f2f6e436937058dfb39..55bfc941d17ad40fc27ee1330fd9b2d2d8527da8 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -74,6 +74,8 @@ #include #include #include +#include +#include /** * sk_filter_trim_cap - run a packet through a socket filter @@ -1729,6 +1731,40 @@ static const struct bpf_func_proto bpf_skb_load_bytes_proto = { .arg4_type = ARG_CONST_SIZE, }; +BPF_CALL_4(bpf_flow_dissector_load_bytes, + const struct bpf_flow_dissector *, ctx, u32, offset, + void *, to, u32, len) +{ + void *ptr; + + if (unlikely(offset > 0xffff)) + goto err_clear; + + if (unlikely(!ctx->skb)) + goto err_clear; + + ptr = skb_header_pointer(ctx->skb, offset, len, to); + if (unlikely(!ptr)) + goto err_clear; + if (ptr != to) + memcpy(to, ptr, len); + + return 0; +err_clear: + memset(to, 0, len); + return -EFAULT; +} + +static const struct bpf_func_proto bpf_flow_dissector_load_bytes_proto = { + .func = bpf_flow_dissector_load_bytes, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_PTR_TO_CTX, + .arg2_type = ARG_ANYTHING, + .arg3_type = ARG_PTR_TO_UNINIT_MEM, + .arg4_type = ARG_CONST_SIZE, +}; + BPF_CALL_5(bpf_skb_load_bytes_relative, const struct sk_buff *, skb, u32, offset, void *, to, u32, len, u32, start_header) { @@ -2015,7 +2051,7 @@ static inline int __bpf_tx_skb(struct net_device *dev, struct sk_buff *skb) { int ret; - if (unlikely(__this_cpu_read(xmit_recursion) > XMIT_RECURSION_LIMIT)) { + if (dev_xmit_recursion()) { net_crit_ratelimited("bpf: recursion limit reached on datapath, buggy bpf program?\n"); kfree_skb(skb); return -ENETDOWN; @@ -2023,9 +2059,9 @@ static inline int __bpf_tx_skb(struct net_device *dev, struct sk_buff *skb) skb->dev = dev; - __this_cpu_inc(xmit_recursion); + dev_xmit_recursion_inc(); ret = dev_queue_xmit(skb); - __this_cpu_dec(xmit_recursion); + dev_xmit_recursion_dec(); return ret; } @@ -2963,42 +2999,128 @@ static u32 bpf_skb_net_base_len(const struct sk_buff *skb) } } -static int bpf_skb_net_grow(struct sk_buff *skb, u32 len_diff) +#define BPF_F_ADJ_ROOM_ENCAP_L3_MASK (BPF_F_ADJ_ROOM_ENCAP_L3_IPV4 | \ + BPF_F_ADJ_ROOM_ENCAP_L3_IPV6) + +#define BPF_F_ADJ_ROOM_MASK (BPF_F_ADJ_ROOM_FIXED_GSO | \ + BPF_F_ADJ_ROOM_ENCAP_L3_MASK | \ + BPF_F_ADJ_ROOM_ENCAP_L4_GRE | \ + BPF_F_ADJ_ROOM_ENCAP_L4_UDP | \ + BPF_F_ADJ_ROOM_ENCAP_L2( \ + BPF_ADJ_ROOM_ENCAP_L2_MASK)) + +static int bpf_skb_net_grow(struct sk_buff *skb, u32 off, u32 len_diff, + u64 flags) { - u32 off = skb_mac_header_len(skb) + bpf_skb_net_base_len(skb); + u8 inner_mac_len = flags >> BPF_ADJ_ROOM_ENCAP_L2_SHIFT; + bool encap = flags & BPF_F_ADJ_ROOM_ENCAP_L3_MASK; + u16 mac_len = 0, inner_net = 0, inner_trans = 0; + unsigned int gso_type = SKB_GSO_DODGY; int ret; - if (skb_is_gso(skb) && !skb_is_gso_tcp(skb)) - return -ENOTSUPP; + if (skb_is_gso(skb) && !skb_is_gso_tcp(skb)) { + /* udp gso_size delineates datagrams, only allow if fixed */ + if (!(skb_shinfo(skb)->gso_type & SKB_GSO_UDP_L4) || + !(flags & BPF_F_ADJ_ROOM_FIXED_GSO)) + return -ENOTSUPP; + } - ret = skb_cow(skb, len_diff); + ret = skb_cow_head(skb, len_diff); if (unlikely(ret < 0)) return ret; + if (encap) { + if (skb->protocol != htons(ETH_P_IP) && + skb->protocol != htons(ETH_P_IPV6)) + return -ENOTSUPP; + + if (flags & BPF_F_ADJ_ROOM_ENCAP_L3_IPV4 && + flags & BPF_F_ADJ_ROOM_ENCAP_L3_IPV6) + return -EINVAL; + + if (flags & BPF_F_ADJ_ROOM_ENCAP_L4_GRE && + flags & BPF_F_ADJ_ROOM_ENCAP_L4_UDP) + return -EINVAL; + + if (skb->encapsulation) + return -EALREADY; + + mac_len = skb->network_header - skb->mac_header; + inner_net = skb->network_header; + if (inner_mac_len > len_diff) + return -EINVAL; + inner_trans = skb->transport_header; + } + ret = bpf_skb_net_hdr_push(skb, off, len_diff); if (unlikely(ret < 0)) return ret; + if (encap) { + skb->inner_mac_header = inner_net - inner_mac_len; + skb->inner_network_header = inner_net; + skb->inner_transport_header = inner_trans; + skb_set_inner_protocol(skb, skb->protocol); + + skb->encapsulation = 1; + skb_set_network_header(skb, mac_len); + + if (flags & BPF_F_ADJ_ROOM_ENCAP_L4_UDP) + gso_type |= SKB_GSO_UDP_TUNNEL; + else if (flags & BPF_F_ADJ_ROOM_ENCAP_L4_GRE) + gso_type |= SKB_GSO_GRE; + else if (flags & BPF_F_ADJ_ROOM_ENCAP_L3_IPV6) + gso_type |= SKB_GSO_IPXIP6; + else if (flags & BPF_F_ADJ_ROOM_ENCAP_L3_IPV4) + gso_type |= SKB_GSO_IPXIP4; + + if (flags & BPF_F_ADJ_ROOM_ENCAP_L4_GRE || + flags & BPF_F_ADJ_ROOM_ENCAP_L4_UDP) { + int nh_len = flags & BPF_F_ADJ_ROOM_ENCAP_L3_IPV6 ? + sizeof(struct ipv6hdr) : + sizeof(struct iphdr); + + skb_set_transport_header(skb, mac_len + nh_len); + } + + /* Match skb->protocol to new outer l3 protocol */ + if (skb->protocol == htons(ETH_P_IP) && + flags & BPF_F_ADJ_ROOM_ENCAP_L3_IPV6) + skb->protocol = htons(ETH_P_IPV6); + else if (skb->protocol == htons(ETH_P_IPV6) && + flags & BPF_F_ADJ_ROOM_ENCAP_L3_IPV4) + skb->protocol = htons(ETH_P_IP); + } + if (skb_is_gso(skb)) { struct skb_shared_info *shinfo = skb_shinfo(skb); /* Due to header grow, MSS needs to be downgraded. */ - skb_decrease_gso_size(shinfo, len_diff); + if (!(flags & BPF_F_ADJ_ROOM_FIXED_GSO)) + skb_decrease_gso_size(shinfo, len_diff); + /* Header must be checked, and gso_segs recomputed. */ - shinfo->gso_type |= SKB_GSO_DODGY; + shinfo->gso_type |= gso_type; shinfo->gso_segs = 0; } return 0; } -static int bpf_skb_net_shrink(struct sk_buff *skb, u32 len_diff) +static int bpf_skb_net_shrink(struct sk_buff *skb, u32 off, u32 len_diff, + u64 flags) { - u32 off = skb_mac_header_len(skb) + bpf_skb_net_base_len(skb); int ret; - if (skb_is_gso(skb) && !skb_is_gso_tcp(skb)) - return -ENOTSUPP; + if (flags & ~BPF_F_ADJ_ROOM_FIXED_GSO) + return -EINVAL; + + if (skb_is_gso(skb) && !skb_is_gso_tcp(skb)) { + /* udp gso_size delineates datagrams, only allow if fixed */ + if (!(skb_shinfo(skb)->gso_type & SKB_GSO_UDP_L4) || + !(flags & BPF_F_ADJ_ROOM_FIXED_GSO)) + return -ENOTSUPP; + } ret = skb_unclone(skb, GFP_ATOMIC); if (unlikely(ret < 0)) @@ -3012,7 +3134,9 @@ static int bpf_skb_net_shrink(struct sk_buff *skb, u32 len_diff) struct skb_shared_info *shinfo = skb_shinfo(skb); /* Due to header shrink, MSS can be upgraded. */ - skb_increase_gso_size(shinfo, len_diff); + if (!(flags & BPF_F_ADJ_ROOM_FIXED_GSO)) + skb_increase_gso_size(shinfo, len_diff); + /* Header must be checked, and gso_segs recomputed. */ shinfo->gso_type |= SKB_GSO_DODGY; shinfo->gso_segs = 0; @@ -3027,49 +3151,50 @@ static u32 __bpf_skb_max_len(const struct sk_buff *skb) SKB_MAX_ALLOC; } -static int bpf_skb_adjust_net(struct sk_buff *skb, s32 len_diff) +BPF_CALL_4(bpf_skb_adjust_room, struct sk_buff *, skb, s32, len_diff, + u32, mode, u64, flags) { - bool trans_same = skb->transport_header == skb->network_header; u32 len_cur, len_diff_abs = abs(len_diff); u32 len_min = bpf_skb_net_base_len(skb); u32 len_max = __bpf_skb_max_len(skb); __be16 proto = skb->protocol; bool shrink = len_diff < 0; + u32 off; int ret; + if (unlikely(flags & ~BPF_F_ADJ_ROOM_MASK)) + return -EINVAL; if (unlikely(len_diff_abs > 0xfffU)) return -EFAULT; if (unlikely(proto != htons(ETH_P_IP) && proto != htons(ETH_P_IPV6))) return -ENOTSUPP; + off = skb_mac_header_len(skb); + switch (mode) { + case BPF_ADJ_ROOM_NET: + off += bpf_skb_net_base_len(skb); + break; + case BPF_ADJ_ROOM_MAC: + break; + default: + return -ENOTSUPP; + } + len_cur = skb->len - skb_network_offset(skb); - if (skb_transport_header_was_set(skb) && !trans_same) - len_cur = skb_network_header_len(skb); if ((shrink && (len_diff_abs >= len_cur || len_cur - len_diff_abs < len_min)) || (!shrink && (skb->len + len_diff_abs > len_max && !skb_is_gso(skb)))) return -ENOTSUPP; - ret = shrink ? bpf_skb_net_shrink(skb, len_diff_abs) : - bpf_skb_net_grow(skb, len_diff_abs); + ret = shrink ? bpf_skb_net_shrink(skb, off, len_diff_abs, flags) : + bpf_skb_net_grow(skb, off, len_diff_abs, flags); bpf_compute_data_pointers(skb); return ret; } -BPF_CALL_4(bpf_skb_adjust_room, struct sk_buff *, skb, s32, len_diff, - u32, mode, u64, flags) -{ - if (unlikely(flags)) - return -EINVAL; - if (likely(mode == BPF_ADJ_ROOM_NET)) - return bpf_skb_adjust_net(skb, len_diff); - - return -ENOTSUPP; -} - static const struct bpf_func_proto bpf_skb_adjust_room_proto = { .func = bpf_skb_adjust_room, .gpl_only = false, @@ -4355,8 +4480,7 @@ BPF_CALL_2(bpf_sock_ops_cb_flags_set, struct bpf_sock_ops_kern *, bpf_sock, if (!IS_ENABLED(CONFIG_INET) || !sk_fullsock(sk)) return -EINVAL; - if (val) - tcp_sk(sk)->bpf_sock_ops_cb_flags = val; + tcp_sk(sk)->bpf_sock_ops_cb_flags = val; return argval & (~BPF_SOCK_OPS_ALL_CB_FLAGS); } @@ -4482,11 +4606,11 @@ static int bpf_fib_set_fwd_params(struct bpf_fib_lookup *params, static int bpf_ipv4_fib_lookup(struct net *net, struct bpf_fib_lookup *params, u32 flags, bool check_mtu) { + struct fib_nh_common *nhc; struct in_device *in_dev; struct neighbour *neigh; struct net_device *dev; struct fib_result res; - struct fib_nh *nh; struct flowi4 fl4; int err; u32 mtu; @@ -4559,22 +4683,33 @@ static int bpf_ipv4_fib_lookup(struct net *net, struct bpf_fib_lookup *params, return BPF_FIB_LKUP_RET_FRAG_NEEDED; } - nh = &res.fi->fib_nh[res.nh_sel]; + nhc = res.nhc; /* do not handle lwt encaps right now */ - if (nh->nh_lwtstate) + if (nhc->nhc_lwtstate) return BPF_FIB_LKUP_RET_UNSUPP_LWT; - dev = nh->nh_dev; - if (nh->nh_gw) - params->ipv4_dst = nh->nh_gw; + dev = nhc->nhc_dev; params->rt_metric = res.fi->fib_priority; /* xdp and cls_bpf programs are run in RCU-bh so * rcu_read_lock_bh is not needed here */ - neigh = __ipv4_neigh_lookup_noref(dev, (__force u32)params->ipv4_dst); + if (likely(nhc->nhc_gw_family != AF_INET6)) { + if (nhc->nhc_gw_family) + params->ipv4_dst = nhc->nhc_gw.ipv4; + + neigh = __ipv4_neigh_lookup_noref(dev, + (__force u32)params->ipv4_dst); + } else { + struct in6_addr *dst = (struct in6_addr *)params->ipv6_dst; + + params->family = AF_INET6; + *dst = nhc->nhc_gw.ipv6; + neigh = __ipv6_neigh_lookup_noref_stub(dev, dst); + } + if (!neigh) return BPF_FIB_LKUP_RET_NO_NEIGH; @@ -4588,13 +4723,13 @@ static int bpf_ipv6_fib_lookup(struct net *net, struct bpf_fib_lookup *params, { struct in6_addr *src = (struct in6_addr *) params->ipv6_src; struct in6_addr *dst = (struct in6_addr *) params->ipv6_dst; + struct fib6_result res = {}; struct neighbour *neigh; struct net_device *dev; struct inet6_dev *idev; - struct fib6_info *f6i; struct flowi6 fl6; int strict = 0; - int oif; + int oif, err; u32 mtu; /* link local addresses are never forwarded */ @@ -4636,61 +4771,57 @@ static int bpf_ipv6_fib_lookup(struct net *net, struct bpf_fib_lookup *params, if (unlikely(!tb)) return BPF_FIB_LKUP_RET_NOT_FWDED; - f6i = ipv6_stub->fib6_table_lookup(net, tb, oif, &fl6, strict); + err = ipv6_stub->fib6_table_lookup(net, tb, oif, &fl6, &res, + strict); } else { fl6.flowi6_mark = 0; fl6.flowi6_secid = 0; fl6.flowi6_tun_key.tun_id = 0; fl6.flowi6_uid = sock_net_uid(net, NULL); - f6i = ipv6_stub->fib6_lookup(net, oif, &fl6, strict); + err = ipv6_stub->fib6_lookup(net, oif, &fl6, &res, strict); } - if (unlikely(IS_ERR_OR_NULL(f6i) || f6i == net->ipv6.fib6_null_entry)) + if (unlikely(err || IS_ERR_OR_NULL(res.f6i) || + res.f6i == net->ipv6.fib6_null_entry)) return BPF_FIB_LKUP_RET_NOT_FWDED; - if (unlikely(f6i->fib6_flags & RTF_REJECT)) { - switch (f6i->fib6_type) { - case RTN_BLACKHOLE: - return BPF_FIB_LKUP_RET_BLACKHOLE; - case RTN_UNREACHABLE: - return BPF_FIB_LKUP_RET_UNREACHABLE; - case RTN_PROHIBIT: - return BPF_FIB_LKUP_RET_PROHIBIT; - default: - return BPF_FIB_LKUP_RET_NOT_FWDED; - } - } - - if (f6i->fib6_type != RTN_UNICAST) + switch (res.fib6_type) { + /* only unicast is forwarded */ + case RTN_UNICAST: + break; + case RTN_BLACKHOLE: + return BPF_FIB_LKUP_RET_BLACKHOLE; + case RTN_UNREACHABLE: + return BPF_FIB_LKUP_RET_UNREACHABLE; + case RTN_PROHIBIT: + return BPF_FIB_LKUP_RET_PROHIBIT; + default: return BPF_FIB_LKUP_RET_NOT_FWDED; + } - if (f6i->fib6_nsiblings && fl6.flowi6_oif == 0) - f6i = ipv6_stub->fib6_multipath_select(net, f6i, &fl6, - fl6.flowi6_oif, NULL, - strict); + ipv6_stub->fib6_select_path(net, &res, &fl6, fl6.flowi6_oif, + fl6.flowi6_oif != 0, NULL, strict); if (check_mtu) { - mtu = ipv6_stub->ip6_mtu_from_fib6(f6i, dst, src); + mtu = ipv6_stub->ip6_mtu_from_fib6(&res, dst, src); if (params->tot_len > mtu) return BPF_FIB_LKUP_RET_FRAG_NEEDED; } - if (f6i->fib6_nh.nh_lwtstate) + if (res.nh->fib_nh_lws) return BPF_FIB_LKUP_RET_UNSUPP_LWT; - if (f6i->fib6_flags & RTF_GATEWAY) - *dst = f6i->fib6_nh.nh_gw; + if (res.nh->fib_nh_gw_family) + *dst = res.nh->fib_nh_gw6; - dev = f6i->fib6_nh.nh_dev; - params->rt_metric = f6i->fib6_metric; + dev = res.nh->fib_nh_dev; + params->rt_metric = res.f6i->fib6_metric; /* xdp and cls_bpf programs are run in RCU-bh so rcu_read_lock_bh is - * not needed here. Can not use __ipv6_neigh_lookup_noref here - * because we need to get nd_tbl via the stub + * not needed here. */ - neigh = ___neigh_lookup_noref(ipv6_stub->nd_tbl, neigh_key_eq128, - ndisc_hashfn, dst, dev); + neigh = __ipv6_neigh_lookup_noref_stub(dev, dst); if (!neigh) return BPF_FIB_LKUP_RET_NO_NEIGH; @@ -5158,15 +5289,15 @@ static struct sock *sk_lookup(struct net *net, struct bpf_sock_tuple *tuple, return sk; } -/* bpf_sk_lookup performs the core lookup for different types of sockets, +/* bpf_skc_lookup performs the core lookup for different types of sockets, * taking a reference on the socket if it doesn't have the flag SOCK_RCU_FREE. * Returns the socket as an 'unsigned long' to simplify the casting in the * callers to satisfy BPF_CALL declarations. */ -static unsigned long -__bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len, - struct net *caller_net, u32 ifindex, u8 proto, u64 netns_id, - u64 flags) +static struct sock * +__bpf_skc_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len, + struct net *caller_net, u32 ifindex, u8 proto, u64 netns_id, + u64 flags) { struct sock *sk = NULL; u8 family = AF_UNSPEC; @@ -5194,15 +5325,27 @@ __bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len, put_net(net); } +out: + return sk; +} + +static struct sock * +__bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len, + struct net *caller_net, u32 ifindex, u8 proto, u64 netns_id, + u64 flags) +{ + struct sock *sk = __bpf_skc_lookup(skb, tuple, len, caller_net, + ifindex, proto, netns_id, flags); + if (sk) sk = sk_to_full_sk(sk); -out: - return (unsigned long) sk; + + return sk; } -static unsigned long -bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len, - u8 proto, u64 netns_id, u64 flags) +static struct sock * +bpf_skc_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len, + u8 proto, u64 netns_id, u64 flags) { struct net *caller_net; int ifindex; @@ -5215,14 +5358,47 @@ bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len, ifindex = 0; } - return __bpf_sk_lookup(skb, tuple, len, caller_net, ifindex, - proto, netns_id, flags); + return __bpf_skc_lookup(skb, tuple, len, caller_net, ifindex, proto, + netns_id, flags); } +static struct sock * +bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len, + u8 proto, u64 netns_id, u64 flags) +{ + struct sock *sk = bpf_skc_lookup(skb, tuple, len, proto, netns_id, + flags); + + if (sk) + sk = sk_to_full_sk(sk); + + return sk; +} + +BPF_CALL_5(bpf_skc_lookup_tcp, struct sk_buff *, skb, + struct bpf_sock_tuple *, tuple, u32, len, u64, netns_id, u64, flags) +{ + return (unsigned long)bpf_skc_lookup(skb, tuple, len, IPPROTO_TCP, + netns_id, flags); +} + +static const struct bpf_func_proto bpf_skc_lookup_tcp_proto = { + .func = bpf_skc_lookup_tcp, + .gpl_only = false, + .pkt_access = true, + .ret_type = RET_PTR_TO_SOCK_COMMON_OR_NULL, + .arg1_type = ARG_PTR_TO_CTX, + .arg2_type = ARG_PTR_TO_MEM, + .arg3_type = ARG_CONST_SIZE, + .arg4_type = ARG_ANYTHING, + .arg5_type = ARG_ANYTHING, +}; + BPF_CALL_5(bpf_sk_lookup_tcp, struct sk_buff *, skb, struct bpf_sock_tuple *, tuple, u32, len, u64, netns_id, u64, flags) { - return bpf_sk_lookup(skb, tuple, len, IPPROTO_TCP, netns_id, flags); + return (unsigned long)bpf_sk_lookup(skb, tuple, len, IPPROTO_TCP, + netns_id, flags); } static const struct bpf_func_proto bpf_sk_lookup_tcp_proto = { @@ -5240,7 +5416,8 @@ static const struct bpf_func_proto bpf_sk_lookup_tcp_proto = { BPF_CALL_5(bpf_sk_lookup_udp, struct sk_buff *, skb, struct bpf_sock_tuple *, tuple, u32, len, u64, netns_id, u64, flags) { - return bpf_sk_lookup(skb, tuple, len, IPPROTO_UDP, netns_id, flags); + return (unsigned long)bpf_sk_lookup(skb, tuple, len, IPPROTO_UDP, + netns_id, flags); } static const struct bpf_func_proto bpf_sk_lookup_udp_proto = { @@ -5275,8 +5452,9 @@ BPF_CALL_5(bpf_xdp_sk_lookup_udp, struct xdp_buff *, ctx, struct net *caller_net = dev_net(ctx->rxq->dev); int ifindex = ctx->rxq->dev->ifindex; - return __bpf_sk_lookup(NULL, tuple, len, caller_net, ifindex, - IPPROTO_UDP, netns_id, flags); + return (unsigned long)__bpf_sk_lookup(NULL, tuple, len, caller_net, + ifindex, IPPROTO_UDP, netns_id, + flags); } static const struct bpf_func_proto bpf_xdp_sk_lookup_udp_proto = { @@ -5291,14 +5469,38 @@ static const struct bpf_func_proto bpf_xdp_sk_lookup_udp_proto = { .arg5_type = ARG_ANYTHING, }; +BPF_CALL_5(bpf_xdp_skc_lookup_tcp, struct xdp_buff *, ctx, + struct bpf_sock_tuple *, tuple, u32, len, u32, netns_id, u64, flags) +{ + struct net *caller_net = dev_net(ctx->rxq->dev); + int ifindex = ctx->rxq->dev->ifindex; + + return (unsigned long)__bpf_skc_lookup(NULL, tuple, len, caller_net, + ifindex, IPPROTO_TCP, netns_id, + flags); +} + +static const struct bpf_func_proto bpf_xdp_skc_lookup_tcp_proto = { + .func = bpf_xdp_skc_lookup_tcp, + .gpl_only = false, + .pkt_access = true, + .ret_type = RET_PTR_TO_SOCK_COMMON_OR_NULL, + .arg1_type = ARG_PTR_TO_CTX, + .arg2_type = ARG_PTR_TO_MEM, + .arg3_type = ARG_CONST_SIZE, + .arg4_type = ARG_ANYTHING, + .arg5_type = ARG_ANYTHING, +}; + BPF_CALL_5(bpf_xdp_sk_lookup_tcp, struct xdp_buff *, ctx, struct bpf_sock_tuple *, tuple, u32, len, u32, netns_id, u64, flags) { struct net *caller_net = dev_net(ctx->rxq->dev); int ifindex = ctx->rxq->dev->ifindex; - return __bpf_sk_lookup(NULL, tuple, len, caller_net, ifindex, - IPPROTO_TCP, netns_id, flags); + return (unsigned long)__bpf_sk_lookup(NULL, tuple, len, caller_net, + ifindex, IPPROTO_TCP, netns_id, + flags); } static const struct bpf_func_proto bpf_xdp_sk_lookup_tcp_proto = { @@ -5313,11 +5515,31 @@ static const struct bpf_func_proto bpf_xdp_sk_lookup_tcp_proto = { .arg5_type = ARG_ANYTHING, }; +BPF_CALL_5(bpf_sock_addr_skc_lookup_tcp, struct bpf_sock_addr_kern *, ctx, + struct bpf_sock_tuple *, tuple, u32, len, u64, netns_id, u64, flags) +{ + return (unsigned long)__bpf_skc_lookup(NULL, tuple, len, + sock_net(ctx->sk), 0, + IPPROTO_TCP, netns_id, flags); +} + +static const struct bpf_func_proto bpf_sock_addr_skc_lookup_tcp_proto = { + .func = bpf_sock_addr_skc_lookup_tcp, + .gpl_only = false, + .ret_type = RET_PTR_TO_SOCK_COMMON_OR_NULL, + .arg1_type = ARG_PTR_TO_CTX, + .arg2_type = ARG_PTR_TO_MEM, + .arg3_type = ARG_CONST_SIZE, + .arg4_type = ARG_ANYTHING, + .arg5_type = ARG_ANYTHING, +}; + BPF_CALL_5(bpf_sock_addr_sk_lookup_tcp, struct bpf_sock_addr_kern *, ctx, struct bpf_sock_tuple *, tuple, u32, len, u64, netns_id, u64, flags) { - return __bpf_sk_lookup(NULL, tuple, len, sock_net(ctx->sk), 0, - IPPROTO_TCP, netns_id, flags); + return (unsigned long)__bpf_sk_lookup(NULL, tuple, len, + sock_net(ctx->sk), 0, IPPROTO_TCP, + netns_id, flags); } static const struct bpf_func_proto bpf_sock_addr_sk_lookup_tcp_proto = { @@ -5334,8 +5556,9 @@ static const struct bpf_func_proto bpf_sock_addr_sk_lookup_tcp_proto = { BPF_CALL_5(bpf_sock_addr_sk_lookup_udp, struct bpf_sock_addr_kern *, ctx, struct bpf_sock_tuple *, tuple, u32, len, u64, netns_id, u64, flags) { - return __bpf_sk_lookup(NULL, tuple, len, sock_net(ctx->sk), 0, - IPPROTO_UDP, netns_id, flags); + return (unsigned long)__bpf_sk_lookup(NULL, tuple, len, + sock_net(ctx->sk), 0, IPPROTO_UDP, + netns_id, flags); } static const struct bpf_func_proto bpf_sock_addr_sk_lookup_udp_proto = { @@ -5463,6 +5686,74 @@ static const struct bpf_func_proto bpf_skb_ecn_set_ce_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, }; + +BPF_CALL_5(bpf_tcp_check_syncookie, struct sock *, sk, void *, iph, u32, iph_len, + struct tcphdr *, th, u32, th_len) +{ +#ifdef CONFIG_SYN_COOKIES + u32 cookie; + int ret; + + if (unlikely(th_len < sizeof(*th))) + return -EINVAL; + + /* sk_listener() allows TCP_NEW_SYN_RECV, which makes no sense here. */ + if (sk->sk_protocol != IPPROTO_TCP || sk->sk_state != TCP_LISTEN) + return -EINVAL; + + if (!sock_net(sk)->ipv4.sysctl_tcp_syncookies) + return -EINVAL; + + if (!th->ack || th->rst || th->syn) + return -ENOENT; + + if (tcp_synq_no_recent_overflow(sk)) + return -ENOENT; + + cookie = ntohl(th->ack_seq) - 1; + + switch (sk->sk_family) { + case AF_INET: + if (unlikely(iph_len < sizeof(struct iphdr))) + return -EINVAL; + + ret = __cookie_v4_check((struct iphdr *)iph, th, cookie); + break; + +#if IS_BUILTIN(CONFIG_IPV6) + case AF_INET6: + if (unlikely(iph_len < sizeof(struct ipv6hdr))) + return -EINVAL; + + ret = __cookie_v6_check((struct ipv6hdr *)iph, th, cookie); + break; +#endif /* CONFIG_IPV6 */ + + default: + return -EPROTONOSUPPORT; + } + + if (ret > 0) + return 0; + + return -ENOENT; +#else + return -ENOTSUPP; +#endif +} + +static const struct bpf_func_proto bpf_tcp_check_syncookie_proto = { + .func = bpf_tcp_check_syncookie, + .gpl_only = true, + .pkt_access = true, + .ret_type = RET_INTEGER, + .arg1_type = ARG_PTR_TO_SOCK_COMMON, + .arg2_type = ARG_PTR_TO_MEM, + .arg3_type = ARG_CONST_SIZE, + .arg4_type = ARG_PTR_TO_MEM, + .arg5_type = ARG_CONST_SIZE, +}; + #endif /* CONFIG_INET */ bool bpf_helper_changes_pkt_data(void *func) @@ -5588,6 +5879,8 @@ sock_addr_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_sock_addr_sk_lookup_udp_proto; case BPF_FUNC_sk_release: return &bpf_sk_release_proto; + case BPF_FUNC_skc_lookup_tcp: + return &bpf_sock_addr_skc_lookup_tcp_proto; #endif /* CONFIG_INET */ default: return bpf_base_func_proto(func_id); @@ -5611,6 +5904,9 @@ sk_filter_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) } } +const struct bpf_func_proto bpf_sk_storage_get_proto __weak; +const struct bpf_func_proto bpf_sk_storage_delete_proto __weak; + static const struct bpf_func_proto * cg_skb_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) { @@ -5619,6 +5915,10 @@ cg_skb_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_get_local_storage_proto; case BPF_FUNC_sk_fullsock: return &bpf_sk_fullsock_proto; + case BPF_FUNC_sk_storage_get: + return &bpf_sk_storage_get_proto; + case BPF_FUNC_sk_storage_delete: + return &bpf_sk_storage_delete_proto; #ifdef CONFIG_INET case BPF_FUNC_tcp_sock: return &bpf_tcp_sock_proto; @@ -5700,6 +6000,10 @@ tc_cls_act_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_skb_fib_lookup_proto; case BPF_FUNC_sk_fullsock: return &bpf_sk_fullsock_proto; + case BPF_FUNC_sk_storage_get: + return &bpf_sk_storage_get_proto; + case BPF_FUNC_sk_storage_delete: + return &bpf_sk_storage_delete_proto; #ifdef CONFIG_XFRM case BPF_FUNC_skb_get_xfrm_state: return &bpf_skb_get_xfrm_state_proto; @@ -5721,6 +6025,12 @@ tc_cls_act_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_tcp_sock_proto; case BPF_FUNC_get_listener_sock: return &bpf_get_listener_sock_proto; + case BPF_FUNC_skc_lookup_tcp: + return &bpf_skc_lookup_tcp_proto; + case BPF_FUNC_tcp_check_syncookie: + return &bpf_tcp_check_syncookie_proto; + case BPF_FUNC_skb_ecn_set_ce: + return &bpf_skb_ecn_set_ce_proto; #endif default: return bpf_base_func_proto(func_id); @@ -5756,6 +6066,10 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_xdp_sk_lookup_tcp_proto; case BPF_FUNC_sk_release: return &bpf_sk_release_proto; + case BPF_FUNC_skc_lookup_tcp: + return &bpf_xdp_skc_lookup_tcp_proto; + case BPF_FUNC_tcp_check_syncookie: + return &bpf_tcp_check_syncookie_proto; #endif default: return bpf_base_func_proto(func_id); @@ -5848,6 +6162,8 @@ sk_skb_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_sk_lookup_udp_proto; case BPF_FUNC_sk_release: return &bpf_sk_release_proto; + case BPF_FUNC_skc_lookup_tcp: + return &bpf_skc_lookup_tcp_proto; #endif default: return bpf_base_func_proto(func_id); @@ -5859,7 +6175,7 @@ flow_dissector_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) { switch (func_id) { case BPF_FUNC_skb_load_bytes: - return &bpf_skb_load_bytes_proto; + return &bpf_flow_dissector_load_bytes_proto; default: return bpf_base_func_proto(func_id); } @@ -5986,9 +6302,7 @@ static bool bpf_skb_is_valid_access(int off, int size, enum bpf_access_type type return false; break; case bpf_ctx_range_ptr(struct __sk_buff, flow_keys): - if (size != sizeof(__u64)) - return false; - break; + return false; case bpf_ctx_range(struct __sk_buff, tstamp): if (size != sizeof(__u64)) return false; @@ -6023,7 +6337,6 @@ static bool sk_filter_is_valid_access(int off, int size, case bpf_ctx_range(struct __sk_buff, data): case bpf_ctx_range(struct __sk_buff, data_meta): case bpf_ctx_range(struct __sk_buff, data_end): - case bpf_ctx_range_ptr(struct __sk_buff, flow_keys): case bpf_ctx_range_till(struct __sk_buff, family, local_port): case bpf_ctx_range(struct __sk_buff, tstamp): case bpf_ctx_range(struct __sk_buff, wire_len): @@ -6050,7 +6363,6 @@ static bool cg_skb_is_valid_access(int off, int size, switch (off) { case bpf_ctx_range(struct __sk_buff, tc_classid): case bpf_ctx_range(struct __sk_buff, data_meta): - case bpf_ctx_range_ptr(struct __sk_buff, flow_keys): case bpf_ctx_range(struct __sk_buff, wire_len): return false; case bpf_ctx_range(struct __sk_buff, data): @@ -6096,7 +6408,6 @@ static bool lwt_is_valid_access(int off, int size, case bpf_ctx_range(struct __sk_buff, tc_classid): case bpf_ctx_range_till(struct __sk_buff, family, local_port): case bpf_ctx_range(struct __sk_buff, data_meta): - case bpf_ctx_range_ptr(struct __sk_buff, flow_keys): case bpf_ctx_range(struct __sk_buff, tstamp): case bpf_ctx_range(struct __sk_buff, wire_len): return false; @@ -6339,7 +6650,6 @@ static bool tc_cls_act_is_valid_access(int off, int size, case bpf_ctx_range(struct __sk_buff, data_end): info->reg_type = PTR_TO_PACKET_END; break; - case bpf_ctx_range_ptr(struct __sk_buff, flow_keys): case bpf_ctx_range_till(struct __sk_buff, family, local_port): return false; } @@ -6541,7 +6851,6 @@ static bool sk_skb_is_valid_access(int off, int size, switch (off) { case bpf_ctx_range(struct __sk_buff, tc_classid): case bpf_ctx_range(struct __sk_buff, data_meta): - case bpf_ctx_range_ptr(struct __sk_buff, flow_keys): case bpf_ctx_range(struct __sk_buff, tstamp): case bpf_ctx_range(struct __sk_buff, wire_len): return false; @@ -6615,24 +6924,65 @@ static bool flow_dissector_is_valid_access(int off, int size, const struct bpf_prog *prog, struct bpf_insn_access_aux *info) { + const int size_default = sizeof(__u32); + + if (off < 0 || off >= sizeof(struct __sk_buff)) + return false; + if (type == BPF_WRITE) return false; switch (off) { case bpf_ctx_range(struct __sk_buff, data): + if (size != size_default) + return false; info->reg_type = PTR_TO_PACKET; - break; + return true; case bpf_ctx_range(struct __sk_buff, data_end): + if (size != size_default) + return false; info->reg_type = PTR_TO_PACKET_END; - break; + return true; case bpf_ctx_range_ptr(struct __sk_buff, flow_keys): + if (size != sizeof(__u64)) + return false; info->reg_type = PTR_TO_FLOW_KEYS; - break; + return true; default: return false; } +} - return bpf_skb_is_valid_access(off, size, type, prog, info); +static u32 flow_dissector_convert_ctx_access(enum bpf_access_type type, + const struct bpf_insn *si, + struct bpf_insn *insn_buf, + struct bpf_prog *prog, + u32 *target_size) + +{ + struct bpf_insn *insn = insn_buf; + + switch (si->off) { + case offsetof(struct __sk_buff, data): + *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_flow_dissector, data), + si->dst_reg, si->src_reg, + offsetof(struct bpf_flow_dissector, data)); + break; + + case offsetof(struct __sk_buff, data_end): + *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_flow_dissector, data_end), + si->dst_reg, si->src_reg, + offsetof(struct bpf_flow_dissector, data_end)); + break; + + case offsetof(struct __sk_buff, flow_keys): + *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_flow_dissector, flow_keys), + si->dst_reg, si->src_reg, + offsetof(struct bpf_flow_dissector, flow_keys)); + break; + } + + return insn - insn_buf; } static u32 bpf_convert_ctx_access(enum bpf_access_type type, @@ -6939,15 +7289,6 @@ static u32 bpf_convert_ctx_access(enum bpf_access_type type, skc_num, 2, target_size)); break; - case offsetof(struct __sk_buff, flow_keys): - off = si->off; - off -= offsetof(struct __sk_buff, flow_keys); - off += offsetof(struct sk_buff, cb); - off += offsetof(struct qdisc_skb_cb, flow_keys); - *insn++ = BPF_LDX_MEM(BPF_SIZEOF(void *), si->dst_reg, - si->src_reg, off); - break; - case offsetof(struct __sk_buff, tstamp): BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, tstamp) != 8); @@ -7952,7 +8293,7 @@ const struct bpf_prog_ops sk_msg_prog_ops = { const struct bpf_verifier_ops flow_dissector_verifier_ops = { .get_func_proto = flow_dissector_func_proto, .is_valid_access = flow_dissector_is_valid_access, - .convert_ctx_access = bpf_convert_ctx_access, + .convert_ctx_access = flow_dissector_convert_ctx_access, }; const struct bpf_prog_ops flow_dissector_prog_ops = { diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c index 94a450b2191a9e25ca79534f9caa68a442daa80c..9ca784c592ac8c9c58282289a81889fbe4658a9e 100644 --- a/net/core/flow_dissector.c +++ b/net/core/flow_dissector.c @@ -65,6 +65,45 @@ void skb_flow_dissector_init(struct flow_dissector *flow_dissector, } EXPORT_SYMBOL(skb_flow_dissector_init); +int skb_flow_dissector_prog_query(const union bpf_attr *attr, + union bpf_attr __user *uattr) +{ + __u32 __user *prog_ids = u64_to_user_ptr(attr->query.prog_ids); + u32 prog_id, prog_cnt = 0, flags = 0; + struct bpf_prog *attached; + struct net *net; + + if (attr->query.query_flags) + return -EINVAL; + + net = get_net_ns_by_fd(attr->query.target_fd); + if (IS_ERR(net)) + return PTR_ERR(net); + + rcu_read_lock(); + attached = rcu_dereference(net->flow_dissector_prog); + if (attached) { + prog_cnt = 1; + prog_id = attached->aux->id; + } + rcu_read_unlock(); + + put_net(net); + + if (copy_to_user(&uattr->query.attach_flags, &flags, sizeof(flags))) + return -EFAULT; + if (copy_to_user(&uattr->query.prog_cnt, &prog_cnt, sizeof(prog_cnt))) + return -EFAULT; + + if (!attr->query.prog_cnt || !prog_ids || !prog_cnt) + return 0; + + if (copy_to_user(prog_ids, &prog_id, sizeof(u32))) + return -EFAULT; + + return 0; +} + int skb_flow_dissector_bpf_prog_attach(const union bpf_attr *attr, struct bpf_prog *prog) { @@ -683,50 +722,30 @@ static void __skb_flow_bpf_to_target(const struct bpf_flow_keys *flow_keys, } } -bool __skb_flow_bpf_dissect(struct bpf_prog *prog, - const struct sk_buff *skb, - struct flow_dissector *flow_dissector, - struct bpf_flow_keys *flow_keys) +bool bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx, + __be16 proto, int nhoff, int hlen) { - struct bpf_skb_data_end cb_saved; - struct bpf_skb_data_end *cb; + struct bpf_flow_keys *flow_keys = ctx->flow_keys; u32 result; - /* Note that even though the const qualifier is discarded - * throughout the execution of the BPF program, all changes(the - * control block) are reverted after the BPF program returns. - * Therefore, __skb_flow_dissect does not alter the skb. - */ - - cb = (struct bpf_skb_data_end *)skb->cb; - - /* Save Control Block */ - memcpy(&cb_saved, cb, sizeof(cb_saved)); - memset(cb, 0, sizeof(*cb)); - /* Pass parameters to the BPF program */ memset(flow_keys, 0, sizeof(*flow_keys)); - cb->qdisc_cb.flow_keys = flow_keys; - flow_keys->n_proto = skb->protocol; - flow_keys->nhoff = skb_network_offset(skb); + flow_keys->n_proto = proto; + flow_keys->nhoff = nhoff; flow_keys->thoff = flow_keys->nhoff; - bpf_compute_data_pointers((struct sk_buff *)skb); - result = BPF_PROG_RUN(prog, skb); + result = BPF_PROG_RUN(prog, ctx); - /* Restore state */ - memcpy(cb, &cb_saved, sizeof(cb_saved)); - - flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, - skb_network_offset(skb), skb->len); + flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen); flow_keys->thoff = clamp_t(u16, flow_keys->thoff, - flow_keys->nhoff, skb->len); + flow_keys->nhoff, hlen); return result == BPF_OK; } /** * __skb_flow_dissect - extract the flow_keys struct and return it + * @net: associated network namespace, derived from @skb if NULL * @skb: sk_buff to extract the flow from, can be NULL if the rest are specified * @flow_dissector: list of keys to dissect * @target_container: target structure to put dissected values into @@ -734,6 +753,8 @@ bool __skb_flow_bpf_dissect(struct bpf_prog *prog, * @proto: protocol for which to get the flow, if @data is NULL use skb->protocol * @nhoff: network header offset, if @data is NULL use skb_network_offset(skb) * @hlen: packet header length, if @data is NULL use skb_headlen(skb) + * @flags: flags that control the dissection process, e.g. + * FLOW_DISSECTOR_F_STOP_AT_L3. * * The function will try to retrieve individual keys into target specified * by flow_dissector from either the skbuff or a raw buffer specified by the @@ -741,7 +762,8 @@ bool __skb_flow_bpf_dissect(struct bpf_prog *prog, * * Caller must take care of zeroing target container memory. */ -bool __skb_flow_dissect(const struct sk_buff *skb, +bool __skb_flow_dissect(const struct net *net, + const struct sk_buff *skb, struct flow_dissector *flow_dissector, void *target_container, void *data, __be16 proto, int nhoff, int hlen, @@ -754,6 +776,7 @@ bool __skb_flow_dissect(const struct sk_buff *skb, struct flow_dissector_key_icmp *key_icmp; struct flow_dissector_key_tags *key_tags; struct flow_dissector_key_vlan *key_vlan; + struct bpf_prog *attached = NULL; enum flow_dissect_ret fdret; enum flow_dissector_key_id dissector_vlan = FLOW_DISSECTOR_KEY_MAX; int num_hdrs = 0; @@ -796,22 +819,39 @@ bool __skb_flow_dissect(const struct sk_buff *skb, target_container); if (skb) { - struct bpf_flow_keys flow_keys; - struct bpf_prog *attached = NULL; + if (!net) { + if (skb->dev) + net = dev_net(skb->dev); + else if (skb->sk) + net = sock_net(skb->sk); + } + } + WARN_ON_ONCE(!net); + if (net) { rcu_read_lock(); - - if (skb->dev) - attached = rcu_dereference(dev_net(skb->dev)->flow_dissector_prog); - else if (skb->sk) - attached = rcu_dereference(sock_net(skb->sk)->flow_dissector_prog); - else - WARN_ON_ONCE(1); + attached = rcu_dereference(net->flow_dissector_prog); if (attached) { - ret = __skb_flow_bpf_dissect(attached, skb, - flow_dissector, - &flow_keys); + struct bpf_flow_keys flow_keys; + struct bpf_flow_dissector ctx = { + .flow_keys = &flow_keys, + .data = data, + .data_end = data + hlen, + }; + __be16 n_proto = proto; + + if (skb) { + ctx.skb = skb; + /* we can't use 'proto' in the skb case + * because it might be set to skb->vlan_proto + * which has been pulled from the data + */ + n_proto = skb->protocol; + } + + ret = bpf_flow_dissect(attached, &ctx, n_proto, nhoff, + hlen); __skb_flow_bpf_to_target(&flow_keys, flow_dissector, target_container); rcu_read_unlock(); @@ -1408,8 +1448,8 @@ u32 __skb_get_hash_symmetric(const struct sk_buff *skb) __flow_hash_secret_init(); memset(&keys, 0, sizeof(keys)); - __skb_flow_dissect(skb, &flow_keys_dissector_symmetric, &keys, - NULL, 0, 0, 0, + __skb_flow_dissect(NULL, skb, &flow_keys_dissector_symmetric, + &keys, NULL, 0, 0, 0, FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL); return __flow_hash_from_keys(&keys, hashrnd); @@ -1510,7 +1550,8 @@ u32 skb_get_poff(const struct sk_buff *skb) { struct flow_keys_basic keys; - if (!skb_flow_dissect_flow_keys_basic(skb, &keys, NULL, 0, 0, 0, 0)) + if (!skb_flow_dissect_flow_keys_basic(NULL, skb, &keys, + NULL, 0, 0, 0, 0)) return 0; return __skb_get_poff(skb, skb->data, &keys, skb_headlen(skb)); diff --git a/net/core/gen_stats.c b/net/core/gen_stats.c index ac679f74ba4754e69c51dd9b3fb9b23611782300..9bf1b9ad17806dfaa579317408e5c4707d014cc0 100644 --- a/net/core/gen_stats.c +++ b/net/core/gen_stats.c @@ -291,6 +291,7 @@ __gnet_stats_copy_queue_cpu(struct gnet_stats_queue *qstats, for_each_possible_cpu(i) { const struct gnet_stats_queue *qcpu = per_cpu_ptr(q, i); + qstats->qlen = 0; qstats->backlog += qcpu->backlog; qstats->drops += qcpu->drops; qstats->requeues += qcpu->requeues; @@ -306,6 +307,7 @@ void __gnet_stats_copy_queue(struct gnet_stats_queue *qstats, if (cpu) { __gnet_stats_copy_queue_cpu(qstats, cpu); } else { + qstats->qlen = q->qlen; qstats->backlog = q->backlog; qstats->drops = q->drops; qstats->requeues = q->requeues; diff --git a/net/core/lwt_bpf.c b/net/core/lwt_bpf.c index 126d31ff5ee343e3248e1bdc9ba238f46a2a69c4..1c94f529f4a13022a2532fb9fa91dbafbd57802c 100644 --- a/net/core/lwt_bpf.c +++ b/net/core/lwt_bpf.c @@ -18,6 +18,7 @@ #include #include #include +#include struct bpf_lwt_prog { struct bpf_prog *prog; @@ -342,8 +343,8 @@ static int bpf_parse_prog(struct nlattr *attr, struct bpf_lwt_prog *prog, int ret; u32 fd; - ret = nla_parse_nested(tb, LWT_BPF_PROG_MAX, attr, bpf_prog_policy, - NULL); + ret = nla_parse_nested_deprecated(tb, LWT_BPF_PROG_MAX, attr, + bpf_prog_policy, NULL); if (ret < 0) return ret; @@ -384,7 +385,8 @@ static int bpf_build_state(struct nlattr *nla, if (family != AF_INET && family != AF_INET6) return -EAFNOSUPPORT; - ret = nla_parse_nested(tb, LWT_BPF_MAX, nla, bpf_nl_policy, extack); + ret = nla_parse_nested_deprecated(tb, LWT_BPF_MAX, nla, bpf_nl_policy, + extack); if (ret < 0) return ret; @@ -452,7 +454,7 @@ static int bpf_fill_lwt_prog(struct sk_buff *skb, int attr, if (!prog->prog) return 0; - nest = nla_nest_start(skb, attr); + nest = nla_nest_start_noflag(skb, attr); if (!nest) return -EMSGSIZE; diff --git a/net/core/lwtunnel.c b/net/core/lwtunnel.c index 19b557bd294bec9777acd58c7c0d017822e32294..69e249fbc02f8363141f07a723fe52d77653ea96 100644 --- a/net/core/lwtunnel.c +++ b/net/core/lwtunnel.c @@ -26,7 +26,7 @@ #include #include #include -#include +#include #ifdef CONFIG_MODULES @@ -223,7 +223,8 @@ void lwtstate_free(struct lwtunnel_state *lws) } EXPORT_SYMBOL_GPL(lwtstate_free); -int lwtunnel_fill_encap(struct sk_buff *skb, struct lwtunnel_state *lwtstate) +int lwtunnel_fill_encap(struct sk_buff *skb, struct lwtunnel_state *lwtstate, + int encap_attr, int encap_type_attr) { const struct lwtunnel_encap_ops *ops; struct nlattr *nest; @@ -236,7 +237,7 @@ int lwtunnel_fill_encap(struct sk_buff *skb, struct lwtunnel_state *lwtstate) lwtstate->type > LWTUNNEL_ENCAP_MAX) return 0; - nest = nla_nest_start(skb, RTA_ENCAP); + nest = nla_nest_start_noflag(skb, encap_attr); if (!nest) return -EMSGSIZE; @@ -250,7 +251,7 @@ int lwtunnel_fill_encap(struct sk_buff *skb, struct lwtunnel_state *lwtstate) if (ret) goto nla_put_failure; nla_nest_end(skb, nest); - ret = nla_put_u16(skb, RTA_ENCAP_TYPE, lwtstate->type); + ret = nla_put_u16(skb, encap_type_attr, lwtstate->type); if (ret) goto nla_put_failure; diff --git a/net/core/neighbour.c b/net/core/neighbour.c index 30f6fd8f68e0dc42801686ede3886f366ee1732b..dfa871061f147f4087194ac29e89f2f98a4e462d 100644 --- a/net/core/neighbour.c +++ b/net/core/neighbour.c @@ -31,6 +31,7 @@ #include #include #include +#include #include #include #include @@ -663,6 +664,8 @@ out: out_tbl_unlock: write_unlock_bh(&tbl->lock); out_neigh_release: + if (!exempt_from_gc) + atomic_dec(&tbl->gc_entries); neigh_release(n); goto out; } @@ -1862,7 +1865,8 @@ static int neigh_add(struct sk_buff *skb, struct nlmsghdr *nlh, int err; ASSERT_RTNL(); - err = nlmsg_parse(nlh, sizeof(*ndm), tb, NDA_MAX, nda_policy, extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*ndm), tb, NDA_MAX, + nda_policy, extack); if (err < 0) goto out; @@ -1920,6 +1924,11 @@ static int neigh_add(struct sk_buff *skb, struct nlmsghdr *nlh, goto out; } + if (tbl->allow_add && !tbl->allow_add(dev, extack)) { + err = -EINVAL; + goto out; + } + neigh = neigh_lookup(tbl, dst, dev); if (neigh == NULL) { bool exempt_from_gc; @@ -1974,7 +1983,7 @@ static int neightbl_fill_parms(struct sk_buff *skb, struct neigh_parms *parms) { struct nlattr *nest; - nest = nla_nest_start(skb, NDTA_PARMS); + nest = nla_nest_start_noflag(skb, NDTA_PARMS); if (nest == NULL) return -ENOBUFS; @@ -2176,8 +2185,8 @@ static int neightbl_set(struct sk_buff *skb, struct nlmsghdr *nlh, bool found = false; int err, tidx; - err = nlmsg_parse(nlh, sizeof(*ndtmsg), tb, NDTA_MAX, - nl_neightbl_policy, extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*ndtmsg), tb, NDTA_MAX, + nl_neightbl_policy, extack); if (err < 0) goto errout; @@ -2214,8 +2223,9 @@ static int neightbl_set(struct sk_buff *skb, struct nlmsghdr *nlh, struct neigh_parms *p; int i, ifindex = 0; - err = nla_parse_nested(tbp, NDTPA_MAX, tb[NDTA_PARMS], - nl_ntbl_parm_policy, extack); + err = nla_parse_nested_deprecated(tbp, NDTPA_MAX, + tb[NDTA_PARMS], + nl_ntbl_parm_policy, extack); if (err < 0) goto errout_tbl_lock; @@ -2655,11 +2665,12 @@ static int neigh_valid_dump_req(const struct nlmsghdr *nlh, return -EINVAL; } - err = nlmsg_parse_strict(nlh, sizeof(struct ndmsg), tb, NDA_MAX, - nda_policy, extack); + err = nlmsg_parse_deprecated_strict(nlh, sizeof(struct ndmsg), + tb, NDA_MAX, nda_policy, + extack); } else { - err = nlmsg_parse(nlh, sizeof(struct ndmsg), tb, NDA_MAX, - nda_policy, extack); + err = nlmsg_parse_deprecated(nlh, sizeof(struct ndmsg), tb, + NDA_MAX, nda_policy, extack); } if (err < 0) return err; @@ -2759,8 +2770,8 @@ static int neigh_valid_get_req(const struct nlmsghdr *nlh, return -EINVAL; } - err = nlmsg_parse_strict(nlh, sizeof(struct ndmsg), tb, NDA_MAX, - nda_policy, extack); + err = nlmsg_parse_deprecated_strict(nlh, sizeof(struct ndmsg), tb, + NDA_MAX, nda_policy, extack); if (err < 0) return err; @@ -2982,7 +2993,13 @@ int neigh_xmit(int index, struct net_device *dev, if (!tbl) goto out; rcu_read_lock_bh(); - neigh = __neigh_lookup_noref(tbl, addr, dev); + if (index == NEIGH_ARP_TABLE) { + u32 key = *((u32 *)addr); + + neigh = __ipv4_neigh_lookup_noref(dev, key); + } else { + neigh = __neigh_lookup_noref(tbl, addr, dev); + } if (!neigh) neigh = __neigh_create(tbl, addr, dev, false); err = PTR_ERR(neigh); diff --git a/net/core/net-sysfs.c b/net/core/net-sysfs.c index 530e5b04b97d7a545e441e35cb8a5bffc84be01b..d9c4360257cebb6e1567a77c0da7b64dc4b5c06c 100644 --- a/net/core/net-sysfs.c +++ b/net/core/net-sysfs.c @@ -754,9 +754,9 @@ static ssize_t store_rps_map(struct netdev_rx_queue *queue, rcu_assign_pointer(queue->rps_map, map); if (map) - static_key_slow_inc(&rps_needed); + static_branch_inc(&rps_needed); if (old_map) - static_key_slow_dec(&rps_needed); + static_branch_dec(&rps_needed); mutex_unlock(&rps_map_mutex); diff --git a/net/core/net_namespace.c b/net/core/net_namespace.c index 7e6dcc6257011d8b60e132e97a0db229c39d1daf..711b161505ac1afcc2aa2dd5d07b33b5a2f1ac89 100644 --- a/net/core/net_namespace.c +++ b/net/core/net_namespace.c @@ -682,8 +682,8 @@ static int rtnl_net_newid(struct sk_buff *skb, struct nlmsghdr *nlh, struct net *peer; int nsid, err; - err = nlmsg_parse(nlh, sizeof(struct rtgenmsg), tb, NETNSA_MAX, - rtnl_net_policy, extack); + err = nlmsg_parse_deprecated(nlh, sizeof(struct rtgenmsg), tb, + NETNSA_MAX, rtnl_net_policy, extack); if (err < 0) return err; if (!tb[NETNSA_NSID]) { @@ -787,11 +787,13 @@ static int rtnl_net_valid_getid_req(struct sk_buff *skb, int i, err; if (!netlink_strict_get_check(skb)) - return nlmsg_parse(nlh, sizeof(struct rtgenmsg), tb, NETNSA_MAX, - rtnl_net_policy, extack); + return nlmsg_parse_deprecated(nlh, sizeof(struct rtgenmsg), + tb, NETNSA_MAX, rtnl_net_policy, + extack); - err = nlmsg_parse_strict(nlh, sizeof(struct rtgenmsg), tb, NETNSA_MAX, - rtnl_net_policy, extack); + err = nlmsg_parse_deprecated_strict(nlh, sizeof(struct rtgenmsg), tb, + NETNSA_MAX, rtnl_net_policy, + extack); if (err) return err; @@ -839,7 +841,7 @@ static int rtnl_net_getid(struct sk_buff *skb, struct nlmsghdr *nlh, peer = get_net_ns_by_fd(nla_get_u32(tb[NETNSA_FD])); nla = tb[NETNSA_FD]; } else if (tb[NETNSA_NSID]) { - peer = get_net_ns_by_id(net, nla_get_u32(tb[NETNSA_NSID])); + peer = get_net_ns_by_id(net, nla_get_s32(tb[NETNSA_NSID])); if (!peer) peer = ERR_PTR(-ENOENT); nla = tb[NETNSA_NSID]; @@ -929,8 +931,9 @@ static int rtnl_valid_dump_net_req(const struct nlmsghdr *nlh, struct sock *sk, struct nlattr *tb[NETNSA_MAX + 1]; int err, i; - err = nlmsg_parse_strict(nlh, sizeof(struct rtgenmsg), tb, NETNSA_MAX, - rtnl_net_policy, extack); + err = nlmsg_parse_deprecated_strict(nlh, sizeof(struct rtgenmsg), tb, + NETNSA_MAX, rtnl_net_policy, + extack); if (err < 0) return err; diff --git a/net/core/netpoll.c b/net/core/netpoll.c index bf5446192d6aeef01d0bdb376a732a559dc192af..a0f05416657bbcf52b0e9554540f0a1345cfa28f 100644 --- a/net/core/netpoll.c +++ b/net/core/netpoll.c @@ -323,7 +323,7 @@ void netpoll_send_skb_on_dev(struct netpoll *np, struct sk_buff *skb, if (skb_queue_len(&npinfo->txq) == 0 && !netpoll_owner_active(dev)) { struct netdev_queue *txq; - txq = netdev_pick_tx(dev, skb, NULL); + txq = netdev_core_pick_tx(dev, skb, NULL); /* try until next clock tick */ for (tries = jiffies_to_usecs(1)/USEC_PER_POLL; diff --git a/net/core/netprio_cgroup.c b/net/core/netprio_cgroup.c index b9057478d69c8ad02ea7b4ba8d6f612e7792a738..7e3d0d99dfaefe620aefb298bcedbce32cf41f5c 100644 --- a/net/core/netprio_cgroup.c +++ b/net/core/netprio_cgroup.c @@ -301,6 +301,4 @@ static int __init init_cgroup_netprio(void) register_netdevice_notifier(&netprio_device_notifier); return 0; } - subsys_initcall(init_cgroup_netprio); -MODULE_LICENSE("GPL v2"); diff --git a/net/core/pktgen.c b/net/core/pktgen.c index f3f5a78cd062973197dc3d514f3c701fd63dc20d..319ad5490fb3428205c870e5edaa1dc861b62d5f 100644 --- a/net/core/pktgen.c +++ b/net/core/pktgen.c @@ -2521,7 +2521,7 @@ static int pktgen_output_ipsec(struct sk_buff *skb, struct pktgen_dev *pkt_dev) skb->_skb_refdst = (unsigned long)&pkt_dev->xdst.u.dst | SKB_DST_NOREF; rcu_read_lock_bh(); - err = x->outer_mode->output(x, skb); + err = pktgen_xfrm_outer_mode_output(x, skb); rcu_read_unlock_bh(); if (err) { XFRM_INC_STATS(net, LINUX_MIB_XFRMOUTSTATEMODEERROR); diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c index 220c56e936592495656962050d285bb1c0024b37..2bd12afb92978650acae30d094de82dc6d28eafa 100644 --- a/net/core/rtnetlink.c +++ b/net/core/rtnetlink.c @@ -634,7 +634,7 @@ static int rtnl_link_slave_info_fill(struct sk_buff *skb, if (nla_put_string(skb, IFLA_INFO_SLAVE_KIND, ops->kind) < 0) return -EMSGSIZE; if (ops->fill_slave_info) { - slave_data = nla_nest_start(skb, IFLA_INFO_SLAVE_DATA); + slave_data = nla_nest_start_noflag(skb, IFLA_INFO_SLAVE_DATA); if (!slave_data) return -EMSGSIZE; err = ops->fill_slave_info(skb, master_dev, dev); @@ -666,7 +666,7 @@ static int rtnl_link_info_fill(struct sk_buff *skb, return err; } if (ops->fill_info) { - data = nla_nest_start(skb, IFLA_INFO_DATA); + data = nla_nest_start_noflag(skb, IFLA_INFO_DATA); if (data == NULL) return -EMSGSIZE; err = ops->fill_info(skb, dev); @@ -686,7 +686,7 @@ static int rtnl_link_fill(struct sk_buff *skb, const struct net_device *dev) struct nlattr *linkinfo; int err = -EMSGSIZE; - linkinfo = nla_nest_start(skb, IFLA_LINKINFO); + linkinfo = nla_nest_start_noflag(skb, IFLA_LINKINFO); if (linkinfo == NULL) goto out; @@ -755,7 +755,7 @@ int rtnetlink_put_metrics(struct sk_buff *skb, u32 *metrics) struct nlattr *mx; int i, valid = 0; - mx = nla_nest_start(skb, RTA_METRICS); + mx = nla_nest_start_noflag(skb, RTA_METRICS); if (mx == NULL) return -ENOBUFS; @@ -1036,12 +1036,12 @@ static int rtnl_vf_ports_fill(struct sk_buff *skb, struct net_device *dev) int vf; int err; - vf_ports = nla_nest_start(skb, IFLA_VF_PORTS); + vf_ports = nla_nest_start_noflag(skb, IFLA_VF_PORTS); if (!vf_ports) return -EMSGSIZE; for (vf = 0; vf < dev_num_vf(dev->dev.parent); vf++) { - vf_port = nla_nest_start(skb, IFLA_VF_PORT); + vf_port = nla_nest_start_noflag(skb, IFLA_VF_PORT); if (!vf_port) goto nla_put_failure; if (nla_put_u32(skb, IFLA_PORT_VF, vf)) @@ -1070,7 +1070,7 @@ static int rtnl_port_self_fill(struct sk_buff *skb, struct net_device *dev) struct nlattr *port_self; int err; - port_self = nla_nest_start(skb, IFLA_PORT_SELF); + port_self = nla_nest_start_noflag(skb, IFLA_PORT_SELF); if (!port_self) return -EMSGSIZE; @@ -1247,7 +1247,7 @@ static noinline_for_stack int rtnl_fill_vfinfo(struct sk_buff *skb, vf_linkstate.link_state = ivi.linkstate; vf_rss_query_en.setting = ivi.rss_query_en; vf_trust.setting = ivi.trusted; - vf = nla_nest_start(skb, IFLA_VF_INFO); + vf = nla_nest_start_noflag(skb, IFLA_VF_INFO); if (!vf) goto nla_put_vfinfo_failure; if (nla_put(skb, IFLA_VF_MAC, sizeof(vf_mac), &vf_mac) || @@ -1266,7 +1266,7 @@ static noinline_for_stack int rtnl_fill_vfinfo(struct sk_buff *skb, nla_put(skb, IFLA_VF_TRUST, sizeof(vf_trust), &vf_trust)) goto nla_put_vf_failure; - vfvlanlist = nla_nest_start(skb, IFLA_VF_VLAN_LIST); + vfvlanlist = nla_nest_start_noflag(skb, IFLA_VF_VLAN_LIST); if (!vfvlanlist) goto nla_put_vf_failure; if (nla_put(skb, IFLA_VF_VLAN_INFO, sizeof(vf_vlan_info), @@ -1279,7 +1279,7 @@ static noinline_for_stack int rtnl_fill_vfinfo(struct sk_buff *skb, if (dev->netdev_ops->ndo_get_vf_stats) dev->netdev_ops->ndo_get_vf_stats(dev, vfs_num, &vf_stats); - vfstats = nla_nest_start(skb, IFLA_VF_STATS); + vfstats = nla_nest_start_noflag(skb, IFLA_VF_STATS); if (!vfstats) goto nla_put_vf_failure; if (nla_put_u64_64bit(skb, IFLA_VF_STATS_RX_PACKETS, @@ -1329,7 +1329,7 @@ static noinline_for_stack int rtnl_fill_vf(struct sk_buff *skb, if (!dev->netdev_ops->ndo_get_vf_config) return 0; - vfinfo = nla_nest_start(skb, IFLA_VFINFO_LIST); + vfinfo = nla_nest_start_noflag(skb, IFLA_VFINFO_LIST); if (!vfinfo) return -EMSGSIZE; @@ -1414,7 +1414,7 @@ static int rtnl_xdp_fill(struct sk_buff *skb, struct net_device *dev) int err; u8 mode; - xdp = nla_nest_start(skb, IFLA_XDP); + xdp = nla_nest_start_noflag(skb, IFLA_XDP); if (!xdp) return -EMSGSIZE; @@ -1541,7 +1541,7 @@ static int rtnl_fill_link_af(struct sk_buff *skb, const struct rtnl_af_ops *af_ops; struct nlattr *af_spec; - af_spec = nla_nest_start(skb, IFLA_AF_SPEC); + af_spec = nla_nest_start_noflag(skb, IFLA_AF_SPEC); if (!af_spec) return -EMSGSIZE; @@ -1552,7 +1552,7 @@ static int rtnl_fill_link_af(struct sk_buff *skb, if (!af_ops->fill_link_af) continue; - af = nla_nest_start(skb, af_ops->family); + af = nla_nest_start_noflag(skb, af_ops->family); if (!af) return -EMSGSIZE; @@ -1797,8 +1797,7 @@ static const struct rtnl_link_ops *linkinfo_to_kind_ops(const struct nlattr *nla const struct rtnl_link_ops *ops = NULL; struct nlattr *linfo[IFLA_INFO_MAX + 1]; - if (nla_parse_nested(linfo, IFLA_INFO_MAX, nla, - ifla_info_policy, NULL) < 0) + if (nla_parse_nested_deprecated(linfo, IFLA_INFO_MAX, nla, ifla_info_policy, NULL) < 0) return NULL; if (linfo[IFLA_INFO_KIND]) { @@ -1897,8 +1896,9 @@ static int rtnl_valid_dump_ifinfo_req(const struct nlmsghdr *nlh, return -EINVAL; } - return nlmsg_parse_strict(nlh, sizeof(*ifm), tb, IFLA_MAX, - ifla_policy, extack); + return nlmsg_parse_deprecated_strict(nlh, sizeof(*ifm), tb, + IFLA_MAX, ifla_policy, + extack); } /* A hack to preserve kernel<->userspace interface. @@ -1911,7 +1911,8 @@ static int rtnl_valid_dump_ifinfo_req(const struct nlmsghdr *nlh, hdrlen = nlmsg_len(nlh) < sizeof(struct ifinfomsg) ? sizeof(struct rtgenmsg) : sizeof(struct ifinfomsg); - return nlmsg_parse(nlh, hdrlen, tb, IFLA_MAX, ifla_policy, extack); + return nlmsg_parse_deprecated(nlh, hdrlen, tb, IFLA_MAX, ifla_policy, + extack); } static int rtnl_dump_ifinfo(struct sk_buff *skb, struct netlink_callback *cb) @@ -2019,7 +2020,8 @@ out_err: int rtnl_nla_parse_ifla(struct nlattr **tb, const struct nlattr *head, int len, struct netlink_ext_ack *exterr) { - return nla_parse(tb, IFLA_MAX, head, len, ifla_policy, exterr); + return nla_parse_deprecated(tb, IFLA_MAX, head, len, ifla_policy, + exterr); } EXPORT_SYMBOL(rtnl_nla_parse_ifla); @@ -2564,8 +2566,10 @@ static int do_setlink(const struct sk_buff *skb, err = -EINVAL; goto errout; } - err = nla_parse_nested(vfinfo, IFLA_VF_MAX, attr, - ifla_vf_policy, NULL); + err = nla_parse_nested_deprecated(vfinfo, IFLA_VF_MAX, + attr, + ifla_vf_policy, + NULL); if (err < 0) goto errout; err = do_setvfinfo(dev, vfinfo); @@ -2592,8 +2596,10 @@ static int do_setlink(const struct sk_buff *skb, err = -EINVAL; goto errout; } - err = nla_parse_nested(port, IFLA_PORT_MAX, attr, - ifla_port_policy, NULL); + err = nla_parse_nested_deprecated(port, IFLA_PORT_MAX, + attr, + ifla_port_policy, + NULL); if (err < 0) goto errout; if (!port[IFLA_PORT_VF]) { @@ -2612,9 +2618,9 @@ static int do_setlink(const struct sk_buff *skb, if (tb[IFLA_PORT_SELF]) { struct nlattr *port[IFLA_PORT_MAX+1]; - err = nla_parse_nested(port, IFLA_PORT_MAX, - tb[IFLA_PORT_SELF], ifla_port_policy, - NULL); + err = nla_parse_nested_deprecated(port, IFLA_PORT_MAX, + tb[IFLA_PORT_SELF], + ifla_port_policy, NULL); if (err < 0) goto errout; @@ -2661,8 +2667,9 @@ static int do_setlink(const struct sk_buff *skb, struct nlattr *xdp[IFLA_XDP_MAX + 1]; u32 xdp_flags = 0; - err = nla_parse_nested(xdp, IFLA_XDP_MAX, tb[IFLA_XDP], - ifla_xdp_policy, NULL); + err = nla_parse_nested_deprecated(xdp, IFLA_XDP_MAX, + tb[IFLA_XDP], + ifla_xdp_policy, NULL); if (err < 0) goto errout; @@ -2716,8 +2723,8 @@ static int rtnl_setlink(struct sk_buff *skb, struct nlmsghdr *nlh, struct nlattr *tb[IFLA_MAX+1]; char ifname[IFNAMSIZ]; - err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFLA_MAX, ifla_policy, - extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*ifm), tb, IFLA_MAX, + ifla_policy, extack); if (err < 0) goto errout; @@ -2813,7 +2820,8 @@ static int rtnl_dellink(struct sk_buff *skb, struct nlmsghdr *nlh, int err; int netnsid = -1; - err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFLA_MAX, ifla_policy, extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*ifm), tb, IFLA_MAX, + ifla_policy, extack); if (err < 0) return err; @@ -2990,7 +2998,8 @@ static int __rtnl_newlink(struct sk_buff *skb, struct nlmsghdr *nlh, #ifdef CONFIG_MODULES replay: #endif - err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFLA_MAX, ifla_policy, extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*ifm), tb, IFLA_MAX, + ifla_policy, extack); if (err < 0) return err; @@ -3024,9 +3033,9 @@ replay: return err; if (tb[IFLA_LINKINFO]) { - err = nla_parse_nested(linkinfo, IFLA_INFO_MAX, - tb[IFLA_LINKINFO], ifla_info_policy, - NULL); + err = nla_parse_nested_deprecated(linkinfo, IFLA_INFO_MAX, + tb[IFLA_LINKINFO], + ifla_info_policy, NULL); if (err < 0) return err; } else @@ -3046,9 +3055,9 @@ replay: return -EINVAL; if (ops->maxtype && linkinfo[IFLA_INFO_DATA]) { - err = nla_parse_nested(attr, ops->maxtype, - linkinfo[IFLA_INFO_DATA], - ops->policy, extack); + err = nla_parse_nested_deprecated(attr, ops->maxtype, + linkinfo[IFLA_INFO_DATA], + ops->policy, extack); if (err < 0) return err; data = attr; @@ -3067,9 +3076,11 @@ replay: if (m_ops->slave_maxtype && linkinfo[IFLA_INFO_SLAVE_DATA]) { - err = nla_parse_nested(slave_attr, m_ops->slave_maxtype, - linkinfo[IFLA_INFO_SLAVE_DATA], - m_ops->slave_policy, extack); + err = nla_parse_nested_deprecated(slave_attr, + m_ops->slave_maxtype, + linkinfo[IFLA_INFO_SLAVE_DATA], + m_ops->slave_policy, + extack); if (err < 0) return err; slave_data = slave_attr; @@ -3250,8 +3261,8 @@ static int rtnl_valid_getlink_req(struct sk_buff *skb, } if (!netlink_strict_get_check(skb)) - return nlmsg_parse(nlh, sizeof(*ifm), tb, IFLA_MAX, ifla_policy, - extack); + return nlmsg_parse_deprecated(nlh, sizeof(*ifm), tb, IFLA_MAX, + ifla_policy, extack); ifm = nlmsg_data(nlh); if (ifm->__ifi_pad || ifm->ifi_type || ifm->ifi_flags || @@ -3260,8 +3271,8 @@ static int rtnl_valid_getlink_req(struct sk_buff *skb, return -EINVAL; } - err = nlmsg_parse_strict(nlh, sizeof(*ifm), tb, IFLA_MAX, ifla_policy, - extack); + err = nlmsg_parse_deprecated_strict(nlh, sizeof(*ifm), tb, IFLA_MAX, + ifla_policy, extack); if (err) return err; @@ -3366,7 +3377,7 @@ static u16 rtnl_calcit(struct sk_buff *skb, struct nlmsghdr *nlh) hdrlen = nlmsg_len(nlh) < sizeof(struct ifinfomsg) ? sizeof(struct rtgenmsg) : sizeof(struct ifinfomsg); - if (nlmsg_parse(nlh, hdrlen, tb, IFLA_MAX, ifla_policy, NULL) >= 0) { + if (nlmsg_parse_deprecated(nlh, hdrlen, tb, IFLA_MAX, ifla_policy, NULL) >= 0) { if (tb[IFLA_EXT_MASK]) ext_filter_mask = nla_get_u32(tb[IFLA_EXT_MASK]); } @@ -3569,7 +3580,7 @@ errout: rtnl_set_sk_err(net, RTNLGRP_NEIGH, err); } -/** +/* * ndo_dflt_fdb_add - default netdevice operation to add an FDB entry */ int ndo_dflt_fdb_add(struct ndmsg *ndm, @@ -3639,7 +3650,8 @@ static int rtnl_fdb_add(struct sk_buff *skb, struct nlmsghdr *nlh, u16 vid; int err; - err = nlmsg_parse(nlh, sizeof(*ndm), tb, NDA_MAX, NULL, extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*ndm), tb, NDA_MAX, NULL, + extack); if (err < 0) return err; @@ -3708,7 +3720,7 @@ out: return err; } -/** +/* * ndo_dflt_fdb_del - default netdevice operation to delete an FDB entry */ int ndo_dflt_fdb_del(struct ndmsg *ndm, @@ -3749,7 +3761,8 @@ static int rtnl_fdb_del(struct sk_buff *skb, struct nlmsghdr *nlh, if (!netlink_capable(skb, CAP_NET_ADMIN)) return -EPERM; - err = nlmsg_parse(nlh, sizeof(*ndm), tb, NDA_MAX, NULL, extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*ndm), tb, NDA_MAX, NULL, + extack); if (err < 0) return err; @@ -3847,8 +3860,11 @@ skip: /** * ndo_dflt_fdb_dump - default netdevice operation to dump an FDB table. - * @nlh: netlink message header + * @skb: socket buffer to store message in + * @cb: netlink callback * @dev: netdevice + * @filter_dev: ignored + * @idx: the number of FDB table entries dumped is added to *@idx * * Default netdevice operation to dump the existing unicast address list. * Returns number of addresses from list put in skb. @@ -3895,8 +3911,8 @@ static int valid_fdb_dump_strict(const struct nlmsghdr *nlh, return -EINVAL; } - err = nlmsg_parse_strict(nlh, sizeof(struct ndmsg), tb, NDA_MAX, - NULL, extack); + err = nlmsg_parse_deprecated_strict(nlh, sizeof(struct ndmsg), tb, + NDA_MAX, NULL, extack); if (err < 0) return err; @@ -3948,8 +3964,9 @@ static int valid_fdb_dump_legacy(const struct nlmsghdr *nlh, nla_attr_size(sizeof(u32)))) { struct ifinfomsg *ifm; - err = nlmsg_parse(nlh, sizeof(struct ifinfomsg), tb, IFLA_MAX, - ifla_policy, extack); + err = nlmsg_parse_deprecated(nlh, sizeof(struct ifinfomsg), + tb, IFLA_MAX, ifla_policy, + extack); if (err < 0) { return -EINVAL; } else if (err == 0) { @@ -4088,8 +4105,8 @@ static int valid_fdb_get_strict(const struct nlmsghdr *nlh, return -EINVAL; } - err = nlmsg_parse_strict(nlh, sizeof(struct ndmsg), tb, NDA_MAX, - nda_policy, extack); + err = nlmsg_parse_deprecated_strict(nlh, sizeof(struct ndmsg), tb, + NDA_MAX, nda_policy, extack); if (err < 0) return err; @@ -4270,7 +4287,7 @@ int ndo_dflt_bridge_getlink(struct sk_buff *skb, u32 pid, u32 seq, nla_put_u32(skb, IFLA_LINK, dev_get_iflink(dev)))) goto nla_put_failure; - br_afspec = nla_nest_start(skb, IFLA_AF_SPEC); + br_afspec = nla_nest_start_noflag(skb, IFLA_AF_SPEC); if (!br_afspec) goto nla_put_failure; @@ -4294,7 +4311,7 @@ int ndo_dflt_bridge_getlink(struct sk_buff *skb, u32 pid, u32 seq, } nla_nest_end(skb, br_afspec); - protinfo = nla_nest_start(skb, IFLA_PROTINFO | NLA_F_NESTED); + protinfo = nla_nest_start(skb, IFLA_PROTINFO); if (!protinfo) goto nla_put_failure; @@ -4351,11 +4368,14 @@ static int valid_bridge_getlink_req(const struct nlmsghdr *nlh, return -EINVAL; } - err = nlmsg_parse_strict(nlh, sizeof(struct ifinfomsg), tb, - IFLA_MAX, ifla_policy, extack); + err = nlmsg_parse_deprecated_strict(nlh, + sizeof(struct ifinfomsg), + tb, IFLA_MAX, ifla_policy, + extack); } else { - err = nlmsg_parse(nlh, sizeof(struct ifinfomsg), tb, - IFLA_MAX, ifla_policy, extack); + err = nlmsg_parse_deprecated(nlh, sizeof(struct ifinfomsg), + tb, IFLA_MAX, ifla_policy, + extack); } if (err < 0) return err; @@ -4773,8 +4793,8 @@ static int rtnl_fill_statsinfo(struct sk_buff *skb, struct net_device *dev, if (ops && ops->fill_linkxstats) { *idxattr = IFLA_STATS_LINK_XSTATS; - attr = nla_nest_start(skb, - IFLA_STATS_LINK_XSTATS); + attr = nla_nest_start_noflag(skb, + IFLA_STATS_LINK_XSTATS); if (!attr) goto nla_put_failure; @@ -4796,8 +4816,8 @@ static int rtnl_fill_statsinfo(struct sk_buff *skb, struct net_device *dev, ops = master->rtnl_link_ops; if (ops && ops->fill_linkxstats) { *idxattr = IFLA_STATS_LINK_XSTATS_SLAVE; - attr = nla_nest_start(skb, - IFLA_STATS_LINK_XSTATS_SLAVE); + attr = nla_nest_start_noflag(skb, + IFLA_STATS_LINK_XSTATS_SLAVE); if (!attr) goto nla_put_failure; @@ -4812,7 +4832,8 @@ static int rtnl_fill_statsinfo(struct sk_buff *skb, struct net_device *dev, if (stats_attr_valid(filter_mask, IFLA_STATS_LINK_OFFLOAD_XSTATS, *idxattr)) { *idxattr = IFLA_STATS_LINK_OFFLOAD_XSTATS; - attr = nla_nest_start(skb, IFLA_STATS_LINK_OFFLOAD_XSTATS); + attr = nla_nest_start_noflag(skb, + IFLA_STATS_LINK_OFFLOAD_XSTATS); if (!attr) goto nla_put_failure; @@ -4831,7 +4852,7 @@ static int rtnl_fill_statsinfo(struct sk_buff *skb, struct net_device *dev, struct rtnl_af_ops *af_ops; *idxattr = IFLA_STATS_AF_SPEC; - attr = nla_nest_start(skb, IFLA_STATS_AF_SPEC); + attr = nla_nest_start_noflag(skb, IFLA_STATS_AF_SPEC); if (!attr) goto nla_put_failure; @@ -4841,7 +4862,8 @@ static int rtnl_fill_statsinfo(struct sk_buff *skb, struct net_device *dev, struct nlattr *af; int err; - af = nla_nest_start(skb, af_ops->family); + af = nla_nest_start_noflag(skb, + af_ops->family); if (!af) { rcu_read_unlock(); goto nla_put_failure; diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 40796b8bf820450f5d0cce38986bd29137e2fd05..e89be628269371f8f4e9f4b1ef9df06e31d26a6a 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -77,6 +77,8 @@ #include #include +#include "datagram.h" + struct kmem_cache *skbuff_head_cache __ro_after_init; static struct kmem_cache *skbuff_fclone_cache __ro_after_init; #ifdef CONFIG_SKB_EXTENSIONS @@ -256,6 +258,33 @@ nodata: } EXPORT_SYMBOL(__alloc_skb); +/* Caller must provide SKB that is memset cleared */ +static struct sk_buff *__build_skb_around(struct sk_buff *skb, + void *data, unsigned int frag_size) +{ + struct skb_shared_info *shinfo; + unsigned int size = frag_size ? : ksize(data); + + size -= SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); + + /* Assumes caller memset cleared SKB */ + skb->truesize = SKB_TRUESIZE(size); + refcount_set(&skb->users, 1); + skb->head = data; + skb->data = data; + skb_reset_tail_pointer(skb); + skb->end = skb->tail + size; + skb->mac_header = (typeof(skb->mac_header))~0U; + skb->transport_header = (typeof(skb->transport_header))~0U; + + /* make sure we initialize shinfo sequentially */ + shinfo = skb_shinfo(skb); + memset(shinfo, 0, offsetof(struct skb_shared_info, dataref)); + atomic_set(&shinfo->dataref, 1); + + return skb; +} + /** * __build_skb - build a network buffer * @data: data buffer provided by caller @@ -277,32 +306,15 @@ EXPORT_SYMBOL(__alloc_skb); */ struct sk_buff *__build_skb(void *data, unsigned int frag_size) { - struct skb_shared_info *shinfo; struct sk_buff *skb; - unsigned int size = frag_size ? : ksize(data); skb = kmem_cache_alloc(skbuff_head_cache, GFP_ATOMIC); - if (!skb) + if (unlikely(!skb)) return NULL; - size -= SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); - memset(skb, 0, offsetof(struct sk_buff, tail)); - skb->truesize = SKB_TRUESIZE(size); - refcount_set(&skb->users, 1); - skb->head = data; - skb->data = data; - skb_reset_tail_pointer(skb); - skb->end = skb->tail + size; - skb->mac_header = (typeof(skb->mac_header))~0U; - skb->transport_header = (typeof(skb->transport_header))~0U; - - /* make sure we initialize shinfo sequentially */ - shinfo = skb_shinfo(skb); - memset(shinfo, 0, offsetof(struct skb_shared_info, dataref)); - atomic_set(&shinfo->dataref, 1); - return skb; + return __build_skb_around(skb, data, frag_size); } /* build_skb() is wrapper over __build_skb(), that specifically @@ -323,6 +335,29 @@ struct sk_buff *build_skb(void *data, unsigned int frag_size) } EXPORT_SYMBOL(build_skb); +/** + * build_skb_around - build a network buffer around provided skb + * @skb: sk_buff provide by caller, must be memset cleared + * @data: data buffer provided by caller + * @frag_size: size of data, or 0 if head was kmalloced + */ +struct sk_buff *build_skb_around(struct sk_buff *skb, + void *data, unsigned int frag_size) +{ + if (unlikely(!skb)) + return NULL; + + skb = __build_skb_around(skb, data, frag_size); + + if (skb && frag_size) { + skb->head_frag = 1; + if (page_is_pfmemalloc(virt_to_head_page(data))) + skb->pfmemalloc = 1; + } + return skb; +} +EXPORT_SYMBOL(build_skb_around); + #define NAPI_SKB_CACHE_SIZE 64 struct napi_alloc_cache { @@ -1105,9 +1140,6 @@ void sock_zerocopy_put_abort(struct ubuf_info *uarg, bool have_uref) } EXPORT_SYMBOL_GPL(sock_zerocopy_put_abort); -extern int __zerocopy_sg_from_iter(struct sock *sk, struct sk_buff *skb, - struct iov_iter *from, size_t length); - int skb_zerocopy_iter_dgram(struct sk_buff *skb, struct msghdr *msg, int len) { return __zerocopy_sg_from_iter(skb->sk, skb, &msg->msg_iter, len); diff --git a/net/core/sock.c b/net/core/sock.c index 067878a1e4c51363e065e13ccdb2b9d03c6a9c5f..75b1c950b49fde74187835bdb3b4952b43ba19af 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -137,6 +137,7 @@ #include #include +#include #include @@ -1709,6 +1710,10 @@ static void __sk_destruct(struct rcu_head *head) sock_disable_timestamp(sk, SK_FLAGS_TIMESTAMP); +#ifdef CONFIG_BPF_SYSCALL + bpf_sk_storage_free(sk); +#endif + if (atomic_read(&sk->sk_omem_alloc)) pr_debug("%s: optmem leakage (%d bytes) detected\n", __func__, atomic_read(&sk->sk_omem_alloc)); @@ -2977,39 +2982,44 @@ bool lock_sock_fast(struct sock *sk) } EXPORT_SYMBOL(lock_sock_fast); -int sock_get_timestamp(struct sock *sk, struct timeval __user *userstamp) +int sock_gettstamp(struct socket *sock, void __user *userstamp, + bool timeval, bool time32) { - struct timeval tv; + struct sock *sk = sock->sk; + struct timespec64 ts; sock_enable_timestamp(sk, SOCK_TIMESTAMP); - tv = ktime_to_timeval(sock_read_timestamp(sk)); - if (tv.tv_sec == -1) + ts = ktime_to_timespec64(sock_read_timestamp(sk)); + if (ts.tv_sec == -1) return -ENOENT; - if (tv.tv_sec == 0) { + if (ts.tv_sec == 0) { ktime_t kt = ktime_get_real(); - sock_write_timestamp(sk, kt); - tv = ktime_to_timeval(kt); + sock_write_timestamp(sk, kt);; + ts = ktime_to_timespec64(kt); } - return copy_to_user(userstamp, &tv, sizeof(tv)) ? -EFAULT : 0; -} -EXPORT_SYMBOL(sock_get_timestamp); -int sock_get_timestampns(struct sock *sk, struct timespec __user *userstamp) -{ - struct timespec ts; + if (timeval) + ts.tv_nsec /= 1000; - sock_enable_timestamp(sk, SOCK_TIMESTAMP); - ts = ktime_to_timespec(sock_read_timestamp(sk)); - if (ts.tv_sec == -1) - return -ENOENT; - if (ts.tv_sec == 0) { - ktime_t kt = ktime_get_real(); - sock_write_timestamp(sk, kt); - ts = ktime_to_timespec(sk->sk_stamp); +#ifdef CONFIG_COMPAT_32BIT_TIME + if (time32) + return put_old_timespec32(&ts, userstamp); +#endif +#ifdef CONFIG_SPARC64 + /* beware of padding in sparc64 timeval */ + if (timeval && !in_compat_syscall()) { + struct __kernel_old_timeval __user tv = { + .tv_sec = ts.tv_sec, + .tv_usec = ts.tv_nsec, + }; + if (copy_to_user(userstamp, &tv, sizeof(tv))) + return -EFAULT; + return 0; } - return copy_to_user(userstamp, &ts, sizeof(ts)) ? -EFAULT : 0; +#endif + return put_timespec64(&ts, userstamp); } -EXPORT_SYMBOL(sock_get_timestampns); +EXPORT_SYMBOL(sock_gettstamp); void sock_enable_timestamp(struct sock *sk, int flag) { diff --git a/net/core/sock_reuseport.c b/net/core/sock_reuseport.c index d8fe3e549373d92524561728f375b2ce73bf0db7..dc4aefdf2a084b0ec46cc14415dcfda00a9aaa3d 100644 --- a/net/core/sock_reuseport.c +++ b/net/core/sock_reuseport.c @@ -144,6 +144,8 @@ static void reuseport_free_rcu(struct rcu_head *head) * reuseport_add_sock - Add a socket to the reuseport group of another. * @sk: New socket to add to the group. * @sk2: Socket belonging to the existing reuseport group. + * @bind_inany: Whether or not the group is bound to a local INANY address. + * * May return ENOMEM and not add socket to group under memory pressure. */ int reuseport_add_sock(struct sock *sk, struct sock *sk2, bool bind_inany) diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index 84bf2861f45f76f162d661298991f13ac0e8b592..1a2685694abd537d7ae304754b84b237928fd298 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -95,12 +95,12 @@ static int rps_sock_flow_sysctl(struct ctl_table *table, int write, if (sock_table != orig_sock_table) { rcu_assign_pointer(rps_sock_flow_table, sock_table); if (sock_table) { - static_key_slow_inc(&rps_needed); - static_key_slow_inc(&rfs_needed); + static_branch_inc(&rps_needed); + static_branch_inc(&rfs_needed); } if (orig_sock_table) { - static_key_slow_dec(&rps_needed); - static_key_slow_dec(&rfs_needed); + static_branch_dec(&rps_needed); + static_branch_dec(&rfs_needed); synchronize_rcu(); vfree(orig_sock_table); } diff --git a/net/dcb/dcbnl.c b/net/dcb/dcbnl.c index a556cd708885a798ba13e1cf8443c8cf96251e28..ceff9d22deea3f7ebe000a9b6dcb0393f0af772d 100644 --- a/net/dcb/dcbnl.c +++ b/net/dcb/dcbnl.c @@ -241,12 +241,13 @@ static int dcbnl_getpfccfg(struct net_device *netdev, struct nlmsghdr *nlh, if (!netdev->dcbnl_ops->getpfccfg) return -EOPNOTSUPP; - ret = nla_parse_nested(data, DCB_PFC_UP_ATTR_MAX, - tb[DCB_ATTR_PFC_CFG], dcbnl_pfc_up_nest, NULL); + ret = nla_parse_nested_deprecated(data, DCB_PFC_UP_ATTR_MAX, + tb[DCB_ATTR_PFC_CFG], + dcbnl_pfc_up_nest, NULL); if (ret) return ret; - nest = nla_nest_start(skb, DCB_ATTR_PFC_CFG); + nest = nla_nest_start_noflag(skb, DCB_ATTR_PFC_CFG); if (!nest) return -EMSGSIZE; @@ -299,12 +300,13 @@ static int dcbnl_getcap(struct net_device *netdev, struct nlmsghdr *nlh, if (!netdev->dcbnl_ops->getcap) return -EOPNOTSUPP; - ret = nla_parse_nested(data, DCB_CAP_ATTR_MAX, tb[DCB_ATTR_CAP], - dcbnl_cap_nest, NULL); + ret = nla_parse_nested_deprecated(data, DCB_CAP_ATTR_MAX, + tb[DCB_ATTR_CAP], dcbnl_cap_nest, + NULL); if (ret) return ret; - nest = nla_nest_start(skb, DCB_ATTR_CAP); + nest = nla_nest_start_noflag(skb, DCB_ATTR_CAP); if (!nest) return -EMSGSIZE; @@ -343,12 +345,13 @@ static int dcbnl_getnumtcs(struct net_device *netdev, struct nlmsghdr *nlh, if (!netdev->dcbnl_ops->getnumtcs) return -EOPNOTSUPP; - ret = nla_parse_nested(data, DCB_NUMTCS_ATTR_MAX, tb[DCB_ATTR_NUMTCS], - dcbnl_numtcs_nest, NULL); + ret = nla_parse_nested_deprecated(data, DCB_NUMTCS_ATTR_MAX, + tb[DCB_ATTR_NUMTCS], + dcbnl_numtcs_nest, NULL); if (ret) return ret; - nest = nla_nest_start(skb, DCB_ATTR_NUMTCS); + nest = nla_nest_start_noflag(skb, DCB_ATTR_NUMTCS); if (!nest) return -EMSGSIZE; @@ -388,8 +391,9 @@ static int dcbnl_setnumtcs(struct net_device *netdev, struct nlmsghdr *nlh, if (!netdev->dcbnl_ops->setnumtcs) return -EOPNOTSUPP; - ret = nla_parse_nested(data, DCB_NUMTCS_ATTR_MAX, tb[DCB_ATTR_NUMTCS], - dcbnl_numtcs_nest, NULL); + ret = nla_parse_nested_deprecated(data, DCB_NUMTCS_ATTR_MAX, + tb[DCB_ATTR_NUMTCS], + dcbnl_numtcs_nest, NULL); if (ret) return ret; @@ -447,8 +451,9 @@ static int dcbnl_getapp(struct net_device *netdev, struct nlmsghdr *nlh, if (!tb[DCB_ATTR_APP]) return -EINVAL; - ret = nla_parse_nested(app_tb, DCB_APP_ATTR_MAX, tb[DCB_ATTR_APP], - dcbnl_app_nest, NULL); + ret = nla_parse_nested_deprecated(app_tb, DCB_APP_ATTR_MAX, + tb[DCB_ATTR_APP], dcbnl_app_nest, + NULL); if (ret) return ret; @@ -479,7 +484,7 @@ static int dcbnl_getapp(struct net_device *netdev, struct nlmsghdr *nlh, up = dcb_getapp(netdev, &app); } - app_nest = nla_nest_start(skb, DCB_ATTR_APP); + app_nest = nla_nest_start_noflag(skb, DCB_ATTR_APP); if (!app_nest) return -EMSGSIZE; @@ -515,8 +520,9 @@ static int dcbnl_setapp(struct net_device *netdev, struct nlmsghdr *nlh, if (!tb[DCB_ATTR_APP]) return -EINVAL; - ret = nla_parse_nested(app_tb, DCB_APP_ATTR_MAX, tb[DCB_ATTR_APP], - dcbnl_app_nest, NULL); + ret = nla_parse_nested_deprecated(app_tb, DCB_APP_ATTR_MAX, + tb[DCB_ATTR_APP], dcbnl_app_nest, + NULL); if (ret) return ret; @@ -573,12 +579,13 @@ static int __dcbnl_pg_getcfg(struct net_device *netdev, struct nlmsghdr *nlh, !netdev->dcbnl_ops->getpgbwgcfgrx) return -EOPNOTSUPP; - ret = nla_parse_nested(pg_tb, DCB_PG_ATTR_MAX, tb[DCB_ATTR_PG_CFG], - dcbnl_pg_nest, NULL); + ret = nla_parse_nested_deprecated(pg_tb, DCB_PG_ATTR_MAX, + tb[DCB_ATTR_PG_CFG], dcbnl_pg_nest, + NULL); if (ret) return ret; - pg_nest = nla_nest_start(skb, DCB_ATTR_PG_CFG); + pg_nest = nla_nest_start_noflag(skb, DCB_ATTR_PG_CFG); if (!pg_nest) return -EMSGSIZE; @@ -593,12 +600,13 @@ static int __dcbnl_pg_getcfg(struct net_device *netdev, struct nlmsghdr *nlh, data = pg_tb[DCB_PG_ATTR_TC_ALL]; else data = pg_tb[i]; - ret = nla_parse_nested(param_tb, DCB_TC_ATTR_PARAM_MAX, data, - dcbnl_tc_param_nest, NULL); + ret = nla_parse_nested_deprecated(param_tb, + DCB_TC_ATTR_PARAM_MAX, data, + dcbnl_tc_param_nest, NULL); if (ret) goto err_pg; - param_nest = nla_nest_start(skb, i); + param_nest = nla_nest_start_noflag(skb, i); if (!param_nest) goto err_pg; @@ -730,8 +738,9 @@ static int dcbnl_setpfccfg(struct net_device *netdev, struct nlmsghdr *nlh, if (!netdev->dcbnl_ops->setpfccfg) return -EOPNOTSUPP; - ret = nla_parse_nested(data, DCB_PFC_UP_ATTR_MAX, - tb[DCB_ATTR_PFC_CFG], dcbnl_pfc_up_nest, NULL); + ret = nla_parse_nested_deprecated(data, DCB_PFC_UP_ATTR_MAX, + tb[DCB_ATTR_PFC_CFG], + dcbnl_pfc_up_nest, NULL); if (ret) return ret; @@ -786,8 +795,9 @@ static int __dcbnl_pg_setcfg(struct net_device *netdev, struct nlmsghdr *nlh, !netdev->dcbnl_ops->setpgbwgcfgrx) return -EOPNOTSUPP; - ret = nla_parse_nested(pg_tb, DCB_PG_ATTR_MAX, tb[DCB_ATTR_PG_CFG], - dcbnl_pg_nest, NULL); + ret = nla_parse_nested_deprecated(pg_tb, DCB_PG_ATTR_MAX, + tb[DCB_ATTR_PG_CFG], dcbnl_pg_nest, + NULL); if (ret) return ret; @@ -795,8 +805,10 @@ static int __dcbnl_pg_setcfg(struct net_device *netdev, struct nlmsghdr *nlh, if (!pg_tb[i]) continue; - ret = nla_parse_nested(param_tb, DCB_TC_ATTR_PARAM_MAX, - pg_tb[i], dcbnl_tc_param_nest, NULL); + ret = nla_parse_nested_deprecated(param_tb, + DCB_TC_ATTR_PARAM_MAX, + pg_tb[i], + dcbnl_tc_param_nest, NULL); if (ret) return ret; @@ -884,12 +896,13 @@ static int dcbnl_bcn_getcfg(struct net_device *netdev, struct nlmsghdr *nlh, !netdev->dcbnl_ops->getbcncfg) return -EOPNOTSUPP; - ret = nla_parse_nested(bcn_tb, DCB_BCN_ATTR_MAX, tb[DCB_ATTR_BCN], - dcbnl_bcn_nest, NULL); + ret = nla_parse_nested_deprecated(bcn_tb, DCB_BCN_ATTR_MAX, + tb[DCB_ATTR_BCN], dcbnl_bcn_nest, + NULL); if (ret) return ret; - bcn_nest = nla_nest_start(skb, DCB_ATTR_BCN); + bcn_nest = nla_nest_start_noflag(skb, DCB_ATTR_BCN); if (!bcn_nest) return -EMSGSIZE; @@ -943,8 +956,9 @@ static int dcbnl_bcn_setcfg(struct net_device *netdev, struct nlmsghdr *nlh, !netdev->dcbnl_ops->setbcnrp) return -EOPNOTSUPP; - ret = nla_parse_nested(data, DCB_BCN_ATTR_MAX, tb[DCB_ATTR_BCN], - dcbnl_pfc_up_nest, NULL); + ret = nla_parse_nested_deprecated(data, DCB_BCN_ATTR_MAX, + tb[DCB_ATTR_BCN], dcbnl_pfc_up_nest, + NULL); if (ret) return ret; @@ -1002,7 +1016,7 @@ static int dcbnl_build_peer_app(struct net_device *netdev, struct sk_buff* skb, */ err = -EMSGSIZE; - app = nla_nest_start(skb, app_nested_type); + app = nla_nest_start_noflag(skb, app_nested_type); if (!app) goto nla_put_failure; @@ -1036,7 +1050,7 @@ static int dcbnl_ieee_fill(struct sk_buff *skb, struct net_device *netdev) if (nla_put_string(skb, DCB_ATTR_IFNAME, netdev->name)) return -EMSGSIZE; - ieee = nla_nest_start(skb, DCB_ATTR_IEEE); + ieee = nla_nest_start_noflag(skb, DCB_ATTR_IEEE); if (!ieee) return -EMSGSIZE; @@ -1106,7 +1120,7 @@ static int dcbnl_ieee_fill(struct sk_buff *skb, struct net_device *netdev) return -EMSGSIZE; } - app = nla_nest_start(skb, DCB_ATTR_IEEE_APP_TABLE); + app = nla_nest_start_noflag(skb, DCB_ATTR_IEEE_APP_TABLE); if (!app) return -EMSGSIZE; @@ -1174,13 +1188,13 @@ static int dcbnl_cee_pg_fill(struct sk_buff *skb, struct net_device *dev, u8 pgid, up_map, prio, tc_pct; const struct dcbnl_rtnl_ops *ops = dev->dcbnl_ops; int i = dir ? DCB_ATTR_CEE_TX_PG : DCB_ATTR_CEE_RX_PG; - struct nlattr *pg = nla_nest_start(skb, i); + struct nlattr *pg = nla_nest_start_noflag(skb, i); if (!pg) return -EMSGSIZE; for (i = DCB_PG_ATTR_TC_0; i <= DCB_PG_ATTR_TC_7; i++) { - struct nlattr *tc_nest = nla_nest_start(skb, i); + struct nlattr *tc_nest = nla_nest_start_noflag(skb, i); if (!tc_nest) return -EMSGSIZE; @@ -1231,7 +1245,7 @@ static int dcbnl_cee_fill(struct sk_buff *skb, struct net_device *netdev) if (nla_put_string(skb, DCB_ATTR_IFNAME, netdev->name)) goto nla_put_failure; - cee = nla_nest_start(skb, DCB_ATTR_CEE); + cee = nla_nest_start_noflag(skb, DCB_ATTR_CEE); if (!cee) goto nla_put_failure; @@ -1250,7 +1264,8 @@ static int dcbnl_cee_fill(struct sk_buff *skb, struct net_device *netdev) /* local pfc */ if (ops->getpfccfg) { - struct nlattr *pfc_nest = nla_nest_start(skb, DCB_ATTR_CEE_PFC); + struct nlattr *pfc_nest = nla_nest_start_noflag(skb, + DCB_ATTR_CEE_PFC); if (!pfc_nest) goto nla_put_failure; @@ -1265,14 +1280,14 @@ static int dcbnl_cee_fill(struct sk_buff *skb, struct net_device *netdev) /* local app */ spin_lock_bh(&dcb_lock); - app = nla_nest_start(skb, DCB_ATTR_CEE_APP_TABLE); + app = nla_nest_start_noflag(skb, DCB_ATTR_CEE_APP_TABLE); if (!app) goto dcb_unlock; list_for_each_entry(itr, &dcb_app_list, list) { if (itr->ifindex == netdev->ifindex) { - struct nlattr *app_nest = nla_nest_start(skb, - DCB_ATTR_APP); + struct nlattr *app_nest = nla_nest_start_noflag(skb, + DCB_ATTR_APP); if (!app_nest) goto dcb_unlock; @@ -1305,7 +1320,8 @@ static int dcbnl_cee_fill(struct sk_buff *skb, struct net_device *netdev) /* features flags */ if (ops->getfeatcfg) { - struct nlattr *feat = nla_nest_start(skb, DCB_ATTR_CEE_FEAT); + struct nlattr *feat = nla_nest_start_noflag(skb, + DCB_ATTR_CEE_FEAT); if (!feat) goto nla_put_failure; @@ -1429,8 +1445,9 @@ static int dcbnl_ieee_set(struct net_device *netdev, struct nlmsghdr *nlh, if (!tb[DCB_ATTR_IEEE]) return -EINVAL; - err = nla_parse_nested(ieee, DCB_ATTR_IEEE_MAX, tb[DCB_ATTR_IEEE], - dcbnl_ieee_policy, NULL); + err = nla_parse_nested_deprecated(ieee, DCB_ATTR_IEEE_MAX, + tb[DCB_ATTR_IEEE], + dcbnl_ieee_policy, NULL); if (err) return err; @@ -1529,8 +1546,9 @@ static int dcbnl_ieee_del(struct net_device *netdev, struct nlmsghdr *nlh, if (!tb[DCB_ATTR_IEEE]) return -EINVAL; - err = nla_parse_nested(ieee, DCB_ATTR_IEEE_MAX, tb[DCB_ATTR_IEEE], - dcbnl_ieee_policy, NULL); + err = nla_parse_nested_deprecated(ieee, DCB_ATTR_IEEE_MAX, + tb[DCB_ATTR_IEEE], + dcbnl_ieee_policy, NULL); if (err) return err; @@ -1602,12 +1620,13 @@ static int dcbnl_getfeatcfg(struct net_device *netdev, struct nlmsghdr *nlh, if (!tb[DCB_ATTR_FEATCFG]) return -EINVAL; - ret = nla_parse_nested(data, DCB_FEATCFG_ATTR_MAX, - tb[DCB_ATTR_FEATCFG], dcbnl_featcfg_nest, NULL); + ret = nla_parse_nested_deprecated(data, DCB_FEATCFG_ATTR_MAX, + tb[DCB_ATTR_FEATCFG], + dcbnl_featcfg_nest, NULL); if (ret) return ret; - nest = nla_nest_start(skb, DCB_ATTR_FEATCFG); + nest = nla_nest_start_noflag(skb, DCB_ATTR_FEATCFG); if (!nest) return -EMSGSIZE; @@ -1646,8 +1665,9 @@ static int dcbnl_setfeatcfg(struct net_device *netdev, struct nlmsghdr *nlh, if (!tb[DCB_ATTR_FEATCFG]) return -EINVAL; - ret = nla_parse_nested(data, DCB_FEATCFG_ATTR_MAX, - tb[DCB_ATTR_FEATCFG], dcbnl_featcfg_nest, NULL); + ret = nla_parse_nested_deprecated(data, DCB_FEATCFG_ATTR_MAX, + tb[DCB_ATTR_FEATCFG], + dcbnl_featcfg_nest, NULL); if (ret) goto err; @@ -1736,8 +1756,8 @@ static int dcb_doit(struct sk_buff *skb, struct nlmsghdr *nlh, if ((nlh->nlmsg_type == RTM_SETDCB) && !netlink_capable(skb, CAP_NET_ADMIN)) return -EPERM; - ret = nlmsg_parse(nlh, sizeof(*dcb), tb, DCB_ATTR_MAX, - dcbnl_rtnl_policy, extack); + ret = nlmsg_parse_deprecated(nlh, sizeof(*dcb), tb, DCB_ATTR_MAX, + dcbnl_rtnl_policy, extack); if (ret < 0) return ret; diff --git a/net/dccp/ipv4.c b/net/dccp/ipv4.c index 26a21d97b6b078558b125134ebe326bf27dc08d6..004535e4c070879288c8605592689d54f99d9fc1 100644 --- a/net/dccp/ipv4.c +++ b/net/dccp/ipv4.c @@ -991,6 +991,7 @@ static const struct proto_ops inet_dccp_ops = { /* FIXME: work on tcp_poll to rename it to inet_csk_poll */ .poll = dccp_poll, .ioctl = inet_ioctl, + .gettstamp = sock_gettstamp, /* FIXME: work on inet_listen to rename it to sock_common_listen */ .listen = inet_dccp_listen, .shutdown = inet_shutdown, diff --git a/net/dccp/ipv6.c b/net/dccp/ipv6.c index 57d84e9b7b6fc820a4616e3ac326633f14e2fd95..c4e4d13010623bd1177527b4fc8b10c504ff8692 100644 --- a/net/dccp/ipv6.c +++ b/net/dccp/ipv6.c @@ -1075,6 +1075,7 @@ static const struct proto_ops inet6_dccp_ops = { .getname = inet6_getname, .poll = dccp_poll, .ioctl = inet6_ioctl, + .gettstamp = sock_gettstamp, .listen = inet_dccp_listen, .shutdown = inet_shutdown, .setsockopt = sock_common_setsockopt, diff --git a/net/decnet/af_decnet.c b/net/decnet/af_decnet.c index bdccc46a2921924cd948d4fdc5ccbd04ef63f5a7..c1fa4785c4c255a04023e35e88eb2e15c96f6a3d 100644 --- a/net/decnet/af_decnet.c +++ b/net/decnet/af_decnet.c @@ -444,7 +444,7 @@ static void dn_destruct(struct sock *sk) skb_queue_purge(&scp->other_xmit_queue); skb_queue_purge(&scp->other_receive_queue); - dst_release(rcu_dereference_check(sk->sk_dst_cache, 1)); + dst_release(rcu_dereference_protected(sk->sk_dst_cache, 1)); } static unsigned long dn_memory_pressure; diff --git a/net/decnet/dn_dev.c b/net/decnet/dn_dev.c index 0962f9201baa14854be26a44fc5c9d807d700295..cca7ae712995918bfae744d49677e24c9d3085ab 100644 --- a/net/decnet/dn_dev.c +++ b/net/decnet/dn_dev.c @@ -583,8 +583,8 @@ static int dn_nl_deladdr(struct sk_buff *skb, struct nlmsghdr *nlh, if (!net_eq(net, &init_net)) goto errout; - err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, dn_ifa_policy, - extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*ifm), tb, IFA_MAX, + dn_ifa_policy, extack); if (err < 0) goto errout; @@ -629,8 +629,8 @@ static int dn_nl_newaddr(struct sk_buff *skb, struct nlmsghdr *nlh, if (!net_eq(net, &init_net)) return -EINVAL; - err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, dn_ifa_policy, - extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*ifm), tb, IFA_MAX, + dn_ifa_policy, extack); if (err < 0) return err; diff --git a/net/decnet/dn_fib.c b/net/decnet/dn_fib.c index 6cd3737593a6b63c01b8e2bc2096e9924d15d978..77fbf8e9df4b7aba28ef772f9fbe30762de024d8 100644 --- a/net/decnet/dn_fib.c +++ b/net/decnet/dn_fib.c @@ -42,7 +42,7 @@ #include #include #include -#include +#include #define RT_MIN_TABLE 1 @@ -517,8 +517,8 @@ static int dn_fib_rtm_delroute(struct sk_buff *skb, struct nlmsghdr *nlh, if (!net_eq(net, &init_net)) return -EINVAL; - err = nlmsg_parse(nlh, sizeof(*r), attrs, RTA_MAX, rtm_dn_policy, - extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*r), attrs, RTA_MAX, + rtm_dn_policy, extack); if (err < 0) return err; @@ -544,8 +544,8 @@ static int dn_fib_rtm_newroute(struct sk_buff *skb, struct nlmsghdr *nlh, if (!net_eq(net, &init_net)) return -EINVAL; - err = nlmsg_parse(nlh, sizeof(*r), attrs, RTA_MAX, rtm_dn_policy, - extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*r), attrs, RTA_MAX, + rtm_dn_policy, extack); if (err < 0) return err; diff --git a/net/decnet/dn_route.c b/net/decnet/dn_route.c index 950613ee7881bc22ee0ebb106dbbacc4c35b7172..664584763c367174a6c4ac3769bd9fd9571009b9 100644 --- a/net/decnet/dn_route.c +++ b/net/decnet/dn_route.c @@ -1651,8 +1651,8 @@ static int dn_cache_getroute(struct sk_buff *in_skb, struct nlmsghdr *nlh, if (!net_eq(net, &init_net)) return -EINVAL; - err = nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, rtm_dn_policy, - extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*rtm), tb, RTA_MAX, + rtm_dn_policy, extack); if (err < 0) return err; diff --git a/net/decnet/dn_table.c b/net/decnet/dn_table.c index f0710b5d037d09d5e2e48cd842e7c999a9061fc7..33fefb0aebca4687a3d119f70ba56a587d238b17 100644 --- a/net/decnet/dn_table.c +++ b/net/decnet/dn_table.c @@ -348,7 +348,8 @@ static int dn_fib_dump_info(struct sk_buff *skb, u32 portid, u32 seq, int event, struct rtnexthop *nhp; struct nlattr *mp_head; - if (!(mp_head = nla_nest_start(skb, RTA_MULTIPATH))) + mp_head = nla_nest_start_noflag(skb, RTA_MULTIPATH); + if (!mp_head) goto errout; for_nexthops(fi) { diff --git a/net/dns_resolver/dns_query.c b/net/dns_resolver/dns_query.c index 76338c38738a748aeaa899ebf30247c0f69741a8..19aa32fc1802b44a971eecd6f7b9f967ec2900d9 100644 --- a/net/dns_resolver/dns_query.c +++ b/net/dns_resolver/dns_query.c @@ -94,8 +94,6 @@ int dns_query(const char *type, const char *name, size_t namelen, desclen += typelen + 1; } - if (!namelen) - namelen = strnlen(name, 256); if (namelen < 3 || namelen > 255) return -EINVAL; desclen += namelen + 1; diff --git a/net/dsa/Kconfig b/net/dsa/Kconfig index fab49132345f1e7a57c76e18f356e83578d69871..cf855352a440eda113c23fc0ad1eafabebbbb4f9 100644 --- a/net/dsa/Kconfig +++ b/net/dsa/Kconfig @@ -4,60 +4,117 @@ config HAVE_NET_DSA # Drivers must select NET_DSA and the appropriate tagging format -config NET_DSA +menuconfig NET_DSA tristate "Distributed Switch Architecture" depends on HAVE_NET_DSA depends on BRIDGE || BRIDGE=n select NET_SWITCHDEV select PHYLINK + select NET_DEVLINK ---help--- Say Y if you want to enable support for the hardware switches supported by the Distributed Switch Architecture. if NET_DSA -config NET_DSA_LEGACY - bool "Support for older platform device and Device Tree registration" - default y - ---help--- - Say Y if you want to enable support for the older platform device and - deprecated Device Tree binding registration. +# tagging formats +config NET_DSA_TAG_8021Q + tristate "Tag driver for switches using custom 802.1Q VLAN headers" + select VLAN_8021Q + help + Unlike the other tagging protocols, the 802.1Q config option simply + provides helpers for other tagging implementations that might rely on + VLAN in one way or another. It is not a complete solution. - This feature is scheduled for removal in 4.17. + Drivers which use these helpers should select this as dependency. + +config NET_DSA_TAG_BRCM_COMMON + tristate + default n -# tagging formats config NET_DSA_TAG_BRCM - bool + tristate "Tag driver for Broadcom switches using in-frame headers" + select NET_DSA_TAG_BRCM_COMMON + help + Say Y if you want to enable support for tagging frames for the + Broadcom switches which place the tag after the MAC source address. + config NET_DSA_TAG_BRCM_PREPEND - bool + tristate "Tag driver for Broadcom switches using prepended headers" + select NET_DSA_TAG_BRCM_COMMON + help + Say Y if you want to enable support for tagging frames for the + Broadcom switches which places the tag before the Ethernet header + (prepended). + +config NET_DSA_TAG_GSWIP + tristate "Tag driver for Lantiq / Intel GSWIP switches" + help + Say Y or M if you want to enable support for tagging frames for the + Lantiq / Intel GSWIP switches. config NET_DSA_TAG_DSA - bool + tristate "Tag driver for Marvell switches using DSA headers" + help + Say Y or M if you want to enable support for tagging frames for the + Marvell switches which use DSA headers. config NET_DSA_TAG_EDSA - bool + tristate "Tag driver for Marvell switches using EtherType DSA headers" + help + Say Y or M if you want to enable support for tagging frames for the + Marvell switches which use EtherType DSA headers. -config NET_DSA_TAG_GSWIP - bool +config NET_DSA_TAG_MTK + tristate "Tag driver for Mediatek switches" + help + Say Y or M if you want to enable support for tagging frames for + Mediatek switches. + +config NET_DSA_TAG_KSZ_COMMON + tristate + default n config NET_DSA_TAG_KSZ - bool + tristate "Tag driver for Microchip 9893 family of switches" + select NET_DSA_TAG_KSZ_COMMON + help + Say Y if you want to enable support for tagging frames for the + Microchip 9893 family of switches. config NET_DSA_TAG_KSZ9477 - bool - select NET_DSA_TAG_KSZ + tristate "Tag driver for Microchip 9477 family of switches" + select NET_DSA_TAG_KSZ_COMMON + help + Say Y if you want to enable support for tagging frames for the + Microchip 9477 family of switches. -config NET_DSA_TAG_LAN9303 - bool +config NET_DSA_TAG_QCA + tristate "Tag driver for Qualcomm Atheros QCA8K switches" + help + Say Y or M if you want to enable support for tagging frames for + the Qualcomm Atheros QCA8K switches. -config NET_DSA_TAG_MTK - bool +config NET_DSA_TAG_LAN9303 + tristate "Tag driver for SMSC/Microchip LAN9303 family of switches" + help + Say Y or M if you want to enable support for tagging frames for the + SMSC/Microchip LAN9303 family of switches. + +config NET_DSA_TAG_SJA1105 + tristate "Tag driver for NXP SJA1105 switches" + select NET_DSA_TAG_8021Q + help + Say Y or M if you want to enable support for tagging frames with the + NXP SJA1105 switch family. Both the native tagging protocol (which + is only for link-local traffic) as well as non-native tagging (based + on a custom 802.1Q VLAN header) are available. config NET_DSA_TAG_TRAILER - bool - -config NET_DSA_TAG_QCA - bool + tristate "Tag driver for switches using a trailer tag" + help + Say Y or M if you want to enable support for tagging frames at + with a trailed. e.g. Marvell 88E6060. endif diff --git a/net/dsa/Makefile b/net/dsa/Makefile index 6e721f7a2947f1a48716f1cff04b394c04658c2b..c342f54715ba1a94fba27cf3dfde3311cf8cdffd 100644 --- a/net/dsa/Makefile +++ b/net/dsa/Makefile @@ -2,16 +2,16 @@ # the core obj-$(CONFIG_NET_DSA) += dsa_core.o dsa_core-y += dsa.o dsa2.o master.o port.o slave.o switch.o -dsa_core-$(CONFIG_NET_DSA_LEGACY) += legacy.o # tagging formats -dsa_core-$(CONFIG_NET_DSA_TAG_BRCM) += tag_brcm.o -dsa_core-$(CONFIG_NET_DSA_TAG_BRCM_PREPEND) += tag_brcm.o -dsa_core-$(CONFIG_NET_DSA_TAG_DSA) += tag_dsa.o -dsa_core-$(CONFIG_NET_DSA_TAG_EDSA) += tag_edsa.o -dsa_core-$(CONFIG_NET_DSA_TAG_GSWIP) += tag_gswip.o -dsa_core-$(CONFIG_NET_DSA_TAG_KSZ) += tag_ksz.o -dsa_core-$(CONFIG_NET_DSA_TAG_LAN9303) += tag_lan9303.o -dsa_core-$(CONFIG_NET_DSA_TAG_MTK) += tag_mtk.o -dsa_core-$(CONFIG_NET_DSA_TAG_QCA) += tag_qca.o -dsa_core-$(CONFIG_NET_DSA_TAG_TRAILER) += tag_trailer.o +obj-$(CONFIG_NET_DSA_TAG_8021Q) += tag_8021q.o +obj-$(CONFIG_NET_DSA_TAG_BRCM_COMMON) += tag_brcm.o +obj-$(CONFIG_NET_DSA_TAG_DSA) += tag_dsa.o +obj-$(CONFIG_NET_DSA_TAG_EDSA) += tag_edsa.o +obj-$(CONFIG_NET_DSA_TAG_GSWIP) += tag_gswip.o +obj-$(CONFIG_NET_DSA_TAG_KSZ_COMMON) += tag_ksz.o +obj-$(CONFIG_NET_DSA_TAG_LAN9303) += tag_lan9303.o +obj-$(CONFIG_NET_DSA_TAG_MTK) += tag_mtk.o +obj-$(CONFIG_NET_DSA_TAG_QCA) += tag_qca.o +obj-$(CONFIG_NET_DSA_TAG_SJA1105) += tag_sja1105.o +obj-$(CONFIG_NET_DSA_TAG_TRAILER) += tag_trailer.o diff --git a/net/dsa/dsa.c b/net/dsa/dsa.c index 36de4f2a3366b4a56d50d3c3ac26a61c0c89548b..1fc782fab393cd756c7dde34ec126a4a64f28735 100644 --- a/net/dsa/dsa.c +++ b/net/dsa/dsa.c @@ -27,6 +27,9 @@ #include "dsa_priv.h" +static LIST_HEAD(dsa_tag_drivers_list); +static DEFINE_MUTEX(dsa_tag_drivers_lock); + static struct sk_buff *dsa_slave_notag_xmit(struct sk_buff *skb, struct net_device *dev) { @@ -35,106 +38,103 @@ static struct sk_buff *dsa_slave_notag_xmit(struct sk_buff *skb, } static const struct dsa_device_ops none_ops = { + .name = "none", + .proto = DSA_TAG_PROTO_NONE, .xmit = dsa_slave_notag_xmit, .rcv = NULL, }; -const struct dsa_device_ops *dsa_device_ops[DSA_TAG_LAST] = { -#ifdef CONFIG_NET_DSA_TAG_BRCM - [DSA_TAG_PROTO_BRCM] = &brcm_netdev_ops, -#endif -#ifdef CONFIG_NET_DSA_TAG_BRCM_PREPEND - [DSA_TAG_PROTO_BRCM_PREPEND] = &brcm_prepend_netdev_ops, -#endif -#ifdef CONFIG_NET_DSA_TAG_DSA - [DSA_TAG_PROTO_DSA] = &dsa_netdev_ops, -#endif -#ifdef CONFIG_NET_DSA_TAG_EDSA - [DSA_TAG_PROTO_EDSA] = &edsa_netdev_ops, -#endif -#ifdef CONFIG_NET_DSA_TAG_GSWIP - [DSA_TAG_PROTO_GSWIP] = &gswip_netdev_ops, -#endif -#ifdef CONFIG_NET_DSA_TAG_KSZ9477 - [DSA_TAG_PROTO_KSZ9477] = &ksz9477_netdev_ops, - [DSA_TAG_PROTO_KSZ9893] = &ksz9893_netdev_ops, -#endif -#ifdef CONFIG_NET_DSA_TAG_LAN9303 - [DSA_TAG_PROTO_LAN9303] = &lan9303_netdev_ops, -#endif -#ifdef CONFIG_NET_DSA_TAG_MTK - [DSA_TAG_PROTO_MTK] = &mtk_netdev_ops, -#endif -#ifdef CONFIG_NET_DSA_TAG_QCA - [DSA_TAG_PROTO_QCA] = &qca_netdev_ops, -#endif -#ifdef CONFIG_NET_DSA_TAG_TRAILER - [DSA_TAG_PROTO_TRAILER] = &trailer_netdev_ops, -#endif - [DSA_TAG_PROTO_NONE] = &none_ops, -}; +DSA_TAG_DRIVER(none_ops); -const char *dsa_tag_protocol_to_str(const struct dsa_device_ops *ops) +static void dsa_tag_driver_register(struct dsa_tag_driver *dsa_tag_driver, + struct module *owner) +{ + dsa_tag_driver->owner = owner; + + mutex_lock(&dsa_tag_drivers_lock); + list_add_tail(&dsa_tag_driver->list, &dsa_tag_drivers_list); + mutex_unlock(&dsa_tag_drivers_lock); +} + +void dsa_tag_drivers_register(struct dsa_tag_driver *dsa_tag_driver_array[], + unsigned int count, struct module *owner) { - const char *protocol_name[DSA_TAG_LAST] = { -#ifdef CONFIG_NET_DSA_TAG_BRCM - [DSA_TAG_PROTO_BRCM] = "brcm", -#endif -#ifdef CONFIG_NET_DSA_TAG_BRCM_PREPEND - [DSA_TAG_PROTO_BRCM_PREPEND] = "brcm-prepend", -#endif -#ifdef CONFIG_NET_DSA_TAG_DSA - [DSA_TAG_PROTO_DSA] = "dsa", -#endif -#ifdef CONFIG_NET_DSA_TAG_EDSA - [DSA_TAG_PROTO_EDSA] = "edsa", -#endif -#ifdef CONFIG_NET_DSA_TAG_GSWIP - [DSA_TAG_PROTO_GSWIP] = "gswip", -#endif -#ifdef CONFIG_NET_DSA_TAG_KSZ9477 - [DSA_TAG_PROTO_KSZ9477] = "ksz9477", - [DSA_TAG_PROTO_KSZ9893] = "ksz9893", -#endif -#ifdef CONFIG_NET_DSA_TAG_LAN9303 - [DSA_TAG_PROTO_LAN9303] = "lan9303", -#endif -#ifdef CONFIG_NET_DSA_TAG_MTK - [DSA_TAG_PROTO_MTK] = "mtk", -#endif -#ifdef CONFIG_NET_DSA_TAG_QCA - [DSA_TAG_PROTO_QCA] = "qca", -#endif -#ifdef CONFIG_NET_DSA_TAG_TRAILER - [DSA_TAG_PROTO_TRAILER] = "trailer", -#endif - [DSA_TAG_PROTO_NONE] = "none", - }; unsigned int i; - BUILD_BUG_ON(ARRAY_SIZE(protocol_name) != DSA_TAG_LAST); + for (i = 0; i < count; i++) + dsa_tag_driver_register(dsa_tag_driver_array[i], owner); +} - for (i = 0; i < ARRAY_SIZE(dsa_device_ops); i++) - if (ops == dsa_device_ops[i]) - return protocol_name[i]; +static void dsa_tag_driver_unregister(struct dsa_tag_driver *dsa_tag_driver) +{ + mutex_lock(&dsa_tag_drivers_lock); + list_del(&dsa_tag_driver->list); + mutex_unlock(&dsa_tag_drivers_lock); +} +EXPORT_SYMBOL_GPL(dsa_tag_drivers_register); - return protocol_name[DSA_TAG_PROTO_NONE]; +void dsa_tag_drivers_unregister(struct dsa_tag_driver *dsa_tag_driver_array[], + unsigned int count) +{ + unsigned int i; + + for (i = 0; i < count; i++) + dsa_tag_driver_unregister(dsa_tag_driver_array[i]); +} +EXPORT_SYMBOL_GPL(dsa_tag_drivers_unregister); + +const char *dsa_tag_protocol_to_str(const struct dsa_device_ops *ops) +{ + return ops->name; }; -const struct dsa_device_ops *dsa_resolve_tag_protocol(int tag_protocol) +const struct dsa_device_ops *dsa_tag_driver_get(int tag_protocol) { + struct dsa_tag_driver *dsa_tag_driver; const struct dsa_device_ops *ops; + char module_name[128]; + bool found = false; + + snprintf(module_name, 127, "%s%d", DSA_TAG_DRIVER_ALIAS, + tag_protocol); + + request_module(module_name); + + mutex_lock(&dsa_tag_drivers_lock); + list_for_each_entry(dsa_tag_driver, &dsa_tag_drivers_list, list) { + ops = dsa_tag_driver->ops; + if (ops->proto == tag_protocol) { + found = true; + break; + } + } - if (tag_protocol >= DSA_TAG_LAST) - return ERR_PTR(-EINVAL); - ops = dsa_device_ops[tag_protocol]; + if (found) { + if (!try_module_get(dsa_tag_driver->owner)) + ops = ERR_PTR(-ENOPROTOOPT); + } else { + ops = ERR_PTR(-ENOPROTOOPT); + } - if (!ops) - return ERR_PTR(-ENOPROTOOPT); + mutex_unlock(&dsa_tag_drivers_lock); return ops; } +void dsa_tag_driver_put(const struct dsa_device_ops *ops) +{ + struct dsa_tag_driver *dsa_tag_driver; + + mutex_lock(&dsa_tag_drivers_lock); + list_for_each_entry(dsa_tag_driver, &dsa_tag_drivers_list, list) { + if (dsa_tag_driver->ops == ops) { + module_put(dsa_tag_driver->owner); + break; + } + } + mutex_unlock(&dsa_tag_drivers_lock); +} + static int dev_is_class(struct device *dev, void *class) { if (dev->class != NULL && !strcmp(dev->class->name, class)) @@ -344,23 +344,28 @@ static int __init dsa_init_module(void) rc = dsa_slave_register_notifier(); if (rc) - return rc; - - rc = dsa_legacy_register(); - if (rc) - return rc; + goto register_notifier_fail; dev_add_pack(&dsa_pack_type); + dsa_tag_driver_register(&DSA_TAG_DRIVER_NAME(none_ops), + THIS_MODULE); + return 0; + +register_notifier_fail: + destroy_workqueue(dsa_owq); + + return rc; } module_init(dsa_init_module); static void __exit dsa_cleanup_module(void) { + dsa_tag_driver_unregister(&DSA_TAG_DRIVER_NAME(none_ops)); + dsa_slave_unregister_notifier(); dev_remove_pack(&dsa_pack_type); - dsa_legacy_unregister(); destroy_workqueue(dsa_owq); } module_exit(dsa_cleanup_module); diff --git a/net/dsa/dsa2.c b/net/dsa/dsa2.c index c00ee464afc78fbe23de6be18be576df9175be0d..3b5f434cad3f51378bee9dd067a3cf0ed200f8d3 100644 --- a/net/dsa/dsa2.c +++ b/net/dsa/dsa2.c @@ -18,6 +18,7 @@ #include #include #include +#include #include "dsa_priv.h" @@ -257,14 +258,39 @@ static void dsa_tree_teardown_default_cpu(struct dsa_switch_tree *dst) static int dsa_port_setup(struct dsa_port *dp) { + enum devlink_port_flavour flavour; struct dsa_switch *ds = dp->ds; - int err = 0; + struct dsa_switch_tree *dst = ds->dst; + int err; + + if (dp->type == DSA_PORT_TYPE_UNUSED) + return 0; memset(&dp->devlink_port, 0, sizeof(dp->devlink_port)); + dp->mac = of_get_mac_address(dp->dn); - if (dp->type != DSA_PORT_TYPE_UNUSED) - err = devlink_port_register(ds->devlink, &dp->devlink_port, - dp->index); + switch (dp->type) { + case DSA_PORT_TYPE_CPU: + flavour = DEVLINK_PORT_FLAVOUR_CPU; + break; + case DSA_PORT_TYPE_DSA: + flavour = DEVLINK_PORT_FLAVOUR_DSA; + break; + case DSA_PORT_TYPE_USER: /* fall-through */ + default: + flavour = DEVLINK_PORT_FLAVOUR_PHYSICAL; + break; + } + + /* dp->index is used now as port_number. However + * CPU and DSA ports should have separate numbering + * independent from front panel port numbers. + */ + devlink_port_attrs_set(&dp->devlink_port, flavour, + dp->index, false, 0, + (const char *) &dst->index, sizeof(dst->index)); + err = devlink_port_register(ds->devlink, &dp->devlink_port, + dp->index); if (err) return err; @@ -272,13 +298,6 @@ static int dsa_port_setup(struct dsa_port *dp) case DSA_PORT_TYPE_UNUSED: break; case DSA_PORT_TYPE_CPU: - /* dp->index is used now as port_number. However - * CPU ports should have separate numbering - * independent from front panel port numbers. - */ - devlink_port_attrs_set(&dp->devlink_port, - DEVLINK_PORT_FLAVOUR_CPU, - dp->index, false, 0); err = dsa_port_link_register_of(dp); if (err) { dev_err(ds->dev, "failed to setup link for port %d.%d\n", @@ -287,13 +306,6 @@ static int dsa_port_setup(struct dsa_port *dp) } break; case DSA_PORT_TYPE_DSA: - /* dp->index is used now as port_number. However - * DSA ports should have separate numbering - * independent from front panel port numbers. - */ - devlink_port_attrs_set(&dp->devlink_port, - DEVLINK_PORT_FLAVOUR_DSA, - dp->index, false, 0); err = dsa_port_link_register_of(dp); if (err) { dev_err(ds->dev, "failed to setup link for port %d.%d\n", @@ -302,9 +314,6 @@ static int dsa_port_setup(struct dsa_port *dp) } break; case DSA_PORT_TYPE_USER: - devlink_port_attrs_set(&dp->devlink_port, - DEVLINK_PORT_FLAVOUR_PHYSICAL, - dp->index, false, 0); err = dsa_slave_create(dp); if (err) dev_err(ds->dev, "failed to create slave for port %d.%d\n", @@ -326,6 +335,8 @@ static void dsa_port_teardown(struct dsa_port *dp) case DSA_PORT_TYPE_UNUSED: break; case DSA_PORT_TYPE_CPU: + dsa_tag_driver_put(dp->tag_ops); + /* fall-through */ case DSA_PORT_TYPE_DSA: dsa_port_link_unregister_of(dp); break; @@ -360,14 +371,14 @@ static int dsa_switch_setup(struct dsa_switch *ds) if (err) return err; - err = ds->ops->setup(ds); - if (err < 0) - return err; - err = dsa_switch_register_notifier(ds); if (err) return err; + err = ds->ops->setup(ds); + if (err < 0) + return err; + if (!ds->slave_mii_bus && ds->ops->phy_read) { ds->slave_mii_bus = devm_mdiobus_alloc(ds->dev); if (!ds->slave_mii_bus) @@ -568,13 +579,14 @@ static int dsa_port_parse_cpu(struct dsa_port *dp, struct net_device *master) enum dsa_tag_protocol tag_protocol; tag_protocol = ds->ops->get_tag_protocol(ds, dp->index); - tag_ops = dsa_resolve_tag_protocol(tag_protocol); + tag_ops = dsa_tag_driver_get(tag_protocol); if (IS_ERR(tag_ops)) { dev_warn(ds->dev, "No tagger for this switch\n"); return PTR_ERR(tag_ops); } dp->type = DSA_PORT_TYPE_CPU; + dp->filter = tag_ops->filter; dp->rcv = tag_ops->rcv; dp->tag_ops = tag_ops; dp->master = master; diff --git a/net/dsa/dsa_priv.h b/net/dsa/dsa_priv.h index 093b7d145eb14d7c8b581c1f1438fa270d9800e4..8f122232464608fc01dd6a5d95962ec494579620 100644 --- a/net/dsa/dsa_priv.h +++ b/net/dsa/dsa_priv.h @@ -84,22 +84,12 @@ struct dsa_slave_priv { }; /* dsa.c */ -const struct dsa_device_ops *dsa_resolve_tag_protocol(int tag_protocol); +const struct dsa_device_ops *dsa_tag_driver_get(int tag_protocol); +void dsa_tag_driver_put(const struct dsa_device_ops *ops); + bool dsa_schedule_work(struct work_struct *work); const char *dsa_tag_protocol_to_str(const struct dsa_device_ops *ops); -/* legacy.c */ -#if IS_ENABLED(CONFIG_NET_DSA_LEGACY) -int dsa_legacy_register(void); -void dsa_legacy_unregister(void); -#else -static inline int dsa_legacy_register(void) -{ - return 0; -} - -static inline void dsa_legacy_unregister(void) { } -#endif int dsa_legacy_fdb_add(struct ndmsg *ndm, struct nlattr *tb[], struct net_device *dev, const unsigned char *addr, u16 vid, @@ -169,6 +159,8 @@ int dsa_port_vlan_add(struct dsa_port *dp, struct switchdev_trans *trans); int dsa_port_vlan_del(struct dsa_port *dp, const struct switchdev_obj_port_vlan *vlan); +int dsa_port_vid_add(struct dsa_port *dp, u16 vid, u16 flags); +int dsa_port_vid_del(struct dsa_port *dp, u16 vid); int dsa_port_link_register_of(struct dsa_port *dp); void dsa_port_link_unregister_of(struct dsa_port *dp); @@ -182,6 +174,8 @@ int dsa_slave_resume(struct net_device *slave_dev); int dsa_slave_register_notifier(void); void dsa_slave_unregister_notifier(void); +void *dsa_defer_xmit(struct sk_buff *skb, struct net_device *dev); + static inline struct dsa_port *dsa_slave_to_port(const struct net_device *dev) { struct dsa_slave_priv *p = netdev_priv(dev); @@ -200,34 +194,4 @@ dsa_slave_to_master(const struct net_device *dev) /* switch.c */ int dsa_switch_register_notifier(struct dsa_switch *ds); void dsa_switch_unregister_notifier(struct dsa_switch *ds); - -/* tag_brcm.c */ -extern const struct dsa_device_ops brcm_netdev_ops; -extern const struct dsa_device_ops brcm_prepend_netdev_ops; - -/* tag_dsa.c */ -extern const struct dsa_device_ops dsa_netdev_ops; - -/* tag_edsa.c */ -extern const struct dsa_device_ops edsa_netdev_ops; - -/* tag_gswip.c */ -extern const struct dsa_device_ops gswip_netdev_ops; - -/* tag_ksz.c */ -extern const struct dsa_device_ops ksz9477_netdev_ops; -extern const struct dsa_device_ops ksz9893_netdev_ops; - -/* tag_lan9303.c */ -extern const struct dsa_device_ops lan9303_netdev_ops; - -/* tag_mtk.c */ -extern const struct dsa_device_ops mtk_netdev_ops; - -/* tag_qca.c */ -extern const struct dsa_device_ops qca_netdev_ops; - -/* tag_trailer.c */ -extern const struct dsa_device_ops trailer_netdev_ops; - #endif diff --git a/net/dsa/legacy.c b/net/dsa/legacy.c deleted file mode 100644 index cb42939db7767bf1fc748c5da7b0f76a135a1171..0000000000000000000000000000000000000000 --- a/net/dsa/legacy.c +++ /dev/null @@ -1,745 +0,0 @@ -/* - * net/dsa/legacy.c - Hardware switch handling - * Copyright (c) 2008-2009 Marvell Semiconductor - * Copyright (c) 2013 Florian Fainelli - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "dsa_priv.h" - -/* switch driver registration ***********************************************/ -static DEFINE_MUTEX(dsa_switch_drivers_mutex); -static LIST_HEAD(dsa_switch_drivers); - -void register_switch_driver(struct dsa_switch_driver *drv) -{ - mutex_lock(&dsa_switch_drivers_mutex); - list_add_tail(&drv->list, &dsa_switch_drivers); - mutex_unlock(&dsa_switch_drivers_mutex); -} -EXPORT_SYMBOL_GPL(register_switch_driver); - -void unregister_switch_driver(struct dsa_switch_driver *drv) -{ - mutex_lock(&dsa_switch_drivers_mutex); - list_del_init(&drv->list); - mutex_unlock(&dsa_switch_drivers_mutex); -} -EXPORT_SYMBOL_GPL(unregister_switch_driver); - -static const struct dsa_switch_ops * -dsa_switch_probe(struct device *parent, struct device *host_dev, int sw_addr, - const char **_name, void **priv) -{ - const struct dsa_switch_ops *ret; - struct list_head *list; - const char *name; - - ret = NULL; - name = NULL; - - mutex_lock(&dsa_switch_drivers_mutex); - list_for_each(list, &dsa_switch_drivers) { - const struct dsa_switch_ops *ops; - struct dsa_switch_driver *drv; - - drv = list_entry(list, struct dsa_switch_driver, list); - ops = drv->ops; - - name = ops->probe(parent, host_dev, sw_addr, priv); - if (name != NULL) { - ret = ops; - break; - } - } - mutex_unlock(&dsa_switch_drivers_mutex); - - *_name = name; - - return ret; -} - -/* basic switch operations **************************************************/ -static int dsa_cpu_dsa_setups(struct dsa_switch *ds) -{ - int ret, port; - - for (port = 0; port < ds->num_ports; port++) { - if (!(dsa_is_cpu_port(ds, port) || dsa_is_dsa_port(ds, port))) - continue; - - ret = dsa_port_link_register_of(&ds->ports[port]); - if (ret) - return ret; - } - return 0; -} - -static int dsa_switch_setup_one(struct dsa_switch *ds, - struct net_device *master) -{ - const struct dsa_switch_ops *ops = ds->ops; - struct dsa_switch_tree *dst = ds->dst; - struct dsa_chip_data *cd = ds->cd; - bool valid_name_found = false; - int index = ds->index; - struct dsa_port *dp; - int i, ret; - - /* - * Validate supplied switch configuration. - */ - for (i = 0; i < ds->num_ports; i++) { - char *name; - - dp = &ds->ports[i]; - - name = cd->port_names[i]; - if (name == NULL) - continue; - dp->name = name; - - if (!strcmp(name, "cpu")) { - if (dst->cpu_dp) { - netdev_err(master, - "multiple cpu ports?!\n"); - return -EINVAL; - } - dst->cpu_dp = &ds->ports[i]; - dst->cpu_dp->master = master; - dp->type = DSA_PORT_TYPE_CPU; - } else if (!strcmp(name, "dsa")) { - dp->type = DSA_PORT_TYPE_DSA; - } else { - dp->type = DSA_PORT_TYPE_USER; - } - valid_name_found = true; - } - - if (!valid_name_found && i == ds->num_ports) - return -EINVAL; - - /* Make the built-in MII bus mask match the number of ports, - * switch drivers can override this later - */ - ds->phys_mii_mask |= dsa_user_ports(ds); - - /* - * If the CPU connects to this switch, set the switch tree - * tagging protocol to the preferred tagging format of this - * switch. - */ - if (dst->cpu_dp->ds == ds) { - const struct dsa_device_ops *tag_ops; - enum dsa_tag_protocol tag_protocol; - - tag_protocol = ops->get_tag_protocol(ds, dst->cpu_dp->index); - tag_ops = dsa_resolve_tag_protocol(tag_protocol); - if (IS_ERR(tag_ops)) - return PTR_ERR(tag_ops); - - dst->cpu_dp->tag_ops = tag_ops; - - /* Few copies for faster access in master receive hot path */ - dst->cpu_dp->rcv = dst->cpu_dp->tag_ops->rcv; - dst->cpu_dp->dst = dst; - } - - memcpy(ds->rtable, cd->rtable, sizeof(ds->rtable)); - - /* - * Do basic register setup. - */ - ret = ops->setup(ds); - if (ret < 0) - return ret; - - ret = dsa_switch_register_notifier(ds); - if (ret) - return ret; - - if (!ds->slave_mii_bus && ops->phy_read) { - ds->slave_mii_bus = devm_mdiobus_alloc(ds->dev); - if (!ds->slave_mii_bus) - return -ENOMEM; - dsa_slave_mii_bus_init(ds); - - ret = mdiobus_register(ds->slave_mii_bus); - if (ret < 0) - return ret; - } - - /* - * Create network devices for physical switch ports. - */ - for (i = 0; i < ds->num_ports; i++) { - ds->ports[i].dn = cd->port_dn[i]; - ds->ports[i].cpu_dp = dst->cpu_dp; - - if (!dsa_is_user_port(ds, i)) - continue; - - ret = dsa_slave_create(&ds->ports[i]); - if (ret < 0) - netdev_err(master, "[%d]: can't create dsa slave device for port %d(%s): %d\n", - index, i, cd->port_names[i], ret); - } - - /* Perform configuration of the CPU and DSA ports */ - ret = dsa_cpu_dsa_setups(ds); - if (ret < 0) - netdev_err(master, "[%d] : can't configure CPU and DSA ports\n", - index); - - return 0; -} - -static struct dsa_switch * -dsa_switch_setup(struct dsa_switch_tree *dst, struct net_device *master, - int index, struct device *parent, struct device *host_dev) -{ - struct dsa_chip_data *cd = dst->pd->chip + index; - const struct dsa_switch_ops *ops; - struct dsa_switch *ds; - int ret; - const char *name; - void *priv; - - /* - * Probe for switch model. - */ - ops = dsa_switch_probe(parent, host_dev, cd->sw_addr, &name, &priv); - if (!ops) { - netdev_err(master, "[%d]: could not detect attached switch\n", - index); - return ERR_PTR(-EINVAL); - } - netdev_info(master, "[%d]: detected a %s switch\n", - index, name); - - - /* - * Allocate and initialise switch state. - */ - ds = dsa_switch_alloc(parent, DSA_MAX_PORTS); - if (!ds) - return ERR_PTR(-ENOMEM); - - ds->dst = dst; - ds->index = index; - ds->cd = cd; - ds->ops = ops; - ds->priv = priv; - - ret = dsa_switch_setup_one(ds, master); - if (ret) - return ERR_PTR(ret); - - return ds; -} - -static void dsa_switch_destroy(struct dsa_switch *ds) -{ - int port; - - /* Destroy network devices for physical switch ports. */ - for (port = 0; port < ds->num_ports; port++) { - if (!dsa_is_user_port(ds, port)) - continue; - - if (!ds->ports[port].slave) - continue; - - dsa_slave_destroy(ds->ports[port].slave); - } - - /* Disable configuration of the CPU and DSA ports */ - for (port = 0; port < ds->num_ports; port++) { - if (!(dsa_is_cpu_port(ds, port) || dsa_is_dsa_port(ds, port))) - continue; - dsa_port_link_unregister_of(&ds->ports[port]); - } - - if (ds->slave_mii_bus && ds->ops->phy_read) - mdiobus_unregister(ds->slave_mii_bus); - - dsa_switch_unregister_notifier(ds); -} - -/* platform driver init and cleanup *****************************************/ -static int dev_is_class(struct device *dev, void *class) -{ - if (dev->class != NULL && !strcmp(dev->class->name, class)) - return 1; - - return 0; -} - -static struct device *dev_find_class(struct device *parent, char *class) -{ - if (dev_is_class(parent, class)) { - get_device(parent); - return parent; - } - - return device_find_child(parent, class, dev_is_class); -} - -struct mii_bus *dsa_host_dev_to_mii_bus(struct device *dev) -{ - struct device *d; - - d = dev_find_class(dev, "mdio_bus"); - if (d != NULL) { - struct mii_bus *bus; - - bus = to_mii_bus(d); - put_device(d); - - return bus; - } - - return NULL; -} -EXPORT_SYMBOL_GPL(dsa_host_dev_to_mii_bus); - -#ifdef CONFIG_OF -static int dsa_of_setup_routing_table(struct dsa_platform_data *pd, - struct dsa_chip_data *cd, - int chip_index, int port_index, - struct device_node *link) -{ - const __be32 *reg; - int link_sw_addr; - struct device_node *parent_sw; - int len; - - parent_sw = of_get_parent(link); - if (!parent_sw) - return -EINVAL; - - reg = of_get_property(parent_sw, "reg", &len); - if (!reg || (len != sizeof(*reg) * 2)) - return -EINVAL; - - /* - * Get the destination switch number from the second field of its 'reg' - * property, i.e. for "reg = <0x19 1>" sw_addr is '1'. - */ - link_sw_addr = be32_to_cpup(reg + 1); - - if (link_sw_addr >= pd->nr_chips) - return -EINVAL; - - cd->rtable[link_sw_addr] = port_index; - - return 0; -} - -static int dsa_of_probe_links(struct dsa_platform_data *pd, - struct dsa_chip_data *cd, - int chip_index, int port_index, - struct device_node *port, - const char *port_name) -{ - struct device_node *link; - int link_index; - int ret; - - for (link_index = 0;; link_index++) { - link = of_parse_phandle(port, "link", link_index); - if (!link) - break; - - if (!strcmp(port_name, "dsa") && pd->nr_chips > 1) { - ret = dsa_of_setup_routing_table(pd, cd, chip_index, - port_index, link); - if (ret) - return ret; - } - } - return 0; -} - -static void dsa_of_free_platform_data(struct dsa_platform_data *pd) -{ - int i; - int port_index; - - for (i = 0; i < pd->nr_chips; i++) { - port_index = 0; - while (port_index < DSA_MAX_PORTS) { - kfree(pd->chip[i].port_names[port_index]); - port_index++; - } - - /* Drop our reference to the MDIO bus device */ - put_device(pd->chip[i].host_dev); - } - kfree(pd->chip); -} - -static int dsa_of_probe(struct device *dev) -{ - struct device_node *np = dev->of_node; - struct device_node *child, *mdio, *ethernet, *port; - struct mii_bus *mdio_bus, *mdio_bus_switch; - struct net_device *ethernet_dev; - struct dsa_platform_data *pd; - struct dsa_chip_data *cd; - const char *port_name; - int chip_index, port_index; - const unsigned int *sw_addr, *port_reg; - u32 eeprom_len; - int ret; - - mdio = of_parse_phandle(np, "dsa,mii-bus", 0); - if (!mdio) - return -EINVAL; - - mdio_bus = of_mdio_find_bus(mdio); - if (!mdio_bus) - return -EPROBE_DEFER; - - ethernet = of_parse_phandle(np, "dsa,ethernet", 0); - if (!ethernet) { - ret = -EINVAL; - goto out_put_mdio; - } - - ethernet_dev = of_find_net_device_by_node(ethernet); - if (!ethernet_dev) { - ret = -EPROBE_DEFER; - goto out_put_mdio; - } - - pd = kzalloc(sizeof(*pd), GFP_KERNEL); - if (!pd) { - ret = -ENOMEM; - goto out_put_ethernet; - } - - dev->platform_data = pd; - pd->of_netdev = ethernet_dev; - pd->nr_chips = of_get_available_child_count(np); - if (pd->nr_chips > DSA_MAX_SWITCHES) - pd->nr_chips = DSA_MAX_SWITCHES; - - pd->chip = kcalloc(pd->nr_chips, sizeof(struct dsa_chip_data), - GFP_KERNEL); - if (!pd->chip) { - ret = -ENOMEM; - goto out_free; - } - - chip_index = -1; - for_each_available_child_of_node(np, child) { - int i; - - chip_index++; - cd = &pd->chip[chip_index]; - - cd->of_node = child; - - /* Initialize the routing table */ - for (i = 0; i < DSA_MAX_SWITCHES; ++i) - cd->rtable[i] = DSA_RTABLE_NONE; - - /* When assigning the host device, increment its refcount */ - cd->host_dev = get_device(&mdio_bus->dev); - - sw_addr = of_get_property(child, "reg", NULL); - if (!sw_addr) - continue; - - cd->sw_addr = be32_to_cpup(sw_addr); - if (cd->sw_addr >= PHY_MAX_ADDR) - continue; - - if (!of_property_read_u32(child, "eeprom-length", &eeprom_len)) - cd->eeprom_len = eeprom_len; - - mdio = of_parse_phandle(child, "mii-bus", 0); - if (mdio) { - mdio_bus_switch = of_mdio_find_bus(mdio); - if (!mdio_bus_switch) { - ret = -EPROBE_DEFER; - goto out_free_chip; - } - - /* Drop the mdio_bus device ref, replacing the host - * device with the mdio_bus_switch device, keeping - * the refcount from of_mdio_find_bus() above. - */ - put_device(cd->host_dev); - cd->host_dev = &mdio_bus_switch->dev; - } - - for_each_available_child_of_node(child, port) { - port_reg = of_get_property(port, "reg", NULL); - if (!port_reg) - continue; - - port_index = be32_to_cpup(port_reg); - if (port_index >= DSA_MAX_PORTS) - break; - - port_name = of_get_property(port, "label", NULL); - if (!port_name) - continue; - - cd->port_dn[port_index] = port; - - cd->port_names[port_index] = kstrdup(port_name, - GFP_KERNEL); - if (!cd->port_names[port_index]) { - ret = -ENOMEM; - goto out_free_chip; - } - - ret = dsa_of_probe_links(pd, cd, chip_index, - port_index, port, port_name); - if (ret) - goto out_free_chip; - - } - } - - /* The individual chips hold their own refcount on the mdio bus, - * so drop ours */ - put_device(&mdio_bus->dev); - - return 0; - -out_free_chip: - dsa_of_free_platform_data(pd); -out_free: - kfree(pd); - dev->platform_data = NULL; -out_put_ethernet: - put_device(ðernet_dev->dev); -out_put_mdio: - put_device(&mdio_bus->dev); - return ret; -} - -static void dsa_of_remove(struct device *dev) -{ - struct dsa_platform_data *pd = dev->platform_data; - - if (!dev->of_node) - return; - - dsa_of_free_platform_data(pd); - put_device(&pd->of_netdev->dev); - kfree(pd); -} -#else -static inline int dsa_of_probe(struct device *dev) -{ - return 0; -} - -static inline void dsa_of_remove(struct device *dev) -{ -} -#endif - -static int dsa_setup_dst(struct dsa_switch_tree *dst, struct net_device *dev, - struct device *parent, struct dsa_platform_data *pd) -{ - int i; - unsigned configured = 0; - - dst->pd = pd; - - for (i = 0; i < pd->nr_chips; i++) { - struct dsa_switch *ds; - - ds = dsa_switch_setup(dst, dev, i, parent, pd->chip[i].host_dev); - if (IS_ERR(ds)) { - netdev_err(dev, "[%d]: couldn't create dsa switch instance (error %ld)\n", - i, PTR_ERR(ds)); - continue; - } - - dst->ds[i] = ds; - - ++configured; - } - - /* - * If no switch was found, exit cleanly - */ - if (!configured) - return -EPROBE_DEFER; - - return dsa_master_setup(dst->cpu_dp->master, dst->cpu_dp); -} - -static int dsa_probe(struct platform_device *pdev) -{ - struct dsa_platform_data *pd = pdev->dev.platform_data; - struct net_device *dev; - struct dsa_switch_tree *dst; - int ret; - - if (pdev->dev.of_node) { - ret = dsa_of_probe(&pdev->dev); - if (ret) - return ret; - - pd = pdev->dev.platform_data; - } - - if (pd == NULL || (pd->netdev == NULL && pd->of_netdev == NULL)) - return -EINVAL; - - if (pd->of_netdev) { - dev = pd->of_netdev; - dev_hold(dev); - } else { - dev = dsa_dev_to_net_device(pd->netdev); - } - if (dev == NULL) { - ret = -EPROBE_DEFER; - goto out; - } - - if (dev->dsa_ptr != NULL) { - dev_put(dev); - ret = -EEXIST; - goto out; - } - - dst = devm_kzalloc(&pdev->dev, sizeof(*dst), GFP_KERNEL); - if (dst == NULL) { - dev_put(dev); - ret = -ENOMEM; - goto out; - } - - platform_set_drvdata(pdev, dst); - - ret = dsa_setup_dst(dst, dev, &pdev->dev, pd); - if (ret) { - dev_put(dev); - goto out; - } - - return 0; - -out: - dsa_of_remove(&pdev->dev); - - return ret; -} - -static void dsa_remove_dst(struct dsa_switch_tree *dst) -{ - int i; - - dsa_master_teardown(dst->cpu_dp->master); - - for (i = 0; i < dst->pd->nr_chips; i++) { - struct dsa_switch *ds = dst->ds[i]; - - if (ds) - dsa_switch_destroy(ds); - } - - dev_put(dst->cpu_dp->master); -} - -static int dsa_remove(struct platform_device *pdev) -{ - struct dsa_switch_tree *dst = platform_get_drvdata(pdev); - - dsa_remove_dst(dst); - dsa_of_remove(&pdev->dev); - - return 0; -} - -static void dsa_shutdown(struct platform_device *pdev) -{ -} - -#ifdef CONFIG_PM_SLEEP -static int dsa_suspend(struct device *d) -{ - struct dsa_switch_tree *dst = dev_get_drvdata(d); - int i, ret = 0; - - for (i = 0; i < dst->pd->nr_chips; i++) { - struct dsa_switch *ds = dst->ds[i]; - - if (ds != NULL) - ret = dsa_switch_suspend(ds); - } - - return ret; -} - -static int dsa_resume(struct device *d) -{ - struct dsa_switch_tree *dst = dev_get_drvdata(d); - int i, ret = 0; - - for (i = 0; i < dst->pd->nr_chips; i++) { - struct dsa_switch *ds = dst->ds[i]; - - if (ds != NULL) - ret = dsa_switch_resume(ds); - } - - return ret; -} -#endif - -static SIMPLE_DEV_PM_OPS(dsa_pm_ops, dsa_suspend, dsa_resume); - -static const struct of_device_id dsa_of_match_table[] = { - { .compatible = "marvell,dsa", }, - {} -}; -MODULE_DEVICE_TABLE(of, dsa_of_match_table); - -static struct platform_driver dsa_driver = { - .probe = dsa_probe, - .remove = dsa_remove, - .shutdown = dsa_shutdown, - .driver = { - .name = "dsa", - .of_match_table = dsa_of_match_table, - .pm = &dsa_pm_ops, - }, -}; - -int dsa_legacy_register(void) -{ - return platform_driver_register(&dsa_driver); -} - -void dsa_legacy_unregister(void) -{ - platform_driver_unregister(&dsa_driver); -} diff --git a/net/dsa/port.c b/net/dsa/port.c index caeef4c99dc0756154d996fda344b44127de1b63..ed8ba9daa3baa3acf9a83d51199ec6c2cb87b3b6 100644 --- a/net/dsa/port.c +++ b/net/dsa/port.c @@ -154,19 +154,67 @@ void dsa_port_bridge_leave(struct dsa_port *dp, struct net_device *br) dsa_port_set_state_now(dp, BR_STATE_FORWARDING); } +static bool dsa_port_can_apply_vlan_filtering(struct dsa_port *dp, + bool vlan_filtering) +{ + struct dsa_switch *ds = dp->ds; + int i; + + if (!ds->vlan_filtering_is_global) + return true; + + /* For cases where enabling/disabling VLAN awareness is global to the + * switch, we need to handle the case where multiple bridges span + * different ports of the same switch device and one of them has a + * different setting than what is being requested. + */ + for (i = 0; i < ds->num_ports; i++) { + struct net_device *other_bridge; + + other_bridge = dsa_to_port(ds, i)->bridge_dev; + if (!other_bridge) + continue; + /* If it's the same bridge, it also has same + * vlan_filtering setting => no need to check + */ + if (other_bridge == dp->bridge_dev) + continue; + if (br_vlan_enabled(other_bridge) != vlan_filtering) { + dev_err(ds->dev, "VLAN filtering is a global setting\n"); + return false; + } + } + return true; +} + int dsa_port_vlan_filtering(struct dsa_port *dp, bool vlan_filtering, struct switchdev_trans *trans) { struct dsa_switch *ds = dp->ds; + int err; /* bridge skips -EOPNOTSUPP, so skip the prepare phase */ if (switchdev_trans_ph_prepare(trans)) return 0; - if (ds->ops->port_vlan_filtering) - return ds->ops->port_vlan_filtering(ds, dp->index, - vlan_filtering); + if (!ds->ops->port_vlan_filtering) + return 0; + + if (!dsa_port_can_apply_vlan_filtering(dp, vlan_filtering)) + return -EINVAL; + + if (dsa_port_is_vlan_filtering(dp) == vlan_filtering) + return 0; + + err = ds->ops->port_vlan_filtering(ds, dp->index, + vlan_filtering); + if (err) + return err; + if (ds->vlan_filtering_is_global) + ds->vlan_filtering = vlan_filtering; + else + dp->vlan_filtering = vlan_filtering; return 0; } @@ -322,6 +370,39 @@ int dsa_port_vlan_del(struct dsa_port *dp, return 0; } +int dsa_port_vid_add(struct dsa_port *dp, u16 vid, u16 flags) +{ + struct switchdev_obj_port_vlan vlan = { + .obj.id = SWITCHDEV_OBJ_ID_PORT_VLAN, + .flags = flags, + .vid_begin = vid, + .vid_end = vid, + }; + struct switchdev_trans trans; + int err; + + trans.ph_prepare = true; + err = dsa_port_vlan_add(dp, &vlan, &trans); + if (err == -EOPNOTSUPP) + return 0; + + trans.ph_prepare = false; + return dsa_port_vlan_add(dp, &vlan, &trans); +} +EXPORT_SYMBOL(dsa_port_vid_add); + +int dsa_port_vid_del(struct dsa_port *dp, u16 vid) +{ + struct switchdev_obj_port_vlan vlan = { + .obj.id = SWITCHDEV_OBJ_ID_PORT_VLAN, + .vid_begin = vid, + .vid_end = vid, + }; + + return dsa_port_vlan_del(dp, &vlan); +} +EXPORT_SYMBOL(dsa_port_vid_del); + static struct phy_device *dsa_port_get_phy_device(struct dsa_port *dp) { struct device_node *phy_dn; diff --git a/net/dsa/slave.c b/net/dsa/slave.c index 093eef6f25995e29d6c06988148d36f598c5b4b7..fe7b6a62e8f1dba6857cdc051826d71ee67fdb8c 100644 --- a/net/dsa/slave.c +++ b/net/dsa/slave.c @@ -120,6 +120,9 @@ static int dsa_slave_close(struct net_device *dev) struct net_device *master = dsa_slave_to_master(dev); struct dsa_port *dp = dsa_slave_to_port(dev); + cancel_work_sync(&dp->xmit_work); + skb_queue_purge(&dp->xmit_queue); + phylink_stop(dp->pl); dsa_port_disable(dp); @@ -379,6 +382,13 @@ static int dsa_slave_get_port_parent_id(struct net_device *dev, struct dsa_switch *ds = dp->ds; struct dsa_switch_tree *dst = ds->dst; + /* For non-legacy ports, devlink is used and it takes + * care of the name generation. This ndo implementation + * should be removed with legacy support. + */ + if (dp->ds->devlink) + return -EOPNOTSUPP; + ppid->id_len = sizeof(dst->index); memcpy(&ppid->id, &dst->index, ppid->id_len); @@ -423,6 +433,24 @@ static void dsa_skb_tx_timestamp(struct dsa_slave_priv *p, kfree_skb(clone); } +netdev_tx_t dsa_enqueue_skb(struct sk_buff *skb, struct net_device *dev) +{ + /* SKB for netpoll still need to be mangled with the protocol-specific + * tag to be successfully transmitted + */ + if (unlikely(netpoll_tx_running(dev))) + return dsa_slave_netpoll_send_skb(dev, skb); + + /* Queue the SKB for transmission on the parent interface, but + * do not modify its EtherType + */ + skb->dev = dsa_slave_to_master(dev); + dev_queue_xmit(skb); + + return NETDEV_TX_OK; +} +EXPORT_SYMBOL_GPL(dsa_enqueue_skb); + static netdev_tx_t dsa_slave_xmit(struct sk_buff *skb, struct net_device *dev) { struct dsa_slave_priv *p = netdev_priv(dev); @@ -445,23 +473,37 @@ static netdev_tx_t dsa_slave_xmit(struct sk_buff *skb, struct net_device *dev) */ nskb = p->xmit(skb, dev); if (!nskb) { - kfree_skb(skb); + if (!DSA_SKB_CB(skb)->deferred_xmit) + kfree_skb(skb); return NETDEV_TX_OK; } - /* SKB for netpoll still need to be mangled with the protocol-specific - * tag to be successfully transmitted - */ - if (unlikely(netpoll_tx_running(dev))) - return dsa_slave_netpoll_send_skb(dev, nskb); + return dsa_enqueue_skb(nskb, dev); +} - /* Queue the SKB for transmission on the parent interface, but - * do not modify its EtherType - */ - nskb->dev = dsa_slave_to_master(dev); - dev_queue_xmit(nskb); +void *dsa_defer_xmit(struct sk_buff *skb, struct net_device *dev) +{ + struct dsa_port *dp = dsa_slave_to_port(dev); - return NETDEV_TX_OK; + DSA_SKB_CB(skb)->deferred_xmit = true; + + skb_queue_tail(&dp->xmit_queue, skb); + schedule_work(&dp->xmit_work); + return NULL; +} +EXPORT_SYMBOL_GPL(dsa_defer_xmit); + +static void dsa_port_xmit_work(struct work_struct *work) +{ + struct dsa_port *dp = container_of(work, struct dsa_port, xmit_work); + struct dsa_switch *ds = dp->ds; + struct sk_buff *skb; + + if (unlikely(!ds->ops->port_deferred_xmit)) + return; + + while ((skb = skb_dequeue(&dp->xmit_queue)) != NULL) + ds->ops->port_deferred_xmit(ds, dp->index, skb); } /* ethtool operations *******************************************************/ @@ -736,6 +778,13 @@ static int dsa_slave_get_phys_port_name(struct net_device *dev, { struct dsa_port *dp = dsa_slave_to_port(dev); + /* For non-legacy ports, devlink is used and it takes + * care of the name generation. This ndo implementation + * should be removed with legacy support. + */ + if (dp->ds->devlink) + return -EOPNOTSUPP; + if (snprintf(name, len, "p%d", dp->index) >= len) return -EINVAL; @@ -764,27 +813,25 @@ static int dsa_slave_add_cls_matchall(struct net_device *dev, struct dsa_mall_tc_entry *mall_tc_entry; __be16 protocol = cls->common.protocol; struct dsa_switch *ds = dp->ds; - struct net_device *to_dev; - const struct tc_action *a; + struct flow_action_entry *act; struct dsa_port *to_dp; int err = -EOPNOTSUPP; if (!ds->ops->port_mirror_add) return err; - if (!tcf_exts_has_one_action(cls->exts)) + if (!flow_offload_has_one_action(&cls->rule->action)) return err; - a = tcf_exts_first_action(cls->exts); + act = &cls->rule->action.entries[0]; - if (is_tcf_mirred_egress_mirror(a) && protocol == htons(ETH_P_ALL)) { + if (act->id == FLOW_ACTION_MIRRED && protocol == htons(ETH_P_ALL)) { struct dsa_mall_mirror_tc_entry *mirror; - to_dev = tcf_mirred_dev(a); - if (!to_dev) + if (!act->dev) return -EINVAL; - if (!dsa_slave_dev_check(to_dev)) + if (!dsa_slave_dev_check(act->dev)) return -EOPNOTSUPP; mall_tc_entry = kzalloc(sizeof(*mall_tc_entry), GFP_KERNEL); @@ -795,7 +842,7 @@ static int dsa_slave_add_cls_matchall(struct net_device *dev, mall_tc_entry->type = DSA_PORT_MALL_MIRROR; mirror = &mall_tc_entry->mirror; - to_dp = dsa_slave_to_port(to_dev); + to_dp = dsa_slave_to_port(act->dev); mirror->to_local_port = to_dp->index; mirror->ingress = ingress; @@ -987,13 +1034,6 @@ static int dsa_slave_vlan_rx_add_vid(struct net_device *dev, __be16 proto, u16 vid) { struct dsa_port *dp = dsa_slave_to_port(dev); - struct switchdev_obj_port_vlan vlan = { - .vid_begin = vid, - .vid_end = vid, - /* This API only allows programming tagged, non-PVID VIDs */ - .flags = 0, - }; - struct switchdev_trans trans; struct bridge_vlan_info info; int ret; @@ -1010,25 +1050,14 @@ static int dsa_slave_vlan_rx_add_vid(struct net_device *dev, __be16 proto, return -EBUSY; } - trans.ph_prepare = true; - ret = dsa_port_vlan_add(dp, &vlan, &trans); - if (ret == -EOPNOTSUPP) - return 0; - - trans.ph_prepare = false; - return dsa_port_vlan_add(dp, &vlan, &trans); + /* This API only allows programming tagged, non-PVID VIDs */ + return dsa_port_vid_add(dp, vid, 0); } static int dsa_slave_vlan_rx_kill_vid(struct net_device *dev, __be16 proto, u16 vid) { struct dsa_port *dp = dsa_slave_to_port(dev); - struct switchdev_obj_port_vlan vlan = { - .vid_begin = vid, - .vid_end = vid, - /* This API only allows programming tagged, non-PVID VIDs */ - .flags = 0, - }; struct bridge_vlan_info info; int ret; @@ -1045,7 +1074,7 @@ static int dsa_slave_vlan_rx_kill_vid(struct net_device *dev, __be16 proto, return -EBUSY; } - ret = dsa_port_vlan_del(dp, &vlan); + ret = dsa_port_vid_del(dp, vid); if (ret == -EOPNOTSUPP) ret = 0; @@ -1096,6 +1125,13 @@ int dsa_legacy_fdb_del(struct ndmsg *ndm, struct nlattr *tb[], return dsa_port_fdb_del(dp, addr, vid); } +static struct devlink_port *dsa_slave_get_devlink_port(struct net_device *dev) +{ + struct dsa_port *dp = dsa_slave_to_port(dev); + + return dp->ds->devlink ? &dp->devlink_port : NULL; +} + static const struct net_device_ops dsa_slave_netdev_ops = { .ndo_open = dsa_slave_open, .ndo_stop = dsa_slave_close, @@ -1119,6 +1155,7 @@ static const struct net_device_ops dsa_slave_netdev_ops = { .ndo_get_port_parent_id = dsa_slave_get_port_parent_id, .ndo_vlan_rx_add_vid = dsa_slave_vlan_rx_add_vid, .ndo_vlan_rx_kill_vid = dsa_slave_vlan_rx_kill_vid, + .ndo_get_devlink_port = dsa_slave_get_devlink_port, }; static struct device_type dsa_type = { @@ -1283,9 +1320,9 @@ static int dsa_slave_phy_setup(struct net_device *slave_dev) phy_flags = ds->ops->get_phy_flags(ds, dp->index); ret = phylink_of_phy_connect(dp->pl, port_dn, phy_flags); - if (ret == -ENODEV) { - /* We could not connect to a designated PHY or SFP, so use the - * switch internal MDIO bus instead + if (ret == -ENODEV && ds->slave_mii_bus) { + /* We could not connect to a designated PHY or SFP, so try to + * use the switch internal MDIO bus instead */ ret = dsa_slave_phy_connect(slave_dev, dp->index); if (ret) { @@ -1297,7 +1334,7 @@ static int dsa_slave_phy_setup(struct net_device *slave_dev) } } - return 0; + return ret; } static struct lock_class_key dsa_slave_netdev_xmit_lock_key; @@ -1316,6 +1353,9 @@ int dsa_slave_suspend(struct net_device *slave_dev) if (!netif_running(slave_dev)) return 0; + cancel_work_sync(&dp->xmit_work); + skb_queue_purge(&dp->xmit_queue); + netif_device_detach(slave_dev); rtnl_lock(); @@ -1378,7 +1418,10 @@ int dsa_slave_create(struct dsa_port *port) NETIF_F_HW_VLAN_CTAG_FILTER; slave_dev->hw_features |= NETIF_F_HW_TC; slave_dev->ethtool_ops = &dsa_slave_ethtool_ops; - eth_hw_addr_inherit(slave_dev, master); + if (!IS_ERR_OR_NULL(port->mac)) + ether_addr_copy(slave_dev->dev_addr, port->mac); + else + eth_hw_addr_inherit(slave_dev, master); slave_dev->priv_flags |= IFF_NO_QUEUE; slave_dev->netdev_ops = &dsa_slave_netdev_ops; slave_dev->min_mtu = 0; @@ -1400,6 +1443,8 @@ int dsa_slave_create(struct dsa_port *port) } p->dp = port; INIT_LIST_HEAD(&p->mall_tc_list); + INIT_WORK(&port->xmit_work, dsa_port_xmit_work); + skb_queue_head_init(&port->xmit_queue); p->xmit = cpu_dp->tag_ops->xmit; port->slave = slave_dev; diff --git a/net/dsa/switch.c b/net/dsa/switch.c index e1fae969aa737cc88556e5c5b0fb7f336571c3ce..7d8cd9bc0ecc8bbf31a246884c386729de1471a0 100644 --- a/net/dsa/switch.c +++ b/net/dsa/switch.c @@ -10,6 +10,7 @@ * (at your option) any later version. */ +#include #include #include #include @@ -71,6 +72,9 @@ static int dsa_switch_bridge_join(struct dsa_switch *ds, static int dsa_switch_bridge_leave(struct dsa_switch *ds, struct dsa_notifier_bridge_info *info) { + bool unset_vlan_filtering = br_vlan_enabled(info->br); + int err, i; + if (ds->index == info->sw_index && ds->ops->port_bridge_leave) ds->ops->port_bridge_leave(ds, info->port, info->br); @@ -78,6 +82,31 @@ static int dsa_switch_bridge_leave(struct dsa_switch *ds, ds->ops->crosschip_bridge_leave(ds, info->sw_index, info->port, info->br); + /* If the bridge was vlan_filtering, the bridge core doesn't trigger an + * event for changing vlan_filtering setting upon slave ports leaving + * it. That is a good thing, because that lets us handle it and also + * handle the case where the switch's vlan_filtering setting is global + * (not per port). When that happens, the correct moment to trigger the + * vlan_filtering callback is only when the last port left this bridge. + */ + if (unset_vlan_filtering && ds->vlan_filtering_is_global) { + for (i = 0; i < ds->num_ports; i++) { + if (i == info->port) + continue; + if (dsa_to_port(ds, i)->bridge_dev == info->br) { + unset_vlan_filtering = false; + break; + } + } + } + if (unset_vlan_filtering) { + struct switchdev_trans trans = {0}; + + err = dsa_port_vlan_filtering(&ds->ports[info->port], + false, &trans); + if (err && err != EOPNOTSUPP) + return err; + } return 0; } @@ -196,7 +225,7 @@ static int dsa_port_vlan_check(struct dsa_switch *ds, int port, if (!dp->bridge_dev) return err; - /* dsa_slave_vlan_rx_{add,kill}_vid() cannot use the prepare pharse and + /* dsa_slave_vlan_rx_{add,kill}_vid() cannot use the prepare phase and * already checks whether there is an overlapping bridge VLAN entry * with the same VID, so here we only need to check that if we are * adding a bridge VLAN entry there is not an overlapping VLAN device diff --git a/net/dsa/tag_8021q.c b/net/dsa/tag_8021q.c new file mode 100644 index 0000000000000000000000000000000000000000..8ae48c7e1e762523a7ec2e71867b1cd5aa0a652c --- /dev/null +++ b/net/dsa/tag_8021q.c @@ -0,0 +1,222 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2019, Vladimir Oltean + * + * This module is not a complete tagger implementation. It only provides + * primitives for taggers that rely on 802.1Q VLAN tags to use. The + * dsa_8021q_netdev_ops is registered for API compliance and not used + * directly by callers. + */ +#include +#include + +#include "dsa_priv.h" + +/* Allocating two VLAN tags per port - one for the RX VID and + * the other for the TX VID - see below + */ +#define DSA_8021Q_VID_RANGE (DSA_MAX_SWITCHES * DSA_MAX_PORTS) +#define DSA_8021Q_VID_BASE (VLAN_N_VID - 2 * DSA_8021Q_VID_RANGE - 1) +#define DSA_8021Q_RX_VID_BASE (DSA_8021Q_VID_BASE) +#define DSA_8021Q_TX_VID_BASE (DSA_8021Q_VID_BASE + DSA_8021Q_VID_RANGE) + +/* Returns the VID to be inserted into the frame from xmit for switch steering + * instructions on egress. Encodes switch ID and port ID. + */ +u16 dsa_8021q_tx_vid(struct dsa_switch *ds, int port) +{ + return DSA_8021Q_TX_VID_BASE + (DSA_MAX_PORTS * ds->index) + port; +} +EXPORT_SYMBOL_GPL(dsa_8021q_tx_vid); + +/* Returns the VID that will be installed as pvid for this switch port, sent as + * tagged egress towards the CPU port and decoded by the rcv function. + */ +u16 dsa_8021q_rx_vid(struct dsa_switch *ds, int port) +{ + return DSA_8021Q_RX_VID_BASE + (DSA_MAX_PORTS * ds->index) + port; +} +EXPORT_SYMBOL_GPL(dsa_8021q_rx_vid); + +/* Returns the decoded switch ID from the RX VID. */ +int dsa_8021q_rx_switch_id(u16 vid) +{ + return ((vid - DSA_8021Q_RX_VID_BASE) / DSA_MAX_PORTS); +} +EXPORT_SYMBOL_GPL(dsa_8021q_rx_switch_id); + +/* Returns the decoded port ID from the RX VID. */ +int dsa_8021q_rx_source_port(u16 vid) +{ + return ((vid - DSA_8021Q_RX_VID_BASE) % DSA_MAX_PORTS); +} +EXPORT_SYMBOL_GPL(dsa_8021q_rx_source_port); + +/* RX VLAN tagging (left) and TX VLAN tagging (right) setup shown for a single + * front-panel switch port (here swp0). + * + * Port identification through VLAN (802.1Q) tags has different requirements + * for it to work effectively: + * - On RX (ingress from network): each front-panel port must have a pvid + * that uniquely identifies it, and the egress of this pvid must be tagged + * towards the CPU port, so that software can recover the source port based + * on the VID in the frame. But this would only work for standalone ports; + * if bridged, this VLAN setup would break autonomous forwarding and would + * force all switched traffic to pass through the CPU. So we must also make + * the other front-panel ports members of this VID we're adding, albeit + * we're not making it their PVID (they'll still have their own). + * By the way - just because we're installing the same VID in multiple + * switch ports doesn't mean that they'll start to talk to one another, even + * while not bridged: the final forwarding decision is still an AND between + * the L2 forwarding information (which is limiting forwarding in this case) + * and the VLAN-based restrictions (of which there are none in this case, + * since all ports are members). + * - On TX (ingress from CPU and towards network) we are faced with a problem. + * If we were to tag traffic (from within DSA) with the port's pvid, all + * would be well, assuming the switch ports were standalone. Frames would + * have no choice but to be directed towards the correct front-panel port. + * But because we also want the RX VLAN to not break bridging, then + * inevitably that means that we have to give them a choice (of what + * front-panel port to go out on), and therefore we cannot steer traffic + * based on the RX VID. So what we do is simply install one more VID on the + * front-panel and CPU ports, and profit off of the fact that steering will + * work just by virtue of the fact that there is only one other port that's + * a member of the VID we're tagging the traffic with - the desired one. + * + * So at the end, each front-panel port will have one RX VID (also the PVID), + * the RX VID of all other front-panel ports, and one TX VID. Whereas the CPU + * port will have the RX and TX VIDs of all front-panel ports, and on top of + * that, is also tagged-input and tagged-output (VLAN trunk). + * + * CPU port CPU port + * +-------------+-----+-------------+ +-------------+-----+-------------+ + * | RX VID | | | | TX VID | | | + * | of swp0 | | | | of swp0 | | | + * | +-----+ | | +-----+ | + * | ^ T | | | Tagged | + * | | | | | ingress | + * | +-------+---+---+-------+ | | +-----------+ | + * | | | | | | | | Untagged | + * | | U v U v U v | | v egress | + * | +-----+ +-----+ +-----+ +-----+ | | +-----+ +-----+ +-----+ +-----+ | + * | | | | | | | | | | | | | | | | | | | | + * | |PVID | | | | | | | | | | | | | | | | | | + * +-+-----+-+-----+-+-----+-+-----+-+ +-+-----+-+-----+-+-----+-+-----+-+ + * swp0 swp1 swp2 swp3 swp0 swp1 swp2 swp3 + */ +int dsa_port_setup_8021q_tagging(struct dsa_switch *ds, int port, bool enabled) +{ + int upstream = dsa_upstream_port(ds, port); + struct dsa_port *dp = &ds->ports[port]; + struct dsa_port *upstream_dp = &ds->ports[upstream]; + u16 rx_vid = dsa_8021q_rx_vid(ds, port); + u16 tx_vid = dsa_8021q_tx_vid(ds, port); + int i, err; + + /* The CPU port is implicitly configured by + * configuring the front-panel ports + */ + if (!dsa_is_user_port(ds, port)) + return 0; + + /* Add this user port's RX VID to the membership list of all others + * (including itself). This is so that bridging will not be hindered. + * L2 forwarding rules still take precedence when there are no VLAN + * restrictions, so there are no concerns about leaking traffic. + */ + for (i = 0; i < ds->num_ports; i++) { + struct dsa_port *other_dp = &ds->ports[i]; + u16 flags; + + if (i == upstream) + /* CPU port needs to see this port's RX VID + * as tagged egress. + */ + flags = 0; + else if (i == port) + /* The RX VID is pvid on this port */ + flags = BRIDGE_VLAN_INFO_UNTAGGED | + BRIDGE_VLAN_INFO_PVID; + else + /* The RX VID is a regular VLAN on all others */ + flags = BRIDGE_VLAN_INFO_UNTAGGED; + + if (enabled) + err = dsa_port_vid_add(other_dp, rx_vid, flags); + else + err = dsa_port_vid_del(other_dp, rx_vid); + if (err) { + dev_err(ds->dev, "Failed to apply RX VID %d to port %d: %d\n", + rx_vid, port, err); + return err; + } + } + /* Finally apply the TX VID on this port and on the CPU port */ + if (enabled) + err = dsa_port_vid_add(dp, tx_vid, BRIDGE_VLAN_INFO_UNTAGGED); + else + err = dsa_port_vid_del(dp, tx_vid); + if (err) { + dev_err(ds->dev, "Failed to apply TX VID %d on port %d: %d\n", + tx_vid, port, err); + return err; + } + if (enabled) + err = dsa_port_vid_add(upstream_dp, tx_vid, 0); + else + err = dsa_port_vid_del(upstream_dp, tx_vid); + if (err) { + dev_err(ds->dev, "Failed to apply TX VID %d on port %d: %d\n", + tx_vid, upstream, err); + return err; + } + + return 0; +} +EXPORT_SYMBOL_GPL(dsa_port_setup_8021q_tagging); + +struct sk_buff *dsa_8021q_xmit(struct sk_buff *skb, struct net_device *netdev, + u16 tpid, u16 tci) +{ + /* skb->data points at skb_mac_header, which + * is fine for vlan_insert_tag. + */ + return vlan_insert_tag(skb, htons(tpid), tci); +} +EXPORT_SYMBOL_GPL(dsa_8021q_xmit); + +struct sk_buff *dsa_8021q_rcv(struct sk_buff *skb, struct net_device *netdev, + struct packet_type *pt, u16 *tpid, u16 *tci) +{ + struct vlan_ethhdr *tag; + + if (unlikely(!pskb_may_pull(skb, VLAN_HLEN))) + return NULL; + + tag = vlan_eth_hdr(skb); + *tpid = ntohs(tag->h_vlan_proto); + *tci = ntohs(tag->h_vlan_TCI); + + /* skb->data points in the middle of the VLAN tag, + * after tpid and before tci. This is because so far, + * ETH_HLEN (DMAC, SMAC, EtherType) bytes were pulled. + * There are 2 bytes of VLAN tag left in skb->data, and upper + * layers expect the 'real' EtherType to be consumed as well. + * Coincidentally, a VLAN header is also of the same size as + * the number of bytes that need to be pulled. + */ + skb_pull_rcsum(skb, VLAN_HLEN); + + return skb; +} +EXPORT_SYMBOL_GPL(dsa_8021q_rcv); + +static const struct dsa_device_ops dsa_8021q_netdev_ops = { + .name = "8021q", + .proto = DSA_TAG_PROTO_8021Q, + .overhead = VLAN_HLEN, +}; + +MODULE_LICENSE("GPL v2"); +MODULE_ALIAS_DSA_TAG_DRIVER(DSA_TAG_PROTO_8021Q); + +module_dsa_tag_driver(dsa_8021q_netdev_ops); diff --git a/net/dsa/tag_brcm.c b/net/dsa/tag_brcm.c index 4aa1d368a5ae6b983889e608f97d10a54063c301..d52db5f2c72171cdc8f5a1b389642362170a4913 100644 --- a/net/dsa/tag_brcm.c +++ b/net/dsa/tag_brcm.c @@ -1,12 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0+ /* * Broadcom tag support * * Copyright (C) 2014 Broadcom Corporation - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. */ #include @@ -59,6 +55,9 @@ #define BRCM_EG_TC_MASK 0x7 #define BRCM_EG_PID_MASK 0x1f +#if IS_ENABLED(CONFIG_NET_DSA_TAG_BRCM) || \ + IS_ENABLED(CONFIG_NET_DSA_TAG_BRCM_PREPEND) + static struct sk_buff *brcm_tag_xmit_ll(struct sk_buff *skb, struct net_device *dev, unsigned int offset) @@ -143,8 +142,9 @@ static struct sk_buff *brcm_tag_rcv_ll(struct sk_buff *skb, return skb; } +#endif -#ifdef CONFIG_NET_DSA_TAG_BRCM +#if IS_ENABLED(CONFIG_NET_DSA_TAG_BRCM) static struct sk_buff *brcm_tag_xmit(struct sk_buff *skb, struct net_device *dev) { @@ -171,14 +171,19 @@ static struct sk_buff *brcm_tag_rcv(struct sk_buff *skb, struct net_device *dev, return nskb; } -const struct dsa_device_ops brcm_netdev_ops = { +static const struct dsa_device_ops brcm_netdev_ops = { + .name = "brcm", + .proto = DSA_TAG_PROTO_BRCM, .xmit = brcm_tag_xmit, .rcv = brcm_tag_rcv, .overhead = BRCM_TAG_LEN, }; + +DSA_TAG_DRIVER(brcm_netdev_ops); +MODULE_ALIAS_DSA_TAG_DRIVER(DSA_TAG_PROTO_BRCM); #endif -#ifdef CONFIG_NET_DSA_TAG_BRCM_PREPEND +#if IS_ENABLED(CONFIG_NET_DSA_TAG_BRCM_PREPEND) static struct sk_buff *brcm_tag_xmit_prepend(struct sk_buff *skb, struct net_device *dev) { @@ -194,9 +199,27 @@ static struct sk_buff *brcm_tag_rcv_prepend(struct sk_buff *skb, return brcm_tag_rcv_ll(skb, dev, pt, ETH_HLEN); } -const struct dsa_device_ops brcm_prepend_netdev_ops = { +static const struct dsa_device_ops brcm_prepend_netdev_ops = { + .name = "brcm-prepend", + .proto = DSA_TAG_PROTO_BRCM_PREPEND, .xmit = brcm_tag_xmit_prepend, .rcv = brcm_tag_rcv_prepend, .overhead = BRCM_TAG_LEN, }; #endif + +DSA_TAG_DRIVER(brcm_prepend_netdev_ops); +MODULE_ALIAS_DSA_TAG_DRIVER(DSA_TAG_PROTO_BRCM_PREPEND); + +static struct dsa_tag_driver *dsa_tag_driver_array[] = { +#if IS_ENABLED(CONFIG_NET_DSA_TAG_BRCM) + &DSA_TAG_DRIVER_NAME(brcm_netdev_ops), +#endif +#if IS_ENABLED(CONFIG_NET_DSA_TAG_BRCM_PREPEND) + &DSA_TAG_DRIVER_NAME(brcm_prepend_netdev_ops), +#endif +}; + +module_dsa_tag_drivers(dsa_tag_driver_array); + +MODULE_LICENSE("GPL"); diff --git a/net/dsa/tag_dsa.c b/net/dsa/tag_dsa.c index 67ff3fae18d8cf05de24578a79df22c7becaac0a..7ddec9794477c1f0798ea58bf85740843bf3d960 100644 --- a/net/dsa/tag_dsa.c +++ b/net/dsa/tag_dsa.c @@ -1,11 +1,7 @@ +// SPDX-License-Identifier: GPL-2.0+ /* * net/dsa/tag_dsa.c - (Non-ethertype) DSA tagging * Copyright (c) 2008-2009 Marvell Semiconductor - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. */ #include @@ -154,9 +150,16 @@ static int dsa_tag_flow_dissect(const struct sk_buff *skb, __be16 *proto, return 0; } -const struct dsa_device_ops dsa_netdev_ops = { +static const struct dsa_device_ops dsa_netdev_ops = { + .name = "dsa", + .proto = DSA_TAG_PROTO_DSA, .xmit = dsa_xmit, .rcv = dsa_rcv, .flow_dissect = dsa_tag_flow_dissect, .overhead = DSA_HLEN, }; + +MODULE_LICENSE("GPL"); +MODULE_ALIAS_DSA_TAG_DRIVER(DSA_TAG_PROTO_DSA); + +module_dsa_tag_driver(dsa_netdev_ops); diff --git a/net/dsa/tag_edsa.c b/net/dsa/tag_edsa.c index 234585ec116ef2bed26bb0126f83c26b3abc5e47..e8eaa804ccb9ef68c4603fa88c24a12930f7fb3a 100644 --- a/net/dsa/tag_edsa.c +++ b/net/dsa/tag_edsa.c @@ -1,11 +1,7 @@ +// SPDX-License-Identifier: GPL-2.0+ /* * net/dsa/tag_edsa.c - Ethertype DSA tagging * Copyright (c) 2008-2009 Marvell Semiconductor - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. */ #include @@ -173,9 +169,16 @@ static int edsa_tag_flow_dissect(const struct sk_buff *skb, __be16 *proto, return 0; } -const struct dsa_device_ops edsa_netdev_ops = { +static const struct dsa_device_ops edsa_netdev_ops = { + .name = "edsa", + .proto = DSA_TAG_PROTO_EDSA, .xmit = edsa_xmit, .rcv = edsa_rcv, .flow_dissect = edsa_tag_flow_dissect, .overhead = EDSA_HLEN, }; + +MODULE_LICENSE("GPL"); +MODULE_ALIAS_DSA_TAG_DRIVER(DSA_TAG_PROTO_EDSA); + +module_dsa_tag_driver(edsa_netdev_ops); diff --git a/net/dsa/tag_gswip.c b/net/dsa/tag_gswip.c index cb6f82ffe5eb5d54c5697d27a59e849bfc2b753a..b678160bbd66e5992c7b7b28a819c25f6dd3ec50 100644 --- a/net/dsa/tag_gswip.c +++ b/net/dsa/tag_gswip.c @@ -103,8 +103,15 @@ static struct sk_buff *gswip_tag_rcv(struct sk_buff *skb, return skb; } -const struct dsa_device_ops gswip_netdev_ops = { +static const struct dsa_device_ops gswip_netdev_ops = { + .name = "gwsip", + .proto = DSA_TAG_PROTO_GSWIP, .xmit = gswip_tag_xmit, .rcv = gswip_tag_rcv, .overhead = GSWIP_RX_HEADER_LEN, }; + +MODULE_LICENSE("GPL"); +MODULE_ALIAS_DSA_TAG_DRIVER(DSA_TAG_PROTO_GSWIP); + +module_dsa_tag_driver(gswip_netdev_ops); diff --git a/net/dsa/tag_ksz.c b/net/dsa/tag_ksz.c index de246c93d3bbd752a8ca432edfc0b179a7893be7..b4872b87d4a6b4971d9ca815be6de3c93ffaab06 100644 --- a/net/dsa/tag_ksz.c +++ b/net/dsa/tag_ksz.c @@ -1,11 +1,7 @@ +// SPDX-License-Identifier: GPL-2.0+ /* * net/dsa/tag_ksz.c - Microchip KSZ Switch tag format handling * Copyright (c) 2017 Microchip Technology - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. */ #include @@ -137,12 +133,17 @@ static struct sk_buff *ksz9477_rcv(struct sk_buff *skb, struct net_device *dev, return ksz_common_rcv(skb, dev, port, len); } -const struct dsa_device_ops ksz9477_netdev_ops = { +static const struct dsa_device_ops ksz9477_netdev_ops = { + .name = "ksz9477", + .proto = DSA_TAG_PROTO_KSZ9477, .xmit = ksz9477_xmit, .rcv = ksz9477_rcv, .overhead = KSZ9477_INGRESS_TAG_LEN, }; +DSA_TAG_DRIVER(ksz9477_netdev_ops); +MODULE_ALIAS_DSA_TAG_DRIVER(DSA_TAG_PROTO_KSZ9477); + #define KSZ9893_TAIL_TAG_OVERRIDE BIT(5) #define KSZ9893_TAIL_TAG_LOOKUP BIT(6) @@ -170,8 +171,22 @@ static struct sk_buff *ksz9893_xmit(struct sk_buff *skb, return nskb; } -const struct dsa_device_ops ksz9893_netdev_ops = { +static const struct dsa_device_ops ksz9893_netdev_ops = { + .name = "ksz9893", + .proto = DSA_TAG_PROTO_KSZ9893, .xmit = ksz9893_xmit, .rcv = ksz9477_rcv, .overhead = KSZ_INGRESS_TAG_LEN, }; + +DSA_TAG_DRIVER(ksz9893_netdev_ops); +MODULE_ALIAS_DSA_TAG_DRIVER(DSA_TAG_PROTO_KSZ9893); + +static struct dsa_tag_driver *dsa_tag_driver_array[] = { + &DSA_TAG_DRIVER_NAME(ksz9477_netdev_ops), + &DSA_TAG_DRIVER_NAME(ksz9893_netdev_ops), +}; + +module_dsa_tag_drivers(dsa_tag_driver_array); + +MODULE_LICENSE("GPL"); diff --git a/net/dsa/tag_lan9303.c b/net/dsa/tag_lan9303.c index f48889e46ff7e8922962f0a3c2ed19d5086a763c..eb0e7a32e53dcdcd3be18512203d33698201617c 100644 --- a/net/dsa/tag_lan9303.c +++ b/net/dsa/tag_lan9303.c @@ -1,15 +1,6 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (C) 2017 Pengutronix, Juergen Borleis - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * */ #include #include @@ -137,8 +128,15 @@ static struct sk_buff *lan9303_rcv(struct sk_buff *skb, struct net_device *dev, return skb; } -const struct dsa_device_ops lan9303_netdev_ops = { +static const struct dsa_device_ops lan9303_netdev_ops = { + .name = "lan9303", + .proto = DSA_TAG_PROTO_LAN9303, .xmit = lan9303_xmit, .rcv = lan9303_rcv, .overhead = LAN9303_TAG_LEN, }; + +MODULE_LICENSE("GPL"); +MODULE_ALIAS_DSA_TAG_DRIVER(DSA_TAG_PROTO_LAN9303); + +module_dsa_tag_driver(lan9303_netdev_ops); diff --git a/net/dsa/tag_mtk.c b/net/dsa/tag_mtk.c index f39f4dfeda34c17a9741dd371a9d35b062b38cab..b5705cba831848189c8981e1027e0ad62eba0efe 100644 --- a/net/dsa/tag_mtk.c +++ b/net/dsa/tag_mtk.c @@ -1,15 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Mediatek DSA Tag support * Copyright (C) 2017 Landen Chao * Sean Wang - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 and - * only version 2 as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include @@ -105,9 +98,16 @@ static int mtk_tag_flow_dissect(const struct sk_buff *skb, __be16 *proto, return 0; } -const struct dsa_device_ops mtk_netdev_ops = { +static const struct dsa_device_ops mtk_netdev_ops = { + .name = "mtk", + .proto = DSA_TAG_PROTO_MTK, .xmit = mtk_tag_xmit, .rcv = mtk_tag_rcv, .flow_dissect = mtk_tag_flow_dissect, .overhead = MTK_HDR_LEN, }; + +MODULE_LICENSE("GPL"); +MODULE_ALIAS_DSA_TAG_DRIVER(DSA_TAG_PROTO_MTK); + +module_dsa_tag_driver(mtk_netdev_ops); diff --git a/net/dsa/tag_qca.c b/net/dsa/tag_qca.c index 85c22ada47449d580ee2a175c0729f4bad2cad61..c95885215525193517a6745040e39a046bf6c494 100644 --- a/net/dsa/tag_qca.c +++ b/net/dsa/tag_qca.c @@ -1,14 +1,6 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2015, The Linux Foundation. All rights reserved. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 and - * only version 2 as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include @@ -107,9 +99,16 @@ static int qca_tag_flow_dissect(const struct sk_buff *skb, __be16 *proto, return 0; } -const struct dsa_device_ops qca_netdev_ops = { +static const struct dsa_device_ops qca_netdev_ops = { + .name = "qca", + .proto = DSA_TAG_PROTO_QCA, .xmit = qca_tag_xmit, .rcv = qca_tag_rcv, .flow_dissect = qca_tag_flow_dissect, .overhead = QCA_HDR_LEN, }; + +MODULE_LICENSE("GPL"); +MODULE_ALIAS_DSA_TAG_DRIVER(DSA_TAG_PROTO_QCA); + +module_dsa_tag_driver(qca_netdev_ops); diff --git a/net/dsa/tag_sja1105.c b/net/dsa/tag_sja1105.c new file mode 100644 index 0000000000000000000000000000000000000000..969402c7dbf1af96f441d8ae0fa5675ab86b2a63 --- /dev/null +++ b/net/dsa/tag_sja1105.c @@ -0,0 +1,131 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2019, Vladimir Oltean + */ +#include +#include +#include +#include +#include "dsa_priv.h" + +/* Similar to is_link_local_ether_addr(hdr->h_dest) but also covers PTP */ +static inline bool sja1105_is_link_local(const struct sk_buff *skb) +{ + const struct ethhdr *hdr = eth_hdr(skb); + u64 dmac = ether_addr_to_u64(hdr->h_dest); + + if ((dmac & SJA1105_LINKLOCAL_FILTER_A_MASK) == + SJA1105_LINKLOCAL_FILTER_A) + return true; + if ((dmac & SJA1105_LINKLOCAL_FILTER_B_MASK) == + SJA1105_LINKLOCAL_FILTER_B) + return true; + return false; +} + +/* This is the first time the tagger sees the frame on RX. + * Figure out if we can decode it, and if we can, annotate skb->cb with how we + * plan to do that, so we don't need to check again in the rcv function. + */ +static bool sja1105_filter(const struct sk_buff *skb, struct net_device *dev) +{ + if (sja1105_is_link_local(skb)) { + SJA1105_SKB_CB(skb)->type = SJA1105_FRAME_TYPE_LINK_LOCAL; + return true; + } + if (!dsa_port_is_vlan_filtering(dev->dsa_ptr)) { + SJA1105_SKB_CB(skb)->type = SJA1105_FRAME_TYPE_NORMAL; + return true; + } + return false; +} + +static struct sk_buff *sja1105_xmit(struct sk_buff *skb, + struct net_device *netdev) +{ + struct dsa_port *dp = dsa_slave_to_port(netdev); + struct dsa_switch *ds = dp->ds; + u16 tx_vid = dsa_8021q_tx_vid(ds, dp->index); + u8 pcp = skb->priority; + + /* Transmitting management traffic does not rely upon switch tagging, + * but instead SPI-installed management routes. Part 2 of this + * is the .port_deferred_xmit driver callback. + */ + if (unlikely(sja1105_is_link_local(skb))) + return dsa_defer_xmit(skb, netdev); + + /* If we are under a vlan_filtering bridge, IP termination on + * switch ports based on 802.1Q tags is simply too brittle to + * be passable. So just defer to the dsa_slave_notag_xmit + * implementation. + */ + if (dsa_port_is_vlan_filtering(dp)) + return skb; + + return dsa_8021q_xmit(skb, netdev, ETH_P_SJA1105, + ((pcp << VLAN_PRIO_SHIFT) | tx_vid)); +} + +static struct sk_buff *sja1105_rcv(struct sk_buff *skb, + struct net_device *netdev, + struct packet_type *pt) +{ + struct ethhdr *hdr = eth_hdr(skb); + u64 source_port, switch_id; + struct sk_buff *nskb; + u16 tpid, vid, tci; + bool is_tagged; + + nskb = dsa_8021q_rcv(skb, netdev, pt, &tpid, &tci); + is_tagged = (nskb && tpid == ETH_P_SJA1105); + + skb->priority = (tci & VLAN_PRIO_MASK) >> VLAN_PRIO_SHIFT; + vid = tci & VLAN_VID_MASK; + + skb->offload_fwd_mark = 1; + + if (SJA1105_SKB_CB(skb)->type == SJA1105_FRAME_TYPE_LINK_LOCAL) { + /* Management traffic path. Switch embeds the switch ID and + * port ID into bytes of the destination MAC, courtesy of + * the incl_srcpt options. + */ + source_port = hdr->h_dest[3]; + switch_id = hdr->h_dest[4]; + /* Clear the DMAC bytes that were mangled by the switch */ + hdr->h_dest[3] = 0; + hdr->h_dest[4] = 0; + } else { + /* Normal traffic path. */ + source_port = dsa_8021q_rx_source_port(vid); + switch_id = dsa_8021q_rx_switch_id(vid); + } + + skb->dev = dsa_master_find_slave(netdev, switch_id, source_port); + if (!skb->dev) { + netdev_warn(netdev, "Couldn't decode source port\n"); + return NULL; + } + + /* Delete/overwrite fake VLAN header, DSA expects to not find + * it there, see dsa_switch_rcv: skb_push(skb, ETH_HLEN). + */ + if (is_tagged) + memmove(skb->data - ETH_HLEN, skb->data - ETH_HLEN - VLAN_HLEN, + ETH_HLEN - VLAN_HLEN); + + return skb; +} + +static struct dsa_device_ops sja1105_netdev_ops = { + .name = "sja1105", + .proto = DSA_TAG_PROTO_SJA1105, + .xmit = sja1105_xmit, + .rcv = sja1105_rcv, + .filter = sja1105_filter, + .overhead = VLAN_HLEN, +}; + +MODULE_LICENSE("GPL v2"); +MODULE_ALIAS_DSA_TAG_DRIVER(DSA_TAG_PROTO_SJA1105); + +module_dsa_tag_driver(sja1105_netdev_ops); diff --git a/net/dsa/tag_trailer.c b/net/dsa/tag_trailer.c index b40756ed6e574c4afe85e4000321e4d86c4269f3..4f8ab62f0208475f910771abdeac7af8603b019e 100644 --- a/net/dsa/tag_trailer.c +++ b/net/dsa/tag_trailer.c @@ -1,11 +1,7 @@ +// SPDX-License-Identifier: GPL-2.0+ /* * net/dsa/tag_trailer.c - Trailer tag format handling * Copyright (c) 2008-2009 Marvell Semiconductor - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. */ #include @@ -81,8 +77,15 @@ static struct sk_buff *trailer_rcv(struct sk_buff *skb, struct net_device *dev, return skb; } -const struct dsa_device_ops trailer_netdev_ops = { +static const struct dsa_device_ops trailer_netdev_ops = { + .name = "trailer", + .proto = DSA_TAG_PROTO_TRAILER, .xmit = trailer_xmit, .rcv = trailer_rcv, .overhead = 4, }; + +MODULE_LICENSE("GPL"); +MODULE_ALIAS_DSA_TAG_DRIVER(DSA_TAG_PROTO_TRAILER); + +module_dsa_tag_driver(trailer_netdev_ops); diff --git a/net/ethernet/eth.c b/net/ethernet/eth.c index f7a3d7a171c78419421b3976d9d510bbd701236c..4b2b222377ace9235294cbbc7cf6456485723769 100644 --- a/net/ethernet/eth.c +++ b/net/ethernet/eth.c @@ -119,13 +119,14 @@ EXPORT_SYMBOL(eth_header); /** * eth_get_headlen - determine the length of header for an ethernet frame + * @dev: pointer to network device * @data: pointer to start of frame * @len: total length of frame * * Make a best effort attempt to pull the length for all of the headers for * a given frame in a linear buffer. */ -u32 eth_get_headlen(void *data, unsigned int len) +u32 eth_get_headlen(const struct net_device *dev, void *data, unsigned int len) { const unsigned int flags = FLOW_DISSECTOR_F_PARSE_1ST_FRAG; const struct ethhdr *eth = (const struct ethhdr *)data; @@ -136,8 +137,9 @@ u32 eth_get_headlen(void *data, unsigned int len) return len; /* parse any remaining L2/L3 headers, check for L4 */ - if (!skb_flow_dissect_flow_keys_basic(NULL, &keys, data, eth->h_proto, - sizeof(*eth), len, flags)) + if (!skb_flow_dissect_flow_keys_basic(dev_net(dev), NULL, &keys, data, + eth->h_proto, sizeof(*eth), + len, flags)) return max_t(u32, keys.control.thoff, sizeof(*eth)); /* parse for any L4 headers */ @@ -183,8 +185,12 @@ __be16 eth_type_trans(struct sk_buff *skb, struct net_device *dev) * at all, so we check here whether one of those tagging * variants has been configured on the receiving interface, * and if so, set skb->protocol without looking at the packet. + * The DSA tagging protocol may be able to decode some but not all + * traffic (for example only for management). In that case give it the + * option to filter the packets from which it can decode source port + * information. */ - if (unlikely(netdev_uses_dsa(dev))) + if (unlikely(netdev_uses_dsa(dev)) && dsa_can_decode(skb, dev)) return htons(ETH_P_XDSA); if (likely(eth_proto_is_802_3(eth->h_proto))) @@ -554,7 +560,7 @@ int eth_platform_get_mac_address(struct device *dev, u8 *mac_addr) addr = NULL; if (dp) addr = of_get_mac_address(dp); - if (!addr) + if (IS_ERR_OR_NULL(addr)) addr = arch_get_platform_mac_address(); if (!addr) diff --git a/net/hsr/Makefile b/net/hsr/Makefile index 9ae972a820f4683efb16eb8fd2c5d69e75c12049..e45757fc477f0b26092e4bb5fe8f1cccd3ea733a 100644 --- a/net/hsr/Makefile +++ b/net/hsr/Makefile @@ -6,3 +6,4 @@ obj-$(CONFIG_HSR) += hsr.o hsr-y := hsr_main.o hsr_framereg.o hsr_device.o \ hsr_netlink.o hsr_slave.o hsr_forward.o +hsr-$(CONFIG_DEBUG_FS) += hsr_debugfs.o diff --git a/net/hsr/hsr_debugfs.c b/net/hsr/hsr_debugfs.c new file mode 100644 index 0000000000000000000000000000000000000000..94447974a3c05d735cad2d26648f432a45e3697a --- /dev/null +++ b/net/hsr/hsr_debugfs.c @@ -0,0 +1,119 @@ +/* + * hsr_debugfs code + * Copyright (C) 2019 Texas Instruments Incorporated + * + * Author(s): + * Murali Karicheri + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License as + * published by the Free Software Foundation version 2. + * + * This program is distributed "as is" WITHOUT ANY WARRANTY of any + * kind, whether express or implied; without even the implied warranty + * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + */ +#include +#include +#include +#include "hsr_main.h" +#include "hsr_framereg.h" + +static void print_mac_address(struct seq_file *sfp, unsigned char *mac) +{ + seq_printf(sfp, "%02x:%02x:%02x:%02x:%02x:%02x:", + mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]); +} + +/* hsr_node_table_show - Formats and prints node_table entries */ +static int +hsr_node_table_show(struct seq_file *sfp, void *data) +{ + struct hsr_priv *priv = (struct hsr_priv *)sfp->private; + struct hsr_node *node; + + seq_puts(sfp, "Node Table entries\n"); + seq_puts(sfp, "MAC-Address-A, MAC-Address-B, time_in[A], "); + seq_puts(sfp, "time_in[B], Address-B port\n"); + rcu_read_lock(); + list_for_each_entry_rcu(node, &priv->node_db, mac_list) { + /* skip self node */ + if (hsr_addr_is_self(priv, node->macaddress_A)) + continue; + print_mac_address(sfp, &node->macaddress_A[0]); + seq_puts(sfp, " "); + print_mac_address(sfp, &node->macaddress_B[0]); + seq_printf(sfp, "0x%lx, ", node->time_in[HSR_PT_SLAVE_A]); + seq_printf(sfp, "0x%lx ", node->time_in[HSR_PT_SLAVE_B]); + seq_printf(sfp, "0x%x\n", node->addr_B_port); + } + rcu_read_unlock(); + return 0; +} + +/* hsr_node_table_open - Open the node_table file + * + * Description: + * This routine opens a debugfs file node_table of specific hsr device + */ +static int +hsr_node_table_open(struct inode *inode, struct file *filp) +{ + return single_open(filp, hsr_node_table_show, inode->i_private); +} + +static const struct file_operations hsr_fops = { + .owner = THIS_MODULE, + .open = hsr_node_table_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; + +/* hsr_debugfs_init - create hsr node_table file for dumping + * the node table + * + * Description: + * When debugfs is configured this routine sets up the node_table file per + * hsr device for dumping the node_table entries + */ +int hsr_debugfs_init(struct hsr_priv *priv, struct net_device *hsr_dev) +{ + int rc = -1; + struct dentry *de = NULL; + + de = debugfs_create_dir(hsr_dev->name, NULL); + if (!de) { + pr_err("Cannot create hsr debugfs root\n"); + return rc; + } + + priv->node_tbl_root = de; + + de = debugfs_create_file("node_table", S_IFREG | 0444, + priv->node_tbl_root, priv, + &hsr_fops); + if (!de) { + pr_err("Cannot create hsr node_table directory\n"); + return rc; + } + priv->node_tbl_file = de; + + return 0; +} + +/* hsr_debugfs_term - Tear down debugfs intrastructure + * + * Description: + * When Debufs is configured this routine removes debugfs file system + * elements that are specific to hsr + */ +void +hsr_debugfs_term(struct hsr_priv *priv) +{ + debugfs_remove(priv->node_tbl_file); + priv->node_tbl_file = NULL; + debugfs_remove(priv->node_tbl_root); + priv->node_tbl_root = NULL; +} diff --git a/net/hsr/hsr_device.c b/net/hsr/hsr_device.c index a97bf326b231fe05c7562ca49279fc3fda9d53c0..15c72065df79103fa78ee47db8f076a62529b055 100644 --- a/net/hsr/hsr_device.c +++ b/net/hsr/hsr_device.c @@ -1,9 +1,5 @@ +// SPDX-License-Identifier: GPL-2.0 /* Copyright 2011-2014 Autronica Fire and Security AS - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License as published by the Free - * Software Foundation; either version 2 of the License, or (at your option) - * any later version. * * Author(s): * 2011-2014 Arvid Brodin, arvid.brodin@alten.se @@ -23,7 +19,6 @@ #include "hsr_main.h" #include "hsr_forward.h" - static bool is_admin_up(struct net_device *dev) { return dev && (dev->flags & IFF_UP); @@ -68,7 +63,7 @@ static bool hsr_check_carrier(struct hsr_port *master) rcu_read_lock(); hsr_for_each_port(master->hsr, port) - if ((port->type != HSR_PT_MASTER) && is_slave_up(port->dev)) { + if (port->type != HSR_PT_MASTER && is_slave_up(port->dev)) { has_carrier = true; break; } @@ -82,7 +77,6 @@ static bool hsr_check_carrier(struct hsr_port *master) return has_carrier; } - static void hsr_check_announce(struct net_device *hsr_dev, unsigned char old_operstate) { @@ -90,15 +84,14 @@ static void hsr_check_announce(struct net_device *hsr_dev, hsr = netdev_priv(hsr_dev); - if ((hsr_dev->operstate == IF_OPER_UP) - && (old_operstate != IF_OPER_UP)) { + if (hsr_dev->operstate == IF_OPER_UP && old_operstate != IF_OPER_UP) { /* Went up */ hsr->announce_count = 0; mod_timer(&hsr->announce_timer, jiffies + msecs_to_jiffies(HSR_ANNOUNCE_INTERVAL)); } - if ((hsr_dev->operstate != IF_OPER_UP) && (old_operstate == IF_OPER_UP)) + if (hsr_dev->operstate != IF_OPER_UP && old_operstate == IF_OPER_UP) /* Went down */ del_timer(&hsr->announce_timer); } @@ -136,7 +129,6 @@ int hsr_get_max_mtu(struct hsr_priv *hsr) return mtu_max - HSR_HLEN; } - static int hsr_dev_change_mtu(struct net_device *dev, int new_mtu) { struct hsr_priv *hsr; @@ -191,14 +183,12 @@ static int hsr_dev_open(struct net_device *dev) return 0; } - static int hsr_dev_close(struct net_device *dev) { /* Nothing to do here. */ return 0; } - static netdev_features_t hsr_features_recompute(struct hsr_priv *hsr, netdev_features_t features) { @@ -231,7 +221,6 @@ static netdev_features_t hsr_fix_features(struct net_device *dev, return hsr_features_recompute(hsr, features); } - static int hsr_dev_xmit(struct sk_buff *skb, struct net_device *dev) { struct hsr_priv *hsr = netdev_priv(dev); @@ -244,14 +233,13 @@ static int hsr_dev_xmit(struct sk_buff *skb, struct net_device *dev) return NETDEV_TX_OK; } - static const struct header_ops hsr_header_ops = { .create = eth_header, .parse = eth_header_parse, }; static void send_hsr_supervision_frame(struct hsr_port *master, - u8 type, u8 hsrVer) + u8 type, u8 hsr_ver) { struct sk_buff *skb; int hlen, tlen; @@ -262,39 +250,38 @@ static void send_hsr_supervision_frame(struct hsr_port *master, hlen = LL_RESERVED_SPACE(master->dev); tlen = master->dev->needed_tailroom; - skb = dev_alloc_skb( - sizeof(struct hsr_tag) + - sizeof(struct hsr_sup_tag) + - sizeof(struct hsr_sup_payload) + hlen + tlen); + skb = dev_alloc_skb(sizeof(struct hsr_tag) + + sizeof(struct hsr_sup_tag) + + sizeof(struct hsr_sup_payload) + hlen + tlen); - if (skb == NULL) + if (!skb) return; skb_reserve(skb, hlen); skb->dev = master->dev; - skb->protocol = htons(hsrVer ? ETH_P_HSR : ETH_P_PRP); + skb->protocol = htons(hsr_ver ? ETH_P_HSR : ETH_P_PRP); skb->priority = TC_PRIO_CONTROL; - if (dev_hard_header(skb, skb->dev, (hsrVer ? ETH_P_HSR : ETH_P_PRP), + if (dev_hard_header(skb, skb->dev, (hsr_ver ? ETH_P_HSR : ETH_P_PRP), master->hsr->sup_multicast_addr, skb->dev->dev_addr, skb->len) <= 0) goto out; skb_reset_mac_header(skb); - if (hsrVer > 0) { + if (hsr_ver > 0) { hsr_tag = skb_put(skb, sizeof(struct hsr_tag)); hsr_tag->encap_proto = htons(ETH_P_PRP); set_hsr_tag_LSDU_size(hsr_tag, HSR_V1_SUP_LSDUSIZE); } hsr_stag = skb_put(skb, sizeof(struct hsr_sup_tag)); - set_hsr_stag_path(hsr_stag, (hsrVer ? 0x0 : 0xf)); - set_hsr_stag_HSR_Ver(hsr_stag, hsrVer); + set_hsr_stag_path(hsr_stag, (hsr_ver ? 0x0 : 0xf)); + set_hsr_stag_HSR_ver(hsr_stag, hsr_ver); /* From HSRv1 on we have separate supervision sequence numbers. */ spin_lock_irqsave(&master->hsr->seqnr_lock, irqflags); - if (hsrVer > 0) { + if (hsr_ver > 0) { hsr_stag->sequence_nr = htons(master->hsr->sup_sequence_nr); hsr_tag->sequence_nr = htons(master->hsr->sequence_nr); master->hsr->sup_sequence_nr++; @@ -305,13 +292,14 @@ static void send_hsr_supervision_frame(struct hsr_port *master, } spin_unlock_irqrestore(&master->hsr->seqnr_lock, irqflags); - hsr_stag->HSR_TLV_Type = type; + hsr_stag->HSR_TLV_type = type; /* TODO: Why 12 in HSRv0? */ - hsr_stag->HSR_TLV_Length = hsrVer ? sizeof(struct hsr_sup_payload) : 12; + hsr_stag->HSR_TLV_length = + hsr_ver ? sizeof(struct hsr_sup_payload) : 12; /* Payload: MacAddressA */ hsr_sp = skb_put(skb, sizeof(struct hsr_sup_payload)); - ether_addr_copy(hsr_sp->MacAddressA, master->dev->dev_addr); + ether_addr_copy(hsr_sp->macaddress_A, master->dev->dev_addr); if (skb_put_padto(skb, ETH_ZLEN + HSR_HLEN)) return; @@ -324,7 +312,6 @@ out: kfree_skb(skb); } - /* Announce (supervision frame) timer function */ static void hsr_announce(struct timer_list *t) @@ -338,15 +325,15 @@ static void hsr_announce(struct timer_list *t) rcu_read_lock(); master = hsr_port_get_hsr(hsr, HSR_PT_MASTER); - if (hsr->announce_count < 3 && hsr->protVersion == 0) { + if (hsr->announce_count < 3 && hsr->prot_version == 0) { send_hsr_supervision_frame(master, HSR_TLV_ANNOUNCE, - hsr->protVersion); + hsr->prot_version); hsr->announce_count++; interval = msecs_to_jiffies(HSR_ANNOUNCE_INTERVAL); } else { send_hsr_supervision_frame(master, HSR_TLV_LIFE_CHECK, - hsr->protVersion); + hsr->prot_version); interval = msecs_to_jiffies(HSR_LIFE_CHECK_INTERVAL); } @@ -357,7 +344,6 @@ static void hsr_announce(struct timer_list *t) rcu_read_unlock(); } - /* According to comments in the declaration of struct net_device, this function * is "Called from unregister, can be used to call free_netdev". Ok then... */ @@ -368,6 +354,8 @@ static void hsr_dev_destroy(struct net_device *hsr_dev) hsr = netdev_priv(hsr_dev); + hsr_debugfs_term(hsr); + rtnl_lock(); hsr_for_each_port(hsr, port) hsr_del_port(port); @@ -423,7 +411,6 @@ void hsr_dev_setup(struct net_device *dev) dev->features |= NETIF_F_NETNS_LOCAL; } - /* Return true if dev is a HSR master; return false otherwise. */ inline bool is_hsr_master(struct net_device *dev) @@ -467,7 +454,7 @@ int hsr_dev_finalize(struct net_device *hsr_dev, struct net_device *slave[2], ether_addr_copy(hsr->sup_multicast_addr, def_multicast_addr); hsr->sup_multicast_addr[ETH_ALEN - 1] = multicast_spec; - hsr->protVersion = protocol_version; + hsr->prot_version = protocol_version; /* FIXME: should I modify the value of these? * @@ -498,6 +485,9 @@ int hsr_dev_finalize(struct net_device *hsr_dev, struct net_device *slave[2], goto fail; mod_timer(&hsr->prune_timer, jiffies + msecs_to_jiffies(PRUNE_PERIOD)); + res = hsr_debugfs_init(hsr, hsr_dev); + if (res) + goto fail; return 0; diff --git a/net/hsr/hsr_device.h b/net/hsr/hsr_device.h index 9975e31bbb827b7f7617372a992e596addaaf6c6..6d7759c4f5f98cc0b00a4287bfe03dea066dae5b 100644 --- a/net/hsr/hsr_device.h +++ b/net/hsr/hsr_device.h @@ -1,9 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* Copyright 2011-2014 Autronica Fire and Security AS - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License as published by the Free - * Software Foundation; either version 2 of the License, or (at your option) - * any later version. * * Author(s): * 2011-2014 Arvid Brodin, arvid.brodin@alten.se diff --git a/net/hsr/hsr_forward.c b/net/hsr/hsr_forward.c index 04b5450c5a5572e875f7900a3676fd80259b9b4b..ddd9605bad04cbe35c5e247a3f8674c3428624ee 100644 --- a/net/hsr/hsr_forward.c +++ b/net/hsr/hsr_forward.c @@ -1,9 +1,5 @@ +// SPDX-License-Identifier: GPL-2.0 /* Copyright 2011-2014 Autronica Fire and Security AS - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License as published by the Free - * Software Foundation; either version 2 of the License, or (at your option) - * any later version. * * Author(s): * 2011-2014 Arvid Brodin, arvid.brodin@alten.se @@ -17,7 +13,6 @@ #include "hsr_main.h" #include "hsr_framereg.h" - struct hsr_node; struct hsr_frame_info { @@ -32,7 +27,6 @@ struct hsr_frame_info { bool is_local_exclusive; }; - /* The uses I can see for these HSR supervision frames are: * 1) Use the frames that are sent after node initialization ("HSR_TLV.Type = * 22") to reset any sequence_nr counters belonging to that node. Useful if @@ -50,46 +44,45 @@ struct hsr_frame_info { */ static bool is_supervision_frame(struct hsr_priv *hsr, struct sk_buff *skb) { - struct ethhdr *ethHdr; - struct hsr_sup_tag *hsrSupTag; - struct hsrv1_ethhdr_sp *hsrV1Hdr; + struct ethhdr *eth_hdr; + struct hsr_sup_tag *hsr_sup_tag; + struct hsrv1_ethhdr_sp *hsr_V1_hdr; WARN_ON_ONCE(!skb_mac_header_was_set(skb)); - ethHdr = (struct ethhdr *) skb_mac_header(skb); + eth_hdr = (struct ethhdr *)skb_mac_header(skb); /* Correct addr? */ - if (!ether_addr_equal(ethHdr->h_dest, + if (!ether_addr_equal(eth_hdr->h_dest, hsr->sup_multicast_addr)) return false; /* Correct ether type?. */ - if (!(ethHdr->h_proto == htons(ETH_P_PRP) - || ethHdr->h_proto == htons(ETH_P_HSR))) + if (!(eth_hdr->h_proto == htons(ETH_P_PRP) || + eth_hdr->h_proto == htons(ETH_P_HSR))) return false; /* Get the supervision header from correct location. */ - if (ethHdr->h_proto == htons(ETH_P_HSR)) { /* Okay HSRv1. */ - hsrV1Hdr = (struct hsrv1_ethhdr_sp *) skb_mac_header(skb); - if (hsrV1Hdr->hsr.encap_proto != htons(ETH_P_PRP)) + if (eth_hdr->h_proto == htons(ETH_P_HSR)) { /* Okay HSRv1. */ + hsr_V1_hdr = (struct hsrv1_ethhdr_sp *)skb_mac_header(skb); + if (hsr_V1_hdr->hsr.encap_proto != htons(ETH_P_PRP)) return false; - hsrSupTag = &hsrV1Hdr->hsr_sup; + hsr_sup_tag = &hsr_V1_hdr->hsr_sup; } else { - hsrSupTag = &((struct hsrv0_ethhdr_sp *) skb_mac_header(skb))->hsr_sup; + hsr_sup_tag = + &((struct hsrv0_ethhdr_sp *)skb_mac_header(skb))->hsr_sup; } - if ((hsrSupTag->HSR_TLV_Type != HSR_TLV_ANNOUNCE) && - (hsrSupTag->HSR_TLV_Type != HSR_TLV_LIFE_CHECK)) + if (hsr_sup_tag->HSR_TLV_type != HSR_TLV_ANNOUNCE && + hsr_sup_tag->HSR_TLV_type != HSR_TLV_LIFE_CHECK) return false; - if ((hsrSupTag->HSR_TLV_Length != 12) && - (hsrSupTag->HSR_TLV_Length != - sizeof(struct hsr_sup_payload))) + if (hsr_sup_tag->HSR_TLV_length != 12 && + hsr_sup_tag->HSR_TLV_length != sizeof(struct hsr_sup_payload)) return false; return true; } - static struct sk_buff *create_stripped_skb(struct sk_buff *skb_in, struct hsr_frame_info *frame) { @@ -100,7 +93,7 @@ static struct sk_buff *create_stripped_skb(struct sk_buff *skb_in, skb_pull(skb_in, HSR_HLEN); skb = __pskb_copy(skb_in, skb_headroom(skb_in) - HSR_HLEN, GFP_ATOMIC); skb_push(skb_in, HSR_HLEN); - if (skb == NULL) + if (!skb) return NULL; skb_reset_mac_header(skb); @@ -108,7 +101,7 @@ static struct sk_buff *create_stripped_skb(struct sk_buff *skb_in, if (skb->ip_summed == CHECKSUM_PARTIAL) skb->csum_start -= HSR_HLEN; - copylen = 2*ETH_ALEN; + copylen = 2 * ETH_ALEN; if (frame->is_vlan) copylen += VLAN_HLEN; src = skb_mac_header(skb_in); @@ -127,9 +120,8 @@ static struct sk_buff *frame_get_stripped_skb(struct hsr_frame_info *frame, return skb_clone(frame->skb_std, GFP_ATOMIC); } - static void hsr_fill_tag(struct sk_buff *skb, struct hsr_frame_info *frame, - struct hsr_port *port, u8 protoVersion) + struct hsr_port *port, u8 proto_version) { struct hsr_ethhdr *hsr_ethhdr; int lane_id; @@ -144,13 +136,13 @@ static void hsr_fill_tag(struct sk_buff *skb, struct hsr_frame_info *frame, if (frame->is_vlan) lsdu_size -= 4; - hsr_ethhdr = (struct hsr_ethhdr *) skb_mac_header(skb); + hsr_ethhdr = (struct hsr_ethhdr *)skb_mac_header(skb); set_hsr_tag_path(&hsr_ethhdr->hsr_tag, lane_id); set_hsr_tag_LSDU_size(&hsr_ethhdr->hsr_tag, lsdu_size); hsr_ethhdr->hsr_tag.sequence_nr = htons(frame->sequence_nr); hsr_ethhdr->hsr_tag.encap_proto = hsr_ethhdr->ethhdr.h_proto; - hsr_ethhdr->ethhdr.h_proto = htons(protoVersion ? + hsr_ethhdr->ethhdr.h_proto = htons(proto_version ? ETH_P_HSR : ETH_P_PRP); } @@ -164,7 +156,7 @@ static struct sk_buff *create_tagged_skb(struct sk_buff *skb_o, /* Create the new skb with enough headroom to fit the HSR tag */ skb = __pskb_copy(skb_o, skb_headroom(skb_o) + HSR_HLEN, GFP_ATOMIC); - if (skb == NULL) + if (!skb) return NULL; skb_reset_mac_header(skb); @@ -180,7 +172,7 @@ static struct sk_buff *create_tagged_skb(struct sk_buff *skb_o, memmove(dst, src, movelen); skb_reset_mac_header(skb); - hsr_fill_tag(skb, frame, port, port->hsr->protVersion); + hsr_fill_tag(skb, frame, port, port->hsr->prot_version); return skb; } @@ -194,7 +186,7 @@ static struct sk_buff *frame_get_tagged_skb(struct hsr_frame_info *frame, if (frame->skb_hsr) return skb_clone(frame->skb_hsr, GFP_ATOMIC); - if ((port->type != HSR_PT_SLAVE_A) && (port->type != HSR_PT_SLAVE_B)) { + if (port->type != HSR_PT_SLAVE_A && port->type != HSR_PT_SLAVE_B) { WARN_ONCE(1, "HSR: Bug: trying to create a tagged frame for a non-ring port"); return NULL; } @@ -202,7 +194,6 @@ static struct sk_buff *frame_get_tagged_skb(struct hsr_frame_info *frame, return create_tagged_skb(frame->skb_std, frame, port); } - static void hsr_deliver_master(struct sk_buff *skb, struct net_device *dev, struct hsr_node *node_src) { @@ -237,7 +228,6 @@ static int hsr_xmit(struct sk_buff *skb, struct hsr_port *port, return dev_queue_xmit(skb); } - /* Forward the frame through all devices except: * - Back through the receiving device * - If it's a HSR frame: through a device where it has passed before @@ -260,11 +250,11 @@ static void hsr_forward_do(struct hsr_frame_info *frame) continue; /* Don't deliver locally unless we should */ - if ((port->type == HSR_PT_MASTER) && !frame->is_local_dest) + if (port->type == HSR_PT_MASTER && !frame->is_local_dest) continue; /* Deliver frames directly addressed to us to master only */ - if ((port->type != HSR_PT_MASTER) && frame->is_local_exclusive) + if (port->type != HSR_PT_MASTER && frame->is_local_exclusive) continue; /* Don't send frame over port where it has been sent before */ @@ -272,7 +262,7 @@ static void hsr_forward_do(struct hsr_frame_info *frame) frame->sequence_nr)) continue; - if (frame->is_supervision && (port->type == HSR_PT_MASTER)) { + if (frame->is_supervision && port->type == HSR_PT_MASTER) { hsr_handle_sup_frame(frame->skb_hsr, frame->node_src, frame->port_rcv); @@ -283,7 +273,7 @@ static void hsr_forward_do(struct hsr_frame_info *frame) skb = frame_get_tagged_skb(frame, port); else skb = frame_get_stripped_skb(frame, port); - if (skb == NULL) { + if (!skb) { /* FIXME: Record the dropped frame? */ continue; } @@ -296,7 +286,6 @@ static void hsr_forward_do(struct hsr_frame_info *frame) } } - static void check_local_dest(struct hsr_priv *hsr, struct sk_buff *skb, struct hsr_frame_info *frame) { @@ -307,16 +296,15 @@ static void check_local_dest(struct hsr_priv *hsr, struct sk_buff *skb, frame->is_local_exclusive = false; } - if ((skb->pkt_type == PACKET_HOST) || - (skb->pkt_type == PACKET_MULTICAST) || - (skb->pkt_type == PACKET_BROADCAST)) { + if (skb->pkt_type == PACKET_HOST || + skb->pkt_type == PACKET_MULTICAST || + skb->pkt_type == PACKET_BROADCAST) { frame->is_local_dest = true; } else { frame->is_local_dest = false; } } - static int hsr_fill_frame_info(struct hsr_frame_info *frame, struct sk_buff *skb, struct hsr_port *port) { @@ -325,18 +313,18 @@ static int hsr_fill_frame_info(struct hsr_frame_info *frame, frame->is_supervision = is_supervision_frame(port->hsr, skb); frame->node_src = hsr_get_node(port, skb, frame->is_supervision); - if (frame->node_src == NULL) + if (!frame->node_src) return -1; /* Unknown node and !is_supervision, or no mem */ - ethhdr = (struct ethhdr *) skb_mac_header(skb); + ethhdr = (struct ethhdr *)skb_mac_header(skb); frame->is_vlan = false; if (ethhdr->h_proto == htons(ETH_P_8021Q)) { frame->is_vlan = true; /* FIXME: */ WARN_ONCE(1, "HSR: VLAN not yet supported"); } - if (ethhdr->h_proto == htons(ETH_P_PRP) - || ethhdr->h_proto == htons(ETH_P_HSR)) { + if (ethhdr->h_proto == htons(ETH_P_PRP) || + ethhdr->h_proto == htons(ETH_P_HSR)) { frame->skb_std = NULL; frame->skb_hsr = skb; frame->sequence_nr = hsr_get_skb_sequence_nr(skb); @@ -371,10 +359,17 @@ void hsr_forward_skb(struct sk_buff *skb, struct hsr_port *port) goto out_drop; hsr_register_frame_in(frame.node_src, port, frame.sequence_nr); hsr_forward_do(&frame); + /* Gets called for ingress frames as well as egress from master port. + * So check and increment stats for master port only here. + */ + if (port->type == HSR_PT_MASTER) { + port->dev->stats.tx_packets++; + port->dev->stats.tx_bytes += skb->len; + } - if (frame.skb_hsr != NULL) + if (frame.skb_hsr) kfree_skb(frame.skb_hsr); - if (frame.skb_std != NULL) + if (frame.skb_std) kfree_skb(frame.skb_std); return; diff --git a/net/hsr/hsr_forward.h b/net/hsr/hsr_forward.h index 5c5bc4b6b75f8955cd2388a087a6fbea5ab7aae0..51a69295566cf683d2879493ab3f9857297941a2 100644 --- a/net/hsr/hsr_forward.h +++ b/net/hsr/hsr_forward.h @@ -1,9 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* Copyright 2011-2014 Autronica Fire and Security AS - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License as published by the Free - * Software Foundation; either version 2 of the License, or (at your option) - * any later version. * * Author(s): * 2011-2014 Arvid Brodin, arvid.brodin@alten.se diff --git a/net/hsr/hsr_framereg.c b/net/hsr/hsr_framereg.c index 9af16cb68f76b92bd929d4d941a5e8c9a04d779e..9fa9abd83018a73e54b238343eb00fa00dd6d53b 100644 --- a/net/hsr/hsr_framereg.c +++ b/net/hsr/hsr_framereg.c @@ -1,9 +1,5 @@ +// SPDX-License-Identifier: GPL-2.0 /* Copyright 2011-2014 Autronica Fire and Security AS - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License as published by the Free - * Software Foundation; either version 2 of the License, or (at your option) - * any later version. * * Author(s): * 2011-2014 Arvid Brodin, arvid.brodin@alten.se @@ -22,23 +18,8 @@ #include "hsr_framereg.h" #include "hsr_netlink.h" - -struct hsr_node { - struct list_head mac_list; - unsigned char MacAddressA[ETH_ALEN]; - unsigned char MacAddressB[ETH_ALEN]; - /* Local slave through which AddrB frames are received from this node */ - enum hsr_port_type AddrB_port; - unsigned long time_in[HSR_PT_PORTS]; - bool time_in_stale[HSR_PT_PORTS]; - u16 seq_out[HSR_PT_PORTS]; - struct rcu_head rcu_head; -}; - - /* TODO: use hash lists for mac addresses (linux/jhash.h)? */ - /* seq_nr_after(a, b) - return true if a is after (higher in sequence than) b, * false otherwise. */ @@ -47,16 +28,16 @@ static bool seq_nr_after(u16 a, u16 b) /* Remove inconsistency where * seq_nr_after(a, b) == seq_nr_before(a, b) */ - if ((int) b - a == 32768) + if ((int)b - a == 32768) return false; - return (((s16) (b - a)) < 0); + return (((s16)(b - a)) < 0); } + #define seq_nr_before(a, b) seq_nr_after((b), (a)) #define seq_nr_after_or_eq(a, b) (!seq_nr_before((a), (b))) #define seq_nr_before_or_eq(a, b) (!seq_nr_after((a), (b))) - bool hsr_addr_is_self(struct hsr_priv *hsr, unsigned char *addr) { struct hsr_node *node; @@ -68,9 +49,9 @@ bool hsr_addr_is_self(struct hsr_priv *hsr, unsigned char *addr) return false; } - if (ether_addr_equal(addr, node->MacAddressA)) + if (ether_addr_equal(addr, node->macaddress_A)) return true; - if (ether_addr_equal(addr, node->MacAddressB)) + if (ether_addr_equal(addr, node->macaddress_B)) return true; return false; @@ -78,20 +59,19 @@ bool hsr_addr_is_self(struct hsr_priv *hsr, unsigned char *addr) /* Search for mac entry. Caller must hold rcu read lock. */ -static struct hsr_node *find_node_by_AddrA(struct list_head *node_db, - const unsigned char addr[ETH_ALEN]) +static struct hsr_node *find_node_by_addr_A(struct list_head *node_db, + const unsigned char addr[ETH_ALEN]) { struct hsr_node *node; list_for_each_entry_rcu(node, node_db, mac_list) { - if (ether_addr_equal(node->MacAddressA, addr)) + if (ether_addr_equal(node->macaddress_A, addr)) return node; } return NULL; } - /* Helper for device init; the self_node_db is used in hsr_rcv() to recognize * frames from self that's been looped over the HSR ring. */ @@ -105,12 +85,12 @@ int hsr_create_self_node(struct list_head *self_node_db, if (!node) return -ENOMEM; - ether_addr_copy(node->MacAddressA, addr_a); - ether_addr_copy(node->MacAddressB, addr_b); + ether_addr_copy(node->macaddress_A, addr_a); + ether_addr_copy(node->macaddress_B, addr_b); rcu_read_lock(); oldnode = list_first_or_null_rcu(self_node_db, - struct hsr_node, mac_list); + struct hsr_node, mac_list); if (oldnode) { list_replace_rcu(&oldnode->mac_list, &node->mac_list); rcu_read_unlock(); @@ -137,7 +117,7 @@ void hsr_del_node(struct list_head *self_node_db) } } -/* Allocate an hsr_node and add it to node_db. 'addr' is the node's AddressA; +/* Allocate an hsr_node and add it to node_db. 'addr' is the node's address_A; * seq_out is used to initialize filtering of outgoing duplicate frames * originating from the newly added node. */ @@ -152,7 +132,7 @@ struct hsr_node *hsr_add_node(struct list_head *node_db, unsigned char addr[], if (!node) return NULL; - ether_addr_copy(node->MacAddressA, addr); + ether_addr_copy(node->macaddress_A, addr); /* We are only interested in time diffs here, so use current jiffies * as initialization. (0 could trigger an spurious ring error warning). @@ -181,19 +161,19 @@ struct hsr_node *hsr_get_node(struct hsr_port *port, struct sk_buff *skb, if (!skb_mac_header_was_set(skb)) return NULL; - ethhdr = (struct ethhdr *) skb_mac_header(skb); + ethhdr = (struct ethhdr *)skb_mac_header(skb); list_for_each_entry_rcu(node, node_db, mac_list) { - if (ether_addr_equal(node->MacAddressA, ethhdr->h_source)) + if (ether_addr_equal(node->macaddress_A, ethhdr->h_source)) return node; - if (ether_addr_equal(node->MacAddressB, ethhdr->h_source)) + if (ether_addr_equal(node->macaddress_B, ethhdr->h_source)) return node; } /* Everyone may create a node entry, connected node to a HSR device. */ - if (ethhdr->h_proto == htons(ETH_P_PRP) - || ethhdr->h_proto == htons(ETH_P_HSR)) { + if (ethhdr->h_proto == htons(ETH_P_PRP) || + ethhdr->h_proto == htons(ETH_P_HSR)) { /* Use the existing sequence_nr from the tag as starting point * for filtering duplicate frames. */ @@ -210,8 +190,8 @@ struct hsr_node *hsr_get_node(struct hsr_port *port, struct sk_buff *skb, return hsr_add_node(node_db, ethhdr->h_source, seq_out); } -/* Use the Supervision frame's info about an eventual MacAddressB for merging - * nodes that has previously had their MacAddressB registered as a separate +/* Use the Supervision frame's info about an eventual macaddress_B for merging + * nodes that has previously had their macaddress_B registered as a separate * node. */ void hsr_handle_sup_frame(struct sk_buff *skb, struct hsr_node *node_curr, @@ -223,7 +203,7 @@ void hsr_handle_sup_frame(struct sk_buff *skb, struct hsr_node *node_curr, struct list_head *node_db; int i; - ethhdr = (struct ethhdr *) skb_mac_header(skb); + ethhdr = (struct ethhdr *)skb_mac_header(skb); /* Leave the ethernet header. */ skb_pull(skb, sizeof(struct ethhdr)); @@ -235,14 +215,14 @@ void hsr_handle_sup_frame(struct sk_buff *skb, struct hsr_node *node_curr, /* And leave the HSR sup tag. */ skb_pull(skb, sizeof(struct hsr_sup_tag)); - hsr_sp = (struct hsr_sup_payload *) skb->data; + hsr_sp = (struct hsr_sup_payload *)skb->data; - /* Merge node_curr (registered on MacAddressB) into node_real */ + /* Merge node_curr (registered on macaddress_B) into node_real */ node_db = &port_rcv->hsr->node_db; - node_real = find_node_by_AddrA(node_db, hsr_sp->MacAddressA); + node_real = find_node_by_addr_A(node_db, hsr_sp->macaddress_A); if (!node_real) /* No frame received from AddrA of this node yet */ - node_real = hsr_add_node(node_db, hsr_sp->MacAddressA, + node_real = hsr_add_node(node_db, hsr_sp->macaddress_A, HSR_SEQNR_START - 1); if (!node_real) goto done; /* No mem */ @@ -250,17 +230,18 @@ void hsr_handle_sup_frame(struct sk_buff *skb, struct hsr_node *node_curr, /* Node has already been merged */ goto done; - ether_addr_copy(node_real->MacAddressB, ethhdr->h_source); + ether_addr_copy(node_real->macaddress_B, ethhdr->h_source); for (i = 0; i < HSR_PT_PORTS; i++) { if (!node_curr->time_in_stale[i] && time_after(node_curr->time_in[i], node_real->time_in[i])) { node_real->time_in[i] = node_curr->time_in[i]; - node_real->time_in_stale[i] = node_curr->time_in_stale[i]; + node_real->time_in_stale[i] = + node_curr->time_in_stale[i]; } if (seq_nr_after(node_curr->seq_out[i], node_real->seq_out[i])) node_real->seq_out[i] = node_curr->seq_out[i]; } - node_real->AddrB_port = port_rcv->type; + node_real->addr_B_port = port_rcv->type; list_del_rcu(&node_curr->mac_list); kfree_rcu(node_curr, rcu_head); @@ -269,11 +250,10 @@ done: skb_push(skb, sizeof(struct hsrv1_ethhdr_sp)); } - /* 'skb' is a frame meant for this host, that is to be passed to upper layers. * * If the frame was sent by a node's B interface, replace the source - * address with that node's "official" address (MacAddressA) so that upper + * address with that node's "official" address (macaddress_A) so that upper * layers recognize where it came from. */ void hsr_addr_subst_source(struct hsr_node *node, struct sk_buff *skb) @@ -283,7 +263,7 @@ void hsr_addr_subst_source(struct hsr_node *node, struct sk_buff *skb) return; } - memcpy(ð_hdr(skb)->h_source, node->MacAddressA, ETH_ALEN); + memcpy(ð_hdr(skb)->h_source, node->macaddress_A, ETH_ALEN); } /* 'skb' is a frame meant for another host. @@ -308,18 +288,18 @@ void hsr_addr_subst_dest(struct hsr_node *node_src, struct sk_buff *skb, if (!is_unicast_ether_addr(eth_hdr(skb)->h_dest)) return; - node_dst = find_node_by_AddrA(&port->hsr->node_db, eth_hdr(skb)->h_dest); + node_dst = find_node_by_addr_A(&port->hsr->node_db, + eth_hdr(skb)->h_dest); if (!node_dst) { WARN_ONCE(1, "%s: Unknown node\n", __func__); return; } - if (port->type != node_dst->AddrB_port) + if (port->type != node_dst->addr_B_port) return; - ether_addr_copy(eth_hdr(skb)->h_dest, node_dst->MacAddressB); + ether_addr_copy(eth_hdr(skb)->h_dest, node_dst->macaddress_B); } - void hsr_register_frame_in(struct hsr_node *node, struct hsr_port *port, u16 sequence_nr) { @@ -352,7 +332,6 @@ int hsr_register_frame_out(struct hsr_port *port, struct hsr_node *node, return 0; } - static struct hsr_port *get_late_port(struct hsr_priv *hsr, struct hsr_node *node) { @@ -373,7 +352,6 @@ static struct hsr_port *get_late_port(struct hsr_priv *hsr, return NULL; } - /* Remove stale sequence_nr records. Called by timer every * HSR_LIFE_CHECK_INTERVAL (two seconds or so). */ @@ -392,9 +370,9 @@ void hsr_prune_nodes(struct timer_list *t) time_b = node->time_in[HSR_PT_SLAVE_B]; /* Check for timestamps old enough to risk wrap-around */ - if (time_after(jiffies, time_a + MAX_JIFFY_OFFSET/2)) + if (time_after(jiffies, time_a + MAX_JIFFY_OFFSET / 2)) node->time_in_stale[HSR_PT_SLAVE_A] = true; - if (time_after(jiffies, time_b + MAX_JIFFY_OFFSET/2)) + if (time_after(jiffies, time_b + MAX_JIFFY_OFFSET / 2)) node->time_in_stale[HSR_PT_SLAVE_B] = true; /* Get age of newest frame from node. @@ -409,26 +387,29 @@ void hsr_prune_nodes(struct timer_list *t) /* Warn of ring error only as long as we get frames at all */ if (time_is_after_jiffies(timestamp + - msecs_to_jiffies(1.5*MAX_SLAVE_DIFF))) { + msecs_to_jiffies(1.5 * MAX_SLAVE_DIFF))) { rcu_read_lock(); port = get_late_port(hsr, node); - if (port != NULL) - hsr_nl_ringerror(hsr, node->MacAddressA, port); + if (port) + hsr_nl_ringerror(hsr, node->macaddress_A, port); rcu_read_unlock(); } /* Prune old entries */ if (time_is_before_jiffies(timestamp + - msecs_to_jiffies(HSR_NODE_FORGET_TIME))) { - hsr_nl_nodedown(hsr, node->MacAddressA); + msecs_to_jiffies(HSR_NODE_FORGET_TIME))) { + hsr_nl_nodedown(hsr, node->macaddress_A); list_del_rcu(&node->mac_list); /* Note that we need to free this entry later: */ kfree_rcu(node, rcu_head); } } rcu_read_unlock(); -} + /* Restart timer */ + mod_timer(&hsr->prune_timer, + jiffies + msecs_to_jiffies(PRUNE_PERIOD)); +} void *hsr_get_next_node(struct hsr_priv *hsr, void *_pos, unsigned char addr[ETH_ALEN]) @@ -439,20 +420,19 @@ void *hsr_get_next_node(struct hsr_priv *hsr, void *_pos, node = list_first_or_null_rcu(&hsr->node_db, struct hsr_node, mac_list); if (node) - ether_addr_copy(addr, node->MacAddressA); + ether_addr_copy(addr, node->macaddress_A); return node; } node = _pos; list_for_each_entry_continue_rcu(node, &hsr->node_db, mac_list) { - ether_addr_copy(addr, node->MacAddressA); + ether_addr_copy(addr, node->macaddress_A); return node; } return NULL; } - int hsr_get_node_data(struct hsr_priv *hsr, const unsigned char *addr, unsigned char addr_b[ETH_ALEN], @@ -466,15 +446,14 @@ int hsr_get_node_data(struct hsr_priv *hsr, struct hsr_port *port; unsigned long tdiff; - rcu_read_lock(); - node = find_node_by_AddrA(&hsr->node_db, addr); + node = find_node_by_addr_A(&hsr->node_db, addr); if (!node) { rcu_read_unlock(); return -ENOENT; /* No such entry */ } - ether_addr_copy(addr_b, node->MacAddressB); + ether_addr_copy(addr_b, node->macaddress_B); tdiff = jiffies - node->time_in[HSR_PT_SLAVE_A]; if (node->time_in_stale[HSR_PT_SLAVE_A]) @@ -500,8 +479,8 @@ int hsr_get_node_data(struct hsr_priv *hsr, *if1_seq = node->seq_out[HSR_PT_SLAVE_B]; *if2_seq = node->seq_out[HSR_PT_SLAVE_A]; - if (node->AddrB_port != HSR_PT_NONE) { - port = hsr_port_get_hsr(hsr, node->AddrB_port); + if (node->addr_B_port != HSR_PT_NONE) { + port = hsr_port_get_hsr(hsr, node->addr_B_port); *addr_b_ifindex = port->dev->ifindex; } else { *addr_b_ifindex = -1; diff --git a/net/hsr/hsr_framereg.h b/net/hsr/hsr_framereg.h index 531fd3dfcac1efae09302284c15e08b4c80a4d33..a3bdcdab469d5383f77f3d41e77beb33f5860cec 100644 --- a/net/hsr/hsr_framereg.h +++ b/net/hsr/hsr_framereg.h @@ -1,9 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* Copyright 2011-2014 Autronica Fire and Security AS - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License as published by the Free - * Software Foundation; either version 2 of the License, or (at your option) - * any later version. * * Author(s): * 2011-2014 Arvid Brodin, arvid.brodin@alten.se @@ -52,4 +48,16 @@ int hsr_get_node_data(struct hsr_priv *hsr, int *if2_age, u16 *if2_seq); +struct hsr_node { + struct list_head mac_list; + unsigned char macaddress_A[ETH_ALEN]; + unsigned char macaddress_B[ETH_ALEN]; + /* Local slave through which AddrB frames are received from this node */ + enum hsr_port_type addr_B_port; + unsigned long time_in[HSR_PT_PORTS]; + bool time_in_stale[HSR_PT_PORTS]; + u16 seq_out[HSR_PT_PORTS]; + struct rcu_head rcu_head; +}; + #endif /* __HSR_FRAMEREG_H */ diff --git a/net/hsr/hsr_main.c b/net/hsr/hsr_main.c index cd37d0011b424824fd113ffd4da59f36c116996a..b9988a662ee1ac14c3ad8b96ea18a2dca92f5254 100644 --- a/net/hsr/hsr_main.c +++ b/net/hsr/hsr_main.c @@ -1,9 +1,5 @@ +// SPDX-License-Identifier: GPL-2.0 /* Copyright 2011-2014 Autronica Fire and Security AS - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License as published by the Free - * Software Foundation; either version 2 of the License, or (at your option) - * any later version. * * Author(s): * 2011-2014 Arvid Brodin, arvid.brodin@alten.se @@ -19,7 +15,6 @@ #include "hsr_framereg.h" #include "hsr_slave.h" - static int hsr_netdev_notify(struct notifier_block *nb, unsigned long event, void *ptr) { @@ -31,12 +26,12 @@ static int hsr_netdev_notify(struct notifier_block *nb, unsigned long event, dev = netdev_notifier_info_to_dev(ptr); port = hsr_port_get_rtnl(dev); - if (port == NULL) { + if (!port) { if (!is_hsr_master(dev)) return NOTIFY_DONE; /* Not an HSR device */ hsr = netdev_priv(dev); port = hsr_port_get_hsr(hsr, HSR_PT_MASTER); - if (port == NULL) { + if (!port) { /* Resend of notification concerning removed device? */ return NOTIFY_DONE; } @@ -63,7 +58,8 @@ static int hsr_netdev_notify(struct notifier_block *nb, unsigned long event, if (port->type == HSR_PT_SLAVE_A) { ether_addr_copy(master->dev->dev_addr, dev->dev_addr); - call_netdevice_notifiers(NETDEV_CHANGEADDR, master->dev); + call_netdevice_notifiers(NETDEV_CHANGEADDR, + master->dev); } /* Make sure we recognize frames from ourselves in hsr_rcv() */ @@ -97,7 +93,6 @@ static int hsr_netdev_notify(struct notifier_block *nb, unsigned long event, return NOTIFY_DONE; } - struct hsr_port *hsr_port_get_hsr(struct hsr_priv *hsr, enum hsr_port_type pt) { struct hsr_port *port; @@ -112,7 +107,6 @@ static struct notifier_block hsr_nb = { .notifier_call = hsr_netdev_notify, /* Slave event notifications */ }; - static int __init hsr_init(void) { int res; diff --git a/net/hsr/hsr_main.h b/net/hsr/hsr_main.h index 9b9909e89e9e855851b2fa6ab8bed321cdcfafc7..96fac696a1e1a602fdf46c4df74016b7baf50008 100644 --- a/net/hsr/hsr_main.h +++ b/net/hsr/hsr_main.h @@ -1,9 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* Copyright 2011-2014 Autronica Fire and Security AS - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License as published by the Free - * Software Foundation; either version 2 of the License, or (at your option) - * any later version. * * Author(s): * 2011-2014 Arvid Brodin, arvid.brodin@alten.se @@ -15,7 +11,6 @@ #include #include - /* Time constants as specified in the HSR specification (IEC-62439-3 2010) * Table 8. * All values in milliseconds. @@ -24,7 +19,6 @@ #define HSR_NODE_FORGET_TIME 60000 /* ms */ #define HSR_ANNOUNCE_INTERVAL 100 /* ms */ - /* By how much may slave1 and slave2 timestamps of latest received frame from * each node differ before we notify of communication problem? */ @@ -32,17 +26,14 @@ #define HSR_SEQNR_START (USHRT_MAX - 1024) #define HSR_SUP_SEQNR_START (HSR_SEQNR_START / 2) - /* How often shall we check for broken ring and remove node entries older than * HSR_NODE_FORGET_TIME? */ #define PRUNE_PERIOD 3000 /* ms */ - #define HSR_TLV_ANNOUNCE 22 #define HSR_TLV_LIFE_CHECK 23 - /* HSR Tag. * As defined in IEC-62439-3:2010, the HSR tag is really { ethertype = 0x88FB, * path, LSDU_size, sequence Nr }. But we let eth_header() create { h_dest, @@ -83,15 +74,14 @@ static inline u16 get_hsr_tag_LSDU_size(struct hsr_tag *ht) static inline void set_hsr_tag_path(struct hsr_tag *ht, u16 path) { - ht->path_and_LSDU_size = htons( - (ntohs(ht->path_and_LSDU_size) & 0x0FFF) | (path << 12)); + ht->path_and_LSDU_size = + htons((ntohs(ht->path_and_LSDU_size) & 0x0FFF) | (path << 12)); } static inline void set_hsr_tag_LSDU_size(struct hsr_tag *ht, u16 LSDU_size) { - ht->path_and_LSDU_size = htons( - (ntohs(ht->path_and_LSDU_size) & 0xF000) | - (LSDU_size & 0x0FFF)); + ht->path_and_LSDU_size = htons((ntohs(ht->path_and_LSDU_size) & + 0xF000) | (LSDU_size & 0x0FFF)); } struct hsr_ethhdr { @@ -99,39 +89,38 @@ struct hsr_ethhdr { struct hsr_tag hsr_tag; } __packed; - /* HSR Supervision Frame data types. * Field names as defined in the IEC:2010 standard for HSR. */ struct hsr_sup_tag { - __be16 path_and_HSR_Ver; + __be16 path_and_HSR_ver; __be16 sequence_nr; - __u8 HSR_TLV_Type; - __u8 HSR_TLV_Length; + __u8 HSR_TLV_type; + __u8 HSR_TLV_length; } __packed; struct hsr_sup_payload { - unsigned char MacAddressA[ETH_ALEN]; + unsigned char macaddress_A[ETH_ALEN]; } __packed; static inline u16 get_hsr_stag_path(struct hsr_sup_tag *hst) { - return get_hsr_tag_path((struct hsr_tag *) hst); + return get_hsr_tag_path((struct hsr_tag *)hst); } static inline u16 get_hsr_stag_HSR_ver(struct hsr_sup_tag *hst) { - return get_hsr_tag_LSDU_size((struct hsr_tag *) hst); + return get_hsr_tag_LSDU_size((struct hsr_tag *)hst); } static inline void set_hsr_stag_path(struct hsr_sup_tag *hst, u16 path) { - set_hsr_tag_path((struct hsr_tag *) hst, path); + set_hsr_tag_path((struct hsr_tag *)hst, path); } -static inline void set_hsr_stag_HSR_Ver(struct hsr_sup_tag *hst, u16 HSR_Ver) +static inline void set_hsr_stag_HSR_ver(struct hsr_sup_tag *hst, u16 HSR_ver) { - set_hsr_tag_LSDU_size((struct hsr_tag *) hst, HSR_Ver); + set_hsr_tag_LSDU_size((struct hsr_tag *)hst, HSR_ver); } struct hsrv0_ethhdr_sp { @@ -145,7 +134,6 @@ struct hsrv1_ethhdr_sp { struct hsr_sup_tag hsr_sup; } __packed; - enum hsr_port_type { HSR_PT_NONE = 0, /* Must be 0, used by framereg */ HSR_PT_SLAVE_A, @@ -171,10 +159,14 @@ struct hsr_priv { struct timer_list prune_timer; int announce_count; u16 sequence_nr; - u16 sup_sequence_nr; /* For HSRv1 separate seq_nr for supervision */ - u8 protVersion; /* Indicate if HSRv0 or HSRv1. */ + u16 sup_sequence_nr; /* For HSRv1 separate seq_nr for supervision */ + u8 prot_version; /* Indicate if HSRv0 or HSRv1. */ spinlock_t seqnr_lock; /* locking for sequence_nr */ unsigned char sup_multicast_addr[ETH_ALEN]; +#ifdef CONFIG_DEBUG_FS + struct dentry *node_tbl_root; + struct dentry *node_tbl_file; +#endif }; #define hsr_for_each_port(hsr, port) \ @@ -187,8 +179,22 @@ static inline u16 hsr_get_skb_sequence_nr(struct sk_buff *skb) { struct hsr_ethhdr *hsr_ethhdr; - hsr_ethhdr = (struct hsr_ethhdr *) skb_mac_header(skb); + hsr_ethhdr = (struct hsr_ethhdr *)skb_mac_header(skb); return ntohs(hsr_ethhdr->hsr_tag.sequence_nr); } +#if IS_ENABLED(CONFIG_DEBUG_FS) +int hsr_debugfs_init(struct hsr_priv *priv, struct net_device *hsr_dev); +void hsr_debugfs_term(struct hsr_priv *priv); +#else +static inline int hsr_debugfs_init(struct hsr_priv *priv, + struct net_device *hsr_dev) +{ + return 0; +} + +static inline void hsr_debugfs_term(struct hsr_priv *priv) +{} +#endif + #endif /* __HSR_PRIVATE_H */ diff --git a/net/hsr/hsr_netlink.c b/net/hsr/hsr_netlink.c index b9cce0fd56966bbd8a16c8bc51555dfab467b6d0..8f8337f893badcbe854108fba737ac9cf45a32ee 100644 --- a/net/hsr/hsr_netlink.c +++ b/net/hsr/hsr_netlink.c @@ -1,9 +1,5 @@ +// SPDX-License-Identifier: GPL-2.0 /* Copyright 2011-2014 Autronica Fire and Security AS - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License as published by the Free - * Software Foundation; either version 2 of the License, or (at your option) - * any later version. * * Author(s): * 2011-2014 Arvid Brodin, arvid.brodin@alten.se @@ -28,7 +24,6 @@ static const struct nla_policy hsr_policy[IFLA_HSR_MAX + 1] = { [IFLA_HSR_SEQ_NR] = { .type = NLA_U16 }, }; - /* Here, it seems a netdevice has already been allocated for us, and the * hsr_dev_setup routine has been executed. Nice! */ @@ -47,12 +42,14 @@ static int hsr_newlink(struct net *src_net, struct net_device *dev, netdev_info(dev, "HSR: Slave1 device not specified\n"); return -EINVAL; } - link[0] = __dev_get_by_index(src_net, nla_get_u32(data[IFLA_HSR_SLAVE1])); + link[0] = __dev_get_by_index(src_net, + nla_get_u32(data[IFLA_HSR_SLAVE1])); if (!data[IFLA_HSR_SLAVE2]) { netdev_info(dev, "HSR: Slave2 device not specified\n"); return -EINVAL; } - link[1] = __dev_get_by_index(src_net, nla_get_u32(data[IFLA_HSR_SLAVE2])); + link[1] = __dev_get_by_index(src_net, + nla_get_u32(data[IFLA_HSR_SLAVE2])); if (!link[0] || !link[1]) return -ENODEV; @@ -119,8 +116,6 @@ static struct rtnl_link_ops hsr_link_ops __read_mostly = { .fill_info = hsr_fill_info, }; - - /* attribute policy */ static const struct nla_policy hsr_genl_policy[HSR_A_MAX + 1] = { [HSR_A_NODE_ADDR] = { .len = ETH_ALEN }, @@ -138,8 +133,6 @@ static const struct genl_multicast_group hsr_mcgrps[] = { { .name = "hsr-network", }, }; - - /* This is called if for some node with MAC address addr, we only get frames * over one of the slave interfaces. This would indicate an open network ring * (i.e. a link has failed somewhere). @@ -156,7 +149,8 @@ void hsr_nl_ringerror(struct hsr_priv *hsr, unsigned char addr[ETH_ALEN], if (!skb) goto fail; - msg_head = genlmsg_put(skb, 0, 0, &hsr_genl_family, 0, HSR_C_RING_ERROR); + msg_head = genlmsg_put(skb, 0, 0, &hsr_genl_family, 0, + HSR_C_RING_ERROR); if (!msg_head) goto nla_put_failure; @@ -201,7 +195,6 @@ void hsr_nl_nodedown(struct hsr_priv *hsr, unsigned char addr[ETH_ALEN]) if (!msg_head) goto nla_put_failure; - res = nla_put(skb, HSR_A_NODE_ADDR, ETH_ALEN, addr); if (res < 0) goto nla_put_failure; @@ -221,7 +214,6 @@ fail: rcu_read_unlock(); } - /* HSR_C_GET_NODE_STATUS lets userspace query the internal HSR node table * about the status of a specific node in the network, defined by its MAC * address. @@ -260,15 +252,13 @@ static int hsr_get_node_status(struct sk_buff *skb_in, struct genl_info *info) goto invalid; hsr_dev = __dev_get_by_index(genl_info_net(info), - nla_get_u32(info->attrs[HSR_A_IFINDEX])); + nla_get_u32(info->attrs[HSR_A_IFINDEX])); if (!hsr_dev) goto invalid; if (!is_hsr_master(hsr_dev)) goto invalid; - /* Send reply */ - skb_out = genlmsg_new(NLMSG_GOODSIZE, GFP_KERNEL); if (!skb_out) { res = -ENOMEM; @@ -276,8 +266,8 @@ static int hsr_get_node_status(struct sk_buff *skb_in, struct genl_info *info) } msg_head = genlmsg_put(skb_out, NETLINK_CB(skb_in).portid, - info->snd_seq, &hsr_genl_family, 0, - HSR_C_SET_NODE_STATUS); + info->snd_seq, &hsr_genl_family, 0, + HSR_C_SET_NODE_STATUS); if (!msg_head) { res = -ENOMEM; goto nla_put_failure; @@ -289,28 +279,30 @@ static int hsr_get_node_status(struct sk_buff *skb_in, struct genl_info *info) hsr = netdev_priv(hsr_dev); res = hsr_get_node_data(hsr, - (unsigned char *) nla_data(info->attrs[HSR_A_NODE_ADDR]), - hsr_node_addr_b, - &addr_b_ifindex, - &hsr_node_if1_age, - &hsr_node_if1_seq, - &hsr_node_if2_age, - &hsr_node_if2_seq); + (unsigned char *) + nla_data(info->attrs[HSR_A_NODE_ADDR]), + hsr_node_addr_b, + &addr_b_ifindex, + &hsr_node_if1_age, + &hsr_node_if1_seq, + &hsr_node_if2_age, + &hsr_node_if2_seq); if (res < 0) goto nla_put_failure; res = nla_put(skb_out, HSR_A_NODE_ADDR, ETH_ALEN, - nla_data(info->attrs[HSR_A_NODE_ADDR])); + nla_data(info->attrs[HSR_A_NODE_ADDR])); if (res < 0) goto nla_put_failure; if (addr_b_ifindex > -1) { res = nla_put(skb_out, HSR_A_NODE_ADDR_B, ETH_ALEN, - hsr_node_addr_b); + hsr_node_addr_b); if (res < 0) goto nla_put_failure; - res = nla_put_u32(skb_out, HSR_A_ADDR_B_IFINDEX, addr_b_ifindex); + res = nla_put_u32(skb_out, HSR_A_ADDR_B_IFINDEX, + addr_b_ifindex); if (res < 0) goto nla_put_failure; } @@ -392,9 +384,7 @@ static int hsr_get_node_list(struct sk_buff *skb_in, struct genl_info *info) if (!is_hsr_master(hsr_dev)) goto invalid; - /* Send reply */ - skb_out = genlmsg_new(NLMSG_GOODSIZE, GFP_KERNEL); if (!skb_out) { res = -ENOMEM; @@ -402,8 +392,8 @@ static int hsr_get_node_list(struct sk_buff *skb_in, struct genl_info *info) } msg_head = genlmsg_put(skb_out, NETLINK_CB(skb_in).portid, - info->snd_seq, &hsr_genl_family, 0, - HSR_C_SET_NODE_LIST); + info->snd_seq, &hsr_genl_family, 0, + HSR_C_SET_NODE_LIST); if (!msg_head) { res = -ENOMEM; goto nla_put_failure; @@ -444,19 +434,18 @@ fail: return res; } - static const struct genl_ops hsr_ops[] = { { .cmd = HSR_C_GET_NODE_STATUS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = 0, - .policy = hsr_genl_policy, .doit = hsr_get_node_status, .dumpit = NULL, }, { .cmd = HSR_C_GET_NODE_LIST, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = 0, - .policy = hsr_genl_policy, .doit = hsr_get_node_list, .dumpit = NULL, }, @@ -467,6 +456,7 @@ static struct genl_family hsr_genl_family __ro_after_init = { .name = "HSR", .version = 1, .maxattr = HSR_A_MAX, + .policy = hsr_genl_policy, .module = THIS_MODULE, .ops = hsr_ops, .n_ops = ARRAY_SIZE(hsr_ops), diff --git a/net/hsr/hsr_netlink.h b/net/hsr/hsr_netlink.h index 3f6b95b5b6b8841b045493468ccce641ea55d258..1121bb192a18e8be5a15320907327b427f3b4c29 100644 --- a/net/hsr/hsr_netlink.h +++ b/net/hsr/hsr_netlink.h @@ -1,9 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* Copyright 2011-2014 Autronica Fire and Security AS - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License as published by the Free - * Software Foundation; either version 2 of the License, or (at your option) - * any later version. * * Author(s): * 2011-2014 Arvid Brodin, arvid.brodin@alten.se diff --git a/net/hsr/hsr_slave.c b/net/hsr/hsr_slave.c index 56080da4aa77ef581f4a3c23b4179d72e511980b..88b6705ded837165cd06544d98ea4fe513747f66 100644 --- a/net/hsr/hsr_slave.c +++ b/net/hsr/hsr_slave.c @@ -1,9 +1,5 @@ +// SPDX-License-Identifier: GPL-2.0 /* Copyright 2011-2014 Autronica Fire and Security AS - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License as published by the Free - * Software Foundation; either version 2 of the License, or (at your option) - * any later version. * * Author(s): * 2011-2014 Arvid Brodin, arvid.brodin@alten.se @@ -18,7 +14,6 @@ #include "hsr_forward.h" #include "hsr_framereg.h" - static rx_handler_result_t hsr_handle_frame(struct sk_buff **pskb) { struct sk_buff *skb = *pskb; @@ -61,12 +56,11 @@ bool hsr_port_exists(const struct net_device *dev) return rcu_access_pointer(dev->rx_handler) == hsr_handle_frame; } - static int hsr_check_dev_ok(struct net_device *dev) { /* Don't allow HSR on non-ethernet like devices */ - if ((dev->flags & IFF_LOOPBACK) || (dev->type != ARPHRD_ETHER) || - (dev->addr_len != ETH_ALEN)) { + if ((dev->flags & IFF_LOOPBACK) || dev->type != ARPHRD_ETHER || + dev->addr_len != ETH_ALEN) { netdev_info(dev, "Cannot use loopback or non-ethernet device as HSR slave.\n"); return -EINVAL; } @@ -99,7 +93,6 @@ static int hsr_check_dev_ok(struct net_device *dev) return 0; } - /* Setup device to be added to the HSR bridge. */ static int hsr_portdev_setup(struct net_device *dev, struct hsr_port *port) { @@ -143,11 +136,11 @@ int hsr_add_port(struct hsr_priv *hsr, struct net_device *dev, } port = hsr_port_get_hsr(hsr, type); - if (port != NULL) + if (port) return -EBUSY; /* This port already exists */ port = kzalloc(sizeof(*port), GFP_KERNEL); - if (port == NULL) + if (!port) return -ENOMEM; if (type != HSR_PT_MASTER) { @@ -184,7 +177,7 @@ void hsr_del_port(struct hsr_port *port) list_del_rcu(&port->port_list); if (port != master) { - if (master != NULL) { + if (master) { netdev_update_features(master->dev); dev_set_mtu(master->dev, hsr_get_max_mtu(hsr)); } diff --git a/net/hsr/hsr_slave.h b/net/hsr/hsr_slave.h index 3ccfbf71c92ef608f9ed5b65ce3e490c4d7abda7..64b54952959207809ffa7632c989f49c88c73143 100644 --- a/net/hsr/hsr_slave.h +++ b/net/hsr/hsr_slave.h @@ -1,11 +1,6 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* Copyright 2011-2014 Autronica Fire and Security AS * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License as published by the Free - * Software Foundation; either version 2 of the License, or (at your option) - * any later version. - * - * Author(s): * 2011-2014 Arvid Brodin, arvid.brodin@alten.se */ diff --git a/net/ieee802154/ieee802154.h b/net/ieee802154/ieee802154.h index a5d7515b7f6256424eb26ce4f07250571ce7f6f7..bc147bc8e36ac38767f2519976ebe3f68d08b364 100644 --- a/net/ieee802154/ieee802154.h +++ b/net/ieee802154/ieee802154.h @@ -20,7 +20,6 @@ void ieee802154_nl_exit(void); #define IEEE802154_OP(_cmd, _func) \ { \ .cmd = _cmd, \ - .policy = ieee802154_policy, \ .doit = _func, \ .dumpit = NULL, \ .flags = GENL_ADMIN_PERM, \ @@ -29,7 +28,6 @@ void ieee802154_nl_exit(void); #define IEEE802154_DUMP(_cmd, _func, _dump) \ { \ .cmd = _cmd, \ - .policy = ieee802154_policy, \ .doit = _func, \ .dumpit = _dump, \ } diff --git a/net/ieee802154/netlink.c b/net/ieee802154/netlink.c index 96636e3b7aa9f36773aee5740134dc1c78d30715..098d67439b6d3e81eb6b85ac4aa0487a72223a5e 100644 --- a/net/ieee802154/netlink.c +++ b/net/ieee802154/netlink.c @@ -136,6 +136,7 @@ struct genl_family nl802154_family __ro_after_init = { .name = IEEE802154_NL_NAME, .version = 1, .maxattr = IEEE802154_ATTR_MAX, + .policy = ieee802154_policy, .module = THIS_MODULE, .ops = ieee802154_ops, .n_ops = ARRAY_SIZE(ieee802154_ops), diff --git a/net/ieee802154/nl802154.c b/net/ieee802154/nl802154.c index 99f6c254ea777c8786421d069f8f2414d0eb34e5..e4c4174f9efb994e25a53e34eeb01cda84da5bc7 100644 --- a/net/ieee802154/nl802154.c +++ b/net/ieee802154/nl802154.c @@ -247,9 +247,11 @@ nl802154_prepare_wpan_dev_dump(struct sk_buff *skb, rtnl_lock(); if (!cb->args[0]) { - err = nlmsg_parse(cb->nlh, GENL_HDRLEN + nl802154_fam.hdrsize, - genl_family_attrbuf(&nl802154_fam), - nl802154_fam.maxattr, nl802154_policy, NULL); + err = nlmsg_parse_deprecated(cb->nlh, + GENL_HDRLEN + nl802154_fam.hdrsize, + genl_family_attrbuf(&nl802154_fam), + nl802154_fam.maxattr, + nl802154_policy, NULL); if (err) goto out_unlock; @@ -312,7 +314,7 @@ static inline void *nl802154hdr_put(struct sk_buff *skb, u32 portid, u32 seq, static int nl802154_put_flags(struct sk_buff *msg, int attr, u32 mask) { - struct nlattr *nl_flags = nla_nest_start(msg, attr); + struct nlattr *nl_flags = nla_nest_start_noflag(msg, attr); int i; if (!nl_flags) @@ -338,7 +340,7 @@ nl802154_send_wpan_phy_channels(struct cfg802154_registered_device *rdev, struct nlattr *nl_page; unsigned long page; - nl_page = nla_nest_start(msg, NL802154_ATTR_CHANNELS_SUPPORTED); + nl_page = nla_nest_start_noflag(msg, NL802154_ATTR_CHANNELS_SUPPORTED); if (!nl_page) return -ENOBUFS; @@ -360,11 +362,11 @@ nl802154_put_capabilities(struct sk_buff *msg, struct nlattr *nl_caps, *nl_channels; int i; - nl_caps = nla_nest_start(msg, NL802154_ATTR_WPAN_PHY_CAPS); + nl_caps = nla_nest_start_noflag(msg, NL802154_ATTR_WPAN_PHY_CAPS); if (!nl_caps) return -ENOBUFS; - nl_channels = nla_nest_start(msg, NL802154_CAP_ATTR_CHANNELS); + nl_channels = nla_nest_start_noflag(msg, NL802154_CAP_ATTR_CHANNELS); if (!nl_channels) return -ENOBUFS; @@ -380,8 +382,8 @@ nl802154_put_capabilities(struct sk_buff *msg, if (rdev->wpan_phy.flags & WPAN_PHY_FLAG_CCA_ED_LEVEL) { struct nlattr *nl_ed_lvls; - nl_ed_lvls = nla_nest_start(msg, - NL802154_CAP_ATTR_CCA_ED_LEVELS); + nl_ed_lvls = nla_nest_start_noflag(msg, + NL802154_CAP_ATTR_CCA_ED_LEVELS); if (!nl_ed_lvls) return -ENOBUFS; @@ -396,7 +398,8 @@ nl802154_put_capabilities(struct sk_buff *msg, if (rdev->wpan_phy.flags & WPAN_PHY_FLAG_TXPOWER) { struct nlattr *nl_tx_pwrs; - nl_tx_pwrs = nla_nest_start(msg, NL802154_CAP_ATTR_TX_POWERS); + nl_tx_pwrs = nla_nest_start_noflag(msg, + NL802154_CAP_ATTR_TX_POWERS); if (!nl_tx_pwrs) return -ENOBUFS; @@ -504,7 +507,7 @@ static int nl802154_send_wpan_phy(struct cfg802154_registered_device *rdev, if (nl802154_put_capabilities(msg, rdev)) goto nla_put_failure; - nl_cmds = nla_nest_start(msg, NL802154_ATTR_SUPPORTED_COMMANDS); + nl_cmds = nla_nest_start_noflag(msg, NL802154_ATTR_SUPPORTED_COMMANDS); if (!nl_cmds) goto nla_put_failure; @@ -561,8 +564,10 @@ static int nl802154_dump_wpan_phy_parse(struct sk_buff *skb, struct nl802154_dump_wpan_phy_state *state) { struct nlattr **tb = genl_family_attrbuf(&nl802154_fam); - int ret = nlmsg_parse(cb->nlh, GENL_HDRLEN + nl802154_fam.hdrsize, tb, - nl802154_fam.maxattr, nl802154_policy, NULL); + int ret = nlmsg_parse_deprecated(cb->nlh, + GENL_HDRLEN + nl802154_fam.hdrsize, + tb, nl802154_fam.maxattr, + nl802154_policy, NULL); /* TODO check if we can handle error here, * we have no backward compatibility @@ -693,7 +698,8 @@ ieee802154_llsec_send_key_id(struct sk_buff *msg, switch (desc->mode) { case NL802154_KEY_ID_MODE_IMPLICIT: - nl_dev_addr = nla_nest_start(msg, NL802154_KEY_ID_ATTR_IMPLICIT); + nl_dev_addr = nla_nest_start_noflag(msg, + NL802154_KEY_ID_ATTR_IMPLICIT); if (!nl_dev_addr) return -ENOBUFS; @@ -768,7 +774,7 @@ static int nl802154_get_llsec_params(struct sk_buff *msg, params.frame_counter)) return -ENOBUFS; - nl_key_id = nla_nest_start(msg, NL802154_ATTR_SEC_OUT_KEY_ID); + nl_key_id = nla_nest_start_noflag(msg, NL802154_ATTR_SEC_OUT_KEY_ID); if (!nl_key_id) return -ENOBUFS; @@ -1306,8 +1312,7 @@ ieee802154_llsec_parse_dev_addr(struct nlattr *nla, { struct nlattr *attrs[NL802154_DEV_ADDR_ATTR_MAX + 1]; - if (!nla || nla_parse_nested(attrs, NL802154_DEV_ADDR_ATTR_MAX, nla, - nl802154_dev_addr_policy, NULL)) + if (!nla || nla_parse_nested_deprecated(attrs, NL802154_DEV_ADDR_ATTR_MAX, nla, nl802154_dev_addr_policy, NULL)) return -EINVAL; if (!attrs[NL802154_DEV_ADDR_ATTR_PAN_ID] || @@ -1346,8 +1351,7 @@ ieee802154_llsec_parse_key_id(struct nlattr *nla, { struct nlattr *attrs[NL802154_KEY_ID_ATTR_MAX + 1]; - if (!nla || nla_parse_nested(attrs, NL802154_KEY_ID_ATTR_MAX, nla, - nl802154_key_id_policy, NULL)) + if (!nla || nla_parse_nested_deprecated(attrs, NL802154_KEY_ID_ATTR_MAX, nla, nl802154_key_id_policy, NULL)) return -EINVAL; if (!attrs[NL802154_KEY_ID_ATTR_MODE]) @@ -1455,11 +1459,11 @@ static int nl802154_send_key(struct sk_buff *msg, u32 cmd, u32 portid, if (nla_put_u32(msg, NL802154_ATTR_IFINDEX, dev->ifindex)) goto nla_put_failure; - nl_key = nla_nest_start(msg, NL802154_ATTR_SEC_KEY); + nl_key = nla_nest_start_noflag(msg, NL802154_ATTR_SEC_KEY); if (!nl_key) goto nla_put_failure; - nl_key_id = nla_nest_start(msg, NL802154_KEY_ATTR_ID); + nl_key_id = nla_nest_start_noflag(msg, NL802154_KEY_ATTR_ID); if (!nl_key_id) goto nla_put_failure; @@ -1562,9 +1566,7 @@ static int nl802154_add_llsec_key(struct sk_buff *skb, struct genl_info *info) struct ieee802154_llsec_key_id id = { }; u32 commands[NL802154_CMD_FRAME_NR_IDS / 32] = { }; - if (nla_parse_nested(attrs, NL802154_KEY_ATTR_MAX, - info->attrs[NL802154_ATTR_SEC_KEY], - nl802154_key_policy, info->extack)) + if (nla_parse_nested_deprecated(attrs, NL802154_KEY_ATTR_MAX, info->attrs[NL802154_ATTR_SEC_KEY], nl802154_key_policy, info->extack)) return -EINVAL; if (!attrs[NL802154_KEY_ATTR_USAGE_FRAMES] || @@ -1612,9 +1614,7 @@ static int nl802154_del_llsec_key(struct sk_buff *skb, struct genl_info *info) struct nlattr *attrs[NL802154_KEY_ATTR_MAX + 1]; struct ieee802154_llsec_key_id id; - if (nla_parse_nested(attrs, NL802154_KEY_ATTR_MAX, - info->attrs[NL802154_ATTR_SEC_KEY], - nl802154_key_policy, info->extack)) + if (nla_parse_nested_deprecated(attrs, NL802154_KEY_ATTR_MAX, info->attrs[NL802154_ATTR_SEC_KEY], nl802154_key_policy, info->extack)) return -EINVAL; if (ieee802154_llsec_parse_key_id(attrs[NL802154_KEY_ATTR_ID], &id) < 0) @@ -1639,7 +1639,7 @@ static int nl802154_send_device(struct sk_buff *msg, u32 cmd, u32 portid, if (nla_put_u32(msg, NL802154_ATTR_IFINDEX, dev->ifindex)) goto nla_put_failure; - nl_device = nla_nest_start(msg, NL802154_ATTR_SEC_DEVICE); + nl_device = nla_nest_start_noflag(msg, NL802154_ATTR_SEC_DEVICE); if (!nl_device) goto nla_put_failure; @@ -1728,8 +1728,7 @@ ieee802154_llsec_parse_device(struct nlattr *nla, { struct nlattr *attrs[NL802154_DEV_ATTR_MAX + 1]; - if (!nla || nla_parse_nested(attrs, NL802154_DEV_ATTR_MAX, - nla, nl802154_dev_policy, NULL)) + if (!nla || nla_parse_nested_deprecated(attrs, NL802154_DEV_ATTR_MAX, nla, nl802154_dev_policy, NULL)) return -EINVAL; memset(dev, 0, sizeof(*dev)); @@ -1780,9 +1779,7 @@ static int nl802154_del_llsec_dev(struct sk_buff *skb, struct genl_info *info) struct nlattr *attrs[NL802154_DEV_ATTR_MAX + 1]; __le64 extended_addr; - if (nla_parse_nested(attrs, NL802154_DEV_ATTR_MAX, - info->attrs[NL802154_ATTR_SEC_DEVICE], - nl802154_dev_policy, info->extack)) + if (nla_parse_nested_deprecated(attrs, NL802154_DEV_ATTR_MAX, info->attrs[NL802154_ATTR_SEC_DEVICE], nl802154_dev_policy, info->extack)) return -EINVAL; if (!attrs[NL802154_DEV_ATTR_EXTENDED_ADDR]) @@ -1808,7 +1805,7 @@ static int nl802154_send_devkey(struct sk_buff *msg, u32 cmd, u32 portid, if (nla_put_u32(msg, NL802154_ATTR_IFINDEX, dev->ifindex)) goto nla_put_failure; - nl_devkey = nla_nest_start(msg, NL802154_ATTR_SEC_DEVKEY); + nl_devkey = nla_nest_start_noflag(msg, NL802154_ATTR_SEC_DEVKEY); if (!nl_devkey) goto nla_put_failure; @@ -1818,7 +1815,7 @@ static int nl802154_send_devkey(struct sk_buff *msg, u32 cmd, u32 portid, devkey->frame_counter)) goto nla_put_failure; - nl_key_id = nla_nest_start(msg, NL802154_DEVKEY_ATTR_ID); + nl_key_id = nla_nest_start_noflag(msg, NL802154_DEVKEY_ATTR_ID); if (!nl_key_id) goto nla_put_failure; @@ -1908,9 +1905,7 @@ static int nl802154_add_llsec_devkey(struct sk_buff *skb, struct genl_info *info __le64 extended_addr; if (!info->attrs[NL802154_ATTR_SEC_DEVKEY] || - nla_parse_nested(attrs, NL802154_DEVKEY_ATTR_MAX, - info->attrs[NL802154_ATTR_SEC_DEVKEY], - nl802154_devkey_policy, info->extack) < 0) + nla_parse_nested_deprecated(attrs, NL802154_DEVKEY_ATTR_MAX, info->attrs[NL802154_ATTR_SEC_DEVKEY], nl802154_devkey_policy, info->extack) < 0) return -EINVAL; if (!attrs[NL802154_DEVKEY_ATTR_FRAME_COUNTER] || @@ -1940,9 +1935,7 @@ static int nl802154_del_llsec_devkey(struct sk_buff *skb, struct genl_info *info struct ieee802154_llsec_device_key key; __le64 extended_addr; - if (nla_parse_nested(attrs, NL802154_DEVKEY_ATTR_MAX, - info->attrs[NL802154_ATTR_SEC_DEVKEY], - nl802154_devkey_policy, info->extack)) + if (nla_parse_nested_deprecated(attrs, NL802154_DEVKEY_ATTR_MAX, info->attrs[NL802154_ATTR_SEC_DEVKEY], nl802154_devkey_policy, info->extack)) return -EINVAL; if (!attrs[NL802154_DEVKEY_ATTR_EXTENDED_ADDR]) @@ -1976,7 +1969,7 @@ static int nl802154_send_seclevel(struct sk_buff *msg, u32 cmd, u32 portid, if (nla_put_u32(msg, NL802154_ATTR_IFINDEX, dev->ifindex)) goto nla_put_failure; - nl_seclevel = nla_nest_start(msg, NL802154_ATTR_SEC_LEVEL); + nl_seclevel = nla_nest_start_noflag(msg, NL802154_ATTR_SEC_LEVEL); if (!nl_seclevel) goto nla_put_failure; @@ -2062,8 +2055,7 @@ llsec_parse_seclevel(struct nlattr *nla, struct ieee802154_llsec_seclevel *sl) { struct nlattr *attrs[NL802154_SECLEVEL_ATTR_MAX + 1]; - if (!nla || nla_parse_nested(attrs, NL802154_SECLEVEL_ATTR_MAX, - nla, nl802154_seclevel_policy, NULL)) + if (!nla || nla_parse_nested_deprecated(attrs, NL802154_SECLEVEL_ATTR_MAX, nla, nl802154_seclevel_policy, NULL)) return -EINVAL; memset(sl, 0, sizeof(*sl)); @@ -2217,131 +2209,131 @@ static void nl802154_post_doit(const struct genl_ops *ops, struct sk_buff *skb, static const struct genl_ops nl802154_ops[] = { { .cmd = NL802154_CMD_GET_WPAN_PHY, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_get_wpan_phy, .dumpit = nl802154_dump_wpan_phy, .done = nl802154_dump_wpan_phy_done, - .policy = nl802154_policy, /* can be retrieved by unprivileged users */ .internal_flags = NL802154_FLAG_NEED_WPAN_PHY | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_GET_INTERFACE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_get_interface, .dumpit = nl802154_dump_interface, - .policy = nl802154_policy, /* can be retrieved by unprivileged users */ .internal_flags = NL802154_FLAG_NEED_WPAN_DEV | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_NEW_INTERFACE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_new_interface, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_WPAN_PHY | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_DEL_INTERFACE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_del_interface, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_WPAN_DEV | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_SET_CHANNEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_set_channel, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_WPAN_PHY | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_SET_CCA_MODE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_set_cca_mode, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_WPAN_PHY | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_SET_CCA_ED_LEVEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_set_cca_ed_level, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_WPAN_PHY | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_SET_TX_POWER, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_set_tx_power, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_WPAN_PHY | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_SET_WPAN_PHY_NETNS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_wpan_phy_netns, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_WPAN_PHY | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_SET_PAN_ID, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_set_pan_id, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_SET_SHORT_ADDR, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_set_short_addr, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_SET_BACKOFF_EXPONENT, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_set_backoff_exponent, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_SET_MAX_CSMA_BACKOFFS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_set_max_csma_backoffs, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_SET_MAX_FRAME_RETRIES, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_set_max_frame_retries, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_SET_LBT_MODE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_set_lbt_mode, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_SET_ACKREQ_DEFAULT, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_set_ackreq_default, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, @@ -2349,33 +2341,33 @@ static const struct genl_ops nl802154_ops[] = { #ifdef CONFIG_IEEE802154_NL802154_EXPERIMENTAL { .cmd = NL802154_CMD_SET_SEC_PARAMS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_set_llsec_params, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_GET_SEC_KEY, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, /* TODO .doit by matching key id? */ .dumpit = nl802154_dump_llsec_key, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_NEW_SEC_KEY, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_add_llsec_key, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_DEL_SEC_KEY, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_del_llsec_key, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, @@ -2383,25 +2375,25 @@ static const struct genl_ops nl802154_ops[] = { /* TODO unique identifier must short+pan OR extended_addr */ { .cmd = NL802154_CMD_GET_SEC_DEV, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, /* TODO .doit by matching extended_addr? */ .dumpit = nl802154_dump_llsec_dev, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_NEW_SEC_DEV, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_add_llsec_dev, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_DEL_SEC_DEV, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_del_llsec_dev, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, @@ -2409,51 +2401,51 @@ static const struct genl_ops nl802154_ops[] = { /* TODO remove complete devkey, put it as nested? */ { .cmd = NL802154_CMD_GET_SEC_DEVKEY, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, /* TODO doit by matching ??? */ .dumpit = nl802154_dump_llsec_devkey, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_NEW_SEC_DEVKEY, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_add_llsec_devkey, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_DEL_SEC_DEVKEY, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_del_llsec_devkey, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_GET_SEC_LEVEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, /* TODO .doit by matching frame_type? */ .dumpit = nl802154_dump_llsec_seclevel, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_NEW_SEC_LEVEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl802154_add_llsec_seclevel, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, }, { .cmd = NL802154_CMD_DEL_SEC_LEVEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, /* TODO match frame_type only? */ .doit = nl802154_del_llsec_seclevel, - .policy = nl802154_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL802154_FLAG_NEED_NETDEV | NL802154_FLAG_NEED_RTNL, @@ -2466,6 +2458,7 @@ static struct genl_family nl802154_fam __ro_after_init = { .hdrsize = 0, /* no private header */ .version = 1, /* no particular meaning now */ .maxattr = NL802154_ATTR_MAX, + .policy = nl802154_policy, .netnsok = true, .pre_doit = nl802154_pre_doit, .post_doit = nl802154_post_doit, diff --git a/net/ieee802154/socket.c b/net/ieee802154/socket.c index bc6b912603f164b981e761974e4e64f39b01dc8f..ce2dfb99753760094133cb0d1e5bb344772192b2 100644 --- a/net/ieee802154/socket.c +++ b/net/ieee802154/socket.c @@ -164,10 +164,6 @@ static int ieee802154_sock_ioctl(struct socket *sock, unsigned int cmd, struct sock *sk = sock->sk; switch (cmd) { - case SIOCGSTAMP: - return sock_get_timestamp(sk, (struct timeval __user *)arg); - case SIOCGSTAMPNS: - return sock_get_timestampns(sk, (struct timespec __user *)arg); case SIOCGIFADDR: case SIOCSIFADDR: return ieee802154_dev_ioctl(sk, (struct ifreq __user *)arg, @@ -426,6 +422,7 @@ static const struct proto_ops ieee802154_raw_ops = { .getname = sock_no_getname, .poll = datagram_poll, .ioctl = ieee802154_sock_ioctl, + .gettstamp = sock_gettstamp, .listen = sock_no_listen, .shutdown = sock_no_shutdown, .setsockopt = sock_common_setsockopt, @@ -988,6 +985,7 @@ static const struct proto_ops ieee802154_dgram_ops = { .getname = sock_no_getname, .poll = datagram_poll, .ioctl = ieee802154_sock_ioctl, + .gettstamp = sock_gettstamp, .listen = sock_no_listen, .shutdown = sock_no_shutdown, .setsockopt = sock_common_setsockopt, diff --git a/net/ipv4/Kconfig b/net/ipv4/Kconfig index 32cae39cdff63125cb7213ae6644e8ff885796c0..8108e97d42854e3f0b118f68e463a9f22fc94e22 100644 --- a/net/ipv4/Kconfig +++ b/net/ipv4/Kconfig @@ -304,7 +304,7 @@ config NET_IPVTI tristate "Virtual (secure) IP: tunneling" select INET_TUNNEL select NET_IP_TUNNEL - depends on INET_XFRM_MODE_TUNNEL + select XFRM ---help--- Tunneling means encapsulating data of one protocol type within another protocol and sending it over a channel that understands the @@ -396,33 +396,6 @@ config INET_TUNNEL tristate default n -config INET_XFRM_MODE_TRANSPORT - tristate "IP: IPsec transport mode" - default y - select XFRM - ---help--- - Support for IPsec transport mode. - - If unsure, say Y. - -config INET_XFRM_MODE_TUNNEL - tristate "IP: IPsec tunnel mode" - default y - select XFRM - ---help--- - Support for IPsec tunnel mode. - - If unsure, say Y. - -config INET_XFRM_MODE_BEET - tristate "IP: IPsec BEET mode" - default y - select XFRM - ---help--- - Support for IPsec BEET mode. - - If unsure, say Y. - config INET_DIAG tristate "INET: socket monitoring interface" default y diff --git a/net/ipv4/Makefile b/net/ipv4/Makefile index 58629314eae95c1e09c5f27f00426dcf1017abb4..000a61994c8f2ff1505aaa95b559359cb020870a 100644 --- a/net/ipv4/Makefile +++ b/net/ipv4/Makefile @@ -37,10 +37,7 @@ obj-$(CONFIG_INET_ESP) += esp4.o obj-$(CONFIG_INET_ESP_OFFLOAD) += esp4_offload.o obj-$(CONFIG_INET_IPCOMP) += ipcomp.o obj-$(CONFIG_INET_XFRM_TUNNEL) += xfrm4_tunnel.o -obj-$(CONFIG_INET_XFRM_MODE_BEET) += xfrm4_mode_beet.o obj-$(CONFIG_INET_TUNNEL) += tunnel4.o -obj-$(CONFIG_INET_XFRM_MODE_TRANSPORT) += xfrm4_mode_transport.o -obj-$(CONFIG_INET_XFRM_MODE_TUNNEL) += xfrm4_mode_tunnel.o obj-$(CONFIG_IP_PNP) += ipconfig.o obj-$(CONFIG_NETFILTER) += netfilter.o netfilter/ obj-$(CONFIG_INET_DIAG) += inet_diag.o diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c index eab3ebde981e78a6a0a4852c3b4374c02ede1187..5183a2daba643b9c07f279f10672d98cd59ed437 100644 --- a/net/ipv4/af_inet.c +++ b/net/ipv4/af_inet.c @@ -136,6 +136,10 @@ void inet_sock_destruct(struct sock *sk) struct inet_sock *inet = inet_sk(sk); __skb_queue_purge(&sk->sk_receive_queue); + if (sk->sk_rx_skb_cache) { + __kfree_skb(sk->sk_rx_skb_cache); + sk->sk_rx_skb_cache = NULL; + } __skb_queue_purge(&sk->sk_error_queue); sk_mem_reclaim(sk); @@ -156,7 +160,7 @@ void inet_sock_destruct(struct sock *sk) WARN_ON(sk->sk_forward_alloc); kfree(rcu_dereference_protected(inet->inet_opt, 1)); - dst_release(rcu_dereference_check(sk->sk_dst_cache, 1)); + dst_release(rcu_dereference_protected(sk->sk_dst_cache, 1)); dst_release(sk->sk_rx_dst); sk_refcnt_debug_dec(sk); } @@ -911,12 +915,6 @@ int inet_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) struct rtentry rt; switch (cmd) { - case SIOCGSTAMP: - err = sock_get_timestamp(sk, (struct timeval __user *)arg); - break; - case SIOCGSTAMPNS: - err = sock_get_timestampns(sk, (struct timespec __user *)arg); - break; case SIOCADDRT: case SIOCDELRT: if (copy_from_user(&rt, p, sizeof(struct rtentry))) @@ -988,6 +986,7 @@ const struct proto_ops inet_stream_ops = { .getname = inet_getname, .poll = tcp_poll, .ioctl = inet_ioctl, + .gettstamp = sock_gettstamp, .listen = inet_listen, .shutdown = inet_shutdown, .setsockopt = sock_common_setsockopt, @@ -1023,6 +1022,7 @@ const struct proto_ops inet_dgram_ops = { .getname = inet_getname, .poll = udp_poll, .ioctl = inet_ioctl, + .gettstamp = sock_gettstamp, .listen = sock_no_listen, .shutdown = inet_shutdown, .setsockopt = sock_common_setsockopt, @@ -1055,6 +1055,7 @@ static const struct proto_ops inet_sockraw_ops = { .getname = inet_getname, .poll = datagram_poll, .ioctl = inet_ioctl, + .gettstamp = sock_gettstamp, .listen = sock_no_listen, .shutdown = inet_shutdown, .setsockopt = sock_common_setsockopt, diff --git a/net/ipv4/bpfilter/sockopt.c b/net/ipv4/bpfilter/sockopt.c index 1e976bb93d99821ba63e444017bd839f231b7092..15427163a041d4dc9d4f7e35ee4d207fd6e5ba2a 100644 --- a/net/ipv4/bpfilter/sockopt.c +++ b/net/ipv4/bpfilter/sockopt.c @@ -77,5 +77,4 @@ static int __init bpfilter_sockopt_init(void) return 0; } - -module_init(bpfilter_sockopt_init); +device_initcall(bpfilter_sockopt_init); diff --git a/net/ipv4/devinet.c b/net/ipv4/devinet.c index eb514f312e6fa3f3aa531914f2b4c9973cc3afc6..701c5d113a34d7066c6e0e8d7f53fcaafa08cd23 100644 --- a/net/ipv4/devinet.c +++ b/net/ipv4/devinet.c @@ -621,8 +621,8 @@ static int inet_rtm_deladdr(struct sk_buff *skb, struct nlmsghdr *nlh, ASSERT_RTNL(); - err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, ifa_ipv4_policy, - extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*ifm), tb, IFA_MAX, + ifa_ipv4_policy, extack); if (err < 0) goto errout; @@ -793,8 +793,8 @@ static struct in_ifaddr *rtm_to_ifaddr(struct net *net, struct nlmsghdr *nlh, struct in_device *in_dev; int err; - err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, ifa_ipv4_policy, - extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*ifm), tb, IFA_MAX, + ifa_ipv4_policy, extack); if (err < 0) goto errout; @@ -1689,8 +1689,8 @@ static int inet_valid_dump_ifaddr_req(const struct nlmsghdr *nlh, fillargs->flags |= NLM_F_DUMP_FILTERED; } - err = nlmsg_parse_strict(nlh, sizeof(*ifm), tb, IFA_MAX, - ifa_ipv4_policy, extack); + err = nlmsg_parse_deprecated_strict(nlh, sizeof(*ifm), tb, IFA_MAX, + ifa_ipv4_policy, extack); if (err < 0) return err; @@ -1906,7 +1906,8 @@ static int inet_validate_link_af(const struct net_device *dev, if (dev && !__in_dev_get_rcu(dev)) return -EAFNOSUPPORT; - err = nla_parse_nested(tb, IFLA_INET_MAX, nla, inet_af_policy, NULL); + err = nla_parse_nested_deprecated(tb, IFLA_INET_MAX, nla, + inet_af_policy, NULL); if (err < 0) return err; @@ -1934,7 +1935,7 @@ static int inet_set_link_af(struct net_device *dev, const struct nlattr *nla) if (!in_dev) return -EAFNOSUPPORT; - if (nla_parse_nested(tb, IFLA_INET_MAX, nla, NULL, NULL) < 0) + if (nla_parse_nested_deprecated(tb, IFLA_INET_MAX, nla, NULL, NULL) < 0) BUG(); if (tb[IFLA_INET_CONF]) { @@ -2076,11 +2077,13 @@ static int inet_netconf_valid_get_req(struct sk_buff *skb, } if (!netlink_strict_get_check(skb)) - return nlmsg_parse(nlh, sizeof(struct netconfmsg), tb, - NETCONFA_MAX, devconf_ipv4_policy, extack); + return nlmsg_parse_deprecated(nlh, sizeof(struct netconfmsg), + tb, NETCONFA_MAX, + devconf_ipv4_policy, extack); - err = nlmsg_parse_strict(nlh, sizeof(struct netconfmsg), tb, - NETCONFA_MAX, devconf_ipv4_policy, extack); + err = nlmsg_parse_deprecated_strict(nlh, sizeof(struct netconfmsg), + tb, NETCONFA_MAX, + devconf_ipv4_policy, extack); if (err) return err; diff --git a/net/ipv4/esp4_offload.c b/net/ipv4/esp4_offload.c index d3170a8001b2a15bffe5a37ab9e0b556663454a4..8edcfa66d1e5ed0f224ba271d6898f1837fc5c76 100644 --- a/net/ipv4/esp4_offload.c +++ b/net/ipv4/esp4_offload.c @@ -109,6 +109,44 @@ static void esp4_gso_encap(struct xfrm_state *x, struct sk_buff *skb) xo->proto = proto; } +static struct sk_buff *xfrm4_tunnel_gso_segment(struct xfrm_state *x, + struct sk_buff *skb, + netdev_features_t features) +{ + __skb_push(skb, skb->mac_len); + return skb_mac_gso_segment(skb, features); +} + +static struct sk_buff *xfrm4_transport_gso_segment(struct xfrm_state *x, + struct sk_buff *skb, + netdev_features_t features) +{ + const struct net_offload *ops; + struct sk_buff *segs = ERR_PTR(-EINVAL); + struct xfrm_offload *xo = xfrm_offload(skb); + + skb->transport_header += x->props.header_len; + ops = rcu_dereference(inet_offloads[xo->proto]); + if (likely(ops && ops->callbacks.gso_segment)) + segs = ops->callbacks.gso_segment(skb, features); + + return segs; +} + +static struct sk_buff *xfrm4_outer_mode_gso_segment(struct xfrm_state *x, + struct sk_buff *skb, + netdev_features_t features) +{ + switch (x->outer_mode.encap) { + case XFRM_MODE_TUNNEL: + return xfrm4_tunnel_gso_segment(x, skb, features); + case XFRM_MODE_TRANSPORT: + return xfrm4_transport_gso_segment(x, skb, features); + } + + return ERR_PTR(-EOPNOTSUPP); +} + static struct sk_buff *esp4_gso_segment(struct sk_buff *skb, netdev_features_t features) { @@ -140,14 +178,16 @@ static struct sk_buff *esp4_gso_segment(struct sk_buff *skb, skb->encap_hdr_csum = 1; - if (!(features & NETIF_F_HW_ESP) || x->xso.dev != skb->dev) + if ((!(skb->dev->gso_partial_features & NETIF_F_HW_ESP) && + !(features & NETIF_F_HW_ESP)) || x->xso.dev != skb->dev) esp_features = features & ~(NETIF_F_SG | NETIF_F_CSUM_MASK); - else if (!(features & NETIF_F_HW_ESP_TX_CSUM)) + else if (!(features & NETIF_F_HW_ESP_TX_CSUM) && + !(skb->dev->gso_partial_features & NETIF_F_HW_ESP_TX_CSUM)) esp_features = features & ~NETIF_F_CSUM_MASK; xo->flags |= XFRM_GSO_SEGMENT; - return x->outer_mode->gso_segment(x, skb, esp_features); + return xfrm4_outer_mode_gso_segment(x, skb, esp_features); } static int esp_input_tail(struct xfrm_state *x, struct sk_buff *skb) @@ -183,7 +223,9 @@ static int esp_xmit(struct xfrm_state *x, struct sk_buff *skb, netdev_features_ if (!xo) return -EINVAL; - if (!(features & NETIF_F_HW_ESP) || x->xso.dev != skb->dev) { + if ((!(features & NETIF_F_HW_ESP) && + !(skb->dev->gso_partial_features & NETIF_F_HW_ESP)) || + x->xso.dev != skb->dev) { xo->flags |= CRYPTO_FALLBACK; hw_offload = false; } diff --git a/net/ipv4/fib_frontend.c b/net/ipv4/fib_frontend.c index ed14ec2455847f2f104cb33fcd6a64cd37b87b40..b298255f6fdb5fc866c7214f0c04aff264f7e55e 100644 --- a/net/ipv4/fib_frontend.c +++ b/net/ipv4/fib_frontend.c @@ -307,7 +307,7 @@ __be32 fib_compute_spec_dst(struct sk_buff *skb) .flowi4_mark = vmark ? skb->mark : 0, }; if (!fib_lookup(net, &fl4, &res, 0)) - return FIB_RES_PREFSRC(net, res); + return fib_result_prefsrc(net, &res); } else { scope = RT_SCOPE_LINK; } @@ -324,16 +324,16 @@ bool fib_info_nh_uses_dev(struct fib_info *fi, const struct net_device *dev) for (ret = 0; ret < fi->fib_nhs; ret++) { struct fib_nh *nh = &fi->fib_nh[ret]; - if (nh->nh_dev == dev) { + if (nh->fib_nh_dev == dev) { dev_match = true; break; - } else if (l3mdev_master_ifindex_rcu(nh->nh_dev) == dev->ifindex) { + } else if (l3mdev_master_ifindex_rcu(nh->fib_nh_dev) == dev->ifindex) { dev_match = true; break; } } #else - if (fi->fib_nh[0].nh_dev == dev) + if (fi->fib_nh[0].fib_nh_dev == dev) dev_match = true; #endif @@ -390,7 +390,7 @@ static int __fib_validate_source(struct sk_buff *skb, __be32 src, __be32 dst, dev_match = fib_info_nh_uses_dev(res.fi, dev); if (dev_match) { - ret = FIB_RES_NH(res).nh_scope >= RT_SCOPE_HOST; + ret = FIB_RES_NHC(res)->nhc_scope >= RT_SCOPE_HOST; return ret; } if (no_addr) @@ -402,7 +402,7 @@ static int __fib_validate_source(struct sk_buff *skb, __be32 src, __be32 dst, ret = 0; if (fib_lookup(net, &fl4, &res, FIB_LOOKUP_IGNORE_LINKSTATE) == 0) { if (res.type == RTN_UNICAST) - ret = FIB_RES_NH(res).nh_scope >= RT_SCOPE_HOST; + ret = FIB_RES_NHC(res)->nhc_scope >= RT_SCOPE_HOST; } return ret; @@ -558,7 +558,8 @@ static int rtentry_to_fib_config(struct net *net, int cmd, struct rtentry *rt, if (rt->rt_gateway.sa_family == AF_INET && addr) { unsigned int addr_type; - cfg->fc_gw = addr; + cfg->fc_gw4 = addr; + cfg->fc_gw_family = AF_INET; addr_type = inet_addr_type_table(net, addr, cfg->fc_table); if (rt->rt_flags & RTF_GATEWAY && addr_type == RTN_UNICAST) @@ -568,7 +569,7 @@ static int rtentry_to_fib_config(struct net *net, int cmd, struct rtentry *rt, if (cmd == SIOCDELRT) return 0; - if (rt->rt_flags & RTF_GATEWAY && !cfg->fc_gw) + if (rt->rt_flags & RTF_GATEWAY && !cfg->fc_gw_family) return -EINVAL; if (cfg->fc_scope == RT_SCOPE_NOWHERE) @@ -664,16 +665,61 @@ const struct nla_policy rtm_ipv4_policy[RTA_MAX + 1] = { [RTA_DPORT] = { .type = NLA_U16 }, }; +int fib_gw_from_via(struct fib_config *cfg, struct nlattr *nla, + struct netlink_ext_ack *extack) +{ + struct rtvia *via; + int alen; + + if (nla_len(nla) < offsetof(struct rtvia, rtvia_addr)) { + NL_SET_ERR_MSG(extack, "Invalid attribute length for RTA_VIA"); + return -EINVAL; + } + + via = nla_data(nla); + alen = nla_len(nla) - offsetof(struct rtvia, rtvia_addr); + + switch (via->rtvia_family) { + case AF_INET: + if (alen != sizeof(__be32)) { + NL_SET_ERR_MSG(extack, "Invalid IPv4 address in RTA_VIA"); + return -EINVAL; + } + cfg->fc_gw_family = AF_INET; + cfg->fc_gw4 = *((__be32 *)via->rtvia_addr); + break; + case AF_INET6: +#ifdef CONFIG_IPV6 + if (alen != sizeof(struct in6_addr)) { + NL_SET_ERR_MSG(extack, "Invalid IPv6 address in RTA_VIA"); + return -EINVAL; + } + cfg->fc_gw_family = AF_INET6; + cfg->fc_gw6 = *((struct in6_addr *)via->rtvia_addr); +#else + NL_SET_ERR_MSG(extack, "IPv6 support not enabled in kernel"); + return -EINVAL; +#endif + break; + default: + NL_SET_ERR_MSG(extack, "Unsupported address family in RTA_VIA"); + return -EINVAL; + } + + return 0; +} + static int rtm_to_fib_config(struct net *net, struct sk_buff *skb, struct nlmsghdr *nlh, struct fib_config *cfg, struct netlink_ext_ack *extack) { + bool has_gw = false, has_via = false; struct nlattr *attr; int err, remaining; struct rtmsg *rtm; - err = nlmsg_validate(nlh, sizeof(*rtm), RTA_MAX, rtm_ipv4_policy, - extack); + err = nlmsg_validate_deprecated(nlh, sizeof(*rtm), RTA_MAX, + rtm_ipv4_policy, extack); if (err < 0) goto errout; @@ -708,12 +754,17 @@ static int rtm_to_fib_config(struct net *net, struct sk_buff *skb, cfg->fc_oif = nla_get_u32(attr); break; case RTA_GATEWAY: - cfg->fc_gw = nla_get_be32(attr); + has_gw = true; + cfg->fc_gw4 = nla_get_be32(attr); + if (cfg->fc_gw4) + cfg->fc_gw_family = AF_INET; break; case RTA_VIA: - NL_SET_ERR_MSG(extack, "IPv4 does not support RTA_VIA attribute"); - err = -EINVAL; - goto errout; + has_via = true; + err = fib_gw_from_via(cfg, attr, extack); + if (err) + goto errout; + break; case RTA_PRIORITY: cfg->fc_priority = nla_get_u32(attr); break; @@ -752,6 +803,12 @@ static int rtm_to_fib_config(struct net *net, struct sk_buff *skb, } } + if (has_gw && has_via) { + NL_SET_ERR_MSG(extack, + "Nexthop configuration can not contain both GATEWAY and VIA"); + goto errout; + } + return 0; errout: return err; @@ -839,8 +896,8 @@ int ip_valid_fib_dump_req(struct net *net, const struct nlmsghdr *nlh, filter->rt_type = rtm->rtm_type; filter->table_id = rtm->rtm_table; - err = nlmsg_parse_strict(nlh, sizeof(*rtm), tb, RTA_MAX, - rtm_ipv4_policy, extack); + err = nlmsg_parse_deprecated_strict(nlh, sizeof(*rtm), tb, RTA_MAX, + rtm_ipv4_policy, extack); if (err < 0) return err; diff --git a/net/ipv4/fib_lookup.h b/net/ipv4/fib_lookup.h index e6ff282bb7f42c2a941ae6c3f2785ca989150ad1..7945f0534db72ecda20ab7f77ae499a0270d3cd1 100644 --- a/net/ipv4/fib_lookup.h +++ b/net/ipv4/fib_lookup.h @@ -45,6 +45,7 @@ static inline void fib_result_assign(struct fib_result *res, { /* we used to play games with refcounts, but we now use RCU */ res->fi = fi; + res->nhc = fib_info_nhc(fi, 0); } struct fib_prop { diff --git a/net/ipv4/fib_semantics.c b/net/ipv4/fib_semantics.c index 8e185b5a2bf6e7c15e83b122ec5e0a93d165273d..d3da6a10f86f4b7e0d34b3851862a290c0b33d40 100644 --- a/net/ipv4/fib_semantics.c +++ b/net/ipv4/fib_semantics.c @@ -41,10 +41,12 @@ #include #include #include +#include #include -#include +#include #include #include +#include #include "fib_lookup.h" @@ -157,12 +159,12 @@ static void rt_fibinfo_free(struct rtable __rcu **rtp) dst_release_immediate(&rt->dst); } -static void free_nh_exceptions(struct fib_nh *nh) +static void free_nh_exceptions(struct fib_nh_common *nhc) { struct fnhe_hash_bucket *hash; int i; - hash = rcu_dereference_protected(nh->nh_exceptions, 1); + hash = rcu_dereference_protected(nhc->nhc_exceptions, 1); if (!hash) return; for (i = 0; i < FNHE_HASH_SIZE; i++) { @@ -204,18 +206,34 @@ static void rt_fibinfo_free_cpus(struct rtable __rcu * __percpu *rtp) free_percpu(rtp); } +void fib_nh_common_release(struct fib_nh_common *nhc) +{ + if (nhc->nhc_dev) + dev_put(nhc->nhc_dev); + + lwtstate_put(nhc->nhc_lwtstate); + rt_fibinfo_free_cpus(nhc->nhc_pcpu_rth_output); + rt_fibinfo_free(&nhc->nhc_rth_input); + free_nh_exceptions(nhc); +} +EXPORT_SYMBOL_GPL(fib_nh_common_release); + +void fib_nh_release(struct net *net, struct fib_nh *fib_nh) +{ +#ifdef CONFIG_IP_ROUTE_CLASSID + if (fib_nh->nh_tclassid) + net->ipv4.fib_num_tclassid_users--; +#endif + fib_nh_common_release(&fib_nh->nh_common); +} + /* Release a nexthop info record */ static void free_fib_info_rcu(struct rcu_head *head) { struct fib_info *fi = container_of(head, struct fib_info, rcu); change_nexthops(fi) { - if (nexthop_nh->nh_dev) - dev_put(nexthop_nh->nh_dev); - lwtstate_put(nexthop_nh->nh_lwtstate); - free_nh_exceptions(nexthop_nh); - rt_fibinfo_free_cpus(nexthop_nh->nh_pcpu_rth_output); - rt_fibinfo_free(&nexthop_nh->nh_rth_input); + fib_nh_release(fi->fib_net, nexthop_nh); } endfor_nexthops(fi); ip_fib_metrics_put(fi->fib_metrics); @@ -230,12 +248,7 @@ void free_fib_info(struct fib_info *fi) return; } fib_info_cnt--; -#ifdef CONFIG_IP_ROUTE_CLASSID - change_nexthops(fi) { - if (nexthop_nh->nh_tclassid) - fi->fib_net->ipv4.fib_num_tclassid_users--; - } endfor_nexthops(fi); -#endif + call_rcu(&fi->rcu, free_fib_info_rcu); } EXPORT_SYMBOL_GPL(free_fib_info); @@ -248,7 +261,7 @@ void fib_release_info(struct fib_info *fi) if (fi->fib_prefsrc) hlist_del(&fi->fib_lhash); change_nexthops(fi) { - if (!nexthop_nh->nh_dev) + if (!nexthop_nh->fib_nh_dev) continue; hlist_del(&nexthop_nh->nh_hash); } endfor_nexthops(fi) @@ -263,18 +276,27 @@ static inline int nh_comp(const struct fib_info *fi, const struct fib_info *ofi) const struct fib_nh *onh = ofi->fib_nh; for_nexthops(fi) { - if (nh->nh_oif != onh->nh_oif || - nh->nh_gw != onh->nh_gw || - nh->nh_scope != onh->nh_scope || + if (nh->fib_nh_oif != onh->fib_nh_oif || + nh->fib_nh_gw_family != onh->fib_nh_gw_family || + nh->fib_nh_scope != onh->fib_nh_scope || #ifdef CONFIG_IP_ROUTE_MULTIPATH - nh->nh_weight != onh->nh_weight || + nh->fib_nh_weight != onh->fib_nh_weight || #endif #ifdef CONFIG_IP_ROUTE_CLASSID nh->nh_tclassid != onh->nh_tclassid || #endif - lwtunnel_cmp_encap(nh->nh_lwtstate, onh->nh_lwtstate) || - ((nh->nh_flags ^ onh->nh_flags) & ~RTNH_COMPARE_MASK)) + lwtunnel_cmp_encap(nh->fib_nh_lws, onh->fib_nh_lws) || + ((nh->fib_nh_flags ^ onh->fib_nh_flags) & ~RTNH_COMPARE_MASK)) + return -1; + + if (nh->fib_nh_gw_family == AF_INET && + nh->fib_nh_gw4 != onh->fib_nh_gw4) return -1; + + if (nh->fib_nh_gw_family == AF_INET6 && + ipv6_addr_cmp(&nh->fib_nh_gw6, &onh->fib_nh_gw6)) + return -1; + onh++; } endfor_nexthops(fi); return 0; @@ -298,7 +320,7 @@ static inline unsigned int fib_info_hashfn(const struct fib_info *fi) val ^= (__force u32)fi->fib_prefsrc; val ^= fi->fib_priority; for_nexthops(fi) { - val ^= fib_devindex_hashfn(nh->nh_oif); + val ^= fib_devindex_hashfn(nh->fib_nh_oif); } endfor_nexthops(fi) return (val ^ (val >> 7) ^ (val >> 12)) & mask; @@ -347,9 +369,9 @@ int ip_fib_check_default(__be32 gw, struct net_device *dev) hash = fib_devindex_hashfn(dev->ifindex); head = &fib_info_devhash[hash]; hlist_for_each_entry(nh, head, nh_hash) { - if (nh->nh_dev == dev && - nh->nh_gw == gw && - !(nh->nh_flags & RTNH_F_DEAD)) { + if (nh->fib_nh_dev == dev && + nh->fib_nh_gw4 == gw && + !(nh->fib_nh_flags & RTNH_F_DEAD)) { spin_unlock(&fib_info_lock); return 0; } @@ -384,10 +406,10 @@ static inline size_t fib_nlmsg_size(struct fib_info *fi) /* grab encap info */ for_nexthops(fi) { - if (nh->nh_lwtstate) { + if (nh->fib_nh_lws) { /* RTA_ENCAP_TYPE */ nh_encapsize += lwtunnel_get_encap_size( - nh->nh_lwtstate); + nh->fib_nh_lws); /* RTA_ENCAP */ nh_encapsize += nla_total_size(2); } @@ -435,10 +457,18 @@ static int fib_detect_death(struct fib_info *fi, int order, struct fib_info **last_resort, int *last_idx, int dflt) { + const struct fib_nh_common *nhc = fib_info_nhc(fi, 0); struct neighbour *n; int state = NUD_NONE; - n = neigh_lookup(&arp_tbl, &fi->fib_nh[0].nh_gw, fi->fib_dev); + if (likely(nhc->nhc_gw_family == AF_INET)) + n = neigh_lookup(&arp_tbl, &nhc->nhc_gw.ipv4, nhc->nhc_dev); + else if (nhc->nhc_gw_family == AF_INET6) + n = neigh_lookup(ipv6_stub->nd_tbl, &nhc->nhc_gw.ipv6, + nhc->nhc_dev); + else + n = NULL; + if (n) { state = n->nud_state; neigh_release(n); @@ -457,6 +487,75 @@ static int fib_detect_death(struct fib_info *fi, int order, return 1; } +int fib_nh_common_init(struct fib_nh_common *nhc, struct nlattr *encap, + u16 encap_type, void *cfg, gfp_t gfp_flags, + struct netlink_ext_ack *extack) +{ + int err; + + nhc->nhc_pcpu_rth_output = alloc_percpu_gfp(struct rtable __rcu *, + gfp_flags); + if (!nhc->nhc_pcpu_rth_output) + return -ENOMEM; + + if (encap) { + struct lwtunnel_state *lwtstate; + + if (encap_type == LWTUNNEL_ENCAP_NONE) { + NL_SET_ERR_MSG(extack, "LWT encap type not specified"); + err = -EINVAL; + goto lwt_failure; + } + err = lwtunnel_build_state(encap_type, encap, nhc->nhc_family, + cfg, &lwtstate, extack); + if (err) + goto lwt_failure; + + nhc->nhc_lwtstate = lwtstate_get(lwtstate); + } + + return 0; + +lwt_failure: + rt_fibinfo_free_cpus(nhc->nhc_pcpu_rth_output); + nhc->nhc_pcpu_rth_output = NULL; + return err; +} +EXPORT_SYMBOL_GPL(fib_nh_common_init); + +int fib_nh_init(struct net *net, struct fib_nh *nh, + struct fib_config *cfg, int nh_weight, + struct netlink_ext_ack *extack) +{ + int err; + + nh->fib_nh_family = AF_INET; + + err = fib_nh_common_init(&nh->nh_common, cfg->fc_encap, + cfg->fc_encap_type, cfg, GFP_KERNEL, extack); + if (err) + return err; + + nh->fib_nh_oif = cfg->fc_oif; + nh->fib_nh_gw_family = cfg->fc_gw_family; + if (cfg->fc_gw_family == AF_INET) + nh->fib_nh_gw4 = cfg->fc_gw4; + else if (cfg->fc_gw_family == AF_INET6) + nh->fib_nh_gw6 = cfg->fc_gw6; + + nh->fib_nh_flags = cfg->fc_flags; + +#ifdef CONFIG_IP_ROUTE_CLASSID + nh->nh_tclassid = cfg->fc_flow; + if (nh->nh_tclassid) + net->ipv4.fib_num_tclassid_users++; +#endif +#ifdef CONFIG_IP_ROUTE_MULTIPATH + nh->fib_nh_weight = nh_weight; +#endif + return 0; +} + #ifdef CONFIG_IP_ROUTE_MULTIPATH static int fib_count_nexthops(struct rtnexthop *rtnh, int remaining, @@ -483,11 +582,15 @@ static int fib_get_nhs(struct fib_info *fi, struct rtnexthop *rtnh, int remaining, struct fib_config *cfg, struct netlink_ext_ack *extack) { + struct net *net = fi->fib_net; + struct fib_config fib_cfg; int ret; change_nexthops(fi) { int attrlen; + memset(&fib_cfg, 0, sizeof(fib_cfg)); + if (!rtnh_ok(rtnh, remaining)) { NL_SET_ERR_MSG(extack, "Invalid nexthop configuration - extra data after nexthop"); @@ -500,56 +603,73 @@ static int fib_get_nhs(struct fib_info *fi, struct rtnexthop *rtnh, return -EINVAL; } - nexthop_nh->nh_flags = - (cfg->fc_flags & ~0xFF) | rtnh->rtnh_flags; - nexthop_nh->nh_oif = rtnh->rtnh_ifindex; - nexthop_nh->nh_weight = rtnh->rtnh_hops + 1; + fib_cfg.fc_flags = (cfg->fc_flags & ~0xFF) | rtnh->rtnh_flags; + fib_cfg.fc_oif = rtnh->rtnh_ifindex; attrlen = rtnh_attrlen(rtnh); if (attrlen > 0) { - struct nlattr *nla, *attrs = rtnh_attrs(rtnh); + struct nlattr *nla, *nlav, *attrs = rtnh_attrs(rtnh); nla = nla_find(attrs, attrlen, RTA_GATEWAY); - nexthop_nh->nh_gw = nla ? nla_get_in_addr(nla) : 0; -#ifdef CONFIG_IP_ROUTE_CLASSID - nla = nla_find(attrs, attrlen, RTA_FLOW); - nexthop_nh->nh_tclassid = nla ? nla_get_u32(nla) : 0; - if (nexthop_nh->nh_tclassid) - fi->fib_net->ipv4.fib_num_tclassid_users++; -#endif - nla = nla_find(attrs, attrlen, RTA_ENCAP); + nlav = nla_find(attrs, attrlen, RTA_VIA); + if (nla && nlav) { + NL_SET_ERR_MSG(extack, + "Nexthop configuration can not contain both GATEWAY and VIA"); + return -EINVAL; + } if (nla) { - struct lwtunnel_state *lwtstate; - struct nlattr *nla_entype; - - nla_entype = nla_find(attrs, attrlen, - RTA_ENCAP_TYPE); - if (!nla_entype) { - NL_SET_BAD_ATTR(extack, nla); - NL_SET_ERR_MSG(extack, - "Encap type is missing"); - goto err_inval; - } - - ret = lwtunnel_build_state(nla_get_u16( - nla_entype), - nla, AF_INET, cfg, - &lwtstate, extack); + fib_cfg.fc_gw4 = nla_get_in_addr(nla); + if (fib_cfg.fc_gw4) + fib_cfg.fc_gw_family = AF_INET; + } else if (nlav) { + ret = fib_gw_from_via(&fib_cfg, nlav, extack); if (ret) goto errout; - nexthop_nh->nh_lwtstate = - lwtstate_get(lwtstate); } + + nla = nla_find(attrs, attrlen, RTA_FLOW); + if (nla) + fib_cfg.fc_flow = nla_get_u32(nla); + + fib_cfg.fc_encap = nla_find(attrs, attrlen, RTA_ENCAP); + nla = nla_find(attrs, attrlen, RTA_ENCAP_TYPE); + if (nla) + fib_cfg.fc_encap_type = nla_get_u16(nla); } + ret = fib_nh_init(net, nexthop_nh, &fib_cfg, + rtnh->rtnh_hops + 1, extack); + if (ret) + goto errout; + rtnh = rtnh_next(rtnh, &remaining); } endfor_nexthops(fi); - return 0; - -err_inval: ret = -EINVAL; - + if (cfg->fc_oif && fi->fib_nh->fib_nh_oif != cfg->fc_oif) { + NL_SET_ERR_MSG(extack, + "Nexthop device index does not match RTA_OIF"); + goto errout; + } + if (cfg->fc_gw_family) { + if (cfg->fc_gw_family != fi->fib_nh->fib_nh_gw_family || + (cfg->fc_gw_family == AF_INET && + fi->fib_nh->fib_nh_gw4 != cfg->fc_gw4) || + (cfg->fc_gw_family == AF_INET6 && + ipv6_addr_cmp(&fi->fib_nh->fib_nh_gw6, &cfg->fc_gw6))) { + NL_SET_ERR_MSG(extack, + "Nexthop gateway does not match RTA_GATEWAY or RTA_VIA"); + goto errout; + } + } +#ifdef CONFIG_IP_ROUTE_CLASSID + if (cfg->fc_flow && fi->fib_nh->nh_tclassid != cfg->fc_flow) { + NL_SET_ERR_MSG(extack, + "Nexthop class id does not match RTA_FLOW"); + goto errout; + } +#endif + ret = 0; errout: return ret; } @@ -558,49 +678,51 @@ static void fib_rebalance(struct fib_info *fi) { int total; int w; - struct in_device *in_dev; if (fi->fib_nhs < 2) return; total = 0; for_nexthops(fi) { - if (nh->nh_flags & RTNH_F_DEAD) + if (nh->fib_nh_flags & RTNH_F_DEAD) continue; - in_dev = __in_dev_get_rtnl(nh->nh_dev); - - if (in_dev && - IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN(in_dev) && - nh->nh_flags & RTNH_F_LINKDOWN) + if (ip_ignore_linkdown(nh->fib_nh_dev) && + nh->fib_nh_flags & RTNH_F_LINKDOWN) continue; - total += nh->nh_weight; + total += nh->fib_nh_weight; } endfor_nexthops(fi); w = 0; change_nexthops(fi) { int upper_bound; - in_dev = __in_dev_get_rtnl(nexthop_nh->nh_dev); - - if (nexthop_nh->nh_flags & RTNH_F_DEAD) { + if (nexthop_nh->fib_nh_flags & RTNH_F_DEAD) { upper_bound = -1; - } else if (in_dev && - IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN(in_dev) && - nexthop_nh->nh_flags & RTNH_F_LINKDOWN) { + } else if (ip_ignore_linkdown(nexthop_nh->fib_nh_dev) && + nexthop_nh->fib_nh_flags & RTNH_F_LINKDOWN) { upper_bound = -1; } else { - w += nexthop_nh->nh_weight; + w += nexthop_nh->fib_nh_weight; upper_bound = DIV_ROUND_CLOSEST_ULL((u64)w << 31, total) - 1; } - atomic_set(&nexthop_nh->nh_upper_bound, upper_bound); + atomic_set(&nexthop_nh->fib_nh_upper_bound, upper_bound); } endfor_nexthops(fi); } #else /* CONFIG_IP_ROUTE_MULTIPATH */ +static int fib_get_nhs(struct fib_info *fi, struct rtnexthop *rtnh, + int remaining, struct fib_config *cfg, + struct netlink_ext_ack *extack) +{ + NL_SET_ERR_MSG(extack, "Multipath support not enabled in kernel"); + + return -EINVAL; +} + #define fib_rebalance(fi) do { } while (0) #endif /* CONFIG_IP_ROUTE_MULTIPATH */ @@ -620,7 +742,7 @@ static int fib_encap_match(u16 encap_type, ret = lwtunnel_build_state(encap_type, encap, AF_INET, cfg, &lwtstate, extack); if (!ret) { - result = lwtunnel_cmp_encap(lwtstate, nh->nh_lwtstate); + result = lwtunnel_cmp_encap(lwtstate, nh->fib_nh_lws); lwtstate_free(lwtstate); } @@ -638,7 +760,7 @@ int fib_nh_match(struct fib_config *cfg, struct fib_info *fi, if (cfg->fc_priority && cfg->fc_priority != fi->fib_priority) return 1; - if (cfg->fc_oif || cfg->fc_gw) { + if (cfg->fc_oif || cfg->fc_gw_family) { if (cfg->fc_encap) { if (fib_encap_match(cfg->fc_encap_type, cfg->fc_encap, fi->fib_nh, cfg, extack)) @@ -649,10 +771,20 @@ int fib_nh_match(struct fib_config *cfg, struct fib_info *fi, cfg->fc_flow != fi->fib_nh->nh_tclassid) return 1; #endif - if ((!cfg->fc_oif || cfg->fc_oif == fi->fib_nh->nh_oif) && - (!cfg->fc_gw || cfg->fc_gw == fi->fib_nh->nh_gw)) - return 0; - return 1; + if ((cfg->fc_oif && cfg->fc_oif != fi->fib_nh->fib_nh_oif) || + (cfg->fc_gw_family && + cfg->fc_gw_family != fi->fib_nh->fib_nh_gw_family)) + return 1; + + if (cfg->fc_gw_family == AF_INET && + cfg->fc_gw4 != fi->fib_nh->fib_nh_gw4) + return 1; + + if (cfg->fc_gw_family == AF_INET6 && + ipv6_addr_cmp(&cfg->fc_gw6, &fi->fib_nh->fib_nh_gw6)) + return 1; + + return 0; } #ifdef CONFIG_IP_ROUTE_MULTIPATH @@ -668,16 +800,48 @@ int fib_nh_match(struct fib_config *cfg, struct fib_info *fi, if (!rtnh_ok(rtnh, remaining)) return -EINVAL; - if (rtnh->rtnh_ifindex && rtnh->rtnh_ifindex != nh->nh_oif) + if (rtnh->rtnh_ifindex && rtnh->rtnh_ifindex != nh->fib_nh_oif) return 1; attrlen = rtnh_attrlen(rtnh); if (attrlen > 0) { - struct nlattr *nla, *attrs = rtnh_attrs(rtnh); + struct nlattr *nla, *nlav, *attrs = rtnh_attrs(rtnh); nla = nla_find(attrs, attrlen, RTA_GATEWAY); - if (nla && nla_get_in_addr(nla) != nh->nh_gw) - return 1; + nlav = nla_find(attrs, attrlen, RTA_VIA); + if (nla && nlav) { + NL_SET_ERR_MSG(extack, + "Nexthop configuration can not contain both GATEWAY and VIA"); + return -EINVAL; + } + + if (nla) { + if (nh->fib_nh_gw_family != AF_INET || + nla_get_in_addr(nla) != nh->fib_nh_gw4) + return 1; + } else if (nlav) { + struct fib_config cfg2; + int err; + + err = fib_gw_from_via(&cfg2, nlav, extack); + if (err) + return err; + + switch (nh->fib_nh_gw_family) { + case AF_INET: + if (cfg2.fc_gw_family != AF_INET || + cfg2.fc_gw4 != nh->fib_nh_gw4) + return 1; + break; + case AF_INET6: + if (cfg2.fc_gw_family != AF_INET6 || + ipv6_addr_cmp(&cfg2.fc_gw6, + &nh->fib_nh_gw6)) + return 1; + break; + } + } + #ifdef CONFIG_IP_ROUTE_CLASSID nla = nla_find(attrs, attrlen, RTA_FLOW); if (nla && nla_get_u32(nla) != nh->nh_tclassid) @@ -731,6 +895,30 @@ bool fib_metrics_match(struct fib_config *cfg, struct fib_info *fi) return true; } +static int fib_check_nh_v6_gw(struct net *net, struct fib_nh *nh, + u32 table, struct netlink_ext_ack *extack) +{ + struct fib6_config cfg = { + .fc_table = table, + .fc_flags = nh->fib_nh_flags | RTF_GATEWAY, + .fc_ifindex = nh->fib_nh_oif, + .fc_gateway = nh->fib_nh_gw6, + }; + struct fib6_nh fib6_nh = {}; + int err; + + err = ipv6_stub->fib6_nh_init(net, &fib6_nh, &cfg, GFP_KERNEL, extack); + if (!err) { + nh->fib_nh_dev = fib6_nh.fib_nh_dev; + dev_hold(nh->fib_nh_dev); + nh->fib_nh_oif = nh->fib_nh_dev->ifindex; + nh->fib_nh_scope = RT_SCOPE_LINK; + + ipv6_stub->fib6_nh_release(&fib6_nh); + } + + return err; +} /* * Picture @@ -775,133 +963,152 @@ bool fib_metrics_match(struct fib_config *cfg, struct fib_info *fi) * | * |-> {local prefix} (terminal node) */ -static int fib_check_nh(struct fib_config *cfg, struct fib_nh *nh, - struct netlink_ext_ack *extack) +static int fib_check_nh_v4_gw(struct net *net, struct fib_nh *nh, u32 table, + u8 scope, struct netlink_ext_ack *extack) { - int err = 0; - struct net *net; struct net_device *dev; + struct fib_result res; + int err; - net = cfg->fc_nlinfo.nl_net; - if (nh->nh_gw) { - struct fib_result res; - - if (nh->nh_flags & RTNH_F_ONLINK) { - unsigned int addr_type; + if (nh->fib_nh_flags & RTNH_F_ONLINK) { + unsigned int addr_type; - if (cfg->fc_scope >= RT_SCOPE_LINK) { - NL_SET_ERR_MSG(extack, - "Nexthop has invalid scope"); - return -EINVAL; - } - dev = __dev_get_by_index(net, nh->nh_oif); - if (!dev) { - NL_SET_ERR_MSG(extack, "Nexthop device required for onlink"); - return -ENODEV; - } - if (!(dev->flags & IFF_UP)) { - NL_SET_ERR_MSG(extack, - "Nexthop device is not up"); - return -ENETDOWN; - } - addr_type = inet_addr_type_dev_table(net, dev, nh->nh_gw); - if (addr_type != RTN_UNICAST) { - NL_SET_ERR_MSG(extack, - "Nexthop has invalid gateway"); - return -EINVAL; - } - if (!netif_carrier_ok(dev)) - nh->nh_flags |= RTNH_F_LINKDOWN; - nh->nh_dev = dev; - dev_hold(dev); - nh->nh_scope = RT_SCOPE_LINK; - return 0; + if (scope >= RT_SCOPE_LINK) { + NL_SET_ERR_MSG(extack, "Nexthop has invalid scope"); + return -EINVAL; } - rcu_read_lock(); - { - struct fib_table *tbl = NULL; - struct flowi4 fl4 = { - .daddr = nh->nh_gw, - .flowi4_scope = cfg->fc_scope + 1, - .flowi4_oif = nh->nh_oif, - .flowi4_iif = LOOPBACK_IFINDEX, - }; - - /* It is not necessary, but requires a bit of thinking */ - if (fl4.flowi4_scope < RT_SCOPE_LINK) - fl4.flowi4_scope = RT_SCOPE_LINK; - - if (cfg->fc_table) - tbl = fib_get_table(net, cfg->fc_table); - - if (tbl) - err = fib_table_lookup(tbl, &fl4, &res, - FIB_LOOKUP_IGNORE_LINKSTATE | - FIB_LOOKUP_NOREF); - - /* on error or if no table given do full lookup. This - * is needed for example when nexthops are in the local - * table rather than the given table - */ - if (!tbl || err) { - err = fib_lookup(net, &fl4, &res, - FIB_LOOKUP_IGNORE_LINKSTATE); - } - - if (err) { - NL_SET_ERR_MSG(extack, - "Nexthop has invalid gateway"); - rcu_read_unlock(); - return err; - } + dev = __dev_get_by_index(net, nh->fib_nh_oif); + if (!dev) { + NL_SET_ERR_MSG(extack, "Nexthop device required for onlink"); + return -ENODEV; } - err = -EINVAL; - if (res.type != RTN_UNICAST && res.type != RTN_LOCAL) { - NL_SET_ERR_MSG(extack, "Nexthop has invalid gateway"); - goto out; + if (!(dev->flags & IFF_UP)) { + NL_SET_ERR_MSG(extack, "Nexthop device is not up"); + return -ENETDOWN; } - nh->nh_scope = res.scope; - nh->nh_oif = FIB_RES_OIF(res); - nh->nh_dev = dev = FIB_RES_DEV(res); - if (!dev) { - NL_SET_ERR_MSG(extack, - "No egress device for nexthop gateway"); - goto out; + addr_type = inet_addr_type_dev_table(net, dev, nh->fib_nh_gw4); + if (addr_type != RTN_UNICAST) { + NL_SET_ERR_MSG(extack, "Nexthop has invalid gateway"); + return -EINVAL; } - dev_hold(dev); if (!netif_carrier_ok(dev)) - nh->nh_flags |= RTNH_F_LINKDOWN; - err = (dev->flags & IFF_UP) ? 0 : -ENETDOWN; - } else { - struct in_device *in_dev; - - if (nh->nh_flags & (RTNH_F_PERVASIVE | RTNH_F_ONLINK)) { - NL_SET_ERR_MSG(extack, - "Invalid flags for nexthop - PERVASIVE and ONLINK can not be set"); - return -EINVAL; + nh->fib_nh_flags |= RTNH_F_LINKDOWN; + nh->fib_nh_dev = dev; + dev_hold(dev); + nh->fib_nh_scope = RT_SCOPE_LINK; + return 0; + } + rcu_read_lock(); + { + struct fib_table *tbl = NULL; + struct flowi4 fl4 = { + .daddr = nh->fib_nh_gw4, + .flowi4_scope = scope + 1, + .flowi4_oif = nh->fib_nh_oif, + .flowi4_iif = LOOPBACK_IFINDEX, + }; + + /* It is not necessary, but requires a bit of thinking */ + if (fl4.flowi4_scope < RT_SCOPE_LINK) + fl4.flowi4_scope = RT_SCOPE_LINK; + + if (table) + tbl = fib_get_table(net, table); + + if (tbl) + err = fib_table_lookup(tbl, &fl4, &res, + FIB_LOOKUP_IGNORE_LINKSTATE | + FIB_LOOKUP_NOREF); + + /* on error or if no table given do full lookup. This + * is needed for example when nexthops are in the local + * table rather than the given table + */ + if (!tbl || err) { + err = fib_lookup(net, &fl4, &res, + FIB_LOOKUP_IGNORE_LINKSTATE); } - rcu_read_lock(); - err = -ENODEV; - in_dev = inetdev_by_index(net, nh->nh_oif); - if (!in_dev) - goto out; - err = -ENETDOWN; - if (!(in_dev->dev->flags & IFF_UP)) { - NL_SET_ERR_MSG(extack, "Device for nexthop is not up"); + + if (err) { + NL_SET_ERR_MSG(extack, "Nexthop has invalid gateway"); goto out; } - nh->nh_dev = in_dev->dev; - dev_hold(nh->nh_dev); - nh->nh_scope = RT_SCOPE_HOST; - if (!netif_carrier_ok(nh->nh_dev)) - nh->nh_flags |= RTNH_F_LINKDOWN; - err = 0; } + + err = -EINVAL; + if (res.type != RTN_UNICAST && res.type != RTN_LOCAL) { + NL_SET_ERR_MSG(extack, "Nexthop has invalid gateway"); + goto out; + } + nh->fib_nh_scope = res.scope; + nh->fib_nh_oif = FIB_RES_OIF(res); + nh->fib_nh_dev = dev = FIB_RES_DEV(res); + if (!dev) { + NL_SET_ERR_MSG(extack, + "No egress device for nexthop gateway"); + goto out; + } + dev_hold(dev); + if (!netif_carrier_ok(dev)) + nh->fib_nh_flags |= RTNH_F_LINKDOWN; + err = (dev->flags & IFF_UP) ? 0 : -ENETDOWN; out: rcu_read_unlock(); return err; } +static int fib_check_nh_nongw(struct net *net, struct fib_nh *nh, + struct netlink_ext_ack *extack) +{ + struct in_device *in_dev; + int err; + + if (nh->fib_nh_flags & (RTNH_F_PERVASIVE | RTNH_F_ONLINK)) { + NL_SET_ERR_MSG(extack, + "Invalid flags for nexthop - PERVASIVE and ONLINK can not be set"); + return -EINVAL; + } + + rcu_read_lock(); + + err = -ENODEV; + in_dev = inetdev_by_index(net, nh->fib_nh_oif); + if (!in_dev) + goto out; + err = -ENETDOWN; + if (!(in_dev->dev->flags & IFF_UP)) { + NL_SET_ERR_MSG(extack, "Device for nexthop is not up"); + goto out; + } + + nh->fib_nh_dev = in_dev->dev; + dev_hold(nh->fib_nh_dev); + nh->fib_nh_scope = RT_SCOPE_HOST; + if (!netif_carrier_ok(nh->fib_nh_dev)) + nh->fib_nh_flags |= RTNH_F_LINKDOWN; + err = 0; +out: + rcu_read_unlock(); + return err; +} + +static int fib_check_nh(struct fib_config *cfg, struct fib_nh *nh, + struct netlink_ext_ack *extack) +{ + struct net *net = cfg->fc_nlinfo.nl_net; + u32 table = cfg->fc_table; + int err; + + if (nh->fib_nh_gw_family == AF_INET) + err = fib_check_nh_v4_gw(net, nh, table, cfg->fc_scope, extack); + else if (nh->fib_nh_gw_family == AF_INET6) + err = fib_check_nh_v6_gw(net, nh, table, extack); + else + err = fib_check_nh_nongw(net, nh, extack); + + return err; +} + static inline unsigned int fib_laddr_hashfn(__be32 val) { unsigned int mask = (fib_info_hash_size - 1); @@ -986,14 +1193,29 @@ static void fib_info_hash_move(struct hlist_head *new_info_hash, __be32 fib_info_update_nh_saddr(struct net *net, struct fib_nh *nh) { - nh->nh_saddr = inet_select_addr(nh->nh_dev, - nh->nh_gw, + nh->nh_saddr = inet_select_addr(nh->fib_nh_dev, + nh->fib_nh_gw4, nh->nh_parent->fib_scope); nh->nh_saddr_genid = atomic_read(&net->ipv4.dev_addr_genid); return nh->nh_saddr; } +__be32 fib_result_prefsrc(struct net *net, struct fib_result *res) +{ + struct fib_nh_common *nhc = res->nhc; + struct fib_nh *nh; + + if (res->fi->fib_prefsrc) + return res->fi->fib_prefsrc; + + nh = container_of(nhc, struct fib_nh, nh_common); + if (nh->nh_saddr_genid == atomic_read(&net->ipv4.dev_addr_genid)) + return nh->nh_saddr; + + return fib_info_update_nh_saddr(net, nh); +} + static bool fib_valid_prefsrc(struct fib_config *cfg, __be32 fib_prefsrc) { if (cfg->fc_type != RTN_LOCAL || !cfg->fc_dst || @@ -1096,72 +1318,18 @@ struct fib_info *fib_create_info(struct fib_config *cfg, fi->fib_nhs = nhs; change_nexthops(fi) { nexthop_nh->nh_parent = fi; - nexthop_nh->nh_pcpu_rth_output = alloc_percpu(struct rtable __rcu *); - if (!nexthop_nh->nh_pcpu_rth_output) - goto failure; } endfor_nexthops(fi) - if (cfg->fc_mp) { -#ifdef CONFIG_IP_ROUTE_MULTIPATH + if (cfg->fc_mp) err = fib_get_nhs(fi, cfg->fc_mp, cfg->fc_mp_len, cfg, extack); - if (err != 0) - goto failure; - if (cfg->fc_oif && fi->fib_nh->nh_oif != cfg->fc_oif) { - NL_SET_ERR_MSG(extack, - "Nexthop device index does not match RTA_OIF"); - goto err_inval; - } - if (cfg->fc_gw && fi->fib_nh->nh_gw != cfg->fc_gw) { - NL_SET_ERR_MSG(extack, - "Nexthop gateway does not match RTA_GATEWAY"); - goto err_inval; - } -#ifdef CONFIG_IP_ROUTE_CLASSID - if (cfg->fc_flow && fi->fib_nh->nh_tclassid != cfg->fc_flow) { - NL_SET_ERR_MSG(extack, - "Nexthop class id does not match RTA_FLOW"); - goto err_inval; - } -#endif -#else - NL_SET_ERR_MSG(extack, - "Multipath support not enabled in kernel"); - goto err_inval; -#endif - } else { - struct fib_nh *nh = fi->fib_nh; - - if (cfg->fc_encap) { - struct lwtunnel_state *lwtstate; - - if (cfg->fc_encap_type == LWTUNNEL_ENCAP_NONE) { - NL_SET_ERR_MSG(extack, - "LWT encap type not specified"); - goto err_inval; - } - err = lwtunnel_build_state(cfg->fc_encap_type, - cfg->fc_encap, AF_INET, cfg, - &lwtstate, extack); - if (err) - goto failure; + else + err = fib_nh_init(net, fi->fib_nh, cfg, 1, extack); - nh->nh_lwtstate = lwtstate_get(lwtstate); - } - nh->nh_oif = cfg->fc_oif; - nh->nh_gw = cfg->fc_gw; - nh->nh_flags = cfg->fc_flags; -#ifdef CONFIG_IP_ROUTE_CLASSID - nh->nh_tclassid = cfg->fc_flow; - if (nh->nh_tclassid) - fi->fib_net->ipv4.fib_num_tclassid_users++; -#endif -#ifdef CONFIG_IP_ROUTE_MULTIPATH - nh->nh_weight = 1; -#endif - } + if (err != 0) + goto failure; if (fib_props[cfg->fc_type].error) { - if (cfg->fc_gw || cfg->fc_oif || cfg->fc_mp) { + if (cfg->fc_gw_family || cfg->fc_oif || cfg->fc_mp) { NL_SET_ERR_MSG(extack, "Gateway, device and multipath can not be specified for this route type"); goto err_inval; @@ -1195,15 +1363,15 @@ struct fib_info *fib_create_info(struct fib_config *cfg, "Route with host scope can not have multiple nexthops"); goto err_inval; } - if (nh->nh_gw) { + if (nh->fib_nh_gw_family) { NL_SET_ERR_MSG(extack, "Route with host scope can not have a gateway"); goto err_inval; } - nh->nh_scope = RT_SCOPE_NOWHERE; - nh->nh_dev = dev_get_by_index(net, fi->fib_nh->nh_oif); + nh->fib_nh_scope = RT_SCOPE_NOWHERE; + nh->fib_nh_dev = dev_get_by_index(net, fi->fib_nh->fib_nh_oif); err = -ENODEV; - if (!nh->nh_dev) + if (!nh->fib_nh_dev) goto failure; } else { int linkdown = 0; @@ -1212,7 +1380,7 @@ struct fib_info *fib_create_info(struct fib_config *cfg, err = fib_check_nh(cfg, nexthop_nh, extack); if (err != 0) goto failure; - if (nexthop_nh->nh_flags & RTNH_F_LINKDOWN) + if (nexthop_nh->fib_nh_flags & RTNH_F_LINKDOWN) linkdown++; } endfor_nexthops(fi) if (linkdown == fi->fib_nhs) @@ -1226,6 +1394,8 @@ struct fib_info *fib_create_info(struct fib_config *cfg, change_nexthops(fi) { fib_info_update_nh_saddr(net, nexthop_nh); + if (nexthop_nh->fib_nh_gw_family == AF_INET6) + fi->fib_nh_is_v6 = true; } endfor_nexthops(fi) fib_rebalance(fi); @@ -1254,9 +1424,9 @@ link_it: struct hlist_head *head; unsigned int hash; - if (!nexthop_nh->nh_dev) + if (!nexthop_nh->fib_nh_dev) continue; - hash = fib_devindex_hashfn(nexthop_nh->nh_dev->ifindex); + hash = fib_devindex_hashfn(nexthop_nh->fib_nh_dev->ifindex); head = &fib_info_devhash[hash]; hlist_add_head(&nexthop_nh->nh_hash, head); } endfor_nexthops(fi) @@ -1275,6 +1445,141 @@ failure: return ERR_PTR(err); } +int fib_nexthop_info(struct sk_buff *skb, const struct fib_nh_common *nhc, + unsigned char *flags, bool skip_oif) +{ + if (nhc->nhc_flags & RTNH_F_DEAD) + *flags |= RTNH_F_DEAD; + + if (nhc->nhc_flags & RTNH_F_LINKDOWN) { + *flags |= RTNH_F_LINKDOWN; + + rcu_read_lock(); + switch (nhc->nhc_family) { + case AF_INET: + if (ip_ignore_linkdown(nhc->nhc_dev)) + *flags |= RTNH_F_DEAD; + break; + case AF_INET6: + if (ip6_ignore_linkdown(nhc->nhc_dev)) + *flags |= RTNH_F_DEAD; + break; + } + rcu_read_unlock(); + } + + switch (nhc->nhc_gw_family) { + case AF_INET: + if (nla_put_in_addr(skb, RTA_GATEWAY, nhc->nhc_gw.ipv4)) + goto nla_put_failure; + break; + case AF_INET6: + /* if gateway family does not match nexthop family + * gateway is encoded as RTA_VIA + */ + if (nhc->nhc_gw_family != nhc->nhc_family) { + int alen = sizeof(struct in6_addr); + struct nlattr *nla; + struct rtvia *via; + + nla = nla_reserve(skb, RTA_VIA, alen + 2); + if (!nla) + goto nla_put_failure; + + via = nla_data(nla); + via->rtvia_family = AF_INET6; + memcpy(via->rtvia_addr, &nhc->nhc_gw.ipv6, alen); + } else if (nla_put_in6_addr(skb, RTA_GATEWAY, + &nhc->nhc_gw.ipv6) < 0) { + goto nla_put_failure; + } + break; + } + + *flags |= (nhc->nhc_flags & RTNH_F_ONLINK); + if (nhc->nhc_flags & RTNH_F_OFFLOAD) + *flags |= RTNH_F_OFFLOAD; + + if (!skip_oif && nhc->nhc_dev && + nla_put_u32(skb, RTA_OIF, nhc->nhc_dev->ifindex)) + goto nla_put_failure; + + if (nhc->nhc_lwtstate && + lwtunnel_fill_encap(skb, nhc->nhc_lwtstate, + RTA_ENCAP, RTA_ENCAP_TYPE) < 0) + goto nla_put_failure; + + return 0; + +nla_put_failure: + return -EMSGSIZE; +} +EXPORT_SYMBOL_GPL(fib_nexthop_info); + +#if IS_ENABLED(CONFIG_IP_ROUTE_MULTIPATH) || IS_ENABLED(CONFIG_IPV6) +int fib_add_nexthop(struct sk_buff *skb, const struct fib_nh_common *nhc, + int nh_weight) +{ + const struct net_device *dev = nhc->nhc_dev; + struct rtnexthop *rtnh; + unsigned char flags = 0; + + rtnh = nla_reserve_nohdr(skb, sizeof(*rtnh)); + if (!rtnh) + goto nla_put_failure; + + rtnh->rtnh_hops = nh_weight - 1; + rtnh->rtnh_ifindex = dev ? dev->ifindex : 0; + + if (fib_nexthop_info(skb, nhc, &flags, true) < 0) + goto nla_put_failure; + + rtnh->rtnh_flags = flags; + + /* length of rtnetlink header + attributes */ + rtnh->rtnh_len = nlmsg_get_pos(skb) - (void *)rtnh; + + return 0; + +nla_put_failure: + return -EMSGSIZE; +} +EXPORT_SYMBOL_GPL(fib_add_nexthop); +#endif + +#ifdef CONFIG_IP_ROUTE_MULTIPATH +static int fib_add_multipath(struct sk_buff *skb, struct fib_info *fi) +{ + struct nlattr *mp; + + mp = nla_nest_start_noflag(skb, RTA_MULTIPATH); + if (!mp) + goto nla_put_failure; + + for_nexthops(fi) { + if (fib_add_nexthop(skb, &nh->nh_common, nh->fib_nh_weight) < 0) + goto nla_put_failure; +#ifdef CONFIG_IP_ROUTE_CLASSID + if (nh->nh_tclassid && + nla_put_u32(skb, RTA_FLOW, nh->nh_tclassid)) + goto nla_put_failure; +#endif + } endfor_nexthops(fi); + + nla_nest_end(skb, mp); + + return 0; + +nla_put_failure: + return -EMSGSIZE; +} +#else +static int fib_add_multipath(struct sk_buff *skb, struct fib_info *fi) +{ + return 0; +} +#endif + int fib_dump_info(struct sk_buff *skb, u32 portid, u32 seq, int event, u32 tb_id, u8 type, __be32 dst, int dst_len, u8 tos, struct fib_info *fi, unsigned int flags) @@ -1315,80 +1620,23 @@ int fib_dump_info(struct sk_buff *skb, u32 portid, u32 seq, int event, nla_put_in_addr(skb, RTA_PREFSRC, fi->fib_prefsrc)) goto nla_put_failure; if (fi->fib_nhs == 1) { - if (fi->fib_nh->nh_gw && - nla_put_in_addr(skb, RTA_GATEWAY, fi->fib_nh->nh_gw)) - goto nla_put_failure; - if (fi->fib_nh->nh_oif && - nla_put_u32(skb, RTA_OIF, fi->fib_nh->nh_oif)) + struct fib_nh *nh = &fi->fib_nh[0]; + unsigned char flags = 0; + + if (fib_nexthop_info(skb, &nh->nh_common, &flags, false) < 0) goto nla_put_failure; - if (fi->fib_nh->nh_flags & RTNH_F_LINKDOWN) { - struct in_device *in_dev; - - rcu_read_lock(); - in_dev = __in_dev_get_rcu(fi->fib_nh->nh_dev); - if (in_dev && - IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN(in_dev)) - rtm->rtm_flags |= RTNH_F_DEAD; - rcu_read_unlock(); - } - if (fi->fib_nh->nh_flags & RTNH_F_OFFLOAD) - rtm->rtm_flags |= RTNH_F_OFFLOAD; + + rtm->rtm_flags = flags; #ifdef CONFIG_IP_ROUTE_CLASSID - if (fi->fib_nh[0].nh_tclassid && - nla_put_u32(skb, RTA_FLOW, fi->fib_nh[0].nh_tclassid)) + if (nh->nh_tclassid && + nla_put_u32(skb, RTA_FLOW, nh->nh_tclassid)) goto nla_put_failure; #endif - if (fi->fib_nh->nh_lwtstate && - lwtunnel_fill_encap(skb, fi->fib_nh->nh_lwtstate) < 0) + } else { + if (fib_add_multipath(skb, fi) < 0) goto nla_put_failure; } -#ifdef CONFIG_IP_ROUTE_MULTIPATH - if (fi->fib_nhs > 1) { - struct rtnexthop *rtnh; - struct nlattr *mp; - mp = nla_nest_start(skb, RTA_MULTIPATH); - if (!mp) - goto nla_put_failure; - - for_nexthops(fi) { - rtnh = nla_reserve_nohdr(skb, sizeof(*rtnh)); - if (!rtnh) - goto nla_put_failure; - - rtnh->rtnh_flags = nh->nh_flags & 0xFF; - if (nh->nh_flags & RTNH_F_LINKDOWN) { - struct in_device *in_dev; - - rcu_read_lock(); - in_dev = __in_dev_get_rcu(nh->nh_dev); - if (in_dev && - IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN(in_dev)) - rtnh->rtnh_flags |= RTNH_F_DEAD; - rcu_read_unlock(); - } - rtnh->rtnh_hops = nh->nh_weight - 1; - rtnh->rtnh_ifindex = nh->nh_oif; - - if (nh->nh_gw && - nla_put_in_addr(skb, RTA_GATEWAY, nh->nh_gw)) - goto nla_put_failure; -#ifdef CONFIG_IP_ROUTE_CLASSID - if (nh->nh_tclassid && - nla_put_u32(skb, RTA_FLOW, nh->nh_tclassid)) - goto nla_put_failure; -#endif - if (nh->nh_lwtstate && - lwtunnel_fill_encap(skb, nh->nh_lwtstate) < 0) - goto nla_put_failure; - - /* length of rtnetlink header + attributes */ - rtnh->rtnh_len = nlmsg_get_pos(skb) - (void *) rtnh; - } endfor_nexthops(fi); - - nla_nest_end(skb, mp); - } -#endif nlmsg_end(skb, nlh); return 0; @@ -1427,28 +1675,26 @@ int fib_sync_down_addr(struct net_device *dev, __be32 local) return ret; } -static int call_fib_nh_notifiers(struct fib_nh *fib_nh, +static int call_fib_nh_notifiers(struct fib_nh *nh, enum fib_event_type event_type) { - struct in_device *in_dev = __in_dev_get_rtnl(fib_nh->nh_dev); + bool ignore_link_down = ip_ignore_linkdown(nh->fib_nh_dev); struct fib_nh_notifier_info info = { - .fib_nh = fib_nh, + .fib_nh = nh, }; switch (event_type) { case FIB_EVENT_NH_ADD: - if (fib_nh->nh_flags & RTNH_F_DEAD) + if (nh->fib_nh_flags & RTNH_F_DEAD) break; - if (IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN(in_dev) && - fib_nh->nh_flags & RTNH_F_LINKDOWN) + if (ignore_link_down && nh->fib_nh_flags & RTNH_F_LINKDOWN) break; - return call_fib4_notifiers(dev_net(fib_nh->nh_dev), event_type, + return call_fib4_notifiers(dev_net(nh->fib_nh_dev), event_type, &info.info); case FIB_EVENT_NH_DEL: - if ((in_dev && IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN(in_dev) && - fib_nh->nh_flags & RTNH_F_LINKDOWN) || - (fib_nh->nh_flags & RTNH_F_DEAD)) - return call_fib4_notifiers(dev_net(fib_nh->nh_dev), + if ((ignore_link_down && nh->fib_nh_flags & RTNH_F_LINKDOWN) || + (nh->fib_nh_flags & RTNH_F_DEAD)) + return call_fib4_notifiers(dev_net(nh->fib_nh_dev), event_type, &info.info); default: break; @@ -1467,12 +1713,12 @@ static int call_fib_nh_notifiers(struct fib_nh *fib_nh, * - if the new MTU is greater than the PMTU, don't make any change * - otherwise, unlock and set PMTU */ -static void nh_update_mtu(struct fib_nh *nh, u32 new, u32 orig) +static void nh_update_mtu(struct fib_nh_common *nhc, u32 new, u32 orig) { struct fnhe_hash_bucket *bucket; int i; - bucket = rcu_dereference_protected(nh->nh_exceptions, 1); + bucket = rcu_dereference_protected(nhc->nhc_exceptions, 1); if (!bucket) return; @@ -1502,8 +1748,8 @@ void fib_sync_mtu(struct net_device *dev, u32 orig_mtu) struct fib_nh *nh; hlist_for_each_entry(nh, head, nh_hash) { - if (nh->nh_dev == dev) - nh_update_mtu(nh, dev->mtu, orig_mtu); + if (nh->fib_nh_dev == dev) + nh_update_mtu(&nh->nh_common, dev->mtu, orig_mtu); } } @@ -1530,22 +1776,22 @@ int fib_sync_down_dev(struct net_device *dev, unsigned long event, bool force) int dead; BUG_ON(!fi->fib_nhs); - if (nh->nh_dev != dev || fi == prev_fi) + if (nh->fib_nh_dev != dev || fi == prev_fi) continue; prev_fi = fi; dead = 0; change_nexthops(fi) { - if (nexthop_nh->nh_flags & RTNH_F_DEAD) + if (nexthop_nh->fib_nh_flags & RTNH_F_DEAD) dead++; - else if (nexthop_nh->nh_dev == dev && - nexthop_nh->nh_scope != scope) { + else if (nexthop_nh->fib_nh_dev == dev && + nexthop_nh->fib_nh_scope != scope) { switch (event) { case NETDEV_DOWN: case NETDEV_UNREGISTER: - nexthop_nh->nh_flags |= RTNH_F_DEAD; + nexthop_nh->fib_nh_flags |= RTNH_F_DEAD; /* fall through */ case NETDEV_CHANGE: - nexthop_nh->nh_flags |= RTNH_F_LINKDOWN; + nexthop_nh->fib_nh_flags |= RTNH_F_LINKDOWN; break; } call_fib_nh_notifiers(nexthop_nh, @@ -1554,7 +1800,7 @@ int fib_sync_down_dev(struct net_device *dev, unsigned long event, bool force) } #ifdef CONFIG_IP_ROUTE_MULTIPATH if (event == NETDEV_UNREGISTER && - nexthop_nh->nh_dev == dev) { + nexthop_nh->fib_nh_dev == dev) { dead = fi->fib_nhs; break; } @@ -1614,8 +1860,8 @@ static void fib_select_default(const struct flowi4 *flp, struct fib_result *res) if (next_fi->fib_scope != res->scope || fa->fa_type != RTN_UNICAST) continue; - if (!next_fi->fib_nh[0].nh_gw || - next_fi->fib_nh[0].nh_scope != RT_SCOPE_LINK) + if (!next_fi->fib_nh[0].fib_nh_gw4 || + next_fi->fib_nh[0].fib_nh_scope != RT_SCOPE_LINK) continue; fib_alias_accessed(fa); @@ -1658,7 +1904,7 @@ out: * Dead device goes up. We wake up dead nexthops. * It takes sense only on multipath routes. */ -int fib_sync_up(struct net_device *dev, unsigned int nh_flags) +int fib_sync_up(struct net_device *dev, unsigned char nh_flags) { struct fib_info *prev_fi; unsigned int hash; @@ -1686,24 +1932,24 @@ int fib_sync_up(struct net_device *dev, unsigned int nh_flags) int alive; BUG_ON(!fi->fib_nhs); - if (nh->nh_dev != dev || fi == prev_fi) + if (nh->fib_nh_dev != dev || fi == prev_fi) continue; prev_fi = fi; alive = 0; change_nexthops(fi) { - if (!(nexthop_nh->nh_flags & nh_flags)) { + if (!(nexthop_nh->fib_nh_flags & nh_flags)) { alive++; continue; } - if (!nexthop_nh->nh_dev || - !(nexthop_nh->nh_dev->flags & IFF_UP)) + if (!nexthop_nh->fib_nh_dev || + !(nexthop_nh->fib_nh_dev->flags & IFF_UP)) continue; - if (nexthop_nh->nh_dev != dev || + if (nexthop_nh->fib_nh_dev != dev || !__in_dev_get_rtnl(dev)) continue; alive++; - nexthop_nh->nh_flags &= ~nh_flags; + nexthop_nh->fib_nh_flags &= ~nh_flags; call_fib_nh_notifiers(nexthop_nh, FIB_EVENT_NH_ADD); } endfor_nexthops(fi) @@ -1723,13 +1969,19 @@ static bool fib_good_nh(const struct fib_nh *nh) { int state = NUD_REACHABLE; - if (nh->nh_scope == RT_SCOPE_LINK) { + if (nh->fib_nh_scope == RT_SCOPE_LINK) { struct neighbour *n; rcu_read_lock_bh(); - n = __ipv4_neigh_lookup_noref(nh->nh_dev, - (__force u32)nh->nh_gw); + if (likely(nh->fib_nh_gw_family == AF_INET)) + n = __ipv4_neigh_lookup_noref(nh->fib_nh_dev, + (__force u32)nh->fib_nh_gw4); + else if (nh->fib_nh_gw_family == AF_INET6) + n = __ipv6_neigh_lookup_noref_stub(nh->fib_nh_dev, + &nh->fib_nh_gw6); + else + n = NULL; if (n) state = n->nud_state; @@ -1745,20 +1997,22 @@ void fib_select_multipath(struct fib_result *res, int hash) struct net *net = fi->fib_net; bool first = false; - for_nexthops(fi) { + change_nexthops(fi) { if (net->ipv4.sysctl_fib_multipath_use_neigh) { - if (!fib_good_nh(nh)) + if (!fib_good_nh(nexthop_nh)) continue; if (!first) { res->nh_sel = nhsel; + res->nhc = &nexthop_nh->nh_common; first = true; } } - if (hash > atomic_read(&nh->nh_upper_bound)) + if (hash > atomic_read(&nexthop_nh->fib_nh_upper_bound)) continue; res->nh_sel = nhsel; + res->nhc = &nexthop_nh->nh_common; return; } endfor_nexthops(fi); } @@ -1785,5 +2039,5 @@ void fib_select_path(struct net *net, struct fib_result *res, check_saddr: if (!fl4->saddr) - fl4->saddr = FIB_RES_PREFSRC(net, *res); + fl4->saddr = fib_result_prefsrc(net, res); } diff --git a/net/ipv4/fib_trie.c b/net/ipv4/fib_trie.c index a573e37e0615b1b1b5093ae3ada97f07c44f27cb..334f723bdf802d2b169fc99631e7af2be7e90fb5 100644 --- a/net/ipv4/fib_trie.c +++ b/net/ipv4/fib_trie.c @@ -183,14 +183,16 @@ struct trie { }; static struct key_vector *resize(struct trie *t, struct key_vector *tn); -static size_t tnode_free_size; +static unsigned int tnode_free_size; /* - * synchronize_rcu after call_rcu for that many pages; it should be especially - * useful before resizing the root node with PREEMPT_NONE configs; the value was - * obtained experimentally, aiming to avoid visible slowdown. + * synchronize_rcu after call_rcu for outstanding dirty memory; it should be + * especially useful before resizing the root node with PREEMPT_NONE configs; + * the value was obtained experimentally, aiming to avoid visible slowdown. */ -static const int sync_pages = 128; +unsigned int sysctl_fib_sync_mem = 512 * 1024; +unsigned int sysctl_fib_sync_mem_min = 64 * 1024; +unsigned int sysctl_fib_sync_mem_max = 64 * 1024 * 1024; static struct kmem_cache *fn_alias_kmem __ro_after_init; static struct kmem_cache *trie_leaf_kmem __ro_after_init; @@ -504,7 +506,7 @@ static void tnode_free(struct key_vector *tn) tn = container_of(head, struct tnode, rcu)->kv; } - if (tnode_free_size >= PAGE_SIZE * sync_pages) { + if (tnode_free_size >= sysctl_fib_sync_mem) { tnode_free_size = 0; synchronize_rcu(); } @@ -1468,19 +1470,17 @@ found: if (fi->fib_flags & RTNH_F_DEAD) continue; for (nhsel = 0; nhsel < fi->fib_nhs; nhsel++) { - const struct fib_nh *nh = &fi->fib_nh[nhsel]; - struct in_device *in_dev = __in_dev_get_rcu(nh->nh_dev); + struct fib_nh_common *nhc = fib_info_nhc(fi, nhsel); - if (nh->nh_flags & RTNH_F_DEAD) + if (nhc->nhc_flags & RTNH_F_DEAD) continue; - if (in_dev && - IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN(in_dev) && - nh->nh_flags & RTNH_F_LINKDOWN && + if (ip_ignore_linkdown(nhc->nhc_dev) && + nhc->nhc_flags & RTNH_F_LINKDOWN && !(fib_flags & FIB_LOOKUP_IGNORE_LINKSTATE)) continue; if (!(flp->flowi4_flags & FLOWI_FLAG_SKIP_NH_OIF)) { if (flp->flowi4_oif && - flp->flowi4_oif != nh->nh_oif) + flp->flowi4_oif != nhc->nhc_oif) continue; } @@ -1490,6 +1490,7 @@ found: res->prefix = htonl(n->key); res->prefixlen = KEYLENGTH - fa->fa_slen; res->nh_sel = nhsel; + res->nhc = nhc; res->type = fa->fa_type; res->scope = fi->fib_scope; res->fi = fi; @@ -1498,7 +1499,7 @@ found: #ifdef CONFIG_IP_FIB_TRIE_STATS this_cpu_inc(stats->semantic_match_passed); #endif - trace_fib_table_lookup(tb->tb_id, flp, nh, err); + trace_fib_table_lookup(tb->tb_id, flp, nhc, err); return err; } @@ -2651,7 +2652,7 @@ static unsigned int fib_flag_trans(int type, __be32 mask, const struct fib_info if (type == RTN_UNREACHABLE || type == RTN_PROHIBIT) flags = RTF_REJECT; - if (fi && fi->fib_nh->nh_gw) + if (fi && fi->fib_nh->fib_nh_gw4) flags |= RTF_GATEWAY; if (mask == htonl(0xFFFFFFFF)) flags |= RTF_HOST; @@ -2702,7 +2703,7 @@ static int fib_route_seq_show(struct seq_file *seq, void *v) "%d\t%08X\t%d\t%u\t%u", fi->fib_dev ? fi->fib_dev->name : "*", prefix, - fi->fib_nh->nh_gw, flags, 0, 0, + fi->fib_nh->fib_nh_gw4, flags, 0, 0, fi->fib_priority, mask, (fi->fib_advmss ? diff --git a/net/ipv4/fou.c b/net/ipv4/fou.c index 12ce6c526d72bd15a16a9415ed992c25039d415a..ca95051317ed62d699c01a4764c5dc915e2699a3 100644 --- a/net/ipv4/fou.c +++ b/net/ipv4/fou.c @@ -137,7 +137,7 @@ static int gue_udp_recv(struct sock *sk, struct sk_buff *skb) break; case 1: { - /* Direct encasulation of IPv4 or IPv6 */ + /* Direct encapsulation of IPv4 or IPv6 */ int prot; @@ -171,9 +171,7 @@ static int gue_udp_recv(struct sock *sk, struct sk_buff *skb) /* guehdr may change after pull */ guehdr = (struct guehdr *)&udp_hdr(skb)[1]; - hdrlen = sizeof(struct guehdr) + optlen; - - if (guehdr->version != 0 || validate_gue_flags(guehdr, optlen)) + if (validate_gue_flags(guehdr, optlen)) goto drop; hdrlen = sizeof(struct guehdr) + optlen; @@ -501,15 +499,45 @@ out_unlock: return err; } -static int fou_add_to_port_list(struct net *net, struct fou *fou) +static bool fou_cfg_cmp(struct fou *fou, struct fou_cfg *cfg) +{ + struct sock *sk = fou->sock->sk; + struct udp_port_cfg *udp_cfg = &cfg->udp_config; + + if (fou->family != udp_cfg->family || + fou->port != udp_cfg->local_udp_port || + sk->sk_dport != udp_cfg->peer_udp_port || + sk->sk_bound_dev_if != udp_cfg->bind_ifindex) + return false; + + if (fou->family == AF_INET) { + if (sk->sk_rcv_saddr != udp_cfg->local_ip.s_addr || + sk->sk_daddr != udp_cfg->peer_ip.s_addr) + return false; + else + return true; +#if IS_ENABLED(CONFIG_IPV6) + } else { + if (ipv6_addr_cmp(&sk->sk_v6_rcv_saddr, &udp_cfg->local_ip6) || + ipv6_addr_cmp(&sk->sk_v6_daddr, &udp_cfg->peer_ip6)) + return false; + else + return true; +#endif + } + + return false; +} + +static int fou_add_to_port_list(struct net *net, struct fou *fou, + struct fou_cfg *cfg) { struct fou_net *fn = net_generic(net, fou_net_id); struct fou *fout; mutex_lock(&fn->fou_lock); list_for_each_entry(fout, &fn->fou_list, list) { - if (fou->port == fout->port && - fou->family == fout->family) { + if (fou_cfg_cmp(fout, cfg)) { mutex_unlock(&fn->fou_lock); return -EALREADY; } @@ -587,7 +615,7 @@ static int fou_create(struct net *net, struct fou_cfg *cfg, sk->sk_allocation = GFP_ATOMIC; - err = fou_add_to_port_list(net, fou); + err = fou_add_to_port_list(net, fou, cfg); if (err) goto error; @@ -607,14 +635,12 @@ error: static int fou_destroy(struct net *net, struct fou_cfg *cfg) { struct fou_net *fn = net_generic(net, fou_net_id); - __be16 port = cfg->udp_config.local_udp_port; - u8 family = cfg->udp_config.family; int err = -EINVAL; struct fou *fou; mutex_lock(&fn->fou_lock); list_for_each_entry(fou, &fn->fou_list, list) { - if (fou->port == port && fou->family == family) { + if (fou_cfg_cmp(fou, cfg)) { fou_release(fou); err = 0; break; @@ -628,16 +654,27 @@ static int fou_destroy(struct net *net, struct fou_cfg *cfg) static struct genl_family fou_nl_family; static const struct nla_policy fou_nl_policy[FOU_ATTR_MAX + 1] = { - [FOU_ATTR_PORT] = { .type = NLA_U16, }, - [FOU_ATTR_AF] = { .type = NLA_U8, }, - [FOU_ATTR_IPPROTO] = { .type = NLA_U8, }, - [FOU_ATTR_TYPE] = { .type = NLA_U8, }, - [FOU_ATTR_REMCSUM_NOPARTIAL] = { .type = NLA_FLAG, }, + [FOU_ATTR_PORT] = { .type = NLA_U16, }, + [FOU_ATTR_AF] = { .type = NLA_U8, }, + [FOU_ATTR_IPPROTO] = { .type = NLA_U8, }, + [FOU_ATTR_TYPE] = { .type = NLA_U8, }, + [FOU_ATTR_REMCSUM_NOPARTIAL] = { .type = NLA_FLAG, }, + [FOU_ATTR_LOCAL_V4] = { .type = NLA_U32, }, + [FOU_ATTR_PEER_V4] = { .type = NLA_U32, }, + [FOU_ATTR_LOCAL_V6] = { .type = sizeof(struct in6_addr), }, + [FOU_ATTR_PEER_V6] = { .type = sizeof(struct in6_addr), }, + [FOU_ATTR_PEER_PORT] = { .type = NLA_U16, }, + [FOU_ATTR_IFINDEX] = { .type = NLA_S32, }, }; static int parse_nl_config(struct genl_info *info, struct fou_cfg *cfg) { + bool has_local = false, has_peer = false; + struct nlattr *attr; + int ifindex; + __be16 port; + memset(cfg, 0, sizeof(*cfg)); cfg->udp_config.family = AF_INET; @@ -659,8 +696,7 @@ static int parse_nl_config(struct genl_info *info, } if (info->attrs[FOU_ATTR_PORT]) { - __be16 port = nla_get_be16(info->attrs[FOU_ATTR_PORT]); - + port = nla_get_be16(info->attrs[FOU_ATTR_PORT]); cfg->udp_config.local_udp_port = port; } @@ -673,6 +709,52 @@ static int parse_nl_config(struct genl_info *info, if (info->attrs[FOU_ATTR_REMCSUM_NOPARTIAL]) cfg->flags |= FOU_F_REMCSUM_NOPARTIAL; + if (cfg->udp_config.family == AF_INET) { + if (info->attrs[FOU_ATTR_LOCAL_V4]) { + attr = info->attrs[FOU_ATTR_LOCAL_V4]; + cfg->udp_config.local_ip.s_addr = nla_get_in_addr(attr); + has_local = true; + } + + if (info->attrs[FOU_ATTR_PEER_V4]) { + attr = info->attrs[FOU_ATTR_PEER_V4]; + cfg->udp_config.peer_ip.s_addr = nla_get_in_addr(attr); + has_peer = true; + } +#if IS_ENABLED(CONFIG_IPV6) + } else { + if (info->attrs[FOU_ATTR_LOCAL_V6]) { + attr = info->attrs[FOU_ATTR_LOCAL_V6]; + cfg->udp_config.local_ip6 = nla_get_in6_addr(attr); + has_local = true; + } + + if (info->attrs[FOU_ATTR_PEER_V6]) { + attr = info->attrs[FOU_ATTR_PEER_V6]; + cfg->udp_config.peer_ip6 = nla_get_in6_addr(attr); + has_peer = true; + } +#endif + } + + if (has_peer) { + if (info->attrs[FOU_ATTR_PEER_PORT]) { + port = nla_get_be16(info->attrs[FOU_ATTR_PEER_PORT]); + cfg->udp_config.peer_udp_port = port; + } else { + return -EINVAL; + } + } + + if (info->attrs[FOU_ATTR_IFINDEX]) { + if (!has_local) + return -EINVAL; + + ifindex = nla_get_s32(info->attrs[FOU_ATTR_IFINDEX]); + + cfg->udp_config.bind_ifindex = ifindex; + } + return 0; } @@ -704,15 +786,37 @@ static int fou_nl_cmd_rm_port(struct sk_buff *skb, struct genl_info *info) static int fou_fill_info(struct fou *fou, struct sk_buff *msg) { + struct sock *sk = fou->sock->sk; + if (nla_put_u8(msg, FOU_ATTR_AF, fou->sock->sk->sk_family) || nla_put_be16(msg, FOU_ATTR_PORT, fou->port) || + nla_put_be16(msg, FOU_ATTR_PEER_PORT, sk->sk_dport) || nla_put_u8(msg, FOU_ATTR_IPPROTO, fou->protocol) || - nla_put_u8(msg, FOU_ATTR_TYPE, fou->type)) + nla_put_u8(msg, FOU_ATTR_TYPE, fou->type) || + nla_put_s32(msg, FOU_ATTR_IFINDEX, sk->sk_bound_dev_if)) return -1; if (fou->flags & FOU_F_REMCSUM_NOPARTIAL) if (nla_put_flag(msg, FOU_ATTR_REMCSUM_NOPARTIAL)) return -1; + + if (fou->sock->sk->sk_family == AF_INET) { + if (nla_put_in_addr(msg, FOU_ATTR_LOCAL_V4, sk->sk_rcv_saddr)) + return -1; + + if (nla_put_in_addr(msg, FOU_ATTR_PEER_V4, sk->sk_daddr)) + return -1; +#if IS_ENABLED(CONFIG_IPV6) + } else { + if (nla_put_in6_addr(msg, FOU_ATTR_LOCAL_V6, + &sk->sk_v6_rcv_saddr)) + return -1; + + if (nla_put_in6_addr(msg, FOU_ATTR_PEER_V6, &sk->sk_v6_daddr)) + return -1; +#endif + } + return 0; } @@ -765,7 +869,7 @@ static int fou_nl_cmd_get_port(struct sk_buff *skb, struct genl_info *info) ret = -ESRCH; mutex_lock(&fn->fou_lock); list_for_each_entry(fout, &fn->fou_list, list) { - if (port == fout->port && family == fout->family) { + if (fou_cfg_cmp(fout, &cfg)) { ret = fou_dump_info(fout, info->snd_portid, info->snd_seq, 0, msg, info->genlhdr->cmd); @@ -809,21 +913,21 @@ static int fou_nl_dump(struct sk_buff *skb, struct netlink_callback *cb) static const struct genl_ops fou_nl_ops[] = { { .cmd = FOU_CMD_ADD, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = fou_nl_cmd_add_port, - .policy = fou_nl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = FOU_CMD_DEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = fou_nl_cmd_rm_port, - .policy = fou_nl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = FOU_CMD_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = fou_nl_cmd_get_port, .dumpit = fou_nl_dump, - .policy = fou_nl_policy, }, }; @@ -832,6 +936,7 @@ static struct genl_family fou_nl_family __ro_after_init = { .name = FOU_GENL_NAME, .version = FOU_GENL_VERSION, .maxattr = FOU_ATTR_MAX, + .policy = fou_nl_policy, .netnsok = true, .module = THIS_MODULE, .ops = fou_nl_ops, @@ -1035,7 +1140,7 @@ static int gue_err(struct sk_buff *skb, u32 info) case 0: /* Full GUE header present */ break; case 1: { - /* Direct encasulation of IPv4 or IPv6 */ + /* Direct encapsulation of IPv4 or IPv6 */ skb_set_transport_header(skb, -(int)sizeof(struct icmphdr)); switch (((struct iphdr *)guehdr)->version) { diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c index 6ea523d71947779b68d599de11a2a74535f578f0..a175e3e7ae97c53d0d9d8392589a86cdbc4c886e 100644 --- a/net/ipv4/inet_connection_sock.c +++ b/net/ipv4/inet_connection_sock.c @@ -564,7 +564,7 @@ struct dst_entry *inet_csk_route_req(const struct sock *sk, rt = ip_route_output_flow(net, fl4, sk); if (IS_ERR(rt)) goto no_route; - if (opt && opt->opt.is_strictroute && rt->rt_uses_gateway) + if (opt && opt->opt.is_strictroute && rt->rt_gw_family) goto route_err; rcu_read_unlock(); return &rt->dst; @@ -602,7 +602,7 @@ struct dst_entry *inet_csk_route_child_sock(const struct sock *sk, rt = ip_route_output_flow(net, fl4, sk); if (IS_ERR(rt)) goto no_route; - if (opt && opt->opt.is_strictroute && rt->rt_uses_gateway) + if (opt && opt->opt.is_strictroute && rt->rt_gw_family) goto route_err; return &rt->dst; diff --git a/net/ipv4/ip_forward.c b/net/ipv4/ip_forward.c index 00ec819f949b5e76ea96be901a697f4e12d5cf4d..06f6f280b9ff38864020e14a8a728b5258e65b0b 100644 --- a/net/ipv4/ip_forward.c +++ b/net/ipv4/ip_forward.c @@ -123,7 +123,7 @@ int ip_forward(struct sk_buff *skb) rt = skb_rtable(skb); - if (opt->is_strictroute && rt->rt_uses_gateway) + if (opt->is_strictroute && rt->rt_gw_family) goto sr_failed; IPCB(skb)->flags |= IPSKB_FORWARDED; diff --git a/net/ipv4/ip_input.c b/net/ipv4/ip_input.c index 1132d6d1796a4f7c947da76b9b39e7fbe11d3399..ed97724c5e3380f73b1a1bf86241ba4d39828efd 100644 --- a/net/ipv4/ip_input.c +++ b/net/ipv4/ip_input.c @@ -130,6 +130,7 @@ #include #include #include +#include #include #include @@ -188,6 +189,8 @@ bool ip_call_ra_chain(struct sk_buff *skb) return false; } +INDIRECT_CALLABLE_DECLARE(int udp_rcv(struct sk_buff *)); +INDIRECT_CALLABLE_DECLARE(int tcp_v4_rcv(struct sk_buff *)); void ip_protocol_deliver_rcu(struct net *net, struct sk_buff *skb, int protocol) { const struct net_protocol *ipprot; @@ -205,7 +208,8 @@ resubmit: } nf_reset(skb); } - ret = ipprot->handler(skb); + ret = INDIRECT_CALL_2(ipprot->handler, tcp_v4_rcv, udp_rcv, + skb); if (ret < 0) { protocol = -ret; goto resubmit; @@ -305,6 +309,8 @@ drop: return true; } +INDIRECT_CALLABLE_DECLARE(int udp_v4_early_demux(struct sk_buff *)); +INDIRECT_CALLABLE_DECLARE(int tcp_v4_early_demux(struct sk_buff *)); static int ip_rcv_finish_core(struct net *net, struct sock *sk, struct sk_buff *skb, struct net_device *dev) { @@ -322,7 +328,8 @@ static int ip_rcv_finish_core(struct net *net, struct sock *sk, ipprot = rcu_dereference(inet_protos[protocol]); if (ipprot && (edemux = READ_ONCE(ipprot->early_demux))) { - err = edemux(skb); + err = INDIRECT_CALL_2(edemux, tcp_v4_early_demux, + udp_v4_early_demux, skb); if (unlikely(err)) goto drop_error; /* must reload iph, skb->head might have changed */ diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index e8bb2e85c5a471f018193b8a08735606ec7f0526..ac880beda8a76e971b1b64d743319aca7dddd972 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -188,7 +188,7 @@ static int ip_finish_output2(struct net *net, struct sock *sk, struct sk_buff *s struct net_device *dev = dst->dev; unsigned int hh_len = LL_RESERVED_SPACE(dev); struct neighbour *neigh; - u32 nexthop; + bool is_v6gw = false; if (rt->rt_type == RTN_MULTICAST) { IP_UPD_PO_STATS(net, IPSTATS_MIB_OUTMCAST, skb->len); @@ -218,16 +218,13 @@ static int ip_finish_output2(struct net *net, struct sock *sk, struct sk_buff *s } rcu_read_lock_bh(); - nexthop = (__force u32) rt_nexthop(rt, ip_hdr(skb)->daddr); - neigh = __ipv4_neigh_lookup_noref(dev, nexthop); - if (unlikely(!neigh)) - neigh = __neigh_create(&arp_tbl, &nexthop, dev, false); + neigh = ip_neigh_for_gw(rt, skb, &is_v6gw); if (!IS_ERR(neigh)) { int res; sock_confirm_neigh(skb, neigh); - res = neigh_output(neigh, skb); - + /* if crossing protocols, can not use the cached header */ + res = neigh_output(neigh, skb, is_v6gw); rcu_read_unlock_bh(); return res; } @@ -472,7 +469,7 @@ int __ip_queue_xmit(struct sock *sk, struct sk_buff *skb, struct flowi *fl, skb_dst_set_noref(skb, &rt->dst); packet_routed: - if (inet_opt && inet_opt->opt.is_strictroute && rt->rt_uses_gateway) + if (inet_opt && inet_opt->opt.is_strictroute && rt->rt_gw_family) goto no_route; /* OK, we know where to send it, allocate and build IP header. */ @@ -694,11 +691,8 @@ int ip_do_fragment(struct net *net, struct sock *sk, struct sk_buff *skb, return 0; } - while (frag) { - skb = frag->next; - kfree_skb(frag); - frag = skb; - } + kfree_skb_list(frag); + IP_INC_STATS(net, IPSTATS_MIB_FRAGFAILS); return err; diff --git a/net/ipv4/ip_tunnel_core.c b/net/ipv4/ip_tunnel_core.c index c3f3d28d1087eea2660df456b0b0b3327873eca4..30c1c264bdfc26ffb32e770ab13df7adb24b9da3 100644 --- a/net/ipv4/ip_tunnel_core.c +++ b/net/ipv4/ip_tunnel_core.c @@ -239,8 +239,8 @@ static int ip_tun_build_state(struct nlattr *attr, struct nlattr *tb[LWTUNNEL_IP_MAX + 1]; int err; - err = nla_parse_nested(tb, LWTUNNEL_IP_MAX, attr, ip_tun_policy, - extack); + err = nla_parse_nested_deprecated(tb, LWTUNNEL_IP_MAX, attr, + ip_tun_policy, extack); if (err < 0) return err; @@ -356,8 +356,8 @@ static int ip6_tun_build_state(struct nlattr *attr, struct nlattr *tb[LWTUNNEL_IP6_MAX + 1]; int err; - err = nla_parse_nested(tb, LWTUNNEL_IP6_MAX, attr, ip6_tun_policy, - extack); + err = nla_parse_nested_deprecated(tb, LWTUNNEL_IP6_MAX, attr, + ip6_tun_policy, extack); if (err < 0) return err; diff --git a/net/ipv4/ip_vti.c b/net/ipv4/ip_vti.c index 35d8346742e2cc2bd7dd242501870a7681da0f96..254a42e83ff987bbfb406906762cac6f596c2047 100644 --- a/net/ipv4/ip_vti.c +++ b/net/ipv4/ip_vti.c @@ -50,7 +50,7 @@ static unsigned int vti_net_id __read_mostly; static int vti_tunnel_init(struct net_device *dev); static int vti_input(struct sk_buff *skb, int nexthdr, __be32 spi, - int encap_type) + int encap_type, bool update_skb_dev) { struct ip_tunnel *tunnel; const struct iphdr *iph = ip_hdr(skb); @@ -65,6 +65,9 @@ static int vti_input(struct sk_buff *skb, int nexthdr, __be32 spi, XFRM_TUNNEL_SKB_CB(skb)->tunnel.ip4 = tunnel; + if (update_skb_dev) + skb->dev = tunnel->dev; + return xfrm_input(skb, nexthdr, spi, encap_type); } @@ -74,47 +77,28 @@ drop: return 0; } -static int vti_input_ipip(struct sk_buff *skb, int nexthdr, __be32 spi, - int encap_type) +static int vti_input_proto(struct sk_buff *skb, int nexthdr, __be32 spi, + int encap_type) { - struct ip_tunnel *tunnel; - const struct iphdr *iph = ip_hdr(skb); - struct net *net = dev_net(skb->dev); - struct ip_tunnel_net *itn = net_generic(net, vti_net_id); - - tunnel = ip_tunnel_lookup(itn, skb->dev->ifindex, TUNNEL_NO_KEY, - iph->saddr, iph->daddr, 0); - if (tunnel) { - if (!xfrm4_policy_check(NULL, XFRM_POLICY_IN, skb)) - goto drop; - - XFRM_TUNNEL_SKB_CB(skb)->tunnel.ip4 = tunnel; - - skb->dev = tunnel->dev; - - return xfrm_input(skb, nexthdr, spi, encap_type); - } - - return -EINVAL; -drop: - kfree_skb(skb); - return 0; + return vti_input(skb, nexthdr, spi, encap_type, false); } -static int vti_rcv(struct sk_buff *skb) +static int vti_rcv(struct sk_buff *skb, __be32 spi, bool update_skb_dev) { XFRM_SPI_SKB_CB(skb)->family = AF_INET; XFRM_SPI_SKB_CB(skb)->daddroff = offsetof(struct iphdr, daddr); - return vti_input(skb, ip_hdr(skb)->protocol, 0, 0); + return vti_input(skb, ip_hdr(skb)->protocol, spi, 0, update_skb_dev); } -static int vti_rcv_ipip(struct sk_buff *skb) +static int vti_rcv_proto(struct sk_buff *skb) { - XFRM_SPI_SKB_CB(skb)->family = AF_INET; - XFRM_SPI_SKB_CB(skb)->daddroff = offsetof(struct iphdr, daddr); + return vti_rcv(skb, 0, false); +} - return vti_input_ipip(skb, ip_hdr(skb)->protocol, ip_hdr(skb)->saddr, 0); +static int vti_rcv_tunnel(struct sk_buff *skb) +{ + return vti_rcv(skb, ip_hdr(skb)->saddr, true); } static int vti_rcv_cb(struct sk_buff *skb, int err) @@ -123,7 +107,7 @@ static int vti_rcv_cb(struct sk_buff *skb, int err) struct net_device *dev; struct pcpu_sw_netstats *tstats; struct xfrm_state *x; - struct xfrm_mode *inner_mode; + const struct xfrm_mode *inner_mode; struct ip_tunnel *tunnel = XFRM_TUNNEL_SKB_CB(skb)->tunnel.ip4; u32 orig_mark = skb->mark; int ret; @@ -142,7 +126,7 @@ static int vti_rcv_cb(struct sk_buff *skb, int err) x = xfrm_input_state(skb); - inner_mode = x->inner_mode; + inner_mode = &x->inner_mode; if (x->sel.family == AF_UNSPEC) { inner_mode = xfrm_ip2inner_mode(x, XFRM_MODE_SKB_CB(skb)->protocol); @@ -153,7 +137,7 @@ static int vti_rcv_cb(struct sk_buff *skb, int err) } } - family = inner_mode->afinfo->family; + family = inner_mode->family; skb->mark = be32_to_cpu(tunnel->parms.i_key); ret = xfrm_policy_check(NULL, XFRM_POLICY_IN, skb, family); @@ -447,31 +431,31 @@ static void __net_init vti_fb_tunnel_init(struct net_device *dev) } static struct xfrm4_protocol vti_esp4_protocol __read_mostly = { - .handler = vti_rcv, - .input_handler = vti_input, + .handler = vti_rcv_proto, + .input_handler = vti_input_proto, .cb_handler = vti_rcv_cb, .err_handler = vti4_err, .priority = 100, }; static struct xfrm4_protocol vti_ah4_protocol __read_mostly = { - .handler = vti_rcv, - .input_handler = vti_input, + .handler = vti_rcv_proto, + .input_handler = vti_input_proto, .cb_handler = vti_rcv_cb, .err_handler = vti4_err, .priority = 100, }; static struct xfrm4_protocol vti_ipcomp4_protocol __read_mostly = { - .handler = vti_rcv, - .input_handler = vti_input, + .handler = vti_rcv_proto, + .input_handler = vti_input_proto, .cb_handler = vti_rcv_cb, .err_handler = vti4_err, .priority = 100, }; static struct xfrm_tunnel ipip_handler __read_mostly = { - .handler = vti_rcv_ipip, + .handler = vti_rcv_tunnel, .err_handler = vti4_err, .priority = 0, }; diff --git a/net/ipv4/ipmr.c b/net/ipv4/ipmr.c index 2c931120c49402ad0bb3e946873cac66540ef0d7..2c61e10a60e33679d308699040d585fac187d77d 100644 --- a/net/ipv4/ipmr.c +++ b/net/ipv4/ipmr.c @@ -66,7 +66,7 @@ #include #include #include -#include +#include #include @@ -373,7 +373,6 @@ static const struct rhashtable_params ipmr_rht_params = { .key_offset = offsetof(struct mfc_cache, cmparg), .key_len = sizeof(struct mfc_cache_cmp_arg), .nelem_hint = 3, - .locks_mul = 1, .obj_cmpfn = ipmr_hash_cmp, .automatic_shrinking = true, }; @@ -2499,8 +2498,8 @@ static int ipmr_rtm_valid_getroute_req(struct sk_buff *skb, } if (!netlink_strict_get_check(skb)) - return nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, - rtm_ipv4_policy, extack); + return nlmsg_parse_deprecated(nlh, sizeof(*rtm), tb, RTA_MAX, + rtm_ipv4_policy, extack); rtm = nlmsg_data(nlh); if ((rtm->rtm_src_len && rtm->rtm_src_len != 32) || @@ -2511,8 +2510,8 @@ static int ipmr_rtm_valid_getroute_req(struct sk_buff *skb, return -EINVAL; } - err = nlmsg_parse_strict(nlh, sizeof(*rtm), tb, RTA_MAX, - rtm_ipv4_policy, extack); + err = nlmsg_parse_deprecated_strict(nlh, sizeof(*rtm), tb, RTA_MAX, + rtm_ipv4_policy, extack); if (err) return err; @@ -2675,8 +2674,8 @@ static int rtm_to_ipmr_mfcc(struct net *net, struct nlmsghdr *nlh, struct rtmsg *rtm; int ret, rem; - ret = nlmsg_validate(nlh, sizeof(*rtm), RTA_MAX, rtm_ipmr_policy, - extack); + ret = nlmsg_validate_deprecated(nlh, sizeof(*rtm), RTA_MAX, + rtm_ipmr_policy, extack); if (ret < 0) goto out; rtm = nlmsg_data(nlh); @@ -2784,7 +2783,7 @@ static bool ipmr_fill_vif(struct mr_table *mrt, u32 vifid, struct sk_buff *skb) return true; vif = &mrt->vif_table[vifid]; - vif_nest = nla_nest_start(skb, IPMRA_VIF); + vif_nest = nla_nest_start_noflag(skb, IPMRA_VIF); if (!vif_nest) return false; if (nla_put_u32(skb, IPMRA_VIFA_IFINDEX, vif->dev->ifindex) || @@ -2868,7 +2867,7 @@ static int ipmr_rtm_dumplink(struct sk_buff *skb, struct netlink_callback *cb) memset(hdr, 0, sizeof(*hdr)); hdr->ifi_family = RTNL_FAMILY_IPMR; - af = nla_nest_start(skb, IFLA_AF_SPEC); + af = nla_nest_start_noflag(skb, IFLA_AF_SPEC); if (!af) { nlmsg_cancel(skb, nlh); goto out; @@ -2879,7 +2878,7 @@ static int ipmr_rtm_dumplink(struct sk_buff *skb, struct netlink_callback *cb) goto out; } - vifs = nla_nest_start(skb, IPMRA_TABLE_VIFS); + vifs = nla_nest_start_noflag(skb, IPMRA_TABLE_VIFS); if (!vifs) { nla_nest_end(skb, af); nlmsg_end(skb, nlh); diff --git a/net/ipv4/ipmr_base.c b/net/ipv4/ipmr_base.c index 3e614cc824f77948493a7867009591b3b09a0179..ea48bd15a575bf38ce44ab7c86bab4f0935bd5b8 100644 --- a/net/ipv4/ipmr_base.c +++ b/net/ipv4/ipmr_base.c @@ -228,7 +228,7 @@ int mr_fill_mroute(struct mr_table *mrt, struct sk_buff *skb, if (c->mfc_flags & MFC_OFFLOAD) rtm->rtm_flags |= RTNH_F_OFFLOAD; - mp_attr = nla_nest_start(skb, RTA_MULTIPATH); + mp_attr = nla_nest_start_noflag(skb, RTA_MULTIPATH); if (!mp_attr) return -EMSGSIZE; @@ -335,8 +335,6 @@ next_entry2: } spin_unlock_bh(lock); err = 0; - e = 0; - out: cb->args[1] = e; return err; @@ -374,6 +372,7 @@ int mr_rtm_dumproute(struct sk_buff *skb, struct netlink_callback *cb, err = mr_table_dump(mrt, skb, cb, fill, lock, filter); if (err < 0) break; + cb->args[1] = 0; next_table: t++; } diff --git a/net/ipv4/netfilter/Kconfig b/net/ipv4/netfilter/Kconfig index c98391d492006d251c3e8c0f025cbbb370bbcc9f..1412b029f37f9444733762ae4d7571d61842e834 100644 --- a/net/ipv4/netfilter/Kconfig +++ b/net/ipv4/netfilter/Kconfig @@ -27,14 +27,6 @@ config NF_TABLES_IPV4 if NF_TABLES_IPV4 -config NFT_CHAIN_ROUTE_IPV4 - tristate "IPv4 nf_tables route chain support" - help - This option enables the "route" chain for IPv4 in nf_tables. This - chain type is used to force packet re-routing after mangling header - fields such as the source, destination, type of service and - the packet mark. - config NFT_REJECT_IPV4 select NF_REJECT_IPV4 default NFT_REJECT @@ -232,16 +224,10 @@ if IP_NF_NAT config IP_NF_TARGET_MASQUERADE tristate "MASQUERADE target support" - select NF_NAT_MASQUERADE - default m if NETFILTER_ADVANCED=n + select NETFILTER_XT_TARGET_MASQUERADE help - Masquerading is a special case of NAT: all outgoing connections are - changed to seem to come from a particular interface's address, and - if the interface goes down, those connections are lost. This is - only useful for dialup accounts with dynamic IP address (ie. your IP - address will be different on next dialup). - - To compile it as a module, choose M here. If unsure, say N. + This is a backwards-compat option for the user's convenience + (e.g. when running oldconfig). It selects NETFILTER_XT_TARGET_MASQUERADE. config IP_NF_TARGET_NETMAP tristate "NETMAP target support" diff --git a/net/ipv4/netfilter/Makefile b/net/ipv4/netfilter/Makefile index e241f5188ebef0e36237f3144c6b827a8e3eadcb..c50e0ec095d2193bca5403abb4afcac726126dea 100644 --- a/net/ipv4/netfilter/Makefile +++ b/net/ipv4/netfilter/Makefile @@ -24,7 +24,6 @@ nf_nat_snmp_basic-y := nf_nat_snmp_basic.asn1.o nf_nat_snmp_basic_main.o $(obj)/nf_nat_snmp_basic_main.o: $(obj)/nf_nat_snmp_basic.asn1.h obj-$(CONFIG_NF_NAT_SNMP_BASIC) += nf_nat_snmp_basic.o -obj-$(CONFIG_NFT_CHAIN_ROUTE_IPV4) += nft_chain_route_ipv4.o obj-$(CONFIG_NFT_REJECT_IPV4) += nft_reject_ipv4.o obj-$(CONFIG_NFT_FIB_IPV4) += nft_fib_ipv4.o obj-$(CONFIG_NFT_DUP_IPV4) += nft_dup_ipv4.o @@ -49,7 +48,6 @@ obj-$(CONFIG_IP_NF_MATCH_RPFILTER) += ipt_rpfilter.o # targets obj-$(CONFIG_IP_NF_TARGET_CLUSTERIP) += ipt_CLUSTERIP.o obj-$(CONFIG_IP_NF_TARGET_ECN) += ipt_ECN.o -obj-$(CONFIG_IP_NF_TARGET_MASQUERADE) += ipt_MASQUERADE.o obj-$(CONFIG_IP_NF_TARGET_REJECT) += ipt_REJECT.o obj-$(CONFIG_IP_NF_TARGET_SYNPROXY) += ipt_SYNPROXY.o diff --git a/net/ipv4/netfilter/nf_nat_h323.c b/net/ipv4/netfilter/nf_nat_h323.c index 4e6b53ab6c3376a8f5ef363f9b2d9ea72cb95b55..7875c98072eb93fb0f555feda39e8433b9c37719 100644 --- a/net/ipv4/netfilter/nf_nat_h323.c +++ b/net/ipv4/netfilter/nf_nat_h323.c @@ -631,4 +631,4 @@ module_exit(fini); MODULE_AUTHOR("Jing Min Zhao "); MODULE_DESCRIPTION("H.323 NAT helper"); MODULE_LICENSE("GPL"); -MODULE_ALIAS("ip_nat_h323"); +MODULE_ALIAS_NF_NAT_HELPER("h323"); diff --git a/net/ipv4/netfilter/nf_nat_pptp.c b/net/ipv4/netfilter/nf_nat_pptp.c index 68b4d450391b8ba9c46b7f02e9243692b3ba4115..e17b4ee7604c98a732229175dd3ea8e109c3fb0f 100644 --- a/net/ipv4/netfilter/nf_nat_pptp.c +++ b/net/ipv4/netfilter/nf_nat_pptp.c @@ -37,7 +37,7 @@ MODULE_LICENSE("GPL"); MODULE_AUTHOR("Harald Welte "); MODULE_DESCRIPTION("Netfilter NAT helper module for PPTP"); -MODULE_ALIAS("ip_nat_pptp"); +MODULE_ALIAS_NF_NAT_HELPER("pptp"); static void pptp_nat_expected(struct nf_conn *ct, struct nf_conntrack_expect *exp) diff --git a/net/ipv4/netfilter/nft_chain_route_ipv4.c b/net/ipv4/netfilter/nft_chain_route_ipv4.c deleted file mode 100644 index 7d82934c46f424bdbd8e0c0280b082c43858567f..0000000000000000000000000000000000000000 --- a/net/ipv4/netfilter/nft_chain_route_ipv4.c +++ /dev/null @@ -1,89 +0,0 @@ -/* - * Copyright (c) 2008 Patrick McHardy - * Copyright (c) 2012 Pablo Neira Ayuso - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -static unsigned int nf_route_table_hook(void *priv, - struct sk_buff *skb, - const struct nf_hook_state *state) -{ - unsigned int ret; - struct nft_pktinfo pkt; - u32 mark; - __be32 saddr, daddr; - u_int8_t tos; - const struct iphdr *iph; - int err; - - nft_set_pktinfo(&pkt, skb, state); - nft_set_pktinfo_ipv4(&pkt, skb); - - mark = skb->mark; - iph = ip_hdr(skb); - saddr = iph->saddr; - daddr = iph->daddr; - tos = iph->tos; - - ret = nft_do_chain(&pkt, priv); - if (ret != NF_DROP && ret != NF_STOLEN) { - iph = ip_hdr(skb); - - if (iph->saddr != saddr || - iph->daddr != daddr || - skb->mark != mark || - iph->tos != tos) { - err = ip_route_me_harder(state->net, skb, RTN_UNSPEC); - if (err < 0) - ret = NF_DROP_ERR(err); - } - } - return ret; -} - -static const struct nft_chain_type nft_chain_route_ipv4 = { - .name = "route", - .type = NFT_CHAIN_T_ROUTE, - .family = NFPROTO_IPV4, - .owner = THIS_MODULE, - .hook_mask = (1 << NF_INET_LOCAL_OUT), - .hooks = { - [NF_INET_LOCAL_OUT] = nf_route_table_hook, - }, -}; - -static int __init nft_chain_route_init(void) -{ - nft_register_chain_type(&nft_chain_route_ipv4); - - return 0; -} - -static void __exit nft_chain_route_exit(void) -{ - nft_unregister_chain_type(&nft_chain_route_ipv4); -} - -module_init(nft_chain_route_init); -module_exit(nft_chain_route_exit); - -MODULE_LICENSE("GPL"); -MODULE_AUTHOR("Patrick McHardy "); -MODULE_ALIAS_NFT_CHAIN(AF_INET, "route"); diff --git a/net/ipv4/route.c b/net/ipv4/route.c index 6fdf1c195d8e3a0e32af0359794f798457a21cb3..11ddc276776e357b7998142f9e80327ea7e89c03 100644 --- a/net/ipv4/route.c +++ b/net/ipv4/route.c @@ -434,37 +434,46 @@ static struct neighbour *ipv4_neigh_lookup(const struct dst_entry *dst, struct sk_buff *skb, const void *daddr) { + const struct rtable *rt = container_of(dst, struct rtable, dst); struct net_device *dev = dst->dev; - const __be32 *pkey = daddr; - const struct rtable *rt; struct neighbour *n; - rt = (const struct rtable *) dst; - if (rt->rt_gateway) - pkey = (const __be32 *) &rt->rt_gateway; - else if (skb) - pkey = &ip_hdr(skb)->daddr; + rcu_read_lock_bh(); + + if (likely(rt->rt_gw_family == AF_INET)) { + n = ip_neigh_gw4(dev, rt->rt_gw4); + } else if (rt->rt_gw_family == AF_INET6) { + n = ip_neigh_gw6(dev, &rt->rt_gw6); + } else { + __be32 pkey; + + pkey = skb ? ip_hdr(skb)->daddr : *((__be32 *) daddr); + n = ip_neigh_gw4(dev, pkey); + } + + if (n && !refcount_inc_not_zero(&n->refcnt)) + n = NULL; + + rcu_read_unlock_bh(); - n = __ipv4_neigh_lookup(dev, *(__force u32 *)pkey); - if (n) - return n; - return neigh_create(&arp_tbl, pkey, dev); + return n; } static void ipv4_confirm_neigh(const struct dst_entry *dst, const void *daddr) { + const struct rtable *rt = container_of(dst, struct rtable, dst); struct net_device *dev = dst->dev; const __be32 *pkey = daddr; - const struct rtable *rt; - rt = (const struct rtable *)dst; - if (rt->rt_gateway) - pkey = (const __be32 *)&rt->rt_gateway; - else if (!daddr || + if (rt->rt_gw_family == AF_INET) { + pkey = (const __be32 *)&rt->rt_gw4; + } else if (rt->rt_gw_family == AF_INET6) { + return __ipv6_confirm_neigh_stub(dev, &rt->rt_gw6); + } else if (!daddr || (rt->rt_flags & - (RTCF_MULTICAST | RTCF_BROADCAST | RTCF_LOCAL))) + (RTCF_MULTICAST | RTCF_BROADCAST | RTCF_LOCAL))) { return; - + } __ipv4_confirm_neigh(dev, *(__force u32 *)pkey); } @@ -500,15 +509,17 @@ EXPORT_SYMBOL(ip_idents_reserve); void __ip_select_ident(struct net *net, struct iphdr *iph, int segs) { - static u32 ip_idents_hashrnd __read_mostly; u32 hash, id; - net_get_random_once(&ip_idents_hashrnd, sizeof(ip_idents_hashrnd)); + /* Note the following code is not safe, but this is okay. */ + if (unlikely(siphash_key_is_zero(&net->ipv4.ip_id_key))) + get_random_bytes(&net->ipv4.ip_id_key, + sizeof(net->ipv4.ip_id_key)); - hash = jhash_3words((__force u32)iph->daddr, + hash = siphash_3u32((__force u32)iph->daddr, (__force u32)iph->saddr, - iph->protocol ^ net_hash_mix(net), - ip_idents_hashrnd); + iph->protocol, + &net->ipv4.ip_id_key); id = ip_idents_reserve(hash, segs); iph->id = htons(id); } @@ -627,13 +638,14 @@ static void fill_route_from_fnhe(struct rtable *rt, struct fib_nh_exception *fnh if (fnhe->fnhe_gw) { rt->rt_flags |= RTCF_REDIRECTED; - rt->rt_gateway = fnhe->fnhe_gw; - rt->rt_uses_gateway = 1; + rt->rt_gw_family = AF_INET; + rt->rt_gw4 = fnhe->fnhe_gw; } } -static void update_or_create_fnhe(struct fib_nh *nh, __be32 daddr, __be32 gw, - u32 pmtu, bool lock, unsigned long expires) +static void update_or_create_fnhe(struct fib_nh_common *nhc, __be32 daddr, + __be32 gw, u32 pmtu, bool lock, + unsigned long expires) { struct fnhe_hash_bucket *hash; struct fib_nh_exception *fnhe; @@ -642,17 +654,17 @@ static void update_or_create_fnhe(struct fib_nh *nh, __be32 daddr, __be32 gw, unsigned int i; int depth; - genid = fnhe_genid(dev_net(nh->nh_dev)); + genid = fnhe_genid(dev_net(nhc->nhc_dev)); hval = fnhe_hashfun(daddr); spin_lock_bh(&fnhe_lock); - hash = rcu_dereference(nh->nh_exceptions); + hash = rcu_dereference(nhc->nhc_exceptions); if (!hash) { hash = kcalloc(FNHE_HASH_SIZE, sizeof(*hash), GFP_ATOMIC); if (!hash) goto out_unlock; - rcu_assign_pointer(nh->nh_exceptions, hash); + rcu_assign_pointer(nhc->nhc_exceptions, hash); } hash += hval; @@ -704,13 +716,13 @@ static void update_or_create_fnhe(struct fib_nh *nh, __be32 daddr, __be32 gw, * stale, so anyone caching it rechecks if this exception * applies to them. */ - rt = rcu_dereference(nh->nh_rth_input); + rt = rcu_dereference(nhc->nhc_rth_input); if (rt) rt->dst.obsolete = DST_OBSOLETE_KILL; for_each_possible_cpu(i) { struct rtable __rcu **prt; - prt = per_cpu_ptr(nh->nh_pcpu_rth_output, i); + prt = per_cpu_ptr(nhc->nhc_pcpu_rth_output, i); rt = rcu_dereference(*prt); if (rt) rt->dst.obsolete = DST_OBSOLETE_KILL; @@ -745,7 +757,7 @@ static void __ip_do_redirect(struct rtable *rt, struct sk_buff *skb, struct flow return; } - if (rt->rt_gateway != old_gw) + if (rt->rt_gw_family != AF_INET || rt->rt_gw4 != old_gw) return; in_dev = __in_dev_get_rcu(dev); @@ -776,9 +788,9 @@ static void __ip_do_redirect(struct rtable *rt, struct sk_buff *skb, struct flow neigh_event_send(n, NULL); } else { if (fib_lookup(net, fl4, &res, 0) == 0) { - struct fib_nh *nh = &FIB_RES_NH(res); + struct fib_nh_common *nhc = FIB_RES_NHC(res); - update_or_create_fnhe(nh, fl4->daddr, new_gw, + update_or_create_fnhe(nhc, fl4->daddr, new_gw, 0, false, jiffies + ip_rt_gc_timeout); } @@ -1025,9 +1037,9 @@ static void __ip_rt_update_pmtu(struct rtable *rt, struct flowi4 *fl4, u32 mtu) rcu_read_lock(); if (fib_lookup(dev_net(dst->dev), fl4, &res, 0) == 0) { - struct fib_nh *nh = &FIB_RES_NH(res); + struct fib_nh_common *nhc = FIB_RES_NHC(res); - update_or_create_fnhe(nh, fl4->daddr, 0, mtu, lock, + update_or_create_fnhe(nhc, fl4->daddr, 0, mtu, lock, jiffies + ip_rt_mtu_expires); } rcu_read_unlock(); @@ -1176,7 +1188,7 @@ static struct dst_entry *ipv4_dst_check(struct dst_entry *dst, u32 cookie) * * When a PMTU/redirect information update invalidates a route, * this is indicated by setting obsolete to DST_OBSOLETE_KILL or - * DST_OBSOLETE_DEAD by dst_free(). + * DST_OBSOLETE_DEAD. */ if (dst->obsolete != DST_OBSOLETE_FORCE_CHK || rt_is_expired(rt)) return NULL; @@ -1261,7 +1273,7 @@ void ip_rt_get_source(u8 *addr, struct sk_buff *skb, struct rtable *rt) rcu_read_lock(); if (fib_lookup(dev_net(rt->dst.dev), &fl4, &res, 0) == 0) - src = FIB_RES_PREFSRC(dev_net(rt->dst.dev), res); + src = fib_result_prefsrc(dev_net(rt->dst.dev), &res); else src = inet_select_addr(rt->dst.dev, rt_nexthop(rt, iph->daddr), @@ -1304,7 +1316,7 @@ static unsigned int ipv4_mtu(const struct dst_entry *dst) mtu = READ_ONCE(dst->dev->mtu); if (unlikely(ip_mtu_locked(dst))) { - if (rt->rt_uses_gateway && mtu > 576) + if (rt->rt_gw_family && mtu > 576) mtu = 576; } @@ -1313,7 +1325,7 @@ static unsigned int ipv4_mtu(const struct dst_entry *dst) return mtu - lwtunnel_headroom(dst->lwtstate, mtu); } -static void ip_del_fnhe(struct fib_nh *nh, __be32 daddr) +static void ip_del_fnhe(struct fib_nh_common *nhc, __be32 daddr) { struct fnhe_hash_bucket *hash; struct fib_nh_exception *fnhe, __rcu **fnhe_p; @@ -1321,7 +1333,7 @@ static void ip_del_fnhe(struct fib_nh *nh, __be32 daddr) spin_lock_bh(&fnhe_lock); - hash = rcu_dereference_protected(nh->nh_exceptions, + hash = rcu_dereference_protected(nhc->nhc_exceptions, lockdep_is_held(&fnhe_lock)); hash += hval; @@ -1347,9 +1359,10 @@ static void ip_del_fnhe(struct fib_nh *nh, __be32 daddr) spin_unlock_bh(&fnhe_lock); } -static struct fib_nh_exception *find_exception(struct fib_nh *nh, __be32 daddr) +static struct fib_nh_exception *find_exception(struct fib_nh_common *nhc, + __be32 daddr) { - struct fnhe_hash_bucket *hash = rcu_dereference(nh->nh_exceptions); + struct fnhe_hash_bucket *hash = rcu_dereference(nhc->nhc_exceptions); struct fib_nh_exception *fnhe; u32 hval; @@ -1363,7 +1376,7 @@ static struct fib_nh_exception *find_exception(struct fib_nh *nh, __be32 daddr) if (fnhe->fnhe_daddr == daddr) { if (fnhe->fnhe_expires && time_after(jiffies, fnhe->fnhe_expires)) { - ip_del_fnhe(nh, daddr); + ip_del_fnhe(nhc, daddr); break; } return fnhe; @@ -1380,9 +1393,9 @@ static struct fib_nh_exception *find_exception(struct fib_nh *nh, __be32 daddr) u32 ip_mtu_from_fib_result(struct fib_result *res, __be32 daddr) { + struct fib_nh_common *nhc = res->nhc; + struct net_device *dev = nhc->nhc_dev; struct fib_info *fi = res->fi; - struct fib_nh *nh = &fi->fib_nh[res->nh_sel]; - struct net_device *dev = nh->nh_dev; u32 mtu = 0; if (dev_net(dev)->ipv4.sysctl_ip_fwd_use_pmtu || @@ -1392,7 +1405,7 @@ u32 ip_mtu_from_fib_result(struct fib_result *res, __be32 daddr) if (likely(!mtu)) { struct fib_nh_exception *fnhe; - fnhe = find_exception(nh, daddr); + fnhe = find_exception(nhc, daddr); if (fnhe && !time_after_eq(jiffies, fnhe->fnhe_expires)) mtu = fnhe->fnhe_pmtu; } @@ -1400,7 +1413,7 @@ u32 ip_mtu_from_fib_result(struct fib_result *res, __be32 daddr) if (likely(!mtu)) mtu = min(READ_ONCE(dev->mtu), IP_MAX_MTU); - return mtu - lwtunnel_headroom(nh->nh_lwtstate, mtu); + return mtu - lwtunnel_headroom(nhc->nhc_lwtstate, mtu); } static bool rt_bind_exception(struct rtable *rt, struct fib_nh_exception *fnhe, @@ -1431,8 +1444,10 @@ static bool rt_bind_exception(struct rtable *rt, struct fib_nh_exception *fnhe, orig = NULL; } fill_route_from_fnhe(rt, fnhe); - if (!rt->rt_gateway) - rt->rt_gateway = daddr; + if (!rt->rt_gw4) { + rt->rt_gw4 = daddr; + rt->rt_gw_family = AF_INET; + } if (do_cache) { dst_hold(&rt->dst); @@ -1451,15 +1466,15 @@ static bool rt_bind_exception(struct rtable *rt, struct fib_nh_exception *fnhe, return ret; } -static bool rt_cache_route(struct fib_nh *nh, struct rtable *rt) +static bool rt_cache_route(struct fib_nh_common *nhc, struct rtable *rt) { struct rtable *orig, *prev, **p; bool ret = true; if (rt_is_input_route(rt)) { - p = (struct rtable **)&nh->nh_rth_input; + p = (struct rtable **)&nhc->nhc_rth_input; } else { - p = (struct rtable **)raw_cpu_ptr(nh->nh_pcpu_rth_output); + p = (struct rtable **)raw_cpu_ptr(nhc->nhc_pcpu_rth_output); } orig = *p; @@ -1555,30 +1570,42 @@ static void rt_set_nexthop(struct rtable *rt, __be32 daddr, bool cached = false; if (fi) { - struct fib_nh *nh = &FIB_RES_NH(*res); + struct fib_nh_common *nhc = FIB_RES_NHC(*res); - if (nh->nh_gw && nh->nh_scope == RT_SCOPE_LINK) { - rt->rt_gateway = nh->nh_gw; - rt->rt_uses_gateway = 1; + if (nhc->nhc_gw_family && nhc->nhc_scope == RT_SCOPE_LINK) { + rt->rt_gw_family = nhc->nhc_gw_family; + /* only INET and INET6 are supported */ + if (likely(nhc->nhc_gw_family == AF_INET)) + rt->rt_gw4 = nhc->nhc_gw.ipv4; + else + rt->rt_gw6 = nhc->nhc_gw.ipv6; } + ip_dst_init_metrics(&rt->dst, fi->fib_metrics); #ifdef CONFIG_IP_ROUTE_CLASSID - rt->dst.tclassid = nh->nh_tclassid; + { + struct fib_nh *nh; + + nh = container_of(nhc, struct fib_nh, nh_common); + rt->dst.tclassid = nh->nh_tclassid; + } #endif - rt->dst.lwtstate = lwtstate_get(nh->nh_lwtstate); + rt->dst.lwtstate = lwtstate_get(nhc->nhc_lwtstate); if (unlikely(fnhe)) cached = rt_bind_exception(rt, fnhe, daddr, do_cache); else if (do_cache) - cached = rt_cache_route(nh, rt); + cached = rt_cache_route(nhc, rt); if (unlikely(!cached)) { /* Routes we intend to cache in nexthop exception or * FIB nexthop have the DST_NOCACHE bit clear. * However, if we are unsuccessful at storing this * route into the cache we really need to set it. */ - if (!rt->rt_gateway) - rt->rt_gateway = daddr; + if (!rt->rt_gw4) { + rt->rt_gw_family = AF_INET; + rt->rt_gw4 = daddr; + } rt_add_uncached_list(rt); } } else @@ -1611,8 +1638,8 @@ struct rtable *rt_dst_alloc(struct net_device *dev, rt->rt_iif = 0; rt->rt_pmtu = 0; rt->rt_mtu_locked = 0; - rt->rt_gateway = 0; - rt->rt_uses_gateway = 0; + rt->rt_gw_family = 0; + rt->rt_gw4 = 0; INIT_LIST_HEAD(&rt->rt_uncached); rt->dst.output = ip_output; @@ -1725,6 +1752,8 @@ static int __mkroute_input(struct sk_buff *skb, struct in_device *in_dev, __be32 daddr, __be32 saddr, u32 tos) { + struct fib_nh_common *nhc = FIB_RES_NHC(*res); + struct net_device *dev = nhc->nhc_dev; struct fib_nh_exception *fnhe; struct rtable *rth; int err; @@ -1733,7 +1762,7 @@ static int __mkroute_input(struct sk_buff *skb, u32 itag = 0; /* get a working reference to the output device */ - out_dev = __in_dev_get_rcu(FIB_RES_DEV(*res)); + out_dev = __in_dev_get_rcu(dev); if (!out_dev) { net_crit_ratelimited("Bug in ip_route_input_slow(). Please report.\n"); return -EINVAL; @@ -1750,10 +1779,14 @@ static int __mkroute_input(struct sk_buff *skb, do_cache = res->fi && !itag; if (out_dev == in_dev && err && IN_DEV_TX_REDIRECTS(out_dev) && - skb->protocol == htons(ETH_P_IP) && - (IN_DEV_SHARED_MEDIA(out_dev) || - inet_addr_onlink(out_dev, saddr, FIB_RES_GW(*res)))) - IPCB(skb)->flags |= IPSKB_DOREDIRECT; + skb->protocol == htons(ETH_P_IP)) { + __be32 gw; + + gw = nhc->nhc_gw_family == AF_INET ? nhc->nhc_gw.ipv4 : 0; + if (IN_DEV_SHARED_MEDIA(out_dev) || + inet_addr_onlink(out_dev, saddr, gw)) + IPCB(skb)->flags |= IPSKB_DOREDIRECT; + } if (skb->protocol != htons(ETH_P_IP)) { /* Not IP (i.e. ARP). Do not create route, if it is @@ -1770,12 +1803,12 @@ static int __mkroute_input(struct sk_buff *skb, } } - fnhe = find_exception(&FIB_RES_NH(*res), daddr); + fnhe = find_exception(nhc, daddr); if (do_cache) { if (fnhe) rth = rcu_dereference(fnhe->fnhe_rth_input); else - rth = rcu_dereference(FIB_RES_NH(*res).nh_rth_input); + rth = rcu_dereference(nhc->nhc_rth_input); if (rt_cache_valid(rth)) { skb_dst_set_noref(skb, &rth->dst); goto out; @@ -2069,7 +2102,9 @@ local_input: do_cache = false; if (res->fi) { if (!itag) { - rth = rcu_dereference(FIB_RES_NH(*res).nh_rth_input); + struct fib_nh_common *nhc = FIB_RES_NHC(*res); + + rth = rcu_dereference(nhc->nhc_rth_input); if (rt_cache_valid(rth)) { skb_dst_set_noref(skb, &rth->dst); err = 0; @@ -2099,16 +2134,16 @@ local_input: } if (do_cache) { - struct fib_nh *nh = &FIB_RES_NH(*res); + struct fib_nh_common *nhc = FIB_RES_NHC(*res); - rth->dst.lwtstate = lwtstate_get(nh->nh_lwtstate); + rth->dst.lwtstate = lwtstate_get(nhc->nhc_lwtstate); if (lwtunnel_input_redirect(rth->dst.lwtstate)) { WARN_ON(rth->dst.input == lwtunnel_input); rth->dst.lwtstate->orig_input = rth->dst.input; rth->dst.input = lwtunnel_input; } - if (unlikely(!rt_cache_route(nh, rth))) + if (unlikely(!rt_cache_route(nhc, rth))) rt_add_uncached_list(rth); } skb_dst_set(skb, &rth->dst); @@ -2279,10 +2314,10 @@ static struct rtable *__mkroute_output(const struct fib_result *res, fnhe = NULL; do_cache &= fi != NULL; if (fi) { + struct fib_nh_common *nhc = FIB_RES_NHC(*res); struct rtable __rcu **prth; - struct fib_nh *nh = &FIB_RES_NH(*res); - fnhe = find_exception(nh, fl4->daddr); + fnhe = find_exception(nhc, fl4->daddr); if (!do_cache) goto add; if (fnhe) { @@ -2290,12 +2325,12 @@ static struct rtable *__mkroute_output(const struct fib_result *res, } else { if (unlikely(fl4->flowi4_flags & FLOWI_FLAG_KNOWN_NH && - !(nh->nh_gw && - nh->nh_scope == RT_SCOPE_LINK))) { + !(nhc->nhc_gw_family && + nhc->nhc_scope == RT_SCOPE_LINK))) { do_cache = false; goto add; } - prth = raw_cpu_ptr(nh->nh_pcpu_rth_output); + prth = raw_cpu_ptr(nhc->nhc_pcpu_rth_output); } rth = rcu_dereference(*prth); if (rt_cache_valid(rth) && dst_hold_safe(&rth->dst)) @@ -2600,8 +2635,11 @@ struct dst_entry *ipv4_blackhole_route(struct net *net, struct dst_entry *dst_or rt->rt_genid = rt_genid_ipv4(net); rt->rt_flags = ort->rt_flags; rt->rt_type = ort->rt_type; - rt->rt_gateway = ort->rt_gateway; - rt->rt_uses_gateway = ort->rt_uses_gateway; + rt->rt_gw_family = ort->rt_gw_family; + if (rt->rt_gw_family == AF_INET) + rt->rt_gw4 = ort->rt_gw4; + else if (rt->rt_gw_family == AF_INET6) + rt->rt_gw6 = ort->rt_gw6; INIT_LIST_HEAD(&rt->rt_uncached); } @@ -2680,9 +2718,22 @@ static int rt_fill_info(struct net *net, __be32 dst, __be32 src, if (nla_put_in_addr(skb, RTA_PREFSRC, fl4->saddr)) goto nla_put_failure; } - if (rt->rt_uses_gateway && - nla_put_in_addr(skb, RTA_GATEWAY, rt->rt_gateway)) + if (rt->rt_gw_family == AF_INET && + nla_put_in_addr(skb, RTA_GATEWAY, rt->rt_gw4)) { goto nla_put_failure; + } else if (rt->rt_gw_family == AF_INET6) { + int alen = sizeof(struct in6_addr); + struct nlattr *nla; + struct rtvia *via; + + nla = nla_reserve(skb, RTA_VIA, alen + 2); + if (!nla) + goto nla_put_failure; + + via = nla_data(nla); + via->rtvia_family = AF_INET6; + memcpy(via->rtvia_addr, &rt->rt_gw6, alen); + } expires = rt->dst.expires; if (expires) { @@ -2819,8 +2870,8 @@ static int inet_rtm_valid_getroute_req(struct sk_buff *skb, } if (!netlink_strict_get_check(skb)) - return nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, - rtm_ipv4_policy, extack); + return nlmsg_parse_deprecated(nlh, sizeof(*rtm), tb, RTA_MAX, + rtm_ipv4_policy, extack); rtm = nlmsg_data(nlh); if ((rtm->rtm_src_len && rtm->rtm_src_len != 32) || @@ -2838,8 +2889,8 @@ static int inet_rtm_valid_getroute_req(struct sk_buff *skb, return -EINVAL; } - err = nlmsg_parse_strict(nlh, sizeof(*rtm), tb, RTA_MAX, - rtm_ipv4_policy, extack); + err = nlmsg_parse_deprecated_strict(nlh, sizeof(*rtm), tb, RTA_MAX, + rtm_ipv4_policy, extack); if (err) return err; diff --git a/net/ipv4/syncookies.c b/net/ipv4/syncookies.c index e531344611a01acf1edbfc01a91f9797dda95831..008545f63667e017fada429d6055b67c88acd985 100644 --- a/net/ipv4/syncookies.c +++ b/net/ipv4/syncookies.c @@ -216,16 +216,15 @@ struct sock *tcp_get_cookie_sock(struct sock *sk, struct sk_buff *skb, refcount_set(&req->rsk_refcnt, 1); tcp_sk(child)->tsoffset = tsoff; sock_rps_save_rxhash(child, skb); - if (!inet_csk_reqsk_queue_add(sk, req, child)) { - bh_unlock_sock(child); - sock_put(child); - child = NULL; - reqsk_put(req); - } - } else { - reqsk_free(req); + if (inet_csk_reqsk_queue_add(sk, req, child)) + return child; + + bh_unlock_sock(child); + sock_put(child); } - return child; + __reqsk_free(req); + + return NULL; } EXPORT_SYMBOL(tcp_get_cookie_sock); diff --git a/net/ipv4/sysctl_net_ipv4.c b/net/ipv4/sysctl_net_ipv4.c index eeb4041fa5f905fb0f7c91ea6d74851ae97259f8..875867b64d6a6597bf4fcd3498ed55741cbe33f7 100644 --- a/net/ipv4/sysctl_net_ipv4.c +++ b/net/ipv4/sysctl_net_ipv4.c @@ -550,6 +550,15 @@ static struct ctl_table ipv4_table[] = { .mode = 0644, .proc_handler = proc_doulongvec_minmax, }, + { + .procname = "fib_sync_mem", + .data = &sysctl_fib_sync_mem, + .maxlen = sizeof(sysctl_fib_sync_mem), + .mode = 0644, + .proc_handler = proc_douintvec_minmax, + .extra1 = &sysctl_fib_sync_mem_min, + .extra2 = &sysctl_fib_sync_mem_max, + }, { } }; diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 6baa6dc1b13b0b94b1da238668b93e167cf444fe..1fa15beb83806696a6b4f5eddb7e45c13a2dea45 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -457,18 +457,6 @@ void tcp_init_sock(struct sock *sk) } EXPORT_SYMBOL(tcp_init_sock); -void tcp_init_transfer(struct sock *sk, int bpf_op) -{ - struct inet_connection_sock *icsk = inet_csk(sk); - - tcp_mtup_init(sk); - icsk->icsk_af_ops->rebuild_header(sk); - tcp_init_metrics(sk); - tcp_call_bpf(sk, bpf_op, 0, NULL); - tcp_init_congestion_control(sk); - tcp_init_buffer_space(sk); -} - static void tcp_tx_timestamp(struct sock *sk, u16 tsflags) { struct sk_buff *skb = tcp_write_queue_tail(sk); @@ -865,6 +853,18 @@ struct sk_buff *sk_stream_alloc_skb(struct sock *sk, int size, gfp_t gfp, { struct sk_buff *skb; + if (likely(!size)) { + skb = sk->sk_tx_skb_cache; + if (skb && !skb_cloned(skb)) { + skb->truesize = SKB_TRUESIZE(skb_end_offset(skb)); + sk->sk_tx_skb_cache = NULL; + pskb_trim(skb, 0); + INIT_LIST_HEAD(&skb->tcp_tsorted_anchor); + skb_shinfo(skb)->tx_flags = 0; + memset(TCP_SKB_CB(skb), 0, sizeof(struct tcp_skb_cb)); + return skb; + } + } /* The TCP header must be at least 32-bit aligned. */ size = ALIGN(size, 4); @@ -1098,30 +1098,6 @@ int tcp_sendpage(struct sock *sk, struct page *page, int offset, } EXPORT_SYMBOL(tcp_sendpage); -/* Do not bother using a page frag for very small frames. - * But use this heuristic only for the first skb in write queue. - * - * Having no payload in skb->head allows better SACK shifting - * in tcp_shift_skb_data(), reducing sack/rack overhead, because - * write queue has less skbs. - * Each skb can hold up to MAX_SKB_FRAGS * 32Kbytes, or ~0.5 MB. - * This also speeds up tso_fragment(), since it wont fallback - * to tcp_fragment(). - */ -static int linear_payload_sz(bool first_skb) -{ - if (first_skb) - return SKB_WITH_OVERHEAD(2048 - MAX_TCP_HEADER); - return 0; -} - -static int select_size(bool first_skb, bool zc) -{ - if (zc) - return 0; - return linear_payload_sz(first_skb); -} - void tcp_free_fastopen_req(struct tcp_sock *tp) { if (tp->fastopen_req) { @@ -1272,7 +1248,6 @@ restart: if (copy <= 0 || !tcp_skb_can_collapse_to(skb)) { bool first_skb; - int linear; new_segment: if (!sk_stream_memory_free(sk)) @@ -1283,8 +1258,7 @@ new_segment: goto restart; } first_skb = tcp_rtx_and_write_queues_empty(sk); - linear = select_size(first_skb, zc); - skb = sk_stream_alloc_skb(sk, linear, sk->sk_allocation, + skb = sk_stream_alloc_skb(sk, 0, sk->sk_allocation, first_skb); if (!skb) goto wait_for_memory; @@ -2552,6 +2526,11 @@ void tcp_write_queue_purge(struct sock *sk) sk_wmem_free_skb(sk, skb); } tcp_rtx_queue_purge(sk); + skb = sk->sk_tx_skb_cache; + if (skb) { + __kfree_skb(skb); + sk->sk_tx_skb_cache = NULL; + } INIT_LIST_HEAD(&tcp_sk(sk)->tsorted_sent_queue); sk_mem_reclaim(sk); tcp_clear_all_retrans_hints(tcp_sk(sk)); @@ -2587,6 +2566,10 @@ int tcp_disconnect(struct sock *sk, int flags) tcp_clear_xmit_timers(sk); __skb_queue_purge(&sk->sk_receive_queue); + if (sk->sk_rx_skb_cache) { + __kfree_skb(sk->sk_rx_skb_cache); + sk->sk_rx_skb_cache = NULL; + } tp->copied_seq = tp->rcv_nxt; tp->urg_data = 0; tcp_write_queue_purge(sk); diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 731d3045b50a0fb9a89c887a154db9a3da8c7ddd..077d9abdfcf503b39c6d7e15fdec18ce408c4119 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -2252,7 +2252,7 @@ static bool tcp_skb_spurious_retrans(const struct tcp_sock *tp, */ static inline bool tcp_packet_delayed(const struct tcp_sock *tp) { - return !tp->retrans_stamp || + return tp->retrans_stamp && tcp_tsopt_ecr_before(tp, tp->retrans_stamp); } @@ -3521,7 +3521,7 @@ static void tcp_xmit_recovery(struct sock *sk, int rexmit) { struct tcp_sock *tp = tcp_sk(sk); - if (rexmit == REXMIT_NONE) + if (rexmit == REXMIT_NONE || sk->sk_state == TCP_SYN_SENT) return; if (unlikely(rexmit == 2)) { @@ -5647,6 +5647,32 @@ discard: } EXPORT_SYMBOL(tcp_rcv_established); +void tcp_init_transfer(struct sock *sk, int bpf_op) +{ + struct inet_connection_sock *icsk = inet_csk(sk); + struct tcp_sock *tp = tcp_sk(sk); + + tcp_mtup_init(sk); + icsk->icsk_af_ops->rebuild_header(sk); + tcp_init_metrics(sk); + + /* Initialize the congestion window to start the transfer. + * Cut cwnd down to 1 per RFC5681 if SYN or SYN-ACK has been + * retransmitted. In light of RFC6298 more aggressive 1sec + * initRTO, we only reset cwnd when more than 1 SYN/SYN-ACK + * retransmission has occurred. + */ + if (tp->total_retrans > 1 && tp->undo_marker) + tp->snd_cwnd = 1; + else + tp->snd_cwnd = tcp_init_cwnd(tp, __sk_dst_get(sk)); + tp->snd_cwnd_stamp = tcp_jiffies32; + + tcp_call_bpf(sk, bpf_op, 0, NULL); + tcp_init_congestion_control(sk); + tcp_init_buffer_space(sk); +} + void tcp_finish_connect(struct sock *sk, struct sk_buff *skb) { struct tcp_sock *tp = tcp_sk(sk); @@ -5748,6 +5774,21 @@ static void smc_check_reset_syn(struct tcp_sock *tp) #endif } +static void tcp_try_undo_spurious_syn(struct sock *sk) +{ + struct tcp_sock *tp = tcp_sk(sk); + u32 syn_stamp; + + /* undo_marker is set when SYN or SYNACK times out. The timeout is + * spurious if the ACK's timestamp option echo value matches the + * original SYN timestamp. + */ + syn_stamp = tp->retrans_stamp; + if (tp->undo_marker && syn_stamp && tp->rx_opt.saw_tstamp && + syn_stamp == tp->rx_opt.rcv_tsecr) + tp->undo_marker = 0; +} + static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb, const struct tcphdr *th) { @@ -5815,6 +5856,7 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb, tcp_ecn_rcv_synack(tp, th); tcp_init_wl(tp, TCP_SKB_CB(skb)->seq); + tcp_try_undo_spurious_syn(sk); tcp_ack(sk, skb, FLAG_SLOWPATH); /* Ok.. it's good. Set up sequence numbers and @@ -5973,6 +6015,27 @@ reset_and_undo: return 1; } +static void tcp_rcv_synrecv_state_fastopen(struct sock *sk) +{ + tcp_try_undo_loss(sk, false); + inet_csk(sk)->icsk_retransmits = 0; + + /* Once we leave TCP_SYN_RECV or TCP_FIN_WAIT_1, + * we no longer need req so release it. + */ + reqsk_fastopen_remove(sk, tcp_sk(sk)->fastopen_rsk, false); + + /* Re-arm the timer because data may have been sent out. + * This is similar to the regular data transmission case + * when new data has just been ack'ed. + * + * (TFO) - we could try to be more aggressive and + * retransmitting any data sooner based on when they + * are sent out. + */ + tcp_rearm_rto(sk); +} + /* * This function implements the receiving procedure of RFC 793 for * all states except ESTABLISHED and TIME_WAIT. @@ -6069,22 +6132,11 @@ int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb) if (!tp->srtt_us) tcp_synack_rtt_meas(sk, req); - /* Once we leave TCP_SYN_RECV, we no longer need req - * so release it. - */ if (req) { - inet_csk(sk)->icsk_retransmits = 0; - reqsk_fastopen_remove(sk, req, false); - /* Re-arm the timer because data may have been sent out. - * This is similar to the regular data transmission case - * when new data has just been ack'ed. - * - * (TFO) - we could try to be more aggressive and - * retransmitting any data sooner based on when they - * are sent out. - */ - tcp_rearm_rto(sk); + tcp_rcv_synrecv_state_fastopen(sk); } else { + tcp_try_undo_spurious_syn(sk); + tp->retrans_stamp = 0; tcp_init_transfer(sk, BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB); tp->copied_seq = tp->rcv_nxt; } @@ -6119,16 +6171,9 @@ int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb) case TCP_FIN_WAIT1: { int tmo; - /* If we enter the TCP_FIN_WAIT1 state and we are a - * Fast Open socket and this is the first acceptable - * ACK we have received, this would have acknowledged - * our SYNACK so stop the SYNACK timer. - */ - if (req) { - /* We no longer need the request sock. */ - reqsk_fastopen_remove(sk, req, false); - tcp_rearm_rto(sk); - } + if (req) + tcp_rcv_synrecv_state_fastopen(sk); + if (tp->snd_una != tp->write_seq) break; @@ -6263,6 +6308,11 @@ static inline void pr_drop_req(struct request_sock *req, __u16 port, int family) * congestion control: Linux DCTCP asserts ECT on all packets, * including SYN, which is most optimal solution; however, * others, such as FreeBSD do not. + * + * Exception: At least one of the reserved bits of the TCP header (th->res1) is + * set, indicating the use of a future TCP extension (such as AccECN). See + * RFC8311 §4.3 which updates RFC3168 to allow the development of such + * extensions. */ static void tcp_ecn_create_request(struct request_sock *req, const struct sk_buff *skb, @@ -6282,7 +6332,7 @@ static void tcp_ecn_create_request(struct request_sock *req, ecn_ok_dst = dst_feature(dst, DST_FEATURE_ECN_MASK); ecn_ok = net->ipv4.sysctl_tcp_ecn || ecn_ok_dst; - if ((!ect && ecn_ok) || tcp_ca_needs_ecn(listen_sk) || + if (((!ect || th->res1) && ecn_ok) || tcp_ca_needs_ecn(listen_sk) || (ecn_ok_dst & DST_FEATURE_ECN_CA) || tcp_bpf_ca_needs_ecn((struct sock *)req)) inet_rsk(req)->ecn_ok = 1; @@ -6298,7 +6348,7 @@ static void tcp_openreq_init(struct request_sock *req, req->cookie_ts = 0; tcp_rsk(req)->rcv_isn = TCP_SKB_CB(skb)->seq; tcp_rsk(req)->rcv_nxt = TCP_SKB_CB(skb)->seq + 1; - tcp_rsk(req)->snt_synack = tcp_clock_us(); + tcp_rsk(req)->snt_synack = 0; tcp_rsk(req)->last_oow_ack_time = 0; req->mss = rx_opt->mss_clamp; req->ts_recent = rx_opt->saw_tstamp ? rx_opt->rcv_tsval : 0; @@ -6502,8 +6552,7 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops, reqsk_fastopen_remove(fastopen_sk, req, false); bh_unlock_sock(fastopen_sk); sock_put(fastopen_sk); - reqsk_put(req); - goto drop; + goto drop_and_free; } sk->sk_data_ready(sk); bh_unlock_sock(fastopen_sk); @@ -6527,7 +6576,7 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops, drop_and_release: dst_release(dst); drop_and_free: - reqsk_free(req); + __reqsk_free(req); drop: tcp_listendrop(sk); return 0; diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index a2896944aa377b7feef6417720348c02c3d8eecb..af81e4a6a8d8eac9aad551a129384ff6b1bf2f6c 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -1785,6 +1785,7 @@ static void tcp_v4_fill_cb(struct sk_buff *skb, const struct iphdr *iph, int tcp_v4_rcv(struct sk_buff *skb) { struct net *net = dev_net(skb->dev); + struct sk_buff *skb_to_free; int sdif = inet_sdif(skb); const struct iphdr *iph; const struct tcphdr *th; @@ -1916,11 +1917,17 @@ process: tcp_segs_in(tcp_sk(sk), skb); ret = 0; if (!sock_owned_by_user(sk)) { + skb_to_free = sk->sk_rx_skb_cache; + sk->sk_rx_skb_cache = NULL; ret = tcp_v4_do_rcv(sk, skb); - } else if (tcp_add_backlog(sk, skb)) { - goto discard_and_relse; + } else { + if (tcp_add_backlog(sk, skb)) + goto discard_and_relse; + skb_to_free = NULL; } bh_unlock_sock(sk); + if (skb_to_free) + __kfree_skb(skb_to_free); put_and_return: if (refcounted) diff --git a/net/ipv4/tcp_metrics.c b/net/ipv4/tcp_metrics.c index b467a7cabf40e6af2ce69649f3ca48635bf06319..c4848e7a0aad123bcb6589be6247fc5007796699 100644 --- a/net/ipv4/tcp_metrics.c +++ b/net/ipv4/tcp_metrics.c @@ -512,16 +512,6 @@ reset: inet_csk(sk)->icsk_rto = TCP_TIMEOUT_FALLBACK; } - /* Cut cwnd down to 1 per RFC5681 if SYN or SYN-ACK has been - * retransmitted. In light of RFC6298 more aggressive 1sec - * initRTO, we only reset cwnd when more than 1 SYN/SYN-ACK - * retransmission has occurred. - */ - if (tp->total_retrans > 1) - tp->snd_cwnd = 1; - else - tp->snd_cwnd = tcp_init_cwnd(tp, dst); - tp->snd_cwnd_stamp = tcp_jiffies32; } bool tcp_peer_is_proven(struct request_sock *req, struct dst_entry *dst) @@ -658,7 +648,7 @@ static int tcp_metrics_fill_info(struct sk_buff *msg, { int n = 0; - nest = nla_nest_start(msg, TCP_METRICS_ATTR_VALS); + nest = nla_nest_start_noflag(msg, TCP_METRICS_ATTR_VALS); if (!nest) goto nla_put_failure; for (i = 0; i < TCP_METRIC_MAX_KERNEL + 1; i++) { @@ -951,14 +941,14 @@ static int tcp_metrics_nl_cmd_del(struct sk_buff *skb, struct genl_info *info) static const struct genl_ops tcp_metrics_nl_ops[] = { { .cmd = TCP_METRICS_CMD_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = tcp_metrics_nl_cmd_get, .dumpit = tcp_metrics_nl_dump, - .policy = tcp_metrics_nl_policy, }, { .cmd = TCP_METRICS_CMD_DEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = tcp_metrics_nl_cmd_del, - .policy = tcp_metrics_nl_policy, .flags = GENL_ADMIN_PERM, }, }; @@ -968,6 +958,7 @@ static struct genl_family tcp_metrics_nl_family __ro_after_init = { .name = TCP_METRICS_GENL_NAME, .version = TCP_METRICS_GENL_VERSION, .maxattr = TCP_METRICS_ATTR_MAX, + .policy = tcp_metrics_nl_policy, .netnsok = true, .module = THIS_MODULE, .ops = tcp_metrics_nl_ops, diff --git a/net/ipv4/tcp_minisocks.c b/net/ipv4/tcp_minisocks.c index 79900f783e0d8a713712b0bf47700f16eebab4db..9c2a0d36fb200b04885cf90493274659640d561b 100644 --- a/net/ipv4/tcp_minisocks.c +++ b/net/ipv4/tcp_minisocks.c @@ -522,6 +522,11 @@ struct sock *tcp_create_openreq_child(const struct sock *sk, newtp->rx_opt.ts_recent_stamp = 0; newtp->tcp_header_len = sizeof(struct tcphdr); } + if (req->num_timeout) { + newtp->undo_marker = treq->snt_isn; + newtp->retrans_stamp = div_u64(treq->snt_synack, + USEC_PER_SEC / TCP_TS_HZ); + } newtp->tsoffset = treq->ts_off; #ifdef CONFIG_TCP_MD5SIG newtp->md5sig_info = NULL; /*XXX*/ diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index 4522579aaca2d24b7be61abac658a3c8cbd2b960..0c4ed66dc1bf3441a9b7e5c9c37692333c92f17c 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -52,12 +52,8 @@ void tcp_mstamp_refresh(struct tcp_sock *tp) { u64 val = tcp_clock_ns(); - if (val > tp->tcp_clock_cache) - tp->tcp_clock_cache = val; - - val = div_u64(val, NSEC_PER_USEC); - if (val > tp->tcp_mstamp) - tp->tcp_mstamp = val; + tp->tcp_clock_cache = val; + tp->tcp_mstamp = div_u64(val, NSEC_PER_USEC); } static bool tcp_write_xmit(struct sock *sk, unsigned int mss_now, int nonagle, @@ -3092,7 +3088,6 @@ void tcp_send_fin(struct sock *sk) tskb = skb_rb_last(&sk->tcp_rtx_queue); if (tskb) { -coalesce: TCP_SKB_CB(tskb)->tcp_flags |= TCPHDR_FIN; TCP_SKB_CB(tskb)->end_seq++; tp->write_seq++; @@ -3108,11 +3103,9 @@ coalesce: } } else { skb = alloc_skb_fclone(MAX_TCP_HEADER, sk->sk_allocation); - if (unlikely(!skb)) { - if (tskb) - goto coalesce; + if (unlikely(!skb)) return; - } + INIT_LIST_HEAD(&skb->tcp_tsorted_anchor); skb_reserve(skb, MAX_TCP_HEADER); sk_forced_mem_schedule(sk, skb->truesize); @@ -3254,7 +3247,11 @@ struct sk_buff *tcp_make_synack(const struct sock *sk, struct dst_entry *dst, skb->skb_mstamp_ns = cookie_init_timestamp(req); else #endif + { skb->skb_mstamp_ns = tcp_clock_ns(); + if (!tcp_rsk(req)->snt_synack) /* Timestamp first SYNACK */ + tcp_rsk(req)->snt_synack = tcp_skb_timestamp_us(skb); + } #ifdef CONFIG_TCP_MD5SIG rcu_read_lock(); diff --git a/net/ipv4/tcp_timer.c b/net/ipv4/tcp_timer.c index f0c86398e6a7285bfb9393b516a149b3202a652c..2ac23da42dd277b4c360307781f53594bd9a0c10 100644 --- a/net/ipv4/tcp_timer.c +++ b/net/ipv4/tcp_timer.c @@ -393,6 +393,9 @@ static void tcp_fastopen_synack_timer(struct sock *sk) tcp_write_err(sk); return; } + /* Lower cwnd after certain SYNACK timeout like tcp_init_transfer() */ + if (icsk->icsk_retransmits == 1) + tcp_enter_loss(sk); /* XXX (TFO) - Unlike regular SYN-ACK retransmit, we ignore error * returned from rtx_syn_ack() to make it more persistent like * regular retransmit because if the child socket has been accepted diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index 372fdc5381a98e0d8a673ef1649323f91764ad8e..3c58ba02af7dec49f454d6998a3436b46c248084 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -1631,7 +1631,7 @@ int udp_ioctl(struct sock *sk, int cmd, unsigned long arg) EXPORT_SYMBOL(udp_ioctl); struct sk_buff *__skb_recv_udp(struct sock *sk, unsigned int flags, - int noblock, int *peeked, int *off, int *err) + int noblock, int *off, int *err) { struct sk_buff_head *sk_queue = &sk->sk_receive_queue; struct sk_buff_head *queue; @@ -1650,13 +1650,11 @@ struct sk_buff *__skb_recv_udp(struct sock *sk, unsigned int flags, break; error = -EAGAIN; - *peeked = 0; do { spin_lock_bh(&queue->lock); skb = __skb_try_recv_from_queue(sk, queue, flags, udp_skb_destructor, - peeked, off, err, - &last); + off, err, &last); if (skb) { spin_unlock_bh(&queue->lock); return skb; @@ -1677,8 +1675,7 @@ struct sk_buff *__skb_recv_udp(struct sock *sk, unsigned int flags, skb = __skb_try_recv_from_queue(sk, queue, flags, udp_skb_dtor_locked, - peeked, off, err, - &last); + off, err, &last); spin_unlock(&sk_queue->lock); spin_unlock_bh(&queue->lock); if (skb) @@ -1713,8 +1710,7 @@ int udp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int noblock, DECLARE_SOCKADDR(struct sockaddr_in *, sin, msg->msg_name); struct sk_buff *skb; unsigned int ulen, copied; - int peeked, peeking, off; - int err; + int off, err, peeking = flags & MSG_PEEK; int is_udplite = IS_UDPLITE(sk); bool checksum_valid = false; @@ -1722,9 +1718,8 @@ int udp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int noblock, return ip_recv_error(sk, msg, len, addr_len); try_again: - peeking = flags & MSG_PEEK; off = sk_peek_offset(sk, flags); - skb = __skb_recv_udp(sk, flags, noblock, &peeked, &off, &err); + skb = __skb_recv_udp(sk, flags, noblock, &off, &err); if (!skb) return err; @@ -1762,7 +1757,7 @@ try_again: } if (unlikely(err)) { - if (!peeked) { + if (!peeking) { atomic_inc(&sk->sk_drops); UDP_INC_STATS(sock_net(sk), UDP_MIB_INERRORS, is_udplite); @@ -1771,7 +1766,7 @@ try_again: return err; } - if (!peeked) + if (!peeking) UDP_INC_STATS(sock_net(sk), UDP_MIB_INDATAGRAMS, is_udplite); diff --git a/net/ipv4/xfrm4_mode_beet.c b/net/ipv4/xfrm4_mode_beet.c deleted file mode 100644 index 856d2dfdb44b6be0d9d9be732d1e26f7db69792c..0000000000000000000000000000000000000000 --- a/net/ipv4/xfrm4_mode_beet.c +++ /dev/null @@ -1,155 +0,0 @@ -/* - * xfrm4_mode_beet.c - BEET mode encapsulation for IPv4. - * - * Copyright (c) 2006 Diego Beltrami - * Miika Komu - * Herbert Xu - * Abhinav Pathak - * Jeff Ahrenholz - */ - -#include -#include -#include -#include -#include -#include -#include -#include - -static void xfrm4_beet_make_header(struct sk_buff *skb) -{ - struct iphdr *iph = ip_hdr(skb); - - iph->ihl = 5; - iph->version = 4; - - iph->protocol = XFRM_MODE_SKB_CB(skb)->protocol; - iph->tos = XFRM_MODE_SKB_CB(skb)->tos; - - iph->id = XFRM_MODE_SKB_CB(skb)->id; - iph->frag_off = XFRM_MODE_SKB_CB(skb)->frag_off; - iph->ttl = XFRM_MODE_SKB_CB(skb)->ttl; -} - -/* Add encapsulation header. - * - * The top IP header will be constructed per draft-nikander-esp-beet-mode-06.txt. - */ -static int xfrm4_beet_output(struct xfrm_state *x, struct sk_buff *skb) -{ - struct ip_beet_phdr *ph; - struct iphdr *top_iph; - int hdrlen, optlen; - - hdrlen = 0; - optlen = XFRM_MODE_SKB_CB(skb)->optlen; - if (unlikely(optlen)) - hdrlen += IPV4_BEET_PHMAXLEN - (optlen & 4); - - skb_set_network_header(skb, -x->props.header_len - - hdrlen + (XFRM_MODE_SKB_CB(skb)->ihl - sizeof(*top_iph))); - if (x->sel.family != AF_INET6) - skb->network_header += IPV4_BEET_PHMAXLEN; - skb->mac_header = skb->network_header + - offsetof(struct iphdr, protocol); - skb->transport_header = skb->network_header + sizeof(*top_iph); - - xfrm4_beet_make_header(skb); - - ph = __skb_pull(skb, XFRM_MODE_SKB_CB(skb)->ihl - hdrlen); - - top_iph = ip_hdr(skb); - - if (unlikely(optlen)) { - BUG_ON(optlen < 0); - - ph->padlen = 4 - (optlen & 4); - ph->hdrlen = optlen / 8; - ph->nexthdr = top_iph->protocol; - if (ph->padlen) - memset(ph + 1, IPOPT_NOP, ph->padlen); - - top_iph->protocol = IPPROTO_BEETPH; - top_iph->ihl = sizeof(struct iphdr) / 4; - } - - top_iph->saddr = x->props.saddr.a4; - top_iph->daddr = x->id.daddr.a4; - - return 0; -} - -static int xfrm4_beet_input(struct xfrm_state *x, struct sk_buff *skb) -{ - struct iphdr *iph; - int optlen = 0; - int err = -EINVAL; - - if (unlikely(XFRM_MODE_SKB_CB(skb)->protocol == IPPROTO_BEETPH)) { - struct ip_beet_phdr *ph; - int phlen; - - if (!pskb_may_pull(skb, sizeof(*ph))) - goto out; - - ph = (struct ip_beet_phdr *)skb->data; - - phlen = sizeof(*ph) + ph->padlen; - optlen = ph->hdrlen * 8 + (IPV4_BEET_PHMAXLEN - phlen); - if (optlen < 0 || optlen & 3 || optlen > 250) - goto out; - - XFRM_MODE_SKB_CB(skb)->protocol = ph->nexthdr; - - if (!pskb_may_pull(skb, phlen)) - goto out; - __skb_pull(skb, phlen); - } - - skb_push(skb, sizeof(*iph)); - skb_reset_network_header(skb); - skb_mac_header_rebuild(skb); - - xfrm4_beet_make_header(skb); - - iph = ip_hdr(skb); - - iph->ihl += optlen / 4; - iph->tot_len = htons(skb->len); - iph->daddr = x->sel.daddr.a4; - iph->saddr = x->sel.saddr.a4; - iph->check = 0; - iph->check = ip_fast_csum(skb_network_header(skb), iph->ihl); - err = 0; -out: - return err; -} - -static struct xfrm_mode xfrm4_beet_mode = { - .input2 = xfrm4_beet_input, - .input = xfrm_prepare_input, - .output2 = xfrm4_beet_output, - .output = xfrm4_prepare_output, - .owner = THIS_MODULE, - .encap = XFRM_MODE_BEET, - .flags = XFRM_MODE_FLAG_TUNNEL, -}; - -static int __init xfrm4_beet_init(void) -{ - return xfrm_register_mode(&xfrm4_beet_mode, AF_INET); -} - -static void __exit xfrm4_beet_exit(void) -{ - int err; - - err = xfrm_unregister_mode(&xfrm4_beet_mode, AF_INET); - BUG_ON(err); -} - -module_init(xfrm4_beet_init); -module_exit(xfrm4_beet_exit); -MODULE_LICENSE("GPL"); -MODULE_ALIAS_XFRM_MODE(AF_INET, XFRM_MODE_BEET); diff --git a/net/ipv4/xfrm4_mode_transport.c b/net/ipv4/xfrm4_mode_transport.c deleted file mode 100644 index 1ad2c2c4e250f84b1ad73020c727ea8b68b3e0d3..0000000000000000000000000000000000000000 --- a/net/ipv4/xfrm4_mode_transport.c +++ /dev/null @@ -1,114 +0,0 @@ -/* - * xfrm4_mode_transport.c - Transport mode encapsulation for IPv4. - * - * Copyright (c) 2004-2006 Herbert Xu - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include - -/* Add encapsulation header. - * - * The IP header will be moved forward to make space for the encapsulation - * header. - */ -static int xfrm4_transport_output(struct xfrm_state *x, struct sk_buff *skb) -{ - struct iphdr *iph = ip_hdr(skb); - int ihl = iph->ihl * 4; - - skb_set_inner_transport_header(skb, skb_transport_offset(skb)); - - skb_set_network_header(skb, -x->props.header_len); - skb->mac_header = skb->network_header + - offsetof(struct iphdr, protocol); - skb->transport_header = skb->network_header + ihl; - __skb_pull(skb, ihl); - memmove(skb_network_header(skb), iph, ihl); - return 0; -} - -/* Remove encapsulation header. - * - * The IP header will be moved over the top of the encapsulation header. - * - * On entry, skb->h shall point to where the IP header should be and skb->nh - * shall be set to where the IP header currently is. skb->data shall point - * to the start of the payload. - */ -static int xfrm4_transport_input(struct xfrm_state *x, struct sk_buff *skb) -{ - int ihl = skb->data - skb_transport_header(skb); - - if (skb->transport_header != skb->network_header) { - memmove(skb_transport_header(skb), - skb_network_header(skb), ihl); - skb->network_header = skb->transport_header; - } - ip_hdr(skb)->tot_len = htons(skb->len + ihl); - skb_reset_transport_header(skb); - return 0; -} - -static struct sk_buff *xfrm4_transport_gso_segment(struct xfrm_state *x, - struct sk_buff *skb, - netdev_features_t features) -{ - const struct net_offload *ops; - struct sk_buff *segs = ERR_PTR(-EINVAL); - struct xfrm_offload *xo = xfrm_offload(skb); - - skb->transport_header += x->props.header_len; - ops = rcu_dereference(inet_offloads[xo->proto]); - if (likely(ops && ops->callbacks.gso_segment)) - segs = ops->callbacks.gso_segment(skb, features); - - return segs; -} - -static void xfrm4_transport_xmit(struct xfrm_state *x, struct sk_buff *skb) -{ - struct xfrm_offload *xo = xfrm_offload(skb); - - skb_reset_mac_len(skb); - pskb_pull(skb, skb->mac_len + sizeof(struct iphdr) + x->props.header_len); - - if (xo->flags & XFRM_GSO_SEGMENT) { - skb_reset_transport_header(skb); - skb->transport_header -= x->props.header_len; - } -} - -static struct xfrm_mode xfrm4_transport_mode = { - .input = xfrm4_transport_input, - .output = xfrm4_transport_output, - .gso_segment = xfrm4_transport_gso_segment, - .xmit = xfrm4_transport_xmit, - .owner = THIS_MODULE, - .encap = XFRM_MODE_TRANSPORT, -}; - -static int __init xfrm4_transport_init(void) -{ - return xfrm_register_mode(&xfrm4_transport_mode, AF_INET); -} - -static void __exit xfrm4_transport_exit(void) -{ - int err; - - err = xfrm_unregister_mode(&xfrm4_transport_mode, AF_INET); - BUG_ON(err); -} - -module_init(xfrm4_transport_init); -module_exit(xfrm4_transport_exit); -MODULE_LICENSE("GPL"); -MODULE_ALIAS_XFRM_MODE(AF_INET, XFRM_MODE_TRANSPORT); diff --git a/net/ipv4/xfrm4_mode_tunnel.c b/net/ipv4/xfrm4_mode_tunnel.c deleted file mode 100644 index 2a9764bd17196966b41755269e03f487cb757288..0000000000000000000000000000000000000000 --- a/net/ipv4/xfrm4_mode_tunnel.c +++ /dev/null @@ -1,152 +0,0 @@ -/* - * xfrm4_mode_tunnel.c - Tunnel mode encapsulation for IPv4. - * - * Copyright (c) 2004-2006 Herbert Xu - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -static inline void ipip_ecn_decapsulate(struct sk_buff *skb) -{ - struct iphdr *inner_iph = ipip_hdr(skb); - - if (INET_ECN_is_ce(XFRM_MODE_SKB_CB(skb)->tos)) - IP_ECN_set_ce(inner_iph); -} - -/* Add encapsulation header. - * - * The top IP header will be constructed per RFC 2401. - */ -static int xfrm4_mode_tunnel_output(struct xfrm_state *x, struct sk_buff *skb) -{ - struct dst_entry *dst = skb_dst(skb); - struct iphdr *top_iph; - int flags; - - skb_set_inner_network_header(skb, skb_network_offset(skb)); - skb_set_inner_transport_header(skb, skb_transport_offset(skb)); - - skb_set_network_header(skb, -x->props.header_len); - skb->mac_header = skb->network_header + - offsetof(struct iphdr, protocol); - skb->transport_header = skb->network_header + sizeof(*top_iph); - top_iph = ip_hdr(skb); - - top_iph->ihl = 5; - top_iph->version = 4; - - top_iph->protocol = xfrm_af2proto(skb_dst(skb)->ops->family); - - /* DS disclosing depends on XFRM_SA_XFLAG_DONT_ENCAP_DSCP */ - if (x->props.extra_flags & XFRM_SA_XFLAG_DONT_ENCAP_DSCP) - top_iph->tos = 0; - else - top_iph->tos = XFRM_MODE_SKB_CB(skb)->tos; - top_iph->tos = INET_ECN_encapsulate(top_iph->tos, - XFRM_MODE_SKB_CB(skb)->tos); - - flags = x->props.flags; - if (flags & XFRM_STATE_NOECN) - IP_ECN_clear(top_iph); - - top_iph->frag_off = (flags & XFRM_STATE_NOPMTUDISC) ? - 0 : (XFRM_MODE_SKB_CB(skb)->frag_off & htons(IP_DF)); - - top_iph->ttl = ip4_dst_hoplimit(xfrm_dst_child(dst)); - - top_iph->saddr = x->props.saddr.a4; - top_iph->daddr = x->id.daddr.a4; - ip_select_ident(dev_net(dst->dev), skb, NULL); - - return 0; -} - -static int xfrm4_mode_tunnel_input(struct xfrm_state *x, struct sk_buff *skb) -{ - int err = -EINVAL; - - if (XFRM_MODE_SKB_CB(skb)->protocol != IPPROTO_IPIP) - goto out; - - if (!pskb_may_pull(skb, sizeof(struct iphdr))) - goto out; - - err = skb_unclone(skb, GFP_ATOMIC); - if (err) - goto out; - - if (x->props.flags & XFRM_STATE_DECAP_DSCP) - ipv4_copy_dscp(XFRM_MODE_SKB_CB(skb)->tos, ipip_hdr(skb)); - if (!(x->props.flags & XFRM_STATE_NOECN)) - ipip_ecn_decapsulate(skb); - - skb_reset_network_header(skb); - skb_mac_header_rebuild(skb); - if (skb->mac_len) - eth_hdr(skb)->h_proto = skb->protocol; - - err = 0; - -out: - return err; -} - -static struct sk_buff *xfrm4_mode_tunnel_gso_segment(struct xfrm_state *x, - struct sk_buff *skb, - netdev_features_t features) -{ - __skb_push(skb, skb->mac_len); - return skb_mac_gso_segment(skb, features); -} - -static void xfrm4_mode_tunnel_xmit(struct xfrm_state *x, struct sk_buff *skb) -{ - struct xfrm_offload *xo = xfrm_offload(skb); - - if (xo->flags & XFRM_GSO_SEGMENT) - skb->transport_header = skb->network_header + - sizeof(struct iphdr); - - skb_reset_mac_len(skb); - pskb_pull(skb, skb->mac_len + x->props.header_len); -} - -static struct xfrm_mode xfrm4_tunnel_mode = { - .input2 = xfrm4_mode_tunnel_input, - .input = xfrm_prepare_input, - .output2 = xfrm4_mode_tunnel_output, - .output = xfrm4_prepare_output, - .gso_segment = xfrm4_mode_tunnel_gso_segment, - .xmit = xfrm4_mode_tunnel_xmit, - .owner = THIS_MODULE, - .encap = XFRM_MODE_TUNNEL, - .flags = XFRM_MODE_FLAG_TUNNEL, -}; - -static int __init xfrm4_mode_tunnel_init(void) -{ - return xfrm_register_mode(&xfrm4_tunnel_mode, AF_INET); -} - -static void __exit xfrm4_mode_tunnel_exit(void) -{ - int err; - - err = xfrm_unregister_mode(&xfrm4_tunnel_mode, AF_INET); - BUG_ON(err); -} - -module_init(xfrm4_mode_tunnel_init); -module_exit(xfrm4_mode_tunnel_exit); -MODULE_LICENSE("GPL"); -MODULE_ALIAS_XFRM_MODE(AF_INET, XFRM_MODE_TUNNEL); diff --git a/net/ipv4/xfrm4_output.c b/net/ipv4/xfrm4_output.c index be980c195fc55b0a23f144eb55641cb16a2eb2de..9bb8905088c75619ebde66739dd43896a90528fd 100644 --- a/net/ipv4/xfrm4_output.c +++ b/net/ipv4/xfrm4_output.c @@ -58,21 +58,6 @@ int xfrm4_extract_output(struct xfrm_state *x, struct sk_buff *skb) return xfrm4_extract_header(skb); } -int xfrm4_prepare_output(struct xfrm_state *x, struct sk_buff *skb) -{ - int err; - - err = xfrm_inner_extract_output(x, skb); - if (err) - return err; - - IPCB(skb)->flags |= IPSKB_XFRM_TUNNEL_SIZE; - skb->protocol = htons(ETH_P_IP); - - return x->outer_mode->output2(x, skb); -} -EXPORT_SYMBOL(xfrm4_prepare_output); - int xfrm4_output_finish(struct sock *sk, struct sk_buff *skb) { memset(IPCB(skb), 0, sizeof(*IPCB(skb))); @@ -87,6 +72,8 @@ int xfrm4_output_finish(struct sock *sk, struct sk_buff *skb) static int __xfrm4_output(struct net *net, struct sock *sk, struct sk_buff *skb) { struct xfrm_state *x = skb_dst(skb)->xfrm; + const struct xfrm_state_afinfo *afinfo; + int ret = -EAFNOSUPPORT; #ifdef CONFIG_NETFILTER if (!x) { @@ -95,7 +82,15 @@ static int __xfrm4_output(struct net *net, struct sock *sk, struct sk_buff *skb) } #endif - return x->outer_mode->afinfo->output_finish(sk, skb); + rcu_read_lock(); + afinfo = xfrm_state_afinfo_get_rcu(x->outer_mode.family); + if (likely(afinfo)) + ret = afinfo->output_finish(sk, skb); + else + kfree_skb(skb); + rcu_read_unlock(); + + return ret; } int xfrm4_output(struct net *net, struct sock *sk, struct sk_buff *skb) diff --git a/net/ipv4/xfrm4_policy.c b/net/ipv4/xfrm4_policy.c index 2b144b92ae46a430d184fca120616166e8010f53..cdef8f9a3b0166942c3bad33b649dffa97578ca8 100644 --- a/net/ipv4/xfrm4_policy.c +++ b/net/ipv4/xfrm4_policy.c @@ -12,7 +12,6 @@ #include #include #include -#include #include #include #include @@ -69,17 +68,6 @@ static int xfrm4_get_saddr(struct net *net, int oif, return 0; } -static int xfrm4_get_tos(const struct flowi *fl) -{ - return IPTOS_RT_MASK & fl->u.ip4.flowi4_tos; /* Strip ECN bits */ -} - -static int xfrm4_init_path(struct xfrm_dst *path, struct dst_entry *dst, - int nfheader_len) -{ - return 0; -} - static int xfrm4_fill_dst(struct xfrm_dst *xdst, struct net_device *dev, const struct flowi *fl) { @@ -97,8 +85,11 @@ static int xfrm4_fill_dst(struct xfrm_dst *xdst, struct net_device *dev, xdst->u.rt.rt_flags = rt->rt_flags & (RTCF_BROADCAST | RTCF_MULTICAST | RTCF_LOCAL); xdst->u.rt.rt_type = rt->rt_type; - xdst->u.rt.rt_gateway = rt->rt_gateway; - xdst->u.rt.rt_uses_gateway = rt->rt_uses_gateway; + xdst->u.rt.rt_gw_family = rt->rt_gw_family; + if (rt->rt_gw_family == AF_INET) + xdst->u.rt.rt_gw4 = rt->rt_gw4; + else if (rt->rt_gw_family == AF_INET6) + xdst->u.rt.rt_gw6 = rt->rt_gw6; xdst->u.rt.rt_pmtu = rt->rt_pmtu; xdst->u.rt.rt_mtu_locked = rt->rt_mtu_locked; INIT_LIST_HEAD(&xdst->u.rt.rt_uncached); @@ -107,120 +98,6 @@ static int xfrm4_fill_dst(struct xfrm_dst *xdst, struct net_device *dev, return 0; } -static void -_decode_session4(struct sk_buff *skb, struct flowi *fl, int reverse) -{ - const struct iphdr *iph = ip_hdr(skb); - int ihl = iph->ihl; - u8 *xprth = skb_network_header(skb) + ihl * 4; - struct flowi4 *fl4 = &fl->u.ip4; - int oif = 0; - - if (skb_dst(skb)) - oif = skb_dst(skb)->dev->ifindex; - - memset(fl4, 0, sizeof(struct flowi4)); - fl4->flowi4_mark = skb->mark; - fl4->flowi4_oif = reverse ? skb->skb_iif : oif; - - fl4->flowi4_proto = iph->protocol; - fl4->daddr = reverse ? iph->saddr : iph->daddr; - fl4->saddr = reverse ? iph->daddr : iph->saddr; - fl4->flowi4_tos = iph->tos; - - if (!ip_is_fragment(iph)) { - switch (iph->protocol) { - case IPPROTO_UDP: - case IPPROTO_UDPLITE: - case IPPROTO_TCP: - case IPPROTO_SCTP: - case IPPROTO_DCCP: - if (xprth + 4 < skb->data || - pskb_may_pull(skb, xprth + 4 - skb->data)) { - __be16 *ports; - - xprth = skb_network_header(skb) + ihl * 4; - ports = (__be16 *)xprth; - - fl4->fl4_sport = ports[!!reverse]; - fl4->fl4_dport = ports[!reverse]; - } - break; - - case IPPROTO_ICMP: - if (xprth + 2 < skb->data || - pskb_may_pull(skb, xprth + 2 - skb->data)) { - u8 *icmp; - - xprth = skb_network_header(skb) + ihl * 4; - icmp = xprth; - - fl4->fl4_icmp_type = icmp[0]; - fl4->fl4_icmp_code = icmp[1]; - } - break; - - case IPPROTO_ESP: - if (xprth + 4 < skb->data || - pskb_may_pull(skb, xprth + 4 - skb->data)) { - __be32 *ehdr; - - xprth = skb_network_header(skb) + ihl * 4; - ehdr = (__be32 *)xprth; - - fl4->fl4_ipsec_spi = ehdr[0]; - } - break; - - case IPPROTO_AH: - if (xprth + 8 < skb->data || - pskb_may_pull(skb, xprth + 8 - skb->data)) { - __be32 *ah_hdr; - - xprth = skb_network_header(skb) + ihl * 4; - ah_hdr = (__be32 *)xprth; - - fl4->fl4_ipsec_spi = ah_hdr[1]; - } - break; - - case IPPROTO_COMP: - if (xprth + 4 < skb->data || - pskb_may_pull(skb, xprth + 4 - skb->data)) { - __be16 *ipcomp_hdr; - - xprth = skb_network_header(skb) + ihl * 4; - ipcomp_hdr = (__be16 *)xprth; - - fl4->fl4_ipsec_spi = htonl(ntohs(ipcomp_hdr[1])); - } - break; - - case IPPROTO_GRE: - if (xprth + 12 < skb->data || - pskb_may_pull(skb, xprth + 12 - skb->data)) { - __be16 *greflags; - __be32 *gre_hdr; - - xprth = skb_network_header(skb) + ihl * 4; - greflags = (__be16 *)xprth; - gre_hdr = (__be32 *)xprth; - - if (greflags[0] & GRE_KEY) { - if (greflags[0] & GRE_CSUM) - gre_hdr++; - fl4->fl4_gre_key = gre_hdr[1]; - } - } - break; - - default: - fl4->fl4_ipsec_spi = 0; - break; - } - } -} - static void xfrm4_update_pmtu(struct dst_entry *dst, struct sock *sk, struct sk_buff *skb, u32 mtu) { @@ -273,9 +150,6 @@ static const struct xfrm_policy_afinfo xfrm4_policy_afinfo = { .dst_ops = &xfrm4_dst_ops_template, .dst_lookup = xfrm4_dst_lookup, .get_saddr = xfrm4_get_saddr, - .decode_session = _decode_session4, - .get_tos = xfrm4_get_tos, - .init_path = xfrm4_init_path, .fill_dst = xfrm4_fill_dst, .blackhole_route = ipv4_blackhole_route, }; diff --git a/net/ipv4/xfrm4_protocol.c b/net/ipv4/xfrm4_protocol.c index 35c54865dc42399da6be67e036692ed56b0ba333..bcab48944c15b390c1f7aaad11db1f3895094289 100644 --- a/net/ipv4/xfrm4_protocol.c +++ b/net/ipv4/xfrm4_protocol.c @@ -46,7 +46,7 @@ static inline struct xfrm4_protocol __rcu **proto_handlers(u8 protocol) handler != NULL; \ handler = rcu_dereference(handler->next)) \ -int xfrm4_rcv_cb(struct sk_buff *skb, u8 protocol, int err) +static int xfrm4_rcv_cb(struct sk_buff *skb, u8 protocol, int err) { int ret; struct xfrm4_protocol *handler; @@ -61,7 +61,6 @@ int xfrm4_rcv_cb(struct sk_buff *skb, u8 protocol, int err) return 0; } -EXPORT_SYMBOL(xfrm4_rcv_cb); int xfrm4_rcv_encap(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type) diff --git a/net/ipv6/Kconfig b/net/ipv6/Kconfig index 613282c65a10750cccc5985be0ffc65a68138ae1..cd915e332c985c1ae139bfca9470c0d0c26b8655 100644 --- a/net/ipv6/Kconfig +++ b/net/ipv6/Kconfig @@ -135,44 +135,11 @@ config INET6_TUNNEL tristate default n -config INET6_XFRM_MODE_TRANSPORT - tristate "IPv6: IPsec transport mode" - default IPV6 - select XFRM - ---help--- - Support for IPsec transport mode. - - If unsure, say Y. - -config INET6_XFRM_MODE_TUNNEL - tristate "IPv6: IPsec tunnel mode" - default IPV6 - select XFRM - ---help--- - Support for IPsec tunnel mode. - - If unsure, say Y. - -config INET6_XFRM_MODE_BEET - tristate "IPv6: IPsec BEET mode" - default IPV6 - select XFRM - ---help--- - Support for IPsec BEET mode. - - If unsure, say Y. - -config INET6_XFRM_MODE_ROUTEOPTIMIZATION - tristate "IPv6: MIPv6 route optimization mode" - select XFRM - ---help--- - Support for MIPv6 route optimization mode. - config IPV6_VTI tristate "Virtual (secure) IPv6: tunneling" select IPV6_TUNNEL select NET_IP_TUNNEL - depends on INET6_XFRM_MODE_TUNNEL + select XFRM ---help--- Tunneling means encapsulating data of one protocol type within another protocol and sending it over a channel that understands the diff --git a/net/ipv6/Makefile b/net/ipv6/Makefile index e0026fa1261bcb277586ddbbffde615200d19604..8ccf355140154d8f4d91dd9f688a4758da978caa 100644 --- a/net/ipv6/Makefile +++ b/net/ipv6/Makefile @@ -35,10 +35,6 @@ obj-$(CONFIG_INET6_ESP_OFFLOAD) += esp6_offload.o obj-$(CONFIG_INET6_IPCOMP) += ipcomp6.o obj-$(CONFIG_INET6_XFRM_TUNNEL) += xfrm6_tunnel.o obj-$(CONFIG_INET6_TUNNEL) += tunnel6.o -obj-$(CONFIG_INET6_XFRM_MODE_TRANSPORT) += xfrm6_mode_transport.o -obj-$(CONFIG_INET6_XFRM_MODE_TUNNEL) += xfrm6_mode_tunnel.o -obj-$(CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION) += xfrm6_mode_ro.o -obj-$(CONFIG_INET6_XFRM_MODE_BEET) += xfrm6_mode_beet.o obj-$(CONFIG_IPV6_MIP6) += mip6.o obj-$(CONFIG_IPV6_ILA) += ila/ obj-$(CONFIG_NETFILTER) += netfilter/ diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c index 4ae17a966ae3d714fdf711d847e4e3ca73458ec2..f96d1de79509d63bdebcd25247a636822fb35570 100644 --- a/net/ipv6/addrconf.c +++ b/net/ipv6/addrconf.c @@ -173,7 +173,8 @@ static int addrconf_ifdown(struct net_device *dev, int how); static struct fib6_info *addrconf_get_prefix_route(const struct in6_addr *pfx, int plen, const struct net_device *dev, - u32 flags, u32 noflags); + u32 flags, u32 noflags, + bool no_gw); static void addrconf_dad_start(struct inet6_ifaddr *ifp); static void addrconf_dad_work(struct work_struct *w); @@ -610,11 +611,13 @@ static int inet6_netconf_valid_get_req(struct sk_buff *skb, } if (!netlink_strict_get_check(skb)) - return nlmsg_parse(nlh, sizeof(struct netconfmsg), tb, - NETCONFA_MAX, devconf_ipv6_policy, extack); + return nlmsg_parse_deprecated(nlh, sizeof(struct netconfmsg), + tb, NETCONFA_MAX, + devconf_ipv6_policy, extack); - err = nlmsg_parse_strict(nlh, sizeof(struct netconfmsg), tb, - NETCONFA_MAX, devconf_ipv6_policy, extack); + err = nlmsg_parse_deprecated_strict(nlh, sizeof(struct netconfmsg), + tb, NETCONFA_MAX, + devconf_ipv6_policy, extack); if (err) return err; @@ -1230,10 +1233,8 @@ cleanup_prefix_route(struct inet6_ifaddr *ifp, unsigned long expires, bool del_r { struct fib6_info *f6i; - f6i = addrconf_get_prefix_route(&ifp->addr, - ifp->prefix_len, - ifp->idev->dev, - 0, RTF_GATEWAY | RTF_DEFAULT); + f6i = addrconf_get_prefix_route(&ifp->addr, ifp->prefix_len, + ifp->idev->dev, 0, RTF_DEFAULT, true); if (f6i) { if (del_rt) ip6_del_rt(dev_net(ifp->idev->dev), f6i); @@ -2402,7 +2403,8 @@ addrconf_prefix_route(struct in6_addr *pfx, int plen, u32 metric, static struct fib6_info *addrconf_get_prefix_route(const struct in6_addr *pfx, int plen, const struct net_device *dev, - u32 flags, u32 noflags) + u32 flags, u32 noflags, + bool no_gw) { struct fib6_node *fn; struct fib6_info *rt = NULL; @@ -2419,7 +2421,9 @@ static struct fib6_info *addrconf_get_prefix_route(const struct in6_addr *pfx, goto out; for_each_fib6_node_rt_rcu(fn) { - if (rt->fib6_nh.nh_dev->ifindex != dev->ifindex) + if (rt->fib6_nh.fib_nh_dev->ifindex != dev->ifindex) + continue; + if (no_gw && rt->fib6_nh.fib_nh_gw_family) continue; if ((rt->fib6_flags & flags) != flags) continue; @@ -2717,7 +2721,7 @@ void addrconf_prefix_rcv(struct net_device *dev, u8 *opt, int len, bool sllao) pinfo->prefix_len, dev, RTF_ADDRCONF | RTF_PREFIX_RT, - RTF_GATEWAY | RTF_DEFAULT); + RTF_DEFAULT, true); if (rt) { /* Autoconf prefix route */ @@ -4563,8 +4567,8 @@ inet6_rtm_deladdr(struct sk_buff *skb, struct nlmsghdr *nlh, u32 ifa_flags; int err; - err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, ifa_ipv6_policy, - extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*ifm), tb, IFA_MAX, + ifa_ipv6_policy, extack); if (err < 0) return err; @@ -4588,10 +4592,8 @@ static int modify_prefix_route(struct inet6_ifaddr *ifp, struct fib6_info *f6i; u32 prio; - f6i = addrconf_get_prefix_route(&ifp->addr, - ifp->prefix_len, - ifp->idev->dev, - 0, RTF_GATEWAY | RTF_DEFAULT); + f6i = addrconf_get_prefix_route(&ifp->addr, ifp->prefix_len, + ifp->idev->dev, 0, RTF_DEFAULT, true); if (!f6i) return -ENOENT; @@ -4729,8 +4731,8 @@ inet6_rtm_newaddr(struct sk_buff *skb, struct nlmsghdr *nlh, struct ifa6_config cfg; int err; - err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, ifa_ipv6_policy, - extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*ifm), tb, IFA_MAX, + ifa_ipv6_policy, extack); if (err < 0) return err; @@ -5086,8 +5088,8 @@ static int inet6_valid_dump_ifaddr_req(const struct nlmsghdr *nlh, fillargs->flags |= NLM_F_DUMP_FILTERED; } - err = nlmsg_parse_strict(nlh, sizeof(*ifm), tb, IFA_MAX, - ifa_ipv6_policy, extack); + err = nlmsg_parse_deprecated_strict(nlh, sizeof(*ifm), tb, IFA_MAX, + ifa_ipv6_policy, extack); if (err < 0) return err; @@ -5237,11 +5239,11 @@ static int inet6_rtm_valid_getaddr_req(struct sk_buff *skb, } if (!netlink_strict_get_check(skb)) - return nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, - ifa_ipv6_policy, extack); + return nlmsg_parse_deprecated(nlh, sizeof(*ifm), tb, IFA_MAX, + ifa_ipv6_policy, extack); - err = nlmsg_parse_strict(nlh, sizeof(*ifm), tb, IFA_MAX, - ifa_ipv6_policy, extack); + err = nlmsg_parse_deprecated_strict(nlh, sizeof(*ifm), tb, IFA_MAX, + ifa_ipv6_policy, extack); if (err) return err; @@ -5667,8 +5669,8 @@ static int inet6_validate_link_af(const struct net_device *dev, if (dev && !__in6_dev_get(dev)) return -EAFNOSUPPORT; - return nla_parse_nested(tb, IFLA_INET6_MAX, nla, inet6_af_policy, - NULL); + return nla_parse_nested_deprecated(tb, IFLA_INET6_MAX, nla, + inet6_af_policy, NULL); } static int check_addr_gen_mode(int mode) @@ -5700,7 +5702,7 @@ static int inet6_set_link_af(struct net_device *dev, const struct nlattr *nla) if (!idev) return -EAFNOSUPPORT; - if (nla_parse_nested(tb, IFLA_INET6_MAX, nla, NULL, NULL) < 0) + if (nla_parse_nested_deprecated(tb, IFLA_INET6_MAX, nla, NULL, NULL) < 0) BUG(); if (tb[IFLA_INET6_TOKEN]) { @@ -5752,7 +5754,7 @@ static int inet6_fill_ifinfo(struct sk_buff *skb, struct inet6_dev *idev, nla_put_u8(skb, IFLA_OPERSTATE, netif_running(dev) ? dev->operstate : IF_OPER_DOWN)) goto nla_put_failure; - protoinfo = nla_nest_start(skb, IFLA_PROTINFO); + protoinfo = nla_nest_start_noflag(skb, IFLA_PROTINFO); if (!protoinfo) goto nla_put_failure; @@ -5972,7 +5974,8 @@ static void __ipv6_ifa_notify(int event, struct inet6_ifaddr *ifp) struct fib6_info *rt; rt = addrconf_get_prefix_route(&ifp->peer_addr, 128, - ifp->idev->dev, 0, 0); + ifp->idev->dev, 0, 0, + false); if (rt) ip6_del_rt(net, rt); } diff --git a/net/ipv6/addrconf_core.c b/net/ipv6/addrconf_core.c index 6c79af056d9b7b7a4595d8a8764839d2204e9ac5..763a947e0d14d3455492b0a13c96153c6936535d 100644 --- a/net/ipv6/addrconf_core.c +++ b/net/ipv6/addrconf_core.c @@ -5,7 +5,7 @@ #include #include -#include +#include #include /* if ipv6 module registers this function is used by xfrm to force all @@ -144,43 +144,53 @@ static struct fib6_table *eafnosupport_fib6_get_table(struct net *net, u32 id) return NULL; } -static struct fib6_info * +static int eafnosupport_fib6_table_lookup(struct net *net, struct fib6_table *table, - int oif, struct flowi6 *fl6, int flags) + int oif, struct flowi6 *fl6, + struct fib6_result *res, int flags) { - return NULL; + return -EAFNOSUPPORT; } -static struct fib6_info * +static int eafnosupport_fib6_lookup(struct net *net, int oif, struct flowi6 *fl6, - int flags) + struct fib6_result *res, int flags) { - return NULL; + return -EAFNOSUPPORT; } -static struct fib6_info * -eafnosupport_fib6_multipath_select(const struct net *net, struct fib6_info *f6i, - struct flowi6 *fl6, int oif, - const struct sk_buff *skb, int strict) +static void +eafnosupport_fib6_select_path(const struct net *net, struct fib6_result *res, + struct flowi6 *fl6, int oif, bool have_oif_match, + const struct sk_buff *skb, int strict) { - return f6i; } static u32 -eafnosupport_ip6_mtu_from_fib6(struct fib6_info *f6i, struct in6_addr *daddr, - struct in6_addr *saddr) +eafnosupport_ip6_mtu_from_fib6(const struct fib6_result *res, + const struct in6_addr *daddr, + const struct in6_addr *saddr) { return 0; } +static int eafnosupport_fib6_nh_init(struct net *net, struct fib6_nh *fib6_nh, + struct fib6_config *cfg, gfp_t gfp_flags, + struct netlink_ext_ack *extack) +{ + NL_SET_ERR_MSG(extack, "IPv6 support not enabled in kernel"); + return -EAFNOSUPPORT; +} + const struct ipv6_stub *ipv6_stub __read_mostly = &(struct ipv6_stub) { .ipv6_dst_lookup = eafnosupport_ipv6_dst_lookup, .ipv6_route_input = eafnosupport_ipv6_route_input, .fib6_get_table = eafnosupport_fib6_get_table, .fib6_table_lookup = eafnosupport_fib6_table_lookup, .fib6_lookup = eafnosupport_fib6_lookup, - .fib6_multipath_select = eafnosupport_fib6_multipath_select, + .fib6_select_path = eafnosupport_fib6_select_path, .ip6_mtu_from_fib6 = eafnosupport_ip6_mtu_from_fib6, + .fib6_nh_init = eafnosupport_fib6_nh_init, }; EXPORT_SYMBOL_GPL(ipv6_stub); diff --git a/net/ipv6/addrlabel.c b/net/ipv6/addrlabel.c index 1766325423b5dad4d8c95c85605dc571248ba6d1..642fc6ac13d229230a51d8513b60fc31c00b7bc4 100644 --- a/net/ipv6/addrlabel.c +++ b/net/ipv6/addrlabel.c @@ -383,8 +383,8 @@ static int ip6addrlbl_newdel(struct sk_buff *skb, struct nlmsghdr *nlh, u32 label; int err = 0; - err = nlmsg_parse(nlh, sizeof(*ifal), tb, IFAL_MAX, ifal_policy, - extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*ifal), tb, IFAL_MAX, + ifal_policy, extack); if (err < 0) return err; @@ -537,8 +537,8 @@ static int ip6addrlbl_valid_get_req(struct sk_buff *skb, } if (!netlink_strict_get_check(skb)) - return nlmsg_parse(nlh, sizeof(*ifal), tb, IFAL_MAX, - ifal_policy, extack); + return nlmsg_parse_deprecated(nlh, sizeof(*ifal), tb, + IFAL_MAX, ifal_policy, extack); ifal = nlmsg_data(nlh); if (ifal->__ifal_reserved || ifal->ifal_flags || ifal->ifal_seq) { @@ -546,8 +546,8 @@ static int ip6addrlbl_valid_get_req(struct sk_buff *skb, return -EINVAL; } - err = nlmsg_parse_strict(nlh, sizeof(*ifal), tb, IFAL_MAX, - ifal_policy, extack); + err = nlmsg_parse_deprecated_strict(nlh, sizeof(*ifal), tb, IFAL_MAX, + ifal_policy, extack); if (err) return err; diff --git a/net/ipv6/af_inet6.c b/net/ipv6/af_inet6.c index 2f45d2a3e3a33e7e43a3384d1a8fe7a683580919..c04ae282f4e4747090314fc44f041a2f00f8efb6 100644 --- a/net/ipv6/af_inet6.c +++ b/net/ipv6/af_inet6.c @@ -56,6 +56,7 @@ #include #include #include +#include #include #ifdef CONFIG_IPV6_TUNNEL #include @@ -546,12 +547,6 @@ int inet6_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) struct net *net = sock_net(sk); switch (cmd) { - case SIOCGSTAMP: - return sock_get_timestamp(sk, (struct timeval __user *)arg); - - case SIOCGSTAMPNS: - return sock_get_timestampns(sk, (struct timespec __user *)arg); - case SIOCADDRT: case SIOCDELRT: @@ -584,6 +579,7 @@ const struct proto_ops inet6_stream_ops = { .getname = inet6_getname, .poll = tcp_poll, /* ok */ .ioctl = inet6_ioctl, /* must change */ + .gettstamp = sock_gettstamp, .listen = inet_listen, /* ok */ .shutdown = inet_shutdown, /* ok */ .setsockopt = sock_common_setsockopt, /* ok */ @@ -617,6 +613,7 @@ const struct proto_ops inet6_dgram_ops = { .getname = inet6_getname, .poll = udp_poll, /* ok */ .ioctl = inet6_ioctl, /* must change */ + .gettstamp = sock_gettstamp, .listen = sock_no_listen, /* ok */ .shutdown = inet_shutdown, /* ok */ .setsockopt = sock_common_setsockopt, /* ok */ @@ -847,6 +844,17 @@ static int __net_init inet6_net_init(struct net *net) net->ipv6.sysctl.bindv6only = 0; net->ipv6.sysctl.icmpv6_time = 1*HZ; net->ipv6.sysctl.icmpv6_echo_ignore_all = 0; + net->ipv6.sysctl.icmpv6_echo_ignore_multicast = 0; + net->ipv6.sysctl.icmpv6_echo_ignore_anycast = 0; + + /* By default, rate limit error messages. + * Except for pmtu discovery, it would break it. + * proc_do_large_bitmap needs pointer to the bitmap. + */ + bitmap_set(net->ipv6.sysctl.icmpv6_ratemask, 0, ICMPV6_ERRMSG_MAX + 1); + bitmap_clear(net->ipv6.sysctl.icmpv6_ratemask, ICMPV6_PKT_TOOBIG, 1); + net->ipv6.sysctl.icmpv6_ratemask_ptr = net->ipv6.sysctl.icmpv6_ratemask; + net->ipv6.sysctl.flowlabel_consistency = 1; net->ipv6.sysctl.auto_flowlabels = IP6_DEFAULT_AUTO_FLOW_LABELS; net->ipv6.sysctl.idgen_retries = 3; @@ -914,8 +922,10 @@ static const struct ipv6_stub ipv6_stub_impl = { .fib6_get_table = fib6_get_table, .fib6_table_lookup = fib6_table_lookup, .fib6_lookup = fib6_lookup, - .fib6_multipath_select = fib6_multipath_select, + .fib6_select_path = fib6_select_path, .ip6_mtu_from_fib6 = ip6_mtu_from_fib6, + .fib6_nh_init = fib6_nh_init, + .fib6_nh_release = fib6_nh_release, .udpv6_encap_enable = udpv6_encap_enable, .ndisc_send_na = ndisc_send_na, .nd_tbl = &nd_tbl, diff --git a/net/ipv6/esp6_offload.c b/net/ipv6/esp6_offload.c index cb99f6fb79b798702ecb87be006044fe06dda3c2..d453cf417b03df29a919ca40360d2a9269c159ca 100644 --- a/net/ipv6/esp6_offload.c +++ b/net/ipv6/esp6_offload.c @@ -136,6 +136,44 @@ static void esp6_gso_encap(struct xfrm_state *x, struct sk_buff *skb) xo->proto = proto; } +static struct sk_buff *xfrm6_tunnel_gso_segment(struct xfrm_state *x, + struct sk_buff *skb, + netdev_features_t features) +{ + __skb_push(skb, skb->mac_len); + return skb_mac_gso_segment(skb, features); +} + +static struct sk_buff *xfrm6_transport_gso_segment(struct xfrm_state *x, + struct sk_buff *skb, + netdev_features_t features) +{ + const struct net_offload *ops; + struct sk_buff *segs = ERR_PTR(-EINVAL); + struct xfrm_offload *xo = xfrm_offload(skb); + + skb->transport_header += x->props.header_len; + ops = rcu_dereference(inet6_offloads[xo->proto]); + if (likely(ops && ops->callbacks.gso_segment)) + segs = ops->callbacks.gso_segment(skb, features); + + return segs; +} + +static struct sk_buff *xfrm6_outer_mode_gso_segment(struct xfrm_state *x, + struct sk_buff *skb, + netdev_features_t features) +{ + switch (x->outer_mode.encap) { + case XFRM_MODE_TUNNEL: + return xfrm6_tunnel_gso_segment(x, skb, features); + case XFRM_MODE_TRANSPORT: + return xfrm6_transport_gso_segment(x, skb, features); + } + + return ERR_PTR(-EOPNOTSUPP); +} + static struct sk_buff *esp6_gso_segment(struct sk_buff *skb, netdev_features_t features) { @@ -174,7 +212,7 @@ static struct sk_buff *esp6_gso_segment(struct sk_buff *skb, xo->flags |= XFRM_GSO_SEGMENT; - return x->outer_mode->gso_segment(x, skb, esp_features); + return xfrm6_outer_mode_gso_segment(x, skb, esp_features); } static int esp6_input_tail(struct xfrm_state *x, struct sk_buff *skb) diff --git a/net/ipv6/fib6_rules.c b/net/ipv6/fib6_rules.c index f590446595d8b879de9ee384021311deed7c943a..06d1b7763600ca6b28f74f527e9d9a424eaf7555 100644 --- a/net/ipv6/fib6_rules.c +++ b/net/ipv6/fib6_rules.c @@ -61,16 +61,16 @@ unsigned int fib6_rules_seq_read(struct net *net) } /* called with rcu lock held; no reference taken on fib6_info */ -struct fib6_info *fib6_lookup(struct net *net, int oif, struct flowi6 *fl6, - int flags) +int fib6_lookup(struct net *net, int oif, struct flowi6 *fl6, + struct fib6_result *res, int flags) { - struct fib6_info *f6i; int err; if (net->ipv6.fib6_has_custom_rules) { struct fib_lookup_arg arg = { .lookup_ptr = fib6_table_lookup, .lookup_data = &oif, + .result = res, .flags = FIB_LOOKUP_NOREF, }; @@ -78,19 +78,15 @@ struct fib6_info *fib6_lookup(struct net *net, int oif, struct flowi6 *fl6, err = fib_rules_lookup(net->ipv6.fib6_rules_ops, flowi6_to_flowi(fl6), flags, &arg); - if (err) - return ERR_PTR(err); - - f6i = arg.result ? : net->ipv6.fib6_null_entry; } else { - f6i = fib6_table_lookup(net, net->ipv6.fib6_local_tbl, - oif, fl6, flags); - if (!f6i || f6i == net->ipv6.fib6_null_entry) - f6i = fib6_table_lookup(net, net->ipv6.fib6_main_tbl, - oif, fl6, flags); + err = fib6_table_lookup(net, net->ipv6.fib6_local_tbl, oif, + fl6, res, flags); + if (err || res->f6i == net->ipv6.fib6_null_entry) + err = fib6_table_lookup(net, net->ipv6.fib6_main_tbl, + oif, fl6, res, flags); } - return f6i; + return err; } struct dst_entry *fib6_rule_lookup(struct net *net, struct flowi6 *fl6, @@ -98,9 +94,11 @@ struct dst_entry *fib6_rule_lookup(struct net *net, struct flowi6 *fl6, int flags, pol_lookup_t lookup) { if (net->ipv6.fib6_has_custom_rules) { + struct fib6_result res = {}; struct fib_lookup_arg arg = { .lookup_ptr = lookup, .lookup_data = skb, + .result = &res, .flags = FIB_LOOKUP_NOREF, }; @@ -110,8 +108,8 @@ struct dst_entry *fib6_rule_lookup(struct net *net, struct flowi6 *fl6, fib_rules_lookup(net->ipv6.fib6_rules_ops, flowi6_to_flowi(fl6), flags, &arg); - if (arg.result) - return arg.result; + if (res.rt6) + return &res.rt6->dst; } else { struct rt6_info *rt; @@ -157,11 +155,11 @@ static int fib6_rule_saddr(struct net *net, struct fib_rule *rule, int flags, static int fib6_rule_action_alt(struct fib_rule *rule, struct flowi *flp, int flags, struct fib_lookup_arg *arg) { + struct fib6_result *res = arg->result; struct flowi6 *flp6 = &flp->u.ip6; struct net *net = rule->fr_net; struct fib6_table *table; - struct fib6_info *f6i; - int err = -EAGAIN, *oif; + int err, *oif; u32 tb_id; switch (rule->action) { @@ -182,14 +180,12 @@ static int fib6_rule_action_alt(struct fib_rule *rule, struct flowi *flp, return -EAGAIN; oif = (int *)arg->lookup_data; - f6i = fib6_table_lookup(net, table, *oif, flp6, flags); - if (f6i != net->ipv6.fib6_null_entry) { + err = fib6_table_lookup(net, table, *oif, flp6, res, flags); + if (!err && res->f6i != net->ipv6.fib6_null_entry) err = fib6_rule_saddr(net, rule, flags, flp6, - fib6_info_nh_dev(f6i)); - - if (likely(!err)) - arg->result = f6i; - } + res->nh->fib_nh_dev); + else + err = -EAGAIN; return err; } @@ -197,6 +193,7 @@ static int fib6_rule_action_alt(struct fib_rule *rule, struct flowi *flp, static int __fib6_rule_action(struct fib_rule *rule, struct flowi *flp, int flags, struct fib_lookup_arg *arg) { + struct fib6_result *res = arg->result; struct flowi6 *flp6 = &flp->u.ip6; struct rt6_info *rt = NULL; struct fib6_table *table; @@ -251,7 +248,7 @@ again: discard_pkt: dst_hold(&rt->dst); out: - arg->result = rt; + res->rt6 = rt; return err; } @@ -266,9 +263,13 @@ static int fib6_rule_action(struct fib_rule *rule, struct flowi *flp, static bool fib6_rule_suppress(struct fib_rule *rule, struct fib_lookup_arg *arg) { - struct rt6_info *rt = (struct rt6_info *) arg->result; + struct fib6_result *res = arg->result; + struct rt6_info *rt = res->rt6; struct net_device *dev = NULL; + if (!rt) + return false; + if (rt->rt6i_idev) dev = rt->rt6i_idev->dev; diff --git a/net/ipv6/icmp.c b/net/ipv6/icmp.c index 802faa2fcc0ed10435d08db62b6f3ca3922ac1b2..afb915807cd0109e02df6e0858319c0ff78c33e8 100644 --- a/net/ipv6/icmp.c +++ b/net/ipv6/icmp.c @@ -168,22 +168,21 @@ static bool is_ineligible(const struct sk_buff *skb) return false; } -static bool icmpv6_mask_allow(int type) +static bool icmpv6_mask_allow(struct net *net, int type) { - /* Informational messages are not limited. */ - if (type & ICMPV6_INFOMSG_MASK) + if (type > ICMPV6_MSG_MAX) return true; - /* Do not limit pmtu discovery, it would break it. */ - if (type == ICMPV6_PKT_TOOBIG) + /* Limit if icmp type is set in ratemask. */ + if (!test_bit(type, net->ipv6.sysctl.icmpv6_ratemask)) return true; return false; } -static bool icmpv6_global_allow(int type) +static bool icmpv6_global_allow(struct net *net, int type) { - if (icmpv6_mask_allow(type)) + if (icmpv6_mask_allow(net, type)) return true; if (icmp_global_allow()) @@ -202,7 +201,7 @@ static bool icmpv6_xrlim_allow(struct sock *sk, u8 type, struct dst_entry *dst; bool res = false; - if (icmpv6_mask_allow(type)) + if (icmpv6_mask_allow(net, type)) return true; /* @@ -511,7 +510,7 @@ static void icmp6_send(struct sk_buff *skb, u8 type, u8 code, __u32 info, local_bh_disable(); /* Check global sysctl_icmp_msgs_per_sec ratelimit */ - if (!(skb->dev->flags&IFF_LOOPBACK) && !icmpv6_global_allow(type)) + if (!(skb->dev->flags & IFF_LOOPBACK) && !icmpv6_global_allow(net, type)) goto out_bh_enable; mip6_addr_swap(skb); @@ -683,12 +682,20 @@ static void icmpv6_echo_reply(struct sk_buff *skb) struct dst_entry *dst; struct ipcm6_cookie ipc6; u32 mark = IP6_REPLY_MARK(net, skb->mark); + bool acast; + + if (ipv6_addr_is_multicast(&ipv6_hdr(skb)->daddr) && + net->ipv6.sysctl.icmpv6_echo_ignore_multicast) + return; saddr = &ipv6_hdr(skb)->daddr; + acast = ipv6_anycast_destination(skb_dst(skb), saddr); + if (acast && net->ipv6.sysctl.icmpv6_echo_ignore_anycast) + return; + if (!ipv6_unicast_destination(skb) && - !(net->ipv6.sysctl.anycast_src_echo_reply && - ipv6_anycast_destination(skb_dst(skb), saddr))) + !(net->ipv6.sysctl.anycast_src_echo_reply && acast)) saddr = NULL; memcpy(&tmp_hdr, icmph, sizeof(tmp_hdr)); @@ -723,6 +730,11 @@ static void icmpv6_echo_reply(struct sk_buff *skb) if (IS_ERR(dst)) goto out; + /* Check the ratelimit */ + if ((!(skb->dev->flags & IFF_LOOPBACK) && !icmpv6_global_allow(net, ICMPV6_ECHO_REPLY)) || + !icmpv6_xrlim_allow(sk, ICMPV6_ECHO_REPLY, &fl6)) + goto out_dst_release; + idev = __in6_dev_get(skb->dev); msg.skb = skb; @@ -743,6 +755,7 @@ static void icmpv6_echo_reply(struct sk_buff *skb) icmpv6_push_pending_frames(sk, &fl6, &tmp_hdr, skb->len + sizeof(struct icmp6hdr)); } +out_dst_release: dst_release(dst); out: icmpv6_xmit_unlock(sk); @@ -1115,6 +1128,27 @@ static struct ctl_table ipv6_icmp_table_template[] = { .mode = 0644, .proc_handler = proc_dointvec, }, + { + .procname = "echo_ignore_multicast", + .data = &init_net.ipv6.sysctl.icmpv6_echo_ignore_multicast, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = proc_dointvec, + }, + { + .procname = "echo_ignore_anycast", + .data = &init_net.ipv6.sysctl.icmpv6_echo_ignore_anycast, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = proc_dointvec, + }, + { + .procname = "ratemask", + .data = &init_net.ipv6.sysctl.icmpv6_ratemask_ptr, + .maxlen = ICMPV6_MSG_MAX + 1, + .mode = 0644, + .proc_handler = proc_do_large_bitmap, + }, { }, }; @@ -1129,6 +1163,9 @@ struct ctl_table * __net_init ipv6_icmp_sysctl_init(struct net *net) if (table) { table[0].data = &net->ipv6.sysctl.icmpv6_time; table[1].data = &net->ipv6.sysctl.icmpv6_echo_ignore_all; + table[2].data = &net->ipv6.sysctl.icmpv6_echo_ignore_multicast; + table[3].data = &net->ipv6.sysctl.icmpv6_echo_ignore_anycast; + table[4].data = &net->ipv6.sysctl.icmpv6_ratemask_ptr; } return table; } diff --git a/net/ipv6/ila/ila_lwt.c b/net/ipv6/ila/ila_lwt.c index 3d56a2fb6f86f1edea73a1c2153ff5d1ed8de451..422dcc691f71cb878f2f6e273bfcf3f2ef6065c3 100644 --- a/net/ipv6/ila/ila_lwt.c +++ b/net/ipv6/ila/ila_lwt.c @@ -146,7 +146,8 @@ static int ila_build_state(struct nlattr *nla, if (family != AF_INET6) return -EINVAL; - ret = nla_parse_nested(tb, ILA_ATTR_MAX, nla, ila_nl_policy, extack); + ret = nla_parse_nested_deprecated(tb, ILA_ATTR_MAX, nla, + ila_nl_policy, extack); if (ret < 0) return ret; diff --git a/net/ipv6/ila/ila_main.c b/net/ipv6/ila/ila_main.c index 18fac76b952052367309ee2ff4b226d7d10415d7..257d2b681246aa863b6cd4400f9ad73e1155c4aa 100644 --- a/net/ipv6/ila/ila_main.c +++ b/net/ipv6/ila/ila_main.c @@ -16,29 +16,29 @@ static const struct nla_policy ila_nl_policy[ILA_ATTR_MAX + 1] = { static const struct genl_ops ila_nl_ops[] = { { .cmd = ILA_CMD_ADD, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = ila_xlat_nl_cmd_add_mapping, - .policy = ila_nl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = ILA_CMD_DEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = ila_xlat_nl_cmd_del_mapping, - .policy = ila_nl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = ILA_CMD_FLUSH, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = ila_xlat_nl_cmd_flush, - .policy = ila_nl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = ILA_CMD_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = ila_xlat_nl_cmd_get_mapping, .start = ila_xlat_nl_dump_start, .dumpit = ila_xlat_nl_dump, .done = ila_xlat_nl_dump_done, - .policy = ila_nl_policy, }, }; @@ -49,6 +49,7 @@ struct genl_family ila_nl_family __ro_after_init = { .name = ILA_GENL_NAME, .version = ILA_GENL_VERSION, .maxattr = ILA_ATTR_MAX, + .policy = ila_nl_policy, .netnsok = true, .parallel_ops = true, .module = THIS_MODULE, diff --git a/net/ipv6/ip6_fib.c b/net/ipv6/ip6_fib.c index 91247a6fc67ff7de1106d028b315a559e53e47f4..08e0390e001c270ae21013f3fd3ef3bf2a52e039 100644 --- a/net/ipv6/ip6_fib.c +++ b/net/ipv6/ip6_fib.c @@ -162,7 +162,7 @@ struct fib6_info *fib6_info_alloc(gfp_t gfp_flags) } INIT_LIST_HEAD(&f6i->fib6_siblings); - atomic_inc(&f6i->fib6_ref); + refcount_set(&f6i->fib6_ref, 1); return f6i; } @@ -175,10 +175,7 @@ void fib6_info_destroy_rcu(struct rcu_head *head) WARN_ON(f6i->fib6_node); bucket = rcu_dereference_protected(f6i->rt6i_exception_bucket, 1); - if (bucket) { - f6i->rt6i_exception_bucket = NULL; - kfree(bucket); - } + kfree(bucket); if (f6i->rt6i_pcpu) { int cpu; @@ -199,10 +196,7 @@ void fib6_info_destroy_rcu(struct rcu_head *head) free_percpu(f6i->rt6i_pcpu); } - lwtstate_put(f6i->fib6_nh.nh_lwtstate); - - if (f6i->fib6_nh.nh_dev) - dev_put(f6i->fib6_nh.nh_dev); + fib6_nh_release(&f6i->fib6_nh); ip_fib_metrics_put(f6i->fib6_metrics); @@ -357,10 +351,11 @@ struct dst_entry *fib6_rule_lookup(struct net *net, struct flowi6 *fl6, } /* called with rcu lock held; no reference taken on fib6_info */ -struct fib6_info *fib6_lookup(struct net *net, int oif, struct flowi6 *fl6, - int flags) +int fib6_lookup(struct net *net, int oif, struct flowi6 *fl6, + struct fib6_result *res, int flags) { - return fib6_table_lookup(net, net->ipv6.fib6_main_tbl, oif, fl6, flags); + return fib6_table_lookup(net, net->ipv6.fib6_main_tbl, oif, fl6, + res, flags); } static void __net_init fib6_tables_init(struct net *net) @@ -851,8 +846,8 @@ insert_above: RCU_INIT_POINTER(in->parent, pn); in->leaf = fn->leaf; - atomic_inc(&rcu_dereference_protected(in->leaf, - lockdep_is_held(&table->tb6_lock))->fib6_ref); + fib6_info_hold(rcu_dereference_protected(in->leaf, + lockdep_is_held(&table->tb6_lock))); /* update parent pointer */ if (dir) @@ -932,7 +927,7 @@ static void fib6_purge_rt(struct fib6_info *rt, struct fib6_node *fn, { struct fib6_table *table = rt->fib6_table; - if (atomic_read(&rt->fib6_ref) != 1) { + if (refcount_read(&rt->fib6_ref) != 1) { /* This route is used as dummy address holder in some split * nodes. It is not leaked, but it still holds other resources, * which must be released in time. So, scan ascendant nodes @@ -945,7 +940,7 @@ static void fib6_purge_rt(struct fib6_info *rt, struct fib6_node *fn, struct fib6_info *new_leaf; if (!(fn->fn_flags & RTN_RTINFO) && leaf == rt) { new_leaf = fib6_find_prefix(net, table, fn); - atomic_inc(&new_leaf->fib6_ref); + fib6_info_hold(new_leaf); rcu_assign_pointer(fn->leaf, new_leaf); fib6_info_release(rt); @@ -1111,7 +1106,7 @@ add: return err; rcu_assign_pointer(rt->fib6_next, iter); - atomic_inc(&rt->fib6_ref); + fib6_info_hold(rt); rcu_assign_pointer(rt->fib6_node, fn); rcu_assign_pointer(*ins, rt); if (!info->skip_notify) @@ -1139,7 +1134,7 @@ add: if (err) return err; - atomic_inc(&rt->fib6_ref); + fib6_info_hold(rt); rcu_assign_pointer(rt->fib6_node, fn); rt->fib6_next = iter->fib6_next; rcu_assign_pointer(*ins, rt); @@ -1281,7 +1276,7 @@ int fib6_add(struct fib6_node *root, struct fib6_info *rt, if (!sfn) goto failure; - atomic_inc(&info->nl_net->ipv6.fib6_null_entry->fib6_ref); + fib6_info_hold(info->nl_net->ipv6.fib6_null_entry); rcu_assign_pointer(sfn->leaf, info->nl_net->ipv6.fib6_null_entry); sfn->fn_flags = RTN_ROOT; @@ -1324,7 +1319,7 @@ int fib6_add(struct fib6_node *root, struct fib6_info *rt, rcu_assign_pointer(fn->leaf, info->nl_net->ipv6.fib6_null_entry); } else { - atomic_inc(&rt->fib6_ref); + fib6_info_hold(rt); rcu_assign_pointer(fn->leaf, rt); } } @@ -2295,6 +2290,7 @@ static int ipv6_route_seq_show(struct seq_file *seq, void *v) { struct fib6_info *rt = v; struct ipv6_route_iter *iter = seq->private; + unsigned int flags = rt->fib6_flags; const struct net_device *dev; seq_printf(seq, "%pi6 %02x ", &rt->fib6_dst.addr, rt->fib6_dst.plen); @@ -2304,15 +2300,17 @@ static int ipv6_route_seq_show(struct seq_file *seq, void *v) #else seq_puts(seq, "00000000000000000000000000000000 00 "); #endif - if (rt->fib6_flags & RTF_GATEWAY) - seq_printf(seq, "%pi6", &rt->fib6_nh.nh_gw); - else + if (rt->fib6_nh.fib_nh_gw_family) { + flags |= RTF_GATEWAY; + seq_printf(seq, "%pi6", &rt->fib6_nh.fib_nh_gw6); + } else { seq_puts(seq, "00000000000000000000000000000000"); + } - dev = rt->fib6_nh.nh_dev; + dev = rt->fib6_nh.fib_nh_dev; seq_printf(seq, " %08x %08x %08x %08x %8s\n", - rt->fib6_metric, atomic_read(&rt->fib6_ref), 0, - rt->fib6_flags, dev ? dev->name : ""); + rt->fib6_metric, refcount_read(&rt->fib6_ref), 0, + flags, dev ? dev->name : ""); iter->w.leaf = NULL; return 0; } diff --git a/net/ipv6/ip6_input.c b/net/ipv6/ip6_input.c index c7ed2b6d5a1dd192cc776231ede06cef8f78065d..b50b1af1f53074142523c417317e8da0ee661a45 100644 --- a/net/ipv6/ip6_input.c +++ b/net/ipv6/ip6_input.c @@ -29,6 +29,7 @@ #include #include #include +#include #include #include @@ -47,6 +48,8 @@ #include #include +INDIRECT_CALLABLE_DECLARE(void udp_v6_early_demux(struct sk_buff *)); +INDIRECT_CALLABLE_DECLARE(void tcp_v6_early_demux(struct sk_buff *)); static void ip6_rcv_finish_core(struct net *net, struct sock *sk, struct sk_buff *skb) { @@ -57,7 +60,8 @@ static void ip6_rcv_finish_core(struct net *net, struct sock *sk, ipprot = rcu_dereference(inet6_protos[ipv6_hdr(skb)->nexthdr]); if (ipprot && (edemux = READ_ONCE(ipprot->early_demux))) - edemux(skb); + INDIRECT_CALL_2(edemux, tcp_v6_early_demux, + udp_v6_early_demux, skb); } if (!skb_valid_dst(skb)) ip6_route_input(skb); @@ -316,6 +320,9 @@ void ipv6_list_rcv(struct list_head *head, struct packet_type *pt, ip6_sublist_rcv(&sublist, curr_dev, curr_net); } +INDIRECT_CALLABLE_DECLARE(int udpv6_rcv(struct sk_buff *)); +INDIRECT_CALLABLE_DECLARE(int tcp_v6_rcv(struct sk_buff *)); + /* * Deliver the packet to the host */ @@ -391,7 +398,8 @@ resubmit_final: !xfrm6_policy_check(NULL, XFRM_POLICY_IN, skb)) goto discard; - ret = ipprot->handler(skb); + ret = INDIRECT_CALL_2(ipprot->handler, tcp_v6_rcv, udpv6_rcv, + skb); if (ret > 0) { if (ipprot->flags & INET6_PROTO_FINAL) { /* Not an extension header, most likely UDP diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index e51f3c648b094afe1d60a518db36a42444c4c55d..adef2236abe2e767602a9d4afbf23fc6db5750dd 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -117,7 +117,7 @@ static int ip6_finish_output2(struct net *net, struct sock *sk, struct sk_buff * neigh = __neigh_create(&nd_tbl, nexthop, dst->dev, false); if (!IS_ERR(neigh)) { sock_confirm_neigh(skb, neigh); - ret = neigh_output(neigh, skb); + ret = neigh_output(neigh, skb, false); rcu_read_unlock_bh(); return ret; } diff --git a/net/ipv6/ip6_vti.c b/net/ipv6/ip6_vti.c index 8b6eefff2f7eaf624d854064b5f4b31121ba5079..218a0dedc8f48e72f3b9df03033875d3c873c6e2 100644 --- a/net/ipv6/ip6_vti.c +++ b/net/ipv6/ip6_vti.c @@ -342,7 +342,7 @@ static int vti6_rcv_cb(struct sk_buff *skb, int err) struct net_device *dev; struct pcpu_sw_netstats *tstats; struct xfrm_state *x; - struct xfrm_mode *inner_mode; + const struct xfrm_mode *inner_mode; struct ip6_tnl *t = XFRM_TUNNEL_SKB_CB(skb)->tunnel.ip6; u32 orig_mark = skb->mark; int ret; @@ -361,7 +361,7 @@ static int vti6_rcv_cb(struct sk_buff *skb, int err) x = xfrm_input_state(skb); - inner_mode = x->inner_mode; + inner_mode = &x->inner_mode; if (x->sel.family == AF_UNSPEC) { inner_mode = xfrm_ip2inner_mode(x, XFRM_MODE_SKB_CB(skb)->protocol); @@ -372,7 +372,7 @@ static int vti6_rcv_cb(struct sk_buff *skb, int err) } } - family = inner_mode->afinfo->family; + family = inner_mode->family; skb->mark = be32_to_cpu(t->parms.i_key); ret = xfrm_policy_check(NULL, XFRM_POLICY_IN, skb, family); diff --git a/net/ipv6/ip6mr.c b/net/ipv6/ip6mr.c index e4dd57976737a81549861c8ff5d9e9dcb7793a8d..4e69847ed5bef4438499a800546e22fbed2962a7 100644 --- a/net/ipv6/ip6mr.c +++ b/net/ipv6/ip6mr.c @@ -355,7 +355,6 @@ static const struct rhashtable_params ip6mr_rht_params = { .key_offset = offsetof(struct mfc6_cache, cmparg), .key_len = sizeof(struct mfc6_cache_cmp_arg), .nelem_hint = 3, - .locks_mul = 1, .obj_cmpfn = ip6mr_hash_cmp, .automatic_shrinking = true, }; diff --git a/net/ipv6/ndisc.c b/net/ipv6/ndisc.c index 659ecf4e4b3cfe8362d1bae95782cda551276611..4c8e2ea8bf193ac006a7dcb4e7e94dff2a394b0e 100644 --- a/net/ipv6/ndisc.c +++ b/net/ipv6/ndisc.c @@ -77,6 +77,8 @@ static u32 ndisc_hash(const void *pkey, const struct net_device *dev, __u32 *hash_rnd); static bool ndisc_key_eq(const struct neighbour *neigh, const void *pkey); +static bool ndisc_allow_add(const struct net_device *dev, + struct netlink_ext_ack *extack); static int ndisc_constructor(struct neighbour *neigh); static void ndisc_solicit(struct neighbour *neigh, struct sk_buff *skb); static void ndisc_error_report(struct neighbour *neigh, struct sk_buff *skb); @@ -117,6 +119,7 @@ struct neigh_table nd_tbl = { .pconstructor = pndisc_constructor, .pdestructor = pndisc_destructor, .proxy_redo = pndisc_redo, + .allow_add = ndisc_allow_add, .id = "ndisc_cache", .parms = { .tbl = &nd_tbl, @@ -392,6 +395,20 @@ static void pndisc_destructor(struct pneigh_entry *n) ipv6_dev_mc_dec(dev, &maddr); } +/* called with rtnl held */ +static bool ndisc_allow_add(const struct net_device *dev, + struct netlink_ext_ack *extack) +{ + struct inet6_dev *idev = __in6_dev_get(dev); + + if (!idev || idev->cnf.disable_ipv6) { + NL_SET_ERR_MSG(extack, "IPv6 is disabled on this device"); + return false; + } + + return true; +} + static struct sk_buff *ndisc_alloc_skb(struct net_device *dev, int len) { @@ -1276,8 +1293,8 @@ static void ndisc_router_discovery(struct sk_buff *skb) rt = rt6_get_dflt_router(net, &ipv6_hdr(skb)->saddr, skb->dev); if (rt) { - neigh = ip6_neigh_lookup(&rt->fib6_nh.nh_gw, - rt->fib6_nh.nh_dev, NULL, + neigh = ip6_neigh_lookup(&rt->fib6_nh.fib_nh_gw6, + rt->fib6_nh.fib_nh_dev, NULL, &ipv6_hdr(skb)->saddr); if (!neigh) { ND_PRINTK(0, err, @@ -1306,8 +1323,8 @@ static void ndisc_router_discovery(struct sk_buff *skb) return; } - neigh = ip6_neigh_lookup(&rt->fib6_nh.nh_gw, - rt->fib6_nh.nh_dev, NULL, + neigh = ip6_neigh_lookup(&rt->fib6_nh.fib_nh_gw6, + rt->fib6_nh.fib_nh_dev, NULL, &ipv6_hdr(skb)->saddr); if (!neigh) { ND_PRINTK(0, err, diff --git a/net/ipv6/netfilter/Kconfig b/net/ipv6/netfilter/Kconfig index ddc99a1653aa790352f9c1a4b11edcb61abbcd1a..086fc669279ecdf71d4724436aaa1c54796cda4c 100644 --- a/net/ipv6/netfilter/Kconfig +++ b/net/ipv6/netfilter/Kconfig @@ -23,14 +23,6 @@ config NF_TABLES_IPV6 if NF_TABLES_IPV6 -config NFT_CHAIN_ROUTE_IPV6 - tristate "IPv6 nf_tables route chain support" - help - This option enables the "route" chain for IPv6 in nf_tables. This - chain type is used to force packet re-routing after mangling header - fields such as the source, destination, flowlabel, hop-limit and - the packet mark. - config NFT_REJECT_IPV6 select NF_REJECT_IPV6 default NFT_REJECT @@ -278,15 +270,10 @@ if IP6_NF_NAT config IP6_NF_TARGET_MASQUERADE tristate "MASQUERADE target support" - select NF_NAT_MASQUERADE + select NETFILTER_XT_TARGET_MASQUERADE help - Masquerading is a special case of NAT: all outgoing connections are - changed to seem to come from a particular interface's address, and - if the interface goes down, those connections are lost. This is - only useful for dialup accounts with dynamic IP address (ie. your IP - address will be different on next dialup). - - To compile it as a module, choose M here. If unsure, say N. + This is a backwards-compat option for the user's convenience + (e.g. when running oldconfig). It selects NETFILTER_XT_TARGET_MASQUERADE. config IP6_NF_TARGET_NPT tristate "NPT (Network Prefix translation) target support" diff --git a/net/ipv6/netfilter/Makefile b/net/ipv6/netfilter/Makefile index 3853c648ebaaa5fe0c5fad8d3fb36bab5e3a5977..731a74c60dca36c64c7a71872e63d4123c51ff05 100644 --- a/net/ipv6/netfilter/Makefile +++ b/net/ipv6/netfilter/Makefile @@ -27,7 +27,6 @@ obj-$(CONFIG_NF_REJECT_IPV6) += nf_reject_ipv6.o obj-$(CONFIG_NF_DUP_IPV6) += nf_dup_ipv6.o # nf_tables -obj-$(CONFIG_NFT_CHAIN_ROUTE_IPV6) += nft_chain_route_ipv6.o obj-$(CONFIG_NFT_REJECT_IPV6) += nft_reject_ipv6.o obj-$(CONFIG_NFT_DUP_IPV6) += nft_dup_ipv6.o obj-$(CONFIG_NFT_FIB_IPV6) += nft_fib_ipv6.o @@ -47,7 +46,6 @@ obj-$(CONFIG_IP6_NF_MATCH_RT) += ip6t_rt.o obj-$(CONFIG_IP6_NF_MATCH_SRH) += ip6t_srh.o # targets -obj-$(CONFIG_IP6_NF_TARGET_MASQUERADE) += ip6t_MASQUERADE.o obj-$(CONFIG_IP6_NF_TARGET_NPT) += ip6t_NPT.o obj-$(CONFIG_IP6_NF_TARGET_REJECT) += ip6t_REJECT.o obj-$(CONFIG_IP6_NF_TARGET_SYNPROXY) += ip6t_SYNPROXY.o diff --git a/net/ipv6/netfilter/ip6t_MASQUERADE.c b/net/ipv6/netfilter/ip6t_MASQUERADE.c deleted file mode 100644 index 29c7f1915a96cba89c357eeaa9de0b8c29aa9f28..0000000000000000000000000000000000000000 --- a/net/ipv6/netfilter/ip6t_MASQUERADE.c +++ /dev/null @@ -1,81 +0,0 @@ -/* - * Copyright (c) 2011 Patrick McHardy - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * Based on Rusty Russell's IPv6 MASQUERADE target. Development of IPv6 - * NAT funded by Astaro. - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -static unsigned int -masquerade_tg6(struct sk_buff *skb, const struct xt_action_param *par) -{ - return nf_nat_masquerade_ipv6(skb, par->targinfo, xt_out(par)); -} - -static int masquerade_tg6_checkentry(const struct xt_tgchk_param *par) -{ - const struct nf_nat_range2 *range = par->targinfo; - - if (range->flags & NF_NAT_RANGE_MAP_IPS) - return -EINVAL; - return nf_ct_netns_get(par->net, par->family); -} - -static void masquerade_tg6_destroy(const struct xt_tgdtor_param *par) -{ - nf_ct_netns_put(par->net, par->family); -} - -static struct xt_target masquerade_tg6_reg __read_mostly = { - .name = "MASQUERADE", - .family = NFPROTO_IPV6, - .checkentry = masquerade_tg6_checkentry, - .destroy = masquerade_tg6_destroy, - .target = masquerade_tg6, - .targetsize = sizeof(struct nf_nat_range), - .table = "nat", - .hooks = 1 << NF_INET_POST_ROUTING, - .me = THIS_MODULE, -}; - -static int __init masquerade_tg6_init(void) -{ - int err; - - err = xt_register_target(&masquerade_tg6_reg); - if (err) - return err; - - err = nf_nat_masquerade_ipv6_register_notifier(); - if (err) - xt_unregister_target(&masquerade_tg6_reg); - - return err; -} -static void __exit masquerade_tg6_exit(void) -{ - nf_nat_masquerade_ipv6_unregister_notifier(); - xt_unregister_target(&masquerade_tg6_reg); -} - -module_init(masquerade_tg6_init); -module_exit(masquerade_tg6_exit); - -MODULE_LICENSE("GPL"); -MODULE_AUTHOR("Patrick McHardy "); -MODULE_DESCRIPTION("Xtables: automatic address SNAT"); diff --git a/net/ipv6/netfilter/nft_chain_route_ipv6.c b/net/ipv6/netfilter/nft_chain_route_ipv6.c deleted file mode 100644 index da3f1f8cb325cbdcb26665abe21fd818384769e1..0000000000000000000000000000000000000000 --- a/net/ipv6/netfilter/nft_chain_route_ipv6.c +++ /dev/null @@ -1,91 +0,0 @@ -/* - * Copyright (c) 2008 Patrick McHardy - * Copyright (c) 2012 Pablo Neira Ayuso - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * Development of this code funded by Astaro AG (http://www.astaro.com/) - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -static unsigned int nf_route_table_hook(void *priv, - struct sk_buff *skb, - const struct nf_hook_state *state) -{ - unsigned int ret; - struct nft_pktinfo pkt; - struct in6_addr saddr, daddr; - u_int8_t hop_limit; - u32 mark, flowlabel; - int err; - - nft_set_pktinfo(&pkt, skb, state); - nft_set_pktinfo_ipv6(&pkt, skb); - - /* save source/dest address, mark, hoplimit, flowlabel, priority */ - memcpy(&saddr, &ipv6_hdr(skb)->saddr, sizeof(saddr)); - memcpy(&daddr, &ipv6_hdr(skb)->daddr, sizeof(daddr)); - mark = skb->mark; - hop_limit = ipv6_hdr(skb)->hop_limit; - - /* flowlabel and prio (includes version, which shouldn't change either */ - flowlabel = *((u32 *)ipv6_hdr(skb)); - - ret = nft_do_chain(&pkt, priv); - if (ret != NF_DROP && ret != NF_STOLEN && - (memcmp(&ipv6_hdr(skb)->saddr, &saddr, sizeof(saddr)) || - memcmp(&ipv6_hdr(skb)->daddr, &daddr, sizeof(daddr)) || - skb->mark != mark || - ipv6_hdr(skb)->hop_limit != hop_limit || - flowlabel != *((u_int32_t *)ipv6_hdr(skb)))) { - err = ip6_route_me_harder(state->net, skb); - if (err < 0) - ret = NF_DROP_ERR(err); - } - - return ret; -} - -static const struct nft_chain_type nft_chain_route_ipv6 = { - .name = "route", - .type = NFT_CHAIN_T_ROUTE, - .family = NFPROTO_IPV6, - .owner = THIS_MODULE, - .hook_mask = (1 << NF_INET_LOCAL_OUT), - .hooks = { - [NF_INET_LOCAL_OUT] = nf_route_table_hook, - }, -}; - -static int __init nft_chain_route_init(void) -{ - nft_register_chain_type(&nft_chain_route_ipv6); - - return 0; -} - -static void __exit nft_chain_route_exit(void) -{ - nft_unregister_chain_type(&nft_chain_route_ipv6); -} - -module_init(nft_chain_route_init); -module_exit(nft_chain_route_exit); - -MODULE_LICENSE("GPL"); -MODULE_AUTHOR("Patrick McHardy "); -MODULE_ALIAS_NFT_CHAIN(AF_INET6, "route"); diff --git a/net/ipv6/output_core.c b/net/ipv6/output_core.c index 4fe7c90962ddae3356200376aa911bab6d75bb48..868ae23dbae195de0e81fcad2566af2420294d7e 100644 --- a/net/ipv6/output_core.c +++ b/net/ipv6/output_core.c @@ -10,15 +10,25 @@ #include #include -static u32 __ipv6_select_ident(struct net *net, u32 hashrnd, +static u32 __ipv6_select_ident(struct net *net, const struct in6_addr *dst, const struct in6_addr *src) { + const struct { + struct in6_addr dst; + struct in6_addr src; + } __aligned(SIPHASH_ALIGNMENT) combined = { + .dst = *dst, + .src = *src, + }; u32 hash, id; - hash = __ipv6_addr_jhash(dst, hashrnd); - hash = __ipv6_addr_jhash(src, hash); - hash ^= net_hash_mix(net); + /* Note the following code is not safe, but this is okay. */ + if (unlikely(siphash_key_is_zero(&net->ipv4.ip_id_key))) + get_random_bytes(&net->ipv4.ip_id_key, + sizeof(net->ipv4.ip_id_key)); + + hash = siphash(&combined, sizeof(combined), &net->ipv4.ip_id_key); /* Treat id of 0 as unset and if we get 0 back from ip_idents_reserve, * set the hight order instead thus minimizing possible future @@ -41,7 +51,6 @@ static u32 __ipv6_select_ident(struct net *net, u32 hashrnd, */ __be32 ipv6_proxy_select_ident(struct net *net, struct sk_buff *skb) { - static u32 ip6_proxy_idents_hashrnd __read_mostly; struct in6_addr buf[2]; struct in6_addr *addrs; u32 id; @@ -53,11 +62,7 @@ __be32 ipv6_proxy_select_ident(struct net *net, struct sk_buff *skb) if (!addrs) return 0; - net_get_random_once(&ip6_proxy_idents_hashrnd, - sizeof(ip6_proxy_idents_hashrnd)); - - id = __ipv6_select_ident(net, ip6_proxy_idents_hashrnd, - &addrs[1], &addrs[0]); + id = __ipv6_select_ident(net, &addrs[1], &addrs[0]); return htonl(id); } EXPORT_SYMBOL_GPL(ipv6_proxy_select_ident); @@ -66,12 +71,9 @@ __be32 ipv6_select_ident(struct net *net, const struct in6_addr *daddr, const struct in6_addr *saddr) { - static u32 ip6_idents_hashrnd __read_mostly; u32 id; - net_get_random_once(&ip6_idents_hashrnd, sizeof(ip6_idents_hashrnd)); - - id = __ipv6_select_ident(net, ip6_idents_hashrnd, daddr, saddr); + id = __ipv6_select_ident(net, daddr, saddr); return htonl(id); } EXPORT_SYMBOL(ipv6_select_ident); diff --git a/net/ipv6/raw.c b/net/ipv6/raw.c index 5a426226c762dd9e401fe7e90fde0ce2fd0e6dc6..84dbe21b71e5ea39bda5fc73003501e0bb30e6f8 100644 --- a/net/ipv6/raw.c +++ b/net/ipv6/raw.c @@ -1356,6 +1356,7 @@ const struct proto_ops inet6_sockraw_ops = { .getname = inet6_getname, .poll = datagram_poll, /* ok */ .ioctl = inet6_ioctl, /* must change */ + .gettstamp = sock_gettstamp, .listen = sock_no_listen, /* ok */ .shutdown = inet_shutdown, /* ok */ .setsockopt = sock_common_setsockopt, /* ok */ diff --git a/net/ipv6/route.c b/net/ipv6/route.c index 0520aca3354b8e47d24e14c7ff2949892435f138..23a20d62daac29e3252725b8cf95d1d1c2b567c4 100644 --- a/net/ipv6/route.c +++ b/net/ipv6/route.c @@ -59,7 +59,7 @@ #include #include #include -#include +#include #include #include #include @@ -102,14 +102,15 @@ static void ip6_rt_update_pmtu(struct dst_entry *dst, struct sock *sk, struct sk_buff *skb, u32 mtu); static void rt6_do_redirect(struct dst_entry *dst, struct sock *sk, struct sk_buff *skb); -static int rt6_score_route(struct fib6_info *rt, int oif, int strict); +static int rt6_score_route(const struct fib6_nh *nh, u32 fib6_flags, int oif, + int strict); static size_t rt6_nlmsg_size(struct fib6_info *rt); static int rt6_fill_node(struct net *net, struct sk_buff *skb, struct fib6_info *rt, struct dst_entry *dst, struct in6_addr *dest, struct in6_addr *src, int iif, int type, u32 portid, u32 seq, unsigned int flags); -static struct rt6_info *rt6_find_cached_rt(struct fib6_info *rt, +static struct rt6_info *rt6_find_cached_rt(const struct fib6_result *res, struct in6_addr *daddr, struct in6_addr *saddr); @@ -295,7 +296,7 @@ static const struct fib6_info fib6_null_entry_template = { .fib6_flags = (RTF_REJECT | RTF_NONEXTHOP), .fib6_protocol = RTPROT_KERNEL, .fib6_metric = ~(u32)0, - .fib6_ref = ATOMIC_INIT(1), + .fib6_ref = REFCOUNT_INIT(1), .fib6_type = RTN_UNREACHABLE, .fib6_metrics = (struct dst_metrics *)&dst_default_metrics, }; @@ -424,13 +425,15 @@ static bool rt6_check_expired(const struct rt6_info *rt) return false; } -struct fib6_info *fib6_multipath_select(const struct net *net, - struct fib6_info *match, - struct flowi6 *fl6, int oif, - const struct sk_buff *skb, - int strict) +void fib6_select_path(const struct net *net, struct fib6_result *res, + struct flowi6 *fl6, int oif, bool have_oif_match, + const struct sk_buff *skb, int strict) { struct fib6_info *sibling, *next_sibling; + struct fib6_info *match = res->f6i; + + if (!match->fib6_nsiblings || have_oif_match) + goto out; /* We might have already computed the hash for ICMPv6 errors. In such * case it will always be non-zero. Otherwise now is the time to do it. @@ -438,61 +441,89 @@ struct fib6_info *fib6_multipath_select(const struct net *net, if (!fl6->mp_hash) fl6->mp_hash = rt6_multipath_hash(net, fl6, skb, NULL); - if (fl6->mp_hash <= atomic_read(&match->fib6_nh.nh_upper_bound)) - return match; + if (fl6->mp_hash <= atomic_read(&match->fib6_nh.fib_nh_upper_bound)) + goto out; list_for_each_entry_safe(sibling, next_sibling, &match->fib6_siblings, fib6_siblings) { + const struct fib6_nh *nh = &sibling->fib6_nh; int nh_upper_bound; - nh_upper_bound = atomic_read(&sibling->fib6_nh.nh_upper_bound); + nh_upper_bound = atomic_read(&nh->fib_nh_upper_bound); if (fl6->mp_hash > nh_upper_bound) continue; - if (rt6_score_route(sibling, oif, strict) < 0) + if (rt6_score_route(nh, sibling->fib6_flags, oif, strict) < 0) break; match = sibling; break; } - return match; +out: + res->f6i = match; + res->nh = &match->fib6_nh; } /* * Route lookup. rcu_read_lock() should be held. */ -static inline struct fib6_info *rt6_device_match(struct net *net, - struct fib6_info *rt, - const struct in6_addr *saddr, - int oif, - int flags) +static bool __rt6_device_match(struct net *net, const struct fib6_nh *nh, + const struct in6_addr *saddr, int oif, int flags) { - struct fib6_info *sprt; + const struct net_device *dev; - if (!oif && ipv6_addr_any(saddr) && - !(rt->fib6_nh.nh_flags & RTNH_F_DEAD)) - return rt; + if (nh->fib_nh_flags & RTNH_F_DEAD) + return false; - for (sprt = rt; sprt; sprt = rcu_dereference(sprt->fib6_next)) { - const struct net_device *dev = sprt->fib6_nh.nh_dev; + dev = nh->fib_nh_dev; + if (oif) { + if (dev->ifindex == oif) + return true; + } else { + if (ipv6_chk_addr(net, saddr, dev, + flags & RT6_LOOKUP_F_IFACE)) + return true; + } - if (sprt->fib6_nh.nh_flags & RTNH_F_DEAD) - continue; + return false; +} - if (oif) { - if (dev->ifindex == oif) - return sprt; - } else { - if (ipv6_chk_addr(net, saddr, dev, - flags & RT6_LOOKUP_F_IFACE)) - return sprt; +static void rt6_device_match(struct net *net, struct fib6_result *res, + const struct in6_addr *saddr, int oif, int flags) +{ + struct fib6_info *f6i = res->f6i; + struct fib6_info *spf6i; + struct fib6_nh *nh; + + if (!oif && ipv6_addr_any(saddr)) { + nh = &f6i->fib6_nh; + if (!(nh->fib_nh_flags & RTNH_F_DEAD)) + goto out; + } + + for (spf6i = f6i; spf6i; spf6i = rcu_dereference(spf6i->fib6_next)) { + nh = &spf6i->fib6_nh; + if (__rt6_device_match(net, nh, saddr, oif, flags)) { + res->f6i = spf6i; + goto out; } } - if (oif && flags & RT6_LOOKUP_F_IFACE) - return net->ipv6.fib6_null_entry; + if (oif && flags & RT6_LOOKUP_F_IFACE) { + res->f6i = net->ipv6.fib6_null_entry; + nh = &res->f6i->fib6_nh; + goto out; + } - return rt->fib6_nh.nh_flags & RTNH_F_DEAD ? net->ipv6.fib6_null_entry : rt; + nh = &f6i->fib6_nh; + if (nh->fib_nh_flags & RTNH_F_DEAD) { + res->f6i = net->ipv6.fib6_null_entry; + nh = &res->f6i->fib6_nh; + } +out: + res->nh = nh; + res->fib6_type = res->f6i->fib6_type; + res->fib6_flags = res->f6i->fib6_flags; } #ifdef CONFIG_IPV6_ROUTER_PREF @@ -514,7 +545,7 @@ static void rt6_probe_deferred(struct work_struct *w) kfree(work); } -static void rt6_probe(struct fib6_info *rt) +static void rt6_probe(struct fib6_nh *fib6_nh) { struct __rt6_probe_work *work = NULL; const struct in6_addr *nh_gw; @@ -530,11 +561,11 @@ static void rt6_probe(struct fib6_info *rt) * Router Reachability Probe MUST be rate-limited * to no more than one per minute. */ - if (!rt || !(rt->fib6_flags & RTF_GATEWAY)) + if (fib6_nh->fib_nh_gw_family) return; - nh_gw = &rt->fib6_nh.nh_gw; - dev = rt->fib6_nh.nh_dev; + nh_gw = &fib6_nh->fib_nh_gw6; + dev = fib6_nh->fib_nh_dev; rcu_read_lock_bh(); idev = __in6_dev_get(dev); neigh = __ipv6_neigh_lookup_noref(dev, nh_gw); @@ -551,13 +582,13 @@ static void rt6_probe(struct fib6_info *rt) __neigh_set_probe_once(neigh); } write_unlock(&neigh->lock); - } else if (time_after(jiffies, rt->last_probe + + } else if (time_after(jiffies, fib6_nh->last_probe + idev->cnf.rtr_probe_interval)) { work = kmalloc(sizeof(*work), GFP_ATOMIC); } if (work) { - rt->last_probe = jiffies; + fib6_nh->last_probe = jiffies; INIT_WORK(&work->work, rt6_probe_deferred); work->target = *nh_gw; dev_hold(dev); @@ -569,7 +600,7 @@ out: rcu_read_unlock_bh(); } #else -static inline void rt6_probe(struct fib6_info *rt) +static inline void rt6_probe(struct fib6_nh *fib6_nh) { } #endif @@ -577,27 +608,14 @@ static inline void rt6_probe(struct fib6_info *rt) /* * Default Router Selection (RFC 2461 6.3.6) */ -static inline int rt6_check_dev(struct fib6_info *rt, int oif) -{ - const struct net_device *dev = rt->fib6_nh.nh_dev; - - if (!oif || dev->ifindex == oif) - return 2; - return 0; -} - -static inline enum rt6_nud_state rt6_check_neigh(struct fib6_info *rt) +static enum rt6_nud_state rt6_check_neigh(const struct fib6_nh *fib6_nh) { enum rt6_nud_state ret = RT6_NUD_FAIL_HARD; struct neighbour *neigh; - if (rt->fib6_flags & RTF_NONEXTHOP || - !(rt->fib6_flags & RTF_GATEWAY)) - return RT6_NUD_SUCCEED; - rcu_read_lock_bh(); - neigh = __ipv6_neigh_lookup_noref(rt->fib6_nh.nh_dev, - &rt->fib6_nh.nh_gw); + neigh = __ipv6_neigh_lookup_noref(fib6_nh->fib_nh_dev, + &fib6_nh->fib_nh_gw6); if (neigh) { read_lock(&neigh->lock); if (neigh->nud_state & NUD_VALID) @@ -618,58 +636,44 @@ static inline enum rt6_nud_state rt6_check_neigh(struct fib6_info *rt) return ret; } -static int rt6_score_route(struct fib6_info *rt, int oif, int strict) +static int rt6_score_route(const struct fib6_nh *nh, u32 fib6_flags, int oif, + int strict) { - int m; + int m = 0; + + if (!oif || nh->fib_nh_dev->ifindex == oif) + m = 2; - m = rt6_check_dev(rt, oif); if (!m && (strict & RT6_LOOKUP_F_IFACE)) return RT6_NUD_FAIL_HARD; #ifdef CONFIG_IPV6_ROUTER_PREF - m |= IPV6_DECODE_PREF(IPV6_EXTRACT_PREF(rt->fib6_flags)) << 2; + m |= IPV6_DECODE_PREF(IPV6_EXTRACT_PREF(fib6_flags)) << 2; #endif - if (strict & RT6_LOOKUP_F_REACHABLE) { - int n = rt6_check_neigh(rt); + if ((strict & RT6_LOOKUP_F_REACHABLE) && + !(fib6_flags & RTF_NONEXTHOP) && nh->fib_nh_gw_family) { + int n = rt6_check_neigh(nh); if (n < 0) return n; } return m; } -/* called with rc_read_lock held */ -static inline bool fib6_ignore_linkdown(const struct fib6_info *f6i) +static bool find_match(struct fib6_nh *nh, u32 fib6_flags, + int oif, int strict, int *mpri, bool *do_rr) { - const struct net_device *dev = fib6_info_nh_dev(f6i); + bool match_do_rr = false; bool rc = false; - - if (dev) { - const struct inet6_dev *idev = __in6_dev_get(dev); - - rc = !!idev->cnf.ignore_routes_with_linkdown; - } - - return rc; -} - -static struct fib6_info *find_match(struct fib6_info *rt, int oif, int strict, - int *mpri, struct fib6_info *match, - bool *do_rr) -{ int m; - bool match_do_rr = false; - if (rt->fib6_nh.nh_flags & RTNH_F_DEAD) + if (nh->fib_nh_flags & RTNH_F_DEAD) goto out; - if (fib6_ignore_linkdown(rt) && - rt->fib6_nh.nh_flags & RTNH_F_LINKDOWN && + if (ip6_ignore_linkdown(nh->fib_nh_dev) && + nh->fib_nh_flags & RTNH_F_LINKDOWN && !(strict & RT6_LOOKUP_F_IGNORE_LINKSTATE)) goto out; - if (fib6_check_expired(rt)) - goto out; - - m = rt6_score_route(rt, oif, strict); + m = rt6_score_route(nh, fib6_flags, oif, strict); if (m == RT6_NUD_FAIL_DO_RR) { match_do_rr = true; m = 0; /* lowest valid score */ @@ -678,67 +682,82 @@ static struct fib6_info *find_match(struct fib6_info *rt, int oif, int strict, } if (strict & RT6_LOOKUP_F_REACHABLE) - rt6_probe(rt); + rt6_probe(nh); /* note that m can be RT6_NUD_FAIL_PROBE at this point */ if (m > *mpri) { *do_rr = match_do_rr; *mpri = m; - match = rt; + rc = true; } out: - return match; + return rc; } -static struct fib6_info *find_rr_leaf(struct fib6_node *fn, - struct fib6_info *leaf, - struct fib6_info *rr_head, - u32 metric, int oif, int strict, - bool *do_rr) +static void __find_rr_leaf(struct fib6_info *f6i_start, + struct fib6_info *nomatch, u32 metric, + struct fib6_result *res, struct fib6_info **cont, + int oif, int strict, bool *do_rr, int *mpri) { - struct fib6_info *rt, *match, *cont; - int mpri = -1; + struct fib6_info *f6i; - match = NULL; - cont = NULL; - for (rt = rr_head; rt; rt = rcu_dereference(rt->fib6_next)) { - if (rt->fib6_metric != metric) { - cont = rt; - break; + for (f6i = f6i_start; + f6i && f6i != nomatch; + f6i = rcu_dereference(f6i->fib6_next)) { + struct fib6_nh *nh; + + if (cont && f6i->fib6_metric != metric) { + *cont = f6i; + return; } - match = find_match(rt, oif, strict, &mpri, match, do_rr); - } + if (fib6_check_expired(f6i)) + continue; - for (rt = leaf; rt && rt != rr_head; - rt = rcu_dereference(rt->fib6_next)) { - if (rt->fib6_metric != metric) { - cont = rt; - break; + nh = &f6i->fib6_nh; + if (find_match(nh, f6i->fib6_flags, oif, strict, mpri, do_rr)) { + res->f6i = f6i; + res->nh = nh; + res->fib6_flags = f6i->fib6_flags; + res->fib6_type = f6i->fib6_type; } - - match = find_match(rt, oif, strict, &mpri, match, do_rr); } +} - if (match || !cont) - return match; +static void find_rr_leaf(struct fib6_node *fn, struct fib6_info *leaf, + struct fib6_info *rr_head, int oif, int strict, + bool *do_rr, struct fib6_result *res) +{ + u32 metric = rr_head->fib6_metric; + struct fib6_info *cont = NULL; + int mpri = -1; - for (rt = cont; rt; rt = rcu_dereference(rt->fib6_next)) - match = find_match(rt, oif, strict, &mpri, match, do_rr); + __find_rr_leaf(rr_head, NULL, metric, res, &cont, + oif, strict, do_rr, &mpri); - return match; + __find_rr_leaf(leaf, rr_head, metric, res, &cont, + oif, strict, do_rr, &mpri); + + if (res->f6i || !cont) + return; + + __find_rr_leaf(cont, NULL, metric, res, NULL, + oif, strict, do_rr, &mpri); } -static struct fib6_info *rt6_select(struct net *net, struct fib6_node *fn, - int oif, int strict) +static void rt6_select(struct net *net, struct fib6_node *fn, int oif, + struct fib6_result *res, int strict) { struct fib6_info *leaf = rcu_dereference(fn->leaf); - struct fib6_info *match, *rt0; + struct fib6_info *rt0; bool do_rr = false; int key_plen; + /* make sure this function or its helpers sets f6i */ + res->f6i = NULL; + if (!leaf || leaf == net->ipv6.fib6_null_entry) - return net->ipv6.fib6_null_entry; + goto out; rt0 = rcu_dereference(fn->rr_ptr); if (!rt0) @@ -755,11 +774,9 @@ static struct fib6_info *rt6_select(struct net *net, struct fib6_node *fn, key_plen = rt0->fib6_src.plen; #endif if (fn->fn_bit != key_plen) - return net->ipv6.fib6_null_entry; - - match = find_rr_leaf(fn, leaf, rt0, rt0->fib6_metric, oif, strict, - &do_rr); + goto out; + find_rr_leaf(fn, leaf, rt0, oif, strict, &do_rr, res); if (do_rr) { struct fib6_info *next = rcu_dereference(rt0->fib6_next); @@ -776,12 +793,19 @@ static struct fib6_info *rt6_select(struct net *net, struct fib6_node *fn, } } - return match ? match : net->ipv6.fib6_null_entry; +out: + if (!res->f6i) { + res->f6i = net->ipv6.fib6_null_entry; + res->nh = &res->f6i->fib6_nh; + res->fib6_flags = res->f6i->fib6_flags; + res->fib6_type = res->f6i->fib6_type; + } } -static bool rt6_is_gw_or_nonexthop(const struct fib6_info *rt) +static bool rt6_is_gw_or_nonexthop(const struct fib6_result *res) { - return (rt->fib6_flags & (RTF_NONEXTHOP | RTF_GATEWAY)); + return (res->f6i->fib6_flags & RTF_NONEXTHOP) || + res->nh->fib_nh_gw_family; } #ifdef CONFIG_IPV6_ROUTE_INFO @@ -865,17 +889,17 @@ int rt6_route_rcv(struct net_device *dev, u8 *opt, int len, */ /* called with rcu_lock held */ -static struct net_device *ip6_rt_get_dev_rcu(struct fib6_info *rt) +static struct net_device *ip6_rt_get_dev_rcu(const struct fib6_result *res) { - struct net_device *dev = rt->fib6_nh.nh_dev; + struct net_device *dev = res->nh->fib_nh_dev; - if (rt->fib6_flags & (RTF_LOCAL | RTF_ANYCAST)) { + if (res->fib6_flags & (RTF_LOCAL | RTF_ANYCAST)) { /* for copies of local routes, dst->dev needs to be the * device if it is a master device, the master device if * device is enslaved, and the loopback as the default */ if (netif_is_l3_slave(dev) && - !rt6_need_strict(&rt->fib6_dst.addr)) + !rt6_need_strict(&res->f6i->fib6_dst.addr)) dev = l3mdev_master_dev_rcu(dev); else if (!netif_is_l3_master(dev)) dev = dev_net(dev)->loopback_dev; @@ -921,11 +945,11 @@ static unsigned short fib6_info_dst_flags(struct fib6_info *rt) return flags; } -static void ip6_rt_init_dst_reject(struct rt6_info *rt, struct fib6_info *ort) +static void ip6_rt_init_dst_reject(struct rt6_info *rt, u8 fib6_type) { - rt->dst.error = ip6_rt_type_to_error(ort->fib6_type); + rt->dst.error = ip6_rt_type_to_error(fib6_type); - switch (ort->fib6_type) { + switch (fib6_type) { case RTN_BLACKHOLE: rt->dst.output = dst_discard_out; rt->dst.input = dst_discard; @@ -943,26 +967,28 @@ static void ip6_rt_init_dst_reject(struct rt6_info *rt, struct fib6_info *ort) } } -static void ip6_rt_init_dst(struct rt6_info *rt, struct fib6_info *ort) +static void ip6_rt_init_dst(struct rt6_info *rt, const struct fib6_result *res) { - if (ort->fib6_flags & RTF_REJECT) { - ip6_rt_init_dst_reject(rt, ort); + struct fib6_info *f6i = res->f6i; + + if (res->fib6_flags & RTF_REJECT) { + ip6_rt_init_dst_reject(rt, res->fib6_type); return; } rt->dst.error = 0; rt->dst.output = ip6_output; - if (ort->fib6_type == RTN_LOCAL || ort->fib6_type == RTN_ANYCAST) { + if (res->fib6_type == RTN_LOCAL || res->fib6_type == RTN_ANYCAST) { rt->dst.input = ip6_input; - } else if (ipv6_addr_type(&ort->fib6_dst.addr) & IPV6_ADDR_MULTICAST) { + } else if (ipv6_addr_type(&f6i->fib6_dst.addr) & IPV6_ADDR_MULTICAST) { rt->dst.input = ip6_mc_input; } else { rt->dst.input = ip6_forward; } - if (ort->fib6_nh.nh_lwtstate) { - rt->dst.lwtstate = lwtstate_get(ort->fib6_nh.nh_lwtstate); + if (res->nh->fib_nh_lws) { + rt->dst.lwtstate = lwtstate_get(res->nh->fib_nh_lws); lwtunnel_set_redirect(&rt->dst); } @@ -977,20 +1003,25 @@ static void rt6_set_from(struct rt6_info *rt, struct fib6_info *from) ip_dst_init_metrics(&rt->dst, from->fib6_metrics); } -/* Caller must already hold reference to @ort */ -static void ip6_rt_copy_init(struct rt6_info *rt, struct fib6_info *ort) +/* Caller must already hold reference to f6i in result */ +static void ip6_rt_copy_init(struct rt6_info *rt, const struct fib6_result *res) { - struct net_device *dev = fib6_info_nh_dev(ort); + const struct fib6_nh *nh = res->nh; + const struct net_device *dev = nh->fib_nh_dev; + struct fib6_info *f6i = res->f6i; - ip6_rt_init_dst(rt, ort); + ip6_rt_init_dst(rt, res); - rt->rt6i_dst = ort->fib6_dst; + rt->rt6i_dst = f6i->fib6_dst; rt->rt6i_idev = dev ? in6_dev_get(dev) : NULL; - rt->rt6i_gateway = ort->fib6_nh.nh_gw; - rt->rt6i_flags = ort->fib6_flags; - rt6_set_from(rt, ort); + rt->rt6i_flags = res->fib6_flags; + if (nh->fib_nh_gw_family) { + rt->rt6i_gateway = nh->fib_nh_gw6; + rt->rt6i_flags |= RTF_GATEWAY; + } + rt6_set_from(rt, f6i); #ifdef CONFIG_IPV6_SUBTREES - rt->rt6i_src = ort->fib6_src; + rt->rt6i_src = f6i->fib6_src; #endif } @@ -1012,14 +1043,13 @@ static struct fib6_node* fib6_backtrack(struct fib6_node *fn, } } -static bool ip6_hold_safe(struct net *net, struct rt6_info **prt, - bool null_fallback) +static bool ip6_hold_safe(struct net *net, struct rt6_info **prt) { struct rt6_info *rt = *prt; if (dst_hold_safe(&rt->dst)) return true; - if (null_fallback) { + if (net) { rt = net->ipv6.ip6_null_entry; dst_hold(&rt->dst); } else { @@ -1030,22 +1060,24 @@ static bool ip6_hold_safe(struct net *net, struct rt6_info **prt, } /* called with rcu_lock held */ -static struct rt6_info *ip6_create_rt_rcu(struct fib6_info *rt) +static struct rt6_info *ip6_create_rt_rcu(const struct fib6_result *res) { - unsigned short flags = fib6_info_dst_flags(rt); - struct net_device *dev = rt->fib6_nh.nh_dev; + struct net_device *dev = res->nh->fib_nh_dev; + struct fib6_info *f6i = res->f6i; + unsigned short flags; struct rt6_info *nrt; - if (!fib6_info_hold_safe(rt)) + if (!fib6_info_hold_safe(f6i)) goto fallback; + flags = fib6_info_dst_flags(f6i); nrt = ip6_dst_alloc(dev_net(dev), dev, flags); if (!nrt) { - fib6_info_release(rt); + fib6_info_release(f6i); goto fallback; } - ip6_rt_copy_init(nrt, rt); + ip6_rt_copy_init(nrt, res); return nrt; fallback: @@ -1060,7 +1092,7 @@ static struct rt6_info *ip6_pol_route_lookup(struct net *net, const struct sk_buff *skb, int flags) { - struct fib6_info *f6i; + struct fib6_result res = {}; struct fib6_node *fn; struct rt6_info *rt; @@ -1070,37 +1102,38 @@ static struct rt6_info *ip6_pol_route_lookup(struct net *net, rcu_read_lock(); fn = fib6_node_lookup(&table->tb6_root, &fl6->daddr, &fl6->saddr); restart: - f6i = rcu_dereference(fn->leaf); - if (!f6i) { - f6i = net->ipv6.fib6_null_entry; - } else { - f6i = rt6_device_match(net, f6i, &fl6->saddr, - fl6->flowi6_oif, flags); - if (f6i->fib6_nsiblings && fl6->flowi6_oif == 0) - f6i = fib6_multipath_select(net, f6i, fl6, - fl6->flowi6_oif, skb, - flags); - } - if (f6i == net->ipv6.fib6_null_entry) { + res.f6i = rcu_dereference(fn->leaf); + if (!res.f6i) + res.f6i = net->ipv6.fib6_null_entry; + else + rt6_device_match(net, &res, &fl6->saddr, fl6->flowi6_oif, + flags); + + if (res.f6i == net->ipv6.fib6_null_entry) { fn = fib6_backtrack(fn, &fl6->saddr); if (fn) goto restart; + + rt = net->ipv6.ip6_null_entry; + dst_hold(&rt->dst); + goto out; } - trace_fib6_table_lookup(net, f6i, table, fl6); + fib6_select_path(net, &res, fl6, fl6->flowi6_oif, + fl6->flowi6_oif != 0, skb, flags); /* Search through exception table */ - rt = rt6_find_cached_rt(f6i, &fl6->daddr, &fl6->saddr); + rt = rt6_find_cached_rt(&res, &fl6->daddr, &fl6->saddr); if (rt) { - if (ip6_hold_safe(net, &rt, true)) + if (ip6_hold_safe(net, &rt)) dst_use_noref(&rt->dst, jiffies); - } else if (f6i == net->ipv6.fib6_null_entry) { - rt = net->ipv6.ip6_null_entry; - dst_hold(&rt->dst); } else { - rt = ip6_create_rt_rcu(f6i); + rt = ip6_create_rt_rcu(&res); } +out: + trace_fib6_table_lookup(net, &res, table, fl6); + rcu_read_unlock(); return rt; @@ -1166,10 +1199,11 @@ int ip6_ins_rt(struct net *net, struct fib6_info *rt) return __ip6_ins_rt(rt, &info, NULL); } -static struct rt6_info *ip6_rt_cache_alloc(struct fib6_info *ort, +static struct rt6_info *ip6_rt_cache_alloc(const struct fib6_result *res, const struct in6_addr *daddr, const struct in6_addr *saddr) { + struct fib6_info *f6i = res->f6i; struct net_device *dev; struct rt6_info *rt; @@ -1177,25 +1211,25 @@ static struct rt6_info *ip6_rt_cache_alloc(struct fib6_info *ort, * Clone the route. */ - if (!fib6_info_hold_safe(ort)) + if (!fib6_info_hold_safe(f6i)) return NULL; - dev = ip6_rt_get_dev_rcu(ort); + dev = ip6_rt_get_dev_rcu(res); rt = ip6_dst_alloc(dev_net(dev), dev, 0); if (!rt) { - fib6_info_release(ort); + fib6_info_release(f6i); return NULL; } - ip6_rt_copy_init(rt, ort); + ip6_rt_copy_init(rt, res); rt->rt6i_flags |= RTF_CACHE; rt->dst.flags |= DST_HOST; rt->rt6i_dst.addr = *daddr; rt->rt6i_dst.plen = 128; - if (!rt6_is_gw_or_nonexthop(ort)) { - if (ort->fib6_dst.plen != 128 && - ipv6_addr_equal(&ort->fib6_dst.addr, daddr)) + if (!rt6_is_gw_or_nonexthop(res)) { + if (f6i->fib6_dst.plen != 128 && + ipv6_addr_equal(&f6i->fib6_dst.addr, daddr)) rt->rt6i_flags |= RTF_ANYCAST; #ifdef CONFIG_IPV6_SUBTREES if (rt->rt6i_src.plen && saddr) { @@ -1208,55 +1242,56 @@ static struct rt6_info *ip6_rt_cache_alloc(struct fib6_info *ort, return rt; } -static struct rt6_info *ip6_rt_pcpu_alloc(struct fib6_info *rt) +static struct rt6_info *ip6_rt_pcpu_alloc(const struct fib6_result *res) { - unsigned short flags = fib6_info_dst_flags(rt); + struct fib6_info *f6i = res->f6i; + unsigned short flags = fib6_info_dst_flags(f6i); struct net_device *dev; struct rt6_info *pcpu_rt; - if (!fib6_info_hold_safe(rt)) + if (!fib6_info_hold_safe(f6i)) return NULL; rcu_read_lock(); - dev = ip6_rt_get_dev_rcu(rt); + dev = ip6_rt_get_dev_rcu(res); pcpu_rt = ip6_dst_alloc(dev_net(dev), dev, flags); rcu_read_unlock(); if (!pcpu_rt) { - fib6_info_release(rt); + fib6_info_release(f6i); return NULL; } - ip6_rt_copy_init(pcpu_rt, rt); + ip6_rt_copy_init(pcpu_rt, res); pcpu_rt->rt6i_flags |= RTF_PCPU; return pcpu_rt; } /* It should be called with rcu_read_lock() acquired */ -static struct rt6_info *rt6_get_pcpu_route(struct fib6_info *rt) +static struct rt6_info *rt6_get_pcpu_route(const struct fib6_result *res) { struct rt6_info *pcpu_rt, **p; - p = this_cpu_ptr(rt->rt6i_pcpu); + p = this_cpu_ptr(res->f6i->rt6i_pcpu); pcpu_rt = *p; if (pcpu_rt) - ip6_hold_safe(NULL, &pcpu_rt, false); + ip6_hold_safe(NULL, &pcpu_rt); return pcpu_rt; } static struct rt6_info *rt6_make_pcpu_route(struct net *net, - struct fib6_info *rt) + const struct fib6_result *res) { struct rt6_info *pcpu_rt, *prev, **p; - pcpu_rt = ip6_rt_pcpu_alloc(rt); + pcpu_rt = ip6_rt_pcpu_alloc(res); if (!pcpu_rt) { dst_hold(&net->ipv6.ip6_null_entry->dst); return net->ipv6.ip6_null_entry; } dst_hold(&pcpu_rt->dst); - p = this_cpu_ptr(rt->rt6i_pcpu); + p = this_cpu_ptr(res->f6i->rt6i_pcpu); prev = cmpxchg(p, NULL, pcpu_rt); BUG_ON(prev); @@ -1397,14 +1432,15 @@ __rt6_find_exception_rcu(struct rt6_exception_bucket **bucket, return NULL; } -static unsigned int fib6_mtu(const struct fib6_info *rt) +static unsigned int fib6_mtu(const struct fib6_result *res) { + const struct fib6_nh *nh = res->nh; unsigned int mtu; - if (rt->fib6_pmtu) { - mtu = rt->fib6_pmtu; + if (res->f6i->fib6_pmtu) { + mtu = res->f6i->fib6_pmtu; } else { - struct net_device *dev = fib6_info_nh_dev(rt); + struct net_device *dev = nh->fib_nh_dev; struct inet6_dev *idev; rcu_read_lock(); @@ -1415,26 +1451,27 @@ static unsigned int fib6_mtu(const struct fib6_info *rt) mtu = min_t(unsigned int, mtu, IP6_MAX_MTU); - return mtu - lwtunnel_headroom(rt->fib6_nh.nh_lwtstate, mtu); + return mtu - lwtunnel_headroom(nh->fib_nh_lws, mtu); } static int rt6_insert_exception(struct rt6_info *nrt, - struct fib6_info *ort) + const struct fib6_result *res) { struct net *net = dev_net(nrt->dst.dev); struct rt6_exception_bucket *bucket; struct in6_addr *src_key = NULL; struct rt6_exception *rt6_ex; + struct fib6_info *f6i = res->f6i; int err = 0; spin_lock_bh(&rt6_exception_lock); - if (ort->exception_bucket_flushed) { + if (f6i->exception_bucket_flushed) { err = -EINVAL; goto out; } - bucket = rcu_dereference_protected(ort->rt6i_exception_bucket, + bucket = rcu_dereference_protected(f6i->rt6i_exception_bucket, lockdep_is_held(&rt6_exception_lock)); if (!bucket) { bucket = kcalloc(FIB6_EXCEPTION_BUCKET_SIZE, sizeof(*bucket), @@ -1443,24 +1480,24 @@ static int rt6_insert_exception(struct rt6_info *nrt, err = -ENOMEM; goto out; } - rcu_assign_pointer(ort->rt6i_exception_bucket, bucket); + rcu_assign_pointer(f6i->rt6i_exception_bucket, bucket); } #ifdef CONFIG_IPV6_SUBTREES - /* rt6i_src.plen != 0 indicates ort is in subtree + /* fib6_src.plen != 0 indicates f6i is in subtree * and exception table is indexed by a hash of - * both rt6i_dst and rt6i_src. + * both fib6_dst and fib6_src. * Otherwise, the exception table is indexed by - * a hash of only rt6i_dst. + * a hash of only fib6_dst. */ - if (ort->fib6_src.plen) + if (f6i->fib6_src.plen) src_key = &nrt->rt6i_src.addr; #endif - /* rt6_mtu_change() might lower mtu on ort. + /* rt6_mtu_change() might lower mtu on f6i. * Only insert this exception route if its mtu - * is less than ort's mtu value. + * is less than f6i's mtu value. */ - if (dst_metric_raw(&nrt->dst, RTAX_MTU) >= fib6_mtu(ort)) { + if (dst_metric_raw(&nrt->dst, RTAX_MTU) >= fib6_mtu(res)) { err = -EINVAL; goto out; } @@ -1489,9 +1526,9 @@ out: /* Update fn->fn_sernum to invalidate all cached dst */ if (!err) { - spin_lock_bh(&ort->fib6_table->tb6_lock); - fib6_update_sernum(net, ort); - spin_unlock_bh(&ort->fib6_table->tb6_lock); + spin_lock_bh(&f6i->fib6_table->tb6_lock); + fib6_update_sernum(net, f6i); + spin_unlock_bh(&f6i->fib6_table->tb6_lock); fib6_force_start_gc(net); } @@ -1528,33 +1565,33 @@ out: /* Find cached rt in the hash table inside passed in rt * Caller has to hold rcu_read_lock() */ -static struct rt6_info *rt6_find_cached_rt(struct fib6_info *rt, +static struct rt6_info *rt6_find_cached_rt(const struct fib6_result *res, struct in6_addr *daddr, struct in6_addr *saddr) { struct rt6_exception_bucket *bucket; struct in6_addr *src_key = NULL; struct rt6_exception *rt6_ex; - struct rt6_info *res = NULL; + struct rt6_info *ret = NULL; - bucket = rcu_dereference(rt->rt6i_exception_bucket); + bucket = rcu_dereference(res->f6i->rt6i_exception_bucket); #ifdef CONFIG_IPV6_SUBTREES - /* rt6i_src.plen != 0 indicates rt is in subtree + /* fib6i_src.plen != 0 indicates f6i is in subtree * and exception table is indexed by a hash of - * both rt6i_dst and rt6i_src. + * both fib6_dst and fib6_src. * Otherwise, the exception table is indexed by - * a hash of only rt6i_dst. + * a hash of only fib6_dst. */ - if (rt->fib6_src.plen) + if (res->f6i->fib6_src.plen) src_key = saddr; #endif rt6_ex = __rt6_find_exception_rcu(&bucket, daddr, src_key); if (rt6_ex && !rt6_check_expired(rt6_ex->rt6i)) - res = rt6_ex->rt6i; + ret = rt6_ex->rt6i; - return res; + return ret; } /* Remove the passed in cached rt from the hash table that contains it */ @@ -1802,11 +1839,10 @@ void rt6_age_exceptions(struct fib6_info *rt, } /* must be called with rcu lock held */ -struct fib6_info *fib6_table_lookup(struct net *net, struct fib6_table *table, - int oif, struct flowi6 *fl6, int strict) +int fib6_table_lookup(struct net *net, struct fib6_table *table, int oif, + struct flowi6 *fl6, struct fib6_result *res, int strict) { struct fib6_node *fn, *saved_fn; - struct fib6_info *f6i; fn = fib6_node_lookup(&table->tb6_root, &fl6->daddr, &fl6->saddr); saved_fn = fn; @@ -1815,8 +1851,8 @@ struct fib6_info *fib6_table_lookup(struct net *net, struct fib6_table *table, oif = 0; redo_rt6_select: - f6i = rt6_select(net, fn, oif, strict); - if (f6i == net->ipv6.fib6_null_entry) { + rt6_select(net, fn, oif, res, strict); + if (res->f6i == net->ipv6.fib6_null_entry) { fn = fib6_backtrack(fn, &fl6->saddr); if (fn) goto redo_rt6_select; @@ -1828,16 +1864,16 @@ redo_rt6_select: } } - trace_fib6_table_lookup(net, f6i, table, fl6); + trace_fib6_table_lookup(net, res, table, fl6); - return f6i; + return 0; } struct rt6_info *ip6_pol_route(struct net *net, struct fib6_table *table, int oif, struct flowi6 *fl6, const struct sk_buff *skb, int flags) { - struct fib6_info *f6i; + struct fib6_result res = {}; struct rt6_info *rt; int strict = 0; @@ -1848,27 +1884,26 @@ struct rt6_info *ip6_pol_route(struct net *net, struct fib6_table *table, rcu_read_lock(); - f6i = fib6_table_lookup(net, table, oif, fl6, strict); - if (f6i->fib6_nsiblings) - f6i = fib6_multipath_select(net, f6i, fl6, oif, skb, strict); - - if (f6i == net->ipv6.fib6_null_entry) { + fib6_table_lookup(net, table, oif, fl6, &res, strict); + if (res.f6i == net->ipv6.fib6_null_entry) { rt = net->ipv6.ip6_null_entry; rcu_read_unlock(); dst_hold(&rt->dst); return rt; } + fib6_select_path(net, &res, fl6, oif, false, skb, strict); + /*Search through exception table */ - rt = rt6_find_cached_rt(f6i, &fl6->daddr, &fl6->saddr); + rt = rt6_find_cached_rt(&res, &fl6->daddr, &fl6->saddr); if (rt) { - if (ip6_hold_safe(net, &rt, true)) + if (ip6_hold_safe(net, &rt)) dst_use_noref(&rt->dst, jiffies); rcu_read_unlock(); return rt; } else if (unlikely((fl6->flowi6_flags & FLOWI_FLAG_KNOWN_NH) && - !(f6i->fib6_flags & RTF_GATEWAY))) { + !res.nh->fib_nh_gw_family)) { /* Create a RTF_CACHE clone which will not be * owned by the fib6 tree. It is for the special case where * the daddr in the skb during the neighbor look-up is different @@ -1876,7 +1911,7 @@ struct rt6_info *ip6_pol_route(struct net *net, struct fib6_table *table, */ struct rt6_info *uncached_rt; - uncached_rt = ip6_rt_cache_alloc(f6i, &fl6->daddr, NULL); + uncached_rt = ip6_rt_cache_alloc(&res, &fl6->daddr, NULL); rcu_read_unlock(); @@ -1898,10 +1933,10 @@ struct rt6_info *ip6_pol_route(struct net *net, struct fib6_table *table, struct rt6_info *pcpu_rt; local_bh_disable(); - pcpu_rt = rt6_get_pcpu_route(f6i); + pcpu_rt = rt6_get_pcpu_route(&res); if (!pcpu_rt) - pcpu_rt = rt6_make_pcpu_route(net, f6i); + pcpu_rt = rt6_make_pcpu_route(net, &res); local_bh_enable(); rcu_read_unlock(); @@ -2320,19 +2355,23 @@ static void __ip6_rt_update_pmtu(struct dst_entry *dst, const struct sock *sk, if (rt6->rt6i_flags & RTF_CACHE) rt6_update_exception_stamp_rt(rt6); } else if (daddr) { - struct fib6_info *from; + struct fib6_result res = {}; struct rt6_info *nrt6; rcu_read_lock(); - from = rcu_dereference(rt6->from); - if (!from) { + res.f6i = rcu_dereference(rt6->from); + if (!res.f6i) { rcu_read_unlock(); return; } - nrt6 = ip6_rt_cache_alloc(from, daddr, saddr); + res.nh = &res.f6i->fib6_nh; + res.fib6_flags = res.f6i->fib6_flags; + res.fib6_type = res.f6i->fib6_type; + + nrt6 = ip6_rt_cache_alloc(&res, daddr, saddr); if (nrt6) { rt6_do_update_pmtu(nrt6, mtu); - if (rt6_insert_exception(nrt6, from)) + if (rt6_insert_exception(nrt6, &res)) dst_release_immediate(&nrt6->dst); } rcu_read_unlock(); @@ -2405,6 +2444,36 @@ void ip6_sk_dst_store_flow(struct sock *sk, struct dst_entry *dst, NULL); } +static bool ip6_redirect_nh_match(const struct fib6_result *res, + struct flowi6 *fl6, + const struct in6_addr *gw, + struct rt6_info **ret) +{ + const struct fib6_nh *nh = res->nh; + + if (nh->fib_nh_flags & RTNH_F_DEAD || !nh->fib_nh_gw_family || + fl6->flowi6_oif != nh->fib_nh_dev->ifindex) + return false; + + /* rt_cache's gateway might be different from its 'parent' + * in the case of an ip redirect. + * So we keep searching in the exception table if the gateway + * is different. + */ + if (!ipv6_addr_equal(gw, &nh->fib_nh_gw6)) { + struct rt6_info *rt_cache; + + rt_cache = rt6_find_cached_rt(res, &fl6->daddr, &fl6->saddr); + if (rt_cache && + ipv6_addr_equal(gw, &rt_cache->rt6i_gateway)) { + *ret = rt_cache; + return true; + } + return false; + } + return true; +} + /* Handle redirects */ struct ip6rd_flowi { struct flowi6 fl6; @@ -2418,7 +2487,8 @@ static struct rt6_info *__ip6_route_redirect(struct net *net, int flags) { struct ip6rd_flowi *rdfl = (struct ip6rd_flowi *)fl6; - struct rt6_info *ret = NULL, *rt_cache; + struct rt6_info *ret = NULL; + struct fib6_result res = {}; struct fib6_info *rt; struct fib6_node *fn; @@ -2436,34 +2506,15 @@ static struct rt6_info *__ip6_route_redirect(struct net *net, fn = fib6_node_lookup(&table->tb6_root, &fl6->daddr, &fl6->saddr); restart: for_each_fib6_node_rt_rcu(fn) { - if (rt->fib6_nh.nh_flags & RTNH_F_DEAD) - continue; + res.f6i = rt; + res.nh = &rt->fib6_nh; + if (fib6_check_expired(rt)) continue; if (rt->fib6_flags & RTF_REJECT) break; - if (!(rt->fib6_flags & RTF_GATEWAY)) - continue; - if (fl6->flowi6_oif != rt->fib6_nh.nh_dev->ifindex) - continue; - /* rt_cache's gateway might be different from its 'parent' - * in the case of an ip redirect. - * So we keep searching in the exception table if the gateway - * is different. - */ - if (!ipv6_addr_equal(&rdfl->gateway, &rt->fib6_nh.nh_gw)) { - rt_cache = rt6_find_cached_rt(rt, - &fl6->daddr, - &fl6->saddr); - if (rt_cache && - ipv6_addr_equal(&rdfl->gateway, - &rt_cache->rt6i_gateway)) { - ret = rt_cache; - break; - } - continue; - } - break; + if (ip6_redirect_nh_match(&res, fl6, &rdfl->gateway, &ret)) + goto out; } if (!rt) @@ -2479,15 +2530,20 @@ restart: goto restart; } + res.f6i = rt; + res.nh = &rt->fib6_nh; out: - if (ret) - ip6_hold_safe(net, &ret, true); - else - ret = ip6_create_rt_rcu(rt); + if (ret) { + ip6_hold_safe(net, &ret); + } else { + res.fib6_flags = res.f6i->fib6_flags; + res.fib6_type = res.f6i->fib6_type; + ret = ip6_create_rt_rcu(&res); + } rcu_read_unlock(); - trace_fib6_table_lookup(net, rt, table, fl6); + trace_fib6_table_lookup(net, &res, table, fl6); return ret; }; @@ -2605,12 +2661,15 @@ out: * based on ip6_dst_mtu_forward and exception logic of * rt6_find_cached_rt; called with rcu_read_lock */ -u32 ip6_mtu_from_fib6(struct fib6_info *f6i, struct in6_addr *daddr, - struct in6_addr *saddr) +u32 ip6_mtu_from_fib6(const struct fib6_result *res, + const struct in6_addr *daddr, + const struct in6_addr *saddr) { struct rt6_exception_bucket *bucket; + const struct fib6_nh *nh = res->nh; + struct fib6_info *f6i = res->f6i; + const struct in6_addr *src_key; struct rt6_exception *rt6_ex; - struct in6_addr *src_key; struct inet6_dev *idev; u32 mtu = 0; @@ -2632,7 +2691,7 @@ u32 ip6_mtu_from_fib6(struct fib6_info *f6i, struct in6_addr *daddr, mtu = dst_metric_raw(&rt6_ex->rt6i->dst, RTAX_MTU); if (likely(!mtu)) { - struct net_device *dev = fib6_info_nh_dev(f6i); + struct net_device *dev = nh->fib_nh_dev; mtu = IPV6_MIN_MTU; idev = __in6_dev_get(dev); @@ -2642,7 +2701,7 @@ u32 ip6_mtu_from_fib6(struct fib6_info *f6i, struct in6_addr *daddr, mtu = min_t(unsigned int, mtu, IP6_MAX_MTU); out: - return mtu - lwtunnel_headroom(fib6_info_nh_lwt(f6i), mtu); + return mtu - lwtunnel_headroom(nh->fib_nh_lws, mtu); } struct dst_entry *icmp6_dst_alloc(struct net_device *dev, @@ -2898,17 +2957,143 @@ out: return err; } +static bool fib6_is_reject(u32 flags, struct net_device *dev, int addr_type) +{ + if ((flags & RTF_REJECT) || + (dev && (dev->flags & IFF_LOOPBACK) && + !(addr_type & IPV6_ADDR_LOOPBACK) && + !(flags & RTF_LOCAL))) + return true; + + return false; +} + +int fib6_nh_init(struct net *net, struct fib6_nh *fib6_nh, + struct fib6_config *cfg, gfp_t gfp_flags, + struct netlink_ext_ack *extack) +{ + struct net_device *dev = NULL; + struct inet6_dev *idev = NULL; + int addr_type; + int err; + + fib6_nh->fib_nh_family = AF_INET6; + + err = -ENODEV; + if (cfg->fc_ifindex) { + dev = dev_get_by_index(net, cfg->fc_ifindex); + if (!dev) + goto out; + idev = in6_dev_get(dev); + if (!idev) + goto out; + } + + if (cfg->fc_flags & RTNH_F_ONLINK) { + if (!dev) { + NL_SET_ERR_MSG(extack, + "Nexthop device required for onlink"); + goto out; + } + + if (!(dev->flags & IFF_UP)) { + NL_SET_ERR_MSG(extack, "Nexthop device is not up"); + err = -ENETDOWN; + goto out; + } + + fib6_nh->fib_nh_flags |= RTNH_F_ONLINK; + } + + fib6_nh->fib_nh_weight = 1; + + /* We cannot add true routes via loopback here, + * they would result in kernel looping; promote them to reject routes + */ + addr_type = ipv6_addr_type(&cfg->fc_dst); + if (fib6_is_reject(cfg->fc_flags, dev, addr_type)) { + /* hold loopback dev/idev if we haven't done so. */ + if (dev != net->loopback_dev) { + if (dev) { + dev_put(dev); + in6_dev_put(idev); + } + dev = net->loopback_dev; + dev_hold(dev); + idev = in6_dev_get(dev); + if (!idev) { + err = -ENODEV; + goto out; + } + } + goto set_dev; + } + + if (cfg->fc_flags & RTF_GATEWAY) { + err = ip6_validate_gw(net, cfg, &dev, &idev, extack); + if (err) + goto out; + + fib6_nh->fib_nh_gw6 = cfg->fc_gateway; + fib6_nh->fib_nh_gw_family = AF_INET6; + } + + err = -ENODEV; + if (!dev) + goto out; + + if (idev->cnf.disable_ipv6) { + NL_SET_ERR_MSG(extack, "IPv6 is disabled on nexthop device"); + err = -EACCES; + goto out; + } + + if (!(dev->flags & IFF_UP) && !cfg->fc_ignore_dev_down) { + NL_SET_ERR_MSG(extack, "Nexthop device is not up"); + err = -ENETDOWN; + goto out; + } + + if (!(cfg->fc_flags & (RTF_LOCAL | RTF_ANYCAST)) && + !netif_carrier_ok(dev)) + fib6_nh->fib_nh_flags |= RTNH_F_LINKDOWN; + + err = fib_nh_common_init(&fib6_nh->nh_common, cfg->fc_encap, + cfg->fc_encap_type, cfg, gfp_flags, extack); + if (err) + goto out; +set_dev: + fib6_nh->fib_nh_dev = dev; + fib6_nh->fib_nh_oif = dev->ifindex; + err = 0; +out: + if (idev) + in6_dev_put(idev); + + if (err) { + lwtstate_put(fib6_nh->fib_nh_lws); + fib6_nh->fib_nh_lws = NULL; + if (dev) + dev_put(dev); + } + + return err; +} + +void fib6_nh_release(struct fib6_nh *fib6_nh) +{ + fib_nh_common_release(&fib6_nh->nh_common); +} + static struct fib6_info *ip6_route_info_create(struct fib6_config *cfg, gfp_t gfp_flags, struct netlink_ext_ack *extack) { struct net *net = cfg->fc_nlinfo.nl_net; struct fib6_info *rt = NULL; - struct net_device *dev = NULL; - struct inet6_dev *idev = NULL; struct fib6_table *table; - int addr_type; int err = -EINVAL; + int addr_type; /* RTF_PCPU is an internal flag; can not be set by userspace */ if (cfg->fc_flags & RTF_PCPU) { @@ -2942,33 +3127,6 @@ static struct fib6_info *ip6_route_info_create(struct fib6_config *cfg, goto out; } #endif - if (cfg->fc_ifindex) { - err = -ENODEV; - dev = dev_get_by_index(net, cfg->fc_ifindex); - if (!dev) - goto out; - idev = in6_dev_get(dev); - if (!idev) - goto out; - } - - if (cfg->fc_metric == 0) - cfg->fc_metric = IP6_RT_PRIO_USER; - - if (cfg->fc_flags & RTNH_F_ONLINK) { - if (!dev) { - NL_SET_ERR_MSG(extack, - "Nexthop device required for onlink"); - err = -ENODEV; - goto out; - } - - if (!(dev->flags & IFF_UP)) { - NL_SET_ERR_MSG(extack, "Nexthop device is not up"); - err = -ENETDOWN; - goto out; - } - } err = -ENOBUFS; if (cfg->fc_nlinfo.nlh && @@ -3012,18 +3170,10 @@ static struct fib6_info *ip6_route_info_create(struct fib6_config *cfg, cfg->fc_protocol = RTPROT_BOOT; rt->fib6_protocol = cfg->fc_protocol; - addr_type = ipv6_addr_type(&cfg->fc_dst); - - if (cfg->fc_encap) { - struct lwtunnel_state *lwtstate; - - err = lwtunnel_build_state(cfg->fc_encap_type, - cfg->fc_encap, AF_INET6, cfg, - &lwtstate, extack); - if (err) - goto out; - rt->fib6_nh.nh_lwtstate = lwtstate_get(lwtstate); - } + rt->fib6_table = table; + rt->fib6_metric = cfg->fc_metric; + rt->fib6_type = cfg->fc_type; + rt->fib6_flags = cfg->fc_flags & ~RTF_GATEWAY; ipv6_addr_prefix(&rt->fib6_dst.addr, &cfg->fc_dst, cfg->fc_dst_len); rt->fib6_dst.plen = cfg->fc_dst_len; @@ -3034,62 +3184,20 @@ static struct fib6_info *ip6_route_info_create(struct fib6_config *cfg, ipv6_addr_prefix(&rt->fib6_src.addr, &cfg->fc_src, cfg->fc_src_len); rt->fib6_src.plen = cfg->fc_src_len; #endif - - rt->fib6_metric = cfg->fc_metric; - rt->fib6_nh.nh_weight = 1; - - rt->fib6_type = cfg->fc_type; + err = fib6_nh_init(net, &rt->fib6_nh, cfg, gfp_flags, extack); + if (err) + goto out; /* We cannot add true routes via loopback here, - they would result in kernel looping; promote them to reject routes + * they would result in kernel looping; promote them to reject routes */ - if ((cfg->fc_flags & RTF_REJECT) || - (dev && (dev->flags & IFF_LOOPBACK) && - !(addr_type & IPV6_ADDR_LOOPBACK) && - !(cfg->fc_flags & RTF_LOCAL))) { - /* hold loopback dev/idev if we haven't done so. */ - if (dev != net->loopback_dev) { - if (dev) { - dev_put(dev); - in6_dev_put(idev); - } - dev = net->loopback_dev; - dev_hold(dev); - idev = in6_dev_get(dev); - if (!idev) { - err = -ENODEV; - goto out; - } - } - rt->fib6_flags = RTF_REJECT|RTF_NONEXTHOP; - goto install_route; - } - - if (cfg->fc_flags & RTF_GATEWAY) { - err = ip6_validate_gw(net, cfg, &dev, &idev, extack); - if (err) - goto out; - - rt->fib6_nh.nh_gw = cfg->fc_gateway; - } - - err = -ENODEV; - if (!dev) - goto out; - - if (idev->cnf.disable_ipv6) { - NL_SET_ERR_MSG(extack, "IPv6 is disabled on nexthop device"); - err = -EACCES; - goto out; - } - - if (!(dev->flags & IFF_UP)) { - NL_SET_ERR_MSG(extack, "Nexthop device is not up"); - err = -ENETDOWN; - goto out; - } + addr_type = ipv6_addr_type(&cfg->fc_dst); + if (fib6_is_reject(cfg->fc_flags, rt->fib6_nh.fib_nh_dev, addr_type)) + rt->fib6_flags = RTF_REJECT | RTF_NONEXTHOP; if (!ipv6_addr_any(&cfg->fc_prefsrc)) { + struct net_device *dev = fib6_info_nh_dev(rt); + if (!ipv6_chk_addr(net, &cfg->fc_prefsrc, dev, 0)) { NL_SET_ERR_MSG(extack, "Invalid source address"); err = -EINVAL; @@ -3100,26 +3208,8 @@ static struct fib6_info *ip6_route_info_create(struct fib6_config *cfg, } else rt->fib6_prefsrc.plen = 0; - rt->fib6_flags = cfg->fc_flags; - -install_route: - if (!(rt->fib6_flags & (RTF_LOCAL | RTF_ANYCAST)) && - !netif_carrier_ok(dev)) - rt->fib6_nh.nh_flags |= RTNH_F_LINKDOWN; - rt->fib6_nh.nh_flags |= (cfg->fc_flags & RTNH_F_ONLINK); - rt->fib6_nh.nh_dev = dev; - rt->fib6_table = table; - - if (idev) - in6_dev_put(idev); - return rt; out: - if (dev) - dev_put(dev); - if (idev) - in6_dev_put(idev); - fib6_info_release(rt); return ERR_PTR(err); } @@ -3260,10 +3350,16 @@ static int ip6_route_del(struct fib6_config *cfg, if (fn) { for_each_fib6_node_rt_rcu(fn) { + struct fib6_nh *nh; + if (cfg->fc_flags & RTF_CACHE) { + struct fib6_result res = { + .f6i = rt, + }; int rc; - rt_cache = rt6_find_cached_rt(rt, &cfg->fc_dst, + rt_cache = rt6_find_cached_rt(&res, + &cfg->fc_dst, &cfg->fc_src); if (rt_cache) { rc = ip6_del_cached_rt(rt_cache, cfg); @@ -3274,12 +3370,14 @@ static int ip6_route_del(struct fib6_config *cfg, } continue; } + + nh = &rt->fib6_nh; if (cfg->fc_ifindex && - (!rt->fib6_nh.nh_dev || - rt->fib6_nh.nh_dev->ifindex != cfg->fc_ifindex)) + (!nh->fib_nh_dev || + nh->fib_nh_dev->ifindex != cfg->fc_ifindex)) continue; if (cfg->fc_flags & RTF_GATEWAY && - !ipv6_addr_equal(&cfg->fc_gateway, &rt->fib6_nh.nh_gw)) + !ipv6_addr_equal(&cfg->fc_gateway, &nh->fib_nh_gw6)) continue; if (cfg->fc_metric && cfg->fc_metric != rt->fib6_metric) continue; @@ -3305,10 +3403,10 @@ static void rt6_do_redirect(struct dst_entry *dst, struct sock *sk, struct sk_bu { struct netevent_redirect netevent; struct rt6_info *rt, *nrt = NULL; + struct fib6_result res = {}; struct ndisc_options ndopts; struct inet6_dev *in6_dev; struct neighbour *neigh; - struct fib6_info *from; struct rd_msg *msg; int optlen, on_link; u8 *lladdr; @@ -3391,11 +3489,14 @@ static void rt6_do_redirect(struct dst_entry *dst, struct sock *sk, struct sk_bu NDISC_REDIRECT, &ndopts); rcu_read_lock(); - from = rcu_dereference(rt->from); - if (!from) + res.f6i = rcu_dereference(rt->from); + if (!res.f6i) goto out; - nrt = ip6_rt_cache_alloc(from, &msg->dest, NULL); + res.nh = &res.f6i->fib6_nh; + res.fib6_flags = res.f6i->fib6_flags; + res.fib6_type = res.f6i->fib6_type; + nrt = ip6_rt_cache_alloc(&res, &msg->dest, NULL); if (!nrt) goto out; @@ -3406,7 +3507,7 @@ static void rt6_do_redirect(struct dst_entry *dst, struct sock *sk, struct sk_bu nrt->rt6i_gateway = *(struct in6_addr *)neigh->primary_key; /* rt6_insert_exception() will take care of duplicated exceptions */ - if (rt6_insert_exception(nrt, from)) { + if (rt6_insert_exception(nrt, &res)) { dst_release_immediate(&nrt->dst); goto out; } @@ -3444,11 +3545,12 @@ static struct fib6_info *rt6_get_route_info(struct net *net, goto out; for_each_fib6_node_rt_rcu(fn) { - if (rt->fib6_nh.nh_dev->ifindex != ifindex) + if (rt->fib6_nh.fib_nh_dev->ifindex != ifindex) continue; - if ((rt->fib6_flags & (RTF_ROUTEINFO|RTF_GATEWAY)) != (RTF_ROUTEINFO|RTF_GATEWAY)) + if (!(rt->fib6_flags & RTF_ROUTEINFO) || + !rt->fib6_nh.fib_nh_gw_family) continue; - if (!ipv6_addr_equal(&rt->fib6_nh.nh_gw, gwaddr)) + if (!ipv6_addr_equal(&rt->fib6_nh.fib_nh_gw6, gwaddr)) continue; if (!fib6_info_hold_safe(rt)) continue; @@ -3506,9 +3608,11 @@ struct fib6_info *rt6_get_dflt_router(struct net *net, rcu_read_lock(); for_each_fib6_node_rt_rcu(&table->tb6_root) { - if (dev == rt->fib6_nh.nh_dev && + struct fib6_nh *nh = &rt->fib6_nh; + + if (dev == nh->fib_nh_dev && ((rt->fib6_flags & (RTF_ADDRCONF | RTF_DEFAULT)) == (RTF_ADDRCONF | RTF_DEFAULT)) && - ipv6_addr_equal(&rt->fib6_nh.nh_gw, addr)) + ipv6_addr_equal(&nh->fib_nh_gw6, addr)) break; } if (rt && !fib6_info_hold_safe(rt)) @@ -3599,7 +3703,7 @@ static void rtmsg_to_fib6_config(struct net *net, .fc_table = l3mdev_fib_table_by_index(net, rtmsg->rtmsg_ifindex) ? : RT6_TABLE_MAIN, .fc_ifindex = rtmsg->rtmsg_ifindex, - .fc_metric = rtmsg->rtmsg_metric, + .fc_metric = rtmsg->rtmsg_metric ? : IP6_RT_PRIO_USER, .fc_expires = rtmsg->rtmsg_info, .fc_dst_len = rtmsg->rtmsg_dst_len, .fc_src_len = rtmsg->rtmsg_src_len, @@ -3721,36 +3825,26 @@ struct fib6_info *addrconf_f6i_alloc(struct net *net, const struct in6_addr *addr, bool anycast, gfp_t gfp_flags) { - u32 tb_id; - struct net_device *dev = idev->dev; - struct fib6_info *f6i; - - f6i = fib6_info_alloc(gfp_flags); - if (!f6i) - return ERR_PTR(-ENOMEM); + struct fib6_config cfg = { + .fc_table = l3mdev_fib_table(idev->dev) ? : RT6_TABLE_LOCAL, + .fc_ifindex = idev->dev->ifindex, + .fc_flags = RTF_UP | RTF_ADDRCONF | RTF_NONEXTHOP, + .fc_dst = *addr, + .fc_dst_len = 128, + .fc_protocol = RTPROT_KERNEL, + .fc_nlinfo.nl_net = net, + .fc_ignore_dev_down = true, + }; - f6i->fib6_metrics = ip_fib_metrics_init(net, NULL, 0, NULL); - f6i->dst_nocount = true; - f6i->dst_host = true; - f6i->fib6_protocol = RTPROT_KERNEL; - f6i->fib6_flags = RTF_UP | RTF_NONEXTHOP; if (anycast) { - f6i->fib6_type = RTN_ANYCAST; - f6i->fib6_flags |= RTF_ANYCAST; + cfg.fc_type = RTN_ANYCAST; + cfg.fc_flags |= RTF_ANYCAST; } else { - f6i->fib6_type = RTN_LOCAL; - f6i->fib6_flags |= RTF_LOCAL; + cfg.fc_type = RTN_LOCAL; + cfg.fc_flags |= RTF_LOCAL; } - f6i->fib6_nh.nh_gw = *addr; - dev_hold(dev); - f6i->fib6_nh.nh_dev = dev; - f6i->fib6_dst.addr = *addr; - f6i->fib6_dst.plen = 128; - tb_id = l3mdev_fib_table(idev->dev) ? : RT6_TABLE_LOCAL; - f6i->fib6_table = fib6_get_table(net, tb_id); - - return f6i; + return ip6_route_info_create(&cfg, gfp_flags, NULL); } /* remove deleted ip from prefsrc entries */ @@ -3766,7 +3860,7 @@ static int fib6_remove_prefsrc(struct fib6_info *rt, void *arg) struct net *net = ((struct arg_dev_net_ip *)arg)->net; struct in6_addr *addr = ((struct arg_dev_net_ip *)arg)->addr; - if (((void *)rt->fib6_nh.nh_dev == dev || !dev) && + if (((void *)rt->fib6_nh.fib_nh_dev == dev || !dev) && rt != net->ipv6.fib6_null_entry && ipv6_addr_equal(addr, &rt->fib6_prefsrc.addr)) { spin_lock_bh(&rt6_exception_lock); @@ -3788,7 +3882,7 @@ void rt6_remove_prefsrc(struct inet6_ifaddr *ifp) fib6_clean_all(net, fib6_remove_prefsrc, &adni); } -#define RTF_RA_ROUTER (RTF_ADDRCONF | RTF_DEFAULT | RTF_GATEWAY) +#define RTF_RA_ROUTER (RTF_ADDRCONF | RTF_DEFAULT) /* Remove routers and update dst entries when gateway turn into host. */ static int fib6_clean_tohost(struct fib6_info *rt, void *arg) @@ -3796,7 +3890,8 @@ static int fib6_clean_tohost(struct fib6_info *rt, void *arg) struct in6_addr *gateway = (struct in6_addr *)arg; if (((rt->fib6_flags & RTF_RA_ROUTER) == RTF_RA_ROUTER) && - ipv6_addr_equal(gateway, &rt->fib6_nh.nh_gw)) { + rt->fib6_nh.fib_nh_gw_family && + ipv6_addr_equal(gateway, &rt->fib6_nh.fib_nh_gw6)) { return -1; } @@ -3817,7 +3912,7 @@ void rt6_clean_tohost(struct net *net, struct in6_addr *gateway) struct arg_netdev_event { const struct net_device *dev; union { - unsigned int nh_flags; + unsigned char nh_flags; unsigned long event; }; }; @@ -3844,9 +3939,9 @@ static struct fib6_info *rt6_multipath_first_sibling(const struct fib6_info *rt) static bool rt6_is_dead(const struct fib6_info *rt) { - if (rt->fib6_nh.nh_flags & RTNH_F_DEAD || - (rt->fib6_nh.nh_flags & RTNH_F_LINKDOWN && - fib6_ignore_linkdown(rt))) + if (rt->fib6_nh.fib_nh_flags & RTNH_F_DEAD || + (rt->fib6_nh.fib_nh_flags & RTNH_F_LINKDOWN && + ip6_ignore_linkdown(rt->fib6_nh.fib_nh_dev))) return true; return false; @@ -3858,11 +3953,11 @@ static int rt6_multipath_total_weight(const struct fib6_info *rt) int total = 0; if (!rt6_is_dead(rt)) - total += rt->fib6_nh.nh_weight; + total += rt->fib6_nh.fib_nh_weight; list_for_each_entry(iter, &rt->fib6_siblings, fib6_siblings) { if (!rt6_is_dead(iter)) - total += iter->fib6_nh.nh_weight; + total += iter->fib6_nh.fib_nh_weight; } return total; @@ -3873,11 +3968,11 @@ static void rt6_upper_bound_set(struct fib6_info *rt, int *weight, int total) int upper_bound = -1; if (!rt6_is_dead(rt)) { - *weight += rt->fib6_nh.nh_weight; + *weight += rt->fib6_nh.fib_nh_weight; upper_bound = DIV_ROUND_CLOSEST_ULL((u64) (*weight) << 31, total) - 1; } - atomic_set(&rt->fib6_nh.nh_upper_bound, upper_bound); + atomic_set(&rt->fib6_nh.fib_nh_upper_bound, upper_bound); } static void rt6_multipath_upper_bound_set(struct fib6_info *rt, int total) @@ -3920,8 +4015,9 @@ static int fib6_ifup(struct fib6_info *rt, void *p_arg) const struct arg_netdev_event *arg = p_arg; struct net *net = dev_net(arg->dev); - if (rt != net->ipv6.fib6_null_entry && rt->fib6_nh.nh_dev == arg->dev) { - rt->fib6_nh.nh_flags &= ~arg->nh_flags; + if (rt != net->ipv6.fib6_null_entry && + rt->fib6_nh.fib_nh_dev == arg->dev) { + rt->fib6_nh.fib_nh_flags &= ~arg->nh_flags; fib6_update_sernum_upto_root(net, rt); rt6_multipath_rebalance(rt); } @@ -3929,7 +4025,7 @@ static int fib6_ifup(struct fib6_info *rt, void *p_arg) return 0; } -void rt6_sync_up(struct net_device *dev, unsigned int nh_flags) +void rt6_sync_up(struct net_device *dev, unsigned char nh_flags) { struct arg_netdev_event arg = { .dev = dev, @@ -3949,10 +4045,10 @@ static bool rt6_multipath_uses_dev(const struct fib6_info *rt, { struct fib6_info *iter; - if (rt->fib6_nh.nh_dev == dev) + if (rt->fib6_nh.fib_nh_dev == dev) return true; list_for_each_entry(iter, &rt->fib6_siblings, fib6_siblings) - if (iter->fib6_nh.nh_dev == dev) + if (iter->fib6_nh.fib_nh_dev == dev) return true; return false; @@ -3973,12 +4069,12 @@ static unsigned int rt6_multipath_dead_count(const struct fib6_info *rt, struct fib6_info *iter; unsigned int dead = 0; - if (rt->fib6_nh.nh_dev == down_dev || - rt->fib6_nh.nh_flags & RTNH_F_DEAD) + if (rt->fib6_nh.fib_nh_dev == down_dev || + rt->fib6_nh.fib_nh_flags & RTNH_F_DEAD) dead++; list_for_each_entry(iter, &rt->fib6_siblings, fib6_siblings) - if (iter->fib6_nh.nh_dev == down_dev || - iter->fib6_nh.nh_flags & RTNH_F_DEAD) + if (iter->fib6_nh.fib_nh_dev == down_dev || + iter->fib6_nh.fib_nh_flags & RTNH_F_DEAD) dead++; return dead; @@ -3986,15 +4082,15 @@ static unsigned int rt6_multipath_dead_count(const struct fib6_info *rt, static void rt6_multipath_nh_flags_set(struct fib6_info *rt, const struct net_device *dev, - unsigned int nh_flags) + unsigned char nh_flags) { struct fib6_info *iter; - if (rt->fib6_nh.nh_dev == dev) - rt->fib6_nh.nh_flags |= nh_flags; + if (rt->fib6_nh.fib_nh_dev == dev) + rt->fib6_nh.fib_nh_flags |= nh_flags; list_for_each_entry(iter, &rt->fib6_siblings, fib6_siblings) - if (iter->fib6_nh.nh_dev == dev) - iter->fib6_nh.nh_flags |= nh_flags; + if (iter->fib6_nh.fib_nh_dev == dev) + iter->fib6_nh.fib_nh_flags |= nh_flags; } /* called with write lock held for table with rt */ @@ -4009,12 +4105,12 @@ static int fib6_ifdown(struct fib6_info *rt, void *p_arg) switch (arg->event) { case NETDEV_UNREGISTER: - return rt->fib6_nh.nh_dev == dev ? -1 : 0; + return rt->fib6_nh.fib_nh_dev == dev ? -1 : 0; case NETDEV_DOWN: if (rt->should_flush) return -1; if (!rt->fib6_nsiblings) - return rt->fib6_nh.nh_dev == dev ? -1 : 0; + return rt->fib6_nh.fib_nh_dev == dev ? -1 : 0; if (rt6_multipath_uses_dev(rt, dev)) { unsigned int count; @@ -4030,10 +4126,10 @@ static int fib6_ifdown(struct fib6_info *rt, void *p_arg) } return -2; case NETDEV_CHANGE: - if (rt->fib6_nh.nh_dev != dev || + if (rt->fib6_nh.fib_nh_dev != dev || rt->fib6_flags & (RTF_LOCAL | RTF_ANYCAST)) break; - rt->fib6_nh.nh_flags |= RTNH_F_LINKDOWN; + rt->fib6_nh.fib_nh_flags |= RTNH_F_LINKDOWN; rt6_multipath_rebalance(rt); break; } @@ -4089,7 +4185,7 @@ static int rt6_mtu_change_route(struct fib6_info *rt, void *p_arg) Since RFC 1981 doesn't include administrative MTU increase update PMTU increase is a MUST. (i.e. jumbo frame) */ - if (rt->fib6_nh.nh_dev == arg->dev && + if (rt->fib6_nh.fib_nh_dev == arg->dev && !fib6_metric_locked(rt, RTAX_MTU)) { u32 mtu = rt->fib6_pmtu; @@ -4143,8 +4239,8 @@ static int rtm_to_fib6_config(struct sk_buff *skb, struct nlmsghdr *nlh, unsigned int pref; int err; - err = nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, rtm_ipv6_policy, - extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*rtm), tb, RTA_MAX, + rtm_ipv6_policy, extack); if (err < 0) goto errout; @@ -4380,7 +4476,7 @@ static int ip6_route_multipath_add(struct fib6_config *cfg, goto cleanup; } - rt->fib6_nh.nh_weight = rtnh->rtnh_hops + 1; + rt->fib6_nh.fib_nh_weight = rtnh->rtnh_hops + 1; err = ip6_route_info_append(info->nl_net, &rt6_nh_list, rt, &r_cfg); @@ -4530,6 +4626,9 @@ static int inet6_rtm_newroute(struct sk_buff *skb, struct nlmsghdr *nlh, if (err < 0) return err; + if (cfg.fc_metric == 0) + cfg.fc_metric = IP6_RT_PRIO_USER; + if (cfg.fc_mp) return ip6_route_multipath_add(&cfg, extack); else @@ -4544,7 +4643,7 @@ static size_t rt6_nlmsg_size(struct fib6_info *rt) nexthop_len = nla_total_size(0) /* RTA_MULTIPATH */ + NLA_ALIGN(sizeof(struct rtnexthop)) + nla_total_size(16) /* RTA_GATEWAY */ - + lwtunnel_get_encap_size(rt->fib6_nh.nh_lwtstate); + + lwtunnel_get_encap_size(rt->fib6_nh.fib_nh_lws); nexthop_len *= rt->fib6_nsiblings; } @@ -4562,77 +4661,10 @@ static size_t rt6_nlmsg_size(struct fib6_info *rt) + nla_total_size(sizeof(struct rta_cacheinfo)) + nla_total_size(TCP_CA_NAME_MAX) /* RTAX_CC_ALGO */ + nla_total_size(1) /* RTA_PREF */ - + lwtunnel_get_encap_size(rt->fib6_nh.nh_lwtstate) + + lwtunnel_get_encap_size(rt->fib6_nh.fib_nh_lws) + nexthop_len; } -static int rt6_nexthop_info(struct sk_buff *skb, struct fib6_info *rt, - unsigned int *flags, bool skip_oif) -{ - if (rt->fib6_nh.nh_flags & RTNH_F_DEAD) - *flags |= RTNH_F_DEAD; - - if (rt->fib6_nh.nh_flags & RTNH_F_LINKDOWN) { - *flags |= RTNH_F_LINKDOWN; - - rcu_read_lock(); - if (fib6_ignore_linkdown(rt)) - *flags |= RTNH_F_DEAD; - rcu_read_unlock(); - } - - if (rt->fib6_flags & RTF_GATEWAY) { - if (nla_put_in6_addr(skb, RTA_GATEWAY, &rt->fib6_nh.nh_gw) < 0) - goto nla_put_failure; - } - - *flags |= (rt->fib6_nh.nh_flags & RTNH_F_ONLINK); - if (rt->fib6_nh.nh_flags & RTNH_F_OFFLOAD) - *flags |= RTNH_F_OFFLOAD; - - /* not needed for multipath encoding b/c it has a rtnexthop struct */ - if (!skip_oif && rt->fib6_nh.nh_dev && - nla_put_u32(skb, RTA_OIF, rt->fib6_nh.nh_dev->ifindex)) - goto nla_put_failure; - - if (rt->fib6_nh.nh_lwtstate && - lwtunnel_fill_encap(skb, rt->fib6_nh.nh_lwtstate) < 0) - goto nla_put_failure; - - return 0; - -nla_put_failure: - return -EMSGSIZE; -} - -/* add multipath next hop */ -static int rt6_add_nexthop(struct sk_buff *skb, struct fib6_info *rt) -{ - const struct net_device *dev = rt->fib6_nh.nh_dev; - struct rtnexthop *rtnh; - unsigned int flags = 0; - - rtnh = nla_reserve_nohdr(skb, sizeof(*rtnh)); - if (!rtnh) - goto nla_put_failure; - - rtnh->rtnh_hops = rt->fib6_nh.nh_weight - 1; - rtnh->rtnh_ifindex = dev ? dev->ifindex : 0; - - if (rt6_nexthop_info(skb, rt, &flags, true) < 0) - goto nla_put_failure; - - rtnh->rtnh_flags = flags; - - /* length of rtnetlink header + attributes */ - rtnh->rtnh_len = nlmsg_get_pos(skb) - (void *)rtnh; - - return 0; - -nla_put_failure: - return -EMSGSIZE; -} - static int rt6_fill_node(struct net *net, struct sk_buff *skb, struct fib6_info *rt, struct dst_entry *dst, struct in6_addr *dest, struct in6_addr *src, @@ -4745,23 +4777,30 @@ static int rt6_fill_node(struct net *net, struct sk_buff *skb, struct fib6_info *sibling, *next_sibling; struct nlattr *mp; - mp = nla_nest_start(skb, RTA_MULTIPATH); + mp = nla_nest_start_noflag(skb, RTA_MULTIPATH); if (!mp) goto nla_put_failure; - if (rt6_add_nexthop(skb, rt) < 0) + if (fib_add_nexthop(skb, &rt->fib6_nh.nh_common, + rt->fib6_nh.fib_nh_weight) < 0) goto nla_put_failure; list_for_each_entry_safe(sibling, next_sibling, &rt->fib6_siblings, fib6_siblings) { - if (rt6_add_nexthop(skb, sibling) < 0) + if (fib_add_nexthop(skb, &sibling->fib6_nh.nh_common, + sibling->fib6_nh.fib_nh_weight) < 0) goto nla_put_failure; } nla_nest_end(skb, mp); } else { - if (rt6_nexthop_info(skb, rt, &rtm->rtm_flags, false) < 0) + unsigned char nh_flags = 0; + + if (fib_nexthop_info(skb, &rt->fib6_nh.nh_common, + &nh_flags, false) < 0) goto nla_put_failure; + + rtm->rtm_flags |= nh_flags; } if (rt6_flags & RTF_EXPIRES) { @@ -4787,7 +4826,7 @@ nla_put_failure: static bool fib6_info_uses_dev(const struct fib6_info *f6i, const struct net_device *dev) { - if (f6i->fib6_nh.nh_dev == dev) + if (f6i->fib6_nh.fib_nh_dev == dev) return true; if (f6i->fib6_nsiblings) { @@ -4795,7 +4834,7 @@ static bool fib6_info_uses_dev(const struct fib6_info *f6i, list_for_each_entry_safe(sibling, next_sibling, &f6i->fib6_siblings, fib6_siblings) { - if (sibling->fib6_nh.nh_dev == dev) + if (sibling->fib6_nh.fib_nh_dev == dev) return true; } } @@ -4847,8 +4886,8 @@ static int inet6_rtm_valid_getroute_req(struct sk_buff *skb, } if (!netlink_strict_get_check(skb)) - return nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, - rtm_ipv6_policy, extack); + return nlmsg_parse_deprecated(nlh, sizeof(*rtm), tb, RTA_MAX, + rtm_ipv6_policy, extack); rtm = nlmsg_data(nlh); if ((rtm->rtm_src_len && rtm->rtm_src_len != 128) || @@ -4864,8 +4903,8 @@ static int inet6_rtm_valid_getroute_req(struct sk_buff *skb, return -EINVAL; } - err = nlmsg_parse_strict(nlh, sizeof(*rtm), tb, RTA_MAX, - rtm_ipv6_policy, extack); + err = nlmsg_parse_deprecated_strict(nlh, sizeof(*rtm), tb, RTA_MAX, + rtm_ipv6_policy, extack); if (err) return err; @@ -5084,7 +5123,7 @@ static int ip6_route_dev_notify(struct notifier_block *this, return NOTIFY_OK; if (event == NETDEV_REGISTER) { - net->ipv6.fib6_null_entry->fib6_nh.nh_dev = dev; + net->ipv6.fib6_null_entry->fib6_nh.fib_nh_dev = dev; net->ipv6.ip6_null_entry->dst.dev = dev; net->ipv6.ip6_null_entry->rt6i_idev = in6_dev_get(dev); #ifdef CONFIG_IPV6_MULTIPLE_TABLES @@ -5419,7 +5458,7 @@ void __init ip6_route_init_special_entries(void) /* Registering of the loopback is done before this portion of code, * the loopback reference in rt6_info will not be taken, do it * manually for init_net */ - init_net.ipv6.fib6_null_entry->fib6_nh.nh_dev = init_net.loopback_dev; + init_net.ipv6.fib6_null_entry->fib6_nh.fib_nh_dev = init_net.loopback_dev; init_net.ipv6.ip6_null_entry->dst.dev = init_net.loopback_dev; init_net.ipv6.ip6_null_entry->rt6i_idev = in6_dev_get(init_net.loopback_dev); #ifdef CONFIG_IPV6_MULTIPLE_TABLES diff --git a/net/ipv6/seg6.c b/net/ipv6/seg6.c index 9b2f272ca164919aa4c9a6559852f55a10ce24aa..0c5479ef9b38ef0cffd47ec225007134708334b4 100644 --- a/net/ipv6/seg6.c +++ b/net/ipv6/seg6.c @@ -398,28 +398,28 @@ static struct pernet_operations ip6_segments_ops = { static const struct genl_ops seg6_genl_ops[] = { { .cmd = SEG6_CMD_SETHMAC, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = seg6_genl_sethmac, - .policy = seg6_genl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = SEG6_CMD_DUMPHMAC, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .start = seg6_genl_dumphmac_start, .dumpit = seg6_genl_dumphmac, .done = seg6_genl_dumphmac_done, - .policy = seg6_genl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = SEG6_CMD_SET_TUNSRC, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = seg6_genl_set_tunsrc, - .policy = seg6_genl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = SEG6_CMD_GET_TUNSRC, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = seg6_genl_get_tunsrc, - .policy = seg6_genl_policy, .flags = GENL_ADMIN_PERM, }, }; @@ -429,6 +429,7 @@ static struct genl_family seg6_genl_family __ro_after_init = { .name = SEG6_GENL_NAME, .version = SEG6_GENL_VERSION, .maxattr = SEG6_ATTR_MAX, + .policy = seg6_genl_policy, .netnsok = true, .parallel_ops = true, .ops = seg6_genl_ops, diff --git a/net/ipv6/seg6_iptunnel.c b/net/ipv6/seg6_iptunnel.c index ee5403cbe655ee2d51eef2cf4913ef39a36b67b1..7a525fda897820edf197d59c624ff787f9501747 100644 --- a/net/ipv6/seg6_iptunnel.c +++ b/net/ipv6/seg6_iptunnel.c @@ -396,8 +396,8 @@ static int seg6_build_state(struct nlattr *nla, if (family != AF_INET && family != AF_INET6) return -EINVAL; - err = nla_parse_nested(tb, SEG6_IPTUNNEL_MAX, nla, - seg6_iptunnel_policy, extack); + err = nla_parse_nested_deprecated(tb, SEG6_IPTUNNEL_MAX, nla, + seg6_iptunnel_policy, extack); if (err < 0) return err; diff --git a/net/ipv6/seg6_local.c b/net/ipv6/seg6_local.c index 60325dbfe88b0ca8ffb08cc89666642c52739480..78155fdb8c3622861f0f2b7a5f98535203671375 100644 --- a/net/ipv6/seg6_local.c +++ b/net/ipv6/seg6_local.c @@ -823,8 +823,9 @@ static int parse_nla_bpf(struct nlattr **attrs, struct seg6_local_lwt *slwt) int ret; u32 fd; - ret = nla_parse_nested(tb, SEG6_LOCAL_BPF_PROG_MAX, - attrs[SEG6_LOCAL_BPF], bpf_prog_policy, NULL); + ret = nla_parse_nested_deprecated(tb, SEG6_LOCAL_BPF_PROG_MAX, + attrs[SEG6_LOCAL_BPF], + bpf_prog_policy, NULL); if (ret < 0) return ret; @@ -853,7 +854,7 @@ static int put_nla_bpf(struct sk_buff *skb, struct seg6_local_lwt *slwt) if (!slwt->bpf.prog) return 0; - nest = nla_nest_start(skb, SEG6_LOCAL_BPF); + nest = nla_nest_start_noflag(skb, SEG6_LOCAL_BPF); if (!nest) return -EMSGSIZE; @@ -959,8 +960,8 @@ static int seg6_local_build_state(struct nlattr *nla, unsigned int family, if (family != AF_INET6) return -EINVAL; - err = nla_parse_nested(tb, SEG6_LOCAL_MAX, nla, seg6_local_policy, - extack); + err = nla_parse_nested_deprecated(tb, SEG6_LOCAL_MAX, nla, + seg6_local_policy, extack); if (err < 0) return err; diff --git a/net/ipv6/sit.c b/net/ipv6/sit.c index b2109b74857d053f52b06c42698cc393d5838609..971d60bf964009d70b4a59b6876f8ac827ebde2a 100644 --- a/net/ipv6/sit.c +++ b/net/ipv6/sit.c @@ -1084,7 +1084,7 @@ static void ipip6_tunnel_bind_dev(struct net_device *dev) if (!tdev && tunnel->parms.link) tdev = __dev_get_by_index(tunnel->net, tunnel->parms.link); - if (tdev) { + if (tdev && !netif_is_l3_master(tdev)) { int t_hlen = tunnel->hlen + sizeof(struct iphdr); dev->hard_header_len = tdev->hard_header_len + sizeof(struct iphdr); diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c index 44d431849d391d6903d263ae547fc9bed1e67aa7..beaf284563015ef0677c39fc056e6ecde3518920 100644 --- a/net/ipv6/tcp_ipv6.c +++ b/net/ipv6/tcp_ipv6.c @@ -43,6 +43,7 @@ #include #include #include +#include #include #include @@ -90,6 +91,18 @@ static struct tcp_md5sig_key *tcp_v6_md5_do_lookup(const struct sock *sk, } #endif +/* Helper returning the inet6 address from a given tcp socket. + * It can be used in TCP stack instead of inet6_sk(sk). + * This avoids a dereference and allow compiler optimizations. + * It is a specialized version of inet6_sk_generic(). + */ +static struct ipv6_pinfo *tcp_inet6_sk(const struct sock *sk) +{ + unsigned int offset = sizeof(struct tcp6_sock) - sizeof(struct ipv6_pinfo); + + return (struct ipv6_pinfo *)(((u8 *)sk) + offset); +} + static void inet6_sk_rx_dst_set(struct sock *sk, const struct sk_buff *skb) { struct dst_entry *dst = skb_dst(skb); @@ -99,7 +112,7 @@ static void inet6_sk_rx_dst_set(struct sock *sk, const struct sk_buff *skb) sk->sk_rx_dst = dst; inet_sk(sk)->rx_dst_ifindex = skb->skb_iif; - inet6_sk(sk)->rx_dst_cookie = rt6_get_cookie(rt); + tcp_inet6_sk(sk)->rx_dst_cookie = rt6_get_cookie(rt); } } @@ -138,7 +151,7 @@ static int tcp_v6_connect(struct sock *sk, struct sockaddr *uaddr, struct sockaddr_in6 *usin = (struct sockaddr_in6 *) uaddr; struct inet_sock *inet = inet_sk(sk); struct inet_connection_sock *icsk = inet_csk(sk); - struct ipv6_pinfo *np = inet6_sk(sk); + struct ipv6_pinfo *np = tcp_inet6_sk(sk); struct tcp_sock *tp = tcp_sk(sk); struct in6_addr *saddr = NULL, *final_p, final; struct ipv6_txoptions *opt; @@ -390,7 +403,7 @@ static int tcp_v6_err(struct sk_buff *skb, struct inet6_skb_parm *opt, if (sk->sk_state == TCP_CLOSE) goto out; - if (ipv6_hdr(skb)->hop_limit < inet6_sk(sk)->min_hopcount) { + if (ipv6_hdr(skb)->hop_limit < tcp_inet6_sk(sk)->min_hopcount) { __NET_INC_STATS(net, LINUX_MIB_TCPMINTTLDROP); goto out; } @@ -405,7 +418,7 @@ static int tcp_v6_err(struct sk_buff *skb, struct inet6_skb_parm *opt, goto out; } - np = inet6_sk(sk); + np = tcp_inet6_sk(sk); if (type == NDISC_REDIRECT) { if (!sock_owned_by_user(sk)) { @@ -478,7 +491,7 @@ static int tcp_v6_send_synack(const struct sock *sk, struct dst_entry *dst, enum tcp_synack_type synack_type) { struct inet_request_sock *ireq = inet_rsk(req); - struct ipv6_pinfo *np = inet6_sk(sk); + struct ipv6_pinfo *np = tcp_inet6_sk(sk); struct ipv6_txoptions *opt; struct flowi6 *fl6 = &fl->u.ip6; struct sk_buff *skb; @@ -737,7 +750,7 @@ static void tcp_v6_init_req(struct request_sock *req, { bool l3_slave = ipv6_l3mdev_skb(TCP_SKB_CB(skb)->header.h6.flags); struct inet_request_sock *ireq = inet_rsk(req); - const struct ipv6_pinfo *np = inet6_sk(sk_listener); + const struct ipv6_pinfo *np = tcp_inet6_sk(sk_listener); ireq->ir_v6_rmt_addr = ipv6_hdr(skb)->saddr; ireq->ir_v6_loc_addr = ipv6_hdr(skb)->daddr; @@ -1066,9 +1079,8 @@ static struct sock *tcp_v6_syn_recv_sock(const struct sock *sk, struct sk_buff * { struct inet_request_sock *ireq; struct ipv6_pinfo *newnp; - const struct ipv6_pinfo *np = inet6_sk(sk); + const struct ipv6_pinfo *np = tcp_inet6_sk(sk); struct ipv6_txoptions *opt; - struct tcp6_sock *newtcp6sk; struct inet_sock *newinet; struct tcp_sock *newtp; struct sock *newsk; @@ -1088,11 +1100,10 @@ static struct sock *tcp_v6_syn_recv_sock(const struct sock *sk, struct sk_buff * if (!newsk) return NULL; - newtcp6sk = (struct tcp6_sock *)newsk; - inet_sk(newsk)->pinet6 = &newtcp6sk->inet6; + inet_sk(newsk)->pinet6 = tcp_inet6_sk(newsk); newinet = inet_sk(newsk); - newnp = inet6_sk(newsk); + newnp = tcp_inet6_sk(newsk); newtp = tcp_sk(newsk); memcpy(newnp, np, sizeof(struct ipv6_pinfo)); @@ -1156,12 +1167,11 @@ static struct sock *tcp_v6_syn_recv_sock(const struct sock *sk, struct sk_buff * ip6_dst_store(newsk, dst, NULL, NULL); inet6_sk_rx_dst_set(newsk, skb); - newtcp6sk = (struct tcp6_sock *)newsk; - inet_sk(newsk)->pinet6 = &newtcp6sk->inet6; + inet_sk(newsk)->pinet6 = tcp_inet6_sk(newsk); newtp = tcp_sk(newsk); newinet = inet_sk(newsk); - newnp = inet6_sk(newsk); + newnp = tcp_inet6_sk(newsk); memcpy(newnp, np, sizeof(struct ipv6_pinfo)); @@ -1276,9 +1286,9 @@ out: */ static int tcp_v6_do_rcv(struct sock *sk, struct sk_buff *skb) { - struct ipv6_pinfo *np = inet6_sk(sk); - struct tcp_sock *tp; + struct ipv6_pinfo *np = tcp_inet6_sk(sk); struct sk_buff *opt_skb = NULL; + struct tcp_sock *tp; /* Imagine: socket is IPv6. IPv4 packet arrives, goes to IPv4 receive handler and backlogged. @@ -1426,8 +1436,9 @@ static void tcp_v6_fill_cb(struct sk_buff *skb, const struct ipv6hdr *hdr, skb->tstamp || skb_hwtstamps(skb)->hwtstamp; } -static int tcp_v6_rcv(struct sk_buff *skb) +INDIRECT_CALLABLE_SCOPE int tcp_v6_rcv(struct sk_buff *skb) { + struct sk_buff *skb_to_free; int sdif = inet6_sdif(skb); const struct tcphdr *th; const struct ipv6hdr *hdr; @@ -1524,7 +1535,7 @@ process: return 0; } } - if (hdr->hop_limit < inet6_sk(sk)->min_hopcount) { + if (hdr->hop_limit < tcp_inet6_sk(sk)->min_hopcount) { __NET_INC_STATS(net, LINUX_MIB_TCPMINTTLDROP); goto discard_and_relse; } @@ -1554,12 +1565,17 @@ process: tcp_segs_in(tcp_sk(sk), skb); ret = 0; if (!sock_owned_by_user(sk)) { + skb_to_free = sk->sk_rx_skb_cache; + sk->sk_rx_skb_cache = NULL; ret = tcp_v6_do_rcv(sk, skb); - } else if (tcp_add_backlog(sk, skb)) { - goto discard_and_relse; + } else { + if (tcp_add_backlog(sk, skb)) + goto discard_and_relse; + skb_to_free = NULL; } bh_unlock_sock(sk); - + if (skb_to_free) + __kfree_skb(skb_to_free); put_and_return: if (refcounted) sock_put(sk); @@ -1639,7 +1655,7 @@ do_time_wait: goto discard_it; } -static void tcp_v6_early_demux(struct sk_buff *skb) +INDIRECT_CALLABLE_SCOPE void tcp_v6_early_demux(struct sk_buff *skb) { const struct ipv6hdr *hdr; const struct tcphdr *th; @@ -1669,7 +1685,7 @@ static void tcp_v6_early_demux(struct sk_buff *skb) struct dst_entry *dst = READ_ONCE(sk->sk_rx_dst); if (dst) - dst = dst_check(dst, inet6_sk(sk)->rx_dst_cookie); + dst = dst_check(dst, tcp_inet6_sk(sk)->rx_dst_cookie); if (dst && inet_sk(sk)->rx_dst_ifindex == skb->skb_iif) skb_dst_set_noref(skb, dst); diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c index 622eeaf5732b39b97752eefb864133e46b27a15d..07fa579dfb96c642873d5904d5309ccfbfde8167 100644 --- a/net/ipv6/udp.c +++ b/net/ipv6/udp.c @@ -36,6 +36,7 @@ #include #include #include +#include #include #include @@ -285,8 +286,7 @@ int udpv6_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, struct inet_sock *inet = inet_sk(sk); struct sk_buff *skb; unsigned int ulen, copied; - int peeked, peeking, off; - int err; + int off, err, peeking = flags & MSG_PEEK; int is_udplite = IS_UDPLITE(sk); struct udp_mib __percpu *mib; bool checksum_valid = false; @@ -299,9 +299,8 @@ int udpv6_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, return ipv6_recv_rxpmtu(sk, msg, len, addr_len); try_again: - peeking = flags & MSG_PEEK; off = sk_peek_offset(sk, flags); - skb = __skb_recv_udp(sk, flags, noblock, &peeked, &off, &err); + skb = __skb_recv_udp(sk, flags, noblock, &off, &err); if (!skb) return err; @@ -340,14 +339,14 @@ try_again: goto csum_copy_err; } if (unlikely(err)) { - if (!peeked) { + if (!peeking) { atomic_inc(&sk->sk_drops); SNMP_INC_STATS(mib, UDP_MIB_INERRORS); } kfree_skb(skb); return err; } - if (!peeked) + if (!peeking) SNMP_INC_STATS(mib, UDP_MIB_INDATAGRAMS); sock_recv_ts_and_drops(msg, sk, skb); @@ -982,7 +981,7 @@ static struct sock *__udp6_lib_demux_lookup(struct net *net, return NULL; } -static void udp_v6_early_demux(struct sk_buff *skb) +INDIRECT_CALLABLE_SCOPE void udp_v6_early_demux(struct sk_buff *skb) { struct net *net = dev_net(skb->dev); const struct udphdr *uh; @@ -1023,7 +1022,7 @@ static void udp_v6_early_demux(struct sk_buff *skb) } } -static __inline__ int udpv6_rcv(struct sk_buff *skb) +INDIRECT_CALLABLE_SCOPE int udpv6_rcv(struct sk_buff *skb) { return __udp6_lib_rcv(skb, &udp_table, IPPROTO_UDP); } diff --git a/net/ipv6/xfrm6_mode_beet.c b/net/ipv6/xfrm6_mode_beet.c deleted file mode 100644 index 57fd314ec2b89f69a871c46e7e3f2ad2d8a520c6..0000000000000000000000000000000000000000 --- a/net/ipv6/xfrm6_mode_beet.c +++ /dev/null @@ -1,131 +0,0 @@ -/* - * xfrm6_mode_beet.c - BEET mode encapsulation for IPv6. - * - * Copyright (c) 2006 Diego Beltrami - * Miika Komu - * Herbert Xu - * Abhinav Pathak - * Jeff Ahrenholz - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -static void xfrm6_beet_make_header(struct sk_buff *skb) -{ - struct ipv6hdr *iph = ipv6_hdr(skb); - - iph->version = 6; - - memcpy(iph->flow_lbl, XFRM_MODE_SKB_CB(skb)->flow_lbl, - sizeof(iph->flow_lbl)); - iph->nexthdr = XFRM_MODE_SKB_CB(skb)->protocol; - - ipv6_change_dsfield(iph, 0, XFRM_MODE_SKB_CB(skb)->tos); - iph->hop_limit = XFRM_MODE_SKB_CB(skb)->ttl; -} - -/* Add encapsulation header. - * - * The top IP header will be constructed per draft-nikander-esp-beet-mode-06.txt. - */ -static int xfrm6_beet_output(struct xfrm_state *x, struct sk_buff *skb) -{ - struct ipv6hdr *top_iph; - struct ip_beet_phdr *ph; - int optlen, hdr_len; - - hdr_len = 0; - optlen = XFRM_MODE_SKB_CB(skb)->optlen; - if (unlikely(optlen)) - hdr_len += IPV4_BEET_PHMAXLEN - (optlen & 4); - - skb_set_network_header(skb, -x->props.header_len - hdr_len); - if (x->sel.family != AF_INET6) - skb->network_header += IPV4_BEET_PHMAXLEN; - skb->mac_header = skb->network_header + - offsetof(struct ipv6hdr, nexthdr); - skb->transport_header = skb->network_header + sizeof(*top_iph); - ph = __skb_pull(skb, XFRM_MODE_SKB_CB(skb)->ihl - hdr_len); - - xfrm6_beet_make_header(skb); - - top_iph = ipv6_hdr(skb); - if (unlikely(optlen)) { - - BUG_ON(optlen < 0); - - ph->padlen = 4 - (optlen & 4); - ph->hdrlen = optlen / 8; - ph->nexthdr = top_iph->nexthdr; - if (ph->padlen) - memset(ph + 1, IPOPT_NOP, ph->padlen); - - top_iph->nexthdr = IPPROTO_BEETPH; - } - - top_iph->saddr = *(struct in6_addr *)&x->props.saddr; - top_iph->daddr = *(struct in6_addr *)&x->id.daddr; - return 0; -} - -static int xfrm6_beet_input(struct xfrm_state *x, struct sk_buff *skb) -{ - struct ipv6hdr *ip6h; - int size = sizeof(struct ipv6hdr); - int err; - - err = skb_cow_head(skb, size + skb->mac_len); - if (err) - goto out; - - __skb_push(skb, size); - skb_reset_network_header(skb); - skb_mac_header_rebuild(skb); - - xfrm6_beet_make_header(skb); - - ip6h = ipv6_hdr(skb); - ip6h->payload_len = htons(skb->len - size); - ip6h->daddr = x->sel.daddr.in6; - ip6h->saddr = x->sel.saddr.in6; - err = 0; -out: - return err; -} - -static struct xfrm_mode xfrm6_beet_mode = { - .input2 = xfrm6_beet_input, - .input = xfrm_prepare_input, - .output2 = xfrm6_beet_output, - .output = xfrm6_prepare_output, - .owner = THIS_MODULE, - .encap = XFRM_MODE_BEET, - .flags = XFRM_MODE_FLAG_TUNNEL, -}; - -static int __init xfrm6_beet_init(void) -{ - return xfrm_register_mode(&xfrm6_beet_mode, AF_INET6); -} - -static void __exit xfrm6_beet_exit(void) -{ - int err; - - err = xfrm_unregister_mode(&xfrm6_beet_mode, AF_INET6); - BUG_ON(err); -} - -module_init(xfrm6_beet_init); -module_exit(xfrm6_beet_exit); -MODULE_LICENSE("GPL"); -MODULE_ALIAS_XFRM_MODE(AF_INET6, XFRM_MODE_BEET); diff --git a/net/ipv6/xfrm6_mode_ro.c b/net/ipv6/xfrm6_mode_ro.c deleted file mode 100644 index da28e4407b8faa91eeb7517e87e9b644ae3f841a..0000000000000000000000000000000000000000 --- a/net/ipv6/xfrm6_mode_ro.c +++ /dev/null @@ -1,85 +0,0 @@ -/* - * xfrm6_mode_ro.c - Route optimization mode for IPv6. - * - * Copyright (C)2003-2006 Helsinki University of Technology - * Copyright (C)2003-2006 USAGI/WIDE Project - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . - */ -/* - * Authors: - * Noriaki TAKAMIYA @USAGI - * Masahide NAKAMURA @USAGI - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include - -/* Add route optimization header space. - * - * The IP header and mutable extension headers will be moved forward to make - * space for the route optimization header. - */ -static int xfrm6_ro_output(struct xfrm_state *x, struct sk_buff *skb) -{ - struct ipv6hdr *iph; - u8 *prevhdr; - int hdr_len; - - iph = ipv6_hdr(skb); - - hdr_len = x->type->hdr_offset(x, skb, &prevhdr); - if (hdr_len < 0) - return hdr_len; - skb_set_mac_header(skb, (prevhdr - x->props.header_len) - skb->data); - skb_set_network_header(skb, -x->props.header_len); - skb->transport_header = skb->network_header + hdr_len; - __skb_pull(skb, hdr_len); - memmove(ipv6_hdr(skb), iph, hdr_len); - - x->lastused = ktime_get_real_seconds(); - - return 0; -} - -static struct xfrm_mode xfrm6_ro_mode = { - .output = xfrm6_ro_output, - .owner = THIS_MODULE, - .encap = XFRM_MODE_ROUTEOPTIMIZATION, -}; - -static int __init xfrm6_ro_init(void) -{ - return xfrm_register_mode(&xfrm6_ro_mode, AF_INET6); -} - -static void __exit xfrm6_ro_exit(void) -{ - int err; - - err = xfrm_unregister_mode(&xfrm6_ro_mode, AF_INET6); - BUG_ON(err); -} - -module_init(xfrm6_ro_init); -module_exit(xfrm6_ro_exit); -MODULE_LICENSE("GPL"); -MODULE_ALIAS_XFRM_MODE(AF_INET6, XFRM_MODE_ROUTEOPTIMIZATION); diff --git a/net/ipv6/xfrm6_mode_transport.c b/net/ipv6/xfrm6_mode_transport.c deleted file mode 100644 index 3c29da5defe6c357ff04ca4adead1a9fee208f08..0000000000000000000000000000000000000000 --- a/net/ipv6/xfrm6_mode_transport.c +++ /dev/null @@ -1,121 +0,0 @@ -/* - * xfrm6_mode_transport.c - Transport mode encapsulation for IPv6. - * - * Copyright (C) 2002 USAGI/WIDE Project - * Copyright (c) 2004-2006 Herbert Xu - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include - -/* Add encapsulation header. - * - * The IP header and mutable extension headers will be moved forward to make - * space for the encapsulation header. - */ -static int xfrm6_transport_output(struct xfrm_state *x, struct sk_buff *skb) -{ - struct ipv6hdr *iph; - u8 *prevhdr; - int hdr_len; - - iph = ipv6_hdr(skb); - skb_set_inner_transport_header(skb, skb_transport_offset(skb)); - - hdr_len = x->type->hdr_offset(x, skb, &prevhdr); - if (hdr_len < 0) - return hdr_len; - skb_set_mac_header(skb, (prevhdr - x->props.header_len) - skb->data); - skb_set_network_header(skb, -x->props.header_len); - skb->transport_header = skb->network_header + hdr_len; - __skb_pull(skb, hdr_len); - memmove(ipv6_hdr(skb), iph, hdr_len); - return 0; -} - -/* Remove encapsulation header. - * - * The IP header will be moved over the top of the encapsulation header. - * - * On entry, skb->h shall point to where the IP header should be and skb->nh - * shall be set to where the IP header currently is. skb->data shall point - * to the start of the payload. - */ -static int xfrm6_transport_input(struct xfrm_state *x, struct sk_buff *skb) -{ - int ihl = skb->data - skb_transport_header(skb); - - if (skb->transport_header != skb->network_header) { - memmove(skb_transport_header(skb), - skb_network_header(skb), ihl); - skb->network_header = skb->transport_header; - } - ipv6_hdr(skb)->payload_len = htons(skb->len + ihl - - sizeof(struct ipv6hdr)); - skb_reset_transport_header(skb); - return 0; -} - -static struct sk_buff *xfrm4_transport_gso_segment(struct xfrm_state *x, - struct sk_buff *skb, - netdev_features_t features) -{ - const struct net_offload *ops; - struct sk_buff *segs = ERR_PTR(-EINVAL); - struct xfrm_offload *xo = xfrm_offload(skb); - - skb->transport_header += x->props.header_len; - ops = rcu_dereference(inet6_offloads[xo->proto]); - if (likely(ops && ops->callbacks.gso_segment)) - segs = ops->callbacks.gso_segment(skb, features); - - return segs; -} - -static void xfrm6_transport_xmit(struct xfrm_state *x, struct sk_buff *skb) -{ - struct xfrm_offload *xo = xfrm_offload(skb); - - skb_reset_mac_len(skb); - pskb_pull(skb, skb->mac_len + sizeof(struct ipv6hdr) + x->props.header_len); - - if (xo->flags & XFRM_GSO_SEGMENT) { - skb_reset_transport_header(skb); - skb->transport_header -= x->props.header_len; - } -} - - -static struct xfrm_mode xfrm6_transport_mode = { - .input = xfrm6_transport_input, - .output = xfrm6_transport_output, - .gso_segment = xfrm4_transport_gso_segment, - .xmit = xfrm6_transport_xmit, - .owner = THIS_MODULE, - .encap = XFRM_MODE_TRANSPORT, -}; - -static int __init xfrm6_transport_init(void) -{ - return xfrm_register_mode(&xfrm6_transport_mode, AF_INET6); -} - -static void __exit xfrm6_transport_exit(void) -{ - int err; - - err = xfrm_unregister_mode(&xfrm6_transport_mode, AF_INET6); - BUG_ON(err); -} - -module_init(xfrm6_transport_init); -module_exit(xfrm6_transport_exit); -MODULE_LICENSE("GPL"); -MODULE_ALIAS_XFRM_MODE(AF_INET6, XFRM_MODE_TRANSPORT); diff --git a/net/ipv6/xfrm6_mode_tunnel.c b/net/ipv6/xfrm6_mode_tunnel.c deleted file mode 100644 index de1b0b8c53b0ba26836d40a53dfa6fe077c7ebef..0000000000000000000000000000000000000000 --- a/net/ipv6/xfrm6_mode_tunnel.c +++ /dev/null @@ -1,151 +0,0 @@ -/* - * xfrm6_mode_tunnel.c - Tunnel mode encapsulation for IPv6. - * - * Copyright (C) 2002 USAGI/WIDE Project - * Copyright (c) 2004-2006 Herbert Xu - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -static inline void ipip6_ecn_decapsulate(struct sk_buff *skb) -{ - struct ipv6hdr *inner_iph = ipipv6_hdr(skb); - - if (INET_ECN_is_ce(XFRM_MODE_SKB_CB(skb)->tos)) - IP6_ECN_set_ce(skb, inner_iph); -} - -/* Add encapsulation header. - * - * The top IP header will be constructed per RFC 2401. - */ -static int xfrm6_mode_tunnel_output(struct xfrm_state *x, struct sk_buff *skb) -{ - struct dst_entry *dst = skb_dst(skb); - struct ipv6hdr *top_iph; - int dsfield; - - skb_set_inner_network_header(skb, skb_network_offset(skb)); - skb_set_inner_transport_header(skb, skb_transport_offset(skb)); - - skb_set_network_header(skb, -x->props.header_len); - skb->mac_header = skb->network_header + - offsetof(struct ipv6hdr, nexthdr); - skb->transport_header = skb->network_header + sizeof(*top_iph); - top_iph = ipv6_hdr(skb); - - top_iph->version = 6; - - memcpy(top_iph->flow_lbl, XFRM_MODE_SKB_CB(skb)->flow_lbl, - sizeof(top_iph->flow_lbl)); - top_iph->nexthdr = xfrm_af2proto(skb_dst(skb)->ops->family); - - if (x->props.extra_flags & XFRM_SA_XFLAG_DONT_ENCAP_DSCP) - dsfield = 0; - else - dsfield = XFRM_MODE_SKB_CB(skb)->tos; - dsfield = INET_ECN_encapsulate(dsfield, XFRM_MODE_SKB_CB(skb)->tos); - if (x->props.flags & XFRM_STATE_NOECN) - dsfield &= ~INET_ECN_MASK; - ipv6_change_dsfield(top_iph, 0, dsfield); - top_iph->hop_limit = ip6_dst_hoplimit(xfrm_dst_child(dst)); - top_iph->saddr = *(struct in6_addr *)&x->props.saddr; - top_iph->daddr = *(struct in6_addr *)&x->id.daddr; - return 0; -} - -#define for_each_input_rcu(head, handler) \ - for (handler = rcu_dereference(head); \ - handler != NULL; \ - handler = rcu_dereference(handler->next)) - - -static int xfrm6_mode_tunnel_input(struct xfrm_state *x, struct sk_buff *skb) -{ - int err = -EINVAL; - - if (XFRM_MODE_SKB_CB(skb)->protocol != IPPROTO_IPV6) - goto out; - if (!pskb_may_pull(skb, sizeof(struct ipv6hdr))) - goto out; - - err = skb_unclone(skb, GFP_ATOMIC); - if (err) - goto out; - - if (x->props.flags & XFRM_STATE_DECAP_DSCP) - ipv6_copy_dscp(ipv6_get_dsfield(ipv6_hdr(skb)), - ipipv6_hdr(skb)); - if (!(x->props.flags & XFRM_STATE_NOECN)) - ipip6_ecn_decapsulate(skb); - - skb_reset_network_header(skb); - skb_mac_header_rebuild(skb); - if (skb->mac_len) - eth_hdr(skb)->h_proto = skb->protocol; - - err = 0; - -out: - return err; -} - -static struct sk_buff *xfrm6_mode_tunnel_gso_segment(struct xfrm_state *x, - struct sk_buff *skb, - netdev_features_t features) -{ - __skb_push(skb, skb->mac_len); - return skb_mac_gso_segment(skb, features); -} - -static void xfrm6_mode_tunnel_xmit(struct xfrm_state *x, struct sk_buff *skb) -{ - struct xfrm_offload *xo = xfrm_offload(skb); - - if (xo->flags & XFRM_GSO_SEGMENT) - skb->transport_header = skb->network_header + sizeof(struct ipv6hdr); - - skb_reset_mac_len(skb); - pskb_pull(skb, skb->mac_len + x->props.header_len); -} - -static struct xfrm_mode xfrm6_tunnel_mode = { - .input2 = xfrm6_mode_tunnel_input, - .input = xfrm_prepare_input, - .output2 = xfrm6_mode_tunnel_output, - .output = xfrm6_prepare_output, - .gso_segment = xfrm6_mode_tunnel_gso_segment, - .xmit = xfrm6_mode_tunnel_xmit, - .owner = THIS_MODULE, - .encap = XFRM_MODE_TUNNEL, - .flags = XFRM_MODE_FLAG_TUNNEL, -}; - -static int __init xfrm6_mode_tunnel_init(void) -{ - return xfrm_register_mode(&xfrm6_tunnel_mode, AF_INET6); -} - -static void __exit xfrm6_mode_tunnel_exit(void) -{ - int err; - - err = xfrm_unregister_mode(&xfrm6_tunnel_mode, AF_INET6); - BUG_ON(err); -} - -module_init(xfrm6_mode_tunnel_init); -module_exit(xfrm6_mode_tunnel_exit); -MODULE_LICENSE("GPL"); -MODULE_ALIAS_XFRM_MODE(AF_INET6, XFRM_MODE_TUNNEL); diff --git a/net/ipv6/xfrm6_output.c b/net/ipv6/xfrm6_output.c index 6a74080005cf6acf15fa59d6d3dd14cbf01a1781..8ad5e54eb8ca201061d48aaa75938231658b3cc4 100644 --- a/net/ipv6/xfrm6_output.c +++ b/net/ipv6/xfrm6_output.c @@ -111,21 +111,6 @@ int xfrm6_extract_output(struct xfrm_state *x, struct sk_buff *skb) return xfrm6_extract_header(skb); } -int xfrm6_prepare_output(struct xfrm_state *x, struct sk_buff *skb) -{ - int err; - - err = xfrm_inner_extract_output(x, skb); - if (err) - return err; - - skb->ignore_df = 1; - skb->protocol = htons(ETH_P_IPV6); - - return x->outer_mode->output2(x, skb); -} -EXPORT_SYMBOL(xfrm6_prepare_output); - int xfrm6_output_finish(struct sock *sk, struct sk_buff *skb) { memset(IP6CB(skb), 0, sizeof(*IP6CB(skb))); @@ -137,11 +122,28 @@ int xfrm6_output_finish(struct sock *sk, struct sk_buff *skb) return xfrm_output(sk, skb); } +static int __xfrm6_output_state_finish(struct xfrm_state *x, struct sock *sk, + struct sk_buff *skb) +{ + const struct xfrm_state_afinfo *afinfo; + int ret = -EAFNOSUPPORT; + + rcu_read_lock(); + afinfo = xfrm_state_afinfo_get_rcu(x->outer_mode.family); + if (likely(afinfo)) + ret = afinfo->output_finish(sk, skb); + else + kfree_skb(skb); + rcu_read_unlock(); + + return ret; +} + static int __xfrm6_output_finish(struct net *net, struct sock *sk, struct sk_buff *skb) { struct xfrm_state *x = skb_dst(skb)->xfrm; - return x->outer_mode->afinfo->output_finish(sk, skb); + return __xfrm6_output_state_finish(x, sk, skb); } static int __xfrm6_output(struct net *net, struct sock *sk, struct sk_buff *skb) @@ -183,7 +185,7 @@ static int __xfrm6_output(struct net *net, struct sock *sk, struct sk_buff *skb) __xfrm6_output_finish); skip_frag: - return x->outer_mode->afinfo->output_finish(sk, skb); + return __xfrm6_output_state_finish(x, sk, skb); } int xfrm6_output(struct net *net, struct sock *sk, struct sk_buff *skb) diff --git a/net/ipv6/xfrm6_policy.c b/net/ipv6/xfrm6_policy.c index 769f8f78d3b8955a509cedc1f3823dbb7d3dce44..699e0730ce8e86545ef2812da316f8e726e4a75d 100644 --- a/net/ipv6/xfrm6_policy.c +++ b/net/ipv6/xfrm6_policy.c @@ -22,9 +22,6 @@ #include #include #include -#if IS_ENABLED(CONFIG_IPV6_MIP6) -#include -#endif static struct dst_entry *xfrm6_dst_lookup(struct net *net, int tos, int oif, const xfrm_address_t *saddr, @@ -71,24 +68,6 @@ static int xfrm6_get_saddr(struct net *net, int oif, return 0; } -static int xfrm6_get_tos(const struct flowi *fl) -{ - return 0; -} - -static int xfrm6_init_path(struct xfrm_dst *path, struct dst_entry *dst, - int nfheader_len) -{ - if (dst->ops->family == AF_INET6) { - struct rt6_info *rt = (struct rt6_info *)dst; - path->path_cookie = rt6_get_cookie(rt); - } - - path->u.rt6.rt6i_nfheader_len = nfheader_len; - - return 0; -} - static int xfrm6_fill_dst(struct xfrm_dst *xdst, struct net_device *dev, const struct flowi *fl) { @@ -118,108 +97,6 @@ static int xfrm6_fill_dst(struct xfrm_dst *xdst, struct net_device *dev, return 0; } -static inline void -_decode_session6(struct sk_buff *skb, struct flowi *fl, int reverse) -{ - struct flowi6 *fl6 = &fl->u.ip6; - int onlyproto = 0; - const struct ipv6hdr *hdr = ipv6_hdr(skb); - u32 offset = sizeof(*hdr); - struct ipv6_opt_hdr *exthdr; - const unsigned char *nh = skb_network_header(skb); - u16 nhoff = IP6CB(skb)->nhoff; - int oif = 0; - u8 nexthdr; - - if (!nhoff) - nhoff = offsetof(struct ipv6hdr, nexthdr); - - nexthdr = nh[nhoff]; - - if (skb_dst(skb)) - oif = skb_dst(skb)->dev->ifindex; - - memset(fl6, 0, sizeof(struct flowi6)); - fl6->flowi6_mark = skb->mark; - fl6->flowi6_oif = reverse ? skb->skb_iif : oif; - - fl6->daddr = reverse ? hdr->saddr : hdr->daddr; - fl6->saddr = reverse ? hdr->daddr : hdr->saddr; - - while (nh + offset + sizeof(*exthdr) < skb->data || - pskb_may_pull(skb, nh + offset + sizeof(*exthdr) - skb->data)) { - nh = skb_network_header(skb); - exthdr = (struct ipv6_opt_hdr *)(nh + offset); - - switch (nexthdr) { - case NEXTHDR_FRAGMENT: - onlyproto = 1; - /* fall through */ - case NEXTHDR_ROUTING: - case NEXTHDR_HOP: - case NEXTHDR_DEST: - offset += ipv6_optlen(exthdr); - nexthdr = exthdr->nexthdr; - exthdr = (struct ipv6_opt_hdr *)(nh + offset); - break; - - case IPPROTO_UDP: - case IPPROTO_UDPLITE: - case IPPROTO_TCP: - case IPPROTO_SCTP: - case IPPROTO_DCCP: - if (!onlyproto && (nh + offset + 4 < skb->data || - pskb_may_pull(skb, nh + offset + 4 - skb->data))) { - __be16 *ports; - - nh = skb_network_header(skb); - ports = (__be16 *)(nh + offset); - fl6->fl6_sport = ports[!!reverse]; - fl6->fl6_dport = ports[!reverse]; - } - fl6->flowi6_proto = nexthdr; - return; - - case IPPROTO_ICMPV6: - if (!onlyproto && (nh + offset + 2 < skb->data || - pskb_may_pull(skb, nh + offset + 2 - skb->data))) { - u8 *icmp; - - nh = skb_network_header(skb); - icmp = (u8 *)(nh + offset); - fl6->fl6_icmp_type = icmp[0]; - fl6->fl6_icmp_code = icmp[1]; - } - fl6->flowi6_proto = nexthdr; - return; - -#if IS_ENABLED(CONFIG_IPV6_MIP6) - case IPPROTO_MH: - offset += ipv6_optlen(exthdr); - if (!onlyproto && (nh + offset + 3 < skb->data || - pskb_may_pull(skb, nh + offset + 3 - skb->data))) { - struct ip6_mh *mh; - - nh = skb_network_header(skb); - mh = (struct ip6_mh *)(nh + offset); - fl6->fl6_mh_type = mh->ip6mh_type; - } - fl6->flowi6_proto = nexthdr; - return; -#endif - - /* XXX Why are there these headers? */ - case IPPROTO_AH: - case IPPROTO_ESP: - case IPPROTO_COMP: - default: - fl6->fl6_ipsec_spi = 0; - fl6->flowi6_proto = nexthdr; - return; - } - } -} - static void xfrm6_update_pmtu(struct dst_entry *dst, struct sock *sk, struct sk_buff *skb, u32 mtu) { @@ -291,9 +168,6 @@ static const struct xfrm_policy_afinfo xfrm6_policy_afinfo = { .dst_ops = &xfrm6_dst_ops_template, .dst_lookup = xfrm6_dst_lookup, .get_saddr = xfrm6_get_saddr, - .decode_session = _decode_session6, - .get_tos = xfrm6_get_tos, - .init_path = xfrm6_init_path, .fill_dst = xfrm6_fill_dst, .blackhole_route = ip6_blackhole_route, }; diff --git a/net/ipv6/xfrm6_protocol.c b/net/ipv6/xfrm6_protocol.c index cc979b702c898cb32405edd78d74dba9e1ac849e..aaacac7fdbce8c8891b3523c14f6f4edcd514609 100644 --- a/net/ipv6/xfrm6_protocol.c +++ b/net/ipv6/xfrm6_protocol.c @@ -46,7 +46,7 @@ static inline struct xfrm6_protocol __rcu **proto_handlers(u8 protocol) handler != NULL; \ handler = rcu_dereference(handler->next)) \ -int xfrm6_rcv_cb(struct sk_buff *skb, u8 protocol, int err) +static int xfrm6_rcv_cb(struct sk_buff *skb, u8 protocol, int err) { int ret; struct xfrm6_protocol *handler; @@ -61,7 +61,6 @@ int xfrm6_rcv_cb(struct sk_buff *skb, u8 protocol, int err) return 0; } -EXPORT_SYMBOL(xfrm6_rcv_cb); static int xfrm6_esp_rcv(struct sk_buff *skb) { diff --git a/net/l2tp/l2tp_core.c b/net/l2tp/l2tp_core.c index 52b5a2797c0c6e85e0cd2f8203616b536b86d178..e4dec03a19fe99d9a39e6b87fb26881e0bd09bba 100644 --- a/net/l2tp/l2tp_core.c +++ b/net/l2tp/l2tp_core.c @@ -1735,7 +1735,8 @@ static __net_exit void l2tp_exit_net(struct net *net) } rcu_read_unlock_bh(); - flush_workqueue(l2tp_wq); + if (l2tp_wq) + flush_workqueue(l2tp_wq); rcu_barrier(); for (hash = 0; hash < L2TP_HASH_SIZE_2; hash++) diff --git a/net/l2tp/l2tp_ip.c b/net/l2tp/l2tp_ip.c index d4c60523c549d33c6ea29641a37a63040b70530d..2cac910c1cd4e5eaba136db2877403ad9618324e 100644 --- a/net/l2tp/l2tp_ip.c +++ b/net/l2tp/l2tp_ip.c @@ -618,6 +618,7 @@ static const struct proto_ops l2tp_ip_ops = { .getname = l2tp_ip_getname, .poll = datagram_poll, .ioctl = inet_ioctl, + .gettstamp = sock_gettstamp, .listen = sock_no_listen, .shutdown = inet_shutdown, .setsockopt = sock_common_setsockopt, diff --git a/net/l2tp/l2tp_ip6.c b/net/l2tp/l2tp_ip6.c index 37a69df17cab982e427e158966b8059886ddd456..4ec546cc1dd6c86bdb3aa4e570c6aecd34e02971 100644 --- a/net/l2tp/l2tp_ip6.c +++ b/net/l2tp/l2tp_ip6.c @@ -752,6 +752,7 @@ static const struct proto_ops l2tp_ip6_ops = { .getname = l2tp_ip6_getname, .poll = datagram_poll, .ioctl = inet6_ioctl, + .gettstamp = sock_gettstamp, .listen = sock_no_listen, .shutdown = inet_shutdown, .setsockopt = sock_common_setsockopt, diff --git a/net/l2tp/l2tp_netlink.c b/net/l2tp/l2tp_netlink.c index edbd5d1fbcde63fd9de6fef070b2787fcd5d993c..6acc7f869b0c81614fbb764fdccf6d23118aea69 100644 --- a/net/l2tp/l2tp_netlink.c +++ b/net/l2tp/l2tp_netlink.c @@ -345,7 +345,7 @@ static int l2tp_nl_tunnel_send(struct sk_buff *skb, u32 portid, u32 seq, int fla nla_put_u16(skb, L2TP_ATTR_ENCAP_TYPE, tunnel->encap)) goto nla_put_failure; - nest = nla_nest_start(skb, L2TP_ATTR_STATS); + nest = nla_nest_start_noflag(skb, L2TP_ATTR_STATS); if (nest == NULL) goto nla_put_failure; @@ -742,7 +742,7 @@ static int l2tp_nl_session_send(struct sk_buff *skb, u32 portid, u32 seq, int fl session->reorder_timeout, L2TP_ATTR_PAD))) goto nla_put_failure; - nest = nla_nest_start(skb, L2TP_ATTR_STATS); + nest = nla_nest_start_noflag(skb, L2TP_ATTR_STATS); if (nest == NULL) goto nla_put_failure; @@ -915,58 +915,58 @@ static const struct nla_policy l2tp_nl_policy[L2TP_ATTR_MAX + 1] = { static const struct genl_ops l2tp_nl_ops[] = { { .cmd = L2TP_CMD_NOOP, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = l2tp_nl_cmd_noop, - .policy = l2tp_nl_policy, /* can be retrieved by unprivileged users */ }, { .cmd = L2TP_CMD_TUNNEL_CREATE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = l2tp_nl_cmd_tunnel_create, - .policy = l2tp_nl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = L2TP_CMD_TUNNEL_DELETE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = l2tp_nl_cmd_tunnel_delete, - .policy = l2tp_nl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = L2TP_CMD_TUNNEL_MODIFY, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = l2tp_nl_cmd_tunnel_modify, - .policy = l2tp_nl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = L2TP_CMD_TUNNEL_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = l2tp_nl_cmd_tunnel_get, .dumpit = l2tp_nl_cmd_tunnel_dump, - .policy = l2tp_nl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = L2TP_CMD_SESSION_CREATE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = l2tp_nl_cmd_session_create, - .policy = l2tp_nl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = L2TP_CMD_SESSION_DELETE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = l2tp_nl_cmd_session_delete, - .policy = l2tp_nl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = L2TP_CMD_SESSION_MODIFY, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = l2tp_nl_cmd_session_modify, - .policy = l2tp_nl_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = L2TP_CMD_SESSION_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = l2tp_nl_cmd_session_get, .dumpit = l2tp_nl_cmd_session_dump, - .policy = l2tp_nl_policy, .flags = GENL_ADMIN_PERM, }, }; @@ -976,6 +976,7 @@ static struct genl_family l2tp_nl_family __ro_after_init = { .version = L2TP_GENL_VERSION, .hdrsize = 0, .maxattr = L2TP_ATTR_MAX, + .policy = l2tp_nl_policy, .netnsok = true, .module = THIS_MODULE, .ops = l2tp_nl_ops, diff --git a/net/l2tp/l2tp_ppp.c b/net/l2tp/l2tp_ppp.c index 04d9946dcdba648f8e4dce9c0c0211f7036ae135..f36cae785e8217be3e9aed72d5fe7c01939511b6 100644 --- a/net/l2tp/l2tp_ppp.c +++ b/net/l2tp/l2tp_ppp.c @@ -1070,7 +1070,6 @@ static int pppol2tp_ioctl(struct socket *sock, unsigned int cmd, { struct pppol2tp_ioc_stats stats; struct l2tp_session *session; - int val; switch (cmd) { case PPPIOCGMRU: @@ -1097,7 +1096,7 @@ static int pppol2tp_ioctl(struct socket *sock, unsigned int cmd, if (!session->session_id && !session->peer_session_id) return -ENOSYS; - if (get_user(val, (int __user *)arg)) + if (!access_ok((int __user *)arg, sizeof(int))) return -EFAULT; break; diff --git a/net/mac80211/cfg.c b/net/mac80211/cfg.c index 09dd1c2860fc030f584fb895ca12483a13592823..52e6a091b7e40800a989fdb12fe6acfffb00124f 100644 --- a/net/mac80211/cfg.c +++ b/net/mac80211/cfg.c @@ -351,6 +351,36 @@ static int ieee80211_set_noack_map(struct wiphy *wiphy, return 0; } +static int ieee80211_set_tx(struct ieee80211_sub_if_data *sdata, + const u8 *mac_addr, u8 key_idx) +{ + struct ieee80211_local *local = sdata->local; + struct ieee80211_key *key; + struct sta_info *sta; + int ret = -EINVAL; + + if (!wiphy_ext_feature_isset(local->hw.wiphy, + NL80211_EXT_FEATURE_EXT_KEY_ID)) + return -EINVAL; + + sta = sta_info_get_bss(sdata, mac_addr); + + if (!sta) + return -EINVAL; + + if (sta->ptk_idx == key_idx) + return 0; + + mutex_lock(&local->key_mtx); + key = key_mtx_dereference(local, sta->ptk[key_idx]); + + if (key && key->conf.flags & IEEE80211_KEY_FLAG_NO_AUTO_TX) + ret = ieee80211_set_tx_key(key); + + mutex_unlock(&local->key_mtx); + return ret; +} + static int ieee80211_add_key(struct wiphy *wiphy, struct net_device *dev, u8 key_idx, bool pairwise, const u8 *mac_addr, struct key_params *params) @@ -365,6 +395,9 @@ static int ieee80211_add_key(struct wiphy *wiphy, struct net_device *dev, if (!ieee80211_sdata_running(sdata)) return -ENETDOWN; + if (pairwise && params->mode == NL80211_KEY_SET_TX) + return ieee80211_set_tx(sdata, mac_addr, key_idx); + /* reject WEP and TKIP keys if WEP failed to initialize */ switch (params->cipher) { case WLAN_CIPHER_SUITE_WEP40: @@ -396,6 +429,9 @@ static int ieee80211_add_key(struct wiphy *wiphy, struct net_device *dev, if (pairwise) key->conf.flags |= IEEE80211_KEY_FLAG_PAIRWISE; + if (params->mode == NL80211_KEY_NO_TX) + key->conf.flags |= IEEE80211_KEY_FLAG_NO_AUTO_TX; + mutex_lock(&local->sta_mtx); if (mac_addr) { @@ -1421,6 +1457,15 @@ static int sta_apply_parameters(struct ieee80211_local *local, if (params->listen_interval >= 0) sta->listen_interval = params->listen_interval; + if (params->sta_modify_mask & STATION_PARAM_APPLY_STA_TXPOWER) { + sta->sta.txpwr.type = params->txpwr.type; + if (params->txpwr.type == NL80211_TX_POWER_LIMITED) + sta->sta.txpwr.power = params->txpwr.power; + ret = drv_sta_set_txpwr(local, sdata, sta); + if (ret) + return ret; + } + if (params->supported_rates) { ieee80211_parse_bitrates(&sdata->vif.bss_conf.chandef, sband, params->supported_rates, @@ -3990,4 +4035,5 @@ const struct cfg80211_ops mac80211_config_ops = { .get_ftm_responder_stats = ieee80211_get_ftm_responder_stats, .start_pmsr = ieee80211_start_pmsr, .abort_pmsr = ieee80211_abort_pmsr, + .probe_mesh_link = ieee80211_probe_mesh_link, }; diff --git a/net/mac80211/debugfs.c b/net/mac80211/debugfs.c index 2d43bc127043617329fd8296ce230e81f5260412..0d462206eef60b1932af8310a7e4fdd83c183056 100644 --- a/net/mac80211/debugfs.c +++ b/net/mac80211/debugfs.c @@ -150,6 +150,58 @@ static const struct file_operations aqm_ops = { .llseek = default_llseek, }; +static ssize_t force_tx_status_read(struct file *file, + char __user *user_buf, + size_t count, + loff_t *ppos) +{ + struct ieee80211_local *local = file->private_data; + char buf[3]; + int len = 0; + + len = scnprintf(buf, sizeof(buf), "%d\n", (int)local->force_tx_status); + + return simple_read_from_buffer(user_buf, count, ppos, + buf, len); +} + +static ssize_t force_tx_status_write(struct file *file, + const char __user *user_buf, + size_t count, + loff_t *ppos) +{ + struct ieee80211_local *local = file->private_data; + char buf[3]; + size_t len; + + if (count > sizeof(buf)) + return -EINVAL; + + if (copy_from_user(buf, user_buf, count)) + return -EFAULT; + + buf[sizeof(buf) - 1] = '\0'; + len = strlen(buf); + if (len > 0 && buf[len - 1] == '\n') + buf[len - 1] = 0; + + if (buf[0] == '0' && buf[1] == '\0') + local->force_tx_status = 0; + else if (buf[0] == '1' && buf[1] == '\0') + local->force_tx_status = 1; + else + return -EINVAL; + + return count; +} + +static const struct file_operations force_tx_status_ops = { + .write = force_tx_status_write, + .read = force_tx_status_read, + .open = simple_open, + .llseek = default_llseek, +}; + #ifdef CONFIG_PM static ssize_t reset_write(struct file *file, const char __user *user_buf, size_t count, loff_t *ppos) @@ -221,6 +273,7 @@ static const char *hw_flag_names[] = { FLAG(TX_STATUS_NO_AMPDU_LEN), FLAG(SUPPORTS_MULTI_BSSID), FLAG(SUPPORTS_ONLY_HE_MULTI_BSSID), + FLAG(EXT_KEY_ID_NATIVE), #undef FLAG }; @@ -382,6 +435,7 @@ void debugfs_hw_add(struct ieee80211_local *local) DEBUGFS_ADD(hwflags); DEBUGFS_ADD(user_power); DEBUGFS_ADD(power); + DEBUGFS_ADD_MODE(force_tx_status, 0600); if (local->ops->wake_tx_queue) DEBUGFS_ADD_MODE(aqm, 0600); diff --git a/net/mac80211/driver-ops.c b/net/mac80211/driver-ops.c index bb886e7db47f1dcf409a4746d7ee47be8c679462..839c0022a29cc8fd5b539e5c0f4d5266cc1c3b28 100644 --- a/net/mac80211/driver-ops.c +++ b/net/mac80211/driver-ops.c @@ -138,6 +138,27 @@ int drv_sta_state(struct ieee80211_local *local, return ret; } +__must_check +int drv_sta_set_txpwr(struct ieee80211_local *local, + struct ieee80211_sub_if_data *sdata, + struct sta_info *sta) +{ + int ret = -EOPNOTSUPP; + + might_sleep(); + + sdata = get_bss_sdata(sdata); + if (!check_sdata_in_driver(sdata)) + return -EIO; + + trace_drv_sta_set_txpwr(local, sdata, &sta->sta); + if (local->ops->sta_set_txpwr) + ret = local->ops->sta_set_txpwr(&local->hw, &sdata->vif, + &sta->sta); + trace_drv_return_int(local, ret); + return ret; +} + void drv_sta_rc_update(struct ieee80211_local *local, struct ieee80211_sub_if_data *sdata, struct ieee80211_sta *sta, u32 changed) diff --git a/net/mac80211/driver-ops.h b/net/mac80211/driver-ops.h index ae4f0be3b393ba727b95060bb7148ec0cd961440..c2d8b5451a5ec6c52245ed306d5e0d4e607b3c67 100644 --- a/net/mac80211/driver-ops.h +++ b/net/mac80211/driver-ops.h @@ -529,6 +529,11 @@ int drv_sta_state(struct ieee80211_local *local, enum ieee80211_sta_state old_state, enum ieee80211_sta_state new_state); +__must_check +int drv_sta_set_txpwr(struct ieee80211_local *local, + struct ieee80211_sub_if_data *sdata, + struct sta_info *sta); + void drv_sta_rc_update(struct ieee80211_local *local, struct ieee80211_sub_if_data *sdata, struct ieee80211_sta *sta, u32 changed); diff --git a/net/mac80211/ieee80211_i.h b/net/mac80211/ieee80211_i.h index e170f986d226891542bb351d6d3ea30dde44fb7b..073a8235ae1bd8cfcd28640a31bbb66bb4d22f96 100644 --- a/net/mac80211/ieee80211_i.h +++ b/net/mac80211/ieee80211_i.h @@ -1269,7 +1269,7 @@ struct ieee80211_local { /* * Key mutex, protects sdata's key_list and sta_info's - * key pointers (write access, they're RCU.) + * key pointers and ptk_idx (write access, they're RCU.) */ struct mutex key_mtx; @@ -1384,6 +1384,7 @@ struct ieee80211_local { struct dentry *rcdir; struct dentry *keys; } debugfs; + bool force_tx_status; #endif /* @@ -1505,7 +1506,6 @@ struct ieee802_11_elems { const struct ieee80211_bss_max_idle_period_ie *max_idle_period_ie; const struct ieee80211_multiple_bssid_configuration *mbssid_config_ie; const struct ieee80211_bssid_index *bssid_index; - const u8 *nontransmitted_bssid_profile; u8 max_bssid_indicator; u8 dtim_count; u8 dtim_period; @@ -1761,7 +1761,8 @@ netdev_tx_t ieee80211_subif_start_xmit(struct sk_buff *skb, struct net_device *dev); void __ieee80211_subif_start_xmit(struct sk_buff *skb, struct net_device *dev, - u32 info_flags); + u32 info_flags, + u32 ctrl_flags); void ieee80211_purge_tx_queue(struct ieee80211_hw *hw, struct sk_buff_head *skbs); struct sk_buff * @@ -1778,6 +1779,8 @@ void ieee80211_clear_fast_xmit(struct sta_info *sta); int ieee80211_tx_control_port(struct wiphy *wiphy, struct net_device *dev, const u8 *buf, size_t len, const u8 *dest, __be16 proto, bool unencrypted); +int ieee80211_probe_mesh_link(struct wiphy *wiphy, struct net_device *dev, + const u8 *buf, size_t len); /* HT */ void ieee80211_apply_htcap_overrides(struct ieee80211_sub_if_data *sdata, diff --git a/net/mac80211/iface.c b/net/mac80211/iface.c index 02d2e6f11e936fb54814b909849116de173a91c9..410685d38c4603c7acc2dccce7aa0f8359a3ee47 100644 --- a/net/mac80211/iface.c +++ b/net/mac80211/iface.c @@ -1133,8 +1133,7 @@ static void ieee80211_uninit(struct net_device *dev) static u16 ieee80211_netdev_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { return ieee80211_select_queue(IEEE80211_DEV_TO_SUB_IF(dev), skb); } @@ -1179,8 +1178,7 @@ static const struct net_device_ops ieee80211_dataif_ops = { static u16 ieee80211_monitor_select_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev, - select_queue_fallback_t fallback) + struct net_device *sb_dev) { struct ieee80211_sub_if_data *sdata = IEEE80211_DEV_TO_SUB_IF(dev); struct ieee80211_local *local = sdata->local; @@ -1227,6 +1225,7 @@ static void ieee80211_if_setup(struct net_device *dev) static void ieee80211_if_setup_no_queue(struct net_device *dev) { ieee80211_if_setup(dev); + dev->features |= NETIF_F_LLTX; dev->priv_flags |= IFF_NO_QUEUE; } @@ -1764,13 +1763,13 @@ int ieee80211_if_add(struct ieee80211_local *local, const char *name, txq_size += sizeof(struct txq_info) + local->hw.txq_data_size; - if (local->ops->wake_tx_queue) + if (local->ops->wake_tx_queue) { if_setup = ieee80211_if_setup_no_queue; - else + } else { if_setup = ieee80211_if_setup; - - if (local->hw.queues >= IEEE80211_NUM_ACS) - txqs = IEEE80211_NUM_ACS; + if (local->hw.queues >= IEEE80211_NUM_ACS) + txqs = IEEE80211_NUM_ACS; + } ndev = alloc_netdev_mqs(size + txq_size, name, name_assign_type, diff --git a/net/mac80211/key.c b/net/mac80211/key.c index 37e372896230a08c6a9214f88ce54e7ad823d352..20bf9db7a3886de7601fabd0e0ff2c721e54e30e 100644 --- a/net/mac80211/key.c +++ b/net/mac80211/key.c @@ -140,6 +140,12 @@ static int ieee80211_key_enable_hw_accel(struct ieee80211_key *key) * so clear that flag now to avoid trying to remove * it again later. */ + if (key->flags & KEY_FLAG_UPLOADED_TO_HARDWARE && + !(key->conf.flags & (IEEE80211_KEY_FLAG_GENERATE_MMIC | + IEEE80211_KEY_FLAG_PUT_MIC_SPACE | + IEEE80211_KEY_FLAG_RESERVE_TAILROOM))) + increment_tailroom_need_count(sdata); + key->flags &= ~KEY_FLAG_UPLOADED_TO_HARDWARE; return -EINVAL; } @@ -179,9 +185,9 @@ static int ieee80211_key_enable_hw_accel(struct ieee80211_key *key) if (!ret) { key->flags |= KEY_FLAG_UPLOADED_TO_HARDWARE; - if (!((key->conf.flags & (IEEE80211_KEY_FLAG_GENERATE_MMIC | - IEEE80211_KEY_FLAG_PUT_MIC_SPACE)) || - (key->conf.flags & IEEE80211_KEY_FLAG_RESERVE_TAILROOM))) + if (!(key->conf.flags & (IEEE80211_KEY_FLAG_GENERATE_MMIC | + IEEE80211_KEY_FLAG_PUT_MIC_SPACE | + IEEE80211_KEY_FLAG_RESERVE_TAILROOM))) decrease_tailroom_need_count(sdata, 1); WARN_ON((key->conf.flags & IEEE80211_KEY_FLAG_PUT_IV_SPACE) && @@ -242,9 +248,9 @@ static void ieee80211_key_disable_hw_accel(struct ieee80211_key *key) sta = key->sta; sdata = key->sdata; - if (!((key->conf.flags & (IEEE80211_KEY_FLAG_GENERATE_MMIC | - IEEE80211_KEY_FLAG_PUT_MIC_SPACE)) || - (key->conf.flags & IEEE80211_KEY_FLAG_RESERVE_TAILROOM))) + if (!(key->conf.flags & (IEEE80211_KEY_FLAG_GENERATE_MMIC | + IEEE80211_KEY_FLAG_PUT_MIC_SPACE | + IEEE80211_KEY_FLAG_RESERVE_TAILROOM))) increment_tailroom_need_count(sdata); key->flags &= ~KEY_FLAG_UPLOADED_TO_HARDWARE; @@ -258,9 +264,24 @@ static void ieee80211_key_disable_hw_accel(struct ieee80211_key *key) sta ? sta->sta.addr : bcast_addr, ret); } +int ieee80211_set_tx_key(struct ieee80211_key *key) +{ + struct sta_info *sta = key->sta; + struct ieee80211_local *local = key->local; + struct ieee80211_key *old; + + assert_key_lock(local); + + old = key_mtx_dereference(local, sta->ptk[sta->ptk_idx]); + sta->ptk_idx = key->conf.keyidx; + ieee80211_check_fast_xmit(sta); + + return 0; +} + static int ieee80211_hw_key_replace(struct ieee80211_key *old_key, struct ieee80211_key *new_key, - bool ptk0rekey) + bool pairwise) { struct ieee80211_sub_if_data *sdata; struct ieee80211_local *local; @@ -277,8 +298,9 @@ static int ieee80211_hw_key_replace(struct ieee80211_key *old_key, assert_key_lock(old_key->local); sta = old_key->sta; - /* PTK only using key ID 0 needs special handling on rekey */ - if (new_key && sta && ptk0rekey) { + /* Unicast rekey without Extended Key ID needs special handling */ + if (new_key && sta && pairwise && + rcu_access_pointer(sta->ptk[sta->ptk_idx]) == old_key) { local = old_key->local; sdata = old_key->sdata; @@ -394,10 +416,6 @@ static int ieee80211_key_replace(struct ieee80211_sub_if_data *sdata, if (old) { idx = old->conf.keyidx; - /* TODO: proper implement and test "Extended Key ID for - * Individually Addressed Frames" from IEEE 802.11-2016. - * Till then always assume only key ID 0 is used for - * pairwise keys.*/ ret = ieee80211_hw_key_replace(old, new, pairwise); } else { /* new must be provided in case old is not */ @@ -414,15 +432,20 @@ static int ieee80211_key_replace(struct ieee80211_sub_if_data *sdata, if (sta) { if (pairwise) { rcu_assign_pointer(sta->ptk[idx], new); - sta->ptk_idx = idx; - if (new) { + if (new && + !(new->conf.flags & IEEE80211_KEY_FLAG_NO_AUTO_TX)) { + sta->ptk_idx = idx; clear_sta_flag(sta, WLAN_STA_BLOCK_BA); ieee80211_check_fast_xmit(sta); } } else { rcu_assign_pointer(sta->gtk[idx], new); } - if (new) + /* Only needed for transition from no key -> key. + * Still triggers unnecessary when using Extended Key ID + * and installing the second key ID the first time. + */ + if (new && !old) ieee80211_check_fast_rx(sta); } else { defunikey = old && @@ -738,16 +761,34 @@ int ieee80211_key_link(struct ieee80211_key *key, * can cause warnings to appear. */ bool delay_tailroom = sdata->vif.type == NL80211_IFTYPE_STATION; - int ret; + int ret = -EOPNOTSUPP; mutex_lock(&sdata->local->key_mtx); - if (sta && pairwise) + if (sta && pairwise) { + struct ieee80211_key *alt_key; + old_key = key_mtx_dereference(sdata->local, sta->ptk[idx]); - else if (sta) + alt_key = key_mtx_dereference(sdata->local, sta->ptk[idx ^ 1]); + + /* The rekey code assumes that the old and new key are using + * the same cipher. Enforce the assumption for pairwise keys. + */ + if (key && + ((alt_key && alt_key->conf.cipher != key->conf.cipher) || + (old_key && old_key->conf.cipher != key->conf.cipher))) + goto out; + } else if (sta) { old_key = key_mtx_dereference(sdata->local, sta->gtk[idx]); - else + } else { old_key = key_mtx_dereference(sdata->local, sdata->keys[idx]); + } + + /* Non-pairwise keys must also not switch the cipher on rekey */ + if (!pairwise) { + if (key && old_key && old_key->conf.cipher != key->conf.cipher) + goto out; + } /* * Silently accept key re-installation without really installing the @@ -1187,9 +1228,9 @@ void ieee80211_remove_key(struct ieee80211_key_conf *keyconf) if (key->flags & KEY_FLAG_UPLOADED_TO_HARDWARE) { key->flags &= ~KEY_FLAG_UPLOADED_TO_HARDWARE; - if (!((key->conf.flags & (IEEE80211_KEY_FLAG_GENERATE_MMIC | - IEEE80211_KEY_FLAG_PUT_MIC_SPACE)) || - (key->conf.flags & IEEE80211_KEY_FLAG_RESERVE_TAILROOM))) + if (!(key->conf.flags & (IEEE80211_KEY_FLAG_GENERATE_MMIC | + IEEE80211_KEY_FLAG_PUT_MIC_SPACE | + IEEE80211_KEY_FLAG_RESERVE_TAILROOM))) increment_tailroom_need_count(key->sdata); } diff --git a/net/mac80211/key.h b/net/mac80211/key.h index ebdb80b85dc3aa3b12da1f8925df3552a51674b3..f06fbd03d23566795db87ba52df192cdeee6936f 100644 --- a/net/mac80211/key.h +++ b/net/mac80211/key.h @@ -18,6 +18,7 @@ #define NUM_DEFAULT_KEYS 4 #define NUM_DEFAULT_MGMT_KEYS 2 +#define INVALID_PTK_KEYIDX 2 /* Keyidx always pointing to a NULL key for PTK */ struct ieee80211_local; struct ieee80211_sub_if_data; @@ -146,6 +147,7 @@ ieee80211_key_alloc(u32 cipher, int idx, size_t key_len, int ieee80211_key_link(struct ieee80211_key *key, struct ieee80211_sub_if_data *sdata, struct sta_info *sta); +int ieee80211_set_tx_key(struct ieee80211_key *key); void ieee80211_key_free(struct ieee80211_key *key, bool delay_tailroom); void ieee80211_key_free_unused(struct ieee80211_key *key); void ieee80211_set_default_key(struct ieee80211_sub_if_data *sdata, int idx, diff --git a/net/mac80211/main.c b/net/mac80211/main.c index 800e67615e2a65d1a60690e4933f672839cb1216..2b608044ae23be931715e02c7119d41407f24e52 100644 --- a/net/mac80211/main.c +++ b/net/mac80211/main.c @@ -1051,6 +1051,22 @@ int ieee80211_register_hw(struct ieee80211_hw *hw) } } + /* Enable Extended Key IDs when driver allowed it, or when it + * supports neither HW crypto nor A-MPDUs + */ + if ((!local->ops->set_key && + !ieee80211_hw_check(hw, AMPDU_AGGREGATION)) || + ieee80211_hw_check(&local->hw, EXT_KEY_ID_NATIVE)) + wiphy_ext_feature_set(local->hw.wiphy, + NL80211_EXT_FEATURE_EXT_KEY_ID); + + /* Mac80211 and therefore all cards only using SW crypto are able to + * handle PTK rekeys correctly + */ + if (!local->ops->set_key) + wiphy_ext_feature_set(local->hw.wiphy, + NL80211_EXT_FEATURE_CAN_REPLACE_PTK0); + /* * Calculate scan IE length -- we need this to alloc * memory and to subtract from the driver limit. It diff --git a/net/mac80211/mesh.h b/net/mac80211/mesh.h index 574c3891c4b2e8b6d7c6031811acdb50b94355be..88535a2e62bcfcd754eb02ba6d76a3eb8fc65241 100644 --- a/net/mac80211/mesh.h +++ b/net/mac80211/mesh.h @@ -278,6 +278,8 @@ mesh_path_add(struct ieee80211_sub_if_data *sdata, const u8 *dst); int mesh_path_add_gate(struct mesh_path *mpath); int mesh_path_send_to_gates(struct mesh_path *mpath); int mesh_gate_num(struct ieee80211_sub_if_data *sdata); +u32 airtime_link_metric_get(struct ieee80211_local *local, + struct sta_info *sta); /* Mesh plinks */ void mesh_neighbour_update(struct ieee80211_sub_if_data *sdata, diff --git a/net/mac80211/mesh_hwmp.c b/net/mac80211/mesh_hwmp.c index f7517668e77a0fdfd74003af482371ac6fc14f04..bf8e13cd5fd16f31fc7c059b64da456c3428db14 100644 --- a/net/mac80211/mesh_hwmp.c +++ b/net/mac80211/mesh_hwmp.c @@ -318,8 +318,8 @@ void ieee80211s_update_metric(struct ieee80211_local *local, cfg80211_calculate_bitrate(&rinfo)); } -static u32 airtime_link_metric_get(struct ieee80211_local *local, - struct sta_info *sta) +u32 airtime_link_metric_get(struct ieee80211_local *local, + struct sta_info *sta) { /* This should be adjusted for each device */ int device_constant = 1 << ARITH_SHIFT; @@ -1130,16 +1130,17 @@ int mesh_nexthop_resolve(struct ieee80211_sub_if_data *sdata, struct mesh_path *mpath; struct sk_buff *skb_to_free = NULL; u8 *target_addr = hdr->addr3; - int err = 0; /* Nulls are only sent to peers for PS and should be pre-addressed */ if (ieee80211_is_qos_nullfunc(hdr->frame_control)) return 0; - rcu_read_lock(); - err = mesh_nexthop_lookup(sdata, skb); - if (!err) - goto endlookup; + /* Allow injected packets to bypass mesh routing */ + if (info->control.flags & IEEE80211_TX_CTRL_SKIP_MPATH_LOOKUP) + return 0; + + if (!mesh_nexthop_lookup(sdata, skb)) + return 0; /* no nexthop found, start resolving */ mpath = mesh_path_lookup(sdata, target_addr); @@ -1147,8 +1148,7 @@ int mesh_nexthop_resolve(struct ieee80211_sub_if_data *sdata, mpath = mesh_path_add(sdata, target_addr); if (IS_ERR(mpath)) { mesh_path_discard_frame(sdata, skb); - err = PTR_ERR(mpath); - goto endlookup; + return PTR_ERR(mpath); } } @@ -1161,13 +1161,10 @@ int mesh_nexthop_resolve(struct ieee80211_sub_if_data *sdata, info->flags |= IEEE80211_TX_INTFL_NEED_TXPROCESSING; ieee80211_set_qos_hdr(sdata, skb); skb_queue_tail(&mpath->frame_queue, skb); - err = -ENOENT; if (skb_to_free) mesh_path_discard_frame(sdata, skb_to_free); -endlookup: - rcu_read_unlock(); - return err; + return -ENOENT; } /** @@ -1187,13 +1184,10 @@ int mesh_nexthop_lookup(struct ieee80211_sub_if_data *sdata, struct sta_info *next_hop; struct ieee80211_hdr *hdr = (struct ieee80211_hdr *) skb->data; u8 *target_addr = hdr->addr3; - int err = -ENOENT; - rcu_read_lock(); mpath = mesh_path_lookup(sdata, target_addr); - if (!mpath || !(mpath->flags & MESH_PATH_ACTIVE)) - goto endlookup; + return -ENOENT; if (time_after(jiffies, mpath->exp_time - @@ -1208,12 +1202,10 @@ int mesh_nexthop_lookup(struct ieee80211_sub_if_data *sdata, memcpy(hdr->addr1, next_hop->sta.addr, ETH_ALEN); memcpy(hdr->addr2, sdata->vif.addr, ETH_ALEN); ieee80211_mps_set_frame_flags(sdata, next_hop, hdr); - err = 0; + return 0; } -endlookup: - rcu_read_unlock(); - return err; + return -ENOENT; } void mesh_path_timer(struct timer_list *t) diff --git a/net/mac80211/mesh_pathtbl.c b/net/mac80211/mesh_pathtbl.c index b76a2aefa9ec05e5162ab565a108b5b98848116f..796b069ad251286860ce6f863d878db74eebae44 100644 --- a/net/mac80211/mesh_pathtbl.c +++ b/net/mac80211/mesh_pathtbl.c @@ -217,7 +217,7 @@ static struct mesh_path *mpath_lookup(struct mesh_table *tbl, const u8 *dst, { struct mesh_path *mpath; - mpath = rhashtable_lookup_fast(&tbl->rhead, dst, mesh_rht_params); + mpath = rhashtable_lookup(&tbl->rhead, dst, mesh_rht_params); if (mpath && mpath_expired(mpath)) { spin_lock_bh(&mpath->state_lock); diff --git a/net/mac80211/mlme.c b/net/mac80211/mlme.c index 2dbcf5d5512efe26de6387cc851cd9fc801a1293..b7a9fe3d5fcb75dfb15e27c536d19066993b0c6d 100644 --- a/net/mac80211/mlme.c +++ b/net/mac80211/mlme.c @@ -1188,9 +1188,6 @@ static void ieee80211_chswitch_work(struct work_struct *work) goto out; } - /* XXX: shouldn't really modify cfg80211-owned data! */ - ifmgd->associated->channel = sdata->csa_chandef.chan; - ifmgd->csa_waiting_bcn = true; ieee80211_sta_reset_beacon_monitor(sdata); diff --git a/net/mac80211/rc80211_minstrel_ht.c b/net/mac80211/rc80211_minstrel_ht.c index ccaf951e4e3122c1291f88f12dded5341f712b09..8b168724c5e7ead9ac04b5efa689e1855b9bc842 100644 --- a/net/mac80211/rc80211_minstrel_ht.c +++ b/net/mac80211/rc80211_minstrel_ht.c @@ -51,8 +51,13 @@ MINSTREL_MAX_STREAMS * _sgi + \ _streams - 1 +#define _MAX(a, b) (((a)>(b))?(a):(b)) + +#define GROUP_SHIFT(duration) \ + _MAX(0, 16 - __builtin_clz(duration)) + /* MCS rate information for an MCS group */ -#define MCS_GROUP(_streams, _sgi, _ht40, _s) \ +#define __MCS_GROUP(_streams, _sgi, _ht40, _s) \ [GROUP_IDX(_streams, _sgi, _ht40)] = { \ .streams = _streams, \ .shift = _s, \ @@ -72,6 +77,13 @@ } \ } +#define MCS_GROUP_SHIFT(_streams, _sgi, _ht40) \ + GROUP_SHIFT(MCS_DURATION(_streams, _sgi, _ht40 ? 54 : 26)) + +#define MCS_GROUP(_streams, _sgi, _ht40) \ + __MCS_GROUP(_streams, _sgi, _ht40, \ + MCS_GROUP_SHIFT(_streams, _sgi, _ht40)) + #define VHT_GROUP_IDX(_streams, _sgi, _bw) \ (MINSTREL_VHT_GROUP_0 + \ MINSTREL_MAX_STREAMS * 2 * (_bw) + \ @@ -81,7 +93,7 @@ #define BW2VBPS(_bw, r3, r2, r1) \ (_bw == BW_80 ? r3 : _bw == BW_40 ? r2 : r1) -#define VHT_GROUP(_streams, _sgi, _bw, _s) \ +#define __VHT_GROUP(_streams, _sgi, _bw, _s) \ [VHT_GROUP_IDX(_streams, _sgi, _bw)] = { \ .streams = _streams, \ .shift = _s, \ @@ -114,6 +126,14 @@ } \ } +#define VHT_GROUP_SHIFT(_streams, _sgi, _bw) \ + GROUP_SHIFT(MCS_DURATION(_streams, _sgi, \ + BW2VBPS(_bw, 117, 54, 26))) + +#define VHT_GROUP(_streams, _sgi, _bw) \ + __VHT_GROUP(_streams, _sgi, _bw, \ + VHT_GROUP_SHIFT(_streams, _sgi, _bw)) + #define CCK_DURATION(_bitrate, _short, _len) \ (1000 * (10 /* SIFS */ + \ (_short ? 72 + 24 : 144 + 48) + \ @@ -129,7 +149,7 @@ CCK_ACK_DURATION(55, _short) >> _s, \ CCK_ACK_DURATION(110, _short) >> _s -#define CCK_GROUP(_s) \ +#define __CCK_GROUP(_s) \ [MINSTREL_CCK_GROUP] = { \ .streams = 1, \ .flags = 0, \ @@ -140,6 +160,12 @@ } \ } +#define CCK_GROUP_SHIFT \ + GROUP_SHIFT(CCK_ACK_DURATION(10, false)) + +#define CCK_GROUP __CCK_GROUP(CCK_GROUP_SHIFT) + + static bool minstrel_vht_only = true; module_param(minstrel_vht_only, bool, 0644); MODULE_PARM_DESC(minstrel_vht_only, @@ -154,47 +180,57 @@ MODULE_PARM_DESC(minstrel_vht_only, * BW -> SGI -> #streams */ const struct mcs_group minstrel_mcs_groups[] = { - MCS_GROUP(1, 0, BW_20, 5), - MCS_GROUP(2, 0, BW_20, 4), - MCS_GROUP(3, 0, BW_20, 4), - - MCS_GROUP(1, 1, BW_20, 5), - MCS_GROUP(2, 1, BW_20, 4), - MCS_GROUP(3, 1, BW_20, 4), - - MCS_GROUP(1, 0, BW_40, 4), - MCS_GROUP(2, 0, BW_40, 4), - MCS_GROUP(3, 0, BW_40, 4), - - MCS_GROUP(1, 1, BW_40, 4), - MCS_GROUP(2, 1, BW_40, 4), - MCS_GROUP(3, 1, BW_40, 4), - - CCK_GROUP(8), - - VHT_GROUP(1, 0, BW_20, 5), - VHT_GROUP(2, 0, BW_20, 4), - VHT_GROUP(3, 0, BW_20, 4), - - VHT_GROUP(1, 1, BW_20, 5), - VHT_GROUP(2, 1, BW_20, 4), - VHT_GROUP(3, 1, BW_20, 4), - - VHT_GROUP(1, 0, BW_40, 4), - VHT_GROUP(2, 0, BW_40, 4), - VHT_GROUP(3, 0, BW_40, 4), - - VHT_GROUP(1, 1, BW_40, 4), - VHT_GROUP(2, 1, BW_40, 4), - VHT_GROUP(3, 1, BW_40, 4), - - VHT_GROUP(1, 0, BW_80, 4), - VHT_GROUP(2, 0, BW_80, 4), - VHT_GROUP(3, 0, BW_80, 4), - - VHT_GROUP(1, 1, BW_80, 4), - VHT_GROUP(2, 1, BW_80, 4), - VHT_GROUP(3, 1, BW_80, 4), + MCS_GROUP(1, 0, BW_20), + MCS_GROUP(2, 0, BW_20), + MCS_GROUP(3, 0, BW_20), + MCS_GROUP(4, 0, BW_20), + + MCS_GROUP(1, 1, BW_20), + MCS_GROUP(2, 1, BW_20), + MCS_GROUP(3, 1, BW_20), + MCS_GROUP(4, 1, BW_20), + + MCS_GROUP(1, 0, BW_40), + MCS_GROUP(2, 0, BW_40), + MCS_GROUP(3, 0, BW_40), + MCS_GROUP(4, 0, BW_40), + + MCS_GROUP(1, 1, BW_40), + MCS_GROUP(2, 1, BW_40), + MCS_GROUP(3, 1, BW_40), + MCS_GROUP(4, 1, BW_40), + + CCK_GROUP, + + VHT_GROUP(1, 0, BW_20), + VHT_GROUP(2, 0, BW_20), + VHT_GROUP(3, 0, BW_20), + VHT_GROUP(4, 0, BW_20), + + VHT_GROUP(1, 1, BW_20), + VHT_GROUP(2, 1, BW_20), + VHT_GROUP(3, 1, BW_20), + VHT_GROUP(4, 1, BW_20), + + VHT_GROUP(1, 0, BW_40), + VHT_GROUP(2, 0, BW_40), + VHT_GROUP(3, 0, BW_40), + VHT_GROUP(4, 0, BW_40), + + VHT_GROUP(1, 1, BW_40), + VHT_GROUP(2, 1, BW_40), + VHT_GROUP(3, 1, BW_40), + VHT_GROUP(4, 1, BW_40), + + VHT_GROUP(1, 0, BW_80), + VHT_GROUP(2, 0, BW_80), + VHT_GROUP(3, 0, BW_80), + VHT_GROUP(4, 0, BW_80), + + VHT_GROUP(1, 1, BW_80), + VHT_GROUP(2, 1, BW_80), + VHT_GROUP(3, 1, BW_80), + VHT_GROUP(4, 1, BW_80), }; static u8 sample_table[SAMPLE_COLUMNS][MCS_GROUP_RATES] __read_mostly; diff --git a/net/mac80211/rc80211_minstrel_ht.h b/net/mac80211/rc80211_minstrel_ht.h index 26b7a3244b478ec329dafe198454b66c9dcbdd2f..f762e5ba7c2ed65095f0f05c2ffbf495b3dc4ff4 100644 --- a/net/mac80211/rc80211_minstrel_ht.h +++ b/net/mac80211/rc80211_minstrel_ht.h @@ -13,7 +13,7 @@ * The number of streams can be changed to 2 to reduce code * size and memory footprint. */ -#define MINSTREL_MAX_STREAMS 3 +#define MINSTREL_MAX_STREAMS 4 #define MINSTREL_HT_STREAM_GROUPS 4 /* BW(=2) * SGI(=2) */ #define MINSTREL_VHT_STREAM_GROUPS 6 /* BW(=3) * SGI(=2) */ diff --git a/net/mac80211/rx.c b/net/mac80211/rx.c index bf0b187f994e9c56e191d2045f405cb6e6bac336..25577ede2986ec5616a94501145a9734ec402521 100644 --- a/net/mac80211/rx.c +++ b/net/mac80211/rx.c @@ -1005,23 +1005,43 @@ static int ieee80211_get_mmie_keyidx(struct sk_buff *skb) return -1; } -static int ieee80211_get_cs_keyid(const struct ieee80211_cipher_scheme *cs, - struct sk_buff *skb) +static int ieee80211_get_keyid(struct sk_buff *skb, + const struct ieee80211_cipher_scheme *cs) { struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)skb->data; __le16 fc; int hdrlen; + int minlen; + u8 key_idx_off; + u8 key_idx_shift; u8 keyid; fc = hdr->frame_control; hdrlen = ieee80211_hdrlen(fc); - if (skb->len < hdrlen + cs->hdr_len) + if (cs) { + minlen = hdrlen + cs->hdr_len; + key_idx_off = hdrlen + cs->key_idx_off; + key_idx_shift = cs->key_idx_shift; + } else { + /* WEP, TKIP, CCMP and GCMP */ + minlen = hdrlen + IEEE80211_WEP_IV_LEN; + key_idx_off = hdrlen + 3; + key_idx_shift = 6; + } + + if (unlikely(skb->len < minlen)) return -EINVAL; - skb_copy_bits(skb, hdrlen + cs->key_idx_off, &keyid, 1); - keyid &= cs->key_idx_mask; - keyid >>= cs->key_idx_shift; + skb_copy_bits(skb, key_idx_off, &keyid, 1); + + if (cs) + keyid &= cs->key_idx_mask; + keyid >>= key_idx_shift; + + /* cs could use more than the usual two bits for the keyid */ + if (unlikely(keyid >= NUM_DEFAULT_KEYS)) + return -EINVAL; return keyid; } @@ -1860,9 +1880,9 @@ ieee80211_rx_h_decrypt(struct ieee80211_rx_data *rx) struct ieee80211_rx_status *status = IEEE80211_SKB_RXCB(skb); struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)skb->data; int keyidx; - int hdrlen; ieee80211_rx_result result = RX_DROP_UNUSABLE; struct ieee80211_key *sta_ptk = NULL; + struct ieee80211_key *ptk_idx = NULL; int mmie_keyidx = -1; __le16 fc; const struct ieee80211_cipher_scheme *cs = NULL; @@ -1900,21 +1920,24 @@ ieee80211_rx_h_decrypt(struct ieee80211_rx_data *rx) if (rx->sta) { int keyid = rx->sta->ptk_idx; + sta_ptk = rcu_dereference(rx->sta->ptk[keyid]); - if (ieee80211_has_protected(fc) && rx->sta->cipher_scheme) { + if (ieee80211_has_protected(fc)) { cs = rx->sta->cipher_scheme; - keyid = ieee80211_get_cs_keyid(cs, rx->skb); + keyid = ieee80211_get_keyid(rx->skb, cs); + if (unlikely(keyid < 0)) return RX_DROP_UNUSABLE; + + ptk_idx = rcu_dereference(rx->sta->ptk[keyid]); } - sta_ptk = rcu_dereference(rx->sta->ptk[keyid]); } if (!ieee80211_has_protected(fc)) mmie_keyidx = ieee80211_get_mmie_keyidx(rx->skb); if (!is_multicast_ether_addr(hdr->addr1) && sta_ptk) { - rx->key = sta_ptk; + rx->key = ptk_idx ? ptk_idx : sta_ptk; if ((status->flag & RX_FLAG_DECRYPTED) && (status->flag & RX_FLAG_IV_STRIPPED)) return RX_CONTINUE; @@ -1974,8 +1997,6 @@ ieee80211_rx_h_decrypt(struct ieee80211_rx_data *rx) } return RX_CONTINUE; } else { - u8 keyid; - /* * The device doesn't give us the IV so we won't be * able to look up the key. That's ok though, we @@ -1989,23 +2010,10 @@ ieee80211_rx_h_decrypt(struct ieee80211_rx_data *rx) (status->flag & RX_FLAG_IV_STRIPPED)) return RX_CONTINUE; - hdrlen = ieee80211_hdrlen(fc); - - if (cs) { - keyidx = ieee80211_get_cs_keyid(cs, rx->skb); + keyidx = ieee80211_get_keyid(rx->skb, cs); - if (unlikely(keyidx < 0)) - return RX_DROP_UNUSABLE; - } else { - if (rx->skb->len < 8 + hdrlen) - return RX_DROP_UNUSABLE; /* TODO: count this? */ - /* - * no need to call ieee80211_wep_get_keyidx, - * it verifies a bunch of things we've done already - */ - skb_copy_bits(rx->skb, hdrlen + 3, &keyid, 1); - keyidx = keyid >> 6; - } + if (unlikely(keyidx < 0)) + return RX_DROP_UNUSABLE; /* check per-station GTK first, if multicast packet */ if (is_multicast_ether_addr(hdr->addr1) && rx->sta) @@ -4050,12 +4058,8 @@ void ieee80211_check_fast_rx(struct sta_info *sta) case WLAN_CIPHER_SUITE_GCMP_256: break; default: - /* we also don't want to deal with WEP or cipher scheme - * since those require looking up the key idx in the - * frame, rather than assuming the PTK is used - * (we need to revisit this once we implement the real - * PTK index, which is now valid in the spec, but we - * haven't implemented that part yet) + /* We also don't want to deal with + * WEP or cipher scheme. */ goto clear_rcu; } diff --git a/net/mac80211/sta_info.c b/net/mac80211/sta_info.c index 11f058987a54393edd87b7e04f88c74084bd70de..a4932ee3595c2f0e03248ab59b13f576a6783faf 100644 --- a/net/mac80211/sta_info.c +++ b/net/mac80211/sta_info.c @@ -347,6 +347,15 @@ struct sta_info *sta_info_alloc(struct ieee80211_sub_if_data *sdata, sta->sta.max_rx_aggregation_subframes = local->hw.max_rx_aggregation_subframes; + /* Extended Key ID needs to install keys for keyid 0 and 1 Rx-only. + * The Tx path starts to use a key as soon as the key slot ptk_idx + * references to is not NULL. To not use the initial Rx-only key + * prematurely for Tx initialize ptk_idx to an impossible PTK keyid + * which always will refer to a NULL key. + */ + BUILD_BUG_ON(ARRAY_SIZE(sta->ptk) <= INVALID_PTK_KEYIDX); + sta->ptk_idx = INVALID_PTK_KEYIDX; + sta->local = local; sta->sdata = sdata; sta->rx_stats.last_rx = jiffies; @@ -2373,6 +2382,12 @@ void sta_set_sinfo(struct sta_info *sta, struct station_info *sinfo, sinfo->filled |= BIT_ULL(NL80211_STA_INFO_ACK_SIGNAL_AVG); } + + if (ieee80211_vif_is_mesh(&sdata->vif)) { + sinfo->filled |= BIT_ULL(NL80211_STA_INFO_AIRTIME_LINK_METRIC); + sinfo->airtime_link_metric = + airtime_link_metric_get(local, sta); + } } u32 sta_get_expected_throughput(struct sta_info *sta) diff --git a/net/mac80211/tdls.c b/net/mac80211/tdls.c index d30690d79a586b075d3eea2b23749154148a7c38..24c37f91ca461ec932cb9c6806aec25fedb8405b 100644 --- a/net/mac80211/tdls.c +++ b/net/mac80211/tdls.c @@ -1056,7 +1056,7 @@ ieee80211_tdls_prep_mgmt_packet(struct wiphy *wiphy, struct net_device *dev, /* disable bottom halves when entering the Tx path */ local_bh_disable(); - __ieee80211_subif_start_xmit(skb, dev, flags); + __ieee80211_subif_start_xmit(skb, dev, flags, 0); local_bh_enable(); return ret; diff --git a/net/mac80211/trace.h b/net/mac80211/trace.h index 8ba70d26b82e4a6ece6b81662f7e748ae365648a..3bb4459b52c780ef7cc5a645661337f4c78417a0 100644 --- a/net/mac80211/trace.h +++ b/net/mac80211/trace.h @@ -828,6 +828,36 @@ TRACE_EVENT(drv_sta_state, ) ); +TRACE_EVENT(drv_sta_set_txpwr, + TP_PROTO(struct ieee80211_local *local, + struct ieee80211_sub_if_data *sdata, + struct ieee80211_sta *sta), + + TP_ARGS(local, sdata, sta), + + TP_STRUCT__entry( + LOCAL_ENTRY + VIF_ENTRY + STA_ENTRY + __field(s16, txpwr) + __field(u8, type) + ), + + TP_fast_assign( + LOCAL_ASSIGN; + VIF_ASSIGN; + STA_ASSIGN; + __entry->txpwr = sta->txpwr.power; + __entry->type = sta->txpwr.type; + ), + + TP_printk( + LOCAL_PR_FMT VIF_PR_FMT STA_PR_FMT " txpwr: %d type %d", + LOCAL_PR_ARG, VIF_PR_ARG, STA_PR_ARG, + __entry->txpwr, __entry->type + ) +); + TRACE_EVENT(drv_sta_rc_update, TP_PROTO(struct ieee80211_local *local, struct ieee80211_sub_if_data *sdata, diff --git a/net/mac80211/tx.c b/net/mac80211/tx.c index 2e816dd67be72d161bf1959554d293f2f6725673..dd220b977025cb8149fb91af86a838e376037970 100644 --- a/net/mac80211/tx.c +++ b/net/mac80211/tx.c @@ -1399,11 +1399,15 @@ static void ieee80211_txq_enqueue(struct ieee80211_local *local, { struct fq *fq = &local->fq; struct fq_tin *tin = &txqi->tin; + u32 flow_idx = fq_flow_idx(fq, skb); ieee80211_set_skb_enqueue_time(skb); - fq_tin_enqueue(fq, tin, skb, + + spin_lock_bh(&fq->lock); + fq_tin_enqueue(fq, tin, flow_idx, skb, fq_skb_free_func, fq_flow_get_default_func); + spin_unlock_bh(&fq->lock); } static bool fq_vlan_filter_func(struct fq *fq, struct fq_tin *tin, @@ -1590,7 +1594,6 @@ static bool ieee80211_queue_skb(struct ieee80211_local *local, struct sta_info *sta, struct sk_buff *skb) { - struct fq *fq = &local->fq; struct ieee80211_vif *vif; struct txq_info *txqi; @@ -1608,9 +1611,7 @@ static bool ieee80211_queue_skb(struct ieee80211_local *local, if (!txqi) return false; - spin_lock_bh(&fq->lock); ieee80211_txq_enqueue(local, txqi, skb); - spin_unlock_bh(&fq->lock); schedule_and_wake_txq(local, txqi); @@ -2431,6 +2432,7 @@ static int ieee80211_lookup_ra_sta(struct ieee80211_sub_if_data *sdata, * @sdata: virtual interface to build the header for * @skb: the skb to build the header in * @info_flags: skb flags to set + * @ctrl_flags: info control flags to set * * This function takes the skb with 802.3 header and reformats the header to * the appropriate IEEE 802.11 header based on which interface the packet is @@ -2446,7 +2448,7 @@ static int ieee80211_lookup_ra_sta(struct ieee80211_sub_if_data *sdata, */ static struct sk_buff *ieee80211_build_hdr(struct ieee80211_sub_if_data *sdata, struct sk_buff *skb, u32 info_flags, - struct sta_info *sta) + struct sta_info *sta, u32 ctrl_flags) { struct ieee80211_local *local = sdata->local; struct ieee80211_tx_info *info; @@ -2470,6 +2472,11 @@ static struct sk_buff *ieee80211_build_hdr(struct ieee80211_sub_if_data *sdata, if (IS_ERR(sta)) sta = NULL; +#ifdef CONFIG_MAC80211_DEBUGFS + if (local->force_tx_status) + info_flags |= IEEE80211_TX_CTL_REQ_TX_STATUS; +#endif + /* convert Ethernet header to proper 802.11 header (based on * operation mode) */ ethertype = (skb->data[12] << 8) | skb->data[13]; @@ -2600,6 +2607,13 @@ static struct sk_buff *ieee80211_build_hdr(struct ieee80211_sub_if_data *sdata, goto free; } band = chanctx_conf->def.chan->band; + + /* For injected frames, fill RA right away as nexthop lookup + * will be skipped. + */ + if ((ctrl_flags & IEEE80211_TX_CTRL_SKIP_MPATH_LOOKUP) && + is_zero_ether_addr(hdr.addr1)) + memcpy(hdr.addr1, skb->data, ETH_ALEN); break; #endif case NL80211_IFTYPE_STATION: @@ -2818,6 +2832,7 @@ static struct sk_buff *ieee80211_build_hdr(struct ieee80211_sub_if_data *sdata, info->flags = info_flags; info->ack_frame_id = info_id; info->band = band; + info->control.flags = ctrl_flags; return skb; free: @@ -3000,23 +3015,15 @@ void ieee80211_check_fast_xmit(struct sta_info *sta) switch (build.key->conf.cipher) { case WLAN_CIPHER_SUITE_CCMP: case WLAN_CIPHER_SUITE_CCMP_256: - /* add fixed key ID */ - if (gen_iv) { - (build.hdr + build.hdr_len)[3] = - 0x20 | (build.key->conf.keyidx << 6); + if (gen_iv) build.pn_offs = build.hdr_len; - } if (gen_iv || iv_spc) build.hdr_len += IEEE80211_CCMP_HDR_LEN; break; case WLAN_CIPHER_SUITE_GCMP: case WLAN_CIPHER_SUITE_GCMP_256: - /* add fixed key ID */ - if (gen_iv) { - (build.hdr + build.hdr_len)[3] = - 0x20 | (build.key->conf.keyidx << 6); + if (gen_iv) build.pn_offs = build.hdr_len; - } if (gen_iv || iv_spc) build.hdr_len += IEEE80211_GCMP_HDR_LEN; break; @@ -3222,6 +3229,7 @@ static bool ieee80211_amsdu_aggregate(struct ieee80211_sub_if_data *sdata, int max_frags = local->hw.max_tx_fragments; int max_amsdu_len = sta->sta.max_amsdu_len; int orig_truesize; + u32 flow_idx; __be16 len; void *data; bool ret = false; @@ -3250,6 +3258,8 @@ static bool ieee80211_amsdu_aggregate(struct ieee80211_sub_if_data *sdata, max_amsdu_len = min_t(int, max_amsdu_len, sta->sta.max_tid_amsdu_len[tid]); + flow_idx = fq_flow_idx(fq, skb); + spin_lock_bh(&fq->lock); /* TODO: Ideally aggregation should be done on dequeue to remain @@ -3257,7 +3267,8 @@ static bool ieee80211_amsdu_aggregate(struct ieee80211_sub_if_data *sdata, */ tin = &txqi->tin; - flow = fq_flow_classify(fq, tin, skb, fq_flow_get_default_func); + flow = fq_flow_classify(fq, tin, flow_idx, skb, + fq_flow_get_default_func); head = skb_peek_tail(&flow->queue); if (!head || skb_is_gso(head)) goto out; @@ -3386,6 +3397,7 @@ static void ieee80211_xmit_fast_finish(struct ieee80211_sub_if_data *sdata, pn = atomic64_inc_return(&key->conf.tx_pn); crypto_hdr[0] = pn; crypto_hdr[1] = pn >> 8; + crypto_hdr[3] = 0x20 | (key->conf.keyidx << 6); crypto_hdr[4] = pn >> 16; crypto_hdr[5] = pn >> 24; crypto_hdr[6] = pn >> 32; @@ -3478,6 +3490,11 @@ static bool ieee80211_xmit_fast(struct ieee80211_sub_if_data *sdata, (tid_tx ? IEEE80211_TX_CTL_AMPDU : 0); info->control.flags = IEEE80211_TX_CTRL_FAST_XMIT; +#ifdef CONFIG_MAC80211_DEBUGFS + if (local->force_tx_status) + info->flags |= IEEE80211_TX_CTL_REQ_TX_STATUS; +#endif + if (hdr->frame_control & cpu_to_le16(IEEE80211_STYPE_QOS_DATA)) { tid = skb->priority & IEEE80211_QOS_CTL_TAG1D_MASK; *ieee80211_get_qos_ctl(hdr) = tid; @@ -3533,6 +3550,7 @@ struct sk_buff *ieee80211_tx_dequeue(struct ieee80211_hw *hw, ieee80211_tx_result r; struct ieee80211_vif *vif = txq->vif; +begin: spin_lock_bh(&fq->lock); if (test_bit(IEEE80211_TXQ_STOP, &txqi->flags) || @@ -3549,11 +3567,12 @@ struct sk_buff *ieee80211_tx_dequeue(struct ieee80211_hw *hw, if (skb) goto out; -begin: skb = fq_tin_dequeue(fq, tin, fq_tin_dequeue_func); if (!skb) goto out; + spin_unlock_bh(&fq->lock); + hdr = (struct ieee80211_hdr *)skb->data; info = IEEE80211_SKB_CB(skb); @@ -3598,8 +3617,11 @@ begin: skb = __skb_dequeue(&tx.skbs); - if (!skb_queue_empty(&tx.skbs)) + if (!skb_queue_empty(&tx.skbs)) { + spin_lock_bh(&fq->lock); skb_queue_splice_tail(&tx.skbs, &txqi->frags); + spin_unlock_bh(&fq->lock); + } } if (skb_has_frag_list(skb) && @@ -3638,6 +3660,7 @@ begin: } IEEE80211_SKB_CB(skb)->control.vif = vif; + return skb; out: spin_unlock_bh(&fq->lock); @@ -3783,9 +3806,11 @@ EXPORT_SYMBOL(ieee80211_txq_schedule_start); void __ieee80211_subif_start_xmit(struct sk_buff *skb, struct net_device *dev, - u32 info_flags) + u32 info_flags, + u32 ctrl_flags) { struct ieee80211_sub_if_data *sdata = IEEE80211_DEV_TO_SUB_IF(dev); + struct ieee80211_local *local = sdata->local; struct sta_info *sta; struct sk_buff *next; @@ -3799,7 +3824,15 @@ void __ieee80211_subif_start_xmit(struct sk_buff *skb, if (ieee80211_lookup_ra_sta(sdata, skb, &sta)) goto out_free; - if (!IS_ERR_OR_NULL(sta)) { + if (IS_ERR(sta)) + sta = NULL; + + if (local->ops->wake_tx_queue) { + u16 queue = __ieee80211_select_queue(sdata, sta, skb); + skb_set_queue_mapping(skb, queue); + } + + if (sta) { struct ieee80211_fast_tx *fast_tx; sk_pacing_shift_update(skb->sk, sdata->local->hw.tx_sk_pacing_shift); @@ -3848,7 +3881,8 @@ void __ieee80211_subif_start_xmit(struct sk_buff *skb, skb->prev = NULL; skb->next = NULL; - skb = ieee80211_build_hdr(sdata, skb, info_flags, sta); + skb = ieee80211_build_hdr(sdata, skb, info_flags, + sta, ctrl_flags); if (IS_ERR(skb)) goto out; @@ -3988,9 +4022,9 @@ netdev_tx_t ieee80211_subif_start_xmit(struct sk_buff *skb, __skb_queue_head_init(&queue); ieee80211_convert_to_unicast(skb, dev, &queue); while ((skb = __skb_dequeue(&queue))) - __ieee80211_subif_start_xmit(skb, dev, 0); + __ieee80211_subif_start_xmit(skb, dev, 0, 0); } else { - __ieee80211_subif_start_xmit(skb, dev, 0); + __ieee80211_subif_start_xmit(skb, dev, 0, 0); } return NETDEV_TX_OK; @@ -4015,7 +4049,7 @@ ieee80211_build_data_template(struct ieee80211_sub_if_data *sdata, goto out; } - skb = ieee80211_build_hdr(sdata, skb, info_flags, sta); + skb = ieee80211_build_hdr(sdata, skb, info_flags, sta, 0); if (IS_ERR(skb)) goto out; @@ -5052,7 +5086,36 @@ int ieee80211_tx_control_port(struct wiphy *wiphy, struct net_device *dev, skb_reset_mac_header(skb); local_bh_disable(); - __ieee80211_subif_start_xmit(skb, skb->dev, flags); + __ieee80211_subif_start_xmit(skb, skb->dev, flags, 0); + local_bh_enable(); + + return 0; +} + +int ieee80211_probe_mesh_link(struct wiphy *wiphy, struct net_device *dev, + const u8 *buf, size_t len) +{ + struct ieee80211_sub_if_data *sdata = IEEE80211_DEV_TO_SUB_IF(dev); + struct ieee80211_local *local = sdata->local; + struct sk_buff *skb; + + skb = dev_alloc_skb(local->hw.extra_tx_headroom + len + + 30 + /* header size */ + 18); /* 11s header size */ + if (!skb) + return -ENOMEM; + + skb_reserve(skb, local->hw.extra_tx_headroom); + skb_put_data(skb, buf, len); + + skb->dev = dev; + skb->protocol = htons(ETH_P_802_3); + skb_reset_network_header(skb); + skb_reset_mac_header(skb); + + local_bh_disable(); + __ieee80211_subif_start_xmit(skb, skb->dev, 0, + IEEE80211_TX_CTRL_SKIP_MPATH_LOOKUP); local_bh_enable(); return 0; diff --git a/net/mac80211/util.c b/net/mac80211/util.c index 4c16559725651df36ef9335ded5954fcdfd35650..cba4633cd6cfd02a838450bad524465579b18f8d 100644 --- a/net/mac80211/util.c +++ b/net/mac80211/util.c @@ -894,10 +894,10 @@ EXPORT_SYMBOL(ieee80211_queue_delayed_work); static u32 _ieee802_11_parse_elems_crc(const u8 *start, size_t len, bool action, struct ieee802_11_elems *elems, - u64 filter, u32 crc, u8 *transmitter_bssid, - u8 *bss_bssid) + u64 filter, u32 crc, + const struct element *check_inherit) { - const struct element *elem, *sub; + const struct element *elem; bool calc_crc = filter != 0; DECLARE_BITMAP(seen_elems, 256); const u8 *ie; @@ -910,6 +910,11 @@ _ieee802_11_parse_elems_crc(const u8 *start, size_t len, bool action, u8 elen = elem->datalen; const u8 *pos = elem->data; + if (check_inherit && + !cfg80211_is_element_inherited(elem, + check_inherit)) + continue; + switch (id) { case WLAN_EID_SSID: case WLAN_EID_SUPP_RATES: @@ -1208,57 +1213,6 @@ _ieee802_11_parse_elems_crc(const u8 *start, size_t len, bool action, if (elen >= sizeof(*elems->max_idle_period_ie)) elems->max_idle_period_ie = (void *)pos; break; - case WLAN_EID_MULTIPLE_BSSID: - if (!bss_bssid || !transmitter_bssid || elen < 4) - break; - - elems->max_bssid_indicator = pos[0]; - - for_each_element(sub, pos + 1, elen - 1) { - u8 sub_len = sub->datalen; - u8 new_bssid[ETH_ALEN]; - const u8 *index; - - /* - * we only expect the "non-transmitted BSSID - * profile" subelement (subelement id 0) - */ - if (sub->id != 0 || sub->datalen < 4) { - /* not a valid BSS profile */ - continue; - } - - if (sub->data[0] != WLAN_EID_NON_TX_BSSID_CAP || - sub->data[1] != 2) { - /* The first element of the - * Nontransmitted BSSID Profile is not - * the Nontransmitted BSSID Capability - * element. - */ - continue; - } - - /* found a Nontransmitted BSSID Profile */ - index = cfg80211_find_ie(WLAN_EID_MULTI_BSSID_IDX, - sub->data, sub_len); - if (!index || index[1] < 1 || index[2] == 0) { - /* Invalid MBSSID Index element */ - continue; - } - - cfg80211_gen_new_bssid(transmitter_bssid, - pos[0], - index[2], - new_bssid); - if (ether_addr_equal(new_bssid, bss_bssid)) { - elems->nontransmitted_bssid_profile = - (void *)sub; - elems->bssid_index_len = index[1]; - elems->bssid_index = (void *)&index[2]; - break; - } - } - break; case WLAN_EID_EXTENSION: if (pos[0] == WLAN_EID_EXT_HE_MU_EDCA && elen >= (sizeof(*elems->mu_edca_param_set) + 1)) { @@ -1300,26 +1254,108 @@ _ieee802_11_parse_elems_crc(const u8 *start, size_t len, bool action, return crc; } +static size_t ieee802_11_find_bssid_profile(const u8 *start, size_t len, + struct ieee802_11_elems *elems, + u8 *transmitter_bssid, + u8 *bss_bssid, + u8 *nontransmitted_profile) +{ + const struct element *elem, *sub; + size_t profile_len = 0; + bool found = false; + + if (!bss_bssid || !transmitter_bssid) + return profile_len; + + for_each_element_id(elem, WLAN_EID_MULTIPLE_BSSID, start, len) { + if (elem->datalen < 2) + continue; + + for_each_element(sub, elem->data + 1, elem->datalen - 1) { + u8 new_bssid[ETH_ALEN]; + const u8 *index; + + if (sub->id != 0 || sub->datalen < 4) { + /* not a valid BSS profile */ + continue; + } + + if (sub->data[0] != WLAN_EID_NON_TX_BSSID_CAP || + sub->data[1] != 2) { + /* The first element of the + * Nontransmitted BSSID Profile is not + * the Nontransmitted BSSID Capability + * element. + */ + continue; + } + + memset(nontransmitted_profile, 0, len); + profile_len = cfg80211_merge_profile(start, len, + elem, + sub, + nontransmitted_profile, + len); + + /* found a Nontransmitted BSSID Profile */ + index = cfg80211_find_ie(WLAN_EID_MULTI_BSSID_IDX, + nontransmitted_profile, + profile_len); + if (!index || index[1] < 1 || index[2] == 0) { + /* Invalid MBSSID Index element */ + continue; + } + + cfg80211_gen_new_bssid(transmitter_bssid, + elem->data[0], + index[2], + new_bssid); + if (ether_addr_equal(new_bssid, bss_bssid)) { + found = true; + elems->bssid_index_len = index[1]; + elems->bssid_index = (void *)&index[2]; + break; + } + } + } + + return found ? profile_len : 0; +} + u32 ieee802_11_parse_elems_crc(const u8 *start, size_t len, bool action, struct ieee802_11_elems *elems, u64 filter, u32 crc, u8 *transmitter_bssid, u8 *bss_bssid) { + const struct element *non_inherit = NULL; + u8 *nontransmitted_profile; + int nontransmitted_profile_len = 0; + memset(elems, 0, sizeof(*elems)); elems->ie_start = start; elems->total_len = len; + nontransmitted_profile = kmalloc(len, GFP_ATOMIC); + if (nontransmitted_profile) { + nontransmitted_profile_len = + ieee802_11_find_bssid_profile(start, len, elems, + transmitter_bssid, + bss_bssid, + nontransmitted_profile); + non_inherit = + cfg80211_find_ext_elem(WLAN_EID_EXT_NON_INHERITANCE, + nontransmitted_profile, + nontransmitted_profile_len); + } + crc = _ieee802_11_parse_elems_crc(start, len, action, elems, filter, - crc, transmitter_bssid, bss_bssid); + crc, non_inherit); /* Override with nontransmitted profile, if found */ - if (transmitter_bssid && elems->nontransmitted_bssid_profile) { - const u8 *profile = elems->nontransmitted_bssid_profile; - - _ieee802_11_parse_elems_crc(&profile[2], profile[1], - action, elems, 0, 0, - transmitter_bssid, bss_bssid); - } + if (nontransmitted_profile_len) + _ieee802_11_parse_elems_crc(nontransmitted_profile, + nontransmitted_profile_len, + action, elems, 0, 0, NULL); if (elems->tim && !elems->parse_error) { const struct ieee80211_tim_ie *tim_ie = elems->tim; @@ -1339,6 +1375,8 @@ u32 ieee802_11_parse_elems_crc(const u8 *start, size_t len, bool action, offsetofend(struct ieee80211_bssid_index, dtim_count)) elems->dtim_count = elems->bssid_index->dtim_count; + kfree(nontransmitted_profile); + return crc; } diff --git a/net/mac80211/wme.c b/net/mac80211/wme.c index 5f7c96368b11915f1a1c6037267b7b03b9957aa8..6a3187883c4b74ea4266cf15dd45811d208eb246 100644 --- a/net/mac80211/wme.c +++ b/net/mac80211/wme.c @@ -141,6 +141,42 @@ u16 ieee80211_select_queue_80211(struct ieee80211_sub_if_data *sdata, return ieee80211_downgrade_queue(sdata, NULL, skb); } +u16 __ieee80211_select_queue(struct ieee80211_sub_if_data *sdata, + struct sta_info *sta, struct sk_buff *skb) +{ + struct mac80211_qos_map *qos_map; + bool qos; + + /* all mesh/ocb stations are required to support WME */ + if (sdata->vif.type == NL80211_IFTYPE_MESH_POINT || + sdata->vif.type == NL80211_IFTYPE_OCB) + qos = true; + else if (sta) + qos = sta->sta.wme; + else + qos = false; + + if (!qos) { + skb->priority = 0; /* required for correct WPA/11i MIC */ + return IEEE80211_AC_BE; + } + + if (skb->protocol == sdata->control_port_protocol) { + skb->priority = 7; + goto downgrade; + } + + /* use the data classifier to determine what 802.1d tag the + * data frame has */ + qos_map = rcu_dereference(sdata->qos_map); + skb->priority = cfg80211_classify8021d(skb, qos_map ? + &qos_map->qos_map : NULL); + + downgrade: + return ieee80211_downgrade_queue(sdata, sta, skb); +} + + /* Indicate which queue to use. */ u16 ieee80211_select_queue(struct ieee80211_sub_if_data *sdata, struct sk_buff *skb) @@ -148,10 +184,12 @@ u16 ieee80211_select_queue(struct ieee80211_sub_if_data *sdata, struct ieee80211_local *local = sdata->local; struct sta_info *sta = NULL; const u8 *ra = NULL; - bool qos = false; - struct mac80211_qos_map *qos_map; u16 ret; + /* when using iTXQ, we can do this later */ + if (local->ops->wake_tx_queue) + return 0; + if (local->hw.queues < IEEE80211_NUM_ACS || skb->len < 6) { skb->priority = 0; /* required for correct WPA/11i MIC */ return 0; @@ -161,10 +199,8 @@ u16 ieee80211_select_queue(struct ieee80211_sub_if_data *sdata, switch (sdata->vif.type) { case NL80211_IFTYPE_AP_VLAN: sta = rcu_dereference(sdata->u.vlan.sta); - if (sta) { - qos = sta->sta.wme; + if (sta) break; - } /* fall through */ case NL80211_IFTYPE_AP: ra = skb->data; @@ -172,56 +208,26 @@ u16 ieee80211_select_queue(struct ieee80211_sub_if_data *sdata, case NL80211_IFTYPE_WDS: ra = sdata->u.wds.remote_addr; break; -#ifdef CONFIG_MAC80211_MESH - case NL80211_IFTYPE_MESH_POINT: - qos = true; - break; -#endif case NL80211_IFTYPE_STATION: /* might be a TDLS station */ sta = sta_info_get(sdata, skb->data); if (sta) - qos = sta->sta.wme; + break; ra = sdata->u.mgd.bssid; break; case NL80211_IFTYPE_ADHOC: ra = skb->data; break; - case NL80211_IFTYPE_OCB: - /* all stations are required to support WME */ - qos = true; - break; default: break; } - if (!sta && ra && !is_multicast_ether_addr(ra)) { + if (!sta && ra && !is_multicast_ether_addr(ra)) sta = sta_info_get(sdata, ra); - if (sta) - qos = sta->sta.wme; - } - if (!qos) { - skb->priority = 0; /* required for correct WPA/11i MIC */ - ret = IEEE80211_AC_BE; - goto out; - } + ret = __ieee80211_select_queue(sdata, sta, skb); - if (skb->protocol == sdata->control_port_protocol) { - skb->priority = 7; - goto downgrade; - } - - /* use the data classifier to determine what 802.1d tag the - * data frame has */ - qos_map = rcu_dereference(sdata->qos_map); - skb->priority = cfg80211_classify8021d(skb, qos_map ? - &qos_map->qos_map : NULL); - - downgrade: - ret = ieee80211_downgrade_queue(sdata, sta, skb); - out: rcu_read_unlock(); return ret; } diff --git a/net/mac80211/wme.h b/net/mac80211/wme.h index 80151edc51955d913ef2b00608f29960557d73c7..b1b1439cb91b5c9c78c9cfa882e21b6de38df2e9 100644 --- a/net/mac80211/wme.h +++ b/net/mac80211/wme.h @@ -16,6 +16,8 @@ u16 ieee80211_select_queue_80211(struct ieee80211_sub_if_data *sdata, struct sk_buff *skb, struct ieee80211_hdr *hdr); +u16 __ieee80211_select_queue(struct ieee80211_sub_if_data *sdata, + struct sta_info *sta, struct sk_buff *skb); u16 ieee80211_select_queue(struct ieee80211_sub_if_data *sdata, struct sk_buff *skb); void ieee80211_set_qos_hdr(struct ieee80211_sub_if_data *sdata, diff --git a/net/mpls/af_mpls.c b/net/mpls/af_mpls.c index f7c544592ec872e9ead8db7e941f5e46861c3baf..baa098291fb02e8e11b4c7240c3967763d28d999 100644 --- a/net/mpls/af_mpls.c +++ b/net/mpls/af_mpls.c @@ -22,8 +22,8 @@ #if IS_ENABLED(CONFIG_IPV6) #include #endif -#include -#include +#include +#include #include "internal.h" /* max memory we will use for mpls_route */ @@ -1223,11 +1223,13 @@ static int mpls_netconf_valid_get_req(struct sk_buff *skb, } if (!netlink_strict_get_check(skb)) - return nlmsg_parse(nlh, sizeof(struct netconfmsg), tb, - NETCONFA_MAX, devconf_mpls_policy, extack); + return nlmsg_parse_deprecated(nlh, sizeof(struct netconfmsg), + tb, NETCONFA_MAX, + devconf_mpls_policy, extack); - err = nlmsg_parse_strict(nlh, sizeof(struct netconfmsg), tb, - NETCONFA_MAX, devconf_mpls_policy, extack); + err = nlmsg_parse_deprecated_strict(nlh, sizeof(struct netconfmsg), + tb, NETCONFA_MAX, + devconf_mpls_policy, extack); if (err) return err; @@ -1788,8 +1790,8 @@ static int rtm_to_route_config(struct sk_buff *skb, int index; int err; - err = nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, rtm_mpls_policy, - extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*rtm), tb, RTA_MAX, + rtm_mpls_policy, extack); if (err < 0) goto errout; @@ -2017,7 +2019,7 @@ static int mpls_dump_route(struct sk_buff *skb, u32 portid, u32 seq, int event, u8 linkdown = 0; u8 dead = 0; - mp = nla_nest_start(skb, RTA_MULTIPATH); + mp = nla_nest_start_noflag(skb, RTA_MULTIPATH); if (!mp) goto nla_put_failure; @@ -2106,8 +2108,8 @@ static int mpls_valid_fib_dump_req(struct net *net, const struct nlmsghdr *nlh, cb->answer_flags = NLM_F_DUMP_FILTERED; } - err = nlmsg_parse_strict(nlh, sizeof(*rtm), tb, RTA_MAX, - rtm_mpls_policy, extack); + err = nlmsg_parse_deprecated_strict(nlh, sizeof(*rtm), tb, RTA_MAX, + rtm_mpls_policy, extack); if (err < 0) return err; @@ -2290,8 +2292,8 @@ static int mpls_valid_getroute_req(struct sk_buff *skb, } if (!netlink_strict_get_check(skb)) - return nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, - rtm_mpls_policy, extack); + return nlmsg_parse_deprecated(nlh, sizeof(*rtm), tb, RTA_MAX, + rtm_mpls_policy, extack); rtm = nlmsg_data(nlh); if ((rtm->rtm_dst_len && rtm->rtm_dst_len != 20) || @@ -2306,8 +2308,8 @@ static int mpls_valid_getroute_req(struct sk_buff *skb, return -EINVAL; } - err = nlmsg_parse_strict(nlh, sizeof(*rtm), tb, RTA_MAX, - rtm_mpls_policy, extack); + err = nlmsg_parse_deprecated_strict(nlh, sizeof(*rtm), tb, RTA_MAX, + rtm_mpls_policy, extack); if (err) return err; diff --git a/net/mpls/mpls_iptunnel.c b/net/mpls/mpls_iptunnel.c index f3a8557494d60e4d1ffe1f89fa32ea00c13eabab..951b52d5835bf4e84e844ff25cffa9ba82fd7328 100644 --- a/net/mpls/mpls_iptunnel.c +++ b/net/mpls/mpls_iptunnel.c @@ -137,10 +137,14 @@ static int mpls_xmit(struct sk_buff *skb) mpls_stats_inc_outucastpkts(out_dev, skb); - if (rt) - err = neigh_xmit(NEIGH_ARP_TABLE, out_dev, &rt->rt_gateway, - skb); - else if (rt6) { + if (rt) { + if (rt->rt_gw_family == AF_INET) + err = neigh_xmit(NEIGH_ARP_TABLE, out_dev, &rt->rt_gw4, + skb); + else if (rt->rt_gw_family == AF_INET6) + err = neigh_xmit(NEIGH_ND_TABLE, out_dev, &rt->rt_gw6, + skb); + } else if (rt6) { if (ipv6_addr_v4mapped(&rt6->rt6i_gateway)) { /* 6PE (RFC 4798) */ err = neigh_xmit(NEIGH_ARP_TABLE, out_dev, &rt6->rt6i_gateway.s6_addr32[3], @@ -174,8 +178,8 @@ static int mpls_build_state(struct nlattr *nla, u8 n_labels; int ret; - ret = nla_parse_nested(tb, MPLS_IPTUNNEL_MAX, nla, - mpls_iptunnel_policy, extack); + ret = nla_parse_nested_deprecated(tb, MPLS_IPTUNNEL_MAX, nla, + mpls_iptunnel_policy, extack); if (ret < 0) return ret; diff --git a/net/ncsi/ncsi-netlink.c b/net/ncsi/ncsi-netlink.c index bad17bba8ba786f589212a2575d346850bab6300..7fc4feddafa34e91ebb8d17f311c73cef84756fe 100644 --- a/net/ncsi/ncsi-netlink.c +++ b/net/ncsi/ncsi-netlink.c @@ -79,7 +79,7 @@ static int ncsi_write_channel_info(struct sk_buff *skb, nla_put_u32(skb, NCSI_CHANNEL_ATTR_VERSION_MINOR, nc->version.alpha2); nla_put_string(skb, NCSI_CHANNEL_ATTR_VERSION_STR, nc->version.fw_name); - vid_nest = nla_nest_start(skb, NCSI_CHANNEL_ATTR_VLAN_LIST); + vid_nest = nla_nest_start_noflag(skb, NCSI_CHANNEL_ATTR_VLAN_LIST); if (!vid_nest) return -ENOMEM; ncf = &nc->vlan_filter; @@ -113,19 +113,19 @@ static int ncsi_write_package_info(struct sk_buff *skb, NCSI_FOR_EACH_PACKAGE(ndp, np) { if (np->id != id) continue; - pnest = nla_nest_start(skb, NCSI_PKG_ATTR); + pnest = nla_nest_start_noflag(skb, NCSI_PKG_ATTR); if (!pnest) return -ENOMEM; nla_put_u32(skb, NCSI_PKG_ATTR_ID, np->id); if ((0x1 << np->id) == ndp->package_whitelist) nla_put_flag(skb, NCSI_PKG_ATTR_FORCED); - cnest = nla_nest_start(skb, NCSI_PKG_ATTR_CHANNEL_LIST); + cnest = nla_nest_start_noflag(skb, NCSI_PKG_ATTR_CHANNEL_LIST); if (!cnest) { nla_nest_cancel(skb, pnest); return -ENOMEM; } NCSI_FOR_EACH_CHANNEL(np, nc) { - nest = nla_nest_start(skb, NCSI_CHANNEL_ATTR); + nest = nla_nest_start_noflag(skb, NCSI_CHANNEL_ATTR); if (!nest) { nla_nest_cancel(skb, cnest); nla_nest_cancel(skb, pnest); @@ -187,7 +187,7 @@ static int ncsi_pkg_info_nl(struct sk_buff *msg, struct genl_info *info) package_id = nla_get_u32(info->attrs[NCSI_ATTR_PACKAGE_ID]); - attr = nla_nest_start(skb, NCSI_ATTR_PACKAGE_LIST); + attr = nla_nest_start_noflag(skb, NCSI_ATTR_PACKAGE_LIST); if (!attr) { kfree_skb(skb); return -EMSGSIZE; @@ -220,8 +220,8 @@ static int ncsi_pkg_info_all_nl(struct sk_buff *skb, void *hdr; int rc; - rc = genlmsg_parse(cb->nlh, &ncsi_genl_family, attrs, NCSI_ATTR_MAX, - ncsi_genl_policy, NULL); + rc = genlmsg_parse_deprecated(cb->nlh, &ncsi_genl_family, attrs, NCSI_ATTR_MAX, + ncsi_genl_policy, NULL); if (rc) return rc; @@ -250,7 +250,7 @@ static int ncsi_pkg_info_all_nl(struct sk_buff *skb, goto err; } - attr = nla_nest_start(skb, NCSI_ATTR_PACKAGE_LIST); + attr = nla_nest_start_noflag(skb, NCSI_ATTR_PACKAGE_LIST); if (!attr) { rc = -EMSGSIZE; goto err; @@ -723,38 +723,38 @@ static int ncsi_set_channel_mask_nl(struct sk_buff *msg, static const struct genl_ops ncsi_ops[] = { { .cmd = NCSI_CMD_PKG_INFO, - .policy = ncsi_genl_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = ncsi_pkg_info_nl, .dumpit = ncsi_pkg_info_all_nl, .flags = 0, }, { .cmd = NCSI_CMD_SET_INTERFACE, - .policy = ncsi_genl_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = ncsi_set_interface_nl, .flags = GENL_ADMIN_PERM, }, { .cmd = NCSI_CMD_CLEAR_INTERFACE, - .policy = ncsi_genl_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = ncsi_clear_interface_nl, .flags = GENL_ADMIN_PERM, }, { .cmd = NCSI_CMD_SEND_CMD, - .policy = ncsi_genl_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = ncsi_send_cmd_nl, .flags = GENL_ADMIN_PERM, }, { .cmd = NCSI_CMD_SET_PACKAGE_MASK, - .policy = ncsi_genl_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = ncsi_set_package_mask_nl, .flags = GENL_ADMIN_PERM, }, { .cmd = NCSI_CMD_SET_CHANNEL_MASK, - .policy = ncsi_genl_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = ncsi_set_channel_mask_nl, .flags = GENL_ADMIN_PERM, }, @@ -764,6 +764,7 @@ static struct genl_family ncsi_genl_family __ro_after_init = { .name = "NCSI", .version = 0, .maxattr = NCSI_ATTR_MAX, + .policy = ncsi_genl_policy, .module = THIS_MODULE, .ops = ncsi_ops, .n_ops = ARRAY_SIZE(ncsi_ops), diff --git a/net/netfilter/Kconfig b/net/netfilter/Kconfig index 6548271209a05c2fce99628c9b23d2cedbf8a087..02b281d3c167e51959be632d120cee43749c63de 100644 --- a/net/netfilter/Kconfig +++ b/net/netfilter/Kconfig @@ -404,11 +404,6 @@ config NF_NAT forms of full Network Address Port Translation. This can be controlled by iptables, ip6tables or nft. -config NF_NAT_NEEDED - bool - depends on NF_NAT - default y - config NF_NAT_AMANDA tristate depends on NF_CONNTRACK && NF_NAT @@ -1002,6 +997,20 @@ config NETFILTER_XT_TARGET_REDIRECT To compile it as a module, choose M here. If unsure, say N. +config NETFILTER_XT_TARGET_MASQUERADE + tristate "MASQUERADE target support" + depends on NF_NAT + default m if NETFILTER_ADVANCED=n + select NF_NAT_MASQUERADE + help + Masquerading is a special case of NAT: all outgoing connections are + changed to seem to come from a particular interface's address, and + if the interface goes down, those connections are lost. This is + only useful for dialup accounts with dynamic IP address (ie. your IP + address will be different on next dialup). + + To compile it as a module, choose M here. If unsure, say N. + config NETFILTER_XT_TARGET_TEE tristate '"TEE" - packet cloning to alternate destination' depends on NETFILTER_ADVANCED diff --git a/net/netfilter/Makefile b/net/netfilter/Makefile index 4894a85cdd0b02c94cfd8b3b0c8ca60c3257d537..72cca6b489604a5bdbf7fb42add5c7fabfe82f16 100644 --- a/net/netfilter/Makefile +++ b/net/netfilter/Makefile @@ -77,7 +77,8 @@ obj-$(CONFIG_NF_DUP_NETDEV) += nf_dup_netdev.o nf_tables-objs := nf_tables_core.o nf_tables_api.o nft_chain_filter.o \ nf_tables_trace.o nft_immediate.o nft_cmp.o nft_range.o \ nft_bitwise.o nft_byteorder.o nft_payload.o nft_lookup.o \ - nft_dynset.o nft_meta.o nft_rt.o nft_exthdr.o + nft_dynset.o nft_meta.o nft_rt.o nft_exthdr.o \ + nft_chain_route.o nf_tables_set-objs := nf_tables_set_core.o \ nft_set_hash.o nft_set_bitmap.o nft_set_rbtree.o @@ -147,6 +148,7 @@ obj-$(CONFIG_NETFILTER_XT_TARGET_NFLOG) += xt_NFLOG.o obj-$(CONFIG_NETFILTER_XT_TARGET_NFQUEUE) += xt_NFQUEUE.o obj-$(CONFIG_NETFILTER_XT_TARGET_RATEEST) += xt_RATEEST.o obj-$(CONFIG_NETFILTER_XT_TARGET_REDIRECT) += xt_REDIRECT.o +obj-$(CONFIG_NETFILTER_XT_TARGET_MASQUERADE) += xt_MASQUERADE.o obj-$(CONFIG_NETFILTER_XT_TARGET_SECMARK) += xt_SECMARK.o obj-$(CONFIG_NETFILTER_XT_TARGET_TPROXY) += xt_TPROXY.o obj-$(CONFIG_NETFILTER_XT_TARGET_TCPMSS) += xt_TCPMSS.o diff --git a/net/netfilter/core.c b/net/netfilter/core.c index 93aaec3a54ecdf701597457d2eddd6039275add0..71f06900473e9181b2d58b1daa9d42352c7382e6 100644 --- a/net/netfilter/core.c +++ b/net/netfilter/core.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include "nf_internals.h" diff --git a/net/netfilter/ipset/ip_set_bitmap_gen.h b/net/netfilter/ipset/ip_set_bitmap_gen.h index 257ca393e6f222f5bb881b547a4a53281d26379f..38ef2ea838cbfada1b997af032d55d4e3e504ca6 100644 --- a/net/netfilter/ipset/ip_set_bitmap_gen.h +++ b/net/netfilter/ipset/ip_set_bitmap_gen.h @@ -99,7 +99,7 @@ mtype_head(struct ip_set *set, struct sk_buff *skb) struct nlattr *nested; size_t memsize = mtype_memsize(map, set->dsize) + set->ext_size; - nested = ipset_nest_start(skb, IPSET_ATTR_DATA); + nested = nla_nest_start(skb, IPSET_ATTR_DATA); if (!nested) goto nla_put_failure; if (mtype_do_head(skb, map) || @@ -109,7 +109,7 @@ mtype_head(struct ip_set *set, struct sk_buff *skb) goto nla_put_failure; if (unlikely(ip_set_put_flags(skb, set))) goto nla_put_failure; - ipset_nest_end(skb, nested); + nla_nest_end(skb, nested); return 0; nla_put_failure: @@ -213,7 +213,7 @@ mtype_list(const struct ip_set *set, u32 id, first = cb->args[IPSET_CB_ARG0]; int ret = 0; - adt = ipset_nest_start(skb, IPSET_ATTR_ADT); + adt = nla_nest_start(skb, IPSET_ATTR_ADT); if (!adt) return -EMSGSIZE; /* Extensions may be replaced */ @@ -230,7 +230,7 @@ mtype_list(const struct ip_set *set, #endif ip_set_timeout_expired(ext_timeout(x, set)))) continue; - nested = ipset_nest_start(skb, IPSET_ATTR_DATA); + nested = nla_nest_start(skb, IPSET_ATTR_DATA); if (!nested) { if (id == first) { nla_nest_cancel(skb, adt); @@ -244,9 +244,9 @@ mtype_list(const struct ip_set *set, goto nla_put_failure; if (ip_set_put_extensions(skb, set, x, mtype_is_filled(x))) goto nla_put_failure; - ipset_nest_end(skb, nested); + nla_nest_end(skb, nested); } - ipset_nest_end(skb, adt); + nla_nest_end(skb, adt); /* Set listing finished */ cb->args[IPSET_CB_ARG0] = 0; @@ -259,7 +259,7 @@ nla_put_failure: cb->args[IPSET_CB_ARG0] = 0; ret = -EMSGSIZE; } - ipset_nest_end(skb, adt); + nla_nest_end(skb, adt); out: rcu_read_unlock(); return ret; diff --git a/net/netfilter/ipset/ip_set_core.c b/net/netfilter/ipset/ip_set_core.c index 45a257695befe57693c53a484f6be028e81fecf1..3f4a4936f63c0859b9461b3bd068cef4cfd9d318 100644 --- a/net/netfilter/ipset/ip_set_core.c +++ b/net/netfilter/ipset/ip_set_core.c @@ -299,8 +299,7 @@ ip_set_get_ipaddr4(struct nlattr *nla, __be32 *ipaddr) if (unlikely(!flag_nested(nla))) return -IPSET_ERR_PROTOCOL; - if (nla_parse_nested(tb, IPSET_ATTR_IPADDR_MAX, nla, - ipaddr_policy, NULL)) + if (nla_parse_nested_deprecated(tb, IPSET_ATTR_IPADDR_MAX, nla, ipaddr_policy, NULL)) return -IPSET_ERR_PROTOCOL; if (unlikely(!ip_set_attr_netorder(tb, IPSET_ATTR_IPADDR_IPV4))) return -IPSET_ERR_PROTOCOL; @@ -318,8 +317,7 @@ ip_set_get_ipaddr6(struct nlattr *nla, union nf_inet_addr *ipaddr) if (unlikely(!flag_nested(nla))) return -IPSET_ERR_PROTOCOL; - if (nla_parse_nested(tb, IPSET_ATTR_IPADDR_MAX, nla, - ipaddr_policy, NULL)) + if (nla_parse_nested_deprecated(tb, IPSET_ATTR_IPADDR_MAX, nla, ipaddr_policy, NULL)) return -IPSET_ERR_PROTOCOL; if (unlikely(!ip_set_attr_netorder(tb, IPSET_ATTR_IPADDR_IPV6))) return -IPSET_ERR_PROTOCOL; @@ -939,8 +937,7 @@ static int ip_set_create(struct net *net, struct sock *ctnl, /* Without holding any locks, create private part. */ if (attr[IPSET_ATTR_DATA] && - nla_parse_nested(tb, IPSET_ATTR_CREATE_MAX, attr[IPSET_ATTR_DATA], - set->type->create_policy, NULL)) { + nla_parse_nested_deprecated(tb, IPSET_ATTR_CREATE_MAX, attr[IPSET_ATTR_DATA], set->type->create_policy, NULL)) { ret = -IPSET_ERR_PROTOCOL; goto put_out; } @@ -1298,8 +1295,9 @@ dump_init(struct netlink_callback *cb, struct ip_set_net *inst) ip_set_id_t index; /* Second pass, so parser can't fail */ - nla_parse(cda, IPSET_ATTR_CMD_MAX, attr, nlh->nlmsg_len - min_len, - ip_set_setname_policy, NULL); + nla_parse_deprecated(cda, IPSET_ATTR_CMD_MAX, attr, + nlh->nlmsg_len - min_len, ip_set_setname_policy, + NULL); cb->args[IPSET_CB_PROTO] = nla_get_u8(cda[IPSET_ATTR_PROTOCOL]); if (cda[IPSET_ATTR_SETNAME]) { @@ -1546,8 +1544,9 @@ call_ad(struct sock *ctnl, struct sk_buff *skb, struct ip_set *set, memcpy(&errmsg->msg, nlh, nlh->nlmsg_len); cmdattr = (void *)&errmsg->msg + min_len; - nla_parse(cda, IPSET_ATTR_CMD_MAX, cmdattr, - nlh->nlmsg_len - min_len, ip_set_adt_policy, NULL); + nla_parse_deprecated(cda, IPSET_ATTR_CMD_MAX, cmdattr, + nlh->nlmsg_len - min_len, + ip_set_adt_policy, NULL); errline = nla_data(cda[IPSET_ATTR_LINENO]); @@ -1592,9 +1591,7 @@ static int ip_set_uadd(struct net *net, struct sock *ctnl, struct sk_buff *skb, use_lineno = !!attr[IPSET_ATTR_LINENO]; if (attr[IPSET_ATTR_DATA]) { - if (nla_parse_nested(tb, IPSET_ATTR_ADT_MAX, - attr[IPSET_ATTR_DATA], - set->type->adt_policy, NULL)) + if (nla_parse_nested_deprecated(tb, IPSET_ATTR_ADT_MAX, attr[IPSET_ATTR_DATA], set->type->adt_policy, NULL)) return -IPSET_ERR_PROTOCOL; ret = call_ad(ctnl, skb, set, tb, IPSET_ADD, flags, use_lineno); @@ -1605,8 +1602,7 @@ static int ip_set_uadd(struct net *net, struct sock *ctnl, struct sk_buff *skb, memset(tb, 0, sizeof(tb)); if (nla_type(nla) != IPSET_ATTR_DATA || !flag_nested(nla) || - nla_parse_nested(tb, IPSET_ATTR_ADT_MAX, nla, - set->type->adt_policy, NULL)) + nla_parse_nested_deprecated(tb, IPSET_ATTR_ADT_MAX, nla, set->type->adt_policy, NULL)) return -IPSET_ERR_PROTOCOL; ret = call_ad(ctnl, skb, set, tb, IPSET_ADD, flags, use_lineno); @@ -1647,9 +1643,7 @@ static int ip_set_udel(struct net *net, struct sock *ctnl, struct sk_buff *skb, use_lineno = !!attr[IPSET_ATTR_LINENO]; if (attr[IPSET_ATTR_DATA]) { - if (nla_parse_nested(tb, IPSET_ATTR_ADT_MAX, - attr[IPSET_ATTR_DATA], - set->type->adt_policy, NULL)) + if (nla_parse_nested_deprecated(tb, IPSET_ATTR_ADT_MAX, attr[IPSET_ATTR_DATA], set->type->adt_policy, NULL)) return -IPSET_ERR_PROTOCOL; ret = call_ad(ctnl, skb, set, tb, IPSET_DEL, flags, use_lineno); @@ -1660,8 +1654,7 @@ static int ip_set_udel(struct net *net, struct sock *ctnl, struct sk_buff *skb, memset(tb, 0, sizeof(*tb)); if (nla_type(nla) != IPSET_ATTR_DATA || !flag_nested(nla) || - nla_parse_nested(tb, IPSET_ATTR_ADT_MAX, nla, - set->type->adt_policy, NULL)) + nla_parse_nested_deprecated(tb, IPSET_ATTR_ADT_MAX, nla, set->type->adt_policy, NULL)) return -IPSET_ERR_PROTOCOL; ret = call_ad(ctnl, skb, set, tb, IPSET_DEL, flags, use_lineno); @@ -1692,8 +1685,7 @@ static int ip_set_utest(struct net *net, struct sock *ctnl, struct sk_buff *skb, if (!set) return -ENOENT; - if (nla_parse_nested(tb, IPSET_ATTR_ADT_MAX, attr[IPSET_ATTR_DATA], - set->type->adt_policy, NULL)) + if (nla_parse_nested_deprecated(tb, IPSET_ATTR_ADT_MAX, attr[IPSET_ATTR_DATA], set->type->adt_policy, NULL)) return -IPSET_ERR_PROTOCOL; rcu_read_lock_bh(); diff --git a/net/netfilter/ipset/ip_set_hash_gen.h b/net/netfilter/ipset/ip_set_hash_gen.h index 2c9609929c7167e86c02dfab5d0bc8a9a78ab8b0..01d51f775f12bdafa9c78da2bd7041d6b52dfae1 100644 --- a/net/netfilter/ipset/ip_set_hash_gen.h +++ b/net/netfilter/ipset/ip_set_hash_gen.h @@ -1057,7 +1057,7 @@ mtype_head(struct ip_set *set, struct sk_buff *skb) htable_bits = t->htable_bits; rcu_read_unlock_bh(); - nested = ipset_nest_start(skb, IPSET_ATTR_DATA); + nested = nla_nest_start(skb, IPSET_ATTR_DATA); if (!nested) goto nla_put_failure; if (nla_put_net32(skb, IPSET_ATTR_HASHSIZE, @@ -1079,7 +1079,7 @@ mtype_head(struct ip_set *set, struct sk_buff *skb) goto nla_put_failure; if (unlikely(ip_set_put_flags(skb, set))) goto nla_put_failure; - ipset_nest_end(skb, nested); + nla_nest_end(skb, nested); return 0; nla_put_failure: @@ -1124,7 +1124,7 @@ mtype_list(const struct ip_set *set, void *incomplete; int i, ret = 0; - atd = ipset_nest_start(skb, IPSET_ATTR_ADT); + atd = nla_nest_start(skb, IPSET_ATTR_ADT); if (!atd) return -EMSGSIZE; @@ -1150,7 +1150,7 @@ mtype_list(const struct ip_set *set, continue; pr_debug("list hash %lu hbucket %p i %u, data %p\n", cb->args[IPSET_CB_ARG0], n, i, e); - nested = ipset_nest_start(skb, IPSET_ATTR_DATA); + nested = nla_nest_start(skb, IPSET_ATTR_DATA); if (!nested) { if (cb->args[IPSET_CB_ARG0] == first) { nla_nest_cancel(skb, atd); @@ -1163,10 +1163,10 @@ mtype_list(const struct ip_set *set, goto nla_put_failure; if (ip_set_put_extensions(skb, set, e, true)) goto nla_put_failure; - ipset_nest_end(skb, nested); + nla_nest_end(skb, nested); } } - ipset_nest_end(skb, atd); + nla_nest_end(skb, atd); /* Set listing finished */ cb->args[IPSET_CB_ARG0] = 0; @@ -1180,7 +1180,7 @@ nla_put_failure: cb->args[IPSET_CB_ARG0] = 0; ret = -EMSGSIZE; } else { - ipset_nest_end(skb, atd); + nla_nest_end(skb, atd); } out: rcu_read_unlock(); diff --git a/net/netfilter/ipset/ip_set_list_set.c b/net/netfilter/ipset/ip_set_list_set.c index 8da228da53ae98e0befbcede2d3b3c8f394aefc2..4f894165cdcdf6de4e3c5721e2663931c0680a86 100644 --- a/net/netfilter/ipset/ip_set_list_set.c +++ b/net/netfilter/ipset/ip_set_list_set.c @@ -466,7 +466,7 @@ list_set_head(struct ip_set *set, struct sk_buff *skb) struct nlattr *nested; size_t memsize = list_set_memsize(map, set->dsize) + set->ext_size; - nested = ipset_nest_start(skb, IPSET_ATTR_DATA); + nested = nla_nest_start(skb, IPSET_ATTR_DATA); if (!nested) goto nla_put_failure; if (nla_put_net32(skb, IPSET_ATTR_SIZE, htonl(map->size)) || @@ -476,7 +476,7 @@ list_set_head(struct ip_set *set, struct sk_buff *skb) goto nla_put_failure; if (unlikely(ip_set_put_flags(skb, set))) goto nla_put_failure; - ipset_nest_end(skb, nested); + nla_nest_end(skb, nested); return 0; nla_put_failure: @@ -494,7 +494,7 @@ list_set_list(const struct ip_set *set, struct set_elem *e; int ret = 0; - atd = ipset_nest_start(skb, IPSET_ATTR_ADT); + atd = nla_nest_start(skb, IPSET_ATTR_ADT); if (!atd) return -EMSGSIZE; @@ -506,7 +506,7 @@ list_set_list(const struct ip_set *set, i++; continue; } - nested = ipset_nest_start(skb, IPSET_ATTR_DATA); + nested = nla_nest_start(skb, IPSET_ATTR_DATA); if (!nested) goto nla_put_failure; ip_set_name_byindex(map->net, e->id, name); @@ -514,11 +514,11 @@ list_set_list(const struct ip_set *set, goto nla_put_failure; if (ip_set_put_extensions(skb, set, e, true)) goto nla_put_failure; - ipset_nest_end(skb, nested); + nla_nest_end(skb, nested); i++; } - ipset_nest_end(skb, atd); + nla_nest_end(skb, atd); /* Set listing finished */ cb->args[IPSET_CB_ARG0] = 0; goto out; @@ -531,7 +531,7 @@ nla_put_failure: ret = -EMSGSIZE; } else { cb->args[IPSET_CB_ARG0] = i; - ipset_nest_end(skb, atd); + nla_nest_end(skb, atd); } out: rcu_read_unlock(); diff --git a/net/netfilter/ipvs/ip_vs_ctl.c b/net/netfilter/ipvs/ip_vs_ctl.c index 053cd96b9c7634a2b6029a0afac6d8cfade43bec..0e887159425cad035e6b278d7bbee3623f09fb08 100644 --- a/net/netfilter/ipvs/ip_vs_ctl.c +++ b/net/netfilter/ipvs/ip_vs_ctl.c @@ -831,6 +831,10 @@ __ip_vs_update_dest(struct ip_vs_service *svc, struct ip_vs_dest *dest, conn_flags = udest->conn_flags & IP_VS_CONN_F_DEST_MASK; conn_flags |= IP_VS_CONN_F_INACTIVE; + /* set the tunnel info */ + dest->tun_type = udest->tun_type; + dest->tun_port = udest->tun_port; + /* set the IP_VS_CONN_F_NOOUTPUT flag if not masquerading/NAT */ if ((conn_flags & IP_VS_CONN_F_FWD_MASK) != IP_VS_CONN_F_MASQ) { conn_flags |= IP_VS_CONN_F_NOOUTPUT; @@ -987,6 +991,13 @@ ip_vs_add_dest(struct ip_vs_service *svc, struct ip_vs_dest_user_kern *udest) return -ERANGE; } + if (udest->tun_type == IP_VS_CONN_F_TUNNEL_TYPE_GUE) { + if (udest->tun_port == 0) { + pr_err("%s(): tunnel port is zero\n", __func__); + return -EINVAL; + } + } + ip_vs_addr_copy(udest->af, &daddr, &udest->addr); /* We use function that requires RCU lock */ @@ -1051,6 +1062,13 @@ ip_vs_edit_dest(struct ip_vs_service *svc, struct ip_vs_dest_user_kern *udest) return -ERANGE; } + if (udest->tun_type == IP_VS_CONN_F_TUNNEL_TYPE_GUE) { + if (udest->tun_port == 0) { + pr_err("%s(): tunnel port is zero\n", __func__); + return -EINVAL; + } + } + ip_vs_addr_copy(udest->af, &daddr, &udest->addr); /* We use function that requires RCU lock */ @@ -2333,6 +2351,7 @@ static void ip_vs_copy_udest_compat(struct ip_vs_dest_user_kern *udest, udest->u_threshold = udest_compat->u_threshold; udest->l_threshold = udest_compat->l_threshold; udest->af = AF_INET; + udest->tun_type = IP_VS_CONN_F_TUNNEL_TYPE_IPIP; } static int @@ -2890,12 +2909,14 @@ static const struct nla_policy ip_vs_dest_policy[IPVS_DEST_ATTR_MAX + 1] = { [IPVS_DEST_ATTR_PERSIST_CONNS] = { .type = NLA_U32 }, [IPVS_DEST_ATTR_STATS] = { .type = NLA_NESTED }, [IPVS_DEST_ATTR_ADDR_FAMILY] = { .type = NLA_U16 }, + [IPVS_DEST_ATTR_TUN_TYPE] = { .type = NLA_U8 }, + [IPVS_DEST_ATTR_TUN_PORT] = { .type = NLA_U16 }, }; static int ip_vs_genl_fill_stats(struct sk_buff *skb, int container_type, struct ip_vs_kstats *kstats) { - struct nlattr *nl_stats = nla_nest_start(skb, container_type); + struct nlattr *nl_stats = nla_nest_start_noflag(skb, container_type); if (!nl_stats) return -EMSGSIZE; @@ -2925,7 +2946,7 @@ nla_put_failure: static int ip_vs_genl_fill_stats64(struct sk_buff *skb, int container_type, struct ip_vs_kstats *kstats) { - struct nlattr *nl_stats = nla_nest_start(skb, container_type); + struct nlattr *nl_stats = nla_nest_start_noflag(skb, container_type); if (!nl_stats) return -EMSGSIZE; @@ -2971,7 +2992,7 @@ static int ip_vs_genl_fill_service(struct sk_buff *skb, struct ip_vs_kstats kstats; char *sched_name; - nl_service = nla_nest_start(skb, IPVS_CMD_ATTR_SERVICE); + nl_service = nla_nest_start_noflag(skb, IPVS_CMD_ATTR_SERVICE); if (!nl_service) return -EMSGSIZE; @@ -3095,8 +3116,7 @@ static int ip_vs_genl_parse_service(struct netns_ipvs *ipvs, /* Parse mandatory identifying service fields first */ if (nla == NULL || - nla_parse_nested(attrs, IPVS_SVC_ATTR_MAX, nla, - ip_vs_svc_policy, NULL)) + nla_parse_nested_deprecated(attrs, IPVS_SVC_ATTR_MAX, nla, ip_vs_svc_policy, NULL)) return -EINVAL; nla_af = attrs[IPVS_SVC_ATTR_AF]; @@ -3182,7 +3202,7 @@ static int ip_vs_genl_fill_dest(struct sk_buff *skb, struct ip_vs_dest *dest) struct nlattr *nl_dest; struct ip_vs_kstats kstats; - nl_dest = nla_nest_start(skb, IPVS_CMD_ATTR_DEST); + nl_dest = nla_nest_start_noflag(skb, IPVS_CMD_ATTR_DEST); if (!nl_dest) return -EMSGSIZE; @@ -3193,6 +3213,10 @@ static int ip_vs_genl_fill_dest(struct sk_buff *skb, struct ip_vs_dest *dest) IP_VS_CONN_F_FWD_MASK)) || nla_put_u32(skb, IPVS_DEST_ATTR_WEIGHT, atomic_read(&dest->weight)) || + nla_put_u8(skb, IPVS_DEST_ATTR_TUN_TYPE, + dest->tun_type) || + nla_put_be16(skb, IPVS_DEST_ATTR_TUN_PORT, + dest->tun_port) || nla_put_u32(skb, IPVS_DEST_ATTR_U_THRESH, dest->u_threshold) || nla_put_u32(skb, IPVS_DEST_ATTR_L_THRESH, dest->l_threshold) || nla_put_u32(skb, IPVS_DEST_ATTR_ACTIVE_CONNS, @@ -3254,8 +3278,7 @@ static int ip_vs_genl_dump_dests(struct sk_buff *skb, mutex_lock(&__ip_vs_mutex); /* Try to find the service for which to dump destinations */ - if (nlmsg_parse(cb->nlh, GENL_HDRLEN, attrs, IPVS_CMD_ATTR_MAX, - ip_vs_cmd_policy, cb->extack)) + if (nlmsg_parse_deprecated(cb->nlh, GENL_HDRLEN, attrs, IPVS_CMD_ATTR_MAX, ip_vs_cmd_policy, cb->extack)) goto out_err; @@ -3291,8 +3314,7 @@ static int ip_vs_genl_parse_dest(struct ip_vs_dest_user_kern *udest, /* Parse mandatory identifying destination fields first */ if (nla == NULL || - nla_parse_nested(attrs, IPVS_DEST_ATTR_MAX, nla, - ip_vs_dest_policy, NULL)) + nla_parse_nested_deprecated(attrs, IPVS_DEST_ATTR_MAX, nla, ip_vs_dest_policy, NULL)) return -EINVAL; nla_addr = attrs[IPVS_DEST_ATTR_ADDR]; @@ -3315,12 +3337,14 @@ static int ip_vs_genl_parse_dest(struct ip_vs_dest_user_kern *udest, /* If a full entry was requested, check for the additional fields */ if (full_entry) { struct nlattr *nla_fwd, *nla_weight, *nla_u_thresh, - *nla_l_thresh; + *nla_l_thresh, *nla_tun_type, *nla_tun_port; nla_fwd = attrs[IPVS_DEST_ATTR_FWD_METHOD]; nla_weight = attrs[IPVS_DEST_ATTR_WEIGHT]; nla_u_thresh = attrs[IPVS_DEST_ATTR_U_THRESH]; nla_l_thresh = attrs[IPVS_DEST_ATTR_L_THRESH]; + nla_tun_type = attrs[IPVS_DEST_ATTR_TUN_TYPE]; + nla_tun_port = attrs[IPVS_DEST_ATTR_TUN_PORT]; if (!(nla_fwd && nla_weight && nla_u_thresh && nla_l_thresh)) return -EINVAL; @@ -3330,6 +3354,12 @@ static int ip_vs_genl_parse_dest(struct ip_vs_dest_user_kern *udest, udest->weight = nla_get_u32(nla_weight); udest->u_threshold = nla_get_u32(nla_u_thresh); udest->l_threshold = nla_get_u32(nla_l_thresh); + + if (nla_tun_type) + udest->tun_type = nla_get_u8(nla_tun_type); + + if (nla_tun_port) + udest->tun_port = nla_get_be16(nla_tun_port); } return 0; @@ -3340,7 +3370,7 @@ static int ip_vs_genl_fill_daemon(struct sk_buff *skb, __u32 state, { struct nlattr *nl_daemon; - nl_daemon = nla_nest_start(skb, IPVS_CMD_ATTR_DAEMON); + nl_daemon = nla_nest_start_noflag(skb, IPVS_CMD_ATTR_DAEMON); if (!nl_daemon) return -EMSGSIZE; @@ -3528,9 +3558,7 @@ static int ip_vs_genl_set_daemon(struct sk_buff *skb, struct genl_info *info) struct nlattr *daemon_attrs[IPVS_DAEMON_ATTR_MAX + 1]; if (!info->attrs[IPVS_CMD_ATTR_DAEMON] || - nla_parse_nested(daemon_attrs, IPVS_DAEMON_ATTR_MAX, - info->attrs[IPVS_CMD_ATTR_DAEMON], - ip_vs_daemon_policy, info->extack)) + nla_parse_nested_deprecated(daemon_attrs, IPVS_DAEMON_ATTR_MAX, info->attrs[IPVS_CMD_ATTR_DAEMON], ip_vs_daemon_policy, info->extack)) goto out; if (cmd == IPVS_CMD_NEW_DAEMON) @@ -3774,94 +3802,98 @@ out: static const struct genl_ops ip_vs_genl_ops[] = { { .cmd = IPVS_CMD_NEW_SERVICE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = ip_vs_cmd_policy, .doit = ip_vs_genl_set_cmd, }, { .cmd = IPVS_CMD_SET_SERVICE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = ip_vs_cmd_policy, .doit = ip_vs_genl_set_cmd, }, { .cmd = IPVS_CMD_DEL_SERVICE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = ip_vs_cmd_policy, .doit = ip_vs_genl_set_cmd, }, { .cmd = IPVS_CMD_GET_SERVICE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, .doit = ip_vs_genl_get_cmd, .dumpit = ip_vs_genl_dump_services, - .policy = ip_vs_cmd_policy, }, { .cmd = IPVS_CMD_NEW_DEST, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = ip_vs_cmd_policy, .doit = ip_vs_genl_set_cmd, }, { .cmd = IPVS_CMD_SET_DEST, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = ip_vs_cmd_policy, .doit = ip_vs_genl_set_cmd, }, { .cmd = IPVS_CMD_DEL_DEST, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = ip_vs_cmd_policy, .doit = ip_vs_genl_set_cmd, }, { .cmd = IPVS_CMD_GET_DEST, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = ip_vs_cmd_policy, .dumpit = ip_vs_genl_dump_dests, }, { .cmd = IPVS_CMD_NEW_DAEMON, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = ip_vs_cmd_policy, .doit = ip_vs_genl_set_daemon, }, { .cmd = IPVS_CMD_DEL_DAEMON, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = ip_vs_cmd_policy, .doit = ip_vs_genl_set_daemon, }, { .cmd = IPVS_CMD_GET_DAEMON, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, .dumpit = ip_vs_genl_dump_daemons, }, { .cmd = IPVS_CMD_SET_CONFIG, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = ip_vs_cmd_policy, .doit = ip_vs_genl_set_cmd, }, { .cmd = IPVS_CMD_GET_CONFIG, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, .doit = ip_vs_genl_get_cmd, }, { .cmd = IPVS_CMD_GET_INFO, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, .doit = ip_vs_genl_get_cmd, }, { .cmd = IPVS_CMD_ZERO, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = ip_vs_cmd_policy, .doit = ip_vs_genl_set_cmd, }, { .cmd = IPVS_CMD_FLUSH, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, .doit = ip_vs_genl_set_cmd, }, @@ -3872,6 +3904,7 @@ static struct genl_family ip_vs_genl_family __ro_after_init = { .name = IPVS_GENL_NAME, .version = IPVS_GENL_VERSION, .maxattr = IPVS_CMD_ATTR_MAX, + .policy = ip_vs_cmd_policy, .netnsok = true, /* Make ipvsadm to work on netns */ .module = THIS_MODULE, .ops = ip_vs_genl_ops, diff --git a/net/netfilter/ipvs/ip_vs_xmit.c b/net/netfilter/ipvs/ip_vs_xmit.c index 175349fcf91f5627fdd5143d06bcacda0050ed47..8d6f94b6777293dd08c19f9522c594d33a3710b6 100644 --- a/net/netfilter/ipvs/ip_vs_xmit.c +++ b/net/netfilter/ipvs/ip_vs_xmit.c @@ -32,6 +32,7 @@ #include #include /* for tcphdr */ #include +#include #include /* for csum_tcpudp_magic */ #include #include /* for icmp_send */ @@ -382,6 +383,10 @@ __ip_vs_get_out_rt(struct netns_ipvs *ipvs, int skb_af, struct sk_buff *skb, mtu = dst_mtu(&rt->dst); } else { mtu = dst_mtu(&rt->dst) - sizeof(struct iphdr); + if (!dest) + goto err_put; + if (dest->tun_type == IP_VS_CONN_F_TUNNEL_TYPE_GUE) + mtu -= sizeof(struct udphdr) + sizeof(struct guehdr); if (mtu < 68) { IP_VS_DBG_RL("%s(): mtu less than 68\n", __func__); goto err_put; @@ -533,6 +538,10 @@ __ip_vs_get_out_rt_v6(struct netns_ipvs *ipvs, int skb_af, struct sk_buff *skb, mtu = dst_mtu(&rt->dst); else { mtu = dst_mtu(&rt->dst) - sizeof(struct ipv6hdr); + if (!dest) + goto err_put; + if (dest->tun_type == IP_VS_CONN_F_TUNNEL_TYPE_GUE) + mtu -= sizeof(struct udphdr) + sizeof(struct guehdr); if (mtu < IPV6_MIN_MTU) { IP_VS_DBG_RL("%s(): mtu less than %d\n", __func__, IPV6_MIN_MTU); @@ -989,6 +998,41 @@ static inline int __tun_gso_type_mask(int encaps_af, int orig_af) } } +static int +ipvs_gue_encap(struct net *net, struct sk_buff *skb, + struct ip_vs_conn *cp, __u8 *next_protocol) +{ + __be16 dport; + __be16 sport = udp_flow_src_port(net, skb, 0, 0, false); + struct udphdr *udph; /* Our new UDP header */ + struct guehdr *gueh; /* Our new GUE header */ + + skb_push(skb, sizeof(struct guehdr)); + + gueh = (struct guehdr *)skb->data; + + gueh->control = 0; + gueh->version = 0; + gueh->hlen = 0; + gueh->flags = 0; + gueh->proto_ctype = *next_protocol; + + skb_push(skb, sizeof(struct udphdr)); + skb_reset_transport_header(skb); + + udph = udp_hdr(skb); + + dport = cp->dest->tun_port; + udph->dest = dport; + udph->source = sport; + udph->len = htons(skb->len); + udph->check = 0; + + *next_protocol = IPPROTO_UDP; + + return 0; +} + /* * IP Tunneling transmitter * @@ -1025,6 +1069,7 @@ ip_vs_tunnel_xmit(struct sk_buff *skb, struct ip_vs_conn *cp, struct iphdr *iph; /* Our new IP header */ unsigned int max_headroom; /* The extra header space needed */ int ret, local; + int tun_type, gso_type; EnterFunction(10); @@ -1046,6 +1091,11 @@ ip_vs_tunnel_xmit(struct sk_buff *skb, struct ip_vs_conn *cp, */ max_headroom = LL_RESERVED_SPACE(tdev) + sizeof(struct iphdr); + tun_type = cp->dest->tun_type; + + if (tun_type == IP_VS_CONN_F_TUNNEL_TYPE_GUE) + max_headroom += sizeof(struct udphdr) + sizeof(struct guehdr); + /* We only care about the df field if sysctl_pmtu_disc(ipvs) is set */ dfp = sysctl_pmtu_disc(ipvs) ? &df : NULL; skb = ip_vs_prepare_tunneled_skb(skb, cp->af, max_headroom, @@ -1054,11 +1104,20 @@ ip_vs_tunnel_xmit(struct sk_buff *skb, struct ip_vs_conn *cp, if (IS_ERR(skb)) goto tx_error; - if (iptunnel_handle_offloads(skb, __tun_gso_type_mask(AF_INET, cp->af))) + gso_type = __tun_gso_type_mask(AF_INET, cp->af); + if (tun_type == IP_VS_CONN_F_TUNNEL_TYPE_GUE) + gso_type |= SKB_GSO_UDP_TUNNEL; + + if (iptunnel_handle_offloads(skb, gso_type)) goto tx_error; skb->transport_header = skb->network_header; + skb_set_inner_ipproto(skb, next_protocol); + + if (tun_type == IP_VS_CONN_F_TUNNEL_TYPE_GUE) + ipvs_gue_encap(net, skb, cp, &next_protocol); + skb_push(skb, sizeof(struct iphdr)); skb_reset_network_header(skb); memset(&(IPCB(skb)->opt), 0, sizeof(IPCB(skb)->opt)); @@ -1102,6 +1161,8 @@ int ip_vs_tunnel_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp, struct ip_vs_protocol *pp, struct ip_vs_iphdr *ipvsh) { + struct netns_ipvs *ipvs = cp->ipvs; + struct net *net = ipvs->net; struct rt6_info *rt; /* Route to the other host */ struct in6_addr saddr; /* Source for tunnel */ struct net_device *tdev; /* Device to other host */ @@ -1112,10 +1173,11 @@ ip_vs_tunnel_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp, struct ipv6hdr *iph; /* Our new IP header */ unsigned int max_headroom; /* The extra header space needed */ int ret, local; + int tun_type, gso_type; EnterFunction(10); - local = __ip_vs_get_out_rt_v6(cp->ipvs, cp->af, skb, cp->dest, + local = __ip_vs_get_out_rt_v6(ipvs, cp->af, skb, cp->dest, &cp->daddr.in6, &saddr, ipvsh, 1, IP_VS_RT_MODE_LOCAL | @@ -1134,17 +1196,31 @@ ip_vs_tunnel_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp, */ max_headroom = LL_RESERVED_SPACE(tdev) + sizeof(struct ipv6hdr); + tun_type = cp->dest->tun_type; + + if (tun_type == IP_VS_CONN_F_TUNNEL_TYPE_GUE) + max_headroom += sizeof(struct udphdr) + sizeof(struct guehdr); + skb = ip_vs_prepare_tunneled_skb(skb, cp->af, max_headroom, &next_protocol, &payload_len, &dsfield, &ttl, NULL); if (IS_ERR(skb)) goto tx_error; - if (iptunnel_handle_offloads(skb, __tun_gso_type_mask(AF_INET6, cp->af))) + gso_type = __tun_gso_type_mask(AF_INET6, cp->af); + if (tun_type == IP_VS_CONN_F_TUNNEL_TYPE_GUE) + gso_type |= SKB_GSO_UDP_TUNNEL; + + if (iptunnel_handle_offloads(skb, gso_type)) goto tx_error; skb->transport_header = skb->network_header; + skb_set_inner_ipproto(skb, next_protocol); + + if (tun_type == IP_VS_CONN_F_TUNNEL_TYPE_GUE) + ipvs_gue_encap(net, skb, cp, &next_protocol); + skb_push(skb, sizeof(struct ipv6hdr)); skb_reset_network_header(skb); memset(&(IPCB(skb)->opt), 0, sizeof(IPCB(skb)->opt)); @@ -1167,7 +1243,7 @@ ip_vs_tunnel_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp, ret = ip_vs_tunnel_xmit_prepare(skb, cp); if (ret == NF_ACCEPT) - ip6_local_out(cp->ipvs->net, skb->sk, skb); + ip6_local_out(net, skb->sk, skb); else if (ret == NF_DROP) kfree_skb(skb); diff --git a/net/netfilter/nf_conntrack_amanda.c b/net/netfilter/nf_conntrack_amanda.c index f2681ec5b5f6eaa1deb3e17a704eeccd46491c5c..dbec6fca0d9eb25735a78c7c13ad52dad713820c 100644 --- a/net/netfilter/nf_conntrack_amanda.c +++ b/net/netfilter/nf_conntrack_amanda.c @@ -28,11 +28,13 @@ static unsigned int master_timeout __read_mostly = 300; static char *ts_algo = "kmp"; +#define HELPER_NAME "amanda" + MODULE_AUTHOR("Brian J. Murrell "); MODULE_DESCRIPTION("Amanda connection tracking module"); MODULE_LICENSE("GPL"); MODULE_ALIAS("ip_conntrack_amanda"); -MODULE_ALIAS_NFCT_HELPER("amanda"); +MODULE_ALIAS_NFCT_HELPER(HELPER_NAME); module_param(master_timeout, uint, 0600); MODULE_PARM_DESC(master_timeout, "timeout for the master connection"); @@ -179,13 +181,14 @@ static const struct nf_conntrack_expect_policy amanda_exp_policy = { static struct nf_conntrack_helper amanda_helper[2] __read_mostly = { { - .name = "amanda", + .name = HELPER_NAME, .me = THIS_MODULE, .help = amanda_help, .tuple.src.l3num = AF_INET, .tuple.src.u.udp.port = cpu_to_be16(10080), .tuple.dst.protonum = IPPROTO_UDP, .expect_policy = &amanda_exp_policy, + .nat_mod_name = NF_NAT_HELPER_NAME(HELPER_NAME), }, { .name = "amanda", @@ -195,6 +198,7 @@ static struct nf_conntrack_helper amanda_helper[2] __read_mostly = { .tuple.src.u.udp.port = cpu_to_be16(10080), .tuple.dst.protonum = IPPROTO_UDP, .expect_policy = &amanda_exp_policy, + .nat_mod_name = NF_NAT_HELPER_NAME(HELPER_NAME), }, }; diff --git a/net/netfilter/nf_conntrack_expect.c b/net/netfilter/nf_conntrack_expect.c index 334d6e5b77621314f273bc190d5b704a43342c61..59c18804a10a0deda45a33b2ab24fa39b0710b0c 100644 --- a/net/netfilter/nf_conntrack_expect.c +++ b/net/netfilter/nf_conntrack_expect.c @@ -336,7 +336,7 @@ void nf_ct_expect_init(struct nf_conntrack_expect *exp, unsigned int class, exp->tuple.dst.u.all = *dst; -#ifdef CONFIG_NF_NAT_NEEDED +#if IS_ENABLED(CONFIG_NF_NAT) memset(&exp->saved_addr, 0, sizeof(exp->saved_addr)); memset(&exp->saved_proto, 0, sizeof(exp->saved_proto)); #endif diff --git a/net/netfilter/nf_conntrack_ftp.c b/net/netfilter/nf_conntrack_ftp.c index a11c304fb7713b36eacd21f421b72e4a4db1b9e7..32aeac1c4760cae52b4e9028a1ebaa221c478c73 100644 --- a/net/netfilter/nf_conntrack_ftp.c +++ b/net/netfilter/nf_conntrack_ftp.c @@ -29,11 +29,13 @@ #include #include +#define HELPER_NAME "ftp" + MODULE_LICENSE("GPL"); MODULE_AUTHOR("Rusty Russell "); MODULE_DESCRIPTION("ftp connection tracking helper"); MODULE_ALIAS("ip_conntrack_ftp"); -MODULE_ALIAS_NFCT_HELPER("ftp"); +MODULE_ALIAS_NFCT_HELPER(HELPER_NAME); /* This is slow, but it's simple. --RR */ static char *ftp_buffer; @@ -588,12 +590,14 @@ static int __init nf_conntrack_ftp_init(void) /* FIXME should be configurable whether IPv4 and IPv6 FTP connections are tracked or not - YK */ for (i = 0; i < ports_c; i++) { - nf_ct_helper_init(&ftp[2 * i], AF_INET, IPPROTO_TCP, "ftp", - FTP_PORT, ports[i], ports[i], &ftp_exp_policy, - 0, help, nf_ct_ftp_from_nlattr, THIS_MODULE); - nf_ct_helper_init(&ftp[2 * i + 1], AF_INET6, IPPROTO_TCP, "ftp", - FTP_PORT, ports[i], ports[i], &ftp_exp_policy, - 0, help, nf_ct_ftp_from_nlattr, THIS_MODULE); + nf_ct_helper_init(&ftp[2 * i], AF_INET, IPPROTO_TCP, + HELPER_NAME, FTP_PORT, ports[i], ports[i], + &ftp_exp_policy, 0, help, + nf_ct_ftp_from_nlattr, THIS_MODULE); + nf_ct_helper_init(&ftp[2 * i + 1], AF_INET6, IPPROTO_TCP, + HELPER_NAME, FTP_PORT, ports[i], ports[i], + &ftp_exp_policy, 0, help, + nf_ct_ftp_from_nlattr, THIS_MODULE); } ret = nf_conntrack_helpers_register(ftp, ports_c * 2); diff --git a/net/netfilter/nf_conntrack_helper.c b/net/netfilter/nf_conntrack_helper.c index 274baf1dab87060454c0aa46f4b0e280c4f01340..918df7f71c8f349f3c29ad77aa419201dbea434e 100644 --- a/net/netfilter/nf_conntrack_helper.c +++ b/net/netfilter/nf_conntrack_helper.c @@ -42,6 +42,9 @@ module_param_named(nf_conntrack_helper, nf_ct_auto_assign_helper, bool, 0644); MODULE_PARM_DESC(nf_conntrack_helper, "Enable automatic conntrack helper assignment (default 0)"); +static DEFINE_MUTEX(nf_ct_nat_helpers_mutex); +static struct list_head nf_ct_nat_helpers __read_mostly; + /* Stupid hash, but collision free for the default registrations of the * helpers currently in the kernel. */ static unsigned int helper_hash(const struct nf_conntrack_tuple *tuple) @@ -130,6 +133,70 @@ void nf_conntrack_helper_put(struct nf_conntrack_helper *helper) } EXPORT_SYMBOL_GPL(nf_conntrack_helper_put); +static struct nf_conntrack_nat_helper * +nf_conntrack_nat_helper_find(const char *mod_name) +{ + struct nf_conntrack_nat_helper *cur; + bool found = false; + + list_for_each_entry_rcu(cur, &nf_ct_nat_helpers, list) { + if (!strcmp(cur->mod_name, mod_name)) { + found = true; + break; + } + } + return found ? cur : NULL; +} + +int +nf_nat_helper_try_module_get(const char *name, u16 l3num, u8 protonum) +{ + struct nf_conntrack_helper *h; + struct nf_conntrack_nat_helper *nat; + char mod_name[NF_CT_HELPER_NAME_LEN]; + int ret = 0; + + rcu_read_lock(); + h = __nf_conntrack_helper_find(name, l3num, protonum); + if (!h) { + rcu_read_unlock(); + return -ENOENT; + } + + nat = nf_conntrack_nat_helper_find(h->nat_mod_name); + if (!nat) { + snprintf(mod_name, sizeof(mod_name), "%s", h->nat_mod_name); + rcu_read_unlock(); + request_module(mod_name); + + rcu_read_lock(); + nat = nf_conntrack_nat_helper_find(mod_name); + if (!nat) { + rcu_read_unlock(); + return -ENOENT; + } + } + + if (!try_module_get(nat->module)) + ret = -ENOENT; + + rcu_read_unlock(); + return ret; +} +EXPORT_SYMBOL_GPL(nf_nat_helper_try_module_get); + +void nf_nat_helper_put(struct nf_conntrack_helper *helper) +{ + struct nf_conntrack_nat_helper *nat; + + nat = nf_conntrack_nat_helper_find(helper->nat_mod_name); + if (WARN_ON_ONCE(!nat)) + return; + + module_put(nat->module); +} +EXPORT_SYMBOL_GPL(nf_nat_helper_put); + struct nf_conn_help * nf_ct_helper_ext_add(struct nf_conn *ct, gfp_t gfp) { @@ -430,6 +497,8 @@ void nf_ct_helper_init(struct nf_conntrack_helper *helper, helper->help = help; helper->from_nlattr = from_nlattr; helper->me = module; + snprintf(helper->nat_mod_name, sizeof(helper->nat_mod_name), + NF_NAT_HELPER_PREFIX "%s", name); if (spec_port == default_port) snprintf(helper->name, sizeof(helper->name), "%s", name); @@ -466,6 +535,22 @@ void nf_conntrack_helpers_unregister(struct nf_conntrack_helper *helper, } EXPORT_SYMBOL_GPL(nf_conntrack_helpers_unregister); +void nf_nat_helper_register(struct nf_conntrack_nat_helper *nat) +{ + mutex_lock(&nf_ct_nat_helpers_mutex); + list_add_rcu(&nat->list, &nf_ct_nat_helpers); + mutex_unlock(&nf_ct_nat_helpers_mutex); +} +EXPORT_SYMBOL_GPL(nf_nat_helper_register); + +void nf_nat_helper_unregister(struct nf_conntrack_nat_helper *nat) +{ + mutex_lock(&nf_ct_nat_helpers_mutex); + list_del_rcu(&nat->list); + mutex_unlock(&nf_ct_nat_helpers_mutex); +} +EXPORT_SYMBOL_GPL(nf_nat_helper_unregister); + static const struct nf_ct_ext_type helper_extend = { .len = sizeof(struct nf_conn_help), .align = __alignof__(struct nf_conn_help), @@ -493,6 +578,7 @@ int nf_conntrack_helper_init(void) goto out_extend; } + INIT_LIST_HEAD(&nf_ct_nat_helpers); return 0; out_extend: kvfree(nf_ct_helper_hash); diff --git a/net/netfilter/nf_conntrack_irc.c b/net/netfilter/nf_conntrack_irc.c index 4099f4d79bae7b675df541547d70f9d793794c16..79e5014b3b0d940c6c7f0700435eaedd9d4c2031 100644 --- a/net/netfilter/nf_conntrack_irc.c +++ b/net/netfilter/nf_conntrack_irc.c @@ -42,11 +42,13 @@ unsigned int (*nf_nat_irc_hook)(struct sk_buff *skb, struct nf_conntrack_expect *exp) __read_mostly; EXPORT_SYMBOL_GPL(nf_nat_irc_hook); +#define HELPER_NAME "irc" + MODULE_AUTHOR("Harald Welte "); MODULE_DESCRIPTION("IRC (DCC) connection tracking helper"); MODULE_LICENSE("GPL"); MODULE_ALIAS("ip_conntrack_irc"); -MODULE_ALIAS_NFCT_HELPER("irc"); +MODULE_ALIAS_NFCT_HELPER(HELPER_NAME); module_param_array(ports, ushort, &ports_c, 0400); MODULE_PARM_DESC(ports, "port numbers of IRC servers"); @@ -259,7 +261,7 @@ static int __init nf_conntrack_irc_init(void) ports[ports_c++] = IRC_PORT; for (i = 0; i < ports_c; i++) { - nf_ct_helper_init(&irc[i], AF_INET, IPPROTO_TCP, "irc", + nf_ct_helper_init(&irc[i], AF_INET, IPPROTO_TCP, HELPER_NAME, IRC_PORT, ports[i], i, &irc_exp_policy, 0, help, NULL, THIS_MODULE); } diff --git a/net/netfilter/nf_conntrack_netlink.c b/net/netfilter/nf_conntrack_netlink.c index d7f61b0547c65c5e85a2080481906d2918a1eddf..8dcc064d518dc8e779ef3d77b278bce7f0b7d6c6 100644 --- a/net/netfilter/nf_conntrack_netlink.c +++ b/net/netfilter/nf_conntrack_netlink.c @@ -46,7 +46,7 @@ #include #include #include -#ifdef CONFIG_NF_NAT_NEEDED +#if IS_ENABLED(CONFIG_NF_NAT) #include #include #endif @@ -63,7 +63,7 @@ static int ctnetlink_dump_tuples_proto(struct sk_buff *skb, int ret = 0; struct nlattr *nest_parms; - nest_parms = nla_nest_start(skb, CTA_TUPLE_PROTO | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, CTA_TUPLE_PROTO); if (!nest_parms) goto nla_put_failure; if (nla_put_u8(skb, CTA_PROTO_NUM, tuple->dst.protonum)) @@ -104,7 +104,7 @@ static int ctnetlink_dump_tuples_ip(struct sk_buff *skb, int ret = 0; struct nlattr *nest_parms; - nest_parms = nla_nest_start(skb, CTA_TUPLE_IP | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, CTA_TUPLE_IP); if (!nest_parms) goto nla_put_failure; @@ -187,7 +187,7 @@ static int ctnetlink_dump_protoinfo(struct sk_buff *skb, struct nf_conn *ct) if (!l4proto->to_nlattr) return 0; - nest_proto = nla_nest_start(skb, CTA_PROTOINFO | NLA_F_NESTED); + nest_proto = nla_nest_start(skb, CTA_PROTOINFO); if (!nest_proto) goto nla_put_failure; @@ -215,7 +215,7 @@ static int ctnetlink_dump_helpinfo(struct sk_buff *skb, if (!helper) goto out; - nest_helper = nla_nest_start(skb, CTA_HELP | NLA_F_NESTED); + nest_helper = nla_nest_start(skb, CTA_HELP); if (!nest_helper) goto nla_put_failure; if (nla_put_string(skb, CTA_HELP_NAME, helper->name)) @@ -249,7 +249,7 @@ dump_counters(struct sk_buff *skb, struct nf_conn_acct *acct, bytes = atomic64_read(&counter[dir].bytes); } - nest_count = nla_nest_start(skb, attr | NLA_F_NESTED); + nest_count = nla_nest_start(skb, attr); if (!nest_count) goto nla_put_failure; @@ -293,7 +293,7 @@ ctnetlink_dump_timestamp(struct sk_buff *skb, const struct nf_conn *ct) if (!tstamp) return 0; - nest_count = nla_nest_start(skb, CTA_TIMESTAMP | NLA_F_NESTED); + nest_count = nla_nest_start(skb, CTA_TIMESTAMP); if (!nest_count) goto nla_put_failure; @@ -337,7 +337,7 @@ static int ctnetlink_dump_secctx(struct sk_buff *skb, const struct nf_conn *ct) return 0; ret = -1; - nest_secctx = nla_nest_start(skb, CTA_SECCTX | NLA_F_NESTED); + nest_secctx = nla_nest_start(skb, CTA_SECCTX); if (!nest_secctx) goto nla_put_failure; @@ -397,7 +397,7 @@ static int ctnetlink_dump_master(struct sk_buff *skb, const struct nf_conn *ct) if (!(ct->status & IPS_EXPECTED)) return 0; - nest_parms = nla_nest_start(skb, CTA_TUPLE_MASTER | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, CTA_TUPLE_MASTER); if (!nest_parms) goto nla_put_failure; if (ctnetlink_dump_tuples(skb, master_tuple(ct)) < 0) @@ -415,7 +415,7 @@ dump_ct_seq_adj(struct sk_buff *skb, const struct nf_ct_seqadj *seq, int type) { struct nlattr *nest_parms; - nest_parms = nla_nest_start(skb, type | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, type); if (!nest_parms) goto nla_put_failure; @@ -467,7 +467,7 @@ static int ctnetlink_dump_ct_synproxy(struct sk_buff *skb, struct nf_conn *ct) if (!synproxy) return 0; - nest_parms = nla_nest_start(skb, CTA_SYNPROXY | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, CTA_SYNPROXY); if (!nest_parms) goto nla_put_failure; @@ -528,7 +528,7 @@ ctnetlink_fill_info(struct sk_buff *skb, u32 portid, u32 seq, u32 type, zone = nf_ct_zone(ct); - nest_parms = nla_nest_start(skb, CTA_TUPLE_ORIG | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, CTA_TUPLE_ORIG); if (!nest_parms) goto nla_put_failure; if (ctnetlink_dump_tuples(skb, nf_ct_tuple(ct, IP_CT_DIR_ORIGINAL)) < 0) @@ -538,7 +538,7 @@ ctnetlink_fill_info(struct sk_buff *skb, u32 portid, u32 seq, u32 type, goto nla_put_failure; nla_nest_end(skb, nest_parms); - nest_parms = nla_nest_start(skb, CTA_TUPLE_REPLY | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, CTA_TUPLE_REPLY); if (!nest_parms) goto nla_put_failure; if (ctnetlink_dump_tuples(skb, nf_ct_tuple(ct, IP_CT_DIR_REPLY)) < 0) @@ -658,7 +658,7 @@ static size_t ctnetlink_nlmsg_size(const struct nf_conn *ct) + nla_total_size(0) /* CTA_HELP */ + nla_total_size(NF_CT_HELPER_NAME_LEN) /* CTA_HELP_NAME */ + ctnetlink_secctx_size(ct) -#ifdef CONFIG_NF_NAT_NEEDED +#if IS_ENABLED(CONFIG_NF_NAT) + 2 * nla_total_size(0) /* CTA_NAT_SEQ_ADJ_ORIG|REPL */ + 6 * nla_total_size(sizeof(u_int32_t)) /* CTA_NAT_SEQ_OFFSET */ #endif @@ -720,7 +720,7 @@ ctnetlink_conntrack_event(unsigned int events, struct nf_ct_event *item) zone = nf_ct_zone(ct); - nest_parms = nla_nest_start(skb, CTA_TUPLE_ORIG | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, CTA_TUPLE_ORIG); if (!nest_parms) goto nla_put_failure; if (ctnetlink_dump_tuples(skb, nf_ct_tuple(ct, IP_CT_DIR_ORIGINAL)) < 0) @@ -730,7 +730,7 @@ ctnetlink_conntrack_event(unsigned int events, struct nf_ct_event *item) goto nla_put_failure; nla_nest_end(skb, nest_parms); - nest_parms = nla_nest_start(skb, CTA_TUPLE_REPLY | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, CTA_TUPLE_REPLY); if (!nest_parms) goto nla_put_failure; if (ctnetlink_dump_tuples(skb, nf_ct_tuple(ct, IP_CT_DIR_REPLY)) < 0) @@ -1020,12 +1020,12 @@ static int ctnetlink_parse_tuple_ip(struct nlattr *attr, struct nlattr *tb[CTA_IP_MAX+1]; int ret = 0; - ret = nla_parse_nested(tb, CTA_IP_MAX, attr, NULL, NULL); + ret = nla_parse_nested_deprecated(tb, CTA_IP_MAX, attr, NULL, NULL); if (ret < 0) return ret; - ret = nla_validate_nested(attr, CTA_IP_MAX, - cta_ip_nla_policy, NULL); + ret = nla_validate_nested_deprecated(attr, CTA_IP_MAX, + cta_ip_nla_policy, NULL); if (ret) return ret; @@ -1052,8 +1052,8 @@ static int ctnetlink_parse_tuple_proto(struct nlattr *attr, struct nlattr *tb[CTA_PROTO_MAX+1]; int ret = 0; - ret = nla_parse_nested(tb, CTA_PROTO_MAX, attr, proto_nla_policy, - NULL); + ret = nla_parse_nested_deprecated(tb, CTA_PROTO_MAX, attr, + proto_nla_policy, NULL); if (ret < 0) return ret; @@ -1065,8 +1065,9 @@ static int ctnetlink_parse_tuple_proto(struct nlattr *attr, l4proto = nf_ct_l4proto_find(tuple->dst.protonum); if (likely(l4proto->nlattr_to_tuple)) { - ret = nla_validate_nested(attr, CTA_PROTO_MAX, - l4proto->nla_policy, NULL); + ret = nla_validate_nested_deprecated(attr, CTA_PROTO_MAX, + l4proto->nla_policy, + NULL); if (ret == 0) ret = l4proto->nlattr_to_tuple(tb, tuple); } @@ -1129,8 +1130,8 @@ ctnetlink_parse_tuple(const struct nlattr * const cda[], memset(tuple, 0, sizeof(*tuple)); - err = nla_parse_nested(tb, CTA_TUPLE_MAX, cda[type], tuple_nla_policy, - NULL); + err = nla_parse_nested_deprecated(tb, CTA_TUPLE_MAX, cda[type], + tuple_nla_policy, NULL); if (err < 0) return err; @@ -1180,7 +1181,8 @@ static int ctnetlink_parse_help(const struct nlattr *attr, char **helper_name, int err; struct nlattr *tb[CTA_HELP_MAX+1]; - err = nla_parse_nested(tb, CTA_HELP_MAX, attr, help_nla_policy, NULL); + err = nla_parse_nested_deprecated(tb, CTA_HELP_MAX, attr, + help_nla_policy, NULL); if (err < 0) return err; @@ -1498,7 +1500,7 @@ static int ctnetlink_get_ct_unconfirmed(struct net *net, struct sock *ctnl, return -EOPNOTSUPP; } -#ifdef CONFIG_NF_NAT_NEEDED +#if IS_ENABLED(CONFIG_NF_NAT) static int ctnetlink_parse_nat_setup(struct nf_conn *ct, enum nf_nat_manip_type manip, @@ -1590,7 +1592,7 @@ ctnetlink_change_status(struct nf_conn *ct, const struct nlattr * const cda[]) static int ctnetlink_setup_nat(struct nf_conn *ct, const struct nlattr * const cda[]) { -#ifdef CONFIG_NF_NAT_NEEDED +#if IS_ENABLED(CONFIG_NF_NAT) int ret; if (!cda[CTA_NAT_DST] && !cda[CTA_NAT_SRC]) @@ -1721,8 +1723,8 @@ static int ctnetlink_change_protoinfo(struct nf_conn *ct, struct nlattr *tb[CTA_PROTOINFO_MAX+1]; int err = 0; - err = nla_parse_nested(tb, CTA_PROTOINFO_MAX, attr, protoinfo_policy, - NULL); + err = nla_parse_nested_deprecated(tb, CTA_PROTOINFO_MAX, attr, + protoinfo_policy, NULL); if (err < 0) return err; @@ -1745,7 +1747,8 @@ static int change_seq_adj(struct nf_ct_seqadj *seq, int err; struct nlattr *cda[CTA_SEQADJ_MAX+1]; - err = nla_parse_nested(cda, CTA_SEQADJ_MAX, attr, seqadj_policy, NULL); + err = nla_parse_nested_deprecated(cda, CTA_SEQADJ_MAX, attr, + seqadj_policy, NULL); if (err < 0) return err; @@ -1822,8 +1825,9 @@ static int ctnetlink_change_synproxy(struct nf_conn *ct, if (!synproxy) return 0; - err = nla_parse_nested(tb, CTA_SYNPROXY_MAX, cda[CTA_SYNPROXY], - synproxy_policy, NULL); + err = nla_parse_nested_deprecated(tb, CTA_SYNPROXY_MAX, + cda[CTA_SYNPROXY], synproxy_policy, + NULL); if (err < 0) return err; @@ -2373,7 +2377,7 @@ ctnetlink_glue_build_size(const struct nf_conn *ct) + nla_total_size(0) /* CTA_HELP */ + nla_total_size(NF_CT_HELPER_NAME_LEN) /* CTA_HELP_NAME */ + ctnetlink_secctx_size(ct) -#ifdef CONFIG_NF_NAT_NEEDED +#if IS_ENABLED(CONFIG_NF_NAT) + 2 * nla_total_size(0) /* CTA_NAT_SEQ_ADJ_ORIG|REPL */ + 6 * nla_total_size(sizeof(u_int32_t)) /* CTA_NAT_SEQ_OFFSET */ #endif @@ -2400,7 +2404,7 @@ static int __ctnetlink_glue_build(struct sk_buff *skb, struct nf_conn *ct) zone = nf_ct_zone(ct); - nest_parms = nla_nest_start(skb, CTA_TUPLE_ORIG | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, CTA_TUPLE_ORIG); if (!nest_parms) goto nla_put_failure; if (ctnetlink_dump_tuples(skb, nf_ct_tuple(ct, IP_CT_DIR_ORIGINAL)) < 0) @@ -2410,7 +2414,7 @@ static int __ctnetlink_glue_build(struct sk_buff *skb, struct nf_conn *ct) goto nla_put_failure; nla_nest_end(skb, nest_parms); - nest_parms = nla_nest_start(skb, CTA_TUPLE_REPLY | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, CTA_TUPLE_REPLY); if (!nest_parms) goto nla_put_failure; if (ctnetlink_dump_tuples(skb, nf_ct_tuple(ct, IP_CT_DIR_REPLY)) < 0) @@ -2472,7 +2476,7 @@ ctnetlink_glue_build(struct sk_buff *skb, struct nf_conn *ct, { struct nlattr *nest_parms; - nest_parms = nla_nest_start(skb, ct_attr | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, ct_attr); if (!nest_parms) goto nla_put_failure; @@ -2553,7 +2557,8 @@ ctnetlink_glue_parse(const struct nlattr *attr, struct nf_conn *ct) struct nlattr *cda[CTA_MAX+1]; int ret; - ret = nla_parse_nested(cda, CTA_MAX, attr, ct_nla_policy, NULL); + ret = nla_parse_nested_deprecated(cda, CTA_MAX, attr, ct_nla_policy, + NULL); if (ret < 0) return ret; @@ -2586,8 +2591,8 @@ ctnetlink_glue_attach_expect(const struct nlattr *attr, struct nf_conn *ct, struct nf_conntrack_expect *exp; int err; - err = nla_parse_nested(cda, CTA_EXPECT_MAX, attr, exp_nla_policy, - NULL); + err = nla_parse_nested_deprecated(cda, CTA_EXPECT_MAX, attr, + exp_nla_policy, NULL); if (err < 0) return err; @@ -2644,7 +2649,7 @@ static int ctnetlink_exp_dump_tuple(struct sk_buff *skb, { struct nlattr *nest_parms; - nest_parms = nla_nest_start(skb, type | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, type); if (!nest_parms) goto nla_put_failure; if (ctnetlink_dump_tuples(skb, tuple) < 0) @@ -2671,7 +2676,7 @@ static int ctnetlink_exp_dump_mask(struct sk_buff *skb, m.src.u.all = mask->src.u.all; m.dst.protonum = tuple->dst.protonum; - nest_parms = nla_nest_start(skb, CTA_EXPECT_MASK | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, CTA_EXPECT_MASK); if (!nest_parms) goto nla_put_failure; @@ -2722,7 +2727,7 @@ ctnetlink_exp_dump_expect(struct sk_buff *skb, struct nf_conn *master = exp->master; long timeout = ((long)exp->timeout.expires - (long)jiffies) / HZ; struct nf_conn_help *help; -#ifdef CONFIG_NF_NAT_NEEDED +#if IS_ENABLED(CONFIG_NF_NAT) struct nlattr *nest_parms; struct nf_conntrack_tuple nat_tuple = {}; #endif @@ -2740,10 +2745,10 @@ ctnetlink_exp_dump_expect(struct sk_buff *skb, CTA_EXPECT_MASTER) < 0) goto nla_put_failure; -#ifdef CONFIG_NF_NAT_NEEDED +#if IS_ENABLED(CONFIG_NF_NAT) if (!nf_inet_addr_cmp(&exp->saved_addr, &any_addr) || exp->saved_proto.all) { - nest_parms = nla_nest_start(skb, CTA_EXPECT_NAT | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, CTA_EXPECT_NAT); if (!nest_parms) goto nla_put_failure; @@ -3204,13 +3209,13 @@ ctnetlink_parse_expect_nat(const struct nlattr *attr, struct nf_conntrack_expect *exp, u_int8_t u3) { -#ifdef CONFIG_NF_NAT_NEEDED +#if IS_ENABLED(CONFIG_NF_NAT) struct nlattr *tb[CTA_EXPECT_NAT_MAX+1]; struct nf_conntrack_tuple nat_tuple = {}; int err; - err = nla_parse_nested(tb, CTA_EXPECT_NAT_MAX, attr, - exp_nat_nla_policy, NULL); + err = nla_parse_nested_deprecated(tb, CTA_EXPECT_NAT_MAX, attr, + exp_nat_nla_policy, NULL); if (err < 0) return err; diff --git a/net/netfilter/nf_conntrack_proto_dccp.c b/net/netfilter/nf_conntrack_proto_dccp.c index 6fca805875051282f6aefae37565f36dcb4cfa3f..7491aa4c35669d2200eba6006dc093777343b2a0 100644 --- a/net/netfilter/nf_conntrack_proto_dccp.c +++ b/net/netfilter/nf_conntrack_proto_dccp.c @@ -598,7 +598,7 @@ static int dccp_to_nlattr(struct sk_buff *skb, struct nlattr *nla, struct nlattr *nest_parms; spin_lock_bh(&ct->lock); - nest_parms = nla_nest_start(skb, CTA_PROTOINFO_DCCP | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, CTA_PROTOINFO_DCCP); if (!nest_parms) goto nla_put_failure; if (nla_put_u8(skb, CTA_PROTOINFO_DCCP_STATE, ct->proto.dccp.state) || @@ -639,8 +639,8 @@ static int nlattr_to_dccp(struct nlattr *cda[], struct nf_conn *ct) if (!attr) return 0; - err = nla_parse_nested(tb, CTA_PROTOINFO_DCCP_MAX, attr, - dccp_nla_policy, NULL); + err = nla_parse_nested_deprecated(tb, CTA_PROTOINFO_DCCP_MAX, attr, + dccp_nla_policy, NULL); if (err < 0) return err; diff --git a/net/netfilter/nf_conntrack_proto_sctp.c b/net/netfilter/nf_conntrack_proto_sctp.c index a7818101ad806a8b89741ed0d465afb432d62e39..5b8dde2664123a663924f11a6da832ad98c749a8 100644 --- a/net/netfilter/nf_conntrack_proto_sctp.c +++ b/net/netfilter/nf_conntrack_proto_sctp.c @@ -520,7 +520,7 @@ static int sctp_to_nlattr(struct sk_buff *skb, struct nlattr *nla, struct nlattr *nest_parms; spin_lock_bh(&ct->lock); - nest_parms = nla_nest_start(skb, CTA_PROTOINFO_SCTP | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, CTA_PROTOINFO_SCTP); if (!nest_parms) goto nla_put_failure; @@ -563,8 +563,8 @@ static int nlattr_to_sctp(struct nlattr *cda[], struct nf_conn *ct) if (!attr) return 0; - err = nla_parse_nested(tb, CTA_PROTOINFO_SCTP_MAX, attr, - sctp_nla_policy, NULL); + err = nla_parse_nested_deprecated(tb, CTA_PROTOINFO_SCTP_MAX, attr, + sctp_nla_policy, NULL); if (err < 0) return err; diff --git a/net/netfilter/nf_conntrack_proto_tcp.c b/net/netfilter/nf_conntrack_proto_tcp.c index a06875a466a49a34b637137666f6500b4492bc3d..7ba01d8ee1652c472c66ac7ba01cca78a0b755c7 100644 --- a/net/netfilter/nf_conntrack_proto_tcp.c +++ b/net/netfilter/nf_conntrack_proto_tcp.c @@ -1192,7 +1192,7 @@ static int tcp_to_nlattr(struct sk_buff *skb, struct nlattr *nla, struct nf_ct_tcp_flags tmp = {}; spin_lock_bh(&ct->lock); - nest_parms = nla_nest_start(skb, CTA_PROTOINFO_TCP | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, CTA_PROTOINFO_TCP); if (!nest_parms) goto nla_put_failure; @@ -1248,8 +1248,8 @@ static int nlattr_to_tcp(struct nlattr *cda[], struct nf_conn *ct) if (!pattr) return 0; - err = nla_parse_nested(tb, CTA_PROTOINFO_TCP_MAX, pattr, - tcp_nla_policy, NULL); + err = nla_parse_nested_deprecated(tb, CTA_PROTOINFO_TCP_MAX, pattr, + tcp_nla_policy, NULL); if (err < 0) return err; diff --git a/net/netfilter/nf_conntrack_sane.c b/net/netfilter/nf_conntrack_sane.c index 5072ff96ab3342e2183067d39333c161cd0b1848..83306648dd0fe5cf305508b44f2e599246fa3262 100644 --- a/net/netfilter/nf_conntrack_sane.c +++ b/net/netfilter/nf_conntrack_sane.c @@ -30,10 +30,12 @@ #include #include +#define HELPER_NAME "sane" + MODULE_LICENSE("GPL"); MODULE_AUTHOR("Michal Schmidt "); MODULE_DESCRIPTION("SANE connection tracking helper"); -MODULE_ALIAS_NFCT_HELPER("sane"); +MODULE_ALIAS_NFCT_HELPER(HELPER_NAME); static char *sane_buffer; @@ -195,12 +197,12 @@ static int __init nf_conntrack_sane_init(void) /* FIXME should be configurable whether IPv4 and IPv6 connections are tracked or not - YK */ for (i = 0; i < ports_c; i++) { - nf_ct_helper_init(&sane[2 * i], AF_INET, IPPROTO_TCP, "sane", - SANE_PORT, ports[i], ports[i], + nf_ct_helper_init(&sane[2 * i], AF_INET, IPPROTO_TCP, + HELPER_NAME, SANE_PORT, ports[i], ports[i], &sane_exp_policy, 0, help, NULL, THIS_MODULE); - nf_ct_helper_init(&sane[2 * i + 1], AF_INET6, IPPROTO_TCP, "sane", - SANE_PORT, ports[i], ports[i], + nf_ct_helper_init(&sane[2 * i + 1], AF_INET6, IPPROTO_TCP, + HELPER_NAME, SANE_PORT, ports[i], ports[i], &sane_exp_policy, 0, help, NULL, THIS_MODULE); } diff --git a/net/netfilter/nf_conntrack_sip.c b/net/netfilter/nf_conntrack_sip.c index 39fcc1ed18f3501b3120fc9aeffbe44e27fda933..c30c883c370bb04889010d5654807529c0a6eac2 100644 --- a/net/netfilter/nf_conntrack_sip.c +++ b/net/netfilter/nf_conntrack_sip.c @@ -30,11 +30,13 @@ #include #include +#define HELPER_NAME "sip" + MODULE_LICENSE("GPL"); MODULE_AUTHOR("Christian Hentschel "); MODULE_DESCRIPTION("SIP connection tracking helper"); MODULE_ALIAS("ip_conntrack_sip"); -MODULE_ALIAS_NFCT_HELPER("sip"); +MODULE_ALIAS_NFCT_HELPER(HELPER_NAME); #define MAX_PORTS 8 static unsigned short ports[MAX_PORTS]; @@ -928,7 +930,7 @@ static int set_expected_rtp_rtcp(struct sk_buff *skb, unsigned int protoff, nfct_help(exp->master)->helper != nfct_help(ct)->helper || exp->class != class) break; -#ifdef CONFIG_NF_NAT_NEEDED +#if IS_ENABLED(CONFIG_NF_NAT) if (!direct_rtp && (!nf_inet_addr_cmp(&exp->saved_addr, &exp->tuple.dst.u3) || exp->saved_proto.udp.port != exp->tuple.dst.u.udp.port) && @@ -1669,21 +1671,21 @@ static int __init nf_conntrack_sip_init(void) ports[ports_c++] = SIP_PORT; for (i = 0; i < ports_c; i++) { - nf_ct_helper_init(&sip[4 * i], AF_INET, IPPROTO_UDP, "sip", - SIP_PORT, ports[i], i, sip_exp_policy, - SIP_EXPECT_MAX, sip_help_udp, + nf_ct_helper_init(&sip[4 * i], AF_INET, IPPROTO_UDP, + HELPER_NAME, SIP_PORT, ports[i], i, + sip_exp_policy, SIP_EXPECT_MAX, sip_help_udp, NULL, THIS_MODULE); - nf_ct_helper_init(&sip[4 * i + 1], AF_INET, IPPROTO_TCP, "sip", - SIP_PORT, ports[i], i, sip_exp_policy, - SIP_EXPECT_MAX, sip_help_tcp, + nf_ct_helper_init(&sip[4 * i + 1], AF_INET, IPPROTO_TCP, + HELPER_NAME, SIP_PORT, ports[i], i, + sip_exp_policy, SIP_EXPECT_MAX, sip_help_tcp, NULL, THIS_MODULE); - nf_ct_helper_init(&sip[4 * i + 2], AF_INET6, IPPROTO_UDP, "sip", - SIP_PORT, ports[i], i, sip_exp_policy, - SIP_EXPECT_MAX, sip_help_udp, + nf_ct_helper_init(&sip[4 * i + 2], AF_INET6, IPPROTO_UDP, + HELPER_NAME, SIP_PORT, ports[i], i, + sip_exp_policy, SIP_EXPECT_MAX, sip_help_udp, NULL, THIS_MODULE); - nf_ct_helper_init(&sip[4 * i + 3], AF_INET6, IPPROTO_TCP, "sip", - SIP_PORT, ports[i], i, sip_exp_policy, - SIP_EXPECT_MAX, sip_help_tcp, + nf_ct_helper_init(&sip[4 * i + 3], AF_INET6, IPPROTO_TCP, + HELPER_NAME, SIP_PORT, ports[i], i, + sip_exp_policy, SIP_EXPECT_MAX, sip_help_tcp, NULL, THIS_MODULE); } diff --git a/net/netfilter/nf_conntrack_standalone.c b/net/netfilter/nf_conntrack_standalone.c index c2ae14c720b4e44780ea954988d9e830eaf4f7c4..e0d392cb3075ab8ee82f69f6fd03f002f9901316 100644 --- a/net/netfilter/nf_conntrack_standalone.c +++ b/net/netfilter/nf_conntrack_standalone.c @@ -511,6 +511,8 @@ static void nf_conntrack_standalone_fini_proc(struct net *net) /* Log invalid packets of a given protocol */ static int log_invalid_proto_min __read_mostly; static int log_invalid_proto_max __read_mostly = 255; +static int zero; +static int one = 1; /* size the user *wants to set */ static unsigned int nf_conntrack_htable_size_user __read_mostly; @@ -624,9 +626,11 @@ static struct ctl_table nf_ct_sysctl_table[] = { [NF_SYSCTL_CT_CHECKSUM] = { .procname = "nf_conntrack_checksum", .data = &init_net.ct.sysctl_checksum, - .maxlen = sizeof(unsigned int), + .maxlen = sizeof(int), .mode = 0644, - .proc_handler = proc_dointvec, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + .extra2 = &one, }, [NF_SYSCTL_CT_LOG_INVALID] = { .procname = "nf_conntrack_log_invalid", @@ -647,33 +651,41 @@ static struct ctl_table nf_ct_sysctl_table[] = { [NF_SYSCTL_CT_ACCT] = { .procname = "nf_conntrack_acct", .data = &init_net.ct.sysctl_acct, - .maxlen = sizeof(unsigned int), + .maxlen = sizeof(int), .mode = 0644, - .proc_handler = proc_dointvec, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + .extra2 = &one, }, [NF_SYSCTL_CT_HELPER] = { .procname = "nf_conntrack_helper", .data = &init_net.ct.sysctl_auto_assign_helper, - .maxlen = sizeof(unsigned int), + .maxlen = sizeof(int), .mode = 0644, - .proc_handler = proc_dointvec, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + .extra2 = &one, }, #ifdef CONFIG_NF_CONNTRACK_EVENTS [NF_SYSCTL_CT_EVENTS] = { .procname = "nf_conntrack_events", .data = &init_net.ct.sysctl_events, - .maxlen = sizeof(unsigned int), + .maxlen = sizeof(int), .mode = 0644, - .proc_handler = proc_dointvec, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + .extra2 = &one, }, #endif #ifdef CONFIG_NF_CONNTRACK_TIMESTAMP [NF_SYSCTL_CT_TIMESTAMP] = { .procname = "nf_conntrack_timestamp", .data = &init_net.ct.sysctl_tstamp, - .maxlen = sizeof(unsigned int), + .maxlen = sizeof(int), .mode = 0644, - .proc_handler = proc_dointvec, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + .extra2 = &one, }, #endif [NF_SYSCTL_CT_PROTO_TIMEOUT_GENERIC] = { @@ -744,15 +756,19 @@ static struct ctl_table nf_ct_sysctl_table[] = { }, [NF_SYSCTL_CT_PROTO_TCP_LOOSE] = { .procname = "nf_conntrack_tcp_loose", - .maxlen = sizeof(unsigned int), + .maxlen = sizeof(int), .mode = 0644, - .proc_handler = proc_dointvec, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + .extra2 = &one, }, [NF_SYSCTL_CT_PROTO_TCP_LIBERAL] = { .procname = "nf_conntrack_tcp_be_liberal", - .maxlen = sizeof(unsigned int), + .maxlen = sizeof(int), .mode = 0644, - .proc_handler = proc_dointvec, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + .extra2 = &one, }, [NF_SYSCTL_CT_PROTO_TCP_MAX_RETRANS] = { .procname = "nf_conntrack_tcp_max_retrans", @@ -887,7 +903,9 @@ static struct ctl_table nf_ct_sysctl_table[] = { .procname = "nf_conntrack_dccp_loose", .maxlen = sizeof(int), .mode = 0644, - .proc_handler = proc_dointvec, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + .extra2 = &one, }, #endif #ifdef CONFIG_NF_CT_PROTO_GRE diff --git a/net/netfilter/nf_conntrack_tftp.c b/net/netfilter/nf_conntrack_tftp.c index 548b673b3625cc55eec26bb096253b592e21c188..6977cb91ae9aa91d9a365ad1c39f8821f17510b4 100644 --- a/net/netfilter/nf_conntrack_tftp.c +++ b/net/netfilter/nf_conntrack_tftp.c @@ -20,11 +20,13 @@ #include #include +#define HELPER_NAME "tftp" + MODULE_AUTHOR("Magnus Boden "); MODULE_DESCRIPTION("TFTP connection tracking helper"); MODULE_LICENSE("GPL"); MODULE_ALIAS("ip_conntrack_tftp"); -MODULE_ALIAS_NFCT_HELPER("tftp"); +MODULE_ALIAS_NFCT_HELPER(HELPER_NAME); #define MAX_PORTS 8 static unsigned short ports[MAX_PORTS]; @@ -119,12 +121,14 @@ static int __init nf_conntrack_tftp_init(void) ports[ports_c++] = TFTP_PORT; for (i = 0; i < ports_c; i++) { - nf_ct_helper_init(&tftp[2 * i], AF_INET, IPPROTO_UDP, "tftp", - TFTP_PORT, ports[i], i, &tftp_exp_policy, - 0, tftp_help, NULL, THIS_MODULE); - nf_ct_helper_init(&tftp[2 * i + 1], AF_INET6, IPPROTO_UDP, "tftp", - TFTP_PORT, ports[i], i, &tftp_exp_policy, - 0, tftp_help, NULL, THIS_MODULE); + nf_ct_helper_init(&tftp[2 * i], AF_INET, IPPROTO_UDP, + HELPER_NAME, TFTP_PORT, ports[i], i, + &tftp_exp_policy, 0, tftp_help, NULL, + THIS_MODULE); + nf_ct_helper_init(&tftp[2 * i + 1], AF_INET6, IPPROTO_UDP, + HELPER_NAME, TFTP_PORT, ports[i], i, + &tftp_exp_policy, 0, tftp_help, NULL, + THIS_MODULE); } ret = nf_conntrack_helpers_register(tftp, ports_c * 2); diff --git a/net/netfilter/nf_conntrack_timeout.c b/net/netfilter/nf_conntrack_timeout.c index 91fbd183da2dcbfa5ea22bdb4c92938ef89b458c..edac8ea4436dd6b1b4f59399e0306e5c920048db 100644 --- a/net/netfilter/nf_conntrack_timeout.c +++ b/net/netfilter/nf_conntrack_timeout.c @@ -48,6 +48,95 @@ void nf_ct_untimeout(struct net *net, struct nf_ct_timeout *timeout) } EXPORT_SYMBOL_GPL(nf_ct_untimeout); +static void __nf_ct_timeout_put(struct nf_ct_timeout *timeout) +{ + typeof(nf_ct_timeout_put_hook) timeout_put; + + timeout_put = rcu_dereference(nf_ct_timeout_put_hook); + if (timeout_put) + timeout_put(timeout); +} + +int nf_ct_set_timeout(struct net *net, struct nf_conn *ct, + u8 l3num, u8 l4num, const char *timeout_name) +{ + typeof(nf_ct_timeout_find_get_hook) timeout_find_get; + struct nf_ct_timeout *timeout; + struct nf_conn_timeout *timeout_ext; + const char *errmsg = NULL; + int ret = 0; + + rcu_read_lock(); + timeout_find_get = rcu_dereference(nf_ct_timeout_find_get_hook); + if (!timeout_find_get) { + ret = -ENOENT; + errmsg = "Timeout policy base is empty"; + goto out; + } + + timeout = timeout_find_get(net, timeout_name); + if (!timeout) { + ret = -ENOENT; + pr_info_ratelimited("No such timeout policy \"%s\"\n", + timeout_name); + goto out; + } + + if (timeout->l3num != l3num) { + ret = -EINVAL; + pr_info_ratelimited("Timeout policy `%s' can only be used by " + "L%d protocol number %d\n", + timeout_name, 3, timeout->l3num); + goto err_put_timeout; + } + /* Make sure the timeout policy matches any existing protocol tracker, + * otherwise default to generic. + */ + if (timeout->l4proto->l4proto != l4num) { + ret = -EINVAL; + pr_info_ratelimited("Timeout policy `%s' can only be used by " + "L%d protocol number %d\n", + timeout_name, 4, timeout->l4proto->l4proto); + goto err_put_timeout; + } + timeout_ext = nf_ct_timeout_ext_add(ct, timeout, GFP_ATOMIC); + if (!timeout_ext) { + ret = -ENOMEM; + goto err_put_timeout; + } + + rcu_read_unlock(); + return ret; + +err_put_timeout: + __nf_ct_timeout_put(timeout); +out: + rcu_read_unlock(); + if (errmsg) + pr_info_ratelimited("%s\n", errmsg); + return ret; +} +EXPORT_SYMBOL_GPL(nf_ct_set_timeout); + +void nf_ct_destroy_timeout(struct nf_conn *ct) +{ + struct nf_conn_timeout *timeout_ext; + typeof(nf_ct_timeout_put_hook) timeout_put; + + rcu_read_lock(); + timeout_put = rcu_dereference(nf_ct_timeout_put_hook); + + if (timeout_put) { + timeout_ext = nf_ct_timeout_find(ct); + if (timeout_ext) { + timeout_put(timeout_ext->timeout); + RCU_INIT_POINTER(timeout_ext->timeout, NULL); + } + } + rcu_read_unlock(); +} +EXPORT_SYMBOL_GPL(nf_ct_destroy_timeout); + static const struct nf_ct_ext_type timeout_extend = { .len = sizeof(struct nf_conn_timeout), .align = __alignof__(struct nf_conn_timeout), diff --git a/net/netfilter/nf_flow_table_ip.c b/net/netfilter/nf_flow_table_ip.c index 1d291a51cd45b74e5f70b34c304e519f2c41875b..6452550d187fecd919c483cea627ed1adbf274cc 100644 --- a/net/netfilter/nf_flow_table_ip.c +++ b/net/netfilter/nf_flow_table_ip.c @@ -235,13 +235,10 @@ nf_flow_offload_ip_hook(void *priv, struct sk_buff *skb, if (tuplehash == NULL) return NF_ACCEPT; - outdev = dev_get_by_index_rcu(state->net, tuplehash->tuple.oifidx); - if (!outdev) - return NF_ACCEPT; - dir = tuplehash->tuple.dir; flow = container_of(tuplehash, struct flow_offload, tuplehash[dir]); rt = (struct rtable *)flow->tuplehash[dir].tuple.dst_cache; + outdev = rt->dst.dev; if (unlikely(nf_flow_exceeds_mtu(skb, flow->tuplehash[dir].tuple.mtu)) && (ip_hdr(skb)->frag_off & htons(IP_DF)) != 0) @@ -452,13 +449,10 @@ nf_flow_offload_ipv6_hook(void *priv, struct sk_buff *skb, if (tuplehash == NULL) return NF_ACCEPT; - outdev = dev_get_by_index_rcu(state->net, tuplehash->tuple.oifidx); - if (!outdev) - return NF_ACCEPT; - dir = tuplehash->tuple.dir; flow = container_of(tuplehash, struct flow_offload, tuplehash[dir]); rt = (struct rt6_info *)flow->tuplehash[dir].tuple.dst_cache; + outdev = rt->dst.dev; if (unlikely(nf_flow_exceeds_mtu(skb, flow->tuplehash[dir].tuple.mtu))) return NF_ACCEPT; diff --git a/net/netfilter/nf_internals.h b/net/netfilter/nf_internals.h index e15779fd58e3a5c8425c0f8ff5296f2bfdf338c8..d6c43902ebd766c1653befa5929659e5c4658606 100644 --- a/net/netfilter/nf_internals.h +++ b/net/netfilter/nf_internals.h @@ -7,9 +7,6 @@ #include /* nf_queue.c */ -int nf_queue(struct sk_buff *skb, struct nf_hook_state *state, - const struct nf_hook_entries *entries, unsigned int index, - unsigned int verdict); void nf_queue_nf_hook_drop(struct net *net); /* nf_log.c */ diff --git a/net/netfilter/nf_nat_amanda.c b/net/netfilter/nf_nat_amanda.c index e4d61a7a52589d85d23185bba1919dfd188e62c2..4e59416ea7090fd75473e7cb1f37b49523825106 100644 --- a/net/netfilter/nf_nat_amanda.c +++ b/net/netfilter/nf_nat_amanda.c @@ -19,10 +19,15 @@ #include #include +#define NAT_HELPER_NAME "amanda" + MODULE_AUTHOR("Brian J. Murrell "); MODULE_DESCRIPTION("Amanda NAT helper"); MODULE_LICENSE("GPL"); -MODULE_ALIAS("ip_nat_amanda"); +MODULE_ALIAS_NF_NAT_HELPER(NAT_HELPER_NAME); + +static struct nf_conntrack_nat_helper nat_helper_amanda = + NF_CT_NAT_HELPER_INIT(NAT_HELPER_NAME); static unsigned int help(struct sk_buff *skb, enum ip_conntrack_info ctinfo, @@ -74,6 +79,7 @@ static unsigned int help(struct sk_buff *skb, static void __exit nf_nat_amanda_fini(void) { + nf_nat_helper_unregister(&nat_helper_amanda); RCU_INIT_POINTER(nf_nat_amanda_hook, NULL); synchronize_rcu(); } @@ -81,6 +87,7 @@ static void __exit nf_nat_amanda_fini(void) static int __init nf_nat_amanda_init(void) { BUG_ON(nf_nat_amanda_hook != NULL); + nf_nat_helper_register(&nat_helper_amanda); RCU_INIT_POINTER(nf_nat_amanda_hook, help); return 0; } diff --git a/net/netfilter/nf_nat_core.c b/net/netfilter/nf_nat_core.c index 000952719adfdf49bf35a53dd800c6cecf45c14f..cd94481e6c0795fd55337ff98f467b78336c87ea 100644 --- a/net/netfilter/nf_nat_core.c +++ b/net/netfilter/nf_nat_core.c @@ -890,8 +890,8 @@ static int nfnetlink_parse_nat_proto(struct nlattr *attr, struct nlattr *tb[CTA_PROTONAT_MAX+1]; int err; - err = nla_parse_nested(tb, CTA_PROTONAT_MAX, attr, - protonat_nla_policy, NULL); + err = nla_parse_nested_deprecated(tb, CTA_PROTONAT_MAX, attr, + protonat_nla_policy, NULL); if (err < 0) return err; @@ -949,7 +949,8 @@ nfnetlink_parse_nat(const struct nlattr *nat, memset(range, 0, sizeof(*range)); - err = nla_parse_nested(tb, CTA_NAT_MAX, nat, nat_nla_policy, NULL); + err = nla_parse_nested_deprecated(tb, CTA_NAT_MAX, nat, + nat_nla_policy, NULL); if (err < 0) return err; @@ -1014,7 +1015,7 @@ static struct nf_ct_helper_expectfn follow_master_nat = { .expectfn = nf_nat_follow_master, }; -int nf_nat_register_fn(struct net *net, const struct nf_hook_ops *ops, +int nf_nat_register_fn(struct net *net, u8 pf, const struct nf_hook_ops *ops, const struct nf_hook_ops *orig_nat_ops, unsigned int ops_count) { struct nat_net *nat_net = net_generic(net, nat_net_id); @@ -1024,14 +1025,12 @@ int nf_nat_register_fn(struct net *net, const struct nf_hook_ops *ops, struct nf_hook_ops *nat_ops; int i, ret; - if (WARN_ON_ONCE(ops->pf >= ARRAY_SIZE(nat_net->nat_proto_net))) + if (WARN_ON_ONCE(pf >= ARRAY_SIZE(nat_net->nat_proto_net))) return -EINVAL; - nat_proto_net = &nat_net->nat_proto_net[ops->pf]; + nat_proto_net = &nat_net->nat_proto_net[pf]; for (i = 0; i < ops_count; i++) { - if (WARN_ON(orig_nat_ops[i].pf != ops->pf)) - return -EINVAL; if (orig_nat_ops[i].hooknum == hooknum) { hooknum = i; break; @@ -1091,8 +1090,8 @@ int nf_nat_register_fn(struct net *net, const struct nf_hook_ops *ops, return ret; } -void nf_nat_unregister_fn(struct net *net, const struct nf_hook_ops *ops, - unsigned int ops_count) +void nf_nat_unregister_fn(struct net *net, u8 pf, const struct nf_hook_ops *ops, + unsigned int ops_count) { struct nat_net *nat_net = net_generic(net, nat_net_id); struct nf_nat_hooks_net *nat_proto_net; @@ -1101,10 +1100,10 @@ void nf_nat_unregister_fn(struct net *net, const struct nf_hook_ops *ops, int hooknum = ops->hooknum; int i; - if (ops->pf >= ARRAY_SIZE(nat_net->nat_proto_net)) + if (pf >= ARRAY_SIZE(nat_net->nat_proto_net)) return; - nat_proto_net = &nat_net->nat_proto_net[ops->pf]; + nat_proto_net = &nat_net->nat_proto_net[pf]; mutex_lock(&nf_nat_proto_mutex); if (WARN_ON(nat_proto_net->users == 0)) diff --git a/net/netfilter/nf_nat_ftp.c b/net/netfilter/nf_nat_ftp.c index 5063cbf1689c879dbdf62a5ccd52df48e9bbede8..0ea6b1bc52dea993c575591227dcc1e1c23c91aa 100644 --- a/net/netfilter/nf_nat_ftp.c +++ b/net/netfilter/nf_nat_ftp.c @@ -21,13 +21,18 @@ #include #include +#define NAT_HELPER_NAME "ftp" + MODULE_LICENSE("GPL"); MODULE_AUTHOR("Rusty Russell "); MODULE_DESCRIPTION("ftp NAT helper"); -MODULE_ALIAS("ip_nat_ftp"); +MODULE_ALIAS_NF_NAT_HELPER(NAT_HELPER_NAME); /* FIXME: Time out? --RR */ +static struct nf_conntrack_nat_helper nat_helper_ftp = + NF_CT_NAT_HELPER_INIT(NAT_HELPER_NAME); + static int nf_nat_ftp_fmt_cmd(struct nf_conn *ct, enum nf_ct_ftp_type type, char *buffer, size_t buflen, union nf_inet_addr *addr, u16 port) @@ -124,6 +129,7 @@ out: static void __exit nf_nat_ftp_fini(void) { + nf_nat_helper_unregister(&nat_helper_ftp); RCU_INIT_POINTER(nf_nat_ftp_hook, NULL); synchronize_rcu(); } @@ -131,6 +137,7 @@ static void __exit nf_nat_ftp_fini(void) static int __init nf_nat_ftp_init(void) { BUG_ON(nf_nat_ftp_hook != NULL); + nf_nat_helper_register(&nat_helper_ftp); RCU_INIT_POINTER(nf_nat_ftp_hook, nf_nat_ftp); return 0; } diff --git a/net/netfilter/nf_nat_irc.c b/net/netfilter/nf_nat_irc.c index 3aa35a43100d576d6927637f03a7999ab4ac9dad..d87cbe5e03ec9c3d6aa67e7973a988e15fa78364 100644 --- a/net/netfilter/nf_nat_irc.c +++ b/net/netfilter/nf_nat_irc.c @@ -23,10 +23,15 @@ #include #include +#define NAT_HELPER_NAME "irc" + MODULE_AUTHOR("Harald Welte "); MODULE_DESCRIPTION("IRC (DCC) NAT helper"); MODULE_LICENSE("GPL"); -MODULE_ALIAS("ip_nat_irc"); +MODULE_ALIAS_NF_NAT_HELPER(NAT_HELPER_NAME); + +static struct nf_conntrack_nat_helper nat_helper_irc = + NF_CT_NAT_HELPER_INIT(NAT_HELPER_NAME); static unsigned int help(struct sk_buff *skb, enum ip_conntrack_info ctinfo, @@ -96,6 +101,7 @@ static unsigned int help(struct sk_buff *skb, static void __exit nf_nat_irc_fini(void) { + nf_nat_helper_unregister(&nat_helper_irc); RCU_INIT_POINTER(nf_nat_irc_hook, NULL); synchronize_rcu(); } @@ -103,6 +109,7 @@ static void __exit nf_nat_irc_fini(void) static int __init nf_nat_irc_init(void) { BUG_ON(nf_nat_irc_hook != NULL); + nf_nat_helper_register(&nat_helper_irc); RCU_INIT_POINTER(nf_nat_irc_hook, help); return 0; } diff --git a/net/netfilter/nf_nat_masquerade.c b/net/netfilter/nf_nat_masquerade.c index d85c4d902e7b1deca67975abdf13b3bf8326351f..8e8a65d46345b292c8fcac2fc7fcdfd1be1c5557 100644 --- a/net/netfilter/nf_nat_masquerade.c +++ b/net/netfilter/nf_nat_masquerade.c @@ -7,12 +7,10 @@ #include #include -#include -#include +#include static DEFINE_MUTEX(masq_mutex); -static unsigned int masq_refcnt4 __read_mostly; -static unsigned int masq_refcnt6 __read_mostly; +static unsigned int masq_refcnt __read_mostly; unsigned int nf_nat_masquerade_ipv4(struct sk_buff *skb, unsigned int hooknum, @@ -137,56 +135,6 @@ static struct notifier_block masq_inet_notifier = { .notifier_call = masq_inet_event, }; -int nf_nat_masquerade_ipv4_register_notifier(void) -{ - int ret = 0; - - mutex_lock(&masq_mutex); - if (WARN_ON_ONCE(masq_refcnt4 == UINT_MAX)) { - ret = -EOVERFLOW; - goto out_unlock; - } - - /* check if the notifier was already set */ - if (++masq_refcnt4 > 1) - goto out_unlock; - - /* Register for device down reports */ - ret = register_netdevice_notifier(&masq_dev_notifier); - if (ret) - goto err_dec; - /* Register IP address change reports */ - ret = register_inetaddr_notifier(&masq_inet_notifier); - if (ret) - goto err_unregister; - - mutex_unlock(&masq_mutex); - return ret; - -err_unregister: - unregister_netdevice_notifier(&masq_dev_notifier); -err_dec: - masq_refcnt4--; -out_unlock: - mutex_unlock(&masq_mutex); - return ret; -} -EXPORT_SYMBOL_GPL(nf_nat_masquerade_ipv4_register_notifier); - -void nf_nat_masquerade_ipv4_unregister_notifier(void) -{ - mutex_lock(&masq_mutex); - /* check if the notifier still has clients */ - if (--masq_refcnt4 > 0) - goto out_unlock; - - unregister_netdevice_notifier(&masq_dev_notifier); - unregister_inetaddr_notifier(&masq_inet_notifier); -out_unlock: - mutex_unlock(&masq_mutex); -} -EXPORT_SYMBOL_GPL(nf_nat_masquerade_ipv4_unregister_notifier); - #if IS_ENABLED(CONFIG_IPV6) static atomic_t v6_worker_count __read_mostly; @@ -322,44 +270,68 @@ static struct notifier_block masq_inet6_notifier = { .notifier_call = masq_inet6_event, }; -int nf_nat_masquerade_ipv6_register_notifier(void) +static int nf_nat_masquerade_ipv6_register_notifier(void) +{ + return register_inet6addr_notifier(&masq_inet6_notifier); +} +#else +static inline int nf_nat_masquerade_ipv6_register_notifier(void) { return 0; } +#endif + +int nf_nat_masquerade_inet_register_notifiers(void) { int ret = 0; mutex_lock(&masq_mutex); - if (WARN_ON_ONCE(masq_refcnt6 == UINT_MAX)) { + if (WARN_ON_ONCE(masq_refcnt == UINT_MAX)) { ret = -EOVERFLOW; goto out_unlock; } - /* check if the notifier is already set */ - if (++masq_refcnt6 > 1) + /* check if the notifier was already set */ + if (++masq_refcnt > 1) goto out_unlock; - ret = register_inet6addr_notifier(&masq_inet6_notifier); + /* Register for device down reports */ + ret = register_netdevice_notifier(&masq_dev_notifier); if (ret) goto err_dec; + /* Register IP address change reports */ + ret = register_inetaddr_notifier(&masq_inet_notifier); + if (ret) + goto err_unregister; + + ret = nf_nat_masquerade_ipv6_register_notifier(); + if (ret) + goto err_unreg_inet; mutex_unlock(&masq_mutex); return ret; +err_unreg_inet: + unregister_inetaddr_notifier(&masq_inet_notifier); +err_unregister: + unregister_netdevice_notifier(&masq_dev_notifier); err_dec: - masq_refcnt6--; + masq_refcnt--; out_unlock: mutex_unlock(&masq_mutex); return ret; } -EXPORT_SYMBOL_GPL(nf_nat_masquerade_ipv6_register_notifier); +EXPORT_SYMBOL_GPL(nf_nat_masquerade_inet_register_notifiers); -void nf_nat_masquerade_ipv6_unregister_notifier(void) +void nf_nat_masquerade_inet_unregister_notifiers(void) { mutex_lock(&masq_mutex); - /* check if the notifier still has clients */ - if (--masq_refcnt6 > 0) + /* check if the notifiers still have clients */ + if (--masq_refcnt > 0) goto out_unlock; + unregister_netdevice_notifier(&masq_dev_notifier); + unregister_inetaddr_notifier(&masq_inet_notifier); +#if IS_ENABLED(CONFIG_IPV6) unregister_inet6addr_notifier(&masq_inet6_notifier); +#endif out_unlock: mutex_unlock(&masq_mutex); } -EXPORT_SYMBOL_GPL(nf_nat_masquerade_ipv6_unregister_notifier); -#endif +EXPORT_SYMBOL_GPL(nf_nat_masquerade_inet_unregister_notifiers); diff --git a/net/netfilter/nf_nat_proto.c b/net/netfilter/nf_nat_proto.c index 62743da3004fa631e6034dc6eb2ff98fcc1b9cd5..84f5c90a7f21f5eb9c23bed36fd31b3159623618 100644 --- a/net/netfilter/nf_nat_proto.c +++ b/net/netfilter/nf_nat_proto.c @@ -725,7 +725,7 @@ nf_nat_ipv4_local_fn(void *priv, struct sk_buff *skb, return ret; } -static const struct nf_hook_ops nf_nat_ipv4_ops[] = { +const struct nf_hook_ops nf_nat_ipv4_ops[] = { /* Before packet filtering, change destination */ { .hook = nf_nat_ipv4_in, @@ -758,13 +758,14 @@ static const struct nf_hook_ops nf_nat_ipv4_ops[] = { int nf_nat_ipv4_register_fn(struct net *net, const struct nf_hook_ops *ops) { - return nf_nat_register_fn(net, ops, nf_nat_ipv4_ops, ARRAY_SIZE(nf_nat_ipv4_ops)); + return nf_nat_register_fn(net, ops->pf, ops, nf_nat_ipv4_ops, + ARRAY_SIZE(nf_nat_ipv4_ops)); } EXPORT_SYMBOL_GPL(nf_nat_ipv4_register_fn); void nf_nat_ipv4_unregister_fn(struct net *net, const struct nf_hook_ops *ops) { - nf_nat_unregister_fn(net, ops, ARRAY_SIZE(nf_nat_ipv4_ops)); + nf_nat_unregister_fn(net, ops->pf, ops, ARRAY_SIZE(nf_nat_ipv4_ops)); } EXPORT_SYMBOL_GPL(nf_nat_ipv4_unregister_fn); @@ -925,20 +926,6 @@ nf_nat_ipv6_out(void *priv, struct sk_buff *skb, return ret; } -static int nat_route_me_harder(struct net *net, struct sk_buff *skb) -{ -#ifdef CONFIG_IPV6_MODULE - const struct nf_ipv6_ops *v6_ops = nf_get_ipv6_ops(); - - if (!v6_ops) - return -EHOSTUNREACH; - - return v6_ops->route_me_harder(net, skb); -#else - return ip6_route_me_harder(net, skb); -#endif -} - static unsigned int nf_nat_ipv6_local_fn(void *priv, struct sk_buff *skb, const struct nf_hook_state *state) @@ -958,7 +945,7 @@ nf_nat_ipv6_local_fn(void *priv, struct sk_buff *skb, if (!nf_inet_addr_cmp(&ct->tuplehash[dir].tuple.dst.u3, &ct->tuplehash[!dir].tuple.src.u3)) { - err = nat_route_me_harder(state->net, skb); + err = nf_ip6_route_me_harder(state->net, skb); if (err < 0) ret = NF_DROP_ERR(err); } @@ -977,7 +964,7 @@ nf_nat_ipv6_local_fn(void *priv, struct sk_buff *skb, return ret; } -static const struct nf_hook_ops nf_nat_ipv6_ops[] = { +const struct nf_hook_ops nf_nat_ipv6_ops[] = { /* Before packet filtering, change destination */ { .hook = nf_nat_ipv6_in, @@ -1010,14 +997,44 @@ static const struct nf_hook_ops nf_nat_ipv6_ops[] = { int nf_nat_ipv6_register_fn(struct net *net, const struct nf_hook_ops *ops) { - return nf_nat_register_fn(net, ops, nf_nat_ipv6_ops, + return nf_nat_register_fn(net, ops->pf, ops, nf_nat_ipv6_ops, ARRAY_SIZE(nf_nat_ipv6_ops)); } EXPORT_SYMBOL_GPL(nf_nat_ipv6_register_fn); void nf_nat_ipv6_unregister_fn(struct net *net, const struct nf_hook_ops *ops) { - nf_nat_unregister_fn(net, ops, ARRAY_SIZE(nf_nat_ipv6_ops)); + nf_nat_unregister_fn(net, ops->pf, ops, ARRAY_SIZE(nf_nat_ipv6_ops)); } EXPORT_SYMBOL_GPL(nf_nat_ipv6_unregister_fn); #endif /* CONFIG_IPV6 */ + +#if defined(CONFIG_NF_TABLES_INET) && IS_ENABLED(CONFIG_NFT_NAT) +int nf_nat_inet_register_fn(struct net *net, const struct nf_hook_ops *ops) +{ + int ret; + + if (WARN_ON_ONCE(ops->pf != NFPROTO_INET)) + return -EINVAL; + + ret = nf_nat_register_fn(net, NFPROTO_IPV6, ops, nf_nat_ipv6_ops, + ARRAY_SIZE(nf_nat_ipv6_ops)); + if (ret) + return ret; + + ret = nf_nat_register_fn(net, NFPROTO_IPV4, ops, nf_nat_ipv4_ops, + ARRAY_SIZE(nf_nat_ipv4_ops)); + if (ret) + nf_nat_ipv6_unregister_fn(net, ops); + + return ret; +} +EXPORT_SYMBOL_GPL(nf_nat_inet_register_fn); + +void nf_nat_inet_unregister_fn(struct net *net, const struct nf_hook_ops *ops) +{ + nf_nat_unregister_fn(net, NFPROTO_IPV4, ops, ARRAY_SIZE(nf_nat_ipv4_ops)); + nf_nat_unregister_fn(net, NFPROTO_IPV6, ops, ARRAY_SIZE(nf_nat_ipv6_ops)); +} +EXPORT_SYMBOL_GPL(nf_nat_inet_unregister_fn); +#endif /* NFT INET NAT */ diff --git a/net/netfilter/nf_nat_sip.c b/net/netfilter/nf_nat_sip.c index aa1be643d7a09510c90102c3b2550c8f1ad51023..464387b3600fbb9b0bc8cd14f72383d098bf1368 100644 --- a/net/netfilter/nf_nat_sip.c +++ b/net/netfilter/nf_nat_sip.c @@ -24,11 +24,15 @@ #include #include +#define NAT_HELPER_NAME "sip" + MODULE_LICENSE("GPL"); MODULE_AUTHOR("Christian Hentschel "); MODULE_DESCRIPTION("SIP NAT helper"); -MODULE_ALIAS("ip_nat_sip"); +MODULE_ALIAS_NF_NAT_HELPER(NAT_HELPER_NAME); +static struct nf_conntrack_nat_helper nat_helper_sip = + NF_CT_NAT_HELPER_INIT(NAT_HELPER_NAME); static unsigned int mangle_packet(struct sk_buff *skb, unsigned int protoff, unsigned int dataoff, @@ -656,8 +660,8 @@ static struct nf_ct_helper_expectfn sip_nat = { static void __exit nf_nat_sip_fini(void) { + nf_nat_helper_unregister(&nat_helper_sip); RCU_INIT_POINTER(nf_nat_sip_hooks, NULL); - nf_ct_helper_expectfn_unregister(&sip_nat); synchronize_rcu(); } @@ -675,6 +679,7 @@ static const struct nf_nat_sip_hooks sip_hooks = { static int __init nf_nat_sip_init(void) { BUG_ON(nf_nat_sip_hooks != NULL); + nf_nat_helper_register(&nat_helper_sip); RCU_INIT_POINTER(nf_nat_sip_hooks, &sip_hooks); nf_ct_helper_expectfn_register(&sip_nat); return 0; diff --git a/net/netfilter/nf_nat_tftp.c b/net/netfilter/nf_nat_tftp.c index 7f67e1d5310d3c12790ac6f70ca207d79d71d9b3..e633b3863e33409aebba3031348e64d7b6c034bd 100644 --- a/net/netfilter/nf_nat_tftp.c +++ b/net/netfilter/nf_nat_tftp.c @@ -13,10 +13,15 @@ #include #include +#define NAT_HELPER_NAME "tftp" + MODULE_AUTHOR("Magnus Boden "); MODULE_DESCRIPTION("TFTP NAT helper"); MODULE_LICENSE("GPL"); -MODULE_ALIAS("ip_nat_tftp"); +MODULE_ALIAS_NF_NAT_HELPER(NAT_HELPER_NAME); + +static struct nf_conntrack_nat_helper nat_helper_tftp = + NF_CT_NAT_HELPER_INIT(NAT_HELPER_NAME); static unsigned int help(struct sk_buff *skb, enum ip_conntrack_info ctinfo, @@ -37,6 +42,7 @@ static unsigned int help(struct sk_buff *skb, static void __exit nf_nat_tftp_fini(void) { + nf_nat_helper_unregister(&nat_helper_tftp); RCU_INIT_POINTER(nf_nat_tftp_hook, NULL); synchronize_rcu(); } @@ -44,6 +50,7 @@ static void __exit nf_nat_tftp_fini(void) static int __init nf_nat_tftp_init(void) { BUG_ON(nf_nat_tftp_hook != NULL); + nf_nat_helper_register(&nat_helper_tftp); RCU_INIT_POINTER(nf_nat_tftp_hook, help); return 0; } diff --git a/net/netfilter/nf_queue.c b/net/netfilter/nf_queue.c index a36a77bae1d6d2c79459dc72826c066a442f60c3..9dc1d6e04946f59ea34377a14834bd711c4cd092 100644 --- a/net/netfilter/nf_queue.c +++ b/net/netfilter/nf_queue.c @@ -240,6 +240,7 @@ int nf_queue(struct sk_buff *skb, struct nf_hook_state *state, return 0; } +EXPORT_SYMBOL_GPL(nf_queue); static unsigned int nf_iterate(struct sk_buff *skb, struct nf_hook_state *state, diff --git a/net/netfilter/nf_tables_api.c b/net/netfilter/nf_tables_api.c index 1606eaa5ae0da368f4a692264456e18dfe27ec8d..d98416e83d4ef2299d24c341c76642cef5f8e15d 100644 --- a/net/netfilter/nf_tables_api.c +++ b/net/netfilter/nf_tables_api.c @@ -53,7 +53,6 @@ static const struct rhashtable_params nft_chain_ht_params = { .hashfn = nft_chain_hash, .obj_hashfn = nft_chain_hash_obj, .obj_cmpfn = nft_chain_hash_cmp, - .locks_mul = 1, .automatic_shrinking = true, }; @@ -1201,7 +1200,7 @@ static int nft_dump_stats(struct sk_buff *skb, struct nft_stats __percpu *stats) total.pkts += pkts; total.bytes += bytes; } - nest = nla_nest_start(skb, NFTA_CHAIN_COUNTERS); + nest = nla_nest_start_noflag(skb, NFTA_CHAIN_COUNTERS); if (nest == NULL) goto nla_put_failure; @@ -1249,7 +1248,7 @@ static int nf_tables_fill_chain_info(struct sk_buff *skb, struct net *net, const struct nf_hook_ops *ops = &basechain->ops; struct nlattr *nest; - nest = nla_nest_start(skb, NFTA_CHAIN_HOOK); + nest = nla_nest_start_noflag(skb, NFTA_CHAIN_HOOK); if (nest == NULL) goto nla_put_failure; if (nla_put_be32(skb, NFTA_HOOK_HOOKNUM, htonl(ops->hooknum))) @@ -1421,8 +1420,8 @@ static struct nft_stats __percpu *nft_stats_alloc(const struct nlattr *attr) struct nft_stats *stats; int err; - err = nla_parse_nested(tb, NFTA_COUNTER_MAX, attr, nft_counter_policy, - NULL); + err = nla_parse_nested_deprecated(tb, NFTA_COUNTER_MAX, attr, + nft_counter_policy, NULL); if (err < 0) return ERR_PTR(err); @@ -1526,8 +1525,9 @@ static int nft_chain_parse_hook(struct net *net, lockdep_assert_held(&net->nft.commit_mutex); lockdep_nfnl_nft_mutex_not_held(); - err = nla_parse_nested(ha, NFTA_HOOK_MAX, nla[NFTA_CHAIN_HOOK], - nft_hook_policy, NULL); + err = nla_parse_nested_deprecated(ha, NFTA_HOOK_MAX, + nla[NFTA_CHAIN_HOOK], + nft_hook_policy, NULL); if (err < 0) return err; @@ -2060,7 +2060,8 @@ static int nf_tables_fill_expr_info(struct sk_buff *skb, goto nla_put_failure; if (expr->ops->dump) { - struct nlattr *data = nla_nest_start(skb, NFTA_EXPR_DATA); + struct nlattr *data = nla_nest_start_noflag(skb, + NFTA_EXPR_DATA); if (data == NULL) goto nla_put_failure; if (expr->ops->dump(skb, expr) < 0) @@ -2079,7 +2080,7 @@ int nft_expr_dump(struct sk_buff *skb, unsigned int attr, { struct nlattr *nest; - nest = nla_nest_start(skb, attr); + nest = nla_nest_start_noflag(skb, attr); if (!nest) goto nla_put_failure; if (nf_tables_fill_expr_info(skb, expr) < 0) @@ -2105,7 +2106,8 @@ static int nf_tables_expr_parse(const struct nft_ctx *ctx, struct nlattr *tb[NFTA_EXPR_MAX + 1]; int err; - err = nla_parse_nested(tb, NFTA_EXPR_MAX, nla, nft_expr_policy, NULL); + err = nla_parse_nested_deprecated(tb, NFTA_EXPR_MAX, nla, + nft_expr_policy, NULL); if (err < 0) return err; @@ -2114,8 +2116,9 @@ static int nf_tables_expr_parse(const struct nft_ctx *ctx, return PTR_ERR(type); if (tb[NFTA_EXPR_DATA]) { - err = nla_parse_nested(info->tb, type->maxattr, - tb[NFTA_EXPR_DATA], type->policy, NULL); + err = nla_parse_nested_deprecated(info->tb, type->maxattr, + tb[NFTA_EXPR_DATA], + type->policy, NULL); if (err < 0) goto err1; } else @@ -2290,7 +2293,7 @@ static int nf_tables_fill_rule_info(struct sk_buff *skb, struct net *net, goto nla_put_failure; } - list = nla_nest_start(skb, NFTA_RULE_EXPRESSIONS); + list = nla_nest_start_noflag(skb, NFTA_RULE_EXPRESSIONS); if (list == NULL) goto nla_put_failure; nft_rule_for_each_expr(expr, next, rule) { @@ -3194,9 +3197,7 @@ static int nf_msecs_to_jiffies64(const struct nlattr *nla, u64 *result) static __be64 nf_jiffies64_to_msecs(u64 input) { - u64 ms = jiffies64_to_nsecs(input); - - return cpu_to_be64(div_u64(ms, NSEC_PER_MSEC)); + return cpu_to_be64(jiffies64_to_msecs(input)); } static int nf_tables_fill_set(struct sk_buff *skb, const struct nft_ctx *ctx, @@ -3261,7 +3262,7 @@ static int nf_tables_fill_set(struct sk_buff *skb, const struct nft_ctx *ctx, if (nla_put(skb, NFTA_SET_USERDATA, set->udlen, set->udata)) goto nla_put_failure; - desc = nla_nest_start(skb, NFTA_SET_DESC); + desc = nla_nest_start_noflag(skb, NFTA_SET_DESC); if (desc == NULL) goto nla_put_failure; if (set->size && @@ -3439,15 +3440,14 @@ err: return err; } -static int nf_tables_set_desc_parse(const struct nft_ctx *ctx, - struct nft_set_desc *desc, +static int nf_tables_set_desc_parse(struct nft_set_desc *desc, const struct nlattr *nla) { struct nlattr *da[NFTA_SET_DESC_MAX + 1]; int err; - err = nla_parse_nested(da, NFTA_SET_DESC_MAX, nla, - nft_set_desc_policy, NULL); + err = nla_parse_nested_deprecated(da, NFTA_SET_DESC_MAX, nla, + nft_set_desc_policy, NULL); if (err < 0) return err; @@ -3566,7 +3566,7 @@ static int nf_tables_newset(struct net *net, struct sock *nlsk, policy = ntohl(nla_get_be32(nla[NFTA_SET_POLICY])); if (nla[NFTA_SET_DESC] != NULL) { - err = nf_tables_set_desc_parse(&ctx, &desc, nla[NFTA_SET_DESC]); + err = nf_tables_set_desc_parse(&desc, nla[NFTA_SET_DESC]); if (err < 0) return err; } @@ -3786,8 +3786,8 @@ bind: } EXPORT_SYMBOL_GPL(nf_tables_bind_set); -void nf_tables_unbind_set(const struct nft_ctx *ctx, struct nft_set *set, - struct nft_set_binding *binding, bool event) +static void nf_tables_unbind_set(const struct nft_ctx *ctx, struct nft_set *set, + struct nft_set_binding *binding, bool event) { list_del_rcu(&binding->list); @@ -3798,7 +3798,6 @@ void nf_tables_unbind_set(const struct nft_ctx *ctx, struct nft_set *set, GFP_KERNEL); } } -EXPORT_SYMBOL_GPL(nf_tables_unbind_set); void nf_tables_deactivate_set(const struct nft_ctx *ctx, struct nft_set *set, struct nft_set_binding *binding, @@ -3913,7 +3912,7 @@ static int nf_tables_fill_setelem(struct sk_buff *skb, unsigned char *b = skb_tail_pointer(skb); struct nlattr *nest; - nest = nla_nest_start(skb, NFTA_LIST_ELEM); + nest = nla_nest_start_noflag(skb, NFTA_LIST_ELEM); if (nest == NULL) goto nla_put_failure; @@ -4057,7 +4056,7 @@ static int nf_tables_dump_set(struct sk_buff *skb, struct netlink_callback *cb) if (nla_put_string(skb, NFTA_SET_ELEM_LIST_SET, set->name)) goto nla_put_failure; - nest = nla_nest_start(skb, NFTA_SET_ELEM_LIST_ELEMENTS); + nest = nla_nest_start_noflag(skb, NFTA_SET_ELEM_LIST_ELEMENTS); if (nest == NULL) goto nla_put_failure; @@ -4129,7 +4128,7 @@ static int nf_tables_fill_setelem_info(struct sk_buff *skb, if (nla_put_string(skb, NFTA_SET_NAME, set->name)) goto nla_put_failure; - nest = nla_nest_start(skb, NFTA_SET_ELEM_LIST_ELEMENTS); + nest = nla_nest_start_noflag(skb, NFTA_SET_ELEM_LIST_ELEMENTS); if (nest == NULL) goto nla_put_failure; @@ -4174,8 +4173,8 @@ static int nft_get_set_elem(struct nft_ctx *ctx, struct nft_set *set, void *priv; int err; - err = nla_parse_nested(nla, NFTA_SET_ELEM_MAX, attr, - nft_set_elem_policy, NULL); + err = nla_parse_nested_deprecated(nla, NFTA_SET_ELEM_MAX, attr, + nft_set_elem_policy, NULL); if (err < 0) return err; @@ -4406,8 +4405,8 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set, u8 ulen; int err; - err = nla_parse_nested(nla, NFTA_SET_ELEM_MAX, attr, - nft_set_elem_policy, NULL); + err = nla_parse_nested_deprecated(nla, NFTA_SET_ELEM_MAX, attr, + nft_set_elem_policy, NULL); if (err < 0) return err; @@ -4700,8 +4699,8 @@ static int nft_del_setelem(struct nft_ctx *ctx, struct nft_set *set, void *priv; int err; - err = nla_parse_nested(nla, NFTA_SET_ELEM_MAX, attr, - nft_set_elem_policy, NULL); + err = nla_parse_nested_deprecated(nla, NFTA_SET_ELEM_MAX, attr, + nft_set_elem_policy, NULL); if (err < 0) goto err1; @@ -4975,8 +4974,8 @@ static struct nft_object *nft_obj_init(const struct nft_ctx *ctx, goto err1; if (attr) { - err = nla_parse_nested(tb, type->maxattr, attr, type->policy, - NULL); + err = nla_parse_nested_deprecated(tb, type->maxattr, attr, + type->policy, NULL); if (err < 0) goto err2; } else { @@ -5019,7 +5018,7 @@ static int nft_object_dump(struct sk_buff *skb, unsigned int attr, { struct nlattr *nest; - nest = nla_nest_start(skb, attr); + nest = nla_nest_start_noflag(skb, attr); if (!nest) goto nla_put_failure; if (obj->ops->dump(skb, obj, reset) < 0) @@ -5552,8 +5551,8 @@ static int nf_tables_flowtable_parse_hook(const struct nft_ctx *ctx, int hooknum, priority; int err, n = 0, i; - err = nla_parse_nested(tb, NFTA_FLOWTABLE_HOOK_MAX, attr, - nft_flowtable_hook_policy, NULL); + err = nla_parse_nested_deprecated(tb, NFTA_FLOWTABLE_HOOK_MAX, attr, + nft_flowtable_hook_policy, NULL); if (err < 0) return err; @@ -5836,14 +5835,14 @@ static int nf_tables_fill_flowtable_info(struct sk_buff *skb, struct net *net, NFTA_FLOWTABLE_PAD)) goto nla_put_failure; - nest = nla_nest_start(skb, NFTA_FLOWTABLE_HOOK); + nest = nla_nest_start_noflag(skb, NFTA_FLOWTABLE_HOOK); if (!nest) goto nla_put_failure; if (nla_put_be32(skb, NFTA_FLOWTABLE_HOOK_NUM, htonl(flowtable->hooknum)) || nla_put_be32(skb, NFTA_FLOWTABLE_HOOK_PRIORITY, htonl(flowtable->priority))) goto nla_put_failure; - nest_devs = nla_nest_start(skb, NFTA_FLOWTABLE_HOOK_DEVS); + nest_devs = nla_nest_start_noflag(skb, NFTA_FLOWTABLE_HOOK_DEVS); if (!nest_devs) goto nla_put_failure; @@ -7210,8 +7209,8 @@ static int nft_verdict_init(const struct nft_ctx *ctx, struct nft_data *data, struct nft_chain *chain; int err; - err = nla_parse_nested(tb, NFTA_VERDICT_MAX, nla, nft_verdict_policy, - NULL); + err = nla_parse_nested_deprecated(tb, NFTA_VERDICT_MAX, nla, + nft_verdict_policy, NULL); if (err < 0) return err; @@ -7269,7 +7268,7 @@ int nft_verdict_dump(struct sk_buff *skb, int type, const struct nft_verdict *v) { struct nlattr *nest; - nest = nla_nest_start(skb, type); + nest = nla_nest_start_noflag(skb, type); if (!nest) goto nla_put_failure; @@ -7341,7 +7340,8 @@ int nft_data_init(const struct nft_ctx *ctx, struct nlattr *tb[NFTA_DATA_MAX + 1]; int err; - err = nla_parse_nested(tb, NFTA_DATA_MAX, nla, nft_data_policy, NULL); + err = nla_parse_nested_deprecated(tb, NFTA_DATA_MAX, nla, + nft_data_policy, NULL); if (err < 0) return err; @@ -7382,7 +7382,7 @@ int nft_data_dump(struct sk_buff *skb, int attr, const struct nft_data *data, struct nlattr *nest; int err; - nest = nla_nest_start(skb, attr); + nest = nla_nest_start_noflag(skb, attr); if (nest == NULL) return -1; @@ -7534,6 +7534,7 @@ static int __init nf_tables_module_init(void) if (err < 0) goto err5; + nft_chain_route_init(); return err; err5: rhltable_destroy(&nft_objname_ht); @@ -7553,6 +7554,7 @@ static void __exit nf_tables_module_exit(void) nfnetlink_subsys_unregister(&nf_tables_subsys); unregister_netdevice_notifier(&nf_tables_flowtable_notifier); nft_chain_filter_fini(); + nft_chain_route_fini(); unregister_pernet_subsys(&nf_tables_net_ops); cancel_work_sync(&trans_destroy_work); rcu_barrier(); diff --git a/net/netfilter/nf_tables_set_core.c b/net/netfilter/nf_tables_set_core.c index 814789644bd3af697377c70fe16db78877453c75..a9fce8d1005150348ef00b635a9c4d5983b7ddba 100644 --- a/net/netfilter/nf_tables_set_core.c +++ b/net/netfilter/nf_tables_set_core.c @@ -1,4 +1,5 @@ /* SPDX-License-Identifier: GPL-2.0 */ +#include #include static int __init nf_tables_set_module_init(void) diff --git a/net/netfilter/nfnetlink.c b/net/netfilter/nfnetlink.c index 916913454624f2740212c62d1c0ce61bc8ae6f73..92077d4591090426ae6fd9335064e95214e76829 100644 --- a/net/netfilter/nfnetlink.c +++ b/net/netfilter/nfnetlink.c @@ -206,8 +206,9 @@ replay: return -ENOMEM; } - err = nla_parse(cda, ss->cb[cb_id].attr_count, attr, attrlen, - ss->cb[cb_id].policy, extack); + err = nla_parse_deprecated(cda, ss->cb[cb_id].attr_count, + attr, attrlen, + ss->cb[cb_id].policy, extack); if (err < 0) { rcu_read_unlock(); return err; @@ -421,8 +422,10 @@ replay: goto ack; } - err = nla_parse(cda, ss->cb[cb_id].attr_count, attr, - attrlen, ss->cb[cb_id].policy, NULL); + err = nla_parse_deprecated(cda, + ss->cb[cb_id].attr_count, + attr, attrlen, + ss->cb[cb_id].policy, NULL); if (err < 0) goto ack; @@ -520,8 +523,8 @@ static void nfnetlink_rcv_skb_batch(struct sk_buff *skb, struct nlmsghdr *nlh) if (skb->len < NLMSG_HDRLEN + sizeof(struct nfgenmsg)) return; - err = nla_parse(cda, NFNL_BATCH_MAX, attr, attrlen, nfnl_batch_policy, - NULL); + err = nla_parse_deprecated(cda, NFNL_BATCH_MAX, attr, attrlen, + nfnl_batch_policy, NULL); if (err < 0) { netlink_ack(skb, nlh, err, NULL); return; diff --git a/net/netfilter/nfnetlink_acct.c b/net/netfilter/nfnetlink_acct.c index 8fa8bf7c48e642344af5c695f998457aae1c6e7c..02c877432d71feae4b002d35d07313862562a58b 100644 --- a/net/netfilter/nfnetlink_acct.c +++ b/net/netfilter/nfnetlink_acct.c @@ -248,8 +248,8 @@ static int nfnl_acct_start(struct netlink_callback *cb) if (!attr) return 0; - err = nla_parse_nested(tb, NFACCT_FILTER_MAX, attr, filter_policy, - NULL); + err = nla_parse_nested_deprecated(tb, NFACCT_FILTER_MAX, attr, + filter_policy, NULL); if (err < 0) return err; diff --git a/net/netfilter/nfnetlink_cthelper.c b/net/netfilter/nfnetlink_cthelper.c index e5d27b2e4ebac9d7256ad53657167e3b87052f94..17eb473a626b903b9701b036c3704b9463f9e2c5 100644 --- a/net/netfilter/nfnetlink_cthelper.c +++ b/net/netfilter/nfnetlink_cthelper.c @@ -78,8 +78,8 @@ nfnl_cthelper_parse_tuple(struct nf_conntrack_tuple *tuple, int err; struct nlattr *tb[NFCTH_TUPLE_MAX+1]; - err = nla_parse_nested(tb, NFCTH_TUPLE_MAX, attr, - nfnl_cthelper_tuple_pol, NULL); + err = nla_parse_nested_deprecated(tb, NFCTH_TUPLE_MAX, attr, + nfnl_cthelper_tuple_pol, NULL); if (err < 0) return err; @@ -139,8 +139,8 @@ nfnl_cthelper_expect_policy(struct nf_conntrack_expect_policy *expect_policy, int err; struct nlattr *tb[NFCTH_POLICY_MAX+1]; - err = nla_parse_nested(tb, NFCTH_POLICY_MAX, attr, - nfnl_cthelper_expect_pol, NULL); + err = nla_parse_nested_deprecated(tb, NFCTH_POLICY_MAX, attr, + nfnl_cthelper_expect_pol, NULL); if (err < 0) return err; @@ -176,8 +176,9 @@ nfnl_cthelper_parse_expect_policy(struct nf_conntrack_helper *helper, struct nlattr *tb[NFCTH_POLICY_SET_MAX+1]; unsigned int class_max; - ret = nla_parse_nested(tb, NFCTH_POLICY_SET_MAX, attr, - nfnl_cthelper_expect_policy_set, NULL); + ret = nla_parse_nested_deprecated(tb, NFCTH_POLICY_SET_MAX, attr, + nfnl_cthelper_expect_policy_set, + NULL); if (ret < 0) return ret; @@ -289,8 +290,8 @@ nfnl_cthelper_update_policy_one(const struct nf_conntrack_expect_policy *policy, struct nlattr *tb[NFCTH_POLICY_MAX + 1]; int err; - err = nla_parse_nested(tb, NFCTH_POLICY_MAX, attr, - nfnl_cthelper_expect_pol, NULL); + err = nla_parse_nested_deprecated(tb, NFCTH_POLICY_MAX, attr, + nfnl_cthelper_expect_pol, NULL); if (err < 0) return err; @@ -361,8 +362,9 @@ static int nfnl_cthelper_update_policy(struct nf_conntrack_helper *helper, unsigned int class_max; int err; - err = nla_parse_nested(tb, NFCTH_POLICY_SET_MAX, attr, - nfnl_cthelper_expect_policy_set, NULL); + err = nla_parse_nested_deprecated(tb, NFCTH_POLICY_SET_MAX, attr, + nfnl_cthelper_expect_policy_set, + NULL); if (err < 0) return err; @@ -462,7 +464,7 @@ nfnl_cthelper_dump_tuple(struct sk_buff *skb, { struct nlattr *nest_parms; - nest_parms = nla_nest_start(skb, NFCTH_TUPLE | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, NFCTH_TUPLE); if (nest_parms == NULL) goto nla_put_failure; @@ -487,7 +489,7 @@ nfnl_cthelper_dump_policy(struct sk_buff *skb, int i; struct nlattr *nest_parms1, *nest_parms2; - nest_parms1 = nla_nest_start(skb, NFCTH_POLICY | NLA_F_NESTED); + nest_parms1 = nla_nest_start(skb, NFCTH_POLICY); if (nest_parms1 == NULL) goto nla_put_failure; @@ -496,8 +498,7 @@ nfnl_cthelper_dump_policy(struct sk_buff *skb, goto nla_put_failure; for (i = 0; i < helper->expect_class_max + 1; i++) { - nest_parms2 = nla_nest_start(skb, - (NFCTH_POLICY_SET+i) | NLA_F_NESTED); + nest_parms2 = nla_nest_start(skb, (NFCTH_POLICY_SET + i)); if (nest_parms2 == NULL) goto nla_put_failure; diff --git a/net/netfilter/nfnetlink_cttimeout.c b/net/netfilter/nfnetlink_cttimeout.c index c69b11ca5aadb1a075c55f1dfdf10ef8d1edcf25..427b411c57394e0e1c851e512a5a2ee3399bae2a 100644 --- a/net/netfilter/nfnetlink_cttimeout.c +++ b/net/netfilter/nfnetlink_cttimeout.c @@ -59,8 +59,11 @@ ctnl_timeout_parse_policy(void *timeout, if (!tb) return -ENOMEM; - ret = nla_parse_nested(tb, l4proto->ctnl_timeout.nlattr_max, attr, - l4proto->ctnl_timeout.nla_policy, NULL); + ret = nla_parse_nested_deprecated(tb, + l4proto->ctnl_timeout.nlattr_max, + attr, + l4proto->ctnl_timeout.nla_policy, + NULL); if (ret < 0) goto err; @@ -184,7 +187,7 @@ ctnl_timeout_fill_info(struct sk_buff *skb, u32 portid, u32 seq, u32 type, htonl(refcount_read(&timeout->refcnt)))) goto nla_put_failure; - nest_parms = nla_nest_start(skb, CTA_TIMEOUT_DATA | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, CTA_TIMEOUT_DATA); if (!nest_parms) goto nla_put_failure; @@ -401,7 +404,7 @@ cttimeout_default_fill_info(struct net *net, struct sk_buff *skb, u32 portid, nla_put_u8(skb, CTA_TIMEOUT_L4PROTO, l4proto->l4proto)) goto nla_put_failure; - nest_parms = nla_nest_start(skb, CTA_TIMEOUT_DATA | NLA_F_NESTED); + nest_parms = nla_nest_start(skb, CTA_TIMEOUT_DATA); if (!nest_parms) goto nla_put_failure; diff --git a/net/netfilter/nfnetlink_osf.c b/net/netfilter/nfnetlink_osf.c index 1f1d90c1716b5bd697b18870e0bb0a7f9d7e63ec..7b827bcb412c90c90dcf030bad8f445c59396d08 100644 --- a/net/netfilter/nfnetlink_osf.c +++ b/net/netfilter/nfnetlink_osf.c @@ -255,9 +255,9 @@ nf_osf_match(const struct sk_buff *skb, u_int8_t family, } EXPORT_SYMBOL_GPL(nf_osf_match); -const char *nf_osf_find(const struct sk_buff *skb, - const struct list_head *nf_osf_fingers, - const int ttl_check) +bool nf_osf_find(const struct sk_buff *skb, + const struct list_head *nf_osf_fingers, + const int ttl_check, struct nf_osf_data *data) { const struct iphdr *ip = ip_hdr(skb); const struct nf_osf_user_finger *f; @@ -265,24 +265,24 @@ const char *nf_osf_find(const struct sk_buff *skb, const struct nf_osf_finger *kf; struct nf_osf_hdr_ctx ctx; const struct tcphdr *tcp; - const char *genre = NULL; memset(&ctx, 0, sizeof(ctx)); tcp = nf_osf_hdr_ctx_init(&ctx, skb, ip, opts); if (!tcp) - return NULL; + return false; list_for_each_entry_rcu(kf, &nf_osf_fingers[ctx.df], finger_entry) { f = &kf->finger; if (!nf_osf_match_one(skb, f, ttl_check, &ctx)) continue; - genre = f->genre; + data->genre = f->genre; + data->version = f->version; break; } - return genre; + return true; } EXPORT_SYMBOL_GPL(nf_osf_find); diff --git a/net/netfilter/nfnetlink_queue.c b/net/netfilter/nfnetlink_queue.c index e057b2961d313cd426f2f2d37ed7e1a40c101174..27dac47b29c2964f8db3552555aa4a1e91094c72 100644 --- a/net/netfilter/nfnetlink_queue.c +++ b/net/netfilter/nfnetlink_queue.c @@ -351,7 +351,7 @@ static int nfqnl_put_bridge(struct nf_queue_entry *entry, struct sk_buff *skb) if (skb_vlan_tag_present(entskb)) { struct nlattr *nest; - nest = nla_nest_start(skb, NFQA_VLAN | NLA_F_NESTED); + nest = nla_nest_start(skb, NFQA_VLAN); if (!nest) goto nla_put_failure; @@ -1139,8 +1139,9 @@ static int nfqa_parse_bridge(struct nf_queue_entry *entry, struct nlattr *tb[NFQA_VLAN_MAX + 1]; int err; - err = nla_parse_nested(tb, NFQA_VLAN_MAX, nfqa[NFQA_VLAN], - nfqa_vlan_policy, NULL); + err = nla_parse_nested_deprecated(tb, NFQA_VLAN_MAX, + nfqa[NFQA_VLAN], + nfqa_vlan_policy, NULL); if (err < 0) return err; diff --git a/net/netfilter/nft_chain_nat.c b/net/netfilter/nft_chain_nat.c index ee4852088d509a154091f41e213d3c5a91c094c4..2f89bde3c61cb93690238c8c53ccad3556806417 100644 --- a/net/netfilter/nft_chain_nat.c +++ b/net/netfilter/nft_chain_nat.c @@ -74,6 +74,36 @@ static const struct nft_chain_type nft_chain_nat_ipv6 = { }; #endif +#ifdef CONFIG_NF_TABLES_INET +static int nft_nat_inet_reg(struct net *net, const struct nf_hook_ops *ops) +{ + return nf_nat_inet_register_fn(net, ops); +} + +static void nft_nat_inet_unreg(struct net *net, const struct nf_hook_ops *ops) +{ + nf_nat_inet_unregister_fn(net, ops); +} + +static const struct nft_chain_type nft_chain_nat_inet = { + .name = "nat", + .type = NFT_CHAIN_T_NAT, + .family = NFPROTO_INET, + .hook_mask = (1 << NF_INET_PRE_ROUTING) | + (1 << NF_INET_LOCAL_IN) | + (1 << NF_INET_LOCAL_OUT) | + (1 << NF_INET_POST_ROUTING), + .hooks = { + [NF_INET_PRE_ROUTING] = nft_nat_do_chain, + [NF_INET_LOCAL_IN] = nft_nat_do_chain, + [NF_INET_LOCAL_OUT] = nft_nat_do_chain, + [NF_INET_POST_ROUTING] = nft_nat_do_chain, + }, + .ops_register = nft_nat_inet_reg, + .ops_unregister = nft_nat_inet_unreg, +}; +#endif + static int __init nft_chain_nat_init(void) { #ifdef CONFIG_NF_TABLES_IPV6 @@ -82,6 +112,9 @@ static int __init nft_chain_nat_init(void) #ifdef CONFIG_NF_TABLES_IPV4 nft_register_chain_type(&nft_chain_nat_ipv4); #endif +#ifdef CONFIG_NF_TABLES_INET + nft_register_chain_type(&nft_chain_nat_inet); +#endif return 0; } @@ -94,6 +127,9 @@ static void __exit nft_chain_nat_exit(void) #ifdef CONFIG_NF_TABLES_IPV6 nft_unregister_chain_type(&nft_chain_nat_ipv6); #endif +#ifdef CONFIG_NF_TABLES_INET + nft_unregister_chain_type(&nft_chain_nat_inet); +#endif } module_init(nft_chain_nat_init); diff --git a/net/netfilter/nft_chain_route.c b/net/netfilter/nft_chain_route.c new file mode 100644 index 0000000000000000000000000000000000000000..8826bbe71136c86bd1020e0598bb4509aa793421 --- /dev/null +++ b/net/netfilter/nft_chain_route.c @@ -0,0 +1,169 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#ifdef CONFIG_NF_TABLES_IPV4 +static unsigned int nf_route_table_hook4(void *priv, + struct sk_buff *skb, + const struct nf_hook_state *state) +{ + const struct iphdr *iph; + struct nft_pktinfo pkt; + __be32 saddr, daddr; + unsigned int ret; + u32 mark; + int err; + u8 tos; + + nft_set_pktinfo(&pkt, skb, state); + nft_set_pktinfo_ipv4(&pkt, skb); + + mark = skb->mark; + iph = ip_hdr(skb); + saddr = iph->saddr; + daddr = iph->daddr; + tos = iph->tos; + + ret = nft_do_chain(&pkt, priv); + if (ret == NF_ACCEPT) { + iph = ip_hdr(skb); + + if (iph->saddr != saddr || + iph->daddr != daddr || + skb->mark != mark || + iph->tos != tos) { + err = ip_route_me_harder(state->net, skb, RTN_UNSPEC); + if (err < 0) + ret = NF_DROP_ERR(err); + } + } + return ret; +} + +static const struct nft_chain_type nft_chain_route_ipv4 = { + .name = "route", + .type = NFT_CHAIN_T_ROUTE, + .family = NFPROTO_IPV4, + .hook_mask = (1 << NF_INET_LOCAL_OUT), + .hooks = { + [NF_INET_LOCAL_OUT] = nf_route_table_hook4, + }, +}; +#endif + +#ifdef CONFIG_NF_TABLES_IPV6 +static unsigned int nf_route_table_hook6(void *priv, + struct sk_buff *skb, + const struct nf_hook_state *state) +{ + struct in6_addr saddr, daddr; + struct nft_pktinfo pkt; + u32 mark, flowlabel; + unsigned int ret; + u8 hop_limit; + int err; + + nft_set_pktinfo(&pkt, skb, state); + nft_set_pktinfo_ipv6(&pkt, skb); + + /* save source/dest address, mark, hoplimit, flowlabel, priority */ + memcpy(&saddr, &ipv6_hdr(skb)->saddr, sizeof(saddr)); + memcpy(&daddr, &ipv6_hdr(skb)->daddr, sizeof(daddr)); + mark = skb->mark; + hop_limit = ipv6_hdr(skb)->hop_limit; + + /* flowlabel and prio (includes version, which shouldn't change either)*/ + flowlabel = *((u32 *)ipv6_hdr(skb)); + + ret = nft_do_chain(&pkt, priv); + if (ret == NF_ACCEPT && + (memcmp(&ipv6_hdr(skb)->saddr, &saddr, sizeof(saddr)) || + memcmp(&ipv6_hdr(skb)->daddr, &daddr, sizeof(daddr)) || + skb->mark != mark || + ipv6_hdr(skb)->hop_limit != hop_limit || + flowlabel != *((u32 *)ipv6_hdr(skb)))) { + err = nf_ip6_route_me_harder(state->net, skb); + if (err < 0) + ret = NF_DROP_ERR(err); + } + + return ret; +} + +static const struct nft_chain_type nft_chain_route_ipv6 = { + .name = "route", + .type = NFT_CHAIN_T_ROUTE, + .family = NFPROTO_IPV6, + .hook_mask = (1 << NF_INET_LOCAL_OUT), + .hooks = { + [NF_INET_LOCAL_OUT] = nf_route_table_hook6, + }, +}; +#endif + +#ifdef CONFIG_NF_TABLES_INET +static unsigned int nf_route_table_inet(void *priv, + struct sk_buff *skb, + const struct nf_hook_state *state) +{ + struct nft_pktinfo pkt; + + switch (state->pf) { + case NFPROTO_IPV4: + return nf_route_table_hook4(priv, skb, state); + case NFPROTO_IPV6: + return nf_route_table_hook6(priv, skb, state); + default: + nft_set_pktinfo(&pkt, skb, state); + break; + } + + return nft_do_chain(&pkt, priv); +} + +static const struct nft_chain_type nft_chain_route_inet = { + .name = "route", + .type = NFT_CHAIN_T_ROUTE, + .family = NFPROTO_INET, + .hook_mask = (1 << NF_INET_LOCAL_OUT), + .hooks = { + [NF_INET_LOCAL_OUT] = nf_route_table_inet, + }, +}; +#endif + +void __init nft_chain_route_init(void) +{ +#ifdef CONFIG_NF_TABLES_IPV6 + nft_register_chain_type(&nft_chain_route_ipv6); +#endif +#ifdef CONFIG_NF_TABLES_IPV4 + nft_register_chain_type(&nft_chain_route_ipv4); +#endif +#ifdef CONFIG_NF_TABLES_INET + nft_register_chain_type(&nft_chain_route_inet); +#endif +} + +void __exit nft_chain_route_fini(void) +{ +#ifdef CONFIG_NF_TABLES_IPV6 + nft_unregister_chain_type(&nft_chain_route_ipv6); +#endif +#ifdef CONFIG_NF_TABLES_IPV4 + nft_unregister_chain_type(&nft_chain_route_ipv4); +#endif +#ifdef CONFIG_NF_TABLES_INET + nft_unregister_chain_type(&nft_chain_route_inet); +#endif +} diff --git a/net/netfilter/nft_compat.c b/net/netfilter/nft_compat.c index 469f9da5073bf61ea7ac50ea02deeea4b4400d15..276f1f2d6de149bee039643bf57947bce69e65d5 100644 --- a/net/netfilter/nft_compat.c +++ b/net/netfilter/nft_compat.c @@ -198,8 +198,8 @@ static int nft_parse_compat(const struct nlattr *attr, u16 *proto, bool *inv) u32 flags; int err; - err = nla_parse_nested(tb, NFTA_RULE_COMPAT_MAX, attr, - nft_rule_compat_policy, NULL); + err = nla_parse_nested_deprecated(tb, NFTA_RULE_COMPAT_MAX, attr, + nft_rule_compat_policy, NULL); if (err < 0) return err; diff --git a/net/netfilter/nft_ct.c b/net/netfilter/nft_ct.c index 7b717fad6cdc8ebbd70cb68bb212c3cb51a93733..f043936763f391878c76f7c7f0b6d51ac1694a81 100644 --- a/net/netfilter/nft_ct.c +++ b/net/netfilter/nft_ct.c @@ -178,6 +178,11 @@ static void nft_ct_get_eval(const struct nft_expr *expr, return; } #endif + case NFT_CT_ID: + if (!nf_ct_is_confirmed(ct)) + goto err; + *dest = nf_ct_get_id(ct); + return; default: break; } @@ -479,6 +484,9 @@ static int nft_ct_get_init(const struct nft_ctx *ctx, len = sizeof(u16); break; #endif + case NFT_CT_ID: + len = sizeof(u32); + break; default: return -EOPNOTSUPP; } @@ -797,9 +805,11 @@ nft_ct_timeout_parse_policy(void *timeouts, if (!tb) return -ENOMEM; - ret = nla_parse_nested(tb, l4proto->ctnl_timeout.nlattr_max, - attr, l4proto->ctnl_timeout.nla_policy, - NULL); + ret = nla_parse_nested_deprecated(tb, + l4proto->ctnl_timeout.nlattr_max, + attr, + l4proto->ctnl_timeout.nla_policy, + NULL); if (ret < 0) goto err; @@ -928,7 +938,7 @@ static int nft_ct_timeout_obj_dump(struct sk_buff *skb, nla_put_be16(skb, NFTA_CT_TIMEOUT_L3PROTO, htons(timeout->l3num))) return -1; - nest_params = nla_nest_start(skb, NFTA_CT_TIMEOUT_DATA | NLA_F_NESTED); + nest_params = nla_nest_start(skb, NFTA_CT_TIMEOUT_DATA); if (!nest_params) return -1; diff --git a/net/netfilter/nft_dynset.c b/net/netfilter/nft_dynset.c index e461007558e8900f16bfb2ca3944d23987d5e669..8394560aa695a1390848ea670da2cb12e0b72f39 100644 --- a/net/netfilter/nft_dynset.c +++ b/net/netfilter/nft_dynset.c @@ -28,6 +28,23 @@ struct nft_dynset { struct nft_set_binding binding; }; +static int nft_expr_clone(struct nft_expr *dst, struct nft_expr *src) +{ + int err; + + if (src->ops->clone) { + dst->ops = src->ops; + err = src->ops->clone(dst, src); + if (err < 0) + return err; + } else { + memcpy(dst, src, src->ops->size); + } + + __module_get(src->ops->type->owner); + return 0; +} + static void *nft_dynset_new(struct nft_set *set, const struct nft_expr *expr, struct nft_regs *regs) { diff --git a/net/netfilter/nft_masq.c b/net/netfilter/nft_masq.c index bee156eaa4008202c9fc4ebc1548e98dbd881d43..86fd90085eafa4fc511929ce9c3a989b0469e50b 100644 --- a/net/netfilter/nft_masq.c +++ b/net/netfilter/nft_masq.c @@ -14,8 +14,7 @@ #include #include #include -#include -#include +#include struct nft_masq { u32 flags; @@ -196,28 +195,73 @@ static struct nft_expr_type nft_masq_ipv6_type __read_mostly = { static int __init nft_masq_module_init_ipv6(void) { - int ret = nft_register_expr(&nft_masq_ipv6_type); - - if (ret) - return ret; - - ret = nf_nat_masquerade_ipv6_register_notifier(); - if (ret < 0) - nft_unregister_expr(&nft_masq_ipv6_type); - - return ret; + return nft_register_expr(&nft_masq_ipv6_type); } static void nft_masq_module_exit_ipv6(void) { nft_unregister_expr(&nft_masq_ipv6_type); - nf_nat_masquerade_ipv6_unregister_notifier(); } #else static inline int nft_masq_module_init_ipv6(void) { return 0; } static inline void nft_masq_module_exit_ipv6(void) {} #endif +#ifdef CONFIG_NF_TABLES_INET +static void nft_masq_inet_eval(const struct nft_expr *expr, + struct nft_regs *regs, + const struct nft_pktinfo *pkt) +{ + switch (nft_pf(pkt)) { + case NFPROTO_IPV4: + return nft_masq_ipv4_eval(expr, regs, pkt); + case NFPROTO_IPV6: + return nft_masq_ipv6_eval(expr, regs, pkt); + } + + WARN_ON_ONCE(1); +} + +static void +nft_masq_inet_destroy(const struct nft_ctx *ctx, const struct nft_expr *expr) +{ + nf_ct_netns_put(ctx->net, NFPROTO_INET); +} + +static struct nft_expr_type nft_masq_inet_type; +static const struct nft_expr_ops nft_masq_inet_ops = { + .type = &nft_masq_inet_type, + .size = NFT_EXPR_SIZE(sizeof(struct nft_masq)), + .eval = nft_masq_inet_eval, + .init = nft_masq_init, + .destroy = nft_masq_inet_destroy, + .dump = nft_masq_dump, + .validate = nft_masq_validate, +}; + +static struct nft_expr_type nft_masq_inet_type __read_mostly = { + .family = NFPROTO_INET, + .name = "masq", + .ops = &nft_masq_inet_ops, + .policy = nft_masq_policy, + .maxattr = NFTA_MASQ_MAX, + .owner = THIS_MODULE, +}; + +static int __init nft_masq_module_init_inet(void) +{ + return nft_register_expr(&nft_masq_inet_type); +} + +static void nft_masq_module_exit_inet(void) +{ + nft_unregister_expr(&nft_masq_inet_type); +} +#else +static inline int nft_masq_module_init_inet(void) { return 0; } +static inline void nft_masq_module_exit_inet(void) {} +#endif + static int __init nft_masq_module_init(void) { int ret; @@ -226,15 +270,23 @@ static int __init nft_masq_module_init(void) if (ret < 0) return ret; + ret = nft_masq_module_init_inet(); + if (ret < 0) { + nft_masq_module_exit_ipv6(); + return ret; + } + ret = nft_register_expr(&nft_masq_ipv4_type); if (ret < 0) { + nft_masq_module_exit_inet(); nft_masq_module_exit_ipv6(); return ret; } - ret = nf_nat_masquerade_ipv4_register_notifier(); + ret = nf_nat_masquerade_inet_register_notifiers(); if (ret < 0) { nft_masq_module_exit_ipv6(); + nft_masq_module_exit_inet(); nft_unregister_expr(&nft_masq_ipv4_type); return ret; } @@ -245,8 +297,9 @@ static int __init nft_masq_module_init(void) static void __exit nft_masq_module_exit(void) { nft_masq_module_exit_ipv6(); + nft_masq_module_exit_inet(); nft_unregister_expr(&nft_masq_ipv4_type); - nf_nat_masquerade_ipv4_unregister_notifier(); + nf_nat_masquerade_inet_unregister_notifiers(); } module_init(nft_masq_module_init); diff --git a/net/netfilter/nft_nat.c b/net/netfilter/nft_nat.c index e93aed9bda887dff7db5c0312aa6ad66d388fdcf..d90d421826aa28546d3c6f2a1b0e1b92f9b1caca 100644 --- a/net/netfilter/nft_nat.c +++ b/net/netfilter/nft_nat.c @@ -140,7 +140,7 @@ static int nft_nat_init(const struct nft_ctx *ctx, const struct nft_expr *expr, return -EINVAL; family = ntohl(nla_get_be32(tb[NFTA_NAT_FAMILY])); - if (family != ctx->family) + if (ctx->family != NFPROTO_INET && ctx->family != family) return -EOPNOTSUPP; switch (family) { @@ -278,13 +278,67 @@ static struct nft_expr_type nft_nat_type __read_mostly = { .owner = THIS_MODULE, }; +#ifdef CONFIG_NF_TABLES_INET +static void nft_nat_inet_eval(const struct nft_expr *expr, + struct nft_regs *regs, + const struct nft_pktinfo *pkt) +{ + const struct nft_nat *priv = nft_expr_priv(expr); + + if (priv->family == nft_pf(pkt)) + nft_nat_eval(expr, regs, pkt); +} + +static const struct nft_expr_ops nft_nat_inet_ops = { + .type = &nft_nat_type, + .size = NFT_EXPR_SIZE(sizeof(struct nft_nat)), + .eval = nft_nat_inet_eval, + .init = nft_nat_init, + .destroy = nft_nat_destroy, + .dump = nft_nat_dump, + .validate = nft_nat_validate, +}; + +static struct nft_expr_type nft_inet_nat_type __read_mostly = { + .name = "nat", + .family = NFPROTO_INET, + .ops = &nft_nat_inet_ops, + .policy = nft_nat_policy, + .maxattr = NFTA_NAT_MAX, + .owner = THIS_MODULE, +}; + +static int nft_nat_inet_module_init(void) +{ + return nft_register_expr(&nft_inet_nat_type); +} + +static void nft_nat_inet_module_exit(void) +{ + nft_unregister_expr(&nft_inet_nat_type); +} +#else +static int nft_nat_inet_module_init(void) { return 0; } +static void nft_nat_inet_module_exit(void) { } +#endif + static int __init nft_nat_module_init(void) { - return nft_register_expr(&nft_nat_type); + int ret = nft_nat_inet_module_init(); + + if (ret) + return ret; + + ret = nft_register_expr(&nft_nat_type); + if (ret) + nft_nat_inet_module_exit(); + + return ret; } static void __exit nft_nat_module_exit(void) { + nft_nat_inet_module_exit(); nft_unregister_expr(&nft_nat_type); } diff --git a/net/netfilter/nft_osf.c b/net/netfilter/nft_osf.c index b13618c764ec296377778ee405b9067515ada25a..87b60d6617ef1d55703f01cbf7a7a849ef515645 100644 --- a/net/netfilter/nft_osf.c +++ b/net/netfilter/nft_osf.c @@ -7,11 +7,13 @@ struct nft_osf { enum nft_registers dreg:8; u8 ttl; + u32 flags; }; static const struct nla_policy nft_osf_policy[NFTA_OSF_MAX + 1] = { [NFTA_OSF_DREG] = { .type = NLA_U32 }, [NFTA_OSF_TTL] = { .type = NLA_U8 }, + [NFTA_OSF_FLAGS] = { .type = NLA_U32 }, }; static void nft_osf_eval(const struct nft_expr *expr, struct nft_regs *regs, @@ -20,9 +22,10 @@ static void nft_osf_eval(const struct nft_expr *expr, struct nft_regs *regs, struct nft_osf *priv = nft_expr_priv(expr); u32 *dest = ®s->data[priv->dreg]; struct sk_buff *skb = pkt->skb; + char os_match[NFT_OSF_MAXGENRELEN + 1]; const struct tcphdr *tcp; + struct nf_osf_data data; struct tcphdr _tcph; - const char *os_name; tcp = skb_header_pointer(skb, ip_hdrlen(skb), sizeof(struct tcphdr), &_tcph); @@ -35,11 +38,17 @@ static void nft_osf_eval(const struct nft_expr *expr, struct nft_regs *regs, return; } - os_name = nf_osf_find(skb, nf_osf_fingers, priv->ttl); - if (!os_name) + if (!nf_osf_find(skb, nf_osf_fingers, priv->ttl, &data)) { strncpy((char *)dest, "unknown", NFT_OSF_MAXGENRELEN); - else - strncpy((char *)dest, os_name, NFT_OSF_MAXGENRELEN); + } else { + if (priv->flags & NFT_OSF_F_VERSION) + snprintf(os_match, NFT_OSF_MAXGENRELEN, "%s:%s", + data.genre, data.version); + else + strlcpy(os_match, data.genre, NFT_OSF_MAXGENRELEN); + + strncpy((char *)dest, os_match, NFT_OSF_MAXGENRELEN); + } } static int nft_osf_init(const struct nft_ctx *ctx, @@ -47,6 +56,7 @@ static int nft_osf_init(const struct nft_ctx *ctx, const struct nlattr * const tb[]) { struct nft_osf *priv = nft_expr_priv(expr); + u32 flags; int err; u8 ttl; @@ -57,6 +67,13 @@ static int nft_osf_init(const struct nft_ctx *ctx, priv->ttl = ttl; } + if (tb[NFTA_OSF_FLAGS]) { + flags = ntohl(nla_get_be32(tb[NFTA_OSF_FLAGS])); + if (flags != NFT_OSF_F_VERSION) + return -EINVAL; + priv->flags = flags; + } + priv->dreg = nft_parse_register(tb[NFTA_OSF_DREG]); err = nft_validate_register_store(ctx, priv->dreg, NULL, NFT_DATA_VALUE, NFT_OSF_MAXGENRELEN); @@ -73,6 +90,9 @@ static int nft_osf_dump(struct sk_buff *skb, const struct nft_expr *expr) if (nla_put_u8(skb, NFTA_OSF_TTL, priv->ttl)) goto nla_put_failure; + if (nla_put_be32(skb, NFTA_OSF_FLAGS, ntohl(priv->flags))) + goto nla_put_failure; + if (nft_dump_register(skb, NFTA_OSF_DREG, priv->dreg)) goto nla_put_failure; diff --git a/net/netfilter/nft_redir.c b/net/netfilter/nft_redir.c index a340cd8a751b483766e4ed7274ce0fb2c2b193e2..da74fdc4a6844f2294ba952188469b0cee64aefd 100644 --- a/net/netfilter/nft_redir.c +++ b/net/netfilter/nft_redir.c @@ -82,7 +82,7 @@ static int nft_redir_init(const struct nft_ctx *ctx, return nf_ct_netns_get(ctx->net, ctx->family); } -int nft_redir_dump(struct sk_buff *skb, const struct nft_expr *expr) +static int nft_redir_dump(struct sk_buff *skb, const struct nft_expr *expr) { const struct nft_redir *priv = nft_expr_priv(expr); @@ -202,6 +202,55 @@ static struct nft_expr_type nft_redir_ipv6_type __read_mostly = { }; #endif +#ifdef CONFIG_NF_TABLES_INET +static void nft_redir_inet_eval(const struct nft_expr *expr, + struct nft_regs *regs, + const struct nft_pktinfo *pkt) +{ + switch (nft_pf(pkt)) { + case NFPROTO_IPV4: + return nft_redir_ipv4_eval(expr, regs, pkt); + case NFPROTO_IPV6: + return nft_redir_ipv6_eval(expr, regs, pkt); + } + + WARN_ON_ONCE(1); +} + +static void +nft_redir_inet_destroy(const struct nft_ctx *ctx, const struct nft_expr *expr) +{ + nf_ct_netns_put(ctx->net, NFPROTO_INET); +} + +static struct nft_expr_type nft_redir_inet_type; +static const struct nft_expr_ops nft_redir_inet_ops = { + .type = &nft_redir_inet_type, + .size = NFT_EXPR_SIZE(sizeof(struct nft_redir)), + .eval = nft_redir_inet_eval, + .init = nft_redir_init, + .destroy = nft_redir_inet_destroy, + .dump = nft_redir_dump, + .validate = nft_redir_validate, +}; + +static struct nft_expr_type nft_redir_inet_type __read_mostly = { + .family = NFPROTO_INET, + .name = "redir", + .ops = &nft_redir_inet_ops, + .policy = nft_redir_policy, + .maxattr = NFTA_MASQ_MAX, + .owner = THIS_MODULE, +}; + +static int __init nft_redir_module_init_inet(void) +{ + return nft_register_expr(&nft_redir_inet_type); +} +#else +static inline int nft_redir_module_init_inet(void) { return 0; } +#endif + static int __init nft_redir_module_init(void) { int ret = nft_register_expr(&nft_redir_ipv4_type); @@ -217,6 +266,15 @@ static int __init nft_redir_module_init(void) } #endif + ret = nft_redir_module_init_inet(); + if (ret < 0) { + nft_unregister_expr(&nft_redir_ipv4_type); +#ifdef CONFIG_NF_TABLES_IPV6 + nft_unregister_expr(&nft_redir_ipv6_type); +#endif + return ret; + } + return ret; } @@ -226,6 +284,9 @@ static void __exit nft_redir_module_exit(void) #ifdef CONFIG_NF_TABLES_IPV6 nft_unregister_expr(&nft_redir_ipv6_type); #endif +#ifdef CONFIG_NF_TABLES_INET + nft_unregister_expr(&nft_redir_inet_type); +#endif } module_init(nft_redir_module_init); diff --git a/net/netfilter/nft_tunnel.c b/net/netfilter/nft_tunnel.c index b113fcac94e1c46be2fae6dd3eec2c5cbb19f40a..3d4c2ae605a8e2c3ceee669570775cd5ebbb3e1e 100644 --- a/net/netfilter/nft_tunnel.c +++ b/net/netfilter/nft_tunnel.c @@ -166,8 +166,8 @@ static int nft_tunnel_obj_ip_init(const struct nft_ctx *ctx, struct nlattr *tb[NFTA_TUNNEL_KEY_IP_MAX + 1]; int err; - err = nla_parse_nested(tb, NFTA_TUNNEL_KEY_IP_MAX, attr, - nft_tunnel_ip_policy, NULL); + err = nla_parse_nested_deprecated(tb, NFTA_TUNNEL_KEY_IP_MAX, attr, + nft_tunnel_ip_policy, NULL); if (err < 0) return err; @@ -195,8 +195,8 @@ static int nft_tunnel_obj_ip6_init(const struct nft_ctx *ctx, struct nlattr *tb[NFTA_TUNNEL_KEY_IP6_MAX + 1]; int err; - err = nla_parse_nested(tb, NFTA_TUNNEL_KEY_IP6_MAX, attr, - nft_tunnel_ip6_policy, NULL); + err = nla_parse_nested_deprecated(tb, NFTA_TUNNEL_KEY_IP6_MAX, attr, + nft_tunnel_ip6_policy, NULL); if (err < 0) return err; @@ -231,8 +231,8 @@ static int nft_tunnel_obj_vxlan_init(const struct nlattr *attr, struct nlattr *tb[NFTA_TUNNEL_KEY_VXLAN_MAX + 1]; int err; - err = nla_parse_nested(tb, NFTA_TUNNEL_KEY_VXLAN_MAX, attr, - nft_tunnel_opts_vxlan_policy, NULL); + err = nla_parse_nested_deprecated(tb, NFTA_TUNNEL_KEY_VXLAN_MAX, attr, + nft_tunnel_opts_vxlan_policy, NULL); if (err < 0) return err; @@ -260,8 +260,9 @@ static int nft_tunnel_obj_erspan_init(const struct nlattr *attr, uint8_t hwid, dir; int err, version; - err = nla_parse_nested(tb, NFTA_TUNNEL_KEY_ERSPAN_MAX, attr, - nft_tunnel_opts_erspan_policy, NULL); + err = nla_parse_nested_deprecated(tb, NFTA_TUNNEL_KEY_ERSPAN_MAX, + attr, nft_tunnel_opts_erspan_policy, + NULL); if (err < 0) return err; @@ -309,8 +310,8 @@ static int nft_tunnel_obj_opts_init(const struct nft_ctx *ctx, struct nlattr *tb[NFTA_TUNNEL_KEY_OPTS_MAX + 1]; int err; - err = nla_parse_nested(tb, NFTA_TUNNEL_KEY_OPTS_MAX, attr, - nft_tunnel_opts_policy, NULL); + err = nla_parse_nested_deprecated(tb, NFTA_TUNNEL_KEY_OPTS_MAX, attr, + nft_tunnel_opts_policy, NULL); if (err < 0) return err; @@ -437,7 +438,7 @@ static int nft_tunnel_ip_dump(struct sk_buff *skb, struct ip_tunnel_info *info) struct nlattr *nest; if (info->mode & IP_TUNNEL_INFO_IPV6) { - nest = nla_nest_start(skb, NFTA_TUNNEL_KEY_IP6); + nest = nla_nest_start_noflag(skb, NFTA_TUNNEL_KEY_IP6); if (!nest) return -1; @@ -448,7 +449,7 @@ static int nft_tunnel_ip_dump(struct sk_buff *skb, struct ip_tunnel_info *info) nla_nest_end(skb, nest); } else { - nest = nla_nest_start(skb, NFTA_TUNNEL_KEY_IP); + nest = nla_nest_start_noflag(skb, NFTA_TUNNEL_KEY_IP); if (!nest) return -1; @@ -468,7 +469,7 @@ static int nft_tunnel_opts_dump(struct sk_buff *skb, struct nft_tunnel_opts *opts = &priv->opts; struct nlattr *nest; - nest = nla_nest_start(skb, NFTA_TUNNEL_KEY_OPTS); + nest = nla_nest_start_noflag(skb, NFTA_TUNNEL_KEY_OPTS); if (!nest) return -1; diff --git a/net/netfilter/x_tables.c b/net/netfilter/x_tables.c index e5e5c64df8d1a94b4e448c2a146c7c112f2e0e58..0a6656ed153430a5b585675a760e4c0c322ab8c8 100644 --- a/net/netfilter/x_tables.c +++ b/net/netfilter/x_tables.c @@ -227,7 +227,7 @@ xt_request_find_match(uint8_t nfproto, const char *name, uint8_t revision) EXPORT_SYMBOL_GPL(xt_request_find_match); /* Find target, grabs ref. Returns ERR_PTR() on error. */ -struct xt_target *xt_find_target(u8 af, const char *name, u8 revision) +static struct xt_target *xt_find_target(u8 af, const char *name, u8 revision) { struct xt_target *t; int err = -ENOENT; @@ -255,7 +255,6 @@ struct xt_target *xt_find_target(u8 af, const char *name, u8 revision) return ERR_PTR(err); } -EXPORT_SYMBOL(xt_find_target); struct xt_target *xt_request_find_target(u8 af, const char *name, u8 revision) { diff --git a/net/netfilter/xt_CT.c b/net/netfilter/xt_CT.c index 0fa863f57575376e16aa59859fa5e392266e715a..d59cb4730facf71c64065961268bf63b54c7ca9c 100644 --- a/net/netfilter/xt_CT.c +++ b/net/netfilter/xt_CT.c @@ -103,85 +103,24 @@ xt_ct_set_helper(struct nf_conn *ct, const char *helper_name, return 0; } -#ifdef CONFIG_NF_CONNTRACK_TIMEOUT -static void __xt_ct_tg_timeout_put(struct nf_ct_timeout *timeout) -{ - typeof(nf_ct_timeout_put_hook) timeout_put; - - timeout_put = rcu_dereference(nf_ct_timeout_put_hook); - if (timeout_put) - timeout_put(timeout); -} -#endif - static int xt_ct_set_timeout(struct nf_conn *ct, const struct xt_tgchk_param *par, const char *timeout_name) { #ifdef CONFIG_NF_CONNTRACK_TIMEOUT - typeof(nf_ct_timeout_find_get_hook) timeout_find_get; const struct nf_conntrack_l4proto *l4proto; - struct nf_ct_timeout *timeout; - struct nf_conn_timeout *timeout_ext; - const char *errmsg = NULL; - int ret = 0; u8 proto; - rcu_read_lock(); - timeout_find_get = rcu_dereference(nf_ct_timeout_find_get_hook); - if (timeout_find_get == NULL) { - ret = -ENOENT; - errmsg = "Timeout policy base is empty"; - goto out; - } - proto = xt_ct_find_proto(par); if (!proto) { - ret = -EINVAL; - errmsg = "You must specify a L4 protocol and not use inversions on it"; - goto out; - } - - timeout = timeout_find_get(par->net, timeout_name); - if (timeout == NULL) { - ret = -ENOENT; - pr_info_ratelimited("No such timeout policy \"%s\"\n", - timeout_name); - goto out; - } - - if (timeout->l3num != par->family) { - ret = -EINVAL; - pr_info_ratelimited("Timeout policy `%s' can only be used by L%d protocol number %d\n", - timeout_name, 3, timeout->l3num); - goto err_put_timeout; + pr_info_ratelimited("You must specify a L4 protocol and not " + "use inversions on it"); + return -EINVAL; } - /* Make sure the timeout policy matches any existing protocol tracker, - * otherwise default to generic. - */ l4proto = nf_ct_l4proto_find(proto); - if (timeout->l4proto->l4proto != l4proto->l4proto) { - ret = -EINVAL; - pr_info_ratelimited("Timeout policy `%s' can only be used by L%d protocol number %d\n", - timeout_name, 4, timeout->l4proto->l4proto); - goto err_put_timeout; - } - timeout_ext = nf_ct_timeout_ext_add(ct, timeout, GFP_ATOMIC); - if (!timeout_ext) { - ret = -ENOMEM; - goto err_put_timeout; - } + return nf_ct_set_timeout(par->net, ct, par->family, l4proto->l4proto, + timeout_name); - rcu_read_unlock(); - return ret; - -err_put_timeout: - __xt_ct_tg_timeout_put(timeout); -out: - rcu_read_unlock(); - if (errmsg) - pr_info_ratelimited("%s\n", errmsg); - return ret; #else return -EOPNOTSUPP; #endif @@ -328,26 +267,6 @@ static int xt_ct_tg_check_v2(const struct xt_tgchk_param *par) return xt_ct_tg_check(par, par->targinfo); } -static void xt_ct_destroy_timeout(struct nf_conn *ct) -{ -#ifdef CONFIG_NF_CONNTRACK_TIMEOUT - struct nf_conn_timeout *timeout_ext; - typeof(nf_ct_timeout_put_hook) timeout_put; - - rcu_read_lock(); - timeout_put = rcu_dereference(nf_ct_timeout_put_hook); - - if (timeout_put) { - timeout_ext = nf_ct_timeout_find(ct); - if (timeout_ext) { - timeout_put(timeout_ext->timeout); - RCU_INIT_POINTER(timeout_ext->timeout, NULL); - } - } - rcu_read_unlock(); -#endif -} - static void xt_ct_tg_destroy(const struct xt_tgdtor_param *par, struct xt_ct_target_info_v1 *info) { @@ -361,7 +280,7 @@ static void xt_ct_tg_destroy(const struct xt_tgdtor_param *par, nf_ct_netns_put(par->net, par->family); - xt_ct_destroy_timeout(ct); + nf_ct_destroy_timeout(ct); nf_ct_put(info->ct); } } diff --git a/net/ipv4/netfilter/ipt_MASQUERADE.c b/net/netfilter/xt_MASQUERADE.c similarity index 52% rename from net/ipv4/netfilter/ipt_MASQUERADE.c rename to net/netfilter/xt_MASQUERADE.c index fd3f9e8a74daf4954d675eaf2cc381196facc4f4..ece20d832adc0adb994a92df9cf323feaebb9e81 100644 --- a/net/ipv4/netfilter/ipt_MASQUERADE.c +++ b/net/netfilter/xt_MASQUERADE.c @@ -9,20 +9,10 @@ * published by the Free Software Foundation. */ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt -#include -#include -#include -#include #include -#include -#include -#include -#include -#include -#include #include #include -#include +#include MODULE_LICENSE("GPL"); MODULE_AUTHOR("Netfilter Core Team "); @@ -64,38 +54,78 @@ static void masquerade_tg_destroy(const struct xt_tgdtor_param *par) nf_ct_netns_put(par->net, par->family); } -static struct xt_target masquerade_tg_reg __read_mostly = { - .name = "MASQUERADE", - .family = NFPROTO_IPV4, - .target = masquerade_tg, - .targetsize = sizeof(struct nf_nat_ipv4_multi_range_compat), - .table = "nat", - .hooks = 1 << NF_INET_POST_ROUTING, - .checkentry = masquerade_tg_check, - .destroy = masquerade_tg_destroy, - .me = THIS_MODULE, +#if IS_ENABLED(CONFIG_IPV6) +static unsigned int +masquerade_tg6(struct sk_buff *skb, const struct xt_action_param *par) +{ + return nf_nat_masquerade_ipv6(skb, par->targinfo, xt_out(par)); +} + +static int masquerade_tg6_checkentry(const struct xt_tgchk_param *par) +{ + const struct nf_nat_range2 *range = par->targinfo; + + if (range->flags & NF_NAT_RANGE_MAP_IPS) + return -EINVAL; + + return nf_ct_netns_get(par->net, par->family); +} +#endif + +static struct xt_target masquerade_tg_reg[] __read_mostly = { + { +#if IS_ENABLED(CONFIG_IPV6) + .name = "MASQUERADE", + .family = NFPROTO_IPV6, + .target = masquerade_tg6, + .targetsize = sizeof(struct nf_nat_range), + .table = "nat", + .hooks = 1 << NF_INET_POST_ROUTING, + .checkentry = masquerade_tg6_checkentry, + .destroy = masquerade_tg_destroy, + .me = THIS_MODULE, + }, { +#endif + .name = "MASQUERADE", + .family = NFPROTO_IPV4, + .target = masquerade_tg, + .targetsize = sizeof(struct nf_nat_ipv4_multi_range_compat), + .table = "nat", + .hooks = 1 << NF_INET_POST_ROUTING, + .checkentry = masquerade_tg_check, + .destroy = masquerade_tg_destroy, + .me = THIS_MODULE, + } }; static int __init masquerade_tg_init(void) { int ret; - ret = xt_register_target(&masquerade_tg_reg); + ret = xt_register_targets(masquerade_tg_reg, + ARRAY_SIZE(masquerade_tg_reg)); if (ret) return ret; - ret = nf_nat_masquerade_ipv4_register_notifier(); - if (ret) - xt_unregister_target(&masquerade_tg_reg); + ret = nf_nat_masquerade_inet_register_notifiers(); + if (ret) { + xt_unregister_targets(masquerade_tg_reg, + ARRAY_SIZE(masquerade_tg_reg)); + return ret; + } return ret; } static void __exit masquerade_tg_exit(void) { - xt_unregister_target(&masquerade_tg_reg); - nf_nat_masquerade_ipv4_unregister_notifier(); + xt_unregister_targets(masquerade_tg_reg, ARRAY_SIZE(masquerade_tg_reg)); + nf_nat_masquerade_inet_unregister_notifiers(); } module_init(masquerade_tg_init); module_exit(masquerade_tg_exit); +#if IS_ENABLED(CONFIG_IPV6) +MODULE_ALIAS("ip6t_MASQUERADE"); +#endif +MODULE_ALIAS("ipt_MASQUERADE"); diff --git a/net/netfilter/xt_connlabel.c b/net/netfilter/xt_connlabel.c index 4fa4efd243532a5507cf9eba4a8a08cb7e63e01b..893374ac3758a56bfdf9affbd5e27948704b5ac0 100644 --- a/net/netfilter/xt_connlabel.c +++ b/net/netfilter/xt_connlabel.c @@ -15,7 +15,7 @@ MODULE_LICENSE("GPL"); MODULE_AUTHOR("Florian Westphal "); -MODULE_DESCRIPTION("Xtables: add/match connection trackling labels"); +MODULE_DESCRIPTION("Xtables: add/match connection tracking labels"); MODULE_ALIAS("ipt_connlabel"); MODULE_ALIAS("ip6t_connlabel"); diff --git a/net/netfilter/xt_hashlimit.c b/net/netfilter/xt_hashlimit.c index 8d86e39d6280a6112ca37ecd1732da70819b1f98..a30536b17ee143f36e04fad4b8ae115b5e3edfe4 100644 --- a/net/netfilter/xt_hashlimit.c +++ b/net/netfilter/xt_hashlimit.c @@ -288,8 +288,7 @@ static int htable_create(struct net *net, struct hashlimit_cfg3 *cfg, size = 16; } /* FIXME: don't use vmalloc() here or anywhere else -HW */ - hinfo = vmalloc(sizeof(struct xt_hashlimit_htable) + - sizeof(struct hlist_head) * size); + hinfo = vmalloc(struct_size(hinfo, hash, size)); if (hinfo == NULL) return -ENOMEM; *out_hinfo = hinfo; diff --git a/net/netlabel/netlabel_calipso.c b/net/netlabel/netlabel_calipso.c index 4d748975117dba1b3889e40f48a4229bcefe2459..1de87172885d22bca351f321f28fbf8e2fea435b 100644 --- a/net/netlabel/netlabel_calipso.c +++ b/net/netlabel/netlabel_calipso.c @@ -321,29 +321,29 @@ static int netlbl_calipso_remove(struct sk_buff *skb, struct genl_info *info) static const struct genl_ops netlbl_calipso_ops[] = { { .cmd = NLBL_CALIPSO_C_ADD, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = calipso_genl_policy, .doit = netlbl_calipso_add, .dumpit = NULL, }, { .cmd = NLBL_CALIPSO_C_REMOVE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = calipso_genl_policy, .doit = netlbl_calipso_remove, .dumpit = NULL, }, { .cmd = NLBL_CALIPSO_C_LIST, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = 0, - .policy = calipso_genl_policy, .doit = netlbl_calipso_list, .dumpit = NULL, }, { .cmd = NLBL_CALIPSO_C_LISTALL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = 0, - .policy = calipso_genl_policy, .doit = NULL, .dumpit = netlbl_calipso_listall, }, @@ -354,6 +354,7 @@ static struct genl_family netlbl_calipso_gnl_family __ro_after_init = { .name = NETLBL_NLTYPE_CALIPSO_NAME, .version = NETLBL_PROTO_VERSION, .maxattr = NLBL_CALIPSO_A_MAX, + .policy = calipso_genl_policy, .module = THIS_MODULE, .ops = netlbl_calipso_ops, .n_ops = ARRAY_SIZE(netlbl_calipso_ops), diff --git a/net/netlabel/netlabel_cipso_v4.c b/net/netlabel/netlabel_cipso_v4.c index 9aacf2da3d98feaf94319f7315b255323f4416f2..5d1121981d0bacf0e525b13f9774251ea5c70525 100644 --- a/net/netlabel/netlabel_cipso_v4.c +++ b/net/netlabel/netlabel_cipso_v4.c @@ -99,9 +99,10 @@ static int netlbl_cipsov4_add_common(struct genl_info *info, doi_def->doi = nla_get_u32(info->attrs[NLBL_CIPSOV4_A_DOI]); - if (nla_validate_nested(info->attrs[NLBL_CIPSOV4_A_TAGLST], - NLBL_CIPSOV4_A_MAX, - netlbl_cipsov4_genl_policy, NULL) != 0) + if (nla_validate_nested_deprecated(info->attrs[NLBL_CIPSOV4_A_TAGLST], + NLBL_CIPSOV4_A_MAX, + netlbl_cipsov4_genl_policy, + NULL) != 0) return -EINVAL; nla_for_each_nested(nla, info->attrs[NLBL_CIPSOV4_A_TAGLST], nla_rem) @@ -146,9 +147,10 @@ static int netlbl_cipsov4_add_std(struct genl_info *info, !info->attrs[NLBL_CIPSOV4_A_MLSLVLLST]) return -EINVAL; - if (nla_validate_nested(info->attrs[NLBL_CIPSOV4_A_MLSLVLLST], - NLBL_CIPSOV4_A_MAX, - netlbl_cipsov4_genl_policy, NULL) != 0) + if (nla_validate_nested_deprecated(info->attrs[NLBL_CIPSOV4_A_MLSLVLLST], + NLBL_CIPSOV4_A_MAX, + netlbl_cipsov4_genl_policy, + NULL) != 0) return -EINVAL; doi_def = kmalloc(sizeof(*doi_def), GFP_KERNEL); @@ -170,9 +172,10 @@ static int netlbl_cipsov4_add_std(struct genl_info *info, info->attrs[NLBL_CIPSOV4_A_MLSLVLLST], nla_a_rem) if (nla_type(nla_a) == NLBL_CIPSOV4_A_MLSLVL) { - if (nla_validate_nested(nla_a, NLBL_CIPSOV4_A_MAX, - netlbl_cipsov4_genl_policy, - NULL) != 0) + if (nla_validate_nested_deprecated(nla_a, + NLBL_CIPSOV4_A_MAX, + netlbl_cipsov4_genl_policy, + NULL) != 0) goto add_std_failure; nla_for_each_nested(nla_b, nla_a, nla_b_rem) switch (nla_type(nla_b)) { @@ -234,19 +237,20 @@ static int netlbl_cipsov4_add_std(struct genl_info *info, } if (info->attrs[NLBL_CIPSOV4_A_MLSCATLST]) { - if (nla_validate_nested(info->attrs[NLBL_CIPSOV4_A_MLSCATLST], - NLBL_CIPSOV4_A_MAX, - netlbl_cipsov4_genl_policy, NULL) != 0) + if (nla_validate_nested_deprecated(info->attrs[NLBL_CIPSOV4_A_MLSCATLST], + NLBL_CIPSOV4_A_MAX, + netlbl_cipsov4_genl_policy, + NULL) != 0) goto add_std_failure; nla_for_each_nested(nla_a, info->attrs[NLBL_CIPSOV4_A_MLSCATLST], nla_a_rem) if (nla_type(nla_a) == NLBL_CIPSOV4_A_MLSCAT) { - if (nla_validate_nested(nla_a, - NLBL_CIPSOV4_A_MAX, - netlbl_cipsov4_genl_policy, - NULL) != 0) + if (nla_validate_nested_deprecated(nla_a, + NLBL_CIPSOV4_A_MAX, + netlbl_cipsov4_genl_policy, + NULL) != 0) goto add_std_failure; nla_for_each_nested(nla_b, nla_a, nla_b_rem) switch (nla_type(nla_b)) { @@ -498,7 +502,7 @@ list_start: if (ret_val != 0) goto list_failure_lock; - nla_a = nla_nest_start(ans_skb, NLBL_CIPSOV4_A_TAGLST); + nla_a = nla_nest_start_noflag(ans_skb, NLBL_CIPSOV4_A_TAGLST); if (nla_a == NULL) { ret_val = -ENOMEM; goto list_failure_lock; @@ -517,7 +521,8 @@ list_start: switch (doi_def->type) { case CIPSO_V4_MAP_TRANS: - nla_a = nla_nest_start(ans_skb, NLBL_CIPSOV4_A_MLSLVLLST); + nla_a = nla_nest_start_noflag(ans_skb, + NLBL_CIPSOV4_A_MLSLVLLST); if (nla_a == NULL) { ret_val = -ENOMEM; goto list_failure_lock; @@ -529,7 +534,8 @@ list_start: CIPSO_V4_INV_LVL) continue; - nla_b = nla_nest_start(ans_skb, NLBL_CIPSOV4_A_MLSLVL); + nla_b = nla_nest_start_noflag(ans_skb, + NLBL_CIPSOV4_A_MLSLVL); if (nla_b == NULL) { ret_val = -ENOMEM; goto list_retry; @@ -548,7 +554,8 @@ list_start: } nla_nest_end(ans_skb, nla_a); - nla_a = nla_nest_start(ans_skb, NLBL_CIPSOV4_A_MLSCATLST); + nla_a = nla_nest_start_noflag(ans_skb, + NLBL_CIPSOV4_A_MLSCATLST); if (nla_a == NULL) { ret_val = -ENOMEM; goto list_retry; @@ -560,7 +567,8 @@ list_start: CIPSO_V4_INV_CAT) continue; - nla_b = nla_nest_start(ans_skb, NLBL_CIPSOV4_A_MLSCAT); + nla_b = nla_nest_start_noflag(ans_skb, + NLBL_CIPSOV4_A_MLSCAT); if (nla_b == NULL) { ret_val = -ENOMEM; goto list_retry; @@ -733,29 +741,29 @@ static int netlbl_cipsov4_remove(struct sk_buff *skb, struct genl_info *info) static const struct genl_ops netlbl_cipsov4_ops[] = { { .cmd = NLBL_CIPSOV4_C_ADD, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = netlbl_cipsov4_genl_policy, .doit = netlbl_cipsov4_add, .dumpit = NULL, }, { .cmd = NLBL_CIPSOV4_C_REMOVE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = netlbl_cipsov4_genl_policy, .doit = netlbl_cipsov4_remove, .dumpit = NULL, }, { .cmd = NLBL_CIPSOV4_C_LIST, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = 0, - .policy = netlbl_cipsov4_genl_policy, .doit = netlbl_cipsov4_list, .dumpit = NULL, }, { .cmd = NLBL_CIPSOV4_C_LISTALL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = 0, - .policy = netlbl_cipsov4_genl_policy, .doit = NULL, .dumpit = netlbl_cipsov4_listall, }, @@ -766,6 +774,7 @@ static struct genl_family netlbl_cipsov4_gnl_family __ro_after_init = { .name = NETLBL_NLTYPE_CIPSOV4_NAME, .version = NETLBL_PROTO_VERSION, .maxattr = NLBL_CIPSOV4_A_MAX, + .policy = netlbl_cipsov4_genl_policy, .module = THIS_MODULE, .ops = netlbl_cipsov4_ops, .n_ops = ARRAY_SIZE(netlbl_cipsov4_ops), diff --git a/net/netlabel/netlabel_mgmt.c b/net/netlabel/netlabel_mgmt.c index 21e0095b1d1422aff42e068490819fd3ac540c58..cae04f207782498d9d7244d1495ee79ecc1400c0 100644 --- a/net/netlabel/netlabel_mgmt.c +++ b/net/netlabel/netlabel_mgmt.c @@ -315,7 +315,7 @@ static int netlbl_mgmt_listentry(struct sk_buff *skb, switch (entry->def.type) { case NETLBL_NLTYPE_ADDRSELECT: - nla_a = nla_nest_start(skb, NLBL_MGMT_A_SELECTORLIST); + nla_a = nla_nest_start_noflag(skb, NLBL_MGMT_A_SELECTORLIST); if (nla_a == NULL) return -ENOMEM; @@ -323,7 +323,8 @@ static int netlbl_mgmt_listentry(struct sk_buff *skb, struct netlbl_domaddr4_map *map4; struct in_addr addr_struct; - nla_b = nla_nest_start(skb, NLBL_MGMT_A_ADDRSELECTOR); + nla_b = nla_nest_start_noflag(skb, + NLBL_MGMT_A_ADDRSELECTOR); if (nla_b == NULL) return -ENOMEM; @@ -357,7 +358,8 @@ static int netlbl_mgmt_listentry(struct sk_buff *skb, netlbl_af6list_foreach_rcu(iter6, &entry->def.addrsel->list6) { struct netlbl_domaddr6_map *map6; - nla_b = nla_nest_start(skb, NLBL_MGMT_A_ADDRSELECTOR); + nla_b = nla_nest_start_noflag(skb, + NLBL_MGMT_A_ADDRSELECTOR); if (nla_b == NULL) return -ENOMEM; @@ -772,57 +774,57 @@ version_failure: static const struct genl_ops netlbl_mgmt_genl_ops[] = { { .cmd = NLBL_MGMT_C_ADD, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = netlbl_mgmt_genl_policy, .doit = netlbl_mgmt_add, .dumpit = NULL, }, { .cmd = NLBL_MGMT_C_REMOVE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = netlbl_mgmt_genl_policy, .doit = netlbl_mgmt_remove, .dumpit = NULL, }, { .cmd = NLBL_MGMT_C_LISTALL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = 0, - .policy = netlbl_mgmt_genl_policy, .doit = NULL, .dumpit = netlbl_mgmt_listall, }, { .cmd = NLBL_MGMT_C_ADDDEF, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = netlbl_mgmt_genl_policy, .doit = netlbl_mgmt_adddef, .dumpit = NULL, }, { .cmd = NLBL_MGMT_C_REMOVEDEF, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = netlbl_mgmt_genl_policy, .doit = netlbl_mgmt_removedef, .dumpit = NULL, }, { .cmd = NLBL_MGMT_C_LISTDEF, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = 0, - .policy = netlbl_mgmt_genl_policy, .doit = netlbl_mgmt_listdef, .dumpit = NULL, }, { .cmd = NLBL_MGMT_C_PROTOCOLS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = 0, - .policy = netlbl_mgmt_genl_policy, .doit = NULL, .dumpit = netlbl_mgmt_protocols, }, { .cmd = NLBL_MGMT_C_VERSION, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = 0, - .policy = netlbl_mgmt_genl_policy, .doit = netlbl_mgmt_version, .dumpit = NULL, }, @@ -833,6 +835,7 @@ static struct genl_family netlbl_mgmt_gnl_family __ro_after_init = { .name = NETLBL_NLTYPE_MGMT_NAME, .version = NETLBL_PROTO_VERSION, .maxattr = NLBL_MGMT_A_MAX, + .policy = netlbl_mgmt_genl_policy, .module = THIS_MODULE, .ops = netlbl_mgmt_genl_ops, .n_ops = ARRAY_SIZE(netlbl_mgmt_genl_ops), diff --git a/net/netlabel/netlabel_unlabeled.c b/net/netlabel/netlabel_unlabeled.c index c92894c3e40a34b34bdf533b1712fd1c1909595e..b87dd34e183573032bb27f7993ae0068c199af10 100644 --- a/net/netlabel/netlabel_unlabeled.c +++ b/net/netlabel/netlabel_unlabeled.c @@ -1317,57 +1317,57 @@ unlabel_staticlistdef_return: static const struct genl_ops netlbl_unlabel_genl_ops[] = { { .cmd = NLBL_UNLABEL_C_STATICADD, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = netlbl_unlabel_genl_policy, .doit = netlbl_unlabel_staticadd, .dumpit = NULL, }, { .cmd = NLBL_UNLABEL_C_STATICREMOVE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = netlbl_unlabel_genl_policy, .doit = netlbl_unlabel_staticremove, .dumpit = NULL, }, { .cmd = NLBL_UNLABEL_C_STATICLIST, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = 0, - .policy = netlbl_unlabel_genl_policy, .doit = NULL, .dumpit = netlbl_unlabel_staticlist, }, { .cmd = NLBL_UNLABEL_C_STATICADDDEF, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = netlbl_unlabel_genl_policy, .doit = netlbl_unlabel_staticadddef, .dumpit = NULL, }, { .cmd = NLBL_UNLABEL_C_STATICREMOVEDEF, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = netlbl_unlabel_genl_policy, .doit = netlbl_unlabel_staticremovedef, .dumpit = NULL, }, { .cmd = NLBL_UNLABEL_C_STATICLISTDEF, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = 0, - .policy = netlbl_unlabel_genl_policy, .doit = NULL, .dumpit = netlbl_unlabel_staticlistdef, }, { .cmd = NLBL_UNLABEL_C_ACCEPT, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = netlbl_unlabel_genl_policy, .doit = netlbl_unlabel_accept, .dumpit = NULL, }, { .cmd = NLBL_UNLABEL_C_LIST, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = 0, - .policy = netlbl_unlabel_genl_policy, .doit = netlbl_unlabel_list, .dumpit = NULL, }, @@ -1378,6 +1378,7 @@ static struct genl_family netlbl_unlabel_gnl_family __ro_after_init = { .name = NETLBL_NLTYPE_UNLABELED_NAME, .version = NETLBL_PROTO_VERSION, .maxattr = NLBL_UNLABEL_A_MAX, + .policy = netlbl_unlabel_genl_policy, .module = THIS_MODULE, .ops = netlbl_unlabel_genl_ops, .n_ops = ARRAY_SIZE(netlbl_unlabel_genl_ops), diff --git a/net/netlink/genetlink.c b/net/netlink/genetlink.c index cb69d35c8e6adc31e07c07f58170c48b6d9e9c3a..efccd1ac9a666c6f344682081a720edd236c111a 100644 --- a/net/netlink/genetlink.c +++ b/net/netlink/genetlink.c @@ -536,6 +536,28 @@ static int genl_family_rcv_msg(const struct genl_family *family, if (ops->dumpit == NULL) return -EOPNOTSUPP; + if (!(ops->validate & GENL_DONT_VALIDATE_DUMP)) { + int hdrlen = GENL_HDRLEN + family->hdrsize; + + if (nlh->nlmsg_len < nlmsg_msg_size(hdrlen)) + return -EINVAL; + + if (family->maxattr) { + unsigned int validate = NL_VALIDATE_STRICT; + + if (ops->validate & + GENL_DONT_VALIDATE_DUMP_STRICT) + validate = NL_VALIDATE_LIBERAL; + rc = __nla_validate(nlmsg_attrdata(nlh, hdrlen), + nlmsg_attrlen(nlh, hdrlen), + family->maxattr, + family->policy, + validate, extack); + if (rc) + return rc; + } + } + if (!family->parallel_ops) { struct netlink_dump_control c = { .module = family->module, @@ -577,8 +599,13 @@ static int genl_family_rcv_msg(const struct genl_family *family, attrbuf = family->attrbuf; if (attrbuf) { - err = nlmsg_parse(nlh, hdrlen, attrbuf, family->maxattr, - ops->policy, extack); + enum netlink_validation validate = NL_VALIDATE_STRICT; + + if (ops->validate & GENL_DONT_VALIDATE_STRICT) + validate = NL_VALIDATE_LIBERAL; + + err = __nlmsg_parse(nlh, hdrlen, attrbuf, family->maxattr, + family->policy, validate, extack); if (err < 0) goto out; } @@ -665,7 +692,7 @@ static int ctrl_fill_info(const struct genl_family *family, u32 portid, u32 seq, struct nlattr *nla_ops; int i; - nla_ops = nla_nest_start(skb, CTRL_ATTR_OPS); + nla_ops = nla_nest_start_noflag(skb, CTRL_ATTR_OPS); if (nla_ops == NULL) goto nla_put_failure; @@ -678,10 +705,10 @@ static int ctrl_fill_info(const struct genl_family *family, u32 portid, u32 seq, op_flags |= GENL_CMD_CAP_DUMP; if (ops->doit) op_flags |= GENL_CMD_CAP_DO; - if (ops->policy) + if (family->policy) op_flags |= GENL_CMD_CAP_HASPOL; - nest = nla_nest_start(skb, i + 1); + nest = nla_nest_start_noflag(skb, i + 1); if (nest == NULL) goto nla_put_failure; @@ -699,7 +726,7 @@ static int ctrl_fill_info(const struct genl_family *family, u32 portid, u32 seq, struct nlattr *nla_grps; int i; - nla_grps = nla_nest_start(skb, CTRL_ATTR_MCAST_GROUPS); + nla_grps = nla_nest_start_noflag(skb, CTRL_ATTR_MCAST_GROUPS); if (nla_grps == NULL) goto nla_put_failure; @@ -709,7 +736,7 @@ static int ctrl_fill_info(const struct genl_family *family, u32 portid, u32 seq, grp = &family->mcgrps[i]; - nest = nla_nest_start(skb, i + 1); + nest = nla_nest_start_noflag(skb, i + 1); if (nest == NULL) goto nla_put_failure; @@ -749,11 +776,11 @@ static int ctrl_fill_mcgrp_info(const struct genl_family *family, nla_put_u16(skb, CTRL_ATTR_FAMILY_ID, family->id)) goto nla_put_failure; - nla_grps = nla_nest_start(skb, CTRL_ATTR_MCAST_GROUPS); + nla_grps = nla_nest_start_noflag(skb, CTRL_ATTR_MCAST_GROUPS); if (nla_grps == NULL) goto nla_put_failure; - nest = nla_nest_start(skb, 1); + nest = nla_nest_start_noflag(skb, 1); if (nest == NULL) goto nla_put_failure; @@ -938,9 +965,9 @@ static int genl_ctrl_event(int event, const struct genl_family *family, static const struct genl_ops genl_ctrl_ops[] = { { .cmd = CTRL_CMD_GETFAMILY, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = ctrl_getfamily, .dumpit = ctrl_dumpfamily, - .policy = ctrl_policy, }, }; @@ -958,6 +985,7 @@ static struct genl_family genl_ctrl __ro_after_init = { .name = "nlctrl", .version = 0x2, .maxattr = CTRL_ATTR_MAX, + .policy = ctrl_policy, .netnsok = true, }; diff --git a/net/netrom/af_netrom.c b/net/netrom/af_netrom.c index 71ffd1a6dc7c6063c00f4c82f985fe9fc0d80dc0..167c09e1ea90b2b4fd96e4c2a84915a800dafb01 100644 --- a/net/netrom/af_netrom.c +++ b/net/netrom/af_netrom.c @@ -1199,7 +1199,6 @@ static int nr_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) { struct sock *sk = sock->sk; void __user *argp = (void __user *)arg; - int ret; switch (cmd) { case TIOCOUTQ: { @@ -1225,18 +1224,6 @@ static int nr_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) return put_user(amount, (int __user *)argp); } - case SIOCGSTAMP: - lock_sock(sk); - ret = sock_get_timestamp(sk, argp); - release_sock(sk); - return ret; - - case SIOCGSTAMPNS: - lock_sock(sk); - ret = sock_get_timestampns(sk, argp); - release_sock(sk); - return ret; - case SIOCGIFADDR: case SIOCSIFADDR: case SIOCGIFDSTADDR: @@ -1362,6 +1349,7 @@ static const struct proto_ops nr_proto_ops = { .getname = nr_getname, .poll = datagram_poll, .ioctl = nr_ioctl, + .gettstamp = sock_gettstamp, .listen = nr_listen, .shutdown = sock_no_shutdown, .setsockopt = nr_setsockopt, diff --git a/net/nfc/netlink.c b/net/nfc/netlink.c index 376181cc1defc66f448106925d9188c41654cab4..04a8e47674ec14165ab6dd18928107e3748ade8e 100644 --- a/net/nfc/netlink.c +++ b/net/nfc/netlink.c @@ -119,9 +119,10 @@ static struct nfc_dev *__get_device_from_cb(struct netlink_callback *cb) int rc; u32 idx; - rc = nlmsg_parse(cb->nlh, GENL_HDRLEN + nfc_genl_family.hdrsize, - attrbuf, nfc_genl_family.maxattr, nfc_genl_policy, - NULL); + rc = nlmsg_parse_deprecated(cb->nlh, + GENL_HDRLEN + nfc_genl_family.hdrsize, + attrbuf, nfc_genl_family.maxattr, + nfc_genl_policy, NULL); if (rc < 0) return ERR_PTR(rc); @@ -392,7 +393,7 @@ int nfc_genl_llc_send_sdres(struct nfc_dev *dev, struct hlist_head *sdres_list) if (nla_put_u32(msg, NFC_ATTR_DEVICE_INDEX, dev->idx)) goto nla_put_failure; - sdp_attr = nla_nest_start(msg, NFC_ATTR_LLC_SDP); + sdp_attr = nla_nest_start_noflag(msg, NFC_ATTR_LLC_SDP); if (sdp_attr == NULL) { rc = -ENOMEM; goto nla_put_failure; @@ -402,7 +403,7 @@ int nfc_genl_llc_send_sdres(struct nfc_dev *dev, struct hlist_head *sdres_list) hlist_for_each_entry_safe(sdres, n, sdres_list, node) { pr_debug("uri: %s, sap: %d\n", sdres->uri, sdres->sap); - uri_attr = nla_nest_start(msg, i++); + uri_attr = nla_nest_start_noflag(msg, i++); if (uri_attr == NULL) { rc = -ENOMEM; goto nla_put_failure; @@ -1177,8 +1178,9 @@ static int nfc_genl_llc_sdreq(struct sk_buff *skb, struct genl_info *info) tlvs_len = 0; nla_for_each_nested(attr, info->attrs[NFC_ATTR_LLC_SDP], rem) { - rc = nla_parse_nested(sdp_attrs, NFC_SDP_ATTR_MAX, attr, - nfc_sdp_genl_policy, info->extack); + rc = nla_parse_nested_deprecated(sdp_attrs, NFC_SDP_ATTR_MAX, + attr, nfc_sdp_genl_policy, + info->extack); if (rc != 0) { rc = -EINVAL; @@ -1667,102 +1669,102 @@ EXPORT_SYMBOL(nfc_vendor_cmd_reply); static const struct genl_ops nfc_genl_ops[] = { { .cmd = NFC_CMD_GET_DEVICE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nfc_genl_get_device, .dumpit = nfc_genl_dump_devices, .done = nfc_genl_dump_devices_done, - .policy = nfc_genl_policy, }, { .cmd = NFC_CMD_DEV_UP, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nfc_genl_dev_up, - .policy = nfc_genl_policy, }, { .cmd = NFC_CMD_DEV_DOWN, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nfc_genl_dev_down, - .policy = nfc_genl_policy, }, { .cmd = NFC_CMD_START_POLL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nfc_genl_start_poll, - .policy = nfc_genl_policy, }, { .cmd = NFC_CMD_STOP_POLL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nfc_genl_stop_poll, - .policy = nfc_genl_policy, }, { .cmd = NFC_CMD_DEP_LINK_UP, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nfc_genl_dep_link_up, - .policy = nfc_genl_policy, }, { .cmd = NFC_CMD_DEP_LINK_DOWN, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nfc_genl_dep_link_down, - .policy = nfc_genl_policy, }, { .cmd = NFC_CMD_GET_TARGET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .dumpit = nfc_genl_dump_targets, .done = nfc_genl_dump_targets_done, - .policy = nfc_genl_policy, }, { .cmd = NFC_CMD_LLC_GET_PARAMS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nfc_genl_llc_get_params, - .policy = nfc_genl_policy, }, { .cmd = NFC_CMD_LLC_SET_PARAMS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nfc_genl_llc_set_params, - .policy = nfc_genl_policy, }, { .cmd = NFC_CMD_LLC_SDREQ, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nfc_genl_llc_sdreq, - .policy = nfc_genl_policy, }, { .cmd = NFC_CMD_FW_DOWNLOAD, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nfc_genl_fw_download, - .policy = nfc_genl_policy, }, { .cmd = NFC_CMD_ENABLE_SE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nfc_genl_enable_se, - .policy = nfc_genl_policy, }, { .cmd = NFC_CMD_DISABLE_SE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nfc_genl_disable_se, - .policy = nfc_genl_policy, }, { .cmd = NFC_CMD_GET_SE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .dumpit = nfc_genl_dump_ses, .done = nfc_genl_dump_ses_done, - .policy = nfc_genl_policy, }, { .cmd = NFC_CMD_SE_IO, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nfc_genl_se_io, - .policy = nfc_genl_policy, }, { .cmd = NFC_CMD_ACTIVATE_TARGET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nfc_genl_activate_target, - .policy = nfc_genl_policy, }, { .cmd = NFC_CMD_VENDOR, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nfc_genl_vendor_cmd, - .policy = nfc_genl_policy, }, { .cmd = NFC_CMD_DEACTIVATE_TARGET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nfc_genl_deactivate_target, - .policy = nfc_genl_policy, }, }; @@ -1771,6 +1773,7 @@ static struct genl_family nfc_genl_family __ro_after_init = { .name = NFC_GENL_NAME, .version = NFC_GENL_VERSION, .maxattr = NFC_ATTR_MAX, + .policy = nfc_genl_policy, .module = THIS_MODULE, .ops = nfc_genl_ops, .n_ops = ARRAY_SIZE(nfc_genl_ops), diff --git a/net/openvswitch/actions.c b/net/openvswitch/actions.c index e47ebbbe71b802b9428de3f824b8ce64265c031d..2c151bb322c1284edb40bcaef00c6ad6a29bc1a3 100644 --- a/net/openvswitch/actions.c +++ b/net/openvswitch/actions.c @@ -169,6 +169,10 @@ static int clone_execute(struct datapath *dp, struct sk_buff *skb, const struct nlattr *actions, int len, bool last, bool clone_flow_key); +static int do_execute_actions(struct datapath *dp, struct sk_buff *skb, + struct sw_flow_key *key, + const struct nlattr *attr, int len); + static void update_ethertype(struct sk_buff *skb, struct ethhdr *hdr, __be16 ethertype) { @@ -1213,6 +1217,40 @@ static int execute_recirc(struct datapath *dp, struct sk_buff *skb, return clone_execute(dp, skb, key, recirc_id, NULL, 0, last, true); } +static int execute_check_pkt_len(struct datapath *dp, struct sk_buff *skb, + struct sw_flow_key *key, + const struct nlattr *attr, bool last) +{ + const struct nlattr *actions, *cpl_arg; + const struct check_pkt_len_arg *arg; + int rem = nla_len(attr); + bool clone_flow_key; + + /* The first netlink attribute in 'attr' is always + * 'OVS_CHECK_PKT_LEN_ATTR_ARG'. + */ + cpl_arg = nla_data(attr); + arg = nla_data(cpl_arg); + + if (skb->len <= arg->pkt_len) { + /* Second netlink attribute in 'attr' is always + * 'OVS_CHECK_PKT_LEN_ATTR_ACTIONS_IF_LESS_EQUAL'. + */ + actions = nla_next(cpl_arg, &rem); + clone_flow_key = !arg->exec_for_lesser_equal; + } else { + /* Third netlink attribute in 'attr' is always + * 'OVS_CHECK_PKT_LEN_ATTR_ACTIONS_IF_GREATER'. + */ + actions = nla_next(cpl_arg, &rem); + actions = nla_next(actions, &rem); + clone_flow_key = !arg->exec_for_greater; + } + + return clone_execute(dp, skb, key, 0, nla_data(actions), + nla_len(actions), last, clone_flow_key); +} + /* Execute a list of actions against 'skb'. */ static int do_execute_actions(struct datapath *dp, struct sk_buff *skb, struct sw_flow_key *key, @@ -1374,6 +1412,16 @@ static int do_execute_actions(struct datapath *dp, struct sk_buff *skb, break; } + + case OVS_ACTION_ATTR_CHECK_PKT_LEN: { + bool last = nla_is_last(a, rem); + + err = execute_check_pkt_len(dp, skb, key, a, last); + if (last) + return err; + + break; + } } if (unlikely(err)) { diff --git a/net/openvswitch/conntrack.c b/net/openvswitch/conntrack.c index 1b6896896fff35ff8f3640ece128cf5a7e30c0dd..333ec5f298fe5fe80122dd59460c0a9c55758dfc 100644 --- a/net/openvswitch/conntrack.c +++ b/net/openvswitch/conntrack.c @@ -24,11 +24,12 @@ #include #include #include +#include #include #include #include -#ifdef CONFIG_NF_NAT_NEEDED +#if IS_ENABLED(CONFIG_NF_NAT) #include #endif @@ -73,7 +74,8 @@ struct ovs_conntrack_info { u32 eventmask; /* Mask of 1 << IPCT_*. */ struct md_mark mark; struct md_labels labels; -#ifdef CONFIG_NF_NAT_NEEDED + char timeout[CTNL_TIMEOUT_NAME_MAX]; +#if IS_ENABLED(CONFIG_NF_NAT) struct nf_nat_range2 range; /* Only present for SRC NAT and DST NAT. */ #endif }; @@ -719,7 +721,7 @@ static bool skb_nfct_cached(struct net *net, return ct_executed; } -#ifdef CONFIG_NF_NAT_NEEDED +#if IS_ENABLED(CONFIG_NF_NAT) /* Modelled after nf_nat_ipv[46]_fn(). * range is only used for new, uninitialized NAT state. * Returns either NF_ACCEPT or NF_DROP. @@ -901,7 +903,7 @@ static int ovs_ct_nat(struct net *net, struct sw_flow_key *key, return err; } -#else /* !CONFIG_NF_NAT_NEEDED */ +#else /* !CONFIG_NF_NAT */ static int ovs_ct_nat(struct net *net, struct sw_flow_key *key, const struct ovs_conntrack_info *info, struct sk_buff *skb, struct nf_conn *ct, @@ -990,6 +992,12 @@ static int __ovs_ct_lookup(struct net *net, struct sw_flow_key *key, GFP_ATOMIC); if (err) return err; + + /* helper installed, add seqadj if NAT is required */ + if (info->nat && !nfct_seqadj(ct)) { + if (!nfct_seqadj_ext_add(ct)) + return -EINVAL; + } } /* Call the helper only if: @@ -1299,6 +1307,7 @@ static int ovs_ct_add_helper(struct ovs_conntrack_info *info, const char *name, { struct nf_conntrack_helper *helper; struct nf_conn_help *help; + int ret = 0; helper = nf_conntrack_helper_try_module_get(name, info->family, key->ip.proto); @@ -1313,16 +1322,24 @@ static int ovs_ct_add_helper(struct ovs_conntrack_info *info, const char *name, return -ENOMEM; } +#ifdef CONFIG_NF_NAT_NEEDED + if (info->nat) { + ret = nf_nat_helper_try_module_get(name, info->family, + key->ip.proto); + if (ret) { + nf_conntrack_helper_put(helper); + OVS_NLERR(log, "Failed to load \"%s\" NAT helper, error: %d", + name, ret); + return ret; + } + } +#endif rcu_assign_pointer(help->helper, helper); info->helper = helper; - - if (info->nat) - request_module("ip_nat_%s", name); - - return 0; + return ret; } -#ifdef CONFIG_NF_NAT_NEEDED +#if IS_ENABLED(CONFIG_NF_NAT) static int parse_nat(const struct nlattr *attr, struct ovs_conntrack_info *info, bool log) { @@ -1459,12 +1476,14 @@ static const struct ovs_ct_len_tbl ovs_ct_attr_lens[OVS_CT_ATTR_MAX + 1] = { .maxlen = sizeof(struct md_labels) }, [OVS_CT_ATTR_HELPER] = { .minlen = 1, .maxlen = NF_CT_HELPER_NAME_LEN }, -#ifdef CONFIG_NF_NAT_NEEDED +#if IS_ENABLED(CONFIG_NF_NAT) /* NAT length is checked when parsing the nested attributes. */ [OVS_CT_ATTR_NAT] = { .minlen = 0, .maxlen = INT_MAX }, #endif [OVS_CT_ATTR_EVENTMASK] = { .minlen = sizeof(u32), .maxlen = sizeof(u32) }, + [OVS_CT_ATTR_TIMEOUT] = { .minlen = 1, + .maxlen = CTNL_TIMEOUT_NAME_MAX }, }; static int parse_ct(const struct nlattr *attr, struct ovs_conntrack_info *info, @@ -1537,7 +1556,7 @@ static int parse_ct(const struct nlattr *attr, struct ovs_conntrack_info *info, return -EINVAL; } break; -#ifdef CONFIG_NF_NAT_NEEDED +#if IS_ENABLED(CONFIG_NF_NAT) case OVS_CT_ATTR_NAT: { int err = parse_nat(a, info, log); @@ -1550,6 +1569,15 @@ static int parse_ct(const struct nlattr *attr, struct ovs_conntrack_info *info, info->have_eventmask = true; info->eventmask = nla_get_u32(a); break; +#ifdef CONFIG_NF_CONNTRACK_TIMEOUT + case OVS_CT_ATTR_TIMEOUT: + memcpy(info->timeout, nla_data(a), nla_len(a)); + if (!memchr(info->timeout, '\0', nla_len(a))) { + OVS_NLERR(log, "Invalid conntrack helper"); + return -EINVAL; + } + break; +#endif default: OVS_NLERR(log, "Unknown conntrack attr (%d)", @@ -1631,6 +1659,14 @@ int ovs_ct_copy_action(struct net *net, const struct nlattr *attr, OVS_NLERR(log, "Failed to allocate conntrack template"); return -ENOMEM; } + + if (ct_info.timeout[0]) { + if (nf_ct_set_timeout(net, ct_info.ct, family, key->ip.proto, + ct_info.timeout)) + pr_info_ratelimited("Failed to associated timeout " + "policy `%s'\n", ct_info.timeout); + } + if (helper) { err = ovs_ct_add_helper(&ct_info, helper, key, log); if (err) @@ -1650,13 +1686,13 @@ err_free_ct: return err; } -#ifdef CONFIG_NF_NAT_NEEDED +#if IS_ENABLED(CONFIG_NF_NAT) static bool ovs_ct_nat_to_attr(const struct ovs_conntrack_info *info, struct sk_buff *skb) { struct nlattr *start; - start = nla_nest_start(skb, OVS_CT_ATTR_NAT); + start = nla_nest_start_noflag(skb, OVS_CT_ATTR_NAT); if (!start) return false; @@ -1723,7 +1759,7 @@ int ovs_ct_action_to_attr(const struct ovs_conntrack_info *ct_info, { struct nlattr *start; - start = nla_nest_start(skb, OVS_ACTION_ATTR_CT); + start = nla_nest_start_noflag(skb, OVS_ACTION_ATTR_CT); if (!start) return -EMSGSIZE; @@ -1751,8 +1787,12 @@ int ovs_ct_action_to_attr(const struct ovs_conntrack_info *ct_info, if (ct_info->have_eventmask && nla_put_u32(skb, OVS_CT_ATTR_EVENTMASK, ct_info->eventmask)) return -EMSGSIZE; + if (ct_info->timeout[0]) { + if (nla_put_string(skb, OVS_CT_ATTR_TIMEOUT, ct_info->timeout)) + return -EMSGSIZE; + } -#ifdef CONFIG_NF_NAT_NEEDED +#if IS_ENABLED(CONFIG_NF_NAT) if (ct_info->nat && !ovs_ct_nat_to_attr(ct_info, skb)) return -EMSGSIZE; #endif @@ -1770,10 +1810,18 @@ void ovs_ct_free_action(const struct nlattr *a) static void __ovs_ct_free_action(struct ovs_conntrack_info *ct_info) { - if (ct_info->helper) + if (ct_info->helper) { +#ifdef CONFIG_NF_NAT_NEEDED + if (ct_info->nat) + nf_nat_helper_put(ct_info->helper); +#endif nf_conntrack_helper_put(ct_info->helper); - if (ct_info->ct) + } + if (ct_info->ct) { + if (ct_info->timeout[0]) + nf_ct_destroy_timeout(ct_info->ct); nf_ct_tmpl_free(ct_info->ct); + } } #if IS_ENABLED(CONFIG_NETFILTER_CONNCOUNT) @@ -2126,7 +2174,11 @@ static int ovs_ct_limit_cmd_get(struct sk_buff *skb, struct genl_info *info) if (IS_ERR(reply)) return PTR_ERR(reply); - nla_reply = nla_nest_start(reply, OVS_CT_LIMIT_ATTR_ZONE_LIMIT); + nla_reply = nla_nest_start_noflag(reply, OVS_CT_LIMIT_ATTR_ZONE_LIMIT); + if (!nla_reply) { + err = -EMSGSIZE; + goto exit_err; + } if (a[OVS_CT_LIMIT_ATTR_ZONE_LIMIT]) { err = ovs_ct_limit_get_zone_limit( @@ -2152,20 +2204,20 @@ exit_err: static struct genl_ops ct_limit_genl_ops[] = { { .cmd = OVS_CT_LIMIT_CMD_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, /* Requires CAP_NET_ADMIN * privilege. */ - .policy = ct_limit_policy, .doit = ovs_ct_limit_cmd_set, }, { .cmd = OVS_CT_LIMIT_CMD_DEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, /* Requires CAP_NET_ADMIN * privilege. */ - .policy = ct_limit_policy, .doit = ovs_ct_limit_cmd_del, }, { .cmd = OVS_CT_LIMIT_CMD_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = 0, /* OK for unprivileged users. */ - .policy = ct_limit_policy, .doit = ovs_ct_limit_cmd_get, }, }; @@ -2179,6 +2231,7 @@ struct genl_family dp_ct_limit_genl_family __ro_after_init = { .name = OVS_CT_LIMIT_FAMILY, .version = OVS_CT_LIMIT_VERSION, .maxattr = OVS_CT_LIMIT_ATTR_MAX, + .policy = ct_limit_policy, .netnsok = true, .parallel_ops = true, .ops = ct_limit_genl_ops, diff --git a/net/openvswitch/datapath.c b/net/openvswitch/datapath.c index 9dd158ab51b310e28354237ff3bcd101a8b829f5..dc9ff936722102264d3fd1631d0f886928ee9b7f 100644 --- a/net/openvswitch/datapath.c +++ b/net/openvswitch/datapath.c @@ -455,7 +455,8 @@ static int queue_userspace_packet(struct datapath *dp, struct sk_buff *skb, upcall->dp_ifindex = dp_ifindex; err = ovs_nla_put_key(key, key, OVS_PACKET_ATTR_KEY, false, user_skb); - BUG_ON(err); + if (err) + goto out; if (upcall_info->userdata) __nla_put(user_skb, OVS_PACKET_ATTR_USERDATA, @@ -463,19 +464,22 @@ static int queue_userspace_packet(struct datapath *dp, struct sk_buff *skb, nla_data(upcall_info->userdata)); if (upcall_info->egress_tun_info) { - nla = nla_nest_start(user_skb, OVS_PACKET_ATTR_EGRESS_TUN_KEY); + nla = nla_nest_start_noflag(user_skb, + OVS_PACKET_ATTR_EGRESS_TUN_KEY); if (!nla) { err = -EMSGSIZE; goto out; } err = ovs_nla_put_tunnel_info(user_skb, upcall_info->egress_tun_info); - BUG_ON(err); + if (err) + goto out; + nla_nest_end(user_skb, nla); } if (upcall_info->actions_len) { - nla = nla_nest_start(user_skb, OVS_PACKET_ATTR_ACTIONS); + nla = nla_nest_start_noflag(user_skb, OVS_PACKET_ATTR_ACTIONS); if (!nla) { err = -EMSGSIZE; goto out; @@ -638,8 +642,8 @@ static const struct nla_policy packet_policy[OVS_PACKET_ATTR_MAX + 1] = { static const struct genl_ops dp_packet_genl_ops[] = { { .cmd = OVS_PACKET_CMD_EXECUTE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_UNS_ADMIN_PERM, /* Requires CAP_NET_ADMIN privilege. */ - .policy = packet_policy, .doit = ovs_packet_cmd_execute } }; @@ -649,6 +653,7 @@ static struct genl_family dp_packet_genl_family __ro_after_init = { .name = OVS_PACKET_FAMILY, .version = OVS_PACKET_VERSION, .maxattr = OVS_PACKET_ATTR_MAX, + .policy = packet_policy, .netnsok = true, .parallel_ops = true, .ops = dp_packet_genl_ops, @@ -776,7 +781,7 @@ static int ovs_flow_cmd_fill_actions(const struct sw_flow *flow, * This can only fail for dump operations because the skb is always * properly sized for single flows. */ - start = nla_nest_start(skb, OVS_FLOW_ATTR_ACTIONS); + start = nla_nest_start_noflag(skb, OVS_FLOW_ATTR_ACTIONS); if (start) { const struct sw_flow_actions *sf_acts; @@ -1374,8 +1379,8 @@ static int ovs_flow_cmd_dump(struct sk_buff *skb, struct netlink_callback *cb) u32 ufid_flags; int err; - err = genlmsg_parse(cb->nlh, &dp_flow_genl_family, a, - OVS_FLOW_ATTR_MAX, flow_policy, NULL); + err = genlmsg_parse_deprecated(cb->nlh, &dp_flow_genl_family, a, + OVS_FLOW_ATTR_MAX, flow_policy, NULL); if (err) return err; ufid_flags = ovs_nla_get_ufid_flags(a[OVS_FLOW_ATTR_UFID_FLAGS]); @@ -1423,24 +1428,24 @@ static const struct nla_policy flow_policy[OVS_FLOW_ATTR_MAX + 1] = { static const struct genl_ops dp_flow_genl_ops[] = { { .cmd = OVS_FLOW_CMD_NEW, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_UNS_ADMIN_PERM, /* Requires CAP_NET_ADMIN privilege. */ - .policy = flow_policy, .doit = ovs_flow_cmd_new }, { .cmd = OVS_FLOW_CMD_DEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_UNS_ADMIN_PERM, /* Requires CAP_NET_ADMIN privilege. */ - .policy = flow_policy, .doit = ovs_flow_cmd_del }, { .cmd = OVS_FLOW_CMD_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = 0, /* OK for unprivileged users. */ - .policy = flow_policy, .doit = ovs_flow_cmd_get, .dumpit = ovs_flow_cmd_dump }, { .cmd = OVS_FLOW_CMD_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_UNS_ADMIN_PERM, /* Requires CAP_NET_ADMIN privilege. */ - .policy = flow_policy, .doit = ovs_flow_cmd_set, }, }; @@ -1450,6 +1455,7 @@ static struct genl_family dp_flow_genl_family __ro_after_init = { .name = OVS_FLOW_FAMILY, .version = OVS_FLOW_VERSION, .maxattr = OVS_FLOW_ATTR_MAX, + .policy = flow_policy, .netnsok = true, .parallel_ops = true, .ops = dp_flow_genl_ops, @@ -1816,24 +1822,24 @@ static const struct nla_policy datapath_policy[OVS_DP_ATTR_MAX + 1] = { static const struct genl_ops dp_datapath_genl_ops[] = { { .cmd = OVS_DP_CMD_NEW, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_UNS_ADMIN_PERM, /* Requires CAP_NET_ADMIN privilege. */ - .policy = datapath_policy, .doit = ovs_dp_cmd_new }, { .cmd = OVS_DP_CMD_DEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_UNS_ADMIN_PERM, /* Requires CAP_NET_ADMIN privilege. */ - .policy = datapath_policy, .doit = ovs_dp_cmd_del }, { .cmd = OVS_DP_CMD_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = 0, /* OK for unprivileged users. */ - .policy = datapath_policy, .doit = ovs_dp_cmd_get, .dumpit = ovs_dp_cmd_dump }, { .cmd = OVS_DP_CMD_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_UNS_ADMIN_PERM, /* Requires CAP_NET_ADMIN privilege. */ - .policy = datapath_policy, .doit = ovs_dp_cmd_set, }, }; @@ -1843,6 +1849,7 @@ static struct genl_family dp_datapath_genl_family __ro_after_init = { .name = OVS_DATAPATH_FAMILY, .version = OVS_DATAPATH_VERSION, .maxattr = OVS_DP_ATTR_MAX, + .policy = datapath_policy, .netnsok = true, .parallel_ops = true, .ops = dp_datapath_genl_ops, @@ -2259,24 +2266,24 @@ static const struct nla_policy vport_policy[OVS_VPORT_ATTR_MAX + 1] = { static const struct genl_ops dp_vport_genl_ops[] = { { .cmd = OVS_VPORT_CMD_NEW, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_UNS_ADMIN_PERM, /* Requires CAP_NET_ADMIN privilege. */ - .policy = vport_policy, .doit = ovs_vport_cmd_new }, { .cmd = OVS_VPORT_CMD_DEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_UNS_ADMIN_PERM, /* Requires CAP_NET_ADMIN privilege. */ - .policy = vport_policy, .doit = ovs_vport_cmd_del }, { .cmd = OVS_VPORT_CMD_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = 0, /* OK for unprivileged users. */ - .policy = vport_policy, .doit = ovs_vport_cmd_get, .dumpit = ovs_vport_cmd_dump }, { .cmd = OVS_VPORT_CMD_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_UNS_ADMIN_PERM, /* Requires CAP_NET_ADMIN privilege. */ - .policy = vport_policy, .doit = ovs_vport_cmd_set, }, }; @@ -2286,6 +2293,7 @@ struct genl_family dp_vport_genl_family __ro_after_init = { .name = OVS_VPORT_FAMILY, .version = OVS_VPORT_VERSION, .maxattr = OVS_VPORT_ATTR_MAX, + .policy = vport_policy, .netnsok = true, .parallel_ops = true, .ops = dp_vport_genl_ops, diff --git a/net/openvswitch/flow_netlink.c b/net/openvswitch/flow_netlink.c index 4bdf5e3ac2087a67e715ebd720159205697668a8..54eb80dd2dc6f099bd903aefe58bf97ba875be4c 100644 --- a/net/openvswitch/flow_netlink.c +++ b/net/openvswitch/flow_netlink.c @@ -91,6 +91,7 @@ static bool actions_may_change_flow(const struct nlattr *actions) case OVS_ACTION_ATTR_SET: case OVS_ACTION_ATTR_SET_MASKED: case OVS_ACTION_ATTR_METER: + case OVS_ACTION_ATTR_CHECK_PKT_LEN: default: return true; } @@ -403,6 +404,7 @@ static const struct ovs_len_tbl ovs_tunnel_key_lens[OVS_TUNNEL_KEY_ATTR_MAX + 1] [OVS_TUNNEL_KEY_ATTR_IPV6_SRC] = { .len = sizeof(struct in6_addr) }, [OVS_TUNNEL_KEY_ATTR_IPV6_DST] = { .len = sizeof(struct in6_addr) }, [OVS_TUNNEL_KEY_ATTR_ERSPAN_OPTS] = { .len = OVS_ATTR_VARIABLE }, + [OVS_TUNNEL_KEY_ATTR_IPV4_INFO_BRIDGE] = { .len = 0 }, }; static const struct ovs_len_tbl @@ -666,6 +668,7 @@ static int ip_tun_from_nlattr(const struct nlattr *attr, bool log) { bool ttl = false, ipv4 = false, ipv6 = false; + bool info_bridge_mode = false; __be16 tun_flags = 0; int opts_type = 0; struct nlattr *a; @@ -782,6 +785,10 @@ static int ip_tun_from_nlattr(const struct nlattr *attr, tun_flags |= TUNNEL_ERSPAN_OPT; opts_type = type; break; + case OVS_TUNNEL_KEY_ATTR_IPV4_INFO_BRIDGE: + info_bridge_mode = true; + ipv4 = true; + break; default: OVS_NLERR(log, "Unknown IP tunnel attribute %d", type); @@ -812,16 +819,29 @@ static int ip_tun_from_nlattr(const struct nlattr *attr, OVS_NLERR(log, "IP tunnel dst address not specified"); return -EINVAL; } - if (ipv4 && !match->key->tun_key.u.ipv4.dst) { - OVS_NLERR(log, "IPv4 tunnel dst address is zero"); - return -EINVAL; + if (ipv4) { + if (info_bridge_mode) { + if (match->key->tun_key.u.ipv4.src || + match->key->tun_key.u.ipv4.dst || + match->key->tun_key.tp_src || + match->key->tun_key.tp_dst || + match->key->tun_key.ttl || + match->key->tun_key.tos || + tun_flags & ~TUNNEL_KEY) { + OVS_NLERR(log, "IPv4 tun info is not correct"); + return -EINVAL; + } + } else if (!match->key->tun_key.u.ipv4.dst) { + OVS_NLERR(log, "IPv4 tunnel dst address is zero"); + return -EINVAL; + } } if (ipv6 && ipv6_addr_any(&match->key->tun_key.u.ipv6.dst)) { OVS_NLERR(log, "IPv6 tunnel dst address is zero"); return -EINVAL; } - if (!ttl) { + if (!ttl && !info_bridge_mode) { OVS_NLERR(log, "IP tunnel TTL not specified."); return -EINVAL; } @@ -836,7 +856,7 @@ static int vxlan_opt_to_nlattr(struct sk_buff *skb, const struct vxlan_metadata *opts = tun_opts; struct nlattr *nla; - nla = nla_nest_start(skb, OVS_TUNNEL_KEY_ATTR_VXLAN_OPTS); + nla = nla_nest_start_noflag(skb, OVS_TUNNEL_KEY_ATTR_VXLAN_OPTS); if (!nla) return -EMSGSIZE; @@ -850,12 +870,17 @@ static int vxlan_opt_to_nlattr(struct sk_buff *skb, static int __ip_tun_to_nlattr(struct sk_buff *skb, const struct ip_tunnel_key *output, const void *tun_opts, int swkey_tun_opts_len, - unsigned short tun_proto) + unsigned short tun_proto, u8 mode) { if (output->tun_flags & TUNNEL_KEY && nla_put_be64(skb, OVS_TUNNEL_KEY_ATTR_ID, output->tun_id, OVS_TUNNEL_KEY_ATTR_PAD)) return -EMSGSIZE; + + if (mode & IP_TUNNEL_INFO_BRIDGE) + return nla_put_flag(skb, OVS_TUNNEL_KEY_ATTR_IPV4_INFO_BRIDGE) + ? -EMSGSIZE : 0; + switch (tun_proto) { case AF_INET: if (output->u.ipv4.src && @@ -918,17 +943,17 @@ static int __ip_tun_to_nlattr(struct sk_buff *skb, static int ip_tun_to_nlattr(struct sk_buff *skb, const struct ip_tunnel_key *output, const void *tun_opts, int swkey_tun_opts_len, - unsigned short tun_proto) + unsigned short tun_proto, u8 mode) { struct nlattr *nla; int err; - nla = nla_nest_start(skb, OVS_KEY_ATTR_TUNNEL); + nla = nla_nest_start_noflag(skb, OVS_KEY_ATTR_TUNNEL); if (!nla) return -EMSGSIZE; err = __ip_tun_to_nlattr(skb, output, tun_opts, swkey_tun_opts_len, - tun_proto); + tun_proto, mode); if (err) return err; @@ -942,7 +967,7 @@ int ovs_nla_put_tunnel_info(struct sk_buff *skb, return __ip_tun_to_nlattr(skb, &tun_info->key, ip_tunnel_info_opts(tun_info), tun_info->options_len, - ip_tunnel_info_af(tun_info)); + ip_tunnel_info_af(tun_info), tun_info->mode); } static int encode_vlan_from_nlattrs(struct sw_flow_match *match, @@ -1932,7 +1957,7 @@ static int nsh_key_to_nlattr(const struct ovs_key_nsh *nsh, bool is_mask, { struct nlattr *start; - start = nla_nest_start(skb, OVS_KEY_ATTR_NSH); + start = nla_nest_start_noflag(skb, OVS_KEY_ATTR_NSH); if (!start) return -EMSGSIZE; @@ -1980,7 +2005,7 @@ static int __ovs_nla_put_key(const struct sw_flow_key *swkey, opts = TUN_METADATA_OPTS(output, swkey->tun_opts_len); if (ip_tun_to_nlattr(skb, &output->tun_key, opts, - swkey->tun_opts_len, swkey->tun_proto)) + swkey->tun_opts_len, swkey->tun_proto, 0)) goto nla_put_failure; } @@ -2015,14 +2040,15 @@ static int __ovs_nla_put_key(const struct sw_flow_key *swkey, if (swkey->eth.vlan.tci || eth_type_vlan(swkey->eth.type)) { if (ovs_nla_put_vlan(skb, &output->eth.vlan, is_mask)) goto nla_put_failure; - encap = nla_nest_start(skb, OVS_KEY_ATTR_ENCAP); + encap = nla_nest_start_noflag(skb, OVS_KEY_ATTR_ENCAP); if (!swkey->eth.vlan.tci) goto unencap; if (swkey->eth.cvlan.tci || eth_type_vlan(swkey->eth.type)) { if (ovs_nla_put_vlan(skb, &output->eth.cvlan, is_mask)) goto nla_put_failure; - in_encap = nla_nest_start(skb, OVS_KEY_ATTR_ENCAP); + in_encap = nla_nest_start_noflag(skb, + OVS_KEY_ATTR_ENCAP); if (!swkey->eth.cvlan.tci) goto unencap; } @@ -2201,7 +2227,7 @@ int ovs_nla_put_key(const struct sw_flow_key *swkey, int err; struct nlattr *nla; - nla = nla_nest_start(skb, attr); + nla = nla_nest_start_noflag(skb, attr); if (!nla) return -EMSGSIZE; err = __ovs_nla_put_key(swkey, output, is_mask, skb); @@ -2605,6 +2631,8 @@ static int validate_and_copy_set_tun(const struct nlattr *attr, tun_info->mode = IP_TUNNEL_INFO_TX; if (key.tun_proto == AF_INET6) tun_info->mode |= IP_TUNNEL_INFO_IPV6; + else if (key.tun_proto == AF_INET && key.tun_key.u.ipv4.dst == 0) + tun_info->mode |= IP_TUNNEL_INFO_BRIDGE; tun_info->key = key.tun_key; /* We need to store the options in the action itself since @@ -2826,8 +2854,8 @@ static int validate_userspace(const struct nlattr *attr) struct nlattr *a[OVS_USERSPACE_ATTR_MAX + 1]; int error; - error = nla_parse_nested(a, OVS_USERSPACE_ATTR_MAX, attr, - userspace_policy, NULL); + error = nla_parse_nested_deprecated(a, OVS_USERSPACE_ATTR_MAX, attr, + userspace_policy, NULL); if (error) return error; @@ -2838,6 +2866,88 @@ static int validate_userspace(const struct nlattr *attr) return 0; } +static const struct nla_policy cpl_policy[OVS_CHECK_PKT_LEN_ATTR_MAX + 1] = { + [OVS_CHECK_PKT_LEN_ATTR_PKT_LEN] = {.type = NLA_U16 }, + [OVS_CHECK_PKT_LEN_ATTR_ACTIONS_IF_GREATER] = {.type = NLA_NESTED }, + [OVS_CHECK_PKT_LEN_ATTR_ACTIONS_IF_LESS_EQUAL] = {.type = NLA_NESTED }, +}; + +static int validate_and_copy_check_pkt_len(struct net *net, + const struct nlattr *attr, + const struct sw_flow_key *key, + struct sw_flow_actions **sfa, + __be16 eth_type, __be16 vlan_tci, + bool log, bool last) +{ + const struct nlattr *acts_if_greater, *acts_if_lesser_eq; + struct nlattr *a[OVS_CHECK_PKT_LEN_ATTR_MAX + 1]; + struct check_pkt_len_arg arg; + int nested_acts_start; + int start, err; + + err = nla_parse_deprecated_strict(a, OVS_CHECK_PKT_LEN_ATTR_MAX, + nla_data(attr), nla_len(attr), + cpl_policy, NULL); + if (err) + return err; + + if (!a[OVS_CHECK_PKT_LEN_ATTR_PKT_LEN] || + !nla_get_u16(a[OVS_CHECK_PKT_LEN_ATTR_PKT_LEN])) + return -EINVAL; + + acts_if_lesser_eq = a[OVS_CHECK_PKT_LEN_ATTR_ACTIONS_IF_LESS_EQUAL]; + acts_if_greater = a[OVS_CHECK_PKT_LEN_ATTR_ACTIONS_IF_GREATER]; + + /* Both the nested action should be present. */ + if (!acts_if_greater || !acts_if_lesser_eq) + return -EINVAL; + + /* validation done, copy the nested actions. */ + start = add_nested_action_start(sfa, OVS_ACTION_ATTR_CHECK_PKT_LEN, + log); + if (start < 0) + return start; + + arg.pkt_len = nla_get_u16(a[OVS_CHECK_PKT_LEN_ATTR_PKT_LEN]); + arg.exec_for_lesser_equal = + last || !actions_may_change_flow(acts_if_lesser_eq); + arg.exec_for_greater = + last || !actions_may_change_flow(acts_if_greater); + + err = ovs_nla_add_action(sfa, OVS_CHECK_PKT_LEN_ATTR_ARG, &arg, + sizeof(arg), log); + if (err) + return err; + + nested_acts_start = add_nested_action_start(sfa, + OVS_CHECK_PKT_LEN_ATTR_ACTIONS_IF_LESS_EQUAL, log); + if (nested_acts_start < 0) + return nested_acts_start; + + err = __ovs_nla_copy_actions(net, acts_if_lesser_eq, key, sfa, + eth_type, vlan_tci, log); + + if (err) + return err; + + add_nested_action_end(*sfa, nested_acts_start); + + nested_acts_start = add_nested_action_start(sfa, + OVS_CHECK_PKT_LEN_ATTR_ACTIONS_IF_GREATER, log); + if (nested_acts_start < 0) + return nested_acts_start; + + err = __ovs_nla_copy_actions(net, acts_if_greater, key, sfa, + eth_type, vlan_tci, log); + + if (err) + return err; + + add_nested_action_end(*sfa, nested_acts_start); + add_nested_action_end(*sfa, start); + return 0; +} + static int copy_action(const struct nlattr *from, struct sw_flow_actions **sfa, bool log) { @@ -2884,6 +2994,7 @@ static int __ovs_nla_copy_actions(struct net *net, const struct nlattr *attr, [OVS_ACTION_ATTR_POP_NSH] = 0, [OVS_ACTION_ATTR_METER] = sizeof(u32), [OVS_ACTION_ATTR_CLONE] = (u32)-1, + [OVS_ACTION_ATTR_CHECK_PKT_LEN] = (u32)-1, }; const struct ovs_action_push_vlan *vlan; int type = nla_type(a); @@ -3085,6 +3196,19 @@ static int __ovs_nla_copy_actions(struct net *net, const struct nlattr *attr, break; } + case OVS_ACTION_ATTR_CHECK_PKT_LEN: { + bool last = nla_is_last(a, rem); + + err = validate_and_copy_check_pkt_len(net, a, key, sfa, + eth_type, + vlan_tci, log, + last); + if (err) + return err; + skip_copy = true; + break; + } + default: OVS_NLERR(log, "Unknown Action type %d", type); return -EINVAL; @@ -3130,7 +3254,7 @@ static int sample_action_to_attr(const struct nlattr *attr, const struct sample_arg *arg; struct nlattr *actions; - start = nla_nest_start(skb, OVS_ACTION_ATTR_SAMPLE); + start = nla_nest_start_noflag(skb, OVS_ACTION_ATTR_SAMPLE); if (!start) return -EMSGSIZE; @@ -3143,7 +3267,7 @@ static int sample_action_to_attr(const struct nlattr *attr, goto out; } - ac_start = nla_nest_start(skb, OVS_SAMPLE_ATTR_ACTIONS); + ac_start = nla_nest_start_noflag(skb, OVS_SAMPLE_ATTR_ACTIONS); if (!ac_start) { err = -EMSGSIZE; goto out; @@ -3169,7 +3293,7 @@ static int clone_action_to_attr(const struct nlattr *attr, struct nlattr *start; int err = 0, rem = nla_len(attr); - start = nla_nest_start(skb, OVS_ACTION_ATTR_CLONE); + start = nla_nest_start_noflag(skb, OVS_ACTION_ATTR_CLONE); if (!start) return -EMSGSIZE; @@ -3183,6 +3307,75 @@ static int clone_action_to_attr(const struct nlattr *attr, return err; } +static int check_pkt_len_action_to_attr(const struct nlattr *attr, + struct sk_buff *skb) +{ + struct nlattr *start, *ac_start = NULL; + const struct check_pkt_len_arg *arg; + const struct nlattr *a, *cpl_arg; + int err = 0, rem = nla_len(attr); + + start = nla_nest_start_noflag(skb, OVS_ACTION_ATTR_CHECK_PKT_LEN); + if (!start) + return -EMSGSIZE; + + /* The first nested attribute in 'attr' is always + * 'OVS_CHECK_PKT_LEN_ATTR_ARG'. + */ + cpl_arg = nla_data(attr); + arg = nla_data(cpl_arg); + + if (nla_put_u16(skb, OVS_CHECK_PKT_LEN_ATTR_PKT_LEN, arg->pkt_len)) { + err = -EMSGSIZE; + goto out; + } + + /* Second nested attribute in 'attr' is always + * 'OVS_CHECK_PKT_LEN_ATTR_ACTIONS_IF_LESS_EQUAL'. + */ + a = nla_next(cpl_arg, &rem); + ac_start = nla_nest_start_noflag(skb, + OVS_CHECK_PKT_LEN_ATTR_ACTIONS_IF_LESS_EQUAL); + if (!ac_start) { + err = -EMSGSIZE; + goto out; + } + + err = ovs_nla_put_actions(nla_data(a), nla_len(a), skb); + if (err) { + nla_nest_cancel(skb, ac_start); + goto out; + } else { + nla_nest_end(skb, ac_start); + } + + /* Third nested attribute in 'attr' is always + * OVS_CHECK_PKT_LEN_ATTR_ACTIONS_IF_GREATER. + */ + a = nla_next(a, &rem); + ac_start = nla_nest_start_noflag(skb, + OVS_CHECK_PKT_LEN_ATTR_ACTIONS_IF_GREATER); + if (!ac_start) { + err = -EMSGSIZE; + goto out; + } + + err = ovs_nla_put_actions(nla_data(a), nla_len(a), skb); + if (err) { + nla_nest_cancel(skb, ac_start); + goto out; + } else { + nla_nest_end(skb, ac_start); + } + + nla_nest_end(skb, start); + return 0; + +out: + nla_nest_cancel(skb, start); + return err; +} + static int set_action_to_attr(const struct nlattr *a, struct sk_buff *skb) { const struct nlattr *ovs_key = nla_data(a); @@ -3195,14 +3388,14 @@ static int set_action_to_attr(const struct nlattr *a, struct sk_buff *skb) struct ovs_tunnel_info *ovs_tun = nla_data(ovs_key); struct ip_tunnel_info *tun_info = &ovs_tun->tun_dst->u.tun_info; - start = nla_nest_start(skb, OVS_ACTION_ATTR_SET); + start = nla_nest_start_noflag(skb, OVS_ACTION_ATTR_SET); if (!start) return -EMSGSIZE; err = ip_tun_to_nlattr(skb, &tun_info->key, ip_tunnel_info_opts(tun_info), tun_info->options_len, - ip_tunnel_info_af(tun_info)); + ip_tunnel_info_af(tun_info), tun_info->mode); if (err) return err; nla_nest_end(skb, start); @@ -3227,7 +3420,7 @@ static int masked_set_action_to_set_action_attr(const struct nlattr *a, /* Revert the conversion we did from a non-masked set action to * masked set action. */ - nla = nla_nest_start(skb, OVS_ACTION_ATTR_SET); + nla = nla_nest_start_noflag(skb, OVS_ACTION_ATTR_SET); if (!nla) return -EMSGSIZE; @@ -3277,6 +3470,12 @@ int ovs_nla_put_actions(const struct nlattr *attr, int len, struct sk_buff *skb) return err; break; + case OVS_ACTION_ATTR_CHECK_PKT_LEN: + err = check_pkt_len_action_to_attr(a, skb); + if (err) + return err; + break; + default: if (nla_put(skb, type, nla_len(a), nla_data(a))) return -EMSGSIZE; diff --git a/net/openvswitch/meter.c b/net/openvswitch/meter.c index 43849d752a1e72e5bcefa74d72ca78a84fd97a12..bb67238f0340fdcc8cb80be190ab81913e28c6c5 100644 --- a/net/openvswitch/meter.c +++ b/net/openvswitch/meter.c @@ -127,7 +127,7 @@ static int ovs_meter_cmd_reply_stats(struct sk_buff *reply, u32 meter_id, OVS_METER_ATTR_PAD)) goto error; - nla = nla_nest_start(reply, OVS_METER_ATTR_BANDS); + nla = nla_nest_start_noflag(reply, OVS_METER_ATTR_BANDS); if (!nla) goto error; @@ -136,7 +136,7 @@ static int ovs_meter_cmd_reply_stats(struct sk_buff *reply, u32 meter_id, for (i = 0; i < meter->n_bands; ++i, ++band) { struct nlattr *band_nla; - band_nla = nla_nest_start(reply, OVS_BAND_ATTR_UNSPEC); + band_nla = nla_nest_start_noflag(reply, OVS_BAND_ATTR_UNSPEC); if (!band_nla || nla_put(reply, OVS_BAND_ATTR_STATS, sizeof(struct ovs_flow_stats), &band->stats)) @@ -166,11 +166,11 @@ static int ovs_meter_cmd_features(struct sk_buff *skb, struct genl_info *info) nla_put_u32(reply, OVS_METER_ATTR_MAX_BANDS, DP_MAX_BANDS)) goto nla_put_failure; - nla = nla_nest_start(reply, OVS_METER_ATTR_BANDS); + nla = nla_nest_start_noflag(reply, OVS_METER_ATTR_BANDS); if (!nla) goto nla_put_failure; - band_nla = nla_nest_start(reply, OVS_BAND_ATTR_UNSPEC); + band_nla = nla_nest_start_noflag(reply, OVS_BAND_ATTR_UNSPEC); if (!band_nla) goto nla_put_failure; /* Currently only DROP band type is supported. */ @@ -227,9 +227,9 @@ static struct dp_meter *dp_meter_create(struct nlattr **a) struct nlattr *attr[OVS_BAND_ATTR_MAX + 1]; u32 band_max_delta_t; - err = nla_parse((struct nlattr **)&attr, OVS_BAND_ATTR_MAX, - nla_data(nla), nla_len(nla), band_policy, - NULL); + err = nla_parse_deprecated((struct nlattr **)&attr, + OVS_BAND_ATTR_MAX, nla_data(nla), + nla_len(nla), band_policy, NULL); if (err) goto exit_free_meter; @@ -526,27 +526,27 @@ bool ovs_meter_execute(struct datapath *dp, struct sk_buff *skb, static struct genl_ops dp_meter_genl_ops[] = { { .cmd = OVS_METER_CMD_FEATURES, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = 0, /* OK for unprivileged users. */ - .policy = meter_policy, .doit = ovs_meter_cmd_features }, { .cmd = OVS_METER_CMD_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, /* Requires CAP_NET_ADMIN * privilege. */ - .policy = meter_policy, .doit = ovs_meter_cmd_set, }, { .cmd = OVS_METER_CMD_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = 0, /* OK for unprivileged users. */ - .policy = meter_policy, .doit = ovs_meter_cmd_get, }, { .cmd = OVS_METER_CMD_DEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, /* Requires CAP_NET_ADMIN * privilege. */ - .policy = meter_policy, .doit = ovs_meter_cmd_del }, }; @@ -560,6 +560,7 @@ struct genl_family dp_meter_genl_family __ro_after_init = { .name = OVS_METER_FAMILY, .version = OVS_METER_VERSION, .maxattr = OVS_METER_ATTR_MAX, + .policy = meter_policy, .netnsok = true, .parallel_ops = true, .ops = dp_meter_genl_ops, diff --git a/net/openvswitch/vport-vxlan.c b/net/openvswitch/vport-vxlan.c index 8f16f11f7ad3e004b39eba94d269fc0f67c96143..f3c54871f9e14f8e6414bb73f0785a0f80660479 100644 --- a/net/openvswitch/vport-vxlan.c +++ b/net/openvswitch/vport-vxlan.c @@ -43,7 +43,7 @@ static int vxlan_get_options(const struct vport *vport, struct sk_buff *skb) if (vxlan->cfg.flags & VXLAN_F_GBP) { struct nlattr *exts; - exts = nla_nest_start(skb, OVS_TUNNEL_ATTR_EXTENSION); + exts = nla_nest_start_noflag(skb, OVS_TUNNEL_ATTR_EXTENSION); if (!exts) return -EMSGSIZE; @@ -70,8 +70,8 @@ static int vxlan_configure_exts(struct vport *vport, struct nlattr *attr, if (nla_len(attr) < sizeof(struct nlattr)) return -EINVAL; - err = nla_parse_nested(exts, OVS_VXLAN_EXT_MAX, attr, exts_policy, - NULL); + err = nla_parse_nested_deprecated(exts, OVS_VXLAN_EXT_MAX, attr, + exts_policy, NULL); if (err < 0) return err; diff --git a/net/openvswitch/vport.c b/net/openvswitch/vport.c index 19f6765566e727d8e02655f0dd6c2f0f971f64e3..258ce3b7b4524bdade7472607dd784e28d612a6c 100644 --- a/net/openvswitch/vport.c +++ b/net/openvswitch/vport.c @@ -319,7 +319,7 @@ int ovs_vport_get_options(const struct vport *vport, struct sk_buff *skb) if (!vport->ops->get_options) return 0; - nla = nla_nest_start(skb, OVS_VPORT_ATTR_OPTIONS); + nla = nla_nest_start_noflag(skb, OVS_VPORT_ATTR_OPTIONS); if (!nla) return -EMSGSIZE; diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c index 9b81813dd16af490859ab32ae59cf919c9e15ed6..90d4e3ce00e51a48406ee48ccb251f310f47337d 100644 --- a/net/packet/af_packet.c +++ b/net/packet/af_packet.c @@ -275,24 +275,22 @@ static bool packet_use_direct_xmit(const struct packet_sock *po) return po->xmit == packet_direct_xmit; } -static u16 __packet_pick_tx_queue(struct net_device *dev, struct sk_buff *skb, - struct net_device *sb_dev) -{ - return dev_pick_tx_cpu_id(dev, skb, sb_dev, NULL); -} - static u16 packet_pick_tx_queue(struct sk_buff *skb) { struct net_device *dev = skb->dev; const struct net_device_ops *ops = dev->netdev_ops; + int cpu = raw_smp_processor_id(); u16 queue_index; +#ifdef CONFIG_XPS + skb->sender_cpu = cpu + 1; +#endif + skb_record_rx_queue(skb, cpu % dev->real_num_tx_queues); if (ops->ndo_select_queue) { - queue_index = ops->ndo_select_queue(dev, skb, NULL, - __packet_pick_tx_queue); + queue_index = ops->ndo_select_queue(dev, skb, NULL); queue_index = netdev_cap_txqueue(dev, queue_index); } else { - queue_index = __packet_pick_tx_queue(dev, skb, NULL); + queue_index = netdev_pick_tx(dev, skb, NULL); } return queue_index; @@ -4090,11 +4088,6 @@ static int packet_ioctl(struct socket *sock, unsigned int cmd, spin_unlock_bh(&sk->sk_receive_queue.lock); return put_user(amount, (int __user *)arg); } - case SIOCGSTAMP: - return sock_get_timestamp(sk, (struct timeval __user *)arg); - case SIOCGSTAMPNS: - return sock_get_timestampns(sk, (struct timespec __user *)arg); - #ifdef CONFIG_INET case SIOCADDRT: case SIOCDELRT: @@ -4470,6 +4463,7 @@ static const struct proto_ops packet_ops_spkt = { .getname = packet_getname_spkt, .poll = datagram_poll, .ioctl = packet_ioctl, + .gettstamp = sock_gettstamp, .listen = sock_no_listen, .shutdown = sock_no_shutdown, .setsockopt = sock_no_setsockopt, @@ -4491,6 +4485,7 @@ static const struct proto_ops packet_ops = { .getname = packet_getname, .poll = packet_poll, .ioctl = packet_ioctl, + .gettstamp = sock_gettstamp, .listen = sock_no_listen, .shutdown = sock_no_shutdown, .setsockopt = packet_setsockopt, diff --git a/net/packet/diag.c b/net/packet/diag.c index 7ef1c881ae7417e500630df398a900dddd598092..98abfd8644a45b9b667bdeddb02767e4d411dab9 100644 --- a/net/packet/diag.c +++ b/net/packet/diag.c @@ -39,7 +39,7 @@ static int pdiag_put_mclist(const struct packet_sock *po, struct sk_buff *nlskb) struct nlattr *mca; struct packet_mclist *ml; - mca = nla_nest_start(nlskb, PACKET_DIAG_MCLIST); + mca = nla_nest_start_noflag(nlskb, PACKET_DIAG_MCLIST); if (!mca) return -EMSGSIZE; diff --git a/net/phonet/pn_netlink.c b/net/phonet/pn_netlink.c index 871eaf2cb85e4d4bec7dd01292c4241dff82c29e..be92d936b5d5670a9f85aab6687c65d5bb2e570f 100644 --- a/net/phonet/pn_netlink.c +++ b/net/phonet/pn_netlink.c @@ -79,8 +79,8 @@ static int addr_doit(struct sk_buff *skb, struct nlmsghdr *nlh, ASSERT_RTNL(); - err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, ifa_phonet_policy, - extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*ifm), tb, IFA_MAX, + ifa_phonet_policy, extack); if (err < 0) return err; @@ -246,8 +246,8 @@ static int route_doit(struct sk_buff *skb, struct nlmsghdr *nlh, ASSERT_RTNL(); - err = nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, rtm_phonet_policy, - extack); + err = nlmsg_parse_deprecated(nlh, sizeof(*rtm), tb, RTA_MAX, + rtm_phonet_policy, extack); if (err < 0) return err; diff --git a/net/psample/psample.c b/net/psample/psample.c index 64f95624f21939c946e0bb256a40d825588e0fd2..a107b24056682038dc767e60a359163752af0f33 100644 --- a/net/psample/psample.c +++ b/net/psample/psample.c @@ -100,6 +100,7 @@ static int psample_nl_cmd_get_group_dumpit(struct sk_buff *msg, static const struct genl_ops psample_nl_ops[] = { { .cmd = PSAMPLE_CMD_GET_GROUP, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .dumpit = psample_nl_cmd_get_group_dumpit, /* can be retrieved by unprivileged users */ } diff --git a/net/qrtr/qrtr.c b/net/qrtr/qrtr.c index b37e6e0a1026428f4547076b7cb3f0e2eab2898a..dd0e97f4f6c021092c3472beab58fde32d35edb5 100644 --- a/net/qrtr/qrtr.c +++ b/net/qrtr/qrtr.c @@ -968,9 +968,6 @@ static int qrtr_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) break; } break; - case SIOCGSTAMP: - rc = sock_get_timestamp(sk, argp); - break; case SIOCADDRT: case SIOCDELRT: case SIOCSIFADDR: @@ -1033,6 +1030,7 @@ static const struct proto_ops qrtr_proto_ops = { .recvmsg = qrtr_recvmsg, .getname = qrtr_getname, .ioctl = qrtr_ioctl, + .gettstamp = sock_gettstamp, .poll = datagram_poll, .shutdown = sock_no_shutdown, .setsockopt = sock_no_setsockopt, @@ -1093,7 +1091,8 @@ static int qrtr_addr_doit(struct sk_buff *skb, struct nlmsghdr *nlh, ASSERT_RTNL(); - rc = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, qrtr_policy, extack); + rc = nlmsg_parse_deprecated(nlh, sizeof(*ifm), tb, IFA_MAX, + qrtr_policy, extack); if (rc < 0) return rc; diff --git a/net/rds/tcp.c b/net/rds/tcp.c index faf726e00e27c75b11721dbc55518ca60bdf00a6..66121bc6f34efe44d66c2940a35f8e1fae274682 100644 --- a/net/rds/tcp.c +++ b/net/rds/tcp.c @@ -551,7 +551,7 @@ static __net_init int rds_tcp_init_net(struct net *net) tbl = kmemdup(rds_tcp_sysctl_table, sizeof(rds_tcp_sysctl_table), GFP_KERNEL); if (!tbl) { - pr_warn("could not set allocate syctl table\n"); + pr_warn("could not set allocate sysctl table\n"); return -ENOMEM; } rtn->ctl_table = tbl; diff --git a/net/rose/af_rose.c b/net/rose/af_rose.c index c96f63ffe31e5fe543b4d033d4342a1e405c6d7b..e274bc6e1458abb5cba556d936187540024fddb7 100644 --- a/net/rose/af_rose.c +++ b/net/rose/af_rose.c @@ -1301,12 +1301,6 @@ static int rose_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) return put_user(amount, (unsigned int __user *) argp); } - case SIOCGSTAMP: - return sock_get_timestamp(sk, (struct timeval __user *) argp); - - case SIOCGSTAMPNS: - return sock_get_timestampns(sk, (struct timespec __user *) argp); - case SIOCGIFADDR: case SIOCSIFADDR: case SIOCGIFDSTADDR: @@ -1474,6 +1468,7 @@ static const struct proto_ops rose_proto_ops = { .getname = rose_getname, .poll = datagram_poll, .ioctl = rose_ioctl, + .gettstamp = sock_gettstamp, .listen = rose_listen, .shutdown = sock_no_shutdown, .setsockopt = rose_setsockopt, diff --git a/net/rxrpc/local_object.c b/net/rxrpc/local_object.c index 01959db51445ca00e6044d8a849e698c4ab17a33..b67dec9454980c46ba7afaa69064782f15d4a803 100644 --- a/net/rxrpc/local_object.c +++ b/net/rxrpc/local_object.c @@ -180,7 +180,7 @@ static int rxrpc_open_socket(struct rxrpc_local *local, struct net *net) /* Fall through and set IPv4 options too otherwise we don't get * errors from IPv4 packets sent through the IPv6 socket. */ - + /* Fall through */ case AF_INET: /* we want to receive ICMP errors */ opt = 1; diff --git a/net/sched/act_api.c b/net/sched/act_api.c index 5a87e271d35a2416b3589888bcfadee0c31b2142..683fcc00da491e581ec5768e00269a494d3311a9 100644 --- a/net/sched/act_api.c +++ b/net/sched/act_api.c @@ -242,7 +242,7 @@ static int tcf_dump_walker(struct tcf_idrinfo *idrinfo, struct sk_buff *skb, (unsigned long)p->tcfa_tm.lastuse)) continue; - nest = nla_nest_start(skb, n_i); + nest = nla_nest_start_noflag(skb, n_i); if (!nest) { index--; goto nla_put_failure; @@ -299,7 +299,7 @@ static int tcf_del_walker(struct tcf_idrinfo *idrinfo, struct sk_buff *skb, struct tc_action *p; unsigned long id = 1; - nest = nla_nest_start(skb, 0); + nest = nla_nest_start_noflag(skb, 0); if (nest == NULL) goto nla_put_failure; if (nla_put_string(skb, TCA_KIND, ops->kind)) @@ -776,7 +776,7 @@ tcf_action_dump_1(struct sk_buff *skb, struct tc_action *a, int bind, int ref) } rcu_read_unlock(); - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (nest == NULL) goto nla_put_failure; err = tcf_action_dump_old(skb, a, bind, ref); @@ -800,7 +800,7 @@ int tcf_action_dump(struct sk_buff *skb, struct tc_action *actions[], for (i = 0; i < TCA_ACT_MAX_PRIO && actions[i]; i++) { a = actions[i]; - nest = nla_nest_start(skb, a->order); + nest = nla_nest_start_noflag(skb, a->order); if (nest == NULL) goto nla_put_failure; err = tcf_action_dump_1(skb, a, bind, ref); @@ -849,7 +849,8 @@ struct tc_action *tcf_action_init_1(struct net *net, struct tcf_proto *tp, int err; if (name == NULL) { - err = nla_parse_nested(tb, TCA_ACT_MAX, nla, NULL, extack); + err = nla_parse_nested_deprecated(tb, TCA_ACT_MAX, nla, NULL, + extack); if (err < 0) goto err_out; err = -EINVAL; @@ -964,7 +965,8 @@ int tcf_action_init(struct net *net, struct tcf_proto *tp, struct nlattr *nla, int err; int i; - err = nla_parse_nested(tb, TCA_ACT_MAX_PRIO, nla, NULL, extack); + err = nla_parse_nested_deprecated(tb, TCA_ACT_MAX_PRIO, nla, NULL, + extack); if (err < 0) return err; @@ -1052,7 +1054,7 @@ static int tca_get_fill(struct sk_buff *skb, struct tc_action *actions[], t->tca__pad1 = 0; t->tca__pad2 = 0; - nest = nla_nest_start(skb, TCA_ACT_TAB); + nest = nla_nest_start_noflag(skb, TCA_ACT_TAB); if (!nest) goto out_nlmsg_trim; @@ -1099,7 +1101,7 @@ static struct tc_action *tcf_action_get_1(struct net *net, struct nlattr *nla, int index; int err; - err = nla_parse_nested(tb, TCA_ACT_MAX, nla, NULL, extack); + err = nla_parse_nested_deprecated(tb, TCA_ACT_MAX, nla, NULL, extack); if (err < 0) goto err_out; @@ -1153,7 +1155,7 @@ static int tca_action_flush(struct net *net, struct nlattr *nla, b = skb_tail_pointer(skb); - err = nla_parse_nested(tb, TCA_ACT_MAX, nla, NULL, extack); + err = nla_parse_nested_deprecated(tb, TCA_ACT_MAX, nla, NULL, extack); if (err < 0) goto err_out; @@ -1176,7 +1178,7 @@ static int tca_action_flush(struct net *net, struct nlattr *nla, t->tca__pad1 = 0; t->tca__pad2 = 0; - nest = nla_nest_start(skb, TCA_ACT_TAB); + nest = nla_nest_start_noflag(skb, TCA_ACT_TAB); if (!nest) { NL_SET_ERR_MSG(extack, "Failed to add new netlink message"); goto out_module_put; @@ -1282,7 +1284,8 @@ tca_action_gd(struct net *net, struct nlattr *nla, struct nlmsghdr *n, size_t attr_size = 0; struct tc_action *actions[TCA_ACT_MAX_PRIO] = {}; - ret = nla_parse_nested(tb, TCA_ACT_MAX_PRIO, nla, NULL, extack); + ret = nla_parse_nested_deprecated(tb, TCA_ACT_MAX_PRIO, nla, NULL, + extack); if (ret < 0) return ret; @@ -1384,8 +1387,8 @@ static int tc_ctl_action(struct sk_buff *skb, struct nlmsghdr *n, !netlink_capable(skb, CAP_NET_ADMIN)) return -EPERM; - ret = nlmsg_parse(n, sizeof(struct tcamsg), tca, TCA_ROOT_MAX, NULL, - extack); + ret = nlmsg_parse_deprecated(n, sizeof(struct tcamsg), tca, + TCA_ROOT_MAX, NULL, extack); if (ret < 0) return ret; @@ -1436,13 +1439,12 @@ static struct nlattr *find_dump_kind(struct nlattr **nla) if (tb1 == NULL) return NULL; - if (nla_parse(tb, TCA_ACT_MAX_PRIO, nla_data(tb1), - NLMSG_ALIGN(nla_len(tb1)), NULL, NULL) < 0) + if (nla_parse_deprecated(tb, TCA_ACT_MAX_PRIO, nla_data(tb1), NLMSG_ALIGN(nla_len(tb1)), NULL, NULL) < 0) return NULL; if (tb[1] == NULL) return NULL; - if (nla_parse_nested(tb2, TCA_ACT_MAX, tb[1], NULL, NULL) < 0) + if (nla_parse_nested_deprecated(tb2, TCA_ACT_MAX, tb[1], NULL, NULL) < 0) return NULL; kind = tb2[TCA_ACT_KIND]; @@ -1466,8 +1468,8 @@ static int tc_dump_action(struct sk_buff *skb, struct netlink_callback *cb) u32 msecs_since = 0; u32 act_count = 0; - ret = nlmsg_parse(cb->nlh, sizeof(struct tcamsg), tb, TCA_ROOT_MAX, - tcaa_policy, cb->extack); + ret = nlmsg_parse_deprecated(cb->nlh, sizeof(struct tcamsg), tb, + TCA_ROOT_MAX, tcaa_policy, cb->extack); if (ret < 0) return ret; @@ -1508,7 +1510,7 @@ static int tc_dump_action(struct sk_buff *skb, struct netlink_callback *cb) if (!count_attr) goto out_module_put; - nest = nla_nest_start(skb, TCA_ACT_TAB); + nest = nla_nest_start_noflag(skb, TCA_ACT_TAB); if (nest == NULL) goto out_module_put; diff --git a/net/sched/act_bpf.c b/net/sched/act_bpf.c index 3841156aa09f778c285765b51342cd5d218a34ac..a0c77faca04bc540609fbbf5c72d8d5267387041 100644 --- a/net/sched/act_bpf.c +++ b/net/sched/act_bpf.c @@ -293,7 +293,8 @@ static int tcf_bpf_init(struct net *net, struct nlattr *nla, if (!nla) return -EINVAL; - ret = nla_parse_nested(tb, TCA_ACT_BPF_MAX, nla, act_bpf_policy, NULL); + ret = nla_parse_nested_deprecated(tb, TCA_ACT_BPF_MAX, nla, + act_bpf_policy, NULL); if (ret < 0) return ret; diff --git a/net/sched/act_connmark.c b/net/sched/act_connmark.c index 32ae0cd6e31c67e36793081ac11371c2250eb0f1..8838575cd53696ff8884a9e9a517c7f5fb353f11 100644 --- a/net/sched/act_connmark.c +++ b/net/sched/act_connmark.c @@ -111,8 +111,8 @@ static int tcf_connmark_init(struct net *net, struct nlattr *nla, if (!nla) return -EINVAL; - ret = nla_parse_nested(tb, TCA_CONNMARK_MAX, nla, connmark_policy, - NULL); + ret = nla_parse_nested_deprecated(tb, TCA_CONNMARK_MAX, nla, + connmark_policy, NULL); if (ret < 0) return ret; diff --git a/net/sched/act_csum.c b/net/sched/act_csum.c index 0c77e7bdf6d5d282965eb3970f3b8934e570e0d4..14bb525e355eaa64889d01fee68aaff93618d688 100644 --- a/net/sched/act_csum.c +++ b/net/sched/act_csum.c @@ -61,7 +61,8 @@ static int tcf_csum_init(struct net *net, struct nlattr *nla, if (nla == NULL) return -EINVAL; - err = nla_parse_nested(tb, TCA_CSUM_MAX, nla, csum_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_CSUM_MAX, nla, csum_policy, + NULL); if (err < 0) return err; diff --git a/net/sched/act_gact.c b/net/sched/act_gact.c index e540e31069d746106eb82c5ea3c2f99f9438cbf4..75492b07f32481f014b69b048f4c839bde251123 100644 --- a/net/sched/act_gact.c +++ b/net/sched/act_gact.c @@ -74,7 +74,8 @@ static int tcf_gact_init(struct net *net, struct nlattr *nla, if (nla == NULL) return -EINVAL; - err = nla_parse_nested(tb, TCA_GACT_MAX, nla, gact_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_GACT_MAX, nla, gact_policy, + NULL); if (err < 0) return err; diff --git a/net/sched/act_ife.c b/net/sched/act_ife.c index 31c6ffb6abe7c607972de7b80fe77f03f8782723..12489f60a979aee97193015159fc299df2586296 100644 --- a/net/sched/act_ife.c +++ b/net/sched/act_ife.c @@ -387,7 +387,7 @@ static int dump_metalist(struct sk_buff *skb, struct tcf_ife_info *ife) if (list_empty(&ife->metalist)) return 0; - nest = nla_nest_start(skb, TCA_IFE_METALST); + nest = nla_nest_start_noflag(skb, TCA_IFE_METALST); if (!nest) goto out_nlmsg_trim; @@ -486,7 +486,8 @@ static int tcf_ife_init(struct net *net, struct nlattr *nla, int ret = 0; int err; - err = nla_parse_nested(tb, TCA_IFE_MAX, nla, ife_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_IFE_MAX, nla, ife_policy, + NULL); if (err < 0) return err; @@ -567,8 +568,9 @@ static int tcf_ife_init(struct net *net, struct nlattr *nla, INIT_LIST_HEAD(&ife->metalist); if (tb[TCA_IFE_METALST]) { - err = nla_parse_nested(tb2, IFE_META_MAX, tb[TCA_IFE_METALST], - NULL, NULL); + err = nla_parse_nested_deprecated(tb2, IFE_META_MAX, + tb[TCA_IFE_METALST], NULL, + NULL); if (err) goto metadata_parse_err; err = populate_metalist(ife, tb2, exists, rtnl_held); diff --git a/net/sched/act_ipt.c b/net/sched/act_ipt.c index 04a0b5c611943a4e10bfa0855f1c3928e4e141de..ae6e28ab1cd755fb234aaa963b900285b660ac0c 100644 --- a/net/sched/act_ipt.c +++ b/net/sched/act_ipt.c @@ -113,7 +113,8 @@ static int __tcf_ipt_init(struct net *net, unsigned int id, struct nlattr *nla, if (nla == NULL) return -EINVAL; - err = nla_parse_nested(tb, TCA_IPT_MAX, nla, ipt_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_IPT_MAX, nla, ipt_policy, + NULL); if (err < 0) return err; diff --git a/net/sched/act_mirred.c b/net/sched/act_mirred.c index 17cc6bd4c57c3a6f12786c3d1109c6e48af185dd..c329390342f40112a3d93d0b295e998d30df5aa3 100644 --- a/net/sched/act_mirred.c +++ b/net/sched/act_mirred.c @@ -111,7 +111,8 @@ static int tcf_mirred_init(struct net *net, struct nlattr *nla, NL_SET_ERR_MSG_MOD(extack, "Mirred requires attributes to be passed"); return -EINVAL; } - ret = nla_parse_nested(tb, TCA_MIRRED_MAX, nla, mirred_policy, extack); + ret = nla_parse_nested_deprecated(tb, TCA_MIRRED_MAX, nla, + mirred_policy, extack); if (ret < 0) return ret; if (!tb[TCA_MIRRED_PARMS]) { diff --git a/net/sched/act_nat.c b/net/sched/act_nat.c index e91bb8eb81ec5e2e7dfe86d2832cdb6530f8e327..51bd1ba02380e0e8226292de877ea70dc2d2343d 100644 --- a/net/sched/act_nat.c +++ b/net/sched/act_nat.c @@ -52,7 +52,8 @@ static int tcf_nat_init(struct net *net, struct nlattr *nla, struct nlattr *est, if (nla == NULL) return -EINVAL; - err = nla_parse_nested(tb, TCA_NAT_MAX, nla, nat_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_NAT_MAX, nla, nat_policy, + NULL); if (err < 0) return err; diff --git a/net/sched/act_pedit.c b/net/sched/act_pedit.c index 287793abfaf9bae9aba9c4f23552890c795010c1..d790c02b9c6c10706f805656bce14bb224f8b3ef 100644 --- a/net/sched/act_pedit.c +++ b/net/sched/act_pedit.c @@ -70,8 +70,9 @@ static struct tcf_pedit_key_ex *tcf_pedit_keys_ex_parse(struct nlattr *nla, goto err_out; } - err = nla_parse_nested(tb, TCA_PEDIT_KEY_EX_MAX, ka, - pedit_key_ex_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_PEDIT_KEY_EX_MAX, + ka, pedit_key_ex_policy, + NULL); if (err) goto err_out; @@ -108,14 +109,15 @@ err_out: static int tcf_pedit_key_ex_dump(struct sk_buff *skb, struct tcf_pedit_key_ex *keys_ex, int n) { - struct nlattr *keys_start = nla_nest_start(skb, TCA_PEDIT_KEYS_EX); + struct nlattr *keys_start = nla_nest_start_noflag(skb, + TCA_PEDIT_KEYS_EX); if (!keys_start) goto nla_failure; for (; n > 0; n--) { struct nlattr *key_start; - key_start = nla_nest_start(skb, TCA_PEDIT_KEY_EX); + key_start = nla_nest_start_noflag(skb, TCA_PEDIT_KEY_EX); if (!key_start) goto nla_failure; @@ -157,7 +159,8 @@ static int tcf_pedit_init(struct net *net, struct nlattr *nla, return -EINVAL; } - err = nla_parse_nested(tb, TCA_PEDIT_MAX, nla, pedit_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_PEDIT_MAX, nla, + pedit_policy, NULL); if (err < 0) return err; diff --git a/net/sched/act_police.c b/net/sched/act_police.c index 2b8581f6ab510100e66fc3e2445ed6460ff65323..61731944742ae564d6f1c300d0bbad5cd1530cb7 100644 --- a/net/sched/act_police.c +++ b/net/sched/act_police.c @@ -22,42 +22,7 @@ #include #include #include - -struct tcf_police_params { - int tcfp_result; - u32 tcfp_ewma_rate; - s64 tcfp_burst; - u32 tcfp_mtu; - s64 tcfp_mtu_ptoks; - struct psched_ratecfg rate; - bool rate_present; - struct psched_ratecfg peak; - bool peak_present; - struct rcu_head rcu; -}; - -struct tcf_police { - struct tc_action common; - struct tcf_police_params __rcu *params; - - spinlock_t tcfp_lock ____cacheline_aligned_in_smp; - s64 tcfp_toks; - s64 tcfp_ptoks; - s64 tcfp_t_c; -}; - -#define to_police(pc) ((struct tcf_police *)pc) - -/* old policer structure from before tc actions */ -struct tc_police_compat { - u32 index; - int action; - u32 limit; - u32 burst; - u32 mtu; - struct tc_ratespec rate; - struct tc_ratespec peakrate; -}; +#include /* Each policer is serialized by its individual spinlock */ @@ -100,7 +65,8 @@ static int tcf_police_init(struct net *net, struct nlattr *nla, if (nla == NULL) return -EINVAL; - err = nla_parse_nested(tb, TCA_POLICE_MAX, nla, police_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_POLICE_MAX, nla, + police_policy, NULL); if (err < 0) return err; @@ -316,6 +282,20 @@ static void tcf_police_cleanup(struct tc_action *a) kfree_rcu(p, rcu); } +static void tcf_police_stats_update(struct tc_action *a, + u64 bytes, u32 packets, + u64 lastuse, bool hw) +{ + struct tcf_police *police = to_police(a); + struct tcf_t *tm = &police->tcf_tm; + + _bstats_cpu_update(this_cpu_ptr(a->cpu_bstats), bytes, packets); + if (hw) + _bstats_cpu_update(this_cpu_ptr(a->cpu_bstats_hw), + bytes, packets); + tm->lastuse = max_t(u64, tm->lastuse, lastuse); +} + static int tcf_police_dump(struct sk_buff *skb, struct tc_action *a, int bind, int ref) { @@ -379,6 +359,7 @@ static struct tc_action_ops act_police_ops = { .kind = "police", .id = TCA_ID_POLICE, .owner = THIS_MODULE, + .stats_update = tcf_police_stats_update, .act = tcf_police_act, .dump = tcf_police_dump, .init = tcf_police_init, diff --git a/net/sched/act_sample.c b/net/sched/act_sample.c index 0f82d50ea23245be1ce34fcce1cdb4a048c1af17..b2faa43c1ac7665f5e21afdbcdd19a17d9c8f402 100644 --- a/net/sched/act_sample.c +++ b/net/sched/act_sample.c @@ -53,7 +53,8 @@ static int tcf_sample_init(struct net *net, struct nlattr *nla, if (!nla) return -EINVAL; - ret = nla_parse_nested(tb, TCA_SAMPLE_MAX, nla, sample_policy, NULL); + ret = nla_parse_nested_deprecated(tb, TCA_SAMPLE_MAX, nla, + sample_policy, NULL); if (ret < 0) return ret; if (!tb[TCA_SAMPLE_PARMS] || !tb[TCA_SAMPLE_RATE] || diff --git a/net/sched/act_simple.c b/net/sched/act_simple.c index 23c8ca5615e59b85d22bacda25a22cc02e54d9f6..ead480e6014c1022015129a415d0d14b3bc0c0c7 100644 --- a/net/sched/act_simple.c +++ b/net/sched/act_simple.c @@ -104,7 +104,8 @@ static int tcf_simp_init(struct net *net, struct nlattr *nla, if (nla == NULL) return -EINVAL; - err = nla_parse_nested(tb, TCA_DEF_MAX, nla, simple_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_DEF_MAX, nla, simple_policy, + NULL); if (err < 0) return err; diff --git a/net/sched/act_skbedit.c b/net/sched/act_skbedit.c index 7e1d261a31d2e73460f5d8b2d1d7869f18a1ea24..7ec159b9536405f043c30834d2fd18c0e7f22f8e 100644 --- a/net/sched/act_skbedit.c +++ b/net/sched/act_skbedit.c @@ -114,7 +114,8 @@ static int tcf_skbedit_init(struct net *net, struct nlattr *nla, if (nla == NULL) return -EINVAL; - err = nla_parse_nested(tb, TCA_SKBEDIT_MAX, nla, skbedit_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_SKBEDIT_MAX, nla, + skbedit_policy, NULL); if (err < 0) return err; diff --git a/net/sched/act_skbmod.c b/net/sched/act_skbmod.c index 1d4c324d0a42bd2cd707a987a5595f3119f66155..186ef98c828f7ae0f32c3a674a8bb0ab1306b76b 100644 --- a/net/sched/act_skbmod.c +++ b/net/sched/act_skbmod.c @@ -102,7 +102,8 @@ static int tcf_skbmod_init(struct net *net, struct nlattr *nla, if (!nla) return -EINVAL; - err = nla_parse_nested(tb, TCA_SKBMOD_MAX, nla, skbmod_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_SKBMOD_MAX, nla, + skbmod_policy, NULL); if (err < 0) return err; diff --git a/net/sched/act_tunnel_key.c b/net/sched/act_tunnel_key.c index d5aaf90a39712982685cbed5f60d16576324f17b..6a9070511ee8454cab75c83bc6d6796ea194da53 100644 --- a/net/sched/act_tunnel_key.c +++ b/net/sched/act_tunnel_key.c @@ -76,8 +76,9 @@ tunnel_key_copy_geneve_opt(const struct nlattr *nla, void *dst, int dst_len, int err, data_len, opt_len; u8 *data; - err = nla_parse_nested(tb, TCA_TUNNEL_KEY_ENC_OPT_GENEVE_MAX, - nla, geneve_opt_policy, extack); + err = nla_parse_nested_deprecated(tb, + TCA_TUNNEL_KEY_ENC_OPT_GENEVE_MAX, + nla, geneve_opt_policy, extack); if (err < 0) return err; @@ -125,8 +126,8 @@ static int tunnel_key_copy_opts(const struct nlattr *nla, u8 *dst, int err, rem, opt_len, len = nla_len(nla), opts_len = 0; const struct nlattr *attr, *head = nla_data(nla); - err = nla_validate(head, len, TCA_TUNNEL_KEY_ENC_OPTS_MAX, - enc_opts_policy, extack); + err = nla_validate_deprecated(head, len, TCA_TUNNEL_KEY_ENC_OPTS_MAX, + enc_opts_policy, extack); if (err) return err; @@ -235,8 +236,8 @@ static int tunnel_key_init(struct net *net, struct nlattr *nla, return -EINVAL; } - err = nla_parse_nested(tb, TCA_TUNNEL_KEY_MAX, nla, tunnel_key_policy, - extack); + err = nla_parse_nested_deprecated(tb, TCA_TUNNEL_KEY_MAX, nla, + tunnel_key_policy, extack); if (err < 0) { NL_SET_ERR_MSG(extack, "Failed to parse nested tunnel key attributes"); return err; @@ -426,7 +427,7 @@ static int tunnel_key_geneve_opts_dump(struct sk_buff *skb, u8 *src = (u8 *)(info + 1); struct nlattr *start; - start = nla_nest_start(skb, TCA_TUNNEL_KEY_ENC_OPTS_GENEVE); + start = nla_nest_start_noflag(skb, TCA_TUNNEL_KEY_ENC_OPTS_GENEVE); if (!start) return -EMSGSIZE; @@ -460,7 +461,7 @@ static int tunnel_key_opts_dump(struct sk_buff *skb, if (!info->options_len) return 0; - start = nla_nest_start(skb, TCA_TUNNEL_KEY_ENC_OPTS); + start = nla_nest_start_noflag(skb, TCA_TUNNEL_KEY_ENC_OPTS); if (!start) return -EMSGSIZE; diff --git a/net/sched/act_vlan.c b/net/sched/act_vlan.c index 0f40d0a74423b8d91bf8bb8838eb382846710851..39bd9fa3e4551cdc7e49e69aa48cc5a82eba9c7e 100644 --- a/net/sched/act_vlan.c +++ b/net/sched/act_vlan.c @@ -124,7 +124,8 @@ static int tcf_vlan_init(struct net *net, struct nlattr *nla, if (!nla) return -EINVAL; - err = nla_parse_nested(tb, TCA_VLAN_MAX, nla, vlan_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_VLAN_MAX, nla, vlan_policy, + NULL); if (err < 0) return err; diff --git a/net/sched/cls_api.c b/net/sched/cls_api.c index 99ae30c177c76783dae71bf7955f4d4d0bb3b639..d4699156974a6811a60e327f37b9d7a80aef7fde 100644 --- a/net/sched/cls_api.c +++ b/net/sched/cls_api.c @@ -37,6 +37,8 @@ #include #include #include +#include +#include #include extern const struct nla_policy rtm_tca_policy[TCA_MAX + 1]; @@ -2006,7 +2008,8 @@ static int tc_new_tfilter(struct sk_buff *skb, struct nlmsghdr *n, replay: tp_created = 0; - err = nlmsg_parse(n, sizeof(*t), tca, TCA_MAX, rtm_tca_policy, extack); + err = nlmsg_parse_deprecated(n, sizeof(*t), tca, TCA_MAX, + rtm_tca_policy, extack); if (err < 0) return err; @@ -2217,7 +2220,8 @@ static int tc_del_tfilter(struct sk_buff *skb, struct nlmsghdr *n, if (!netlink_ns_capable(skb, net->user_ns, CAP_NET_ADMIN)) return -EPERM; - err = nlmsg_parse(n, sizeof(*t), tca, TCA_MAX, rtm_tca_policy, extack); + err = nlmsg_parse_deprecated(n, sizeof(*t), tca, TCA_MAX, + rtm_tca_policy, extack); if (err < 0) return err; @@ -2366,7 +2370,8 @@ static int tc_get_tfilter(struct sk_buff *skb, struct nlmsghdr *n, int err; bool rtnl_held = false; - err = nlmsg_parse(n, sizeof(*t), tca, TCA_MAX, rtm_tca_policy, extack); + err = nlmsg_parse_deprecated(n, sizeof(*t), tca, TCA_MAX, + rtm_tca_policy, extack); if (err < 0) return err; @@ -2558,8 +2563,8 @@ static int tc_dump_tfilter(struct sk_buff *skb, struct netlink_callback *cb) if (nlmsg_len(cb->nlh) < sizeof(*tcm)) return skb->len; - err = nlmsg_parse(cb->nlh, sizeof(*tcm), tca, TCA_MAX, NULL, - cb->extack); + err = nlmsg_parse_deprecated(cb->nlh, sizeof(*tcm), tca, TCA_MAX, + NULL, cb->extack); if (err) return err; @@ -2806,7 +2811,8 @@ static int tc_ctl_chain(struct sk_buff *skb, struct nlmsghdr *n, return -EPERM; replay: - err = nlmsg_parse(n, sizeof(*t), tca, TCA_MAX, rtm_tca_policy, extack); + err = nlmsg_parse_deprecated(n, sizeof(*t), tca, TCA_MAX, + rtm_tca_policy, extack); if (err < 0) return err; @@ -2937,8 +2943,8 @@ static int tc_dump_chain(struct sk_buff *skb, struct netlink_callback *cb) if (nlmsg_len(cb->nlh) < sizeof(*tcm)) return skb->len; - err = nlmsg_parse(cb->nlh, sizeof(*tcm), tca, TCA_MAX, rtm_tca_policy, - cb->extack); + err = nlmsg_parse_deprecated(cb->nlh, sizeof(*tcm), tca, TCA_MAX, + rtm_tca_policy, cb->extack); if (err) return err; @@ -3111,7 +3117,7 @@ int tcf_exts_dump(struct sk_buff *skb, struct tcf_exts *exts) * tc data even if iproute2 was newer - jhs */ if (exts->type != TCA_OLD_COMPAT) { - nest = nla_nest_start(skb, exts->action); + nest = nla_nest_start_noflag(skb, exts->action); if (nest == NULL) goto nla_put_failure; @@ -3120,7 +3126,7 @@ int tcf_exts_dump(struct sk_buff *skb, struct tcf_exts *exts) nla_nest_end(skb, nest); } else if (exts->police) { struct tc_action *act = tcf_exts_first_act(exts); - nest = nla_nest_start(skb, exts->police); + nest = nla_nest_start_noflag(skb, exts->police); if (nest == NULL || !act) goto nla_put_failure; if (tcf_action_dump_old(skb, act, 0, 0) < 0) @@ -3229,7 +3235,6 @@ int tc_setup_flow_action(struct flow_action *flow_action, entry->tunnel = tcf_tunnel_info(act); } else if (is_tcf_tunnel_release(act)) { entry->id = FLOW_ACTION_TUNNEL_DECAP; - entry->tunnel = tcf_tunnel_info(act); } else if (is_tcf_pedit(act)) { for (k = 0; k < tcf_pedit_nkeys(act); k++) { switch (tcf_pedit_cmd(act, k)) { @@ -3254,6 +3259,18 @@ int tc_setup_flow_action(struct flow_action *flow_action, } else if (is_tcf_skbedit_mark(act)) { entry->id = FLOW_ACTION_MARK; entry->mark = tcf_skbedit_mark(act); + } else if (is_tcf_sample(act)) { + entry->id = FLOW_ACTION_SAMPLE; + entry->sample.psample_group = + tcf_sample_psample_group(act); + entry->sample.trunc_size = tcf_sample_trunc_size(act); + entry->sample.truncate = tcf_sample_truncate(act); + entry->sample.rate = tcf_sample_rate(act); + } else if (is_tcf_police(act)) { + entry->id = FLOW_ACTION_POLICE; + entry->police.burst = tcf_police_tcfp_burst(act); + entry->police.rate_bytes_ps = + tcf_police_rate_bytes_ps(act); } else { goto err_out; } diff --git a/net/sched/cls_basic.c b/net/sched/cls_basic.c index 687b0af67878b6436550768170784262b5e5b2be..923863f3b0d8fc5021897da3a4f3db82cc8a515b 100644 --- a/net/sched/cls_basic.c +++ b/net/sched/cls_basic.c @@ -185,8 +185,8 @@ static int basic_change(struct net *net, struct sk_buff *in_skb, if (tca[TCA_OPTIONS] == NULL) return -EINVAL; - err = nla_parse_nested(tb, TCA_BASIC_MAX, tca[TCA_OPTIONS], - basic_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_BASIC_MAX, tca[TCA_OPTIONS], + basic_policy, NULL); if (err < 0) return err; @@ -288,7 +288,7 @@ static int basic_dump(struct net *net, struct tcf_proto *tp, void *fh, t->tcm_handle = f->handle; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (nest == NULL) goto nla_put_failure; diff --git a/net/sched/cls_bpf.c b/net/sched/cls_bpf.c index b4ac58039cb1c3757f9f4e57ee1fb0ce34cc1be0..27365ed3fe0b8b35031676ba90eb57778838e475 100644 --- a/net/sched/cls_bpf.c +++ b/net/sched/cls_bpf.c @@ -157,8 +157,7 @@ static int cls_bpf_offload_cmd(struct tcf_proto *tp, struct cls_bpf_prog *prog, skip_sw = prog && tc_skip_sw(prog->gen_flags); obj = prog ?: oldprog; - tc_cls_common_offload_init(&cls_bpf.common, tp, obj->gen_flags, - extack); + tc_cls_common_offload_init(&cls_bpf.common, tp, obj->gen_flags, extack); cls_bpf.command = TC_CLSBPF_OFFLOAD; cls_bpf.exts = &obj->exts; cls_bpf.prog = prog ? prog->filter : NULL; @@ -468,8 +467,8 @@ static int cls_bpf_change(struct net *net, struct sk_buff *in_skb, if (tca[TCA_OPTIONS] == NULL) return -EINVAL; - ret = nla_parse_nested(tb, TCA_BPF_MAX, tca[TCA_OPTIONS], bpf_policy, - NULL); + ret = nla_parse_nested_deprecated(tb, TCA_BPF_MAX, tca[TCA_OPTIONS], + bpf_policy, NULL); if (ret < 0) return ret; @@ -591,7 +590,7 @@ static int cls_bpf_dump(struct net *net, struct tcf_proto *tp, void *fh, cls_bpf_offload_update_stats(tp, prog); - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (nest == NULL) goto nla_put_failure; diff --git a/net/sched/cls_cgroup.c b/net/sched/cls_cgroup.c index 4c1567854f954e9914409a1972e08220985bf486..35659127e5a3af83b913d7cae63770a353198584 100644 --- a/net/sched/cls_cgroup.c +++ b/net/sched/cls_cgroup.c @@ -32,6 +32,8 @@ static int cls_cgroup_classify(struct sk_buff *skb, const struct tcf_proto *tp, struct cls_cgroup_head *head = rcu_dereference_bh(tp->root); u32 classid = task_get_classid(skb); + if (unlikely(!head)) + return -1; if (!classid) return -1; if (!tcf_em_tree_match(skb, &head->ematches, NULL)) @@ -104,8 +106,9 @@ static int cls_cgroup_change(struct net *net, struct sk_buff *in_skb, goto errout; new->handle = handle; new->tp = tp; - err = nla_parse_nested(tb, TCA_CGROUP_MAX, tca[TCA_OPTIONS], - cgroup_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_CGROUP_MAX, + tca[TCA_OPTIONS], cgroup_policy, + NULL); if (err < 0) goto errout; @@ -176,7 +179,7 @@ static int cls_cgroup_dump(struct net *net, struct tcf_proto *tp, void *fh, t->tcm_handle = head->handle; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (nest == NULL) goto nla_put_failure; diff --git a/net/sched/cls_flow.c b/net/sched/cls_flow.c index eece1ee26930c7da736739dd5dba3145c79d0127..7bb79ec5b176f634e1bb879a2591a9575a61e6ae 100644 --- a/net/sched/cls_flow.c +++ b/net/sched/cls_flow.c @@ -408,7 +408,8 @@ static int flow_change(struct net *net, struct sk_buff *in_skb, if (opt == NULL) return -EINVAL; - err = nla_parse_nested(tb, TCA_FLOW_MAX, opt, flow_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_FLOW_MAX, opt, flow_policy, + NULL); if (err < 0) return err; @@ -629,7 +630,7 @@ static int flow_dump(struct net *net, struct tcf_proto *tp, void *fh, t->tcm_handle = f->handle; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (nest == NULL) goto nla_put_failure; diff --git a/net/sched/cls_flower.c b/net/sched/cls_flower.c index c04247b403ed288b6c168ecbef16fa2399bd3326..f6685fc53119a99ec9cb04d5bc1961cdace943c5 100644 --- a/net/sched/cls_flower.c +++ b/net/sched/cls_flower.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include @@ -75,6 +76,7 @@ struct fl_flow_mask { struct list_head filters; struct rcu_work rwork; struct list_head list; + refcount_t refcnt; }; struct fl_flow_tmplt { @@ -86,7 +88,9 @@ struct fl_flow_tmplt { struct cls_fl_head { struct rhashtable ht; + spinlock_t masks_lock; /* Protect masks list */ struct list_head masks; + struct list_head hw_filters; struct rcu_work rwork; struct idr handle_idr; }; @@ -99,11 +103,18 @@ struct cls_fl_filter { struct tcf_result res; struct fl_flow_key key; struct list_head list; + struct list_head hw_list; u32 handle; u32 flags; u32 in_hw_count; struct rcu_work rwork; struct net_device *hw_dev; + /* Flower classifier is unlocked, which means that its reference counter + * can be changed concurrently without any kind of external + * synchronization. Use atomic reference counter to be concurrency-safe. + */ + refcount_t refcnt; + bool deleted; }; static const struct rhashtable_params mask_ht_params = { @@ -304,7 +315,9 @@ static int fl_init(struct tcf_proto *tp) if (!head) return -ENOBUFS; + spin_lock_init(&head->masks_lock); INIT_LIST_HEAD_RCU(&head->masks); + INIT_LIST_HEAD(&head->hw_filters); rcu_assign_pointer(tp->root, head); idr_init(&head->handle_idr); @@ -313,6 +326,7 @@ static int fl_init(struct tcf_proto *tp) static void fl_mask_free(struct fl_flow_mask *mask) { + WARN_ON(!list_empty(&mask->filters)); rhashtable_destroy(&mask->ht); kfree(mask); } @@ -325,22 +339,32 @@ static void fl_mask_free_work(struct work_struct *work) fl_mask_free(mask); } -static bool fl_mask_put(struct cls_fl_head *head, struct fl_flow_mask *mask, - bool async) +static bool fl_mask_put(struct cls_fl_head *head, struct fl_flow_mask *mask) { - if (!list_empty(&mask->filters)) + if (!refcount_dec_and_test(&mask->refcnt)) return false; rhashtable_remove_fast(&head->ht, &mask->ht_node, mask_ht_params); + + spin_lock(&head->masks_lock); list_del_rcu(&mask->list); - if (async) - tcf_queue_work(&mask->rwork, fl_mask_free_work); - else - fl_mask_free(mask); + spin_unlock(&head->masks_lock); + + tcf_queue_work(&mask->rwork, fl_mask_free_work); return true; } +static struct cls_fl_head *fl_head_dereference(struct tcf_proto *tp) +{ + /* Flower classifier only changes root pointer during init and destroy. + * Users must obtain reference to tcf_proto instance before calling its + * API, so tp->root pointer is protected from concurrent call to + * fl_destroy() by reference counting. + */ + return rcu_dereference_raw(tp->root); +} + static void __fl_destroy_filter(struct cls_fl_filter *f) { tcf_exts_destroy(&f->exts); @@ -353,37 +377,50 @@ static void fl_destroy_filter_work(struct work_struct *work) struct cls_fl_filter *f = container_of(to_rcu_work(work), struct cls_fl_filter, rwork); - rtnl_lock(); __fl_destroy_filter(f); - rtnl_unlock(); } static void fl_hw_destroy_filter(struct tcf_proto *tp, struct cls_fl_filter *f, - struct netlink_ext_ack *extack) + bool rtnl_held, struct netlink_ext_ack *extack) { struct tc_cls_flower_offload cls_flower = {}; struct tcf_block *block = tp->chain->block; + if (!rtnl_held) + rtnl_lock(); + tc_cls_common_offload_init(&cls_flower.common, tp, f->flags, extack); cls_flower.command = TC_CLSFLOWER_DESTROY; cls_flower.cookie = (unsigned long) f; tc_setup_cb_call(block, TC_SETUP_CLSFLOWER, &cls_flower, false); + spin_lock(&tp->lock); + list_del_init(&f->hw_list); tcf_block_offload_dec(block, &f->flags); + spin_unlock(&tp->lock); + + if (!rtnl_held) + rtnl_unlock(); } static int fl_hw_replace_filter(struct tcf_proto *tp, - struct cls_fl_filter *f, + struct cls_fl_filter *f, bool rtnl_held, struct netlink_ext_ack *extack) { + struct cls_fl_head *head = fl_head_dereference(tp); struct tc_cls_flower_offload cls_flower = {}; struct tcf_block *block = tp->chain->block; bool skip_sw = tc_skip_sw(f->flags); - int err; + int err = 0; + + if (!rtnl_held) + rtnl_lock(); cls_flower.rule = flow_rule_alloc(tcf_exts_num_actions(&f->exts)); - if (!cls_flower.rule) - return -ENOMEM; + if (!cls_flower.rule) { + err = -ENOMEM; + goto errout; + } tc_cls_common_offload_init(&cls_flower.common, tp, f->flags, extack); cls_flower.command = TC_CLSFLOWER_REPLACE; @@ -396,35 +433,51 @@ static int fl_hw_replace_filter(struct tcf_proto *tp, err = tc_setup_flow_action(&cls_flower.rule->action, &f->exts); if (err) { kfree(cls_flower.rule); - if (skip_sw) { + if (skip_sw) NL_SET_ERR_MSG_MOD(extack, "Failed to setup flow action"); - return err; - } - return 0; + else + err = 0; + goto errout; } err = tc_setup_cb_call(block, TC_SETUP_CLSFLOWER, &cls_flower, skip_sw); kfree(cls_flower.rule); if (err < 0) { - fl_hw_destroy_filter(tp, f, NULL); - return err; + fl_hw_destroy_filter(tp, f, true, NULL); + goto errout; } else if (err > 0) { f->in_hw_count = err; + err = 0; + spin_lock(&tp->lock); tcf_block_offload_inc(block, &f->flags); + spin_unlock(&tp->lock); } - if (skip_sw && !(f->flags & TCA_CLS_FLAGS_IN_HW)) - return -EINVAL; + if (skip_sw && !(f->flags & TCA_CLS_FLAGS_IN_HW)) { + err = -EINVAL; + goto errout; + } - return 0; + spin_lock(&tp->lock); + list_add(&f->hw_list, &head->hw_filters); + spin_unlock(&tp->lock); +errout: + if (!rtnl_held) + rtnl_unlock(); + + return err; } -static void fl_hw_update_stats(struct tcf_proto *tp, struct cls_fl_filter *f) +static void fl_hw_update_stats(struct tcf_proto *tp, struct cls_fl_filter *f, + bool rtnl_held) { struct tc_cls_flower_offload cls_flower = {}; struct tcf_block *block = tp->chain->block; + if (!rtnl_held) + rtnl_lock(); + tc_cls_common_offload_init(&cls_flower.common, tp, f->flags, NULL); cls_flower.command = TC_CLSFLOWER_STATS; cls_flower.cookie = (unsigned long) f; @@ -435,27 +488,81 @@ static void fl_hw_update_stats(struct tcf_proto *tp, struct cls_fl_filter *f) tcf_exts_stats_update(&f->exts, cls_flower.stats.bytes, cls_flower.stats.pkts, cls_flower.stats.lastused); + + if (!rtnl_held) + rtnl_unlock(); } -static bool __fl_delete(struct tcf_proto *tp, struct cls_fl_filter *f, - struct netlink_ext_ack *extack) +static void __fl_put(struct cls_fl_filter *f) { - struct cls_fl_head *head = rtnl_dereference(tp->root); - bool async = tcf_exts_get_net(&f->exts); - bool last; + if (!refcount_dec_and_test(&f->refcnt)) + return; + + if (tcf_exts_get_net(&f->exts)) + tcf_queue_work(&f->rwork, fl_destroy_filter_work); + else + __fl_destroy_filter(f); +} + +static struct cls_fl_filter *__fl_get(struct cls_fl_head *head, u32 handle) +{ + struct cls_fl_filter *f; + rcu_read_lock(); + f = idr_find(&head->handle_idr, handle); + if (f && !refcount_inc_not_zero(&f->refcnt)) + f = NULL; + rcu_read_unlock(); + + return f; +} + +static struct cls_fl_filter *fl_get_next_filter(struct tcf_proto *tp, + unsigned long *handle) +{ + struct cls_fl_head *head = fl_head_dereference(tp); + struct cls_fl_filter *f; + + rcu_read_lock(); + while ((f = idr_get_next_ul(&head->handle_idr, handle))) { + /* don't return filters that are being deleted */ + if (refcount_inc_not_zero(&f->refcnt)) + break; + ++(*handle); + } + rcu_read_unlock(); + + return f; +} + +static int __fl_delete(struct tcf_proto *tp, struct cls_fl_filter *f, + bool *last, bool rtnl_held, + struct netlink_ext_ack *extack) +{ + struct cls_fl_head *head = fl_head_dereference(tp); + + *last = false; + + spin_lock(&tp->lock); + if (f->deleted) { + spin_unlock(&tp->lock); + return -ENOENT; + } + + f->deleted = true; + rhashtable_remove_fast(&f->mask->ht, &f->ht_node, + f->mask->filter_ht_params); idr_remove(&head->handle_idr, f->handle); list_del_rcu(&f->list); - last = fl_mask_put(head, f->mask, async); + spin_unlock(&tp->lock); + + *last = fl_mask_put(head, f->mask); if (!tc_skip_hw(f->flags)) - fl_hw_destroy_filter(tp, f, extack); + fl_hw_destroy_filter(tp, f, rtnl_held, extack); tcf_unbind_filter(tp, &f->res); - if (async) - tcf_queue_work(&f->rwork, fl_destroy_filter_work); - else - __fl_destroy_filter(f); + __fl_put(f); - return last; + return 0; } static void fl_destroy_sleepable(struct work_struct *work) @@ -472,13 +579,15 @@ static void fl_destroy_sleepable(struct work_struct *work) static void fl_destroy(struct tcf_proto *tp, bool rtnl_held, struct netlink_ext_ack *extack) { - struct cls_fl_head *head = rtnl_dereference(tp->root); + struct cls_fl_head *head = fl_head_dereference(tp); struct fl_flow_mask *mask, *next_mask; struct cls_fl_filter *f, *next; + bool last; list_for_each_entry_safe(mask, next_mask, &head->masks, list) { list_for_each_entry_safe(f, next, &mask->filters, list) { - if (__fl_delete(tp, f, extack)) + __fl_delete(tp, f, &last, rtnl_held, extack); + if (last) break; } } @@ -488,11 +597,18 @@ static void fl_destroy(struct tcf_proto *tp, bool rtnl_held, tcf_queue_work(&head->rwork, fl_destroy_sleepable); } +static void fl_put(struct tcf_proto *tp, void *arg) +{ + struct cls_fl_filter *f = arg; + + __fl_put(f); +} + static void *fl_get(struct tcf_proto *tp, u32 handle) { - struct cls_fl_head *head = rtnl_dereference(tp->root); + struct cls_fl_head *head = fl_head_dereference(tp); - return idr_find(&head->handle_idr, handle); + return __fl_get(head, handle); } static const struct nla_policy fl_policy[TCA_FLOWER_MAX + 1] = { @@ -768,8 +884,9 @@ static int fl_set_geneve_opt(const struct nlattr *nla, struct fl_flow_key *key, return -EINVAL; } - err = nla_parse_nested(tb, TCA_FLOWER_KEY_ENC_OPT_GENEVE_MAX, - nla, geneve_opt_policy, extack); + err = nla_parse_nested_deprecated(tb, + TCA_FLOWER_KEY_ENC_OPT_GENEVE_MAX, + nla, geneve_opt_policy, extack); if (err < 0) return err; @@ -831,18 +948,18 @@ static int fl_set_enc_opt(struct nlattr **tb, struct fl_flow_key *key, const struct nlattr *nla_enc_key, *nla_opt_key, *nla_opt_msk = NULL; int err, option_len, key_depth, msk_depth = 0; - err = nla_validate_nested(tb[TCA_FLOWER_KEY_ENC_OPTS], - TCA_FLOWER_KEY_ENC_OPTS_MAX, - enc_opts_policy, extack); + err = nla_validate_nested_deprecated(tb[TCA_FLOWER_KEY_ENC_OPTS], + TCA_FLOWER_KEY_ENC_OPTS_MAX, + enc_opts_policy, extack); if (err) return err; nla_enc_key = nla_data(tb[TCA_FLOWER_KEY_ENC_OPTS]); if (tb[TCA_FLOWER_KEY_ENC_OPTS_MASK]) { - err = nla_validate_nested(tb[TCA_FLOWER_KEY_ENC_OPTS_MASK], - TCA_FLOWER_KEY_ENC_OPTS_MAX, - enc_opts_policy, extack); + err = nla_validate_nested_deprecated(tb[TCA_FLOWER_KEY_ENC_OPTS_MASK], + TCA_FLOWER_KEY_ENC_OPTS_MAX, + enc_opts_policy, extack); if (err) return err; @@ -1227,12 +1344,18 @@ static struct fl_flow_mask *fl_create_new_mask(struct cls_fl_head *head, INIT_LIST_HEAD_RCU(&newmask->filters); - err = rhashtable_insert_fast(&head->ht, &newmask->ht_node, - mask_ht_params); + refcount_set(&newmask->refcnt, 1); + err = rhashtable_replace_fast(&head->ht, &mask->ht_node, + &newmask->ht_node, mask_ht_params); if (err) goto errout_destroy; + /* Wait until any potential concurrent users of mask are finished */ + synchronize_rcu(); + + spin_lock(&head->masks_lock); list_add_tail_rcu(&newmask->list, &head->masks); + spin_unlock(&head->masks_lock); return newmask; @@ -1250,41 +1373,77 @@ static int fl_check_assign_mask(struct cls_fl_head *head, struct fl_flow_mask *mask) { struct fl_flow_mask *newmask; + int ret = 0; + + rcu_read_lock(); - fnew->mask = rhashtable_lookup_fast(&head->ht, mask, mask_ht_params); + /* Insert mask as temporary node to prevent concurrent creation of mask + * with same key. Any concurrent lookups with same key will return + * -EAGAIN because mask's refcnt is zero. It is safe to insert + * stack-allocated 'mask' to masks hash table because we call + * synchronize_rcu() before returning from this function (either in case + * of error or after replacing it with heap-allocated mask in + * fl_create_new_mask()). + */ + fnew->mask = rhashtable_lookup_get_insert_fast(&head->ht, + &mask->ht_node, + mask_ht_params); if (!fnew->mask) { - if (fold) - return -EINVAL; + rcu_read_unlock(); + + if (fold) { + ret = -EINVAL; + goto errout_cleanup; + } newmask = fl_create_new_mask(head, mask); - if (IS_ERR(newmask)) - return PTR_ERR(newmask); + if (IS_ERR(newmask)) { + ret = PTR_ERR(newmask); + goto errout_cleanup; + } fnew->mask = newmask; + return 0; + } else if (IS_ERR(fnew->mask)) { + ret = PTR_ERR(fnew->mask); } else if (fold && fold->mask != fnew->mask) { - return -EINVAL; + ret = -EINVAL; + } else if (!refcount_inc_not_zero(&fnew->mask->refcnt)) { + /* Mask was deleted concurrently, try again */ + ret = -EAGAIN; } + rcu_read_unlock(); + return ret; - return 0; +errout_cleanup: + rhashtable_remove_fast(&head->ht, &mask->ht_node, + mask_ht_params); + /* Wait until any potential concurrent users of mask are finished */ + synchronize_rcu(); + return ret; } static int fl_set_parms(struct net *net, struct tcf_proto *tp, struct cls_fl_filter *f, struct fl_flow_mask *mask, unsigned long base, struct nlattr **tb, struct nlattr *est, bool ovr, - struct fl_flow_tmplt *tmplt, + struct fl_flow_tmplt *tmplt, bool rtnl_held, struct netlink_ext_ack *extack) { int err; - err = tcf_exts_validate(net, tp, tb, est, &f->exts, ovr, true, + err = tcf_exts_validate(net, tp, tb, est, &f->exts, ovr, rtnl_held, extack); if (err < 0) return err; if (tb[TCA_FLOWER_CLASSID]) { f->res.classid = nla_get_u32(tb[TCA_FLOWER_CLASSID]); + if (!rtnl_held) + rtnl_lock(); tcf_bind_filter(tp, &f->res, base); + if (!rtnl_held) + rtnl_unlock(); } err = fl_set_key(net, tb, &f->key, &mask->key, extack); @@ -1302,25 +1461,52 @@ static int fl_set_parms(struct net *net, struct tcf_proto *tp, return 0; } +static int fl_ht_insert_unique(struct cls_fl_filter *fnew, + struct cls_fl_filter *fold, + bool *in_ht) +{ + struct fl_flow_mask *mask = fnew->mask; + int err; + + err = rhashtable_lookup_insert_fast(&mask->ht, + &fnew->ht_node, + mask->filter_ht_params); + if (err) { + *in_ht = false; + /* It is okay if filter with same key exists when + * overwriting. + */ + return fold && err == -EEXIST ? 0 : err; + } + + *in_ht = true; + return 0; +} + static int fl_change(struct net *net, struct sk_buff *in_skb, struct tcf_proto *tp, unsigned long base, u32 handle, struct nlattr **tca, void **arg, bool ovr, bool rtnl_held, struct netlink_ext_ack *extack) { - struct cls_fl_head *head = rtnl_dereference(tp->root); + struct cls_fl_head *head = fl_head_dereference(tp); struct cls_fl_filter *fold = *arg; struct cls_fl_filter *fnew; struct fl_flow_mask *mask; struct nlattr **tb; + bool in_ht; int err; - if (!tca[TCA_OPTIONS]) - return -EINVAL; + if (!tca[TCA_OPTIONS]) { + err = -EINVAL; + goto errout_fold; + } mask = kzalloc(sizeof(struct fl_flow_mask), GFP_KERNEL); - if (!mask) - return -ENOBUFS; + if (!mask) { + err = -ENOBUFS; + goto errout_fold; + } tb = kcalloc(TCA_FLOWER_MAX + 1, sizeof(struct nlattr *), GFP_KERNEL); if (!tb) { @@ -1328,8 +1514,8 @@ static int fl_change(struct net *net, struct sk_buff *in_skb, goto errout_mask_alloc; } - err = nla_parse_nested(tb, TCA_FLOWER_MAX, tca[TCA_OPTIONS], - fl_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_FLOWER_MAX, + tca[TCA_OPTIONS], fl_policy, NULL); if (err < 0) goto errout_tb; @@ -1343,6 +1529,8 @@ static int fl_change(struct net *net, struct sk_buff *in_skb, err = -ENOBUFS; goto errout_tb; } + INIT_LIST_HEAD(&fnew->hw_list); + refcount_set(&fnew->refcnt, 1); err = tcf_exts_init(&fnew->exts, net, TCA_FLOWER_ACT, 0); if (err < 0) @@ -1358,7 +1546,7 @@ static int fl_change(struct net *net, struct sk_buff *in_skb, } err = fl_set_parms(net, tp, fnew, mask, base, tb, tca[TCA_RATE], ovr, - tp->chain->tmplt_priv, extack); + tp->chain->tmplt_priv, rtnl_held, extack); if (err) goto errout; @@ -1366,169 +1554,247 @@ static int fl_change(struct net *net, struct sk_buff *in_skb, if (err) goto errout; - if (!handle) { - handle = 1; - err = idr_alloc_u32(&head->handle_idr, fnew, &handle, - INT_MAX, GFP_KERNEL); - } else if (!fold) { - /* user specifies a handle and it doesn't exist */ - err = idr_alloc_u32(&head->handle_idr, fnew, &handle, - handle, GFP_KERNEL); - } + err = fl_ht_insert_unique(fnew, fold, &in_ht); if (err) goto errout_mask; - fnew->handle = handle; - - if (!fold && __fl_lookup(fnew->mask, &fnew->mkey)) { - err = -EEXIST; - goto errout_idr; - } - - err = rhashtable_insert_fast(&fnew->mask->ht, &fnew->ht_node, - fnew->mask->filter_ht_params); - if (err) - goto errout_idr; if (!tc_skip_hw(fnew->flags)) { - err = fl_hw_replace_filter(tp, fnew, extack); + err = fl_hw_replace_filter(tp, fnew, rtnl_held, extack); if (err) - goto errout_mask_ht; + goto errout_ht; } if (!tc_in_hw(fnew->flags)) fnew->flags |= TCA_CLS_FLAGS_NOT_IN_HW; + spin_lock(&tp->lock); + + /* tp was deleted concurrently. -EAGAIN will cause caller to lookup + * proto again or create new one, if necessary. + */ + if (tp->deleting) { + err = -EAGAIN; + goto errout_hw; + } + if (fold) { + /* Fold filter was deleted concurrently. Retry lookup. */ + if (fold->deleted) { + err = -EAGAIN; + goto errout_hw; + } + + fnew->handle = handle; + + if (!in_ht) { + struct rhashtable_params params = + fnew->mask->filter_ht_params; + + err = rhashtable_insert_fast(&fnew->mask->ht, + &fnew->ht_node, + params); + if (err) + goto errout_hw; + in_ht = true; + } + + refcount_inc(&fnew->refcnt); rhashtable_remove_fast(&fold->mask->ht, &fold->ht_node, fold->mask->filter_ht_params); - if (!tc_skip_hw(fold->flags)) - fl_hw_destroy_filter(tp, fold, NULL); - } - - *arg = fnew; - - if (fold) { idr_replace(&head->handle_idr, fnew, fnew->handle); list_replace_rcu(&fold->list, &fnew->list); + fold->deleted = true; + + spin_unlock(&tp->lock); + + fl_mask_put(head, fold->mask); + if (!tc_skip_hw(fold->flags)) + fl_hw_destroy_filter(tp, fold, rtnl_held, NULL); tcf_unbind_filter(tp, &fold->res); - tcf_exts_get_net(&fold->exts); - tcf_queue_work(&fold->rwork, fl_destroy_filter_work); + /* Caller holds reference to fold, so refcnt is always > 0 + * after this. + */ + refcount_dec(&fold->refcnt); + __fl_put(fold); } else { + if (handle) { + /* user specifies a handle and it doesn't exist */ + err = idr_alloc_u32(&head->handle_idr, fnew, &handle, + handle, GFP_ATOMIC); + + /* Filter with specified handle was concurrently + * inserted after initial check in cls_api. This is not + * necessarily an error if NLM_F_EXCL is not set in + * message flags. Returning EAGAIN will cause cls_api to + * try to update concurrently inserted rule. + */ + if (err == -ENOSPC) + err = -EAGAIN; + } else { + handle = 1; + err = idr_alloc_u32(&head->handle_idr, fnew, &handle, + INT_MAX, GFP_ATOMIC); + } + if (err) + goto errout_hw; + + refcount_inc(&fnew->refcnt); + fnew->handle = handle; list_add_tail_rcu(&fnew->list, &fnew->mask->filters); + spin_unlock(&tp->lock); } + *arg = fnew; + kfree(tb); kfree(mask); return 0; -errout_mask_ht: - rhashtable_remove_fast(&fnew->mask->ht, &fnew->ht_node, - fnew->mask->filter_ht_params); - -errout_idr: - if (!fold) - idr_remove(&head->handle_idr, fnew->handle); - +errout_ht: + spin_lock(&tp->lock); +errout_hw: + fnew->deleted = true; + spin_unlock(&tp->lock); + if (!tc_skip_hw(fnew->flags)) + fl_hw_destroy_filter(tp, fnew, rtnl_held, NULL); + if (in_ht) + rhashtable_remove_fast(&fnew->mask->ht, &fnew->ht_node, + fnew->mask->filter_ht_params); errout_mask: - fl_mask_put(head, fnew->mask, false); - + fl_mask_put(head, fnew->mask); errout: - tcf_exts_destroy(&fnew->exts); - kfree(fnew); + __fl_put(fnew); errout_tb: kfree(tb); errout_mask_alloc: kfree(mask); +errout_fold: + if (fold) + __fl_put(fold); return err; } static int fl_delete(struct tcf_proto *tp, void *arg, bool *last, bool rtnl_held, struct netlink_ext_ack *extack) { - struct cls_fl_head *head = rtnl_dereference(tp->root); + struct cls_fl_head *head = fl_head_dereference(tp); struct cls_fl_filter *f = arg; + bool last_on_mask; + int err = 0; - rhashtable_remove_fast(&f->mask->ht, &f->ht_node, - f->mask->filter_ht_params); - __fl_delete(tp, f, extack); + err = __fl_delete(tp, f, &last_on_mask, rtnl_held, extack); *last = list_empty(&head->masks); - return 0; + __fl_put(f); + + return err; } static void fl_walk(struct tcf_proto *tp, struct tcf_walker *arg, bool rtnl_held) { - struct cls_fl_head *head = rtnl_dereference(tp->root); struct cls_fl_filter *f; arg->count = arg->skip; - while ((f = idr_get_next_ul(&head->handle_idr, - &arg->cookie)) != NULL) { + while ((f = fl_get_next_filter(tp, &arg->cookie)) != NULL) { if (arg->fn(tp, f, arg) < 0) { + __fl_put(f); arg->stop = 1; break; } - arg->cookie = f->handle + 1; + __fl_put(f); + arg->cookie++; arg->count++; } } +static struct cls_fl_filter * +fl_get_next_hw_filter(struct tcf_proto *tp, struct cls_fl_filter *f, bool add) +{ + struct cls_fl_head *head = fl_head_dereference(tp); + + spin_lock(&tp->lock); + if (list_empty(&head->hw_filters)) { + spin_unlock(&tp->lock); + return NULL; + } + + if (!f) + f = list_entry(&head->hw_filters, struct cls_fl_filter, + hw_list); + list_for_each_entry_continue(f, &head->hw_filters, hw_list) { + if (!(add && f->deleted) && refcount_inc_not_zero(&f->refcnt)) { + spin_unlock(&tp->lock); + return f; + } + } + + spin_unlock(&tp->lock); + return NULL; +} + static int fl_reoffload(struct tcf_proto *tp, bool add, tc_setup_cb_t *cb, void *cb_priv, struct netlink_ext_ack *extack) { - struct cls_fl_head *head = rtnl_dereference(tp->root); struct tc_cls_flower_offload cls_flower = {}; struct tcf_block *block = tp->chain->block; - struct fl_flow_mask *mask; - struct cls_fl_filter *f; + struct cls_fl_filter *f = NULL; int err; - list_for_each_entry(mask, &head->masks, list) { - list_for_each_entry(f, &mask->filters, list) { - if (tc_skip_hw(f->flags)) - continue; - - cls_flower.rule = - flow_rule_alloc(tcf_exts_num_actions(&f->exts)); - if (!cls_flower.rule) - return -ENOMEM; - - tc_cls_common_offload_init(&cls_flower.common, tp, - f->flags, extack); - cls_flower.command = add ? - TC_CLSFLOWER_REPLACE : TC_CLSFLOWER_DESTROY; - cls_flower.cookie = (unsigned long)f; - cls_flower.rule->match.dissector = &mask->dissector; - cls_flower.rule->match.mask = &mask->key; - cls_flower.rule->match.key = &f->mkey; - - err = tc_setup_flow_action(&cls_flower.rule->action, - &f->exts); - if (err) { - kfree(cls_flower.rule); - if (tc_skip_sw(f->flags)) { - NL_SET_ERR_MSG_MOD(extack, "Failed to setup flow action"); - return err; - } - continue; + /* hw_filters list can only be changed by hw offload functions after + * obtaining rtnl lock. Make sure it is not changed while reoffload is + * iterating it. + */ + ASSERT_RTNL(); + + while ((f = fl_get_next_hw_filter(tp, f, add))) { + cls_flower.rule = + flow_rule_alloc(tcf_exts_num_actions(&f->exts)); + if (!cls_flower.rule) { + __fl_put(f); + return -ENOMEM; + } + + tc_cls_common_offload_init(&cls_flower.common, tp, f->flags, + extack); + cls_flower.command = add ? + TC_CLSFLOWER_REPLACE : TC_CLSFLOWER_DESTROY; + cls_flower.cookie = (unsigned long)f; + cls_flower.rule->match.dissector = &f->mask->dissector; + cls_flower.rule->match.mask = &f->mask->key; + cls_flower.rule->match.key = &f->mkey; + + err = tc_setup_flow_action(&cls_flower.rule->action, &f->exts); + if (err) { + kfree(cls_flower.rule); + if (tc_skip_sw(f->flags)) { + NL_SET_ERR_MSG_MOD(extack, "Failed to setup flow action"); + __fl_put(f); + return err; } + goto next_flow; + } - cls_flower.classid = f->res.classid; + cls_flower.classid = f->res.classid; - err = cb(TC_SETUP_CLSFLOWER, &cls_flower, cb_priv); - kfree(cls_flower.rule); + err = cb(TC_SETUP_CLSFLOWER, &cls_flower, cb_priv); + kfree(cls_flower.rule); - if (err) { - if (add && tc_skip_sw(f->flags)) - return err; - continue; + if (err) { + if (add && tc_skip_sw(f->flags)) { + __fl_put(f); + return err; } - - tc_cls_offload_cnt_update(block, &f->in_hw_count, - &f->flags, add); + goto next_flow; } + + spin_lock(&tp->lock); + tc_cls_offload_cnt_update(block, &f->in_hw_count, &f->flags, + add); + spin_unlock(&tp->lock); +next_flow: + __fl_put(f); } return 0; @@ -1587,8 +1853,8 @@ static void *fl_tmplt_create(struct net *net, struct tcf_chain *chain, tb = kcalloc(TCA_FLOWER_MAX + 1, sizeof(struct nlattr *), GFP_KERNEL); if (!tb) return ERR_PTR(-ENOBUFS); - err = nla_parse_nested(tb, TCA_FLOWER_MAX, tca[TCA_OPTIONS], - fl_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_FLOWER_MAX, + tca[TCA_OPTIONS], fl_policy, NULL); if (err) goto errout_tb; @@ -1786,7 +2052,7 @@ static int fl_dump_key_geneve_opt(struct sk_buff *skb, struct nlattr *nest; int opt_off = 0; - nest = nla_nest_start(skb, TCA_FLOWER_KEY_ENC_OPTS_GENEVE); + nest = nla_nest_start_noflag(skb, TCA_FLOWER_KEY_ENC_OPTS_GENEVE); if (!nest) goto nla_put_failure; @@ -1822,7 +2088,7 @@ static int fl_dump_key_options(struct sk_buff *skb, int enc_opt_type, if (!enc_opts->len) return 0; - nest = nla_nest_start(skb, enc_opt_type); + nest = nla_nest_start_noflag(skb, enc_opt_type); if (!nest) goto nla_put_failure; @@ -2061,31 +2327,37 @@ static int fl_dump(struct net *net, struct tcf_proto *tp, void *fh, struct cls_fl_filter *f = fh; struct nlattr *nest; struct fl_flow_key *key, *mask; + bool skip_hw; if (!f) return skb->len; t->tcm_handle = f->handle; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (!nest) goto nla_put_failure; + spin_lock(&tp->lock); + if (f->res.classid && nla_put_u32(skb, TCA_FLOWER_CLASSID, f->res.classid)) - goto nla_put_failure; + goto nla_put_failure_locked; key = &f->key; mask = &f->mask->key; + skip_hw = tc_skip_hw(f->flags); if (fl_dump_key(skb, net, key, mask)) - goto nla_put_failure; - - if (!tc_skip_hw(f->flags)) - fl_hw_update_stats(tp, f); + goto nla_put_failure_locked; if (f->flags && nla_put_u32(skb, TCA_FLOWER_FLAGS, f->flags)) - goto nla_put_failure; + goto nla_put_failure_locked; + + spin_unlock(&tp->lock); + + if (!skip_hw) + fl_hw_update_stats(tp, f, rtnl_held); if (nla_put_u32(skb, TCA_FLOWER_IN_HW_COUNT, f->in_hw_count)) goto nla_put_failure; @@ -2100,6 +2372,8 @@ static int fl_dump(struct net *net, struct tcf_proto *tp, void *fh, return skb->len; +nla_put_failure_locked: + spin_unlock(&tp->lock); nla_put_failure: nla_nest_cancel(skb, nest); return -1; @@ -2111,7 +2385,7 @@ static int fl_tmplt_dump(struct sk_buff *skb, struct net *net, void *tmplt_priv) struct fl_flow_key *key, *mask; struct nlattr *nest; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (!nest) goto nla_put_failure; @@ -2144,6 +2418,7 @@ static struct tcf_proto_ops cls_fl_ops __read_mostly = { .init = fl_init, .destroy = fl_destroy, .get = fl_get, + .put = fl_put, .change = fl_change, .delete = fl_delete, .walk = fl_walk, @@ -2154,6 +2429,7 @@ static struct tcf_proto_ops cls_fl_ops __read_mostly = { .tmplt_destroy = fl_tmplt_destroy, .tmplt_dump = fl_tmplt_dump, .owner = THIS_MODULE, + .flags = TCF_PROTO_OPS_DOIT_UNLOCKED, }; static int __init cls_fl_init(void) diff --git a/net/sched/cls_fw.c b/net/sched/cls_fw.c index ad036b00427d92859393773eb0a9d99e2da14e4b..1d0b39c3932f945b7e7e221ec13055374bc5ba9a 100644 --- a/net/sched/cls_fw.c +++ b/net/sched/cls_fw.c @@ -263,7 +263,8 @@ static int fw_change(struct net *net, struct sk_buff *in_skb, if (!opt) return handle ? -EINVAL : 0; /* Succeed if it is old method. */ - err = nla_parse_nested(tb, TCA_FW_MAX, opt, fw_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_FW_MAX, opt, fw_policy, + NULL); if (err < 0) return err; @@ -402,7 +403,7 @@ static int fw_dump(struct net *net, struct tcf_proto *tp, void *fh, if (!f->res.classid && !tcf_exts_has_actions(&f->exts)) return skb->len; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (nest == NULL) goto nla_put_failure; diff --git a/net/sched/cls_matchall.c b/net/sched/cls_matchall.c index a13bc351a4148f40f434b25a9adffc4cf9137548..1e98a517fb0b47f59c2cdf0235245e36c08a4222 100644 --- a/net/sched/cls_matchall.c +++ b/net/sched/cls_matchall.c @@ -32,6 +32,9 @@ static int mall_classify(struct sk_buff *skb, const struct tcf_proto *tp, { struct cls_mall_head *head = rcu_dereference_bh(tp->root); + if (unlikely(!head)) + return -1; + if (tc_skip_sw(head->flags)) return -1; @@ -89,12 +92,29 @@ static int mall_replace_hw_filter(struct tcf_proto *tp, bool skip_sw = tc_skip_sw(head->flags); int err; + cls_mall.rule = flow_rule_alloc(tcf_exts_num_actions(&head->exts)); + if (!cls_mall.rule) + return -ENOMEM; + tc_cls_common_offload_init(&cls_mall.common, tp, head->flags, extack); cls_mall.command = TC_CLSMATCHALL_REPLACE; - cls_mall.exts = &head->exts; cls_mall.cookie = cookie; + err = tc_setup_flow_action(&cls_mall.rule->action, &head->exts); + if (err) { + kfree(cls_mall.rule); + mall_destroy_hw_filter(tp, head, cookie, NULL); + if (skip_sw) + NL_SET_ERR_MSG_MOD(extack, "Failed to setup flow action"); + else + err = 0; + + return err; + } + err = tc_setup_cb_call(block, TC_SETUP_CLSMATCHALL, &cls_mall, skip_sw); + kfree(cls_mall.rule); + if (err < 0) { mall_destroy_hw_filter(tp, head, cookie, NULL); return err; @@ -181,8 +201,8 @@ static int mall_change(struct net *net, struct sk_buff *in_skb, if (head) return -EEXIST; - err = nla_parse_nested(tb, TCA_MATCHALL_MAX, tca[TCA_OPTIONS], - mall_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_MATCHALL_MAX, + tca[TCA_OPTIONS], mall_policy, NULL); if (err < 0) return err; @@ -272,13 +292,27 @@ static int mall_reoffload(struct tcf_proto *tp, bool add, tc_setup_cb_t *cb, if (tc_skip_hw(head->flags)) return 0; + cls_mall.rule = flow_rule_alloc(tcf_exts_num_actions(&head->exts)); + if (!cls_mall.rule) + return -ENOMEM; + tc_cls_common_offload_init(&cls_mall.common, tp, head->flags, extack); cls_mall.command = add ? TC_CLSMATCHALL_REPLACE : TC_CLSMATCHALL_DESTROY; - cls_mall.exts = &head->exts; cls_mall.cookie = (unsigned long)head; + err = tc_setup_flow_action(&cls_mall.rule->action, &head->exts); + if (err) { + kfree(cls_mall.rule); + if (add && tc_skip_sw(head->flags)) { + NL_SET_ERR_MSG_MOD(extack, "Failed to setup flow action"); + return err; + } + } + err = cb(TC_SETUP_CLSMATCHALL, &cls_mall, cb_priv); + kfree(cls_mall.rule); + if (err) { if (add && tc_skip_sw(head->flags)) return err; @@ -290,6 +324,23 @@ static int mall_reoffload(struct tcf_proto *tp, bool add, tc_setup_cb_t *cb, return 0; } +static void mall_stats_hw_filter(struct tcf_proto *tp, + struct cls_mall_head *head, + unsigned long cookie) +{ + struct tc_cls_matchall_offload cls_mall = {}; + struct tcf_block *block = tp->chain->block; + + tc_cls_common_offload_init(&cls_mall.common, tp, head->flags, NULL); + cls_mall.command = TC_CLSMATCHALL_STATS; + cls_mall.cookie = cookie; + + tc_setup_cb_call(block, TC_SETUP_CLSMATCHALL, &cls_mall, false); + + tcf_exts_stats_update(&head->exts, cls_mall.stats.bytes, + cls_mall.stats.pkts, cls_mall.stats.lastused); +} + static int mall_dump(struct net *net, struct tcf_proto *tp, void *fh, struct sk_buff *skb, struct tcmsg *t, bool rtnl_held) { @@ -301,9 +352,12 @@ static int mall_dump(struct net *net, struct tcf_proto *tp, void *fh, if (!head) return skb->len; + if (!tc_skip_hw(head->flags)) + mall_stats_hw_filter(tp, head, (unsigned long)head); + t->tcm_handle = head->handle; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (!nest) goto nla_put_failure; diff --git a/net/sched/cls_route.c b/net/sched/cls_route.c index f006af23b64a8e85c066f404a2761c85b2d3a77a..eeff5bbfb91291c9dab0273eef4b8c991fdaec73 100644 --- a/net/sched/cls_route.c +++ b/net/sched/cls_route.c @@ -484,7 +484,8 @@ static int route4_change(struct net *net, struct sk_buff *in_skb, if (opt == NULL) return handle ? -EINVAL : 0; - err = nla_parse_nested(tb, TCA_ROUTE4_MAX, opt, route4_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_ROUTE4_MAX, opt, + route4_policy, NULL); if (err < 0) return err; @@ -607,7 +608,7 @@ static int route4_dump(struct net *net, struct tcf_proto *tp, void *fh, t->tcm_handle = f->handle; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (nest == NULL) goto nla_put_failure; diff --git a/net/sched/cls_rsvp.h b/net/sched/cls_rsvp.h index 0719a21d9c4127fae62754986e2d620b3c99422f..a4688bb92f43ed86c5d15ade166e596c9445101a 100644 --- a/net/sched/cls_rsvp.h +++ b/net/sched/cls_rsvp.h @@ -497,7 +497,8 @@ static int rsvp_change(struct net *net, struct sk_buff *in_skb, if (opt == NULL) return handle ? -EINVAL : 0; - err = nla_parse_nested(tb, TCA_RSVP_MAX, opt, rsvp_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_RSVP_MAX, opt, rsvp_policy, + NULL); if (err < 0) return err; @@ -706,7 +707,7 @@ static int rsvp_dump(struct net *net, struct tcf_proto *tp, void *fh, t->tcm_handle = f->handle; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (nest == NULL) goto nla_put_failure; diff --git a/net/sched/cls_tcindex.c b/net/sched/cls_tcindex.c index 24e0a62a65ccbe720f0648f1b68fdbbf4eea2dde..9f4f4203c3880118a74614c72e02cf27d882e48c 100644 --- a/net/sched/cls_tcindex.c +++ b/net/sched/cls_tcindex.c @@ -510,7 +510,8 @@ tcindex_change(struct net *net, struct sk_buff *in_skb, if (!opt) return 0; - err = nla_parse_nested(tb, TCA_TCINDEX_MAX, opt, tcindex_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_TCINDEX_MAX, opt, + tcindex_policy, NULL); if (err < 0) return err; @@ -601,7 +602,7 @@ static int tcindex_dump(struct net *net, struct tcf_proto *tp, void *fh, tp, fh, skb, t, p, r); pr_debug("p->perfect %p p->h %p\n", p->perfect, p->h); - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (nest == NULL) goto nla_put_failure; diff --git a/net/sched/cls_u32.c b/net/sched/cls_u32.c index 48e76a3acf8a77457cd85381b1b434720420e419..4b8710a266cca9a06e7390f7f36bd226b1bc9cb9 100644 --- a/net/sched/cls_u32.c +++ b/net/sched/cls_u32.c @@ -847,7 +847,7 @@ static struct tc_u_knode *u32_init_knode(struct net *net, struct tcf_proto *tp, /* Similarly success statistics must be moved as pointers */ new->pcpu_success = n->pcpu_success; #endif - memcpy(&new->sel, s, sizeof(*s) + s->nkeys*sizeof(struct tc_u32_key)); + memcpy(&new->sel, s, struct_size(s, keys, s->nkeys)); if (tcf_exts_init(&new->exts, net, TCA_U32_ACT, TCA_U32_POLICE)) { kfree(new); @@ -884,7 +884,8 @@ static int u32_change(struct net *net, struct sk_buff *in_skb, } } - err = nla_parse_nested(tb, TCA_U32_MAX, opt, u32_policy, extack); + err = nla_parse_nested_deprecated(tb, TCA_U32_MAX, opt, u32_policy, + extack); if (err < 0) return err; @@ -1294,7 +1295,7 @@ static int u32_dump(struct net *net, struct tcf_proto *tp, void *fh, t->tcm_handle = n->handle; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (nest == NULL) goto nla_put_failure; diff --git a/net/sched/em_ipt.c b/net/sched/em_ipt.c index a5f34e930eff5206bfd0b0990045395f60d7aa0a..60c26b8294b5ceee2eda04490485fe737a765b28 100644 --- a/net/sched/em_ipt.c +++ b/net/sched/em_ipt.c @@ -120,8 +120,8 @@ static int em_ipt_change(struct net *net, void *data, int data_len, struct xt_match *match; int mdata_len, ret; - ret = nla_parse(tb, TCA_EM_IPT_MAX, data, data_len, em_ipt_policy, - NULL); + ret = nla_parse_deprecated(tb, TCA_EM_IPT_MAX, data, data_len, + em_ipt_policy, NULL); if (ret < 0) return ret; diff --git a/net/sched/em_meta.c b/net/sched/em_meta.c index d6e97115500bcda961014071178886b120b65952..28dfa8f2a4ea0467b7c12f0a71f49743b39ba32d 100644 --- a/net/sched/em_meta.c +++ b/net/sched/em_meta.c @@ -912,7 +912,8 @@ static int em_meta_change(struct net *net, void *data, int len, struct tcf_meta_hdr *hdr; struct meta_match *meta = NULL; - err = nla_parse(tb, TCA_EM_META_MAX, data, len, meta_policy, NULL); + err = nla_parse_deprecated(tb, TCA_EM_META_MAX, data, len, + meta_policy, NULL); if (err < 0) goto errout; diff --git a/net/sched/ematch.c b/net/sched/ematch.c index 1331a4c2d8ff55fdc802b85b819939836feda4e1..7b86c2a44746deb1d6f2d727ef6d6e0078a66fc3 100644 --- a/net/sched/ematch.c +++ b/net/sched/ematch.c @@ -314,7 +314,8 @@ int tcf_em_tree_validate(struct tcf_proto *tp, struct nlattr *nla, if (!nla) return 0; - err = nla_parse_nested(tb, TCA_EMATCH_TREE_MAX, nla, em_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_EMATCH_TREE_MAX, nla, + em_policy, NULL); if (err < 0) goto errout; @@ -440,14 +441,14 @@ int tcf_em_tree_dump(struct sk_buff *skb, struct tcf_ematch_tree *tree, int tlv) struct nlattr *top_start; struct nlattr *list_start; - top_start = nla_nest_start(skb, tlv); + top_start = nla_nest_start_noflag(skb, tlv); if (top_start == NULL) goto nla_put_failure; if (nla_put(skb, TCA_EMATCH_TREE_HDR, sizeof(tree->hdr), &tree->hdr)) goto nla_put_failure; - list_start = nla_nest_start(skb, TCA_EMATCH_TREE_LIST); + list_start = nla_nest_start_noflag(skb, TCA_EMATCH_TREE_LIST); if (list_start == NULL) goto nla_put_failure; diff --git a/net/sched/sch_api.c b/net/sched/sch_api.c index fb8f138b97763bdf917b202ba1b93ad773d0407f..607e84d67c332cc6db818d1cce5d66a4874eeeee 100644 --- a/net/sched/sch_api.c +++ b/net/sched/sch_api.c @@ -479,7 +479,8 @@ static struct qdisc_size_table *qdisc_get_stab(struct nlattr *opt, u16 *tab = NULL; int err; - err = nla_parse_nested(tb, TCA_STAB_MAX, opt, stab_policy, extack); + err = nla_parse_nested_deprecated(tb, TCA_STAB_MAX, opt, stab_policy, + extack); if (err < 0) return ERR_PTR(err); if (!tb[TCA_STAB_BASE]) { @@ -542,7 +543,7 @@ static int qdisc_dump_stab(struct sk_buff *skb, struct qdisc_size_table *stab) { struct nlattr *nest; - nest = nla_nest_start(skb, TCA_STAB); + nest = nla_nest_start_noflag(skb, TCA_STAB); if (nest == NULL) goto nla_put_failure; if (nla_put(skb, TCA_STAB_BASE, sizeof(stab->szopts), &stab->szopts)) @@ -998,6 +999,19 @@ static void notify_and_destroy(struct net *net, struct sk_buff *skb, qdisc_put(old); } +static void qdisc_clear_nolock(struct Qdisc *sch) +{ + sch->flags &= ~TCQ_F_NOLOCK; + if (!(sch->flags & TCQ_F_CPUSTATS)) + return; + + free_percpu(sch->cpu_bstats); + free_percpu(sch->cpu_qstats); + sch->cpu_bstats = NULL; + sch->cpu_qstats = NULL; + sch->flags &= ~TCQ_F_CPUSTATS; +} + /* Graft qdisc "new" to class "classid" of qdisc "parent" or * to device "dev". * @@ -1076,7 +1090,7 @@ skip: /* Only support running class lockless if parent is lockless */ if (new && (new->flags & TCQ_F_NOLOCK) && parent && !(parent->flags & TCQ_F_NOLOCK)) - new->flags &= ~TCQ_F_NOLOCK; + qdisc_clear_nolock(new); if (!cops || !cops->graft) return -EOPNOTSUPP; @@ -1410,8 +1424,8 @@ static int tc_get_qdisc(struct sk_buff *skb, struct nlmsghdr *n, !netlink_ns_capable(skb, net->user_ns, CAP_NET_ADMIN)) return -EPERM; - err = nlmsg_parse(n, sizeof(*tcm), tca, TCA_MAX, rtm_tca_policy, - extack); + err = nlmsg_parse_deprecated(n, sizeof(*tcm), tca, TCA_MAX, + rtm_tca_policy, extack); if (err < 0) return err; @@ -1495,8 +1509,8 @@ static int tc_modify_qdisc(struct sk_buff *skb, struct nlmsghdr *n, replay: /* Reinit, just in case something touches this. */ - err = nlmsg_parse(n, sizeof(*tcm), tca, TCA_MAX, rtm_tca_policy, - extack); + err = nlmsg_parse_deprecated(n, sizeof(*tcm), tca, TCA_MAX, + rtm_tca_policy, extack); if (err < 0) return err; @@ -1730,8 +1744,8 @@ static int tc_dump_qdisc(struct sk_buff *skb, struct netlink_callback *cb) idx = 0; ASSERT_RTNL(); - err = nlmsg_parse(nlh, sizeof(struct tcmsg), tca, TCA_MAX, - rtm_tca_policy, cb->extack); + err = nlmsg_parse_deprecated(nlh, sizeof(struct tcmsg), tca, TCA_MAX, + rtm_tca_policy, cb->extack); if (err < 0) return err; @@ -1959,8 +1973,8 @@ static int tc_ctl_tclass(struct sk_buff *skb, struct nlmsghdr *n, !netlink_ns_capable(skb, net->user_ns, CAP_NET_ADMIN)) return -EPERM; - err = nlmsg_parse(n, sizeof(*tcm), tca, TCA_MAX, rtm_tca_policy, - extack); + err = nlmsg_parse_deprecated(n, sizeof(*tcm), tca, TCA_MAX, + rtm_tca_policy, extack); if (err < 0) return err; diff --git a/net/sched/sch_atm.c b/net/sched/sch_atm.c index d714d3747bcb3ced42cbfddb3988db7363e34e1c..ae506c7906cd8afda910098aa91e08fde1b5fad3 100644 --- a/net/sched/sch_atm.c +++ b/net/sched/sch_atm.c @@ -223,7 +223,8 @@ static int atm_tc_change(struct Qdisc *sch, u32 classid, u32 parent, if (opt == NULL) return -EINVAL; - error = nla_parse_nested(tb, TCA_ATM_MAX, opt, atm_policy, NULL); + error = nla_parse_nested_deprecated(tb, TCA_ATM_MAX, opt, atm_policy, + NULL); if (error < 0) return error; @@ -609,7 +610,7 @@ static int atm_tc_dump_class(struct Qdisc *sch, unsigned long cl, tcm->tcm_handle = flow->common.classid; tcm->tcm_info = flow->q->handle; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (nest == NULL) goto nla_put_failure; diff --git a/net/sched/sch_cake.c b/net/sched/sch_cake.c index 259d97bc2abd39df8df646c2ebc34ea272e1fd70..53a80bc6b13a10cd0f3826b5c87a4e8baf5ccfd5 100644 --- a/net/sched/sch_cake.c +++ b/net/sched/sch_cake.c @@ -2531,7 +2531,8 @@ static int cake_change(struct Qdisc *sch, struct nlattr *opt, if (!opt) return -EINVAL; - err = nla_parse_nested(tb, TCA_CAKE_MAX, opt, cake_policy, extack); + err = nla_parse_nested_deprecated(tb, TCA_CAKE_MAX, opt, cake_policy, + extack); if (err < 0) return err; @@ -2735,7 +2736,7 @@ static int cake_dump(struct Qdisc *sch, struct sk_buff *skb) struct cake_sched_data *q = qdisc_priv(sch); struct nlattr *opts; - opts = nla_nest_start(skb, TCA_OPTIONS); + opts = nla_nest_start_noflag(skb, TCA_OPTIONS); if (!opts) goto nla_put_failure; @@ -2806,7 +2807,7 @@ nla_put_failure: static int cake_dump_stats(struct Qdisc *sch, struct gnet_dump *d) { - struct nlattr *stats = nla_nest_start(d->skb, TCA_STATS_APP); + struct nlattr *stats = nla_nest_start_noflag(d->skb, TCA_STATS_APP); struct cake_sched_data *q = qdisc_priv(sch); struct nlattr *tstats, *ts; int i; @@ -2836,7 +2837,7 @@ static int cake_dump_stats(struct Qdisc *sch, struct gnet_dump *d) #undef PUT_STAT_U32 #undef PUT_STAT_U64 - tstats = nla_nest_start(d->skb, TCA_CAKE_STATS_TIN_STATS); + tstats = nla_nest_start_noflag(d->skb, TCA_CAKE_STATS_TIN_STATS); if (!tstats) goto nla_put_failure; @@ -2853,7 +2854,7 @@ static int cake_dump_stats(struct Qdisc *sch, struct gnet_dump *d) for (i = 0; i < q->tin_cnt; i++) { struct cake_tin_data *b = &q->tins[q->tin_order[i]]; - ts = nla_nest_start(d->skb, i + 1); + ts = nla_nest_start_noflag(d->skb, i + 1); if (!ts) goto nla_put_failure; @@ -2973,7 +2974,7 @@ static int cake_dump_class_stats(struct Qdisc *sch, unsigned long cl, if (flow) { ktime_t now = ktime_get(); - stats = nla_nest_start(d->skb, TCA_STATS_APP); + stats = nla_nest_start_noflag(d->skb, TCA_STATS_APP); if (!stats) return -1; diff --git a/net/sched/sch_cbq.c b/net/sched/sch_cbq.c index 114b9048ea7e3682106c6e65644d4d0992e20461..ba4b33b74dd81267de25859e2dd94731f9097507 100644 --- a/net/sched/sch_cbq.c +++ b/net/sched/sch_cbq.c @@ -1149,7 +1149,8 @@ static int cbq_init(struct Qdisc *sch, struct nlattr *opt, return -EINVAL; } - err = nla_parse_nested(tb, TCA_CBQ_MAX, opt, cbq_policy, extack); + err = nla_parse_nested_deprecated(tb, TCA_CBQ_MAX, opt, cbq_policy, + extack); if (err < 0) return err; @@ -1305,7 +1306,7 @@ static int cbq_dump(struct Qdisc *sch, struct sk_buff *skb) struct cbq_sched_data *q = qdisc_priv(sch); struct nlattr *nest; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (nest == NULL) goto nla_put_failure; if (cbq_dump_attr(skb, &q->link) < 0) @@ -1340,7 +1341,7 @@ cbq_dump_class(struct Qdisc *sch, unsigned long arg, tcm->tcm_handle = cl->common.classid; tcm->tcm_info = cl->q->handle; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (nest == NULL) goto nla_put_failure; if (cbq_dump_attr(skb, cl) < 0) @@ -1473,7 +1474,8 @@ cbq_change_class(struct Qdisc *sch, u32 classid, u32 parentid, struct nlattr **t return -EINVAL; } - err = nla_parse_nested(tb, TCA_CBQ_MAX, opt, cbq_policy, extack); + err = nla_parse_nested_deprecated(tb, TCA_CBQ_MAX, opt, cbq_policy, + extack); if (err < 0) return err; diff --git a/net/sched/sch_cbs.c b/net/sched/sch_cbs.c index c6a502933fe78b250b971416caeffd796d3f31d3..8077c846f5bffccac74178fdf796ecb58ebc5890 100644 --- a/net/sched/sch_cbs.c +++ b/net/sched/sch_cbs.c @@ -61,16 +61,20 @@ #include #include #include +#include #include #include #include +static LIST_HEAD(cbs_list); +static DEFINE_SPINLOCK(cbs_list_lock); + #define BYTES_PER_KBIT (1000LL / 8) struct cbs_sched_data { bool offload; int queue; - s64 port_rate; /* in bytes/s */ + atomic64_t port_rate; /* in bytes/s */ s64 last; /* timestamp in ns */ s64 credits; /* in bytes */ s32 locredit; /* in bytes */ @@ -82,6 +86,7 @@ struct cbs_sched_data { struct sk_buff **to_free); struct sk_buff *(*dequeue)(struct Qdisc *sch); struct Qdisc *qdisc; + struct list_head cbs_list; }; static int cbs_child_enqueue(struct sk_buff *skb, struct Qdisc *sch, @@ -181,6 +186,11 @@ static struct sk_buff *cbs_dequeue_soft(struct Qdisc *sch) s64 credits; int len; + if (atomic64_read(&q->port_rate) == -1) { + WARN_ONCE(1, "cbs: dequeue() called with unknown port rate."); + return NULL; + } + if (q->credits < 0) { credits = timediff_to_credits(now - q->last, q->idleslope); @@ -207,7 +217,8 @@ static struct sk_buff *cbs_dequeue_soft(struct Qdisc *sch) /* As sendslope is a negative number, this will decrease the * amount of q->credits. */ - credits = credits_from_len(len, q->sendslope, q->port_rate); + credits = credits_from_len(len, q->sendslope, + atomic64_read(&q->port_rate)); credits += q->credits; q->credits = max_t(s64, credits, q->locredit); @@ -294,6 +305,50 @@ static int cbs_enable_offload(struct net_device *dev, struct cbs_sched_data *q, return 0; } +static void cbs_set_port_rate(struct net_device *dev, struct cbs_sched_data *q) +{ + struct ethtool_link_ksettings ecmd; + int port_rate = -1; + + if (!__ethtool_get_link_ksettings(dev, &ecmd) && + ecmd.base.speed != SPEED_UNKNOWN) + port_rate = ecmd.base.speed * 1000 * BYTES_PER_KBIT; + + atomic64_set(&q->port_rate, port_rate); + netdev_dbg(dev, "cbs: set %s's port_rate to: %lld, linkspeed: %d\n", + dev->name, (long long)atomic64_read(&q->port_rate), + ecmd.base.speed); +} + +static int cbs_dev_notifier(struct notifier_block *nb, unsigned long event, + void *ptr) +{ + struct net_device *dev = netdev_notifier_info_to_dev(ptr); + struct cbs_sched_data *q; + struct net_device *qdev; + bool found = false; + + ASSERT_RTNL(); + + if (event != NETDEV_UP && event != NETDEV_CHANGE) + return NOTIFY_DONE; + + spin_lock(&cbs_list_lock); + list_for_each_entry(q, &cbs_list, cbs_list) { + qdev = qdisc_dev(q->qdisc); + if (qdev == dev) { + found = true; + break; + } + } + spin_unlock(&cbs_list_lock); + + if (found) + cbs_set_port_rate(dev, q); + + return NOTIFY_DONE; +} + static int cbs_change(struct Qdisc *sch, struct nlattr *opt, struct netlink_ext_ack *extack) { @@ -303,7 +358,8 @@ static int cbs_change(struct Qdisc *sch, struct nlattr *opt, struct tc_cbs_qopt *qopt; int err; - err = nla_parse_nested(tb, TCA_CBS_MAX, opt, cbs_policy, extack); + err = nla_parse_nested_deprecated(tb, TCA_CBS_MAX, opt, cbs_policy, + extack); if (err < 0) return err; @@ -315,16 +371,7 @@ static int cbs_change(struct Qdisc *sch, struct nlattr *opt, qopt = nla_data(tb[TCA_CBS_PARMS]); if (!qopt->offload) { - struct ethtool_link_ksettings ecmd; - s64 link_speed; - - if (!__ethtool_get_link_ksettings(dev, &ecmd)) - link_speed = ecmd.base.speed; - else - link_speed = SPEED_1000; - - q->port_rate = link_speed * 1000 * BYTES_PER_KBIT; - + cbs_set_port_rate(dev, q); cbs_disable_offload(dev, q); } else { err = cbs_enable_offload(dev, q, qopt, extack); @@ -347,6 +394,7 @@ static int cbs_init(struct Qdisc *sch, struct nlattr *opt, { struct cbs_sched_data *q = qdisc_priv(sch); struct net_device *dev = qdisc_dev(sch); + int err; if (!opt) { NL_SET_ERR_MSG(extack, "Missing CBS qdisc options which are mandatory"); @@ -367,7 +415,17 @@ static int cbs_init(struct Qdisc *sch, struct nlattr *opt, qdisc_watchdog_init(&q->watchdog, sch); - return cbs_change(sch, opt, extack); + err = cbs_change(sch, opt, extack); + if (err) + return err; + + if (!q->offload) { + spin_lock(&cbs_list_lock); + list_add(&q->cbs_list, &cbs_list); + spin_unlock(&cbs_list_lock); + } + + return 0; } static void cbs_destroy(struct Qdisc *sch) @@ -375,8 +433,11 @@ static void cbs_destroy(struct Qdisc *sch) struct cbs_sched_data *q = qdisc_priv(sch); struct net_device *dev = qdisc_dev(sch); - qdisc_watchdog_cancel(&q->watchdog); + spin_lock(&cbs_list_lock); + list_del(&q->cbs_list); + spin_unlock(&cbs_list_lock); + qdisc_watchdog_cancel(&q->watchdog); cbs_disable_offload(dev, q); if (q->qdisc) @@ -389,7 +450,7 @@ static int cbs_dump(struct Qdisc *sch, struct sk_buff *skb) struct tc_cbs_qopt opt = { }; struct nlattr *nest; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (!nest) goto nla_put_failure; @@ -487,14 +548,24 @@ static struct Qdisc_ops cbs_qdisc_ops __read_mostly = { .owner = THIS_MODULE, }; +static struct notifier_block cbs_device_notifier = { + .notifier_call = cbs_dev_notifier, +}; + static int __init cbs_module_init(void) { + int err = register_netdevice_notifier(&cbs_device_notifier); + + if (err) + return err; + return register_qdisc(&cbs_qdisc_ops); } static void __exit cbs_module_exit(void) { unregister_qdisc(&cbs_qdisc_ops); + unregister_netdevice_notifier(&cbs_device_notifier); } module_init(cbs_module_init) module_exit(cbs_module_exit) diff --git a/net/sched/sch_choke.c b/net/sched/sch_choke.c index eafc0d17d1748cf01f33d206cb46dde6ed81116c..370dbcf49e8b8ab15653d753fcafa5019a62e22a 100644 --- a/net/sched/sch_choke.c +++ b/net/sched/sch_choke.c @@ -358,7 +358,8 @@ static int choke_change(struct Qdisc *sch, struct nlattr *opt, if (opt == NULL) return -EINVAL; - err = nla_parse_nested(tb, TCA_CHOKE_MAX, opt, choke_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_CHOKE_MAX, opt, + choke_policy, NULL); if (err < 0) return err; @@ -452,7 +453,7 @@ static int choke_dump(struct Qdisc *sch, struct sk_buff *skb) .Scell_log = q->parms.Scell_log, }; - opts = nla_nest_start(skb, TCA_OPTIONS); + opts = nla_nest_start_noflag(skb, TCA_OPTIONS); if (opts == NULL) goto nla_put_failure; diff --git a/net/sched/sch_codel.c b/net/sched/sch_codel.c index 17cd81f84b5de9d9f53acf2c9078a7618927ddc5..25ef172c23dfa31d05729c76a58ade3bb4967d5e 100644 --- a/net/sched/sch_codel.c +++ b/net/sched/sch_codel.c @@ -141,7 +141,8 @@ static int codel_change(struct Qdisc *sch, struct nlattr *opt, if (!opt) return -EINVAL; - err = nla_parse_nested(tb, TCA_CODEL_MAX, opt, codel_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_CODEL_MAX, opt, + codel_policy, NULL); if (err < 0) return err; @@ -217,7 +218,7 @@ static int codel_dump(struct Qdisc *sch, struct sk_buff *skb) struct codel_sched_data *q = qdisc_priv(sch); struct nlattr *opts; - opts = nla_nest_start(skb, TCA_OPTIONS); + opts = nla_nest_start_noflag(skb, TCA_OPTIONS); if (opts == NULL) goto nla_put_failure; diff --git a/net/sched/sch_drr.c b/net/sched/sch_drr.c index 430df9a55ec4e9742786fb869ab3acf28e84f5ed..ffcd6654c39dcb556fe9157b7706a4298bf18862 100644 --- a/net/sched/sch_drr.c +++ b/net/sched/sch_drr.c @@ -70,7 +70,8 @@ static int drr_change_class(struct Qdisc *sch, u32 classid, u32 parentid, return -EINVAL; } - err = nla_parse_nested(tb, TCA_DRR_MAX, opt, drr_policy, extack); + err = nla_parse_nested_deprecated(tb, TCA_DRR_MAX, opt, drr_policy, + extack); if (err < 0) return err; @@ -244,7 +245,7 @@ static int drr_dump_class(struct Qdisc *sch, unsigned long arg, tcm->tcm_handle = cl->common.classid; tcm->tcm_info = cl->qdisc->handle; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (nest == NULL) goto nla_put_failure; if (nla_put_u32(skb, TCA_DRR_QUANTUM, cl->quantum)) diff --git a/net/sched/sch_dsmark.c b/net/sched/sch_dsmark.c index 42471464ded3fdf28b712e2d16b7cb4f34939cc0..3deeb06eaecfa64bdcb11af73c6b7136c51f0350 100644 --- a/net/sched/sch_dsmark.c +++ b/net/sched/sch_dsmark.c @@ -132,7 +132,8 @@ static int dsmark_change(struct Qdisc *sch, u32 classid, u32 parent, if (!opt) goto errout; - err = nla_parse_nested(tb, TCA_DSMARK_MAX, opt, dsmark_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_DSMARK_MAX, opt, + dsmark_policy, NULL); if (err < 0) goto errout; @@ -353,7 +354,8 @@ static int dsmark_init(struct Qdisc *sch, struct nlattr *opt, if (err) return err; - err = nla_parse_nested(tb, TCA_DSMARK_MAX, opt, dsmark_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_DSMARK_MAX, opt, + dsmark_policy, NULL); if (err < 0) goto errout; @@ -432,7 +434,7 @@ static int dsmark_dump_class(struct Qdisc *sch, unsigned long cl, tcm->tcm_handle = TC_H_MAKE(TC_H_MAJ(sch->handle), cl - 1); tcm->tcm_info = p->q->handle; - opts = nla_nest_start(skb, TCA_OPTIONS); + opts = nla_nest_start_noflag(skb, TCA_OPTIONS); if (opts == NULL) goto nla_put_failure; if (nla_put_u8(skb, TCA_DSMARK_MASK, p->mv[cl - 1].mask) || @@ -451,7 +453,7 @@ static int dsmark_dump(struct Qdisc *sch, struct sk_buff *skb) struct dsmark_qdisc_data *p = qdisc_priv(sch); struct nlattr *opts = NULL; - opts = nla_nest_start(skb, TCA_OPTIONS); + opts = nla_nest_start_noflag(skb, TCA_OPTIONS); if (opts == NULL) goto nla_put_failure; if (nla_put_u16(skb, TCA_DSMARK_INDICES, p->indices)) diff --git a/net/sched/sch_etf.c b/net/sched/sch_etf.c index 1150f22983dfe6e5e3b8aaf6619b22702341b6cd..db0c2ba1d1562854826582cca0bafbab24cae243 100644 --- a/net/sched/sch_etf.c +++ b/net/sched/sch_etf.c @@ -351,7 +351,8 @@ static int etf_init(struct Qdisc *sch, struct nlattr *opt, return -EINVAL; } - err = nla_parse_nested(tb, TCA_ETF_MAX, opt, etf_policy, extack); + err = nla_parse_nested_deprecated(tb, TCA_ETF_MAX, opt, etf_policy, + extack); if (err < 0) return err; @@ -460,7 +461,7 @@ static int etf_dump(struct Qdisc *sch, struct sk_buff *skb) struct tc_etf_qopt opt = { }; struct nlattr *nest; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (!nest) goto nla_put_failure; diff --git a/net/sched/sch_fq.c b/net/sched/sch_fq.c index 1a662f2bb7bb7bb5507107f61657d44fa28ca991..26a94e5cd5dfae34109649b04a1ebcaafa0f545b 100644 --- a/net/sched/sch_fq.c +++ b/net/sched/sch_fq.c @@ -54,10 +54,23 @@ #include #include +struct fq_skb_cb { + u64 time_to_send; +}; + +static inline struct fq_skb_cb *fq_skb_cb(struct sk_buff *skb) +{ + qdisc_cb_private_validate(skb, sizeof(struct fq_skb_cb)); + return (struct fq_skb_cb *)qdisc_skb_cb(skb)->data; +} + /* - * Per flow structure, dynamically allocated + * Per flow structure, dynamically allocated. + * If packets have monotically increasing time_to_send, they are placed in O(1) + * in linear list (head,tail), otherwise are placed in a rbtree (t_root). */ struct fq_flow { + struct rb_root t_root; struct sk_buff *head; /* list of skbs for this flow : first skb */ union { struct sk_buff *tail; /* last skb in the list */ @@ -257,6 +270,17 @@ static struct fq_flow *fq_classify(struct sk_buff *skb, struct fq_sched_data *q) */ sk = (struct sock *)((hash << 1) | 1UL); skb_orphan(skb); + } else if (sk->sk_state == TCP_CLOSE) { + unsigned long hash = skb_get_hash(skb) & q->orphan_mask; + /* + * Sockets in TCP_CLOSE are non connected. + * Typical use case is UDP sockets, they can send packets + * with sendto() to many different destinations. + * We probably could use a generic bit advertising + * non connected sockets, instead of sk_state == TCP_CLOSE, + * if we care enough. + */ + sk = (struct sock *)((hash << 1) | 1UL); } root = &q->fq_root[hash_ptr(sk, q->fq_trees_log)]; @@ -277,7 +301,7 @@ static struct fq_flow *fq_classify(struct sk_buff *skb, struct fq_sched_data *q) * It not, we need to refill credit with * initial quantum */ - if (unlikely(skb->sk && + if (unlikely(skb->sk == sk && f->socket_hash != sk->sk_hash)) { f->credit = q->initial_quantum; f->socket_hash = sk->sk_hash; @@ -298,9 +322,11 @@ static struct fq_flow *fq_classify(struct sk_buff *skb, struct fq_sched_data *q) q->stat_allocation_errors++; return &q->internal; } + /* f->t_root is already zeroed after kmem_cache_zalloc() */ + fq_flow_set_detached(f); f->sk = sk; - if (skb->sk) + if (skb->sk == sk) f->socket_hash = sk->sk_hash; f->credit = q->initial_quantum; @@ -312,14 +338,40 @@ static struct fq_flow *fq_classify(struct sk_buff *skb, struct fq_sched_data *q) return f; } +static struct sk_buff *fq_peek(struct fq_flow *flow) +{ + struct sk_buff *skb = skb_rb_first(&flow->t_root); + struct sk_buff *head = flow->head; + + if (!skb) + return head; + + if (!head) + return skb; + + if (fq_skb_cb(skb)->time_to_send < fq_skb_cb(head)->time_to_send) + return skb; + return head; +} + +static void fq_erase_head(struct Qdisc *sch, struct fq_flow *flow, + struct sk_buff *skb) +{ + if (skb == flow->head) { + flow->head = skb->next; + } else { + rb_erase(&skb->rbnode, &flow->t_root); + skb->dev = qdisc_dev(sch); + } +} /* remove one skb from head of flow queue */ static struct sk_buff *fq_dequeue_head(struct Qdisc *sch, struct fq_flow *flow) { - struct sk_buff *skb = flow->head; + struct sk_buff *skb = fq_peek(flow); if (skb) { - flow->head = skb->next; + fq_erase_head(sch, flow, skb); skb_mark_not_on_list(skb); flow->qlen--; qdisc_qstats_backlog_dec(sch, skb); @@ -330,15 +382,36 @@ static struct sk_buff *fq_dequeue_head(struct Qdisc *sch, struct fq_flow *flow) static void flow_queue_add(struct fq_flow *flow, struct sk_buff *skb) { - struct sk_buff *head = flow->head; + struct rb_node **p, *parent; + struct sk_buff *head, *aux; - skb->next = NULL; - if (!head) - flow->head = skb; - else - flow->tail->next = skb; + fq_skb_cb(skb)->time_to_send = skb->tstamp ?: ktime_get_ns(); + + head = flow->head; + if (!head || + fq_skb_cb(skb)->time_to_send >= fq_skb_cb(flow->tail)->time_to_send) { + if (!head) + flow->head = skb; + else + flow->tail->next = skb; + flow->tail = skb; + skb->next = NULL; + return; + } + + p = &flow->t_root.rb_node; + parent = NULL; - flow->tail = skb; + while (*p) { + parent = *p; + aux = rb_to_skb(parent); + if (fq_skb_cb(skb)->time_to_send >= fq_skb_cb(aux)->time_to_send) + p = &parent->rb_right; + else + p = &parent->rb_left; + } + rb_link_node(&skb->rbnode, parent, p); + rb_insert_color(&skb->rbnode, &flow->t_root); } static int fq_enqueue(struct sk_buff *skb, struct Qdisc *sch, @@ -450,9 +523,9 @@ begin: goto begin; } - skb = f->head; + skb = fq_peek(f); if (skb) { - u64 time_next_packet = max_t(u64, ktime_to_ns(skb->tstamp), + u64 time_next_packet = max_t(u64, fq_skb_cb(skb)->time_to_send, f->time_next_packet); if (now < time_next_packet) { @@ -533,6 +606,15 @@ out: static void fq_flow_purge(struct fq_flow *flow) { + struct rb_node *p = rb_first(&flow->t_root); + + while (p) { + struct sk_buff *skb = rb_to_skb(p); + + p = rb_next(p); + rb_erase(&skb->rbnode, &flow->t_root); + rtnl_kfree_skbs(skb, skb); + } rtnl_kfree_skbs(flow->head, flow->tail); flow->head = NULL; flow->qlen = 0; @@ -684,7 +766,8 @@ static int fq_change(struct Qdisc *sch, struct nlattr *opt, if (!opt) return -EINVAL; - err = nla_parse_nested(tb, TCA_FQ_MAX, opt, fq_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_FQ_MAX, opt, fq_policy, + NULL); if (err < 0) return err; @@ -823,7 +906,7 @@ static int fq_dump(struct Qdisc *sch, struct sk_buff *skb) u64 ce_threshold = q->ce_threshold; struct nlattr *opts; - opts = nla_nest_start(skb, TCA_OPTIONS); + opts = nla_nest_start_noflag(skb, TCA_OPTIONS); if (opts == NULL) goto nla_put_failure; diff --git a/net/sched/sch_fq_codel.c b/net/sched/sch_fq_codel.c index cd04d40c30b672ef755b66a50ff42bf8240367db..08d85370b97c1e3e5e13504fe1b1a757bc082a5c 100644 --- a/net/sched/sch_fq_codel.c +++ b/net/sched/sch_fq_codel.c @@ -387,8 +387,8 @@ static int fq_codel_change(struct Qdisc *sch, struct nlattr *opt, if (!opt) return -EINVAL; - err = nla_parse_nested(tb, TCA_FQ_CODEL_MAX, opt, fq_codel_policy, - NULL); + err = nla_parse_nested_deprecated(tb, TCA_FQ_CODEL_MAX, opt, + fq_codel_policy, NULL); if (err < 0) return err; if (tb[TCA_FQ_CODEL_FLOWS]) { @@ -527,7 +527,7 @@ static int fq_codel_dump(struct Qdisc *sch, struct sk_buff *skb) struct fq_codel_sched_data *q = qdisc_priv(sch); struct nlattr *opts; - opts = nla_nest_start(skb, TCA_OPTIONS); + opts = nla_nest_start_noflag(skb, TCA_OPTIONS); if (opts == NULL) goto nla_put_failure; diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c index a117d92605580c1d325eb171b1f3e72ba4321f52..cce1e9ee85af04109068f1e3025f587674864ffa 100644 --- a/net/sched/sch_generic.c +++ b/net/sched/sch_generic.c @@ -32,6 +32,7 @@ #include #include #include +#include #include /* Qdisc to use by default */ @@ -68,7 +69,7 @@ static inline struct sk_buff *__skb_dequeue_bad_txq(struct Qdisc *q) skb = __skb_dequeue(&q->skb_bad_txq); if (qdisc_is_percpu_stats(q)) { qdisc_qstats_cpu_backlog_dec(q, skb); - qdisc_qstats_atomic_qlen_dec(q); + qdisc_qstats_cpu_qlen_dec(q); } else { qdisc_qstats_backlog_dec(q, skb); q->q.qlen--; @@ -108,7 +109,7 @@ static inline void qdisc_enqueue_skb_bad_txq(struct Qdisc *q, if (qdisc_is_percpu_stats(q)) { qdisc_qstats_cpu_backlog_inc(q, skb); - qdisc_qstats_atomic_qlen_inc(q); + qdisc_qstats_cpu_qlen_inc(q); } else { qdisc_qstats_backlog_inc(q, skb); q->q.qlen++; @@ -118,52 +119,36 @@ static inline void qdisc_enqueue_skb_bad_txq(struct Qdisc *q, spin_unlock(lock); } -static inline int __dev_requeue_skb(struct sk_buff *skb, struct Qdisc *q) +static inline void dev_requeue_skb(struct sk_buff *skb, struct Qdisc *q) { - while (skb) { - struct sk_buff *next = skb->next; - - __skb_queue_tail(&q->gso_skb, skb); - q->qstats.requeues++; - qdisc_qstats_backlog_inc(q, skb); - q->q.qlen++; /* it's still part of the queue */ + spinlock_t *lock = NULL; - skb = next; + if (q->flags & TCQ_F_NOLOCK) { + lock = qdisc_lock(q); + spin_lock(lock); } - __netif_schedule(q); - return 0; -} - -static inline int dev_requeue_skb_locked(struct sk_buff *skb, struct Qdisc *q) -{ - spinlock_t *lock = qdisc_lock(q); - - spin_lock(lock); while (skb) { struct sk_buff *next = skb->next; __skb_queue_tail(&q->gso_skb, skb); - qdisc_qstats_cpu_requeues_inc(q); - qdisc_qstats_cpu_backlog_inc(q, skb); - qdisc_qstats_atomic_qlen_inc(q); + /* it's still part of the queue */ + if (qdisc_is_percpu_stats(q)) { + qdisc_qstats_cpu_requeues_inc(q); + qdisc_qstats_cpu_backlog_inc(q, skb); + qdisc_qstats_cpu_qlen_inc(q); + } else { + q->qstats.requeues++; + qdisc_qstats_backlog_inc(q, skb); + q->q.qlen++; + } skb = next; } - spin_unlock(lock); - + if (lock) + spin_unlock(lock); __netif_schedule(q); - - return 0; -} - -static inline int dev_requeue_skb(struct sk_buff *skb, struct Qdisc *q) -{ - if (q->flags & TCQ_F_NOLOCK) - return dev_requeue_skb_locked(skb, q); - else - return __dev_requeue_skb(skb, q); } static void try_bulk_dequeue_skb(struct Qdisc *q, @@ -252,7 +237,7 @@ static struct sk_buff *dequeue_skb(struct Qdisc *q, bool *validate, skb = __skb_dequeue(&q->gso_skb); if (qdisc_is_percpu_stats(q)) { qdisc_qstats_cpu_backlog_dec(q, skb); - qdisc_qstats_atomic_qlen_dec(q); + qdisc_qstats_cpu_qlen_dec(q); } else { qdisc_qstats_backlog_dec(q, skb); q->q.qlen--; @@ -457,6 +442,7 @@ static void dev_watchdog(struct timer_list *t) } if (some_queue_timedout) { + trace_net_dev_xmit_timeout(dev, i); WARN_ONCE(1, KERN_INFO "NETDEV WATCHDOG: %s (%s): transmit queue %u timed out\n", dev->name, netdev_drivername(dev), i); dev->netdev_ops->ndo_tx_timeout(dev); @@ -645,11 +631,7 @@ static int pfifo_fast_enqueue(struct sk_buff *skb, struct Qdisc *qdisc, if (unlikely(err)) return qdisc_drop_cpu(skb, qdisc, to_free); - qdisc_qstats_atomic_qlen_inc(qdisc); - /* Note: skb can not be used after skb_array_produce(), - * so we better not use qdisc_qstats_cpu_backlog_inc() - */ - this_cpu_add(qdisc->cpu_qstats->backlog, pkt_len); + qdisc_update_stats_at_enqueue(qdisc, pkt_len); return NET_XMIT_SUCCESS; } @@ -668,9 +650,9 @@ static struct sk_buff *pfifo_fast_dequeue(struct Qdisc *qdisc) skb = __skb_array_consume(q); } if (likely(skb)) { - qdisc_qstats_cpu_backlog_dec(qdisc, skb); - qdisc_bstats_cpu_update(qdisc, skb); - qdisc_qstats_atomic_qlen_dec(qdisc); + qdisc_update_stats_at_dequeue(qdisc, skb); + } else { + qdisc->empty = true; } return skb; @@ -714,6 +696,7 @@ static void pfifo_fast_reset(struct Qdisc *qdisc) struct gnet_stats_queue *q = per_cpu_ptr(qdisc->cpu_qstats, i); q->backlog = 0; + q->qlen = 0; } } @@ -880,6 +863,7 @@ struct Qdisc *qdisc_alloc(struct netdev_queue *dev_queue, sch->enqueue = ops->enqueue; sch->dequeue = ops->dequeue; sch->dev_queue = dev_queue; + sch->empty = true; dev_hold(dev); refcount_set(&sch->refcnt, 1); diff --git a/net/sched/sch_gred.c b/net/sched/sch_gred.c index 234afbf9115b7fbb7713fe12f74ea099fb69e8e2..dfa657da100f31d1171af406c4f8745933a00955 100644 --- a/net/sched/sch_gred.c +++ b/net/sched/sch_gred.c @@ -538,7 +538,8 @@ static void gred_vq_apply(struct gred_sched *table, const struct nlattr *entry) struct nlattr *tb[TCA_GRED_VQ_MAX + 1]; u32 dp; - nla_parse_nested(tb, TCA_GRED_VQ_MAX, entry, gred_vq_policy, NULL); + nla_parse_nested_deprecated(tb, TCA_GRED_VQ_MAX, entry, + gred_vq_policy, NULL); dp = nla_get_u32(tb[TCA_GRED_VQ_DP]); @@ -568,8 +569,8 @@ static int gred_vq_validate(struct gred_sched *table, u32 cdp, int err; u32 dp; - err = nla_parse_nested(tb, TCA_GRED_VQ_MAX, entry, gred_vq_policy, - extack); + err = nla_parse_nested_deprecated(tb, TCA_GRED_VQ_MAX, entry, + gred_vq_policy, extack); if (err < 0) return err; @@ -610,8 +611,8 @@ static int gred_vqs_validate(struct gred_sched *table, u32 cdp, const struct nlattr *attr; int rem, err; - err = nla_validate_nested(vqs, TCA_GRED_VQ_ENTRY_MAX, - gred_vqe_policy, extack); + err = nla_validate_nested_deprecated(vqs, TCA_GRED_VQ_ENTRY_MAX, + gred_vqe_policy, extack); if (err < 0) return err; @@ -650,7 +651,8 @@ static int gred_change(struct Qdisc *sch, struct nlattr *opt, if (opt == NULL) return -EINVAL; - err = nla_parse_nested(tb, TCA_GRED_MAX, opt, gred_policy, extack); + err = nla_parse_nested_deprecated(tb, TCA_GRED_MAX, opt, gred_policy, + extack); if (err < 0) return err; @@ -737,7 +739,8 @@ static int gred_init(struct Qdisc *sch, struct nlattr *opt, if (!opt) return -EINVAL; - err = nla_parse_nested(tb, TCA_GRED_MAX, opt, gred_policy, extack); + err = nla_parse_nested_deprecated(tb, TCA_GRED_MAX, opt, gred_policy, + extack); if (err < 0) return err; @@ -772,7 +775,7 @@ static int gred_dump(struct Qdisc *sch, struct sk_buff *skb) if (gred_offload_dump_stats(sch)) goto nla_put_failure; - opts = nla_nest_start(skb, TCA_OPTIONS); + opts = nla_nest_start_noflag(skb, TCA_OPTIONS); if (opts == NULL) goto nla_put_failure; if (nla_put(skb, TCA_GRED_DPS, sizeof(sopt), &sopt)) @@ -790,7 +793,7 @@ static int gred_dump(struct Qdisc *sch, struct sk_buff *skb) goto nla_put_failure; /* Old style all-in-one dump of VQs */ - parms = nla_nest_start(skb, TCA_GRED_PARMS); + parms = nla_nest_start_noflag(skb, TCA_GRED_PARMS); if (parms == NULL) goto nla_put_failure; @@ -841,7 +844,7 @@ append_opt: nla_nest_end(skb, parms); /* Dump the VQs again, in more structured way */ - vqs = nla_nest_start(skb, TCA_GRED_VQ_LIST); + vqs = nla_nest_start_noflag(skb, TCA_GRED_VQ_LIST); if (!vqs) goto nla_put_failure; @@ -852,7 +855,7 @@ append_opt: if (!q) continue; - vq = nla_nest_start(skb, TCA_GRED_VQ_ENTRY); + vq = nla_nest_start_noflag(skb, TCA_GRED_VQ_ENTRY); if (!vq) goto nla_put_failure; diff --git a/net/sched/sch_hfsc.c b/net/sched/sch_hfsc.c index d2ab463f22ae8b122ae43d1969cf795fb11c05b3..433f2190960fe88235a5c6381760596cfc93305a 100644 --- a/net/sched/sch_hfsc.c +++ b/net/sched/sch_hfsc.c @@ -926,7 +926,8 @@ hfsc_change_class(struct Qdisc *sch, u32 classid, u32 parentid, if (opt == NULL) return -EINVAL; - err = nla_parse_nested(tb, TCA_HFSC_MAX, opt, hfsc_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_HFSC_MAX, opt, hfsc_policy, + NULL); if (err < 0) return err; @@ -1300,7 +1301,7 @@ hfsc_dump_class(struct Qdisc *sch, unsigned long arg, struct sk_buff *skb, if (cl->level == 0) tcm->tcm_info = cl->qdisc->handle; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (nest == NULL) goto nla_put_failure; if (hfsc_dump_curves(skb, cl) < 0) diff --git a/net/sched/sch_hhf.c b/net/sched/sch_hhf.c index 9d6a47697406091ba6fda460889f01923f658c77..a28e09b1609ca1f68088d87fd06c895bdeb63d5d 100644 --- a/net/sched/sch_hhf.c +++ b/net/sched/sch_hhf.c @@ -518,7 +518,8 @@ static int hhf_change(struct Qdisc *sch, struct nlattr *opt, if (!opt) return -EINVAL; - err = nla_parse_nested(tb, TCA_HHF_MAX, opt, hhf_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_HHF_MAX, opt, hhf_policy, + NULL); if (err < 0) return err; @@ -654,7 +655,7 @@ static int hhf_dump(struct Qdisc *sch, struct sk_buff *skb) struct hhf_sched_data *q = qdisc_priv(sch); struct nlattr *opts; - opts = nla_nest_start(skb, TCA_OPTIONS); + opts = nla_nest_start_noflag(skb, TCA_OPTIONS); if (opts == NULL) goto nla_put_failure; diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c index 2f9883b196e8e6b10abd9b623b6285274a003ff6..909370049fcafbcd1b83ee9c2529d1be1eaac943 100644 --- a/net/sched/sch_htb.c +++ b/net/sched/sch_htb.c @@ -165,7 +165,8 @@ struct htb_sched { /* non shaped skbs; let them go directly thru */ struct qdisc_skb_head direct_queue; - long direct_pkts; + u32 direct_pkts; + u32 overlimits; struct qdisc_watchdog watchdog; @@ -533,8 +534,10 @@ htb_change_class_mode(struct htb_sched *q, struct htb_class *cl, s64 *diff) if (new_mode == cl->cmode) return; - if (new_mode == HTB_CANT_SEND) + if (new_mode == HTB_CANT_SEND) { cl->overlimits++; + q->overlimits++; + } if (cl->prio_activity) { /* not necessary: speed optimization */ if (cl->cmode != HTB_CANT_SEND) @@ -937,7 +940,6 @@ ok: goto ok; } } - qdisc_qstats_overlimit(sch); if (likely(next_event > q->now)) qdisc_watchdog_schedule_ns(&q->watchdog, next_event); else @@ -1012,7 +1014,8 @@ static int htb_init(struct Qdisc *sch, struct nlattr *opt, if (err) return err; - err = nla_parse_nested(tb, TCA_HTB_MAX, opt, htb_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_HTB_MAX, opt, htb_policy, + NULL); if (err < 0) return err; @@ -1047,6 +1050,7 @@ static int htb_dump(struct Qdisc *sch, struct sk_buff *skb) struct nlattr *nest; struct tc_htb_glob gopt; + sch->qstats.overlimits = q->overlimits; /* Its safe to not acquire qdisc lock. As we hold RTNL, * no change can happen on the qdisc parameters. */ @@ -1057,7 +1061,7 @@ static int htb_dump(struct Qdisc *sch, struct sk_buff *skb) gopt.defcls = q->defcls; gopt.debug = 0; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (nest == NULL) goto nla_put_failure; if (nla_put(skb, TCA_HTB_INIT, sizeof(gopt), &gopt) || @@ -1086,7 +1090,7 @@ static int htb_dump_class(struct Qdisc *sch, unsigned long arg, if (!cl->level && cl->leaf.q) tcm->tcm_info = cl->leaf.q->handle; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (nest == NULL) goto nla_put_failure; @@ -1310,7 +1314,8 @@ static int htb_change_class(struct Qdisc *sch, u32 classid, if (!opt) goto failure; - err = nla_parse_nested(tb, TCA_HTB_MAX, opt, htb_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_HTB_MAX, opt, htb_policy, + NULL); if (err < 0) goto failure; diff --git a/net/sched/sch_ingress.c b/net/sched/sch_ingress.c index ce3f55259d0d1a9024e0d553f0a0b3e101697430..0bac926b46c7643d1af858a50fe54142c1b9f04c 100644 --- a/net/sched/sch_ingress.c +++ b/net/sched/sch_ingress.c @@ -106,7 +106,7 @@ static int ingress_dump(struct Qdisc *sch, struct sk_buff *skb) { struct nlattr *nest; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (nest == NULL) goto nla_put_failure; diff --git a/net/sched/sch_mqprio.c b/net/sched/sch_mqprio.c index ea0dc112b38dd4ac43d0fdc15f4742583c380991..d05086dc38667a3524c9a4d7c3ee9a89a4d00ed1 100644 --- a/net/sched/sch_mqprio.c +++ b/net/sched/sch_mqprio.c @@ -125,8 +125,9 @@ static int parse_attr(struct nlattr *tb[], int maxtype, struct nlattr *nla, int nested_len = nla_len(nla) - NLA_ALIGN(len); if (nested_len >= nla_attr_size(0)) - return nla_parse(tb, maxtype, nla_data(nla) + NLA_ALIGN(len), - nested_len, policy, NULL); + return nla_parse_deprecated(tb, maxtype, + nla_data(nla) + NLA_ALIGN(len), + nested_len, policy, NULL); memset(tb, 0, sizeof(struct nlattr *) * (maxtype + 1)); return 0; @@ -349,7 +350,7 @@ static int dump_rates(struct mqprio_sched *priv, int i; if (priv->flags & TC_MQPRIO_F_MIN_RATE) { - nest = nla_nest_start(skb, TCA_MQPRIO_MIN_RATE64); + nest = nla_nest_start_noflag(skb, TCA_MQPRIO_MIN_RATE64); if (!nest) goto nla_put_failure; @@ -363,7 +364,7 @@ static int dump_rates(struct mqprio_sched *priv, } if (priv->flags & TC_MQPRIO_F_MAX_RATE) { - nest = nla_nest_start(skb, TCA_MQPRIO_MAX_RATE64); + nest = nla_nest_start_noflag(skb, TCA_MQPRIO_MAX_RATE64); if (!nest) goto nla_put_failure; diff --git a/net/sched/sch_netem.c b/net/sched/sch_netem.c index cc9d8133afcdb7fe1b6b4c35f9d96d636ea11035..78aa76b0da2eac4e704e91f06adf71bb80292fb7 100644 --- a/net/sched/sch_netem.c +++ b/net/sched/sch_netem.c @@ -935,8 +935,9 @@ static int parse_attr(struct nlattr *tb[], int maxtype, struct nlattr *nla, } if (nested_len >= nla_attr_size(0)) - return nla_parse(tb, maxtype, nla_data(nla) + NLA_ALIGN(len), - nested_len, policy, NULL); + return nla_parse_deprecated(tb, maxtype, + nla_data(nla) + NLA_ALIGN(len), + nested_len, policy, NULL); memset(tb, 0, sizeof(struct nlattr *) * (maxtype + 1)); return 0; @@ -1079,7 +1080,7 @@ static int dump_loss_model(const struct netem_sched_data *q, { struct nlattr *nest; - nest = nla_nest_start(skb, TCA_NETEM_LOSS); + nest = nla_nest_start_noflag(skb, TCA_NETEM_LOSS); if (nest == NULL) goto nla_put_failure; diff --git a/net/sched/sch_pie.c b/net/sched/sch_pie.c index 1cc0c7b74aa3531fa649df1142f7dbff3034ab23..8fa129d3943e32ade1306ae40990ddc730001d2f 100644 --- a/net/sched/sch_pie.c +++ b/net/sched/sch_pie.c @@ -216,7 +216,8 @@ static int pie_change(struct Qdisc *sch, struct nlattr *opt, if (!opt) return -EINVAL; - err = nla_parse_nested(tb, TCA_PIE_MAX, opt, pie_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_PIE_MAX, opt, pie_policy, + NULL); if (err < 0) return err; @@ -491,7 +492,7 @@ static int pie_dump(struct Qdisc *sch, struct sk_buff *skb) struct pie_sched_data *q = qdisc_priv(sch); struct nlattr *opts; - opts = nla_nest_start(skb, TCA_OPTIONS); + opts = nla_nest_start_noflag(skb, TCA_OPTIONS); if (!opts) goto nla_put_failure; diff --git a/net/sched/sch_qfq.c b/net/sched/sch_qfq.c index 1589364b54da11dc241212dee190dad741d9d9bc..3f9e8b425ac6e4dba4ebd45e7243adb0911f4bf9 100644 --- a/net/sched/sch_qfq.c +++ b/net/sched/sch_qfq.c @@ -410,8 +410,8 @@ static int qfq_change_class(struct Qdisc *sch, u32 classid, u32 parentid, return -EINVAL; } - err = nla_parse_nested(tb, TCA_QFQ_MAX, tca[TCA_OPTIONS], qfq_policy, - NULL); + err = nla_parse_nested_deprecated(tb, TCA_QFQ_MAX, tca[TCA_OPTIONS], + qfq_policy, NULL); if (err < 0) return err; @@ -619,7 +619,7 @@ static int qfq_dump_class(struct Qdisc *sch, unsigned long arg, tcm->tcm_handle = cl->common.classid; tcm->tcm_info = cl->qdisc->handle; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (nest == NULL) goto nla_put_failure; if (nla_put_u32(skb, TCA_QFQ_WEIGHT, cl->agg->class_weight) || diff --git a/net/sched/sch_red.c b/net/sched/sch_red.c index 4e8c0abf619459f396b91fc587271b7938e48330..1e68a13bb66b6e846a79ff1a10e473f31cfd0310 100644 --- a/net/sched/sch_red.c +++ b/net/sched/sch_red.c @@ -205,7 +205,8 @@ static int red_change(struct Qdisc *sch, struct nlattr *opt, if (opt == NULL) return -EINVAL; - err = nla_parse_nested(tb, TCA_RED_MAX, opt, red_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_RED_MAX, opt, red_policy, + NULL); if (err < 0) return err; @@ -318,7 +319,7 @@ static int red_dump(struct Qdisc *sch, struct sk_buff *skb) if (err) goto nla_put_failure; - opts = nla_nest_start(skb, TCA_OPTIONS); + opts = nla_nest_start_noflag(skb, TCA_OPTIONS); if (opts == NULL) goto nla_put_failure; if (nla_put(skb, TCA_RED_PARMS, sizeof(opt), &opt) || diff --git a/net/sched/sch_sfb.c b/net/sched/sch_sfb.c index 2419fdb759667a5c124f2018a310aabe9318b257..b245d6a2068d04b1433a8d6740b4edcf1dd2e872 100644 --- a/net/sched/sch_sfb.c +++ b/net/sched/sch_sfb.c @@ -499,7 +499,8 @@ static int sfb_change(struct Qdisc *sch, struct nlattr *opt, int err; if (opt) { - err = nla_parse_nested(tb, TCA_SFB_MAX, opt, sfb_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_SFB_MAX, opt, + sfb_policy, NULL); if (err < 0) return -EINVAL; @@ -580,7 +581,7 @@ static int sfb_dump(struct Qdisc *sch, struct sk_buff *skb) }; sch->qstats.backlog = q->qdisc->qstats.backlog; - opts = nla_nest_start(skb, TCA_OPTIONS); + opts = nla_nest_start_noflag(skb, TCA_OPTIONS); if (opts == NULL) goto nla_put_failure; if (nla_put(skb, TCA_SFB_PARMS, sizeof(opt), &opt)) diff --git a/net/sched/sch_taprio.c b/net/sched/sch_taprio.c index c7041999eb5d348e7520dab451205fda7a72d22b..9ecfb8f5902a4ff285dbcf14497c293a13a76696 100644 --- a/net/sched/sch_taprio.c +++ b/net/sched/sch_taprio.c @@ -13,13 +13,18 @@ #include #include #include +#include #include #include +#include #include #include #include #include +static LIST_HEAD(taprio_list); +static DEFINE_SPINLOCK(taprio_list_lock); + #define TAPRIO_ALL_GATES_OPEN -1 struct sched_entry { @@ -37,24 +42,88 @@ struct sched_entry { u8 command; }; +struct sched_gate_list { + struct rcu_head rcu; + struct list_head entries; + size_t num_entries; + ktime_t cycle_close_time; + s64 cycle_time; + s64 cycle_time_extension; + s64 base_time; +}; + struct taprio_sched { struct Qdisc **qdiscs; struct Qdisc *root; - s64 base_time; int clockid; - int picos_per_byte; /* Using picoseconds because for 10Gbps+ - * speeds it's sub-nanoseconds per byte - */ - size_t num_entries; + atomic64_t picos_per_byte; /* Using picoseconds because for 10Gbps+ + * speeds it's sub-nanoseconds per byte + */ /* Protects the update side of the RCU protected current_entry */ spinlock_t current_entry_lock; struct sched_entry __rcu *current_entry; - struct list_head entries; + struct sched_gate_list __rcu *oper_sched; + struct sched_gate_list __rcu *admin_sched; ktime_t (*get_time)(void); struct hrtimer advance_timer; + struct list_head taprio_list; }; +static ktime_t sched_base_time(const struct sched_gate_list *sched) +{ + if (!sched) + return KTIME_MAX; + + return ns_to_ktime(sched->base_time); +} + +static void taprio_free_sched_cb(struct rcu_head *head) +{ + struct sched_gate_list *sched = container_of(head, struct sched_gate_list, rcu); + struct sched_entry *entry, *n; + + if (!sched) + return; + + list_for_each_entry_safe(entry, n, &sched->entries, list) { + list_del(&entry->list); + kfree(entry); + } + + kfree(sched); +} + +static void switch_schedules(struct taprio_sched *q, + struct sched_gate_list **admin, + struct sched_gate_list **oper) +{ + rcu_assign_pointer(q->oper_sched, *admin); + rcu_assign_pointer(q->admin_sched, NULL); + + if (*oper) + call_rcu(&(*oper)->rcu, taprio_free_sched_cb); + + *oper = *admin; + *admin = NULL; +} + +static ktime_t get_cycle_time(struct sched_gate_list *sched) +{ + struct sched_entry *entry; + ktime_t cycle = 0; + + if (sched->cycle_time != 0) + return sched->cycle_time; + + list_for_each_entry(entry, &sched->entries, list) + cycle = ktime_add_ns(cycle, entry->interval); + + sched->cycle_time = cycle; + + return cycle; +} + static int taprio_enqueue(struct sk_buff *skb, struct Qdisc *sch, struct sk_buff **to_free) { @@ -85,7 +154,7 @@ static struct sk_buff *taprio_peek(struct Qdisc *sch) rcu_read_lock(); entry = rcu_dereference(q->current_entry); - gate_mask = entry ? entry->gate_mask : -1; + gate_mask = entry ? entry->gate_mask : TAPRIO_ALL_GATES_OPEN; rcu_read_unlock(); if (!gate_mask) @@ -107,7 +176,7 @@ static struct sk_buff *taprio_peek(struct Qdisc *sch) tc = netdev_get_prio_tc_map(dev, prio); if (!(gate_mask & BIT(tc))) - return NULL; + continue; return skb; } @@ -117,18 +186,30 @@ static struct sk_buff *taprio_peek(struct Qdisc *sch) static inline int length_to_duration(struct taprio_sched *q, int len) { - return (len * q->picos_per_byte) / 1000; + return div_u64(len * atomic64_read(&q->picos_per_byte), 1000); +} + +static void taprio_set_budget(struct taprio_sched *q, struct sched_entry *entry) +{ + atomic_set(&entry->budget, + div64_u64((u64)entry->interval * 1000, + atomic64_read(&q->picos_per_byte))); } static struct sk_buff *taprio_dequeue(struct Qdisc *sch) { struct taprio_sched *q = qdisc_priv(sch); struct net_device *dev = qdisc_dev(sch); + struct sk_buff *skb = NULL; struct sched_entry *entry; - struct sk_buff *skb; u32 gate_mask; int i; + if (atomic64_read(&q->picos_per_byte) == -1) { + WARN_ONCE(1, "taprio: dequeue() called with unknown picos per byte."); + return NULL; + } + rcu_read_lock(); entry = rcu_dereference(q->current_entry); /* if there's no entry, it means that the schedule didn't @@ -137,10 +218,9 @@ static struct sk_buff *taprio_dequeue(struct Qdisc *sch) * "AdminGateSates" */ gate_mask = entry ? entry->gate_mask : TAPRIO_ALL_GATES_OPEN; - rcu_read_unlock(); if (!gate_mask) - return NULL; + goto done; for (i = 0; i < dev->num_tx_queues; i++) { struct Qdisc *child = q->qdiscs[i]; @@ -171,39 +251,81 @@ static struct sk_buff *taprio_dequeue(struct Qdisc *sch) */ if (gate_mask != TAPRIO_ALL_GATES_OPEN && ktime_after(guard, entry->close_time)) - return NULL; + continue; /* ... and no budget. */ if (gate_mask != TAPRIO_ALL_GATES_OPEN && atomic_sub_return(len, &entry->budget) < 0) - return NULL; + continue; skb = child->ops->dequeue(child); if (unlikely(!skb)) - return NULL; + goto done; qdisc_bstats_update(sch, skb); qdisc_qstats_backlog_dec(sch, skb); sch->q.qlen--; - return skb; + goto done; } - return NULL; +done: + rcu_read_unlock(); + + return skb; } -static bool should_restart_cycle(const struct taprio_sched *q, +static bool should_restart_cycle(const struct sched_gate_list *oper, const struct sched_entry *entry) { - WARN_ON(!entry); + if (list_is_last(&entry->list, &oper->entries)) + return true; + + if (ktime_compare(entry->close_time, oper->cycle_close_time) == 0) + return true; + + return false; +} + +static bool should_change_schedules(const struct sched_gate_list *admin, + const struct sched_gate_list *oper, + ktime_t close_time) +{ + ktime_t next_base_time, extension_time; + + if (!admin) + return false; - return list_is_last(&entry->list, &q->entries); + next_base_time = sched_base_time(admin); + + /* This is the simple case, the close_time would fall after + * the next schedule base_time. + */ + if (ktime_compare(next_base_time, close_time) <= 0) + return true; + + /* This is the cycle_time_extension case, if the close_time + * plus the amount that can be extended would fall after the + * next schedule base_time, we can extend the current schedule + * for that amount. + */ + extension_time = ktime_add_ns(close_time, oper->cycle_time_extension); + + /* FIXME: the IEEE 802.1Q-2018 Specification isn't clear about + * how precisely the extension should be made. So after + * conformance testing, this logic may change. + */ + if (ktime_compare(next_base_time, extension_time) <= 0) + return true; + + return false; } static enum hrtimer_restart advance_sched(struct hrtimer *timer) { struct taprio_sched *q = container_of(timer, struct taprio_sched, advance_timer); + struct sched_gate_list *oper, *admin; struct sched_entry *entry, *next; struct Qdisc *sch = q->root; ktime_t close_time; @@ -211,29 +333,49 @@ static enum hrtimer_restart advance_sched(struct hrtimer *timer) spin_lock(&q->current_entry_lock); entry = rcu_dereference_protected(q->current_entry, lockdep_is_held(&q->current_entry_lock)); + oper = rcu_dereference_protected(q->oper_sched, + lockdep_is_held(&q->current_entry_lock)); + admin = rcu_dereference_protected(q->admin_sched, + lockdep_is_held(&q->current_entry_lock)); - /* This is the case that it's the first time that the schedule - * runs, so it only happens once per schedule. The first entry - * is pre-calculated during the schedule initialization. + if (!oper) + switch_schedules(q, &admin, &oper); + + /* This can happen in two cases: 1. this is the very first run + * of this function (i.e. we weren't running any schedule + * previously); 2. The previous schedule just ended. The first + * entry of all schedules are pre-calculated during the + * schedule initialization. */ - if (unlikely(!entry)) { - next = list_first_entry(&q->entries, struct sched_entry, + if (unlikely(!entry || entry->close_time == oper->base_time)) { + next = list_first_entry(&oper->entries, struct sched_entry, list); close_time = next->close_time; goto first_run; } - if (should_restart_cycle(q, entry)) - next = list_first_entry(&q->entries, struct sched_entry, + if (should_restart_cycle(oper, entry)) { + next = list_first_entry(&oper->entries, struct sched_entry, list); - else + oper->cycle_close_time = ktime_add_ns(oper->cycle_close_time, + oper->cycle_time); + } else { next = list_next_entry(entry, list); + } close_time = ktime_add_ns(entry->close_time, next->interval); + close_time = min_t(ktime_t, close_time, oper->cycle_close_time); + + if (should_change_schedules(admin, oper, close_time)) { + /* Set things so the next time this runs, the new + * schedule runs. + */ + close_time = sched_base_time(admin); + switch_schedules(q, &admin, &oper); + } next->close_time = close_time; - atomic_set(&next->budget, - (next->interval * 1000) / q->picos_per_byte); + taprio_set_budget(q, next); first_run: rcu_assign_pointer(q->current_entry, next); @@ -263,10 +405,12 @@ static const struct nla_policy taprio_policy[TCA_TAPRIO_ATTR_MAX + 1] = { [TCA_TAPRIO_ATTR_PRIOMAP] = { .len = sizeof(struct tc_mqprio_qopt) }, - [TCA_TAPRIO_ATTR_SCHED_ENTRY_LIST] = { .type = NLA_NESTED }, - [TCA_TAPRIO_ATTR_SCHED_BASE_TIME] = { .type = NLA_S64 }, - [TCA_TAPRIO_ATTR_SCHED_SINGLE_ENTRY] = { .type = NLA_NESTED }, - [TCA_TAPRIO_ATTR_SCHED_CLOCKID] = { .type = NLA_S32 }, + [TCA_TAPRIO_ATTR_SCHED_ENTRY_LIST] = { .type = NLA_NESTED }, + [TCA_TAPRIO_ATTR_SCHED_BASE_TIME] = { .type = NLA_S64 }, + [TCA_TAPRIO_ATTR_SCHED_SINGLE_ENTRY] = { .type = NLA_NESTED }, + [TCA_TAPRIO_ATTR_SCHED_CLOCKID] = { .type = NLA_S32 }, + [TCA_TAPRIO_ATTR_SCHED_CYCLE_TIME] = { .type = NLA_S64 }, + [TCA_TAPRIO_ATTR_SCHED_CYCLE_TIME_EXTENSION] = { .type = NLA_S64 }, }; static int fill_sched_entry(struct nlattr **tb, struct sched_entry *entry, @@ -302,8 +446,8 @@ static int parse_sched_entry(struct nlattr *n, struct sched_entry *entry, struct nlattr *tb[TCA_TAPRIO_SCHED_ENTRY_MAX + 1] = { }; int err; - err = nla_parse_nested(tb, TCA_TAPRIO_SCHED_ENTRY_MAX, n, - entry_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_TAPRIO_SCHED_ENTRY_MAX, n, + entry_policy, NULL); if (err < 0) { NL_SET_ERR_MSG(extack, "Could not parse nested entry"); return -EINVAL; @@ -314,70 +458,8 @@ static int parse_sched_entry(struct nlattr *n, struct sched_entry *entry, return fill_sched_entry(tb, entry, extack); } -/* Returns the number of entries in case of success */ -static int parse_sched_single_entry(struct nlattr *n, - struct taprio_sched *q, - struct netlink_ext_ack *extack) -{ - struct nlattr *tb_entry[TCA_TAPRIO_SCHED_ENTRY_MAX + 1] = { }; - struct nlattr *tb_list[TCA_TAPRIO_SCHED_MAX + 1] = { }; - struct sched_entry *entry; - bool found = false; - u32 index; - int err; - - err = nla_parse_nested(tb_list, TCA_TAPRIO_SCHED_MAX, - n, entry_list_policy, NULL); - if (err < 0) { - NL_SET_ERR_MSG(extack, "Could not parse nested entry"); - return -EINVAL; - } - - if (!tb_list[TCA_TAPRIO_SCHED_ENTRY]) { - NL_SET_ERR_MSG(extack, "Single-entry must include an entry"); - return -EINVAL; - } - - err = nla_parse_nested(tb_entry, TCA_TAPRIO_SCHED_ENTRY_MAX, - tb_list[TCA_TAPRIO_SCHED_ENTRY], - entry_policy, NULL); - if (err < 0) { - NL_SET_ERR_MSG(extack, "Could not parse nested entry"); - return -EINVAL; - } - - if (!tb_entry[TCA_TAPRIO_SCHED_ENTRY_INDEX]) { - NL_SET_ERR_MSG(extack, "Entry must specify an index\n"); - return -EINVAL; - } - - index = nla_get_u32(tb_entry[TCA_TAPRIO_SCHED_ENTRY_INDEX]); - if (index >= q->num_entries) { - NL_SET_ERR_MSG(extack, "Index for single entry exceeds number of entries in schedule"); - return -EINVAL; - } - - list_for_each_entry(entry, &q->entries, list) { - if (entry->index == index) { - found = true; - break; - } - } - - if (!found) { - NL_SET_ERR_MSG(extack, "Could not find entry"); - return -ENOENT; - } - - err = fill_sched_entry(tb_entry, entry, extack); - if (err < 0) - return err; - - return q->num_entries; -} - static int parse_sched_list(struct nlattr *list, - struct taprio_sched *q, + struct sched_gate_list *sched, struct netlink_ext_ack *extack) { struct nlattr *n; @@ -407,64 +489,42 @@ static int parse_sched_list(struct nlattr *list, return err; } - list_add_tail(&entry->list, &q->entries); + list_add_tail(&entry->list, &sched->entries); i++; } - q->num_entries = i; + sched->num_entries = i; return i; } -/* Returns the number of entries in case of success */ -static int parse_taprio_opt(struct nlattr **tb, struct taprio_sched *q, - struct netlink_ext_ack *extack) +static int parse_taprio_schedule(struct nlattr **tb, + struct sched_gate_list *new, + struct netlink_ext_ack *extack) { int err = 0; - int clockid; - if (tb[TCA_TAPRIO_ATTR_SCHED_ENTRY_LIST] && - tb[TCA_TAPRIO_ATTR_SCHED_SINGLE_ENTRY]) - return -EINVAL; - - if (tb[TCA_TAPRIO_ATTR_SCHED_SINGLE_ENTRY] && q->num_entries == 0) - return -EINVAL; - - if (q->clockid == -1 && !tb[TCA_TAPRIO_ATTR_SCHED_CLOCKID]) - return -EINVAL; + if (tb[TCA_TAPRIO_ATTR_SCHED_SINGLE_ENTRY]) { + NL_SET_ERR_MSG(extack, "Adding a single entry is not supported"); + return -ENOTSUPP; + } if (tb[TCA_TAPRIO_ATTR_SCHED_BASE_TIME]) - q->base_time = nla_get_s64( - tb[TCA_TAPRIO_ATTR_SCHED_BASE_TIME]); + new->base_time = nla_get_s64(tb[TCA_TAPRIO_ATTR_SCHED_BASE_TIME]); - if (tb[TCA_TAPRIO_ATTR_SCHED_CLOCKID]) { - clockid = nla_get_s32(tb[TCA_TAPRIO_ATTR_SCHED_CLOCKID]); + if (tb[TCA_TAPRIO_ATTR_SCHED_CYCLE_TIME_EXTENSION]) + new->cycle_time_extension = nla_get_s64(tb[TCA_TAPRIO_ATTR_SCHED_CYCLE_TIME_EXTENSION]); - /* We only support static clockids and we don't allow - * for it to be modified after the first init. - */ - if (clockid < 0 || (q->clockid != -1 && q->clockid != clockid)) - return -EINVAL; - - q->clockid = clockid; - } + if (tb[TCA_TAPRIO_ATTR_SCHED_CYCLE_TIME]) + new->cycle_time = nla_get_s64(tb[TCA_TAPRIO_ATTR_SCHED_CYCLE_TIME]); if (tb[TCA_TAPRIO_ATTR_SCHED_ENTRY_LIST]) err = parse_sched_list( - tb[TCA_TAPRIO_ATTR_SCHED_ENTRY_LIST], q, extack); - else if (tb[TCA_TAPRIO_ATTR_SCHED_SINGLE_ENTRY]) - err = parse_sched_single_entry( - tb[TCA_TAPRIO_ATTR_SCHED_SINGLE_ENTRY], q, extack); - - /* parse_sched_* return the number of entries in the schedule, - * a schedule with zero entries is an error. - */ - if (err == 0) { - NL_SET_ERR_MSG(extack, "The schedule should contain at least one entry"); - return -EINVAL; - } + tb[TCA_TAPRIO_ATTR_SCHED_ENTRY_LIST], new, extack); + if (err < 0) + return err; - return err; + return 0; } static int taprio_parse_mqprio_opt(struct net_device *dev, @@ -473,11 +533,17 @@ static int taprio_parse_mqprio_opt(struct net_device *dev, { int i, j; - if (!qopt) { + if (!qopt && !dev->num_tc) { NL_SET_ERR_MSG(extack, "'mqprio' configuration is necessary"); return -EINVAL; } + /* If num_tc is already set, it means that the user already + * configured the mqprio part + */ + if (dev->num_tc) + return 0; + /* Verify num_tc is not out of max range */ if (qopt->num_tc > TC_MAX_QUEUE) { NL_SET_ERR_MSG(extack, "Number of traffic classes is outside valid range"); @@ -523,76 +589,141 @@ static int taprio_parse_mqprio_opt(struct net_device *dev, return 0; } -static ktime_t taprio_get_start_time(struct Qdisc *sch) +static int taprio_get_start_time(struct Qdisc *sch, + struct sched_gate_list *sched, + ktime_t *start) { struct taprio_sched *q = qdisc_priv(sch); - struct sched_entry *entry; ktime_t now, base, cycle; s64 n; - base = ns_to_ktime(q->base_time); - cycle = 0; - - /* Calculate the cycle_time, by summing all the intervals. - */ - list_for_each_entry(entry, &q->entries, list) - cycle = ktime_add_ns(cycle, entry->interval); + base = sched_base_time(sched); + now = q->get_time(); - if (!cycle) - return base; + if (ktime_after(base, now)) { + *start = base; + return 0; + } - now = q->get_time(); + cycle = get_cycle_time(sched); - if (ktime_after(base, now)) - return base; + /* The qdisc is expected to have at least one sched_entry. Moreover, + * any entry must have 'interval' > 0. Thus if the cycle time is zero, + * something went really wrong. In that case, we should warn about this + * inconsistent state and return error. + */ + if (WARN_ON(!cycle)) + return -EFAULT; /* Schedule the start time for the beginning of the next * cycle. */ n = div64_s64(ktime_sub_ns(now, base), cycle); - - return ktime_add_ns(base, (n + 1) * cycle); + *start = ktime_add_ns(base, (n + 1) * cycle); + return 0; } -static void taprio_start_sched(struct Qdisc *sch, ktime_t start) +static void setup_first_close_time(struct taprio_sched *q, + struct sched_gate_list *sched, ktime_t base) { - struct taprio_sched *q = qdisc_priv(sch); struct sched_entry *first; - unsigned long flags; + ktime_t cycle; - spin_lock_irqsave(&q->current_entry_lock, flags); + first = list_first_entry(&sched->entries, + struct sched_entry, list); + + cycle = get_cycle_time(sched); - first = list_first_entry(&q->entries, struct sched_entry, - list); + /* FIXME: find a better place to do this */ + sched->cycle_close_time = ktime_add_ns(base, cycle); - first->close_time = ktime_add_ns(start, first->interval); - atomic_set(&first->budget, - (first->interval * 1000) / q->picos_per_byte); + first->close_time = ktime_add_ns(base, first->interval); + taprio_set_budget(q, first); rcu_assign_pointer(q->current_entry, NULL); +} - spin_unlock_irqrestore(&q->current_entry_lock, flags); +static void taprio_start_sched(struct Qdisc *sch, + ktime_t start, struct sched_gate_list *new) +{ + struct taprio_sched *q = qdisc_priv(sch); + ktime_t expires; + + expires = hrtimer_get_expires(&q->advance_timer); + if (expires == 0) + expires = KTIME_MAX; + + /* If the new schedule starts before the next expiration, we + * reprogram it to the earliest one, so we change the admin + * schedule to the operational one at the right time. + */ + start = min_t(ktime_t, start, expires); hrtimer_start(&q->advance_timer, start, HRTIMER_MODE_ABS); } +static void taprio_set_picos_per_byte(struct net_device *dev, + struct taprio_sched *q) +{ + struct ethtool_link_ksettings ecmd; + int picos_per_byte = -1; + + if (!__ethtool_get_link_ksettings(dev, &ecmd) && + ecmd.base.speed != SPEED_UNKNOWN) + picos_per_byte = div64_s64(NSEC_PER_SEC * 1000LL * 8, + ecmd.base.speed * 1000 * 1000); + + atomic64_set(&q->picos_per_byte, picos_per_byte); + netdev_dbg(dev, "taprio: set %s's picos_per_byte to: %lld, linkspeed: %d\n", + dev->name, (long long)atomic64_read(&q->picos_per_byte), + ecmd.base.speed); +} + +static int taprio_dev_notifier(struct notifier_block *nb, unsigned long event, + void *ptr) +{ + struct net_device *dev = netdev_notifier_info_to_dev(ptr); + struct net_device *qdev; + struct taprio_sched *q; + bool found = false; + + ASSERT_RTNL(); + + if (event != NETDEV_UP && event != NETDEV_CHANGE) + return NOTIFY_DONE; + + spin_lock(&taprio_list_lock); + list_for_each_entry(q, &taprio_list, taprio_list) { + qdev = qdisc_dev(q->root); + if (qdev == dev) { + found = true; + break; + } + } + spin_unlock(&taprio_list_lock); + + if (found) + taprio_set_picos_per_byte(dev, q); + + return NOTIFY_DONE; +} + static int taprio_change(struct Qdisc *sch, struct nlattr *opt, struct netlink_ext_ack *extack) { struct nlattr *tb[TCA_TAPRIO_ATTR_MAX + 1] = { }; + struct sched_gate_list *oper, *admin, *new_admin; struct taprio_sched *q = qdisc_priv(sch); struct net_device *dev = qdisc_dev(sch); struct tc_mqprio_qopt *mqprio = NULL; - struct ethtool_link_ksettings ecmd; - int i, err, size; - s64 link_speed; + int i, err, clockid; + unsigned long flags; ktime_t start; - err = nla_parse_nested(tb, TCA_TAPRIO_ATTR_MAX, opt, - taprio_policy, extack); + err = nla_parse_nested_deprecated(tb, TCA_TAPRIO_ATTR_MAX, opt, + taprio_policy, extack); if (err < 0) return err; - err = -EINVAL; if (tb[TCA_TAPRIO_ATTR_PRIOMAP]) mqprio = nla_data(tb[TCA_TAPRIO_ATTR_PRIOMAP]); @@ -600,13 +731,78 @@ static int taprio_change(struct Qdisc *sch, struct nlattr *opt, if (err < 0) return err; - /* A schedule with less than one entry is an error */ - size = parse_taprio_opt(tb, q, extack); - if (size < 0) - return size; + new_admin = kzalloc(sizeof(*new_admin), GFP_KERNEL); + if (!new_admin) { + NL_SET_ERR_MSG(extack, "Not enough memory for a new schedule"); + return -ENOMEM; + } + INIT_LIST_HEAD(&new_admin->entries); - hrtimer_init(&q->advance_timer, q->clockid, HRTIMER_MODE_ABS); - q->advance_timer.function = advance_sched; + rcu_read_lock(); + oper = rcu_dereference(q->oper_sched); + admin = rcu_dereference(q->admin_sched); + rcu_read_unlock(); + + if (mqprio && (oper || admin)) { + NL_SET_ERR_MSG(extack, "Changing the traffic mapping of a running schedule is not supported"); + err = -ENOTSUPP; + goto free_sched; + } + + err = parse_taprio_schedule(tb, new_admin, extack); + if (err < 0) + goto free_sched; + + if (new_admin->num_entries == 0) { + NL_SET_ERR_MSG(extack, "There should be at least one entry in the schedule"); + err = -EINVAL; + goto free_sched; + } + + if (tb[TCA_TAPRIO_ATTR_SCHED_CLOCKID]) { + clockid = nla_get_s32(tb[TCA_TAPRIO_ATTR_SCHED_CLOCKID]); + + /* We only support static clockids and we don't allow + * for it to be modified after the first init. + */ + if (clockid < 0 || + (q->clockid != -1 && q->clockid != clockid)) { + NL_SET_ERR_MSG(extack, "Changing the 'clockid' of a running schedule is not supported"); + err = -ENOTSUPP; + goto free_sched; + } + + q->clockid = clockid; + } + + if (q->clockid == -1 && !tb[TCA_TAPRIO_ATTR_SCHED_CLOCKID]) { + NL_SET_ERR_MSG(extack, "Specifying a 'clockid' is mandatory"); + err = -EINVAL; + goto free_sched; + } + + taprio_set_picos_per_byte(dev, q); + + /* Protects against enqueue()/dequeue() */ + spin_lock_bh(qdisc_lock(sch)); + + if (!hrtimer_active(&q->advance_timer)) { + hrtimer_init(&q->advance_timer, q->clockid, HRTIMER_MODE_ABS); + q->advance_timer.function = advance_sched; + } + + if (mqprio) { + netdev_set_num_tc(dev, mqprio->num_tc); + for (i = 0; i < mqprio->num_tc; i++) + netdev_set_tc_queue(dev, i, + mqprio->count[i], + mqprio->offset[i]); + + /* Always use supplied priority mappings */ + for (i = 0; i < TC_BITMASK + 1; i++) + netdev_set_prio_tc_map(dev, i, + mqprio->prio_tc_map[i]); + } switch (q->clockid) { case CLOCK_REALTIME: @@ -622,65 +818,52 @@ static int taprio_change(struct Qdisc *sch, struct nlattr *opt, q->get_time = ktime_get_clocktai; break; default: - return -ENOTSUPP; + NL_SET_ERR_MSG(extack, "Invalid 'clockid'"); + err = -EINVAL; + goto unlock; } - for (i = 0; i < dev->num_tx_queues; i++) { - struct netdev_queue *dev_queue; - struct Qdisc *qdisc; - - dev_queue = netdev_get_tx_queue(dev, i); - qdisc = qdisc_create_dflt(dev_queue, - &pfifo_qdisc_ops, - TC_H_MAKE(TC_H_MAJ(sch->handle), - TC_H_MIN(i + 1)), - extack); - if (!qdisc) - return -ENOMEM; + err = taprio_get_start_time(sch, new_admin, &start); + if (err < 0) { + NL_SET_ERR_MSG(extack, "Internal error: failed get start time"); + goto unlock; + } - if (i < dev->real_num_tx_queues) - qdisc_hash_add(qdisc, false); + setup_first_close_time(q, new_admin, start); - q->qdiscs[i] = qdisc; - } + /* Protects against advance_sched() */ + spin_lock_irqsave(&q->current_entry_lock, flags); - if (mqprio) { - netdev_set_num_tc(dev, mqprio->num_tc); - for (i = 0; i < mqprio->num_tc; i++) - netdev_set_tc_queue(dev, i, - mqprio->count[i], - mqprio->offset[i]); + taprio_start_sched(sch, start, new_admin); - /* Always use supplied priority mappings */ - for (i = 0; i < TC_BITMASK + 1; i++) - netdev_set_prio_tc_map(dev, i, - mqprio->prio_tc_map[i]); - } + rcu_assign_pointer(q->admin_sched, new_admin); + if (admin) + call_rcu(&admin->rcu, taprio_free_sched_cb); + new_admin = NULL; - if (!__ethtool_get_link_ksettings(dev, &ecmd)) - link_speed = ecmd.base.speed; - else - link_speed = SPEED_1000; + spin_unlock_irqrestore(&q->current_entry_lock, flags); - q->picos_per_byte = div64_s64(NSEC_PER_SEC * 1000LL * 8, - link_speed * 1000 * 1000); + err = 0; - start = taprio_get_start_time(sch); - if (!start) - return 0; +unlock: + spin_unlock_bh(qdisc_lock(sch)); - taprio_start_sched(sch, start); +free_sched: + kfree(new_admin); - return 0; + return err; } static void taprio_destroy(struct Qdisc *sch) { struct taprio_sched *q = qdisc_priv(sch); struct net_device *dev = qdisc_dev(sch); - struct sched_entry *entry, *n; unsigned int i; + spin_lock(&taprio_list_lock); + list_del(&q->taprio_list); + spin_unlock(&taprio_list_lock); + hrtimer_cancel(&q->advance_timer); if (q->qdiscs) { @@ -693,10 +876,11 @@ static void taprio_destroy(struct Qdisc *sch) netdev_set_num_tc(dev, 0); - list_for_each_entry_safe(entry, n, &q->entries, list) { - list_del(&entry->list); - kfree(entry); - } + if (q->oper_sched) + call_rcu(&q->oper_sched->rcu, taprio_free_sched_cb); + + if (q->admin_sched) + call_rcu(&q->admin_sched->rcu, taprio_free_sched_cb); } static int taprio_init(struct Qdisc *sch, struct nlattr *opt, @@ -704,12 +888,12 @@ static int taprio_init(struct Qdisc *sch, struct nlattr *opt, { struct taprio_sched *q = qdisc_priv(sch); struct net_device *dev = qdisc_dev(sch); + int i; - INIT_LIST_HEAD(&q->entries); spin_lock_init(&q->current_entry_lock); - /* We may overwrite the configuration later */ hrtimer_init(&q->advance_timer, CLOCK_TAI, HRTIMER_MODE_ABS); + q->advance_timer.function = advance_sched; q->root = sch; @@ -735,6 +919,29 @@ static int taprio_init(struct Qdisc *sch, struct nlattr *opt, if (!opt) return -EINVAL; + spin_lock(&taprio_list_lock); + list_add(&q->taprio_list, &taprio_list); + spin_unlock(&taprio_list_lock); + + for (i = 0; i < dev->num_tx_queues; i++) { + struct netdev_queue *dev_queue; + struct Qdisc *qdisc; + + dev_queue = netdev_get_tx_queue(dev, i); + qdisc = qdisc_create_dflt(dev_queue, + &pfifo_qdisc_ops, + TC_H_MAKE(TC_H_MAJ(sch->handle), + TC_H_MIN(i + 1)), + extack); + if (!qdisc) + return -ENOMEM; + + if (i < dev->real_num_tx_queues) + qdisc_hash_add(qdisc, false); + + q->qdiscs[i] = qdisc; + } + return taprio_change(sch, opt, extack); } @@ -781,7 +988,7 @@ static int dump_entry(struct sk_buff *msg, { struct nlattr *item; - item = nla_nest_start(msg, TCA_TAPRIO_SCHED_ENTRY); + item = nla_nest_start_noflag(msg, TCA_TAPRIO_SCHED_ENTRY); if (!item) return -ENOSPC; @@ -806,15 +1013,55 @@ nla_put_failure: return -1; } +static int dump_schedule(struct sk_buff *msg, + const struct sched_gate_list *root) +{ + struct nlattr *entry_list; + struct sched_entry *entry; + + if (nla_put_s64(msg, TCA_TAPRIO_ATTR_SCHED_BASE_TIME, + root->base_time, TCA_TAPRIO_PAD)) + return -1; + + if (nla_put_s64(msg, TCA_TAPRIO_ATTR_SCHED_CYCLE_TIME, + root->cycle_time, TCA_TAPRIO_PAD)) + return -1; + + if (nla_put_s64(msg, TCA_TAPRIO_ATTR_SCHED_CYCLE_TIME_EXTENSION, + root->cycle_time_extension, TCA_TAPRIO_PAD)) + return -1; + + entry_list = nla_nest_start_noflag(msg, + TCA_TAPRIO_ATTR_SCHED_ENTRY_LIST); + if (!entry_list) + goto error_nest; + + list_for_each_entry(entry, &root->entries, list) { + if (dump_entry(msg, entry) < 0) + goto error_nest; + } + + nla_nest_end(msg, entry_list); + return 0; + +error_nest: + nla_nest_cancel(msg, entry_list); + return -1; +} + static int taprio_dump(struct Qdisc *sch, struct sk_buff *skb) { struct taprio_sched *q = qdisc_priv(sch); struct net_device *dev = qdisc_dev(sch); + struct sched_gate_list *oper, *admin; struct tc_mqprio_qopt opt = { 0 }; - struct nlattr *nest, *entry_list; - struct sched_entry *entry; + struct nlattr *nest, *sched_nest; unsigned int i; + rcu_read_lock(); + oper = rcu_dereference(q->oper_sched); + admin = rcu_dereference(q->admin_sched); + opt.num_tc = netdev_get_num_tc(dev); memcpy(opt.prio_tc_map, dev->prio_tc_map, sizeof(opt.prio_tc_map)); @@ -823,36 +1070,45 @@ static int taprio_dump(struct Qdisc *sch, struct sk_buff *skb) opt.offset[i] = dev->tc_to_txq[i].offset; } - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (!nest) - return -ENOSPC; + goto start_error; if (nla_put(skb, TCA_TAPRIO_ATTR_PRIOMAP, sizeof(opt), &opt)) goto options_error; - if (nla_put_s64(skb, TCA_TAPRIO_ATTR_SCHED_BASE_TIME, - q->base_time, TCA_TAPRIO_PAD)) + if (nla_put_s32(skb, TCA_TAPRIO_ATTR_SCHED_CLOCKID, q->clockid)) goto options_error; - if (nla_put_s32(skb, TCA_TAPRIO_ATTR_SCHED_CLOCKID, q->clockid)) + if (oper && dump_schedule(skb, oper)) goto options_error; - entry_list = nla_nest_start(skb, TCA_TAPRIO_ATTR_SCHED_ENTRY_LIST); - if (!entry_list) + if (!admin) + goto done; + + sched_nest = nla_nest_start_noflag(skb, TCA_TAPRIO_ATTR_ADMIN_SCHED); + if (!sched_nest) goto options_error; - list_for_each_entry(entry, &q->entries, list) { - if (dump_entry(skb, entry) < 0) - goto options_error; - } + if (dump_schedule(skb, admin)) + goto admin_error; + + nla_nest_end(skb, sched_nest); - nla_nest_end(skb, entry_list); +done: + rcu_read_unlock(); return nla_nest_end(skb, nest); +admin_error: + nla_nest_cancel(skb, sched_nest); + options_error: nla_nest_cancel(skb, nest); - return -1; + +start_error: + rcu_read_unlock(); + return -ENOSPC; } static struct Qdisc *taprio_leaf(struct Qdisc *sch, unsigned long cl) @@ -939,6 +1195,7 @@ static struct Qdisc_ops taprio_qdisc_ops __read_mostly = { .id = "taprio", .priv_size = sizeof(struct taprio_sched), .init = taprio_init, + .change = taprio_change, .destroy = taprio_destroy, .peek = taprio_peek, .dequeue = taprio_dequeue, @@ -947,14 +1204,24 @@ static struct Qdisc_ops taprio_qdisc_ops __read_mostly = { .owner = THIS_MODULE, }; +static struct notifier_block taprio_device_notifier = { + .notifier_call = taprio_dev_notifier, +}; + static int __init taprio_module_init(void) { + int err = register_netdevice_notifier(&taprio_device_notifier); + + if (err) + return err; + return register_qdisc(&taprio_qdisc_ops); } static void __exit taprio_module_exit(void) { unregister_qdisc(&taprio_qdisc_ops); + unregister_netdevice_notifier(&taprio_device_notifier); } module_init(taprio_module_init); diff --git a/net/sched/sch_tbf.c b/net/sched/sch_tbf.c index f71578dbb9e39292329e06d98535c764043acd55..c09c0d855846ba2a4f2e0c031af22bae1069bdcc 100644 --- a/net/sched/sch_tbf.c +++ b/net/sched/sch_tbf.c @@ -308,7 +308,8 @@ static int tbf_change(struct Qdisc *sch, struct nlattr *opt, s64 buffer, mtu; u64 rate64 = 0, prate64 = 0; - err = nla_parse_nested(tb, TCA_TBF_MAX, opt, tbf_policy, NULL); + err = nla_parse_nested_deprecated(tb, TCA_TBF_MAX, opt, tbf_policy, + NULL); if (err < 0) return err; @@ -448,7 +449,7 @@ static int tbf_dump(struct Qdisc *sch, struct sk_buff *skb) struct tc_tbf_qopt opt; sch->qstats.backlog = q->qdisc->qstats.backlog; - nest = nla_nest_start(skb, TCA_OPTIONS); + nest = nla_nest_start_noflag(skb, TCA_OPTIONS); if (nest == NULL) goto nla_put_failure; diff --git a/net/sctp/ipv6.c b/net/sctp/ipv6.c index 6200cd2b4b996b156aad38e3d94f0e87335f9cad..188c47eb206ea830641d5f1cf9746743b2f23781 100644 --- a/net/sctp/ipv6.c +++ b/net/sctp/ipv6.c @@ -1030,6 +1030,7 @@ static const struct proto_ops inet6_seqpacket_ops = { .getname = sctp_getname, .poll = sctp_poll, .ioctl = inet6_ioctl, + .gettstamp = sock_gettstamp, .listen = sctp_inet_listen, .shutdown = inet_shutdown, .setsockopt = sock_common_setsockopt, diff --git a/net/sctp/protocol.c b/net/sctp/protocol.c index 951afdeea5e92c7cab48f53482d307c3f9893d89..f0631bf486b655486028cca166a167ce10da1879 100644 --- a/net/sctp/protocol.c +++ b/net/sctp/protocol.c @@ -1026,6 +1026,7 @@ static const struct proto_ops inet_seqpacket_ops = { .getname = inet_getname, /* Semantics are different. */ .poll = sctp_poll, .ioctl = inet_ioctl, + .gettstamp = sock_gettstamp, .listen = sctp_inet_listen, .shutdown = inet_shutdown, /* Looks harmless. */ .setsockopt = sock_common_setsockopt, /* IP_SOL IP_OPTION is a problem */ diff --git a/net/sctp/sm_statefuns.c b/net/sctp/sm_statefuns.c index 713a669d205858a39d65af31396ab74abe138dc2..e3f4abe6134e58ff5558c5b3349f26a76da549e9 100644 --- a/net/sctp/sm_statefuns.c +++ b/net/sctp/sm_statefuns.c @@ -6431,13 +6431,15 @@ static int sctp_eat_data(const struct sctp_association *asoc, * in sctp_ulpevent_make_rcvmsg will drop the frame if we grow our * memory usage too much */ - if (*sk->sk_prot_creator->memory_pressure) { + if (sk_under_memory_pressure(sk)) { if (sctp_tsnmap_has_gap(map) && (sctp_tsnmap_get_ctsn(map) + 1) == tsn) { pr_debug("%s: under pressure, reneging for tsn:%u\n", __func__, tsn); deliver = SCTP_CMD_RENEGE; - } + } else { + sk_mem_reclaim(sk); + } } /* diff --git a/net/sctp/socket.c b/net/sctp/socket.c index 4583fa914e62aedaf2ef29c5cf668f0caee4eade..e4e892cc5644811b876d5a1cbd25e4fe3ef30136 100644 --- a/net/sctp/socket.c +++ b/net/sctp/socket.c @@ -1913,7 +1913,10 @@ static int sctp_sendmsg_to_asoc(struct sctp_association *asoc, if (sctp_wspace(asoc) < (int)msg_len) sctp_prsctp_prune(asoc, sinfo, msg_len - sctp_wspace(asoc)); - if (sctp_wspace(asoc) <= 0) { + if (sk_under_memory_pressure(sk)) + sk_mem_reclaim(sk); + + if (sctp_wspace(asoc) <= 0 || !sk_wmem_schedule(sk, msg_len)) { timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT); err = sctp_wait_for_sndbuf(asoc, &timeo, msg_len); if (err) @@ -8931,7 +8934,10 @@ static int sctp_wait_for_sndbuf(struct sctp_association *asoc, long *timeo_p, goto do_error; if (signal_pending(current)) goto do_interrupted; - if ((int)msg_len <= sctp_wspace(asoc)) + if (sk_under_memory_pressure(sk)) + sk_mem_reclaim(sk); + if ((int)msg_len <= sctp_wspace(asoc) && + sk_wmem_schedule(sk, msg_len)) break; /* Let another process have a go. Since we are going diff --git a/net/sctp/stream_interleave.c b/net/sctp/stream_interleave.c index 102c6fefe38c93be2760ca7df0adfe62bead1313..25e0b7e5189c5be42fcd0dd8f669baedd921abdd 100644 --- a/net/sctp/stream_interleave.c +++ b/net/sctp/stream_interleave.c @@ -484,14 +484,15 @@ static struct sctp_ulpevent *sctp_intl_order(struct sctp_ulpq *ulpq, } static int sctp_enqueue_event(struct sctp_ulpq *ulpq, - struct sctp_ulpevent *event) + struct sk_buff_head *skb_list) { - struct sk_buff *skb = sctp_event2skb(event); struct sock *sk = ulpq->asoc->base.sk; struct sctp_sock *sp = sctp_sk(sk); - struct sk_buff_head *skb_list; + struct sctp_ulpevent *event; + struct sk_buff *skb; - skb_list = (struct sk_buff_head *)skb->prev; + skb = __skb_peek(skb_list); + event = sctp_skb2event(skb); if (sk->sk_shutdown & RCV_SHUTDOWN && (sk->sk_shutdown & SEND_SHUTDOWN || @@ -858,19 +859,24 @@ static int sctp_ulpevent_idata(struct sctp_ulpq *ulpq, if (!(event->msg_flags & SCTP_DATA_UNORDERED)) { event = sctp_intl_reasm(ulpq, event); - if (event && event->msg_flags & MSG_EOR) { + if (event) { skb_queue_head_init(&temp); __skb_queue_tail(&temp, sctp_event2skb(event)); - event = sctp_intl_order(ulpq, event); + if (event->msg_flags & MSG_EOR) + event = sctp_intl_order(ulpq, event); } } else { event = sctp_intl_reasm_uo(ulpq, event); + if (event) { + skb_queue_head_init(&temp); + __skb_queue_tail(&temp, sctp_event2skb(event)); + } } if (event) { event_eor = (event->msg_flags & MSG_EOR) ? 1 : 0; - sctp_enqueue_event(ulpq, event); + sctp_enqueue_event(ulpq, &temp); } return event_eor; @@ -944,20 +950,27 @@ out: static void sctp_intl_start_pd(struct sctp_ulpq *ulpq, gfp_t gfp) { struct sctp_ulpevent *event; + struct sk_buff_head temp; if (!skb_queue_empty(&ulpq->reasm)) { do { event = sctp_intl_retrieve_first(ulpq); - if (event) - sctp_enqueue_event(ulpq, event); + if (event) { + skb_queue_head_init(&temp); + __skb_queue_tail(&temp, sctp_event2skb(event)); + sctp_enqueue_event(ulpq, &temp); + } } while (event); } if (!skb_queue_empty(&ulpq->reasm_uo)) { do { event = sctp_intl_retrieve_first_uo(ulpq); - if (event) - sctp_enqueue_event(ulpq, event); + if (event) { + skb_queue_head_init(&temp); + __skb_queue_tail(&temp, sctp_event2skb(event)); + sctp_enqueue_event(ulpq, &temp); + } } while (event); } } @@ -1059,7 +1072,7 @@ static void sctp_intl_reap_ordered(struct sctp_ulpq *ulpq, __u16 sid) if (event) { sctp_intl_retrieve_ordered(ulpq, event); - sctp_enqueue_event(ulpq, event); + sctp_enqueue_event(ulpq, &temp); } } @@ -1298,6 +1311,15 @@ static void sctp_handle_iftsn(struct sctp_ulpq *ulpq, struct sctp_chunk *chunk) ntohl(skip->mid), skip->flags); } +static int do_ulpq_tail_event(struct sctp_ulpq *ulpq, struct sctp_ulpevent *event) +{ + struct sk_buff_head temp; + + skb_queue_head_init(&temp); + __skb_queue_tail(&temp, sctp_event2skb(event)); + return sctp_ulpq_tail_event(ulpq, &temp); +} + static struct sctp_stream_interleave sctp_stream_interleave_0 = { .data_chunk_len = sizeof(struct sctp_data_chunk), .ftsn_chunk_len = sizeof(struct sctp_fwdtsn_chunk), @@ -1306,7 +1328,7 @@ static struct sctp_stream_interleave sctp_stream_interleave_0 = { .assign_number = sctp_chunk_assign_ssn, .validate_data = sctp_validate_data, .ulpevent_data = sctp_ulpq_tail_data, - .enqueue_event = sctp_ulpq_tail_event, + .enqueue_event = do_ulpq_tail_event, .renege_events = sctp_ulpq_renege, .start_pd = sctp_ulpq_partial_delivery, .abort_pd = sctp_ulpq_abort_pd, @@ -1317,6 +1339,16 @@ static struct sctp_stream_interleave sctp_stream_interleave_0 = { .handle_ftsn = sctp_handle_fwdtsn, }; +static int do_sctp_enqueue_event(struct sctp_ulpq *ulpq, + struct sctp_ulpevent *event) +{ + struct sk_buff_head temp; + + skb_queue_head_init(&temp); + __skb_queue_tail(&temp, sctp_event2skb(event)); + return sctp_enqueue_event(ulpq, &temp); +} + static struct sctp_stream_interleave sctp_stream_interleave_1 = { .data_chunk_len = sizeof(struct sctp_idata_chunk), .ftsn_chunk_len = sizeof(struct sctp_ifwdtsn_chunk), @@ -1325,7 +1357,7 @@ static struct sctp_stream_interleave sctp_stream_interleave_1 = { .assign_number = sctp_chunk_assign_mid, .validate_data = sctp_validate_idata, .ulpevent_data = sctp_ulpevent_idata, - .enqueue_event = sctp_enqueue_event, + .enqueue_event = do_sctp_enqueue_event, .renege_events = sctp_renege_events, .start_pd = sctp_intl_start_pd, .abort_pd = sctp_intl_abort_pd, diff --git a/net/sctp/ulpevent.c b/net/sctp/ulpevent.c index 8cb7d9858270a617e46e32e988babf86196ef84c..c2a7478587ab4fe53e3e30178b604b3eb46c5f8d 100644 --- a/net/sctp/ulpevent.c +++ b/net/sctp/ulpevent.c @@ -634,8 +634,9 @@ struct sctp_ulpevent *sctp_ulpevent_make_rcvmsg(struct sctp_association *asoc, gfp_t gfp) { struct sctp_ulpevent *event = NULL; - struct sk_buff *skb; - size_t padding, len; + struct sk_buff *skb = chunk->skb; + struct sock *sk = asoc->base.sk; + size_t padding, datalen; int rx_count; /* @@ -646,15 +647,12 @@ struct sctp_ulpevent *sctp_ulpevent_make_rcvmsg(struct sctp_association *asoc, if (asoc->ep->rcvbuf_policy) rx_count = atomic_read(&asoc->rmem_alloc); else - rx_count = atomic_read(&asoc->base.sk->sk_rmem_alloc); + rx_count = atomic_read(&sk->sk_rmem_alloc); - if (rx_count >= asoc->base.sk->sk_rcvbuf) { + datalen = ntohs(chunk->chunk_hdr->length); - if ((asoc->base.sk->sk_userlocks & SOCK_RCVBUF_LOCK) || - (!sk_rmem_schedule(asoc->base.sk, chunk->skb, - chunk->skb->truesize))) - goto fail; - } + if (rx_count >= sk->sk_rcvbuf || !sk_rmem_schedule(sk, skb, datalen)) + goto fail; /* Clone the original skb, sharing the data. */ skb = skb_clone(chunk->skb, gfp); @@ -681,8 +679,7 @@ struct sctp_ulpevent *sctp_ulpevent_make_rcvmsg(struct sctp_association *asoc, * The sender should never pad with more than 3 bytes. The receiver * MUST ignore the padding bytes. */ - len = ntohs(chunk->chunk_hdr->length); - padding = SCTP_PAD4(len) - len; + padding = SCTP_PAD4(datalen) - datalen; /* Fixup cloned skb with just this chunks data. */ skb_trim(skb, chunk->chunk_end - padding - skb->data); diff --git a/net/sctp/ulpqueue.c b/net/sctp/ulpqueue.c index 5dde92101743768e05682991759b5713ecbde906..a212fe079c07e17dd533b71bc3dbb5526c89d8c0 100644 --- a/net/sctp/ulpqueue.c +++ b/net/sctp/ulpqueue.c @@ -116,12 +116,13 @@ int sctp_ulpq_tail_data(struct sctp_ulpq *ulpq, struct sctp_chunk *chunk, event = sctp_ulpq_reasm(ulpq, event); /* Do ordering if needed. */ - if ((event) && (event->msg_flags & MSG_EOR)) { + if (event) { /* Create a temporary list to collect chunks on. */ skb_queue_head_init(&temp); __skb_queue_tail(&temp, sctp_event2skb(event)); - event = sctp_ulpq_order(ulpq, event); + if (event->msg_flags & MSG_EOR) + event = sctp_ulpq_order(ulpq, event); } /* Send event to the ULP. 'event' is the sctp_ulpevent for @@ -129,7 +130,7 @@ int sctp_ulpq_tail_data(struct sctp_ulpq *ulpq, struct sctp_chunk *chunk, */ if (event) { event_eor = (event->msg_flags & MSG_EOR) ? 1 : 0; - sctp_ulpq_tail_event(ulpq, event); + sctp_ulpq_tail_event(ulpq, &temp); } return event_eor; @@ -193,18 +194,17 @@ static int sctp_ulpq_clear_pd(struct sctp_ulpq *ulpq) return sctp_clear_pd(ulpq->asoc->base.sk, ulpq->asoc); } -/* If the SKB of 'event' is on a list, it is the first such member - * of that list. - */ -int sctp_ulpq_tail_event(struct sctp_ulpq *ulpq, struct sctp_ulpevent *event) +int sctp_ulpq_tail_event(struct sctp_ulpq *ulpq, struct sk_buff_head *skb_list) { struct sock *sk = ulpq->asoc->base.sk; struct sctp_sock *sp = sctp_sk(sk); - struct sk_buff_head *queue, *skb_list; - struct sk_buff *skb = sctp_event2skb(event); + struct sctp_ulpevent *event; + struct sk_buff_head *queue; + struct sk_buff *skb; int clear_pd = 0; - skb_list = (struct sk_buff_head *) skb->prev; + skb = __skb_peek(skb_list); + event = sctp_skb2event(skb); /* If the socket is just going to throw this away, do not * even try to deliver it. @@ -257,13 +257,7 @@ int sctp_ulpq_tail_event(struct sctp_ulpq *ulpq, struct sctp_ulpevent *event) } } - /* If we are harvesting multiple skbs they will be - * collected on a list. - */ - if (skb_list) - skb_queue_splice_tail_init(skb_list, queue); - else - __skb_queue_tail(queue, skb); + skb_queue_splice_tail_init(skb_list, queue); /* Did we just complete partial delivery and need to get * rolling again? Move pending data to the receive @@ -738,25 +732,25 @@ void sctp_ulpq_reasm_flushtsn(struct sctp_ulpq *ulpq, __u32 fwd_tsn) static void sctp_ulpq_reasm_drain(struct sctp_ulpq *ulpq) { struct sctp_ulpevent *event = NULL; - struct sk_buff_head temp; if (skb_queue_empty(&ulpq->reasm)) return; while ((event = sctp_ulpq_retrieve_reassembled(ulpq)) != NULL) { - /* Do ordering if needed. */ - if ((event) && (event->msg_flags & MSG_EOR)) { - skb_queue_head_init(&temp); - __skb_queue_tail(&temp, sctp_event2skb(event)); + struct sk_buff_head temp; + + skb_queue_head_init(&temp); + __skb_queue_tail(&temp, sctp_event2skb(event)); + /* Do ordering if needed. */ + if (event->msg_flags & MSG_EOR) event = sctp_ulpq_order(ulpq, event); - } /* Send event to the ULP. 'event' is the * sctp_ulpevent for very first SKB on the temp' list. */ if (event) - sctp_ulpq_tail_event(ulpq, event); + sctp_ulpq_tail_event(ulpq, &temp); } } @@ -956,7 +950,7 @@ static void sctp_ulpq_reap_ordered(struct sctp_ulpq *ulpq, __u16 sid) if (event) { /* see if we have more ordered that we can deliver */ sctp_ulpq_retrieve_ordered(ulpq, event); - sctp_ulpq_tail_event(ulpq, event); + sctp_ulpq_tail_event(ulpq, &temp); } } @@ -1082,7 +1076,11 @@ void sctp_ulpq_partial_delivery(struct sctp_ulpq *ulpq, event = sctp_ulpq_retrieve_first(ulpq); /* Send event to the ULP. */ if (event) { - sctp_ulpq_tail_event(ulpq, event); + struct sk_buff_head temp; + + skb_queue_head_init(&temp); + __skb_queue_tail(&temp, sctp_event2skb(event)); + sctp_ulpq_tail_event(ulpq, &temp); sctp_ulpq_set_pd(ulpq); return; } @@ -1106,7 +1104,8 @@ void sctp_ulpq_renege(struct sctp_ulpq *ulpq, struct sctp_chunk *chunk, freed += sctp_ulpq_renege_frags(ulpq, needed - freed); } /* If able to free enough room, accept this chunk. */ - if (freed >= needed) { + if (sk_rmem_schedule(asoc->base.sk, chunk->skb, needed) && + freed >= needed) { int retval = sctp_ulpq_tail_data(ulpq, chunk, gfp); /* * Enter partial delivery if chunk has not been diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c index 6f869ef49b3226806ab7f7973821870d77618004..086d9913975ddeabcbfa1a80bd51501ee8d304f0 100644 --- a/net/smc/af_smc.c +++ b/net/smc/af_smc.c @@ -134,11 +134,9 @@ static int smc_release(struct socket *sock) smc = smc_sk(sk); /* cleanup for a dangling non-blocking connect */ - if (smc->connect_info && sk->sk_state == SMC_INIT) + if (smc->connect_nonblock && sk->sk_state == SMC_INIT) tcp_abort(smc->clcsock->sk, ECONNABORTED); flush_work(&smc->connect_work); - kfree(smc->connect_info); - smc->connect_info = NULL; if (sk->sk_state == SMC_LISTEN) /* smc_close_non_accepted() is called and acquires @@ -460,6 +458,7 @@ static int smc_connect_fallback(struct smc_sock *smc, int reason_code) smc_switch_to_fallback(smc); smc->fallback_rsn = reason_code; smc_copy_sock_settings_to_clc(smc); + smc->connect_nonblock = 0; if (smc->sk.sk_state == SMC_INIT) smc->sk.sk_state = SMC_ACTIVE; return 0; @@ -499,46 +498,41 @@ static int smc_connect_abort(struct smc_sock *smc, int reason_code, mutex_unlock(&smc_client_lgr_pending); smc_conn_free(&smc->conn); + smc->connect_nonblock = 0; return reason_code; } /* check if there is a rdma device available for this connection. */ /* called for connect and listen */ -static int smc_check_rdma(struct smc_sock *smc, struct smc_ib_device **ibdev, - u8 *ibport, unsigned short vlan_id, u8 gid[]) +static int smc_find_rdma_device(struct smc_sock *smc, struct smc_init_info *ini) { - int reason_code = 0; - /* PNET table look up: search active ib_device and port * within same PNETID that also contains the ethernet device * used for the internal TCP socket */ - smc_pnet_find_roce_resource(smc->clcsock->sk, ibdev, ibport, vlan_id, - gid); - if (!(*ibdev)) - reason_code = SMC_CLC_DECL_CNFERR; /* configuration error */ - - return reason_code; + smc_pnet_find_roce_resource(smc->clcsock->sk, ini); + if (!ini->ib_dev) + return SMC_CLC_DECL_NOSMCRDEV; + return 0; } /* check if there is an ISM device available for this connection. */ /* called for connect and listen */ -static int smc_check_ism(struct smc_sock *smc, struct smcd_dev **ismdev) +static int smc_find_ism_device(struct smc_sock *smc, struct smc_init_info *ini) { /* Find ISM device with same PNETID as connecting interface */ - smc_pnet_find_ism_resource(smc->clcsock->sk, ismdev); - if (!(*ismdev)) - return SMC_CLC_DECL_CNFERR; /* configuration error */ + smc_pnet_find_ism_resource(smc->clcsock->sk, ini); + if (!ini->ism_dev) + return SMC_CLC_DECL_NOSMCDDEV; return 0; } /* Check for VLAN ID and register it on ISM device just for CLC handshake */ static int smc_connect_ism_vlan_setup(struct smc_sock *smc, - struct smcd_dev *ismdev, - unsigned short vlan_id) + struct smc_init_info *ini) { - if (vlan_id && smc_ism_get_vlan(ismdev, vlan_id)) - return SMC_CLC_DECL_CNFERR; + if (ini->vlan_id && smc_ism_get_vlan(ini->ism_dev, ini->vlan_id)) + return SMC_CLC_DECL_ISMVLANERR; return 0; } @@ -546,12 +540,11 @@ static int smc_connect_ism_vlan_setup(struct smc_sock *smc, * used, the VLAN ID will be registered again during the connection setup. */ static int smc_connect_ism_vlan_cleanup(struct smc_sock *smc, bool is_smcd, - struct smcd_dev *ismdev, - unsigned short vlan_id) + struct smc_init_info *ini) { if (!is_smcd) return 0; - if (vlan_id && smc_ism_put_vlan(ismdev, vlan_id)) + if (ini->vlan_id && smc_ism_put_vlan(ini->ism_dev, ini->vlan_id)) return SMC_CLC_DECL_CNFERR; return 0; } @@ -559,13 +552,12 @@ static int smc_connect_ism_vlan_cleanup(struct smc_sock *smc, bool is_smcd, /* CLC handshake during connect */ static int smc_connect_clc(struct smc_sock *smc, int smc_type, struct smc_clc_msg_accept_confirm *aclc, - struct smc_ib_device *ibdev, u8 ibport, - u8 gid[], struct smcd_dev *ismdev) + struct smc_init_info *ini) { int rc = 0; /* do inband token exchange */ - rc = smc_clc_send_proposal(smc, smc_type, ibdev, ibport, gid, ismdev); + rc = smc_clc_send_proposal(smc, smc_type, ini); if (rc) return rc; /* receive SMC Accept CLC message */ @@ -576,23 +568,19 @@ static int smc_connect_clc(struct smc_sock *smc, int smc_type, /* setup for RDMA connection of client */ static int smc_connect_rdma(struct smc_sock *smc, struct smc_clc_msg_accept_confirm *aclc, - struct smc_ib_device *ibdev, u8 ibport) + struct smc_init_info *ini) { - int local_contact = SMC_FIRST_CONTACT; struct smc_link *link; int reason_code = 0; + ini->is_smcd = false; + ini->ib_lcl = &aclc->lcl; + ini->ib_clcqpn = ntoh24(aclc->qpn); + ini->srv_first_contact = aclc->hdr.flag; + mutex_lock(&smc_client_lgr_pending); - local_contact = smc_conn_create(smc, false, aclc->hdr.flag, ibdev, - ibport, ntoh24(aclc->qpn), &aclc->lcl, - NULL, 0); - if (local_contact < 0) { - if (local_contact == -ENOMEM) - reason_code = SMC_CLC_DECL_MEM;/* insufficient memory*/ - else if (local_contact == -ENOLINK) - reason_code = SMC_CLC_DECL_SYNCERR; /* synchr. error */ - else - reason_code = SMC_CLC_DECL_INTERR; /* other error */ + reason_code = smc_conn_create(smc, ini); + if (reason_code) { mutex_unlock(&smc_client_lgr_pending); return reason_code; } @@ -602,45 +590,48 @@ static int smc_connect_rdma(struct smc_sock *smc, /* create send buffer and rmb */ if (smc_buf_create(smc, false)) - return smc_connect_abort(smc, SMC_CLC_DECL_MEM, local_contact); + return smc_connect_abort(smc, SMC_CLC_DECL_MEM, + ini->cln_first_contact); - if (local_contact == SMC_FIRST_CONTACT) + if (ini->cln_first_contact == SMC_FIRST_CONTACT) smc_link_save_peer_info(link, aclc); if (smc_rmb_rtoken_handling(&smc->conn, aclc)) return smc_connect_abort(smc, SMC_CLC_DECL_ERR_RTOK, - local_contact); + ini->cln_first_contact); smc_close_init(smc); smc_rx_init(smc); - if (local_contact == SMC_FIRST_CONTACT) { + if (ini->cln_first_contact == SMC_FIRST_CONTACT) { if (smc_ib_ready_link(link)) return smc_connect_abort(smc, SMC_CLC_DECL_ERR_RDYLNK, - local_contact); + ini->cln_first_contact); } else { if (smc_reg_rmb(link, smc->conn.rmb_desc, true)) return smc_connect_abort(smc, SMC_CLC_DECL_ERR_REGRMB, - local_contact); + ini->cln_first_contact); } smc_rmb_sync_sg_for_device(&smc->conn); reason_code = smc_clc_send_confirm(smc); if (reason_code) - return smc_connect_abort(smc, reason_code, local_contact); + return smc_connect_abort(smc, reason_code, + ini->cln_first_contact); smc_tx_init(smc); - if (local_contact == SMC_FIRST_CONTACT) { + if (ini->cln_first_contact == SMC_FIRST_CONTACT) { /* QP confirmation over RoCE fabric */ reason_code = smc_clnt_conf_first_link(smc); if (reason_code) return smc_connect_abort(smc, reason_code, - local_contact); + ini->cln_first_contact); } mutex_unlock(&smc_client_lgr_pending); smc_copy_sock_settings_to_clc(smc); + smc->connect_nonblock = 0; if (smc->sk.sk_state == SMC_INIT) smc->sk.sk_state = SMC_ACTIVE; @@ -650,23 +641,26 @@ static int smc_connect_rdma(struct smc_sock *smc, /* setup for ISM connection of client */ static int smc_connect_ism(struct smc_sock *smc, struct smc_clc_msg_accept_confirm *aclc, - struct smcd_dev *ismdev) + struct smc_init_info *ini) { - int local_contact = SMC_FIRST_CONTACT; int rc = 0; + ini->is_smcd = true; + ini->ism_gid = aclc->gid; + ini->srv_first_contact = aclc->hdr.flag; + /* there is only one lgr role for SMC-D; use server lock */ mutex_lock(&smc_server_lgr_pending); - local_contact = smc_conn_create(smc, true, aclc->hdr.flag, NULL, 0, 0, - NULL, ismdev, aclc->gid); - if (local_contact < 0) { + rc = smc_conn_create(smc, ini); + if (rc) { mutex_unlock(&smc_server_lgr_pending); - return SMC_CLC_DECL_MEM; + return rc; } /* Create send and receive buffers */ if (smc_buf_create(smc, true)) - return smc_connect_abort(smc, SMC_CLC_DECL_MEM, local_contact); + return smc_connect_abort(smc, SMC_CLC_DECL_MEM, + ini->cln_first_contact); smc_conn_save_peer_info(smc, aclc); smc_close_init(smc); @@ -675,10 +669,11 @@ static int smc_connect_ism(struct smc_sock *smc, rc = smc_clc_send_confirm(smc); if (rc) - return smc_connect_abort(smc, rc, local_contact); + return smc_connect_abort(smc, rc, ini->cln_first_contact); mutex_unlock(&smc_server_lgr_pending); smc_copy_sock_settings_to_clc(smc); + smc->connect_nonblock = 0; if (smc->sk.sk_state == SMC_INIT) smc->sk.sk_state = SMC_ACTIVE; @@ -690,13 +685,9 @@ static int __smc_connect(struct smc_sock *smc) { bool ism_supported = false, rdma_supported = false; struct smc_clc_msg_accept_confirm aclc; - struct smc_ib_device *ibdev; - struct smcd_dev *ismdev; - u8 gid[SMC_GID_SIZE]; - unsigned short vlan; + struct smc_init_info ini = {0}; int smc_type; int rc = 0; - u8 ibport; sock_hold(&smc->sk); /* sock put in passive closing */ @@ -711,20 +702,21 @@ static int __smc_connect(struct smc_sock *smc) if (using_ipsec(smc)) return smc_connect_decline_fallback(smc, SMC_CLC_DECL_IPSEC); - /* check for VLAN ID */ - if (smc_vlan_by_tcpsk(smc->clcsock, &vlan)) - return smc_connect_decline_fallback(smc, SMC_CLC_DECL_CNFERR); + /* get vlan id from IP device */ + if (smc_vlan_by_tcpsk(smc->clcsock, &ini)) + return smc_connect_decline_fallback(smc, + SMC_CLC_DECL_GETVLANERR); /* check if there is an ism device available */ - if (!smc_check_ism(smc, &ismdev) && - !smc_connect_ism_vlan_setup(smc, ismdev, vlan)) { + if (!smc_find_ism_device(smc, &ini) && + !smc_connect_ism_vlan_setup(smc, &ini)) { /* ISM is supported for this connection */ ism_supported = true; smc_type = SMC_TYPE_D; } /* check if there is a rdma device available */ - if (!smc_check_rdma(smc, &ibdev, &ibport, vlan, gid)) { + if (!smc_find_rdma_device(smc, &ini)) { /* RDMA is supported for this connection */ rdma_supported = true; if (ism_supported) @@ -738,25 +730,25 @@ static int __smc_connect(struct smc_sock *smc) return smc_connect_decline_fallback(smc, SMC_CLC_DECL_NOSMCDEV); /* perform CLC handshake */ - rc = smc_connect_clc(smc, smc_type, &aclc, ibdev, ibport, gid, ismdev); + rc = smc_connect_clc(smc, smc_type, &aclc, &ini); if (rc) { - smc_connect_ism_vlan_cleanup(smc, ism_supported, ismdev, vlan); + smc_connect_ism_vlan_cleanup(smc, ism_supported, &ini); return smc_connect_decline_fallback(smc, rc); } /* depending on previous steps, connect using rdma or ism */ if (rdma_supported && aclc.hdr.path == SMC_TYPE_R) - rc = smc_connect_rdma(smc, &aclc, ibdev, ibport); + rc = smc_connect_rdma(smc, &aclc, &ini); else if (ism_supported && aclc.hdr.path == SMC_TYPE_D) - rc = smc_connect_ism(smc, &aclc, ismdev); + rc = smc_connect_ism(smc, &aclc, &ini); else rc = SMC_CLC_DECL_MODEUNSUPP; if (rc) { - smc_connect_ism_vlan_cleanup(smc, ism_supported, ismdev, vlan); + smc_connect_ism_vlan_cleanup(smc, ism_supported, &ini); return smc_connect_decline_fallback(smc, rc); } - smc_connect_ism_vlan_cleanup(smc, ism_supported, ismdev, vlan); + smc_connect_ism_vlan_cleanup(smc, ism_supported, &ini); return 0; } @@ -764,17 +756,30 @@ static void smc_connect_work(struct work_struct *work) { struct smc_sock *smc = container_of(work, struct smc_sock, connect_work); - int rc; + long timeo = smc->sk.sk_sndtimeo; + int rc = 0; - lock_sock(&smc->sk); - rc = kernel_connect(smc->clcsock, &smc->connect_info->addr, - smc->connect_info->alen, smc->connect_info->flags); + if (!timeo) + timeo = MAX_SCHEDULE_TIMEOUT; + lock_sock(smc->clcsock->sk); if (smc->clcsock->sk->sk_err) { smc->sk.sk_err = smc->clcsock->sk->sk_err; - goto out; - } - if (rc < 0) { - smc->sk.sk_err = -rc; + } else if ((1 << smc->clcsock->sk->sk_state) & + (TCPF_SYN_SENT | TCP_SYN_RECV)) { + rc = sk_stream_wait_connect(smc->clcsock->sk, &timeo); + if ((rc == -EPIPE) && + ((1 << smc->clcsock->sk->sk_state) & + (TCPF_ESTABLISHED | TCPF_CLOSE_WAIT))) + rc = 0; + } + release_sock(smc->clcsock->sk); + lock_sock(&smc->sk); + if (rc != 0 || smc->sk.sk_err) { + smc->sk.sk_state = SMC_CLOSED; + if (rc == -EPIPE || rc == -EAGAIN) + smc->sk.sk_err = EPIPE; + else if (signal_pending(current)) + smc->sk.sk_err = -sock_intr_errno(timeo); goto out; } @@ -791,8 +796,6 @@ out: smc->sk.sk_write_space(&smc->sk); } } - kfree(smc->connect_info); - smc->connect_info = NULL; release_sock(&smc->sk); } @@ -825,26 +828,18 @@ static int smc_connect(struct socket *sock, struct sockaddr *addr, smc_copy_sock_settings_to_clc(smc); tcp_sk(smc->clcsock->sk)->syn_smc = 1; + if (smc->connect_nonblock) { + rc = -EALREADY; + goto out; + } + rc = kernel_connect(smc->clcsock, addr, alen, flags); + if (rc && rc != -EINPROGRESS) + goto out; if (flags & O_NONBLOCK) { - if (smc->connect_info) { - rc = -EALREADY; - goto out; - } - smc->connect_info = kzalloc(alen + 2 * sizeof(int), GFP_KERNEL); - if (!smc->connect_info) { - rc = -ENOMEM; - goto out; - } - smc->connect_info->alen = alen; - smc->connect_info->flags = flags ^ O_NONBLOCK; - memcpy(&smc->connect_info->addr, addr, alen); - schedule_work(&smc->connect_work); + if (schedule_work(&smc->connect_work)) + smc->connect_nonblock = 1; rc = -EINPROGRESS; } else { - rc = kernel_connect(smc->clcsock, addr, alen, flags); - if (rc) - goto out; - rc = __smc_connect(smc); if (rc < 0) goto out; @@ -1116,7 +1111,7 @@ static void smc_listen_decline(struct smc_sock *new_smc, int reason_code, } /* listen worker: check prefixes */ -static int smc_listen_rdma_check(struct smc_sock *new_smc, +static int smc_listen_prfx_check(struct smc_sock *new_smc, struct smc_clc_msg_proposal *pclc) { struct smc_clc_msg_proposal_prefix *pclc_prfx; @@ -1124,25 +1119,21 @@ static int smc_listen_rdma_check(struct smc_sock *new_smc, pclc_prfx = smc_clc_proposal_get_prefix(pclc); if (smc_clc_prfx_match(newclcsock, pclc_prfx)) - return SMC_CLC_DECL_CNFERR; + return SMC_CLC_DECL_DIFFPREFIX; return 0; } /* listen worker: initialize connection and buffers */ static int smc_listen_rdma_init(struct smc_sock *new_smc, - struct smc_clc_msg_proposal *pclc, - struct smc_ib_device *ibdev, u8 ibport, - int *local_contact) + struct smc_init_info *ini) { + int rc; + /* allocate connection / link group */ - *local_contact = smc_conn_create(new_smc, false, 0, ibdev, ibport, 0, - &pclc->lcl, NULL, 0); - if (*local_contact < 0) { - if (*local_contact == -ENOMEM) - return SMC_CLC_DECL_MEM;/* insufficient memory*/ - return SMC_CLC_DECL_INTERR; /* other error */ - } + rc = smc_conn_create(new_smc, ini); + if (rc) + return rc; /* create send buffer and rmb */ if (smc_buf_create(new_smc, false)) @@ -1154,33 +1145,30 @@ static int smc_listen_rdma_init(struct smc_sock *new_smc, /* listen worker: initialize connection and buffers for SMC-D */ static int smc_listen_ism_init(struct smc_sock *new_smc, struct smc_clc_msg_proposal *pclc, - struct smcd_dev *ismdev, - int *local_contact) + struct smc_init_info *ini) { struct smc_clc_msg_smcd *pclc_smcd; + int rc; pclc_smcd = smc_get_clc_msg_smcd(pclc); - *local_contact = smc_conn_create(new_smc, true, 0, NULL, 0, 0, NULL, - ismdev, pclc_smcd->gid); - if (*local_contact < 0) { - if (*local_contact == -ENOMEM) - return SMC_CLC_DECL_MEM;/* insufficient memory*/ - return SMC_CLC_DECL_INTERR; /* other error */ - } + ini->ism_gid = pclc_smcd->gid; + rc = smc_conn_create(new_smc, ini); + if (rc) + return rc; /* Check if peer can be reached via ISM device */ if (smc_ism_cantalk(new_smc->conn.lgr->peer_gid, new_smc->conn.lgr->vlan_id, new_smc->conn.lgr->smcd)) { - if (*local_contact == SMC_FIRST_CONTACT) + if (ini->cln_first_contact == SMC_FIRST_CONTACT) smc_lgr_forget(new_smc->conn.lgr); smc_conn_free(&new_smc->conn); - return SMC_CLC_DECL_CNFERR; + return SMC_CLC_DECL_SMCDNOTALK; } /* Create send and receive buffers */ if (smc_buf_create(new_smc, true)) { - if (*local_contact == SMC_FIRST_CONTACT) + if (ini->cln_first_contact == SMC_FIRST_CONTACT) smc_lgr_forget(new_smc->conn.lgr); smc_conn_free(&new_smc->conn); return SMC_CLC_DECL_MEM; @@ -1244,15 +1232,10 @@ static void smc_listen_work(struct work_struct *work) struct socket *newclcsock = new_smc->clcsock; struct smc_clc_msg_accept_confirm cclc; struct smc_clc_msg_proposal *pclc; - struct smc_ib_device *ibdev; + struct smc_init_info ini = {0}; bool ism_supported = false; - struct smcd_dev *ismdev; u8 buf[SMC_CLC_MAX_LEN]; - int local_contact = 0; - unsigned short vlan; - int reason_code = 0; int rc = 0; - u8 ibport; if (new_smc->listen_smc->sk.sk_state != SMC_LISTEN) return smc_listen_out_err(new_smc); @@ -1274,17 +1257,26 @@ static void smc_listen_work(struct work_struct *work) * wait for and receive SMC Proposal CLC message */ pclc = (struct smc_clc_msg_proposal *)&buf; - reason_code = smc_clc_wait_msg(new_smc, pclc, SMC_CLC_MAX_LEN, - SMC_CLC_PROPOSAL, CLC_WAIT_TIME); - if (reason_code) { - smc_listen_decline(new_smc, reason_code, 0); - return; - } + rc = smc_clc_wait_msg(new_smc, pclc, SMC_CLC_MAX_LEN, + SMC_CLC_PROPOSAL, CLC_WAIT_TIME); + if (rc) + goto out_decl; /* IPSec connections opt out of SMC-R optimizations */ if (using_ipsec(new_smc)) { - smc_listen_decline(new_smc, SMC_CLC_DECL_IPSEC, 0); - return; + rc = SMC_CLC_DECL_IPSEC; + goto out_decl; + } + + /* check for matching IP prefix and subnet length */ + rc = smc_listen_prfx_check(new_smc, pclc); + if (rc) + goto out_decl; + + /* get vlan id from IP device */ + if (smc_vlan_by_tcpsk(new_smc->clcsock, &ini)) { + rc = SMC_CLC_DECL_GETVLANERR; + goto out_decl; } mutex_lock(&smc_server_lgr_pending); @@ -1293,59 +1285,73 @@ static void smc_listen_work(struct work_struct *work) smc_tx_init(new_smc); /* check if ISM is available */ - if ((pclc->hdr.path == SMC_TYPE_D || pclc->hdr.path == SMC_TYPE_B) && - !smc_check_ism(new_smc, &ismdev) && - !smc_listen_ism_init(new_smc, pclc, ismdev, &local_contact)) { - ism_supported = true; + if (pclc->hdr.path == SMC_TYPE_D || pclc->hdr.path == SMC_TYPE_B) { + ini.is_smcd = true; /* prepare ISM check */ + rc = smc_find_ism_device(new_smc, &ini); + if (!rc) + rc = smc_listen_ism_init(new_smc, pclc, &ini); + if (!rc) + ism_supported = true; + else if (pclc->hdr.path == SMC_TYPE_D) + goto out_unlock; /* skip RDMA and decline */ } /* check if RDMA is available */ - if (!ism_supported && - ((pclc->hdr.path != SMC_TYPE_R && pclc->hdr.path != SMC_TYPE_B) || - smc_vlan_by_tcpsk(new_smc->clcsock, &vlan) || - smc_check_rdma(new_smc, &ibdev, &ibport, vlan, NULL) || - smc_listen_rdma_check(new_smc, pclc) || - smc_listen_rdma_init(new_smc, pclc, ibdev, ibport, - &local_contact) || - smc_listen_rdma_reg(new_smc, local_contact))) { - /* SMC not supported, decline */ - mutex_unlock(&smc_server_lgr_pending); - smc_listen_decline(new_smc, SMC_CLC_DECL_MODEUNSUPP, - local_contact); - return; + if (!ism_supported) { /* SMC_TYPE_R or SMC_TYPE_B */ + /* prepare RDMA check */ + memset(&ini, 0, sizeof(ini)); + ini.is_smcd = false; + ini.ib_lcl = &pclc->lcl; + rc = smc_find_rdma_device(new_smc, &ini); + if (rc) { + /* no RDMA device found */ + if (pclc->hdr.path == SMC_TYPE_B) + /* neither ISM nor RDMA device found */ + rc = SMC_CLC_DECL_NOSMCDEV; + goto out_unlock; + } + rc = smc_listen_rdma_init(new_smc, &ini); + if (rc) + goto out_unlock; + rc = smc_listen_rdma_reg(new_smc, ini.cln_first_contact); + if (rc) + goto out_unlock; } /* send SMC Accept CLC message */ - rc = smc_clc_send_accept(new_smc, local_contact); - if (rc) { - mutex_unlock(&smc_server_lgr_pending); - smc_listen_decline(new_smc, rc, local_contact); - return; - } + rc = smc_clc_send_accept(new_smc, ini.cln_first_contact); + if (rc) + goto out_unlock; /* SMC-D does not need this lock any more */ if (ism_supported) mutex_unlock(&smc_server_lgr_pending); /* receive SMC Confirm CLC message */ - reason_code = smc_clc_wait_msg(new_smc, &cclc, sizeof(cclc), - SMC_CLC_CONFIRM, CLC_WAIT_TIME); - if (reason_code) { + rc = smc_clc_wait_msg(new_smc, &cclc, sizeof(cclc), + SMC_CLC_CONFIRM, CLC_WAIT_TIME); + if (rc) { if (!ism_supported) - mutex_unlock(&smc_server_lgr_pending); - smc_listen_decline(new_smc, reason_code, local_contact); - return; + goto out_unlock; + goto out_decl; } /* finish worker */ if (!ism_supported) { - rc = smc_listen_rdma_finish(new_smc, &cclc, local_contact); + rc = smc_listen_rdma_finish(new_smc, &cclc, + ini.cln_first_contact); mutex_unlock(&smc_server_lgr_pending); if (rc) return; } smc_conn_save_peer_info(new_smc, &cclc); smc_listen_out_connected(new_smc); + return; + +out_unlock: + mutex_unlock(&smc_server_lgr_pending); +out_decl: + smc_listen_decline(new_smc, rc, ini.cln_first_contact); } static void smc_tcp_listen_work(struct work_struct *work) @@ -1591,8 +1597,8 @@ static __poll_t smc_poll(struct file *file, struct socket *sock, poll_table *wait) { struct sock *sk = sock->sk; - __poll_t mask = 0; struct smc_sock *smc; + __poll_t mask = 0; if (!sk) return EPOLLNVAL; @@ -1602,8 +1608,6 @@ static __poll_t smc_poll(struct file *file, struct socket *sock, /* delegate to CLC child sock */ mask = smc->clcsock->ops->poll(file, smc->clcsock, wait); sk->sk_err = smc->clcsock->sk->sk_err; - if (sk->sk_err) - mask |= EPOLLERR; } else { if (sk->sk_state != SMC_CLOSED) sock_poll_wait(file, sock, wait); @@ -1614,9 +1618,14 @@ static __poll_t smc_poll(struct file *file, struct socket *sock, mask |= EPOLLHUP; if (sk->sk_state == SMC_LISTEN) { /* woken up by sk_data_ready in smc_listen_work() */ - mask = smc_accept_poll(sk); + mask |= smc_accept_poll(sk); + } else if (smc->use_fallback) { /* as result of connect_work()*/ + mask |= smc->clcsock->ops->poll(file, smc->clcsock, + wait); + sk->sk_err = smc->clcsock->sk->sk_err; } else { - if (atomic_read(&smc->conn.sndbuf_space) || + if ((sk->sk_state != SMC_INIT && + atomic_read(&smc->conn.sndbuf_space)) || sk->sk_shutdown & SEND_SHUTDOWN) { mask |= EPOLLOUT | EPOLLWRNORM; } else { diff --git a/net/smc/smc.h b/net/smc/smc.h index adbdf195eb085f5507444340fb1f8ddcb69548d5..878313f8d6c1772849d0877f50e4190d9fe2170b 100644 --- a/net/smc/smc.h +++ b/net/smc/smc.h @@ -190,18 +190,11 @@ struct smc_connection { u64 peer_token; /* SMC-D token of peer */ }; -struct smc_connect_info { - int flags; - int alen; - struct sockaddr addr; -}; - struct smc_sock { /* smc sock container */ struct sock sk; struct socket *clcsock; /* internal tcp socket */ struct smc_connection conn; /* smc connection */ struct smc_sock *listen_smc; /* listen parent */ - struct smc_connect_info *connect_info; /* connect address & flags */ struct work_struct connect_work; /* handle non-blocking connect*/ struct work_struct tcp_listen_work;/* handle tcp socket accepts */ struct work_struct smc_listen_work;/* prepare new accept socket */ @@ -219,6 +212,10 @@ struct smc_sock { /* smc sock container */ * started, waiting for unsent * data to be sent */ + u8 connect_nonblock : 1; + /* non-blocking connect in + * flight + */ struct mutex clcsock_release_lock; /* protects clcsock of a listen * socket diff --git a/net/smc/smc_clc.c b/net/smc/smc_clc.c index d53fd588d1f5a4450819934c83cf960dc29ff46b..745afd82f281186f2ca1b580081714893507595f 100644 --- a/net/smc/smc_clc.c +++ b/net/smc/smc_clc.c @@ -385,8 +385,7 @@ int smc_clc_send_decline(struct smc_sock *smc, u32 peer_diag_info) /* send CLC PROPOSAL message across internal TCP socket */ int smc_clc_send_proposal(struct smc_sock *smc, int smc_type, - struct smc_ib_device *ibdev, u8 ibport, u8 gid[], - struct smcd_dev *ismdev) + struct smc_init_info *ini) { struct smc_clc_ipv6_prefix ipv6_prfx[SMC_CLC_MAX_V6_PREFIX]; struct smc_clc_msg_proposal_prefix pclc_prfx; @@ -416,8 +415,9 @@ int smc_clc_send_proposal(struct smc_sock *smc, int smc_type, /* add SMC-R specifics */ memcpy(pclc.lcl.id_for_peer, local_systemid, sizeof(local_systemid)); - memcpy(&pclc.lcl.gid, gid, SMC_GID_SIZE); - memcpy(&pclc.lcl.mac, &ibdev->mac[ibport - 1], ETH_ALEN); + memcpy(&pclc.lcl.gid, ini->ib_gid, SMC_GID_SIZE); + memcpy(&pclc.lcl.mac, &ini->ib_dev->mac[ini->ib_port - 1], + ETH_ALEN); pclc.iparea_offset = htons(0); } if (smc_type == SMC_TYPE_D || smc_type == SMC_TYPE_B) { @@ -425,7 +425,7 @@ int smc_clc_send_proposal(struct smc_sock *smc, int smc_type, memset(&pclc_smcd, 0, sizeof(pclc_smcd)); plen += sizeof(pclc_smcd); pclc.iparea_offset = htons(SMC_CLC_PROPOSAL_MAX_OFFSET); - pclc_smcd.gid = ismdev->local_gid; + pclc_smcd.gid = ini->ism_dev->local_gid; } pclc.hdr.length = htons(plen); diff --git a/net/smc/smc_clc.h b/net/smc/smc_clc.h index 24658e8c0de42bfa182cd7379bb79850eb3936a5..ca209272e5fa698df643807174b1ce3501ed6215 100644 --- a/net/smc/smc_clc.h +++ b/net/smc/smc_clc.h @@ -34,16 +34,22 @@ #define SMC_CLC_DECL_CNFERR 0x03000000 /* configuration error */ #define SMC_CLC_DECL_PEERNOSMC 0x03010000 /* peer did not indicate SMC */ #define SMC_CLC_DECL_IPSEC 0x03020000 /* IPsec usage */ -#define SMC_CLC_DECL_NOSMCDEV 0x03030000 /* no SMC device found */ +#define SMC_CLC_DECL_NOSMCDEV 0x03030000 /* no SMC device found (R or D) */ +#define SMC_CLC_DECL_NOSMCDDEV 0x03030001 /* no SMC-D device found */ +#define SMC_CLC_DECL_NOSMCRDEV 0x03030002 /* no SMC-R device found */ +#define SMC_CLC_DECL_SMCDNOTALK 0x03030003 /* SMC-D dev can't talk to peer */ #define SMC_CLC_DECL_MODEUNSUPP 0x03040000 /* smc modes do not match (R or D)*/ #define SMC_CLC_DECL_RMBE_EC 0x03050000 /* peer has eyecatcher in RMBE */ #define SMC_CLC_DECL_OPTUNSUPP 0x03060000 /* fastopen sockopt not supported */ +#define SMC_CLC_DECL_DIFFPREFIX 0x03070000 /* IP prefix / subnet mismatch */ +#define SMC_CLC_DECL_GETVLANERR 0x03080000 /* err to get vlan id of ip device*/ +#define SMC_CLC_DECL_ISMVLANERR 0x03090000 /* err to reg vlan id on ism dev */ #define SMC_CLC_DECL_SYNCERR 0x04000000 /* synchronization error */ #define SMC_CLC_DECL_PEERDECL 0x05000000 /* peer declined during handshake */ -#define SMC_CLC_DECL_INTERR 0x99990000 /* internal error */ -#define SMC_CLC_DECL_ERR_RTOK 0x99990001 /* rtoken handling failed */ -#define SMC_CLC_DECL_ERR_RDYLNK 0x99990002 /* ib ready link failed */ -#define SMC_CLC_DECL_ERR_REGRMB 0x99990003 /* reg rmb failed */ +#define SMC_CLC_DECL_INTERR 0x09990000 /* internal error */ +#define SMC_CLC_DECL_ERR_RTOK 0x09990001 /* rtoken handling failed */ +#define SMC_CLC_DECL_ERR_RDYLNK 0x09990002 /* ib ready link failed */ +#define SMC_CLC_DECL_ERR_REGRMB 0x09990003 /* reg rmb failed */ struct smc_clc_msg_hdr { /* header1 of clc messages */ u8 eyecatcher[4]; /* eye catcher */ @@ -179,6 +185,7 @@ smc_get_clc_msg_smcd(struct smc_clc_msg_proposal *prop) } struct smcd_dev; +struct smc_init_info; int smc_clc_prfx_match(struct socket *clcsock, struct smc_clc_msg_proposal_prefix *prop); @@ -186,8 +193,7 @@ int smc_clc_wait_msg(struct smc_sock *smc, void *buf, int buflen, u8 expected_type, unsigned long timeout); int smc_clc_send_decline(struct smc_sock *smc, u32 peer_diag_info); int smc_clc_send_proposal(struct smc_sock *smc, int smc_type, - struct smc_ib_device *smcibdev, u8 ibport, u8 gid[], - struct smcd_dev *ismdev); + struct smc_init_info *ini); int smc_clc_send_confirm(struct smc_sock *smc); int smc_clc_send_accept(struct smc_sock *smc, int srv_first_contact); diff --git a/net/smc/smc_core.c b/net/smc/smc_core.c index 53a17cfa61af8d96e62fe11e9508584d71596061..2d2850adc2a3fa7a6a8667d79add5678beb130ca 100644 --- a/net/smc/smc_core.c +++ b/net/smc/smc_core.c @@ -195,10 +195,7 @@ static void smc_lgr_free_work(struct work_struct *work) } /* create a new SMC link group */ -static int smc_lgr_create(struct smc_sock *smc, bool is_smcd, - struct smc_ib_device *smcibdev, u8 ibport, - char *peer_systemid, unsigned short vlan_id, - struct smcd_dev *smcismdev, u64 peer_gid) +static int smc_lgr_create(struct smc_sock *smc, struct smc_init_info *ini) { struct smc_link_group *lgr; struct smc_link *lnk; @@ -206,20 +203,21 @@ static int smc_lgr_create(struct smc_sock *smc, bool is_smcd, int rc = 0; int i; - if (is_smcd && vlan_id) { - rc = smc_ism_get_vlan(smcismdev, vlan_id); - if (rc) + if (ini->is_smcd && ini->vlan_id) { + if (smc_ism_get_vlan(ini->ism_dev, ini->vlan_id)) { + rc = SMC_CLC_DECL_ISMVLANERR; goto out; + } } lgr = kzalloc(sizeof(*lgr), GFP_KERNEL); if (!lgr) { - rc = -ENOMEM; + rc = SMC_CLC_DECL_MEM; goto out; } - lgr->is_smcd = is_smcd; + lgr->is_smcd = ini->is_smcd; lgr->sync_err = 0; - lgr->vlan_id = vlan_id; + lgr->vlan_id = ini->vlan_id; rwlock_init(&lgr->sndbufs_lock); rwlock_init(&lgr->rmbs_lock); rwlock_init(&lgr->conns_lock); @@ -231,29 +229,32 @@ static int smc_lgr_create(struct smc_sock *smc, bool is_smcd, memcpy(&lgr->id, (u8 *)&smc_lgr_list.num, SMC_LGR_ID_SIZE); INIT_DELAYED_WORK(&lgr->free_work, smc_lgr_free_work); lgr->conns_all = RB_ROOT; - if (is_smcd) { + if (ini->is_smcd) { /* SMC-D specific settings */ - lgr->peer_gid = peer_gid; - lgr->smcd = smcismdev; + lgr->peer_gid = ini->ism_gid; + lgr->smcd = ini->ism_dev; } else { /* SMC-R specific settings */ lgr->role = smc->listen_smc ? SMC_SERV : SMC_CLNT; - memcpy(lgr->peer_systemid, peer_systemid, SMC_SYSTEMID_LEN); + memcpy(lgr->peer_systemid, ini->ib_lcl->id_for_peer, + SMC_SYSTEMID_LEN); lnk = &lgr->lnk[SMC_SINGLE_LINK]; /* initialize link */ lnk->state = SMC_LNK_ACTIVATING; lnk->link_id = SMC_SINGLE_LINK; - lnk->smcibdev = smcibdev; - lnk->ibport = ibport; - lnk->path_mtu = smcibdev->pattr[ibport - 1].active_mtu; - if (!smcibdev->initialized) - smc_ib_setup_per_ibdev(smcibdev); + lnk->smcibdev = ini->ib_dev; + lnk->ibport = ini->ib_port; + lnk->path_mtu = + ini->ib_dev->pattr[ini->ib_port - 1].active_mtu; + if (!ini->ib_dev->initialized) + smc_ib_setup_per_ibdev(ini->ib_dev); get_random_bytes(rndvec, sizeof(rndvec)); lnk->psn_initial = rndvec[0] + (rndvec[1] << 8) + (rndvec[2] << 16); rc = smc_ib_determine_gid(lnk->smcibdev, lnk->ibport, - vlan_id, lnk->gid, &lnk->sgid_index); + ini->vlan_id, lnk->gid, + &lnk->sgid_index); if (rc) goto free_lgr; rc = smc_llc_link_init(lnk); @@ -289,6 +290,12 @@ clear_llc_lnk: free_lgr: kfree(lgr); out: + if (rc < 0) { + if (rc == -ENOMEM) + rc = SMC_CLC_DECL_MEM; + else + rc = SMC_CLC_DECL_INTERR; + } return rc; } @@ -528,13 +535,13 @@ void smc_smcd_terminate(struct smcd_dev *dev, u64 peer_gid, unsigned short vlan) /* Determine vlan of internal TCP socket. * @vlan_id: address to store the determined vlan id into */ -int smc_vlan_by_tcpsk(struct socket *clcsock, unsigned short *vlan_id) +int smc_vlan_by_tcpsk(struct socket *clcsock, struct smc_init_info *ini) { struct dst_entry *dst = sk_dst_get(clcsock->sk); struct net_device *ndev; int i, nest_lvl, rc = 0; - *vlan_id = 0; + ini->vlan_id = 0; if (!dst) { rc = -ENOTCONN; goto out; @@ -546,7 +553,7 @@ int smc_vlan_by_tcpsk(struct socket *clcsock, unsigned short *vlan_id) ndev = dst->dev; if (is_vlan_dev(ndev)) { - *vlan_id = vlan_dev_vlan_id(ndev); + ini->vlan_id = vlan_dev_vlan_id(ndev); goto out_rel; } @@ -560,7 +567,7 @@ int smc_vlan_by_tcpsk(struct socket *clcsock, unsigned short *vlan_id) lower = lower->next; ndev = (struct net_device *)netdev_lower_get_next(ndev, &lower); if (is_vlan_dev(ndev)) { - *vlan_id = vlan_dev_vlan_id(ndev); + ini->vlan_id = vlan_dev_vlan_id(ndev); break; } } @@ -594,24 +601,16 @@ static bool smcd_lgr_match(struct smc_link_group *lgr, } /* create a new SMC connection (and a new link group if necessary) */ -int smc_conn_create(struct smc_sock *smc, bool is_smcd, int srv_first_contact, - struct smc_ib_device *smcibdev, u8 ibport, u32 clcqpn, - struct smc_clc_msg_local *lcl, struct smcd_dev *smcd, - u64 peer_gid) +int smc_conn_create(struct smc_sock *smc, struct smc_init_info *ini) { struct smc_connection *conn = &smc->conn; - int local_contact = SMC_FIRST_CONTACT; struct smc_link_group *lgr; - unsigned short vlan_id; enum smc_lgr_role role; int rc = 0; + ini->cln_first_contact = SMC_FIRST_CONTACT; role = smc->listen_smc ? SMC_SERV : SMC_CLNT; - rc = smc_vlan_by_tcpsk(smc->clcsock, &vlan_id); - if (rc) - return rc; - - if ((role == SMC_CLNT) && srv_first_contact) + if (role == SMC_CLNT && ini->srv_first_contact) /* create new link group as well */ goto create; @@ -619,14 +618,15 @@ int smc_conn_create(struct smc_sock *smc, bool is_smcd, int srv_first_contact, spin_lock_bh(&smc_lgr_list.lock); list_for_each_entry(lgr, &smc_lgr_list.list, list) { write_lock_bh(&lgr->conns_lock); - if ((is_smcd ? smcd_lgr_match(lgr, smcd, peer_gid) : - smcr_lgr_match(lgr, lcl, role, clcqpn)) && + if ((ini->is_smcd ? + smcd_lgr_match(lgr, ini->ism_dev, ini->ism_gid) : + smcr_lgr_match(lgr, ini->ib_lcl, role, ini->ib_clcqpn)) && !lgr->sync_err && - lgr->vlan_id == vlan_id && + lgr->vlan_id == ini->vlan_id && (role == SMC_CLNT || lgr->conns_num < SMC_RMBS_PER_LGR_MAX)) { /* link group found */ - local_contact = SMC_REUSE_CONTACT; + ini->cln_first_contact = SMC_REUSE_CONTACT; conn->lgr = lgr; smc_lgr_register_conn(conn); /* add smc conn to lgr */ if (delayed_work_pending(&lgr->free_work)) @@ -638,19 +638,18 @@ int smc_conn_create(struct smc_sock *smc, bool is_smcd, int srv_first_contact, } spin_unlock_bh(&smc_lgr_list.lock); - if (role == SMC_CLNT && !srv_first_contact && - (local_contact == SMC_FIRST_CONTACT)) { + if (role == SMC_CLNT && !ini->srv_first_contact && + ini->cln_first_contact == SMC_FIRST_CONTACT) { /* Server reuses a link group, but Client wants to start * a new one * send out_of_sync decline, reason synchr. error */ - return -ENOLINK; + return SMC_CLC_DECL_SYNCERR; } create: - if (local_contact == SMC_FIRST_CONTACT) { - rc = smc_lgr_create(smc, is_smcd, smcibdev, ibport, - lcl->id_for_peer, vlan_id, smcd, peer_gid); + if (ini->cln_first_contact == SMC_FIRST_CONTACT) { + rc = smc_lgr_create(smc, ini); if (rc) goto out; smc_lgr_register_conn(conn); /* add smc conn to lgr */ @@ -658,7 +657,7 @@ create: conn->local_tx_ctrl.common.type = SMC_CDC_MSG_TYPE; conn->local_tx_ctrl.len = SMC_WR_TX_SIZE; conn->urg_state = SMC_URG_READ; - if (is_smcd) { + if (ini->is_smcd) { conn->rx_off = sizeof(struct smcd_cdc_msg); smcd_cdc_rx_init(conn); /* init tasklet for this conn */ } @@ -667,7 +666,7 @@ create: #endif out: - return rc ? rc : local_contact; + return rc; } /* convert the RMB size into the compressed notation - minimum 16K. diff --git a/net/smc/smc_core.h b/net/smc/smc_core.h index 8806d2afa6edb1178f4ad2904aa1346ee42bcae6..c00ac61dc129f8c39351a5666ab706fa5d013d19 100644 --- a/net/smc/smc_core.h +++ b/net/smc/smc_core.h @@ -229,6 +229,24 @@ struct smc_link_group { }; }; +struct smc_clc_msg_local; + +struct smc_init_info { + u8 is_smcd; + unsigned short vlan_id; + int srv_first_contact; + int cln_first_contact; + /* SMC-R */ + struct smc_clc_msg_local *ib_lcl; + struct smc_ib_device *ib_dev; + u8 ib_gid[SMC_GID_SIZE]; + u8 ib_port; + u32 ib_clcqpn; + /* SMC-D */ + u64 ism_gid; + struct smcd_dev *ism_dev; +}; + /* Find the connection associated with the given alert token in the link group. * To use rbtrees we have to implement our own search core. * Requires @conns_lock @@ -281,13 +299,10 @@ void smc_sndbuf_sync_sg_for_cpu(struct smc_connection *conn); void smc_sndbuf_sync_sg_for_device(struct smc_connection *conn); void smc_rmb_sync_sg_for_cpu(struct smc_connection *conn); void smc_rmb_sync_sg_for_device(struct smc_connection *conn); -int smc_vlan_by_tcpsk(struct socket *clcsock, unsigned short *vlan_id); +int smc_vlan_by_tcpsk(struct socket *clcsock, struct smc_init_info *ini); void smc_conn_free(struct smc_connection *conn); -int smc_conn_create(struct smc_sock *smc, bool is_smcd, int srv_first_contact, - struct smc_ib_device *smcibdev, u8 ibport, u32 clcqpn, - struct smc_clc_msg_local *lcl, struct smcd_dev *smcd, - u64 peer_gid); +int smc_conn_create(struct smc_sock *smc, struct smc_init_info *ini); void smcd_conn_free(struct smc_connection *conn); void smc_lgr_schedule_free_work_fast(struct smc_link_group *lgr); void smc_core_exit(void); diff --git a/net/smc/smc_pnet.c b/net/smc/smc_pnet.c index 0285c7f9e79b6edb6a288be8bb50092a55bc7cfb..bab2da8cf17af875db0348a80b1d0488272f4990 100644 --- a/net/smc/smc_pnet.c +++ b/net/smc/smc_pnet.c @@ -26,6 +26,7 @@ #include "smc_pnet.h" #include "smc_ib.h" #include "smc_ism.h" +#include "smc_core.h" #define SMC_ASCII_BLANK 32 @@ -611,28 +612,28 @@ static int smc_pnet_flush(struct sk_buff *skb, struct genl_info *info) static const struct genl_ops smc_pnet_ops[] = { { .cmd = SMC_PNETID_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = smc_pnet_policy, .doit = smc_pnet_get, .dumpit = smc_pnet_dump, .start = smc_pnet_dump_start }, { .cmd = SMC_PNETID_ADD, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = smc_pnet_policy, .doit = smc_pnet_add }, { .cmd = SMC_PNETID_DEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = smc_pnet_policy, .doit = smc_pnet_del }, { .cmd = SMC_PNETID_FLUSH, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = smc_pnet_policy, .doit = smc_pnet_flush } }; @@ -643,6 +644,7 @@ static struct genl_family smc_pnet_nl_family __ro_after_init = { .name = SMCR_GENL_FAMILY_NAME, .version = SMCR_GENL_FAMILY_VERSION, .maxattr = SMC_PNETID_MAX, + .policy = smc_pnet_policy, .netnsok = true, .module = THIS_MODULE, .ops = smc_pnet_ops, @@ -759,8 +761,7 @@ static int smc_pnet_find_ndev_pnetid_by_table(struct net_device *ndev, * IB device and port */ static void smc_pnet_find_rdma_dev(struct net_device *netdev, - struct smc_ib_device **smcibdev, - u8 *ibport, unsigned short vlan_id, u8 gid[]) + struct smc_init_info *ini) { struct smc_ib_device *ibdev; @@ -780,10 +781,10 @@ static void smc_pnet_find_rdma_dev(struct net_device *netdev, dev_put(ndev); if (netdev == ndev && smc_ib_port_active(ibdev, i) && - !smc_ib_determine_gid(ibdev, i, vlan_id, gid, - NULL)) { - *smcibdev = ibdev; - *ibport = i; + !smc_ib_determine_gid(ibdev, i, ini->vlan_id, + ini->ib_gid, NULL)) { + ini->ib_dev = ibdev; + ini->ib_port = i; break; } } @@ -798,9 +799,7 @@ static void smc_pnet_find_rdma_dev(struct net_device *netdev, * If nothing found, try to use handshake device */ static void smc_pnet_find_roce_by_pnetid(struct net_device *ndev, - struct smc_ib_device **smcibdev, - u8 *ibport, unsigned short vlan_id, - u8 gid[]) + struct smc_init_info *ini) { u8 ndev_pnetid[SMC_MAX_PNETID_LEN]; struct smc_ib_device *ibdev; @@ -810,7 +809,7 @@ static void smc_pnet_find_roce_by_pnetid(struct net_device *ndev, if (smc_pnetid_by_dev_port(ndev->dev.parent, ndev->dev_port, ndev_pnetid) && smc_pnet_find_ndev_pnetid_by_table(ndev, ndev_pnetid)) { - smc_pnet_find_rdma_dev(ndev, smcibdev, ibport, vlan_id, gid); + smc_pnet_find_rdma_dev(ndev, ini); return; /* pnetid could not be determined */ } @@ -821,10 +820,10 @@ static void smc_pnet_find_roce_by_pnetid(struct net_device *ndev, continue; if (smc_pnet_match(ibdev->pnetid[i - 1], ndev_pnetid) && smc_ib_port_active(ibdev, i) && - !smc_ib_determine_gid(ibdev, i, vlan_id, gid, - NULL)) { - *smcibdev = ibdev; - *ibport = i; + !smc_ib_determine_gid(ibdev, i, ini->vlan_id, + ini->ib_gid, NULL)) { + ini->ib_dev = ibdev; + ini->ib_port = i; goto out; } } @@ -834,7 +833,7 @@ out: } static void smc_pnet_find_ism_by_pnetid(struct net_device *ndev, - struct smcd_dev **smcismdev) + struct smc_init_info *ini) { u8 ndev_pnetid[SMC_MAX_PNETID_LEN]; struct smcd_dev *ismdev; @@ -848,7 +847,7 @@ static void smc_pnet_find_ism_by_pnetid(struct net_device *ndev, spin_lock(&smcd_dev_list.lock); list_for_each_entry(ismdev, &smcd_dev_list.list, list) { if (smc_pnet_match(ismdev->pnetid, ndev_pnetid)) { - *smcismdev = ismdev; + ini->ism_dev = ismdev; break; } } @@ -859,21 +858,18 @@ static void smc_pnet_find_ism_by_pnetid(struct net_device *ndev, * determine ib_device and port belonging to used internal TCP socket * ethernet interface. */ -void smc_pnet_find_roce_resource(struct sock *sk, - struct smc_ib_device **smcibdev, u8 *ibport, - unsigned short vlan_id, u8 gid[]) +void smc_pnet_find_roce_resource(struct sock *sk, struct smc_init_info *ini) { struct dst_entry *dst = sk_dst_get(sk); - *smcibdev = NULL; - *ibport = 0; - + ini->ib_dev = NULL; + ini->ib_port = 0; if (!dst) goto out; if (!dst->dev) goto out_rel; - smc_pnet_find_roce_by_pnetid(dst->dev, smcibdev, ibport, vlan_id, gid); + smc_pnet_find_roce_by_pnetid(dst->dev, ini); out_rel: dst_release(dst); @@ -881,17 +877,17 @@ out: return; } -void smc_pnet_find_ism_resource(struct sock *sk, struct smcd_dev **smcismdev) +void smc_pnet_find_ism_resource(struct sock *sk, struct smc_init_info *ini) { struct dst_entry *dst = sk_dst_get(sk); - *smcismdev = NULL; + ini->ism_dev = NULL; if (!dst) goto out; if (!dst->dev) goto out_rel; - smc_pnet_find_ism_by_pnetid(dst->dev, smcismdev); + smc_pnet_find_ism_by_pnetid(dst->dev, ini); out_rel: dst_release(dst); diff --git a/net/smc/smc_pnet.h b/net/smc/smc_pnet.h index 5eac42fb45d06c98eaf7180c3af6bf21021a5a17..4564e4d69c2e563d3c519434f01c4fb203781242 100644 --- a/net/smc/smc_pnet.h +++ b/net/smc/smc_pnet.h @@ -18,6 +18,7 @@ struct smc_ib_device; struct smcd_dev; +struct smc_init_info; /** * struct smc_pnettable - SMC PNET table anchor @@ -43,9 +44,7 @@ int smc_pnet_init(void) __init; int smc_pnet_net_init(struct net *net); void smc_pnet_exit(void); void smc_pnet_net_exit(struct net *net); -void smc_pnet_find_roce_resource(struct sock *sk, - struct smc_ib_device **smcibdev, u8 *ibport, - unsigned short vlan_id, u8 gid[]); -void smc_pnet_find_ism_resource(struct sock *sk, struct smcd_dev **smcismdev); +void smc_pnet_find_roce_resource(struct sock *sk, struct smc_init_info *ini); +void smc_pnet_find_ism_resource(struct sock *sk, struct smc_init_info *ini); #endif diff --git a/net/socket.c b/net/socket.c index 8255f5bda0aa07dbeb78460d5bbf52a9f4510fd3..472fbefa5d9bbf080f96d6f1b88748fb056733a8 100644 --- a/net/socket.c +++ b/net/socket.c @@ -90,6 +90,7 @@ #include #include #include +#include #include #include @@ -108,6 +109,13 @@ #include #include +/* proto_ops for ipv4 and ipv6 use the same {recv,send}msg function */ +#if IS_ENABLED(CONFIG_INET) +#define INDIRECT_CALL_INET4(f, f1, ...) INDIRECT_CALL_1(f, f1, __VA_ARGS__) +#else +#define INDIRECT_CALL_INET4(f, f1, ...) f(__VA_ARGS__) +#endif + #ifdef CONFIG_NET_RX_BUSY_POLL unsigned int sysctl_net_busy_read __read_mostly; unsigned int sysctl_net_busy_poll __read_mostly; @@ -645,10 +653,12 @@ EXPORT_SYMBOL(__sock_tx_timestamp); * Sends @msg through @sock, passing through LSM. * Returns the number of bytes sent, or an error code. */ - +INDIRECT_CALLABLE_DECLARE(int inet_sendmsg(struct socket *, struct msghdr *, + size_t)); static inline int sock_sendmsg_nosec(struct socket *sock, struct msghdr *msg) { - int ret = sock->ops->sendmsg(sock, msg, msg_data_left(msg)); + int ret = INDIRECT_CALL_INET4(sock->ops->sendmsg, inet_sendmsg, sock, + msg, msg_data_left(msg)); BUG_ON(ret == -EIOCBQUEUED); return ret; } @@ -874,11 +884,13 @@ EXPORT_SYMBOL_GPL(__sock_recv_ts_and_drops); * Receives @msg from @sock, passing through LSM. Returns the total number * of bytes received, or an error. */ - +INDIRECT_CALLABLE_DECLARE(int inet_recvmsg(struct socket *, struct msghdr *, + size_t , int )); static inline int sock_recvmsg_nosec(struct socket *sock, struct msghdr *msg, int flags) { - return sock->ops->recvmsg(sock, msg, msg_data_left(msg), flags); + return INDIRECT_CALL_INET4(sock->ops->recvmsg, inet_recvmsg, sock, msg, + msg_data_left(msg), flags); } int sock_recvmsg(struct socket *sock, struct msghdr *msg, int flags) @@ -1164,6 +1176,26 @@ static long sock_ioctl(struct file *file, unsigned cmd, unsigned long arg) err = open_related_ns(&net->ns, get_net_ns); break; + case SIOCGSTAMP_OLD: + case SIOCGSTAMPNS_OLD: + if (!sock->ops->gettstamp) { + err = -ENOIOCTLCMD; + break; + } + err = sock->ops->gettstamp(sock, argp, + cmd == SIOCGSTAMP_OLD, + !IS_ENABLED(CONFIG_64BIT)); + break; + case SIOCGSTAMP_NEW: + case SIOCGSTAMPNS_NEW: + if (!sock->ops->gettstamp) { + err = -ENOIOCTLCMD; + break; + } + err = sock->ops->gettstamp(sock, argp, + cmd == SIOCGSTAMP_NEW, + false); + break; default: err = sock_do_ioctl(net, sock, cmd, arg); break; @@ -2916,38 +2948,6 @@ void socket_seq_show(struct seq_file *seq) #endif /* CONFIG_PROC_FS */ #ifdef CONFIG_COMPAT -static int do_siocgstamp(struct net *net, struct socket *sock, - unsigned int cmd, void __user *up) -{ - mm_segment_t old_fs = get_fs(); - struct timeval ktv; - int err; - - set_fs(KERNEL_DS); - err = sock_do_ioctl(net, sock, cmd, (unsigned long)&ktv); - set_fs(old_fs); - if (!err) - err = compat_put_timeval(&ktv, up); - - return err; -} - -static int do_siocgstampns(struct net *net, struct socket *sock, - unsigned int cmd, void __user *up) -{ - mm_segment_t old_fs = get_fs(); - struct timespec kts; - int err; - - set_fs(KERNEL_DS); - err = sock_do_ioctl(net, sock, cmd, (unsigned long)&kts); - set_fs(old_fs); - if (!err) - err = compat_put_timespec(&kts, up); - - return err; -} - static int compat_dev_ifconf(struct net *net, struct compat_ifconf __user *uifc32) { struct compat_ifconf ifc32; @@ -3347,10 +3347,13 @@ static int compat_sock_ioctl_trans(struct file *file, struct socket *sock, case SIOCADDRT: case SIOCDELRT: return routing_ioctl(net, sock, cmd, argp); - case SIOCGSTAMP: - return do_siocgstamp(net, sock, cmd, argp); - case SIOCGSTAMPNS: - return do_siocgstampns(net, sock, cmd, argp); + case SIOCGSTAMP_OLD: + case SIOCGSTAMPNS_OLD: + if (!sock->ops->gettstamp) + return -ENOIOCTLCMD; + return sock->ops->gettstamp(sock, argp, cmd == SIOCGSTAMP_OLD, + !COMPAT_USE_64BIT_TIME); + case SIOCBONDSLAVEINFOQUERY: case SIOCBONDINFOQUERY: case SIOCSHWTSTAMP: @@ -3368,6 +3371,8 @@ static int compat_sock_ioctl_trans(struct file *file, struct socket *sock, case SIOCADDDLCI: case SIOCDELDLCI: case SIOCGSKNS: + case SIOCGSTAMP_NEW: + case SIOCGSTAMPNS_NEW: return sock_ioctl(file, cmd, arg); case SIOCGIFFLAGS: diff --git a/net/strparser/strparser.c b/net/strparser/strparser.c index fa6c977b4c41a4a0b8deeb99c3e5d0d03c55de2b..e137698e8aef7bb246d6b8bd9c269881f2f19d5b 100644 --- a/net/strparser/strparser.c +++ b/net/strparser/strparser.c @@ -14,7 +14,8 @@ #include #include #include -#include +#include +#include #include #include #include @@ -297,7 +298,7 @@ static int __strp_recv(read_descriptor_t *desc, struct sk_buff *orig_skb, break; } - /* Positive extra indicates ore bytes than needed for the + /* Positive extra indicates more bytes than needed for the * message */ @@ -545,7 +546,7 @@ void strp_check_rcv(struct strparser *strp) } EXPORT_SYMBOL_GPL(strp_check_rcv); -static int __init strp_mod_init(void) +static int __init strp_dev_init(void) { strp_wq = create_singlethread_workqueue("kstrp"); if (unlikely(!strp_wq)) @@ -553,11 +554,4 @@ static int __init strp_mod_init(void) return 0; } - -static void __exit strp_mod_exit(void) -{ - destroy_workqueue(strp_wq); -} -module_init(strp_mod_init); -module_exit(strp_mod_exit); -MODULE_LICENSE("GPL"); +device_initcall(strp_dev_init); diff --git a/net/tipc/bcast.c b/net/tipc/bcast.c index d8026543bf4ce53634cb2d6fc114e57cdb74b21f..6c997d4a62189347650c43b9d2891a6883b0be6f 100644 --- a/net/tipc/bcast.c +++ b/net/tipc/bcast.c @@ -54,7 +54,9 @@ const char tipc_bclink_name[] = "broadcast-link"; * @dests: array keeping number of reachable destinations per bearer * @primary_bearer: a bearer having links to all broadcast destinations, if any * @bcast_support: indicates if primary bearer, if any, supports broadcast + * @force_bcast: forces broadcast for multicast traffic * @rcast_support: indicates if all peer nodes support replicast + * @force_rcast: forces replicast for multicast traffic * @rc_ratio: dest count as percentage of cluster size where send method changes * @bc_threshold: calculated from rc_ratio; if dests > threshold use broadcast */ @@ -64,7 +66,9 @@ struct tipc_bc_base { int dests[MAX_BEARERS]; int primary_bearer; bool bcast_support; + bool force_bcast; bool rcast_support; + bool force_rcast; int rc_ratio; int bc_threshold; }; @@ -216,9 +220,24 @@ static void tipc_bcast_select_xmit_method(struct net *net, int dests, } /* Can current method be changed ? */ method->expires = jiffies + TIPC_METHOD_EXPIRE; - if (method->mandatory || time_before(jiffies, exp)) + if (method->mandatory) return; + if (!(tipc_net(net)->capabilities & TIPC_MCAST_RBCTL) && + time_before(jiffies, exp)) + return; + + /* Configuration as force 'broadcast' method */ + if (bb->force_bcast) { + method->rcast = false; + return; + } + /* Configuration as force 'replicast' method */ + if (bb->force_rcast) { + method->rcast = true; + return; + } + /* Configuration as 'autoselect' or default method */ /* Determine method to use now */ method->rcast = dests <= bb->bc_threshold; } @@ -281,6 +300,63 @@ static int tipc_rcast_xmit(struct net *net, struct sk_buff_head *pkts, return 0; } +/* tipc_mcast_send_sync - deliver a dummy message with SYN bit + * @net: the applicable net namespace + * @skb: socket buffer to copy + * @method: send method to be used + * @dests: destination nodes for message. + * @cong_link_cnt: returns number of encountered congested destination links + * Returns 0 if success, otherwise errno + */ +static int tipc_mcast_send_sync(struct net *net, struct sk_buff *skb, + struct tipc_mc_method *method, + struct tipc_nlist *dests, + u16 *cong_link_cnt) +{ + struct tipc_msg *hdr, *_hdr; + struct sk_buff_head tmpq; + struct sk_buff *_skb; + + /* Is a cluster supporting with new capabilities ? */ + if (!(tipc_net(net)->capabilities & TIPC_MCAST_RBCTL)) + return 0; + + hdr = buf_msg(skb); + if (msg_user(hdr) == MSG_FRAGMENTER) + hdr = msg_get_wrapped(hdr); + if (msg_type(hdr) != TIPC_MCAST_MSG) + return 0; + + /* Allocate dummy message */ + _skb = tipc_buf_acquire(MCAST_H_SIZE, GFP_KERNEL); + if (!_skb) + return -ENOMEM; + + /* Preparing for 'synching' header */ + msg_set_syn(hdr, 1); + + /* Copy skb's header into a dummy header */ + skb_copy_to_linear_data(_skb, hdr, MCAST_H_SIZE); + skb_orphan(_skb); + + /* Reverse method for dummy message */ + _hdr = buf_msg(_skb); + msg_set_size(_hdr, MCAST_H_SIZE); + msg_set_is_rcast(_hdr, !msg_is_rcast(hdr)); + + skb_queue_head_init(&tmpq); + __skb_queue_tail(&tmpq, _skb); + if (method->rcast) + tipc_bcast_xmit(net, &tmpq, cong_link_cnt); + else + tipc_rcast_xmit(net, &tmpq, dests, cong_link_cnt); + + /* This queue should normally be empty by now */ + __skb_queue_purge(&tmpq); + + return 0; +} + /* tipc_mcast_xmit - deliver message to indicated destination nodes * and to identified node local sockets * @net: the applicable net namespace @@ -296,6 +372,9 @@ int tipc_mcast_xmit(struct net *net, struct sk_buff_head *pkts, u16 *cong_link_cnt) { struct sk_buff_head inputq, localq; + bool rcast = method->rcast; + struct tipc_msg *hdr; + struct sk_buff *skb; int rc = 0; skb_queue_head_init(&inputq); @@ -309,6 +388,18 @@ int tipc_mcast_xmit(struct net *net, struct sk_buff_head *pkts, /* Send according to determined transmit method */ if (dests->remote) { tipc_bcast_select_xmit_method(net, dests->remote, method); + + skb = skb_peek(pkts); + hdr = buf_msg(skb); + if (msg_user(hdr) == MSG_FRAGMENTER) + hdr = msg_get_wrapped(hdr); + msg_set_is_rcast(hdr, method->rcast); + + /* Switch method ? */ + if (rcast != method->rcast) + tipc_mcast_send_sync(net, skb, method, + dests, cong_link_cnt); + if (method->rcast) rc = tipc_rcast_xmit(net, pkts, dests, cong_link_cnt); else @@ -485,10 +576,63 @@ static int tipc_bc_link_set_queue_limits(struct net *net, u32 limit) return 0; } +static int tipc_bc_link_set_broadcast_mode(struct net *net, u32 bc_mode) +{ + struct tipc_bc_base *bb = tipc_bc_base(net); + + switch (bc_mode) { + case BCLINK_MODE_BCAST: + if (!bb->bcast_support) + return -ENOPROTOOPT; + + bb->force_bcast = true; + bb->force_rcast = false; + break; + case BCLINK_MODE_RCAST: + if (!bb->rcast_support) + return -ENOPROTOOPT; + + bb->force_bcast = false; + bb->force_rcast = true; + break; + case BCLINK_MODE_SEL: + if (!bb->bcast_support || !bb->rcast_support) + return -ENOPROTOOPT; + + bb->force_bcast = false; + bb->force_rcast = false; + break; + default: + return -EINVAL; + } + + return 0; +} + +static int tipc_bc_link_set_broadcast_ratio(struct net *net, u32 bc_ratio) +{ + struct tipc_bc_base *bb = tipc_bc_base(net); + + if (!bb->bcast_support || !bb->rcast_support) + return -ENOPROTOOPT; + + if (bc_ratio > 100 || bc_ratio <= 0) + return -EINVAL; + + bb->rc_ratio = bc_ratio; + tipc_bcast_lock(net); + tipc_bcbase_calc_bc_threshold(net); + tipc_bcast_unlock(net); + + return 0; +} + int tipc_nl_bc_link_set(struct net *net, struct nlattr *attrs[]) { int err; u32 win; + u32 bc_mode; + u32 bc_ratio; struct nlattr *props[TIPC_NLA_PROP_MAX + 1]; if (!attrs[TIPC_NLA_LINK_PROP]) @@ -498,12 +642,28 @@ int tipc_nl_bc_link_set(struct net *net, struct nlattr *attrs[]) if (err) return err; - if (!props[TIPC_NLA_PROP_WIN]) + if (!props[TIPC_NLA_PROP_WIN] && + !props[TIPC_NLA_PROP_BROADCAST] && + !props[TIPC_NLA_PROP_BROADCAST_RATIO]) { return -EOPNOTSUPP; + } + + if (props[TIPC_NLA_PROP_BROADCAST]) { + bc_mode = nla_get_u32(props[TIPC_NLA_PROP_BROADCAST]); + err = tipc_bc_link_set_broadcast_mode(net, bc_mode); + } + + if (!err && props[TIPC_NLA_PROP_BROADCAST_RATIO]) { + bc_ratio = nla_get_u32(props[TIPC_NLA_PROP_BROADCAST_RATIO]); + err = tipc_bc_link_set_broadcast_ratio(net, bc_ratio); + } - win = nla_get_u32(props[TIPC_NLA_PROP_WIN]); + if (!err && props[TIPC_NLA_PROP_WIN]) { + win = nla_get_u32(props[TIPC_NLA_PROP_WIN]); + err = tipc_bc_link_set_queue_limits(net, win); + } - return tipc_bc_link_set_queue_limits(net, win); + return err; } int tipc_bcast_init(struct net *net) @@ -529,7 +689,7 @@ int tipc_bcast_init(struct net *net) goto enomem; bb->link = l; tn->bcl = l; - bb->rc_ratio = 25; + bb->rc_ratio = 10; bb->rcast_support = true; return 0; enomem: @@ -576,3 +736,108 @@ void tipc_nlist_purge(struct tipc_nlist *nl) nl->remote = 0; nl->local = false; } + +u32 tipc_bcast_get_broadcast_mode(struct net *net) +{ + struct tipc_bc_base *bb = tipc_bc_base(net); + + if (bb->force_bcast) + return BCLINK_MODE_BCAST; + + if (bb->force_rcast) + return BCLINK_MODE_RCAST; + + if (bb->bcast_support && bb->rcast_support) + return BCLINK_MODE_SEL; + + return 0; +} + +u32 tipc_bcast_get_broadcast_ratio(struct net *net) +{ + struct tipc_bc_base *bb = tipc_bc_base(net); + + return bb->rc_ratio; +} + +void tipc_mcast_filter_msg(struct net *net, struct sk_buff_head *defq, + struct sk_buff_head *inputq) +{ + struct sk_buff *skb, *_skb, *tmp; + struct tipc_msg *hdr, *_hdr; + bool match = false; + u32 node, port; + + skb = skb_peek(inputq); + if (!skb) + return; + + hdr = buf_msg(skb); + + if (likely(!msg_is_syn(hdr) && skb_queue_empty(defq))) + return; + + node = msg_orignode(hdr); + if (node == tipc_own_addr(net)) + return; + + port = msg_origport(hdr); + + /* Has the twin SYN message already arrived ? */ + skb_queue_walk(defq, _skb) { + _hdr = buf_msg(_skb); + if (msg_orignode(_hdr) != node) + continue; + if (msg_origport(_hdr) != port) + continue; + match = true; + break; + } + + if (!match) { + if (!msg_is_syn(hdr)) + return; + __skb_dequeue(inputq); + __skb_queue_tail(defq, skb); + return; + } + + /* Deliver non-SYN message from other link, otherwise queue it */ + if (!msg_is_syn(hdr)) { + if (msg_is_rcast(hdr) != msg_is_rcast(_hdr)) + return; + __skb_dequeue(inputq); + __skb_queue_tail(defq, skb); + return; + } + + /* Queue non-SYN/SYN message from same link */ + if (msg_is_rcast(hdr) == msg_is_rcast(_hdr)) { + __skb_dequeue(inputq); + __skb_queue_tail(defq, skb); + return; + } + + /* Matching SYN messages => return the one with data, if any */ + __skb_unlink(_skb, defq); + if (msg_data_sz(hdr)) { + kfree_skb(_skb); + } else { + __skb_dequeue(inputq); + kfree_skb(skb); + __skb_queue_tail(inputq, _skb); + } + + /* Deliver subsequent non-SYN messages from same peer */ + skb_queue_walk_safe(defq, _skb, tmp) { + _hdr = buf_msg(_skb); + if (msg_orignode(_hdr) != node) + continue; + if (msg_origport(_hdr) != port) + continue; + if (msg_is_syn(_hdr)) + break; + __skb_unlink(_skb, defq); + __skb_queue_tail(inputq, _skb); + } +} diff --git a/net/tipc/bcast.h b/net/tipc/bcast.h index 751530ab0c494f0b84c2c6d40fd6db20773f5367..dadad953e2bef5bb88483a22f902e3289b921e13 100644 --- a/net/tipc/bcast.h +++ b/net/tipc/bcast.h @@ -48,6 +48,10 @@ extern const char tipc_bclink_name[]; #define TIPC_METHOD_EXPIRE msecs_to_jiffies(5000) +#define BCLINK_MODE_BCAST 0x1 +#define BCLINK_MODE_RCAST 0x2 +#define BCLINK_MODE_SEL 0x4 + struct tipc_nlist { struct list_head list; u32 self; @@ -63,11 +67,13 @@ void tipc_nlist_del(struct tipc_nlist *nl, u32 node); /* Cookie to be used between socket and broadcast layer * @rcast: replicast (instead of broadcast) was used at previous xmit * @mandatory: broadcast/replicast indication was set by user + * @deferredq: defer queue to make message in order * @expires: re-evaluate non-mandatory transmit method if we are past this */ struct tipc_mc_method { bool rcast; bool mandatory; + struct sk_buff_head deferredq; unsigned long expires; }; @@ -92,6 +98,12 @@ int tipc_nl_add_bc_link(struct net *net, struct tipc_nl_msg *msg); int tipc_nl_bc_link_set(struct net *net, struct nlattr *attrs[]); int tipc_bclink_reset_stats(struct net *net); +u32 tipc_bcast_get_broadcast_mode(struct net *net); +u32 tipc_bcast_get_broadcast_ratio(struct net *net); + +void tipc_mcast_filter_msg(struct net *net, struct sk_buff_head *defq, + struct sk_buff_head *inputq); + static inline void tipc_bcast_lock(struct net *net) { spin_lock_bh(&tipc_net(net)->bclock); diff --git a/net/tipc/bearer.c b/net/tipc/bearer.c index d27f30a9a01d8d34b3f46d6fc12f2d192d6f3494..2bed6589f41ecab6da76c118975a6a509018519e 100644 --- a/net/tipc/bearer.c +++ b/net/tipc/bearer.c @@ -687,14 +687,14 @@ static int __tipc_nl_add_bearer(struct tipc_nl_msg *msg, if (!hdr) return -EMSGSIZE; - attrs = nla_nest_start(msg->skb, TIPC_NLA_BEARER); + attrs = nla_nest_start_noflag(msg->skb, TIPC_NLA_BEARER); if (!attrs) goto msg_full; if (nla_put_string(msg->skb, TIPC_NLA_BEARER_NAME, bearer->name)) goto attr_msg_full; - prop = nla_nest_start(msg->skb, TIPC_NLA_BEARER_PROP); + prop = nla_nest_start_noflag(msg->skb, TIPC_NLA_BEARER_PROP); if (!prop) goto prop_msg_full; if (nla_put_u32(msg->skb, TIPC_NLA_PROP_PRIO, bearer->priority)) @@ -776,9 +776,9 @@ int tipc_nl_bearer_get(struct sk_buff *skb, struct genl_info *info) if (!info->attrs[TIPC_NLA_BEARER]) return -EINVAL; - err = nla_parse_nested(attrs, TIPC_NLA_BEARER_MAX, - info->attrs[TIPC_NLA_BEARER], - tipc_nl_bearer_policy, info->extack); + err = nla_parse_nested_deprecated(attrs, TIPC_NLA_BEARER_MAX, + info->attrs[TIPC_NLA_BEARER], + tipc_nl_bearer_policy, info->extack); if (err) return err; @@ -825,9 +825,9 @@ int __tipc_nl_bearer_disable(struct sk_buff *skb, struct genl_info *info) if (!info->attrs[TIPC_NLA_BEARER]) return -EINVAL; - err = nla_parse_nested(attrs, TIPC_NLA_BEARER_MAX, - info->attrs[TIPC_NLA_BEARER], - tipc_nl_bearer_policy, info->extack); + err = nla_parse_nested_deprecated(attrs, TIPC_NLA_BEARER_MAX, + info->attrs[TIPC_NLA_BEARER], + tipc_nl_bearer_policy, info->extack); if (err) return err; @@ -870,9 +870,9 @@ int __tipc_nl_bearer_enable(struct sk_buff *skb, struct genl_info *info) if (!info->attrs[TIPC_NLA_BEARER]) return -EINVAL; - err = nla_parse_nested(attrs, TIPC_NLA_BEARER_MAX, - info->attrs[TIPC_NLA_BEARER], - tipc_nl_bearer_policy, info->extack); + err = nla_parse_nested_deprecated(attrs, TIPC_NLA_BEARER_MAX, + info->attrs[TIPC_NLA_BEARER], + tipc_nl_bearer_policy, info->extack); if (err) return err; @@ -921,9 +921,9 @@ int tipc_nl_bearer_add(struct sk_buff *skb, struct genl_info *info) if (!info->attrs[TIPC_NLA_BEARER]) return -EINVAL; - err = nla_parse_nested(attrs, TIPC_NLA_BEARER_MAX, - info->attrs[TIPC_NLA_BEARER], - tipc_nl_bearer_policy, info->extack); + err = nla_parse_nested_deprecated(attrs, TIPC_NLA_BEARER_MAX, + info->attrs[TIPC_NLA_BEARER], + tipc_nl_bearer_policy, info->extack); if (err) return err; @@ -964,9 +964,9 @@ int __tipc_nl_bearer_set(struct sk_buff *skb, struct genl_info *info) if (!info->attrs[TIPC_NLA_BEARER]) return -EINVAL; - err = nla_parse_nested(attrs, TIPC_NLA_BEARER_MAX, - info->attrs[TIPC_NLA_BEARER], - tipc_nl_bearer_policy, info->extack); + err = nla_parse_nested_deprecated(attrs, TIPC_NLA_BEARER_MAX, + info->attrs[TIPC_NLA_BEARER], + tipc_nl_bearer_policy, info->extack); if (err) return err; @@ -1033,14 +1033,14 @@ static int __tipc_nl_add_media(struct tipc_nl_msg *msg, if (!hdr) return -EMSGSIZE; - attrs = nla_nest_start(msg->skb, TIPC_NLA_MEDIA); + attrs = nla_nest_start_noflag(msg->skb, TIPC_NLA_MEDIA); if (!attrs) goto msg_full; if (nla_put_string(msg->skb, TIPC_NLA_MEDIA_NAME, media->name)) goto attr_msg_full; - prop = nla_nest_start(msg->skb, TIPC_NLA_MEDIA_PROP); + prop = nla_nest_start_noflag(msg->skb, TIPC_NLA_MEDIA_PROP); if (!prop) goto prop_msg_full; if (nla_put_u32(msg->skb, TIPC_NLA_PROP_PRIO, media->priority)) @@ -1107,9 +1107,9 @@ int tipc_nl_media_get(struct sk_buff *skb, struct genl_info *info) if (!info->attrs[TIPC_NLA_MEDIA]) return -EINVAL; - err = nla_parse_nested(attrs, TIPC_NLA_MEDIA_MAX, - info->attrs[TIPC_NLA_MEDIA], - tipc_nl_media_policy, info->extack); + err = nla_parse_nested_deprecated(attrs, TIPC_NLA_MEDIA_MAX, + info->attrs[TIPC_NLA_MEDIA], + tipc_nl_media_policy, info->extack); if (err) return err; @@ -1155,9 +1155,9 @@ int __tipc_nl_media_set(struct sk_buff *skb, struct genl_info *info) if (!info->attrs[TIPC_NLA_MEDIA]) return -EINVAL; - err = nla_parse_nested(attrs, TIPC_NLA_MEDIA_MAX, - info->attrs[TIPC_NLA_MEDIA], - tipc_nl_media_policy, info->extack); + err = nla_parse_nested_deprecated(attrs, TIPC_NLA_MEDIA_MAX, + info->attrs[TIPC_NLA_MEDIA], + tipc_nl_media_policy, info->extack); if (!attrs[TIPC_NLA_MEDIA_NAME]) return -EINVAL; diff --git a/net/tipc/core.c b/net/tipc/core.c index 5b38f516428147d6e2f1e03a1d3f9e2bfe9a769f..27cccd101ef663476f0b582487eae9d79533994e 100644 --- a/net/tipc/core.c +++ b/net/tipc/core.c @@ -43,6 +43,7 @@ #include "net.h" #include "socket.h" #include "bcast.h" +#include "node.h" #include @@ -59,6 +60,7 @@ static int __net_init tipc_init_net(struct net *net) tn->node_addr = 0; tn->trial_addr = 0; tn->addr_trial_end = 0; + tn->capabilities = TIPC_NODE_CAPABILITIES; memset(tn->node_id, 0, sizeof(tn->node_id)); memset(tn->node_id_string, 0, sizeof(tn->node_id_string)); tn->mon_threshold = TIPC_DEF_MON_THRESHOLD; diff --git a/net/tipc/core.h b/net/tipc/core.h index 8020a6c360ffd2eeef2430308f2ba3b13b5d13cc..7a68e1b6a066922b54bbc50c3ad1bff7e65c0973 100644 --- a/net/tipc/core.h +++ b/net/tipc/core.h @@ -122,6 +122,9 @@ struct tipc_net { /* Topology subscription server */ struct tipc_topsrv *topsrv; atomic_t subscription_count; + + /* Cluster capabilities */ + u16 capabilities; }; static inline struct tipc_net *tipc_net(struct net *net) diff --git a/net/tipc/group.c b/net/tipc/group.c index 63f39201e41e699104d838f206d43f7f24806d3a..992be6113676988291157fb81c65b5ad605e522b 100644 --- a/net/tipc/group.c +++ b/net/tipc/group.c @@ -917,7 +917,7 @@ void tipc_group_member_evt(struct tipc_group *grp, int tipc_group_fill_sock_diag(struct tipc_group *grp, struct sk_buff *skb) { - struct nlattr *group = nla_nest_start(skb, TIPC_NLA_SOCK_GROUP); + struct nlattr *group = nla_nest_start_noflag(skb, TIPC_NLA_SOCK_GROUP); if (!group) return -EMSGSIZE; diff --git a/net/tipc/link.c b/net/tipc/link.c index 131aa2f0fd27c46e14f024b317dd65c786b0bea4..f5cd986e1e504229936ebe097997fd34556c2ba4 100644 --- a/net/tipc/link.c +++ b/net/tipc/link.c @@ -151,6 +151,7 @@ struct tipc_link { /* Failover/synch */ u16 drop_point; struct sk_buff *failover_reasm_skb; + struct sk_buff_head failover_deferdq; /* Max packet negotiation */ u16 mtu; @@ -209,6 +210,7 @@ enum { }; #define TIPC_BC_RETR_LIM msecs_to_jiffies(10) /* [ms] */ +#define TIPC_UC_RETR_TIME (jiffies + msecs_to_jiffies(1)) /* * Interval between NACKs when packets arrive out of order @@ -246,6 +248,10 @@ static int tipc_link_build_nack_msg(struct tipc_link *l, static void tipc_link_build_bc_init_msg(struct tipc_link *l, struct sk_buff_head *xmitq); static bool tipc_link_release_pkts(struct tipc_link *l, u16 to); +static u16 tipc_build_gap_ack_blks(struct tipc_link *l, void *data); +static void tipc_link_advance_transmq(struct tipc_link *l, u16 acked, u16 gap, + struct tipc_gap_ack_blks *ga, + struct sk_buff_head *xmitq); /* * Simple non-static link routines (i.e. referenced outside this file) @@ -493,6 +499,7 @@ bool tipc_link_create(struct net *net, char *if_name, int bearer_id, __skb_queue_head_init(&l->transmq); __skb_queue_head_init(&l->backlogq); __skb_queue_head_init(&l->deferdq); + __skb_queue_head_init(&l->failover_deferdq); skb_queue_head_init(&l->wakeupq); skb_queue_head_init(l->inputq); return true; @@ -885,6 +892,7 @@ void tipc_link_reset(struct tipc_link *l) __skb_queue_purge(&l->transmq); __skb_queue_purge(&l->deferdq); __skb_queue_purge(&l->backlogq); + __skb_queue_purge(&l->failover_deferdq); l->backlog[TIPC_LOW_IMPORTANCE].len = 0; l->backlog[TIPC_MEDIUM_IMPORTANCE].len = 0; l->backlog[TIPC_HIGH_IMPORTANCE].len = 0; @@ -1156,34 +1164,14 @@ static bool tipc_data_input(struct tipc_link *l, struct sk_buff *skb, * Consumes buffer */ static int tipc_link_input(struct tipc_link *l, struct sk_buff *skb, - struct sk_buff_head *inputq) + struct sk_buff_head *inputq, + struct sk_buff **reasm_skb) { struct tipc_msg *hdr = buf_msg(skb); - struct sk_buff **reasm_skb = &l->reasm_buf; struct sk_buff *iskb; struct sk_buff_head tmpq; int usr = msg_user(hdr); - int rc = 0; int pos = 0; - int ipos = 0; - - if (unlikely(usr == TUNNEL_PROTOCOL)) { - if (msg_type(hdr) == SYNCH_MSG) { - __skb_queue_purge(&l->deferdq); - goto drop; - } - if (!tipc_msg_extract(skb, &iskb, &ipos)) - return rc; - kfree_skb(skb); - skb = iskb; - hdr = buf_msg(skb); - if (less(msg_seqno(hdr), l->drop_point)) - goto drop; - if (tipc_data_input(l, skb, inputq)) - return rc; - usr = msg_user(hdr); - reasm_skb = &l->failover_reasm_skb; - } if (usr == MSG_BUNDLER) { skb_queue_head_init(&tmpq); @@ -1208,11 +1196,66 @@ static int tipc_link_input(struct tipc_link *l, struct sk_buff *skb, tipc_link_bc_init_rcv(l->bc_rcvlink, hdr); tipc_bcast_unlock(l->net); } -drop: + kfree_skb(skb); return 0; } +/* tipc_link_tnl_rcv() - receive TUNNEL_PROTOCOL message, drop or process the + * inner message along with the ones in the old link's + * deferdq + * @l: tunnel link + * @skb: TUNNEL_PROTOCOL message + * @inputq: queue to put messages ready for delivery + */ +static int tipc_link_tnl_rcv(struct tipc_link *l, struct sk_buff *skb, + struct sk_buff_head *inputq) +{ + struct sk_buff **reasm_skb = &l->failover_reasm_skb; + struct sk_buff_head *fdefq = &l->failover_deferdq; + struct tipc_msg *hdr = buf_msg(skb); + struct sk_buff *iskb; + int ipos = 0; + int rc = 0; + u16 seqno; + + /* SYNCH_MSG */ + if (msg_type(hdr) == SYNCH_MSG) + goto drop; + + /* FAILOVER_MSG */ + if (!tipc_msg_extract(skb, &iskb, &ipos)) { + pr_warn_ratelimited("Cannot extract FAILOVER_MSG, defq: %d\n", + skb_queue_len(fdefq)); + return rc; + } + + do { + seqno = buf_seqno(iskb); + + if (unlikely(less(seqno, l->drop_point))) { + kfree_skb(iskb); + continue; + } + + if (unlikely(seqno != l->drop_point)) { + __tipc_skb_queue_sorted(fdefq, seqno, iskb); + continue; + } + + l->drop_point++; + + if (!tipc_data_input(l, iskb, inputq)) + rc |= tipc_link_input(l, iskb, inputq, reasm_skb); + if (unlikely(rc)) + break; + } while ((iskb = __tipc_skb_dequeue(fdefq, l->drop_point))); + +drop: + kfree_skb(skb); + return rc; +} + static bool tipc_link_release_pkts(struct tipc_link *l, u16 acked) { bool released = false; @@ -1228,6 +1271,106 @@ static bool tipc_link_release_pkts(struct tipc_link *l, u16 acked) return released; } +/* tipc_build_gap_ack_blks - build Gap ACK blocks + * @l: tipc link that data have come with gaps in sequence if any + * @data: data buffer to store the Gap ACK blocks after built + * + * returns the actual allocated memory size + */ +static u16 tipc_build_gap_ack_blks(struct tipc_link *l, void *data) +{ + struct sk_buff *skb = skb_peek(&l->deferdq); + struct tipc_gap_ack_blks *ga = data; + u16 len, expect, seqno = 0; + u8 n = 0; + + if (!skb) + goto exit; + + expect = buf_seqno(skb); + skb_queue_walk(&l->deferdq, skb) { + seqno = buf_seqno(skb); + if (unlikely(more(seqno, expect))) { + ga->gacks[n].ack = htons(expect - 1); + ga->gacks[n].gap = htons(seqno - expect); + if (++n >= MAX_GAP_ACK_BLKS) { + pr_info_ratelimited("Too few Gap ACK blocks!\n"); + goto exit; + } + } else if (unlikely(less(seqno, expect))) { + pr_warn("Unexpected skb in deferdq!\n"); + continue; + } + expect = seqno + 1; + } + + /* last block */ + ga->gacks[n].ack = htons(seqno); + ga->gacks[n].gap = 0; + n++; + +exit: + len = tipc_gap_ack_blks_sz(n); + ga->len = htons(len); + ga->gack_cnt = n; + return len; +} + +/* tipc_link_advance_transmq - advance TIPC link transmq queue by releasing + * acked packets, also doing retransmissions if + * gaps found + * @l: tipc link with transmq queue to be advanced + * @acked: seqno of last packet acked by peer without any gaps before + * @gap: # of gap packets + * @ga: buffer pointer to Gap ACK blocks from peer + * @xmitq: queue for accumulating the retransmitted packets if any + */ +static void tipc_link_advance_transmq(struct tipc_link *l, u16 acked, u16 gap, + struct tipc_gap_ack_blks *ga, + struct sk_buff_head *xmitq) +{ + struct sk_buff *skb, *_skb, *tmp; + struct tipc_msg *hdr; + u16 bc_ack = l->bc_rcvlink->rcv_nxt - 1; + u16 ack = l->rcv_nxt - 1; + u16 seqno; + u16 n = 0; + + skb_queue_walk_safe(&l->transmq, skb, tmp) { + seqno = buf_seqno(skb); + +next_gap_ack: + if (less_eq(seqno, acked)) { + /* release skb */ + __skb_unlink(skb, &l->transmq); + kfree_skb(skb); + } else if (less_eq(seqno, acked + gap)) { + /* retransmit skb */ + if (time_before(jiffies, TIPC_SKB_CB(skb)->nxt_retr)) + continue; + TIPC_SKB_CB(skb)->nxt_retr = TIPC_UC_RETR_TIME; + + _skb = __pskb_copy(skb, MIN_H_SIZE, GFP_ATOMIC); + if (!_skb) + continue; + hdr = buf_msg(_skb); + msg_set_ack(hdr, ack); + msg_set_bcast_ack(hdr, bc_ack); + _skb->priority = TC_PRIO_CONTROL; + __skb_queue_tail(xmitq, _skb); + l->stats.retransmitted++; + } else { + /* retry with Gap ACK blocks if any */ + if (!ga || n >= ga->gack_cnt) + break; + acked = ntohs(ga->gacks[n].ack); + gap = ntohs(ga->gacks[n].gap); + n++; + goto next_gap_ack; + } + } +} + /* tipc_link_build_state_msg: prepare link state message for transmission * * Note that sending of broadcast ack is coordinated among nodes, to reduce @@ -1282,6 +1425,7 @@ static int tipc_link_build_nack_msg(struct tipc_link *l, struct sk_buff_head *xmitq) { u32 def_cnt = ++l->stats.deferred_recv; + u32 defq_len = skb_queue_len(&l->deferdq); int match1, match2; if (link_is_bc_rcvlink(l)) { @@ -1292,7 +1436,7 @@ static int tipc_link_build_nack_msg(struct tipc_link *l, return 0; } - if ((skb_queue_len(&l->deferdq) == 1) || !(def_cnt % TIPC_NACK_INTV)) + if (defq_len >= 3 && !((defq_len - 3) % 16)) tipc_link_build_proto_msg(l, STATE_MSG, 0, 0, 0, 0, 0, xmitq); return 0; } @@ -1306,29 +1450,29 @@ int tipc_link_rcv(struct tipc_link *l, struct sk_buff *skb, struct sk_buff_head *xmitq) { struct sk_buff_head *defq = &l->deferdq; - struct tipc_msg *hdr; + struct tipc_msg *hdr = buf_msg(skb); u16 seqno, rcv_nxt, win_lim; int rc = 0; + /* Verify and update link state */ + if (unlikely(msg_user(hdr) == LINK_PROTOCOL)) + return tipc_link_proto_rcv(l, skb, xmitq); + + /* Don't send probe at next timeout expiration */ + l->silent_intv_cnt = 0; + do { hdr = buf_msg(skb); seqno = msg_seqno(hdr); rcv_nxt = l->rcv_nxt; win_lim = rcv_nxt + TIPC_MAX_LINK_WIN; - /* Verify and update link state */ - if (unlikely(msg_user(hdr) == LINK_PROTOCOL)) - return tipc_link_proto_rcv(l, skb, xmitq); - if (unlikely(!link_is_up(l))) { if (l->state == LINK_ESTABLISHING) rc = TIPC_LINK_UP_EVT; goto drop; } - /* Don't send probe at next timeout expiration */ - l->silent_intv_cnt = 0; - /* Drop if outside receive window */ if (unlikely(less(seqno, rcv_nxt) || more(seqno, win_lim))) { l->stats.duplicates++; @@ -1353,13 +1497,16 @@ int tipc_link_rcv(struct tipc_link *l, struct sk_buff *skb, /* Deliver packet */ l->rcv_nxt++; l->stats.recv_pkts++; - if (!tipc_data_input(l, skb, l->inputq)) - rc |= tipc_link_input(l, skb, l->inputq); + + if (unlikely(msg_user(hdr) == TUNNEL_PROTOCOL)) + rc |= tipc_link_tnl_rcv(l, skb, l->inputq); + else if (!tipc_data_input(l, skb, l->inputq)) + rc |= tipc_link_input(l, skb, l->inputq, &l->reasm_buf); if (unlikely(++l->rcv_unacked >= TIPC_MIN_LINK_WIN)) rc |= tipc_link_build_state_msg(l, xmitq); if (unlikely(rc & ~TIPC_LINK_SND_STATE)) break; - } while ((skb = __skb_dequeue(defq))); + } while ((skb = __tipc_skb_dequeue(defq, l->rcv_nxt))); return rc; drop: @@ -1380,6 +1527,7 @@ static void tipc_link_build_proto_msg(struct tipc_link *l, int mtyp, bool probe, struct tipc_mon_state *mstate = &l->mon_state; int dlen = 0; void *data; + u16 glen = 0; /* Don't send protocol message during reset or link failover */ if (tipc_link_is_blocked(l)) @@ -1392,8 +1540,8 @@ static void tipc_link_build_proto_msg(struct tipc_link *l, int mtyp, bool probe, rcvgap = buf_seqno(skb_peek(dfq)) - l->rcv_nxt; skb = tipc_msg_create(LINK_PROTOCOL, mtyp, INT_H_SIZE, - tipc_max_domain_size, l->addr, - tipc_own_addr(l->net), 0, 0, 0); + tipc_max_domain_size + MAX_GAP_ACK_BLKS_SZ, + l->addr, tipc_own_addr(l->net), 0, 0, 0); if (!skb) return; @@ -1420,9 +1568,11 @@ static void tipc_link_build_proto_msg(struct tipc_link *l, int mtyp, bool probe, msg_set_bc_gap(hdr, link_bc_rcv_gap(bcl)); msg_set_probe(hdr, probe); msg_set_is_keepalive(hdr, probe || probe_reply); - tipc_mon_prep(l->net, data, &dlen, mstate, l->bearer_id); - msg_set_size(hdr, INT_H_SIZE + dlen); - skb_trim(skb, INT_H_SIZE + dlen); + if (l->peer_caps & TIPC_GAP_ACK_BLOCK) + glen = tipc_build_gap_ack_blks(l, data); + tipc_mon_prep(l->net, data + glen, &dlen, mstate, l->bearer_id); + msg_set_size(hdr, INT_H_SIZE + glen + dlen); + skb_trim(skb, INT_H_SIZE + glen + dlen); l->stats.sent_states++; l->rcv_unacked = 0; } else { @@ -1481,6 +1631,7 @@ void tipc_link_create_dummy_tnl_msg(struct tipc_link *l, void tipc_link_tnl_prepare(struct tipc_link *l, struct tipc_link *tnl, int mtyp, struct sk_buff_head *xmitq) { + struct sk_buff_head *fdefq = &tnl->failover_deferdq; struct sk_buff *skb, *tnlskb; struct tipc_msg *hdr, tnlhdr; struct sk_buff_head *queue = &l->transmq; @@ -1508,7 +1659,11 @@ void tipc_link_tnl_prepare(struct tipc_link *l, struct tipc_link *tnl, /* Initialize reusable tunnel packet header */ tipc_msg_init(tipc_own_addr(l->net), &tnlhdr, TUNNEL_PROTOCOL, mtyp, INT_H_SIZE, l->addr); - pktcnt = skb_queue_len(&l->transmq) + skb_queue_len(&l->backlogq); + if (mtyp == SYNCH_MSG) + pktcnt = l->snd_nxt - buf_seqno(skb_peek(&l->transmq)); + else + pktcnt = skb_queue_len(&l->transmq); + pktcnt += skb_queue_len(&l->backlogq); msg_set_msgcnt(&tnlhdr, pktcnt); msg_set_bearer_id(&tnlhdr, l->peer_bearer_id); tnl: @@ -1539,6 +1694,49 @@ tnl: tnl->drop_point = l->rcv_nxt; tnl->failover_reasm_skb = l->reasm_buf; l->reasm_buf = NULL; + + /* Failover the link's deferdq */ + if (unlikely(!skb_queue_empty(fdefq))) { + pr_warn("Link failover deferdq not empty: %d!\n", + skb_queue_len(fdefq)); + __skb_queue_purge(fdefq); + } + skb_queue_splice_init(&l->deferdq, fdefq); + } +} + +/** + * tipc_link_failover_prepare() - prepare tnl for link failover + * + * This is a special version of the precursor - tipc_link_tnl_prepare(), + * see the tipc_node_link_failover() for details + * + * @l: failover link + * @tnl: tunnel link + * @xmitq: queue for messages to be xmited + */ +void tipc_link_failover_prepare(struct tipc_link *l, struct tipc_link *tnl, + struct sk_buff_head *xmitq) +{ + struct sk_buff_head *fdefq = &tnl->failover_deferdq; + + tipc_link_create_dummy_tnl_msg(tnl, xmitq); + + /* This failover link enpoint was never established before, + * so it has not received anything from peer. + * Otherwise, it must be a normal failover situation or the + * node has entered SELF_DOWN_PEER_LEAVING and both peer nodes + * would have to start over from scratch instead. + */ + WARN_ON(l && tipc_link_is_up(l)); + tnl->drop_point = 1; + tnl->failover_reasm_skb = NULL; + + /* Initiate the link's failover deferdq */ + if (unlikely(!skb_queue_empty(fdefq))) { + pr_warn("Link failover deferdq not empty: %d!\n", + skb_queue_len(fdefq)); + __skb_queue_purge(fdefq); } } @@ -1592,6 +1790,7 @@ static int tipc_link_proto_rcv(struct tipc_link *l, struct sk_buff *skb, struct sk_buff_head *xmitq) { struct tipc_msg *hdr = buf_msg(skb); + struct tipc_gap_ack_blks *ga = NULL; u16 rcvgap = 0; u16 ack = msg_ack(hdr); u16 gap = msg_seq_gap(hdr); @@ -1602,6 +1801,7 @@ static int tipc_link_proto_rcv(struct tipc_link *l, struct sk_buff *skb, u16 dlen = msg_data_sz(hdr); int mtyp = msg_type(hdr); bool reply = msg_probe(hdr); + u16 glen = 0; void *data; char *if_name; int rc = 0; @@ -1699,7 +1899,17 @@ static int tipc_link_proto_rcv(struct tipc_link *l, struct sk_buff *skb, rc = TIPC_LINK_UP_EVT; break; } - tipc_mon_rcv(l->net, data, dlen, l->addr, + + /* Receive Gap ACK blocks from peer if any */ + if (l->peer_caps & TIPC_GAP_ACK_BLOCK) { + ga = (struct tipc_gap_ack_blks *)data; + glen = ntohs(ga->len); + /* sanity check: if failed, ignore Gap ACK blocks */ + if (glen != tipc_gap_ack_blks_sz(ga->gack_cnt)) + ga = NULL; + } + + tipc_mon_rcv(l->net, data + glen, dlen - glen, l->addr, &l->mon_state, l->bearer_id); /* Send NACK if peer has sent pkts we haven't received yet */ @@ -1708,13 +1918,12 @@ static int tipc_link_proto_rcv(struct tipc_link *l, struct sk_buff *skb, if (rcvgap || reply) tipc_link_build_proto_msg(l, STATE_MSG, 0, reply, rcvgap, 0, 0, xmitq); - tipc_link_release_pkts(l, ack); + + tipc_link_advance_transmq(l, ack, gap, ga, xmitq); /* If NACK, retransmit will now start at right position */ - if (gap) { - rc = tipc_link_retrans(l, l, ack + 1, ack + gap, xmitq); + if (gap) l->stats.recv_nacks++; - } tipc_link_advance_backlog(l, xmitq); if (unlikely(!skb_queue_empty(&l->wakeupq))) @@ -1974,8 +2183,8 @@ int tipc_nl_parse_link_prop(struct nlattr *prop, struct nlattr *props[]) { int err; - err = nla_parse_nested(props, TIPC_NLA_PROP_MAX, prop, - tipc_nl_prop_policy, NULL); + err = nla_parse_nested_deprecated(props, TIPC_NLA_PROP_MAX, prop, + tipc_nl_prop_policy, NULL); if (err) return err; @@ -2054,7 +2263,7 @@ static int __tipc_nl_add_stats(struct sk_buff *skb, struct tipc_stats *s) (s->accu_queue_sz / s->queue_sz_counts) : 0} }; - stats = nla_nest_start(skb, TIPC_NLA_LINK_STATS); + stats = nla_nest_start_noflag(skb, TIPC_NLA_LINK_STATS); if (!stats) return -EMSGSIZE; @@ -2086,7 +2295,7 @@ int __tipc_nl_add_link(struct net *net, struct tipc_nl_msg *msg, if (!hdr) return -EMSGSIZE; - attrs = nla_nest_start(msg->skb, TIPC_NLA_LINK); + attrs = nla_nest_start_noflag(msg->skb, TIPC_NLA_LINK); if (!attrs) goto msg_full; @@ -2108,7 +2317,7 @@ int __tipc_nl_add_link(struct net *net, struct tipc_nl_msg *msg, if (nla_put_flag(msg->skb, TIPC_NLA_LINK_ACTIVE)) goto attr_msg_full; - prop = nla_nest_start(msg->skb, TIPC_NLA_LINK_PROP); + prop = nla_nest_start_noflag(msg->skb, TIPC_NLA_LINK_PROP); if (!prop) goto attr_msg_full; if (nla_put_u32(msg->skb, TIPC_NLA_PROP_PRIO, link->priority)) @@ -2175,7 +2384,7 @@ static int __tipc_nl_add_bc_link_stat(struct sk_buff *skb, (stats->accu_queue_sz / stats->queue_sz_counts) : 0} }; - nest = nla_nest_start(skb, TIPC_NLA_LINK_STATS); + nest = nla_nest_start_noflag(skb, TIPC_NLA_LINK_STATS); if (!nest) return -EMSGSIZE; @@ -2199,6 +2408,8 @@ int tipc_nl_add_bc_link(struct net *net, struct tipc_nl_msg *msg) struct nlattr *attrs; struct nlattr *prop; struct tipc_net *tn = net_generic(net, tipc_net_id); + u32 bc_mode = tipc_bcast_get_broadcast_mode(net); + u32 bc_ratio = tipc_bcast_get_broadcast_ratio(net); struct tipc_link *bcl = tn->bcl; if (!bcl) @@ -2213,7 +2424,7 @@ int tipc_nl_add_bc_link(struct net *net, struct tipc_nl_msg *msg) return -EMSGSIZE; } - attrs = nla_nest_start(msg->skb, TIPC_NLA_LINK); + attrs = nla_nest_start_noflag(msg->skb, TIPC_NLA_LINK); if (!attrs) goto msg_full; @@ -2230,11 +2441,17 @@ int tipc_nl_add_bc_link(struct net *net, struct tipc_nl_msg *msg) if (nla_put_u32(msg->skb, TIPC_NLA_LINK_TX, 0)) goto attr_msg_full; - prop = nla_nest_start(msg->skb, TIPC_NLA_LINK_PROP); + prop = nla_nest_start_noflag(msg->skb, TIPC_NLA_LINK_PROP); if (!prop) goto attr_msg_full; if (nla_put_u32(msg->skb, TIPC_NLA_PROP_WIN, bcl->window)) goto prop_msg_full; + if (nla_put_u32(msg->skb, TIPC_NLA_PROP_BROADCAST, bc_mode)) + goto prop_msg_full; + if (bc_mode & BCLINK_MODE_SEL) + if (nla_put_u32(msg->skb, TIPC_NLA_PROP_BROADCAST_RATIO, + bc_ratio)) + goto prop_msg_full; nla_nest_end(msg->skb, prop); err = __tipc_nl_add_bc_link_stat(msg->skb, &bcl->stats); diff --git a/net/tipc/link.h b/net/tipc/link.h index 8439e0ee53a8af2643f0cccacd57d5a677c1ecd1..adcad65e761ce5a66ffa64e803062fe063ddb713 100644 --- a/net/tipc/link.h +++ b/net/tipc/link.h @@ -90,6 +90,8 @@ void tipc_link_tnl_prepare(struct tipc_link *l, struct tipc_link *tnl, int mtyp, struct sk_buff_head *xmitq); void tipc_link_create_dummy_tnl_msg(struct tipc_link *tnl, struct sk_buff_head *xmitq); +void tipc_link_failover_prepare(struct tipc_link *l, struct tipc_link *tnl, + struct sk_buff_head *xmitq); void tipc_link_build_reset_msg(struct tipc_link *l, struct sk_buff_head *xmitq); int tipc_link_fsm_evt(struct tipc_link *l, int evt); bool tipc_link_is_up(struct tipc_link *l); diff --git a/net/tipc/monitor.c b/net/tipc/monitor.c index 67f69389ec179bff3518b35c9d546685d9d2bc41..6a6eae88442f36652902e16f7c0fbb63bb27c17e 100644 --- a/net/tipc/monitor.c +++ b/net/tipc/monitor.c @@ -696,7 +696,7 @@ static int __tipc_nl_add_monitor_peer(struct tipc_peer *peer, if (!hdr) return -EMSGSIZE; - attrs = nla_nest_start(msg->skb, TIPC_NLA_MON_PEER); + attrs = nla_nest_start_noflag(msg->skb, TIPC_NLA_MON_PEER); if (!attrs) goto msg_full; @@ -785,7 +785,7 @@ int __tipc_nl_add_monitor(struct net *net, struct tipc_nl_msg *msg, if (!hdr) return -EMSGSIZE; - attrs = nla_nest_start(msg->skb, TIPC_NLA_MON); + attrs = nla_nest_start_noflag(msg->skb, TIPC_NLA_MON); if (!attrs) goto msg_full; diff --git a/net/tipc/msg.h b/net/tipc/msg.h index d7e4b8b93f9d39b8335ab4751c29baea807add66..8de02ad6e352cc17b0c825530ba613e586eebc09 100644 --- a/net/tipc/msg.h +++ b/net/tipc/msg.h @@ -117,6 +117,37 @@ struct tipc_msg { __be32 hdr[15]; }; +/* struct tipc_gap_ack - TIPC Gap ACK block + * @ack: seqno of the last consecutive packet in link deferdq + * @gap: number of gap packets since the last ack + * + * E.g: + * link deferdq: 1 2 3 4 10 11 13 14 15 20 + * --> Gap ACK blocks: <4, 5>, <11, 1>, <15, 4>, <20, 0> + */ +struct tipc_gap_ack { + __be16 ack; + __be16 gap; +}; + +/* struct tipc_gap_ack_blks + * @len: actual length of the record + * @gack_cnt: number of Gap ACK blocks in the record + * @gacks: array of Gap ACK blocks + */ +struct tipc_gap_ack_blks { + __be16 len; + u8 gack_cnt; + u8 reserved; + struct tipc_gap_ack gacks[]; +}; + +#define tipc_gap_ack_blks_sz(n) (sizeof(struct tipc_gap_ack_blks) + \ + sizeof(struct tipc_gap_ack) * (n)) + +#define MAX_GAP_ACK_BLKS 32 +#define MAX_GAP_ACK_BLKS_SZ tipc_gap_ack_blks_sz(MAX_GAP_ACK_BLKS) + static inline struct tipc_msg *buf_msg(struct sk_buff *skb) { return (struct tipc_msg *)skb->data; @@ -257,6 +288,16 @@ static inline void msg_set_src_droppable(struct tipc_msg *m, u32 d) msg_set_bits(m, 0, 18, 1, d); } +static inline bool msg_is_rcast(struct tipc_msg *m) +{ + return msg_bits(m, 0, 18, 0x1); +} + +static inline void msg_set_is_rcast(struct tipc_msg *m, bool d) +{ + msg_set_bits(m, 0, 18, 0x1, d); +} + static inline void msg_set_size(struct tipc_msg *m, u32 sz) { m->hdr[0] = htonl((msg_word(m, 0) & ~0x1ffff) | sz); @@ -1110,4 +1151,25 @@ static inline void tipc_skb_queue_splice_tail_init(struct sk_buff_head *list, tipc_skb_queue_splice_tail(&tmp, head); } +/* __tipc_skb_dequeue() - dequeue the head skb according to expected seqno + * @list: list to be dequeued from + * @seqno: seqno of the expected msg + * + * returns skb dequeued from the list if its seqno is less than or equal to + * the expected one, otherwise the skb is still hold + * + * Note: must be used with appropriate locks held only + */ +static inline struct sk_buff *__tipc_skb_dequeue(struct sk_buff_head *list, + u16 seqno) +{ + struct sk_buff *skb = skb_peek(list); + + if (skb && less_eq(buf_seqno(skb), seqno)) { + __skb_unlink(skb, list); + return skb; + } + return NULL; +} + #endif diff --git a/net/tipc/name_table.c b/net/tipc/name_table.c index 89993afe0fbd38713dd3d0499cc79e6c3e159b4d..66a65c2cdb23c09960e71885635412163a6dcecd 100644 --- a/net/tipc/name_table.c +++ b/net/tipc/name_table.c @@ -829,11 +829,11 @@ static int __tipc_nl_add_nametable_publ(struct tipc_nl_msg *msg, if (!hdr) return -EMSGSIZE; - attrs = nla_nest_start(msg->skb, TIPC_NLA_NAME_TABLE); + attrs = nla_nest_start_noflag(msg->skb, TIPC_NLA_NAME_TABLE); if (!attrs) goto msg_full; - b = nla_nest_start(msg->skb, TIPC_NLA_NAME_TABLE_PUBL); + b = nla_nest_start_noflag(msg->skb, TIPC_NLA_NAME_TABLE_PUBL); if (!b) goto attr_msg_full; diff --git a/net/tipc/net.c b/net/tipc/net.c index 7ce1e86b024f09cb7345840d4a4d8e3949c5107d..85707c185360ccc98478ff17070bddb4cdfd0641 100644 --- a/net/tipc/net.c +++ b/net/tipc/net.c @@ -187,7 +187,7 @@ static int __tipc_nl_add_net(struct net *net, struct tipc_nl_msg *msg) if (!hdr) return -EMSGSIZE; - attrs = nla_nest_start(msg->skb, TIPC_NLA_NET); + attrs = nla_nest_start_noflag(msg->skb, TIPC_NLA_NET); if (!attrs) goto msg_full; @@ -245,9 +245,9 @@ int __tipc_nl_net_set(struct sk_buff *skb, struct genl_info *info) if (!info->attrs[TIPC_NLA_NET]) return -EINVAL; - err = nla_parse_nested(attrs, TIPC_NLA_NET_MAX, - info->attrs[TIPC_NLA_NET], tipc_nl_net_policy, - info->extack); + err = nla_parse_nested_deprecated(attrs, TIPC_NLA_NET_MAX, + info->attrs[TIPC_NLA_NET], + tipc_nl_net_policy, info->extack); if (err) return err; diff --git a/net/tipc/netlink.c b/net/tipc/netlink.c index 99ee419210baf4374d438b138ab922034950ba7e..99bd166bccec6941ff031e5be0fe4dbf300cd6f9 100644 --- a/net/tipc/netlink.c +++ b/net/tipc/netlink.c @@ -110,7 +110,9 @@ const struct nla_policy tipc_nl_prop_policy[TIPC_NLA_PROP_MAX + 1] = { [TIPC_NLA_PROP_UNSPEC] = { .type = NLA_UNSPEC }, [TIPC_NLA_PROP_PRIO] = { .type = NLA_U32 }, [TIPC_NLA_PROP_TOL] = { .type = NLA_U32 }, - [TIPC_NLA_PROP_WIN] = { .type = NLA_U32 } + [TIPC_NLA_PROP_WIN] = { .type = NLA_U32 }, + [TIPC_NLA_PROP_BROADCAST] = { .type = NLA_U32 }, + [TIPC_NLA_PROP_BROADCAST_RATIO] = { .type = NLA_U32 } }; const struct nla_policy tipc_nl_bearer_policy[TIPC_NLA_BEARER_MAX + 1] = { @@ -141,115 +143,115 @@ const struct nla_policy tipc_nl_udp_policy[TIPC_NLA_UDP_MAX + 1] = { static const struct genl_ops tipc_genl_v2_ops[] = { { .cmd = TIPC_NL_BEARER_DISABLE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = tipc_nl_bearer_disable, - .policy = tipc_nl_policy, }, { .cmd = TIPC_NL_BEARER_ENABLE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = tipc_nl_bearer_enable, - .policy = tipc_nl_policy, }, { .cmd = TIPC_NL_BEARER_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = tipc_nl_bearer_get, .dumpit = tipc_nl_bearer_dump, - .policy = tipc_nl_policy, }, { .cmd = TIPC_NL_BEARER_ADD, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = tipc_nl_bearer_add, - .policy = tipc_nl_policy, }, { .cmd = TIPC_NL_BEARER_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = tipc_nl_bearer_set, - .policy = tipc_nl_policy, }, { .cmd = TIPC_NL_SOCK_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .start = tipc_dump_start, .dumpit = tipc_nl_sk_dump, .done = tipc_dump_done, - .policy = tipc_nl_policy, }, { .cmd = TIPC_NL_PUBL_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .dumpit = tipc_nl_publ_dump, - .policy = tipc_nl_policy, }, { .cmd = TIPC_NL_LINK_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = tipc_nl_node_get_link, .dumpit = tipc_nl_node_dump_link, - .policy = tipc_nl_policy, }, { .cmd = TIPC_NL_LINK_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = tipc_nl_node_set_link, - .policy = tipc_nl_policy, }, { .cmd = TIPC_NL_LINK_RESET_STATS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = tipc_nl_node_reset_link_stats, - .policy = tipc_nl_policy, }, { .cmd = TIPC_NL_MEDIA_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = tipc_nl_media_get, .dumpit = tipc_nl_media_dump, - .policy = tipc_nl_policy, }, { .cmd = TIPC_NL_MEDIA_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = tipc_nl_media_set, - .policy = tipc_nl_policy, }, { .cmd = TIPC_NL_NODE_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .dumpit = tipc_nl_node_dump, - .policy = tipc_nl_policy, }, { .cmd = TIPC_NL_NET_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .dumpit = tipc_nl_net_dump, - .policy = tipc_nl_policy, }, { .cmd = TIPC_NL_NET_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = tipc_nl_net_set, - .policy = tipc_nl_policy, }, { .cmd = TIPC_NL_NAME_TABLE_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .dumpit = tipc_nl_name_table_dump, - .policy = tipc_nl_policy, }, { .cmd = TIPC_NL_MON_SET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = tipc_nl_node_set_monitor, - .policy = tipc_nl_policy, }, { .cmd = TIPC_NL_MON_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = tipc_nl_node_get_monitor, .dumpit = tipc_nl_node_dump_monitor, - .policy = tipc_nl_policy, }, { .cmd = TIPC_NL_MON_PEER_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .dumpit = tipc_nl_node_dump_monitor_peer, - .policy = tipc_nl_policy, }, { .cmd = TIPC_NL_PEER_REMOVE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = tipc_nl_peer_rm, - .policy = tipc_nl_policy, }, #ifdef CONFIG_TIPC_MEDIA_UDP { .cmd = TIPC_NL_UDP_GET_REMOTEIP, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .dumpit = tipc_udp_nl_dump_remoteip, - .policy = tipc_nl_policy, }, #endif }; @@ -259,6 +261,7 @@ struct genl_family tipc_genl_family __ro_after_init = { .version = TIPC_GENL_V2_VERSION, .hdrsize = 0, .maxattr = TIPC_NLA_MAX, + .policy = tipc_nl_policy, .netnsok = true, .module = THIS_MODULE, .ops = tipc_genl_v2_ops, @@ -273,8 +276,8 @@ int tipc_nlmsg_parse(const struct nlmsghdr *nlh, struct nlattr ***attr) if (!*attr) return -EOPNOTSUPP; - return nlmsg_parse(nlh, GENL_HDRLEN, *attr, maxattr, tipc_nl_policy, - NULL); + return nlmsg_parse_deprecated(nlh, GENL_HDRLEN, *attr, maxattr, + tipc_nl_policy, NULL); } int __init tipc_netlink_start(void) diff --git a/net/tipc/netlink_compat.c b/net/tipc/netlink_compat.c index 340a6e7c43a7d39c596de4f1b0045e4200edc2a3..c6a04c09d075c92f5f4333f7a7924d384c93a314 100644 --- a/net/tipc/netlink_compat.c +++ b/net/tipc/netlink_compat.c @@ -328,9 +328,9 @@ static int __tipc_nl_compat_doit(struct tipc_nl_compat_cmd_doit *cmd, if (err) goto doit_out; - err = nla_parse(attrbuf, tipc_genl_family.maxattr, - (const struct nlattr *)trans_buf->data, - trans_buf->len, NULL, NULL); + err = nla_parse_deprecated(attrbuf, tipc_genl_family.maxattr, + (const struct nlattr *)trans_buf->data, + trans_buf->len, NULL, NULL); if (err) goto doit_out; @@ -378,8 +378,8 @@ static int tipc_nl_compat_bearer_dump(struct tipc_nl_compat_msg *msg, if (!attrs[TIPC_NLA_BEARER]) return -EINVAL; - err = nla_parse_nested(bearer, TIPC_NLA_BEARER_MAX, - attrs[TIPC_NLA_BEARER], NULL, NULL); + err = nla_parse_nested_deprecated(bearer, TIPC_NLA_BEARER_MAX, + attrs[TIPC_NLA_BEARER], NULL, NULL); if (err) return err; @@ -399,7 +399,7 @@ static int tipc_nl_compat_bearer_enable(struct tipc_nl_compat_cmd_doit *cmd, b = (struct tipc_bearer_config *)TLV_DATA(msg->req); - bearer = nla_nest_start(skb, TIPC_NLA_BEARER); + bearer = nla_nest_start_noflag(skb, TIPC_NLA_BEARER); if (!bearer) return -EMSGSIZE; @@ -419,7 +419,7 @@ static int tipc_nl_compat_bearer_enable(struct tipc_nl_compat_cmd_doit *cmd, return -EMSGSIZE; if (ntohl(b->priority) <= TIPC_MAX_LINK_PRI) { - prop = nla_nest_start(skb, TIPC_NLA_BEARER_PROP); + prop = nla_nest_start_noflag(skb, TIPC_NLA_BEARER_PROP); if (!prop) return -EMSGSIZE; if (nla_put_u32(skb, TIPC_NLA_PROP_PRIO, ntohl(b->priority))) @@ -441,7 +441,7 @@ static int tipc_nl_compat_bearer_disable(struct tipc_nl_compat_cmd_doit *cmd, name = (char *)TLV_DATA(msg->req); - bearer = nla_nest_start(skb, TIPC_NLA_BEARER); + bearer = nla_nest_start_noflag(skb, TIPC_NLA_BEARER); if (!bearer) return -EMSGSIZE; @@ -514,24 +514,26 @@ static int tipc_nl_compat_link_stat_dump(struct tipc_nl_compat_msg *msg, if (!attrs[TIPC_NLA_LINK]) return -EINVAL; - err = nla_parse_nested(link, TIPC_NLA_LINK_MAX, attrs[TIPC_NLA_LINK], - NULL, NULL); + err = nla_parse_nested_deprecated(link, TIPC_NLA_LINK_MAX, + attrs[TIPC_NLA_LINK], NULL, NULL); if (err) return err; if (!link[TIPC_NLA_LINK_PROP]) return -EINVAL; - err = nla_parse_nested(prop, TIPC_NLA_PROP_MAX, - link[TIPC_NLA_LINK_PROP], NULL, NULL); + err = nla_parse_nested_deprecated(prop, TIPC_NLA_PROP_MAX, + link[TIPC_NLA_LINK_PROP], NULL, + NULL); if (err) return err; if (!link[TIPC_NLA_LINK_STATS]) return -EINVAL; - err = nla_parse_nested(stats, TIPC_NLA_STATS_MAX, - link[TIPC_NLA_LINK_STATS], NULL, NULL); + err = nla_parse_nested_deprecated(stats, TIPC_NLA_STATS_MAX, + link[TIPC_NLA_LINK_STATS], NULL, + NULL); if (err) return err; @@ -645,8 +647,8 @@ static int tipc_nl_compat_link_dump(struct tipc_nl_compat_msg *msg, if (!attrs[TIPC_NLA_LINK]) return -EINVAL; - err = nla_parse_nested(link, TIPC_NLA_LINK_MAX, attrs[TIPC_NLA_LINK], - NULL, NULL); + err = nla_parse_nested_deprecated(link, TIPC_NLA_LINK_MAX, + attrs[TIPC_NLA_LINK], NULL, NULL); if (err) return err; @@ -685,7 +687,7 @@ static int tipc_nl_compat_media_set(struct sk_buff *skb, lc = (struct tipc_link_config *)TLV_DATA(msg->req); - media = nla_nest_start(skb, TIPC_NLA_MEDIA); + media = nla_nest_start_noflag(skb, TIPC_NLA_MEDIA); if (!media) return -EMSGSIZE; @@ -696,7 +698,7 @@ static int tipc_nl_compat_media_set(struct sk_buff *skb, if (nla_put_string(skb, TIPC_NLA_MEDIA_NAME, lc->name)) return -EMSGSIZE; - prop = nla_nest_start(skb, TIPC_NLA_MEDIA_PROP); + prop = nla_nest_start_noflag(skb, TIPC_NLA_MEDIA_PROP); if (!prop) return -EMSGSIZE; @@ -717,7 +719,7 @@ static int tipc_nl_compat_bearer_set(struct sk_buff *skb, lc = (struct tipc_link_config *)TLV_DATA(msg->req); - bearer = nla_nest_start(skb, TIPC_NLA_BEARER); + bearer = nla_nest_start_noflag(skb, TIPC_NLA_BEARER); if (!bearer) return -EMSGSIZE; @@ -728,7 +730,7 @@ static int tipc_nl_compat_bearer_set(struct sk_buff *skb, if (nla_put_string(skb, TIPC_NLA_BEARER_NAME, lc->name)) return -EMSGSIZE; - prop = nla_nest_start(skb, TIPC_NLA_BEARER_PROP); + prop = nla_nest_start_noflag(skb, TIPC_NLA_BEARER_PROP); if (!prop) return -EMSGSIZE; @@ -748,14 +750,14 @@ static int __tipc_nl_compat_link_set(struct sk_buff *skb, lc = (struct tipc_link_config *)TLV_DATA(msg->req); - link = nla_nest_start(skb, TIPC_NLA_LINK); + link = nla_nest_start_noflag(skb, TIPC_NLA_LINK); if (!link) return -EMSGSIZE; if (nla_put_string(skb, TIPC_NLA_LINK_NAME, lc->name)) return -EMSGSIZE; - prop = nla_nest_start(skb, TIPC_NLA_LINK_PROP); + prop = nla_nest_start_noflag(skb, TIPC_NLA_LINK_PROP); if (!prop) return -EMSGSIZE; @@ -811,7 +813,7 @@ static int tipc_nl_compat_link_reset_stats(struct tipc_nl_compat_cmd_doit *cmd, name = (char *)TLV_DATA(msg->req); - link = nla_nest_start(skb, TIPC_NLA_LINK); + link = nla_nest_start_noflag(skb, TIPC_NLA_LINK); if (!link) return -EMSGSIZE; @@ -869,16 +871,18 @@ static int tipc_nl_compat_name_table_dump(struct tipc_nl_compat_msg *msg, if (!attrs[TIPC_NLA_NAME_TABLE]) return -EINVAL; - err = nla_parse_nested(nt, TIPC_NLA_NAME_TABLE_MAX, - attrs[TIPC_NLA_NAME_TABLE], NULL, NULL); + err = nla_parse_nested_deprecated(nt, TIPC_NLA_NAME_TABLE_MAX, + attrs[TIPC_NLA_NAME_TABLE], NULL, + NULL); if (err) return err; if (!nt[TIPC_NLA_NAME_TABLE_PUBL]) return -EINVAL; - err = nla_parse_nested(publ, TIPC_NLA_PUBL_MAX, - nt[TIPC_NLA_NAME_TABLE_PUBL], NULL, NULL); + err = nla_parse_nested_deprecated(publ, TIPC_NLA_PUBL_MAX, + nt[TIPC_NLA_NAME_TABLE_PUBL], NULL, + NULL); if (err) return err; @@ -937,8 +941,8 @@ static int __tipc_nl_compat_publ_dump(struct tipc_nl_compat_msg *msg, if (!attrs[TIPC_NLA_PUBL]) return -EINVAL; - err = nla_parse_nested(publ, TIPC_NLA_PUBL_MAX, attrs[TIPC_NLA_PUBL], - NULL, NULL); + err = nla_parse_nested_deprecated(publ, TIPC_NLA_PUBL_MAX, + attrs[TIPC_NLA_PUBL], NULL, NULL); if (err) return err; @@ -973,7 +977,7 @@ static int tipc_nl_compat_publ_dump(struct tipc_nl_compat_msg *msg, u32 sock) return -EMSGSIZE; } - nest = nla_nest_start(args, TIPC_NLA_SOCK); + nest = nla_nest_start_noflag(args, TIPC_NLA_SOCK); if (!nest) { kfree_skb(args); return -EMSGSIZE; @@ -1007,8 +1011,8 @@ static int tipc_nl_compat_sk_dump(struct tipc_nl_compat_msg *msg, if (!attrs[TIPC_NLA_SOCK]) return -EINVAL; - err = nla_parse_nested(sock, TIPC_NLA_SOCK_MAX, attrs[TIPC_NLA_SOCK], - NULL, NULL); + err = nla_parse_nested_deprecated(sock, TIPC_NLA_SOCK_MAX, + attrs[TIPC_NLA_SOCK], NULL, NULL); if (err) return err; @@ -1019,8 +1023,9 @@ static int tipc_nl_compat_sk_dump(struct tipc_nl_compat_msg *msg, u32 node; struct nlattr *con[TIPC_NLA_CON_MAX + 1]; - err = nla_parse_nested(con, TIPC_NLA_CON_MAX, - sock[TIPC_NLA_SOCK_CON], NULL, NULL); + err = nla_parse_nested_deprecated(con, TIPC_NLA_CON_MAX, + sock[TIPC_NLA_SOCK_CON], + NULL, NULL); if (err) return err; @@ -1059,8 +1064,8 @@ static int tipc_nl_compat_media_dump(struct tipc_nl_compat_msg *msg, if (!attrs[TIPC_NLA_MEDIA]) return -EINVAL; - err = nla_parse_nested(media, TIPC_NLA_MEDIA_MAX, - attrs[TIPC_NLA_MEDIA], NULL, NULL); + err = nla_parse_nested_deprecated(media, TIPC_NLA_MEDIA_MAX, + attrs[TIPC_NLA_MEDIA], NULL, NULL); if (err) return err; @@ -1079,8 +1084,8 @@ static int tipc_nl_compat_node_dump(struct tipc_nl_compat_msg *msg, if (!attrs[TIPC_NLA_NODE]) return -EINVAL; - err = nla_parse_nested(node, TIPC_NLA_NODE_MAX, attrs[TIPC_NLA_NODE], - NULL, NULL); + err = nla_parse_nested_deprecated(node, TIPC_NLA_NODE_MAX, + attrs[TIPC_NLA_NODE], NULL, NULL); if (err) return err; @@ -1100,7 +1105,7 @@ static int tipc_nl_compat_net_set(struct tipc_nl_compat_cmd_doit *cmd, val = ntohl(*(__be32 *)TLV_DATA(msg->req)); - net = nla_nest_start(skb, TIPC_NLA_NET); + net = nla_nest_start_noflag(skb, TIPC_NLA_NET); if (!net) return -EMSGSIZE; @@ -1126,8 +1131,8 @@ static int tipc_nl_compat_net_dump(struct tipc_nl_compat_msg *msg, if (!attrs[TIPC_NLA_NET]) return -EINVAL; - err = nla_parse_nested(net, TIPC_NLA_NET_MAX, attrs[TIPC_NLA_NET], - NULL, NULL); + err = nla_parse_nested_deprecated(net, TIPC_NLA_NET_MAX, + attrs[TIPC_NLA_NET], NULL, NULL); if (err) return err; @@ -1300,6 +1305,7 @@ send: static const struct genl_ops tipc_genl_compat_ops[] = { { .cmd = TIPC_GENL_CMD, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = tipc_nl_compat_recv, }, }; diff --git a/net/tipc/node.c b/net/tipc/node.c index dd3b6dc17662fc42eb0b567501c6b9e8bee67031..9e106d3ed187eca5a728b8c44ffebfe871dd98a6 100644 --- a/net/tipc/node.c +++ b/net/tipc/node.c @@ -375,14 +375,20 @@ static struct tipc_node *tipc_node_create(struct net *net, u32 addr, if (n->capabilities == capabilities) goto exit; /* Same node may come back with new capabilities */ - write_lock_bh(&n->lock); + tipc_node_write_lock(n); n->capabilities = capabilities; for (bearer_id = 0; bearer_id < MAX_BEARERS; bearer_id++) { l = n->links[bearer_id].link; if (l) tipc_link_update_caps(l, capabilities); } - write_unlock_bh(&n->lock); + tipc_node_write_unlock_fast(n); + + /* Calculate cluster capabilities */ + tn->capabilities = TIPC_NODE_CAPABILITIES; + list_for_each_entry_rcu(temp_node, &tn->node_list, list) { + tn->capabilities &= temp_node->capabilities; + } goto exit; } n = kzalloc(sizeof(*n), GFP_ATOMIC); @@ -433,6 +439,11 @@ static struct tipc_node *tipc_node_create(struct net *net, u32 addr, break; } list_add_tail_rcu(&n->list, &temp_node->list); + /* Calculate cluster capabilities */ + tn->capabilities = TIPC_NODE_CAPABILITIES; + list_for_each_entry_rcu(temp_node, &tn->node_list, list) { + tn->capabilities &= temp_node->capabilities; + } trace_tipc_node_create(n, true, " "); exit: spin_unlock_bh(&tn->node_list_lock); @@ -589,6 +600,7 @@ static void tipc_node_clear_links(struct tipc_node *node) */ static bool tipc_node_cleanup(struct tipc_node *peer) { + struct tipc_node *temp_node; struct tipc_net *tn = tipc_net(peer->net); bool deleted = false; @@ -604,6 +616,13 @@ static bool tipc_node_cleanup(struct tipc_node *peer) deleted = true; } tipc_node_write_unlock(peer); + + /* Calculate cluster capabilities */ + tn->capabilities = TIPC_NODE_CAPABILITIES; + list_for_each_entry_rcu(temp_node, &tn->node_list, list) { + tn->capabilities &= temp_node->capabilities; + } + spin_unlock_bh(&tn->node_list_lock); return deleted; } @@ -695,7 +714,6 @@ static void __tipc_node_link_up(struct tipc_node *n, int bearer_id, *slot0 = bearer_id; *slot1 = bearer_id; tipc_node_fsm_evt(n, SELF_ESTABL_CONTACT_EVT); - n->failover_sent = false; n->action_flags |= TIPC_NOTIFY_NODE_UP; tipc_link_set_active(nl, true); tipc_bcast_add_peer(n->net, nl, xmitq); @@ -737,6 +755,45 @@ static void tipc_node_link_up(struct tipc_node *n, int bearer_id, tipc_node_write_unlock(n); } +/** + * tipc_node_link_failover() - start failover in case "half-failover" + * + * This function is only called in a very special situation where link + * failover can be already started on peer node but not on this node. + * This can happen when e.g. + * 1. Both links <1A-2A>, <1B-2B> down + * 2. Link endpoint 2A up, but 1A still down (e.g. due to network + * disturbance, wrong session, etc.) + * 3. Link <1B-2B> up + * 4. Link endpoint 2A down (e.g. due to link tolerance timeout) + * 5. Node B starts failover onto link <1B-2B> + * + * ==> Node A does never start link/node failover! + * + * @n: tipc node structure + * @l: link peer endpoint failingover (- can be NULL) + * @tnl: tunnel link + * @xmitq: queue for messages to be xmited on tnl link later + */ +static void tipc_node_link_failover(struct tipc_node *n, struct tipc_link *l, + struct tipc_link *tnl, + struct sk_buff_head *xmitq) +{ + /* Avoid to be "self-failover" that can never end */ + if (!tipc_link_is_up(tnl)) + return; + + tipc_link_fsm_evt(tnl, LINK_SYNCH_END_EVT); + tipc_node_fsm_evt(n, NODE_SYNCH_END_EVT); + + n->sync_point = tipc_link_rcv_nxt(tnl) + (U16_MAX / 2 - 1); + tipc_link_failover_prepare(l, tnl, xmitq); + + if (l) + tipc_link_fsm_evt(l, LINK_FAILOVER_BEGIN_EVT); + tipc_node_fsm_evt(n, NODE_FAILOVER_BEGIN_EVT); +} + /** * __tipc_node_link_down - handle loss of link */ @@ -1340,7 +1397,7 @@ static int __tipc_nl_add_node(struct tipc_nl_msg *msg, struct tipc_node *node) if (!hdr) return -EMSGSIZE; - attrs = nla_nest_start(msg->skb, TIPC_NLA_NODE); + attrs = nla_nest_start_noflag(msg->skb, TIPC_NLA_NODE); if (!attrs) goto msg_full; @@ -1656,14 +1713,16 @@ static bool tipc_node_check_state(struct tipc_node *n, struct sk_buff *skb, tipc_skb_queue_splice_tail_init(tipc_link_inputq(pl), tipc_link_inputq(l)); } + /* If parallel link was already down, and this happened before - * the tunnel link came up, FAILOVER was never sent. Ensure that - * FAILOVER is sent to get peer out of NODE_FAILINGOVER state. + * the tunnel link came up, node failover was never started. + * Ensure that a FAILOVER_MSG is sent to get peer out of + * NODE_FAILINGOVER state, also this node must accept + * TUNNEL_MSGs from peer. */ - if (n->state != NODE_FAILINGOVER && !n->failover_sent) { - tipc_link_create_dummy_tnl_msg(l, xmitq); - n->failover_sent = true; - } + if (n->state != NODE_FAILINGOVER) + tipc_node_link_failover(n, pl, l, xmitq); + /* If pkts arrive out of order, use lowest calculated syncpt */ if (less(syncpt, n->sync_point)) n->sync_point = syncpt; @@ -1866,9 +1925,9 @@ int tipc_nl_peer_rm(struct sk_buff *skb, struct genl_info *info) if (!info->attrs[TIPC_NLA_NET]) return -EINVAL; - err = nla_parse_nested(attrs, TIPC_NLA_NET_MAX, - info->attrs[TIPC_NLA_NET], tipc_nl_net_policy, - info->extack); + err = nla_parse_nested_deprecated(attrs, TIPC_NLA_NET_MAX, + info->attrs[TIPC_NLA_NET], + tipc_nl_net_policy, info->extack); if (err) return err; @@ -2024,9 +2083,9 @@ int tipc_nl_node_set_link(struct sk_buff *skb, struct genl_info *info) if (!info->attrs[TIPC_NLA_LINK]) return -EINVAL; - err = nla_parse_nested(attrs, TIPC_NLA_LINK_MAX, - info->attrs[TIPC_NLA_LINK], - tipc_nl_link_policy, info->extack); + err = nla_parse_nested_deprecated(attrs, TIPC_NLA_LINK_MAX, + info->attrs[TIPC_NLA_LINK], + tipc_nl_link_policy, info->extack); if (err) return err; @@ -2100,9 +2159,9 @@ int tipc_nl_node_get_link(struct sk_buff *skb, struct genl_info *info) if (!info->attrs[TIPC_NLA_LINK]) return -EINVAL; - err = nla_parse_nested(attrs, TIPC_NLA_LINK_MAX, - info->attrs[TIPC_NLA_LINK], - tipc_nl_link_policy, info->extack); + err = nla_parse_nested_deprecated(attrs, TIPC_NLA_LINK_MAX, + info->attrs[TIPC_NLA_LINK], + tipc_nl_link_policy, info->extack); if (err) return err; @@ -2165,9 +2224,9 @@ int tipc_nl_node_reset_link_stats(struct sk_buff *skb, struct genl_info *info) if (!info->attrs[TIPC_NLA_LINK]) return -EINVAL; - err = nla_parse_nested(attrs, TIPC_NLA_LINK_MAX, - info->attrs[TIPC_NLA_LINK], - tipc_nl_link_policy, info->extack); + err = nla_parse_nested_deprecated(attrs, TIPC_NLA_LINK_MAX, + info->attrs[TIPC_NLA_LINK], + tipc_nl_link_policy, info->extack); if (err) return err; @@ -2305,9 +2364,10 @@ int tipc_nl_node_set_monitor(struct sk_buff *skb, struct genl_info *info) if (!info->attrs[TIPC_NLA_MON]) return -EINVAL; - err = nla_parse_nested(attrs, TIPC_NLA_MON_MAX, - info->attrs[TIPC_NLA_MON], - tipc_nl_monitor_policy, info->extack); + err = nla_parse_nested_deprecated(attrs, TIPC_NLA_MON_MAX, + info->attrs[TIPC_NLA_MON], + tipc_nl_monitor_policy, + info->extack); if (err) return err; @@ -2334,7 +2394,7 @@ static int __tipc_nl_add_monitor_prop(struct net *net, struct tipc_nl_msg *msg) if (!hdr) return -EMSGSIZE; - attrs = nla_nest_start(msg->skb, TIPC_NLA_MON); + attrs = nla_nest_start_noflag(msg->skb, TIPC_NLA_MON); if (!attrs) goto msg_full; @@ -2425,9 +2485,10 @@ int tipc_nl_node_dump_monitor_peer(struct sk_buff *skb, if (!attrs[TIPC_NLA_MON]) return -EINVAL; - err = nla_parse_nested(mon, TIPC_NLA_MON_MAX, - attrs[TIPC_NLA_MON], - tipc_nl_monitor_policy, NULL); + err = nla_parse_nested_deprecated(mon, TIPC_NLA_MON_MAX, + attrs[TIPC_NLA_MON], + tipc_nl_monitor_policy, + NULL); if (err) return err; diff --git a/net/tipc/node.h b/net/tipc/node.h index 4f59a30e989a75827365e2acddc5a9c45b86694d..c0bf49ea3de46ce91ba1757c8efe912d077da92d 100644 --- a/net/tipc/node.h +++ b/net/tipc/node.h @@ -51,7 +51,9 @@ enum { TIPC_BLOCK_FLOWCTL = (1 << 3), TIPC_BCAST_RCAST = (1 << 4), TIPC_NODE_ID128 = (1 << 5), - TIPC_LINK_PROTO_SEQNO = (1 << 6) + TIPC_LINK_PROTO_SEQNO = (1 << 6), + TIPC_MCAST_RBCTL = (1 << 7), + TIPC_GAP_ACK_BLOCK = (1 << 8) }; #define TIPC_NODE_CAPABILITIES (TIPC_SYN_BIT | \ @@ -60,7 +62,9 @@ enum { TIPC_BCAST_RCAST | \ TIPC_BLOCK_FLOWCTL | \ TIPC_NODE_ID128 | \ - TIPC_LINK_PROTO_SEQNO) + TIPC_LINK_PROTO_SEQNO | \ + TIPC_MCAST_RBCTL | \ + TIPC_GAP_ACK_BLOCK) #define INVALID_BEARER_ID -1 void tipc_node_stop(struct net *net); diff --git a/net/tipc/socket.c b/net/tipc/socket.c index b542f14ed444bfcedac61ef2d1eda45d1af1add2..145e4decb0c9ab10470e3a4de66a2c046d0c6f70 100644 --- a/net/tipc/socket.c +++ b/net/tipc/socket.c @@ -485,6 +485,7 @@ static int tipc_sk_create(struct net *net, struct socket *sock, tsk_set_unreturnable(tsk, true); if (sock->type == SOCK_DGRAM) tsk_set_unreliable(tsk, true); + __skb_queue_head_init(&tsk->mc_method.deferredq); } trace_tipc_sk_create(sk, NULL, TIPC_DUMP_NONE, " "); @@ -582,6 +583,7 @@ static int tipc_release(struct socket *sock) sk->sk_shutdown = SHUTDOWN_MASK; tipc_sk_leave(tsk); tipc_sk_withdraw(tsk, 0, NULL); + __skb_queue_purge(&tsk->mc_method.deferredq); sk_stop_timer(sk, &sk->sk_timer); tipc_sk_remove(tsk); @@ -2149,6 +2151,7 @@ static void tipc_sk_filter_rcv(struct sock *sk, struct sk_buff *skb, struct tipc_msg *hdr = buf_msg(skb); struct net *net = sock_net(sk); struct sk_buff_head inputq; + int mtyp = msg_type(hdr); int limit, err = TIPC_OK; trace_tipc_sk_filter_rcv(sk, skb, TIPC_DUMP_ALL, " "); @@ -2162,6 +2165,9 @@ static void tipc_sk_filter_rcv(struct sock *sk, struct sk_buff *skb, if (unlikely(grp)) tipc_group_filter_msg(grp, &inputq, xmitq); + if (unlikely(!grp) && mtyp == TIPC_MCAST_MSG) + tipc_mcast_filter_msg(net, &tsk->mc_method.deferredq, &inputq); + /* Validate and add to receive buffer if there is space */ while ((skb = __skb_dequeue(&inputq))) { hdr = buf_msg(skb); @@ -3064,6 +3070,9 @@ static int tipc_getsockopt(struct socket *sock, int lvl, int opt, case TIPC_SOCK_RECVQ_DEPTH: value = skb_queue_len(&sk->sk_receive_queue); break; + case TIPC_SOCK_RECVQ_USED: + value = sk_rmem_alloc_get(sk); + break; case TIPC_GROUP_JOIN: seq.type = 0; if (tsk->group) @@ -3264,7 +3273,7 @@ static int __tipc_nl_add_sk_con(struct sk_buff *skb, struct tipc_sock *tsk) peer_node = tsk_peer_node(tsk); peer_port = tsk_peer_port(tsk); - nest = nla_nest_start(skb, TIPC_NLA_SOCK_CON); + nest = nla_nest_start_noflag(skb, TIPC_NLA_SOCK_CON); if (!nest) return -EMSGSIZE; @@ -3323,7 +3332,7 @@ static int __tipc_nl_add_sk(struct sk_buff *skb, struct netlink_callback *cb, if (!hdr) goto msg_cancel; - attrs = nla_nest_start(skb, TIPC_NLA_SOCK); + attrs = nla_nest_start_noflag(skb, TIPC_NLA_SOCK); if (!attrs) goto genlmsg_cancel; @@ -3428,7 +3437,7 @@ int tipc_sk_fill_sock_diag(struct sk_buff *skb, struct netlink_callback *cb, if (!(sk_filter_state & (1 << sk->sk_state))) return 0; - attrs = nla_nest_start(skb, TIPC_NLA_SOCK); + attrs = nla_nest_start_noflag(skb, TIPC_NLA_SOCK); if (!attrs) goto msg_cancel; @@ -3446,7 +3455,7 @@ int tipc_sk_fill_sock_diag(struct sk_buff *skb, struct netlink_callback *cb, TIPC_NLA_SOCK_PAD)) goto attr_msg_cancel; - stat = nla_nest_start(skb, TIPC_NLA_SOCK_STAT); + stat = nla_nest_start_noflag(skb, TIPC_NLA_SOCK_STAT); if (!stat) goto attr_msg_cancel; @@ -3503,7 +3512,7 @@ static int __tipc_nl_add_sk_publ(struct sk_buff *skb, if (!hdr) goto msg_cancel; - attrs = nla_nest_start(skb, TIPC_NLA_PUBL); + attrs = nla_nest_start_noflag(skb, TIPC_NLA_PUBL); if (!attrs) goto genlmsg_cancel; @@ -3590,9 +3599,9 @@ int tipc_nl_publ_dump(struct sk_buff *skb, struct netlink_callback *cb) if (!attrs[TIPC_NLA_SOCK]) return -EINVAL; - err = nla_parse_nested(sock, TIPC_NLA_SOCK_MAX, - attrs[TIPC_NLA_SOCK], - tipc_nl_sock_policy, NULL); + err = nla_parse_nested_deprecated(sock, TIPC_NLA_SOCK_MAX, + attrs[TIPC_NLA_SOCK], + tipc_nl_sock_policy, NULL); if (err) return err; diff --git a/net/tipc/udp_media.c b/net/tipc/udp_media.c index 4d85d71f16e2abcee0f2abd84020f9b471f3e8fa..7fc02d84c4f1087a08cffae68812a793ab199bd2 100644 --- a/net/tipc/udp_media.c +++ b/net/tipc/udp_media.c @@ -44,7 +44,7 @@ #include #include #include -#include +#include #include #include "core.h" #include "addr.h" @@ -354,25 +354,21 @@ static int tipc_udp_recv(struct sock *sk, struct sk_buff *skb) skb_pull(skb, sizeof(struct udphdr)); hdr = buf_msg(skb); - rcu_read_lock(); - b = rcu_dereference_rtnl(ub->bearer); + b = rcu_dereference(ub->bearer); if (!b) - goto rcu_out; + goto out; if (b && test_bit(0, &b->up)) { tipc_rcv(sock_net(sk), skb, b); - rcu_read_unlock(); return 0; } if (unlikely(msg_user(hdr) == LINK_CONFIG)) { err = tipc_udp_rcast_disc(b, skb); if (err) - goto rcu_out; + goto out; } -rcu_out: - rcu_read_unlock(); out: kfree_skb(skb); return 0; @@ -451,9 +447,9 @@ int tipc_udp_nl_dump_remoteip(struct sk_buff *skb, struct netlink_callback *cb) if (!attrs[TIPC_NLA_BEARER]) return -EINVAL; - err = nla_parse_nested(battrs, TIPC_NLA_BEARER_MAX, - attrs[TIPC_NLA_BEARER], - tipc_nl_bearer_policy, NULL); + err = nla_parse_nested_deprecated(battrs, TIPC_NLA_BEARER_MAX, + attrs[TIPC_NLA_BEARER], + tipc_nl_bearer_policy, NULL); if (err) return err; @@ -527,7 +523,7 @@ int tipc_udp_nl_add_bearer_data(struct tipc_nl_msg *msg, struct tipc_bearer *b) if (!ub) return -ENODEV; - nest = nla_nest_start(msg->skb, TIPC_NLA_BEARER_UDP_OPTS); + nest = nla_nest_start_noflag(msg->skb, TIPC_NLA_BEARER_UDP_OPTS); if (!nest) goto msg_full; @@ -605,8 +601,7 @@ int tipc_udp_nl_bearer_add(struct tipc_bearer *b, struct nlattr *attr) struct nlattr *opts[TIPC_NLA_UDP_MAX + 1]; struct udp_media_addr *dst; - if (nla_parse_nested(opts, TIPC_NLA_UDP_MAX, attr, - tipc_nl_udp_policy, NULL)) + if (nla_parse_nested_deprecated(opts, TIPC_NLA_UDP_MAX, attr, tipc_nl_udp_policy, NULL)) return -EINVAL; if (!opts[TIPC_NLA_UDP_REMOTE]) @@ -659,9 +654,7 @@ static int tipc_udp_enable(struct net *net, struct tipc_bearer *b, if (!attrs[TIPC_NLA_BEARER_UDP_OPTS]) goto err; - if (nla_parse_nested(opts, TIPC_NLA_UDP_MAX, - attrs[TIPC_NLA_BEARER_UDP_OPTS], - tipc_nl_udp_policy, NULL)) + if (nla_parse_nested_deprecated(opts, TIPC_NLA_UDP_MAX, attrs[TIPC_NLA_BEARER_UDP_OPTS], tipc_nl_udp_policy, NULL)) goto err; if (!opts[TIPC_NLA_UDP_LOCAL] || !opts[TIPC_NLA_UDP_REMOTE]) { diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index 14dedb24fa7b6ff1e7fe99f9adbe114eeabf15c1..e225c81e6b357ea40028723bc890f3277672cfa8 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -89,22 +89,6 @@ static void tls_device_gc_task(struct work_struct *work) } } -static void tls_device_attach(struct tls_context *ctx, struct sock *sk, - struct net_device *netdev) -{ - if (sk->sk_destruct != tls_device_sk_destruct) { - refcount_set(&ctx->refcount, 1); - dev_hold(netdev); - ctx->netdev = netdev; - spin_lock_irq(&tls_device_lock); - list_add_tail(&ctx->list, &tls_device_list); - spin_unlock_irq(&tls_device_lock); - - ctx->sk_destruct = sk->sk_destruct; - sk->sk_destruct = tls_device_sk_destruct; - } -} - static void tls_device_queue_ctx_destruction(struct tls_context *ctx) { unsigned long flags; @@ -199,7 +183,7 @@ static void tls_icsk_clean_acked(struct sock *sk, u32 acked_seq) * socket and no in-flight SKBs associated with this * socket, so it is safe to free all the resources. */ -void tls_device_sk_destruct(struct sock *sk) +static void tls_device_sk_destruct(struct sock *sk) { struct tls_context *tls_ctx = tls_get_ctx(sk); struct tls_offload_context_tx *ctx = tls_offload_ctx_tx(tls_ctx); @@ -217,7 +201,6 @@ void tls_device_sk_destruct(struct sock *sk) if (refcount_dec_and_test(&tls_ctx->refcount)) tls_device_queue_ctx_destruction(tls_ctx); } -EXPORT_SYMBOL(tls_device_sk_destruct); void tls_device_free_resources_tx(struct sock *sk) { @@ -584,7 +567,7 @@ void handle_device_resync(struct sock *sk, u32 seq, u64 rcd_sn) rx_ctx = tls_offload_ctx_rx(tls_ctx); resync_req = atomic64_read(&rx_ctx->resync_req); - req_seq = ntohl(resync_req >> 32) - ((u32)TLS_HEADER_SIZE - 1); + req_seq = (resync_req >> 32) - ((u32)TLS_HEADER_SIZE - 1); is_req_pending = resync_req; if (unlikely(is_req_pending) && req_seq == seq && @@ -699,6 +682,22 @@ int tls_device_decrypted(struct sock *sk, struct sk_buff *skb) tls_device_reencrypt(sk, skb); } +static void tls_device_attach(struct tls_context *ctx, struct sock *sk, + struct net_device *netdev) +{ + if (sk->sk_destruct != tls_device_sk_destruct) { + refcount_set(&ctx->refcount, 1); + dev_hold(netdev); + ctx->netdev = netdev; + spin_lock_irq(&tls_device_lock); + list_add_tail(&ctx->list, &tls_device_list); + spin_unlock_irq(&tls_device_lock); + + ctx->sk_destruct = sk->sk_destruct; + sk->sk_destruct = tls_device_sk_destruct; + } +} + int tls_set_device_offload(struct sock *sk, struct tls_context *ctx) { u16 nonce_size, tag_size, iv_size, rec_seq_size; @@ -882,8 +881,6 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx) } if (!(netdev->features & NETIF_F_HW_TLS_RX)) { - pr_err_ratelimited("%s: netdev %s with no TLS offload\n", - __func__, netdev->name); rc = -ENOTSUPP; goto release_netdev; } @@ -911,11 +908,8 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx) rc = netdev->tlsdev_ops->tls_dev_add(netdev, sk, TLS_OFFLOAD_CTX_DIR_RX, &ctx->crypto_recv.info, tcp_sk(sk)->copied_seq); - if (rc) { - pr_err_ratelimited("%s: The netdev has refused to offload this socket\n", - __func__); + if (rc) goto free_sw_resources; - } tls_device_attach(ctx, sk, netdev); goto release_netdev; diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c index 478603f43964d557146ae141ba45d4b0cae538fd..fc81ae18cc4482b207cab84cc2f87ecef16a8dce 100644 --- a/net/tls/tls_main.c +++ b/net/tls/tls_main.c @@ -490,27 +490,32 @@ static int do_tls_setsockopt_conf(struct sock *sk, char __user *optval, switch (crypto_info->cipher_type) { case TLS_CIPHER_AES_GCM_128: + optsize = sizeof(struct tls12_crypto_info_aes_gcm_128); + break; case TLS_CIPHER_AES_GCM_256: { - optsize = crypto_info->cipher_type == TLS_CIPHER_AES_GCM_128 ? - sizeof(struct tls12_crypto_info_aes_gcm_128) : - sizeof(struct tls12_crypto_info_aes_gcm_256); - if (optlen != optsize) { - rc = -EINVAL; - goto err_crypto_info; - } - rc = copy_from_user(crypto_info + 1, optval + sizeof(*crypto_info), - optlen - sizeof(*crypto_info)); - if (rc) { - rc = -EFAULT; - goto err_crypto_info; - } + optsize = sizeof(struct tls12_crypto_info_aes_gcm_256); break; } + case TLS_CIPHER_AES_CCM_128: + optsize = sizeof(struct tls12_crypto_info_aes_ccm_128); + break; default: rc = -EINVAL; goto err_crypto_info; } + if (optlen != optsize) { + rc = -EINVAL; + goto err_crypto_info; + } + + rc = copy_from_user(crypto_info + 1, optval + sizeof(*crypto_info), + optlen - sizeof(*crypto_info)); + if (rc) { + rc = -EFAULT; + goto err_crypto_info; + } + if (tx) { #ifdef CONFIG_TLS_DEVICE rc = tls_set_device_offload(sk, ctx); diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index 29d6af43dd249dd72c175ac7401a308ef6193c2c..c02293fb10e60a7b904c82980ba74c4dbd0af171 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -42,8 +42,6 @@ #include #include -#define MAX_IV_SIZE TLS_CIPHER_AES_GCM_128_IV_SIZE - static int __skb_nsg(struct sk_buff *skb, int offset, int len, unsigned int recursion_level) { @@ -225,7 +223,7 @@ static int tls_do_decryption(struct sock *sk, /* Using skb->sk to push sk through to crypto async callback * handler. This allows propagating errors up to the socket * if needed. It _must_ be cleared in the async handler - * before kfree_skb is called. We _know_ skb->sk is NULL + * before consume_skb is called. We _know_ skb->sk is NULL * because it is a clone from strparser. */ skb->sk = sk; @@ -479,11 +477,18 @@ static int tls_do_encryption(struct sock *sk, struct tls_rec *rec = ctx->open_rec; struct sk_msg *msg_en = &rec->msg_encrypted; struct scatterlist *sge = sk_msg_elem(msg_en, start); - int rc; + int rc, iv_offset = 0; + + /* For CCM based ciphers, first byte of IV is a constant */ + if (prot->cipher_type == TLS_CIPHER_AES_CCM_128) { + rec->iv_data[0] = TLS_AES_CCM_IV_B0_BYTE; + iv_offset = 1; + } + + memcpy(&rec->iv_data[iv_offset], tls_ctx->tx.iv, + prot->iv_size + prot->salt_size); - memcpy(rec->iv_data, tls_ctx->tx.iv, sizeof(rec->iv_data)); - xor_iv_with_seq(prot->version, rec->iv_data, - tls_ctx->tx.rec_seq); + xor_iv_with_seq(prot->version, rec->iv_data, tls_ctx->tx.rec_seq); sge->offset += prot->prepend_size; sge->length -= prot->prepend_size; @@ -1344,6 +1349,7 @@ static int decrypt_internal(struct sock *sk, struct sk_buff *skb, struct scatterlist *sgout = NULL; const int data_len = rxm->full_len - prot->overhead_size + prot->tail_size; + int iv_offset = 0; if (*zc && (out_iov || out_sg)) { if (out_iov) @@ -1386,18 +1392,25 @@ static int decrypt_internal(struct sock *sk, struct sk_buff *skb, aad = (u8 *)(sgout + n_sgout); iv = aad + prot->aad_size; + /* For CCM based ciphers, first byte of nonce+iv is always '2' */ + if (prot->cipher_type == TLS_CIPHER_AES_CCM_128) { + iv[0] = 2; + iv_offset = 1; + } + /* Prepare IV */ err = skb_copy_bits(skb, rxm->offset + TLS_HEADER_SIZE, - iv + TLS_CIPHER_AES_GCM_128_SALT_SIZE, + iv + iv_offset + prot->salt_size, prot->iv_size); if (err < 0) { kfree(mem); return err; } if (prot->version == TLS_1_3_VERSION) - memcpy(iv, tls_ctx->rx.iv, crypto_aead_ivsize(ctx->aead_recv)); + memcpy(iv + iv_offset, tls_ctx->rx.iv, + crypto_aead_ivsize(ctx->aead_recv)); else - memcpy(iv, tls_ctx->rx.iv, TLS_CIPHER_AES_GCM_128_SALT_SIZE); + memcpy(iv + iv_offset, tls_ctx->rx.iv, prot->salt_size); xor_iv_with_seq(prot->version, iv, tls_ctx->rx.rec_seq); @@ -1524,7 +1537,7 @@ static bool tls_sw_advance_skb(struct sock *sk, struct sk_buff *skb, rxm->full_len -= len; return false; } - kfree_skb(skb); + consume_skb(skb); } /* Finished with message */ @@ -1633,7 +1646,7 @@ static int process_rx_list(struct tls_sw_context_rx *ctx, if (!is_peek) { skb_unlink(skb, &ctx->rx_list); - kfree_skb(skb); + consume_skb(skb); } skb = next_skb; @@ -2144,14 +2157,15 @@ int tls_set_sw_offload(struct sock *sk, struct tls_context *ctx, int tx) struct tls_crypto_info *crypto_info; struct tls12_crypto_info_aes_gcm_128 *gcm_128_info; struct tls12_crypto_info_aes_gcm_256 *gcm_256_info; + struct tls12_crypto_info_aes_ccm_128 *ccm_128_info; struct tls_sw_context_tx *sw_ctx_tx = NULL; struct tls_sw_context_rx *sw_ctx_rx = NULL; struct cipher_context *cctx; struct crypto_aead **aead; struct strp_callbacks cb; - u16 nonce_size, tag_size, iv_size, rec_seq_size; + u16 nonce_size, tag_size, iv_size, rec_seq_size, salt_size; struct crypto_tfm *tfm; - char *iv, *rec_seq, *key, *salt; + char *iv, *rec_seq, *key, *salt, *cipher_name; size_t keysize; int rc = 0; @@ -2216,6 +2230,8 @@ int tls_set_sw_offload(struct sock *sk, struct tls_context *ctx, int tx) keysize = TLS_CIPHER_AES_GCM_128_KEY_SIZE; key = gcm_128_info->key; salt = gcm_128_info->salt; + salt_size = TLS_CIPHER_AES_GCM_128_SALT_SIZE; + cipher_name = "gcm(aes)"; break; } case TLS_CIPHER_AES_GCM_256: { @@ -2231,6 +2247,25 @@ int tls_set_sw_offload(struct sock *sk, struct tls_context *ctx, int tx) keysize = TLS_CIPHER_AES_GCM_256_KEY_SIZE; key = gcm_256_info->key; salt = gcm_256_info->salt; + salt_size = TLS_CIPHER_AES_GCM_256_SALT_SIZE; + cipher_name = "gcm(aes)"; + break; + } + case TLS_CIPHER_AES_CCM_128: { + nonce_size = TLS_CIPHER_AES_CCM_128_IV_SIZE; + tag_size = TLS_CIPHER_AES_CCM_128_TAG_SIZE; + iv_size = TLS_CIPHER_AES_CCM_128_IV_SIZE; + iv = ((struct tls12_crypto_info_aes_ccm_128 *)crypto_info)->iv; + rec_seq_size = TLS_CIPHER_AES_CCM_128_REC_SEQ_SIZE; + rec_seq = + ((struct tls12_crypto_info_aes_ccm_128 *)crypto_info)->rec_seq; + ccm_128_info = + (struct tls12_crypto_info_aes_ccm_128 *)crypto_info; + keysize = TLS_CIPHER_AES_CCM_128_KEY_SIZE; + key = ccm_128_info->key; + salt = ccm_128_info->salt; + salt_size = TLS_CIPHER_AES_CCM_128_SALT_SIZE; + cipher_name = "ccm(aes)"; break; } default: @@ -2260,16 +2295,16 @@ int tls_set_sw_offload(struct sock *sk, struct tls_context *ctx, int tx) prot->overhead_size = prot->prepend_size + prot->tag_size + prot->tail_size; prot->iv_size = iv_size; - cctx->iv = kmalloc(iv_size + TLS_CIPHER_AES_GCM_128_SALT_SIZE, - GFP_KERNEL); + prot->salt_size = salt_size; + cctx->iv = kmalloc(iv_size + salt_size, GFP_KERNEL); if (!cctx->iv) { rc = -ENOMEM; goto free_priv; } /* Note: 128 & 256 bit salt are the same size */ - memcpy(cctx->iv, salt, TLS_CIPHER_AES_GCM_128_SALT_SIZE); - memcpy(cctx->iv + TLS_CIPHER_AES_GCM_128_SALT_SIZE, iv, iv_size); prot->rec_seq_size = rec_seq_size; + memcpy(cctx->iv, salt, salt_size); + memcpy(cctx->iv + salt_size, iv, iv_size); cctx->rec_seq = kmemdup(rec_seq, rec_seq_size, GFP_KERNEL); if (!cctx->rec_seq) { rc = -ENOMEM; @@ -2277,7 +2312,7 @@ int tls_set_sw_offload(struct sock *sk, struct tls_context *ctx, int tx) } if (!*aead) { - *aead = crypto_alloc_aead("gcm(aes)", 0, 0); + *aead = crypto_alloc_aead(cipher_name, 0, 0); if (IS_ERR(*aead)) { rc = PTR_ERR(*aead); *aead = NULL; diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index ddb838a1b74c0bf04fb8dda43fa2bb471925ad5c..e68d7454f2e3bcf9b278c56c945ad711e68dfdd9 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2040,8 +2040,8 @@ static int unix_dgram_recvmsg(struct socket *sock, struct msghdr *msg, struct unix_sock *u = unix_sk(sk); struct sk_buff *skb, *last; long timeo; + int skip; int err; - int peeked, skip; err = -EOPNOTSUPP; if (flags&MSG_OOB) @@ -2053,8 +2053,8 @@ static int unix_dgram_recvmsg(struct socket *sock, struct msghdr *msg, mutex_lock(&u->iolock); skip = sk_peek_offset(sk, flags); - skb = __skb_try_recv_datagram(sk, flags, NULL, &peeked, &skip, - &err, &last); + skb = __skb_try_recv_datagram(sk, flags, NULL, &skip, &err, + &last); if (skb) break; diff --git a/net/wimax/stack.c b/net/wimax/stack.c index a6307813b6d5adddc58f3a2683a6a9c6f94ced2d..4969de6728860b625a16e060e0f6e616a7f5856c 100644 --- a/net/wimax/stack.c +++ b/net/wimax/stack.c @@ -419,26 +419,26 @@ static const struct nla_policy wimax_gnl_policy[WIMAX_GNL_ATTR_MAX + 1] = { static const struct genl_ops wimax_gnl_ops[] = { { .cmd = WIMAX_GNL_OP_MSG_FROM_USER, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = wimax_gnl_policy, .doit = wimax_gnl_doit_msg_from_user, }, { .cmd = WIMAX_GNL_OP_RESET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = wimax_gnl_policy, .doit = wimax_gnl_doit_reset, }, { .cmd = WIMAX_GNL_OP_RFKILL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = wimax_gnl_policy, .doit = wimax_gnl_doit_rfkill, }, { .cmd = WIMAX_GNL_OP_STATE_GET, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_ADMIN_PERM, - .policy = wimax_gnl_policy, .doit = wimax_gnl_doit_state_get, }, }; @@ -582,6 +582,7 @@ struct genl_family wimax_gnl_family __ro_after_init = { .version = WIMAX_GNL_VERSION, .hdrsize = 0, .maxattr = WIMAX_GNL_ATTR_MAX, + .policy = wimax_gnl_policy, .module = THIS_MODULE, .ops = wimax_gnl_ops, .n_ops = ARRAY_SIZE(wimax_gnl_ops), diff --git a/net/wireless/nl80211.c b/net/wireless/nl80211.c index 47e30a58566c2817696655212a8da4c5fc00f00e..fffe4b371e2353a1a5588f309cebc48a28994051 100644 --- a/net/wireless/nl80211.c +++ b/net/wireless/nl80211.c @@ -331,6 +331,11 @@ const struct nla_policy nl80211_policy[NUM_NL80211_ATTR] = { .len = NL80211_MAX_SUPP_RATES }, [NL80211_ATTR_STA_PLINK_ACTION] = NLA_POLICY_MAX(NLA_U8, NUM_NL80211_PLINK_ACTIONS - 1), + [NL80211_ATTR_STA_TX_POWER_SETTING] = + NLA_POLICY_RANGE(NLA_U8, + NL80211_TX_POWER_AUTOMATIC, + NL80211_TX_POWER_FIXED), + [NL80211_ATTR_STA_TX_POWER] = { .type = NLA_S16 }, [NL80211_ATTR_STA_VLAN] = { .type = NLA_U32 }, [NL80211_ATTR_MNTR_FLAGS] = { /* NLA_NESTED can't be empty */ }, [NL80211_ATTR_MESH_ID] = { .type = NLA_BINARY, @@ -553,6 +558,7 @@ static const struct nla_policy nl80211_key_policy[NL80211_KEY_MAX + 1] = { [NL80211_KEY_DEFAULT_MGMT] = { .type = NLA_FLAG }, [NL80211_KEY_TYPE] = NLA_POLICY_MAX(NLA_U32, NUM_NL80211_KEYTYPES - 1), [NL80211_KEY_DEFAULT_TYPES] = { .type = NLA_NESTED }, + [NL80211_KEY_MODE] = NLA_POLICY_RANGE(NLA_U8, 0, NL80211_KEY_SET_TX), }; /* policy for the key default flags */ @@ -617,12 +623,21 @@ nl80211_rekey_policy[NUM_NL80211_REKEY_DATA] = { [NL80211_REKEY_DATA_REPLAY_CTR] = { .len = NL80211_REPLAY_CTR_LEN }, }; +static const struct nla_policy +nl80211_match_band_rssi_policy[NUM_NL80211_BANDS] = { + [NL80211_BAND_2GHZ] = { .type = NLA_S32 }, + [NL80211_BAND_5GHZ] = { .type = NLA_S32 }, + [NL80211_BAND_60GHZ] = { .type = NLA_S32 }, +}; + static const struct nla_policy nl80211_match_policy[NL80211_SCHED_SCAN_MATCH_ATTR_MAX + 1] = { [NL80211_SCHED_SCAN_MATCH_ATTR_SSID] = { .type = NLA_BINARY, .len = IEEE80211_MAX_SSID_LEN }, [NL80211_SCHED_SCAN_MATCH_ATTR_BSSID] = { .len = ETH_ALEN }, [NL80211_SCHED_SCAN_MATCH_ATTR_RSSI] = { .type = NLA_U32 }, + [NL80211_SCHED_SCAN_MATCH_PER_BAND_RSSI] = + NLA_POLICY_NESTED(nl80211_match_band_rssi_policy), }; static const struct nla_policy @@ -688,9 +703,11 @@ int nl80211_prepare_wdev_dump(struct netlink_callback *cb, int err; if (!cb->args[0]) { - err = nlmsg_parse(cb->nlh, GENL_HDRLEN + nl80211_fam.hdrsize, - genl_family_attrbuf(&nl80211_fam), - nl80211_fam.maxattr, nl80211_policy, NULL); + err = nlmsg_parse_deprecated(cb->nlh, + GENL_HDRLEN + nl80211_fam.hdrsize, + genl_family_attrbuf(&nl80211_fam), + nl80211_fam.maxattr, + nl80211_policy, NULL); if (err) return err; @@ -740,13 +757,13 @@ static int nl80211_msg_put_wmm_rules(struct sk_buff *msg, { int j; struct nlattr *nl_wmm_rules = - nla_nest_start(msg, NL80211_FREQUENCY_ATTR_WMM); + nla_nest_start_noflag(msg, NL80211_FREQUENCY_ATTR_WMM); if (!nl_wmm_rules) goto nla_put_failure; for (j = 0; j < IEEE80211_NUM_ACS; j++) { - struct nlattr *nl_wmm_rule = nla_nest_start(msg, j); + struct nlattr *nl_wmm_rule = nla_nest_start_noflag(msg, j); if (!nl_wmm_rule) goto nla_put_failure; @@ -875,7 +892,7 @@ static bool nl80211_put_txq_stats(struct sk_buff *msg, return false; \ } while (0) - txqattr = nla_nest_start(msg, attrtype); + txqattr = nla_nest_start_noflag(msg, attrtype); if (!txqattr) return false; @@ -910,8 +927,9 @@ static int nl80211_parse_key_new(struct genl_info *info, struct nlattr *key, struct key_parse *k) { struct nlattr *tb[NL80211_KEY_MAX + 1]; - int err = nla_parse_nested(tb, NL80211_KEY_MAX, key, - nl80211_key_policy, info->extack); + int err = nla_parse_nested_deprecated(tb, NL80211_KEY_MAX, key, + nl80211_key_policy, + info->extack); if (err) return err; @@ -947,10 +965,11 @@ static int nl80211_parse_key_new(struct genl_info *info, struct nlattr *key, if (tb[NL80211_KEY_DEFAULT_TYPES]) { struct nlattr *kdt[NUM_NL80211_KEY_DEFAULT_TYPES]; - err = nla_parse_nested(kdt, NUM_NL80211_KEY_DEFAULT_TYPES - 1, - tb[NL80211_KEY_DEFAULT_TYPES], - nl80211_key_default_policy, - info->extack); + err = nla_parse_nested_deprecated(kdt, + NUM_NL80211_KEY_DEFAULT_TYPES - 1, + tb[NL80211_KEY_DEFAULT_TYPES], + nl80211_key_default_policy, + info->extack); if (err) return err; @@ -958,6 +977,9 @@ static int nl80211_parse_key_new(struct genl_info *info, struct nlattr *key, k->def_multi = kdt[NL80211_KEY_DEFAULT_TYPE_MULTICAST]; } + if (tb[NL80211_KEY_MODE]) + k->p.mode = nla_get_u8(tb[NL80211_KEY_MODE]); + return 0; } @@ -994,11 +1016,11 @@ static int nl80211_parse_key_old(struct genl_info *info, struct key_parse *k) if (info->attrs[NL80211_ATTR_KEY_DEFAULT_TYPES]) { struct nlattr *kdt[NUM_NL80211_KEY_DEFAULT_TYPES]; - int err = nla_parse_nested(kdt, - NUM_NL80211_KEY_DEFAULT_TYPES - 1, - info->attrs[NL80211_ATTR_KEY_DEFAULT_TYPES], - nl80211_key_default_policy, - info->extack); + int err = nla_parse_nested_deprecated(kdt, + NUM_NL80211_KEY_DEFAULT_TYPES - 1, + info->attrs[NL80211_ATTR_KEY_DEFAULT_TYPES], + nl80211_key_default_policy, + info->extack); if (err) return err; @@ -1187,7 +1209,7 @@ static struct ieee80211_channel *nl80211_get_valid_chan(struct wiphy *wiphy, static int nl80211_put_iftypes(struct sk_buff *msg, u32 attr, u16 ifmodes) { - struct nlattr *nl_modes = nla_nest_start(msg, attr); + struct nlattr *nl_modes = nla_nest_start_noflag(msg, attr); int i; if (!nl_modes) @@ -1215,8 +1237,8 @@ static int nl80211_put_iface_combinations(struct wiphy *wiphy, struct nlattr *nl_combis; int i, j; - nl_combis = nla_nest_start(msg, - NL80211_ATTR_INTERFACE_COMBINATIONS); + nl_combis = nla_nest_start_noflag(msg, + NL80211_ATTR_INTERFACE_COMBINATIONS); if (!nl_combis) goto nla_put_failure; @@ -1226,18 +1248,19 @@ static int nl80211_put_iface_combinations(struct wiphy *wiphy, c = &wiphy->iface_combinations[i]; - nl_combi = nla_nest_start(msg, i + 1); + nl_combi = nla_nest_start_noflag(msg, i + 1); if (!nl_combi) goto nla_put_failure; - nl_limits = nla_nest_start(msg, NL80211_IFACE_COMB_LIMITS); + nl_limits = nla_nest_start_noflag(msg, + NL80211_IFACE_COMB_LIMITS); if (!nl_limits) goto nla_put_failure; for (j = 0; j < c->n_limits; j++) { struct nlattr *nl_limit; - nl_limit = nla_nest_start(msg, j + 1); + nl_limit = nla_nest_start_noflag(msg, j + 1); if (!nl_limit) goto nla_put_failure; if (nla_put_u32(msg, NL80211_IFACE_LIMIT_MAX, @@ -1290,7 +1313,8 @@ static int nl80211_send_wowlan_tcp_caps(struct cfg80211_registered_device *rdev, if (!tcp) return 0; - nl_tcp = nla_nest_start(msg, NL80211_WOWLAN_TRIG_TCP_CONNECTION); + nl_tcp = nla_nest_start_noflag(msg, + NL80211_WOWLAN_TRIG_TCP_CONNECTION); if (!nl_tcp) return -ENOBUFS; @@ -1330,7 +1354,8 @@ static int nl80211_send_wowlan(struct sk_buff *msg, if (!rdev->wiphy.wowlan) return 0; - nl_wowlan = nla_nest_start(msg, NL80211_ATTR_WOWLAN_TRIGGERS_SUPPORTED); + nl_wowlan = nla_nest_start_noflag(msg, + NL80211_ATTR_WOWLAN_TRIGGERS_SUPPORTED); if (!nl_wowlan) return -ENOBUFS; @@ -1459,7 +1484,8 @@ static int nl80211_send_band_rateinfo(struct sk_buff *msg, if (sband->n_iftype_data) { struct nlattr *nl_iftype_data = - nla_nest_start(msg, NL80211_BAND_ATTR_IFTYPE_DATA); + nla_nest_start_noflag(msg, + NL80211_BAND_ATTR_IFTYPE_DATA); int err; if (!nl_iftype_data) @@ -1468,7 +1494,7 @@ static int nl80211_send_band_rateinfo(struct sk_buff *msg, for (i = 0; i < sband->n_iftype_data; i++) { struct nlattr *iftdata; - iftdata = nla_nest_start(msg, i + 1); + iftdata = nla_nest_start_noflag(msg, i + 1); if (!iftdata) return -ENOBUFS; @@ -1484,12 +1510,12 @@ static int nl80211_send_band_rateinfo(struct sk_buff *msg, } /* add bitrates */ - nl_rates = nla_nest_start(msg, NL80211_BAND_ATTR_RATES); + nl_rates = nla_nest_start_noflag(msg, NL80211_BAND_ATTR_RATES); if (!nl_rates) return -ENOBUFS; for (i = 0; i < sband->n_bitrates; i++) { - nl_rate = nla_nest_start(msg, i); + nl_rate = nla_nest_start_noflag(msg, i); if (!nl_rate) return -ENOBUFS; @@ -1522,12 +1548,12 @@ nl80211_send_mgmt_stypes(struct sk_buff *msg, if (!mgmt_stypes) return 0; - nl_ifs = nla_nest_start(msg, NL80211_ATTR_TX_FRAME_TYPES); + nl_ifs = nla_nest_start_noflag(msg, NL80211_ATTR_TX_FRAME_TYPES); if (!nl_ifs) return -ENOBUFS; for (ift = 0; ift < NUM_NL80211_IFTYPES; ift++) { - nl_ftypes = nla_nest_start(msg, ift); + nl_ftypes = nla_nest_start_noflag(msg, ift); if (!nl_ftypes) return -ENOBUFS; i = 0; @@ -1545,12 +1571,12 @@ nl80211_send_mgmt_stypes(struct sk_buff *msg, nla_nest_end(msg, nl_ifs); - nl_ifs = nla_nest_start(msg, NL80211_ATTR_RX_FRAME_TYPES); + nl_ifs = nla_nest_start_noflag(msg, NL80211_ATTR_RX_FRAME_TYPES); if (!nl_ifs) return -ENOBUFS; for (ift = 0; ift < NUM_NL80211_IFTYPES; ift++) { - nl_ftypes = nla_nest_start(msg, ift); + nl_ftypes = nla_nest_start_noflag(msg, ift); if (!nl_ftypes) return -ENOBUFS; i = 0; @@ -1668,7 +1694,7 @@ nl80211_send_pmsr_ftm_capa(const struct cfg80211_pmsr_capabilities *cap, if (!cap->ftm.supported) return 0; - ftm = nla_nest_start(msg, NL80211_PMSR_TYPE_FTM); + ftm = nla_nest_start_noflag(msg, NL80211_PMSR_TYPE_FTM); if (!ftm) return -ENOBUFS; @@ -1716,7 +1742,7 @@ static int nl80211_send_pmsr_capa(struct cfg80211_registered_device *rdev, * will genlmsg_cancel() if we fail */ - pmsr = nla_nest_start(msg, NL80211_ATTR_PEER_MEASUREMENTS); + pmsr = nla_nest_start_noflag(msg, NL80211_ATTR_PEER_MEASUREMENTS); if (!pmsr) return -ENOBUFS; @@ -1731,7 +1757,7 @@ static int nl80211_send_pmsr_capa(struct cfg80211_registered_device *rdev, nla_put_flag(msg, NL80211_PMSR_ATTR_RANDOMIZE_MAC_ADDR)) return -ENOBUFS; - caps = nla_nest_start(msg, NL80211_PMSR_ATTR_TYPE_CAPA); + caps = nla_nest_start_noflag(msg, NL80211_PMSR_ATTR_TYPE_CAPA); if (!caps) return -ENOBUFS; @@ -1892,7 +1918,8 @@ static int nl80211_send_wiphy(struct cfg80211_registered_device *rdev, break; /* fall through */ case 3: - nl_bands = nla_nest_start(msg, NL80211_ATTR_WIPHY_BANDS); + nl_bands = nla_nest_start_noflag(msg, + NL80211_ATTR_WIPHY_BANDS); if (!nl_bands) goto nla_put_failure; @@ -1905,7 +1932,7 @@ static int nl80211_send_wiphy(struct cfg80211_registered_device *rdev, if (!sband) continue; - nl_band = nla_nest_start(msg, band); + nl_band = nla_nest_start_noflag(msg, band); if (!nl_band) goto nla_put_failure; @@ -1919,15 +1946,16 @@ static int nl80211_send_wiphy(struct cfg80211_registered_device *rdev, /* fall through */ default: /* add frequencies */ - nl_freqs = nla_nest_start( - msg, NL80211_BAND_ATTR_FREQS); + nl_freqs = nla_nest_start_noflag(msg, + NL80211_BAND_ATTR_FREQS); if (!nl_freqs) goto nla_put_failure; for (i = state->chan_start - 1; i < sband->n_channels; i++) { - nl_freq = nla_nest_start(msg, i); + nl_freq = nla_nest_start_noflag(msg, + i); if (!nl_freq) goto nla_put_failure; @@ -1972,7 +2000,8 @@ static int nl80211_send_wiphy(struct cfg80211_registered_device *rdev, break; /* fall through */ case 4: - nl_cmds = nla_nest_start(msg, NL80211_ATTR_SUPPORTED_COMMANDS); + nl_cmds = nla_nest_start_noflag(msg, + NL80211_ATTR_SUPPORTED_COMMANDS); if (!nl_cmds) goto nla_put_failure; @@ -2120,7 +2149,8 @@ static int nl80211_send_wiphy(struct cfg80211_registered_device *rdev, const struct nl80211_vendor_cmd_info *info; struct nlattr *nested; - nested = nla_nest_start(msg, NL80211_ATTR_VENDOR_DATA); + nested = nla_nest_start_noflag(msg, + NL80211_ATTR_VENDOR_DATA); if (!nested) goto nla_put_failure; @@ -2136,8 +2166,8 @@ static int nl80211_send_wiphy(struct cfg80211_registered_device *rdev, const struct nl80211_vendor_cmd_info *info; struct nlattr *nested; - nested = nla_nest_start(msg, - NL80211_ATTR_VENDOR_EVENTS); + nested = nla_nest_start_noflag(msg, + NL80211_ATTR_VENDOR_EVENTS); if (!nested) goto nla_put_failure; @@ -2174,7 +2204,8 @@ static int nl80211_send_wiphy(struct cfg80211_registered_device *rdev, struct nlattr *nested; u32 bss_select_support = rdev->wiphy.bss_select_support; - nested = nla_nest_start(msg, NL80211_ATTR_BSS_SELECT); + nested = nla_nest_start_noflag(msg, + NL80211_ATTR_BSS_SELECT); if (!nested) goto nla_put_failure; @@ -2196,8 +2227,8 @@ static int nl80211_send_wiphy(struct cfg80211_registered_device *rdev, rdev->wiphy.iftype_ext_capab) { struct nlattr *nested_ext_capab, *nested; - nested = nla_nest_start(msg, - NL80211_ATTR_IFTYPE_EXT_CAPA); + nested = nla_nest_start_noflag(msg, + NL80211_ATTR_IFTYPE_EXT_CAPA); if (!nested) goto nla_put_failure; @@ -2207,7 +2238,8 @@ static int nl80211_send_wiphy(struct cfg80211_registered_device *rdev, capab = &rdev->wiphy.iftype_ext_capab[i]; - nested_ext_capab = nla_nest_start(msg, i); + nested_ext_capab = nla_nest_start_noflag(msg, + i); if (!nested_ext_capab || nla_put_u32(msg, NL80211_ATTR_IFTYPE, capab->iftype) || @@ -2289,8 +2321,10 @@ static int nl80211_dump_wiphy_parse(struct sk_buff *skb, struct nl80211_dump_wiphy_state *state) { struct nlattr **tb = genl_family_attrbuf(&nl80211_fam); - int ret = nlmsg_parse(cb->nlh, GENL_HDRLEN + nl80211_fam.hdrsize, tb, - nl80211_fam.maxattr, nl80211_policy, NULL); + int ret = nlmsg_parse_deprecated(cb->nlh, + GENL_HDRLEN + nl80211_fam.hdrsize, + tb, nl80211_fam.maxattr, + nl80211_policy, NULL); /* ignore parse errors for backward compatibility */ if (ret) return 0; @@ -2733,10 +2767,11 @@ static int nl80211_set_wiphy(struct sk_buff *skb, struct genl_info *info) nla_for_each_nested(nl_txq_params, info->attrs[NL80211_ATTR_WIPHY_TXQ_PARAMS], rem_txq_params) { - result = nla_parse_nested(tb, NL80211_TXQ_ATTR_MAX, - nl_txq_params, - txq_params_policy, - info->extack); + result = nla_parse_nested_deprecated(tb, + NL80211_TXQ_ATTR_MAX, + nl_txq_params, + txq_params_policy, + info->extack); if (result) return result; result = parse_txq_params(tb, &txq_params); @@ -3193,8 +3228,7 @@ static int parse_monitor_flags(struct nlattr *nla, u32 *mntrflags) if (!nla) return -EINVAL; - if (nla_parse_nested(flags, NL80211_MNTR_FLAG_MAX, nla, - mntr_flags_policy, NULL)) + if (nla_parse_nested_deprecated(flags, NL80211_MNTR_FLAG_MAX, nla, mntr_flags_policy, NULL)) return -EINVAL; for (flag = 1; flag <= NL80211_MNTR_FLAG_MAX; flag++) @@ -3521,7 +3555,7 @@ static void get_key_callback(void *c, struct key_params *params) params->cipher))) goto nla_put_failure; - key = nla_nest_start(cookie->msg, NL80211_ATTR_KEY); + key = nla_nest_start_noflag(cookie->msg, NL80211_ATTR_KEY); if (!key) goto nla_put_failure; @@ -3634,8 +3668,11 @@ static int nl80211_set_key(struct sk_buff *skb, struct genl_info *info) if (key.idx < 0) return -EINVAL; - /* only support setting default key */ - if (!key.def && !key.defmgmt) + /* Only support setting default key and + * Extended Key ID action NL80211_KEY_SET_TX. + */ + if (!key.def && !key.defmgmt && + !(key.p.mode == NL80211_KEY_SET_TX)) return -EINVAL; wdev_lock(dev->ieee80211_ptr); @@ -3659,7 +3696,7 @@ static int nl80211_set_key(struct sk_buff *skb, struct genl_info *info) #ifdef CONFIG_CFG80211_WEXT dev->ieee80211_ptr->wext.default_key = key.idx; #endif - } else { + } else if (key.defmgmt) { if (key.def_uni || !key.def_multi) { err = -EINVAL; goto out; @@ -3681,8 +3718,25 @@ static int nl80211_set_key(struct sk_buff *skb, struct genl_info *info) #ifdef CONFIG_CFG80211_WEXT dev->ieee80211_ptr->wext.default_mgmt_key = key.idx; #endif - } + } else if (key.p.mode == NL80211_KEY_SET_TX && + wiphy_ext_feature_isset(&rdev->wiphy, + NL80211_EXT_FEATURE_EXT_KEY_ID)) { + u8 *mac_addr = NULL; + + if (info->attrs[NL80211_ATTR_MAC]) + mac_addr = nla_data(info->attrs[NL80211_ATTR_MAC]); + + if (!mac_addr || key.idx < 0 || key.idx > 1) { + err = -EINVAL; + goto out; + } + err = rdev_add_key(rdev, dev, key.idx, + NL80211_KEYTYPE_PAIRWISE, + mac_addr, &key.p); + } else { + err = -EINVAL; + } out: wdev_unlock(dev->ieee80211_ptr); @@ -3843,8 +3897,7 @@ static struct cfg80211_acl_data *parse_acl_data(struct wiphy *wiphy, if (n_entries > wiphy->max_acl_mac_addrs) return ERR_PTR(-ENOTSUPP); - acl = kzalloc(sizeof(*acl) + (sizeof(struct mac_address) * n_entries), - GFP_KERNEL); + acl = kzalloc(struct_size(acl, mac_addrs, n_entries), GFP_KERNEL); if (!acl) return ERR_PTR(-ENOMEM); @@ -4054,8 +4107,10 @@ static int nl80211_parse_tx_bitrate_mask(struct genl_info *info, sband = rdev->wiphy.bands[band]; if (sband == NULL) return -EINVAL; - err = nla_parse_nested(tb, NL80211_TXRATE_MAX, tx_rates, - nl80211_txattr_policy, info->extack); + err = nla_parse_nested_deprecated(tb, NL80211_TXRATE_MAX, + tx_rates, + nl80211_txattr_policy, + info->extack); if (err) return err; if (tb[NL80211_TXRATE_LEGACY]) { @@ -4223,9 +4278,10 @@ static int nl80211_parse_beacon(struct cfg80211_registered_device *rdev, if (attrs[NL80211_ATTR_FTM_RESPONDER]) { struct nlattr *tb[NL80211_FTM_RESP_ATTR_MAX + 1]; - err = nla_parse_nested(tb, NL80211_FTM_RESP_ATTR_MAX, - attrs[NL80211_ATTR_FTM_RESPONDER], - NULL, NULL); + err = nla_parse_nested_deprecated(tb, + NL80211_FTM_RESP_ATTR_MAX, + attrs[NL80211_ATTR_FTM_RESPONDER], + NULL, NULL); if (err) return err; @@ -4633,8 +4689,7 @@ static int parse_station_flags(struct genl_info *info, if (!nla) return 0; - if (nla_parse_nested(flags, NL80211_STA_FLAG_MAX, nla, - sta_flags_policy, info->extack)) + if (nla_parse_nested_deprecated(flags, NL80211_STA_FLAG_MAX, nla, sta_flags_policy, info->extack)) return -EINVAL; /* @@ -4686,7 +4741,7 @@ bool nl80211_put_sta_rate(struct sk_buff *msg, struct rate_info *info, int attr) u16 bitrate_compat; enum nl80211_rate_info rate_flg; - rate = nla_nest_start(msg, attr); + rate = nla_nest_start_noflag(msg, attr); if (!rate) return false; @@ -4773,7 +4828,7 @@ static bool nl80211_put_signal(struct sk_buff *msg, u8 mask, s8 *signal, if (!mask) return true; - attr = nla_nest_start(msg, id); + attr = nla_nest_start_noflag(msg, id); if (!attr) return false; @@ -4808,7 +4863,7 @@ static int nl80211_send_station(struct sk_buff *msg, u32 cmd, u32 portid, nla_put_u32(msg, NL80211_ATTR_GENERATION, sinfo->generation)) goto nla_put_failure; - sinfoattr = nla_nest_start(msg, NL80211_ATTR_STA_INFO); + sinfoattr = nla_nest_start_noflag(msg, NL80211_ATTR_STA_INFO); if (!sinfoattr) goto nla_put_failure; @@ -4889,6 +4944,7 @@ static int nl80211_send_station(struct sk_buff *msg, u32 cmd, u32 portid, PUT_SINFO(TX_RETRIES, tx_retries, u32); PUT_SINFO(TX_FAILED, tx_failed, u32); PUT_SINFO(EXPECTED_THROUGHPUT, expected_throughput, u32); + PUT_SINFO(AIRTIME_LINK_METRIC, airtime_link_metric, u32); PUT_SINFO(BEACON_LOSS, beacon_loss_count, u32); PUT_SINFO(LOCAL_PM, local_pm, u32); PUT_SINFO(PEER_PM, peer_pm, u32); @@ -4896,7 +4952,8 @@ static int nl80211_send_station(struct sk_buff *msg, u32 cmd, u32 portid, PUT_SINFO(CONNECTED_TO_GATE, connected_to_gate, u8); if (sinfo->filled & BIT_ULL(NL80211_STA_INFO_BSS_PARAM)) { - bss_param = nla_nest_start(msg, NL80211_STA_INFO_BSS_PARAM); + bss_param = nla_nest_start_noflag(msg, + NL80211_STA_INFO_BSS_PARAM); if (!bss_param) goto nla_put_failure; @@ -4939,7 +4996,8 @@ static int nl80211_send_station(struct sk_buff *msg, u32 cmd, u32 portid, struct nlattr *tidsattr; int tid; - tidsattr = nla_nest_start(msg, NL80211_STA_INFO_TID_STATS); + tidsattr = nla_nest_start_noflag(msg, + NL80211_STA_INFO_TID_STATS); if (!tidsattr) goto nla_put_failure; @@ -4952,7 +5010,7 @@ static int nl80211_send_station(struct sk_buff *msg, u32 cmd, u32 portid, if (!tidstats->filled) continue; - tidattr = nla_nest_start(msg, tid + 1); + tidattr = nla_nest_start_noflag(msg, tid + 1); if (!tidattr) goto nla_put_failure; @@ -5300,8 +5358,9 @@ static int nl80211_parse_sta_wme(struct genl_info *info, return 0; nla = info->attrs[NL80211_ATTR_STA_WME]; - err = nla_parse_nested(tb, NL80211_STA_WME_MAX, nla, - nl80211_sta_wme_policy, info->extack); + err = nla_parse_nested_deprecated(tb, NL80211_STA_WME_MAX, nla, + nl80211_sta_wme_policy, + info->extack); if (err) return err; @@ -5387,6 +5446,36 @@ static int nl80211_set_station_tdls(struct genl_info *info, return nl80211_parse_sta_wme(info, params); } +static int nl80211_parse_sta_txpower_setting(struct genl_info *info, + struct station_parameters *params) +{ + struct cfg80211_registered_device *rdev = info->user_ptr[0]; + int idx; + + if (info->attrs[NL80211_ATTR_STA_TX_POWER_SETTING]) { + if (!rdev->ops->set_tx_power || + !wiphy_ext_feature_isset(&rdev->wiphy, + NL80211_EXT_FEATURE_STA_TX_PWR)) + return -EOPNOTSUPP; + + idx = NL80211_ATTR_STA_TX_POWER_SETTING; + params->txpwr.type = nla_get_u8(info->attrs[idx]); + + if (params->txpwr.type == NL80211_TX_POWER_LIMITED) { + idx = NL80211_ATTR_STA_TX_POWER; + + if (info->attrs[idx]) + params->txpwr.power = + nla_get_s16(info->attrs[idx]); + else + return -EINVAL; + } + params->sta_modify_mask |= STATION_PARAM_APPLY_STA_TXPOWER; + } + + return 0; +} + static int nl80211_set_station(struct sk_buff *skb, struct genl_info *info) { struct cfg80211_registered_device *rdev = info->user_ptr[0]; @@ -5480,6 +5569,10 @@ static int nl80211_set_station(struct sk_buff *skb, struct genl_info *info) NL80211_EXT_FEATURE_AIRTIME_FAIRNESS)) return -EOPNOTSUPP; + err = nl80211_parse_sta_txpower_setting(info, ¶ms); + if (err) + return err; + /* Include parameters for TDLS peer (will check later) */ err = nl80211_set_station_tdls(info, ¶ms); if (err) @@ -5617,6 +5710,10 @@ static int nl80211_new_station(struct sk_buff *skb, struct genl_info *info) NL80211_EXT_FEATURE_AIRTIME_FAIRNESS)) return -EOPNOTSUPP; + err = nl80211_parse_sta_txpower_setting(info, ¶ms); + if (err) + return err; + err = nl80211_parse_sta_channel_info(info, ¶ms); if (err) return err; @@ -5799,7 +5896,7 @@ static int nl80211_send_mpath(struct sk_buff *msg, u32 portid, u32 seq, nla_put_u32(msg, NL80211_ATTR_GENERATION, pinfo->generation)) goto nla_put_failure; - pinfoattr = nla_nest_start(msg, NL80211_ATTR_MPATH_INFO); + pinfoattr = nla_nest_start_noflag(msg, NL80211_ATTR_MPATH_INFO); if (!pinfoattr) goto nla_put_failure; if ((pinfo->filled & MPATH_INFO_FRAME_QLEN) && @@ -6250,7 +6347,7 @@ static int nl80211_get_mesh_config(struct sk_buff *skb, NL80211_CMD_GET_MESH_CONFIG); if (!hdr) goto out; - pinfoattr = nla_nest_start(msg, NL80211_ATTR_MESH_CONFIG); + pinfoattr = nla_nest_start_noflag(msg, NL80211_ATTR_MESH_CONFIG); if (!pinfoattr) goto nla_put_failure; if (nla_put_u32(msg, NL80211_ATTR_IFINDEX, dev->ifindex) || @@ -6403,9 +6500,7 @@ do { \ if (!info->attrs[NL80211_ATTR_MESH_CONFIG]) return -EINVAL; - if (nla_parse_nested(tb, NL80211_MESHCONF_ATTR_MAX, - info->attrs[NL80211_ATTR_MESH_CONFIG], - nl80211_meshconf_params_policy, info->extack)) + if (nla_parse_nested_deprecated(tb, NL80211_MESHCONF_ATTR_MAX, info->attrs[NL80211_ATTR_MESH_CONFIG], nl80211_meshconf_params_policy, info->extack)) return -EINVAL; /* This makes sure that there aren't more than 32 mesh config @@ -6538,9 +6633,7 @@ static int nl80211_parse_mesh_setup(struct genl_info *info, if (!info->attrs[NL80211_ATTR_MESH_SETUP]) return -EINVAL; - if (nla_parse_nested(tb, NL80211_MESH_SETUP_ATTR_MAX, - info->attrs[NL80211_ATTR_MESH_SETUP], - nl80211_mesh_setup_params_policy, info->extack)) + if (nla_parse_nested_deprecated(tb, NL80211_MESH_SETUP_ATTR_MAX, info->attrs[NL80211_ATTR_MESH_SETUP], nl80211_mesh_setup_params_policy, info->extack)) return -EINVAL; if (tb[NL80211_MESH_SETUP_ENABLE_VENDOR_SYNC]) @@ -6629,7 +6722,7 @@ static int nl80211_put_regdom(const struct ieee80211_regdomain *regdom, nla_put_u8(msg, NL80211_ATTR_DFS_REGION, regdom->dfs_region))) goto nla_put_failure; - nl_reg_rules = nla_nest_start(msg, NL80211_ATTR_REG_RULES); + nl_reg_rules = nla_nest_start_noflag(msg, NL80211_ATTR_REG_RULES); if (!nl_reg_rules) goto nla_put_failure; @@ -6644,7 +6737,7 @@ static int nl80211_put_regdom(const struct ieee80211_regdomain *regdom, freq_range = ®_rule->freq_range; power_rule = ®_rule->power_rule; - nl_reg_rule = nla_nest_start(msg, i); + nl_reg_rule = nla_nest_start_noflag(msg, i); if (!nl_reg_rule) goto nla_put_failure; @@ -6882,7 +6975,7 @@ static int nl80211_set_reg(struct sk_buff *skb, struct genl_info *info) struct nlattr *nl_reg_rule; char *alpha2; int rem_reg_rules, r; - u32 num_rules = 0, rule_idx = 0, size_of_regd; + u32 num_rules = 0, rule_idx = 0; enum nl80211_dfs_regions dfs_region = NL80211_DFS_UNSET; struct ieee80211_regdomain *rd; @@ -6907,10 +7000,7 @@ static int nl80211_set_reg(struct sk_buff *skb, struct genl_info *info) if (!reg_is_valid_request(alpha2)) return -EINVAL; - size_of_regd = sizeof(struct ieee80211_regdomain) + - num_rules * sizeof(struct ieee80211_reg_rule); - - rd = kzalloc(size_of_regd, GFP_KERNEL); + rd = kzalloc(struct_size(rd, reg_rules, num_rules), GFP_KERNEL); if (!rd) return -ENOMEM; @@ -6927,9 +7017,9 @@ static int nl80211_set_reg(struct sk_buff *skb, struct genl_info *info) nla_for_each_nested(nl_reg_rule, info->attrs[NL80211_ATTR_REG_RULES], rem_reg_rules) { - r = nla_parse_nested(tb, NL80211_REG_RULE_ATTR_MAX, - nl_reg_rule, reg_rule_policy, - info->extack); + r = nla_parse_nested_deprecated(tb, NL80211_REG_RULE_ATTR_MAX, + nl_reg_rule, reg_rule_policy, + info->extack); if (r) goto bad_reg; r = parse_reg_rule(tb, &rd->reg_rules[rule_idx]); @@ -7000,8 +7090,9 @@ static int parse_bss_select(struct nlattr *nla, struct wiphy *wiphy, if (!nla_ok(nest, nla_len(nest))) return -EINVAL; - err = nla_parse_nested(attr, NL80211_BSS_SELECT_ATTR_MAX, nest, - nl80211_bss_select_policy, NULL); + err = nla_parse_nested_deprecated(attr, NL80211_BSS_SELECT_ATTR_MAX, + nest, nl80211_bss_select_policy, + NULL); if (err) return err; @@ -7494,8 +7585,10 @@ nl80211_parse_sched_scan_plans(struct wiphy *wiphy, int n_plans, if (WARN_ON(i >= n_plans)) return -EINVAL; - err = nla_parse_nested(plan, NL80211_SCHED_SCAN_PLAN_MAX, - attr, nl80211_plan_policy, NULL); + err = nla_parse_nested_deprecated(plan, + NL80211_SCHED_SCAN_PLAN_MAX, + attr, nl80211_plan_policy, + NULL); if (err) return err; @@ -7537,6 +7630,41 @@ nl80211_parse_sched_scan_plans(struct wiphy *wiphy, int n_plans, return 0; } +static int +nl80211_parse_sched_scan_per_band_rssi(struct wiphy *wiphy, + struct cfg80211_match_set *match_sets, + struct nlattr *tb_band_rssi, + s32 rssi_thold) +{ + struct nlattr *attr; + int i, tmp, ret = 0; + + if (!wiphy_ext_feature_isset(wiphy, + NL80211_EXT_FEATURE_SCHED_SCAN_BAND_SPECIFIC_RSSI_THOLD)) { + if (tb_band_rssi) + ret = -EOPNOTSUPP; + else + for (i = 0; i < NUM_NL80211_BANDS; i++) + match_sets->per_band_rssi_thold[i] = + NL80211_SCAN_RSSI_THOLD_OFF; + return ret; + } + + for (i = 0; i < NUM_NL80211_BANDS; i++) + match_sets->per_band_rssi_thold[i] = rssi_thold; + + nla_for_each_nested(attr, tb_band_rssi, tmp) { + enum nl80211_band band = nla_type(attr); + + if (band < 0 || band >= NUM_NL80211_BANDS) + return -EINVAL; + + match_sets->per_band_rssi_thold[band] = nla_get_s32(attr); + } + + return 0; +} + static struct cfg80211_sched_scan_request * nl80211_parse_sched_scan(struct wiphy *wiphy, struct wireless_dev *wdev, struct nlattr **attrs, int max_match_sets) @@ -7581,10 +7709,11 @@ nl80211_parse_sched_scan(struct wiphy *wiphy, struct wireless_dev *wdev, tmp) { struct nlattr *rssi; - err = nla_parse_nested(tb, - NL80211_SCHED_SCAN_MATCH_ATTR_MAX, - attr, nl80211_match_policy, - NULL); + err = nla_parse_nested_deprecated(tb, + NL80211_SCHED_SCAN_MATCH_ATTR_MAX, + attr, + nl80211_match_policy, + NULL); if (err) return ERR_PTR(err); @@ -7768,51 +7897,64 @@ nl80211_parse_sched_scan(struct wiphy *wiphy, struct wireless_dev *wdev, tmp) { struct nlattr *ssid, *bssid, *rssi; - err = nla_parse_nested(tb, - NL80211_SCHED_SCAN_MATCH_ATTR_MAX, - attr, nl80211_match_policy, - NULL); + err = nla_parse_nested_deprecated(tb, + NL80211_SCHED_SCAN_MATCH_ATTR_MAX, + attr, + nl80211_match_policy, + NULL); if (err) goto out_free; ssid = tb[NL80211_SCHED_SCAN_MATCH_ATTR_SSID]; bssid = tb[NL80211_SCHED_SCAN_MATCH_ATTR_BSSID]; - if (ssid || bssid) { - if (WARN_ON(i >= n_match_sets)) { - /* this indicates a programming error, - * the loop above should have verified - * things properly - */ + + if (!ssid && !bssid) { + i++; + continue; + } + + if (WARN_ON(i >= n_match_sets)) { + /* this indicates a programming error, + * the loop above should have verified + * things properly + */ + err = -EINVAL; + goto out_free; + } + + if (ssid) { + if (nla_len(ssid) > IEEE80211_MAX_SSID_LEN) { err = -EINVAL; goto out_free; } - - if (ssid) { - if (nla_len(ssid) > IEEE80211_MAX_SSID_LEN) { - err = -EINVAL; - goto out_free; - } - memcpy(request->match_sets[i].ssid.ssid, - nla_data(ssid), nla_len(ssid)); - request->match_sets[i].ssid.ssid_len = - nla_len(ssid); - } - if (bssid) { - if (nla_len(bssid) != ETH_ALEN) { - err = -EINVAL; - goto out_free; - } - memcpy(request->match_sets[i].bssid, - nla_data(bssid), ETH_ALEN); + memcpy(request->match_sets[i].ssid.ssid, + nla_data(ssid), nla_len(ssid)); + request->match_sets[i].ssid.ssid_len = + nla_len(ssid); + } + if (bssid) { + if (nla_len(bssid) != ETH_ALEN) { + err = -EINVAL; + goto out_free; } + memcpy(request->match_sets[i].bssid, + nla_data(bssid), ETH_ALEN); + } - /* special attribute - old implementation w/a */ + /* special attribute - old implementation w/a */ + request->match_sets[i].rssi_thold = default_match_rssi; + rssi = tb[NL80211_SCHED_SCAN_MATCH_ATTR_RSSI]; + if (rssi) request->match_sets[i].rssi_thold = - default_match_rssi; - rssi = tb[NL80211_SCHED_SCAN_MATCH_ATTR_RSSI]; - if (rssi) - request->match_sets[i].rssi_thold = - nla_get_s32(rssi); - } + nla_get_s32(rssi); + + /* Parse per band RSSI attribute */ + err = nl80211_parse_sched_scan_per_band_rssi(wiphy, + &request->match_sets[i], + tb[NL80211_SCHED_SCAN_MATCH_PER_BAND_RSSI], + request->match_sets[i].rssi_thold); + if (err) + goto out_free; + i++; } @@ -8061,7 +8203,7 @@ static int nl80211_notify_radar_detection(struct sk_buff *skb, cfg80211_sched_dfs_chan_update(rdev); - memcpy(&rdev->radar_chandef, &chandef, sizeof(chandef)); + rdev->radar_chandef = chandef; /* Propagate this notification to other radios as well */ queue_work(cfg80211_wq, &rdev->propagate_radar_detect_wk); @@ -8143,9 +8285,9 @@ static int nl80211_channel_switch(struct sk_buff *skb, struct genl_info *info) if (err) return err; - err = nla_parse_nested(csa_attrs, NL80211_ATTR_MAX, - info->attrs[NL80211_ATTR_CSA_IES], - nl80211_policy, info->extack); + err = nla_parse_nested_deprecated(csa_attrs, NL80211_ATTR_MAX, + info->attrs[NL80211_ATTR_CSA_IES], + nl80211_policy, info->extack); if (err) return err; @@ -8269,7 +8411,7 @@ static int nl80211_send_bss(struct sk_buff *msg, struct netlink_callback *cb, NL80211_ATTR_PAD)) goto nla_put_failure; - bss = nla_nest_start(msg, NL80211_ATTR_BSS); + bss = nla_nest_start_noflag(msg, NL80211_ATTR_BSS); if (!bss) goto nla_put_failure; if ((!is_zero_ether_addr(res->bssid) && @@ -8446,7 +8588,7 @@ static int nl80211_send_survey(struct sk_buff *msg, u32 portid, u32 seq, if (nla_put_u32(msg, NL80211_ATTR_IFINDEX, dev->ifindex)) goto nla_put_failure; - infoattr = nla_nest_start(msg, NL80211_ATTR_SURVEY_INFO); + infoattr = nla_nest_start_noflag(msg, NL80211_ATTR_SURVEY_INFO); if (!infoattr) goto nla_put_failure; @@ -9287,7 +9429,7 @@ __cfg80211_alloc_vendor_skb(struct cfg80211_registered_device *rdev, goto nla_put_failure; } - data = nla_nest_start(skb, attr); + data = nla_nest_start_noflag(skb, attr); if (!data) goto nla_put_failure; @@ -9420,9 +9562,10 @@ static int nl80211_testmode_dump(struct sk_buff *skb, } else { struct nlattr **attrbuf = genl_family_attrbuf(&nl80211_fam); - err = nlmsg_parse(cb->nlh, GENL_HDRLEN + nl80211_fam.hdrsize, - attrbuf, nl80211_fam.maxattr, - nl80211_policy, NULL); + err = nlmsg_parse_deprecated(cb->nlh, + GENL_HDRLEN + nl80211_fam.hdrsize, + attrbuf, nl80211_fam.maxattr, + nl80211_policy, NULL); if (err) goto out_err; @@ -9461,7 +9604,7 @@ static int nl80211_testmode_dump(struct sk_buff *skb, break; } - tmdata = nla_nest_start(skb, NL80211_ATTR_TESTDATA); + tmdata = nla_nest_start_noflag(skb, NL80211_ATTR_TESTDATA); if (!tmdata) { genlmsg_cancel(skb, hdr); break; @@ -10546,8 +10689,9 @@ static int nl80211_set_cqm(struct sk_buff *skb, struct genl_info *info) if (!cqm) return -EINVAL; - err = nla_parse_nested(attrs, NL80211_ATTR_CQM_MAX, cqm, - nl80211_attr_cqm_policy, info->extack); + err = nla_parse_nested_deprecated(attrs, NL80211_ATTR_CQM_MAX, cqm, + nl80211_attr_cqm_policy, + info->extack); if (err) return err; @@ -10739,12 +10883,12 @@ static int nl80211_send_wowlan_patterns(struct sk_buff *msg, if (!wowlan->n_patterns) return 0; - nl_pats = nla_nest_start(msg, NL80211_WOWLAN_TRIG_PKT_PATTERN); + nl_pats = nla_nest_start_noflag(msg, NL80211_WOWLAN_TRIG_PKT_PATTERN); if (!nl_pats) return -ENOBUFS; for (i = 0; i < wowlan->n_patterns; i++) { - nl_pat = nla_nest_start(msg, i + 1); + nl_pat = nla_nest_start_noflag(msg, i + 1); if (!nl_pat) return -ENOBUFS; pat_len = wowlan->patterns[i].pattern_len; @@ -10770,7 +10914,8 @@ static int nl80211_send_wowlan_tcp(struct sk_buff *msg, if (!tcp) return 0; - nl_tcp = nla_nest_start(msg, NL80211_WOWLAN_TRIG_TCP_CONNECTION); + nl_tcp = nla_nest_start_noflag(msg, + NL80211_WOWLAN_TRIG_TCP_CONNECTION); if (!nl_tcp) return -ENOBUFS; @@ -10814,7 +10959,7 @@ static int nl80211_send_wowlan_nd(struct sk_buff *msg, if (!req) return 0; - nd = nla_nest_start(msg, NL80211_WOWLAN_TRIG_NET_DETECT); + nd = nla_nest_start_noflag(msg, NL80211_WOWLAN_TRIG_NET_DETECT); if (!nd) return -ENOBUFS; @@ -10840,7 +10985,7 @@ static int nl80211_send_wowlan_nd(struct sk_buff *msg, return -ENOBUFS; } - freqs = nla_nest_start(msg, NL80211_ATTR_SCAN_FREQUENCIES); + freqs = nla_nest_start_noflag(msg, NL80211_ATTR_SCAN_FREQUENCIES); if (!freqs) return -ENOBUFS; @@ -10852,12 +10997,13 @@ static int nl80211_send_wowlan_nd(struct sk_buff *msg, nla_nest_end(msg, freqs); if (req->n_match_sets) { - matches = nla_nest_start(msg, NL80211_ATTR_SCHED_SCAN_MATCH); + matches = nla_nest_start_noflag(msg, + NL80211_ATTR_SCHED_SCAN_MATCH); if (!matches) return -ENOBUFS; for (i = 0; i < req->n_match_sets; i++) { - match = nla_nest_start(msg, i); + match = nla_nest_start_noflag(msg, i); if (!match) return -ENOBUFS; @@ -10870,12 +11016,12 @@ static int nl80211_send_wowlan_nd(struct sk_buff *msg, nla_nest_end(msg, matches); } - scan_plans = nla_nest_start(msg, NL80211_ATTR_SCHED_SCAN_PLANS); + scan_plans = nla_nest_start_noflag(msg, NL80211_ATTR_SCHED_SCAN_PLANS); if (!scan_plans) return -ENOBUFS; for (i = 0; i < req->n_scan_plans; i++) { - scan_plan = nla_nest_start(msg, i + 1); + scan_plan = nla_nest_start_noflag(msg, i + 1); if (!scan_plan) return -ENOBUFS; @@ -10924,7 +11070,8 @@ static int nl80211_get_wowlan(struct sk_buff *skb, struct genl_info *info) if (rdev->wiphy.wowlan_config) { struct nlattr *nl_wowlan; - nl_wowlan = nla_nest_start(msg, NL80211_ATTR_WOWLAN_TRIGGERS); + nl_wowlan = nla_nest_start_noflag(msg, + NL80211_ATTR_WOWLAN_TRIGGERS); if (!nl_wowlan) goto nla_put_failure; @@ -10982,8 +11129,8 @@ static int nl80211_parse_wowlan_tcp(struct cfg80211_registered_device *rdev, if (!rdev->wiphy.wowlan->tcp) return -EINVAL; - err = nla_parse_nested(tb, MAX_NL80211_WOWLAN_TCP, attr, - nl80211_wowlan_tcp_policy, NULL); + err = nla_parse_nested_deprecated(tb, MAX_NL80211_WOWLAN_TCP, attr, + nl80211_wowlan_tcp_policy, NULL); if (err) return err; @@ -11128,8 +11275,8 @@ static int nl80211_parse_wowlan_nd(struct cfg80211_registered_device *rdev, goto out; } - err = nla_parse_nested(tb, NL80211_ATTR_MAX, attr, nl80211_policy, - NULL); + err = nla_parse_nested_deprecated(tb, NL80211_ATTR_MAX, attr, + nl80211_policy, NULL); if (err) goto out; @@ -11164,9 +11311,9 @@ static int nl80211_set_wowlan(struct sk_buff *skb, struct genl_info *info) goto set_wakeup; } - err = nla_parse_nested(tb, MAX_NL80211_WOWLAN_TRIG, - info->attrs[NL80211_ATTR_WOWLAN_TRIGGERS], - nl80211_wowlan_policy, info->extack); + err = nla_parse_nested_deprecated(tb, MAX_NL80211_WOWLAN_TRIG, + info->attrs[NL80211_ATTR_WOWLAN_TRIGGERS], + nl80211_wowlan_policy, info->extack); if (err) return err; @@ -11248,9 +11395,11 @@ static int nl80211_set_wowlan(struct sk_buff *skb, struct genl_info *info) rem) { u8 *mask_pat; - err = nla_parse_nested(pat_tb, MAX_NL80211_PKTPAT, pat, - nl80211_packet_pattern_policy, - info->extack); + err = nla_parse_nested_deprecated(pat_tb, + MAX_NL80211_PKTPAT, + pat, + nl80211_packet_pattern_policy, + info->extack); if (err) goto error; @@ -11358,12 +11507,12 @@ static int nl80211_send_coalesce_rules(struct sk_buff *msg, if (!rdev->coalesce->n_rules) return 0; - nl_rules = nla_nest_start(msg, NL80211_ATTR_COALESCE_RULE); + nl_rules = nla_nest_start_noflag(msg, NL80211_ATTR_COALESCE_RULE); if (!nl_rules) return -ENOBUFS; for (i = 0; i < rdev->coalesce->n_rules; i++) { - nl_rule = nla_nest_start(msg, i + 1); + nl_rule = nla_nest_start_noflag(msg, i + 1); if (!nl_rule) return -ENOBUFS; @@ -11376,13 +11525,13 @@ static int nl80211_send_coalesce_rules(struct sk_buff *msg, rule->condition)) return -ENOBUFS; - nl_pats = nla_nest_start(msg, - NL80211_ATTR_COALESCE_RULE_PKT_PATTERN); + nl_pats = nla_nest_start_noflag(msg, + NL80211_ATTR_COALESCE_RULE_PKT_PATTERN); if (!nl_pats) return -ENOBUFS; for (j = 0; j < rule->n_patterns; j++) { - nl_pat = nla_nest_start(msg, j + 1); + nl_pat = nla_nest_start_noflag(msg, j + 1); if (!nl_pat) return -ENOBUFS; pat_len = rule->patterns[j].pattern_len; @@ -11463,8 +11612,8 @@ static int nl80211_parse_coalesce_rule(struct cfg80211_registered_device *rdev, int rem, pat_len, mask_len, pkt_offset, n_patterns = 0; struct nlattr *pat_tb[NUM_NL80211_PKTPAT]; - err = nla_parse_nested(tb, NL80211_ATTR_COALESCE_RULE_MAX, rule, - nl80211_coalesce_policy, NULL); + err = nla_parse_nested_deprecated(tb, NL80211_ATTR_COALESCE_RULE_MAX, + rule, nl80211_coalesce_policy, NULL); if (err) return err; @@ -11499,8 +11648,10 @@ static int nl80211_parse_coalesce_rule(struct cfg80211_registered_device *rdev, rem) { u8 *mask_pat; - err = nla_parse_nested(pat_tb, MAX_NL80211_PKTPAT, pat, - nl80211_packet_pattern_policy, NULL); + err = nla_parse_nested_deprecated(pat_tb, MAX_NL80211_PKTPAT, + pat, + nl80211_packet_pattern_policy, + NULL); if (err) return err; @@ -11622,9 +11773,9 @@ static int nl80211_set_rekey_data(struct sk_buff *skb, struct genl_info *info) if (!info->attrs[NL80211_ATTR_REKEY_DATA]) return -EINVAL; - err = nla_parse_nested(tb, MAX_NL80211_REKEY_DATA, - info->attrs[NL80211_ATTR_REKEY_DATA], - nl80211_rekey_policy, info->extack); + err = nla_parse_nested_deprecated(tb, MAX_NL80211_REKEY_DATA, + info->attrs[NL80211_ATTR_REKEY_DATA], + nl80211_rekey_policy, info->extack); if (err) return err; @@ -11936,9 +12087,10 @@ static int nl80211_nan_add_func(struct sk_buff *skb, if (!info->attrs[NL80211_ATTR_NAN_FUNC]) return -EINVAL; - err = nla_parse_nested(tb, NL80211_NAN_FUNC_ATTR_MAX, - info->attrs[NL80211_ATTR_NAN_FUNC], - nl80211_nan_func_policy, info->extack); + err = nla_parse_nested_deprecated(tb, NL80211_NAN_FUNC_ATTR_MAX, + info->attrs[NL80211_ATTR_NAN_FUNC], + nl80211_nan_func_policy, + info->extack); if (err) return err; @@ -12034,9 +12186,11 @@ static int nl80211_nan_add_func(struct sk_buff *skb, if (tb[NL80211_NAN_FUNC_SRF]) { struct nlattr *srf_tb[NUM_NL80211_NAN_SRF_ATTR]; - err = nla_parse_nested(srf_tb, NL80211_NAN_SRF_ATTR_MAX, - tb[NL80211_NAN_FUNC_SRF], - nl80211_nan_srf_policy, info->extack); + err = nla_parse_nested_deprecated(srf_tb, + NL80211_NAN_SRF_ATTR_MAX, + tb[NL80211_NAN_FUNC_SRF], + nl80211_nan_srf_policy, + info->extack); if (err) goto out; @@ -12134,7 +12288,7 @@ out: NL80211_ATTR_PAD)) goto nla_put_failure; - func_attr = nla_nest_start(msg, NL80211_ATTR_NAN_FUNC); + func_attr = nla_nest_start_noflag(msg, NL80211_ATTR_NAN_FUNC); if (!func_attr) goto nla_put_failure; @@ -12251,11 +12405,12 @@ void cfg80211_nan_match(struct wireless_dev *wdev, nla_put(msg, NL80211_ATTR_MAC, ETH_ALEN, match->addr)) goto nla_put_failure; - match_attr = nla_nest_start(msg, NL80211_ATTR_NAN_MATCH); + match_attr = nla_nest_start_noflag(msg, NL80211_ATTR_NAN_MATCH); if (!match_attr) goto nla_put_failure; - local_func_attr = nla_nest_start(msg, NL80211_NAN_MATCH_FUNC_LOCAL); + local_func_attr = nla_nest_start_noflag(msg, + NL80211_NAN_MATCH_FUNC_LOCAL); if (!local_func_attr) goto nla_put_failure; @@ -12264,7 +12419,8 @@ void cfg80211_nan_match(struct wireless_dev *wdev, nla_nest_end(msg, local_func_attr); - peer_func_attr = nla_nest_start(msg, NL80211_NAN_MATCH_FUNC_PEER); + peer_func_attr = nla_nest_start_noflag(msg, + NL80211_NAN_MATCH_FUNC_PEER); if (!peer_func_attr) goto nla_put_failure; @@ -12330,7 +12486,7 @@ void cfg80211_nan_func_terminated(struct wireless_dev *wdev, NL80211_ATTR_PAD)) goto nla_put_failure; - func_attr = nla_nest_start(msg, NL80211_ATTR_NAN_FUNC); + func_attr = nla_nest_start_noflag(msg, NL80211_ATTR_NAN_FUNC); if (!func_attr) goto nla_put_failure; @@ -12567,8 +12723,10 @@ static int nl80211_prepare_vendor_dump(struct sk_buff *skb, return 0; } - err = nlmsg_parse(cb->nlh, GENL_HDRLEN + nl80211_fam.hdrsize, attrbuf, - nl80211_fam.maxattr, nl80211_policy, NULL); + err = nlmsg_parse_deprecated(cb->nlh, + GENL_HDRLEN + nl80211_fam.hdrsize, + attrbuf, nl80211_fam.maxattr, + nl80211_policy, NULL); if (err) return err; @@ -12679,7 +12837,8 @@ static int nl80211_vendor_cmd_dump(struct sk_buff *skb, break; } - vendor_data = nla_nest_start(skb, NL80211_ATTR_VENDOR_DATA); + vendor_data = nla_nest_start_noflag(skb, + NL80211_ATTR_VENDOR_DATA); if (!vendor_data) { genlmsg_cancel(skb, hdr); break; @@ -13223,7 +13382,8 @@ static int nl80211_get_ftm_responder_stats(struct sk_buff *skb, if (nla_put_u32(msg, NL80211_ATTR_IFINDEX, dev->ifindex)) goto nla_put_failure; - ftm_stats_attr = nla_nest_start(msg, NL80211_ATTR_FTM_RESPONDER_STATS); + ftm_stats_attr = nla_nest_start_noflag(msg, + NL80211_ATTR_FTM_RESPONDER_STATS); if (!ftm_stats_attr) goto nla_put_failure; @@ -13259,6 +13419,72 @@ nla_put_failure: return -ENOBUFS; } +static int nl80211_update_owe_info(struct sk_buff *skb, struct genl_info *info) +{ + struct cfg80211_registered_device *rdev = info->user_ptr[0]; + struct cfg80211_update_owe_info owe_info; + struct net_device *dev = info->user_ptr[1]; + + if (!rdev->ops->update_owe_info) + return -EOPNOTSUPP; + + if (!info->attrs[NL80211_ATTR_STATUS_CODE] || + !info->attrs[NL80211_ATTR_MAC]) + return -EINVAL; + + memset(&owe_info, 0, sizeof(owe_info)); + owe_info.status = nla_get_u16(info->attrs[NL80211_ATTR_STATUS_CODE]); + nla_memcpy(owe_info.peer, info->attrs[NL80211_ATTR_MAC], ETH_ALEN); + + if (info->attrs[NL80211_ATTR_IE]) { + owe_info.ie = nla_data(info->attrs[NL80211_ATTR_IE]); + owe_info.ie_len = nla_len(info->attrs[NL80211_ATTR_IE]); + } + + return rdev_update_owe_info(rdev, dev, &owe_info); +} + +static int nl80211_probe_mesh_link(struct sk_buff *skb, struct genl_info *info) +{ + struct cfg80211_registered_device *rdev = info->user_ptr[0]; + struct net_device *dev = info->user_ptr[1]; + struct wireless_dev *wdev = dev->ieee80211_ptr; + struct station_info sinfo = {}; + const u8 *buf; + size_t len; + u8 *dest; + int err; + + if (!rdev->ops->probe_mesh_link || !rdev->ops->get_station) + return -EOPNOTSUPP; + + if (!info->attrs[NL80211_ATTR_MAC] || + !info->attrs[NL80211_ATTR_FRAME]) { + GENL_SET_ERR_MSG(info, "Frame or MAC missing"); + return -EINVAL; + } + + if (wdev->iftype != NL80211_IFTYPE_MESH_POINT) + return -EOPNOTSUPP; + + dest = nla_data(info->attrs[NL80211_ATTR_MAC]); + buf = nla_data(info->attrs[NL80211_ATTR_FRAME]); + len = nla_len(info->attrs[NL80211_ATTR_FRAME]); + + if (len < sizeof(struct ethhdr)) + return -EINVAL; + + if (!ether_addr_equal(buf, dest) || is_multicast_ether_addr(buf) || + !ether_addr_equal(buf + ETH_ALEN, dev->dev_addr)) + return -EINVAL; + + err = rdev_get_station(rdev, dev, dest, &sinfo); + if (err) + return err; + + return rdev_probe_mesh_link(rdev, dev, dest, buf, len); +} + #define NL80211_FLAG_NEED_WIPHY 0x01 #define NL80211_FLAG_NEED_NETDEV 0x02 #define NL80211_FLAG_NEED_RTNL 0x04 @@ -13365,66 +13591,66 @@ static void nl80211_post_doit(const struct genl_ops *ops, struct sk_buff *skb, static const struct genl_ops nl80211_ops[] = { { .cmd = NL80211_CMD_GET_WIPHY, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_get_wiphy, .dumpit = nl80211_dump_wiphy, .done = nl80211_dump_wiphy_done, - .policy = nl80211_policy, /* can be retrieved by unprivileged users */ .internal_flags = NL80211_FLAG_NEED_WIPHY | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_WIPHY, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_wiphy, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_GET_INTERFACE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_get_interface, .dumpit = nl80211_dump_interface, - .policy = nl80211_policy, /* can be retrieved by unprivileged users */ .internal_flags = NL80211_FLAG_NEED_WDEV | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_INTERFACE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_interface, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_NEW_INTERFACE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_new_interface, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WIPHY | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_DEL_INTERFACE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_del_interface, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WDEV | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_GET_KEY, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_get_key, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_KEY, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_key, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL | @@ -13432,8 +13658,8 @@ static const struct genl_ops nl80211_ops[] = { }, { .cmd = NL80211_CMD_NEW_KEY, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_new_key, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL | @@ -13441,15 +13667,15 @@ static const struct genl_ops nl80211_ops[] = { }, { .cmd = NL80211_CMD_DEL_KEY, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_del_key, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_BEACON, - .policy = nl80211_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_UNS_ADMIN_PERM, .doit = nl80211_set_beacon, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | @@ -13457,7 +13683,7 @@ static const struct genl_ops nl80211_ops[] = { }, { .cmd = NL80211_CMD_START_AP, - .policy = nl80211_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_UNS_ADMIN_PERM, .doit = nl80211_start_ap, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | @@ -13465,7 +13691,7 @@ static const struct genl_ops nl80211_ops[] = { }, { .cmd = NL80211_CMD_STOP_AP, - .policy = nl80211_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .flags = GENL_UNS_ADMIN_PERM, .doit = nl80211_stop_ap, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | @@ -13473,172 +13699,172 @@ static const struct genl_ops nl80211_ops[] = { }, { .cmd = NL80211_CMD_GET_STATION, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_get_station, .dumpit = nl80211_dump_station, - .policy = nl80211_policy, .internal_flags = NL80211_FLAG_NEED_NETDEV | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_STATION, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_station, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_NEW_STATION, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_new_station, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_DEL_STATION, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_del_station, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_GET_MPATH, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_get_mpath, .dumpit = nl80211_dump_mpath, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_GET_MPP, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_get_mpp, .dumpit = nl80211_dump_mpp, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_MPATH, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_mpath, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_NEW_MPATH, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_new_mpath, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_DEL_MPATH, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_del_mpath, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_BSS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_bss, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_GET_REG, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_get_reg_do, .dumpit = nl80211_get_reg_dump, - .policy = nl80211_policy, .internal_flags = NL80211_FLAG_NEED_RTNL, /* can be retrieved by unprivileged users */ }, #ifdef CONFIG_CFG80211_CRDA_SUPPORT { .cmd = NL80211_CMD_SET_REG, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_reg, - .policy = nl80211_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_RTNL, }, #endif { .cmd = NL80211_CMD_REQ_SET_REG, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_req_set_reg, - .policy = nl80211_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = NL80211_CMD_RELOAD_REGDB, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_reload_regdb, - .policy = nl80211_policy, .flags = GENL_ADMIN_PERM, }, { .cmd = NL80211_CMD_GET_MESH_CONFIG, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_get_mesh_config, - .policy = nl80211_policy, /* can be retrieved by unprivileged users */ .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_MESH_CONFIG, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_update_mesh_config, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_TRIGGER_SCAN, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_trigger_scan, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_ABORT_SCAN, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_abort_scan, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_GET_SCAN, - .policy = nl80211_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .dumpit = nl80211_dump_scan, }, { .cmd = NL80211_CMD_START_SCHED_SCAN, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_start_sched_scan, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_STOP_SCHED_SCAN, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_stop_sched_scan, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_AUTHENTICATE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_authenticate, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL | @@ -13646,8 +13872,8 @@ static const struct genl_ops nl80211_ops[] = { }, { .cmd = NL80211_CMD_ASSOCIATE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_associate, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL | @@ -13655,32 +13881,32 @@ static const struct genl_ops nl80211_ops[] = { }, { .cmd = NL80211_CMD_DEAUTHENTICATE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_deauthenticate, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_DISASSOCIATE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_disassociate, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_JOIN_IBSS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_join_ibss, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_LEAVE_IBSS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_leave_ibss, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, @@ -13688,9 +13914,9 @@ static const struct genl_ops nl80211_ops[] = { #ifdef CONFIG_NL80211_TESTMODE { .cmd = NL80211_CMD_TESTMODE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_testmode_do, .dumpit = nl80211_testmode_dump, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WIPHY | NL80211_FLAG_NEED_RTNL, @@ -13698,8 +13924,8 @@ static const struct genl_ops nl80211_ops[] = { #endif { .cmd = NL80211_CMD_CONNECT, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_connect, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL | @@ -13707,8 +13933,8 @@ static const struct genl_ops nl80211_ops[] = { }, { .cmd = NL80211_CMD_UPDATE_CONNECT_PARAMS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_update_connect_params, - .policy = nl80211_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL | @@ -13716,29 +13942,29 @@ static const struct genl_ops nl80211_ops[] = { }, { .cmd = NL80211_CMD_DISCONNECT, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_disconnect, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_WIPHY_NETNS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_wiphy_netns, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WIPHY | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_GET_SURVEY, - .policy = nl80211_policy, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .dumpit = nl80211_dump_survey, }, { .cmd = NL80211_CMD_SET_PMKSA, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_setdel_pmksa, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL | @@ -13746,136 +13972,136 @@ static const struct genl_ops nl80211_ops[] = { }, { .cmd = NL80211_CMD_DEL_PMKSA, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_setdel_pmksa, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_FLUSH_PMKSA, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_flush_pmksa, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_REMAIN_ON_CHANNEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_remain_on_channel, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_CANCEL_REMAIN_ON_CHANNEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_cancel_remain_on_channel, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_TX_BITRATE_MASK, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_tx_bitrate_mask, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_REGISTER_FRAME, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_register_mgmt, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WDEV | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_FRAME, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_tx_mgmt, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_FRAME_WAIT_CANCEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_tx_mgmt_cancel_wait, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_POWER_SAVE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_power_save, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_GET_POWER_SAVE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_get_power_save, - .policy = nl80211_policy, /* can be retrieved by unprivileged users */ .internal_flags = NL80211_FLAG_NEED_NETDEV | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_CQM, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_cqm, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_CHANNEL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_channel, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_WDS_PEER, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_wds_peer, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_JOIN_MESH, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_join_mesh, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_LEAVE_MESH, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_leave_mesh, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_JOIN_OCB, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_join_ocb, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_LEAVE_OCB, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_leave_ocb, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, @@ -13883,16 +14109,16 @@ static const struct genl_ops nl80211_ops[] = { #ifdef CONFIG_PM { .cmd = NL80211_CMD_GET_WOWLAN, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_get_wowlan, - .policy = nl80211_policy, /* can be retrieved by unprivileged users */ .internal_flags = NL80211_FLAG_NEED_WIPHY | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_WOWLAN, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_wowlan, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WIPHY | NL80211_FLAG_NEED_RTNL, @@ -13900,8 +14126,8 @@ static const struct genl_ops nl80211_ops[] = { #endif { .cmd = NL80211_CMD_SET_REKEY_OFFLOAD, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_rekey_data, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL | @@ -13909,189 +14135,189 @@ static const struct genl_ops nl80211_ops[] = { }, { .cmd = NL80211_CMD_TDLS_MGMT, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_tdls_mgmt, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_TDLS_OPER, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_tdls_oper, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_UNEXPECTED_FRAME, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_register_unexpected_frame, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_PROBE_CLIENT, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_probe_client, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_REGISTER_BEACONS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_register_beacons, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WIPHY | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_NOACK_MAP, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_noack_map, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_START_P2P_DEVICE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_start_p2p_device, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WDEV | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_STOP_P2P_DEVICE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_stop_p2p_device, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_START_NAN, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_start_nan, - .policy = nl80211_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WDEV | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_STOP_NAN, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_stop_nan, - .policy = nl80211_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_ADD_NAN_FUNCTION, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_nan_add_func, - .policy = nl80211_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_DEL_NAN_FUNCTION, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_nan_del_func, - .policy = nl80211_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_CHANGE_NAN_CONFIG, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_nan_change_config, - .policy = nl80211_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_MCAST_RATE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_mcast_rate, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_MAC_ACL, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_mac_acl, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_RADAR_DETECT, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_start_radar_detection, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_GET_PROTOCOL_FEATURES, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_get_protocol_features, - .policy = nl80211_policy, }, { .cmd = NL80211_CMD_UPDATE_FT_IES, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_update_ft_ies, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_CRIT_PROTOCOL_START, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_crit_protocol_start, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_CRIT_PROTOCOL_STOP, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_crit_protocol_stop, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_GET_COALESCE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_get_coalesce, - .policy = nl80211_policy, .internal_flags = NL80211_FLAG_NEED_WIPHY | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_COALESCE, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_coalesce, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WIPHY | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_CHANNEL_SWITCH, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_channel_switch, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_VENDOR, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_vendor_cmd, .dumpit = nl80211_vendor_cmd_dump, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WIPHY | NL80211_FLAG_NEED_RTNL | @@ -14099,102 +14325,116 @@ static const struct genl_ops nl80211_ops[] = { }, { .cmd = NL80211_CMD_SET_QOS_MAP, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_qos_map, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_ADD_TX_TS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_add_tx_ts, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_DEL_TX_TS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_del_tx_ts, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_TDLS_CHANNEL_SWITCH, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_tdls_channel_switch, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_TDLS_CANCEL_CHANNEL_SWITCH, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_tdls_cancel_channel_switch, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_MULTICAST_TO_UNICAST, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_multicast_to_unicast, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_SET_PMK, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_set_pmk, - .policy = nl80211_policy, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL | NL80211_FLAG_CLEAR_SKB, }, { .cmd = NL80211_CMD_DEL_PMK, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_del_pmk, - .policy = nl80211_policy, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_EXTERNAL_AUTH, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_external_auth, - .policy = nl80211_policy, .flags = GENL_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_CONTROL_PORT_FRAME, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_tx_control_port, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_GET_FTM_RESPONDER_STATS, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_get_ftm_responder_stats, - .policy = nl80211_policy, .internal_flags = NL80211_FLAG_NEED_NETDEV | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_PEER_MEASUREMENT_START, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_pmsr_start, - .policy = nl80211_policy, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_WDEV_UP | NL80211_FLAG_NEED_RTNL, }, { .cmd = NL80211_CMD_NOTIFY_RADAR, + .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP, .doit = nl80211_notify_radar_detection, - .policy = nl80211_policy, + .flags = GENL_UNS_ADMIN_PERM, + .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | + NL80211_FLAG_NEED_RTNL, + }, + { + .cmd = NL80211_CMD_UPDATE_OWE_INFO, + .doit = nl80211_update_owe_info, + .flags = GENL_ADMIN_PERM, + .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | + NL80211_FLAG_NEED_RTNL, + }, + { + .cmd = NL80211_CMD_PROBE_MESH_LINK, + .doit = nl80211_probe_mesh_link, .flags = GENL_UNS_ADMIN_PERM, .internal_flags = NL80211_FLAG_NEED_NETDEV_UP | NL80211_FLAG_NEED_RTNL, @@ -14206,6 +14446,7 @@ static struct genl_family nl80211_fam __ro_after_init = { .hdrsize = 0, /* no private header */ .version = 1, /* no particular meaning now */ .maxattr = NL80211_ATTR_MAX, + .policy = nl80211_policy, .netnsok = true, .pre_doit = nl80211_pre_doit, .post_doit = nl80211_post_doit, @@ -14269,7 +14510,7 @@ static int nl80211_add_scan_req(struct sk_buff *msg, if (WARN_ON(!req)) return 0; - nest = nla_nest_start(msg, NL80211_ATTR_SCAN_SSIDS); + nest = nla_nest_start_noflag(msg, NL80211_ATTR_SCAN_SSIDS); if (!nest) goto nla_put_failure; for (i = 0; i < req->n_ssids; i++) { @@ -14278,7 +14519,7 @@ static int nl80211_add_scan_req(struct sk_buff *msg, } nla_nest_end(msg, nest); - nest = nla_nest_start(msg, NL80211_ATTR_SCAN_FREQUENCIES); + nest = nla_nest_start_noflag(msg, NL80211_ATTR_SCAN_FREQUENCIES); if (!nest) goto nla_put_failure; for (i = 0; i < req->n_channels; i++) { @@ -14540,7 +14781,7 @@ static void nl80211_send_mlme_event(struct cfg80211_registered_device *rdev, if (uapsd_queues >= 0) { struct nlattr *nla_wmm = - nla_nest_start(msg, NL80211_ATTR_STA_WME); + nla_nest_start_noflag(msg, NL80211_ATTR_STA_WME); if (!nla_wmm) goto nla_put_failure; @@ -14981,7 +15222,7 @@ void nl80211_send_beacon_hint_event(struct wiphy *wiphy, goto nla_put_failure; /* Before */ - nl_freq = nla_nest_start(msg, NL80211_ATTR_FREQ_BEFORE); + nl_freq = nla_nest_start_noflag(msg, NL80211_ATTR_FREQ_BEFORE); if (!nl_freq) goto nla_put_failure; @@ -14990,7 +15231,7 @@ void nl80211_send_beacon_hint_event(struct wiphy *wiphy, nla_nest_end(msg, nl_freq); /* After */ - nl_freq = nla_nest_start(msg, NL80211_ATTR_FREQ_AFTER); + nl_freq = nla_nest_start_noflag(msg, NL80211_ATTR_FREQ_AFTER); if (!nl_freq) goto nla_put_failure; @@ -15424,7 +15665,7 @@ static struct sk_buff *cfg80211_prepare_cqm(struct net_device *dev, if (mac && nla_put(msg, NL80211_ATTR_MAC, ETH_ALEN, mac)) goto nla_put_failure; - cb[1] = nla_nest_start(msg, NL80211_ATTR_CQM); + cb[1] = nla_nest_start_noflag(msg, NL80211_ATTR_CQM); if (!cb[1]) goto nla_put_failure; @@ -15585,7 +15826,7 @@ static void nl80211_gtk_rekey_notify(struct cfg80211_registered_device *rdev, nla_put(msg, NL80211_ATTR_MAC, ETH_ALEN, bssid)) goto nla_put_failure; - rekey_attr = nla_nest_start(msg, NL80211_ATTR_REKEY_DATA); + rekey_attr = nla_nest_start_noflag(msg, NL80211_ATTR_REKEY_DATA); if (!rekey_attr) goto nla_put_failure; @@ -15640,7 +15881,7 @@ nl80211_pmksa_candidate_notify(struct cfg80211_registered_device *rdev, nla_put_u32(msg, NL80211_ATTR_IFINDEX, netdev->ifindex)) goto nla_put_failure; - attr = nla_nest_start(msg, NL80211_ATTR_PMKSA_CANDIDATE); + attr = nla_nest_start_noflag(msg, NL80211_ATTR_PMKSA_CANDIDATE); if (!attr) goto nla_put_failure; @@ -15727,6 +15968,11 @@ void cfg80211_ch_switch_notify(struct net_device *dev, wdev->chandef = *chandef; wdev->preset_chandef = *chandef; + + if (wdev->iftype == NL80211_IFTYPE_STATION && + !WARN_ON(!wdev->current_bss)) + wdev->current_bss->pub.channel = chandef->chan; + nl80211_ch_switch_notify(rdev, dev, chandef, GFP_KERNEL, NL80211_CMD_CH_SWITCH_NOTIFY, 0); } @@ -15945,15 +16191,15 @@ static int cfg80211_net_detect_results(struct sk_buff *msg, struct nlattr *nl_results, *nl_match, *nl_freqs; int i, j; - nl_results = nla_nest_start( - msg, NL80211_WOWLAN_TRIG_NET_DETECT_RESULTS); + nl_results = nla_nest_start_noflag(msg, + NL80211_WOWLAN_TRIG_NET_DETECT_RESULTS); if (!nl_results) return -EMSGSIZE; for (i = 0; i < nd->n_matches; i++) { struct cfg80211_wowlan_nd_match *match = nd->matches[i]; - nl_match = nla_nest_start(msg, i); + nl_match = nla_nest_start_noflag(msg, i); if (!nl_match) break; @@ -15971,8 +16217,8 @@ static int cfg80211_net_detect_results(struct sk_buff *msg, } if (match->n_channels) { - nl_freqs = nla_nest_start( - msg, NL80211_ATTR_SCAN_FREQUENCIES); + nl_freqs = nla_nest_start_noflag(msg, + NL80211_ATTR_SCAN_FREQUENCIES); if (!nl_freqs) { nla_nest_cancel(msg, nl_match); goto out; @@ -16031,7 +16277,8 @@ void cfg80211_report_wowlan_wakeup(struct wireless_dev *wdev, if (wakeup) { struct nlattr *reasons; - reasons = nla_nest_start(msg, NL80211_ATTR_WOWLAN_TRIGGERS); + reasons = nla_nest_start_noflag(msg, + NL80211_ATTR_WOWLAN_TRIGGERS); if (!reasons) goto free_msg; @@ -16370,6 +16617,46 @@ int cfg80211_external_auth_request(struct net_device *dev, } EXPORT_SYMBOL(cfg80211_external_auth_request); +void cfg80211_update_owe_info_event(struct net_device *netdev, + struct cfg80211_update_owe_info *owe_info, + gfp_t gfp) +{ + struct wiphy *wiphy = netdev->ieee80211_ptr->wiphy; + struct cfg80211_registered_device *rdev = wiphy_to_rdev(wiphy); + struct sk_buff *msg; + void *hdr; + + trace_cfg80211_update_owe_info_event(wiphy, netdev, owe_info); + + msg = nlmsg_new(NLMSG_DEFAULT_SIZE, gfp); + if (!msg) + return; + + hdr = nl80211hdr_put(msg, 0, 0, 0, NL80211_CMD_UPDATE_OWE_INFO); + if (!hdr) + goto nla_put_failure; + + if (nla_put_u32(msg, NL80211_ATTR_WIPHY, rdev->wiphy_idx) || + nla_put_u32(msg, NL80211_ATTR_IFINDEX, netdev->ifindex) || + nla_put(msg, NL80211_ATTR_MAC, ETH_ALEN, owe_info->peer)) + goto nla_put_failure; + + if (!owe_info->ie_len || + nla_put(msg, NL80211_ATTR_IE, owe_info->ie_len, owe_info->ie)) + goto nla_put_failure; + + genlmsg_end(msg, hdr); + + genlmsg_multicast_netns(&nl80211_fam, wiphy_net(&rdev->wiphy), msg, 0, + NL80211_MCGRP_MLME, gfp); + return; + +nla_put_failure: + genlmsg_cancel(msg, hdr); + nlmsg_free(msg); +} +EXPORT_SYMBOL(cfg80211_update_owe_info_event); + /* initialisation/exit functions */ int __init nl80211_init(void) diff --git a/net/wireless/pmsr.c b/net/wireless/pmsr.c index 5e2ab01d325c8f2dee653868a6c212bdef6e8f08..1b190475359a794754b922acb485d07524cec777 100644 --- a/net/wireless/pmsr.c +++ b/net/wireless/pmsr.c @@ -25,7 +25,8 @@ static int pmsr_parse_ftm(struct cfg80211_registered_device *rdev, } /* no validation needed - was already done via nested policy */ - nla_parse_nested(tb, NL80211_PMSR_FTM_REQ_ATTR_MAX, ftmreq, NULL, NULL); + nla_parse_nested_deprecated(tb, NL80211_PMSR_FTM_REQ_ATTR_MAX, ftmreq, + NULL, NULL); if (tb[NL80211_PMSR_FTM_REQ_ATTR_PREAMBLE]) preamble = nla_get_u32(tb[NL80211_PMSR_FTM_REQ_ATTR_PREAMBLE]); @@ -139,7 +140,8 @@ static int pmsr_parse_peer(struct cfg80211_registered_device *rdev, int err, rem; /* no validation needed - was already done via nested policy */ - nla_parse_nested(tb, NL80211_PMSR_PEER_ATTR_MAX, peer, NULL, NULL); + nla_parse_nested_deprecated(tb, NL80211_PMSR_PEER_ATTR_MAX, peer, + NULL, NULL); if (!tb[NL80211_PMSR_PEER_ATTR_ADDR] || !tb[NL80211_PMSR_PEER_ATTR_CHAN] || @@ -154,9 +156,9 @@ static int pmsr_parse_peer(struct cfg80211_registered_device *rdev, /* reuse info->attrs */ memset(info->attrs, 0, sizeof(*info->attrs) * (NL80211_ATTR_MAX + 1)); /* need to validate here, we don't want to have validation recursion */ - err = nla_parse_nested(info->attrs, NL80211_ATTR_MAX, - tb[NL80211_PMSR_PEER_ATTR_CHAN], - nl80211_policy, info->extack); + err = nla_parse_nested_deprecated(info->attrs, NL80211_ATTR_MAX, + tb[NL80211_PMSR_PEER_ATTR_CHAN], + nl80211_policy, info->extack); if (err) return err; @@ -165,9 +167,9 @@ static int pmsr_parse_peer(struct cfg80211_registered_device *rdev, return err; /* no validation needed - was already done via nested policy */ - nla_parse_nested(req, NL80211_PMSR_REQ_ATTR_MAX, - tb[NL80211_PMSR_PEER_ATTR_REQ], - NULL, NULL); + nla_parse_nested_deprecated(req, NL80211_PMSR_REQ_ATTR_MAX, + tb[NL80211_PMSR_PEER_ATTR_REQ], NULL, + NULL); if (!req[NL80211_PMSR_REQ_ATTR_DATA]) { NL_SET_ERR_MSG_ATTR(info->extack, @@ -420,22 +422,22 @@ static int nl80211_pmsr_send_result(struct sk_buff *msg, { struct nlattr *pmsr, *peers, *peer, *resp, *data, *typedata; - pmsr = nla_nest_start(msg, NL80211_ATTR_PEER_MEASUREMENTS); + pmsr = nla_nest_start_noflag(msg, NL80211_ATTR_PEER_MEASUREMENTS); if (!pmsr) goto error; - peers = nla_nest_start(msg, NL80211_PMSR_ATTR_PEERS); + peers = nla_nest_start_noflag(msg, NL80211_PMSR_ATTR_PEERS); if (!peers) goto error; - peer = nla_nest_start(msg, 1); + peer = nla_nest_start_noflag(msg, 1); if (!peer) goto error; if (nla_put(msg, NL80211_PMSR_PEER_ATTR_ADDR, ETH_ALEN, res->addr)) goto error; - resp = nla_nest_start(msg, NL80211_PMSR_PEER_ATTR_RESP); + resp = nla_nest_start_noflag(msg, NL80211_PMSR_PEER_ATTR_RESP); if (!resp) goto error; @@ -452,11 +454,11 @@ static int nl80211_pmsr_send_result(struct sk_buff *msg, if (res->final && nla_put_flag(msg, NL80211_PMSR_RESP_ATTR_FINAL)) goto error; - data = nla_nest_start(msg, NL80211_PMSR_RESP_ATTR_DATA); + data = nla_nest_start_noflag(msg, NL80211_PMSR_RESP_ATTR_DATA); if (!data) goto error; - typedata = nla_nest_start(msg, res->type); + typedata = nla_nest_start_noflag(msg, res->type); if (!typedata) goto error; diff --git a/net/wireless/rdev-ops.h b/net/wireless/rdev-ops.h index 5cb48d135fab26a6c51ad88a0365abb1cccc5df8..e853a4fe6f97f2cd716cdd5253a4d47f364b9d8a 100644 --- a/net/wireless/rdev-ops.h +++ b/net/wireless/rdev-ops.h @@ -77,7 +77,8 @@ static inline int rdev_add_key(struct cfg80211_registered_device *rdev, struct key_params *params) { int ret; - trace_rdev_add_key(&rdev->wiphy, netdev, key_index, pairwise, mac_addr); + trace_rdev_add_key(&rdev->wiphy, netdev, key_index, pairwise, + mac_addr, params->mode); ret = rdev->ops->add_key(&rdev->wiphy, netdev, key_index, pairwise, mac_addr, params); trace_rdev_return_int(&rdev->wiphy, ret); @@ -1272,4 +1273,30 @@ rdev_abort_pmsr(struct cfg80211_registered_device *rdev, trace_rdev_return_void(&rdev->wiphy); } +static inline int rdev_update_owe_info(struct cfg80211_registered_device *rdev, + struct net_device *dev, + struct cfg80211_update_owe_info *oweinfo) +{ + int ret = -EOPNOTSUPP; + + trace_rdev_update_owe_info(&rdev->wiphy, dev, oweinfo); + if (rdev->ops->update_owe_info) + ret = rdev->ops->update_owe_info(&rdev->wiphy, dev, oweinfo); + trace_rdev_return_int(&rdev->wiphy, ret); + return ret; +} + +static inline int +rdev_probe_mesh_link(struct cfg80211_registered_device *rdev, + struct net_device *dev, const u8 *dest, + const void *buf, size_t len) +{ + int ret; + + trace_rdev_probe_mesh_link(&rdev->wiphy, dev, dest, buf, len); + ret = rdev->ops->probe_mesh_link(&rdev->wiphy, dev, buf, len); + trace_rdev_return_int(&rdev->wiphy, ret); + return ret; +} + #endif /* __CFG80211_RDEV_OPS */ diff --git a/net/wireless/reg.c b/net/wireless/reg.c index a6fd5ce199da197aeaffc6b71b75703fe811974b..4831ad745f912dd0e82e0a87b49414a175a28c68 100644 --- a/net/wireless/reg.c +++ b/net/wireless/reg.c @@ -427,14 +427,10 @@ static const struct ieee80211_regdomain * reg_copy_regd(const struct ieee80211_regdomain *src_regd) { struct ieee80211_regdomain *regd; - int size_of_regd; unsigned int i; - size_of_regd = - sizeof(struct ieee80211_regdomain) + - src_regd->n_reg_rules * sizeof(struct ieee80211_reg_rule); - - regd = kzalloc(size_of_regd, GFP_KERNEL); + regd = kzalloc(struct_size(regd, reg_rules, src_regd->n_reg_rules), + GFP_KERNEL); if (!regd) return ERR_PTR(-ENOMEM); @@ -948,12 +944,10 @@ static int regdb_query_country(const struct fwdb_header *db, unsigned int ptr = be16_to_cpu(country->coll_ptr) << 2; struct fwdb_collection *coll = (void *)((u8 *)db + ptr); struct ieee80211_regdomain *regdom; - unsigned int size_of_regd, i; - - size_of_regd = sizeof(struct ieee80211_regdomain) + - coll->n_rules * sizeof(struct ieee80211_reg_rule); + unsigned int i; - regdom = kzalloc(size_of_regd, GFP_KERNEL); + regdom = kzalloc(struct_size(regdom, reg_rules, coll->n_rules), + GFP_KERNEL); if (!regdom) return -ENOMEM; @@ -1489,7 +1483,7 @@ static struct ieee80211_regdomain * regdom_intersect(const struct ieee80211_regdomain *rd1, const struct ieee80211_regdomain *rd2) { - int r, size_of_regd; + int r; unsigned int x, y; unsigned int num_rules = 0; const struct ieee80211_reg_rule *rule1, *rule2; @@ -1520,10 +1514,7 @@ regdom_intersect(const struct ieee80211_regdomain *rd1, if (!num_rules) return NULL; - size_of_regd = sizeof(struct ieee80211_regdomain) + - num_rules * sizeof(struct ieee80211_reg_rule); - - rd = kzalloc(size_of_regd, GFP_KERNEL); + rd = kzalloc(struct_size(rd, reg_rules, num_rules), GFP_KERNEL); if (!rd) return NULL; diff --git a/net/wireless/scan.c b/net/wireless/scan.c index 04d888628f29dcca952d38d48b785d5e2c56dfef..c04f5451f89b114cb98c43b1b3ce6724858f2a4f 100644 --- a/net/wireless/scan.c +++ b/net/wireless/scan.c @@ -179,12 +179,63 @@ static bool __cfg80211_unlink_bss(struct cfg80211_registered_device *rdev, return true; } +bool cfg80211_is_element_inherited(const struct element *elem, + const struct element *non_inherit_elem) +{ + u8 id_len, ext_id_len, i, loop_len, id; + const u8 *list; + + if (elem->id == WLAN_EID_MULTIPLE_BSSID) + return false; + + if (!non_inherit_elem || non_inherit_elem->datalen < 2) + return true; + + /* + * non inheritance element format is: + * ext ID (56) | IDs list len | list | extension IDs list len | list + * Both lists are optional. Both lengths are mandatory. + * This means valid length is: + * elem_len = 1 (extension ID) + 2 (list len fields) + list lengths + */ + id_len = non_inherit_elem->data[1]; + if (non_inherit_elem->datalen < 3 + id_len) + return true; + + ext_id_len = non_inherit_elem->data[2 + id_len]; + if (non_inherit_elem->datalen < 3 + id_len + ext_id_len) + return true; + + if (elem->id == WLAN_EID_EXTENSION) { + if (!ext_id_len) + return true; + loop_len = ext_id_len; + list = &non_inherit_elem->data[3 + id_len]; + id = elem->data[0]; + } else { + if (!id_len) + return true; + loop_len = id_len; + list = &non_inherit_elem->data[2]; + id = elem->id; + } + + for (i = 0; i < loop_len; i++) { + if (list[i] == id) + return false; + } + + return true; +} +EXPORT_SYMBOL(cfg80211_is_element_inherited); + static size_t cfg80211_gen_new_ie(const u8 *ie, size_t ielen, const u8 *subelement, size_t subie_len, u8 *new_ie, gfp_t gfp) { u8 *pos, *tmp; const u8 *tmp_old, *tmp_new; + const struct element *non_inherit_elem; u8 *sub_copy; /* copy subelement as we need to change its content to @@ -203,6 +254,11 @@ static size_t cfg80211_gen_new_ie(const u8 *ie, size_t ielen, pos += (tmp_new[1] + 2); } + /* get non inheritance list if exists */ + non_inherit_elem = + cfg80211_find_ext_elem(WLAN_EID_EXT_NON_INHERITANCE, + sub_copy, subie_len); + /* go through IEs in ie (skip SSID) and subelement, * merge them into new_ie */ @@ -223,8 +279,11 @@ static size_t cfg80211_gen_new_ie(const u8 *ie, size_t ielen, subie_len); if (!tmp) { + const struct element *old_elem = (void *)tmp_old; + /* ie in old ie but not in subelement */ - if (tmp_old[0] != WLAN_EID_MULTIPLE_BSSID) { + if (cfg80211_is_element_inherited(old_elem, + non_inherit_elem)) { memcpy(pos, tmp_old, tmp_old[1] + 2); pos += tmp_old[1] + 2; } @@ -268,8 +327,7 @@ static size_t cfg80211_gen_new_ie(const u8 *ie, size_t ielen, tmp_new = sub_copy; while (tmp_new + tmp_new[1] + 2 - sub_copy <= subie_len) { if (!(tmp_new[0] == WLAN_EID_NON_TX_BSSID_CAP || - tmp_new[0] == WLAN_EID_SSID || - tmp_new[0] == WLAN_EID_MULTI_BSSID_IDX)) { + tmp_new[0] == WLAN_EID_SSID)) { memcpy(pos, tmp_new, tmp_new[1] + 2); pos += tmp_new[1] + 2; } @@ -1397,6 +1455,78 @@ cfg80211_inform_single_bss_data(struct wiphy *wiphy, return &res->pub; } +static const struct element +*cfg80211_get_profile_continuation(const u8 *ie, size_t ielen, + const struct element *mbssid_elem, + const struct element *sub_elem) +{ + const u8 *mbssid_end = mbssid_elem->data + mbssid_elem->datalen; + const struct element *next_mbssid; + const struct element *next_sub; + + next_mbssid = cfg80211_find_elem(WLAN_EID_MULTIPLE_BSSID, + mbssid_end, + ielen - (mbssid_end - ie)); + + /* + * If is is not the last subelement in current MBSSID IE or there isn't + * a next MBSSID IE - profile is complete. + */ + if ((sub_elem->data + sub_elem->datalen < mbssid_end - 1) || + !next_mbssid) + return NULL; + + /* For any length error, just return NULL */ + + if (next_mbssid->datalen < 4) + return NULL; + + next_sub = (void *)&next_mbssid->data[1]; + + if (next_mbssid->data + next_mbssid->datalen < + next_sub->data + next_sub->datalen) + return NULL; + + if (next_sub->id != 0 || next_sub->datalen < 2) + return NULL; + + /* + * Check if the first element in the next sub element is a start + * of a new profile + */ + return next_sub->data[0] == WLAN_EID_NON_TX_BSSID_CAP ? + NULL : next_mbssid; +} + +size_t cfg80211_merge_profile(const u8 *ie, size_t ielen, + const struct element *mbssid_elem, + const struct element *sub_elem, + u8 *merged_ie, size_t max_copy_len) +{ + size_t copied_len = sub_elem->datalen; + const struct element *next_mbssid; + + if (sub_elem->datalen > max_copy_len) + return 0; + + memcpy(merged_ie, sub_elem->data, sub_elem->datalen); + + while ((next_mbssid = cfg80211_get_profile_continuation(ie, ielen, + mbssid_elem, + sub_elem))) { + const struct element *next_sub = (void *)&next_mbssid->data[1]; + + if (copied_len + next_sub->datalen > max_copy_len) + break; + memcpy(merged_ie + copied_len, next_sub->data, + next_sub->datalen); + copied_len += next_sub->datalen; + } + + return copied_len; +} +EXPORT_SYMBOL(cfg80211_merge_profile); + static void cfg80211_parse_mbssid_data(struct wiphy *wiphy, struct cfg80211_inform_bss *data, enum cfg80211_bss_frame_type ftype, @@ -1410,7 +1540,8 @@ static void cfg80211_parse_mbssid_data(struct wiphy *wiphy, const struct element *elem, *sub; size_t new_ie_len; u8 new_bssid[ETH_ALEN]; - u8 *new_ie; + u8 *new_ie, *profile; + u64 seen_indices = 0; u16 capability; struct cfg80211_bss *bss; @@ -1428,10 +1559,16 @@ static void cfg80211_parse_mbssid_data(struct wiphy *wiphy, if (!new_ie) return; + profile = kmalloc(ielen, gfp); + if (!profile) + goto out; + for_each_element_id(elem, WLAN_EID_MULTIPLE_BSSID, ie, ielen) { if (elem->datalen < 4) continue; for_each_element(sub, elem->data + 1, elem->datalen - 1) { + u8 profile_len; + if (sub->id != 0 || sub->datalen < 4) { /* not a valid BSS profile */ continue; @@ -1446,16 +1583,31 @@ static void cfg80211_parse_mbssid_data(struct wiphy *wiphy, continue; } + memset(profile, 0, ielen); + profile_len = cfg80211_merge_profile(ie, ielen, + elem, + sub, + profile, + ielen); + /* found a Nontransmitted BSSID Profile */ mbssid_index_ie = cfg80211_find_ie (WLAN_EID_MULTI_BSSID_IDX, - sub->data, sub->datalen); + profile, profile_len); if (!mbssid_index_ie || mbssid_index_ie[1] < 1 || - mbssid_index_ie[2] == 0) { + mbssid_index_ie[2] == 0 || + mbssid_index_ie[2] > 46) { /* No valid Multiple BSSID-Index element */ continue; } + if (seen_indices & BIT(mbssid_index_ie[2])) + /* We don't support legacy split of a profile */ + net_dbg_ratelimited("Partial info for BSSID index %d\n", + mbssid_index_ie[2]); + + seen_indices |= BIT(mbssid_index_ie[2]); + non_tx_data->bssid_index = mbssid_index_ie[2]; non_tx_data->max_bssid_indicator = elem->data[0]; @@ -1464,13 +1616,14 @@ static void cfg80211_parse_mbssid_data(struct wiphy *wiphy, non_tx_data->bssid_index, new_bssid); memset(new_ie, 0, IEEE80211_MAX_DATA_LEN); - new_ie_len = cfg80211_gen_new_ie(ie, ielen, sub->data, - sub->datalen, new_ie, + new_ie_len = cfg80211_gen_new_ie(ie, ielen, + profile, + profile_len, new_ie, gfp); if (!new_ie_len) continue; - capability = get_unaligned_le16(sub->data + 2); + capability = get_unaligned_le16(profile + 2); bss = cfg80211_inform_single_bss_data(wiphy, data, ftype, new_bssid, tsf, @@ -1486,7 +1639,9 @@ static void cfg80211_parse_mbssid_data(struct wiphy *wiphy, } } +out: kfree(new_ie); + kfree(profile); } struct cfg80211_bss * diff --git a/net/wireless/trace.h b/net/wireless/trace.h index 44b2ce1bb13af846d467565e6b441e7c34d8fbf1..2abfff925aac82d6705a2ec370026bf371529cef 100644 --- a/net/wireless/trace.h +++ b/net/wireless/trace.h @@ -430,22 +430,43 @@ DECLARE_EVENT_CLASS(key_handle, BOOL_TO_STR(__entry->pairwise), MAC_PR_ARG(mac_addr)) ); -DEFINE_EVENT(key_handle, rdev_add_key, +DEFINE_EVENT(key_handle, rdev_get_key, TP_PROTO(struct wiphy *wiphy, struct net_device *netdev, u8 key_index, bool pairwise, const u8 *mac_addr), TP_ARGS(wiphy, netdev, key_index, pairwise, mac_addr) ); -DEFINE_EVENT(key_handle, rdev_get_key, +DEFINE_EVENT(key_handle, rdev_del_key, TP_PROTO(struct wiphy *wiphy, struct net_device *netdev, u8 key_index, bool pairwise, const u8 *mac_addr), TP_ARGS(wiphy, netdev, key_index, pairwise, mac_addr) ); -DEFINE_EVENT(key_handle, rdev_del_key, +TRACE_EVENT(rdev_add_key, TP_PROTO(struct wiphy *wiphy, struct net_device *netdev, u8 key_index, - bool pairwise, const u8 *mac_addr), - TP_ARGS(wiphy, netdev, key_index, pairwise, mac_addr) + bool pairwise, const u8 *mac_addr, u8 mode), + TP_ARGS(wiphy, netdev, key_index, pairwise, mac_addr, mode), + TP_STRUCT__entry( + WIPHY_ENTRY + NETDEV_ENTRY + MAC_ENTRY(mac_addr) + __field(u8, key_index) + __field(bool, pairwise) + __field(u8, mode) + ), + TP_fast_assign( + WIPHY_ASSIGN; + NETDEV_ASSIGN; + MAC_ASSIGN(mac_addr, mac_addr); + __entry->key_index = key_index; + __entry->pairwise = pairwise; + __entry->mode = mode; + ), + TP_printk(WIPHY_PR_FMT ", " NETDEV_PR_FMT ", key_index: %u, " + "mode: %u, pairwise: %s, mac addr: " MAC_PR_FMT, + WIPHY_PR_ARG, NETDEV_PR_ARG, __entry->key_index, + __entry->mode, BOOL_TO_STR(__entry->pairwise), + MAC_PR_ARG(mac_addr)) ); TRACE_EVENT(rdev_set_default_key, @@ -3362,6 +3383,62 @@ TRACE_EVENT(cfg80211_pmsr_complete, WIPHY_PR_ARG, WDEV_PR_ARG, (unsigned long long)__entry->cookie) ); + +TRACE_EVENT(rdev_update_owe_info, + TP_PROTO(struct wiphy *wiphy, struct net_device *netdev, + struct cfg80211_update_owe_info *owe_info), + TP_ARGS(wiphy, netdev, owe_info), + TP_STRUCT__entry(WIPHY_ENTRY + NETDEV_ENTRY + MAC_ENTRY(peer) + __field(u16, status) + __dynamic_array(u8, ie, owe_info->ie_len)), + TP_fast_assign(WIPHY_ASSIGN; + NETDEV_ASSIGN; + MAC_ASSIGN(peer, owe_info->peer); + __entry->status = owe_info->status; + memcpy(__get_dynamic_array(ie), + owe_info->ie, owe_info->ie_len);), + TP_printk(WIPHY_PR_FMT ", " NETDEV_PR_FMT ", peer: " MAC_PR_FMT + " status %d", WIPHY_PR_ARG, NETDEV_PR_ARG, MAC_PR_ARG(peer), + __entry->status) +); + +TRACE_EVENT(cfg80211_update_owe_info_event, + TP_PROTO(struct wiphy *wiphy, struct net_device *netdev, + struct cfg80211_update_owe_info *owe_info), + TP_ARGS(wiphy, netdev, owe_info), + TP_STRUCT__entry(WIPHY_ENTRY + NETDEV_ENTRY + MAC_ENTRY(peer) + __dynamic_array(u8, ie, owe_info->ie_len)), + TP_fast_assign(WIPHY_ASSIGN; + NETDEV_ASSIGN; + MAC_ASSIGN(peer, owe_info->peer); + memcpy(__get_dynamic_array(ie), owe_info->ie, + owe_info->ie_len);), + TP_printk(WIPHY_PR_FMT ", " NETDEV_PR_FMT ", peer: " MAC_PR_FMT, + WIPHY_PR_ARG, NETDEV_PR_ARG, MAC_PR_ARG(peer)) +); + +TRACE_EVENT(rdev_probe_mesh_link, + TP_PROTO(struct wiphy *wiphy, struct net_device *netdev, + const u8 *dest, const u8 *buf, size_t len), + TP_ARGS(wiphy, netdev, dest, buf, len), + TP_STRUCT__entry( + WIPHY_ENTRY + NETDEV_ENTRY + MAC_ENTRY(dest) + ), + TP_fast_assign( + WIPHY_ASSIGN; + NETDEV_ASSIGN; + MAC_ASSIGN(dest, dest); + ), + TP_printk(WIPHY_PR_FMT ", " NETDEV_PR_FMT ", " MAC_PR_FMT, + WIPHY_PR_ARG, NETDEV_PR_ARG, MAC_PR_ARG(dest)) +); + #endif /* !__RDEV_OPS_TRACE || TRACE_HEADER_MULTI_READ */ #undef TRACE_INCLUDE_PATH diff --git a/net/wireless/util.c b/net/wireless/util.c index 75899b62bdc9ed2116a1420a6035c904307f4838..cf63b635afc098eeafe7462bd41b86a7e433b85c 100644 --- a/net/wireless/util.c +++ b/net/wireless/util.c @@ -237,14 +237,23 @@ int cfg80211_validate_key_settings(struct cfg80211_registered_device *rdev, case WLAN_CIPHER_SUITE_CCMP_256: case WLAN_CIPHER_SUITE_GCMP: case WLAN_CIPHER_SUITE_GCMP_256: - /* Disallow pairwise keys with non-zero index unless it's WEP - * or a vendor specific cipher (because current deployments use - * pairwise WEP keys with non-zero indices and for vendor - * specific ciphers this should be validated in the driver or - * hardware level - but 802.11i clearly specifies to use zero) + /* IEEE802.11-2016 allows only 0 and - when using Extended Key + * ID - 1 as index for pairwise keys. + * @NL80211_KEY_NO_TX is only allowed for pairwise keys when + * the driver supports Extended Key ID. + * @NL80211_KEY_SET_TX can't be set when installing and + * validating a key. */ - if (pairwise && key_idx) + if (params->mode == NL80211_KEY_NO_TX) { + if (!wiphy_ext_feature_isset(&rdev->wiphy, + NL80211_EXT_FEATURE_EXT_KEY_ID)) + return -EINVAL; + else if (!pairwise || key_idx < 0 || key_idx > 1) + return -EINVAL; + } else if ((pairwise && key_idx) || + params->mode == NL80211_KEY_SET_TX) { return -EINVAL; + } break; case WLAN_CIPHER_SUITE_AES_CMAC: case WLAN_CIPHER_SUITE_BIP_CMAC_256: diff --git a/net/wireless/wext-compat.c b/net/wireless/wext-compat.c index d522787c735409adca94d315da27133f50a9dd37..46e4d69db8453c35ec1d3895dfc52bb1baa08c2c 100644 --- a/net/wireless/wext-compat.c +++ b/net/wireless/wext-compat.c @@ -353,9 +353,6 @@ static int cfg80211_wext_siwretry(struct net_device *dev, changed |= WIPHY_PARAM_RETRY_SHORT; } - if (!changed) - return 0; - err = rdev_set_wiphy_params(rdev, changed); if (err) { wdev->wiphy->retry_short = oshort; diff --git a/net/x25/af_x25.c b/net/x25/af_x25.c index 20a511398389d686383e9d8c74434b3f2ad40f3f..0ea48a52ce798f059b5c89529193f91124438e17 100644 --- a/net/x25/af_x25.c +++ b/net/x25/af_x25.c @@ -1398,18 +1398,6 @@ static int x25_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) break; } - case SIOCGSTAMP: - rc = -EINVAL; - if (sk) - rc = sock_get_timestamp(sk, - (struct timeval __user *)argp); - break; - case SIOCGSTAMPNS: - rc = -EINVAL; - if (sk) - rc = sock_get_timestampns(sk, - (struct timespec __user *)argp); - break; case SIOCGIFADDR: case SIOCSIFADDR: case SIOCGIFDSTADDR: @@ -1681,8 +1669,6 @@ static int compat_x25_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) { void __user *argp = compat_ptr(arg); - struct sock *sk = sock->sk; - int rc = -ENOIOCTLCMD; switch(cmd) { @@ -1690,18 +1676,6 @@ static int compat_x25_ioctl(struct socket *sock, unsigned int cmd, case TIOCINQ: rc = x25_ioctl(sock, cmd, (unsigned long)argp); break; - case SIOCGSTAMP: - rc = -EINVAL; - if (sk) - rc = compat_sock_get_timestamp(sk, - (struct timeval __user*)argp); - break; - case SIOCGSTAMPNS: - rc = -EINVAL; - if (sk) - rc = compat_sock_get_timestampns(sk, - (struct timespec __user*)argp); - break; case SIOCGIFADDR: case SIOCSIFADDR: case SIOCGIFDSTADDR: @@ -1765,6 +1739,7 @@ static const struct proto_ops x25_proto_ops = { #ifdef CONFIG_COMPAT .compat_ioctl = compat_x25_ioctl, #endif + .gettstamp = sock_gettstamp, .listen = x25_listen, .shutdown = sock_no_shutdown, .setsockopt = x25_setsockopt, diff --git a/net/xdp/xsk_queue.h b/net/xdp/xsk_queue.h index 610c0bdc0c2b883d08e706bbd8b0ce3109b09d1e..88b9ae24658d1efd39286d9a42357efe3b25ee42 100644 --- a/net/xdp/xsk_queue.h +++ b/net/xdp/xsk_queue.h @@ -43,6 +43,48 @@ struct xsk_queue { u64 invalid_descs; }; +/* The structure of the shared state of the rings are the same as the + * ring buffer in kernel/events/ring_buffer.c. For the Rx and completion + * ring, the kernel is the producer and user space is the consumer. For + * the Tx and fill rings, the kernel is the consumer and user space is + * the producer. + * + * producer consumer + * + * if (LOAD ->consumer) { LOAD ->producer + * (A) smp_rmb() (C) + * STORE $data LOAD $data + * smp_wmb() (B) smp_mb() (D) + * STORE ->producer STORE ->consumer + * } + * + * (A) pairs with (D), and (B) pairs with (C). + * + * Starting with (B), it protects the data from being written after + * the producer pointer. If this barrier was missing, the consumer + * could observe the producer pointer being set and thus load the data + * before the producer has written the new data. The consumer would in + * this case load the old data. + * + * (C) protects the consumer from speculatively loading the data before + * the producer pointer actually has been read. If we do not have this + * barrier, some architectures could load old data as speculative loads + * are not discarded as the CPU does not know there is a dependency + * between ->producer and data. + * + * (A) is a control dependency that separates the load of ->consumer + * from the stores of $data. In case ->consumer indicates there is no + * room in the buffer to store $data we do not. So no barrier is needed. + * + * (D) protects the load of the data to be observed to happen after the + * store of the consumer pointer. If we did not have this memory + * barrier, the producer could observe the consumer pointer being set + * and overwrite the data with a new value before the consumer got the + * chance to read the old value. The consumer would thus miss reading + * the old entry and very likely read the new entry twice, once right + * now and again after circling through the ring. + */ + /* Common functions operating for both RXTX and umem queues */ static inline u64 xskq_nb_invalid_descs(struct xsk_queue *q) @@ -106,6 +148,7 @@ static inline u64 *xskq_validate_addr(struct xsk_queue *q, u64 *addr) static inline u64 *xskq_peek_addr(struct xsk_queue *q, u64 *addr) { if (q->cons_tail == q->cons_head) { + smp_mb(); /* D, matches A */ WRITE_ONCE(q->ring->consumer, q->cons_tail); q->cons_head = q->cons_tail + xskq_nb_avail(q, RX_BATCH_SIZE); @@ -128,10 +171,11 @@ static inline int xskq_produce_addr(struct xsk_queue *q, u64 addr) if (xskq_nb_free(q, q->prod_tail, 1) == 0) return -ENOSPC; + /* A, matches D */ ring->desc[q->prod_tail++ & q->ring_mask] = addr; /* Order producer and data */ - smp_wmb(); + smp_wmb(); /* B, matches C */ WRITE_ONCE(q->ring->producer, q->prod_tail); return 0; @@ -144,6 +188,7 @@ static inline int xskq_produce_addr_lazy(struct xsk_queue *q, u64 addr) if (xskq_nb_free(q, q->prod_head, LAZY_UPDATE_THRESHOLD) == 0) return -ENOSPC; + /* A, matches D */ ring->desc[q->prod_head++ & q->ring_mask] = addr; return 0; } @@ -152,7 +197,7 @@ static inline void xskq_produce_flush_addr_n(struct xsk_queue *q, u32 nb_entries) { /* Order producer and data */ - smp_wmb(); + smp_wmb(); /* B, matches C */ q->prod_tail += nb_entries; WRITE_ONCE(q->ring->producer, q->prod_tail); @@ -163,6 +208,7 @@ static inline int xskq_reserve_addr(struct xsk_queue *q) if (xskq_nb_free(q, q->prod_head, 1) == 0) return -ENOSPC; + /* A, matches D */ q->prod_head++; return 0; } @@ -204,11 +250,12 @@ static inline struct xdp_desc *xskq_peek_desc(struct xsk_queue *q, struct xdp_desc *desc) { if (q->cons_tail == q->cons_head) { + smp_mb(); /* D, matches A */ WRITE_ONCE(q->ring->consumer, q->cons_tail); q->cons_head = q->cons_tail + xskq_nb_avail(q, RX_BATCH_SIZE); /* Order consumer and data */ - smp_rmb(); + smp_rmb(); /* C, matches B */ } return xskq_validate_desc(q, desc); @@ -228,6 +275,7 @@ static inline int xskq_produce_batch_desc(struct xsk_queue *q, if (xskq_nb_free(q, q->prod_head, 1) == 0) return -ENOSPC; + /* A, matches D */ idx = (q->prod_head++) & q->ring_mask; ring->desc[idx].addr = addr; ring->desc[idx].len = len; @@ -238,7 +286,7 @@ static inline int xskq_produce_batch_desc(struct xsk_queue *q, static inline void xskq_produce_flush_desc(struct xsk_queue *q) { /* Order producer and data */ - smp_wmb(); + smp_wmb(); /* B, matches C */ q->prod_tail = q->prod_head, WRITE_ONCE(q->ring->producer, q->prod_tail); diff --git a/net/xfrm/Kconfig b/net/xfrm/Kconfig index 5d43aaa1702738670b593edfe182437dfd5b1d3b..1ec8071226b2d1b800f711177500e6f7871dcd5d 100644 --- a/net/xfrm/Kconfig +++ b/net/xfrm/Kconfig @@ -3,7 +3,7 @@ # config XFRM bool - depends on NET + depends on INET select GRO_CELLS select SKB_EXTENSIONS @@ -15,9 +15,9 @@ config XFRM_ALGO select XFRM select CRYPTO +if INET config XFRM_USER tristate "Transformation user configuration interface" - depends on INET select XFRM_ALGO ---help--- Support for Transformation(XFRM) user configuration interface @@ -56,7 +56,7 @@ config XFRM_MIGRATE config XFRM_STATISTICS bool "Transformation statistics" - depends on INET && XFRM && PROC_FS + depends on XFRM && PROC_FS ---help--- This statistics is not a SNMP/MIB specification but shows statistics about transformation error (or almost error) factor @@ -95,3 +95,5 @@ config NET_KEY_MIGRATE . If unsure, say N. + +endif # INET diff --git a/net/xfrm/xfrm_device.c b/net/xfrm/xfrm_device.c index b8736f56e7f7b6c14e35e3bb9ada09ded462e61c..b24cd86a02c3693ec6eb27f4fb03b5830bd420ce 100644 --- a/net/xfrm/xfrm_device.c +++ b/net/xfrm/xfrm_device.c @@ -23,6 +23,60 @@ #include #ifdef CONFIG_XFRM_OFFLOAD +static void __xfrm_transport_prep(struct xfrm_state *x, struct sk_buff *skb, + unsigned int hsize) +{ + struct xfrm_offload *xo = xfrm_offload(skb); + + skb_reset_mac_len(skb); + pskb_pull(skb, skb->mac_len + hsize + x->props.header_len); + + if (xo->flags & XFRM_GSO_SEGMENT) { + skb_reset_transport_header(skb); + skb->transport_header -= x->props.header_len; + } +} + +static void __xfrm_mode_tunnel_prep(struct xfrm_state *x, struct sk_buff *skb, + unsigned int hsize) + +{ + struct xfrm_offload *xo = xfrm_offload(skb); + + if (xo->flags & XFRM_GSO_SEGMENT) + skb->transport_header = skb->network_header + hsize; + + skb_reset_mac_len(skb); + pskb_pull(skb, skb->mac_len + x->props.header_len); +} + +/* Adjust pointers into the packet when IPsec is done at layer2 */ +static void xfrm_outer_mode_prep(struct xfrm_state *x, struct sk_buff *skb) +{ + switch (x->outer_mode.encap) { + case XFRM_MODE_TUNNEL: + if (x->outer_mode.family == AF_INET) + return __xfrm_mode_tunnel_prep(x, skb, + sizeof(struct iphdr)); + if (x->outer_mode.family == AF_INET6) + return __xfrm_mode_tunnel_prep(x, skb, + sizeof(struct ipv6hdr)); + break; + case XFRM_MODE_TRANSPORT: + if (x->outer_mode.family == AF_INET) + return __xfrm_transport_prep(x, skb, + sizeof(struct iphdr)); + if (x->outer_mode.family == AF_INET6) + return __xfrm_transport_prep(x, skb, + sizeof(struct ipv6hdr)); + break; + case XFRM_MODE_ROUTEOPTIMIZATION: + case XFRM_MODE_IN_TRIGGER: + case XFRM_MODE_BEET: + break; + } +} + struct sk_buff *validate_xmit_xfrm(struct sk_buff *skb, netdev_features_t features, bool *again) { int err; @@ -78,7 +132,8 @@ struct sk_buff *validate_xmit_xfrm(struct sk_buff *skb, netdev_features_t featur } if (!skb->next) { - x->outer_mode->xmit(x, skb); + esp_features |= skb->dev->gso_partial_features; + xfrm_outer_mode_prep(x, skb); xo->flags |= XFRM_DEV_RESUME; @@ -101,12 +156,14 @@ struct sk_buff *validate_xmit_xfrm(struct sk_buff *skb, netdev_features_t featur do { struct sk_buff *nskb = skb2->next; + + esp_features |= skb->dev->gso_partial_features; skb_mark_not_on_list(skb2); xo = xfrm_offload(skb2); xo->flags |= XFRM_DEV_RESUME; - x->outer_mode->xmit(x, skb2); + xfrm_outer_mode_prep(x, skb2); err = x->type_offload->xmit(x, skb2, esp_features); if (!err) { @@ -247,7 +304,7 @@ void xfrm_dev_resume(struct sk_buff *skb) unsigned long flags; rcu_read_lock(); - txq = netdev_pick_tx(dev, skb, NULL); + txq = netdev_core_pick_tx(dev, skb, NULL); HARD_TX_LOCK(dev, txq, smp_processor_id()); if (!netif_xmit_frozen_or_stopped(txq)) diff --git a/net/xfrm/xfrm_inout.h b/net/xfrm/xfrm_inout.h new file mode 100644 index 0000000000000000000000000000000000000000..c7b0318938e227b8d69f84959bcbcaeb7512ddcb --- /dev/null +++ b/net/xfrm/xfrm_inout.h @@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#include +#include +#include + +#ifndef XFRM_INOUT_H +#define XFRM_INOUT_H 1 + +static inline void xfrm6_beet_make_header(struct sk_buff *skb) +{ + struct ipv6hdr *iph = ipv6_hdr(skb); + + iph->version = 6; + + memcpy(iph->flow_lbl, XFRM_MODE_SKB_CB(skb)->flow_lbl, + sizeof(iph->flow_lbl)); + iph->nexthdr = XFRM_MODE_SKB_CB(skb)->protocol; + + ipv6_change_dsfield(iph, 0, XFRM_MODE_SKB_CB(skb)->tos); + iph->hop_limit = XFRM_MODE_SKB_CB(skb)->ttl; +} + +static inline void xfrm4_beet_make_header(struct sk_buff *skb) +{ + struct iphdr *iph = ip_hdr(skb); + + iph->ihl = 5; + iph->version = 4; + + iph->protocol = XFRM_MODE_SKB_CB(skb)->protocol; + iph->tos = XFRM_MODE_SKB_CB(skb)->tos; + + iph->id = XFRM_MODE_SKB_CB(skb)->id; + iph->frag_off = XFRM_MODE_SKB_CB(skb)->frag_off; + iph->ttl = XFRM_MODE_SKB_CB(skb)->ttl; +} + +#endif diff --git a/net/xfrm/xfrm_input.c b/net/xfrm/xfrm_input.c index b3b613660d441605a8c8383cf05119db82af1ea0..314973aaa4143ef01832bbe39a15f672375df01a 100644 --- a/net/xfrm/xfrm_input.c +++ b/net/xfrm/xfrm_input.c @@ -21,6 +21,8 @@ #include #include +#include "xfrm_inout.h" + struct xfrm_trans_tasklet { struct tasklet_struct tasklet; struct sk_buff_head queue; @@ -166,35 +168,299 @@ int xfrm_parse_spi(struct sk_buff *skb, u8 nexthdr, __be32 *spi, __be32 *seq) } EXPORT_SYMBOL(xfrm_parse_spi); -int xfrm_prepare_input(struct xfrm_state *x, struct sk_buff *skb) +static int xfrm4_remove_beet_encap(struct xfrm_state *x, struct sk_buff *skb) +{ + struct iphdr *iph; + int optlen = 0; + int err = -EINVAL; + + if (unlikely(XFRM_MODE_SKB_CB(skb)->protocol == IPPROTO_BEETPH)) { + struct ip_beet_phdr *ph; + int phlen; + + if (!pskb_may_pull(skb, sizeof(*ph))) + goto out; + + ph = (struct ip_beet_phdr *)skb->data; + + phlen = sizeof(*ph) + ph->padlen; + optlen = ph->hdrlen * 8 + (IPV4_BEET_PHMAXLEN - phlen); + if (optlen < 0 || optlen & 3 || optlen > 250) + goto out; + + XFRM_MODE_SKB_CB(skb)->protocol = ph->nexthdr; + + if (!pskb_may_pull(skb, phlen)) + goto out; + __skb_pull(skb, phlen); + } + + skb_push(skb, sizeof(*iph)); + skb_reset_network_header(skb); + skb_mac_header_rebuild(skb); + + xfrm4_beet_make_header(skb); + + iph = ip_hdr(skb); + + iph->ihl += optlen / 4; + iph->tot_len = htons(skb->len); + iph->daddr = x->sel.daddr.a4; + iph->saddr = x->sel.saddr.a4; + iph->check = 0; + iph->check = ip_fast_csum(skb_network_header(skb), iph->ihl); + err = 0; +out: + return err; +} + +static void ipip_ecn_decapsulate(struct sk_buff *skb) +{ + struct iphdr *inner_iph = ipip_hdr(skb); + + if (INET_ECN_is_ce(XFRM_MODE_SKB_CB(skb)->tos)) + IP_ECN_set_ce(inner_iph); +} + +static int xfrm4_remove_tunnel_encap(struct xfrm_state *x, struct sk_buff *skb) { - struct xfrm_mode *inner_mode = x->inner_mode; + int err = -EINVAL; + + if (XFRM_MODE_SKB_CB(skb)->protocol != IPPROTO_IPIP) + goto out; + + if (!pskb_may_pull(skb, sizeof(struct iphdr))) + goto out; + + err = skb_unclone(skb, GFP_ATOMIC); + if (err) + goto out; + + if (x->props.flags & XFRM_STATE_DECAP_DSCP) + ipv4_copy_dscp(XFRM_MODE_SKB_CB(skb)->tos, ipip_hdr(skb)); + if (!(x->props.flags & XFRM_STATE_NOECN)) + ipip_ecn_decapsulate(skb); + + skb_reset_network_header(skb); + skb_mac_header_rebuild(skb); + if (skb->mac_len) + eth_hdr(skb)->h_proto = skb->protocol; + + err = 0; + +out: + return err; +} + +static void ipip6_ecn_decapsulate(struct sk_buff *skb) +{ + struct ipv6hdr *inner_iph = ipipv6_hdr(skb); + + if (INET_ECN_is_ce(XFRM_MODE_SKB_CB(skb)->tos)) + IP6_ECN_set_ce(skb, inner_iph); +} + +static int xfrm6_remove_tunnel_encap(struct xfrm_state *x, struct sk_buff *skb) +{ + int err = -EINVAL; + + if (XFRM_MODE_SKB_CB(skb)->protocol != IPPROTO_IPV6) + goto out; + if (!pskb_may_pull(skb, sizeof(struct ipv6hdr))) + goto out; + + err = skb_unclone(skb, GFP_ATOMIC); + if (err) + goto out; + + if (x->props.flags & XFRM_STATE_DECAP_DSCP) + ipv6_copy_dscp(ipv6_get_dsfield(ipv6_hdr(skb)), + ipipv6_hdr(skb)); + if (!(x->props.flags & XFRM_STATE_NOECN)) + ipip6_ecn_decapsulate(skb); + + skb_reset_network_header(skb); + skb_mac_header_rebuild(skb); + if (skb->mac_len) + eth_hdr(skb)->h_proto = skb->protocol; + + err = 0; + +out: + return err; +} + +static int xfrm6_remove_beet_encap(struct xfrm_state *x, struct sk_buff *skb) +{ + struct ipv6hdr *ip6h; + int size = sizeof(struct ipv6hdr); int err; - err = x->outer_mode->afinfo->extract_input(x, skb); + err = skb_cow_head(skb, size + skb->mac_len); if (err) + goto out; + + __skb_push(skb, size); + skb_reset_network_header(skb); + skb_mac_header_rebuild(skb); + + xfrm6_beet_make_header(skb); + + ip6h = ipv6_hdr(skb); + ip6h->payload_len = htons(skb->len - size); + ip6h->daddr = x->sel.daddr.in6; + ip6h->saddr = x->sel.saddr.in6; + err = 0; +out: + return err; +} + +/* Remove encapsulation header. + * + * The IP header will be moved over the top of the encapsulation + * header. + * + * On entry, the transport header shall point to where the IP header + * should be and the network header shall be set to where the IP + * header currently is. skb->data shall point to the start of the + * payload. + */ +static int +xfrm_inner_mode_encap_remove(struct xfrm_state *x, + const struct xfrm_mode *inner_mode, + struct sk_buff *skb) +{ + switch (inner_mode->encap) { + case XFRM_MODE_BEET: + if (inner_mode->family == AF_INET) + return xfrm4_remove_beet_encap(x, skb); + if (inner_mode->family == AF_INET6) + return xfrm6_remove_beet_encap(x, skb); + break; + case XFRM_MODE_TUNNEL: + if (inner_mode->family == AF_INET) + return xfrm4_remove_tunnel_encap(x, skb); + if (inner_mode->family == AF_INET6) + return xfrm6_remove_tunnel_encap(x, skb); + break; + } + + WARN_ON_ONCE(1); + return -EOPNOTSUPP; +} + +static int xfrm_prepare_input(struct xfrm_state *x, struct sk_buff *skb) +{ + const struct xfrm_mode *inner_mode = &x->inner_mode; + const struct xfrm_state_afinfo *afinfo; + int err = -EAFNOSUPPORT; + + rcu_read_lock(); + afinfo = xfrm_state_afinfo_get_rcu(x->outer_mode.family); + if (likely(afinfo)) + err = afinfo->extract_input(x, skb); + + if (err) { + rcu_read_unlock(); return err; + } if (x->sel.family == AF_UNSPEC) { inner_mode = xfrm_ip2inner_mode(x, XFRM_MODE_SKB_CB(skb)->protocol); - if (inner_mode == NULL) + if (!inner_mode) { + rcu_read_unlock(); return -EAFNOSUPPORT; + } } - skb->protocol = inner_mode->afinfo->eth_proto; - return inner_mode->input2(x, skb); + afinfo = xfrm_state_afinfo_get_rcu(inner_mode->family); + if (unlikely(!afinfo)) { + rcu_read_unlock(); + return -EAFNOSUPPORT; + } + + skb->protocol = afinfo->eth_proto; + rcu_read_unlock(); + return xfrm_inner_mode_encap_remove(x, inner_mode, skb); +} + +/* Remove encapsulation header. + * + * The IP header will be moved over the top of the encapsulation header. + * + * On entry, skb_transport_header() shall point to where the IP header + * should be and skb_network_header() shall be set to where the IP header + * currently is. skb->data shall point to the start of the payload. + */ +static int xfrm4_transport_input(struct xfrm_state *x, struct sk_buff *skb) +{ + int ihl = skb->data - skb_transport_header(skb); + + if (skb->transport_header != skb->network_header) { + memmove(skb_transport_header(skb), + skb_network_header(skb), ihl); + skb->network_header = skb->transport_header; + } + ip_hdr(skb)->tot_len = htons(skb->len + ihl); + skb_reset_transport_header(skb); + return 0; +} + +static int xfrm6_transport_input(struct xfrm_state *x, struct sk_buff *skb) +{ +#if IS_ENABLED(CONFIG_IPV6) + int ihl = skb->data - skb_transport_header(skb); + + if (skb->transport_header != skb->network_header) { + memmove(skb_transport_header(skb), + skb_network_header(skb), ihl); + skb->network_header = skb->transport_header; + } + ipv6_hdr(skb)->payload_len = htons(skb->len + ihl - + sizeof(struct ipv6hdr)); + skb_reset_transport_header(skb); + return 0; +#else + WARN_ON_ONCE(1); + return -EAFNOSUPPORT; +#endif +} + +static int xfrm_inner_mode_input(struct xfrm_state *x, + const struct xfrm_mode *inner_mode, + struct sk_buff *skb) +{ + switch (inner_mode->encap) { + case XFRM_MODE_BEET: + case XFRM_MODE_TUNNEL: + return xfrm_prepare_input(x, skb); + case XFRM_MODE_TRANSPORT: + if (inner_mode->family == AF_INET) + return xfrm4_transport_input(x, skb); + if (inner_mode->family == AF_INET6) + return xfrm6_transport_input(x, skb); + break; + case XFRM_MODE_ROUTEOPTIMIZATION: + WARN_ON_ONCE(1); + break; + default: + WARN_ON_ONCE(1); + break; + } + + return -EOPNOTSUPP; } -EXPORT_SYMBOL(xfrm_prepare_input); int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type) { + const struct xfrm_state_afinfo *afinfo; struct net *net = dev_net(skb->dev); + const struct xfrm_mode *inner_mode; int err; __be32 seq; __be32 seq_hi; struct xfrm_state *x = NULL; xfrm_address_t *daddr; - struct xfrm_mode *inner_mode; u32 mark = skb->mark; unsigned int family = AF_UNSPEC; int decaps = 0; @@ -216,7 +482,7 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type) goto drop; } - family = x->outer_mode->afinfo->family; + family = x->outer_mode.family; /* An encap_type of -1 indicates async resumption. */ if (encap_type == -1) { @@ -400,7 +666,7 @@ resume: XFRM_MODE_SKB_CB(skb)->protocol = nexthdr; - inner_mode = x->inner_mode; + inner_mode = &x->inner_mode; if (x->sel.family == AF_UNSPEC) { inner_mode = xfrm_ip2inner_mode(x, XFRM_MODE_SKB_CB(skb)->protocol); @@ -410,12 +676,12 @@ resume: } } - if (inner_mode->input(x, skb)) { + if (xfrm_inner_mode_input(x, inner_mode, skb)) { XFRM_INC_STATS(net, LINUX_MIB_XFRMINSTATEMODEERROR); goto drop; } - if (x->outer_mode->flags & XFRM_MODE_FLAG_TUNNEL) { + if (x->outer_mode.flags & XFRM_MODE_FLAG_TUNNEL) { decaps = 1; break; } @@ -425,7 +691,7 @@ resume: * transport mode so the outer address is identical. */ daddr = &x->id.daddr; - family = x->outer_mode->afinfo->family; + family = x->outer_mode.family; err = xfrm_parse_spi(skb, nexthdr, &spi, &seq); if (err < 0) { @@ -453,7 +719,12 @@ resume: if (xo) xfrm_gro = xo->flags & XFRM_GRO; - err = x->inner_mode->afinfo->transport_finish(skb, xfrm_gro || async); + err = -EAFNOSUPPORT; + rcu_read_lock(); + afinfo = xfrm_state_afinfo_get_rcu(x->inner_mode.family); + if (likely(afinfo)) + err = afinfo->transport_finish(skb, xfrm_gro || async); + rcu_read_unlock(); if (xfrm_gro) { sp = skb_sec_path(skb); if (sp) diff --git a/net/xfrm/xfrm_interface.c b/net/xfrm/xfrm_interface.c index 85fec98676d34abdee2524d7f0ea0fe24b8b6a9d..ad3a2555c517f961c26712c4fcab60728bcf5798 100644 --- a/net/xfrm/xfrm_interface.c +++ b/net/xfrm/xfrm_interface.c @@ -255,8 +255,8 @@ static void xfrmi_scrub_packet(struct sk_buff *skb, bool xnet) static int xfrmi_rcv_cb(struct sk_buff *skb, int err) { + const struct xfrm_mode *inner_mode; struct pcpu_sw_netstats *tstats; - struct xfrm_mode *inner_mode; struct net_device *dev; struct xfrm_state *x; struct xfrm_if *xi; @@ -284,7 +284,7 @@ static int xfrmi_rcv_cb(struct sk_buff *skb, int err) xnet = !net_eq(xi->net, dev_net(skb->dev)); if (xnet) { - inner_mode = x->inner_mode; + inner_mode = &x->inner_mode; if (x->sel.family == AF_UNSPEC) { inner_mode = xfrm_ip2inner_mode(x, XFRM_MODE_SKB_CB(skb)->protocol); @@ -296,7 +296,7 @@ static int xfrmi_rcv_cb(struct sk_buff *skb, int err) } if (!xfrm_policy_check(NULL, XFRM_POLICY_IN, skb, - inner_mode->afinfo->family)) + inner_mode->family)) return -EPERM; } diff --git a/net/xfrm/xfrm_output.c b/net/xfrm/xfrm_output.c index 9333153bafda9052c6c68f257ae0d2685d3d5441..a55510f9ff35a711af26e416c6168370fac04ac6 100644 --- a/net/xfrm/xfrm_output.c +++ b/net/xfrm/xfrm_output.c @@ -17,9 +17,13 @@ #include #include #include +#include #include +#include "xfrm_inout.h" + static int xfrm_output2(struct net *net, struct sock *sk, struct sk_buff *skb); +static int xfrm_inner_extract_output(struct xfrm_state *x, struct sk_buff *skb); static int xfrm_skb_check_space(struct sk_buff *skb) { @@ -50,6 +54,360 @@ static struct dst_entry *skb_dst_pop(struct sk_buff *skb) return child; } +/* Add encapsulation header. + * + * The IP header will be moved forward to make space for the encapsulation + * header. + */ +static int xfrm4_transport_output(struct xfrm_state *x, struct sk_buff *skb) +{ + struct iphdr *iph = ip_hdr(skb); + int ihl = iph->ihl * 4; + + skb_set_inner_transport_header(skb, skb_transport_offset(skb)); + + skb_set_network_header(skb, -x->props.header_len); + skb->mac_header = skb->network_header + + offsetof(struct iphdr, protocol); + skb->transport_header = skb->network_header + ihl; + __skb_pull(skb, ihl); + memmove(skb_network_header(skb), iph, ihl); + return 0; +} + +/* Add encapsulation header. + * + * The IP header and mutable extension headers will be moved forward to make + * space for the encapsulation header. + */ +static int xfrm6_transport_output(struct xfrm_state *x, struct sk_buff *skb) +{ +#if IS_ENABLED(CONFIG_IPV6) + struct ipv6hdr *iph; + u8 *prevhdr; + int hdr_len; + + iph = ipv6_hdr(skb); + skb_set_inner_transport_header(skb, skb_transport_offset(skb)); + + hdr_len = x->type->hdr_offset(x, skb, &prevhdr); + if (hdr_len < 0) + return hdr_len; + skb_set_mac_header(skb, + (prevhdr - x->props.header_len) - skb->data); + skb_set_network_header(skb, -x->props.header_len); + skb->transport_header = skb->network_header + hdr_len; + __skb_pull(skb, hdr_len); + memmove(ipv6_hdr(skb), iph, hdr_len); + return 0; +#else + WARN_ON_ONCE(1); + return -EAFNOSUPPORT; +#endif +} + +/* Add route optimization header space. + * + * The IP header and mutable extension headers will be moved forward to make + * space for the route optimization header. + */ +static int xfrm6_ro_output(struct xfrm_state *x, struct sk_buff *skb) +{ +#if IS_ENABLED(CONFIG_IPV6) + struct ipv6hdr *iph; + u8 *prevhdr; + int hdr_len; + + iph = ipv6_hdr(skb); + + hdr_len = x->type->hdr_offset(x, skb, &prevhdr); + if (hdr_len < 0) + return hdr_len; + skb_set_mac_header(skb, + (prevhdr - x->props.header_len) - skb->data); + skb_set_network_header(skb, -x->props.header_len); + skb->transport_header = skb->network_header + hdr_len; + __skb_pull(skb, hdr_len); + memmove(ipv6_hdr(skb), iph, hdr_len); + + x->lastused = ktime_get_real_seconds(); + + return 0; +#else + WARN_ON_ONCE(1); + return -EAFNOSUPPORT; +#endif +} + +/* Add encapsulation header. + * + * The top IP header will be constructed per draft-nikander-esp-beet-mode-06.txt. + */ +static int xfrm4_beet_encap_add(struct xfrm_state *x, struct sk_buff *skb) +{ + struct ip_beet_phdr *ph; + struct iphdr *top_iph; + int hdrlen, optlen; + + hdrlen = 0; + optlen = XFRM_MODE_SKB_CB(skb)->optlen; + if (unlikely(optlen)) + hdrlen += IPV4_BEET_PHMAXLEN - (optlen & 4); + + skb_set_network_header(skb, -x->props.header_len - hdrlen + + (XFRM_MODE_SKB_CB(skb)->ihl - sizeof(*top_iph))); + if (x->sel.family != AF_INET6) + skb->network_header += IPV4_BEET_PHMAXLEN; + skb->mac_header = skb->network_header + + offsetof(struct iphdr, protocol); + skb->transport_header = skb->network_header + sizeof(*top_iph); + + xfrm4_beet_make_header(skb); + + ph = __skb_pull(skb, XFRM_MODE_SKB_CB(skb)->ihl - hdrlen); + + top_iph = ip_hdr(skb); + + if (unlikely(optlen)) { + if (WARN_ON(optlen < 0)) + return -EINVAL; + + ph->padlen = 4 - (optlen & 4); + ph->hdrlen = optlen / 8; + ph->nexthdr = top_iph->protocol; + if (ph->padlen) + memset(ph + 1, IPOPT_NOP, ph->padlen); + + top_iph->protocol = IPPROTO_BEETPH; + top_iph->ihl = sizeof(struct iphdr) / 4; + } + + top_iph->saddr = x->props.saddr.a4; + top_iph->daddr = x->id.daddr.a4; + + return 0; +} + +/* Add encapsulation header. + * + * The top IP header will be constructed per RFC 2401. + */ +static int xfrm4_tunnel_encap_add(struct xfrm_state *x, struct sk_buff *skb) +{ + struct dst_entry *dst = skb_dst(skb); + struct iphdr *top_iph; + int flags; + + skb_set_inner_network_header(skb, skb_network_offset(skb)); + skb_set_inner_transport_header(skb, skb_transport_offset(skb)); + + skb_set_network_header(skb, -x->props.header_len); + skb->mac_header = skb->network_header + + offsetof(struct iphdr, protocol); + skb->transport_header = skb->network_header + sizeof(*top_iph); + top_iph = ip_hdr(skb); + + top_iph->ihl = 5; + top_iph->version = 4; + + top_iph->protocol = xfrm_af2proto(skb_dst(skb)->ops->family); + + /* DS disclosing depends on XFRM_SA_XFLAG_DONT_ENCAP_DSCP */ + if (x->props.extra_flags & XFRM_SA_XFLAG_DONT_ENCAP_DSCP) + top_iph->tos = 0; + else + top_iph->tos = XFRM_MODE_SKB_CB(skb)->tos; + top_iph->tos = INET_ECN_encapsulate(top_iph->tos, + XFRM_MODE_SKB_CB(skb)->tos); + + flags = x->props.flags; + if (flags & XFRM_STATE_NOECN) + IP_ECN_clear(top_iph); + + top_iph->frag_off = (flags & XFRM_STATE_NOPMTUDISC) ? + 0 : (XFRM_MODE_SKB_CB(skb)->frag_off & htons(IP_DF)); + + top_iph->ttl = ip4_dst_hoplimit(xfrm_dst_child(dst)); + + top_iph->saddr = x->props.saddr.a4; + top_iph->daddr = x->id.daddr.a4; + ip_select_ident(dev_net(dst->dev), skb, NULL); + + return 0; +} + +#if IS_ENABLED(CONFIG_IPV6) +static int xfrm6_tunnel_encap_add(struct xfrm_state *x, struct sk_buff *skb) +{ + struct dst_entry *dst = skb_dst(skb); + struct ipv6hdr *top_iph; + int dsfield; + + skb_set_inner_network_header(skb, skb_network_offset(skb)); + skb_set_inner_transport_header(skb, skb_transport_offset(skb)); + + skb_set_network_header(skb, -x->props.header_len); + skb->mac_header = skb->network_header + + offsetof(struct ipv6hdr, nexthdr); + skb->transport_header = skb->network_header + sizeof(*top_iph); + top_iph = ipv6_hdr(skb); + + top_iph->version = 6; + + memcpy(top_iph->flow_lbl, XFRM_MODE_SKB_CB(skb)->flow_lbl, + sizeof(top_iph->flow_lbl)); + top_iph->nexthdr = xfrm_af2proto(skb_dst(skb)->ops->family); + + if (x->props.extra_flags & XFRM_SA_XFLAG_DONT_ENCAP_DSCP) + dsfield = 0; + else + dsfield = XFRM_MODE_SKB_CB(skb)->tos; + dsfield = INET_ECN_encapsulate(dsfield, XFRM_MODE_SKB_CB(skb)->tos); + if (x->props.flags & XFRM_STATE_NOECN) + dsfield &= ~INET_ECN_MASK; + ipv6_change_dsfield(top_iph, 0, dsfield); + top_iph->hop_limit = ip6_dst_hoplimit(xfrm_dst_child(dst)); + top_iph->saddr = *(struct in6_addr *)&x->props.saddr; + top_iph->daddr = *(struct in6_addr *)&x->id.daddr; + return 0; +} + +static int xfrm6_beet_encap_add(struct xfrm_state *x, struct sk_buff *skb) +{ + struct ipv6hdr *top_iph; + struct ip_beet_phdr *ph; + int optlen, hdr_len; + + hdr_len = 0; + optlen = XFRM_MODE_SKB_CB(skb)->optlen; + if (unlikely(optlen)) + hdr_len += IPV4_BEET_PHMAXLEN - (optlen & 4); + + skb_set_network_header(skb, -x->props.header_len - hdr_len); + if (x->sel.family != AF_INET6) + skb->network_header += IPV4_BEET_PHMAXLEN; + skb->mac_header = skb->network_header + + offsetof(struct ipv6hdr, nexthdr); + skb->transport_header = skb->network_header + sizeof(*top_iph); + ph = __skb_pull(skb, XFRM_MODE_SKB_CB(skb)->ihl - hdr_len); + + xfrm6_beet_make_header(skb); + + top_iph = ipv6_hdr(skb); + if (unlikely(optlen)) { + if (WARN_ON(optlen < 0)) + return -EINVAL; + + ph->padlen = 4 - (optlen & 4); + ph->hdrlen = optlen / 8; + ph->nexthdr = top_iph->nexthdr; + if (ph->padlen) + memset(ph + 1, IPOPT_NOP, ph->padlen); + + top_iph->nexthdr = IPPROTO_BEETPH; + } + + top_iph->saddr = *(struct in6_addr *)&x->props.saddr; + top_iph->daddr = *(struct in6_addr *)&x->id.daddr; + return 0; +} +#endif + +/* Add encapsulation header. + * + * On exit, the transport header will be set to the start of the + * encapsulation header to be filled in by x->type->output and the mac + * header will be set to the nextheader (protocol for IPv4) field of the + * extension header directly preceding the encapsulation header, or in + * its absence, that of the top IP header. + * The value of the network header will always point to the top IP header + * while skb->data will point to the payload. + */ +static int xfrm4_prepare_output(struct xfrm_state *x, struct sk_buff *skb) +{ + int err; + + err = xfrm_inner_extract_output(x, skb); + if (err) + return err; + + IPCB(skb)->flags |= IPSKB_XFRM_TUNNEL_SIZE; + skb->protocol = htons(ETH_P_IP); + + switch (x->outer_mode.encap) { + case XFRM_MODE_BEET: + return xfrm4_beet_encap_add(x, skb); + case XFRM_MODE_TUNNEL: + return xfrm4_tunnel_encap_add(x, skb); + } + + WARN_ON_ONCE(1); + return -EOPNOTSUPP; +} + +static int xfrm6_prepare_output(struct xfrm_state *x, struct sk_buff *skb) +{ +#if IS_ENABLED(CONFIG_IPV6) + int err; + + err = xfrm_inner_extract_output(x, skb); + if (err) + return err; + + skb->ignore_df = 1; + skb->protocol = htons(ETH_P_IPV6); + + switch (x->outer_mode.encap) { + case XFRM_MODE_BEET: + return xfrm6_beet_encap_add(x, skb); + case XFRM_MODE_TUNNEL: + return xfrm6_tunnel_encap_add(x, skb); + default: + WARN_ON_ONCE(1); + return -EOPNOTSUPP; + } +#endif + WARN_ON_ONCE(1); + return -EAFNOSUPPORT; +} + +static int xfrm_outer_mode_output(struct xfrm_state *x, struct sk_buff *skb) +{ + switch (x->outer_mode.encap) { + case XFRM_MODE_BEET: + case XFRM_MODE_TUNNEL: + if (x->outer_mode.family == AF_INET) + return xfrm4_prepare_output(x, skb); + if (x->outer_mode.family == AF_INET6) + return xfrm6_prepare_output(x, skb); + break; + case XFRM_MODE_TRANSPORT: + if (x->outer_mode.family == AF_INET) + return xfrm4_transport_output(x, skb); + if (x->outer_mode.family == AF_INET6) + return xfrm6_transport_output(x, skb); + break; + case XFRM_MODE_ROUTEOPTIMIZATION: + if (x->outer_mode.family == AF_INET6) + return xfrm6_ro_output(x, skb); + WARN_ON_ONCE(1); + break; + default: + WARN_ON_ONCE(1); + break; + } + + return -EOPNOTSUPP; +} + +#if IS_ENABLED(CONFIG_NET_PKTGEN) +int pktgen_xfrm_outer_mode_output(struct xfrm_state *x, struct sk_buff *skb) +{ + return xfrm_outer_mode_output(x, skb); +} +EXPORT_SYMBOL_GPL(pktgen_xfrm_outer_mode_output); +#endif + static int xfrm_output_one(struct sk_buff *skb, int err) { struct dst_entry *dst = skb_dst(skb); @@ -68,7 +426,7 @@ static int xfrm_output_one(struct sk_buff *skb, int err) skb->mark = xfrm_smark_get(skb->mark, x); - err = x->outer_mode->output(x, skb); + err = xfrm_outer_mode_output(x, skb); if (err) { XFRM_INC_STATS(net, LINUX_MIB_XFRMOUTSTATEMODEERROR); goto error_nolock; @@ -131,7 +489,7 @@ resume: } skb_dst_set(skb, dst); x = dst->xfrm; - } while (x && !(x->outer_mode->flags & XFRM_MODE_FLAG_TUNNEL)); + } while (x && !(x->outer_mode.flags & XFRM_MODE_FLAG_TUNNEL)); return 0; @@ -258,20 +616,29 @@ out: } EXPORT_SYMBOL_GPL(xfrm_output); -int xfrm_inner_extract_output(struct xfrm_state *x, struct sk_buff *skb) +static int xfrm_inner_extract_output(struct xfrm_state *x, struct sk_buff *skb) { - struct xfrm_mode *inner_mode; + const struct xfrm_state_afinfo *afinfo; + const struct xfrm_mode *inner_mode; + int err = -EAFNOSUPPORT; + if (x->sel.family == AF_UNSPEC) inner_mode = xfrm_ip2inner_mode(x, xfrm_af2proto(skb_dst(skb)->ops->family)); else - inner_mode = x->inner_mode; + inner_mode = &x->inner_mode; if (inner_mode == NULL) return -EAFNOSUPPORT; - return inner_mode->afinfo->extract_output(x, skb); + + rcu_read_lock(); + afinfo = xfrm_state_afinfo_get_rcu(inner_mode->family); + if (likely(afinfo)) + err = afinfo->extract_output(x, skb); + rcu_read_unlock(); + + return err; } -EXPORT_SYMBOL_GPL(xfrm_inner_extract_output); void xfrm_local_error(struct sk_buff *skb, int mtu) { diff --git a/net/xfrm/xfrm_policy.c b/net/xfrm/xfrm_policy.c index a6b58df7a70f614084f38f3591592650db5829c7..410233c5681e55d51b72b254be4e08ea913f3ebb 100644 --- a/net/xfrm/xfrm_policy.c +++ b/net/xfrm/xfrm_policy.c @@ -27,10 +27,14 @@ #include #include #include +#include #include #include #include #include +#if IS_ENABLED(CONFIG_IPV6_MIP6) +#include +#endif #ifdef CONFIG_XFRM_STATISTICS #include #endif @@ -2450,18 +2454,10 @@ xfrm_tmpl_resolve(struct xfrm_policy **pols, int npols, const struct flowi *fl, static int xfrm_get_tos(const struct flowi *fl, int family) { - const struct xfrm_policy_afinfo *afinfo; - int tos; + if (family == AF_INET) + return IPTOS_RT_MASK & fl->u.ip4.flowi4_tos; - afinfo = xfrm_policy_get_afinfo(family); - if (!afinfo) - return 0; - - tos = afinfo->get_tos(fl); - - rcu_read_unlock(); - - return tos; + return 0; } static inline struct xfrm_dst *xfrm_alloc_dst(struct net *net, int family) @@ -2499,21 +2495,14 @@ static inline struct xfrm_dst *xfrm_alloc_dst(struct net *net, int family) return xdst; } -static inline int xfrm_init_path(struct xfrm_dst *path, struct dst_entry *dst, - int nfheader_len) +static void xfrm_init_path(struct xfrm_dst *path, struct dst_entry *dst, + int nfheader_len) { - const struct xfrm_policy_afinfo *afinfo = - xfrm_policy_get_afinfo(dst->ops->family); - int err; - - if (!afinfo) - return -EINVAL; - - err = afinfo->init_path(path, dst, nfheader_len); - - rcu_read_unlock(); - - return err; + if (dst->ops->family == AF_INET6) { + struct rt6_info *rt = (struct rt6_info *)dst; + path->path_cookie = rt6_get_cookie(rt); + path->u.rt6.rt6i_nfheader_len = nfheader_len; + } } static inline int xfrm_fill_dst(struct xfrm_dst *xdst, struct net_device *dev, @@ -2545,10 +2534,11 @@ static struct dst_entry *xfrm_bundle_create(struct xfrm_policy *policy, const struct flowi *fl, struct dst_entry *dst) { + const struct xfrm_state_afinfo *afinfo; + const struct xfrm_mode *inner_mode; struct net *net = xp_net(policy); unsigned long now = jiffies; struct net_device *dev; - struct xfrm_mode *inner_mode; struct xfrm_dst *xdst_prev = NULL; struct xfrm_dst *xdst0 = NULL; int i = 0; @@ -2594,7 +2584,7 @@ static struct dst_entry *xfrm_bundle_create(struct xfrm_policy *policy, goto put_states; } } else - inner_mode = xfrm[i]->inner_mode; + inner_mode = &xfrm[i]->inner_mode; xdst->route = dst; dst_copy_metrics(dst1, dst); @@ -2622,7 +2612,14 @@ static struct dst_entry *xfrm_bundle_create(struct xfrm_policy *policy, dst1->lastuse = now; dst1->input = dst_discard; - dst1->output = inner_mode->afinfo->output; + + rcu_read_lock(); + afinfo = xfrm_state_afinfo_get_rcu(inner_mode->family); + if (likely(afinfo)) + dst1->output = afinfo->output; + else + dst1->output = dst_discard_out; + rcu_read_unlock(); xdst_prev = xdst; @@ -3263,20 +3260,229 @@ xfrm_policy_ok(const struct xfrm_tmpl *tmpl, const struct sec_path *sp, int star return start; } +static void +decode_session4(struct sk_buff *skb, struct flowi *fl, bool reverse) +{ + const struct iphdr *iph = ip_hdr(skb); + u8 *xprth = skb_network_header(skb) + iph->ihl * 4; + struct flowi4 *fl4 = &fl->u.ip4; + int oif = 0; + + if (skb_dst(skb)) + oif = skb_dst(skb)->dev->ifindex; + + memset(fl4, 0, sizeof(struct flowi4)); + fl4->flowi4_mark = skb->mark; + fl4->flowi4_oif = reverse ? skb->skb_iif : oif; + + if (!ip_is_fragment(iph)) { + switch (iph->protocol) { + case IPPROTO_UDP: + case IPPROTO_UDPLITE: + case IPPROTO_TCP: + case IPPROTO_SCTP: + case IPPROTO_DCCP: + if (xprth + 4 < skb->data || + pskb_may_pull(skb, xprth + 4 - skb->data)) { + __be16 *ports; + + xprth = skb_network_header(skb) + iph->ihl * 4; + ports = (__be16 *)xprth; + + fl4->fl4_sport = ports[!!reverse]; + fl4->fl4_dport = ports[!reverse]; + } + break; + case IPPROTO_ICMP: + if (xprth + 2 < skb->data || + pskb_may_pull(skb, xprth + 2 - skb->data)) { + u8 *icmp; + + xprth = skb_network_header(skb) + iph->ihl * 4; + icmp = xprth; + + fl4->fl4_icmp_type = icmp[0]; + fl4->fl4_icmp_code = icmp[1]; + } + break; + case IPPROTO_ESP: + if (xprth + 4 < skb->data || + pskb_may_pull(skb, xprth + 4 - skb->data)) { + __be32 *ehdr; + + xprth = skb_network_header(skb) + iph->ihl * 4; + ehdr = (__be32 *)xprth; + + fl4->fl4_ipsec_spi = ehdr[0]; + } + break; + case IPPROTO_AH: + if (xprth + 8 < skb->data || + pskb_may_pull(skb, xprth + 8 - skb->data)) { + __be32 *ah_hdr; + + xprth = skb_network_header(skb) + iph->ihl * 4; + ah_hdr = (__be32 *)xprth; + + fl4->fl4_ipsec_spi = ah_hdr[1]; + } + break; + case IPPROTO_COMP: + if (xprth + 4 < skb->data || + pskb_may_pull(skb, xprth + 4 - skb->data)) { + __be16 *ipcomp_hdr; + + xprth = skb_network_header(skb) + iph->ihl * 4; + ipcomp_hdr = (__be16 *)xprth; + + fl4->fl4_ipsec_spi = htonl(ntohs(ipcomp_hdr[1])); + } + break; + case IPPROTO_GRE: + if (xprth + 12 < skb->data || + pskb_may_pull(skb, xprth + 12 - skb->data)) { + __be16 *greflags; + __be32 *gre_hdr; + + xprth = skb_network_header(skb) + iph->ihl * 4; + greflags = (__be16 *)xprth; + gre_hdr = (__be32 *)xprth; + + if (greflags[0] & GRE_KEY) { + if (greflags[0] & GRE_CSUM) + gre_hdr++; + fl4->fl4_gre_key = gre_hdr[1]; + } + } + break; + default: + fl4->fl4_ipsec_spi = 0; + break; + } + } + fl4->flowi4_proto = iph->protocol; + fl4->daddr = reverse ? iph->saddr : iph->daddr; + fl4->saddr = reverse ? iph->daddr : iph->saddr; + fl4->flowi4_tos = iph->tos; +} + +#if IS_ENABLED(CONFIG_IPV6) +static void +decode_session6(struct sk_buff *skb, struct flowi *fl, bool reverse) +{ + struct flowi6 *fl6 = &fl->u.ip6; + int onlyproto = 0; + const struct ipv6hdr *hdr = ipv6_hdr(skb); + u32 offset = sizeof(*hdr); + struct ipv6_opt_hdr *exthdr; + const unsigned char *nh = skb_network_header(skb); + u16 nhoff = IP6CB(skb)->nhoff; + int oif = 0; + u8 nexthdr; + + if (!nhoff) + nhoff = offsetof(struct ipv6hdr, nexthdr); + + nexthdr = nh[nhoff]; + + if (skb_dst(skb)) + oif = skb_dst(skb)->dev->ifindex; + + memset(fl6, 0, sizeof(struct flowi6)); + fl6->flowi6_mark = skb->mark; + fl6->flowi6_oif = reverse ? skb->skb_iif : oif; + + fl6->daddr = reverse ? hdr->saddr : hdr->daddr; + fl6->saddr = reverse ? hdr->daddr : hdr->saddr; + + while (nh + offset + sizeof(*exthdr) < skb->data || + pskb_may_pull(skb, nh + offset + sizeof(*exthdr) - skb->data)) { + nh = skb_network_header(skb); + exthdr = (struct ipv6_opt_hdr *)(nh + offset); + + switch (nexthdr) { + case NEXTHDR_FRAGMENT: + onlyproto = 1; + /* fall through */ + case NEXTHDR_ROUTING: + case NEXTHDR_HOP: + case NEXTHDR_DEST: + offset += ipv6_optlen(exthdr); + nexthdr = exthdr->nexthdr; + exthdr = (struct ipv6_opt_hdr *)(nh + offset); + break; + case IPPROTO_UDP: + case IPPROTO_UDPLITE: + case IPPROTO_TCP: + case IPPROTO_SCTP: + case IPPROTO_DCCP: + if (!onlyproto && (nh + offset + 4 < skb->data || + pskb_may_pull(skb, nh + offset + 4 - skb->data))) { + __be16 *ports; + + nh = skb_network_header(skb); + ports = (__be16 *)(nh + offset); + fl6->fl6_sport = ports[!!reverse]; + fl6->fl6_dport = ports[!reverse]; + } + fl6->flowi6_proto = nexthdr; + return; + case IPPROTO_ICMPV6: + if (!onlyproto && (nh + offset + 2 < skb->data || + pskb_may_pull(skb, nh + offset + 2 - skb->data))) { + u8 *icmp; + + nh = skb_network_header(skb); + icmp = (u8 *)(nh + offset); + fl6->fl6_icmp_type = icmp[0]; + fl6->fl6_icmp_code = icmp[1]; + } + fl6->flowi6_proto = nexthdr; + return; +#if IS_ENABLED(CONFIG_IPV6_MIP6) + case IPPROTO_MH: + offset += ipv6_optlen(exthdr); + if (!onlyproto && (nh + offset + 3 < skb->data || + pskb_may_pull(skb, nh + offset + 3 - skb->data))) { + struct ip6_mh *mh; + + nh = skb_network_header(skb); + mh = (struct ip6_mh *)(nh + offset); + fl6->fl6_mh_type = mh->ip6mh_type; + } + fl6->flowi6_proto = nexthdr; + return; +#endif + /* XXX Why are there these headers? */ + case IPPROTO_AH: + case IPPROTO_ESP: + case IPPROTO_COMP: + default: + fl6->fl6_ipsec_spi = 0; + fl6->flowi6_proto = nexthdr; + return; + } + } +} +#endif + int __xfrm_decode_session(struct sk_buff *skb, struct flowi *fl, unsigned int family, int reverse) { - const struct xfrm_policy_afinfo *afinfo = xfrm_policy_get_afinfo(family); - int err; - - if (unlikely(afinfo == NULL)) + switch (family) { + case AF_INET: + decode_session4(skb, fl, reverse); + break; +#if IS_ENABLED(CONFIG_IPV6) + case AF_INET6: + decode_session6(skb, fl, reverse); + break; +#endif + default: return -EAFNOSUPPORT; + } - afinfo->decode_session(skb, fl, reverse); - - err = security_xfrm_decode_session(skb, &fl->flowi_secid); - rcu_read_unlock(); - return err; + return security_xfrm_decode_session(skb, &fl->flowi_secid); } EXPORT_SYMBOL(__xfrm_decode_session); diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c index c62f712fdaf71d51bf605e4412ec0274d4a0f633..c5d81316330b34adbc13a6a059b765de10c5c595 100644 --- a/net/xfrm/xfrm_state.c +++ b/net/xfrm/xfrm_state.c @@ -173,7 +173,7 @@ static DEFINE_SPINLOCK(xfrm_state_gc_lock); int __xfrm_state_delete(struct xfrm_state *x); int km_query(struct xfrm_state *x, struct xfrm_tmpl *t, struct xfrm_policy *pol); -bool km_is_alive(const struct km_event *c); +static bool km_is_alive(const struct km_event *c); void km_state_expired(struct xfrm_state *x, int hard, u32 portid); static DEFINE_SPINLOCK(xfrm_type_lock); @@ -330,100 +330,67 @@ static void xfrm_put_type_offload(const struct xfrm_type_offload *type) module_put(type->owner); } -static DEFINE_SPINLOCK(xfrm_mode_lock); -int xfrm_register_mode(struct xfrm_mode *mode, int family) -{ - struct xfrm_state_afinfo *afinfo; - struct xfrm_mode **modemap; - int err; - - if (unlikely(mode->encap >= XFRM_MODE_MAX)) - return -EINVAL; - - afinfo = xfrm_state_get_afinfo(family); - if (unlikely(afinfo == NULL)) - return -EAFNOSUPPORT; - - err = -EEXIST; - modemap = afinfo->mode_map; - spin_lock_bh(&xfrm_mode_lock); - if (modemap[mode->encap]) - goto out; - - err = -ENOENT; - if (!try_module_get(afinfo->owner)) - goto out; - - mode->afinfo = afinfo; - modemap[mode->encap] = mode; - err = 0; - -out: - spin_unlock_bh(&xfrm_mode_lock); - rcu_read_unlock(); - return err; -} -EXPORT_SYMBOL(xfrm_register_mode); - -int xfrm_unregister_mode(struct xfrm_mode *mode, int family) -{ - struct xfrm_state_afinfo *afinfo; - struct xfrm_mode **modemap; - int err; - - if (unlikely(mode->encap >= XFRM_MODE_MAX)) - return -EINVAL; - - afinfo = xfrm_state_get_afinfo(family); - if (unlikely(afinfo == NULL)) - return -EAFNOSUPPORT; - - err = -ENOENT; - modemap = afinfo->mode_map; - spin_lock_bh(&xfrm_mode_lock); - if (likely(modemap[mode->encap] == mode)) { - modemap[mode->encap] = NULL; - module_put(mode->afinfo->owner); - err = 0; - } - - spin_unlock_bh(&xfrm_mode_lock); - rcu_read_unlock(); - return err; -} -EXPORT_SYMBOL(xfrm_unregister_mode); - -static struct xfrm_mode *xfrm_get_mode(unsigned int encap, int family) -{ - struct xfrm_state_afinfo *afinfo; - struct xfrm_mode *mode; - int modload_attempted = 0; +static const struct xfrm_mode xfrm4_mode_map[XFRM_MODE_MAX] = { + [XFRM_MODE_BEET] = { + .encap = XFRM_MODE_BEET, + .flags = XFRM_MODE_FLAG_TUNNEL, + .family = AF_INET, + }, + [XFRM_MODE_TRANSPORT] = { + .encap = XFRM_MODE_TRANSPORT, + .family = AF_INET, + }, + [XFRM_MODE_TUNNEL] = { + .encap = XFRM_MODE_TUNNEL, + .flags = XFRM_MODE_FLAG_TUNNEL, + .family = AF_INET, + }, +}; + +static const struct xfrm_mode xfrm6_mode_map[XFRM_MODE_MAX] = { + [XFRM_MODE_BEET] = { + .encap = XFRM_MODE_BEET, + .flags = XFRM_MODE_FLAG_TUNNEL, + .family = AF_INET6, + }, + [XFRM_MODE_ROUTEOPTIMIZATION] = { + .encap = XFRM_MODE_ROUTEOPTIMIZATION, + .family = AF_INET6, + }, + [XFRM_MODE_TRANSPORT] = { + .encap = XFRM_MODE_TRANSPORT, + .family = AF_INET6, + }, + [XFRM_MODE_TUNNEL] = { + .encap = XFRM_MODE_TUNNEL, + .flags = XFRM_MODE_FLAG_TUNNEL, + .family = AF_INET6, + }, +}; + +static const struct xfrm_mode *xfrm_get_mode(unsigned int encap, int family) +{ + const struct xfrm_mode *mode; if (unlikely(encap >= XFRM_MODE_MAX)) return NULL; -retry: - afinfo = xfrm_state_get_afinfo(family); - if (unlikely(afinfo == NULL)) - return NULL; - - mode = READ_ONCE(afinfo->mode_map[encap]); - if (unlikely(mode && !try_module_get(mode->owner))) - mode = NULL; - - rcu_read_unlock(); - if (!mode && !modload_attempted) { - request_module("xfrm-mode-%d-%d", family, encap); - modload_attempted = 1; - goto retry; + switch (family) { + case AF_INET: + mode = &xfrm4_mode_map[encap]; + if (mode->family == family) + return mode; + break; + case AF_INET6: + mode = &xfrm6_mode_map[encap]; + if (mode->family == family) + return mode; + break; + default: + break; } - return mode; -} - -static void xfrm_put_mode(struct xfrm_mode *mode) -{ - module_put(mode->owner); + return NULL; } void xfrm_state_free(struct xfrm_state *x) @@ -444,12 +411,6 @@ static void ___xfrm_state_destroy(struct xfrm_state *x) kfree(x->coaddr); kfree(x->replay_esn); kfree(x->preplay_esn); - if (x->inner_mode) - xfrm_put_mode(x->inner_mode); - if (x->inner_mode_iaf) - xfrm_put_mode(x->inner_mode_iaf); - if (x->outer_mode) - xfrm_put_mode(x->outer_mode); if (x->type_offload) xfrm_put_type_offload(x->type_offload); if (x->type) { @@ -591,8 +552,6 @@ struct xfrm_state *xfrm_state_alloc(struct net *net) x->lft.hard_packet_limit = XFRM_INF; x->replay_maxage = 0; x->replay_maxdiff = 0; - x->inner_mode = NULL; - x->inner_mode_iaf = NULL; spin_lock_init(&x->lock); } return x; @@ -2072,7 +2031,7 @@ int km_report(struct net *net, u8 proto, struct xfrm_selector *sel, xfrm_address } EXPORT_SYMBOL(km_report); -bool km_is_alive(const struct km_event *c) +static bool km_is_alive(const struct km_event *c) { struct xfrm_mgr *km; bool is_alive = false; @@ -2088,7 +2047,6 @@ bool km_is_alive(const struct km_event *c) return is_alive; } -EXPORT_SYMBOL(km_is_alive); int xfrm_user_policy(struct sock *sk, int optname, u8 __user *optval, int optlen) { @@ -2201,6 +2159,7 @@ struct xfrm_state_afinfo *xfrm_state_afinfo_get_rcu(unsigned int family) return rcu_dereference(xfrm_state_afinfo[family]); } +EXPORT_SYMBOL_GPL(xfrm_state_afinfo_get_rcu); struct xfrm_state_afinfo *xfrm_state_get_afinfo(unsigned int family) { @@ -2248,8 +2207,9 @@ int xfrm_state_mtu(struct xfrm_state *x, int mtu) int __xfrm_init_state(struct xfrm_state *x, bool init_replay, bool offload) { - struct xfrm_state_afinfo *afinfo; - struct xfrm_mode *inner_mode; + const struct xfrm_state_afinfo *afinfo; + const struct xfrm_mode *inner_mode; + const struct xfrm_mode *outer_mode; int family = x->props.family; int err; @@ -2275,25 +2235,22 @@ int __xfrm_init_state(struct xfrm_state *x, bool init_replay, bool offload) goto error; if (!(inner_mode->flags & XFRM_MODE_FLAG_TUNNEL) && - family != x->sel.family) { - xfrm_put_mode(inner_mode); + family != x->sel.family) goto error; - } - x->inner_mode = inner_mode; + x->inner_mode = *inner_mode; } else { - struct xfrm_mode *inner_mode_iaf; + const struct xfrm_mode *inner_mode_iaf; int iafamily = AF_INET; inner_mode = xfrm_get_mode(x->props.mode, x->props.family); if (inner_mode == NULL) goto error; - if (!(inner_mode->flags & XFRM_MODE_FLAG_TUNNEL)) { - xfrm_put_mode(inner_mode); + if (!(inner_mode->flags & XFRM_MODE_FLAG_TUNNEL)) goto error; - } - x->inner_mode = inner_mode; + + x->inner_mode = *inner_mode; if (x->props.family == AF_INET) iafamily = AF_INET6; @@ -2301,9 +2258,7 @@ int __xfrm_init_state(struct xfrm_state *x, bool init_replay, bool offload) inner_mode_iaf = xfrm_get_mode(x->props.mode, iafamily); if (inner_mode_iaf) { if (inner_mode_iaf->flags & XFRM_MODE_FLAG_TUNNEL) - x->inner_mode_iaf = inner_mode_iaf; - else - xfrm_put_mode(inner_mode_iaf); + x->inner_mode_iaf = *inner_mode_iaf; } } @@ -2317,12 +2272,13 @@ int __xfrm_init_state(struct xfrm_state *x, bool init_replay, bool offload) if (err) goto error; - x->outer_mode = xfrm_get_mode(x->props.mode, family); - if (x->outer_mode == NULL) { + outer_mode = xfrm_get_mode(x->props.mode, family); + if (!outer_mode) { err = -EPROTONOSUPPORT; goto error; } + x->outer_mode = *outer_mode; if (init_replay) { err = xfrm_init_replay(x); if (err) diff --git a/net/xfrm/xfrm_user.c b/net/xfrm/xfrm_user.c index 6916931b1de1ce6ea3f3a419a0e84aa353142eae..eb8d14389601646d44ae438058f9a429585e40eb 100644 --- a/net/xfrm/xfrm_user.c +++ b/net/xfrm/xfrm_user.c @@ -1006,8 +1006,8 @@ static int xfrm_dump_sa(struct sk_buff *skb, struct netlink_callback *cb) u8 proto = 0; int err; - err = nlmsg_parse(cb->nlh, 0, attrs, XFRMA_MAX, xfrma_policy, - cb->extack); + err = nlmsg_parse_deprecated(cb->nlh, 0, attrs, XFRMA_MAX, + xfrma_policy, cb->extack); if (err < 0) return err; @@ -2644,9 +2644,9 @@ static int xfrm_user_rcv_msg(struct sk_buff *skb, struct nlmsghdr *nlh, } } - err = nlmsg_parse(nlh, xfrm_msg_min[type], attrs, - link->nla_max ? : XFRMA_MAX, - link->nla_pol ? : xfrma_policy, extack); + err = nlmsg_parse_deprecated(nlh, xfrm_msg_min[type], attrs, + link->nla_max ? : XFRMA_MAX, + link->nla_pol ? : xfrma_policy, extack); if (err < 0) return err; diff --git a/samples/bpf/.gitignore b/samples/bpf/.gitignore index dbb817dbacfcaefd42daa9d55ca04addb05fc83a..c7498457595a152b0579c0e710dee1766db251d0 100644 --- a/samples/bpf/.gitignore +++ b/samples/bpf/.gitignore @@ -1,5 +1,6 @@ cpustat fds_example +hbm lathist lwt_len_hist map_perf_test @@ -44,5 +45,6 @@ xdp_redirect_cpu xdp_redirect_map xdp_router_ipv4 xdp_rxq_info +xdp_sample_pkts xdp_tx_iptunnel xdpsock diff --git a/samples/bpf/asm_goto_workaround.h b/samples/bpf/asm_goto_workaround.h index 5cd7c1d1a5d5637247cf51ba86fd4543d7190a46..7409722727ca16fcffa9c7bc5fe03f6a749cc4bf 100644 --- a/samples/bpf/asm_goto_workaround.h +++ b/samples/bpf/asm_goto_workaround.h @@ -13,4 +13,5 @@ #define asm_volatile_goto(x...) asm volatile("invalid use of asm_volatile_goto") #endif +#define volatile(x...) volatile("") #endif diff --git a/samples/bpf/offwaketime_user.c b/samples/bpf/offwaketime_user.c index f06063af9fcb9a9fe4ca2eb90c68669b2ad6ca11..bb315ce1b8660cf86fd70a2b3d917d0742064706 100644 --- a/samples/bpf/offwaketime_user.c +++ b/samples/bpf/offwaketime_user.c @@ -28,6 +28,11 @@ static void print_ksym(__u64 addr) if (!addr) return; sym = ksym_search(addr); + if (!sym) { + printf("ksym not found. Is kallsyms loaded?\n"); + return; + } + if (PRINT_RAW_ADDR) printf("%s/%llx;", sym->name, addr); else diff --git a/samples/bpf/sampleip_user.c b/samples/bpf/sampleip_user.c index 216c7ecbbbe9ef201ff3e7225e0cbf71a4f44718..23b90a45c80251ca3765f75873d1a85d423cf1ab 100644 --- a/samples/bpf/sampleip_user.c +++ b/samples/bpf/sampleip_user.c @@ -109,6 +109,11 @@ static void print_ip_map(int fd) for (i = 0; i < max; i++) { if (counts[i].ip > PAGE_OFFSET) { sym = ksym_search(counts[i].ip); + if (!sym) { + printf("ksym not found. Is kallsyms loaded?\n"); + continue; + } + printf("0x%-17llx %-32s %u\n", counts[i].ip, sym->name, counts[i].count); } else { diff --git a/samples/bpf/spintest_user.c b/samples/bpf/spintest_user.c index 8d3e9cfa190978e56d00bde53a22d3f72f9e9b7e..2556af2d9b3e8e2725bfc2c062f99d588b7f727a 100644 --- a/samples/bpf/spintest_user.c +++ b/samples/bpf/spintest_user.c @@ -37,8 +37,13 @@ int main(int ac, char **argv) bpf_map_lookup_elem(map_fd[0], &next_key, &value); assert(next_key == value); sym = ksym_search(value); - printf(" %s", sym->name); key = next_key; + if (!sym) { + printf("ksym not found. Is kallsyms loaded?\n"); + continue; + } + + printf(" %s", sym->name); } if (key) printf("\n"); diff --git a/samples/bpf/trace_event_user.c b/samples/bpf/trace_event_user.c index d08046ab81f043505e0ea42a2e8c85661ae68f76..d4178f60e075f099efe340c4d22b05ce4df013ef 100644 --- a/samples/bpf/trace_event_user.c +++ b/samples/bpf/trace_event_user.c @@ -34,6 +34,11 @@ static void print_ksym(__u64 addr) if (!addr) return; sym = ksym_search(addr); + if (!sym) { + printf("ksym not found. Is kallsyms loaded?\n"); + return; + } + printf("%s;", sym->name); if (!strcmp(sym->name, "sys_read")) sys_read_seen = true; diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh index dc0e8c5a140239c6a3bb13ccfbca5ab522328a28..e3c06b9482a2597598b5d13e61582b31026e6596 100755 --- a/scripts/link-vmlinux.sh +++ b/scripts/link-vmlinux.sh @@ -35,7 +35,7 @@ set -e info() { if [ "${quiet}" != "silent_" ]; then - printf " %-7s %s\n" ${1} ${2} + printf " %-7s %s\n" "${1}" "${2}" fi } @@ -91,6 +91,25 @@ vmlinux_link() fi } +# generate .BTF typeinfo from DWARF debuginfo +gen_btf() +{ + local pahole_ver; + + if ! [ -x "$(command -v ${PAHOLE})" ]; then + info "BTF" "${1}: pahole (${PAHOLE}) is not available" + return 0 + fi + + pahole_ver=$(${PAHOLE} --version | sed -E 's/v([0-9]+)\.([0-9]+)/\1\2/') + if [ "${pahole_ver}" -lt "113" ]; then + info "BTF" "${1}: pahole version $(${PAHOLE} --version) is too old, need at least v1.13" + return 0 + fi + + info "BTF" ${1} + LLVM_OBJCOPY=${OBJCOPY} ${PAHOLE} -J ${1} +} # Create ${2} .o file with all symbols from the ${1} object file kallsyms() @@ -248,6 +267,10 @@ fi info LD vmlinux vmlinux_link "${kallsymso}" vmlinux +if [ -n "${CONFIG_DEBUG_INFO_BTF}" ]; then + gen_btf vmlinux +fi + if [ -n "${CONFIG_BUILDTIME_EXTABLE_SORT}" ]; then info SORTEX vmlinux sortextable vmlinux diff --git a/tools/arch/arm64/include/asm/barrier.h b/tools/arch/arm64/include/asm/barrier.h index 378c051fa1776534b0e1fca7e56567f9f8a3d2d4..3b9b41331c4f16ec4339ab20e32d434d89336c37 100644 --- a/tools/arch/arm64/include/asm/barrier.h +++ b/tools/arch/arm64/include/asm/barrier.h @@ -14,6 +14,16 @@ #define wmb() asm volatile("dmb ishst" ::: "memory") #define rmb() asm volatile("dmb ishld" ::: "memory") +/* + * Kernel uses dmb variants on arm64 for smp_*() barriers. Pretty much the same + * implementation as above mb()/wmb()/rmb(), though for the latter kernel uses + * dsb. In any case, should above mb()/wmb()/rmb() change, make sure the below + * smp_*() don't. + */ +#define smp_mb() asm volatile("dmb ish" ::: "memory") +#define smp_wmb() asm volatile("dmb ishst" ::: "memory") +#define smp_rmb() asm volatile("dmb ishld" ::: "memory") + #define smp_store_release(p, v) \ do { \ union { typeof(*p) __val; char __c[1]; } __u = \ diff --git a/tools/arch/x86/include/asm/barrier.h b/tools/arch/x86/include/asm/barrier.h index 58919868473c134f63a2ad42bd64bac8ac46fabd..0adf295dd5b6aa9e8639d1956f481827a61fd2ca 100644 --- a/tools/arch/x86/include/asm/barrier.h +++ b/tools/arch/x86/include/asm/barrier.h @@ -21,9 +21,12 @@ #define rmb() asm volatile("lock; addl $0,0(%%esp)" ::: "memory") #define wmb() asm volatile("lock; addl $0,0(%%esp)" ::: "memory") #elif defined(__x86_64__) -#define mb() asm volatile("mfence":::"memory") -#define rmb() asm volatile("lfence":::"memory") +#define mb() asm volatile("mfence" ::: "memory") +#define rmb() asm volatile("lfence" ::: "memory") #define wmb() asm volatile("sfence" ::: "memory") +#define smp_rmb() barrier() +#define smp_wmb() barrier() +#define smp_mb() asm volatile("lock; addl $0,-132(%%rsp)" ::: "memory", "cc") #endif #if defined(__x86_64__) diff --git a/tools/bpf/bpftool/.gitignore b/tools/bpf/bpftool/.gitignore index 67167e44b72662d15fe9336359589e1d8f6fe5b8..8248b8dd89d4b9cd4b2d9622623d43f2140a42f2 100644 --- a/tools/bpf/bpftool/.gitignore +++ b/tools/bpf/bpftool/.gitignore @@ -1,5 +1,5 @@ *.d -bpftool +/bpftool bpftool*.8 bpf-helpers.* FEATURE-DUMP.bpftool diff --git a/tools/bpf/bpftool/Documentation/bpftool-btf.rst b/tools/bpf/bpftool/Documentation/bpftool-btf.rst new file mode 100644 index 0000000000000000000000000000000000000000..2dbc1413fabd8caa116ffff4df999d3ec0c4dece --- /dev/null +++ b/tools/bpf/bpftool/Documentation/bpftool-btf.rst @@ -0,0 +1,222 @@ +================ +bpftool-btf +================ +------------------------------------------------------------------------------- +tool for inspection of BTF data +------------------------------------------------------------------------------- + +:Manual section: 8 + +SYNOPSIS +======== + + **bpftool** [*OPTIONS*] **btf** *COMMAND* + + *OPTIONS* := { { **-j** | **--json** } [{ **-p** | **--pretty** }] } + + *COMMANDS* := { **dump** | **help** } + +BTF COMMANDS +============= + +| **bpftool** **btf dump** *BTF_SRC* +| **bpftool** **btf help** +| +| *BTF_SRC* := { **id** *BTF_ID* | **prog** *PROG* | **map** *MAP* [{**key** | **value** | **kv** | **all**}] | **file** *FILE* } +| *MAP* := { **id** *MAP_ID* | **pinned** *FILE* } +| *PROG* := { **id** *PROG_ID* | **pinned** *FILE* | **tag** *PROG_TAG* } + +DESCRIPTION +=========== + **bpftool btf dump** *BTF_SRC* + Dump BTF entries from a given *BTF_SRC*. + + When **id** is specified, BTF object with that ID will be + loaded and all its BTF types emitted. + + When **map** is provided, it's expected that map has + associated BTF object with BTF types describing key and + value. It's possible to select whether to dump only BTF + type(s) associated with key (**key**), value (**value**), + both key and value (**kv**), or all BTF types present in + associated BTF object (**all**). If not specified, **kv** + is assumed. + + When **prog** is provided, it's expected that program has + associated BTF object with BTF types. + + When specifying *FILE*, an ELF file is expected, containing + .BTF section with well-defined BTF binary format data, + typically produced by clang or pahole. + + **bpftool btf help** + Print short help message. + +OPTIONS +======= + -h, --help + Print short generic help message (similar to **bpftool help**). + + -V, --version + Print version number (similar to **bpftool version**). + + -j, --json + Generate JSON output. For commands that cannot produce JSON, this + option has no effect. + + -p, --pretty + Generate human-readable JSON output. Implies **-j**. + +EXAMPLES +======== +**# bpftool btf dump id 1226** +:: + + [1] PTR '(anon)' type_id=2 + [2] STRUCT 'dummy_tracepoint_args' size=16 vlen=2 + 'pad' type_id=3 bits_offset=0 + 'sock' type_id=4 bits_offset=64 + [3] INT 'long long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none) + [4] PTR '(anon)' type_id=5 + [5] FWD 'sock' fwd_kind=union + +This gives an example of default output for all supported BTF kinds. + +**$ cat prog.c** +:: + + struct fwd_struct; + + enum my_enum { + VAL1 = 3, + VAL2 = 7, + }; + + typedef struct my_struct my_struct_t; + + struct my_struct { + const unsigned int const_int_field; + int bitfield_field: 4; + char arr_field[16]; + const struct fwd_struct *restrict fwd_field; + enum my_enum enum_field; + volatile my_struct_t *typedef_ptr_field; + }; + + union my_union { + int a; + struct my_struct b; + }; + + struct my_struct struct_global_var __attribute__((section("data_sec"))) = { + .bitfield_field = 3, + .enum_field = VAL1, + }; + int global_var __attribute__((section("data_sec"))) = 7; + + __attribute__((noinline)) + int my_func(union my_union *arg1, int arg2) + { + static int static_var __attribute__((section("data_sec"))) = 123; + static_var++; + return static_var; + } + +**$ bpftool btf dump file prog.o** +:: + + [1] PTR '(anon)' type_id=2 + [2] UNION 'my_union' size=48 vlen=2 + 'a' type_id=3 bits_offset=0 + 'b' type_id=4 bits_offset=0 + [3] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED + [4] STRUCT 'my_struct' size=48 vlen=6 + 'const_int_field' type_id=5 bits_offset=0 + 'bitfield_field' type_id=3 bits_offset=32 bitfield_size=4 + 'arr_field' type_id=8 bits_offset=40 + 'fwd_field' type_id=10 bits_offset=192 + 'enum_field' type_id=14 bits_offset=256 + 'typedef_ptr_field' type_id=15 bits_offset=320 + [5] CONST '(anon)' type_id=6 + [6] INT 'unsigned int' size=4 bits_offset=0 nr_bits=32 encoding=(none) + [7] INT 'char' size=1 bits_offset=0 nr_bits=8 encoding=SIGNED + [8] ARRAY '(anon)' type_id=7 index_type_id=9 nr_elems=16 + [9] INT '__ARRAY_SIZE_TYPE__' size=4 bits_offset=0 nr_bits=32 encoding=(none) + [10] RESTRICT '(anon)' type_id=11 + [11] PTR '(anon)' type_id=12 + [12] CONST '(anon)' type_id=13 + [13] FWD 'fwd_struct' fwd_kind=union + [14] ENUM 'my_enum' size=4 vlen=2 + 'VAL1' val=3 + 'VAL2' val=7 + [15] PTR '(anon)' type_id=16 + [16] VOLATILE '(anon)' type_id=17 + [17] TYPEDEF 'my_struct_t' type_id=4 + [18] FUNC_PROTO '(anon)' ret_type_id=3 vlen=2 + 'arg1' type_id=1 + 'arg2' type_id=3 + [19] FUNC 'my_func' type_id=18 + [20] VAR 'struct_global_var' type_id=4, linkage=global-alloc + [21] VAR 'global_var' type_id=3, linkage=global-alloc + [22] VAR 'my_func.static_var' type_id=3, linkage=static + [23] DATASEC 'data_sec' size=0 vlen=3 + type_id=20 offset=0 size=48 + type_id=21 offset=0 size=4 + type_id=22 offset=52 size=4 + +The following commands print BTF types associated with specified map's key, +value, both key and value, and all BTF types, respectively. By default, both +key and value types will be printed. + +**# bpftool btf dump map id 123 key** + +:: + + [39] TYPEDEF 'u32' type_id=37 + +**# bpftool btf dump map id 123 value** + +:: + + [86] PTR '(anon)' type_id=87 + +**# bpftool btf dump map id 123 kv** + +:: + + [39] TYPEDEF 'u32' type_id=37 + [86] PTR '(anon)' type_id=87 + +**# bpftool btf dump map id 123 all** + +:: + + [1] PTR '(anon)' type_id=0 + . + . + . + [2866] ARRAY '(anon)' type_id=52 index_type_id=51 nr_elems=4 + +All the standard ways to specify map or program are supported: + +**# bpftool btf dump map id 123** + +**# bpftool btf dump map pinned /sys/fs/bpf/map_name** + +**# bpftool btf dump prog id 456** + +**# bpftool btf dump prog tag b88e0a09b1d9759d** + +**# bpftool btf dump prog pinned /sys/fs/bpf/prog_name** + +SEE ALSO +======== + **bpf**\ (2), + **bpf-helpers**\ (7), + **bpftool**\ (8), + **bpftool-map**\ (8), + **bpftool-prog**\ (8), + **bpftool-cgroup**\ (8), + **bpftool-feature**\ (8), + **bpftool-net**\ (8), + **bpftool-perf**\ (8) diff --git a/tools/bpf/bpftool/Documentation/bpftool-cgroup.rst b/tools/bpf/bpftool/Documentation/bpftool-cgroup.rst index 9bb9ace54ba83b55694590a1eab7c4d23a0d7112..ac26876389c2383297185ddc1d3f3f1ac0067d8b 100644 --- a/tools/bpf/bpftool/Documentation/bpftool-cgroup.rst +++ b/tools/bpf/bpftool/Documentation/bpftool-cgroup.rst @@ -29,7 +29,7 @@ CGROUP COMMANDS | *PROG* := { **id** *PROG_ID* | **pinned** *FILE* | **tag** *PROG_TAG* } | *ATTACH_TYPE* := { **ingress** | **egress** | **sock_create** | **sock_ops** | **device** | | **bind4** | **bind6** | **post_bind4** | **post_bind6** | **connect4** | **connect6** | -| **sendmsg4** | **sendmsg6** } +| **sendmsg4** | **sendmsg6** | **sysctl** } | *ATTACH_FLAGS* := { **multi** | **override** } DESCRIPTION @@ -85,7 +85,8 @@ DESCRIPTION **sendmsg4** call to sendto(2), sendmsg(2), sendmmsg(2) for an unconnected udp4 socket (since 4.18); **sendmsg6** call to sendto(2), sendmsg(2), sendmmsg(2) for an - unconnected udp6 socket (since 4.18). + unconnected udp6 socket (since 4.18); + **sysctl** sysctl access (since 5.2). **bpftool cgroup detach** *CGROUP* *ATTACH_TYPE* *PROG* Detach *PROG* from the cgroup *CGROUP* and attach type @@ -99,7 +100,7 @@ OPTIONS -h, --help Print short generic help message (similar to **bpftool help**). - -v, --version + -V, --version Print version number (similar to **bpftool version**). -j, --json @@ -144,4 +145,5 @@ SEE ALSO **bpftool-map**\ (8), **bpftool-feature**\ (8), **bpftool-net**\ (8), - **bpftool-perf**\ (8) + **bpftool-perf**\ (8), + **bpftool-btf**\ (8) diff --git a/tools/bpf/bpftool/Documentation/bpftool-feature.rst b/tools/bpf/bpftool/Documentation/bpftool-feature.rst index 82de03dd8f5204440cce0c4936236d9521cae8c8..14180e8870824304c54abb9cd67c82919f158e90 100644 --- a/tools/bpf/bpftool/Documentation/bpftool-feature.rst +++ b/tools/bpf/bpftool/Documentation/bpftool-feature.rst @@ -63,7 +63,7 @@ OPTIONS -h, --help Print short generic help message (similar to **bpftool help**). - -v, --version + -V, --version Print version number (similar to **bpftool version**). -j, --json @@ -82,4 +82,5 @@ SEE ALSO **bpftool-map**\ (8), **bpftool-cgroup**\ (8), **bpftool-net**\ (8), - **bpftool-perf**\ (8) + **bpftool-perf**\ (8), + **bpftool-btf**\ (8) diff --git a/tools/bpf/bpftool/Documentation/bpftool-map.rst b/tools/bpf/bpftool/Documentation/bpftool-map.rst index 5c984ffc9f01e8bcece0499dde198eb51c321197..13ef27b39f200ff4239741201deb95d37067464c 100644 --- a/tools/bpf/bpftool/Documentation/bpftool-map.rst +++ b/tools/bpf/bpftool/Documentation/bpftool-map.rst @@ -135,7 +135,7 @@ OPTIONS -h, --help Print short generic help message (similar to **bpftool help**). - -v, --version + -V, --version Print version number (similar to **bpftool version**). -j, --json @@ -258,4 +258,5 @@ SEE ALSO **bpftool-cgroup**\ (8), **bpftool-feature**\ (8), **bpftool-net**\ (8), - **bpftool-perf**\ (8) + **bpftool-perf**\ (8), + **bpftool-btf**\ (8) diff --git a/tools/bpf/bpftool/Documentation/bpftool-net.rst b/tools/bpf/bpftool/Documentation/bpftool-net.rst index 779dab3650eeb20047d4a9114d2c8da8bf796dca..934580850f42b0f74a91b1323020db187652df9c 100644 --- a/tools/bpf/bpftool/Documentation/bpftool-net.rst +++ b/tools/bpf/bpftool/Documentation/bpftool-net.rst @@ -55,7 +55,7 @@ OPTIONS -h, --help Print short generic help message (similar to **bpftool help**). - -v, --version + -V, --version Print version number (similar to **bpftool version**). -j, --json @@ -143,4 +143,5 @@ SEE ALSO **bpftool-map**\ (8), **bpftool-cgroup**\ (8), **bpftool-feature**\ (8), - **bpftool-perf**\ (8) + **bpftool-perf**\ (8), + **bpftool-btf**\ (8) diff --git a/tools/bpf/bpftool/Documentation/bpftool-perf.rst b/tools/bpf/bpftool/Documentation/bpftool-perf.rst index bca5590a80d057dfb7b1c1c1b5d581536f88de27..0c7576523a217150c4239b7e903ba82fe53baaab 100644 --- a/tools/bpf/bpftool/Documentation/bpftool-perf.rst +++ b/tools/bpf/bpftool/Documentation/bpftool-perf.rst @@ -43,7 +43,7 @@ OPTIONS -h, --help Print short generic help message (similar to **bpftool help**). - -v, --version + -V, --version Print version number (similar to **bpftool version**). -j, --json @@ -85,4 +85,5 @@ SEE ALSO **bpftool-map**\ (8), **bpftool-cgroup**\ (8), **bpftool-feature**\ (8), - **bpftool-net**\ (8) + **bpftool-net**\ (8), + **bpftool-btf**\ (8) diff --git a/tools/bpf/bpftool/Documentation/bpftool-prog.rst b/tools/bpf/bpftool/Documentation/bpftool-prog.rst index 9386bd6e03969ea9fc2b0a01ac20dc8448954d20..e8118544d118282b63bfc27e515ea53ae9903d96 100644 --- a/tools/bpf/bpftool/Documentation/bpftool-prog.rst +++ b/tools/bpf/bpftool/Documentation/bpftool-prog.rst @@ -25,7 +25,7 @@ PROG COMMANDS | **bpftool** **prog dump xlated** *PROG* [{**file** *FILE* | **opcodes** | **visual** | **linum**}] | **bpftool** **prog dump jited** *PROG* [{**file** *FILE* | **opcodes** | **linum**}] | **bpftool** **prog pin** *PROG* *FILE* -| **bpftool** **prog { load | loadall }** *OBJ* *PATH* [**type** *TYPE*] [**map** {**idx** *IDX* | **name** *NAME*} *MAP*] [**dev** *NAME*] +| **bpftool** **prog { load | loadall }** *OBJ* *PATH* [**type** *TYPE*] [**map** {**idx** *IDX* | **name** *NAME*} *MAP*] [**dev** *NAME*] [**pinmaps** *MAP_DIR*] | **bpftool** **prog attach** *PROG* *ATTACH_TYPE* [*MAP*] | **bpftool** **prog detach** *PROG* *ATTACH_TYPE* [*MAP*] | **bpftool** **prog tracelog** @@ -39,7 +39,8 @@ PROG COMMANDS | **cgroup/sock** | **cgroup/dev** | **lwt_in** | **lwt_out** | **lwt_xmit** | | **lwt_seg6local** | **sockops** | **sk_skb** | **sk_msg** | **lirc_mode2** | | **cgroup/bind4** | **cgroup/bind6** | **cgroup/post_bind4** | **cgroup/post_bind6** | -| **cgroup/connect4** | **cgroup/connect6** | **cgroup/sendmsg4** | **cgroup/sendmsg6** +| **cgroup/connect4** | **cgroup/connect6** | **cgroup/sendmsg4** | **cgroup/sendmsg6** | +| **cgroup/sysctl** | } | *ATTACH_TYPE* := { | **msg_verdict** | **stream_verdict** | **stream_parser** | **flow_dissector** @@ -56,6 +57,14 @@ DESCRIPTION Output will start with program ID followed by program type and zero or more named attributes (depending on kernel version). + Since Linux 5.1 the kernel can collect statistics on BPF + programs (such as the total time spent running the program, + and the number of times it was run). If available, bpftool + shows such statistics. However, the kernel does not collect + them by defaults, as it slightly impacts performance on each + program run. Activation or deactivation of the feature is + performed via the **kernel.bpf_stats_enabled** sysctl knob. + **bpftool prog dump xlated** *PROG* [{ **file** *FILE* | **opcodes** | **visual** | **linum** }] Dump eBPF instructions of the program from the kernel. By default, eBPF will be disassembled and printed to standard @@ -144,7 +153,7 @@ OPTIONS -h, --help Print short generic help message (similar to **bpftool help**). - -v, --version + -V, --version Print version number (similar to **bpftool version**). -j, --json @@ -262,4 +271,5 @@ SEE ALSO **bpftool-cgroup**\ (8), **bpftool-feature**\ (8), **bpftool-net**\ (8), - **bpftool-perf**\ (8) + **bpftool-perf**\ (8), + **bpftool-btf**\ (8) diff --git a/tools/bpf/bpftool/Documentation/bpftool.rst b/tools/bpf/bpftool/Documentation/bpftool.rst index 4f2188845dd8338f435db3c7ac07e742e9b3ecbd..3e562d7fd56f942c657fdb7015de606190ea5a26 100644 --- a/tools/bpf/bpftool/Documentation/bpftool.rst +++ b/tools/bpf/bpftool/Documentation/bpftool.rst @@ -49,7 +49,7 @@ OPTIONS -h, --help Print short help message (similar to **bpftool help**). - -v, --version + -V, --version Print version number (similar to **bpftool version**). -j, --json @@ -76,4 +76,5 @@ SEE ALSO **bpftool-cgroup**\ (8), **bpftool-feature**\ (8), **bpftool-net**\ (8), - **bpftool-perf**\ (8) + **bpftool-perf**\ (8), + **bpftool-btf**\ (8) diff --git a/tools/bpf/bpftool/bash-completion/bpftool b/tools/bpf/bpftool/bash-completion/bpftool index b803827d01e8843941d40de02e819a3af8eee079..50e402a5a9c86276f2b37526f43c0ede5c1ba8a6 100644 --- a/tools/bpf/bpftool/bash-completion/bpftool +++ b/tools/bpf/bpftool/bash-completion/bpftool @@ -217,6 +217,7 @@ _bpftool() done cur=${words[cword]} prev=${words[cword - 1]} + pprev=${words[cword - 2]} local object=${words[1]} command=${words[2]} @@ -272,17 +273,17 @@ _bpftool() "$cur" ) ) return 0 ;; - *) - _bpftool_once_attr 'file' - if _bpftool_search_list 'xlated'; then - COMPREPLY+=( $( compgen -W 'opcodes visual linum' -- \ - "$cur" ) ) - else - COMPREPLY+=( $( compgen -W 'opcodes linum' -- \ - "$cur" ) ) - fi - return 0 - ;; + *) + _bpftool_once_attr 'file' + if _bpftool_search_list 'xlated'; then + COMPREPLY+=( $( compgen -W 'opcodes visual linum' -- \ + "$cur" ) ) + else + COMPREPLY+=( $( compgen -W 'opcodes linum' -- \ + "$cur" ) ) + fi + return 0 + ;; esac ;; pin) @@ -370,7 +371,8 @@ _bpftool() lirc_mode2 cgroup/bind4 cgroup/bind6 \ cgroup/connect4 cgroup/connect6 \ cgroup/sendmsg4 cgroup/sendmsg6 \ - cgroup/post_bind4 cgroup/post_bind6" -- \ + cgroup/post_bind4 cgroup/post_bind6 \ + cgroup/sysctl" -- \ "$cur" ) ) return 0 ;; @@ -606,6 +608,51 @@ _bpftool() ;; esac ;; + btf) + local PROG_TYPE='id pinned tag' + local MAP_TYPE='id pinned' + case $command in + dump) + case $prev in + $command) + COMPREPLY+=( $( compgen -W "id map prog file" -- \ + "$cur" ) ) + return 0 + ;; + prog) + COMPREPLY=( $( compgen -W "$PROG_TYPE" -- "$cur" ) ) + return 0 + ;; + map) + COMPREPLY=( $( compgen -W "$MAP_TYPE" -- "$cur" ) ) + return 0 + ;; + id) + case $pprev in + prog) + _bpftool_get_prog_ids + ;; + map) + _bpftool_get_map_ids + ;; + esac + return 0 + ;; + *) + if [[ $cword == 6 ]] && [[ ${words[3]} == "map" ]]; then + COMPREPLY+=( $( compgen -W 'key value kv all' -- \ + "$cur" ) ) + fi + return 0 + ;; + esac + ;; + *) + [[ $prev == $object ]] && \ + COMPREPLY=( $( compgen -W 'dump help' -- "$cur" ) ) + ;; + esac + ;; cgroup) case $command in show|list) @@ -619,7 +666,7 @@ _bpftool() attach|detach) local ATTACH_TYPES='ingress egress sock_create sock_ops \ device bind4 bind6 post_bind4 post_bind6 connect4 \ - connect6 sendmsg4 sendmsg6' + connect6 sendmsg4 sendmsg6 sysctl' local ATTACH_FLAGS='multi override' local PROG_TYPE='id pinned tag' case $prev in @@ -629,7 +676,7 @@ _bpftool() ;; ingress|egress|sock_create|sock_ops|device|bind4|bind6|\ post_bind4|post_bind6|connect4|connect6|sendmsg4|\ - sendmsg6) + sendmsg6|sysctl) COMPREPLY=( $( compgen -W "$PROG_TYPE" -- \ "$cur" ) ) return 0 diff --git a/tools/bpf/bpftool/btf.c b/tools/bpf/bpftool/btf.c new file mode 100644 index 0000000000000000000000000000000000000000..58a2cd002a4b18dc0ed87d64266a2fef13107ab2 --- /dev/null +++ b/tools/bpf/bpftool/btf.c @@ -0,0 +1,586 @@ +// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +/* Copyright (C) 2019 Facebook */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "btf.h" +#include "json_writer.h" +#include "main.h" + +static const char * const btf_kind_str[NR_BTF_KINDS] = { + [BTF_KIND_UNKN] = "UNKNOWN", + [BTF_KIND_INT] = "INT", + [BTF_KIND_PTR] = "PTR", + [BTF_KIND_ARRAY] = "ARRAY", + [BTF_KIND_STRUCT] = "STRUCT", + [BTF_KIND_UNION] = "UNION", + [BTF_KIND_ENUM] = "ENUM", + [BTF_KIND_FWD] = "FWD", + [BTF_KIND_TYPEDEF] = "TYPEDEF", + [BTF_KIND_VOLATILE] = "VOLATILE", + [BTF_KIND_CONST] = "CONST", + [BTF_KIND_RESTRICT] = "RESTRICT", + [BTF_KIND_FUNC] = "FUNC", + [BTF_KIND_FUNC_PROTO] = "FUNC_PROTO", + [BTF_KIND_VAR] = "VAR", + [BTF_KIND_DATASEC] = "DATASEC", +}; + +static const char *btf_int_enc_str(__u8 encoding) +{ + switch (encoding) { + case 0: + return "(none)"; + case BTF_INT_SIGNED: + return "SIGNED"; + case BTF_INT_CHAR: + return "CHAR"; + case BTF_INT_BOOL: + return "BOOL"; + default: + return "UNKN"; + } +} + +static const char *btf_var_linkage_str(__u32 linkage) +{ + switch (linkage) { + case BTF_VAR_STATIC: + return "static"; + case BTF_VAR_GLOBAL_ALLOCATED: + return "global-alloc"; + default: + return "(unknown)"; + } +} + +static const char *btf_str(const struct btf *btf, __u32 off) +{ + if (!off) + return "(anon)"; + return btf__name_by_offset(btf, off) ? : "(invalid)"; +} + +static int dump_btf_type(const struct btf *btf, __u32 id, + const struct btf_type *t) +{ + json_writer_t *w = json_wtr; + int kind, safe_kind; + + kind = BTF_INFO_KIND(t->info); + safe_kind = kind <= BTF_KIND_MAX ? kind : BTF_KIND_UNKN; + + if (json_output) { + jsonw_start_object(w); + jsonw_uint_field(w, "id", id); + jsonw_string_field(w, "kind", btf_kind_str[safe_kind]); + jsonw_string_field(w, "name", btf_str(btf, t->name_off)); + } else { + printf("[%u] %s '%s'", id, btf_kind_str[safe_kind], + btf_str(btf, t->name_off)); + } + + switch (BTF_INFO_KIND(t->info)) { + case BTF_KIND_INT: { + __u32 v = *(__u32 *)(t + 1); + const char *enc; + + enc = btf_int_enc_str(BTF_INT_ENCODING(v)); + + if (json_output) { + jsonw_uint_field(w, "size", t->size); + jsonw_uint_field(w, "bits_offset", BTF_INT_OFFSET(v)); + jsonw_uint_field(w, "nr_bits", BTF_INT_BITS(v)); + jsonw_string_field(w, "encoding", enc); + } else { + printf(" size=%u bits_offset=%u nr_bits=%u encoding=%s", + t->size, BTF_INT_OFFSET(v), BTF_INT_BITS(v), + enc); + } + break; + } + case BTF_KIND_PTR: + case BTF_KIND_CONST: + case BTF_KIND_VOLATILE: + case BTF_KIND_RESTRICT: + case BTF_KIND_TYPEDEF: + if (json_output) + jsonw_uint_field(w, "type_id", t->type); + else + printf(" type_id=%u", t->type); + break; + case BTF_KIND_ARRAY: { + const struct btf_array *arr = (const void *)(t + 1); + + if (json_output) { + jsonw_uint_field(w, "type_id", arr->type); + jsonw_uint_field(w, "index_type_id", arr->index_type); + jsonw_uint_field(w, "nr_elems", arr->nelems); + } else { + printf(" type_id=%u index_type_id=%u nr_elems=%u", + arr->type, arr->index_type, arr->nelems); + } + break; + } + case BTF_KIND_STRUCT: + case BTF_KIND_UNION: { + const struct btf_member *m = (const void *)(t + 1); + __u16 vlen = BTF_INFO_VLEN(t->info); + int i; + + if (json_output) { + jsonw_uint_field(w, "size", t->size); + jsonw_uint_field(w, "vlen", vlen); + jsonw_name(w, "members"); + jsonw_start_array(w); + } else { + printf(" size=%u vlen=%u", t->size, vlen); + } + for (i = 0; i < vlen; i++, m++) { + const char *name = btf_str(btf, m->name_off); + __u32 bit_off, bit_sz; + + if (BTF_INFO_KFLAG(t->info)) { + bit_off = BTF_MEMBER_BIT_OFFSET(m->offset); + bit_sz = BTF_MEMBER_BITFIELD_SIZE(m->offset); + } else { + bit_off = m->offset; + bit_sz = 0; + } + + if (json_output) { + jsonw_start_object(w); + jsonw_string_field(w, "name", name); + jsonw_uint_field(w, "type_id", m->type); + jsonw_uint_field(w, "bits_offset", bit_off); + if (bit_sz) { + jsonw_uint_field(w, "bitfield_size", + bit_sz); + } + jsonw_end_object(w); + } else { + printf("\n\t'%s' type_id=%u bits_offset=%u", + name, m->type, bit_off); + if (bit_sz) + printf(" bitfield_size=%u", bit_sz); + } + } + if (json_output) + jsonw_end_array(w); + break; + } + case BTF_KIND_ENUM: { + const struct btf_enum *v = (const void *)(t + 1); + __u16 vlen = BTF_INFO_VLEN(t->info); + int i; + + if (json_output) { + jsonw_uint_field(w, "size", t->size); + jsonw_uint_field(w, "vlen", vlen); + jsonw_name(w, "values"); + jsonw_start_array(w); + } else { + printf(" size=%u vlen=%u", t->size, vlen); + } + for (i = 0; i < vlen; i++, v++) { + const char *name = btf_str(btf, v->name_off); + + if (json_output) { + jsonw_start_object(w); + jsonw_string_field(w, "name", name); + jsonw_uint_field(w, "val", v->val); + jsonw_end_object(w); + } else { + printf("\n\t'%s' val=%u", name, v->val); + } + } + if (json_output) + jsonw_end_array(w); + break; + } + case BTF_KIND_FWD: { + const char *fwd_kind = BTF_INFO_KIND(t->info) ? "union" + : "struct"; + + if (json_output) + jsonw_string_field(w, "fwd_kind", fwd_kind); + else + printf(" fwd_kind=%s", fwd_kind); + break; + } + case BTF_KIND_FUNC: + if (json_output) + jsonw_uint_field(w, "type_id", t->type); + else + printf(" type_id=%u", t->type); + break; + case BTF_KIND_FUNC_PROTO: { + const struct btf_param *p = (const void *)(t + 1); + __u16 vlen = BTF_INFO_VLEN(t->info); + int i; + + if (json_output) { + jsonw_uint_field(w, "ret_type_id", t->type); + jsonw_uint_field(w, "vlen", vlen); + jsonw_name(w, "params"); + jsonw_start_array(w); + } else { + printf(" ret_type_id=%u vlen=%u", t->type, vlen); + } + for (i = 0; i < vlen; i++, p++) { + const char *name = btf_str(btf, p->name_off); + + if (json_output) { + jsonw_start_object(w); + jsonw_string_field(w, "name", name); + jsonw_uint_field(w, "type_id", p->type); + jsonw_end_object(w); + } else { + printf("\n\t'%s' type_id=%u", name, p->type); + } + } + if (json_output) + jsonw_end_array(w); + break; + } + case BTF_KIND_VAR: { + const struct btf_var *v = (const void *)(t + 1); + const char *linkage; + + linkage = btf_var_linkage_str(v->linkage); + + if (json_output) { + jsonw_uint_field(w, "type_id", t->type); + jsonw_string_field(w, "linkage", linkage); + } else { + printf(" type_id=%u, linkage=%s", t->type, linkage); + } + break; + } + case BTF_KIND_DATASEC: { + const struct btf_var_secinfo *v = (const void *)(t+1); + __u16 vlen = BTF_INFO_VLEN(t->info); + int i; + + if (json_output) { + jsonw_uint_field(w, "size", t->size); + jsonw_uint_field(w, "vlen", vlen); + jsonw_name(w, "vars"); + jsonw_start_array(w); + } else { + printf(" size=%u vlen=%u", t->size, vlen); + } + for (i = 0; i < vlen; i++, v++) { + if (json_output) { + jsonw_start_object(w); + jsonw_uint_field(w, "type_id", v->type); + jsonw_uint_field(w, "offset", v->offset); + jsonw_uint_field(w, "size", v->size); + jsonw_end_object(w); + } else { + printf("\n\ttype_id=%u offset=%u size=%u", + v->type, v->offset, v->size); + } + } + if (json_output) + jsonw_end_array(w); + break; + } + default: + break; + } + + if (json_output) + jsonw_end_object(json_wtr); + else + printf("\n"); + + return 0; +} + +static int dump_btf_raw(const struct btf *btf, + __u32 *root_type_ids, int root_type_cnt) +{ + const struct btf_type *t; + int i; + + if (json_output) { + jsonw_start_object(json_wtr); + jsonw_name(json_wtr, "types"); + jsonw_start_array(json_wtr); + } + + if (root_type_cnt) { + for (i = 0; i < root_type_cnt; i++) { + t = btf__type_by_id(btf, root_type_ids[i]); + dump_btf_type(btf, root_type_ids[i], t); + } + } else { + int cnt = btf__get_nr_types(btf); + + for (i = 1; i <= cnt; i++) { + t = btf__type_by_id(btf, i); + dump_btf_type(btf, i, t); + } + } + + if (json_output) { + jsonw_end_array(json_wtr); + jsonw_end_object(json_wtr); + } + return 0; +} + +static bool check_btf_endianness(GElf_Ehdr *ehdr) +{ + static unsigned int const endian = 1; + + switch (ehdr->e_ident[EI_DATA]) { + case ELFDATA2LSB: + return *(unsigned char const *)&endian == 1; + case ELFDATA2MSB: + return *(unsigned char const *)&endian == 0; + default: + return 0; + } +} + +static int btf_load_from_elf(const char *path, struct btf **btf) +{ + int err = -1, fd = -1, idx = 0; + Elf_Data *btf_data = NULL; + Elf_Scn *scn = NULL; + Elf *elf = NULL; + GElf_Ehdr ehdr; + + if (elf_version(EV_CURRENT) == EV_NONE) { + p_err("failed to init libelf for %s", path); + return -1; + } + + fd = open(path, O_RDONLY); + if (fd < 0) { + p_err("failed to open %s: %s", path, strerror(errno)); + return -1; + } + + elf = elf_begin(fd, ELF_C_READ, NULL); + if (!elf) { + p_err("failed to open %s as ELF file", path); + goto done; + } + if (!gelf_getehdr(elf, &ehdr)) { + p_err("failed to get EHDR from %s", path); + goto done; + } + if (!check_btf_endianness(&ehdr)) { + p_err("non-native ELF endianness is not supported"); + goto done; + } + if (!elf_rawdata(elf_getscn(elf, ehdr.e_shstrndx), NULL)) { + p_err("failed to get e_shstrndx from %s\n", path); + goto done; + } + + while ((scn = elf_nextscn(elf, scn)) != NULL) { + GElf_Shdr sh; + char *name; + + idx++; + if (gelf_getshdr(scn, &sh) != &sh) { + p_err("failed to get section(%d) header from %s", + idx, path); + goto done; + } + name = elf_strptr(elf, ehdr.e_shstrndx, sh.sh_name); + if (!name) { + p_err("failed to get section(%d) name from %s", + idx, path); + goto done; + } + if (strcmp(name, BTF_ELF_SEC) == 0) { + btf_data = elf_getdata(scn, 0); + if (!btf_data) { + p_err("failed to get section(%d, %s) data from %s", + idx, name, path); + goto done; + } + break; + } + } + + if (!btf_data) { + p_err("%s ELF section not found in %s", BTF_ELF_SEC, path); + goto done; + } + + *btf = btf__new(btf_data->d_buf, btf_data->d_size); + if (IS_ERR(*btf)) { + err = PTR_ERR(*btf); + *btf = NULL; + p_err("failed to load BTF data from %s: %s", + path, strerror(err)); + goto done; + } + + err = 0; +done: + if (err) { + if (*btf) { + btf__free(*btf); + *btf = NULL; + } + } + if (elf) + elf_end(elf); + close(fd); + return err; +} + +static int do_dump(int argc, char **argv) +{ + struct btf *btf = NULL; + __u32 root_type_ids[2]; + int root_type_cnt = 0; + __u32 btf_id = -1; + const char *src; + int fd = -1; + int err; + + if (!REQ_ARGS(2)) { + usage(); + return -1; + } + src = GET_ARG(); + + if (is_prefix(src, "map")) { + struct bpf_map_info info = {}; + __u32 len = sizeof(info); + + if (!REQ_ARGS(2)) { + usage(); + return -1; + } + + fd = map_parse_fd_and_info(&argc, &argv, &info, &len); + if (fd < 0) + return -1; + + btf_id = info.btf_id; + if (argc && is_prefix(*argv, "key")) { + root_type_ids[root_type_cnt++] = info.btf_key_type_id; + NEXT_ARG(); + } else if (argc && is_prefix(*argv, "value")) { + root_type_ids[root_type_cnt++] = info.btf_value_type_id; + NEXT_ARG(); + } else if (argc && is_prefix(*argv, "all")) { + NEXT_ARG(); + } else if (argc && is_prefix(*argv, "kv")) { + root_type_ids[root_type_cnt++] = info.btf_key_type_id; + root_type_ids[root_type_cnt++] = info.btf_value_type_id; + NEXT_ARG(); + } else { + root_type_ids[root_type_cnt++] = info.btf_key_type_id; + root_type_ids[root_type_cnt++] = info.btf_value_type_id; + } + } else if (is_prefix(src, "prog")) { + struct bpf_prog_info info = {}; + __u32 len = sizeof(info); + + if (!REQ_ARGS(2)) { + usage(); + return -1; + } + + fd = prog_parse_fd(&argc, &argv); + if (fd < 0) + return -1; + + err = bpf_obj_get_info_by_fd(fd, &info, &len); + if (err) { + p_err("can't get prog info: %s", strerror(errno)); + goto done; + } + + btf_id = info.btf_id; + } else if (is_prefix(src, "id")) { + char *endptr; + + btf_id = strtoul(*argv, &endptr, 0); + if (*endptr) { + p_err("can't parse %s as ID", **argv); + return -1; + } + NEXT_ARG(); + } else if (is_prefix(src, "file")) { + err = btf_load_from_elf(*argv, &btf); + if (err) + goto done; + NEXT_ARG(); + } else { + err = -1; + p_err("unrecognized BTF source specifier: '%s'", src); + goto done; + } + + if (!btf) { + err = btf__get_from_id(btf_id, &btf); + if (err) { + p_err("get btf by id (%u): %s", btf_id, strerror(err)); + goto done; + } + if (!btf) { + err = ENOENT; + p_err("can't find btf with ID (%u)", btf_id); + goto done; + } + } + + dump_btf_raw(btf, root_type_ids, root_type_cnt); + +done: + close(fd); + btf__free(btf); + return err; +} + +static int do_help(int argc, char **argv) +{ + if (json_output) { + jsonw_null(json_wtr); + return 0; + } + + fprintf(stderr, + "Usage: %s btf dump BTF_SRC\n" + " %s btf help\n" + "\n" + " BTF_SRC := { id BTF_ID | prog PROG | map MAP [{key | value | kv | all}] | file FILE }\n" + " " HELP_SPEC_MAP "\n" + " " HELP_SPEC_PROGRAM "\n" + " " HELP_SPEC_OPTIONS "\n" + "", + bin_name, bin_name); + + return 0; +} + +static const struct cmd cmds[] = { + { "help", do_help }, + { "dump", do_dump }, + { 0 } +}; + +int do_btf(int argc, char **argv) +{ + return cmd_select(cmds, argc, argv, do_help); +} diff --git a/tools/bpf/bpftool/btf_dumper.c b/tools/bpf/bpftool/btf_dumper.c index e63bce0755ebe605870af4b6b6e9322006f64a3d..8cafb9b314672fe238cb3d4e40b9dafcba1f2301 100644 --- a/tools/bpf/bpftool/btf_dumper.c +++ b/tools/bpf/bpftool/btf_dumper.c @@ -309,6 +309,48 @@ static int btf_dumper_struct(const struct btf_dumper *d, __u32 type_id, return ret; } +static int btf_dumper_var(const struct btf_dumper *d, __u32 type_id, + __u8 bit_offset, const void *data) +{ + const struct btf_type *t = btf__type_by_id(d->btf, type_id); + int ret; + + jsonw_start_object(d->jw); + jsonw_name(d->jw, btf__name_by_offset(d->btf, t->name_off)); + ret = btf_dumper_do_type(d, t->type, bit_offset, data); + jsonw_end_object(d->jw); + + return ret; +} + +static int btf_dumper_datasec(const struct btf_dumper *d, __u32 type_id, + const void *data) +{ + struct btf_var_secinfo *vsi; + const struct btf_type *t; + int ret = 0, i, vlen; + + t = btf__type_by_id(d->btf, type_id); + if (!t) + return -EINVAL; + + vlen = BTF_INFO_VLEN(t->info); + vsi = (struct btf_var_secinfo *)(t + 1); + + jsonw_start_object(d->jw); + jsonw_name(d->jw, btf__name_by_offset(d->btf, t->name_off)); + jsonw_start_array(d->jw); + for (i = 0; i < vlen; i++) { + ret = btf_dumper_do_type(d, vsi[i].type, 0, data + vsi[i].offset); + if (ret) + break; + } + jsonw_end_array(d->jw); + jsonw_end_object(d->jw); + + return ret; +} + static int btf_dumper_do_type(const struct btf_dumper *d, __u32 type_id, __u8 bit_offset, const void *data) { @@ -341,6 +383,10 @@ static int btf_dumper_do_type(const struct btf_dumper *d, __u32 type_id, case BTF_KIND_CONST: case BTF_KIND_RESTRICT: return btf_dumper_modifier(d, type_id, bit_offset, data); + case BTF_KIND_VAR: + return btf_dumper_var(d, type_id, bit_offset, data); + case BTF_KIND_DATASEC: + return btf_dumper_datasec(d, type_id, data); default: jsonw_printf(d->jw, "(unsupported-kind"); return -EINVAL; @@ -377,6 +423,7 @@ static int __btf_dumper_type_only(const struct btf *btf, __u32 type_id, { const struct btf_type *proto_type; const struct btf_array *array; + const struct btf_var *var; const struct btf_type *t; if (!type_id) { @@ -440,6 +487,18 @@ static int __btf_dumper_type_only(const struct btf *btf, __u32 type_id, if (pos == -1) return -1; break; + case BTF_KIND_VAR: + var = (struct btf_var *)(t + 1); + if (var->linkage == BTF_VAR_STATIC) + BTF_PRINT_ARG("static "); + BTF_PRINT_TYPE(t->type); + BTF_PRINT_ARG(" %s", + btf__name_by_offset(btf, t->name_off)); + break; + case BTF_KIND_DATASEC: + BTF_PRINT_ARG("section (\"%s\") ", + btf__name_by_offset(btf, t->name_off)); + break; case BTF_KIND_UNKN: default: return -1; diff --git a/tools/bpf/bpftool/cgroup.c b/tools/bpf/bpftool/cgroup.c index 4b5c8da2a7c0943d975e1bed242b9be31d29d1ab..7e22f115c8c1af352984961a3a6c0dc336d4b864 100644 --- a/tools/bpf/bpftool/cgroup.c +++ b/tools/bpf/bpftool/cgroup.c @@ -25,7 +25,7 @@ " ATTACH_TYPE := { ingress | egress | sock_create |\n" \ " sock_ops | device | bind4 | bind6 |\n" \ " post_bind4 | post_bind6 | connect4 |\n" \ - " connect6 | sendmsg4 | sendmsg6 }" + " connect6 | sendmsg4 | sendmsg6 | sysctl }" static const char * const attach_type_strings[] = { [BPF_CGROUP_INET_INGRESS] = "ingress", @@ -41,6 +41,7 @@ static const char * const attach_type_strings[] = { [BPF_CGROUP_INET6_POST_BIND] = "post_bind6", [BPF_CGROUP_UDP4_SENDMSG] = "sendmsg4", [BPF_CGROUP_UDP6_SENDMSG] = "sendmsg6", + [BPF_CGROUP_SYSCTL] = "sysctl", [__MAX_BPF_ATTACH_TYPE] = NULL, }; @@ -248,6 +249,13 @@ static int do_show_tree_fn(const char *fpath, const struct stat *sb, for (type = 0; type < __MAX_BPF_ATTACH_TYPE; type++) show_attached_bpf_progs(cgroup_fd, type, ftw->level); + if (errno == EINVAL) + /* Last attach type does not support query. + * Do not report an error for this, especially because batch + * mode would stop processing commands. + */ + errno = 0; + if (json_output) { jsonw_end_array(json_wtr); jsonw_end_object(json_wtr); diff --git a/tools/bpf/bpftool/main.c b/tools/bpf/bpftool/main.c index a9d5e9e6a7327d9a8af33825e8daa93f4c06fc4e..1ac1fc520e6ab6a7f789b1b658351f3840d30e8b 100644 --- a/tools/bpf/bpftool/main.c +++ b/tools/bpf/bpftool/main.c @@ -56,7 +56,7 @@ static int do_help(int argc, char **argv) " %s batch file FILE\n" " %s version\n" "\n" - " OBJECT := { prog | map | cgroup | perf | net | feature }\n" + " OBJECT := { prog | map | cgroup | perf | net | feature | btf }\n" " " HELP_SPEC_OPTIONS "\n" "", bin_name, bin_name, bin_name); @@ -188,6 +188,7 @@ static const struct cmd cmds[] = { { "perf", do_perf }, { "net", do_net }, { "feature", do_feature }, + { "btf", do_btf }, { "version", do_version }, { 0 } }; diff --git a/tools/bpf/bpftool/main.h b/tools/bpf/bpftool/main.h index d7dd84d3c660eb62b60f2d762fb0e74a5524ff42..3d63feb7f852fb8cb9a23a5b6b51b25d860ffceb 100644 --- a/tools/bpf/bpftool/main.h +++ b/tools/bpf/bpftool/main.h @@ -73,6 +73,7 @@ static const char * const prog_type_name[] = { [BPF_PROG_TYPE_LIRC_MODE2] = "lirc_mode2", [BPF_PROG_TYPE_SK_REUSEPORT] = "sk_reuseport", [BPF_PROG_TYPE_FLOW_DISSECTOR] = "flow_dissector", + [BPF_PROG_TYPE_CGROUP_SYSCTL] = "cgroup_sysctl", }; extern const char * const map_type_name[]; @@ -149,6 +150,7 @@ int do_perf(int argc, char **arg); int do_net(int argc, char **arg); int do_tracelog(int argc, char **arg); int do_feature(int argc, char **argv); +int do_btf(int argc, char **argv); int parse_u32_arg(int *argc, char ***argv, __u32 *val, const char *what); int prog_parse_fd(int *argc, char ***argv); diff --git a/tools/bpf/bpftool/map.c b/tools/bpf/bpftool/map.c index 994a7e0d16fb54a0dbd68cfaaf350b54927418b5..3ec82904ccec6aa974353d20f96875267a20ffee 100644 --- a/tools/bpf/bpftool/map.c +++ b/tools/bpf/bpftool/map.c @@ -46,6 +46,7 @@ const char * const map_type_name[] = { [BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE] = "percpu_cgroup_storage", [BPF_MAP_TYPE_QUEUE] = "queue", [BPF_MAP_TYPE_STACK] = "stack", + [BPF_MAP_TYPE_SK_STORAGE] = "sk_storage", }; const size_t map_type_name_size = ARRAY_SIZE(map_type_name); @@ -153,11 +154,13 @@ static int do_dump_btf(const struct btf_dumper *d, /* start of key-value pair */ jsonw_start_object(d->jw); - jsonw_name(d->jw, "key"); + if (map_info->btf_key_type_id) { + jsonw_name(d->jw, "key"); - ret = btf_dumper_type(d, map_info->btf_key_type_id, key); - if (ret) - goto err_end_obj; + ret = btf_dumper_type(d, map_info->btf_key_type_id, key); + if (ret) + goto err_end_obj; + } if (!map_is_per_cpu(map_info->type)) { jsonw_name(d->jw, "value"); @@ -259,20 +262,20 @@ static void print_entry_json(struct bpf_map_info *info, unsigned char *key, } static void print_entry_error(struct bpf_map_info *info, unsigned char *key, - const char *value) + const char *error_msg) { - int value_size = strlen(value); + int msg_size = strlen(error_msg); bool single_line, break_names; - break_names = info->key_size > 16 || value_size > 16; - single_line = info->key_size + value_size <= 24 && !break_names; + break_names = info->key_size > 16 || msg_size > 16; + single_line = info->key_size + msg_size <= 24 && !break_names; printf("key:%c", break_names ? '\n' : ' '); fprint_hex(stdout, key, info->key_size, " "); printf(single_line ? " " : "\n"); - printf("value:%c%s", break_names ? '\n' : ' ', value); + printf("value:%c%s", break_names ? '\n' : ' ', error_msg); printf("\n"); } @@ -296,11 +299,7 @@ static void print_entry_plain(struct bpf_map_info *info, unsigned char *key, if (info->value_size) { printf("value:%c", break_names ? '\n' : ' '); - if (value) - fprint_hex(stdout, value, info->value_size, - " "); - else - printf(""); + fprint_hex(stdout, value, info->value_size, " "); } printf("\n"); @@ -319,11 +318,8 @@ static void print_entry_plain(struct bpf_map_info *info, unsigned char *key, for (i = 0; i < n; i++) { printf("value (CPU %02d):%c", i, info->value_size > 16 ? '\n' : ' '); - if (value) - fprint_hex(stdout, value + i * step, - info->value_size, " "); - else - printf(""); + fprint_hex(stdout, value + i * step, + info->value_size, " "); printf("\n"); } } @@ -536,6 +532,9 @@ static int show_map_close_json(int fd, struct bpf_map_info *info) } close(fd); + if (info->btf_id) + jsonw_int_field(json_wtr, "btf_id", info->btf_id); + if (!hash_empty(map_table.table)) { struct pinned_obj *obj; @@ -602,15 +601,19 @@ static int show_map_close_plain(int fd, struct bpf_map_info *info) } close(fd); - printf("\n"); if (!hash_empty(map_table.table)) { struct pinned_obj *obj; hash_for_each_possible(map_table.table, obj, hash, info->id) { if (obj->id == info->id) - printf("\tpinned %s\n", obj->path); + printf("\n\tpinned %s", obj->path); } } + + if (info->btf_id) + printf("\n\tbtf_id %d", info->btf_id); + + printf("\n"); return 0; } @@ -720,11 +723,16 @@ static int dump_map_elem(int fd, void *key, void *value, jsonw_string_field(json_wtr, "error", strerror(lookup_errno)); jsonw_end_object(json_wtr); } else { - if (errno == ENOENT) - print_entry_plain(map_info, key, NULL); - else - print_entry_error(map_info, key, - strerror(lookup_errno)); + const char *msg = NULL; + + if (lookup_errno == ENOENT) + msg = ""; + else if (lookup_errno == ENOSPC && + map_info->type == BPF_MAP_TYPE_REUSEPORT_SOCKARRAY) + msg = ""; + + print_entry_error(map_info, key, + msg ? : strerror(lookup_errno)); } return 0; @@ -778,6 +786,10 @@ static int do_dump(int argc, char **argv) } } + if (info.type == BPF_MAP_TYPE_REUSEPORT_SOCKARRAY && + info.value_size != 8) + p_info("Warning: cannot read values from %s map with value_size != 8", + map_type_name[info.type]); while (true) { err = bpf_map_get_next_key(fd, prev_key, key); if (err) { diff --git a/tools/bpf/bpftool/net.c b/tools/bpf/bpftool/net.c index db0e7de49d49e2f4bac139f87ee9811bfd71be4e..67e99c56bc88c0e4684755f97a562020c0731bf0 100644 --- a/tools/bpf/bpftool/net.c +++ b/tools/bpf/bpftool/net.c @@ -3,6 +3,7 @@ #define _GNU_SOURCE #include +#include #include #include #include @@ -12,6 +13,8 @@ #include #include #include +#include +#include #include #include @@ -48,6 +51,10 @@ struct bpf_filter_t { int ifindex; }; +struct bpf_attach_info { + __u32 flow_dissector_id; +}; + static int dump_link_nlmsg(void *cookie, void *msg, struct nlattr **tb) { struct bpf_netdev_t *netinfo = cookie; @@ -180,8 +187,45 @@ out: return 0; } +static int query_flow_dissector(struct bpf_attach_info *attach_info) +{ + __u32 attach_flags; + __u32 prog_ids[1]; + __u32 prog_cnt; + int err; + int fd; + + fd = open("/proc/self/ns/net", O_RDONLY); + if (fd < 0) { + p_err("can't open /proc/self/ns/net: %d", + strerror(errno)); + return -1; + } + prog_cnt = ARRAY_SIZE(prog_ids); + err = bpf_prog_query(fd, BPF_FLOW_DISSECTOR, 0, + &attach_flags, prog_ids, &prog_cnt); + close(fd); + if (err) { + if (errno == EINVAL) { + /* Older kernel's don't support querying + * flow dissector programs. + */ + errno = 0; + return 0; + } + p_err("can't query prog: %s", strerror(errno)); + return -1; + } + + if (prog_cnt == 1) + attach_info->flow_dissector_id = prog_ids[0]; + + return 0; +} + static int do_show(int argc, char **argv) { + struct bpf_attach_info attach_info = {}; int i, sock, ret, filter_idx = -1; struct bpf_netdev_t dev_array; unsigned int nl_pid; @@ -199,6 +243,10 @@ static int do_show(int argc, char **argv) usage(); } + ret = query_flow_dissector(&attach_info); + if (ret) + return -1; + sock = libbpf_netlink_open(&nl_pid); if (sock < 0) { fprintf(stderr, "failed to open netlink sock\n"); @@ -227,6 +275,12 @@ static int do_show(int argc, char **argv) } NET_END_ARRAY("\n"); } + + NET_START_ARRAY("flow_dissector", "%s:\n"); + if (attach_info.flow_dissector_id > 0) + NET_DUMP_UINT("id", "id %u", attach_info.flow_dissector_id); + NET_END_ARRAY("\n"); + NET_END_OBJECT; if (json_output) jsonw_end_array(json_wtr); diff --git a/tools/bpf/bpftool/prog.c b/tools/bpf/bpftool/prog.c index d2be5a06c339155f355058946f655f38f97b5db9..fc495b27f0fcb98cd852f08c7cc6523bd54a9925 100644 --- a/tools/bpf/bpftool/prog.c +++ b/tools/bpf/bpftool/prog.c @@ -249,6 +249,9 @@ static void print_prog_json(struct bpf_prog_info *info, int fd) if (info->nr_map_ids) show_prog_maps(fd, info->nr_map_ids); + if (info->btf_id) + jsonw_int_field(json_wtr, "btf_id", info->btf_id); + if (!hash_empty(prog_table.table)) { struct pinned_obj *obj; @@ -319,6 +322,9 @@ static void print_prog_plain(struct bpf_prog_info *info, int fd) } } + if (info->btf_id) + printf("\n\tbtf_id %d", info->btf_id); + printf("\n"); } @@ -1054,7 +1060,7 @@ static int do_help(int argc, char **argv) " tracepoint | raw_tracepoint | xdp | perf_event | cgroup/skb |\n" " cgroup/sock | cgroup/dev | lwt_in | lwt_out | lwt_xmit |\n" " lwt_seg6local | sockops | sk_skb | sk_msg | lirc_mode2 |\n" - " sk_reuseport | flow_dissector |\n" + " sk_reuseport | flow_dissector | cgroup/sysctl |\n" " cgroup/bind4 | cgroup/bind6 | cgroup/post_bind4 |\n" " cgroup/post_bind6 | cgroup/connect4 | cgroup/connect6 |\n" " cgroup/sendmsg4 | cgroup/sendmsg6 }\n" diff --git a/tools/bpf/bpftool/xlated_dumper.c b/tools/bpf/bpftool/xlated_dumper.c index 7073dbe1ff27eee87201ead853228d7be1b12028..0bb17bf88b188378cd863e733683a0e9da131c77 100644 --- a/tools/bpf/bpftool/xlated_dumper.c +++ b/tools/bpf/bpftool/xlated_dumper.c @@ -195,6 +195,9 @@ static const char *print_imm(void *private_data, if (insn->src_reg == BPF_PSEUDO_MAP_FD) snprintf(dd->scratch_buff, sizeof(dd->scratch_buff), "map[id:%u]", insn->imm); + else if (insn->src_reg == BPF_PSEUDO_MAP_VALUE) + snprintf(dd->scratch_buff, sizeof(dd->scratch_buff), + "map[id:%u][0]+%u", insn->imm, (insn + 1)->imm); else snprintf(dd->scratch_buff, sizeof(dd->scratch_buff), "0x%llx", (unsigned long long)full_imm); diff --git a/tools/include/linux/filter.h b/tools/include/linux/filter.h index cce0b02c0e286030171c261df2af713ab15ce75f..ca28b6ab8db7c7d315d2e128331d7223c2113825 100644 --- a/tools/include/linux/filter.h +++ b/tools/include/linux/filter.h @@ -278,10 +278,29 @@ .off = 0, \ .imm = ((__u64) (IMM)) >> 32 }) +#define BPF_LD_IMM64_RAW_FULL(DST, SRC, OFF1, OFF2, IMM1, IMM2) \ + ((struct bpf_insn) { \ + .code = BPF_LD | BPF_DW | BPF_IMM, \ + .dst_reg = DST, \ + .src_reg = SRC, \ + .off = OFF1, \ + .imm = IMM1 }), \ + ((struct bpf_insn) { \ + .code = 0, /* zero is reserved opcode */ \ + .dst_reg = 0, \ + .src_reg = 0, \ + .off = OFF2, \ + .imm = IMM2 }) + /* pseudo BPF_LD_IMM64 insn used to refer to process-local map_fd */ #define BPF_LD_MAP_FD(DST, MAP_FD) \ - BPF_LD_IMM64_RAW(DST, BPF_PSEUDO_MAP_FD, MAP_FD) + BPF_LD_IMM64_RAW_FULL(DST, BPF_PSEUDO_MAP_FD, 0, 0, \ + MAP_FD, 0) + +#define BPF_LD_MAP_VALUE(DST, MAP_FD, VALUE_OFF) \ + BPF_LD_IMM64_RAW_FULL(DST, BPF_PSEUDO_MAP_VALUE, 0, 0, \ + MAP_FD, VALUE_OFF) /* Relative call */ diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 929c8e537a14a517c0a3c7ca5b6b15353d622c30..72336bac7573368e7bce974626b68569c46eaabb 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -105,6 +105,7 @@ enum bpf_cmd { BPF_BTF_GET_FD_BY_ID, BPF_TASK_FD_QUERY, BPF_MAP_LOOKUP_AND_DELETE_ELEM, + BPF_MAP_FREEZE, }; enum bpf_map_type { @@ -132,6 +133,7 @@ enum bpf_map_type { BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE, BPF_MAP_TYPE_QUEUE, BPF_MAP_TYPE_STACK, + BPF_MAP_TYPE_SK_STORAGE, }; /* Note that tracing related programs such as @@ -166,6 +168,8 @@ enum bpf_prog_type { BPF_PROG_TYPE_LIRC_MODE2, BPF_PROG_TYPE_SK_REUSEPORT, BPF_PROG_TYPE_FLOW_DISSECTOR, + BPF_PROG_TYPE_CGROUP_SYSCTL, + BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE, }; enum bpf_attach_type { @@ -187,6 +191,7 @@ enum bpf_attach_type { BPF_CGROUP_UDP6_SENDMSG, BPF_LIRC_MODE2, BPF_FLOW_DISSECTOR, + BPF_CGROUP_SYSCTL, __MAX_BPF_ATTACH_TYPE }; @@ -255,8 +260,19 @@ enum bpf_attach_type { */ #define BPF_F_ANY_ALIGNMENT (1U << 1) -/* when bpf_ldimm64->src_reg == BPF_PSEUDO_MAP_FD, bpf_ldimm64->imm == fd */ +/* When BPF ldimm64's insn[0].src_reg != 0 then this can have + * two extensions: + * + * insn[0].src_reg: BPF_PSEUDO_MAP_FD BPF_PSEUDO_MAP_VALUE + * insn[0].imm: map fd map fd + * insn[1].imm: 0 offset into value + * insn[0].off: 0 0 + * insn[1].off: 0 0 + * ldimm64 rewrite: address of map address of map[0]+offset + * verifier type: CONST_PTR_TO_MAP PTR_TO_MAP_VALUE + */ #define BPF_PSEUDO_MAP_FD 1 +#define BPF_PSEUDO_MAP_VALUE 2 /* when bpf_call->src_reg == BPF_PSEUDO_CALL, bpf_call->imm == pc-relative * offset to another bpf function @@ -283,7 +299,7 @@ enum bpf_attach_type { #define BPF_OBJ_NAME_LEN 16U -/* Flags for accessing BPF object */ +/* Flags for accessing BPF object from syscall side. */ #define BPF_F_RDONLY (1U << 3) #define BPF_F_WRONLY (1U << 4) @@ -293,6 +309,10 @@ enum bpf_attach_type { /* Zero-initialize hash function seed. This should only be used for testing. */ #define BPF_F_ZERO_SEED (1U << 6) +/* Flags for accessing BPF object from program side. */ +#define BPF_F_RDONLY_PROG (1U << 7) +#define BPF_F_WRONLY_PROG (1U << 8) + /* flags for BPF_PROG_QUERY */ #define BPF_F_QUERY_EFFECTIVE (1U << 0) @@ -396,6 +416,13 @@ union bpf_attr { __aligned_u64 data_out; __u32 repeat; __u32 duration; + __u32 ctx_size_in; /* input: len of ctx_in */ + __u32 ctx_size_out; /* input/output: len of ctx_out + * returns ENOSPC if ctx_out + * is too small. + */ + __aligned_u64 ctx_in; + __aligned_u64 ctx_out; } test; struct { /* anonymous struct used by BPF_*_GET_*_ID */ @@ -1478,13 +1505,31 @@ union bpf_attr { * Grow or shrink the room for data in the packet associated to * *skb* by *len_diff*, and according to the selected *mode*. * - * There is a single supported mode at this time: + * There are two supported modes at this time: + * + * * **BPF_ADJ_ROOM_MAC**: Adjust room at the mac layer + * (room space is added or removed below the layer 2 header). * * * **BPF_ADJ_ROOM_NET**: Adjust room at the network layer * (room space is added or removed below the layer 3 header). * - * All values for *flags* are reserved for future usage, and must - * be left at zero. + * The following flags are supported at this time: + * + * * **BPF_F_ADJ_ROOM_FIXED_GSO**: Do not adjust gso_size. + * Adjusting mss in this way is not allowed for datagrams. + * + * * **BPF_F_ADJ_ROOM_ENCAP_L3_IPV4 **: + * * **BPF_F_ADJ_ROOM_ENCAP_L3_IPV6 **: + * Any new space is reserved to hold a tunnel header. + * Configure skb offsets and other fields accordingly. + * + * * **BPF_F_ADJ_ROOM_ENCAP_L4_GRE **: + * * **BPF_F_ADJ_ROOM_ENCAP_L4_UDP **: + * Use with ENCAP_L3 flags to further specify the tunnel type. + * + * * **BPF_F_ADJ_ROOM_ENCAP_L2(len) **: + * Use with ENCAP_L3/L4 flags to further specify the tunnel + * type; **len** is the length of the inner MAC header. * * A call to this helper is susceptible to change the underlaying * packet buffer. Therefore, at load time, all checks on pointers @@ -1694,12 +1739,19 @@ union bpf_attr { * error if an eBPF program tries to set a callback that is not * supported in the current kernel. * - * The supported callback values that *argval* can combine are: + * *argval* is a flag array which can combine these flags: * * * **BPF_SOCK_OPS_RTO_CB_FLAG** (retransmission time out) * * **BPF_SOCK_OPS_RETRANS_CB_FLAG** (retransmission) * * **BPF_SOCK_OPS_STATE_CB_FLAG** (TCP state change) * + * Therefore, this function can be used to clear a callback flag by + * setting the appropriate bit to zero. e.g. to disable the RTO + * callback: + * + * **bpf_sock_ops_cb_flags_set(bpf_sock,** + * **bpf_sock->bpf_sock_ops_cb_flags & ~BPF_SOCK_OPS_RTO_CB_FLAG)** + * * Here are some examples of where one could call such eBPF * program: * @@ -2431,6 +2483,190 @@ union bpf_attr { * Return * A **struct bpf_sock** pointer on success, or **NULL** in * case of failure. + * + * struct bpf_sock *bpf_skc_lookup_tcp(void *ctx, struct bpf_sock_tuple *tuple, u32 tuple_size, u64 netns, u64 flags) + * Description + * Look for TCP socket matching *tuple*, optionally in a child + * network namespace *netns*. The return value must be checked, + * and if non-**NULL**, released via **bpf_sk_release**\ (). + * + * This function is identical to bpf_sk_lookup_tcp, except that it + * also returns timewait or request sockets. Use bpf_sk_fullsock + * or bpf_tcp_socket to access the full structure. + * + * This helper is available only if the kernel was compiled with + * **CONFIG_NET** configuration option. + * Return + * Pointer to **struct bpf_sock**, or **NULL** in case of failure. + * For sockets with reuseport option, the **struct bpf_sock** + * result is from **reuse->socks**\ [] using the hash of the tuple. + * + * int bpf_tcp_check_syncookie(struct bpf_sock *sk, void *iph, u32 iph_len, struct tcphdr *th, u32 th_len) + * Description + * Check whether iph and th contain a valid SYN cookie ACK for + * the listening socket in sk. + * + * iph points to the start of the IPv4 or IPv6 header, while + * iph_len contains sizeof(struct iphdr) or sizeof(struct ip6hdr). + * + * th points to the start of the TCP header, while th_len contains + * sizeof(struct tcphdr). + * + * Return + * 0 if iph and th are a valid SYN cookie ACK, or a negative error + * otherwise. + * + * int bpf_sysctl_get_name(struct bpf_sysctl *ctx, char *buf, size_t buf_len, u64 flags) + * Description + * Get name of sysctl in /proc/sys/ and copy it into provided by + * program buffer *buf* of size *buf_len*. + * + * The buffer is always NUL terminated, unless it's zero-sized. + * + * If *flags* is zero, full name (e.g. "net/ipv4/tcp_mem") is + * copied. Use **BPF_F_SYSCTL_BASE_NAME** flag to copy base name + * only (e.g. "tcp_mem"). + * Return + * Number of character copied (not including the trailing NUL). + * + * **-E2BIG** if the buffer wasn't big enough (*buf* will contain + * truncated name in this case). + * + * int bpf_sysctl_get_current_value(struct bpf_sysctl *ctx, char *buf, size_t buf_len) + * Description + * Get current value of sysctl as it is presented in /proc/sys + * (incl. newline, etc), and copy it as a string into provided + * by program buffer *buf* of size *buf_len*. + * + * The whole value is copied, no matter what file position user + * space issued e.g. sys_read at. + * + * The buffer is always NUL terminated, unless it's zero-sized. + * Return + * Number of character copied (not including the trailing NUL). + * + * **-E2BIG** if the buffer wasn't big enough (*buf* will contain + * truncated name in this case). + * + * **-EINVAL** if current value was unavailable, e.g. because + * sysctl is uninitialized and read returns -EIO for it. + * + * int bpf_sysctl_get_new_value(struct bpf_sysctl *ctx, char *buf, size_t buf_len) + * Description + * Get new value being written by user space to sysctl (before + * the actual write happens) and copy it as a string into + * provided by program buffer *buf* of size *buf_len*. + * + * User space may write new value at file position > 0. + * + * The buffer is always NUL terminated, unless it's zero-sized. + * Return + * Number of character copied (not including the trailing NUL). + * + * **-E2BIG** if the buffer wasn't big enough (*buf* will contain + * truncated name in this case). + * + * **-EINVAL** if sysctl is being read. + * + * int bpf_sysctl_set_new_value(struct bpf_sysctl *ctx, const char *buf, size_t buf_len) + * Description + * Override new value being written by user space to sysctl with + * value provided by program in buffer *buf* of size *buf_len*. + * + * *buf* should contain a string in same form as provided by user + * space on sysctl write. + * + * User space may write new value at file position > 0. To override + * the whole sysctl value file position should be set to zero. + * Return + * 0 on success. + * + * **-E2BIG** if the *buf_len* is too big. + * + * **-EINVAL** if sysctl is being read. + * + * int bpf_strtol(const char *buf, size_t buf_len, u64 flags, long *res) + * Description + * Convert the initial part of the string from buffer *buf* of + * size *buf_len* to a long integer according to the given base + * and save the result in *res*. + * + * The string may begin with an arbitrary amount of white space + * (as determined by isspace(3)) followed by a single optional '-' + * sign. + * + * Five least significant bits of *flags* encode base, other bits + * are currently unused. + * + * Base must be either 8, 10, 16 or 0 to detect it automatically + * similar to user space strtol(3). + * Return + * Number of characters consumed on success. Must be positive but + * no more than buf_len. + * + * **-EINVAL** if no valid digits were found or unsupported base + * was provided. + * + * **-ERANGE** if resulting value was out of range. + * + * int bpf_strtoul(const char *buf, size_t buf_len, u64 flags, unsigned long *res) + * Description + * Convert the initial part of the string from buffer *buf* of + * size *buf_len* to an unsigned long integer according to the + * given base and save the result in *res*. + * + * The string may begin with an arbitrary amount of white space + * (as determined by isspace(3)). + * + * Five least significant bits of *flags* encode base, other bits + * are currently unused. + * + * Base must be either 8, 10, 16 or 0 to detect it automatically + * similar to user space strtoul(3). + * Return + * Number of characters consumed on success. Must be positive but + * no more than buf_len. + * + * **-EINVAL** if no valid digits were found or unsupported base + * was provided. + * + * **-ERANGE** if resulting value was out of range. + * + * void *bpf_sk_storage_get(struct bpf_map *map, struct bpf_sock *sk, void *value, u64 flags) + * Description + * Get a bpf-local-storage from a sk. + * + * Logically, it could be thought of getting the value from + * a *map* with *sk* as the **key**. From this + * perspective, the usage is not much different from + * **bpf_map_lookup_elem(map, &sk)** except this + * helper enforces the key must be a **bpf_fullsock()** + * and the map must be a BPF_MAP_TYPE_SK_STORAGE also. + * + * Underneath, the value is stored locally at *sk* instead of + * the map. The *map* is used as the bpf-local-storage **type**. + * The bpf-local-storage **type** (i.e. the *map*) is searched + * against all bpf-local-storages residing at sk. + * + * An optional *flags* (BPF_SK_STORAGE_GET_F_CREATE) can be + * used such that a new bpf-local-storage will be + * created if one does not exist. *value* can be used + * together with BPF_SK_STORAGE_GET_F_CREATE to specify + * the initial value of a bpf-local-storage. If *value* is + * NULL, the new bpf-local-storage will be zero initialized. + * Return + * A bpf-local-storage pointer is returned on success. + * + * **NULL** if not found or there was an error in adding + * a new bpf-local-storage. + * + * int bpf_sk_storage_delete(struct bpf_map *map, struct bpf_sock *sk) + * Description + * Delete a bpf-local-storage from a sk. + * Return + * 0 on success. + * + * **-ENOENT** if the bpf-local-storage cannot be found. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -2531,7 +2767,17 @@ union bpf_attr { FN(sk_fullsock), \ FN(tcp_sock), \ FN(skb_ecn_set_ce), \ - FN(get_listener_sock), + FN(get_listener_sock), \ + FN(skc_lookup_tcp), \ + FN(tcp_check_syncookie), \ + FN(sysctl_get_name), \ + FN(sysctl_get_current_value), \ + FN(sysctl_get_new_value), \ + FN(sysctl_set_new_value), \ + FN(strtol), \ + FN(strtoul), \ + FN(sk_storage_get), \ + FN(sk_storage_delete), /* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call @@ -2590,9 +2836,30 @@ enum bpf_func_id { /* Current network namespace */ #define BPF_F_CURRENT_NETNS (-1L) +/* BPF_FUNC_skb_adjust_room flags. */ +#define BPF_F_ADJ_ROOM_FIXED_GSO (1ULL << 0) + +#define BPF_ADJ_ROOM_ENCAP_L2_MASK 0xff +#define BPF_ADJ_ROOM_ENCAP_L2_SHIFT 56 + +#define BPF_F_ADJ_ROOM_ENCAP_L3_IPV4 (1ULL << 1) +#define BPF_F_ADJ_ROOM_ENCAP_L3_IPV6 (1ULL << 2) +#define BPF_F_ADJ_ROOM_ENCAP_L4_GRE (1ULL << 3) +#define BPF_F_ADJ_ROOM_ENCAP_L4_UDP (1ULL << 4) +#define BPF_F_ADJ_ROOM_ENCAP_L2(len) (((__u64)len & \ + BPF_ADJ_ROOM_ENCAP_L2_MASK) \ + << BPF_ADJ_ROOM_ENCAP_L2_SHIFT) + +/* BPF_FUNC_sysctl_get_name flags. */ +#define BPF_F_SYSCTL_BASE_NAME (1ULL << 0) + +/* BPF_FUNC_sk_storage_get flags */ +#define BPF_SK_STORAGE_GET_F_CREATE (1ULL << 0) + /* Mode for BPF_FUNC_skb_adjust_room helper. */ enum bpf_adj_room_mode { BPF_ADJ_ROOM_NET, + BPF_ADJ_ROOM_MAC, }; /* Mode for BPF_FUNC_skb_load_bytes_relative helper. */ @@ -3218,4 +3485,14 @@ struct bpf_line_info { struct bpf_spin_lock { __u32 val; }; + +struct bpf_sysctl { + __u32 write; /* Sysctl is being read (= 0) or written (= 1). + * Allows 1,2,4-byte read, but no write. + */ + __u32 file_pos; /* Sysctl file position to read from, write to. + * Allows 1,2,4-byte read an 4-byte write. + */ +}; + #endif /* _UAPI__LINUX_BPF_H__ */ diff --git a/tools/include/uapi/linux/btf.h b/tools/include/uapi/linux/btf.h index 7b7475ef2f175c9279915aa8d9eb014845533bd9..9310652ca4f96bb56c6f7f561332bcff01511f3a 100644 --- a/tools/include/uapi/linux/btf.h +++ b/tools/include/uapi/linux/btf.h @@ -39,11 +39,11 @@ struct btf_type { * struct, union and fwd */ __u32 info; - /* "size" is used by INT, ENUM, STRUCT and UNION. + /* "size" is used by INT, ENUM, STRUCT, UNION and DATASEC. * "size" tells the size of the type it is describing. * * "type" is used by PTR, TYPEDEF, VOLATILE, CONST, RESTRICT, - * FUNC and FUNC_PROTO. + * FUNC, FUNC_PROTO and VAR. * "type" is a type_id referring to another type. */ union { @@ -70,8 +70,10 @@ struct btf_type { #define BTF_KIND_RESTRICT 11 /* Restrict */ #define BTF_KIND_FUNC 12 /* Function */ #define BTF_KIND_FUNC_PROTO 13 /* Function Proto */ -#define BTF_KIND_MAX 13 -#define NR_BTF_KINDS 14 +#define BTF_KIND_VAR 14 /* Variable */ +#define BTF_KIND_DATASEC 15 /* Section */ +#define BTF_KIND_MAX BTF_KIND_DATASEC +#define NR_BTF_KINDS (BTF_KIND_MAX + 1) /* For some specific BTF_KIND, "struct btf_type" is immediately * followed by extra data. @@ -138,4 +140,26 @@ struct btf_param { __u32 type; }; +enum { + BTF_VAR_STATIC = 0, + BTF_VAR_GLOBAL_ALLOCATED, +}; + +/* BTF_KIND_VAR is followed by a single "struct btf_var" to describe + * additional information related to the variable such as its linkage. + */ +struct btf_var { + __u32 linkage; +}; + +/* BTF_KIND_DATASEC is followed by multiple "struct btf_var_secinfo" + * to describe all BTF_KIND_VAR types it contains along with it's + * in-section offset as well as size. + */ +struct btf_var_secinfo { + __u32 type; + __u32 offset; + __u32 size; +}; + #endif /* _UAPI__LINUX_BTF_H__ */ diff --git a/tools/lib/bpf/.gitignore b/tools/lib/bpf/.gitignore index fecb78afea3feb634750bf823811f1d5a70b3482..d9e9dec04605e619431c4929e409b261b651667a 100644 --- a/tools/lib/bpf/.gitignore +++ b/tools/lib/bpf/.gitignore @@ -1,4 +1,5 @@ libbpf_version.h +libbpf.pc FEATURE-DUMP.libbpf test_libbpf libbpf.so.* diff --git a/tools/lib/bpf/Makefile b/tools/lib/bpf/Makefile index 8e7c56e9590fbb0dfa2d76780696206b166ec440..f91639bf5650c18cd36daef9650625758d478f39 100644 --- a/tools/lib/bpf/Makefile +++ b/tools/lib/bpf/Makefile @@ -3,7 +3,7 @@ BPF_VERSION = 0 BPF_PATCHLEVEL = 0 -BPF_EXTRAVERSION = 2 +BPF_EXTRAVERSION = 3 MAKEFLAGS += --no-print-directory @@ -90,6 +90,7 @@ LIBBPF_VERSION = $(BPF_VERSION).$(BPF_PATCHLEVEL).$(BPF_EXTRAVERSION) LIB_TARGET = libbpf.a libbpf.so.$(LIBBPF_VERSION) LIB_FILE = libbpf.a libbpf.so* +PC_FILE = libbpf.pc # Set compile option CFLAGS ifdef EXTRA_CFLAGS @@ -134,13 +135,14 @@ VERSION_SCRIPT := libbpf.map LIB_TARGET := $(addprefix $(OUTPUT),$(LIB_TARGET)) LIB_FILE := $(addprefix $(OUTPUT),$(LIB_FILE)) +PC_FILE := $(addprefix $(OUTPUT),$(PC_FILE)) GLOBAL_SYM_COUNT = $(shell readelf -s --wide $(BPF_IN) | \ awk '/GLOBAL/ && /DEFAULT/ && !/UND/ {s++} END{print s}') VERSIONED_SYM_COUNT = $(shell readelf -s --wide $(OUTPUT)libbpf.so | \ grep -Eo '[^ ]+@LIBBPF_' | cut -d@ -f1 | sort -u | wc -l) -CMD_TARGETS = $(LIB_TARGET) +CMD_TARGETS = $(LIB_TARGET) $(PC_FILE) CXX_TEST_TARGET = $(OUTPUT)test_libbpf @@ -187,6 +189,12 @@ $(OUTPUT)libbpf.a: $(BPF_IN) $(OUTPUT)test_libbpf: test_libbpf.cpp $(OUTPUT)libbpf.a $(QUIET_LINK)$(CXX) $(INCLUDES) $^ -lelf -o $@ +$(OUTPUT)libbpf.pc: + $(QUIET_GEN)sed -e "s|@PREFIX@|$(prefix)|" \ + -e "s|@LIBDIR@|$(libdir_SQ)|" \ + -e "s|@VERSION@|$(LIBBPF_VERSION)|" \ + < libbpf.pc.template > $@ + check: check_abi check_abi: $(OUTPUT)libbpf.so @@ -222,9 +230,14 @@ install_headers: $(call do_install,bpf.h,$(prefix)/include/bpf,644); \ $(call do_install,libbpf.h,$(prefix)/include/bpf,644); \ $(call do_install,btf.h,$(prefix)/include/bpf,644); \ + $(call do_install,libbpf_util.h,$(prefix)/include/bpf,644); \ $(call do_install,xsk.h,$(prefix)/include/bpf,644); -install: install_lib +install_pkgconfig: $(PC_FILE) + $(call QUIET_INSTALL, $(PC_FILE)) \ + $(call do_install,$(PC_FILE),$(libdir_SQ)/pkgconfig,644) + +install: install_lib install_pkgconfig ### Cleaning rules @@ -234,7 +247,7 @@ config-clean: clean: $(call QUIET_CLEAN, libbpf) $(RM) $(TARGETS) $(CXX_TEST_TARGET) \ - *.o *~ *.a *.so *.so.$(VERSION) .*.d .*.cmd LIBBPF-CFLAGS + *.o *~ *.a *.so *.so.$(VERSION) .*.d .*.cmd *.pc LIBBPF-CFLAGS $(call QUIET_CLEAN, core-gen) $(RM) $(OUTPUT)FEATURE-DUMP.libbpf diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c index 9cd015574e83828d20ace576fe7190f68a924829..c4a48086dc9a5a6f926367dc1b05a4a25e62e41b 100644 --- a/tools/lib/bpf/bpf.c +++ b/tools/lib/bpf/bpf.c @@ -46,6 +46,8 @@ # define __NR_bpf 349 # elif defined(__s390__) # define __NR_bpf 351 +# elif defined(__arc__) +# define __NR_bpf 280 # else # error __NR_bpf not defined. libbpf does not support your arch. # endif @@ -79,7 +81,6 @@ static inline int sys_bpf_prog_load(union bpf_attr *attr, unsigned int size) int bpf_create_map_xattr(const struct bpf_create_map_attr *create_attr) { - __u32 name_len = create_attr->name ? strlen(create_attr->name) : 0; union bpf_attr attr; memset(&attr, '\0', sizeof(attr)); @@ -89,8 +90,9 @@ int bpf_create_map_xattr(const struct bpf_create_map_attr *create_attr) attr.value_size = create_attr->value_size; attr.max_entries = create_attr->max_entries; attr.map_flags = create_attr->map_flags; - memcpy(attr.map_name, create_attr->name, - min(name_len, BPF_OBJ_NAME_LEN - 1)); + if (create_attr->name) + memcpy(attr.map_name, create_attr->name, + min(strlen(create_attr->name), BPF_OBJ_NAME_LEN - 1)); attr.numa_node = create_attr->numa_node; attr.btf_fd = create_attr->btf_fd; attr.btf_key_type_id = create_attr->btf_key_type_id; @@ -155,7 +157,6 @@ int bpf_create_map_in_map_node(enum bpf_map_type map_type, const char *name, int key_size, int inner_map_fd, int max_entries, __u32 map_flags, int node) { - __u32 name_len = name ? strlen(name) : 0; union bpf_attr attr; memset(&attr, '\0', sizeof(attr)); @@ -166,7 +167,9 @@ int bpf_create_map_in_map_node(enum bpf_map_type map_type, const char *name, attr.inner_map_fd = inner_map_fd; attr.max_entries = max_entries; attr.map_flags = map_flags; - memcpy(attr.map_name, name, min(name_len, BPF_OBJ_NAME_LEN - 1)); + if (name) + memcpy(attr.map_name, name, + min(strlen(name), BPF_OBJ_NAME_LEN - 1)); if (node >= 0) { attr.map_flags |= BPF_F_NUMA_NODE; @@ -216,18 +219,15 @@ int bpf_load_program_xattr(const struct bpf_load_program_attr *load_attr, void *finfo = NULL, *linfo = NULL; union bpf_attr attr; __u32 log_level; - __u32 name_len; int fd; if (!load_attr || !log_buf != !log_buf_sz) return -EINVAL; log_level = load_attr->log_level; - if (log_level > 2 || (log_level && !log_buf)) + if (log_level > (4 | 2 | 1) || (log_level && !log_buf)) return -EINVAL; - name_len = load_attr->name ? strlen(load_attr->name) : 0; - memset(&attr, 0, sizeof(attr)); attr.prog_type = load_attr->prog_type; attr.expected_attach_type = load_attr->expected_attach_type; @@ -253,8 +253,9 @@ int bpf_load_program_xattr(const struct bpf_load_program_attr *load_attr, attr.line_info_rec_size = load_attr->line_info_rec_size; attr.line_info_cnt = load_attr->line_info_cnt; attr.line_info = ptr_to_u64(load_attr->line_info); - memcpy(attr.prog_name, load_attr->name, - min(name_len, BPF_OBJ_NAME_LEN - 1)); + if (load_attr->name) + memcpy(attr.prog_name, load_attr->name, + min(strlen(load_attr->name), BPF_OBJ_NAME_LEN - 1)); fd = sys_bpf_prog_load(&attr, sizeof(attr)); if (fd >= 0) @@ -429,6 +430,16 @@ int bpf_map_get_next_key(int fd, const void *key, void *next_key) return sys_bpf(BPF_MAP_GET_NEXT_KEY, &attr, sizeof(attr)); } +int bpf_map_freeze(int fd) +{ + union bpf_attr attr; + + memset(&attr, 0, sizeof(attr)); + attr.map_fd = fd; + + return sys_bpf(BPF_MAP_FREEZE, &attr, sizeof(attr)); +} + int bpf_obj_pin(int fd, const char *pathname) { union bpf_attr attr; @@ -545,10 +556,15 @@ int bpf_prog_test_run_xattr(struct bpf_prog_test_run_attr *test_attr) attr.test.data_out = ptr_to_u64(test_attr->data_out); attr.test.data_size_in = test_attr->data_size_in; attr.test.data_size_out = test_attr->data_size_out; + attr.test.ctx_in = ptr_to_u64(test_attr->ctx_in); + attr.test.ctx_out = ptr_to_u64(test_attr->ctx_out); + attr.test.ctx_size_in = test_attr->ctx_size_in; + attr.test.ctx_size_out = test_attr->ctx_size_out; attr.test.repeat = test_attr->repeat; ret = sys_bpf(BPF_PROG_TEST_RUN, &attr, sizeof(attr)); test_attr->data_size_out = attr.test.data_size_out; + test_attr->ctx_size_out = attr.test.ctx_size_out; test_attr->retval = attr.test.retval; test_attr->duration = attr.test.duration; return ret; diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h index 6ffdd79bea89df1e8e2d015130f15a93e504ff94..9593fec75652e571b6c5447d3c3aa47786fadcaf 100644 --- a/tools/lib/bpf/bpf.h +++ b/tools/lib/bpf/bpf.h @@ -26,6 +26,7 @@ #include #include #include +#include #ifdef __cplusplus extern "C" { @@ -92,7 +93,7 @@ struct bpf_load_program_attr { #define MAPS_RELAX_COMPAT 0x01 /* Recommend log buffer size */ -#define BPF_LOG_BUF_SIZE (256 * 1024) +#define BPF_LOG_BUF_SIZE (UINT32_MAX >> 8) /* verifier maximum in kernels <= 5.1 */ LIBBPF_API int bpf_load_program_xattr(const struct bpf_load_program_attr *load_attr, char *log_buf, size_t log_buf_sz); @@ -117,6 +118,7 @@ LIBBPF_API int bpf_map_lookup_and_delete_elem(int fd, const void *key, void *value); LIBBPF_API int bpf_map_delete_elem(int fd, const void *key); LIBBPF_API int bpf_map_get_next_key(int fd, const void *key, void *next_key); +LIBBPF_API int bpf_map_freeze(int fd); LIBBPF_API int bpf_obj_pin(int fd, const char *pathname); LIBBPF_API int bpf_obj_get(const char *pathname); LIBBPF_API int bpf_prog_attach(int prog_fd, int attachable_fd, @@ -135,6 +137,11 @@ struct bpf_prog_test_run_attr { * out: length of data_out */ __u32 retval; /* out: return code of the BPF program */ __u32 duration; /* out: average per repetition in ns */ + const void *ctx_in; /* optional */ + __u32 ctx_size_in; + void *ctx_out; /* optional */ + __u32 ctx_size_out; /* in: max length of ctx_out + * out: length of cxt_out */ }; LIBBPF_API int bpf_prog_test_run_xattr(struct bpf_prog_test_run_attr *test_attr); diff --git a/tools/lib/bpf/btf.c b/tools/lib/bpf/btf.c index cf119c9b6f2700e790ec02e16b524f6b3a4cf581..75eaf10b9e1aa6fd3ceba642013c8a1f1815ae46 100644 --- a/tools/lib/bpf/btf.c +++ b/tools/lib/bpf/btf.c @@ -24,6 +24,8 @@ ((k) == BTF_KIND_CONST) || \ ((k) == BTF_KIND_RESTRICT)) +#define IS_VAR(k) ((k) == BTF_KIND_VAR) + static struct btf_type btf_void; struct btf { @@ -212,6 +214,10 @@ static int btf_type_size(struct btf_type *t) return base_size + vlen * sizeof(struct btf_member); case BTF_KIND_FUNC_PROTO: return base_size + vlen * sizeof(struct btf_param); + case BTF_KIND_VAR: + return base_size + sizeof(struct btf_var); + case BTF_KIND_DATASEC: + return base_size + vlen * sizeof(struct btf_var_secinfo); default: pr_debug("Unsupported BTF_KIND:%u\n", BTF_INFO_KIND(t->info)); return -EINVAL; @@ -283,6 +289,7 @@ __s64 btf__resolve_size(const struct btf *btf, __u32 type_id) case BTF_KIND_STRUCT: case BTF_KIND_UNION: case BTF_KIND_ENUM: + case BTF_KIND_DATASEC: size = t->size; goto done; case BTF_KIND_PTR: @@ -292,6 +299,7 @@ __s64 btf__resolve_size(const struct btf *btf, __u32 type_id) case BTF_KIND_VOLATILE: case BTF_KIND_CONST: case BTF_KIND_RESTRICT: + case BTF_KIND_VAR: type_id = t->type; break; case BTF_KIND_ARRAY: @@ -326,7 +334,8 @@ int btf__resolve_type(const struct btf *btf, __u32 type_id) t = btf__type_by_id(btf, type_id); while (depth < MAX_RESOLVE_DEPTH && !btf_type_is_void_or_null(t) && - IS_MODIFIER(BTF_INFO_KIND(t->info))) { + (IS_MODIFIER(BTF_INFO_KIND(t->info)) || + IS_VAR(BTF_INFO_KIND(t->info)))) { type_id = t->type; t = btf__type_by_id(btf, type_id); depth++; @@ -408,6 +417,92 @@ done: return btf; } +static int compare_vsi_off(const void *_a, const void *_b) +{ + const struct btf_var_secinfo *a = _a; + const struct btf_var_secinfo *b = _b; + + return a->offset - b->offset; +} + +static int btf_fixup_datasec(struct bpf_object *obj, struct btf *btf, + struct btf_type *t) +{ + __u32 size = 0, off = 0, i, vars = BTF_INFO_VLEN(t->info); + const char *name = btf__name_by_offset(btf, t->name_off); + const struct btf_type *t_var; + struct btf_var_secinfo *vsi; + struct btf_var *var; + int ret; + + if (!name) { + pr_debug("No name found in string section for DATASEC kind.\n"); + return -ENOENT; + } + + ret = bpf_object__section_size(obj, name, &size); + if (ret || !size || (t->size && t->size != size)) { + pr_debug("Invalid size for section %s: %u bytes\n", name, size); + return -ENOENT; + } + + t->size = size; + + for (i = 0, vsi = (struct btf_var_secinfo *)(t + 1); + i < vars; i++, vsi++) { + t_var = btf__type_by_id(btf, vsi->type); + var = (struct btf_var *)(t_var + 1); + + if (BTF_INFO_KIND(t_var->info) != BTF_KIND_VAR) { + pr_debug("Non-VAR type seen in section %s\n", name); + return -EINVAL; + } + + if (var->linkage == BTF_VAR_STATIC) + continue; + + name = btf__name_by_offset(btf, t_var->name_off); + if (!name) { + pr_debug("No name found in string section for VAR kind\n"); + return -ENOENT; + } + + ret = bpf_object__variable_offset(obj, name, &off); + if (ret) { + pr_debug("No offset found in symbol table for VAR %s\n", name); + return -ENOENT; + } + + vsi->offset = off; + } + + qsort(t + 1, vars, sizeof(*vsi), compare_vsi_off); + return 0; +} + +int btf__finalize_data(struct bpf_object *obj, struct btf *btf) +{ + int err = 0; + __u32 i; + + for (i = 1; i <= btf->nr_types; i++) { + struct btf_type *t = btf->types[i]; + + /* Loader needs to fix up some of the things compiler + * couldn't get its hands on while emitting BTF. This + * is section size and global variable offset. We use + * the info from the ELF itself for this purpose. + */ + if (BTF_INFO_KIND(t->info) == BTF_KIND_DATASEC) { + err = btf_fixup_datasec(obj, btf, t); + if (err) + break; + } + } + + return err; +} + int btf__load(struct btf *btf) { __u32 log_buf_size = BPF_LOG_BUF_SIZE; @@ -1259,8 +1354,16 @@ static struct btf_dedup *btf_dedup_new(struct btf *btf, struct btf_ext *btf_ext, } /* special BTF "void" type is made canonical immediately */ d->map[0] = 0; - for (i = 1; i <= btf->nr_types; i++) - d->map[i] = BTF_UNPROCESSED_ID; + for (i = 1; i <= btf->nr_types; i++) { + struct btf_type *t = d->btf->types[i]; + __u16 kind = BTF_INFO_KIND(t->info); + + /* VAR and DATASEC are never deduped and are self-canonical */ + if (kind == BTF_KIND_VAR || kind == BTF_KIND_DATASEC) + d->map[i] = i; + else + d->map[i] = BTF_UNPROCESSED_ID; + } d->hypot_map = malloc(sizeof(__u32) * (1 + btf->nr_types)); if (!d->hypot_map) { @@ -1851,6 +1954,8 @@ static int btf_dedup_prim_type(struct btf_dedup *d, __u32 type_id) case BTF_KIND_UNION: case BTF_KIND_FUNC: case BTF_KIND_FUNC_PROTO: + case BTF_KIND_VAR: + case BTF_KIND_DATASEC: return 0; case BTF_KIND_INT: @@ -2604,6 +2709,7 @@ static int btf_dedup_remap_type(struct btf_dedup *d, __u32 type_id) case BTF_KIND_PTR: case BTF_KIND_TYPEDEF: case BTF_KIND_FUNC: + case BTF_KIND_VAR: r = btf_dedup_remap_type_id(d, t->type); if (r < 0) return r; @@ -2658,6 +2764,20 @@ static int btf_dedup_remap_type(struct btf_dedup *d, __u32 type_id) break; } + case BTF_KIND_DATASEC: { + struct btf_var_secinfo *var = (struct btf_var_secinfo *)(t + 1); + __u16 vlen = BTF_INFO_VLEN(t->info); + + for (i = 0; i < vlen; i++) { + r = btf_dedup_remap_type_id(d, var->type); + if (r < 0) + return r; + var->type = r; + var++; + } + break; + } + default: return -EINVAL; } diff --git a/tools/lib/bpf/btf.h b/tools/lib/bpf/btf.h index 28a1e1e59861927dc31783b74ee11be302667f32..c7b399e81fcefae69ac07e48dd103811924d1e29 100644 --- a/tools/lib/bpf/btf.h +++ b/tools/lib/bpf/btf.h @@ -21,6 +21,8 @@ struct btf; struct btf_ext; struct btf_type; +struct bpf_object; + /* * The .BTF.ext ELF section layout defined as * struct btf_ext_header @@ -57,6 +59,7 @@ struct btf_ext_header { LIBBPF_API void btf__free(struct btf *btf); LIBBPF_API struct btf *btf__new(__u8 *data, __u32 size); +LIBBPF_API int btf__finalize_data(struct bpf_object *obj, struct btf *btf); LIBBPF_API int btf__load(struct btf *btf); LIBBPF_API __s32 btf__find_by_name(const struct btf *btf, const char *type_name); diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 11c25d9ea43124fc6e67fab5a7dc8b93f5dc9e4a..11a65db4b93fbf15320148c5f8a05da84febe385 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -7,6 +7,7 @@ * Copyright (C) 2015 Wang Nan * Copyright (C) 2015 Huawei Inc. * Copyright (C) 2017 Nicira, Inc. + * Copyright (C) 2019 Isovalent, Inc. */ #ifndef _GNU_SOURCE @@ -52,6 +53,11 @@ #define BPF_FS_MAGIC 0xcafe4a11 #endif +/* vsprintf() in __base_pr() uses nonliteral format string. It may break + * compilation if user enables corresponding warning. Disable it explicitly. + */ +#pragma GCC diagnostic ignored "-Wformat-nonliteral" + #define __printf(a, b) __attribute__((format(printf, a, b))) static int __base_pr(enum libbpf_print_level level, const char *format, @@ -120,6 +126,8 @@ static inline __u64 ptr_to_u64(const void *ptr) struct bpf_capabilities { /* v4.14: kernel support for program & map names. */ __u32 name:1; + /* v5.2: kernel support for global data sections. */ + __u32 global_data:1; }; /* @@ -144,6 +152,7 @@ struct bpf_program { enum { RELO_LD64, RELO_CALL, + RELO_DATA, } type; int insn_idx; union { @@ -152,6 +161,7 @@ struct bpf_program { }; } *reloc_desc; int nr_reloc; + int log_level; struct { int nr; @@ -176,6 +186,19 @@ struct bpf_program { __u32 line_info_cnt; }; +enum libbpf_map_type { + LIBBPF_MAP_UNSPEC, + LIBBPF_MAP_DATA, + LIBBPF_MAP_BSS, + LIBBPF_MAP_RODATA, +}; + +static const char * const libbpf_type_to_btf_name[] = { + [LIBBPF_MAP_DATA] = ".data", + [LIBBPF_MAP_BSS] = ".bss", + [LIBBPF_MAP_RODATA] = ".rodata", +}; + struct bpf_map { int fd; char *name; @@ -187,11 +210,18 @@ struct bpf_map { __u32 btf_value_type_id; void *priv; bpf_map_clear_priv_t clear_priv; + enum libbpf_map_type libbpf_type; +}; + +struct bpf_secdata { + void *rodata; + void *data; }; static LIST_HEAD(bpf_objects_list); struct bpf_object { + char name[BPF_OBJ_NAME_LEN]; char license[64]; __u32 kern_version; @@ -199,6 +229,7 @@ struct bpf_object { size_t nr_programs; struct bpf_map *maps; size_t nr_maps; + struct bpf_secdata sections; bool loaded; bool has_pseudo_calls; @@ -214,6 +245,9 @@ struct bpf_object { Elf *elf; GElf_Ehdr ehdr; Elf_Data *symbols; + Elf_Data *data; + Elf_Data *rodata; + Elf_Data *bss; size_t strtabidx; struct { GElf_Shdr shdr; @@ -222,6 +256,9 @@ struct bpf_object { int nr_reloc; int maps_shndx; int text_shndx; + int data_shndx; + int rodata_shndx; + int bss_shndx; } efile; /* * All loaded bpf_object is linked in a list, which is @@ -443,6 +480,7 @@ static struct bpf_object *bpf_object__new(const char *path, size_t obj_buf_sz) { struct bpf_object *obj; + char *end; obj = calloc(1, sizeof(struct bpf_object) + strlen(path) + 1); if (!obj) { @@ -451,8 +489,14 @@ static struct bpf_object *bpf_object__new(const char *path, } strcpy(obj->path, path); - obj->efile.fd = -1; + /* Using basename() GNU version which doesn't modify arg. */ + strncpy(obj->name, basename((void *)path), + sizeof(obj->name) - 1); + end = strchr(obj->name, '.'); + if (end) + *end = 0; + obj->efile.fd = -1; /* * Caller of this function should also calls * bpf_object__elf_finish() after data collection to return @@ -462,6 +506,9 @@ static struct bpf_object *bpf_object__new(const char *path, obj->efile.obj_buf = obj_buf; obj->efile.obj_buf_sz = obj_buf_sz; obj->efile.maps_shndx = -1; + obj->efile.data_shndx = -1; + obj->efile.rodata_shndx = -1; + obj->efile.bss_shndx = -1; obj->loaded = false; @@ -480,6 +527,9 @@ static void bpf_object__elf_finish(struct bpf_object *obj) obj->efile.elf = NULL; } obj->efile.symbols = NULL; + obj->efile.data = NULL; + obj->efile.rodata = NULL; + obj->efile.bss = NULL; zfree(&obj->efile.reloc); obj->efile.nr_reloc = 0; @@ -621,27 +671,182 @@ static bool bpf_map_type__is_map_in_map(enum bpf_map_type type) return false; } +static int bpf_object_search_section_size(const struct bpf_object *obj, + const char *name, size_t *d_size) +{ + const GElf_Ehdr *ep = &obj->efile.ehdr; + Elf *elf = obj->efile.elf; + Elf_Scn *scn = NULL; + int idx = 0; + + while ((scn = elf_nextscn(elf, scn)) != NULL) { + const char *sec_name; + Elf_Data *data; + GElf_Shdr sh; + + idx++; + if (gelf_getshdr(scn, &sh) != &sh) { + pr_warning("failed to get section(%d) header from %s\n", + idx, obj->path); + return -EIO; + } + + sec_name = elf_strptr(elf, ep->e_shstrndx, sh.sh_name); + if (!sec_name) { + pr_warning("failed to get section(%d) name from %s\n", + idx, obj->path); + return -EIO; + } + + if (strcmp(name, sec_name)) + continue; + + data = elf_getdata(scn, 0); + if (!data) { + pr_warning("failed to get section(%d) data from %s(%s)\n", + idx, name, obj->path); + return -EIO; + } + + *d_size = data->d_size; + return 0; + } + + return -ENOENT; +} + +int bpf_object__section_size(const struct bpf_object *obj, const char *name, + __u32 *size) +{ + int ret = -ENOENT; + size_t d_size; + + *size = 0; + if (!name) { + return -EINVAL; + } else if (!strcmp(name, ".data")) { + if (obj->efile.data) + *size = obj->efile.data->d_size; + } else if (!strcmp(name, ".bss")) { + if (obj->efile.bss) + *size = obj->efile.bss->d_size; + } else if (!strcmp(name, ".rodata")) { + if (obj->efile.rodata) + *size = obj->efile.rodata->d_size; + } else { + ret = bpf_object_search_section_size(obj, name, &d_size); + if (!ret) + *size = d_size; + } + + return *size ? 0 : ret; +} + +int bpf_object__variable_offset(const struct bpf_object *obj, const char *name, + __u32 *off) +{ + Elf_Data *symbols = obj->efile.symbols; + const char *sname; + size_t si; + + if (!name || !off) + return -EINVAL; + + for (si = 0; si < symbols->d_size / sizeof(GElf_Sym); si++) { + GElf_Sym sym; + + if (!gelf_getsym(symbols, si, &sym)) + continue; + if (GELF_ST_BIND(sym.st_info) != STB_GLOBAL || + GELF_ST_TYPE(sym.st_info) != STT_OBJECT) + continue; + + sname = elf_strptr(obj->efile.elf, obj->efile.strtabidx, + sym.st_name); + if (!sname) { + pr_warning("failed to get sym name string for var %s\n", + name); + return -EIO; + } + if (strcmp(name, sname) == 0) { + *off = sym.st_value; + return 0; + } + } + + return -ENOENT; +} + +static bool bpf_object__has_maps(const struct bpf_object *obj) +{ + return obj->efile.maps_shndx >= 0 || + obj->efile.data_shndx >= 0 || + obj->efile.rodata_shndx >= 0 || + obj->efile.bss_shndx >= 0; +} + +static int +bpf_object__init_internal_map(struct bpf_object *obj, struct bpf_map *map, + enum libbpf_map_type type, Elf_Data *data, + void **data_buff) +{ + struct bpf_map_def *def = &map->def; + char map_name[BPF_OBJ_NAME_LEN]; + + map->libbpf_type = type; + map->offset = ~(typeof(map->offset))0; + snprintf(map_name, sizeof(map_name), "%.8s%.7s", obj->name, + libbpf_type_to_btf_name[type]); + map->name = strdup(map_name); + if (!map->name) { + pr_warning("failed to alloc map name\n"); + return -ENOMEM; + } + + def->type = BPF_MAP_TYPE_ARRAY; + def->key_size = sizeof(int); + def->value_size = data->d_size; + def->max_entries = 1; + def->map_flags = type == LIBBPF_MAP_RODATA ? + BPF_F_RDONLY_PROG : 0; + if (data_buff) { + *data_buff = malloc(data->d_size); + if (!*data_buff) { + zfree(&map->name); + pr_warning("failed to alloc map content buffer\n"); + return -ENOMEM; + } + memcpy(*data_buff, data->d_buf, data->d_size); + } + + pr_debug("map %td is \"%s\"\n", map - obj->maps, map->name); + return 0; +} + static int bpf_object__init_maps(struct bpf_object *obj, int flags) { + int i, map_idx, map_def_sz = 0, nr_syms, nr_maps = 0, nr_maps_glob = 0; bool strict = !(flags & MAPS_RELAX_COMPAT); - int i, map_idx, map_def_sz, nr_maps = 0; - Elf_Scn *scn; - Elf_Data *data = NULL; Elf_Data *symbols = obj->efile.symbols; + Elf_Data *data = NULL; + int ret = 0; - if (obj->efile.maps_shndx < 0) - return -EINVAL; if (!symbols) return -EINVAL; + nr_syms = symbols->d_size / sizeof(GElf_Sym); - scn = elf_getscn(obj->efile.elf, obj->efile.maps_shndx); - if (scn) - data = elf_getdata(scn, NULL); - if (!scn || !data) { - pr_warning("failed to get Elf_Data from map section %d\n", - obj->efile.maps_shndx); - return -EINVAL; + if (obj->efile.maps_shndx >= 0) { + Elf_Scn *scn = elf_getscn(obj->efile.elf, + obj->efile.maps_shndx); + + if (scn) + data = elf_getdata(scn, NULL); + if (!scn || !data) { + pr_warning("failed to get Elf_Data from map section %d\n", + obj->efile.maps_shndx); + return -EINVAL; + } } /* @@ -651,7 +856,16 @@ bpf_object__init_maps(struct bpf_object *obj, int flags) * * TODO: Detect array of map and report error. */ - for (i = 0; i < symbols->d_size / sizeof(GElf_Sym); i++) { + if (obj->caps.global_data) { + if (obj->efile.data_shndx >= 0) + nr_maps_glob++; + if (obj->efile.rodata_shndx >= 0) + nr_maps_glob++; + if (obj->efile.bss_shndx >= 0) + nr_maps_glob++; + } + + for (i = 0; data && i < nr_syms; i++) { GElf_Sym sym; if (!gelf_getsym(symbols, i, &sym)) @@ -661,22 +875,24 @@ bpf_object__init_maps(struct bpf_object *obj, int flags) nr_maps++; } - /* Alloc obj->maps and fill nr_maps. */ - pr_debug("maps in %s: %d maps in %zd bytes\n", obj->path, - nr_maps, data->d_size); - - if (!nr_maps) + if (!nr_maps && !nr_maps_glob) return 0; /* Assume equally sized map definitions */ - map_def_sz = data->d_size / nr_maps; - if (!data->d_size || (data->d_size % nr_maps) != 0) { - pr_warning("unable to determine map definition size " - "section %s, %d maps in %zd bytes\n", - obj->path, nr_maps, data->d_size); - return -EINVAL; + if (data) { + pr_debug("maps in %s: %d maps in %zd bytes\n", obj->path, + nr_maps, data->d_size); + + map_def_sz = data->d_size / nr_maps; + if (!data->d_size || (data->d_size % nr_maps) != 0) { + pr_warning("unable to determine map definition size " + "section %s, %d maps in %zd bytes\n", + obj->path, nr_maps, data->d_size); + return -EINVAL; + } } + nr_maps += nr_maps_glob; obj->maps = calloc(nr_maps, sizeof(obj->maps[0])); if (!obj->maps) { pr_warning("alloc maps for object failed\n"); @@ -697,7 +913,7 @@ bpf_object__init_maps(struct bpf_object *obj, int flags) /* * Fill obj->maps using data in "maps" section. */ - for (i = 0, map_idx = 0; i < symbols->d_size / sizeof(GElf_Sym); i++) { + for (i = 0, map_idx = 0; data && i < nr_syms; i++) { GElf_Sym sym; const char *map_name; struct bpf_map_def *def; @@ -710,6 +926,8 @@ bpf_object__init_maps(struct bpf_object *obj, int flags) map_name = elf_strptr(obj->efile.elf, obj->efile.strtabidx, sym.st_name); + + obj->maps[map_idx].libbpf_type = LIBBPF_MAP_UNSPEC; obj->maps[map_idx].offset = sym.st_value; if (sym.st_value + map_def_sz > data->d_size) { pr_warning("corrupted maps section in %s: last map \"%s\" too small\n", @@ -758,8 +976,31 @@ bpf_object__init_maps(struct bpf_object *obj, int flags) map_idx++; } - qsort(obj->maps, obj->nr_maps, sizeof(obj->maps[0]), compare_bpf_map); - return 0; + if (!obj->caps.global_data) + goto finalize; + + /* + * Populate rest of obj->maps with libbpf internal maps. + */ + if (obj->efile.data_shndx >= 0) + ret = bpf_object__init_internal_map(obj, &obj->maps[map_idx++], + LIBBPF_MAP_DATA, + obj->efile.data, + &obj->sections.data); + if (!ret && obj->efile.rodata_shndx >= 0) + ret = bpf_object__init_internal_map(obj, &obj->maps[map_idx++], + LIBBPF_MAP_RODATA, + obj->efile.rodata, + &obj->sections.rodata); + if (!ret && obj->efile.bss_shndx >= 0) + ret = bpf_object__init_internal_map(obj, &obj->maps[map_idx++], + LIBBPF_MAP_BSS, + obj->efile.bss, NULL); +finalize: + if (!ret) + qsort(obj->maps, obj->nr_maps, sizeof(obj->maps[0]), + compare_bpf_map); + return ret; } static bool section_have_execinstr(struct bpf_object *obj, int idx) @@ -785,6 +1026,7 @@ static int bpf_object__elf_collect(struct bpf_object *obj, int flags) Elf *elf = obj->efile.elf; GElf_Ehdr *ep = &obj->efile.ehdr; Elf_Data *btf_ext_data = NULL; + Elf_Data *btf_data = NULL; Elf_Scn *scn = NULL; int idx = 0, err = 0; @@ -828,32 +1070,18 @@ static int bpf_object__elf_collect(struct bpf_object *obj, int flags) (int)sh.sh_link, (unsigned long)sh.sh_flags, (int)sh.sh_type); - if (strcmp(name, "license") == 0) + if (strcmp(name, "license") == 0) { err = bpf_object__init_license(obj, data->d_buf, data->d_size); - else if (strcmp(name, "version") == 0) + } else if (strcmp(name, "version") == 0) { err = bpf_object__init_kversion(obj, data->d_buf, data->d_size); - else if (strcmp(name, "maps") == 0) + } else if (strcmp(name, "maps") == 0) { obj->efile.maps_shndx = idx; - else if (strcmp(name, BTF_ELF_SEC) == 0) { - obj->btf = btf__new(data->d_buf, data->d_size); - if (IS_ERR(obj->btf)) { - pr_warning("Error loading ELF section %s: %ld. Ignored and continue.\n", - BTF_ELF_SEC, PTR_ERR(obj->btf)); - obj->btf = NULL; - continue; - } - err = btf__load(obj->btf); - if (err) { - pr_warning("Error loading %s into kernel: %d. Ignored and continue.\n", - BTF_ELF_SEC, err); - btf__free(obj->btf); - obj->btf = NULL; - err = 0; - } + } else if (strcmp(name, BTF_ELF_SEC) == 0) { + btf_data = data; } else if (strcmp(name, BTF_EXT_ELF_SEC) == 0) { btf_ext_data = data; } else if (sh.sh_type == SHT_SYMTAB) { @@ -865,20 +1093,28 @@ static int bpf_object__elf_collect(struct bpf_object *obj, int flags) obj->efile.symbols = data; obj->efile.strtabidx = sh.sh_link; } - } else if ((sh.sh_type == SHT_PROGBITS) && - (sh.sh_flags & SHF_EXECINSTR) && - (data->d_size > 0)) { - if (strcmp(name, ".text") == 0) - obj->efile.text_shndx = idx; - err = bpf_object__add_program(obj, data->d_buf, - data->d_size, name, idx); - if (err) { - char errmsg[STRERR_BUFSIZE]; - char *cp = libbpf_strerror_r(-err, errmsg, - sizeof(errmsg)); - - pr_warning("failed to alloc program %s (%s): %s", - name, obj->path, cp); + } else if (sh.sh_type == SHT_PROGBITS && data->d_size > 0) { + if (sh.sh_flags & SHF_EXECINSTR) { + if (strcmp(name, ".text") == 0) + obj->efile.text_shndx = idx; + err = bpf_object__add_program(obj, data->d_buf, + data->d_size, name, idx); + if (err) { + char errmsg[STRERR_BUFSIZE]; + char *cp = libbpf_strerror_r(-err, errmsg, + sizeof(errmsg)); + + pr_warning("failed to alloc program %s (%s): %s", + name, obj->path, cp); + } + } else if (strcmp(name, ".data") == 0) { + obj->efile.data = data; + obj->efile.data_shndx = idx; + } else if (strcmp(name, ".rodata") == 0) { + obj->efile.rodata = data; + obj->efile.rodata_shndx = idx; + } else { + pr_debug("skip section(%d) %s\n", idx, name); } } else if (sh.sh_type == SHT_REL) { void *reloc = obj->efile.reloc; @@ -906,6 +1142,9 @@ static int bpf_object__elf_collect(struct bpf_object *obj, int flags) obj->efile.reloc[n].shdr = sh; obj->efile.reloc[n].data = data; } + } else if (sh.sh_type == SHT_NOBITS && strcmp(name, ".bss") == 0) { + obj->efile.bss = data; + obj->efile.bss_shndx = idx; } else { pr_debug("skip section(%d) %s\n", idx, name); } @@ -917,6 +1156,25 @@ static int bpf_object__elf_collect(struct bpf_object *obj, int flags) pr_warning("Corrupted ELF file: index of strtab invalid\n"); return LIBBPF_ERRNO__FORMAT; } + if (btf_data) { + obj->btf = btf__new(btf_data->d_buf, btf_data->d_size); + if (IS_ERR(obj->btf)) { + pr_warning("Error loading ELF section %s: %ld. Ignored and continue.\n", + BTF_ELF_SEC, PTR_ERR(obj->btf)); + obj->btf = NULL; + } else { + err = btf__finalize_data(obj, obj->btf); + if (!err) + err = btf__load(obj->btf); + if (err) { + pr_warning("Error finalizing and loading %s into kernel: %d. Ignored and continue.\n", + BTF_ELF_SEC, err); + btf__free(obj->btf); + obj->btf = NULL; + err = 0; + } + } + } if (btf_ext_data) { if (!obj->btf) { pr_debug("Ignore ELF section %s because its depending ELF section %s is not found.\n", @@ -932,7 +1190,7 @@ static int bpf_object__elf_collect(struct bpf_object *obj, int flags) } } } - if (obj->efile.maps_shndx >= 0) { + if (bpf_object__has_maps(obj)) { err = bpf_object__init_maps(obj, flags); if (err) goto out; @@ -968,13 +1226,46 @@ bpf_object__find_program_by_title(struct bpf_object *obj, const char *title) return NULL; } +static bool bpf_object__shndx_is_data(const struct bpf_object *obj, + int shndx) +{ + return shndx == obj->efile.data_shndx || + shndx == obj->efile.bss_shndx || + shndx == obj->efile.rodata_shndx; +} + +static bool bpf_object__shndx_is_maps(const struct bpf_object *obj, + int shndx) +{ + return shndx == obj->efile.maps_shndx; +} + +static bool bpf_object__relo_in_known_section(const struct bpf_object *obj, + int shndx) +{ + return shndx == obj->efile.text_shndx || + bpf_object__shndx_is_maps(obj, shndx) || + bpf_object__shndx_is_data(obj, shndx); +} + +static enum libbpf_map_type +bpf_object__section_to_libbpf_map_type(const struct bpf_object *obj, int shndx) +{ + if (shndx == obj->efile.data_shndx) + return LIBBPF_MAP_DATA; + else if (shndx == obj->efile.bss_shndx) + return LIBBPF_MAP_BSS; + else if (shndx == obj->efile.rodata_shndx) + return LIBBPF_MAP_RODATA; + else + return LIBBPF_MAP_UNSPEC; +} + static int bpf_program__collect_reloc(struct bpf_program *prog, GElf_Shdr *shdr, Elf_Data *data, struct bpf_object *obj) { Elf_Data *symbols = obj->efile.symbols; - int text_shndx = obj->efile.text_shndx; - int maps_shndx = obj->efile.maps_shndx; struct bpf_map *maps = obj->maps; size_t nr_maps = obj->nr_maps; int i, nrels; @@ -994,7 +1285,10 @@ bpf_program__collect_reloc(struct bpf_program *prog, GElf_Shdr *shdr, GElf_Sym sym; GElf_Rel rel; unsigned int insn_idx; + unsigned int shdr_idx; struct bpf_insn *insns = prog->insns; + enum libbpf_map_type type; + const char *name; size_t map_idx; if (!gelf_getrel(data, i, &rel)) { @@ -1009,13 +1303,18 @@ bpf_program__collect_reloc(struct bpf_program *prog, GElf_Shdr *shdr, GELF_R_SYM(rel.r_info)); return -LIBBPF_ERRNO__FORMAT; } - pr_debug("relo for %lld value %lld name %d\n", + + name = elf_strptr(obj->efile.elf, obj->efile.strtabidx, + sym.st_name) ? : ""; + + pr_debug("relo for %lld value %lld name %d (\'%s\')\n", (long long) (rel.r_info >> 32), - (long long) sym.st_value, sym.st_name); + (long long) sym.st_value, sym.st_name, name); - if (sym.st_shndx != maps_shndx && sym.st_shndx != text_shndx) { - pr_warning("Program '%s' contains non-map related relo data pointing to section %u\n", - prog->section_name, sym.st_shndx); + shdr_idx = sym.st_shndx; + if (!bpf_object__relo_in_known_section(obj, shdr_idx)) { + pr_warning("Program '%s' contains unrecognized relo data pointing to section %u\n", + prog->section_name, shdr_idx); return -LIBBPF_ERRNO__RELOC; } @@ -1040,24 +1339,45 @@ bpf_program__collect_reloc(struct bpf_program *prog, GElf_Shdr *shdr, return -LIBBPF_ERRNO__RELOC; } - /* TODO: 'maps' is sorted. We can use bsearch to make it faster. */ - for (map_idx = 0; map_idx < nr_maps; map_idx++) { - if (maps[map_idx].offset == sym.st_value) { - pr_debug("relocation: find map %zd (%s) for insn %u\n", - map_idx, maps[map_idx].name, insn_idx); - break; + if (bpf_object__shndx_is_maps(obj, shdr_idx) || + bpf_object__shndx_is_data(obj, shdr_idx)) { + type = bpf_object__section_to_libbpf_map_type(obj, shdr_idx); + if (type != LIBBPF_MAP_UNSPEC) { + if (GELF_ST_BIND(sym.st_info) == STB_GLOBAL) { + pr_warning("bpf: relocation: not yet supported relo for non-static global \'%s\' variable found in insns[%d].code 0x%x\n", + name, insn_idx, insns[insn_idx].code); + return -LIBBPF_ERRNO__RELOC; + } + if (!obj->caps.global_data) { + pr_warning("bpf: relocation: kernel does not support global \'%s\' variable access in insns[%d]\n", + name, insn_idx); + return -LIBBPF_ERRNO__RELOC; + } } - } - if (map_idx >= nr_maps) { - pr_warning("bpf relocation: map_idx %d large than %d\n", - (int)map_idx, (int)nr_maps - 1); - return -LIBBPF_ERRNO__RELOC; - } + for (map_idx = 0; map_idx < nr_maps; map_idx++) { + if (maps[map_idx].libbpf_type != type) + continue; + if (type != LIBBPF_MAP_UNSPEC || + (type == LIBBPF_MAP_UNSPEC && + maps[map_idx].offset == sym.st_value)) { + pr_debug("relocation: find map %zd (%s) for insn %u\n", + map_idx, maps[map_idx].name, insn_idx); + break; + } + } - prog->reloc_desc[i].type = RELO_LD64; - prog->reloc_desc[i].insn_idx = insn_idx; - prog->reloc_desc[i].map_idx = map_idx; + if (map_idx >= nr_maps) { + pr_warning("bpf relocation: map_idx %d large than %d\n", + (int)map_idx, (int)nr_maps - 1); + return -LIBBPF_ERRNO__RELOC; + } + + prog->reloc_desc[i].type = type != LIBBPF_MAP_UNSPEC ? + RELO_DATA : RELO_LD64; + prog->reloc_desc[i].insn_idx = insn_idx; + prog->reloc_desc[i].map_idx = map_idx; + } } return 0; } @@ -1065,18 +1385,27 @@ bpf_program__collect_reloc(struct bpf_program *prog, GElf_Shdr *shdr, static int bpf_map_find_btf_info(struct bpf_map *map, const struct btf *btf) { struct bpf_map_def *def = &map->def; - __u32 key_type_id, value_type_id; + __u32 key_type_id = 0, value_type_id = 0; int ret; - ret = btf__get_map_kv_tids(btf, map->name, def->key_size, - def->value_size, &key_type_id, - &value_type_id); - if (ret) + if (!bpf_map__is_internal(map)) { + ret = btf__get_map_kv_tids(btf, map->name, def->key_size, + def->value_size, &key_type_id, + &value_type_id); + } else { + /* + * LLVM annotates global data differently in BTF, that is, + * only as '.data', '.bss' or '.rodata'. + */ + ret = btf__find_by_name(btf, + libbpf_type_to_btf_name[map->libbpf_type]); + } + if (ret < 0) return ret; map->btf_key_type_id = key_type_id; - map->btf_value_type_id = value_type_id; - + map->btf_value_type_id = bpf_map__is_internal(map) ? + ret : value_type_id; return 0; } @@ -1181,10 +1510,96 @@ bpf_object__probe_name(struct bpf_object *obj) return 0; } +static int +bpf_object__probe_global_data(struct bpf_object *obj) +{ + struct bpf_load_program_attr prg_attr; + struct bpf_create_map_attr map_attr; + char *cp, errmsg[STRERR_BUFSIZE]; + struct bpf_insn insns[] = { + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 16), + BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 42), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }; + int ret, map; + + memset(&map_attr, 0, sizeof(map_attr)); + map_attr.map_type = BPF_MAP_TYPE_ARRAY; + map_attr.key_size = sizeof(int); + map_attr.value_size = 32; + map_attr.max_entries = 1; + + map = bpf_create_map_xattr(&map_attr); + if (map < 0) { + cp = libbpf_strerror_r(errno, errmsg, sizeof(errmsg)); + pr_warning("Error in %s():%s(%d). Couldn't create simple array map.\n", + __func__, cp, errno); + return -errno; + } + + insns[0].imm = map; + + memset(&prg_attr, 0, sizeof(prg_attr)); + prg_attr.prog_type = BPF_PROG_TYPE_SOCKET_FILTER; + prg_attr.insns = insns; + prg_attr.insns_cnt = ARRAY_SIZE(insns); + prg_attr.license = "GPL"; + + ret = bpf_load_program_xattr(&prg_attr, NULL, 0); + if (ret >= 0) { + obj->caps.global_data = 1; + close(ret); + } + + close(map); + return 0; +} + static int bpf_object__probe_caps(struct bpf_object *obj) { - return bpf_object__probe_name(obj); + int (*probe_fn[])(struct bpf_object *obj) = { + bpf_object__probe_name, + bpf_object__probe_global_data, + }; + int i, ret; + + for (i = 0; i < ARRAY_SIZE(probe_fn); i++) { + ret = probe_fn[i](obj); + if (ret < 0) + return ret; + } + + return 0; +} + +static int +bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map) +{ + char *cp, errmsg[STRERR_BUFSIZE]; + int err, zero = 0; + __u8 *data; + + /* Nothing to do here since kernel already zero-initializes .bss map. */ + if (map->libbpf_type == LIBBPF_MAP_BSS) + return 0; + + data = map->libbpf_type == LIBBPF_MAP_DATA ? + obj->sections.data : obj->sections.rodata; + + err = bpf_map_update_elem(map->fd, &zero, data, 0); + /* Freeze .rodata map as read-only from syscall side. */ + if (!err && map->libbpf_type == LIBBPF_MAP_RODATA) { + err = bpf_map_freeze(map->fd); + if (err) { + cp = libbpf_strerror_r(errno, errmsg, sizeof(errmsg)); + pr_warning("Error freezing map(%s) as read-only: %s\n", + map->name, cp); + err = 0; + } + } + return err; } static int @@ -1244,6 +1659,7 @@ bpf_object__create_maps(struct bpf_object *obj) size_t j; err = *pfd; +err_out: cp = libbpf_strerror_r(errno, errmsg, sizeof(errmsg)); pr_warning("failed to create map (name: '%s'): %s\n", map->name, cp); @@ -1251,6 +1667,15 @@ bpf_object__create_maps(struct bpf_object *obj) zclose(obj->maps[j].fd); return err; } + + if (bpf_map__is_internal(map)) { + err = bpf_object__populate_internal_map(obj, map); + if (err < 0) { + zclose(*pfd); + goto err_out; + } + } + pr_debug("create map %s: fd=%d\n", map->name, *pfd); } @@ -1405,21 +1830,29 @@ bpf_program__relocate(struct bpf_program *prog, struct bpf_object *obj) return 0; for (i = 0; i < prog->nr_reloc; i++) { - if (prog->reloc_desc[i].type == RELO_LD64) { + if (prog->reloc_desc[i].type == RELO_LD64 || + prog->reloc_desc[i].type == RELO_DATA) { + bool relo_data = prog->reloc_desc[i].type == RELO_DATA; struct bpf_insn *insns = prog->insns; int insn_idx, map_idx; insn_idx = prog->reloc_desc[i].insn_idx; map_idx = prog->reloc_desc[i].map_idx; - if (insn_idx >= (int)prog->insns_cnt) { + if (insn_idx + 1 >= (int)prog->insns_cnt) { pr_warning("relocation out of range: '%s'\n", prog->section_name); return -LIBBPF_ERRNO__RELOC; } - insns[insn_idx].src_reg = BPF_PSEUDO_MAP_FD; + + if (!relo_data) { + insns[insn_idx].src_reg = BPF_PSEUDO_MAP_FD; + } else { + insns[insn_idx].src_reg = BPF_PSEUDO_MAP_VALUE; + insns[insn_idx + 1].imm = insns[insn_idx].imm; + } insns[insn_idx].imm = obj->maps[map_idx].fd; - } else { + } else if (prog->reloc_desc[i].type == RELO_CALL) { err = bpf_program__reloc_text(prog, obj, &prog->reloc_desc[i]); if (err) @@ -1494,6 +1927,7 @@ load_program(struct bpf_program *prog, struct bpf_insn *insns, int insns_cnt, { struct bpf_load_program_attr load_attr; char *cp, errmsg[STRERR_BUFSIZE]; + int log_buf_size = BPF_LOG_BUF_SIZE; char *log_buf; int ret; @@ -1514,21 +1948,30 @@ load_program(struct bpf_program *prog, struct bpf_insn *insns, int insns_cnt, load_attr.line_info = prog->line_info; load_attr.line_info_rec_size = prog->line_info_rec_size; load_attr.line_info_cnt = prog->line_info_cnt; + load_attr.log_level = prog->log_level; if (!load_attr.insns || !load_attr.insns_cnt) return -EINVAL; - log_buf = malloc(BPF_LOG_BUF_SIZE); +retry_load: + log_buf = malloc(log_buf_size); if (!log_buf) pr_warning("Alloc log buffer for bpf loader error, continue without log\n"); - ret = bpf_load_program_xattr(&load_attr, log_buf, BPF_LOG_BUF_SIZE); + ret = bpf_load_program_xattr(&load_attr, log_buf, log_buf_size); if (ret >= 0) { + if (load_attr.log_level) + pr_debug("verifier log:\n%s", log_buf); *pfd = ret; ret = 0; goto out; } + if (errno == ENOSPC) { + log_buf_size <<= 1; + free(log_buf); + goto retry_load; + } ret = -LIBBPF_ERRNO__LOAD; cp = libbpf_strerror_r(errno, errmsg, sizeof(errmsg)); pr_warning("load bpf program failed: %s\n", cp); @@ -1693,7 +2136,9 @@ static bool bpf_prog_type__needs_kver(enum bpf_prog_type type) case BPF_PROG_TYPE_UNSPEC: case BPF_PROG_TYPE_TRACEPOINT: case BPF_PROG_TYPE_RAW_TRACEPOINT: + case BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE: case BPF_PROG_TYPE_PERF_EVENT: + case BPF_PROG_TYPE_CGROUP_SYSCTL: return false; case BPF_PROG_TYPE_KPROBE: default: @@ -1729,6 +2174,7 @@ __bpf_object__open(const char *path, void *obj_buf, size_t obj_buf_sz, CHECK_ERR(bpf_object__elf_init(obj), err, out); CHECK_ERR(bpf_object__check_endianness(obj), err, out); + CHECK_ERR(bpf_object__probe_caps(obj), err, out); CHECK_ERR(bpf_object__elf_collect(obj, flags), err, out); CHECK_ERR(bpf_object__collect_reloc(obj), err, out); CHECK_ERR(bpf_object__validate(obj, needs_kver), err, out); @@ -1822,7 +2268,6 @@ int bpf_object__load(struct bpf_object *obj) obj->loaded = true; - CHECK_ERR(bpf_object__probe_caps(obj), err, out); CHECK_ERR(bpf_object__create_maps(obj), err, out); CHECK_ERR(bpf_object__relocate(obj), err, out); CHECK_ERR(bpf_object__load_progs(obj), err, out); @@ -2303,6 +2748,9 @@ void bpf_object__close(struct bpf_object *obj) obj->maps[i].priv = NULL; obj->maps[i].clear_priv = NULL; } + + zfree(&obj->sections.rodata); + zfree(&obj->sections.data); zfree(&obj->maps); obj->nr_maps = 0; @@ -2631,6 +3079,8 @@ static const struct { BPF_CGROUP_UDP4_SENDMSG), BPF_EAPROG_SEC("cgroup/sendmsg6", BPF_PROG_TYPE_CGROUP_SOCK_ADDR, BPF_CGROUP_UDP6_SENDMSG), + BPF_EAPROG_SEC("cgroup/sysctl", BPF_PROG_TYPE_CGROUP_SYSCTL, + BPF_CGROUP_SYSCTL), }; #undef BPF_PROG_SEC_IMPL @@ -2780,6 +3230,11 @@ bool bpf_map__is_offload_neutral(struct bpf_map *map) return map->def.type == BPF_MAP_TYPE_PERF_EVENT_ARRAY; } +bool bpf_map__is_internal(struct bpf_map *map) +{ + return map->libbpf_type != LIBBPF_MAP_UNSPEC; +} + void bpf_map__set_ifindex(struct bpf_map *map, __u32 ifindex) { map->map_ifindex = ifindex; @@ -2938,6 +3393,7 @@ int bpf_prog_load_xattr(const struct bpf_prog_load_attr *attr, bpf_program__set_expected_attach_type(prog, expected_attach_type); + prog->log_level = attr->log_level; if (!first_prog) first_prog = prog; } diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h index c70785cc8ef560165e323abcf9e83f8cc05eec31..c5ff00515ce7d606a1c68ba119cdc091a3731e00 100644 --- a/tools/lib/bpf/libbpf.h +++ b/tools/lib/bpf/libbpf.h @@ -75,6 +75,10 @@ struct bpf_object *__bpf_object__open_xattr(struct bpf_object_open_attr *attr, LIBBPF_API struct bpf_object *bpf_object__open_buffer(void *obj_buf, size_t obj_buf_sz, const char *name); +int bpf_object__section_size(const struct bpf_object *obj, const char *name, + __u32 *size); +int bpf_object__variable_offset(const struct bpf_object *obj, const char *name, + __u32 *off); LIBBPF_API int bpf_object__pin_maps(struct bpf_object *obj, const char *path); LIBBPF_API int bpf_object__unpin_maps(struct bpf_object *obj, const char *path); @@ -301,6 +305,7 @@ LIBBPF_API void *bpf_map__priv(struct bpf_map *map); LIBBPF_API int bpf_map__reuse_fd(struct bpf_map *map, int fd); LIBBPF_API int bpf_map__resize(struct bpf_map *map, __u32 max_entries); LIBBPF_API bool bpf_map__is_offload_neutral(struct bpf_map *map); +LIBBPF_API bool bpf_map__is_internal(struct bpf_map *map); LIBBPF_API void bpf_map__set_ifindex(struct bpf_map *map, __u32 ifindex); LIBBPF_API int bpf_map__pin(struct bpf_map *map, const char *path); LIBBPF_API int bpf_map__unpin(struct bpf_map *map, const char *path); @@ -314,6 +319,7 @@ struct bpf_prog_load_attr { enum bpf_prog_type prog_type; enum bpf_attach_type expected_attach_type; int ifindex; + int log_level; }; LIBBPF_API int bpf_prog_load_xattr(const struct bpf_prog_load_attr *attr, diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map index f3ce50500cf2985edae7202b5dbf09b63c402d3f..673001787cba26f32572ae2e8dc2dc0a8bc9663f 100644 --- a/tools/lib/bpf/libbpf.map +++ b/tools/lib/bpf/libbpf.map @@ -157,3 +157,10 @@ LIBBPF_0.0.2 { bpf_program__bpil_addr_to_offs; bpf_program__bpil_offs_to_addr; } LIBBPF_0.0.1; + +LIBBPF_0.0.3 { + global: + bpf_map__is_internal; + bpf_map_freeze; + btf__finalize_data; +} LIBBPF_0.0.2; diff --git a/tools/lib/bpf/libbpf.pc.template b/tools/lib/bpf/libbpf.pc.template new file mode 100644 index 0000000000000000000000000000000000000000..ac17fcef2108ec81dc8800d97e3cd4f6c127b253 --- /dev/null +++ b/tools/lib/bpf/libbpf.pc.template @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) + +prefix=@PREFIX@ +libdir=@LIBDIR@ +includedir=${prefix}/include + +Name: libbpf +Description: BPF library +Version: @VERSION@ +Libs: -L${libdir} -lbpf +Requires.private: libelf +Cflags: -I${includedir} diff --git a/tools/lib/bpf/libbpf_probes.c b/tools/lib/bpf/libbpf_probes.c index 8c3a1c04dcb23bee42bda6cdbca5f82982b14407..a2c64a9ce1a63cf40306d7a70f6f607624dd8de1 100644 --- a/tools/lib/bpf/libbpf_probes.c +++ b/tools/lib/bpf/libbpf_probes.c @@ -9,6 +9,7 @@ #include #include +#include #include #include @@ -93,10 +94,12 @@ probe_load(enum bpf_prog_type prog_type, const struct bpf_insn *insns, case BPF_PROG_TYPE_CGROUP_DEVICE: case BPF_PROG_TYPE_SK_MSG: case BPF_PROG_TYPE_RAW_TRACEPOINT: + case BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE: case BPF_PROG_TYPE_LWT_SEG6LOCAL: case BPF_PROG_TYPE_LIRC_MODE2: case BPF_PROG_TYPE_SK_REUSEPORT: case BPF_PROG_TYPE_FLOW_DISSECTOR: + case BPF_PROG_TYPE_CGROUP_SYSCTL: default: break; } @@ -129,11 +132,65 @@ bool bpf_probe_prog_type(enum bpf_prog_type prog_type, __u32 ifindex) return errno != EINVAL && errno != EOPNOTSUPP; } +static int load_btf(void) +{ +#define BTF_INFO_ENC(kind, kind_flag, vlen) \ + ((!!(kind_flag) << 31) | ((kind) << 24) | ((vlen) & BTF_MAX_VLEN)) +#define BTF_TYPE_ENC(name, info, size_or_type) \ + (name), (info), (size_or_type) +#define BTF_INT_ENC(encoding, bits_offset, nr_bits) \ + ((encoding) << 24 | (bits_offset) << 16 | (nr_bits)) +#define BTF_TYPE_INT_ENC(name, encoding, bits_offset, bits, sz) \ + BTF_TYPE_ENC(name, BTF_INFO_ENC(BTF_KIND_INT, 0, 0), sz), \ + BTF_INT_ENC(encoding, bits_offset, bits) +#define BTF_MEMBER_ENC(name, type, bits_offset) \ + (name), (type), (bits_offset) + + const char btf_str_sec[] = "\0bpf_spin_lock\0val\0cnt\0l"; + /* struct bpf_spin_lock { + * int val; + * }; + * struct val { + * int cnt; + * struct bpf_spin_lock l; + * }; + */ + __u32 btf_raw_types[] = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + /* struct bpf_spin_lock */ /* [2] */ + BTF_TYPE_ENC(1, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 1), 4), + BTF_MEMBER_ENC(15, 1, 0), /* int val; */ + /* struct val */ /* [3] */ + BTF_TYPE_ENC(15, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 2), 8), + BTF_MEMBER_ENC(19, 1, 0), /* int cnt; */ + BTF_MEMBER_ENC(23, 2, 32),/* struct bpf_spin_lock l; */ + }; + struct btf_header btf_hdr = { + .magic = BTF_MAGIC, + .version = BTF_VERSION, + .hdr_len = sizeof(struct btf_header), + .type_len = sizeof(btf_raw_types), + .str_off = sizeof(btf_raw_types), + .str_len = sizeof(btf_str_sec), + }; + __u8 raw_btf[sizeof(struct btf_header) + sizeof(btf_raw_types) + + sizeof(btf_str_sec)]; + + memcpy(raw_btf, &btf_hdr, sizeof(btf_hdr)); + memcpy(raw_btf + sizeof(btf_hdr), btf_raw_types, sizeof(btf_raw_types)); + memcpy(raw_btf + sizeof(btf_hdr) + sizeof(btf_raw_types), + btf_str_sec, sizeof(btf_str_sec)); + + return bpf_load_btf(raw_btf, sizeof(raw_btf), 0, 0, 0); +} + bool bpf_probe_map_type(enum bpf_map_type map_type, __u32 ifindex) { int key_size, value_size, max_entries, map_flags; + __u32 btf_key_type_id = 0, btf_value_type_id = 0; struct bpf_create_map_attr attr = {}; - int fd = -1, fd_inner; + int fd = -1, btf_fd = -1, fd_inner; key_size = sizeof(__u32); value_size = sizeof(__u32); @@ -159,6 +216,16 @@ bool bpf_probe_map_type(enum bpf_map_type map_type, __u32 ifindex) case BPF_MAP_TYPE_STACK: key_size = 0; break; + case BPF_MAP_TYPE_SK_STORAGE: + btf_key_type_id = 1; + btf_value_type_id = 3; + value_size = 8; + max_entries = 0; + map_flags = BPF_F_NO_PREALLOC; + btf_fd = load_btf(); + if (btf_fd < 0) + return false; + break; case BPF_MAP_TYPE_UNSPEC: case BPF_MAP_TYPE_HASH: case BPF_MAP_TYPE_ARRAY: @@ -204,11 +271,18 @@ bool bpf_probe_map_type(enum bpf_map_type map_type, __u32 ifindex) attr.max_entries = max_entries; attr.map_flags = map_flags; attr.map_ifindex = ifindex; + if (btf_fd >= 0) { + attr.btf_fd = btf_fd; + attr.btf_key_type_id = btf_key_type_id; + attr.btf_value_type_id = btf_value_type_id; + } fd = bpf_create_map_xattr(&attr); } if (fd >= 0) close(fd); + if (btf_fd >= 0) + close(btf_fd); return fd >= 0; } diff --git a/tools/lib/bpf/libbpf_util.h b/tools/lib/bpf/libbpf_util.h index 81ecda0cb9c92307a4d8235c715497087aa819ce..da94c4cb2e4db5094e9b3e3a91ae05f8b1d9bbf6 100644 --- a/tools/lib/bpf/libbpf_util.h +++ b/tools/lib/bpf/libbpf_util.h @@ -23,6 +23,36 @@ do { \ #define pr_info(fmt, ...) __pr(LIBBPF_INFO, fmt, ##__VA_ARGS__) #define pr_debug(fmt, ...) __pr(LIBBPF_DEBUG, fmt, ##__VA_ARGS__) +/* Use these barrier functions instead of smp_[rw]mb() when they are + * used in a libbpf header file. That way they can be built into the + * application that uses libbpf. + */ +#if defined(__i386__) || defined(__x86_64__) +# define libbpf_smp_rmb() asm volatile("" : : : "memory") +# define libbpf_smp_wmb() asm volatile("" : : : "memory") +# define libbpf_smp_mb() \ + asm volatile("lock; addl $0,-4(%%rsp)" : : : "memory", "cc") +/* Hinders stores to be observed before older loads. */ +# define libbpf_smp_rwmb() asm volatile("" : : : "memory") +#elif defined(__aarch64__) +# define libbpf_smp_rmb() asm volatile("dmb ishld" : : : "memory") +# define libbpf_smp_wmb() asm volatile("dmb ishst" : : : "memory") +# define libbpf_smp_mb() asm volatile("dmb ish" : : : "memory") +# define libbpf_smp_rwmb() libbpf_smp_mb() +#elif defined(__arm__) +/* These are only valid for armv7 and above */ +# define libbpf_smp_rmb() asm volatile("dmb ish" : : : "memory") +# define libbpf_smp_wmb() asm volatile("dmb ishst" : : : "memory") +# define libbpf_smp_mb() asm volatile("dmb ish" : : : "memory") +# define libbpf_smp_rwmb() libbpf_smp_mb() +#else +/* Architecture missing native barrier functions. */ +# define libbpf_smp_rmb() __sync_synchronize() +# define libbpf_smp_wmb() __sync_synchronize() +# define libbpf_smp_mb() __sync_synchronize() +# define libbpf_smp_rwmb() __sync_synchronize() +#endif + #ifdef __cplusplus } /* extern "C" */ #endif diff --git a/tools/lib/bpf/xsk.c b/tools/lib/bpf/xsk.c index 8d0078b65486f45730f3d967c84a6709afb23cdb..a3d1a302bc9cb36788612c2033bb667737aee455 100644 --- a/tools/lib/bpf/xsk.c +++ b/tools/lib/bpf/xsk.c @@ -248,8 +248,7 @@ int xsk_umem__create(struct xsk_umem **umem_ptr, void *umem_area, __u64 size, return 0; out_mmap: - munmap(umem->fill, - off.fr.desc + umem->config.fill_size * sizeof(__u64)); + munmap(map, off.fr.desc + umem->config.fill_size * sizeof(__u64)); out_socket: close(umem->fd); out_umem_alloc: @@ -259,7 +258,8 @@ out_umem_alloc: static int xsk_load_xdp_prog(struct xsk_socket *xsk) { - char bpf_log_buf[BPF_LOG_BUF_SIZE]; + static const int log_buf_size = 16 * 1024; + char log_buf[log_buf_size]; int err, prog_fd; /* This is the C-program: @@ -308,10 +308,10 @@ static int xsk_load_xdp_prog(struct xsk_socket *xsk) size_t insns_cnt = sizeof(prog) / sizeof(struct bpf_insn); prog_fd = bpf_load_program(BPF_PROG_TYPE_XDP, prog, insns_cnt, - "LGPL-2.1 or BSD-2-Clause", 0, bpf_log_buf, - BPF_LOG_BUF_SIZE); + "LGPL-2.1 or BSD-2-Clause", 0, log_buf, + log_buf_size); if (prog_fd < 0) { - pr_warning("BPF log buffer:\n%s", bpf_log_buf); + pr_warning("BPF log buffer:\n%s", log_buf); return prog_fd; } @@ -387,21 +387,17 @@ static void xsk_delete_bpf_maps(struct xsk_socket *xsk) { close(xsk->qidconf_map_fd); close(xsk->xsks_map_fd); + xsk->qidconf_map_fd = -1; + xsk->xsks_map_fd = -1; } -static int xsk_update_bpf_maps(struct xsk_socket *xsk, int qidconf_value, - int xsks_value) +static int xsk_lookup_bpf_maps(struct xsk_socket *xsk) { - bool qidconf_map_updated = false, xsks_map_updated = false; + __u32 i, *map_ids, num_maps, prog_len = sizeof(struct bpf_prog_info); + __u32 map_len = sizeof(struct bpf_map_info); struct bpf_prog_info prog_info = {}; - __u32 prog_len = sizeof(prog_info); struct bpf_map_info map_info; - __u32 map_len = sizeof(map_info); - __u32 *map_ids; - int reset_value = 0; - __u32 num_maps; - unsigned int i; - int err; + int fd, err; err = bpf_obj_get_info_by_fd(xsk->prog_fd, &prog_info, &prog_len); if (err) @@ -422,66 +418,71 @@ static int xsk_update_bpf_maps(struct xsk_socket *xsk, int qidconf_value, goto out_map_ids; for (i = 0; i < prog_info.nr_map_ids; i++) { - int fd; + if (xsk->qidconf_map_fd != -1 && xsk->xsks_map_fd != -1) + break; fd = bpf_map_get_fd_by_id(map_ids[i]); - if (fd < 0) { - err = -errno; - goto out_maps; - } + if (fd < 0) + continue; err = bpf_obj_get_info_by_fd(fd, &map_info, &map_len); - if (err) - goto out_maps; + if (err) { + close(fd); + continue; + } if (!strcmp(map_info.name, "qidconf_map")) { - err = bpf_map_update_elem(fd, &xsk->queue_id, - &qidconf_value, 0); - if (err) - goto out_maps; - qidconf_map_updated = true; xsk->qidconf_map_fd = fd; - } else if (!strcmp(map_info.name, "xsks_map")) { - err = bpf_map_update_elem(fd, &xsk->queue_id, - &xsks_value, 0); - if (err) - goto out_maps; - xsks_map_updated = true; + continue; + } + + if (!strcmp(map_info.name, "xsks_map")) { xsk->xsks_map_fd = fd; + continue; } - if (qidconf_map_updated && xsks_map_updated) - break; + close(fd); } - if (!(qidconf_map_updated && xsks_map_updated)) { + err = 0; + if (xsk->qidconf_map_fd < 0 || xsk->xsks_map_fd < 0) { err = -ENOENT; - goto out_maps; + xsk_delete_bpf_maps(xsk); } - err = 0; - goto out_success; - -out_maps: - if (qidconf_map_updated) - (void)bpf_map_update_elem(xsk->qidconf_map_fd, &xsk->queue_id, - &reset_value, 0); - if (xsks_map_updated) - (void)bpf_map_update_elem(xsk->xsks_map_fd, &xsk->queue_id, - &reset_value, 0); -out_success: - if (qidconf_map_updated) - close(xsk->qidconf_map_fd); - if (xsks_map_updated) - close(xsk->xsks_map_fd); out_map_ids: free(map_ids); return err; } +static void xsk_clear_bpf_maps(struct xsk_socket *xsk) +{ + int qid = false; + + bpf_map_update_elem(xsk->qidconf_map_fd, &xsk->queue_id, &qid, 0); + bpf_map_delete_elem(xsk->xsks_map_fd, &xsk->queue_id); +} + +static int xsk_set_bpf_maps(struct xsk_socket *xsk) +{ + int qid = true, fd = xsk->fd, err; + + err = bpf_map_update_elem(xsk->qidconf_map_fd, &xsk->queue_id, &qid, 0); + if (err) + goto out; + + err = bpf_map_update_elem(xsk->xsks_map_fd, &xsk->queue_id, &fd, 0); + if (err) + goto out; + + return 0; +out: + xsk_clear_bpf_maps(xsk); + return err; +} + static int xsk_setup_xdp_prog(struct xsk_socket *xsk) { - bool prog_attached = false; __u32 prog_id = 0; int err; @@ -491,7 +492,6 @@ static int xsk_setup_xdp_prog(struct xsk_socket *xsk) return err; if (!prog_id) { - prog_attached = true; err = xsk_create_bpf_maps(xsk); if (err) return err; @@ -501,20 +501,21 @@ static int xsk_setup_xdp_prog(struct xsk_socket *xsk) goto out_maps; } else { xsk->prog_fd = bpf_prog_get_fd_by_id(prog_id); + err = xsk_lookup_bpf_maps(xsk); + if (err) + goto out_load; } - err = xsk_update_bpf_maps(xsk, true, xsk->fd); + err = xsk_set_bpf_maps(xsk); if (err) goto out_load; return 0; out_load: - if (prog_attached) - close(xsk->prog_fd); + close(xsk->prog_fd); out_maps: - if (prog_attached) - xsk_delete_bpf_maps(xsk); + xsk_delete_bpf_maps(xsk); return err; } @@ -523,11 +524,11 @@ int xsk_socket__create(struct xsk_socket **xsk_ptr, const char *ifname, struct xsk_ring_cons *rx, struct xsk_ring_prod *tx, const struct xsk_socket_config *usr_config) { + void *rx_map = NULL, *tx_map = NULL; struct sockaddr_xdp sxdp = {}; struct xdp_mmap_offsets off; struct xsk_socket *xsk; socklen_t optlen; - void *map; int err; if (!umem || !xsk_ptr || !rx || !tx) @@ -593,40 +594,40 @@ int xsk_socket__create(struct xsk_socket **xsk_ptr, const char *ifname, } if (rx) { - map = xsk_mmap(NULL, off.rx.desc + - xsk->config.rx_size * sizeof(struct xdp_desc), - PROT_READ | PROT_WRITE, - MAP_SHARED | MAP_POPULATE, - xsk->fd, XDP_PGOFF_RX_RING); - if (map == MAP_FAILED) { + rx_map = xsk_mmap(NULL, off.rx.desc + + xsk->config.rx_size * sizeof(struct xdp_desc), + PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_POPULATE, + xsk->fd, XDP_PGOFF_RX_RING); + if (rx_map == MAP_FAILED) { err = -errno; goto out_socket; } rx->mask = xsk->config.rx_size - 1; rx->size = xsk->config.rx_size; - rx->producer = map + off.rx.producer; - rx->consumer = map + off.rx.consumer; - rx->ring = map + off.rx.desc; + rx->producer = rx_map + off.rx.producer; + rx->consumer = rx_map + off.rx.consumer; + rx->ring = rx_map + off.rx.desc; } xsk->rx = rx; if (tx) { - map = xsk_mmap(NULL, off.tx.desc + - xsk->config.tx_size * sizeof(struct xdp_desc), - PROT_READ | PROT_WRITE, - MAP_SHARED | MAP_POPULATE, - xsk->fd, XDP_PGOFF_TX_RING); - if (map == MAP_FAILED) { + tx_map = xsk_mmap(NULL, off.tx.desc + + xsk->config.tx_size * sizeof(struct xdp_desc), + PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_POPULATE, + xsk->fd, XDP_PGOFF_TX_RING); + if (tx_map == MAP_FAILED) { err = -errno; goto out_mmap_rx; } tx->mask = xsk->config.tx_size - 1; tx->size = xsk->config.tx_size; - tx->producer = map + off.tx.producer; - tx->consumer = map + off.tx.consumer; - tx->ring = map + off.tx.desc; + tx->producer = tx_map + off.tx.producer; + tx->consumer = tx_map + off.tx.consumer; + tx->ring = tx_map + off.tx.desc; tx->cached_cons = xsk->config.tx_size; } xsk->tx = tx; @@ -642,6 +643,9 @@ int xsk_socket__create(struct xsk_socket **xsk_ptr, const char *ifname, goto out_mmap_tx; } + xsk->qidconf_map_fd = -1; + xsk->xsks_map_fd = -1; + if (!(xsk->config.libbpf_flags & XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD)) { err = xsk_setup_xdp_prog(xsk); if (err) @@ -653,13 +657,11 @@ int xsk_socket__create(struct xsk_socket **xsk_ptr, const char *ifname, out_mmap_tx: if (tx) - munmap(xsk->tx, - off.tx.desc + + munmap(tx_map, off.tx.desc + xsk->config.tx_size * sizeof(struct xdp_desc)); out_mmap_rx: if (rx) - munmap(xsk->rx, - off.rx.desc + + munmap(rx_map, off.rx.desc + xsk->config.rx_size * sizeof(struct xdp_desc)); out_socket: if (--umem->refcount) @@ -684,9 +686,9 @@ int xsk_umem__delete(struct xsk_umem *umem) optlen = sizeof(off); err = getsockopt(umem->fd, SOL_XDP, XDP_MMAP_OFFSETS, &off, &optlen); if (!err) { - munmap(umem->fill->ring, + munmap(umem->fill->ring - off.fr.desc, off.fr.desc + umem->config.fill_size * sizeof(__u64)); - munmap(umem->comp->ring, + munmap(umem->comp->ring - off.cr.desc, off.cr.desc + umem->config.comp_size * sizeof(__u64)); } @@ -698,6 +700,7 @@ int xsk_umem__delete(struct xsk_umem *umem) void xsk_socket__delete(struct xsk_socket *xsk) { + size_t desc_sz = sizeof(struct xdp_desc); struct xdp_mmap_offsets off; socklen_t optlen; int err; @@ -705,19 +708,21 @@ void xsk_socket__delete(struct xsk_socket *xsk) if (!xsk) return; - (void)xsk_update_bpf_maps(xsk, 0, 0); + xsk_clear_bpf_maps(xsk); + xsk_delete_bpf_maps(xsk); optlen = sizeof(off); err = getsockopt(xsk->fd, SOL_XDP, XDP_MMAP_OFFSETS, &off, &optlen); if (!err) { - if (xsk->rx) - munmap(xsk->rx->ring, - off.rx.desc + - xsk->config.rx_size * sizeof(struct xdp_desc)); - if (xsk->tx) - munmap(xsk->tx->ring, - off.tx.desc + - xsk->config.tx_size * sizeof(struct xdp_desc)); + if (xsk->rx) { + munmap(xsk->rx->ring - off.rx.desc, + off.rx.desc + xsk->config.rx_size * desc_sz); + } + if (xsk->tx) { + munmap(xsk->tx->ring - off.tx.desc, + off.tx.desc + xsk->config.tx_size * desc_sz); + } + } xsk->umem->refcount--; diff --git a/tools/lib/bpf/xsk.h b/tools/lib/bpf/xsk.h index a497f00e29628e39d7104d5cc6003212f2f50d17..82ea71a0f3ec639e65aa938db6149d9e32c55f72 100644 --- a/tools/lib/bpf/xsk.h +++ b/tools/lib/bpf/xsk.h @@ -16,6 +16,7 @@ #include #include "libbpf.h" +#include "libbpf_util.h" #ifdef __cplusplus extern "C" { @@ -36,6 +37,10 @@ struct name { \ DEFINE_XSK_RING(xsk_ring_prod); DEFINE_XSK_RING(xsk_ring_cons); +/* For a detailed explanation on the memory barriers associated with the + * ring, please take a look at net/xdp/xsk_queue.h. + */ + struct xsk_umem; struct xsk_socket; @@ -105,7 +110,7 @@ static inline __u32 xsk_cons_nb_avail(struct xsk_ring_cons *r, __u32 nb) static inline size_t xsk_ring_prod__reserve(struct xsk_ring_prod *prod, size_t nb, __u32 *idx) { - if (unlikely(xsk_prod_nb_free(prod, nb) < nb)) + if (xsk_prod_nb_free(prod, nb) < nb) return 0; *idx = prod->cached_prod; @@ -116,10 +121,10 @@ static inline size_t xsk_ring_prod__reserve(struct xsk_ring_prod *prod, static inline void xsk_ring_prod__submit(struct xsk_ring_prod *prod, size_t nb) { - /* Make sure everything has been written to the ring before signalling - * this to the kernel. + /* Make sure everything has been written to the ring before indicating + * this to the kernel by writing the producer pointer. */ - smp_wmb(); + libbpf_smp_wmb(); *prod->producer += nb; } @@ -129,11 +134,11 @@ static inline size_t xsk_ring_cons__peek(struct xsk_ring_cons *cons, { size_t entries = xsk_cons_nb_avail(cons, nb); - if (likely(entries > 0)) { + if (entries > 0) { /* Make sure we do not speculatively read the data before * we have received the packet buffers from the ring. */ - smp_rmb(); + libbpf_smp_rmb(); *idx = cons->cached_cons; cons->cached_cons += entries; @@ -144,6 +149,11 @@ static inline size_t xsk_ring_cons__peek(struct xsk_ring_cons *cons, static inline void xsk_ring_cons__release(struct xsk_ring_cons *cons, size_t nb) { + /* Make sure data has been read before indicating we are done + * with the entries by updating the consumer pointer. + */ + libbpf_smp_rwmb(); + *cons->consumer += nb; } diff --git a/tools/testing/selftests/bpf/.gitignore b/tools/testing/selftests/bpf/.gitignore index 3b74d23fffabe46e5fac0f78b4c28be5dce6d13c..41e8a689aa77744e91282bd14a98c9d16a20dc2a 100644 --- a/tools/testing/selftests/bpf/.gitignore +++ b/tools/testing/selftests/bpf/.gitignore @@ -30,4 +30,5 @@ test_netcnt test_section_names test_tcpnotify_user test_libbpf +test_tcp_check_syncookie_user alu32 diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 2aed37ea61a4cba8f2e871686d7a94590cac3326..66f2dca1dee19cf73596757d8b39eab7b3651687 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -23,7 +23,7 @@ TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_lpm_map test test_align test_verifier_log test_dev_cgroup test_tcpbpf_user \ test_sock test_btf test_sockmap test_lirc_mode2_user get_cgroup_id_user \ test_socket_cookie test_cgroup_storage test_select_reuseport test_section_names \ - test_netcnt test_tcpnotify_user test_sock_fields + test_netcnt test_tcpnotify_user test_sock_fields test_sysctl BPF_OBJ_FILES = $(patsubst %.c,%.o, $(notdir $(wildcard progs/*.c))) TEST_GEN_FILES = $(BPF_OBJ_FILES) @@ -51,7 +51,10 @@ TEST_PROGS := test_kmod.sh \ test_skb_cgroup_id.sh \ test_flow_dissector.sh \ test_xdp_vlan.sh \ - test_lwt_ip_encap.sh + test_lwt_ip_encap.sh \ + test_tcp_check_syncookie.sh \ + test_tc_tunnel.sh \ + test_tc_edt.sh TEST_PROGS_EXTENDED := with_addr.sh \ with_tunnels.sh \ @@ -60,7 +63,7 @@ TEST_PROGS_EXTENDED := with_addr.sh \ # Compile but not part of 'make run_tests' TEST_GEN_PROGS_EXTENDED = test_libbpf_open test_sock_addr test_skb_cgroup_id_user \ - flow_dissector_load test_flow_dissector + flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user include ../lib.mk @@ -69,7 +72,9 @@ TEST_CUSTOM_PROGS = $(OUTPUT)/urandom_read all: $(TEST_CUSTOM_PROGS) $(OUTPUT)/urandom_read: $(OUTPUT)/%: %.c - $(CC) -o $@ -static $< -Wl,--build-id + $(CC) -o $@ $< -Wl,--build-id + +$(OUTPUT)/test_maps: map_tests/*.c BPFOBJ := $(OUTPUT)/libbpf.a @@ -90,6 +95,7 @@ $(OUTPUT)/get_cgroup_id_user: cgroup_helpers.c $(OUTPUT)/test_cgroup_storage: cgroup_helpers.c $(OUTPUT)/test_netcnt: cgroup_helpers.c $(OUTPUT)/test_sock_fields: cgroup_helpers.c +$(OUTPUT)/test_sysctl: cgroup_helpers.c .PHONY: force @@ -206,7 +212,7 @@ ifeq ($(DWARF2BTF),y) endif PROG_TESTS_H := $(OUTPUT)/prog_tests/tests.h -$(OUTPUT)/test_progs: $(PROG_TESTS_H) +test_progs.c: $(PROG_TESTS_H) $(OUTPUT)/test_progs: CFLAGS += $(TEST_PROGS_CFLAGS) $(OUTPUT)/test_progs: prog_tests/*.c @@ -228,8 +234,29 @@ $(PROG_TESTS_H): $(PROG_TESTS_DIR) $(PROG_TESTS_FILES) echo '#endif' \ ) > $(PROG_TESTS_H)) +TEST_MAPS_CFLAGS := -I. -I$(OUTPUT) +MAP_TESTS_DIR = $(OUTPUT)/map_tests +$(MAP_TESTS_DIR): + mkdir -p $@ +MAP_TESTS_H := $(MAP_TESTS_DIR)/tests.h +test_maps.c: $(MAP_TESTS_H) +$(OUTPUT)/test_maps: CFLAGS += $(TEST_MAPS_CFLAGS) +MAP_TESTS_FILES := $(wildcard map_tests/*.c) +$(MAP_TESTS_H): $(MAP_TESTS_DIR) $(MAP_TESTS_FILES) + $(shell ( cd map_tests/; \ + echo '/* Generated header, do not edit */'; \ + echo '#ifdef DECLARE'; \ + ls *.c 2> /dev/null | \ + sed -e 's@\([^\.]*\)\.c@extern void test_\1(void);@'; \ + echo '#endif'; \ + echo '#ifdef CALL'; \ + ls *.c 2> /dev/null | \ + sed -e 's@\([^\.]*\)\.c@test_\1();@'; \ + echo '#endif' \ + ) > $(MAP_TESTS_H)) + VERIFIER_TESTS_H := $(OUTPUT)/verifier/tests.h -$(OUTPUT)/test_verifier: $(VERIFIER_TESTS_H) +test_verifier.c: $(VERIFIER_TESTS_H) $(OUTPUT)/test_verifier: CFLAGS += $(TEST_VERIFIER_CFLAGS) VERIFIER_TESTS_DIR = $(OUTPUT)/verifier @@ -247,4 +274,4 @@ $(OUTPUT)/verifier/tests.h: $(VERIFIER_TESTS_DIR) $(VERIFIER_TEST_FILES) ) > $(VERIFIER_TESTS_H)) EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(ALU32_BUILD_DIR) \ - $(VERIFIER_TESTS_H) $(PROG_TESTS_H) + $(VERIFIER_TESTS_H) $(PROG_TESTS_H) $(MAP_TESTS_H) diff --git a/tools/testing/selftests/bpf/bpf_helpers.h b/tools/testing/selftests/bpf/bpf_helpers.h index c81fc350f7ad46ad60d53ac3dd8121059020f9a6..6e80b66d7fb1d7a715782bdbffe74fbea005d0a6 100644 --- a/tools/testing/selftests/bpf/bpf_helpers.h +++ b/tools/testing/selftests/bpf/bpf_helpers.h @@ -9,14 +9,14 @@ #define SEC(NAME) __attribute__((section(NAME), used)) /* helper functions called from eBPF programs written in C */ -static void *(*bpf_map_lookup_elem)(void *map, void *key) = +static void *(*bpf_map_lookup_elem)(void *map, const void *key) = (void *) BPF_FUNC_map_lookup_elem; -static int (*bpf_map_update_elem)(void *map, void *key, void *value, +static int (*bpf_map_update_elem)(void *map, const void *key, const void *value, unsigned long long flags) = (void *) BPF_FUNC_map_update_elem; -static int (*bpf_map_delete_elem)(void *map, void *key) = +static int (*bpf_map_delete_elem)(void *map, const void *key) = (void *) BPF_FUNC_map_delete_elem; -static int (*bpf_map_push_elem)(void *map, void *value, +static int (*bpf_map_push_elem)(void *map, const void *value, unsigned long long flags) = (void *) BPF_FUNC_map_push_elem; static int (*bpf_map_pop_elem)(void *map, void *value) = @@ -159,6 +159,11 @@ static struct bpf_sock *(*bpf_sk_lookup_tcp)(void *ctx, int size, unsigned long long netns_id, unsigned long long flags) = (void *) BPF_FUNC_sk_lookup_tcp; +static struct bpf_sock *(*bpf_skc_lookup_tcp)(void *ctx, + struct bpf_sock_tuple *tuple, + int size, unsigned long long netns_id, + unsigned long long flags) = + (void *) BPF_FUNC_skc_lookup_tcp; static struct bpf_sock *(*bpf_sk_lookup_udp)(void *ctx, struct bpf_sock_tuple *tuple, int size, unsigned long long netns_id, @@ -184,6 +189,33 @@ static struct bpf_sock *(*bpf_get_listener_sock)(struct bpf_sock *sk) = (void *) BPF_FUNC_get_listener_sock; static int (*bpf_skb_ecn_set_ce)(void *ctx) = (void *) BPF_FUNC_skb_ecn_set_ce; +static int (*bpf_tcp_check_syncookie)(struct bpf_sock *sk, + void *ip, int ip_len, void *tcp, int tcp_len) = + (void *) BPF_FUNC_tcp_check_syncookie; +static int (*bpf_sysctl_get_name)(void *ctx, char *buf, + unsigned long long buf_len, + unsigned long long flags) = + (void *) BPF_FUNC_sysctl_get_name; +static int (*bpf_sysctl_get_current_value)(void *ctx, char *buf, + unsigned long long buf_len) = + (void *) BPF_FUNC_sysctl_get_current_value; +static int (*bpf_sysctl_get_new_value)(void *ctx, char *buf, + unsigned long long buf_len) = + (void *) BPF_FUNC_sysctl_get_new_value; +static int (*bpf_sysctl_set_new_value)(void *ctx, const char *buf, + unsigned long long buf_len) = + (void *) BPF_FUNC_sysctl_set_new_value; +static int (*bpf_strtol)(const char *buf, unsigned long long buf_len, + unsigned long long flags, long *res) = + (void *) BPF_FUNC_strtol; +static int (*bpf_strtoul)(const char *buf, unsigned long long buf_len, + unsigned long long flags, unsigned long *res) = + (void *) BPF_FUNC_strtoul; +static void *(*bpf_sk_storage_get)(void *map, struct bpf_sock *sk, + void *value, __u64 flags) = + (void *) BPF_FUNC_sk_storage_get; +static int (*bpf_sk_storage_delete)(void *map, struct bpf_sock *sk) = + (void *)BPF_FUNC_sk_storage_delete; /* llvm builtin functions that eBPF C program may use to * emit BPF_LD_ABS and BPF_LD_IND instructions @@ -274,6 +306,9 @@ static int (*bpf_skb_adjust_room)(void *ctx, __s32 len_diff, __u32 mode, #elif defined(__TARGET_ARCH_s930x) #define bpf_target_s930x #define bpf_target_defined +#elif defined(__TARGET_ARCH_arm) + #define bpf_target_arm + #define bpf_target_defined #elif defined(__TARGET_ARCH_arm64) #define bpf_target_arm64 #define bpf_target_defined @@ -296,6 +331,8 @@ static int (*bpf_skb_adjust_room)(void *ctx, __s32 len_diff, __u32 mode, #define bpf_target_x86 #elif defined(__s390x__) #define bpf_target_s930x +#elif defined(__arm__) + #define bpf_target_arm #elif defined(__aarch64__) #define bpf_target_arm64 #elif defined(__mips__) @@ -333,6 +370,19 @@ static int (*bpf_skb_adjust_room)(void *ctx, __s32 len_diff, __u32 mode, #define PT_REGS_SP(x) ((x)->gprs[15]) #define PT_REGS_IP(x) ((x)->psw.addr) +#elif defined(bpf_target_arm) + +#define PT_REGS_PARM1(x) ((x)->uregs[0]) +#define PT_REGS_PARM2(x) ((x)->uregs[1]) +#define PT_REGS_PARM3(x) ((x)->uregs[2]) +#define PT_REGS_PARM4(x) ((x)->uregs[3]) +#define PT_REGS_PARM5(x) ((x)->uregs[4]) +#define PT_REGS_RET(x) ((x)->uregs[14]) +#define PT_REGS_FP(x) ((x)->uregs[11]) /* Works only with CONFIG_FRAME_POINTER */ +#define PT_REGS_RC(x) ((x)->uregs[0]) +#define PT_REGS_SP(x) ((x)->uregs[13]) +#define PT_REGS_IP(x) ((x)->uregs[12]) + #elif defined(bpf_target_arm64) #define PT_REGS_PARM1(x) ((x)->regs[0]) diff --git a/tools/testing/selftests/bpf/config b/tools/testing/selftests/bpf/config index 37f947ec44ed91533489572b2bd053ded3f080cd..f7a0744db31e1bb3211e0b17ca73b7e57941e5b7 100644 --- a/tools/testing/selftests/bpf/config +++ b/tools/testing/selftests/bpf/config @@ -23,3 +23,14 @@ CONFIG_LWTUNNEL=y CONFIG_BPF_STREAM_PARSER=y CONFIG_XDP_SOCKETS=y CONFIG_FTRACE_SYSCALLS=y +CONFIG_IPV6_TUNNEL=y +CONFIG_IPV6_GRE=y +CONFIG_NET_FOU=m +CONFIG_NET_FOU_IP_TUNNELS=y +CONFIG_IPV6_FOU=m +CONFIG_IPV6_FOU_TUNNEL=m +CONFIG_MPLS=y +CONFIG_NET_MPLS_GSO=m +CONFIG_MPLS_ROUTING=m +CONFIG_MPLS_IPTUNNEL=m +CONFIG_IPV6_SIT=m diff --git a/tools/testing/selftests/bpf/flow_dissector_load.c b/tools/testing/selftests/bpf/flow_dissector_load.c index 77cafa66d0487840a81b02871d85f8d6b7082389..3fd83b9dc1bfe2db2e334f2eba8e010b1900e5cd 100644 --- a/tools/testing/selftests/bpf/flow_dissector_load.c +++ b/tools/testing/selftests/bpf/flow_dissector_load.c @@ -26,7 +26,7 @@ static void load_and_attach_program(void) struct bpf_object *obj; ret = bpf_flow_load(&obj, cfg_path_name, cfg_section_name, - cfg_map_name, &prog_fd); + cfg_map_name, NULL, &prog_fd, NULL); if (ret) error(1, 0, "bpf_flow_load %s", cfg_path_name); @@ -52,7 +52,7 @@ static void detach_program(void) sprintf(command, "rm -r %s", cfg_pin_path); ret = system(command); if (ret) - error(1, errno, command); + error(1, errno, "%s", command); } static void parse_opts(int argc, char **argv) diff --git a/tools/testing/selftests/bpf/flow_dissector_load.h b/tools/testing/selftests/bpf/flow_dissector_load.h index 41dd6959feb0daf34562cdf8ef344c966340d901..daeaeb5188940540bad2aaedc055a22bc6f550c7 100644 --- a/tools/testing/selftests/bpf/flow_dissector_load.h +++ b/tools/testing/selftests/bpf/flow_dissector_load.h @@ -9,10 +9,12 @@ static inline int bpf_flow_load(struct bpf_object **obj, const char *path, const char *section_name, const char *map_name, - int *prog_fd) + const char *keys_map_name, + int *prog_fd, + int *keys_fd) { struct bpf_program *prog, *main_prog; - struct bpf_map *prog_array; + struct bpf_map *prog_array, *keys; int prog_array_fd; int ret, fd, i; @@ -23,19 +25,29 @@ static inline int bpf_flow_load(struct bpf_object **obj, main_prog = bpf_object__find_program_by_title(*obj, section_name); if (!main_prog) - return ret; + return -1; *prog_fd = bpf_program__fd(main_prog); if (*prog_fd < 0) - return ret; + return -1; prog_array = bpf_object__find_map_by_name(*obj, map_name); if (!prog_array) - return ret; + return -1; prog_array_fd = bpf_map__fd(prog_array); if (prog_array_fd < 0) - return ret; + return -1; + + if (keys_map_name && keys_fd) { + keys = bpf_object__find_map_by_name(*obj, keys_map_name); + if (!keys) + return -1; + + *keys_fd = bpf_map__fd(keys); + if (*keys_fd < 0) + return -1; + } i = 0; bpf_object__for_each_program(prog, *obj) { diff --git a/tools/testing/selftests/bpf/map_tests/sk_storage_map.c b/tools/testing/selftests/bpf/map_tests/sk_storage_map.c new file mode 100644 index 0000000000000000000000000000000000000000..e569edc679d881944f8b61fbe8eeec58cabee032 --- /dev/null +++ b/tools/testing/selftests/bpf/map_tests/sk_storage_map.c @@ -0,0 +1,629 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2019 Facebook */ +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include +#include + +static struct bpf_create_map_attr xattr = { + .name = "sk_storage_map", + .map_type = BPF_MAP_TYPE_SK_STORAGE, + .map_flags = BPF_F_NO_PREALLOC, + .max_entries = 0, + .key_size = 4, + .value_size = 8, + .btf_key_type_id = 1, + .btf_value_type_id = 3, + .btf_fd = -1, +}; + +static unsigned int nr_sk_threads_done; +static unsigned int nr_sk_threads_err; +static unsigned int nr_sk_per_thread = 4096; +static unsigned int nr_sk_threads = 4; +static int sk_storage_map = -1; +static unsigned int stop; +static int runtime_s = 5; + +static bool is_stopped(void) +{ + return READ_ONCE(stop); +} + +static unsigned int threads_err(void) +{ + return READ_ONCE(nr_sk_threads_err); +} + +static void notify_thread_err(void) +{ + __sync_add_and_fetch(&nr_sk_threads_err, 1); +} + +static bool wait_for_threads_err(void) +{ + while (!is_stopped() && !threads_err()) + usleep(500); + + return !is_stopped(); +} + +static unsigned int threads_done(void) +{ + return READ_ONCE(nr_sk_threads_done); +} + +static void notify_thread_done(void) +{ + __sync_add_and_fetch(&nr_sk_threads_done, 1); +} + +static void notify_thread_redo(void) +{ + __sync_sub_and_fetch(&nr_sk_threads_done, 1); +} + +static bool wait_for_threads_done(void) +{ + while (threads_done() != nr_sk_threads && !is_stopped() && + !threads_err()) + usleep(50); + + return !is_stopped() && !threads_err(); +} + +static bool wait_for_threads_redo(void) +{ + while (threads_done() && !is_stopped() && !threads_err()) + usleep(50); + + return !is_stopped() && !threads_err(); +} + +static bool wait_for_map(void) +{ + while (READ_ONCE(sk_storage_map) == -1 && !is_stopped()) + usleep(50); + + return !is_stopped(); +} + +static bool wait_for_map_close(void) +{ + while (READ_ONCE(sk_storage_map) != -1 && !is_stopped()) + ; + + return !is_stopped(); +} + +static int load_btf(void) +{ + const char btf_str_sec[] = "\0bpf_spin_lock\0val\0cnt\0l"; + __u32 btf_raw_types[] = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + /* struct bpf_spin_lock */ /* [2] */ + BTF_TYPE_ENC(1, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 1), 4), + BTF_MEMBER_ENC(15, 1, 0), /* int val; */ + /* struct val */ /* [3] */ + BTF_TYPE_ENC(15, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 2), 8), + BTF_MEMBER_ENC(19, 1, 0), /* int cnt; */ + BTF_MEMBER_ENC(23, 2, 32),/* struct bpf_spin_lock l; */ + }; + struct btf_header btf_hdr = { + .magic = BTF_MAGIC, + .version = BTF_VERSION, + .hdr_len = sizeof(struct btf_header), + .type_len = sizeof(btf_raw_types), + .str_off = sizeof(btf_raw_types), + .str_len = sizeof(btf_str_sec), + }; + __u8 raw_btf[sizeof(struct btf_header) + sizeof(btf_raw_types) + + sizeof(btf_str_sec)]; + + memcpy(raw_btf, &btf_hdr, sizeof(btf_hdr)); + memcpy(raw_btf + sizeof(btf_hdr), btf_raw_types, sizeof(btf_raw_types)); + memcpy(raw_btf + sizeof(btf_hdr) + sizeof(btf_raw_types), + btf_str_sec, sizeof(btf_str_sec)); + + return bpf_load_btf(raw_btf, sizeof(raw_btf), 0, 0, 0); +} + +static int create_sk_storage_map(void) +{ + int btf_fd, map_fd; + + btf_fd = load_btf(); + CHECK(btf_fd == -1, "bpf_load_btf", "btf_fd:%d errno:%d\n", + btf_fd, errno); + xattr.btf_fd = btf_fd; + + map_fd = bpf_create_map_xattr(&xattr); + xattr.btf_fd = -1; + close(btf_fd); + CHECK(map_fd == -1, + "bpf_create_map_xattr()", "errno:%d\n", errno); + + return map_fd; +} + +static void *insert_close_thread(void *arg) +{ + struct { + int cnt; + int lock; + } value = { .cnt = 0xeB9F, .lock = 0, }; + int i, map_fd, err, *sk_fds; + + sk_fds = malloc(sizeof(*sk_fds) * nr_sk_per_thread); + if (!sk_fds) { + notify_thread_err(); + return ERR_PTR(-ENOMEM); + } + + for (i = 0; i < nr_sk_per_thread; i++) + sk_fds[i] = -1; + + while (!is_stopped()) { + if (!wait_for_map()) + goto close_all; + + map_fd = READ_ONCE(sk_storage_map); + for (i = 0; i < nr_sk_per_thread && !is_stopped(); i++) { + sk_fds[i] = socket(AF_INET6, SOCK_STREAM, 0); + if (sk_fds[i] == -1) { + err = -errno; + fprintf(stderr, "socket(): errno:%d\n", errno); + goto errout; + } + err = bpf_map_update_elem(map_fd, &sk_fds[i], &value, + BPF_NOEXIST); + if (err) { + err = -errno; + fprintf(stderr, + "bpf_map_update_elem(): errno:%d\n", + errno); + goto errout; + } + } + + notify_thread_done(); + wait_for_map_close(); + +close_all: + for (i = 0; i < nr_sk_per_thread; i++) { + close(sk_fds[i]); + sk_fds[i] = -1; + } + + notify_thread_redo(); + } + + free(sk_fds); + return NULL; + +errout: + for (i = 0; i < nr_sk_per_thread && sk_fds[i] != -1; i++) + close(sk_fds[i]); + free(sk_fds); + notify_thread_err(); + return ERR_PTR(err); +} + +static int do_sk_storage_map_stress_free(void) +{ + int i, map_fd = -1, err = 0, nr_threads_created = 0; + pthread_t *sk_thread_ids; + void *thread_ret; + + sk_thread_ids = malloc(sizeof(pthread_t) * nr_sk_threads); + if (!sk_thread_ids) { + fprintf(stderr, "malloc(sk_threads): NULL\n"); + return -ENOMEM; + } + + for (i = 0; i < nr_sk_threads; i++) { + err = pthread_create(&sk_thread_ids[i], NULL, + insert_close_thread, NULL); + if (err) { + err = -errno; + goto done; + } + nr_threads_created++; + } + + while (!is_stopped()) { + map_fd = create_sk_storage_map(); + WRITE_ONCE(sk_storage_map, map_fd); + + if (!wait_for_threads_done()) + break; + + WRITE_ONCE(sk_storage_map, -1); + close(map_fd); + map_fd = -1; + + if (!wait_for_threads_redo()) + break; + } + +done: + WRITE_ONCE(stop, 1); + for (i = 0; i < nr_threads_created; i++) { + pthread_join(sk_thread_ids[i], &thread_ret); + if (IS_ERR(thread_ret) && !err) { + err = PTR_ERR(thread_ret); + fprintf(stderr, "threads#%u: err:%d\n", i, err); + } + } + free(sk_thread_ids); + + if (map_fd != -1) + close(map_fd); + + return err; +} + +static void *update_thread(void *arg) +{ + struct { + int cnt; + int lock; + } value = { .cnt = 0xeB9F, .lock = 0, }; + int map_fd = READ_ONCE(sk_storage_map); + int sk_fd = *(int *)arg; + int err = 0; /* Suppress compiler false alarm */ + + while (!is_stopped()) { + err = bpf_map_update_elem(map_fd, &sk_fd, &value, 0); + if (err && errno != EAGAIN) { + err = -errno; + fprintf(stderr, "bpf_map_update_elem: %d %d\n", + err, errno); + break; + } + } + + if (!is_stopped()) { + notify_thread_err(); + return ERR_PTR(err); + } + + return NULL; +} + +static void *delete_thread(void *arg) +{ + int map_fd = READ_ONCE(sk_storage_map); + int sk_fd = *(int *)arg; + int err = 0; /* Suppress compiler false alarm */ + + while (!is_stopped()) { + err = bpf_map_delete_elem(map_fd, &sk_fd); + if (err && errno != ENOENT) { + err = -errno; + fprintf(stderr, "bpf_map_delete_elem: %d %d\n", + err, errno); + break; + } + } + + if (!is_stopped()) { + notify_thread_err(); + return ERR_PTR(err); + } + + return NULL; +} + +static int do_sk_storage_map_stress_change(void) +{ + int i, sk_fd, map_fd = -1, err = 0, nr_threads_created = 0; + pthread_t *sk_thread_ids; + void *thread_ret; + + sk_thread_ids = malloc(sizeof(pthread_t) * nr_sk_threads); + if (!sk_thread_ids) { + fprintf(stderr, "malloc(sk_threads): NULL\n"); + return -ENOMEM; + } + + sk_fd = socket(AF_INET6, SOCK_STREAM, 0); + if (sk_fd == -1) { + err = -errno; + goto done; + } + + map_fd = create_sk_storage_map(); + WRITE_ONCE(sk_storage_map, map_fd); + + for (i = 0; i < nr_sk_threads; i++) { + if (i & 0x1) + err = pthread_create(&sk_thread_ids[i], NULL, + update_thread, &sk_fd); + else + err = pthread_create(&sk_thread_ids[i], NULL, + delete_thread, &sk_fd); + if (err) { + err = -errno; + goto done; + } + nr_threads_created++; + } + + wait_for_threads_err(); + +done: + WRITE_ONCE(stop, 1); + for (i = 0; i < nr_threads_created; i++) { + pthread_join(sk_thread_ids[i], &thread_ret); + if (IS_ERR(thread_ret) && !err) { + err = PTR_ERR(thread_ret); + fprintf(stderr, "threads#%u: err:%d\n", i, err); + } + } + free(sk_thread_ids); + + if (sk_fd != -1) + close(sk_fd); + close(map_fd); + + return err; +} + +static void stop_handler(int signum) +{ + if (signum != SIGALRM) + printf("stopping...\n"); + WRITE_ONCE(stop, 1); +} + +#define BPF_SK_STORAGE_MAP_TEST_NR_THREADS "BPF_SK_STORAGE_MAP_TEST_NR_THREADS" +#define BPF_SK_STORAGE_MAP_TEST_SK_PER_THREAD "BPF_SK_STORAGE_MAP_TEST_SK_PER_THREAD" +#define BPF_SK_STORAGE_MAP_TEST_RUNTIME_S "BPF_SK_STORAGE_MAP_TEST_RUNTIME_S" +#define BPF_SK_STORAGE_MAP_TEST_NAME "BPF_SK_STORAGE_MAP_TEST_NAME" + +static void test_sk_storage_map_stress_free(void) +{ + struct rlimit rlim_old, rlim_new = {}; + int err; + + getrlimit(RLIMIT_NOFILE, &rlim_old); + + signal(SIGTERM, stop_handler); + signal(SIGINT, stop_handler); + if (runtime_s > 0) { + signal(SIGALRM, stop_handler); + alarm(runtime_s); + } + + if (rlim_old.rlim_cur < nr_sk_threads * nr_sk_per_thread) { + rlim_new.rlim_cur = nr_sk_threads * nr_sk_per_thread + 128; + rlim_new.rlim_max = rlim_new.rlim_cur + 128; + err = setrlimit(RLIMIT_NOFILE, &rlim_new); + CHECK(err, "setrlimit(RLIMIT_NOFILE)", "rlim_new:%lu errno:%d", + rlim_new.rlim_cur, errno); + } + + err = do_sk_storage_map_stress_free(); + + signal(SIGTERM, SIG_DFL); + signal(SIGINT, SIG_DFL); + if (runtime_s > 0) { + signal(SIGALRM, SIG_DFL); + alarm(0); + } + + if (rlim_new.rlim_cur) + setrlimit(RLIMIT_NOFILE, &rlim_old); + + CHECK(err, "test_sk_storage_map_stress_free", "err:%d\n", err); +} + +static void test_sk_storage_map_stress_change(void) +{ + int err; + + signal(SIGTERM, stop_handler); + signal(SIGINT, stop_handler); + if (runtime_s > 0) { + signal(SIGALRM, stop_handler); + alarm(runtime_s); + } + + err = do_sk_storage_map_stress_change(); + + signal(SIGTERM, SIG_DFL); + signal(SIGINT, SIG_DFL); + if (runtime_s > 0) { + signal(SIGALRM, SIG_DFL); + alarm(0); + } + + CHECK(err, "test_sk_storage_map_stress_change", "err:%d\n", err); +} + +static void test_sk_storage_map_basic(void) +{ + struct { + int cnt; + int lock; + } value = { .cnt = 0xeB9f, .lock = 0, }, lookup_value; + struct bpf_create_map_attr bad_xattr; + int btf_fd, map_fd, sk_fd, err; + + btf_fd = load_btf(); + CHECK(btf_fd == -1, "bpf_load_btf", "btf_fd:%d errno:%d\n", + btf_fd, errno); + xattr.btf_fd = btf_fd; + + sk_fd = socket(AF_INET6, SOCK_STREAM, 0); + CHECK(sk_fd == -1, "socket()", "sk_fd:%d errno:%d\n", + sk_fd, errno); + + map_fd = bpf_create_map_xattr(&xattr); + CHECK(map_fd == -1, "bpf_create_map_xattr(good_xattr)", + "map_fd:%d errno:%d\n", map_fd, errno); + + /* Add new elem */ + memcpy(&lookup_value, &value, sizeof(value)); + err = bpf_map_update_elem(map_fd, &sk_fd, &value, + BPF_NOEXIST | BPF_F_LOCK); + CHECK(err, "bpf_map_update_elem(BPF_NOEXIST|BPF_F_LOCK)", + "err:%d errno:%d\n", err, errno); + err = bpf_map_lookup_elem_flags(map_fd, &sk_fd, &lookup_value, + BPF_F_LOCK); + CHECK(err || lookup_value.cnt != value.cnt, + "bpf_map_lookup_elem_flags(BPF_F_LOCK)", + "err:%d errno:%d cnt:%x(%x)\n", + err, errno, lookup_value.cnt, value.cnt); + + /* Bump the cnt and update with BPF_EXIST | BPF_F_LOCK */ + value.cnt += 1; + err = bpf_map_update_elem(map_fd, &sk_fd, &value, + BPF_EXIST | BPF_F_LOCK); + CHECK(err, "bpf_map_update_elem(BPF_EXIST|BPF_F_LOCK)", + "err:%d errno:%d\n", err, errno); + err = bpf_map_lookup_elem_flags(map_fd, &sk_fd, &lookup_value, + BPF_F_LOCK); + CHECK(err || lookup_value.cnt != value.cnt, + "bpf_map_lookup_elem_flags(BPF_F_LOCK)", + "err:%d errno:%d cnt:%x(%x)\n", + err, errno, lookup_value.cnt, value.cnt); + + /* Bump the cnt and update with BPF_EXIST */ + value.cnt += 1; + err = bpf_map_update_elem(map_fd, &sk_fd, &value, BPF_EXIST); + CHECK(err, "bpf_map_update_elem(BPF_EXIST)", + "err:%d errno:%d\n", err, errno); + err = bpf_map_lookup_elem_flags(map_fd, &sk_fd, &lookup_value, + BPF_F_LOCK); + CHECK(err || lookup_value.cnt != value.cnt, + "bpf_map_lookup_elem_flags(BPF_F_LOCK)", + "err:%d errno:%d cnt:%x(%x)\n", + err, errno, lookup_value.cnt, value.cnt); + + /* Update with BPF_NOEXIST */ + value.cnt += 1; + err = bpf_map_update_elem(map_fd, &sk_fd, &value, + BPF_NOEXIST | BPF_F_LOCK); + CHECK(!err || errno != EEXIST, + "bpf_map_update_elem(BPF_NOEXIST|BPF_F_LOCK)", + "err:%d errno:%d\n", err, errno); + err = bpf_map_update_elem(map_fd, &sk_fd, &value, BPF_NOEXIST); + CHECK(!err || errno != EEXIST, "bpf_map_update_elem(BPF_NOEXIST)", + "err:%d errno:%d\n", err, errno); + value.cnt -= 1; + err = bpf_map_lookup_elem_flags(map_fd, &sk_fd, &lookup_value, + BPF_F_LOCK); + CHECK(err || lookup_value.cnt != value.cnt, + "bpf_map_lookup_elem_flags(BPF_F_LOCK)", + "err:%d errno:%d cnt:%x(%x)\n", + err, errno, lookup_value.cnt, value.cnt); + + /* Bump the cnt again and update with map_flags == 0 */ + value.cnt += 1; + err = bpf_map_update_elem(map_fd, &sk_fd, &value, 0); + CHECK(err, "bpf_map_update_elem()", "err:%d errno:%d\n", + err, errno); + err = bpf_map_lookup_elem_flags(map_fd, &sk_fd, &lookup_value, + BPF_F_LOCK); + CHECK(err || lookup_value.cnt != value.cnt, + "bpf_map_lookup_elem_flags(BPF_F_LOCK)", + "err:%d errno:%d cnt:%x(%x)\n", + err, errno, lookup_value.cnt, value.cnt); + + /* Test delete elem */ + err = bpf_map_delete_elem(map_fd, &sk_fd); + CHECK(err, "bpf_map_delete_elem()", "err:%d errno:%d\n", + err, errno); + err = bpf_map_lookup_elem_flags(map_fd, &sk_fd, &lookup_value, + BPF_F_LOCK); + CHECK(!err || errno != ENOENT, + "bpf_map_lookup_elem_flags(BPF_F_LOCK)", + "err:%d errno:%d\n", err, errno); + err = bpf_map_delete_elem(map_fd, &sk_fd); + CHECK(!err || errno != ENOENT, "bpf_map_delete_elem()", + "err:%d errno:%d\n", err, errno); + + memcpy(&bad_xattr, &xattr, sizeof(xattr)); + bad_xattr.btf_key_type_id = 0; + err = bpf_create_map_xattr(&bad_xattr); + CHECK(!err || errno != EINVAL, "bap_create_map_xattr(bad_xattr)", + "err:%d errno:%d\n", err, errno); + + memcpy(&bad_xattr, &xattr, sizeof(xattr)); + bad_xattr.btf_key_type_id = 3; + err = bpf_create_map_xattr(&bad_xattr); + CHECK(!err || errno != EINVAL, "bap_create_map_xattr(bad_xattr)", + "err:%d errno:%d\n", err, errno); + + memcpy(&bad_xattr, &xattr, sizeof(xattr)); + bad_xattr.max_entries = 1; + err = bpf_create_map_xattr(&bad_xattr); + CHECK(!err || errno != EINVAL, "bap_create_map_xattr(bad_xattr)", + "err:%d errno:%d\n", err, errno); + + memcpy(&bad_xattr, &xattr, sizeof(xattr)); + bad_xattr.map_flags = 0; + err = bpf_create_map_xattr(&bad_xattr); + CHECK(!err || errno != EINVAL, "bap_create_map_xattr(bad_xattr)", + "err:%d errno:%d\n", err, errno); + + xattr.btf_fd = -1; + close(btf_fd); + close(map_fd); + close(sk_fd); +} + +void test_sk_storage_map(void) +{ + const char *test_name, *env_opt; + bool test_ran = false; + + test_name = getenv(BPF_SK_STORAGE_MAP_TEST_NAME); + + env_opt = getenv(BPF_SK_STORAGE_MAP_TEST_NR_THREADS); + if (env_opt) + nr_sk_threads = atoi(env_opt); + + env_opt = getenv(BPF_SK_STORAGE_MAP_TEST_SK_PER_THREAD); + if (env_opt) + nr_sk_per_thread = atoi(env_opt); + + env_opt = getenv(BPF_SK_STORAGE_MAP_TEST_RUNTIME_S); + if (env_opt) + runtime_s = atoi(env_opt); + + if (!test_name || !strcmp(test_name, "basic")) { + test_sk_storage_map_basic(); + test_ran = true; + } + if (!test_name || !strcmp(test_name, "stress_free")) { + test_sk_storage_map_stress_free(); + test_ran = true; + } + if (!test_name || !strcmp(test_name, "stress_change")) { + test_sk_storage_map_stress_change(); + test_ran = true; + } + + if (test_ran) + printf("%s:PASS\n", __func__); + else + CHECK(1, "Invalid test_name", "%s\n", test_name); +} diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_obj_id.c b/tools/testing/selftests/bpf/prog_tests/bpf_obj_id.c index a64f7a02139c258c33704fcd6c575d9c341235c0..cb827383db4d5614799ca83b998a094b25415191 100644 --- a/tools/testing/selftests/bpf/prog_tests/bpf_obj_id.c +++ b/tools/testing/selftests/bpf/prog_tests/bpf_obj_id.c @@ -73,7 +73,7 @@ void test_bpf_obj_id(void) info_len != sizeof(struct bpf_map_info) || strcmp((char *)map_infos[i].name, expected_map_name), "get-map-info(fd)", - "err %d errno %d type %d(%d) info_len %u(%Zu) key_size %u value_size %u max_entries %u map_flags %X name %s(%s)\n", + "err %d errno %d type %d(%d) info_len %u(%zu) key_size %u value_size %u max_entries %u map_flags %X name %s(%s)\n", err, errno, map_infos[i].type, BPF_MAP_TYPE_ARRAY, info_len, sizeof(struct bpf_map_info), @@ -117,7 +117,7 @@ void test_bpf_obj_id(void) *(int *)(long)prog_infos[i].map_ids != map_infos[i].id || strcmp((char *)prog_infos[i].name, expected_prog_name), "get-prog-info(fd)", - "err %d errno %d i %d type %d(%d) info_len %u(%Zu) jit_enabled %d jited_prog_len %u xlated_prog_len %u jited_prog %d xlated_prog %d load_time %lu(%lu) uid %u(%u) nr_map_ids %u(%u) map_id %u(%u) name %s(%s)\n", + "err %d errno %d i %d type %d(%d) info_len %u(%zu) jit_enabled %d jited_prog_len %u xlated_prog_len %u jited_prog %d xlated_prog %d load_time %lu(%lu) uid %u(%u) nr_map_ids %u(%u) map_id %u(%u) name %s(%s)\n", err, errno, i, prog_infos[i].type, BPF_PROG_TYPE_SOCKET_FILTER, info_len, sizeof(struct bpf_prog_info), @@ -185,7 +185,7 @@ void test_bpf_obj_id(void) memcmp(&prog_info, &prog_infos[i], info_len) || *(int *)(long)prog_info.map_ids != saved_map_id, "get-prog-info(next_id->fd)", - "err %d errno %d info_len %u(%Zu) memcmp %d map_id %u(%u)\n", + "err %d errno %d info_len %u(%zu) memcmp %d map_id %u(%u)\n", err, errno, info_len, sizeof(struct bpf_prog_info), memcmp(&prog_info, &prog_infos[i], info_len), *(int *)(long)prog_info.map_ids, saved_map_id); @@ -231,7 +231,7 @@ void test_bpf_obj_id(void) memcmp(&map_info, &map_infos[i], info_len) || array_value != array_magic_value, "check get-map-info(next_id->fd)", - "err %d errno %d info_len %u(%Zu) memcmp %d array_value %llu(%llu)\n", + "err %d errno %d info_len %u(%zu) memcmp %d array_value %llu(%llu)\n", err, errno, info_len, sizeof(struct bpf_map_info), memcmp(&map_info, &map_infos[i], info_len), array_value, array_magic_value); diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c b/tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c new file mode 100644 index 0000000000000000000000000000000000000000..23b159d95c3f7f2c1f19e1bfb306302bbfa66f20 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c @@ -0,0 +1,49 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2019 Facebook +#include +static int libbpf_debug_print(enum libbpf_print_level level, + const char *format, va_list args) +{ + if (level != LIBBPF_DEBUG) + return 0; + + if (!strstr(format, "verifier log")) + return 0; + return vfprintf(stderr, "%s", args); +} + +static int check_load(const char *file) +{ + struct bpf_prog_load_attr attr; + struct bpf_object *obj; + int err, prog_fd; + + memset(&attr, 0, sizeof(struct bpf_prog_load_attr)); + attr.file = file; + attr.prog_type = BPF_PROG_TYPE_SCHED_CLS; + attr.log_level = 4; + err = bpf_prog_load_xattr(&attr, &obj, &prog_fd); + bpf_object__close(obj); + if (err) + error_cnt++; + return err; +} + +void test_bpf_verif_scale(void) +{ + const char *file1 = "./test_verif_scale1.o"; + const char *file2 = "./test_verif_scale2.o"; + const char *file3 = "./test_verif_scale3.o"; + int err; + + if (verifier_stats) + libbpf_set_print(libbpf_debug_print); + + err = check_load(file1); + err |= check_load(file2); + err |= check_load(file3); + if (!err) + printf("test_verif_scale:OK\n"); + else + printf("test_verif_scale:FAIL\n"); +} diff --git a/tools/testing/selftests/bpf/prog_tests/flow_dissector.c b/tools/testing/selftests/bpf/prog_tests/flow_dissector.c index fc818bc1d7294454093a949b3e5f394ca9879982..8b54adfd626449a1b4a6621d6b5c2dad5ec75083 100644 --- a/tools/testing/selftests/bpf/prog_tests/flow_dissector.c +++ b/tools/testing/selftests/bpf/prog_tests/flow_dissector.c @@ -1,8 +1,11 @@ // SPDX-License-Identifier: GPL-2.0 #include +#include +#include +#include #define CHECK_FLOW_KEYS(desc, got, expected) \ - CHECK(memcmp(&got, &expected, sizeof(got)) != 0, \ + CHECK_ATTR(memcmp(&got, &expected, sizeof(got)) != 0, \ desc, \ "nhoff=%u/%u " \ "thoff=%u/%u " \ @@ -10,6 +13,7 @@ "is_frag=%u/%u " \ "is_first_frag=%u/%u " \ "is_encap=%u/%u " \ + "ip_proto=0x%x/0x%x " \ "n_proto=0x%x/0x%x " \ "sport=%u/%u " \ "dport=%u/%u\n", \ @@ -19,53 +23,32 @@ got.is_frag, expected.is_frag, \ got.is_first_frag, expected.is_first_frag, \ got.is_encap, expected.is_encap, \ + got.ip_proto, expected.ip_proto, \ got.n_proto, expected.n_proto, \ got.sport, expected.sport, \ got.dport, expected.dport) -static struct bpf_flow_keys pkt_v4_flow_keys = { - .nhoff = 0, - .thoff = sizeof(struct iphdr), - .addr_proto = ETH_P_IP, - .ip_proto = IPPROTO_TCP, - .n_proto = __bpf_constant_htons(ETH_P_IP), -}; - -static struct bpf_flow_keys pkt_v6_flow_keys = { - .nhoff = 0, - .thoff = sizeof(struct ipv6hdr), - .addr_proto = ETH_P_IPV6, - .ip_proto = IPPROTO_TCP, - .n_proto = __bpf_constant_htons(ETH_P_IPV6), -}; - -#define VLAN_HLEN 4 +struct ipv4_pkt { + struct ethhdr eth; + struct iphdr iph; + struct tcphdr tcp; +} __packed; -static struct { +struct svlan_ipv4_pkt { struct ethhdr eth; __u16 vlan_tci; __u16 vlan_proto; struct iphdr iph; struct tcphdr tcp; -} __packed pkt_vlan_v4 = { - .eth.h_proto = __bpf_constant_htons(ETH_P_8021Q), - .vlan_proto = __bpf_constant_htons(ETH_P_IP), - .iph.ihl = 5, - .iph.protocol = IPPROTO_TCP, - .iph.tot_len = __bpf_constant_htons(MAGIC_BYTES), - .tcp.urg_ptr = 123, - .tcp.doff = 5, -}; +} __packed; -static struct bpf_flow_keys pkt_vlan_v4_flow_keys = { - .nhoff = VLAN_HLEN, - .thoff = VLAN_HLEN + sizeof(struct iphdr), - .addr_proto = ETH_P_IP, - .ip_proto = IPPROTO_TCP, - .n_proto = __bpf_constant_htons(ETH_P_IP), -}; +struct ipv6_pkt { + struct ethhdr eth; + struct ipv6hdr iph; + struct tcphdr tcp; +} __packed; -static struct { +struct dvlan_ipv6_pkt { struct ethhdr eth; __u16 vlan_tci; __u16 vlan_proto; @@ -73,68 +56,213 @@ static struct { __u16 vlan_proto2; struct ipv6hdr iph; struct tcphdr tcp; -} __packed pkt_vlan_v6 = { - .eth.h_proto = __bpf_constant_htons(ETH_P_8021AD), - .vlan_proto = __bpf_constant_htons(ETH_P_8021Q), - .vlan_proto2 = __bpf_constant_htons(ETH_P_IPV6), - .iph.nexthdr = IPPROTO_TCP, - .iph.payload_len = __bpf_constant_htons(MAGIC_BYTES), - .tcp.urg_ptr = 123, - .tcp.doff = 5, +} __packed; + +struct test { + const char *name; + union { + struct ipv4_pkt ipv4; + struct svlan_ipv4_pkt svlan_ipv4; + struct ipv6_pkt ipv6; + struct dvlan_ipv6_pkt dvlan_ipv6; + } pkt; + struct bpf_flow_keys keys; }; -static struct bpf_flow_keys pkt_vlan_v6_flow_keys = { - .nhoff = VLAN_HLEN * 2, - .thoff = VLAN_HLEN * 2 + sizeof(struct ipv6hdr), - .addr_proto = ETH_P_IPV6, - .ip_proto = IPPROTO_TCP, - .n_proto = __bpf_constant_htons(ETH_P_IPV6), +#define VLAN_HLEN 4 + +struct test tests[] = { + { + .name = "ipv4", + .pkt.ipv4 = { + .eth.h_proto = __bpf_constant_htons(ETH_P_IP), + .iph.ihl = 5, + .iph.protocol = IPPROTO_TCP, + .iph.tot_len = __bpf_constant_htons(MAGIC_BYTES), + .tcp.doff = 5, + }, + .keys = { + .nhoff = ETH_HLEN, + .thoff = ETH_HLEN + sizeof(struct iphdr), + .addr_proto = ETH_P_IP, + .ip_proto = IPPROTO_TCP, + .n_proto = __bpf_constant_htons(ETH_P_IP), + }, + }, + { + .name = "ipv6", + .pkt.ipv6 = { + .eth.h_proto = __bpf_constant_htons(ETH_P_IPV6), + .iph.nexthdr = IPPROTO_TCP, + .iph.payload_len = __bpf_constant_htons(MAGIC_BYTES), + .tcp.doff = 5, + }, + .keys = { + .nhoff = ETH_HLEN, + .thoff = ETH_HLEN + sizeof(struct ipv6hdr), + .addr_proto = ETH_P_IPV6, + .ip_proto = IPPROTO_TCP, + .n_proto = __bpf_constant_htons(ETH_P_IPV6), + }, + }, + { + .name = "802.1q-ipv4", + .pkt.svlan_ipv4 = { + .eth.h_proto = __bpf_constant_htons(ETH_P_8021Q), + .vlan_proto = __bpf_constant_htons(ETH_P_IP), + .iph.ihl = 5, + .iph.protocol = IPPROTO_TCP, + .iph.tot_len = __bpf_constant_htons(MAGIC_BYTES), + .tcp.doff = 5, + }, + .keys = { + .nhoff = ETH_HLEN + VLAN_HLEN, + .thoff = ETH_HLEN + VLAN_HLEN + sizeof(struct iphdr), + .addr_proto = ETH_P_IP, + .ip_proto = IPPROTO_TCP, + .n_proto = __bpf_constant_htons(ETH_P_IP), + }, + }, + { + .name = "802.1ad-ipv6", + .pkt.dvlan_ipv6 = { + .eth.h_proto = __bpf_constant_htons(ETH_P_8021AD), + .vlan_proto = __bpf_constant_htons(ETH_P_8021Q), + .vlan_proto2 = __bpf_constant_htons(ETH_P_IPV6), + .iph.nexthdr = IPPROTO_TCP, + .iph.payload_len = __bpf_constant_htons(MAGIC_BYTES), + .tcp.doff = 5, + }, + .keys = { + .nhoff = ETH_HLEN + VLAN_HLEN * 2, + .thoff = ETH_HLEN + VLAN_HLEN * 2 + + sizeof(struct ipv6hdr), + .addr_proto = ETH_P_IPV6, + .ip_proto = IPPROTO_TCP, + .n_proto = __bpf_constant_htons(ETH_P_IPV6), + }, + }, }; +static int create_tap(const char *ifname) +{ + struct ifreq ifr = { + .ifr_flags = IFF_TAP | IFF_NO_PI | IFF_NAPI | IFF_NAPI_FRAGS, + }; + int fd, ret; + + strncpy(ifr.ifr_name, ifname, sizeof(ifr.ifr_name)); + + fd = open("/dev/net/tun", O_RDWR); + if (fd < 0) + return -1; + + ret = ioctl(fd, TUNSETIFF, &ifr); + if (ret) + return -1; + + return fd; +} + +static int tx_tap(int fd, void *pkt, size_t len) +{ + struct iovec iov[] = { + { + .iov_len = len, + .iov_base = pkt, + }, + }; + return writev(fd, iov, ARRAY_SIZE(iov)); +} + +static int ifup(const char *ifname) +{ + struct ifreq ifr = {}; + int sk, ret; + + strncpy(ifr.ifr_name, ifname, sizeof(ifr.ifr_name)); + + sk = socket(PF_INET, SOCK_DGRAM, 0); + if (sk < 0) + return -1; + + ret = ioctl(sk, SIOCGIFFLAGS, &ifr); + if (ret) { + close(sk); + return -1; + } + + ifr.ifr_flags |= IFF_UP; + ret = ioctl(sk, SIOCSIFFLAGS, &ifr); + if (ret) { + close(sk); + return -1; + } + + close(sk); + return 0; +} + void test_flow_dissector(void) { - struct bpf_flow_keys flow_keys; + int i, err, prog_fd, keys_fd = -1, tap_fd; struct bpf_object *obj; - __u32 duration, retval; - int err, prog_fd; - __u32 size; + __u32 duration = 0; err = bpf_flow_load(&obj, "./bpf_flow.o", "flow_dissector", - "jmp_table", &prog_fd); + "jmp_table", "last_dissection", &prog_fd, &keys_fd); if (err) { error_cnt++; return; } - err = bpf_prog_test_run(prog_fd, 10, &pkt_v4, sizeof(pkt_v4), - &flow_keys, &size, &retval, &duration); - CHECK(size != sizeof(flow_keys) || err || retval != 1, "ipv4", - "err %d errno %d retval %d duration %d size %u/%lu\n", - err, errno, retval, duration, size, sizeof(flow_keys)); - CHECK_FLOW_KEYS("ipv4_flow_keys", flow_keys, pkt_v4_flow_keys); - - err = bpf_prog_test_run(prog_fd, 10, &pkt_v6, sizeof(pkt_v6), - &flow_keys, &size, &retval, &duration); - CHECK(size != sizeof(flow_keys) || err || retval != 1, "ipv6", - "err %d errno %d retval %d duration %d size %u/%lu\n", - err, errno, retval, duration, size, sizeof(flow_keys)); - CHECK_FLOW_KEYS("ipv6_flow_keys", flow_keys, pkt_v6_flow_keys); - - err = bpf_prog_test_run(prog_fd, 10, &pkt_vlan_v4, sizeof(pkt_vlan_v4), - &flow_keys, &size, &retval, &duration); - CHECK(size != sizeof(flow_keys) || err || retval != 1, "vlan_ipv4", - "err %d errno %d retval %d duration %d size %u/%lu\n", - err, errno, retval, duration, size, sizeof(flow_keys)); - CHECK_FLOW_KEYS("vlan_ipv4_flow_keys", flow_keys, - pkt_vlan_v4_flow_keys); - - err = bpf_prog_test_run(prog_fd, 10, &pkt_vlan_v6, sizeof(pkt_vlan_v6), - &flow_keys, &size, &retval, &duration); - CHECK(size != sizeof(flow_keys) || err || retval != 1, "vlan_ipv6", - "err %d errno %d retval %d duration %d size %u/%lu\n", - err, errno, retval, duration, size, sizeof(flow_keys)); - CHECK_FLOW_KEYS("vlan_ipv6_flow_keys", flow_keys, - pkt_vlan_v6_flow_keys); + for (i = 0; i < ARRAY_SIZE(tests); i++) { + struct bpf_flow_keys flow_keys; + struct bpf_prog_test_run_attr tattr = { + .prog_fd = prog_fd, + .data_in = &tests[i].pkt, + .data_size_in = sizeof(tests[i].pkt), + .data_out = &flow_keys, + }; + + err = bpf_prog_test_run_xattr(&tattr); + CHECK_ATTR(tattr.data_size_out != sizeof(flow_keys) || + err || tattr.retval != 1, + tests[i].name, + "err %d errno %d retval %d duration %d size %u/%lu\n", + err, errno, tattr.retval, tattr.duration, + tattr.data_size_out, sizeof(flow_keys)); + CHECK_FLOW_KEYS(tests[i].name, flow_keys, tests[i].keys); + } + + /* Do the same tests but for skb-less flow dissector. + * We use a known path in the net/tun driver that calls + * eth_get_headlen and we manually export bpf_flow_keys + * via BPF map in this case. + */ + + err = bpf_prog_attach(prog_fd, 0, BPF_FLOW_DISSECTOR, 0); + CHECK(err, "bpf_prog_attach", "err %d errno %d", err, errno); + + tap_fd = create_tap("tap0"); + CHECK(tap_fd < 0, "create_tap", "tap_fd %d errno %d", tap_fd, errno); + err = ifup("tap0"); + CHECK(err, "ifup", "err %d errno %d", err, errno); + + for (i = 0; i < ARRAY_SIZE(tests); i++) { + struct bpf_flow_keys flow_keys = {}; + struct bpf_prog_test_run_attr tattr = {}; + __u32 key = 0; + + err = tx_tap(tap_fd, &tests[i].pkt, sizeof(tests[i].pkt)); + CHECK(err < 0, "tx_tap", "err %d errno %d", err, errno); + + err = bpf_map_lookup_elem(keys_fd, &key, &flow_keys); + CHECK_ATTR(err, tests[i].name, "bpf_map_lookup_elem %d\n", err); + + CHECK_ATTR(err, tests[i].name, "skb-less err %d\n", err); + CHECK_FLOW_KEYS(tests[i].name, flow_keys, tests[i].keys); + } bpf_object__close(obj); } diff --git a/tools/testing/selftests/bpf/prog_tests/flow_dissector_load_bytes.c b/tools/testing/selftests/bpf/prog_tests/flow_dissector_load_bytes.c new file mode 100644 index 0000000000000000000000000000000000000000..dc5ef155ec282b3403df8db16ad81ceb7e4cc1a1 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/flow_dissector_load_bytes.c @@ -0,0 +1,48 @@ +// SPDX-License-Identifier: GPL-2.0 +#include + +void test_flow_dissector_load_bytes(void) +{ + struct bpf_flow_keys flow_keys; + __u32 duration = 0, retval, size; + struct bpf_insn prog[] = { + // BPF_REG_1 - 1st argument: context + // BPF_REG_2 - 2nd argument: offset, start at first byte + BPF_MOV64_IMM(BPF_REG_2, 0), + // BPF_REG_3 - 3rd argument: destination, reserve byte on stack + BPF_ALU64_REG(BPF_MOV, BPF_REG_3, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, -1), + // BPF_REG_4 - 4th argument: copy one byte + BPF_MOV64_IMM(BPF_REG_4, 1), + // bpf_skb_load_bytes(ctx, sizeof(pkt_v4), ptr, 1) + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, + BPF_FUNC_skb_load_bytes), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), + // if (ret == 0) return BPF_DROP (2) + BPF_MOV64_IMM(BPF_REG_0, BPF_DROP), + BPF_EXIT_INSN(), + // if (ret != 0) return BPF_OK (0) + BPF_MOV64_IMM(BPF_REG_0, BPF_OK), + BPF_EXIT_INSN(), + }; + int fd, err; + + /* make sure bpf_skb_load_bytes is not allowed from skb-less context + */ + fd = bpf_load_program(BPF_PROG_TYPE_FLOW_DISSECTOR, prog, + ARRAY_SIZE(prog), "GPL", 0, NULL, 0); + CHECK(fd < 0, + "flow_dissector-bpf_skb_load_bytes-load", + "fd %d errno %d\n", + fd, errno); + + err = bpf_prog_test_run(fd, 1, &pkt_v4, sizeof(pkt_v4), + &flow_keys, &size, &retval, &duration); + CHECK(size != sizeof(flow_keys) || err || retval != 1, + "flow_dissector-bpf_skb_load_bytes", + "err %d errno %d retval %d duration %d size %u/%zu\n", + err, errno, retval, duration, size, sizeof(flow_keys)); + + if (fd >= -1) + close(fd); +} diff --git a/tools/testing/selftests/bpf/prog_tests/get_stack_raw_tp.c b/tools/testing/selftests/bpf/prog_tests/get_stack_raw_tp.c index d7bb5beb1c57c112092b5e76e0755185f9303f8d..c2a0a9d5591b4b5ea3436bfe490689c7756e50db 100644 --- a/tools/testing/selftests/bpf/prog_tests/get_stack_raw_tp.c +++ b/tools/testing/selftests/bpf/prog_tests/get_stack_raw_tp.c @@ -39,7 +39,7 @@ static int get_stack_print_output(void *data, int size) } else { for (i = 0; i < num_stack; i++) { ks = ksym_search(raw_data[i]); - if (strcmp(ks->name, nonjit_func) == 0) { + if (ks && (strcmp(ks->name, nonjit_func) == 0)) { found = true; break; } @@ -56,7 +56,7 @@ static int get_stack_print_output(void *data, int size) } else { for (i = 0; i < num_stack; i++) { ks = ksym_search(e->kern_stack[i]); - if (strcmp(ks->name, nonjit_func) == 0) { + if (ks && (strcmp(ks->name, nonjit_func) == 0)) { good_kern_stack = true; break; } diff --git a/tools/testing/selftests/bpf/prog_tests/global_data.c b/tools/testing/selftests/bpf/prog_tests/global_data.c new file mode 100644 index 0000000000000000000000000000000000000000..d011079fb0bfe6dbd6c963c342787fc25e97b819 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/global_data.c @@ -0,0 +1,157 @@ +// SPDX-License-Identifier: GPL-2.0 +#include + +static void test_global_data_number(struct bpf_object *obj, __u32 duration) +{ + int i, err, map_fd; + uint64_t num; + + map_fd = bpf_find_map(__func__, obj, "result_number"); + if (map_fd < 0) { + error_cnt++; + return; + } + + struct { + char *name; + uint32_t key; + uint64_t num; + } tests[] = { + { "relocate .bss reference", 0, 0 }, + { "relocate .data reference", 1, 42 }, + { "relocate .rodata reference", 2, 24 }, + { "relocate .bss reference", 3, 0 }, + { "relocate .data reference", 4, 0xffeeff }, + { "relocate .rodata reference", 5, 0xabab }, + { "relocate .bss reference", 6, 1234 }, + { "relocate .bss reference", 7, 0 }, + { "relocate .rodata reference", 8, 0xab }, + { "relocate .rodata reference", 9, 0x1111111111111111 }, + { "relocate .rodata reference", 10, ~0 }, + }; + + for (i = 0; i < sizeof(tests) / sizeof(tests[0]); i++) { + err = bpf_map_lookup_elem(map_fd, &tests[i].key, &num); + CHECK(err || num != tests[i].num, tests[i].name, + "err %d result %lx expected %lx\n", + err, num, tests[i].num); + } +} + +static void test_global_data_string(struct bpf_object *obj, __u32 duration) +{ + int i, err, map_fd; + char str[32]; + + map_fd = bpf_find_map(__func__, obj, "result_string"); + if (map_fd < 0) { + error_cnt++; + return; + } + + struct { + char *name; + uint32_t key; + char str[32]; + } tests[] = { + { "relocate .rodata reference", 0, "abcdefghijklmnopqrstuvwxyz" }, + { "relocate .data reference", 1, "abcdefghijklmnopqrstuvwxyz" }, + { "relocate .bss reference", 2, "" }, + { "relocate .data reference", 3, "abcdexghijklmnopqrstuvwxyz" }, + { "relocate .bss reference", 4, "\0\0hello" }, + }; + + for (i = 0; i < sizeof(tests) / sizeof(tests[0]); i++) { + err = bpf_map_lookup_elem(map_fd, &tests[i].key, str); + CHECK(err || memcmp(str, tests[i].str, sizeof(str)), + tests[i].name, "err %d result \'%s\' expected \'%s\'\n", + err, str, tests[i].str); + } +} + +struct foo { + __u8 a; + __u32 b; + __u64 c; +}; + +static void test_global_data_struct(struct bpf_object *obj, __u32 duration) +{ + int i, err, map_fd; + struct foo val; + + map_fd = bpf_find_map(__func__, obj, "result_struct"); + if (map_fd < 0) { + error_cnt++; + return; + } + + struct { + char *name; + uint32_t key; + struct foo val; + } tests[] = { + { "relocate .rodata reference", 0, { 42, 0xfefeefef, 0x1111111111111111ULL, } }, + { "relocate .bss reference", 1, { } }, + { "relocate .rodata reference", 2, { } }, + { "relocate .data reference", 3, { 41, 0xeeeeefef, 0x2111111111111111ULL, } }, + }; + + for (i = 0; i < sizeof(tests) / sizeof(tests[0]); i++) { + err = bpf_map_lookup_elem(map_fd, &tests[i].key, &val); + CHECK(err || memcmp(&val, &tests[i].val, sizeof(val)), + tests[i].name, "err %d result { %u, %u, %llu } expected { %u, %u, %llu }\n", + err, val.a, val.b, val.c, tests[i].val.a, tests[i].val.b, tests[i].val.c); + } +} + +static void test_global_data_rdonly(struct bpf_object *obj, __u32 duration) +{ + int err = -ENOMEM, map_fd, zero = 0; + struct bpf_map *map; + __u8 *buff; + + map = bpf_object__find_map_by_name(obj, "test_glo.rodata"); + if (!map || !bpf_map__is_internal(map)) { + error_cnt++; + return; + } + + map_fd = bpf_map__fd(map); + if (map_fd < 0) { + error_cnt++; + return; + } + + buff = malloc(bpf_map__def(map)->value_size); + if (buff) + err = bpf_map_update_elem(map_fd, &zero, buff, 0); + free(buff); + CHECK(!err || errno != EPERM, "test .rodata read-only map", + "err %d errno %d\n", err, errno); +} + +void test_global_data(void) +{ + const char *file = "./test_global_data.o"; + __u32 duration = 0, retval; + struct bpf_object *obj; + int err, prog_fd; + + err = bpf_prog_load(file, BPF_PROG_TYPE_SCHED_CLS, &obj, &prog_fd); + if (CHECK(err, "load program", "error %d loading %s\n", err, file)) + return; + + err = bpf_prog_test_run(prog_fd, 1, &pkt_v4, sizeof(pkt_v4), + NULL, NULL, &retval, &duration); + CHECK(err || retval, "pass global data run", + "err %d errno %d retval %d duration %d\n", + err, errno, retval, duration); + + test_global_data_number(obj, duration); + test_global_data_string(obj, duration); + test_global_data_struct(obj, duration); + test_global_data_rdonly(obj, duration); + + bpf_object__close(obj); +} diff --git a/tools/testing/selftests/bpf/prog_tests/raw_tp_writable_reject_nbd_invalid.c b/tools/testing/selftests/bpf/prog_tests/raw_tp_writable_reject_nbd_invalid.c new file mode 100644 index 0000000000000000000000000000000000000000..9807336a301683ac5c11e69ca922e6ddc79cc653 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/raw_tp_writable_reject_nbd_invalid.c @@ -0,0 +1,42 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include + +void test_raw_tp_writable_reject_nbd_invalid(void) +{ + __u32 duration = 0; + char error[4096]; + int bpf_fd = -1, tp_fd = -1; + + const struct bpf_insn program[] = { + /* r6 is our tp buffer */ + BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0), + /* one byte beyond the end of the nbd_request struct */ + BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_6, + sizeof(struct nbd_request)), + BPF_EXIT_INSN(), + }; + + struct bpf_load_program_attr load_attr = { + .prog_type = BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE, + .license = "GPL v2", + .insns = program, + .insns_cnt = sizeof(program) / sizeof(struct bpf_insn), + .log_level = 2, + }; + + bpf_fd = bpf_load_program_xattr(&load_attr, error, sizeof(error)); + if (CHECK(bpf_fd < 0, "bpf_raw_tracepoint_writable load", + "failed: %d errno %d\n", bpf_fd, errno)) + return; + + tp_fd = bpf_raw_tracepoint_open("nbd_send_request", bpf_fd); + if (CHECK(tp_fd >= 0, "bpf_raw_tracepoint_writable open", + "erroneously succeeded\n")) + goto out_bpffd; + + close(tp_fd); +out_bpffd: + close(bpf_fd); +} diff --git a/tools/testing/selftests/bpf/prog_tests/raw_tp_writable_test_run.c b/tools/testing/selftests/bpf/prog_tests/raw_tp_writable_test_run.c new file mode 100644 index 0000000000000000000000000000000000000000..5c45424cac5f81ebe9ae71fbac561638b2d12064 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/raw_tp_writable_test_run.c @@ -0,0 +1,80 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include + +void test_raw_tp_writable_test_run(void) +{ + __u32 duration = 0; + char error[4096]; + + const struct bpf_insn trace_program[] = { + BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0), + BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6, 0), + BPF_MOV64_IMM(BPF_REG_0, 42), + BPF_STX_MEM(BPF_W, BPF_REG_6, BPF_REG_0, 0), + BPF_EXIT_INSN(), + }; + + struct bpf_load_program_attr load_attr = { + .prog_type = BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE, + .license = "GPL v2", + .insns = trace_program, + .insns_cnt = sizeof(trace_program) / sizeof(struct bpf_insn), + .log_level = 2, + }; + + int bpf_fd = bpf_load_program_xattr(&load_attr, error, sizeof(error)); + if (CHECK(bpf_fd < 0, "bpf_raw_tracepoint_writable loaded", + "failed: %d errno %d\n", bpf_fd, errno)) + return; + + const struct bpf_insn skb_program[] = { + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }; + + struct bpf_load_program_attr skb_load_attr = { + .prog_type = BPF_PROG_TYPE_SOCKET_FILTER, + .license = "GPL v2", + .insns = skb_program, + .insns_cnt = sizeof(skb_program) / sizeof(struct bpf_insn), + }; + + int filter_fd = + bpf_load_program_xattr(&skb_load_attr, error, sizeof(error)); + if (CHECK(filter_fd < 0, "test_program_loaded", "failed: %d errno %d\n", + filter_fd, errno)) + goto out_bpffd; + + int tp_fd = bpf_raw_tracepoint_open("bpf_test_finish", bpf_fd); + if (CHECK(tp_fd < 0, "bpf_raw_tracepoint_writable opened", + "failed: %d errno %d\n", tp_fd, errno)) + goto out_filterfd; + + char test_skb[128] = { + 0, + }; + + __u32 prog_ret; + int err = bpf_prog_test_run(filter_fd, 1, test_skb, sizeof(test_skb), 0, + 0, &prog_ret, 0); + CHECK(err != 42, "test_run", + "tracepoint did not modify return value\n"); + CHECK(prog_ret != 0, "test_run_ret", + "socket_filter did not return 0\n"); + + close(tp_fd); + + err = bpf_prog_test_run(filter_fd, 1, test_skb, sizeof(test_skb), 0, 0, + &prog_ret, 0); + CHECK(err != 0, "test_run_notrace", + "test_run failed with %d errno %d\n", err, errno); + CHECK(prog_ret != 0, "test_run_ret_notrace", + "socket_filter did not return 0\n"); + +out_filterfd: + close(filter_fd); +out_bpffd: + close(bpf_fd); +} diff --git a/tools/testing/selftests/bpf/prog_tests/skb_ctx.c b/tools/testing/selftests/bpf/prog_tests/skb_ctx.c new file mode 100644 index 0000000000000000000000000000000000000000..e95baa32e2778daba75b5af14eb09f8992289df7 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/skb_ctx.c @@ -0,0 +1,89 @@ +// SPDX-License-Identifier: GPL-2.0 +#include + +void test_skb_ctx(void) +{ + struct __sk_buff skb = { + .cb[0] = 1, + .cb[1] = 2, + .cb[2] = 3, + .cb[3] = 4, + .cb[4] = 5, + .priority = 6, + }; + struct bpf_prog_test_run_attr tattr = { + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .ctx_in = &skb, + .ctx_size_in = sizeof(skb), + .ctx_out = &skb, + .ctx_size_out = sizeof(skb), + }; + struct bpf_object *obj; + int err; + int i; + + err = bpf_prog_load("./test_skb_ctx.o", BPF_PROG_TYPE_SCHED_CLS, &obj, + &tattr.prog_fd); + if (CHECK_ATTR(err, "load", "err %d errno %d\n", err, errno)) + return; + + /* ctx_in != NULL, ctx_size_in == 0 */ + + tattr.ctx_size_in = 0; + err = bpf_prog_test_run_xattr(&tattr); + CHECK_ATTR(err == 0, "ctx_size_in", "err %d errno %d\n", err, errno); + tattr.ctx_size_in = sizeof(skb); + + /* ctx_out != NULL, ctx_size_out == 0 */ + + tattr.ctx_size_out = 0; + err = bpf_prog_test_run_xattr(&tattr); + CHECK_ATTR(err == 0, "ctx_size_out", "err %d errno %d\n", err, errno); + tattr.ctx_size_out = sizeof(skb); + + /* non-zero [len, tc_index] fields should be rejected*/ + + skb.len = 1; + err = bpf_prog_test_run_xattr(&tattr); + CHECK_ATTR(err == 0, "len", "err %d errno %d\n", err, errno); + skb.len = 0; + + skb.tc_index = 1; + err = bpf_prog_test_run_xattr(&tattr); + CHECK_ATTR(err == 0, "tc_index", "err %d errno %d\n", err, errno); + skb.tc_index = 0; + + /* non-zero [hash, sk] fields should be rejected */ + + skb.hash = 1; + err = bpf_prog_test_run_xattr(&tattr); + CHECK_ATTR(err == 0, "hash", "err %d errno %d\n", err, errno); + skb.hash = 0; + + skb.sk = (struct bpf_sock *)1; + err = bpf_prog_test_run_xattr(&tattr); + CHECK_ATTR(err == 0, "sk", "err %d errno %d\n", err, errno); + skb.sk = 0; + + err = bpf_prog_test_run_xattr(&tattr); + CHECK_ATTR(err != 0 || tattr.retval, + "run", + "err %d errno %d retval %d\n", + err, errno, tattr.retval); + + CHECK_ATTR(tattr.ctx_size_out != sizeof(skb), + "ctx_size_out", + "incorrect output size, want %lu have %u\n", + sizeof(skb), tattr.ctx_size_out); + + for (i = 0; i < 5; i++) + CHECK_ATTR(skb.cb[i] != i + 2, + "ctx_out_cb", + "skb->cb[i] == %d, expected %d\n", + skb.cb[i], i + 2); + CHECK_ATTR(skb.priority != 7, + "ctx_out_priority", + "skb->priority == %d, expected %d\n", + skb.priority, 7); +} diff --git a/tools/testing/selftests/bpf/prog_tests/stacktrace_build_id_nmi.c b/tools/testing/selftests/bpf/prog_tests/stacktrace_build_id_nmi.c index 8a114bb1c379040b81bd70c87e10ab8fd9401af1..1c1a2f75f3d828da1288ecaa52657851c0600be8 100644 --- a/tools/testing/selftests/bpf/prog_tests/stacktrace_build_id_nmi.c +++ b/tools/testing/selftests/bpf/prog_tests/stacktrace_build_id_nmi.c @@ -1,13 +1,25 @@ // SPDX-License-Identifier: GPL-2.0 #include +static __u64 read_perf_max_sample_freq(void) +{ + __u64 sample_freq = 5000; /* fallback to 5000 on error */ + FILE *f; + + f = fopen("/proc/sys/kernel/perf_event_max_sample_rate", "r"); + if (f == NULL) + return sample_freq; + fscanf(f, "%llu", &sample_freq); + fclose(f); + return sample_freq; +} + void test_stacktrace_build_id_nmi(void) { int control_map_fd, stackid_hmap_fd, stackmap_fd, stack_amap_fd; const char *file = "./test_stacktrace_build_id.o"; int err, pmu_fd, prog_fd; struct perf_event_attr attr = { - .sample_freq = 5000, .freq = 1, .type = PERF_TYPE_HARDWARE, .config = PERF_COUNT_HW_CPU_CYCLES, @@ -20,6 +32,8 @@ void test_stacktrace_build_id_nmi(void) int build_id_matches = 0; int retry = 1; + attr.sample_freq = read_perf_max_sample_freq(); + retry: err = bpf_prog_load(file, BPF_PROG_TYPE_PERF_EVENT, &obj, &prog_fd); if (CHECK(err, "prog_load", "err %d errno %d\n", err, errno)) diff --git a/tools/testing/selftests/bpf/progs/bpf_flow.c b/tools/testing/selftests/bpf/progs/bpf_flow.c index 75b17cada53937e5e237e00049a1bc22176cb83c..81ad9a0b29d06a32251c505361401c25433a2796 100644 --- a/tools/testing/selftests/bpf/progs/bpf_flow.c +++ b/tools/testing/selftests/bpf/progs/bpf_flow.c @@ -64,6 +64,25 @@ struct bpf_map_def SEC("maps") jmp_table = { .max_entries = 8 }; +struct bpf_map_def SEC("maps") last_dissection = { + .type = BPF_MAP_TYPE_ARRAY, + .key_size = sizeof(__u32), + .value_size = sizeof(struct bpf_flow_keys), + .max_entries = 1, +}; + +static __always_inline int export_flow_keys(struct bpf_flow_keys *keys, + int ret) +{ + struct bpf_flow_keys *val; + __u32 key = 0; + + val = bpf_map_lookup_elem(&last_dissection, &key); + if (val) + memcpy(val, keys, sizeof(*val)); + return ret; +} + static __always_inline void *bpf_flow_dissect_get_header(struct __sk_buff *skb, __u16 hdr_size, void *buffer) @@ -109,10 +128,10 @@ static __always_inline int parse_eth_proto(struct __sk_buff *skb, __be16 proto) break; default: /* Protocol not supported */ - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); } - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); } SEC("flow_dissector") @@ -139,8 +158,8 @@ static __always_inline int parse_ip_proto(struct __sk_buff *skb, __u8 proto) case IPPROTO_ICMP: icmp = bpf_flow_dissect_get_header(skb, sizeof(*icmp), &_icmp); if (!icmp) - return BPF_DROP; - return BPF_OK; + return export_flow_keys(keys, BPF_DROP); + return export_flow_keys(keys, BPF_OK); case IPPROTO_IPIP: keys->is_encap = true; return parse_eth_proto(skb, bpf_htons(ETH_P_IP)); @@ -150,11 +169,11 @@ static __always_inline int parse_ip_proto(struct __sk_buff *skb, __u8 proto) case IPPROTO_GRE: gre = bpf_flow_dissect_get_header(skb, sizeof(*gre), &_gre); if (!gre) - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); if (bpf_htons(gre->flags & GRE_VERSION)) /* Only inspect standard GRE packets with version 0 */ - return BPF_OK; + return export_flow_keys(keys, BPF_OK); keys->thoff += sizeof(*gre); /* Step over GRE Flags and Proto */ if (GRE_IS_CSUM(gre->flags)) @@ -170,7 +189,7 @@ static __always_inline int parse_ip_proto(struct __sk_buff *skb, __u8 proto) eth = bpf_flow_dissect_get_header(skb, sizeof(*eth), &_eth); if (!eth) - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); keys->thoff += sizeof(*eth); @@ -181,31 +200,31 @@ static __always_inline int parse_ip_proto(struct __sk_buff *skb, __u8 proto) case IPPROTO_TCP: tcp = bpf_flow_dissect_get_header(skb, sizeof(*tcp), &_tcp); if (!tcp) - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); if (tcp->doff < 5) - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); if ((__u8 *)tcp + (tcp->doff << 2) > data_end) - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); keys->sport = tcp->source; keys->dport = tcp->dest; - return BPF_OK; + return export_flow_keys(keys, BPF_OK); case IPPROTO_UDP: case IPPROTO_UDPLITE: udp = bpf_flow_dissect_get_header(skb, sizeof(*udp), &_udp); if (!udp) - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); keys->sport = udp->source; keys->dport = udp->dest; - return BPF_OK; + return export_flow_keys(keys, BPF_OK); default: - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); } - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); } static __always_inline int parse_ipv6_proto(struct __sk_buff *skb, __u8 nexthdr) @@ -225,7 +244,7 @@ static __always_inline int parse_ipv6_proto(struct __sk_buff *skb, __u8 nexthdr) return parse_ip_proto(skb, nexthdr); } - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); } PROG(IP)(struct __sk_buff *skb) @@ -238,11 +257,11 @@ PROG(IP)(struct __sk_buff *skb) iph = bpf_flow_dissect_get_header(skb, sizeof(*iph), &_iph); if (!iph) - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); /* IP header cannot be smaller than 20 bytes */ if (iph->ihl < 5) - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); keys->addr_proto = ETH_P_IP; keys->ipv4_src = iph->saddr; @@ -250,7 +269,7 @@ PROG(IP)(struct __sk_buff *skb) keys->thoff += iph->ihl << 2; if (data + keys->thoff > data_end) - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); if (iph->frag_off & bpf_htons(IP_MF | IP_OFFSET)) { keys->is_frag = true; @@ -264,7 +283,7 @@ PROG(IP)(struct __sk_buff *skb) } if (done) - return BPF_OK; + return export_flow_keys(keys, BPF_OK); return parse_ip_proto(skb, iph->protocol); } @@ -276,7 +295,7 @@ PROG(IPV6)(struct __sk_buff *skb) ip6h = bpf_flow_dissect_get_header(skb, sizeof(*ip6h), &_ip6h); if (!ip6h) - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); keys->addr_proto = ETH_P_IPV6; memcpy(&keys->ipv6_src, &ip6h->saddr, 2*sizeof(ip6h->saddr)); @@ -288,11 +307,12 @@ PROG(IPV6)(struct __sk_buff *skb) PROG(IPV6OP)(struct __sk_buff *skb) { + struct bpf_flow_keys *keys = skb->flow_keys; struct ipv6_opt_hdr *ip6h, _ip6h; ip6h = bpf_flow_dissect_get_header(skb, sizeof(*ip6h), &_ip6h); if (!ip6h) - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); /* hlen is in 8-octets and does not include the first 8 bytes * of the header @@ -309,7 +329,7 @@ PROG(IPV6FR)(struct __sk_buff *skb) fragh = bpf_flow_dissect_get_header(skb, sizeof(*fragh), &_fragh); if (!fragh) - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); keys->thoff += sizeof(*fragh); keys->is_frag = true; @@ -321,13 +341,14 @@ PROG(IPV6FR)(struct __sk_buff *skb) PROG(MPLS)(struct __sk_buff *skb) { + struct bpf_flow_keys *keys = skb->flow_keys; struct mpls_label *mpls, _mpls; mpls = bpf_flow_dissect_get_header(skb, sizeof(*mpls), &_mpls); if (!mpls) - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); - return BPF_OK; + return export_flow_keys(keys, BPF_OK); } PROG(VLAN)(struct __sk_buff *skb) @@ -339,10 +360,10 @@ PROG(VLAN)(struct __sk_buff *skb) if (keys->n_proto == bpf_htons(ETH_P_8021AD)) { vlan = bpf_flow_dissect_get_header(skb, sizeof(*vlan), &_vlan); if (!vlan) - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); if (vlan->h_vlan_encapsulated_proto != bpf_htons(ETH_P_8021Q)) - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); keys->nhoff += sizeof(*vlan); keys->thoff += sizeof(*vlan); @@ -350,14 +371,14 @@ PROG(VLAN)(struct __sk_buff *skb) vlan = bpf_flow_dissect_get_header(skb, sizeof(*vlan), &_vlan); if (!vlan) - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); keys->nhoff += sizeof(*vlan); keys->thoff += sizeof(*vlan); /* Only allow 8021AD + 8021Q double tagging and no triple tagging.*/ if (vlan->h_vlan_encapsulated_proto == bpf_htons(ETH_P_8021AD) || vlan->h_vlan_encapsulated_proto == bpf_htons(ETH_P_8021Q)) - return BPF_DROP; + return export_flow_keys(keys, BPF_DROP); keys->n_proto = vlan->h_vlan_encapsulated_proto; return parse_eth_proto(skb, vlan->h_vlan_encapsulated_proto); diff --git a/tools/testing/selftests/bpf/progs/test_global_data.c b/tools/testing/selftests/bpf/progs/test_global_data.c new file mode 100644 index 0000000000000000000000000000000000000000..5ab14e941980c8a71674e14607fe8996da102f43 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_global_data.c @@ -0,0 +1,106 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2019 Isovalent, Inc. + +#include +#include +#include + +#include "bpf_helpers.h" + +struct bpf_map_def SEC("maps") result_number = { + .type = BPF_MAP_TYPE_ARRAY, + .key_size = sizeof(__u32), + .value_size = sizeof(__u64), + .max_entries = 11, +}; + +struct bpf_map_def SEC("maps") result_string = { + .type = BPF_MAP_TYPE_ARRAY, + .key_size = sizeof(__u32), + .value_size = 32, + .max_entries = 5, +}; + +struct foo { + __u8 a; + __u32 b; + __u64 c; +}; + +struct bpf_map_def SEC("maps") result_struct = { + .type = BPF_MAP_TYPE_ARRAY, + .key_size = sizeof(__u32), + .value_size = sizeof(struct foo), + .max_entries = 5, +}; + +/* Relocation tests for __u64s. */ +static __u64 num0; +static __u64 num1 = 42; +static const __u64 num2 = 24; +static __u64 num3 = 0; +static __u64 num4 = 0xffeeff; +static const __u64 num5 = 0xabab; +static const __u64 num6 = 0xab; + +/* Relocation tests for strings. */ +static const char str0[32] = "abcdefghijklmnopqrstuvwxyz"; +static char str1[32] = "abcdefghijklmnopqrstuvwxyz"; +static char str2[32]; + +/* Relocation tests for structs. */ +static const struct foo struct0 = { + .a = 42, + .b = 0xfefeefef, + .c = 0x1111111111111111ULL, +}; +static struct foo struct1; +static const struct foo struct2; +static struct foo struct3 = { + .a = 41, + .b = 0xeeeeefef, + .c = 0x2111111111111111ULL, +}; + +#define test_reloc(map, num, var) \ + do { \ + __u32 key = num; \ + bpf_map_update_elem(&result_##map, &key, var, 0); \ + } while (0) + +SEC("static_data_load") +int load_static_data(struct __sk_buff *skb) +{ + static const __u64 bar = ~0; + + test_reloc(number, 0, &num0); + test_reloc(number, 1, &num1); + test_reloc(number, 2, &num2); + test_reloc(number, 3, &num3); + test_reloc(number, 4, &num4); + test_reloc(number, 5, &num5); + num4 = 1234; + test_reloc(number, 6, &num4); + test_reloc(number, 7, &num0); + test_reloc(number, 8, &num6); + + test_reloc(string, 0, str0); + test_reloc(string, 1, str1); + test_reloc(string, 2, str2); + str1[5] = 'x'; + test_reloc(string, 3, str1); + __builtin_memcpy(&str2[2], "hello", sizeof("hello")); + test_reloc(string, 4, str2); + + test_reloc(struct, 0, &struct0); + test_reloc(struct, 1, &struct1); + test_reloc(struct, 2, &struct2); + test_reloc(struct, 3, &struct3); + + test_reloc(number, 9, &struct0.c); + test_reloc(number, 10, &bar); + + return TC_ACT_OK; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/test_jhash.h b/tools/testing/selftests/bpf/progs/test_jhash.h new file mode 100644 index 0000000000000000000000000000000000000000..3d12c11a8d479536c8d5e58e8116845406e9af7a --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_jhash.h @@ -0,0 +1,70 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2019 Facebook + +typedef unsigned int u32; + +static __attribute__((always_inline)) u32 rol32(u32 word, unsigned int shift) +{ + return (word << shift) | (word >> ((-shift) & 31)); +} + +#define __jhash_mix(a, b, c) \ +{ \ + a -= c; a ^= rol32(c, 4); c += b; \ + b -= a; b ^= rol32(a, 6); a += c; \ + c -= b; c ^= rol32(b, 8); b += a; \ + a -= c; a ^= rol32(c, 16); c += b; \ + b -= a; b ^= rol32(a, 19); a += c; \ + c -= b; c ^= rol32(b, 4); b += a; \ +} + +#define __jhash_final(a, b, c) \ +{ \ + c ^= b; c -= rol32(b, 14); \ + a ^= c; a -= rol32(c, 11); \ + b ^= a; b -= rol32(a, 25); \ + c ^= b; c -= rol32(b, 16); \ + a ^= c; a -= rol32(c, 4); \ + b ^= a; b -= rol32(a, 14); \ + c ^= b; c -= rol32(b, 24); \ +} + +#define JHASH_INITVAL 0xdeadbeef + +static ATTR +u32 jhash(const void *key, u32 length, u32 initval) +{ + u32 a, b, c; + const unsigned char *k = key; + + a = b = c = JHASH_INITVAL + length + initval; + + while (length > 12) { + a += *(volatile u32 *)(k); + b += *(volatile u32 *)(k + 4); + c += *(volatile u32 *)(k + 8); + __jhash_mix(a, b, c); + length -= 12; + k += 12; + } + switch (length) { + case 12: c += (u32)k[11]<<24; + case 11: c += (u32)k[10]<<16; + case 10: c += (u32)k[9]<<8; + case 9: c += k[8]; + case 8: b += (u32)k[7]<<24; + case 7: b += (u32)k[6]<<16; + case 6: b += (u32)k[5]<<8; + case 5: b += k[4]; + case 4: a += (u32)k[3]<<24; + case 3: a += (u32)k[2]<<16; + case 2: a += (u32)k[1]<<8; + case 1: a += k[0]; + c ^= a; + __jhash_final(a, b, c); + case 0: /* Nothing left to add */ + break; + } + + return c; +} diff --git a/tools/testing/selftests/bpf/progs/test_skb_ctx.c b/tools/testing/selftests/bpf/progs/test_skb_ctx.c new file mode 100644 index 0000000000000000000000000000000000000000..7a80960d7df134602a8f27ac6e10b1d12347ae03 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_skb_ctx.c @@ -0,0 +1,21 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include "bpf_helpers.h" + +int _version SEC("version") = 1; +char _license[] SEC("license") = "GPL"; + +SEC("skb_ctx") +int process(struct __sk_buff *skb) +{ + #pragma clang loop unroll(full) + for (int i = 0; i < 5; i++) { + if (skb->cb[i] != i + 1) + return 1; + skb->cb[i]++; + } + skb->priority++; + + return 0; +} diff --git a/tools/testing/selftests/bpf/progs/test_sock_fields_kern.c b/tools/testing/selftests/bpf/progs/test_sock_fields_kern.c index 37328f1485384b756c3120f428a3348206a9eb47..1c39e4ccb7f1bf208b7debd62c633f1f719794c8 100644 --- a/tools/testing/selftests/bpf/progs/test_sock_fields_kern.c +++ b/tools/testing/selftests/bpf/progs/test_sock_fields_kern.c @@ -55,6 +55,31 @@ struct bpf_map_def SEC("maps") linum_map = { .max_entries = __NR_BPF_LINUM_ARRAY_IDX, }; +struct bpf_spinlock_cnt { + struct bpf_spin_lock lock; + __u32 cnt; +}; + +struct bpf_map_def SEC("maps") sk_pkt_out_cnt = { + .type = BPF_MAP_TYPE_SK_STORAGE, + .key_size = sizeof(int), + .value_size = sizeof(struct bpf_spinlock_cnt), + .max_entries = 0, + .map_flags = BPF_F_NO_PREALLOC, +}; + +BPF_ANNOTATE_KV_PAIR(sk_pkt_out_cnt, int, struct bpf_spinlock_cnt); + +struct bpf_map_def SEC("maps") sk_pkt_out_cnt10 = { + .type = BPF_MAP_TYPE_SK_STORAGE, + .key_size = sizeof(int), + .value_size = sizeof(struct bpf_spinlock_cnt), + .max_entries = 0, + .map_flags = BPF_F_NO_PREALLOC, +}; + +BPF_ANNOTATE_KV_PAIR(sk_pkt_out_cnt10, int, struct bpf_spinlock_cnt); + static bool is_loopback6(__u32 *a6) { return !a6[0] && !a6[1] && !a6[2] && a6[3] == bpf_htonl(1); @@ -120,7 +145,9 @@ static void tpcpy(struct bpf_tcp_sock *dst, SEC("cgroup_skb/egress") int egress_read_sock_fields(struct __sk_buff *skb) { + struct bpf_spinlock_cnt cli_cnt_init = { .lock = 0, .cnt = 0xeB9F }; __u32 srv_idx = ADDR_SRV_IDX, cli_idx = ADDR_CLI_IDX, result_idx; + struct bpf_spinlock_cnt *pkt_out_cnt, *pkt_out_cnt10; struct sockaddr_in6 *srv_sa6, *cli_sa6; struct bpf_tcp_sock *tp, *tp_ret; struct bpf_sock *sk, *sk_ret; @@ -161,6 +188,32 @@ int egress_read_sock_fields(struct __sk_buff *skb) skcpy(sk_ret, sk); tpcpy(tp_ret, tp); + if (result_idx == EGRESS_SRV_IDX) { + /* The userspace has created it for srv sk */ + pkt_out_cnt = bpf_sk_storage_get(&sk_pkt_out_cnt, sk, 0, 0); + pkt_out_cnt10 = bpf_sk_storage_get(&sk_pkt_out_cnt10, sk, + 0, 0); + } else { + pkt_out_cnt = bpf_sk_storage_get(&sk_pkt_out_cnt, sk, + &cli_cnt_init, + BPF_SK_STORAGE_GET_F_CREATE); + pkt_out_cnt10 = bpf_sk_storage_get(&sk_pkt_out_cnt10, + sk, &cli_cnt_init, + BPF_SK_STORAGE_GET_F_CREATE); + } + + if (!pkt_out_cnt || !pkt_out_cnt10) + RETURN; + + /* Even both cnt and cnt10 have lock defined in their BTF, + * intentionally one cnt takes lock while one does not + * as a test for the spinlock support in BPF_MAP_TYPE_SK_STORAGE. + */ + pkt_out_cnt->cnt += 1; + bpf_spin_lock(&pkt_out_cnt10->lock); + pkt_out_cnt10->cnt += 10; + bpf_spin_unlock(&pkt_out_cnt10->lock); + RETURN; } diff --git a/tools/testing/selftests/bpf/progs/test_sysctl_prog.c b/tools/testing/selftests/bpf/progs/test_sysctl_prog.c new file mode 100644 index 0000000000000000000000000000000000000000..a295cad805d7d4ccdc30277376737e57718fffd3 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_sysctl_prog.c @@ -0,0 +1,70 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2019 Facebook + +#include +#include + +#include +#include + +#include "bpf_helpers.h" +#include "bpf_util.h" + +/* Max supported length of a string with unsigned long in base 10 (pow2 - 1). */ +#define MAX_ULONG_STR_LEN 0xF + +/* Max supported length of sysctl value string (pow2). */ +#define MAX_VALUE_STR_LEN 0x40 + +static __always_inline int is_tcp_mem(struct bpf_sysctl *ctx) +{ + char tcp_mem_name[] = "net/ipv4/tcp_mem"; + unsigned char i; + char name[64]; + int ret; + + memset(name, 0, sizeof(name)); + ret = bpf_sysctl_get_name(ctx, name, sizeof(name), 0); + if (ret < 0 || ret != sizeof(tcp_mem_name) - 1) + return 0; + +#pragma clang loop unroll(full) + for (i = 0; i < sizeof(tcp_mem_name); ++i) + if (name[i] != tcp_mem_name[i]) + return 0; + + return 1; +} + +SEC("cgroup/sysctl") +int sysctl_tcp_mem(struct bpf_sysctl *ctx) +{ + unsigned long tcp_mem[3] = {0, 0, 0}; + char value[MAX_VALUE_STR_LEN]; + unsigned char i, off = 0; + int ret; + + if (ctx->write) + return 0; + + if (!is_tcp_mem(ctx)) + return 0; + + ret = bpf_sysctl_get_current_value(ctx, value, MAX_VALUE_STR_LEN); + if (ret < 0 || ret >= MAX_VALUE_STR_LEN) + return 0; + +#pragma clang loop unroll(full) + for (i = 0; i < ARRAY_SIZE(tcp_mem); ++i) { + ret = bpf_strtoul(value + off, MAX_ULONG_STR_LEN, 0, + tcp_mem + i); + if (ret <= 0 || ret > MAX_ULONG_STR_LEN) + return 0; + off += ret & MAX_ULONG_STR_LEN; + } + + + return tcp_mem[0] < tcp_mem[1] && tcp_mem[1] < tcp_mem[2]; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/test_tc_edt.c b/tools/testing/selftests/bpf/progs/test_tc_edt.c new file mode 100644 index 0000000000000000000000000000000000000000..3af64c470d64328922b97d38f17993a1d33d2380 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_tc_edt.c @@ -0,0 +1,109 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include +#include "bpf_helpers.h" +#include "bpf_endian.h" + +/* the maximum delay we are willing to add (drop packets beyond that) */ +#define TIME_HORIZON_NS (2000 * 1000 * 1000) +#define NS_PER_SEC 1000000000 +#define ECN_HORIZON_NS 5000000 +#define THROTTLE_RATE_BPS (5 * 1000 * 1000) + +/* flow_key => last_tstamp timestamp used */ +struct bpf_map_def SEC("maps") flow_map = { + .type = BPF_MAP_TYPE_HASH, + .key_size = sizeof(uint32_t), + .value_size = sizeof(uint64_t), + .max_entries = 1, +}; + +static inline int throttle_flow(struct __sk_buff *skb) +{ + int key = 0; + uint64_t *last_tstamp = bpf_map_lookup_elem(&flow_map, &key); + uint64_t delay_ns = ((uint64_t)skb->len) * NS_PER_SEC / + THROTTLE_RATE_BPS; + uint64_t now = bpf_ktime_get_ns(); + uint64_t tstamp, next_tstamp = 0; + + if (last_tstamp) + next_tstamp = *last_tstamp + delay_ns; + + tstamp = skb->tstamp; + if (tstamp < now) + tstamp = now; + + /* should we throttle? */ + if (next_tstamp <= tstamp) { + if (bpf_map_update_elem(&flow_map, &key, &tstamp, BPF_ANY)) + return TC_ACT_SHOT; + return TC_ACT_OK; + } + + /* do not queue past the time horizon */ + if (next_tstamp - now >= TIME_HORIZON_NS) + return TC_ACT_SHOT; + + /* set ecn bit, if needed */ + if (next_tstamp - now >= ECN_HORIZON_NS) + bpf_skb_ecn_set_ce(skb); + + if (bpf_map_update_elem(&flow_map, &key, &next_tstamp, BPF_EXIST)) + return TC_ACT_SHOT; + skb->tstamp = next_tstamp; + + return TC_ACT_OK; +} + +static inline int handle_tcp(struct __sk_buff *skb, struct tcphdr *tcp) +{ + void *data_end = (void *)(long)skb->data_end; + + /* drop malformed packets */ + if ((void *)(tcp + 1) > data_end) + return TC_ACT_SHOT; + + if (tcp->dest == bpf_htons(9000)) + return throttle_flow(skb); + + return TC_ACT_OK; +} + +static inline int handle_ipv4(struct __sk_buff *skb) +{ + void *data_end = (void *)(long)skb->data_end; + void *data = (void *)(long)skb->data; + struct iphdr *iph; + uint32_t ihl; + + /* drop malformed packets */ + if (data + sizeof(struct ethhdr) > data_end) + return TC_ACT_SHOT; + iph = (struct iphdr *)(data + sizeof(struct ethhdr)); + if ((void *)(iph + 1) > data_end) + return TC_ACT_SHOT; + ihl = iph->ihl * 4; + if (((void *)iph) + ihl > data_end) + return TC_ACT_SHOT; + + if (iph->protocol == IPPROTO_TCP) + return handle_tcp(skb, (struct tcphdr *)(((void *)iph) + ihl)); + + return TC_ACT_OK; +} + +SEC("cls_test") int tc_prog(struct __sk_buff *skb) +{ + if (skb->protocol == bpf_htons(ETH_P_IP)) + return handle_ipv4(skb); + + return TC_ACT_OK; +} + +char __license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/test_tc_tunnel.c b/tools/testing/selftests/bpf/progs/test_tc_tunnel.c new file mode 100644 index 0000000000000000000000000000000000000000..74370e7e286d5fe2f6c0b244a7fa7382242d2dfa --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_tc_tunnel.c @@ -0,0 +1,536 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* In-place tunneling */ + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "bpf_endian.h" +#include "bpf_helpers.h" + +static const int cfg_port = 8000; + +static const int cfg_udp_src = 20000; + +#define UDP_PORT 5555 +#define MPLS_OVER_UDP_PORT 6635 +#define ETH_OVER_UDP_PORT 7777 + +/* MPLS label 1000 with S bit (last label) set and ttl of 255. */ +static const __u32 mpls_label = __bpf_constant_htonl(1000 << 12 | + MPLS_LS_S_MASK | 0xff); + +struct gre_hdr { + __be16 flags; + __be16 protocol; +} __attribute__((packed)); + +union l4hdr { + struct udphdr udp; + struct gre_hdr gre; +}; + +struct v4hdr { + struct iphdr ip; + union l4hdr l4hdr; + __u8 pad[16]; /* enough space for L2 header */ +} __attribute__((packed)); + +struct v6hdr { + struct ipv6hdr ip; + union l4hdr l4hdr; + __u8 pad[16]; /* enough space for L2 header */ +} __attribute__((packed)); + +static __always_inline void set_ipv4_csum(struct iphdr *iph) +{ + __u16 *iph16 = (__u16 *)iph; + __u32 csum; + int i; + + iph->check = 0; + +#pragma clang loop unroll(full) + for (i = 0, csum = 0; i < sizeof(*iph) >> 1; i++) + csum += *iph16++; + + iph->check = ~((csum & 0xffff) + (csum >> 16)); +} + +static __always_inline int encap_ipv4(struct __sk_buff *skb, __u8 encap_proto, + __u16 l2_proto) +{ + __u16 udp_dst = UDP_PORT; + struct iphdr iph_inner; + struct v4hdr h_outer; + struct tcphdr tcph; + int olen, l2_len; + int tcp_off; + __u64 flags; + + /* Most tests encapsulate a packet into a tunnel with the same + * network protocol, and derive the outer header fields from + * the inner header. + * + * The 6in4 case tests different inner and outer protocols. As + * the inner is ipv6, but the outer expects an ipv4 header as + * input, manually build a struct iphdr based on the ipv6hdr. + */ + if (encap_proto == IPPROTO_IPV6) { + const __u32 saddr = (192 << 24) | (168 << 16) | (1 << 8) | 1; + const __u32 daddr = (192 << 24) | (168 << 16) | (1 << 8) | 2; + struct ipv6hdr iph6_inner; + + /* Read the IPv6 header */ + if (bpf_skb_load_bytes(skb, ETH_HLEN, &iph6_inner, + sizeof(iph6_inner)) < 0) + return TC_ACT_OK; + + /* Derive the IPv4 header fields from the IPv6 header */ + memset(&iph_inner, 0, sizeof(iph_inner)); + iph_inner.version = 4; + iph_inner.ihl = 5; + iph_inner.tot_len = bpf_htons(sizeof(iph6_inner) + + bpf_ntohs(iph6_inner.payload_len)); + iph_inner.ttl = iph6_inner.hop_limit - 1; + iph_inner.protocol = iph6_inner.nexthdr; + iph_inner.saddr = __bpf_constant_htonl(saddr); + iph_inner.daddr = __bpf_constant_htonl(daddr); + + tcp_off = sizeof(iph6_inner); + } else { + if (bpf_skb_load_bytes(skb, ETH_HLEN, &iph_inner, + sizeof(iph_inner)) < 0) + return TC_ACT_OK; + + tcp_off = sizeof(iph_inner); + } + + /* filter only packets we want */ + if (iph_inner.ihl != 5 || iph_inner.protocol != IPPROTO_TCP) + return TC_ACT_OK; + + if (bpf_skb_load_bytes(skb, ETH_HLEN + tcp_off, + &tcph, sizeof(tcph)) < 0) + return TC_ACT_OK; + + if (tcph.dest != __bpf_constant_htons(cfg_port)) + return TC_ACT_OK; + + olen = sizeof(h_outer.ip); + l2_len = 0; + + flags = BPF_F_ADJ_ROOM_FIXED_GSO | BPF_F_ADJ_ROOM_ENCAP_L3_IPV4; + + switch (l2_proto) { + case ETH_P_MPLS_UC: + l2_len = sizeof(mpls_label); + udp_dst = MPLS_OVER_UDP_PORT; + break; + case ETH_P_TEB: + l2_len = ETH_HLEN; + udp_dst = ETH_OVER_UDP_PORT; + break; + } + flags |= BPF_F_ADJ_ROOM_ENCAP_L2(l2_len); + + switch (encap_proto) { + case IPPROTO_GRE: + flags |= BPF_F_ADJ_ROOM_ENCAP_L4_GRE; + olen += sizeof(h_outer.l4hdr.gre); + h_outer.l4hdr.gre.protocol = bpf_htons(l2_proto); + h_outer.l4hdr.gre.flags = 0; + break; + case IPPROTO_UDP: + flags |= BPF_F_ADJ_ROOM_ENCAP_L4_UDP; + olen += sizeof(h_outer.l4hdr.udp); + h_outer.l4hdr.udp.source = __bpf_constant_htons(cfg_udp_src); + h_outer.l4hdr.udp.dest = bpf_htons(udp_dst); + h_outer.l4hdr.udp.check = 0; + h_outer.l4hdr.udp.len = bpf_htons(bpf_ntohs(iph_inner.tot_len) + + sizeof(h_outer.l4hdr.udp) + + l2_len); + break; + case IPPROTO_IPIP: + case IPPROTO_IPV6: + break; + default: + return TC_ACT_OK; + } + + /* add L2 encap (if specified) */ + switch (l2_proto) { + case ETH_P_MPLS_UC: + *((__u32 *)((__u8 *)&h_outer + olen)) = mpls_label; + break; + case ETH_P_TEB: + if (bpf_skb_load_bytes(skb, 0, (__u8 *)&h_outer + olen, + ETH_HLEN)) + return TC_ACT_SHOT; + break; + } + olen += l2_len; + + /* add room between mac and network header */ + if (bpf_skb_adjust_room(skb, olen, BPF_ADJ_ROOM_MAC, flags)) + return TC_ACT_SHOT; + + /* prepare new outer network header */ + h_outer.ip = iph_inner; + h_outer.ip.tot_len = bpf_htons(olen + + bpf_ntohs(h_outer.ip.tot_len)); + h_outer.ip.protocol = encap_proto; + + set_ipv4_csum((void *)&h_outer.ip); + + /* store new outer network header */ + if (bpf_skb_store_bytes(skb, ETH_HLEN, &h_outer, olen, + BPF_F_INVALIDATE_HASH) < 0) + return TC_ACT_SHOT; + + /* if changing outer proto type, update eth->h_proto */ + if (encap_proto == IPPROTO_IPV6) { + struct ethhdr eth; + + if (bpf_skb_load_bytes(skb, 0, ð, sizeof(eth)) < 0) + return TC_ACT_SHOT; + eth.h_proto = bpf_htons(ETH_P_IP); + if (bpf_skb_store_bytes(skb, 0, ð, sizeof(eth), 0) < 0) + return TC_ACT_SHOT; + } + + return TC_ACT_OK; +} + +static __always_inline int encap_ipv6(struct __sk_buff *skb, __u8 encap_proto, + __u16 l2_proto) +{ + __u16 udp_dst = UDP_PORT; + struct ipv6hdr iph_inner; + struct v6hdr h_outer; + struct tcphdr tcph; + int olen, l2_len; + __u16 tot_len; + __u64 flags; + + if (bpf_skb_load_bytes(skb, ETH_HLEN, &iph_inner, + sizeof(iph_inner)) < 0) + return TC_ACT_OK; + + /* filter only packets we want */ + if (bpf_skb_load_bytes(skb, ETH_HLEN + sizeof(iph_inner), + &tcph, sizeof(tcph)) < 0) + return TC_ACT_OK; + + if (tcph.dest != __bpf_constant_htons(cfg_port)) + return TC_ACT_OK; + + olen = sizeof(h_outer.ip); + l2_len = 0; + + flags = BPF_F_ADJ_ROOM_FIXED_GSO | BPF_F_ADJ_ROOM_ENCAP_L3_IPV6; + + switch (l2_proto) { + case ETH_P_MPLS_UC: + l2_len = sizeof(mpls_label); + udp_dst = MPLS_OVER_UDP_PORT; + break; + case ETH_P_TEB: + l2_len = ETH_HLEN; + udp_dst = ETH_OVER_UDP_PORT; + break; + } + flags |= BPF_F_ADJ_ROOM_ENCAP_L2(l2_len); + + switch (encap_proto) { + case IPPROTO_GRE: + flags |= BPF_F_ADJ_ROOM_ENCAP_L4_GRE; + olen += sizeof(h_outer.l4hdr.gre); + h_outer.l4hdr.gre.protocol = bpf_htons(l2_proto); + h_outer.l4hdr.gre.flags = 0; + break; + case IPPROTO_UDP: + flags |= BPF_F_ADJ_ROOM_ENCAP_L4_UDP; + olen += sizeof(h_outer.l4hdr.udp); + h_outer.l4hdr.udp.source = __bpf_constant_htons(cfg_udp_src); + h_outer.l4hdr.udp.dest = bpf_htons(udp_dst); + tot_len = bpf_ntohs(iph_inner.payload_len) + sizeof(iph_inner) + + sizeof(h_outer.l4hdr.udp); + h_outer.l4hdr.udp.check = 0; + h_outer.l4hdr.udp.len = bpf_htons(tot_len); + break; + case IPPROTO_IPV6: + break; + default: + return TC_ACT_OK; + } + + /* add L2 encap (if specified) */ + switch (l2_proto) { + case ETH_P_MPLS_UC: + *((__u32 *)((__u8 *)&h_outer + olen)) = mpls_label; + break; + case ETH_P_TEB: + if (bpf_skb_load_bytes(skb, 0, (__u8 *)&h_outer + olen, + ETH_HLEN)) + return TC_ACT_SHOT; + break; + } + olen += l2_len; + + /* add room between mac and network header */ + if (bpf_skb_adjust_room(skb, olen, BPF_ADJ_ROOM_MAC, flags)) + return TC_ACT_SHOT; + + /* prepare new outer network header */ + h_outer.ip = iph_inner; + h_outer.ip.payload_len = bpf_htons(olen + + bpf_ntohs(h_outer.ip.payload_len)); + + h_outer.ip.nexthdr = encap_proto; + + /* store new outer network header */ + if (bpf_skb_store_bytes(skb, ETH_HLEN, &h_outer, olen, + BPF_F_INVALIDATE_HASH) < 0) + return TC_ACT_SHOT; + + return TC_ACT_OK; +} + +SEC("encap_ipip_none") +int __encap_ipip_none(struct __sk_buff *skb) +{ + if (skb->protocol == __bpf_constant_htons(ETH_P_IP)) + return encap_ipv4(skb, IPPROTO_IPIP, ETH_P_IP); + else + return TC_ACT_OK; +} + +SEC("encap_gre_none") +int __encap_gre_none(struct __sk_buff *skb) +{ + if (skb->protocol == __bpf_constant_htons(ETH_P_IP)) + return encap_ipv4(skb, IPPROTO_GRE, ETH_P_IP); + else + return TC_ACT_OK; +} + +SEC("encap_gre_mpls") +int __encap_gre_mpls(struct __sk_buff *skb) +{ + if (skb->protocol == __bpf_constant_htons(ETH_P_IP)) + return encap_ipv4(skb, IPPROTO_GRE, ETH_P_MPLS_UC); + else + return TC_ACT_OK; +} + +SEC("encap_gre_eth") +int __encap_gre_eth(struct __sk_buff *skb) +{ + if (skb->protocol == __bpf_constant_htons(ETH_P_IP)) + return encap_ipv4(skb, IPPROTO_GRE, ETH_P_TEB); + else + return TC_ACT_OK; +} + +SEC("encap_udp_none") +int __encap_udp_none(struct __sk_buff *skb) +{ + if (skb->protocol == __bpf_constant_htons(ETH_P_IP)) + return encap_ipv4(skb, IPPROTO_UDP, ETH_P_IP); + else + return TC_ACT_OK; +} + +SEC("encap_udp_mpls") +int __encap_udp_mpls(struct __sk_buff *skb) +{ + if (skb->protocol == __bpf_constant_htons(ETH_P_IP)) + return encap_ipv4(skb, IPPROTO_UDP, ETH_P_MPLS_UC); + else + return TC_ACT_OK; +} + +SEC("encap_udp_eth") +int __encap_udp_eth(struct __sk_buff *skb) +{ + if (skb->protocol == __bpf_constant_htons(ETH_P_IP)) + return encap_ipv4(skb, IPPROTO_UDP, ETH_P_TEB); + else + return TC_ACT_OK; +} + +SEC("encap_sit_none") +int __encap_sit_none(struct __sk_buff *skb) +{ + if (skb->protocol == __bpf_constant_htons(ETH_P_IPV6)) + return encap_ipv4(skb, IPPROTO_IPV6, ETH_P_IP); + else + return TC_ACT_OK; +} + +SEC("encap_ip6tnl_none") +int __encap_ip6tnl_none(struct __sk_buff *skb) +{ + if (skb->protocol == __bpf_constant_htons(ETH_P_IPV6)) + return encap_ipv6(skb, IPPROTO_IPV6, ETH_P_IPV6); + else + return TC_ACT_OK; +} + +SEC("encap_ip6gre_none") +int __encap_ip6gre_none(struct __sk_buff *skb) +{ + if (skb->protocol == __bpf_constant_htons(ETH_P_IPV6)) + return encap_ipv6(skb, IPPROTO_GRE, ETH_P_IPV6); + else + return TC_ACT_OK; +} + +SEC("encap_ip6gre_mpls") +int __encap_ip6gre_mpls(struct __sk_buff *skb) +{ + if (skb->protocol == __bpf_constant_htons(ETH_P_IPV6)) + return encap_ipv6(skb, IPPROTO_GRE, ETH_P_MPLS_UC); + else + return TC_ACT_OK; +} + +SEC("encap_ip6gre_eth") +int __encap_ip6gre_eth(struct __sk_buff *skb) +{ + if (skb->protocol == __bpf_constant_htons(ETH_P_IPV6)) + return encap_ipv6(skb, IPPROTO_GRE, ETH_P_TEB); + else + return TC_ACT_OK; +} + +SEC("encap_ip6udp_none") +int __encap_ip6udp_none(struct __sk_buff *skb) +{ + if (skb->protocol == __bpf_constant_htons(ETH_P_IPV6)) + return encap_ipv6(skb, IPPROTO_UDP, ETH_P_IPV6); + else + return TC_ACT_OK; +} + +SEC("encap_ip6udp_mpls") +int __encap_ip6udp_mpls(struct __sk_buff *skb) +{ + if (skb->protocol == __bpf_constant_htons(ETH_P_IPV6)) + return encap_ipv6(skb, IPPROTO_UDP, ETH_P_MPLS_UC); + else + return TC_ACT_OK; +} + +SEC("encap_ip6udp_eth") +int __encap_ip6udp_eth(struct __sk_buff *skb) +{ + if (skb->protocol == __bpf_constant_htons(ETH_P_IPV6)) + return encap_ipv6(skb, IPPROTO_UDP, ETH_P_TEB); + else + return TC_ACT_OK; +} + +static int decap_internal(struct __sk_buff *skb, int off, int len, char proto) +{ + char buf[sizeof(struct v6hdr)]; + struct gre_hdr greh; + struct udphdr udph; + int olen = len; + + switch (proto) { + case IPPROTO_IPIP: + case IPPROTO_IPV6: + break; + case IPPROTO_GRE: + olen += sizeof(struct gre_hdr); + if (bpf_skb_load_bytes(skb, off + len, &greh, sizeof(greh)) < 0) + return TC_ACT_OK; + switch (bpf_ntohs(greh.protocol)) { + case ETH_P_MPLS_UC: + olen += sizeof(mpls_label); + break; + case ETH_P_TEB: + olen += ETH_HLEN; + break; + } + break; + case IPPROTO_UDP: + olen += sizeof(struct udphdr); + if (bpf_skb_load_bytes(skb, off + len, &udph, sizeof(udph)) < 0) + return TC_ACT_OK; + switch (bpf_ntohs(udph.dest)) { + case MPLS_OVER_UDP_PORT: + olen += sizeof(mpls_label); + break; + case ETH_OVER_UDP_PORT: + olen += ETH_HLEN; + break; + } + break; + default: + return TC_ACT_OK; + } + + if (bpf_skb_adjust_room(skb, -olen, BPF_ADJ_ROOM_MAC, + BPF_F_ADJ_ROOM_FIXED_GSO)) + return TC_ACT_SHOT; + + return TC_ACT_OK; +} + +static int decap_ipv4(struct __sk_buff *skb) +{ + struct iphdr iph_outer; + + if (bpf_skb_load_bytes(skb, ETH_HLEN, &iph_outer, + sizeof(iph_outer)) < 0) + return TC_ACT_OK; + + if (iph_outer.ihl != 5) + return TC_ACT_OK; + + return decap_internal(skb, ETH_HLEN, sizeof(iph_outer), + iph_outer.protocol); +} + +static int decap_ipv6(struct __sk_buff *skb) +{ + struct ipv6hdr iph_outer; + + if (bpf_skb_load_bytes(skb, ETH_HLEN, &iph_outer, + sizeof(iph_outer)) < 0) + return TC_ACT_OK; + + return decap_internal(skb, ETH_HLEN, sizeof(iph_outer), + iph_outer.nexthdr); +} + +SEC("decap") +int decap_f(struct __sk_buff *skb) +{ + switch (skb->protocol) { + case __bpf_constant_htons(ETH_P_IP): + return decap_ipv4(skb); + case __bpf_constant_htons(ETH_P_IPV6): + return decap_ipv6(skb); + default: + /* does not match, ignore */ + return TC_ACT_OK; + } +} + +char __license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/test_tcp_check_syncookie_kern.c b/tools/testing/selftests/bpf/progs/test_tcp_check_syncookie_kern.c new file mode 100644 index 0000000000000000000000000000000000000000..1ab095bcacd844ea1ed57ad728b166a425e363ad --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_tcp_check_syncookie_kern.c @@ -0,0 +1,129 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2018 Facebook +// Copyright (c) 2019 Cloudflare + +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "bpf_helpers.h" +#include "bpf_endian.h" + +struct bpf_map_def SEC("maps") results = { + .type = BPF_MAP_TYPE_ARRAY, + .key_size = sizeof(__u32), + .value_size = sizeof(__u64), + .max_entries = 1, +}; + +static __always_inline void check_syncookie(void *ctx, void *data, + void *data_end) +{ + struct bpf_sock_tuple tup; + struct bpf_sock *sk; + struct ethhdr *ethh; + struct iphdr *ipv4h; + struct ipv6hdr *ipv6h; + struct tcphdr *tcph; + int ret; + __u32 key = 0; + __u64 value = 1; + + ethh = data; + if (ethh + 1 > data_end) + return; + + switch (bpf_ntohs(ethh->h_proto)) { + case ETH_P_IP: + ipv4h = data + sizeof(struct ethhdr); + if (ipv4h + 1 > data_end) + return; + + if (ipv4h->ihl != 5) + return; + + tcph = data + sizeof(struct ethhdr) + sizeof(struct iphdr); + if (tcph + 1 > data_end) + return; + + tup.ipv4.saddr = ipv4h->saddr; + tup.ipv4.daddr = ipv4h->daddr; + tup.ipv4.sport = tcph->source; + tup.ipv4.dport = tcph->dest; + + sk = bpf_skc_lookup_tcp(ctx, &tup, sizeof(tup.ipv4), + BPF_F_CURRENT_NETNS, 0); + if (!sk) + return; + + if (sk->state != BPF_TCP_LISTEN) + goto release; + + ret = bpf_tcp_check_syncookie(sk, ipv4h, sizeof(*ipv4h), + tcph, sizeof(*tcph)); + break; + + case ETH_P_IPV6: + ipv6h = data + sizeof(struct ethhdr); + if (ipv6h + 1 > data_end) + return; + + if (ipv6h->nexthdr != IPPROTO_TCP) + return; + + tcph = data + sizeof(struct ethhdr) + sizeof(struct ipv6hdr); + if (tcph + 1 > data_end) + return; + + memcpy(tup.ipv6.saddr, &ipv6h->saddr, sizeof(tup.ipv6.saddr)); + memcpy(tup.ipv6.daddr, &ipv6h->daddr, sizeof(tup.ipv6.daddr)); + tup.ipv6.sport = tcph->source; + tup.ipv6.dport = tcph->dest; + + sk = bpf_skc_lookup_tcp(ctx, &tup, sizeof(tup.ipv6), + BPF_F_CURRENT_NETNS, 0); + if (!sk) + return; + + if (sk->state != BPF_TCP_LISTEN) + goto release; + + ret = bpf_tcp_check_syncookie(sk, ipv6h, sizeof(*ipv6h), + tcph, sizeof(*tcph)); + break; + + default: + return; + } + + if (ret == 0) + bpf_map_update_elem(&results, &key, &value, 0); + +release: + bpf_sk_release(sk); +} + +SEC("clsact/check_syncookie") +int check_syncookie_clsact(struct __sk_buff *skb) +{ + check_syncookie(skb, (void *)(long)skb->data, + (void *)(long)skb->data_end); + return TC_ACT_OK; +} + +SEC("xdp/check_syncookie") +int check_syncookie_xdp(struct xdp_md *ctx) +{ + check_syncookie(ctx, (void *)(long)ctx->data, + (void *)(long)ctx->data_end); + return XDP_PASS; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/test_tcpbpf_kern.c b/tools/testing/selftests/bpf/progs/test_tcpbpf_kern.c index 74f73b33a7b07d0ec21e6e6ba53291ea197f6e75..c7c3240e0dd40beec59dc94777844c5e1c0d882f 100644 --- a/tools/testing/selftests/bpf/progs/test_tcpbpf_kern.c +++ b/tools/testing/selftests/bpf/progs/test_tcpbpf_kern.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0 #include #include +#include #include #include #include @@ -9,7 +10,6 @@ #include #include #include -#include #include "bpf_helpers.h" #include "bpf_endian.h" #include "test_tcpbpf.h" diff --git a/tools/testing/selftests/bpf/progs/test_tcpnotify_kern.c b/tools/testing/selftests/bpf/progs/test_tcpnotify_kern.c index edbca203ce2dd972a6c5edd91eca9036a2fbf84a..ec6db6e64c41ac06a929f03e709e88ba23359141 100644 --- a/tools/testing/selftests/bpf/progs/test_tcpnotify_kern.c +++ b/tools/testing/selftests/bpf/progs/test_tcpnotify_kern.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0 #include #include +#include #include #include #include @@ -9,7 +10,6 @@ #include #include #include -#include #include "bpf_helpers.h" #include "bpf_endian.h" #include "test_tcpnotify.h" diff --git a/tools/testing/selftests/bpf/progs/test_verif_scale1.c b/tools/testing/selftests/bpf/progs/test_verif_scale1.c new file mode 100644 index 0000000000000000000000000000000000000000..f3236ce35f313492e3e10e5c46cafbaa9aa08021 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_verif_scale1.c @@ -0,0 +1,30 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2019 Facebook +#include +#include "bpf_helpers.h" +#define ATTR __attribute__((noinline)) +#include "test_jhash.h" + +SEC("scale90_noinline") +int balancer_ingress(struct __sk_buff *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + void *ptr; + int ret = 0, nh_off, i = 0; + + nh_off = 14; + + /* pragma unroll doesn't work on large loops */ + +#define C do { \ + ptr = data + i; \ + if (ptr + nh_off > data_end) \ + break; \ + ctx->tc_index = jhash(ptr, nh_off, ctx->cb[0] + i++); \ + } while (0); +#define C30 C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C; + C30;C30;C30; /* 90 calls */ + return 0; +} +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/test_verif_scale2.c b/tools/testing/selftests/bpf/progs/test_verif_scale2.c new file mode 100644 index 0000000000000000000000000000000000000000..77830693eccb21061441191c6641fee2201afc51 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_verif_scale2.c @@ -0,0 +1,30 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2019 Facebook +#include +#include "bpf_helpers.h" +#define ATTR __attribute__((always_inline)) +#include "test_jhash.h" + +SEC("scale90_inline") +int balancer_ingress(struct __sk_buff *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + void *ptr; + int ret = 0, nh_off, i = 0; + + nh_off = 14; + + /* pragma unroll doesn't work on large loops */ + +#define C do { \ + ptr = data + i; \ + if (ptr + nh_off > data_end) \ + break; \ + ctx->tc_index = jhash(ptr, nh_off, ctx->cb[0] + i++); \ + } while (0); +#define C30 C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C; + C30;C30;C30; /* 90 calls */ + return 0; +} +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/test_verif_scale3.c b/tools/testing/selftests/bpf/progs/test_verif_scale3.c new file mode 100644 index 0000000000000000000000000000000000000000..1848da04ea412237d8432bb54f4ff532c4412d42 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_verif_scale3.c @@ -0,0 +1,30 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2019 Facebook +#include +#include "bpf_helpers.h" +#define ATTR __attribute__((noinline)) +#include "test_jhash.h" + +SEC("scale90_noinline32") +int balancer_ingress(struct __sk_buff *ctx) +{ + void *data_end = (void *)(long)ctx->data_end; + void *data = (void *)(long)ctx->data; + void *ptr; + int ret = 0, nh_off, i = 0; + + nh_off = 32; + + /* pragma unroll doesn't work on large loops */ + +#define C do { \ + ptr = data + i; \ + if (ptr + nh_off > data_end) \ + break; \ + ctx->tc_index = jhash(ptr, nh_off, ctx->cb[0] + i++); \ + } while (0); +#define C30 C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C;C; + C30;C30;C30; /* 90 calls */ + return 0; +} +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_btf.c b/tools/testing/selftests/bpf/test_btf.c index ec5794e4205bc04dabcaa20532cfe4cd2fcf6bbd..42c1ce9889453257c0cef5f62895985f0f273104 100644 --- a/tools/testing/selftests/bpf/test_btf.c +++ b/tools/testing/selftests/bpf/test_btf.c @@ -24,6 +24,7 @@ #include "bpf_rlimit.h" #include "bpf_util.h" +#include "test_btf.h" #define MAX_INSNS 512 #define MAX_SUBPROGS 16 @@ -58,63 +59,6 @@ static int __base_pr(enum libbpf_print_level level __attribute__((unused)), return vfprintf(stderr, format, args); } -#define BTF_INFO_ENC(kind, kind_flag, vlen) \ - ((!!(kind_flag) << 31) | ((kind) << 24) | ((vlen) & BTF_MAX_VLEN)) - -#define BTF_TYPE_ENC(name, info, size_or_type) \ - (name), (info), (size_or_type) - -#define BTF_INT_ENC(encoding, bits_offset, nr_bits) \ - ((encoding) << 24 | (bits_offset) << 16 | (nr_bits)) -#define BTF_TYPE_INT_ENC(name, encoding, bits_offset, bits, sz) \ - BTF_TYPE_ENC(name, BTF_INFO_ENC(BTF_KIND_INT, 0, 0), sz), \ - BTF_INT_ENC(encoding, bits_offset, bits) - -#define BTF_FWD_ENC(name, kind_flag) \ - BTF_TYPE_ENC(name, BTF_INFO_ENC(BTF_KIND_FWD, kind_flag, 0), 0) - -#define BTF_ARRAY_ENC(type, index_type, nr_elems) \ - (type), (index_type), (nr_elems) -#define BTF_TYPE_ARRAY_ENC(type, index_type, nr_elems) \ - BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_ARRAY, 0, 0), 0), \ - BTF_ARRAY_ENC(type, index_type, nr_elems) - -#define BTF_STRUCT_ENC(name, nr_elems, sz) \ - BTF_TYPE_ENC(name, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, nr_elems), sz) - -#define BTF_UNION_ENC(name, nr_elems, sz) \ - BTF_TYPE_ENC(name, BTF_INFO_ENC(BTF_KIND_UNION, 0, nr_elems), sz) - -#define BTF_MEMBER_ENC(name, type, bits_offset) \ - (name), (type), (bits_offset) -#define BTF_ENUM_ENC(name, val) (name), (val) -#define BTF_MEMBER_OFFSET(bitfield_size, bits_offset) \ - ((bitfield_size) << 24 | (bits_offset)) - -#define BTF_TYPEDEF_ENC(name, type) \ - BTF_TYPE_ENC(name, BTF_INFO_ENC(BTF_KIND_TYPEDEF, 0, 0), type) - -#define BTF_PTR_ENC(type) \ - BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_PTR, 0, 0), type) - -#define BTF_CONST_ENC(type) \ - BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_CONST, 0, 0), type) - -#define BTF_VOLATILE_ENC(type) \ - BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_VOLATILE, 0, 0), type) - -#define BTF_RESTRICT_ENC(type) \ - BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_RESTRICT, 0, 0), type) - -#define BTF_FUNC_PROTO_ENC(ret_type, nargs) \ - BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_FUNC_PROTO, 0, nargs), ret_type) - -#define BTF_FUNC_PROTO_ARG_ENC(name, type) \ - (name), (type) - -#define BTF_FUNC_ENC(name, func_proto) \ - BTF_TYPE_ENC(name, BTF_INFO_ENC(BTF_KIND_FUNC, 0, 0), func_proto) - #define BTF_END_RAW 0xdeadbeef #define NAME_TBD 0xdeadb33f @@ -291,7 +235,6 @@ static struct btf_raw_test raw_tests[] = { .value_type_id = 3, .max_entries = 4, }, - { .descr = "struct test #3 Invalid member offset", .raw_types = { @@ -319,7 +262,664 @@ static struct btf_raw_test raw_tests[] = { .btf_load_err = true, .err_str = "Invalid member bits_offset", }, - +/* + * struct A { + * unsigned long long m; + * int n; + * char o; + * [3 bytes hole] + * int p[8]; + * }; + */ +{ + .descr = "global data test #1", + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + /* unsigned long long */ + BTF_TYPE_INT_ENC(0, 0, 0, 64, 8), /* [2] */ + /* char */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 8, 1), /* [3] */ + /* int[8] */ + BTF_TYPE_ARRAY_ENC(1, 1, 8), /* [4] */ + /* struct A { */ /* [5] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 4), 48), + BTF_MEMBER_ENC(NAME_TBD, 2, 0), /* unsigned long long m;*/ + BTF_MEMBER_ENC(NAME_TBD, 1, 64),/* int n; */ + BTF_MEMBER_ENC(NAME_TBD, 3, 96),/* char o; */ + BTF_MEMBER_ENC(NAME_TBD, 4, 128),/* int p[8] */ + /* } */ + BTF_END_RAW, + }, + .str_sec = "\0A\0m\0n\0o\0p", + .str_sec_size = sizeof("\0A\0m\0n\0o\0p"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = "struct_test1_map", + .key_size = sizeof(int), + .value_size = 48, + .key_type_id = 1, + .value_type_id = 5, + .max_entries = 4, +}, +/* + * struct A { + * unsigned long long m; + * int n; + * char o; + * [3 bytes hole] + * int p[8]; + * }; + * static struct A t; <- in .bss + */ +{ + .descr = "global data test #2", + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + /* unsigned long long */ + BTF_TYPE_INT_ENC(0, 0, 0, 64, 8), /* [2] */ + /* char */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 8, 1), /* [3] */ + /* int[8] */ + BTF_TYPE_ARRAY_ENC(1, 1, 8), /* [4] */ + /* struct A { */ /* [5] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 4), 48), + BTF_MEMBER_ENC(NAME_TBD, 2, 0), /* unsigned long long m;*/ + BTF_MEMBER_ENC(NAME_TBD, 1, 64),/* int n; */ + BTF_MEMBER_ENC(NAME_TBD, 3, 96),/* char o; */ + BTF_MEMBER_ENC(NAME_TBD, 4, 128),/* int p[8] */ + /* } */ + /* static struct A t */ + BTF_VAR_ENC(NAME_TBD, 5, 0), /* [6] */ + /* .bss section */ /* [7] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 48), + BTF_VAR_SECINFO_ENC(6, 0, 48), + BTF_END_RAW, + }, + .str_sec = "\0A\0m\0n\0o\0p\0t\0.bss", + .str_sec_size = sizeof("\0A\0m\0n\0o\0p\0t\0.bss"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 48, + .key_type_id = 0, + .value_type_id = 7, + .max_entries = 1, +}, +{ + .descr = "global data test #3", + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + /* static int t */ + BTF_VAR_ENC(NAME_TBD, 1, 0), /* [2] */ + /* .bss section */ /* [3] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 4), + BTF_VAR_SECINFO_ENC(2, 0, 4), + BTF_END_RAW, + }, + .str_sec = "\0t\0.bss", + .str_sec_size = sizeof("\0t\0.bss"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 4, + .key_type_id = 0, + .value_type_id = 3, + .max_entries = 1, +}, +{ + .descr = "global data test #4, unsupported linkage", + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + /* static int t */ + BTF_VAR_ENC(NAME_TBD, 1, 2), /* [2] */ + /* .bss section */ /* [3] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 4), + BTF_VAR_SECINFO_ENC(2, 0, 4), + BTF_END_RAW, + }, + .str_sec = "\0t\0.bss", + .str_sec_size = sizeof("\0t\0.bss"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 4, + .key_type_id = 0, + .value_type_id = 3, + .max_entries = 1, + .btf_load_err = true, + .err_str = "Linkage not supported", +}, +{ + .descr = "global data test #5, invalid var type", + .raw_types = { + /* static void t */ + BTF_VAR_ENC(NAME_TBD, 0, 0), /* [1] */ + /* .bss section */ /* [2] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 4), + BTF_VAR_SECINFO_ENC(1, 0, 4), + BTF_END_RAW, + }, + .str_sec = "\0t\0.bss", + .str_sec_size = sizeof("\0t\0.bss"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 4, + .key_type_id = 0, + .value_type_id = 2, + .max_entries = 1, + .btf_load_err = true, + .err_str = "Invalid type_id", +}, +{ + .descr = "global data test #6, invalid var type (fwd type)", + .raw_types = { + /* union A */ + BTF_TYPE_ENC(NAME_TBD, + BTF_INFO_ENC(BTF_KIND_FWD, 1, 0), 0), /* [1] */ + /* static union A t */ + BTF_VAR_ENC(NAME_TBD, 1, 0), /* [2] */ + /* .bss section */ /* [3] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 4), + BTF_VAR_SECINFO_ENC(2, 0, 4), + BTF_END_RAW, + }, + .str_sec = "\0A\0t\0.bss", + .str_sec_size = sizeof("\0A\0t\0.bss"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 4, + .key_type_id = 0, + .value_type_id = 2, + .max_entries = 1, + .btf_load_err = true, + .err_str = "Invalid type", +}, +{ + .descr = "global data test #7, invalid var type (fwd type)", + .raw_types = { + /* union A */ + BTF_TYPE_ENC(NAME_TBD, + BTF_INFO_ENC(BTF_KIND_FWD, 1, 0), 0), /* [1] */ + /* static union A t */ + BTF_VAR_ENC(NAME_TBD, 1, 0), /* [2] */ + /* .bss section */ /* [3] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 4), + BTF_VAR_SECINFO_ENC(1, 0, 4), + BTF_END_RAW, + }, + .str_sec = "\0A\0t\0.bss", + .str_sec_size = sizeof("\0A\0t\0.bss"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 4, + .key_type_id = 0, + .value_type_id = 2, + .max_entries = 1, + .btf_load_err = true, + .err_str = "Invalid type", +}, +{ + .descr = "global data test #8, invalid var size", + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + /* unsigned long long */ + BTF_TYPE_INT_ENC(0, 0, 0, 64, 8), /* [2] */ + /* char */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 8, 1), /* [3] */ + /* int[8] */ + BTF_TYPE_ARRAY_ENC(1, 1, 8), /* [4] */ + /* struct A { */ /* [5] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 4), 48), + BTF_MEMBER_ENC(NAME_TBD, 2, 0), /* unsigned long long m;*/ + BTF_MEMBER_ENC(NAME_TBD, 1, 64),/* int n; */ + BTF_MEMBER_ENC(NAME_TBD, 3, 96),/* char o; */ + BTF_MEMBER_ENC(NAME_TBD, 4, 128),/* int p[8] */ + /* } */ + /* static struct A t */ + BTF_VAR_ENC(NAME_TBD, 5, 0), /* [6] */ + /* .bss section */ /* [7] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 48), + BTF_VAR_SECINFO_ENC(6, 0, 47), + BTF_END_RAW, + }, + .str_sec = "\0A\0m\0n\0o\0p\0t\0.bss", + .str_sec_size = sizeof("\0A\0m\0n\0o\0p\0t\0.bss"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 48, + .key_type_id = 0, + .value_type_id = 7, + .max_entries = 1, + .btf_load_err = true, + .err_str = "Invalid size", +}, +{ + .descr = "global data test #9, invalid var size", + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + /* unsigned long long */ + BTF_TYPE_INT_ENC(0, 0, 0, 64, 8), /* [2] */ + /* char */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 8, 1), /* [3] */ + /* int[8] */ + BTF_TYPE_ARRAY_ENC(1, 1, 8), /* [4] */ + /* struct A { */ /* [5] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 4), 48), + BTF_MEMBER_ENC(NAME_TBD, 2, 0), /* unsigned long long m;*/ + BTF_MEMBER_ENC(NAME_TBD, 1, 64),/* int n; */ + BTF_MEMBER_ENC(NAME_TBD, 3, 96),/* char o; */ + BTF_MEMBER_ENC(NAME_TBD, 4, 128),/* int p[8] */ + /* } */ + /* static struct A t */ + BTF_VAR_ENC(NAME_TBD, 5, 0), /* [6] */ + /* .bss section */ /* [7] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 46), + BTF_VAR_SECINFO_ENC(6, 0, 48), + BTF_END_RAW, + }, + .str_sec = "\0A\0m\0n\0o\0p\0t\0.bss", + .str_sec_size = sizeof("\0A\0m\0n\0o\0p\0t\0.bss"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 48, + .key_type_id = 0, + .value_type_id = 7, + .max_entries = 1, + .btf_load_err = true, + .err_str = "Invalid size", +}, +{ + .descr = "global data test #10, invalid var size", + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + /* unsigned long long */ + BTF_TYPE_INT_ENC(0, 0, 0, 64, 8), /* [2] */ + /* char */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 8, 1), /* [3] */ + /* int[8] */ + BTF_TYPE_ARRAY_ENC(1, 1, 8), /* [4] */ + /* struct A { */ /* [5] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 4), 48), + BTF_MEMBER_ENC(NAME_TBD, 2, 0), /* unsigned long long m;*/ + BTF_MEMBER_ENC(NAME_TBD, 1, 64),/* int n; */ + BTF_MEMBER_ENC(NAME_TBD, 3, 96),/* char o; */ + BTF_MEMBER_ENC(NAME_TBD, 4, 128),/* int p[8] */ + /* } */ + /* static struct A t */ + BTF_VAR_ENC(NAME_TBD, 5, 0), /* [6] */ + /* .bss section */ /* [7] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 46), + BTF_VAR_SECINFO_ENC(6, 0, 46), + BTF_END_RAW, + }, + .str_sec = "\0A\0m\0n\0o\0p\0t\0.bss", + .str_sec_size = sizeof("\0A\0m\0n\0o\0p\0t\0.bss"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 48, + .key_type_id = 0, + .value_type_id = 7, + .max_entries = 1, + .btf_load_err = true, + .err_str = "Invalid size", +}, +{ + .descr = "global data test #11, multiple section members", + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + /* unsigned long long */ + BTF_TYPE_INT_ENC(0, 0, 0, 64, 8), /* [2] */ + /* char */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 8, 1), /* [3] */ + /* int[8] */ + BTF_TYPE_ARRAY_ENC(1, 1, 8), /* [4] */ + /* struct A { */ /* [5] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 4), 48), + BTF_MEMBER_ENC(NAME_TBD, 2, 0), /* unsigned long long m;*/ + BTF_MEMBER_ENC(NAME_TBD, 1, 64),/* int n; */ + BTF_MEMBER_ENC(NAME_TBD, 3, 96),/* char o; */ + BTF_MEMBER_ENC(NAME_TBD, 4, 128),/* int p[8] */ + /* } */ + /* static struct A t */ + BTF_VAR_ENC(NAME_TBD, 5, 0), /* [6] */ + /* static int u */ + BTF_VAR_ENC(NAME_TBD, 1, 0), /* [7] */ + /* .bss section */ /* [8] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 2), 62), + BTF_VAR_SECINFO_ENC(6, 10, 48), + BTF_VAR_SECINFO_ENC(7, 58, 4), + BTF_END_RAW, + }, + .str_sec = "\0A\0m\0n\0o\0p\0t\0u\0.bss", + .str_sec_size = sizeof("\0A\0m\0n\0o\0p\0t\0u\0.bss"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 62, + .key_type_id = 0, + .value_type_id = 8, + .max_entries = 1, +}, +{ + .descr = "global data test #12, invalid offset", + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + /* unsigned long long */ + BTF_TYPE_INT_ENC(0, 0, 0, 64, 8), /* [2] */ + /* char */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 8, 1), /* [3] */ + /* int[8] */ + BTF_TYPE_ARRAY_ENC(1, 1, 8), /* [4] */ + /* struct A { */ /* [5] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 4), 48), + BTF_MEMBER_ENC(NAME_TBD, 2, 0), /* unsigned long long m;*/ + BTF_MEMBER_ENC(NAME_TBD, 1, 64),/* int n; */ + BTF_MEMBER_ENC(NAME_TBD, 3, 96),/* char o; */ + BTF_MEMBER_ENC(NAME_TBD, 4, 128),/* int p[8] */ + /* } */ + /* static struct A t */ + BTF_VAR_ENC(NAME_TBD, 5, 0), /* [6] */ + /* static int u */ + BTF_VAR_ENC(NAME_TBD, 1, 0), /* [7] */ + /* .bss section */ /* [8] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 2), 62), + BTF_VAR_SECINFO_ENC(6, 10, 48), + BTF_VAR_SECINFO_ENC(7, 60, 4), + BTF_END_RAW, + }, + .str_sec = "\0A\0m\0n\0o\0p\0t\0u\0.bss", + .str_sec_size = sizeof("\0A\0m\0n\0o\0p\0t\0u\0.bss"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 62, + .key_type_id = 0, + .value_type_id = 8, + .max_entries = 1, + .btf_load_err = true, + .err_str = "Invalid offset+size", +}, +{ + .descr = "global data test #13, invalid offset", + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + /* unsigned long long */ + BTF_TYPE_INT_ENC(0, 0, 0, 64, 8), /* [2] */ + /* char */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 8, 1), /* [3] */ + /* int[8] */ + BTF_TYPE_ARRAY_ENC(1, 1, 8), /* [4] */ + /* struct A { */ /* [5] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 4), 48), + BTF_MEMBER_ENC(NAME_TBD, 2, 0), /* unsigned long long m;*/ + BTF_MEMBER_ENC(NAME_TBD, 1, 64),/* int n; */ + BTF_MEMBER_ENC(NAME_TBD, 3, 96),/* char o; */ + BTF_MEMBER_ENC(NAME_TBD, 4, 128),/* int p[8] */ + /* } */ + /* static struct A t */ + BTF_VAR_ENC(NAME_TBD, 5, 0), /* [6] */ + /* static int u */ + BTF_VAR_ENC(NAME_TBD, 1, 0), /* [7] */ + /* .bss section */ /* [8] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 2), 62), + BTF_VAR_SECINFO_ENC(6, 10, 48), + BTF_VAR_SECINFO_ENC(7, 12, 4), + BTF_END_RAW, + }, + .str_sec = "\0A\0m\0n\0o\0p\0t\0u\0.bss", + .str_sec_size = sizeof("\0A\0m\0n\0o\0p\0t\0u\0.bss"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 62, + .key_type_id = 0, + .value_type_id = 8, + .max_entries = 1, + .btf_load_err = true, + .err_str = "Invalid offset", +}, +{ + .descr = "global data test #14, invalid offset", + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + /* unsigned long long */ + BTF_TYPE_INT_ENC(0, 0, 0, 64, 8), /* [2] */ + /* char */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 8, 1), /* [3] */ + /* int[8] */ + BTF_TYPE_ARRAY_ENC(1, 1, 8), /* [4] */ + /* struct A { */ /* [5] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 4), 48), + BTF_MEMBER_ENC(NAME_TBD, 2, 0), /* unsigned long long m;*/ + BTF_MEMBER_ENC(NAME_TBD, 1, 64),/* int n; */ + BTF_MEMBER_ENC(NAME_TBD, 3, 96),/* char o; */ + BTF_MEMBER_ENC(NAME_TBD, 4, 128),/* int p[8] */ + /* } */ + /* static struct A t */ + BTF_VAR_ENC(NAME_TBD, 5, 0), /* [6] */ + /* static int u */ + BTF_VAR_ENC(NAME_TBD, 1, 0), /* [7] */ + /* .bss section */ /* [8] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 2), 62), + BTF_VAR_SECINFO_ENC(7, 58, 4), + BTF_VAR_SECINFO_ENC(6, 10, 48), + BTF_END_RAW, + }, + .str_sec = "\0A\0m\0n\0o\0p\0t\0u\0.bss", + .str_sec_size = sizeof("\0A\0m\0n\0o\0p\0t\0u\0.bss"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 62, + .key_type_id = 0, + .value_type_id = 8, + .max_entries = 1, + .btf_load_err = true, + .err_str = "Invalid offset", +}, +{ + .descr = "global data test #15, not var kind", + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + BTF_VAR_ENC(NAME_TBD, 1, 0), /* [2] */ + /* .bss section */ /* [3] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 4), + BTF_VAR_SECINFO_ENC(1, 0, 4), + BTF_END_RAW, + }, + .str_sec = "\0A\0t\0.bss", + .str_sec_size = sizeof("\0A\0t\0.bss"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 4, + .key_type_id = 0, + .value_type_id = 3, + .max_entries = 1, + .btf_load_err = true, + .err_str = "Not a VAR kind member", +}, +{ + .descr = "global data test #16, invalid var referencing sec", + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + BTF_VAR_ENC(NAME_TBD, 5, 0), /* [2] */ + BTF_VAR_ENC(NAME_TBD, 2, 0), /* [3] */ + /* a section */ /* [4] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 4), + BTF_VAR_SECINFO_ENC(3, 0, 4), + /* a section */ /* [5] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 4), + BTF_VAR_SECINFO_ENC(6, 0, 4), + BTF_VAR_ENC(NAME_TBD, 1, 0), /* [6] */ + BTF_END_RAW, + }, + .str_sec = "\0A\0t\0s\0a\0a", + .str_sec_size = sizeof("\0A\0t\0s\0a\0a"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 4, + .key_type_id = 0, + .value_type_id = 4, + .max_entries = 1, + .btf_load_err = true, + .err_str = "Invalid type_id", +}, +{ + .descr = "global data test #17, invalid var referencing var", + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + BTF_VAR_ENC(NAME_TBD, 1, 0), /* [2] */ + BTF_VAR_ENC(NAME_TBD, 2, 0), /* [3] */ + /* a section */ /* [4] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 4), + BTF_VAR_SECINFO_ENC(3, 0, 4), + BTF_END_RAW, + }, + .str_sec = "\0A\0t\0s\0a\0a", + .str_sec_size = sizeof("\0A\0t\0s\0a\0a"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 4, + .key_type_id = 0, + .value_type_id = 4, + .max_entries = 1, + .btf_load_err = true, + .err_str = "Invalid type_id", +}, +{ + .descr = "global data test #18, invalid var loop", + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + BTF_VAR_ENC(NAME_TBD, 2, 0), /* [2] */ + /* .bss section */ /* [3] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 4), + BTF_VAR_SECINFO_ENC(2, 0, 4), + BTF_END_RAW, + }, + .str_sec = "\0A\0t\0aaa", + .str_sec_size = sizeof("\0A\0t\0aaa"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 4, + .key_type_id = 0, + .value_type_id = 4, + .max_entries = 1, + .btf_load_err = true, + .err_str = "Invalid type_id", +}, +{ + .descr = "global data test #19, invalid var referencing var", + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + BTF_VAR_ENC(NAME_TBD, 3, 0), /* [2] */ + BTF_VAR_ENC(NAME_TBD, 1, 0), /* [3] */ + BTF_END_RAW, + }, + .str_sec = "\0A\0t\0s\0a\0a", + .str_sec_size = sizeof("\0A\0t\0s\0a\0a"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 4, + .key_type_id = 0, + .value_type_id = 4, + .max_entries = 1, + .btf_load_err = true, + .err_str = "Invalid type_id", +}, +{ + .descr = "global data test #20, invalid ptr referencing var", + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + /* PTR type_id=3 */ /* [2] */ + BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_PTR, 0, 0), 3), + BTF_VAR_ENC(NAME_TBD, 1, 0), /* [3] */ + BTF_END_RAW, + }, + .str_sec = "\0A\0t\0s\0a\0a", + .str_sec_size = sizeof("\0A\0t\0s\0a\0a"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 4, + .key_type_id = 0, + .value_type_id = 4, + .max_entries = 1, + .btf_load_err = true, + .err_str = "Invalid type_id", +}, +{ + .descr = "global data test #21, var included in struct", + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + /* struct A { */ /* [2] */ + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 2), sizeof(int) * 2), + BTF_MEMBER_ENC(NAME_TBD, 1, 0), /* int m; */ + BTF_MEMBER_ENC(NAME_TBD, 3, 32),/* VAR type_id=3; */ + /* } */ + BTF_VAR_ENC(NAME_TBD, 1, 0), /* [3] */ + BTF_END_RAW, + }, + .str_sec = "\0A\0t\0s\0a\0a", + .str_sec_size = sizeof("\0A\0t\0s\0a\0a"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 4, + .key_type_id = 0, + .value_type_id = 4, + .max_entries = 1, + .btf_load_err = true, + .err_str = "Invalid member", +}, +{ + .descr = "global data test #22, array of var", + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + BTF_TYPE_ARRAY_ENC(3, 1, 4), /* [2] */ + BTF_VAR_ENC(NAME_TBD, 1, 0), /* [3] */ + BTF_END_RAW, + }, + .str_sec = "\0A\0t\0s\0a\0a", + .str_sec_size = sizeof("\0A\0t\0s\0a\0a"), + .map_type = BPF_MAP_TYPE_ARRAY, + .map_name = ".bss", + .key_size = sizeof(int), + .value_size = 4, + .key_type_id = 0, + .value_type_id = 4, + .max_entries = 1, + .btf_load_err = true, + .err_str = "Invalid elem", +}, /* Test member exceeds the size of struct. * * struct A { @@ -3677,6 +4277,7 @@ struct pprint_mapv { } aenum; uint32_t ui32b; uint32_t bits2c:2; + uint8_t si8_4[2][2]; }; #ifdef __SIZEOF_INT128__ @@ -3729,7 +4330,7 @@ static struct btf_raw_test pprint_test_template[] = { BTF_ENUM_ENC(NAME_TBD, 2), BTF_ENUM_ENC(NAME_TBD, 3), /* struct pprint_mapv */ /* [16] */ - BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 10), 40), + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 11), 40), BTF_MEMBER_ENC(NAME_TBD, 11, 0), /* uint32_t ui32 */ BTF_MEMBER_ENC(NAME_TBD, 10, 32), /* uint16_t ui16 */ BTF_MEMBER_ENC(NAME_TBD, 12, 64), /* int32_t si32 */ @@ -3740,9 +4341,12 @@ static struct btf_raw_test pprint_test_template[] = { BTF_MEMBER_ENC(NAME_TBD, 15, 192), /* aenum */ BTF_MEMBER_ENC(NAME_TBD, 11, 224), /* uint32_t ui32b */ BTF_MEMBER_ENC(NAME_TBD, 6, 256), /* bits2c */ + BTF_MEMBER_ENC(NAME_TBD, 17, 264), /* si8_4 */ + BTF_TYPE_ARRAY_ENC(18, 1, 2), /* [17] */ + BTF_TYPE_ARRAY_ENC(1, 1, 2), /* [18] */ BTF_END_RAW, }, - BTF_STR_SEC("\0unsigned char\0unsigned short\0unsigned int\0int\0unsigned long long\0uint8_t\0uint16_t\0uint32_t\0int32_t\0uint64_t\0ui64\0ui8a\0ENUM_ZERO\0ENUM_ONE\0ENUM_TWO\0ENUM_THREE\0pprint_mapv\0ui32\0ui16\0si32\0unused_bits2a\0bits28\0unused_bits2b\0aenum\0ui32b\0bits2c"), + BTF_STR_SEC("\0unsigned char\0unsigned short\0unsigned int\0int\0unsigned long long\0uint8_t\0uint16_t\0uint32_t\0int32_t\0uint64_t\0ui64\0ui8a\0ENUM_ZERO\0ENUM_ONE\0ENUM_TWO\0ENUM_THREE\0pprint_mapv\0ui32\0ui16\0si32\0unused_bits2a\0bits28\0unused_bits2b\0aenum\0ui32b\0bits2c\0si8_4"), .key_size = sizeof(unsigned int), .value_size = sizeof(struct pprint_mapv), .key_type_id = 3, /* unsigned int */ @@ -3791,7 +4395,7 @@ static struct btf_raw_test pprint_test_template[] = { BTF_ENUM_ENC(NAME_TBD, 2), BTF_ENUM_ENC(NAME_TBD, 3), /* struct pprint_mapv */ /* [16] */ - BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_STRUCT, 1, 10), 40), + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_STRUCT, 1, 11), 40), BTF_MEMBER_ENC(NAME_TBD, 11, BTF_MEMBER_OFFSET(0, 0)), /* uint32_t ui32 */ BTF_MEMBER_ENC(NAME_TBD, 10, BTF_MEMBER_OFFSET(0, 32)), /* uint16_t ui16 */ BTF_MEMBER_ENC(NAME_TBD, 12, BTF_MEMBER_OFFSET(0, 64)), /* int32_t si32 */ @@ -3802,9 +4406,12 @@ static struct btf_raw_test pprint_test_template[] = { BTF_MEMBER_ENC(NAME_TBD, 15, BTF_MEMBER_OFFSET(0, 192)), /* aenum */ BTF_MEMBER_ENC(NAME_TBD, 11, BTF_MEMBER_OFFSET(0, 224)), /* uint32_t ui32b */ BTF_MEMBER_ENC(NAME_TBD, 6, BTF_MEMBER_OFFSET(2, 256)), /* bits2c */ + BTF_MEMBER_ENC(NAME_TBD, 17, 264), /* si8_4 */ + BTF_TYPE_ARRAY_ENC(18, 1, 2), /* [17] */ + BTF_TYPE_ARRAY_ENC(1, 1, 2), /* [18] */ BTF_END_RAW, }, - BTF_STR_SEC("\0unsigned char\0unsigned short\0unsigned int\0int\0unsigned long long\0uint8_t\0uint16_t\0uint32_t\0int32_t\0uint64_t\0ui64\0ui8a\0ENUM_ZERO\0ENUM_ONE\0ENUM_TWO\0ENUM_THREE\0pprint_mapv\0ui32\0ui16\0si32\0unused_bits2a\0bits28\0unused_bits2b\0aenum\0ui32b\0bits2c"), + BTF_STR_SEC("\0unsigned char\0unsigned short\0unsigned int\0int\0unsigned long long\0uint8_t\0uint16_t\0uint32_t\0int32_t\0uint64_t\0ui64\0ui8a\0ENUM_ZERO\0ENUM_ONE\0ENUM_TWO\0ENUM_THREE\0pprint_mapv\0ui32\0ui16\0si32\0unused_bits2a\0bits28\0unused_bits2b\0aenum\0ui32b\0bits2c\0si8_4"), .key_size = sizeof(unsigned int), .value_size = sizeof(struct pprint_mapv), .key_type_id = 3, /* unsigned int */ @@ -3855,7 +4462,7 @@ static struct btf_raw_test pprint_test_template[] = { BTF_ENUM_ENC(NAME_TBD, 2), BTF_ENUM_ENC(NAME_TBD, 3), /* struct pprint_mapv */ /* [16] */ - BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_STRUCT, 1, 10), 40), + BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_STRUCT, 1, 11), 40), BTF_MEMBER_ENC(NAME_TBD, 11, BTF_MEMBER_OFFSET(0, 0)), /* uint32_t ui32 */ BTF_MEMBER_ENC(NAME_TBD, 10, BTF_MEMBER_OFFSET(0, 32)), /* uint16_t ui16 */ BTF_MEMBER_ENC(NAME_TBD, 12, BTF_MEMBER_OFFSET(0, 64)), /* int32_t si32 */ @@ -3866,13 +4473,16 @@ static struct btf_raw_test pprint_test_template[] = { BTF_MEMBER_ENC(NAME_TBD, 15, BTF_MEMBER_OFFSET(0, 192)), /* aenum */ BTF_MEMBER_ENC(NAME_TBD, 11, BTF_MEMBER_OFFSET(0, 224)), /* uint32_t ui32b */ BTF_MEMBER_ENC(NAME_TBD, 17, BTF_MEMBER_OFFSET(2, 256)), /* bits2c */ + BTF_MEMBER_ENC(NAME_TBD, 20, BTF_MEMBER_OFFSET(0, 264)), /* si8_4 */ /* typedef unsigned int ___int */ /* [17] */ BTF_TYPEDEF_ENC(NAME_TBD, 18), BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_VOLATILE, 0, 0), 6), /* [18] */ BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_CONST, 0, 0), 15), /* [19] */ + BTF_TYPE_ARRAY_ENC(21, 1, 2), /* [20] */ + BTF_TYPE_ARRAY_ENC(1, 1, 2), /* [21] */ BTF_END_RAW, }, - BTF_STR_SEC("\0unsigned char\0unsigned short\0unsigned int\0int\0unsigned long long\0uint8_t\0uint16_t\0uint32_t\0int32_t\0uint64_t\0ui64\0ui8a\0ENUM_ZERO\0ENUM_ONE\0ENUM_TWO\0ENUM_THREE\0pprint_mapv\0ui32\0ui16\0si32\0unused_bits2a\0bits28\0unused_bits2b\0aenum\0ui32b\0bits2c\0___int"), + BTF_STR_SEC("\0unsigned char\0unsigned short\0unsigned int\0int\0unsigned long long\0uint8_t\0uint16_t\0uint32_t\0int32_t\0uint64_t\0ui64\0ui8a\0ENUM_ZERO\0ENUM_ONE\0ENUM_TWO\0ENUM_THREE\0pprint_mapv\0ui32\0ui16\0si32\0unused_bits2a\0bits28\0unused_bits2b\0aenum\0ui32b\0bits2c\0___int\0si8_4"), .key_size = sizeof(unsigned int), .value_size = sizeof(struct pprint_mapv), .key_type_id = 3, /* unsigned int */ @@ -4007,6 +4617,10 @@ static void set_pprint_mapv(enum pprint_mapv_kind_t mapv_kind, v->aenum = i & 0x03; v->ui32b = 4; v->bits2c = 1; + v->si8_4[0][0] = (cpu + i) & 0xff; + v->si8_4[0][1] = (cpu + i + 1) & 0xff; + v->si8_4[1][0] = (cpu + i + 2) & 0xff; + v->si8_4[1][1] = (cpu + i + 3) & 0xff; v = (void *)v + rounded_value_size; } } @@ -4040,7 +4654,7 @@ ssize_t get_pprint_expected_line(enum pprint_mapv_kind_t mapv_kind, nexpected_line = snprintf(expected_line, line_size, "%s%u: {%u,0,%d,0x%x,0x%x,0x%x," "{%lu|[%u,%u,%u,%u,%u,%u,%u,%u]},%s," - "%u,0x%x}\n", + "%u,0x%x,[[%d,%d],[%d,%d]]}\n", percpu_map ? "\tcpu" : "", percpu_map ? cpu : next_key, v->ui32, v->si32, @@ -4054,7 +4668,9 @@ ssize_t get_pprint_expected_line(enum pprint_mapv_kind_t mapv_kind, v->ui8a[6], v->ui8a[7], pprint_enum_str[v->aenum], v->ui32b, - v->bits2c); + v->bits2c, + v->si8_4[0][0], v->si8_4[0][1], + v->si8_4[1][0], v->si8_4[1][1]); } #ifdef __SIZEOF_INT128__ @@ -5965,6 +6581,51 @@ const struct btf_dedup_test dedup_tests[] = { .dont_resolve_fwds = false, }, }, +{ + .descr = "dedup: datasec and vars pass-through", + .input = { + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + /* static int t */ + BTF_VAR_ENC(NAME_NTH(2), 1, 0), /* [2] */ + /* .bss section */ /* [3] */ + BTF_TYPE_ENC(NAME_NTH(1), BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 4), + BTF_VAR_SECINFO_ENC(2, 0, 4), + /* int, referenced from [5] */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [4] */ + /* another static int t */ + BTF_VAR_ENC(NAME_NTH(2), 4, 0), /* [5] */ + /* another .bss section */ /* [6] */ + BTF_TYPE_ENC(NAME_NTH(1), BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 4), + BTF_VAR_SECINFO_ENC(5, 0, 4), + BTF_END_RAW, + }, + BTF_STR_SEC("\0.bss\0t"), + }, + .expect = { + .raw_types = { + /* int */ + BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + /* static int t */ + BTF_VAR_ENC(NAME_NTH(2), 1, 0), /* [2] */ + /* .bss section */ /* [3] */ + BTF_TYPE_ENC(NAME_NTH(1), BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 4), + BTF_VAR_SECINFO_ENC(2, 0, 4), + /* another static int t */ + BTF_VAR_ENC(NAME_NTH(2), 1, 0), /* [4] */ + /* another .bss section */ /* [5] */ + BTF_TYPE_ENC(NAME_NTH(1), BTF_INFO_ENC(BTF_KIND_DATASEC, 0, 1), 4), + BTF_VAR_SECINFO_ENC(4, 0, 4), + BTF_END_RAW, + }, + BTF_STR_SEC("\0.bss\0t"), + }, + .opts = { + .dont_resolve_fwds = false, + .dedup_table_size = 1 + }, +}, }; @@ -5994,6 +6655,10 @@ static int btf_type_size(const struct btf_type *t) return base_size + vlen * sizeof(struct btf_member); case BTF_KIND_FUNC_PROTO: return base_size + vlen * sizeof(struct btf_param); + case BTF_KIND_VAR: + return base_size + sizeof(struct btf_var); + case BTF_KIND_DATASEC: + return base_size + vlen * sizeof(struct btf_var_secinfo); default: fprintf(stderr, "Unsupported BTF_KIND:%u\n", kind); return -EINVAL; diff --git a/tools/testing/selftests/bpf/test_btf.h b/tools/testing/selftests/bpf/test_btf.h new file mode 100644 index 0000000000000000000000000000000000000000..2023725f19628128fdbd9194772b09ba87eebefd --- /dev/null +++ b/tools/testing/selftests/bpf/test_btf.h @@ -0,0 +1,69 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2019 Facebook */ + +#ifndef _TEST_BTF_H +#define _TEST_BTF_H + +#define BTF_INFO_ENC(kind, kind_flag, vlen) \ + ((!!(kind_flag) << 31) | ((kind) << 24) | ((vlen) & BTF_MAX_VLEN)) + +#define BTF_TYPE_ENC(name, info, size_or_type) \ + (name), (info), (size_or_type) + +#define BTF_INT_ENC(encoding, bits_offset, nr_bits) \ + ((encoding) << 24 | (bits_offset) << 16 | (nr_bits)) +#define BTF_TYPE_INT_ENC(name, encoding, bits_offset, bits, sz) \ + BTF_TYPE_ENC(name, BTF_INFO_ENC(BTF_KIND_INT, 0, 0), sz), \ + BTF_INT_ENC(encoding, bits_offset, bits) + +#define BTF_FWD_ENC(name, kind_flag) \ + BTF_TYPE_ENC(name, BTF_INFO_ENC(BTF_KIND_FWD, kind_flag, 0), 0) + +#define BTF_ARRAY_ENC(type, index_type, nr_elems) \ + (type), (index_type), (nr_elems) +#define BTF_TYPE_ARRAY_ENC(type, index_type, nr_elems) \ + BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_ARRAY, 0, 0), 0), \ + BTF_ARRAY_ENC(type, index_type, nr_elems) + +#define BTF_STRUCT_ENC(name, nr_elems, sz) \ + BTF_TYPE_ENC(name, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, nr_elems), sz) + +#define BTF_UNION_ENC(name, nr_elems, sz) \ + BTF_TYPE_ENC(name, BTF_INFO_ENC(BTF_KIND_UNION, 0, nr_elems), sz) + +#define BTF_VAR_ENC(name, type, linkage) \ + BTF_TYPE_ENC(name, BTF_INFO_ENC(BTF_KIND_VAR, 0, 0), type), (linkage) +#define BTF_VAR_SECINFO_ENC(type, offset, size) \ + (type), (offset), (size) + +#define BTF_MEMBER_ENC(name, type, bits_offset) \ + (name), (type), (bits_offset) +#define BTF_ENUM_ENC(name, val) (name), (val) +#define BTF_MEMBER_OFFSET(bitfield_size, bits_offset) \ + ((bitfield_size) << 24 | (bits_offset)) + +#define BTF_TYPEDEF_ENC(name, type) \ + BTF_TYPE_ENC(name, BTF_INFO_ENC(BTF_KIND_TYPEDEF, 0, 0), type) + +#define BTF_PTR_ENC(type) \ + BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_PTR, 0, 0), type) + +#define BTF_CONST_ENC(type) \ + BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_CONST, 0, 0), type) + +#define BTF_VOLATILE_ENC(type) \ + BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_VOLATILE, 0, 0), type) + +#define BTF_RESTRICT_ENC(type) \ + BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_RESTRICT, 0, 0), type) + +#define BTF_FUNC_PROTO_ENC(ret_type, nargs) \ + BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_FUNC_PROTO, 0, nargs), ret_type) + +#define BTF_FUNC_PROTO_ARG_ENC(name, type) \ + (name), (type) + +#define BTF_FUNC_ENC(name, func_proto) \ + BTF_TYPE_ENC(name, BTF_INFO_ENC(BTF_KIND_FUNC, 0, 0), func_proto) + +#endif /* _TEST_BTF_H */ diff --git a/tools/testing/selftests/bpf/test_libbpf_open.c b/tools/testing/selftests/bpf/test_libbpf_open.c index 65cbd30704b5ab05bfce44f4058919e4a6bdc039..9e9db202d218a70af43aa7c72e31a12d5ffda509 100644 --- a/tools/testing/selftests/bpf/test_libbpf_open.c +++ b/tools/testing/selftests/bpf/test_libbpf_open.c @@ -11,6 +11,8 @@ static const char *__doc__ = #include #include +#include "bpf_rlimit.h" + static const struct option long_options[] = { {"help", no_argument, NULL, 'h' }, {"debug", no_argument, NULL, 'D' }, diff --git a/tools/testing/selftests/bpf/test_lwt_ip_encap.sh b/tools/testing/selftests/bpf/test_lwt_ip_encap.sh index d4d3391cc13af24af27bf6c8a01420ae9e5ffb95..acf7a74f97cd9bb61420205d34cea328f2380084 100755 --- a/tools/testing/selftests/bpf/test_lwt_ip_encap.sh +++ b/tools/testing/selftests/bpf/test_lwt_ip_encap.sh @@ -129,6 +129,24 @@ setup() ip link set veth7 netns ${NS2} ip link set veth8 netns ${NS3} + if [ ! -z "${VRF}" ] ; then + ip -netns ${NS1} link add red type vrf table 1001 + ip -netns ${NS1} link set red up + ip -netns ${NS1} route add table 1001 unreachable default metric 8192 + ip -netns ${NS1} -6 route add table 1001 unreachable default metric 8192 + ip -netns ${NS1} link set veth1 vrf red + ip -netns ${NS1} link set veth5 vrf red + + ip -netns ${NS2} link add red type vrf table 1001 + ip -netns ${NS2} link set red up + ip -netns ${NS2} route add table 1001 unreachable default metric 8192 + ip -netns ${NS2} -6 route add table 1001 unreachable default metric 8192 + ip -netns ${NS2} link set veth2 vrf red + ip -netns ${NS2} link set veth3 vrf red + ip -netns ${NS2} link set veth6 vrf red + ip -netns ${NS2} link set veth7 vrf red + fi + # configure addesses: the top route (1-2-3-4) ip -netns ${NS1} addr add ${IPv4_1}/24 dev veth1 ip -netns ${NS2} addr add ${IPv4_2}/24 dev veth2 @@ -163,29 +181,29 @@ setup() # NS1 # top route - ip -netns ${NS1} route add ${IPv4_2}/32 dev veth1 - ip -netns ${NS1} route add default dev veth1 via ${IPv4_2} # go top by default - ip -netns ${NS1} -6 route add ${IPv6_2}/128 dev veth1 - ip -netns ${NS1} -6 route add default dev veth1 via ${IPv6_2} # go top by default + ip -netns ${NS1} route add ${IPv4_2}/32 dev veth1 ${VRF} + ip -netns ${NS1} route add default dev veth1 via ${IPv4_2} ${VRF} # go top by default + ip -netns ${NS1} -6 route add ${IPv6_2}/128 dev veth1 ${VRF} + ip -netns ${NS1} -6 route add default dev veth1 via ${IPv6_2} ${VRF} # go top by default # bottom route - ip -netns ${NS1} route add ${IPv4_6}/32 dev veth5 - ip -netns ${NS1} route add ${IPv4_7}/32 dev veth5 via ${IPv4_6} - ip -netns ${NS1} route add ${IPv4_8}/32 dev veth5 via ${IPv4_6} - ip -netns ${NS1} -6 route add ${IPv6_6}/128 dev veth5 - ip -netns ${NS1} -6 route add ${IPv6_7}/128 dev veth5 via ${IPv6_6} - ip -netns ${NS1} -6 route add ${IPv6_8}/128 dev veth5 via ${IPv6_6} + ip -netns ${NS1} route add ${IPv4_6}/32 dev veth5 ${VRF} + ip -netns ${NS1} route add ${IPv4_7}/32 dev veth5 via ${IPv4_6} ${VRF} + ip -netns ${NS1} route add ${IPv4_8}/32 dev veth5 via ${IPv4_6} ${VRF} + ip -netns ${NS1} -6 route add ${IPv6_6}/128 dev veth5 ${VRF} + ip -netns ${NS1} -6 route add ${IPv6_7}/128 dev veth5 via ${IPv6_6} ${VRF} + ip -netns ${NS1} -6 route add ${IPv6_8}/128 dev veth5 via ${IPv6_6} ${VRF} # NS2 # top route - ip -netns ${NS2} route add ${IPv4_1}/32 dev veth2 - ip -netns ${NS2} route add ${IPv4_4}/32 dev veth3 - ip -netns ${NS2} -6 route add ${IPv6_1}/128 dev veth2 - ip -netns ${NS2} -6 route add ${IPv6_4}/128 dev veth3 + ip -netns ${NS2} route add ${IPv4_1}/32 dev veth2 ${VRF} + ip -netns ${NS2} route add ${IPv4_4}/32 dev veth3 ${VRF} + ip -netns ${NS2} -6 route add ${IPv6_1}/128 dev veth2 ${VRF} + ip -netns ${NS2} -6 route add ${IPv6_4}/128 dev veth3 ${VRF} # bottom route - ip -netns ${NS2} route add ${IPv4_5}/32 dev veth6 - ip -netns ${NS2} route add ${IPv4_8}/32 dev veth7 - ip -netns ${NS2} -6 route add ${IPv6_5}/128 dev veth6 - ip -netns ${NS2} -6 route add ${IPv6_8}/128 dev veth7 + ip -netns ${NS2} route add ${IPv4_5}/32 dev veth6 ${VRF} + ip -netns ${NS2} route add ${IPv4_8}/32 dev veth7 ${VRF} + ip -netns ${NS2} -6 route add ${IPv6_5}/128 dev veth6 ${VRF} + ip -netns ${NS2} -6 route add ${IPv6_8}/128 dev veth7 ${VRF} # NS3 # top route @@ -207,16 +225,16 @@ setup() ip -netns ${NS3} tunnel add gre_dev mode gre remote ${IPv4_1} local ${IPv4_GRE} ttl 255 ip -netns ${NS3} link set gre_dev up ip -netns ${NS3} addr add ${IPv4_GRE} dev gre_dev - ip -netns ${NS1} route add ${IPv4_GRE}/32 dev veth5 via ${IPv4_6} - ip -netns ${NS2} route add ${IPv4_GRE}/32 dev veth7 via ${IPv4_8} + ip -netns ${NS1} route add ${IPv4_GRE}/32 dev veth5 via ${IPv4_6} ${VRF} + ip -netns ${NS2} route add ${IPv4_GRE}/32 dev veth7 via ${IPv4_8} ${VRF} # configure IPv6 GRE device in NS3, and a route to it via the "bottom" route ip -netns ${NS3} -6 tunnel add name gre6_dev mode ip6gre remote ${IPv6_1} local ${IPv6_GRE} ttl 255 ip -netns ${NS3} link set gre6_dev up ip -netns ${NS3} -6 addr add ${IPv6_GRE} nodad dev gre6_dev - ip -netns ${NS1} -6 route add ${IPv6_GRE}/128 dev veth5 via ${IPv6_6} - ip -netns ${NS2} -6 route add ${IPv6_GRE}/128 dev veth7 via ${IPv6_8} + ip -netns ${NS1} -6 route add ${IPv6_GRE}/128 dev veth5 via ${IPv6_6} ${VRF} + ip -netns ${NS2} -6 route add ${IPv6_GRE}/128 dev veth7 via ${IPv6_8} ${VRF} # rp_filter gets confused by what these tests are doing, so disable it ip netns exec ${NS1} sysctl -wq net.ipv4.conf.all.rp_filter=0 @@ -244,18 +262,18 @@ trap cleanup EXIT remove_routes_to_gredev() { - ip -netns ${NS1} route del ${IPv4_GRE} dev veth5 - ip -netns ${NS2} route del ${IPv4_GRE} dev veth7 - ip -netns ${NS1} -6 route del ${IPv6_GRE}/128 dev veth5 - ip -netns ${NS2} -6 route del ${IPv6_GRE}/128 dev veth7 + ip -netns ${NS1} route del ${IPv4_GRE} dev veth5 ${VRF} + ip -netns ${NS2} route del ${IPv4_GRE} dev veth7 ${VRF} + ip -netns ${NS1} -6 route del ${IPv6_GRE}/128 dev veth5 ${VRF} + ip -netns ${NS2} -6 route del ${IPv6_GRE}/128 dev veth7 ${VRF} } add_unreachable_routes_to_gredev() { - ip -netns ${NS1} route add unreachable ${IPv4_GRE}/32 - ip -netns ${NS2} route add unreachable ${IPv4_GRE}/32 - ip -netns ${NS1} -6 route add unreachable ${IPv6_GRE}/128 - ip -netns ${NS2} -6 route add unreachable ${IPv6_GRE}/128 + ip -netns ${NS1} route add unreachable ${IPv4_GRE}/32 ${VRF} + ip -netns ${NS2} route add unreachable ${IPv4_GRE}/32 ${VRF} + ip -netns ${NS1} -6 route add unreachable ${IPv6_GRE}/128 ${VRF} + ip -netns ${NS2} -6 route add unreachable ${IPv6_GRE}/128 ${VRF} } test_ping() @@ -265,10 +283,10 @@ test_ping() local RET=0 if [ "${PROTO}" == "IPv4" ] ; then - ip netns exec ${NS1} ping -c 1 -W 1 -I ${IPv4_SRC} ${IPv4_DST} 2>&1 > /dev/null + ip netns exec ${NS1} ping -c 1 -W 1 -I veth1 ${IPv4_DST} 2>&1 > /dev/null RET=$? elif [ "${PROTO}" == "IPv6" ] ; then - ip netns exec ${NS1} ping6 -c 1 -W 6 -I ${IPv6_SRC} ${IPv6_DST} 2>&1 > /dev/null + ip netns exec ${NS1} ping6 -c 1 -W 6 -I veth1 ${IPv6_DST} 2>&1 > /dev/null RET=$? else echo " test_ping: unknown PROTO: ${PROTO}" @@ -328,7 +346,7 @@ test_gso() test_egress() { local readonly ENCAP=$1 - echo "starting egress ${ENCAP} encap test" + echo "starting egress ${ENCAP} encap test ${VRF}" setup # by default, pings work @@ -336,26 +354,35 @@ test_egress() test_ping IPv6 0 # remove NS2->DST routes, ping fails - ip -netns ${NS2} route del ${IPv4_DST}/32 dev veth3 - ip -netns ${NS2} -6 route del ${IPv6_DST}/128 dev veth3 + ip -netns ${NS2} route del ${IPv4_DST}/32 dev veth3 ${VRF} + ip -netns ${NS2} -6 route del ${IPv6_DST}/128 dev veth3 ${VRF} test_ping IPv4 1 test_ping IPv6 1 # install replacement routes (LWT/eBPF), pings succeed if [ "${ENCAP}" == "IPv4" ] ; then - ip -netns ${NS1} route add ${IPv4_DST} encap bpf xmit obj test_lwt_ip_encap.o sec encap_gre dev veth1 - ip -netns ${NS1} -6 route add ${IPv6_DST} encap bpf xmit obj test_lwt_ip_encap.o sec encap_gre dev veth1 + ip -netns ${NS1} route add ${IPv4_DST} encap bpf xmit obj \ + test_lwt_ip_encap.o sec encap_gre dev veth1 ${VRF} + ip -netns ${NS1} -6 route add ${IPv6_DST} encap bpf xmit obj \ + test_lwt_ip_encap.o sec encap_gre dev veth1 ${VRF} elif [ "${ENCAP}" == "IPv6" ] ; then - ip -netns ${NS1} route add ${IPv4_DST} encap bpf xmit obj test_lwt_ip_encap.o sec encap_gre6 dev veth1 - ip -netns ${NS1} -6 route add ${IPv6_DST} encap bpf xmit obj test_lwt_ip_encap.o sec encap_gre6 dev veth1 + ip -netns ${NS1} route add ${IPv4_DST} encap bpf xmit obj \ + test_lwt_ip_encap.o sec encap_gre6 dev veth1 ${VRF} + ip -netns ${NS1} -6 route add ${IPv6_DST} encap bpf xmit obj \ + test_lwt_ip_encap.o sec encap_gre6 dev veth1 ${VRF} else echo " unknown encap ${ENCAP}" TEST_STATUS=1 fi test_ping IPv4 0 test_ping IPv6 0 - test_gso IPv4 - test_gso IPv6 + + # skip GSO tests with VRF: VRF routing needs properly assigned + # source IP/device, which is easy to do with ping and hard with dd/nc. + if [ -z "${VRF}" ] ; then + test_gso IPv4 + test_gso IPv6 + fi # a negative test: remove routes to GRE devices: ping fails remove_routes_to_gredev @@ -374,7 +401,7 @@ test_egress() test_ingress() { local readonly ENCAP=$1 - echo "starting ingress ${ENCAP} encap test" + echo "starting ingress ${ENCAP} encap test ${VRF}" setup # need to wait a bit for IPv6 to autoconf, otherwise @@ -385,18 +412,22 @@ test_ingress() test_ping IPv6 0 # remove NS2->DST routes, pings fail - ip -netns ${NS2} route del ${IPv4_DST}/32 dev veth3 - ip -netns ${NS2} -6 route del ${IPv6_DST}/128 dev veth3 + ip -netns ${NS2} route del ${IPv4_DST}/32 dev veth3 ${VRF} + ip -netns ${NS2} -6 route del ${IPv6_DST}/128 dev veth3 ${VRF} test_ping IPv4 1 test_ping IPv6 1 # install replacement routes (LWT/eBPF), pings succeed if [ "${ENCAP}" == "IPv4" ] ; then - ip -netns ${NS2} route add ${IPv4_DST} encap bpf in obj test_lwt_ip_encap.o sec encap_gre dev veth2 - ip -netns ${NS2} -6 route add ${IPv6_DST} encap bpf in obj test_lwt_ip_encap.o sec encap_gre dev veth2 + ip -netns ${NS2} route add ${IPv4_DST} encap bpf in obj \ + test_lwt_ip_encap.o sec encap_gre dev veth2 ${VRF} + ip -netns ${NS2} -6 route add ${IPv6_DST} encap bpf in obj \ + test_lwt_ip_encap.o sec encap_gre dev veth2 ${VRF} elif [ "${ENCAP}" == "IPv6" ] ; then - ip -netns ${NS2} route add ${IPv4_DST} encap bpf in obj test_lwt_ip_encap.o sec encap_gre6 dev veth2 - ip -netns ${NS2} -6 route add ${IPv6_DST} encap bpf in obj test_lwt_ip_encap.o sec encap_gre6 dev veth2 + ip -netns ${NS2} route add ${IPv4_DST} encap bpf in obj \ + test_lwt_ip_encap.o sec encap_gre6 dev veth2 ${VRF} + ip -netns ${NS2} -6 route add ${IPv6_DST} encap bpf in obj \ + test_lwt_ip_encap.o sec encap_gre6 dev veth2 ${VRF} else echo "FAIL: unknown encap ${ENCAP}" TEST_STATUS=1 @@ -418,6 +449,13 @@ test_ingress() process_test_results } +VRF="" +test_egress IPv4 +test_egress IPv6 +test_ingress IPv4 +test_ingress IPv6 + +VRF="vrf red" test_egress IPv4 test_egress IPv6 test_ingress IPv4 diff --git a/tools/testing/selftests/bpf/test_maps.c b/tools/testing/selftests/bpf/test_maps.c index 3c627771f9654b676a6eca5a143ffcaf0d65d8bc..246f745cb006b36169632d0554c4d7b2070f7a55 100644 --- a/tools/testing/selftests/bpf/test_maps.c +++ b/tools/testing/selftests/bpf/test_maps.c @@ -27,6 +27,7 @@ #include "bpf_util.h" #include "bpf_rlimit.h" +#include "test_maps.h" #ifndef ENOTSUPP #define ENOTSUPP 524 @@ -36,15 +37,6 @@ static int skips; static int map_flags; -#define CHECK(condition, tag, format...) ({ \ - int __ret = !!(condition); \ - if (__ret) { \ - printf("%s(%d):FAIL:%s ", __func__, __LINE__, tag); \ - printf(format); \ - exit(-1); \ - } \ -}) - static void test_hashmap(unsigned int task, void *data) { long long key, next_key, first_key, value; @@ -1703,6 +1695,10 @@ static void run_all_tests(void) test_map_in_map(); } +#define DECLARE +#include +#undef DECLARE + int main(void) { srand(time(NULL)); @@ -1713,6 +1709,10 @@ int main(void) map_flags = BPF_F_NO_PREALLOC; run_all_tests(); +#define CALL +#include +#undef CALL + printf("test_maps: OK, %d SKIPPED\n", skips); return 0; } diff --git a/tools/testing/selftests/bpf/test_maps.h b/tools/testing/selftests/bpf/test_maps.h new file mode 100644 index 0000000000000000000000000000000000000000..77d8587ac4edb5dd1640e74771efd0f509e1cb46 --- /dev/null +++ b/tools/testing/selftests/bpf/test_maps.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _TEST_MAPS_H +#define _TEST_MAPS_H + +#include +#include + +#define CHECK(condition, tag, format...) ({ \ + int __ret = !!(condition); \ + if (__ret) { \ + printf("%s(%d):FAIL:%s ", __func__, __LINE__, tag); \ + printf(format); \ + exit(-1); \ + } \ +}) + +#endif diff --git a/tools/testing/selftests/bpf/test_offload.py b/tools/testing/selftests/bpf/test_offload.py index 84bea3985d64fb154ac6bdcac4a31e166e175834..425f9ed27c3b1b9de7e733358c3035b595dbef21 100755 --- a/tools/testing/selftests/bpf/test_offload.py +++ b/tools/testing/selftests/bpf/test_offload.py @@ -1,6 +1,7 @@ #!/usr/bin/python3 # Copyright (C) 2017 Netronome Systems, Inc. +# Copyright (c) 2019 Mellanox Technologies. All rights reserved # # This software is licensed under the GNU General License Version 2, # June 1991 as shown in the file COPYING in the top-level directory of this @@ -15,10 +16,12 @@ from datetime import datetime import argparse +import errno import json import os import pprint import random +import re import string import struct import subprocess @@ -306,6 +309,8 @@ class DebugfsDir: _, out = cmd('ls ' + path) for f in out.split(): + if f == "ports": + continue p = os.path.join(path, f) if os.path.isfile(p): _, out = cmd('cat %s/%s' % (path, f)) @@ -321,42 +326,112 @@ class DebugfsDir: return dfs -class NetdevSim: +class NetdevSimDev: """ - Class for netdevsim netdevice and its attributes. + Class for netdevsim bus device and its attributes. """ - def __init__(self, link=None): - self.link = link + def __init__(self, port_count=1): + addr = 0 + while True: + try: + with open("/sys/bus/netdevsim/new_device", "w") as f: + f.write("%u %u" % (addr, port_count)) + except OSError as e: + if e.errno == errno.ENOSPC: + addr += 1 + continue + raise e + break + self.addr = addr + + # As probe of netdevsim device might happen from a workqueue, + # so wait here until all netdevs appear. + self.wait_for_netdevs(port_count) + + ret, out = cmd("udevadm settle", fail=False) + if ret: + raise Exception("udevadm settle failed") + ifnames = self.get_ifnames() - self.dev = self._netdevsim_create() devs.append(self) + self.dfs_dir = "/sys/kernel/debug/netdevsim/netdevsim%u/" % addr + + self.nsims = [] + for port_index in range(port_count): + self.nsims.append(NetdevSim(self, port_index, ifnames[port_index])) + + def get_ifnames(self): + ifnames = [] + listdir = os.listdir("/sys/bus/netdevsim/devices/netdevsim%u/net/" % self.addr) + for ifname in listdir: + ifnames.append(ifname) + ifnames.sort() + return ifnames + + def wait_for_netdevs(self, port_count): + timeout = 5 + timeout_start = time.time() + + while True: + try: + ifnames = self.get_ifnames() + except FileNotFoundError as e: + ifnames = [] + if len(ifnames) == port_count: + break + if time.time() < timeout_start + timeout: + continue + raise Exception("netdevices did not appear within timeout") - self.ns = "" + def dfs_num_bound_progs(self): + path = os.path.join(self.dfs_dir, "bpf_bound_progs") + _, progs = cmd('ls %s' % (path)) + return len(progs.split()) - self.dfs_dir = '/sys/kernel/debug/netdevsim/%s' % (self.dev['ifname']) - self.sdev_dir = self.dfs_dir + '/sdev/' - self.dfs_refresh() + def dfs_get_bound_progs(self, expected): + progs = DebugfsDir(os.path.join(self.dfs_dir, "bpf_bound_progs")) + if expected is not None: + if len(progs) != expected: + fail(True, "%d BPF programs bound, expected %d" % + (len(progs), expected)) + return progs - def __getitem__(self, key): - return self.dev[key] + def remove(self): + with open("/sys/bus/netdevsim/del_device", "w") as f: + f.write("%u" % self.addr) + devs.remove(self) - def _netdevsim_create(self): - link = "" if self.link is None else "link " + self.link.dev['ifname'] - _, old = ip("link show") - ip("link add sim%d {link} type netdevsim".format(link=link)) - _, new = ip("link show") + def remove_nsim(self, nsim): + self.nsims.remove(nsim) + with open("/sys/bus/netdevsim/devices/netdevsim%u/del_port" % self.addr ,"w") as f: + f.write("%u" % nsim.port_index) - for dev in new: - f = filter(lambda x: x["ifname"] == dev["ifname"], old) - if len(list(f)) == 0: - return dev +class NetdevSim: + """ + Class for netdevsim netdevice and its attributes. + """ + + def __init__(self, nsimdev, port_index, ifname): + # In case udev renamed the netdev to according to new schema, + # check if the name matches the port_index. + nsimnamere = re.compile("eni\d+np(\d+)") + match = nsimnamere.match(ifname) + if match and int(match.groups()[0]) != port_index + 1: + raise Exception("netdevice name mismatches the expected one") + + self.nsimdev = nsimdev + self.port_index = port_index + self.ns = "" + self.dfs_dir = "%s/ports/%u/" % (nsimdev.dfs_dir, port_index) + self.dfs_refresh() + _, [self.dev] = ip("link show dev %s" % ifname) - raise Exception("failed to create netdevsim device") + def __getitem__(self, key): + return self.dev[key] def remove(self): - devs.remove(self) - ip("link del dev %s" % (self.dev["ifname"]), ns=self.ns) + self.nsimdev.remove_nsim(self) def dfs_refresh(self): self.dfs = DebugfsDir(self.dfs_dir) @@ -367,22 +442,9 @@ class NetdevSim: _, data = cmd('cat %s' % (path)) return data.strip() - def dfs_num_bound_progs(self): - path = os.path.join(self.sdev_dir, "bpf_bound_progs") - _, progs = cmd('ls %s' % (path)) - return len(progs.split()) - - def dfs_get_bound_progs(self, expected): - progs = DebugfsDir(os.path.join(self.sdev_dir, "bpf_bound_progs")) - if expected is not None: - if len(progs) != expected: - fail(True, "%d BPF programs bound, expected %d" % - (len(progs), expected)) - return progs - def wait_for_flush(self, bound=0, total=0, n_retry=20): for i in range(n_retry): - nbound = self.dfs_num_bound_progs() + nbound = self.nsimdev.dfs_num_bound_progs() nprogs = len(bpftool_prog_list()) if nbound == bound and nprogs == total: return @@ -612,7 +674,7 @@ def test_spurios_extack(sim, obj, skip_hw, needle): include_stderr=True) check_no_extack(res, needle) -def test_multi_prog(sim, obj, modename, modeid): +def test_multi_prog(simdev, sim, obj, modename, modeid): start_test("Test multi-attachment XDP - %s + offload..." % (modename or "default", )) sim.set_xdp(obj, "offload") @@ -668,11 +730,12 @@ def test_multi_prog(sim, obj, modename, modeid): check_multi_basic(two_xdps) start_test("Test multi-attachment XDP - device remove...") - sim.remove() + simdev.remove() - sim = NetdevSim() + simdev = NetdevSimDev() + sim, = simdev.nsims sim.set_ethtool_tc_offloads(True) - return sim + return [simdev, sim] # Parse command line parser = argparse.ArgumentParser() @@ -729,12 +792,14 @@ try: bytecode = bpf_bytecode("1,6 0 0 4294967295,") start_test("Test destruction of generic XDP...") - sim = NetdevSim() + simdev = NetdevSimDev() + sim, = simdev.nsims sim.set_xdp(obj, "generic") - sim.remove() + simdev.remove() bpftool_prog_list_wait(expected=0) - sim = NetdevSim() + simdev = NetdevSimDev() + sim, = simdev.nsims sim.tc_add_ingress() start_test("Test TC non-offloaded...") @@ -744,7 +809,7 @@ try: start_test("Test TC non-offloaded isn't getting bound...") ret, _ = sim.cls_bpf_add_filter(obj, fail=False) fail(ret != 0, "Software TC filter did not load") - sim.dfs_get_bound_progs(expected=0) + simdev.dfs_get_bound_progs(expected=0) sim.tc_flush_filters() @@ -761,7 +826,7 @@ try: start_test("Test TC offload by default...") ret, _ = sim.cls_bpf_add_filter(obj, fail=False) fail(ret != 0, "Software TC filter did not load") - sim.dfs_get_bound_progs(expected=0) + simdev.dfs_get_bound_progs(expected=0) ingress = sim.tc_show_ingress(expected=1) fltr = ingress[0] fail(not fltr["in_hw"], "Filter not offloaded by default") @@ -771,7 +836,7 @@ try: start_test("Test TC cBPF bytcode tries offload by default...") ret, _ = sim.cls_bpf_add_filter(bytecode, fail=False) fail(ret != 0, "Software TC filter did not load") - sim.dfs_get_bound_progs(expected=0) + simdev.dfs_get_bound_progs(expected=0) ingress = sim.tc_show_ingress(expected=1) fltr = ingress[0] fail(not fltr["in_hw"], "Bytecode not offloaded by default") @@ -839,7 +904,7 @@ try: check_verifier_log(err, "[netdevsim] Hello from netdevsim!") start_test("Test TC offload basics...") - dfs = sim.dfs_get_bound_progs(expected=1) + dfs = simdev.dfs_get_bound_progs(expected=1) progs = bpftool_prog_list(expected=1) ingress = sim.tc_show_ingress(expected=1) @@ -874,18 +939,20 @@ try: start_test("Test destroying device gets rid of TC filters...") sim.cls_bpf_add_filter(obj, skip_sw=True) - sim.remove() + simdev.remove() bpftool_prog_list_wait(expected=0) - sim = NetdevSim() + simdev = NetdevSimDev() + sim, = simdev.nsims sim.set_ethtool_tc_offloads(True) start_test("Test destroying device gets rid of XDP...") sim.set_xdp(obj, "offload") - sim.remove() + simdev.remove() bpftool_prog_list_wait(expected=0) - sim = NetdevSim() + simdev = NetdevSimDev() + sim, = simdev.nsims sim.set_ethtool_tc_offloads(True) start_test("Test XDP prog reporting...") @@ -971,7 +1038,7 @@ try: check_verifier_log(err, "[netdevsim] Hello from netdevsim!") start_test("Test XDP offload is device bound...") - dfs = sim.dfs_get_bound_progs(expected=1) + dfs = simdev.dfs_get_bound_progs(expected=1) dprog = dfs[0] fail(prog["id"] != link_xdp["id"], "Program IDs don't match") @@ -990,7 +1057,8 @@ try: bpftool_prog_list_wait(expected=0) start_test("Test attempt to use a program for a wrong device...") - sim2 = NetdevSim() + simdev2 = NetdevSimDev() + sim2, = simdev2.nsims sim2.set_xdp(obj, "offload") pin_file, pinned = pin_prog("/sys/fs/bpf/tmp") @@ -998,7 +1066,7 @@ try: fail=False, include_stderr=True) fail(ret == 0, "Pinned program loaded for a different device accepted") check_extack_nsim(err, "program bound to different dev.", args) - sim2.remove() + simdev2.remove() ret, _, err = sim.set_xdp(pinned, "offload", fail=False, include_stderr=True) fail(ret == 0, "Pinned program loaded for a removed device accepted") @@ -1006,9 +1074,9 @@ try: rm(pin_file) bpftool_prog_list_wait(expected=0) - sim = test_multi_prog(sim, obj, "", 1) - sim = test_multi_prog(sim, obj, "drv", 1) - sim = test_multi_prog(sim, obj, "generic", 2) + simdev, sim = test_multi_prog(simdev, sim, obj, "", 1) + simdev, sim = test_multi_prog(simdev, sim, obj, "drv", 1) + simdev, sim = test_multi_prog(simdev, sim, obj, "generic", 2) start_test("Test mixing of TC and XDP...") sim.tc_add_ingress() @@ -1055,15 +1123,15 @@ try: start_test("Test if netdev removal waits for translation...") delay_msec = 500 - sim.dfs["bpf_bind_verifier_delay"] = delay_msec + sim.dfs["dev/bpf_bind_verifier_delay"] = delay_msec start = time.time() cmd_line = "tc filter add dev %s ingress bpf %s da skip_sw" % \ (sim['ifname'], obj) tc_proc = cmd(cmd_line, background=True, fail=False) # Wait for the verifier to start - while sim.dfs_num_bound_progs() <= 2: + while simdev.dfs_num_bound_progs() <= 2: pass - sim.remove() + simdev.remove() end = time.time() ret, _ = cmd_result(tc_proc, fail=False) time_diff = end - start @@ -1078,7 +1146,8 @@ try: clean_up() bpftool_prog_list_wait(expected=0) - sim = NetdevSim() + simdev = NetdevSimDev() + sim, = simdev.nsims map_obj = bpf_obj("sample_map_ret0.o") start_test("Test loading program with maps...") sim.set_xdp(map_obj, "offload", JSON=False) # map fixup msg breaks JSON @@ -1100,7 +1169,7 @@ try: prog_file, _ = pin_prog("/sys/fs/bpf/tmp_prog") map_file, _ = pin_map("/sys/fs/bpf/tmp_map", idx=1, expected=2) - sim.remove() + simdev.remove() start_test("Test bpftool bound info reporting (removed dev)...") check_dev_info_removed(prog_file=prog_file, map_file=map_file) @@ -1109,7 +1178,8 @@ try: clean_up() bpftool_prog_list_wait(expected=0) - sim = NetdevSim() + simdev = NetdevSimDev() + sim, = simdev.nsims start_test("Test map update (no flags)...") sim.set_xdp(map_obj, "offload", JSON=False) # map fixup msg breaks JSON @@ -1190,27 +1260,29 @@ try: start_test("Test map remove...") sim.unset_xdp("offload") bpftool_map_list_wait(expected=0) - sim.remove() + simdev.remove() - sim = NetdevSim() + simdev = NetdevSimDev() + sim, = simdev.nsims sim.set_xdp(map_obj, "offload", JSON=False) # map fixup msg breaks JSON - sim.remove() + simdev.remove() bpftool_map_list_wait(expected=0) start_test("Test map creation fail path...") - sim = NetdevSim() + simdev = NetdevSimDev() + sim, = simdev.nsims sim.dfs["bpf_map_accept"] = "N" ret, _ = sim.set_xdp(map_obj, "offload", JSON=False, fail=False) fail(ret == 0, "netdevsim didn't refuse to create a map with offload disabled") - sim.remove() + simdev.remove() start_test("Test multi-dev ASIC program reuse...") - simA = NetdevSim() - simB1 = NetdevSim() - simB2 = NetdevSim(link=simB1) - simB3 = NetdevSim(link=simB1) + simdevA = NetdevSimDev() + simA, = simdevA.nsims + simdevB = NetdevSimDev(3) + simB1, simB2, simB3 = simdevB.nsims sims = (simA, simB1, simB2, simB3) simB = (simB1, simB2, simB3) @@ -1222,13 +1294,13 @@ try: progB = bpf_pinned("/sys/fs/bpf/nsimB") simA.set_xdp(progA, "offload", JSON=False) - for d in simB: + for d in simdevB.nsims: d.set_xdp(progB, "offload", JSON=False) start_test("Test multi-dev ASIC cross-dev replace...") ret, _ = simA.set_xdp(progB, "offload", force=True, JSON=False, fail=False) fail(ret == 0, "cross-ASIC program allowed") - for d in simB: + for d in simdevB.nsims: ret, _ = d.set_xdp(progA, "offload", force=True, JSON=False, fail=False) fail(ret == 0, "cross-ASIC program allowed") @@ -1240,7 +1312,7 @@ try: fail=False, include_stderr=True) fail(ret == 0, "cross-ASIC program allowed") check_extack_nsim(err, "program bound to different dev.", args) - for d in simB: + for d in simdevB.nsims: ret, _, err = d.set_xdp(progA, "offload", force=True, JSON=False, fail=False, include_stderr=True) fail(ret == 0, "cross-ASIC program allowed") @@ -1277,7 +1349,7 @@ try: start_test("Test multi-dev ASIC cross-dev destruction...") bpftool_prog_list_wait(expected=2) - simA.remove() + simdevA.remove() bpftool_prog_list_wait(expected=1) ifnameB = bpftool("prog show %s" % (progB))[1]["dev"]["ifname"] @@ -1295,6 +1367,7 @@ try: fail(ifnameB != simB3['ifname'], "program not bound to remaining device") simB3.remove() + simdevB.remove() bpftool_prog_list_wait(expected=0) start_test("Test multi-dev ASIC cross-dev destruction - orphaned...") diff --git a/tools/testing/selftests/bpf/test_progs.c b/tools/testing/selftests/bpf/test_progs.c index 5d10aee9e27744f75db45baf5889f56381ce287c..bf5c90998916a60961ede884b9ea2d5393813cdd 100644 --- a/tools/testing/selftests/bpf/test_progs.c +++ b/tools/testing/selftests/bpf/test_progs.c @@ -9,6 +9,7 @@ int error_cnt, pass_cnt; bool jit_enabled; +bool verifier_stats = false; struct ipv4_packet pkt_v4 = { .eth.h_proto = __bpf_constant_htons(ETH_P_IP), @@ -162,12 +163,15 @@ void *spin_lock_thread(void *arg) #include #undef DECLARE -int main(void) +int main(int ac, char **av) { srand(time(NULL)); jit_enabled = is_jit_enabled(); + if (ac == 2 && strcmp(av[1], "-s") == 0) + verifier_stats = true; + #define CALL #include #undef CALL diff --git a/tools/testing/selftests/bpf/test_progs.h b/tools/testing/selftests/bpf/test_progs.h index 51a07367cd433f5ae9055083b4d1a7e6868fdae0..f095e1d4c657f4ee56a2198ce26561274d9b7115 100644 --- a/tools/testing/selftests/bpf/test_progs.h +++ b/tools/testing/selftests/bpf/test_progs.h @@ -40,6 +40,7 @@ typedef __u16 __sum16; extern int error_cnt, pass_cnt; extern bool jit_enabled; +extern bool verifier_stats; #define MAGIC_BYTES 123 diff --git a/tools/testing/selftests/bpf/test_section_names.c b/tools/testing/selftests/bpf/test_section_names.c index 7c4f41572b1cfd26f437ab64ba32a798e49efb38..bebd4fbca1f43e7102ec9dd66c0828d3aebfc60a 100644 --- a/tools/testing/selftests/bpf/test_section_names.c +++ b/tools/testing/selftests/bpf/test_section_names.c @@ -119,6 +119,11 @@ static struct sec_name_test tests[] = { {0, BPF_PROG_TYPE_CGROUP_SOCK_ADDR, BPF_CGROUP_UDP6_SENDMSG}, {0, BPF_CGROUP_UDP6_SENDMSG}, }, + { + "cgroup/sysctl", + {0, BPF_PROG_TYPE_CGROUP_SYSCTL, BPF_CGROUP_SYSCTL}, + {0, BPF_CGROUP_SYSCTL}, + }, }; static int test_prog_type_by_name(const struct sec_name_test *test) diff --git a/tools/testing/selftests/bpf/test_sock_fields.c b/tools/testing/selftests/bpf/test_sock_fields.c index dcae7f664dce0827f5ddda4bc5f693d191def9b0..e089477fa0a3ec98cdf29450f8d133fe7b492933 100644 --- a/tools/testing/selftests/bpf/test_sock_fields.c +++ b/tools/testing/selftests/bpf/test_sock_fields.c @@ -35,6 +35,11 @@ enum bpf_linum_array_idx { __NR_BPF_LINUM_ARRAY_IDX, }; +struct bpf_spinlock_cnt { + struct bpf_spin_lock lock; + __u32 cnt; +}; + #define CHECK(condition, tag, format...) ({ \ int __ret = !!(condition); \ if (__ret) { \ @@ -50,6 +55,8 @@ enum bpf_linum_array_idx { #define DATA_LEN sizeof(DATA) static struct sockaddr_in6 srv_sa6, cli_sa6; +static int sk_pkt_out_cnt10_fd; +static int sk_pkt_out_cnt_fd; static int linum_map_fd; static int addr_map_fd; static int tp_map_fd; @@ -220,28 +227,90 @@ static void check_result(void) "Unexpected listen_tp", "Check listen_tp output. ingress_linum:%u", ingress_linum); - CHECK(srv_tp.data_segs_out != 1 || + CHECK(srv_tp.data_segs_out != 2 || srv_tp.data_segs_in || srv_tp.snd_cwnd != 10 || srv_tp.total_retrans || - srv_tp.bytes_acked != DATA_LEN, + srv_tp.bytes_acked != 2 * DATA_LEN, "Unexpected srv_tp", "Check srv_tp output. egress_linum:%u", egress_linum); CHECK(cli_tp.data_segs_out || - cli_tp.data_segs_in != 1 || + cli_tp.data_segs_in != 2 || cli_tp.snd_cwnd != 10 || cli_tp.total_retrans || - cli_tp.bytes_received != DATA_LEN, + cli_tp.bytes_received != 2 * DATA_LEN, "Unexpected cli_tp", "Check cli_tp output. egress_linum:%u", egress_linum); } +static void check_sk_pkt_out_cnt(int accept_fd, int cli_fd) +{ + struct bpf_spinlock_cnt pkt_out_cnt = {}, pkt_out_cnt10 = {}; + int err; + + pkt_out_cnt.cnt = ~0; + pkt_out_cnt10.cnt = ~0; + err = bpf_map_lookup_elem(sk_pkt_out_cnt_fd, &accept_fd, &pkt_out_cnt); + if (!err) + err = bpf_map_lookup_elem(sk_pkt_out_cnt10_fd, &accept_fd, + &pkt_out_cnt10); + + /* The bpf prog only counts for fullsock and + * passive conneciton did not become fullsock until 3WHS + * had been finished. + * The bpf prog only counted two data packet out but we + * specially init accept_fd's pkt_out_cnt by 2 in + * init_sk_storage(). Hence, 4 here. + */ + CHECK(err || pkt_out_cnt.cnt != 4 || pkt_out_cnt10.cnt != 40, + "bpf_map_lookup_elem(sk_pkt_out_cnt, &accept_fd)", + "err:%d errno:%d pkt_out_cnt:%u pkt_out_cnt10:%u", + err, errno, pkt_out_cnt.cnt, pkt_out_cnt10.cnt); + + pkt_out_cnt.cnt = ~0; + pkt_out_cnt10.cnt = ~0; + err = bpf_map_lookup_elem(sk_pkt_out_cnt_fd, &cli_fd, &pkt_out_cnt); + if (!err) + err = bpf_map_lookup_elem(sk_pkt_out_cnt10_fd, &cli_fd, + &pkt_out_cnt10); + /* Active connection is fullsock from the beginning. + * 1 SYN and 1 ACK during 3WHS + * 2 Acks on data packet. + * + * The bpf_prog initialized it to 0xeB9F. + */ + CHECK(err || pkt_out_cnt.cnt != 0xeB9F + 4 || + pkt_out_cnt10.cnt != 0xeB9F + 40, + "bpf_map_lookup_elem(sk_pkt_out_cnt, &cli_fd)", + "err:%d errno:%d pkt_out_cnt:%u pkt_out_cnt10:%u", + err, errno, pkt_out_cnt.cnt, pkt_out_cnt10.cnt); +} + +static void init_sk_storage(int sk_fd, __u32 pkt_out_cnt) +{ + struct bpf_spinlock_cnt scnt = {}; + int err; + + scnt.cnt = pkt_out_cnt; + err = bpf_map_update_elem(sk_pkt_out_cnt_fd, &sk_fd, &scnt, + BPF_NOEXIST); + CHECK(err, "bpf_map_update_elem(sk_pkt_out_cnt_fd)", + "err:%d errno:%d", err, errno); + + scnt.cnt *= 10; + err = bpf_map_update_elem(sk_pkt_out_cnt10_fd, &sk_fd, &scnt, + BPF_NOEXIST); + CHECK(err, "bpf_map_update_elem(sk_pkt_out_cnt10_fd)", + "err:%d errno:%d", err, errno); +} + static void test(void) { int listen_fd, cli_fd, accept_fd, epfd, err; struct epoll_event ev; socklen_t addrlen; + int i; addrlen = sizeof(struct sockaddr_in6); ev.events = EPOLLIN; @@ -308,24 +377,30 @@ static void test(void) accept_fd, errno); close(listen_fd); - /* Send some data from accept_fd to cli_fd */ - err = send(accept_fd, DATA, DATA_LEN, 0); - CHECK(err != DATA_LEN, "send(accept_fd)", "err:%d errno:%d", - err, errno); - - /* Have some timeout in recv(cli_fd). Just in case. */ ev.data.fd = cli_fd; err = epoll_ctl(epfd, EPOLL_CTL_ADD, cli_fd, &ev); CHECK(err, "epoll_ctl(EPOLL_CTL_ADD, cli_fd)", "err:%d errno:%d", err, errno); - err = epoll_wait(epfd, &ev, 1, 1000); - CHECK(err != 1 || ev.data.fd != cli_fd, - "epoll_wait(cli_fd)", "err:%d errno:%d ev.data.fd:%d cli_fd:%d", - err, errno, ev.data.fd, cli_fd); + init_sk_storage(accept_fd, 2); - err = recv(cli_fd, NULL, 0, MSG_TRUNC); - CHECK(err, "recv(cli_fd)", "err:%d errno:%d", err, errno); + for (i = 0; i < 2; i++) { + /* Send some data from accept_fd to cli_fd */ + err = send(accept_fd, DATA, DATA_LEN, 0); + CHECK(err != DATA_LEN, "send(accept_fd)", "err:%d errno:%d", + err, errno); + + /* Have some timeout in recv(cli_fd). Just in case. */ + err = epoll_wait(epfd, &ev, 1, 1000); + CHECK(err != 1 || ev.data.fd != cli_fd, + "epoll_wait(cli_fd)", "err:%d errno:%d ev.data.fd:%d cli_fd:%d", + err, errno, ev.data.fd, cli_fd); + + err = recv(cli_fd, NULL, 0, MSG_TRUNC); + CHECK(err, "recv(cli_fd)", "err:%d errno:%d", err, errno); + } + + check_sk_pkt_out_cnt(accept_fd, cli_fd); close(epfd); close(accept_fd); @@ -395,6 +470,14 @@ int main(int argc, char **argv) CHECK(!map, "cannot find linum_map", "(null)"); linum_map_fd = bpf_map__fd(map); + map = bpf_object__find_map_by_name(obj, "sk_pkt_out_cnt"); + CHECK(!map, "cannot find sk_pkt_out_cnt", "(null)"); + sk_pkt_out_cnt_fd = bpf_map__fd(map); + + map = bpf_object__find_map_by_name(obj, "sk_pkt_out_cnt10"); + CHECK(!map, "cannot find sk_pkt_out_cnt10", "(null)"); + sk_pkt_out_cnt10_fd = bpf_map__fd(map); + test(); bpf_object__close(obj); diff --git a/tools/testing/selftests/bpf/test_sysctl.c b/tools/testing/selftests/bpf/test_sysctl.c new file mode 100644 index 0000000000000000000000000000000000000000..a3bebd7c68ddc6262c5eaf6021bd8c8304b6a006 --- /dev/null +++ b/tools/testing/selftests/bpf/test_sysctl.c @@ -0,0 +1,1567 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2019 Facebook + +#include +#include +#include +#include +#include +#include + +#include + +#include +#include + +#include "bpf_rlimit.h" +#include "bpf_util.h" +#include "cgroup_helpers.h" + +#define CG_PATH "/foo" +#define MAX_INSNS 512 +#define FIXUP_SYSCTL_VALUE 0 + +char bpf_log_buf[BPF_LOG_BUF_SIZE]; + +struct sysctl_test { + const char *descr; + size_t fixup_value_insn; + struct bpf_insn insns[MAX_INSNS]; + const char *prog_file; + enum bpf_attach_type attach_type; + const char *sysctl; + int open_flags; + const char *newval; + const char *oldval; + enum { + LOAD_REJECT, + ATTACH_REJECT, + OP_EPERM, + SUCCESS, + } result; +}; + +static struct sysctl_test tests[] = { + { + .descr = "sysctl wrong attach_type", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + }, + .attach_type = 0, + .sysctl = "kernel/ostype", + .open_flags = O_RDONLY, + .result = ATTACH_REJECT, + }, + { + .descr = "sysctl:read allow all", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "kernel/ostype", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + .descr = "sysctl:read deny all", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "kernel/ostype", + .open_flags = O_RDONLY, + .result = OP_EPERM, + }, + { + .descr = "ctx:write sysctl:read read ok", + .insns = { + /* If (write) */ + BPF_LDX_MEM(BPF_W, BPF_REG_7, BPF_REG_1, + offsetof(struct bpf_sysctl, write)), + BPF_JMP_IMM(BPF_JNE, BPF_REG_7, 1, 2), + + /* return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_JMP_A(1), + + /* else return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "kernel/ostype", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + .descr = "ctx:write sysctl:write read ok", + .insns = { + /* If (write) */ + BPF_LDX_MEM(BPF_B, BPF_REG_7, BPF_REG_1, + offsetof(struct bpf_sysctl, write)), + BPF_JMP_IMM(BPF_JNE, BPF_REG_7, 1, 2), + + /* return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_JMP_A(1), + + /* else return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "kernel/domainname", + .open_flags = O_WRONLY, + .newval = "(none)", /* same as default, should fail anyway */ + .result = OP_EPERM, + }, + { + .descr = "ctx:write sysctl:read write reject", + .insns = { + /* write = X */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, + offsetof(struct bpf_sysctl, write)), + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "kernel/ostype", + .open_flags = O_RDONLY, + .result = LOAD_REJECT, + }, + { + .descr = "ctx:file_pos sysctl:read read ok", + .insns = { + /* If (file_pos == X) */ + BPF_LDX_MEM(BPF_W, BPF_REG_7, BPF_REG_1, + offsetof(struct bpf_sysctl, file_pos)), + BPF_JMP_IMM(BPF_JNE, BPF_REG_7, 0, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "kernel/ostype", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + .descr = "ctx:file_pos sysctl:read read ok narrow", + .insns = { + /* If (file_pos == X) */ + BPF_LDX_MEM(BPF_B, BPF_REG_7, BPF_REG_1, + offsetof(struct bpf_sysctl, file_pos)), + BPF_JMP_IMM(BPF_JNE, BPF_REG_7, 0, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "kernel/ostype", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + .descr = "ctx:file_pos sysctl:read write ok", + .insns = { + /* file_pos = X */ + BPF_MOV64_IMM(BPF_REG_0, 2), + BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, + offsetof(struct bpf_sysctl, file_pos)), + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "kernel/ostype", + .open_flags = O_RDONLY, + .oldval = "nux\n", + .result = SUCCESS, + }, + { + .descr = "sysctl_get_name sysctl_value:base ok", + .insns = { + /* sysctl_get_name arg2 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + + BPF_MOV64_REG(BPF_REG_2, BPF_REG_7), + + /* sysctl_get_name arg3 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_3, 8), + + /* sysctl_get_name arg4 (flags) */ + BPF_MOV64_IMM(BPF_REG_4, BPF_F_SYSCTL_BASE_NAME), + + /* sysctl_get_name(ctx, buf, buf_len, flags) */ + BPF_EMIT_CALL(BPF_FUNC_sysctl_get_name), + + /* if (ret == expected && */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, sizeof("tcp_mem") - 1, 6), + /* buf == "tcp_mem\0") */ + BPF_LD_IMM64(BPF_REG_8, 0x006d656d5f706374ULL), + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 0), + BPF_JMP_REG(BPF_JNE, BPF_REG_8, BPF_REG_9, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/tcp_mem", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + .descr = "sysctl_get_name sysctl_value:base E2BIG truncated", + .insns = { + /* sysctl_get_name arg2 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + + BPF_MOV64_REG(BPF_REG_2, BPF_REG_7), + + /* sysctl_get_name arg3 (buf_len) too small */ + BPF_MOV64_IMM(BPF_REG_3, 7), + + /* sysctl_get_name arg4 (flags) */ + BPF_MOV64_IMM(BPF_REG_4, BPF_F_SYSCTL_BASE_NAME), + + /* sysctl_get_name(ctx, buf, buf_len, flags) */ + BPF_EMIT_CALL(BPF_FUNC_sysctl_get_name), + + /* if (ret == expected && */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, -E2BIG, 6), + + /* buf[0:7] == "tcp_me\0") */ + BPF_LD_IMM64(BPF_REG_8, 0x00656d5f706374ULL), + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 0), + BPF_JMP_REG(BPF_JNE, BPF_REG_8, BPF_REG_9, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/tcp_mem", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + .descr = "sysctl_get_name sysctl:full ok", + .insns = { + /* sysctl_get_name arg2 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -24), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 8), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 16), + + BPF_MOV64_REG(BPF_REG_2, BPF_REG_7), + + /* sysctl_get_name arg3 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_3, 17), + + /* sysctl_get_name arg4 (flags) */ + BPF_MOV64_IMM(BPF_REG_4, 0), + + /* sysctl_get_name(ctx, buf, buf_len, flags) */ + BPF_EMIT_CALL(BPF_FUNC_sysctl_get_name), + + /* if (ret == expected && */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 16, 14), + + /* buf[0:8] == "net/ipv4" && */ + BPF_LD_IMM64(BPF_REG_8, 0x347670692f74656eULL), + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 0), + BPF_JMP_REG(BPF_JNE, BPF_REG_8, BPF_REG_9, 10), + + /* buf[8:16] == "/tcp_mem" && */ + BPF_LD_IMM64(BPF_REG_8, 0x6d656d5f7063742fULL), + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 8), + BPF_JMP_REG(BPF_JNE, BPF_REG_8, BPF_REG_9, 6), + + /* buf[16:24] == "\0") */ + BPF_LD_IMM64(BPF_REG_8, 0x0ULL), + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 16), + BPF_JMP_REG(BPF_JNE, BPF_REG_8, BPF_REG_9, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/tcp_mem", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + .descr = "sysctl_get_name sysctl:full E2BIG truncated", + .insns = { + /* sysctl_get_name arg2 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -16), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 8), + + BPF_MOV64_REG(BPF_REG_2, BPF_REG_7), + + /* sysctl_get_name arg3 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_3, 16), + + /* sysctl_get_name arg4 (flags) */ + BPF_MOV64_IMM(BPF_REG_4, 0), + + /* sysctl_get_name(ctx, buf, buf_len, flags) */ + BPF_EMIT_CALL(BPF_FUNC_sysctl_get_name), + + /* if (ret == expected && */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, -E2BIG, 10), + + /* buf[0:8] == "net/ipv4" && */ + BPF_LD_IMM64(BPF_REG_8, 0x347670692f74656eULL), + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 0), + BPF_JMP_REG(BPF_JNE, BPF_REG_8, BPF_REG_9, 6), + + /* buf[8:16] == "/tcp_me\0") */ + BPF_LD_IMM64(BPF_REG_8, 0x00656d5f7063742fULL), + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 8), + BPF_JMP_REG(BPF_JNE, BPF_REG_8, BPF_REG_9, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/tcp_mem", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + .descr = "sysctl_get_name sysctl:full E2BIG truncated small", + .insns = { + /* sysctl_get_name arg2 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + + BPF_MOV64_REG(BPF_REG_2, BPF_REG_7), + + /* sysctl_get_name arg3 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_3, 7), + + /* sysctl_get_name arg4 (flags) */ + BPF_MOV64_IMM(BPF_REG_4, 0), + + /* sysctl_get_name(ctx, buf, buf_len, flags) */ + BPF_EMIT_CALL(BPF_FUNC_sysctl_get_name), + + /* if (ret == expected && */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, -E2BIG, 6), + + /* buf[0:8] == "net/ip\0") */ + BPF_LD_IMM64(BPF_REG_8, 0x000070692f74656eULL), + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 0), + BPF_JMP_REG(BPF_JNE, BPF_REG_8, BPF_REG_9, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/tcp_mem", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + .descr = "sysctl_get_current_value sysctl:read ok, gt", + .insns = { + /* sysctl_get_current_value arg2 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_REG(BPF_REG_2, BPF_REG_7), + + /* sysctl_get_current_value arg3 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_3, 8), + + /* sysctl_get_current_value(ctx, buf, buf_len) */ + BPF_EMIT_CALL(BPF_FUNC_sysctl_get_current_value), + + /* if (ret == expected && */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 6, 6), + + /* buf[0:6] == "Linux\n\0") */ + BPF_LD_IMM64(BPF_REG_8, 0x000a78756e694cULL), + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 0), + BPF_JMP_REG(BPF_JNE, BPF_REG_8, BPF_REG_9, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "kernel/ostype", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + .descr = "sysctl_get_current_value sysctl:read ok, eq", + .insns = { + /* sysctl_get_current_value arg2 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_STX_MEM(BPF_B, BPF_REG_7, BPF_REG_0, 7), + + BPF_MOV64_REG(BPF_REG_2, BPF_REG_7), + + /* sysctl_get_current_value arg3 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_3, 7), + + /* sysctl_get_current_value(ctx, buf, buf_len) */ + BPF_EMIT_CALL(BPF_FUNC_sysctl_get_current_value), + + /* if (ret == expected && */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 6, 6), + + /* buf[0:6] == "Linux\n\0") */ + BPF_LD_IMM64(BPF_REG_8, 0x000a78756e694cULL), + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 0), + BPF_JMP_REG(BPF_JNE, BPF_REG_8, BPF_REG_9, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "kernel/ostype", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + .descr = "sysctl_get_current_value sysctl:read E2BIG truncated", + .insns = { + /* sysctl_get_current_value arg2 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_STX_MEM(BPF_H, BPF_REG_7, BPF_REG_0, 6), + + BPF_MOV64_REG(BPF_REG_2, BPF_REG_7), + + /* sysctl_get_current_value arg3 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_3, 6), + + /* sysctl_get_current_value(ctx, buf, buf_len) */ + BPF_EMIT_CALL(BPF_FUNC_sysctl_get_current_value), + + /* if (ret == expected && */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, -E2BIG, 6), + + /* buf[0:6] == "Linux\0") */ + BPF_LD_IMM64(BPF_REG_8, 0x000078756e694cULL), + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 0), + BPF_JMP_REG(BPF_JNE, BPF_REG_8, BPF_REG_9, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "kernel/ostype", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + .descr = "sysctl_get_current_value sysctl:read EINVAL", + .insns = { + /* sysctl_get_current_value arg2 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + + BPF_MOV64_REG(BPF_REG_2, BPF_REG_7), + + /* sysctl_get_current_value arg3 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_3, 8), + + /* sysctl_get_current_value(ctx, buf, buf_len) */ + BPF_EMIT_CALL(BPF_FUNC_sysctl_get_current_value), + + /* if (ret == expected && */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, -EINVAL, 4), + + /* buf[0:8] is NUL-filled) */ + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 0), + BPF_JMP_IMM(BPF_JNE, BPF_REG_9, 0, 2), + + /* return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_JMP_A(1), + + /* else return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv6/conf/lo/stable_secret", /* -EIO */ + .open_flags = O_RDONLY, + .result = OP_EPERM, + }, + { + .descr = "sysctl_get_current_value sysctl:write ok", + .fixup_value_insn = 6, + .insns = { + /* sysctl_get_current_value arg2 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + + BPF_MOV64_REG(BPF_REG_2, BPF_REG_7), + + /* sysctl_get_current_value arg3 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_3, 8), + + /* sysctl_get_current_value(ctx, buf, buf_len) */ + BPF_EMIT_CALL(BPF_FUNC_sysctl_get_current_value), + + /* if (ret == expected && */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 4, 6), + + /* buf[0:4] == expected) */ + BPF_LD_IMM64(BPF_REG_8, FIXUP_SYSCTL_VALUE), + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 0), + BPF_JMP_REG(BPF_JNE, BPF_REG_8, BPF_REG_9, 2), + + /* return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_JMP_A(1), + + /* else return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/route/mtu_expires", + .open_flags = O_WRONLY, + .newval = "600", /* same as default, should fail anyway */ + .result = OP_EPERM, + }, + { + .descr = "sysctl_get_new_value sysctl:read EINVAL", + .insns = { + /* sysctl_get_new_value arg2 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + + BPF_MOV64_REG(BPF_REG_2, BPF_REG_7), + + /* sysctl_get_new_value arg3 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_3, 8), + + /* sysctl_get_new_value(ctx, buf, buf_len) */ + BPF_EMIT_CALL(BPF_FUNC_sysctl_get_new_value), + + /* if (ret == expected) */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, -EINVAL, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/tcp_mem", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + .descr = "sysctl_get_new_value sysctl:write ok", + .insns = { + /* sysctl_get_new_value arg2 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + + BPF_MOV64_REG(BPF_REG_2, BPF_REG_7), + + /* sysctl_get_new_value arg3 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_3, 4), + + /* sysctl_get_new_value(ctx, buf, buf_len) */ + BPF_EMIT_CALL(BPF_FUNC_sysctl_get_new_value), + + /* if (ret == expected && */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 3, 4), + + /* buf[0:4] == "606\0") */ + BPF_LDX_MEM(BPF_W, BPF_REG_9, BPF_REG_7, 0), + BPF_JMP_IMM(BPF_JNE, BPF_REG_9, 0x00363036, 2), + + /* return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_JMP_A(1), + + /* else return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/route/mtu_expires", + .open_flags = O_WRONLY, + .newval = "606", + .result = OP_EPERM, + }, + { + .descr = "sysctl_get_new_value sysctl:write ok long", + .insns = { + /* sysctl_get_new_value arg2 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -24), + + BPF_MOV64_REG(BPF_REG_2, BPF_REG_7), + + /* sysctl_get_new_value arg3 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_3, 24), + + /* sysctl_get_new_value(ctx, buf, buf_len) */ + BPF_EMIT_CALL(BPF_FUNC_sysctl_get_new_value), + + /* if (ret == expected && */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 23, 14), + + /* buf[0:8] == "3000000 " && */ + BPF_LD_IMM64(BPF_REG_8, 0x2030303030303033ULL), + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 0), + BPF_JMP_REG(BPF_JNE, BPF_REG_8, BPF_REG_9, 10), + + /* buf[8:16] == "4000000 " && */ + BPF_LD_IMM64(BPF_REG_8, 0x2030303030303034ULL), + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 8), + BPF_JMP_REG(BPF_JNE, BPF_REG_8, BPF_REG_9, 6), + + /* buf[16:24] == "6000000\0") */ + BPF_LD_IMM64(BPF_REG_8, 0x0030303030303036ULL), + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 16), + BPF_JMP_REG(BPF_JNE, BPF_REG_8, BPF_REG_9, 2), + + /* return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_JMP_A(1), + + /* else return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/tcp_mem", + .open_flags = O_WRONLY, + .newval = "3000000 4000000 6000000", + .result = OP_EPERM, + }, + { + .descr = "sysctl_get_new_value sysctl:write E2BIG", + .insns = { + /* sysctl_get_new_value arg2 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_STX_MEM(BPF_B, BPF_REG_7, BPF_REG_0, 3), + + BPF_MOV64_REG(BPF_REG_2, BPF_REG_7), + + /* sysctl_get_new_value arg3 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_3, 3), + + /* sysctl_get_new_value(ctx, buf, buf_len) */ + BPF_EMIT_CALL(BPF_FUNC_sysctl_get_new_value), + + /* if (ret == expected && */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, -E2BIG, 4), + + /* buf[0:3] == "60\0") */ + BPF_LDX_MEM(BPF_W, BPF_REG_9, BPF_REG_7, 0), + BPF_JMP_IMM(BPF_JNE, BPF_REG_9, 0x003036, 2), + + /* return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_JMP_A(1), + + /* else return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/route/mtu_expires", + .open_flags = O_WRONLY, + .newval = "606", + .result = OP_EPERM, + }, + { + .descr = "sysctl_set_new_value sysctl:read EINVAL", + .insns = { + /* sysctl_set_new_value arg2 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, 0x00303036), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + + BPF_MOV64_REG(BPF_REG_2, BPF_REG_7), + + /* sysctl_set_new_value arg3 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_3, 3), + + /* sysctl_set_new_value(ctx, buf, buf_len) */ + BPF_EMIT_CALL(BPF_FUNC_sysctl_set_new_value), + + /* if (ret == expected) */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, -EINVAL, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/route/mtu_expires", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + .descr = "sysctl_set_new_value sysctl:write ok", + .fixup_value_insn = 2, + .insns = { + /* sysctl_set_new_value arg2 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, FIXUP_SYSCTL_VALUE), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + + BPF_MOV64_REG(BPF_REG_2, BPF_REG_7), + + /* sysctl_set_new_value arg3 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_3, 3), + + /* sysctl_set_new_value(ctx, buf, buf_len) */ + BPF_EMIT_CALL(BPF_FUNC_sysctl_set_new_value), + + /* if (ret == expected) */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/route/mtu_expires", + .open_flags = O_WRONLY, + .newval = "606", + .result = SUCCESS, + }, + { + "bpf_strtoul one number string", + .insns = { + /* arg1 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, 0x00303036), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + + BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), + + /* arg2 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_2, 4), + + /* arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + + /* arg4 (res) */ + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + BPF_MOV64_REG(BPF_REG_4, BPF_REG_7), + + BPF_EMIT_CALL(BPF_FUNC_strtoul), + + /* if (ret == expected && */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 3, 4), + /* res == expected) */ + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 0), + BPF_JMP_IMM(BPF_JNE, BPF_REG_9, 600, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/route/mtu_expires", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + "bpf_strtoul multi number string", + .insns = { + /* arg1 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + /* "600 602\0" */ + BPF_LD_IMM64(BPF_REG_0, 0x0032303620303036ULL), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), + + /* arg2 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_2, 8), + + /* arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + + /* arg4 (res) */ + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + BPF_MOV64_REG(BPF_REG_4, BPF_REG_7), + + BPF_EMIT_CALL(BPF_FUNC_strtoul), + + /* if (ret == expected && */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 3, 18), + /* res == expected) */ + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 0), + BPF_JMP_IMM(BPF_JNE, BPF_REG_9, 600, 16), + + /* arg1 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_ALU64_REG(BPF_ADD, BPF_REG_7, BPF_REG_0), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), + + /* arg2 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_2, 8), + BPF_ALU64_REG(BPF_SUB, BPF_REG_2, BPF_REG_0), + + /* arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + + /* arg4 (res) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -16), + BPF_MOV64_REG(BPF_REG_4, BPF_REG_7), + + BPF_EMIT_CALL(BPF_FUNC_strtoul), + + /* if (ret == expected && */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 4, 4), + /* res == expected) */ + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 0), + BPF_JMP_IMM(BPF_JNE, BPF_REG_9, 602, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/tcp_mem", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + "bpf_strtoul buf_len = 0, reject", + .insns = { + /* arg1 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, 0x00303036), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + + BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), + + /* arg2 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_2, 0), + + /* arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + + /* arg4 (res) */ + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + BPF_MOV64_REG(BPF_REG_4, BPF_REG_7), + + BPF_EMIT_CALL(BPF_FUNC_strtoul), + + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/route/mtu_expires", + .open_flags = O_RDONLY, + .result = LOAD_REJECT, + }, + { + "bpf_strtoul supported base, ok", + .insns = { + /* arg1 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, 0x00373730), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + + BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), + + /* arg2 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_2, 4), + + /* arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 8), + + /* arg4 (res) */ + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + BPF_MOV64_REG(BPF_REG_4, BPF_REG_7), + + BPF_EMIT_CALL(BPF_FUNC_strtoul), + + /* if (ret == expected && */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 3, 4), + /* res == expected) */ + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 0), + BPF_JMP_IMM(BPF_JNE, BPF_REG_9, 63, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/route/mtu_expires", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + "bpf_strtoul unsupported base, EINVAL", + .insns = { + /* arg1 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, 0x00303036), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + + BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), + + /* arg2 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_2, 4), + + /* arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 3), + + /* arg4 (res) */ + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + BPF_MOV64_REG(BPF_REG_4, BPF_REG_7), + + BPF_EMIT_CALL(BPF_FUNC_strtoul), + + /* if (ret == expected) */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, -EINVAL, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/route/mtu_expires", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + "bpf_strtoul buf with spaces only, EINVAL", + .insns = { + /* arg1 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, 0x090a0c0d), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + + BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), + + /* arg2 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_2, 4), + + /* arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + + /* arg4 (res) */ + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + BPF_MOV64_REG(BPF_REG_4, BPF_REG_7), + + BPF_EMIT_CALL(BPF_FUNC_strtoul), + + /* if (ret == expected) */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, -EINVAL, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/route/mtu_expires", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + "bpf_strtoul negative number, EINVAL", + .insns = { + /* arg1 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, 0x00362d0a), /* " -6\0" */ + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + + BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), + + /* arg2 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_2, 4), + + /* arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + + /* arg4 (res) */ + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + BPF_MOV64_REG(BPF_REG_4, BPF_REG_7), + + BPF_EMIT_CALL(BPF_FUNC_strtoul), + + /* if (ret == expected) */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, -EINVAL, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/route/mtu_expires", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + "bpf_strtol negative number, ok", + .insns = { + /* arg1 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, 0x00362d0a), /* " -6\0" */ + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + + BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), + + /* arg2 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_2, 4), + + /* arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 10), + + /* arg4 (res) */ + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + BPF_MOV64_REG(BPF_REG_4, BPF_REG_7), + + BPF_EMIT_CALL(BPF_FUNC_strtol), + + /* if (ret == expected && */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 3, 4), + /* res == expected) */ + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 0), + BPF_JMP_IMM(BPF_JNE, BPF_REG_9, -6, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/route/mtu_expires", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + "bpf_strtol hex number, ok", + .insns = { + /* arg1 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, 0x65667830), /* "0xfe" */ + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + + BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), + + /* arg2 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_2, 4), + + /* arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + + /* arg4 (res) */ + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + BPF_MOV64_REG(BPF_REG_4, BPF_REG_7), + + BPF_EMIT_CALL(BPF_FUNC_strtol), + + /* if (ret == expected && */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 4, 4), + /* res == expected) */ + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 0), + BPF_JMP_IMM(BPF_JNE, BPF_REG_9, 254, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/route/mtu_expires", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + "bpf_strtol max long", + .insns = { + /* arg1 (buf) 9223372036854775807 */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -24), + BPF_LD_IMM64(BPF_REG_0, 0x3032373333323239ULL), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + BPF_LD_IMM64(BPF_REG_0, 0x3537373435383633ULL), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 8), + BPF_LD_IMM64(BPF_REG_0, 0x0000000000373038ULL), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 16), + + BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), + + /* arg2 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_2, 19), + + /* arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + + /* arg4 (res) */ + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + BPF_MOV64_REG(BPF_REG_4, BPF_REG_7), + + BPF_EMIT_CALL(BPF_FUNC_strtol), + + /* if (ret == expected && */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 19, 6), + /* res == expected) */ + BPF_LD_IMM64(BPF_REG_8, 0x7fffffffffffffffULL), + BPF_LDX_MEM(BPF_DW, BPF_REG_9, BPF_REG_7, 0), + BPF_JMP_REG(BPF_JNE, BPF_REG_8, BPF_REG_9, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/route/mtu_expires", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + "bpf_strtol overflow, ERANGE", + .insns = { + /* arg1 (buf) 9223372036854775808 */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -24), + BPF_LD_IMM64(BPF_REG_0, 0x3032373333323239ULL), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + BPF_LD_IMM64(BPF_REG_0, 0x3537373435383633ULL), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 8), + BPF_LD_IMM64(BPF_REG_0, 0x0000000000383038ULL), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 16), + + BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), + + /* arg2 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_2, 19), + + /* arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + + /* arg4 (res) */ + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + BPF_MOV64_REG(BPF_REG_4, BPF_REG_7), + + BPF_EMIT_CALL(BPF_FUNC_strtol), + + /* if (ret == expected) */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, -ERANGE, 2), + + /* return ALLOW; */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_JMP_A(1), + + /* else return DENY; */ + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/route/mtu_expires", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, + { + "C prog: deny all writes", + .prog_file = "./test_sysctl_prog.o", + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/tcp_mem", + .open_flags = O_WRONLY, + .newval = "123 456 789", + .result = OP_EPERM, + }, + { + "C prog: deny access by name", + .prog_file = "./test_sysctl_prog.o", + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/route/mtu_expires", + .open_flags = O_RDONLY, + .result = OP_EPERM, + }, + { + "C prog: read tcp_mem", + .prog_file = "./test_sysctl_prog.o", + .attach_type = BPF_CGROUP_SYSCTL, + .sysctl = "net/ipv4/tcp_mem", + .open_flags = O_RDONLY, + .result = SUCCESS, + }, +}; + +static size_t probe_prog_length(const struct bpf_insn *fp) +{ + size_t len; + + for (len = MAX_INSNS - 1; len > 0; --len) + if (fp[len].code != 0 || fp[len].imm != 0) + break; + return len + 1; +} + +static int fixup_sysctl_value(const char *buf, size_t buf_len, + struct bpf_insn *prog, size_t insn_num) +{ + uint32_t value_num = 0; + uint8_t c, i; + + if (buf_len > sizeof(value_num)) { + log_err("Value is too big (%zd) to use in fixup", buf_len); + return -1; + } + + for (i = 0; i < buf_len; ++i) { + c = buf[i]; + value_num |= (c << i * 8); + } + + prog[insn_num].imm = value_num; + + return 0; +} + +static int load_sysctl_prog_insns(struct sysctl_test *test, + const char *sysctl_path) +{ + struct bpf_insn *prog = test->insns; + struct bpf_load_program_attr attr; + int ret; + + memset(&attr, 0, sizeof(struct bpf_load_program_attr)); + attr.prog_type = BPF_PROG_TYPE_CGROUP_SYSCTL; + attr.insns = prog; + attr.insns_cnt = probe_prog_length(attr.insns); + attr.license = "GPL"; + + if (test->fixup_value_insn) { + char buf[128]; + ssize_t len; + int fd; + + fd = open(sysctl_path, O_RDONLY | O_CLOEXEC); + if (fd < 0) { + log_err("open(%s) failed", sysctl_path); + return -1; + } + len = read(fd, buf, sizeof(buf)); + if (len == -1) { + log_err("read(%s) failed", sysctl_path); + close(fd); + return -1; + } + close(fd); + if (fixup_sysctl_value(buf, len, prog, test->fixup_value_insn)) + return -1; + } + + ret = bpf_load_program_xattr(&attr, bpf_log_buf, BPF_LOG_BUF_SIZE); + if (ret < 0 && test->result != LOAD_REJECT) { + log_err(">>> Loading program error.\n" + ">>> Verifier output:\n%s\n-------\n", bpf_log_buf); + } + + return ret; +} + +static int load_sysctl_prog_file(struct sysctl_test *test) +{ + struct bpf_prog_load_attr attr; + struct bpf_object *obj; + int prog_fd; + + memset(&attr, 0, sizeof(struct bpf_prog_load_attr)); + attr.file = test->prog_file; + attr.prog_type = BPF_PROG_TYPE_CGROUP_SYSCTL; + + if (bpf_prog_load_xattr(&attr, &obj, &prog_fd)) { + if (test->result != LOAD_REJECT) + log_err(">>> Loading program (%s) error.\n", + test->prog_file); + return -1; + } + + return prog_fd; +} + +static int load_sysctl_prog(struct sysctl_test *test, const char *sysctl_path) +{ + return test->prog_file + ? load_sysctl_prog_file(test) + : load_sysctl_prog_insns(test, sysctl_path); +} + +static int access_sysctl(const char *sysctl_path, + const struct sysctl_test *test) +{ + int err = 0; + int fd; + + fd = open(sysctl_path, test->open_flags | O_CLOEXEC); + if (fd < 0) + return fd; + + if (test->open_flags == O_RDONLY) { + char buf[128]; + + if (read(fd, buf, sizeof(buf)) == -1) + goto err; + if (test->oldval && + strncmp(buf, test->oldval, strlen(test->oldval))) { + log_err("Read value %s != %s", buf, test->oldval); + goto err; + } + } else if (test->open_flags == O_WRONLY) { + if (!test->newval) { + log_err("New value for sysctl is not set"); + goto err; + } + if (write(fd, test->newval, strlen(test->newval)) == -1) + goto err; + } else { + log_err("Unexpected sysctl access: neither read nor write"); + goto err; + } + + goto out; +err: + err = -1; +out: + close(fd); + return err; +} + +static int run_test_case(int cgfd, struct sysctl_test *test) +{ + enum bpf_attach_type atype = test->attach_type; + char sysctl_path[128]; + int progfd = -1; + int err = 0; + + printf("Test case: %s .. ", test->descr); + + snprintf(sysctl_path, sizeof(sysctl_path), "/proc/sys/%s", + test->sysctl); + + progfd = load_sysctl_prog(test, sysctl_path); + if (progfd < 0) { + if (test->result == LOAD_REJECT) + goto out; + else + goto err; + } + + if (bpf_prog_attach(progfd, cgfd, atype, BPF_F_ALLOW_OVERRIDE) == -1) { + if (test->result == ATTACH_REJECT) + goto out; + else + goto err; + } + + if (access_sysctl(sysctl_path, test) == -1) { + if (test->result == OP_EPERM && errno == EPERM) + goto out; + else + goto err; + } + + if (test->result != SUCCESS) { + log_err("Unexpected failure"); + goto err; + } + + goto out; +err: + err = -1; +out: + /* Detaching w/o checking return code: best effort attempt. */ + if (progfd != -1) + bpf_prog_detach(cgfd, atype); + close(progfd); + printf("[%s]\n", err ? "FAIL" : "PASS"); + return err; +} + +static int run_tests(int cgfd) +{ + int passes = 0; + int fails = 0; + int i; + + for (i = 0; i < ARRAY_SIZE(tests); ++i) { + if (run_test_case(cgfd, &tests[i])) + ++fails; + else + ++passes; + } + printf("Summary: %d PASSED, %d FAILED\n", passes, fails); + return fails ? -1 : 0; +} + +int main(int argc, char **argv) +{ + int cgfd = -1; + int err = 0; + + if (setup_cgroup_environment()) + goto err; + + cgfd = create_and_get_cgroup(CG_PATH); + if (cgfd < 0) + goto err; + + if (join_cgroup(CG_PATH)) + goto err; + + if (run_tests(cgfd)) + goto err; + + goto out; +err: + err = -1; +out: + close(cgfd); + cleanup_cgroup_environment(); + return err; +} diff --git a/tools/testing/selftests/bpf/test_tc_edt.sh b/tools/testing/selftests/bpf/test_tc_edt.sh new file mode 100755 index 0000000000000000000000000000000000000000..f38567ef694b6e34cae650cad1676e1fe7391d6e --- /dev/null +++ b/tools/testing/selftests/bpf/test_tc_edt.sh @@ -0,0 +1,99 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# This test installs a TC bpf program that throttles a TCP flow +# with dst port = 9000 down to 5MBps. Then it measures actual +# throughput of the flow. + +if [[ $EUID -ne 0 ]]; then + echo "This script must be run as root" + echo "FAIL" + exit 1 +fi + +# check that nc, dd, and timeout are present +command -v nc >/dev/null 2>&1 || \ + { echo >&2 "nc is not available"; exit 1; } +command -v dd >/dev/null 2>&1 || \ + { echo >&2 "nc is not available"; exit 1; } +command -v timeout >/dev/null 2>&1 || \ + { echo >&2 "timeout is not available"; exit 1; } + +readonly NS_SRC="ns-src-$(mktemp -u XXXXXX)" +readonly NS_DST="ns-dst-$(mktemp -u XXXXXX)" + +readonly IP_SRC="172.16.1.100" +readonly IP_DST="172.16.2.100" + +cleanup() +{ + ip netns del ${NS_SRC} + ip netns del ${NS_DST} +} + +trap cleanup EXIT + +set -e # exit on error + +ip netns add "${NS_SRC}" +ip netns add "${NS_DST}" +ip link add veth_src type veth peer name veth_dst +ip link set veth_src netns ${NS_SRC} +ip link set veth_dst netns ${NS_DST} + +ip -netns ${NS_SRC} addr add ${IP_SRC}/24 dev veth_src +ip -netns ${NS_DST} addr add ${IP_DST}/24 dev veth_dst + +ip -netns ${NS_SRC} link set dev veth_src up +ip -netns ${NS_DST} link set dev veth_dst up + +ip -netns ${NS_SRC} route add ${IP_DST}/32 dev veth_src +ip -netns ${NS_DST} route add ${IP_SRC}/32 dev veth_dst + +# set up TC on TX +ip netns exec ${NS_SRC} tc qdisc add dev veth_src root fq +ip netns exec ${NS_SRC} tc qdisc add dev veth_src clsact +ip netns exec ${NS_SRC} tc filter add dev veth_src egress \ + bpf da obj test_tc_edt.o sec cls_test + + +# start the listener +ip netns exec ${NS_DST} bash -c \ + "nc -4 -l -s ${IP_DST} -p 9000 >/dev/null &" +declare -i NC_PID=$! +sleep 1 + +declare -ir TIMEOUT=20 +declare -ir EXPECTED_BPS=5000000 + +# run the load, capture RX bytes on DST +declare -ir RX_BYTES_START=$( ip netns exec ${NS_DST} \ + cat /sys/class/net/veth_dst/statistics/rx_bytes ) + +set +e +ip netns exec ${NS_SRC} bash -c "timeout ${TIMEOUT} dd if=/dev/zero \ + bs=1000 count=1000000 > /dev/tcp/${IP_DST}/9000 2>/dev/null" +set -e + +declare -ir RX_BYTES_END=$( ip netns exec ${NS_DST} \ + cat /sys/class/net/veth_dst/statistics/rx_bytes ) + +declare -ir ACTUAL_BPS=$(( ($RX_BYTES_END - $RX_BYTES_START) / $TIMEOUT )) + +echo $TIMEOUT $ACTUAL_BPS $EXPECTED_BPS | \ + awk '{printf "elapsed: %d sec; bps difference: %.2f%%\n", + $1, ($2-$3)*100.0/$3}' + +# Pass the test if the actual bps is within 1% of the expected bps. +# The difference is usually about 0.1% on a 20-sec test, and ==> zero +# the longer the test runs. +declare -ir RES=$( echo $ACTUAL_BPS $EXPECTED_BPS | \ + awk 'function abs(x){return ((x < 0.0) ? -x : x)} + {if (abs(($1-$2)*100.0/$2) > 1.0) { print "1" } + else { print "0"} }' ) +if [ "${RES}" == "0" ] ; then + echo "PASS" +else + echo "FAIL" + exit 1 +fi diff --git a/tools/testing/selftests/bpf/test_tc_tunnel.sh b/tools/testing/selftests/bpf/test_tc_tunnel.sh new file mode 100755 index 0000000000000000000000000000000000000000..ff0d31d38061fa747f4171dd85fdacdd94fcbdc6 --- /dev/null +++ b/tools/testing/selftests/bpf/test_tc_tunnel.sh @@ -0,0 +1,290 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# In-place tunneling + +# must match the port that the bpf program filters on +readonly port=8000 + +readonly ns_prefix="ns-$$-" +readonly ns1="${ns_prefix}1" +readonly ns2="${ns_prefix}2" + +readonly ns1_v4=192.168.1.1 +readonly ns2_v4=192.168.1.2 +readonly ns1_v6=fd::1 +readonly ns2_v6=fd::2 + +# Must match port used by bpf program +readonly udpport=5555 +# MPLSoverUDP +readonly mplsudpport=6635 +readonly mplsproto=137 + +readonly infile="$(mktemp)" +readonly outfile="$(mktemp)" + +setup() { + ip netns add "${ns1}" + ip netns add "${ns2}" + + ip link add dev veth1 mtu 1500 netns "${ns1}" type veth \ + peer name veth2 mtu 1500 netns "${ns2}" + + ip netns exec "${ns1}" ethtool -K veth1 tso off + + ip -netns "${ns1}" link set veth1 up + ip -netns "${ns2}" link set veth2 up + + ip -netns "${ns1}" -4 addr add "${ns1_v4}/24" dev veth1 + ip -netns "${ns2}" -4 addr add "${ns2_v4}/24" dev veth2 + ip -netns "${ns1}" -6 addr add "${ns1_v6}/64" dev veth1 nodad + ip -netns "${ns2}" -6 addr add "${ns2_v6}/64" dev veth2 nodad + + # clamp route to reserve room for tunnel headers + ip -netns "${ns1}" -4 route flush table main + ip -netns "${ns1}" -6 route flush table main + ip -netns "${ns1}" -4 route add "${ns2_v4}" mtu 1458 dev veth1 + ip -netns "${ns1}" -6 route add "${ns2_v6}" mtu 1438 dev veth1 + + sleep 1 + + dd if=/dev/urandom of="${infile}" bs="${datalen}" count=1 status=none +} + +cleanup() { + ip netns del "${ns2}" + ip netns del "${ns1}" + + if [[ -f "${outfile}" ]]; then + rm "${outfile}" + fi + if [[ -f "${infile}" ]]; then + rm "${infile}" + fi +} + +server_listen() { + ip netns exec "${ns2}" nc "${netcat_opt}" -l -p "${port}" > "${outfile}" & + server_pid=$! + sleep 0.2 +} + +client_connect() { + ip netns exec "${ns1}" timeout 2 nc "${netcat_opt}" -w 1 "${addr2}" "${port}" < "${infile}" + echo $? +} + +verify_data() { + wait "${server_pid}" + # sha1sum returns two fields [sha1] [filepath] + # convert to bash array and access first elem + insum=($(sha1sum ${infile})) + outsum=($(sha1sum ${outfile})) + if [[ "${insum[0]}" != "${outsum[0]}" ]]; then + echo "data mismatch" + exit 1 + fi +} + +set -e + +# no arguments: automated test, run all +if [[ "$#" -eq "0" ]]; then + echo "ipip" + $0 ipv4 ipip none 100 + + echo "ip6ip6" + $0 ipv6 ip6tnl none 100 + + echo "sit" + $0 ipv6 sit none 100 + + for mac in none mpls eth ; do + echo "ip gre $mac" + $0 ipv4 gre $mac 100 + + echo "ip6 gre $mac" + $0 ipv6 ip6gre $mac 100 + + echo "ip gre $mac gso" + $0 ipv4 gre $mac 2000 + + echo "ip6 gre $mac gso" + $0 ipv6 ip6gre $mac 2000 + + echo "ip udp $mac" + $0 ipv4 udp $mac 100 + + echo "ip6 udp $mac" + $0 ipv6 ip6udp $mac 100 + + echo "ip udp $mac gso" + $0 ipv4 udp $mac 2000 + + echo "ip6 udp $mac gso" + $0 ipv6 ip6udp $mac 2000 + done + + echo "OK. All tests passed" + exit 0 +fi + +if [[ "$#" -ne "4" ]]; then + echo "Usage: $0" + echo " or: $0 " + exit 1 +fi + +case "$1" in +"ipv4") + readonly addr1="${ns1_v4}" + readonly addr2="${ns2_v4}" + readonly ipproto=4 + readonly netcat_opt=-${ipproto} + readonly foumod=fou + readonly foutype=ipip + readonly fouproto=4 + readonly fouproto_mpls=${mplsproto} + readonly gretaptype=gretap + ;; +"ipv6") + readonly addr1="${ns1_v6}" + readonly addr2="${ns2_v6}" + readonly ipproto=6 + readonly netcat_opt=-${ipproto} + readonly foumod=fou6 + readonly foutype=ip6tnl + readonly fouproto="41 -6" + readonly fouproto_mpls="${mplsproto} -6" + readonly gretaptype=ip6gretap + ;; +*) + echo "unknown arg: $1" + exit 1 + ;; +esac + +readonly tuntype=$2 +readonly mac=$3 +readonly datalen=$4 + +echo "encap ${addr1} to ${addr2}, type ${tuntype}, mac ${mac} len ${datalen}" + +trap cleanup EXIT + +setup + +# basic communication works +echo "test basic connectivity" +server_listen +client_connect +verify_data + +# clientside, insert bpf program to encap all TCP to port ${port} +# client can no longer connect +ip netns exec "${ns1}" tc qdisc add dev veth1 clsact +ip netns exec "${ns1}" tc filter add dev veth1 egress \ + bpf direct-action object-file ./test_tc_tunnel.o \ + section "encap_${tuntype}_${mac}" +echo "test bpf encap without decap (expect failure)" +server_listen +! client_connect + +if [[ "$tuntype" =~ "udp" ]]; then + # Set up fou tunnel. + ttype="${foutype}" + targs="encap fou encap-sport auto encap-dport $udpport" + # fou may be a module; allow this to fail. + modprobe "${foumod}" ||true + if [[ "$mac" == "mpls" ]]; then + dport=${mplsudpport} + dproto=${fouproto_mpls} + tmode="mode any ttl 255" + else + dport=${udpport} + dproto=${fouproto} + fi + ip netns exec "${ns2}" ip fou add port $dport ipproto ${dproto} + targs="encap fou encap-sport auto encap-dport $dport" +elif [[ "$tuntype" =~ "gre" && "$mac" == "eth" ]]; then + ttype=$gretaptype +else + ttype=$tuntype + targs="" +fi + +# tunnel address family differs from inner for SIT +if [[ "${tuntype}" == "sit" ]]; then + link_addr1="${ns1_v4}" + link_addr2="${ns2_v4}" +else + link_addr1="${addr1}" + link_addr2="${addr2}" +fi + +# serverside, insert decap module +# server is still running +# client can connect again +ip netns exec "${ns2}" ip link add name testtun0 type "${ttype}" \ + ${tmode} remote "${link_addr1}" local "${link_addr2}" $targs + +expect_tun_fail=0 + +if [[ "$tuntype" == "ip6udp" && "$mac" == "mpls" ]]; then + # No support for MPLS IPv6 fou tunnel; expect failure. + expect_tun_fail=1 +elif [[ "$tuntype" =~ "udp" && "$mac" == "eth" ]]; then + # No support for TEB fou tunnel; expect failure. + expect_tun_fail=1 +elif [[ "$tuntype" =~ "gre" && "$mac" == "eth" ]]; then + # Share ethernet address between tunnel/veth2 so L2 decap works. + ethaddr=$(ip netns exec "${ns2}" ip link show veth2 | \ + awk '/ether/ { print $2 }') + ip netns exec "${ns2}" ip link set testtun0 address $ethaddr +elif [[ "$mac" == "mpls" ]]; then + modprobe mpls_iptunnel ||true + modprobe mpls_gso ||true + ip netns exec "${ns2}" sysctl -qw net.mpls.platform_labels=65536 + ip netns exec "${ns2}" ip -f mpls route add 1000 dev lo + ip netns exec "${ns2}" ip link set lo up + ip netns exec "${ns2}" sysctl -qw net.mpls.conf.testtun0.input=1 + ip netns exec "${ns2}" sysctl -qw net.ipv4.conf.lo.rp_filter=0 +fi + +# Because packets are decapped by the tunnel they arrive on testtun0 from +# the IP stack perspective. Ensure reverse path filtering is disabled +# otherwise we drop the TCP SYN as arriving on testtun0 instead of the +# expected veth2 (veth2 is where 192.168.1.2 is configured). +ip netns exec "${ns2}" sysctl -qw net.ipv4.conf.all.rp_filter=0 +# rp needs to be disabled for both all and testtun0 as the rp value is +# selected as the max of the "all" and device-specific values. +ip netns exec "${ns2}" sysctl -qw net.ipv4.conf.testtun0.rp_filter=0 +ip netns exec "${ns2}" ip link set dev testtun0 up +if [[ "$expect_tun_fail" == 1 ]]; then + # This tunnel mode is not supported, so we expect failure. + echo "test bpf encap with tunnel device decap (expect failure)" + ! client_connect +else + echo "test bpf encap with tunnel device decap" + client_connect + verify_data + server_listen +fi + +# bpf_skb_net_shrink does not take tunnel flags yet, cannot update L3. +if [[ "${tuntype}" == "sit" ]]; then + echo OK + exit 0 +fi + +# serverside, use BPF for decap +ip netns exec "${ns2}" ip link del dev testtun0 +ip netns exec "${ns2}" tc qdisc add dev veth2 clsact +ip netns exec "${ns2}" tc filter add dev veth2 ingress \ + bpf direct-action object-file ./test_tc_tunnel.o section decap +echo "test bpf encap with bpf decap" +client_connect +verify_data + +echo OK diff --git a/tools/testing/selftests/bpf/test_tcp_check_syncookie.sh b/tools/testing/selftests/bpf/test_tcp_check_syncookie.sh new file mode 100755 index 0000000000000000000000000000000000000000..d48e51716d196b360c3919822a24be97c7147440 --- /dev/null +++ b/tools/testing/selftests/bpf/test_tcp_check_syncookie.sh @@ -0,0 +1,81 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2018 Facebook +# Copyright (c) 2019 Cloudflare + +set -eu + +wait_for_ip() +{ + local _i + printf "Wait for IP %s to become available " "$1" + for _i in $(seq ${MAX_PING_TRIES}); do + printf "." + if ns1_exec ping -c 1 -W 1 "$1" >/dev/null 2>&1; then + echo " OK" + return + fi + sleep 1 + done + echo 1>&2 "ERROR: Timeout waiting for test IP to become available." + exit 1 +} + +get_prog_id() +{ + awk '/ id / {sub(/.* id /, "", $0); print($1)}' +} + +ns1_exec() +{ + ip netns exec ns1 "$@" +} + +setup() +{ + ip netns add ns1 + ns1_exec ip link set lo up + + ns1_exec sysctl -w net.ipv4.tcp_syncookies=2 + + wait_for_ip 127.0.0.1 + wait_for_ip ::1 +} + +cleanup() +{ + ip netns del ns1 2>/dev/null || : +} + +main() +{ + trap cleanup EXIT 2 3 6 15 + setup + + printf "Testing clsact..." + ns1_exec tc qdisc add dev "${TEST_IF}" clsact + ns1_exec tc filter add dev "${TEST_IF}" ingress \ + bpf obj "${BPF_PROG_OBJ}" sec "${CLSACT_SECTION}" da + + BPF_PROG_ID=$(ns1_exec tc filter show dev "${TEST_IF}" ingress | \ + get_prog_id) + ns1_exec "${PROG}" "${BPF_PROG_ID}" + ns1_exec tc qdisc del dev "${TEST_IF}" clsact + + printf "Testing XDP..." + ns1_exec ip link set "${TEST_IF}" xdp \ + object "${BPF_PROG_OBJ}" section "${XDP_SECTION}" + BPF_PROG_ID=$(ns1_exec ip link show "${TEST_IF}" | get_prog_id) + ns1_exec "${PROG}" "${BPF_PROG_ID}" +} + +DIR=$(dirname $0) +TEST_IF=lo +MAX_PING_TRIES=5 +BPF_PROG_OBJ="${DIR}/test_tcp_check_syncookie_kern.o" +CLSACT_SECTION="clsact/check_syncookie" +XDP_SECTION="xdp/check_syncookie" +BPF_PROG_ID=0 +PROG="${DIR}/test_tcp_check_syncookie_user" + +main diff --git a/tools/testing/selftests/bpf/test_tcp_check_syncookie_user.c b/tools/testing/selftests/bpf/test_tcp_check_syncookie_user.c new file mode 100644 index 0000000000000000000000000000000000000000..87829c86c7469564a5b3ed06c2fcdc961e7cca12 --- /dev/null +++ b/tools/testing/selftests/bpf/test_tcp_check_syncookie_user.c @@ -0,0 +1,212 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2018 Facebook +// Copyright (c) 2019 Cloudflare + +#include +#include +#include + +#include +#include +#include +#include + +#include +#include + +#include "bpf_rlimit.h" +#include "cgroup_helpers.h" + +static int start_server(const struct sockaddr *addr, socklen_t len) +{ + int fd; + + fd = socket(addr->sa_family, SOCK_STREAM, 0); + if (fd == -1) { + log_err("Failed to create server socket"); + goto out; + } + + if (bind(fd, addr, len) == -1) { + log_err("Failed to bind server socket"); + goto close_out; + } + + if (listen(fd, 128) == -1) { + log_err("Failed to listen on server socket"); + goto close_out; + } + + goto out; + +close_out: + close(fd); + fd = -1; +out: + return fd; +} + +static int connect_to_server(int server_fd) +{ + struct sockaddr_storage addr; + socklen_t len = sizeof(addr); + int fd = -1; + + if (getsockname(server_fd, (struct sockaddr *)&addr, &len)) { + log_err("Failed to get server addr"); + goto out; + } + + fd = socket(addr.ss_family, SOCK_STREAM, 0); + if (fd == -1) { + log_err("Failed to create client socket"); + goto out; + } + + if (connect(fd, (const struct sockaddr *)&addr, len) == -1) { + log_err("Fail to connect to server"); + goto close_out; + } + + goto out; + +close_out: + close(fd); + fd = -1; +out: + return fd; +} + +static int get_map_fd_by_prog_id(int prog_id) +{ + struct bpf_prog_info info = {}; + __u32 info_len = sizeof(info); + __u32 map_ids[1]; + int prog_fd = -1; + int map_fd = -1; + + prog_fd = bpf_prog_get_fd_by_id(prog_id); + if (prog_fd < 0) { + log_err("Failed to get fd by prog id %d", prog_id); + goto err; + } + + info.nr_map_ids = 1; + info.map_ids = (__u64)(unsigned long)map_ids; + + if (bpf_obj_get_info_by_fd(prog_fd, &info, &info_len)) { + log_err("Failed to get info by prog fd %d", prog_fd); + goto err; + } + + if (!info.nr_map_ids) { + log_err("No maps found for prog fd %d", prog_fd); + goto err; + } + + map_fd = bpf_map_get_fd_by_id(map_ids[0]); + if (map_fd < 0) + log_err("Failed to get fd by map id %d", map_ids[0]); +err: + if (prog_fd >= 0) + close(prog_fd); + return map_fd; +} + +static int run_test(int server_fd, int results_fd) +{ + int client = -1, srv_client = -1; + int ret = 0; + __u32 key = 0; + __u64 value = 0; + + if (bpf_map_update_elem(results_fd, &key, &value, 0) < 0) { + log_err("Can't clear results"); + goto err; + } + + client = connect_to_server(server_fd); + if (client == -1) + goto err; + + srv_client = accept(server_fd, NULL, 0); + if (srv_client == -1) { + log_err("Can't accept connection"); + goto err; + } + + if (bpf_map_lookup_elem(results_fd, &key, &value) < 0) { + log_err("Can't lookup result"); + goto err; + } + + if (value != 1) { + log_err("Didn't match syncookie: %llu", value); + goto err; + } + + goto out; + +err: + ret = 1; +out: + close(client); + close(srv_client); + return ret; +} + +int main(int argc, char **argv) +{ + struct sockaddr_in addr4; + struct sockaddr_in6 addr6; + int server = -1; + int server_v6 = -1; + int results = -1; + int err = 0; + + if (argc < 2) { + fprintf(stderr, "Usage: %s prog_id\n", argv[0]); + exit(1); + } + + results = get_map_fd_by_prog_id(atoi(argv[1])); + if (results < 0) { + log_err("Can't get map"); + goto err; + } + + memset(&addr4, 0, sizeof(addr4)); + addr4.sin_family = AF_INET; + addr4.sin_addr.s_addr = htonl(INADDR_LOOPBACK); + addr4.sin_port = 0; + + memset(&addr6, 0, sizeof(addr6)); + addr6.sin6_family = AF_INET6; + addr6.sin6_addr = in6addr_loopback; + addr6.sin6_port = 0; + + server = start_server((const struct sockaddr *)&addr4, sizeof(addr4)); + if (server == -1) + goto err; + + server_v6 = start_server((const struct sockaddr *)&addr6, + sizeof(addr6)); + if (server_v6 == -1) + goto err; + + if (run_test(server, results)) + goto err; + + if (run_test(server_v6, results)) + goto err; + + printf("ok\n"); + goto out; +err: + err = 1; +out: + close(server); + close(server_v6); + close(results); + return err; +} diff --git a/tools/testing/selftests/bpf/test_verifier.c b/tools/testing/selftests/bpf/test_verifier.c index 477a9dcf9ffff4b5b47a73759ba00d92d13b0106..ccd896b98cacc56be75be3e9e4053a590cd14121 100644 --- a/tools/testing/selftests/bpf/test_verifier.c +++ b/tools/testing/selftests/bpf/test_verifier.c @@ -47,11 +47,13 @@ #include "bpf_rlimit.h" #include "bpf_rand.h" #include "bpf_util.h" +#include "test_btf.h" #include "../../../include/linux/filter.h" #define MAX_INSNS BPF_MAXINSNS +#define MAX_TEST_INSNS 1000000 #define MAX_FIXUPS 8 -#define MAX_NR_MAPS 14 +#define MAX_NR_MAPS 18 #define MAX_TEST_RUNS 8 #define POINTER_VALUE 0xcafe4all #define TEST_DATA_LEN 64 @@ -66,6 +68,7 @@ static int skips; struct bpf_test { const char *descr; struct bpf_insn insns[MAX_INSNS]; + struct bpf_insn *fill_insns; int fixup_map_hash_8b[MAX_FIXUPS]; int fixup_map_hash_48b[MAX_FIXUPS]; int fixup_map_hash_16b[MAX_FIXUPS]; @@ -80,9 +83,14 @@ struct bpf_test { int fixup_cgroup_storage[MAX_FIXUPS]; int fixup_percpu_cgroup_storage[MAX_FIXUPS]; int fixup_map_spin_lock[MAX_FIXUPS]; + int fixup_map_array_ro[MAX_FIXUPS]; + int fixup_map_array_wo[MAX_FIXUPS]; + int fixup_map_array_small[MAX_FIXUPS]; + int fixup_sk_storage_map[MAX_FIXUPS]; const char *errstr; const char *errstr_unpriv; uint32_t retval, retval_unpriv, insn_processed; + int prog_len; enum { UNDEF, ACCEPT, @@ -119,10 +127,11 @@ struct other_val { static void bpf_fill_ld_abs_vlan_push_pop(struct bpf_test *self) { - /* test: {skb->data[0], vlan_push} x 68 + {skb->data[0], vlan_pop} x 68 */ + /* test: {skb->data[0], vlan_push} x 51 + {skb->data[0], vlan_pop} x 51 */ #define PUSH_CNT 51 - unsigned int len = BPF_MAXINSNS; - struct bpf_insn *insn = self->insns; + /* jump range is limited to 16 bit. PUSH_CNT of ld_abs needs room */ + unsigned int len = (1 << 15) - PUSH_CNT * 2 * 5 * 6; + struct bpf_insn *insn = self->fill_insns; int i = 0, j, k = 0; insn[i++] = BPF_MOV64_REG(BPF_REG_6, BPF_REG_1); @@ -156,12 +165,14 @@ loop: for (; i < len - 1; i++) insn[i] = BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 0xbef); insn[len - 1] = BPF_EXIT_INSN(); + self->prog_len = len; } static void bpf_fill_jump_around_ld_abs(struct bpf_test *self) { - struct bpf_insn *insn = self->insns; - unsigned int len = BPF_MAXINSNS; + struct bpf_insn *insn = self->fill_insns; + /* jump range is limited to 16 bit. every ld_abs is replaced by 6 insns */ + unsigned int len = (1 << 15) / 6; int i = 0; insn[i++] = BPF_MOV64_REG(BPF_REG_6, BPF_REG_1); @@ -171,11 +182,12 @@ static void bpf_fill_jump_around_ld_abs(struct bpf_test *self) while (i < len - 1) insn[i++] = BPF_LD_ABS(BPF_B, 1); insn[i] = BPF_EXIT_INSN(); + self->prog_len = i + 1; } static void bpf_fill_rand_ld_dw(struct bpf_test *self) { - struct bpf_insn *insn = self->insns; + struct bpf_insn *insn = self->fill_insns; uint64_t res = 0; int i = 0; @@ -193,12 +205,83 @@ static void bpf_fill_rand_ld_dw(struct bpf_test *self) insn[i++] = BPF_ALU64_IMM(BPF_RSH, BPF_REG_1, 32); insn[i++] = BPF_ALU64_REG(BPF_XOR, BPF_REG_0, BPF_REG_1); insn[i] = BPF_EXIT_INSN(); + self->prog_len = i + 1; res ^= (res >> 32); self->retval = (uint32_t)res; } +/* test the sequence of 1k jumps */ +static void bpf_fill_scale1(struct bpf_test *self) +{ + struct bpf_insn *insn = self->fill_insns; + int i = 0, k = 0; + + insn[i++] = BPF_MOV64_REG(BPF_REG_6, BPF_REG_1); + /* test to check that the sequence of 1024 jumps is acceptable */ + while (k++ < 1024) { + insn[i++] = BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, + BPF_FUNC_get_prandom_u32); + insn[i++] = BPF_JMP_IMM(BPF_JGT, BPF_REG_0, bpf_semi_rand_get(), 2); + insn[i++] = BPF_MOV64_REG(BPF_REG_1, BPF_REG_10); + insn[i++] = BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_6, + -8 * (k % 64 + 1)); + } + /* every jump adds 1024 steps to insn_processed, so to stay exactly + * within 1m limit add MAX_TEST_INSNS - 1025 MOVs and 1 EXIT + */ + while (i < MAX_TEST_INSNS - 1025) + insn[i++] = BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 42); + insn[i] = BPF_EXIT_INSN(); + self->prog_len = i + 1; + self->retval = 42; +} + +/* test the sequence of 1k jumps in inner most function (function depth 8)*/ +static void bpf_fill_scale2(struct bpf_test *self) +{ + struct bpf_insn *insn = self->fill_insns; + int i = 0, k = 0; + +#define FUNC_NEST 7 + for (k = 0; k < FUNC_NEST; k++) { + insn[i++] = BPF_CALL_REL(1); + insn[i++] = BPF_EXIT_INSN(); + } + insn[i++] = BPF_MOV64_REG(BPF_REG_6, BPF_REG_1); + /* test to check that the sequence of 1024 jumps is acceptable */ + while (k++ < 1024) { + insn[i++] = BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, + BPF_FUNC_get_prandom_u32); + insn[i++] = BPF_JMP_IMM(BPF_JGT, BPF_REG_0, bpf_semi_rand_get(), 2); + insn[i++] = BPF_MOV64_REG(BPF_REG_1, BPF_REG_10); + insn[i++] = BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_6, + -8 * (k % (64 - 4 * FUNC_NEST) + 1)); + } + /* every jump adds 1024 steps to insn_processed, so to stay exactly + * within 1m limit add MAX_TEST_INSNS - 1025 MOVs and 1 EXIT + */ + while (i < MAX_TEST_INSNS - 1025) + insn[i++] = BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 42); + insn[i] = BPF_EXIT_INSN(); + self->prog_len = i + 1; + self->retval = 42; +} + +static void bpf_fill_scale(struct bpf_test *self) +{ + switch (self->retval) { + case 1: + return bpf_fill_scale1(self); + case 2: + return bpf_fill_scale2(self); + default: + self->prog_len = 0; + break; + } +} + /* BPF_SK_LOOKUP contains 13 instructions, if you need to fix up maps */ -#define BPF_SK_LOOKUP \ +#define BPF_SK_LOOKUP(func) \ /* struct bpf_sock_tuple tuple = {} */ \ BPF_MOV64_IMM(BPF_REG_2, 0), \ BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_2, -8), \ @@ -207,13 +290,13 @@ static void bpf_fill_rand_ld_dw(struct bpf_test *self) BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_2, -32), \ BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_2, -40), \ BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_2, -48), \ - /* sk = sk_lookup_tcp(ctx, &tuple, sizeof tuple, 0, 0) */ \ + /* sk = func(ctx, &tuple, sizeof tuple, 0, 0) */ \ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), \ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -48), \ BPF_MOV64_IMM(BPF_REG_3, sizeof(struct bpf_sock_tuple)), \ BPF_MOV64_IMM(BPF_REG_4, 0), \ BPF_MOV64_IMM(BPF_REG_5, 0), \ - BPF_EMIT_CALL(BPF_FUNC_sk_lookup_tcp) + BPF_EMIT_CALL(BPF_FUNC_ ## func) /* BPF_DIRECT_PKT_R2 contains 7 instructions, it initializes default return * value into 0 and does necessary preparation for direct packet access @@ -277,13 +360,15 @@ static bool skip_unsupported_map(enum bpf_map_type map_type) return false; } -static int create_map(uint32_t type, uint32_t size_key, - uint32_t size_value, uint32_t max_elem) +static int __create_map(uint32_t type, uint32_t size_key, + uint32_t size_value, uint32_t max_elem, + uint32_t extra_flags) { int fd; fd = bpf_create_map(type, size_key, size_value, max_elem, - type == BPF_MAP_TYPE_HASH ? BPF_F_NO_PREALLOC : 0); + (type == BPF_MAP_TYPE_HASH ? + BPF_F_NO_PREALLOC : 0) | extra_flags); if (fd < 0) { if (skip_unsupported_map(type)) return -1; @@ -293,6 +378,12 @@ static int create_map(uint32_t type, uint32_t size_key, return fd; } +static int create_map(uint32_t type, uint32_t size_key, + uint32_t size_value, uint32_t max_elem) +{ + return __create_map(type, size_key, size_value, max_elem, 0); +} + static void update_map(int fd, int index) { struct test_val value = { @@ -408,24 +499,6 @@ static int create_cgroup_storage(bool percpu) return fd; } -#define BTF_INFO_ENC(kind, kind_flag, vlen) \ - ((!!(kind_flag) << 31) | ((kind) << 24) | ((vlen) & BTF_MAX_VLEN)) -#define BTF_TYPE_ENC(name, info, size_or_type) \ - (name), (info), (size_or_type) -#define BTF_INT_ENC(encoding, bits_offset, nr_bits) \ - ((encoding) << 24 | (bits_offset) << 16 | (nr_bits)) -#define BTF_TYPE_INT_ENC(name, encoding, bits_offset, bits, sz) \ - BTF_TYPE_ENC(name, BTF_INFO_ENC(BTF_KIND_INT, 0, 0), sz), \ - BTF_INT_ENC(encoding, bits_offset, bits) -#define BTF_MEMBER_ENC(name, type, bits_offset) \ - (name), (type), (bits_offset) - -struct btf_raw_data { - __u32 raw_types[64]; - const char *str_sec; - __u32 str_sec_size; -}; - /* struct bpf_spin_lock { * int val; * }; @@ -500,6 +573,31 @@ static int create_map_spin_lock(void) return fd; } +static int create_sk_storage_map(void) +{ + struct bpf_create_map_attr attr = { + .name = "test_map", + .map_type = BPF_MAP_TYPE_SK_STORAGE, + .key_size = 4, + .value_size = 8, + .max_entries = 0, + .map_flags = BPF_F_NO_PREALLOC, + .btf_key_type_id = 1, + .btf_value_type_id = 3, + }; + int fd, btf_fd; + + btf_fd = load_btf(); + if (btf_fd < 0) + return -1; + attr.btf_fd = btf_fd; + fd = bpf_create_map_xattr(&attr); + close(attr.btf_fd); + if (fd < 0) + printf("Failed to create sk_storage_map\n"); + return fd; +} + static char bpf_vlog[UINT_MAX >> 8]; static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, @@ -519,9 +617,15 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, int *fixup_cgroup_storage = test->fixup_cgroup_storage; int *fixup_percpu_cgroup_storage = test->fixup_percpu_cgroup_storage; int *fixup_map_spin_lock = test->fixup_map_spin_lock; + int *fixup_map_array_ro = test->fixup_map_array_ro; + int *fixup_map_array_wo = test->fixup_map_array_wo; + int *fixup_map_array_small = test->fixup_map_array_small; + int *fixup_sk_storage_map = test->fixup_sk_storage_map; - if (test->fill_helper) + if (test->fill_helper) { + test->fill_insns = calloc(MAX_TEST_INSNS, sizeof(struct bpf_insn)); test->fill_helper(test); + } /* Allocating HTs with 1 elem is fine here, since we only test * for verifier and not do a runtime lookup, so the only thing @@ -642,6 +746,42 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type, fixup_map_spin_lock++; } while (*fixup_map_spin_lock); } + if (*fixup_map_array_ro) { + map_fds[14] = __create_map(BPF_MAP_TYPE_ARRAY, sizeof(int), + sizeof(struct test_val), 1, + BPF_F_RDONLY_PROG); + update_map(map_fds[14], 0); + do { + prog[*fixup_map_array_ro].imm = map_fds[14]; + fixup_map_array_ro++; + } while (*fixup_map_array_ro); + } + if (*fixup_map_array_wo) { + map_fds[15] = __create_map(BPF_MAP_TYPE_ARRAY, sizeof(int), + sizeof(struct test_val), 1, + BPF_F_WRONLY_PROG); + update_map(map_fds[15], 0); + do { + prog[*fixup_map_array_wo].imm = map_fds[15]; + fixup_map_array_wo++; + } while (*fixup_map_array_wo); + } + if (*fixup_map_array_small) { + map_fds[16] = __create_map(BPF_MAP_TYPE_ARRAY, sizeof(int), + 1, 1, 0); + update_map(map_fds[16], 0); + do { + prog[*fixup_map_array_small].imm = map_fds[16]; + fixup_map_array_small++; + } while (*fixup_map_array_small); + } + if (*fixup_sk_storage_map) { + map_fds[17] = create_sk_storage_map(); + do { + prog[*fixup_sk_storage_map].imm = map_fds[17]; + fixup_sk_storage_map++; + } while (*fixup_sk_storage_map); + } } static int set_admin(bool admin) @@ -718,12 +858,17 @@ static void do_test_single(struct bpf_test *test, bool unpriv, prog_type = BPF_PROG_TYPE_SOCKET_FILTER; fixup_skips = skips; do_test_fixup(test, prog_type, prog, map_fds); + if (test->fill_insns) { + prog = test->fill_insns; + prog_len = test->prog_len; + } else { + prog_len = probe_filter_length(prog); + } /* If there were some map skips during fixup due to missing bpf * features, skip this test. */ if (fixup_skips != skips) return; - prog_len = probe_filter_length(prog); pflags = 0; if (test->flags & F_LOAD_WITH_STRICT_ALIGNMENT) @@ -731,7 +876,7 @@ static void do_test_single(struct bpf_test *test, bool unpriv, if (test->flags & F_NEEDS_EFFICIENT_UNALIGNED_ACCESS) pflags |= BPF_F_ANY_ALIGNMENT; fd_prog = bpf_verify_program(prog_type, prog, prog_len, pflags, - "GPL", 0, bpf_vlog, sizeof(bpf_vlog), 1); + "GPL", 0, bpf_vlog, sizeof(bpf_vlog), 4); if (fd_prog < 0 && !bpf_probe_prog_type(prog_type, 0)) { printf("SKIP (unsupported program type %d)\n", prog_type); skips++; @@ -830,6 +975,8 @@ static void do_test_single(struct bpf_test *test, bool unpriv, goto fail_log; } close_fds: + if (test->fill_insns) + free(test->fill_insns); close(fd_prog); for (i = 0; i < MAX_NR_MAPS; i++) close(map_fds[i]); diff --git a/tools/testing/selftests/bpf/trace_helpers.c b/tools/testing/selftests/bpf/trace_helpers.c index 4cdb63bf0521d546c65802cd7567a46a12e6769d..9a9fc6c9b70b5660ee83bca010b98b4af704dd2f 100644 --- a/tools/testing/selftests/bpf/trace_helpers.c +++ b/tools/testing/selftests/bpf/trace_helpers.c @@ -52,6 +52,10 @@ struct ksym *ksym_search(long key) int start = 0, end = sym_cnt; int result; + /* kallsyms not loaded. return NULL */ + if (sym_cnt <= 0) + return NULL; + while (start < end) { size_t mid = start + (end - start) / 2; diff --git a/tools/testing/selftests/bpf/urandom_read.c b/tools/testing/selftests/bpf/urandom_read.c index 9de8b7cb4e6df6b5929c915566dff74163efbcf2..db781052758d3774b42cb891b86b09320e3d95d5 100644 --- a/tools/testing/selftests/bpf/urandom_read.c +++ b/tools/testing/selftests/bpf/urandom_read.c @@ -7,11 +7,19 @@ #define BUF_SIZE 256 +static __attribute__((noinline)) +void urandom_read(int fd, int count) +{ + char buf[BUF_SIZE]; + int i; + + for (i = 0; i < count; ++i) + read(fd, buf, BUF_SIZE); +} + int main(int argc, char *argv[]) { int fd = open("/dev/urandom", O_RDONLY); - int i; - char buf[BUF_SIZE]; int count = 4; if (fd < 0) @@ -20,8 +28,7 @@ int main(int argc, char *argv[]) if (argc == 2) count = atoi(argv[1]); - for (i = 0; i < count; ++i) - read(fd, buf, BUF_SIZE); + urandom_read(fd, count); close(fd); return 0; diff --git a/tools/testing/selftests/bpf/verifier/array_access.c b/tools/testing/selftests/bpf/verifier/array_access.c index 0dcecaf3ec6f547b0f60a33621f36e2d9780d88a..bcb83196e459bed011b9df3b1dbb3404ac9a6088 100644 --- a/tools/testing/selftests/bpf/verifier/array_access.c +++ b/tools/testing/selftests/bpf/verifier/array_access.c @@ -217,3 +217,162 @@ .result = REJECT, .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, +{ + "valid read map access into a read-only array 1", + .insns = { + BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), + BPF_LD_MAP_FD(BPF_REG_1, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), + BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .fixup_map_array_ro = { 3 }, + .result = ACCEPT, + .retval = 28, +}, +{ + "valid read map access into a read-only array 2", + .insns = { + BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), + BPF_LD_MAP_FD(BPF_REG_1, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 6), + + BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), + BPF_MOV64_IMM(BPF_REG_2, 4), + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_MOV64_IMM(BPF_REG_4, 0), + BPF_MOV64_IMM(BPF_REG_5, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, + BPF_FUNC_csum_diff), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_array_ro = { 3 }, + .result = ACCEPT, + .retval = -29, +}, +{ + "invalid write map access into a read-only array 1", + .insns = { + BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), + BPF_LD_MAP_FD(BPF_REG_1, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), + BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 42), + BPF_EXIT_INSN(), + }, + .fixup_map_array_ro = { 3 }, + .result = REJECT, + .errstr = "write into map forbidden", +}, +{ + "invalid write map access into a read-only array 2", + .insns = { + BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), + BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), + BPF_LD_MAP_FD(BPF_REG_1, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 5), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_MOV64_IMM(BPF_REG_2, 0), + BPF_MOV64_REG(BPF_REG_3, BPF_REG_0), + BPF_MOV64_IMM(BPF_REG_4, 8), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, + BPF_FUNC_skb_load_bytes), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_array_ro = { 4 }, + .result = REJECT, + .errstr = "write into map forbidden", +}, +{ + "valid write map access into a write-only array 1", + .insns = { + BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), + BPF_LD_MAP_FD(BPF_REG_1, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), + BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 42), + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + }, + .fixup_map_array_wo = { 3 }, + .result = ACCEPT, + .retval = 1, +}, +{ + "valid write map access into a write-only array 2", + .insns = { + BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), + BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), + BPF_LD_MAP_FD(BPF_REG_1, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 5), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_MOV64_IMM(BPF_REG_2, 0), + BPF_MOV64_REG(BPF_REG_3, BPF_REG_0), + BPF_MOV64_IMM(BPF_REG_4, 8), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, + BPF_FUNC_skb_load_bytes), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_array_wo = { 4 }, + .result = ACCEPT, + .retval = 0, +}, +{ + "invalid read map access into a write-only array 1", + .insns = { + BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), + BPF_LD_MAP_FD(BPF_REG_1, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), + BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .fixup_map_array_wo = { 3 }, + .result = REJECT, + .errstr = "read from map forbidden", +}, +{ + "invalid read map access into a write-only array 2", + .insns = { + BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), + BPF_LD_MAP_FD(BPF_REG_1, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 6), + + BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), + BPF_MOV64_IMM(BPF_REG_2, 4), + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_MOV64_IMM(BPF_REG_4, 0), + BPF_MOV64_IMM(BPF_REG_5, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, + BPF_FUNC_csum_diff), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .fixup_map_array_wo = { 3 }, + .result = REJECT, + .errstr = "read from map forbidden", +}, diff --git a/tools/testing/selftests/bpf/verifier/ctx_skb.c b/tools/testing/selftests/bpf/verifier/ctx_skb.c index c660deb582f169b117a24685d429230dc847521d..b0fda2877119c4af08277bd0f329f238c193313c 100644 --- a/tools/testing/selftests/bpf/verifier/ctx_skb.c +++ b/tools/testing/selftests/bpf/verifier/ctx_skb.c @@ -705,7 +705,6 @@ .errstr = "invalid bpf_context access", .result = REJECT, .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, - .flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS, }, { "check cb access: half, wrong type", diff --git a/tools/testing/selftests/bpf/verifier/direct_value_access.c b/tools/testing/selftests/bpf/verifier/direct_value_access.c new file mode 100644 index 0000000000000000000000000000000000000000..b9fb28e8e2243da75d2ef98141fcf216517761eb --- /dev/null +++ b/tools/testing/selftests/bpf/verifier/direct_value_access.c @@ -0,0 +1,347 @@ +{ + "direct map access, write test 1", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0), + BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 4242), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = ACCEPT, + .retval = 1, +}, +{ + "direct map access, write test 2", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 8), + BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 4242), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = ACCEPT, + .retval = 1, +}, +{ + "direct map access, write test 3", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 8), + BPF_ST_MEM(BPF_DW, BPF_REG_1, 8, 4242), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = ACCEPT, + .retval = 1, +}, +{ + "direct map access, write test 4", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 40), + BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 4242), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = ACCEPT, + .retval = 1, +}, +{ + "direct map access, write test 5", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 32), + BPF_ST_MEM(BPF_DW, BPF_REG_1, 8, 4242), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = ACCEPT, + .retval = 1, +}, +{ + "direct map access, write test 6", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 40), + BPF_ST_MEM(BPF_DW, BPF_REG_1, 4, 4242), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = REJECT, + .errstr = "R1 min value is outside of the array range", +}, +{ + "direct map access, write test 7", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_MAP_VALUE(BPF_REG_1, 0, -1), + BPF_ST_MEM(BPF_DW, BPF_REG_1, 4, 4242), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = REJECT, + .errstr = "direct value offset of 4294967295 is not allowed", +}, +{ + "direct map access, write test 8", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 1), + BPF_ST_MEM(BPF_DW, BPF_REG_1, -1, 4242), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = ACCEPT, + .retval = 1, +}, +{ + "direct map access, write test 9", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 48), + BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 4242), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = REJECT, + .errstr = "invalid access to map value pointer", +}, +{ + "direct map access, write test 10", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 47), + BPF_ST_MEM(BPF_B, BPF_REG_1, 0, 4), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = ACCEPT, + .retval = 1, +}, +{ + "direct map access, write test 11", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 48), + BPF_ST_MEM(BPF_B, BPF_REG_1, 0, 4), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = REJECT, + .errstr = "invalid access to map value pointer", +}, +{ + "direct map access, write test 12", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_MAP_VALUE(BPF_REG_1, 0, (1<<29)), + BPF_ST_MEM(BPF_B, BPF_REG_1, 0, 4), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = REJECT, + .errstr = "direct value offset of 536870912 is not allowed", +}, +{ + "direct map access, write test 13", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_MAP_VALUE(BPF_REG_1, 0, (1<<29)-1), + BPF_ST_MEM(BPF_B, BPF_REG_1, 0, 4), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = REJECT, + .errstr = "invalid access to map value pointer, value_size=48 off=536870911", +}, +{ + "direct map access, write test 14", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 47), + BPF_LD_MAP_VALUE(BPF_REG_2, 0, 46), + BPF_ST_MEM(BPF_H, BPF_REG_2, 0, 0xffff), + BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1, 0), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1, 3 }, + .result = ACCEPT, + .retval = 0xff, +}, +{ + "direct map access, write test 15", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 46), + BPF_LD_MAP_VALUE(BPF_REG_2, 0, 46), + BPF_ST_MEM(BPF_H, BPF_REG_2, 0, 0xffff), + BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_1, 0), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1, 3 }, + .result = ACCEPT, + .retval = 0xffff, +}, +{ + "direct map access, write test 16", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 46), + BPF_LD_MAP_VALUE(BPF_REG_2, 0, 47), + BPF_ST_MEM(BPF_H, BPF_REG_2, 0, 0xffff), + BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_1, 0), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1, 3 }, + .result = REJECT, + .errstr = "invalid access to map value, value_size=48 off=47 size=2", +}, +{ + "direct map access, write test 17", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 46), + BPF_LD_MAP_VALUE(BPF_REG_2, 0, 46), + BPF_ST_MEM(BPF_H, BPF_REG_2, 1, 0xffff), + BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_1, 0), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1, 3 }, + .result = REJECT, + .errstr = "invalid access to map value, value_size=48 off=47 size=2", +}, +{ + "direct map access, write test 18", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0), + BPF_ST_MEM(BPF_H, BPF_REG_1, 0, 42), + BPF_EXIT_INSN(), + }, + .fixup_map_array_small = { 1 }, + .result = REJECT, + .errstr = "R1 min value is outside of the array range", +}, +{ + "direct map access, write test 19", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0), + BPF_ST_MEM(BPF_B, BPF_REG_1, 0, 42), + BPF_EXIT_INSN(), + }, + .fixup_map_array_small = { 1 }, + .result = ACCEPT, + .retval = 1, +}, +{ + "direct map access, write test 20", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 1), + BPF_ST_MEM(BPF_B, BPF_REG_1, 0, 42), + BPF_EXIT_INSN(), + }, + .fixup_map_array_small = { 1 }, + .result = REJECT, + .errstr = "invalid access to map value pointer", +}, +{ + "direct map access, invalid insn test 1", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_VALUE, 0, 1, 0, 47), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = REJECT, + .errstr = "invalid bpf_ld_imm64 insn", +}, +{ + "direct map access, invalid insn test 2", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_VALUE, 1, 0, 0, 47), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = REJECT, + .errstr = "BPF_LD_IMM64 uses reserved fields", +}, +{ + "direct map access, invalid insn test 3", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_VALUE, ~0, 0, 0, 47), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = REJECT, + .errstr = "BPF_LD_IMM64 uses reserved fields", +}, +{ + "direct map access, invalid insn test 4", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_VALUE, 0, ~0, 0, 47), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = REJECT, + .errstr = "invalid bpf_ld_imm64 insn", +}, +{ + "direct map access, invalid insn test 5", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_VALUE, ~0, ~0, 0, 47), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = REJECT, + .errstr = "invalid bpf_ld_imm64 insn", +}, +{ + "direct map access, invalid insn test 6", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_FD, ~0, 0, 0, 0), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = REJECT, + .errstr = "BPF_LD_IMM64 uses reserved fields", +}, +{ + "direct map access, invalid insn test 7", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_FD, 0, ~0, 0, 0), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = REJECT, + .errstr = "invalid bpf_ld_imm64 insn", +}, +{ + "direct map access, invalid insn test 8", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_FD, ~0, ~0, 0, 0), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = REJECT, + .errstr = "invalid bpf_ld_imm64 insn", +}, +{ + "direct map access, invalid insn test 9", + .insns = { + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_FD, 0, 0, 0, 47), + BPF_EXIT_INSN(), + }, + .fixup_map_array_48b = { 1 }, + .result = REJECT, + .errstr = "unrecognized bpf_ld_imm64 insn", +}, diff --git a/tools/testing/selftests/bpf/verifier/int_ptr.c b/tools/testing/selftests/bpf/verifier/int_ptr.c new file mode 100644 index 0000000000000000000000000000000000000000..ca3b4729df66ca3b86fd4a1ba377601d81e31336 --- /dev/null +++ b/tools/testing/selftests/bpf/verifier/int_ptr.c @@ -0,0 +1,160 @@ +{ + "ARG_PTR_TO_LONG uninitialized", + .insns = { + /* bpf_strtoul arg1 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, 0x00303036), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + + BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), + + /* bpf_strtoul arg2 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_2, 4), + + /* bpf_strtoul arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + + /* bpf_strtoul arg4 (res) */ + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_REG(BPF_REG_4, BPF_REG_7), + + /* bpf_strtoul() */ + BPF_EMIT_CALL(BPF_FUNC_strtoul), + + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + }, + .result = REJECT, + .prog_type = BPF_PROG_TYPE_CGROUP_SYSCTL, + .errstr = "invalid indirect read from stack off -16+0 size 8", +}, +{ + "ARG_PTR_TO_LONG half-uninitialized", + .insns = { + /* bpf_strtoul arg1 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, 0x00303036), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + + BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), + + /* bpf_strtoul arg2 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_2, 4), + + /* bpf_strtoul arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + + /* bpf_strtoul arg4 (res) */ + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_STX_MEM(BPF_W, BPF_REG_7, BPF_REG_0, 0), + BPF_MOV64_REG(BPF_REG_4, BPF_REG_7), + + /* bpf_strtoul() */ + BPF_EMIT_CALL(BPF_FUNC_strtoul), + + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + }, + .result = REJECT, + .prog_type = BPF_PROG_TYPE_CGROUP_SYSCTL, + .errstr = "invalid indirect read from stack off -16+4 size 8", +}, +{ + "ARG_PTR_TO_LONG misaligned", + .insns = { + /* bpf_strtoul arg1 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, 0x00303036), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + + BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), + + /* bpf_strtoul arg2 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_2, 4), + + /* bpf_strtoul arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + + /* bpf_strtoul arg4 (res) */ + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -12), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_STX_MEM(BPF_W, BPF_REG_7, BPF_REG_0, 0), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 4), + BPF_MOV64_REG(BPF_REG_4, BPF_REG_7), + + /* bpf_strtoul() */ + BPF_EMIT_CALL(BPF_FUNC_strtoul), + + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + }, + .result = REJECT, + .prog_type = BPF_PROG_TYPE_CGROUP_SYSCTL, + .errstr = "misaligned stack access off (0x0; 0x0)+-20+0 size 8", +}, +{ + "ARG_PTR_TO_LONG size < sizeof(long)", + .insns = { + /* bpf_strtoul arg1 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -16), + BPF_MOV64_IMM(BPF_REG_0, 0x00303036), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + + BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), + + /* bpf_strtoul arg2 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_2, 4), + + /* bpf_strtoul arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + + /* bpf_strtoul arg4 (res) */ + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, 12), + BPF_STX_MEM(BPF_W, BPF_REG_7, BPF_REG_0, 0), + BPF_MOV64_REG(BPF_REG_4, BPF_REG_7), + + /* bpf_strtoul() */ + BPF_EMIT_CALL(BPF_FUNC_strtoul), + + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + }, + .result = REJECT, + .prog_type = BPF_PROG_TYPE_CGROUP_SYSCTL, + .errstr = "invalid stack type R4 off=-4 access_size=8", +}, +{ + "ARG_PTR_TO_LONG initialized", + .insns = { + /* bpf_strtoul arg1 (buf) */ + BPF_MOV64_REG(BPF_REG_7, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_MOV64_IMM(BPF_REG_0, 0x00303036), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + + BPF_MOV64_REG(BPF_REG_1, BPF_REG_7), + + /* bpf_strtoul arg2 (buf_len) */ + BPF_MOV64_IMM(BPF_REG_2, 4), + + /* bpf_strtoul arg3 (flags) */ + BPF_MOV64_IMM(BPF_REG_3, 0), + + /* bpf_strtoul arg4 (res) */ + BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8), + BPF_STX_MEM(BPF_DW, BPF_REG_7, BPF_REG_0, 0), + BPF_MOV64_REG(BPF_REG_4, BPF_REG_7), + + /* bpf_strtoul() */ + BPF_EMIT_CALL(BPF_FUNC_strtoul), + + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_CGROUP_SYSCTL, +}, diff --git a/tools/testing/selftests/bpf/verifier/jit.c b/tools/testing/selftests/bpf/verifier/jit.c index be488b4495a330750c79a0ed44840512acf79869..c33adf344faeb3f70706358cbe782e29cd0acdf5 100644 --- a/tools/testing/selftests/bpf/verifier/jit.c +++ b/tools/testing/selftests/bpf/verifier/jit.c @@ -86,3 +86,22 @@ .result = ACCEPT, .retval = 2, }, +{ + "jit: jsgt, jslt", + .insns = { + BPF_LD_IMM64(BPF_REG_1, 0x80000000ULL), + BPF_LD_IMM64(BPF_REG_2, 0x0ULL), + BPF_JMP_REG(BPF_JSGT, BPF_REG_1, BPF_REG_2, 2), + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + + BPF_JMP_REG(BPF_JSLT, BPF_REG_2, BPF_REG_1, 2), + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + + BPF_MOV64_IMM(BPF_REG_0, 2), + BPF_EXIT_INSN(), + }, + .result = ACCEPT, + .retval = 2, +}, diff --git a/tools/testing/selftests/bpf/verifier/ld_dw.c b/tools/testing/selftests/bpf/verifier/ld_dw.c index d2c75b889598a777384459a8acf538f09f3d5f35..0f18e62f0099bbcf2e6bb982138c42cda40b0d76 100644 --- a/tools/testing/selftests/bpf/verifier/ld_dw.c +++ b/tools/testing/selftests/bpf/verifier/ld_dw.c @@ -34,3 +34,12 @@ .result = ACCEPT, .retval = 5, }, +{ + "ld_dw: xor semi-random 64 bit imms, test 5", + .insns = { }, + .data = { }, + .fill_helper = bpf_fill_rand_ld_dw, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .result = ACCEPT, + .retval = 1000000 - 6, +}, diff --git a/tools/testing/selftests/bpf/verifier/raw_tp_writable.c b/tools/testing/selftests/bpf/verifier/raw_tp_writable.c new file mode 100644 index 0000000000000000000000000000000000000000..95b5d70a1dc126a3e5871e8d060df294571cf7e1 --- /dev/null +++ b/tools/testing/selftests/bpf/verifier/raw_tp_writable.c @@ -0,0 +1,34 @@ +{ + "raw_tracepoint_writable: reject variable offset", + .insns = { + /* r6 is our tp buffer */ + BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0), + + BPF_LD_MAP_FD(BPF_REG_1, 0), + /* move the key (== 0) to r10-8 */ + BPF_MOV32_IMM(BPF_REG_0, 0), + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8), + BPF_STX_MEM(BPF_DW, BPF_REG_2, BPF_REG_0, 0), + /* lookup in the map */ + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, + BPF_FUNC_map_lookup_elem), + + /* exit clean if null */ + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), + BPF_EXIT_INSN(), + + /* shift the buffer pointer to a variable location */ + BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, 0), + BPF_ALU64_REG(BPF_ADD, BPF_REG_6, BPF_REG_0), + /* clobber whatever's there */ + BPF_MOV64_IMM(BPF_REG_7, 4242), + BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_7, 0), + + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .fixup_map_hash_8b = { 1, }, + .prog_type = BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE, + .errstr = "R6 invalid variable buffer offset: off=0, var_off=(0x0; 0xffffffff)", +}, diff --git a/tools/testing/selftests/bpf/verifier/ref_tracking.c b/tools/testing/selftests/bpf/verifier/ref_tracking.c index 923f2110072d6f1f4a124824228c082d43f094d8..ebcbf154c4600d7c9f8fc6d1acc63061cbaaa1df 100644 --- a/tools/testing/selftests/bpf/verifier/ref_tracking.c +++ b/tools/testing/selftests/bpf/verifier/ref_tracking.c @@ -1,7 +1,18 @@ { "reference tracking: leak potential reference", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), + BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), /* leak reference */ + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .errstr = "Unreleased reference", + .result = REJECT, +}, +{ + "reference tracking: leak potential reference to sock_common", + .insns = { + BPF_SK_LOOKUP(skc_lookup_tcp), BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), /* leak reference */ BPF_EXIT_INSN(), }, @@ -12,7 +23,7 @@ { "reference tracking: leak potential reference on stack", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_4, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, -8), BPF_STX_MEM(BPF_DW, BPF_REG_4, BPF_REG_0, 0), @@ -26,7 +37,7 @@ { "reference tracking: leak potential reference on stack 2", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_4, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, -8), BPF_STX_MEM(BPF_DW, BPF_REG_4, BPF_REG_0, 0), @@ -41,7 +52,18 @@ { "reference tracking: zero potential reference", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), + BPF_MOV64_IMM(BPF_REG_0, 0), /* leak reference */ + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .errstr = "Unreleased reference", + .result = REJECT, +}, +{ + "reference tracking: zero potential reference to sock_common", + .insns = { + BPF_SK_LOOKUP(skc_lookup_tcp), BPF_MOV64_IMM(BPF_REG_0, 0), /* leak reference */ BPF_EXIT_INSN(), }, @@ -52,7 +74,7 @@ { "reference tracking: copy and zero potential references", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_7, BPF_REG_0), BPF_MOV64_IMM(BPF_REG_0, 0), BPF_MOV64_IMM(BPF_REG_7, 0), /* leak reference */ @@ -65,7 +87,7 @@ { "reference tracking: release reference without check", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), /* reference in r0 may be NULL */ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), BPF_MOV64_IMM(BPF_REG_2, 0), @@ -76,10 +98,36 @@ .errstr = "type=sock_or_null expected=sock", .result = REJECT, }, +{ + "reference tracking: release reference to sock_common without check", + .insns = { + BPF_SK_LOOKUP(skc_lookup_tcp), + /* reference in r0 may be NULL */ + BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), + BPF_MOV64_IMM(BPF_REG_2, 0), + BPF_EMIT_CALL(BPF_FUNC_sk_release), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .errstr = "type=sock_common_or_null expected=sock", + .result = REJECT, +}, { "reference tracking: release reference", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), + BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), + BPF_EMIT_CALL(BPF_FUNC_sk_release), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .result = ACCEPT, +}, +{ + "reference tracking: release reference to sock_common", + .insns = { + BPF_SK_LOOKUP(skc_lookup_tcp), BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), BPF_EMIT_CALL(BPF_FUNC_sk_release), @@ -91,7 +139,7 @@ { "reference tracking: release reference 2", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), BPF_EXIT_INSN(), @@ -104,7 +152,7 @@ { "reference tracking: release reference twice", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), @@ -120,7 +168,7 @@ { "reference tracking: release reference twice inside branch", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3), /* goto end */ @@ -147,7 +195,7 @@ BPF_EXIT_INSN(), BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_2, offsetof(struct __sk_buff, mark)), - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_JMP_IMM(BPF_JEQ, BPF_REG_6, 0, 1), /* mark == 0? */ /* Leak reference in R0 */ BPF_EXIT_INSN(), @@ -175,7 +223,7 @@ BPF_EXIT_INSN(), BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_2, offsetof(struct __sk_buff, mark)), - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_JMP_IMM(BPF_JEQ, BPF_REG_6, 0, 4), /* mark == 0? */ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), /* sk NULL? */ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), @@ -193,7 +241,7 @@ { "reference tracking in call: free reference in subprog", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), /* unchecked reference */ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 2), BPF_MOV64_IMM(BPF_REG_0, 0), @@ -211,7 +259,7 @@ { "reference tracking in call: free reference in subprog and outside", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), /* unchecked reference */ BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 3), @@ -241,7 +289,7 @@ /* subprog 1 */ BPF_MOV64_REG(BPF_REG_6, BPF_REG_4), - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), /* spill unchecked sk_ptr into stack of caller */ BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_0, 0), BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), @@ -262,7 +310,7 @@ BPF_EXIT_INSN(), /* subprog 1 */ - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_EXIT_INSN(), /* return sk */ }, .prog_type = BPF_PROG_TYPE_SCHED_CLS, @@ -291,7 +339,7 @@ BPF_EXIT_INSN(), /* subprog 2 */ - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_EXIT_INSN(), }, .prog_type = BPF_PROG_TYPE_SCHED_CLS, @@ -324,7 +372,7 @@ BPF_EXIT_INSN(), /* subprog 2 */ - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_EXIT_INSN(), }, .prog_type = BPF_PROG_TYPE_SCHED_CLS, @@ -334,7 +382,7 @@ "reference tracking: allow LD_ABS", .insns = { BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), BPF_EMIT_CALL(BPF_FUNC_sk_release), @@ -350,7 +398,7 @@ "reference tracking: forbid LD_ABS while holding reference", .insns = { BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_LD_ABS(BPF_B, 0), BPF_LD_ABS(BPF_H, 0), BPF_LD_ABS(BPF_W, 0), @@ -367,7 +415,7 @@ "reference tracking: allow LD_IND", .insns = { BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), BPF_EMIT_CALL(BPF_FUNC_sk_release), @@ -384,7 +432,7 @@ "reference tracking: forbid LD_IND while holding reference", .insns = { BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_4, BPF_REG_0), BPF_MOV64_IMM(BPF_REG_7, 1), BPF_LD_IND(BPF_W, BPF_REG_7, -0x200000), @@ -402,7 +450,7 @@ "reference tracking: check reference or tail call", .insns = { BPF_MOV64_REG(BPF_REG_7, BPF_REG_1), - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), /* if (sk) bpf_sk_release() */ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 7), @@ -424,7 +472,7 @@ "reference tracking: release reference then tail call", .insns = { BPF_MOV64_REG(BPF_REG_7, BPF_REG_1), - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), /* if (sk) bpf_sk_release() */ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 1), @@ -446,7 +494,7 @@ .insns = { BPF_MOV64_REG(BPF_REG_7, BPF_REG_1), /* Look up socket and store in REG_6 */ - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), /* bpf_tail_call() */ BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), BPF_MOV64_IMM(BPF_REG_3, 2), @@ -470,7 +518,7 @@ .insns = { BPF_MOV64_REG(BPF_REG_7, BPF_REG_1), /* Look up socket and store in REG_6 */ - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), /* if (!sk) goto end */ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 7), @@ -492,7 +540,7 @@ { "reference tracking: mangle and release sock_or_null", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 5), BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), @@ -506,7 +554,7 @@ { "reference tracking: mangle and release sock", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 5), @@ -520,7 +568,7 @@ { "reference tracking: access member", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3), BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_0, 4), @@ -534,7 +582,7 @@ { "reference tracking: write to member", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 5), BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), @@ -553,7 +601,7 @@ { "reference tracking: invalid 64-bit access of member", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3), BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_0, 0), @@ -568,7 +616,7 @@ { "reference tracking: access after release", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_1, BPF_REG_0), BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), BPF_EMIT_CALL(BPF_FUNC_sk_release), @@ -608,7 +656,7 @@ { "reference tracking: use ptr from bpf_tcp_sock() after release", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), BPF_EXIT_INSN(), BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), @@ -631,7 +679,7 @@ { "reference tracking: use ptr from bpf_sk_fullsock() after release", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), BPF_EXIT_INSN(), BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), @@ -654,7 +702,7 @@ { "reference tracking: use ptr from bpf_sk_fullsock(tp) after release", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), BPF_EXIT_INSN(), BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), @@ -681,7 +729,7 @@ { "reference tracking: use sk after bpf_sk_release(tp)", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), BPF_EXIT_INSN(), BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), @@ -703,7 +751,7 @@ { "reference tracking: use ptr from bpf_get_listener_sock() after bpf_sk_release(sk)", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), BPF_EXIT_INSN(), BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), @@ -725,7 +773,7 @@ { "reference tracking: bpf_sk_release(listen_sk)", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), BPF_EXIT_INSN(), BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), @@ -750,7 +798,7 @@ /* !bpf_sk_fullsock(sk) is checked but !bpf_tcp_sock(sk) is not checked */ "reference tracking: tp->snd_cwnd after bpf_sk_fullsock(sk) and bpf_tcp_sock(sk)", .insns = { - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1), BPF_EXIT_INSN(), BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), diff --git a/tools/testing/selftests/bpf/verifier/scale.c b/tools/testing/selftests/bpf/verifier/scale.c new file mode 100644 index 0000000000000000000000000000000000000000..7f868d4802e0e1feb48ec399efdfae0e576c699e --- /dev/null +++ b/tools/testing/selftests/bpf/verifier/scale.c @@ -0,0 +1,18 @@ +{ + "scale: scale test 1", + .insns = { }, + .data = { }, + .fill_helper = bpf_fill_scale, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .result = ACCEPT, + .retval = 1, +}, +{ + "scale: scale test 2", + .insns = { }, + .data = { }, + .fill_helper = bpf_fill_scale, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .result = ACCEPT, + .retval = 2, +}, diff --git a/tools/testing/selftests/bpf/verifier/sock.c b/tools/testing/selftests/bpf/verifier/sock.c index 416436231fab011aeebfbed0f0f253ad25bef912..b31cd2cf50d0fab72ba38e08877e4b1471cd407d 100644 --- a/tools/testing/selftests/bpf/verifier/sock.c +++ b/tools/testing/selftests/bpf/verifier/sock.c @@ -382,3 +382,119 @@ .result = REJECT, .errstr = "reference has not been acquired before", }, +{ + "sk_storage_get(map, skb->sk, NULL, 0): value == NULL", + .insns = { + BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), + BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + BPF_MOV64_IMM(BPF_REG_4, 0), + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), + BPF_LD_MAP_FD(BPF_REG_1, 0), + BPF_EMIT_CALL(BPF_FUNC_sk_storage_get), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .fixup_sk_storage_map = { 11 }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .result = ACCEPT, +}, +{ + "sk_storage_get(map, skb->sk, 1, 1): value == 1", + .insns = { + BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), + BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + BPF_MOV64_IMM(BPF_REG_4, 1), + BPF_MOV64_IMM(BPF_REG_3, 1), + BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), + BPF_LD_MAP_FD(BPF_REG_1, 0), + BPF_EMIT_CALL(BPF_FUNC_sk_storage_get), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .fixup_sk_storage_map = { 11 }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .result = REJECT, + .errstr = "R3 type=inv expected=fp", +}, +{ + "sk_storage_get(map, skb->sk, &stack_value, 1): stack_value", + .insns = { + BPF_MOV64_IMM(BPF_REG_2, 0), + BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_2, -8), + BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), + BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + BPF_MOV64_IMM(BPF_REG_4, 1), + BPF_MOV64_REG(BPF_REG_3, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, -8), + BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), + BPF_LD_MAP_FD(BPF_REG_1, 0), + BPF_EMIT_CALL(BPF_FUNC_sk_storage_get), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .fixup_sk_storage_map = { 14 }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .result = ACCEPT, +}, +{ + "sk_storage_get(map, skb->sk, &stack_value, 1): partially init stack_value", + .insns = { + BPF_MOV64_IMM(BPF_REG_2, 0), + BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_2, -8), + BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, offsetof(struct __sk_buff, sk)), + BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 2), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + BPF_EMIT_CALL(BPF_FUNC_sk_fullsock), + BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 2), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + BPF_MOV64_IMM(BPF_REG_4, 1), + BPF_MOV64_REG(BPF_REG_3, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, -8), + BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), + BPF_LD_MAP_FD(BPF_REG_1, 0), + BPF_EMIT_CALL(BPF_FUNC_sk_storage_get), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .fixup_sk_storage_map = { 14 }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .result = REJECT, + .errstr = "invalid indirect read from stack", +}, +{ + "bpf_map_lookup_elem(smap, &key)", + .insns = { + BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), + BPF_LD_MAP_FD(BPF_REG_1, 0), + BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .fixup_sk_storage_map = { 3 }, + .prog_type = BPF_PROG_TYPE_SCHED_CLS, + .result = REJECT, + .errstr = "cannot pass map_type 24 into func bpf_map_lookup_elem", +}, diff --git a/tools/testing/selftests/bpf/verifier/unpriv.c b/tools/testing/selftests/bpf/verifier/unpriv.c index dbaf5be947b2be8ea3007cae023ef94be600a553..91bb77c24a2ef3bb392ce2763e90d7a05f2af34d 100644 --- a/tools/testing/selftests/bpf/verifier/unpriv.c +++ b/tools/testing/selftests/bpf/verifier/unpriv.c @@ -242,7 +242,7 @@ .insns = { BPF_MOV64_REG(BPF_REG_8, BPF_REG_1), /* struct bpf_sock *sock = bpf_sock_lookup(...); */ - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), /* u64 foo; */ /* void *target = &foo; */ @@ -276,7 +276,7 @@ .insns = { BPF_MOV64_REG(BPF_REG_8, BPF_REG_1), /* struct bpf_sock *sock = bpf_sock_lookup(...); */ - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), /* u64 foo; */ /* void *target = &foo; */ @@ -307,7 +307,7 @@ .insns = { BPF_MOV64_REG(BPF_REG_8, BPF_REG_1), /* struct bpf_sock *sock = bpf_sock_lookup(...); */ - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), /* u64 foo; */ /* void *target = &foo; */ @@ -339,7 +339,7 @@ .insns = { BPF_MOV64_REG(BPF_REG_8, BPF_REG_1), /* struct bpf_sock *sock = bpf_sock_lookup(...); */ - BPF_SK_LOOKUP, + BPF_SK_LOOKUP(sk_lookup_tcp), BPF_MOV64_REG(BPF_REG_2, BPF_REG_0), /* u64 foo; */ /* void *target = &foo; */ diff --git a/tools/testing/selftests/bpf/verifier/var_off.c b/tools/testing/selftests/bpf/verifier/var_off.c index 1e536ff121a5c0199a0316b9aec76ed1d0563313..8504ac9378098962be2d7bbec6f04e0907e5699d 100644 --- a/tools/testing/selftests/bpf/verifier/var_off.c +++ b/tools/testing/selftests/bpf/verifier/var_off.c @@ -40,7 +40,35 @@ .prog_type = BPF_PROG_TYPE_LWT_IN, }, { - "indirect variable-offset stack access", + "indirect variable-offset stack access, unbounded", + .insns = { + BPF_MOV64_IMM(BPF_REG_2, 6), + BPF_MOV64_IMM(BPF_REG_3, 28), + /* Fill the top 16 bytes of the stack. */ + BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, 0), + BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), + /* Get an unknown value. */ + BPF_LDX_MEM(BPF_DW, BPF_REG_4, BPF_REG_1, offsetof(struct bpf_sock_ops, + bytes_received)), + /* Check the lower bound but don't check the upper one. */ + BPF_JMP_IMM(BPF_JSLT, BPF_REG_4, 0, 4), + /* Point the lower bound to initialized stack. Offset is now in range + * from fp-16 to fp+0x7fffffffffffffef, i.e. max value is unbounded. + */ + BPF_ALU64_IMM(BPF_SUB, BPF_REG_4, 16), + BPF_ALU64_REG(BPF_ADD, BPF_REG_4, BPF_REG_10), + BPF_MOV64_IMM(BPF_REG_5, 8), + /* Dereference it indirectly. */ + BPF_EMIT_CALL(BPF_FUNC_getsockopt), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .errstr = "R4 unbounded indirect variable offset stack access", + .result = REJECT, + .prog_type = BPF_PROG_TYPE_SOCK_OPS, +}, +{ + "indirect variable-offset stack access, max out of bound", .insns = { /* Fill the top 8 bytes of the stack */ BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), @@ -60,7 +88,161 @@ BPF_EXIT_INSN(), }, .fixup_map_hash_8b = { 5 }, - .errstr = "variable stack read R2", + .errstr = "R2 max value is outside of stack bound", + .result = REJECT, + .prog_type = BPF_PROG_TYPE_LWT_IN, +}, +{ + "indirect variable-offset stack access, min out of bound", + .insns = { + /* Fill the top 8 bytes of the stack */ + BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), + /* Get an unknown value */ + BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, 0), + /* Make it small and 4-byte aligned */ + BPF_ALU64_IMM(BPF_AND, BPF_REG_2, 4), + BPF_ALU64_IMM(BPF_SUB, BPF_REG_2, 516), + /* add it to fp. We now have either fp-516 or fp-512, but + * we don't know which + */ + BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_10), + /* dereference it indirectly */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .fixup_map_hash_8b = { 5 }, + .errstr = "R2 min value is outside of stack bound", + .result = REJECT, + .prog_type = BPF_PROG_TYPE_LWT_IN, +}, +{ + "indirect variable-offset stack access, max_off+size > max_initialized", + .insns = { + /* Fill only the second from top 8 bytes of the stack. */ + BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, 0), + /* Get an unknown value. */ + BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, 0), + /* Make it small and 4-byte aligned. */ + BPF_ALU64_IMM(BPF_AND, BPF_REG_2, 4), + BPF_ALU64_IMM(BPF_SUB, BPF_REG_2, 16), + /* Add it to fp. We now have either fp-12 or fp-16, but we don't know + * which. fp-12 size 8 is partially uninitialized stack. + */ + BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_10), + /* Dereference it indirectly. */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .fixup_map_hash_8b = { 5 }, + .errstr = "invalid indirect read from stack var_off", + .result = REJECT, + .prog_type = BPF_PROG_TYPE_LWT_IN, +}, +{ + "indirect variable-offset stack access, min_off < min_initialized", + .insns = { + /* Fill only the top 8 bytes of the stack. */ + BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), + /* Get an unknown value */ + BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, 0), + /* Make it small and 4-byte aligned. */ + BPF_ALU64_IMM(BPF_AND, BPF_REG_2, 4), + BPF_ALU64_IMM(BPF_SUB, BPF_REG_2, 16), + /* Add it to fp. We now have either fp-12 or fp-16, but we don't know + * which. fp-16 size 8 is partially uninitialized stack. + */ + BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_10), + /* Dereference it indirectly. */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .fixup_map_hash_8b = { 5 }, + .errstr = "invalid indirect read from stack var_off", .result = REJECT, .prog_type = BPF_PROG_TYPE_LWT_IN, }, +{ + "indirect variable-offset stack access, priv vs unpriv", + .insns = { + /* Fill the top 16 bytes of the stack. */ + BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, 0), + BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), + /* Get an unknown value. */ + BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, 0), + /* Make it small and 4-byte aligned. */ + BPF_ALU64_IMM(BPF_AND, BPF_REG_2, 4), + BPF_ALU64_IMM(BPF_SUB, BPF_REG_2, 16), + /* Add it to fp. We now have either fp-12 or fp-16, we don't know + * which, but either way it points to initialized stack. + */ + BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_10), + /* Dereference it indirectly. */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .fixup_map_hash_8b = { 6 }, + .errstr_unpriv = "R2 stack pointer arithmetic goes out of range, prohibited for !root", + .result_unpriv = REJECT, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_CGROUP_SKB, +}, +{ + "indirect variable-offset stack access, uninitialized", + .insns = { + BPF_MOV64_IMM(BPF_REG_2, 6), + BPF_MOV64_IMM(BPF_REG_3, 28), + /* Fill the top 16 bytes of the stack. */ + BPF_ST_MEM(BPF_W, BPF_REG_10, -16, 0), + BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), + /* Get an unknown value. */ + BPF_LDX_MEM(BPF_W, BPF_REG_4, BPF_REG_1, 0), + /* Make it small and 4-byte aligned. */ + BPF_ALU64_IMM(BPF_AND, BPF_REG_4, 4), + BPF_ALU64_IMM(BPF_SUB, BPF_REG_4, 16), + /* Add it to fp. We now have either fp-12 or fp-16, we don't know + * which, but either way it points to initialized stack. + */ + BPF_ALU64_REG(BPF_ADD, BPF_REG_4, BPF_REG_10), + BPF_MOV64_IMM(BPF_REG_5, 8), + /* Dereference it indirectly. */ + BPF_EMIT_CALL(BPF_FUNC_getsockopt), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .errstr = "invalid indirect read from stack var_off", + .result = REJECT, + .prog_type = BPF_PROG_TYPE_SOCK_OPS, +}, +{ + "indirect variable-offset stack access, ok", + .insns = { + /* Fill the top 16 bytes of the stack. */ + BPF_ST_MEM(BPF_DW, BPF_REG_10, -16, 0), + BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0), + /* Get an unknown value. */ + BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, 0), + /* Make it small and 4-byte aligned. */ + BPF_ALU64_IMM(BPF_AND, BPF_REG_2, 4), + BPF_ALU64_IMM(BPF_SUB, BPF_REG_2, 16), + /* Add it to fp. We now have either fp-12 or fp-16, we don't know + * which, but either way it points to initialized stack. + */ + BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_10), + /* Dereference it indirectly. */ + BPF_LD_MAP_FD(BPF_REG_1, 0), + BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .fixup_map_hash_8b = { 6 }, + .result = ACCEPT, + .prog_type = BPF_PROG_TYPE_LWT_IN, +}, diff --git a/tools/testing/selftests/drivers/net/mlxsw/qos_ets_strict.sh b/tools/testing/selftests/drivers/net/mlxsw/qos_ets_strict.sh new file mode 100755 index 0000000000000000000000000000000000000000..6d1790b5de7ac3a8218992798ed751a1d5ec253c --- /dev/null +++ b/tools/testing/selftests/drivers/net/mlxsw/qos_ets_strict.sh @@ -0,0 +1,311 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# A test for strict prioritization of traffic in the switch. Run two streams of +# traffic, each through a different ingress port, one tagged with PCP of 1, the +# other with PCP of 2. Both streams converge at one egress port, where they are +# assigned TC of, respectively, 1 and 2, with strict priority configured between +# them. In H3, we expect to see (almost) exclusively the high-priority traffic. +# +# Please see qos_mc_aware.sh for an explanation of why we use mausezahn and +# counters instead of just running iperf3. +# +# +---------------------------+ +-----------------------------+ +# | H1 | | H2 | +# | $h1.111 + | | + $h2.222 | +# | 192.0.2.33/28 | | | | 192.0.2.65/28 | +# | e-qos-map 0:1 | | | | e-qos-map 0:2 | +# | | | | | | +# | $h1 + | | + $h2 | +# +-----------------|---------+ +---------|-------------------+ +# | | +# +-----------------|-------------------------------------|-------------------+ +# | $swp1 + + $swp2 | +# | >1Gbps | | >1Gbps | +# | +---------------|-----------+ +----------|----------------+ | +# | | $swp1.111 + | | + $swp2.222 | | +# | | BR111 | SW | BR222 | | +# | | $swp3.111 + | | + $swp3.222 | | +# | +---------------|-----------+ +----------|----------------+ | +# | \_____________________________________/ | +# | | | +# | + $swp3 | +# | | 1Gbps bottleneck | +# | | ETS: (up n->tc n for n in 0..7) | +# | | strict priority | +# +------------------------------------|--------------------------------------+ +# | +# +--------------------|--------------------+ +# | + $h3 H3 | +# | / \ | +# | / \ | +# | $h3.111 + + $h3.222 | +# | 192.0.2.34/28 192.0.2.66/28 | +# +-----------------------------------------+ + +ALL_TESTS=" + ping_ipv4 + test_ets_strict +" + +lib_dir=$(dirname $0)/../../../net/forwarding + +NUM_NETIFS=6 +source $lib_dir/lib.sh +source $lib_dir/devlink_lib.sh +source qos_lib.sh + +h1_create() +{ + simple_if_init $h1 + mtu_set $h1 10000 + + vlan_create $h1 111 v$h1 192.0.2.33/28 + ip link set dev $h1.111 type vlan egress-qos-map 0:1 +} + +h1_destroy() +{ + vlan_destroy $h1 111 + + mtu_restore $h1 + simple_if_fini $h1 +} + +h2_create() +{ + simple_if_init $h2 + mtu_set $h2 10000 + + vlan_create $h2 222 v$h2 192.0.2.65/28 + ip link set dev $h2.222 type vlan egress-qos-map 0:2 +} + +h2_destroy() +{ + vlan_destroy $h2 222 + + mtu_restore $h2 + simple_if_fini $h2 +} + +h3_create() +{ + simple_if_init $h3 + mtu_set $h3 10000 + + vlan_create $h3 111 v$h3 192.0.2.34/28 + vlan_create $h3 222 v$h3 192.0.2.66/28 +} + +h3_destroy() +{ + vlan_destroy $h3 222 + vlan_destroy $h3 111 + + mtu_restore $h3 + simple_if_fini $h3 +} + +switch_create() +{ + ip link set dev $swp1 up + mtu_set $swp1 10000 + + ip link set dev $swp2 up + mtu_set $swp2 10000 + + # prio n -> TC n, strict scheduling + lldptool -T -i $swp3 -V ETS-CFG up2tc=0:0,1:1,2:2,3:3,4:4,5:5,6:6,7:7 + lldptool -T -i $swp3 -V ETS-CFG tsa=$( + )"0:strict,"$( + )"1:strict,"$( + )"2:strict,"$( + )"3:strict,"$( + )"4:strict,"$( + )"5:strict,"$( + )"6:strict,"$( + )"7:strict" + sleep 1 + + ip link set dev $swp3 up + mtu_set $swp3 10000 + ethtool -s $swp3 speed 1000 autoneg off + + vlan_create $swp1 111 + vlan_create $swp2 222 + vlan_create $swp3 111 + vlan_create $swp3 222 + + ip link add name br111 up type bridge vlan_filtering 0 + ip link set dev $swp1.111 master br111 + ip link set dev $swp3.111 master br111 + + ip link add name br222 up type bridge vlan_filtering 0 + ip link set dev $swp2.222 master br222 + ip link set dev $swp3.222 master br222 + + # Make sure that ingress quotas are smaller than egress so that there is + # room for both streams of traffic to be admitted to shared buffer. + devlink_pool_size_thtype_set 0 dynamic 10000000 + devlink_pool_size_thtype_set 4 dynamic 10000000 + + devlink_port_pool_th_set $swp1 0 6 + devlink_tc_bind_pool_th_set $swp1 1 ingress 0 6 + + devlink_port_pool_th_set $swp2 0 6 + devlink_tc_bind_pool_th_set $swp2 2 ingress 0 6 + + devlink_tc_bind_pool_th_set $swp3 1 egress 4 7 + devlink_tc_bind_pool_th_set $swp3 2 egress 4 7 + devlink_port_pool_th_set $swp3 4 7 +} + +switch_destroy() +{ + devlink_port_pool_th_restore $swp3 4 + devlink_tc_bind_pool_th_restore $swp3 2 egress + devlink_tc_bind_pool_th_restore $swp3 1 egress + + devlink_tc_bind_pool_th_restore $swp2 2 ingress + devlink_port_pool_th_restore $swp2 0 + + devlink_tc_bind_pool_th_restore $swp1 1 ingress + devlink_port_pool_th_restore $swp1 0 + + devlink_pool_size_thtype_restore 4 + devlink_pool_size_thtype_restore 0 + + ip link del dev br222 + ip link del dev br111 + + vlan_destroy $swp3 222 + vlan_destroy $swp3 111 + vlan_destroy $swp2 222 + vlan_destroy $swp1 111 + + ethtool -s $swp3 autoneg on + mtu_restore $swp3 + ip link set dev $swp3 down + lldptool -T -i $swp3 -V ETS-CFG up2tc=0:0,1:0,2:0,3:0,4:0,5:0,6:0,7:0 + + mtu_restore $swp2 + ip link set dev $swp2 down + + mtu_restore $swp1 + ip link set dev $swp1 down +} + +setup_prepare() +{ + h1=${NETIFS[p1]} + swp1=${NETIFS[p2]} + + swp2=${NETIFS[p3]} + h2=${NETIFS[p4]} + + swp3=${NETIFS[p5]} + h3=${NETIFS[p6]} + + h3mac=$(mac_get $h3) + + vrf_prepare + + h1_create + h2_create + h3_create + switch_create +} + +cleanup() +{ + pre_cleanup + + switch_destroy + h3_destroy + h2_destroy + h1_destroy + + vrf_cleanup +} + +ping_ipv4() +{ + ping_test $h1 192.0.2.34 " from H1" + ping_test $h2 192.0.2.66 " from H2" +} + +rel() +{ + local old=$1; shift + local new=$1; shift + + bc <<< " + scale=2 + ret = 100 * $new / $old + if (ret > 0) { ret } else { 0 } + " +} + +test_ets_strict() +{ + RET=0 + + # Run high-prio traffic on its own. + start_traffic $h2.222 192.0.2.65 192.0.2.66 $h3mac + local -a rate_2 + rate_2=($(measure_rate $swp2 $h3 rx_octets_prio_2 "prio 2")) + check_err $? "Could not get high enough prio-2 ingress rate" + local rate_2_in=${rate_2[0]} + local rate_2_eg=${rate_2[1]} + stop_traffic # $h2.222 + + # Start low-prio stream. + start_traffic $h1.111 192.0.2.33 192.0.2.34 $h3mac + + local -a rate_1 + rate_1=($(measure_rate $swp1 $h3 rx_octets_prio_1 "prio 1")) + check_err $? "Could not get high enough prio-1 ingress rate" + local rate_1_in=${rate_1[0]} + local rate_1_eg=${rate_1[1]} + + # High-prio and low-prio on their own should have about the same + # throughput. + local rel21=$(rel $rate_1_eg $rate_2_eg) + check_err $(bc <<< "$rel21 < 95") + check_err $(bc <<< "$rel21 > 105") + + # Start the high-prio stream--now both streams run. + start_traffic $h2.222 192.0.2.65 192.0.2.66 $h3mac + rate_3=($(measure_rate $swp2 $h3 rx_octets_prio_2 "prio 2 w/ 1")) + check_err $? "Could not get high enough prio-2 ingress rate with prio-1" + local rate_3_in=${rate_3[0]} + local rate_3_eg=${rate_3[1]} + stop_traffic # $h2.222 + + stop_traffic # $h1.111 + + # High-prio should have about the same throughput whether or not + # low-prio is in the system. + local rel32=$(rel $rate_2_eg $rate_3_eg) + check_err $(bc <<< "$rel32 < 95") + + log_test "strict priority" + echo "Ingress to switch:" + echo " p1 in rate $(humanize $rate_1_in)" + echo " p2 in rate $(humanize $rate_2_in)" + echo " p2 in rate w/ p1 $(humanize $rate_3_in)" + echo "Egress from switch:" + echo " p1 eg rate $(humanize $rate_1_eg)" + echo " p2 eg rate $(humanize $rate_2_eg) ($rel21% of p1)" + echo " p2 eg rate w/ p1 $(humanize $rate_3_eg) ($rel32% of p2)" +} + +trap cleanup EXIT + +setup_prepare +setup_wait + +tests_run + +exit $EXIT_STATUS diff --git a/tools/testing/selftests/drivers/net/mlxsw/qos_lib.sh b/tools/testing/selftests/drivers/net/mlxsw/qos_lib.sh new file mode 100644 index 0000000000000000000000000000000000000000..e80be65799adf6ff3b46e0883f14bfbc3f11d7b4 --- /dev/null +++ b/tools/testing/selftests/drivers/net/mlxsw/qos_lib.sh @@ -0,0 +1,98 @@ +# SPDX-License-Identifier: GPL-2.0 + +humanize() +{ + local speed=$1; shift + + for unit in bps Kbps Mbps Gbps; do + if (($(echo "$speed < 1024" | bc))); then + break + fi + + speed=$(echo "scale=1; $speed / 1024" | bc) + done + + echo "$speed${unit}" +} + +rate() +{ + local t0=$1; shift + local t1=$1; shift + local interval=$1; shift + + echo $((8 * (t1 - t0) / interval)) +} + +start_traffic() +{ + local h_in=$1; shift # Where the traffic egresses the host + local sip=$1; shift + local dip=$1; shift + local dmac=$1; shift + + $MZ $h_in -p 8000 -A $sip -B $dip -c 0 \ + -a own -b $dmac -t udp -q & + sleep 1 +} + +stop_traffic() +{ + # Suppress noise from killing mausezahn. + { kill %% && wait %%; } 2>/dev/null +} + +check_rate() +{ + local rate=$1; shift + local min=$1; shift + local what=$1; shift + + if ((rate > min)); then + return 0 + fi + + echo "$what $(humanize $ir) < $(humanize $min)" > /dev/stderr + return 1 +} + +measure_rate() +{ + local sw_in=$1; shift # Where the traffic ingresses the switch + local host_in=$1; shift # Where it ingresses another host + local counter=$1; shift # Counter to use for measurement + local what=$1; shift + + local interval=10 + local i + local ret=0 + + # Dips in performance might cause momentary ingress rate to drop below + # 1Gbps. That wouldn't saturate egress and MC would thus get through, + # seemingly winning bandwidth on account of UC. Demand at least 2Gbps + # average ingress rate to somewhat mitigate this. + local min_ingress=2147483648 + + for i in {5..0}; do + local t0=$(ethtool_stats_get $host_in $counter) + local u0=$(ethtool_stats_get $sw_in $counter) + sleep $interval + local t1=$(ethtool_stats_get $host_in $counter) + local u1=$(ethtool_stats_get $sw_in $counter) + + local ir=$(rate $u0 $u1 $interval) + local er=$(rate $t0 $t1 $interval) + + if check_rate $ir $min_ingress "$what ingress rate"; then + break + fi + + # Fail the test if we can't get the throughput. + if ((i == 0)); then + ret=1 + fi + done + + echo $ir $er + return $ret +} diff --git a/tools/testing/selftests/drivers/net/mlxsw/qos_mc_aware.sh b/tools/testing/selftests/drivers/net/mlxsw/qos_mc_aware.sh index 117f6f35d72fac57ad87f0ed4772a50511d45cb4..71231ad2dbfb5d2d4c3082a6669578dab59fcaef 100755 --- a/tools/testing/selftests/drivers/net/mlxsw/qos_mc_aware.sh +++ b/tools/testing/selftests/drivers/net/mlxsw/qos_mc_aware.sh @@ -67,6 +67,8 @@ lib_dir=$(dirname $0)/../../../net/forwarding NUM_NETIFS=6 source $lib_dir/lib.sh +source $lib_dir/devlink_lib.sh +source qos_lib.sh h1_create() { @@ -140,10 +142,28 @@ switch_create() ip link set dev br111 up ip link set dev $swp2.111 master br111 ip link set dev $swp3.111 master br111 + + # Make sure that ingress quotas are smaller than egress so that there is + # room for both streams of traffic to be admitted to shared buffer. + devlink_port_pool_th_set $swp1 0 5 + devlink_tc_bind_pool_th_set $swp1 0 ingress 0 5 + + devlink_port_pool_th_set $swp2 0 5 + devlink_tc_bind_pool_th_set $swp2 1 ingress 0 5 + + devlink_port_pool_th_set $swp3 4 12 } switch_destroy() { + devlink_port_pool_th_restore $swp3 4 + + devlink_tc_bind_pool_th_restore $swp2 1 ingress + devlink_port_pool_th_restore $swp2 0 + + devlink_tc_bind_pool_th_restore $swp1 0 ingress + devlink_port_pool_th_restore $swp1 0 + ip link del dev br111 ip link del dev br1 @@ -201,107 +221,28 @@ ping_ipv4() ping_test $h2 192.0.2.130 } -humanize() -{ - local speed=$1; shift - - for unit in bps Kbps Mbps Gbps; do - if (($(echo "$speed < 1024" | bc))); then - break - fi - - speed=$(echo "scale=1; $speed / 1024" | bc) - done - - echo "$speed${unit}" -} - -rate() -{ - local t0=$1; shift - local t1=$1; shift - local interval=$1; shift - - echo $((8 * (t1 - t0) / interval)) -} - -check_rate() -{ - local rate=$1; shift - local min=$1; shift - local what=$1; shift - - if ((rate > min)); then - return 0 - fi - - echo "$what $(humanize $ir) < $(humanize $min_ingress)" > /dev/stderr - return 1 -} - -measure_uc_rate() -{ - local what=$1; shift - - local interval=10 - local i - local ret=0 - - # Dips in performance might cause momentary ingress rate to drop below - # 1Gbps. That wouldn't saturate egress and MC would thus get through, - # seemingly winning bandwidth on account of UC. Demand at least 2Gbps - # average ingress rate to somewhat mitigate this. - local min_ingress=2147483648 - - $MZ $h2.111 -p 8000 -A 192.0.2.129 -B 192.0.2.130 -c 0 \ - -a own -b $h3mac -t udp -q & - sleep 1 - - for i in {5..0}; do - local t0=$(ethtool_stats_get $h3 rx_octets_prio_1) - local u0=$(ethtool_stats_get $swp2 rx_octets_prio_1) - sleep $interval - local t1=$(ethtool_stats_get $h3 rx_octets_prio_1) - local u1=$(ethtool_stats_get $swp2 rx_octets_prio_1) - - local ir=$(rate $u0 $u1 $interval) - local er=$(rate $t0 $t1 $interval) - - if check_rate $ir $min_ingress "$what ingress rate"; then - break - fi - - # Fail the test if we can't get the throughput. - if ((i == 0)); then - ret=1 - fi - done - - # Suppress noise from killing mausezahn. - { kill %% && wait; } 2>/dev/null - - echo $ir $er - exit $ret -} - test_mc_aware() { RET=0 local -a uc_rate - uc_rate=($(measure_uc_rate "UC-only")) + start_traffic $h2.111 192.0.2.129 192.0.2.130 $h3mac + uc_rate=($(measure_rate $swp2 $h3 rx_octets_prio_1 "UC-only")) check_err $? "Could not get high enough UC-only ingress rate" + stop_traffic local ucth1=${uc_rate[1]} - $MZ $h1 -p 8000 -c 0 -a own -b bc -t udp -q & + start_traffic $h1 own bc bc local d0=$(date +%s) local t0=$(ethtool_stats_get $h3 rx_octets_prio_0) local u0=$(ethtool_stats_get $swp1 rx_octets_prio_0) local -a uc_rate_2 - uc_rate_2=($(measure_uc_rate "UC+MC")) + start_traffic $h2.111 192.0.2.129 192.0.2.130 $h3mac + uc_rate_2=($(measure_rate $swp2 $h3 rx_octets_prio_1 "UC+MC")) check_err $? "Could not get high enough UC+MC ingress rate" + stop_traffic local ucth2=${uc_rate_2[1]} local d1=$(date +%s) @@ -319,8 +260,7 @@ test_mc_aware() local mc_ir=$(rate $u0 $u1 $interval) local mc_er=$(rate $t0 $t1 $interval) - # Suppress noise from killing mausezahn. - { kill %% && wait; } 2>/dev/null + stop_traffic log_test "UC performace under MC overload" @@ -344,8 +284,7 @@ test_uc_aware() { RET=0 - $MZ $h2.111 -p 8000 -A 192.0.2.129 -B 192.0.2.130 -c 0 \ - -a own -b $h3mac -t udp -q & + start_traffic $h2.111 192.0.2.129 192.0.2.130 $h3mac local d0=$(date +%s) local t0=$(ethtool_stats_get $h3 rx_octets_prio_1) @@ -375,8 +314,7 @@ test_uc_aware() ((attempts == passes)) check_err $? - # Suppress noise from killing mausezahn. - { kill %% && wait; } 2>/dev/null + stop_traffic log_test "MC performace under UC overload" echo " ingress UC throughput $(humanize ${uc_ir})" diff --git a/tools/testing/selftests/drivers/net/mlxsw/rtnetlink.sh b/tools/testing/selftests/drivers/net/mlxsw/rtnetlink.sh index a6c196c8534cea2a49d600aa23b39070bfd481f0..1c30f302a1e75e42999ac97159673122cd026839 100755 --- a/tools/testing/selftests/drivers/net/mlxsw/rtnetlink.sh +++ b/tools/testing/selftests/drivers/net/mlxsw/rtnetlink.sh @@ -27,6 +27,7 @@ ALL_TESTS=" lag_dev_deletion_test vlan_interface_uppers_test bridge_extern_learn_test + neigh_offload_test devlink_reload_test " NUM_NETIFS=2 @@ -581,6 +582,31 @@ bridge_extern_learn_test() ip link del dev br0 } +neigh_offload_test() +{ + # Test that IPv4 and IPv6 neighbour entries are marked as offloaded + RET=0 + + ip -4 address add 192.0.2.1/24 dev $swp1 + ip -6 address add 2001:db8:1::1/64 dev $swp1 + + ip -4 neigh add 192.0.2.2 lladdr de:ad:be:ef:13:37 nud perm dev $swp1 + ip -6 neigh add 2001:db8:1::2 lladdr de:ad:be:ef:13:37 nud perm \ + dev $swp1 + + ip -4 neigh show dev $swp1 | grep 192.0.2.2 | grep -q offload + check_err $? "ipv4 neigh entry not marked as offloaded when should" + ip -6 neigh show dev $swp1 | grep 2001:db8:1::2 | grep -q offload + check_err $? "ipv6 neigh entry not marked as offloaded when should" + + log_test "neighbour offload indication" + + ip -6 neigh del 2001:db8:1::2 dev $swp1 + ip -4 neigh del 192.0.2.2 dev $swp1 + ip -6 address del 2001:db8:1::1/64 dev $swp1 + ip -4 address del 192.0.2.1/24 dev $swp1 +} + devlink_reload_test() { # Test that after executing all the above configuration tests, a diff --git a/tools/testing/selftests/drivers/net/mlxsw/spectrum-2/tc_flower.sh b/tools/testing/selftests/drivers/net/mlxsw/spectrum-2/tc_flower.sh index a372b2f60874d2a62905f763777717009fd31143..fb850e0ec8375f143c8da0459ee48ca778bb3a29 100755 --- a/tools/testing/selftests/drivers/net/mlxsw/spectrum-2/tc_flower.sh +++ b/tools/testing/selftests/drivers/net/mlxsw/spectrum-2/tc_flower.sh @@ -12,6 +12,7 @@ ALL_TESTS="single_mask_test identical_filters_test two_masks_test \ delta_two_masks_one_key_test delta_simple_rehash_test \ bloom_simple_test bloom_complex_test bloom_delta_test" NUM_NETIFS=2 +source $lib_dir/lib.sh source $lib_dir/tc_common.sh source $lib_dir/devlink_lib.sh diff --git a/tools/testing/selftests/drivers/net/mlxsw/spectrum/devlink_resources.sh b/tools/testing/selftests/drivers/net/mlxsw/spectrum/devlink_resources.sh index b1fe960e398a8da6846e36289c8cb178b03bcc56..6f2683cbc7d5ce8202ec571418bcb5595dfa5101 100755 --- a/tools/testing/selftests/drivers/net/mlxsw/spectrum/devlink_resources.sh +++ b/tools/testing/selftests/drivers/net/mlxsw/spectrum/devlink_resources.sh @@ -1,7 +1,10 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0 +lib_dir=$(dirname $0)/../../../../net/forwarding + NUM_NETIFS=1 +source $lib_dir/lib.sh source devlink_lib_spectrum.sh setup_prepare() diff --git a/tools/testing/selftests/drivers/net/mlxsw/spectrum/resource_scale.sh b/tools/testing/selftests/drivers/net/mlxsw/spectrum/resource_scale.sh index e7ffc79561b7770fc9918afc1a72fe2ccb96a146..43ba1b438f6dbad35bdcb3b65b494f7addd25025 100755 --- a/tools/testing/selftests/drivers/net/mlxsw/spectrum/resource_scale.sh +++ b/tools/testing/selftests/drivers/net/mlxsw/spectrum/resource_scale.sh @@ -1,8 +1,11 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0 +lib_dir=$(dirname $0)/../../../../net/forwarding + NUM_NETIFS=6 -source ../../../../net/forwarding/tc_common.sh +source $lib_dir/lib.sh +source $lib_dir/tc_common.sh source devlink_lib_spectrum.sh current_test="" diff --git a/tools/testing/selftests/net/config b/tools/testing/selftests/net/config index e9c860d0041614fbb3df603e53bc0e460a753a11..47404044860189010c12e6e2cfeecbf3d0e7dcef 100644 --- a/tools/testing/selftests/net/config +++ b/tools/testing/selftests/net/config @@ -7,9 +7,7 @@ CONFIG_NET_L3_MASTER_DEV=y CONFIG_IPV6=y CONFIG_IPV6_MULTIPLE_TABLES=y CONFIG_VETH=y -CONFIG_INET_XFRM_MODE_TUNNEL=y CONFIG_NET_IPVTI=y -CONFIG_INET6_XFRM_MODE_TUNNEL=y CONFIG_IPV6_VTI=y CONFIG_DUMMY=y CONFIG_BRIDGE=y diff --git a/tools/testing/selftests/net/fib_tests.sh b/tools/testing/selftests/net/fib_tests.sh index 0d2a5f4f1e63829f3ca8dfcd8695b91409823f7f..9457aaeae092b7b4145fa7bdc9b6b9fa6acb3e4e 100755 --- a/tools/testing/selftests/net/fib_tests.sh +++ b/tools/testing/selftests/net/fib_tests.sh @@ -9,7 +9,8 @@ ret=0 ksft_skip=4 # all tests in this script. Can be overridden with -t option -TESTS="unregister down carrier nexthop ipv6_rt ipv4_rt ipv6_addr_metric ipv4_addr_metric ipv6_route_metrics ipv4_route_metrics" +TESTS="unregister down carrier nexthop ipv6_rt ipv4_rt ipv6_addr_metric ipv4_addr_metric ipv6_route_metrics ipv4_route_metrics ipv4_route_v6_gw" + VERBOSE=0 PAUSE_ON_FAIL=no PAUSE=no @@ -48,6 +49,7 @@ setup() { set -e ip netns add ns1 + ip netns set ns1 auto $IP link set dev lo up ip netns exec ns1 sysctl -qw net.ipv4.ip_forward=1 ip netns exec ns1 sysctl -qw net.ipv6.conf.all.forwarding=1 @@ -707,6 +709,7 @@ route_setup() set -e ip netns add ns2 + ip netns set ns2 auto ip -netns ns2 link set dev lo up ip netns exec ns2 sysctl -qw net.ipv4.ip_forward=1 ip netns exec ns2 sysctl -qw net.ipv6.conf.all.forwarding=1 @@ -1428,6 +1431,70 @@ ipv4_route_metrics_test() route_cleanup } +ipv4_route_v6_gw_test() +{ + local rc + + echo + echo "IPv4 route with IPv6 gateway tests" + + route_setup + sleep 2 + + # + # single path route + # + run_cmd "$IP ro add 172.16.104.0/24 via inet6 2001:db8:101::2" + rc=$? + log_test $rc 0 "Single path route with IPv6 gateway" + if [ $rc -eq 0 ]; then + check_route "172.16.104.0/24 via inet6 2001:db8:101::2 dev veth1" + fi + + run_cmd "ip netns exec ns1 ping -w1 -c1 172.16.104.1" + log_test $rc 0 "Single path route with IPv6 gateway - ping" + + run_cmd "$IP ro del 172.16.104.0/24 via inet6 2001:db8:101::2" + rc=$? + log_test $rc 0 "Single path route delete" + if [ $rc -eq 0 ]; then + check_route "172.16.112.0/24" + fi + + # + # multipath - v6 then v4 + # + run_cmd "$IP ro add 172.16.104.0/24 nexthop via inet6 2001:db8:101::2 dev veth1 nexthop via 172.16.103.2 dev veth3" + rc=$? + log_test $rc 0 "Multipath route add - v6 nexthop then v4" + if [ $rc -eq 0 ]; then + check_route "172.16.104.0/24 nexthop via inet6 2001:db8:101::2 dev veth1 weight 1 nexthop via 172.16.103.2 dev veth3 weight 1" + fi + + run_cmd "$IP ro del 172.16.104.0/24 nexthop via 172.16.103.2 dev veth3 nexthop via inet6 2001:db8:101::2 dev veth1" + log_test $? 2 " Multipath route delete - nexthops in wrong order" + + run_cmd "$IP ro del 172.16.104.0/24 nexthop via inet6 2001:db8:101::2 dev veth1 nexthop via 172.16.103.2 dev veth3" + log_test $? 0 " Multipath route delete exact match" + + # + # multipath - v4 then v6 + # + run_cmd "$IP ro add 172.16.104.0/24 nexthop via 172.16.103.2 dev veth3 nexthop via inet6 2001:db8:101::2 dev veth1" + rc=$? + log_test $rc 0 "Multipath route add - v4 nexthop then v6" + if [ $rc -eq 0 ]; then + check_route "172.16.104.0/24 nexthop via 172.16.103.2 dev veth3 weight 1 nexthop via inet6 2001:db8:101::2 dev veth1 weight 1" + fi + + run_cmd "$IP ro del 172.16.104.0/24 nexthop via inet6 2001:db8:101::2 dev veth1 nexthop via 172.16.103.2 dev veth3" + log_test $? 2 " Multipath route delete - nexthops in wrong order" + + run_cmd "$IP ro del 172.16.104.0/24 nexthop via 172.16.103.2 dev veth3 nexthop via inet6 2001:db8:101::2 dev veth1" + log_test $? 0 " Multipath route delete exact match" + + route_cleanup +} ################################################################################ # usage @@ -1497,6 +1564,7 @@ do ipv4_addr_metric) ipv4_addr_metric_test;; ipv6_route_metrics) ipv6_route_metrics_test;; ipv4_route_metrics) ipv4_route_metrics_test;; + ipv4_route_v6_gw) ipv4_route_v6_gw_test;; help) echo "Test names: $TESTS"; exit 0;; esac diff --git a/tools/testing/selftests/net/forwarding/bridge_igmp.sh b/tools/testing/selftests/net/forwarding/bridge_igmp.sh new file mode 100755 index 0000000000000000000000000000000000000000..88d2472ba15133fc8afde226d1d29ddb7ebb6eb8 --- /dev/null +++ b/tools/testing/selftests/net/forwarding/bridge_igmp.sh @@ -0,0 +1,152 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +ALL_TESTS="reportleave_test" +NUM_NETIFS=4 +CHECK_TC="yes" +TEST_GROUP="239.10.10.10" +TEST_GROUP_MAC="01:00:5e:0a:0a:0a" +source lib.sh + +h1_create() +{ + simple_if_init $h1 192.0.2.1/24 2001:db8:1::1/64 +} + +h1_destroy() +{ + simple_if_fini $h1 192.0.2.1/24 2001:db8:1::1/64 +} + +h2_create() +{ + simple_if_init $h2 192.0.2.2/24 2001:db8:1::2/64 +} + +h2_destroy() +{ + simple_if_fini $h2 192.0.2.2/24 2001:db8:1::2/64 +} + +switch_create() +{ + ip link add dev br0 type bridge mcast_snooping 1 mcast_querier 1 + + ip link set dev $swp1 master br0 + ip link set dev $swp2 master br0 + + ip link set dev br0 up + ip link set dev $swp1 up + ip link set dev $swp2 up +} + +switch_destroy() +{ + ip link set dev $swp2 down + ip link set dev $swp1 down + + ip link del dev br0 +} + +setup_prepare() +{ + h1=${NETIFS[p1]} + swp1=${NETIFS[p2]} + + swp2=${NETIFS[p3]} + h2=${NETIFS[p4]} + + vrf_prepare + + h1_create + h2_create + + switch_create +} + +cleanup() +{ + pre_cleanup + + switch_destroy + + # Always cleanup the mcast group + ip address del dev $h2 $TEST_GROUP/32 2>&1 1>/dev/null + + h2_destroy + h1_destroy + + vrf_cleanup +} + +# return 0 if the packet wasn't seen on host2_if or 1 if it was +mcast_packet_test() +{ + local mac=$1 + local ip=$2 + local host1_if=$3 + local host2_if=$4 + local seen=0 + + # Add an ACL on `host2_if` which will tell us whether the packet + # was received by it or not. + tc qdisc add dev $host2_if ingress + tc filter add dev $host2_if ingress protocol ip pref 1 handle 101 \ + flower dst_mac $mac action drop + + $MZ $host1_if -c 1 -p 64 -b $mac -B $ip -t udp "dp=4096,sp=2048" -q + sleep 1 + + tc -j -s filter show dev $host2_if ingress \ + | jq -e ".[] | select(.options.handle == 101) \ + | select(.options.actions[0].stats.packets == 1)" &> /dev/null + if [[ $? -eq 0 ]]; then + seen=1 + fi + + tc filter del dev $host2_if ingress protocol ip pref 1 handle 101 flower + tc qdisc del dev $host2_if ingress + + return $seen +} + +reportleave_test() +{ + RET=0 + ip address add dev $h2 $TEST_GROUP/32 autojoin + check_err $? "Could not join $TEST_GROUP" + + sleep 5 + bridge mdb show dev br0 | grep $TEST_GROUP 1>/dev/null + check_err $? "Report didn't create mdb entry for $TEST_GROUP" + + mcast_packet_test $TEST_GROUP_MAC $TEST_GROUP $h1 $h2 + check_fail $? "Traffic to $TEST_GROUP wasn't forwarded" + + log_test "IGMP report $TEST_GROUP" + + RET=0 + bridge mdb show dev br0 | grep $TEST_GROUP 1>/dev/null + check_err $? "mdb entry for $TEST_GROUP is missing" + + ip address del dev $h2 $TEST_GROUP/32 + check_err $? "Could not leave $TEST_GROUP" + + sleep 5 + bridge mdb show dev br0 | grep $TEST_GROUP 1>/dev/null + check_fail $? "Leave didn't delete mdb entry for $TEST_GROUP" + + mcast_packet_test $TEST_GROUP_MAC $TEST_GROUP $h1 $h2 + check_err $? "Traffic to $TEST_GROUP was forwarded without mdb entry" + + log_test "IGMP leave $TEST_GROUP" +} + +trap cleanup EXIT + +setup_prepare +setup_wait + +tests_run + +exit $EXIT_STATUS diff --git a/tools/testing/selftests/net/forwarding/devlink_lib.sh b/tools/testing/selftests/net/forwarding/devlink_lib.sh index 57cf8914910d897f2514d604062365f640d09c87..8553a67a23221205e81334c9b99ee68425d1b872 100644 --- a/tools/testing/selftests/net/forwarding/devlink_lib.sh +++ b/tools/testing/selftests/net/forwarding/devlink_lib.sh @@ -1,22 +1,11 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0 -############################################################################## -# Source library - -relative_path="${BASH_SOURCE%/*}" -if [[ "$relative_path" == "${BASH_SOURCE}" ]]; then - relative_path="." -fi - -source "$relative_path/lib.sh" - ############################################################################## # Defines -DEVLINK_DEV=$(devlink port show | grep "${NETIFS[p1]}" | \ - grep -v "${NETIFS[p1]}[0-9]" | cut -d" " -f1 | \ - rev | cut -d"/" -f2- | rev) +DEVLINK_DEV=$(devlink port show "${NETIFS[p1]}" -j \ + | jq -r '.port | keys[]' | cut -d/ -f-2) if [ -z "$DEVLINK_DEV" ]; then echo "SKIP: ${NETIFS[p1]} has no devlink device registered for it" exit 1 @@ -106,3 +95,98 @@ devlink_reload() grep -c "size_new") check_err $still_pending "Failed reload - There are still unset sizes" } + +declare -A DEVLINK_ORIG + +devlink_port_pool_threshold() +{ + local port=$1; shift + local pool=$1; shift + + devlink sb port pool show $port pool $pool -j \ + | jq '.port_pool."'"$port"'"[].threshold' +} + +devlink_port_pool_th_set() +{ + local port=$1; shift + local pool=$1; shift + local th=$1; shift + local key="port_pool($port,$pool).threshold" + + DEVLINK_ORIG[$key]=$(devlink_port_pool_threshold $port $pool) + devlink sb port pool set $port pool $pool th $th +} + +devlink_port_pool_th_restore() +{ + local port=$1; shift + local pool=$1; shift + local key="port_pool($port,$pool).threshold" + + devlink sb port pool set $port pool $pool th ${DEVLINK_ORIG[$key]} +} + +devlink_pool_size_thtype() +{ + local pool=$1; shift + + devlink sb pool show "$DEVLINK_DEV" pool $pool -j \ + | jq -r '.pool[][] | (.size, .thtype)' +} + +devlink_pool_size_thtype_set() +{ + local pool=$1; shift + local thtype=$1; shift + local size=$1; shift + local key="pool($pool).size_thtype" + + DEVLINK_ORIG[$key]=$(devlink_pool_size_thtype $pool) + devlink sb pool set "$DEVLINK_DEV" pool $pool size $size thtype $thtype +} + +devlink_pool_size_thtype_restore() +{ + local pool=$1; shift + local key="pool($pool).size_thtype" + local -a orig=(${DEVLINK_ORIG[$key]}) + + devlink sb pool set "$DEVLINK_DEV" pool $pool \ + size ${orig[0]} thtype ${orig[1]} +} + +devlink_tc_bind_pool_th() +{ + local port=$1; shift + local tc=$1; shift + local dir=$1; shift + + devlink sb tc bind show $port tc $tc type $dir -j \ + | jq -r '.tc_bind[][] | (.pool, .threshold)' +} + +devlink_tc_bind_pool_th_set() +{ + local port=$1; shift + local tc=$1; shift + local dir=$1; shift + local pool=$1; shift + local th=$1; shift + local key="tc_bind($port,$dir,$tc).pool_th" + + DEVLINK_ORIG[$key]=$(devlink_tc_bind_pool_th $port $tc $dir) + devlink sb tc bind set $port tc $tc type $dir pool $pool th $th +} + +devlink_tc_bind_pool_th_restore() +{ + local port=$1; shift + local tc=$1; shift + local dir=$1; shift + local key="tc_bind($port,$dir,$tc).pool_th" + local -a orig=(${DEVLINK_ORIG[$key]}) + + devlink sb tc bind set $port tc $tc type $dir \ + pool ${orig[0]} th ${orig[1]} +} diff --git a/tools/testing/selftests/net/forwarding/loopback.sh b/tools/testing/selftests/net/forwarding/loopback.sh new file mode 100755 index 0000000000000000000000000000000000000000..6e4626ae71b0caf2d8ec6d3c3bfb02f5892fa69b --- /dev/null +++ b/tools/testing/selftests/net/forwarding/loopback.sh @@ -0,0 +1,94 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +ALL_TESTS="loopback_test" +NUM_NETIFS=2 +source tc_common.sh +source lib.sh + +h1_create() +{ + simple_if_init $h1 192.0.2.1/24 + tc qdisc add dev $h1 clsact +} + +h1_destroy() +{ + tc qdisc del dev $h1 clsact + simple_if_fini $h1 192.0.2.1/24 +} + +h2_create() +{ + simple_if_init $h2 +} + +h2_destroy() +{ + simple_if_fini $h2 +} + +loopback_test() +{ + RET=0 + + tc filter add dev $h1 ingress protocol arp pref 1 handle 101 flower \ + skip_hw arp_op reply arp_tip 192.0.2.1 action drop + + $MZ $h1 -c 1 -t arp -q + + tc_check_packets "dev $h1 ingress" 101 1 + check_fail $? "Matched on a filter without loopback setup" + + ethtool -K $h1 loopback on + check_err $? "Failed to enable loopback" + + setup_wait_dev $h1 + + $MZ $h1 -c 1 -t arp -q + + tc_check_packets "dev $h1 ingress" 101 1 + check_err $? "Did not match on filter with loopback" + + ethtool -K $h1 loopback off + check_err $? "Failed to disable loopback" + + $MZ $h1 -c 1 -t arp -q + + tc_check_packets "dev $h1 ingress" 101 2 + check_fail $? "Matched on a filter after loopback was removed" + + tc filter del dev $h1 ingress protocol arp pref 1 handle 101 flower + + log_test "loopback" +} + +setup_prepare() +{ + h1=${NETIFS[p1]} + h2=${NETIFS[p2]} + + vrf_prepare + + h1_create + h2_create +} + +cleanup() +{ + pre_cleanup + + h2_destroy + h1_destroy + + vrf_cleanup +} + +trap cleanup EXIT + +setup_prepare +setup_wait + +tests_run + +exit $EXIT_STATUS diff --git a/tools/testing/selftests/net/forwarding/router_multicast.sh b/tools/testing/selftests/net/forwarding/router_multicast.sh index 109e6d7851690e3e33cf831e1810734448c6aba5..57e90c873a2cf4e6e326ed57b6410414a9dcb597 100755 --- a/tools/testing/selftests/net/forwarding/router_multicast.sh +++ b/tools/testing/selftests/net/forwarding/router_multicast.sh @@ -28,7 +28,7 @@ # +------------------+ +------------------+ # -ALL_TESTS="mcast_v4 mcast_v6" +ALL_TESTS="mcast_v4 mcast_v6 rpf_v4 rpf_v6" NUM_NETIFS=6 source lib.sh source tc_common.sh @@ -46,10 +46,14 @@ h1_create() ip route add 2001:db8:2::/64 vrf v$h1 nexthop via 2001:db8:1::1 ip route add 2001:db8:3::/64 vrf v$h1 nexthop via 2001:db8:1::1 + + tc qdisc add dev $h1 ingress } h1_destroy() { + tc qdisc del dev $h1 ingress + ip route del 2001:db8:3::/64 vrf v$h1 ip route del 2001:db8:2::/64 vrf v$h1 @@ -124,10 +128,14 @@ router_create() ip address add 2001:db8:1::1/64 dev $rp1 ip address add 2001:db8:2::1/64 dev $rp2 ip address add 2001:db8:3::1/64 dev $rp3 + + tc qdisc add dev $rp3 ingress } router_destroy() { + tc qdisc del dev $rp3 ingress + ip address del 2001:db8:3::1/64 dev $rp3 ip address del 2001:db8:2::1/64 dev $rp2 ip address del 2001:db8:1::1/64 dev $rp1 @@ -301,6 +309,103 @@ mcast_v6() log_test "mcast IPv6" } +rpf_v4() +{ + # Add a multicast route from first router port to the other two. Send + # matching packets and test that both hosts receive them. Then, send + # the same packets via the third router port and test that they do not + # reach any host due to RPF check. A filter with 'skip_hw' is added to + # test that devices capable of multicast routing offload trap those + # packets. The filter is essentialy a NOP in other scenarios. + + RET=0 + + tc filter add dev $h1 ingress protocol ip pref 1 handle 1 flower \ + dst_ip 225.1.2.3 ip_proto udp dst_port 12345 action drop + tc filter add dev $h2 ingress protocol ip pref 1 handle 1 flower \ + dst_ip 225.1.2.3 ip_proto udp dst_port 12345 action drop + tc filter add dev $h3 ingress protocol ip pref 1 handle 1 flower \ + dst_ip 225.1.2.3 ip_proto udp dst_port 12345 action drop + tc filter add dev $rp3 ingress protocol ip pref 1 handle 1 flower \ + skip_hw dst_ip 225.1.2.3 ip_proto udp dst_port 12345 action pass + + create_mcast_sg $rp1 198.51.100.2 225.1.2.3 $rp2 $rp3 + + $MZ $h1 -c 5 -p 128 -t udp "ttl=10,sp=54321,dp=12345" \ + -a 00:11:22:33:44:55 -b 01:00:5e:01:02:03 \ + -A 198.51.100.2 -B 225.1.2.3 -q + + tc_check_packets "dev $h2 ingress" 1 5 + check_err $? "Multicast not received on first host" + tc_check_packets "dev $h3 ingress" 1 5 + check_err $? "Multicast not received on second host" + + $MZ $h3 -c 5 -p 128 -t udp "ttl=10,sp=54321,dp=12345" \ + -a 00:11:22:33:44:55 -b 01:00:5e:01:02:03 \ + -A 198.51.100.2 -B 225.1.2.3 -q + + tc_check_packets "dev $h1 ingress" 1 0 + check_err $? "Multicast received on first host when should not" + tc_check_packets "dev $h2 ingress" 1 5 + check_err $? "Multicast received on second host when should not" + tc_check_packets "dev $rp3 ingress" 1 5 + check_err $? "Packets not trapped due to RPF check" + + delete_mcast_sg $rp1 198.51.100.2 225.1.2.3 $rp2 $rp3 + + tc filter del dev $rp3 ingress protocol ip pref 1 handle 1 flower + tc filter del dev $h3 ingress protocol ip pref 1 handle 1 flower + tc filter del dev $h2 ingress protocol ip pref 1 handle 1 flower + tc filter del dev $h1 ingress protocol ip pref 1 handle 1 flower + + log_test "RPF IPv4" +} + +rpf_v6() +{ + RET=0 + + tc filter add dev $h1 ingress protocol ipv6 pref 1 handle 1 flower \ + dst_ip ff0e::3 ip_proto udp dst_port 12345 action drop + tc filter add dev $h2 ingress protocol ipv6 pref 1 handle 1 flower \ + dst_ip ff0e::3 ip_proto udp dst_port 12345 action drop + tc filter add dev $h3 ingress protocol ipv6 pref 1 handle 1 flower \ + dst_ip ff0e::3 ip_proto udp dst_port 12345 action drop + tc filter add dev $rp3 ingress protocol ipv6 pref 1 handle 1 flower \ + skip_hw dst_ip ff0e::3 ip_proto udp dst_port 12345 action pass + + create_mcast_sg $rp1 2001:db8:1::2 ff0e::3 $rp2 $rp3 + + $MZ $h1 -6 -c 5 -p 128 -t udp "ttl=10,sp=54321,dp=12345" \ + -a 00:11:22:33:44:55 -b 33:33:00:00:00:03 \ + -A 2001:db8:1::2 -B ff0e::3 -q + + tc_check_packets "dev $h2 ingress" 1 5 + check_err $? "Multicast not received on first host" + tc_check_packets "dev $h3 ingress" 1 5 + check_err $? "Multicast not received on second host" + + $MZ $h3 -6 -c 5 -p 128 -t udp "ttl=10,sp=54321,dp=12345" \ + -a 00:11:22:33:44:55 -b 33:33:00:00:00:03 \ + -A 2001:db8:1::2 -B ff0e::3 -q + + tc_check_packets "dev $h1 ingress" 1 0 + check_err $? "Multicast received on first host when should not" + tc_check_packets "dev $h2 ingress" 1 5 + check_err $? "Multicast received on second host when should not" + tc_check_packets "dev $rp3 ingress" 1 5 + check_err $? "Packets not trapped due to RPF check" + + delete_mcast_sg $rp1 2001:db8:1::2 ff0e::3 $rp2 $rp3 + + tc filter del dev $rp3 ingress protocol ipv6 pref 1 handle 1 flower + tc filter del dev $h3 ingress protocol ipv6 pref 1 handle 1 flower + tc filter del dev $h2 ingress protocol ipv6 pref 1 handle 1 flower + tc filter del dev $h1 ingress protocol ipv6 pref 1 handle 1 flower + + log_test "RPF IPv6" +} + trap cleanup EXIT setup_prepare diff --git a/tools/testing/selftests/net/forwarding/tc_flower.sh b/tools/testing/selftests/net/forwarding/tc_flower.sh index 20d1077e5a3de9cb8ad9186d866837421b84d518..29bcfa84aec710702cc82ceb1e64c6ee098a8803 100755 --- a/tools/testing/selftests/net/forwarding/tc_flower.sh +++ b/tools/testing/selftests/net/forwarding/tc_flower.sh @@ -2,7 +2,7 @@ # SPDX-License-Identifier: GPL-2.0 ALL_TESTS="match_dst_mac_test match_src_mac_test match_dst_ip_test \ - match_src_ip_test match_ip_flags_test" + match_src_ip_test match_ip_flags_test match_pcp_test match_vlan_test" NUM_NETIFS=2 source tc_common.sh source lib.sh @@ -219,6 +219,63 @@ match_ip_flags_test() log_test "ip_flags match ($tcflags)" } +match_pcp_test() +{ + RET=0 + + vlan_create $h2 85 v$h2 192.0.2.11/24 + + tc filter add dev $h2 ingress protocol 802.1q pref 1 handle 101 \ + flower vlan_prio 6 $tcflags dst_mac $h2mac action drop + tc filter add dev $h2 ingress protocol 802.1q pref 2 handle 102 \ + flower vlan_prio 7 $tcflags dst_mac $h2mac action drop + + $MZ $h1 -c 1 -p 64 -a $h1mac -b $h2mac -B 192.0.2.11 -Q 7:85 -t ip -q + $MZ $h1 -c 1 -p 64 -a $h1mac -b $h2mac -B 192.0.2.11 -Q 0:85 -t ip -q + + tc_check_packets "dev $h2 ingress" 101 0 + check_err $? "Matched on specified PCP when should not" + + tc_check_packets "dev $h2 ingress" 102 1 + check_err $? "Did not match on specified PCP" + + tc filter del dev $h2 ingress protocol 802.1q pref 2 handle 102 flower + tc filter del dev $h2 ingress protocol 802.1q pref 1 handle 101 flower + + vlan_destroy $h2 85 + + log_test "PCP match ($tcflags)" +} + +match_vlan_test() +{ + RET=0 + + vlan_create $h2 85 v$h2 192.0.2.11/24 + vlan_create $h2 75 v$h2 192.0.2.10/24 + + tc filter add dev $h2 ingress protocol 802.1q pref 1 handle 101 \ + flower vlan_id 75 $tcflags action drop + tc filter add dev $h2 ingress protocol 802.1q pref 2 handle 102 \ + flower vlan_id 85 $tcflags action drop + + $MZ $h1 -c 1 -p 64 -a $h1mac -b $h2mac -B 192.0.2.11 -Q 0:85 -t ip -q + + tc_check_packets "dev $h2 ingress" 101 0 + check_err $? "Matched on specified VLAN when should not" + + tc_check_packets "dev $h2 ingress" 102 1 + check_err $? "Did not match on specified VLAN" + + tc filter del dev $h2 ingress protocol 802.1q pref 2 handle 102 flower + tc filter del dev $h2 ingress protocol 802.1q pref 1 handle 101 flower + + vlan_destroy $h2 75 + vlan_destroy $h2 85 + + log_test "VLAN match ($tcflags)" +} + setup_prepare() { h1=${NETIFS[p1]} diff --git a/tools/testing/selftests/net/forwarding/tc_vlan_modify.sh b/tools/testing/selftests/net/forwarding/tc_vlan_modify.sh new file mode 100755 index 0000000000000000000000000000000000000000..45378905cb972bda2097d5749b3b7caaeb786f0d --- /dev/null +++ b/tools/testing/selftests/net/forwarding/tc_vlan_modify.sh @@ -0,0 +1,164 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +ALL_TESTS=" + vlan_modify_ingress + vlan_modify_egress +" + +NUM_NETIFS=4 +CHECK_TC="yes" +source lib.sh + +h1_create() +{ + simple_if_init $h1 192.0.2.1/28 2001:db8:1::1/64 + vlan_create $h1 85 v$h1 192.0.2.17/28 2001:db8:2::1/64 +} + +h1_destroy() +{ + vlan_destroy $h1 85 + simple_if_fini $h1 192.0.2.1/28 2001:db8:1::1/64 +} + +h2_create() +{ + simple_if_init $h2 192.0.2.2/28 2001:db8:1::2/64 + vlan_create $h2 65 v$h2 192.0.2.18/28 2001:db8:2::2/64 +} + +h2_destroy() +{ + vlan_destroy $h2 65 + simple_if_fini $h2 192.0.2.2/28 2001:db8:1::2/64 +} + +switch_create() +{ + ip link add dev br0 type bridge vlan_filtering 1 mcast_snooping 0 + + ip link set dev $swp1 master br0 + ip link set dev $swp2 master br0 + + ip link set dev br0 up + ip link set dev $swp1 up + ip link set dev $swp2 up + + bridge vlan add dev $swp1 vid 85 + bridge vlan add dev $swp2 vid 65 + + bridge vlan add dev $swp2 vid 85 + bridge vlan add dev $swp1 vid 65 + + tc qdisc add dev $swp1 clsact + tc qdisc add dev $swp2 clsact +} + +switch_destroy() +{ + tc qdisc del dev $swp2 clsact + tc qdisc del dev $swp1 clsact + + bridge vlan del vid 65 dev $swp1 + bridge vlan del vid 85 dev $swp2 + + bridge vlan del vid 65 dev $swp2 + bridge vlan del vid 85 dev $swp1 + + ip link set dev $swp2 down + ip link set dev $swp1 down + + ip link del dev br0 +} + +setup_prepare() +{ + h1=${NETIFS[p1]} + swp1=${NETIFS[p2]} + + swp2=${NETIFS[p3]} + h2=${NETIFS[p4]} + + vrf_prepare + + h1_create + h2_create + + switch_create +} + +cleanup() +{ + pre_cleanup + + switch_destroy + + h2_destroy + h1_destroy + + vrf_cleanup +} + +vlan_modify_ingress() +{ + RET=0 + + ping_do $h1.85 192.0.2.18 + check_fail $? "ping between two different vlans passed when should not" + + ping6_do $h1.85 2001:db8:2::2 + check_fail $? "ping6 between two different vlans passed when should not" + + tc filter add dev $swp1 ingress protocol all pref 1 handle 1 \ + flower action vlan modify id 65 + tc filter add dev $swp2 ingress protocol all pref 1 handle 1 \ + flower action vlan modify id 85 + + ping_do $h1.85 192.0.2.18 + check_err $? "ping between two different vlans failed when should not" + + ping6_do $h1.85 2001:db8:2::2 + check_err $? "ping6 between two different vlans failed when should not" + + log_test "VLAN modify at ingress" + + tc filter del dev $swp2 ingress protocol all pref 1 handle 1 flower + tc filter del dev $swp1 ingress protocol all pref 1 handle 1 flower +} + +vlan_modify_egress() +{ + RET=0 + + ping_do $h1.85 192.0.2.18 + check_fail $? "ping between two different vlans passed when should not" + + ping6_do $h1.85 2001:db8:2::2 + check_fail $? "ping6 between two different vlans passed when should not" + + tc filter add dev $swp1 egress protocol all pref 1 handle 1 \ + flower action vlan modify id 85 + tc filter add dev $swp2 egress protocol all pref 1 handle 1 \ + flower action vlan modify id 65 + + ping_do $h1.85 192.0.2.18 + check_err $? "ping between two different vlans failed when should not" + + ping6_do $h1.85 2001:db8:2::2 + check_err $? "ping6 between two different vlans failed when should not" + + log_test "VLAN modify at egress" + + tc filter del dev $swp2 egress protocol all pref 1 handle 1 flower + tc filter del dev $swp1 egress protocol all pref 1 handle 1 flower +} + +trap cleanup EXIT + +setup_prepare +setup_wait + +tests_run + +exit $EXIT_STATUS diff --git a/tools/testing/selftests/net/pmtu.sh b/tools/testing/selftests/net/pmtu.sh index 912b2dc50be3d8ee88f51a41e8c440524ecddc4a..524b15dabb3cc257c08a7bacf5150e22f2c50640 100755 --- a/tools/testing/selftests/net/pmtu.sh +++ b/tools/testing/selftests/net/pmtu.sh @@ -116,6 +116,10 @@ # Kselftest framework requirement - SKIP code is 4. ksft_skip=4 +PAUSE_ON_FAIL=no +VERBOSE=0 +TRACING=0 + # Some systems don't have a ping6 binary anymore which ping6 > /dev/null 2>&1 && ping6=$(which ping6) || ping6=$(which ping) @@ -222,6 +226,23 @@ err_flush() { err_buf= } +run_cmd() { + cmd="$*" + + if [ "$VERBOSE" = "1" ]; then + printf " COMMAND: $cmd\n" + fi + + out="$($cmd 2>&1)" + rc=$? + if [ "$VERBOSE" = "1" -a -n "$out" ]; then + echo " $out" + echo + fi + + return $rc +} + # Find the auto-generated name for this namespace nsname() { eval echo \$NS_$1 @@ -258,22 +279,22 @@ setup_fou_or_gue() { fi fi - ${ns_a} ip fou add port 5555 ipproto ${ipproto} || return 2 - ${ns_a} ip link add ${encap}_a type ${type} ${mode} local ${a_addr} remote ${b_addr} encap ${encap} encap-sport auto encap-dport 5556 || return 2 + run_cmd ${ns_a} ip fou add port 5555 ipproto ${ipproto} || return 2 + run_cmd ${ns_a} ip link add ${encap}_a type ${type} ${mode} local ${a_addr} remote ${b_addr} encap ${encap} encap-sport auto encap-dport 5556 || return 2 - ${ns_b} ip fou add port 5556 ipproto ${ipproto} - ${ns_b} ip link add ${encap}_b type ${type} ${mode} local ${b_addr} remote ${a_addr} encap ${encap} encap-sport auto encap-dport 5555 + run_cmd ${ns_b} ip fou add port 5556 ipproto ${ipproto} + run_cmd ${ns_b} ip link add ${encap}_b type ${type} ${mode} local ${b_addr} remote ${a_addr} encap ${encap} encap-sport auto encap-dport 5555 if [ "${inner}" = "4" ]; then - ${ns_a} ip addr add ${tunnel4_a_addr}/${tunnel4_mask} dev ${encap}_a - ${ns_b} ip addr add ${tunnel4_b_addr}/${tunnel4_mask} dev ${encap}_b + run_cmd ${ns_a} ip addr add ${tunnel4_a_addr}/${tunnel4_mask} dev ${encap}_a + run_cmd ${ns_b} ip addr add ${tunnel4_b_addr}/${tunnel4_mask} dev ${encap}_b else - ${ns_a} ip addr add ${tunnel6_a_addr}/${tunnel6_mask} dev ${encap}_a - ${ns_b} ip addr add ${tunnel6_b_addr}/${tunnel6_mask} dev ${encap}_b + run_cmd ${ns_a} ip addr add ${tunnel6_a_addr}/${tunnel6_mask} dev ${encap}_a + run_cmd ${ns_b} ip addr add ${tunnel6_b_addr}/${tunnel6_mask} dev ${encap}_b fi - ${ns_a} ip link set ${encap}_a up - ${ns_b} ip link set ${encap}_b up + run_cmd ${ns_a} ip link set ${encap}_a up + run_cmd ${ns_b} ip link set ${encap}_b up } setup_fou44() { @@ -319,17 +340,17 @@ setup_namespaces() { } setup_veth() { - ${ns_a} ip link add veth_a type veth peer name veth_b || return 1 - ${ns_a} ip link set veth_b netns ${NS_B} + run_cmd ${ns_a} ip link add veth_a type veth peer name veth_b || return 1 + run_cmd ${ns_a} ip link set veth_b netns ${NS_B} - ${ns_a} ip addr add ${veth4_a_addr}/${veth4_mask} dev veth_a - ${ns_b} ip addr add ${veth4_b_addr}/${veth4_mask} dev veth_b + run_cmd ${ns_a} ip addr add ${veth4_a_addr}/${veth4_mask} dev veth_a + run_cmd ${ns_b} ip addr add ${veth4_b_addr}/${veth4_mask} dev veth_b - ${ns_a} ip addr add ${veth6_a_addr}/${veth6_mask} dev veth_a - ${ns_b} ip addr add ${veth6_b_addr}/${veth6_mask} dev veth_b + run_cmd ${ns_a} ip addr add ${veth6_a_addr}/${veth6_mask} dev veth_a + run_cmd ${ns_b} ip addr add ${veth6_b_addr}/${veth6_mask} dev veth_b - ${ns_a} ip link set veth_a up - ${ns_b} ip link set veth_b up + run_cmd ${ns_a} ip link set veth_a up + run_cmd ${ns_b} ip link set veth_b up } setup_vti() { @@ -342,14 +363,14 @@ setup_vti() { [ ${proto} -eq 6 ] && vti_type="vti6" || vti_type="vti" - ${ns_a} ip link add vti${proto}_a type ${vti_type} local ${veth_a_addr} remote ${veth_b_addr} key 10 || return 1 - ${ns_b} ip link add vti${proto}_b type ${vti_type} local ${veth_b_addr} remote ${veth_a_addr} key 10 + run_cmd ${ns_a} ip link add vti${proto}_a type ${vti_type} local ${veth_a_addr} remote ${veth_b_addr} key 10 || return 1 + run_cmd ${ns_b} ip link add vti${proto}_b type ${vti_type} local ${veth_b_addr} remote ${veth_a_addr} key 10 - ${ns_a} ip addr add ${vti_a_addr}/${vti_mask} dev vti${proto}_a - ${ns_b} ip addr add ${vti_b_addr}/${vti_mask} dev vti${proto}_b + run_cmd ${ns_a} ip addr add ${vti_a_addr}/${vti_mask} dev vti${proto}_a + run_cmd ${ns_b} ip addr add ${vti_b_addr}/${vti_mask} dev vti${proto}_b - ${ns_a} ip link set vti${proto}_a up - ${ns_b} ip link set vti${proto}_b up + run_cmd ${ns_a} ip link set vti${proto}_a up + run_cmd ${ns_b} ip link set vti${proto}_b up } setup_vti4() { @@ -375,17 +396,17 @@ setup_vxlan_or_geneve() { opts_b="" fi - ${ns_a} ip link add ${type}_a type ${type} id 1 ${opts_a} remote ${b_addr} ${opts} || return 1 - ${ns_b} ip link add ${type}_b type ${type} id 1 ${opts_b} remote ${a_addr} ${opts} + run_cmd ${ns_a} ip link add ${type}_a type ${type} id 1 ${opts_a} remote ${b_addr} ${opts} || return 1 + run_cmd ${ns_b} ip link add ${type}_b type ${type} id 1 ${opts_b} remote ${a_addr} ${opts} - ${ns_a} ip addr add ${tunnel4_a_addr}/${tunnel4_mask} dev ${type}_a - ${ns_b} ip addr add ${tunnel4_b_addr}/${tunnel4_mask} dev ${type}_b + run_cmd ${ns_a} ip addr add ${tunnel4_a_addr}/${tunnel4_mask} dev ${type}_a + run_cmd ${ns_b} ip addr add ${tunnel4_b_addr}/${tunnel4_mask} dev ${type}_b - ${ns_a} ip addr add ${tunnel6_a_addr}/${tunnel6_mask} dev ${type}_a - ${ns_b} ip addr add ${tunnel6_b_addr}/${tunnel6_mask} dev ${type}_b + run_cmd ${ns_a} ip addr add ${tunnel6_a_addr}/${tunnel6_mask} dev ${type}_a + run_cmd ${ns_b} ip addr add ${tunnel6_b_addr}/${tunnel6_mask} dev ${type}_b - ${ns_a} ip link set ${type}_a up - ${ns_b} ip link set ${type}_b up + run_cmd ${ns_a} ip link set ${type}_a up + run_cmd ${ns_b} ip link set ${type}_b up } setup_geneve4() { @@ -409,15 +430,15 @@ setup_xfrm() { veth_a_addr="${2}" veth_b_addr="${3}" - ${ns_a} ip -${proto} xfrm state add src ${veth_a_addr} dst ${veth_b_addr} spi 0x1000 proto esp aead "rfc4106(gcm(aes))" 0x0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f 128 mode tunnel || return 1 - ${ns_a} ip -${proto} xfrm state add src ${veth_b_addr} dst ${veth_a_addr} spi 0x1001 proto esp aead "rfc4106(gcm(aes))" 0x0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f 128 mode tunnel - ${ns_a} ip -${proto} xfrm policy add dir out mark 10 tmpl src ${veth_a_addr} dst ${veth_b_addr} proto esp mode tunnel - ${ns_a} ip -${proto} xfrm policy add dir in mark 10 tmpl src ${veth_b_addr} dst ${veth_a_addr} proto esp mode tunnel + run_cmd "${ns_a} ip -${proto} xfrm state add src ${veth_a_addr} dst ${veth_b_addr} spi 0x1000 proto esp aead 'rfc4106(gcm(aes))' 0x0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f 128 mode tunnel" || return 1 + run_cmd "${ns_a} ip -${proto} xfrm state add src ${veth_b_addr} dst ${veth_a_addr} spi 0x1001 proto esp aead 'rfc4106(gcm(aes))' 0x0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f 128 mode tunnel" + run_cmd "${ns_a} ip -${proto} xfrm policy add dir out mark 10 tmpl src ${veth_a_addr} dst ${veth_b_addr} proto esp mode tunnel" + run_cmd "${ns_a} ip -${proto} xfrm policy add dir in mark 10 tmpl src ${veth_b_addr} dst ${veth_a_addr} proto esp mode tunnel" - ${ns_b} ip -${proto} xfrm state add src ${veth_a_addr} dst ${veth_b_addr} spi 0x1000 proto esp aead "rfc4106(gcm(aes))" 0x0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f 128 mode tunnel - ${ns_b} ip -${proto} xfrm state add src ${veth_b_addr} dst ${veth_a_addr} spi 0x1001 proto esp aead "rfc4106(gcm(aes))" 0x0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f 128 mode tunnel - ${ns_b} ip -${proto} xfrm policy add dir out mark 10 tmpl src ${veth_b_addr} dst ${veth_a_addr} proto esp mode tunnel - ${ns_b} ip -${proto} xfrm policy add dir in mark 10 tmpl src ${veth_a_addr} dst ${veth_b_addr} proto esp mode tunnel + run_cmd "${ns_b} ip -${proto} xfrm state add src ${veth_a_addr} dst ${veth_b_addr} spi 0x1000 proto esp aead 'rfc4106(gcm(aes))' 0x0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f 128 mode tunnel" + run_cmd "${ns_b} ip -${proto} xfrm state add src ${veth_b_addr} dst ${veth_a_addr} spi 0x1001 proto esp aead 'rfc4106(gcm(aes))' 0x0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f 128 mode tunnel" + run_cmd "${ns_b} ip -${proto} xfrm policy add dir out mark 10 tmpl src ${veth_b_addr} dst ${veth_a_addr} proto esp mode tunnel" + run_cmd "${ns_b} ip -${proto} xfrm policy add dir in mark 10 tmpl src ${veth_a_addr} dst ${veth_b_addr} proto esp mode tunnel" } setup_xfrm4() { @@ -481,7 +502,7 @@ setup() { } trace() { - [ $tracing -eq 0 ] && return + [ $TRACING -eq 0 ] && return for arg do [ "${ns_cmd}" = "" ] && ns_cmd="${arg}" && continue @@ -597,8 +618,8 @@ test_pmtu_ipvX() { mtu "${ns_b}" veth_B-R2 1500 # Create route exceptions - ${ns_a} ${ping} -q -M want -i 0.1 -w 1 -s 1800 ${dst1} > /dev/null - ${ns_a} ${ping} -q -M want -i 0.1 -w 1 -s 1800 ${dst2} > /dev/null + run_cmd ${ns_a} ${ping} -q -M want -i 0.1 -w 1 -s 1800 ${dst1} + run_cmd ${ns_a} ${ping} -q -M want -i 0.1 -w 1 -s 1800 ${dst2} # Check that exceptions have been created with the correct PMTU pmtu_1="$(route_get_dst_pmtu_from_exception "${ns_a}" ${dst1})" @@ -630,7 +651,7 @@ test_pmtu_ipvX() { # Decrease remote MTU on path via R2, get new exception mtu "${ns_r2}" veth_R2-B 400 mtu "${ns_b}" veth_B-R2 400 - ${ns_a} ${ping} -q -M want -i 0.1 -w 1 -s 1400 ${dst2} > /dev/null + run_cmd ${ns_a} ${ping} -q -M want -i 0.1 -w 1 -s 1400 ${dst2} pmtu_2="$(route_get_dst_pmtu_from_exception "${ns_a}" ${dst2})" check_pmtu_value "lock 552" "${pmtu_2}" "exceeding MTU, with MTU < min_pmtu" || return 1 @@ -647,7 +668,7 @@ test_pmtu_ipvX() { check_pmtu_value "1500" "${pmtu_2}" "increasing local MTU" || return 1 # Get new exception - ${ns_a} ${ping} -q -M want -i 0.1 -w 1 -s 1400 ${dst2} > /dev/null + run_cmd ${ns_a} ${ping} -q -M want -i 0.1 -w 1 -s 1400 ${dst2} pmtu_2="$(route_get_dst_pmtu_from_exception "${ns_a}" ${dst2})" check_pmtu_value "lock 552" "${pmtu_2}" "exceeding MTU, with MTU < min_pmtu" || return 1 } @@ -696,7 +717,7 @@ test_pmtu_ipvX_over_vxlanY_or_geneveY_exception() { mtu "${ns_a}" ${type}_a $((${ll_mtu} + 1000)) mtu "${ns_b}" ${type}_b $((${ll_mtu} + 1000)) - ${ns_a} ${ping} -q -M want -i 0.1 -w 1 -s $((${ll_mtu} + 500)) ${dst} > /dev/null + run_cmd ${ns_a} ${ping} -q -M want -i 0.1 -w 1 -s $((${ll_mtu} + 500)) ${dst} # Check that exception was created pmtu="$(route_get_dst_pmtu_from_exception "${ns_a}" ${dst})" @@ -776,7 +797,7 @@ test_pmtu_ipvX_over_fouY_or_gueY() { mtu "${ns_a}" ${encap}_a $((${ll_mtu} + 1000)) mtu "${ns_b}" ${encap}_b $((${ll_mtu} + 1000)) - ${ns_a} ${ping} -q -M want -i 0.1 -w 1 -s $((${ll_mtu} + 500)) ${dst} > /dev/null + run_cmd ${ns_a} ${ping} -q -M want -i 0.1 -w 1 -s $((${ll_mtu} + 500)) ${dst} # Check that exception was created pmtu="$(route_get_dst_pmtu_from_exception "${ns_a}" ${dst})" @@ -834,13 +855,13 @@ test_pmtu_vti4_exception() { # Send DF packet without exceeding link layer MTU, check that no # exception is created - ${ns_a} ping -q -M want -i 0.1 -w 1 -s ${ping_payload} ${tunnel4_b_addr} > /dev/null + run_cmd ${ns_a} ping -q -M want -i 0.1 -w 1 -s ${ping_payload} ${tunnel4_b_addr} pmtu="$(route_get_dst_pmtu_from_exception "${ns_a}" ${tunnel4_b_addr})" check_pmtu_value "" "${pmtu}" "sending packet smaller than PMTU (IP payload length ${esp_payload_rfc4106})" || return 1 # Now exceed link layer MTU by one byte, check that exception is created # with the right PMTU value - ${ns_a} ping -q -M want -i 0.1 -w 1 -s $((ping_payload + 1)) ${tunnel4_b_addr} > /dev/null + run_cmd ${ns_a} ping -q -M want -i 0.1 -w 1 -s $((ping_payload + 1)) ${tunnel4_b_addr} pmtu="$(route_get_dst_pmtu_from_exception "${ns_a}" ${tunnel4_b_addr})" check_pmtu_value "${esp_payload_rfc4106}" "${pmtu}" "exceeding PMTU (IP payload length $((esp_payload_rfc4106 + 1)))" } @@ -856,7 +877,7 @@ test_pmtu_vti6_exception() { mtu "${ns_b}" veth_b 4000 mtu "${ns_a}" vti6_a 5000 mtu "${ns_b}" vti6_b 5000 - ${ns_a} ${ping6} -q -i 0.1 -w 1 -s 60000 ${tunnel6_b_addr} > /dev/null + run_cmd ${ns_a} ${ping6} -q -i 0.1 -w 1 -s 60000 ${tunnel6_b_addr} # Check that exception was created pmtu="$(route_get_dst_pmtu_from_exception "${ns_a}" ${tunnel6_b_addr})" @@ -902,9 +923,9 @@ test_pmtu_vti6_default_mtu() { test_pmtu_vti4_link_add_mtu() { setup namespaces || return 2 - ${ns_a} ip link add vti4_a type vti local ${veth4_a_addr} remote ${veth4_b_addr} key 10 + run_cmd ${ns_a} ip link add vti4_a type vti local ${veth4_a_addr} remote ${veth4_b_addr} key 10 [ $? -ne 0 ] && err " vti not supported" && return 2 - ${ns_a} ip link del vti4_a + run_cmd ${ns_a} ip link del vti4_a fail=0 @@ -912,7 +933,7 @@ test_pmtu_vti4_link_add_mtu() { max=$((65535 - 20)) # Check invalid values first for v in $((min - 1)) $((max + 1)); do - ${ns_a} ip link add vti4_a mtu ${v} type vti local ${veth4_a_addr} remote ${veth4_b_addr} key 10 2>/dev/null + run_cmd ${ns_a} ip link add vti4_a mtu ${v} type vti local ${veth4_a_addr} remote ${veth4_b_addr} key 10 # This can fail, or MTU can be adjusted to a proper value [ $? -ne 0 ] && continue mtu="$(link_get_mtu "${ns_a}" vti4_a)" @@ -920,14 +941,14 @@ test_pmtu_vti4_link_add_mtu() { err " vti tunnel created with invalid MTU ${mtu}" fail=1 fi - ${ns_a} ip link del vti4_a + run_cmd ${ns_a} ip link del vti4_a done # Now check valid values for v in ${min} 1300 ${max}; do - ${ns_a} ip link add vti4_a mtu ${v} type vti local ${veth4_a_addr} remote ${veth4_b_addr} key 10 + run_cmd ${ns_a} ip link add vti4_a mtu ${v} type vti local ${veth4_a_addr} remote ${veth4_b_addr} key 10 mtu="$(link_get_mtu "${ns_a}" vti4_a)" - ${ns_a} ip link del vti4_a + run_cmd ${ns_a} ip link del vti4_a if [ "${mtu}" != "${v}" ]; then err " vti MTU ${mtu} doesn't match configured value ${v}" fail=1 @@ -940,9 +961,9 @@ test_pmtu_vti4_link_add_mtu() { test_pmtu_vti6_link_add_mtu() { setup namespaces || return 2 - ${ns_a} ip link add vti6_a type vti6 local ${veth6_a_addr} remote ${veth6_b_addr} key 10 + run_cmd ${ns_a} ip link add vti6_a type vti6 local ${veth6_a_addr} remote ${veth6_b_addr} key 10 [ $? -ne 0 ] && err " vti6 not supported" && return 2 - ${ns_a} ip link del vti6_a + run_cmd ${ns_a} ip link del vti6_a fail=0 @@ -950,7 +971,7 @@ test_pmtu_vti6_link_add_mtu() { max=$((65535 - 40)) # Check invalid values first for v in $((min - 1)) $((max + 1)); do - ${ns_a} ip link add vti6_a mtu ${v} type vti6 local ${veth6_a_addr} remote ${veth6_b_addr} key 10 2>/dev/null + run_cmd ${ns_a} ip link add vti6_a mtu ${v} type vti6 local ${veth6_a_addr} remote ${veth6_b_addr} key 10 # This can fail, or MTU can be adjusted to a proper value [ $? -ne 0 ] && continue mtu="$(link_get_mtu "${ns_a}" vti6_a)" @@ -958,14 +979,14 @@ test_pmtu_vti6_link_add_mtu() { err " vti6 tunnel created with invalid MTU ${v}" fail=1 fi - ${ns_a} ip link del vti6_a + run_cmd ${ns_a} ip link del vti6_a done # Now check valid values for v in 68 1280 1300 $((65535 - 40)); do - ${ns_a} ip link add vti6_a mtu ${v} type vti6 local ${veth6_a_addr} remote ${veth6_b_addr} key 10 + run_cmd ${ns_a} ip link add vti6_a mtu ${v} type vti6 local ${veth6_a_addr} remote ${veth6_b_addr} key 10 mtu="$(link_get_mtu "${ns_a}" vti6_a)" - ${ns_a} ip link del vti6_a + run_cmd ${ns_a} ip link del vti6_a if [ "${mtu}" != "${v}" ]; then err " vti6 MTU ${mtu} doesn't match configured value ${v}" fail=1 @@ -978,19 +999,19 @@ test_pmtu_vti6_link_add_mtu() { test_pmtu_vti6_link_change_mtu() { setup namespaces || return 2 - ${ns_a} ip link add dummy0 mtu 1500 type dummy + run_cmd ${ns_a} ip link add dummy0 mtu 1500 type dummy [ $? -ne 0 ] && err " dummy not supported" && return 2 - ${ns_a} ip link add dummy1 mtu 3000 type dummy - ${ns_a} ip link set dummy0 up - ${ns_a} ip link set dummy1 up + run_cmd ${ns_a} ip link add dummy1 mtu 3000 type dummy + run_cmd ${ns_a} ip link set dummy0 up + run_cmd ${ns_a} ip link set dummy1 up - ${ns_a} ip addr add ${dummy6_0_addr}/${dummy6_mask} dev dummy0 - ${ns_a} ip addr add ${dummy6_1_addr}/${dummy6_mask} dev dummy1 + run_cmd ${ns_a} ip addr add ${dummy6_0_addr}/${dummy6_mask} dev dummy0 + run_cmd ${ns_a} ip addr add ${dummy6_1_addr}/${dummy6_mask} dev dummy1 fail=0 # Create vti6 interface bound to device, passing MTU, check it - ${ns_a} ip link add vti6_a mtu 1300 type vti6 remote ${dummy6_0_addr} local ${dummy6_0_addr} + run_cmd ${ns_a} ip link add vti6_a mtu 1300 type vti6 remote ${dummy6_0_addr} local ${dummy6_0_addr} mtu="$(link_get_mtu "${ns_a}" vti6_a)" if [ ${mtu} -ne 1300 ]; then err " vti6 MTU ${mtu} doesn't match configured value 1300" @@ -999,7 +1020,7 @@ test_pmtu_vti6_link_change_mtu() { # Move to another device with different MTU, without passing MTU, check # MTU is adjusted - ${ns_a} ip link set vti6_a type vti6 remote ${dummy6_1_addr} local ${dummy6_1_addr} + run_cmd ${ns_a} ip link set vti6_a type vti6 remote ${dummy6_1_addr} local ${dummy6_1_addr} mtu="$(link_get_mtu "${ns_a}" vti6_a)" if [ ${mtu} -ne $((3000 - 40)) ]; then err " vti MTU ${mtu} is not dummy MTU 3000 minus IPv6 header length" @@ -1007,7 +1028,7 @@ test_pmtu_vti6_link_change_mtu() { fi # Move it back, passing MTU, check MTU is not overridden - ${ns_a} ip link set vti6_a mtu 1280 type vti6 remote ${dummy6_0_addr} local ${dummy6_0_addr} + run_cmd ${ns_a} ip link set vti6_a mtu 1280 type vti6 remote ${dummy6_0_addr} local ${dummy6_0_addr} mtu="$(link_get_mtu "${ns_a}" vti6_a)" if [ ${mtu} -ne 1280 ]; then err " vti6 MTU ${mtu} doesn't match configured value 1280" @@ -1052,7 +1073,7 @@ test_cleanup_vxlanX_exception() { # Fill exception cache for multiple CPUs (2) # we can always use inner IPv4 for that for cpu in ${cpu_list}; do - taskset --cpu-list ${cpu} ${ns_a} ping -q -M want -i 0.1 -w 1 -s $((${ll_mtu} + 500)) ${tunnel4_b_addr} > /dev/null + run_cmd taskset --cpu-list ${cpu} ${ns_a} ping -q -M want -i 0.1 -w 1 -s $((${ll_mtu} + 500)) ${tunnel4_b_addr} done ${ns_a} ip link del dev veth_A-R1 & @@ -1084,29 +1105,33 @@ usage() { exit 1 } +################################################################################ +# exitcode=0 desc=0 + +while getopts :ptv o +do + case $o in + p) PAUSE_ON_FAIL=yes;; + v) VERBOSE=1;; + t) if which tcpdump > /dev/null 2>&1; then + TRACING=1 + else + echo "=== tcpdump not available, tracing disabled" + fi + ;; + *) usage;; + esac +done +shift $(($OPTIND-1)) + IFS=" " -tracing=0 for arg do - if [ "${arg}" != "${arg#--*}" ]; then - opt="${arg#--}" - if [ "${opt}" = "trace" ]; then - if which tcpdump > /dev/null 2>&1; then - tracing=1 - else - echo "=== tcpdump not available, tracing disabled" - fi - else - usage - fi - else - # Check first that all requested tests are available before - # running any - command -v > /dev/null "test_${arg}" || { echo "=== Test ${arg} not found"; usage; } - fi + # Check first that all requested tests are available before running any + command -v > /dev/null "test_${arg}" || { echo "=== Test ${arg} not found"; usage; } done trap cleanup EXIT @@ -1124,6 +1149,11 @@ for t in ${tests}; do ( unset IFS + + if [ "$VERBOSE" = "1" ]; then + printf "\n##########################################################################\n\n" + fi + eval test_${name} ret=$? cleanup @@ -1132,6 +1162,11 @@ for t in ${tests}; do printf "TEST: %-60s [ OK ]\n" "${t}" elif [ $ret -eq 1 ]; then printf "TEST: %-60s [FAIL]\n" "${t}" + if [ "${PAUSE_ON_FAIL}" = "yes" ]; then + echo + echo "Pausing. Hit enter to continue" + read a + fi err_flush exit 1 elif [ $ret -eq 2 ]; then diff --git a/tools/testing/selftests/net/rtnetlink.sh b/tools/testing/selftests/net/rtnetlink.sh index b447803f3f8ae067ea4da2782a268ff72dace936..b25c9fe019d28211abcd282384ec9cda0945436d 100755 --- a/tools/testing/selftests/net/rtnetlink.sh +++ b/tools/testing/selftests/net/rtnetlink.sh @@ -696,9 +696,9 @@ kci_test_ipsec_offload() algo="aead rfc4106(gcm(aes)) 0x3132333435363738393031323334353664636261 128" srcip=192.168.123.3 dstip=192.168.123.4 - dev=simx1 - sysfsd=/sys/kernel/debug/netdevsim/$dev + sysfsd=/sys/kernel/debug/netdevsim/netdevsim0/ports/0/ sysfsf=$sysfsd/ipsec + sysfsnet=/sys/bus/netdevsim/devices/netdevsim0/net/ # setup netdevsim since dummydev doesn't have offload support modprobe netdevsim @@ -708,7 +708,11 @@ kci_test_ipsec_offload() return 1 fi - ip link add $dev type netdevsim + echo "0" > /sys/bus/netdevsim/new_device + while [ ! -d $sysfsnet ] ; do :; done + udevadm settle + dev=`ls $sysfsnet` + ip addr add $srcip dev $dev ip link set $dev up if [ ! -d $sysfsd ] ; then @@ -781,7 +785,6 @@ EOF fi # clean up any leftovers - ip link del $dev rmmod netdevsim if [ $ret -ne 0 ]; then diff --git a/tools/testing/selftests/netfilter/Makefile b/tools/testing/selftests/netfilter/Makefile index a37cb1192c6a6bc6080c829b63768e6ba52f8dd1..3e6d1bcc28949aebae0c7504357ec3eb2244822e 100644 --- a/tools/testing/selftests/netfilter/Makefile +++ b/tools/testing/selftests/netfilter/Makefile @@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 # Makefile for netfilter selftests -TEST_PROGS := nft_trans_stress.sh nft_nat.sh conntrack_icmp_related.sh +TEST_PROGS := nft_trans_stress.sh nft_nat.sh bridge_brouter.sh \ + conntrack_icmp_related.sh include ../lib.mk diff --git a/tools/testing/selftests/netfilter/bridge_brouter.sh b/tools/testing/selftests/netfilter/bridge_brouter.sh new file mode 100755 index 0000000000000000000000000000000000000000..29f3955b9af7e8b357e1b806e7ca496b1c1bc924 --- /dev/null +++ b/tools/testing/selftests/netfilter/bridge_brouter.sh @@ -0,0 +1,146 @@ +#!/bin/bash +# +# This test is for bridge 'brouting', i.e. make some packets being routed +# rather than getting bridged even though they arrive on interface that is +# part of a bridge. + +# eth0 br0 eth0 +# setup is: ns1 <-> ns0 <-> ns2 + +# Kselftest framework requirement - SKIP code is 4. +ksft_skip=4 +ret=0 + +ebtables -V > /dev/null 2>&1 +if [ $? -ne 0 ];then + echo "SKIP: Could not run test without ebtables" + exit $ksft_skip +fi + +ip -Version > /dev/null 2>&1 +if [ $? -ne 0 ];then + echo "SKIP: Could not run test without ip tool" + exit $ksft_skip +fi + +ip netns add ns0 +ip netns add ns1 +ip netns add ns2 + +ip link add veth0 netns ns0 type veth peer name eth0 netns ns1 +if [ $? -ne 0 ]; then + echo "SKIP: Can't create veth device" + exit $ksft_skip +fi +ip link add veth1 netns ns0 type veth peer name eth0 netns ns2 + +ip -net ns0 link set lo up +ip -net ns0 link set veth0 up +ip -net ns0 link set veth1 up + +ip -net ns0 link add br0 type bridge +if [ $? -ne 0 ]; then + echo "SKIP: Can't create bridge br0" + exit $ksft_skip +fi + +ip -net ns0 link set veth0 master br0 +ip -net ns0 link set veth1 master br0 +ip -net ns0 link set br0 up +ip -net ns0 addr add 10.0.0.1/24 dev br0 + +# place both in same subnet, ns1 and ns2 connected via ns0:br0 +for i in 1 2; do + ip -net ns$i link set lo up + ip -net ns$i link set eth0 up + ip -net ns$i addr add 10.0.0.1$i/24 dev eth0 +done + +test_ebtables_broute() +{ + local cipt + + # redirect is needed so the dstmac is rewritten to the bridge itself, + # ip stack won't process OTHERHOST (foreign unicast mac) packets. + ip netns exec ns0 ebtables -t broute -A BROUTING -p ipv4 --ip-protocol icmp -j redirect --redirect-target=DROP + if [ $? -ne 0 ]; then + echo "SKIP: Could not add ebtables broute redirect rule" + return $ksft_skip + fi + + # ping netns1, expected to not work (ip forwarding is off) + ip netns exec ns1 ping -q -c 1 10.0.0.12 > /dev/null 2>&1 + if [ $? -eq 0 ]; then + echo "ERROR: ping works, should have failed" 1>&2 + return 1 + fi + + # enable forwarding on both interfaces. + # neither needs an ip address, but at least the bridge needs + # an ip address in same network segment as ns1 and ns2 (ns0 + # needs to be able to determine route for to-be-forwarded packet). + ip netns exec ns0 sysctl -q net.ipv4.conf.veth0.forwarding=1 + ip netns exec ns0 sysctl -q net.ipv4.conf.veth1.forwarding=1 + + sleep 1 + + ip netns exec ns1 ping -q -c 1 10.0.0.12 > /dev/null + if [ $? -ne 0 ]; then + echo "ERROR: ping did not work, but it should (broute+forward)" 1>&2 + return 1 + fi + + echo "PASS: ns1/ns2 connectivity with active broute rule" + ip netns exec ns0 ebtables -t broute -F + + # ping netns1, expected to work (frames are bridged) + ip netns exec ns1 ping -q -c 1 10.0.0.12 > /dev/null + if [ $? -ne 0 ]; then + echo "ERROR: ping did not work, but it should (bridged)" 1>&2 + return 1 + fi + + ip netns exec ns0 ebtables -t filter -A FORWARD -p ipv4 --ip-protocol icmp -j DROP + + # ping netns1, expected to not work (DROP in bridge forward) + ip netns exec ns1 ping -q -c 1 10.0.0.12 > /dev/null 2>&1 + if [ $? -eq 0 ]; then + echo "ERROR: ping works, should have failed (icmp forward drop)" 1>&2 + return 1 + fi + + # re-activate brouter + ip netns exec ns0 ebtables -t broute -A BROUTING -p ipv4 --ip-protocol icmp -j redirect --redirect-target=DROP + + ip netns exec ns2 ping -q -c 1 10.0.0.11 > /dev/null + if [ $? -ne 0 ]; then + echo "ERROR: ping did not work, but it should (broute+forward 2)" 1>&2 + return 1 + fi + + echo "PASS: ns1/ns2 connectivity with active broute rule and bridge forward drop" + return 0 +} + +# test basic connectivity +ip netns exec ns1 ping -c 1 -q 10.0.0.12 > /dev/null +if [ $? -ne 0 ]; then + echo "ERROR: Could not reach ns2 from ns1" 1>&2 + ret=1 +fi + +ip netns exec ns2 ping -c 1 -q 10.0.0.11 > /dev/null +if [ $? -ne 0 ]; then + echo "ERROR: Could not reach ns1 from ns2" 1>&2 + ret=1 +fi + +if [ $ret -eq 0 ];then + echo "PASS: netns connectivity: ns1 and ns2 can reach each other" +fi + +test_ebtables_broute +ret=$? +for i in 0 1 2; do ip netns del ns$i;done + +exit $ret diff --git a/tools/testing/selftests/netfilter/nft_nat.sh b/tools/testing/selftests/netfilter/nft_nat.sh index 3194007cf8d1bf3f456d9e4594417ffe2f9d56fd..21159f5f33628f0b1d0d58599dc973657e3193e5 100755 --- a/tools/testing/selftests/netfilter/nft_nat.sh +++ b/tools/testing/selftests/netfilter/nft_nat.sh @@ -6,6 +6,7 @@ # Kselftest framework requirement - SKIP code is 4. ksft_skip=4 ret=0 +test_inet_nat=true nft --version > /dev/null 2>&1 if [ $? -ne 0 ];then @@ -141,17 +142,24 @@ reset_counters() test_local_dnat6() { + local family=$1 local lret=0 + local IPF="" + + if [ $family = "inet" ];then + IPF="ip6" + fi + ip netns exec ns0 nft -f - </dev/null +table $family nat { chain output { type nat hook output priority 0; policy accept; - ip daddr 10.0.1.99 dnat to 10.0.2.99 + ip daddr 10.0.1.99 dnat $IPF to 10.0.2.99 } } EOF + if [ $? -ne 0 ]; then + if [ $family = "inet" ];then + echo "SKIP: inet nat tests" + test_inet_nat=false + return $ksft_skip + fi + echo "SKIP: Could not add add $family dnat hook" + return $ksft_skip + fi + # ping netns1, expect rewrite to netns2 ip netns exec ns0 ping -q -c 1 10.0.1.99 > /dev/null if [ $? -ne 0 ]; then @@ -264,9 +289,9 @@ EOF fi done - test $lret -eq 0 && echo "PASS: ping to ns1 was NATted to ns2" + test $lret -eq 0 && echo "PASS: ping to ns1 was $family NATted to ns2" - ip netns exec ns0 nft flush chain ip nat output + ip netns exec ns0 nft flush chain $family nat output reset_counters ip netns exec ns0 ping -q -c 1 10.0.1.99 > /dev/null @@ -313,7 +338,7 @@ EOF fi done - test $lret -eq 0 && echo "PASS: ping to ns1 OK after nat output chain flush" + test $lret -eq 0 && echo "PASS: ping to ns1 OK after $family nat output chain flush" return $lret } @@ -321,6 +346,7 @@ EOF test_masquerade6() { + local family=$1 local natflags=$1 local lret=0 @@ -352,16 +378,25 @@ test_masquerade6() # add masquerading rule ip netns exec ns0 nft -f - < /dev/null # ping ns2->ns1 if [ $? -ne 0 ] ; then +<<<<<<< HEAD + echo "ERROR: cannot ping ns1 from ns2 with active $family masquerading" +======= echo "ERROR: cannot ping ns1 from ns2 with active ipv6 masquerade $natflags" +>>>>>>> cd8dead0c39457e58ec1d36db93aedca811d48f1 lret=1 fi @@ -398,6 +433,9 @@ EOF fi done +<<<<<<< HEAD + ip netns exec ns0 nft flush chain $family nat postrouting +======= ip netns exec ns2 ping -q -c 1 dead:1::99 > /dev/null # ping ns2->ns1 if [ $? -ne 0 ] ; then echo "ERROR: cannot ping ns1 from ns2 with active ipv6 masquerade $natflags (attempt 2)" @@ -405,19 +443,28 @@ EOF fi ip netns exec ns0 nft flush chain ip6 nat postrouting +>>>>>>> cd8dead0c39457e58ec1d36db93aedca811d48f1 if [ $? -ne 0 ]; then - echo "ERROR: Could not flush ip6 nat postrouting" 1>&2 + echo "ERROR: Could not flush $family nat postrouting" 1>&2 lret=1 fi +<<<<<<< HEAD + test $lret -eq 0 && echo "PASS: $family IPv6 masquerade for ns2" +======= test $lret -eq 0 && echo "PASS: IPv6 masquerade $natflags for ns2" +>>>>>>> cd8dead0c39457e58ec1d36db93aedca811d48f1 return $lret } test_masquerade() { +<<<<<<< HEAD + local family=$1 +======= local natflags=$1 +>>>>>>> cd8dead0c39457e58ec1d36db93aedca811d48f1 local lret=0 ip netns exec ns0 sysctl net.ipv4.conf.veth0.forwarding=1 > /dev/null @@ -448,16 +495,25 @@ test_masquerade() # add masquerading rule ip netns exec ns0 nft -f - < /dev/null # ping ns2->ns1 if [ $? -ne 0 ] ; then +<<<<<<< HEAD + echo "ERROR: cannot ping ns1 from ns2 with active $family masquerading" +======= echo "ERROR: cannot ping ns1 from ns2 with active ip masquere $natflags" +>>>>>>> cd8dead0c39457e58ec1d36db93aedca811d48f1 lret=1 fi @@ -493,6 +549,9 @@ EOF fi done +<<<<<<< HEAD + ip netns exec ns0 nft flush chain $family nat postrouting +======= ip netns exec ns2 ping -q -c 1 10.0.1.99 > /dev/null # ping ns2->ns1 if [ $? -ne 0 ] ; then echo "ERROR: cannot ping ns1 from ns2 with active ip masquerade $natflags (attempt 2)" @@ -500,18 +559,24 @@ EOF fi ip netns exec ns0 nft flush chain ip nat postrouting +>>>>>>> cd8dead0c39457e58ec1d36db93aedca811d48f1 if [ $? -ne 0 ]; then - echo "ERROR: Could not flush nat postrouting" 1>&2 + echo "ERROR: Could not flush $family nat postrouting" 1>&2 lret=1 fi +<<<<<<< HEAD + test $lret -eq 0 && echo "PASS: $family IP masquerade for ns2" +======= test $lret -eq 0 && echo "PASS: IP masquerade $natflags for ns2" +>>>>>>> cd8dead0c39457e58ec1d36db93aedca811d48f1 return $lret } test_redirect6() { + local family=$1 local lret=0 ip netns exec ns0 sysctl net.ipv6.conf.all.forwarding=1 > /dev/null @@ -541,16 +606,21 @@ test_redirect6() # add redirect rule ip netns exec ns0 nft -f - < /dev/null # ping ns2->ns1 if [ $? -ne 0 ] ; then - echo "ERROR: cannot ping ns1 from ns2 with active ip6 redirect" + echo "ERROR: cannot ping ns1 from ns2 via ipv6 with active $family redirect" lret=1 fi @@ -574,19 +644,20 @@ EOF fi done - ip netns exec ns0 nft delete table ip6 nat + ip netns exec ns0 nft delete table $family nat if [ $? -ne 0 ]; then - echo "ERROR: Could not delete ip6 nat table" 1>&2 + echo "ERROR: Could not delete $family nat table" 1>&2 lret=1 fi - test $lret -eq 0 && echo "PASS: IPv6 redirection for ns2" + test $lret -eq 0 && echo "PASS: $family IPv6 redirection for ns2" return $lret } test_redirect() { + local family=$1 local lret=0 ip netns exec ns0 sysctl net.ipv4.conf.veth0.forwarding=1 > /dev/null @@ -617,16 +688,21 @@ test_redirect() # add redirect rule ip netns exec ns0 nft -f - < /dev/null # ping ns2->ns1 if [ $? -ne 0 ] ; then - echo "ERROR: cannot ping ns1 from ns2 with active ip redirect" + echo "ERROR: cannot ping ns1 from ns2 with active $family ip redirect" lret=1 fi @@ -651,13 +727,13 @@ EOF fi done - ip netns exec ns0 nft delete table ip nat + ip netns exec ns0 nft delete table $family nat if [ $? -ne 0 ]; then - echo "ERROR: Could not delete nat table" 1>&2 + echo "ERROR: Could not delete $family nat table" 1>&2 lret=1 fi - test $lret -eq 0 && echo "PASS: IP redirection for ns2" + test $lret -eq 0 && echo "PASS: $family IP redirection for ns2" return $lret } @@ -760,20 +836,34 @@ if [ $ret -eq 0 ];then fi reset_counters -test_local_dnat -test_local_dnat6 +test_local_dnat ip +test_local_dnat6 ip6 +reset_counters +$test_inet_nat && test_local_dnat inet +$test_inet_nat && test_local_dnat6 inet reset_counters +<<<<<<< HEAD +test_masquerade ip +test_masquerade6 ip6 +reset_counters +$test_inet_nat && test_masquerade inet +$test_inet_nat && test_masquerade6 inet +======= test_masquerade "" test_masquerade6 "" reset_counters test_masquerade "fully-random" test_masquerade6 "fully-random" +>>>>>>> cd8dead0c39457e58ec1d36db93aedca811d48f1 reset_counters -test_redirect -test_redirect6 +test_redirect ip +test_redirect6 ip6 +reset_counters +$test_inet_nat && test_redirect inet +$test_inet_nat && test_redirect6 inet for i in 0 1 2; do ip netns del ns$i;done diff --git a/tools/testing/selftests/tc-testing/tc-tests/actions/pedit.json b/tools/testing/selftests/tc-testing/tc-tests/actions/pedit.json index b73ceb9e28b1dad6362473d02896071db5fa169f..0d319f1d01db40ad0d408bccc3bc19fa902eaabc 100644 --- a/tools/testing/selftests/tc-testing/tc-tests/actions/pedit.json +++ b/tools/testing/selftests/tc-testing/tc-tests/actions/pedit.json @@ -47,5 +47,908 @@ "teardown": [ "$TC actions flush action pedit" ] + }, + { + "id": "377e", + "name": "Add pedit action with RAW_OP offset u32", + "category": [ + "actions", + "pedit", + "raw_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit munge offset 12 u32 set 0x90abcdef", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": "12: val 90abcdef mask 00000000", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "a0ca", + "name": "Add pedit action with RAW_OP offset u32 (INVALID)", + "category": [ + "actions", + "pedit", + "raw_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit munge offset 2 u32 set 0x12345678", + "expExitCode": "255", + "verifyCmd": "/bin/true", + "matchPattern": " ", + "matchCount": "0", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "dd8a", + "name": "Add pedit action with RAW_OP offset u16 u16", + "category": [ + "actions", + "pedit", + "raw_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit munge offset 12 u16 set 0x1234 munge offset 14 u16 set 0x5678", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": "val 12340000 mask 0000ffff.*val 00005678 mask ffff0000", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "53db", + "name": "Add pedit action with RAW_OP offset u16 (INVALID)", + "category": [ + "actions", + "pedit", + "raw_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit munge offset 15 u16 set 0x1234", + "expExitCode": "255", + "verifyCmd": "/bin/true", + "matchPattern": " ", + "matchCount": "0", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "5c7e", + "name": "Add pedit action with RAW_OP offset u8 add value", + "category": [ + "actions", + "pedit", + "raw_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit ex munge offset 16 u8 add 0xf", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": " 16: add 0f000000 mask 00ffffff", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "2893", + "name": "Add pedit action with RAW_OP offset u8 quad", + "category": [ + "actions", + "pedit", + "raw_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit munge offset 12 u8 set 0x12 munge offset 13 u8 set 0x34 munge offset 14 u8 set 0x56 munge offset 15 u8 set 0x78", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": "val 12000000 mask 00ffffff.*val 00340000 mask ff00ffff.*val 00005600 mask ffff00ff.*val 00000078 mask ffffff00", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "3a07", + "name": "Add pedit action with RAW_OP offset u8-u16-u8", + "category": [ + "actions", + "pedit", + "raw_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit munge offset 0 u8 set 0x12 munge offset 1 u16 set 0x3456 munge offset 3 u8 set 0x78", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": "val 12000000 mask 00ffffff.*val 00345600 mask ff0000ff.*val 00000078 mask ffffff00", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "ab0f", + "name": "Add pedit action with RAW_OP offset u16-u8-u8", + "category": [ + "actions", + "pedit", + "raw_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit munge offset 0 u16 set 0x1234 munge offset 2 u8 set 0x56 munge offset 3 u8 set 0x78", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": "val 12340000 mask 0000ffff.*val 00005600 mask ffff00ff.*val 00000078 mask ffffff00", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "9d12", + "name": "Add pedit action with RAW_OP offset u32 set u16 clear u8 invert", + "category": [ + "actions", + "pedit", + "raw_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit munge offset 0 u32 set 0x12345678 munge offset 1 u16 clear munge offset 2 u8 invert", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": "val 12345678 mask 00000000.*val 00000000 mask ff0000ff.*val 0000ff00 mask ffffffff", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "ebfa", + "name": "Add pedit action with RAW_OP offset overflow u32 (INVALID)", + "category": [ + "actions", + "pedit", + "raw_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit munge offset 0xffffffffffffffffffffffffffffffffffffffffff u32 set 0x1", + "expExitCode": "255", + "verifyCmd": "/bin/true", + "matchPattern": " ", + "matchCount": "0", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "f512", + "name": "Add pedit action with RAW_OP offset u16 at offmask shift set", + "category": [ + "actions", + "pedit", + "raw_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit munge offset 12 u16 at 12 ffff 1 set 0xaaaa", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": " 12: val aaaa0000 mask 0000ffff", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "c2cb", + "name": "Add pedit action with RAW_OP offset u32 retain value", + "category": [ + "actions", + "pedit", + "raw_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit munge offset 12 u32 set 0x12345678 retain 0xff00", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": " 12: val 00005600 mask ffff00ff", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "86d4", + "name": "Add pedit action with LAYERED_OP eth set src & dst", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit ex munge eth src set 11:22:33:44:55:66 munge eth dst set ff:ee:dd:cc:bb:aa", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": "eth\\+4: val 00001122 mask ffff0000.*eth\\+8: val 33445566 mask 00000000.*eth\\+0: val ffeeddcc mask 00000000.*eth\\+4: val bbaa0000 mask 0000ffff", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "c715", + "name": "Add pedit action with LAYERED_OP eth set src (INVALID)", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit ex munge eth src set %e:11:m2:33:x4:-5", + "expExitCode": "255", + "verifyCmd": "/bin/true", + "matchPattern": " ", + "matchCount": "0", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "ba22", + "name": "Add pedit action with LAYERED_OP eth type set/clear sequence", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit ex munge eth type set 0x1 munge eth type clear munge eth type set 0x1 munge eth type clear munge eth type set 0x1 munge eth type clear munge eth type set 0x1 munge eth type clear munge eth type set 0x1 munge eth type clear munge eth type set 0x1 munge eth type clear", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": "eth\\+12: val 00010000 mask 0000ffff.*eth\\+12: val 00000000 mask 0000ffff.*eth\\+12: val 00010000 mask 0000ffff.*eth\\+12: val 00000000 mask 0000ffff.*eth\\+12: val 00010000 mask 0000ffff.*eth\\+12: val 00000000 mask 0000ffff.*eth\\+12: val 00010000 mask 0000ffff.*eth\\+12: val 00000000 mask 0000ffff.*eth\\+12: val 00010000 mask 0000ffff.*eth\\+12: val 00000000 mask 0000ffff.*eth\\+12: val 00010000 mask 0000ffff.*eth\\+12: val 00000000 mask 0000ffff", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "5810", + "name": "Add pedit action with LAYERED_OP ip set src & dst", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit munge ip src set 18.52.86.120 munge ip dst set 18.52.86.120", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": " 12: val 12345678 mask 00000000.* 16: val 12345678 mask 00000000", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "1092", + "name": "Add pedit action with LAYERED_OP ip set ihl & dsfield", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit munge ip ihl set 0xff munge ip dsfield set 0xff", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": " 0: val 0f000000 mask f0ffffff.* 0: val 00ff0000 mask ff00ffff", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "02d8", + "name": "Add pedit action with LAYERED_OP ip set ttl & protocol", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit munge ip ttl set 0x1 munge ip protocol set 0xff", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": " 8: val 01000000 mask 00ffffff.* 8: val 00ff0000 mask ff00ffff", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "3e2d", + "name": "Add pedit action with LAYERED_OP ip set ttl (INVALID)", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit munge ip ttl set 300", + "expExitCode": "255", + "verifyCmd": "/bin/true", + "matchPattern": " ", + "matchCount": "0", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "31ae", + "name": "Add pedit action with LAYERED_OP ip ttl clear/set", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit munge ip ttl clear munge ip ttl set 0x1", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": " 8: val 00000000 mask 00ffffff.* 8: val 01000000 mask 00ffffff", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "486f", + "name": "Add pedit action with LAYERED_OP ip set duplicate fields", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit munge ip ttl set 0x1 munge ip ttl set 0x1", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": " 8: val 01000000 mask 00ffffff.* 8: val 01000000 mask 00ffffff", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "e790", + "name": "Add pedit action with LAYERED_OP ip set ce, df, mf, firstfrag, nofrag fields", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit munge ip ce set 0xff munge ip df set 0xff munge ip mf set 0xff munge ip firstfrag set 0xff munge ip nofrag set 0xff", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": " 4: val 00008000 mask ffff7fff.* 4: val 00004000 mask ffffbfff.* 4: val 00002000 mask ffffdfff.* 4: val 00001f00 mask ffffe0ff.* 4: val 00003f00 mask ffffc0ff", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "6829", + "name": "Add pedit action with LAYERED_OP beyond ip set dport & sport", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit munge ip dport set 0x1234 munge ip sport set 0x5678", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": " 20: val 00001234 mask ffff0000.* 20: val 56780000 mask 0000ffff", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "afd8", + "name": "Add pedit action with LAYERED_OP beyond ip set icmp_type & icmp_code", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit munge ip icmp_type set 0xff munge ip icmp_code set 0xff", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": " 20: val ff000000 mask 00ffffff.* 20: val ff000000 mask 00ffffff", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "3143", + "name": "Add pedit action with LAYERED_OP beyond ip set dport (INVALID)", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit ex munge ip dport set 0x1234", + "expExitCode": "255", + "verifyCmd": "/bin/true", + "matchPattern": " ", + "matchCount": "0", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "fc1f", + "name": "Add pedit action with LAYERED_OP ip6 set src & dst", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit ex munge ip6 src set 2001:0db8:0:f101::1 munge ip6 dst set 2001:0db8:0:f101::1", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": "ipv6\\+8: val 20010db8 mask 00000000.*ipv6\\+12: val 0000f101 mask 00000000.*ipv6\\+16: val 00000000 mask 00000000.*ipv6\\+20: val 00000001 mask 00000000.*ipv6\\+24: val 20010db8 mask 00000000.*ipv6\\+28: val 0000f101 mask 00000000.*ipv6\\+32: val 00000000 mask 00000000.*ipv6\\+36: val 00000001 mask 00000000", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "6d34", + "name": "Add pedit action with LAYERED_OP ip6 dst retain value (INVALID)", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit ex munge ip6 dst set 2001:0db8:0:f101::1 retain 0xff0000", + "expExitCode": "255", + "verifyCmd": "/bin/true", + "matchPattern": " ", + "matchCount": "0", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "6f5e", + "name": "Add pedit action with LAYERED_OP ip6 flow_lbl", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit ex munge ip6 flow_lbl set 0xfffff", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": "ipv6\\+0: val 0007ffff mask fff80000", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "6795", + "name": "Add pedit action with LAYERED_OP ip6 set payload_len, nexthdr, hoplimit", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit ex munge ip6 payload_len set 0xffff munge ip6 nexthdr set 0xff munge ip6 hoplimit set 0xff", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": "ipv6\\+4: val ffff0000 mask 0000ffff.*ipv6\\+4: val 0000ff00 mask ffff00ff.*ipv6\\+4: val 000000ff mask ffffff00", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "1442", + "name": "Add pedit action with LAYERED_OP tcp set dport & sport", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit ex munge tcp dport set 4789 munge tcp sport set 1", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": "tcp\\+0: val 000012b5 mask ffff0000.*tcp\\+0: val 00010000 mask 0000ffff", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "b7ac", + "name": "Add pedit action with LAYERED_OP tcp sport set (INVALID)", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit ex munge tcp sport set -200", + "expExitCode": "255", + "verifyCmd": "/bin/true", + "matchPattern": " ", + "matchCount": "0", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "cfcc", + "name": "Add pedit action with LAYERED_OP tcp flags set", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit ex munge tcp flags set 0x16", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": "tcp\\+12: val 00160000 mask ff00ffff", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "3bc4", + "name": "Add pedit action with LAYERED_OP tcp set dport, sport & flags fields", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit ex munge tcp dport set 4789 munge tcp sport set 1 munge tcp flags set 0x1", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": "tcp\\+0: val 000012b5 mask ffff0000.*tcp\\+0: val 00010000 mask 0000ffff.*tcp\\+12: val 00010000 mask ff00ffff", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "f1c8", + "name": "Add pedit action with LAYERED_OP udp set dport & sport", + "category": [ + "actions", + "pedit", + "layered_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit ex munge udp dport set 4789 munge udp sport set 4789", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": "udp\\+0: val 000012b5 mask ffff0000.*udp\\+0: val 12b50000 mask 0000ffff", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "d784", + "name": "Add pedit action with mixed RAW/LAYERED_OP #1", + "category": [ + "actions", + "pedit", + "layered_op", + "raw_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit ex munge eth src set 11:22:33:44:55:66 munge ip ttl set 0xff munge tcp flags clear munge offset 15 u8 add 40 retain 0xf0 munge udp dport add 1", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": "eth\\+4: val 00001122 mask ffff0000.*eth\\+8: val 33445566 mask 00000000.*ipv4\\+8: val ff000000 mask 00ffffff.*tcp\\+12: val 00000000 mask ff00ffff.* 12: add 00000020 mask ffffff0f.*udp\\+0: add 00000001 mask ffff0000", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] + }, + { + "id": "70ca", + "name": "Add pedit action with mixed RAW/LAYERED_OP #2", + "category": [ + "actions", + "pedit", + "layered_op", + "raw_op" + ], + "setup": [ + [ + "$TC actions flush action pedit", + 0, + 1, + 255 + ] + ], + "cmdUnderTest": "$TC actions add action pedit ex munge eth src set 11:22:33:44:55:66 munge eth dst set ff:ee:dd:cc:bb:aa munge ip6 payload_len set 0xffff munge ip6 nexthdr set 0xff munge ip6 hoplimit preserve munge offset 0 u8 set 0x12 munge offset 1 u16 set 0x3456 munge offset 3 u8 set 0x78 munge ip ttl set 0xaa munge ip protocol set 0xff", + "expExitCode": "0", + "verifyCmd": "$TC actions list action pedit | grep 'key '", + "matchPattern": "eth\\+4: val 00001122 mask ffff0000.*eth\\+8: val 33445566 mask 00000000.*eth\\+0: val ffeeddcc mask 00000000.*eth\\+4: val bbaa0000 mask 0000ffff.*ipv6\\+4: val ffff0000 mask 0000ffff.*ipv6\\+4: val 0000ff00 mask ffff00ff.*ipv6\\+4: val 00000000 mask ffffffff.* 0: val 12000000 mask 00ffffff.* 0: val 00345600 mask ff0000ff.* 0: val 00000078 mask ffffff00.*ipv4\\+8: val aa000000 mask 00ffffff.*ipv4\\+8: val 00ff0000 mask ff00ffff", + "matchCount": "1", + "teardown": [ + "$TC actions flush action pedit" + ] } + ] diff --git a/tools/testing/selftests/tc-testing/tc-tests/filters/tests.json b/tools/testing/selftests/tc-testing/tc-tests/filters/tests.json index 2d096b2abf2c943064ae86e674d2a7a65f49757d..e2f92cefb8d5ef06237e87ffc88cad19667afbac 100644 --- a/tools/testing/selftests/tc-testing/tc-tests/filters/tests.json +++ b/tools/testing/selftests/tc-testing/tc-tests/filters/tests.json @@ -58,5 +58,25 @@ "$TC qdisc del dev $DEV2 ingress", "/bin/rm $BATCH_FILE" ] + }, + { + "id": "4cbd", + "name": "Try to add filter with duplicate key", + "category": [ + "filter", + "flower" + ], + "setup": [ + "$TC qdisc add dev $DEV2 ingress", + "$TC filter add dev $DEV2 protocol ip prio 1 parent ffff: flower dst_mac e4:11:22:11:4a:51 src_mac e4:11:22:11:4a:50 ip_proto tcp src_ip 1.1.1.1 dst_ip 2.2.2.2 action drop" + ], + "cmdUnderTest": "$TC filter add dev $DEV2 protocol ip prio 1 parent ffff: flower dst_mac e4:11:22:11:4a:51 src_mac e4:11:22:11:4a:50 ip_proto tcp src_ip 1.1.1.1 dst_ip 2.2.2.2 action drop", + "expExitCode": "2", + "verifyCmd": "$TC -s filter show dev $DEV2 ingress", + "matchPattern": "filter protocol ip pref 1 flower chain 0 handle", + "matchCount": "1", + "teardown": [ + "$TC qdisc del dev $DEV2 ingress" + ] } ]