diff options
Diffstat (limited to 'source/documentation/backports')
| -rw-r--r-- | source/documentation/backports/hacking.rst | 406 | ||||
| -rw-r--r-- | source/documentation/backports/hacking/todo.rst | 120 |
2 files changed, 526 insertions, 0 deletions
diff --git a/source/documentation/backports/hacking.rst b/source/documentation/backports/hacking.rst new file mode 100644 index 0000000..72c64a1 --- /dev/null +++ b/source/documentation/backports/hacking.rst @@ -0,0 +1,406 @@ +Documentation/backports/hacking +=============================== + +.. note:: + + the backports project's process is being reworked and redefined. + Some of the information in this page may be outdated and will be + updated when the changes in the process are finalized. + +The project has a :doc:`To-Do List <hacking/todo>` and a :doc:`License +<../../license>`. + +Philosophy +---------- + +The project aims to optimize the backporting process by not only +providing backported drivers but by also aiming to `automatically +backport the Linux kernel +<http://www.do-not-panic.com/2012/08/automatically-backporting-linux-kernel.html>`__ +by `optimizing backporting collateral evolutions +<http://www.do-not-panic.com/2012/08/optimizing-backporting-collateral.html>`__. + +Backports development flow +-------------------------- + +#. The `backports.git + <https://git.kernel.org/cgit/linux/kernel/git/backports/backports.git>`__ + "master" development branch always tracks "linux-next", allowing it + to track all the development trees. + + - This ensures that, at the close of each merge window, the state of + the backports will be very close to the state of the first release + candidate. + +#. At the close of each merge window, the Backports Project creates a new branch. + + - This new branch tracks the progress of the impending release + throughout the release candidate evaluation period, to the major + release, and on to its lifetime as a stable kernel. + +The backports project thus makes three kinds of backports releases possible: + +- those derived from linux-next +- those derived from the most recent release candidate (if any) +- those derived from recent stable kernels. + +Backports git tree tags and branches +------------------------------------ + +The project uses two types of git tags: + +- daily tags following linux-next +- stable tags following either Linus' tree for rc releases or Greg's + stable tree for stable extra version releases + +The project uses a master branch and a series of stable branches, a +stable branch to track each stable release of the kernel. The master +branch always tags linux-next, for example the backports tag +backports-20141114 supports using linux-next tag next-20141114. Once +Linus makes a release candidate of the kernel a respective *stable +backports branch* is created based on the version of Linux Linus +released, this backports branch will be based on a recent backports tag +which was tracking linux-next. Future release candidates of the same +version of Linux as well as stable release of Linux under the same +version will be backported using the respective *backports stable +branch* created. + +If any of this is fuzzy please refer to the `Increasing Automation in +the Backporting of Linux Drivers Using Coccinelle +<https://github.com/mcgrof/paper-backports/raw/master/paper.pdf>`__ +paper, in particular the sections: + +- 1A - A. The Linux kernel development model +- 1B - A brief history of the Linux kernel backports project + +Git trees you will need +======================= + +The project backports a few subsystems down to older kernels. To be able +to synchronize backporting the latest and greatest the `linux-next.git +<https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git>`__ +tree is used as its main source for kernel updates. backports brings a +general kernel compatibility part containing some backported c functions +and header files in addition to patches for code in linux-next to make +it work on older kernel versions and with the general kernel +compatibility part. You will need first the `backports.git +<https://git.kernel.org/cgit/linux/kernel/git/backports/backports.git>`__ +tree. Once cloned you can run:: + + ./devel/backports-update-manager + +That will get you all other trees and sources required for development, +which are: + +- linux-next - Stephen Rothwell's tree which tracks all development trees daily +- linux - Linus' tree +- linux-stable - Greg's stable tree for stable extra version releases + +This will use git clone --reference to minimize disk usage. + +You can generate backports based on different trees depending on your +needs. We are supporting build based on `linux-next.git +<https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/>`__, +`linux.git +<https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/>`__ +and `linux-stable.git +<https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/>`__. +You can also generate builds based on your own tree, but then you +probably have to patch backports yourself to add support for that. + +Tool prerequisites +------------------ + +git, python patch and coccinelle are needed to generate a backports +release. The tools will complain if you do not have any of the required +dependencies. + +Generate new release +-------------------- + +To generate a own backport release based on linux-next you need the +following git trees:: + + git://git.kernel.org/pub/scm/linux/kernel/git/backports/backports.git + git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git + +As linux-next changes often you should start with a working version by +using a tagged version from linux-next.git and backports.git with the +same date in it like backports-20130607 and next-20130607. This is a +combination where someone already tested that the patches are still +applying. + +After you have checked out some version of these git trees, you can +start generating a tar by running ./gentree.py in the backports tree. +This will take the path to the already checked out linux-next kernel as +the second last argument and the path to a directory where the newly +created release should be written to as the last parameter. The releases +are not created in place any more, but in a new path specified in the +last parameter:: + + ./gentree.py --clean --git-revision v3.x /path/to/your/linux-next /path/to/your/newly/created/release + +Adding new driver +================= + +This is a list of steps to be done when adding a new driver. Not all +steps are needed for every new driver it depends on what type of driver +is added. You can take the patch adding support for the ieee802154 +subsystem as a reference: +https://git.kernel.org/cgit/linux/kernel/git/backports/backports.git/commit/?id=a42aa4d474e5859164a00bf451970fb58b676fcf + +You should target the master branch of the backports repository. + +#. Add directories and files that should be copied from the mainline kernel to ``copy-list`` +#. Add patches to ``patches/collateral-evolutions/`` if needed, it is better to add or modify a header file in ``backport/backport-include/`` than adding a patch. +#. Add reference to the newly added driver directory in ``backport/Makefile.kernel`` if needed (just needed for new substems) +#. Add reference to the newly added driver directory in ``backport/Kconfig.sources`` if needed (just needed for new subsystems) +#. Add a defconfig file for this driver in ``backport/defconfigs/`` and modify other defconfig files if appropriate, e.g. if you add a wifi driver add it to ``backport/defconfigs/wifi`` +#. If the driver does not build on all kernel versions specify a minimum kernel version in ``dependencies`` + +Backports currently supports the latest minor version of every mainline +kernel starting from 2.6.24. You should check that backports still +builds after making your changes. To do so you should install all +supported kernel with the script in ``devel/get-compat-kernels`` and +then run ``devel/ckmake`` to build backports against every kernel +version. + +Backporting a kernel feature +---------------------------- + +Features being backported should either be backported into a respective +backports-3.x.c file under backports/compat/ and/or their respective +header file under backport/backport-include/. + +Backports Kconfig operation +--------------------------- + +Go read the `backports kconfig operation +<https://git.kernel.org/cgit/linux/kernel/git/backports/backports.git/tree/devel/doc/kconfig-operation>`__ +documentation. + +Backports Makefile operation +---------------------------- + +Go read the `backports Makefile operation +<https://git.kernel.org/cgit/linux/kernel/git/backports/backports.git/tree/devel/doc/makefile-operation>`__ +documentation. + +Sending patches +--------------- + +backports contributions follow the contribution model implemented by the +Linux kernel. Patches or pull requests for backports must have be +signed-offed. If you don't sign off on them **they will not accepted**. +This means adding a line that says "Signed-off-by: Name " at the end of +each commit, indicating that you wrote the code and have the right to +pass it on as an open source patch. For exact definition of what the +Signed-off-by tag is you can read the definition of the "**Developer's +Certificate of Origin 1.1**", which you can read here: + +http://gerrit.googlecode.com/svn/documentation/2.0/user-signedoffby.html + +Remember there are three trees. linux-next itself is a conglomeration of +kernel git trees itself, so patches for linux-next.git should be sent to +each respective subsystem for which the patches are targeted for. So for +example for 802.11 you will want to send them to John Linville and cc +linux-wireless, for further guidelines on this see the `Submitting +Patches guidelines for 802.11 +<http://wireless.kernel.org/en/developers/Documentation/SubmittingPatches>`__. +As another example, for bluetooth you will want to send them to Gustavo +Padovan and cc the linux-bluetooth mailing list. If your patch touches +on others areas of the kernel refer to the MAINTAINERS file on the +kernel. + +For backports.git please send patches against to:: + + To: hauke@hauke-m.de, mcgrof@kernel.org + CC: backports@vger.kernel.org + Subject: [PATCH] backports: fix foo + +Patches are preferred sent with a clear commit log entry, if unfamiliar +with how to send patches please refer to `a git guide +<http://wireless.kernel.org/en/developers/Documentation/git-guide>`__. + +Requirements for contributions +------------------------------ + +The code quality, and coding style is precisely the same as followed by +the Linux kernel, for that refer to `the Linux kernel SubmittingPatches +<https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/SubmittingPatches>`__ +guide. Other than this, since we a supporting backporting you are +expected to refer to the respective upstream commit which you are +providing a backport for. This should describe both the git commit +sha1sum on Linux, and the commit description. The description should +also include the Linux kernel version which merged the commit upstream +first, this can be used to help understand what target kernels the +backport work was for. An example of good commit log follows:: + + commit 54584122b3279700f5ef2ba4f260a84273215947 + Author: Stefan Assmann <sassmann@kpanic.de> + Date: Fri May 16 13:21:19 2014 +0200 + + backports: handle new get_module_info and get_module_eeprom pointers in struct ethtool_ops + + In kernel 3.5 struct ethtool_ops received 2 new function pointers + get_module_info and get_module_eeprom. Address this by putting ifdef around + the code. + + commit 41c3cb6d20f0252308e9796fa4f3dacb4960de91 + Author: Stuart Hodgson <smhodgson@solarflare.com> + Date: Thu Apr 19 09:44:42 2012 +0100 + + ethtool: Extend the ethtool API to obtain plugin module eeprom data + + git describe --contains 41c3cb6d20f0252308e9796fa4f3dacb4960de91 + v3.5-rc1~109^2~124^2~2 + + Signed-off-by: Stefan Assmann <sassmann@kpanic.de> + Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de> + +Backporting preference for EXPORT_SYMBOL_GPL() and proprietary drivers +---------------------------------------------------------------------- + +This framework was designed by intent since its inception **only** for +*upstream* Linux kernel drivers to avoid excuses about the difficulty to +support **upstream** Linux kernel development while also supporting +**users on older kernels**. To ensure the intent is respected currently +all symbols that we do work on to backport are exported via +**EXPORT_SYMBOL_GPL()** as we do work to backport them for the supported +kernels. By using EXPORT_SYMBOL_GPL() we make it **clear** through the +only current technical means we have that if you use the backported +symbols your software **is considered derivative works of the Linux +kernel**. + +Backporting more subsystems +--------------------------- + +Feel free to any backport subsystems. The requirement is you at least +list its kernel dependency and also that you have tested compilation +success with ckmake. If you are adding a new subsystem we are in hopes +you will also help maintain it. + +Generating backports stats +-------------------------- + +To help backports developers evaluate the gains of use of Coccinelle +SmPL a tool has been written as part of the work done towards the +`Increasing Automation in the Backporting of Linux Drivers Using +Coccinelle paper +<http://coccinelle.lip6.fr/papers/backport_edcc15.pdf>`__, the tool has +a temporary git tree on github, integration of the tool into the +upstream backports tree is desirable but porting of the tool to Python +must be done first. Below are instructions on how to use the tool to get +stats on the latest tree. The statistics give a slew of metrics over the +gains of using of Coccinelle towards automatically backporting the Linux +kernel on each release. + +First git clone upstream backports git tree, and have a linux-next tree +available (using Linus' tree is possible as well, you just need to use +the respective stable backports branch). Then generate a backports +release using --gitdebug so that each step of the backports process +creates a commit. This lets the tool evaluate the impact of using +Coccinelle at each step. Assuming you have backports cloned on +~/backports/ and the latest linux-next tree with the respective +linux-next tag associated with the latest backports tree tag you would +do:: + + cd linux-next git reset --hard next-20151218 + cd ~/backports/ + mkdir -p ~/build/ rm -rf ~/build/backports-20151218 ./gentree.py --gitdebug ~/linux-next ~/build/backports-20151218 + +Now go get the temporary backports stats tool and copy over the required +binary and script:: + + cd ~/ git clone https://github.com/mcgrof/backports-cocci-stats.git cd backports-cocci-stats make + cp gen-cocci-stats.sh ~/build/backports-20151218/ cp clean ~/build/backports-20151218/ + +Now generate the stats, you should see something like this:: + + mcgrof@ergon ~/build/backports-20151218 (git::master)$ ./gen-cocci-stats.sh + Development and Maintenance efficiency metrics: + + dev-efficiency maint-efficiency diff-wc diffstat clean SmPL-Patch + 0.555556 2.33333 21 5 9 0065-ndisc_send_na-argument.cocci + 2.75 8 64 22 8 0062-iff-no-queue.cocci + 0.666667 1.88889 17 6 9 skb_no_xmit_more.cocci + 1.16667 2.79167 67 28 24 ptp_getsettime64.cocci + 0.142857 1 14 2 14 features_check.cocci + 1.18182 5.45455 60 13 11 0055-netdev-tstats.cocci + 1.55906 3.55118 451 198 127 0054-struct-proto_ops-sig.cocci + 0.666667 1.88889 17 6 9 no-pfmemalloc.cocci + 0.634146 1.4878 61 26 41 set_vf_rate.cocci + 3.75 10.625 85 30 8 igb_pci_error_handlers.cocci + 1.07692 4.23077 55 14 13 ethtool_cmd_mdix.cocci + 0.588235 1.23529 21 10 17 rxnfc.cocci + 0.285714 1.53571 43 8 28 get_module.cocci + 0.285714 1.53571 43 8 28 ethtool_eee.cocci + 0.714286 2.28571 16 5 7 skb_no_fcs.cocci + 0.25 1.59375 51 8 32 set_vf_spoofchk.cocci + 0.428571 2.85714 40 6 14 sriov_configure.cocci + 0.87037 2.7037 146 47 54 0031-sk_data_ready.cocci + 4.4 17.9 179 44 10 genl-const.cocci + 7.11111 49.6667 447 64 9 0019-usb_driver_lpm.cocci + 0.571429 4.14286 58 8 14 get_ts_info.cocci + 11.5333 45.4 681 173 15 0001-netlink-portid.cocci + 2.96667 16.35 981 178 60 0002-no_dmabuf.cocci + 0.444444 1.77778 32 8 18 0002-gpio-parent.cocci + 0.512821 1.89744 74 20 39 0002-group_attr_bus.cocci + �� 0.769231 2.79487 109 30 39 0001-group_attr_class.cocci + 1.47184 5.6484 3711 967 657 all-SmPL.cocci + + Patch total diff wc -l: 3075 SmPL total diff wc -l: 3711 Total total diff wc -l: 6786 + Patch diff % contribution: 45.3139 SmPL diff % contribution: 54.6861 + +Now say you wanted to generate stats for a newer release, say for stats +based on linux-next tag next-20160122, you would do:: + + cd ~/linux-next/ + git reset --hard next-20160122 + cd ~/backports/ + ./gentree.py --gitdebug ~/linux-next/ ~/build/backports-20160122 + cd ~/build/backports-20160122 + cp ~/backports-cocci-stats/gen-cocci-stats.sh . + cp ~/backports-cocci-stats/clean . + ./gen-cocci-stats.sh + + Development and Maintenance efficiency metrics: + + dev-efficiency maint-efficiency diff-wc diffstat clean SmPL-Patch + 0.4 2.6 13 2 5 0067-mdio-addr.cocci + 0.555556 2.33333 21 5 9 0065-ndisc_send_na-argument.cocci + 2.75 8 64 22 8 0062-iff-no-queue.cocci + 0.666667 1.88889 17 6 9 skb_no_xmit_more.cocci + 1.16667 2.79167 67 28 24 ptp_getsettime64.cocci + 0.142857 1 14 2 14 features_check.cocci + 1.18182 5.45455 60 13 11 0055-netdev-tstats.cocci + 1.55906 3.55118 451 198 127 0054-struct-proto_ops-sig.cocci + 0.666667 1.88889 17 6 9 no-pfmemalloc.cocci + 0.634146 1.4878 61 26 41 set_vf_rate.cocci + 3.75 10.625 85 30 8 igb_pci_error_handlers.cocci + 1.07692 4.23077 55 14 13 ethtool_cmd_mdix.cocci + 0.588235 1.23529 21 10 17 rxnfc.cocci + 0.285714 1.53571 43 8 28 get_module.cocci + 0.285714 1.53571 43 8 28 ethtool_eee.cocci + 0.714286 2.28571 16 5 7 skb_no_fcs.cocci + 0.25 1.59375 51 8 32 set_vf_spoofchk.cocci + 0.428571 2.85714 40 6 14 sriov_configure.cocci + 0.87037 2.7037 146 47 54 0031-sk_data_ready.cocci + 4.4 17.9 179 44 10 genl-const.cocci + 7.11111 49.6667 447 64 9 0019-usb_driver_lpm.cocci + 0.571429 4.14286 58 8 14 get_ts_info.cocci + 11.5333 45.4 681 173 15 0001-netlink-portid.cocci + 2.96667 16.35 981 178 60 0002-no_dmabuf.cocci + 0.444444 1.77778 32 8 18 0002-gpio-parent.cocci + 0.512821 1.89744 74 20 39 0002-group_attr_bus.cocci + 0.769231 2.79487 109 30 39 0001-group_attr_class.cocci + 1.46375 5.62538 3724 969 662 all-SmPL.cocci + + Patch total diff wc -l: 3087 SmPL total diff wc -l: 3724 Total total diff wc -l: 6811 + Patch diff % contribution: 45.3237 SmPL diff % contribution: 54.6763 + +See also +-------- + +- :doc:`To-Do List <hacking/todo>` +- :doc:`License <../../license>` diff --git a/source/documentation/backports/hacking/todo.rst b/source/documentation/backports/hacking/todo.rst new file mode 100644 index 0000000..07244b0 --- /dev/null +++ b/source/documentation/backports/hacking/todo.rst @@ -0,0 +1,120 @@ +Documentation/backports/hacking/todo +==================================== + +The Linux kernel backports TODO list. + +make localmodconfig support +--------------------------- + +The Linux kernel has *make localmodconfig* support, random users +shouldn't be expected to know what modules they need, so in order to +allow users to get only the drivers they need built we'd need something +similar to *make localmodconfig*. There are a few ways to do this. The +best way would be to get upstream a mapping of a kernel object of a +device to list its respective Kconfig. We'd also need a map of Kconfig +changes over time as the kernel advances so that if a user is on an +older system their old kobject Kconfig could be mapped to a new one. +This should require changes upstream. + +Other simpler solutions are to rely on 'lspci -k' output and map this to +respective device drivers but this solution would be PCI bus specific. +We should review how the upstream *make localmodconfig* currently works +and improve upon that to enable support for it on backports. + +Who's working on this? +~~~~~~~~~~~~~~~~~~~~~~ + +No one. + +Module namespaces +----------------- + +Module namespace can consist of either adding prefixes to existing +symbols or modifying the Linux kernel module data structure so that upon +request for a symbol it's own namespace can be inspected for its +requested namespace. The Linux kernel backports project embraces +prefixing backported versions of symbols by prefixing backported +versions of symbols with *backports\_*. A macro LINUX_BACKPORT() was +added to help with this. For example:: + + #define crc8 LINUX_BACKPORT(crc8) + +Andi Kleen introduced the modular structure modification version of +`module namespaces +<http://thread.gmane.org/gmane.linux.network/78674>`__ back in 2007 but +`Rusty Russell nacked these patches +<http://article.gmane.org/gmane.linux.kernel/606885>`__. Because of this +we're left with remapping modules to their own namespaces without +modifying the in-kernel module structure. It **may** be possible to +somehow provide module namespaces without requiring modifications to the +in-kernel module structure, but such mechanisms would end up being +hacks. We need to evaluate if we can figure a clever way to support this +that is not so error prone or if we want to respinning Andi's patches +with the intent to help support kernel backporting. + +Original motivation for module namespaces +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Here's a brain dump from Andi about the original motivation behind +module namespaces: + +The original intent was to provide a cleaner export symbol API. + +There are roughly two classes of exported symbols: + +- Generally usable, well designed, interfaces intended to be used by + multiple modules (e.g. functions for drivers to register themselves or + library functions) +- Special purpose exports that are only really for a single module, but + are not intended as a general interface. These interfaces are usually + not really clean and reusable. + +The idea was to mark the later special purpose exports with the name of +the module that is supposed to them. They wouldn't be available to +everybody else and wouldn't become part of the general kernel module +API. + +Motivations +~~~~~~~~~~~ + +At times Linux distributions may choose, for whatever support-nightmare +reasons, to support two versions of a subsystem module and have two +separate device drivers use a different version of that subsystem. The +reason this is ill-advised is we stand to gain by having development +done on the upstream and latest versions of each subsystem and drivers; +that backports framework allows you to get that. An example helps to +clarify here. Suppose a distribution wants to ship with an old version +of mac80211 and have the iwlwifi driver use it, but for ath9k they'd +want to ship the newer version of mac80211 and new version of ath9k. +This cannot be supported unless module namespaces are created. + +Another more reasonable motivation is to avoid regressions with features +/ symbols being updated only for backported drivers. Non-backported +drivers will use the non-backported version of the routine. This should +avoid regressions with non-backported drivers using these routines. + +Who's working on this? +~~~~~~~~~~~~~~~~~~~~~~ + +No one. + +DKMS support +------------ + +Someone seems to want add support for `DKMS +<https://en.wikipedia.org/wiki/Dynamic_Kernel_Module_Support>`__. Who +was that? What benefits does this bring? Isn't DKMS for proprietary +modules? If so we don't want this support. We don't want proprietary +drivers using the Linux kernel backports framework. + +DKMS is for proprietary modules only because all proprietary modules are +out-of-tree, and DKMS is for all out-of-tree modules, whether they are +proprietary or not. Since Linux kernel backports are out-of-tree, DKMS +brings the benefit of automagically keeping existing backports active +when a new kernel is installed, rather than going back to the in-tree +modules with every new kernel. + +Who's working on this? +~~~~~~~~~~~~~~~~~~~~~~ + +No one |