How to Contribute (Draft)
[Note: please feel free to edit these pages at will. Absolutely none of the following is an official statement of policy. Only after collaboration, much discussion and revision will this draft go on the mpir webpage as an official presentation of our policy.]
This page describes different ways you can contribute to mpir, guidelines for submission and details of the submission process.
Ways to contribute to mpir
There are a number of ways you can contribute to mpir:
1. Large projects
The number of large projects is very limited, as they are difficult to manage and integrate. But there are a number of such projects and we are always interested in thinking about new suggestions.
Currently people are working on: a new integer format for arrays (the fmpz integer type), a new generic C layer (called mpir_n) to eventually take over from the irregular mpn layer, a multithreaded library of functions using OpenMPI and a new FFT for large integer multiplication.
More details can be found here: Large projects
To initiate a large project, first garner support from the mpir list http://groups.google.co.uk/group/mpir-devel/topics?start=
This may involve writing proof of principle code demonstrating how the idea will work, and giving performance figures.
Next, copy the entire development source tree from trunk in the repository, make a new directory in branches with the name of the project and implement it there. Once the project is ready to be merged into trunk, announce this on the list and the process can begin.
2. New functions
Note that mpir is not a general purpose number theory library, so arbitrary functionality will not be added. To submit a new function, first announce it on the mpir list. Anyone objecting should do so within a day or two. If there are no objections, go ahead and implement the function and submit it.
First make a trac ticket explaining what you are working on. Nothing happens in mpir without a trac ticket.
We operate on a one function, one file basis. So your new function should be contained in an appropriately named file, e.g. a new mpz function mpz_blah, would be in a file mpz_blah.c and included in the mpz directory. Simply place the file directly into trunk. No branch needs to be started unless your function relies on functionality from a large project.
It will be helpful if you can add a wiki page with a description of the function you are implementing. On the main page of the wiki http://wiki.mpir.org/ is a section entitled Structure. It is a hierarchical set of pages reflecting the structure of mpir. Find the appropriate page and add a link to the page you created for your function.
There is also the following page listing all the current functions being worked on. Please also add a link to the page for your new function there: New functions
3. Performance optimisation
Performance optimisations whether they be function, module or project wide are always received with enthusiasm. This work can be done directly in trunk, however the best way to proceed is to check out the repository, get the optimisation working, then commit to trunk. Basically trunk should build at all times for everyone.
Performance optimisations which require considerable complication of the code or which speed one platform or function up, but slow down others will not become part of a release. We have a no speed regressions rule. It is important that each performance optimisation is carefully profiled on a range of platforms (if relevant) before moving on to the next optimisation.
Module or project wide optimisations should be fully demonstrated with proof of principle code before rolling them out across the code base.
4. Build systems
We support MSVC, OSX Xcode and various other platforms as first class citizens.
Submissions to support new platforms are always welcome, but we encourage integration of the build systems with the existing code base to the greatest extent possible. Exceptions exist when such integration would cause uglification of code.
The general rule is, if a change or optimisation to the code supporting platform X would have to be propagated to the code for another platform Y, then the code should be integrated, unless it proves technically infeasible to do so, or if the result makes the code hard to maintain or understand.
Hacks to the build system are strongly discouraged.
Build system projects are described here: Build systems
5. Documentation
Developer and user documentation can always be improved. In particular it is great to have clear explanations and lots of examples. All new features have to be documented to be included, but existing documentation can often be improved after the fact.
The documentation should be confined to /doc with developer documentation in /doc/devel and user and general documentation in /doc.
6. Improving the code quality
Projects to improve the code quality vary. One example is converting x86 assembly code from gas (at&t) to intel format. Another example might be rewriting the build system in python, or consolidating the code for two different build systems into a single system. Yet another example might be turning fully unrolled loops into partially unrolled loops so that overrun is never possible, thus anticipating and preventing future bugs. A final example might be extending the integer format so it can handle much larger integers, or improving the FFT so it can multiply integers of say terabytes in size.
7. Auxilliary features
Auxilliary features might include example programs in the examples directory, benchmarking tools, test platforms and other tools distributed with mpir.
The website may also list additional projects which are not part of mpir but which arose from the project or are associated with the project.
Guidelines for submissions
The process that code must go through to end up in an mpir release is detailed in the next section, but no matter what that process is, it is essentially designed to maintain quality control and performance. In this section, the immutable principles behind that process are outlined.
Each piece of code making it into an mpir release must satisfy certain requirements:
Licensing
Roughly speaking, contributions must be compatible with an overall license of "GPL v2 or later", i.e. not version 3. Please see the licensing page for more details: Licensing
Tickets in trac
Nothing gets worked on without a trac ticket. See http://trac.mpir.org/mpir_trac/ The ticket should describe what it is you are going to do, before you do it. It is also highly encouraged to put information on the wiki pages about the work so others can see what is currently being worked on.
Code commenting
All code must be commented so that it can be ascertained how the code works. This means that all performance hacks should be commented, the basic flow of the code should be clear from the comments, and if the code implements a mathematical algorithm, the in-code comments should briefly note which parts of the code implement which parts of the algorithm.
Functions which implement a mathematical algorithm must have accompanying documentation immediately before the function implementation stating what is assumed about the inputs, what algorithm is used if it isn't obvious (including full references to papers if applicable) and what properties the outputs have. It is highly encouraged to put a runtime complexity if this is known or can be calculated, if the function is implementing a non-trivial mathematical algorithm. Proofs of correctness should also be supplied for any claims made in the code comments that are non-obvious, e.g. comments to the effect that wraparound can never occur, or a carry will never occur. This is only needed when it is not reasonably obvious why it is so.
Testing
All mpn or mpir_n level functions should have corresponding tests in the test program /tests/devel/try. Tests should pass for a wide range of the different kinds of data that the program can generate. Any new function should be tested for many hours with such data. If assembly language is used to speed up such functions, such tests should be done on all relevant platforms.
All higher level functions must have a test in the directory /tests which is run by make check.
Both low and high level functions must be tested extensively with random data. In particular we implement branch level testing, which means that each branch in the code must have been observed to be taken many times without crashing or incorrect results. This can be done by placing traces in the code at each branch and removing them one by one as the branch is observed to be taken sufficiently many times.
Test code should make non-trivial tests of the correctness of the code. This can often be done by implementing some consistency check, e.g. computing some checksum or comapring against a naive well understood implementation of the same computation, perhaps even involving a different simpler algorithm.
The tests that were performed must be described in the trac ticket associated with the function/module, before the ticket can be closed and the test code should also be adequately commented.
Performance
The basic rule is that no performance regressions are allowed. As part of documenting progress in the trac ticket, a description of how the function performs must be given. At this point we don't have a profiling platform, so this is done ad hoc. However code should be checked for peformance with small, intermediate and large data and non-symmetric data where applicable. Where a number of degrees of freedom are possible, a matrix of performance values should be given. This matrix should contain a significant number of data points. It is also useful to attach this data in the form of a table or graph to the trac system.
An mpirbench score improvement is never a sufficient indication of performance. Each function should be individually profiled.
Documentation
Each function which is intended for consumption by users of the library should be carefully documented in the user manual in /doc, including any specific assumptions about inputs that the code makes or any limitations the implementation has. Any function or macro written for use internally by developers should be documented in /doc/devel
Build testing
In most cases, build testing will be done by the release manager, Michael Abshoff and by automated scripts. There is documentation in /doc describing how to add new functions to the mpir build system. Build testing is the final step, and the release manager reserves the right to back code out before release that causes difficult build problems.
The submission process
Types of submission
There are two types of submission to mpir. The first type of submission is where someone external to the project submits code for possible inclusion in mpir. The submission process here covers the process that goes from them suggesting that it be included, to it being rewritten or integrated into mpir, to it finally being published in a release.
The second type of submission is for code written specifically for mpir, and developed directly in the trunk or one of the branches.
Who is in control
The mpir project is run by a developer friendly community. There is no "maintainer" as such, only a release manager who essentially manages build testing and is responsible for a range of quality control issues. Everything else is done by the community itself. Each developer must manage their own testing, documentation and profiling unless they have an explicit agreement with someone else to do it for them. Volunteers are usually available to help out if asked.
Announcing the submission
The first step in making a submission is to post to the development list http://groups.google.co.uk/group/mpir-devel/topics?start= announcing what it is you intend to work on. This involves stating what it is you want to do and give sufficient details of how it will be accomplished so people can evaluate. If there are no objections within a day or two, you are free to start work on it.
If you are making an external submission, once this process of discussion is complete, a volunteer will hopefully step forward to begin work on merging your patch if you don't wish to join the community and do it yourself. Volunteers are allowed to be paid for such work, if it doesn't conflict with their employment rules or visa conditions. Donations can also be made to the mpir project via SAGE http://www.sagemath.org/ and earmarked specially for mpir development. We acknowledge with gratitude, past and current contributions of funding, code and hardware here Contributions.
Note that at times, in order to convince the developers that what you propose is a good idea, you may have to write proof of principle code and give extensive timings to support your claims. No vote will be taken as to whether your idea is good or not. Objections will be raised by concerned developers as part of the normal process of evaluating what you propose. Some things will never receive approval, but if all serious objections have been dealt with or there simply aren't any, work can begin.
Everything has a trac ticket
The second step is to write a trac ticket, if one doesn't already exist, covering what it is you are going to do, and outlining how it will be done.
At the end of the work, infomation about testing and profiling should be added to the trac ticket. A list of files and functions that were modified should also be provided.
When you completely finish resolving a ticket, close it, but only after adding the ticket to the relevant section on the wiki Tasks. If you don't add the details there, it becomes hard to give credit to the appropriate people when the release manager issues a release. It also becomes difficult for the release manager to solve build issues, as he doesn't know what has been modified since the last release.
Do make new tickets when you find bugs or when you think of new features you'd like to see. All bugs affecting trunk are blockers. Do fix bugs whenever you can. The number of bugs should never grow too large as they will hold up releases. It is not the job of the release manager to fix bugs.
Update the wiki
Anything that is going to take you a while to do should be described on the wiki. Start with Structure and find the page corresponding to the part of mpir you are working on. Add an entry about what you are doing and link to it. Also add a link to this information to New functions if you are adding a new function to a module.
Check out a working copy from svn
To begin work check out the svn repository from http://flint.svn.sourceforge.net/svnroot/mpir-trunk/ Instructions for doing this can be found here: Getting the code Simple instructions for using svn can be found here: http://www.linuxfromscratch.org/blfs/edguide/chapter03.html
Note that large projects will generally add a new branch to svn and begin working there instead.
Committing code
Once the code is working and you'd like to commit, check that mpir still builds, if relevant, and then you can commit to trunk. If you already have access to svn as a trusted developer, then just go ahead and make as many commits as you need to. Always fill out a log message. It is always better to make lots of incremental improvements that don't break anything than one huge commit, unless you are a big project.
If you haven't yet been given svn write privileges, someone will need to inspect your code for quality. Ask one of the active developers on the list to check your code. This can happen by making it available somewhere for them to download, or by emailing them a tarball or patch. Usually the developer will either submit your code for you, or they will help you refine it.
Another way to proceed is to find someone with commit privileges and work with them on the code. This works best if you can get login access to the same development machine with edit privileges on the same local copy of the repository or if you agree to make patches and send them to each other.
Eventually someone may recommend that you be given write privileges to svn. If your coding ability is up to scratch and you'd like to be a trusted developer you will be given an svn account allowing you to commit code for yourself and others whenever you want. Essentially any trusted developer can commit to svn, but everyone still has to go through the full submission process. Basically all submissions to the project are equally welcome and will only make it to a release if they can satisfy the criteria in the previous section.
Alert the release manager
Once you have finished work on something it is good practice to let the release manager know and give him a list of tickets that you've completed. If your work was very successful, you are encouraged to brag about it on the list. Developers like to hear about successes. It is especially important to alert developers if you have implemented something which they can use.
Once you have let the release manager know that you are done, assuming you have completed all documentation, testing and profiling and any necessary reviews (see below) are complete, your submission is complete and you should see it acknowledged in the next official release if all goes well.
Official releases
Once a release draws near, mpir-trunk will be copied and called mpir-X.Y.Z, for some release number X.Y.Z. Development in the new mpir-trunk can then continue as normal, but only finishing work, bug fixing, build testing and reverting unwanted code or features will occur from that point in mpir-X.Y.Z. Michael Abshoff will eventually sign off on the release candidate and it will be available for beta testing. Finally it will become an official release, available as a tarball from the website. We anticipate releases quite regularly.
If code is backed out of a release, it will remain in trunk (unless it is deemed to be awful or has been orhpaned), so it can be worked on for possible inclusion in the next release.
Merges
From time to time, a major project will announce to the list that they wish to merge with trunk. For a short time, this will take precedence over all other development work, until the merge is complete.
Mathematical algorithms : review procedures
The above processes are how day-to-day work gets accomplished on mpir. But certain kinds of development require additional procedures.
Any code which implements a mathematical algorithm must be reviewed by someone with the necessary mathematical and coding expertise. This review process is similar to getting a paper published, except that you will know who is reviewing your code. The primary objective of the reviewer is to verify that your code does what you claim it does in the documentation. If a specific algorithm is implemented, their job is to verify that the code does what the algorithm says it should. The second objective of the reviewer is to check that your testing, profiling and documentation are adequate. Note that not all code needs review in this way, only code that claims to implement a non-trivial mathematical algorithm.
Community decisions
On occasion, the mpir project may need to make a decision which affects the mpir community, or a question might arise as to the best way to proceed in a certain situation. Generally a process of discussion continues until concensus is reached in the relevant subcommunity, i.e. until all meaningful objections are dealt with.
It is expected that any large scale projects or modifications to the project will undergo a period of discussion before any work is commenced. But if no serious objections are raised in that time, the work can go ahead. No vote needs to occur. Projects should do everything they can to make it clear what is being done, however, and any changes to design decisions made along the way. The wiki pages are useful for that, and may avoid large amounts of work being reverted or lost due to objections raised after work has begun.
Panel of advisors
In situations where concensus is not reached, one option we may consider is to appoint a panel of outside advisors to make recommendations and in some situations to resolve the issue. This panel of scientific and computational advisors will consist of experts not currently active in the mpir community, but well known for their input into computational mathematics.
Eventually it might even make sense to appoint a permanent panel of mpir and non-mpir experts who will take charge of assigning external referees to review the implementation of important algorithms in mpir.