One of the problems I had while designing a library was the API documentation. The traditional way was to create separate files that contain documentation and update it when it is needed to. Although this sounds simple, it is not. It is really hard to keep documentation up to date when dealing with libraries under heavy development. As you may have already noticed, there are may real world libraries with out-dated documentation, and the most important reason for this, seems to be the separation of documentation and source code.
While exploring alternative ways of documenting (for the gnutls libraries), gdoc by Michael Zucchi seemed really promising. The idea behind this program is to document the API within the source code and extract it later, to create the documentation. The idea is simple, and it is really easy to keep the documentation up to date. Since I only know LaTeX, I've modified gdoc to be able to produce LaTeX output. This can be found at the gnutls git repository.
As an example of the gdoc output see the gnutls documentation (function references) at gnutls site.
See also DOC++.