The Budapest Open Access Initiative has a comprehensive Guide to Institutional Repository Software.

Two of the larger systems are Dspace and Eprints.

Laval University has developed its own open source repository system, Archimede, which is diffirent from although inspired by Dspace.

Material Repositories - Rafts or Battleships? is a useful article about one group's attempts to find a suitable repository system.