Hackage Stats: The Past Year

16Mar11

It’s been a while since I’ve crunched the Hackage / Cabal logs. Here’s an update, covering March 2010 – Feb 2011.

Hackage

By March 2011, there were 2894 packages on Hackage. There were 2.02 million “cabal installs” in the 12 months to March 2011 (doubling the download rate of the previous year). There have been just shy of 4 million total “cabal installs” total, since Hackage went live. By 2011, Hackage was averaging over 200k downloads a month, more than double the same period a year ago.

Hackage continues to grow very quickly, averaging 18 package releases a day in March 2011, compared with 12 a day a year ago.

The Long Tail

The long tail of package popularity is visible on the log/log scale of downloads versus package rank, where the top 10% of packages, get 80% of the downloads

.

Libraries

The top 15 libraries in 2010 were:

  1. parsec (30.3k downloads)
  2. HTTP (27.1k)
  3. network (26.8k)
  4. mtl (26.1k)
  5. binary (23.9k)
  6. zlib (23.5k)
  7. transformers (20.7k)
  8. utf8-string (19.7k)
  9. QuickCheck (19.4k)
  10. Cabal (17.5k)
  11. text (16.6k)
  12. haskell-src-exts (15.2k)
  13. deepseq (15k)
  14. regex-base (14.5k)
  15. hslogger (14k)

Apps

The top 15 apps in 2010

  1. cabal-install (25.9k downloads
  2. haddock (12.5k)
  3. xmonad (10.4k)
  4. cpphs (10.4k)
  5. happy (9.7k)
  6. alex (8.3k)
  7. pandoc (6.8k)
  8. darcs (6.4k)
  9. snap (5.5k)
  10. hlint (5k)
  11. leksah (4.5k)
  12. happstack (4k)
  13. xmobar (3.9k)
  14. yesod (3.1k)
  15. cabal2arch (2.6k)

Some noticeable changes: the 3 core web frameworks are way up in the list.

  • xml is the most popular xml library (12k) vs HaXml (7.6k), hexpat (4.3k), hxt (3.1k).
  • For graphics, cairo is the clear winner (12.5k), gtk (8k), wxcore (3k)
  • vector is the most popular arrays library (10k)
  • cereal is closing in on binary (9.7k)
  • attoparsec is climbing (8.3k)
  • HDBC remains the most popular datbase layer (6.5k)

The yearly totals for each package are available here.


Filed under: News   |  Leave a Comment
Tags:

The Year In Haskell

20Dec09

Spend some time to catch up on what’s been happening in Haskell this year. In this edition of the Web News, we look back on the major Haskell news of 2009. You can also read this edition in PDF form.

The Haskell Web News brings you what’s been happening with the Haskell programming language as voted by readers of The Haskell Reddit.

Community

Haskell 2010: A major event in 2009 was the release of the  Haskell 2010 standard — the first official revision to the language since Haskell ’98, and the first result of the Haskell Prime process. The new language standard will form the core of a rolling language upgrade in coming years.

The New Logo: Along with the new language standard came a new logo for Haskell – again, the first since Fritz Ruehr’s proposal back in the day. To produce the logo, a community competition was launched, and via anonymous condorcet voting from -cafe@ members, a winner was chosen!

Hackathons go global: For the past 3 years, a series of Haskell hackathons have been held in Europe, one or two a year, as  the Hac series. In 2009, we held 4 hackathons, and for the first time, two in North America: in Portland, Edinburgh Philadelphia and Utrecht. The next Hackathon is in Zurich in March 2010.Hac icon

New user groups: For a long time the Haskell community relied on conferences for our user groups. By the end of 2009 however, the Haskell user group concept has become widespread, with nearly 30 groups around the world, including ten groups in North America, and twelve in Europe, and six across Australia, Asia and South America.

Summer of Code: Haskell.org participated in the Google Summer of Code, for the fourth consecutive year. This year, Google funded development of 5 Haskell projects:   Gergely Patai to improve space profiling, Isaac Dupree to work on Haddock, Niklas Broberg on Language.Haskell, Petr Ročkai on scaling Darcs to large repositories, and Thomas ten Cate on EclipseFP. In addition, Jane Street funded a Haskell project for the first time, under the Jane Street Summer Projects series, supporting development of the Lambda Cube 3D rendering engine.

Books

Real World Haskell: a year ago, Real World Haskell was published, the first O’Reilly book on Haskell, the first Haskell book for professional programmers, and went on to win the Jolt Award for  best technical book of the year. It has since also been translated into Japanese. It was notable also for having the entire book available in pre-pub and post-pub as a wiki.

LYAH: a quirky and accessible online book was produced, Learn You a Haskell for Great Good, becoming something of an internet sensation. LYAH is set to be published in book form in 2010.beta

Functional Programming Fundamentals: Erik Meijer produced a 13 part video lecture series based on Graham Hutton’s Programming in Haskell, which went on to have 500k downloads in 2009.

Compilers and Infrastructure

GHC 6.12: Version 6.12 of the primary Haskell implementation, GHC, occured in 2009, with major improvements for parallel performance, support for ThreadScope profiling, dynamic linking on Linux, and with I/O libraries Unicode-aware. The improvements to parallel performance where benchmarked, and documented in the paper “Runtime Support for Multicore Haskell“. The GHC team also published their latest status report.

An LLVM backend for GHC? In something of a tour de force, David Terei designed an LLVM backend for GHC, implemented and benchmarked it :: PDF, finding that the current code gen was competitive, but that LLVM wins out on the numeric-based loops typical in data parallel code.

Dataflow Optimizing Backend: Norman Ramsey and João Dias presented a new dataflow optimizing backend for GHC, making it easier to do some very aggressive low level optimizations late in the compilation phase.

Data Parallelism: Data Parallel Haskell progressed further, and GHC 6.12 is significantly better than prior GHC’s. The current status of the data parallel extensions to Haskell was described at the HIM.

Supercompilation for GHC? Yet more work on the optimization front, a design was developed for adding a supercompilation phase to GHC, making possible whole new classes of aggressive code transformation optimizations that were previously quite difficult.

GHC with shared libraries: Well-Typed, funded by the Industrial Haskell Group, completed the work of enabling shared libraries to be produced by GHC.  That means “Hello world” in Haskell is now only an 11k executable using GHC.

JHC progress: There was a new major release of jhc, the whole program optimizing Haskell compiler.

UHC: At the Utrecht Hackathon on April, the first release of UHC, the Utrecht Haskell Compiler was announced, a new Haskell compiler with a GRIN-based intermediate representation, which makes extensive use of attribute grammars in its construction.

Libraries and Distribution

The Haskell Platform: in 2009, a major effort was launched to provide a single “batteries included” library suite for Haskell, suiting commercial, open source and research needs. The result is The Haskell Platform, (also now available in Debian), a single installer for a complete GHC-based Haskell system, including extensive libraries. By the year’s end there had been 450 thousand downloads of the binary installers!

Hackage: Hackage is the central repository for Haskell libraries and applications, and, along with Cabal, provides a publishing, distribution, and dependency resolution system for all Haskell code. Hackage continues to grow in 2009, faster than in 2008. There have been 3500+ package updates this year, so we now have 1750 Haskell packages available on Hackage in total, up from 750 a year ago. The size of Hackage is phenomenal, as you can see when you visualize the Haskell Universe. Earlier in the year, we reached one million Haskell downloads on Hackage. November 2009 was the first month with more than 100k downloads from Hackage in a single month.

Industry

The Industrial Haskell Group: For the first time, in 2009, a consortium of companies came together to fund continued development of Haskell and its toolchain. The result is the IHG, launched earlier in the year. The initial round of funding resulted in several improvements to the ecosystem. The group is seeking to expand membership in 2010 to further consolidate the commercial strength Haskell. The Birth of the IHG was presented at the “Commercial Users of Functional Programming” workshop.

CUFP: This year’s Commercial Users of Haskell workshop was held in Edinburgh, and included talks on Real World Haskell, teleconferencing on maps in Haskell, FP at Facebook. Next year’s CUFP will be held in Baltimore.

Industrial reports: As part of Galois‘ 10th birthday, Don Stewart presented a talk at the LondonHUG on Engineering Large Projects in Haskell, celebrating a decade of use of Haskell by Galois. TypLAB, a new startup  talked about why they use Haskell. Facebook released its lex-pass tool to automate changes to a PHP codebase, by writing abstract-syntax-tree transformers in Haskell. And Starling Software described building a real time financial trading system in Haskell :: PDF. Tom Hawkin’s Atom EDSL for control systems went into production use in trucks and buses, and is starting to be used on a NASA runtime monitoring project. The Cryptol release got slashdotted. Tupil.com talked about their experiences building commercial web apps in Haskell.

Conferences

The major conferences of the year were in Edinburgh, and for the first time we have video of the entire week, including videos of the International Conference of Functional Programming,  the Haskell Symposium, the Commercial Users of Functional Programming,  and the Haskell Implementors Meeting. Hooray for Malcolm Wallace for compiling these videos.

Libraries

New IO library: the major news in libraries included a rewrite of the IO library (to support native unicode encodings on IO, in-memory IO buffers, and more). See Simon Marlow Wander Through GHC 6.12’s New IO Library.

Associated Types for Optimizations: Another popular post was on self-optimizing data structures, by Don Stewart, showing how to statically specialize the layout of Haskell container data types, improving data density and performance.

The Text Library:  A major revision of the Haskell text library was announced by Bryan O’Sullivan, giving the Haskell community a packed, unicode text type suitable for large scale unicode text processing.

LLVM adventures: One of the most interesting new libraries of 2009 was the high level LLVM bindings for Haskell. Lennart Augustsson and Bryan O’Sullivan had various adventures with it, including beating GHC (and GCC) and producing a BASIC embedded in Haskell via LLVM.

New tools

Vacuum: The biggest tools of the year included vacuum, Matt Morrow’s tool for extracting graphs of heap objects in running Haskell processes. With it you can vizualize Haskell data structures live in GHCi.

Benchmarking: Another great new tool is criterion, a library for statistically robust microbenchmarking, by Bryan O’Sullivan. You can read more about it in the release notes.

The Leksah IDE:major announcement was made at the Utrecht Hackathon: Leksah, a Haskell IDE in Haskell by Juergen Nicklisch-Franken.

ThreadScope: Finally, another key new tool addresses the problem of profiling in a multicore setting. The result is threadscope (a tool for profiling concurrent and parallel Haskell programs). ThreadScope lets you visualize graphically events in the GHC runtime system, including GC, thread migration, execution, spark activity and more. You can watch this video to see it at work.

Haskell in New Domains

Linux Kernel: Thomas DuBuisson showed us how to write Linux Kernel Modules in Haskell.

Barrelfish OS: Microsoft’s new Barrelfish OS was built on domain specific languages written in Haskell :: PDF

seL4: the first machine-verified microkernel: the Secure Microkernel Project (seL4) announced it’s major result: the world’s first formal machine-checked proof of a kernel, and they used Haskell as a key part of their development methodology.

iPhones: GHC was modified to allow native Haskell on iPhone.

Haskell for Theorem Proving: Haskabelle was released, a tool to convert Haskell source into Isabelle theory files, suitable for theorem proving activities.  A little fun with Haskabelle can be had.

Open Source

xmonad: The xmonad tiling window manager continued to be the most popular open source app on Hackage in 2009, with the xmonad 0.9 release occurring.

gitit: another significant new project is gitit, a feature-rich wiki running on top of git or darcs, with pandoc support (meaning you can write your wiki in markdown/tex/html/…)

happstack: the happstack web framework (the Haskell equivalent of Django) was in active development, with happstack 0.4 released. happstack powers gitit, and patch-tag.com, amongst other things. You can also read about why Happstack scales, and SQL fails!

darcs: the darcs revision control system world was extremely active this year, with darcs 2.2 and darcs 2.3 released (along with a Summer of code project) and great strides in scalability via a new storage mechanism.

Tutorials

The top 7 tutorials of the year:

Advocacy

And finally, the most popular pieces of advocacy for the year:

Haskell 10 years ago

And the news of 10 years ago? Here’s a selection of the top news in 1999.

HaskellDB: Daan Leijen announced a preview of HaskellDB, the type safe database library for Haskell, now on Hackage.

Edison data structures: Chris Okasaki announced the first public release of Edison, a library of efficient purely functional data structures.

Functional Graphs: Martin.Erwig announced the Functional Graph Library.

Haskell 98: Simon Peyton-Jones announced that the Haskell 98 standard was done.

Hugs : Mark Jones announced a beta release of Hugs 98.

GHC 4:  Simon Marlow announced the GHC 4.02.

c2hs: Manuel Chakravarty announced the C->Haskell 0.7.2 – beta release

Craft of FP: Simon Thompson announced the Haskell: The Craft of Functional Programming: Second Edition

TclHaskell. Meurig Sage announced the TclHaskell, an FFI binding to Tcl.

This is just some of the new stuff that has happened in Haskell this year. You can stay up to date by following the Haskell Reddit. See you in 2010!


Filed under: News   |  3 Comments
Tags: , , , ,

What’s new in Haskell? December 2009

05Dec09

What’s been happening with the Haskell programming language for the past month, as voted by readers of The Haskell Reddit. This is the first edition of the Web News, so feedback on the content, schedule and goals is welcome. Read more about the goals for the project on the about page.

Haskell 2010

The biggest news of November for the Haskell community was the announcement of the release of the new Haskell language revision: Haskell 2010. The Haskell Prime process, which produced this revision, calls for regular releases of newly standardised extensions, and this first revision declared as standard Haskell:

The end of n+k patterns in standard Haskell completes one of the longest sagas in the language’s history, with efforts to remove the feature more than 10 years ago .

The next steps for the standard are a revised language report, and for compilers to adopt the features. The Haskell 2011 group is also forming to investigate the next round of standardisation. (And see the Haskell 2010 slashdotting for fun…)

The packages that split Hackage

There are more than 1700 packages on Hackage now, with some quite complicated dependencies set up. Duncan Coutts looked at how to construct a maximal install plan, such that the largest fraction of Hackage could be installed without conflict. He found that a small number of key packages cause “splits” in the install scheme, as significant numbers of packages depend on incompatible versions of these conflicters (e.g. HaXml 1.13. or 1.19). Duncan was able to find a coherent install plan on unix for 1453 packages in the end, and since then a new stable version of HaXml has been released, which will further improve the overall coherency.

NoSlow: Microbenchmarks for array libraries

Designing the perfect array library in Haskell has been a hot topic for the last couple of years, and a number of competitors have emerged looking for the right balance of performance, flexibility and a rich API. Roman Leschinskiy announced a project, NoSlow, to methodically compare the performance of the various array libraries, with published results based on Bryan O’Sullivan’s criterion benchmarking tool.

An LLVM backend for GHC :: PDF

David Terei published his undergraduate honors thesis, “LLVM for GHC”, at the University of New South Wales, describing his design for porting GHC to use LLVM for code generation, finding that it mapped “remarkably well”. He then goes on to implement and benchmark the resulting compiler, finding that LLVM was generally similar to the existing code generator for the majority of programs, with the exception of array benchmarks, which were significantly better under LLVM. An amazing achievement for such a short project!

GHC 6.12.1 Release Candidate 2

The second release candidate for the GHC 6.12 series was announced, including installers for Windows and OS X. The upcoming 6.12 release features some significant improvements to data parallel arrays, new profiling tools, including support for ThreadScope, full support for shared libraries (as funded by the IHG), improvements  to the performance of parallel and concurrent programs in general, and support for unicode Handles (as well as other encodings directly in the Handle layer).

Interviews and Media

In the broader media, we saw an interview with John Hughes at InfoQ, contrasting Erlang and Haskell as well as a surprise Haskell talk at Supercomputing 09, where Haskell was used for high performance computational biology.

Finally, congratulations to Graham Hutton and Erik Meijer, as the series of C9 lectures, based on “Programming in Haskell” (the book), reaches a quarter million downloads.

New Haskell Software

In November, there were 287 package released on Hackage, bringing the total number of Haskell packages to 1720. We had a record 143 thousand downloads from Hackage in November, breaking the previous monthly record of 95 thousand set in October 2009, and bringing the total downloads from Hackage to 1.53M.

The top 15 most popular libraries in November were:

  1. HTTP
  2. utf8-string
  3. zlib
  4. binary
  5. haskell-src-exts
  6. QuickCheck
  7. ghc-paths
  8. X11
  9. parsec
  10. haskeline
  11. xmonad-contrib
  12. time
  13. network
  14. terminfo
  15. uniplate

And the top 15 most popular applications are:

  1. xmonad
  2. cabal-install
  3. haddock
  4. cpphs
  5. hscolour
  6. happy
  7. yi
  8. xmobar
  9. darcs
  10. alex
  11. hlint
  12. leksah
  13. pandoc
  14. happstack
  15. gitit

The two most popular Haskell project releases, as voted online, were:

Articles and Tutorials

The most popular new Haskell articles and tutorials online this month were:

You shouldn’t do that

And, finally, news that perhaps shouldn’t have been:

Next edition: a 2009 Haskell retrospective.


Filed under: News   |  2 Comments
Tags: , , , , ,