The Chrome Application is open source:
The Ledger Wallet itself is open spec:
The firmware is not open source - as vendor NDAs prevent us from releasing a source code that would be of any use to our users. However, the specifications are fully open and detailed, and all cryptographic operations are deterministic, which allow any user to verify that the card is answering what it should and that there is no side channel.
We believe that validating both the card behavior and the clients behavior against the specifications should provide enough confidence to the users - also, this approach is widely used in the smartcard industry, typically to obtain security certifications for banking and identity products.