In my experience, validation/conformance testing is primarily of value to the developer, rather than the end user. Conformance testing demonstrates the degree to which the compiler has implemented the language specification. The benefit to a developer is the ability to precisely characterize portability issues if the need arises to change compilers. This in turn mitigates risks associated with changing vendors or platforms.
The end user usually cannot tell whether company X or company Y compiler was used just by looking at the MMI. Quality products are the result of quality processes which encompass much more than a specific compiler or code generator. Customers requiring the highest levels of integrity may want to investigate the FAA Certification process for DO-178B Level A software (software which could lead to the loss of an aircraft).
The views herein are the opinion of the author and do not necessarily reflect those of The Boeing Company, nor of its suppliers or customers.
> From: Roger Racine[SMTP:[log in to unmask]]
> Reply To: Roger Racine
> Sent: Tuesday, May 18, 1999 1:05 PM
> To: [log in to unmask]
> Subject: Conformance Testing
> Our company is involved in an Army project planning to use Ada. An Army
> officer wrote a memo describing his concerns over the loss of DoD
> involvement in what used to be validation of compilers. A very rough
> synopsis of his concerns is:
> * Since the testing is now being done by a commercial group (the ARA), the
> testing might be less rigorous than when it was done by the AJPO, so as to
> satisfy the commercial sector, which has less need for rigor than the DoD.
> This might lead to less assurance that the compiler is defect-free, which
> then leads to the conclusion that the end system will be less reliable.
> We are being asked to comment on this memo, and would like any information
> regarding the Conformance Testing that might alleviate this concern. For
> example, is ISO, or any independent organization, involved in the
> definition of the tests?
> Since the government dropped testing, has testing continued?
> Basically, I could use any ammunition on why conformance testing is either
> A) just as good as it used to be; or B) why conformance testing is not a
> major factor in end-system reliability.
> Pointers to web sites or technical articles would also be welcome.
> Roger Racine
> Roger Racine
> Draper Laboratory, MS 31
> 555 Technology Sq.
> Cambridge, MA 02139