Our tech lab reviews several financial applications every year from a variety of vendors that are focused on the financial institution market space. The majority of these applications perform poorly to some extent in either security and/or usability. Here are three key tips for vendors to keep in mind when they or their clients ask us to do an assessment of their application.

1. Make sure the application actually works as it would in a production environment. Make sure it is reasonable in terms of performance. The idea of performing our lab assessment is to model risks in a real world simulation. Thus, if the system is not configured and working as it would in a real deployment, then the validity of the test is poor. Many of the applications we test simply do not function as expected. Many times, their performance is so slow and horrible that it impacts the availability metric. Basically, by the time it is submitted for the complete application assessment or risk assessment, it should work and be installed in a QA environment just as it would be in production. If there are any variances, be prepared with a document that explains them and their anticipated effects. Be ready to discuss and defend your assertions with a team of deeply technical engineers.

2. Do the basics. Make sure you meet an established baseline like PCI, ISO or some other basic security measure. That means ensuring that controls are in use to provide for confidentiality, integrity and availability. That means that you are protecting the data properly during transit, storage and processing. That means that you and/or your client have an idea about how to provide preventative, detective and responsive capabilities around your product. Make sure your documentation clearly explains any security assumptions or add-on products required.

3. Be ready to handle issues. If/When we find a security issue, be it overflows, input problems, and/or best practice variances, be ready to mitigate the issue and submit a fix. Many times it takes months for vendors to handle the issues we find and this is certainly NOT good for their relationship with the client. Almost every full assessment our lab conducts involves some kind of deployment timeline and crunch from the customer. Nothing seems to go worse for vendors whose products we test as when an issue is found and they become unresponsive to us and/or their client. Seriously, JUST DON’T DO THIS. Be prepared to apply resources to fix issues when we test the application. Very few applications (less than 2%) pass through the lab process without some sort of issue. This is NOT a basic process, it is a seriously deep, complex and heavily leveraged process for finding holes and measuring impact. Be prepared.

I hope this post helps both clients and vendors be better prepared for their testing. I think it gives the basic ideas for the approaches that we know do not work. We really want your applications to be secure, thus the level of detail we apply. Let us know if you have any questions. We are also about to open the lab registration window for 1Q09, so if you have applications you would like tested, let us know and we will try and get them on the schedule.