Brain's Crust

Offline web application

2011-02-11T01:03:00.000+01:00

Some time ago I was thinking about the issues that envolve the modern trend of having all the applications online somewhere on the "net". GoogleDocs, Picasa, e-banking, online calendars and more seem to be the right way to go ... unless you find yourself somewhere without a reliable connection.

One of the most powerfull buzzwords nowadays is "cloud computing" and the "everything-in-the-cloud" pattern. But I bet almost everyone has at least once been in a situation where he needed to work with his favourite web tool but he simply couldn't since he lacked a good quality connection. What then ? Wouldn't it be awesome if he could just use the tool localy and once connected synchronize the data without him even knowing ?

There are several basic ways how to achieve the required behaviour. The modern one would be to use HTML5, but due to low browser support and the fact that many users/institutions use old versions of browsers this is not usually a good road to take. Another solution if you already have a fully functional web application is to deploy it on a localy installed application/web server and implementing the tier responsible for data synchronization. Last but not least interesting is the somehow forgotten solution of creating a desktop application (thick client) that uses the business logic stored on the server through some middleware technology (with delayed evaluation untill online) or SOAP.

So I've started searching the web for some articles that cover the topic and here are some of the most interesting I've found:
Synchronizing a Web Client database (Google Calendar)
Operational transformation explained
HTML5: Client-side Storage
Making the Case for Local Database and Synchronization

One of the basic problems that we need to cover is the synchronization of databases and since "reinventing the wheel" is a commonly known anti-pattern I've searched for some tools that can do the job effectively:
SymetricDS
Daffodil Replicator

Also there are some interesting frameworks and technologies that might come handy:
SmartGWT and SmartClient Framework
JavaFX

Business intelligence - concepts

2010-11-28T02:46:00.000+01:00

Since the ancient times people knew the importance of knowledge. It is a well-known fact that information if used properly means power. Today we are overwhelmed by information. It is everywhere and we now posses the technology required to gather it. But to gain the hidden power we have to use some techniques to help us manage, understand and benefit from all the data we have. To properly use the information is essential for banks, insurance companies, medical facilities and many many others. The importance of managing information lead to the new IT science field called "business intelligence". In short it can be described as the way of gathering, analyzing and using information for business benefits (statistics, estimations).

The first mention in technical literature about "business intelligence" comes from IBM in the 50's where it is described as the ability to perceive relations between presented data in a way that helps us achieve our goals through actions. The data is processed by a computer system, analyzed and the analysis results are passed to people that have the knowledge to use them for a specific benefit.

Today's understanding of BI is a little bit different but the main idea remains. It is more described as concepts and methods used to improve decision making based on data provided by systems managing and processing data. Since BI is a fast growing field there emerged numerous sub fields that although use different methods and algorithms have something in common. They all process data created at a different time, places, systems and for a different purpose. They all focus on "secondary data processing". A short list follows:

data warehousing (gathering and storing semantically similar data from different resources)
reporting (presenting analysis of data in an easy-2-understand manner)
OLAP (analytical online data processing)
EIS (executive information systems)
MIS (management information systems)
data mining (finding patterns and relations in potentially unrelated data)

Data is usually directly processed by the application that uses or generates it. The main purpose of data in these applications is usually only of supportive nature that matters only during the execution of a specific task but once the task is successfully done a big part of the data is of no use. This is called "primary data processing" and once it is done the data can be thrown away or stored in a sort of data archive where we have to ensure additional security. This was the main idea of using data in applications.

But once the amount of data started to grow some people realized that by looking at the archived data they might discover some useful information that was hidden before. The problem at that time was that the amount of data required to get relevant information out of it was bigger than the technology could handle and it kept growing. Luckily in the 90's the technology caught up and we finally got into our hands the resources to process and take advantage of all the gathered data. Investments came into the recycling of old data, purifying it and using it for business improvement.

At first the same methods that were used for both primary and secondary data processing but it soon became clear that both approaches differ. The primary processing relied on transactions with the typical CRUD operations and main aim on maintaining the data consistent, secured and accessible at any time by multiple users. This is called OLTP (on-line transaction processing) and is typical for working with small amounts of data. Systems for secondary data processing don't typically require that amount of security and concurrency since they mostly only read data but in much more bigger chunks thus they require specific optimizations for these operations. The structure of the data also varies since primary data is application oriented (sales, finance, investments, etc.) but secondary data is more "object-centric" (customer, product) gathering similar data from several applications.

In the next article we will look a little bit closer on the approaches used to process such data.

Software engineering 12 - Proposals and estimations

2010-01-11T13:15:00.001+01:00

Introduction
We discussed about the importance of communicating with the client and developing a software product. But we missed an important part which is: "How to get a contract ?". This feature requires loads of soft-skills, good reputation and even then don't expect crowds standing at your door that want to hire your services. And event after acquiring a client we need to make him "an offer he won't be able to resist" but that would of course bring us profit for that's what business is about. Creating time/costs estimates that can be fulfilled is essential as they are the first thing that every businessman looks at.

Creating offer
When creating a offer to the client there are 2 situations that you can encounter. You're either creating an offer for a completely new product or your offer concerns the modification of an existing already developed one. Clients hire software developing companies for 2 basic reasons:

they need to make modifications to the system that are too large or expensive in relation to time/money that they can't be part of the standard "system modification request" policy
they have an unmaintained product or they ended a contract with the original contractor

When creating an offer there are several steps that need to be taken (in order):

make a record about the new offer
declare responsibilities
create the offer
analyze and review the offer
describe the communication protocols that will be used
create estimates of the time/labor requirements

Sometimes offers can seem misleading when they don't include the ideas, materials and processes that were used for estimates generation. Businessmen don't usually conform to time/cost estimates at first if they are defined in longer time intervals or bigger sums. They have the feeling that things can always be done faster and cheaper and having the methodology and data used included in the offer helps convincing.

The content of an offer should be understandable, clean and exact in such a way the client gets a first overview of what is it he'll be paying for. Everything that is not clearly defined might lead to potential future conflicts that are very bad for your reputation and if you don't want to loose the client also to an unpredicted increase in costs due to satisfying his need. As people tend to say: "The client is always right." The basic items that you should cover in your offer are:

extent of work/project
labor required (ideally in "man-days")
deadlines/milestones
quality description
resource requirements
potential risks
marginal conditions
everything else to make the "people in charge" make adequate decisions

While establishing the project extents it is common to use structured and . Defining the extent of things we will implement (positive limits) and we won't (negative limits) is good to have for future negotiations on the price of additional features and changes. You should always avoid abstract limitations and rather use some countable way of doing it (instead of "fast" use "server's response under 5s"). Try to be defensive in terms of not promising things you can't do but don't be too restrictive or you might not get the contract. Part of the positive limitation is a set of all the feature we ply to implement. To know the right balance is the know-how that only experience can give you. Nevertheless it's good to include clauses saying that some items on will be specified later on as part of specification/analysis. Resource requirements for development, documentation and things related to the construction process should be included for they are part of the overall costs. One crucial part of every offer is to declare some conditions that would apply to the client as the requirements for cooperation which is essential for every phase of the development process (requirements gathering, specification, user acceptance testing).

Estimates

Everything around proposals, creating offers and attempting to convince the client you're the right one to get the contract is making correct assumptions about time, costs and resources required. That helps to intrigue the client and avoid problematic projects or those that are "out of our league". This quality is one of the basics of a good IT manager and having a pocket full of techniques and materials that help is key to success. There are some guidelines that can be found on the internet, also articles and discussions describing their experience is helpful but what helps the most is own experience and a history of previous project with all the project plans, analyses, resource management history, etc. Creating a summary after each project that would contain the system characteristics, time spend on various phases and activities, encountered problems and risks, time/cost summaries and everything you find useful will soon result in an essential knowledge base. This summary should be based mostly on measurements (time, size, effort, quality) and usually project management tools have mechanisms to support this which makes it really easy to do (BugZilla, Jira, SVNStat, etc). The estimates done in the initial phases of the project are very rough and they are modified during the various phases of the project or as reaction to an unpredicted event but keep in mind that for the client the estimations given in the offer are the ones that count the most so a common practice is to keep a time/cost reserve that can handle such situations.

Conclusion

Creating an offer for the client is a complex process that needs the involvement of management as well as IT staff to create a realistic plan capable of successful completion. Correct estimates are what can help projects success of fail and measures need to take place to help create them. Having summaries of previous projects is the logical place where to start an archiving them is essential. Always remember that it's business and the main goal is to gain "profit" (cash, reputation, knowledge).

Software engineering 11 - Project management

2010-01-10T13:32:00.001+01:00

Introduction
Project management is not directly part of the essential parts of the software development process. Is is more of a supportive part for it helps coordinate and manage the rest of the software development phases which means you can live without it but it would take a huge amount of luck and self-discipline to succeed this way on larger projects.

Project management is understood slightly different by various world class software companies:

NASA (First begin by start planning the whole project and understand the problems, requirements and work that is about to be done. Technical approach needs to be defined by choosing an adequate life-time model, methods, activities and products. Then the project plan should be finished defining the relations with management, organization, estimates, time-planning. After that it's time for implementing the project plan and project management's role is to monitor, control and maintain the plan of software development. The last responsibility of project management is the projects closure after everything is done or when the project is canceled.)
SYBASE (By Sybase the project management tasks are 3 consecutive actions. The project "initiation" which defines the problems and analyzes potential solutions. It is also responsible for creating a project plan. Then it is "execution" which aims at implementing the plan while monitoring and managing the overall progress. Finally there is "closeout" of the project.

Software management program control
This is an overall look at the requirements of project management as seen by some "best practices":

Software management planning should include

definitions of product goals (establish functionality, criteria of success)
how will the product be structured
project plan analysis and its testing
time estimates for the project (schedules)
estimating costs ("costing the plan")
frequent plan revisions based on the environment changes

Implementation of the plan
Methods for including changes into the plan and project
Methods for achieving project goals (resources, quality, etc.)
Coordination of the effort of various groups and individuals
Ensure adequate quality control mechanisms to assure the high quality of the product

Project management is in essence strongly connected with all the various phases of the software development for it's the central management point (requirements, architecture, configuration management, engineering process, quality assurance).

Basic "howto"
The basic point where to begin is establishing a project organization that should be absolutely clear. Absolutely essential is to understand "what is it we're going to do" to avoid problems with doing wrong things or the good things the wrong way. The key to successful project management is to decompose the whole work into small and compact units for it's way easier to make a time/cost estimate on a small problem than on the whole one. Also creating an analysis and resource estimation/planning is more easy on small pieces. After the decomposition we need to assign available resources to he various units or set an order in which resources will be assigned. During the latter phases we need to have tools and mechanisms to monitor and measure progress, report and archive so we can react to situations when something goes not as planned and we have materials for further usage (future planning, etc.).

Conclusion (tips for manager)
There are several hints for IT managers that can represent the whole idea behind a successfull project management:

you must create and then maintain a plan in advance for an adequate long time
you need to be clear about deadlines and 3rd party companies obligations
your employees need always to know what to do in the next few days/weeks so they can plan their time effectively
you need to pay attention to the effectivity of your employees and create a good working environment
you need to have everything well organized
you need to keep benchmarking the progress and keep the "economic" aspect in mind at all times
you need to understand the system, its complexity and be capable of making decisions when the client pushes you to a commitment of extent/deadlines
you have to keep in touch with the client, attend all meetings, keep detailed records of them and various discussions, hand over protocols, etc.
you need to keep an overview and index of all problems, potential risks and of course a proposal on their solution/elimination
most of all you must pay attention to whatever happens in the project and keep track of the extent of the project

Articles
SAFE Project Handbook

Checklists and templates
CxCheck_MostCommonSchedRisks.txt
CxCheck_CompleteListOfSchedRisks.txt
PDD.doc
CxTemp_top10risks.doc
Risk_mgmt_plan_template.doc
Status_Report.doc
Status_Report_simple.doc

Software engineering 10 - Maintenance

2010-01-09T23:15:00.000+01:00

Introduction

Every software system an IT development company delivers is delivered in a state that should resemble the extent given by the contract the client and the supplier made (usually based on the requirements or specification). The system is then accepted by the client and being run in production fulfilling it's job. It is possible to assume that a product in regular usage contains only a small number of additional defects that were not yet discovered or unrepaired. Nevertheless applications created to order don't last the same their entire lifetime and new features are being desired. Small changes, errors that show themselves only after several months in production or when situation changes (more users/data to handle, changes in enterprise business process, etc.) lead to the need of the softwares evolution. Most of the times the "guys" that perform such actions are the developers of the original system for they possess the required knowledge base.

New releases

It is common that the company developing a product keeps involved even after the delivery of the final product by offering "maintenance services" to the product. Both parties, the developing company and the customer, benefit from such an arrangement for the customer can have changes done quite fast and the IT company has jobs for a small percent of its staff and money for low effort since they have everything ready (the environment, all the documents, knowledge, etc.). The most common scenario is that the IT company keeps continuously improving the product making regular releases of new versions (new features, requested changes) and some minor releases for requests of critical importance (bug/defect fixes, security vulnerabilities).

ISO/IEC 14764 specifies 4 main types of maintenance:

Corrective (repairing found defects and bugs)
Adaptive (keep the software updated with the current environment/infrastructure)
Perfective (improve maintainability or performance)
Preventive (detection and repair of latent defects before they become serious defects)

Maintenance as system development life-cycle

Maintenance of a software product is very similar to the overall software development process with all its typical phases but on a minor scope. Every new request that is to be implemented can be looked at as an individual software project with the rest of the system being the platform where we want our new product to be integrated. It's simple as that and the same techniques and facts we mentioned in the previous articles apply. All the new requests need to be documented, analyzed, solutions designed, implemented and tested.

Maintaining 3rd party product

Sometimes an IT company is asked or obliged to maintain a product that is not of their making. Such situations usually come when you're hired instead of an another company that was a supplier for several years and your projects are tightly coupled with the current infrastructure (some companies are only specialized on maintenance but in some cases creating a whole new product might result more effective than maintaining the existing ones).

In such cases it is essential to have:

a complete specification to estimate the extent of the work that is being asked from you
architecture and design documents which you can follow to keep a unified state of code/functionality
clearly defined practices and processes to follow

Those are the minimum requirements without which you should really think about rebuilding everything from scratch otherwise you might spend horrendous amount of time and money by simply trying to understand the current system you're maintaining. Also be careful for the changes you're implementing as part of maintenance tend to silently break the architecture piece by piece.

Conclusion
Maintenance is a very effective process with quite exact estimates that client and developer can benefit from if the one providing maintenance is the same subject that created the maintained product. On the other hand for an external subject maintenance can become a nightmare if the product developers didn't follow the commonly used practices of software engineering (documentation, manageability of the whole project and all it's phases, etc.). While maintaining a product the focus is on quality and performance of all the various maintenance aspects and it's hard to keep it all organized. Also many people are not keen on maintenance since it doesn't involve too much creativity for the changes done to the product tend to be of more cosmetic nature. That might be the reasons why maintenance leads to a somehow slow destruction of all the nice software engineering practices that were done while the main development as many people to "delete what they don't understand", get fed up with the project or simply don't feel enthusiastic about it. Believe it or not but after several years of maintenance software usually looks completely different (even it's internals) that the original specified product. And worst of all the documentation of the changes rarely last more than a year.

Software engineering 09 - Development environment

2010-01-09T16:54:00.000+01:00

Introduction
Every software project that runs through the various phases of development requires a environment in which the development takes place. A well setup environment is one of the greatest factors that helps the project to success. As we could see on the LCO/LCA/IOC diagram environment setup is done at the very beginning of the project for it's the logical place where to start.

The basic objective of the development environment is to provide a platform where analysis and development can take part and which models the client's infrastructure where our final product will live for the rest of its lifetime. One might think that having one environment for development purposes is more than enough but they would be mistaken.

Motivation

The base motivation behind spending costs on establishing an development environment is the fact that the software product we're about to create will be used on a specific client's platform and will be integrated into his infrastructure. Usually simply developing at client's place is not an option for various reasons so we need to create a similar environment of our own where we'll be developing and testing the product. Also we need a place where we can present the final product to the client and where acceptance tests will take part before it goes to production. Having a single environment is not ideal for it can usually be used for only one phase (either development or testing, etc.) so the solution is obvious - create several environments with tasks attached so they server their purpose only. Some simple environments we might want to setup are:

development environment (for development purposes, coding, prototyping)
integration environment (simulates client's infrastructure and helps in testing the integration of the product with other technologies/services)
testing environment (benchmarking, automated tests, regular builds and continual integration)
(pre)acceptance environment (acceptance testing, presenting to client/users)
production environment (final stage where we test the product in real action)

There is one part to the various environments people tend to forget and that's the social aspect of it. Part of every environment is having a comfortable working environment (furniture, privacy, space, good collective, team-building, etc.) and quality tools (hardware, literature, libraries, sets of 3rd party tools, automatization, etc.).

Continuous integration
Continuous integration is a nice feature to have in your development environment for it saves quite a lot of time by automatically running builds of the product and test scenarios on them. Otherwise such actions need to be performed manually which costs your programmers/testers valuable time. The basic actions of continuous integration are building the product, running automated tests and scenarios and publishing the results. The results need to be of course revised by a human being but by means of automatic warnings on errors/bugs/non-standard situations the time required is minimal.

We should note that the environment should also have mechanisms for every possible/relevant configuration units, should support ways of creating backups and their restoring, maintain an index of all the software backups, should be able to provide smaller or bigger delivers of the product in development, define processes for installation, modification and accessing and ideally support techniques for auditing.

The importance of backuping can never be stressed enough for whenever shared data, repository, hardware or people get corrupt you can always rebuild the whole infrastructure from the backups in an economic way.

Build process

When establishing the build process for your software you should make sure he has the following capabilities for it's the regular builds you'll use to create a supply for your client and presentations (no non-IT guy will be pleased to know the installation requires several hours of shell configurations after which a BSOD appears):

create a supply (choose the finished and stable parts with potentially not all features that can work together on the targets environment)
install the supply (make sure it sets up all the necessary parts so that the system works)
create a supply for user/customer's installation (prepare the product in such way the client is able to install it)
deploy a complete product
repair a small bug/defect in a fast and economic way (sometimes "be bulletproof" meaning you needn't redeploy everything due to a small defect but rather be able to "patch" the already running system)
ability to work with different platforms and configurations (application servers, operating systems, database system, etc.)

Conclusion

When developing a product for a client you should try the best you can to create an environment identical to the one your application will be running on. That way it's easier to test the product and also avoid unexpected defects caused by synchronization, resources or simply human factors. A daily build is nice to have for it helps monitor progress and testing. The supply system should be quite simple and as automatic as possible for it's nice to be able get everything running by 1 simple click. Finally backuping, logging and automatic reporting will save you quite some time when totally unexpected situations happen. Usage of advanced tools (Ant, CruiseControl, Hudson) should be part of your daily developers/IT managements life.

Software engineering 08 - Configuration management

2010-01-09T00:26:00.000+01:00

Introduction
The topic of "configuration management" is closely related to SCM ("supply chain management") and its main goal is to systematically manage the configuration of a software product. The attempt of "configuration management" is to provide order and some rules that can ease coping with additional changes that might be required during various periods of the project lifetime and were not foreseen. One typical case is when clients realize that the software should have additional features or that it doesn't satisfy their needs. In such cases when changes in specification, design and implementation need to happen it is essential for both time, money and customers satisfaction to have a clearly defined process and means to perform such changes. A simple set of problems "configuration management" needs to handle is:

ensure audit of all part of the system or software product
ensure identification of all individual parts as well as the whole system
make sure that changes don't break the product itself
ensure a possibility of determining the state (past/present) of the products configuration

As for almost every aspect of software engineering there is also an ISO (ISO 9000-3:1991) norm that describes the requirements in a more detailed and formal way:

audit and identification of every single software item
audit and identification of the software product
provide data to generate and update every single version of the software product
ensure correctness of making changes to the product
provide report about the state of the configuration
updates of a given software item by more than 1 person

Put in simple words what "configuration management" needs to provide is versioning of the product and a clean transparent process of making changes to it.

Versioning

The basic way of assuring that we keep track of various changes to the software product is by keeping all the versions by some sort of backup after every change with a documentation describing the changes made. Of course backuping the whole system every time a change is made would be a very naive way of doing things so we need to decide which items of the software product we'll be version controlled and which not. Also we need to decide with those unversioned for we always require a way of knowing how did they change during time. The identification of the parts and the system is simply done by a timestamp or version ID every time a change is made. To avoid too much versions which would prove completely unmanageable we create a new version after a set of changes takes part.

Having various versions of a software or its parts is time-saving for work can be done on different versions of the product at the same time without colliding. Also having the possibility of rollback to a concrete version has its benefits for test-driven programming has proven that sometimes it's way easier to restore a backuped version and re-implement from scratch the new functionality than having to debug it. Another great advantage of version control systems usage is that they were all created in such a way they allow better cooperation of people on the project with some techniques of minimizing or solving the typical conflicts of "2 people working on the same thing".

Today versioning is easy-to-do for there are many well-tested version control systems that offer great functionality and are easy to use. Their usage has become an essential part of the development process and they are well integrated with other project management tools to help increase manageability and performance. One of the older ones were RCS (Revision control system) and CVS (Concurrent version system) which are nowadays considered a little bit obsolete. Still used among various mostly open-source projects is the popular SVN (Subversion) and some modern ones are Mercurial and GIT.

Although version control systems have lots of common they can be separated into 2 basic categories based on the type of "repository" they work:

centralized
distributed

The centralized version control systems have a central repository where everything is stored. Each user has it's own local copy of the repository's content where he makes all the required changes and these are then uploaded to the centralized repository to the main branch where of course conflicts need to be resolved. These systems place emphasis on synchronization, tracking and backups. Whenever there is the need of trying something completely different the only way of doing sow without affecting the main branch is to create a separate single one but this is quite unpleasant for the longer the side-branch lives the more conflicts emerge and merging becomes very difficult to do.

On the other hand the distributed version control systems are composed of various repositories owned by each user and sharing is done based on the principle of "web of trust". Changes are made by user to his own repository and others decide whose changes they want to download (the download of a change and its application are 2 separate operations). Nevertheless there is usually a central repository that gathers changes from other repositories. The emphasis here is on changes sharing.

Management of SCR (system change requests)

Today there are no much project where the classic waterfall model of software development can be used for changes are an integral part of the process itself. It is clear then that we require some techniques and tools to handle the incoming changes in an elegant way. System change requests are one constant in the development and as such there should be absolutely clear what it's lifetime is. A simple illustration can be seen on the next picture.

At the very beginning there needs to be a need for change for making changes out of thin air makes no sense at all. The need for change needs to be described and of course identified so we can keep track of it. The request should contain an unique identifier, detailed description ideally with connection to specification and connection to the version in the version control system for every change is related to some. The request is analyzed and reviewed. Based on the evaluation it is either rejected in which case the requester should be informed or is approved. Approved requests are assigned to a software engineer, scheduled, implemented, tested and finally reviewed to make sure we satisfied the initial request.

For management of SCRs there are specialized ticket-based tools like Bugzilla, Trac, JIRA, etc. that are well established and commonly used which helps incorporate such feature into your software project management. Thanks to these systems you can not only benefit from easy management but it helps show statistics and reports of the amount of changes, their approval/rejection, etc. which can be used as one metric of determining the quality and overall management/development performance as well as it can be used for future analyses.

Conclusion

Never underestimate the power of a clean and understandable SCM process. Version control of various software items as well as of the whole product will help you not only while development but also when you need to deploy the software to varying environments. Having the possibility of branching your product at development time is crucial for saving time while trying new ideas and merging things back together or reverting makes it extremely easy to manage even big projects. But as with anything remember the tools and techniques need to be used properly, otherwise they may cause more harm than benefits.

Articles

Little Book of Configuration Management

NASA Software Configuration Management Guidebook

Guide to software configuration management

Templates

NASA_SwCMPlanTemplate.doc

Software engineering 07 - Quality assurance

2010-01-08T17:51:00.000+01:00

What's QA ?

Quality assurance is a nice term in software engineering that many people use to impress but not all of them understand "what's under the hood" of it. Some confuse it with "quality", others suppose it's a synonym to "testing" or "validation and verification". Let's explore these terms a little.

"Quality" is a typical buzz-word you'll hear on almost every management meeting and nobody ever takes some time thinking about it's background. People usually tend to take this as granted without thinking about what quality really means for that specific project. If we have a look at an official definition from ISO 8402-1986 it is probable that we'll end up more confused than before. Here's the definition:

"QUALITY: The totality of features and characteristics of a product or service that bear on its ability to meet stated or implied needs."

Simply stated it says "to have a satisfied customer".

"Validation and verification" is a complex process or possibly a set of activities which's goal is to determine if a specific artifact or system component fulfills all the demands. Validation is usually related to specification as it's main purpose is to compare the specification and final product if they really match. Verification is a more detailed process that makes sure things have been done exactly the way they should have. Simply stated validation is "building the right thing" and verification "building it right". The relation of V&V to quality is simple for V&V is a method of determining and measuring quality.

"Testing" was discussed in one of the previous articles so we should already a quite idea about how it fits into this whole "what is quality assurance problem". It is simply one of many ways of doing V&V. The results of testing are a clean metric about the quality rate of the product.

So "quality assurance" is a set of activities which's goal is to ensure the quality of a product or service by systematic and trustworthy means. As stated by a known art critic John Rushkin: "Quality is never an accident; it's always the result of intelligent effort." Although it would be nice to be able to assure 100% functional and reliable product it is impossible to achieve. What we can do is by means of "quality assurance" make this somehow more probable.

Why bother with quality ?
Simple answer: "Because quality matters." Quality assurance brings several important benefits to every project:

it's is economically effective (more than 50% of extra costs come from the fact of creating a bad-quality product and to repair defects found by customers)
it retains customers and improves profit
it's an advantage against your competitors
it is essential for your business to last
it improves your reputation even world-wide which helps establish yourself on international markets
it's "classy"

A nice scheme that presents the benefits of quality assurance is the "Deming's chain reaction" scheme which puts into relation the quality, productivity, business success and new jobs providing.

Achieving quality

To achieve quality the first thing you need to do is create and initiate an overall "quality program" which should define the policy of quality, establish a department responsible for quality assurance and of course have the management's blessing by showing quality is what they support and require. Then it's essential to create a plan (sometimes called "quality framework") to help the quality achievements. This plan need to take into account both cultural and technical aspects of assuring quality. It should count with:

a clean metric to measure achievements (of people, software, process, etc.)
establishing what is good/wrong/acceptable
what should we focus on
how to proceed
what should we require

There are many standardized "quality assurance frameworks" that will help you but if you're not a big company then the investments into purchasing the framework might not prove that useful and we're not mentioning how overcomplicated some of them are (as many standards are). As a easy-to-follow model you can simply use the classical already mentioned "V-model".

After every quality assurance plan and gaining its results there is the critically important part of analyzing all the results and behaving adequately. It is essential to be self-critic, regularly check and recheck the project, run quality audits and of course keep revising even the quality assurance plans for if we pass a bad plan it tells us nothing about the quality of the product or process.

Conclusion

Quality assurance is something that distinguishes amateur projects form real software engineering. It's main requirement is that it needs to be well planned in advance. It's nothing that will just miraculously appear out of thin air by simply testing your product. The whole process needs to be as pragmatic as possible and we need to think about quality on every level of the process from individuals to the organization as a whole. The most important part is to keep checking everything again and again which is usually the only viable way of assuring the quality of the product or process.

Articles

ESA Software Engineering Standards - Issue 2

NASA Software Assurance Guidebook

Software Quality Assurance Process

NASA Software Formal Inspections Guidebook

Overcoming the NAH Syndrome for Inspection Deployment

Templates

Software Quality Assurance Plan Template

Software engineering 06 - Documentation

2010-01-08T13:52:00.000+01:00

Importance

Until now in the previous "software engineering" articles I tried to stress the relevance of many documents that should accompany every software project. We talked about specification, use cases, feature lists, design and system architecture documents, even the documentation of the source code. But why are these so important for the process of software creation ?

The basic answer is "People tend to forget, documents don't !". Whenever you write something down and keep it somewhere where it can be easily found when needed of course then it should never happen that you are uncertain about something you discussed earlier. Every solution, every decision that have been made are archived and no one can deny that. For some this might seem a problem for this fact applies to both sides, the developers as well as the clients and managers and to be honest sometimes people would like to leave some stuff hidden. Anyway in an ideal world that should never be the case.

The documentation is not only to be some sort of "memory" that is archived and mostly never looked at again but it's main advantage is that it helps in communication among various stakeholders. You needn't spend extent time explaining what it is you have in mind but you simply point your finger onto a paragraph in the document an everything is clear as sunshine. The problem is given, the context is known.

Of course the "archive" feature of documentation is handy for various reasons:

it is the basic source of information for developers, system administrators and users
it helps in maintaining the product (in 2 years there will be almost no-one remembering what reasons lead to the current solution, what communication protocols among objects were used, etc.)
it is essential for future planning, helps people learn from mistakes done in the past, is useful for time estimations

Classification

The documentation of the software engineering process can be divided into 2 basic categories based on the topics and aspects of the development process they cover:

Process documentation
Product documentation

The "process documentation" describes the whole process of development and maintenance of the software product. It should include plans, time schedules, milestone and task estimates, quality assurance documents, standards and best practices, all the important meetings and decisions, reports and many many more. To illustrate why are such documents useful imagine you are creating a time schedule with estimates for a new software product. If you've done a similar one in the past it is very likely that the time estimates will be quite similar hence it gives you a nice place where to start from. Reports can be used to determine if the estimations made at the beginning of the project matched the reality and the deadlines were met, the tasks difficulty was guessed properly, etc. On the other hand some of the process documentation is relevant only during the development and can be discarded after the project is finished (for example working papers meant for communication support between the management and developers like presentations, specific prototypes, etc.).

Product documentation is more user/developer related for its main purpose is to server as a manual of the system. There should be several types of product documentation based on the knowledge base of the potential reader. Also the techniques used should vary based on that. For unexperienced users the documentation should be as simple as possible with lots of screenshots, detailed examples of usage and scenarios that help them understand the most basic features of the product. Experienced users on the other hand usually tend to have a feature/function list with all their capabilities and usage description. We should also distinguish various documents based on the users role for an end-user should have a different documentation (usage manual, feature list) than an administrator (installation and maintenance guide) or developer (API description, design description, communication protocols).

A typical user documentation is composed of:

introductory manual
functional description
system reference manual
system installation document
system administrators' manual

A typical system documentation is composed of:

system requirements and description of their reason
specification
architecture and design
source code and comments
validation documents (test plans, test scenarios, reports, etc.)
maintenance documents (known bugs, infrastructure requirements and dependence, influence on rest of the system, etc)

Requirements and form

The basic requirement for documentation is keeping its consistency during the whole lifetime of a software product for it's one of the basic representations of the system. Although various documents describe the system from various points of view it is essential to keep them all in consistent state (design should reflect requirements, code should reflect design, tests should reflect requirements, etc.) otherwise they wont play their role right and on large software products this can have a significantly deteriorating effect.

The documentation tends to grow off limits so using a good management policies is key to success. To help keep the management under control consider the following:

use an unified and ideally standardized template system
unified language (same naming conventions, names for modules/objects, etc.)
logical document structure
a clearly defined place and means of storing documents
use automatic generation tools (in-code documentation extractors (Doxygen), full-text indexing tools, etc.)
create a "navigation central point" (possibly web page, CMS or a wiki-like system)

As for the documents structure it doesn't really matter which technique or tool you'll use (MS Office, PDF, UML diagrams, emails, issue tracking systems, database tables, etc.). What matters is to create a basic set of rules at the beginning of the development process and them enforce their following. A typical document structure depends on the content but here are some basic stuff that you should include into your documents:

project identification
document identification
author, certifier
document type
version, version history and changes description
distribution list
confidentiality level
abstract and keywords
copyright
list of notions and abbreviations
table of contents
index

Conclusion

Having documentation costs time and money but it's benefits outweigh. Another thing is that it is not sufficient to have documentation but it's also essential to have it well organized so whenever you need to find something you know exactly where to look. Practice shows that in a lot of IT companies documentation tended to be something of very low importance but since we're living in times when products change quite often knowing what is it we've done months ago proves to be more than effective from both, time and money, points of view.

Articles

NASA Software Documentation Standard

Software engineering 05 - Testing

2010-01-07T19:54:00.001+01:00

Introduction

Software testing is a complex set of actions with one main goal which is establishing a belief that the software does only things it is meant to do. This is not it's only function but the most important one. Testing is of course also used for evaluating the performance of the software, its functionality, helps in verification of the system if it achieves the requested or at least acceptable results, etc. It is also essential for code analysis with the attempt to find all weak or buggy spots, potential future problems of bottlenecks. Also helps sell the product for a software that is proven to be 99.9% tested is more likely to be sold than the one without testing.

The importance and reason for testing has changed dramatically during time for in 1960s tests we used mostly for demonstration purposes. Their goal was to show the client what the software is capable of. Soon developers realized (possibly because some of the demonstrations didn't end as expected - Bill Gates' BSOD) that testing can provide them with a way of finding defects in the software that were later debugged. Since 1990s the usage of testing is mostly prevention for as mentioned several times the earlier we discover a defect in the software the less money it costs to repair it. Although a defect is not always that critical it needs to be resolved.

Testing is a complex process that is not related only to code but also to all the aspects of software creation. Design assumptions and interconnection of objects needs to be tested as well as the GUI design/prototypes, functionality and specification (by reviews to verify we are really doing what the client asks).

Testing

Every single test should have one single objective that should be as specific as possible. In case we need to test 2 or more similar objectives then it is usually better to create individual tests or a parametrized test. Complex tests can be composed of several simpler ones by creating advanced test plans and scenarios.

The testing of a software is somehow tricky because to test a software in a way to assure its quality we usually require loads of tests and some advanced management is required. For this purposes we should always have a "test plan" that describes which parts of the system will/won't be tested, how will the tests be conducted (what tools and methods will be used), who is going to be in charge for the tests (setting up responsibilities). Each test consists of 3 basic parts:

execution (prepare the software for testing, setup context, etc.)
testing (run the test)
evaluation (gather statistics and inform about the success/failure of the test)

Every test or a set of tests should be associated to a "test case" which describes the conditions that determine if the test passed or failed. This is quite critical for every test should attempt to cover all the possible contexts and act accordingly. This is a common place for mistakes for every assumption about an impossible case that certainly will never happen needs to be based on some invariants that usually come from the design and implementation of the overall system. So a tip is to "never make assumptions about anything and if we want to do them make damn sure they are right and wont change in time".

Basic principles

When testing a software we should try to keep in mind the following simple facts:

there is no such thing as "complete testing"
to create a test requires experience and creativity
testing is driven by potential risks (more important parts should be more tested)
tests are based on the analysis, design and specification
tests require time, money and motivation
tests should be prepared and run at the right time
keeping track of statistics and code coverage is as important as testing itself

The tests and their creation can be divided into 2 simple groups based on the knowledge base of the subject in charge of the creation. On one hand we have tests created by people that understand the software and are quite familiar with the implementation and how thing really work inside. This is called "white-box testing" and is usually done by developers themselves at the time of the implementation as part of the debugging process just to test if the piece of code they just wrote does what is supposed to do. The other approach is called "black-box testing" and logically it's the exact opposite. The test designer knows nothing about the code internals and all he knows is the interface and what it's functionality ought to be based on the specifications or application design. This sort of testing is usually done by professional testers, users or by other developers and is better in a way that the tester can't make any assumptions based on the implementation. One great benefit over "white-box testing" is that implementation changes don't break the test if the interface remains the same which is a useful feature.

As mentioned in the facts about testing one crucial aspect is the exact timing of tests meaning good time planning is essential for the tests to play they part in the overall process so we can benefit from these. Simply said if we start testing late then there's quite a big chance we won't test the software properly mostly due to time pressure because deadlines need to be fulfilled. On the other hand if we start testing too early there is a possibility we'll be testing an unfinished and unstable piece of software which would result in a waste of time, money and testers work. To know when to test what part is some sort of magic and a sign of a good and experienced IT manager but a nice hint is the "V-model".

To avoid problems with test planning we need to remember that tests are created and performed by humans which means that no tests can be absolute, no good testing can be done without adequate resources. Good testing requires excellent cooperation between testers, developers and most of all management. A common mistake is the belief that testers are the ones responsible for the quality of the product. This is not really true for they are usually under pressure from the management that schedules test runs and deliveries without taking into account the real-life project limitations. Also lots of companies don't really invest into qualifying testers or they rate their performance by the number of "bugs" they found which leads to their focus on small and not-critical problems just to earn enough points. Another bad testing practice is limiting tests to a specific area of interest (specification based testing, function testing, etc.) or by exactly specifying which atomic parts to test and how leaving little space for testers creativity. Over-detailed tests cause lots of trouble too for they are hard to maintain and usually redundant which leads to too long test runs.

Another thing that should be taken into consideration are the mentioned resources that management needs to provide for the testing to have effect. To create an adequate estimate is also a great problem and sometimes people tend to under or over estimate these. As an example there have been quite a lot of studies with similar results that say that the amount of users testing an application is related to the quality of the testing and that to have more than 5 users test your product is unnecessary for it doesn't provide such extra benefits.

Basic testing strategies

It is not easy to create a universal guideline for creating test plans and making thus testing effective independent on the type or extent of a project. This is probably the reason why are experienced IT managers so important for this is some sort of magic that requires lots of experience on various projects. Anyway there are some tips that might be a nice to follow for starters:

Begin with the basic tests. Test with values you know shouldn't break anything just to verify everything does what it should. Most software products work for quite some time without breaking anything and Windows users are used to restart the service from time to time. Anyway you can blame Windows if thats your softwares main platform.
Test slightly everything first. You should always run some preliminary tests first to make sure the whole system works as expected most of the time. If it does there is just a slim chance there's something wrong with the design which are the worst problems to solve.
Run complex and detailed tests as soon as possible. If the software passed all the basic tests it's the right time to start testing in a more aggressive fashion. Create detailed tests based on the simple ones, create complex testing scenarios with contexts that are not so likely to happen. This usually discovers small implementation mistakes, bugs and possible weak points of your software.
Modify test boundaries. After some time the amount of tests is more than overwhelming and their execution takes quite a lot of time. Also there probably is quite a lot of redundance for many complex tests do the same job as the a set of simpler ones. The elimination of unnecessary tests will improve the situation significantly.
Exploratory testing. The most useful testing advice is to keep creating new tests all the time. A nice practice is let every developer create a new test or a set of tests on weekly basis. That way we can discover forgotten untested places.

Reporting

Testing would be almost useless without reports that record the test results as well as all the defects software has. These reports should have a standardized form which can be easily achieved by the usage of a bug-tracking system (BugZilla, Trac). The basic and simple rule is that every issue should be reported and ideally traceable by connecting it to the version of code currently tested. The issue needs to be reported in such a way that enables others to reconstruct and simulate it and most of all repair it. This can be achieved by appending logs, scenarios, detailed results, etc. Also a note describing it's relevance and effect it has on the system is advisable.

Automating tests and regression testing

Ever since testing became an integral part of software development people though of ways how to make the testing process automatic for several reasons. First people needn't to run the tests by hand every time something changed, tests could be reused and of course test runs could be planned in advance for times when the system was not overused. Today regression tests (running old test scenarios on new versions of code) is one of the basic ways to determine new changes are consistent with the rest of the system. This leads to the principles of "continuous integration" (smoke testing), configuration testing on various software/hardware, stress and load testing, etc. Some specialized tools can be used (for example Hudson) to do the job.

Conclusion
Testing helps create a good and high-quality product if the resources are provided. Many people think of testing as an easy click-and-see job for they don't see the experience and creativity required. Also it is important to attempt to have a high "test code coverage" (some norms require a percentage) but we need to keep in ming there is no such thing as 100% test coverage for we may never know what can happen and we usually don't test the platform and tools we use. Anyway even the testing software we use might be buggy!

Articles

What Is Software Testing? And Why Is It So Hard?

Little Book of Testing - Volume I

Little Book of Testing - Volume II

Software Test Planning and Management Guide

Checklists and templates

CxCheck_TestPlan.pdf

CxTemp_TestPlan.doc

IEEE_TestPlanTemplate.pdf

MIL-STD-498_SwTestDescriptionTemplate.doc

MIL-STD-498_SwTestPlanTemplate.doc

MIL-STD-498_SwTestReportTemplate.doc

SPAWAR_SwTestPlanTemplate.doc

SPAWAR_SwTestReportTemplate.doc

Software engineering 04 - Construction

2010-01-07T17:26:00.000+01:00

Introduction
Finally we got to the part that for most of the managers and clients is what IT is supposed to do which is the construction of the software product they are paying for. The development process can live without detailed specifications, designs and advanced analysis but without several hundred/thousand lines of working code no software can be produced - it's simple as that. This is probably the reason why this phase is the place where the majority of the investments and costs is located. Some of the modern agile methodologies of software development, for example the well-known "extreme programming", believe that most of the design and analysis can be done while implementing the system (which is the parallel development principle we described in the previous articles) and that is what is usually done nowadays, although it definitely depends on the system we are creating for a complex software with reliability as its main concern will require more thinking than a simple web presentation.

Lines of code as main metric
It is always an issue to determine the metrics to monitor the progress of a software construction. The most common one is the number of lines of code a person/team has done in a period of time. Of course this is relative for the quality matters a lot but by "pair programming" or "collective code ownership" practices we can diminish bad code and hence use this simple metric as a nice overview. But there is one basic thing we need to keep in mind that is related to the estimates we proposed in the specification/design phases for it is proven that the amount of new code decreases in a logarithmic fashion as can be seen on the following picture.

Similar statistics follow almost every project. In the first phase of the development it's an explosion of new code for we start (usually) from scratch and when using modern programming languages about a 50% portion of code is of supportive/platform dependent nature. After some time we have the working corpse of our new system and we start making adjustments, modifications, add-ons and debugging which takes lots of time but on small portions of code. Keeping this in mind is crucial for a lot of people believe that if at the beginning of the project the rate of new code is 1000 lines/week then if this number falls dramatically it is a signal that overall workers performance is diminishing.

Implementing rules
The implementation part as being the phase where everything is physically made needs to be effective, quite fast and most of all high-quality. Clean and understandable code might even fulfill the same job as the design documents and architecture description. It's the heart of the system and if done badly there's nothing that can save the project, except for new reimplementation based on design of course. A complete and comprehensive list of methods, tips and guidelines can be found in Steve Mcconnells book "Code Complete" which should be a bible for every software developer, but here are some of the basic rules you might want to follow:

start coding only when it's absolutely clear "what to do" and "how"
always follow the architecture and design
use "the best tool available" that can handle the job
use "the best solutions available" (frameworks, libraries, design patterns, best common practices, 3rd party software)
know the basic concepts of software construction
write tests to verify your code
use syntax checkers, static code analysis tools to find bugs and mistakes in advance
maintain the code by versioning, documentation
document "WHY" and not "WHAT" for that should be clear from the context
at all costs avoid quick and simple partial solutions with the intention to repair them later for you'll probably never will ("TODO,FIXME" comments are almost always left untouched due to time pressure)
understand the "business" behind the software for if you don't share the client's point of view you have almost no chance of satisfying his needs

Model/Domain/Test driven development

There are several basic approaches to software construction which differ mainly by their point of view on the overall construction process and the units that the system is composed of. The more traditional one is the "model driven development" while the new and sort of still experimental ones are "domain driven development" and "test driven development".

The "model driven development" suggests creating abstractions that model concrete real concepts than the computer/algorithmic ones meaning the models represent a real-life object/service/situation more than some computer specific issues/techniques. That way it's easier to design them and see the various interactions among each module and somehow hides the technical stuff under a nice cover of abstraction. This approach also encourages reusability of already implemented models and their composition into new products. This leads to money saves in the future as some commonly used real-life models are implemented once and then reused for later projects. To benefit from such development we clearly need to have the models implemented as independently as possible and as specific as the real-life stuff they model.

Similar but a little bit different is the approach of "domain driven development" which is based on 2 basic premises:

complex domain designs should be based on a model
for most software projects, the primary focus should be on the domain and domain logic (as opposed to the particular technology used to implement the system)

The slight difference here is that we don't focus on individual independent models that we later combine into the final product but we create models related to the problem specifically. In simple words it's about mapping business domain concepts into software artifacts. It's more of an addition and perfection of the plain model driven development and as such all the benefits stay.

One particularly interesting approach is the one of "test driven development" which looks at the implementation process in a different way. It's main idea is that developers should first create the tests for the small pieces of software they are creating thus preparing some quality expectations and then implement the piece while continuously testing each version against the tests. This approach has several benefits like the need of developers to think about the software as the user would, creates a well tested software and help assure the overall quality of the product. Also it takes advantage of the premise that small pieces of code are better tested and that debugging is more expensive than simply reverting to the previous working version. The increased costs for loads of tests, the main argument against, is somehow pointless because some tests we would have to do anyway. Nevertheless this approach is more useful for small projects or projects composed of many individual parts that needn't large complex full functional tests to determine success. Also note that many tests are created by developers themselves and they usually make some assumptions about the test context which might create untested holes - so don't believe that test driven development assures quality by itself.

Documentation, versioning and refactoring

When concerning the software construction there are several things developers ought to do. Apart from the typically administrative ones like setting up means and forms of communication, create code style rules and prepare tools to enforce them, set the basic rules for typical practices, establish naming conventions, etc., there are 3 basic matters we should always keep in mind.

Documentation is the basic and probably most useful one. Having good documentation helps reduce costs and most of all helps manage problem situations (change of assignee, bug detection/elimination, problem solving, logical errors, etc.). The paradigm of "self-documenting code" is the basics. Good clean code and understandable naming conventions are essential and with additional information from the design papers are developers strongest weapon when dealing with problems. There is no need to document every single step for the code itself serves as the main source of information but describing functionality, pre/post conditions and invariants of larger function blocks is never a bad thing.

Code versioning is also one of the techniques that helps solve possible problems as well as its benefit for cooperation among developers. It does not only document changes but allow a quick and simple mechanism to rollback to a specific version if for example changes in design need to be made rather than reimplementing from the actual state.

Least but not last is refactoring which can simply be described as code cleanup. It should be done after a block large enough or module has been completed and properly tested. It involves removal of unnecessary or dead blocks of code, some style standardizations, possibly some performance tuning and other more of aesthetic tasks that don't affect the external functional behavior.

Conclusion

Attempt to create a clean and well documented code that strictly follows the already prepared design. Don't implement design changes directly but make sure you modify the architecture design documents so that they reflect the implementation and vice versa. And most of all "Write code for people, not for the machines !".

Articles

"Best Practices" Columns by Steve McConnell

Checklists

CxCheck_Conditionals.txt

CxCheck_ConstructingRoutine.txt

CxCheck_ControlStructureIssues.txt

CxCheck_DataCreation.txt

CxCheck_DocumentingCode.txt

CxCheck_HighQualityModules.txt

CxCheck_HighQualityRoutines.txt

CxCheck_Layout.txt

CxCheck_Loops.txt

CxCheck_NamingData.txt

CxCheck_OrganizingCode.txt

CxCheck_UnusualControlStructures.txt

CxCheck_UsingData.txt

Software engineering 03 - Architecture and Design

2010-01-07T00:12:00.002+01:00

Introduction

The creation of the architecture and design is almost as important as other parts of a software engineering process. It's hard to believe that in many IT companies only a smart portion of time and money is assigned to architecture design. This aspect might come from the fact that the client usually thinks that from the specification and requirements it is absolutely clear what the design of the application should look like. The fact is that the specification tells only "what is it we want" and the design should provide us with a specific manual on "with what technical tools and principles we achieve that". This is essential for understanding for all the main stakeholders because some might think the architecture design is just a duplicate of the specification project so in simple words "doing the same thing we've already done".

It's not a standstill
The design takes approximately the same amount of time and money as does the specification and someone might think that all the team forces deal with this phase while no "concrete" work is done. As shown on the well-known diagram of the software creation this is not true.

The creation of the architecture and design requires a very good knowledge base, advanced software engineering and programming skills, overall IT knowledge and most of all experience. We usually have only some such people in the team and while this subset creates the design other work can be done simultaneously. Such actions could be:

learning the technologies that will be used for the implementation, special courses and training
getting to know the clients infrastructure
prepare the development platform for the implementation (version control system, test and production servers, software for additional functionality, etc.)
prototyping of the already done designs and verifying their correctness
start implementations of already designed parts

Basic concepts

Creating a software project design is one of the more complicated phases of the software creation process and it is not easy to give specific guidelines that would give instantaneous success. There are 5 basic design concepts that everyone should keep in mind when designing a modern application:

decomposition (split the application into small individual modules that have a clear set of responsibilities and are more or less logically independent on each another)
abstraction (attempt to identify common properties and functionality for some of the decomposed system parts)
encapsulation (make a difference between objects structure and behavior, protect internal parts from the outer elements)
cohesion ("high", the amount of responsibilities an independent module fulfills, we try to make it as highest as possible in which case the module can take care of all it's functionality by himself)
coupling ("low", metric to determine the dependency between various modules, we try to make it as low as possible in which case objects are more independent)

The concepts above can be easily achieved by the usage of some "modern" programming techniques like object-oriented programming. OOP became popular in the 90' and was considered the solution to all the common design, maintainability and reusability problems mostly thanks to the nowadays overestimated object inheritance. This concept was seen as the greatest benefit of OOP but during time became too abused and nowadays is considered more of bad practice for whenever we use complex inheritance schemes we limit our possibilities to make additional changes without the need to modify the architecture. Programmers recommend to use object composition instead and many frameworks attempt to encourage that. One great example from the Java world is the Spring Framework or EJB with its object injection.

Design patterns

When speaking about OOP we have to mention the concept of existing design patterns. These are universal platform independent patterns that are meant to solve many of the common design issues. The knowledge of some of the basic ones is essential for creating a good, reliable architecture. On the other hand the misuse can result in more and complex future problems or useless extra implementation. One example for all is the notorious "singleton" which is one of the worst possible design patterns but also one of the most used ones. It's main disadvantage is the fact that it somehow supplies the same functionality as a global variable, keeps it's state throughout the whole run of the application and creates quite tight coupling among objects.

Stakeholders and how they see it

There exist various points of view at the design process per stakeholder and what it should concern. If we divide the stakeholders into several categories we can identify some basic concerns:

customer

schedule and budget estimation
feasibility and risk management
requirements traceability
progress tracing

user

consistency with requirements and usage scenarios
future requirement growth accommodation
performance
reliability
interoperability

architect and system engineer

requirements traceability
support of tradeoff analyses
completeness
consistency of architecture

developer

sufficient detail for design
reference for selecting/assembling components
maintain interoperability with existing systems

maintainer

guidance on software modification
guidance on architecture evolution
maintain interoperability with existing systems

Another way of showing the various views on the design process is the "4+1 view model" which shows 4 basic views that are interconnected among themselves and by use-cases and scenarios. Logical view is end-users view and is mostly concerned with pure functionality. The development view is programmers view and its mostly concerned with the whole software management. The process view is integrators point of view and its main concern is scalability and performance. The last view is physical view which is the view of system engineers who are concerned with topology and communications.

Integration

One very important part of the design which is usually neglected is the important integration of the new product with the rest of the clients system in such a way these can cooperate and take advantage of each others features. There are many ways on how to implement such integration but one of the most common are file transfer, shared databases, remote procedure calls or messaging.

Documents

The whole design should be also well documented in an understandable form. The product of this phase should be a complete manual that can be handled to a team of coders and if followed it should give exactly the required final product. This document should also be verified and confirmed by the client. Of course some might object that by preparing such a detailed document there is the risk that the client will take our results and find some other subject to implement it. The only motivation for such an act might be the attempt to lower costs but real-life experience shows this usually comes at the cost of lower quality of the final product for the new team doesn't have the experience gained while gathering requirements, specifying the software and design creation.

Conclusion

The design is not something that should be taken likely as it can prevent re-implementations, gives the developers and coders an approximately exact overview about the software and can significantly reduce costs. Although it really depends on the scope of the project for creating detailed design and architecture for small projects might be a waste of time since they need to have some formal style. As for some good practices that we might want to keep in mind:

Program to interface, not implementation !
Favor object composition over inheritance.
Avoid using singletons.
Think twice, program once !
Keep it DRY, shy and tell the other guy !

Articles
An Introduction to Software Architecture
On the Criteria To Be Used in Decomposing Systems into Modules
Architectural Blueprints - The “4+1” View Model of SW Architecture

Checklists and templates
CxCheck_SwArchitecture.txt
CxCheck_HighLevelDesign.txt
CxCheck_HighQualityModules.txt
SwDesignSpec.doc
MIL-STD-498_InterfaceReqsSpecification.doc
MIL-STD-498_InterfaceDesignDescrption.doc
MIL-STD-498_SwDesignDescription.doc
MIL-STD-498_SysSubsysSpecification.doc
MIL-STD-498_SysSubsysDesignDescription.doc

Software engineering 02 - Requirements

2010-01-06T12:00:00.003+01:00

Why are they so important

Requirements are one of the basics of software engineering. There have been many projects that failed due to the bad understanding about what were the requirements for the software and many other cost much more for the simple reason of constant requirement modifications during development. To realize what the exact requirements are in the beginning of the project is essential for it is proven that changes to projects cost lots more in the later phases than in the early ones.

As can be seen on the graph if we have the requirements wrong then it is far cheaper to realize the defects soon for the changing a list of "what to do" on paper is quite cheap while doing such changes to a fully operational system will take several man-days to fulfill.

Overview

Requirements gathering is quite a complex process that most of all requires good social skills. It's not a "once-per-project" phase but it happens throughout the whole lifetime of a software on different levels of formality. The whole process can be divided into four categories of actions:

Elicitation (meetings, dealings, user observation, documents creation and marking ups, etc.)
Analysis (brainstorming, debates, group-thinking, etc.)
Specification (problems decomposition, documents creation, establishing processes, notations, etc.)
Verification (meetings, presentations, dealings about the extents of projects, etc.)

It usually follows the easy pattern of first meeting the client and debate about his idea of requirements. Everything should be recorded as it is later used among the developers for brainstorming and analysis. The results of these collective meetings is put onto paper in a standardized form and with some additional information is presented back to the client to determine his satisfaction.

Although this process seems easy and straightforward there are quite a lot of things that can go wrong. For many IT people the communication with a client is a nightmare mostly because the clients describe things from the users point of view without any technical specifications. IT people on the other hand tend to be too specific and overwhelm everyone with their technical knowledge. Where this leads is the feeling that both parties know their goals and feel satisfied after the meeting but in a couple of weeks realize the product doesn't look the what they intended.

To avoid this problem it is good to keep detailed records or summaries about every meeting and let all the parties confirm those for example by signatures. That way whenever a misunderstanding occurs both sides can avoid discussions about "whose fault it was". This leads to the need of being exact as possible.

Extent of requirements
There are many discussions among the various software engineers about what should be part of a projects requirements and how detailed the information needs to be to achieve the best results possible. The answer is pretty simple: "Requirements should cover all the obligations." In short the result of requirements gathering should be a comprehensive list of features and stuff we as developers are obliged to incorporate into the final product. But there is a small drawback caused by the different understanding of this list between the developers and the client. Time has proved that:

The client requests to have everything which we didn't explicitly reject
The developers provide only what is explicitly specified

These two points of view usually don't exactly overlap and there is some "grey zone" between them that can be a place for future problems. The goal of an appropriate requirements gathering using some standardized processes is to erase the difference between the clients and the developers view.

The whole picture

Requirements gathering is part of the initial phase of every project iteration for before we do something we need to know what is it we want. It is said that the requirements form about 10-30% of the overall project costs and time depending on the complexity of the project. Also as mentioned above there is a clear correlation between the time at which requirement changes occur and the costs it takes to implement them. In modern software engineering it is common to start working at architecture design and implementation while the requirements are not fully specified. This saves costs and most of all speeds up the overall development process if done correctly. Of course this applies only to parts of the system we suppose the client is satisfied with and which won't change dramatically while specifying the requirements.

The overall software process usually has 3 main milestones which are quite related to requirements gathering as can be seen on the graph above:

LCO (life cycle objective)
LCA (life cycle assessment)
IOC (initial operational capability)

Techniques

There are many ways of gathering requirements and the set of suitable techniques differs from project to project. These include specification documents, checklists, GUI prototyping, templates, UML, use cases, user stories, polls and questionnaires, etc. But there are some guidelines that if followed can make the requirements gathering efficient and maintainable.

use structured text for better orientation
make sure you communicate with the right stakeholder (client, person responsible)
keep in touch with the client as much as possible
don't let the form overshadow the content
keep the content complex but understandable
all requirements are essential, don't forget about some
remember that there is no "right" way of doing this
always learn from previous mistakes and don't let overconfidence cloud your mind

Articles & Books

When Telepathy Won’t Do: Requirements Engineering Key Practices

Karl Wiegers Describes 10 Requirements Traps to Avoid

Writing Effective Natural Language Requirements Specifications

Be Careful With “Use Cases”

Just Enough Structured Analysis

Checklists

CxCheck_Requirementst.txt

Requirements_review_checklist.doc

Use_case_checklist.doc

Impact_analysis_checklist.doc

Templates

srs_template.doc

use_case_template.doc

vision_and_scope_template.doc

SAFE_BusinessRequirements.doc

SAFE_SystemRequirements.doc

Software engineering 01 - Introduction

2010-01-05T22:11:00.004+01:00

I recently realized there is a deep hole in the knowledge of software engineering among programmers mostly due to the natural nature of these people to be sort of antisocial. Usually programmers tend to underestimate the customers needs and for those starting in the business it takes several falls to realize they have to change their habits a little bit to everyones satisfaction.

Also a lot of managers and people using IT services don't usually have a good understanding about the process required for creating a good and reliable product that will all the "buzz-word" features managers so much like (good design, great performance, fast & cheap to create, etc.).

In the next series of articles I would like to present all the main points that need (or should be) taken into consideration when creating a software product. The intention of the articles is to present the overall basics to IT and non-IT people from both views to help these two camps understand each others priorities, needs and problems.

I have spent a few years in a software company that sort of ignored many of the nowadays common software engineering practices mostly due to really bad communication between the IT department and the management so at some points I might not be that objective but I promise I'll try to keep those at a minimum. As for sources that helped me create these articles I mostly used courses from "The Faculty of Mathematics and Physics, Charles University in Prague" where I'm recently studying for my computer science degree ("Software engineering for real-life" by Tomáš Krátký, Profinit, "Best programming practices" by Lubomír Bulej and "Tools for development and monitoring of software" by Tomáš Kalibera). Another great sources are the wonderful book "Code Complete" by Steve Mcconnell, "The Art of Unix Programming" by Eric Steven Raymond and of course the ultimate sources "Wikipedia" and "Swebok".

In the series I would like to take a look into the following topics:

Requirements (the importance of knowing what the client wants us to do)
Software architecture (how should we think about the design)
Construction (lots of good code is not everything)
Testing (howto make sure we provide a "bulletproof" product)
Quality assurance (quality is what matters)
Configuration management (when clients realize they wanted more than they told us)
Development environment (development and production do really differ)
Maintenance (howto get quite a lot of money without too much work)
Project management (it's all about business)
Estimations and proposals (how do we know we are payed enough)
Improvements (what can we do to make it even nicer)

I will try to follow the "KISS" principle throughout the articles to make things as clear as possible but since I'm not a writer, more of a shy computer guy, feel free to append questions and other comments and I will try to answer them as soon as I'll have time to do so.

So, once upon a time ...

Spring Framework

2010-01-04T15:59:00.002+01:00

It has been some time since me and my coworkers thought about developing web applications in Java. It was no coincidence that at the same time we came across the Spring Framework which according to many users provides more efficient development, lowers costs and takes great advantage of code reusability. At that time we were uncertain about the claims for it was well known that SpringFramework is a great tool but in the wrong hands can deal quite a lot of damage.

We discussed various pros/cons of different tools for Java-based web applications, mostly JSF (Java Server Faces) and the famous SpringFramework (including Spring Web-Flow), and realized there are not so many things you can do that can't be done in the other. The amount of coding also didn't vary so much so where are these claimed Spring's benefits ?

As I can tell, its main power relies in the fact that this framework has been created with the "all-for-testing" paradigm in mind which leads to applicantions that are easily tested (mostly due to Spring's IOC which helps you create different test scenarios without the need to modify the source code).

There are a lot of objections against the usage of SpringFramework and its mostly up to the developer if he chooses to use Spring. Some people are frustraded by the blotted XML configurations which need some special tools to maintain on really big projects, others hate the IOC for itself because it removes one of Javas core features - strong typing (the type control is run at runtime).

My conclussion is as follows:

If you plan to create a quite small blog-like web application SpringFramework (and possibly Java) is too big and too complex for the job. You should use the framework on really big enterprise projects where you need excessive testing capabilities, when you will plan to reuse many of the components and modules that form your application and your development team has more than 1 developer. It's mostly a way of using EJB in a more reusable fashion with quite a long learning curve to take real advantage of whatever Spring Framework provides for your applications. One great advantage is that you are free to combine Spring with other technologies and use only the part you need.

Some additional information about SpringFramework can be found here:

Spring main page - http://www.springsource.org/

Spring tutorial - http://www.roseindia.net/spring/

IBM Spring tutorial - http://www.ibm.com/developerworks/web/library/wa-spring1/

Just another Blog on the horizon

2009-12-10T15:56:00.001+01:00

The situation about blogs is getting calmer that a year ago and I decided the time has come for me to join this "blogger" community. I'm not a writer so please don't expect any novels or heart-breaking stories. The aim of this blog is simply to store some interesting stuff that's in my mind and I wouldn't want to loose.

If by any chance you find something of some interest in any post please feel free to do whatever you want with it. I'm no copyright moron so this whole blog and its content should be following the WTFPL licence.

It is likely there will be more IT stuff than any other since I'm a computer sciences student at Charles University in Prague (Czech Republic) so take that in consideration if you came here in search of some crappy videos or jokes (although I'm not suggesting you won't find any here).

Ok, that's it ... the first entry.

howg