Industry Practices and Tools - Part 2

Code Quality

Code quality can translate into how useful and maintainable your code is: high-quality code can be re-used and re-developed; low-quality code doesn’t last.

Impact of Quality code

• Clarity
• Maintainable
• Documented
• Refactored
• Well-tested
• Extensible
• Efficiency

There are many aspects of code quality that are worth improving.

   The quality of the code can be measured by different aspects.

• Weighted Micro Function Points - One of the newer models(2009) which adjusts function points using weights derived from program flow complexity, operand and operator vocabulary, object usage, and algorithmic intricacy. 

• Halstead Complexity Measures -   Introduced by Maurice Howard Halstead in 1977 as part of his treatise on establishing an empirical science of software development. Halstead made the observation that metrics of the software should reflect the implementation or expression of algorithms in different languages, but be independent of their execution on a specific platform. These metrics are therefore computed statically from the code.

• Cyclomatic Complexity - This is a source code complexity measurement that is being correlated to a number of coding errors. It is calculated by developing a Control Flow Graph of the code that measures the number of linearly-independent paths through a program module.

Lower the Program's cyclomatic complexity, lower the risk to modify and easier to understand. It can be represented using the below formula:

Cyclomatic complexity = E - N + 2*P 
where,
  E = number of edges in the flow graph.
  N = number of nodes in the flow graph.
  P = number of nodes that have exit points

• Lines of code - Source lines of code (SLOC), also known as lines of code (LOC), is a software metric used to measure the size of a computer program by counting the number of lines in the text of the program's source code.

• Lines of code per method - It starts at the method or function level. In Code Complete, Steve McConnell says that the theoretical best maximum limit for a method or function is the number of lines that can fit on one screen (i.e., that a developer can see at one time).

Why do we want to measure software 

quality?

There are many reasons to why we want to measure quality. Some developers like to use metrics to identify areas of the code that could be improved. Others use metric as a quality gate; if the code doesn't satisfy a certain "quality" criteria, the build fails.

Managers may have different reasons to why they want to measure quality. Although they may say otherwise, the most common reasons to why they want software quality to be measured are:

• Lack of trust.
• Fear of being responsible for things they don't understand.
• Find other people to blame in case something goes wrong.
• Try to find a justification for the general dissatisfaction among the people building or using the software.
• Show they are in control.
• Cover their backside, showing that they are doing something to control quality (ticking a box).
• Annual targets and bonuses (normally associated to the percentage of test coverage).

With the exception of a few developers and academics, the people that are keener to find a way to measure software quality are the ones that don't write software on a daily basis. Many of them don't even understand how some of the metrics are calculated, but they want a way to control what they don't understand. How many times have we heard stories about managers who force their teams to keep test coverage above a certain percentage? Do they really know what it means? Would they be able to help the teams to achieve that? Can they distinguish good tests from bad tests?

When do we want to measure quality?

The desire to measure quality is intensified when things are not going well: people are unhappy; some team members are not suitable for the job; too many bugs; there's always problems with new releases; few releases per year; things are taking too long to be done; lack of trust; people are often wondering what others are doing and who is responsible for what. But when asked, almost no one can explain exactly where the software lacks quality or how it could be fixed.

On the other hand, we very rarely see people talking about measuring software quality when the project is going well: team is composed by good professionals; people are working well with each other; people respect and trust the opinion of their teammates; software is released often and with zero or few cosmetic bugs; good communication and team spirit;

Code metrics

Code analysis tools are great when used by developers as an aid to find areas of the system that could be improved. They are great to highlight things that are not always easily seen with a naked eye. They may also be very helpful when working with legacy code. However, the problem with code metrics is that making sure that the code compiles to specific metrics is very different from saying that the code has quality. Having high test coverage is not the same as having good tests. Having loose coupled and high cohesive code doesn't mean code that express the business domain or that is very easy to understand. Code with low cyclomatic complexity doesn't mean code that behaves according business specifications and is bug free.

Non-code related metrics

How long does it take to build a new feature? How hard is to deploy the application? How many bugs are found in production every time we go live? How often do we go live? How fast can we safely and confidently change the software to accommodate the ever changing business requirements? Those are all things that we could associate with code quality, but can they be really associated to it? Are all these things only related to the quality of the code? Let's take two common non-code related metrics as examples.

• Number of bugs: What type of bugs have been raised? Are they related to the bad state of the code base, the lack of skills of the developer that implemented the code, bad requirements, or a complete lack of communication between business, developers, and testers?
• Time to deliver a feature: Why are we unhappy about the time it took us to deliver a feature? Is is because of bad estimations? Is it because we were doing something that we haven't done before and we had to explore different alternatives? Were we blocked by other teams or internal bureaucracy? How does this feature compare to previous features implemented in the system? Are they similar? Completely different?

How to maintain the code quality?

• Documenting the code. Comments improves the readability and the understandability of the code.
• Regular Code Reviews. According to a recent survey, software professionals rank code reviews as the number one way to improve programming. These reviews enable developers to collaborate and share knowledge with each other, which improves their work. Furthermore, reviews
ensure that code adheres to established standards.
• Functional Testing. Functional testing is important because it encourages developers to focus on software functionality from the outset, reducing extraneous code. The aim of software development is to write an application
that delivers exactly what users need.
• Clear Requirements. Most software development projects begin with a requirements document, or story cards in agile development. A project with clear, feasible requirements is much more likely to achieve high quality than ambiguous, poorly specified requirements.

Code Review Tools

• SonarQube
• Sonar Source
• CSS LINT
• Sidekick
• Klockwork
• W3C(Markup validation Service)

Dependency / Package Management

A package manager or package management system is a collection of software tools that automates the process of installing, upgrading, configuring, and removing computer programs for a computer's operating system in a consistent manner.

A package manager deals with packages, distributions of software and data in archive files. Packages contain metadata, such as the software's name, description of its purpose, version number, vendor, checksum, and a list of dependencies necessary for the software to run properly. Upon installation, metadata is stored in a local package database. Package managers typically maintain a database of software dependencies and version information to prevent software mismatches and missing prerequisites. They work closely with software repositories, binary repository managers, and app stores.

Package managers are designed to eliminate the need for manual installs and updates. This can be particularly useful for large enterprises whose operating systems are based on Linux and other Unix-like systems, typically consisting of hundreds or even tens of thousands of distinct software packages.

Dependency Management Tools examples

• NPM -  Each project can use a package.json file setup through NPM and even managed with Gulp(on Node). Dependencies can be updated and optimized right from the terminal. And you can build new projects with dependency files and version numbers automatically pulled from the package.json file.NPM is valuable for more than just dependency management, and it’s practically a must-know tool for modern web development.

• Bower -The package management system Bower runs on NPM which seems a little redundant but there is a difference between the two, notably that NPM offers more features while Bower aims for a reduction in file size and load times for frontend dependencies

• RubyGems - RubyGems is a package manager for Ruby with a high popularity among web developers. The project is open source and inclusive of all free Ruby gems.
  To give a brief overview for beginners, a “gem” is just some code that runs on a Ruby environment. This can lead to programs like Bundler which manage gem versions and keep everything updated.
• RequireJS -  It can be used for loading JS modules quickly including Node modules.
  RequireJS can automatically detect required dependencies based on what you’re using so this might be akin to classic software programming in C/C++ where libraries are included with further libraries.

• Jam - This is a JavaScript package manager with automatic management similar to RequireJS.
  All your dependencies are pulled into a single JS file which lets you add and remove items quickly. Plus these can be updated in the browser regardless of other tools you’re using (like RequireJS).Libraries are updated based on the latest versions through the terminal. Each project can be created automatically with optimized components based on your needs. Jam is free on GitHub and worth a look if you have the time.

• Browserify
• Mantri
• Volo
• Ender
• Pip
• Composer

Build Tools

Build tools are programs that automate the creation of executable applications from source code.

Building incorporates compiling, linking and packaging the code into a usable or executable form. 

In small projects, developers will often manually invoke the build process. This is not practical for larger projects, where it is very hard to keep track of what needs to be built, in what sequence and what dependencies there are in the building process. Using an automation tool allows the build process to be more consistent.

Build automation

•On-demand automation such as a user running a script at the command line

•Scheduled automation such as a continuous integration server running a nightly build

•Triggered automation such as a continuous integration server running a build on every commit to a version-control system.

•A necessary pre-condition for continuous integration and continuous testing
•Accelerate the compile and link processing
•Eliminate redundant tasks
•Minimize "bad builds"
•Documentation has the history of builds and releases in order to investigate issues
•Save time and money, and improve product quality

Building incorporates compiling,linking and packaging the code into a usable or executable form. Basically build automation is the act of scripting or automating a wide variety of tasks that software developers do in their day-to-day activities like: Downloading dependencies. Compiling source code into binary code.

List of Build Tools

Invoke - Invoke is a Python (2.6+ and 3.3+) task execution tool & library, drawing inspiration from various sources to arrive at a powerful & clean feature set. 

Open Build Service - The Open Build Service (OBS) is a generic system to build and distribute packages from sources in an automatic, consistent and reproducible way. 

Webpack - Webpack is a module bundler for modern JavaScript applications. It takes the dependencies and generates a dependency graph allowing web developers to use…

Azure DevOps - The Azure DevOps Project presents a simplified experience where you bring your existing code and Git repository, or choose from one of the sample applications…

Cake - Cake is built on top of the Roslyn compiler which enables you to write your build scripts in C#. Cross platform Cake is available on Windows, Linux…

CMake - CMake is cross-platform free and open-source software for managing the build process of software using a compiler-independent method.

Gradle - Gradle is a project automation tool that builds upon the concepts of Apache Ant and Apache Maven and introduces a Groovy-based domain-specific language…

ANT - Apache Ant is a software tool for automating software build processes. It originally came from the Apache Tomcat project in early 2000.

Maven - Maven is a build automation tool used primarily for Java projects. The word maven means 'accumulator of knowledge' in Yiddish.

MSBuild - MSBuild, also called Microsoft Build Engine, is a build tool for managed code and was part of .NET Framework.

Gulp - Gulp is a build system that can improve how you develop websites by automating common tasks, such as compiling preprocessed CSS, minifying JavaScript and…

Build Tools - Maven - Build Lifecycle

A Build Lifecycle is a well-defined sequence of phases, which define the order in which the goals are to be executed.

The primary(default) life cycle of Maven is used to build the application, using 23 phases.

• Validate

• Initialize

• Generate-sources

• Compile

• Generate-test-sources

• Etc.

Lifecycle	Description
validate	validate the project is correct and all necessary information is available
compile	compile the source code
test	test the compiled source code using a unit testing
package	take the compiled code and package it in its distributable format, such as a JAR
integration-test	deploy the package into an environment where integration tests can be run
verify	verify the package is valid and meets quality criteria
install	install the package into the local repository
deploy	publish to integration or release environment

Maven has following three standard lifecycles:

• clean
• default (or build)
• site

These build phases are executed sequentially to complete the default lifecycle.

Given the build phases above, when the default lifecycle is used, Maven will

1. validate the project
2. compile the sources
3. run those against the tests
4. package the binaries (e.g. jar)
5. run integration tests against that package
6. verify the package
7. install the verified package to the local repository
8. deploy the installed package in a specified environment

To do all those, you only need to call the last build phase to be executed, in this case, deploy:

mvn deploy

Calling a build phase will execute not only that build phase, but also every build phase prior to the called build phase.

Thus, doing:

mvn integration-test

will do every build phase before it (validate, compile, package, etc.), before executing integration-test.

The same command can be used in a multi-module with one or more sub projects. For example:

mvn clean install

This command will traverse into all of the subprojects and run clean, then install including all of the prior steps.

Clean Lifecycle Reference

pre-clean	executes processes needed prior to the actual project cleaning
clean	remove all files generated by the previous build
post-clean	executes processes needed to finalize the project cleaning

Default Lifecycle Reference

validate	validate the project and ensure that all necessary information is available.
initialize	initialize build state, set properties or create directories.
generate-sources	generate any source code.
process-sources	process the source code.
generate-resources	generate resources.
process-resources	copy and process the resources into the destination directory for packaging.
compile	compile the source code.
process-classes	post-process the generated files from compilation.
generate-test-sources	generate any test source code.
process-test-sources	process the test source code.
generate-test-resources	create resources for testing.
process-test-resources	copy and process the resources into the test destination directory.
test-compile	compile the test source code
process-test-classes	post-process the generated files from test compilation.
test	run tests using a unit testing framework.
prepare-package	perform any operations necessary to prepare a package before the packaging.
package	package the compiled code into its distributable format, such as a JAR.
pre-integration-test	perform actions required before integration tests are executed.
integration-test	process and deploy the package into an environment where integration tests can be run.
post-integration-test	perform actions required after integration tests have been executed.
verify	run any checks to verify the package is valid.
install	install the package into the local repository.
deploy	publish the project.

Site Lifecycle Reference

pre-site	executes processes prior to the project site generation
site	generates the project's site documentation
post-site	executes processes to finalize the site generation
site-deploy	deploys the generated site to the web server

Maven Build Profile

Maven build profiles is a set of configuration values which allows us to build our project using different configurations.

We can use Maven build profile to set or override default values of Maven build.

We can use build profile to customize build for different environments such as production vs test environments.

We can set different database connection URL for testing and production environments.

Profiles are specified in pom.xml file using its profiles elements and are triggered in variety of ways.

Example

<project ...

  <modelVersion>4.0.0</modelVersion>

  <groupId>com.company.fee</groupId>
  <artifactId>fee-calculation</artifactId>
  <version>1.0.0</version>

  <profiles>
      <profile>
          <id>test</id>
          <activation>...</activation>
          <build>...</build>
          <modules>...</modules>
          <repositories>...</repositories>
          <pluginRepositories>...</pluginRepositories>
          <dependencies>...</dependencies>
          <reporting>...</reporting>
          <dependencyManagement>...</dependencyManagement>
          <distributionManagement>...</distributionManagement>
      </profile>
  </profiles>

</project>

Use the Content Package Maven plugin to integrate package management tasks into your Maven projects. The plugin goals and parameters enable you to automate many of the tasks that you would normally perform using the Package Manager page or the File Vault command line:

•    Create new packages from files in the file system.
•    Install and uninstall packages on the CRX or CQ server.
•    Build packages that are already defined on the server.
•    Obtain a list of packages that are installed on the server.
•    Remove a package from the server.

Throughout the software industry, several best practices are widely followed. Some of the more commonly used are an iterative development process, requirement management, quality control, and change control.

R is the leading analytics tool in the industry and widely used for statistics and data modeling.

Python is an object-oriented scripting language which is easy to read, write, maintain and is a free open source tool. It was developed by Guido van Rossum in late 1980’s which supports both functional and structured programming methods.

Excel is a basic, popular and widely used analytical tool almost in all industries. Whether you are an expert in Sass, R or Tableau, you will still need to use Excel. Excel becomes important when there is a requirement of analytics on the client’s internal data. It analyzes the complex task that summarizes the data with a preview of pivot tables that helps in filtering the data as per client requirement. Excel has the advance business analytics option which helps in modelling capabilities which have prebuilt options like automatic relationship detection, a creation of DAX measures and time grouping.

References

https://medium.com/@cleverti/why-is-code-quality-such-a-big-deal-for-developers-91bdace85d44

https://codurance.com/2014/12/14/quality-cannot-be-measured/

https://en.wikipedia.org/wiki/Package_manager

https://www.hongkiat.com/blog/manage-dependencies-tools-webdev/

http://www.java2s.com/Tutorials/Java/Maven_Tutorial/1070__Maven_Build_Life_Cycles.htm

https://xebialabs.com/the-ultimate-devops-tool-chest/build/


https://www.techopedia.com/definition/16359/build-tool

https://www.proschoolonline.com/blog/top-10-data-analytics-tools