Formal Methods for Developing and Changing Engineering Systems

I was reading this post: https://levelup.gitconnected.com/how-to-plan-for-success-when-starting-a-new-technical-project-901599161629

It made a lot of sense, but it was missing the critical part of how to plan. Planning should be done at the creation of a system (any system complex enough for both sides of the napkin) and for changes (which are guaranteed to occur). But are there formal methods to do so? Yes.

One time-tested formal method is Systems Engineering. (SE). This is used in projects like aircraft and medical devices. There are specific versions (medical has a set) and aircraft (DO-178C, others). However, they easily scale to whatever system you need. For a very simple system or program, they might be a few pages. However, most systems are complex enough that planning becomes critical. The famous quote is:

“If you fail to plan, you are planning to fail!” — Benjamin Franklin

But they all boil down to “plan your work, work your plan” — catch the errors at the (cheap) spider-monkey stage, not the (expensive) King Kong stage.

Please note SE processes are valid for both new projects and changing a system. They both require the same process. And changing a system is easier for a project built with SE methods because you have the documentation from the start, plus your team has the SE experience.

Part of the SE process is to define what needs to be done, and no more. This means changing the system is easier, because, done properly, you don’t need to change the existing system, just add to it. In the case of needing changes to the existing system, you have an easier task of figuring what/why was done and therefore what needs to be changed.

I happen to be a fan of Uncle Bob (Martin). We are both veterans of the Coding and Process wars. He is one of the Agile founders. I want to emphasise the process below works with Agile. While the full SE process is a heavyweight, each of the concepts can be applied to the general Agile process and to the sprints. The biggest part is getting the requirements nailed down, because they drive the project.

In the medical and aviation fields, following the SE processes are mandated by the FDA and FAA. However, for non-regulated companies, this must be mandated and enforced by management. The problem is a lot of programmers want to start coding right away. This leads to cost and schedule over-runs, not to mention systems that do not meet the requirements and goals. However, when done right, the SW engineers are involved from the start. This creates buy-in. And once the SW engineers see how it actually makes their lives easier, they will be more likely to do it on other projects.

Note Electrical and Mechanical engineers need to plan because they are dealing with physical things. SW is fungible, physical things are not. However, a number of projects need SW to control the physical things, which means planning SW is important for these types of projects.

There are four steps in the process:

1) Requirements — what to do.

This is critical, because that can be a tricky thing to actually specify. See https://tenstep.com/use-these-four-steps-to-gather-requirements/ for a good start. Creating requirements is trickier than it may seem. Sometimes the players are not obvious. The characteristics for a good requirement are:

1a) must be clear

1b) must deal with a single issue

1c) must be testable

1d) must not conflict with other requirements

The link states: “A good requirement states something that is necessary, verifiable, and attainable.”

For example: Assume the system processes 200 records per second — too slow. A requirement would be: the system shall process 1000 records per second. Note in this case, this is a systems-level requirement, probably from management. The system needs to be analyzed to determine if the current hardware will allow this increase, or more/better hardware is needed. The requirements are the result of a systems-level analysis of the problem.

For an excellent tutorial, see https://reqexperts.com/wp-content/uploads/2015/07/writing_good_requirements.htm . NASA: https://www.nasa.gov/seh/appendix-c-how-to-write-a-good-requirement

There are several keywords for requirements. See https://reqexperts.com/2012/10/09/using-the-correct-terms-shall-will-should/ and https://www.ietf.org/rfc/rfc2119.txt for a discussion on words “shall”, “must”, “will”, “should” and “may”.

“Shall” or “Must” are (synonyms) mandatory requirements.

“Should” means that there are valid reasons to ignore a particular requirement, but the implications need to be weighed.

“Will” lays out specific engineering specifications for (generally) physical things, such as the physical dimensions of an avionics engine monitor, or the power consumption/voltage/current, etc. of an engineering unit.

“May” means that the requirement is truly optional. Interoperability is critical with different systems that may or may not implement an optional requirement. These types of requirements must be carefully thought out.

The link states:

• Requirements use shall
• Statements of fact use will (This one can be a bit tricky.)
• Goals use should

Note there are actually levels and types of requirements, from the general to the specific. However, all requirements must satisfy the above conditions. For more detail, read https://acqnotes.com/acqnote/tasks/requirement-types, but the types are below. The link has a good description of the types:

• Functional Requirements (actions)
• Performance Requirements (measurable)
• System Technical Requirements (Allocated and Derived)
• Allocated Requirements: flow directly from the system requirements down to the elements of the system.
• Derived Requirements: dependent on the design solution (and so are sometimes called design requirements). They include internal interface constraints between the elements of the system.
• Specifications (technical details on mainly physical things)

One advantages of proper requirements is the ability to estimate the time and money for each requirement. Taken as a set, you can get a good project budget. Then, you have a tool you can take to upper management.

Look around for requirements for similar systems. This can give you a starting point. Engage the stakeholders. They will be more supportive when they have “skin in the game”. The one caveat is they may want to expand the scope past what they need to be. The SE process gives you the power to point out the time and money for different paths. This tends to reduce the “Kitchen Sink” syndrome once management sees the cost of the parts that are not needed.

2) Design — how to do it.

The design needs to be in enough detail to create the schedule. In Agile, these would include user stories. They also need to be detailed enough to clearly specify the work. For example, the path needed to get 1000 records per second. The details must be worked out in the design phase.

Details might include register definitions for custom hardware, API definitions, state machines, algorithms, webpage design, database schema, methods of verifying logins, etc. All of the actual details required for the system. Do not skimp on this step. Work through all of the details. Storyboard and walk through all of the steps.

This does not need to be complex. Use 3x5 cards. Use powerpoint. Use Adobe tools to create the user experience. Solidworks, a top mechanical engineering tool, uses simple methods (3x5 cards, simple mockups, etc) to create new features. Point is: simple tools work.

Bring in a variety of people to “test drive” the design. Bring in your kids and parents. An 8yo kid and a 65yo should both be able to drive your webpage design. Personally, most webpages drive me up the wall because they have things like “hit the submit button” and there is no submit button!

The design needs to specify the requirements that will be satisfied. All work must be for satisfying one or more requirements. Setting up a matrix of requirements vs design elements is a good way to manage and measure the work. This allows everybody to see the project progress. (A good manager will sprinkle the schedule with a few fun events for important milestones (hint).)

Use reviews throughout the process: design, code, test. Emphasise to your team the goal is to find the holes. Emphasise the team’s ego must be on the result, not the process. Reward both good design/code/etc and finding the holes at the team level. The goal is a successful project and a happy team.

3) Implementation

This is the “work your plan” part. Use whatever method you need. Use continuous testing and integration — this is a good way to manage and measure progress. Use whatever testing method you are used to and is appropriate to the project.

I would suggest putting the task/requirements matrix on the wall, and keep it updated. This shows the team the project’s progress. Celebrate at different milestones, and a big party at the end.

4) Tests

There is a test phase in formal SE . Tests are defined up front (eg exactly how to measure how many records per second are being processed). This does not imply testing should be left to after the code is written. Continuous testing is fine and encouraged. Again — spider-monkey, not King Kong.

Part of the requirements phase is defining of the tests to satisfy the requirements. Obviously, the type of tests and how they get executed is a function of the system. The SE process does not enforce any particular method. But the tests are defined and created to assure the requirements are met. The tests should be run from the start. They will fail, but they will be written. A passing test is one step closer to the end.

Fundamentally, all testing should verify that all requirements are satisfied. This is the fundamental reason for having a matrix of (requirements, tests, work) in order to show progress and the successful end of the project.

There is a formal test called V&V: Verification (did we build the system right — eg as planned), and Validation (eg did we build the right system). Both are critical. You can have the best designed system, but if it does not what is needed, the effort has failed.

Conclusion

A good resource to formal SE is INCOSE.org: the International Council On Systems Engineering. They created the “V” pattern. The left side, going down, is the planning. All requirements and planning is done when you hit the bottom of the V. The right side, going up, are the implementation and testing.: https://www.sebokwiki.org/wiki/System_Life_Cycle_Process_Models:_Vee

Granted, the full SE process is a heavyweight. It has to be, for something that will kill people when the plane falls out of the sky. But you can scale it as needed. Following the process forces you to think through what and how to do it. In the end, it actually makes the project go smoother and more predictable. All of the INCOSE material is on-line. Requirements are actually the hardest part, due to the conditions for a good requirement, and assuring the complete set of requirements. But the effort up-front saves money, time, and those 3am “Oh Sh*t” efforts. Also helps prevent ulcers and upset families from slipped schedules.

See https://en.wikipedia.org/wiki/Systems_engineering for a more complete description. Also INCOSE.org for the gory details. It may also help to hire an SE consultant to teach a one or two day class on the process. Again — you can tailor the process to fit your needs.

Start small, with a project that might take a couple of months and fairly limited scope. As you gain confidence, scale up. The SE process really works when done right. You will find projects more predictable and manageable.