Pushed by the necessity to objectively measure the progress of their verification efforts and the contributions of various verification strategies, IC designers have adopted protection as a metric. Nonetheless, what precisely is being measured is completely different relying on the underlying verification know-how in use.
In Half 1, we outlined the textbook definitions of simulation and formal protection, after which we briefly launched the dangers inherent in arbitrarily merging these metrics collectively. In Half 2, we shared a collection of RTL DUT code examples for instance the difficulty that may happen when evaluating the outcomes from simulation and formal side-by-side.
On this article, we are going to show how even correctly merged 100% code and purposeful protection outcomes can nonetheless tempt you to prematurely conclude that your verification effort is over and it’s protected to declare victory. However the excellent news is that mutation evaluation can exhaustively show that each your DUT and your verification plan itself are bug free. Plus, we are going to summarize suggestions for correctly utilizing simulation and formal protection collectively.
Full code protection achieved—however there may be nonetheless a bug
The instance finite state machine (FSM) and supporting logic in Determine 1 reveals a lure that many fall into—assuming that 100% code protection implies that they’re completed with verification. This assumption is predicated on a false impression of the explanation we measure code protection—the only real worth of code protection is in stating areas of the DUT which haven’t been verified, no matter which engine is getting used.
Determine 1 Here’s a view of 100% code protection of FSM from simulation (left) and formal (proper). However may there nonetheless be a bug within the DUT? Supply: Siemens EDA
On this instance, the 2 outputs are mutex and a property has been written that passes in simulation and is confirmed in formal. On this verification situation, the above outcomes from simulation and formal present full protection on the mutex test for the “out1” and “out2” alerts.
Regardless of these promising outcomes, there are two vital factors to contemplate. One is the significance of a whole check plan. Think about that there’s a requirement of “the FSM to be in solely state 2 for not more than 3 cycles”. On this case, all of the code is traversed, however the purposeful habits is inaccurate. If this requirement was not a part of the check plan, then the above protection stories would idiot the consumer into believing that the verification was full whereas the bug slipped via. Nonetheless, if the requirement made it into the check plan—which was in-turn captured within the following assertion—then it will be correctly examined:
a_st2_3_max: assert property (@(posedge clk) disable iff (!rstn)
$rose(st2) |-> ##[1:3] !st2 );
On this situation, simulation may have utterly missed this purposeful bug if there was no check to cowl this case; and actually, for the vectors run to present the above protection, that was the case, because the assertion handed. Nonetheless, exhaustive formal evaluation would have noticed the purposeful error and reported it as a counter-example to the consumer.
The bigger level of this instance is the cautionary be aware that protection is especially for seeing what isn’t lined, versus the give attention to reaching 100% protection in any respect prices; even when you’ve got 100% code protection, you should still have bugs that different analyses and protection metrics will reveal.
Now let’s go a bit additional and see how model-based mutation protection can assist in revealing such gaps in a verification atmosphere. Mutation protection is generated by the verification software systematically injecting mutations (faults) into a proper mannequin of the DUT, after which checking whether or not any assertion would detect a bug on the corresponding fault level. The mutations are completed electronically, within the mutation software’s reminiscence, whereas the unique supply code is untouched. In impact, mutation protection measures the flexibility of your testbench to seek out bugs; in case your verification atmosphere can’t discover a bug that you’ve got intentionally created within the type of a property, it’s unlikely to seek out bugs you haven’t imagined.
As proven in Determine 2, working the protection evaluation with mutations signifies exactly the place there’s a hole overlaying the requirement, and thus the necessity to add it to the check plan so as to make it full.
Determine 2 The model-based mutation protection reveals the lacking requirement and associated check from the check plan. Supply: Siemens EDA
Recap: Utilizing each formal and simulation code protection collectively
Verification groups use protection knowledge from three forms of engines: formal, simulation, and emulation. All contribute to check plan signoff in addition to protection closure in all points. Specializing in code protection, the next desk outlines a few of the variations between formal and simulation code protection:
Desk 1 The information highlights the variations between formal and simulation code protection. Supply: Siemens EDA
The first concern right here is that mixing code protection from a number of verification engines can masks holes within the testbench of the opposite engine. There are different delicate variations which make merging code protection from completely different engines troublesome. The next suggestions might assist.
Suggestions
The formal and simulation protection flows are versatile, which lets you make use of them in no matter capability most closely fits your wants. Under are some things to contemplate when utilizing code protection from any of the verification engines:
- Ideally, shut code protection for every verification engine individually.
⁰ Give attention to bettering testbench completeness and robustness in every area.
⁰ Code protection knowledge ought to be stored separate in the principle protection database for this goal.
⁰ Moreover, within the formal area, run each proof core (formal protection based mostly on the logic wanted to show a property) and mutation protection to test testbench completeness.
- Take a look at planning is vital for figuring out which verification engine will confirm which components of the design.
⁰ Formal might completely confirm sure modules. When mixing formal and simulation code protection, attempt to preserve it to occasion boundaries, one thing that may be enabled/disabled in a single area versus the opposite.
⁰ When mixing code protection, plan it as a part of the check plan, late within the recreation.
- Know the place your code protection comes from: formal versus simulation.
⁰ Maintain code protection knowledge from every area separate in the principle protection database.
⁰ Reporting should additionally make it clear the place the protection knowledge got here from.
⁰ Solely merge protection knowledge for last stories close to the tip of the undertaking.
- Keep away from writing properties or including vectors to solely trivially hit some particular a part of the design to get to the 100% protection mark.
⁰ This doesn’t usually enhance verification, solely trivially improves protection.
⁰ That is why the check plan is vital. Protection holes level to an incomplete check plan and in the end an incomplete/weak testbench.
⁰ When including properties or vectors to shut code protection holes, verification of options and necessities of the design are your information and will provide you with the very best high quality of verification.
Naturally, code protection from numerous verification engines will probably be used to make selections relating to assembly undertaking milestones. The verification group and undertaking leaders are in the end accountable for their outcomes. When required to combine protection for a given block from a number of verification engines, it is suggested to have a peer assessment on how the code protection was generated to ensure legitimate exams tied to a check plan are used.
Correct merger of outcomes
With correct understanding of the character of the protection metrics being recorded, output from all sources might be mixed to present a holistic image of the progress and high quality of the verification effort. On this collection, we targeted on the commonest aspect of this want—correctly merging the outcomes from formal evaluation with the protection from a constrained-random simulation testbench such that particular person contributors and group leaders will perceive precisely what the information is telling them.
Editor’s Word: That is the final article of the three-part collection on the pitfalls of blending formal and simulation protection. Half 1 outlined the dangers of arbitrarily merging simulation and formal protection. Half 2 shared RTL DUT code examples for instance the difficulty of evaluating the outcomes from simulation and formal side-by-side.
Mark Eslinger is a product engineer within the IC Verification Programs division of Siemens EDA, the place he makes a speciality of assertion-based strategies and formal verification.
Joe Hupcey III is product advertising and marketing supervisor for Siemens EDA’s Design & Verification Applied sciences formal product line of automated functions and superior property checking.
Nicolae Tusinschi is a product supervisor for formal verification options at Siemens EDA.
Associated Content material
