“Secure web environment, maintained in NL, two step authentication” – PSJ –
“Define criteria with Delphi-panel and AHP” – PSJ –
“Decide on criteria sets” – PSJ –
“Define alternatives” – PSJ –
“Score alternatives on criteria” – PSJ –
“Judge results” – PSJ –
“Compare criteria sets” – PSJ –
“Judge analyses” – PSJ –


Objective Ranking Tool (ORT)

The support of the Objective Ranking Tool (ORT), a webbased software application, is multi-functional to make comparisons. ORT helps for example to:

  • prioritise risk locations and circumstances based on a defined threat;
  • define the vulnerability and resilience of organisations to natural hazards and extreme weather;
  • judge alternatives when taking decisions and compare the outcome with the 'ideal' solution;
  • decide in questions of ranking to define the best shop or product;
  • analyse in questions of compliancy which elements score well and which don't;
  • choose the best candidate in recruitment and selection;
  • make the best choice in tender processes;
  • find the best partner in a dating process;
  • choose the best energy- or telecom provider for a specific group of customers.

The support of ORT will lead, in a verifiable and objective process, in a short timeframe, to a ranking of alternatives based on predefined criteria. The highest scoring alternative related to a reference, complies the most to the defined criteria. In the process, in which all stakeholders can be involved, the imphasis is based on facts, meanings and opinions will be made objective as much as possible. The outcome will give input for decision making.

The most important features:

  • the combination of three scientific principles: Similarity Judgment, Analytic Hierarchy Processing and Delphi-panel;
  • involvement of all relevant partners, their interests and the translation to criteria;
  • the common process and analyses, which is needed to develop criteria, will lead to better acceptance of the outcome;
  • ORT is easy to adapt when new information is available;
  • within ORT sensitivity analyses will be possible.

PSJ offers with ORT:

  • the use of a high-quality software application, availability based on license or one time use;
  • proces support and project management;
  • training;
  • substantive support.


The Delphi methode (Rowe, Wright, (2001)) is used to develop alternatives based on a panel of independent experts. The basic principle of Delphi implies that predictions of a structured group of experts will give better outcomes then an unstructured group or individuals.

The advantages of Delphi are:

  • value free input of observations by a group of participants;  
  • discussions and outcomes will not be influenced by one person; 
  • several rounds to share and comment on opinions;
  • involve 'the' experts within a certain field of knowledge.

Within ORT a Delphi-panel is used twice: 

  • when initially developing criteria and criteria sets; 
  • in the process of scoring on alternatives.

The group of experts will be different in both situations.

Analytic Hierarchy Process (AHP)

AHP (Saaty, 1978) is a methode to give weights to criteria. AHP is based on pairwise comparisons and is needed to combine with the Delphi-methode.

Pairwise comparisons are based on the scale of Saaty's between 1-9 where the value '1' an equal importance between two criteria expresses and the value '9' a maximum difference on importance.

After each round the satisfaction of the experts should be judged.

  • is the knowledge used well? 
  • is there added value from the expert during the meeting? 
  • are the results recognised? 
  • which improvements are possible?

AHP offers the possibility for sensitivity analysis which can be applied within ORT as well. 

For research PSJ uses the templates from http://bpmsg.com.

Similarity Judgment

Research from Tversky (1977) concluded that similarity judgments of two phenomena depends on an analysis of the features shared by the two phenomena as well as the features which are unique to each phenomanan. Such a similarity is expressed by a number between '0' and '1' according to the  formula Sij = fi, j / [fi , j + a ( fi , not j ) + b ( fnot i, j )] in which:

  • features in object i, but not in object j: fi, not j; 
  • features in object j, but not in object i: fnot I, j; 
  • common features of both objects: fi, j;
  • the constant 'a' and 'b' counts up to '1’ and are equal (both 0,5) if all possible features are taken into consideration.

The closer the outcome according to the formula to '1' the more equality between objects exists. 

This principle is used within ORT as a mathemetic formula where amendments are added.