Standard Guide for the Overall Greatest Value Evaluation Methodology Concerning Procurement of Computer Products and Services

(Tentative and Unofficial Translation)
28 March 1995

Standard Guide for the Overall Greatest Value Evaluation Methodology Concerning Procurement of Computer Products and Services

(Understandings among related ministries and agencies concerned in procurement)

This standard guide has been compiled as a guide to the basic operational points concerning operation within the rules of the heads of each of the government ministries and agencies after consultation with the Minister of Finance, to ensure that procurement entities contribute to improving efficiency of administrative processing in cases where the overall-greatest-value evaluation method is used for procurement.

Section 1
Rules of the Heads of all Government Ministries and Agencies (After Consultation with the Minister of Finance)

I. Scope of Application

The rules will be applied, in the case where the following designated computer products and services are procured, the issuance of the notice of invitation to tender is made after 1 July 1995.

Subject of Procurement

Computer Products
(1) Computers and Peripheral Equipment

(2) Products that are an important element of (1)

(3) Price of services associated with the provision of computer products in cases where the services concerned do not exceed the price of the products concerned.
Computer Services
(1) Computer Operation and Maintenance

(2) Computer Data Input

(3) Computer System Development

(4) Computer Software Development

(5) Maintenance of (2)-(4)

(6) System Integration Services

(7) Other related services (Including cases in which expert dispatch is the aim of the contract for services (1)-(6)).

(Note): The above-mentioned are referred to as computer products and services (hereafter referred to as products etc.,) as applied in the "Measures on Japanese Public Sector Procurement of Computer Products and Services" (A decision on 20 January 1992 of the 21^st Meeting of the Committee for Drawing Up and Promoting the Action Program. Hereafter referred to as "the Measures".

Scope of Application

Procurement in which products exceeds a projected price of 800,000 SDRs.
In addition to 1., when procurement entities decided to make a bid application where the overall-greatest-value evaluation is suitable, a report is made to the Minister of Finance no less than thirty days prior to the date of the Notice of Procurement.
(Note) 1. In cases where this bidding method is applied, the exchange of SDRs into Japanese currency is calculated based on the following method which is based on the exchange amount publicly notified by the Minister of Finance, which in turn is based on Article 2 of the regulation on "Cabinet Order Stipulating Special Procedures for Government Procurement of Products or Specified Services" (Government Ordinance No. 300 of 18 November 1980)

Conversion of SDRs into Japanese currency =
130,000 SDRs in Japanese currency multiplied by OO/13

(Note) OO indicates the SDRs in units of 10,000, which are the object of conversion.
2. With regard to adjustment of the amount converted to Japanese currency, amounts that are less that 10,000,000 yen are rounded up.

II. Bidding Method

Upon application bidders must supply information on price, performance, function and technology and of the tenders that conform to the following requirements, and through the overall-greatest-value method detailed in III, the tender which receives the highest value will be the successful bidder.
(1) The bid price is within the scope of the limits of the projected price.

(2) The performance, function and technology concerning the bid (hereafter referred to as performance etc.,) fulfill the required minimum stipulations of articles that are viewed indispensable in the performance etc., required stipulations (referred to below as technical specifications), which are enumerated in the Notice of Procurement (including relevant tender documentation, detailed below).
If the bidder with the highest value as explained in 1. above exceeds two persons, the successful bid will be decided by the drawing of lots.

III. The Overall Greatest Value Evaluation Method

The evaluation method of performance etc., is as detailed hereunder.
(1) The technical specifications, which are the object of evaluation, have been designed as evaluation articles from the viewpoint of administrative and business needs to respond to the pertinent procurement aims and contents. They are divided into those articles viewed as essential and other articles.

(2) With regard to essential articles, the minimum requirements should be shown article by article and the bids will fail if they are not met. If they are met, basic points will be designated and additional points will be given to the specification that surpass the minimum level.

(3) Articles other than those considered essential will be designated additional points by article upon evaluation.

(4) The points in each of the evaluation articles are decided by their necessity and importance.
The overall evaluation relating to price and performance etc., results in a value which is attained by dividing the total points for the performance etc., of evaluation categories in the bidders' applications, by the tendered bid price concerned.

IV. Others

In cases where this bidding method is applied, it should be made clear in the notice of invitation to tender that the determination of the successful tender will be made through the greatest overall greatest value method and the characteristics of this method.

Section 2 Guidelines Concerning Overall Greatest Value Evaluation

I. General Matters

The compilation of criteria for overall greatest value evaluation, while referring to the data and comments gained through such methods as the submission of comments, is based on "the Measures" and is designed to ensure transparency and fairness.
With regard to technical requirements and criteria for the award of contracts, tender documentation should be clear and specific, and to this end should be specified in a notice of invitation to tender.
In cases where technical requirements and criteria pertaining to bidding evaluation are stipulated in documents relating to specifications (hereafter specification documents) and documents relating to overall evaluation (hereafter overall evaluation standards), these are delivered immediately as part of the tender documentation, in response to requests from suppliers.

II. Technical Requirements

Technical requirements are divided into those that are essential and other stipulations, and are clearly specified in the tender documentation (including specification documentation).
Technical requirements and criteria for the award of contracts should be specified in the tender documentation, and that notion should be made in the notice of invitation .
Essential requirements are limited to those necessary minimum contents needed in practice by the procuring entities.
Requirements other than those considered essential are limited to those articles for evaluation as laid down in the overall evaluation criteria and those articles not subject to evaluation are not listed.
In the case when, as the results of the data and comments gained from the procedure of the submission of comments, etc., there are performance-related proposals through different methods from suppliers, and when any of the proposals could be adopted, the technical requirements are established not to exclude any specific supplier and consideration is duly given so that it is possible each suppliers performance can be evaluated.
Technical requirements which can be quantified (performance which can be quantified), are in principle specified as figures, and when it is inevitable that they be specified qualitatively , they are recorded to the most specific and detailed extent possible.

III. Evaluation Criteria

Concerning criteria pertaining to bidding, evaluation articles, distribution of points (basic points and points given upon evaluation (hereafter referred to as "additional points") and other articles necessary to evaluation are made clear in the tender documentation (including the overall evaluation criteria).
Evaluation articles and distribution of points are based upon their necessity and importance upon procurement and are established in a suitable fashion while referring to the performance etc., of products etc. that will be practically possible to introduce and have been supplied through the procedure of the submission of comments, etc.
In light of importance and necessity for procurement, items for which it is pointless to surpass the scope of necessary evaluation are excluded from evaluation.
In essential evaluation articles also, in light of the importance and necessity for procurement, it is sufficient that they fulfill minimum requirements and even if the necessary minimum is exceeded, which are beyond the scope of evaluation, no additional points are accrued.
Evaluation articles should be made as detailed and as specific as possible for the contents of evaluation. In this case, with regard to those items which can be shown in a scope (maximum figure) which is evaluated quantitatively, this should be notified in each evaluation article.
The proportional distribution of the sum of basic points and sum of additional points shall be established suitably, ensuring the purpose and usage of products, etc. to be procured.
Evaluation articles are based on the necessity for procurement and articles are chosen from those listed in the attachment.
In cases where specific evaluation articles are established, the articles are placed in the contract pertaining to procurement and the contents limited to those that can be secured, with those articles that cannot be secured not being subject to evaluation articles.

IV. Evaluation

Bidding evaluation shall be based on the tender documentation (including specifications and overall evaluation criteria) and performance etc., which is not specified in the tender documentation will not be evaluated.
The evaluation of performance etc., will be undertaken appropriately in a just and fair examination by the procurement entities.
Furthermore, the investigation undertaken will be of a common criteria for all bidders and there will be no specific methods adopted for evaluating specific bidders.
With regard to essential articles of evaluation, it will be judged whether or not the minimum requirements as specified in the essential stipulations in the tender documentation (including specifications) are fulfilled and then passed or failed.
With regard to those tenders that pass, the basic points and additional points will be given, based on the tender documentation (including the overall evaluation standards).
With regard to articles of evaluation other than those considered essential, it will be judged whether or not the non-essential requirements in the tender documentation (including specification documents) are fulfilled. If the requirements are met additional points will be awarded based on the tender documentation (including the overall evaluation standards).
Evaluations pertaining to qualitative evaluation articles shall be undertaken with due rationality.
Concerning evaluation of performance etc., in the case where tests are enforced for tendered products, in order to ensure that the process is undertaken in a fair and non-discriminatory manner, contents and methods of the tests concerned are clarified in the tender documentation.

V. Others

Recording of a Successful Bid etc., and Information Provision
(1) With regard to a result where performance etc., evaluation relate to the bid application in the overall evaluation or in the case of a successful bid, recording shall be made immediately. In particular, with regard to the results of inspection of technical requirements, each evaluation article and their outcome and the reasons for outcomes shall be recorded and appropriate responses to complaints, etc., from suppliers shall be made.

(2) Where requests are made from unsuccessful bidding suppliers for information on the successful bid, information will be provided on the relative merits of the successful bid (points gained for bid price and performance etc., by the said supplier and the successful bidder).
Ensuring Evaluation Contents
(1) With regard to performance etc., evaluated in the overall evaluation, all details of these shall be described in contracts to ensure they are carried out.

(2) For the inspection on the supply time for products etc., it shall be ensured that the evaluated contents are met.

[Attachment]

(1) Articles relating to Performance and Function, etc.

Processing Capability

(i)Equipment performance factors

a) Capability of computer processing devices (general processing capability): Articles of evaluation are established to evaluate whether the procured system can perform its assigned task. As a rule international standards shall be adopted.
b) Access time to external memory device and transmission speed: In the case where a system's processing capability is controlled by external memory device performance and processing methods for systems which use a virtual memory device or database management system, evaluation articles are established which sufficiently reflect the above conditions for access time and speed.
c) Capability regarding communications: Articles of evaluation are established for the number of lines that can be accommodated and the speed of support telecommunications.
d) Channel and Bus Capability: Articles of evaluation are established for channel and bus numbers, transmission speed and number of connection devices.
e) Other factors of equipment performance: Articles of evaluation are established accordingly for performance factors of peripheral equipment and others in line with the characteristics of the system for procurement.

(ii)System performance value

Responding to system characteristics, etc., articles of evaluation are established for performance values such as those based on standard models, performance values based on special model simulation and the establishment of processing capability values based on benchmarks.

Capacity and Scale

(i) Capacity of Main Memory Device

Articles of evaluation are established for capacity for the whole of the main memory device, with due consideration given to the utilization pattern of the main memory device for operating systems (OS) and user applications.

(ii) Capacity of Direct Access Memory Devices

Articles of evaluation are established for capacity etc., which takes into consideration processing efficiency, safety and capability to maintain functioning of the procured system.

(iii) Software Scale

Articles of evaluation are established in a certain scope for software scale of the procured system.

(iv) Data Capacity

Articles of evaluation are established for the largest possible data processing capacity, in order that processing capabilities can be guaranteed for data capacity as handled by the procured system.

Environmental Conditions

(i) Electric Facilities

Due consideration is made of the various factors with decide electric facility scale and method of supply (use of industrial power, AVR or CVCF), such as equipment standard power, voltage and possible scope of frequency modulation, and articles of evaluation are established for the specifications and performance of electric facilities for equipment.

Depending on the equipment there are cases where comparing the standard power use, the initial load when switching on power is very large, and it is therefore necessary to establish articles of evaluations to take this point into consideration.

(ii) Air Conditions

Articles of evaluation are established for air conditioning facilities.

(iii) Facility Conditions

Factors such as equipment area and weight which are factors deciding the scale and strength of construction facilities have articles of evaluation established for such equipment facility requirements.

(iv) Harmonizing with the Environment

With regard to equipment set-up in general offices, articles of evaluation are established for offices designs, noise levels and vibration that harmonize with the environment.

(v) Development Areas

In cases where places are established for development of systems, articles of evaluation are established for each condition depending on the form of procurement.

(vi) Development Equipment

Articles of evaluation are established for equipment used in development. The place of development is also of concern, but articles of evaluation are established as to whether the equipment can be used for the development of the procured system.

Function

(i) Hardware Equipment Function

Hardware equipment is based on function, and articles of evaluation established for each constituent factor.

a) Central processing device
b) External memory device
c) Input/output device (including special functions)
d) Terminal devices
e) Client Server Work Station
f) Other

(ii) Software Function

Software functions are divided into operating systems and function program products and articles of evaluation established for each of these.

(iii) Communications Function

Articles of evaluation are established for telecommunications protocols and types of support lines (including LAN/WAN)

Standardization

(i) Standardization of Input/Output Media

For methods of recording of information expressions of input/output media (magnetic tape, magnetic disc, optical disk etc.) and presentation of these media, articles of evaluation are established in order that the above conform to JIS standards or ISO standards etc.

(ii) Standardization of Methods of System Development

As for technical methods of development and tools used for development, with due consideration given to the system's life-cycle articles of evaluation are established.

(iii) Standardization of language processing programs

Articles of evaluation are established in accordance with JIS or ISO standard conformity for the describing format of language programs such as FORTRAN, COBAL, C etc., and for the describing format of input/output data and the rules of grammar.

(iv) Communications Function

Articles of evaluation are established for communications protocols to conform to JIS or ISO standards. As there are sections of ISO standards that overlap with standards based on ITU recommendations and since there is also a time lag between recommendations and standardization being established, in analysis ITU recommendations should also be the object of consideration.

(v) Other Standardization

Articles of evaluation are established for conformity to international and domestic standards.

Transferability

(i) Sustainability of Business

Articles of evaluation are established for sustainability and guarantees of existing business.

(ii) Sustainable Usage of Data, Etc.

Articles of evaluation are established for guarantee of the sustainable usage of data etc., and in cases where there is a risk etc., of whether or not conversion is necessary.

(iii) Sustainable System Usage

Articles of evaluation are established for guarantees for sustainable operation of existing systems and in cases where there is a risk etc., of whether program conversion is necessary or not.

(iv) Transfer Conditions

Regarding observance of transfer conditions, articles of evaluation are established for quality etc., during and after the transfer period.

(v) Operating Sustainability

With regard to operating sustainability of existing systems, articles of evaluation are established for cases in which changes occur.

Others

(2) Articles Concerning Reliability and Flexibility, etc.

Performance Reliability

(i) Reliability Indicators

a) Mean Time Between Failure (MTBF): MTBF is the mean time between one failure and another. Articles of evaluation are established for MTBF regarding equipment which forms the basis of a system.
b) Mean Time to Repair (MTTR): MTTR is the mean time taken from system failure to system repair. Articles of evaluation are established for MTTR regarding equipment which forms the basis of a system.
c) Availability: Availability is the ratio of system work to be carried out in unit hours. Articles of evaluation are established for this.
d) Mean Time Between Unscheduled Interruption (MTBUI): Articles of evaluation are established for time until systems are turned off, including hardware and software.
e) Level of Business Understanding: Articles of evaluation are established for knowledge and experience of the relevant business. Articles of evaluation are established for knowledge relating to related laws and regulations that are vital for business as well as knowledge and experience or related business.

(ii) Technology Reliability

a) Technology Reliability at Equipment Level: Articles of evaluation are established for damage detection functions, damage repair functions and damage assessment functions etc., such as error detection, self correction, re-booting, and damage informing at equipment level.
b) System Building Technology: Articles of evaluation are established for the improvement of system reliability through i) the main frame system; ii) the input/output system; iii) the telecommunications system; and iv) the composition of these systems integrated.
c) Technology Reliability at a Business Level: Articles of evaluation are established on a business level for processing correctness, legality and speed, as well as on a technological level.

(iii) Measure to Improve the Quality of User Programs

Articles of evaluation are established for each group of functions, etc., for the purpose of improving the quality of user programs and supporting higher reliability.

a) Function of development support
b) Function of test support
c) Function of software management

(iv) Measures to Improve Operation Reliability

Articles of evaluation are established regarding measures for faulty operation in central equipment operations.

Operational Performance

(i) Operational Performance for Hardware

Articles of evaluation are established for operational performance for identical models for the main equipment of calculation systems.

(ii) Operational Results for Software

Articles of Evaluation are established regarding the operational results of operating systems and each software.

Security

(i) Functions for protection from irregular access

As measures to avoid the intentional leaking and destruction of information, since it is necessary to limit access, articles of evaluation are established regarding protective functions.

(ii) Protective Functions Against Destruction

Articles of evaluation are established regarding conditions for all protective measures against information destruction.

Ease of Maintenance

(i) Function Composition

Articles of evaluation are established regarding ease of maintenance by ensuring that plans for function composition and function division are submitted.

(ii) Documents

Articles of evaluation are established regarding the usability etc., in subsequent function expansion of the documents that are necessary for maintenance by ensuring that such types and their documents contents are submitted.

(iii) Software Management Tools

Articles of evaluation are established regarding whether or not software management tools are used and tool contents, etc.

Expandability

(i) Hardware Expandability

Articles of evaluation are established for factors including system expandability, such as improvement of processing capability, increase in file capacity and increase of direct access terminals.

(ii) Software Expandability

Articles of evaluation are established regarding expandability of a) processing format b) networks c) operating systems, after system development.

a) Expandability of processing format
Expandability upon change to processing format
b) Network expandability
c) Expandability of operating system
d) Ease of introduction of packaging software

Compatibility

(i) Compatibility of Input/Output Media

Articles of evaluation are established regarding the compatibility of equipment of input/output media such as magnetic tapes, floppy discs etc., compatibility of information recording methods and compatibility of internal codes.

(ii) Program Compatibility

Articles of evaluation are established for compatibility at the source program level and the executable program, and compatibility or ease of use of screen or tag creation aid tools.

(ii)Compatibility of Job Control Language

Articles of evaluation are established for compatibility and ease of use of job control language.

Ease of Operation

Articles of evaluation are established regarding for system operation status monitoring and network operation status monitoring functions.

Automation Measures

(i) Operation Automation Function

Articles of evaluation are established for functions, etc., aimed at the automation and energy saving functions of system operation.

(ii) Operation Automation Function

Articles of evaluation are established for the automation and energy saving function, etc., of input and output operation devices.

Others

(3) Articles concerning maintenance and support, etc.

Hardware maintenance

(i) Methods concerning maintenance systems

Articles of evaluation are established for such systems as the preventive maintenance system, response systems in case of trouble, and procurement systems for replacement components. Articles of evaluation are also established for those systems that are being developed at a local level.

(ii) Maintenance Technology

Articles of evaluation are established for self-assessing trouble technology, long-distance damage examination technology as well as automatic restorative technology and all kinds of maintenance technology to ensure systematic and effective trouble detection, assessment and restoration.

Software Support

Articles of evaluation are established for software support systems and the support range, etc.

Documentary Support

Articles of evaluation are established for documentary support, such as whether it is provided, the support media, the time of provision and contents, etc.

User Education

Articles of evaluation are established for the system, subject, timing and methods, etc., of education of users.

Restrictive Conditions

Articles of evaluation are established for restrictive conditions that arise against the procurement side through the procurement concerned.

Others

(4) Articles relating to capability of bidders for development and supply

Development Methodology

Articles of evaluation are established for the existence and suitability of methods of development methodology that are used for supply by bidders. In addition, articles of evaluation are established for concepts, etc., relating to function composition and documents in development methodologies.

Schedule for Realization

The bidder is required to compile a schedule for realization and articles of evaluation are established for the suitability of the schedule of the procurement side.

Project Management

Articles of evaluation are established for the suitability of project personnel level (these are ranked through the Information-Technology Engineers Examination which is a basis for evaluation), system, and personnel plans. Articles of evaluation are also established for the suitability of schedule arrangements.

Techniques/Tools

Articles of evaluation are established concerning the existence and suitability of application plans for tools by process and function.

Existence of an Environment for Development

Articles of evaluation are established concerning the status, etc., of facilities for computers, in cases where all or part of the development process are implemented through supplier side computers.

Development Results

Articles of evaluation are established for the development results of similar systems and similar scales (manhours/expenses).

Reliability of Bidders

(i) Quality Control

Articles of evaluation are established concerning quality control.

- Quality control policies
- Organization
- Responsibility and competence
- Quality control activities
- Quality control tools

(ii) Official Qualifications

Articles of evaluation are established for the acquisition of official qualifications (organizational and personnel), such as ISO9000 and efforts towards acquiring such qualifications.

(iii) Security

Articles of evaluation are established concerning suitability of security systems.

(iv)Employee Education

Articles of evaluation are established for organization, systems, and results pertaining to employee education, etc.

Others

(5) Others

(i) In addition to the above, other appropriate evaluation articles shall be established as the occasion arises.

(ii) For the compilation of specifications relevant to software development, the software development process and process contents shall conform to clearly defined standards based on the ISO/IEC Software Life-Cycle Process (SLCP).

Index | Back | Next