Digital Health Impact Framework User Manual

The Digital Health Impact Framework (DHIF) supplements ADB’s Guidance for Investing in Digital Health and guides people on how to appraise planned digital health investment decisions. It provides a process to help digital health specialists in Ministries of Health, private sector and those working with development partners make well-informed investment decisions. Five supporting illustrative DHIF models accompany the manual as examples for how to decide on digital health investments.

Who is this digital health impact framework (dHiF) manual for? People who need to appraise planned and proposed digital health strategies and investments to brief decision-makers on finding optimal strategies and projects. These decision-makers may be politicians, senior government officials, health service executives, clinical and digital health leaders and managers, and stakeholders, such as donors, who support health systems.
What is the dHiF manual's purpose? DHIF (i) supplements the Guidance for Investing in Digital Health and guides people on how to appraise planned digital health investment decisions, and (ii) provides a process to help digital health specialists and business managers tasked with assembling data and analyses needed to enable well-informed investment decisions. Five supporting illustrative DHIF models accompany the manual as examples of the information produced. Other DHIF models using parts of the methodology are available too.
What is the expected outcome of the manual? Government and health system leaders will (i) take advantage of the DHIF methodology to appraise and develop the foundation of their digital health strategies and projects, large or small; (ii) maximize their benefits and net benefits; and (iii) minimize risks by developing and implementing well-formed digital health strategies and investment plans as well as sustainable digital health programs and projects.
What is the best way to start using dHiF? Select the parts that fit the important issues for specific digital investment decisions. These could be • capital expenditure affordability, • operational expenditure affordability, • all stakeholder costs, • just health care costs, • all stakeholder benefits, • just health care benefits, • comparing digital health investment options, and • risk exposure and mitigation strategies.
It is not essential to apply the manual in its entirety. Choose which parts are appropriate for the needs of the specific digital health investment. Lessons from the three actual DHIF models reinforce this approach. Five illustrative Digital Health Impact Framework models have been developed to use as examples in the manual. These were refined after applying the manual in three different settings, which are also included in the manual. All models are available online at Asia eHealth Information Network's Standards and Interoperability Lab for Asia website at http://www.sil-asia.org/lab-assets/.
iNTrOduCTiON This user manual supports Guidance for Investing in Digital Health, an Asian Development Bank (ADB) publication which sets out a way to find appropriate strategic directions for digital health. It also recognizes a low success rate of digital technology projects in the public sector. About 30% fail totally; between 50% and 60% are partial failures. This leaves, at best, a 20% success rate (World Bank 2016, p. 165). This poor performance and investment return emphasizes the need for rigorous investment appraisal of digital health programs and projects.
The manual describes how to use the digital health impact framework (DHIF) to • identify appropriate, sustainable digital health programs and projects that achieve strategic goals; and • produce information for business cases that decision-makers need for effective investment decisions.
This DHIF manual describes how to create bespoke DHIF models, either partially or in full. It has three main parts: (i) Concepts and Methodology, (ii) Understanding the Digital Health Impact Framework Model, and (iii) Illustrative Digital Health Impact Framework Models.
It is an aid to developing the information and skills needed to set up and use DHIF models. For new users, this manual is an appropriate way ahead to start small-scale DHIF health models, then move on progressively to larger-scale models. Users should increase their DHIF experience gradually to deal with the increasing complexity of digital health. There are five illustrative DHIF models, which are referred to in parts II and III, and are used to illustrate the application of DHIF: • mHealth for telemedicine dermatology, with no extra access for patients who have no previous access, so only for current patient cohorts; • mHealth for telemedicine dermatology, with extra access for patient cohorts who had no previous access, so a universal health coverage (UHC) initiative; • short messaging service (SMS) for advice and checkup reminders for pregnant women; • digital surveillance for malaria; and • interoperable electronic health records (EHRs), small scale for a group of local hospitals.
DHIF has been used in three live settings: • digital health strategic affordability and benefits for a small state; • immunization information system (IIS) options for a small state, outline the case for change, and an ecosystem, developed in collaboration with United Nations Children's Fund (UNICEF) East Asia & Pacific and shown in Appendix 1; and • comparison of local immunization performance.
Data from these has been converted into generic models and used to illustrate DHIF in part III. They are available, along with the initial five illustrative DHIF models, to download and review from the Asia eHealth Information Network (AeHIN) Standards and Interoperability Lab for Asia (SIL-Asia) website at http://www.sil-asia.org/lab-assets/.
Most importantly, users can use individual parts of DHIF as independent modules that match their specific digital health priorities. The three actual DHIF models adopted this approach. DHIF is a generic methodology, but each of its models is bespoke. They are built from specific digital health projects, enabling users to address their specific strategic and investment challenges. Examples of DHIF start-up components are • all stakeholder benefits, • just health care benefits, • capital expenditure affordability, • operational expenditure affordability, • all stakeholder costs, • just health care costs, • comparing digital health investment options, and • risk exposure and mitigation strategies.
Users can choose which parts are appropriate for the needs of their specific digital health investments. Before starting to construct a DHIF model for a proposed digital health project, users should first have a firm understanding of the main components of proposed digital health investment. These include • investment goals linked to health, health care, and digital health strategies; • user requirements; • required functionality; • architecture; • interoperability; • usability; • scope; • network requirements; • data and capacity dependencies from other digital health investments; • data utilization; and • stakeholder engagement, human capacity building, and change management.
They determine many of digital health investments' time lines, resources, costs, benefits, and risks. Without an understanding of these digital health components, DHIF models are limited.
Lessons from the three actual DHIF models are the following: • Start small, and expand sophistication toward full DHIFs slowly, such as steps 1, 3, and 9 of the 10 DHIF steps described in section II-C. • Comparing investment scenarios and options is critical for effective decisions. • Data needed are not all readily available, reinforcing the need to be skilled at estimation. • It is important to fit DHIF models into business cases to secure resources. • Having someone in organizations with modeling skills enables a smooth small start. • Digital health initiatives derived from policy and strategic edicts always require investment choices, options, and decisions that benefit from DHIF.

i. CONCEpTS aNd mETHOdOlOgy a. introduction
Approaches to developing digital health strategies are set out in ADB's Guidance for Investing in Digital Health. A core component is the need to make affordable digital health investment decisions that achieve value for money (vFM); a socioeconomic concept applied to digital health to help improve decision-making and the delivery of services, which advance health as well as health care strategic goals and priorities. Figure 1 shows the two main goals of digital health investment decisions: • Identify a mix of digital health projects that offers the best, affordable fit to health and health care strategies; and • Identify options for each digital health project offer the best, affordable vFM.
This view simplifies the much wider and more complex range of projects and options that may be involved.
DHIF models at the "Which options?" level provide the data needed to support the framework's "Which projects?" role. Without DHIF and its models in place for the "Which options?" it is seldom feasible to deal with the strategic and benefits requirements of "Which projects?".
Good decisions depend on rigorous business cases. Using DHIF models provides substantial components, and these enable prompt adjustments as decision-makers develop, challenge, refine, and appraise their • identifying and describing why a digital health project's estimated vFM is what it is; and • seeking a probable digital health investment's performance, not its potential, which exceeds its probability and is rarely achieved, if ever.
Socioeconomic costs and benefits extend across all stakeholder types. Patients, their carers, families, other residents, citizens, and visitors; health workers; and health care organizations are all stakeholders, so their estimated costs and benefits are included in DHIF models. DHIF calls vFM the socioeconomic return. More details are described later in section II-k.
Financial features are similar to some socioeconomic costs, but are different. Financial costs include changes in cash flow of all stakeholder types. They also include changes in health care organizations' income and expenditure (I&E), an accounting accruals concept. Both of these measures of affordability are investment plans' critical constraints. More details are described later in section I-D.
DHIF sets all estimated costs and benefits as well as financial profiles in individual years across an appropriate time scale. Cost, benefit, and financial curves change across these years, leading either to positive or negative cumulative net benefits and their socioeconomic return at the end of the time scale. Alongside these are financial and accounting affordability profiles and challenges that match financial costs to the finances available.
Estimating and calculating a socioeconomic return is not sufficient. An explanation of how to interpret the socioeconomic return must accompany it. This means understanding how the variables in the DHIF model interact to give a high socioeconomic return, a moderate socioeconomic return, or a negative socioeconomic return. This knowledge and understanding is the most important aspect of DHIF modeling. Decision-makers need it for informed decisions.
Digital health leaders need to be able to make optimal digital health investment decisions that implement their strategies. For each project, it means knowing an estimated probability of it being optimal, why it is optimal or why not, and how to keep it optimal. The potential of digital health is seldom realized fully, if at all. Potential benefits overstate investments' socioeconomic returns and understate its risks. Understanding these relationships is vital, and DHIF helps decision-makers in this quest. It leads on to finding an optimal solution.
As a concept of digital health economics, DHIF is a way of quantifying and estimating the impact of digital health or, in some cases, lack of impact, on the complex changes planned and needed for better health care and people's health. When estimated socioeconomic benefits of these exceed estimated socioeconomic costs, aggregated for all direct stakeholder types, the result is vFM. When they do not achieve it, digital health projects are not worth it, and thus should be modified or abandoned.

C. digital Health impact Framework as a learning process
Everyone has their own way of learning. One way to learn how to apply the DHIF methodology, described later in the manual, is to adopt a structured sequence of • understanding the DHIF process and methodology; • understanding the DHIF model; • reviewing the three actual models to see the DHIF components that users have started from; • reviewing the five illustrative DHIF models, starting with the simplest, SMS and mHealth telemedicine, then progress to more complex models, such as EHRs that compare conventional capital expenditure options with public-private partnership (PPP) options; • understanding how combined and aggregated DHIF summary reports can support digital health investment decisions in their strategic context; • having understood the theory, to then build DHIF models for proposed modest digital health projects, estimate the data needed, and identify data gaps and weaknesses; and • reviewing, developing, and improving actual DHIF models as a step to creating models for complex digital health projects.
Building an initial, simple DHIF is a critical step. An effective approach is to complete it as part of action learning. It is a sequence of learning and implementing solutions to real challenges; a form of learning by doing. Applying DHIF to real digital health investment is a continuous action learning process in itself, as new insights and issues keep emerging as models are constructed, reviewed, and developed. Action learning is encapsulated as (Revans 1980): As DHIF models are developed, the number of variables and calculations often increases, adding to their complexity. In this process, it is easy to drift away from the main principles. Appendix 2 sets out a checklist of common errors and, being aware of them, can help to retain the required rigor and compliance.
Understanding these lays a foundation for using DHIF to accumulate knowledge and analyses about actual digital health projects. Its aggregated and comparative information provides valuable knowledge about digital health performance that can inform future digital health strategies and investment decisions.
In-country facilitation of DHIF models benefits from sharing experience and findings. There is much to be gained by sharing knowledge of DHIF analyses and findings.
Benefiting from digital health enables health and health care to change and improve. Using DHIF to build bespoke socioeconomic and financial models within a common methodology helps to quantify, estimate, understand, and assess how health and health care can change to realize the strategic benefits of digital health. It is a complex set of relationships stretching across several stakeholder types, information and communication technology (ICT), health care resources, and finance.

d. digital Health impact Framework methodology
DHIF combines two generic methodologies. One is cost-benefit analysis (CBA) to estimate each option's vFM. This part of DHIF is a version of the modeling methodology used by eHealth impact studies (Dobrev et al. 2008).
The other is digital health finance and accounting to estimate each option's affordability. These rely on cash flow and accruals accounting that recognize two types of transactions. One is the economic events in a year in which they are incurred, regardless of when cash transactions occur. The other represents events that have no cash transactions, such as depreciation. Comparing these financial and accounting estimates with projects' financial provisions in budgets and financial plans reveals their affordability profiles.
These are dealt with in sequence, with CBA first, then financial analysis next. This is an important part of the methodology. It ensures that vFM is identified as the main goal, with financial affordability as an enabler or, when insufficient, an inhibitor. Understanding the associated costing and NMvs as well as the assumptions and estimates (A&Es) involved are then addressed.

Cost-Benefit Analysis
The CBA component seeks each investment option's good, weak, or negative socioeconomic vFM profile. Socioeconomic costs and benefits are estimated over time for all relevant stakeholders. Two main components are health care ICT and health care organizational costs: • health care ICT includes its estimated costs and benefits of liberated resources; and • health care organizational costs include the resources and benefits, mainly of bringing liberated resources into alternative use or uses.
For small-scale investment, such as mHealth telemedicine in a single specialty, reusing the liberated resource can be relatively straightforward and achieved by clinical teams. For large-scale investment, such as hospital-wide EHRs, there can be numerous, small blocks of liberated resources that comprise in total a strategic shift in utilization. Redeploying these as a strategic whole, however, is a demanding leadership and management task. It is challenging to estimate and achieve, and made more complex in health care's CAS context. DHIF helps to develop and appraise scenarios and options for these complexities.
DHIF has several types of classifications for estimated costs and benefits. There are three types of costs and benefits for each stakeholder type: • extra cash needed for extra resources or realized as a benefit through reduced cash outlays; • redeployable resources to support projects, such as health workers' time for engagement, or realized as a benefit by liberating resources such as health workers' time and occupied hospital beds; and • intangible benefits, with no market price, such as lives saved, illnesses and infections avoided, better quality health care, extra travel time, and saved travel time.
Intangibles can be assigned notional monetary values (NMv) using willingness to pay (WTP) techniques. They can also be assigned weightings and scores instead of NMvs. Both need to be used with considerable care. They are a major part of patients', carers', residents' and visitors' benefits. These are described later in section II-F.
Estimated costs and benefits are also classified by stakeholder type: • patients, carers, residents, and visitors; • health workers; and • health care provider organizations.
Four types of cost and benefit distributions are part of the methodology. First, all costs and benefits with estimated monetary values are distributed across the three stakeholder categories above. It requires estimates of the numbers and types of stakeholder in each group. These reveal the extent of investment by health care organizations for the benefit of patients, carers, residents, and visitors. It is a crucial link to health and health care strategies.
A second classification is a distribution of costs and benefits across extra cash needed and saved, redeployed and redeployable resources, and intangible costs and benefits. Rarely does this distribution show that extra cash requirements of health care's digital health investment leads to net cash benefits for health care organizations. Instead, the extra cash requirements usually exceed the cash saving generated as benefits. It shows that digital health is an investment by health care organizations that aims for better health and health care for patients, carers, and residents. These are described in more detail later in section II-H.
Third is a distribution that allocates benefits to one of three categories: quality, access, and efficiency. This is described in detail later in section II-F.
Fourth is the important distribution between iCT and organizational resources. Many digital health investments require organizational resources that exceed ICT costs. Identifying the balance of these resources is essential for success, because the required organizational resources must be in place in full and at the right times along the time scale to maximize benefits realization from the ICT component. Capacity building, change management, and digital health leadership are critical organizational resources.
All estimated costs and benefits with estimated monetary values are adjusted for differences in the value of money over time using discounted cash flow (DCF) to calculate their net present values (NPv). They are then adjusted for sensitivity, optimism bias, and risk exposure to test their reliability, described later in section II-J.

Financial Analysis
Most digital health investment decisions reach an end stage where affordability becomes a critical determinant. Whichever methods are used to finance investment, leaving financial estimates and requirements late in investment decisions can result in projects reverting to their earlier stages to be reset, diminished, or, worse, abandoned. This is not a good use of resources; DHIF brings the financial aspects forward so they can be considered and iterated in parallel with socioeconomic returns, not in sequence.
When the first set of estimated digital health investment components is complete, their costs and financial benefits are used to estimate each option's financial requirements. Two types of financial costs and benefits are cash flow and I&E that uses the accruals methodology. Credit cards illustrate the difference between cash and accruals. In a month, credit card purchases are accruals. Cash is when payments are made, usually the following month in this example.
depreciation has no cash transaction. It is an I&E cost and balance sheet adjustment, and not included in CBA. Transactions are charged against profit and loss accounts and statements and an equivalent reduction of asset values on balance sheets, so there is no associated cash flow. Obviously, it is not included in cash flow estimates. Other financial costs and savings not included in socioeconomic estimates are transfer payments, such as taxes where no transfers of resources are involved, grants, financial donations, and loan repayments. All are excluded from socioeconomic costs and benefits. Donations of real resources such as infrastructure and services are included in socioeconomic costs. Capital expenditure and extra operational expenditure require extra cash flow as part of financing arrangements. They can be partially offset by savings from ending legacy systems.
Capital expenditure and operational expenditure budgets and provisions in financial plans are conventional sources of finance. Grants and donations are included too. Any estimated shortfalls from the available finance reveal affordability and its part of sustainability issues that need addressing before proceeding with digital health investment initiatives.
Some large-scale digital health investment can be achieved with PPPs, minimizing capital expenditure and loan charges by replacing capital expenditure with leases or service charges. These change financing profiles dramatically by reducing health care organizations' capital expenditure and increasing their operational expenditure. It can change socioeconomic returns too and should not be pursued as an option unless projects offer appropriate vFM and are affordable.

Costing and Notional Monetary Values
Assessing both DHIF costs and benefits relies on costing and pricing methodologies. Where market prices are available, information is derived from salaries and wages, invoices, and suppliers' estimates and quotes. Health care budgets and unit costs derived from health care costing models are a common data source for DHIF also. It is important to distinguish socioeconomic costs and benefits from financial and accounting transactions of cash flow and I&E.
Socioeconomic costs for DHIF are classified as direct. They reflect the estimated resources needed to realize the socioeconomic benefits, also classified as direct. The term reflects the close and direct relationship of projects with their resources and benefits. DHIF does not include second-level, indirect costs and benefits. Examples are the possible additional costs of extra carbon dioxide emissions from increased electricity consumption, and the benefits of the ability of recovered patients to participate in economic and social activities, such as charity-raising fun runs, attending concerts, and increasing consumption in the economy.
Sometimes, digital health can enable patients and carers to return to their families and resume work, a second-level benefit. Where this is explicit, it can be included as benefit in DHIF. An example is where patients return to work on their farms and smallholdings that have had to reduce production because of patients' illnesses. Where patients return to work in organizations earlier as a result of digital health investment, there are too many economic and business variables that obscure the effect of their return to work on increased economic production and productivity, so it is excluded from DHIF models. The required level of information and probable outcomes are seldom available for DHIF models used for business cases and retrospective evaluations.
For redeployed resources, health organizations' budgets, annual accounts, and unit cost models provide some of the information needed for employees' time and inpatient capacity. These are not always in a format that can be used for all costs needed by DHIF. Instead, DHIF needs units of resources, such as hospital bed days and outpatient attendances, and their unit costs. There are two main costing methodologies: • total absorption costing (TAC), with allocated direct and apportioned indirect components; and • variable, stepped, and fixed costing (vSFC).
They are essential to estimate tangible health care and organizational costs of digital health and associated investment. They also can be used to quantify efficiency and productivity benefits.
TAC is usually completed for whole organizations or their significant subsets. It uses all organizations' costs in an accounting period, usually a year, and relies on the cost accounting definitions of direct and indirect costs. These definitions are not the same as the direct and indirect definitions used by socioeconomics. The accounting definitions are • direct costs are allocated, and • indirect costs are apportioned using formulas. TAC uses total expenditure in an accounting period, usually a year, to estimate the unit costs of its range of outputs in the same period. TAC is the main methodology for diagnosis-related group (DRG) costs. It relies on two different techniques.
TAC's direct costs are defined as allocated to activities because they are specific to them and there are data about them. An example is prescribed drugs, where patients, their health care professionals and their specialties, and the price of drugs are known. Another example is where wards or clinical specialties are cost centers; costs of their nurses' pay and employers' costs are direct. Where nurses are assigned to several wards, their costs are direct if there is information about the time they spend on each one, which is not usual, and indirect if this information is not collected. Thus, formulas are used to apportion their costs to wards.
Indirect costs cannot be allocated because there is no information about their links to the activities, workloads, and the units to which they relate. They are apportioned to these units using formulas. Overheads, such as a hospital's energy and building maintenance costs are examples of indirect costs. Another example of apportioned indirect costs is patient costing. Nurses rarely keep detailed records of the time they spend with each patient. This means these costs are classified as indirect, and formulas are created to apportion their costs to patients. Overheads rarely change as a result of digital health investment because all resources assigned to each option are accounted for as direct costs of the project. Consequently, digital health costs rarely include increases or decreases to overheads.
The indirect, apportioned component of unit costs using TAC means all its unit costs are estimates. Consequently, there is no such thing as the cost, only a cost. This means there could be other versions depending on the formulas used to apportion indirect costs. Using TAC's indirect costs in DHIF is an illustration of the reliance on estimates in investment appraisal, and the need for sensitivity, optimism bias, and risk adjustments.
vSFC is a different approach. variable costs are linked to units of workload. As workloads change, variable unit costs remain the same, with total spending on variable costs changing in parallel with workload changes, a correlation of +1. For patient costs, prescribed drugs are a variable cost. Prostheses are variable costs for some patient types. Some medical and surgical supplies are another example of variable costs. Information about their actual use by each patient may not be collected, but as patient numbers change, so does the consumption of medical and surgical supplies, as part of variable costs for each patient. variable costs affect total expenditure similarly for both increases and decreases in workload.
Stepped costs are determined by blocks of workload. For example, a physiotherapist may see 10 patients in a session and may be able to see 11 patients without extra resources. Adding another, increasing total workload to 12 patients may not be feasible within the resources available, so an extra whole session may be needed, with costs stepping up until the new session is full. It will be stepped up again when one extra patient is added.
For benefits arising from reductions in activity, stepped costing works in reverse to estimate cost savings. Starting from 20 physiotherapy patients, resources for a session have to remain in place until the number of patients has reduced to 11. Estimating stepped costs in reverse is more complex than going in the more conventional, upward direction. For example, removing resources and their costs can sometimes take longer than adding resources, so the effects can be different over accounting periods and for digital health projects.
variable costs, such as prescribed drugs and prostheses, change with changes in units of output, so they are straightforward to estimate if data are available. Stepped costs are more complicated and may need separate, bespoke cost models for each type of resource to provide data for DHIF. They can be developed in consultation with health workers who provide each service and understand where the steps are.
Fixed costs are rarely relevant for DHIF. By definition, they remain unchanged as workloads change. As digital health investment changes workloads, fixed costs remain the same, so they can usually be ignored in DHIF.
Where redeployed resources are brought into use as a cost, such as treating more patients, their monetary values can be costed using variable costs and stepped costs if appropriate. Costing benefits of liberated capacity using TAC, as described above, is higher than its variable cost. This means using variable costs gives an estimated monetary net benefit, which is a proxy for the available capacity that does not need extra investment to create. The estimated difference can be seen as an estimate of the net value of the redeployable resource.
Costing methodologies are needed for redeployed resources such as shares of existing computer and network capacities. TAC and vSFC costing may be needed for these. Data may not be widely available in some health systems. Therefore, it may not be practical to compile a TAC for the purposes of a DHIF, so A&Es are common practice.
Intangible costs have no market prices, making NMvs necessary. For example, the NMv of extra travel time can be estimated as a proportion of average national or local wages. These are usually relatively small proportions.
NMvs use WTP techniques. For lives saved and illnesses avoided, WTP may have high NMvs. The various ways to estimate these are described later in section II-H, which also includes a description of weighting and scoring, an alternative to NMvs.

Assumptions and Estimates
DHIF relies extensively on A&Es, even when using them for retrospective evaluations when more, but seldom complete, actual information is available. Sources of A&E data for both socioeconomic and financial aspects include • local information systems, such as catchment populations and communities, users, and accounting and costing data; • local research on health needs and probable responses and benefits; • transferrable research for DHIF components; • estimates of WTP from international, national, and local research; and • judgments by teams that reflect the perspectives of all stakeholder types.
It is important that these are explicit and readily available for decision-makers and stakeholders to review, challenge, and change at every stage of their decision-making stages. An A&E sheet in each DHIF model helps achieve this.

E.
The Structure of a digital Health impact Framework model While DHIF's methodology is generic, DHIF models are bespoke. Each one is rooted in its actual health, health care, and digital health context. These define their individual estimated digital health investment profiles. This is the why digital health investments have to be made within a country's health care context.
All options should be based on estimated probable data. Using probable data means DHIF's results are less than the potential of digital health, which is seldom, if ever, realized, as experience from project evaluations suggests. Figure 2 shows an example of three curves of average socioeconomic returns from 36 evaluations (Jones 2014), mainly EHRs. It shows the difference between unsuccessful digital health and good digital health investments, particularly where there is an additional socioeconomic return for high-level, semantically interoperable solutions that span significant parts of health systems, thus offering additional third-party benefits such as reducing research organizations' project time scales.
DHIF models can estimate a probability of proposed digital health investment to ensure that decisionmakers adopt options that avoid weak curves and follow good, sustainable curves. Four important features of DHIF digital health investment curves are the following: • Most begin with negative socioeconomic returns, often -100%, but not always, so decisionmakers need to know which trajectory their digital health investment proposals are following. • Some of the projects in the successful digital health curves in Figure 2 build from benefits carried over or linked with previous projects, which have changed the negative socioeconomic returns to below about -50% by extending some of their change management initiatives into the early years. • For good digital health, very soon after implementation, utilization expands rapidly and substantial benefits are realized to achieve a steep, rising slope that sustains benefits into the medium and longer term. • When digital health projects do not achieve this, benefits are deferred, and may be unable to recover sufficiently to turn their negative net benefit curves toward success, requiring very demanding decisions about their future. These curves are derived from the equivalent of DHIF's set of interrelated processes. Simplifi ed summaries are in . Figure 3 shows DHIF's high-level structure and process that leads to these results and activities.
Financial, accounting, and aff ordability estimates are made after the fi rst round of net benefi ts is completed. Figure 4 shows the high-level process and structure for this part of the appraisal.  Net capital expenditure and operational expenditure requirements are then compared with the fi nance available in each year to determine aff ordability, as Figure 4 shows. Figure 5 shows the iterative process of fi nding an optimal link between aff ordability and socioeconomic return. It may not always be achieved, so another option may have to be selected for investment. Figure 5 summarizes the process leading to the two main types of investment decisions: proceed or stop.
The three high-level diagrams indicate the extent and range of variables that need assembling and connecting in digital health investment. DHIF enables their combined eff ect to be used to fi nd an optimal relationship between them, if it exists. If it does not, then an option, or an entire project, should not go ahead.
Underpinning the high-level diagrams, the following are some of the components of DHIF's structure: • A&Es for stakeholders, which include º estimated numbers of patients, carers, residents, and visitors; health workers; and health care organizations over time; º their estimated socioeconomic costs and benefi ts allocated to each year; and º appraisal components such as sensitivity, optimism bias, risk exposure, interest rates, depreciation method, and price base; • A&Es are copied as formulas and linked cells to costs, benefi ts, and risk adjustment sheets; • estimated costs and allocations to stakeholders and resource types; • estimated benefi ts and allocations to stakeholders, resource types, as well as quality, access, and effi ciency categories; • costs and benefi ts adjusted to NPvs; • NPv adjusted for sensitivity; • NPv adjusted for optimism bias; • NPv adjusted for risk exposure, but only for prospective DHIF models; • estimates copied to summary sheets; and • charts for probable trajectories and distributions.
It is important to remember the following: • setting up a DHIF model relies on an evolving understanding of the health care CAS in which a proposed digital health investment is planned to operate;

Finance available
Where an a ordability shortfall is identified Review socioeconomic return to find viable a ordability If no match found, move to another option or project.
If an optimal match found, move project to next stage.

Iterate
• DHIF needs an understanding of the scope, range, options, components, and use of proposed digital health investment; • these two understandings improve as DHIF models are developed, so they always have a learning component; • consequently, complete DHIF models are seldom, if ever, achieved at the first attempt; they always needs refining; • DHIF models do not provide forecasts and predictions of digital health investments to be made in the future; • instead, they provide a probable trajectory estimated at points in time that projects may follow.
Decision-makers can use the information to take actions needed to change the trajectory and secure affordable, sustainable, and required vFM; and • when adding and deleting rows and columns in A&E sheets, they should also be changed in the costs, benefits, and risk sheets.
Using DHIF for decisions requires DHIF analysts and digital health investment decision-makers to ensure clarity on differences between estimated probable trajectories. The constructive use of DHIF models for prospective investment decisions must reflect their reliance on the rigor of their A&Es.
Uncertainty is inherent in investment decisions. This is due to the reliance on A&Es and exposure to future events of each health system as a CAS. DHIF's probable trajectories are based on A&Es at points in time. These change over time as more information becomes available, and actual challenges and opportunities arise that shift estimated probable trajectories. An example is where unmitigated risks emerge and change estimated costs, benefits, or time scales, and often all three. These can occur both up to the point when the system goes live, and beyond, into the change management and benefits realization time scales. It is widely recognized that past performance is not a guide to future performance, and the value of investment and the benefits deriving from investment can go up and down. DHIF enables decision-makers to assess and monitor the probable effect of these changes as they emerge.
After digital health systems go live, events such as cybersecurity flaws, changing policies and priorities of third party health care payers, and health ministry strategies can create new financial constraints and opportunities. DHIF models enable decision-makers to construct prompt responses to these changes before they occur. Preparing for these adverse changes is why DHIF's probable trajectories are adjusted for sensitivity, optimism bias, and risk. Taken together, they show a band of probable trajectories.
Forecasts and predictions are statements about the future that can imply a higher degree of certainty than probable trajectories. Some are based on facts or evidence, but not always. The terms "forecasts" and "predictions" convey too high a degree of certainty for digital health investment and do not readily reflect DHIF's range of probable trajectories. These can change as digital health projects move through their stages toward operation and benefits realization.
Most important of all, parts of DHIF can be used as independent modules. This is an important facility for new users who have specific digital health investment themes to address. DHIF has already been used in this way, dealing directly with benefits and affordability of digital health strategic scenarios. These are steps 1, 2, 3, and 9 in the DHIF process shown in Figure 6. This illustrates that each step can be a separate model. It is not essential to apply the manual in its entirety. Each user can choose which parts are appropriate for the needs of their specific digital health investment. This approach was adopted for the three live DHIF models.
ii. uNdErSTaNdiNg THE digiTal HEalTH impaCT FramEWOrk mOdEl a. introduction DHIF has several components. Before using DHIF, it is important to understand its process. This section describes the DHIF objectives and each of its 10 steps.

b. digital Health impact Framework Objectives
The five DHIF objectives are • to reach the end point of a digital health investment decision, which is to find an optimal relationship, if there is one, between socioeconomic net benefits, its socioeconomic return, and financial affordability; • to find the investment crux, the point in time where an estimated cumulative benefits curve crosses the estimated cost curve; • to identify the sustainability of the benefit and socioeconomic return curves beyond the crux, or show that it fails to do this; • to compare digital health projects to find an optimal, affordable strategic mix and fit between socioeconomic return and affordability, if there is one; and • to understand which related health care investments can build on and supplement each other.

C.
digital Health impact Framework's 10 Steps These objectives are reached through the 10 dHiF steps that link estimated socioeconomic costs and benefits with affordability and financial costs. It is crucial that these are understood when building DHIF models. They do not proceed as discrete, irrevocable steps. It is normal to return to previous steps as new information and insights emerge during evaluation and decision-making. They are discrete parts of DHIF's process and their use in digital health business cases and decisions. Figure 6 summarizes the 10 steps.
It is not essential to include all 10 steps as a first-time DHIF user. A constructive way to start a DHIF model is to begin with steps 1, 2, 3, and 9. Extra content can be added to this initial DHIF model as appropriate.
It is essential that decision-makers and DHIF evaluation teams are clear on the process and sequence of activities that they will follow. It can help decision-makers, stakeholders, and evaluators to be clear about the changing status of the findings as well as of the events and reviews that can improve each project's estimated performance. A&Es are essential and inevitable components of DHIF models. Each digital health program and project has its own bespoke investment model and A&Es. Reliance on these A&Es requires that they are stated explicitly and accessibly, and provide links to the way they are applied in DHIF models. This is essential for their rigorous assessment and testing.
The 10 steps can be used for comprehensive evaluations of digital health programs and projects proposed for a digital health strategy. To begin DHIF modeling, not all the steps are needed from the outset. Decision-makers can choose which of the steps they want to start with. An example is a Southeast Asian country that had a wide array of digital health projects available that exceeded its affordability limit. An initial DHIF assessment relied on steps 1, 3, and 9. It scheduled the following: • step 1: identify time scales; • step 3: generic schedule of estimated benefits without quantification; and • step 9: estimate the financial costs and affordability of strategic scenarios, adjusted for sensitivity, optimism bias, and risk.
Using DHIF flexibly can provide information needed to make decisions that begin to set affordable strategies. As a generic methodology to create bespoke DHIF models, it is essential that it fits decisionmakers' contexts. The 10 steps enable decision-makers and modelers to select their relevant components to produce the data they need for the decisions they see as appropriate. DHIF is not prescriptive; it can be a set of choices. Refine and iterate socioeconomic returns and affordability to find an optimal link. There are five possible results from a project's DHIF model: • a preferred strategic scenario or project option is identified with an affordable positive socioeconomic return that passes the sensitivity, optimism bias, and risk exposure adjustments; • a preferred strategic scenario or project option that does not pass the adjustment tests and is not affordable; but, after iteration, an optimal relationship is found; • a preferred strategic of project option that does not emerge from iteration and has to be replaced with another scenario or option; • none of the scenarios or options produces a positive socioeconomic return, so the whole strategy or project does not go ahead, even if it is affordable; and • none of the scenarios or options is affordable, so the strategy of project does not go ahead, even if it has a positive socioeconomic return.
The following are the lessons from the three actual DHIF models of how to begin using and applying DHIF. They reinforce the point that simple steps should be taken first but be guided by the overall framework of the 10-step DHIF methodology: • Start small, and expand sophistication toward full DHIFs slowly, such as steps 1, 3, and 9 of the 10 DHIF steps described in section II-C. • Comparing investment scenarios and options is critical for effective decisions. • Data needed are not all readily available, reinforcing the need to be skilled at estimation. • It is important to fit DHIF models into business cases to secure resources. • Having someone in organizations with modeling skills enables a smooth small start. • Digital health initiatives derived from policy and strategic edicts always require investment choices, options, and decisions that benefit from DHIF.

d. identify Time Scales
The first step in a DHIF model is to set the number of years of the evaluation time scale, usually toward, or up to, a digital health project's life cycle. It can be changed later, if it is seen as too short or too long. For a proposed digital health investment, it starts from the current year. Resources and costs before the current year, such as existing computer and network capacity, are excluded. They are classified as sunk costs.
Four of the illustrative DHIF models have a 6-year time scale; that of the EHRs model is 10 years. All five models are set at the initial decision-making stages and have a relatively long lead times to their go live years. An important assessment is the match between the years after go live to each project's life cycle, by asking: is the DHIF life cycle after the go live year long enough to reflect the time needed to maximize benefit realization?
A set of EHRs and ePrescribing evaluations (Dobrev et al. 2010) found that it can take 8-11 years to achieve a socioeconomic return for successful large-scale digital health projects. Some people find it hard to accept that it can take this long. For an initial prospective DHIF model before a decision to proceed is taken, this view of a shorter lead time underestimates or ignores the times and costs of Instead, they often see digital health as just health ICT and refer only to time scales of suppliers, project managers, and implementation.
These understate the time needed for decision-making, planning, development, procurement, and change management. Setting time scales must allow sufficient years to realize significant benefits too or, in the case of weak projects, to demonstrate that the digital health projects will not deliver a material, substantial, positive socioeconomic return, however long it takes.

E. identify Stakeholders
The main stakeholder types include Stakeholders must be segmented so that features like these are identified and quantified explicitly for each type. Then, DHIF models can identify how and when each stakeholder type could benefit, usually not before the go live year, unless there are specific circumstances, such as rolled over benefits realizations and continuing change management from legacy systems.
The illustrative DHIF models show the distribution of costs and benefits across stakeholder types. This reveals an important feature of most successful digital health: health care organizations incur most of the costs, while benefits can be heavily skewed toward patients and carers for several types of digital health investment. It confirms that the primary aim of digital health investment is to enable health care organizations to improve the health of and health care for patients, carers, residents, and visitors.

F. identify benefits
There are several steps: • define the benefits; • identify the beneficiaries, including patients, carers, residents, and visitors; health workers; and health care organizations; • estimate the numbers of each type of beneficiary; and • assign them to the years in which the benefits are estimated to be realized.
When these are complete, they can be quantified using a methodology to estimate their monetary values and the NMvs of the intangible benefits. Using digital health for malaria surveillance as an example, intangible benefits can extend across a range of stakeholders: • all the residents and visitors in the territories affected by the disease, and people beyond the territories and the country's borders; • residents and visitors identified as most at risk; • patients and their carers; • people who are infected; and • people who die from the disease.
Defining who benefits is a top challenge. An important task is defining the cohort of patients, carers, residents, and visitors who benefit. The next challenge is selecting the basis of assigning NMvs to these benefits, described later. It is not the only way, so it is a matter that needs rigorous attention and explicit reporting to decision-makers for final decisions. DHIF teams need to agree on an appropriate approach. A role for the DHIF model is to facilitate an easy process to assemble different scenarios and options and demonstrate their impact.
There are three ways to distribute benefits: • quality, access, and efficiency; • extra resources, redeployable resources, and intangible benefits; and • stakeholder type.
The first way to distribute benefits is made up of • quality, comprising five main components: º better informed patients; º safer health care; º shorter waiting times up to and within health care; º better integrated health care, supported by sharing information; and º more effective health care, including lives saved as well as illnesses, complications, and infections avoided; • access: º extended health care coverage; • efficiency, comprising two main components: º cash releasing savings, including avoided cost of legacy systems; and º improved productivity, such as more patients diagnosed and treated with fewer resources than before digital health investment, leading to redeployable resources.
Many of the five quality benefits are often classified as intangible. For example, there are few tangible measures for better informed patients, as well as better integrated and more effective health care. Shorter waiting times may rely on NMvs of patients' and carers' time, and safer health care may be measured by estimated fewer adverse events or incidents that release estimated resources, but they need NMvs to provide estimated values for patients. Safer health care may lead to reductions in the costs of insurance policies and claims, but requires time to demonstrate these tangible benefits.
Efficiency benefits can comprise a considerable proportion of the benefits of digital health. For largescale digital health projects, benefits are often dispersed in small amounts across a wide range of health care resources. Redeploying them to realize their benefits simultaneously on a large scale is extremely demanding. Several small-scale initiatives can be a more constructive approach. DHIF modelers and decision-makers should consider the most appropriate change management and benefits realization approaches that can lead to successful results.
Access is a measure related to universal health coverage (UHC). It is an estimate of the number of residents and visitors who are unable to access health care, but now can as a result of digital health investment. It is a stringent definition about extending health care coverage and reach toward UHC. Substantial access benefits for unserved communities are demanding to realize many forms of digital health. They usually need considerable parallel investments in direct health care resources for a large impact. Providing the resources needed often depends on redeploying resources from efficiency gains. For DHIF, access is different from shorter waiting times included in quality. Shorter waiting times apply to residents and visitors who already have access to health care, so reduced waiting times before their appointments and within health care are classified as quality benefits.
The second way to distribute benefits is to consider • extra cash; including avoided waste where cash outlay is reduced, such as one-off lower drug stocks, variable costs saved from duplicating diagnostic procedures, less spending on travel, lower staffing levels needed for EHRs, and replaced ICT service contracts; • redeployable resources; including less travel time, saved stepped costs from duplicating diagnostic procedures, and fewer outpatient visits, hospital admissions, and general practitioner visits; and • intangible benefits; including high NMv components, such as lives saved; avoided illnesses, complications, and infections; and lower unit values, such as time saved with fewer journeys to hospitals and health workers who achieve better job satisfaction and information needed for continuing professional accreditation.
Estimates of extra cash benefits can rely on market prices. Redeployable benefits release resources that can be used for other purposes and rely on unit costs. Examples are health workers' time, fewer hospital bed days needed, and shorter outpatient schedules. They are usually created by improved productivity and efficiencies leading to less waste.
For some digital health projects, intangible benefits can comprise a large proportion of total benefits. In these cases, the need for realism and rigor applied to A&Es are vital. This type of result from a DHIF model needs describing and explaining to decision-makers. Three themes are: • identifying if intangible benefits' estimated values exceed the likely outcome materially, • establishing if the timings of benefit realization reflect the likely outcome based on current knowledge, and • identifying if the NMvs distort intangible benefits' proportions excessively so need reviewing rigorously after WTP is applied.
The third way to distribute benefits is by stakeholder type. The usual three classifications are by patients, carers, residents, and visitors; health workers; and health care organizations.
Identifying benefits depends extensively on a range of factors that need assessing before they can become digital health investment components. They include • behaviors of patients, carers, residents, and visitors; • clinical, political, and executive leadership; • health worker capacity to change some clinical and working practices; • digital health, such as º semantic interoperability, º architecture, º functionality, º usability, º utilization, º reliability, and º cybersecurity; • change management; and • users' general levels of digital health and change management capacity and capabilities.
The prevalence and effectiveness of these factors in DHIF models have a significant role in determining and estimating who benefits and how. It is not a direct relationship because health care is a CAS, so their presence is not enough. Their estimated probable impact in their CAS setting is important also. For rigorous investment decisions, it is vital that decision-makers regularly scrutinize and challenge all A&Es in the DHIF models on which they rely. º health care-related resources; such as extra health workers needed for variable and some stepped costs of more treatments and procedures available from efficiency gains, and using more drugs, medical and surgical supplies for more procedures, more prostheses, and increased patients' and health workers' travel spending from better health-and health care-seeking behavior and referral structures; • resources redeployed from other activities, such as doctors, nurses, and pharmacists allocating time for engagement, training events, health workers' travel time change management, and benefits realization; and • intangible resources, such as increased patients' travel time.
Extra resources are a broad socioeconomic classification. Definitions for financial and affordability depend partly on the accounting policies of countries and health care organizations. An example is mobile devices. Each one may cost less than capital threshold values, but, treated as a group, may exceed them, so the group can be treated as capital. Even if they do, countries and health care organizations can still choose to classify them as operational expenditure.
Classification by stakeholder type has to be consistent with the equivalent benefits grouping. It is usually patients, carers, residents and visitors, health workers, and health care organizations.
Another classification is ICT and organizational costs. ICT costs are not always more than 50% of the total socioeconomic costs, making the deployment of organizational resources of equivalent importance. DHIF enables costs to be allocated to each of the two categories. This second step requires information about the specific health care interactions and benefits for patients. Where this is clear in DHIF models, it can provide data to help with absorbing all socioeconomic costs by each type of benefit. Where benefits are expressed in more general terms, TAC apportionments may be spurious and best avoided. Before attempting this, it is essential that significant experience is gained from a set of completed digital health A&Es, their analysis, and their assessment.
ICT resources comprise assets and operational resources.
Organizational costs are mainly operational expenditure and often redeployed from existing budgets. They are numerous, and include • health professionals' budgeted time deployed to digital health engagement, including setting user requirements; • extra health care resources needed to realize the benefits of liberated health care resources, such as more health workers and hospital bed days; • digital health leaders' and decision-making teams' time deployed to proposed projects; • extra health informatics specialists, extra health analytics specialists, extra ICT support technicians; • project managers; • change management and capacity building, often following on from project management but omitted from resource allocations; • digital health trainers for specific projects and cybersecurity; • health workers' time deployed to training health workers' time deployed to benefits realization; and • out-of-pocket expenses of patients, carers, residents, and visitors, such as extra travel costs and the intangible cost of extra travel time.
Change management and health worker capacity building are vital resources for benefits realization. As health care is a labor-intensive endeavor, at the heart of a successful digital health and care economy will be people, not digits (Docherty, Miller, and Patel 2018).
Change management by people leading to benefits realization and health care transformation is the most challenging part of digital health. Allocating resources for change management must be accompanied with a specific change management methodology.
There are many models to select from, some of which overlap. vital themes in selection include • excellent digital health leadership; • change needs time, but for digital health, a prompt, significant rise in net benefits is needed after implementation to ensure sustainability; • building health workers' and managers' capacity to succeed with change must begin and be substantially complete before implementation; • success needs two parallel strands: º hard components, such as realistic time scales, skills and capacity, commitment, and health workers' time and effort; º soft skills, such as culture, leadership, and motivation (Sirkin, keenan, and Jackson 2005); and • most successful transformation is usually at organizations' peripheries and led by general managers and health workers, and deals with specific local problems and challenges (Docherty, Miller, and Patel 2018).
A DHIF context that the models will be applied to is a combination of the types of benefits and changes that can realize them. It creates a matrix, with process, organic, and strategic changes on the X axis and quality, access, and efficiency benefits on the Y axis, shown in Table 1. The distinctions between process, organic, and strategic changes are not always clear cut. Allocations to cells should use the prime impact of each item. Process change is mainly administrative practices, such as registration and access to records. It is structured and methodical, and mainly realizes efficiency benefits.
These realize mainly quality, and possibly some access benefits.
Strategic change is where health care organizations realize corporate goals, such as providing new services and transferring services and resources from hospital to communities. They are mainly quality and access benefits and can include complex redeployments and transfers of resources and workloads between departments, clinical directorates, and other organizations in health systems, such as from hospitals to primary care.
Boundaries between the cells are not rigid; the matrix is more conceptual than a methodology. However, organic changes often have a significant impact, so engaging and supporting health care professionals to achieve their changes is a crucial change management component and essential for sustained benefits realization. These leadership traits are prevalent in extremely competitive and successful teams. Realizing digital health benefits often requires teams to collaborate and integrate, so the traits need adjusting to fit this setting. It is important to recognize that conventional leadership may not be entirely appropriate for digital health benefits realization and change management. Selecting, developing, and applying an approach needs to be handled with considerable care. The Captain Class findings have revealed a complexity in the choice. Where there is complexity, there is risk.
Where existing assets and resources such as spare capacity on networks are used, they need to be identified and their estimated cost included. For ICT resources, these may be challenging to specify, but still need including in resource requirements.
Donated resources need to be included in socioeconomic costs. Examples are equipment, training, and, in some cases, donations in the form of network capacity from mobile phone operators. They are socioeconomic resources for projects. Donations and grants are excluded from socioeconomic costs but included in affordability and finance.
Other divisions are between capital expenditure and PPP options, and capital expenditure and operational expenditure. When compared with sources of finance, they are part of affordability estimates. Another distinction is between health care costs for ICT and organizational resources, such as engagement, training, and benefits realization.
Financial and accounting costs of depreciation and loan charges are excluded from socioeconomic costs. However, it is important to classify socioeconomic costs as either capital expenditure or operational expenditure. The financial affordability part of DHIF models needs this to estimate depreciation and loan charges in line with organizations' accounting policies.

H. Estimate monetary values of Socioeconomic benefits
Estimating monetary values of socioeconomic benefits is DHIF's most challenging activity. Since many digital health benefits are intangible, they rely on each type of stakeholder's WTP as proxies to estimate NMvs.
Some estimates, such as travel cost savings and liberated health care capacity and other resources, have monetary values. Their total monetary values can be estimated based on total unit or variable costs. An example is the reduced average travel costs for a group of patients and carers to visit hospitals from a specific catchment area. Each of these cost savings estimates is a variable cost per person. This is described in section II-I.
Where resources are estimated as liberated, DHIF uses their average cost using the TAC methodology, also described in section II-I. Examples are the average cost of a DRG, cost of a hospital bed day in each specialty, and average cost of an outpatient appointment in a relevant specialty. They represent estimates of the total cost of the resources for an activity or workload when they are liberated by benefits realization of digital health investment.
Other benefits, such as travel time savings and better informed residents and visitors, have no market prices. For these, WTP can be used to estimate their NMvs. People who have the opportunity to access health and health care data may only have minor WTP, leading to small NMvs per person . These may often be appropriate for benefits for patients, carers, and health workers such as better quality and more efficient health care. It is important not to exaggerate stakeholders' WTP for these types of benefits. Patients, carers, and other residents may already expect these benefits to be in place without digital health. For lives saved and illnesses and infections avoided, WTP and the resulting NMvs can be much higher.

Estimating Monetary Values for Lives Saved and Fewer Illnesses
Intangible benefits are lives saved; avoiding infections, complications, and illness; and better health care. There are no market prices or routine costing information available to estimate their monetary values. Instead, there are two choices for A&Es. One is identifying and weighting probable benefits. Weighting, as is described below, provides insights to the relative scale of NMvs to be assigned to each benefit.
The other is selecting a technique to estimate the NMvs of the benefits. These are needed for each type of beneficiary, not just the health system. There are several methods to estimate some monetary values. They are all variants of WTP as an estimate of prices in the absence of market prices, and are proxies for beneficiaries' views of the value to them. Estimating WTP can include a structured survey of some health workers to identify their WTP for digital health. It is more challenging to survey a group of patients, carers, and other residents.
For lives saved and illnesses avoided, WTP may have high NMvs. There are several ways to estimate these. They include • lives Saved Tool (liST), used for survival of mothers and children (Johns Hopkins School of Public Health 2017); • quality-adjusted life years (QALYs); • disability-adjusted life years (DALYs); • life insurance values; and • earnings, such as average and ranges of income, and proxies of gross domestic product (GDP) or gross national income (GNI) per head.
Each of these has limitations and drawbacks. Estimated NMvs must be tested rigorously before they are used for digital health investment decisions. Then, decision-makers must challenge them at all decision stages to ensure they reflect a realistic and corporate view. Like all estimates and proxies, they are only indicative.
This information can inform decision-makers about the balance of impacts as a subset of overall socioeconomic returns, but their use must recognize the limitations and appropriateness of each one. At the initial stage of digital health investment decisions, an appraisal is prospective, so there is seldom specific, hard evidence about the probable benefits and their realizable extent and timing. This is an important factor in choosing a methodology. Using the information for decisions must reflect the limitations of the A&Es available. It may become more pronounced as the scale, scope, and complexity of the type of digital health investments increase.
Relative weighting can be used where WTP is seen as too unreliable. It can produce different results for WTP for benefits such as lives saved, infections avoided, and better quality health care. It indicates the effect of the choice on socioeconomic returns of valuation methodologies for intangible benefits.
Either weighting or NMvs are appropriate, but it is important to be aware that the choice may affect benefits, net benefits, and socioeconomic returns. Figure 7 illustrates one such difference. It is unlikely to affect affordability, which relies on estimates of tangible extra costs and financial benefits such as cash flow savings.
Identifying NMv criteria must be clearly defined so that appraisal teams and decision-makers can understand, challenge, change, and eventually accept them. They must be defined in service or outputoriented terms, and reflect strategic objectives and performance measures. Methodological errors also have to be avoided. Good practices include • not double-counting by overlapping criteria; • avoiding general, over-used terms that have several different components, such as health care quality; • not double-counting benefits included as costs, such as reliability, which is part of maintenance costs; • including all relevant criteria, both common to all the options and specific to one or a few options, so as to avoid distorted results; • excluding confounders, sometimes called mixing of effects, where the effects of investments on their outcomes are mixed in with the effects of additional factors that distort true relationships; • avoiding benefits drift, where extra benefits that are not strategic are included to show or increase positive socioeconomic returns; and • recognizing that some stakeholders may not perceive benefits from digital health, but see benefits as a basic requirement of a good health service that they already expect without digital health.
While relative weighting is mainly applied directly to benefits without NMvs, the illustrative DHIF models use the technique to estimate both unweighted and weighted unit costs of lives saved and illnesses avoided through digital health investments' total estimated socioeconomic costs. The effect is to show how WTP with NMvs reduce the unit socioeconomic costs of digital health per patient and carer.
Most digital health investment has an array of intangible benefits. Assigning estimated socioeconomic costs to each type requires extremely detailed information. For example, this applies to reliably identifying shares of each type of ICT capital and operational costs for a range of benefits such as lives saved, infections avoided, and better health care quality without details of digital health utilization linked to each benefit type and their reliance on the share of resources. These types of estimates are challenging also at the monitoring and evaluation stages. In earlier DHIF stages, there are seldom, if ever, any data to help with the endeavor.
Estimating the net benefit of NMvs needs a TAC methodology to estimate the total share of estimated costs for NMvs for each stakeholder group. It requires a complex, sophisticated TAC model, and is not part of DHIF. It is complex. The two steps necessary are: • allocate costs that are directly attributable to a benefit; and • apportion costs to benefits using a formula, similar to overhead apportionments.
This second step requires information about the specific health care interactions and benefits for patients. Where this is clear in DHIF models, it can provide data to help with absorbing all socioeconomic costs by each type of benefit. Where benefits are expressed in more general terms, TAC apportionments may be spurious and, so best avoided. Before attempting this, it is essential that significant experience is gained from a set of completed digital health A&Es, as well as their analysis and assessment.
There are more sophisticated WTP techniques such as LiST, QALYs, and DALYs. Other approaches use life and illness insurance data, and also earnings and pensions. There are important issues of comparability and weighting and scoring benefits to be considered too. All these more detailed approaches are considered further in Appendix 5.
A simple example of a QALY calculation and its DCF calculations for NPv and equivalent annual cost (EAC) is where the time scales of options are different. QALYs use estimates of extra life expectancy and health-related quality of life to assess the unit costs (Sassi 2006). Cost effectiveness analysis uses QALYs to assess improvements from specific health interventions, such as a new drug. Table 2 shows an example of simple estimates of QALYs. For investment options with different time scales, DCF uses an equivalent annual cost, instead of NPvs.
In the example, both methods, NPv and EAC, show that drug B offers the best vFM, but, if only $10,000 is available to spend, drug B is not affordable. This triggers a set of iterations and options for decisionmakers that is common in many digital health investment decisions to find an optimal relationship between socioeconomic net benefits and affordability.
There is also a choice between NMvs and weighted and scored benefits. As they are different techniques, the results can differ. A comparison of the two techniques can be presented between preferred strategic scenarios and options of different projects. Decision-makers can then assess the comparisons, and decide which one to use in selecting strategic scenarios and projects' options to go on to the next assessment and decision stage.
The illustrative malaria surveillance DHIF model includes estimated weightings and NMvs for these intangible patient and carer benefits. NMvs' distribution of benefits is very different to the weighted number of beneficiaries. Table 3 shows an example.
continued on next page  Table 3 shows differences between the two approaches, illustrating the need to test and review both methods before using the data for firm digital health investment decisions. While the average percentages for estimated NMvs and weighting in Table 3 are the same at 11%, the distributions are different. For NMvs, "People at risk not infected" exceeds the average +1 and +2 standard deviations. For weightings, two values exceed the average +1 standard deviation and all are within +2, with one value lower than average -1 standard deviation. Weighting has moved "People at risk not infected" to the other end of the distribution.
The National Health Service in Scotland (NHS Scotland) has described a weighted scoring methodology (Government of Scotland 2009, pp. 73-78) that can be used in two ways. One is to estimate the cost apportionments to benefits to show relative net benefits across part or all of the benefits profiles. The other is where NMvs of intangible benefits are not available, and seen as too contentious and unreliable to use.
Weighted scoring is a multi-attribute or multi-criterion analysis. When used to compare options, it requires • identification of all of a project's strategic intangible attributes or criteria; • allocation of weights to each of them to reflect their relative importance; and • allocation of scores to each option to reflect how they perform in relation to each criterion.  Combining the results gives a single weighted score for each option to compare their overall performance in nonmonetary terms. The illustrative DHIF models use the first two steps, but not the scoring step, to show how appraisals can be used without estimated NMvs for intangible benefits. The same approach can be used to evaluate the nonmonetary aspects of strategic scenarios.
The core principle is that weighting is a stakeholder endeavor to reach a set of team judgments. Team members should represent all stakeholders, especially those directly affected by a project and those responsible for its delivery. It is not a matter just for experts. Each individual's weightings have the same value for decisions as those of all other team members.
Weightings are assigned after team discussions, but not necessarily after reaching consensus. These can be reviewed as a team, with each individual able to change their weighting in second and subsequent rounds, similar to a Delphi methodology. An independent team leader should oversee the process, probe opinions, promote consensus where appropriate, and avoid prejudice and inappropriate influence by individual stakeholders.
It is vital that teams understand the methodology and its process. Before starting the weighting, a description of the process and methodology should be prepared, which includes a comprehensible explanation for the rationale for weights and scores, and how they will be used and tested for robustness. If this is not completed effectively and understood by participants, it can cause delays, confusion, and inappropriate decision-making.
Weighting assigns numeric values, not NMvs, to judgments. These judgments should not be arbitrary or subjective, but reflect stakeholders' views supported by objective information. Achieving meaningful results that decision-makers can rely on requires teams • with the relevant knowledge and expertise needed to make credible assessments and judgments of projects and each option's probable impacts; • that can explain their justification for their weights and scores; and • that can describe the weights or scores of dissenting individuals, where there is a lack of consensus.
The six steps are: • identify the relevant intangible benefits and criteria from health and health care, and supporting digital health strategies; • weight criteria to reflect their relative importance and value to the strategic goals; • score the options to reflect their performance against each criterion; • calculate weighted scores; • test results for robustness; and • interpret results.
In the illustrative DHIF model for malaria surveillance, the weighting is applied to the number and types of patient and carer benefits in the benefits sheet. Table 4 shows an example.
The estimated weightings column shows the relative weighting. While people may agree that fewer patient deaths should be weighted more than better informed patients, not everyone will agree that fewer deaths should be weighted five times more. This illustrates the need for each individual's weightings to be combined for a team distribution and reviewing the results.
Having agreed weightings with the team, the data can be used to estimate unit costs by dividing the total socioeconomic costs by the specific weightings, such as lives saved and infections avoided. This technique has a significant drawback. Costs are related also to other tangible benefits, so they need to be assigned to benefits, which is a complex methodology, as described earlier in section II-H.1.
There are two types of comparisons using these unit costs. One is between options for a digital health project to identify a preferred option; the other is between preferred options of different projects to guide decision-makers in choices seeking an optimal socioeconomic return for their combined strategic digital health programs. Decision-makers may want to choose the scenario, option, or project with the lowest unit cost for specified strategic benefits.
Estimates from the "summary" sheets of the DHIF illustrative model for SMS and malaria surveillance show how this can be presented to stakeholders and decision-makers. They show that lives saved as a criterion may not be a sufficient criterion. Life-threatening infections and illnesses avoided may have an equivalent importance for health and health care strategies. These perspectives can be reflected in relative weighting and scoring. Table 5 shows an example of the different effects of NMvs for lives saved and infections avoided. This direct comparison is not sufficient for decision-makers and stakeholders. It should be accompanied by the equivalent data adjusted for sensitivity, optimism bias, and risk.

i. Estimate Socioeconomic Costs
Establishing socioeconomic estimated monetary values to assign to each type of cost has to comply with a definition that differs from conventional, probably more familiar, financial and accounting definitions. Socioeconomic costs exclude transfer costs, such as unrecoverable value-added tax (vAT) and other taxes because there is no exchange of resources. They also exclude grants and cash donations, so the full cost of resources is included. However, donated resources, such as equipment and services, are included.
Depreciation and loan charges for conventionally financed projects are excluded, since these are part of the financial and accounting costs. For PPP options, suppliers' financing costs are included in their prices, often in leases and service fees.
For redeployed resources, health care organization budgets and annual accounts provide some of the information needed, such as employees' time as well as outpatient and inpatient capacities. These are rarely in a format that can be used for all costs. Costs of extra resources needed are derived from estimates of planned ICT costs from sources such as invoices, supplier quotes, and feasibility studies.
Extra health workers' whole time equivalent (WTE) costs are estimated using time, pay rates, and employers' costs, and include employers' pension and social security contributions. Estimated travel, health care costs, and any lost wages of each type of stakeholder are relatively straightforward to estimate using either specific costs or averages. Estimated costs per day or hour should reflect the number of working days in a year. It excludes holidays and average sick leave. An example of working days in a year is about 220 when weekends, national holidays, and annual leave entitlements are excluded. It can be reduced further with a reduction for average sickness rates.
Redeployed resources need various types of cost data. Costs of redeployed health workers, such as those participating in engagement, can be estimated using their respective employers' costs. Other resources, such as reusing liberated health care capacity, require information from costing models. It is a common data source for DHIF, but not one that is always readily available. Redeployed resources can be costed at their variable cost taken from vSFC models. It helps to reveal an estimated net benefit compared with the estimated monetary cost of liberated resources that uses their total unit costs from a TAC model. The difference between them is a proxy for the avoided investment cost of additional health care capacity without digital health. The key approaches to estimating socioeconomic costs have already been described in section I-D.3. They involve the use of vSFC and TAC, and consideration of intangible costs. Socioeconomic costs of digital health are classified as direct. They are the estimated resources needed to realize benefits which are also classified as direct. The term reflects the close relationship of the resources and benefits of a project to the main stakeholders.
Second-level costs and benefits are indirect, and not included. Examples are the possible additional costs of extra carbon dioxide emissions of increased electricity consumption and the benefits of having recovered patients able to participate in social activities, such as charity-raising fun runs and attending concerts, and return to work in large corporate organizations where daily workforce fluctuations seldom change economic outputs.
A wide range of data sources is needed to populate parts of a DHIF model. An example is WHO's annual malaria report (World Health Organization 2017b). Another is the Asia eHealth Information Network (AeHIN) costing tool used for OpenHIE implementation, available at http://sil-asia.org/lab -assets/ the AeHIN Standards and Interoperability Lab for Asia (SIL-Asia) initiative.
When using monetary data from sources like these, it is important that they comply with the DHIF methodology, such as a consistent constant price base and consistent definitions of economic and financial costs. Statistical data and trends about populations and communities may only be available up to 2 or 3 years before the first year of DHIF models, so they may need extending and adjusting to bring them up to date and estimate them into the future.

J. adjust for Sensitivity, Optimism, and risk
As mentioned previously, the World Bank Digital Dividends report (World Bank 2016, p. 165) identified about 30% of public sector digital technology projects as totally failing. Another 50% to 60% are partially successful. It is unwise to assume that digital health projects significantly exceed this performance.
All costs and benefits in prospective appraisals rely on appropriate and realistic A&Es. These are tested using three separate adjustments: sensitivity, optimism bias, and risk exposure. The effects are to increase costs and reduce benefits systematically to see if proposed projects can retain their estimated socioeconomic returns. Table 6 and Figure 8 illustrate the effects in the generic DHIF model for immunization information system (IIS) for a small state.
In this example, the project retains its socioeconomic return after adjusting for sensitivity. The optimism bias adjustment dips into negative territory, then returns to positive, thus passing the test. Its socioeconomic return turns negative when adjusted for risk exposure, failing the test. It highlights the critical need for a rigorous risk mitigation strategy and plan.
For prospective DHIF models for proposed digital health investment, all three adjustments are applied separately to the unadjusted NPv. For retrospective DHIF models, the effects of risk exposure are subsumed into the estimated costs and benefits, so are not applied. Only sensitivity adjustment is needed to reflect the reliance on A&E. Optimism bias adjustments may be needed, depending on the degree of estimation needed for costs and benefits.
DHIF models for existing digital health services can be built from retrospective evaluations into prospective, estimated trajectories. For these, optimism bias and risk exposure are applied only to the prospective estimated costs and benefits.
Sensitivity testing aims to reveal the extent to which estimated positive socioeconomic returns rely on the rigor of the estimates used for their costs and benefits. A simple sensitivity derived from the experiences of investment decision-makers can test the effect of relying on the A&E as sources of estimated socioeconomic return. If the adjustment results in a negative socioeconomic return, the A&Es need reviewing and resetting to be justified and more achievable, if they remain feasible.
NPv = net present value. Years 1-5 shown, years 6-10 not shown, total to years 1-10. Source: Digital Health Impact Framework illustrative models for immunization information system for a small state, summary sheet, 2018. Table 6 continued Optimism bias adjusts for organizations' and DHIF users' inherent tendencies to underestimate costs of, and overestimate benefits from, ICT projects. Adjustments to costs reflect the degrees of ICT tools' coding and functionality complexities in projects. These adjust benefits by deferring the estimated go live year and related benefits. If the result is a negative socioeconomic return, the project needs reviewing, revising, or abandoning.
risk exposure estimates the cost and probability of risk. Its adjustment often results in a negative socioeconomic return. Where a positive return is retained and is materially lower than the unadjusted return, the next step for both results is to set up a risk mitigation plan, and reflect its planned actions in the risk exposure model until it converts the adjusted negative socioeconomic return into a positive. If a positive risk-adjusted socioeconomic return is not achieved after this, projects should stop.
Two important aspects of estimation are Occam's razor (Encyclopaedia Britannica 2017) and outcome switching (Prescription Intelligence 2016). These can be addressed by the three adjustments.
Occam's razor, sometimes called the law of parsimony or law of economy, is an antidote to exaggerated claims. It says "plurality should not be posited without necessity." The principle gives precedence to simplicity and evidence. Of two competing theories, the simpler explanation of an entity is preferred. Entities should adopt a process of paring down information to make finding the truth easier. The tests can help reveal where this degree of rigor has been disregarded.
Outcome switching must be resisted. It is usually linked to a parameter of a clinical trial adjusted after the results are known to improve the numbers and results. It can occur in digital health investment too. Some adverse outcomes are not reported; or some positive, but trivial and nonstrategic, outcomes are added or switched after the fact to enhance net benefits and socioeconomic returns.
Attempting the equivalent for DHIF is tempting, but should be resisted when a socioeconomic return is not big or early enough. It is acceptable in some circumstances, provided the extra benefits are realistic and become strategic. When it happens, they are usually intangible or nonfinancial benefits, and extensively reliant on A&Es. Some digital health projects have unexpected benefits, but it is better to identify these impartially, and at the outset. Where extra benefits are added legitimately, they should be identified clearly along with their A&E methodology. Sometimes, they can increase total benefits to an extent that they pass sensitivity, optimism bias, and risk exposure tests. What must be avoided are projects lacking strategic fit, proceeding when they should either be extensively modified or abandoned.

Sensitivity Analysis
Sensitivity analysis has two main methodologies to identify how uncertainty of underlying mathematical models' outputs can be apportioned to different to inputs and outputs: • deterministic, with causal relationships with no random variables, aiming to represent changes in an underlying process, such as a CAS, and empirically estimate relationships between variables to produce an approximation of reality; and • stochastic, with a random probability distribution or pattern that may not be predicted precisely, so the adjusted model creates a projection of outputs based on a set of random values.

Figure 8: illustrative immunization information System Comparison of Estimated Cumulative Net benefits and Sensitivity, Optimism bias, and risk adjustment
NPv = net present value. Source: Digital Health Impact Framework illustrative models for immunization information system for a small state, "ser sens opt risk" sheet, 2018. Four main ways to test A&E for sensitivity that can be used within DHIF are • one-way sensitivity analysis, varying one estimate at a time to see the effect; • multiway sensitivity analysis, varying more than one parameter at a time, with the advantage of producing a more complete representation of uncertainty; • set all parameters to their estimated extreme values; and • probabilistic sensitivity analysis, a stochastic model which captures a representation of the uncertainty for each component of a model, such as Monte Carlo simulation.
Empirical experience with some digital health investment decision-makers is to test A&Es using large changes to total cost and benefits, using extreme values. Drawn from their experience, their preferred adjustments have been increases in costs of 50% and reductions in benefits of 50%.
A separate adjustment is to increase the DCF discount rate to 6%. A typical rate is between 3% and 3.5%, but some countries use higher rates. This sensitivity adjustment doubles the rate to test its effect on socioeconomic returns. With relatively low discount rates of 3% to 3.5%, this test seldom produces much of a change in net benefit. If it converts positive socioeconomic returns into negatives, the unadjusted socioeconomic returns were probably too low to offer a viable project, and would fail the main sensitivity test, as well as for optimism bias and risk adjustment. Where NPvs of net benefits have a large gap from their X axis, the test is rarely worth completing.

Optimism Bias
Optimism • 40%, when digital health system and interfaces are standard products already fully developed and proven, with practically no new coding; • 100%, when digital health system and interfaces use several standard applications, and add or develop further functionality requiring a substantial degree of new coding; and • 200%, when digital health system and interfaces are new and untried, requiring a high degree of new coding.
These cost increases also affect time scales. Go live dates are deferred too, so benefits are deferred to match. The percentage deferral broadly matches selection from one of the three upper limit rates of 40%, 100%, and 200%.

Risk Exposure
Risk is a complex phenomenon. While it is assessed for each cost and benefit component and time, in practice, these do not change in isolation. They change as part of a set of variables that create other changes attributable to the same risk factor. Training is an example. If this slips in time or is inadequate, it can knock digital health utilization, change management, and benefits realization out of gear. A similar adverse effect can result from ICT or software that does not meet user needs, and ultimately minimizes and defers benefits realization.
It is feasible to construct complex risk exposure models that include the relationships between the ranges of variables, but it is complex. Instead, DHIF uses a simpler risk exposure model that includes socioeconomic cost and benefits as well as financing risks. It is a combination of the estimated costs of each risk multiplied by the probability of the risks materializing. The result is risk exposure. It reveals the need for a rigorous risk mitigation strategy and plan. At the early stages of DHIF models, risk probabilities are high, usually above 50% for costs and below 50% for benefits.
Information about risk probabilities is limited for digital health. The main sources are the few comparisons of retrospective and prospective evaluations of digital health services. Experiences from other countries are valuable. An example is NHS England's National Programme for IT, now largely abandoned after costing approximately £12 billion ($16 billion as at 27 May 2018), plus its considerable, unestimated redeployed cost of hospital resources across the country. The project did not result in realized benefits anywhere near equivalent to the scale of costs (National Audit Office 2011) and illustrates the extent of adjustments needed for digital health investment risk exposure.
Risk-adjusted socioeconomic returns often result in negative numbers. This does not mean that proposed projects should be abandoned. It means that if risks are not mitigated, projects will probably run into insurmountable difficulties that could be catastrophic. Effective risk mitigation strategies, plans, and activities will help to minimize this outcome, but rarely eliminate risk completely.
After decisions are taken to proceed, projects' DHIF models can be updated during projects' time scales, including after implementation, even once started, to test if they are still on track and if net benefits are realized and risk exposure is being mitigated adequately. From the planning, development, implementation, and operational stages, the models eventually provide a foundation for monitoring and evaluation. The chances are that progress will not be on track in some respects. Deferred go live dates, referred to as slippage, are a common phenomenon. It is not a risk but a manifestation of several risks and these need identifying and mitigating. DHIF models can identify where corrections are needed and help digital health leaders and other decision-makers determine and monitor their responses.
A common cost curve encountered by ICT projects is where costs increase when software bugs are fixed later in their implementation cycle. Figure 9 shows the changes in fixing costs as software projects progress across their time scales.
DHIF can identify and quantify some of these phenomena and help ensure that bugs are fixed early and promptly. An example before go live is when stakeholder engagement has been inadequate, slowing down decisions. Another is where user requirements and usability levels are lower than required and likely to lead to underutilization with a knock-on delay, more costly changes, and lower benefits realization. A sound testing environment, for example in standards and interoperability laboratories, can help address these issues and enhance net benefits, so socioeconomic returns improve affordability.

k. Calculate Net benefits, the Socioeconomic returns
For some digital health services, such as EHRs and surveillance, better data quality is an important foundation for benefits. Its components include relevance, accuracy, completeness, reporting volumes, availability, and frequency. A separate risk assessment can be completed by comparing estimated benefits for a target quality level, such as 95%, with a level where benefits break even with costs. Where estimated benefits tumble with a small reduction from the target, risk exposure is high; where the breakeven is well below the target level, risk exposure is low.
Assigning costs and benefits to years depends on project plans and estimates of digital health utilization that leads on to benefits realization. Usually, all benefits accrue in the period after implementation and during the operational phase. Their phasing can match estimated changes in user and beneficiary numbers. Occasionally, benefits can carry over from legacy systems and be enhanced before the go live date.
All costs and benefits are adjusted for the time value of money. DCF, using a discount rate of about 3%, a social time preference rate, provides the NPv of costs and benefits of monetary values (Edejer et al. 2003, p. 70). It is the approach used by the WHO cost effectiveness and strategic planning (WHO-CHOICE) team (WHO 2017a;2002, pp. 106-107).
DCF compares costs and benefits that occur in different time periods by converting them to NPvs. Applied to the social time preference, it follows the principle that people generally prefer to receive goods and services now rather than later. It is measured using a social time preference rate, usually 3% or 3.5%, but possibly higher in some countries. It is an estimate of the rate at which society values the present compared to the future.
The discount rate reduces the value of future costs and benefits in present-day terms. If society has a rate of 3% a year, it values $1 received today as needing $1.03 to be received in a year's time to retain  Table 7 shows an example of estimated costs and benefits over 6 years at their NPvs.
The estimated NPv of costs over 6 years is $29,604,708 after using a discount rate of 3% a year at annual intervals. In DCF, the first year is assigned as year 0, hence not discounted. DHIF models use the DCF formula. A discount formula function from a spreadsheet can start DCF as year 1, so that year 0 is discounted. It is important to use these embedded formulas in DHIF calculations starting from a project's year 2 to leave DCF year "0" not discounted with an NPv rate of 1.000. Table 7 and the illustrative DHIF models are in real terms, so they exclude future estimated general inflation. This is a requirement for DCF, which disregards general inflation. It helps to see real changes in resources and benefits across the years. Adjusted costs and benefits for sensitivity, optimism bias, and risk exposure are also expressed in NPvs.

Estimated costs and benefits in
When these are complete, each year's estimated NPv costs can be deducted from the estimated NPv benefits to provide the annual socioeconomic returns, measured by net benefits and the socioeconomic return as a percentage. It is often -100% before benefits are realized. The total vFM at the end of the time period provides a project's estimated socioeconomic return.
Other information can be shown alongside the net benefits to reveal digital health projects' specific features. Examples are • estimated unit socioeconomic cost of a life saved; • estimated weighted unit socioeconomic cost of a life saved; • estimated unit socioeconomic cost of illness or infection avoided; • weighted unit socioeconomic cost of illness or infection avoided; and • estimated unit socioeconomic costs and benefits for patients and communities.

l. Estimate Financial Costs and affordability
Affordability is often a major constraint on digital health investment. Critical investment criteria are the levels of cost and socioeconomic return that health care can afford. Consequently, socioeconomic costs and benefits without their financing and affordability estimates over the strategic or project time scales will not support effective decision-making.
Digital health investment's socioeconomic and financial costs are not exactly the same. The first step is to convert appropriate socioeconomic costs into their financial equivalents. Socioeconomic costs reflect monetary values of resources, including NMvs. They exclude transfer payments where no resources are involved, such as unrecoverable vAT, grants, and cash donations. They also exclude health care organizations' loan repayments, and depreciation, an accounting transaction where no cash changes hands. An estimated value of donated equipment and its maintenance should be included in socioeconomic costs.
Maintenance of donated equipment should be included in financial costs, together with grants, unrecoverable vAT and other taxes, and cash donations. Depreciation should be included only in financial and accounting costs, not in cash flow estimates, because depreciation is an internal accounting charge to profit and loss accounts and a reduction in asset value.
Two other financial measures are receipts and payments, the components of cash flow. Both need information drawn from extra socioeconomic capital and operational costs and benefits, adjusted as described in the previous paragraphs. For cash flow, spending can be matched to cash flow budgets and financial plans. For I&E, an accounting accruals concept, depreciation, loan charges, and any creditors and debtors needs to be included.
Depreciation is not a cash transaction. It is an annual cost charged against profits, if there are any, or increases losses. It should be determined using the relevant organizations' depreciation policies. The simplest is straight-line depreciation, where net asset values are divided by each asset's estimated life years. Net asset values can be estimated as equal annual costs of the asset, minus its residual value at the end of their useful lives. For ICT, estimated residual values are often minimal, largely due to the effect of obsolescence.
Loan charges are calculated using the estimated borrowing requirement and interest rates for capital expenditure for the appropriate health care organizations. They include annual principal and interest repayments. For PPPs, these are usually included in partners' leasing or service charges, so they are already part of the cash flow and I&E requirement described above.
Debtors and creditors may or may not be relevant. An example is recoverable vAT. It may be paid by health care organizations toward the end of 1 year, but not recovered until the start of the following year. This creates a debtor for I&E, so the cash receipt in the latter year should be included in the preceding year's I&E, but not the previous year's cash receipts. Where resources are received in one year, but not paid for until the year after, the costs should be included in the first year's I&E as a creditor, with cash payments in the following year. It is a simple example of accruals accounting.
Redeployed assets and resources from existing budgets can be excluded from financial and affordability estimates because they are already financed. An example is the time and costs of doctors, nurses, and pharmacists redeployed from their existing activities to be part of digital health's essential stakeholder engagement and project teams. The costs of any locums needed should be included.
Patients, carers, residents, and visitors have affordability constraints also. Their costs and available finance to pay require estimating as well. These can include payments for extra drugs, medicines and surgical supplies, hospital and primary health care charges, and travel costs. If these stakeholders cannot afford these, it diminishes benefits, so socioeconomic returns.
Other agencies involved in digital health investment, such as social care, can be stakeholders. Their financial costs and affordability need estimating when they are included in projects. These should be calculated as part of the stakeholder engagement process, so that overt and explicit decisions can be taken that feed into subsequent digital health decisions.
Each year's total estimated financial capital and operational expenditure of stakeholders can then be compared with their finance available, including grants. They can have different time scales, creating a considerable challenge for affordability assessments. Digital health life cycles are usually much longer than health care organizations' financial plans, which are often limited to about 3-5 years. While this may match small-scale initiatives, such as mHealth, for large-scale digital health projects, such as EHRs, their life cycles extend several years beyond financial plans.
It is common for health care organizations' financial plans to look no more than 3 years ahead. Assessing digital health investment's affordability requires a longer time horizon, so A&Es are needed that can extend the initial financial provisions beyond their shorter time scales.
Assessing affordability uses both cash flow and I&E. There are two assessment questions. First, is there a shortfall on the total capital and operational finance available? Second, is there a shortfall on specific years? If either of these displays a deficit, then decision-makers need to iterate the socioeconomic and financial components to find an optimal relationship where affordability is achieved. This can include seeking and acquiring more finance.
This is a common situation for digital health investment to reach. It shows DHIF's value in two respects. It enables early identification of affordability, a crucial sustainability issue that needs resolving. DHIF models also enable A&Es to be reviewed and changed relatively easily, and new or refined affordable options to be set up and modeled efficiently and speedily.
DHIF's steps 3 and 9 identify benefits, and estimate financial costs and affordability. They have been used to assess the affordability of a digital health strategy in a small state. Digital health affordability is always challenging, but especially demanding to deal with in small states and low-income countries where digital health capital and operational costs can require large proportions of national health care budgets.
The DHIF model for digital health strategy affordability evaluates three scenarios. Scenario 1 is a comprehensive digital health strategy. It is not feasible to provide the non-recurring finance needed. Scenario 2 modifies some of the digital health content, but it does not improve the affordability sufficiently. Scenario 3 is a major reduction to the strategy. It includes only three main initiatives: • foundation investment, which provides no benefits; • EHRs, which provide some quality and efficiency benefits; • laboratory test ordering and reporting, which provide considerable benefits for health workers and patients, because the information content is the most used of all digital health data types; and • a combined strategy of EHRs and laboratory information, which offers the best value for money because it aggregates the benefits and maximizes the utilization of the foundation investment. Table 8 shows the estimated result.
Scenario 3 is more affordable than scenarios 1 and 2, but still not entirely affordable. EHRs and labs individually have enough recurring finance, but insufficient non-recurring finance. A combined strategy offers better foundation investment utilization, but is still not affordable. It offers better value, partly by better utilization of foundation investment, and reveals a common investment challenge: should value be reduced to match affordability, or should finance be increased to realize better value? The next stage is a combination of review and reducing digital health content, and the MOH to provide additional finance.
There are no right or wrong answers; only strategic choices.
m. refine and iterate Socioeconomic returns and affordability to Find an Optimal link, if One Exists The first question for decision-makers is; can the extra socioeconomic costs be reduced to improve affordability while retaining the required benefits? If it cannot, a second question is; can the financing model be changed to increase the finance available to achieve affordability and retain the required benefits? If this cannot be achieved, a new set of options are needed. This can proceed along the following sequence: • reduce socioeconomic spending to match affordability and reduce benefits in line with reductions in digital health coverage, functionality, and human and ICT capacity to identify its socioeconomic return and adjustments, often creating a suboptimal option; • if the socioeconomic return is not acceptable, create different options that comply with affordability constraints; • if these socioeconomic returns are not affordable, seek additional finance; and • assess the impact of each of these on subsequent, dependent digital health projects.
Solutions can mix these approaches. It is appropriate to do this where iteration proves intractable from socioeconomic and financial perspectives.
A common option in response to affordability constraints is to defer ICT spending, especially capital expenditure. This can alleviate short-term affordability constraints, but not longer-term constraints. Like optimism bias, extending resource and cost time lines can also defer benefits, reducing socioeconomic returns for the selected period.
Another option is to seek PPPs. These can reduce capital expenditure, but usually increase operational expenditure, and they may not address recurring affordability challenges.
Digital health has its own dynamics. Each project has its own dynamics that often lay foundations for subsequent projects. When addressing affordability challenges, it is vital that any changes made to comply with a digital health project's affordability requirements do not exacerbate affordability and socioeconomic challenges for other current and future projects. This is a complex set of relationships to deal with, often needing connected DHIF models for a dynamic investment stream. Each of these should be revealed in digital health strategies.
Using DHIF to appraise digital health investment options means they are prospective appraisals that show a probable outcome within a range. It will change over time as estimated events, costs, benefits, and timings convert into actual activities. DHIF can be used for retrospective evaluations also, as part of monitoring and evaluation. Retrospective models include more actual data than prospective models, but both rely extensively on A&Es. For example, NMvs of intangible benefits such as better health and lives saved, and costs of redeployed resources always contain estimates.
An important role of retrospective evaluations is to identify failed and failing projects in time to avoid or minimize sunk cost fallacies (Arkes and Blumer 1985). This is where organizations keep investing in uneconomic projects as decisions are linked to inappropriate psychological attachments to their initial investments, instead of rational decision based on information. It is a form of the idiom of "throwing good money after bad" in futile hopes that significant net benefits will eventually emerge.
Financial models in DHIF aim to identify digital health projects' affordability. Where projects are not affordable, changes are needed from a range of options that include increasing the finance available, decreasing project costs and probable estimated benefits, finding another option, and abandoning projects completely. As a combined socioeconomic and financial model, DHIF enables these iterative decisions to be assessed quickly.
DHIF models for large-scale digital health programs have a complex set of relationships linking numerous variables. To keep track of them all, they need estimating and setting out in an accessible A&E schedule, not obscured in modeling calculations that decision-makers cannot easily find, assess, and change. For a proposed digital health investment, the goal is to find an option with an optimal net benefit like maximum, affordable vFM.
iii. illuSTraTivE digiTal HEalTH impaCT FramEWOrk mOdElS a. introduction Eight DHIF models are now available. Since the draft manual was produced, DHIF has been used in three settings • digital health strategic affordability and benefits for a small state; • immunization information system (IIS) options, a generic case for change and an ecosystem developed in collaboration with UNICEF East Asia & Pacific, and the digital health ecosystem in Appendix 1; and • comparison of local immunization performance with IIS.
These DHIF models and the five initial examples are available online at AeHIN's SIL-Asia website at http://www.sil-asia.org/lab-assets/. Each one is an example of the data needed to start a DHIF model.
As initial entries into DHIF, the three actual models have used some, but not all, DHIF functions. They represent a constructive way for health systems to begin an approach and develop digital health investment appraisal.
This part of the manual uses extracts to show how the models are constructed, and the information needed to support digital health business and investment cases. The five DHIF models, in order of increasing complexity, are: Before commissioning or starting a DHIF model, an appropriate process must be in place. The 10 highlevel steps described earlier are supported by 40 more detailed steps shown in Appendix 3. Awareness and understanding of these steps are important in working through the illustrative DHIF models.
It is essential for users to move on to developing their own DHIF models. The five illustrative DHIF models show how these steps fit together as a strategic initiative. They also show the DHIF's content and structure. Each one uses illustrative data, not actual. Each provides examples of how a DHIF model is constructed, how monetary values can be estimated, and how to test A&Es. The A&E data are illustrative and not intended for transfer into actual models.
All five DHIF models reveal hypothetical challenges, including affordability and risk exposure of projects with attractive but unaffordable socioeconomic returns. In real situations, projects and their DHIF models all need modifying in each subsequent stage in their decision processes. Commentaries and examples from two of the models, SMS for vulnerable pregnant women and EHRs in a hospital, provide extracts to show the types of DHIF content and findings.
The five DHIF models are regularly reviewed and refined. The tables and charts have been copied into the manual to serve as illustrations. Their content will be updated on the SIL-Asia website at http://www. sil-asia.org/lab-assets/, and added to when new examples are shared.

b. digital Health impact Framework Example from the immunization information System Two Options model
The commentary on the IIS two options model aims to guide users into constructing their own DHIF models using their own data from their digital health projects. The IIS model was constructed in collaboration with UNICEF East Asia & Pacific. It includes a summarized case for change that can be part of a business case for investment. It is in DHIF IIS model's Excel file on SIL-Asia's website and includes an IIS ecosystem shown in Appendix 1.
Several countries in East Asia and the Pacific have high immunization rates (UNICEF 2017). Over recent years, many rates have not been consistent and are well below 90% in some districts. Relatively low performers achieve rates over 70% for the first three vital vaccines, diphtheria, pertussis, and tetanus, referred to as DPT3, administered before children are 12 months old. Some district rates are below 50%.
Immunization benefits children, families, and health workers in these countries, while contributing at the same time to UHC initiatives.
There are several options for IIS. Two are set out in the DHIF model, both with an immunization registry and links to supply chain management. One option is almost full national coverage, which is evaluated and then adjusted for a phased option. While there is a strong investment case, several options usually need evaluating. Having two is a good place to start.
The challenge is to find options that offer the best and affordable value for realizing net benefits for children, parents, communities, health workers, and health systems and UHC. DHIF can help identify practical, affordable solutions that can improve relatively high, but inadequate rates and move up low rates as part of initiatives that extend the reach of immunization referred to as to the "last few miles." The start point is A&E. Appendix 3 gives a generic view of the topics that an A&E sheet should contain.
There are more examples later in this section. The IIS A&E sheet has four main sections: • general statistics about patients and communities and some costs, • estimated benefits, • estimated costs, and • core calculation estimates. Table 9 contains statistics and data from the "assumptions & estimates" sheet about children and their vaccination status before and after IIS, as well as costs shown in rows 45 and 47. Cells shaded yellow are where inputs to A&Es can be entered and changed. Supporting spreadsheets from the DHIF models show they work automatically through the DHIF model and link to the summary sheets for socioeconomic return, financial and affordability results, and the charts.
The first step is to set the time line in years and the IIS service go live year (rows 2 and 3). Then, estimate annual populations, numbers of children, vaccination rates, and numbers of children vaccinated and not (rows 3-11). These are for option 1 a comprehensive national IIS implementation.
Rows 13-27 set out the planned IIS coverage. Cell B17 shows this as 100%. The worksheet then shows the estimated impact on vaccination rates from a more efficient and effective service. These are set to a maximum of 16 percentage points, as shown in a "v&i rates tables" rows 50-56, another sheet in the DHIF model. Districts with low vaccination rates cannot improve by the maximum 16 percentage points; rather, the estimate is set to 14 percentage points, so still below an effective immunization rate. Countries with partial IIS investment can replace these with their own data derived from comparison of their locations with and without IIS.
Next is an estimate of the adjustments to the previous estimates for option 2. It is a phased national implementation starting with districts currently with >90% vaccination rates then moving in sequence down the vaccination rate league table (rows 29-38). Phasing defers benefits, but may be more realistic than a planned comprehensive project.
Estimates of the numbers of health workers as IIS users are needed to estimate efficiency gains rows (40-43). These are determined by the planned percentage IIS coverage.
Finally, estimates of changes to vaccine stock levels are needed. One is a reduction in excess stocks by better information and management offered by IIS. The other is for increased stockholding needed for extra immunization coverage, (rows 48-53). Benefits are derived from the increased immunization rates. There is a wide range of components, including better health, avoided travel time and cost, and fewer lost days of education. To ensure comparison with other digital health projects, and to include them in the strategic overview, it is helpful to assign NMvs to patient benefits using common measures. The IIS model uses data from a literature review of life values (Hirth 2000), adjusted for estimates of lower-income countries. For actual appraisals, local data are the best source, if available. The literature review identified a large range, which contributes partly to these estimates being contentious. This can be minimized somewhat by applying estimated NMvs consistently across all the digital health models.
Health care benefits (rows 103-133) include an estimated impact on resources, including fewer hospital admissions (rows 108-121), improved health worker productivity (rows 127-128), and stock management (rows 130-133). None of these are intangible, but they still rely on estimates. While stock management benefits are relatively minor and do not affect to socioeconomic return materially, they provide useful contributions to the financial and affordability estimates. Stock reductions are offset by an extra cost of increased estimated stock holdings needed to support the increased vaccination rate.
They combine resources, such as health workers and facilities, with their costs and the effect of digital health on liberating these resources for redeployment to other health care purposes. They can also include cash savings, but the largest proportion tends to be resource liberation.     In the first year of estimated benefits, there is usually a delay in realization after go live shown in column H. Benefits show a standard reduction of 50% in the go live year to adjust for a general estimated short delay in realization after implementation (row 191). It reveals the assumption more clearly than including it directly in each benefit calculation. A general adjustment has been used for this in all five illustrative DHIF models.
An alternative approach is to adjust each estimated benefit for its specific time after go live. The choice can depend on the information available as well as the degree of precision needed at each stage of DHIF's role in decision-making processes and the required business cases.
This illustrates an important A&E principle of materiality that needs to be set in place when using DHIF. Some A&Es may be small scale and spurious and may not affect the socioeconomic return or financial results materially or significantly. It may be reasonable to omit them from DHIF and treat them as included in contingencies.
Similarly, materiality can change as DHIF reports are used in the sequence of investment decisions. Using a general adjustment, such as the benefits share in the go live year, may be acceptable in the initial decision, but there might have to be a switch to specific adjustments as decisions and data become available and increased precision becomes more critical. Examples of A&Es for costs are in Table 11.
There are two main parts: parents and their children, on the one hand, and health care, on the other. Parents' costs include charges for vaccinations and travel time and costs; health care costs include capital (rows 146-155), other non-recurring costs (rows 157, 159-161), and recurring costs (rows 158, 161, and 166-188).
Capital costs are the two main information systems, IIS and stock management. Other ICT costs include extra computer and network capacity, maintenance, operating costs, and obsolescence.
Cybersecurity costs (row 158) are a combination of ICT tools, training, awareness, and risk response and management. It is an increasingly important requirement and should be consistent with the overall digital health cybersecurity strategy and initiative.   An important feature of the A&Es costs and benefits is the wide range of values. Estimates of some components are relatively minor, with negligible impact on the socioeconomic return. It is not unreasonable to omit these as entries in the DHIF model and reflect them in a contingency rate.
Another important feature is the balance of estimated costs and benefits between extra finance, redeployable resources, and intangibles. Figure 10 shows a large proportion of total IIS benefits arising from estimated NMvs for intangible benefits. Decision-makers need to use this socioeconomic return carefully and cautiously in investment appraisals and decisions. Justifying a significant investment for intangible benefits is a hard sell. Just because they occupy a large proportion does not mean they are wrong. It means they should be rigorously assembled and assessed to ensure the estimates are reasonable. DHIF offers users a helpful way to identify, review, and assess intangible A&Es used for NMvs, and to decide how they would estimate and test them for realism, reliability, and change as required.   The next stage is to make adjustments. Table 12 is an extract from the A&E sheet that includes data in rows 192-211.
Sensitivity (rows 203-205) and optimism bias (rows 207-209) adjustments are relatively fixed. Sensitivity and optimism bias rates are used to adjust the gross NPvs of estimated costs and benefits. Sensitivity is a simple adjustment that increases estimated costs and reduces estimated benefits to test the robustness of socioeconomic returns' estimating reliance. Optimism bias is a similar adjustment, but for modelers' and decision-makers' tendencies too. It uses the percentages to increases estimated costs and defers the go live year, so benefits.
Risk adjustments to costs and benefits (row 210) are not fixed, but derived from risk exposure tables included in the DHIF model. An example follows. One option is to use simple rates instead, which need not be the same for costs and benefits. As digital health projects are high risk, these rates must be high, usually well over a 50% increase on costs and more than a 50% reduction in benefits.
Risk exposure of costs is a combination of socioeconomic and financial risks. It includes parents' estimated shortfall on disposable income as well as estimated capital and revenue finance shortfall of health care organizations. It reflects the affordability risks that all digital health strategies, plans, programs, and projects face.
The discount rate (row 201) is used to adjust all costs and benefits to NPv. This reflects changes in the value of money of over time. A rate of 3% reflects the concept of a social time preference rate. It is usually lower than returns on investment expected from commercial and business investments. Some countries use different rates, and these should be used for their DHIF models.
All estimated costs and benefits are at constant prices (row 192) and exclude future, estimated general inflation. This enables real terms changes over time to be seen. Declaring the price base reinforces this. It is also important to know when DHIF models are updated later in the investment decision cycle so that the price base can be updated also.
Other items that may be technical features include financial data used in affordability estimates, such as depreciation rates and methodology, loan charges and time periods, and transfer payments such as unrecoverable vAT.
A standard adjustment to defer benefits in the go live year is referred to earlier. As more data become available, they may be replaced by specific estimates for each benefit.
Three contingencies in the cells (rows 193-199) are set based on the degree of reliance on estimates. As DHIF models progress through the investment stages and more data become available, contingencies may be reduced.
Data from these tables transfer to the costs and benefit calculations. These are for the main option, then adjusted for option 2, a phased implementation. It starts in districts with current immunization rates of over 90%, moving promptly on to other district with lower rates in sequence. Estimated costs are similar for both options, so curves overlap. Some estimated benefits are deferred for option 2. Figure 11 shows the difference.

Figure 11: Estimated Costs and benefits for Two immunization information System Options
Source: Digital Health Impact Framework model for immunization information system two options "cum net bene option 2" sheet, 2018.     Difference -IIS = immunization information system, v&I = vaccination and immunization. Source: DHIF Model IIS Two Options "risk" Sheet' 2018. Table 14 shows risk-adjusted benefits have a more modest risk exposure estimate than costs. IIS have established benefits. The precise value depends on local circumstances and the starting immunization rate. If the rate before IIS is high, such as over 90%, benefits are not as great as a starting rate between 70% and 85%; if they are lower than 70%, and especially below 50%, IIS benefits are constrained by other inhibitors, such as insufficient health workers. Figure 12 shows the socioeconomic return and adjusted curve for sensitivity, optimism bias, and risk. The IIS two options pass sensitivity and optimism bias tests. This is not unusual for DHIF models for proven digital health systems such as IIS. However, they fail the risk adjustment test. This shows that realizing the high proportions of intangible benefits carries a high risk exposure, but the project may still be viable. It confirms the need for a rigorous risk mitigation strategy. Decision-makers can use this DHIF information to review the project's risk exposure as well as risk mitigation strategy and plan, together with the levels of intangible benefits, to find an optimal relationship between them.
It also confirms the need to identify accurately and completely all the beneficiary types, their benefits, and their NMvs. These perspectives need careful assessment in DHIF models used for investment decisions. Adjusting the IIS model by phasing investment, starting with districts with rates over 90%, creates option 2. The costs are similar, but some benefits are deferred. Figure 13 shows the lower socioeconomic return and its three adjustments.

Figure 13: Comparison of Option 2 from immunization information System Two Options Cost and benefits adjusted for Sensitivity, Optimism bias, and risk
NPv = net present value. Source: Digital Health Impact Framework model for immunization information system two options "ser sens opt risk option 2" sheet, 2018. Option 2's performance curves are slightly flatter. Sensitivity and optimism bias curves are closer to the X axis, and the risk-adjusted curve is similar. The option looks less attractive than the main option. Comparing the two options provides valuable information for decision-makers. The first step should be to refine the A&Es for each option to establish a firmer estimate of the differences. In practice, there should be more than two options.
Decision-makers can then see two investment perspectives: absolute estimated socioeconomic returns, affordability, and risk; and relative estimated socioeconomic returns, affordability, and risk. Taken together, they support effective digital health decisions. Crucially, decision-makers can see behind the estimates and understand their underlying causes for each option.
Sensitivity, optimism bias, and risk adjustments all test the rigor of estimated costs and benefits, their socioeconomic returns, levels of annual finance needed, and affordability challenges. It is feasible for socioeconomic returns to pass some tests and not others, indicating the need to refine and improve estimates or sometimes whole projects.

C. digital Health impact Framework and Electronic Health records: a more Complex Example
DHIF is a generic methodology for producing bespoke socioeconomic and financial models for all types of digital health projects. The components are in both the SMS and EHR models, but the differences in complexity, scope, and scale means that the DHIF models are very different. The bespoke model for EHRs has hundreds of variables stretching across a time line of 10 years or more.
Options for EHRs can often include both capital expenditure and PPP. They have different risk profiles. The illustrative DHIF model applies only one profile to both options, but for actual models each option needs its own specific risk exposure adjustments. PPPs can often include risk sharing between partners, but practical opportunities for health care organizations to share risks with PPP partners are often limited. This may reduce the health sector's risk exposure somewhat, but the relationship between risk and reward may require an increased recurring financial commitment for health care organizations that needs including in both DHIF's socioeconomic returns and affordability components. Transferring risk to PPP partners is not usually a practical proposition.
EHRs reach more patients and carers than most digital health projects. Improvements in health care quality and efficiency benefit them, but the cause and effect are both complex and seldom direct. The CAS for EHRs has several users across numerous clinical conditions and specialties, resources, and variables between EHRs, health workers, and patients that determine the scale, timing, and types of benefits. DHIF enables these to be identified and tested.
EHRs also have more users than most digital health services. Each has specific requirements for user specifications, usability, utilization, information sharing across multidisciplinary teams, and benefits realization. Several users can be leaders and champions. If these are in place, especially leaders in specialties and clinical teams, the chances of success can increase. An example is Benh Bien Quan Thu Hospital in viet Nam (http://www.fvhospital.com). It has an extensive, successful digital health profile.
Costs of EHRs are on the more complex and large-scale end of the DHIF continuum. Resource requirements are large, and their time scales elongated. DHIF models often have to accommodate EHR costs of foundation and enabling investment, such as health information exchange (HIE), unique patient identifier (UPI), and network capacity. The illustrative EHR DHIF model includes some of these. Data from the AeHIN Costing Tool from SIL-Asia used for OpenHIE Implementation are sourced for some of the cost data. When using data from other sources, it is important that the use complies with the DHIF methodology, such as establishing a consistent price base and definitions of economic and financial costs.
For complex investment like EHRs, reliance on redeployed resources is both significant and essential for success. DHIF includes estimated costs for engagement; continuous training; and redeployable health care assets such as inpatient days, outpatient appointments, and emergency department capacity. Estimated costs of bringing these resources into alternative uses are included in the illustrative DHIF model.
The complexity of EHRs often results in achieving annual net benefits a year or two before cumulative net benefits are realized. Figure 14 shows a crux and a large, prompt benefit take-off. The gap can widen as projects drift or encounter unmitigated risks, which is common as DHIF models move along the time scale. Decision criteria can include both the years to reach annual net benefit and cumulative net benefit. Related criteria are the steepness of the benefits and net benefit curves soon after implementation. For a successful investment, these should be steep. If they are not, reaching the crux, shown at year five in Figure 14 may be delayed considerably, or not at all. DHIF models help decision-makers to identify and address these issues.
The curves look appropriate, but the gap between the benefits and the annual net benefit curves dips sharply within a few years of the crux. This is an example of issues revealed by DHIF models that need reviewing. Figure 15 of the annual estimated costs, benefits, and net benefits shows how significant this gap could be after the crux in year four.  Decision-makers will need to know why the curves are the shape they are. There are several possible reasons: • an increased investment in using redeployed resources; • an artefact of the DHIF modeling A&Es; • estimated costs taking a steep rise that needs addressing and smoothing or reducing; • benefit realization easing up after its steep climb; • a natural phenomenon of diminishing returns; or • a combination of some, or all, of these.
These reveal an important role for DHIF users, so it is important that DHIF modelers have these explanations and can convey them to decision-makers. They have to construct DHIF models that can be used to show and explain profiles of socioeconomic returns and financial affordability. Perhaps more importantly, they have to be able to explain the reasons for the results. Why and how estimated socioeconomic return or affordability positions have been reached and how they can be changed are vital commentaries that decision-makers need.
Financing EHRs can be complex too. Capital expenditure can stretch across several years, leading to extended capital finance, loan charges, depreciation, and obsolescence. The illustrative DHIF model for EHRs is a simplified version with capital resources allocated only to 2 years to show the calculation principles. The reality is that capital finance will often be needed for EHRs across a stretch of several years.
Affordability can be demanding to finalize for large-scale digital health projects. Health care organizations' financial perspectives often stretch from the current year's budget to perhaps a 3-year financial plan. EHR life cycles reach well beyond this time limit, sometimes to 15 years or more. The illustrative DHIF includes finance across a 10-year period, beyond financial planning horizons. Affordability assessments beyond the medium term for real DHIF models for EHRs may have to rely on identifying the financial commitments that decision-makers will have to assess as viable or not.  Most digital health projects rely on ICT foundation infrastructure, such as existing government networks and UPIs. Where these resources are already in place and costs were incurred before the first year of a DHIF appraisal, they are classified as sunk costs and excluded from DHIF appraisals. Where capacity of this type of infrastructure needs expanding as part of digital health projects to support their performance, such as extra network and computing capacity, the extra cost incurred should be included in the year that it occurs and apportioned to each project in line with their estimated utilization over the time frame. It can be a complex costing exercise.
When DHIF is used solely for foundation infrastructure appraisals, they are not likely to produce net benefits without their enabled applications. Large, complex DHIF models with long time scales can show these links and help deal with this requirement. They also help refresh ideas and thinking in support of digital health strategies.
Similarly, using sophisticated digital health applications beyond their immediate goals can offer additional strategic benefits. High-level semantic interoperable EHRs can provide research bodies opportunities to reduce their research costs and times. These benefits beyond health care boundaries can have important additional benefits that DHIF's strategic functions can explore.

d. digital Health impact Framework models, intangible benefits, and Notional monetary value
DHIF's consistent methodology enables projects to be assessed as a whole and to find an optimal, sustainable balance of investment across projects identified in digital health strategies by testing strategies' probable vFM and affordability. The five illustrative DHIF models are deliberately designed to demonstrate weaknesses in their digital health plans. They include high levels of risk exposures, affordability limitations, and some imbalances across benefit types. Starting from this perspective conveys a clearer view of DHIF's role in decision-making and health care strengthening.
intangible benefits can comprise a significant proportion of digital health benefits. identifying and quantifying these is important, and preparing an assumptions and Estimates sheet is a starting point. Table 15 shows how statistical data are compiled in the SMS model. More complex digital health projects will have more content, and bigger and more complex tables.   In the illustrative SMS DHIF model, NMvs for a life saved are derived from adjusted findings in the Hirth, Chernew, and Miller (2000) report, as shown in Table 17.
This shows one version of NMvs. It is important that the estimates are shown in A&E sheets so that decision-makers and stakeholders can easily challenge and change both the methodology and the results. Estimates from local research are usually best. If they are not available, local adjustments to international or local research can act as proxies, provided they are used reasonably. Table 18 shows the health care benefits performance.
The estimated demand drop benefits as a direct result of the SMS initiative are derived by estimating the number of patients who will no longer need these services and assigning monetary values from the stakeholder data section of the sheet. There can be numerous components of estimated costs.  Estimates for ICT costs are usually from suppliers' information. Organizational costs are usually derived from a combination of staffing levels; salaries plus employers' costs, such as pension and social security contributions; and time and changes to health care resources, such as inpatient beds and outpatient clinic appointments. They also include project management, change management, and training. Table  19 shows an example of estimated socioeconomic costs from the malaria model. Software 500,000 --500,000 500,000 500,000 500,000 2,000,000 Middleware 500,000 --500,000 500,000 500,000 500,000 2,000,000 Analytical tools 500,000 --500,000 500,000 500,000 500,000 2,000,000  Both costs and benefits are discounted to NPvs and adjusted for sensitivity, optimism bias, and risk exposure. Table 20 shows an extract from the SMS model.
To use the formula for the costs and benefits for each year, the years should be assigned a sequence of numerical values beginning with the first year as "0." This means discounting begins in the first year. The effect of assigning "0" means that the first year's costs and benefits are not discounted.
The formula for NPv is: where n is the number of years and i is the discount rate. An example of converting it into a calculation in a DHIF model is: where cell $C$1 is the discount rate and D1 is the year in the time scale. Copying the formula to each column of the time scale updates the year number. When a set of annual costs and benefits are multiplied by their NPv formula in their column, it produces their NPvs for that year.
The difference between the total, cumulative NPv of costs and benefits for the whole time scale is the socioeconomic return expressed in currency and a percentage. The year when the benefits' NPv exceeds the costs' NPv is the crux, both for annual and cumulative costs and benefits. Figure 16 shows the effect of discounting. Over a 10-year time scale, with a 3% discount rate, the difference increases with each year, but is not large.
Digital health is a risky undertaking and risk is a cost. It manifests itself as cost and time overruns, but there are underlying causes. A simple risk calculation is: Estimated cost of risk × its probabilities of occurring = risk exposure.    • software is clumsy and has limited usability; • functionality is not up to requirements; • interoperability is less than required; • HIE not working as planned; • networks are too slow, limiting usability; • fixing bugs takes too long; • fixing bugs is more costly than expected; • recruiting specialist staff is harder than planned; • retaining specialist staff is harder than expected; • stakeholder engagement takes longer than planned; • implementation takes longer than planned; • procurement takes longer than planned; • supplier services are more expensive than planned; • cybersecurity is not good enough; • utilization is not as large as expected; • inadequate investment in human capacity building and change management; • change management becomes random; • benefits are not realized in full; and • benefits take longer to realize than expected.
All digital health projects are exposed to 100% probability of risk at their initial stages, meaning they risk collapse entirely. In the early days of the first DHIF models, the risk can diminish slightly from 100%, but remains high. A challenge for estimating risk exposure is the lack of data and knowledge about probabilities. When there are no local data about probabilities, risk adjustments should use large estimated values, usually in excess of 50% for risk probabilities. As projects proceed, more information is revealed about risks. It may show that the risk adjustment can be reduced, or that it should be increased.
Large-scale digital health projects can have significant PPP options. These have different socioeconomic and financial profiles and curves. An example from the EHR model shows a comparison of two illustrative options. It reveals the differences between them when swapping capital expenditure for leasing and operational expenditure. Leasing increases operational expenditure and reduces the demand for capital finance. Table 21 shows how financing options can be developed for equivalent socioeconomic benefits.
Typical digital health costs for a set of EHRs and ePrescribing in several European countries are allocated as 42% to ICT, and 58% to organizations' costs, including change and transformation efforts. The range is wide: 14% to 68% for ICT and 32% to 86% for organizational costs (Dobrev et al. 2010). There are no comparable data for ADB member countries.
Average cost profiles show that about 49% of digital health costs need extra finance. About 42% are resources redeployed from existing budgets, which create a significant leadership and management challenge. The other 9% are intangible costs (Dobrev et al. 2010). Decision-makers need to know these components and their proportions. They determine digital health investment financial requirements and resource redeployment arrangements.
Digital health leaders face another considerable challenge with efficiency gains. For successful EHRs, they can be about 79% of benefits. With ePrescribing, they are about 75% (Dobrev et al. 2010). The resources are often distributed in small amounts across a wide range of health care activities. Redeploying them is demanding. An example is where digital health benefits save 2% of nurses' time. How can this be redeployed in small-scale nursing teams? Is it easier when the nursing workforce comprises very large working teams or groups? It may be that it creates spare capacity that enables the nurses to see more patients in response to the continuous growing demand. It may be that the time can be redeployed to provide better health care quality. These are decisions and actions needed from digital health leaders and decision-makers.
Average quality benefits can reach 19% for successful EHRs and 24% for ePrescribing. Access benefits for people who previously did not have access to health care may be as low as about 1% to 2% of total benefits. They can be higher for some types of digital health investment, and are in parallel with investment in extra capacity provided by more health workers, extra drugs and medicines, more medical and surgical supplies, and more health care facilities (Dobrev et al. 2010). This means that ICT investment alone, without investments in human resources and capabilities, medical supplies, and training, is not sufficient. For some digital health projects, investment in liberated and expanded health care capacity may also be needed to maximize benefits.
Better patient safety is an important component of digital health quality benefits. A report by the United States National Quality Forum (2016)   The challenges of not using good basic practices in data usability and presentation, as well as the need to review and change some health care regulations, were emphasized later (Electronic Health Record Association 2017). An example is a requirement to keep medical records and discharge letters as paper documents. This inhibits some digital health benefits and leads to electronic and paper systems operating in parallel systems, increasing costs and diminishing net benefits.
Measure needed to succeed include • a strategy for remaining compliant and giving up paper; • minimizing, or avoiding if possible, paper medical records; • avoiding paper discharge letters; • identify legal documents that require paper; • establish how penned signatures can be used; and, • identify health care regulations that need changing.
These features need consideration as components of, and use by, DHIF models. They require effective change management.

E. digital Health impact Framework models and digital Health Strategy
Before setting up a DHIF model, it is important to keep in mind the outputs needed. Numerous perspectives can be drawn from DHIF models, especially choices and decisions for digital health strategies. Tables 22 and 23 compiled from six DHIF models accompanying the manual show some of these. They have been combined into an illustrative assessment of a digital health strategy. It shows how an overview can be used to test the strategy's vFM and affordability. Two projects, IIS and malaria surveillance, extend across large populations. The other four do not. The small-scale EHR project benefits a local community of hospitals. This accounts for the difference in the monetary scale of the illustrative projects.
The six examples of DHIF models provide decision-makers with some of the information they need for their digital health strategies and investment decisions. Each model uses illustrative data to show a positive socioeconomic return, which then becomes negative when they are stretched across 10-year time scales and adjusted for increased risk exposure due to the extended time scale. Whichever mix of digital health projects are seen as the best strategic fit, the final selection requires significant risk mitigation strategies and plans. Without them, emerging risks can paralyze leaders, project teams, and users' actions.
Digital health has many risk mitigation components. Examples of risk mitigation are • setting realistic goals, not aspirations, for short project time scales; • ensuring effective and comprehensive stakeholder engagement to listen to their needs and concerns; • adequate time allocated to determine stakeholders' information and usability requirements; • widespread general digital health skills and awareness across workforces; • effective and rigorous procurement, including arrangements for seamless changes of suppliers and penalties for poor performance that exceed suppliers' costs to fix poor performance; • investment in appropriate change management extending beyond project management; • adequate parallel ICT investment, such as network and connectivity capacity to sustain prompt transactions; • rigorous business cases and realistic investment decisions; • comprehensive, medium-term health care financing, including take-up when grants terminate; • where appropriate, ensuring other stakeholders, including patients and residents and visitors, can afford digital health; • project plans that recognize that digital health projects do not proceed in a linear manner; • ready, steady, go decisions instead of not ready, steady, go, go again, go again decisions; • digital health leaders who rely on personal power, not position power, and have clear strategic goals, can maximize flexible tactics in a constrained digital health context, and can inspire users; • ensuring that victory is not declared too soon; and • appropriate digital health leadership from political and executive levels through frontline managers and health workers, and an underpinning requirement of risk mitigation.
Failing to have risk mitigation strategies and plans in place contributes to cost and time overruns, as well as diminished benefits. While often seen as risks, overruns are manifestations of a range of many individual digital health risks, such as delays of decisions needed to deal with technical issues. It is important to address each of these components rather than the headline overruns and diminished benefits.
Furthermore, none of the six illustrative DHIF models are shown as affordable. It is typical of the types of issues that DHIF models can reveal in their earlier formats. They provide decision-makers with information they can use to refine investment plans in dialogues with stakeholders. Tables 22 and 23 use data from the six illustrative DHIF models to show how these issues can be identified, with comparisons in percentage terms shown in Table 22 and the estimated numbers in Table 23. The two tables show how combining the six digital health solutions can increase benefits and total estimated socioeconomic returns.
At the ADB workshop on 31 January 2018 in Bangkok, participants were asked to decide on the decision criteria they would use to select which of the six illustrative DHIF models they would retain in an affordable digital health strategy and why. Groups showed a wide range of different approaches that DHIF can support. The following is a summary of their responses: • Challenge and refine the assumptions and estimates in each model before finalizing a decision, an important principle in that every time decision-makers review findings from DHIF models, the assumptions and estimates should be challenged to ensure that the models comply with corporate perspectives. • Consistent with the challenges, return the EHR project to the modelers for explanations that the project fails to have benefits that save lives. • Select digital health projects that save lives. • Assign letters to each project to replace the names and avoid decision bias. • Select projects that benefit that largest number of people. • EHR projects invariably slip in time, so redeploy finance from the project to other projects and negotiate with the ministry of health to reset its financial and affordability base. • know and understand the benefits and beneficiaries of each proposed digital health investment before finalizing investment priorities. • Ensure that all the organizational resources will be in place as needed before agreeing to projects to proceed.
Starting with affordability, each planned project has an affordability gap of varying degrees. Dealing with this involves a mix of the following strategic options: • find additional finance; • find additional solutions, such as PPP for EHRs, shown as leasing option in the EHR model; • cut projects from the strategic plan, such as the two mHealth telemedicine projects that have minimal, relative benefits, but also minimal relative costs, and do not provide a substantial solution for the affordability challenge; • cut the SMS and mHealth telemedicine with extra access projects because they carry large risks; • cut the telemedicine projects because their scales are not big enough; and • expand the scales of the telemedicine projects and reduce other projects.    Digital health investment requires considerable organizational costs. Decision-makers need to ensure that realistic plans are in place to provide these resources, many of which are redeployed from existing budgets. Where this is not practical, projects should be reconsidered or scheduled for a possible cut to release finance for other projects. The malaria surveillance project has the lowest percentage of organizational costs, reinforcing decisions to assign it as the top priority.
Strategic investment in foundation projects, such as UPIs and semantic interoperability, is not included in Table 23. In reality, they would be, increasing costs but providing a platform for additional future benefits. Table 23 could show how each one will contribute to the six projects. The timings and costs of foundation investment can be factored into each of the six projects' estimated time scale, costs, benefits, and socioeconomic return. Apportioning foundation costs to each application is challenging, but should be attempted.
These strategic perspectives then become part of the iteration process identified as step 10 of the DHIF process, illustrated in section II-C. It helps to locate decisions about changing individual investment plans in the strategic context and requirements of digital health. It is an essential comprehensive perspective for digital health leaders.
Some proportions are very small. For some of the six illustrative DHIF models, some cost and benefit values are not appropriate, so they have zero values. In addition to the relative socioeconomic and financial balance of digital health strategies, their scales and scopes are important as well.
The low proportions for access illustrate an important digital health investment theme that has to be integrated with developments in health care models. The strategic overview reveals that these are not in place for telemedicine and EHRs, though they are for SMS, telemedicine with extra access, IIS, and malaria surveillance. The proportion is small, however, because the estimated monetary values of lives saved and other quality benefits are larger. These show the importance of a strategic overview. Decisionmakers can elicit these themes and profiles to ensure that the balances of their digital health investment plans are structured appropriately across benefit categories.
Tables 22 and 23 show the estimated beneficiary numbers and financial estimates for up to 6 years to highlight the medium-term financial planning limitation. It emphasizes the affordability requirements of strategic, large-scale digital health projects' longer time scale.

F. presenting your digital Health impact Framework model and Findings for decisions
Presenting findings from DHIF models is a vital stage in reaching investment decisions. Whether DHIF is used either in full or partially, or for digital health strategies, programs, or projects, presenting findings to inform decision-makers is a vital part of DHIF modeling. It is challenging for DHIF modelers too. A generic business case checklist can help preparations (Sheen and Gallo 2015). It has been expanded to address digital health specifics as follows: • Set out a clear health, health care, and business need and case for change as a compelling, attractive story that appeals to decision-makers. • Ensure that detailed explanations and descriptions are available about why the socioeconomic and financial characteristics combine into the results for question and answer sessions. • Show how the benefits in the DHIF align with strategic goals. • Identify the objectives and requirements of each stakeholder group.
• Confirm which representatives of each stakeholder group are engaged in the DHIF modeling. • Confirm that significant stakeholders involved in the project support the option. • Confirm which subject matter experts in the organization have been engaged in, and support, cost and benefit estimates, time scales, and sensitivity, optimism bias, and risk adjustments. • Ensure that estimated costs are included for all change management, benefits realization, and health and health care transformation. • Ensure all the A&Es are scheduled and accessible to decision-makers. • Compare the DHIF results with the equivalent for realistic strategic scenarios or project options. • Have a concise and clear set of slides and diagrams.
• Ensure that a decision to proceed can move on promptly to the next stage.
When this last step is achieved, the DHIF models can move on to their next stage. The first DHIF model never survives the first grilling by decision-makers because it reveals information and insights they did not have beforehand.
DHIF models enable decision-makers to engage with their digital health projects' components to test their assumptions and ideas well before they are implemented or abandoned, and see the results in simple spreadsheet graphics. Sharing learning from findings and model structures will improve techniques, digital health knowledge and investment decisions. It can be within and between health systems.

glOSSary benefit enablers
They include engagement throughout projects, resources for change, functionality, interoperability, architecture, computer and network capacity, meeting users' requirements, usability, utilization, digital health, and analytic skills and knowledge. benefits The three main types are quality, access, and efficiency.
Quality includes better informed, so healthier patients, safer health care, shorter waiting times, more integrated health care, and more effective health care, including lives saved.
Access is people's opportunities to enter health systems when they were previously denied, so part of equity envisaged by universal health coverage (World Health Organization 2014).
Efficiency includes improved productivity, redeployed resources, neither of which may reduce cash outlays, but can reduce some variable costs, and cash savings by avoiding waste.
The three other classifications of economic benefits are financial, nonfinancial, and intangible, with no market price, so relying on monetary values estimates.
Financial benefits are improvements in cash flow and income and expenditure performance, such as reduced spending and increased income.

Cash flow
Receipts and payments of funds, either as cash, check, or bank transfers.

Cause and effect
Most digital health, especially large-scale services such as electronic health records and ePrescribing, has an indirect effect on health and health care across a cause and effect chain, because most of its information is provided to users who have a choice about how and when to use it, if at all.
Benefits depend on behavior and performance of individuals in the health system and if, when, or how they use digital health information, so how the complex adaptive health care system changes to realize the benefits of digital health.
Dealing with this lack of direct cause and effect between two variables, digital health socioeconomics require reliance on estimates of probabilities of how the cause and effect chain might work, or does not work.
For narrower digital health, such as short messaging service reminders and telemedicine, cause and effect are easier to establish, but can still rely extensively on users' choices, assumptions, and estimates.

Costs
There are several classifications.
Two are information and communication technology and organizational costs, such as stakeholder engagement, setting requirements, training, project management, change management, and benefits realization.
There are three types of socioeconomic costs: extra resources, redeployed resources from existing budgets, and intangible resources where there are no market prices, such as patients' extra travel time.
Financial costs are derived from transactions, accounting, and costing, budgeting, and financial planning.
Socioeconomic costs exclude transfer payments where no resources are provided, such as value-added tax, grants, financial donations, loan charges, and depreciation.
Socioeconomic costs include the purchasing costs of donated equipment.
Financial costs include transfer payments, grants, and financial donations, and may include depreciation on donated equipment but exclude their purchasing cost.
Two costing techniques are total absorption cost to produce average costs, such as costs per inpatient bed day and diagnosis-related group, and variable, semifixed, and fixed costs.
Total absorption cost is used mainly for pricing decisions; variable costs can be used to price additional outputs, so is relevant for digital health investment decisions, because relying extensively on average costs could significantly distort the analysis by adding bias, but when to use them appropriately needs careful consideration.

Crux
Points in time when benefit and cost curves cross, and when benefits exceed costs to create net benefits.
There are two, one for annual costs and benefits, the other for cumulative costs and benefits, which is the measure for socioeconomic return.

digital health finance
Affordability over time is usually a crunch component of digital health investment decisions, and measured in two ways: comparing the net cash flow of digital health and its income and expenditure and balance sheets with their budgets and financial plans.

digital health socioeconomics
value for money is measured as net benefit over time, the difference between estimated cumulative costs and benefits for each stakeholder type using cost benefit analysis methodologies (Sartori et al. 2015).
DHIF is a version of the modeling methodology used by eHealth impact studies (Dobrev et al. 2008) and was derived from the United kingdom's Green Book (HM Treasury 2003) cost benefit analysis methodology, and includes a financial component and a digital health strategy module.
DHIF calls value for money the socioeconomic return to reflect the inclusion of residents and visitors, patients and carers, health workers, and managers as digital health stakeholders.
Return on investment has more to do with projects aiming for financial gain within an entity, and is linked more to DHIF's financial component.
It is also part of monitoring and evaluation (World Health Organization 2016).
discount rate A social time preference rate, usually about 3%, to reflect the value of society's preferences to receive goods and services sooner rather than later, and to defer costs to future generations.

discounted cash flow
Monetary values of costs and benefits that are adjusted to reflect the changing value of money over time due to other investment opportunities, especially earning or saving interest.

Economics and finance
Seeking optimal relationship between socio-economic returns and affordability means essentially to define how many socioeconomic returns an organization or agency can afford.
Where affordability is the core factor in digital health decisions, reducing costs to achieve affordability may have an adverse effect on socioeconomic return, hence the need to find an optimal relationship, an alternative option, or abandon a proposed project.

Finance
To ensure that digital health projects are financially viable and sustainable over their life cycles, and within the resources available.
To identify the need to modify digital health projects so that they are financially viable and sustainable.
To find a sustainable mix of projects that implement digital health strategies.

income and expenditure
An accruals concept and accounting standard to measure profit and loss recognizing economic events regardless of when cash receipts or payments occur, a bit like using credit cards. It also includes costs with no cash transactions, such as depreciation and contingent liabilities.

minimize project costs
The cost of fixing errors and bugs in projects increases as projects move along their life cycles (Callum 2014), so it is important to find opportunities to identify and take corrective action as early as possible in the life cycles of digital health projects.
mixed appraisal A combination of prospective and retrospective appraisals usually used to check digital health projects' socioeconomic and financial profiles to achieve performance and trajectories before, on, or after the go live date, or to provide data for DHIF knowledge bases.
Net present value Each year's estimated costs and benefits reduced by the discount rate.
Discounted cash flow formula:

Option
Part of a prospective socioeconomic appraisal that sets out one way to achieve a strategic goal, including a specification of the project, its components, and its technicalities.

Options
A range of possible digital health solutions for a project and with different costs and benefits, often including options for direct capital expenditure and a publicprivate partnership option.

prospective appraisal
Socioeconomic and financial appraisals of digital health in the future, and investment decisions or estimated trajectories of existing digital health services.
Often used as part of business cases to identify the most optimal option and compare socioeconomic returns with other projects, such as other digital health projects, new drugs, more health workers, and facilities competing for resources.
Prospective appraisals are sometimes called formative or ex ante because they are based on estimates.
They may include counterfactual options that have not yet occurred, or may not.

retrospective appraisal
Socioeconomic and financial digital health evaluations extend into the past.
Usually used for monitoring and evaluation to identify digital health projects' performances and match them to their planned goals.
Retrospective evaluations are sometimes called summative or ex post facto because they are based on actual results, but they can still rely extensively on estimates.

Socioeconomics
To find the crux of digital health costs and benefits for all stakeholder types in a set of project options requires knowing • how long it takes to reach a net benefit, so a socioeconomic return; • whether the socioeconomic return is sustainable over digital health projects' life cycles; • if net benefit is not achievable, whether the digital health project needs modifying or abandoning; and • how and why digital health projects behave and perform, or how they do not.
To assess the risk exposure of digital health to design effective risk mitigation plans.
To identify an appropriate, optimal mix of viable, sustainable digital health projects for digital health strategies' implementation stages.

Stakeholder data
Includes the estimated numbers, timings, and types of patients and residents and visitors, health worker numbers and types, unit costs, marginal and variable costs, and sometimes willingness to pay, which is often small amounts per person , or can be large when it is an estimated value of lives saved or illnesses avoided (Hirth 2000).
Each of these is assigned to years in projects' time lines.

Sunk costs
Resources related to projects before the appraisal period, but not included in the appraisal, such as communication networks already in place as part of a government initiative that a digital health project will rely on.
If extra resources are needed as a result of the project, such as extra network capacity or running costs, they should be included in the appraisal.

Sunk cost fallacy
Organizations continue to invest in uneconomic projects as decisions link to inappropriate psychological attachments to their initial investments.

Time line
The number of years for a digital health investment appraisal, usually similar to the project's estimated life cycle. This diagram illustrates the complexities of a health care ecosystem. The Digital Health Impact Framework can apply this to clarify a project's variables, their relationships, and the investment implications of improving the performance each of the components shown.

appENdix 2: COmmON SOCiOECONOmiC aNd FiNaNCial appraiSal ErrOrS
As digital health appraisal models are developed, the number of variables and calculations increases, adding to the complexity. It is easy to drift away from the main principles. A checklist of common errors (Government of Scotland 2009, pp. 108-109) can help to retain the required rigor and compliance: • vague, qualitative statements of strategies, goals, and objectives that inhibit their quantifi cation and measurement; • not including the resources needed for the project to succeed, especially resources already in place which are redeployed to projects, as these have an opportunity cost and should be costed at their current market value, such as doctors' time for engagement that reduces their time for clinical activities; • not including all existing resources liberated when projects go live, including those arising from productivity gains; • including estimated costs in socioeconomic estimates that should be in fi nancial estimates, such as º capital fi nancing charges, such as loan charges; º double counting capital costs with interest and depreciation charges; º not including the cost of estimated capital cash outlay in the years in which it occurs; and º transfer payments such as social security or redundancy payments, in the socioeconomic estimates instead of the fi nancial; • not including estimated costs and benefi ts to other bodies or budget holders, and not extending the appraisal beyond the horizons of a health care organization; • ignoring displacement, the impact on activities of other health care providers or market competitors; • no clear explanation of the basis of all weights and scores; and • using assumptions and estimates in cost and benefi t calculations that are not on the assumptions and estimates sheet, and not easily accessible by decision-makers and stakeholders for scrutiny. The adkar® model developed by Jeff Hiatt (2006) focuses on and supports every person involved in the changes. Its goal is to ensure that everyone succeeds. It has five steps: • A -awareness of the need to change, • D -desire to support and participate in the changes, • k -knowledge of how to change, • A -ability to implement the changes, and • R -reinforcement to sustain the changes.
ADkAR® emphasizes outcomes and benefits, and directs change management activities. Awareness, desire, knowledge, ability, and reinforcement are part of the outcomes also. It can be used for changes needed across all three DHIF's types of changes: process, organic, and strategic.
Its link to everyone involved provides them with information for their personal development plans for the specific changes needed to realize digital health benefits. ADkAR® is also a source of detailed information needed to monitor progress. It is also the foundation of effective communication strategies and plans.
kotter's 8-Step model of Change is set out in his book Steps to Accelerate Change in Your Organization and is regularly updated. It updates his original 8-step process for change to support the needs of modern organizations. Some of the eight mistakes that organizations make that result in unsuccessful change are part of the inherent risks of digital health • not establishing a great enough sense of urgency, • not creating a powerful enough guiding coalition, appendix 3 continued • mobilizing organizations' key players, and • silencing the most vocal naysayers.
lean Six Sigma (https://www.sixsigma.com) is a technique to improve quality and efficiency. It identifies and removes product and service deficiencies and waste. It can be used to see and solve problems from patients' perspectives so that process improvements make a difference for them.
The Captain Class (Walker 2018) identifies the traits of highly successful captains in sports. Managers and health professionals at health care organizations' peripheries can be likened to captains of sports teams. They work with relatively small teams and are under the umbrella of executives. Seven traits of outstanding sports captains have been identified and their leadership styles analyzed. It offers a different perspective to mainstream leadership models: • extreme doggedness and focus; • testing the limits of rules; • willing to complete thankless jobs in the shadows; • low key, practical, and democratic communication style; • motivates others with passionate, nonverbal displays; • strong convictions and courage to stand apart; and • ironclad emotional control.
appENdix 5: WilliNgNESS TO pay aNd NOTiONal mONETary valuE Appendix 5 sets out five approaches to assessing willingness to pay (WTP) and notional monetary value (NMv), as well as considering the issues of comparability, weighting, and scoring benefits.

Lives Saved
Two main options are using weighted or unweighted benefits for cost allocations and apportionments of planned digital health investments. The two most challenging components are estimating the number of lives saved, and infections and complications avoided.
The starting point is using effectiveness of the delivery strategy as a catalyst for improving service delivery or utilization. It may facilitate or restrict the type of outcome measures available for use in socioeconomic analyses, such as changes in coverage or process indicators. For these, opportunities should be explored to translate coverage data into modeling tools.
For example, for key reproductive, maternal, newborn, and child health interventions, modeling tools are available, such as the Lives Saved Tool (LiST), part of Spectrum, a software package maintained by Avenir Health and funded by the Bill & Melinda Gates Foundation (Avenir Health 2017).
LiST produces estimates of global health impact by modeling outcomes, including • neonatal and child mortality; • maternal mortality; • stillbirths; • preterm, small-for-gestational-age, and low birth weight as birth outcomes; and • nutrition outcomes for stunting, wasting, and anemia.
LiST's framework is based on • demography details, either directly from demographic projections produced by the united Nations population division or from national or subnational demographic estimates; • cause of death information for neonates, children under 5 years old, mothers, and stillbirths, from country-specific World Health Organization (WHO) profiles or estimated using local data sources; • coverage levels for a variety of key health interventions that affect child and maternal mortality; • health status indicators for a national or subnational setting; and • effectiveness estimates for neonatal, child, and maternal interventions from the latest scientific reviews and literature.
These generate estimates of lives saved for individual and packages of interventions. For mHealth solutions anticipated to have a differential effect across subgroups, their specific costs and 2equity, and out-of-pocket spending. They should be explored from the outset.

Quality-Adjusted Life Years
A quality-adjusted life year (QALY) concept estimates the value of a health state that is more desirable, and thus more valuable, to society as a whole (Weinstein, Torrance, and McGuire 2009). It can take the perspective of an individual, communities, or the health system making decisions on their behalf. It is not the same as a value assigned by an individual benefiting the improved health state. QALYs disregard these benefits perceived by individual patients, carers, and residents. It is a version of WTP. There are nine principles: • a decision on resource allocation is being made; • health-related consequences of options can be specified in terms of health states, changes in health states, and durations of health states over time; • resources are limited, and each option has an impact on available resources, so it is an opportunity cost; • decision-makers need to maximize the health of the population within resource constraints; • health is defined as value-weighted time and beyond the relevant time horizon; • value is measured in terms of preference or desirability; • each individual is risk-neutral about longevity and has additive utility over time, an assumption enabling health states to be valued at points in time, disregarding their duration or sequence; • value scores or preferences measured across individuals can be aggregated and used for communities; and • QALYs calculated using the aggregated preference weights can be aggregated across individuals.
Using QALYs for digital health investment decisions has some limitations, especially for projects with numerous, complex variables, assumptions, estimates, and risks. QALY information is rarely available at the early stages of digital health project appraisal. It may be available for some smaller-scale digital health projects using data from their operational stages.
Digital health is a health care resource usually supporting health workers. This creates a set of variables that combine into a complex adaptive system that may have no direct cause-and-effect relationship between digital health and benefits. Where this type of indirect effect can be established, it can be challenging to disentangle the contributions of each of the resources involved, needing very complex costing and system models that may extend over long investment time scales.
Digital health resources support a range of benefits in quality, access, and efficiency groups. These need assumptions and estimates to apportion estimated costs across benefits to find the cost of a QALY. They can be within a very wide range, limiting their reliability for decisions. It is more complex for largescale digital health initiatives, such as electronic health records and ePrescribing. Costs can stretch across a wide range of specialties, diagnosis-related groups, and specific patient types, making cost apportionments complex or spurious.
An estimated QALY value was about $24,777 at 1997 prices, using human capital as a basis. Other QALY values can exceed rules of thumb frequently used to determine whether an intervention produces an acceptable increase in health benefits in exchange for incremental expenditure . A European study (Donaldson et al. 2010) found a wide range of possible QALY values across eight countries, ranging from €17,557 to €76,623, at estimated 2010 prices, using three approaches. Discounted values of a life ranged from €26,016 to €318,983. This wide range reveals a challenge in using QALYs where the choice of a QALY value can be instrumental in determining the socioeconomic value of health-related benefits and net benefits.
A study by Thavorncharoensap et al. (2013) estimated the WTP of QALYs in Thailand as a range of about B59,000 to B285,000, about $3,578 to $17,283 at purchasing power parity. The range is about 0.4-2.1 times Thailand's GDP per capita of about B138,000 in 2008. The study says that the estimates are broadly in line with the Commission on Macroeconomics and Health's range of 1-3 times GDP, though the lower end of the study's range is below the commission's lower end. It also found that QALY values varied for specific health conditions. This indicates the need for careful consideration. One way may be to substitute values from each end of a range and compare the effect on the socioeconomic returns and their adjustments for sensitivity, optimism bias, and risk exposure.
QALYs can only be used for health-related benefits, and require estimated costs to be apportioned to these. Large percentages of digital health benefits are often derived from efficiency gains, and QALYs are not designed to deal with these. As a measure of society's benefit, QALYs may not reflect the NMvs of individuals or groups of patients, carers, and residents, and cannot be used to estimate NMvs for patients' time-saving benefits, such as online booking and fewer journeys. Neither do they provide NMvs for carers, residents, and visitors using information for better health or avoiding illnesses and infections, or for health workers' digital health benefits. Other methods are needed to assess their NMvs.

Disability-Adjusted Life Years
Expressed as the number of years lost due to ill health, disability, or early death, disability-adjusted life years (DALYs) measure overall disease burden. It was conceptualized (Murray and Lopez 1997) in the Global Burden of Disease studies for WHO and the World bank.
This metric combines mortality and morbidity into a single, common measure. It extends the concept of potential years of life lost due to premature death, by including mortality and its equivalent years of healthy life lost, as well as being in states of poor health or disability. One DALY is 1 lost year of healthy life. The sum of DALYs across populations measures the gap between current health status and an ideal health situation where the entire population lives to an advanced age, free of disease and disability.
Before DALYs, the number of years of life lost was available as a single measure of health. When health is included to reflect disability expressed as healthy life lost, years lived with disability produce a different result by using the formula:

Years of life lost + years lived with disability = DALYs
DALYs' measure of time lost due to premature death and time disabled by disease is especially relevant for the effects of chronic illness. It can be weighted to assign additional values to years lived as a young adult. Its concept is that society places more value in education of children's years to improve productivity and enhance a return on investment.
Social weighting is another adjustment. The value of each year of life depends on age. A weighting of 0.1658, discounted at 3%, has been used in Global Burden of Disease studies, meaning that a weighted year of life saved next year is worth about 97% of a year of life saved this year, and so on in subsequent years. The formula is: is the age at which the year is lived. is the value assigned to it relative to an average value of 1. However, relationships between life expectancy, years lost, discounting, and social weighting are complex. They depend on the severity and duration of illness (Disabled World 2017). For early stages of digital health investment, and later stages too, DALYs are not often available. Its use in digital health can be more challenging because digital health does not always have a direct cause and effect with the health of patients, residents, and visitors. Its users intervene and can affect the benefits.
For digital health investment decisions, DALYs have similar limitations as QALYs described above. Limitations of cost apportionments and data across a wide range of specialties, diagnosis-related groups, and specific patients are not likely to be available at early digital health investment stages, if at all. DALYs can only be used for health-related benefits and require estimated costs to be apportioned to these. Large percentages of digital health benefits are often derived from efficiency gains, and DALYs are not appropriate for these. As a measure of society's benefit, DALYs may not reflect individuals' NMvs or NMvs of groups of patients. DALYs cannot be used to estimate NMvs for patients' time-saving benefits, such as online booking and fewer journeys. Neither do they provide NMvs for carers, residents, and visitors using information for better health or illnesses and infections avoided, or for health workers.
Other methods are needed to assess their NMvs.

Life and Illness Insurance
There are many different types of policies and premium levels with life insured values, so this is not a straightforward method for the Digital Health Impact Framework (DHIF). A recent estimated average annual life insurance payout by a major United kingdom insurer (Aviva 2017) was over £32,000, about $43,000. For critical illness, or some extreme types of morbidity, the average payout was about £59,000, equivalent to about $78,000.
This method offers a crude way to estimate NMvs. Using information from people with insurance policies can exclude some communities, such as people on low and no wages, who cannot afford them. These limitations can skew results.
There are two advantages of this method. One is that life insurance and illness factors are rigorously scrutinized by actuaries, increasing their financial reliability. The other is that NMvs using health insurance are derived from market prices and values, albeit from a self-selecting community that is not random and can afford it. It is still an NMv proxy.
Using health insurance is less reliable than using life insurance. Annual premiums can increase significantly over time, more than general price increases in economies, requiring real terms adjustments in DHIF. In addition, they can vary by location and the amounts are relatively small compared to life insurance payouts, so they need an actuarial adjustment for anticipated life expectancies.
While the methodology is easy to understand and has some basis in transactions, care is necessary when using it. Testing the reliability of insurance estimates can use payout outliers and extremes to replace estimates to show the sensitivity effect on estimated benefits and socioeconomic returns. This is in addition to sensitivity, optimism bias, and risk adjustments.

Earnings and Pensions
WTP can often rely on income. Earnings and pensions are the two main income sources in an economy. Main data sources include average income, average pension, gross domestic product (GDP) per head, and gross national income (GNI) per head. Much of these data are available from the World Bank. These compare to estimated salaries and wages, converted from Thai baht: • maximum $208,000; • average $35,730; • median $24,960; and • minimum $3,840 (Salary Explorer 2017).
Part of the difference is explained by the numbers having different denominators. The whole population is used for GDP per head. It overcomes the limitation that not all the population has a salary or wage, such as children in education, unemployed people, volunteers, and people receiving only state pension incomes. These differences show that considerable care is needed in selecting a methodology. Choices can be refined with WTP estimates. Adjustments include linking estimates to specific patient and community groups, using averages and testing the effect of using outlying values, such as excluding maximum wages.
Having selected an approach, examples of using the data include estimating a minimum wage that people would be prepared to pay for better information about their health. For early and subsequent stages of digital health appraisals, it is not often practical to survey patients, carers, and other residents and visitors to elicit their WTP preferences. This means reliance on judgments by teams.
It may be viable to survey a sample of health workers to identify their WTP for digital health benefits, for example asking them how much of a wage rise would be needed to compensate for giving up digital health in their clinical and working practices, or how much of a wage drop they would accept to keep digital health services. Before trying these, a clear description of WTP principles is needed so that participants understand WTP concepts, especially that it is not a step leading to direct payments by them.
There are limitations to using earnings and pensions for WTP. They do not fit with children, volunteers, pensioners, and unemployed people. Applying a WTP to their benefits should reflect their possible different values and those of their carers, but it should also aim to be consistent across all stakeholder types.
Testing the reliability of earnings estimates can include alternative pay levels to replace estimates used to show the effect on estimated benefits and socioeconomic returns. This is in addition to sensitivity, optimism bias, and risk adjustments.

Comparability
A strategic perspective needs a comparison and aggregation of all proposed digital health investments. This requires methodological consistency across all DHIF models in a health system. Whichever method is selected for estimating NMvs of intangible benefits, it must be used consistently for each group of projects. One study (Bergmo 2014) found that the use of QALYs in 17 telemedicine evaluations, a relatively straightforward exercise, had different definitions and that a harmonized approach is needed. It is a principle that applies to all the methods for estimating intangible benefits' NMvs across strategic digital health investment scenarios and project options.

Weighting and Scoring Benefits
If organizations choose not to use NMvs for intangible benefits, they can use weighting and scoring techniques. Teams have to select a suitable scale to assign values to each weighted benefit. Used in combination with estimated values for tangible benefits, they enable two sets of comparison. One is comparing broad estimates of benefits of strategic scenarios; the other is comparisons of benefits of each option for individual digital health projects. Seeking relative differences means assigning scores within a total for all major benefits of each scenario or option. It maintains relativity and is a cardinal method. The total available points are determined by the number of benefits. For example, if a project has 20 benefits, a total of 100 may be appropriate for an average of 5 per benefit. The same benefit range should be used for all scenarios and options to ensure consistency. If one option only has 19 of the 20 benefits, the total points available would be 95, again 5 points per benefit.
The technique applies to both relative weighting and scoring. Relative weighted scores are simple multiplications of each benefit's relative weight by its score. The total for each scenario and option can be divided by their estimated cumulative total costs and cumulative benefits for comparison with other scenarios and options.
Weighting and scoring should be completed by representative groups of stakeholders. These can include decision-makers. It is best if groups meet at a single event so that each stakeholder's issues and themes are available for the others to hear and consider in making their weighting and scoring decisions. DHIF modelers should be excluded from weighting and scoring because they are not stakeholders.
This is only part of the appraisal. It is necessary to present the relative weighted score results for each scenario and option to decision-makers. They can be accompanied with each scenario's and option's unit cost per weighted score, and alongside the estimated values of tangible benefits and their socioeconomic returns derived from costs and tangible benefits. Their financing and affordability profiles should be shown also.