Revista de Administração Revista de Administração
Revista de Administração 2017;52:256-67 DOI: 10.1016/j.rausp.2017.05.005
Information technology
A study on the impact of non-operational mechanisms on the effectiveness of public information technology governance
Um estudo sobre o impacto dos mecanismos não operacionais na efetividade da governança de tecnologia da informação pública
Un estudio acerca de la influencia de los mecanismos no operacionales en la efectividad de la gobernanza de tecnología de la información pública
Leonel Cerqueira Santos, , Carlos Denner dos Santos
Universidade de Brasília, Brasília, DF, Brazil
Received 15 February 2016, Accepted 01 December 2016
Abstract

This study proposes an analysis of non-operational mechanisms that can impact the effectiveness of information technology governance in the Brazilian public administration. A questionnaire was sent to information technology managers and experienced professionals in the area of information technology governance, who work in public organizations. Based on the collected data, we used the structural equations technique to verify the impact of the following mechanisms on the effectiveness of information technology governance: top management support; performance of the information technology steering committee; use of information technology strategic planning; performance of the information technology investment portfolio management. The study indicates that the first three mechanisms do not influence the perceived effectiveness of information technology governance in the Brazilian public sector. However, the performance of the information technology investment portfolio management has a direct and positive consequence on effectiveness; also, as a mediating variable, it significantly affects the performance of the information technology steering committee, increasing the effectiveness of Brazilian public information technology governance. Through the achieved results and the analysis of these effects, this study aims to contribute theoretically to the literature on information technology governance, as there are few studies that examine which mechanisms assist to establish this effectiveness in public organizations, and even less when it comes to the Brazilian public sector.

Resumo

Neste estudo, os autores propõem uma análise dos mecanismos não operacionais que podem impactar na efetividade da governança da tecnologia da informação na administração pública. Nesta pesquisa utilizou-se um questionário, o qual foi aplicado em gestores da área de tecnologia da informação e profissionais experientes na área de governança de tecnologia da informação que trabalham em organizações da administração pública. A partir dos dados coletados, aplicou-se a técnica de equações estruturais para verificar o impacto dos seguintes mecanismos na efetividade da governança de tecnologia da informação: suporte da alta administração; atuação do comitê de direção de tecnologia da informação; utilização do planejamento estratégico de tecnologia da informação; atuação da gestão de portfólio de investimentos de tecnologia da informação. Os resultados indicam que os três primeiros mecanismos não têm influência sobre a efetividade percebida da governança de tecnologia da informação do setor público. Porém, a atuação da gestão de portfólio de investimentos de tecnologia da informação influencia direta e positivamente a efetividade, sendo que também através da sua presença como variável mediadora, é que a atuação do comitê de tecnologia da informação passa a ter efeito significativo sobre a efetividade da governança de tecnologia da informação pública. Com os resultados obtidos e a análise destes efeitos, este trabalho almeja contribuir teoricamente para a literatura de governança de tecnologia da informação dentro do contexto em que existem poucas pesquisas que examinam quais mecanismos contribuem para estabelecer essa efetividade dentro das organizações públicas, sendo este número ainda mais incipiente quando o ambiente se restringe ao setor público brasileiro.

Resumen

En este estudio los autores proponen un análisis de los mecanismos no operacionales que pueden influir en la efectividad de la gobernanza de tecnología de la información en la administración pública. Se ha aplicado un cuestionario a gestores del área de tecnología de la información y a profesionales experimentados en el área de gobierno de tecnología de la información que trabajan en organizaciones de la administración pública. Se ha aplicado a los datos recogidos la técnica de ecuaciones estructurales con el fin de verificar la influencia de los siguientes mecanismos en la efectividad de la gobernanza de tecnología de la información: apoyo de la alta dirección; actuación de la comisión de dirección de tecnología de la información; utilización del plan estratégico de tecnología de la información; actuación de la gestión de cartera de inversiones de tecnología de la información. Los resultados indican que los tres primeros mecanismos no tienen influencia en la efectividad percibida de la gobernanza de tecnología de la información del sector público. Sin embargo, la actuación de la gestión de cartera de inversiones de tecnología de la información influye directa y positivamente en la efectividad; además, es por medio de su presencia como variable mediadora que la actuación de la comisión de tecnología de la información logra tener un efecto significativo sobre la efectividad de la gobernanza de tecnología de la información pública. Con este trabajo, mediante los resultados obtenidos y el análisis de estos efectos, se pretende contribuir teóricamente a la literatura de gobernanza de tecnología de la información, en un contexto en que se observan pocos estudios dedicados a examinar cuáles mecanismos contribuyen a establecer esta efectividad dentro de las organizaciones públicas, y siendo este número aún más reducido cuando el ambiente se limita al sector público en Brasil.

Keywords
Effectiveness, IT Governance, Mechanisms of IT Governance, Stakeholders, Brazilian Public Administration
Palavras-chave
Efetividade, Governança de TI, Mecanismos de Governança de TI, Stakeholders, Administração Pública Federal
Palabras clave
Efectividad, Gobernanza de TI, Mecanismos de gobernanza de TI, Stakeholders, Administración pública federal
Introduction

Brazilian public administration is inspired by the theoretical models of the New Public Management and New Public Governance, which require state efficiency together with the provision of public services of higher quality, and based on social control and warranty of individual rights (Bresser-Pereira, 2007; Imasato, Martins, & Pieranti, 2011).

In this scenario, public organizations are subject to the influence of stakeholders in establishing their institutional activities, since the interested parts affect and are also affected by the achievement of the organizations’ objectives (Freeman & Reed, 1983).

All these influential conditions lead the public sector to improve its corporate governance, as it helps organizations make better decisions, thus enhancing government's operation (Carvalho & Laurindo, 2007; Chan, Sabherwal, & Thatcher, 2006; Cumps et al., 2009; Davenport, 1993; Earl, 1993; Henderson & Venkatraman, 1993; Hirscheim & Sabherwal, 2001; Jorfi, Nor, Najjar, & Jorfi, 2011; Niederman, Brancheau, & Wetherbe, 1991; Nolan & McFarlan, 2005). This idea becomes stronger when the subject is information technology (IT), since it has turned into an integral part of the organizational strategy.

Hence, in the technological field, IT governance is established as part of corporate governance, which is understood as a set of mechanisms of structures, procedures and relationships defined by top management, in order to direct IT actions and to exercise control over their use and management (Information Systems Audit and Control Association [ISACA], 2011; Peterson, 2003; Weill & Ross, 2006).

Several researchers have studied the implementation of mechanisms of IT governance and their impact on the effectiveness of corporate governance, but most of them focus on the private sector (Al-Farsi & Haddadeh, 2015; Ali & Green, 2007, 2012; Bermejo, Tonelli, & Zambalde, 2014; Bowen, Cheung, & Rohde, 2007; Bradley et al., 2012; De Haes & Van Grembergen, 2008; Heindrickson & Santos, 2014; Huang, Zmud, & Price, 2010; Liang, Chiu, Wu, & Straub, 2011; Lunardi, Dolci, Maçada, & Becker, 2014; Nfuka & Rusu, 2010; Rau, 2004; Schlosser, Beimborn, Weitzel, & Wagner, 2015; Vaswani, 2003; Weill & Woodham, 2002).

In view of this context, this research aimed to answer the following question: which non-operational mechanisms affect the level of perceived effectiveness of IT governance in public administration?

Hence, this study sought a theoretical and practical contribution on how IT governance and its mechanisms evolve in public organizations, which differ from the private sector. Government institutions are strongly influenced by the political environment and by the accomplishment of legal norms and provisions, besides having to carry out ceremonies, rites and cultural myths, rather than the search for investment remuneration and financial results (Campbell, McDonald, & Sethibe, 2009; Rodrigues & Souza, 2014; Tonelli, Bermejo, Santos, Zuppo, & Zambalde, 2015).

The next sections present the theoretical framework, the methodology used for data collection, the presentation and analysis of the data, and in the end, the final considerations, with the research limitations and suggestion for future studies.

Theoretical models of public administration: new public management and new public governance

According to Denhardt (2008), the New Public Management (NPM) had its starting point in an academic seminar held in 1968 at the Minnowbrook Convention Center, Syracuse University, New York. Its bases are related to the following factors: focus on the citizen; orientation toward results, through strategic planning and performance indicators; emphasis on transparency and social control; management flexibilization, and appreciation and development of civil servants (Hood, 1995).

For NPM, the relationship between commitment to results for the citizen and responsibility of politicians and civil servants raised the idea of effectiveness in the context of the public sector. This effectiveness moves toward more flexible organizations (reduction of bureaucracy), innovative and efficient, whose main focus for the delivery of public services is the citizen (called “customer”), who receives a more personalized treatment.

In Brazil, NPM began to be implemented in 1990, during President Fernando Collor de Mello's government (1990–1992), and became central during Fernando Henrique Cardoso's first term (1995–2002), through the creation of the Master Plan for Reform of the State Apparatus, which defined the guidelines and established the goals for the public administration reform, justified by the crisis about the State role (Brasil, 1995; Ferrarezzi & Amorim, 2007).

However, there are gaps when NPM makes use of private sector business principles to improve government performance. This inconsistency occurs due to the lack of denominators such as “profit” or “return on investment” in the public sector, which are common in the private environment. Therefore, it is difficult to make a common comparison across the full range of public programs, as these are based on the citizens’ trust in the legitimacy of their political institutions, rather than in the market-centered efficiency (Larsen, 2008).

These gaps in NPM have generated a countermovement called New Public Governance (NPG) that places political values at the center of the debate, as opposed to the strictly instrumental focus of NPM. NPM puts much more emphasis on citizen's participation than on the efficiency of public policies’ implementation (Alford, 2002; Stoker, 2006).

Information technology governance and its effectiveness

The implementation of the New Public Management reinforced the inclusion of measures that combined the achievement of the requested organizational performance and the demand for accountability (Barrett, 2003). This need becomes stronger with the New Public Governance, since it considers citizens as co-producers, who actively operate in the provision of services and decision making, by demanding excellent service rendering (Bovaird, 2007; Fotaki, 2011; Powell, Greener, Szmigin, Doheny, & Mills, 2010; Simmons, Birchall, Doheny, & Powell, 2007).

These ideas are consistent with the concept of public corporate governance, since this has, in its essence, the goal of balancing the competitiveness and productivity of institutions with a responsible and transparent management, which is reflected in the public technology area especially through its new role in the institutional setting. Currently, information technology is considered a critical and strategic asset for organizations, whether public or private, and corporations are increasingly dependent on IT (Campbell et al., 2009; Gelinas, Sutton, & Fedorowicz, 2004; Schlosser et al., 2015).

Therefore, this new concept made IT in public administration involve factors that go far beyond technology – people, structures, processes and, above all, knowledge – and all of them must be articulated so that computing resources, in fact, respond appropriately to the aspirations of public administration and society (Dunleavy, Margetts, Bastow, & Tinkle, 2006, Holden, 2007; Juiz, Guerrero, & Lera, 2014).

Thus, IT plays the role of a fundamental and transforming element of the public organization, being no longer just an object of management, but of governance. Hence, public IT enters the field of IT governance, which emerges –as a field of study and practical discipline – as a subset of corporate governance. It specifies decision rights and the framework of responsibilities in order to stimulate desirable behaviors in the use of IT, being the responsibility of the organization's board of directors and executive management (Weill & Ross, 2006).

In the specific case of the public environment, governance must follow the environment's objective, which is based on responsibilities to the House of Representatives and to taxpayers, acquisition and distribution of power in society, and citizens and stakeholders’ demands for advertising the acts and transparency of public service; in other words, there are activities that do not exist in the private sector or that exceed its level of requirement (Barrett, 2003; Mello, 2006). It must be emphasized that such stakeholders are important because they have resources on which organizations depend, which give them power over these institutions (Pfeffer & Salancik, 1978).

The discussion on the effectiveness of technological governance arises from the need to evaluate the quality of results provided by IT governance in the organizations.

Regarding IT governance, effectiveness is characterized by the achievement of goals related to cost, growth, asset utilization and business flexibility, as well as meeting legal and regulatory requirements (Bowen et al., 2007). According to Weill and Ross (2006), the accomplishment of these objectives is expressed by effective IT governance that meets the following critical factors: transparency, active participation, frequent redesign, establishment of governance at multiple organizational levels, simplicity, existence of a process for treating exceptions and alignment of incentive and reward systems.

Thus, the implementation of effective IT governance involves a set of processes whose purpose is to meet the expectations regarding the organization's critical factors of success (ISACA, 2011). The effectiveness of IT governance, both in public and private environments, will be achieved depending on how the quality of the results derived from decision-making structures, processes and relational mechanisms for the direction and control of IT operations affects the accomplishment of IT goals (Sambamurthy & Zmud, 1999; Torres, 2004), and consequently will be reflected in the organizational objectives.

This effectiveness will be reached through the performance of a set of arrangements and practices associated with structure, processes and relationships (Grembergen, De Haes, & Guldentops, 2004), which act in order to meet the objectives of the organization regarding information technology, and have the following meanings:

  • 1)

    Structural mechanisms – involve the organization and placement of the IT area in the hierarchical structure of the institution, as well as the clear definition of roles and responsibilities of the positions that compose this structure (De Haes & Van Grembergen, 2008);

  • 2)

    Processual mechanisms – deal with strategic decision-making, IT strategic planning and frameworks for monitoring, control and processes (De Haes & Van Grembergen, 2008);

  • 3)

    Relational mechanisms – complement IT governance, incorporating IT relations with other areas of the organization and their users. These mechanisms guarantee the functioning of the structure created for IT and of the established processes (De Haes & Van Grembergen, 2008).

In discussing which and how many mechanisms are relevant for IT governance effectiveness, this research confirms previous studies in this area, which state that among several mechanisms of IT governance listed and documented, only a limited number are truly significant and can impact the effectiveness of public technology governance.

De Haes and Van Grembergen (2008) and Lunardi et al. (2014) examined some of these surveys and concluded that there are five practices which are more effective compared to others, for example, IT steering committees and portfolio management, considering that the research was done in private institutions. On the other hand, through single case studies conducted in large organizations, Bowen, Cheung and Rohde (2007) and Ali and Green (2012) explored the factors that influence the mechanisms of IT governance, showing that its performance is associated with mechanisms such as shared understanding of the objectives between business and IT, the active involvement of IT committees in management and decisions, strategies and policies shared and communicated between business and IT.

In a recent study, Heindrickson and Santos (2014) found that mediation of the process of IT investment portfolio is necessary, so that the perceived effects of performance improvement on the IT committee and on the IT solution manager upon governance effectiveness may be statistically significant.

Therefore, based on these studies, the mechanisms described below, which have a non-operational nature, were examined in this research:

  • 1)

    Relational mechanismTop management support – the commitment of top management regarding initiatives related to IT governance enhances IT success and helps to integrate it with strategies, business processes, stakeholders’ expectations, and institutional mechanisms that ensure the permanence of IT investments over time (Armstrong & Sambamurthy, 1999; Trites, 2004; Weill & Ross, 2004);

  • 2)

    Structural mechanismPerformance of the IT committee –literature argues that the performance of the structural mechanism “IT committee” in planning and aligning responsibilities to the business is fundamental for the support of governance structures, planning and management of processes and information systems; hence, it is an important governance mechanism used by companies that show better performance (Doll & Torkzadeh, 1987; Earl, 1989; Gupta & Raghunathan, 1989; ISACA, 2011; Ragunathan & Ragunathan, 1989; Weill & Ross, 2006). The committee is composed of a team of executives from different areas of the organization with the main function of connecting business and IT strategies, as well as making decisions regarding projects’ selection and prioritization (Castro & Carvalho, 2010; McKeen & Guimarães, 1985; Moraes & Laurindo, 2003; Nolan, 1982).

    Hence, by determining methods, forms of management and investment priorities in IT businesses, this mechanism has the mission of managing resources coming from the stakeholders that take part in the control and provision of institutional resources (De Haes & Van Grembergen, 2009; Hardy, 2003);

  • 3)

    Processual mechanismPerformance of the IT investments portfolio management – the objective of the processual mechanism of IT investments portfolio management is to improve the use of institutional resources in order to select a set of projects and programs that can bring the highest possible return to the organization (De Haes & Van Grembergen, 2008). This mechanism includes prioritization processes, investments and projects in which business and IT are involved, and its development and existence are linked to the mechanism of IT committee (De Haes & Van Grembergen, 2009; Heindrickson & Santos, 2014). Portfolio management and the IT steering committee are interconnected, because the committee operates especially in the phases of prioritization, authorization and review of investments to be made (Archer & Ghasemzadeh, 1999; De Haes & Van Grembergen, 2008; Heindrickson & Santos, 2014).

    Therefore, in the joint performance of committee and portfolio management, stakeholders express their direction and actions inside the organizations are legitimated. That happens because these mechanisms provide the form of allocation/control of IT resources and deal with the trade-off between profitability/cost efficiency and the sustainable, reliable and stable growth of IT (Frooman, 1999; Korac-Kakabadse & Kakabadse, 2001);

  • 4)

    Processual mechanismUse of IT strategic planning – it is a formal process to define and update IT strategy, and concentrates all the methods, processes and implementations necessary for IT to successfully meet the organizational strategies (Lederer & Sethi, 1988).

Based on these mechanisms, the following hypotheses were raised:H1

Top management support has a direct and positive influence on the effectiveness of IT governance.

H2

The performance of the IT committee has a direct and positive influence on the effectiveness of IT governance.

H3

The total effect (direct and indirect effects) of the performance of the IT committee has a direct and positive influence on the effectiveness of IT governance.

H4

The performance of the IT committee has a direct and positive influence on the performance of the IT investments portfolio management.

H5

The performance of the IT investments portfolio management has a direct and positive influence on the effectiveness of IT governance.

H6

The use of IT strategic planning has a direct and positive influence on the effectiveness of IT governance.

Conceptual model

Based on the studies previously mentioned we present in Fig. 1 the model that summarizes the addressed topic, with its variables, relations and hypotheses that were tested. It points out that the mechanisms Performance of the IT Committee and Performance of the IT Investments Portfolio Management were extracted from the model adopted by Heindrickson and Santos (2014), and the mechanisms top management support and use of IT strategic planning were taken from an exploratory study conducted by De Haes and Van Grembergen (2009).

Fig. 1.
(0.12MB).

Conceptual model and hypotheses.

Method

This research used quantitative data collection and quantitative analysis techniques; the methodological process was divided in two phases.

First, a quantitative field research (survey) was carried out through the application of a questionnaire (Appendix A) to the universe of Brazilian federal public administration. This made the study even more complex due to the difficulty of accessing IT governance executives, which we defined as the subjects of the survey.

In the second phase, the statistical technique of structural equations was applied to the collected data, to analyze the participants’ responses.

The choice of the research universe came primarily from the idea that, using the New Public Management and New Public Governance as a basis, the government should undergo a reform that has the announced mission of enhancing the efficiency of public services, improving the management of available resources and increasing its transparency, which influenced the Master Plan for Reform of the State Apparatus (Diniz, Barbosa, Junqueira, & Prado, 2009).

This influence based on efficiency led the Plan to determine, in four distinct groups (Strategic Core, Exclusive Activities, Non-Exclusive Services and Production of Goods and Services for the Market), the activities that should be carried out by the State and, for each group, the most adequate form of management to be implemented.

In our study, we focused on the groups that involve activities carried out exclusively by the State, involving organizations that belong to the Strategic Core and Exclusive Activities groups.

Strategic Core is the sector that defines laws and public policies, and demands their enforcement, being responsible for strategic decision making. It corresponds to the Legislative and Judicial, the Public Ministry and, in the Executive Branch, to the President of the Republic and the ministries responsible for the planning and formulation of public policies (Brasil, 1995).

Exclusive Activities comprise the sectors where services are provided only by the State. These are services in which the extrovert power of the State is exercised, as the power to regulate, to inspect, and to promote, and are exemplified by the collection and control of taxes, the police, and basic social security (Brasil, 1995).

Within this universe the sample was delimited to managers of the information technology area and to experienced professionals in the area of IT governance, in different types of organizations that compose the Brazilian federal public scene.

These individuals answered a questionnaire that was sent with an electronic letter of presentation. The questionnaire (Appendix A) was developed using the Google Docs tool, and the questions were elaborated based on the objectives, activities and results of the governance mechanisms studied, as well as on the attributes of IT governance effectiveness, based on academic studies and best practices of IT governance (Ali & Green, 2007; Archer & Ghasemzadeh, 1999; Castro & Carvalho, 2010; Heindrickson & Santos, 2014; ISACA, 2011; IT Governance Institute, 2007; Karimi, Bhattacherjee, Gupta, & Somers, 2000; Moraes & Laurindo, 2003; Vaswani, 2003; Weill & Ross, 2006; Weill, Woerner, & McDonald, 2009).

The queries asked for responses on a Likert scale (do not know/do not respond, totally disagree, disagree, neither agree nor disagree, agree, totally agree), and were analyzed through the structural equations technique.

Structural equation modeling is a set of techniques and procedures that are an extension of other multivariate techniques, and evaluate the dependency relations between one or more variables, representing those that cannot be measured directly by using other variables called latent constructs.

The associations of these variables can be estimated through covariance and are discriminated in the linear model, which can be separated into two: measurement model, which makes the connection between the measurement instrument (observed variables) and the theoretical construct (latent variables); and structural model, which defines the casual or association relationships between variables and unobserved constructs (latent variables), specifying whether a latent variable causes direct or indirect changes in other latent variables. Fig. 2 shows the structural/measurement model according to the graphical representation of structural equation modelling.

Fig. 2.
(0.21MB).

Structural/measurement model and hypotheses.

Following this model, we did the programming by using the R Statistics free software, with the support of Lavaan package (latent variable analysis), to make the model diagnosis and to complete the hypothesis test proposed in this research.

Results

For data collection, we sent invitations by e-mail. A total of 137 civil servants from 58 federal public organizations, who are managers of the IT area, participated in the study. They are experienced professionals in their working areas or understand the subject. Table 1 shows the distribution of respondents according to important features of the sample.

Table 1.

Characteristics of the sample.

Characteristics of the sample (n=137)
Characteristic  Quantity  Percentage (%) 
Type of public organization
Direct Public Administration of the Executive Branch  59  43.1 
Autarchic and Foundational Public Administration of the Executive Branch  37  27.0 
Judicial Power, Prosecuting Counsel, Public Defense  22  16.1 
Legislative Power and The Federal Court of Accounts  19  13.9 
Working time in the institution
1–3 years  39  28.5 
4–9 years  58  42.3 
10–15 years  10  7.3 
16–20 years  6.6 
20–25 years  6.6 
26–30 years  4.4 
Over 30 years  4.4 
Gender
Male  116  84.7 
Female  21  15.3 
Age group
18–29 years old  17  12.4 
30–39 years old  50  36.5 
40–49 years old  42  30.7 
50–59 years old  20  14.6 
Over 60 years old  0.0 
Education degree
Bachelor  24  17.5 
Specialist  75  54.7 
Master  33  24.1 
Doctorate  3.6 

The sample size rule was respected, given that it should have the minimum value of ten cases per item (manifest variable) of the questionnaire (Nunnally, 1967; Everitt, 1975); therefore, for this research the minimum number of participants should be 120.

The research ensured the confidentiality of the respondents, focusing on their perception about the connection between the mechanisms of IT governance and the effectiveness of the governance in their organizations.

With these data in hand, in addition to the conceptual and structural models, we conducted the analysis through the programming built in the R statistics free software. The initial analysis was divided in two stages: study of the model's identifiability and of the indices that reveal its global fit.

In the case of models with combined latent variables and measurement, such as the model proposed in this study, identifiability was confirmed by applying the t rule and the two-step rule (Bollen, 1989; Marôco, 2010).

Regarding the forms of diagnosing the model's global fit, the following statistical tests were applied, and results are shown in Table 2.

Table 2.

Results of the diagnostic tests for the model's fit.

Statistical test  Result  Significance 
Chi-square test
Minimum function test statistic value  69.293  Rejected hypothesis 
Chi-squared p-value  0.0158a  Rejected hypothesis 
Comparative fit index (CFI)  0.975  Very good fit 
Tucker–Lewis index (TLI)  0.964  Very good fit 
Root mean square error of approximation (RMSEA)
RMSEA  0.061  Accepted hypothesis 
p-value  0.259a  Accepted hypothesis 
a

Obs: p-value0.05.

By analyzing the various tests, we observed that only the chi-square test of adjustment is against the theoretical formulation, since most of the other tests have confirmed the model's adequate fit.

Another item that strengthened the suitable fit of the model and showed its acceptable explanation rate was the determination coefficient (R2). The model's R2 had the value of 0.795, that is, 79.5% of all variability of IT governance effectiveness is explained by the relationships presented in the theoretical formulation.

As to the hypothesis test, as shown in Table 3, all proposed relationships are significant (significance less than or equal to 0.05), and the marked variables were excluded from the analysis. The exceptions regarding significance were found in the following associations: in the relationship between Top Management Support (TMS) and Effectiveness of IT Governance (EITG), whose p-value is equal to 0.166; in the influence of Use of IT Strategic Planning (UITSP) on Effectiveness of IT Governance (EITG), whose p-value was 0.150; and in the relationship between Performance of IT Committee (PITC) and Effectiveness of IT Governance (EITG), whose p-value is equal to 0.980.

Table 3.

Results of the relationship between the model variables.

Variables (n=137)  EITG  TMS  UITSP  PITC  PITIPM 
EITG
Correlation coefficient  1.000  0.304  0.222  −0.031  0.393 
p-value  –  0.166a  0.150a  0.980a  0.016 
PITIPM
Correlation coefficient  –  –  –  0.256  1.000 
p-value  –  –  –  0.01a  – 
a

The p-value has significance level of 5%.

The Performance of IT Committee (PITC) on the Performance of IT Investments Portfolio Management (PITIPM) is significant, with a p-value of 0.001; and also significant is the influence of Performance of IT Investments Portfolio Management (PITIPM) on Effectiveness of IT Governance (EITG), since its p-value is 0.016.

Thus, we can conclude that the observed relationship between the independent variable Performance of IT Committee (PITC) and the dependent variable Effectiveness of IT Governance (EITG) is more effective when this linkage occurs through the mediator variable Performance of IT Investments Portfolio Management (PITIPM) than when there is a direct influence of the committee (PITC) on the Effectiveness of IT Governance (EITG).

Our understanding is strengthened when we observe the negative regression coefficient of the relationship between Performance of IT Committee (PITC) and Effectiveness of IT Governance (EITG) (estimated value=−0.0301), in opposition to the positive coefficient derived from the multiplication of the regression coefficients of the influence of the committee (PITC) on the investments portfolio management (PITIPM), and the influence of PITIPM on EITG (estimated value of total effects=0.256×0.393=0.101).

Continuing the analysis, again it is possible to notice the non-significant role of IT Committee (PITC) on Effectiveness of IT Governance (EITG) when examining the variances of the system components, since this mechanism has a non-significant variance (p-value=0.084).

Therefore, based on the observed statistical evidence, Fig. 3 presents the structural model with the estimates and significance of each hypothesis – where the rejected hypotheses are written in red.

Fig. 3.
(0.21MB).

Structural model with estimated values and significance of hypotheses.

Discussion and final considerations

This study sought to identify and understand the achievement of goals defined by the strategies of the organization for IT governance, through the relationships between the mechanisms of IT governance and its effectiveness.

The hypothesis that top management support has a direct and positive influence on the effectiveness of IT governance was rejected. This happens because the contemporary Brazilian public administration is not fully built according to the New Public Management and New Public Governance models, and shows a combination of features derived from patrimonialism and the Weberian bureaucratic model. New Public Management and New Public Governance idealize top management based on technical and meritocratic competences, inducing modernization through the use of information technologies, and supporting government programs that use technology to promote performance, efficiency, effectiveness, transparency and quality in the relationships with citizens and other stakeholders (Agune & Carlos, 2005; Bresser-Pereira, 2007; Grant & Chau, 2005; Tan and Pan, 2003). And this does not fully happen in the Brazilian federal public sector.

About the Performance of the IT Committee, we emphasize that it is a structural mechanism; therefore it is involved with the organizational composition for the functioning of IT governance, which implies the need for arrangements. And these arrangements are built differently in public and private organizations.

In the private sector, there is a clear definition of the roles and responsibilities of the committees, as they are created with the main objective of monitoring and directing the company's information technology. Having defined their goal, the executives consequently execute their functions within boundaries established inside the group, in order to give return (profit) to the institution. This way of operating explains the positive influence of the performance of the IT committee observed in previous studies in the private environment (Ali & Green, 2007, 2012; Karimi et al., 2000; Lunardi et al., 2014; Vaswani, 2003).

In the public sector this is not clear, because there are many levels of responsibility (and interests) inside and outside the organizations, at local, state and federal levels, which increase tensions and conflicts in doing business (Bermejo et al., 2014; Campbell et al., 2009; Edwards & Clough, 2005). Therefore, the hypothesis that the performance of the IT committee has a direct and positive influence on the effectiveness of IT governance was rejected.

Given this situation, the research found that only through mediation of the IT Investments Portfolio Management the perceived effects of improved performance of the IT committee on effectiveness become statistically significant (H3, H4 and H5 hypotheses).

This interaction of mediation and the significant relationship between IT committee and portfolio management is important and affects the model as a whole, probably because when dealing with the performance mechanism in IT Investments Portfolio Management we use a criterion common to the budget area.

Budget issues may have a significant impact on the process of formulating IT governance policies in the public sector. After all, the budget process is the arena where the allocation of scarce resources of the society is determined and, as such, it is a key element of the policy-making process. Even if the policies have already been approved in other levels, they must go through the budget area to ensure that the necessary resources are available for implementing them effectively (Banco Interamericano de Desenvolvimento [BID], 2007).

Hence, we observe the importance of the performance of IT investments portfolio management, since it aims to prioritize projects and investments, deciding how to split scarce resources among them, taking into account resource limitations and business needs, by aligning them with organizational strategic objectives and considering the high degree of uncertainty associated with the use of technologies (Hall & Liedtka, 2005; Xue, Liang, & Boulton, 2008).

Therefore, the IT committee alone does not impact effectiveness, and only achieves this goal when it receives the responsibility of aligning IT actions and investments, justifying technology investments in face of a rigid budget structure (Kumar, Ajjan, & Niu, 2008).

In addition, the Federal Court of Auditors, institution of the Brazilian government responsible for external control, recommends that public organizations establish an IT committee to provide the allocation of public resources according to their needs and priorities (Brasil, 2008).

Regarding the use of IT strategic planning, there is a difference between discourse and practice. In the speech, strategic planning is said to be the main method for defining actions; however, in practice it happens through institutional processes of isomorphism, best practices, and benchmarking. These processes proliferate in the environment of public institutions, which are systematized and interconnected in a discretionary way that encourages the adoption of models and practices for reasons that go far beyond efficiency (Bermejo, Tonelli, Zambalde, Brito, & Todesco, 2012; Zattoni & Cuomo, 2008).

Limitations and suggestions for future research

This study has restrictions regarding the size of the sample, which is small due to the number of Brazilian federal public organizations that belong to the ‘strategic core’ and ‘exclusive activities’ groups. However, we should highlight as a positive aspect the use of the respondents’ opinions to evaluate the importance of the questionnaire items and their measure.

Concerning suggestions for future research, we propose studies that use Likert type scales and ranking categories directed to experts on the subject; in other words, scales that do not accept absence of answers, since respondents would have the ability and technical expertise to express relevant opinions.

There is also a need for studies on the effectiveness of IT governance and its mechanisms in government institutions that offer non-exclusive services and produce goods and services for the market, as well as expanding the research to state and local levels.

Finally, we suggest as future research the replication of this study with the purpose of validating its results and overcoming the limitations previously mentioned.

Conflicts of interest

The authors declare no conflicts of interest.

References
Agune and Carlos, 2005
R. Agune,J. Carlos
Governo eletrônico e novos processos de trabalho
Gestão pública no Brasil contemporâneo,
Al-Farsi and Haddadeh, 2015
K. Al-Farsi,R.E. Haddadeh
Framing information technology governance in the public sector: Opportunities and challenges
International Journal of Electronic Government Research, 11 (2015), pp. 89-101
Alford, 2002
J. Alford
Defining the client in the public sector: A social exchange perspective
Public Administration Review, 62 (2002), pp. 337-346
Ali and Green, 2007
S. Ali,P. Green
IT Governance mechanisms in public sector organizations: An Australian context
Journal of Global Information Management, 15 (2007), pp. 41-63
Ali and Green, 2012
S. Ali,P. Green
Effective information technology (IT) governance mechanisms: An IT outsourcing perspective
Information Systems Frontiers, 14 (2012), pp. 179-193
Archer and Ghasemzadeh, 1999
N.P. Archer,F. Ghasemzadeh
An integrated framework for project portfolio selection
International Journal of Project Management, 17 (1999), pp. 207-216
Armstrong and Sambamurthy, 1999
C.P. Armstrong,V. Sambamurthy
Information technology assimilation in firms: The influence of senior leadership and IT infrastructures
Information System Research, 10 (1999), pp. 304-331
Banco Interamericano de Desenvolvimento, 2007
Banco Interamericano de Desenvolvimento
A política das políticas públicas: Relatório de progresso econômico e social na América Latina
Elsevier, (2007)
Barrett, 2003
P. Barrett
Achieving better practice: Corporate governance in the public sector
Australian National Audit Office, (2003)
Bermejo et al., 2014
P.H.S. Bermejo,A.O. Tonelli,A.L. Zambalde
Developing IT governance in Brazilian public organizations
International Business Research, 7 (2014), pp. 101-114
Bermejo et al., 2012
P.H.S. Bermejo,A.O. Tonelli,A.L. Zambalde,M.J. Brito,J.L. Todesco
Implementation of information technology governance through IT strategic planning
African Journal of Business Management, 6 (2012), pp. 11118-11179
Bollen, 1989
K.A. Bollen
Structural equations with latent variables
John Wiley & Sons, (1989)
Bovaird, 2007
T. Bovaird
Beyond engagement and participation: User and community coproduction of public services
Public Administration Review, 67 (2007), pp. 846-860
Bowen et al., 2007
P.L. Bowen,M.D. Cheung,F.H. Rohde
Enhancing IT governance practices: A model and case study of an organization's efforts
International Journal of Accounting Information Systems, 8 (2007), pp. 191-221
Bradley et al., 2012
R. Bradley,T. Byrd,J. Pridmore,E. Thrasher,R. Pratt,V. Mbarika
An empirical examination of antecedents and consequences of IT governance in US hospitals
Journal of Information Technology, 27 (2012), pp. 156-177
Brasil, 1995
Brasil
Plano Diretor da Reforma do Aparelho do Estado
Presidência da República/Câmara da Reforma do Estado/Ministério da Administração Federal e da Reforma do Estado, (1995)
Retrieved from http://www.bresserpereira.org.br/documents/mare/planodiretor/planodiretor.pdf
Brasil, 2008
Brasil
Levantamento de auditoria: Situação da governança de tecnologia da informação – TI na administração pública federal
Tribunal de Contas da União, (2008)
Retrieved from http://www.tcu.gov.br/Consultas/Juris/Docs/judoc/Acord/20080814/008-380-2007-1-GP.doc
Bresser-Pereira, 2007
L.C. Bresser-Pereira
The structural public governance model
International Public Management Review, 8 (2007), pp. 16-32
Campbell et al., 2009
J. Campbell,C. McDonald,T. Sethibe
Public and private sector IT Governance: Identifying contextual differences
Australian Journal of Information Systems, 16 (2009), pp. 5-18
Carvalho and Laurindo, 2007
M.M. Carvalho,F.J.B. Laurindo
Estratégia competitiva: Dos conceitos à implementação
2a ed., Atlas, (2007)
Castro and Carvalho, 2010
H.G.D. Castro,M.M.D. Carvalho
Gerenciamento do portfolio de projetos: Um estudo exploratório
Gestão & Produção, 17 (2010), pp. 283-296 http://dx.doi.org/10.1016/j.rbre.2017.05.006
Chan et al., 2006
Y.E. Chan,R. Sabherwal,J.B. Thatcher
Antecedents and outcomes of strategic IS alignment: An empirical investigation
IEEE Transactions on Engineering Management, 3 (2006), pp. 27-47
Cumps et al., 2009
B. Cumps,D. Martens,M. De Backer,R. Haesen,S. Viane,G. Dedene
Inferring comprehensible business/ICT alignment rules
Information and Management, 46 (2009), pp. 116-124
Davenport, 1993
T.H. Davenport
Process innovation: Reengineering work through information technology
Harvard Business School Press, (1993)
De Haes and Van Grembergen, 2008
S. De Haes,W. Van Grembergen
Analyzing the relationship between IT governance and business/IT alignment maturity
Proceedings of the 41th Hawaii international conference on system sciences,
De Haes and Van Grembergen, 2009
S. De Haes,W. Van Grembergen
An exploratory study into IT governance implementations and its impact on business/IT alignment
Information Systems Management, 26 (2009), pp. 123-137
Denhardt, 2008
R.B. Denhardt
Theories of public organization
5th ed., Thomson/Wadsworth, (2008)
Diniz et al., 2009
E.H. Diniz,A.F. Barbosa,A.R.B. Junqueira,O. Prado
O governo eletrônico no Brasil: Perspectiva histórica a partir de um modelo estruturado de análise
RAP – Revista de Administração Pública, 43 (2009), pp. 23-48
Doll and Torkzadeh, 1987
W.J. Doll,G. Torkzadeh
The relationship of MIS steering committees to size of firm and formalization of MIS planning
Communications of the ACM, 30 (1987), pp. 972-978
Dunleavy et al., 2006
P. Dunleavy,H. Margetts,S. Bastow,J. Tinkle
New public management is dead – long live digital-era governance
Journal of Public Administration Research and Theory, 16 (2006), pp. 467-494
Earl, 1989
M.J. Earl
Management strategies for information technology
Prentice-Hall, (1989)
Earl, 1993
M.J. Earl
Experiences in strategic information systems planning
MIS Quarterly, 17 (1993), pp. 1-24
Edwards and Clough, 2005
M. Edwards,R. Clough
Corporate governance and performance: An exploration of the connection in a public sector context Issues Series Paper n. 1
Corporate Governance ARC Project, University of Canberra, (2005)
Everitt, 1975
S. Everitt
Multivariate analysis: The need for data, and other problems
British Journal of Psychiatry, 126 (1975), pp. 237-240
Ferrarezi and Amorim, 2007
E. Ferrarezi,S. Amorim
Concurso inovação na gestão pública federal: Análise de uma trajetória (1996–2006)
Cadernos Enap, 32 (2007), pp. 1-53
Fotaki, 2011
M. Fotaki
Towards developing new partnerships in public services: Users as consumers, citizens and/or co-producers in health and social care in England and Sweden
Public Administration, 89 (2011), pp. 933-955
Freeman and Reed, 1983
R.E. Freeman,D.L. Reed
Stockholders and stakeholders: A new perspective on corporate governance
California Management Review, 25 (1983), pp. 88-106
Frooman, 1999
J. Frooman
Stakeholder influence strategies
Academy of Management Review, 24 (1999), pp. 191-203
Gelinas et al., 2004
U. Gelinas,S. Sutton,J. Fedorowicz
Business processes and information technology
Thompson Learning, (2004)
Grant and Chau, 2005
G. Grant,D. Chau
Developing a generic framework for e-government
Journal of Global Information Management, 13 (2005), pp. 1-30
Gupta and Raghunathan, 1989
Y.P. Gupta,T.S. Raghunathan
Impact of information systems (IS) steering committees on IS planning
Decision Sciences, 20 (1989), pp. 777-793
Hall and Liedtka, 2005
J.A. Hall,S.L. Liedtka
Financial performance, CEO compensation, and large-scale information technology outsourcing decision
Journal of Management Information Systems, 22 (2005), pp. 193-221
Hardy, 2003
G. Hardy
Coordinating IT governance – A new role for IT strategy committees
Information Systems Control Journal, 4 (2003), pp. 21-24
Heindrickson and Santos, 2014
G. Heindrickson,C.D. Santos Jr.
Information technology governance in public organizations: How perceived effectiveness relates to three classical mechanisms
JISTEM – Journal of Information Systems and Technology Management, 11 (2014), pp. 297-326
Henderson and Venkatraman, 1993
J. Henderson,N. Venkatraman
Strategic alignment: Leveraging information technology for transforming organizations
IBM System Journal, 32 (1993), pp. 472-484
Hirschheim and Sabherwal, 2001
R. Hirschheim,R. Sabherwal
Detours in the path toward strategic information systems alignment
California Management Review, 44 (2001), pp. 87-108
Holden, 2007
S.H. Holden
The evolution of federal information technology management literature
Modern public information technology systems: Issues and challenges,
Hood, 1995
C. Hood
The new public management in the 1980s: Variations on a theme
Accounting, Organizations and Society, 20 (1995), pp. 95-109
Huang et al., 2010
R. Huang,R.W. Zmud,R.L. Price
Influencing the effectiveness of IT governance practices through steering committees and communication policies
European Journal of Information Systems, 19 (2010), pp. 288-302
Imasato et al., 2011
T. Imasato,P.E.M. Martins,O.P. Pieranti
Administrative reforms and global managerialism: A critical analysis of three Brazilian state reforms
Canadian Journal of Administrative Sciences, 28 (2011), pp. 174-187
ISACA, 2011
Information Systems Audit and Control Association
COBIT 5: A business framework for the governance and management of enterprise IT
ISACA, (2011)
IT Governance Institute, 2007
IT Governance Institute
About IT Governance framework (CobiT 4.0)
ITGI, (2007)
Jorfi et al., 2011
S. Jorfi,K. Nor,L. Najjar,H. Jorfi
The impact of IT flexibility on strategic alignment (with focus on export)
International Journal of Business & Management, 6 (2011), pp. 264-270 http://dx.doi.org/10.1542/peds.2016-4194
Juiz et al., 2014
C. Juiz,C. Guerrero,I. Lera
Implementing good governance principles for the public sector in information technology governance frameworks
Open Journal of Accounting, 3 (2014), pp. 9-27
Karimi et al., 2000
J. Karimi,A. Bhattacherjee,Y.P. Gupta,T.M. Somers
The effects of MIS steering committees on information technology management sophistication
Journal of Management Information Systems, 17 (2000), pp. 207-230
Korac-Kakabadse and Kakabadse, 2001
N. Korac-Kakabadse,A. Kakabadse
IS/IT governance: Need for an integrated model
Corporate Governance, 1 (2001), pp. 9-11
Kumar et al., 2008
R. Kumar,H. Ajjan,Y. Niu
Information technology portfolio management: Literature review, framework, and research issues
Information Resource Management Journal, 21 (2008), pp. 64-87
Larsen, 2008
G.L. Larsen
Emerging governance at the edge of constrained federalism: Public administrators at the frontier of democracy (PhD thesis)
Portland State University, (2008)
(PhD thesis)
Lederer and Sethi, 1988
A.L. Lederer,V. Sethi
The implementation of strategic information systems planning methodologies
MIS Quarterly, 12 (1988), pp. 445-461
Liang et al., 2011
T. Liang,Y. Chiu,S.P. Wu,D. Straub
The impact of IT governance on organizational performance
Proceedings of the 17th Americas conference on information systems,
Lunardi et al., 2014
G.L. Lunardi,P.C. Dolci,A.C.G. Maçada,J.L. Becker
Análise dos mecanismos de governança de TI mais difundidos entre as empresas brasileiras
Revista Alcance, 21 (2014), pp. 46-76
Marôco, 2010
J. Marôco
Análise de equações estruturais: Fundamentos teóricos softwares e aplicações
Report Number Ltda, (2010)
McKeen and Guimarães, 1985
J.D. McKeen,T. Guimarães
Selecting MIS projects by steering committee
Communications of the ACM, 28 (1985), pp. 1344-1352
Mello, 2006
G.R. Mello
Governança Corporativa no Setor Público Federal Brasileiro
Faculdade de Economia, Administração e Contabilidade, Universidade de São Paulo, (2006)
(Dissertação de Mestrado)
Moraes and Laurindo, 2003
R.O. Moraes,F.J.B. Laurindo
Um estudo de caso de gestão de portfolio de projetos de tecnologia da informação
Gestão & Produção, 10 (2003), pp. 311-328 http://dx.doi.org/10.1016/j.rbre.2017.05.006
Nfuka and Rusu, 2010
E.N. Nfuka,L. Rusu
Critical success factors for effective IT governance in the public sector organizations in a developing country: The case of Tanzania
Proceedings of the 18th European conference on information systems,
Niederman et al., 1991
F. Niederman,J.C. Brancheau,J.C. Wetherbe
Information systems management issues for the 1990's
MIS Quarterly, 15 (1991), pp. 475-500
Nolan, 1982
R.L. Nolan
Managing information systems by committee
Harvard Business Review, 60 (1982), pp. 72-99
Nolan and McFarlan, 2005
R. Nolan,F.W. McFarlan
Information technology and the board of directors
Harvard Business Review, 83 (2005), pp. 96-105
Nunnally, 1967
J.C. Nunnally
Psychometric theory
McGraw-Hill, (1967)
Peterson, 2003
R.R. Peterson
Information strategies and tactics for information technology governance
Strategies for information technology governance,
Pfeffer and Salancik, 1978
J. Pfeffer,G. Salancik
The external control of organizations: A resource dependence perspective
Harper & Row, (1978)
Powell et al., 2010
M. Powell,I. Greener,I. Szmigin,S. Doheny,N. Mills
Broadening the focus of public service consumerism
Public Management Review, 12 (2010), pp. 323-339
Raghunathan and Raghunathan, 1989
B. Raghunathan,T.S. Raghunathan
MIS steering committees: Their effect on information systems planning
Journal of Information Systems, 3 (1989), pp. 104-116
Rau, 2004
K.G. Rau
Effective governance of IT: Design objectives, roles, and relationships
Information Systems Management, 21 (2004), pp. 35-42
Rodrigues and Souza, 2014
J.G.L. Rodrigues,J. Neto Souza
Diretrizes para implantação da governança de tecnologia da informação no setor público brasileiro à luz da teoria Institucional
Revista do Serviço Público, 63 (2014), pp. 475-497
Sambamurthy and Zmud, 1999
V. Sambamurthy,R.W. Zmud
Arrangements for information technology governance: A theory of multiple contingencies
MIS Quarterly, 23 (1999), pp. 261-290
Schlosser et al., 2015
F. Schlosser,D. Beimborn,T. Weitzel,H.T. Wagner
Achieving social alignment between business and IT – An empirical evaluation of the efficacy of IT governance mechanisms
Journal of Information Technology, 30 (2015), pp. 119-135
Simmons et al., 2007
R. Simmons,J. Birchall,S. Doheny,M. Powell
Citizen governance: Opportunities for inclusivity in policy and policy making?
Policy and Politics, 35 (2007), pp. 457-478
Stoker, 2006
G. Stoker
Public value management: A new narrative for networked governance?
American Review of Public Administration, 36 (2006), pp. 41-57
Tan and Pan, 2003
C.W. Tan,S.L. Pan
Managing e-transformation in the public sector: An e-government study of the Inland Revenue Authority of Singapore (IRAS)
European Journal of Information Systems, 12 (2003), pp. 269-281
Tonelli et al., 2015
A.O. Tonelli,P.H.S. Bermejo,P.A. Santos,L. Zuppo,A.L. Zambalde
IT governance in the public sector: A conceptual model
Information Systems Frontiers, (2015), http://dx.doi.org/10.1007/s10796-015-9614-x
Torres, 2004
M.D.F. Torres
Estado, democracia e administração pública no Brasil
Editora FGV, (2004)
Trites, 2004
G. Trites
Director responsibility for IT governance
International Journal of Accounting Information Systems, 5 (2004), pp. 89-99
Van Grembergen et al., 2004
W. Van Grembergen,S. De Haes,E. Guldentops
Structures, processes and relational mechanisms for IT governance
Strategies for information technology governance,
Vaswani, 2003
R. Vaswani
Determinants of effective information technology (IT) governance
School of Business, University of Queensland, (2003)
(Honors Thesis)
Weill and Ross, 2004
P. Weill,J. Ross
IT governance: How top performers manage IT decision rights for superior results
Harvard Business School Press, (2004)
Weill and Ross, 2006
P. Weill,J. Ross
Governança de TI: Tecnologia da informação
Makron Books, (2006)
Weill et al., 2009
P. Weill,S.L. Woerner,M. McDonald
Managing the IT portfolio (update circa 2009): Infrastructure dwindling in the downturn. MIT Sloan Center for Information Systems Research
CISR Research Briefing, IX (2009), pp. 1-3
Weill and Woodham, 2002
P. Weill,P. Woodham
Don’t just lead, govern: Implementing effective IT governance
MIT Sloan CISR Working Paper n. 326, (2002),
Xue et al., 2008
Y. Xue,H. Liang,W. Boulton
Information technology governance in information technology investment decision processes: The impact of investment characteristics, external environment, and internal context
MIS Quarterly, 32 (2008), pp. 67-96
Zattoni and Cuomo, 2008
A. Zattoni,F. Cuomo
Why adopt codes of good governance? A comparison of institutional and efficiency perspectives
Corporate Governance, 16 (2008), pp. 1-15

Peer Review under the responsibility of Departamento de Administração, Faculdade de Economia, Administração e Contabilidade da Universidade de São Paulo – FEA/USP.

Copyright © 2017. Departamento de Administração, Faculdade de Economia, Administração e Contabilidade da Universidade de São Paulo ¿ FEA/USP
Revista de Administração 2017;52:256-67 DOI: 10.1016/j.rausp.2017.05.005