September 11, 2024
Volume 22, issue 4

Download PDF version of this article PDF

Assessing IT Project Success: Perception vs. Reality

We would not be in the digital age if it were not for the recurrent success of IT projects.

João Varajão and António Trigo

Project success has been a hot topic in the scientific and practitioner literature for a long time²³ and has been frequently reported with the assumption that IT projects show low levels of success.^6,11

On the one hand, there are well-known studies (e.g., the Standish Group Chaos reports^16,18) that point out low levels of success over a long period of time. On the other, the reported results are typically based on participants' perceptions and a limited set of criteria related to the Iron Triangle constraints (e.g., scope, time, and cost compliance), without providing evidence of whether project success was objectively evaluated.^20,22 These results have been questioned by several authors (e.g., Glass¹¹), given the samples of the studies and the insufficient criteria used for evaluating success, as they can be potential sources of bias.

This raises research question 1 (RQ1) about whether IT projects still tend to be problematic and unsuccessful endeavors, or can the reported levels of success be biased by the criteria used in the evaluation; and research question 2 (RQ2) regarding which objective measures (beyond the classical scope, time, or cost compliance, or even stakeholder satisfaction) can be used to support project success evaluation and reporting accuracy.

This article addresses these questions by examining IT project success based on data from a survey administered to experienced IT project managers. This study complements existing research by providing practitioners and researchers with a new perspective on IT project results and insights into project success. Namely, the results show that IT projects are achieving high levels of success; they stress the importance of considering a rich set of objective criteria when studying and evaluating project success, focusing on not only project execution, but also post-project outcomes and impacts.

Background

IT plays a central role in modern organizations and is a business core asset essential to improving productivity, reducing costs, and gaining competitive advantages—just to mention a few benefits.^3,24 Organizations must continuously innovate to maintain their competitive position, and the sustainable success of an organization is strongly associated with the success of IT projects such as those related to digital transformation.⁸

Projects are becoming more complex, and IT projects can take on many sizes and forms.²¹ They can encompass custom systems such as software development; COTS (commercial off-the-shelf) implementations such as ERP (enterprise resource planning); systems improvement; process improvement using IT; systems migration; infrastructure enhancement; and consultancy.⁵ Consequently, project success is a complex, multidimensional, dynamic, and relative concept.²³ In fact, the complexity and ambiguity surrounding the definition and measurement of project success have been recognized as problems themselves as awareness of success has evolved.^12,19 This is caused by, for example, differing perspectives on success by the project stakeholders.¹⁴

There are two main components of project success:² management and product. Project management success focuses on the process, particularly on the project's successful realization regarding scope, time, and cost—these three criteria indicate the degree of efficiency and effectiveness of project execution. The second component of project success is mainly related to the project's outcomes and impacts, and focuses on the effects of the resulting deliverables, usually in the post-project stage. Even though project management success and product success are not dependent on each other, the latter may be jeopardized if project management fails. Hence, the project and its resulting products and services cannot be considered in isolation.¹³ Typically, the reports on success found in the literature focus on project management.²⁰

Unfortunately, in recent decades IT projects have not been synonymous with "success."²² In fact, IT often seems like a victim of its own failures, and this perception is widespread.^11,19 The Standish Group reports are a landmark in the development of this vision of "failure." The Standish Group published its first "Chaos" report in 1994,¹⁶ and over time, with periodic publication of such reports, the idea has persisted that IT projects are problematic.¹³ For example, the Chaos report in 2020¹⁸ shows that only 31 percent of projects are successful, 50 percent are challenged (e.g., fail over time), and 19 percent fail completely.

These studies, although often cited, have been questioned and criticized by several researchers,¹¹ mainly because of: nondisclosure of the methodology or the instrument used for collecting data; no explanation of how the sample of participants was selected; inconsistent reporting; and having specific conceptions about the definition of success or failure. Other authors have also reported high levels of project failure (e.g., Frączkowski, et al.¹⁰), while still others have reported high levels of success (e.g., Bilir and Yafez⁴).

In spite of these differences and the apparently contradictory results of these studies, they share several common characteristics. The reported results are typically based on the perceptions of the participants (e.g., project managers);¹⁵ they consider only a few criteria related to project management (e.g., scope, time and cost compliance, and client satisfaction); and they lack evidence of objective evaluation of projects.^15,20

The study presented in this article takes into account these limitations and addresses the literature gap by examining the current status of IT project success and expanding the extant perspective regarding criteria for supporting the reported results, addressing both project management success and product success.

Method

An online questionnaire-based survey was administered to IT project managers. Given the goal of the study, a quantitative approach was used. The data was analyzed by applying descriptive and inferential statistics, using the SPSS (Statistical Package for the Social Sciences) software package.

Measurement instrument

The questionnaire measured several aspects of IT project success. The participants were asked to select one completed project they were involved in and to indicate the characteristics, level of global success achieved, and level of compliance with the scope, time, and cost, as well as vendor (supplier) satisfaction, client satisfaction, deliverables impact, and achievement of benefits verified in the project.

To provide additional support and evidence on the accuracy of the reported success, other variables were also considered, such as the use of deliverables in the post-project stage and variables related to future business, long-term benefits, and long-term relationships and partnerships with stakeholders, namely: hiring of project-related maintenance services; contracting of new projects; and recommendation of the vendor company by the project's client.¹⁵

All the items related to success used a semantic bipolar differential scale. For analysis, the line was divided into eight equal sections and coded from 0 ("I do not know"), 1 ("very low"/"total failure") to 7 ("very high"/"total success"). This scale is more precise than merely asking if there was compliance (e.g., with time or budget), which could have given rise to many interpretations. For example, consider a project with an agreed-upon change in scope between the vendor and the client, causing the project to extend for one month longer than originally planned. Faced with the question, "Was there compliance with time?," some participants might answer "yes" because even though the project ended one month later than planned, there was a justification for the extension. Other participants might answer "no" since the project's duration failed to meet the original plan.

By using a Likert scale, it is possible to indicate the level of compliance regarding each criterion and accommodate those nuances. Moreover, in a specific project, time noncompliance with a two-month delay would probably not be the same as an eight-month delay (and the severity of the delay depends on the specific project). Regarding the additional variables (e.g., the use of the deliverables in the post-project stage), the questions had four possible answers: "Yes," "No," "I do not know," and "Not applicable."

The context validity of the questionnaire was examined before administering the survey. Two professors of IT and project management, two researchers, and two IT project managers pilot-tested the survey. The results allowed for several minor refinements (mainly rewording) to the final questionnaire.

Data collection

To define the study sample, a database of contacts from 500 IT companies was initially created using several sources.¹ To complement the initial list, project managers were sought in the worldwide community of LinkedIn. An invitation containing a brief description of the study's objectives and a link to a Google Forms questionnaire was sent to each company in the database.

To enhance response rate, several project managers identified in LinkedIn were also contacted. Respondents could suggest other participants when filling out the questionnaire, in which case a new invitation to complete the questionnaire was then sent (following a snowball approach). As an incentive for participating in the survey, the participants were informed that a kilogram of rice would be donated to a social institution that fights starvation and poverty for each questionnaire correctly and non-anonymously completed. As a result, more than 100 kilograms of rice were donated. This proved to be both a good incentive for participation in the survey and a valid social contribution. Thus, we strongly recommend it when conducting questionnaire-based studies.

The study generated a total of 202 completed questionnaires, of which nine were unusable because of incomplete responses, leaving 193 complete questionnaires for analysis. Most respondents are male (78.24 percent) and over 35 years old (67.88 percent). The majority have five or more years of experience (70.99 percent. Also, 93.26 percent of the respondents hold graduate or postgraduate degrees, and almost half of them (48.71 percent) have training or certification in project management.

The respondents' companies (vendor side) are of varying sizes, and the sample is evenly split in several of the contextual variables (number of employees, income, international presence, and certifications), thus rendering the analysis more reliable. Almost half of the companies have a project management office or similar organizational unit (49.74 percent), and most companies have an international presence (74.61 percent). Overall, the respondents are experienced project managers, representing a wide spectrum of companies and projects.

Results and Discussion

Project managers were asked to characterize the last completed project carried out for an external client in which they had participated, selected from the set of projects ending at least six months prior (to enable post-project evaluation). The project managers reported being involved in projects of varying types, costs, and duration. Approximately 65 percent of the projects are related to software development, 17.62 percent to packaged software/COTS implementations, and the remaining to other types of IT projects. The majority include both products and services as deliverables (62.69 percent) and involve organizational information systems implementation services (59.59 percent). Most of the projects (about 70 percent) lasted up to 12 months.

The reported projects represent those with budgets of less than EUR 50,000 to those with budgets of more than EUR 500,000. Regarding team size, a wide variety was also observed. Many projects had adopted a PMBOK (Project Management Body of Knowledge) or PMBOK-based approach (25.91 percent), or an agile (Scrum or Scrum-based) approach (36.27 percent). In most of the projects, the respondents have a project manager role (81.87 percent). The client companies for the reported projects are from several industries, services, and sectors. There is, however, a prevalence of large-sized companies (considering the number of employees and annual income), since 58.55 percent have 250 or more employees, and more than 42 percent have a yearly income of EUR 10 million or more.

Levels of success

Figure 1 helps to answer RQ1—Do IT projects still tend to be problematic and unsuccessful endeavors, or can the reported success be biased by the criteria used in the evaluation?. The results indicate that IT projects are achieving high levels of success, with most reaching the top levels (90.16 percent are above the mid-level of the scale, with level 7 corresponding to complete success), and 61.66 percent achieving the top two levels of global success. In projects in the middle point (level 4) or below, the percentage drops to 9.84. Note that levels 1 to 4 (color coded as "lower than 5 or not disclosed" in the figure) are grouped to improve the graphic readability. Taken together, these levels represent a less satisfactory outcome of the project.

Assessing IT Project Success: Perception vs. Reality

Correlations were computed using Spearman's rho statistics to investigate whether there was a statistically significant association between global success and scope, cost, time, vendor satisfaction, deliverables impact, client benefits, and client satisfaction. As expected, positive correlations were found between all the variables (p<.01), since they are directly related to project success.^15,23 Regarding the specific case of the correlations between global success and vendor satisfaction (ρ=.653, p<.01) and client satisfaction (ρ=.714, p<.01), the effects are large (>.50).⁷ Global success also correlates with all the variables related to project execution (time [ρ=.537, p<.01], scope [ρ=.509, p<.01], and cost [ρ=.355]), as well as with the variables related to outcomes and impacts (e.g., deliverables impact [ρ=.490, p<.01] and client benefits [ρ=.751, p<.01]). Note that the larger effects on global success are from client satisfaction (ρ=.714) and client benefits (ρ=.751). The lower effect (a medium effect⁷) is from cost (ρ=.355, p<.01).

These results contradict the general idea about the failure of IT projects, which can be explained by the criteria used for evaluating and reporting success. For example, the classic definition of success by the well-known Standish Group's Chaos Report is: "The project is completed on time and on budget, with all features and functions as initially specified."¹⁶ More recently, project success was redefined by the Standish Group to completion "on time, on budget, with a satisfactory result,"¹⁷, and if a project fails to meet one criterion, it is then considered "challenged." Authors such as Bilir and Yafez⁴ and Varajão et al.²⁵ follow a similar definition. This study does not follow this concept of success, which may (at least partially) explain the differences found in other studies.

Going further

The analyzed projects show high levels of success. The data is self-reported by project managers, which (as in other studies) may result in the risk of bias caused by the selected project characteristics or the respondent's own perspective. Although this limitation is often found in similar studies, it may undermine confidence in results when there is no complementary evidence.

To reduce this risk, and answer RQ2—What objective measures (beyond the traditional scope, time, or cost compliance, or even stakeholder satisfaction) can be used to support project success evaluation and reporting accuracy?—several questions regarding complementary criteria were included in the survey's questionnaire (based on Pereira et al.¹⁵) to mitigate potential bias in results. The questions were defined taking into account that the use of deliverables in the post-project stage, future business, long-term benefits, and long-term relationships and partnerships with stakeholders are strong measures of success.^9,15

The additional questions focused on 1) the use of the deliverables by the client in the post-project stage, 2) the hiring of project-related support or maintenance services by the client, 3) the contracting of new projects by the client, and 4) the recommendation of the vendor by the client (to other potential clients). The answers to these questions provide additional and objective support for improving the accuracy of the reported success. In other words, positive answers to these questions substantiate and complement the reported levels of success.

Regarding the deliverables resulting from the projects, the clients are using them in 81.87 percent of the cases. In three cases (1.55 percent), the deliverables are no longer in use because they have reached the end of their lifecycle. In the remaining cases (16.58 percent), the respondents indicated that the question did not apply to their projects (7.25 percent) or were unaware if the client was still using the deliverables (9.33 percent). In 89.36 percent of cases in which the use of deliverables was currently known (corresponding to 73.06 percent of total cases), the vendors expect that their clients would continue to use them in the future.

In 68.39 percent of the cases, the client hired project-related support or maintenance services. There was no hiring of such services in 11.92 percent of the cases. In the remaining cases, it either did not apply (16.06 percent) or was unknown to the respondents (3.63 percent). The most common duration of maintenance contracts is 12 months (47.90 percent). Contracts of six months or less account for 18.49 percent, while those of more than 12 months account for 33.61 percent. For support services contracts, the duration is most frequently 12 months (50 percent). Contracts of six months or less account for 13.21 percent, and those for more than 12 months account for 36.79 percent.

The contracting of new projects by the client occurred in 58.03 percent of the cases (note that this was the percentage obtained at the time data was gathered); 17.10 percent of participants answered negatively, and 24.87 percent stated that they did not know.

To verify whether there is bias in the selection of projects to report, the participants were asked if, when the project was executed, their company (vendor) was carrying out other projects for the same client. The answer was positive in 59.07 percent of the cases, negative in 34.72 percent, and "did not know" or "not applicable" in the remaining cases. Those who answered positively were then asked about the success of other projects in comparison with the selected project. The vast majority (about 86 percent) indicated that the reported project success was similar to that registered in the other ongoing projects. In about 12 percent of the cases, the success of the other projects was lower, and for the remaining cases, the participants did not know. Thus, it can be considered that no bias was identified in the selection of projects.

In 37.82 percent of the cases, the respondents were aware that the client recommended their company to a potential client, which is also an indicator of success; 59.07 percent responded that they did not know, and 3.11 percent answered that they believed that the client did not recommend them.

In summary, combining the results of the questions about whether the client hired project-related support or maintenance services or contracted new projects (after the project conclusion), 86.01 percent did so. This is a significant indicator that supports the reliability of the high success rates of reported IT projects and should be considered in further studies focusing on project success.

Conclusion

These results challenge the general idea that IT projects are "problematic endeavors" and counter the assumptions that most IT projects fail. On the contrary, this study shows that IT projects are achieving high levels of success. Furthermore, the results are supported by criteria related both to project execution and post-project outcomes and impacts, as well as criteria that provide objective support and evidence of success. The reported high levels of success make sense: We would not be in the digital age if it were not for the recurrent success of IT projects.

This study has significant implications for practice, research, and education by providing new insights into IT project success. It expands the body of knowledge on project management by reporting project success (and not exclusively project management success), grounded in several objective criteria such as deliverables usage by the client in the post-project stage, hiring of project-related support/maintenance services by the client, contracting of new projects by the client, and vendor recommendation by the client to potential clients.

Researchers can find a set of criteria they can use when studying and reporting the success of IT projects, thus expanding the current perspective on evaluation and contributing to more accurate conclusions. For practitioners, this study provides a rich set of criteria that can be used for evaluating their projects, as well as strong evidence of the importance of considering not only project execution, but also post-project outcomes and impacts in the evaluation.

This study represents an advance on earlier work, but it has some limitations. It is based on self-reported evidence from recent experiences of project managers. This means that each project included in this study relies on the memory and opinion of just one project manager responsible for the project. Bias was minimized, however, by asking about the selection of the last completed project and requesting project managers to compare the results with other projects conducted by their company, and by considering several additional objective criteria in the study (e.g., hiring of project-related support or maintenance services).

We encourage future research to study and report the evolution of the success of IT projects by using the criteria proposed here. Also important is examining the results of IT projects by investigating in depth the contributors to vendor and client satisfaction. Considering the common limitation of studies focused on the success of IT projects—the self-reported evidence by project managers—we encourage new studies to involve other stakeholders as participants, such as the users of the project deliverables, the sponsors, or the actual clients. Finally, we recommend studying the influencers of success and success achievement of particular project types (e.g., AI, DevOps, low-code development, etc.) and in particular project contexts (e.g., private and public sectors).

References

1. APDC Digital Business Community. 2018. Global ICT Directory; https://www.apdc.pt/iniciativas/diretorio-tic/.

2. Baccarini, D. 1999. The logical framework method for defining project success. Project Management Journal 30 (4), 25?32; https://www.pmi.org/learning/library/project-success-logical-framework-5309.

3. Barros, L., Tam, C., Varajão, J. 2024. Agile software development projects?unveiling the human-related critical success factors. Information and Software Technology 170 (C); https://dl.acm.org/doi/abs/10.1016/j.infsof.2024.107432.

4. Bilir, C., Yafez, E. 2021. Project success/failure rates in Turkey. International Journal of Information Systems and Project Management 9 (4), 24?40; https://aisel.aisnet.org/ijispm/vol9/iss4/3/.

5. Cadle, J., Yeates, D. 2008. Project Management for Information Systems, fifth edition. Pearson Education.

6. Cecez-Kecmanovic, D., Kautz, K., Abrahall, R. 2014. Reframing success and failure on information systems: a performative perspective. MIS Quarterly 38 (2), 561?588; https://dl.acm.org/doi/10.25300/MISQ/2014/38.2.11.

7. Cohen, J. 1988. Statistical Power Analysis for the Behavioral Sciences, second ed. Lawrence Erlbaum Associates.

8. Colomo-Palacios, R., Gonz?lez-Carrasco, I., López-Cuadrado, J.L., Trigo, A., Varajão, J. 2014. I-Competere: using applied intelligence in search of competency gaps in software project managers. Information Systems Frontiers 16 (4), 607-625; https://dl.acm.org/doi/10.1007/s10796-012-9369-6.

9. DeLone, W.H., McLean, E.R. 2003. The DeLone and McLean Model of information systems success: a ten-year update. Journal of Management Information Systems 19 (4), 9?30; https://dl.acm.org/doi/10.5555/1289765.1289767.

10. Frączkowski, K., Gładysz, B., Kuchta, D., Stanek, S. 2018. Selection of IT projects to be implemented in an organisation to maximise their success probability. Journal of Decision Systems 27 (sup1), 111-122; https://www.tandfonline.com/doi/abs/10.1080/12460125.2018.1464310.

11. Glass, R. 2006. The Standish report: does it really describe a software crisis? Communications of the ACM 49 (8), 15?16; https://cacm.acm.org/practice/the-standish-report/.

12. Jugdev, K., Muller, R. 2005. A retrospective look at our evolving understanding of project success. Project Management Journal 36 (4), 19?31; https://www.pmi.org/learning/library/look-evolving-understanding-project-success-5526.

13. Marnewick, C. 2012. A longitudinal analysis of ICT project success. Proceedings of the South African Institute for Computer Scientists and Information Technologists Conference, 326?334; https://dl.acm.org/doi/10.1145/2389836.2389875.

14. Nelson, R. 2005. Project retrospectives: evaluating project success, failure, and everything in between. MIS Quarterly Executive 4 (3), 361?372; https://aisel.aisnet.org/misqe/vol4/iss3/5/.

15. Pereira, J., Varajão, J., Takagi, N. 2022. Evaluation of information systems project success — insights from practitioners. Information Systems Management 39 (2), 138-155; https://www.tandfonline.com/doi/full/10.1080/10580530.2021.1887982.

16. Standish Group. 1994. Chaos Report 1994. The Standish Group; standishgroup.com.

17. Standish Group. 2015. Chaos Report 2015. The Standish Group; standishgroup.com.

18. Standish Group. 2020. Chaos 2020: beyond infinity overview. The Standish Group; standishgroup.com.

19. Thomas, G., Fern?ndez, W. 2008. Success in IT projects: a matter of definition? International Journal of Project Management 26 (7), 733?742; https://www.sciencedirect.com/science/article/abs/pii/S0263786308000835.

20. Varajão, J., Carvalho, J.A. 2018. Evaluating the success of IS/IT projects: how are companies doing it? International Research Workshop on IT Project Management; https://aisel.aisnet.org/cgi/viewcontent.cgi?article=1003&context=irwitpm2018.

21. Varajão, J., Fernandes, G., Amaral, A. 2023. Linking information systems team resilience to project management success. Project Leadership and Society 4; https://www.sciencedirect.com/science/article/pii/S2666721523000157.

22. Varajão, J., Louren?o, J.C., Gomes, J. 2022. Models and methods for information systems project success evaluation — a review and directions for research. Heliyon 8 (12), e11977; https://www.cell.com/heliyon/fulltext/S2405-8440(22)03265-0?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS2405844022032650%3Fshowall%3Dtrue.

23. Varajão, J., Magalhães, L., Freitas, L., Rocha, P. 2022. Success management — from theory to practice. International Journal of Project Management 40 (5), 481?498; https://www.sciencedirect.com/science/article/pii/S0263786322000473.

24. Varajão, J., Trigo, A. 2016. Evaluation of IS project success in InfSysMakers: an exploratory case study. 37th International Conference on Information Systems; https://repositorium.sdum.uminho.pt/bitstream/1822/43857/1/ICIS-final.pdf.

25. Varajão, J., Trigo, A., Moura, I., Pereira, J.L. 2020. IS project management success in developing countries. Lecture Notes in Business Information Processing 402, 769?780; https://link.springer.com/chapter/10.1007/978-3-030-63396-7_52.

João Varajão is a professor and researcher of information systems and project management at the University of Minho/ALGORITMI Research Centre/LASI. He has published numerous refereed publications, and written and edited books, book chapters, and conference communications. He serves as editor-in-chief, associate editor, and scientific committee member for conferences and international journals.

António Trigo is a professor of management information systems at the ISCTE ? University Institute of Lisbon and a researcher at the ALGORITMI Research Centre/LASI/University of Minho. He has published refereed publications and written books and conference communications. Before joining academia, he worked as a software engineer and project manager.

Originally published in Queue vol. 22, no. 4—
Comment on this article in the ACM Digital Library