Intelligent Design and Validation of an Administrative Automation Evaluation Model in Government Organizations: An Exploratory Mixed-Methods Study

Introduction

Administrative automation has moved beyond the digitization of routine paperwork. In contemporary public administration, automation systems increasingly combine workflow management, data integration, algorithmic decision support, predictive analytics, and artificial intelligence (AI). These systems can support faster service delivery, better information quality, more consistent decision-making, and improved organizational coordination, but they also introduce questions about accountability, transparency, data protection, human capability, and value creation for citizens and public organizations.

AI systems combine learning, inference, and decision-support techniques that can be used for classification, prediction, planning, and knowledge-based assistance (Russell & Norvig, 2021). In organizational settings, AI can support process automation, data-based insight generation, and engagement with employees and service users (Davenport & Ronanki, 2018). Public-sector AI applications can improve service delivery and operational efficiency, but they also create governance, legal, ethical, and organizational challenges (Dwivedi et al., 2021; Tveita & Hustad, 2025; Wirtz et al., 2019). The United Nations E-Government Survey emphasizes that digital transformation and AI need to be accompanied by capacity building, inclusive access, and risk-aware governance (United Nations Department of Economic and Social Affairs, 2024).

A central problem is that many organizations still evaluate administrative automation through narrow technical or financial indicators, such as uptime, transaction speed, or cost reduction. Such indicators are necessary but insufficient. Effective evaluation also requires attention to human experience, user acceptance, information quality, security, business continuity, organizational learning, sustainability, and public value. Human-AI collaboration is particularly important because AI systems do not simply replace human judgment; they can complement human work when organizational decision-making involves uncertainty, complexity, and ambiguity (Jarrahi, 2018).

The present study addresses this gap by developing and validating a comprehensive evaluation model for administrative automation in government organizations. The model integrates qualitative expert knowledge, Delphi consensus, PLS-SEM-based structural validation, and ANFIS-based predictive analysis. Its main contribution is the translation of administrative automation from a technical implementation problem into a multidimensional evaluation framework that can support diagnosis, monitoring, benchmarking, and managerial decision-making.

Methods

Design

The study used an exploratory mixed-methods design. The qualitative phase was used to construct the model, and the quantitative phase was used to validate and test it. This design is appropriate when the target construct is complex, context-sensitive, and insufficiently captured by existing instruments (Creswell & Plano Clark, 2017).

Qualitative Phase

The qualitative phase combined document review and expert input. The expert panel included academic and executive specialists in public administration, information technology, administrative transformation, process management, and human resources. Twelve experts participated in the initial qualitative stage. Data were analyzed through open, axial, and selective coding. Open coding generated initial indicators; axial coding grouped semantically related indicators into components; and selective coding organized the final conceptual structure around the central theme of administrative automation evaluation.

Delphi Procedure

A two-round Delphi procedure was used to refine the indicators and obtain expert consensus. The Delphi panel consisted of 20 experts. In the first round, experts rated the importance and clarity of the 100 indicators. Indicators were retained when they reached a mean importance score of at least 3.5 on a five-point scale and a standard deviation no greater than 1.00. Indicators with lower mean scores or higher dispersion were revised and returned to the experts in the second round. The second round included statistical feedback from round 1, refined wording, and clearer operational definitions.

Quantitative Phase

The quantitative phase used a 41-item questionnaire derived from the qualitative model. The target population included managers and experts in government organizations who had direct experience with administrative automation systems. The final sample consisted of 127 respondents. Items were scored on a five-point Likert scale. PLS-SEM was used because the model was prediction-oriented, involved multiple latent constructs, and was developed from an exploratory model-building process. Reliability, convergent validity, discriminant validity, collinearity, path significance, and overall model fit were evaluated according to standard PLS-SEM reporting principles (Hair et al., 2022; Henseler et al., 2015). Bootstrapping was used to test the statistical significance of structural paths.

ANFIS Predictive Modeling

After the PLS-SEM analysis, the ten construct scores were used as input variables in an ANFIS model. ANFIS combines fuzzy inference with adaptive learning and is useful when relationships are nonlinear, interactive, and difficult to represent with simple linear equations (Jang, 1993). The target variable was the composite score of administrative automation evaluation. The data were divided into training, validation, and test subsets. A Sugeno-type fuzzy inference system was generated by subtractive clustering and trained with a hybrid learning algorithm combining least squares and gradient descent.

Ethical Considerations

The study involved expert interviews and questionnaire responses from adult public-sector personnel. Participation was voluntary, informed consent was obtained before data collection, and responses were anonymized before analysis. Ethics approval was not required for this anonymous, non-interventional study involving adult professionals and no collection of sensitive personal data. No identifiable personal information is reported in the manuscript.

Results

Qualitative Extraction of Indicators, Components, and Dimensions

The qualitative phase produced a four-layer evaluation architecture. At the lowest level, 100 operational indicators were extracted from expert interviews, document review, and iterative coding. These indicators covered process time, error reduction, user acceptance, data quality, auditability, security, business continuity, AI explainability, cost saving, sustainability, innovation, and public value. Through axial coding, the indicators were consolidated into 30 components; through selective coding, the components were organized into 10 higher-order dimensions around the core construct of administrative automation evaluation. The resulting structure is summarized in Table 1.

Note. Indicator codes are condensed from the 100-item indicator bank extracted in the qualitative phase. The full operational wording was used to develop the final questionnaire.

Delphi Panel and Consensus Results

The Delphi process was used to test whether the proposed indicator system was acceptable to experts. The panel consisted of 20 experts: 12 men (60%) and 8 women (40%). Six experts held a master's degree (30%) and 14 held a doctoral degree (70%). Seven experts had 5-10 years of experience in administrative automation (35%), while 13 had more than 10 years of relevant experience (65%). In the first round, most indicators received mean importance scores above 4.00. Indicators with lower mean scores or higher dispersion, including document digitization, ADM decision accuracy, data quality, CO2-saving indicators, cultural acceptance, and innovation-budget share, were revised and returned for the second round. After statistical feedback and refined operational definitions, all indicators reached the final acceptance threshold. Dimension-level consensus is reported in Table 2.

Note. The Delphi panel included 20 experts. All dimensions exceeded the retention threshold after the second round.

Quantitative Sample and Descriptive Statistics

The quantitative phase included 127 public-sector respondents who had direct experience with administrative automation systems. The sample was concentrated in the 35-44 age range, and most respondents had substantial work experience. Automation-system exposure was high: 58.27% reported daily use and 22.83% reported use several times per week. In addition, 79.53% had received automation-related training, 70.87% were permanent or formal employees, 65.35% were men, and 34.65% were women. These characteristics support the relevance of the respondents for evaluating automation systems from an informed organizational perspective.

The construct means were located near the midpoint of the five-point response scale, indicating moderate perceived maturity rather than uniformly strong performance. As shown in Table 3, innovation and continuous improvement had the highest mean score, whereas process performance and efficiency and sustainability/environmental impacts had the lowest means.

Note. Scale range: 1-5. K-S tests did not reject normality for the ten constructs; Lilliefors and Jarque-Bera tests flagged mild departures for selected constructs. Because PLS-SEM is distribution-flexible, bootstrapping was used for inference.

Measurement and Structural Model Results

The measurement model showed uneven but informative psychometric performance. User experience and human resources, governance/transparency/compliance, process performance and efficiency, strategic alignment and public value, and data integration and management showed acceptable reliability and convergent validity. Innovation and continuous improvement, sustainability/environmental impacts, financial outcomes, and risk/security/business continuity showed weaker AVE or Cronbach's alpha values. Discriminant-validity diagnostics suggested partial separation among constructs. The Fornell-Larcker matrix showed overlap among process, technology, data, and governance dimensions, while most HTMT values remained below or near common interpretive thresholds. This pattern supports the model as an integrated diagnostic framework while indicating that future studies need to test second-order or bifactor specifications.

Bootstrapping indicated that all ten paths to the administrative automation evaluation construct were statistically significant. Table 4 reports the combined measurement, structural, and collinearity results. Figure 1 visualizes the standardized path coefficients.

Note. β = standardized path coefficient to administrative automation evaluation. CR = composite reliability; AVE = average variance extracted. Negative coefficients are interpreted cautiously because they may reflect reverse coding, suppression effects, or construct overlap. Model fit was acceptable: SRMR = 0.039 for the saturated model and 0.041 for the estimated model; NFI = 0.91 for the estimated model.

Note. Positive bars indicate positive structural coefficients; lighter negative bars indicate inverse coefficients requiring cautious methodological interpretation.

ANFIS Predictive Modeling

To complement the explanatory PLS-SEM analysis, an ANFIS model was developed to capture nonlinear and interaction-based patterns among the ten constructs. The model used the ten dimension scores as inputs and the composite automation evaluation score as the output. The data were divided into 89 training cases, 19 validation cases, and 19 test cases. A Sugeno-type fuzzy inference system was generated using subtractive clustering with a cluster radius of 0.35. The system contained 89 fuzzy rules and was trained for 80 epochs using a hybrid learning algorithm. Table 5 summarizes the ANFIS configuration, predictive performance, and sensitivity ranking. Figure 2 shows the relative importance of the ten ANFIS inputs.

Note. The near-zero training error indicates very close learning of the training subset; therefore, interpretation emphasizes validation and testing results rather than training fit.

Note. SA = strategic alignment and public value; MS = technology maturity and scalability; UH = user experience and human resources; FO = financial outcomes and cost savings; PP = process performance and efficiency; IN = innovation and continuous improvement; DI = data integration and management; RS = risk, security, and business continuity; SU = sustainability and environmental impacts; GT = governance, transparency, and compliance.

Integrated Interpretation of the Results

Across the qualitative, Delphi, PLS-SEM, and ANFIS stages, the strongest message is that administrative automation in government organizations cannot be evaluated as a narrow technical installation. The qualitative stage established a broad indicator system; the Delphi results confirmed that experts prioritize process performance, user experience, financial outcomes, and governance; the PLS-SEM results showed that all ten dimensions are statistically connected to the evaluation construct; and the ANFIS results showed that prediction is driven mainly by strategic alignment, technology maturity, user experience, financial outcomes, and process performance. Taken together, the model supports a staged evaluation logic: first align automation with public value and organizational strategy; second ensure technological maturity, data integration, and security; third redesign processes and user experience; and fourth use financial, sustainability, and innovation indicators for continuous monitoring.

Discussion

The findings show that evaluating administrative automation in government organizations requires an integrated model that includes technical, process, human, governance, financial, and public-value dimensions. The qualitative findings confirm that automation is not merely a technology deployment issue. It is a system of organizational redesign, data governance, human capability, risk management, and strategic value creation. This interpretation is consistent with public-sector AI research, which emphasizes that AI systems need to be assessed not only by technical functionality but also by their governance implications, organizational fit, and value for public service (Wirtz et al., 2019).

The Delphi findings indicate strong expert agreement on operational and human dimensions. Process performance, user experience, and financial outcomes received high consensus, while sustainability received the lowest retained consensus. This does not mean that sustainability is unimportant; rather, it suggests that in the studied context, respondents and experts may view environmental indicators as less immediate than service speed, error reduction, cost control, and user acceptance. For management, sustainability can therefore be integrated through measurable indicators such as paper reduction, energy efficiency, and CO2-saving estimates.

The PLS-SEM findings support the empirical relevance of the ten-dimensional framework. However, the measurement model also identifies areas that need refinement. Some constructs showed lower AVE or Cronbach's alpha values, suggesting that item wording, item balance, and subcomponent structure require improvement in future versions of the questionnaire. The high correlations between some constructs also indicate that respondents may experience different aspects of automation as interconnected rather than isolated.

The ANFIS findings add a predictive layer to the explanatory model. The strongest predictors were strategic alignment and public value, technology maturity and scalability, user experience and human resources, and financial outcomes. This ranking suggests that automation evaluation begins with the question of whether the technology supports organizational goals and public value. A technically advanced system can still fail if it is not aligned with service priorities, does not scale, is difficult for staff to use, or does not produce measurable financial and operational value.

The negative PLS-SEM coefficients for governance/transparency/compliance, process performance and efficiency, and sustainability are not interpreted as evidence that these dimensions harm automation. A more defensible interpretation is methodological and contextual. Some indicators may have been reverse-coded or framed in terms of risk and burden rather than achievement; high construct overlap can generate suppression effects; and in immature automation environments, stronger governance or performance monitoring may initially expose weaknesses, delays, or compliance costs. Future studies need to verify item direction and test alternative model specifications before drawing causal conclusions.

The findings also align with recent work on AI and administrative support occupations, which stresses that automation affects not only technical tasks but also roles, skills, human factors, and organizational adaptation (Pennathur et al., 2024).

Practical Implications

The model can be used as a diagnostic and monitoring tool in government organizations. At the strategic level, managers can connect automation projects to public value indicators such as service quality, procedural fairness, response time, citizen satisfaction, and transparency. At the technical-process level, organizations need to improve system maturity, API integration, data quality, scalability, and process redesign before simply digitizing existing workflows. At the human level, user-centered design, role-based training, change management, and technical support are necessary to reduce resistance and prevent user error. At the governance level, data-protection controls, audit trails, access management, risk assessment, and periodic compliance review need to be embedded into the automation lifecycle.

Limitations and Future Research

This study has several limitations. First, the quantitative data were based on self-report questionnaires, which can be affected by common-method bias, social desirability, and differences in respondents' digital literacy. Second, the model was validated in a specific public-sector context; therefore, generalization to other government organizations, municipalities, ministries, or countries requires caution. Third, some measurement-model indicators require refinement because several constructs showed lower internal consistency or convergent validity. Fourth, the ANFIS model showed strong training performance but moderate test performance, indicating that future studies need to compare ANFIS with other machine-learning algorithms and use larger datasets, operational logs, and longitudinal indicators.

Future research needs to test the model in multiple organizations and compare results across organizational size, digital maturity, service domain, and governance structure. Longitudinal studies are also needed to examine how automation maturity, user trust, digital culture, and public value evolve after implementation. Future work can add mediating and moderating variables such as digital culture, organizational trust, leadership support, and data-governance maturity.

Conclusion

The study developed and validated a comprehensive model for evaluating administrative automation in government organizations. The final model includes 100 indicators, 30 components, and 10 dimensions. Delphi results confirmed expert consensus for all dimensions, PLS-SEM supported the empirical relevance of the model, and ANFIS demonstrated moderate-to-good predictive capability. The central conclusion is that administrative automation succeeds when technology maturity is combined with strategic alignment, public value, process redesign, user experience, financial outcomes, and governance safeguards. The model therefore serves as a practical decision-support framework for continuous evaluation, benchmarking, and improvement of administrative automation in public organizations.

Ethics Approval and Consent to Participate

Ethics approval was not required for this anonymous, non-interventional study involving adult public-sector personnel and no collection of sensitive personal data. Participation was voluntary, informed consent was obtained from all participants before data collection, and all responses were anonymized and used only for research purposes.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Conflict of Interest

The authors declare that they have no known competing financial or non-financial interests related to this study.

Data Availability

The data that support the findings of this study may be made available from the corresponding author upon reasonable request, subject to institutional permission and confidentiality restrictions.

Author Contributions

Fatemeh Motaghi: conceptualization, data collection, formal analysis, and writing - original draft. Behzad Souki: supervision, methodology, validation, and writing - review and editing. Zahra Moghimi: validation, interpretation, and writing - review and editing.