Usability, Technicality, Interoperability (UTI), a Comprehensive Assessment Framework for Digital Platforms

Cross-organizational digital platforms are being increasingly used in various domains to connect multiple systems and stakeholders for data sharing and joint operations. However, state-of-the-art platform assessment frameworks have gaps in terms of completeness and depth of analysis. We propose the Usability, Technicality, Interoperability (UTI) framework, which integrates such dimensions into a single, comprehensive model. Each dimension is assessed through a bouquet of Key Performance Indicators (KPIs), combining qualitative and quantitative metrics (13 in total). Comparison with expert-defined numerical thresholds allows identifying strengths and weaknesses of the analyzed platform, which is key for choice, utilization and maintenance. To support traceability and reuse of results, UTI implements a lightweight RESTful system architecture able to receive KPIs from heterogeneous sources, store them in a non-relational database and answer queries. We applied UTI to 4 leading-edge, open-source digital platforms (Measurify, Arduino Cloud, ThingsBoard, NextCloud). The assessment (with 30 end-users) involved usability testing, user questionnaires, automated tools, and developer feedback. Results show that UTI is able to spot capabilities and limitations of a wide range of digital platforms, both within and beyond the IoT context, demonstrating its adaptability across domains and use cases. Compared to 2 state-of-the-art frameworks, UTI is able to achieve a significant advancement, particularly across the usability and technicality dimensions, which it covers with more numerous and focused KPIs. Overall, the experimental results confirm UTI's effectiveness in enabling holistic assessment, supporting informed adoption and development decisions, and guiding stakeholders in ensuring reliability, functionality, and continuous improvement.