Εφαρμογή επεξεργασίας, ανάλυσης και πρόβλεψης βασισμένη σε μεγάλα δεδομένα έξυπνων πόλεων
| datacite.contributor.RelatedPerson | Λογοθέτης, Μιχαήλ | |
| datacite.contributor.RelatedPerson | Τόμκος, Ιωάννης | |
| datacite.contributor.Supervisor | Δενάζης, Σπυρίδων | |
| dc.contributor.author | Σταθούλιας, Κωνσταντίνος | |
| dc.contributor.other | Stathoulias, Konstantinos | |
| dc.date.accessioned | 2026-07-09T11:11:52Z | |
| dc.date.available | 2026-07-09T11:11:52Z | |
| dc.date.issued | 2026-07-08 | |
| dc.degree | graduateThesis | |
| dc.description.abstract | Η ανάγκη για αντίληψη του αστικού περιβάλλοντος των έξυπνων πόλεων έχει οδηγήσει στην ανάπτυξη δικτύων αισθητήρων και μηχανισμών συλλογής δεδομένων πραγματικού χρόνου. Η κατάλληλη διαχείριση αυτών των δεδομένων και η εξαγωγή αξιόπιστων προβλέψεων συμβάλλουν στην καλύτερη λήψη αποφάσεων και στη βελτίωση της βιωσιμότητας ενός συνόλου έξυπνων πόλεων. Τα δεδομένα που συλλέχθησαν προέρχονται από το χώροδεδομένων του έργου «Smart Interconnected Cities for Greece 2.0», και συγκεκριμένα από έξι API τερματικά. Στα τερματικά αυτά, από τη στιγμή που έχει πραγματοποιηθεί ενημέρωση για τις διαθέσιμες υπηρεσίες από ένα Metadata Broker, αποστέλλονταν HTTP αιτήματα και τα δεδομένα λαμβάνονταν ως αποκρίσεις που ακολουθούν το πρότυπο NGSI-LD. Έπειτα, τα δεδομένα που συλλέχθησαν, αφού έχουν αποθηκευτεί σε μια κεντρική αποθήκη δεδομένων, υφίστανται κάποιες διαδικασίες επεξεργασίας, ώστε οι χρονοσειρές να έχουν κατάλληλη μορφή να εισαχθούν στα μοντέλα πρόβλεψης. Πέρα από την διαχείριση επεξεργασμένων συνόλων δεδομένων που βασίζονται σε παρατηρήσεις πραγματικού χρόνου, μέσα από την εργασία επιτυγχάνονται η αξιολόγηση δημοφιλών μοντέλων πρόβλεψης και ο ασφαλής διαμοιρασμός δεδομένων σε εξωτερικούς χρήστες. Συγκεκριμένα, τα μοντέλα, βάσει των πληροφοριών που έχουν συλλεχθεί και με χρήση τεχνικών Μηχανικής Χαρακτηριστικών, εκπαιδεύονται ανά χρονοσειρά. Όταν τα μοντέλα πρόβλεψης αποτυπώσουν την εποχικότητα και την τάση της εκάστοτε χρονοσειράς, εξάγονται οι αναπαραστάσεις επεξεργασμένων και προβλεπόμενων τιμών για την αξιολόγησή τους και δημοσιεύονται τα επεξεργασμένα δεδομένα. Η αξιολόγηση βασίζεται στον υπολογισμό της μετρικής του συμμετρικού μέσου απόλυτου ποσοστιαίου σφάλματος sMAPE, κατάλληλη για χρονοσειρές που εκφράζουν διαφορετικά μεγέθη. Τα μοντέλα που τίθενται προς αξιολόγηση είναι τα στατιστικά μοντέλα SARIMAX, Holt-Winters’ και Meta Prophet, καθώς και τα μοντέλα μηχανικής μάθησης XGBoost, GRU και Google TimesFM. Από τα αποτελέσματα έχει προκύψει ότι τα μοντέλα μηχανικής μάθησης είναι πιο αποδοτικά από τα στατιστικά μοντέλα. | |
| dc.description.translatedabstract | Sensors’ installation in urban areas has led to the generation of large volumes of data that describe in real time different aspects of how smart cities function. Thanks to the collection of these data, valuable insights can be derived based on the available information anddecision-making processes can be supported based on forecasting processes, whose inputs consist of curated IoT datasets, represented as timeseries. The inserted time series are assumed to consist of three main components: seasonality, trend, and residuals.Data are fetched from API Hubs that belong to the dataspace of the “Smart Interconnected Cities for Greece 2.0” project, which aims to ensure interoperability, scalability, and the efficient utilization of Greek smart cities data. This project is based on Data Spaces philosophy and leverages FIWARE technologies. The ecosystem contains multiple API Hubs, eachcorresponding to a participating city or institution, and a central Metadata Broker, connected to these hubs. Six API endpoints were used in the raw data fetching process. These six hubs contain data originated from the cities of Patras, Athens, Piraeus, Larissa, Trikala, Kozani and Kastoria with the latter two served through the UoWM API Hub.Firstly, API Hubs publish their metadata to the central Metadata Broker. The broker can create a catalog containing information about the hubs’ availability and geographical location about them. Afterwards, Adaptors are used so they perform service discovery processes bycommunicating with a Metadata Broker. Through the obtained information, the adaptorsdiscover which of the targeted API hubs are available and obtain the corresponding metadata that are required to identify the appropriate data sources. Once the metadata have been retrieved by the adaptors, HTTP requests are sent to the available API endpoints, providing information of the entity type whose past observations are going to be fetched and the time range these were tracked by sensors. Each entity type is associated with the Smart Data Model that defines the semantic context of the observations. The entities are structured based on the NGSI-LD model and contain information about the sensor identifier, the values of the monitored variable and the corresponding timestamps of these observations. These observations are stored in a centralized Data Warehouse, which is represented by a MongoDB database, and are subsequently organized into timeseries, which is going to be utilized for curation and forecasting processes. For the API Hubs of Larissa, Piraeus, and Kozani, multiple HTTP requests are required to retrieve past observations spanning the desired time range, due to restrictions applied on the maximum time range supported by a single API request. This repeated request mechanism is referred to as the polling process.The next stage of the proposed methodology involves the curation of the collected data.Timeseries that exhibit nearly constant values over extended time periods or are described by insufficient density of observations are discarded, as no information about seasonality and trend can be provided for forecasting models. Furthermore, such datasets may indicate sensor malfunctions or communication issues. The remaining timeseries are, later, getting resampled, to a fixed sampling interval, so they can be described by a uniform temporal resolution required for reliable and efficient time series forecasting. This is followed by a missing value handling process, during which timestamps without observations are imputed based on linearinterpolation for non-edge missing values, while forward and backward filling methods are utilized to missing values at the beginning and end of each time series, respectively.As far as anomalies are concerned, STL decomposition takes place and separates each timeseries into its seasonality, trend and residual components. Based on this decomposition, point outliers, level shifts in trend component and abrupt changes in the variance of the residual component can be detected, by distinguishing irregular observations. The data rectification process preserves the initial format of timeseries while replacing anomalies with values reconstructed from the corresponding trend and seasonal components. The entire curation stage contributes to data quality improvement, enabling publication and reuse of curated datasets through platforms such as Kaggle. The curated values get stored in the Data Warehouse.The workflow then proceeds with the operation of forecasting models’ development andtraining. In particular, curated datasets are repeatedly partitioned into multiple rolling train-test windows, where some preceding observations are used for training and some subsequentobservations for testing, allowing the models to be developed and evaluated, respectively. This procedure is referred to as the train/test splitting process. Each train–test split derives a training set of different length. Apart from the sliding window technique, lagged and temporal features are also used for model training. Lags are described as historical observations of the timeseries that enable forecasting models to capture temporal dependencies. In addition, calendar-based features, which may be expressed such as the hour of the day and the day of the week, are incorporated to provide temporal context. It should be noted that model development is performed independently for each time series. As a result, the total number of training procedures equals the number of forecasting models multiplied by the number of available time series.Once the timeseries have been curated, six forecasting models are trained. Specifically, three statistical models and three machine learning models are examined. Statistical models’ results are based on mathematical calculations containing the underlying timeseries components, whereas machine learning ones learn forecasting patterns directly from historical observations for forecast generation. Among the statistical models, SARIMAX combines autoregressive, seasonal and exogenous effects, Holt-Winters’ model is known applying exponentiallydecreasing weights to past observations, emphasizing recent ones and Prophet, which was published by Meta, automatically models timeseries components, using an additive framework.When it comes to machine learning models, XGBoost is a gradient-boosting algorithm that builds ensemble decision trees, GRU, as a type of a Recurrent Neural Network, uses update and reset gates to control information usage and TimesFM, which was published by GoogleResearch, is a pre-trained and decoder-only transformer-based model.The final stage of the methodology focuses on model evaluation and results visualization. In particular, the testing sets that have been produced from the train-test splitting method define the evaluation time period, during which the forecasted values are compared with thecorresponding curated observations. Once the forecasted values are stored in the warehouse, results are visualized using Plotly. Subsequently, comparative analysis of the six forecasting models is based on Symmetric Mean Absolute Percentage Error (sMAPE) calculations, whose results represent the absolute value of deviation between forecasted and actual values relative to the average of actual and forecasted values’ absolute values. This error metric facilitates model evaluation, as 14 different modalities are represented by 74 timeseries in total. The primary total evaluation criterion adopted in the study is the median sMAPE across the 74 timeseries for each forecasting model, since the distribution of forecasting errors contains several extreme values that could bias the mean sMAPE. However, since the expression of trend and seasonality varies across different attributes, an additional comparative analysis of the forecasting models is conducted at the attribute level. Finally, to compare based on all sMAPE results, a statistical analysis consisting of the Friedman test followed by the Nemenyi post-hoc test is performed. Thanks to this analysis, conclusions can be conducted concerning differences between forecasting models and whether these are statistically significant.Based on comparative analysis’ results, machine learning models outperform statistical ones.Specifically, TimesFM achieved the highest forecasting accuracy due to its large-scale pre-training on diverse real-world time series. Moreover, XGBoost is the most stable forecasting model, as it exhibited the lowest deviation between all forecasting errors, so the sMAPE distribution had the lowest interquartile range. The sole exception to the superior forecasting efficiency of machine learning models is the GRU, which in most case exhibits significant smoothing, limiting its ability to capture abrupt temporal variations. Nevertheless, the statistical approaches remain competitive for several attributes, particularly those exhibiting stable trend and seasonal patterns. Especially, SARIMAX could predict the exact moments of occurrence of peaks and valleys of most timeseries. Overall, no single forecasting model consistently outperforms the others across all evaluated attributes, reinforcing the conclusion that the selection of the most suitable forecasting model should depend on the characteristics of the input time series. | |
| dc.identifier.uri | https://hdl.handle.net/10889/32438 | |
| dc.language.iso | el | |
| dc.rights | Attribution-NonCommercial-ShareAlike 3.0 Greece | en |
| dc.rights.uri | http://creativecommons.org/licenses/by-nc-sa/3.0/gr/ | |
| dc.subject | Διασυνδεδεμένες έξυπνες πόλεις | |
| dc.subject | Χώρος δεδομένων | |
| dc.subject | Αποθήκες δεδομένων | |
| dc.subject | Μηχανική χαρακτηριστικών | |
| dc.subject.alternative | Interconnected smart cities | |
| dc.subject.alternative | Dataspace | |
| dc.subject.alternative | Data warehouses | |
| dc.subject.alternative | Feature engineering | |
| dc.subject.alternative | Metadata broker | |
| dc.subject.alternative | FIWARE | |
| dc.subject.alternative | NGSI-LD | |
| dc.subject.alternative | HTTP | |
| dc.subject.alternative | API | |
| dc.subject.alternative | ETL processes | |
| dc.subject.alternative | sMAPE | |
| dc.subject.alternative | Train-test splitting | |
| dc.title | Εφαρμογή επεξεργασίας, ανάλυσης και πρόβλεψης βασισμένη σε μεγάλα δεδομένα έξυπνων πόλεων | |
| dc.title.alternative | Smart cities big data processing, analysis and forecasting application |