2.1 In Mobile and IoT Devices Malware Detection

Malware in mobile and IoT has emerged as a new threat to cybersecurity mainly because these devices are increasingly used in sensitive areas [18, 19, 20, 31]. Subsequently, smartphones, especially those with the Android platform, have become the largest opportunities for attackers, given their ubiquity, availability, and versatility. Traditional malware detection methods, such as signature-based and heuristic approaches, struggle with modern threats like polymorphic and metamorphic malware, which evade detection through obfuscation and dynamic changes. While machine learning models like RNNs, LSTMs, and GRUs improve sequential data analysis [21], they often fail to capture global dependencies and lack explainability, limiting their effectiveness. The number of apps and user actions is continually growing, which, along with other factors, forms an unbounded space for activity, against which it is challenging to defend using traditional approaches based on signatures [22].

Like PCs, IoT devices used today in smart cities for infrastructure monitoring, utility management, and other improvements in people's daily lives are also easily infected by malware. These devices are resource-constrained, often have weak security controls in place, and are members of large complex networks that are challenging to monitor [23]. These devices' flaws can be used to launch Distributed Denial of Service (DDoS) attacks, data theft, or incapacitating essential services such as healthcare and traffic systems [24]. Traditional malware detection techniques, such as signature-based detection, work by recognizing known signatures of malware. Although helpful for inherent attacks, these techniques fail to identify zero-day attacks, polymorphic malware, or a new breed of viruses [25]. For this reason, new techniques to detect these malware classifications that can be applied effectively to future threats are also required. This is true for most ML and DL models that can easily capture data patterns related to malicious activity even if the threats are unknown or in a new variant [26].

2.2 Vision Transformers (ViTs) and Their Application to Malware Detection

Designed to focus on vision, Vision Transformers (or ViTs) are a relatively new addition to deep learners and have demonstrated impressive performance in image classification. The idea that forms the basis for ViTs is to partition an image into smaller segments and process each segment or patch as an individual token. Making tokens of ViTs attend to each other across space also helps ViTs model long-range dependencies in data; it suggests that ViTs are particularly suitable for tasks where the global context is crucial, such as object recognition in images. ViTs have recently been applied to the field of malware detection since they provide the opportunity to capture global dependencies in sequential data, such as opcode sequences originating from Android and IoT malware. Unlike the original Convolutional Neural Networks (CNNs), which extract local spatial features by convolutions, ViTs regard the whole input data as a sequence and the interaction between all patches (or tokens) simultaneously. This allows ViTs to analyze local dependencies in the data and index important patterns of malicious behavior that might go unnoticed at the granular level [27, 28]. This article shows that ViTs have one crucial benefit over other architectures—their ability to capture long-range dependencies. As seen regarding malware analysis, malicious activities usually extend through several instructions, and identifying such activities means recognizing the interactions between far opcodes. Unlike most embeddings that operate on a single opcode, ViTs work on a sequence of all or at least a subset of them, allowing ViTs to detect trop global that characterizes malware behavior [29]. Some recent works have shown that ViTs are more effective than the traditional CNN-based counterparts in certain malware detection [35] tasks because of their better ability to capture these relations [30]. Nevertheless, ViTs have come across some issues when working with sequential data, and more specifically, with opcode sequences. ViTs can capture global dependencies but do not capture temporal dependencies between the opcodes, which this problem heavily relies on due to sequential processing performed by malware. This has been done due to the limitations of the ViTs and the subsequent investigation of more powerful models, such as Recurrent Neural Networks (RNNs), that are effective for sequential data.

2.3 Hybrid Models

Further, the RNN utilizes the ViTs to identify the presence of malware or not. The integration of Vision Transformers (ViTs) and Recurrent Neural Networks (RNNs), namely Long Short-Term Memory (LSTM) networks and Gated Recurrent Units (GRUs), becomes a natural solution to avoid the problems that occur with each model. Although ViTs consider global patterns of the data, RNN memorizes previous inputs' information, which makes them suitable for understanding the sequence of indices in the malware operation.

LSTMs are a subtype of RNN machines that perform exceptionally well with long-term sequence dependencies. Gating mechanisms help to protect from the vanishing gradient problem, which tends to happen if traditional RNNs are used with long sequences by allowing the model to decide which information to remember and which to forget at each step. This makes LSTMs suitable for analyzing dependencies on the timelines of actions in malware samples, given that the temporal order of operations defines malicious behavior. LSTMs are similar to GRUs but allow the data to be updated at each time step, which makes them more computationally extensive but, at the same time, they can get at least as many parameters capturing long-range dependencies in sequential data. The further hybrid models can describe the malware's behavior better, Created with ViTs that deal with global dependencies and LSTMs or GRUs that address sequential ones. It combines this to enhance the effectiveness of the tuple detection model as the model needs to consider the order in which these operations are called when determining the general structure of the malware.

Recent works show that the integration of ViT and RNN has been pioneered in many fields and domains, ranging from time series to natural language processing domains where both spatial and temporal information are essential. In malware detection this approach has proved effective, enabling models to be developed that capture the richness and the multifaceted nature of malware.

2.4 Ensemble Learning for Malware Detection

Ensemble learning is one of the strong concepts in machine learning in which different models are used jointly to enhance the characteristics of the decision. Finally in ensemble methods such as bagging and boosting different individual models are built using different sections of the data or else the same section of data is used by building them using different algorithms and the final decision is constructed from the individual model's decisions. The strength of ensemble methods is that they lessen the problem of overfitting and increase the model generalization since a number of models are used. Smoothing methods like Ada Boosting and Gradient boosting are used where the learning algorithm builds a sequence of models each demonstrating different errors. This they achieve at the expense of a slight slowdown in computational speed and the emergence of higher order polynomials, and a strong learner that can perform well even on complex and noisy data. The rgenetic algorithm [36, 37], involve the construction of a set of models trained on different subsets of the data, and making a decision based on the response of all of the models. Both techniques are more suitable for dealing with imbalanced datasets frequently used in malicious code detection when the number of normal samples is significantly greater than that of malicious samples.

Specifically to the detection of malware, the ensemble learning method proposed in this paper can enhance the performance of each component model which is easily affected by variance and bias when they face complex and unbalanced datasets. As the name suggests ensemble methods can also improve the models' resistance to adversarial attacks or previously unknown strains of malware. In the proposed ViTDroid model, the idea of ensemble learning can be implemented on top of the hybrid ViT-RNN model to enhance the malware detection result's reliability and accuracy.

2.5 Explainability in Malware Detection

Interpretability in the context of a Machine learning model means how easily the decisions made by a model can be explained. In the case of cybersecurity, the crucial steps of identifying that certain software is either malicious or benign are taken by automated systems, thus the respective decisions need to be explainable. This is especially true when deep learning models ViTs and hybrid models are used as most of them are considered "Black Box" models it is difficult to understand their internal mapping mechanisms.

As for this issue, several approaches have been made to enhance the interpretability of deep learning models. Some of them include attention mechanisms, for instance, enabling the models to give priority to portions of the input data, which are most important in making the forecast. In malware detection, attention mechanisms can identify which areas of the opcode sequence are more indicative of malicious activity, thus making the model's decision-making more interpretable.

Furthermore, non-specific interpretation techniques, such as LIME (Local Interpretable Model-Agnostic Explanations) or SHAP (Shapley Additive exPlanations) can generate explanations for the predictions made by a diverse model. LIME relies on the concept of substituting a complex model with a simpler model, to explain the predictions made for an individual instance while SHAP uses concepts of cooperative game theory to explain the level of contribution provided by each feature to the final prediction. These tools enable a user to know why a particular sample has been categorized as either malicious or benign, which is helpful in trust and validation in cyber-security applications [38, 39].

Complementing the ViTDroid model with attention mechanisms and interpretability tools, this research improves the model's trustworthiness and supports the safe utilization of the model's predictions by cybersecurity professionals.

2.6 Data Augmentation and Transfer Learning for Coping with Data Poverty

A major limitation typical of developing machine learning models for malware detection [32, 33, 34] is the availability of labeled data, particularly for new or relatively rare types of malware. This problem can be solved to a certain extent using a data augmentation technique which makes the amount and the variety of data in the training set larger artificially. As mentioned before, common augmentation in image classification includes rotation, scaling, and flipping among them. When dealing with opcode sequences data augmentation can be performed by changing the structure as well as the sequence of opcodes but still pointing toward an evil purpose. This helps the model to generalize more and hence be made more immune to other unseen samples.

Another powerful technique that also corrects the existence of scarce labeled data is known as transfer learning. When deep learning models are trained from scratch, they require more datasets and a longer time to obtain the result; conversely, pre-trained deep learning models also make use of learned features from related tasks or domains. In malware detection, transfer learning [40, 41, 42] has been found effective when the original models from related large datasets from other environments or any general image classification datasets have been used. This enables the model to harness the knowledge that is embedded in the pre-trained model even under conditions where the only available data is labeled.

Figure 1 Proposed architecture.

3.1 Overview of ViTDroid Architecture

ViTDroid is designed to address the limitations of traditional machine learning models in malware detection by leveraging the power of Vision Transformers for capturing global dependencies within opcode sequences. The architecture of ViTDroid combines the strengths of ViTs with Recurrent Neural Networks (RNNs), such as Long Short-Term Memory (LSTM) networks or Gated Recurrent Units (GRUs), to model both the spatial and temporal dynamics of malware execution. ViTDroid combines Vision Transformers (ViTs) and recurrent neural networks (RNNs) to enhance malware detection by leveraging both spatial and temporal dependencies. Malware opcode sequences are tokenized into patches and processed by the Vision Transformer module to capture global dependencies through self-attention mechanisms. The RNN module, using LSTM or GRU, analyzes sequential relationships to identify temporal patterns of malicious activity. Outputs from these modules are integrated using ensemble techniques like boosting and bagging to improve robustness and accuracy. Additionally, explainability tools such as LIME and SHAP enhance transparency by highlighting features that influence predictions, ensuring trust and usability in cybersecurity applications.

At a high level, the ViTDroid architecture consists of several components:

3.2 Input Representation: Opcode Sequences as Patches

To fit the input data into the Vision Transformer, the raw opcode sequences are transformed into patches similar to the image ones. Each opcode is taken as a feature and a window of a fixed size is applied by sliding it over the sequence to produce patches. These patches are then inserted into the fixed length vectors using the embedding layer showing the model what to learn in terms of opcode sequences features.

This conversion allows the Vision Transformer to learn over opcode sequences similar to how it learns over an image where patches are local fragments of the sequence and learn dependencies over the entire sequence by attending to distinct patches simultaneously. It is specifically beneficial for malware detection since the model can gain both the detailed local structures (local opcodes) and the global context (coarse patterns across opcodes) in the malware execution sequence.

3.3 Vision Transformer Module: Global Dependency Capture

The Vision Transformer (ViT) module is the central module of the ViTDroid. In ViTs, the patches focused on capturing global dependencies in the input data which is important in observing the behavior of malware across different instructions or opcodes. It is well known that convolutions are used to learn local features in a standard convolutional model, but this hinders the performance of learning long-range dependencies. To this, end, ViTs do not process the sequence one element at a time; rather, the whole sequence's interaction is captured.

When it comes to ViTDroid, the ViT module learns dependencies across the entire sequence acting on the embedded opcode patches in parallel. This makes it possible for the model to identify other obscure characteristics that reveal the violation of malicious behavior not defined by the local level Opcode. For the same reason, the self-attention mechanism used in ViT models enables an adjustment of weights by which the data will be weighted in a way that is beneficial to focus on important patches while disregarding less relevant ones.

3.4 Hybrid RNN Module: Temporal Dependency Modeling

Although ViTs are excellent in modeling dependencies across the entire globe, malware can have intricate temporal patterns, and any reordering of opcodes that a malware might induce is crucial to identifying its misbehavior. To overcome this, in ViTDroid, there is a combined module, where there is Recurrent Neural Network (RNN) – Long Short-Term Memory (LSTM) or Gated Recurrent Units (GRU). These above-described RNN variants are built to handle sequential data and learn temporal patterns by having a memory of earlier inputs. Malware detection requires both spatial analysis of opcode relationships and temporal analysis of sequential behaviors. Vision Transformers (ViTs) capture global spatial dependencies, identifying subtle malicious patterns, while RNNs like LSTM and GRU excel at modeling temporal relationships critical for detecting chronological malware behaviors. ViTDroid combines these strengths, enabling comprehensive detection of complex malware by addressing both spatial and temporal challenges effectively.

3.5 Ensemble Learning: Improving Accuracy and Stability

In a bid to enhance the distinct and consistent outcomes of ViTDroid, boosting and bagging are combined learning methods that are integrated alongside the tool. Voting operates by taking several models and using them in an ensemble, as the inconsistency and prejudice problems hurt performance. In ViTDroid, such techniques as Gradient Boosting or AdaBoost are used, where in the process of learning several models are trained, and each subsequent one is aimed at improving results on the problems that failed the previous model. This aids in modifying the model to extremely function better in response to sophistical and noisy data inputs.

Further, some modifications like bagging techniques like Random Forests may be applied for parallel use with Vision Transformer and RNN sub-modules. The results are combined by training several models on various portions of the data and then making the consolidated prediction. This ensemble approach decreases the probability of over-fitting and increases the likelihood for ViTDroid to perform well on completely unseen cases, thus making the method more resistant to adversarial attacks and other modifications of the used samples of malware.

3.6 Explainability Layer: Enhancing Model Transparency

Cybersecurity is a domain where it is very important to know how a given model decides that a sample is malicious or benign, especially in the case of inspecting the samples that are classified as malware. To promote transparency of the results, ViTDroid has components of explainability that make the public trust in the output. These mechanisms are used to focus on the specific sections of the opcode sequence which might have a strong bearing on the model's decision. This made it more convenient to shift the focus of security analysts on the special opcodes or patterns that led to the detection of malware reducing the interpretative part of the model.

To improve the general interpretability ViTDroid uses post hoc interpretability tools such as LIME and SHAP. These tools help to determine how important each given feature contributes to the model in question (in this case, each opcode or patch). Thus, the applications of these tools, ViTDroid can help the security professionals with understandable standpoints on why such behaviors are malicious, so that those security professionals can make the right decision on the suspicion.

3.7 Training Process: Data Augmentation and Transfer Learning

To demonstrate that ViTDroid achieves good performance on practical datasets the training is done using modern approaches such as data augmentation and transfer learning. Data augmentation is used because, for malware detection, there are few samples available that are labeled for training. Thus, by creating the new training samples based on the transformations (for example the permutation of the opcodes order) the model can generalize upon the new, unseen types of malware and does not get overtrained on certain variants.

Transfer learning is also used to use models ready on related tasks, for example, image classification or malware detection in other contexts. This approach helps ViTDroid leverage the knowledge embedded in pre-trained models, and perform feature extraction to the particular task of Android and IoT malware classification. The process of transfer learning is most useful when training with a relatively small number of labeled inputs because the model itself can learn more quickly from small sets.

4.1 Experimental Setup

To assess the performance of ViTDroid we used a set of Android and IoT malware samples and the main focus was on smart city IoT devices. The datasets were then cleaned and tokenized to ensure opcode sequences were converted into image patches for the Vision Transformer module. Furthermore, the same sample of malware was split into training and test data sets to measure its generalization capability.

We trained and tested fusing hardware and software environments into one training and testing environment. The experiments were performed on a machine installed with an NVIDIA GPU, which provided resources for training complicated deep-learning models. The model was reproducibly built and experimented on with the help of established deep learning frameworks such as TensorFlow and PyTorch.

4.2 Datasets

To assess the execution of ViTDroid, two separate sets of malware were used, the first set was from Android malware, and the second set was from IoT malware. We chose these datasets to prove that the model works universally focusing on the Android domain and IoT devices in smart cities at that.

4.3 Android Malware Detection Results Random Neural Network Long Short-Term Memory Recursive Neural Network Gated Recurrent Unit

In this section we describe the performance of the ViTDroid model by testing various Recurrent Neural Network (RNN) configurations on the Android malware dataset. As these following experiments focus on judging each RNN variant's capability of mining the temporal characteristics of opcodes in Android malware,

The performance metrics for each model are summarized as follows:

Figure 2 Performance of the LSTM model.

Dependencies that arise with problems such as vanishing gradient make it less suitable for tackling more complex tasks.

Figure 3 Accuracy of the RNN model.

4.4 RNN Model

As can be seen in Figure 3, the training accuracy and test accuracy for the RNN model of opcode patterns demonstrates that the RNN achieves a training accuracy of 97%, meaning it has learned the input-opcode pairs effectively. The test accuracy stands at 95% and thus, we infer that the model has good generalization capability but maps a slight loss in terms of accuracy on unseen data sets. The difference between training and test accuracy is rather small; the model is not overfitting, but at the same time, it is not underfitting. In general, the performance of the RNN is satisfactory but there is still room for improvement to increase the robustness of the approach for new types of malware.

4.5 LSTM Model Performance

The graph on the LSTM model shows that the training phase and testing phase show a high level of accuracy, as shown in Figure 2. The model is also achieving an average training accuracy which is rising slowly and steadily and has reached 98% which demonstrates that the LSTM is learning to identify the op-code sequences of Android malware constantly.

Training accuracy is pretty high, equal to 97%, while the test one is lower 96%, still, which is very good for most cases and means that the model is very good in generalized to the new data. The differences between the training and test accuracy are not high meaning the LSTM model is fitting the data appropriately and isn't overfitting samples it hasn't Seen.

Figure 4 Performance of the GRU model.

This makes the LSTM architecture specifically appropriate for use in malware detection problems given that the ability to detect long-term dependencies and sequential patterns is of paramount importance.

4.6 GRU Model Performanc

As can be seen in Figure 4, the plot for the training accuracy of the GRU model also demonstrates that the specific metric is gradually growing and reaches the value of around 98%, which means that the GRU model is capable of discovering the patterns hidden in the opcode sequences from Android malware. This is due to the training being able to carry out the details of the sequence dependencies properly during the training phase.

The test accuracy of the GRU model converges at 0.96A like the LSTM model. The minor gap between the training and test accuracy indicates that similar to the LSTM model, the GRU model is equally capable of performing well on unseen data without having to overfit the model. The final GRU model is comparable to LSTM in terms of mean accuracy but is less parameterized and therefore much more efficient to compute.

4.7 All Model Comparison

In the case of comparing RNN, LSTM, and GRU models for Android malware detection, all have fared reasonably well in context with the training accuracy where 97% was achieved for RNN and 98% for LSTM as well as GRU models. However, when measured in test accuracy, RNN produced 95% whereas LSTM and GRU models produced 96% each. This makes us conclude that although the RNN facilitates good performance for the training data, it does slightly struggle in extension with other unseen data probably because the facility to capture long-term dependencies is weaker as empathized for LSTM and GRU. The performance of LSTM and GRU were comparable, although LSTM incurred slightly more computational cost; thus, GRU is preferable in practical scenarios. In total, based on the presented results, LSTM and GRU perform more generally than other examined RNN models with GRU being more efficient.

4.8 Model Comparison: IoT Malware Detection

The results of the comparative analysis of RNN, GRU, LSTM, and ViTDroid are shown in the Table 2 and Figure 5, which demonstrates the IoT malware identification score for the offered methods. In the presented experiments, LSTM and ViTDroid showed the highest results with 98.85% accuracy for LSTM and 98% for ViTDroid. The next was GRU with 95% of accuracy and RNN with 94.50%. Once again, ViTDroid had the best values for precision (0.97) and recall (0.98), and the performance of GRU and LSTM was similar. LSTM and ViTDroid emerged as the best out of all and ViTDroid was particularly good concerning both the precision and recall of the overall results and hence, could be effectively used for IoT malware identification.

Figure 5 Model comparison: IoT malware detection.

4.8.2 Model Accuracy Comparison: Android vs IoT Malware

Figure 6 Comparison: Android vs. IoT malware.

As can be seen in Table 3 and Figure 6, the comparison of the model accuracy of Android and IoT malware detection proves that ViTDroid was the best-performing algorithm with an accuracy rate of 99.1% for Android malware and 98% for IoT malware. Similarly, LSTM also gave improved results: 98% for Android and 98.85% for IoT. GRU got 96% for Android and 95% for IoT and RNN got 95% for Android and 94.50% for IoT. In aggregate, Android models were somewhat more accurate than IoT models, which could be a result of the fact that IoT malware is typically more nuanced. However, in both domains, ViTDroid and LSTM showed the best results.

4.9 Additional Metrics for Real-World Deployment

To evaluate the practicality of ViTDroid for real-world deployment, additional metrics such as model complexity, computational efficiency, and inference time were assessed. ViTDroid's hybrid design, combining Vision Transformers and RNNs, has higher complexity compared to standalone models like LSTM or GRU. While its training time (18 minutes per epoch) exceeds that of simpler models (e.g., RNN at 10 minutes per epoch), the added accuracy and robustness justify this overhead. Inference time, critical for real-time detection, was 3.5 ms/sample for ViTDroid, slightly higher than LSTM (2.5 ms/sample) or GRU (2.3 ms/sample). Despite these computational demands, ViTDroid's superior accuracy and explainability make it a strong candidate for deployment in critical security applications.