Complexity-based graph attention network for metamorphic malware detection
| dc.contributor.author | Brezinski, Kenneth | |
| dc.contributor.examiningcommittee | McLeod, Bob (Electrical and Computer Engineering) | |
| dc.contributor.examiningcommittee | Mohammed, Noman (Computer Science) | |
| dc.contributor.examiningcommittee | Dansereau, Richard (Carleton University) | |
| dc.contributor.supervisor | Ferens, Ken | |
| dc.date.accessioned | 2024-08-19T14:03:33Z | |
| dc.date.available | 2024-08-19T14:03:33Z | |
| dc.date.issued | 2024-08-05 | |
| dc.date.submitted | 2024-08-05T22:05:49Z | en_US |
| dc.date.submitted | 2024-08-19T04:20:47Z | en_US |
| dc.degree.discipline | Electrical and Computer Engineering | |
| dc.degree.level | Doctor of Philosophy (Ph.D.) | |
| dc.description.abstract | This thesis work presents a new approach to malware analysis by creating a specialized sandbox environment for executing and monitoring malware on a host operating system. Over 200 malware samples, along with benignware, were tested in this environment. Application Programming Interface (API) calls were traced of how these executable interacted with the host system, which including registry changes, file system access, and thread activity. Two new methods to measure complexity, Mass Radius and Radius of Gyration Fractal Dimension (FD), were developed and added to a Deep Learning model. These complexity methods helped the model converge faster. Tests showed that the Mass Radius FD measure improved convergence and accuracy over the Radius of Gyration FD in identifying malware. The complexity models performed better than standard models across different datasets. The study also found that shorter sequences of API calls and file events were more likely to indicate malicious behavior. Using GNNExplainer, linked API sequences were linked to specific malware techniques, providing deeper insights into the model’s predictions. The complexity models identified flaws in traditional methods and successfully flagged malware that other commercial sandbox methods were not able to identify, lending credence to the sophistication and applicability of this work. | |
| dc.description.note | October 2024 | |
| dc.identifier.uri | http://hdl.handle.net/1993/38391 | |
| dc.language.iso | eng | |
| dc.rights | open access | en_US |
| dc.subject | Deep Learning | |
| dc.subject | Cyber Security | |
| dc.subject | Complexity Analysis | |
| dc.subject | Malware | |
| dc.title | Complexity-based graph attention network for metamorphic malware detection | |
| dc.type | doctoral thesis | en_US |
| local.subject.manitoba | no | |
| oaire.awardNumber | IT15018 | |
| oaire.awardTitle | Mitacs Accelerate | |
| oaire.awardURI | https://www.mitacs.ca/our-programs/accelerate-core-business/ | |
| project.funder.name | Mitacs |