Autoencoder-Based Anomaly Detection on Metadata Metrics for Privacy Enforcement Monitoring
Keywords:
autoencoder, anomaly detection, privacy enforcement, multivariate time series, reconstruction error, metadata metricsAbstract
A deep autoencoder can find outliers in multivariate privacy enforcement system telemetry. Autoencoders condense policy engine compilation time, content filter hit rates, and dataset access frequencies to characterise normal operational behaviour. The reconstruction error may suggest misconfigurations, regressions, or compliance issues. Z-score and exponentially weighted moving averages vs. suggested framework. Higher cardiacity improves early anomaly detection. Observability stacks provide proactive warning and privacy posture assessment without telemetry overhead. Use threshold calibration on labelled event datasets to simplify detection sensitivity vs. false positives testing. The study presents a scalable and adaptable solution to increase enterprise-grade data infrastructure telemetry systems' privacy detection time.
Downloads
References
A. Chalapathy and S. Chawla, "Deep learning for anomaly detection: A survey," arXiv preprint arXiv:1901.03407, 2019.
R. Akoglu, H. Tong, and D. Koutra, "Graph based anomaly detection and description: a survey," Data Mining and Knowledge Discovery, vol. 29, no. 3, pp. 626–688, 2015.
G. E. Hinton and R. R. Salakhutdinov, "Reducing the dimensionality of data with neural networks," Science, vol. 313, no. 5786, pp. 504–507, 2006.
J. An and S. Cho, "Variational autoencoder based anomaly detection using reconstruction probability," Special Lecture on IE, vol. 2, no. 1, pp. 1–18, 2015.
D. P. Kingma and M. Welling, "Auto-encoding variational Bayes," in Proc. Int. Conf. Learning Representations (ICLR), 2014.
T. Kim and C. Ahn, "Deep learning-based anomaly detection for time series data," International Journal of Advanced Computer Science and Applications, vol. 10, no. 1, pp. 112–119, 2019.
M. Goldstein and S. Uchida, "A comparative evaluation of unsupervised anomaly detection algorithms for multivariate data," PloS one, vol. 11, no. 4, p. e0152173, 2016.
M. Malhotra, L. Vig, G. Shroff, and P. Agarwal, "Long short term memory networks for anomaly detection in time series," in Proc. European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, 2015, pp. 89–94.
A. Ruff, R. Vandermeulen, N. Görnitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft, "Deep one-class classification," in Proc. Int. Conf. Machine Learning (ICML), 2018, pp. 4393–4402.
L. I. Kuncheva, "Change detection in streaming multivariate data using principal component analysis," IEEE Trans. Neural Netw., vol. 20, no. 1, pp. 157–166, Jan. 2009.
M. Ahmed, A. N. Mahmood, and J. Hu, "A survey of network anomaly detection techniques," Journal of Network and Computer Applications, vol. 60, pp. 19–31, 2016.
F. Chalapathy, A. Krishnan, and S. Chawla, "Anomaly detection using one-class neural networks," arXiv preprint arXiv:1802.06360, 2018.
S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997.
J. Redmon and A. Farhadi, "YOLOv3: An incremental improvement," arXiv preprint arXiv:1804.02767, 2018.
Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," Nature, vol. 521, pp. 436–444, 2015.
L. Breiman, "Random forests," Machine Learning, vol. 45, no. 1, pp. 5–32, 2001.
J. M. Zaki and M. M. Gaber, Data Stream Mining: A Practical Approach, Springer, 2014.
T. Chen, T. Kornblith, M. Norouzi, and G. Hinton, "A simple framework for contrastive learning of visual representations," in Proc. Int. Conf. Machine Learning (ICML), 2020, pp. 1597–1607.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going deeper with convolutions," in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1–9.
B. Lakshminarayanan, A. Pritzel, and C. Blundell, "Simple and scalable predictive uncertainty estimation using deep ensembles," in Advances in Neural Information Processing Systems (NeurIPS), 2017, pp. 6402–6413.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
