Cutting Edge '25

Harmful Visual Content Detection System for Social Media Platforms

By

Catagories

Play Video

"Harmful content on social media platforms poses a significant threat to user safety, especially when such content includes violent, abusive, or inappropriate visuals that bypass traditional moderation filters. Current content moderation systems often fail to detect nuanced harmful visuals, such as subtle gestures or weapons shown in harmless contexts. Moreover, most existing approaches focus on text-based filtering or rely on basic object detection, which struggle to understand the real context behind the images and videos shared online. This research addresses the pressing need for a more intelligent and context-aware detection system that can help minimize the spread of harmful visuals in real time. To solve this issue, a hybrid AI-based system was developed that integrates YOLOv8 for object detection with a vision-language model to analyze visual context and determine the harmfulness of content. The system detects harmful categories such as alcohol, blood, cigarettes, guns, knives, and insulting gestures in both images and videos. It then classifies the content as harmful or non-harmful based on the scenario. An automatic alert mechanism is also implemented to notify administrators via email when harmful content is detected. The backend was built with Flask, and a user-friendly interface was provided for seamless interaction and visualization of results. The system demonstrated strong performance with high detection accuracy across various test cases, including challenging scenarios with small object sizes, low lighting, and multiple object categories. Evaluation results showed high precision and recall values, and experts praised the contextual understanding achieved by combining object detection with language reasoning. The model successfully flagged harmful content and generated contextual justifications, offering a practical solution for enhancing safety on social media platforms. These results indicate that the proposed system is both effective and scalable for real-time harmful visual content moderation."

Vision Quest

Check out the visionary projects our students have brought up in this year
VisuaLit

VisuaLit is an AI-powered eBook reader that redefines traditional reading by merging visual storytelling, audio narration, and contextual learning into…

VenDoor

The VenDoor application is a fully functional mobile application designed to create a bridge between mobile vendors and their customers…

UniGuide

UniGuide is a student-focused platform that helps individuals make smart educational and career decisions. It offers a comprehensive database of…