TL;DR
This project provides a longitudinal analysis of Arabic music lyrics spanning five decades (1970s–2020s). By leveraging Natural Language Processing (NLP) and automated data scraping, I analyzed 561 songs to quantify shifts in sentiment, the prevalence of explicit language, and the correlation between popularity metrics and lyrical content. The study successfully mapped how regional cultural shifts—such as the rise of Mahraganat and modern rap—have fundamentally reshaped the linguistic landscape of Arabic music.
Problem
While music serves as a primary cultural barometer, the evolution of Arabic lyrics has remained under-explored through a computational lens. There was a lack of empirical data to quantify how sentiment and “edginess” in lyrics have evolved over time. Specifically, it was unclear how modern movements emerging in the 2000s influenced the frequency of offensive language and the overall emotional tone of popular music across the region.
Design Iterations
The project evolved from a baseline sentiment analysis into a multi-dimensional data engineering pipeline:
-
Data Acquisition: I developed a custom scraper utilizing
Spotipyto extract metadata (popularity, release year) andGenius.pyto fetch raw lyrical content. -
Preprocessing Pipeline: I implemented a robust cleaning script (
Data Processing.py) designed to handle Arabic-specific linguistic challenges, including the removal of Tashkeel (diacritics), punctuation normalization, and the unification of visually similar characters (e.g., “ة” to “ه”). -
Analytical Layers:
- Sentiment Analysis: Utilized the CAMeL toolkit to categorize lyrics into positive, negative, or neutral sentiments.
- Linguistic Filtering: Engineered a custom filter to detect and categorize 245 specific offensive terms.
- Trend Mapping: Correlated these findings with Spotify’s popularity metrics to determine if “edgier” content correlates with higher engagement levels.
Technical Details
The backend architecture was built primarily in Python, utilizing a modular design to ensure scalability and clarity:
- Data Engineering: Employed
pandasfor complex dataframe manipulation and thecsvmodule for structured data storage between processing stages. - NLP Pipeline: Integrated
camel_toolsto achieve high-accuracy sentiment analysis tailored specifically for Arabic dialects. - Automation: Automated the collection of 561 tracks by iterating through yearly intervals (1970–2023) to ensure a consistent and representative longitudinal sample.
- Feature Engineering: Developed a “Popularity Score” metric to analyze the growth of music distribution in tandem with the evolution of lyrical content over time.
Results & Key Metrics
The following visualizations illustrate the core findings derived from the analysis:
The analysis yielded several high-impact findings:
- Sentiment Trends: While positive sentiments generally outpaced negative ones, the data revealed a distinct shift in lyrical tone beginning in the early 2000s.
- Offensive Language Spike: A significant increase in offensive language was detected starting in the 1980s, peaking sharply in the 2010s—a trend that correlates directly with the rise of Mahraganat and modern rap movements.
- Popularity Correlation: Spotify data confirmed a massive surge in popularity scores after 2000, coinciding with the diversification and increased “edginess” of lyrical content.
