🚀 Python-Powered K-Pop Idol News Scraping & Visualization with a Minecraft Twist

Scraping K-pop idol news allows fans and analysts to track trending topics, comeback announcements, and social‑media buzz in real time. By using Python’s requests library alongside BeautifulSoup, you can pull article headlines, publish dates, and engagement metrics from Korean entertainment sites such as Naver News or Daum.

Once the raw data is collected, transforming it into a Minecraft‑themed visualization makes the insights both fun and shareable. For example, you can map each idol’s weekly mention count to a specific block type—diamond for high activity, gold for medium, and stone for low—then render a colorful chart that looks like a Minecraft world map.

1. Build a Robust News Scraper

Start by installing requests and beautifulsoup4 via pip, then set a realistic User‑Agent header to avoid being blocked. Loop through paginated search results for a keyword like '아이돌 뉴스' and extract the <a> tags inside each article container, storing the URL, title, and timestamp in a list of dictionaries.

Add error handling with try/except for connection timeouts and implement a delay of 1‑2 seconds between requests using time.sleep to respect the site’s rate limits. The following code snippet shows a basic loop that fetches up to 20 pages and appends each article’s data to a list.

import requests\nfrom bs4 import BeautifulSoup\nimport time\n\nheaders = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0 Safari/537.36'}\nbase_url = 'https://search.naver.com/search.naver?where=news&query=%EC%95%84%EC%9D%B4%EB%8F%84+%EB%89%B4%EC%8A%A4&start={}'\narticles = []\nfor page in range(0, 200, 10):\n url = base_url.format(page)\n try:\n resp = requests.get(url, headers=headers, timeout=10)\n resp.raise_for_status()\n soup = BeautifulSoup(resp.text, 'html.parser')\n for item in soup.select('ul.list_news li'):\n title_tag = item.select_one('a.news_tit')\n if title_tag:\n articles.append({\n 'title': title_tag.get_text(strip=True),\n 'url': title_tag['href'],\n 'timestamp': item.select_one('span.info').get_text(strip=True) if item.select_one('span.info') else None\n })\n time.sleep(1.5)\n except Exception as e:\n print(f'Error on page {page}: {e}')\n continue

2. Clean, Deduplicate, and Store the Data

Load the scraped list into a pandas DataFrame, then convert the timestamp strings to datetime objects using pd.to_datetime with format='%Y.%m.%d %H:%M'. Drop duplicates based on the title and URL columns, and filter out any entries older than 30 days to keep the dataset relevant.

Save the cleaned DataFrame to a CSV file for later analysis, and optionally push it to a SQLite database for quick querying. The example below demonstrates the cleaning steps and the to_csv call.

import pandas as pd\n\n# Assuming 'articles' list from previous step\ndf = pd.DataFrame(articles)\ndf['timestamp'] = pd.to_datetime(df['timestamp'], errors='coerce')\ndf = df.drop_duplicates(subset=['title', 'url'])\ndf = df[df['timestamp'] >= pd.Timestamp.now() - pd.Timedelta(days=30)]\ndf.to_csv('kpop_news_clean.csv', index=False)\n# optional SQLite\nimport sqlite3\nconn = sqlite3.connect('kpop_news.db')\ndf.to_sql('articles', conn, if_exists='replace', index=False)\nconn.close()

3. Create a Minecraft‑Style Visualization

Define a custom colormap that mimics Minecraft block colors: map low mention counts to a stone gray (#808080), medium to gold (#FFD700), and high to diamond blue (#1E90FF). Use matplotlib’s barh function to plot each idol’s weekly mention count, setting the bar color via the colormap based on normalized values.

To add a playful touch, overlay a small Minecraft grass‑block icon in the background of the chart using PIL, then save the figure as a PNG. The code snippet shows how to generate the colormap, create the bar chart, and composite the icon.

import matplotlib.pyplot as plt\nimport numpy as np\nfrom PIL import Image\n\n# Assuming df has columns: idol, mentions\ndf_viz = df.groupby('idol')['mentions'].sum().reset_index()\ndf_viz = df_viz.sort_values('mentions', ascending=False)\n\n# Normalize mentions for colormap\nnorm = plt.Normalize(df_viz['mentions'].min(), df_viz['mentions'].max())\n# Custom Minecraft-inspired colors\ncolors = ['#808080', '#FFD700', '#1E90FF'] # stone, gold, diamond\ncmap = plt.LinearSegmentedColormap.from_list('minecraft', colors, N=256)\n\nfig, ax = plt.subplots(figsize=(8,6))\nbars = ax.barh(df_viz['idol'], df_viz['mentions'], color=cmap(norm(df_viz['mentions'])))\nax.set_xlabel('Weekly Mentions')\nax.set_title('K-Pop Idol Activity (Minecraft Theme)')\n\n# Overlay grass block icon\ntry:\n grass = Image.open('grass_block.png').resize((50,50))\n fig_image = Image.new('RGBA', fig.canvas.get_width_height(), (255,255,255,0))\n fig_image.paste(grass, (10,10))\n fig_image = np.array(fig_image)\n ax.imshow(fig_image, extent=[0, df_viz['mentions'].max(), -0.5, len(df_viz)-0.5], aspect='auto', alpha=0.3)\nexcept FileNotFoundError:\n pass # Icon optional\n\nplt.tight_layout()\nplt.savefig('kpop_minecraft_viz.png', dpi=150)\nplt.close()

Wrap-Up and Next Steps

By combining Python’s scraping power with a Minecraft‑themed visual approach, you turn raw K‑pop news data into an engaging story that resonates with both data enthusiasts and gamers. The workflow—scrape, clean, store, visualize—can be automated with a simple cron job or GitHub Actions to keep the chart updated daily.

Future enhancements could include sentiment analysis using KoNLPy to gauge fan reactions, or integrating the visualization into a Minecraft mod that displays real‑time idol stats as in‑game billboards. Experiment with different block palettes or interactive libraries like Plotly to make the dashboard even more immersive.

🚀 Share files with us securely! : www.simpledrop.net