Pulpie: Cost-Effective Web Content Extraction for Cleaner AI Data

Pulpie is a family of open-source web content extraction models designed to strip boilerplate elements like advertisements, footers, and sidebars from raw HTML, delivering only the main content in HTML or Markdown format. Developed by Feyn, these models aim to provide clean web data efficiently and affordably.

The core innovation lies in Pulpie's architecture. Unlike many leading extractors, which are decoder-based and generate output token by token—requiring the full model to be read from memory at each step—Pulpie models are encoders. They perform a single forward pass over the entire input HTML, classifying each block as either boilerplate or content.

This encoder-based approach makes Pulpie compute-bound rather than memory-bound. This distinction is crucial for cost efficiency, as cheaper GPUs often have a higher ratio of compute power to memory bandwidth. Feyn claims this architecture allows Pulpie to clean one billion webpages for approximately $7,900, a significant reduction compared to the $159,000 cost for a competitor like Dripper.

The motivation for Pulpie emerged from the challenges of building robust AI systems. Noisy web content, often containing irrelevant ads or navigation elements, can degrade the quality of LLM outputs. Feyn's founder noted, "> an ad for 'Gemini on Pixel' slipped into our search results, got passed into LLM context, and ended up in the final answer served to the user."

Such incidents highlight how "bad data kills model intelligence." Pulpie was built to address this fundamental problem by providing a reliable and inexpensive method for data preprocessing.

Pulpie's models are open source and available on Hugging Face, with Feyn providing documentation on their training process and usage. This transparency and accessibility allow developers to integrate the models directly into their data pipelines, enhancing the purity of information fed into AI systems.

For professionals building Retrieval-Augmented Generation (RAG) systems, data analytics platforms, or automated research tools, Pulpie offers a method to preprocess web-scraped data, ensuring that large language models receive high-quality, relevant information. This can lead to more accurate and reliable AI applications, particularly where context purity is critical.