Introduction
Web scraping is a powerful technique for extracting data from websites. With Laravel's elegant syntax and rich ecosystem, you can build robust, scalable scrapers that automate data collection, extraction, and analysis. This guide covers everything from setup to advanced techniques.
---
Why Laravel for Web Scraping?
- Eloquent ORM for easy data storage
- Queue system for handling large-scale scraping
- Artisan commands for scheduled scraping tasks
- Built-in HTTP client with retry logic
- Rich ecosystem of packages
---
Setting Up Your Scraper
Install Required Packages
composer require symfony/dom-crawler
composer require guzzlehttp/guzzle
Create an Artisan Command
php artisan make:command ScrapeWebsite
---
Basic HTML Scraping
use Illuminate\Support\Facades\Http;
use Symfony\Component\DomCrawler\Crawler;
class ScrapeWebsite extends Command
{
protected $signature = 'scrape:website {url}';
public function handle()
{
$url = $this->argument('url');
$response = Http::get($url);
$crawler = new Crawler($response->body());
// Extract all article titles
$titles = $crawler->filter('h2.article-title')
->each(function (Crawler $node) {
return $node->text();
});
foreach ($titles as $title) {
$this->info("Found: {$title}");
}
}
}
---
Storing Scraped Data
// Migration
Schema::create('scraped_articles', function (Blueprint $table) {
$table->id();
$table->string('title');
$table->text('content')->nullable();
$table->string('source_url');
$table->timestamp('scraped_at');
$table->timestamps();
});
// Model
class ScrapedArticle extends Model
{
protected $fillable = ['title', 'content', 'source_url', 'scraped_at'];
}
---
Handling Pagination
public function scrapeAllPages(string $baseUrl): Collection
{
$allData = collect();
$page = 1;
do {
$response = Http::get("{$baseUrl}?page={$page}");
$crawler = new Crawler($response->body());
$items = $crawler->filter('.product-card')->each(function ($node) {
return [
'name' => $node->filter('.name')->text(),
'price' => $node->filter('.price')->text(),
];
});
$allData = $allData->merge($items);
$hasNext = $crawler->filter('.next-page')->count() > 0;
$page++;
sleep(2); // Be respectful!
} while ($hasNext);
return $allData;
}
---
Scheduling Scrapers
// app/Console/Kernel.php
protected function schedule(Schedule $schedule)
{
$schedule->command('scrape:website https://example.com')
->dailyAt('06:00')
->withoutOverlapping();
}
---
Best Practices
1. Respect robots.txt - Always check before scraping
2. Rate limiting - Add delays between requests
3. User-Agent headers - Identify your bot properly
4. Error handling - Implement retry logic
5. Data validation - Validate extracted data before storing
6. Legal compliance - Ensure scraping is permitted
---
Conclusion
Laravel provides an excellent foundation for building web scrapers. Its HTTP client, queue system, and Eloquent ORM create a powerful toolkit for automating data extraction. Remember to scrape responsibly and always respect website terms of service.
