I built a simple web crawler using good old platform threads. It was just a multithreaded crawler, nothing fancy. But then, curiosity struck: “What happens if I use Virtual Threads instead?”

Virtual Threads are one of my favorite recent additions to the Java ecosystem. Switching from platform threads to Virtual Threads dramatically improved the URL processing rate… until the whole thing blew up with an OutOfMemoryError.

Yep. Fast became too fast.

I love exploring Java’s new features. Virtual Threads, Records, Pattern Matching, you name it. But every tool has tradeoffs. In software engineering, solving one problem can introduce another. Our job isn’t just writing code; it’s balancing performance, safety, and maintainability.

This post is about my little hacking session, where I tried to unleash the power of Virtual Threads, only to discover it takes a bit of finesse to avoid turning performance into a memory bomb. I’ll walk you through what happened, how I fixed it, and how you can too.

Before playing with Virtual Threads, I created a very basic crawler using traditional platform threads. The list of URLs to process was predetermined and inserted into an executor queue. Each platform thread would take a new task from the queue, which consisted of: getting a URL, fetching its content from a local server (to eliminate bandwidth and external rate limiting), emulating some processing, and then moving on to the next one.

public class PlatformThreadsCrawler {
//the executor service used to fetch and process data
private final ExecutorService executorService = Executors.newFixedThreadPool(200);
private final HttpClient httpClient = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(10))
.build();
private final AtomicLong processedCount = new AtomicLong(0);
//submit the job in the executor queue and then wait for results
public void crawlUrls(List<String> urls) {
long startTime = System.currentTimeMillis();
// Submit all download tasks
CompletableFuture<?>[] futures = new CompletableFuture[urls.size()];
int index = 0;
for (String url : urls) {
futures[index++] = CompletableFuture.runAsync(
() -> downloadAndProcess(url),
executorService
);
}
// Wait for all to complete
CompletableFuture.allOf(futures).join();
long endTime = System.currentTimeMillis();
System.out.println("\n=== Crawl Complete ===");
System.out.println("Time taken: " + (endTime - startTime) + "ms");
}
private void downloadAndProcess(String url) {
try {
//download
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url))
.timeout(Duration.ofSeconds(10))
.build();
HttpResponse<byte[]> response = httpClient.send(request,
HttpResponse.BodyHandlers.ofByteArray());
//if ok process the data
if (response.statusCode() == 200) {
byte[] content = response.body();
// Simulate some processing work
int result = process(content);
} else {
System.err.println("HTTP " + response.statusCode() + " for: " + url);
}
} catch (Exception e) {
System.err.println("Error downloading " + url + ": " + e.getMessage());
} finally {
long count = processedCount.incrementAndGet();
if (count % 100 == 0) {
System.out.println("Processed " + count + " URLs");
}
}
}
private static int process(byte[] content) {
//emulate some processing logic (omitted here)
}
}

I submitted a list of 20K urls, to be processed by 200 platform threads. For testing, I used a local HTTP server with static routes to simulate pages. I generated the urls starting from this list, and then duplicating it. The goal was to emulate small and medium file download (like simple html and small images).

urls.addAll(List.of(
"http://localhost:8080/data/1kb",    // 1KB
"http://localhost:8080/data/10kb",   // 10KB
"http://localhost:8080/data/100kb",  // 100KB
"http://localhost:8080/data/1mb"     // 1MB
));

The Java heap max size was 1GB, to emulate a scenario in which memory available was not “infinite”. I emulated the file processing with a simple word count logic. Here you can see some stats captured with VisualVM:

Executors.newFixedThreadPool(200)

Executors.newVirtualThreadPerTaskExecutor()

public class ControlledVirtualThreadsCrawler {
private final ExecutorService executorService = 
Executors.newVirtualThreadPerTaskExecutor();
private final Semaphore concurrencyLimit = new Semaphore(500);
private void downloadAndProcess(String url) {
concurrencyLimit.acquire();
try {
// ... existing download and process logic
} finally {
concurrencyLimit.release();
}
}
}