Virtual Threads

Virtual threads are lightweight threads that reduce the effort of writing, maintaining, and debugging high-throughput concurrent applications. They are introduced in Java 19 as a preview feature and can be enabled by using --enable-preview command line option. This article will discuss virtual threads in detail.

Green Threads

• During the time of Java 1.3 and before, JVM used to schedule threads onto a single OS thread used by the JVM.
  • This is called M:1 scheduling, as M Java threads would be scheduled onto 1 OS thread.
• Such threads are called green threads, as the scheduling is done by the Java runtime instead of the OS.

Platform Thread

• A platform thread is implemented as a thin wrapper around an operating system (OS) thread.
  • This is called 1:1 scheduling, because JVMs (version 1.4 and later) in modern OSs would create an OS thread for each Java thread created.
• A platform thread runs Java code on its underlying OS thread, and the platform thread captures its OS thread for the platform thread's entire lifetime.
  • Consequently, the number of available platform threads is limited to the number of OS threads.
• Platform threads typically have a large thread stack and other resources maintained by the operating system.
• Since these threads are expensive to create, they are usually pooled (using a thread pool).
• They are instances of java.lang.Thread class.

Virtual Thread

• Virtual threads are scheduled by the JVM to run on a platform thread.
  • The platform thread in this case is called the virtual thread's carrier.
• Many virtual threads can be scheduled to run on one of the OS threads (or platform threads).
  • This is called M:N scheduling, where M virtual threads are running on N platform threads
• Virtual threads can thus be called a type of green threads discussed earlier but the difference is that this time there could be even millions of virtual threads scheduled to run on a limited number of OS threads.
• Virtual threads too are instances of java.lang.Thread class.
  • You can tell whether a thread is virtual by calling isVirtual() method on a thread object.
  • Virtual threads are always daemon threads, so isDaemon() would always return true for virtual threads
  • Since virtual threads are instances of Thread class, much of your thread-related code will remain unaffected should you choose to run it on virtual threads.
• Virtual threads typically have a shallow call stack, performing as few as a single HTTP client call or a single JDBC query.
• Before ending this section, we will discuss two more related topics -
  • Mounting and Unmounting Virtual Threads
  • Little's law

1 - Mounting and Unmounting Virtual Threads

• The operating system schedules when a platform thread is run.
• However, the Java runtime schedules when a virtual thread is run.
• When the Java runtime schedules a virtual thread, it assigns or mounts the virtual thread on a platform thread.
  • This platform thread is called a carrier.
• After running some code, the virtual thread can unmount from its carrier.
  • This usually happens when the virtual thread performs a blocking I/O operation.
• After a virtual thread unmounts from its carrier, the carrier is free, which means that the Java runtime scheduler can mount a different virtual thread on it.

2 - Pinning a virtual thread

• A virtual thread cannot be unmounted during blocking operations when it is pinned to its carrier.
• A virtual thread is pinned in the following situations:
  • The virtual thread runs code inside a synchronized block or method
  • The virtual thread runs a native method or a foreign function
• Pinning does not make an application incorrect, but it might hinder its scalability.

3 - Little's Law

• The scalability of a stable system is governed by Little's Law, which relates latency, concurrency, and throughput.
• If each request has a duration (or latency) of d, and we can perform N tasks concurrently, then throughput T is given by:

T = N / d

• This law states that to scale up the throughput, we either have to
  • proportionally scale down the latency, or
  • scale up the number of requests we can handle concurrently.
• When we hit the limit on concurrent threads (as in the case of platform threads), the throughput of the thread-per-task model is limited by Little's Law.
• Virtual threads gracefully address this by giving us more concurrent threads rather than asking us to change our programming model.

When to use virtual threads?

• Use virtual threads in high-throughput concurrent applications, especially those that consist of a great number of concurrent tasks that spend much of their time waiting.
  • For example, virtual threads are suitable for running tasks that spend most of the time blocked, often waiting for I/O operations to complete.
  • However, they are not intended for long-running CPU-intensive operations. For these, platform threads are more suitable.
• Virtual threads are not faster threads; they do not run code any faster than platform threads.
  • They exist to provide scale (higher throughput), not speed (lower latency).
    • Throughput - No. of tasks performed at a given point in time
    • Latency - Time taken to complete a task

Code examples of virtual threads

1 - Spawning a single virtual thread

try {
    var builder = Thread.ofVirtual().name("MyThread");
    Runnable task = () -> {
        System.out.println("Running virtual thread: "
            + Thread.currentThread().getName());
    };
    var thread = builder.start(task);
    System.out.println("Started virtual thread: " + thread.getName());
    // wait for the thread to terminate
    thread.join();
} 
catch (InterruptedException e) {
    System.err.println("Caught InterruptedException");
}

Output:

Started virtual thread: MyThread
Running virtual thread: MyThread

2 - Spawning multiple virtual threads using executor

In this code, we are computing sum of 0 till 100 on each of the 8,000 virtual threads and then later summing up the results:

var executor = Executors.newVirtualThreadPerTaskExecutor();

Consumer<Long> sleep = (timeInMs) -> {
	try {
		TimeUnit.MILLISECONDS.sleep(timeInMs);
	}
	catch(InterruptedException ex) {
		System.out.println("InterruptedException occurred.");
	}
};

var futureList =
	IntStream.range(0, 8_000)
	.boxed()
	.map(idx ->
		executor.submit(() -> {
			var sumTill100 =
				IntStream.range(0, 100)
				.map(num -> {
					sleep.accept(5L);
					return num;
				})
				.reduce((a, b) -> a + b).orElse(0);

			System.out.println("Sum till 100 computed by Thread ID " + Thread.currentThread().threadId() + " is " + sumTill100);

			return sumTill100;
		})
	).collect(Collectors.toList());
		
sleep.accept(1000L);    

var sum =
	futureList.stream()
	.map(future -> {
		try {
			return future.get();
		}
		catch (InterruptedException | ExecutionException ex) {
			System.err.println("Caught InterruptedException or ExecutionException");
			return 0;
		}
	})
	.reduce((a, b) -> a + b).orElse(0);

System.out.println("The value of sum (should be 39600000): " + sum);

executor.shutdown();

Output:

Sum till 100 computed by Thread ID 7349 is 4950
Sum till 100 computed by Thread ID 7355 is 4950
...
Sum till 100 computed by Thread ID 8021 is 4950
The value of sum (should be 39600000): 39600000

Best practices for using virtual threads

1 - Do not pool virtual threads

• A thread pool is a group of preconstructed platform threads that are reused when they become available. Some thread pools have a fixed number of threads while others create new threads as needed.
• Do not pool virtual threads. Create one for every application task.
  • Virtual threads are short-lived and have shallow call stacks.
  • They don't need the additional overhead or the functionality of thread pools.

2 - Use semaphores for limited access

• A semaphore restricts the number of threads that can access a physical or logical resource.
• Use semaphores (instead of thread pools of fixed size) if you need to limit concurrency.
  • For example, if you make sure only a specified number of threads can access a limited resource, such as requests to a database.

3 - Avoid pinning virtual threads to their carrier

• As discussed in Pinning a virtual thread section, pinning might hinder an application's scalability.
• Try avoiding frequent and long-lived pinning by:
  • revising synchronized blocks, or
  • methods that run frequently and guard potentially long I/O operations with java.util.concurrent.locks.ReentrantLock.

4 - Use only when you want to increase throughput

• Virtual threads do not make your application faster, they just make it easier to process a large number of concurrent tasks.
  • In other words, use virtual threads when you want to increase throughput.
• For CPU-intensive tasks, it is better to use platform threads.

© 2022 Sumeet Das