Coroutines in LLVM

Warning

This is a work in progress. Compatibility across LLVM releases is notguaranteed.

Introduction

LLVM coroutines are functions that have one or more suspend points.When a suspend point is reached, the execution of a coroutine is suspended andcontrol is returned back to its caller. A suspended coroutine can be resumedto continue execution from the last suspend point or it can be destroyed.

In the following example, we call function f (which may or may not be acoroutine itself) that returns a handle to a suspended coroutine(coroutine handle) that is used by main to resume the coroutine twice andthen destroy it:

  1. define i32 @main() {
  2. entry:
  3.   %hdl = call i8* @f(i32 4)
  4.   call void @llvm.coro.resume(i8* %hdl)
  5.   call void @llvm.coro.resume(i8* %hdl)
  6.   call void @llvm.coro.destroy(i8* %hdl)
  7.   ret i32 0
  8. }

In addition to the function stack frame which exists when a coroutine isexecuting, there is an additional region of storage that contains objects thatkeep the coroutine state when a coroutine is suspended. This region of storageis called the coroutine frame. It is created when a coroutine is calledand destroyed when a coroutine either runs to completion or is destroyedwhile suspended.

LLVM currently supports two styles of coroutine lowering. These stylessupport substantially different sets of features, have substantiallydifferent ABIs, and expect substantially different patterns of frontendcode generation. However, the styles also have a great deal in common.

In all cases, an LLVM coroutine is initially represented as an ordinary LLVMfunction that has calls to coroutine intrinsics defining the structure ofthe coroutine. The coroutine function is then, in the most general case,rewritten by the coroutine lowering passes to become the “ramp function”,the initial entrypoint of the coroutine, which executes until a suspend pointis first reached. The remainder of the original coroutine function is splitout into some number of “resume functions”. Any state which must persistacross suspensions is stored in the coroutine frame. The resume functionsmust somehow be able to handle either a “normal” resumption, which continuesthe normal execution of the coroutine, or an “abnormal” resumption, whichmust unwind the coroutine without attempting to suspend it.

Switched-Resume Lowering

In LLVM’s standard switched-resume lowering, signaled by the use ofllvm.coro.id, the coroutine frame is stored as part of a “coroutineobject” which represents a handle to a particular invocation of thecoroutine. All coroutine objects support a common ABI allowing certainfeatures to be used without knowing anything about the coroutine’simplementation:

  • A coroutine object can be queried to see if it has reached completionwith llvm.coro.done.
  • A coroutine object can be resumed normally if it has not already reachedcompletion with llvm.coro.resume.
  • A coroutine object can be destroyed, invalidating the coroutine object,with llvm.coro.destroy. This must be done separately even if thecoroutine has reached completion normally.
  • “Promise” storage, which is known to have a certain size and alignment,can be projected out of the coroutine object with llvm.coro.promise.The coroutine implementation must have been compiled to define a promiseof the same size and alignment.

In general, interacting with a coroutine object in any of these ways whileit is running has undefined behavior.

The coroutine function is split into three functions, representing threedifferent ways that control can enter the coroutine:

  • the ramp function that is initially invoked, which takes arbitraryarguments and returns a pointer to the coroutine object;
  • a coroutine resume function that is invoked when the coroutine is resumed,which takes a pointer to the coroutine object and returns void;
  • a coroutine destroy function that is invoked when the coroutine isdestroyed, which takes a pointer to the coroutine object and returnsvoid.Because the resume and destroy functions are shared across all suspendpoints, suspend points must store the index of the active suspend inthe coroutine object, and the resume/destroy functions must switch overthat index to get back to the correct point. Hence the name of thislowering.

Pointers to the resume and destroy functions are stored in the coroutineobject at known offsets which are fixed for all coroutines. A completedcoroutine is represented with a null resume function.

There is a somewhat complex protocol of intrinsics for allocating anddeallocating the coroutine object. It is complex in order to allow theallocation to be elided due to inlining. This protocol is discussedin further detail below.

The frontend may generate code to call the coroutine function directly;this will become a call to the ramp function and will return a pointerto the coroutine object. The frontend should always resume or destroythe coroutine using the corresponding intrinsics.

Returned-Continuation Lowering

In returned-continuation lowering, signaled by the use ofllvm.coro.id.retcon or llvm.coro.id.retcon.once, some aspects ofthe ABI must be handled more explicitly by the frontend.

In this lowering, every suspend point takes a list of “yielded values”which are returned back to the caller along with a function pointer,called the continuation function. The coroutine is resumed by simplycalling this continuation function pointer. The original coroutineis divided into the ramp function and then an arbitrary number ofthese continuation functions, one for each suspend point.

LLVM actually supports two closely-related returned-continuationlowerings:

  • In normal returned-continuation lowering, the coroutine may suspenditself multiple times. This means that a continuation functionitself returns another continuation pointer, as well as a list ofyielded values.

The coroutine indicates that it has run to completion by returninga null continuation pointer. Any yielded values will be _undef_should be ignored.

  • In yield-once returned-continuation lowering, the coroutine mustsuspend itself exactly once (or throw an exception). The rampfunction returns a continuation function pointer and yieldedvalues, but the continuation function simply returns _void_when the coroutine has run to completion.

The coroutine frame is maintained in a fixed-size buffer that ispassed to the coro.id intrinsic, which guarantees a certain sizeand alignment statically. The same buffer must be passed to thecontinuation function(s). The coroutine will allocate memory if thebuffer is insufficient, in which case it will need to store atleast that pointer in the buffer; therefore the buffer must alwaysbe at least pointer-sized. How the coroutine uses the buffer mayvary between suspend points.

In addition to the buffer pointer, continuation functions take anargument indicating whether the coroutine is being resumed normally(zero) or abnormally (non-zero).

LLVM is currently ineffective at statically eliminating allocationsafter fully inlining returned-continuation coroutines into a caller.This may be acceptable if LLVM’s coroutine support is primarily beingused for low-level lowering and inlining is expected to be appliedearlier in the pipeline.

Coroutines by Example

The examples below are all of switched-resume coroutines.

Coroutine Representation

Let’s look at an example of an LLVM coroutine with the behavior sketchedby the following pseudo-code.

  1. void *f(int n) {
  2.    for(;;) {
  3.      print(n++);
  4.      <suspend> // returns a coroutine handle on first suspend
  5.    }
  6. }

This coroutine calls some function print with value n as an argument andsuspends execution. Every time this coroutine resumes, it calls print again with an argument one bigger than the last time. This coroutine never completes by itself and must be destroyed explicitly. If we use this coroutine witha main shown in the previous section. It will call print with values 4, 5and 6 after which the coroutine will be destroyed.

The LLVM IR for this coroutine looks like this:

  1. define i8* @f(i32 %n) {
  2. entry:
  3.   %id = call token @llvm.coro.id(i32 0, i8* null, i8* null, i8* null)
  4.   %size = call i32 @llvm.coro.size.i32()
  5.   %alloc = call i8* @malloc(i32 %size)
  6.   %hdl = call noalias i8* @llvm.coro.begin(token %id, i8* %alloc)
  7.   br label %loop
  8. loop:
  9.   %n.val = phi i32 [ %n, %entry ], [ %inc, %loop ]
  10.   %inc = add nsw i32 %n.val, 1
  11.   call void @print(i32 %n.val)
  12.   %0 = call i8 @llvm.coro.suspend(token none, i1 false)
  13.   switch i8 %0, label %suspend [i8 0, label %loop
  14.                                 i8 1, label %cleanup]
  15. cleanup:
  16.   %mem = call i8* @llvm.coro.free(token %id, i8* %hdl)
  17.   call void @free(i8* %mem)
  18.   br label %suspend
  19. suspend:
  20.   %unused = call i1 @llvm.coro.end(i8* %hdl, i1 false)
  21.   ret i8* %hdl
  22. }

The entry block establishes the coroutine frame. The coro.size intrinsic islowered to a constant representing the size required for the coroutine frame.The coro.begin intrinsic initializes the coroutine frame and returns thecoroutine handle. The second parameter of coro.begin is given a block of memoryto be used if the coroutine frame needs to be allocated dynamically.The coro.id intrinsic serves as coroutine identity useful in cases when thecoro.begin intrinsic get duplicated by optimization passes such asjump-threading.

The cleanup block destroys the coroutine frame. The coro.free intrinsic,given the coroutine handle, returns a pointer of the memory block to be freed ornull if the coroutine frame was not allocated dynamically. The _cleanup_block is entered when coroutine runs to completion by itself or destroyed viacall to the coro.destroy intrinsic.

The suspend block contains code to be executed when coroutine runs tocompletion or suspended. The coro.end intrinsic marks the point wherea coroutine needs to return control back to the caller if it is not an initialinvocation of the coroutine.

The loop blocks represents the body of the coroutine. The coro.suspendintrinsic in combination with the following switch indicates what happens tocontrol flow when a coroutine is suspended (default case), resumed (case 0) ordestroyed (case 1).

Coroutine Transformation

One of the steps of coroutine lowering is building the coroutine frame. Thedef-use chains are analyzed to determine which objects need be kept alive acrosssuspend points. In the coroutine shown in the previous section, use of virtual register%n.val is separated from the definition by a suspend point, therefore, itcannot reside on the stack frame since the latter goes away once the coroutineis suspended and control is returned back to the caller. An i32 slot isallocated in the coroutine frame and %n.val is spilled and reloaded from thatslot as needed.

We also store addresses of the resume and destroy functions so that thecoro.resume and coro.destroy intrinsics can resume and destroy the coroutinewhen its identity cannot be determined statically at compile time. For ourexample, the coroutine frame will be:

  1. %f.frame = type { void (%f.frame*)*, void (%f.frame*)*, i32 }

After resume and destroy parts are outlined, function f will contain only thecode responsible for creation and initialization of the coroutine frame andexecution of the coroutine until a suspend point is reached:

  1. define i8* @f(i32 %n) {
  2. entry:
  3.   %id = call token @llvm.coro.id(i32 0, i8* null, i8* null, i8* null)
  4.   %alloc = call noalias i8* @malloc(i32 24)
  5.   %0 = call noalias i8* @llvm.coro.begin(token %id, i8* %alloc)
  6.   %frame = bitcast i8* %0 to %f.frame*
  7.   %1 = getelementptr %f.frame, %f.frame* %frame, i32 0, i32 0
  8.   store void (%f.frame*)* @f.resume, void (%f.frame*)** %1
  9.   %2 = getelementptr %f.frame, %f.frame* %frame, i32 0, i32 1
  10.   store void (%f.frame*)* @f.destroy, void (%f.frame*)** %2
  11.  
  12.   %inc = add nsw i32 %n, 1
  13.   %inc.spill.addr = getelementptr inbounds %f.Frame, %f.Frame* %FramePtr, i32 0, i32 2
  14.   store i32 %inc, i32* %inc.spill.addr
  15.   call void @print(i32 %n)
  16.  
  17.   ret i8* %frame
  18. }

Outlined resume part of the coroutine will reside in function f.resume:

  1. define internal fastcc void @f.resume(%f.frame* %frame.ptr.resume) {
  2. entry:
  3.   %inc.spill.addr = getelementptr %f.frame, %f.frame* %frame.ptr.resume, i64 0, i32 2
  4.   %inc.spill = load i32, i32* %inc.spill.addr, align 4
  5.   %inc = add i32 %n.val, 1
  6.   store i32 %inc, i32* %inc.spill.addr, align 4
  7.   tail call void @print(i32 %inc)
  8.   ret void
  9. }

Whereas function f.destroy will contain the cleanup code for the coroutine:

  1. define internal fastcc void @f.destroy(%f.frame* %frame.ptr.destroy) {
  2. entry:
  3.   %0 = bitcast %f.frame* %frame.ptr.destroy to i8*
  4.   tail call void @free(i8* %0)
  5.   ret void
  6. }

Avoiding Heap Allocations

A particular coroutine usage pattern, which is illustrated by the main_function in the overview section, where a coroutine is created, manipulated anddestroyed by the same calling function, is common for coroutines implementingRAII idiom and is suitable for allocation elision optimization which avoiddynamic allocation by storing the coroutine frame as a static _alloca in itscaller.

In the entry block, we will call coro.alloc intrinsic that will return true_when dynamic allocation is required, and _false if dynamic allocation iselided.

  1. entry:
  2.   %id = call token @llvm.coro.id(i32 0, i8* null, i8* null, i8* null)
  3.   %need.dyn.alloc = call i1 @llvm.coro.alloc(token %id)
  4.   br i1 %need.dyn.alloc, label %dyn.alloc, label %coro.begin
  5. dyn.alloc:
  6.   %size = call i32 @llvm.coro.size.i32()
  7.   %alloc = call i8* @CustomAlloc(i32 %size)
  8.   br label %coro.begin
  9. coro.begin:
  10.   %phi = phi i8* [ null, %entry ], [ %alloc, %dyn.alloc ]
  11.   %hdl = call noalias i8* @llvm.coro.begin(token %id, i8* %phi)

In the cleanup block, we will make freeing the coroutine frame conditional oncoro.free intrinsic. If allocation is elided, coro.free returns _null_thus skipping the deallocation code:

  1. cleanup:
  2.   %mem = call i8* @llvm.coro.free(token %id, i8* %hdl)
  3.   %need.dyn.free = icmp ne i8* %mem, null
  4.   br i1 %need.dyn.free, label %dyn.free, label %if.end
  5. dyn.free:
  6.   call void @CustomFree(i8* %mem)
  7.   br label %if.end
  8. if.end:
  9.   ...

With allocations and deallocations represented as described as above, aftercoroutine heap allocation elision optimization, the resulting main will be:

  1. define i32 @main() {
  2. entry:
  3.   call void @print(i32 4)
  4.   call void @print(i32 5)
  5.   call void @print(i32 6)
  6.   ret i32 0
  7. }

Multiple Suspend Points

Let’s consider the coroutine that has more than one suspend point:

  1. void *f(int n) {
  2.    for(;;) {
  3.      print(n++);
  4.      <suspend>
  5.      print(-n);
  6.      <suspend>
  7.    }
  8. }

Matching LLVM code would look like (with the rest of the code remaining the sameas the code in the previous section):

  1. loop:
  2.   %n.addr = phi i32 [ %n, %entry ], [ %inc, %loop.resume ]
  3.   call void @print(i32 %n.addr) #4
  4.   %2 = call i8 @llvm.coro.suspend(token none, i1 false)
  5.   switch i8 %2, label %suspend [i8 0, label %loop.resume
  6.                                 i8 1, label %cleanup]
  7. loop.resume:
  8.   %inc = add nsw i32 %n.addr, 1
  9.   %sub = xor i32 %n.addr, -1
  10.   call void @print(i32 %sub)
  11.   %3 = call i8 @llvm.coro.suspend(token none, i1 false)
  12.   switch i8 %3, label %suspend [i8 0, label %loop
  13.                                 i8 1, label %cleanup]

In this case, the coroutine frame would include a suspend index that willindicate at which suspend point the coroutine needs to resume. The resumefunction will use an index to jump to an appropriate basic block and will lookas follows:

  1. define internal fastcc void @f.Resume(%f.Frame* %FramePtr) {
  2. entry.Resume:
  3.   %index.addr = getelementptr inbounds %f.Frame, %f.Frame* %FramePtr, i64 0, i32 2
  4.   %index = load i8, i8* %index.addr, align 1
  5.   %switch = icmp eq i8 %index, 0
  6.   %n.addr = getelementptr inbounds %f.Frame, %f.Frame* %FramePtr, i64 0, i32 3
  7.   %n = load i32, i32* %n.addr, align 4
  8.   br i1 %switch, label %loop.resume, label %loop
  9.  
  10. loop.resume:
  11.   %sub = xor i32 %n, -1
  12.   call void @print(i32 %sub)
  13.   br label %suspend
  14. loop:
  15.   %inc = add nsw i32 %n, 1
  16.   store i32 %inc, i32* %n.addr, align 4
  17.   tail call void @print(i32 %inc)
  18.   br label %suspend
  19.  
  20. suspend:
  21.   %storemerge = phi i8 [ 0, %loop ], [ 1, %loop.resume ]
  22.   store i8 %storemerge, i8* %index.addr, align 1
  23.   ret void
  24. }

If different cleanup code needs to get executed for different suspend points,a similar switch will be in the f.destroy function.

Note

Using suspend index in a coroutine state and having a switch in f.resume andf.destroy is one of the possible implementation strategies. We exploredanother option where a distinct f.resume1, f.resume2, etc. are created forevery suspend point, and instead of storing an index, the resume and destroyfunction pointers are updated at every suspend. Early testing showed that thecurrent approach is easier on the optimizer than the latter so it is alowering strategy implemented at the moment.

Distinct Save and Suspend

In the previous example, setting a resume index (or some other state change thatneeds to happen to prepare a coroutine for resumption) happens at the same time asa suspension of a coroutine. However, in certain cases, it is necessary to controlwhen coroutine is prepared for resumption and when it is suspended.

In the following example, a coroutine represents some activity that is drivenby completions of asynchronous operations async_op1 and async_op2 which geta coroutine handle as a parameter and resume the coroutine once asyncoperation is finished.

  1. void g() {
  2.    for (;;)
  3.      if (cond()) {
  4.         async_op1(<coroutine-handle>); // will resume once async_op1 completes
  5.         <suspend>
  6.         do_one();
  7.      }
  8.      else {
  9.         async_op2(<coroutine-handle>); // will resume once async_op2 completes
  10.         <suspend>
  11.         do_two();
  12.      }
  13.    }
  14. }

In this case, coroutine should be ready for resumption prior to a call toasync_op1 and async_op2. The coro.save intrinsic is used to indicate apoint when coroutine should be ready for resumption (namely, when a resume indexshould be stored in the coroutine frame, so that it can be resumed at thecorrect resume point):

  1. if.true:
  2.   %save1 = call token @llvm.coro.save(i8* %hdl)
  3.   call void @async_op1(i8* %hdl)
  4.   %suspend1 = call i1 @llvm.coro.suspend(token %save1, i1 false)
  5.   switch i8 %suspend1, label %suspend [i8 0, label %resume1
  6.                                        i8 1, label %cleanup]
  7. if.false:
  8.   %save2 = call token @llvm.coro.save(i8* %hdl)
  9.   call void @async_op2(i8* %hdl)
  10.   %suspend2 = call i1 @llvm.coro.suspend(token %save2, i1 false)
  11.   switch i8 %suspend1, label %suspend [i8 0, label %resume2
  12.                                        i8 1, label %cleanup]

Coroutine Promise

A coroutine author or a frontend may designate a distinguished alloca that canbe used to communicate with the coroutine. This distinguished alloca is calledcoroutine promise and is provided as the second parameter to thecoro.id intrinsic.

The following coroutine designates a 32 bit integer promise and uses it tostore the current value produced by a coroutine.

  1. define i8* @f(i32 %n) {
  2. entry:
  3.   %promise = alloca i32
  4.   %pv = bitcast i32* %promise to i8*
  5.   %id = call token @llvm.coro.id(i32 0, i8* %pv, i8* null, i8* null)
  6.   %need.dyn.alloc = call i1 @llvm.coro.alloc(token %id)
  7.   br i1 %need.dyn.alloc, label %dyn.alloc, label %coro.begin
  8. dyn.alloc:
  9.   %size = call i32 @llvm.coro.size.i32()
  10.   %alloc = call i8* @malloc(i32 %size)
  11.   br label %coro.begin
  12. coro.begin:
  13.   %phi = phi i8* [ null, %entry ], [ %alloc, %dyn.alloc ]
  14.   %hdl = call noalias i8* @llvm.coro.begin(token %id, i8* %phi)
  15.   br label %loop
  16. loop:
  17.   %n.val = phi i32 [ %n, %coro.begin ], [ %inc, %loop ]
  18.   %inc = add nsw i32 %n.val, 1
  19.   store i32 %n.val, i32* %promise
  20.   %0 = call i8 @llvm.coro.suspend(token none, i1 false)
  21.   switch i8 %0, label %suspend [i8 0, label %loop
  22.                                 i8 1, label %cleanup]
  23. cleanup:
  24.   %mem = call i8* @llvm.coro.free(token %id, i8* %hdl)
  25.   call void @free(i8* %mem)
  26.   br label %suspend
  27. suspend:
  28.   %unused = call i1 @llvm.coro.end(i8* %hdl, i1 false)
  29.   ret i8* %hdl
  30. }

A coroutine consumer can rely on the coro.promise intrinsic to access thecoroutine promise.

  1. define i32 @main() {
  2. entry:
  3.   %hdl = call i8* @f(i32 4)
  4.   %promise.addr.raw = call i8* @llvm.coro.promise(i8* %hdl, i32 4, i1 false)
  5.   %promise.addr = bitcast i8* %promise.addr.raw to i32*
  6.   %val0 = load i32, i32* %promise.addr
  7.   call void @print(i32 %val0)
  8.   call void @llvm.coro.resume(i8* %hdl)
  9.   %val1 = load i32, i32* %promise.addr
  10.   call void @print(i32 %val1)
  11.   call void @llvm.coro.resume(i8* %hdl)
  12.   %val2 = load i32, i32* %promise.addr
  13.   call void @print(i32 %val2)
  14.   call void @llvm.coro.destroy(i8* %hdl)
  15.   ret i32 0
  16. }

After example in this section is compiled, result of the compilation will be:

  1. define i32 @main() {
  2. entry:
  3.   tail call void @print(i32 4)
  4.   tail call void @print(i32 5)
  5.   tail call void @print(i32 6)
  6.   ret i32 0
  7. }

Final Suspend

A coroutine author or a frontend may designate a particular suspend to be final,by setting the second argument of the coro.suspend intrinsic to true.Such a suspend point has two properties:

  • it is possible to check whether a suspended coroutine is at the final suspendpoint via coro.done intrinsic;
  • a resumption of a coroutine stopped at the final suspend point leads toundefined behavior. The only possible action for a coroutine at a finalsuspend point is destroying it via coro.destroy intrinsic.

From the user perspective, the final suspend point represents an idea of acoroutine reaching the end. From the compiler perspective, it is an optimizationopportunity for reducing number of resume points (and therefore switch cases) inthe resume function.

The following is an example of a function that keeps resuming the coroutineuntil the final suspend point is reached after which point the coroutine isdestroyed:

  1. define i32 @main() {
  2. entry:
  3.   %hdl = call i8* @f(i32 4)
  4.   br label %while
  5. while:
  6.   call void @llvm.coro.resume(i8* %hdl)
  7.   %done = call i1 @llvm.coro.done(i8* %hdl)
  8.   br i1 %done, label %end, label %while
  9. end:
  10.   call void @llvm.coro.destroy(i8* %hdl)
  11.   ret i32 0
  12. }

Usually, final suspend point is a frontend injected suspend point that does notcorrespond to any explicitly authored suspend point of the high level language.For example, for a Python generator that has only one suspend point:

  1. def coroutine(n):
  2.   for i in range(n):
  3.     yield i

Python frontend would inject two more suspend points, so that the actual codelooks like this:

  1. void* coroutine(int n) {
  2.   int current_value;
  3.   <designate current_value to be coroutine promise>
  4.   <SUSPEND> // injected suspend point, so that the coroutine starts suspended
  5.   for (int i = 0; i < n; ++i) {
  6.     current_value = i; <SUSPEND>; // corresponds to "yield i"
  7.   }
  8.   <SUSPEND final=true> // injected final suspend point
  9. }

and python iterator next would look like:

  1. int __next__(void* hdl) {
  2.   coro.resume(hdl);
  3.   if (coro.done(hdl)) throw StopIteration();
  4.   return *(int*)coro.promise(hdl, 4, false);
  5. }

Intrinsics

Coroutine Manipulation Intrinsics

Intrinsics described in this section are used to manipulate an existingcoroutine. They can be used in any function which happen to have a pointerto a coroutine frame or a pointer to a coroutine promise.

‘llvm.coro.destroy’ Intrinsic

Syntax:
  1. declare void @llvm.coro.destroy(i8* <handle>)
Overview:

The ‘llvm.coro.destroy’ intrinsic destroys a suspendedswitched-resume coroutine.

Arguments:

The argument is a coroutine handle to a suspended coroutine.

Semantics:

When possible, the coro.destroy intrinsic is replaced with a direct call tothe coroutine destroy function. Otherwise it is replaced with an indirect callbased on the function pointer for the destroy function stored in the coroutineframe. Destroying a coroutine that is not suspended leads to undefined behavior.

‘llvm.coro.resume’ Intrinsic

  1. declare void @llvm.coro.resume(i8* <handle>)
Overview:

The ‘llvm.coro.resume’ intrinsic resumes a suspended switched-resume coroutine.

Arguments:

The argument is a handle to a suspended coroutine.

Semantics:

When possible, the coro.resume intrinsic is replaced with a direct call to thecoroutine resume function. Otherwise it is replaced with an indirect call basedon the function pointer for the resume function stored in the coroutine frame.Resuming a coroutine that is not suspended leads to undefined behavior.

‘llvm.coro.done’ Intrinsic

  1. declare i1 @llvm.coro.done(i8* <handle>)
Overview:

The ‘llvm.coro.done’ intrinsic checks whether a suspendedswitched-resume coroutine is at the final suspend point or not.

Arguments:

The argument is a handle to a suspended coroutine.

Semantics:

Using this intrinsic on a coroutine that does not have a final suspend pointor on a coroutine that is not suspended leads to undefined behavior.

‘llvm.coro.promise’ Intrinsic

  1. declare i8* @llvm.coro.promise(i8* <ptr>, i32 <alignment>, i1 <from>)
Overview:

The ‘llvm.coro.promise’ intrinsic obtains a pointer to acoroutine promise given a switched-resume coroutine handle and vice versa.

Arguments:

The first argument is a handle to a coroutine if from is false. Otherwise,it is a pointer to a coroutine promise.

The second argument is an alignment requirements of the promise.If a frontend designated %promise = alloca i32 as a promise, the alignmentargument to coro.promise should be the alignment of i32 on the targetplatform. If a frontend designated %promise = alloca i32, align 16 as apromise, the alignment argument should be 16.This argument only accepts constants.

The third argument is a boolean indicating a direction of the transformation.If from is true, the intrinsic returns a coroutine handle given a pointerto a promise. If from is false, the intrinsics return a pointer to a promisefrom a coroutine handle. This argument only accepts constants.

Semantics:

Using this intrinsic on a coroutine that does not have a coroutine promiseleads to undefined behavior. It is possible to read and modify coroutinepromise of the coroutine which is currently executing. The coroutine author anda coroutine user are responsible to makes sure there is no data races.

Example:
  1. define i8* @f(i32 %n) {
  2. entry:
  3.   %promise = alloca i32
  4.   %pv = bitcast i32* %promise to i8*
  5.   ; the second argument to coro.id points to the coroutine promise.
  6.   %id = call token @llvm.coro.id(i32 0, i8* %pv, i8* null, i8* null)
  7.   ...
  8.   %hdl = call noalias i8* @llvm.coro.begin(token %id, i8* %alloc)
  9.   ...
  10.   store i32 42, i32* %promise ; store something into the promise
  11.   ...
  12.   ret i8* %hdl
  13. }
  14.  
  15. define i32 @main() {
  16. entry:
  17.   %hdl = call i8* @f(i32 4) ; starts the coroutine and returns its handle
  18.   %promise.addr.raw = call i8* @llvm.coro.promise(i8* %hdl, i32 4, i1 false)
  19.   %promise.addr = bitcast i8* %promise.addr.raw to i32*
  20.   %val = load i32, i32* %promise.addr ; load a value from the promise
  21.   call void @print(i32 %val)
  22.   call void @llvm.coro.destroy(i8* %hdl)
  23.   ret i32 0
  24. }

Coroutine Structure Intrinsics

Intrinsics described in this section are used within a coroutine to describethe coroutine structure. They should not be used outside of a coroutine.

‘llvm.coro.size’ Intrinsic

  1. declare i32 @llvm.coro.size.i32()
  2. declare i64 @llvm.coro.size.i64()
Overview:

The ‘llvm.coro.size’ intrinsic returns the number of bytesrequired to store a coroutine frame. This is only supported forswitched-resume coroutines.

Arguments:

None

Semantics:

The coro.size intrinsic is lowered to a constant representing the size ofthe coroutine frame.

‘llvm.coro.begin’ Intrinsic

  1. declare i8* @llvm.coro.begin(token <id>, i8* <mem>)
Overview:

The ‘llvm.coro.begin’ intrinsic returns an address of the coroutine frame.

Arguments:

The first argument is a token returned by a call to ‘llvm.coro.id’identifying the coroutine.

The second argument is a pointer to a block of memory where coroutine framewill be stored if it is allocated dynamically. This pointer is ignoredfor returned-continuation coroutines.

Semantics:

Depending on the alignment requirements of the objects in the coroutine frameand/or on the codegen compactness reasons the pointer returned from coro.begin_may be at offset to the %mem_ argument. (This could be beneficial ifinstructions that express relative access to data can be more compactly encodedwith small positive and negative offsets).

A frontend should emit exactly one coro.begin intrinsic per coroutine.

‘llvm.coro.free’ Intrinsic

  1. declare i8* @llvm.coro.free(token %id, i8* <frame>)
Overview:

The ‘llvm.coro.free’ intrinsic returns a pointer to a block of memory wherecoroutine frame is stored or null if this instance of a coroutine did not usedynamically allocated memory for its coroutine frame. This intrinsic is notsupported for returned-continuation coroutines.

Arguments:

The first argument is a token returned by a call to ‘llvm.coro.id’identifying the coroutine.

The second argument is a pointer to the coroutine frame. This should be the samepointer that was returned by prior coro.begin call.

Example (custom deallocation function):
  1. cleanup:
  2.   %mem = call i8* @llvm.coro.free(token %id, i8* %frame)
  3.   %mem_not_null = icmp ne i8* %mem, null
  4.   br i1 %mem_not_null, label %if.then, label %if.end
  5. if.then:
  6.   call void @CustomFree(i8* %mem)
  7.   br label %if.end
  8. if.end:
  9.   ret void
Example (standard deallocation functions):
  1. cleanup:
  2.   %mem = call i8* @llvm.coro.free(token %id, i8* %frame)
  3.   call void @free(i8* %mem)
  4.   ret void

‘llvm.coro.alloc’ Intrinsic

  1. declare i1 @llvm.coro.alloc(token <id>)
Overview:

The ‘llvm.coro.alloc’ intrinsic returns true if dynamic allocation isrequired to obtain a memory for the coroutine frame and false otherwise.This is not supported for returned-continuation coroutines.

Arguments:

The first argument is a token returned by a call to ‘llvm.coro.id’identifying the coroutine.

Semantics:

A frontend should emit at most one coro.alloc intrinsic per coroutine.The intrinsic is used to suppress dynamic allocation of the coroutine framewhen possible.

Example:
  1. entry:
  2.   %id = call token @llvm.coro.id(i32 0, i8* null, i8* null, i8* null)
  3.   %dyn.alloc.required = call i1 @llvm.coro.alloc(token %id)
  4.   br i1 %dyn.alloc.required, label %coro.alloc, label %coro.begin
  5.  
  6. coro.alloc:
  7.   %frame.size = call i32 @llvm.coro.size()
  8.   %alloc = call i8* @MyAlloc(i32 %frame.size)
  9.   br label %coro.begin
  10.  
  11. coro.begin:
  12.   %phi = phi i8* [ null, %entry ], [ %alloc, %coro.alloc ]
  13.   %frame = call i8* @llvm.coro.begin(token %id, i8* %phi)

‘llvm.coro.noop’ Intrinsic

  1. declare i8* @llvm.coro.noop()
Overview:

The ‘llvm.coro.noop’ intrinsic returns an address of the coroutine frame ofa coroutine that does nothing when resumed or destroyed.

Arguments:

None

Semantics:

This intrinsic is lowered to refer to a private constant coroutine frame. Theresume and destroy handlers for this frame are empty functions that do nothing.Note that in different translation units llvm.coro.noop may return different pointers.

‘llvm.coro.frame’ Intrinsic

  1. declare i8* @llvm.coro.frame()
Overview:

The ‘llvm.coro.frame’ intrinsic returns an address of the coroutine frame ofthe enclosing coroutine.

Arguments:

None

Semantics:

This intrinsic is lowered to refer to the coro.begin instruction. This isa frontend convenience intrinsic that makes it easier to refer to thecoroutine frame.

‘llvm.coro.id’ Intrinsic

  1. declare token @llvm.coro.id(i32 <align>, i8* <promise>, i8* <coroaddr>,
  2.                                                         i8* <fnaddrs>)
Overview:

The ‘llvm.coro.id’ intrinsic returns a token identifying aswitched-resume coroutine.

Arguments:

The first argument provides information on the alignment of the memory returnedby the allocation function and given to coro.begin by the first argument. Ifthis argument is 0, the memory is assumed to be aligned to 2 sizeof(i8).This argument only accepts constants.

The second argument, if not null, designates a particular alloca instructionto be a coroutine promise.

The third argument is null coming out of the frontend. The CoroEarly pass setsthis argument to point to the function this coro.id belongs to.

The fourth argument is null before coroutine is split, and later is replacedto point to a private global constant array containing function pointers tooutlined resume and destroy parts of the coroutine.

Semantics:

The purpose of this intrinsic is to tie together coro.id, coro.alloc andcoro.begin belonging to the same coroutine to prevent optimization passes fromduplicating any of these instructions unless entire body of the coroutine isduplicated.

A frontend should emit exactly one coro.id intrinsic per coroutine.

‘llvm.coro.id.retcon’ Intrinsic

  1. declare token @llvm.coro.id.retcon(i32 <size>, i32 <align>, i8* <buffer>,
  2.                                    i8* <continuation prototype>,
  3.                                    i8* <alloc>, i8* <dealloc>)
Overview:

The ‘llvm.coro.id.retcon’ intrinsic returns a token identifying amultiple-suspend returned-continuation coroutine.

The ‘result-type sequence’ of the coroutine is defined as follows:

  • if the return type of the coroutine function is void, it is theempty sequence;
  • if the return type of the coroutine function is a struct, it is theelement types of that struct in order;
  • otherwise, it is just the return type of the coroutine function.

The first element of the result-type sequence must be a pointer type;continuation functions will be coerced to this type. The rest ofthe sequence are the ‘yield types’, and any suspends in the coroutinemust take arguments of these types.

Arguments:

The first and second arguments are the expected size and alignment ofthe buffer provided as the third argument. They must be constant.

The fourth argument must be a reference to a global function, calledthe ‘continuation prototype function’. The type, calling convention,and attributes of any continuation functions will be taken from thisdeclaration. The return type of the prototype function must match thereturn type of the current function. The first parameter type must bea pointer type. The second parameter type must be an integer type;it will be used only as a boolean flag.

The fifth argument must be a reference to a global function that willbe used to allocate memory. It may not fail, either by returning nullor throwing an exception. It must take an integer and return a pointer.

The sixth argument must be a reference to a global function that willbe used to deallocate memory. It must take a pointer and return void.

‘llvm.coro.id.retcon.once’ Intrinsic

  1. declare token @llvm.coro.id.retcon.once(i32 <size>, i32 <align>, i8* <buffer>,
  2.                                         i8* <prototype>,
  3.                                         i8* <alloc>, i8* <dealloc>)
Overview:

The ‘llvm.coro.id.retcon.once’ intrinsic returns a token identifying aunique-suspend returned-continuation coroutine.

Arguments:

As for llvm.core.id.retcon, except that the return type of thecontinuation prototype must be void instead of matching thecoroutine’s return type.

‘llvm.coro.end’ Intrinsic

  1. declare i1 @llvm.coro.end(i8* <handle>, i1 <unwind>)
Overview:

The ‘llvm.coro.end’ marks the point where execution of the resume part ofthe coroutine should end and control should return to the caller.

Arguments:

The first argument should refer to the coroutine handle of the enclosingcoroutine. A frontend is allowed to supply null as the first parameter, in thiscase coro-early pass will replace the null with an appropriate coroutinehandle value.

The second argument should be true if this coro.end is in the block that ispart of the unwind sequence leaving the coroutine body due to an exception andfalse otherwise.

Semantics:

The purpose of this intrinsic is to allow frontends to mark the cleanup andother code that is only relevant during the initial invocation of the coroutineand should not be present in resume and destroy parts.

In returned-continuation lowering, llvm.coro.end fully destroys thecoroutine frame. If the second argument is false, it also returns fromthe coroutine with a null continuation pointer, and the next instructionwill be unreachable. If the second argument is true, it falls throughso that the following logic can resume unwinding. In a yield-oncecoroutine, reaching a non-unwind llvm.coro.end without having firstreached a llvm.coro.suspend.retcon has undefined behavior.

The remainder of this section describes the behavior under switched-resumelowering.

This intrinsic is lowered when a coroutine is split intothe start, resume and destroy parts. In the start part, it is a no-op,in resume and destroy parts, it is replaced with ret void instruction andthe rest of the block containing coro.end instruction is discarded.In landing pads it is replaced with an appropriate instruction to unwind tocaller. The handling of coro.end differs depending on whether the target isusing landingpad or WinEH exception model.

For landingpad based exception model, it is expected that frontend uses thecoro.end intrinsic as follows:

  1. ehcleanup:
  2.   %InResumePart = call i1 @llvm.coro.end(i8* null, i1 true)
  3.   br i1 %InResumePart, label %eh.resume, label %cleanup.cont
  4.  
  5. cleanup.cont:
  6.   ; rest of the cleanup
  7.  
  8. eh.resume:
  9.   %exn = load i8*, i8** %exn.slot, align 8
  10.   %sel = load i32, i32* %ehselector.slot, align 4
  11.   %lpad.val = insertvalue { i8*, i32 } undef, i8* %exn, 0
  12.   %lpad.val29 = insertvalue { i8*, i32 } %lpad.val, i32 %sel, 1
  13.   resume { i8*, i32 } %lpad.val29

The CoroSpit pass replaces coro.end with True in the resume functions,thus leading to immediate unwind to the caller, whereas in start function itis replaced with False, thus allowing to proceed to the rest of the cleanupcode that is only needed during initial invocation of the coroutine.

For Windows Exception handling model, a frontend should attach a funclet bundlereferring to an enclosing cleanuppad as follows:

  1. ehcleanup:
  2.   %tok = cleanuppad within none []
  3.   %unused = call i1 @llvm.coro.end(i8* null, i1 true) [ "funclet"(token %tok) ]
  4.   cleanupret from %tok unwind label %RestOfTheCleanup

The CoroSplit pass, if the funclet bundle is present, will insertcleanupret from %tok unwind to caller beforethe coro.end intrinsic and will remove the rest of the block.

The following table summarizes the handling of coro.end intrinsic.

In Start FunctionIn Resume/Destroy Functions
unwind=falsenothingret void
unwind=trueWinEHnothingcleanupret unwind to caller
Landingpadnothingnothing

‘llvm.coro.suspend’ Intrinsic

  1. declare i8 @llvm.coro.suspend(token <save>, i1 <final>)
Overview:

The ‘llvm.coro.suspend’ marks the point where execution of aswitched-resume coroutine is suspended and control is returned backto the caller. Conditional branches consuming the result of thisintrinsic lead to basic blocks where coroutine should proceed whensuspended (-1), resumed (0) or destroyed (1).

Arguments:

The first argument refers to a token of coro.save intrinsic that marks thepoint when coroutine state is prepared for suspension. If none token is passed,the intrinsic behaves as if there were a coro.save immediately precedingthe coro.suspend intrinsic.

The second argument indicates whether this suspension point is final.The second argument only accepts constants. If more than one suspend point isdesignated as final, the resume and destroy branches should lead to the samebasic blocks.

Example (normal suspend point):
  1. %0 = call i8 @llvm.coro.suspend(token none, i1 false)
  2. switch i8 %0, label %suspend [i8 0, label %resume
  3.                               i8 1, label %cleanup]
Example (final suspend point):
  1. while.end:
  2.   %s.final = call i8 @llvm.coro.suspend(token none, i1 true)
  3.   switch i8 %s.final, label %suspend [i8 0, label %trap
  4.                                       i8 1, label %cleanup]
  5. trap:
  6.   call void @llvm.trap()
  7.   unreachable
Semantics:

If a coroutine that was suspended at the suspend point marked by this intrinsicis resumed via coro.resume the control will transfer to the basic blockof the 0-case. If it is resumed via coro.destroy, it will proceed to thebasic block indicated by the 1-case. To suspend, coroutine proceed to thedefault label.

If suspend intrinsic is marked as final, it can consider the true branchunreachable and can perform optimizations that can take advantage of that fact.

‘llvm.coro.save’ Intrinsic

  1. declare token @llvm.coro.save(i8* <handle>)
Overview:

The ‘llvm.coro.save’ marks the point where a coroutine need to update itsstate to prepare for resumption to be considered suspended (and thus eligiblefor resumption).

Arguments:

The first argument points to a coroutine handle of the enclosing coroutine.

Semantics:

Whatever coroutine state changes are required to enable resumption ofthe coroutine from the corresponding suspend point should be done at the pointof coro.save intrinsic.

Example:

Separate save and suspend points are necessary when a coroutine is used torepresent an asynchronous control flow driven by callbacks representingcompletions of asynchronous operations.

In such a case, a coroutine should be ready for resumption prior to a call toasync_op function that may trigger resumption of a coroutine from the same ora different thread possibly prior to async_op call returning control backto the coroutine:

  1. %save1 = call token @llvm.coro.save(i8* %hdl)
  2. call void @async_op1(i8* %hdl)
  3. %suspend1 = call i1 @llvm.coro.suspend(token %save1, i1 false)
  4. switch i8 %suspend1, label %suspend [i8 0, label %resume1
  5.                                      i8 1, label %cleanup]

‘llvm.coro.suspend.retcon’ Intrinsic

  1. declare i1 @llvm.coro.suspend.retcon(...)
Overview:

The ‘llvm.coro.suspend.retcon’ intrinsic marks the point whereexecution of a returned-continuation coroutine is suspended and controlis returned back to the caller.

llvm.coro.suspend.retcon` does not support separate save points;they are not useful when the continuation function is not locallyaccessible. That would be a more appropriate feature for a passconlowering that is not yet implemented.

Arguments:

The types of the arguments must exactly match the yielded-types sequenceof the coroutine. They will be turned into return values from the rampand continuation functions, along with the next continuation function.

Semantics:

The result of the intrinsic indicates whether the coroutine should resumeabnormally (non-zero).

In a normal coroutine, it is undefined behavior if the coroutine executesa call to llvm.coro.suspend.retcon after resuming abnormally.

In a yield-once coroutine, it is undefined behavior if the coroutineexecutes a call to llvm.coro.suspend.retcon after resuming in any way.

‘llvm.coro.param’ Intrinsic

  1. declare i1 @llvm.coro.param(i8* <original>, i8* <copy>)
Overview:

The ‘llvm.coro.param’ is used by a frontend to mark up the code used toconstruct and destruct copies of the parameters. If the optimizer discovers thata particular parameter copy is not used after any suspends, it can remove theconstruction and destruction of the copy by replacing corresponding coro.paramwith i1 false and replacing any use of the copy with the original.

Arguments:

The first argument points to an alloca storing the value of a parameter to acoroutine.

The second argument points to an alloca storing the value of the copy of thatparameter.

Semantics:

The optimizer is free to always replace this intrinsic with i1 true.

The optimizer is also allowed to replace it with i1 false provided that theparameter copy is only used prior to control flow reaching any of the suspendpoints. The code that would be DCE’d if the coro.param is replaced withi1 false is not considered to be a use of the parameter copy.

The frontend can emit this intrinsic if its language rules allow for thisoptimization.

Example:

Consider the following example. A coroutine takes two parameters a and _b_that has a destructor and a move constructor.

  1. struct A { ~A(); A(A&&); bool foo(); void bar(); };
  2.  
  3. task<int> f(A a, A b) {
  4.   if (a.foo())
  5.     return 42;
  6.  
  7.   a.bar();
  8.   co_await read_async(); // introduces suspend point
  9.   b.bar();
  10. }

Note that, uses of b is used after a suspend point and thus must be copiedinto a coroutine frame, whereas a does not have to, since it never usedafter suspend.

A frontend can create parameter copies for a and b as follows:

  1. task<int> f(A a', A b') {
  2.   a = alloca A;
  3.   b = alloca A;
  4.   // move parameters to its copies
  5.   if (coro.param(a', a)) A::A(a, A&& a');
  6.   if (coro.param(b', b)) A::A(b, A&& b');
  7.   ...
  8.   // destroy parameters copies
  9.   if (coro.param(a', a)) A::~A(a);
  10.   if (coro.param(b', b)) A::~A(b);
  11. }

The optimizer can replace coro.param(a’,a) with i1 false and replace all usesof a with a’, since it is not used after suspend.

The optimizer must replace coro.param(b’, b) with i1 true, since b is usedafter suspend and therefore, it has to reside in the coroutine frame.

Coroutine Transformation Passes

CoroEarly

The pass CoroEarly lowers coroutine intrinsics that hide the details of thestructure of the coroutine frame, but, otherwise not needed to be preserved tohelp later coroutine passes. This pass lowers coro.frame, coro.done,and coro.promise intrinsics.

CoroSplit

The pass CoroSplit buides coroutine frame and outlines resume and destroy partsinto separate functions.

CoroElide

The pass CoroElide examines if the inlined coroutine is eligible for heapallocation elision optimization. If so, it replacescoro.begin intrinsic with an address of a coroutine frame placed on its callerand replaces coro.alloc and coro.free intrinsics with false and null_respectively to remove the deallocation code.This pass also replaces _coro.resume and coro.destroy intrinsics with directcalls to resume and destroy functions for a particular coroutine where possible.

CoroCleanup

This pass runs late to lower all coroutine related intrinsics not replaced byearlier passes.

Areas Requiring Attention

  • A coroutine frame is bigger than it could be. Adding stack packing and stackcoloring like optimization on the coroutine frame will result in tightercoroutine frames.
  • Take advantage of the lifetime intrinsics for the data that goes into thecoroutine frame. Leave lifetime intrinsics as is for the data that stays inallocas.
  • The CoroElide optimization pass relies on coroutine ramp function to beinlined. It would be beneficial to split the ramp function further toincrease the chance that it will get inlined into its caller.
  • Design a convention that would make it possible to apply coroutine heapelision optimization across ABI boundaries.
  • Cannot handle coroutines with inalloca parameters (used in x86 on Windows).
  • Alignment is ignored by coro.begin and coro.free intrinsics.
  • Make required changes to make sure that coroutine optimizations work withLTO.
  • More tests, more tests, more tests