bech bech - 2 years ago 63
C# Question

Azure Function fails to load assemblies after running fine for a while

I have a function listening to blob storage that have inserts frequently, though at times rarely enough for the function to go idle.

It is set up in a way that it contains some custom DLL's in the bin folder, and the actual function itself is merely calling a run method in one of those assemblies. This works fine. Most of the time.

My problem is, that if I let it run for a while it will suddenly fail to load my custom assemblies, with the following error message:

Exception while executing function: Functions.EventHandlerFunctionMicrosoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.EventHandlerFunction ---> System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation. ---> System.TypeLoadException : Could not load type 'MyAssembly.MyFunctions.EventHandlerFunction' from assembly 'MyAssembly.MyFunctions, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. at async Submission#0.Run(Stream blob,String name,TraceWriter log) at : 19 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine](TStateMachine& stateMachine) at Submission#0.Run(Stream blob,String name,TraceWriter log) End of inner exception at System.RuntimeMethodHandle.InvokeMethod(Object target,Object[] arguments,Signature sig,Boolean constructor) at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj,Object[] parameters,Object[] arguments) at System.Reflection.RuntimeMethodInfo.Invoke(Object obj,BindingFlags invokeAttr,Binder binder,Object[] parameters,CultureInfo culture) at async Microsoft.Azure.WebJobs.Script.Description.DotNetFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`1.InvokeAsync[TReflected](Object[] arguments) at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeAsync(IFunctionInvoker invoker,Object[] invokeParameters,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance) at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstance instance,IReadOnlyDictionary`2 parameters,TraceWriter traceWriter,CancellationToken…


As you can see from the stacktrace, my assembly (renamed MyAssembly.MyFunctions), suddenly fails to load. It will manage to recover on it's own, but the downtime and "dropped" blobs are somewhere around 1/5th of my blobs.

This seems dumb. While I realize I could mitigate this by reacting to that exception and reinserting the blob, this just seems like a waste of resources.

My guess is that the azure functions runtime sometimes fails to load dlls after being idle. Has anyone experienced this and found a way to solve it?

UPDATE: My Function looks like this:

#r "MyAssembly.MyFunctions.dll"

public static async Task Run(Stream blob, string name, TraceWriter log)
{
try {
await new EventHandlerFunction().Run(name);
} catch (Exception e) {
log.Error(e.Message);
}
}

Answer Source

To anyone stumbling on this:

The answer was that multiple versions of the same assemblies were located in the same function app, in different bin folders (different functions). Apparently, it would sometimes find a different version when loading assemblies, and that would cause the mentioned error.

There are two solutions:

  1. Preferably, start using the VS2017 tooling to build, test, and publish your functions. This puts all assemblies in the same bin folder and the whole development and publishing process resembles that of Webjobs.
  2. Alternatively, put all your assemblies in a shared folder, and reference that rather than specific bin folder for each function.
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download