There seems to be a huge delay in the LM Callbacks, called before and after the actual LLM is requested. Most of the time is spent inside n8n, not the language model. See this post:
Removing the callbacks actually speed up the thing a lot!