Im trying to use tiktoken to get token length for some text but it is not working, getting some error saying Nonetype object has no attribute `new`

[Anasp]

Describe the problem/error/question

I need the tiktoken library. I see that pyodide has it supported, but I tried using the code node and just importing tiktoken and it didn’t work. So i also tried shifting over to javascript and trying some code there and it doesn’t work…

Basically I want to grab that element fit_data and the text from it and just run it through the python code that uses tiktoken to get the token count. If it is more than X tokens, I want to truncate it (I can do that part myself in the code) just the hard part that I am stuck on is getting the correct data element and running my tiktoken encoding on it… See the error please.

What is the error message (if any)?

Problem in node ‘Code‘
AttributeError: ‘NoneType’ object has no attribute ‘new’

Please share your workflow

Share the output returned by the last node

Information on your n8n setup

• n8n version: 1.94.1
• Database (default: SQLite): n/a
• n8n EXECUTIONS_PROCESS setting (default: own, main): manual
• Running n8n via (Docker, npm, n8n cloud, desktop app): website (is this n8n cloud) ?
• Operating system: macos sequioa 15.3.2

[Anasp]

Bumping this.

[King_Samuel_David]

Hello @Anasp

Hope everything is going well, after looking further into this, I found that using the code node, your be limited in using packages, see details here:

Code node documentation | n8n Docs

n8n added Python support in version 1.0. It doesn’t include a Python executable. Instead, n8n provides Python support using Pyodide, which is a port of CPython to WebAssembly. This limits the available Python packages to the Packages included with Pyodide. n8n downloads the package automatically the first time you use it.

But as a workaround, you can use the execute command node,

There are a few steps such as installing python, this video details it https://www.youtube.com/watch?v=Z-3rY3Rku20

And as a step by step,

1. Install Python and Pip as Root

From within your container or host:

For Alpine-based image:

apk add --no-cache python3 py3-pip rust cargo

rust and cargo are required to compile tiktoken on ARM.

---

2. Switch to node user & install tiktoken

su node
pip3 install --user tiktoken --break-system-packages

This installs tiktoken under /home/node/.local/.

---

3. Run a working Execute Command node

Paste this into your n8n Execute Command node:

su node -c "python3 -c \"import tiktoken; data = 'This is a test'; enc = tiktoken.get_encoding('cl100k_base'); print('Token count:', len(enc.encode(data)))\""

Output should be:

Token count: 6

4. (Optional) Add /home/node/.local/bin to PATH

If you ever need to use any CLI tools installed by pip:

echo 'export PATH=$HOME/.local/bin:$PATH' >> /home/node/.profile

I tested and this works, you can also pass in the expression from the edit fields node, it’s abit of a headache, as if your not using root, you can’t sudo with the node users, for me and my dev env I use the root user in my yml, but you can also use this:

docker exec -it --user root n8n sh

To gain access as root to install, and then just switch back to the node user for the pip install for step 2.

Hope this helps,

Samuel