Automatic tests for CI

Questions
1

I was wondering about running a new version of n8n, but it broke my workflow. After looking at the code of that specific node it was clear: it got refactored and something was broken. But a cool project, needs cool CI (bad joke, I guess) - but it really needs it. So I found this. It already runs on every commit, but there aren’t any tests.

Are there any plans to get those nodes tested? So we can be sure, that a code change might not break a node?

Greets, Michael

1 Like
2

Very sorry that you had problems with upgrading!

Yes you are totally right we need definitely more tests! It is on our to-do list for a while now and I hope we can get to it soon. In this case we can sadly not do much and have to trust the community as we simply do not have a self-hosted Jira-Version running. At some point we would have to mock all APIs and then run the tests against them. Sadly do we however have currently not the resources to do that.

Normally do we try to avoid any breaking changes in general. Sometimes it is however needed and then they get added here:

So before upgrading this file should always be checked first, to see if anything has to be done.

3

Hi @jan,

thanks for answering! I read the changelog before upgrading. The issue linked above doesn’t seem to be related to renaming, instead it changed code in how to retrieve credentials and thus, broke. Anyway, I’ll fix that later.

What I’m asking for is a guideline on how n8n’s plan look like for integrating tests? This project is wonderful, but looses trust on every update if anything broke, that could be easily avoided! I know adding tests afterwards is a hard challenge. But I think it could be easily done by:

  1. adding a test-suite, writing tests “as example” (dear dev, look here for an example on how to test your node).
  2. encouraging developers to write tests for bugfixes (“do you please mind, to add some tests?”).
  3. requiring tests for new nodes - prohibiting merge requests for new nodes without a single test.

That said, do you have an example on how to test a node? As soon as I can spend some time I’ll add tests for jira (on-premise version). I’m eager to learn new things, so please guide me (and other developers) in the correct direction. Please share your plans so this can happen :slight_smile:

4

Ah yes. The breaking-changes only contain breaking-changes we are aware of, do knowingly and on purpose. The issue you are experiencing was not supposed to be there so is more a bug.

About testing. That “plan” or “framework” did sadly not get created yet. Once we have it we will also create some examples, documentation and helper-functions to make it easy for other people to do the same and to create tests.

Thanks a lot for your suggestions, they are very helpful and appreciated! Also very great that you offer to help, so also thanks for that! Great to have people like you being part of the community!

I will update here once we have something ready.

5

Hi Jan and Michael,

Just wanted to bump up this conversation to see how we’re doing on automated testing of nodes. Looking at the code itself, I still don’t see a lot of tests done on the nodes themselves, only on some of the core functionality.

I’ve been playing around with Jest a little bit using mocks on our custom nodes, but without any success yet. If I had just an example of how a single node could be tested it would be great so I was wondering if you have anything like that Jan?

Regards,
Pim

1 Like
6

How is it going in May 2022? Are there any recommendations on how to test nodes? Since then, a huge number of nodes have been developed. And if the community does not write tests, then nodes can break silently. Just paying attention to the important issue of nodes testing.

7

Hey all, I know that we had a bunch of engineers implement some basic smoke tests using cypress in an internal hackathon last week.

Perhaps @agobrecht, @alexgrozav or @Michael_Kret can share some additional details on how they approached these tests?

8

Hey! Essentially, we’ve written some tests to ensure that core functionality is working as expected after every merge.

To do this, we are:

  • running n8n locally with a clean database
  • setting up a new user
  • logging in with the new user and storing the logged in session
  • going through the onboarding process
  • creating a function node
  • creating a set node and interacting with the inputs (setting a new field using expressions)
  • executing the workflow
  • checking for successful execution
  • creating a new workflow

To do:

  • duplicating a workflow
  • deleting a workflow
  • exporting a workflow
  • ensure it is working as expected in a CI environment

The code for it can be found here: N8N-3722 Cypress E2E tests by alexgrozav · Pull Request #3490 · n8n-io/n8n · GitHub

1 Like