HTTP Request issue on the records fetch

Questions
, ,
1

Hi Team,

I am trying to get the records from the elastic search using HTTP Request. When I try to get the bulk records for last 14 days.
When I check in Kibana I am able to see around 417 records from the discover, When I execute same query in n8n , I am able to get only 208 records only.

I am able see as below because of time-out or may I know the reason :

image
image638Ă—777 17.2 KB

image

may I know how can we get full records using this node or please suggest me on this.

Thanks

3

Hi @DilipChiru, I am afraid I won’t be able to answer this questions purely based on the information you have provided. My first guess is that you might be using different indices. Kibana might default to querying multiple indices whereas your n8n node might not do this.

Can you confirm the exact structure of an example Elasticsearch document you can’t find using n8n and share the exact workflow you are using to query this document?

Edit: From looking at your screenshot it seems you’re actually seeing the correct number. The value 208 in the took field simply specifies the milliseconds it took Elasticsearch to execute the request as per Search API | Elasticsearch Guide [8.10] | Elastic.

You want to look at the hits field instead:

image
image638Ă—777 19.6 KB

4

Hi @MutedJam,

Thanks for the reply.
Actually, I took the example from the below link,

I have tried using the similar flow, But when I tried to get it. I am receiving the less records towards
“item-list” node. Below is the screenshot related to it

image
image2541Ă—472 63.7 KB

Basically, from the Http-request we have 433 records in last 14days. we can see from the screenshot. When I check from the Kibana discover page for last 14 days with same aggregation and also the respective filter. we can see 433 records. Even When I download into csv . We can see 433 records available from the csv.

Please suggest me on the above scenario to get full records. Currently, we are getting only 326 records.

Thanks

5

Hi @DilipChiru, this looks like you’re splitting out buckets from an aggregation now rather than actual hits/documents. One bucket can contain multiple documents.

You would need to fetch actual documents in your Elasticsearch query for this number to match.

But to be honest, these questions are better placed with the Elasticsearch community. n8n simply uses the data your Elasticsearch instance returns, it does not impact how Elasticsearch works.

6

Hi @MutedJam ,

Thanks for the reply, as mentioned. It is the issue on the Elasticsearch side. I am getting only less output with the API query.

Let me check further with Elasticsearch community.

Thank you for it :slight_smile:

1 Like
7

You’re most welcome.

On a separate note, perhaps you also want to switch to the Elasticsearch node provided by n8n here instead of having to build HTTP Requests? Something like below:
to fetch your individual documents

This example query will return individual documents rather than aggregation buckets, though chances are your actual fields have slightly different names than mine (so you cannot copy this query 1:1).