How to get new items from API?

[Ruriko]

Describe the issue/error/question

I’m trying to get new items from an API and then upload the image but so far I’m only able to get it half working. It is getting the new items but it is skipping quite a few items. I had a look at past executions and noticed it stops at the function node which gives an empty result but it shouldn’t since I know the api has a lot of items frequently updated every few minutes. So how do I get new items without it skipping some of them?

What is the error message (if any)?

Please share the workflow

Share the output returned by the last node

Information on your n8n setup

• **n8n version:**0.182.1
• Database you’re using (default: SQLite):
• Running n8n with the execution process [own(default), main]:
• **Running n8n via [Docker, npm, n8n.cloud, desktop app]:**npm

[MutedJam]

Hi @Ruriko, execution would stop if a node does not return any items. So it would appear your filtering logic doesn’t quite work as expected from the sounds of it. Perhaps you can share a JSON dataset coming from your Set node that you would expect to make it past your Function node?

[Ruriko]

@MutedJam I updated my workflow by deleting the SET node and tested it but it still skipping a few items

[MutedJam]

Could you share some example JSON data you’re sending to the Function node which you’d expect to be passed on by the Function node but which isn’t?

Also, could you share the value of staticData in your node?

[Ruriko]

json data

[
{
"id": 99187184,
"title": "大天狗と薄い本",
"type": "illust",
"image_urls": {
"square_medium": "https://i.pximg.net/c/360x360_70/img-master/img/2022/06/20/20/49/28/99187184_p0_square1200.jpg",
"medium": "https://i.pximg.net/c/540x540_70/img-master/img/2022/06/20/20/49/28/99187184_p0_master1200.jpg",
"large": "https://i.pximg.net/c/600x1200_90/img-master/img/2022/06/20/20/49/28/99187184_p0_master1200.jpg"
},
"caption": "大事なお勉強中の飯綱丸様",
"restrict": 0,
"user": {
"id": 12998,
"name": "ぽて",
"account": "ptkankodori",
"profile_image_urls": {
"medium": "https://i.pximg.net/user-profile/img/2017/12/17/12/19/50/13571974_5304245c067c2e0122748858393815bc_170.png"
},
"is_followed": false
},
"tags": [
{
"name": "東方",
"translated_name": null
},
{
"name": "東方虹龍洞",
"translated_name": null
},
{
"name": "飯綱丸龍",
"translated_name": null
},
{
"name": "天弓千亦",
"translated_name": null
}
],
"tools": [
"SAI"
],
"create_date": "2022-06-20T20:49:28+09:00",
"page_count": 2,
"width": 1073,
"height": 1523,
"sanity_level": 2,
"x_restrict": 0,
"series": null,
"meta_single_page": {
},
"meta_pages": [
{
"image_urls": {
"square_medium": "https://i.pximg.net/c/360x360_70/img-master/img/2022/06/20/20/49/28/99187184_p0_square1200.jpg",
"medium": "https://i.pximg.net/c/540x540_70/img-master/img/2022/06/20/20/49/28/99187184_p0_master1200.jpg",
"large": "https://i.pximg.net/c/600x1200_90/img-master/img/2022/06/20/20/49/28/99187184_p0_master1200.jpg",
"original": "https://i.pximg.net/img-original/img/2022/06/20/20/49/28/99187184_p0.png"
}
},
{
"image_urls": {
"square_medium": "https://i.pximg.net/c/360x360_70/img-master/img/2022/06/20/20/49/28/99187184_p1_square1200.jpg",
"medium": "https://i.pximg.net/c/540x540_70/img-master/img/2022/06/20/20/49/28/99187184_p1_master1200.jpg",
"large": "https://i.pximg.net/c/600x1200_90/img-master/img/2022/06/20/20/49/28/99187184_p1_master1200.jpg",
"original": "https://i.pximg.net/img-original/img/2022/06/20/20/49/28/99187184_p1.png"
}
}
],
"total_view": 219,
"total_bookmarks": 28,
"is_bookmarked": false,
"visible": true,
"is_muted": false
},
{
"id": 99187068,
"title": "ビキニアリスちゃん",
"type": "illust",
"image_urls": {
"square_medium": "https://i.pximg.net/c/360x360_70/img-master/img/2022/06/20/20/45/01/99187068_p0_square1200.jpg",
"medium": "https://i.pximg.net/c/540x540_70/img-master/img/2022/06/20/20/45/01/99187068_p0_master1200.jpg",
"large": "https://i.pximg.net/c/600x1200_90/img-master/img/2022/06/20/20/45/01/99187068_p0_master1200.jpg"
},
"caption": "もうすぐ夏ですね～",
"restrict": 0,
"user": {
"id": 82810629,
"name": "iwasi",
"account": "user_pnru4885",
"profile_image_urls": {
"medium": "https://i.pximg.net/user-profile/img/2022/06/16/14/59/18/22886041_6fb741cbe82d8066258a3b812583b8e5_170.jpg"
},
"is_followed": false
},
"tags": [
{
"name": "東方",
"translated_name": null
},
{
"name": "水着",
"translated_name": null
},
{
"name": "ビキニ",
"translated_name": null
},
{
"name": "アリス・マーガトロイド",
"translated_name": null
},
{
"name": "アリス",
"translated_name": null
}
],
"tools": [
],
"create_date": "2022-06-20T20:45:01+09:00",
"page_count": 1,
"width": 708,
"height": 1000,
"sanity_level": 4,
"x_restrict": 0,
"series": null,
"meta_single_page": {
"original_image_url": "https://i.pximg.net/img-original/img/2022/06/20/20/45/01/99187068_p0.jpg"
},
"meta_pages": [
],
"total_view": 12,
"total_bookmarks": 3,
"is_bookmarked": false,
"visible": true,
"is_muted": false
},
{
"id": 99187042,
"title": "ゆゆ",
"type": "illust",
"image_urls": {
"square_medium": "https://i.pximg.net/c/360x360_70/img-master/img/2022/06/20/20/43/46/99187042_p0_square1200.jpg",
"medium": "https://i.pximg.net/c/540x540_70/img-master/img/2022/06/20/20/43/46/99187042_p0_master1200.jpg",
"large": "https://i.pximg.net/c/600x1200_90/img-master/img/2022/06/20/20/43/46/99187042_p0_master1200.jpg"
},
"caption": "",
"restrict": 0,
"user": {
"id": 70169966,
"name": "ぽ",
"account": "user_anhk5528",
"profile_image_urls": {
"medium": "https://i.pximg.net/user-profile/img/2022/03/01/00/32/35/22308363_216ac3b1b492a9e3cff2fd0448bbc7b4_170.jpg"
},
"is_followed": false
},
"tags": [
{
"name": "東方",
"translated_name": null
},
{
"name": "八雲紫",
"translated_name": null
},
{
"name": "西行寺幽々子",
"translated_name": null
},
{
"name": "女の子",
"translated_name": null
},
{
"name": "東方Project",
"translated_name": null
}
],
"tools": [
],
"create_date": "2022-06-20T20:43:46+09:00",
"page_count": 2,
"width": 1288,
"height": 1323,
"sanity_level": 4,
"x_restrict": 0,
"series": null,
"meta_single_page": {
},
"meta_pages": [
{
"image_urls": {
"square_medium": "https://i.pximg.net/c/360x360_70/img-master/img/2022/06/20/20/43/46/99187042_p0_square1200.jpg",
"medium": "https://i.pximg.net/c/540x540_70/img-master/img/2022/06/20/20/43/46/99187042_p0_master1200.jpg",
"large": "https://i.pximg.net/c/600x1200_90/img-master/img/2022/06/20/20/43/46/99187042_p0_master1200.jpg",
"original": "https://i.pximg.net/img-original/img/2022/06/20/20/43/46/99187042_p0.jpg"
}
},
{
"image_urls": {
"square_medium": "https://i.pximg.net/c/360x360_70/img-master/img/2022/06/20/20/43/46/99187042_p1_square1200.jpg",
"medium": "https://i.pximg.net/c/540x540_70/img-master/img/2022/06/20/20/43/46/99187042_p1_master1200.jpg",
"large": "https://i.pximg.net/c/600x1200_90/img-master/img/2022/06/20/20/43/46/99187042_p1_master1200.jpg",
"original": "https://i.pximg.net/img-original/img/2022/06/20/20/43/46/99187042_p1.jpg"
}
}
],
"total_view": 17,
"total_bookmarks": 3,
"is_bookmarked": false,
"visible": true,
"is_muted": false
},
{
"id": 99187033,
"title": "『東方』古明地こいし",
"type": "illust",
"image_urls": {
"square_medium": "https://i.pximg.net/c/360x360_70/img-master/img/2022/06/20/20/43/21/99187033_p0_square1200.jpg",
"medium": "https://i.pximg.net/c/540x540_70/img-master/img/2022/06/20/20/43/21/99187033_p0_master1200.jpg",
"large": "https://i.pximg.net/c/600x1200_90/img-master/img/2022/06/20/20/43/21/99187033_p0_master1200.jpg"
},
"caption": "こいしちゃん！",
"restrict": 0,
"user": {
"id": 82806402,
"name": "100",
"account": "100_02",
"profile_image_urls": {
"medium": "https://i.pximg.net/user-profile/img/2022/06/16/11/43/55/22885461_fca8ebe56d1e8bce9269386a1b23966b_170.jpg"
},
"is_followed": false
},
"tags": [
{
"name": "東方地霊殿",
"translated_name": null
},
{
"name": "少女",
"translated_name": null
},
{
"name": "二次創作",
"translated_name": null
},
{
"name": "イラスト",
"translated_name": null
},
{
"name": "古明地こいし",
"translated_name": null
},
{
"name": "東方Project",
"translated_name": null
},
{
"name": "東方",
"translated_name": null
}
],
"tools": [
],
"create_date": "2022-06-20T20:43:21+09:00",
"page_count": 1,
"width": 1704,
"height": 1767,
"sanity_level": 2,
"x_restrict": 0,
"series": null,
"meta_single_page": {
"original_image_url": "https://i.pximg.net/img-original/img/2022/06/20/20/43/21/99187033_p0.jpg"
},
"meta_pages": [
],
"total_view": 9,
"total_bookmarks": 2,
"is_bookmarked": false,
"visible": true,
"is_muted": false
},
{
"id": 99187030,
"title": "Hatate's leisure time",
"type": "illust",
"image_urls": {
"square_medium": "https://i.pximg.net/c/360x360_70/img-master/img/2022/06/20/20/43/19/99187030_p0_square1200.jpg",
"medium": "https://i.pximg.net/c/540x540_70/img-master/img/2022/06/20/20/43/19/99187030_p0_master1200.jpg",
"large": "https://i.pximg.net/c/600x1200_90/img-master/img/2022/06/20/20/43/19/99187030_p0_master1200.jpg"
},
"caption": "",
"restrict": 0,
"user": {
"id": 12990233,
"name": "主廚(Chef)",
"account": "joanna890508",
"profile_image_urls": {
"medium": "https://i.pximg.net/user-profile/img/2022/03/19/13/50/24/22406091_d5feb78730d70e82b655a20d06b99823_170.jpg"
},
"is_followed": false
},
"tags": [
{
"name": "姫海棠はたて",
"translated_name": null
},
{
"name": "東方Project",
"translated_name": null
}
],
"tools": [
],
"create_date": "2022-06-20T20:43:19+09:00",
"page_count": 1,
"width": 1414,
"height": 1000,
"sanity_level": 2,
"x_restrict": 0,
"series": null,
"meta_single_page": {
"original_image_url": "https://i.pximg.net/img-original/img/2022/06/20/20/43/19/99187030_p0.jpg"
},
"meta_pages": [
],
"total_view": 10,
"total_bookmarks": 3,
"is_bookmarked": false,
"visible": true,
"is_muted": false
},
{
"id": 99186976,
"title": "東方 幻想郷のA級スナイパー",
"type": "illust",
"image_urls": {
"square_medium": "https://i.pximg.net/c/360x360_70/img-master/img/2022/06/20/20/41/01/99186976_p0_square1200.jpg",
"medium": "https://i.pximg.net/c/540x540_70/img-master/img/2022/06/20/20/41/01/99186976_p0_master1200.jpg",
"large": "https://i.pximg.net/c/600x1200_90/img-master/img/2022/06/20/20/41/01/99186976_p0_master1200.jpg"
},
"caption": "東方mmdイラスト等を模写させていただきまし<br /><br />A級スナイパーと言えば<br />やはりこの人かな…と思いまして<br />ちょいと描いてみました<br /><br />お暇でしたら見てやってください",
"restrict": 0,
"user": {
"id": 26826729,
"name": "にしやん",
"account": "user_uggw4322",
"profile_image_urls": {
"medium": "https://i.pximg.net/user-profile/img/2019/12/28/22/15/31/16763057_6687b73d52a48db055d2dc91cbd7d39b_170.jpg"
},
"is_followed": false
},
"tags": [
{
"name": "東方",
"translated_name": null
},
{
"name": "博麗神社",
"translated_name": null
},
{
"name": "鈴仙優曇院イナバ",
"translated_name": null
},
{
"name": "デューク東郷",
"translated_name": null
},
{
"name": "ゴルゴ13",
"translated_name": null
}
],
"tools": [
],
"create_date": "2022-06-20T20:41:01+09:00",
"page_count": 1,
"width": 1536,
"height": 2048,
"sanity_level": 2,
"x_restrict": 0,
"series": null,
"meta_single_page": {
"original_image_url": "https://i.pximg.net/img-original/img/2022/06/20/20/41/01/99186976_p0.jpg"
},
"meta_pages": [
],
"total_view": 4,
"total_bookmarks": 2,
"is_bookmarked": false,
"visible": true,
"is_muted": false
},
{
"id": 99186955,
"title": "ぴっちり見下ろしひじりん",
"type": "illust",
"image_urls": {
"square_medium": "https://i.pximg.net/c/360x360_70/img-master/img/2022/06/20/20/40/02/99186955_p0_square1200.jpg",
"medium": "https://i.pximg.net/c/540x540_70/img-master/img/2022/06/20/20/40/02/99186955_p0_master1200.jpg",
"large": "https://i.pximg.net/c/600x1200_90/img-master/img/2022/06/20/20/40/02/99186955_p0_master1200.jpg"
},
"caption": "ちょっと蒸れが気になってほしい<br /><br />高画質+全裸差分【FANBOX】<br /><a href="https://www.fanbox.cc/@mammypoko/posts/4011575" target="_blank">https://www.fanbox.cc/@mammypoko/posts/4011575</a>",
"restrict": 0,
"user": {
"id": 864187,
"name": "ぽこ",
"account": "mammypoko",
"profile_image_urls": {
"medium": "https://i.pximg.net/user-profile/img/2020/07/07/13/01/24/18948980_d32f1341c26b95814380be8b88ffeefb_170.png"
},
"is_followed": false
},
"tags": [
{
"name": "東方",
"translated_name": null
},
{
"name": "聖白蓮",
"translated_name": null
},
{
"name": "ぴっちりスーツ",
"translated_name": null
}
],
"tools": [
],
"create_date": "2022-06-20T20:40:02+09:00",
"page_count": 1,
"width": 868,
"height": 1228,
"sanity_level": 6,
"x_restrict": 0,
"series": null,
"meta_single_page": {
"original_image_url": "https://i.pximg.net/img-original/img/2022/06/20/20/40/02/99186955_p0.png"
},
"meta_pages": [
],
"total_view": 125,
"total_bookmarks": 54,
"is_bookmarked": false,
"visible": true,
"is_muted": false
},
{
"id": 99186946,
"title": "モップは武器じゃありません妹様！！",
"type": "illust",
"image_urls": {
"square_medium": "https://i.pximg.net/c/360x360_70/img-master/img/2022/06/20/20/39/42/99186946_p0_square1200.jpg",
"medium": "https://i.pximg.net/c/540x540_70/img-master/img/2022/06/20/20/39/42/99186946_p0_master1200.jpg",
"large": "https://i.pximg.net/c/600x1200_90/img-master/img/2022/06/20/20/39/42/99186946_p0_master1200.jpg"
},
"caption": "咲夜「使い方間違ってますよ！！！」",
"restrict": 0,
"user": {
"id": 24603574,
"name": "宮古さん",
"account": "user_uetn8324",
"profile_image_urls": {
"medium": "https://i.pximg.net/user-profile/img/2020/10/23/06/17/28/19552071_f587657872cdab0bf02b8dffa6114ed3_170.jpg"
},
"is_followed": false
},
"tags": [
{
"name": "東方",
"translated_name": null
},
{
"name": "フランドール・スカーレット",
"translated_name": null
},
{
"name": "メイド服",
"translated_name": null
}
],
"tools": [
],
"create_date": "2022-06-20T20:39:42+09:00",
"page_count": 1,
"width": 4032,
"height": 3024,
"sanity_level": 2,
"x_restrict": 0,
"series": null,
"meta_single_page": {
"original_image_url": "https://i.pximg.net/img-original/img/2022/06/20/20/39/42/99186946_p0.jpg"
},
"meta_pages": [
],
"total_view": 6,
"total_bookmarks": 3,
"is_bookmarked": false,
"visible": true,
"is_muted": false
},
{
"id": 99186747,
"title": "魂魄妖夢",
"type": "illust",
"image_urls": {
"square_medium": "https://i.pximg.net/c/360x360_70/img-master/img/2022/06/20/20/31/03/99186747_p0_square1200.jpg",
"medium": "https://i.pximg.net/c/540x540_70/img-master/img/2022/06/20/20/31/03/99186747_p0_master1200.jpg",
"large": "https://i.pximg.net/c/600x1200_90/img-master/img/2022/06/20/20/31/03/99186747_p0_master1200.jpg"
},
"caption": "参照ヨルハ二号B型画的，也算是妖梦在cos了？",
"restrict": 0,
"user": {
"id": 15655239,
"name": "机智小明",
"account": "821048347",
"profile_image_urls": {
"medium": "https://i.pximg.net/user-profile/img/2020/04/26/12/13/50/18415887_f2a65572a8a3b7a2afac36bb6d6ebe0d_170.png"
},
"is_followed": false
},
"tags": [
{
"name": "東方",
"translated_name": null
},
{
"name": "東方Project",
"translated_name": null
},
{
"name": "魂魄妖夢",
"translated_name": null
},
{
"name": "ヨルハ二号B型",
"translated_name": null
},
{
"name": "コスプレ",
"translated_name": null
}
],
"tools": [
],
"create_date": "2022-06-20T20:31:03+09:00",
"page_count": 1,
"width": 2480,
"height": 3508,
"sanity_level": 2,
"x_restrict": 0,
"series": null,
"meta_single_page": {
"original_image_url": "https://i.pximg.net/img-original/img/2022/06/20/20/31/03/99186747_p0.jpg"
},
"meta_pages": [
],
"total_view": 20,
"total_bookmarks": 4,
"is_bookmarked": false,
"visible": true,
"is_muted": false
},
{
"id": 99186694,
"title": "大ちゃん「私が憧れたチルノちゃんは…そんなこと言わない！」",
"type": "illust",
"image_urls": {
"square_medium": "https://i.pximg.net/c/360x360_70/img-master/img/2022/06/20/20/29/02/99186694_p0_square1200.jpg",
"medium": "https://i.pximg.net/c/540x540_70/img-master/img/2022/06/20/20/29/02/99186694_p0_master1200.jpg",
"large": "https://i.pximg.net/c/600x1200_90/img-master/img/2022/06/20/20/29/02/99186694_p0_master1200.jpg"
},
"caption": "チルノ「あぁ…」<br />ギュゥ…<br />大ちゃん「もう大丈夫だね。早く行ってあげて！…」<br />チルノ「ッ！！…ぁ…はあああああぁああああああ！！！！！！」<br />チルノは猛スピードで飛び立った。振り返らずに…<br /><br />このお話です↓<br /><strong><a href="pixiv://novels/17348972">novel/17348972</a></strong>",
"restrict": 0,
"user": {
"id": 29643592,
"name": "拳パンチ！",
"account": "user_yxst2877",
"profile_image_urls": {
"medium": "https://i.pximg.net/user-profile/img/2020/03/22/01/12/23/18158007_6cdb7e1252811f2599f4eefeaf1d1237_170.jpg"
},
"is_followed": false
},
"tags": [
{
"name": "東方Project",
"translated_name": null
},
{
"name": "大妖精",
"translated_name": null
},
{
"name": "チルノ",
"translated_name": null
}
],
"tools": [
"MediBang Paint"
],
"create_date": "2022-06-20T20:29:02+09:00",
"page_count": 2,
"width": 7015,
"height": 4960,
"sanity_level": 2,
"x_restrict": 0,
"series": null,
"meta_single_page": {
},
"meta_pages": [
{
"image_urls": {
"square_medium": "https://i.pximg.net/c/360x360_70/img-master/img/2022/06/20/20/29/02/99186694_p0_square1200.jpg",
"medium": "https://i.pximg.net/c/540x540_70/img-master/img/2022/06/20/20/29/02/99186694_p0_master1200.jpg",
"large": "https://i.pximg.net/c/600x1200_90/img-master/img/2022/06/20/20/29/02/99186694_p0_master1200.jpg",
"original": "https://i.pximg.net/img-original/img/2022/06/20/20/29/02/99186694_p0.jpg"
}
},
{
"image_urls": {
"square_medium": "https://i.pximg.net/c/360x360_70/img-master/img/2022/06/20/20/29/02/99186694_p1_square1200.jpg",
"medium": "https://i.pximg.net/c/540x540_70/img-master/img/2022/06/20/20/29/02/99186694_p1_master1200.jpg",
"large": "https://i.pximg.net/c/600x1200_90/img-master/img/2022/06/20/20/29/02/99186694_p1_master1200.jpg",
"original": "https://i.pximg.net/img-original/img/2022/06/20/20/29/02/99186694_p1.jpg"
}
}
],
"total_view": 6,
"total_bookmarks": 2,
"is_bookmarked": false,
"visible": true,
"is_muted": false
}
]

I kind don’t know value for staticData since I just used an example function code I found from a template and modified it. I think the value is global

[MutedJam]

Hm, when testing this on my end the Function node let’s all 10 items pass:

So I suspect the problem lies with the static data of your workflow used in your Function code.

So to avoid trouble here and to simplify debugging (which is very painful with static workflow data as it won’t work when manually executing your workflow), you might want to consider alternative approach for your deduplication. I’ve shared a possible approach using Grist (a spreadsheet/database) combination here:

Monitoring RSS feeds with Grist and Telegram: lot of duplicates

Hey @dot, welcome to the community! I am sorry to hear you’re having trouble here. I am not super familiar with Grist, but I think we can nevertheless get this to work. Assuming you are storing the URLs of all feed items in Grist I think you can avail of the Merge’s node Remove Key Matches mode. This allows you to only look at items not (yet) present in your Grist table and should help you avoid duplicates. A workflow could look like so: I’ve tested this and it seems …

The example was built for an RSS feed of course, but you should be able to adjust it to your needs. You wouldn’t need to worry about the loop and probably use the ID field from your API data to avoid duplicates.

[Ruriko]

Have you tested it in production cause that’s when it doesn’t work properly. Testing with the Editor UI works fine. Use the api since the json is probably outdated by now (https://api.adoreanime.com/api/pixiv/?type=search&word=%E6%9D%B1%E6%96%B9&page=1)

[MutedJam]

Hey @Ruriko, this suggests the problem lies indeed with your static workflow data (which is ignored when manually executing a workflow). So in order to reproduce this your specific static data would be required.

But really, with this causing so much of a headache you probably want to consider an alternative approach and avoid using static workflow data here (as suggested above).