Conversation
supportHi, After updating to v1.65.225, I’ve noticed some things changed in search results after having to re-index our whole database. The most worrying part is that some posts aren’t showing up in results at all. Here’s one example: https://macmagazine.com.br/post/2023/12/06/apple-tv-recebe-5-indicacoes-aos-spirit-awards-com-shrinking-the-changeling-e-mais/ Here’s the test for its ID: https://d.pr/i/nLWK9U And a test search for “Spirit Awards”: https://d.pr/i/1DQofL Besides that, I’ve also noticed that the search results’ order has changed a bit from the previous to the new version. Did it change the results’ weights or anything else I can change to make it more or less how it was before? Thank you!
Hi @rfischmann Thank you for this report. In fact, we changed a lot in the algorithm in the new version 1.65, so despite deep testing, shortcomings are not excluded. I would advise you to start by opening the publication with ID 984018 for editing via Edit Post. And just save it. This forces the update of the publication index. After that, try the search again (in Sandbox you can). After that, write to me to see if the publication has appeared in the index. This step is for verification purposes, of course, I don’t want you to search for all non-indexed publications and manually index them. I need this to understand what’s going on. I would also ask you to indicate your PHP version, the amount of script memory from php.ini and the current allowed time_limit. You can view this information in Tools/Website Health. Thank you!
Hi, @epsiloncool ! I’ve edited post ID 984018 and yes, now it shows up on top: https://d.pr/i/20lJe0 I have no idea at all how many other posts would be affected, of course. We’re running under PHP 8.2.18, 512MB memory limit, 300 seconds time limit. Cheers!
Every Sunday, I save a weekly backup of our MySQL database and it’s always growing up by a few MBs. This time, it fell from 928MB to 679MB, and I’m sure it’s related to WP Fast Total Search’s latest update. Its index database is now either very optimized or smaller, or it really didn’t recreate a full index of our posts.
Hi @rfischmann Actually, the new indexing rule removed “revisions” from index. Yes, it makes the index smaller. You can check how much “revision” records were removed from the index on the “Indexing Engine Settings / Indexing Rules”, there is a message like “We also found N records that are not subject to any rule (they will have empty data in the index).” You can compare how much records were removed from the index and compare with backup size change. According to missed posts I would like to ask you to find some more posts that are not searchable (but have to be searchable). If you find them, I would ask you to check some database records then (or ask you to make an Anydesk session for me so I can check some things). It’s really important, because your case can be either single case or repeatable problem. Yesterday I made some more checks for indexing and still was unable to repeat your problem. Thanks.
This, @epsiloncool ? https://d.pr/i/FRfv0f It does make total sense not to include revisions from the index, good idea! That post that wasn’t correctly indexed was identified by us simply because we were looking for it specifically. I haven’t noticed other examples up until now, but I’ll run some tests and see if I can identify any others. As for the “new” search results’ order, look at this example query: https://d.pr/i/OLWydu Before the update, it would show “Os 5 artigos mais lidos no MacMagazine: de 12 a 19 de maio” first, then “Os 5 artigos mais lidos no MacMagazine: de 5 a 12 de maio” second, which both have the exact query in their title AND are the two latest/most recent posts. The first result there, “Retrospectiva 2009: os dez posts mais acessados no MacMagazine durante o ano”, doesn’t even have “mais lidos” in its title — which seems very weird to me.
Another example that I don’t understand the search results’ order at all. Here is our podcast’s tag: https://macmagazine.com.br/categoria/podcast/ If you run a search for “MacMagazine no Ar”, it will bring results in what seems to be a total random order. I’d expect at least the latest podcasts to be on top of the results.
It’s really hard to find a specific post that might have not been indexed out of nowhere. One idea: where can I see how many posts has WP Fast Total Search indexed? Then, I can compare that number to the number of published posts we have.
The total posts (wp_posts records) processed and indexed is displayed here But to say which posts has any index data and which are not, we need to make some DB queries I think. Let me compose one and I will send you so we can make fast check
That number (307815 right now, in our case) doesn’t make much sense to me. We have 82,677 published posts…
Hi @rfischmann Actually, WP and plugins store almost everything in the wp_posts table. Not only pages and posts but a lot of different records too. For example menu items, service information, custom post types etc etc WPFTS tries to index all that info, fortunately most of those additional records are short and does not give any sensible index size increment. Okay Well, could we try to make a fast check of your database? I have constructed the MySQL query for this, and would ask you to run it in your phpMyAdmin to detect those posts that were indexed wrongly or not indexed. select * from (select p.ID post_id, wi.id windex_id, p.post_title, p.post_type, p.post_status, wlog. status , wd_t.n wdt_n, dd_t.n_total wdt_total, dd_t.n_nw wdt_nw, length(p.post_title) title_len, (length(p.post_title) / wd_t.n) av_t, wd_c.n wdc_n, dd_c.n_total wdc_total, dd_c.n_nw wdc_nw, length(p.post_content) content_len, (length(p.post_content) / wd_c.n) av_c, (if(isnull(wi.id),128,0) + if(wlog. status != 3,64,0) + if((dd_t.n_nw != dd_t.n_total) or (dd_t.n_total != wd_t.n) or (dd_c.n_nw != dd_c.n_total) or (dd_c.n_total != wd_c.n), 32, 0) + if(((length(p.post_title) > 0) and (wd_t.n = 0)) or ((length(p.post_content) > 0) and (wd_c.n = 0)), 16, 0) + if(((isnull(wd_t.n) or (wd_t.n < 1)) and (isnull(wd_c.n) or (wd_c.n < 1)) and (p.post_type in ("post", "page"))), 8, 0)) warn_level from wp_posts p left join wpftsi_index wi on p.ID = wi.tid and wi.tsrc = "wp_posts" left join wpftsi_docs wd_t on wd_t.token = "post_title" and wd_t.index_id = wi.id left join wpftsi_docs wd_c on wd_c.token = "post_content" and wd_c.index_id = wi.id left join wpftsi_ilog wlog on wlog.index_id = wi.id left join ( select v.did, sum(if(isnull(w.word),0,1)) n_nw, count(*) n_total from wpftsi_vectors v left join wpftsi_words w on w.id = v.wid group by v.did ) dd_t on dd_t.did = wd_t.id left join ( select v.did, sum(if(isnull(w.word),0,1)) n_nw, count(*) n_total from wpftsi_vectors v left join wpftsi_words w on w.id = v.wid group by v.did ) dd_c on dd_c.did = wd_c.id ) tt where tt.warn_level > 0 order by tt.warn_level desc limit 1000 This MySQL looks scary, but it does simple things: it finds some key data values in WPFTS index and in case something suspicious, it will show the post record in the result with extended data. The main column to check is “warn_level”. The value there means how worse the data we found. 128 means “post does not have any index record at all” 64 means “indexing was broken because of internal error, status not equal to 3” 32 means “calculated number of words in the document does not equal to actually stored number in index” 16 means “post title or post content has some text data, but number of indexed words is 0” We mainly interested for records with warn_level greater than 16. Sometimes (often) wp_posts records may have warn_level = 16 because title has only special characters (those are not indexed normally) or content have only images (those not indexed too). If you found some posts with warn_level = 16, you can check them by post_id in WP Admin to see whether they have indexable text info or not. Ideally if you can send me CSV or screenshot of this MySQL query result. P.S. You may need to change “wp_posts” value in the query to actual name of your wp_posts DB table (sometimes wp_ prefix may be different). Thanks.
Understood about the wp_posts table, @epsiloncool , thanks! The MySQL query took almost 2 minutes to run and it seems to have generated up to the 1,000 results’ limit. I think there are lots of “oembed_cache” lines that are useless, I’m not sure. Here’s the CSV file: https://d.pr/f/oNV1lK
Great, I see nothing too bad, but we can check what happen with some posts. For example in the row 9 we can see post_id = 56355. According to CSV data, this post has big content size (4301 characters), but NO indexed words. Could you open any Post in Edit Post mode, please, and then replace post_id in the URL to 56355 so you can see Edit Post page for this post. (“Criado para explorar o coprocessador M7 de iPhones 5s….”) What can you say about it’s post content? Why it’s not indexed? In case it contains only images this it OK. But in case it has some text, it’s a problem. Let me know, thanks!
Got it. That’s really weird, @epsiloncool . That post_id is actually an image, it seems? https://d.pr/i/7NuAv1 The “Criado para explorar o coprocessador M7 de iPhones 5s, Nike+ Move já está na App Store” post’s ID is actually 419985, as far as I checked directly in its page: view-source: https://macmagazine.com.br/post/2013/11/10/criado-para-explorar-o-coprocessador-m7-de-iphones-5s-nike-move-ja-esta-na-app-store/ I’m a bit confused.
Yes, sorry. I said you windex_id, not post_id. Actual post_id was 419985. So as I can see on the post page it has a lot of texts, but no words in the index. That’s bad. Do you see post_content in Sandbox tester for ID = 419985 ?
No content at all: https://d.pr/i/ZKzSQG I opened that post in the editor and what I’ve noticed is that, as it’s from 2013, it still uses WordPress’ old/classic editor: https://d.pr/i/RWUjLJ I made a small edit, first without converting to blocks, and that seems to be enough to index it: https://d.pr/i/QMLz6Y I then converted it to blocks, didn’t seem to make much difference to its indexation: https://d.pr/i/sGoxMC
Okay, so we located the problem, that’s great. It seems WPFTS does not extract post content correctly for OLD posts (ones created in WP Classic Editor) when Block Editor is active. Let me make some checks on my dev machine, I hope to repeat the same problem here and then I will be able to fix that. Thanks!
Awesome, thanks a lot @epsiloncool ! I hope it’s indeed that. 😊
I was thinking a bit more about it, @epsiloncool , and maybe — unfortunately — it’s not that simple. I just ran some test searches on ever older posts than that one, all made with the Classic Editor, and they’re correctly indexed by WP Fast Total Search. Also, that Spirit Awards one that wasn’t indexed at all is from December 2023, made of course with the Gutenberg editor. 🤔
Thank you for this idea, @rfischmann ! I also found some other posts in your CSV report. Could you please check if those records have some data in Sandbox tester? Could you say were these posts created with Classic editor or Block? 217165, 336337, 452385, 491877 Also I would ask you to NOT update those post, because we would need them to test fixed WPFTS version later. Thanks!
None of those 4 posts have any data in Sandbox tester besides the title, like this one: https://d.pr/i/dV6by0 However, it’s weird that, even like that, it’s showing up in the search results: https://d.pr/i/hAbBeL — I’ve also tested with the other three and all of them show up (unlike that Spirit Awards one). And yes, all 4 posts are old, created with the Classic Editor. I haven’t updated any of them.
Hi @rfischmann It’s okay that those posts are searchable by title, since the title was indexed fine. But they should be not searchable by content. I have one more question. Could you please check the options in the Indexing Engine Settings / Indexing Defaults? Do you have all the options ON?...
Hi @rfischmann Thank you for this report. In fact, we changed a lot in the algorithm in the new version 1.65, so despite deep testing, shortcomings are not excluded. I would advise you to start by opening the publication with ID 984018 for editing via Edit Post. And just save it. This forces the update of the publication index. After that, try the search again (in Sandbox you can). After that, write to me to see if the publication has appeared in the index. This step is for verification purposes, of course, I don’t want you to search for all non-indexed publications and manually index them. I need this to understand what’s going on. I would also ask you to indicate your PHP version, the amount of script memory from php.ini and the current allowed time_limit. You can view this information in Tools/Website Health. Thank you!
Hi, @epsiloncool ! I’ve edited post ID 984018 and yes, now it shows up on top: https://d.pr/i/20lJe0 I have no idea at all how many other posts would be affected, of course. We’re running under PHP 8.2.18, 512MB memory limit, 300 seconds time limit. Cheers!
Every Sunday, I save a weekly backup of our MySQL database and it’s always growing up by a few MBs. This time, it fell from 928MB to 679MB, and I’m sure it’s related to WP Fast Total Search’s latest update. Its index database is now either very optimized or smaller, or it really didn’t recreate a full index of our posts.
Hi @rfischmann Actually, the new indexing rule removed “revisions” from index. Yes, it makes the index smaller. You can check how much “revision” records were removed from the index on the “Indexing Engine Settings / Indexing Rules”, there is a message like “We also found N records that are not subject to any rule (they will have empty data in the index).” You can compare how much records were removed from the index and compare with backup size change. According to missed posts I would like to ask you to find some more posts that are not searchable (but have to be searchable). If you find them, I would ask you to check some database records then (or ask you to make an Anydesk session for me so I can check some things). It’s really important, because your case can be either single case or repeatable problem. Yesterday I made some more checks for indexing and still was unable to repeat your problem. Thanks.
This, @epsiloncool ? https://d.pr/i/FRfv0f It does make total sense not to include revisions from the index, good idea! That post that wasn’t correctly indexed was identified by us simply because we were looking for it specifically. I haven’t noticed other examples up until now, but I’ll run some tests and see if I can identify any others. As for the “new” search results’ order, look at this example query: https://d.pr/i/OLWydu Before the update, it would show “Os 5 artigos mais lidos no MacMagazine: de 12 a 19 de maio” first, then “Os 5 artigos mais lidos no MacMagazine: de 5 a 12 de maio” second, which both have the exact query in their title AND are the two latest/most recent posts. The first result there, “Retrospectiva 2009: os dez posts mais acessados no MacMagazine durante o ano”, doesn’t even have “mais lidos” in its title — which seems very weird to me.
Another example that I don’t understand the search results’ order at all. Here is our podcast’s tag: https://macmagazine.com.br/categoria/podcast/ If you run a search for “MacMagazine no Ar”, it will bring results in what seems to be a total random order. I’d expect at least the latest podcasts to be on top of the results.
It’s really hard to find a specific post that might have not been indexed out of nowhere. One idea: where can I see how many posts has WP Fast Total Search indexed? Then, I can compare that number to the number of published posts we have.
The total posts (wp_posts records) processed and indexed is displayed here But to say which posts has any index data and which are not, we need to make some DB queries I think. Let me compose one and I will send you so we can make fast check
That number (307815 right now, in our case) doesn’t make much sense to me. We have 82,677 published posts…
Hi @rfischmann Actually, WP and plugins store almost everything in the wp_posts table. Not only pages and posts but a lot of different records too. For example menu items, service information, custom post types etc etc WPFTS tries to index all that info, fortunately most of those additional records are short and does not give any sensible index size increment. Okay Well, could we try to make a fast check of your database? I have constructed the MySQL query for this, and would ask you to run it in your phpMyAdmin to detect those posts that were indexed wrongly or not indexed. select * from (select p.ID post_id, wi.id windex_id, p.post_title, p.post_type, p.post_status, wlog. status , wd_t.n wdt_n, dd_t.n_total wdt_total, dd_t.n_nw wdt_nw, length(p.post_title) title_len, (length(p.post_title) / wd_t.n) av_t, wd_c.n wdc_n, dd_c.n_total wdc_total, dd_c.n_nw wdc_nw, length(p.post_content) content_len, (length(p.post_content) / wd_c.n) av_c, (if(isnull(wi.id),128,0) + if(wlog. status != 3,64,0) + if((dd_t.n_nw != dd_t.n_total) or (dd_t.n_total != wd_t.n) or (dd_c.n_nw != dd_c.n_total) or (dd_c.n_total != wd_c.n), 32, 0) + if(((length(p.post_title) > 0) and (wd_t.n = 0)) or ((length(p.post_content) > 0) and (wd_c.n = 0)), 16, 0) + if(((isnull(wd_t.n) or (wd_t.n < 1)) and (isnull(wd_c.n) or (wd_c.n < 1)) and (p.post_type in ("post", "page"))), 8, 0)) warn_level from wp_posts p left join wpftsi_index wi on p.ID = wi.tid and wi.tsrc = "wp_posts" left join wpftsi_docs wd_t on wd_t.token = "post_title" and wd_t.index_id = wi.id left join wpftsi_docs wd_c on wd_c.token = "post_content" and wd_c.index_id = wi.id left join wpftsi_ilog wlog on wlog.index_id = wi.id left join ( select v.did, sum(if(isnull(w.word),0,1)) n_nw, count(*) n_total from wpftsi_vectors v left join wpftsi_words w on w.id = v.wid group by v.did ) dd_t on dd_t.did = wd_t.id left join ( select v.did, sum(if(isnull(w.word),0,1)) n_nw, count(*) n_total from wpftsi_vectors v left join wpftsi_words w on w.id = v.wid group by v.did ) dd_c on dd_c.did = wd_c.id ) tt where tt.warn_level > 0 order by tt.warn_level desc limit 1000 This MySQL looks scary, but it does simple things: it finds some key data values in WPFTS index and in case something suspicious, it will show the post record in the result with extended data. The main column to check is “warn_level”. The value there means how worse the data we found. 128 means “post does not have any index record at all” 64 means “indexing was broken because of internal error, status not equal to 3” 32 means “calculated number of words in the document does not equal to actually stored number in index” 16 means “post title or post content has some text data, but number of indexed words is 0” We mainly interested for records with warn_level greater than 16. Sometimes (often) wp_posts records may have warn_level = 16 because title has only special characters (those are not indexed normally) or content have only images (those not indexed too). If you found...
Understood about the wp_posts table, @epsiloncool , thanks! The MySQL query took almost 2 minutes to run and it seems to have generated up to the 1,000 results’ limit. I think there are lots of “oembed_cache” lines that are useless, I’m not sure. Here’s the CSV file: https://d.pr/f/oNV1lK
Great, I see nothing too bad, but we can check what happen with some posts. For example in the row 9 we can see post_id = 56355. According to CSV data, this post has big content size (4301 characters), but NO indexed words. Could you open any Post in Edit Post mode, please, and then replace post_id in the URL to 56355 so you can see Edit Post page for this post. (“Criado para explorar o coprocessador M7 de iPhones 5s….”) What can you say about it’s post content? Why it’s not indexed? In case it contains only images this it OK. But in case it has some text, it’s a problem. Let me know, thanks!
Got it. That’s really weird, @epsiloncool . That post_id is actually an image, it seems? https://d.pr/i/7NuAv1 The “Criado para explorar o coprocessador M7 de iPhones 5s, Nike+ Move já está na App Store” post’s ID is actually 419985, as far as I checked directly in its page: view-source: https://macmagazine.com.br/post/2013/11/10/criado-para-explorar-o-coprocessador-m7-de-iphones-5s-nike-move-ja-esta-na-app-store/ I’m a bit confused.
Yes, sorry. I said you windex_id, not post_id. Actual post_id was 419985. So as I can see on the post page it has a lot of texts, but no words in the index. That’s bad. Do you see post_content in Sandbox tester for ID = 419985 ?
No content at all: https://d.pr/i/ZKzSQG I opened that post in the editor and what I’ve noticed is that, as it’s from 2013, it still uses WordPress’ old/classic editor: https://d.pr/i/RWUjLJ I made a small edit, first without converting to blocks, and that seems to be enough to index it: https://d.pr/i/QMLz6Y I then converted it to blocks, didn’t seem to make much difference to its indexation: https://d.pr/i/sGoxMC
Okay, so we located the problem, that’s great. It seems WPFTS does not extract post content correctly for OLD posts (ones created in WP Classic Editor) when Block Editor is active. Let me make some checks on my dev machine, I hope to repeat the same problem here and then I will be able to fix that. Thanks!
Awesome, thanks a lot @epsiloncool ! I hope it’s indeed that. 😊
I was thinking a bit more about it, @epsiloncool , and maybe — unfortunately — it’s not that simple. I just ran some test searches on ever older posts than that one, all made with the Classic Editor, and they’re correctly indexed by WP Fast Total Search. Also, that Spirit Awards one that wasn’t indexed at all is from December 2023, made of course with the Gutenberg editor. 🤔
Thank you for this idea, @rfischmann ! I also found some other posts in your CSV report. Could you please check if those records have some data in Sandbox tester? Could you say were these posts created with Classic editor or Block? 217165, 336337, 452385, 491877 Also I would ask you to NOT update those post, because we would need them to test fixed WPFTS version later. Thanks!
None of those 4 posts have any data in Sandbox tester besides the title, like this one: https://d.pr/i/dV6by0 However, it’s weird that, even like that, it’s showing up in the search results: https://d.pr/i/hAbBeL — I’ve also tested with the other three and all of them show up (unlike that Spirit Awards one). And yes, all 4 posts are old, created with the Classic Editor. I haven’t updated any of them.
Hi @rfischmann It’s okay that those posts are searchable by title, since the title was indexed fine. But they should be not searchable by content. I have one more question. Could you please check the options in the Indexing Engine Settings / Indexing Defaults? Do you have all the options ON? At least first 3 options. Please let me know. Thanks!
Yeah, but once again going back to that Spirit Awards issue, wasn’t its title also indexed? https://d.pr/i/nLWK9U Here are my settings: https://d.pr/i/1cqnD0 I’ve disabled shortcodes because we have one that generates a full “widget” of apps from Apple’s App Store, and although it has a cache, when it’s expired, it needs to contact Apple’s server to update it. When it was on, the indexation would never finish because we have hundreds of posts with a few app widgets in each of them [ one example here ], so that was “blocking” it.
Nice, thank you. This fact localizes the problem even more, because for disabled “Index Shortcodes Content” WPFTS uses different piece of the code and I have changed this part of the code recently. Now I know where to dig deeper. In relation to “Spirit Awards issue” it’s still an open question. But I hope we will find an answer soon. Also I would ask you to run that MySQL query again, but I made some changes: added “oembed_cache” to the exclusions, because we really not interested in them. Also I’ve increased limit to 10000 to see more info. Also I’ve grouped records with the same warn_level and post_type to simplier find them. May be we can find something interesting then. select * from (select p.ID post_id, wi.id windex_id, p.post_title, p.post_type, p.post_status, wlog. status , wd_t.n wdt_n, dd_t.n_total wdt_total, dd_t.n_nw wdt_nw, length(p.post_title) title_len, (length(p.post_title) / wd_t.n) av_t, wd_c.n wdc_n, dd_c.n_total wdc_total, dd_c.n_nw wdc_nw, length(p.post_content) content_len, (length(p.post_content) / wd_c.n) av_c, (if(isnull(wi.id),128,0) + if(wlog. status != 3,64,0) + if((dd_t.n_nw != dd_t.n_total) or (dd_t.n_total != wd_t.n) or (dd_c.n_nw != dd_c.n_total) or (dd_c.n_total != wd_c.n), 32, 0) + if(((length(p.post_title) > 0) and (wd_t.n = 0)) or ((length(p.post_content) > 0) and (wd_c.n = 0)), 16, 0) + if(((isnull(wd_t.n) or (wd_t.n < 1)) and (isnull(wd_c.n) or (wd_c.n < 1)) and (p.post_type in ("post", "page"))), 8, 0)) warn_level from wp_posts p left join wpftsi_index wi on p.ID = wi.tid and wi.tsrc = "wp_posts" left join wpftsi_docs wd_t on wd_t.token = "post_title" and wd_t.index_id = wi.id left join wpftsi_docs wd_c on wd_c.token = "post_content" and wd_c.index_id = wi.id left join wpftsi_ilog wlog on wlog.index_id = wi.id left join ( select v.did, sum(if(isnull(w.word),0,1)) n_nw, count(*) n_total from wpftsi_vectors v left join wpftsi_words w on w.id = v.wid group by v.did ) dd_t on dd_t.did = wd_t.id left join ( select v.did, sum(if(isnull(w.word),0,1)) n_nw, count(*) n_total from wpftsi_vectors v left join wpftsi_words w on w.id = v.wid group by v.did ) dd_c on dd_c.did = wd_c.id ) tt where tt.warn_level > 0 and (tt.post_type not in ("oembed_cache")) order by tt.warn_level desc, tt.post_type desc limit 10000 Thank you.
Of course, @epsiloncool , I’m glad to help! Here’s the new CSV for that query: https://d.pr/f/Ttriua It seems that excluding “oembed_cache” made it much smaller, 170 lines total.
On an unrelated note (or not), @epsiloncool , I’ve just noticed that my error_log file is being flooded by this error: [22-May-2024 09:20:03 UTC] PHP Warning: Undefined variable $finish_ts in /home/macmagazine/public_html/wp-content/plugins/fulltext-search/includes/wpfts_core.php on line 3161 If you could please fix it, thanks a lot!
Hi @epsiloncool , I’m back with a new, different example than the ones we’ve seen so far. At this very moment, this article isn’t showing up in our search results: https://macmagazine.com.br/post/2021/06/20/applecare-protection-plan-services-e-applecare-o-que-muda/ The difference from the others is that it’s not old, it was written with Gutenberg and the worst part is that it was edited/updated just yesterday. It seems to be ok in the Sandbox Area: https://d.pr/i/UidYWD But it really doesn’t show up in searches: https://d.pr/i/dhf25d
Hi, @rfischmann It’s very good that you found this new case. I would ask you to execute MySQL query to get some information about this post and index state. Could you please execute this and then show me the result? Thanks select * from (select p.ID post_id, wi.id windex_id, p.post_title, p.post_type, p.post_status, wlog. status , wd_t.n wdt_n, dd_t.n_total wdt_total, dd_t.n_nw wdt_nw, length(p.post_title) title_len, (length(p.post_title) / wd_t.n) av_t, wd_c.n wdc_n, dd_c.n_total wdc_total, dd_c.n_nw wdc_nw, length(p.post_content) content_len, (length(p.post_content) / wd_c.n) av_c, (if(isnull(wi.id),128,0) + if(wlog. status != 3,64,0) + if((dd_t.n_nw != dd_t.n_total) or (dd_t.n_total != wd_t.n) or (dd_c.n_nw != dd_c.n_total) or (dd_c.n_total != wd_c.n), 32, 0) + if(((length(p.post_title) > 0) and (wd_t.n = 0)) or ((length(p.post_content) > 0) and (wd_c.n = 0)), 16, 0) + if(((isnull(wd_t.n) or (wd_t.n < 1)) and (isnull(wd_c.n) or (wd_c.n < 1)) and (p.post_type in ("post", "page"))), 8, 0)) warn_level from wp_posts p left join wpftsi_index wi on p.ID = wi.tid and wi.tsrc = "wp_posts" left join wpftsi_docs wd_t on wd_t.token = "post_title" and wd_t.index_id = wi.id left join wpftsi_docs wd_c on wd_c.token = "post_content" and wd_c.index_id = wi.id left join wpftsi_ilog wlog on wlog.index_id = wi.id left join ( select v.did, sum(if(isnull(w.word),0,1)) n_nw, count(*) n_total from wpftsi_vectors v left join wpftsi_words w on w.id = v.wid group by v.did ) dd_t on dd_t.did = wd_t.id left join ( select v.did, sum(if(isnull(w.word),0,1)) n_nw, count(*) n_total from wpftsi_vectors v left join wpftsi_words w on w.id = v.wid group by v.did ) dd_c on dd_c.did = wd_c.id ) tt where tt.post_id = 795909
Sure, here you go: https://d.pr/f/uKAGgq
Hi @rfischmann I believe we are as close as we can get to the cause of the problem. My initial MySQL query did not detect this situation; it is that the number of words in the document was correctly counted and stored, but in fact the words themselves were not included in the index vector lists. Below I am attaching an improved version of the query, could you run it and attach the result as before – it will identify all unindexed publications of the first type (like Spirit Awards and 795909). All I have to do is find out why exactly these publications were not processed properly. I’m already analyzing my code and trying to replicate the problem on my side. I’m wondering if there are any publications where only the titles were not indexed, but not the content, or vice versa. And also, is the problem related to the length of publications or the presence of special characters in the text (probably this can break queries if the encoding of the database tables does not match). Thank you! select * from (select p.ID post_id, wi.id windex_id, p.post_title, p.post_type, p.post_status, wlog. status , wd_t.n wdt_n, dd_t.n_total wdt_total, dd_t.n_nw wdt_nw, length(p.post_title) title_len, (length(p.post_title) / wd_t.n) av_t, wd_c.n wdc_n, dd_c.n_total wdc_total, dd_c.n_nw wdc_nw, length(p.post_content) content_len, (length(p.post_content) / wd_c.n) av_c, (if(isnull(wi.id),128,0) + if(wlog. status != 3,64,0) + if((ifnull(dd_t.n_nw, 0) != ifnull(dd_t.n_total, 0)) or (ifnull(dd_t.n_total, 0) != ifnull(wd_t.n, 0)) or (ifnull(dd_c.n_nw, 0) != ifnull(dd_c.n_total, 0)) or (ifnull(dd_c.n_total, 0) != ifnull(wd_c.n, 0)), 32, 0) + if(((length(p.post_title) > 0) and (wd_t.n = 0)) or ((length(p.post_content) > 0) and (wd_c.n = 0)), 16, 0) + if(((isnull(wd_t.n) or (wd_t.n < 1)) and (isnull(wd_c.n) or (wd_c.n < 1)) and (p.post_type in ("post", "page"))), 8, 0)) warn_level from wp_posts p left join wpftsi_index wi on p.ID = wi.tid and wi.tsrc = "wp_posts" left join wpftsi_docs wd_t on wd_t.token = "post_title" and wd_t.index_id = wi.id left join wpftsi_docs wd_c on wd_c.token = "post_content" and wd_c.index_id = wi.id left join wpftsi_ilog wlog on wlog.index_id = wi.id left join ( select v.did, sum(if(isnull(w.word),0,1)) n_nw, count(*) n_total from wpftsi_vectors v left join wpftsi_words w on w.id = v.wid group by v.did ) dd_t on dd_t.did = wd_t.id left join ( select v.did, sum(if(isnull(w.word),0,1)) n_nw, count(*) n_total from wpftsi_vectors v left join wpftsi_words w on w.id = v.wid group by v.did ) dd_c on dd_c.did = wd_c.id ) tt where tt.warn_level > 0 and (tt.post_type not in ("oembed_cache")) order by tt.warn_level desc, tt.post_type desc limit 10000
I’m glad to be able to help you figure out what’s going on, @epsiloncool ! I hope you’ll get there. 😊 Here’s the new CSV: https://d.pr/f/i9iGib
Hi @rfischmann I have updated WPFTS to the 1.67.231 but do not update your local copy this moment. Please could you make this short test in order to see difference: Run this MySQL query and record results somewhere select (select count(*) from wpftsi_index) n_index, (select count(*) from wpftsi_docs) n_docs, (select count(*) from wpftsi_words) n_words, (select count(*) from wpftsi_vectors) n_vectors, (select count(*) from wpftsi_tw) n_tw 2. Update your local copy of the WPFTS plugin to the 1.67.231 and Rebuild the Index 3. Please run the MySQL query from #1 again and again record the numbers (should be different, n_vectors should be way large) 4. Run the latest big MySQL query from my previous post and export CSV for me, please. I have a big hope that we found the problem and I fixed it. Let’s see. Thanks!!!
Of course, @epsiloncool , will follow your instructions right now.
It just finished now, @epsiloncool , about 2h15 of processing time. I don’t know what would you consider “way” larger for n_vectors — it is larger, but I don’t know if “way larger”. Here are all the CSV files you’ve requested: https://d.pr/f/677tTj
BTW, at least the Spirit Awards and AppleCare posts are correctly showing up in search results. However, the order of results still bothers me a lot I must say. I wish I could go back to how it was before v1.65. 😛
Hi @rfischmann Thanks for the reports, I am investigating them now. According to search results order – could you please check which options you selected on the pages:
Sure thing: Filter: https://d.pr/i/u9BLLC Relevance: https://d.pr/i/wome94 Sort Results: https://d.pr/i/H1vm1x I didn’t change any of that since before the v1.65 update.
Well, order settings looks good. However, they can be overriden by some other plugin. Could you please explain in words, why the order is bad now? How it should look you think? I don’t remember if I have changed ordering logic in the latest WPFTS update, so reasons may be different. What we can check also is Analytics tab. There is a list of all search queries and we can see details. Please click to the latest query and open the Explained tab. There is a item called “Query Parameters (actually used)”, please check “order” and “orderby” values. In case they are not “relevance” and “DESC”, they was overriden by some 3rd party plugin.
Just checked that and it looks good, @epsiloncool : https://d.pr/i/0rNVsN What’s bothering more, and that changed when I updated to v1.65, are some searches for posts that we post every week. For example, posts with the 5 most read articles of the week. Here’s a search for “mais lidos” (that’s “most read” in Portuguese): https://d.pr/i/Ko7UZO And the same query from the Sandbox Area: https://d.pr/i/LxfGZb As you can see, the first result doesn’t even have “mais lidos” in its title, which is very weird in my opinion. All the other results do have that, but I feel the order is very strange. I’d expect, and that’s how it was before v1.65, that the most recent post with “mais lidos” in its title would be the top result (these two: https://d.pr/i/1gV4SE ). You could say there might be some differences in the posts’ contents that would affect this, but they’re all mostly the same. I look at the results and they don’t make sense to me, the way they’re ordered. And this “mais lidos” is just one example of a few others I’ve came across.
Hi, @rfischmann Okay, for the “mais lidos” query and results I can say next. First article does not have a phrase “mail lidos” in the title, but anyway it got the biggest relevance score. I think because there are some “mais lidos” phrases in the post content. What we can do? We can lower the relevance coefficient of the post content, so existence of the phrase in the title will be MORE significant than existence of the phrase in the content. This is what “cluster weights” for. Currently you can see the “post_title” weight is 0.8 while “post_content” is 0.5. These are default values. I would propose you to set “post_title” to 0.9 and “post_content” to 0.4 and see what happen. I think it should work better. You can even set “post_content” to 0, in this case WPFTS will not try to search phrase in the post content at all. But for my humble opinion it’s not friendly for users (I bet they never use strict phrases for search that you planned/imagined). Please try and let me know how it works in your website. Thank you!
By the way I see no except text under posts in search results which can be a bit unclear for users. Most people use Google/Bing/etc and are used to seeing the context in which the requested phrase appears. You could improve the user experience by adding Smart Excerpts. Just a recommendation.
Yeah, I’d never set “post_content” to 0, sounds like a real bad idea. Making the change you’ve suggested got rid of that weird post on the top, but there are still lots of posts showing up before the latest two. Search from WordPress: https://d.pr/i/tqFPhK Search from Sandbox Area: https://d.pr/i/3UI6QR However, I do understand that, from this point on, it’s up to the plugin’s algorithm to determine each post relevance score based on its settings. I’m just saying that, before v1.65, the two latest posts on that same query would be the top two results.
Oh, I’ll look into your Smart Excerpts suggestion! Thanks! 😊
Hum, just ran a quick test with it and there is some HTML code showing up, don’t know why: https://d.pr/i/5EjLah
It looks like get_the_excerpt() call is screened by htmlspecialchars() or something that converts plaintext to HTML. You need to find it and remove, so the_excerpt() will be shown in the page directly.
In your theme (caards) there is a file template-parts/archive/content.php https://e-wm.org/i/E20240526-040307-001.png In selected code you can find the code <?php echo esc_html( csco_get_post_excerpt() ); ?> I would propose to replace it by simple <?php echo the_excerpt(); ?> That should work. Normally you can copy this file to the child theme and modify it there, so it will not be replaced on next theme update.
That does seem to work, thanks a lot @epsiloncool ! https://d.pr/i/sreaOA It just doesn’t make the search terms bold. Is that also something from our theme itself?
Hi @rfischmann I would propose to check the HTML code of excerpts, in case there is <b> tags, then your theme simple removes font-weight value from this tag so text looks not bold. It can be fixed by adding “font-weight: bold !important” for <b> style (you can do it at the WPFTS Settings / Search & Output / Display tab also. https://e-wm.org/i/E20240526-174426-001.png
Nope, there’s isn’t any tag for bold in the excerpts: https://d.pr/i/oJ3K4k
Hi @rfischmann It looks like the_excerpt() does not work, theme’s excerpt generator is used instead. Very strange, because in your previous screenshot it was working definitely (but it was screened). Could you show me the code where you set up the_excerpt() ? Thanks.
It seems the same to me. The previous screenshot was from this query: https://macmagazine.com.br/?s=apple+computer As per your instructions, I duplicated Caards’ /template-parts/archive/content.php file to my child’s folder and edited this part: https://d.pr/i/3GFkPa
Hi @rfischmann Remember, in this post you wrote: Hum, just ran a quick test with it and there is some HTML code showing up, don’t know why: https://d.pr/i/5EjLah In this post we saw the HTML, but it was screened. I think csco_get_post_excerpt() function may return the content with HTML, but it will be incorrectly terminated. What we should try is to use theme’s internal implementation without screen functionality. Let’s try to replace <?php echo the_excerpt(); ?> by <?php echo csco_get_the_excerpt(2000); ?> and let’s see what happen. Thanks.
That worked, @epsiloncool ! We’re almost there. 😊 However, the excerpts are very short as you can see in this example: https://macmagazine.com.br/?s=adobe+photoshop I’ve already changed from 300 to 2000, here, but it didn’t make any difference: https://d.pr/i/hwUslQ Any idea why? Thank you!