Thread: Major Forum Changes Ahead

Thank you to the folks who came (from my Facebook announcement) to test the revived chat feature.

It went well, and surfaced one problem to fix. There is a race condition where your own post can appear to be duplicated in the chat window. These phantom double-posts only show for the sender of the message, and go away if you refresh the page. It was sort of a known issue before, but testing made it obvious that it is a lot more common than previously thought. This will get fixed tomorrow.

If you want more details about the issue, read on.
The chat page polls the server for new messages and shows them as they get posted. But if you're posting your own message, why wait until it gets to the server and back before showing it? It can be shown right away to the user that posted it without the round trip. Let's call this "fast-display" for the sake of this story. When working well, this makes the chat seem a little bit more responsive to the end user. Well, sometimes the newly-posted message makes the full round trip and shows up before this fast-display code even completes. So when the fast-display code does complete, it places a duplicate "phantom" message for the user after the real message that came back from the server. The fix will be to just get rid of this fast-display technique and solely rely on the round-trip for everything. This simplifies the code A LOT actually, and I'm happy to rip it out (including an entire PHP-file/endpoint). With this testing, it's clear the chat will feel just as (or nearly so) responsive even without this "feature".

Update (Version 1.7.1)
The phantom double-posts in chat described in the previous post have been taken care of.

The chat feature is in a stable, useful state so I'm going to take a break for a few days while I send out applications for a new job.

When I get back I will work on a handful of new features for chat, reproduced here from two posts back:

Eh, there is one small issue I'd like to take care of next. I'm not sure why, but even the old chat suffered from this problem. Sometimes loading the chat page can incorrectly tell the user they are not logged in, and the chat doesn't display. Revisiting the chat page (from the header perhaps) will try again and almost always fixes the problem (if not, try a few more times). But I should be able to fix that. I think somehow the loging check runs before the session is fully complete? I'll try to fix the issue by adding a tiny delay.

Edit: Or maybe I can check to see if the session has started yet and wait until that happens.
PHP: session_status - Manual

Edit 2: Yah there are a few ways to solve this, but looking at it closer I see I had the login check much earlier in the chat page than anywhere else which caused this race condition. I'll just do things the same way I do them everywhere else and that should solve it.

Update (Version 1.7.2)
Navigating to the chat page should no longer occasionally (and incorrectly) tell the user they are not logged in.

I refactored things so the chat page works just like all the others, which gives the session enough time to initialize before checking it. This happens simply because normally pages load and run a good bit more code before working with the session. This is code we needed anyway for chat, so I just swapped the order in which things happened.

On a related note, it has become clear a session can be requested to start, but that it might not actually exist by the time it is referenced. So where I normally start the session for all forum pages, I've added a loop that checks to make sure the session has actually started before then proceeding. This will prevent any more related race conditions in the future.

Update (Version 1.7.3)
I'm working on a minor, obscure bug with the forum when viewed with the Pale Moon browser and it is now looking like a bug in Apache.
Related threads:
sr20-forum loses CSS styling after soft refresh
Response headers empty for 304 Not Modified responses. (Paid member login required)
69580 – Stream not closed cleanly during simple 304 response.

Anyway, while working on that I've been looking at the forum logs and finally addressing some issues revealed in there. I've made a handful of minor improvements by addressing PHP notices and warnings (not many errors to speak of) as I find them. This should all be transparent to users.

While working on the above I found we had 3 forum sub-sections that were actually completely empty. They were converted over in 2017 but never had any threads in them. They were likely vestigial sections from some forum work performed a long, long time ago. So I deleted those. This is where they used to be:






Finally, I came across 132 orphaned posts while digging around in the database. I deleted these orphaned posts which were made in early 2009 and converted over in 2017 but never belonged to an actual thread (likely because the thread was deleted in mid-2009). There was no useful tech info in the posts. It was an off-topic discussion about funny videos/GIFs. And now that I type this out, I am realizing it was probably the original funny video/GIF thread that Vadim deleted by accident and could not recover, so the new version of it was posted. (That new version I've actually looked for a couple times in the past few weeks and for the life of me I can't find it.)
Edit: Oh yah, of course, it's a sticky thread in The Moshpit. I forgot we even had that section...

Technically these posts still exist in the legacy posts DB table, so one day maybe I could recover them and merge them into the new thread, but that is just not something I want to tackle right now.


I will continue to work on these minor issues in the logs a bit while I wait to see what happens with the Apache bug report.

Update (Version 1.7.4)
I'm still looking at the logs and fixing anything I see. There really isn't much left now, so I'm wrapping up this work. I've fixed a handful of notices, warnings, and errors. I've also made some of the session code better and a few other things. Really it was a nice polish job on many areas of the code.

While I was in the logs I saw some probing with URLs that are too long, no doubt by some malicious actor. This is pretty common, doesn't mean anything is wrong with the site, and is mostly noise. But the way the URLs were constructed it seems that a (non-functional) link in the member profile page to non-existent "albums" was giving them ideas. This was a placeholder for me to remember how the old albums looked, worked, etc.

But we don't have member album support right now, and if it ever comes back it'll be a while from now, and only after forum supporter accounts get fleshed out again, if that ever happens. So I've hidden the album stuff from the member profile page, which hopefully stops giving hints to malicious actors for URLs to construct.

Similarly, I've also hidden the "Product Reviews & Ratings" placeholder section. Again, it will return if it gets implemented (but this feature of the forum never took off and it probably does not need to ever come back).

While I was working on the member profile page (here is mine for an example), I also noticed there is another placeholder for a feature (currently not implemented) which was showing for all users, when it really should just be displayed for mods/admins. So I've fixed that.
I also changed the "Join Date" to display just a date (like YYYY-MM-DD) instead of a date-time (like YYYY-MM-DD HH:MM:SS) which makes more sense and is how it should be.
Lastly, I finally spent a little bit of time to implement the "Posts Per Day" calculation which I've wanted to do for a long time. I've buried this a bit here at the end, but this is a big deal (or it is to me at least). This is actually, finally, new useful information/behavior from the forum and honestly should induce a "feature" version bump to 1.8.0 but that felt a little excessive for what is, in some ways, such a tiny feature.

I am going to take a break for a bit, and then hopefully come back to work on the improvements to the new chat feature, as mentioned earlier.

Update (Version 1.7.5)
I noticed the chat feature was cutting anchor tags off of the end of URLs. It turns out the pound/hashtag/number symbol (#) which denotes an anchor tag in a URL weren't allowed at all in chat messages. And after a little bit of debugging I realized it was because I was using HTTP GET requests for the chat messages. This is all sorts of bad form, and I had a note to update chat messages to use HTTP POST in the near future (mostly to add CSRF protection to chat messages). Well, it looks like the future is now.

So, incoming chat messages are now protected with CSRF, the pound symbol (#) is no longer filtered out in general, and URLs with anchor tags no longer get truncated.

its not letting me send messages in chat now lol

logged out an back in still wont go thru

Oh... That's no good. Thank you for reporting it. Let's continue the discussion here: Can't post in chat after recent update.