The last version of Stuffed Tracker was released in the beginning of January 2006. So it’s been 2 months without new versions which is unusual for us (we try to release a new version every month).
But there is nothing to worry about. Ivan, the main developer of Stuffed Tracker, had to take some time off to help with implementing the new interface for Factory Nova and also do some custom development work for our friends from Florida.
He is now back to work on the next version of Stuffed Tracker, but we still don’t expect it to be released at least for another month. I’ll explain why.
The time has come for us to focus on Stuffed Tracker’s perfomance. Don’t get me wrong, even before we’ve released the initial 2.0 version we’ve optimized the speed of the code and the database to the maximum. But as it appears this is still not enough.
Working with statistical data (collecting & analyzing) is one of the hardest tasks in web software development and only because the amounts of data could get huge when popular sites are tracked or you are tracking tens of thousands of keywords on Google (or more). With such volumes you can easily get millions of records in the stats database in just several days and much much more after a month or two of tracking.
Really, everyone has the same problem here. I am still waiting for my invitation to Google Analytics although I signed up for it a week or two after Google launched the service.
So, again, making the database to query a set of data which contains tens of millions of records in real time will be a challenge even if you are doing this on a dedicated server with 2 Xeon processors and 4 Gb of memory.
How can we solve the problem, we were thinking. And eventually we’ve come up with several ideas that we think should work:
1. No real time reports (as an option) — the reports will be prepared by a cron script and a user will view an already prepared report which will load fast and the loading time would not depend on the amount of statistics that you have in the database. This feature is especially challenging because of the dynamic nature of our reports constructor, but we think we know how to solve this.
2. Delayed tracking (as an option) — tracking will get rid of all of its current logic (which currently does mutliple requests in the database, thus increasing its load), it will brainlessly log every tracking request somewhere and then a cron script will process this log and will put the data in the proper database tables, apply required logic, etc.
3. Archiving of the old statistics — we are planning to make it possible to archive the old statistics in some form so that you will be able to view the archives later and delete the data that was already archived to free up the database and increase the overall performance of Stuffed Tracker. It would probably be possible to optionally do this via a shell PHP script (especially important when working with huge databases where a normal PHP script accessed through the web could timeout).
4. AJAX in the reports — to make the reports more responsive, convenient and generally to ease the pain of waiting for the slow reports in case we will still have them somewhere.
As you can probably imagine, these ideas are not fast to implement, but we are confident we need to do them before we will move forward with any other features. This will be a strong foundation for Stuffed Tracker on top of which we will be adding new features and functionality in the future.
option 2 and 3 together seem the most promising to me. Option 1 is of course also very interesting. I think archiving of analytics data is very important as most of the times data older than a year is not really interesting anymore for most clients. So this data could be exported in compressed-format to a file and removed from the database.
please keep in mind when creating functionality that has to be run by a cronjob/taskmanager that people are also using ST on windows-servers.