TLDR; Upgrading YouTrack to 7.0 and getting a “Failed to execute refactoring for entity type: Event” memory error in the Hub upgrade? Add more physical RAM, then add some more, with maybe a little more on top. YT 7.0 upgrade process is memory hungry.
If you’ve read the previous posts on this site, scrolled through my Twitter feed, or spoken to me about issue management, continuous integration, delivery or deployment, then you’ll have probably heard me talk about the Jetbrains product, YouTrack. Heck, I even used it to ticket-manage my wedding back in 2014!
YouTrack is, in a nutshell:
Easy-to-use, fully customisable issue tracking and agile project management tool your development team will love.
(Those are Jetbrain’s words) – and they’re right. It is easy to use and your dev team will love it. Especially if you’ve made them use other management tools, especially that one beginning with a ‘J’. Ouch.
Anyhow, We’ve been running YouTrack 6.5 as a standalone JAR instance, on a linux box, for quite some time. The recent announcement of YouTrack 7.0 with it’s enhanced agile boards and other wonderful improvements meant I was quite looking forward to updating. Along comes the fateful day and having taken another backup (never hurts to have multiple backups) we took the service down and started the upgrade.
Being a standalone JAR file the upgrade was as simple as:
- Stop the YouTrack instance
- Grab the new JAR file (7.0) and put it on the linux box, in the right folder
- Rename the old JAR file (6.5) to “.old” or something.
- Restart the YouTrack instance
Simples, as a meerkat would have you believe.
Scroll back a little, to August 2015, and YouTrack introduced their new user management tool, Hub, which they began using for user auth control. Hub stores its own event log and audit trail, and this meant that when the new YouTrack instance started up, and realised it had some upgrade work to do, it had to include Hub in its plans. And here’s where the problems began.
One of the first things the upgrade script (built into the latest JAR file) will try to do is upgrade the database schema of the Hub database. What we hadn’t realised is quite how big the Hub database had become. It seems having a custom-built dashboard, using the YouTrack API, generates quite a lot of audit events.
During out first upgrade attempt we realised it was taking quite a while and started watching the logs as they were generated. We quickly learnt that Hub was being upgraded first and found that a log file called ‘hub-all.log’ was keeping track of the changes (.youtrack/logs/hub/hub-all.log) – before long this error turned up:
ERROR [sistentEntityStoreRefactorings] Failed to execute refactoring for entity type: Event java.lang.OutOfMemoryError: GC overhead limit exceeded
Again, the commercially exploited Meerkat pops up, and we simply increase the setting in the java command line to grant the service some more memory and start the YouTrack service again. Except no, same problem, same error. So we max out the memory allocation and go again. Same result.
Each time this error came up the upgrade would push on through but we’d end up with a response from the browser that contained a sad face and some awful animated background which made our eyes hurt:
(Don’t bother clicking ‘Hub Integration Settings’ – Hub is unavailable. AKA; no keyboard found, press F1 to continue)
So we start to think about bringing back the 6.5 instance, restoring the DB from a backup (once YT 7.0 has touched the DB it will refuse to launch 6.5 so a restore is your only option to downgrade – and to be fair, JetBrains does warn you of this in the upgrade release notes). Maybe we can clean up the Hub event log. Let me save you the hassle – there is no option to do anything like that. Hub 1.0, which is what YouTrack 6.5 uses, doesn’t have an option to do anything like that (and neither does Hub 2.5, which comes with YouTrack 7 – but that’s a different story)
Nope. No option to do that.
It’s at times like this that you need to bring yourself back from the command line, back from the log files, and think bigger than an SSH session. We’ve run out of memory on the box. The obvious fix, add more memory. Easy. The meerkat would be proud.
We just about doubled the amount of physical memory the server had, much more than it needs to run YouTrack on a day-to-day basis, and lo and behold, it works. The upgrade sails through and soon enough we’re presented with the lovely YT 7.0 start screen. (Watching the upgrade continue on it does eventually clean up the Hub Audit Events table – but only after it’s spent time refactoring it to the new schema!)
And it was worth it, YouTrack 7.0 kicks some ass.