Benchmarking : Performance Matters

I'd like to think I'm a nice guy, although I can be blunt at times. When you're running 200 addons inside of a game that's already demanding on your system, performance matters. It matters a lot. I don't want Tweaker to compete with other addons out there. I don't want to say it's better than addons like Viewport and kgPanels. But the purpose of Tweaker is to allow the user to create a highly customized experience without all the bloat that comes with it. And numbers speak volumes. So it's with these thoughts in mind that I will be making direct comparisons of Tweaker's performance with the performance of various addons that do the same thing.

Keep in mind, Tweaker is more work to configure than your average addon. But I think it's important to get an idea of what you'll be gaining from that extra bit of work you do yourself.

To conduct these tests, I created a benchmarking addon. The code is rather simple.

local frame = CreateFrame("FRAME", "Benchmarker", UIParent) local elapsed = 0 local reportIn = 60 local addons = {} for i = 1,GetNumAddOns() do local name, title, notes, enabled, loadable, reason, security = GetAddOnInfo(i) if enabled or loadable then table.insert(addons, { Name = name, CPU = 0 }) end end table.sort(addons, function(a, b) return a.Name:lower() < b.Name:lower() end) local function ReportAddonUsage(heading) if not heading or type(heading) ~= "string" then if elapsed < reportIn then heading = "Startup Usage Report:" else heading = string.format("Usage Report After %d Seconds:", reportIn) end end if heading and string.len(heading) > 0 then heading = string.format("|cffA0A0FE%s|r", heading) DEFAULT_CHAT_FRAME:AddMessage(heading) end collectgarbage("collect") UpdateAddOnMemoryUsage() UpdateAddOnCPUUsage() for i = 1,table.maxn(addons) do local name = addons[i].Name local mem, totalcpu = GetAddOnMemoryUsage(name), GetAddOnCPUUsage(name) local cpu = totalcpu - addons[i].CPU addons[i].CPU = totalcpu if mem > 0 or cpu > 0 then local str = "%s - %.2f KB - %.2f ms" if elapsed > 0 then str = str.." - %.2f ms/s" end DEFAULT_CHAT_FRAME:AddMessage(string.format(str, name, mem, cpu, totalcpu/elapsed)) end end end frame:SetScript("OnUpdate", function(self, elap) elapsed = elapsed + elap if elapsed >= reportIn then ReportAddonUsage() reportIn = reportIn + 60 end end) frame:RegisterEvent("PLAYER_LOGIN") frame:SetScript("OnEvent", function(self, event) if (GetCVar("scriptProfile") or "0") ~= "1" then SetCVar("scriptProfile", "1") DEFAULT_CHAT_FRAME:AddMessage("CPU measuring was not enabled. Please reload your UI.") end ReportAddonUsage() end)

Addon	Base Startup	During Configuration	Startup After Configuration	Normal Use
Viewport	489.36 KB, 2.04 ms	567.82 KB, 207.57 ms	489.36 KB, 2.44 ms	489.36 KB, 0.01 ms
Tweaker	0.08 KB, 0.00 ms	-	16.48 KB, 0.03 ms	16.48 KB, 0.00 ms
Tweaker (Adding the Black Panel)	-	-	17.59 KB, 0.09 ms	17.59 KB, 0.00 ms
kgPanels (Adding the Black Panel)	-	-	146.23 KB, 2.97 ms	146.16 KB, 0.00 ms

Base startup is on a fresh, new install. During configuration is after opening the configuration windows and tweaking settings around like I would normally do -- nothing excessive, nothing minimal. Since Tweaker's configuration is done in the config.lua file outside of the game, this step doesn't apply for Tweaker. Startup After Configuration is the readings after a reloadui, and normal use is a rough average of letting the game continue to run for awhile after the UI is reloaded, without touching any configuration settings further.

Obviously there's not much to Tweaker at a base install. After doing the configuration, you'll see that the memory jumps quite a bit, but it's still significantly smaller than Viewport's memory footprint. Startup time is also noticeably faster. Since neither do any continual processing, there's no real CPU use after the initial startup.

Viewport does not include a black panel to cover up the area of the screen that the WorldFrame no longer covers. I've added a 3rd and 4th addon listing to demonstrate the memory and cpu requirements for this. You can see that adding a black panel onto Tweaker's configuration only adds another 1.11 KB of memory and 0.06 ms of startup time. Using another full-blown addon like kgPanels for this effect requires an additional 146 KB of memory and adds almost 3 ms onto your startup time.

Keep in mind, this is purely measuring performance. If I were to add more statistic here, I would say it took under 5 minutes to configure Viewport and kgPanels to get this effect. On the other hand, I spent about an hour or two figuring out how to achieve these effect on my own using Tweaker (this time is subjective based on an individual's familiarity with creating addons -- I was a bit rusty and Tweaker also required some tweaks to get it working). Fortunately, with the help of my viewport tutorial, it shouldn't take you nearly that long to achieve your own customized results.

Update: Tweaks to the Benchmarking script. The "heading" before each report is now blue so you can easily see the beginning of each report. The CPU measurement was changed to ms because it measures how many milliseconds the addon is doing something. CPU/s was changed to ms/s because it's how many milliseconds of every second the addon is doing something. The ms/s (former CPU/s) calculation was also fixed. It was previously dividing how many milliseconds were used in the last minute by the number of total seconds that had elapsed. It now divides the total number of milliseconds used by the total number of seconds elapsed. Please be aware that none of these changes affect the previous reports for Viewport since Viewport only responds to user action.. They inaccuracies were found while testing Cartographer, which does have some consistent CPU usage even while the user is doing nothing.