Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Options
Go to last post Go to first unread
Offline elasto  
#1 Posted : Wednesday, April 9, 2014 7:03:40 AM(UTC)
elasto

Joined: 8/23/2011(UTC)
Posts: 245

Thanks: 6 times
Was thanked: 12 time(s) in 11 post(s)
I have a question and thought it'd be of interest to more people than just me so thought I'd post it here ^^

To the best of my understanding, as things stand the Build Service does not directly convert some (most? any?) of the engine's core C# classes - instead substituting more highly optimized platform-specific versions (eg. C++)

Given that the engine is open-source though, what are the implications if people make changes to their own versions of the engine?

A simple example would be the Maths/Matrix/Quaternion classes: A decision was made fairly early on to optimize them, yes, but to hold off from some of the crazier optimizations like unrolling loops if it resulted in the code becoming too ugly and confusing to follow. Instead such craziness could be limited to the built versions.

But what are the implications if someone goes on to change something in their version of Maths/Matrix/Quaternion?

To start with, are the swaps by the build service done at the class level or the method level? ie. if I only so much as add an extra Maths function will my resultant build now be slower for all Maths functions?

Another set of classes I know have more optimized versions are the Entity ones.

So, to reiterate my question: if I were to add functionality to Entities in my local version of the engine - let's say reintroduce the ability for Entities to broadcast messages and subscribe to broadcasts from others - will that suddenly make all Entity processing much slower because it's using a C#->C++ conversion of my Entity class instead of the hand-optimized C++ one?

Or, let's say I don't change the Entity class itself. Let's say I create a PGEntity class that derives from Entity with my messaging layer in it, and then make Entity2D etc. derive from PGEntity instead of Entity. Will simply the fact that Entity2D now changes which class it derives from mean it can't use the hand-optimized version?

If any of that is the case, then I'd suggest (a) there should be a warning at the top of some of these classes/methods to only change if you must, (b) have the build service itself warn that there were changes in such core classes and (c) it'd be an argument for putting as many of these optimizations as possible into the C# code to reduce the impact when it occurs.

But hopefully you'll set my mind at easy and tell me I'm worrying over nothing ^^

Wanna join the discussion?! Login to your forum accountregister a new account. Or Connect via Facebook Twitter Google

Offline Benjamin  
#2 Posted : Friday, April 11, 2014 3:24:47 PM(UTC)
Benjamin

Medals: Admin

Joined: 8/20/2011(UTC)
Posts: 1,421
Location: Hannover

Thanks: 18 times
Was thanked: 97 time(s) in 92 post(s)
Actually every single line is currently converted from C# to optimized C#, to be used on any platform that supports C#.
The same optimized C# code is then used and further changed in preparation to conversion to C++, still every single line you had is there or was converted.

For native C++ things are not working the same way, so now some replacements happen and a lot of the low level classes are replaced, changed or completely removed and get an implementation that makes sense on C++, for example there is no XNA on C++, thus Graphics has a C++ specific implementation (OpenGL), same for JavaScript (WebGL).

Entities are much harder to describe because of many optimizations in place already in the C# version of the open source engine. In general entities should be empty classes that just have a list of components and tags (and IsActive to use it or disconnect it from the EntitiesRunner).

In general all Entities get converted to plain data structures in the optimized C# version, this way execution is way faster, no reflection is needed, no casting, no type conversion or complicated Get, Contains, GetInterpolated, etc. methods are necessary because all the data is right there. We call this Flatten Entities. These are then used to be converted to any other language, which is quite simple because they all just use some data. This way the data can be optimized to fit best to the target platform architecture. Obviously we did not have the time to do much here as we are still busy with the optimized C# side of things, but the goal is to make sure we do not have too many cache misses and structure the data in a nice way the CPU and GPU like. This is something other engines cannot provide automatically, the user has to do all this work himself (or use an engine where he cannot invent new things without first understanding how it all works on the low level).

Currently whenever the build service encounters something it does not understand, it will abort and throw the exception back to the user. This is pretty much the mode we work on most of the time. The compilers after the conversion can also throw warnings and errors, which are also send back to the user. In any case the user can try to fix the problem himself (if the error makes sense and there are ways around it), or he has to ask us to support that specific feature better.

In your example the PGEntity class might not be understood by the Flatten Entities conversion step (it might work already, I don't know your code and potential issues), then you will get an error message and can either try to fix it yourself, or ask us to support it better (which we might not be able to do in a short amount of time if it is a really complex issue and we have other things keeping us busy). The more people ask and need support, the more important things become and the more we will want to support it :) So if you are the only user in the world with PGEntity and everyone else is fine, and it would take 2 months to support your crazy PGEntity because it uses all kinds of strange C#6 language features, PInvokes, whatever, then we most likely are not going to support it, you will have to fix it yourself. If everyone is doing these kinds of things, we have to support it in the long run. Just adding messaging in your example is plain C# code and should work out of the box, try it out ^^

Obviously it will take some time until things are really general. This is why we are currently not so much community focused, but instead do some internal projects and contract work to earn some money and find out the basic functionality we need. At any point in time we are still releasing versions and making the features available to all users. If someone is working on something and needs our priority support, we are also helping as best as we can, the hope is of course that things just work for most guys and we do not need to do custom support much.

Hope this helps.
thanks 1 user thanked Benjamin for this useful post.
elasto on 4/11/2014(UTC)
Offline Benjamin  
#3 Posted : Friday, April 11, 2014 3:34:50 PM(UTC)
Benjamin

Medals: Admin

Joined: 8/20/2011(UTC)
Posts: 1,421
Location: Hannover

Thanks: 18 times
Was thanked: 97 time(s) in 92 post(s)
A quick note on Datatypes, like Color or Matrix: We do not replace those classes because we change them quite often and this would make the replaced classes not longer work with the new app code that requires new functionality. Instead any datatype (which is any struct with some data in it) is optimized, all the properties in all of your classes are converted in the same manner. Unrolling loops is a bit more custom and very hard to detect, so there might be some things that only work on specific method code blocks like operator * (Matrix, Matrix), but if you put the same code somewhere else, it will be optimized in the same way (highly unlikely, the matrix multiplication code is really custom).

Also not all optimizations do work and there is much more work that can be done. We are constantly profiling and finding new issues in all systems, not just the low level code, but also any high level issue or app issue. For example we only noticed recently through CreepyTowers that scene loading was slow and had some problems because it used 15 scenes and it took like 5 seconds to load them all. With one scene loading in less than half a second, we did not notice, but once we looked at the profiler and seen the horrible mess, we immediately fixed it ^^ Now scene loading takes less than 1ms for highly complex scenes, now most of the loading time (which is less than 5% from before in case of CreepyTowers) is spend actually loading files and pushing them to the GPU or into memory for later use.
Offline elasto  
#4 Posted : Saturday, April 12, 2014 4:10:40 AM(UTC)
elasto

Joined: 8/23/2011(UTC)
Posts: 245

Thanks: 6 times
Was thanked: 12 time(s) in 11 post(s)
Great answer. Good to hear that the C# engine code is optimized not at a class level or even method level but at the line (or perhaps 'logical unit') level.

A follow-up question then.

The fact you've built a 'compiler' that converts from C#->optimized-C# then optimized-C#->C++ (and other platforms) is excellent - but it naturally adds in risk. My follow-up question is about the unit testing.

Do the existing C# tests get automatically converted so that the optimized-C# is covered by passing tests, and then do they get converted again into C++ to ensure that is covered by passing tests also? If not, how is it automatically established that the build service itself hasn't introduced a bug in the newly built C#/C++ code? And what about the builds for other platforms - are they covered by any amount of automatic testing too?
Offline Benjamin  
#5 Posted : Saturday, April 12, 2014 2:24:10 PM(UTC)
Benjamin

Medals: Admin

Joined: 8/20/2011(UTC)
Posts: 1,421
Location: Hannover

Thanks: 18 times
Was thanked: 97 time(s) in 92 post(s)
Actually we convert mostly expressions and sometimes statements (which would be one or more lines). We only converted the sample games in the past (so only the lines required to get LogoApp working, which are not many). Recently (beginning of this year) we started to convert every single line including tests (basically all ~150 projects in our main solution) and got it working on the optimized C# output.

However getting the same to run in native C++ is quite a challenge and we only have done so for a couple of tests projects (the most important ones like low level datatypes, collections, plus graphics and rendering logic). Keep in mind that many things just work differently and require even different tests, so we naturally wrote the C++ code with tests first and then tried to convert to the same code, which was more than enough work just for the plain code without tests ^^ Sadly we do not have the man power to get through this now as other things keep us busy, but we will revisit this once our main goals for the milestones have been reached.

I could not integrate our coverage tools into our build report (hopefully that will work soon, we always try to stay at 100% coverage, but not all tools agree on that), but here you can always see the latest build result with all the details:
http://deltaengine.net/a...test-server-build-report

Edited by user Saturday, April 12, 2014 2:25:25 PM(UTC)  | Reason: fixed link

Rss Feed  Atom Feed
Users browsing this topic
OceanSpiders 2.0
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

Powered by YAF.NET | YAF.NET © 2003-2023, Yet Another Forum.NET
This page was generated in 0.103 seconds.