FiXED: Duplicated Node IDs

[REV_BL!TZKR!EG]

While working through the whole thread I am now summarizing the problem.
Therefore let me verify/falsify a couple of assumptions and hypotheses:

Hans, what exactly is this "Remap IDs" under "Tools" in S3D? Wouldn't that help us?

If you use the remap Option a whole unit(file) will geht everywhere New IDs.
So in that case we would have to reassign everything again. Shipwaves, damage files, sounds, etc.

Remap only recreates the IDs _within_ that one selected unit you just opened.
But it is unclear what you mean by "have to reassign everything".
PM just confirmed that there are no other references that need to be reassigned - from where have you got that, Gunnar?
We are JUST NOW talking about this online - so if you could come online and talk this would really bring us forward and clear another misunderstanding.

Quote from: Grey Eminence on 20161212, 20:34:51

What pisses me off is that it seems we are trying to fix years of lazyass work from other modders who
couldn't be bothered to check if there were duplicated IDs for the ships they were creating copying others.

True! And what pisses me off even more is that we only have this assumption and no evidence that the way proposed WILL work.
I admire Blitz for taking all the efforts!!

Grey is correct. Read on why.


Quote from: Philipp_Thomsen on 20161122, 01:04:29

Reason for this is, for some reason some of these IDs are duplicated, so when the game calls for an ID already in use, game crashes.
By replacing these IDs with new IDs, we should be preventing these crashes.

This hypothesis is most apparently wrong. Read on as to why.


Quote from: HanSolo78 on 20161119, 09:22:14

The origin of these duplicated IDs is probably Bad cloning from other units. This tool appears the First time on subsim and I think not many do know it. So Most people were not aware that the new units they did have duplicated IDs with existing ones. It was always in my mind that this could be very complicated ... new units... No one ever had a look to the entire mega mods.

This assumption is partly wrong.
The "problem" of duplicated IDs (if there is one, read on!) has not come to life with the mods or mega-mods!
Reason: I have scanned GWX3 and WAC4 with the D-Finder and they both have duplicated IDs, too!
Even stock 1.4b has! And all them three do not crash.

Conclusion: The crashes do NOT have their root in the duplicated IDs, otherwise GWX3, WAC4 and stock1.4 would crash, too (which they do not)


Quote from: HanSolo78 on 20161122, 14:21:57

What is the root cause of these duplicated IDs?
Who cloned what and where?

Does WAC 4.1 also have these duplicated IDs?
Do also other mods such as GWX have these duplicated IDs?

I sense there might be a totally different problem that came much later than WAC 4.1...

The root is every ship/unit creator. Every unit is cloned from an existing unit. Then the IDs are renewed by the creators. Unfortunately only Most of IDs from the .dat files were only renewed, not all. IDs from all other ship files are more often duplicated.
WAC4.1 has also duplicated IDs, but probably not that much. Gwx I dont know. I dont have it installed. But it does not change one Thing. The problem is real and there is nothing that can be done except solving this problem.

As explained above, this problem is no problem for WAC4, GWX3 and stock.
So it cannot be the IDs that cause WAC5 to crash.

Prozedurmeister was correct when doubting it - he was also asking for proof:

duplicated IDs which probably cause the CTDs as a unit which has duplicated IDs also cause SH3 to collapse.

Is that certain?

Most obviously, following the above argumentation, this is not the case.

It is of course true that Duplicated IDs are no good, and certainly even PACK3D only copied existing IDs.

But I fail to see why GWX3 and WAC4 would not crash - they contain bunches of these duplicated IDs as well!

[HanSolo78]

Then... another question from me. Why so new ships lead to CTDs. New ships contain new different IDs, which are not in Gwx3, WAC4.1 or Stock SH3. There are totally different ships and IDs.

It is easy to create a new theory.
It is definitily prooved that duplicated IDs lead to CTDs.
So... any news theory needs proof.
What about the proof of your new theory? What else could be responsible?

Btw. Happy New Year

[Prozedurmeister]

Who has proved that, Gunnar?

[REV_BL!TZKR!EG]

It is easy to create a new theory.
What about the proof of your new theory? What else could be responsible?

Which new theory?
I am not creating a new theory at all.
I am critically reflecting what is being assumed here.

Then... another question from me. Why so new ships lead to CTDs.

Try as I did and run the Duplicate-Finder on WAC4.1
WAC4 was NEVER reported buggy or crashing. Or was it?
And you will see that WAC4 has loads of duplicate IDs, too. But it does not crash!

So what does that tell you? Judge yourself.

It is definitily prooved that duplicated IDs lead to CTDs.

Then please explain why WAC4 does NOT crash - though it has loads of duplicated IDs.

Also please do not forget to answer all the other open questions - shall I mark them with color?
I fear that you overread them...

[HanSolo78]

@PM

Me, I myself prooved that a duplicated ID in a File lead to CTDs.

@ Blitz

There is only one sentence marked as a question.
And maybe it ist mit clear what I meant with reassign. It means that all the other files like! .val, .sim and so on need a new connection to the dat file. Because the whole dat file gets new IDs with the remap operation.

[REV_BL!TZKR!EG]

Standing in the shoulders of giants, we can reach higher. That's the way to progress

...and so we did...

BACKUP FIRST!

EDiT:
Moved all these instructions to http://www.donitzeliteflotilla.com/forum/index.php?topic=2660




BUT!
Please do not blame us - we have not gotten confirmation that what we coded is wanted!
Reason: Gunnar, you have written posts but have not responded to these more important messages:Re: Problem: Duplicated Ids (Nodes)
Re: Problem: Duplicated Ids (Nodes)
Re: Problem: Duplicated Ids (Nodes)

Okay, so this is a screenshot of our newest version - both tools in one
(and checked only as far as we can know without our questions being answered!)

[Prozedurmeister]

@PM
Me, I myself prooved that a duplicated ID in a File lead to CTDs.

And how?

Have you read what Blitz wrote?
"Then please explain why WAC4 does NOT crash - though it has loads of duplicated IDs."

If what you say is true then why do WAC4, GWX3 and stock not crash - only WAC5alpha crashes!

And in what environment and what circumstances and after which action do these crashes come?
Why do you never answer all that?
Please explain that contradiction!

[Philipp Thomsen]

Here is the breakdown of how to change these manually in case someone wants to help.

1) Run the duplicate ID finder in a specific folder, like crew, sea, objetcs, etc.
2) Copy all the results into google spreadsheets or excel if you have it
3) Open the ID reverser Blitz and Proz made together with the excel list and S3D in the same window, as shown below:



1) First line contain the ID in question, ignore the second line and grab the name of the file that needs fixing from the third line. Open that file in S3D.
2) Copy the ID for that file, toss it in the converter and convert it
3) Copy the result of the conversion into the find function of S3D
4) Hit "NEW" to generate a new ID for that node, and hit CTRL+S to save.
5) Move on to the next file.
6) Profit.

[REV_BL!TZKR!EG]

Sorry, guys. Prozedurmeister and I were so discouraged by the circumstance that our questions remain STILL UNANSWERED,
that we ourselves then neither bothered about testing with the uploaded version.

So yes, we did test within our compiler-environment, and there it worked
but no, the download-version had "WRITE= OFF".

This is now the WRITE-enabled version for all of you to use. We have compared the files in hexeditor WITH THIS VERSION
and the changes are good.

[REV_BL!TZKR!EG]

Ok, the tool apparently works, the IDs are being replaced and opening the new file in S3D doesn't show any error.

However, we have a new problem. The duplicated IDs are being all replaced by a new ID and so this new ID is also a duplicate.

See, the Duplicate finder was run again and a few duplicate IDs were still found.
A lot less duplicates than before... which is weird. I'm not quite sure why.

PT, I coincidently found that you provided a screenshot - I hate that postimage does often not work properly, so I converted it to another pic-hoster:


This screenshot is a perfect example on how frustrating our work is.
A) Weeks ago we did not know nothing about how IDs were written, the format, the headers, the index.
We reverse-engineered all that - stumbling in the dark.
And we still to not know the format of the index, thus we cannot even know if we are doing good.

B) The output of Ref's Duplicate Finder is total crap - which of each duplicate pairs should we work with?
If we took path#2 (second one as duplicate), we would in larger lists only write to the same file again and again - so we fixed that.
If we took path#1 (we thought of that as unlogic), we would in this case only fix 964121 and 964261 - and 177487 would be skipped !!!
Following this unlogic nonsense I decided that we write a routine that searches the IDs WITHIN the file of path#1 and collect these offsets to fix them ALL.

C) What we here do not get is why a specific range of files has apparently the problem that you describe, this happens with...
EXACTLY 15 files in the AIR folder
EXACTLY 1 file in the CREW folder
and so on.

what is so special about these files?

Even if they can't, this already helps a lot.
If the new ids generated indeed work I'm sh3 (haven't tested) then the tool does 90% of the work, we can polish the rest by hand.

I dearly hope so. Losing too much hair on this...

[REV_BL!TZKR!EG]

We are trying to find why certain units "behave" different.
It would appear this is because the images are SUB-EMBEDDED so to say. They contain a sub-link and an EMPTY (!) node-description !!!

Look at this screenshit:

[REV_BL!TZKR!EG]

To Oliver and Peter... I just worked with your updated tool and it definitely cna save us much time!!  :-* :)
Great work from both of you! I just reworked the submarine folder from about 28000 line to about 2900 with your tool!  :)

Many thanks for this final acknowledgement.
Hearing this from you as a teacher really made my heart rejoice

There are some things that can be done  ;)

-Manual ?
-Add on Mods (different GUIs, harder AI)
-What else? Your ideas?

That is all here on this forum already!
I collected everything from our emails that is about features in this thread

Now, lets pack this together and start testing Alpha 9 for CTDs.
If it's all running smoothly, then lets start talking about modding it further.

Good point!

[itadriver]

Maybe I didnt understand the problem.
If I remap ID using S3editor (v0.9.9),  the parent ID in .cam file remain linked to .dat ID.
So I think it is not necessary to reassign.
I tried and the result is as I told above.
Let me know if I'm wrong.
I'm glad to help you.

[itadriver]

You have not to remap a single id.
Open all files (dat, cam ......) and keep them open
Now in .dat click 'tools', 'remap id', 'random', 'ok' .
Then in each orher file (cam, sim, val.....) click 'tools', 'remap id', 'ok' (DO NOT CLICK RANDOM. Click it only in dat file).
When done this, check if the new files have no duplicate id. If so repeat the procedure.
I use this metod because there is no possibility to forget any parent ID ( i think this has been the cause of missed parent id in cam).
I will send you an interior remapped so you will see :)

[itadriver]

Sorry....
I just opened the interiors files and I saw that there are duplicate id in the same file. Do not consider my precedent post.
I will remap the interior.

I need the following files (error while processing file - cant open with S3ditor)
- nss_U-Boot2_rr.cam
- nss_U-Boot7_rr.cam
- nss_U-Boot9_rr.cam

21_rr works :)