Amiga-Development

Please login or register.

Login with username, password and session length
Advanced search  

News:

Created for developers of all Amiga camps

Author Topic: hvl2wav c2pas translation failure  (Read 546 times)

0 Members and 1 Guest are viewing this topic.

magorium

  • Full Member
  • ***
  • Posts: 200
  • Programming is an art form that fights back
    • View Profile
hvl2wav c2pas translation failure
« on: February 11, 2017, 06:07:49 PM »

Since another thread already discussed the basics of C, and some 'hidden' specifics that the language confronts you with i got into a bit of a problem with regards to translating hively player to Pascal.

I've looked at my 'port' a numerous amount of times now and, i simply fail to see where i go wrong. Most probably i have interpreted that the c-code does something but in fact does something different then i was expecting. As can be read from that earlier mentioned thread, that is something i am still struggling with (even after all these years).

The wonderful results can be found here.

I would be obliged if someone with more c knowledge (and preferably a bit of Pascal as well) would be able to point me a bit into the right direction. One word of warning though as the code is a bit tedious and long winded. I've looked at the code for so long that i can't even distinguish spaces from numbers anymore  ;D

In case wondering, i used the hvl2wav sources. That way i could use the original c produced executable to generate a wav file and see what the Pascal translated code produced, compare the wav files to see if both codes produced the same -> my code goes a bit off at some of the effects (i was even unable to determine which effect(s)).

In the end i would like to add the ahi/wav back-end to be used in a player or all platforms (even though the c code can be used to do the same).

regards,
Logged

ALB42

  • Moderator
  • Newbie
  • *****
  • Posts: 22
    • View Profile
Re: hvl2wav c2pas translation failure
« Reply #1 on: February 12, 2017, 02:48:57 PM »

uhhh
I'm sure you will hate that error ;)

https://github.com/magorium/hvlpas/blob/master/replay.pas#L1717

reads

Code: [Select]
    d3 := d3 and (not $80);
    d3 := d4 and (not $80);

and the original C Code is:

Code: [Select]
    d3 &= ~0x80;
    d4 &= ~0x80;

maybe better would be:
Code: [Select]
    d3 := d3 and (not $80);
    d4 := d4 and (not $80);
but thats just a suggestion ;)
(at least for me it works this way)
awesome work btw. I like it.
Logged

magorium

  • Full Member
  • ***
  • Posts: 200
  • Programming is an art form that fights back
    • View Profile
Re: hvl2wav c2pas translation failure
« Reply #2 on: February 12, 2017, 05:26:55 PM »

First of all,thank you very much for your time on this ALB42. It is really appreciated.

uhhh
I'm sure you will hate that error ;)
Yups  :)

Quote
maybe better would be:
Code: [Select]
    d3 := d3 and (not $80);
    d4 := d4 and (not $80);
Oh dear... that is a very nasty typo which i have overlooked dozens of time by now. Thank you for your fresh pair of eyes and make me see that.

Quote
(at least for me it works this way)
awesome work btw. I like it.
Unfortunately, for the test-song i used, it made not a single bit of difference  :o That made some progress, but it seems that was not all what is wrong with my code (stupid me initially corrected the wrong source)  :-[

I've uploaded my test song (a moon light) to your ftp server. Its in pub/hvl, and the zip contains the original .hvl file and both the Pascal produced wav as well as the wav produced by the original hvl2wav program on windows.

Please don't be intimidated by the diff file :-)

Unless you are familiar with the tune(s) you probably won't spot too much difference. There are two hvl tunes with some effects that even sounds strange for the untrained ears (the names eludes me atm, i'll look them up when i get back).

There unfortunately is more then meets the eye. The asm code (relying on processor specifcs) was converted into c (relying on c compiler behavior) which i then converted into Pascal (which has it's own peculiarities).

I'm pretty sure my error lies in either a wrong interpretation of what the c compiler produces or unfamiliar/overlooked behavior from the Pascal compiler. In my experience replayers are a bit nasty to port (good).

PS: feel fee to give away the link to the archive in case it is requested, since it's your bandwidth i'm wasting.
« Last Edit: February 12, 2017, 05:58:55 PM by magorium »
Logged

ALB42

  • Moderator
  • Newbie
  • *****
  • Posts: 22
    • View Profile
Re: hvl2wav c2pas translation failure
« Reply #3 on: February 12, 2017, 09:30:38 PM »

First one is easy:

https://github.com/magorium/hvlpas/blob/master/replay.pas#L1920
Code: [Select]
FXParam := FXParam - $a0-$50;

the ordering is important!
the C Source is:
Code: [Select]
if( (FXParam -= 0xa0-0x50) < 0 ) break; // 1.6

it should be:
Code: [Select]
FXParam := FXParam - ($a0-$50);
to make sure it calculates the later one first, which it will not do on default.



The second was VERY hard, took me 2 hour to figure out whats wrong there ;-)
because its a combination of an error in the C Program and undefined behavior, a real language difference.

The root of course is the pseudorandom generator used here:

https://github.com/magorium/hvlpas/blob/master/replay.pas#L2309
Code: [Select]
voice^.vc_WNRandom := int32( int64(voice^.vc_WNRandom) + int64(2239384) ) ;    // FPC: overflow
voice^.vc_WNRandom := ((((voice^.vc_WNRandom shr 8) or (voice^.vc_WNRandom shl 24)) + 782323) xor 75) - 6735;

1. the C program is not 64 bit safe :-(, so if you compile 64 bit the pseudorandom go havoc

but this is true:

2. the behavior for Pascals "shr" and Cs ">>" is different for negative numbers
(pascal shifts as it would be unsigned value, C fills the leading places with "1", so the number stay negative)

this is need to evaluate again with 32 bit C program

so to fix that you have to change vc_WNRandom to int64 and replace the source with this one
do not forget to introduce the rShift variable.

Code: [Select]
NEED some more fixing
var
  rShift: Int32;

voice^.vc_WNRandom := Int32( int64(voice^.vc_WNRandom) + int64(2239384) ) ;    // FPC: overflow     

if voice^.vc_WNRandom < 0 then
  rShift := (voice^.vc_WNRandom shr 8) or $FF000000
else
  rShift := (voice^.vc_WNRandom shr 8);

voice^.vc_WNRandom := (((rShift or (voice^.vc_WNRandom shl 24)) + 782323) xor 75) - 6735;

pffff... what a mess ;-)
but for me it works now as the C demo does, but I'm not sure the pseudorandom generator does what it should do, but at least the Pascal program now does the same. Pseudorandom generators with shifts an XORs seldom use signed values, at least the ones I know/use.

« Last Edit: February 12, 2017, 10:11:32 PM by ALB42 »
Logged

magorium

  • Full Member
  • ***
  • Posts: 200
  • Programming is an art form that fights back
    • View Profile
Re: hvl2wav c2pas translation failure
« Reply #4 on: February 13, 2017, 06:46:58 PM »

First one is easy:
Oops  :-[

Yes, thank you for spotting this as well as for your explanation.

It's one for the hall of shame as i am familiar with how this c construct works. I seem to have missed this when i re-cleaned the sources with those from my local working dir (not branch).

fwiw: there is another one exactly like that at line 1516

Quote
The second was VERY hard, took me 2 hour to figure out whats wrong there ;-)

Hmz, i wonder if all that casting was a premonition :-)

Quote
1. the C program is not 64 bit safe :-(, so if you compile 64 bit the pseudorandom go havoc
That would propbably have been encountered and/or addressed with the (recent) jessie 64 bit conversion as well (i just noticed that it was worked upon).

Quote
2. the behavior for Pascals "shr" and Cs ">>" is different for negative numbers
(pascal shifts as it would be unsigned value, C fills the leading places with "1", so the number stay negative)
I am somewhat familiar with the differences between c and Pascal with regards to shifting. I just not had any clue that the implementation in the replayer was depending on this behaviour.

TBH i probably would not have been able to find that myself, so thank you very much for having spotted that.

Am i correct when stating that using the (new) SarLongInt intrinsic would have similar results as your manual solution ? , e.g.:

Code: [Select]
    if ( voice^.vc_Waveform = (4-1) ) then
    begin
      // AddRandomMoving
      AudioSource := AudioSource + ( voice^.vc_WNRandom and (2*$280-1) ) and (not 1);  // FPC note: check not(1)
      // GoOnRandom
      voice^.vc_WNRandom := int32( int64(voice^.vc_WNRandom) + int64(2239384) ) ;    // FPC: overflow
//      voice^.vc_WNRandom := ((((voice^.vc_WNRandom sar 8) or (voice^.vc_WNRandom shl 24)) + 782323) xor 75) - 6735;
      voice^.vc_WNRandom := ((( sarLongInt(voice^.vc_WNRandom, 8) or (voice^.vc_WNRandom shl 24)) + 782323) xor 75) - 6735;
    end;

I have not seen the results with this change with my own two eyes (i had to rely on reports from 3th party) but that seems to have been a major improvement over pre-existing wav output.

If i have to trust the reports then this would make about 75 percent of the outputted wav similar to the wav as outputted by the original hvl2wav utility and those values that are different are close to its original counterpart. There are no complete diff blocks anymore, only several values (which seems another indication something is still wrong with one of the applied effects).

I'll apply the fixes to my tree as soon as i'm able to. Please feel free to a pull request if wanted. (fwiw: i'll credit you for the fixes anyway)

Thank you very much for your input, analysis and solutions. At the least you gave me a very good motivation to try my other changes again in order to try locate where else i did go wrong. Feel free to further aid me in this though  :D

regards,
Logged

ALB42

  • Moderator
  • Newbie
  • *****
  • Posts: 22
    • View Profile
Re: hvl2wav c2pas translation failure
« Reply #5 on: February 15, 2017, 08:54:53 PM »

of thanks for that tip with sarLongInt, yes it seems it does the same... I did not know that.

yes it's still not perfectly, but we have a problem:

*"Magic Elixier of mighty bug finding" wears off*

I know, what the problem is: it is some rounding error in the basic waveforms which is rather strange in the C program, I really did get not why it happens and what is wrong there. So I give up for now.

But I can tell you how to find it.

if you insert here
https://github.com/magorium/hvlpas/blob/master/replay.pas#L513
a debug output, something like this
Code: [Select]
lowbuf^  := int8( trunc(low) );     // FPC: requires truncation
lowbuf   := lowbuf + 1;
highbuf^ := int8(  trunc(high) );    // FPC: requires truncation
highbuf  := highbuf + 1;       
if (temp = 25) and (wv=12) then
  writeln(temp, ',',wv,',',i,'-', trunc(low),':',trunc(high));   

and do the same in the C program

Code: [Select]
low  += mid * fre;
low   = clip( low );
if ((temp==25)&&(wv==12))
{
  printf("%i,%i,%i-%i:%i\n", temp,wv,i,(int8)low,(int8)high);
}

and you compare the 2 outputs (windiff or so) you will notice a difference at i>25

pascal:

Pascal______C
25,12,24--128:0
25,12,25--128:0
25,12,26--128:0
25,12,27--128:0
25,12,24--128:0
25,12,25--127:0
25,12,26--128:0
25,12,27--127:0

if you compare the double values before the conversion to integer (so before trunc/(int8)) you will notice, the both doubles seems to be exactly -128.0 even you print many digits it seems to be -128.0, why the C compiler rounds it up to -127. The C code here is again very bad (for example it calculates in double, but the used constants are all single precision), could be the difference come from there.

The visible difference you can see fits to a rounding problem... because if you make a bin diff you can see that always the lower byte of the sound word is different, but only if the high byte is very big.
At least I can not hear a difference when playing the wav. For me this would be ok to keep it like that.
« Last Edit: February 15, 2017, 09:30:57 PM by ALB42 »
Logged

magorium

  • Full Member
  • ***
  • Posts: 200
  • Programming is an art form that fights back
    • View Profile
Re: hvl2wav c2pas translation failure
« Reply #6 on: February 16, 2017, 01:49:43 AM »

of thanks for that tip with sarLongInt, yes it seems it does the same... I did not know that.
I didn't either.... it just seemed odd to me that FPC would not have an implementation for doing an arithmetic shift (and it actually officially didn't until recently  :o).

Learning something new every day  :)

Quote
*"Magic Elixier of mighty bug finding" wears off*

I had a couple of minutes to spare for a quick test that shows what that magix elixer of yours brought sofar:
Code: [Select]
"a moon light.ori.wav" and "a moon light.ori.wav" have 0 different bytes ( 100.0000% similarity )
"a moon light.ori.wav" and "a moon light.pas.wav" have 37074380 different bytes ( 1.9812% similarity )
"a moon light.ori.wav" and "a moon light.pas.fix.d3.wav" have 34541768 different bytes ( 8.6770% similarity )
"a moon light.ori.wav" and "a moon light.pas.fix.fxparam.wav" have 13649319 different bytes ( 63.9133% similarity )
"a moon light.ori.wav" and "a moon light.pas.fix.random.wav" have 2360052 different bytes ( 93.7604% similarity )

That's an improvement of about 92 percent against my initial results and much better then the earlier reported 75 percent similarity  ;D

Hmz... that makes me wonder what my contribution was to that code i've committed   ???

Quote
I know, what the problem is: it is some rounding error in the basic waveforms which is rather strange in the C program, I really did get not why it happens and what is wrong there.
I did a small readup on that and some SO answers seem to indicate that indeed there are 'better' conversion techniques, depending on the compiler (static casts etc).

Quote
But I can tell you how to find it.
Hmz that is very interesting.

I also recognized the pattern when looking at the diff files and therefor was initially suspecting GenWhiteNoise() routine to be the cause for that as that seemed the most likely candidate to me. I wasted quite some cycles on that   :'(

I could have sworn that i've tested the waveforms with running a checksum over the generated waveforms and compare c against Pascal results. I either remember this wrongly or i made a boo boo in the checksum calculation :-S

Quote
At least I can not hear a difference when playing the wav. For me this would be ok to keep it like that.
You are quite right about that.

This is just one of those things that keeps buggin' me until the ends of time, and i won't rest until i've turned every stone in order to be able to pinpoint the whats, wheres and/or whys ... i consider it a narcissistic character flaw  :D (*)

Very important lesson learned/reminded though: always ever doubt your own work as for certain you will keep making mistakes with these kind of conversion. A second set of eyes is a god-bless for such cases.

Thank for that mate !

edit:
(*) changing float64 type inside chelpers unit from double to extended seems to have done the trick.
Code: [Select]
"a moon light.ori.wav" and "a moon light.pas.fix.extended.wav" have 0 different bytes ( 100.0000% similarity )
Weird, as i figured float64 to comply to FPC's type double ? did i go wrong there ?
« Last Edit: February 16, 2017, 02:19:01 AM by magorium »
Logged