The First Message Box
Let's start our unpacked program with OllyDbg, run it, input eight A's into the Name and Key 1 field and press CHECK STAGE 1. What happens is that a warning message is displayed saying that the key is invalid as is presented in the picture below:
When reverse engineering we must always keep a bigger picture in mind - if we would follow each instruction step by step trying to decipher what each instruction does, we would pretty soon get lost. Instead, it's way better to have an abstract overview of what's happening in the higher layer of the program. We must think in terms of a higher programming language like C/C++ instead of a lower programming language like ASM to grasp what the program is doing in a timely fashion. When we have a basic understanding what a certain piece of a program is doing, then we can start looking at the instruction in more detail to really understand what's going on. We must also remember that we can only change the course of the program by our input arguments, which are: Name, Key 1 and Key 2.
So, first we must ask ourselves the right questions: What should happen when the right Name and Key 1 are entered into the input boxes. Most probably, the input field for Key 2 should be activated and no warning message should be produced. A notification message box about the right name and key can be displayed, which indicates that we've inputed the right values. So in order to achieve that, we must first get around the warning box saying Invalid key value.
In order to identify why the warning message is displayed we must backtrace the program execution from the point where the warning message is displayed. We could reverse engineer the whole program's code to really understand it, at which point it wouldn't be very difficult to find the right spot where the warning message is displayed. But we mustn't do that, since it's very time consuming and doesn't really make any sense.
A better way is to find a string "Invalid key value." in the program. This is exactly the warning displayed when we click on the CHECK STAGE 1. So, if we're able to find a point in the program where that warning message box is displayed, we can also determine the reason why the message box was called.
To do that, we must first right-click on the CPU View in OllyDbg and click on: Search For - All referenced text strings. When we do that the program should be analyzed for all text strings used in the program. Upon completion, the following Text Strings are displayed:
We can see our warning message "Invalid key value." being referenced at addresses 0x00401 D 9 B and 0x00401 BF 2. There are also other strings being referenced that are of greater importance to us, since they display other warning messages and also a success message. Those referenced strings are presented here:
Text strings referenced in main:.text, item 1 Address=00401724 Disassembly=push main.00541970 Text string=ASCII "Congratulations !!! Go on !!!" Text strings referenced in main:.text, item 2 Address=00401744 Disassembly=push main.00541990 Text string=ASCII "Incorrect Name/Key pair. Try again." Text strings referenced in main:.text, item 3 Address=004017AA Disassembly=mov esi,main.005419B8 Text string=ASCII "ESETNOD32@ESETNOD32@ESETNOD32@ESETNOD32@ESETNOD32@ESETNOD32@ESETNOD32@ESETNOD32@" Text strings referenced in main:.text, item 4 Address=00401BF2 Disassembly=push main.00541A0C Text string=ASCII "Invalid key value." Text strings referenced in main:.text, item 5 Address=00401D8A Disassembly=push main.00541A20 Text string=ASCII "Congratulations !!! You are done !!!" Text strings referenced in main:.text, item 6 Address=00401D91 Disassembly=push main.00541A48 Text string=ASCII "Incorrect Key 2. Try again." Text strings referenced in main:.text, item 7 Address=00401D9B Disassembly=push main.00541A0C Text string=ASCII "Invalid key value."
The best way to pin-point the exact location where our warning message is called is to take a look at the CPU instructions at address 0x00401 D 9 B and set a breakpoint there. To do that we must right-click on the CPU View and select Go To - Expression and enter our desired address 0x00401 D 9 B, then right-click on the CPU View again and select Breakpoint - Toggle. After that we must press on the CHECK STAGE 1 button again, when our breakpoint should be hit.
But when we click on the button nothing happens. Why? It's probably because the program is using the other referenced string when displaying a popup message box; if we remember correctly there were two text strings saying Invalid key value, at addresses 0x00401 BF 2 and 0x00401 D 9 B. To check whether this is true, we need to set a breakpoint on the address 0x00401 BF 2 and again press the check button. This time, the breakpoint is hit, which can be seen in the picture below:
The picture also displays the prototype of a function that is used to open our message box. The function is residing at address 0x004079 F 0, which takes three parameters that we push to the stack. The first parameter is a number 0, second parameter is a number 0x30 and the last parameter is a pointer to our message string.
When the message box is closed, we jump back to the address 0x00401 B 83. From what we can see, we can be pretty confident that this piece of code only displays the warning message and then returns to the main program, so nothing here indicates that our input values could be used to change the course of a program. That is why we need to backtrace the execution of the program a little further. To do that we must open a Call Stack while program is paused at the breakpoint in previous function somewhere. To do that we must click on Menu and select View - Call stack. The picture presenting the call stack when the execution is paused at the address 0x004018 F 2 is attached below:
We need to look at the first line, where are we currently located. We can see that our current function was called from an address 0x00406237. This is where we need to set our new breakpoint and rerun the program. When we click on the Check stage 1 button, the program execution is stopped at the new breakpoint. The paused execution can be seen in the following picture:
Around the red breakpoint, we can also see the code that was used before the call instruction was called. After careful examination, we can determine that this piece of code doesn't contain anything interesting, because we can't directly change the course of a program without input arguments. So, we need to get one call higher on the call stack. To do that, we must again look at the call stack, which can be seen in the picture below, this time relevant to the breakpoint at address 0x00406237:
The current function was called from the address 0x00406444. The disassembled function situated around this address is presented in the picture below:
Hm, but this code doesn't immediately reveal anything useful, so what's going on. Maybe we missed something. Let's trace the program right before our function is called - right to the address 0x00406237. But this time don't look around this function call, but rather step into it. By doing so we won't be checking up the function stack, but down the function stack, which can also be helpful sometimes. Why is that? It's because not every jump instruction introduces a new Call Stack when viewed in a debugger. This is because the jump can be executed with the use of different instructions, among which only the call instruction actually shows up in the Call Stack. All other jmp instructions will not be shown in the Call Stack.
When we enter the function that is called at address 0x00406237, our program execution is immediately taken to an address 0x00401990, where we need to set our new breakpoint for easier access in the future. The code of the whole function residing at that address is presented here:
00401990 . 55 push ebp 00401991 . 8BEC mov ebp,esp 00401993 . 6A FF push -1 00401995 . 68 B6795100 push main.005179B6 0040199A . 64:A1 00000000 mov eax,dword ptr fs: 004019A0 . 50 push eax 004019A1 . 81EC 18050000 sub esp,518 004019A7 . A1 D0E45500 mov eax,dword ptr ds:[55E4D0] 004019AC . 33C5 xor eax,ebp 004019AE . 8945 F0 mov dword ptr ss:[ebp-10],eax 004019B1 . 53 push ebx 004019B2 . 56 push esi 004019B3 . 57 push edi 004019B4 . 50 push eax 004019B5 . 8D45 F4 lea eax,dword ptr ss:[ebp-C] 004019B8 . 64:A3 00000000 mov dword ptr fs:,eax 004019BE . 8BF1 mov esi,ecx 004019C0 . 89B5 E0FAFFFF mov dword ptr ss:[ebp-520],esi 004019C6 . E8 32E20000 call main.0040FBFD 004019CB . 33C9 xor ecx,ecx 004019CD . 85C0 test eax,eax 004019CF . 0F95C1 setne cl 004019D2 . 85C9 test ecx,ecx 004019D4 . 75 0A jnz short main.004019E0 004019D6 . 68 05400080 push 80004005 004019DB . E8 E0060000 call main.004020C0 004019E0 > 8B10 mov edx,dword ptr ds:[eax] 004019E2 . 8BC8 mov ecx,eax 004019E4 . 8B42 0C mov eax,dword ptr ds:[edx+C] 004019E7 . FFD0 call eax 004019E9 . 83C0 10 add eax,10 004019EC . 8985 ECFAFFFF mov dword ptr ss:[ebp-514],eax 004019F2 . C745 FC 000000>mov dword ptr ss:[ebp-4],0 004019F9 . E8 FFE10000 call main.0040FBFD 004019FE . 33C9 xor ecx,ecx 00401A00 . 85C0 test eax,eax 00401A02 . 0F95C1 setne cl 00401A05 . 85C9 test ecx,ecx 00401A07 . 75 0A jnz short main.00401A13 00401A09 . 68 05400080 push 80004005 00401A0E . E8 AD060000 call main.004020C0 00401A13 > 8B10 mov edx,dword ptr ds:[eax] 00401A15 . 8BC8 mov ecx,eax 00401A17 . 8B42 0C mov eax,dword ptr ds:[edx+C] 00401A1A . FFD0 call eax 00401A1C . 83C0 10 add eax,10 00401A1F . 8985 E8FAFFFF mov dword ptr ss:[ebp-518],eax 00401A25 . 8D95 ECFAFFFF lea edx,dword ptr ss:[ebp-514] 00401A2B . 8D8E 98000000 lea ecx,dword ptr ds:[esi+98] 00401A31 . 52 push edx 00401A32 . C645 FC 01 mov byte ptr ss:[ebp-4],1 00401A36 . 898D E4FAFFFF mov dword ptr ss:[ebp-51C],ecx 00401A3C . E8 2AA20000 call main.0040BC6B 00401A41 . 8D85 E8FAFFFF lea eax,dword ptr ss:[ebp-518] 00401A47 . 8D8E C4030000 lea ecx,dword ptr ds:[esi+3C4] 00401A4D . 50 push eax 00401A4E . 898D DCFAFFFF mov dword ptr ss:[ebp-524],ecx 00401A54 . E8 12A20000 call main.0040BC6B 00401A59 . 8B85 E8FAFFFF mov eax,dword ptr ss:[ebp-518] 00401A5F . BE 01000000 mov esi,1 00401A64 . 3970 FC cmp dword ptr ds:[eax-4],esi 00401A67 . 7E 15 jle short main.00401A7E 00401A69 . 8B48 F4 mov ecx,dword ptr ds:[eax-C] 00401A6C . 51 push ecx 00401A6D . 8D8D E8FAFFFF lea ecx,dword ptr ss:[ebp-518] 00401A73 . E8 D8040000 call main.00401F50 00401A78 . 8B85 E8FAFFFF mov eax,dword ptr ss:[ebp-518] 00401A7E > 50 push eax 00401A7F . 8D95 F0FAFFFF lea edx,dword ptr ss:[ebp-510] 00401A85 . 68 00040000 push 400 00401A8A . 52 push edx 00401A8B . E8 59440F00 call main.004F5EE9 00401A90 . 8B85 ECFAFFFF mov eax,dword ptr ss:[ebp-514] 00401A96 . 83C4 0C add esp,0C 00401A99 . 3970 FC cmp dword ptr ds:[eax-4],esi 00401A9C . 7E 15 jle short main.00401AB3 00401A9E . 8B40 F4 mov eax,dword ptr ds:[eax-C] 00401AA1 . 50 push eax 00401AA2 . 8D8D ECFAFFFF lea ecx,dword ptr ss:[ebp-514] 00401AA8 . E8 A3040000 call main.00401F50 00401AAD . 8B85 ECFAFFFF mov eax,dword ptr ss:[ebp-514] 00401AB3 > 50 push eax 00401AB4 . 8D8D F0FEFFFF lea ecx,dword ptr ss:[ebp-110] 00401ABA . 68 00010000 push 100 00401ABF . 51 push ecx 00401AC0 . E8 24440F00 call main.004F5EE9 00401AC5 . 8D85 F0FAFFFF lea eax,dword ptr ss:[ebp-510] 00401ACB . 83C4 0C add esp,0C 00401ACE . 8D50 01 lea edx,dword ptr ds:[eax+1] 00401AD1 > 8A08 mov cl,byte ptr ds:[eax] 00401AD3 . 40 inc eax 00401AD4 . 84C9 test cl,cl 00401AD6 .^75 F9 jnz short main.00401AD1 00401AD8 . 2BC2 sub eax,edx 00401ADA . 8BF0 mov esi,eax 00401ADC . 8D3C76 lea edi,dword ptr ds:[esi+esi*2] 00401ADF . 03FF add edi,edi 00401AE1 . C1EF 03 shr edi,3 00401AE4 . 85FF test edi,edi 00401AE6 . 0F84 02010000 je main.00401BEE 00401AEC . 83FE 3F cmp esi,3F 00401AEF . 0F86 F9000000 jbe main.00401BEE 00401AF5 . 8D56 0A lea edx,dword ptr ds:[esi+A] 00401AF8 . 52 push edx 00401AF9 . E8 04410F00 call main.004F5C02 00401AFE . 8BD8 mov ebx,eax 00401B00 . 83C4 04 add esp,4 00401B03 . 85DB test ebx,ebx 00401B05 . 74 7C je short main.00401B83 00401B07 . 8BCB mov ecx,ebx 00401B09 . 8BD6 mov edx,esi 00401B0B . 8D85 F0FAFFFF lea eax,dword ptr ss:[ebp-510] 00401B11 . E8 AAF7FFFF call main.004012C0 00401B16 . 8B85 ECFAFFFF mov eax,dword ptr ss:[ebp-514] 00401B1C . 8B40 F4 mov eax,dword ptr ds:[eax-C] 00401B1F . 57 push edi 00401B20 . 53 push ebx 00401B21 . 50 push eax 00401B22 . 8D8D F0FEFFFF lea ecx,dword ptr ss:[ebp-110] 00401B28 . E8 33FCFFFF call main.00401760 00401B2D . 53 push ebx 00401B2E . 8BF0 mov esi,eax 00401B30 . E8 61410F00 call main.004F5C96 00401B35 . 83C4 04 add esp,4 00401B38 . 85F6 test esi,esi 00401B3A . 74 47 je short main.00401B83 00401B3C . 8BB5 E0FAFFFF mov esi,dword ptr ss:[ebp-520] 00401B42 . 6A 01 push 1 00401B44 . 8D8E 94050000 lea ecx,dword ptr ds:[esi+594] 00401B4A . E8 1CC40000 call main.0040DF6B 00401B4F . 6A 01 push 1 00401B51 . 8D8E 08060000 lea ecx,dword ptr ds:[esi+608] 00401B57 . E8 0FC40000 call main.0040DF6B 00401B5C . 6A 00 push 0 00401B5E . 8D8E 50030000 lea ecx,dword ptr ds:[esi+350] 00401B64 . E8 02C40000 call main.0040DF6B 00401B69 . 8B8D E4FAFFFF mov ecx,dword ptr ss:[ebp-51C] 00401B6F . 6A 00 push 0 00401B71 . E8 F5C30000 call main.0040DF6B 00401B76 . 8B8D DCFAFFFF mov ecx,dword ptr ss:[ebp-524] 00401B7C . 6A 00 push 0 00401B7E . E8 E8C30000 call main.0040DF6B 00401B83 > C645 FC 00 mov byte ptr ss:[ebp-4],0 00401B87 . 8B85 E8FAFFFF mov eax,dword ptr ss:[ebp-518] 00401B8D . 83C0 F0 add eax,-10 00401B90 . 8D48 0C lea ecx,dword ptr ds:[eax+C] 00401B93 . 83CA FF or edx,FFFFFFFF 00401B96 . F0:0FC111 lock xadd dword ptr ds:[ecx],edx 00401B9A . 4A dec edx 00401B9B . 85D2 test edx,edx 00401B9D . 7F 0A jg short main.00401BA9 00401B9F . 8B08 mov ecx,dword ptr ds:[eax] 00401BA1 . 8B11 mov edx,dword ptr ds:[ecx] 00401BA3 . 50 push eax 00401BA4 . 8B42 04 mov eax,dword ptr ds:[edx+4] 00401BA7 . FFD0 call eax 00401BA9 > C745 FC FFFFFF>mov dword ptr ss:[ebp-4],-1 00401BB0 . 8B85 ECFAFFFF mov eax,dword ptr ss:[ebp-514] 00401BB6 . 83C0 F0 add eax,-10 00401BB9 . 8D48 0C lea ecx,dword ptr ds:[eax+C] 00401BBC . 83CA FF or edx,FFFFFFFF 00401BBF . F0:0FC111 lock xadd dword ptr ds:[ecx],edx 00401BC3 . 4A dec edx 00401BC4 . 85D2 test edx,edx 00401BC6 . 7F 0A jg short main.00401BD2 00401BC8 . 8B08 mov ecx,dword ptr ds:[eax] 00401BCA . 8B11 mov edx,dword ptr ds:[ecx] 00401BCC . 50 push eax 00401BCD . 8B42 04 mov eax,dword ptr ds:[edx+4] 00401BD0 . FFD0 call eax 00401BD2 > 8B4D F4 mov ecx,dword ptr ss:[ebp-C] 00401BD5 . 64:890D 000000>mov dword ptr fs:,ecx 00401BDC . 59 pop ecx 00401BDD . 5F pop edi 00401BDE . 5E pop esi 00401BDF . 5B pop ebx 00401BE0 . 8B4D F0 mov ecx,dword ptr ss:[ebp-10] 00401BE3 . 33CD xor ecx,ebp 00401BE5 . E8 CF3E0F00 call main.004F5AB9 00401BEA . 8BE5 mov esp,ebp 00401BEC . 5D pop ebp 00401BED . C3 retn 00401BEE > 6A 00 push 0 00401BF0 . 6A 30 push 30 00401BF2 . 68 0C1A5400 push main.00541A0C 00401BF7 . E8 F45D0000 call main.004079F0 00401BFC .^EB 85 jmp short main.00401B83
This code can be quite daunting, but we must break it down to further understand the program. First we must determine where the jump to our message box function occurs. With simple stepping through the code, we can quickly discover that the jump occurs at an address 0x00401 AEF:
00401AEF . 0F86 F9000000 jbe main.00401BEE
We can see the known address 0x00401 BEE that was the entry point of the first function we introduced, the one that is popping-up the Invalid key value message box. What's interesting is what happens immediately before the function call. We need to be looking for any instruction that can alter the program flow based on the user input. After careful observation we can determine that it's the JMP instruction right before the function call that determines if the function will be called or not. Both of the relevant instructions are presented here:
00401AEC . 83FE 3F cmp esi,3F 00401AEF . 0F86 F9000000 jbe main.00401BEE
Ok, we're comparing the value in the register esi with the constant 0x3 F. If the value in the register esi is less than or equal to the value of 0x3 F, then the jbe jump is executed and our message box is displayed. In order to not display the message box, the value in the register esi must be greater than 0x3 F. In order to set the value in the register esi, we must first determine if the value can actually be changed by the input values in fields Name and Key 1. If we look at the register value when we use the input string AAAAAAAA, we can see that it contains a value 0x08. But we can't conclude anything useful from this. We must input some other values and check if the value in the esi is changing - this is in fact true. If we enter the value AAAAAAAAAAAAAAAA in the Key 1 field and we leave Name field at value AAAAAAAA, the esi register will contain the value of 0x16. This indicates that the esi register holds the number of bytes inputted in the Key 1 field.
Thus, we have our first predicate: the number of bytes in the Key 1 field must be greater that 0x3 F, otherwise the first warning message box is displayed, which we don't want. The predicate is:
len(Key 1) > 0x3F