The First Message Box

Let's start our unpacked program with OllyDbg, run it, input eight A's into the Name and Key 1 field and press CHECK STAGE 1. What happens is that a warning message is displayed saying that the key is invalid as is presented in the picture below:

When reverse engineering we must always keep a bigger picture in mind - if we would follow each instruction step by step trying to decipher what each instruction does, we would pretty soon get lost. Instead, it's way better to have an abstract overview of what's happening in the higher layer of the program. We must think in terms of a higher programming language like C/C++ instead of a lower programming language like ASM to grasp what the program is doing in a timely fashion. When we have a basic understanding what a certain piece of a program is doing, then we can start looking at the instruction in more detail to really understand what's going on. We must also remember that we can only change the course of the program by our input arguments, which are: Name, Key 1 and Key 2.

So, first we must ask ourselves the right questions: What should happen when the right Name and Key 1 are entered into the input boxes. Most probably, the input field for Key 2 should be activated and no warning message should be produced. A notification message box about the right name and key can be displayed, which indicates that we've inputed the right values. So in order to achieve that, we must first get around the warning box saying Invalid key value.

In order to identify why the warning message is displayed we must backtrace the program execution from the point where the warning message is displayed. We could reverse engineer the whole program's code to really understand it, at which point it wouldn't be very difficult to find the right spot where the warning message is displayed. But we mustn't do that, since it's very time consuming and doesn't really make any sense.

A better way is to find a string "Invalid key value." in the program. This is exactly the warning displayed when we click on the CHECK STAGE 1. So, if we're able to find a point in the program where that warning message box is displayed, we can also determine the reason why the message box was called.

To do that, we must first right-click on the CPU View in OllyDbg and click on: Search For - All referenced text strings. When we do that the program should be analyzed for all text strings used in the program. Upon completion, the following Text Strings are displayed:

We can see our warning message "Invalid key value." being referenced at addresses 0x00401 D 9 B and 0x00401 BF 2. There are also other strings being referenced that are of greater importance to us, since they display other warning messages and also a success message. Those referenced strings are presented here:

Text strings referenced in main:.text, item 1
Address=00401724
Disassembly=push main.00541970
Text string=ASCII "Congratulations !!! Go on !!!"

Text strings referenced in main:.text, item 2
Address=00401744
Disassembly=push main.00541990
Text string=ASCII "Incorrect Name/Key pair. Try again."

Text strings referenced in main:.text, item 3
Address=004017AA
Disassembly=mov esi,main.005419B8
Text string=ASCII
"ESETNOD32@ESETNOD32@ESETNOD32@ESETNOD32@ESETNOD32@ESETNOD32@ESETNOD32@ESETNOD32@"

Text strings referenced in main:.text, item 4
Address=00401BF2
Disassembly=push main.00541A0C
Text string=ASCII "Invalid key value."

Text strings referenced in main:.text, item 5
Address=00401D8A
Disassembly=push main.00541A20
Text string=ASCII "Congratulations !!! You are done !!!"

Text strings referenced in main:.text, item 6
Address=00401D91
Disassembly=push main.00541A48
Text string=ASCII "Incorrect Key 2. Try again."

Text strings referenced in main:.text, item 7
Address=00401D9B
Disassembly=push main.00541A0C
Text string=ASCII "Invalid key value."

The best way to pin-point the exact location where our warning message is called is to take a look at the CPU instructions at address 0x00401 D 9 B and set a breakpoint there. To do that we must right-click on the CPU View and select Go To - Expression and enter our desired address 0x00401 D 9 B, then right-click on the CPU View again and select Breakpoint - Toggle. After that we must press on the CHECK STAGE 1 button again, when our breakpoint should be hit.

But when we click on the button nothing happens. Why? It's probably because the program is using the other referenced string when displaying a popup message box; if we remember correctly there were two text strings saying Invalid key value, at addresses 0x00401 BF 2 and 0x00401 D 9 B. To check whether this is true, we need to set a breakpoint on the address 0x00401 BF 2 and again press the check button. This time, the breakpoint is hit, which can be seen in the picture below:

The picture also displays the prototype of a function that is used to open our message box. The function is residing at address 0x004079 F 0, which takes three parameters that we push to the stack. The first parameter is a number 0, second parameter is a number 0x30 and the last parameter is a pointer to our message string.

When the message box is closed, we jump back to the address 0x00401 B 83. From what we can see, we can be pretty confident that this piece of code only displays the warning message and then returns to the main program, so nothing here indicates that our input values could be used to change the course of a program. That is why we need to backtrace the execution of the program a little further. To do that we must open a Call Stack while program is paused at the breakpoint in previous function somewhere. To do that we must click on Menu and select View - Call stack. The picture presenting the call stack when the execution is paused at the address 0x004018 F 2 is attached below:

We need to look at the first line, where are we currently located. We can see that our current function was called from an address 0x00406237. This is where we need to set our new breakpoint and rerun the program. When we click on the Check stage 1 button, the program execution is stopped at the new breakpoint. The paused execution can be seen in the following picture:

Around the red breakpoint, we can also see the code that was used before the call instruction was called. After careful examination, we can determine that this piece of code doesn't contain anything interesting, because we can't directly change the course of a program without input arguments. So, we need to get one call higher on the call stack. To do that, we must again look at the call stack, which can be seen in the picture below, this time relevant to the breakpoint at address 0x00406237:

The current function was called from the address 0x00406444. The disassembled function situated around this address is presented in the picture below:

Hm, but this code doesn't immediately reveal anything useful, so what's going on. Maybe we missed something. Let's trace the program right before our function is called - right to the address 0x00406237. But this time don't look around this function call, but rather step into it. By doing so we won't be checking up the function stack, but down the function stack, which can also be helpful sometimes. Why is that? It's because not every jump instruction introduces a new Call Stack when viewed in a debugger. This is because the jump can be executed with the use of different instructions, among which only the call instruction actually shows up in the Call Stack. All other jmp instructions will not be shown in the Call Stack.

When we enter the function that is called at address 0x00406237, our program execution is immediately taken to an address 0x00401990, where we need to set our new breakpoint for easier access in the future. The code of the whole function residing at that address is presented here:

00401990 . 55 push ebp
00401991 . 8BEC mov ebp,esp
00401993 . 6A FF push -1
00401995 . 68 B6795100 push main.005179B6
0040199A . 64:A1 00000000 mov eax,dword ptr fs:[0]
004019A0 . 50 push eax
004019A1 . 81EC 18050000 sub esp,518
004019A7 . A1 D0E45500 mov eax,dword ptr ds:[55E4D0]
004019AC . 33C5 xor eax,ebp
004019AE . 8945 F0 mov dword ptr ss:[ebp-10],eax
004019B1 . 53 push ebx
004019B2 . 56 push esi
004019B3 . 57 push edi
004019B4 . 50 push eax
004019B5 . 8D45 F4 lea eax,dword ptr ss:[ebp-C]
004019B8 . 64:A3 00000000 mov dword ptr fs:[0],eax
004019BE . 8BF1 mov esi,ecx
004019C0 . 89B5 E0FAFFFF mov dword ptr ss:[ebp-520],esi
004019C6 . E8 32E20000 call main.0040FBFD
004019CB . 33C9 xor ecx,ecx
004019CD . 85C0 test eax,eax
004019CF . 0F95C1 setne cl
004019D2 . 85C9 test ecx,ecx
004019D4 . 75 0A jnz short main.004019E0
004019D6 . 68 05400080 push 80004005
004019DB . E8 E0060000 call main.004020C0
004019E0 > 8B10 mov edx,dword ptr ds:[eax]
004019E2 . 8BC8 mov ecx,eax
004019E4 . 8B42 0C mov eax,dword ptr ds:[edx+C]
004019E7 . FFD0 call eax
004019E9 . 83C0 10 add eax,10
004019EC . 8985 ECFAFFFF mov dword ptr ss:[ebp-514],eax
004019F2 . C745 FC 000000>mov dword ptr ss:[ebp-4],0
004019F9 . E8 FFE10000 call main.0040FBFD
004019FE . 33C9 xor ecx,ecx
00401A00 . 85C0 test eax,eax
00401A02 . 0F95C1 setne cl
00401A05 . 85C9 test ecx,ecx
00401A07 . 75 0A jnz short main.00401A13
00401A09 . 68 05400080 push 80004005
00401A0E . E8 AD060000 call main.004020C0
00401A13 > 8B10 mov edx,dword ptr ds:[eax]
00401A15 . 8BC8 mov ecx,eax
00401A17 . 8B42 0C mov eax,dword ptr ds:[edx+C]
00401A1A . FFD0 call eax
00401A1C . 83C0 10 add eax,10
00401A1F . 8985 E8FAFFFF mov dword ptr ss:[ebp-518],eax
00401A25 . 8D95 ECFAFFFF lea edx,dword ptr ss:[ebp-514]
00401A2B . 8D8E 98000000 lea ecx,dword ptr ds:[esi+98]
00401A31 . 52 push edx
00401A32 . C645 FC 01 mov byte ptr ss:[ebp-4],1
00401A36 . 898D E4FAFFFF mov dword ptr ss:[ebp-51C],ecx
00401A3C . E8 2AA20000 call main.0040BC6B
00401A41 . 8D85 E8FAFFFF lea eax,dword ptr ss:[ebp-518]
00401A47 . 8D8E C4030000 lea ecx,dword ptr ds:[esi+3C4]
00401A4D . 50 push eax
00401A4E . 898D DCFAFFFF mov dword ptr ss:[ebp-524],ecx
00401A54 . E8 12A20000 call main.0040BC6B
00401A59 . 8B85 E8FAFFFF mov eax,dword ptr ss:[ebp-518]
00401A5F . BE 01000000 mov esi,1
00401A64 . 3970 FC cmp dword ptr ds:[eax-4],esi
00401A67 . 7E 15 jle short main.00401A7E
00401A69 . 8B48 F4 mov ecx,dword ptr ds:[eax-C]
00401A6C . 51 push ecx
00401A6D . 8D8D E8FAFFFF lea ecx,dword ptr ss:[ebp-518]
00401A73 . E8 D8040000 call main.00401F50
00401A78 . 8B85 E8FAFFFF mov eax,dword ptr ss:[ebp-518]
00401A7E > 50 push eax
00401A7F . 8D95 F0FAFFFF lea edx,dword ptr ss:[ebp-510]
00401A85 . 68 00040000 push 400
00401A8A . 52 push edx
00401A8B . E8 59440F00 call main.004F5EE9
00401A90 . 8B85 ECFAFFFF mov eax,dword ptr ss:[ebp-514]
00401A96 . 83C4 0C add esp,0C
00401A99 . 3970 FC cmp dword ptr ds:[eax-4],esi
00401A9C . 7E 15 jle short main.00401AB3
00401A9E . 8B40 F4 mov eax,dword ptr ds:[eax-C]
00401AA1 . 50 push eax
00401AA2 . 8D8D ECFAFFFF lea ecx,dword ptr ss:[ebp-514]
00401AA8 . E8 A3040000 call main.00401F50
00401AAD . 8B85 ECFAFFFF mov eax,dword ptr ss:[ebp-514]
00401AB3 > 50 push eax
00401AB4 . 8D8D F0FEFFFF lea ecx,dword ptr ss:[ebp-110]
00401ABA . 68 00010000 push 100
00401ABF . 51 push ecx
00401AC0 . E8 24440F00 call main.004F5EE9
00401AC5 . 8D85 F0FAFFFF lea eax,dword ptr ss:[ebp-510]
00401ACB . 83C4 0C add esp,0C
00401ACE . 8D50 01 lea edx,dword ptr ds:[eax+1]
00401AD1 > 8A08 mov cl,byte ptr ds:[eax]
00401AD3 . 40 inc eax
00401AD4 . 84C9 test cl,cl
00401AD6 .^75 F9 jnz short main.00401AD1
00401AD8 . 2BC2 sub eax,edx
00401ADA . 8BF0 mov esi,eax
00401ADC . 8D3C76 lea edi,dword ptr ds:[esi+esi*2]
00401ADF . 03FF add edi,edi
00401AE1 . C1EF 03 shr edi,3
00401AE4 . 85FF test edi,edi
00401AE6 . 0F84 02010000 je main.00401BEE
00401AEC . 83FE 3F cmp esi,3F
00401AEF . 0F86 F9000000 jbe main.00401BEE
00401AF5 . 8D56 0A lea edx,dword ptr ds:[esi+A]
00401AF8 . 52 push edx
00401AF9 . E8 04410F00 call main.004F5C02
00401AFE . 8BD8 mov ebx,eax
00401B00 . 83C4 04 add esp,4
00401B03 . 85DB test ebx,ebx
00401B05 . 74 7C je short main.00401B83
00401B07 . 8BCB mov ecx,ebx
00401B09 . 8BD6 mov edx,esi
00401B0B . 8D85 F0FAFFFF lea eax,dword ptr ss:[ebp-510]
00401B11 . E8 AAF7FFFF call main.004012C0
00401B16 . 8B85 ECFAFFFF mov eax,dword ptr ss:[ebp-514]
00401B1C . 8B40 F4 mov eax,dword ptr ds:[eax-C]
00401B1F . 57 push edi
00401B20 . 53 push ebx
00401B21 . 50 push eax
00401B22 . 8D8D F0FEFFFF lea ecx,dword ptr ss:[ebp-110]
00401B28 . E8 33FCFFFF call main.00401760
00401B2D . 53 push ebx
00401B2E . 8BF0 mov esi,eax
00401B30 . E8 61410F00 call main.004F5C96
00401B35 . 83C4 04 add esp,4
00401B38 . 85F6 test esi,esi
00401B3A . 74 47 je short main.00401B83
00401B3C . 8BB5 E0FAFFFF mov esi,dword ptr ss:[ebp-520]
00401B42 . 6A 01 push 1
00401B44 . 8D8E 94050000 lea ecx,dword ptr ds:[esi+594]
00401B4A . E8 1CC40000 call main.0040DF6B
00401B4F . 6A 01 push 1
00401B51 . 8D8E 08060000 lea ecx,dword ptr ds:[esi+608]
00401B57 . E8 0FC40000 call main.0040DF6B
00401B5C . 6A 00 push 0
00401B5E . 8D8E 50030000 lea ecx,dword ptr ds:[esi+350]
00401B64 . E8 02C40000 call main.0040DF6B
00401B69 . 8B8D E4FAFFFF mov ecx,dword ptr ss:[ebp-51C]
00401B6F . 6A 00 push 0
00401B71 . E8 F5C30000 call main.0040DF6B
00401B76 . 8B8D DCFAFFFF mov ecx,dword ptr ss:[ebp-524]
00401B7C . 6A 00 push 0
00401B7E . E8 E8C30000 call main.0040DF6B
00401B83 > C645 FC 00 mov byte ptr ss:[ebp-4],0
00401B87 . 8B85 E8FAFFFF mov eax,dword ptr ss:[ebp-518]
00401B8D . 83C0 F0 add eax,-10
00401B90 . 8D48 0C lea ecx,dword ptr ds:[eax+C]
00401B93 . 83CA FF or edx,FFFFFFFF
00401B96 . F0:0FC111 lock xadd dword ptr ds:[ecx],edx
00401B9A . 4A dec edx
00401B9B . 85D2 test edx,edx
00401B9D . 7F 0A jg short main.00401BA9
00401B9F . 8B08 mov ecx,dword ptr ds:[eax]
00401BA1 . 8B11 mov edx,dword ptr ds:[ecx]
00401BA3 . 50 push eax
00401BA4 . 8B42 04 mov eax,dword ptr ds:[edx+4]
00401BA7 . FFD0 call eax
00401BA9 > C745 FC FFFFFF>mov dword ptr ss:[ebp-4],-1
00401BB0 . 8B85 ECFAFFFF mov eax,dword ptr ss:[ebp-514]
00401BB6 . 83C0 F0 add eax,-10
00401BB9 . 8D48 0C lea ecx,dword ptr ds:[eax+C]
00401BBC . 83CA FF or edx,FFFFFFFF
00401BBF . F0:0FC111 lock xadd dword ptr ds:[ecx],edx
00401BC3 . 4A dec edx
00401BC4 . 85D2 test edx,edx
00401BC6 . 7F 0A jg short main.00401BD2
00401BC8 . 8B08 mov ecx,dword ptr ds:[eax]
00401BCA . 8B11 mov edx,dword ptr ds:[ecx]
00401BCC . 50 push eax
00401BCD . 8B42 04 mov eax,dword ptr ds:[edx+4]
00401BD0 . FFD0 call eax
00401BD2 > 8B4D F4 mov ecx,dword ptr ss:[ebp-C]
00401BD5 . 64:890D 000000>mov dword ptr fs:[0],ecx
00401BDC . 59 pop ecx
00401BDD . 5F pop edi
00401BDE . 5E pop esi
00401BDF . 5B pop ebx
00401BE0 . 8B4D F0 mov ecx,dword ptr ss:[ebp-10]
00401BE3 . 33CD xor ecx,ebp
00401BE5 . E8 CF3E0F00 call main.004F5AB9
00401BEA . 8BE5 mov esp,ebp
00401BEC . 5D pop ebp
00401BED . C3 retn
00401BEE > 6A 00 push 0
00401BF0 . 6A 30 push 30
00401BF2 . 68 0C1A5400 push main.00541A0C
00401BF7 . E8 F45D0000 call main.004079F0
00401BFC .^EB 85 jmp short main.00401B83

This code can be quite daunting, but we must break it down to further understand the program. First we must determine where the jump to our message box function occurs. With simple stepping through the code, we can quickly discover that the jump occurs at an address 0x00401 AEF:

00401AEF . 0F86 F9000000 jbe main.00401BEE

We can see the known address 0x00401 BEE that was the entry point of the first function we introduced, the one that is popping-up the Invalid key value message box. What's interesting is what happens immediately before the function call. We need to be looking for any instruction that can alter the program flow based on the user input. After careful observation we can determine that it's the JMP instruction right before the function call that determines if the function will be called or not. Both of the relevant instructions are presented here:

00401AEC . 83FE 3F cmp esi,3F
00401AEF . 0F86 F9000000 jbe main.00401BEE

Ok, we're comparing the value in the register esi with the constant 0x3 F. If the value in the register esi is less than or equal to the value of 0x3 F, then the jbe jump is executed and our message box is displayed. In order to not display the message box, the value in the register esi must be greater than 0x3 F. In order to set the value in the register esi, we must first determine if the value can actually be changed by the input values in fields Name and Key 1. If we look at the register value when we use the input string AAAAAAAA, we can see that it contains a value 0x08. But we can't conclude anything useful from this. We must input some other values and check if the value in the esi is changing - this is in fact true. If we enter the value AAAAAAAAAAAAAAAA in the Key 1 field and we leave Name field at value AAAAAAAA, the esi register will contain the value of 0x16. This indicates that the esi register holds the number of bytes inputted in the Key 1 field.

Thus, we have our first predicate: the number of bytes in the Key 1 field must be greater that 0x3 F, otherwise the first warning message box is displayed, which we don't want. The predicate is:

len(Key 1) > 0x3F