Warrior’s Guide to Embedded Debugging – Part I

Properties-for-debug5272

There is nothing more frustrating than having a problem and feeling helpless while some oracle on the other end of a nebulous contact form decides your fate. In the ideal world you would be able to solve your own problems without ever needing outside help, right? Our goal at NetBurner is to make the product in such a way that you rarely need support; not because we don’t want to help you, but to get you farther, faster. We still have a way to go and that’s why we take pride in the quality of our support, blog posts and community forums to fill those remaining gaps.

Given that, there are some common things that you can do to help yourself without waiting for us, or if you’re just a do-it-yourselfer like me. Over and over we find our support engineers end up on a common path when helping to resolve issues. This article is an attempt to empower you to get on that path. If you can solve your problem yourself it’s often a quicker, more satisfying solution for you. And the quicker you get your project to market, the better.

The Basics:

Turning a hex address into a code location.

Starting with some basics. A lot of the tools and techniques described in this document will generate hex addresses. Turning these addresses into code locations is a key skill. When you build a NetBurner project it creates several files, two of which are: a MyProject_APP.s19 file to load on the board AND a MyProject.elf file, which you have probably been ignoring.

Here’s the pitfall. This ELF file holds all the symbol and location information for your project. IT IS VITALLY IMPORTANT that the version of ELF file you use for turning hex addresses into code matches the APP file you have loaded on the board. So, if you’re going to do debugging, you must use the ELF file that was built at the same time as the app file.

  • Here are three tools that will turn hex addresses into code locations.
    1. Winaddr2line – Located in c:nburnpcbin
    2. M68k-elf-addr2line – Located c:nburngcc-m68kbin
    3. SmartTrap – Located in C:nburnincludes

Winaddr2line

On a PC, just open a command window (cmd). Then at the prompt enter “Winaddr2line”. You’ll see the window shown below open.

After you start winaddr2line, you load the ELF file. Then you put the hex addresses you are interested in the Address edit box and hit Decode.

This shows you the code addresses that correspond to the addresses you entered.

TIP: If winaddr2line is not working for you try running it as administrator. To run as an administrator: Hit “Windows+R” to open the Run box. Type “cmd.exe” to open the command window. After typing the command, hit “Ctrl+Shift+Enter” to run it with admin privileges. Simply hitting “Enter” runs the command as a normal user.

M68k-elf-addr2line

For those of you NOT running a Windows OS you will have to use the command line utility M68k-elf-addr2line. It is in C:nburngcc-m68kbin. To use it go to the command line and enter this command:

m68k-elf-addr2line -a -f --demangle -e crashdummy.elf 0x4000ad18 0x4000c48a 0x40027928

You’ll get a response like so:

0x4000ad18
OS_TASK_DLY_OBJ::Wait(os_tcb volatile&, unsigned char, unsigned long)
C:nburnsystemucos.cpp:752
0x4000c48a
OSFifoPend(OS_FIFO*, unsigned short)
C:nburnincludeucos.h:703
0x40027928
TopOfStackKillfunction()
C:nburnsystemucosmcfc.cpp:45

The general format is:

  • Address
  • Functional called
  • Line of code

The command below (m68k-elf-objdump) is what generates the disassembly, which matches the C++ code the corresponding assembly code and the assembly code’s associated addresses. To generate a disassembly file listing the code enter the follow command:

m68k-elf-objdump -d -S crashdummy.elf > crashdummy.lst

If I then open the file, I can find the hex code address I’m looking for…

   inline OS_FIFO_EL *OSFifoPend( OS_FIFO *pFifo, WORD timeout )
      { volatile OS_FIFO_EL * ret; pFifo->Pend(&ret, timeout); return (OS_FIFO_EL *)ret; }
4000c472:   2039 8000 4b70  movel 80004b70 
<ETHER_STK_SECT_END>,%d0
4000c478:   0280 0000 ffff  andil #65535,%d0
4000c47e:   2f00            movel %d0,%sp@-
4000c480:   2f02            movel %d2,%sp@-
4000c482:   4879 8000 4e84  pea 80004e84 <ipRxFifo>
4000c488:   4e93            jsr %a3@
4000c48a:   246e fffc       moveal %fp@(-4),%a2

So, if we have a hex address target for where our code is we then know how to turn that into a code address. How do we go about getting this hex address to begin with?

SmartTrap

When you code “traps” or “crashes” the default monitor trap messages are not very useful. SmartTrap gives you a much more useful output. Usage is simple. When the system traps it outputs debug messages over the console.

smarttrap.h file is included in the default NetBurner install directory C:nburnincludes. In your code turn it on by adding the following in the file where your UserMain() function is located (main.cpp in the examples we provide):

// Put this at the top of the file
#include <smarttrap.h>  

// Add this in UserMain() after the stack has been initialized
// with InitializeStack(), init(), or initWithWeb()
EnableSmartTraps();

Now when a trap is generated SmartTrap will generate an output on the console port that looks like:

-------------------Trap information-----------------------------
Exception Frame/A7 =800028a0
Trap Vector        =Debug interrupt (12)
Format             =04
Status register SR =2014
Fault Status       =00
Faulted PC         =4001a3c0

-------------------Register information-------------------------
A0=40030fe1 A1=40030cb4 A2=4000ba40 A3=4000a288
A4=40006e3c A5=000000a5 A6=800028cc A7=800028a0
D0=00000000 D1=00000054 D2=000000d2 D3=000000d3
D4=000000d4 D5=000000d5 D6=000000d6 D7=000000d7 
SR=2014 PC=4001a3c0
-------------------RTOS information-----------------------------
The OSTCBCur current task control block = 80000670
This looks like a valid TCB
The current running task is: Main#32
-------------------Task information-----------------------------
Task    | State    |Wait| Call Stack
Idle#3f|Ready     |    |40009f4a,4002790c,0
Main#32|Running   |    |4001a3c0,4002790c,0
TCPD#28|Semaphore |0257|4000a906,4001407e,4002790c,0
IP#27|Fifo      |0009|4000acfc,4000c46e,4002790c,0
Enet#26|Fifo      |0024|4000acfc,4002c3cc,4002790c,0
HTTP#2d|Semaphore |0013|4000a906,400170ce,40015778,4002790c,0

-------------------End of Trap Diagnostics----------------------

The trap report shows you the next instruction that was supposed to be executed AFTER the offending instruction (Highlighted in Yellow above). It also shows you the call stack (Highlighted in Green above) for the task that was running when the trap happened. Look for the task entry that says “running”.

Lastly, it will tell you why it trapped (highlighted in Red). In this case it says, “debug interrupt”. On NetBurner’s MCF5441X based platforms the system does a debug trap for null pointers. In the other platforms a null pointer will give you an access error.

TaskScan

TaskScan is a tool that will tell you what all the tasks in your system are doing. Using it is pretty much a two-step process. It is included in the NetBurner tools installed on your computer when you first set up your device.

In your code turn it on by adding the taskmon.h file which is part of a default NetBurner install in C:nburninclude:

#include <taskmon.h>
*
*
// Add this in UserMain() after the stack has been initialized
// with InitializeStack() 
 EnableTaskMonitor();/*Enable the Task scan utility */

Then launch the TaskScan utility. You can do this via the search bar in Windows or open Run (Window + R) > taskscan.exe > Enter. The application window will then open.

Select your matching ELF file via “Browse” and find the device. Hit “scan”.

Pro Tip: if you don’t have the ELF file with you, TaskScan will still work, but will only show hex addresses.

Using UDP Terminal to get messages

Often one uses printf debugging, it lets you print out messages to tell you what your code is thinking. This is a tried and true debugging method that is as old as coding itself. There is just one catch… your serial cord is too short, or your serial console terminal is actually used by your end application and you have no access to it. Never fear, there is a solution!

We can use the NetBurner syslog utility to printf to the syslog UDP port and to monitor it over the local network segment. Just like the others, this tools is located in C:nburninclude .

// Include this in whatever files have functions
// that you want to log
#include <syslog.h>

// This log the data using this function
SysLog(“Use this just like printf”);

Once you’re sending stuff to syslog (UDP broadcast on port 514), you need a method to see the data. I recommend that you use our UDPTerminal.exe utility and set the local port to 514 in the Local (listening) Port field in the application and press Close when done. If you can’t find do the following Run (Window Button + R) > udpterminal.exe. The application window is shown below.

StackChecking

On all NetBurner releases after 2.8 we have enabled stack checking. We’ll want to use it. This will automatically generate a trap when your code exceeds its allocated stack space.

To enable stack checking with the NBEclipse IDE Desktop Application:

First open the app. Open c:nburnincludepredef.h and uncomment:

/ #define UCOS_STACKOVERFLOW (1) /.

Then in the NBEclipse IDE application right click on “Project” and select “Properties”. Under “C/C++ Build”, click “Settings”. Under “GNU C/C++ Compiler” -> “Debugging”, select the checkbox “Enable stack checking”. See image below.

Rebuild the system files by clicking “NBEclipse”-> “Rebuild” modified system files.

Add the following code to your application during the initialization code: EnableOSStackProtector();

Or for the command line:

Modify makefile and add EXTRACXXFLAGS = -fstack-check

Rebuild the system files by typing make cleanall.

Add the following code to your application during the initialization code:

EnableOSStackProtector();

If you’d like more on stack debugging here’s one of our posts on debugging your stack memory overflows.

Part II Preview

Next Month we will continue this article with the following details:

  • The Network dies now what?
    • What exactly died?
    • ARP, Ping, TCCP, buffercount
  • How to hunt down a wild memory overwrite. AKA “who stepped on my variable”
    • The write trap
    • How to get information out of the system no matter how haunted it is
  • Task info from the bottom of the world
    • Debug messages from the bottom of the world
    • Debugging with a NON Maskable interrupt

As always please join in the conversation and leave a comment, suggestion or question below or in our forum!

UPDATE: Find Part II of this guide here!

Share this post

Subscribe to our Newsletter

Get monthly updates from our Learn Blog with the latest in IoT and Embedded technology news, trends, tutorial and best practices. Or just opt in for product change notifications.

Leave a Reply
Click to access the login or register cheese