Debug method - 2: root cause analysis

During firmware development, if something does not work as expected, we need to throw this question first: "hardware issue or firmware issue?" This idea will help to figure out potential root causes.

• Hardware issue?

  • Verify power supply. Use a multimeter to measure each voltage test point such as 3.3V, 5V and 12V. Make sure the multimeter to DC mode and select right voltage range. Since it is DC, red probe goes to + and black probe goes to -(GND). If they are swapped, you would read negative voltage but magnitude would be the same.

  • Continuity check. Use a multimeter to verify electrical connections. https://www.fluke.com/en-us/learn/blog/digital-multimeters/how-to-test-for-continuity

  • Verify serial communication. Use a logic analyzer to observe data line and clock line. Typically, the logic analyzer can decode UART, SPI, I2C and CAN messages.

• Firmware issue?

  • Use a debugger to verify the code flow.

  • Verify clock configuration. The simplest way is using Systick and delay function with blinky application.

  • Use an example project from a microcontroller's vendor. This helps how to use vendor's driver code and pin configuration.

Here are the five steps of root cause analysis.(https://www.mindtools.com/ag6pkn9/root-cause-analysis)

1. Define the problem.

2. Collect data.

3. Identify causal factors.

4. Identify root cause(s).

5. Implement solutions.

Let's apply the root cause analysis to the firmware world example.

1. Problem: The force sensor does not provide the measurement.

2. Data

   1. The sensor datasheet is found(https://www.te.com/commerce/DocumentDelivery/DDEController?Action=showdoc&DocId=Data+Sheet%7FFX29%7FA7%7Fpdf%7FEnglish%7FENG_DS_FX29_A7.pdf%7F20009605-23).

   2. The sensor has four wires(Power, GND, I2C-SCL, I2C-SDA).

   3. The sensor is communicated via I2C. The sensor is I2C slave and the microcontroller is I2C master.

   4. The sensor supports the clock rate of 100kHz ~ 400kHz.

   5. The sensor needs 3.3V DC power supply.

   6. There are two commands(Measurement request and read data)

3. Casual factors

   1. The sensor does not make a good connection with a target.

      1. Wrong pin out

      2. Connectivity issue

   2. Firmware configures a microcontroller as I2C slave instead of master.

   3. Firmware configures wrong clock rate.

   4. The target board does not supply good power.

   5. Firmware send wrong commands including frame format and slave I2C address.

4. Root causes

   1. I2C-SCL does not make a good connection. To validate connectivity, measure resistance between I2C-SCL and GND. The measurement is close to 0 ohm. (root cause#1)

   2. Firmware configures a microcontroller as I2C master correctly.

   3. Firmware configures I2C clock rate as 100kHz which is within sensor's supported range.

   4. Measurement is close to 3.3V.

   5. Firmware configures a command with wrong frame format. (root cause#2)

5. Solution

   1. Fix I2C-SCL wire.

   2. Fix frame format and verify with a logic analyzer.

      1. reference an example code based on Arduino project(https://forums.adafruit.com/viewtopic.php?t=179938). The datasheet did not provide details of commands.

      2. Ask details of commands to TE customer service and receive another datasheet information(https://www.mouser.com/datasheet/2/698/REN_ZSC31014_DST_20160120_1-1999520.pdf).

As an engineer, the problem solving is life. To solve problems more efficiently, we need a method which can guide us to make a problem more specific and quantitative. One of methods is the root cause analysis and I show how to apply this method over real firmware problem above. While writing down the root cause analysis, everything becomes more clear. At the end, I am able to read force value!