CAN-Bus lag issues with increasing busload

General Bugs and troubleshooting
1

Hi,

I’ve got some issues with my RealDash CAN-Bus setup.
Some of the signals are very laggy sometimes.

That means, that sometimes its working as supposed.
But sometimes the signal on Real-Dash Gauge has about 1-2 seconds delay.
It gets worser with increasing CAN-Bus load.

For example: I compare the value for the clutch pressure on Tunerstudio and on Realdash.
TunerStudio value of course changes immediately when I pull the clutch lever, but on RealDash (receiving the signal via CAN-Bus from Megasquirt) sometimes it has a delay of 1-2 seconds.

My hardware setup is the following:

Megasquirt MS3x
RealDash on Android Tablet
Tested with two CAN-Adapters:

  1. USB CAN Analyzer Seeedstudio V8.00 200000 Baudrate
  2. 3RT CAN-Adapter 115200 Baudrate

500k CAN-Bus Baudrate
4 additional CAN-Bus devices (1xDBW-Controller, 3xBreakout Boxes)

With both CAN-Transceiver everything is working perfectly as long as I only got a few messages on the CAN-Bus.
If I then, for example, activate the Megasquirt DBW messages (50-100Hz messages) then it gets laggy with the 3RT CAN-Adapter.

Therefore I tried the USB CAN-Analyzer from Seeedstudio.
This is working better for some reason. But I still get the same issues like descriped above, as soon as I further increase the busload.

Example for increasing the busload:
So even with deactivated (high-frequency) DBW messages, but with CAN realtime data broadcasting with multiple activated messages at high transmission rate, I get CAN-Bus issues with both CAN adaptors.
Which I can see on laggy sensor values.

I only got issues on RealDash.
I didn’t notice any laggy behaviour on Tunerstudio/Megasquirt.
The Megasquirt also receives most signals via CAN-Bus.

Now I wonder, is anyone running a similar setting with the Seedstudio USB CAN-Analyzer without issues?

Any ideas how to improve the performance?

  1. Is it possible to include MASK and FILTER settings for the CAN-Adapter in Real-Dash?
    I’d hope to improve the performance, by accepting only the ID’s which I want to see on Real-Dash.
  2. Or should I test another CAN-Bus adaptor? If so, which one?

That’s an overview over my CAN-Bus devices and sending messages.
That’s my minimum requirement. I’d say it’s not that many messages.

Megasquirt:
11 messages with 5-10 Hz via CAN realtime data broadcasting
2 DBW messages with 50-100Hz

DBW-Controller:
2 Messages w/ interrupt callback on MS3 DBW ID

“Left Side” Breakout-Box
2 Messages with 5 Hz

“Right Side” Breakout-Box
1 Message with 10 Hz

Front-ECU, which has the Pedal-Position-Input for the DBW:
1 Message with 50 Hz (PPS)
1 Message with 5Hz

Regards

Philipp

Edit: Data rate shown in RealDash is 5000 ish

2

Hard to say, but I could think of two reasons:

  • CAN adapters must read CAN line and send to serial, busy CAN network may cause frame drops. I think there is nothing wrong with the adapter, it just cannot process the data fast enough.

  • RealDash has quite large input buffer for incoming CAN frames. If for some reason app is not running fast enough to process all the data, you may occasionally experience 1-2 second lag as you describe. You can experiment if this may be the case by using higher-end phone or tablet to test the connection.

Of course this could be multitude of other issues. We use two CAN lines with MaxxECU on DEATHFISH 2 and never had a single hiccup on that setup. And MaxxECU sends more data faster compared to Megasquirt default settings.

3

Okay thanks for reply.
Got some further questions

  1. What about my idea with adding MASK/Filter settings in RealDash for the CAN-Adapters?
    That generic software of the SeedStudio CAN-Analyzer supports that. Same for the 3RT CAN-Bus android App. So I think it should be generally possible to include it also in RealDash.

Like this the CAN-Transceiver and the Android tablet only have to process the data which will be used for display and datalogging. And in my example … all of that high frequency DBW messages doesn’t have to be processed by the App anymore.
I think that would be quite a booster.

On all my CAN-Bus ECU’s in the network I’m working with masks and filters as well. Which helped to gain a lot of performance.

  1. To your setup: How many reads per second do you have there?
    Also above 5000? And have you tested your setup with CAN Analyzer ?

Regarding the hardware:
Yes okay. I will try different hardware and see what’s the impact.
However my current hardware has following specs:
CPU: Qualcomm Cortex-A53 64-bit Octa-core Processor, 1.8GHz
GPU: Adreno 506
RAM: 2GB LPDDR3

I’d say it’s not so bad. I can also test reduce the dashboard pages, maybe it helps?
Currently I got 5 or 6 pages in the dash with an .rd file with >12Mb.
Not sure if the dash got an impact.

4

This is doable, but I would first prefer to verify that problem is actually there.

Your hardware should be able to run RD with those CAN connections just fine. Maybe you could try with simple, text only dashboard if that helps.

5

I tried it once with only one text gauge + one graph gauge on the active page.
There I still had the same issues. However I had additional 5 pages in the dash.
Not sure If there is an impact by the other pages.

Therefore I will test it with just one page and one text gauge.
I’ll be back at home in three weeks.
If there is anything else I should test or measure then, let me know.

6

Are you using a termination resistor in the CAN network?

1 Like
7

Hi, the MS3x has one 120 Ohm and I got one 120 Ohm at the CAN-Analyzer.
Between those two the distance is the largest (120cm).

As far as I remember I removed the resistors on the other CAN-Bus transceivers.
But I’ll double check that and measure the resistance between High- and Low lines.

1 Like
8

Currently I’m back at home and there I made further tests as requested

  1. CAN-Bus termination is fine. Got only two 120 Ohm resistors.
    One resistor in MS3 and another one at the front CAN-ECU. Those are the end points of my CAN-network.
    Then I measured the resistance between High and Low and Multimeter shows 62 Ohm.
    So that should be all fine. And like said before, all other CAN-Devices work as supposed to, I only got those lag issues on RealDash.

  2. Created a simple Dash in Real Dash - Only showing one single value.
    Still have the same issues.

  3. Then I reloaded my standard Dashboard. Then I also noticed that those high frequency DBW messages from the MS3 do not lag. Only the other signals, which are sent via “CAN realtime data broadcasting” from MS3.
    Propably as the DBW messages have lower ID than the other MS3 messages, and are therefore priorized?

  4. Then I tried different BaudRate Settings on the CANanalyzer.
    Standard was 2.000.000 Baud
    I reduced it to 115.200. For me it looked like, this made the situation even a bit more worse.

  5. In general … same picture like aready descriped.
    With low busload (without those DBW messages) I don’t have issues.
    RealDash then shows ~700 values/s (Data update rate)
    If I activate DBW on the MS3, then it increases to 4300-4800 values/s.

Therefore back to my questions from some weeks ago:

  1. Can you implement CAN-Bus Mask and Filter setting in Realdash?
    I think easiest would be to accept only ID’s, which are defined in the .xml file.
    I mean other IDs can’t be used anyway, if they are not in the xml.

  2. Are you guys really running with up to 5000 values/s data rate, without issues in RealDash?

9

Currently we have not planned to implement that as this is very special case and affects very few users.

We do run up to 20000 values/s on simulator setup, albeit on powerful desktop PCs. I believe that adapter starts buffering the frames when its unable to process them fast enough. That is somewhat difficult to test on.

10

Okay, I’ve reduced now the DBW messages to 50Hz on my DBW controller. All other stuff max on 10Hz. And only the necessary stuff.
Now it’s working stable without delays or lags. But who knows, maybe someday filter settings will be implemented to real dash.

BTW: I’ve bought this USB-C CAN module from ebay (175861965091) for troubleshooting.
It’s works super great, is smaller than the Seedstudio CANalyzer and costs only below 10€. And the best is, that it works with software cangaroo. It’s been super helpful for me for troubleshooting the CAN-Bus. I also created a dbc file, so that I can see the “decoded” messages on my CAN-Bus and also the transmission frequencies of all messages.

11

Interesting looking device, I ordered some of them also for testing. Here is their ‘official’ web page, I think:

An Open-Source USB to CAN Adapter - CANable

As it happens, this slowdown on busy CAN has been bothering me and I have made an small update to RealDash that may increase the maximum throughput of the data. Send me an email to contact@realdash.net if you would like to try the pre-release version.

12

Sound good, thanks.
Unfortunately I’m away for some weeks again. Don’t know yet, how long exactly.
But sure, I’ll test when I’m back. If its not in an official release by then, I’ll contact you.

1 Like
13

I think I may be running into similar limitations with my Haltech Elite ECU setup using seeed can analyzer set to 2Mbaud UART.
Haltech uses 1Mbaud CAN and I have ECU + wideband controller on the same bus, in addition to that I am emulating an IO Box which is required to send 3(or optionally 4) standard length frames every 20ms. That’s 200 frames/s to send.
Last time I checked there is only about 1200 frames/s in total (1000 to receive/200 to transmit), however some values are definitely dropped such as target lambda which often displays stale values (0.92 stale instead of 1.0 current as confirmed by having the tuning software running in parallel on the laptop) or nonsense like 0.3 target for a short period when the real target is 0.98. And I had experienced a couple other values “jump” on screen like coolant temperature which definitely does not experience quick changes.

In addition whenever the car is running and everything is communicating my ECU reports IO box timeout every several seconds (the time between reported errors always varies in the log).

Running Android 8.0 on my joying px5 head unit modified with large heatsink and fan (8x ARM Cortex-A53 @ 1.5 GHz).

This is the last bit that I really wanted to get running on my head unit (the IO Box A/B emulation at 50Hz CAN frame frequency) but unfortunately am struggling to understand what the reason for communication timeouts is.
Is there a suggested way to debug this other than sniffing the canbus with another analyzer/logic analyzer?

14

On too busy CAN, typically these problems go into two categories, the adapter or RealDash:

If the problem is on the adapter:

  • Frames are dropped completely.
  • Frames are sometimes interleaved in serial data causing erroneous readings.

If problem is on the RealDash:

  • Noticeably lag on data experienced as RealDash is unable to process incoming frames fast enough.
  • After certain amount of lag, RealDash also starts dropping frames.

If you believe that problem is on RealDash side, you could try:

  • Minimize calculations on XML ‘conversion’ attribute. While it seems simple enough to process the conversions, when you do that to thousands of frames per second it starts adding up quickly. And as all conversions are processed on single thread, some devices may run out of oomph pretty quickly.
    • Use multiplication instead of division
    • Minimize usage of enum attribute
    • Use conversion ‘V’ instead of B0, B1 …