Sniffing and analysing communication protocol
I decided to put online my RTL2832U reverse-engineering / hacking scripts. I get quite often questions how to sniff USB, analyse data and generate some driver code to test. I hope this example helps. Fortunately RTL2832U USB protocol is also very simple, which is easy to read and understand even with very limited existing knowledge.I like to use native Windows XP and SniffUSB 2.0 to took USB captures. There is multiple USB sniffers available, both software and hardware. Feel free to select one you like, but be aware that these scripts only work for SniffUSB logging format. Making log format conversion scripts should be quite trivial though.
Step-by-step
- Take sniff using SniffUSB (outputs file UsbSnoop.log). Just tune to working TV channel, one sec is enough, log file size increases very rapidly when picture is streaming.
- Use parser.pl to convert SniffUSB log (UsbSnoop.log) more human readable form. (History: In my understanding parser.pl was a part of USBreplay toolkit.)
- ./parser.pl UsbSnoop.log > UsbSnoop.ts
- Lets remove video data as we are not interested of it. It is only control data what we like to get.
- sed -e 's/BULK\[00081\].*$/BULK\[00081\] MPEG2 TS packet data removed/g' UsbSnoop.ts > UsbSnoop.p
- Dump out register writes from the control data.
- ./rtl2832u.py UsbSnoop.p > UsbSnoop.c
rtl2832u.py could currently extract all registers from rtl2832u devices that has tuner e4000, fc2580, fc0013, tua9001 or r820t. There is few more RF tuners used with rtl2832u and in case of unknown tuner it could still dump out rtl2832u itself registers. Generated code could be copy & pasted directly to the Linux drivers in question and make some tests.
Bug hunting
I have found that approach most easiest way to find out and kill Linux driver bugs. Lets take for example RF tuner bug, which causes notable reduced sensitivity. Took sniffs from the working machine, generate code, copy & paste that to the suitable places of non-working driver. Rapidly that non-working driver starts working and it is time to do manual binary search to find out problematic setting and fix it. Easy? Eventually yes.Scripts and an example (rtl2832u + fc0013)
http://palosaari.fi/linux/v4l-dvb/rtl2832u_scripts/- parser.pl, log conversion script
- rtl2832u.py, C-code generator script
- UsbSnoop.log, SniffUSB log
- UsbSnoop.ts, parser.pl log
- UsbSnoop.p, sed log, video stream removed
- UsbSnoop.c, rtl2832u.py log, generated C-code
Another trick I commonly use for this sort of thing is to take the trace, and then have a Perl script find the last value of each register set for a given chip. This allows you to build up a representation of the internal state of the chip under Windows and compare it against a register dump under Linux.
ReplyDeleteDevin
Yeah, building shadow registers by your script. That is very useful in case of you are making 100% reverse-engineered driver from the bus sniffs.
DeleteThere is quite a lot of small tricks to find out different data.
For example demod has only few registers which are programmed differently for each bandwidth. Those are easy to find out taking sniffs against DTV signal generator and then diff programmed registers.
TPS (transmission signal parameters) could be found just dumping all demod registers and check which are changing according to parameters set to signal generator. But for these you will need that modulator...
I prefer to use VirtualBox USB pass-thru and get the trace directly from the Linux USB debugging interface. One advantage of this is that the log comes out directly into the Linux system without having to transfer it around. Another advantage is that I can knock together a fake emulation of the hardware using gadgetfs and the loopback driver, and then use it to inject data into the Windows driver and see how it responds.
ReplyDeleteWhat advantages does SniffUSB have?
Advantages of SniffUSB are ... not very rational. It is simple, free, does the job and is popular. Also I have used it from the very beginning of driver hacking career ~2006. Same applies for most LinuxTV developers I think. But disadvantages are coming larger everyday as it supports only legacy Windows XP. I actually has a Ellisys hardware sniffer too, but I am too lazy to use it because of SniffUSB just works :)
DeleteI have also tried virtual machine solutions, both VMware and VirtualBox.VMware tends to be broken after each Kernel upgrade as its Kernel driver patch touches Kernel sources deeply. Biggest problem with VirtualBox was missing USB2.0 support! Otherwise VirtualBox could be very nice solution.
Dunno if there virtual machine things has changed during last ~3 years...
In my experience the use of virtual machines for USB debug has been a mixed bag. It works reasonably well nowadays for digital TV (where the bandwidth is typically only 10-30Mbps), but still falls down when trying to debug raw analog TV (where the data rates are on the order of 240 Mbps).
DeleteDon't get me wrong - it *has* gotten better. When I first tried debugging a tuner with VirtualBox back in 2009, it locked up the entire system.
Devin
I see, thanks. I've only really hacked on quirky HID devices and other low-speed, USB 1 stuff, so never run into these issues.
DeleteHi Antti,
ReplyDeleteWith your trick , i found a problem in your E4000 module, there is a problem in the sigma delta calculation.
i've fixed the problem with the folowing code
/*Problem in sigma delta
sigma_delta = 0x10000UL * (f_VCO % priv->cfg->clock) / priv->cfg->clock;
*/
//Fixed
f_VCO = c->frequency * e4000_pll_lut[i].mul;
xl=priv->cfg->clock;
sigma_delta = (f_VCO % xl);
sigma_delta=sigma_delta/1000;
xl=xl/1000;
sigma_delta=sigma_delta*65536/xl;
//end of fix
With your code the sigma delta has always a small value due to a bad convertion.
The problem was an improper tuning on some frequencies
I've send you a email with my full code
Best Regard's
Damien CABROL
Nice! I will check that on end of the week as I am on holiday trip currently.
DeleteProblem was 32-bit overflow in calculation. The reason why I have not been able to find that, even there has been very many bug reports, is that I am using 64-bit Kernel and bug appears only in 32-bit. I am now waiting patch from Damien...
Deletesigma_delta = 0x10000UL * (f_vco % priv->cfg->clock) / priv->cfg->clock;
sigma_delta = div_u64(0x10000ULL * (f_vco % priv->cfg->clock), priv->cfg->clock);
Just that bug was the main motivation behind that blog post (as I mentioned in one mail in the mailing list) and seems like it leads the bug fix!
That makes also wonder what I should do in order to improve situation. The fact is that almost every driver I have made has had similar 32-bit vs. 64-bit bugs. Lastly it was Mirics MSI3101 driver one week ago, which Mauro fixed kindly, as I was travelling at the time :) I don't want to invest any new hardware only for testing all kind of architectures...
With the end-of-life reached for Windows XP using SniffUSB 2.0 for analysis becomes a challenge in itself (I've no PC with this OS version anymore). From the previous posts I also understand that using a VM with XP will give it's own challenges too, hence I wonder if someone tried an analysis with other freeware tools running on W7?
ReplyDeleteI found:
1) https://github.com/desowin/usbpcap but am unclear if the documented limitations would impact the required result for the stated purpose. any hint?
2) http://sourceforge.net/projects/usbsnoop/ but there was an issue in W7 reported
I'm aware of expected log format differences, which will require changes in parser.pl
Thanks for sharing insights with a newbie.
Regards
Mike
This comment has been removed by the author.
ReplyDeleteI'll give usbpcap a trail based on response to my question in https://groups.google.com/forum/#!topic/usbpcap/8uUbIoCjdJ0
ReplyDeleteYeah, SniffUSB is a bit legacy, but I use it still. Recently I used Linux kernel sniffer for one driver. It is suitable solution if you use Windows guest in Linux virtual machine host. I used parse_tcpdump_log.pl script from v4l-utils package. That package contains other suitable scripts too.
DeleteHere is example I did, first one takes sniff, second line removes stream data we are not interested (we are interested mainly control commands):
$ sudo ./parse_tcpdump_log.pl --pcap --device usbmon2 > usbsniff.p
$ sed -e 's/EP\=81).*$/EP\=81) stream data removed/g' usbsniff.p > usbsniff.p2
Appreciate the hint. Unfortunately my only Linux host is a tiny Raspberry Pi that can't run a Windows guest VM, hence the OS independent option WireShark & USBPcap sounds still like less effort for my computing environment but I'm still gathering information before jumping to a conclusion.
DeleteDo you think you could start developing Linux kernel using windows desktop and Raspberry Pi? Go on and install native Linux first. It is very hard without.
DeleteWell from my view (without the LINUX hacker background) the goal was to create & compile a device driver. This should be about 150-300 lines of C code. I thought a RPi should be capable enough to handle this task. Did I make a wrong assumption?
ReplyDeleteHacking drivers is not that simple. You will need to compile it many, many, times, even many hundred of times if you make a totally new driver. It is very unlikely you could just look the sniffs and succeed at the first try.
DeleteMy goal is to get Delock 61959 V2 driver working. Was I misunderstanding the complexity and therefore effort from the statement: '
DeleteSi2165 driver was made by Matthias Schwarzott and Si2148 is supported by Si2157 driver I made. So yes, that device could be supported rather easily. Just small changes needed for existing drivers in order to define device profile.'?
Still, it is very unlikely it starts working just right after first test. What you think you like to see from the sniffs? Likely tuner and demod I2C addresses are the first and also easy. But it does not work with that simple unless a very good luck. You likely need to dig with GPIOs and if bad luck then with clock configs. And maybe some more. Even if it is not many lines of code, it could be tens of things to try and adjust.
DeleteI've followed your advice, converted an old HP NC6000 Laptop with Intel Pentium M 1.7 Ghz CPU & 1GB RAM to LUBUNTU 14.04 and obtained the current media tree as explained in http://linuxtv.org/wiki/index.php/Maintaining_Git_trees.
DeleteIssue encountered:
1) can't find any reference to Si2148 in '/Linux-Media/v4l-dvb/drivers/media/tuners/si2157.c' only Si2147 and Si2158
Did I miss a patch? Any hint appreciated.
Current driver does not know Si2148 chip. However, all those Silicon Labs tuners uses quite same firmware API, which means same driver could be used, with minor or no modifications at all.
DeleteThere seems to be support for Si2148 too, but that patch is not yet send for me. I suspect is coming later.
https://tvheadend.org/attachments/2347/0001-tuners-si2157-Si2148-support.patch
An instant bonus are bonus funds that are be} awarded in a restricted state and could also be} wagered with instantly upon grant. In general, making a “pass line” bet on whether or not the shooter will win offers you 50/50 odds. Blackjack is usually about luck, and generally, the home edge , is simply about 2%. 바카라 사이트 Fully adaptable assembly spaces, unparalleled leisure and thrilling facilities.
ReplyDelete