DVB core enhancements - comments please?

As you could remember I initially scheduled my GSoC project to two phases. First one was DVB USB enhancements and second part was for DVB core issues. DVB USB changes are quite ready and I am going to sent upstream merge request in next few days. I will write article about that later in next few days.

I did some planning for the DVB core part recently. I ended-up listing known bugs, insufficient functionality and some new features and sent plan to the Linux-Media mailing list for the comments. No comments so far for that mail - but many of those are discussed earlier too.

DVB core enhancements - comments please?

DVB core enhancements - comments please?

Here is my list of needed DVB core related changes. Feel free to comment - what are not needed or what you would like to see instead. I will try to implement what I can (and what I like most interesting .

general validly checking for demodulator callback input values

• currently each driver needs to validate those
• values are highly hooked up to used television standard
• we can do almost all validly checking inside core
• we can also check if call is possible to perform in given condition
• for example BER is not valid when demod is unlocked
  

suspend / resume support

• support is currently quite missing, all what is done is on interface drivers
• needs power management
• streaming makes it hard
• quite a lot work to get it working in case of streaming is ongoing

use Kernel power management instead of own

• there seems to be Kernel services for power-management
• study if it is wise to use Kernel services instead of own
• own PM is still working very well, at least I dont know any problems

SDR - Softaware Defined Radio support DVB API

• http://comments.gmane.org/gmane.linux.drivers.video-input-infrastructure/44461
• there is existing devices that are SDR (RTL2832U "rtl-sdr")
• SDR is quite near what is digital TV streaming
• study what is needed
• new delivery system for frontend API called SDR?
• some core changes needed, like status (is locked etc)
• how about demuxer?
• stream conversion, inside Kernel?
• what are new parameters needed for DVB API?

DTMB standard support for DVB API

• it is Chinese DTV standard
• I already ran RFC but have been too busy for implementing it :]

LNA (low-noise amplifier) support for DVB API

• there is quite a lot of devices having LNA
• currently not supported => LNA is configured off typically

offer polling method for statistics

• many static counters could not be read as a "one go"
• typical cycle is : start measurement => wait => read counters
• some drivers starts own internal work-queue for polling (complexity)
• some drivers blocks IOCTL when taking measurement (bad)

fix frontend properties

• those has been broken since MFE => SFE change
• currently implemented as a properties per driver
• need to be properties per delivery system
• are broken because driver/chip could support multiple DTV standard

AF9015 driver converted to new DVB USB

AF9015 driver converted!

I finally decided to publish one rather complex DVB USB driver as a demonstration what it looks like. I selected Afatech AF9015 driver as it is one of the most complex and main motivator of whole DVB USB changes. I have also Anysee, AU6610 and EC168 drivers converted, but those are far away complexity compared for AF9015, since not so interesting.

Code size reduced a lot

All-in-all, code size goes down rather much. It was 2084 LOC and now it is 1609 LOC - it is near 500 LOC less. Basically it is all coming from the removed hacks. Now all needed functionality is generalized to the common DVB USB and ugly hacks are removed from the individual driver.

PATCH: af9015: switch to new DVB-USB

Near future plans

I am quite happy for features common DVB USB now supports. I think all but dynamic USB ID are implemented somehow for those I initially planned. There is still much work to do when implementing all correctly and test error paths. There is still at one big technical problem to solve. It is that delayed init which was added to fix firmware loading. I can get it crashing rather easily just repeatedly loading and unloading DVB USB driver... I suspect it is coming from the fact both workqueue running delayed device intialization and USB-core end up same routines. For example workqueue is downloading firmware and at the same time module is unloaded by user. Likely some locking is needed to prevent module unload in that case.

Dynamic USB device ID

Dynamic USB device ID

Reference designs

Nowadays many DVB UDB devices are actually chip vendor reference designs which means those are very similar. Used chips are same and wired together similarly, meaning only new device id is needed for the driver in order to get it working.

It takes too long

There is always significant delay from the device release to the point support is arrived for the distribution. That delay is coming from the different schedules. First it takes time until driver author gets info about new device is needed to add driver. He has to find out hardware or at least someone who can test patch. After that there is Kernel schedule. Kernel has own merge window and release candidate cycle that takes many months. In my understanding simple USB IDs are allowed to add driver during release candidate phase, but usually it still goes through merge window which adds significant delay. And finally there distribution schedule. It is usually needed to upgrade used distribution as they do not upgrade new Kernel version during release cycle. All-in-all, it could take year or so.

Promote the device ID to driver

There is feature called dynamic USB device ID to tackle that delay. User can tell for the Kernel that he wants try if given device driver could drive his new device. For example we could say load dvb_usb_af9015 driver for the device having USB ID 15a4:9016:
echo 15a4 9016 > /sys/bus/usb/drivers/dvb_usb_af9015/new_id

DVB USB and dynamic IDs

Currently DVB USB doesn't support dynamic IDs. Today I did some work in order to add support for it. Unfortunately I did not find out as nice solution as I was hoping. As for now I use .driver_info field from the struct usb_device_id to pass all needed to data to the DVB USB. In practice .driver_info field carries pointer to the struct dvb_usb_device_properties. In normal case all needed data is inside MODULE_DEVICE_TABLE() but in case of dynamic ID there is no entry inside MODULE_DEVICE_TABLE() and thus no needed pointer inside .driver_info.

Solution is to implement own .probe() and set .driver_info for dynamic ID. And finally pass that all to the DVB USB. Not very ugly, nor nice.

What I would like to see is dynamic ID entry which could be added to the driver MODULE_DEVICE_TABLE() similarly as others. Maybe there is even some reserved USB vendor IDs for special purposes that could be used.

struct usb_device_id {
    .idVendor = VENDOR_ID_DYNAMIC
    .idProduct = PRODUCT_ID_DYNAMIC
    .driver_info = <own data>
}

DVB USB firmware download problems

DVB USB firmware download problems

suspend / resume

I listed fixing resume from the suspend as a one topic of the DVB USB to fix. There has been very long time nasty suspend / resume bugs which seems to be related for the firmware downloading. Reason has not been clear thus those were never fixed. I found one bug report back from the 2009 which was sent for me, as my af9015 driver crashes during resume. At that time I didn't have much experience how to hunt bug like that, which was out of my chip driver. I made simplest possible DVB USB driver which just requests firmware download - and crashes - just as a proof of concept. Unfortunately I did not get any help and it left unresolved.
Crash I made 2009: DVB USB resume from suspend crash

Fedora 17 installation to the SSD

As I got very many Kernel crashes during the development and every crash means booting which wastes time around 10 minutes per boot. Also Fedora 17 was just released and I wanted to install it, so I decided to buy new SSD and install Fedora 17! Install goes nicely, taking only minute or two, it is unbelievable how fast SSD is. I have to say hate Gnome3 UI, thus I was earlier running it "fallback" mode but now I decided to give opportunity for Cinnamon.

DVB USB devices on Fedora 17

Finally I started to test how DVB devices were working. It came as a small surprise that plugging DVB USB stick in hangs device around 30 seconds until it appeared. Looked the logs and there was udev errors. It was not totally broken as it still started working but takes 30 seconds and it happened for every DVB USB device which uses firmware.


Jun  1 19:24:24 localhost kernel: [   59.820552] dvb-usb: found a 'Afatech AF9015 DVB-T USB2.0 stick' in cold state, will try to load a firmware
Jun  1 19:24:54 localhost udevd[423]: worker [1581] timeout, kill it
Jun  1 19:24:54 localhost kernel: [   89.853168] dvb-usb: downloading firmware from file 'dvb-usb-af9015.fw'

udev and blocking firmware download at module_init()

Clear suspicion was udev as there was nothing changed inside DVB USB which could break it like that. I downgraded udev to Fedora 16 version and it started working. As I was not any familiar with the udev I decided to open ticket for the Fedora Bugzilla targeting it udev. It didn't take many minutes Kay Sievers replied to Bugzilla, explaining it is Kernel driver issue as it blocks module_init() by downloading firmware. The reason it now arises is udev. Its behavior was changed and we wasn't aware that it was needed to change Kernel drivers not to block module_init().
Fedora Bugzilla: DVB USB device firmware requested in module_init()

The fix

I implemented non-blocking module_init() / USB probe() for DVB USB and problem disappeared. It was done by delaying old probe functionality, which downloads firmware too, using Kernel workqueue and returning always success for the probe() / module_init(). Doing it that way is not prettiest option at least for my mind but there is no any better I am aware. Downside here is that we must deregister device in case of something fails during "old probe". I did that de-registering using usb_driver_release_interface().

suspend / resume fixed too?

I was unable to reproduce old, and very well known, suspend / resume bug with my current hardware and software. Anyhow, it is quite likely coming from the same blocking firmware download issue. Resume from the suspend behaves quite similarly as plugging DVB USB stick to the USB port.

There is still another existing suspend bug. If you are streaming DVB device, like watching television, it will never go to the sleep. It just hangs. That is DVB core issue as in my understanding there should be some mechanism to stop stream before going to sleep. I will fix that later this sumer after DVB USB issues are fixed.