FFRend

Important FFRend bug fix; testers needed please!

2012-01-23T19:22:00.000-08:00

The last released version of FFRend (2.2.1.5) had a bug that caused the entire desktop to flicker whenever a row view was updated. Since just about every command updates one or more row views, the desktop flickered like crazy. You'd think I would have noticed it during testing but I normally run FFRend full-screen. I'm pretty embarrassed about it. Anyway please take the latest version, which fixes this lame bug and some others too.

Download:
FFRend download

I would greatly appreciate it if any FFRend users who hang out here would take the new version out for a spin and report back if they see anything weird. I'm especially worried about the row view code, because that's what caused the desktop flicker bug in the first place. The Freeframe Parameter and MIDI Setup row views are now supposed to remember the scroll position separately for each plugin. Try switching back and forth between a plugin that's scrolled and one that isn't. Do you see any painting artifacts, or does the view paint smoothly?

This version also fixes a bunch of problems related to synchronization of oscillators between plugins, which crept in with V2 as side effects of parallel processing. The earlier versions of V2 would lose sync between plugins at the drop of a hat. It should be a lot better now. The problem was basically that in a pipeline each plugin has its own frame of reference in terms of time, so it's necessary to compensate for that in various places. To debug it I had to make a pair of special Freeframe plugins, one that stamps its parameter directly onto the frame as a number, and another that draws its parameter onto the frame as a waveform, like an oscilloscope. They're pretty handy actually. I can make them available if anyone wants them...

time travel: oscillator slippage

2011-12-31T14:31:00.001-08:00

The V2 parallel-processing rework of FFRend was a huge win overall, but it also introduced subtle bugs, some of which are taking a long time to be discovered. Another one turned up yesterday: oscillator slippage can occur due to time shift between plugins. In a pipeline, the further a given stage is from the output, the further into the future that stage is compared to the output. Any queuing between stages increases the difference even more.

Suppose a project contains two plugins, A and B, each containing one oscillator. A inputs to B, and B is connected to the output. The two oscillators have exactly the same frequency and appear to be in perfect sync. But if we stop FFRend in the debugger and examine the two oscillators, we'll find that they don't have the same value! A's oscillator is ahead by a small amount. This is because A is working ahead of B, i.e. A is operating on a frame that hasn't arrived at B yet. This unavoidable; it's how pipelines (or assembly lines) increase throughput. We need to careful when we say that A and B appear to be in sync. What we really mean is that A and B are in sync in the output! This the synchronization we care about. A and B should be out of sync when viewed from an external point of view (e.g. the debugger) because that's our proof that parallel processing is occurring.

However this necessary time shift means that complications can and do occur. For example, suppose we save the project. As currently coded, we're just copying the current values of oscillators to the project file. But the current values aren't all from the same time! In other words, the current value of A's parameter represents a state several frames ahead of B's state, which is several frames ahead of the output's state. Consequently, if we reopen the project, the oscillators won't be in sync in the output, despite the fact that they were in sync when the project was saved. The error occurs because we're violating an implicit assumption that all the values in the project file are from the same frame of reference.

To solve this problem, the save function needs to compensate the oscillator values for time shift. This should be doable because a) oscillators are periodic functions and can therefore be stepped in either direction with predictable results, and b) each plugin knows exactly how many frames ahead it is relative to the output state.

Note that this problem is unaffected by pause, i.e. the problem occurs regardless of whether the engine was paused when the project was saved. This is because pausing the engine doesn't reset the frame pipeline (unlike stopping the engine, which does); all the frames are frozen in mid-flight, so however far ahead a given plugin was when the engine was paused, it remains that way.

An additional, related problem was observed: inserting a plugin while the engine is running also causes oscillators to slip. Here the issue is even more subtle. The engine can only modify the plugin array while stopped, therefore inserting a plugin necessarily involves stopping and restarting the engine. However stopping the engine resets the pipeline, discarding any frames that were in progress but not yet output. Thus when the engine is restarted, all the plugins initially have the same frame of reference. But the oscillators still have the same states they had at the moment the engine stopped, when the plugins all had different frames of reference! Consequently there's now a mismatch between a plugin's oscillator states and what frame it's processing. The further from the output the plugin is, the larger this mismatch is, and the worse the slippage is.

This problem can be solved similarly to the one described above, by compensating each plugin's oscillators as needed in the case where the engine is restarting.

As previously mentioned, these problems took a long to show up, and even when they did it wasn't simple to verify them, because of time uncertainty. The most reliable tool for observing them was an invention I should have thought of long ago: a Freeframe plugin that simply writes its parameter to the input frame, as text. If you modulate the parameter with an oscillator, you're now stamping each output frame with an unambiguous record of how that oscillator's state changed over time. Most importantly, you're seeing the value the oscillator had when its parent plugin processed that frame, regardless of any time shifts due to pipeline backlog within the engine.

The plugin needs extra parameters for controlling the position of the text with the frame, so that different instances don't overwrite each other's text, but other than that it's totally straightforward. I'm thinking of making a similar plugin that draws the parameter as a waveform, like an oscilloscope.

Tests also show that even within a single plugin, oscillators may slip (become out of phase with each other), though the slippage is of a much lower order of magnitude. Whether they slip or not depends on the numerical relationship between their respective frequencies. For example:


Freq A  Freq B    Slippage
.1      .05       none
.1      .0125     none
.1      .0333333  .0002 after 2 million frames

This type of slippage is due to the use of floating-point in the oscillator, so there's probably not much to be done about it, but more research is needed.

Fauve colorizing plugin released

2011-12-29T22:48:00.000-08:00

Good news everyone! The FFRend project has released an exciting new Freeframe plugin that colorizes video in the style of Fauvism! Some still images demonstrating its effect are shown below.

The second example is taken from this short (15 second) demo clip.

The Fauve plugin is packaged in ckffplugs-1.0.13, available for download here:
http://ffrend.sourceforge.net/download.html
(Freeframe Plugins, Binary Release)
And for those who care about such things the source is there also.

The plugin is a visualization of a histogram but using pseudocolor instead of the customary graph. Pixels are colored according to the popularity of their original colors in the input image, such that more common input colors translate to brighter output colors. The effect enhances edges, adds texture, simplifies and drastically alters the palette, and reduces spatial cohesion. In Luma mode, the plugin:

1. Calculates the frame's luma histogram.
2. Replaces each pixel with a greyscale value corresponding to that pixel's rank in the luma histogram. The histogram is normalized so that pixels in the top rank are assigned white in the output, while pixels in lower ranks receive proportionally darker values.

RGB mode is similar, but does the above operations separately for each color channel, resulting in color output instead of greyscale.
can read about it how it works.

No doubt André Derain is rolling in his grave...

André Derain, The Houses of Parliament (1906)

FFRend adds playlist feature: handy for installations

2011-12-27T12:51:00.000-08:00

The latest version of FFRend (2.2.01) introduces a playlist, better integrates the load-balancing feature, improves memory management, and fixes a number of bugs.

FFRend can now automatically open a series of projects, also known as a playlist. This feature is primarily intended to allow unattended use of FFRend, e.g. in an installation or gallery setting, though it's also helpful for VJs who need a break during a show. The user specifies the list of projects, the interval between project changes, whether the list should be played in sequential or random order, and whether it should loop. The playlist can be passed via the command line, in which case FFRend starts up with the playlist playing. This is particularly useful in combination with the new /fullscreen command-line option, which causes FFRend to start up full-screen on whichever monitor contains the output window.

The Load Balance feature was converted from a modal dialog to a control bar, allowing load to be monitored continuously during normal operation. The command is now View/Load Balance instead of Edit/Load Balance.

FFRend's handling of low memory conditions was improved, and advanced options were added which allow memory use to be limited in various ways. This can be helpful when working with large frame sizes.

Download:
http://ffrend.sourceforge.net/download.html

Release Notes:
http://ffrend.sourceforge.net/relnotes.html

FFRend (Freeframe Renderer) is a free, open-source renderer for Freeframe V1 plugins. It allows you to chain any number of plugins together in any type of routing, automate their parameters using oscillators, and display the output full-screen and/or record it to a file. FFRend has a modular interface, includes MIDI and dual-monitor support, supports plugin authoring via metaplugins, and leverages multicore CPUs by running each plugin in its own thread.

Bifurcations: digital emulations of analog video feedback

2011-12-21T13:12:00.000-08:00

Lately I've been experimenting with digital emulations of analog video feedback, with interesting results. I enclose a link to sample clips below. All the projects in this family share the constraint of having no "content" (i.e. source plugins) other than Radar and WaveGen. No clips or Whorld, just Radar, WaveGen, and feedback (usually multiple luma feedback loops with different effects on each loop). I'm particularly excited about "Starry Night" and "Fireplace".

Bifurcations

However there's an unexpected problem. I'm finding that I can't get the kind of results shown in these clips without using some kind of mirror, kaleidoscope, or similar effect. I get the best results by automating the scale parameters of IscHypnotic. This is contrary to what I know about analog feedback. For analog feedback all I typically need is:

1. Looping: in analog this is created by pointing the video camera at the monitor, but in digital the equivalent is e.g. an alpha mixer, luma key, or chroma key with one of its inputs looped back from the output.

2. Decay offset: in analog this is usually accomplished with a TBC, but in digital various types of color or luma shifts can be substituted.

3. Motion: in analog, the motion typically comes from moving the camera, but in digital a simple pan/spin/zoom should in theory work.

But when I try digital setups this simple, I don't get interesting results unless I make the motion component much more complicated. This isn't necessarily bad, but it does give the output a characteristic look (radial, or at least symmetrical). In the case of IscHypnotic it also introduces significant pixelization, which is one of the main objections to digital feedback.

Another interesting result is enclosed below. The FFRend project that generated this image always converges on the state shown. Perturb it all you like, and it either breaks completely or converges again. The branches are a perfect demonstration of the role of bifurcation in natural systems. If you ever had any doubts that nature is chaotic, this should dispel them.

But why does this particular project do it? I don't have any other projects that create this shape, and I can't get it to work without IscHypnotic either. It's all very puzzling. What is the role of the mirror/kaleidoscope/hypnotic in all this? The "Fireplace" clip suggests that there might be a way around it.

Examine the texture in the "Etched Chaos" clip. At some points it resembles feathers, or scales. Again it's clearly organic, and reminiscent of Mandelbrot and other fractals. There's a critical tipping point where it starts "budding" like a plant, but then it quickly exhausts itself and the moment slips away. Could it be possible to stabilize it?

Sensitivity to initial conditions is just the nature of chaos. I remember from the analog days that the feedback setups were absurdly sensitive. It was fussy work that required patience and a steady hand, and some people were better at it than others. There were magical cameras, magical monitors, even magical cables. Sometimes two cameras of the exact same model behaved drastically differently for no obvious reason. Nonetheless people managed to stabilize intense bifurcations for minutes or hours.

Back in the day, analog feedback purists often told me that I would never get similar results digitally. Electric Sheep was an interesting counter-example, but wasn't directly comparable since it's not even close to being a real-time process. I always held that it was just a question of time, and that parallel processing and Moore's law would eventually catch up. All of these new projects run at 30 FPS at XGA resolution on an i7 PC. The results convince me (and some of my most hard-headed analog purist friends) not only that it's possible to convincingly emulate analog feedback in real time, but that digital feedback has the potential to exceed analog feedback in terms of aesthetic variety.

Load balancing

2011-05-19T13:21:00.000-07:00

FFRend 2.1.00 adds an important performance-related feature (load balancing), and fixes a large number of bugs, some of them serious. Issues marked (V2) are specific to version 2.

For more on how to use load balancing, see the load balancing manual page. For the theory behind load balancing, see my article Pipelining for Plugin Chains.

1. A load balancing dialog was added, which allows multiple threads to be assigned to a plugin. Given sufficient cores, load balancing can greatly improve throughput in the common case where some plugins require significantly more CPU time than others (asymmetrical loading). The dialog is accessed via Edit/Load Balance or Shift+L. Load balance settings are stored in the project. Note that load balancing is only effective for stateless filters: assigning multiple threads to a stateful filter may cause output strobing.

2. Changing the frame rate in the Options dialog caused an incorrect recording length if the record duration was specified in time (as opposed to frames). Fixed.

3. The single step command wasn't updating the slider positions of modulated parameters. Fixed. (V2)

4. In the history view, the zoom setting now persists in the registry. (V2)

5. In the history view, the renderer's display name was changed from Output to Renderer, to be consistent with the other views. (V2)

6. The file browser's list control had an extra border (in XP and up) which appeared as a thin black outline. Fixed.

7. In the file browser, some editing keys (Delete, Insert, Escape, Ctrl+Z) behaved unexpectedly during label editing: they triggered app behaviors instead of edit control behaviors. Fixed.

8. Pressing the Escape key while dragging a plugin tab, patch row, or metaparameter row failed to cancel the drag operation. Fixed. (V2)

9. Pressing the Escape key failed to cancel a drag operation if a dialog row control had focus. Fixed.

10. If an error occurred while restoring the job control state, invalid jobs could result and persist. Partial data is now purged, and the user is given the option to delete the potentially corrupt job control file.

11. Restarting the engine caused frame queues to be destroyed and recreated, incurring needless overhead; the queues are now resized instead. (V2)

12. The options dialog was converted from a dialog to a property sheet, allowing related options to be grouped into pages for easier comprehension.

13. Changing the MIDI device can be slow if the CPU is heavily loaded, so a wait cursor is now shown, and the main frame is explicitly updated to ensure that the options dialog erases cleanly.

14. The sync oscillators command (Edit/Sync Oscillators) was initially removed in V2, but it's been restored. (V2)

15. The graph view was causing a handle leak, in which each graph update would leak two process information handles. The bug could be demonstrated by resizing the graph view. Fixed. (V2)

16. Deleting a plugin or opening a new project could cause a serious memory leak, which would occur while shrinking the frame buffer pool. As many as two frames could leak, depending on whether any of the deleted frames were currently assigned to a DirectDraw surface. Repeatedly undoing and redoing a plugin deletion would eventually demonstrate the bug. Fixed. (V2)

17. Dropping files onto the app from Explorer would leak heap memory, due to OnDropFiles failing to call DragFinish. Fixed.

18. Left-clicking a parameter slider's thumb without moving it added an undo event, even though the parameter value was unchanged. Fixed.

19. Opening a recent file via the file menu could cause the file menu to bounce, i.e. redisplay unexpectedly. This occurred after deleting a row in the Parameter, Patch, MIDI Setup, or Metaparameter views, while one its controls had focus, which would cause a hidden form view to receive focus. Fixed. (V2)

20. If MIDI Setup was showing a plugin-specific page, and one of the page's row controls had focus, attempting to select a main menu command via the Alt key would cause an infinite loop, permanently hanging the app. This is the infamous WM_GETDLGCODE bug, solved by removing the DS_CONTROL style from the MIDI Setup dialog. Fixed.

21. If a metaparameter was added while the Metaparams view was hidden, and the MIDI Setup view was visible and showing its Metaparam page, the new metaparameter wouldn't immediately appear in MIDI Setup. It would appear when the MIDI Setup view was next updated, e.g. by hiding and reshowing it or selecting a different page. This could occur when creating a metaparameter via the main view's context menu. Fixed. (V2)

22. In the MIDI and Metaparameter views, the header columns didn't line up correctly with their corresponding controls, due to a row item coordinate conversion error. Fixed. (V2)

23. Right-clicking the History view's scroll bars showed the view's context menu, instead of the scroll bar context menu. Fixed. (V2)

FFRend V2: multicore version released!

2010-06-20T13:18:00.000-07:00

FFRend 2.0.00 is a near-total rewrite, not an upgrade. The application was completely redesigned from the ground up to take advantage of the parallel-processing capabilities of today's multi-core CPUs.

Major differences

1. Each plugin now runs in its own thread, and given sufficient cores, each plugin runs on its own core, so that throughput is limited only by the slowest plugin (Amdahl's law).
2. The user interface is now much more responsive, even when the CPU is heavily loaded, because rendering no longer occurs in the UI thread.
3. The built-in clip player was replaced by an enhanced version of PlayerFF which allows clips to be opened by path, or by dragging from the Files bar's Clips pane.
4. Monitor quality has been improved and now has two settings (fast or smooth), specified in the options dialog. Even the fast setting is still an improvement over version 1.
5. Multiple instances of the same plugin are now differentiated by decorating their names, e.g. PeteMixer-1, PeterMixer-2, etc.
6. Plugins with more than two inputs are now supported.
7. A Graph view is now available, which shows a dynamically updated graph of your current routing. Note that you must download and install graphviz to use this feature.
8. Two additional new views, History and Queues, provide dynamic information about the rendering engine, allowing you to easily determine which plugins are limiting throughput.
9. The frame rate can now be unlocked, so that the engine renders as fast as possible, or at the monitor refresh rate in full-screen mode.

Minor differences

1. The Record command was moved from the View menu to the File menu, and the Options command moved from the View menu to the Edit menu.
2. The Export and Record commands no longer automatically pause the output.
3. The Bypass command no longer alters the routing. This can cause significantly different behavior, particularly when bypassing source plugins.
4. All sizing control bars now have a system menu option to control whether they're dockable.
5. There's no longer a distinction between Full-Screen and Exclusive modes, i.e. Full-Screen mode is always Exclusive. Consequently the app's UI is no longer accessible in Full-Screen mode without using a dual-monitor setup.
6. Metaplugins are still fully supported. Note however that metaplugins are internally single-threaded, i.e. a metaplugin's component plugins do not run in parallel with each other. Consequently, on a multicore machine, a typical metaplugin will run more slowly than the equivalent project.
7. If a plugin can't render because it's not routed to the output, any parameter modulations it has will not automate their corresponding sliders in the UI.
8. The Global Plugin and Monitor Source Selection features were removed.
9. The app now builds in VC++ 9.0.

FFRell (FFRend in parallel) coming soon

2010-04-24T13:56:00.000-07:00

There's been lots of progress on FFRell, the parallel-processing version of FFRend. I'm currently running FFRell at 1024 x 768 on a purpose-built i7 box (8 CPUs with Hyperthreading) and seeing frame rates from 30 to 60 Hz for quite complex patches. FFRend has been completely rebuilt around a multithreaded pipelined rendering engine. Each plugin runs in its own thread, and frames are routed via thread-safe zero-copy pointer queues.

I started by building a test system called ParaPET (Parallel Plugin Engine Test) which is an interesting project by itself; see the enclosed screen shot. Basically ParaPET is a tool for modeling the problem of optimal parallelism for arbitrary signal routes. The first case I had to deal with was feedback. If plugin A takes input from plugin B, and B also takes input from A, neither plugin can get started and the engine stalls. The solution is to prime one of the plugins, i.e. feed it blank frames until the feedback loop is closed and becomes self-sustaining. But first you need a general way of identifying such cases in all their permutations. ParaPET features a spider object which recursively crawls the plugins looking for feedback.

The other interesting case is when a plugin's output can arrive at a downstream mixer via multiple routes of different lengths. A simple example would be a source plugin that connects to both inputs of a mixer, one directly and the other via an effect. In such cases you'll get staggering, i.e. the plugins won't execute in parallel despite available CPUs. The solution is to add delay to the shorter route, so that the route lengths become equal again. Delay is added to a route by increasing the size of the corresponding frame queue.

This turned out to be a pretty hard problem to solve for all possible cases. Again the solution was recursion. The same crawl that detects feedback also gathers the necessary information for route balancing: for each plugin, 1) its maximum distance from the final output, and 2) for each sink mixer it's connected to, its distance from that mixer (distances are measured in hops, just as in network topology). With this information in hand, for each plugin, we can determine its longest route to a sink mixer, and compensate any shorter routes accordingly.

I came up with dozens of tough asymmetric test cases, including three- and four-input mixers, and the spider handles them all. It works so well I'm considering releasing ParaPET as a separate project. It's not tied to video processing so it would make a good general framework for pipelined parallel execution of interdependent tasks. If you're interested in playing with it let me know...

Once I got the engine working in ParaPET, I started building a new app (ParaFFEn) that used the same thread architecture but with FreeFrame plugins and actual video frames. The goal was to eventually port the engine into FFRend, but after hours of surgery the patient died: FFRend was just too single-thread oriented and couldn't adapt, so I gave up and started rebuilding FFRend from scratch, starting with ParaFFEn's engine.

After a long hard slog, FFRell can now read FFRend projects and run them in full-screen dual-monitor Exclusive mode. All of FFRend's GUI elements are included except the file browser and monitor bars. FFRell also includes the nifty dynamic graph view from ParaPET (courtesy of GraphViz), plus the scrolling execution history and queue views. What's not working includes recording, clipboard support, MIDI, and undo. Also FFRell can't create metaplugins, though it can load them. Plenty to keep me busy this summer. I hope to have a reasonably stable version by Q3.

MIDI control of monitor source

2007-12-08T15:37:00.000-08:00

These are the changes needed to implement MIDI control of monitor source:

in MidiInfo.h:


enum {  // plugin MIDI properties
    MP_BYPASS,
    MP_MONITOR,
    PLUGIN_MIDI_PROPS
};

in MainMidi.cpp:


const int CMainFrame::m_PluginMidiPropName[PLUGIN_MIDI_PROPS] = {
    IDS_MP_BYPASS,
    IDS_MP_MONITOR
};

case MP_MONITOR:
    if (Toggle)
        Val = (GetMonitorSource() != ParmIdx);
    {
        bool    IsMod = IsModified();
        SetMonitorSource(Val >= .5 ? ParmIdx : -1);
        SetModify(IsMod);   // restore modify flag
    }
    break;

The above change also requires a new version of the project file format, otherwise errors will occur in CFFPlugInfo::Serialize due the increased size of m_MidiInfo. This probably could have been avoided if the size of m_MidiInfo had been serialized, but it's too late for that now. The only remaining option is an explicit version test. Note that the version being tested is from the next level up, i.e. it's a CFFProject version, not a CFFPlugInfo version. The CFFPlugInfo version must be increased too, so that the clipboard format changes.


// this was tested and works fine
int MidiInfoSize = Version > 4 ? sizeof(m_MidiInfo) : sizeof(CMidiInfo);
...
ar.Write(m_MidiInfo, MidiInfoSize);
...
ar.Read(m_MidiInfo, MidiInfoSize);

Thumbnail thread exit problem

2007-09-27T15:36:00.000-07:00

If we're in thumbnail view, the thumbnail thread could be running, and we want it to exit, so we can safely launch another instance. We ask it to exit by incrementing the job ID; the thread checks the job ID after each extraction, and exits if the ID changes. This simple method has a weakness: there's a chance we preempted the thread after it checked the ID, but before it posted the bitmap to us, in which case we'll receive a spurious bitmap message. The thread is almost always busy extracting, so the chance is very small, but just in case, we increment the job ID before doing other work, thereby giving the thread more time to exit.

Projects are affected by changing frame rate

2007-07-29T15:35:00.000-07:00

Frame rate has turned out to be a major hassle. It probably would have been a good idea for projects to store the frame rate that was in effect which the project was saved, so that if the project is subsequently run at a different frame rate, the oscillator frequencies could then be compensated. Oh well.

Practically speaking, all of the projects that are currently being prepared for DVD render correctly at 25 FPS ONLY. This includes

big hex 27%
UltraWhorld 3b
UltraWhorld 4b

Just to make things even more confusing, there's a problem with PeteKaleidascope on the XP machine: at Divisions = .03 (horizontal mirroring), it creates ghosts. This can be observed clearly with the UltraWhorld 4b patch. The solution is to use a mirror plugin instead.

Hidden controls retain focus; child dialogs and DS_CONTROL

2007-07-13T14:32:00.000-07:00

Version 1.4.03 (and previous versions) crash deterministically if you load (via drag/drop) a plugin with at least one parameter, then load (again via drag/drop) a plugin with NO parameters, and then spin the mouse wheel, or press one of the arrow keys or editing keys. This turns out to be an example of a more general problem: if a control has focus, hiding its parent dialog does NOT take focus away from the control. The control continues to receive mouse and keyboard input. This can cause unexpected behavior (e.g. the wheel moving an invisible control) or even crash the app. But why do we have hidden dialogs? Here's why.

FFRend makes much use of the row view. This custom UI object is similar to a list view or grid control, but it's implemented as a form view containing a vertical list of child dialogs, one per row. A row dialog is just like any other dialog: it has a resource, contains controls, can be built using the Class Wizard, and can even be tested outside the containing view. The main advantage of this approach is encapsulation: row dialogs derive from a base class which makes it easier to operate on entire rows at once (e.g. for cut/copy/paste). By contrast, in a grid control, everything is contained in a single window, so there's no equivalent to a row object.

FFRend uses the row view to display plugin parameters, patchbay, MIDI setup, and metaparameters. In the case of plugin parameters, there's a complication: each plugin has different parameters, but only one plugin's parameters are visible at a time. FFRend allows each plugin to have its own set of row dialogs, and then shows and hides entire sets of row dialogs as needed. For example, when a plugin is selected, the previously selected plugin's rows are hidden, and the new plugin's rows are shown. This approach is wasteful in terms of memory, windows, and GDI objects, but efficient in terms of CPU usage: hiding/showing windows is cheap compared to creating and destroying them. The approach has another advantage: since we can assume that while a plugin exists, its parameter row dialogs also exist, we can store the parameter and automation data in the row controls, instead of storing them elsewhere and updating the controls on demand. This is elegant, and simplifies undo handling. So that's why we have hidden dialogs.

As it turns out, a dialog has to have the DS_CONTROL style in order to behave well as a child of another window. People often encounter this issue when they try to make a home-grown property sheet, i.e. a series of child dialogs that can be overlaid onto a parent container dialog. The usual problem is that without DS_CONTROL, tabbing doesn't work as expected: the entire child dialog is treated as a single tab stop. DS_CONTROL integrates the child dialog's tab layout into the tab layout of the parent window. FFRend used to handle tabbing in row views explicitly, but now that it's using DS_CONTROL, Windows handles the tabbing.

DS_CONTROL also improves window activation behavior: without DS_CONTROL, showing the parent window unexpectedly focuses the last control in the child dialog, and this can lead to the problem described above (hidden controls with focus). So DS_CONTROL is a very good discovery, and makes the row view UI much more robust, but it's still necessary to test for a hidden control with focus after updating the plugin parameters view. That's not such a big deal.

XP/Uno drops input MIDI running status messages

2007-06-18T13:41:00.000-07:00

The problem occurs when using WhorldRC to switch video clips quickly in the PlayerFF plugin. The problem first appeared when shortcut keys were added to WhorldRC for triggering recent clips. Before that it wasn't possible to switch clips fast enough to cause the bug. It seems that when MIDI messages are output close together in time, Windows uses MIDI running status, otherwise not. This was verified using the DOS mididump program. The running status messages are NOT received by the MIDI input callback under XP with the M-Audio Uno. This is true using both the driver that shipped with the Uno, and the Windows default driver. Note that with the latest M-Audio driver (MA_CMIDI_WDM_4.2.03v4.exe), the callback DOES receive events for the running status messages, but they are strangely garbled.

Attempts to reproduce the problem using the X-Session as input were not succesful. Perhaps it's the particular nature of WhorldRC's output (b0 00 xx b0 01 xx) that causes the bug? The most likely suspects are the Uno itself, its driver, or XP's USB MIDI driver. One way to test this would be by using a different MIDI device.

For the moment the workaround is a hack to WhorldRC: it sends a note off command after each set of bank/clip commands commands. This is a bit wasteful but WhorldRC is a low-bandwidth app and it seems to have no side effects.

meta-meta-meta-plugins

2007-02-03T00:02:00.001-08:00

The MetaFFRend beta is available now. We actually had a bit a of a celebration here tonight. Not only does MetaFFRend work, but we proved that nesting is unlimited. It's easy to make a meta-metaplugin, a metaplugin that contains metaplugin(s) which contain plugins. Or even weirder combinations, like a metaplugin which uses a mixer plugin to do automated mixing between two metaplugins. (!!) The level of nesting is arbitrary, and there's essentially no performance penalty for it, because the overhead of the base plugins is always huge compared to the overhead from nesting.

This is wild stuff, and not so impractical as it might sound. It means if I make a metaplugin that does something you like, and send it to you, you can use it inside one of your own metaplugins, add value to it somehow, and then send it someone else. By its very nature MetaFFRend encourages iterative composition, and distributed creation. It also means that you can simplify your own creative process by using grouping, in other words you can organize your own metaplugins in a hierarchy, with more basic effects at the bottom, e.g. a kaleidescope with automated rotation, and then reuse your building blocks (or other people's) in higher-level metaplugins.

Not only that, but embedding can be nested too, though ONLY if all the metaplugins at each level are "free", i.e. include the word "copyleft" in their author/license field. So for example if I send you one of my embedded metaplugins, and it's copyleft, you can embed it inside one of your metaplugins, and if you specify copyleft too, your metaplugin can also be embedded by someone else, and so on... And at each level the embedding works as expected, so the final plugin can be loaded into a host like any other plugin, and automatically unzips itself recursively the first time you load it. None of the recipients needs to worry about missing plugins. How about that.

The real point here is that it gives you, me, all of us a super-compact way to transport compositions. A complicated metaplugin might be 300K bytes, WITH embedding, without, less than 100K. Compare this to the cost of transporting a 30-second movie file, even compressed. The problem I've been trying to solve is that while Whorld operates in vector space, as soon as I apply effects to it, the output is video, which is a huge storage problem. But now I don't need to store anything. Instead of making a movie, I just send you a metaplugin that contains the Whorld plugin, running through a bunch of effects with automated parameters. It looks THE SAME as it did on my machine, but no goddamn gigantic movie file, and no compression either. You can render it to a movie file if you want, at any resolution you like, and if I used randomness in my Whorld patches (or in my automations), it will do something different every time you run it. The static movie file is replaced by an algorithm that's capable of dynamic behavior, user interaction, etc. Visual effects can be made quickly, without programming, even made live, and then easily transported, to be used and/or modified by others.

What this really does is push Whorld (and any other freeframe visual synths) back into vector space. It's a whole new type of content, all the more because it's N-dimensional (due the nesting capability), and distinct from making videos or coding plugins or VJing. It's metaplugin authoring, I don't know what else to call it, but hey, I'm drunk. Whatever it is, it's wicked object-oriented.

OK I'll shut up now and go drink some water. My head is starting to hurt and not just from programming.

RFC: MetaFFRend / FF meta-plugin authoring

2007-01-19T14:35:00.000-08:00

Request for Comments

OK, now that FFRend 1.3 is out the door, it's time to get serious about MetaFFRend. I hope to spark a bit of pre-coding discussion here, so that once the coding starts we can get it mostly right the first time. Here are some issues that have been bumping around in my head.

Summary

This is a Request For Comments regarding the VJ Forums proposal to turn FFRend into a plugin-authoring tool, somewhat similar to Pete Warden's FreeChain application. The idea is basically to export a FFRend project as a Freeframe plugin. This "meta-plugin" could be loaded into a Freeframe host software, and would behave just as if you were running the equivalent project in FFRend.

The meta-plugin may contain links to the plugins it requires, or alternatively, the plugins may be "bundled" into the meta-plugin. In the latter case, it's easier to distribute the meta-plugin, because there's no dependence on external files. Specifically, a meta-plugin will consist of FFRend's rendering engine, links to (or compressed copies of) the necessary plugins, and the project data (parameter settings, automations, etc).

Linked plugins and bundling

A meta-plugin contains links to other plugins. This works fine as long as the meta-plugin stays on the PC where it was created, but as soon as you distribute it, you have problems. There's very little chance that the absolute paths will happen to be correct on someone else's PC. IMO the solution has two parts. The first part is, the meta-plugin should search for its plugins in an intelligent way, e.g. the following steps, in order:

1) Try the absolute paths from the project data first.
2) Then look for an INI file, e.g. My Documents\FFRend\MetaFFRend.ini, which could contain a plugin path, e.g. PluginPath=C:\whatever;C:\foo;
3) Next, try My Documents\FFRend\Plugins.
4) Finally, look in the same folder the meta-plugin was loaded from.
5) If none of the above works, add an error message to a log file, e.g. My Documents\FFRend\MetaFFRend.log

The second part of the solution is, there should be an option to bundle all the plugins that the meta-plugin requires into the meta-plugin. This could make the meta-plugin quite large, so it probably makes sense to compress the plugins, e.g. using zlib which is open-source. The first time you load a bundled meta-plugin, it will unpack itself to the folder specified in MetaFFRend.ini, or if that fails, to My Documents\FFRend\Plugins. On subsequent loads it will detect that the plugins are already there, and skip the unpack. This means the meta-plugin could be slow to load the first time, but subsequent loads will be fast.

Meta-parameters

Like any other FF plugin, a meta-plugin can expose parameters to the host. These parameters are referred to as "meta-parameters". The consensus on this forum was that each meta-parameter should be able to control multiple things. When I say "things" I mean control targets; the potential targets in FFRend are not only FF parameters, but also modulator settings (frequency, waveform, etc.), bypass, and other properties. The reason why this is such an important capability, is that many hosts severely limit the number of FF parameters a plugin can expose. By allowing "grouping", the meta-plugin can make better use of a limited number of parameters.

Anatomy of a Meta-Plugin

The base MetaFFRend DLL (i.e. without any project data embedded in it) lives in the same folder as FFRend. It has a tag at the end of the DLL file, consisting of 16 characters: BASEMETAFFREND01

When FFRend creates a new meta-plugin, it starts with the base plugin. It removes the end-of-DLL tag and replaces it with the project data, followed by a different tag, also 16 bytes, like so:


char    Id[10];     // must contain characters MetaFFRend
WORD    Version;    // version number
DWORD   DataOfs;    // offset of project data from end of DLL file

What's nice about this scheme is it's easy to read the project data from the DLL: you just read the last 16 bytes, and now you know whether it's a valid meta-plugin, and if so, where to find the project data. Project data is a CArchive, and starts with a CMetaPlugin header:


CString m_PluginName;       // plugin name, 16 characters maximum
CString m_Description;      // a description of the plugin
CString m_AboutText;        // author and license information
int     m_BitDepthMask;     // mask of supported bit depths
int     m_PluginMajorVersion;   // number before decimal point
int     m_PluginMinorVersion;   // number after decimal point
int     m_NumInputFrames;   // number of input frames plugin expects
CDWordArray m_InpTargetIdx; // for each input frame, index of target sub-plugin, or -1 for default
bool    m_IsBundled;        // true if sub-plugins are bundled into meta-plugin
bool    m_IsCompressed;     // true if bundled sub-plugins are compressed
CMetaParamArray m_Param;    // information about each meta-parameter

class CMetaParam:

CString m_ParamName;    // parameter name, 16 characters maximum
float   m_Val;          // parameter's initial value
CMetaParamTargetArray   m_Target;   // array of parameter targets

class CMetaParamTarget:

int     m_PlugIdx;      // index of destination plugin, or -1 for misc. property
int     m_ParamIdx;     // index of destination parameter, or -1 for plugin property
int     m_PropIdx;      // index of property
float   m_RangeStart;   // start of parameter range
float   m_RangeEnd;     // end of parameter range

Heap-trashing bug also in RadialBlur, SpiralBlur & TimeBlur

2007-01-13T01:27:00.005-08:00

I previously reported finding a bug in Pete Warden's Mixer plugin. As it turns out, this bug also occurs in three other Pete plugins: RadialBlur, SpiralBlur, and TimeBlur. The original bug report for PeteMixer is here.

PeteMixer, PeteRadialBlur, PeteSpiralBlur, and PeteTimeBlur all have the same bug: they use MMX movq (64-bit move) in situations where movd (32-bit move) was intended. In all cases the results are the same: the two bytes immediately following the end of the output buffer get trashed, AKA munged, hosed, stomped, toasted, vaporized, etc.

Since the output buffer is almost certainly on the heap, the consequences depend entirely on how the host uses the heap. This probably isn't deterministic, so depending on the circumstances the bug could cause strange and wonderful behavior, or crash the host, or have no effect all.

I searched the plugin sources for other instances of movq, and found no other instances except in a few support modules. I haven't had time to wade through those modules yet, but my guess is they won't have the bug, since they weren't cut from the same cloth, as it were. The support modules that use movq are:

BoxFilter
Radiant
ImageMath

I have built UNOFFICIAL patched versions of the buggy plugins. You can download the binaries here, and the patched source files are here. PeteMixer was patched previously, but I included it in the above downloads for completeness.

I diffed the code carefully, and found no differences between my versions and the original that aren't "good" differences. For testing, I ran the original and the patched plugin side-by-side in FFRend, sent their outputs into a mixer, and A/B'd them to ensure that their output was identical.

Here are the relevant MD5 checksums:

BUGGY VERSIONS:
PeteMixer BUGGY.dll 5d7f65d48627e065f218bab5ee1d42a4
PeteRadialBlur BUGGY.dll 04551870addf5b3c8c14740e1aa7d316
PeteSpiralBlur BUGGY.dll b1e66597fbebeac46e59a208fd893aba
PeteTimeBlur BUGGY.dll c636cb633bc404f050497c4369f70241

PATCHED VERSIONS:
PeteMixer.dll a4c78bfa1a34f895264c9e20e3c6b035
PeteRadialBlur.dll 93592d40a4c2b9470f601c0e20042c1b
PeteSpiralBlur.dll f2f9b4f8d0740d170f87b173d9b1f83f
PeteTimeBlur.dll 031cc8426accc21df0989b34fe884f69

Adapting CSizingControlBar to work without idle time

2007-01-09T22:39:00.000-08:00

Since MFC generally relies on the existence of idle time, an MFC app that doesn't have idle time is arguably incorrectly designed. However, the theoretical multithreaded alternative is so difficult to implement correctly, that in practice, timer-driven MFC apps often do real work in their message loops, potentially using up all their idle time. My app (FFRend) is an example of this: since it processes video frames in OnTimer, if the processing becomes complex enough, idle time drops to zero.

This approach causes serious problems for Cristi Posea's otherwise delightful CSizingControlBar. It also causes problems elsewhere in the app, but these are minor issues related to the UpdateCmdUI mechanism, or the status bar message line. In an app without idle time, CSizingControlBar exhibits the following undesirable behaviors:

1. While the bar is docked, resizing it doesn't work: the bar is not repainted. This occurs because the sizing bar's implementation uses DelayRecalcLayout, which helps prevents flicker, but also relies on idle processing.

2. When the bar is floated, the main frame's layout is not updated. Again, the cause is DelayRecalcLayout. Note that this problem also occurs with other control bars, not only the sizing bar.

3. When the bar is floating, its Close button doesn't work: the bar remains visible, and left-clicking the edge of its frame causes the bar to shrink to a tiny rectangle.

4. The docked bar's close button doesn't react to mouse-overs.

I have found simple solutions for the first three problems. The last problem is still unsolved, but luckily it's the least serious one.

Problems 1 and 2 can both be solved by conditionally calling RecalcLayout from main frame's OnTimer handler. The condition is a simple test for pending idle layout, specifically, if (m_nIdleFlags & idleLayout) is true. I discovered this technique quite by accident in the "Professional UI Solutions" support forum.

 
void CMainFrame::OnTimer(UINT nIDEvent) 
{
    Sleep(40);  // emulate some work that consumes all our idle time

#if ENABLE_NO_IDLE_FIXES    // ck
    // CSizingControlBar uses DelayRecalcLayout, in order to avoid flicker when
    // resizing docked bars.  DelayRecalcLayout requires idle processing, which
    // normally isn't a problem, but this app typically doesn't have idle time.
    // Our solution is to periodically test the idle flags, and if there's idle
    // layout pending, call RecalcLayout.  This also ensures that the layout is
    // updated when controls bars are floated (any bars, not just sizing ones).
    if (m_nIdleFlags & idleLayout)
        RecalcLayout();
#endif

    CFrameWnd::OnTimer(nIDEvent);
}

Problem 3 is solved by customizing CMiniDockFrameWnd to handle WM_SYSMESSAGE. In OnSysMessage, if the ID is SC_CLOSE, hide the dock frame, via ShowWindow(FALSE).


void CSizingDockFrame::OnSysCommand(UINT nID, LPARAM lParam)
{
    CMiniDockFrameWnd::OnSysCommand(nID, lParam);
    // in the default implementation, if there's no idle time, the close button
    // doesn't work: the bar remains visible, and left-clicking the edge of its
    // frame causes the bar to shrink to a tiny rectangle
    if (nID == SC_CLOSE)
        ShowWindow(SW_HIDE);    // but this does work
}

Note that the derived dock frame must be installed in CMainFrame::OnCreate, by setting the CFrameWnd member m_pFloatingFrameClass, immediately after the call to EnableDocking.


    EnableDocking(CBRS_ALIGN_ANY);

    // use a custom dock frame that can handle zero idle time
    m_pFloatingFrameClass = RUNTIME_CLASS(CSizingDockFrame);

With these two simple, lightweight fixes, CSizingControlBar works quite happily in an app without idle time. Ever now and then there's a happy ending. Enjoy!

MetaFFRend : FFRend as a plugin authoring tool

2007-01-06T16:55:00.001-08:00

I plan to re-purpose FFRend, from an effects renderer, to a plugin authoring tool, somewhat like Pete Warden's discontinued FreeChain project. This idea was proposed by Leo Mayberry (AKA KillingFrenzy) on VJ Forums, and seconded by many others. FFRend will be able to output a freeframe "meta-plugin" that behaves identically to a given FFRend project. A meta-plugin contains *links* to other plugins. A meta-plugin is a DLL containing FFRend's rendering engine, plus some project data (which plugins to load, parameter and automation settings, signal routing, etc).

The project is in development, and about 50% done. I already have proof of concept: I can create a meta-plugin that runs a FFRend project from within any freeframe host. The main limitations are, 1) it's hard-coded which project the meta-plugin loads, and 2) no parameters are exposed to the host.

The biggest headache is creating the GUI within FFRend for deciding which parameters the meta-plugin should expose to the host (I call these "meta-parameters"). Ideally a meta-parameter should be able to control different types of properties, not only plugin parameters, but also oscillator settings and so forth, as MIDI already does. Also, it would be nice if a single meta-parameter could control multiple properties (i.e. grouping). This is important because many hosts severely limit the number of freeframe parameters.

The structure of the project data needs to be finalized soon. I'm working on that now. Meanwhile I'm also working on some ease-of-use features for FFRend: a dockable file browser with tabs for Projects, Plugins, and Clips, and a dockable preview window. Both features are commonly found in VJ software.

PeteMixer trashes 2 bytes beyond output buffer

2006-12-25T02:04:00.000-08:00

In version 1.15 of the Freeframe plugins, PeteMixer trashes the 2 bytes immediately following the output buffer. Version 1.14 also has the bug. I believe the bug also occurs in the AFX and VJo versions but I can't verify this directly.

[NOTE: this bug also occurs in PeteRadialBlur, PeteSpiralBlur, and PeteTimeBlur. For more info see here.]

The memory violation only occurs with the MMX version of the plugin. Interestingly, the non-MMX version of PeteMixer also fails, but in a different way: it always outputs black.

In the non-MMX version, the output colors are shifted right 16 when 8 was intended.


//  nOutputBlue>>=16;   // ck: shifting twice as much as needed
    nOutputBlue>>=8;

The MMX error is the use of movq when movd was intended. The output pointer pCurrentOutput is a 32-bit pointer, and it's being incremented by one (i.e. four bytes) for each iteration, which means that on the last iteration, writing 64 bits to *pCurrentOutput trashes the first two bytes of whatever happens to be above the output buffer in memory.


//  movq    [esi],mm7   // ck: TRASHES 2 bytes beyond output buffer
    movd    [esi],mm7   // ck: 32-bit move

The input pointers are also using movq, and while this doesn't overwrite memory it could cause a protection violation.

Theoretically the MMX version also uses more memory bandwidth than it needs to. I benchmarked it, and found that the corrected version is indeed slightly faster. For the original, 1000 calls to ProcessFrameCopy on a 640 x 480 frame took an average of 15.304 ms per frame, while the corrected version took an average of 14.820 per frame: a gain of 3%. I speculate that the gain is so small because the needless memory operations always hit L2 cache.

Heap-trashing bugs are notoriously difficult to find, and this one was no exception. They don't always cause a crash, and even when they do, it typically happens much later, in some unrelated component. In fact it was precisely this symptom--bizarre failures in things that never failed before--that pointed me in the right direction.

One disadvantage of writing such nice free plugins is that they're everywhere. I shudder to think how many mysterious crashes have been unjustly blamed on host applications. Ah, the joys and perils of inline assembler.

You can download an UNOFFICIAL patched DLL here. For more info see the patched source.

SetSurfaceDesc memory leak

2006-11-27T22:46:00.000-08:00

The CVideo object is leaking memory badly, and I'm surprised I never noticed it before. It leaks the entire DirectSurface every time a new video is opened. At 640 x 480 that's around 1 MB per open, which adds up fast. This could explain why Whorld misbehaves after many hours of triggering video clips.

Initially I suspected VfW, but then I verified that VfW definitely cleans up after itself. The problem turns out to be with DirectDraw, specifically with SetSurfaceDesc.

CVideo's constructor creates a default 1 x 1 memory surface. When a video is opened, CVideo attaches this surface to the video frame, using SetSurfaceDesc. This is a major optimization: it allows CVideo to avoid using GDI to blit each video frame to the DirectDraw surface, because the video frame IS the DirectDraw surface. In practice SetSurfaceDesc only needs to be called once, when the video opens, because VfW doesn't change the address of video frame after that. In fact it only changes the address if you open a video with a different frame size or pixel format, sensibly enough. CVideo checks for a change in frame buffer address, and if one occurs, it reattaches its surface to the new address.

According to the MSDN on SetSurface, "The DirectDrawSurface object will not deallocate surface memory that it didn't allocate. Therefore, when the surface memory is no longer needed, it is your responsibility to deallocate it. However, when SetSurfaceDesc is called, DirectDraw frees the original surface memory that it implicitly allocated when creating the surface."

I interpreted this to mean that once you've done at least one SetSurfaceDesc for a surface, you're on your own, as far as memory management. But what happens is, when the video is closed, DirectDraw leaves some object the same size as the frame buffer object allocated. It can't be VfW's frame buffer, because VfW destroys that when you call AVIStreamGetFrameClose. I can't imagine how or why this happens, but it sure isn't documented.

I only found two ways to make DirectDraw release this mysterious hidden frame buffer. The obvious way is to destroy the surface, but I'd prefer not to do this, because it means destroying and re-creating the surface every time a video is opened, which seems wasteful. The other way is to call SetSurfaceDesc again, passing it a 1 x 1 dummy surface. This works fine, and only takes about 50 microseconds. The surface description never changes, so it can even be a static array.


DDSURFACEDESC CVideo::m_DefSurf = {
    sizeof(DDSURFACEDESC),  // dwSize
    DDSD_WIDTH | DDSD_HEIGHT | DDSD_PITCH | DDSD_LPSURFACE | DDSD_PIXELFORMAT,  // dwFlags
    1,  // dwHeight
    1,  // dwWidth
    4,  // lPitch (Width * BitCount / 8)
    0, 0, 0, 0, // dwBackBufferCount, dwMipMapCount, dwAlphaBitDepth, dwReserved
    &m_DefSurfMem, // lpSurface
    {0, 0}, {0, 0}, {0, 0}, {0, 0}, // color keys
    {
        sizeof(DDPIXELFORMAT),  // dwSize
        DDPF_RGB,   // dwFlags
        0,          // dwFourCC
        32,         // dwRGBBitCount
        0xff0000,   // dwRBitMask
        0x00ff00,   // dwGBitMask
        0x0000ff    // dwBBitMask
    }
};
DWORD CVideo::m_DefSurfMem; // pointed to by m_DefSurf.lpSurface
...
void CVideo::Close()
{
    // if surface exists, we must attach it to a default 1 x 1 memory surface,
    // otherwise DirectDraw leaves a mysterious hidden frame buffer allocated
    if (m_Surface != NULL)
        m_Surface->SetSurfaceDesc(&m_DefSurf, 0);   // prevents a major leak
...

Undo performance

2006-11-24T11:08:00.000-08:00

The undo manager uses CArray to implement the undo history. As a result, the performance of undo notification varies significantly depending on whether undo is limited, or unlimited. Performance was measured using a test function that repeatedly generates the same undo event, as shown below. To simulate realistic conditions, the test function was called from the timer hook, and the results were stored in an array and written after the test, avoiding potential interference from file I/O.

If undo is unlimited, notification time is mostly constant, except when the CArray has to grow. Since growing entails copying the entire array to a new memory location, the time required to grow the CArray increases linearly with the number of undoable edits. In a test of 10000 iterations, undo notification took an average of 50 microseconds. The actual samples were nearly indistinguishable from the average, except when the array grew, resulting in peaks which increased linearly, up to 1.6 milliseconds by the end of the test. The time between peaks also increased linearly as expected, due to MFC's heuristic method of computing the grow size. There were also a few seemingly random, unexplained spikes of nearly 2.5 milliseconds.

If undo is limited, notification time is constant. This is because once the limit is reached, adding a new notification deletes the oldest event from the history. Deleting from the front of a CArray requires copying the entire array down one element, but the array size is constant, so there's no memory reallocation, and the time required to do the copy doesn't change. In a test of 10000 iterations, undo notification took an average of 60 microseconds, only 10 microseconds more than the unlimited case. The actual samples were similar to the average, with randomly-spaced peaks up to around 150 microseconds. Again there were some unexplained spikes, though they were an order of magnitude lower, around 250 microseconds.

Note that OnPlugBypass with undo notification commented out takes an average of 38 microseconds, so in all cases undo notification takes longer than other work performed by OnPlugBypass.

Conclusion: undo performance is suboptimal, due to the use of CArray. An implementation based on CList would almost certainly perform better for unlimited undo, and probably the same or slightly better for limited undo. This optimization needs to be weighed against substantially increased complexity in the undo manager, e.g. array indexing would have to be replaced by iteration.

This hypothesis was tested by slapping together a minimally functional CList-based implementation and repeating the test. The result: for unlimited undo, the average time was 48 microseconds, and the actual samples showed only minor deviations, e.g. 80 or 150 microseconds, except for the occasional unexplained 2.5 millisecond spike. On the other hand, the undo manager complications look pretty formidable.


static  const   MAX_SAMPS = 10000;
float   samp[MAX_SAMPS];
int samps = 0;
void CMainFrame::OnTimer(UINT nIDEvent)
{
    if (m_Plugin[0].IsCreated()) {
#if 0   // zero for unlimited undo
        if (!samps)
            m_UndoMgr.SetLevels(100);
#endif
        OnPlugBypass();
        if (samps == MAX_SAMPS) {
            FILE *fp = fopen("undo bench.txt", "wc");
            for (int i = 0; i < samps; i++)
                fprintf(fp, "%d\t%f\n", i, samp[i]);
            fclose(fp);
            exit(0);
        }
    }
    ...

#include "benchmark.h"
extern float samp[];
extern int samps;
void CFFPlugsDlg::OnPlugBypass() 
{
    int sel = GetCurSel();
    if (sel >= 0) {
CBenchmark  b;
        NotifyUndoableEdit(UCODE_BYPASS);
samp[samps++] = float(b.Elapsed());
        BypassPlugin(sel, !IsPluginBypassed(sel));
    }
}

Synchronizing automations to clip length

2006-11-09T21:34:00.000-08:00

The manual method is pretty straightforward, though it does require a calculator. Take FFRend's ideal frame rate (NOT the video clip's frame rate, that doesn't matter), and divide it by the video clip's frame count. Now multiply the result by 100. Enter that number in the Master speed toolbar, and you're all set, though you might also want to pause, rewind the clip, and sync the oscillators.

This scheme redefines the frequency unit, from Hertz to clip passes. A frequency of 1 will repeat once per clip pass, 2 will repeat twice per clip pass, .5 will repeat every other clip pass, etc.

The X 100 accounts for the fact that master speed is a percentage.

For example, if the clip is 1859 frames long, and it's playing at 25 FPS:

Master Speed = 25 / 1859 * 100 = 1.3448

frame buffer bit counts

2006-11-06T12:13:00.000-08:00

PlayerFF works in Resolume and Flowmotion, but not in OpenTZT, because OpenTZT passes 24-bit frames to the plugin, even though the screen resolution is 32-bit. The underlying problem is that you can't use DirectDraw to blit between surfaces with different bit counts. I tell my AVI reader (AviToBmp) to uncompress the video into the best format for the display (by passing AVIStreamGetFrameOpen AVIGETFRAMEF_BESTDISPLAYFMT). I use SetSurfaceDesc to turn the video frame into a DirectDraw surface, which means if my display is set for 32 bits, my video frame is also 32 bits, regardless of the actual color depth of the video. That's optimal if the host frame buffers also have the display's bit count, which you would think they would, but in OpenTZT, they don't for some reason, so the blit fails with error E_NOTIMPL.

AVIStreamGetFrameOpen can be also passed a BITMAPINFO that tells it what format to decompress to. This allows me force the the video format to match the host's format, as follows:


BITMAPINFOHEADER bih;
ZeroMemory(&bih, sizeof(bih));
bih.biSize = sizeof(bih);
bih.biWidth = m_pBmpInfo->bmiHeader.biWidth;
bih.biHeight = m_pBmpInfo->bmiHeader.biHeight;
bih.biPlanes = 1;
bih.biBitCount = 24; // or whatever host wants
m_pGetFrame = AVIStreamGetFrameOpen(m_pStream, &bih);

Another solution is to just accept that PlayerFF won't work in OpenTZT. Most VJ softwares don't need a player plugin anyway, because they already have elaborate media players built into them. Let's not forget that PlayerFF is primarily designed for use in FFRend!

Another problem: OpenTZT and Flowmotion display PlayerFF's output upside-down, but it looks fine in Resolume and FFRend. Something's pretty wrong there...

plugin ID must be unique in Resolume

2006-11-06T12:05:00.000-08:00

It appears that Resolume uses the plugin ID to keep track of its freeframe plugins. It came up because PlayerFF was using FFDemoSrc's plugin ID, and since FFDemoSrc happened to also be in Resolume's plugin folder, selecting PlayerFF actually selected FFDemoSrc instead. Flowmotion doesn't exhibit this behavior. So the freeframe documentation doesn't lie: plugin ID really does need to be unique! One wonders what non-authority is responsible for coordinating this...

PlayerFF: freeframe clip player

2006-11-06T10:02:00.000-08:00

I got my standalone freeframe clip player up last night. It's called PlayerFF (OK maybe it needs a better name). It handles AVI/BMP/JPG/GIF, and has three parameters so far:

Clip Select (which clip you're playing)
Pause (0 is play, any other value is pause)
Position (0 is the start of the clip, 1 is the end)

The clips are hard-coded at the moment. :(

Here's what I propose for clip management. The plugin should have both a "Clip Select" and a "Bank Select" parameter. It will look in the magical folder "\My Documents\PlayerFF". Any clips it finds there will wind up in bank zero, UNLESS the magical folder contains an optional playlist file. The playlist file must be called playlist.txt, and it contains the paths of the clips to load, one per line, with optional bank separators. Clips are loaded in the order they appear in the playlist, or if there's no playlist, in alphabetical order.

:0
C:\temp\avi files\Night Traffic.avi
C:\temp\avi files\earth1.avi
C:\temp\avi files\Boat Ride to Punta Sal (xvid).avi
C:\temp\avi files\01_24_04-med.avi
:1
C:\temp\avi files\tint.avi
C:\temp\avi files\kissinggirls.avi
C:\Chris\images\debbie\DSC_0080.jpg

plugin and project info can have different parameter counts

2006-11-06T09:59:00.000-08:00

I just found a neat bug. I added some parameters to my new PlayerFF plugin, and when I loaded up a FFRend project that uses it, there was garbage in the modulation settings for the new parameters.

It turns out I was assuming that the plugin's number of parameters, and the number of parameters I have information about in the project file, are always the same. That's normally the case of course, but a new version of the plugin with more (or less) parameters violates that assumption. Oops.

And the solution:


// the plugin's number of parameters might not match our info's parameter count,
// for example if it's a different version of the plugin; take whichever is less
int rows = min(GetPluginRows(PlugIdx), Info.m_Parm.GetSize());

how to keep frame counter from clobbering toolbar hints

2006-10-28T01:27:00.000-07:00

in CMainFrame::OnNotify:


case AFX_IDW_TOOLBAR:
    if (nmh->code == TBN_HOTITEMCHANGE) {
        LPNMTBHOTITEM   lpnmhi = (LPNMTBHOTITEM)nmh;
        if (lpnmhi->dwFlags & HICF_ENTERING)    // if entering toolbar
            m_HideFrameCounter = TRUE;  // hide frame counter
        else if (lpnmhi->dwFlags & HICF_LEAVING)    // if leaving toolbar
            m_HideFrameCounter = FALSE; // show frame counter
    }
    break;

How to get a huge file size

2006-10-28T01:26:00.000-07:00


static bool GetFileSizeEx(LPCSTR Path, LARGE_INTEGER& Size)
{
    HANDLE hFile = CreateFile(Path, GENERIC_READ, FILE_SHARE_READ, 
        NULL, OPEN_EXISTING, 0, NULL);
    if (hFile == INVALID_HANDLE_VALUE)
        return(FALSE);
    Size.LowPart = GetFileSize(hFile, (PULONG)&Size.HighPart);
    CloseHandle(hFile);
    return(Size.LowPart != 0xFFFFFFFF || GetLastError() != NO_ERROR);
}

More on matching original big hex

2006-10-20T01:25:00.000-07:00

Considerable progress has been made, using a special version of WhorldFF that reads jumps times from a list, i.e. a file containing a list of the frame numbers at which to do random jumps. The frame numbers were determined by painstaking experimentation.

The key concept is that in the version of FFRend that created original big hex, all of WhorldFF's parameters were initially set by the host. By comparison, in the current version of FFRend, parameters are only set if they differ from their defaults. So for example, number of rings defaults to .5. If it's .5 in the preset, it won't be sent to WhorldFF, so WhorldFF will use the patch's value (1000) instead of 154 (what .5 denormalizes to). The corrected preset uses a value of .500001 rather than .5, because this tricks FFRend into sending WhorldFF the parameter, but is also close enough to the desired number (.5) so that it makes no difference.

Another thing: Tile's Cell Width and Cell height have to start at .54, not .5! No idea why but it's crucial. With this change we get exact matching at 640 x 480, except for very minor variations near the jump points.

Oh and one more thing: original big hex's initial frame offset turns out to 65, not 67. This was obscured by the Tile Cell parameter error.

Minor detail: still # 4454 was mislabeled, it's actually 4554.

Iages 2288 and 2297 aren't correct even at 640 x 480, and since they're suspiciously close to the jump at 2283, it's likely that 2283 is misplaced. The 1105 image is also off (the jump is at 1104, a single frame before!).

qsort-based template class for sorting arrays

2006-10-17T01:06:00.000-07:00


template class CSortArray {
public:
    static  void    Sort(T *a, int Size, bool Desc = FALSE) {
        qsort(a, Size, sizeof(T), Desc ? CmpDesc : CmpAsc);
    }

private:
    static  int CmpAsc(const void *arg1, const void *arg2) {
        if (*(T *)arg1 < *(T *)arg2)
            return(-1);
        if (*(T *)arg1 > *(T *)arg2)
            return(1);
        return(0);
    }
    static  int CmpDesc(const void *arg1, const void *arg2) {
        if (*(T *)arg1 > *(T *)arg2)
            return(-1);
        if (*(T *)arg1 < *(T *)arg2)
            return(1);
        return(0);
    }
};

SetTimer granularity problem, and multimedia alternative

2006-10-15T01:05:00.000-07:00

SetTimer's granularity is 10 ms in w2k, and 15.625 ms in XP. As the following data shows, w2k can achieve exactly 25 Hz, but in XP the best fit is 21.33 Hz. Neither OS can achieve 30 Hz, and worse yet, they fail differently: a period of 33 ms gets you 25 Hz in w2k, and 21.33 Hz in XP.


available fequencies (w2k):
period  1..10   11..20  21..30  31..40  41..50  51..60
freq    100     50      33.33   25      20      16.67

available fequencies (XP):
period  1..15   16..31  32..46  47..62  63..78  79..93
freq    64      32      21.33   16      12.8    10.67

Note: in XP, weird behavior occurs near the above boundaries, e.g. at period = 31, freq oscillates between 31.03 and 30.52.

Multimedia timers appear to have 1 ms granularity, but are significantly more expensive in terms of CPU load. Note that that the callback runs in a system thread and must not post timer messages while OnTimer is running; otherwise if OnTimer consumes more than one timer period's worth of CPU time, the GUI will be non-responsive.


volatile BOOL   m_InTimer;  // true if we're in OnTimer
static void CALLBACK MMTimeProc(UINT uID, UINT uMsg, DWORD dwUser,  DWORD dw1, DWORD dw2)
{
    if (!m_InTimer)
        PostMessage((HWND)dwUser, WM_TIMER, 0, 0);
}

Matching big hex at HD high-res

2006-10-14T01:03:00.000-07:00

At 1920 x 1440, Zoom FF param should be (log(0.997395 * 3) + 1) / 2 = .737994
Must also compensate line width! Should be 3 * 6 = 18
Perfect match with WhorldFF, Kaleidascope and Tile. Problem lies further down the signal chain.
Timeblur is also a perfect match, Solarize is also fine, the problem is with Glow.
Hypothesis: if resolution is doubled, inner and outer radius must both be doubled also.
Result: It works fine up to 1280 x 960. At 1600 x 1200 and above, Glow exhibits unexpected behavior.
Bummer. Looks like we're limited to 1280 x 960.

NOTE that you can't change the desired frame rate in the options dialog without also compensating master speed, otherwise all the automations will be off, e.g. if you double the frame rate you must also double master speed.

Resolume plugins that aren’t compatible with FFRend

2006-10-13T01:03:00.000-07:00

Invalid bit depth (24-bit only):
iua_RectField.dll
resAsciiArt.dll
resCaptureScreen.dll
resChristmasBalls.dll
resLumaImage.dll
resPuzzle.dll
resResolumeBlocks.dll
resTracker.dll
resZxSpectrum.dll

Crashes:
resDelay.dll

Doesn’t work?
resFeedback.dll
resDelayBlend.dll

Making big images

2006-10-13T01:02:00.000-07:00

Recording at 1920 x 1080 (HD) works fine. On the hot rod, I get almost 2 FPS uncompressed and around 1 FPS using XVID. That seems pretty damn slow, but it's about 3000 times faster than Electric Sheep. The question is whether 1080 is good enough for generating poster-sized images, e.g. 11 x 17 at 300 DPI. HD 1080 is only 6.4 x 3.6 at 300 DPI. I tried 5100 x 3300 and it didn't seem to work even on the hot rod. There was memory to spare, so I'm guessing that the exponential increase in compute time would mean render times on the order of an hour per frame, like Electric Sheep's. It would be interesting to see whether it eventually coughs up some frames.

Matching the original big hex recording

2006-10-10T01:02:00.000-07:00

Hue parameter must be .50000001 which fools "don't set parm needlessly" test, so that hue gets set.

Since origin motion is random, WhorldFF does an initial jump even though the tempo is zero. It takes a while to settle down because of damping. The final location is:

x = .001251258888516
y = .563585314493240

After the initial jump it appears to stay put, but NO, it jumps again at around frame 5000. Why? Looks like a divide by zero problem somewhere, maybe CRealTimer::SetFreq isn't handling zero correctly.

It's hopeless, because the illustrated curves patch uses a random oscillator for poly sides. The random jumps are being triggered asynchronously by CRealTimer's thread, which causes both the origin sequence and the poly sides sequence to be unpredictable.

So the original can't be matched. Sorry! One option is to just live with big hex being different every time it'srecorded. That's either a cool feature or a pain in the ass, depending on your point view. Another option is to disable random jumps in the illustrated curves patch. That should make big hex deterministic, though this needs to be proved. (Yes, it's deterministic, provided you remember to restart the app before each recording).

Disabling random jumps also fixes the occasional inconsistencies in big hex's origin motion. Since whorld and the kaleidescope effect both have origin motion, they're adding or subtracting, and sometimes canceling each other out, which causes the origin to lurch, hesitate, or reverse. But is this good or bad behavior? Again it's subjective: it could be interesting, or annoying. I generally like the smooth origin motion better.

Experiment: big hex 66% with Al Fasawz. The animation moves too fast and spoils the mood of the music. Try a master pitch of 27%. That's the maximum speed reduction possible without losing some time blur (the original had time blur = .27, and .27 / .27 = 1.0, which is the maximum Freeframe parameter). Slow enough?

Artifact along right edge

2006-09-25T01:01:00.000-07:00

It's a narrow vertical strip at the far right edge, spanning the entire height of the image, that's shifted vertically by a few a pixels. It appears to come from Pete's glow effect and possibly from other effects as well. Check this and notify him. To remove it from frame captures, crop to 560 x 480 via PS Canvas Size.

Move bug

2006-09-19T01:01:00.000-07:00

After repeated drag moves, plugins weren't matching their titles and parameter rows. Inserting source first was a bad idea. The correct method is: copy source to a temp, delete source, insert destination, copy temp to destination. This also gets rid of the bump source/dest kludge.

Useful debug code that helped solve it:


void CFFPluginArray::Dump()
{
    printf("\n");
    printf("Count = %d, LastLoadedIdx = %d\n", m_Count, m_LastLoadedIdx);
    for (int i = 0; i < m_Count; i++) {
        CString s;
        if (m_Plugin[i].IsLoaded()) {
            CFFPlugin   *pp = &m_Plugin[i].m_Plugin;
            pp->GetPluginName(s);
            printf("%d: '%s' %s\n", i, s, m_Plugin[i].IsBypassed() ? "[BYPASS]" : "");
            for (int j = 0; j < pp->GetNumParams(); j++) {
                pp->GetParamName(j, s);
                if (m_Plugin[i].IsCreated()) {
                    printf("\t'%s' = %g\n", s, m_Plugin[i].m_Instance.GetParam(j));
                } else
                    printf("\t'%s'\n", s);
            }
        } else
            printf("%d: (empty)\n", i);
    }
}

master pitch problems

2006-08-13T01:00:00.000-07:00

Whorld's Master Speed parameter must be compensated for master pitch changes. This is non-trivial because it's a logarithmic value mapped to a linear normalized parameter. The formula to determine the correct Master Speed parameter for a given pitch is:


ff_speed = ((log(whorld_speed) / log(20)) + 1) / 2

where the nominal pitch is 1.0 (100%), half-speed is 0.5 (50%), double-speed is 2.0 (200%), etc.


Some examples:
whorld  FF
1.0     .5
0.5     .384311
2.0     .615689
.6666   .432326211
.27     .281466900

Time Blur must also be compensated. The solution turns out to be trivial: just divide the Time Blur by the pitch, i.e. if Time Blur is .27, and pitch is .5, Time Blur should be .54. The idea is that if you're creating half-speed data, you want to blur twice as many frames to achieve the same effect.

Other Whorld parameters:


ff_zoom = (log(whorld_zoom) + 1) / 2
ff_hue = (whorld_hue / 360)

CNumEdit bug

2006-08-13T00:59:00.000-07:00

A CNumEdit bug was discovered While attempting to compensate Time Blur's parameter for master pitch: CNumEdit notifies the parent before the aux window (aux is a CEditSlider in this case). The parent (a row dialog) sends a notification via SendMessage instead of PostMessage, and since the recipient (main frame) reads the value from the slider, the value is stale, because the slider hasn't been updated yet. The symptom: if a parameter is edited by typing text in the edit control and pressing tab, the slider moves to correct position, but the plugin doesn't respond to the change. The solution: CNumEdit should notify the aux window first, then the parent.

This is also a problem in Whorld, but it's masked because CMasterDlg::OnNotify uses PostMessage. Using SendMessage causes the same bug to appear. With the corrected CNumEdit, CMasterDlg::OnNotify can use SendMessage without problems. This is preferrable since SendMessage is theoretically more efficient.