You can download the utility from here: SWF to SVG/PDF. It is still very experimental and will certainly need some tuning in order to produce perfect output. In order to use it, you need, of course, Python and, also, the SWFTools. You need to add a few lines in lib/Makefile of SWFTools in order to create a dynamic library for Python:

librfxswf.$(SLEXT): $(rfxswf_objects) rfxswf.$(O) drawer.$(O) $(lame_in_source) $(h263_objects) $(as12compiler_in_source) $(as3compiler_objects) libbase$(A)
$(L) -g -shared $@ -o $^ -lz

The utility is split into two parts. The swf_lib.py uses Python ctype in order to communicate with the dynamic library and the swf_to_pdf.py is the main engine that performs the transformation. Currently shapes and text are supported. Eventually, more stuff could be implemented. It should be possible to transform a Flash application completely into an equivalent HTML5 application. I’ll leave that as an exercise for the reader!

You can see here an example of the utility:

• The original PDF.
• The SWF produced by SWFTools’ pdf2swf tool.
• The output SVG produced by the SWF to SVG/PDF utility presented here.
• The output PDF produced by the SWF to SVG/PDF utility presented here.

• Every column of type VARCHAR has an associated character set, which means that data in this column are stored in the specified character set.
• For every connection, the client can negotiate the character set that wants to send and receive. It can do that by issuing a SET NAMES SQL statement.

MySQL handles conversion between character sets when necessary. It can actually get really messy if you do not use the same character set everywhere. To demonstrate this fact, I have written a Python utility that tries combinations of character sets and operations and produces a nice summary in HTML. You can find the summary here: MySQL – character sets comparison tables.

The script uses the following MySQL character sets:

• latin1
• utf8
• greek
• binary

It performs this sequence of operations:

• Flush the table.
• Insert a row of data.
• Select all rows.
• Select the inserted row using a WHERE clause.
• Select the inserted row using a fulltext index.

This sequence is repeated for every combination of the following:

• Column character set.
• Character set during INSERT.
• Character set during SELECT.

The test data used are the first 4 (in order for the fulltext index to work) capital letters of the Greek alphabet. When working with the latin1 and binary character sets, these letters are encoded as UTF-8 (which is plain wrong, but it happens in bad configurations).
The output of the script consists of tables that describe the behavior for each case. The observed behavior fits in 5 categories:

• INSERT triggers error.
• SELECT … WHERE … triggers error.
• SELECT … WHERE … returns empty set.
• Almost correct behavior. Fulltext search does not work.
• Correct behavior.

Notice, also, that by looking at the main diagonal of the tables you can understand how the system behaves when the client uses a specific character set on a column of the specified character set. By looking at other elements of the tables you can understand what will happen if you decide to switch your client from a character set A to character set B. You can see, that most of the time, this switch will cause you trouble.

The add-on implements a JavaScript XPCOM component. This component is registered in chrome.manifest. The component registers an observer for http-on-modify-request topics. The observer’s callback function is called when a new HTTP request is ready to be sent. The function appends into a simple text file located on the user’s desktop a line consisting of the referrer, HTTP method and URL of the HTTP request.

You can download the extension and some other required extensions from here:

• auto count sql data provider extension
• auto count grid view extension
• sql table view extension

The application uses wxPython in order to provide a GUI. Also, it uses pywin32 in order to communicate with the Windows API. Both of these are requirements for executing the application. Also, it features the notion of protected services, which are not allowed to be manipulated. Starting, stopping or changing the start-up type of these services can result to operating system instability and/or boot failure.

The problem was that sometimes my monitor could not go into power saving mode. I had absolutely no idea why such a thing happened and I couldn’t correlate this problem with any user action I was performing each time the problem arose. I knew that I could fix this problem by just clicking Apply in the Power Options control panel applet and I, also, had figured out that the problem disappeared if I adjusted the inactivity period down to 1 minute.

If you don’t want to go into all the technical details that will follow, you may want to know that the cause for this problem is the Adobe Flash player. The problem happens after viewing a Flash video (e.g. YouTube) and there is a bug report for this issue: FP-5216. If you want to learn more please continue reading and/or watch the screen-cast: Part 1 and Part 2.

So, in the last few days, I decided to finally find the cause (and a possible solution) for this problem. The tools and references I used include:

• Process Monitor
• IDA Pro
• SoftICE
• leaked Windows 2000 source code

The process will be described in discrete steps:

1. The first thing I had to do was to discover where the monitor power off timeout was stored, who stored it and who accessed it. I started the Power Options control panel applet, then I started Process Monitor, made a change in the applet, clicked Apply and observed the output of Process Monitor. By inspecting the stack traces I was able to discover that ntoskrnl!NtPowerInformation was the function that stored the power policy into the registry (HKLM\System\CurrentControlSet\Control\Session Manager\Power\AcPolicy). This registry key is binary, so I couldn’t extract any more information from it.
2. The next step was to find about the NtPowerInformation function. So, I looked the Windows 2000 source code for references and I found some. The most interesting was the reference from the Power Options control panel applet itself (win2k/private/windows/shell/cpls/powrprof/powrprof.c). The most interesting function call looked like:
   

   NtPowerInformation(SystemPowerPolicyAc, InputBufferAc, sizeof(SYSTEM_POWER_POLICY), OutputBufferAc, sizeof(SYSTEM_POWER_POLICY));

   So, I had to find more information about the SYSTEM_POWER_POLICY structure. Fortunately, it is defined inside winnt.h and also has a DWORD member named VideoTimeout (offset 0xC0).

3. My next step was to open ntoskrnl.exe into IDA Pro and try to understand what NtPowerInformation did. Turns out that it calls PopApplyPolicy, which in turn does a memcpy of the given SYSTEM_POWER_POLICY structure to PopAcPolicy (which resides in kernel memory).
4. So, I used SoftICE to set a memory breakpoint at PopAcPolicy+0xC0 in order to determine who accessed the VideoTimeout field. The field was accessed from PopDispatchPolicyCallout.
5. Back to IDA Pro where I could see that PopDispatchPolicyCallout was sending the VideoTimeout field as a parameter to a function pointed by PopEventCallout. Now, I just had to learn where this pointer goes.
6. Back to SoftICE, it was easy to see that PopEventCallout contained the address of win32k!UserPowerEventCallout.
7. Now, I started reading the Windows 2000 source code. UserPowerEventCallout was in file win2k/private/ntos/w32/ntuser/kernel/power.c, it calls QueuePowerRequest, which calls xxxUserPowerCalloutWorker, which calls xxxUserPowerEventCalloutWorker, which does something like that:
   

   xxxSystemParametersInfo(SPI_SETLOWPOWERTIMEOUT, VideoTimeout, 0, 0);
   xxxSystemParametersInfo(SPI_SETPOWEROFFTIMEOUT, VideoTimeout, 0, 0);

   I looked for xxxSystemParametersInfo and found it inside win2k/private/ntos/w32/ntuser/kernel/rare.c. This function contains a huge switch statement. The SPI_SETLOWPOWERTIMEOUT and SPI_SETPOWEROFFTIMEOUT cases set two global variables: giLowPowerTimeOutMs and giPowerOffTimeOutMs respectively.

8. Now, the question is: Who reads these variables? Again, it’s time for SoftICE. I installed a memory breakpoint at these variables and within 1 second it was hit! The variables were accessed by win32k!IdleTimerProc.
9. Back to reading source code. IdleTimerProc is defined in win2k/private/ntos/w32/ntuser/kernel/input.c. First it check if it is time to display the screen saver and then it checks if it is time to turn off the monitor.
   • How does it check if it is time to take action? It calls IsTimeFromLastInput (defined in win2k/private/ntos/w32/ntuser/kernel/userk.h), which uses a global structure named glinp of type LASTINPUT. LASTINPUT is defined in win2k/private/ntos/w32/ntuser/kernel/userk.h and contains a field named timeLastInputMessage. This field, probably, holds the timestamp of the last user activity.
   • How does it turns off the monitor? Well, it is simple:
     

     PostMessage(gpqForeground->spwndActive, WM_SYSCOMMAND, SC_MONITORPOWER, LOWPOWER_PHASE);
     PostMessage(gpqForeground->spwndActive, WM_SYSCOMMAND, SC_MONITORPOWER, POWEROFF_PHASE);

     It sends a message to the foreground window. And what happens next? Well, probably the foreground window calls DefWindowProc and the message is handled internally. I looked the source code for references to SC_MONITORPOWER and found that this message is handled by the function xxxSysCommand which resides in win2k/private/ntos/w32/ntuser/kernel/syscmd.c. xxxSysCommand does a DrvSetMonitorPowerState call which probably goes directly to video card driver and that’s it!

10. By now, I had all the information that I needed. The question, though, remains: Where is the problem?
    1. First I thought that it might be a video card driver problem. So, I just wrote a simple Windows application, that posted the above mentioned message into its message loop and… tada… the monitor turned off!
    2. Then, I thought that it might be a rogue foreground window that does not call DefWindowProc. I didn’t check that. Besides the problem usually happened when I had only my desktop appear and no other active foreground windows.
    3. A wild guess now… let’s monitor the glinp structure. I’ve written a small driver that reads kernel memory and a small user-mode application that displays it every few seconds. Let it run and… boom… after exactly 1 minute of inactivity the glinp gets reset. How does that happen?
11. I went to SoftICE and set a memory breakpoint there. A little bit tricky because glinp was changed when I exited SoftICE resulting in an endless loop. Fortunatelly, SoftICE supports conditional breakpoints. After 1 minute, I found that glinp was reset from winlogon!RunScreenSaver.
12. I went to IDA Pro and found out that RunScreenSaver calls SystemParametersInfo(SPI_SETSCREENSAVETIMEOUT) which in turn resets glinp as seen from the source file win2k/private/ntos/w32/ntuser/kernel/rare.c. Actually RunScreenSaver does that only when it was unable to start the screen saver.
13. Why is the screen saver trying to activate itself? It shouldn’t do that. I check the global variable giScreenSaveTimeOutMs with SoftICE. Weird… it’s positive. When I booted the machine it was negative. Negative means that the screen saver is disabled. Somebody changed this value. But the code that changes this value inside win2k/private/ntos/w32/ntuser/kernel/rare.c takes care to maintain the sign of it. Now, somebody is doing something nasty. Let’s reset the value to negative and set a memory breakpoint through SoftICE to it. Now, let’s do our normal job on the computer. Hmm… nothing happens… Now what? Never mind, let’s watch a YouTube video! Bingo… the value is changed. npswf32.dll did it! Adobe Flash player plugin did it! How does it do it? Well…
    

    int x;
    SystemParametersInfo(SPI_GETSCREENSAVETIMEOUT, 0, &x, 0);
    SystemParametersInfo(SPI_SETSCREENSAVETIMEOUT, 0, 0, 0);
    SystemParametersInfo(SPI_SETSCREENSAVETIMEOUT, x, 0, 0);

    The above code snippet, changes the negative value into a positive one!

So the mystery is solved! How do I fix it? I set the screen saver wait time to 20 minutes and then disable it. Also, another way is just to open the Screen Saver control panel applet and click OK every time this problem happens and everything resets back to normal.

While I was investigating this issue, I noticed that there was no available utility that could detect and report installed Windows hooks. So here I present a Windows hooks detector. It is a command line application accompanied by a system driver, that scans, detects and reports installed Windows hooks. The source code and the binary compiled for Windows XP is available here: Windows hooks detector.

Also you can download a demo hook application, that installs local and global hooks for testing purposes. The source code and the binary is available here: Demo Windows hook application

According to the MSDN documentation for Windows hooks:

a hook is a point in the system message-handling mechanism where an application can install a subroutine to monitor the message traffic in the system and process certain types of messages before they reach the target window procedure.

A Windows hook is installed by calling SetWindowsHookEx (source code) and removed by calling UnhookWindowsHookEx. It can either be installed globally or locally. When it is installed globally it intercepts messages for all threads belonging to the same desktop. When it is installed locally it intercepts messages directed to a specific thread.

The Windows API does not offer any way for an application to list the installed hooks. After a hook is installed, it is stored and managed inside kernel structures which cannot be accessed from the user mode. Thus a kernel driver is necessary to access these structures. Furthermore, the whole mechanism of installing, handling and removing hooks is undocumented, which means that there is need to reverse engineer the relevant code. So after spending some time with the IDA Pro Disassembler, SoftICE and a Windows XP image running on VMware the job was done.

An overview for detecting installed global hooks follows:

1. Call PsGetCurrentThread and get the ETHREAD structure of the current thread. ETHREAD is an opaque data structure according to the MSDN documentation.
2. Extract the THREADINFO structure by calling PsGetThreadWin32Thread. Both of them are undocumented.
3. Extract the DESKTOPINFO.
4. There you can a find all the globally installed hooks. They are organized in a array. Each element is a linked list and corresponds to a specific hook (WH_*).

An overview for detecting installed local hooks follows:

1. Given a thread ID.
2. Call PsLookupThreadByThreadId and get the ETHREAD structure of the specified thread.
3. Extract the THREADINFO structure by calling PsGetThreadWin32Thread.
4. There you can a find all the locally installed hooks for the specified thread. They are organized in a array. Each element is a linked list and corresponds to a specific hook (WH_*).

Each hook is represented by a HOOK structure, which is undocumented. The HOOK structure stores the virtual address of the handler function. If the handler resides inside a DLL, then the relative virtual address is stored and the DLL path is stored in an atom table. That’s because a DLL’s base address may be different across processes. Also, by storing the DLL path in the atom table, the system knows which DLL to load if the hook is triggered inside a new process. We could say that the hook DLL is lazily loaded into processes.

Unfortunately, this atom table is internal. Also, there is no exported function of the kernel that accesses it. So the only way to read its contents is to call directly non-exported and undocumented kernel function calls. In order to get the necessary addresses, the utility uses the Debug Help Library to load the PDB of win32k.sys. Two symbols are needed:

• UserGetAtomName: A function that reads the above mentioned internal atom table.
• aatomSysLoaded: An array containing the atoms of hook DLLs. The necessary offset in this array is stored in the HOOK structure.

The utility has been tested with the following versions of win32k.sys: 5.1.2600.2180, 5.1.2600.2770, 5.1.2600.5512. These versions correspond to Windows XP SP2 and SP3. If you don’t have one of those versions and the utility crashes your system, please send me your win32k.sys for analysis. Otherwise, please inform me in order to update the supported versions.

There is one more way to get the DLL path for a hook, although it is less reliable, so I haven’t implemented it. Though, I am going to describe it. After loading a hook DLL the system stores in the PROCESSINFO structure the base address of the loaded DLL. So if the hook DLL is loaded in the inspected process, you can retrieve its base address and from the base address after traversing the module list of the process you can find the DLL path. If the hook DLL has not been loaded yet, then you are out of luck.

If you would like to compile the utility you need the Windows Driver Kit (WDK) and ddkbuild.

When you have to implement the state machine in hardware you have to find a way to map these abstract states into logical signals. There are a lot of possibilities. Two of the most common are:

• Use one signal for each state, e.g. if there are 3 states you must use 3 signals (S0, S1, S2). Each signal requires its own electrical path so this method becomes quickly inefficient as the number of states increases. On the other hand, it is quite easy to derive the transition (next state) functions and the output functions because you don’t have to deal with any special encoding of states, e.g. OUTPUT_A = STATE_1 OR STATE_2 can be easily implemented in hardware with a 2-input OR gate, because every state has its one signal.
• Encode states in binary, e.g. state 0 is encoded as 00, state 1 as 01, state 2 as 10. Each binary digit requires its own electrical path. That way we minimize the number of signals required but we increase the complexity of the transition and output functions.

The utility presented here has two features that can help designers implement their state machines in hardware:

• For a given mapping of abstract states to state encodings, it uses the Quine-McCluskey algorithm in order to find the minimum two-level logic functions that implement the state machine.
• It can try all permutations of state encodings for a given state word length in order to find the one that minimizes the total complexity of all the required functions for the implementation of the state machine. Be aware, that this mode of operation has exponential complexity so it can only be used for relatively small state machines.

You can find the Moore state machine solver and optimizer here: optistate.py

I improved the performance of the above mentioned implementations by:

• using bitwise operations,
• eliminating uses of strings and
• using Petrick’s method for determining all minimum sum-of-products solutions from a prime implicant chart.

I have also run line_profiler in order to pinpoint the lines of code that might need optimization if necessary.

You can find the Python module here: qm.py and the line_profiler output here: qm_line_profiler_output.txt.

Update 2011/05/19: I have improved the complexity calculation method. Before this change the complexity of a NOT gate was the same as the complexity of a 2-input OR gate, so instead of the minimal function NOT B, the function A OR C was produced. I have also written a function that provides an interface in case you would like to use this implementation instead of Robert Dick’s. The interface translates the output to match that of Robert Dick’s implementation.

def t(x,n):
  x = x[1]
  if x == '1':
    return ['X'*n]
  if x == '0':
    return ['X'*n]

  result = []
  for a,b in x:
    tmp = []
    for i in xrange(n):
      if a & 1:
        tmp.append('1')
      elif b & 1:
        tmp.append('X')
      else:
        tmp.append('0')
      a >>= 1
      b >>= 1
    tmp.reverse()
    result.append(''.join(tmp))
  return result

The utility is composed of several components:

• The WebBrowser component is responsible for transferring data from the configured URLs. It supports HTTP cookies, it can submit HTML forms in order to login into web sites and, also, uses the user agent string of Mozilla Firefox in order not to trigger weird behaviour in some web sites.
• The Storage component is responsible for storing the retrieved web pages into an SQLite database. Each web page is compressed using zlib and stored along with its URL and its retrieval timestamp. This component can, also, read from the database the two most recent versions of a web page for comparison.
• The HTMLReport component handles the generation of the HTML report that summarises the changes found in the configured URLs. It uses Mako Templates for Python in order to easily generate the desired HTML output.
• The ConfParser component handles reading the configuration file, which is written in YAML, through PyYAML.
• The BaseDiffEngine component is an abstract component which is the basis for creating components that compare specific web sites. Comparing HTML documents is accomplished through the lxml library. Each URL can be set in the configuration file to use a specific difference engine for comparison.

Along with the sitemon utility come some sample difference engines:

• The Comparison engine compares web sites line-by-line as if someone used the diff command line utility on them.
• The DiffInvision, DiffVBulletin and DiffPHPBB compare the respective forums (Invision’s IP.Board, vBulletin and phpBB). Actually these engines can compare the topic summary pages that these forums provide (so that you can see if a new post has been made).

You can download the sitemon utility along with the above described difference engines from here: sitemon.

Below you can see a sample configuration file for comparing the Dropbox home page after logging into it.

authentication:
  https://www.dropbox.com :
    method: post
    url: https://www.dropbox.com/login
    params:
      login_email: HIDDEN
      login_password: HIDDEN
      remember_me: on
      t: ''
sites:
  - url: https://www.dropbox.com/home
    diff_engine: Comparison
    validations:
      - xpath: //a[contains(@href,"/logout")]
        should_exist: Yes