Reverse Engineering a dwIoControlCode from DeviceIoControl

If you happen to Reverse Engineer Windows stuff, sooner or later you're going to want to translate a numeric dwIoControlCode from your assembly to something more palatable. To that end, head to: http://www.ioctls.net.

Compiling FreeDOS on Windows, using VMWare player

Might as well follow up on the previous post...

Here is how I set up an environment to be able to recompile the FreeDOS kernel, on Windows 7 x64, using the VMWare player.

1. Download the "FreeDOS Base CD" from here. Yes, I am well aware that there is also a "FreeDOS Base CD with source code" available, but we are going to recompile the latest kernel from SVN and in case I decide to upload my VMWare image to help y'all, I want the disk to be as little bloated as possible
2. Create a new VMWare image in Player and select "I will install the OS later" when prompted. A 512MB (0.5 GB) disk is all you need in size, and I'm probably being generous. On the other hand since we are going to compile, setting at least 512 MB RAM for the VM (rather than VMWare's default 16 MB) is probably a good idea. You can also remove virtual floppy and sound card as we don't need them
3. Set the VM to boot from the ISO you just downloaded in step 1, and start it. When prompted by FreeDOS, select "1) Continue to boot from FreeDOS from the CD-ROM" or press Enter, then select "1. Install to harddisk using FreeDOS SETUP (default)" and go through the installation steps. You may have to enter the VMWare BIOS (by pressing the F2 key) to make the CD-ROM first bootable device after partitioning in order to conplete the installation.
4. One installed, FreeDOS should boot from the hard drive and the first bad surprise you'll get is that the default of "2 - Load FreeDOS with EMM386+EMS and SHARE" results in "CONFIG.SYS error in line 24 - Illegal Instruction occured". You should select instead "1- Load FreeDOS with EMM386, no EMS (most UMBs), max RAM free" or "3 - Load FreeDOS including HIMEM XMS-memory driver"
5. Actually, let us start by fixing this inconvenience by issuing "edit fdconfig.sys" and changing the "MENUDEFAULT=2, 5" to "MENUDEFAULT=3, 5". Now that's better.
6. Power off the Virtual Machine and mount its virtual disk for transfer as highlighted in my previous post (or find another way that is convenient to you to transfer files to the disk)
7. Download the latest DOS 32 bit nasm from https://sourceforge.net/projects/nasm/files/DOS%2032-bit%20binaries/ and extract nasm.exe as well as cwsdpmi.exe from the archive to a \BIN directory on the FreeDOS hard drive
8. Download the latest DOS/i386 version of UPX from http://upx.sourceforge.net and extract upx.exe to \BIN
9. Download the latest open-watcom-c-dos-x.y.exe from http://ftp.openwatcom.org/ftp/ and copy it to your virtual FreeDOS hard drive (where is irrelevant)
10. Fetch the latest FreeDOS kernel using svn (eg. using TortoiseSVN on Windows) from https://freedos.svn.sourceforge.net/svnroot/freedos/kernel/trunk and copy the whole kernel\ directory to \src\kernel on your FreeDOS disk
11. Boot the FreeDOS VM again, and now run the open-watcom-c-dos-x.y.exe from wherever you copied it (which may now be listed as open-w~1.exe). Install Open Watcom as "Selective installation" and in "Toolkits and other components" remove "Sample programs". Also in that section, select the "Helpfiles" sub-menu and remove the "DOS Hosted Help Files". The installation will take a little while, but at least it's unattended. At the end, tell Open Watcom NOT to modify the AUTOEXEC.BAT and CONFIG.SYS files
12. Edit AUTOEXEC.BAT and edit the PATH line to have the following:
    set PATH %dosdir%\bin;C:\BIN;C:\WATCOM\BINW
13. Also in AUTOEXEC.BAT, after the line lh doslfn, add:
    set INCLUDE=C:\WATCOM\H
set WATCOM=C:\WATCOM
set EDPATH=C:\WATCOM\EDDAT
set WIPFC=C:\WATCOM\WIPFC
14. Reboot the VM
15. Go to C:\src\kernel and issue "copy config.b config.bat"
16. Edit config.bat and update the following:
    set XNASM=c:\bin\nasm.exe
    Uncomment all the relevant WATCOM options in the file
    It is also recommended to modify config.bat to have:
    set XCPU=386
set XFAT=32
17. Issue the command: build
18. If you followed everything properly, you should end up with a successful build of KERNEL.SYS and other files in C:\src\kernel\bin
    Note that you can clean up a previous build by issuing clean

Bonus: For those who don't want to have to go through all these steps, please find a ready-to-use VMWare image here (Player 4.x, 86 MB, 7z compressed), created using the exact steps above.

Rufus - The bootable USB Formatting Utility

It is my very great pleasure to introduce Rufus (direct downloads here), my own GPL'd version of a bootable USB formatting utility (DOS + ISO).

If you have been using the old HP Utility to create DOS bootable USB, you can throw that old thing away! In a small executable, and with no requirement for an installer, Rufus offers you a much better and up to date interface, with better features, and a DOS creation that doesn't rely on external files. It can also create bootable USB from ISO images. Plus you will find welcome additional features, such as the ability to check your USB stick for bad blocks. Best of all, and as you have come to expect from this site, it is 100% Free Software.

Please make sure you check the official Rufus page, and stop looking further when you need a DOS bootable USB stick. Of course, Rufus is compatible with all versions of Windows starting with Windows XP and lets you use the always awesome FreeDOS alongside the rather old and not up to date Windows Millenium DOS.

Why are you doing this?

Well, the truth of the matter is that, after having used the HPUSBFW utility for some time, it turns out that I really can't stand proprietary software utilities (as well as Windows' glaring shortcomings), so I decided to create my own Open Source version of an equivalent tool. Also, the fact that the many people, who have taken a stab at creating their own DOS bootable USB formatting utility, decided to go closed source doesn't really help. Simple utilities should only be Open Source, period.

Some interesting technical details (or yet another annoying technical rant)

You'd think there wouldn't be much to formatting an USB flash drive for DOS on Windows, but you would be wrong. As I already explained, there's some reason why Windows doesn't do it natively. Also, you may be surprised to hear that Windows doesn't actually provide a public API to format a drive, and instead you have to hijack an undocumented one called FormatEx, and which can be found in fmifs.dll. Then you will find that FormatEx kind of destroys your partition table when you want to use LBA, so you need to fix it manually. There's also this whole business of allowed cluster sizes. And then there's all the usual traps, such as having the partition boot records needing to be patched on XP, because unlike Vista or later, it equates an USB Flash drive to a floppy, as well as Windows' somewhat mysterious handling of Physical Drives vs. Logical Volumes. One will let you access the MBR and the other one the Partition Boot Record, yet, you still need to hold a lock to the latter to be able to access its underlying sectors with the former. Straightforward, it is not.

If you are so inclined, you'll find Rufus' FormatThread() in format.c as a good starting point. Oh, and would be quite ungrateful if I didn't acknowledge other OSS projects, such as ms-sys (boot records handling) or e2fsprogs (bad blocks check) for providing some of the building blocks used by the utility, as well as the talented designers from PC-Unleashed for the Rufus icon.

Finally, for those interested, the acronym stands for "The Reliable USB Formatting Utility, with Source".

Enjoy!

IMPORTANT NOTE: If you want support, please use the Github issue tracker. Any requests for support in the comments will be left unanswered.

Why Windows doesn't let you create an USB MS-DOS startup disk

Today, I'll provide an answer to the following question, which, if you're like me, must have irked you some when formatting an USB flash drive:

Yes why? Why is is that Windows has no problems creating an MS-DOS startup disk for the now useless floppy technology, but does not allow you to do so for the ubiquitous USB memory stick one?
It's not really like this is an impossible feat on Windows, as there exist quite a few tools that allow you to create one already (and I have now added my MUCH BETTER ONE to the mix). So why doesn't Microsoft do it?

If it ain't broke, don't fix it... But don't upgrade it either.

Ah, conservatism. I'm not talking about the political kind here, but rather its no less sinister computer equivalent: "Hey, we worked hard enough to get there. We don't see why we should take the risk of changing existing code, just so newcomers can reap the benefits."

You see, the way Windows creates an MS-DOS startup disk is not by doing anything smart, but by doing something lazy —which some also say is the second smartest thing to do after "smart"— such as including a 1:1 image of a 1.4MB floppy disk in a DLL (diskcopy.dll) and just splashing it wholesale onto a floppy when you request a startup disk.
Oh and yes, that's L-A-Z-Y-lazy since, if you actually have a look at the image file, you'll see that Microsoft didn't even spend much time creating a well thought-out one, but instead picked up the first Windows Millenium Edition (Me) startup disk they had lying around and just proceeded to "delete" stuff until it looked OK enough. The proof of it is that the image still contains much of the "deleted" content from the original disk and for a good read on how exactly the Windows Me Startup disk was "emasculated", you are very much invited to read Daniel Sedory's entry on the subject.

Of course, this whole technique of imaging a whole floppy does fall apart as soon as you're trying to create an unknown size USB bootdisk and this, my little children, is why Windows won't let you create DOS bootable USB flash drives to this day. End of story. Good night.

But wait, there's more!

Alright, alright, I hear you: that was a pretty underwhelming bedtime story. So of course there's more. And to answer your follow up question on why Microsoft didn't just compensate for the issue above by dropping the floppy image from diskcopy.dll and simply include the files themselves, we'll press on with a little bit of history.

As stated previously, the "DOS" files included in the image file that Windows uses when creating an "MS-DOS" startup disk actually come from a Windows Me boot disk, which, amongst its many disappointments, is also well known for crippling the DOS version it came with. As Wikipedia indicates, "One of the most publicized changes from Windows 98 is that Windows Me does not include real mode MS-DOS". Furthermore, departing from any MS-DOS that came prior, "the Autoexec.bat and Config.sys files are used only to set global environment variables". Thus DOS 8.0, which is the one included with Windows Millenium, is indeed seen by many as crippled, whereas DOS 7.1, a.k.a. the one from Windows 98 SE, is often labelled the "last good DOS".

But much more relevant to the USB boot issue is that the Windows Me DOS is not actually able to produce a command prompt on anything else but a floppy media. That is to say, even if you were to create all the required boot records on an USB drive, and then copy the unmodified COMMAND.COM and IO.SYS from the diskcopy.dll floppy image, your drive will never boot. You can try this for yourself, if you have an existing bootable stick, as 7-zip will happily extract the files from diskcopy.dll if you want to copy them over.

That still doesn't explain anything...

"OK fine," you say, "Windows Me is unusable but Microsoft should have the rights to their own DOS stuff anyway, so why don't they either just use the DOS files from Windows 98, or produce an uncrippled new version of the WinMe DOS, and go with that?". That's actually a very valid question, which people have only been able to speculate onto this day. Technically and most likely legally, there doesn't exist a reason why Microsoft shouldn't be able to provide a proper set of DOS files to boot an USB stick with. Furthermore, and this is the trick people use when they want to legally provide MS-DOS files for USB boot on Windows, uncrippling the WinMe DOS files from the diskcopy.dll to get them working on USB is actually trivial and documented here. For the record, even the well known HPUSBFW tool, from HP will patch the COMMAND.COM and IO.SYS files according to the above, when provided with WinMe DOS files. Yet, Microsoft went as far as explicitly discouraging people from doing so...

Thus, the moral of our story is that, while there is no real technical limitation to it, there is no satisfying explanation as to why Microsoft decided to inconvenience millions of their users by not allowing to create a DOS bootable USB drive. However, if you keep watching this blog, you will soon be provided with a 100% Open Source tool that does just that (and more!), so stay tuned.

Bonus

By the way, if you get your hands on an older version of the HPUSBFW tool --the one that is about 400 KB in size, you should be advised that it contains, as a resource, a self extracting UPX compressed executable that will extract the Windows 98 DOS files. The utility itself is designed to only enable extraction of these files if specific HP USB hardware (VID:PID 03F0:0023) is detected, however that can easily be patched to enable any device (F0 03 75 15 → F0 03 90 90 & 23 00 75 0C → 23 90 90 0C) and one could use HPUSBFW as a standalone USB DOS boot utility.

cathash

cathash is a multiplatform MS CAT/Authenticode SHA-1 generation, inspired by cathash by Michel I. Gallant, but not using CryptCATAdminCalcHashFromFileHandle, and thus able to run on UNIX platforms, including big-endian ones.

It is intended for the computation of the custom SHA-1, used by Microsoft and others, to verify the authenticity of Windows executable (or regular files), using the algorithm detailed at the end of the "Windows Authenticode Portable Executable Signature Format" specifications.

The utility was heavily validated against Windows system files (32 and 64 bit), MSVC generated files, MinGW/MinGW-w64 generated files and so on, so it is expected to be fully compliant with the MS SHA-1.
If compiled on Windows, and unless you comment the VALIDATE_HASH define, the program will also validate its computation against the one from CryptCATAdminCalcHashFromFileHandle, for extra safety.

By the way, if you are interested in producing your own implementation, you may want to note that the MS specs omit the fact that the optional extra PE data, starting at SUM_OF_BYTES_HASHED, needs to be padded to the next 8 byte boundary for hashing.

Outside of its upcoming libwdi usage, this utility may come handy for UNIX users needing to validate the authenticity of Windows files.
The source, which comes along with a (signed) 32 bit Windows executable can either be downloaded here (direct link) or here (SourceForge).
You can also directly access cathash.c, to compile it using any of Visual Studio, WDK, MinGW, cygwin or UNIX based gcc.

IMPORTANT NOTE: As reported by Abid Bhat, some Windows XP system files, such as C:\Windows\system32\drivers\update.sys, C:\Windows\Driver Cache\i386\sp3.cab or C:\Windows\Driver Cache\i386\driver.cab, are not getting the expected hash when computed by the current version of cathash. I'll look into it when I get a chance, but it may take a while...

bin2coff

As part of the ongoing effort to add 7z compression to libwdi, I am publishing bin2coff, which is an improved version of the tool of the same name by Vortex. bin2coff is aimed at generating MS COFF object files that are simple wrappers for binary data, and that can be used by the MS (Visual Studio, WDK), MinGW (MinGW32, MinGW-w64) and other linker tools. Such a tool can be especially useful if you want to generate a library that contains a large chunk binary data. For instance, and this is what I plan to use it for in libwdi, you can use it to embed a complete 7z archive in a cross-compiled Win32 static library file, which you will then be able to extract at runtime.

What this version of bin2coff brings is 64 bit compatibility (MinGW-w64), the provision of a size variable besides the binary data, as well as the full sourcecode. A 32 bit Windows executable is also included in the archive, which you can either download here (direct link) or here (SourceForge).

Usage: bin2coff bin obj [label] [64bit]

  bin  : source binary data
  obj  : target object file, in MS COFF format.
  label: identifier for the extern data. If not provided, the name of the
         binary file without extension is used.
  64bit: produce a 64 bit compatible object - symbols are generated without
         leading underscores and machine type is set to x86_x64.

With your linker set properly, typical access from a C source is:

    extern uint8_t  label[]     /* binary data         */
    extern uint32_t label_size  /* size of binary data */

Addon:
If, for some foolish reason, or just because you like it when things blow up in your face, you want to integrate bin2coff generation with libtool, you will find that you need to generate a .lo wrapper. In that case, the following Makefile excerpt might be of interest to you:

wdi_data.lo: bin2coff
# $* is the target without extension - here 'wdi_data'
 bin2coff $*.7z $*.o $(bin2coff_OPTIONS)
# libtool won't let us get away with a mere .o - we have to generate a .lo wrapper.
# What's more, the first comment from the .lo is mandatory and must contain the libtool version
 @echo "# Generated by $(shell $(LIBTOOL) --version | head -n 1)" > $@
 @echo "pic_object='$*.o'" >> $@
 @echo "non_pic_object='$*.o'" >> $@

Access denied and Windows 7

One of the annoying aspects of using NTFS is that, from time to time (eg. if Windows 8 preview corrupted your Windows 7 partition so bad that you had to reinstall the whole OS - see post #7), Windows will decide that you are no longer entitled to access your files, and while performing some operation or other you'll get something like:

There's nothing more infuriating in terms of computing than being denied access to files you should have access to.

The solution? Open an elevated command prompt and issue:

takeown /R  /F <Drive or directory>
Icacls <Drive or directory> /reset /T

Building and running Clang static-analyzer on Windows/MinGW

With the Clang static-analyzer becoming more and more popular these days, MinGW users on Windows might be looking for some way to also bring the Clang goodness to their shores.

However, well, let's just say that the LLVM documentation isn't that intuitive for newcomers, especially if you were expecting to be able to download a nice Windows binary package and roll. This quick recipe on setting up a complete Clang static-analyzer environment for MinGW attempts to remedy that. As usual, what is presented below worked on my environment at the time of the post - I make no guarantee that it will work elsewhere, especially if you choose to deviate from the guide.

Also, before you start, please be aware that this whole process will send you back about 11 GB in terms of disk space, so make sure you have adequate room.

1. Install MinGW (32 bit!) using the latest msys-get-inst (just run the GUI installer) picking up the very latest packages.
   For the installation directory, I used D:\Clang, which I will assume to also be yours for for the rest of the guide - please modify accordingly. With regards to the installation packages, select MinGW Compiler Suite → C compiler (which should be selected by default), MinGW Compiler Suite → C++ compiler, and MinGW Developer Toolkit to ensure that binutils and all the necessary tools to build clang, except Python, are available. You should also select MSYS Basic System, so that a shell is available. You probably also want to play it safe by picking up the pre-packaged repository catalog for MinGW. The retrieval and installation of the packages will take a few minutes.
2. Download Python v2.x (32 bit installer) from http://www.python.org/download/.
   Use D:\Clang\bin as the installation directory and unselect everything but Python Interpreter and Libraries, as this is all you really need from Python.
   IMPORTANT: Using python 3.2.2 produced an error on my machine during the build, so make sure you pick v2.x.
3. [OBSOLETE: You can skip to the next step as per comments below]
   Download the latest (Experimental) Clang Binaries for Mingw32/x86 from http://llvm.org/releases/download.html. Extract its content (bin/, include/ ...) into D:\Clang, overwriting any existing files. We do not care about headers being overwritten as this setup will be used for Clang and Clang only. Always make sure llvm-gcc is installed in the /bin directory of MinGW, unless you want to see a CreateProcess() error when running it (or want to have to duplicate the binutils files which we already got installed from picking up the MinGW Developer Toolkit).
4. After installing an Subversion client such as Tortoise-SVN if needed:
   
   • Checkout llvm from http://llvm.org/svn/llvm-project/llvm/trunk (this is a Subversion URI), into D:\Clang\msys\1.0\src\llvm
   • Checkout clang from http://llvm.org/svn/llvm-project/cfe/trunk, into D:\Clang\msys\1.0\src\llvm\tools\clang
     NB: you may have to checkout clang outside of llvm first, then move it to the right location if your Subversion client doesn't let you checkout a repository inside an existing repository.
5. Open an msys shell by launching D:\Clang\msys\1.0\msys.bat and issue the following commands:

   cd /src
   mkdir build
   cd build
   export CC=gcc
   export CXX=g++
   ../llvm/configure --disable-docs --enable-optimized --enable-targets=x86,x86_64 --prefix=/mingw
   make -j2

   The --prefix ensures that our files will be installed in the bin directory of MinGW and the two export lines avoid the need to install the Clang binaries to be able to recompile (As per now obsolete Step 3). Note that, in the msys shell, /mingw always point to the root directory where you installed MinGW (here D:\clang) regardless of the name you used. I would strongly advise against using a different directory, or not specifying the --prefix option (in which case /usr/local will be used as default), as I ran into issues when I tried (the static-analyzer failed to locate <stdio.h> when parsing sources). As usual, the -j2, which specifies the number of jobs to run in parallel, is there to make use of as many cores as we have (here two). Adjust according to your platform.
   Oh, and if you want to know why we use --enable-optimized, see the very end of this post...
6. Go for a walk or something. Some of the libraries and executables are huge (hundreds of MB in size) so this whole process will take a while. During the various compilation runs I conducted, I chanced in one that froze (one make process using 100% of the CPU and never completing) and another where Windows reported "llvm-tblgen has stopped working", so don't be surprised if the compilation doesn't work quite right first time 'round -- it's fairly heavy on resources.
7. If the compilation completed, then you can run:

   make install -j2

   Note that this operation will take some time too!
8. Copy all the content of D:\Clang\msys\1.0\src\llvm\tools\clang\tools\scan-build, except c++-analyzer to D:\Clang\bin.
   Copy all the content of D:\Clang\msys\1.0\src\llvm\tools\clang\tools\scan-view to D:\Clang\bin.
9. Issue the command:

   ln -s /mingw/bin/ccc-analyzer /mingw/bin/c++-analyzer

   The reason we're re-creating this symbolic link (which on Windows ends up being a duplicate rather than a link anyway) is that the one from the file provided by LLVM only works on UNIX systems.
10. Edit D:\clang\bin\scan-build, line 100, and remove the /bin there to have:
    my $ClangSB = Cwd::realpath("$RealBin/clang");
    If you don't do that, scan-build will look for clang in /mingw/bin/bin and obviously fail to locate it.
11. Finally, since we had to override them, you shouldn't forget to undefine the CC and CXX environmental variables:

    export -n CC CXX
    unset CC CXX

Success looks like what?

$ cd /d/libusb-pbatard
$ scan-build ./autogen.sh
(...)
$ scan-build make
make  all-recursive
make[1]: Entering directory `/d/libusb-pbatard'
Making all in libusb
make[2]: Entering directory `/d/libusb-pbatard/libusb'
  CC     libusb_1_0_la-core.lo
core.c:1626:3: warning: Value stored to 'r' is never read
                r = 0;
                ^   ~
1 warning generated.
  CC     libusb_1_0_la-descriptor.lo
  CC     libusb_1_0_la-io.lo
io.c:1819:8: warning: Call to 'malloc' has an allocation size of 0 bytes
        fds = malloc(sizeof(*fds) * nfds);
              ^      ~~~~~~~~~~~~~~~~~~~
1 warning generated.
  CC     libusb_1_0_la-sync.lo
  CC     libusb_1_0_la-poll_windows.lo
  CC     libusb_1_0_la-windows_usb.lo
os/windows_usb.c:1218:26: warning: Value stored to 'discdevs' during its initialization is never read
        struct discovered_devs *discdevs = *_discdevs;
                                ^          ~~~~~~~~~~
os/windows_usb.c:1297:4: warning: Value stored to 'session_id' is never read
                        session_id = 0;
                        ^            ~
2 warnings generated.
  RC     libusb-1.0.lo
  CC     libusb_1_0_la-threads_windows.lo
os/threads_windows.c:115:2: warning: Value stored to 'prev_pos' is never read
        prev_pos = pos = NULL;
        ^          ~~~~~~~~~~
1 warning generated.
  CCLD   libusb-1.0.la
Creating library file: .libs/libusb-1.0.dll.a
make[2]: Leaving directory `/d/libusb-pbatard/libusb'
Making all in doc
make[2]: Entering directory `/d/libusb-pbatard/doc'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/d/libusb-pbatard/doc'
Making all in examples
make[2]: Entering directory `/d/libusb-pbatard/examples'
  CC     xusb.o
xusb.c: In function 'test_mass_storage':
xusb.c:443:9: warning: variable 'junk' set but not used [-Wunused-but-set-variable]
xusb.c:520:4: warning: Value stored to 'junk' is never read
                        junk = fwrite(data, 1, size, fd);
                        ^      ~~~~~~~~~~~~~~~~~~~~~~~~~
1 warning generated.
  CCLD   xusb.exe
./.libs/lt-xusb.c:692:11: warning: Value stored to 'len' during its initialization is never read
      int len = strlen (new_value);
          ^     ~~~~~~~~~~~~~~~~~~
1 warning generated.
  CC     lsusb.o
  CCLD   lsusb.exe
./.libs/lt-lsusb.c:692:11: warning: Value stored to 'len' during its initialization is never read
      int len = strlen (new_value);
          ^     ~~~~~~~~~~~~~~~~~~
1 warning generated.
make[2]: Leaving directory `/d/libusb-pbatard/examples'
make[2]: Entering directory `/d/libusb-pbatard'
make[2]: Leaving directory `/d/libusb-pbatard'
make[1]: Leaving directory `/d/libusb-pbatard'
scan-build: 8 bugs found.
scan-build: Run 'scan-view /tmp/scan-build-2011-10-24-1' to examine bug reports.

This is a typical result that Clang will (eventually - the analysis takes some time!) churn through. One false positive (that 'junk' variable is meant to be ignored, and only serves to eliminate a warning about not reading the return value from fwrite) and apart from that malloc code, everything looks fairly minor. Even on its very first report from clang, which of course we will act upon, libusb-1.0 on Windows doesn't appear to fare too bad!

Bonus:
If you followed the guide above, chances are you'll be using separate installations for the standard MinGW/MinGW-w64 and the Clangified ones. As such, you may want a title on the msys shell window that reflects that you are in a Clang environment, to identify which is which more easily.
To change the msys commandline window title, just edit your D:\clang\msys\1.0\etc\profile, locate the line:

export PS1='\[\033]0;$MSYSTEM:\w\007

and replace $MSYSTEM with the title you want.

Bonus 2:
Displayed after spending more than one hour of compilation, using gcc:

llvm[0]: ***** Completed Debug+Asserts Build
llvm[0]: ***** Note: Debug build can be 10 times slower than an
llvm[0]: ***** optimized build. Use make ENABLE_OPTIMIZED=1 to
llvm[0]: ***** make an optimized build. Alternatively you can
llvm[0]: ***** configure with --enable-optimized.

Now you tell me!?! Shouldn't that be the kind of info you want to display after someone issues configure?

Avoiding Program Compatibility Assistant warnings on Windows

If you're developing applications on Windows Vista or later, you may end up in a situation where this rather annoying little fellow pops up after your program execution:

"What gives?, you say, my program's not an installer and it doesn't have anything like 'setup' or 'install' in the executable name! Why does Windows pester me and my users with such a warning?" The answer: have a look at your resource file (RC, etc) and especially your versioning strings, such as "FileDescription" or "ProductName". If there's any or part of the taboo keywords in there, such as "setup", "install", "update", Windows assumes that the executable is an installer, and also assumes the worst. Thus, the solution is to never use any or part of these keywords in any of the location that Windows will look for.

For more information on how Windows "heuristically" detects installation programs, have a look at the Installer Detection Technology paragraph from the following TechNet note.

Windows driver utilities and other good stuff

Courtesy of Xiaofan.

If you test driver installations quite often under Windows 7, then PnPUtil from Microsoft can be quite useful.

The RAPR GUI driver store management utility can also help easily identify the inf associated with a driver.

Note that PnPUtil has a bug under Vista that output redirection does not work and it affects RAPR as well. Travis, of libusbK, and libusbDotNet fame has a console version of of driver store management utility located here. The download includes the source as well.
Those who are interested can probably enhance it and create a GUI version for it. Do not use it under XP though.

32 bit apps, Windows x64 and the System32 directory

While adding a hidden magic key feature to Zadig, that deletes the libusb-1.0 DLLs that are installed by the libusbK driver in System32\ and SysWOW64\ (when you are developing for both libwdi and libusb, the multiplication of libusb DLLs becomes a major issue), I found that a 32 bit Zadig application running on x64 Windows would not see the libusb-1.0.dll residing in System32\.

The reason for that is that, from a 32 bit app perspective, Windows maps SysWOW64\ (the directory that, as its name does NOT indicate, contains 32 bit binaries) to System32\, and therefore trying to access the actual System32\ (the directory that contains 64 bit binaries) using %WINDIR%\System32.

If for any reason you want to access the actual System32\ directory from a 32 bit application running on Windows x64, you have to use Sysnative (eg: C:\Windows\Sysnative).

Enabling the Issuer Statement button on a Windows certificate

You may have seen a few of these certificates where clicking on the "Issuer Statement" button brings you to a specific webpage, usually a "Certification Practice Statement", or CPS, on the issuer's webserver. Maybe you've wondered how you could achieve the same on certificates you generate? Well, wonder no more.

It basically all boils down to adding a CPS Policy Identifier in the Certificate Policies field, and the Windows PKI API provides everything you need to do so:

  DWORD dwSize;
  CERT_POLICY_QUALIFIER_INFO certPolicyQualifier;
  CERT_POLICY_INFO certPolicyInfo = { "1.3.6.1.5.5.7.2.1", 1, &certPolicyQualifier };
  CERT_POLICIES_INFO certPolicyInfoArray = { 1, &certPolicyInfo };
  CHAR szCPSName[] = "http://cps-page.yourserver.com";
  CERT_NAME_VALUE certCPSValue;
  CERT_EXTENSION certExtension;
  CERT_EXTENSIONS certExtensionsArray = { 1, &certExtension };

  // Set the CPS Certificate Policies field - this enables the "Issuer Statement" button on the cert
  certCPSValue.dwValueType = CERT_RDN_IA5_STRING;
  certCPSValue.Value.cbData = sizeof(szCPSName);
  certCPSValue.Value.pbData = (BYTE*)szCPSName;
  if ( (!CryptEncodeObject(X509_ASN_ENCODING, X509_NAME_VALUE, (LPVOID)&certCPSValue, NULL, &dwSize))
    || ((pbCPSNotice = (BYTE*)malloc(dwSize)) == NULL)
    || (!CryptEncodeObject(X509_ASN_ENCODING, X509_NAME_VALUE, (LPVOID)&certCPSValue, pbCPSNotice, &dwSize)) ) {
    printf("could not setup CPS\n");
    goto out;
  }

  certPolicyQualifier.pszPolicyQualifierId = szOID_PKIX_POLICY_QUALIFIER_CPS;
  certPolicyQualifier.Qualifier.cbData = dwSize;
  certPolicyQualifier.Qualifier.pbData = pbCPSNotice;
  if ( (!CryptEncodeObject(X509_ASN_ENCODING, X509_CERT_POLICIES, (LPVOID)&certPolicyInfoArray, NULL, &dwSize))
    || ((pbPolicyInfo = (BYTE*)malloc(dwSize)) == NULL)
    || (!CryptEncodeObject(X509_ASN_ENCODING, X509_CERT_POLICIES, (LPVOID)&certPolicyInfoArray, pbPolicyInfo, &dwSize)) ) {
    printf("could not setup Certificate Policies\n");
    goto out;
  }

  certExtension.pszObjId = szOID_CERT_POLICIES;
  certExtension.fCritical = FALSE;
  certExtension.Value.cbData = dwSize;
  certExtension.Value.pbData = pbPolicyInfo;

  // Call CertCreateSelfSignCertificate() with certExtensionsArray as the last parameter or something...
  (...)

out:
  // Cleanup...
  if (pbCPSNotice != NULL) free(pbCPSNotice);
  if (pbPolicyInfo != NULL) free(pbPolicyInfo);

For a real-life example of the above, as well as the setting of other certificate properties such as the Enhanced Key Usage or an URL in the Subject Alternative Name, you may want to have a look at the LGPL pki.c from libwdi, and our CreateSelfSignedCert() function there.

Oh, and we pretty much rewrote basic LPGL redistributable versions of the MakeCat/Inf2Cat, MakeCert, CertMgr and Signtool WDK utilities, so if you need them, there's here as well. For additional details, look here

Programatically setting and applying Local Group Policies on Windows

One of the most annoying aspects of device driver installation on Windows, and one of the main gripe people are expected to have with the current libwdi (which has since been fixed programmatically using the technique exposed below), is the Windows default of creating a system restore point each time it installs a new driver. Having seen restore point creations that take more than 5 minutes, and ending up in failure as a result, since 5 minutes is the default timeout for a point creation, I have been looking at ways to change this default behaviour.

Enters the Local Group Policy. If you're lucky enough to run a version of Windows that enables it, you might be aware of the gpedit mmc tool (launch the command gpedit.msc to access it), which can be used to alter various Windows settings, a.k.a. Group Policies. In general, Group Policies are settings that are applied by Active Directory administrators in a networked environment, but the non networked part, called Local Group Policy can also be used by the end users to control the behaviour of their local machine or user account, should the network administrator decide not to override those. Of special interest to us is the Administrative Templates → System → Device Installation → "Prevent creation of a system restore point during device activity..." setting, which you can access after launching gpedit.msc. It works, and is exactly what we have been looking for.

But then again, we don't want to rely on end users to change this setting themselves, especially if they are running the Basic or Home edition of Windows 7 or Vista where gpedit is not available. We want to do it programmatically.

If you use the RegFormApp tool from NirSoft, to monitor what gpedit actually does when enabling this setting, you find that it simply creates a new DisableSystemRestore DWORD in the registry at:
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Group Policy Objects\{<some-guid>}Machine\Software\Policies\Microsoft\Windows\DeviceInstall\Settings

Brilliant. The much decried Registry might actually prove its worth, if all you have to do to toggle system restore point creation on driver installation, is modify a key there... except, we're talking about a Microsoft way of implementing a solution, and it would be way too simple if Windows' own Local Group Policy settings were stored in the registry. The key created by gpedit there is in fact useless for the programmer, because it's just a some temporary storage for gpedit and not at all where the system looks when deciding whether it should create a restore point. As a matter of fact, the GUID changes every time you run gpedit.

So now we need to figure out a way to set our Local Group Policy programmatically, as gpedit does, and you can trust that it's not going to be as simple as what we were hoping for. Thankfully, if you look hard enough, you may end up on the Microsoft's USGCB Tech Blog page dealing with Utilities for automating Local Group Policy management. These tools are precisely the kind of sample we are looking for, as the ImportRegPol there can pick up a .pol file and apply the LPGO associated with it, and they come with the source.

To cut a long story short, you will need to use an IGroupPolicyObject Interface object, and below is the C source (with no error checking whatsoever) that one can use to disable the creation of a restore point during driver installation on Windows Vista and Windows 7:

#include <gpedit.h>

DWORD val, val_size=sizeof(DWORD);
HRESULT hr;
IGroupPolicyObject* pLGPO;
HKEY machine_key, dsrkey;
// MSVC is finicky about these ones => redefine them
const IID my_IID_IGroupPolicyObject = 
 { 0xea502723, 0xa23d, 0x11d1, {0xa7, 0xd3, 0x0, 0x0, 0xf8, 0x75, 0x71, 0xe3} };
const IID my_CLSID_GroupPolicyObject = 
 { 0xea502722, 0xa23d, 0x11d1, {0xa7, 0xd3, 0x0, 0x0, 0xf8, 0x75, 0x71, 0xe3} };
GUID ext_guid = REGISTRY_EXTENSION_GUID;
// This next one can be any GUID you want
GUID snap_guid = { 0x3d271cfc, 0x2bc6, 0x4ac2, {0xb6, 0x33, 0x3b, 0xdf, 0xf5, 0xbd, 0xab, 0x2a} };

// Create an instance of the IGroupPolicyObject class
CoInitializeEx(NULL, COINIT_APARTMENTTHREADED);
CoCreateInstance(&my_CLSID_GroupPolicyObject, NULL, CLSCTX_INPROC_SERVER,
 &my_IID_IGroupPolicyObject, (LPVOID*)&pLGPO);

// We need the machine LGPO (if C++, no need to go through the lpVtbl table)
pLGPO->lpVtbl->OpenLocalMachineGPO(pLGPO, GPO_OPEN_LOAD_REGISTRY);
pLGPO->lpVtbl->GetRegistryKey(pLGPO, GPO_SECTION_MACHINE, &machine_key);

// The disable System Restore is a DWORD value of Policies\Microsoft\Windows\DeviceInstall\Settings
RegCreateKeyEx(machine_key, "Software\\Policies\\Microsoft\\Windows\\DeviceInstall\\Settings",
 0, NULL, 0, KEY_SET_VALUE | KEY_QUERY_VALUE, NULL, &dsrkey, NULL);
 
// Create the value
val = 1;
RegSetKeyValue(dsrkey, NULL, "DisableSystemRestore", REG_DWORD, &val, sizeof(val));
RegCloseKey(dsrkey);

// Apply policy and free resources
pLGPO->lpVtbl->Save(pLGPO, TRUE, TRUE, &ext_guid, &snap_guid);
RegCloseKey(machine_key);
pLGPO->lpVtbl->Release(pLGPO);

Disabling of system restore during driver installation - coming soon to a libwdi powered application near you!

Windows backup and MediaID.bin

Whoever thought it would be a good idea to make Windows backup dependent on a small MediaId.bin file residing on the backup target, outside of the actual backup directory, oughta get fired.

So, let's say you're using a samba drive as your backup destination, where there already exists a bunch of directories. Windows will create its own WindowsImageBackup directory in there, and that's fine.

Then, a few month later, you're cleaning up the root of your backup directory, and seeing this small MediaID.bin, which doesn't seem to serve any purpose (if it's that small, it can't be that important - plus all it really contains is a 16 bytes payload).

Bad move: now NONE of your backup archives will work.

If you need a unique ID to identify that a backup file is legitimate, how about you add it to the backup file itself instead of creating an extra MediaID.bin, outside of the backup directory, that screams "delete me"?

Virtual Store, Windows 7 and HyperTerminal

Let's do things in order then.

1. In Windows 7 (I think it started with Vista), Microsoft locked down the ability for executables to write into system directories like C:\Windows, C:\Program Files, etc., unless they run as admin. This is probably a good thing.
   
2. However, backward compatibility tells us that there are plenty of programs that want to store settings or write files into their installation directory, which is most likely located in C:\Program Files (x86)
   
3. To avert this issue, Microsoft created the Virtual Store directory, which is located in C:\Users\myuser\AppData\Local\VirtualStore\ which the system uses for write access to restricted directories by overlaying it into system calls. For instance, let's say a copy of XP's HyperTerminal is installed on Windows 7 x64 into C:\Program Files (x86)\HyperTerminal\ with an HT session file called mysession.ht residing in the same directory, with this session file having been previously created and installed into C:\Program Files (x86)\HyperTerminal\ as admin. Having been developed up to XP (and yanked on later versions - more about this later), HyperTerminal saw no qualms about updating session files that reside in C:\Program Files (x86)\HyperTerminal\. And HyperTerminal needs to update your session file, whenever you exit the program, to update the terminal buffer log and other things, so it needs to write the session file every time it is used. What transparently happens the first time HT tries to update the session file is:
   
   1. An exact duplicate of mysession.ht is created as C:\Users\myuser\AppData\Local\VirtualStore\Program Files (x86)\HyperTerminal\mysession.ht, to which "myuser" has R/W access.
   2. On subsequent access from HT, this is the file that is being read and written, the original file from C:\Program Files (x86)\HyperTerminal\mysession.ht being left completely untouched from how it was when first copied there.
   In essence this means that there exists differing duplicates of the same file on your system, which, if you are using programs that have not been updated for Win7/Vista, is something you want to be aware of. For instance, in case you spent a lot of time creating conf/settings files in an application, and thought that a backup of the C:\Program Files (x86)\Some App\ would be enough, you might want to reconsider. This can be quite important if you want to back up application settings (IDA Pro comes to mind) as any app that was designed to store data in C:\Program Files\ will effectively store it in the Virtual Store, and simply backing up the app dir won't be enough.
   
4. What does the above has to do with HyperTerminal then? Well, HyperTerminal was yanked from Vista and Win7, which is bad news if you're working with embedded devices, and say, want to transfer data using a serial console through X/Y/Zmodem... The thing is, the HyperTerminal files from XP work just fine on Win7 (even on Win7 64 bit), but they have a few limitations:
   • if you place your HT session file into a different directory than the one where HT is installed, you will have trouble opening it through a shortcut, and will get prompted about creating or opening a session file every time you run HT. Of course, this constant nagging about creating session files gets old really fast.
   • you cannot change the default startup directory to something you have R/W access to without HT getting into other issues.
   • if you receive data through X\Y\Zmodem and didn't change the directory, it'll end up in the Virtual Store by default, which is why you might want to know where exactly that directory is located.

As a bonus, if you have an XP machine available, here are the files you want to copy to reinstate HyperTerminal in Windows Vista/7:

• C:\Program Files\Windows NT\hypertrm.exe
• C:\Program Files\Windows NT\htrn_jis.dll
• C:\Windows\System32\hypertrm.dll
• C:\Windows\System32\hticons.dll

Copy all those into  C:\Program Files (x86)\HyperTerminal\ and you should be good. And yes, one could solve the Virtual Store problem by installing all these files somewhere other than Program Files, but some people might want to keep their system organized.

Where does Windows device enumeration actually begin?

That's in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum.

i.e., the PCI\ USB\ HID\ branches of a Device Instance ID path all originate from ControlSet\Enum.

Preventing USB devices with different serials to be seen as separate devices

Once again, I have to give props to Xiaofan:

The following is quoted from a USB-IF forum post (the link is now invalid)
http://www.usb.org/phpbb/viewtopic.php?p=33850

"Win2000/XP has a way to ignore the serial number.
This is copied form an AppNote I wrote:
-------------------------------------------------------
Ignore serial number in Win2000 / XP In Windows 2000 and XP,
there is a way to ignore the serial number for a certain device.
This is not recommended in general, but there are reasons to do so.

This is the registry key
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\UsbFlags]
The global way: The entry below does enable the serial numbers despite
its name! It is already there after a fresh installation.
GlobalDisableSerNumGen = 1
We recommend keeping the above setting unchanged.

Now here is the individual way. Create an entry under the
above ...\UsbFlags key. The name must start with "IgnoreHWSerNum"
followed by the vendor and product ID of the device. The value for the
entry is 1 for disable.
Example:
IgnoreHWSerNum0536016A= 1
Our vendor ID (VID) is 0536 (hex), the product ID (PID) depends
on the interface you choose.

Showing hidden devices on Windows

Courtesy of the good people of the libusb-devel mailing list:

1. Set a System (not user) environment variable called:
   DevMgr_Show_NonPresent_Devices with value 1
   
2. Tell device mgr (thru menus) to show hidden devices

See http://www.ftdichip.com/Documents/AppNotes/AN_107_AdvancedDriverOptions_AN_000073.pdf

This trick seems to work fine on Windows 7 as well.

Launching the device manager in one click

A few of us need to have quick access to the device manager always, and while things have improved on Windows 7, we're still not there with regards to one click access. The solution:

C:\Windows\system32\mmc.exe C:\Windows\system32\devmgmt.msc

Windows UAC and bash's "Bad File Number" error

Today's problem: trying to run a freshly compiled setup_driver.exe from MSYS/MinGW produces the following:

sh: ./setup_driver.exe: Bad file number

The reason: Microsoft's implementation of UAC will try to give a free pass to any application that has "setup" in its name, and run it in elevated mode (after asking for confirmation).
This doesn't fare well with most shells running on Windows, as they don't know how to call the UAC elevation process, and you get the error above.

Rename your application to something that does not contain "setup" and your problem is gone.
Also applies to variations of "install" in your name.