Filtering dat files

If you read this blog or the CE newsgroups you should by now know you should never change any files in the PUBLIC or PRIVATE folders of your WINCE installation tree. So what if you want to change a setting in common.reg (which is located in the PUBLIC folder)? Well, add the same entry to your platform.reg (in your PLATFORM\BSP folder) and set it to whatever you like. So, what if you want to delete a setting from common.reg? Again, add the same entry to your platform.reg but precede the key name with a "-". This works because platform.reg is merged into reginit.ini after common.reg, so any duplicate entries or deletion commands (the "-") will override the settings in common.reg.

So, what about dat files? Unfortunately, deleting dat entries with the "-" tag is not supported. Dat files are responsible for the initial copy of files and creation of folders during boot. For instance, dat files are used to copy shortcuts from the \Windows folder to the desktop of your CE device. Now what if you don't want to have icons on your desktop? The only way that seems possible is to actually change the public dat files in place, but you know you should never change any files in the PUBLIC tree! We just have to come up with a different solution...

The solution in this case is to filter the dat files and remove any unwanted lines before creating the final image but after the sysgen phase and without touching the original files in PUBLIC. Windows CE calls out to a number of batch files during the makeimg process to let you write your own scripts to affect the image at different points during the image building process. One of those batch files is "PostFmergeObj.bat" which, as the name suggests, is called just after fmerge merged the "Obj" files (which are in fact "Dat" files... confused already? ;o) into your FLATRELEASEDIR.

By using a bit of scripting magic we can use this batch file to filter out unwanted lines from the final merged InitObj.dat file:

Update:Line 18 is required to get this to work in WEC7/WEC2013

  1. REM   This batch file filters the DAT files
  2. REM   CE merges all DAT files into initobj.tmp
  3. REM   and then transforms initobj.tmp into a
  4. REM   UNICODE version initobj.dat.
  5. REM   Just before the conversion to unicode
  6. REM   the build system calls PostFmergeObj.bat
  7. REM   (this file). We filter out all the
  8. REM   strings defined in PostFmergeObj.txt
  9. REM   from initobj.tmp
  10.  
  11. @echo off
  12. echo PostFmergeObj.bat entry.
  13. pushd %_FLATRELEASEDIR%
  14. echo Deleting root:- entries from initobj.tmp
  15. del initobj.org
  16. ren initobj.tmp initobj.org
  17. findstr /i /v /g:PostFmergeObj.txt initobj.org > initobj.tmp
  18. txt2ucde initobj.tmp initobj.dat
  19. popd
  20. echo PostFmergeObj.bat exit.
  21. @echo on

The core of the script is from line 15 onwards. In line 15 we delete any previous initobj.org file this batch file may have left behind. Then, in line 16, we rename initobj.tmp (all dat files are merged into this file by fmerge) to initobj.org and in line 17 we call the dos utility Findstr:

>findstr /?
Searches for strings in files.

FINDSTR [/B] [/E] [/L] [/R] [/S] [/I] [/X] [/V] [/N] [/M] [/O] [/P] [/F:file]
        [/C:string] [/G:file] [/D:dir list] [/A:color attributes] [/OFF[LINE]]
        strings [[drive:][path]filename[ ...]]

  /B         Matches pattern if at the beginning of a line.
  /E         Matches pattern if at the end of a line.
  /L         Uses search strings literally.
  /R         Uses search strings as regular expressions.
  /S         Searches for matching files in the current directory and all
             subdirectories.
  /I         Specifies that the search is not to be case-sensitive.
  /X         Prints lines that match exactly.
  /V         Prints only lines that do not contain a match.
  /N         Prints the line number before each line that matches.
  /M         Prints only the filename if a file contains a match.
  /O         Prints character offset before each matching line.
  /P         Skip files with non-printable characters.
  /OFF[LINE] Do not skip files with offline attribute set.
  /A:attr    Specifies color attribute with two hex digits. See "color /?"
  /F:file    Reads file list from the specified file(/ stands for console).
  /C:string  Uses specified string as a literal search string.
  /G:file    Gets search strings from the specified file(/ stands for console).
  /D:dir     Search a semicolon delimited list of directories
  strings    Text to be searched for.
  [drive:][path]filename
             Specifies a file or files to search.

Use spaces to separate multiple search strings unless the argument is prefixed
with /C.  For example, 'FINDSTR "hello there" x.y' searches for "hello" or
"there" in file x.y.  'FINDSTR /C:"hello there" x.y' searches for
"hello there" in file x.y.

Regular expression quick reference:
  .        Wildcard: any character
  *        Repeat: zero or more occurances of previous character or class
  ^        Line position: beginning of line
  $        Line position: end of line
  [class]  Character class: any one character in set
  [^class] Inverse class: any one character not in set
  [x-y]    Range: any characters within the specified range
  \x       Escape: literal use of metacharacter x
  \<xyz    Word position: beginning of word
  xyz\>    Word position: end of word

For full information on FINDSTR regular expressions refer to the online Command
Reference.

By calling findstr with options /I and /V we effectively filter out the lines we specify in PostFmergeObj.txt and we pipe the output of findstr into InitObj.tmp. After this batch file returns the build system will call the "Text to Unicode" tool (Txt2ucde.exe) that converts any ASCII text strings in Initobj.tmp to Unicode text strings and creates the file Initobj.dat which then contains the Unicode conversion of InitObj.tmp (see http://msdn.microsoft.com/en-us/library/ms930234.aspx).

Note that you can also use regular expressions to find strings with findstr. Using this you have a very powerful tool to filter whatever file you want filtered and of course this trick doesn't only work on dat files, it works on any ascii text file (like bib, reg, etc).

All you have to do for all this to start working in your build is create PostFmergeObj.bat in your PLATFORM\BSP\FILES folder and copy'n'paste the above script. The Windows CE build system will automatically search for this batch file and if it finds it execute it just after it has merged the various configuration files. Don't forget to create a "PostFmergeObj.txt" file with the lines you want to delete from the final InitObj.dat file in the same folder as the batch file, like this:

root:-Directory("\Windows"):-Directory("www")
Directory("\Windows\www"):-Directory("wwwpub")
root:-Directory("Program Files")
root:-Directory("My Documents")
root:-File("Control Panel.lnk", "\Windows\control.lnk")
Directory("\Windows"):-Directory("Desktop")
Directory("\Windows"):-Directory("Programs")
Directory("\Windows"):-Directory("Recent")
Directory("\Program Files"):-File("Command Prompt.lnk","\Windows\cmd.lnk")
Directory("\Windows\Programs"):-File("Windows Explorer.lnk", "\Windows\explore.lnk")
Directory("\Windows\Programs"):-File("Command Prompt.lnk","\Windows\cmd.lnk")
Directory("\Program Files"):-File("Command Prompt.lnk","\Windows\cmd.lnk")

When you now perform a "make image" this batch file should automatically execute and its output (the "echo" commands in the batch file) should appear in the build output window.

Comments

Thanks,
I was just discussing this subject with someone I work with, and this fills in all the missing information . It would have taken us several days to put it together - so this is a time save.

Shai

That is very good to hear! Glad to be of help.

HI,
I want to use the /g option to extract a couple of things from ini files, basically: the content of text file given as argument to the /g option looks like this:

"DestinationDirectory="
"[*$*.???]"

however, when I issue the command:

findstr /n /i /s /g:finddata.txt *.ini

it hangs there forever :-( Even I added the .out file to send the results, but that did not work. Any suggestions what am I doing wrong? Regards,

If you want to use regular expressions, you've got to add the /r parameter.

Very nice article Michel. This is going to help a lot of people out.

Note that this technique can be used for other files as well, like .db files.

I have taken what you have so graciously provided and used it as a simple and efficient way of clearing off the Desktop.

All of the shortcuts that are typically put on the Desktop are removed, producing the desired "Clean Desktop" outcome,
all without the taboo of editing any of the PUBLIC DAT files.

NOTE: The following method will snag any shortcuts that the PLATFORM may attempt to put on the desktop too.

I only made minor changes to one line in PostFmergeObj.bat (added /L and /C options):

findstr /I /L /V /C /G:PostFmergeObj.txt initobj.org > initobj.tmp

I used the following string in my PostFmergeObj.txt:

"\Windows\Desktop"

And the final tweak I made is...

Rather than put the PostFmergeObj files in your PLATFORM\BSP\FILES folder as directed...
I chose to put the PostFmergeObj files in my OS Design in "OAK\files" under:

_PROJPUBLICROOT (e.g. \WINCE600\OSDesigns\<OSDesignName>\<OSDesignName>\WINCE600\PUBLIC)

so that one set of files can be used by all platforms of my OS Design.

To utilize the "shared files for all platforms of an OS Design" approach I added the following Custom Build Actions section to each platform in my pbxml OS Design file:

      <CustomBuildActions Name="CustomBuildActions">
        <BuildStep Step="PreSysgen">
          <StringCollectionEX Name="Action">
            <String>echo In PreSysgen CustomBuildActions</String>
            <String>echo ALL Platforms for this OS Design use the same Project files...</String>
            <String>echo copy /Y %_PROJPUBLICROOT%\OAK\files\Project.* %_PROJECTOAKROOT%\files</String>
            <String>copy /Y %_PROJPUBLICROOT%\OAK\files\Project.* %_PROJECTOAKROOT%\files</String>
            <String>echo Out PreSysgen CustomBuildActions</String>
          </StringCollectionEX>
        </BuildStep>
        <BuildStep Step="PreMakeImage">
          <StringCollectionEX Name="Action">
            <String>echo In PreMakeImage CustomBuildActions</String>
            <String>copy /Y %_PROJPUBLICROOT%\OAK\files\pegconn.wav %_FLATRELEASEDIR%\infbeg.wav</String>
            <String>copy /Y %_PROJPUBLICROOT%\OAK\files\pegdisc.wav %_FLATRELEASEDIR%\infend.wav</String>
            <String>copy /Y %_PROJPUBLICROOT%\OAK\files\pegdisc.wav %_FLATRELEASEDIR%\infintr.wav</String>
            <String>copy /Y %_PROJPUBLICROOT%\OAK\files\*.dll %_FLATRELEASEDIR%</String>
            <String>copy /Y %_PROJPUBLICROOT%\OAK\files\*.bib %_FLATRELEASEDIR%</String>
            <String>copy /Y %_PROJPUBLICROOT%\OAK\files\*.bmp %_FLATRELEASEDIR%</String>
            <String>copy /Y %_PROJPUBLICROOT%\OAK\files\*.reg %_FLATRELEASEDIR%</String>
            <String>copy /Y %_PROJPUBLICROOT%\OAK\files\*.wav %_FLATRELEASEDIR%</String>
            <String>copy /Y %_PROJPUBLICROOT%\OAK\files\PostFmergeObj.* %_FLATRELEASEDIR%</String>
            <String>echo Out PreMakeImage CustomBuildActions</String>
          </StringCollectionEX>
        </BuildStep>
      </CustomBuildActions>

Adding the PreSysgen CustomBuildActions listed above allows me to also use shared project.bib project.dat and project.reg
files across all PLATFORMS in my OS Design as well.

Thank you for your addition!

seems findstr does not support Japanese characters

Great content - thanks a lot